2024-09-11T01:18:56.560 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-11T01:18:56.578 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T01:18:56.669 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7899900 branch: wip-hemanth-testing-2024-09-10-0723 description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest} thrashers/one workloads/snaps-few-objects} email: hyelloji@redhat.com first_in_suite: false job_id: '7899900' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-hemanth-testing-2024-09-10-0723 ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos service trim min: 5 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore write v2 random: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-hemanth-testing-2024-09-10-0723 sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 owner: scheduled_hyelloji@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 9167 sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-hemanth-testing-2024-09-10-0723 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 targets: smithi012.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKrxytsmSSWI6nG56dznjsc7P5wsT+aWf19CmdWPPjQvnFGVtRI5KzbHYneuAfFwoeT0z6dPRmlrSVwr7N0BbyM= smithi071.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD/5BxFyq5H567Msv+LIeSsl8qqupOIJFr3bEQaRpg7Mo80GuJRdpGbPh94XT92tRK3LA2yIr6v1sWfpfsRZiJs= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 20 thrash_delay: 1 - rados: clients: - client.0 objects: 50 op_weights: copy_from: 50 delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_17:16:14 tube: smithi user: hyelloji verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-11T01:18:56.669 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa; will attempt to use it 2024-09-11T01:18:56.670 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa/tasks 2024-09-11T01:18:56.670 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-11T01:18:56.672 INFO:teuthology.task.internal:Checking packages... 2024-09-11T01:18:56.693 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8f219c476cb4f26fc5719c661c0323719a45cb30' 2024-09-11T01:18:56.693 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T01:18:56.693 INFO:teuthology.packaging:ref: None 2024-09-11T01:18:56.693 INFO:teuthology.packaging:tag: None 2024-09-11T01:18:56.693 INFO:teuthology.packaging:branch: wip-hemanth-testing-2024-09-10-0723 2024-09-11T01:18:56.693 INFO:teuthology.packaging:sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T01:18:56.693 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-hemanth-testing-2024-09-10-0723 2024-09-11T01:18:56.849 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4854.g8f219c47 2024-09-11T01:18:56.850 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-11T01:18:56.857 INFO:teuthology.task.internal:no buildpackages task found 2024-09-11T01:18:56.857 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-11T01:18:56.865 INFO:teuthology.task.internal:Saving configuration 2024-09-11T01:18:56.878 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-11T01:18:56.885 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-11T01:18:56.916 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7899900', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 01:14:05.368525', 'locked_by': 'scheduled_hyelloji@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKrxytsmSSWI6nG56dznjsc7P5wsT+aWf19CmdWPPjQvnFGVtRI5KzbHYneuAfFwoeT0z6dPRmlrSVwr7N0BbyM='} 2024-09-11T01:18:56.947 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7899900', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 01:14:05.370277', 'locked_by': 'scheduled_hyelloji@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD/5BxFyq5H567Msv+LIeSsl8qqupOIJFr3bEQaRpg7Mo80GuJRdpGbPh94XT92tRK3LA2yIr6v1sWfpfsRZiJs='} 2024-09-11T01:18:56.947 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-11T01:18:56.953 INFO:teuthology.task.internal:roles: ubuntu@smithi012.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-11T01:18:56.954 INFO:teuthology.task.internal:roles: ubuntu@smithi071.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-11T01:18:56.954 INFO:teuthology.run_tasks:Running task console_log... 2024-09-11T01:18:57.027 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1693223be0>, signals=[15]) 2024-09-11T01:18:57.027 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-11T01:18:57.034 INFO:teuthology.task.internal:Opening connections... 2024-09-11T01:18:57.034 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi012.front.sepia.ceph.com 2024-09-11T01:18:57.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T01:18:57.119 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi071.front.sepia.ceph.com 2024-09-11T01:18:57.119 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T01:18:57.220 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-11T01:18:57.227 DEBUG:teuthology.orchestra.run.smithi012:> uname -m 2024-09-11T01:18:57.245 INFO:teuthology.orchestra.run.smithi012.stdout:x86_64 2024-09-11T01:18:57.245 DEBUG:teuthology.orchestra.run.smithi012:> cat /etc/os-release 2024-09-11T01:18:57.302 INFO:teuthology.orchestra.run.smithi012.stdout:NAME="CentOS Stream" 2024-09-11T01:18:57.302 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION="9" 2024-09-11T01:18:57.302 INFO:teuthology.orchestra.run.smithi012.stdout:ID="centos" 2024-09-11T01:18:57.302 INFO:teuthology.orchestra.run.smithi012.stdout:ID_LIKE="rhel fedora" 2024-09-11T01:18:57.302 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_ID="9" 2024-09-11T01:18:57.302 INFO:teuthology.orchestra.run.smithi012.stdout:PLATFORM_ID="platform:el9" 2024-09-11T01:18:57.302 INFO:teuthology.orchestra.run.smithi012.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-11T01:18:57.302 INFO:teuthology.orchestra.run.smithi012.stdout:ANSI_COLOR="0;31" 2024-09-11T01:18:57.303 INFO:teuthology.orchestra.run.smithi012.stdout:LOGO="fedora-logo-icon" 2024-09-11T01:18:57.303 INFO:teuthology.orchestra.run.smithi012.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-11T01:18:57.303 INFO:teuthology.orchestra.run.smithi012.stdout:HOME_URL="https://centos.org/" 2024-09-11T01:18:57.303 INFO:teuthology.orchestra.run.smithi012.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-11T01:18:57.303 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-11T01:18:57.303 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-11T01:18:57.303 INFO:teuthology.lock.ops:Updating smithi012.front.sepia.ceph.com on lock server 2024-09-11T01:18:57.330 DEBUG:teuthology.orchestra.run.smithi071:> uname -m 2024-09-11T01:18:57.348 INFO:teuthology.orchestra.run.smithi071.stdout:x86_64 2024-09-11T01:18:57.348 DEBUG:teuthology.orchestra.run.smithi071:> cat /etc/os-release 2024-09-11T01:18:57.407 INFO:teuthology.orchestra.run.smithi071.stdout:NAME="CentOS Stream" 2024-09-11T01:18:57.407 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION="9" 2024-09-11T01:18:57.407 INFO:teuthology.orchestra.run.smithi071.stdout:ID="centos" 2024-09-11T01:18:57.407 INFO:teuthology.orchestra.run.smithi071.stdout:ID_LIKE="rhel fedora" 2024-09-11T01:18:57.407 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_ID="9" 2024-09-11T01:18:57.407 INFO:teuthology.orchestra.run.smithi071.stdout:PLATFORM_ID="platform:el9" 2024-09-11T01:18:57.407 INFO:teuthology.orchestra.run.smithi071.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-11T01:18:57.407 INFO:teuthology.orchestra.run.smithi071.stdout:ANSI_COLOR="0;31" 2024-09-11T01:18:57.407 INFO:teuthology.orchestra.run.smithi071.stdout:LOGO="fedora-logo-icon" 2024-09-11T01:18:57.407 INFO:teuthology.orchestra.run.smithi071.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-11T01:18:57.407 INFO:teuthology.orchestra.run.smithi071.stdout:HOME_URL="https://centos.org/" 2024-09-11T01:18:57.407 INFO:teuthology.orchestra.run.smithi071.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-11T01:18:57.408 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-11T01:18:57.408 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-11T01:18:57.408 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2024-09-11T01:18:57.438 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-11T01:18:57.447 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-11T01:18:57.453 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-11T01:18:57.454 DEBUG:teuthology.orchestra.run.smithi012:> test '!' -e /home/ubuntu/cephtest 2024-09-11T01:18:57.456 DEBUG:teuthology.orchestra.run.smithi071:> test '!' -e /home/ubuntu/cephtest 2024-09-11T01:18:57.473 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-11T01:18:57.479 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-11T01:18:57.479 DEBUG:teuthology.orchestra.run.smithi012:> test -z $(ls -A /var/lib/ceph) 2024-09-11T01:18:57.514 DEBUG:teuthology.orchestra.run.smithi071:> test -z $(ls -A /var/lib/ceph) 2024-09-11T01:18:57.627 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-11T01:18:57.696 INFO:teuthology.run_tasks:Running task kernel... 2024-09-11T01:18:57.710 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-11T01:18:57.711 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-11T01:18:57.711 DEBUG:teuthology.orchestra.run.smithi012:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T01:18:57.711 DEBUG:teuthology.orchestra.run.smithi071:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T01:18:57.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T01:18:57.728 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2024-09-11T01:18:57.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T01:18:57.729 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2024-09-11T01:18:57.786 INFO:teuthology.orchestra.run.smithi012.stdout:5.14.0-503.el9.x86_64 2024-09-11T01:18:57.786 INFO:teuthology.task.kernel:Running kernel on smithi012: 5.14.0-503.el9.x86_64 2024-09-11T01:18:57.786 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum install -y kernel 2024-09-11T01:18:57.787 INFO:teuthology.orchestra.run.smithi071.stdout:5.14.0-503.el9.x86_64 2024-09-11T01:18:57.788 INFO:teuthology.task.kernel:Running kernel on smithi071: 5.14.0-503.el9.x86_64 2024-09-11T01:18:57.788 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum install -y kernel 2024-09-11T01:19:00.056 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.2 MB 00:00 2024-09-11T01:19:02.741 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - BaseOS 2.6 MB/s | 8.2 MB 00:03 2024-09-11T01:19:03.590 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - AppStream 8.7 MB/s | 20 MB 00:02 2024-09-11T01:19:04.833 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - AppStream 21 MB/s | 20 MB 00:00 2024-09-11T01:19:09.024 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.5 MB 00:01 2024-09-11T01:19:10.633 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 19 kB 00:00 2024-09-11T01:19:11.504 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-11T01:19:13.358 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - CRB 1.4 MB/s | 6.5 MB 00:04 2024-09-11T01:19:14.900 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 19 kB 00:00 2024-09-11T01:19:15.796 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-11T01:19:17.167 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-11T01:19:19.282 INFO:teuthology.orchestra.run.smithi012.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-11T01:19:19.282 INFO:teuthology.orchestra.run.smithi012.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-11T01:19:19.371 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-09-11T01:19:19.382 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:19:19.382 INFO:teuthology.orchestra.run.smithi012.stdout: Package Architecture Version Repository Size 2024-09-11T01:19:19.382 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:19:19.382 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-09-11T01:19:19.382 INFO:teuthology.orchestra.run.smithi012.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-11T01:19:19.382 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-09-11T01:19:19.382 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-11T01:19:19.382 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-11T01:19:19.383 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-11T01:19:19.383 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:19:19.383 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-09-11T01:19:19.383 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:19:19.383 INFO:teuthology.orchestra.run.smithi012.stdout:Install 4 Packages 2024-09-11T01:19:19.383 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:19:19.384 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 79 M 2024-09-11T01:19:19.384 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 126 M 2024-09-11T01:19:19.384 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-09-11T01:19:19.519 INFO:teuthology.orchestra.run.smithi012.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 255 kB/s | 23 kB 00:00 2024-09-11T01:19:20.035 INFO:teuthology.orchestra.run.smithi012.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-09-11T01:19:20.352 INFO:teuthology.orchestra.run.smithi012.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 37 MB/s | 35 MB 00:00 2024-09-11T01:19:20.944 INFO:teuthology.orchestra.run.smithi012.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 20 MB/s | 29 MB 00:01 2024-09-11T01:19:20.944 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:19:20.944 INFO:teuthology.orchestra.run.smithi012.stdout:Total 50 MB/s | 79 MB 00:01 2024-09-11T01:19:21.252 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-09-11T01:19:21.660 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-09-11T01:19:22.060 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-09-11T01:19:22.060 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-09-11T01:19:23.056 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-09-11T01:19:23.056 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-09-11T01:19:23.275 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-11T01:19:23.275 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-11T01:19:23.383 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-11T01:19:23.387 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:19:23.387 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-09-11T01:19:23.387 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:19:23.387 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-11T01:19:23.387 INFO:teuthology.orchestra.run.smithi071.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-11T01:19:23.387 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-09-11T01:19:23.387 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-11T01:19:23.387 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-11T01:19:23.388 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-11T01:19:23.388 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:19:23.388 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-11T01:19:23.388 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:19:23.388 INFO:teuthology.orchestra.run.smithi071.stdout:Install 4 Packages 2024-09-11T01:19:23.388 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:19:23.389 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 79 M 2024-09-11T01:19:23.389 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 126 M 2024-09-11T01:19:23.389 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-11T01:19:24.211 INFO:teuthology.orchestra.run.smithi071.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 42 kB/s | 23 kB 00:00 2024-09-11T01:19:25.620 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-09-11T01:19:26.304 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-11T01:19:26.386 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T01:19:27.628 INFO:teuthology.orchestra.run.smithi071.stdout:(2/4): kernel-modules-core-5.14.0-505.el9.x86_6 8.4 MB/s | 29 MB 00:03 2024-09-11T01:19:28.386 INFO:teuthology.orchestra.run.smithi071.stdout:(3/4): kernel-core-5.14.0-505.el9.x86_64.rpm 3.3 MB/s | 16 MB 00:04 2024-09-11T01:19:28.920 INFO:teuthology.orchestra.run.smithi071.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 6.6 MB/s | 35 MB 00:05 2024-09-11T01:19:28.920 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:19:28.920 INFO:teuthology.orchestra.run.smithi071.stdout:Total 14 MB/s | 79 MB 00:05 2024-09-11T01:19:29.186 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T01:19:29.316 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T01:19:29.605 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-11T01:19:29.968 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-11T01:19:29.968 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-11T01:19:30.933 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-11T01:19:30.933 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-11T01:19:33.412 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-11T01:19:34.130 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-11T01:19:34.222 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T01:19:34.441 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T01:19:34.483 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T01:19:37.031 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T01:19:37.168 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T01:19:39.655 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T01:19:41.993 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T01:19:42.110 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T01:19:46.958 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T01:20:18.885 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T01:20:18.926 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-11T01:20:19.410 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T01:20:19.411 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-11T01:20:19.411 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T01:20:19.411 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T01:20:20.243 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T01:20:20.244 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:20:20.244 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-09-11T01:20:20.244 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-11T01:20:20.244 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-11T01:20:20.244 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-11T01:20:20.244 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-11T01:20:20.244 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:20:20.244 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-09-11T01:20:20.636 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T01:20:20.714 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T01:20:20.714 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T01:20:20.714 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-11T01:20:20.714 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T01:20:20.714 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T01:20:20.714 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-11T01:20:20.715 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi012.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T01:20:20.715 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T01:20:20.773 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T01:20:20.773 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T01:20:20.774 DEBUG:teuthology.orchestra.run.smithi012:> sudo rpm -qi grub2-tools 2024-09-11T01:20:20.813 INFO:teuthology.orchestra.run.smithi012.stdout:Name : grub2-tools 2024-09-11T01:20:20.813 INFO:teuthology.orchestra.run.smithi012.stdout:Epoch : 1 2024-09-11T01:20:20.813 INFO:teuthology.orchestra.run.smithi012.stdout:Version : 2.06 2024-09-11T01:20:20.813 INFO:teuthology.orchestra.run.smithi012.stdout:Release : 82.el9 2024-09-11T01:20:20.813 INFO:teuthology.orchestra.run.smithi012.stdout:Architecture: x86_64 2024-09-11T01:20:20.813 INFO:teuthology.orchestra.run.smithi012.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-11T01:20:20.813 INFO:teuthology.orchestra.run.smithi012.stdout:Group : Unspecified 2024-09-11T01:20:20.813 INFO:teuthology.orchestra.run.smithi012.stdout:Size : 8274670 2024-09-11T01:20:20.813 INFO:teuthology.orchestra.run.smithi012.stdout:License : GPLv3+ 2024-09-11T01:20:20.813 INFO:teuthology.orchestra.run.smithi012.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-11T01:20:20.814 INFO:teuthology.orchestra.run.smithi012.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-11T01:20:20.814 INFO:teuthology.orchestra.run.smithi012.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-11T01:20:20.814 INFO:teuthology.orchestra.run.smithi012.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-11T01:20:20.814 INFO:teuthology.orchestra.run.smithi012.stdout:Packager : builder@centos.org 2024-09-11T01:20:20.814 INFO:teuthology.orchestra.run.smithi012.stdout:Vendor : CentOS 2024-09-11T01:20:20.814 INFO:teuthology.orchestra.run.smithi012.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-11T01:20:20.814 INFO:teuthology.orchestra.run.smithi012.stdout:Summary : Support tools for GRUB. 2024-09-11T01:20:20.814 INFO:teuthology.orchestra.run.smithi012.stdout:Description : 2024-09-11T01:20:20.814 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:20:20.814 INFO:teuthology.orchestra.run.smithi012.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-11T01:20:20.814 INFO:teuthology.orchestra.run.smithi012.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-11T01:20:20.814 INFO:teuthology.orchestra.run.smithi012.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-11T01:20:20.814 INFO:teuthology.orchestra.run.smithi012.stdout:hardware devices. 2024-09-11T01:20:20.815 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:20:20.815 INFO:teuthology.orchestra.run.smithi012.stdout:This subpackage provides tools for support of all platforms. 2024-09-11T01:20:20.815 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-11T01:20:20.816 INFO:teuthology.task.kernel:Updating grub on smithi012 to boot 5.14.0-505.el9.x86_64 2024-09-11T01:20:20.816 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-11T01:20:21.434 INFO:teuthology.orchestra.run.smithi012.stderr:Generating grub configuration file ... 2024-09-11T01:20:22.757 INFO:teuthology.orchestra.run.smithi012.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-11T01:20:22.789 INFO:teuthology.orchestra.run.smithi012.stderr:done 2024-09-11T01:20:22.792 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2024-09-11T01:20:22.807 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.RPLoA8MgSs 2024-09-11T01:20:22.807 DEBUG:teuthology.orchestra.run.smithi012:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.RPLoA8MgSs 2024-09-11T01:20:22.874 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.RPLoA8MgSs 2024-09-11T01:20:23.034 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.RPLoA8MgSs is 6KB 2024-09-11T01:20:23.070 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.RPLoA8MgSs 2024-09-11T01:20:23.087 DEBUG:teuthology.orchestra.run.smithi012:> sudo /bin/ls /boot/loader/entries || true 2024-09-11T01:20:23.164 INFO:teuthology.orchestra.run.smithi012.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-11T01:20:23.164 INFO:teuthology.orchestra.run.smithi012.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-11T01:20:23.165 INFO:teuthology.orchestra.run.smithi012.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-11T01:20:23.165 INFO:teuthology.orchestra.run.smithi012.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-11T01:20:23.165 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-11T01:20:23.315 DEBUG:teuthology.orchestra.run.smithi012:> sudo shutdown -r now 2024-09-11T01:20:25.210 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T01:20:25.251 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-11T01:20:25.633 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T01:20:25.633 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-11T01:20:25.634 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T01:20:25.634 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T01:20:26.486 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T01:20:26.486 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:20:26.486 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-11T01:20:26.487 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-11T01:20:26.487 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-11T01:20:26.487 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-11T01:20:26.487 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-11T01:20:26.487 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:20:26.487 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-11T01:20:26.820 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T01:20:26.908 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T01:20:26.908 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T01:20:26.908 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-11T01:20:26.908 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T01:20:26.908 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T01:20:26.908 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-11T01:20:26.909 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi071.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T01:20:26.909 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T01:20:26.967 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T01:20:26.967 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T01:20:26.967 DEBUG:teuthology.orchestra.run.smithi071:> sudo rpm -qi grub2-tools 2024-09-11T01:20:27.009 INFO:teuthology.orchestra.run.smithi071.stdout:Name : grub2-tools 2024-09-11T01:20:27.009 INFO:teuthology.orchestra.run.smithi071.stdout:Epoch : 1 2024-09-11T01:20:27.010 INFO:teuthology.orchestra.run.smithi071.stdout:Version : 2.06 2024-09-11T01:20:27.010 INFO:teuthology.orchestra.run.smithi071.stdout:Release : 82.el9 2024-09-11T01:20:27.010 INFO:teuthology.orchestra.run.smithi071.stdout:Architecture: x86_64 2024-09-11T01:20:27.010 INFO:teuthology.orchestra.run.smithi071.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-11T01:20:27.010 INFO:teuthology.orchestra.run.smithi071.stdout:Group : Unspecified 2024-09-11T01:20:27.010 INFO:teuthology.orchestra.run.smithi071.stdout:Size : 8274670 2024-09-11T01:20:27.010 INFO:teuthology.orchestra.run.smithi071.stdout:License : GPLv3+ 2024-09-11T01:20:27.010 INFO:teuthology.orchestra.run.smithi071.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-11T01:20:27.010 INFO:teuthology.orchestra.run.smithi071.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-11T01:20:27.010 INFO:teuthology.orchestra.run.smithi071.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-11T01:20:27.010 INFO:teuthology.orchestra.run.smithi071.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-11T01:20:27.010 INFO:teuthology.orchestra.run.smithi071.stdout:Packager : builder@centos.org 2024-09-11T01:20:27.010 INFO:teuthology.orchestra.run.smithi071.stdout:Vendor : CentOS 2024-09-11T01:20:27.011 INFO:teuthology.orchestra.run.smithi071.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-11T01:20:27.011 INFO:teuthology.orchestra.run.smithi071.stdout:Summary : Support tools for GRUB. 2024-09-11T01:20:27.011 INFO:teuthology.orchestra.run.smithi071.stdout:Description : 2024-09-11T01:20:27.011 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:20:27.011 INFO:teuthology.orchestra.run.smithi071.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-11T01:20:27.011 INFO:teuthology.orchestra.run.smithi071.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-11T01:20:27.011 INFO:teuthology.orchestra.run.smithi071.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-11T01:20:27.011 INFO:teuthology.orchestra.run.smithi071.stdout:hardware devices. 2024-09-11T01:20:27.011 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:20:27.011 INFO:teuthology.orchestra.run.smithi071.stdout:This subpackage provides tools for support of all platforms. 2024-09-11T01:20:27.012 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-11T01:20:27.012 INFO:teuthology.task.kernel:Updating grub on smithi071 to boot 5.14.0-505.el9.x86_64 2024-09-11T01:20:27.012 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-11T01:20:27.681 INFO:teuthology.orchestra.run.smithi071.stderr:Generating grub configuration file ... 2024-09-11T01:20:28.907 INFO:teuthology.orchestra.run.smithi071.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-11T01:20:28.936 INFO:teuthology.orchestra.run.smithi071.stderr:done 2024-09-11T01:20:28.943 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2024-09-11T01:20:28.959 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.4i4pO0yxdB 2024-09-11T01:20:28.959 DEBUG:teuthology.orchestra.run.smithi071:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.4i4pO0yxdB 2024-09-11T01:20:29.026 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.4i4pO0yxdB 2024-09-11T01:20:29.163 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.4i4pO0yxdB is 6KB 2024-09-11T01:20:29.213 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.4i4pO0yxdB 2024-09-11T01:20:29.228 DEBUG:teuthology.orchestra.run.smithi071:> sudo /bin/ls /boot/loader/entries || true 2024-09-11T01:20:29.296 INFO:teuthology.orchestra.run.smithi071.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-11T01:20:29.296 INFO:teuthology.orchestra.run.smithi071.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-11T01:20:29.296 INFO:teuthology.orchestra.run.smithi071.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-11T01:20:29.297 INFO:teuthology.orchestra.run.smithi071.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-11T01:20:29.298 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-11T01:20:29.448 DEBUG:teuthology.orchestra.run.smithi071:> sudo shutdown -r now 2024-09-11T01:20:53.341 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-11T01:20:53.342 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-09-11T01:20:53.343 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T01:20:59.457 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-11T01:20:59.457 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-09-11T01:20:59.458 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T01:21:11.741 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2024-09-11T01:21:17.885 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-09-11T01:21:20.745 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-09-11T01:21:20.746 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T01:21:26.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-09-11T01:21:26.892 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T01:21:29.981 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-09-11T01:21:39.133 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2024-09-11T01:21:41.984 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-09-11T01:21:41.985 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T01:21:51.139 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-09-11T01:21:51.140 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T01:21:54.499 DEBUG:teuthology.orchestra.run.smithi012:> true 2024-09-11T01:21:55.044 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-09-11T01:21:55.044 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-11T01:21:55.044 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2024-09-11T01:21:55.059 INFO:teuthology.orchestra.run.smithi012.stdout:5.14.0-505.el9.x86_64 2024-09-11T01:21:55.059 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-11T01:21:55.059 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T01:21:55.060 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-11T01:21:56.060 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-11T01:21:56.060 DEBUG:teuthology.orchestra.run.smithi012:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T01:21:56.161 INFO:teuthology.orchestra.run.smithi012.stdout:ttyS1 2024-09-11T01:21:56.194 DEBUG:teuthology.parallel:result is None 2024-09-11T01:22:00.381 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-09-11T01:22:15.382 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-09-11T01:22:15.383 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T01:22:15.775 DEBUG:teuthology.orchestra.run.smithi071:> true 2024-09-11T01:22:16.196 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-09-11T01:22:16.196 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-11T01:22:16.197 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2024-09-11T01:22:16.211 INFO:teuthology.orchestra.run.smithi071.stdout:5.14.0-505.el9.x86_64 2024-09-11T01:22:16.212 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-11T01:22:16.212 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T01:22:16.212 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-11T01:22:17.213 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-11T01:22:17.213 DEBUG:teuthology.orchestra.run.smithi071:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T01:22:17.336 INFO:teuthology.orchestra.run.smithi071.stdout:ttyS1 2024-09-11T01:22:17.369 DEBUG:teuthology.parallel:result is None 2024-09-11T01:22:17.370 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-11T01:22:17.376 INFO:teuthology.task.internal:Creating test directory... 2024-09-11T01:22:17.377 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T01:22:17.380 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T01:22:17.398 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-11T01:22:17.404 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-11T01:22:17.411 INFO:teuthology.task.internal:Creating archive directory... 2024-09-11T01:22:17.411 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T01:22:17.441 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T01:22:17.476 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-11T01:22:17.483 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-11T01:22:17.483 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T01:22:17.509 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T01:22:17.539 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T01:22:17.550 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T01:22:17.552 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T01:22:17.561 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T01:22:17.563 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-11T01:22:17.570 INFO:teuthology.task.internal:Configuring sudo... 2024-09-11T01:22:17.570 DEBUG:teuthology.orchestra.run.smithi012:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T01:22:17.596 DEBUG:teuthology.orchestra.run.smithi071:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T01:22:17.631 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-11T01:22:17.639 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-11T01:22:17.640 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T01:22:17.666 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T01:22:17.689 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T01:22:17.772 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T01:22:17.815 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T01:22:17.908 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T01:22:17.941 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-09-11T01:22:17.941 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T01:22:18.008 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T01:22:18.058 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T01:22:18.146 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T01:22:18.197 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T01:22:18.233 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-11T01:22:18.233 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T01:22:18.301 DEBUG:teuthology.orchestra.run.smithi012:> sudo service rsyslog restart 2024-09-11T01:22:18.303 DEBUG:teuthology.orchestra.run.smithi071:> sudo service rsyslog restart 2024-09-11T01:22:18.354 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T01:22:18.400 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T01:22:18.952 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-11T01:22:18.960 INFO:teuthology.task.internal:Starting timer... 2024-09-11T01:22:18.961 INFO:teuthology.run_tasks:Running task pcp... 2024-09-11T01:22:18.971 INFO:teuthology.run_tasks:Running task selinux... 2024-09-11T01:22:18.982 DEBUG:teuthology.orchestra.run.smithi012:> sudo service auditd rotate 2024-09-11T01:22:19.031 INFO:teuthology.orchestra.run.smithi012.stdout:Rotating logs: 2024-09-11T01:22:19.033 DEBUG:teuthology.orchestra.run.smithi071:> sudo service auditd rotate 2024-09-11T01:22:19.087 INFO:teuthology.orchestra.run.smithi071.stdout:Rotating logs: 2024-09-11T01:22:19.089 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-11T01:22:19.089 DEBUG:teuthology.orchestra.run.smithi012:> /usr/sbin/getenforce 2024-09-11T01:22:19.125 INFO:teuthology.orchestra.run.smithi012.stdout:Permissive 2024-09-11T01:22:19.125 DEBUG:teuthology.orchestra.run.smithi071:> /usr/sbin/getenforce 2024-09-11T01:22:19.145 INFO:teuthology.orchestra.run.smithi071.stdout:Permissive 2024-09-11T01:22:19.145 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi012.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi071.front.sepia.ceph.com': 'permissive'} 2024-09-11T01:22:19.145 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T01:22:19.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T01:22:19.197 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T01:22:19.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T01:22:19.223 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-11T01:22:19.223 DEBUG:teuthology.orchestra.run.smithi012:> sudo /usr/sbin/setenforce permissive 2024-09-11T01:22:19.272 DEBUG:teuthology.orchestra.run.smithi071:> sudo /usr/sbin/setenforce permissive 2024-09-11T01:22:19.307 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-11T01:22:19.317 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-11T01:22:19.320 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-11T01:22:19.527 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-11T01:22:19.542 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-11T01:22:19.544 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi012.front.sepia.ceph.com,smithi071.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-11T01:26:02.722 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi012.front.sepia.ceph.com'), Remote(name='ubuntu@smithi071.front.sepia.ceph.com')] 2024-09-11T01:26:02.724 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-09-11T01:26:02.725 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T01:26:02.814 DEBUG:teuthology.orchestra.run.smithi012:> true 2024-09-11T01:26:02.896 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-09-11T01:26:02.897 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-09-11T01:26:02.897 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T01:26:03.008 DEBUG:teuthology.orchestra.run.smithi071:> true 2024-09-11T01:26:03.087 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-09-11T01:26:03.088 INFO:teuthology.run_tasks:Running task clock... 2024-09-11T01:26:03.099 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-11T01:26:03.100 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T01:26:03.100 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T01:26:03.103 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T01:26:03.103 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T01:26:03.133 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-11T01:26:03.149 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-11T01:26:03.156 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-11T01:26:03.170 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-11T01:26:03.175 INFO:teuthology.orchestra.run.smithi012.stderr:sudo: ntpd: command not found 2024-09-11T01:26:03.187 INFO:teuthology.orchestra.run.smithi012.stdout:506 Cannot talk to daemon 2024-09-11T01:26:03.193 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: ntpd: command not found 2024-09-11T01:26:03.202 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-11T01:26:03.204 INFO:teuthology.orchestra.run.smithi071.stdout:506 Cannot talk to daemon 2024-09-11T01:26:03.217 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-11T01:26:03.219 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-11T01:26:03.232 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-11T01:26:03.273 INFO:teuthology.orchestra.run.smithi012.stderr:bash: line 1: ntpq: command not found 2024-09-11T01:26:03.278 INFO:teuthology.orchestra.run.smithi012.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T01:26:03.278 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================== 2024-09-11T01:26:03.278 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T01:26:03.278 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T01:26:03.278 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T01:26:03.278 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T01:26:03.286 INFO:teuthology.orchestra.run.smithi071.stderr:bash: line 1: ntpq: command not found 2024-09-11T01:26:03.291 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T01:26:03.291 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2024-09-11T01:26:03.291 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T01:26:03.291 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T01:26:03.291 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T01:26:03.291 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T01:26:03.292 INFO:teuthology.run_tasks:Running task install... 2024-09-11T01:26:03.300 DEBUG:teuthology.task.install:project ceph 2024-09-11T01:26:03.300 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}} 2024-09-11T01:26:03.300 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'} 2024-09-11T01:26:03.300 INFO:teuthology.task.install:Using flavor: default 2024-09-11T01:26:03.305 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-11T01:26:03.305 INFO:teuthology.task.install:extra packages: [] 2024-09-11T01:26:03.306 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'tag': None, 'wait_for_package': False} 2024-09-11T01:26:03.306 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T01:26:03.308 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'tag': None, 'wait_for_package': False} 2024-09-11T01:26:03.309 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T01:26:03.456 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/ 2024-09-11T01:26:03.456 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4854.g8f219c47 2024-09-11T01:26:03.472 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/ 2024-09-11T01:26:03.473 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4854.g8f219c47 2024-09-11T01:26:03.586 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-11T01:26:03.586 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-09-11T01:26:03.586 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-11T01:26:03.594 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-11T01:26:03.594 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-11T01:26:03.594 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-11T01:26:03.613 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-11T01:26:03.614 DEBUG:teuthology.orchestra.run.smithi012:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8f219c476cb4f26fc5719c661c0323719a45cb30/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-11T01:26:03.620 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-11T01:26:03.620 DEBUG:teuthology.orchestra.run.smithi071:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8f219c476cb4f26fc5719c661c0323719a45cb30/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-11T01:26:03.680 DEBUG:teuthology.orchestra.run.smithi012:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-11T01:26:03.686 DEBUG:teuthology.orchestra.run.smithi071:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-11T01:26:03.757 DEBUG:teuthology.orchestra.run.smithi012:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-11T01:26:03.760 DEBUG:teuthology.orchestra.run.smithi071:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-11T01:26:03.822 INFO:teuthology.orchestra.run.smithi012.stdout:check_obsoletes = 1 2024-09-11T01:26:03.824 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum clean all 2024-09-11T01:26:03.824 INFO:teuthology.orchestra.run.smithi071.stdout:check_obsoletes = 1 2024-09-11T01:26:03.826 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum clean all 2024-09-11T01:26:04.062 INFO:teuthology.orchestra.run.smithi071.stdout:45 files removed 2024-09-11T01:26:04.065 INFO:teuthology.orchestra.run.smithi012.stdout:45 files removed 2024-09-11T01:26:04.088 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-radosgw 2024-09-11T01:26:04.092 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-radosgw 2024-09-11T01:26:04.638 INFO:teuthology.orchestra.run.smithi071.stdout:ceph packages for x86_64 318 kB/s | 84 kB 00:00 2024-09-11T01:26:04.675 INFO:teuthology.orchestra.run.smithi012.stdout:ceph packages for x86_64 295 kB/s | 84 kB 00:00 2024-09-11T01:26:04.930 INFO:teuthology.orchestra.run.smithi071.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-09-11T01:26:04.976 INFO:teuthology.orchestra.run.smithi012.stdout:ceph noarch packages 95 kB/s | 24 kB 00:00 2024-09-11T01:26:05.198 INFO:teuthology.orchestra.run.smithi071.stdout:ceph source packages 8.4 kB/s | 1.9 kB 00:00 2024-09-11T01:26:05.260 INFO:teuthology.orchestra.run.smithi012.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-11T01:26:05.731 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.2 MB 00:00 2024-09-11T01:26:06.002 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-09-11T01:26:09.494 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - AppStream 8.7 MB/s | 20 MB 00:02 2024-09-11T01:26:09.757 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - AppStream 7.0 MB/s | 20 MB 00:02 2024-09-11T01:26:13.983 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.5 MB 00:00 2024-09-11T01:26:13.994 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-11T01:26:15.487 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - Extras packages 84 kB/s | 19 kB 00:00 2024-09-11T01:26:16.225 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 19 kB 00:01 2024-09-11T01:26:16.616 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-11T01:26:16.970 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 37 MB/s | 23 MB 00:00 2024-09-11T01:26:22.255 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-11T01:26:22.426 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-11T01:26:24.734 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-09-11T01:26:24.745 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:26:24.745 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-09-11T01:26:24.745 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:26:24.745 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-09-11T01:26:24.745 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-radosgw x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 11 M 2024-09-11T01:26:24.745 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrading: 2024-09-11T01:26:24.745 INFO:teuthology.orchestra.run.smithi012.stdout: librados2 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 3.5 M 2024-09-11T01:26:24.745 INFO:teuthology.orchestra.run.smithi012.stdout: librbd1 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 3.2 M 2024-09-11T01:26:24.746 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-09-11T01:26:24.746 INFO:teuthology.orchestra.run.smithi012.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-11T01:26:24.746 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 5.5 M 2024-09-11T01:26:24.746 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-common x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 22 M 2024-09-11T01:26:24.746 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-selinux x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 25 k 2024-09-11T01:26:24.746 INFO:teuthology.orchestra.run.smithi012.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-11T01:26:24.746 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-11T01:26:24.746 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-11T01:26:24.746 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs2 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 755 k 2024-09-11T01:26:24.746 INFO:teuthology.orchestra.run.smithi012.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-11T01:26:24.746 INFO:teuthology.orchestra.run.smithi012.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-11T01:26:24.746 INFO:teuthology.orchestra.run.smithi012.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-11T01:26:24.747 INFO:teuthology.orchestra.run.smithi012.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-11T01:26:24.747 INFO:teuthology.orchestra.run.smithi012.stdout: libradosstriper1 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 507 k 2024-09-11T01:26:24.747 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-11T01:26:24.747 INFO:teuthology.orchestra.run.smithi012.stdout: librgw2 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 5.5 M 2024-09-11T01:26:24.747 INFO:teuthology.orchestra.run.smithi012.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-11T01:26:24.747 INFO:teuthology.orchestra.run.smithi012.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-11T01:26:24.747 INFO:teuthology.orchestra.run.smithi012.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-11T01:26:24.747 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 45 k 2024-09-11T01:26:24.747 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-common x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 143 k 2024-09-11T01:26:24.747 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cephfs x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 161 k 2024-09-11T01:26:24.747 INFO:teuthology.orchestra.run.smithi012.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-11T01:26:24.747 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rados x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 321 k 2024-09-11T01:26:24.748 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rbd x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 306 k 2024-09-11T01:26:24.748 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rgw x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 99 k 2024-09-11T01:26:24.748 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-11T01:26:24.748 INFO:teuthology.orchestra.run.smithi012.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-11T01:26:24.748 INFO:teuthology.orchestra.run.smithi012.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-11T01:26:24.748 INFO:teuthology.orchestra.run.smithi012.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-11T01:26:24.748 INFO:teuthology.orchestra.run.smithi012.stdout:Installing weak dependencies: 2024-09-11T01:26:24.748 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-11T01:26:24.748 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:26:24.748 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-09-11T01:26:24.748 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:26:24.748 INFO:teuthology.orchestra.run.smithi012.stdout:Install 31 Packages 2024-09-11T01:26:24.749 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrade 2 Packages 2024-09-11T01:26:24.749 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:26:24.749 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 62 M 2024-09-11T01:26:24.749 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-09-11T01:26:24.794 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-11T01:26:24.804 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:26:24.804 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-09-11T01:26:24.804 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:26:24.804 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-11T01:26:24.804 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 11 M 2024-09-11T01:26:24.804 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrading: 2024-09-11T01:26:24.805 INFO:teuthology.orchestra.run.smithi071.stdout: librados2 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 3.5 M 2024-09-11T01:26:24.805 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 3.2 M 2024-09-11T01:26:24.805 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-09-11T01:26:24.805 INFO:teuthology.orchestra.run.smithi071.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-11T01:26:24.805 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 5.5 M 2024-09-11T01:26:24.805 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 22 M 2024-09-11T01:26:24.805 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 25 k 2024-09-11T01:26:24.805 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-11T01:26:24.805 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-11T01:26:24.805 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-11T01:26:24.805 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 755 k 2024-09-11T01:26:24.805 INFO:teuthology.orchestra.run.smithi071.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-11T01:26:24.805 INFO:teuthology.orchestra.run.smithi071.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-11T01:26:24.806 INFO:teuthology.orchestra.run.smithi071.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-11T01:26:24.806 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-11T01:26:24.806 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 507 k 2024-09-11T01:26:24.806 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-11T01:26:24.806 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 5.5 M 2024-09-11T01:26:24.806 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-11T01:26:24.806 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-11T01:26:24.806 INFO:teuthology.orchestra.run.smithi071.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-11T01:26:24.806 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 45 k 2024-09-11T01:26:24.806 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 143 k 2024-09-11T01:26:24.806 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 161 k 2024-09-11T01:26:24.806 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-11T01:26:24.806 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 321 k 2024-09-11T01:26:24.807 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 306 k 2024-09-11T01:26:24.807 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 99 k 2024-09-11T01:26:24.807 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-11T01:26:24.807 INFO:teuthology.orchestra.run.smithi071.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-11T01:26:24.807 INFO:teuthology.orchestra.run.smithi071.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-11T01:26:24.807 INFO:teuthology.orchestra.run.smithi071.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-11T01:26:24.807 INFO:teuthology.orchestra.run.smithi071.stdout:Installing weak dependencies: 2024-09-11T01:26:24.807 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-11T01:26:24.807 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:26:24.807 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-11T01:26:24.807 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:26:24.807 INFO:teuthology.orchestra.run.smithi071.stdout:Install 31 Packages 2024-09-11T01:26:24.807 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrade 2 Packages 2024-09-11T01:26:24.808 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:26:24.808 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 62 M 2024-09-11T01:26:24.808 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-11T01:26:25.389 INFO:teuthology.orchestra.run.smithi012.stdout:(1/33): ceph-base-19.3.0-4854.g8f219c47.el9.x86 10 MB/s | 5.5 MB 00:00 2024-09-11T01:26:25.456 INFO:teuthology.orchestra.run.smithi012.stdout:(2/33): ceph-selinux-19.3.0-4854.g8f219c47.el9. 378 kB/s | 25 kB 00:00 2024-09-11T01:26:25.565 INFO:teuthology.orchestra.run.smithi012.stdout:(3/33): libcephfs2-19.3.0-4854.g8f219c47.el9.x8 6.8 MB/s | 755 kB 00:00 2024-09-11T01:26:25.665 INFO:teuthology.orchestra.run.smithi012.stdout:(4/33): libradosstriper1-19.3.0-4854.g8f219c47. 5.0 MB/s | 507 kB 00:00 2024-09-11T01:26:25.807 INFO:teuthology.orchestra.run.smithi012.stdout:(5/33): ceph-radosgw-19.3.0-4854.g8f219c47.el9. 11 MB/s | 11 MB 00:00 2024-09-11T01:26:25.882 INFO:teuthology.orchestra.run.smithi012.stdout:(6/33): python3-ceph-argparse-19.3.0-4854.g8f21 603 kB/s | 45 kB 00:00 2024-09-11T01:26:25.983 INFO:teuthology.orchestra.run.smithi012.stdout:(7/33): python3-ceph-common-19.3.0-4854.g8f219c 1.4 MB/s | 143 kB 00:00 2024-09-11T01:26:26.052 INFO:teuthology.orchestra.run.smithi071.stdout:(1/33): ceph-base-19.3.0-4854.g8f219c47.el9.x86 4.8 MB/s | 5.5 MB 00:01 2024-09-11T01:26:26.074 INFO:teuthology.orchestra.run.smithi012.stdout:(8/33): librgw2-19.3.0-4854.g8f219c47.el9.x86_6 13 MB/s | 5.5 MB 00:00 2024-09-11T01:26:26.099 INFO:teuthology.orchestra.run.smithi012.stdout:(9/33): python3-cephfs-19.3.0-4854.g8f219c47.el 1.4 MB/s | 161 kB 00:00 2024-09-11T01:26:26.102 INFO:teuthology.orchestra.run.smithi071.stdout:(2/33): ceph-selinux-19.3.0-4854.g8f219c47.el9. 505 kB/s | 25 kB 00:00 2024-09-11T01:26:26.125 INFO:teuthology.orchestra.run.smithi012.stdout:(10/33): python3-rados-19.3.0-4854.g8f219c47.el 6.3 MB/s | 321 kB 00:00 2024-09-11T01:26:26.150 INFO:teuthology.orchestra.run.smithi012.stdout:(11/33): python3-rbd-19.3.0-4854.g8f219c47.el9. 5.9 MB/s | 306 kB 00:00 2024-09-11T01:26:26.177 INFO:teuthology.orchestra.run.smithi071.stdout:(3/33): libcephfs2-19.3.0-4854.g8f219c47.el9.x8 9.8 MB/s | 755 kB 00:00 2024-09-11T01:26:26.184 INFO:teuthology.orchestra.run.smithi012.stdout:(12/33): python3-rgw-19.3.0-4854.g8f219c47.el9. 1.7 MB/s | 99 kB 00:00 2024-09-11T01:26:26.252 INFO:teuthology.orchestra.run.smithi071.stdout:(4/33): libradosstriper1-19.3.0-4854.g8f219c47. 6.6 MB/s | 507 kB 00:00 2024-09-11T01:26:26.393 INFO:teuthology.orchestra.run.smithi012.stdout:(13/33): ceph-common-19.3.0-4854.g8f219c47.el9. 14 MB/s | 22 MB 00:01 2024-09-11T01:26:26.501 INFO:teuthology.orchestra.run.smithi012.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 305 kB/s | 107 kB 00:00 2024-09-11T01:26:26.526 INFO:teuthology.orchestra.run.smithi012.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 499 kB/s | 171 kB 00:00 2024-09-11T01:26:26.552 INFO:teuthology.orchestra.run.smithi012.stdout:(16/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 900 kB/s | 45 kB 00:00 2024-09-11T01:26:26.577 INFO:teuthology.orchestra.run.smithi012.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 871 kB/s | 160 kB 00:00 2024-09-11T01:26:26.594 INFO:teuthology.orchestra.run.smithi071.stdout:(5/33): librgw2-19.3.0-4854.g8f219c47.el9.x86_6 16 MB/s | 5.5 MB 00:00 2024-09-11T01:26:26.627 INFO:teuthology.orchestra.run.smithi012.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 847 kB/s | 42 kB 00:00 2024-09-11T01:26:26.653 INFO:teuthology.orchestra.run.smithi012.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-11T01:26:26.703 INFO:teuthology.orchestra.run.smithi012.stdout:(20/33): qatlib-service-24.02.0-1.el9.x86_64.rp 719 kB/s | 36 kB 00:00 2024-09-11T01:26:26.711 INFO:teuthology.orchestra.run.smithi071.stdout:(6/33): ceph-radosgw-19.3.0-4854.g8f219c47.el9. 6.1 MB/s | 11 MB 00:01 2024-09-11T01:26:26.737 INFO:teuthology.orchestra.run.smithi012.stdout:(21/33): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 662 kB 00:00 2024-09-11T01:26:26.737 INFO:teuthology.orchestra.run.smithi071.stdout:(7/33): python3-ceph-argparse-19.3.0-4854.g8f21 318 kB/s | 45 kB 00:00 2024-09-11T01:26:26.761 INFO:teuthology.orchestra.run.smithi012.stdout:(22/33): qatlib-24.02.0-1.el9.x86_64.rpm 1.6 MB/s | 221 kB 00:00 2024-09-11T01:26:26.762 INFO:teuthology.orchestra.run.smithi071.stdout:(8/33): python3-ceph-common-19.3.0-4854.g8f219c 2.8 MB/s | 143 kB 00:00 2024-09-11T01:26:26.787 INFO:teuthology.orchestra.run.smithi012.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 565 kB/s | 47 kB 00:00 2024-09-11T01:26:26.787 INFO:teuthology.orchestra.run.smithi071.stdout:(9/33): python3-cephfs-19.3.0-4854.g8f219c47.el 3.1 MB/s | 161 kB 00:00 2024-09-11T01:26:26.812 INFO:teuthology.orchestra.run.smithi071.stdout:(10/33): python3-rados-19.3.0-4854.g8f219c47.el 6.2 MB/s | 321 kB 00:00 2024-09-11T01:26:26.838 INFO:teuthology.orchestra.run.smithi071.stdout:(11/33): python3-rbd-19.3.0-4854.g8f219c47.el9. 5.9 MB/s | 306 kB 00:00 2024-09-11T01:26:26.854 INFO:teuthology.orchestra.run.smithi012.stdout:(24/33): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.6 MB/s | 309 kB 00:00 2024-09-11T01:26:26.854 INFO:teuthology.orchestra.run.smithi071.stdout:(12/33): python3-rgw-19.3.0-4854.g8f219c47.el9. 2.3 MB/s | 99 kB 00:00 2024-09-11T01:26:26.879 INFO:teuthology.orchestra.run.smithi012.stdout:(25/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 283 kB/s | 26 kB 00:00 2024-09-11T01:26:26.904 INFO:teuthology.orchestra.run.smithi012.stdout:(26/33): liboath-2.6.7-2.el9.x86_64.rpm 965 kB/s | 49 kB 00:00 2024-09-11T01:26:26.932 INFO:teuthology.orchestra.run.smithi012.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-11T01:26:26.964 INFO:teuthology.orchestra.run.smithi012.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-11T01:26:26.997 INFO:teuthology.orchestra.run.smithi012.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-09-11T01:26:27.064 INFO:teuthology.orchestra.run.smithi012.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-11T01:26:27.147 INFO:teuthology.orchestra.run.smithi012.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-11T01:26:27.155 INFO:teuthology.orchestra.run.smithi071.stdout:(13/33): ceph-common-19.3.0-4854.g8f219c47.el9. 9.7 MB/s | 22 MB 00:02 2024-09-11T01:26:27.256 INFO:teuthology.orchestra.run.smithi012.stdout:(32/33): librados2-19.3.0-4854.g8f219c47.el9.x8 13 MB/s | 3.5 MB 00:00 2024-09-11T01:26:27.315 INFO:teuthology.orchestra.run.smithi012.stdout:(33/33): librbd1-19.3.0-4854.g8f219c47.el9.x86_ 13 MB/s | 3.2 MB 00:00 2024-09-11T01:26:27.319 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:26:27.319 INFO:teuthology.orchestra.run.smithi012.stdout:Total 24 MB/s | 62 MB 00:02 2024-09-11T01:26:27.437 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-09-11T01:26:27.478 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-09-11T01:26:27.479 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-09-11T01:26:27.681 INFO:teuthology.orchestra.run.smithi071.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 127 kB/s | 107 kB 00:00 2024-09-11T01:26:27.730 INFO:teuthology.orchestra.run.smithi071.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 195 kB/s | 171 kB 00:00 2024-09-11T01:26:27.872 INFO:teuthology.orchestra.run.smithi071.stdout:(16/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 223 kB/s | 160 kB 00:00 2024-09-11T01:26:28.175 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-09-11T01:26:28.175 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-09-11T01:26:28.289 INFO:teuthology.orchestra.run.smithi071.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 74 kB/s | 45 kB 00:00 2024-09-11T01:26:28.598 INFO:teuthology.orchestra.run.smithi071.stdout:(18/33): librdkafka-1.6.1-102.el9.x86_64.rpm 764 kB/s | 662 kB 00:00 2024-09-11T01:26:28.648 INFO:teuthology.orchestra.run.smithi071.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 377 kB/s | 292 kB 00:00 2024-09-11T01:26:28.873 INFO:teuthology.orchestra.run.smithi071.stdout:(20/33): python3-prettytable-0.7.2-27.el9.noarc 72 kB/s | 42 kB 00:00 2024-09-11T01:26:29.223 INFO:teuthology.orchestra.run.smithi071.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 62 kB/s | 36 kB 00:00 2024-09-11T01:26:29.274 INFO:teuthology.orchestra.run.smithi071.stdout:(22/33): gperftools-libs-2.9.1-2.el9.x86_64.rpm 6.1 MB/s | 309 kB 00:00 2024-09-11T01:26:29.349 INFO:teuthology.orchestra.run.smithi071.stdout:(23/33): qatlib-24.02.0-1.el9.x86_64.rpm 294 kB/s | 221 kB 00:00 2024-09-11T01:26:29.416 INFO:teuthology.orchestra.run.smithi071.stdout:(24/33): libarrow-9.0.0-11.el9.x86_64.rpm 31 MB/s | 4.4 MB 00:00 2024-09-11T01:26:29.433 INFO:teuthology.orchestra.run.smithi071.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 3.2 MB/s | 49 kB 00:00 2024-09-11T01:26:29.450 INFO:teuthology.orchestra.run.smithi071.stdout:(26/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 83 kB/s | 47 kB 00:00 2024-09-11T01:26:29.467 INFO:teuthology.orchestra.run.smithi071.stdout:(27/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 221 kB/s | 26 kB 00:00 2024-09-11T01:26:29.500 INFO:teuthology.orchestra.run.smithi071.stdout:(28/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-09-11T01:26:29.526 INFO:teuthology.orchestra.run.smithi071.stdout:(29/33): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-11T01:26:29.576 INFO:teuthology.orchestra.run.smithi071.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 22 MB/s | 1.6 MB 00:00 2024-09-11T01:26:29.609 INFO:teuthology.orchestra.run.smithi071.stdout:(31/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.1 MB/s | 839 kB 00:00 2024-09-11T01:26:29.734 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-09-11T01:26:29.824 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-11T01:26:29.826 INFO:teuthology.orchestra.run.smithi071.stdout:(32/33): librbd1-19.3.0-4854.g8f219c47.el9.x86_ 13 MB/s | 3.2 MB 00:00 2024-09-11T01:26:29.853 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-11T01:26:29.877 INFO:teuthology.orchestra.run.smithi071.stdout:(33/33): librados2-19.3.0-4854.g8f219c47.el9.x8 9.9 MB/s | 3.5 MB 00:00 2024-09-11T01:26:29.883 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:26:29.883 INFO:teuthology.orchestra.run.smithi071.stdout:Total 12 MB/s | 62 MB 00:05 2024-09-11T01:26:29.886 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-11T01:26:29.917 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-11T01:26:29.939 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-11T01:26:30.004 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-11T01:26:30.039 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-11T01:26:30.039 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-11T01:26:30.140 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-11T01:26:30.199 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-11T01:26:30.562 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-11T01:26:30.671 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 7/35 2024-09-11T01:26:30.721 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-11T01:26:30.721 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-11T01:26:30.757 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 7/35 2024-09-11T01:26:30.847 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rados-2:19.3.0-4854.g8f219c47.el9.x86_64 8/35 2024-09-11T01:26:30.885 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 9/35 2024-09-11T01:26:30.936 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 9/35 2024-09-11T01:26:30.973 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-11T01:26:31.017 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ceph-argparse-2:19.3.0-4854.g8f219c47.el9. 11/35 2024-09-11T01:26:31.089 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cephfs-2:19.3.0-4854.g8f219c47.el9.x86_64 12/35 2024-09-11T01:26:31.117 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86_6 13/35 2024-09-11T01:26:31.181 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86_6 13/35 2024-09-11T01:26:31.215 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-11T01:26:31.328 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-11T01:26:31.358 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-11T01:26:31.400 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-11T01:26:31.431 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/35 2024-09-11T01:26:31.461 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-11T01:26:31.502 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-11T01:26:31.549 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-11T01:26:31.884 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-11T01:26:31.949 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 23/35 2024-09-11T01:26:32.024 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 23/35 2024-09-11T01:26:32.060 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rbd-2:19.3.0-4854.g8f219c47.el9.x86_64 24/35 2024-09-11T01:26:32.098 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-11T01:26:32.225 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-11T01:26:32.309 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-11T01:26:32.340 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-11T01:26:32.375 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-11T01:26:32.398 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-11T01:26:32.423 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-11T01:26:32.600 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-11T01:26:32.627 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-11T01:26:32.651 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 27/35 2024-09-11T01:26:32.653 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-11T01:26:32.714 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 27/35 2024-09-11T01:26:32.795 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rgw-2:19.3.0-4854.g8f219c47.el9.x86_64 28/35 2024-09-11T01:26:32.828 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ceph-common-2:19.3.0-4854.g8f219c47.el9.x8 29/35 2024-09-11T01:26:33.041 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-11T01:26:33.110 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 7/35 2024-09-11T01:26:33.202 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 7/35 2024-09-11T01:26:33.292 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rados-2:19.3.0-4854.g8f219c47.el9.x86_64 8/35 2024-09-11T01:26:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 9/35 2024-09-11T01:26:33.391 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 9/35 2024-09-11T01:26:33.428 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-11T01:26:33.472 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-argparse-2:19.3.0-4854.g8f219c47.el9. 11/35 2024-09-11T01:26:33.542 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cephfs-2:19.3.0-4854.g8f219c47.el9.x86_64 12/35 2024-09-11T01:26:33.572 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86_6 13/35 2024-09-11T01:26:33.636 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86_6 13/35 2024-09-11T01:26:33.670 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-11T01:26:33.781 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-11T01:26:33.813 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-11T01:26:33.855 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-11T01:26:33.886 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/35 2024-09-11T01:26:33.915 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-11T01:26:33.948 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-11T01:26:33.993 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-11T01:26:34.314 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-11T01:26:34.362 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 23/35 2024-09-11T01:26:34.452 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 23/35 2024-09-11T01:26:34.493 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rbd-2:19.3.0-4854.g8f219c47.el9.x86_64 24/35 2024-09-11T01:26:34.536 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-11T01:26:35.026 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-11T01:26:35.056 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 27/35 2024-09-11T01:26:35.127 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 27/35 2024-09-11T01:26:35.205 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rgw-2:19.3.0-4854.g8f219c47.el9.x86_64 28/35 2024-09-11T01:26:35.233 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 30/35 2024-09-11T01:26:35.258 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-common-2:19.3.0-4854.g8f219c47.el9.x8 29/35 2024-09-11T01:26:35.266 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 30/35 2024-09-11T01:26:35.903 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 30/35 2024-09-11T01:26:35.945 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 31/35 2024-09-11T01:26:36.045 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 31/35 2024-09-11T01:26:36.045 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-11T01:26:36.045 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-11T01:26:36.046 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:26:36.080 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 32/35 2024-09-11T01:26:37.567 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 30/35 2024-09-11T01:26:37.604 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 30/35 2024-09-11T01:26:38.178 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 30/35 2024-09-11T01:26:38.215 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 31/35 2024-09-11T01:26:38.310 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 31/35 2024-09-11T01:26:38.310 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-11T01:26:38.310 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-11T01:26:38.310 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:26:38.394 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 32/35 2024-09-11T01:26:46.554 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 32/35 2024-09-11T01:26:46.554 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /sys 2024-09-11T01:26:46.554 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /proc 2024-09-11T01:26:46.555 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /mnt 2024-09-11T01:26:46.555 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /var/tmp 2024-09-11T01:26:46.555 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /home 2024-09-11T01:26:46.555 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /root 2024-09-11T01:26:46.555 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /tmp 2024-09-11T01:26:46.555 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:26:47.612 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 33/35 2024-09-11T01:26:47.640 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 33/35 2024-09-11T01:26:47.640 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T01:26:47.640 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-11T01:26:47.640 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T01:26:47.640 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T01:26:47.640 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:26:47.642 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-11T01:26:47.740 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-11T01:26:47.740 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-11T01:26:48.464 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 32/35 2024-09-11T01:26:48.465 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /sys 2024-09-11T01:26:48.465 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /proc 2024-09-11T01:26:48.465 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /mnt 2024-09-11T01:26:48.465 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /var/tmp 2024-09-11T01:26:48.465 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /home 2024-09-11T01:26:48.465 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /root 2024-09-11T01:26:48.465 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /tmp 2024-09-11T01:26:48.465 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:26:48.988 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-11T01:26:48.988 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 1/35 2024-09-11T01:26:48.988 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 2/35 2024-09-11T01:26:48.988 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 3/35 2024-09-11T01:26:48.988 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 4/35 2024-09-11T01:26:48.988 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 5/35 2024-09-11T01:26:48.988 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86_6 6/35 2024-09-11T01:26:48.988 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 7/35 2024-09-11T01:26:48.989 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4854.g8f219c47.el9. 8/35 2024-09-11T01:26:48.989 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ceph-common-2:19.3.0-4854.g8f219c47.el9.x8 9/35 2024-09-11T01:26:48.989 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cephfs-2:19.3.0-4854.g8f219c47.el9.x86_64 10/35 2024-09-11T01:26:48.989 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rados-2:19.3.0-4854.g8f219c47.el9.x86_64 11/35 2024-09-11T01:26:48.989 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rbd-2:19.3.0-4854.g8f219c47.el9.x86_64 12/35 2024-09-11T01:26:48.989 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rgw-2:19.3.0-4854.g8f219c47.el9.x86_64 13/35 2024-09-11T01:26:48.989 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-11T01:26:48.989 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-11T01:26:48.989 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-11T01:26:48.989 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-11T01:26:48.989 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-11T01:26:48.989 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-11T01:26:48.990 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-11T01:26:48.990 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-11T01:26:48.990 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-11T01:26:48.990 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-11T01:26:48.990 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 24/35 2024-09-11T01:26:48.990 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-11T01:26:48.990 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-11T01:26:48.990 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-11T01:26:48.990 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-11T01:26:48.990 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-11T01:26:48.990 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-11T01:26:48.991 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-11T01:26:48.991 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 32/35 2024-09-11T01:26:48.991 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-11T01:26:48.991 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 34/35 2024-09-11T01:26:49.473 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 33/35 2024-09-11T01:26:49.498 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 33/35 2024-09-11T01:26:49.498 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T01:26:49.498 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-11T01:26:49.498 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T01:26:49.498 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T01:26:49.498 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:26:49.500 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-11T01:26:49.592 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-11T01:26:49.592 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-11T01:26:49.914 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-11T01:26:49.914 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:26:49.914 INFO:teuthology.orchestra.run.smithi012.stdout:Upgraded: 2024-09-11T01:26:49.914 INFO:teuthology.orchestra.run.smithi012.stdout: librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:49.914 INFO:teuthology.orchestra.run.smithi012.stdout: librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:49.914 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-09-11T01:26:49.914 INFO:teuthology.orchestra.run.smithi012.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-11T01:26:49.914 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:49.914 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:49.914 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:49.914 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:49.915 INFO:teuthology.orchestra.run.smithi012.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-11T01:26:49.915 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-11T01:26:49.915 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-11T01:26:49.915 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:49.915 INFO:teuthology.orchestra.run.smithi012.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-11T01:26:49.915 INFO:teuthology.orchestra.run.smithi012.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-11T01:26:49.915 INFO:teuthology.orchestra.run.smithi012.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-11T01:26:49.915 INFO:teuthology.orchestra.run.smithi012.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-11T01:26:49.915 INFO:teuthology.orchestra.run.smithi012.stdout: libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:49.915 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-11T01:26:49.915 INFO:teuthology.orchestra.run.smithi012.stdout: librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:49.915 INFO:teuthology.orchestra.run.smithi012.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-11T01:26:49.915 INFO:teuthology.orchestra.run.smithi012.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-11T01:26:49.916 INFO:teuthology.orchestra.run.smithi012.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-11T01:26:49.916 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:49.916 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:49.916 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cephfs-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:49.916 INFO:teuthology.orchestra.run.smithi012.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-11T01:26:49.916 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rados-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:49.916 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rbd-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:49.916 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rgw-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:49.916 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-11T01:26:49.916 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-11T01:26:49.916 INFO:teuthology.orchestra.run.smithi012.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-11T01:26:49.916 INFO:teuthology.orchestra.run.smithi012.stdout: re2-1:20211101-20.el9.x86_64 2024-09-11T01:26:49.916 INFO:teuthology.orchestra.run.smithi012.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-11T01:26:49.917 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:26:49.917 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-09-11T01:26:50.280 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-test 2024-09-11T01:26:50.753 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-11T01:26:50.753 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 1/35 2024-09-11T01:26:50.753 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 2/35 2024-09-11T01:26:50.753 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 3/35 2024-09-11T01:26:50.753 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 4/35 2024-09-11T01:26:50.753 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 5/35 2024-09-11T01:26:50.753 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86_6 6/35 2024-09-11T01:26:50.753 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 7/35 2024-09-11T01:26:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4854.g8f219c47.el9. 8/35 2024-09-11T01:26:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-common-2:19.3.0-4854.g8f219c47.el9.x8 9/35 2024-09-11T01:26:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cephfs-2:19.3.0-4854.g8f219c47.el9.x86_64 10/35 2024-09-11T01:26:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rados-2:19.3.0-4854.g8f219c47.el9.x86_64 11/35 2024-09-11T01:26:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rbd-2:19.3.0-4854.g8f219c47.el9.x86_64 12/35 2024-09-11T01:26:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rgw-2:19.3.0-4854.g8f219c47.el9.x86_64 13/35 2024-09-11T01:26:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-11T01:26:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-11T01:26:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-11T01:26:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-11T01:26:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-11T01:26:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-11T01:26:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-11T01:26:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-11T01:26:50.755 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-11T01:26:50.755 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-11T01:26:50.755 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 24/35 2024-09-11T01:26:50.755 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-11T01:26:50.755 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-11T01:26:50.756 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-11T01:26:50.756 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-11T01:26:50.757 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-11T01:26:50.757 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-11T01:26:50.757 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-11T01:26:50.757 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 32/35 2024-09-11T01:26:50.757 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-11T01:26:50.757 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 34/35 2024-09-11T01:26:50.862 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:00:28 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:26:51.001 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-09-11T01:26:51.002 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:26:51.002 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-09-11T01:26:51.002 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:26:51.002 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-09-11T01:26:51.002 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-test x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 50 M 2024-09-11T01:26:51.002 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-09-11T01:26:51.003 INFO:teuthology.orchestra.run.smithi012.stdout: libcephsqlite x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 162 k 2024-09-11T01:26:51.003 INFO:teuthology.orchestra.run.smithi012.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-11T01:26:51.003 INFO:teuthology.orchestra.run.smithi012.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-11T01:26:51.003 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:26:51.003 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-09-11T01:26:51.003 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:26:51.003 INFO:teuthology.orchestra.run.smithi012.stdout:Install 4 Packages 2024-09-11T01:26:51.003 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:26:51.004 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 51 M 2024-09-11T01:26:51.004 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 217 M 2024-09-11T01:26:51.004 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-09-11T01:26:51.253 INFO:teuthology.orchestra.run.smithi012.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-09-11T01:26:51.278 INFO:teuthology.orchestra.run.smithi012.stdout:(2/4): libcephsqlite-19.3.0-4854.g8f219c47.el9. 712 kB/s | 162 kB 00:00 2024-09-11T01:26:51.303 INFO:teuthology.orchestra.run.smithi012.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-11T01:26:51.581 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-11T01:26:51.581 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:26:51.581 INFO:teuthology.orchestra.run.smithi071.stdout:Upgraded: 2024-09-11T01:26:51.582 INFO:teuthology.orchestra.run.smithi071.stdout: librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:51.582 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:51.582 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-11T01:26:51.582 INFO:teuthology.orchestra.run.smithi071.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-11T01:26:51.582 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:51.582 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:51.583 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:51.583 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:51.583 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-11T01:26:51.583 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-11T01:26:51.583 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-11T01:26:51.583 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:51.583 INFO:teuthology.orchestra.run.smithi071.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-11T01:26:51.584 INFO:teuthology.orchestra.run.smithi071.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-11T01:26:51.584 INFO:teuthology.orchestra.run.smithi071.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-11T01:26:51.584 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-11T01:26:51.584 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:51.584 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-11T01:26:51.584 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:51.585 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-11T01:26:51.585 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-11T01:26:51.585 INFO:teuthology.orchestra.run.smithi071.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-11T01:26:51.585 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:51.585 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:51.586 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:51.586 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-11T01:26:51.586 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:51.586 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:51.586 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:26:51.586 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-11T01:26:51.586 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-11T01:26:51.587 INFO:teuthology.orchestra.run.smithi071.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-11T01:26:51.587 INFO:teuthology.orchestra.run.smithi071.stdout: re2-1:20211101-20.el9.x86_64 2024-09-11T01:26:51.587 INFO:teuthology.orchestra.run.smithi071.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-11T01:26:51.587 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:26:51.587 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-11T01:26:51.944 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-test 2024-09-11T01:26:52.508 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:30 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:26:52.639 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-11T01:26:52.640 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:26:52.640 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-09-11T01:26:52.640 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:26:52.640 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-11T01:26:52.640 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 50 M 2024-09-11T01:26:52.640 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-09-11T01:26:52.640 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 162 k 2024-09-11T01:26:52.641 INFO:teuthology.orchestra.run.smithi071.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-11T01:26:52.641 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-11T01:26:52.641 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:26:52.641 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-11T01:26:52.641 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:26:52.641 INFO:teuthology.orchestra.run.smithi071.stdout:Install 4 Packages 2024-09-11T01:26:52.641 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:26:52.641 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 51 M 2024-09-11T01:26:52.642 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 217 M 2024-09-11T01:26:52.642 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-11T01:26:52.653 INFO:teuthology.orchestra.run.smithi012.stdout:(4/4): ceph-test-19.3.0-4854.g8f219c47.el9.x86_ 31 MB/s | 50 MB 00:01 2024-09-11T01:26:52.655 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:26:52.656 INFO:teuthology.orchestra.run.smithi012.stdout:Total 31 MB/s | 51 MB 00:01 2024-09-11T01:26:52.684 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-09-11T01:26:52.707 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-09-11T01:26:52.707 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-09-11T01:26:52.812 INFO:teuthology.orchestra.run.smithi071.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.4 MB/s | 304 kB 00:00 2024-09-11T01:26:52.863 INFO:teuthology.orchestra.run.smithi071.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-11T01:26:52.897 INFO:teuthology.orchestra.run.smithi071.stdout:(3/4): libcephsqlite-19.3.0-4854.g8f219c47.el9. 776 kB/s | 162 kB 00:00 2024-09-11T01:26:53.132 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-09-11T01:26:53.133 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-09-11T01:26:53.685 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-09-11T01:26:53.774 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-11T01:26:53.820 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-11T01:26:53.847 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 3/4 2024-09-11T01:26:54.265 INFO:teuthology.orchestra.run.smithi071.stdout:(4/4): ceph-test-19.3.0-4854.g8f219c47.el9.x86_ 32 MB/s | 50 MB 00:01 2024-09-11T01:26:54.268 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:26:54.268 INFO:teuthology.orchestra.run.smithi071.stdout:Total 31 MB/s | 51 MB 00:01 2024-09-11T01:26:54.295 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-11T01:26:54.315 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-11T01:26:54.316 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-11T01:26:54.711 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-11T01:26:54.711 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-11T01:26:55.246 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-11T01:26:55.346 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-11T01:26:55.387 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-11T01:26:55.415 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 3/4 2024-09-11T01:26:58.886 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 3/4 2024-09-11T01:26:58.972 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-test-2:19.3.0-4854.g8f219c47.el9.x86_64 4/4 2024-09-11T01:27:00.154 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 3/4 2024-09-11T01:27:00.238 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-test-2:19.3.0-4854.g8f219c47.el9.x86_64 4/4 2024-09-11T01:27:01.430 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-test-2:19.3.0-4854.g8f219c47.el9.x86_64 4/4 2024-09-11T01:27:01.431 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-test-2:19.3.0-4854.g8f219c47.el9.x86_64 1/4 2024-09-11T01:27:01.431 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 2/4 2024-09-11T01:27:01.431 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-11T01:27:01.751 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-11T01:27:01.752 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:01.752 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-09-11T01:27:01.752 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-test-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:27:01.752 INFO:teuthology.orchestra.run.smithi012.stdout: libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:27:01.752 INFO:teuthology.orchestra.run.smithi012.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-11T01:27:01.752 INFO:teuthology.orchestra.run.smithi012.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-11T01:27:01.752 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:01.752 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-09-11T01:27:02.040 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph 2024-09-11T01:27:02.641 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:00:40 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:27:02.655 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-test-2:19.3.0-4854.g8f219c47.el9.x86_64 4/4 2024-09-11T01:27:02.655 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-test-2:19.3.0-4854.g8f219c47.el9.x86_64 1/4 2024-09-11T01:27:02.655 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 2/4 2024-09-11T01:27:02.655 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-11T01:27:02.785 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-09-11T01:27:02.788 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:27:02.788 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-09-11T01:27:02.788 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:27:02.788 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-09-11T01:27:02.788 INFO:teuthology.orchestra.run.smithi012.stdout: ceph x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 6.6 k 2024-09-11T01:27:02.788 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-09-11T01:27:02.788 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mds x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 2.4 M 2024-09-11T01:27:02.789 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 1.5 M 2024-09-11T01:27:02.789 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 291 k 2024-09-11T01:27:02.789 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mon x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 4.8 M 2024-09-11T01:27:02.789 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-osd x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 17 M 2024-09-11T01:27:02.789 INFO:teuthology.orchestra.run.smithi012.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-11T01:27:02.789 INFO:teuthology.orchestra.run.smithi012.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-11T01:27:02.789 INFO:teuthology.orchestra.run.smithi012.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-11T01:27:02.789 INFO:teuthology.orchestra.run.smithi012.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-11T01:27:02.789 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-11T01:27:02.789 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-11T01:27:02.789 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-11T01:27:02.789 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-11T01:27:02.790 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-11T01:27:02.790 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-11T01:27:02.790 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-collections 2024-09-11T01:27:02.790 INFO:teuthology.orchestra.run.smithi012.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-11T01:27:02.790 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-11T01:27:02.790 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-11T01:27:02.790 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-11T01:27:02.790 INFO:teuthology.orchestra.run.smithi012.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-11T01:27:02.790 INFO:teuthology.orchestra.run.smithi012.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-11T01:27:02.790 INFO:teuthology.orchestra.run.smithi012.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-11T01:27:02.790 INFO:teuthology.orchestra.run.smithi012.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-11T01:27:02.790 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-11T01:27:02.790 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-11T01:27:02.791 INFO:teuthology.orchestra.run.smithi012.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-11T01:27:02.791 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-11T01:27:02.791 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-11T01:27:02.791 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-11T01:27:02.791 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-11T01:27:02.791 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-11T01:27:02.791 INFO:teuthology.orchestra.run.smithi012.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-11T01:27:02.791 INFO:teuthology.orchestra.run.smithi012.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-11T01:27:02.791 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-11T01:27:02.791 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-11T01:27:02.791 INFO:teuthology.orchestra.run.smithi012.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-11T01:27:02.791 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:02.791 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-09-11T01:27:02.792 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:27:02.792 INFO:teuthology.orchestra.run.smithi012.stdout:Install 37 Packages 2024-09-11T01:27:02.792 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:02.792 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 31 M 2024-09-11T01:27:02.792 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 109 M 2024-09-11T01:27:02.792 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-09-11T01:27:02.974 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-11T01:27:02.974 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:02.974 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-11T01:27:02.974 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:27:02.974 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:27:02.975 INFO:teuthology.orchestra.run.smithi071.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-11T01:27:02.975 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-11T01:27:02.975 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:02.975 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-11T01:27:03.265 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph 2024-09-11T01:27:03.412 INFO:teuthology.orchestra.run.smithi012.stdout:(1/37): ceph-19.3.0-4854.g8f219c47.el9.x86_64.r 42 kB/s | 6.6 kB 00:00 2024-09-11T01:27:03.646 INFO:teuthology.orchestra.run.smithi012.stdout:(2/37): ceph-mds-19.3.0-4854.g8f219c47.el9.x86_ 6.1 MB/s | 2.4 MB 00:00 2024-09-11T01:27:03.712 INFO:teuthology.orchestra.run.smithi012.stdout:(3/37): ceph-mgr-19.3.0-4854.g8f219c47.el9.x86_ 3.3 MB/s | 1.5 MB 00:00 2024-09-11T01:27:03.805 INFO:teuthology.orchestra.run.smithi012.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4854.g8f21 3.1 MB/s | 291 kB 00:00 2024-09-11T01:27:03.850 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:41 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:27:03.994 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-11T01:27:03.996 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:03.996 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-09-11T01:27:03.996 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:03.996 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-11T01:27:03.996 INFO:teuthology.orchestra.run.smithi071.stdout: ceph x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 6.6 k 2024-09-11T01:27:03.996 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-09-11T01:27:03.997 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 2.4 M 2024-09-11T01:27:03.997 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 1.5 M 2024-09-11T01:27:03.997 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 291 k 2024-09-11T01:27:03.997 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 4.8 M 2024-09-11T01:27:03.997 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 17 M 2024-09-11T01:27:03.997 INFO:teuthology.orchestra.run.smithi071.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-11T01:27:03.997 INFO:teuthology.orchestra.run.smithi071.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-11T01:27:03.997 INFO:teuthology.orchestra.run.smithi071.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-11T01:27:03.997 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-11T01:27:03.997 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-11T01:27:03.997 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-11T01:27:03.997 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-11T01:27:03.997 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-11T01:27:03.998 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-11T01:27:03.998 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-11T01:27:03.998 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-collections 2024-09-11T01:27:03.998 INFO:teuthology.orchestra.run.smithi071.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-11T01:27:03.998 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-11T01:27:03.998 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-11T01:27:03.998 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-11T01:27:03.998 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-11T01:27:03.998 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-11T01:27:03.998 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-11T01:27:03.998 INFO:teuthology.orchestra.run.smithi071.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-11T01:27:03.998 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-11T01:27:03.999 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-11T01:27:03.999 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-11T01:27:03.999 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-11T01:27:03.999 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-11T01:27:03.999 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-11T01:27:03.999 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-11T01:27:03.999 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-11T01:27:03.999 INFO:teuthology.orchestra.run.smithi071.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-11T01:27:03.999 INFO:teuthology.orchestra.run.smithi071.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-11T01:27:03.999 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-11T01:27:03.999 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-11T01:27:03.999 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-11T01:27:03.999 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:04.000 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-11T01:27:04.000 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:04.000 INFO:teuthology.orchestra.run.smithi071.stdout:Install 37 Packages 2024-09-11T01:27:04.000 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:04.000 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 31 M 2024-09-11T01:27:04.000 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 109 M 2024-09-11T01:27:04.000 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-11T01:27:04.255 INFO:teuthology.orchestra.run.smithi012.stdout:(5/37): ceph-mon-19.3.0-4854.g8f219c47.el9.x86_ 5.7 MB/s | 4.8 MB 00:00 2024-09-11T01:27:04.705 INFO:teuthology.orchestra.run.smithi012.stdout:(6/37): ceph-osd-19.3.0-4854.g8f219c47.el9.x86_ 16 MB/s | 17 MB 00:01 2024-09-11T01:27:04.730 INFO:teuthology.orchestra.run.smithi012.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 274 kB/s | 253 kB 00:00 2024-09-11T01:27:04.879 INFO:teuthology.orchestra.run.smithi071.stdout:(1/37): ceph-19.3.0-4854.g8f219c47.el9.x86_64.r 37 kB/s | 6.6 kB 00:00 2024-09-11T01:27:05.397 INFO:teuthology.orchestra.run.smithi012.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 154 kB/s | 106 kB 00:00 2024-09-11T01:27:05.455 INFO:teuthology.orchestra.run.smithi012.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 186 kB/s | 135 kB 00:00 2024-09-11T01:27:05.496 INFO:teuthology.orchestra.run.smithi071.stdout:(2/37): ceph-mon-19.3.0-4854.g8f219c47.el9.x86_ 7.8 MB/s | 4.8 MB 00:00 2024-09-11T01:27:05.530 INFO:teuthology.orchestra.run.smithi071.stdout:(3/37): ceph-mgr-19.3.0-4854.g8f219c47.el9.x86_ 1.8 MB/s | 1.5 MB 00:00 2024-09-11T01:27:05.663 INFO:teuthology.orchestra.run.smithi071.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4854.g8f21 2.1 MB/s | 291 kB 00:00 2024-09-11T01:27:05.672 INFO:teuthology.orchestra.run.smithi012.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 885 kB/s | 1.2 MB 00:01 2024-09-11T01:27:05.747 INFO:teuthology.orchestra.run.smithi071.stdout:(5/37): ceph-mds-19.3.0-4854.g8f219c47.el9.x86_ 2.3 MB/s | 2.4 MB 00:01 2024-09-11T01:27:05.881 INFO:teuthology.orchestra.run.smithi071.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-09-11T01:27:05.889 INFO:teuthology.orchestra.run.smithi012.stdout:(11/37): lua-5.4.4-4.el9.x86_64.rpm 905 kB/s | 188 kB 00:00 2024-09-11T01:27:05.922 INFO:teuthology.orchestra.run.smithi071.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-11T01:27:05.948 INFO:teuthology.orchestra.run.smithi012.stdout:(12/37): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-09-11T01:27:05.998 INFO:teuthology.orchestra.run.smithi012.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 696 kB/s | 35 kB 00:00 2024-09-11T01:27:06.008 INFO:teuthology.orchestra.run.smithi071.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 1.6 MB/s | 135 kB 00:00 2024-09-11T01:27:06.057 INFO:teuthology.orchestra.run.smithi012.stdout:(14/37): python3-packaging-20.9-5.el9.noarch.rp 1.3 MB/s | 77 kB 00:00 2024-09-11T01:27:06.081 INFO:teuthology.orchestra.run.smithi071.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 1.6 MB/s | 125 kB 00:00 2024-09-11T01:27:06.107 INFO:teuthology.orchestra.run.smithi012.stdout:(15/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-11T01:27:06.157 INFO:teuthology.orchestra.run.smithi071.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 3.0 MB/s | 1.2 MB 00:00 2024-09-11T01:27:06.157 INFO:teuthology.orchestra.run.smithi012.stdout:(16/37): python3-requests-2.25.1-8.el9.noarch.r 164 kB/s | 125 kB 00:00 2024-09-11T01:27:06.181 INFO:teuthology.orchestra.run.smithi071.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.1 MB/s | 215 kB 00:00 2024-09-11T01:27:06.182 INFO:teuthology.orchestra.run.smithi012.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 555 kB/s | 42 kB 00:00 2024-09-11T01:27:06.241 INFO:teuthology.orchestra.run.smithi012.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-09-11T01:27:06.267 INFO:teuthology.orchestra.run.smithi012.stdout:(19/37): python3-urllib3-1.26.5-6.el9.noarch.rp 265 kB/s | 215 kB 00:00 2024-09-11T01:27:06.292 INFO:teuthology.orchestra.run.smithi012.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 867 kB/s | 43 kB 00:00 2024-09-11T01:27:06.307 INFO:teuthology.orchestra.run.smithi071.stdout:(12/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-11T01:27:06.332 INFO:teuthology.orchestra.run.smithi071.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 188 kB 00:00 2024-09-11T01:27:06.334 INFO:teuthology.orchestra.run.smithi012.stdout:(21/37): python3-cherrypy-18.6.1-2.el9.noarch.r 11 MB/s | 358 kB 00:00 2024-09-11T01:27:06.359 INFO:teuthology.orchestra.run.smithi012.stdout:(22/37): lua-devel-5.4.4-4.el9.x86_64.rpm 111 kB/s | 22 kB 00:00 2024-09-11T01:27:06.374 INFO:teuthology.orchestra.run.smithi071.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 518 kB/s | 35 kB 00:00 2024-09-11T01:27:06.384 INFO:teuthology.orchestra.run.smithi012.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 223 kB/s | 11 kB 00:00 2024-09-11T01:27:06.416 INFO:teuthology.orchestra.run.smithi071.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 922 kB/s | 77 kB 00:00 2024-09-11T01:27:06.418 INFO:teuthology.orchestra.run.smithi012.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 910 kB/s | 23 kB 00:00 2024-09-11T01:27:06.441 INFO:teuthology.orchestra.run.smithi071.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 762 kB/s | 51 kB 00:00 2024-09-11T01:27:06.443 INFO:teuthology.orchestra.run.smithi012.stdout:(25/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 975 kB/s | 172 kB 00:00 2024-09-11T01:27:06.469 INFO:teuthology.orchestra.run.smithi012.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 404 kB/s | 19 kB 00:00 2024-09-11T01:27:06.494 INFO:teuthology.orchestra.run.smithi012.stdout:(27/37): python3-jaraco-classes-3.2.1-5.el9.noa 132 kB/s | 18 kB 00:00 2024-09-11T01:27:06.500 INFO:teuthology.orchestra.run.smithi071.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 497 kB/s | 42 kB 00:00 2024-09-11T01:27:06.519 INFO:teuthology.orchestra.run.smithi012.stdout:(28/37): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-09-11T01:27:06.542 INFO:teuthology.orchestra.run.smithi071.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 3.6 MB/s | 151 kB 00:00 2024-09-11T01:27:06.544 INFO:teuthology.orchestra.run.smithi012.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 615 kB/s | 46 kB 00:00 2024-09-11T01:27:06.567 INFO:teuthology.orchestra.run.smithi071.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-09-11T01:27:06.569 INFO:teuthology.orchestra.run.smithi012.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-11T01:27:06.593 INFO:teuthology.orchestra.run.smithi071.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 6.7 MB/s | 172 kB 00:00 2024-09-11T01:27:06.595 INFO:teuthology.orchestra.run.smithi012.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 329 kB/s | 16 kB 00:00 2024-09-11T01:27:06.620 INFO:teuthology.orchestra.run.smithi012.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-11T01:27:06.626 INFO:teuthology.orchestra.run.smithi071.stdout:(21/37): python3-cherrypy-18.6.1-2.el9.noarch.r 11 MB/s | 358 kB 00:00 2024-09-11T01:27:06.643 INFO:teuthology.orchestra.run.smithi071.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 644 kB/s | 11 kB 00:00 2024-09-11T01:27:06.645 INFO:teuthology.orchestra.run.smithi012.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-09-11T01:27:06.660 INFO:teuthology.orchestra.run.smithi071.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 1.0 MB/s | 18 kB 00:00 2024-09-11T01:27:06.670 INFO:teuthology.orchestra.run.smithi012.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-11T01:27:06.677 INFO:teuthology.orchestra.run.smithi071.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 1.3 MB/s | 23 kB 00:00 2024-09-11T01:27:06.687 INFO:teuthology.orchestra.run.smithi012.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-11T01:27:06.702 INFO:teuthology.orchestra.run.smithi071.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 1.1 MB/s | 19 kB 00:00 2024-09-11T01:27:06.713 INFO:teuthology.orchestra.run.smithi012.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-09-11T01:27:06.727 INFO:teuthology.orchestra.run.smithi071.stdout:(26/37): python3-jaraco-text-3.2.0-6.el9.noarch 786 kB/s | 20 kB 00:00 2024-09-11T01:27:06.738 INFO:teuthology.orchestra.run.smithi012.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-09-11T01:27:06.744 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:27:06.744 INFO:teuthology.orchestra.run.smithi012.stdout:Total 7.8 MB/s | 31 MB 00:03 2024-09-11T01:27:06.753 INFO:teuthology.orchestra.run.smithi071.stdout:(27/37): python3-logutils-0.3.5-21.el9.noarch.r 1.8 MB/s | 46 kB 00:00 2024-09-11T01:27:06.770 INFO:teuthology.orchestra.run.smithi071.stdout:(28/37): python3-more-itertools-8.12.0-2.el9.no 4.6 MB/s | 79 kB 00:00 2024-09-11T01:27:06.904 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-09-11T01:27:06.937 INFO:teuthology.orchestra.run.smithi071.stdout:(29/37): ceph-osd-19.3.0-4854.g8f219c47.el9.x86 12 MB/s | 17 MB 00:01 2024-09-11T01:27:06.949 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-09-11T01:27:06.949 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-09-11T01:27:06.970 INFO:teuthology.orchestra.run.smithi071.stdout:(30/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-09-11T01:27:06.995 INFO:teuthology.orchestra.run.smithi071.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 282 kB/s | 16 kB 00:00 2024-09-11T01:27:07.021 INFO:teuthology.orchestra.run.smithi071.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-11T01:27:07.046 INFO:teuthology.orchestra.run.smithi071.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-11T01:27:07.080 INFO:teuthology.orchestra.run.smithi071.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.9 MB/s | 230 kB 00:00 2024-09-11T01:27:07.105 INFO:teuthology.orchestra.run.smithi071.stdout:(35/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 7.2 MB/s | 427 kB 00:00 2024-09-11T01:27:07.130 INFO:teuthology.orchestra.run.smithi071.stdout:(36/37): lua-devel-5.4.4-4.el9.x86_64.rpm 32 kB/s | 22 kB 00:00 2024-09-11T01:27:07.155 INFO:teuthology.orchestra.run.smithi071.stdout:(37/37): python3-zc-lockfile-2.0-10.el9.noarch. 265 kB/s | 20 kB 00:00 2024-09-11T01:27:07.159 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:27:07.159 INFO:teuthology.orchestra.run.smithi071.stdout:Total 9.7 MB/s | 31 MB 00:03 2024-09-11T01:27:07.306 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-11T01:27:07.346 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-11T01:27:07.346 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-11T01:27:07.460 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-09-11T01:27:07.460 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-09-11T01:27:07.815 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-11T01:27:07.815 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-11T01:27:08.855 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-09-11T01:27:08.916 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-11T01:27:08.949 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-11T01:27:08.980 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-11T01:27:09.057 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-11T01:27:09.096 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-11T01:27:09.129 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-11T01:27:09.183 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-11T01:27:09.233 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-11T01:27:09.258 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-11T01:27:09.291 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-11T01:27:09.319 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-11T01:27:09.320 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-11T01:27:09.365 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-11T01:27:09.390 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-11T01:27:09.400 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-11T01:27:09.429 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-11T01:27:09.434 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-11T01:27:09.464 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-11T01:27:09.467 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-11T01:27:09.506 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-11T01:27:09.559 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-11T01:27:09.564 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-11T01:27:09.607 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-11T01:27:09.629 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-11T01:27:09.641 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-11T01:27:09.669 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-11T01:27:09.690 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-11T01:27:09.705 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-11T01:27:09.730 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-11T01:27:09.735 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-11T01:27:09.769 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-11T01:27:09.797 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-11T01:27:09.807 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-11T01:27:09.830 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-11T01:27:09.873 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-11T01:27:09.909 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-11T01:27:09.955 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-11T01:27:09.959 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-11T01:27:09.988 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-11T01:27:10.042 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-11T01:27:10.060 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-11T01:27:10.109 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-11T01:27:10.146 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-11T01:27:10.200 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-11T01:27:10.204 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-11T01:27:10.269 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-11T01:27:10.279 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-11T01:27:10.332 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-11T01:27:10.370 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-11T01:27:10.419 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-11T01:27:10.466 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-11T01:27:10.474 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-11T01:27:10.525 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-11T01:27:10.607 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-11T01:27:10.661 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-11T01:27:10.705 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-11T01:27:10.716 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-11T01:27:10.747 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-11T01:27:10.795 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-11T01:27:10.886 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-11T01:27:10.899 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-11T01:27:10.956 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-11T01:27:11.073 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4854.g8f219c47.el9. 32/37 2024-09-11T01:27:11.084 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-11T01:27:11.109 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 33/37 2024-09-11T01:27:11.140 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 33/37 2024-09-11T01:27:11.140 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T01:27:11.140 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-11T01:27:11.140 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T01:27:11.140 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T01:27:11.141 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:11.247 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4854.g8f219c47.el9. 32/37 2024-09-11T01:27:11.275 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 33/37 2024-09-11T01:27:11.301 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 33/37 2024-09-11T01:27:11.302 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T01:27:11.302 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-11T01:27:11.302 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T01:27:11.302 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T01:27:11.302 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:12.693 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 34/37 2024-09-11T01:27:12.722 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 34/37 2024-09-11T01:27:12.722 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T01:27:12.722 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-11T01:27:12.723 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T01:27:12.723 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T01:27:12.723 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:12.776 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 34/37 2024-09-11T01:27:12.803 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 34/37 2024-09-11T01:27:12.804 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T01:27:12.804 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-11T01:27:12.804 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T01:27:12.804 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T01:27:12.804 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:13.202 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 35/37 2024-09-11T01:27:13.228 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 35/37 2024-09-11T01:27:13.228 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T01:27:13.228 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-11T01:27:13.229 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T01:27:13.229 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T01:27:13.229 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:13.268 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 35/37 2024-09-11T01:27:13.291 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 35/37 2024-09-11T01:27:13.291 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T01:27:13.291 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-11T01:27:13.292 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T01:27:13.292 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T01:27:13.292 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:13.469 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 36/37 2024-09-11T01:27:13.495 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 36/37 2024-09-11T01:27:13.495 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T01:27:13.495 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-11T01:27:13.495 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T01:27:13.495 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T01:27:13.495 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:13.526 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 36/37 2024-09-11T01:27:13.550 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 36/37 2024-09-11T01:27:13.550 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T01:27:13.550 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-11T01:27:13.550 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T01:27:13.550 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T01:27:13.550 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:13.561 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-2:19.3.0-4854.g8f219c47.el9.x86_64 37/37 2024-09-11T01:27:13.615 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-2:19.3.0-4854.g8f219c47.el9.x86_64 37/37 2024-09-11T01:27:15.359 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-2:19.3.0-4854.g8f219c47.el9.x86_64 37/37 2024-09-11T01:27:15.359 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-2:19.3.0-4854.g8f219c47.el9.x86_64 1/37 2024-09-11T01:27:15.359 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 2/37 2024-09-11T01:27:15.359 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 3/37 2024-09-11T01:27:15.359 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 4/37 2024-09-11T01:27:15.359 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 5/37 2024-09-11T01:27:15.359 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4854.g8f219c47.el9. 6/37 2024-09-11T01:27:15.359 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-11T01:27:15.359 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-11T01:27:15.359 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-11T01:27:15.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-11T01:27:15.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-11T01:27:15.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-11T01:27:15.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-11T01:27:15.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-11T01:27:15.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-11T01:27:15.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-11T01:27:15.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-11T01:27:15.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-11T01:27:15.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-11T01:27:15.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-11T01:27:15.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-11T01:27:15.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-11T01:27:15.360 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-11T01:27:15.361 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-11T01:27:15.361 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-11T01:27:15.361 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-11T01:27:15.361 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-11T01:27:15.362 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-11T01:27:15.362 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-11T01:27:15.362 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-11T01:27:15.362 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-11T01:27:15.362 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-11T01:27:15.362 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-11T01:27:15.362 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-11T01:27:15.362 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-11T01:27:15.363 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-11T01:27:15.413 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-2:19.3.0-4854.g8f219c47.el9.x86_64 37/37 2024-09-11T01:27:15.413 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-2:19.3.0-4854.g8f219c47.el9.x86_64 1/37 2024-09-11T01:27:15.413 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 2/37 2024-09-11T01:27:15.413 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 3/37 2024-09-11T01:27:15.413 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 4/37 2024-09-11T01:27:15.413 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 5/37 2024-09-11T01:27:15.413 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4854.g8f219c47.el9. 6/37 2024-09-11T01:27:15.414 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-11T01:27:15.414 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-11T01:27:15.414 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-11T01:27:15.414 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-11T01:27:15.414 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-11T01:27:15.414 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-11T01:27:15.414 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-11T01:27:15.414 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-11T01:27:15.414 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-11T01:27:15.414 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-11T01:27:15.414 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-11T01:27:15.414 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-11T01:27:15.414 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-11T01:27:15.415 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-11T01:27:15.415 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-11T01:27:15.415 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-11T01:27:15.415 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-11T01:27:15.415 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-11T01:27:15.415 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-11T01:27:15.415 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-11T01:27:15.415 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-11T01:27:15.415 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-11T01:27:15.415 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-11T01:27:15.415 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-11T01:27:15.416 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-11T01:27:15.417 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-11T01:27:15.417 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-11T01:27:15.417 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-11T01:27:15.417 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-11T01:27:15.417 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-11T01:27:16.411 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-11T01:27:16.411 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:16.412 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-11T01:27:16.412 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:27:16.412 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:27:16.412 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:27:16.412 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T01:27:16.412 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:27:16.412 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:27:16.412 INFO:teuthology.orchestra.run.smithi071.stdout: lua-5.4.4-4.el9.x86_64 2024-09-11T01:27:16.412 INFO:teuthology.orchestra.run.smithi071.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-11T01:27:16.412 INFO:teuthology.orchestra.run.smithi071.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-11T01:27:16.412 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-11T01:27:16.412 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-11T01:27:16.413 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-11T01:27:16.413 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-11T01:27:16.413 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-11T01:27:16.413 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-11T01:27:16.413 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-11T01:27:16.413 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-11T01:27:16.413 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-11T01:27:16.413 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-11T01:27:16.413 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-11T01:27:16.413 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-11T01:27:16.413 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-11T01:27:16.413 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-11T01:27:16.413 INFO:teuthology.orchestra.run.smithi071.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-11T01:27:16.414 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-11T01:27:16.414 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ply-3.11-14.el9.noarch 2024-09-11T01:27:16.414 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-11T01:27:16.414 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-11T01:27:16.414 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-11T01:27:16.414 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-11T01:27:16.414 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-11T01:27:16.414 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-11T01:27:16.414 INFO:teuthology.orchestra.run.smithi071.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-11T01:27:16.414 INFO:teuthology.orchestra.run.smithi071.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-11T01:27:16.414 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-11T01:27:16.414 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-11T01:27:16.414 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-11T01:27:16.415 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:16.415 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-11T01:27:16.539 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-11T01:27:16.539 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:16.539 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-09-11T01:27:16.539 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:27:16.540 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:27:16.540 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:27:16.540 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T01:27:16.540 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:27:16.540 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:27:16.540 INFO:teuthology.orchestra.run.smithi012.stdout: lua-5.4.4-4.el9.x86_64 2024-09-11T01:27:16.540 INFO:teuthology.orchestra.run.smithi012.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-11T01:27:16.540 INFO:teuthology.orchestra.run.smithi012.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-11T01:27:16.540 INFO:teuthology.orchestra.run.smithi012.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-11T01:27:16.540 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-11T01:27:16.540 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-11T01:27:16.540 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-11T01:27:16.541 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-11T01:27:16.541 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-11T01:27:16.541 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-11T01:27:16.541 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-11T01:27:16.541 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-11T01:27:16.541 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-11T01:27:16.541 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-11T01:27:16.541 INFO:teuthology.orchestra.run.smithi012.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-11T01:27:16.541 INFO:teuthology.orchestra.run.smithi012.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-11T01:27:16.541 INFO:teuthology.orchestra.run.smithi012.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-11T01:27:16.541 INFO:teuthology.orchestra.run.smithi012.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-11T01:27:16.541 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-11T01:27:16.541 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ply-3.11-14.el9.noarch 2024-09-11T01:27:16.541 INFO:teuthology.orchestra.run.smithi012.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-11T01:27:16.542 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-11T01:27:16.542 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-11T01:27:16.542 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-11T01:27:16.542 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-11T01:27:16.542 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-11T01:27:16.542 INFO:teuthology.orchestra.run.smithi012.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-11T01:27:16.542 INFO:teuthology.orchestra.run.smithi012.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-11T01:27:16.542 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-11T01:27:16.542 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-11T01:27:16.542 INFO:teuthology.orchestra.run.smithi012.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-11T01:27:16.542 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:16.542 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-09-11T01:27:16.764 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-base 2024-09-11T01:27:16.874 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-base 2024-09-11T01:27:17.335 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:55 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:27:17.425 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T01:27:17.467 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:00:55 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:27:17.467 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-11T01:27:17.468 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-11T01:27:17.468 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-11T01:27:17.538 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install cephadm 2024-09-11T01:27:17.559 INFO:teuthology.orchestra.run.smithi012.stdout:Package ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T01:27:17.604 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-09-11T01:27:17.605 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2024-09-11T01:27:17.605 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-09-11T01:27:17.669 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install cephadm 2024-09-11T01:27:18.100 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:56 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:27:18.231 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-11T01:27:18.231 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:18.231 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-09-11T01:27:18.232 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:18.232 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-11T01:27:18.232 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 783 k 2024-09-11T01:27:18.232 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:18.232 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-11T01:27:18.232 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:18.232 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-09-11T01:27:18.232 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:18.232 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 783 k 2024-09-11T01:27:18.232 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 790 k 2024-09-11T01:27:18.232 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-11T01:27:18.257 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:00:56 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:27:18.396 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-09-11T01:27:18.396 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:27:18.397 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-09-11T01:27:18.397 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:27:18.397 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-09-11T01:27:18.397 INFO:teuthology.orchestra.run.smithi012.stdout: cephadm noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 783 k 2024-09-11T01:27:18.397 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:18.397 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-09-11T01:27:18.397 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:27:18.397 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2024-09-11T01:27:18.397 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:18.397 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 783 k 2024-09-11T01:27:18.397 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 790 k 2024-09-11T01:27:18.397 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-09-11T01:27:18.491 INFO:teuthology.orchestra.run.smithi071.stdout:cephadm-19.3.0-4854.g8f219c47.el9.noarch.rpm 3.0 MB/s | 783 kB 00:00 2024-09-11T01:27:18.492 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:27:18.492 INFO:teuthology.orchestra.run.smithi071.stdout:Total 2.9 MB/s | 783 kB 00:00 2024-09-11T01:27:18.492 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-11T01:27:18.497 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-11T01:27:18.497 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-11T01:27:18.503 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-11T01:27:18.503 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-11T01:27:18.572 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-11T01:27:18.644 INFO:teuthology.orchestra.run.smithi012.stdout:cephadm-19.3.0-4854.g8f219c47.el9.noarch.rpm 3.1 MB/s | 783 kB 00:00 2024-09-11T01:27:18.645 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:27:18.645 INFO:teuthology.orchestra.run.smithi012.stdout:Total 3.1 MB/s | 783 kB 00:00 2024-09-11T01:27:18.645 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-09-11T01:27:18.651 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-09-11T01:27:18.651 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-09-11T01:27:18.657 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-09-11T01:27:18.657 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-09-11T01:27:18.733 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-09-11T01:27:19.002 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 1/1 2024-09-11T01:27:19.086 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 1/1 2024-09-11T01:27:19.165 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 1/1 2024-09-11T01:27:19.251 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 1/1 2024-09-11T01:27:19.453 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 1/1 2024-09-11T01:27:19.609 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 1/1 2024-09-11T01:27:19.729 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 1/1 2024-09-11T01:27:19.730 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:19.730 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-11T01:27:19.730 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T01:27:19.730 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:19.730 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-11T01:27:19.865 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 1/1 2024-09-11T01:27:19.865 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:19.866 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-09-11T01:27:19.866 INFO:teuthology.orchestra.run.smithi012.stdout: cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T01:27:19.866 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:19.866 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-09-11T01:27:19.949 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-immutable-object-cache 2024-09-11T01:27:20.067 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-immutable-object-cache 2024-09-11T01:27:20.518 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:58 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:27:20.654 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:00:58 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:27:20.655 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-11T01:27:20.656 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:20.656 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-09-11T01:27:20.656 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:20.656 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-11T01:27:20.656 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 145 k 2024-09-11T01:27:20.656 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:20.656 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-11T01:27:20.656 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:20.656 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-09-11T01:27:20.657 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:20.657 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 145 k 2024-09-11T01:27:20.657 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 432 k 2024-09-11T01:27:20.657 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-11T01:27:20.793 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-09-11T01:27:20.794 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:27:20.794 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repo Size 2024-09-11T01:27:20.794 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:27:20.794 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-09-11T01:27:20.794 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 145 k 2024-09-11T01:27:20.794 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:20.794 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-09-11T01:27:20.795 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:27:20.795 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2024-09-11T01:27:20.795 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:20.795 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 145 k 2024-09-11T01:27:20.795 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 432 k 2024-09-11T01:27:20.795 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-09-11T01:27:20.855 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-immutable-object-cache-19.3.0-4854.g8f219c 732 kB/s | 145 kB 00:00 2024-09-11T01:27:20.855 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:27:20.855 INFO:teuthology.orchestra.run.smithi071.stdout:Total 727 kB/s | 145 kB 00:00 2024-09-11T01:27:20.856 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-11T01:27:20.863 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-11T01:27:20.863 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-11T01:27:20.921 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-11T01:27:20.921 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-11T01:27:21.008 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-immutable-object-cache-19.3.0-4854.g8f219c 680 kB/s | 145 kB 00:00 2024-09-11T01:27:21.008 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:27:21.009 INFO:teuthology.orchestra.run.smithi012.stdout:Total 676 kB/s | 145 kB 00:00 2024-09-11T01:27:21.009 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-09-11T01:27:21.017 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-09-11T01:27:21.017 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-09-11T01:27:21.067 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-11T01:27:21.077 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-09-11T01:27:21.077 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-09-11T01:27:21.131 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4854.g8f219c47. 1/1 2024-09-11T01:27:21.156 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4854.g8f219c47. 1/1 2024-09-11T01:27:21.156 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T01:27:21.157 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-11T01:27:21.157 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:21.231 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-09-11T01:27:21.292 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4854.g8f219c47. 1/1 2024-09-11T01:27:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4854.g8f219c47. 1/1 2024-09-11T01:27:21.318 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T01:27:21.318 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-11T01:27:21.318 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:22.088 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4854.g8f219c47. 1/1 2024-09-11T01:27:22.088 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:22.088 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-11T01:27:22.088 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:27:22.088 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:22.088 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-11T01:27:22.253 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4854.g8f219c47. 1/1 2024-09-11T01:27:22.254 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:22.254 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-09-11T01:27:22.254 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-immutable-object-cache-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:27:22.254 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:22.254 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-09-11T01:27:22.332 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr 2024-09-11T01:27:22.486 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-mgr 2024-09-11T01:27:22.913 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:00 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:27:23.003 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T01:27:23.045 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-11T01:27:23.045 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-11T01:27:23.046 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-11T01:27:23.075 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:01 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:27:23.119 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-dashboard 2024-09-11T01:27:23.169 INFO:teuthology.orchestra.run.smithi012.stdout:Package ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T01:27:23.213 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-09-11T01:27:23.214 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2024-09-11T01:27:23.214 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-09-11T01:27:23.278 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-mgr-dashboard 2024-09-11T01:27:23.682 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:01 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:27:23.816 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-11T01:27:23.817 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:23.817 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-09-11T01:27:23.817 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:23.817 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-11T01:27:23.817 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 14 M 2024-09-11T01:27:23.817 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-09-11T01:27:23.817 INFO:teuthology.orchestra.run.smithi071.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-11T01:27:23.818 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 34 k 2024-09-11T01:27:23.818 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 16 k 2024-09-11T01:27:23.818 INFO:teuthology.orchestra.run.smithi071.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-11T01:27:23.818 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-11T01:27:23.818 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-11T01:27:23.818 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-11T01:27:23.818 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-11T01:27:23.818 INFO:teuthology.orchestra.run.smithi071.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-11T01:27:23.818 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-11T01:27:23.818 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-11T01:27:23.818 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:23.818 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-11T01:27:23.818 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:23.819 INFO:teuthology.orchestra.run.smithi071.stdout:Install 12 Packages 2024-09-11T01:27:23.819 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:23.819 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 19 M 2024-09-11T01:27:23.819 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 159 M 2024-09-11T01:27:23.819 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-11T01:27:23.863 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:01 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:27:24.003 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-09-11T01:27:24.004 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:27:24.004 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-09-11T01:27:24.005 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:27:24.005 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-09-11T01:27:24.005 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 14 M 2024-09-11T01:27:24.005 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-09-11T01:27:24.005 INFO:teuthology.orchestra.run.smithi012.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-11T01:27:24.005 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 34 k 2024-09-11T01:27:24.005 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 16 k 2024-09-11T01:27:24.005 INFO:teuthology.orchestra.run.smithi012.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-11T01:27:24.005 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-11T01:27:24.005 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-11T01:27:24.005 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-11T01:27:24.005 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-11T01:27:24.005 INFO:teuthology.orchestra.run.smithi012.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-11T01:27:24.006 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-11T01:27:24.006 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-11T01:27:24.006 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:24.006 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-09-11T01:27:24.006 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:27:24.006 INFO:teuthology.orchestra.run.smithi012.stdout:Install 12 Packages 2024-09-11T01:27:24.006 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:24.006 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 19 M 2024-09-11T01:27:24.007 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 159 M 2024-09-11T01:27:24.007 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-09-11T01:27:24.349 INFO:teuthology.orchestra.run.smithi071.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4854.g8f 156 kB/s | 34 kB 00:00 2024-09-11T01:27:24.366 INFO:teuthology.orchestra.run.smithi071.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4854.g8f2 69 kB/s | 16 kB 00:00 2024-09-11T01:27:24.525 INFO:teuthology.orchestra.run.smithi071.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 269 kB 00:00 2024-09-11T01:27:24.600 INFO:teuthology.orchestra.run.smithi071.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.0 MB/s | 1.0 MB 00:00 2024-09-11T01:27:24.650 INFO:teuthology.orchestra.run.smithi071.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-11T01:27:24.667 INFO:teuthology.orchestra.run.smithi071.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 1.1 MB/s | 19 kB 00:00 2024-09-11T01:27:24.709 INFO:teuthology.orchestra.run.smithi071.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.6 MB/s | 863 kB 00:00 2024-09-11T01:27:24.761 INFO:teuthology.orchestra.run.smithi012.stdout:(1/12): ceph-prometheus-alerts-19.3.0-4854.g8f2 105 kB/s | 16 kB 00:00 2024-09-11T01:27:24.767 INFO:teuthology.orchestra.run.smithi071.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 20 MB/s | 2.0 MB 00:00 2024-09-11T01:27:24.784 INFO:teuthology.orchestra.run.smithi071.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.8 MB/s | 31 kB 00:00 2024-09-11T01:27:24.786 INFO:teuthology.orchestra.run.smithi012.stdout:(2/12): ceph-grafana-dashboards-19.3.0-4854.g8f 189 kB/s | 34 kB 00:00 2024-09-11T01:27:24.810 INFO:teuthology.orchestra.run.smithi071.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.4 MB/s | 144 kB 00:00 2024-09-11T01:27:24.837 INFO:teuthology.orchestra.run.smithi071.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.5 MB/s | 188 kB 00:00 2024-09-11T01:27:25.071 INFO:teuthology.orchestra.run.smithi071.stdout:(12/12): ceph-mgr-dashboard-19.3.0-4854.g8f219c 15 MB/s | 14 MB 00:00 2024-09-11T01:27:25.074 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:27:25.075 INFO:teuthology.orchestra.run.smithi071.stdout:Total 15 MB/s | 19 MB 00:01 2024-09-11T01:27:25.123 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-11T01:27:25.141 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-11T01:27:25.142 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-11T01:27:25.312 INFO:teuthology.orchestra.run.smithi012.stdout:(3/12): ceph-mgr-dashboard-19.3.0-4854.g8f219c4 20 MB/s | 14 MB 00:00 2024-09-11T01:27:25.402 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-11T01:27:25.402 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-11T01:27:25.428 INFO:teuthology.orchestra.run.smithi012.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 420 kB/s | 269 kB 00:00 2024-09-11T01:27:25.487 INFO:teuthology.orchestra.run.smithi012.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.2 MB/s | 548 kB 00:00 2024-09-11T01:27:25.512 INFO:teuthology.orchestra.run.smithi012.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 778 kB/s | 19 kB 00:00 2024-09-11T01:27:25.579 INFO:teuthology.orchestra.run.smithi012.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 31 MB/s | 2.0 MB 00:00 2024-09-11T01:27:25.613 INFO:teuthology.orchestra.run.smithi012.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.5 MB/s | 144 kB 00:00 2024-09-11T01:27:25.638 INFO:teuthology.orchestra.run.smithi012.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-11T01:27:25.663 INFO:teuthology.orchestra.run.smithi012.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.3 MB/s | 188 kB 00:00 2024-09-11T01:27:25.730 INFO:teuthology.orchestra.run.smithi012.stdout:(11/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.0 MB/s | 1.0 MB 00:00 2024-09-11T01:27:25.992 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-11T01:27:26.080 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-11T01:27:26.114 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-11T01:27:26.161 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-11T01:27:26.199 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-11T01:27:26.293 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-11T01:27:26.322 INFO:teuthology.orchestra.run.smithi012.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 855 kB/s | 863 kB 00:01 2024-09-11T01:27:26.327 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:27:26.327 INFO:teuthology.orchestra.run.smithi012.stdout:Total 8.3 MB/s | 19 MB 00:02 2024-09-11T01:27:26.382 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-09-11T01:27:26.405 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-09-11T01:27:26.405 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-09-11T01:27:26.417 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-11T01:27:26.502 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-11T01:27:26.554 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-11T01:27:26.582 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-11T01:27:26.606 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4854.g8f219c47.el9 10/12 2024-09-11T01:27:26.679 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-09-11T01:27:26.680 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-09-11T01:27:27.310 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-09-11T01:27:27.390 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-11T01:27:27.425 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-11T01:27:27.473 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-11T01:27:27.510 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-11T01:27:27.617 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-11T01:27:27.775 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-11T01:27:27.866 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-11T01:27:27.924 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-11T01:27:27.951 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-11T01:27:27.984 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4854.g8f219c47.el9 10/12 2024-09-11T01:27:29.779 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4854.g8f219c47.el 11/12 2024-09-11T01:27:29.835 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.noa 12/12 2024-09-11T01:27:31.361 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4854.g8f219c47.el 11/12 2024-09-11T01:27:31.421 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.noa 12/12 2024-09-11T01:27:31.582 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.noa 12/12 2024-09-11T01:27:31.582 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4854.g8f219c47.el 1/12 2024-09-11T01:27:31.583 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.noa 2/12 2024-09-11T01:27:31.583 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4854.g8f219c47.el9 3/12 2024-09-11T01:27:31.583 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-11T01:27:31.583 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-11T01:27:31.583 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-11T01:27:31.583 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-11T01:27:31.583 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-11T01:27:31.583 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-11T01:27:31.583 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-11T01:27:31.583 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-11T01:27:31.911 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-11T01:27:31.911 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:31.911 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-11T01:27:31.911 INFO:teuthology.orchestra.run.smithi071.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-11T01:27:31.911 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T01:27:31.911 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T01:27:31.911 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T01:27:31.911 INFO:teuthology.orchestra.run.smithi071.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-11T01:27:31.912 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-11T01:27:31.912 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-11T01:27:31.912 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-11T01:27:31.912 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-11T01:27:31.912 INFO:teuthology.orchestra.run.smithi071.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-11T01:27:31.912 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-11T01:27:31.912 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-11T01:27:31.912 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:31.912 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-11T01:27:32.186 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-11T01:27:32.751 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:10 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:27:32.891 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-11T01:27:32.892 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================= 2024-09-11T01:27:32.892 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-09-11T01:27:32.892 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================= 2024-09-11T01:27:32.893 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-11T01:27:32.893 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 7.4 M 2024-09-11T01:27:32.893 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-09-11T01:27:32.893 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-11T01:27:32.893 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-11T01:27:32.893 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-11T01:27:32.893 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-11T01:27:32.893 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-11T01:27:32.893 INFO:teuthology.orchestra.run.smithi071.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-11T01:27:32.893 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-11T01:27:32.893 INFO:teuthology.orchestra.run.smithi071.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-11T01:27:32.893 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-11T01:27:32.894 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-11T01:27:32.894 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-11T01:27:32.894 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:32.894 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-11T01:27:32.894 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================= 2024-09-11T01:27:32.894 INFO:teuthology.orchestra.run.smithi071.stdout:Install 12 Packages 2024-09-11T01:27:32.894 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:32.895 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 38 M 2024-09-11T01:27:32.895 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 212 M 2024-09-11T01:27:32.895 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-11T01:27:33.329 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.noa 12/12 2024-09-11T01:27:33.329 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4854.g8f219c47.el 1/12 2024-09-11T01:27:33.329 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.noa 2/12 2024-09-11T01:27:33.329 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4854.g8f219c47.el9 3/12 2024-09-11T01:27:33.329 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-11T01:27:33.329 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-11T01:27:33.330 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-11T01:27:33.330 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-11T01:27:33.330 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-11T01:27:33.330 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-11T01:27:33.330 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-11T01:27:33.330 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-11T01:27:33.343 INFO:teuthology.orchestra.run.smithi071.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-11T01:27:33.393 INFO:teuthology.orchestra.run.smithi071.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.1 MB/s | 801 kB 00:00 2024-09-11T01:27:33.476 INFO:teuthology.orchestra.run.smithi071.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 246 kB/s | 33 kB 00:00 2024-09-11T01:27:33.526 INFO:teuthology.orchestra.run.smithi071.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-09-11T01:27:33.586 INFO:teuthology.orchestra.run.smithi071.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 656 kB/s | 38 kB 00:00 2024-09-11T01:27:33.669 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-11T01:27:33.670 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:33.670 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-09-11T01:27:33.670 INFO:teuthology.orchestra.run.smithi012.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-11T01:27:33.670 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-grafana-dashboards-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T01:27:33.670 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T01:27:33.670 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-prometheus-alerts-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T01:27:33.670 INFO:teuthology.orchestra.run.smithi012.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-11T01:27:33.670 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-11T01:27:33.670 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-11T01:27:33.670 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-11T01:27:33.670 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-11T01:27:33.670 INFO:teuthology.orchestra.run.smithi012.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-11T01:27:33.670 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-11T01:27:33.671 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-11T01:27:33.671 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:33.671 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-09-11T01:27:33.710 INFO:teuthology.orchestra.run.smithi071.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-48 15 MB/s | 7.4 MB 00:00 2024-09-11T01:27:33.803 INFO:teuthology.orchestra.run.smithi071.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.3 MB/s | 3.0 MB 00:00 2024-09-11T01:27:33.900 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-11T01:27:34.011 INFO:teuthology.orchestra.run.smithi071.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 12 MB/s | 4.9 MB 00:00 2024-09-11T01:27:34.036 INFO:teuthology.orchestra.run.smithi071.stdout:(9/12): python3-devel-3.9.19-8.el9.x86_64.rpm 751 kB/s | 245 kB 00:00 2024-09-11T01:27:34.120 INFO:teuthology.orchestra.run.smithi071.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 5.1 MB 00:00 2024-09-11T01:27:34.145 INFO:teuthology.orchestra.run.smithi071.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.8 MB/s | 250 kB 00:00 2024-09-11T01:27:34.500 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:12 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:27:34.642 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-09-11T01:27:34.643 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================= 2024-09-11T01:27:34.644 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-09-11T01:27:34.644 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================= 2024-09-11T01:27:34.644 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-09-11T01:27:34.644 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 7.4 M 2024-09-11T01:27:34.644 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-09-11T01:27:34.644 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-11T01:27:34.644 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-11T01:27:34.644 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-11T01:27:34.644 INFO:teuthology.orchestra.run.smithi012.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-11T01:27:34.644 INFO:teuthology.orchestra.run.smithi012.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-11T01:27:34.644 INFO:teuthology.orchestra.run.smithi012.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-11T01:27:34.645 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-11T01:27:34.645 INFO:teuthology.orchestra.run.smithi012.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-11T01:27:34.645 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-11T01:27:34.645 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-11T01:27:34.645 INFO:teuthology.orchestra.run.smithi012.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-11T01:27:34.645 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:34.645 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-09-11T01:27:34.645 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================= 2024-09-11T01:27:34.645 INFO:teuthology.orchestra.run.smithi012.stdout:Install 12 Packages 2024-09-11T01:27:34.645 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:34.646 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 38 M 2024-09-11T01:27:34.646 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 212 M 2024-09-11T01:27:34.646 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-09-11T01:27:34.737 INFO:teuthology.orchestra.run.smithi071.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 23 MB/s | 16 MB 00:00 2024-09-11T01:27:34.738 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:27:34.738 INFO:teuthology.orchestra.run.smithi071.stdout:Total 21 MB/s | 38 MB 00:01 2024-09-11T01:27:34.920 INFO:teuthology.orchestra.run.smithi012.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2024-09-11T01:27:34.953 INFO:teuthology.orchestra.run.smithi012.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.7 MB/s | 801 kB 00:00 2024-09-11T01:27:35.090 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-11T01:27:35.117 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-11T01:27:35.117 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-11T01:27:35.237 INFO:teuthology.orchestra.run.smithi012.stdout:(3/12): ceph-mgr-diskprediction-local-19.3.0-48 16 MB/s | 7.4 MB 00:00 2024-09-11T01:27:35.495 INFO:teuthology.orchestra.run.smithi012.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 57 kB/s | 33 kB 00:00 2024-09-11T01:27:35.548 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-11T01:27:35.548 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-11T01:27:35.721 INFO:teuthology.orchestra.run.smithi012.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 38 kB/s | 18 kB 00:00 2024-09-11T01:27:36.063 INFO:teuthology.orchestra.run.smithi012.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 68 kB/s | 38 kB 00:00 2024-09-11T01:27:36.322 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-11T01:27:36.379 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-11T01:27:36.418 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-11T01:27:36.517 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-11T01:27:36.553 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-11T01:27:36.737 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-11T01:27:36.768 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-11T01:27:36.844 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-11T01:27:36.929 INFO:teuthology.orchestra.run.smithi012.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 282 kB/s | 245 kB 00:00 2024-09-11T01:27:37.429 INFO:teuthology.orchestra.run.smithi012.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 2.9 MB/s | 4.9 MB 00:01 2024-09-11T01:27:37.494 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-11T01:27:37.597 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-11T01:27:38.321 INFO:teuthology.orchestra.run.smithi012.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 281 kB/s | 250 kB 00:00 2024-09-11T01:27:38.630 INFO:teuthology.orchestra.run.smithi012.stdout:(10/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rp 833 kB/s | 3.0 MB 00:03 2024-09-11T01:27:38.833 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T01:27:39.072 INFO:teuthology.orchestra.run.smithi012.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.4 MB/s | 5.1 MB 00:02 2024-09-11T01:27:39.872 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-11T01:27:39.908 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f219 12/12 2024-09-11T01:27:40.447 INFO:teuthology.orchestra.run.smithi012.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 7.6 MB/s | 16 MB 00:02 2024-09-11T01:27:40.449 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:27:40.449 INFO:teuthology.orchestra.run.smithi012.stdout:Total 6.6 MB/s | 38 MB 00:05 2024-09-11T01:27:40.822 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-09-11T01:27:40.850 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-09-11T01:27:40.850 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-09-11T01:27:41.307 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-09-11T01:27:41.307 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-09-11T01:27:42.099 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-09-11T01:27:42.175 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-11T01:27:42.220 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-11T01:27:42.325 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-11T01:27:42.363 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-11T01:27:42.495 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f219 12/12 2024-09-11T01:27:42.495 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f219 1/12 2024-09-11T01:27:42.495 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-11T01:27:42.495 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-11T01:27:42.495 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-11T01:27:42.495 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-11T01:27:42.495 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-11T01:27:42.496 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-11T01:27:42.496 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-11T01:27:42.496 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-11T01:27:42.496 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T01:27:42.496 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-11T01:27:42.554 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-11T01:27:42.587 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-11T01:27:42.665 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-11T01:27:42.929 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-11T01:27:42.929 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:42.929 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-11T01:27:42.929 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T01:27:42.929 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-11T01:27:42.929 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-11T01:27:42.929 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-11T01:27:42.930 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-11T01:27:42.930 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-11T01:27:42.930 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-11T01:27:42.930 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-11T01:27:42.930 INFO:teuthology.orchestra.run.smithi071.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-11T01:27:42.930 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-11T01:27:42.930 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-11T01:27:42.930 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-11T01:27:42.930 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:42.930 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-11T01:27:43.345 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-rook 2024-09-11T01:27:43.360 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-11T01:27:43.487 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-11T01:27:43.925 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:21 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:27:44.062 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-11T01:27:44.063 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:44.063 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-09-11T01:27:44.063 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:44.063 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-11T01:27:44.063 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 49 k 2024-09-11T01:27:44.063 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-09-11T01:27:44.063 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-11T01:27:44.063 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-11T01:27:44.064 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-11T01:27:44.064 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-11T01:27:44.064 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-11T01:27:44.064 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-11T01:27:44.064 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-11T01:27:44.064 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-11T01:27:44.064 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-11T01:27:44.064 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-11T01:27:44.064 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-11T01:27:44.064 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-11T01:27:44.064 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:44.064 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-11T01:27:44.064 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:44.064 INFO:teuthology.orchestra.run.smithi071.stdout:Install 13 Packages 2024-09-11T01:27:44.065 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:44.065 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 2.2 M 2024-09-11T01:27:44.065 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 26 M 2024-09-11T01:27:44.065 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-11T01:27:44.499 INFO:teuthology.orchestra.run.smithi071.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 206 kB/s | 19 kB 00:00 2024-09-11T01:27:44.524 INFO:teuthology.orchestra.run.smithi071.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 217 kB/s | 26 kB 00:00 2024-09-11T01:27:44.608 INFO:teuthology.orchestra.run.smithi071.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 159 kB 00:00 2024-09-11T01:27:44.633 INFO:teuthology.orchestra.run.smithi071.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 222 kB 00:00 2024-09-11T01:27:44.700 INFO:teuthology.orchestra.run.smithi071.stdout:(5/13): ceph-mgr-rook-19.3.0-4854.g8f219c47.el9 167 kB/s | 49 kB 00:00 2024-09-11T01:27:44.725 INFO:teuthology.orchestra.run.smithi071.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 584 kB/s | 54 kB 00:00 2024-09-11T01:27:44.750 INFO:teuthology.orchestra.run.smithi071.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.9 MB/s | 279 kB 00:00 2024-09-11T01:27:44.801 INFO:teuthology.orchestra.run.smithi071.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 321 kB/s | 32 kB 00:00 2024-09-11T01:27:44.816 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T01:27:44.826 INFO:teuthology.orchestra.run.smithi071.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 140 kB/s | 14 kB 00:00 2024-09-11T01:27:44.852 INFO:teuthology.orchestra.run.smithi071.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-11T01:27:44.877 INFO:teuthology.orchestra.run.smithi071.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.9 MB/s | 220 kB 00:00 2024-09-11T01:27:44.902 INFO:teuthology.orchestra.run.smithi071.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-11T01:27:44.936 INFO:teuthology.orchestra.run.smithi071.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.7 MB/s | 1.0 MB 00:00 2024-09-11T01:27:44.938 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:27:44.938 INFO:teuthology.orchestra.run.smithi071.stdout:Total 2.5 MB/s | 2.2 MB 00:00 2024-09-11T01:27:45.005 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-11T01:27:45.020 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-11T01:27:45.020 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-11T01:27:45.106 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-11T01:27:45.107 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-11T01:27:45.621 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-11T01:27:45.791 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-11T01:27:45.845 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-11T01:27:45.900 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-11T01:27:45.911 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-11T01:27:45.935 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f219 12/12 2024-09-11T01:27:45.937 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-11T01:27:45.979 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-11T01:27:46.082 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-11T01:27:46.183 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-11T01:27:46.230 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-11T01:27:46.950 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-11T01:27:47.011 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-11T01:27:47.052 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-11T01:27:47.100 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-11T01:27:47.122 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 13/13 2024-09-11T01:27:47.875 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 13/13 2024-09-11T01:27:47.875 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 1/13 2024-09-11T01:27:47.875 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-11T01:27:47.876 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-11T01:27:47.876 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-11T01:27:47.876 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-11T01:27:47.876 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-11T01:27:47.876 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-11T01:27:47.876 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-11T01:27:47.876 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-11T01:27:47.876 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-11T01:27:47.876 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-11T01:27:47.876 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-11T01:27:48.217 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-11T01:27:48.217 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:48.217 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-11T01:27:48.217 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T01:27:48.217 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-11T01:27:48.217 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-11T01:27:48.217 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-11T01:27:48.217 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-11T01:27:48.217 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-11T01:27:48.217 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-11T01:27:48.217 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-11T01:27:48.218 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-11T01:27:48.218 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-11T01:27:48.218 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-11T01:27:48.218 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-11T01:27:48.218 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-11T01:27:48.218 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:48.218 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-11T01:27:48.468 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-cephadm 2024-09-11T01:27:48.578 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f219 12/12 2024-09-11T01:27:48.578 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f219 1/12 2024-09-11T01:27:48.578 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-11T01:27:48.578 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-11T01:27:48.578 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-11T01:27:48.578 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-11T01:27:48.578 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-11T01:27:48.578 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-11T01:27:48.579 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-11T01:27:48.579 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-11T01:27:48.579 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T01:27:48.579 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-11T01:27:48.984 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-11T01:27:48.984 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:48.985 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-09-11T01:27:48.985 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T01:27:48.985 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-11T01:27:48.985 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-11T01:27:48.985 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-11T01:27:48.985 INFO:teuthology.orchestra.run.smithi012.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-11T01:27:48.985 INFO:teuthology.orchestra.run.smithi012.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-11T01:27:48.985 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-11T01:27:48.985 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-11T01:27:48.985 INFO:teuthology.orchestra.run.smithi012.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-11T01:27:48.985 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-11T01:27:48.985 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-11T01:27:48.986 INFO:teuthology.orchestra.run.smithi012.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-11T01:27:48.986 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:48.986 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-09-11T01:27:49.029 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:27 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:27:49.162 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-11T01:27:49.162 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:49.163 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-09-11T01:27:49.163 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:49.163 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-11T01:27:49.163 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 154 k 2024-09-11T01:27:49.163 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-09-11T01:27:49.163 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-11T01:27:49.163 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-11T01:27:49.163 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-11T01:27:49.163 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-11T01:27:49.163 INFO:teuthology.orchestra.run.smithi071.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-11T01:27:49.163 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:49.163 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-11T01:27:49.163 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:49.164 INFO:teuthology.orchestra.run.smithi071.stdout:Install 6 Packages 2024-09-11T01:27:49.164 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:49.164 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 7.0 M 2024-09-11T01:27:49.164 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 33 M 2024-09-11T01:27:49.164 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-11T01:27:49.365 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-mgr-rook 2024-09-11T01:27:49.511 INFO:teuthology.orchestra.run.smithi071.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-09-11T01:27:49.569 INFO:teuthology.orchestra.run.smithi071.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-11T01:27:49.595 INFO:teuthology.orchestra.run.smithi071.stdout:(3/6): ceph-mgr-cephadm-19.3.0-4854.g8f219c47.e 681 kB/s | 154 kB 00:00 2024-09-11T01:27:49.620 INFO:teuthology.orchestra.run.smithi071.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-11T01:27:49.662 INFO:teuthology.orchestra.run.smithi071.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-09-11T01:27:49.745 INFO:teuthology.orchestra.run.smithi071.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-09-11T01:27:49.748 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:27:49.748 INFO:teuthology.orchestra.run.smithi071.stdout:Total 12 MB/s | 7.0 MB 00:00 2024-09-11T01:27:49.818 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-11T01:27:49.828 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-11T01:27:49.829 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-11T01:27:49.926 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-11T01:27:49.928 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-11T01:27:49.961 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:27 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:27:50.103 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-09-11T01:27:50.104 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:27:50.104 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-09-11T01:27:50.104 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:27:50.104 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-09-11T01:27:50.104 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-rook noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 49 k 2024-09-11T01:27:50.105 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-09-11T01:27:50.105 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-11T01:27:50.105 INFO:teuthology.orchestra.run.smithi012.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-11T01:27:50.105 INFO:teuthology.orchestra.run.smithi012.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-11T01:27:50.105 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-11T01:27:50.105 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-11T01:27:50.105 INFO:teuthology.orchestra.run.smithi012.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-11T01:27:50.105 INFO:teuthology.orchestra.run.smithi012.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-11T01:27:50.105 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-11T01:27:50.105 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-11T01:27:50.105 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-11T01:27:50.105 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-11T01:27:50.105 INFO:teuthology.orchestra.run.smithi012.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-11T01:27:50.106 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:50.106 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-09-11T01:27:50.106 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:27:50.106 INFO:teuthology.orchestra.run.smithi012.stdout:Install 13 Packages 2024-09-11T01:27:50.106 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:50.106 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 2.2 M 2024-09-11T01:27:50.107 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 26 M 2024-09-11T01:27:50.107 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-09-11T01:27:50.233 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-11T01:27:50.401 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-11T01:27:50.455 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-11T01:27:50.567 INFO:teuthology.orchestra.run.smithi012.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 211 kB/s | 26 kB 00:00 2024-09-11T01:27:50.592 INFO:teuthology.orchestra.run.smithi012.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 131 kB/s | 19 kB 00:00 2024-09-11T01:27:50.618 INFO:teuthology.orchestra.run.smithi012.stdout:(3/13): ceph-mgr-rook-19.3.0-4854.g8f219c47.el9 286 kB/s | 49 kB 00:00 2024-09-11T01:27:50.710 INFO:teuthology.orchestra.run.smithi012.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 222 kB 00:00 2024-09-11T01:27:50.735 INFO:teuthology.orchestra.run.smithi012.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 159 kB 00:00 2024-09-11T01:27:50.769 INFO:teuthology.orchestra.run.smithi012.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 916 kB/s | 54 kB 00:00 2024-09-11T01:27:50.802 INFO:teuthology.orchestra.run.smithi012.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-11T01:27:50.827 INFO:teuthology.orchestra.run.smithi012.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.3 MB/s | 279 kB 00:00 2024-09-11T01:27:50.852 INFO:teuthology.orchestra.run.smithi012.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 173 kB/s | 14 kB 00:00 2024-09-11T01:27:50.894 INFO:teuthology.orchestra.run.smithi012.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-09-11T01:27:50.920 INFO:teuthology.orchestra.run.smithi012.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.9 MB/s | 220 kB 00:00 2024-09-11T01:27:50.945 INFO:teuthology.orchestra.run.smithi012.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-11T01:27:50.968 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-11T01:27:50.995 INFO:teuthology.orchestra.run.smithi012.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2024-09-11T01:27:50.998 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:27:50.998 INFO:teuthology.orchestra.run.smithi012.stdout:Total 2.5 MB/s | 2.2 MB 00:00 2024-09-11T01:27:51.057 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-11T01:27:51.066 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-09-11T01:27:51.082 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-09-11T01:27:51.082 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-09-11T01:27:51.125 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-11T01:27:51.150 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 6/6 2024-09-11T01:27:51.180 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-09-11T01:27:51.181 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-09-11T01:27:51.830 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-09-11T01:27:51.935 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 6/6 2024-09-11T01:27:51.935 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 1/6 2024-09-11T01:27:51.935 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-11T01:27:51.935 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-11T01:27:51.935 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-11T01:27:51.935 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-11T01:27:52.002 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-11T01:27:52.065 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-11T01:27:52.122 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-11T01:27:52.156 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-11T01:27:52.207 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-11T01:27:52.213 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-11T01:27:52.214 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:52.214 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-11T01:27:52.214 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T01:27:52.214 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-11T01:27:52.214 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-11T01:27:52.214 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-11T01:27:52.214 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-11T01:27:52.214 INFO:teuthology.orchestra.run.smithi071.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-11T01:27:52.214 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:52.214 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-11T01:27:52.313 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-11T01:27:52.424 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-11T01:27:52.483 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-fuse 2024-09-11T01:27:52.484 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-11T01:27:53.045 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:31 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:27:53.175 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-11T01:27:53.176 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:53.176 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-09-11T01:27:53.176 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:53.176 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-11T01:27:53.176 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 900 k 2024-09-11T01:27:53.176 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-09-11T01:27:53.176 INFO:teuthology.orchestra.run.smithi071.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-11T01:27:53.176 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:53.176 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-11T01:27:53.176 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:53.176 INFO:teuthology.orchestra.run.smithi071.stdout:Install 2 Packages 2024-09-11T01:27:53.177 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:53.177 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 980 k 2024-09-11T01:27:53.177 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 2.9 M 2024-09-11T01:27:53.177 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-11T01:27:53.251 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-11T01:27:53.297 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-11T01:27:53.330 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-11T01:27:53.379 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-11T01:27:53.408 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 13/13 2024-09-11T01:27:53.571 INFO:teuthology.orchestra.run.smithi071.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 680 kB/s | 80 kB 00:00 2024-09-11T01:27:53.871 INFO:teuthology.orchestra.run.smithi071.stdout:(2/2): ceph-fuse-19.3.0-4854.g8f219c47.el9.x86_ 2.1 MB/s | 900 kB 00:00 2024-09-11T01:27:53.872 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:27:53.872 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.4 MB/s | 980 kB 00:00 2024-09-11T01:27:53.884 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-11T01:27:53.907 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-11T01:27:53.907 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-11T01:27:53.987 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-11T01:27:53.987 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-11T01:27:54.172 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-11T01:27:54.325 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-11T01:27:54.386 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 2/2 2024-09-11T01:27:54.393 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 13/13 2024-09-11T01:27:54.393 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 1/13 2024-09-11T01:27:54.394 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-11T01:27:54.394 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-11T01:27:54.394 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-11T01:27:54.394 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-11T01:27:54.394 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-11T01:27:54.394 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-11T01:27:54.395 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-11T01:27:54.395 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-11T01:27:54.395 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-11T01:27:54.395 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-11T01:27:54.395 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-11T01:27:54.740 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-11T01:27:54.740 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:54.740 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-09-11T01:27:54.740 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T01:27:54.740 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-11T01:27:54.741 INFO:teuthology.orchestra.run.smithi012.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-11T01:27:54.741 INFO:teuthology.orchestra.run.smithi012.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-11T01:27:54.741 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-11T01:27:54.741 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-11T01:27:54.741 INFO:teuthology.orchestra.run.smithi012.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-11T01:27:54.741 INFO:teuthology.orchestra.run.smithi012.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-11T01:27:54.741 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-11T01:27:54.741 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-11T01:27:54.741 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-11T01:27:54.741 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-11T01:27:54.741 INFO:teuthology.orchestra.run.smithi012.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-11T01:27:54.742 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:54.742 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-09-11T01:27:54.978 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-mgr-cephadm 2024-09-11T01:27:55.088 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 2/2 2024-09-11T01:27:55.088 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 1/2 2024-09-11T01:27:55.348 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-11T01:27:55.349 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:55.349 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-11T01:27:55.349 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-11T01:27:55.349 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:55.349 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-11T01:27:55.568 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:33 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:27:55.625 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-volume 2024-09-11T01:27:55.710 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-09-11T01:27:55.711 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:27:55.711 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-09-11T01:27:55.711 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:27:55.711 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-09-11T01:27:55.711 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 154 k 2024-09-11T01:27:55.711 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-09-11T01:27:55.711 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-11T01:27:55.712 INFO:teuthology.orchestra.run.smithi012.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-11T01:27:55.712 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-11T01:27:55.712 INFO:teuthology.orchestra.run.smithi012.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-11T01:27:55.712 INFO:teuthology.orchestra.run.smithi012.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-11T01:27:55.712 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:55.712 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-09-11T01:27:55.713 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:27:55.713 INFO:teuthology.orchestra.run.smithi012.stdout:Install 6 Packages 2024-09-11T01:27:55.713 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:27:55.714 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 7.0 M 2024-09-11T01:27:55.714 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 33 M 2024-09-11T01:27:55.714 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-09-11T01:27:55.980 INFO:teuthology.orchestra.run.smithi012.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4854.g8f219c47.e 728 kB/s | 154 kB 00:00 2024-09-11T01:27:56.047 INFO:teuthology.orchestra.run.smithi012.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 10 MB/s | 548 kB 00:00 2024-09-11T01:27:56.072 INFO:teuthology.orchestra.run.smithi012.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.2 MB/s | 58 kB 00:00 2024-09-11T01:27:56.097 INFO:teuthology.orchestra.run.smithi012.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 3.1 MB/s | 74 kB 00:00 2024-09-11T01:27:56.206 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:34 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:27:56.347 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-11T01:27:56.348 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:56.348 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-09-11T01:27:56.348 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:56.348 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-11T01:27:56.348 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 285 k 2024-09-11T01:27:56.348 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:56.348 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-11T01:27:56.349 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:56.349 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-09-11T01:27:56.349 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:56.349 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 285 k 2024-09-11T01:27:56.349 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 1.3 M 2024-09-11T01:27:56.349 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-11T01:27:56.548 INFO:teuthology.orchestra.run.smithi012.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 318 kB/s | 248 kB 00:00 2024-09-11T01:27:56.577 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-volume-19.3.0-4854.g8f219c47.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-11T01:27:56.577 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:27:56.577 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-11T01:27:56.577 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-11T01:27:56.585 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-11T01:27:56.585 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-11T01:27:56.612 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-11T01:27:56.613 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-11T01:27:56.847 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-11T01:27:56.919 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 1/1 2024-09-11T01:27:56.935 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 1/1 2024-09-11T01:27:56.936 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T01:27:56.936 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-11T01:27:56.936 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:57.897 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 1/1 2024-09-11T01:27:57.897 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:57.897 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-11T01:27:57.897 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T01:27:57.897 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:57.897 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-11T01:27:58.188 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados-devel 2024-09-11T01:27:58.766 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:36 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:27:58.901 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-11T01:27:58.902 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:58.902 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-09-11T01:27:58.902 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:58.902 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-11T01:27:58.902 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 127 k 2024-09-11T01:27:58.902 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:58.902 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-11T01:27:58.902 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:27:58.902 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-09-11T01:27:58.902 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:27:58.903 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 127 k 2024-09-11T01:27:58.903 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 456 k 2024-09-11T01:27:58.903 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-11T01:27:59.088 INFO:teuthology.orchestra.run.smithi071.stdout:librados-devel-19.3.0-4854.g8f219c47.el9.x86_64 685 kB/s | 127 kB 00:00 2024-09-11T01:27:59.089 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:27:59.089 INFO:teuthology.orchestra.run.smithi071.stdout:Total 679 kB/s | 127 kB 00:00 2024-09-11T01:27:59.089 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-11T01:27:59.094 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-11T01:27:59.094 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-11T01:27:59.163 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-11T01:27:59.163 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-11T01:27:59.322 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-11T01:27:59.427 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librados-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:27:59.853 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:00.097 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:00.097 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:00.097 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-11T01:28:00.097 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:28:00.097 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:00.098 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-11T01:28:00.349 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs2 2024-09-11T01:28:00.921 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:38 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:28:01.011 INFO:teuthology.orchestra.run.smithi071.stdout:Package libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T01:28:01.047 INFO:teuthology.orchestra.run.smithi012.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 1.1 MB/s | 6.0 MB 00:05 2024-09-11T01:28:01.049 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:28:01.050 INFO:teuthology.orchestra.run.smithi012.stdout:Total 1.3 MB/s | 7.0 MB 00:05 2024-09-11T01:28:01.055 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-11T01:28:01.056 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-11T01:28:01.056 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-11T01:28:01.127 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-09-11T01:28:01.140 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-09-11T01:28:01.140 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-09-11T01:28:01.153 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs-devel 2024-09-11T01:28:01.248 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-09-11T01:28:01.248 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-09-11T01:28:01.687 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-09-11T01:28:01.723 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:39 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:28:01.859 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-11T01:28:01.859 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:28:01.860 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-09-11T01:28:01.860 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:28:01.860 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-11T01:28:01.860 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 32 k 2024-09-11T01:28:01.860 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:01.860 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-11T01:28:01.860 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:28:01.860 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-09-11T01:28:01.860 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:01.860 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 32 k 2024-09-11T01:28:01.860 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 142 k 2024-09-11T01:28:01.860 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-11T01:28:01.871 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-11T01:28:01.925 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-11T01:28:02.008 INFO:teuthology.orchestra.run.smithi071.stdout:libcephfs-devel-19.3.0-4854.g8f219c47.el9.x86_6 216 kB/s | 32 kB 00:00 2024-09-11T01:28:02.008 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:28:02.008 INFO:teuthology.orchestra.run.smithi071.stdout:Total 214 kB/s | 32 kB 00:00 2024-09-11T01:28:02.008 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-11T01:28:02.012 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-11T01:28:02.012 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-11T01:28:02.030 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-11T01:28:02.030 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-11T01:28:02.123 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-11T01:28:02.238 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:02.488 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-11T01:28:02.580 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-11T01:28:02.595 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:02.657 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-11T01:28:02.688 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 6/6 2024-09-11T01:28:02.827 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:02.828 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:02.828 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-11T01:28:02.828 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:28:02.828 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:02.828 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-11T01:28:03.006 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados2 2024-09-11T01:28:03.456 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 6/6 2024-09-11T01:28:03.456 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 1/6 2024-09-11T01:28:03.456 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-11T01:28:03.456 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-11T01:28:03.456 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-11T01:28:03.456 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-11T01:28:03.597 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:41 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:28:03.691 INFO:teuthology.orchestra.run.smithi071.stdout:Package librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T01:28:03.736 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-11T01:28:03.737 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-11T01:28:03.737 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-11T01:28:03.748 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-11T01:28:03.748 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:03.748 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-09-11T01:28:03.748 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T01:28:03.748 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-11T01:28:03.748 INFO:teuthology.orchestra.run.smithi012.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-11T01:28:03.749 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-11T01:28:03.749 INFO:teuthology.orchestra.run.smithi012.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-11T01:28:03.749 INFO:teuthology.orchestra.run.smithi012.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-11T01:28:03.749 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:03.749 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-09-11T01:28:03.812 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librbd1 2024-09-11T01:28:03.995 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-fuse 2024-09-11T01:28:04.400 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:42 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:28:04.492 INFO:teuthology.orchestra.run.smithi071.stdout:Package librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T01:28:04.535 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-11T01:28:04.536 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-11T01:28:04.536 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-11T01:28:04.603 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:42 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:28:04.619 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rados 2024-09-11T01:28:04.744 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-09-11T01:28:04.745 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:28:04.745 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-09-11T01:28:04.745 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:28:04.745 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-09-11T01:28:04.745 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-fuse x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 900 k 2024-09-11T01:28:04.745 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-09-11T01:28:04.745 INFO:teuthology.orchestra.run.smithi012.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-11T01:28:04.745 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:04.745 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-09-11T01:28:04.745 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:28:04.745 INFO:teuthology.orchestra.run.smithi012.stdout:Install 2 Packages 2024-09-11T01:28:04.746 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:04.746 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 980 k 2024-09-11T01:28:04.746 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 2.9 M 2024-09-11T01:28:04.746 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-09-11T01:28:04.891 INFO:teuthology.orchestra.run.smithi012.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 649 kB/s | 80 kB 00:00 2024-09-11T01:28:05.032 INFO:teuthology.orchestra.run.smithi012.stdout:(2/2): ceph-fuse-19.3.0-4854.g8f219c47.el9.x86_ 3.3 MB/s | 900 kB 00:00 2024-09-11T01:28:05.033 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:28:05.033 INFO:teuthology.orchestra.run.smithi012.stdout:Total 3.3 MB/s | 980 kB 00:00 2024-09-11T01:28:05.045 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-09-11T01:28:05.068 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-09-11T01:28:05.068 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-09-11T01:28:05.154 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-09-11T01:28:05.154 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-09-11T01:28:05.182 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:43 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:28:05.273 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rados-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T01:28:05.317 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-11T01:28:05.318 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-11T01:28:05.318 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-11T01:28:05.350 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-09-11T01:28:05.389 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rgw 2024-09-11T01:28:05.497 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-11T01:28:05.568 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 2/2 2024-09-11T01:28:05.956 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:43 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:28:06.046 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rgw-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T01:28:06.090 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-11T01:28:06.090 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-11T01:28:06.091 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-11T01:28:06.168 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-cephfs 2024-09-11T01:28:06.262 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 2/2 2024-09-11T01:28:06.262 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 1/2 2024-09-11T01:28:06.555 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-11T01:28:06.555 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:06.555 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-09-11T01:28:06.555 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-11T01:28:06.555 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:06.555 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-09-11T01:28:06.736 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:44 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:28:06.804 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-volume 2024-09-11T01:28:06.827 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-cephfs-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T01:28:06.870 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-11T01:28:06.871 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-11T01:28:06.871 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-11T01:28:06.944 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rbd 2024-09-11T01:28:07.401 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:45 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:28:07.511 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:45 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:28:07.539 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-09-11T01:28:07.540 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:28:07.540 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-09-11T01:28:07.540 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:28:07.540 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-09-11T01:28:07.540 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-volume noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 285 k 2024-09-11T01:28:07.541 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:07.541 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-09-11T01:28:07.541 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:28:07.541 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2024-09-11T01:28:07.541 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:07.541 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 285 k 2024-09-11T01:28:07.541 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 1.3 M 2024-09-11T01:28:07.541 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-09-11T01:28:07.601 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rbd-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T01:28:07.644 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-11T01:28:07.645 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-11T01:28:07.645 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-11T01:28:07.720 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-fuse 2024-09-11T01:28:07.770 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-volume-19.3.0-4854.g8f219c47.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-11T01:28:07.771 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:28:07.771 INFO:teuthology.orchestra.run.smithi012.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-11T01:28:07.771 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-09-11T01:28:07.780 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-09-11T01:28:07.780 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-09-11T01:28:07.807 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-09-11T01:28:07.808 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-09-11T01:28:08.038 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-09-11T01:28:08.106 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 1/1 2024-09-11T01:28:08.123 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 1/1 2024-09-11T01:28:08.123 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T01:28:08.123 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-11T01:28:08.123 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:08.293 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:46 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:28:08.426 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-11T01:28:08.427 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:28:08.427 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-09-11T01:28:08.427 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:28:08.427 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-11T01:28:08.427 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 85 k 2024-09-11T01:28:08.427 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:08.427 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-11T01:28:08.427 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:28:08.427 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-09-11T01:28:08.427 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:08.428 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 85 k 2024-09-11T01:28:08.428 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 231 k 2024-09-11T01:28:08.428 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-11T01:28:08.744 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-fuse-19.3.0-4854.g8f219c47.el9.x86_64.rpm 270 kB/s | 85 kB 00:00 2024-09-11T01:28:08.744 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:28:08.745 INFO:teuthology.orchestra.run.smithi071.stdout:Total 268 kB/s | 85 kB 00:00 2024-09-11T01:28:08.745 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-11T01:28:08.752 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-11T01:28:08.752 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-11T01:28:08.818 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-11T01:28:08.818 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-11T01:28:08.977 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-11T01:28:09.091 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:09.202 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 1/1 2024-09-11T01:28:09.202 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:09.202 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-09-11T01:28:09.203 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T01:28:09.203 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:09.203 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-09-11T01:28:09.459 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:09.479 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install librados-devel 2024-09-11T01:28:09.710 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:09.710 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:09.711 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-11T01:28:09.711 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:28:09.711 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:09.711 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-11T01:28:09.970 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-mirror 2024-09-11T01:28:10.077 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:48 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:28:10.218 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-09-11T01:28:10.218 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:28:10.218 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repo Size 2024-09-11T01:28:10.219 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:28:10.219 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-09-11T01:28:10.219 INFO:teuthology.orchestra.run.smithi012.stdout: librados-devel x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 127 k 2024-09-11T01:28:10.219 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:10.219 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-09-11T01:28:10.219 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:28:10.219 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2024-09-11T01:28:10.219 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:10.219 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 127 k 2024-09-11T01:28:10.219 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 456 k 2024-09-11T01:28:10.219 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-09-11T01:28:10.400 INFO:teuthology.orchestra.run.smithi012.stdout:librados-devel-19.3.0-4854.g8f219c47.el9.x86_64 702 kB/s | 127 kB 00:00 2024-09-11T01:28:10.400 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:28:10.401 INFO:teuthology.orchestra.run.smithi012.stdout:Total 697 kB/s | 127 kB 00:00 2024-09-11T01:28:10.401 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-09-11T01:28:10.407 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-09-11T01:28:10.407 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-09-11T01:28:10.481 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-09-11T01:28:10.481 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-09-11T01:28:10.548 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:48 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:28:10.642 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-09-11T01:28:10.686 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-11T01:28:10.686 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:28:10.686 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-09-11T01:28:10.686 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:28:10.686 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-11T01:28:10.686 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 3.2 M 2024-09-11T01:28:10.687 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:10.687 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-11T01:28:10.687 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:28:10.687 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-09-11T01:28:10.687 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:10.687 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 3.2 M 2024-09-11T01:28:10.688 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 14 M 2024-09-11T01:28:10.688 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-11T01:28:10.743 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librados-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:11.149 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:11.413 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-mirror-19.3.0-4854.g8f219c47.el9.x86_64.rpm 4.4 MB/s | 3.2 MB 00:00 2024-09-11T01:28:11.413 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:28:11.414 INFO:teuthology.orchestra.run.smithi071.stdout:Total 4.4 MB/s | 3.2 MB 00:00 2024-09-11T01:28:11.414 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-11T01:28:11.424 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-11T01:28:11.424 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-11T01:28:11.507 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-11T01:28:11.508 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-11T01:28:11.545 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:11.545 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:11.545 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-09-11T01:28:11.545 INFO:teuthology.orchestra.run.smithi012.stdout: librados-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:28:11.545 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:11.545 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-09-11T01:28:11.788 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install libcephfs2 2024-09-11T01:28:11.945 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-11T01:28:12.014 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:12.039 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:12.039 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T01:28:12.039 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-11T01:28:12.039 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T01:28:12.039 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T01:28:12.039 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:12.378 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:50 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:28:12.472 INFO:teuthology.orchestra.run.smithi012.stdout:Package libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T01:28:12.519 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-09-11T01:28:12.520 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2024-09-11T01:28:12.520 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-09-11T01:28:12.583 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install libcephfs-devel 2024-09-11T01:28:13.120 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:13.120 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:13.120 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-11T01:28:13.120 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:28:13.120 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:13.120 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-11T01:28:13.174 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:51 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:28:13.313 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-09-11T01:28:13.314 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:28:13.314 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repo Size 2024-09-11T01:28:13.315 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:28:13.315 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-09-11T01:28:13.315 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-devel x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 32 k 2024-09-11T01:28:13.315 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:13.315 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-09-11T01:28:13.315 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:28:13.315 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2024-09-11T01:28:13.315 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:13.315 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 32 k 2024-09-11T01:28:13.315 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 142 k 2024-09-11T01:28:13.315 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-09-11T01:28:13.382 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-nbd 2024-09-11T01:28:13.461 INFO:teuthology.orchestra.run.smithi012.stdout:libcephfs-devel-19.3.0-4854.g8f219c47.el9.x86_6 217 kB/s | 32 kB 00:00 2024-09-11T01:28:13.462 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:28:13.462 INFO:teuthology.orchestra.run.smithi012.stdout:Total 216 kB/s | 32 kB 00:00 2024-09-11T01:28:13.462 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-09-11T01:28:13.466 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-09-11T01:28:13.466 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-09-11T01:28:13.495 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-09-11T01:28:13.496 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-09-11T01:28:13.590 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-09-11T01:28:13.703 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephfs-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:13.968 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:51 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:28:14.096 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:14.101 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-11T01:28:14.102 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:28:14.102 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-09-11T01:28:14.102 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:28:14.102 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-11T01:28:14.102 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 171 k 2024-09-11T01:28:14.102 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:14.102 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-11T01:28:14.102 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-11T01:28:14.102 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-09-11T01:28:14.103 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:14.103 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 171 k 2024-09-11T01:28:14.103 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 490 k 2024-09-11T01:28:14.103 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-11T01:28:14.316 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-nbd-19.3.0-4854.g8f219c47.el9.x86_64.rpm 804 kB/s | 171 kB 00:00 2024-09-11T01:28:14.316 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:28:14.316 INFO:teuthology.orchestra.run.smithi071.stdout:Total 798 kB/s | 171 kB 00:00 2024-09-11T01:28:14.317 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-11T01:28:14.324 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-11T01:28:14.324 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-11T01:28:14.335 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephfs-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:14.335 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:14.335 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-09-11T01:28:14.335 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:28:14.335 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:14.335 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-09-11T01:28:14.388 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-11T01:28:14.389 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-11T01:28:14.509 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install librados2 2024-09-11T01:28:14.559 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-11T01:28:14.662 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-nbd-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:15.080 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:15.108 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:53 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:28:15.200 INFO:teuthology.orchestra.run.smithi012.stdout:Package librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T01:28:15.246 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-09-11T01:28:15.247 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2024-09-11T01:28:15.247 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-09-11T01:28:15.312 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install librbd1 2024-09-11T01:28:15.351 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-nbd-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:15.351 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:15.351 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-11T01:28:15.351 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:28:15.351 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:15.351 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-11T01:28:15.576 DEBUG:teuthology.parallel:result is None 2024-09-11T01:28:15.907 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:53 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:28:15.998 INFO:teuthology.orchestra.run.smithi012.stdout:Package librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T01:28:16.044 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-09-11T01:28:16.045 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2024-09-11T01:28:16.045 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-09-11T01:28:16.110 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install python3-rados 2024-09-11T01:28:16.704 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:54 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:28:16.797 INFO:teuthology.orchestra.run.smithi012.stdout:Package python3-rados-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T01:28:16.842 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-09-11T01:28:16.843 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2024-09-11T01:28:16.843 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-09-11T01:28:16.910 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install python3-rgw 2024-09-11T01:28:17.503 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:55 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:28:17.595 INFO:teuthology.orchestra.run.smithi012.stdout:Package python3-rgw-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T01:28:17.641 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-09-11T01:28:17.642 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2024-09-11T01:28:17.642 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-09-11T01:28:17.708 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install python3-cephfs 2024-09-11T01:28:18.301 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:56 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:28:18.394 INFO:teuthology.orchestra.run.smithi012.stdout:Package python3-cephfs-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T01:28:18.440 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-09-11T01:28:18.441 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2024-09-11T01:28:18.441 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-09-11T01:28:18.509 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install python3-rbd 2024-09-11T01:28:19.100 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:57 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:28:19.192 INFO:teuthology.orchestra.run.smithi012.stdout:Package python3-rbd-2:19.3.0-4854.g8f219c47.el9.x86_64 is already installed. 2024-09-11T01:28:19.239 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-09-11T01:28:19.240 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2024-09-11T01:28:19.240 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-09-11T01:28:19.298 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install rbd-fuse 2024-09-11T01:28:19.889 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:57 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:28:20.027 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-09-11T01:28:20.027 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:28:20.028 INFO:teuthology.orchestra.run.smithi012.stdout: Package Architecture Version Repository Size 2024-09-11T01:28:20.028 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:28:20.028 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-09-11T01:28:20.028 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-fuse x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 85 k 2024-09-11T01:28:20.028 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:20.028 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-09-11T01:28:20.028 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:28:20.028 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2024-09-11T01:28:20.028 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:20.028 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 85 k 2024-09-11T01:28:20.028 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 231 k 2024-09-11T01:28:20.029 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-09-11T01:28:20.230 INFO:teuthology.orchestra.run.smithi012.stdout:rbd-fuse-19.3.0-4854.g8f219c47.el9.x86_64.rpm 422 kB/s | 85 kB 00:00 2024-09-11T01:28:20.231 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:28:20.231 INFO:teuthology.orchestra.run.smithi012.stdout:Total 419 kB/s | 85 kB 00:00 2024-09-11T01:28:20.231 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-09-11T01:28:20.239 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-09-11T01:28:20.239 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-09-11T01:28:20.308 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-09-11T01:28:20.309 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-09-11T01:28:20.463 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-09-11T01:28:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:20.958 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:21.216 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:21.217 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:21.217 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-09-11T01:28:21.217 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:28:21.217 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:21.217 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-09-11T01:28:21.479 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install rbd-mirror 2024-09-11T01:28:22.071 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:02:00 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:28:22.214 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-09-11T01:28:22.215 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:28:22.215 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repo Size 2024-09-11T01:28:22.215 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:28:22.215 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-09-11T01:28:22.215 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-mirror x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 3.2 M 2024-09-11T01:28:22.215 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:22.216 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-09-11T01:28:22.216 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:28:22.216 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2024-09-11T01:28:22.216 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:22.216 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 3.2 M 2024-09-11T01:28:22.216 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 14 M 2024-09-11T01:28:22.216 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-09-11T01:28:22.966 INFO:teuthology.orchestra.run.smithi012.stdout:rbd-mirror-19.3.0-4854.g8f219c47.el9.x86_64.rpm 4.2 MB/s | 3.2 MB 00:00 2024-09-11T01:28:22.967 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:28:22.968 INFO:teuthology.orchestra.run.smithi012.stdout:Total 4.2 MB/s | 3.2 MB 00:00 2024-09-11T01:28:22.968 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-09-11T01:28:22.979 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-09-11T01:28:22.979 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-09-11T01:28:23.068 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-09-11T01:28:23.068 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-09-11T01:28:23.533 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-09-11T01:28:23.600 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:23.627 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:23.627 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T01:28:23.627 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-11T01:28:23.628 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T01:28:23.628 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T01:28:23.628 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:24.769 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:24.769 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:24.769 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-09-11T01:28:24.769 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:28:24.769 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:24.769 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-09-11T01:28:25.022 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install rbd-nbd 2024-09-11T01:28:25.619 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:02:03 ago on Wed 11 Sep 2024 01:26:22 AM UTC. 2024-09-11T01:28:25.757 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-09-11T01:28:25.758 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:28:25.758 INFO:teuthology.orchestra.run.smithi012.stdout: Package Architecture Version Repository Size 2024-09-11T01:28:25.758 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:28:25.759 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-09-11T01:28:25.759 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-nbd x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 171 k 2024-09-11T01:28:25.759 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:25.759 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-09-11T01:28:25.759 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-09-11T01:28:25.759 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2024-09-11T01:28:25.759 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:25.759 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 171 k 2024-09-11T01:28:25.759 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 490 k 2024-09-11T01:28:25.759 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-09-11T01:28:25.971 INFO:teuthology.orchestra.run.smithi012.stdout:rbd-nbd-19.3.0-4854.g8f219c47.el9.x86_64.rpm 806 kB/s | 171 kB 00:00 2024-09-11T01:28:25.971 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-09-11T01:28:25.971 INFO:teuthology.orchestra.run.smithi012.stdout:Total 800 kB/s | 171 kB 00:00 2024-09-11T01:28:25.972 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-09-11T01:28:25.980 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-09-11T01:28:25.980 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-09-11T01:28:26.049 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-09-11T01:28:26.049 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-09-11T01:28:26.208 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-09-11T01:28:26.313 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-nbd-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:26.729 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:26.998 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-nbd-2:19.3.0-4854.g8f219c47.el9.x86_64 1/1 2024-09-11T01:28:26.998 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:26.999 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-09-11T01:28:26.999 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-nbd-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T01:28:26.999 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:26.999 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-09-11T01:28:27.251 DEBUG:teuthology.parallel:result is None 2024-09-11T01:28:27.251 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T01:28:27.401 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-11T01:28:27.424 INFO:teuthology.orchestra.run.smithi012.stdout:19.3.0-4854.g8f219c47.el9 2024-09-11T01:28:27.424 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4854.g8f219c47.el9 2024-09-11T01:28:27.425 INFO:teuthology.task.install:The correct ceph version 19.3.0-4854.g8f219c47 is installed. 2024-09-11T01:28:27.426 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T01:28:27.570 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-11T01:28:27.596 INFO:teuthology.orchestra.run.smithi071.stdout:19.3.0-4854.g8f219c47.el9 2024-09-11T01:28:27.597 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4854.g8f219c47.el9 2024-09-11T01:28:27.597 INFO:teuthology.task.install:The correct ceph version 19.3.0-4854.g8f219c47 is installed. 2024-09-11T01:28:27.599 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-11T01:28:27.599 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-09-11T01:28:27.599 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T01:28:27.627 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-11T01:28:27.628 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T01:28:27.666 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-11T01:28:27.666 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-09-11T01:28:27.666 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T01:28:27.695 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T01:28:27.762 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-11T01:28:27.762 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T01:28:27.790 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T01:28:27.857 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-11T01:28:27.858 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-09-11T01:28:27.858 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T01:28:27.885 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T01:28:27.952 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-11T01:28:27.952 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T01:28:27.980 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T01:28:28.045 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-11T01:28:28.046 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-09-11T01:28:28.046 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T01:28:28.072 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T01:28:28.138 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-11T01:28:28.139 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T01:28:28.164 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T01:28:28.229 INFO:teuthology.run_tasks:Running task ceph... 2024-09-11T01:28:28.307 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-11T01:28:28.308 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 777 /var/log/ceph 2024-09-11T01:28:28.310 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 777 /var/log/ceph 2024-09-11T01:28:28.334 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-11T01:28:28.334 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T01:28:28.377 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T01:28:28.402 INFO:tasks.ceph:Creating extra log directories... 2024-09-11T01:28:28.402 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T01:28:28.445 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T01:28:28.477 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-11T01:28:28.478 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-11T01:28:28.478 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7899900', 'branch': 'wip-hemanth-testing-2024-09-10-0723', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest} thrashers/one workloads/snaps-few-objects}', 'email': 'hyelloji@redhat.com', 'first_in_suite': False, 'job_id': '7899900', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-hemanth-testing-2024-09-10-0723'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-hemanth-testing-2024-09-10-0723', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}}, 'owner': 'scheduled_hyelloji@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 9167, 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-hemanth-testing-2024-09-10-0723', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'targets': {'smithi012.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKrxytsmSSWI6nG56dznjsc7P5wsT+aWf19CmdWPPjQvnFGVtRI5KzbHYneuAfFwoeT0z6dPRmlrSVwr7N0BbyM=', 'smithi071.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD/5BxFyq5H567Msv+LIeSsl8qqupOIJFr3bEQaRpg7Mo80GuJRdpGbPh94XT92tRK3LA2yIr6v1sWfpfsRZiJs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 20, 'thrash_delay': 1}}, {'rados': {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 50, 'delete': 50, 'read': 100, 'rollback': 50, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_17:16:14', 'tube': 'smithi', 'user': 'hyelloji', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-11T01:28:28.478 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-11T01:28:28.513 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-11T01:28:28.528 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T01:28:28.570 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T01:28:28.595 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-09-11T01:28:28.595 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T01:28:28.652 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T01:28:28.653 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_1 2024-09-11T01:28:28.709 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T01:28:28.709 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T01:28:28.710 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-11T01:28:28.710 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T01:28:28.710 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T01:28:28.710 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-09-11 01:28:23.886104733 +0000 2024-09-11T01:28:28.710 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-09-11 01:25:42.608937035 +0000 2024-09-11T01:28:28.710 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-09-11 01:25:42.608937035 +0000 2024-09-11T01:28:28.710 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-09-11 01:25:42.608937035 +0000 2024-09-11T01:28:28.710 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T01:28:28.777 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-09-11T01:28:28.777 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-09-11T01:28:28.777 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000281314 s, 1.8 MB/s 2024-09-11T01:28:28.779 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T01:28:28.836 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_2 2024-09-11T01:28:28.896 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T01:28:28.896 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T01:28:28.896 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-11T01:28:28.896 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T01:28:28.896 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T01:28:28.897 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-09-11 01:28:23.886104733 +0000 2024-09-11T01:28:28.897 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-09-11 01:25:42.948934475 +0000 2024-09-11T01:28:28.897 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-09-11 01:25:42.948934475 +0000 2024-09-11T01:28:28.897 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-09-11 01:25:42.948934475 +0000 2024-09-11T01:28:28.897 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T01:28:28.962 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-09-11T01:28:28.962 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-09-11T01:28:28.962 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000194128 s, 2.6 MB/s 2024-09-11T01:28:28.963 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T01:28:29.021 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_3 2024-09-11T01:28:29.078 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T01:28:29.078 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T01:28:29.078 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-11T01:28:29.078 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T01:28:29.078 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T01:28:29.078 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-09-11 01:28:23.887104726 +0000 2024-09-11T01:28:29.078 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-09-11 01:25:43.289931907 +0000 2024-09-11T01:28:29.078 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-09-11 01:25:43.289931907 +0000 2024-09-11T01:28:29.078 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-09-11 01:25:43.289931907 +0000 2024-09-11T01:28:29.079 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T01:28:29.144 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-09-11T01:28:29.144 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-09-11T01:28:29.144 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.00018038 s, 2.8 MB/s 2024-09-11T01:28:29.145 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T01:28:29.203 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_4 2024-09-11T01:28:29.260 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T01:28:29.260 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T01:28:29.260 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-11T01:28:29.260 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T01:28:29.260 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T01:28:29.260 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-09-11 01:28:23.887104726 +0000 2024-09-11T01:28:29.260 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-09-11 01:25:43.617929438 +0000 2024-09-11T01:28:29.260 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-09-11 01:25:43.617929438 +0000 2024-09-11T01:28:29.260 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-09-11 01:25:43.617929438 +0000 2024-09-11T01:28:29.261 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T01:28:29.326 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-09-11T01:28:29.326 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-09-11T01:28:29.326 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000191597 s, 2.7 MB/s 2024-09-11T01:28:29.327 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T01:28:29.384 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T01:28:29.385 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-11T01:28:29.385 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T01:28:29.404 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T01:28:29.404 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_1 2024-09-11T01:28:29.462 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T01:28:29.462 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T01:28:29.462 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-11T01:28:29.462 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T01:28:29.462 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T01:28:29.463 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-09-11 01:28:12.293282906 +0000 2024-09-11T01:28:29.463 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-09-11 01:25:42.322230125 +0000 2024-09-11T01:28:29.463 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-09-11 01:25:42.322230125 +0000 2024-09-11T01:28:29.463 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-09-11 01:25:42.322230125 +0000 2024-09-11T01:28:29.463 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T01:28:29.531 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-09-11T01:28:29.531 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-09-11T01:28:29.531 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000294851 s, 1.7 MB/s 2024-09-11T01:28:29.532 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T01:28:29.592 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_2 2024-09-11T01:28:29.650 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T01:28:29.650 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T01:28:29.650 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-11T01:28:29.651 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T01:28:29.651 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T01:28:29.651 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-09-11 01:28:12.293282906 +0000 2024-09-11T01:28:29.651 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-09-11 01:25:42.657223859 +0000 2024-09-11T01:28:29.651 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-09-11 01:25:42.657223859 +0000 2024-09-11T01:28:29.651 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-09-11 01:25:42.657223859 +0000 2024-09-11T01:28:29.651 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T01:28:29.716 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-09-11T01:28:29.716 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-09-11T01:28:29.716 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000192983 s, 2.7 MB/s 2024-09-11T01:28:29.718 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T01:28:29.777 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_3 2024-09-11T01:28:29.833 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T01:28:29.833 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T01:28:29.833 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-11T01:28:29.833 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T01:28:29.833 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T01:28:29.834 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-09-11 01:28:12.294282887 +0000 2024-09-11T01:28:29.834 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-09-11 01:25:42.992217591 +0000 2024-09-11T01:28:29.834 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-09-11 01:25:42.992217591 +0000 2024-09-11T01:28:29.834 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-09-11 01:25:42.992217591 +0000 2024-09-11T01:28:29.834 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T01:28:29.899 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-09-11T01:28:29.899 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-09-11T01:28:29.899 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000238347 s, 2.1 MB/s 2024-09-11T01:28:29.900 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T01:28:29.957 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_4 2024-09-11T01:28:30.014 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T01:28:30.014 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T01:28:30.014 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-11T01:28:30.014 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T01:28:30.014 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T01:28:30.014 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-09-11 01:28:12.294282887 +0000 2024-09-11T01:28:30.015 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-09-11 01:25:43.311211624 +0000 2024-09-11T01:28:30.015 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-09-11 01:25:43.311211624 +0000 2024-09-11T01:28:30.015 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-09-11 01:25:43.311211624 +0000 2024-09-11T01:28:30.015 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T01:28:30.078 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-09-11T01:28:30.078 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-09-11T01:28:30.078 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.00019495 s, 2.6 MB/s 2024-09-11T01:28:30.079 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T01:28:30.136 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-11T01:28:30.136 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi012.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-11T01:28:30.136 INFO:tasks.ceph:Generating config... 2024-09-11T01:28:30.138 INFO:tasks.ceph:[client] debug monc = 20 2024-09-11T01:28:30.138 INFO:tasks.ceph:[client] debug ms = 1 2024-09-11T01:28:30.138 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-11T01:28:30.138 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-11T01:28:30.138 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-09-11T01:28:30.138 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-09-11T01:28:30.138 INFO:tasks.ceph:[global] ms inject delay type = mon 2024-09-11T01:28:30.138 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-09-11T01:28:30.138 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-09-11T01:28:30.138 INFO:tasks.ceph:[global] ms type = async 2024-09-11T01:28:30.138 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-11T01:28:30.139 INFO:tasks.ceph:[mgr] debug monc = 10 2024-09-11T01:28:30.139 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-11T01:28:30.139 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-11T01:28:30.139 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-11T01:28:30.139 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-11T01:28:30.139 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-11T01:28:30.139 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-11T01:28:30.139 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-11T01:28:30.139 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-11T01:28:30.139 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-11T01:28:30.139 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-11T01:28:30.139 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-11T01:28:30.139 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-11T01:28:30.139 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-11T01:28:30.140 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-09-11T01:28:30.140 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-11T01:28:30.140 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-11T01:28:30.140 INFO:tasks.ceph:[osd] bluestore write v2 random = True 2024-09-11T01:28:30.140 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-11T01:28:30.140 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-11T01:28:30.140 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-11T01:28:30.140 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-11T01:28:30.140 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-11T01:28:30.140 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-11T01:28:30.140 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-11T01:28:30.140 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-11T01:28:30.140 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-11T01:28:30.140 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-11T01:28:30.140 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-11T01:28:30.141 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-11T01:28:30.141 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-11T01:28:30.141 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-11T01:28:30.141 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-11T01:28:30.141 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-11T01:28:30.141 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-11T01:28:30.141 INFO:tasks.ceph:Setting up mon.a... 2024-09-11T01:28:30.141 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-11T01:28:30.199 INFO:teuthology.orchestra.run.smithi012.stdout:creating /etc/ceph/ceph.keyring 2024-09-11T01:28:30.202 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-11T01:28:30.249 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T01:28:30.275 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.12'), ('mon.c', '[v2:172.21.15.12:3301,v1:172.21.15.12:6790]'), ('mon.b', '172.21.15.71')] 2024-09-11T01:28:30.276 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.12,[v2:172.21.15.12:3301,v1:172.21.15.12:6790],172.21.15.71', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-11T01:28:30.277 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-09-11T01:28:30.277 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T01:28:30.332 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.12 --addv c '[v2:172.21.15.12:3301,v1:172.21.15.12:6790]' --add b 172.21.15.71 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-11T01:28:30.422 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-11T01:28:30.422 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: generated fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:28:30.422 INFO:teuthology.orchestra.run.smithi012.stdout:setting min_mon_release = quincy 2024-09-11T01:28:30.422 INFO:teuthology.orchestra.run.smithi012.stdout:epoch 0 2024-09-11T01:28:30.422 INFO:teuthology.orchestra.run.smithi012.stdout:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:28:30.422 INFO:teuthology.orchestra.run.smithi012.stdout:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:28:30.423 INFO:teuthology.orchestra.run.smithi012.stdout:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:28:30.423 INFO:teuthology.orchestra.run.smithi012.stdout:min_mon_release 17 (quincy) 2024-09-11T01:28:30.423 INFO:teuthology.orchestra.run.smithi012.stdout:election_strategy: 1 2024-09-11T01:28:30.423 INFO:teuthology.orchestra.run.smithi012.stdout:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:28:30.423 INFO:teuthology.orchestra.run.smithi012.stdout:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:28:30.423 INFO:teuthology.orchestra.run.smithi012.stdout:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:28:30.423 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-11T01:28:30.424 DEBUG:teuthology.orchestra.run.smithi012:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T01:28:30.440 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab... 2024-09-11T01:28:30.441 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-11T01:28:30.483 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-11T01:28:30.525 INFO:teuthology.orchestra.run.smithi012.stdout:[global] 2024-09-11T01:28:30.525 INFO:teuthology.orchestra.run.smithi012.stdout: chdir = "" 2024-09-11T01:28:30.525 INFO:teuthology.orchestra.run.smithi012.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T01:28:30.525 INFO:teuthology.orchestra.run.smithi012.stdout: auth supported = cephx 2024-09-11T01:28:30.526 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.526 INFO:teuthology.orchestra.run.smithi012.stdout: filestore xattr use omap = true 2024-09-11T01:28:30.526 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.526 INFO:teuthology.orchestra.run.smithi012.stdout: mon clock drift allowed = 1.000 2024-09-11T01:28:30.526 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.526 INFO:teuthology.orchestra.run.smithi012.stdout: osd crush chooseleaf type = 0 2024-09-11T01:28:30.526 INFO:teuthology.orchestra.run.smithi012.stdout: auth debug = true 2024-09-11T01:28:30.526 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.526 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on old message = true 2024-09-11T01:28:30.526 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on bug = true 2024-09-11T01:28:30.526 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.526 INFO:teuthology.orchestra.run.smithi012.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T01:28:30.526 INFO:teuthology.orchestra.run.smithi012.stdout: mon pg warn max object skew = 0 2024-09-11T01:28:30.526 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.527 INFO:teuthology.orchestra.run.smithi012.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T01:28:30.527 INFO:teuthology.orchestra.run.smithi012.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T01:28:30.527 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.527 INFO:teuthology.orchestra.run.smithi012.stdout: osd pool default size = 2 2024-09-11T01:28:30.527 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.527 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd allow primary affinity = true 2024-09-11T01:28:30.527 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd allow pg remap = true 2024-09-11T01:28:30.527 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on legacy crush tunables = false 2024-09-11T01:28:30.527 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on crush straw calc version zero = false 2024-09-11T01:28:30.527 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on no sortbitwise = false 2024-09-11T01:28:30.527 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on osd down out interval zero = false 2024-09-11T01:28:30.527 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on too few osds = false 2024-09-11T01:28:30.527 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T01:28:30.527 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T01:28:30.528 INFO:teuthology.orchestra.run.smithi012.stdout: mon_allow_pool_size_one = true 2024-09-11T01:28:30.528 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.528 INFO:teuthology.orchestra.run.smithi012.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T01:28:30.528 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.528 INFO:teuthology.orchestra.run.smithi012.stdout: osd default data pool replay window = 5 2024-09-11T01:28:30.528 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.528 INFO:teuthology.orchestra.run.smithi012.stdout: mon allow pool delete = true 2024-09-11T01:28:30.528 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.528 INFO:teuthology.orchestra.run.smithi012.stdout: mon cluster log file level = debug 2024-09-11T01:28:30.528 INFO:teuthology.orchestra.run.smithi012.stdout: debug asserts on shutdown = true 2024-09-11T01:28:30.528 INFO:teuthology.orchestra.run.smithi012.stdout: mon health detail to clog = false 2024-09-11T01:28:30.528 INFO:teuthology.orchestra.run.smithi012.stdout: mon host = "172.21.15.12,[v2:172.21.15.12:3301,v1:172.21.15.12:6790],172.21.15.71" 2024-09-11T01:28:30.528 INFO:teuthology.orchestra.run.smithi012.stdout: mon client directed command retry = 5 2024-09-11T01:28:30.528 INFO:teuthology.orchestra.run.smithi012.stdout: mon election default strategy = 1 2024-09-11T01:28:30.528 INFO:teuthology.orchestra.run.smithi012.stdout: ms inject delay max = 1 2024-09-11T01:28:30.528 INFO:teuthology.orchestra.run.smithi012.stdout: ms inject delay probability = 0.005 2024-09-11T01:28:30.529 INFO:teuthology.orchestra.run.smithi012.stdout: ms inject delay type = mon 2024-09-11T01:28:30.529 INFO:teuthology.orchestra.run.smithi012.stdout: ms inject internal delays = 0.002 2024-09-11T01:28:30.529 INFO:teuthology.orchestra.run.smithi012.stdout: ms inject socket failures = 2500 2024-09-11T01:28:30.529 INFO:teuthology.orchestra.run.smithi012.stdout: ms type = async 2024-09-11T01:28:30.529 INFO:teuthology.orchestra.run.smithi012.stdout: fsid = ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:28:30.529 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.529 INFO:teuthology.orchestra.run.smithi012.stdout:[osd] 2024-09-11T01:28:30.529 INFO:teuthology.orchestra.run.smithi012.stdout: osd journal size = 100 2024-09-11T01:28:30.529 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.529 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub load threshold = 5.0 2024-09-11T01:28:30.529 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub max interval = 600 2024-09-11T01:28:30.529 INFO:teuthology.orchestra.run.smithi012.stdout: osd mclock profile = high_recovery_ops 2024-09-11T01:28:30.529 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.529 INFO:teuthology.orchestra.run.smithi012.stdout: osd recover clone overlap = true 2024-09-11T01:28:30.530 INFO:teuthology.orchestra.run.smithi012.stdout: osd recovery max chunk = 1048576 2024-09-11T01:28:30.530 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.530 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug shutdown = true 2024-09-11T01:28:30.530 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug op order = true 2024-09-11T01:28:30.530 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug verify stray on activate = true 2024-09-11T01:28:30.530 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug trim objects = true 2024-09-11T01:28:30.530 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.530 INFO:teuthology.orchestra.run.smithi012.stdout: osd open classes on start = true 2024-09-11T01:28:30.530 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug pg log writeout = true 2024-09-11T01:28:30.530 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.530 INFO:teuthology.orchestra.run.smithi012.stdout: osd deep scrub update digest min age = 30 2024-09-11T01:28:30.530 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.530 INFO:teuthology.orchestra.run.smithi012.stdout: osd map max advance = 10 2024-09-11T01:28:30.530 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.530 INFO:teuthology.orchestra.run.smithi012.stdout: journal zero on create = true 2024-09-11T01:28:30.531 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.531 INFO:teuthology.orchestra.run.smithi012.stdout: filestore ondisk finisher threads = 3 2024-09-11T01:28:30.531 INFO:teuthology.orchestra.run.smithi012.stdout: filestore apply finisher threads = 3 2024-09-11T01:28:30.531 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.531 INFO:teuthology.orchestra.run.smithi012.stdout: bdev debug aio = true 2024-09-11T01:28:30.531 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug misdirected ops = true 2024-09-11T01:28:30.531 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore block size = 96636764160 2024-09-11T01:28:30.531 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore compression algorithm = snappy 2024-09-11T01:28:30.531 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore compression mode = aggressive 2024-09-11T01:28:30.531 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore fsck on mount = True 2024-09-11T01:28:30.531 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore write v2 random = True 2024-09-11T01:28:30.531 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore zero block detection = True 2024-09-11T01:28:30.531 INFO:teuthology.orchestra.run.smithi012.stdout: debug bluefs = 20 2024-09-11T01:28:30.531 INFO:teuthology.orchestra.run.smithi012.stdout: debug bluestore = 20 2024-09-11T01:28:30.531 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-09-11T01:28:30.532 INFO:teuthology.orchestra.run.smithi012.stdout: debug osd = 20 2024-09-11T01:28:30.532 INFO:teuthology.orchestra.run.smithi012.stdout: debug rocksdb = 10 2024-09-11T01:28:30.532 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd backfillfull_ratio = 0.85 2024-09-11T01:28:30.532 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd full ratio = 0.9 2024-09-11T01:28:30.532 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd nearfull ratio = 0.8 2024-09-11T01:28:30.532 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug verify cached snaps = True 2024-09-11T01:28:30.532 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug verify missing on start = True 2024-09-11T01:28:30.532 INFO:teuthology.orchestra.run.smithi012.stdout: osd failsafe full ratio = 0.95 2024-09-11T01:28:30.532 INFO:teuthology.orchestra.run.smithi012.stdout: osd mclock override recovery settings = True 2024-09-11T01:28:30.532 INFO:teuthology.orchestra.run.smithi012.stdout: osd objectstore = bluestore 2024-09-11T01:28:30.532 INFO:teuthology.orchestra.run.smithi012.stdout: osd op queue = debug_random 2024-09-11T01:28:30.532 INFO:teuthology.orchestra.run.smithi012.stdout: osd op queue cut off = debug_random 2024-09-11T01:28:30.532 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.532 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr] 2024-09-11T01:28:30.533 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-09-11T01:28:30.533 INFO:teuthology.orchestra.run.smithi012.stdout: debug mgr = 20 2024-09-11T01:28:30.533 INFO:teuthology.orchestra.run.smithi012.stdout: debug mon = 20 2024-09-11T01:28:30.533 INFO:teuthology.orchestra.run.smithi012.stdout: debug auth = 20 2024-09-11T01:28:30.533 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min pgs per osd = 4 2024-09-11T01:28:30.533 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min bytes per osd = 10 2024-09-11T01:28:30.533 INFO:teuthology.orchestra.run.smithi012.stdout: mgr/telemetry/nag = false 2024-09-11T01:28:30.533 INFO:teuthology.orchestra.run.smithi012.stdout: debug monc = 10 2024-09-11T01:28:30.533 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.533 INFO:teuthology.orchestra.run.smithi012.stdout:[mon] 2024-09-11T01:28:30.533 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-09-11T01:28:30.533 INFO:teuthology.orchestra.run.smithi012.stdout: debug mon = 20 2024-09-11T01:28:30.533 INFO:teuthology.orchestra.run.smithi012.stdout: debug paxos = 20 2024-09-11T01:28:30.533 INFO:teuthology.orchestra.run.smithi012.stdout: debug auth = 20 2024-09-11T01:28:30.533 INFO:teuthology.orchestra.run.smithi012.stdout: mon data avail warn = 5 2024-09-11T01:28:30.534 INFO:teuthology.orchestra.run.smithi012.stdout: mon mgr mkfs grace = 240 2024-09-11T01:28:30.534 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min pgs per osd = 4 2024-09-11T01:28:30.534 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd reporter subtree level = osd 2024-09-11T01:28:30.534 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd prime pg temp = true 2024-09-11T01:28:30.534 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min bytes per osd = 10 2024-09-11T01:28:30.534 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.534 INFO:teuthology.orchestra.run.smithi012.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T01:28:30.534 INFO:teuthology.orchestra.run.smithi012.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T01:28:30.534 INFO:teuthology.orchestra.run.smithi012.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T01:28:30.534 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.534 INFO:teuthology.orchestra.run.smithi012.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T01:28:30.534 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T01:28:30.534 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T01:28:30.534 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.534 INFO:teuthology.orchestra.run.smithi012.stdout: # 1m isn't quite enough 2024-09-11T01:28:30.534 INFO:teuthology.orchestra.run.smithi012.stdout: mon_down_mkfs_grace = 2m 2024-09-11T01:28:30.535 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.535 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_filestore_osds = false 2024-09-11T01:28:30.535 INFO:teuthology.orchestra.run.smithi012.stdout: mon min osdmap epochs = 25 2024-09-11T01:28:30.535 INFO:teuthology.orchestra.run.smithi012.stdout: mon osdmap full prune interval = 2 2024-09-11T01:28:30.535 INFO:teuthology.orchestra.run.smithi012.stdout: mon osdmap full prune min = 15 2024-09-11T01:28:30.535 INFO:teuthology.orchestra.run.smithi012.stdout: mon osdmap full prune txsize = 2 2024-09-11T01:28:30.535 INFO:teuthology.orchestra.run.smithi012.stdout: mon scrub inject crc mismatch = 0.01 2024-09-11T01:28:30.535 INFO:teuthology.orchestra.run.smithi012.stdout: mon scrub inject missing keys = 0.05 2024-09-11T01:28:30.535 INFO:teuthology.orchestra.run.smithi012.stdout: mon scrub interval = 300 2024-09-11T01:28:30.535 INFO:teuthology.orchestra.run.smithi012.stdout: paxos service trim min = 5 2024-09-11T01:28:30.535 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:28:30.535 INFO:teuthology.orchestra.run.smithi012.stdout:[client] 2024-09-11T01:28:30.535 INFO:teuthology.orchestra.run.smithi012.stdout: rgw cache enabled = true 2024-09-11T01:28:30.535 INFO:teuthology.orchestra.run.smithi012.stdout: rgw enable ops log = true 2024-09-11T01:28:30.535 INFO:teuthology.orchestra.run.smithi012.stdout: rgw enable usage log = true 2024-09-11T01:28:30.535 INFO:teuthology.orchestra.run.smithi012.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T01:28:30.536 INFO:teuthology.orchestra.run.smithi012.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T01:28:30.536 INFO:teuthology.orchestra.run.smithi012.stdout: debug monc = 20 2024-09-11T01:28:30.536 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-09-11T01:28:30.536 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.a] 2024-09-11T01:28:30.536 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.c] 2024-09-11T01:28:30.536 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.b] 2024-09-11T01:28:30.536 INFO:teuthology.orchestra.run.smithi071.stdout:[global] 2024-09-11T01:28:30.537 INFO:teuthology.orchestra.run.smithi071.stdout: chdir = "" 2024-09-11T01:28:30.537 INFO:teuthology.orchestra.run.smithi071.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T01:28:30.537 INFO:teuthology.orchestra.run.smithi071.stdout: auth supported = cephx 2024-09-11T01:28:30.537 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.537 INFO:teuthology.orchestra.run.smithi071.stdout: filestore xattr use omap = true 2024-09-11T01:28:30.537 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.537 INFO:teuthology.orchestra.run.smithi071.stdout: mon clock drift allowed = 1.000 2024-09-11T01:28:30.537 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.537 INFO:teuthology.orchestra.run.smithi071.stdout: osd crush chooseleaf type = 0 2024-09-11T01:28:30.537 INFO:teuthology.orchestra.run.smithi071.stdout: auth debug = true 2024-09-11T01:28:30.537 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.537 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on old message = true 2024-09-11T01:28:30.537 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on bug = true 2024-09-11T01:28:30.537 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.538 INFO:teuthology.orchestra.run.smithi071.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T01:28:30.538 INFO:teuthology.orchestra.run.smithi071.stdout: mon pg warn max object skew = 0 2024-09-11T01:28:30.538 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.538 INFO:teuthology.orchestra.run.smithi071.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T01:28:30.538 INFO:teuthology.orchestra.run.smithi071.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T01:28:30.538 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.538 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default size = 2 2024-09-11T01:28:30.538 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.538 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow primary affinity = true 2024-09-11T01:28:30.538 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow pg remap = true 2024-09-11T01:28:30.538 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on legacy crush tunables = false 2024-09-11T01:28:30.538 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on crush straw calc version zero = false 2024-09-11T01:28:30.538 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on no sortbitwise = false 2024-09-11T01:28:30.538 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on osd down out interval zero = false 2024-09-11T01:28:30.539 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on too few osds = false 2024-09-11T01:28:30.539 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T01:28:30.539 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T01:28:30.539 INFO:teuthology.orchestra.run.smithi071.stdout: mon_allow_pool_size_one = true 2024-09-11T01:28:30.539 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.539 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T01:28:30.539 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.539 INFO:teuthology.orchestra.run.smithi071.stdout: osd default data pool replay window = 5 2024-09-11T01:28:30.539 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.539 INFO:teuthology.orchestra.run.smithi071.stdout: mon allow pool delete = true 2024-09-11T01:28:30.539 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.539 INFO:teuthology.orchestra.run.smithi071.stdout: mon cluster log file level = debug 2024-09-11T01:28:30.539 INFO:teuthology.orchestra.run.smithi071.stdout: debug asserts on shutdown = true 2024-09-11T01:28:30.539 INFO:teuthology.orchestra.run.smithi071.stdout: mon health detail to clog = false 2024-09-11T01:28:30.539 INFO:teuthology.orchestra.run.smithi071.stdout: mon host = "172.21.15.12,[v2:172.21.15.12:3301,v1:172.21.15.12:6790],172.21.15.71" 2024-09-11T01:28:30.539 INFO:teuthology.orchestra.run.smithi071.stdout: mon client directed command retry = 5 2024-09-11T01:28:30.540 INFO:teuthology.orchestra.run.smithi071.stdout: mon election default strategy = 1 2024-09-11T01:28:30.540 INFO:teuthology.orchestra.run.smithi071.stdout: ms inject delay max = 1 2024-09-11T01:28:30.540 INFO:teuthology.orchestra.run.smithi071.stdout: ms inject delay probability = 0.005 2024-09-11T01:28:30.540 INFO:teuthology.orchestra.run.smithi071.stdout: ms inject delay type = mon 2024-09-11T01:28:30.540 INFO:teuthology.orchestra.run.smithi071.stdout: ms inject internal delays = 0.002 2024-09-11T01:28:30.540 INFO:teuthology.orchestra.run.smithi071.stdout: ms inject socket failures = 2500 2024-09-11T01:28:30.540 INFO:teuthology.orchestra.run.smithi071.stdout: ms type = async 2024-09-11T01:28:30.540 INFO:teuthology.orchestra.run.smithi071.stdout: fsid = ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:28:30.540 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.540 INFO:teuthology.orchestra.run.smithi071.stdout:[osd] 2024-09-11T01:28:30.540 INFO:teuthology.orchestra.run.smithi071.stdout: osd journal size = 100 2024-09-11T01:28:30.540 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.540 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub load threshold = 5.0 2024-09-11T01:28:30.540 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub max interval = 600 2024-09-11T01:28:30.540 INFO:teuthology.orchestra.run.smithi071.stdout: osd mclock profile = high_recovery_ops 2024-09-11T01:28:30.540 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.541 INFO:teuthology.orchestra.run.smithi071.stdout: osd recover clone overlap = true 2024-09-11T01:28:30.541 INFO:teuthology.orchestra.run.smithi071.stdout: osd recovery max chunk = 1048576 2024-09-11T01:28:30.541 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.541 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug shutdown = true 2024-09-11T01:28:30.541 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug op order = true 2024-09-11T01:28:30.541 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify stray on activate = true 2024-09-11T01:28:30.541 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug trim objects = true 2024-09-11T01:28:30.541 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.541 INFO:teuthology.orchestra.run.smithi071.stdout: osd open classes on start = true 2024-09-11T01:28:30.541 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug pg log writeout = true 2024-09-11T01:28:30.541 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.541 INFO:teuthology.orchestra.run.smithi071.stdout: osd deep scrub update digest min age = 30 2024-09-11T01:28:30.541 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.541 INFO:teuthology.orchestra.run.smithi071.stdout: osd map max advance = 10 2024-09-11T01:28:30.541 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.542 INFO:teuthology.orchestra.run.smithi071.stdout: journal zero on create = true 2024-09-11T01:28:30.542 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.542 INFO:teuthology.orchestra.run.smithi071.stdout: filestore ondisk finisher threads = 3 2024-09-11T01:28:30.542 INFO:teuthology.orchestra.run.smithi071.stdout: filestore apply finisher threads = 3 2024-09-11T01:28:30.542 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.542 INFO:teuthology.orchestra.run.smithi071.stdout: bdev debug aio = true 2024-09-11T01:28:30.542 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug misdirected ops = true 2024-09-11T01:28:30.542 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore block size = 96636764160 2024-09-11T01:28:30.542 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore compression algorithm = snappy 2024-09-11T01:28:30.542 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore compression mode = aggressive 2024-09-11T01:28:30.542 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore fsck on mount = True 2024-09-11T01:28:30.542 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore write v2 random = True 2024-09-11T01:28:30.542 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore zero block detection = True 2024-09-11T01:28:30.542 INFO:teuthology.orchestra.run.smithi071.stdout: debug bluefs = 20 2024-09-11T01:28:30.543 INFO:teuthology.orchestra.run.smithi071.stdout: debug bluestore = 20 2024-09-11T01:28:30.543 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-09-11T01:28:30.543 INFO:teuthology.orchestra.run.smithi071.stdout: debug osd = 20 2024-09-11T01:28:30.543 INFO:teuthology.orchestra.run.smithi071.stdout: debug rocksdb = 10 2024-09-11T01:28:30.543 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd backfillfull_ratio = 0.85 2024-09-11T01:28:30.543 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd full ratio = 0.9 2024-09-11T01:28:30.543 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd nearfull ratio = 0.8 2024-09-11T01:28:30.543 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify cached snaps = True 2024-09-11T01:28:30.543 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify missing on start = True 2024-09-11T01:28:30.543 INFO:teuthology.orchestra.run.smithi071.stdout: osd failsafe full ratio = 0.95 2024-09-11T01:28:30.543 INFO:teuthology.orchestra.run.smithi071.stdout: osd mclock override recovery settings = True 2024-09-11T01:28:30.543 INFO:teuthology.orchestra.run.smithi071.stdout: osd objectstore = bluestore 2024-09-11T01:28:30.543 INFO:teuthology.orchestra.run.smithi071.stdout: osd op queue = debug_random 2024-09-11T01:28:30.544 INFO:teuthology.orchestra.run.smithi071.stdout: osd op queue cut off = debug_random 2024-09-11T01:28:30.544 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.544 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr] 2024-09-11T01:28:30.544 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-09-11T01:28:30.544 INFO:teuthology.orchestra.run.smithi071.stdout: debug mgr = 20 2024-09-11T01:28:30.544 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2024-09-11T01:28:30.544 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2024-09-11T01:28:30.544 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2024-09-11T01:28:30.544 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2024-09-11T01:28:30.544 INFO:teuthology.orchestra.run.smithi071.stdout: mgr/telemetry/nag = false 2024-09-11T01:28:30.544 INFO:teuthology.orchestra.run.smithi071.stdout: debug monc = 10 2024-09-11T01:28:30.544 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.544 INFO:teuthology.orchestra.run.smithi071.stdout:[mon] 2024-09-11T01:28:30.544 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-09-11T01:28:30.545 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2024-09-11T01:28:30.545 INFO:teuthology.orchestra.run.smithi071.stdout: debug paxos = 20 2024-09-11T01:28:30.545 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2024-09-11T01:28:30.545 INFO:teuthology.orchestra.run.smithi071.stdout: mon data avail warn = 5 2024-09-11T01:28:30.545 INFO:teuthology.orchestra.run.smithi071.stdout: mon mgr mkfs grace = 240 2024-09-11T01:28:30.545 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2024-09-11T01:28:30.545 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd reporter subtree level = osd 2024-09-11T01:28:30.545 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd prime pg temp = true 2024-09-11T01:28:30.545 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2024-09-11T01:28:30.545 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.545 INFO:teuthology.orchestra.run.smithi071.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T01:28:30.545 INFO:teuthology.orchestra.run.smithi071.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T01:28:30.545 INFO:teuthology.orchestra.run.smithi071.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T01:28:30.545 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.545 INFO:teuthology.orchestra.run.smithi071.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T01:28:30.546 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T01:28:30.546 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T01:28:30.546 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.546 INFO:teuthology.orchestra.run.smithi071.stdout: # 1m isn't quite enough 2024-09-11T01:28:30.546 INFO:teuthology.orchestra.run.smithi071.stdout: mon_down_mkfs_grace = 2m 2024-09-11T01:28:30.546 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.546 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_filestore_osds = false 2024-09-11T01:28:30.546 INFO:teuthology.orchestra.run.smithi071.stdout: mon min osdmap epochs = 25 2024-09-11T01:28:30.546 INFO:teuthology.orchestra.run.smithi071.stdout: mon osdmap full prune interval = 2 2024-09-11T01:28:30.546 INFO:teuthology.orchestra.run.smithi071.stdout: mon osdmap full prune min = 15 2024-09-11T01:28:30.546 INFO:teuthology.orchestra.run.smithi071.stdout: mon osdmap full prune txsize = 2 2024-09-11T01:28:30.546 INFO:teuthology.orchestra.run.smithi071.stdout: mon scrub inject crc mismatch = 0.01 2024-09-11T01:28:30.546 INFO:teuthology.orchestra.run.smithi071.stdout: mon scrub inject missing keys = 0.05 2024-09-11T01:28:30.546 INFO:teuthology.orchestra.run.smithi071.stdout: mon scrub interval = 300 2024-09-11T01:28:30.547 INFO:teuthology.orchestra.run.smithi071.stdout: paxos service trim min = 5 2024-09-11T01:28:30.547 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-11T01:28:30.547 INFO:teuthology.orchestra.run.smithi071.stdout:[client] 2024-09-11T01:28:30.547 INFO:teuthology.orchestra.run.smithi071.stdout: rgw cache enabled = true 2024-09-11T01:28:30.547 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable ops log = true 2024-09-11T01:28:30.547 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable usage log = true 2024-09-11T01:28:30.547 INFO:teuthology.orchestra.run.smithi071.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T01:28:30.547 INFO:teuthology.orchestra.run.smithi071.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T01:28:30.547 INFO:teuthology.orchestra.run.smithi071.stdout: debug monc = 20 2024-09-11T01:28:30.547 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-09-11T01:28:30.547 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.a] 2024-09-11T01:28:30.547 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.c] 2024-09-11T01:28:30.547 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.b] 2024-09-11T01:28:30.548 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-11T01:28:30.549 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-11T01:28:30.645 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-11T01:28:30.645 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-09-11T01:28:30.645 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-11T01:28:30.661 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-09-11T01:28:30.661 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-11T01:28:30.717 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi012.front.sepia.ceph.com 2024-09-11T01:28:30.717 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-09-11T01:28:30.717 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-11T01:28:30.717 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T01:28:30.793 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-09-11T01:28:30.793 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-11T01:28:30.849 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi071.front.sepia.ceph.com 2024-09-11T01:28:30.849 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-11T01:28:30.849 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-11T01:28:30.849 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T01:28:30.886 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-11T01:28:30.886 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-11T01:28:30.942 INFO:tasks.ceph:Setting up mon nodes... 2024-09-11T01:28:30.943 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-11T01:28:30.943 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-11T01:28:31.067 INFO:teuthology.orchestra.run.smithi071.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-11T01:28:31.070 INFO:tasks.ceph:Setting up mds nodes... 2024-09-11T01:28:31.070 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-11T01:28:31.070 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-11T01:28:31.111 INFO:teuthology.orchestra.run.smithi071.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-11T01:28:31.123 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-11T01:28:31.123 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi012.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-11T01:28:31.123 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-11T01:28:31.150 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T01:28:31.150 INFO:tasks.ceph:role: osd.0 2024-09-11T01:28:31.151 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi012.front.sepia.ceph.com 2024-09-11T01:28:31.151 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-11T01:28:31.225 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T01:28:31.225 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T01:28:31.225 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T01:28:31.225 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T01:28:31.225 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T01:28:31.226 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2024-09-11T01:28:31.226 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T01:28:31.226 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T01:28:31.226 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T01:28:31.226 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T01:28:31.740 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2024-09-11T01:28:31.742 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2024-09-11T01:28:31.743 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-11T01:28:31.882 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-11T01:28:31.918 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-11T01:28:31.987 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T01:28:31.987 INFO:tasks.ceph:role: osd.1 2024-09-11T01:28:31.987 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi012.front.sepia.ceph.com 2024-09-11T01:28:31.987 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-11T01:28:32.062 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T01:28:32.062 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T01:28:32.062 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T01:28:32.062 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T01:28:32.062 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T01:28:32.062 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2024-09-11T01:28:32.063 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T01:28:32.063 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T01:28:32.063 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T01:28:32.063 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T01:28:32.507 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2024-09-11T01:28:32.509 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2024-09-11T01:28:32.510 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-11T01:28:32.593 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-11T01:28:32.663 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-11T01:28:32.728 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T01:28:32.728 INFO:tasks.ceph:role: osd.2 2024-09-11T01:28:32.729 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi012.front.sepia.ceph.com 2024-09-11T01:28:32.729 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-11T01:28:32.801 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T01:28:32.802 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T01:28:32.802 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T01:28:32.802 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T01:28:32.802 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T01:28:32.802 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2024-09-11T01:28:32.802 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T01:28:32.802 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T01:28:32.802 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T01:28:32.802 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T01:28:33.290 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2024-09-11T01:28:33.292 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2024-09-11T01:28:33.292 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-11T01:28:33.410 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-11T01:28:33.442 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T01:28:33.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:33.539+0000 7fee7a20d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-11T01:28:33.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:33.543+0000 7fee7a20d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-11T01:28:33.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:33.543+0000 7fee7a20d640 -1 bdev(0x559a88f71800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-11T01:28:33.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:33.543+0000 7fee7a20d640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-11T01:28:36.457 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-11T01:28:36.486 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T01:28:36.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:36.575+0000 7fe9e0e0c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-11T01:28:36.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:36.576+0000 7fe9e0e0c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-11T01:28:36.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:36.576+0000 7fe9e0e0c640 -1 bdev(0x557c44f5d800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-11T01:28:36.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:36.576+0000 7fe9e0e0c640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-11T01:28:39.988 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-11T01:28:40.018 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T01:28:40.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:40.109+0000 7feb0abae640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-11T01:28:40.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:40.109+0000 7feb0abae640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-11T01:28:40.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:40.109+0000 7feb0abae640 -1 bdev(0x5619ff4ad800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-11T01:28:40.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:40.110+0000 7feb0abae640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-11T01:28:43.270 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-11T01:28:43.301 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-11T01:28:43.333 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-11T01:28:43.333 INFO:tasks.ceph:role: osd.3 2024-09-11T01:28:43.333 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com 2024-09-11T01:28:43.333 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-11T01:28:43.409 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T01:28:43.409 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T01:28:43.409 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T01:28:43.409 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T01:28:43.410 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T01:28:43.410 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-09-11T01:28:43.410 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T01:28:43.410 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T01:28:43.410 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T01:28:43.410 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T01:28:43.912 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-09-11T01:28:43.915 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-09-11T01:28:43.915 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-11T01:28:44.041 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-11T01:28:44.075 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-11T01:28:44.140 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-11T01:28:44.140 INFO:tasks.ceph:role: osd.4 2024-09-11T01:28:44.140 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com 2024-09-11T01:28:44.140 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-11T01:28:44.213 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T01:28:44.213 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T01:28:44.213 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T01:28:44.213 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T01:28:44.213 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T01:28:44.213 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-09-11T01:28:44.213 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T01:28:44.214 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T01:28:44.214 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T01:28:44.214 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T01:28:44.664 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-09-11T01:28:44.666 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-09-11T01:28:44.666 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-11T01:28:44.763 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-11T01:28:44.791 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-11T01:28:44.856 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-11T01:28:44.856 INFO:tasks.ceph:role: osd.5 2024-09-11T01:28:44.856 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi071.front.sepia.ceph.com 2024-09-11T01:28:44.856 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-11T01:28:44.928 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T01:28:44.929 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T01:28:44.929 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T01:28:44.929 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T01:28:44.929 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T01:28:44.929 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-09-11T01:28:44.929 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T01:28:44.929 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T01:28:44.929 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T01:28:44.929 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T01:28:45.429 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-09-11T01:28:45.431 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-09-11T01:28:45.431 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-11T01:28:45.572 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-11T01:28:45.603 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T01:28:45.702 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:28:45.701+0000 7feab98b2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-11T01:28:45.708 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:28:45.706+0000 7feab98b2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-11T01:28:45.708 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:28:45.706+0000 7feab98b2640 -1 bdev(0x55889d71f800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-11T01:28:45.708 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:28:45.706+0000 7feab98b2640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-11T01:28:48.607 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-11T01:28:48.638 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T01:28:48.728 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:28:48.726+0000 7f3df6f20640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-11T01:28:48.729 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:28:48.727+0000 7f3df6f20640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-11T01:28:48.729 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:28:48.727+0000 7f3df6f20640 -1 bdev(0x55734dbbf800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-11T01:28:48.729 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:28:48.727+0000 7f3df6f20640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-11T01:28:52.383 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-11T01:28:52.413 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T01:28:52.505 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:28:52.502+0000 7f79b0c11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-11T01:28:52.505 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:28:52.503+0000 7f79b0c11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-11T01:28:52.505 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:28:52.503+0000 7f79b0c11640 -1 bdev(0x556386fdd800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-11T01:28:52.505 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:28:52.503+0000 7f79b0c11640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-11T01:28:55.657 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-11T01:28:55.684 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-11T01:28:55.684 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-09-11T01:28:55.684 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-11T01:28:55.715 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-09-11T01:28:55.716 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-11T01:28:55.787 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-09-11T01:28:55.787 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-11T01:28:55.860 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-11T01:28:55.861 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-11T01:28:55.888 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-11T01:28:55.888 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-11T01:28:55.955 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-11T01:28:55.955 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-11T01:28:56.023 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-11T01:28:56.023 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-11T01:28:56.089 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-11T01:28:56.089 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-11T01:28:56.146 INFO:tasks.ceph:Adding keys to all mons... 2024-09-11T01:28:56.146 DEBUG:teuthology.orchestra.run.smithi012:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-11T01:28:56.149 DEBUG:teuthology.orchestra.run.smithi071:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-11T01:28:56.190 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.0] 2024-09-11T01:28:56.190 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQBB8uBmJrIyIBAAqL/GyQRLl6FMvibyUnYMTw== 2024-09-11T01:28:56.190 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.1] 2024-09-11T01:28:56.190 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQBE8uBmxbJcIhAAVYxiv1k1UUqZrJHRGi3LLA== 2024-09-11T01:28:56.190 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.2] 2024-09-11T01:28:56.190 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQBI8uBm1F2hBhAAQSqvhurgzbg3TRUoL00diA== 2024-09-11T01:28:56.190 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr.x] 2024-09-11T01:28:56.190 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQA/8uBmFGL8AxAAdR18igH01xTUlC9ILb9JpQ== 2024-09-11T01:28:56.190 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.3] 2024-09-11T01:28:56.190 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQBN8uBmDyDZKRAA23LhLMG/aRzxlv/202WM3A== 2024-09-11T01:28:56.190 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.4] 2024-09-11T01:28:56.190 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQBQ8uBmATphKxAATo2RwMKsiGRVnNaKNXgrQA== 2024-09-11T01:28:56.191 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.5] 2024-09-11T01:28:56.191 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQBU8uBmNZMIHhAAVxtfFODbZEFtxi6JWXatgg== 2024-09-11T01:28:56.191 INFO:teuthology.orchestra.run.smithi012.stdout:[client.0] 2024-09-11T01:28:56.191 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQA/8uBmUHWWBhAAqGCih5EFf5lqv82s8MmWYg== 2024-09-11T01:28:56.211 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.0] 2024-09-11T01:28:56.211 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBB8uBmJrIyIBAAqL/GyQRLl6FMvibyUnYMTw== 2024-09-11T01:28:56.211 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.1] 2024-09-11T01:28:56.211 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBE8uBmxbJcIhAAVYxiv1k1UUqZrJHRGi3LLA== 2024-09-11T01:28:56.211 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.2] 2024-09-11T01:28:56.211 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBI8uBm1F2hBhAAQSqvhurgzbg3TRUoL00diA== 2024-09-11T01:28:56.212 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr.x] 2024-09-11T01:28:56.212 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQA/8uBmFGL8AxAAdR18igH01xTUlC9ILb9JpQ== 2024-09-11T01:28:56.212 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.3] 2024-09-11T01:28:56.212 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBN8uBmDyDZKRAA23LhLMG/aRzxlv/202WM3A== 2024-09-11T01:28:56.212 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.4] 2024-09-11T01:28:56.212 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBQ8uBmATphKxAATo2RwMKsiGRVnNaKNXgrQA== 2024-09-11T01:28:56.212 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.5] 2024-09-11T01:28:56.212 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBU8uBmNZMIHhAAVxtfFODbZEFtxi6JWXatgg== 2024-09-11T01:28:56.212 INFO:teuthology.orchestra.run.smithi071.stdout:[client.0] 2024-09-11T01:28:56.213 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQA/8uBmUHWWBhAAqGCih5EFf5lqv82s8MmWYg== 2024-09-11T01:28:56.214 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T01:28:56.234 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T01:28:56.306 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T01:28:56.308 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T01:28:56.363 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T01:28:56.365 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T01:28:56.417 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T01:28:56.419 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T01:28:56.470 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T01:28:56.473 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T01:28:56.524 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T01:28:56.526 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T01:28:56.576 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T01:28:56.579 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T01:28:56.630 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T01:28:56.632 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T01:28:56.686 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-11T01:28:56.686 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-11T01:28:56.711 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T01:28:56.811 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-11T01:28:56.837 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-11T01:28:56.903 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T01:28:57.001 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-11T01:28:57.027 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-11T01:28:57.052 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T01:28:57.148 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-11T01:28:57.172 DEBUG:teuthology.orchestra.run.smithi012:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-11T01:28:57.174 DEBUG:teuthology.orchestra.run.smithi071:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-11T01:28:57.227 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-11T01:28:57.228 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T01:28:57.228 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T01:28:57.231 INFO:tasks.ceph.mon.a:Started 2024-09-11T01:28:57.231 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-11T01:28:57.231 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-11T01:28:57.233 INFO:tasks.ceph.mon.c:Started 2024-09-11T01:28:57.233 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-11T01:28:57.233 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-11T01:28:57.272 INFO:tasks.ceph.mon.b:Started 2024-09-11T01:28:57.272 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-11T01:28:57.273 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-11T01:28:57.273 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-11T01:28:57.275 INFO:tasks.ceph.mgr.x:Started 2024-09-11T01:28:57.275 DEBUG:tasks.ceph:set 0 configs 2024-09-11T01:28:57.275 INFO:tasks.ceph:Setting crush tunables to default 2024-09-11T01:28:57.275 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster ceph osd crush tunables default 2024-09-11T01:28:57.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.375+0000 7fb3baccd640 10 monclient: get_monmap_and_config 2024-09-11T01:28:57.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.375+0000 7fb3baccd640 10 monclient: build_initial_monmap 2024-09-11T01:28:57.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.375+0000 7fb3baccd640 10 monclient: monmap: 2024-09-11T01:28:57.378 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:28:57.378 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:28:57.378 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:57.377532+0000 2024-09-11T01:28:57.379 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:57.377532+0000 2024-09-11T01:28:57.379 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:28:57.379 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:28:57.379 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:28:57.379 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:28:57.379 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:28:57.379 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:28:57.379 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:28:57.379 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:28:57.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.376+0000 7fb3baccd640 1 Processor -- start 2024-09-11T01:28:57.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.376+0000 7fb3baccd640 1 -- start start 2024-09-11T01:28:57.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.376+0000 7fb3baccd640 10 monclient: init 2024-09-11T01:28:57.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3baccd640 10 monclient: _reopen_session rank -1 2024-09-11T01:28:57.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3baccd640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-11T01:28:57.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3baccd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb3b4138210 addr v1:172.21.15.12:6789/0 2024-09-11T01:28:57.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3baccd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb3b40a6810 0x7fb3b413d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:28:57.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3baccd640 10 monclient(hunting): picked mon.noname-c con 0x7fb3b40a6810 addr v2:172.21.15.71:3300/0 2024-09-11T01:28:57.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3baccd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb3b413d640 0x7fb3b413fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:28:57.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3baccd640 10 monclient(hunting): picked mon.noname-a con 0x7fb3b413d640 addr v2:172.21.15.12:3300/0 2024-09-11T01:28:57.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3baccd640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3b4138d00 con 0x7fb3b4138210 2024-09-11T01:28:57.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3baccd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:28:57.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3baccd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fb3b409c4c0 con 0x7fb3b413d640 2024-09-11T01:28:57.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3baccd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:28:57.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3baccd640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fb3b409ba20 con 0x7fb3b40a6810 2024-09-11T01:28:57.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3baccd640 10 monclient(hunting): _renew_subs 2024-09-11T01:28:57.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3baccd640 10 monclient(hunting): authenticate will time out at 743.820228s 2024-09-11T01:28:57.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3b8a69640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb3b4138210 0x7fb3b41385f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46906/0 (socket says 172.21.15.12:46906) 2024-09-11T01:28:57.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3b8a69640 1 -- 172.21.15.12:0/1597759190 learned_addr learned my addr 172.21.15.12:0/1597759190 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:28:57.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3b926a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb3b413d640 0x7fb3b413fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:28:57.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3b926a640 10 monclient(hunting): get_auth_request con 0x7fb3b413d640 auth_method 0 2024-09-11T01:28:57.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3b926a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:28:57.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3b926a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:28:57.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3b926a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:28:57.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3b926a640 1 -- 172.21.15.12:0/1597759190 >> v2:172.21.15.12:3300/0 conn(0x7fb3b413d640 msgr2=0x7fb3b413fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:28:57.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3b926a640 1 -- 172.21.15.12:0/1597759190 >> v2:172.21.15.12:3300/0 conn(0x7fb3b413d640 msgr2=0x7fb3b413fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:28:57.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3b926a640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.12:3300/0 conn(0x7fb3b413d640 0x7fb3b413fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:28:57.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3b3fff640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.71:3300/0 conn(0x7fb3b40a6810 0x7fb3b413d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:28:57.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3b926a640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.12:3300/0 conn(0x7fb3b413d640 0x7fb3b413fa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:28:57.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3b3fff640 10 monclient(hunting): get_auth_request con 0x7fb3b40a6810 auth_method 0 2024-09-11T01:28:57.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:28:57.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:28:57.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:28:57.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3b3fff640 1 -- 172.21.15.12:0/1597759190 >> v2:172.21.15.71:3300/0 conn(0x7fb3b40a6810 msgr2=0x7fb3b413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:28:57.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3b3fff640 1 -- 172.21.15.12:0/1597759190 >> v2:172.21.15.71:3300/0 conn(0x7fb3b40a6810 msgr2=0x7fb3b413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:28:57.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3b3fff640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.71:3300/0 conn(0x7fb3b40a6810 0x7fb3b413d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:28:57.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.377+0000 7fb3b3fff640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.71:3300/0 conn(0x7fb3b40a6810 0x7fb3b413d100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:28:57.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.379+0000 7fb3b8a69640 1 -- 172.21.15.12:0/1597759190 >> v1:172.21.15.12:6789/0 conn(0x7fb3b4138210 legacy=0x7fb3b41385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:28:57.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.578+0000 7fb3b926a640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.12:3300/0 conn(0x7fb3b413d640 0x7fb3b413fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:28:57.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.578+0000 7fb3b926a640 10 monclient(hunting): get_auth_request con 0x7fb3b413d640 auth_method 0 2024-09-11T01:28:57.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.578+0000 7fb3b926a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:28:57.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.578+0000 7fb3b926a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:28:57.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.578+0000 7fb3b926a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:28:57.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.578+0000 7fb3b926a640 1 -- 172.21.15.12:0/1597759190 >> v2:172.21.15.12:3300/0 conn(0x7fb3b413d640 msgr2=0x7fb3b413fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:28:57.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.578+0000 7fb3b926a640 1 -- 172.21.15.12:0/1597759190 >> v2:172.21.15.12:3300/0 conn(0x7fb3b413d640 msgr2=0x7fb3b413fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:28:57.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.578+0000 7fb3b926a640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.12:3300/0 conn(0x7fb3b413d640 0x7fb3b413fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:28:57.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.578+0000 7fb3b3fff640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.71:3300/0 conn(0x7fb3b40a6810 0x7fb3b413d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:28:57.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.578+0000 7fb3b3fff640 10 monclient(hunting): get_auth_request con 0x7fb3b40a6810 auth_method 0 2024-09-11T01:28:57.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.578+0000 7fb3b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:28:57.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.578+0000 7fb3b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:28:57.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.578+0000 7fb3b3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:28:57.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.578+0000 7fb3b926a640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.12:3300/0 conn(0x7fb3b413d640 0x7fb3b413fa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:28:57.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.578+0000 7fb3b3fff640 1 -- 172.21.15.12:0/1597759190 >> v2:172.21.15.71:3300/0 conn(0x7fb3b40a6810 msgr2=0x7fb3b413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:28:57.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.578+0000 7fb3b3fff640 1 -- 172.21.15.12:0/1597759190 >> v2:172.21.15.71:3300/0 conn(0x7fb3b40a6810 msgr2=0x7fb3b413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:28:57.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.578+0000 7fb3b3fff640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.71:3300/0 conn(0x7fb3b40a6810 0x7fb3b413d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:28:57.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.578+0000 7fb3b3fff640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.71:3300/0 conn(0x7fb3b40a6810 0x7fb3b413d100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:28:57.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.979+0000 7fb3b926a640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.12:3300/0 conn(0x7fb3b413d640 0x7fb3b413fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:28:57.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.979+0000 7fb3b926a640 10 monclient(hunting): get_auth_request con 0x7fb3b413d640 auth_method 0 2024-09-11T01:28:57.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.979+0000 7fb3b926a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:28:57.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.979+0000 7fb3b926a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:28:57.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.979+0000 7fb3b926a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:28:57.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.979+0000 7fb3b926a640 1 -- 172.21.15.12:0/1597759190 >> v2:172.21.15.12:3300/0 conn(0x7fb3b413d640 msgr2=0x7fb3b413fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:28:57.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.979+0000 7fb3b926a640 1 -- 172.21.15.12:0/1597759190 >> v2:172.21.15.12:3300/0 conn(0x7fb3b413d640 msgr2=0x7fb3b413fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:28:57.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.979+0000 7fb3b3fff640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.71:3300/0 conn(0x7fb3b40a6810 0x7fb3b413d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:28:57.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.979+0000 7fb3b926a640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.12:3300/0 conn(0x7fb3b413d640 0x7fb3b413fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:28:57.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.979+0000 7fb3b3fff640 10 monclient(hunting): get_auth_request con 0x7fb3b40a6810 auth_method 0 2024-09-11T01:28:57.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.979+0000 7fb3b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:28:57.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.979+0000 7fb3b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:28:57.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.979+0000 7fb3b3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:28:57.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.979+0000 7fb3b926a640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.12:3300/0 conn(0x7fb3b413d640 0x7fb3b413fa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:28:57.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.979+0000 7fb3b3fff640 1 -- 172.21.15.12:0/1597759190 >> v2:172.21.15.71:3300/0 conn(0x7fb3b40a6810 msgr2=0x7fb3b413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:28:57.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.979+0000 7fb3b3fff640 1 -- 172.21.15.12:0/1597759190 >> v2:172.21.15.71:3300/0 conn(0x7fb3b40a6810 msgr2=0x7fb3b413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:28:57.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.979+0000 7fb3b3fff640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.71:3300/0 conn(0x7fb3b40a6810 0x7fb3b413d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:28:57.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:57.979+0000 7fb3b3fff640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.71:3300/0 conn(0x7fb3b40a6810 0x7fb3b413d100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:28:58.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:58.780+0000 7fb3b926a640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.12:3300/0 conn(0x7fb3b413d640 0x7fb3b413fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:28:58.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:58.780+0000 7fb3b926a640 10 monclient(hunting): get_auth_request con 0x7fb3b413d640 auth_method 0 2024-09-11T01:28:58.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:58.780+0000 7fb3b926a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:28:58.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:58.780+0000 7fb3b926a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:28:58.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:58.780+0000 7fb3b926a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:28:58.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:58.780+0000 7fb3b3fff640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.71:3300/0 conn(0x7fb3b40a6810 0x7fb3b413d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:28:58.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:58.780+0000 7fb3b926a640 1 -- 172.21.15.12:0/1597759190 >> v2:172.21.15.12:3300/0 conn(0x7fb3b413d640 msgr2=0x7fb3b413fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:28:58.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:58.780+0000 7fb3b926a640 1 -- 172.21.15.12:0/1597759190 >> v2:172.21.15.12:3300/0 conn(0x7fb3b413d640 msgr2=0x7fb3b413fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:28:58.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:58.780+0000 7fb3b926a640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.12:3300/0 conn(0x7fb3b413d640 0x7fb3b413fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:28:58.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:58.780+0000 7fb3b3fff640 10 monclient(hunting): get_auth_request con 0x7fb3b40a6810 auth_method 0 2024-09-11T01:28:58.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:58.780+0000 7fb3b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:28:58.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:58.780+0000 7fb3b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:28:58.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:58.780+0000 7fb3b3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:28:58.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:58.780+0000 7fb3b926a640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.12:3300/0 conn(0x7fb3b413d640 0x7fb3b413fa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:28:58.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:58.780+0000 7fb3b3fff640 1 -- 172.21.15.12:0/1597759190 >> v2:172.21.15.71:3300/0 conn(0x7fb3b40a6810 msgr2=0x7fb3b413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:28:58.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:58.780+0000 7fb3b3fff640 1 -- 172.21.15.12:0/1597759190 >> v2:172.21.15.71:3300/0 conn(0x7fb3b40a6810 msgr2=0x7fb3b413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:28:58.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:58.780+0000 7fb3b3fff640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.71:3300/0 conn(0x7fb3b40a6810 0x7fb3b413d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:28:58.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:28:58.780+0000 7fb3b3fff640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.71:3300/0 conn(0x7fb3b40a6810 0x7fb3b413d100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:29:00.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.378+0000 7fb3b27fc640 10 monclient(hunting): tick 2024-09-11T01:29:00.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.378+0000 7fb3b27fc640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:29:00.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.378+0000 7fb3b27fc640 1 monclient(hunting): continuing hunt 2024-09-11T01:29:00.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.378+0000 7fb3b27fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:29:00.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.378+0000 7fb3b27fc640 1 -- 172.21.15.12:0/1597759190 >> v2:172.21.15.71:3300/0 conn(0x7fb3b40a6810 msgr2=0x7fb3b413d100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:29:00.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.378+0000 7fb3b27fc640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.71:3300/0 conn(0x7fb3b40a6810 0x7fb3b413d100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:00.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.378+0000 7fb3b27fc640 1 -- 172.21.15.12:0/1597759190 >> v2:172.21.15.12:3300/0 conn(0x7fb3b413d640 msgr2=0x7fb3b413fa30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:29:00.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.378+0000 7fb3b27fc640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.12:3300/0 conn(0x7fb3b413d640 0x7fb3b413fa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:00.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.378+0000 7fb3b27fc640 1 -- 172.21.15.12:0/1597759190 >> v1:172.21.15.12:6789/0 conn(0x7fb3b4138210 legacy=0x7fb3b41385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:00.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.378+0000 7fb3b27fc640 10 monclient: _add_conns ranks=[2,1] 2024-09-11T01:29:00.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.378+0000 7fb3b27fc640 1 --2- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb398003710 0x7fb398005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:00.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.378+0000 7fb3b27fc640 10 monclient(hunting): picked mon.noname-b con 0x7fb398003710 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:00.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.378+0000 7fb3b27fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb398006270 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:00.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.378+0000 7fb3b27fc640 1 -- 172.21.15.12:0/1597759190 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3b40a6590 con 0x7fb398006270 2024-09-11T01:29:00.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.378+0000 7fb3b27fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:00.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.378+0000 7fb3b27fc640 1 -- 172.21.15.12:0/1597759190 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb3b409c4c0 con 0x7fb398003710 2024-09-11T01:29:00.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.378+0000 7fb3b27fc640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:00.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.378+0000 7fb3b8a69640 1 --2- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb398003710 0x7fb398005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:00.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.378+0000 7fb3b8a69640 10 monclient(hunting): get_auth_request con 0x7fb398003710 auth_method 0 2024-09-11T01:29:00.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.378+0000 7fb3b8a69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:00.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.378+0000 7fb3b8a69640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:00.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.378+0000 7fb3b8a69640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:00.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.378+0000 7fb3b8a69640 1 -- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb398003710 msgr2=0x7fb398005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:29:00.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.378+0000 7fb3b8a69640 1 -- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb398003710 msgr2=0x7fb398005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:29:00.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.378+0000 7fb3b8a69640 1 --2- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb398003710 0x7fb398005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:29:00.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.378+0000 7fb3b8a69640 1 --2- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb398003710 0x7fb398005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:29:00.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.380+0000 7fb3b3fff640 1 -- 172.21.15.12:0/1597759190 >> v1:172.21.15.71:6789/0 conn(0x7fb398006270 legacy=0x7fb398008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:00.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.579+0000 7fb3b8a69640 1 --2- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb398003710 0x7fb398005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:00.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.579+0000 7fb3b8a69640 10 monclient(hunting): get_auth_request con 0x7fb398003710 auth_method 0 2024-09-11T01:29:00.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.579+0000 7fb3b8a69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:00.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.579+0000 7fb3b8a69640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:00.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.579+0000 7fb3b8a69640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:29:00.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.579+0000 7fb3b8a69640 1 -- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb398003710 msgr2=0x7fb398005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:29:00.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.579+0000 7fb3b8a69640 1 -- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb398003710 msgr2=0x7fb398005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:29:00.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.579+0000 7fb3b8a69640 1 --2- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb398003710 0x7fb398005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:29:00.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.579+0000 7fb3b8a69640 1 --2- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb398003710 0x7fb398005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:29:00.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.980+0000 7fb3b8a69640 1 --2- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb398003710 0x7fb398005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:00.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.980+0000 7fb3b8a69640 10 monclient(hunting): get_auth_request con 0x7fb398003710 auth_method 0 2024-09-11T01:29:00.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.980+0000 7fb3b8a69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:00.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.980+0000 7fb3b8a69640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:00.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.980+0000 7fb3b8a69640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:29:00.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.980+0000 7fb3b8a69640 1 -- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb398003710 msgr2=0x7fb398005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:29:00.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.980+0000 7fb3b8a69640 1 -- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb398003710 msgr2=0x7fb398005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:29:00.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.980+0000 7fb3b8a69640 1 --2- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb398003710 0x7fb398005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:29:00.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:00.980+0000 7fb3b8a69640 1 --2- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb398003710 0x7fb398005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:29:01.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:01.781+0000 7fb3b8a69640 1 --2- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb398003710 0x7fb398005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:01.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:01.781+0000 7fb3b8a69640 10 monclient(hunting): get_auth_request con 0x7fb398003710 auth_method 0 2024-09-11T01:29:01.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:01.781+0000 7fb3b8a69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:01.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:01.781+0000 7fb3b8a69640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:01.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:01.781+0000 7fb3b8a69640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:29:01.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:01.781+0000 7fb3b8a69640 1 -- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb398003710 msgr2=0x7fb398005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:29:01.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:01.781+0000 7fb3b8a69640 1 -- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb398003710 msgr2=0x7fb398005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:29:01.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:01.781+0000 7fb3b8a69640 1 --2- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb398003710 0x7fb398005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:29:01.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:01.781+0000 7fb3b8a69640 1 --2- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb398003710 0x7fb398005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:29:03.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.377+0000 7fb3b27fc640 10 monclient(hunting): tick 2024-09-11T01:29:03.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.377+0000 7fb3b27fc640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:29:03.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.377+0000 7fb3b27fc640 1 monclient(hunting): continuing hunt 2024-09-11T01:29:03.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.377+0000 7fb3b27fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:29:03.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.377+0000 7fb3b27fc640 1 -- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb398003710 msgr2=0x7fb398005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:29:03.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.377+0000 7fb3b27fc640 1 --2- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb398003710 0x7fb398005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:03.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.377+0000 7fb3b27fc640 1 -- 172.21.15.12:0/1597759190 >> v1:172.21.15.71:6789/0 conn(0x7fb398006270 legacy=0x7fb398008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:03.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.377+0000 7fb3b926a640 1 -- 172.21.15.12:0/1597759190 reap_dead start 2024-09-11T01:29:03.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.377+0000 7fb3b27fc640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-11T01:29:03.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.377+0000 7fb3b27fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb39800a170 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:03.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.377+0000 7fb3b27fc640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.12:3300/0 conn(0x7fb398005b40 0x7fb398008730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:03.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.377+0000 7fb3b27fc640 10 monclient(hunting): picked mon.noname-a con 0x7fb398005b40 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:03.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.377+0000 7fb3b27fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb398008ee0 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:03.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.377+0000 7fb3b27fc640 1 -- 172.21.15.12:0/1597759190 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb39800b0c0 con 0x7fb39800a170 2024-09-11T01:29:03.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.378+0000 7fb3b27fc640 1 -- 172.21.15.12:0/1597759190 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb39800ada0 con 0x7fb398008ee0 2024-09-11T01:29:03.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.378+0000 7fb3b27fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:03.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.378+0000 7fb3b27fc640 1 -- 172.21.15.12:0/1597759190 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fb3b409c4c0 con 0x7fb398005b40 2024-09-11T01:29:03.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.378+0000 7fb3b27fc640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:03.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.378+0000 7fb3b3fff640 1 --2- 172.21.15.12:0/1597759190 >> v2:172.21.15.12:3300/0 conn(0x7fb398005b40 0x7fb398008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:03.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.378+0000 7fb3b3fff640 10 monclient(hunting): get_auth_request con 0x7fb398005b40 auth_method 0 2024-09-11T01:29:03.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.378+0000 7fb3b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:03.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.378+0000 7fb3b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:03.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.378+0000 7fb3b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:03.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.378+0000 7fb3b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:03.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.378+0000 7fb3b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:03.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.378+0000 7fb3b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:03.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.378+0000 7fb3b3fff640 10 monclient(hunting): handle_auth_done global_id 4104 payload 274 2024-09-11T01:29:03.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.378+0000 7fb3b3fff640 1 -- 172.21.15.12:0/1597759190 >> v1:172.21.15.71:6789/0 conn(0x7fb398008ee0 legacy=0x7fb398009390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:03.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.378+0000 7fb3b3fff640 1 -- 172.21.15.12:0/1597759190 >> v1:172.21.15.12:6789/0 conn(0x7fb39800a170 legacy=0x7fb39800a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:03.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.378+0000 7fb3b3fff640 10 monclient: _finish_hunting 0 2024-09-11T01:29:03.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.378+0000 7fb3b3fff640 1 monclient: found mon.noname-a 2024-09-11T01:29:03.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.378+0000 7fb3b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:03.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.378+0000 7fb3b3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:03.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.378+0000 7fb3b3fff640 1 -- 172.21.15.12:0/1597759190 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb398009b90 con 0x7fb398005b40 2024-09-11T01:29:03.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.381+0000 7fb3b3fff640 1 -- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb398005b40 msgr2=0x7fb398008730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:03.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.381+0000 7fb3b3fff640 1 --2- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb398005b40 0x7fb398008730 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0006e80 tx=0x7fb3a002abf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc95d04fccea47d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:03.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.381+0000 7fb3b37fe640 1 -- 172.21.15.12:0/1597759190 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a002c7f0 con 0x7fb398005b40 2024-09-11T01:29:03.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.381+0000 7fb3b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:03.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.381+0000 7fb3b37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:03.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.381+0000 7fb3b37fe640 10 monclient: dump: 2024-09-11T01:29:03.387 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:03.387 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:03.387 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:03.387 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:03.387 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:03.387 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:03.387 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:03.387 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:03.387 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:03.387 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:03.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.381+0000 7fb3b37fe640 10 monclient: _finish_auth 0 2024-09-11T01:29:03.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.381+0000 7fb3b37fe640 10 monclient: _check_auth_tickets 2024-09-11T01:29:03.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.381+0000 7fb3b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:03.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.381+0000 7fb3b37fe640 1 -- 172.21.15.12:0/1597759190 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3a002c990 con 0x7fb398005b40 2024-09-11T01:29:03.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.381+0000 7fb3b37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:03.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.381+0000 7fb3b37fe640 1 -- 172.21.15.12:0/1597759190 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a002cd60 con 0x7fb398005b40 2024-09-11T01:29:03.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.381+0000 7fb3baccd640 5 monclient: authenticate success, global_id 4104 2024-09-11T01:29:03.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.381+0000 7fb3baccd640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:03.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.381+0000 7fb3baccd640 4 set_mon_vals no callback set 2024-09-11T01:29:03.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.381+0000 7fb3b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:03.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.381+0000 7fb3b37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:03.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.381+0000 7fb3b37fe640 10 monclient: dump: 2024-09-11T01:29:03.388 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:03.388 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:03.389 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:03.389 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:03.389 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:03.389 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:03.389 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:03.389 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:03.389 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:03.389 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:03.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.382+0000 7fb3baccd640 10 monclient: shutdown 2024-09-11T01:29:03.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.382+0000 7fb3baccd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:03.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.382+0000 7fb3baccd640 1 -- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb398005b40 msgr2=0x7fb398008730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:03.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.382+0000 7fb3baccd640 1 --2- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb398005b40 0x7fb398008730 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0006e80 tx=0x7fb3a002abf0 comp rx=0 tx=0).stop 2024-09-11T01:29:03.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.382+0000 7fb3baccd640 1 -- 172.21.15.12:0/1597759190 shutdown_connections 2024-09-11T01:29:03.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.382+0000 7fb3baccd640 1 --2- 172.21.15.12:0/1597759190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb398005b40 0x7fb398008730 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:03.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.382+0000 7fb3baccd640 1 -- 172.21.15.12:0/1597759190 >> 172.21.15.12:0/1597759190 conn(0x7fb3b40aaa80 msgr2=0x7fb3b40aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:03.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.382+0000 7fb3baccd640 1 -- 172.21.15.12:0/1597759190 shutdown_connections 2024-09-11T01:29:03.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.382+0000 7fb3baccd640 1 -- 172.21.15.12:0/1597759190 wait complete. 2024-09-11T01:29:03.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.383+0000 7fb3baccd640 10 monclient: build_initial_monmap 2024-09-11T01:29:03.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.383+0000 7fb3baccd640 10 monclient: monmap: 2024-09-11T01:29:03.390 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:03.390 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:03.390 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:03.390 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:03.390 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:03.390 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:03.390 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:03.390 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:03.391 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:03.391 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:03.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.384+0000 7fb3baccd640 1 Processor -- start 2024-09-11T01:29:03.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.384+0000 7fb3baccd640 1 -- start start 2024-09-11T01:29:03.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.384+0000 7fb3baccd640 10 monclient: init 2024-09-11T01:29:03.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.384+0000 7fb3baccd640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:03.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.384+0000 7fb3baccd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:29:03.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.384+0000 7fb3baccd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb398005b40 0x7fb3b40a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:03.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.384+0000 7fb3baccd640 10 monclient(hunting): picked mon.noname-b con 0x7fb398005b40 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:03.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3baccd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb398008ee0 0x7fb3b4138ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:03.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3baccd640 10 monclient(hunting): picked mon.noname-a con 0x7fb398008ee0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:03.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3baccd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb39800a170 0x7fb3b4139410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3baccd640 10 monclient(hunting): picked mon.noname-c con 0x7fb39800a170 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3baccd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3baccd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb3b406f4f0 con 0x7fb398008ee0 2024-09-11T01:29:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3baccd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3baccd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb3b409c2a0 con 0x7fb398005b40 2024-09-11T01:29:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3baccd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3baccd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb3b406f670 con 0x7fb39800a170 2024-09-11T01:29:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3baccd640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3baccd640 10 monclient(hunting): authenticate will time out at 749.828054s 2024-09-11T01:29:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb398008ee0 0x7fb3b4138ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b926a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb39800a170 0x7fb3b4139410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb398008ee0 0x7fb3b4138ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44178/0 (socket says 172.21.15.12:44178) 2024-09-11T01:29:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b926a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb39800a170 0x7fb3b4139410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51080/0 (socket says 172.21.15.12:51080) 2024-09-11T01:29:03.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b3fff640 1 -- 172.21.15.12:0/3682282985 learned_addr learned my addr 172.21.15.12:0/3682282985 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:03.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b3fff640 10 monclient(hunting): get_auth_request con 0x7fb398008ee0 auth_method 0 2024-09-11T01:29:03.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:03.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:03.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:03.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b926a640 10 monclient(hunting): get_auth_request con 0x7fb39800a170 auth_method 0 2024-09-11T01:29:03.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b8a69640 1 --2- 172.21.15.12:0/3682282985 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb398005b40 0x7fb3b40a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:03.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b926a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:03.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b926a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:03.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b926a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:03.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b8a69640 10 monclient(hunting): get_auth_request con 0x7fb398005b40 auth_method 0 2024-09-11T01:29:03.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b8a69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:03.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b8a69640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:03.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b8a69640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:03.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b926a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:03.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b926a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:03.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b926a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:03.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b8a69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:03.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b8a69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:03.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b8a69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:03.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:03.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:03.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:03.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b926a640 10 monclient(hunting): handle_auth_done global_id 4103 payload 824 2024-09-11T01:29:03.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b926a640 1 -- 172.21.15.12:0/3682282985 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb398005b40 msgr2=0x7fb3b40a4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:03.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b926a640 1 --2- 172.21.15.12:0/3682282985 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb398005b40 0x7fb3b40a4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:03.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b926a640 1 -- 172.21.15.12:0/3682282985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb398008ee0 msgr2=0x7fb3b4138ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:03.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b926a640 1 --2- 172.21.15.12:0/3682282985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb398008ee0 0x7fb3b4138ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:03.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b926a640 10 monclient: _finish_hunting 0 2024-09-11T01:29:03.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b926a640 1 monclient: found mon.noname-c 2024-09-11T01:29:03.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b926a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:03.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b926a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:29:03.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b926a640 1 -- 172.21.15.12:0/3682282985 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3b413bff0 con 0x7fb39800a170 2024-09-11T01:29:03.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b3fff640 1 --2- 172.21.15.12:0/3682282985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb398008ee0 0x7fb3b4138ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:03.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.385+0000 7fb3b8a69640 1 --2- 172.21.15.12:0/3682282985 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb398005b40 0x7fb3b40a4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:03.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.388+0000 7fb3b926a640 1 -- 172.21.15.12:0/3682282985 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb39800a170 msgr2=0x7fb3b4139410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:03.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.388+0000 7fb3b926a640 1 --2- 172.21.15.12:0/3682282985 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb39800a170 0x7fb3b4139410 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb3b40a7050 tx=0x7fb3ac009c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:03.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.388+0000 7fb3b17fa640 1 -- 172.21.15.12:0/3682282985 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac01b070 con 0x7fb39800a170 2024-09-11T01:29:03.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.388+0000 7fb3b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:03.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.388+0000 7fb3b17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:03.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.388+0000 7fb3b17fa640 10 monclient: dump: 2024-09-11T01:29:03.400 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:03.400 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:03.400 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:03.400 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:03.400 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:03.400 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:03.400 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:03.400 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:03.400 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:03.400 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:03.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.388+0000 7fb3b17fa640 10 monclient: _finish_auth 0 2024-09-11T01:29:03.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.388+0000 7fb3b17fa640 10 monclient: _check_auth_tickets 2024-09-11T01:29:03.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.388+0000 7fb3b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:03.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.388+0000 7fb3b17fa640 1 -- 172.21.15.12:0/3682282985 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3ac013b20 con 0x7fb39800a170 2024-09-11T01:29:03.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.388+0000 7fb3b17fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:03.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.388+0000 7fb3b17fa640 1 -- 172.21.15.12:0/3682282985 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac017430 con 0x7fb39800a170 2024-09-11T01:29:03.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.388+0000 7fb3b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:03.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.388+0000 7fb3b17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:03.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.388+0000 7fb3b17fa640 10 monclient: dump: 2024-09-11T01:29:03.401 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:03.401 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:03.401 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:03.401 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:03.401 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:03.401 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:03.401 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:03.402 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.388+0000 7fb3baccd640 5 monclient: authenticate success, global_id 4103 2024-09-11T01:29:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.388+0000 7fb3baccd640 10 monclient: _renew_subs 2024-09-11T01:29:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.388+0000 7fb3baccd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.388+0000 7fb3baccd640 1 -- 172.21.15.12:0/3682282985 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3b413c2d0 con 0x7fb39800a170 2024-09-11T01:29:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.388+0000 7fb3b27fc640 4 set_mon_vals no callback set 2024-09-11T01:29:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.388+0000 7fb3baccd640 10 monclient: _renew_subs 2024-09-11T01:29:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.388+0000 7fb3baccd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.388+0000 7fb3baccd640 1 -- 172.21.15.12:0/3682282985 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3b413bd10 con 0x7fb39800a170 2024-09-11T01:29:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.388+0000 7fb3b17fa640 1 -- 172.21.15.12:0/3682282985 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7fb3ac0175d0 con 0x7fb39800a170 2024-09-11T01:29:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.388+0000 7fb3b17fa640 1 -- 172.21.15.12:0/3682282985 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb3ac020ba0 con 0x7fb39800a170 2024-09-11T01:29:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.389+0000 7fb3baccd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:03.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.389+0000 7fb3baccd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:03.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.389+0000 7fb3baccd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:03.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.389+0000 7fb3baccd640 1 -- 172.21.15.12:0/3682282985 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3b40a9af0 con 0x7fb39800a170 2024-09-11T01:29:03.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.392+0000 7fb3b17fa640 1 -- 172.21.15.12:0/3682282985 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7fb3ac020450 con 0x7fb39800a170 2024-09-11T01:29:03.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.392+0000 7fb3b17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:03.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.392+0000 7fb3b17fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.460+0000 7fb3baccd640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-11T01:29:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.460+0000 7fb3baccd640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-11T01:29:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.460+0000 7fb3baccd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.460+0000 7fb3baccd640 1 -- 172.21.15.12:0/3682282985 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fb3b409d8f0 con 0x7fb39800a170 2024-09-11T01:29:03.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.531+0000 7fb3b17fa640 1 -- 172.21.15.12:0/3682282985 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fb3b409d8f0 con 0x7fb39800a170 2024-09-11T01:29:03.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.531+0000 7fb3b17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-11T01:29:03.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.531+0000 7fb3b17fa640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-11T01:29:03.534 INFO:teuthology.orchestra.run.smithi012.stderr:adjusted tunables profile to default 2024-09-11T01:29:03.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.532+0000 7fb3baccd640 10 monclient: shutdown 2024-09-11T01:29:03.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.532+0000 7fb3baccd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:03.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.532+0000 7fb3baccd640 1 -- 172.21.15.12:0/3682282985 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb39800a170 msgr2=0x7fb3b4139410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:03.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.532+0000 7fb3baccd640 1 --2- 172.21.15.12:0/3682282985 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb39800a170 0x7fb3b4139410 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb3b40a7050 tx=0x7fb3ac009c40 comp rx=0 tx=0).stop 2024-09-11T01:29:03.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.532+0000 7fb3baccd640 1 -- 172.21.15.12:0/3682282985 shutdown_connections 2024-09-11T01:29:03.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.532+0000 7fb3baccd640 1 --2- 172.21.15.12:0/3682282985 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb39800a170 0x7fb3b4139410 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:03.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.532+0000 7fb3baccd640 1 --2- 172.21.15.12:0/3682282985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb398008ee0 0x7fb3b4138ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:03.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.532+0000 7fb3baccd640 1 --2- 172.21.15.12:0/3682282985 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb398005b40 0x7fb3b40a4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:03.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.532+0000 7fb3baccd640 1 -- 172.21.15.12:0/3682282985 >> 172.21.15.12:0/3682282985 conn(0x7fb3b40aaa80 msgr2=0x7fb3b4137410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:03.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.532+0000 7fb3baccd640 1 -- 172.21.15.12:0/3682282985 shutdown_connections 2024-09-11T01:29:03.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:03.532+0000 7fb3baccd640 1 -- 172.21.15.12:0/3682282985 wait complete. 2024-09-11T01:29:03.547 INFO:tasks.ceph:check_enable_crimson: False 2024-09-11T01:29:03.547 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-11T01:29:03.547 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-09-11T01:29:03.547 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-11T01:29:03.575 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-09-11T01:29:03.575 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-11T01:29:03.642 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-09-11T01:29:03.642 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-11T01:29:03.709 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-11T01:29:03.709 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-11T01:29:03.734 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-11T01:29:03.734 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-11T01:29:03.799 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-11T01:29:03.799 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-11T01:29:03.864 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new 616ba1d0-adf6-4fe1-85e3-266e53e21867 0 2024-09-11T01:29:04.002 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:03.999+0000 7f6be42ed640 10 monclient: get_monmap_and_config 2024-09-11T01:29:04.002 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:03.999+0000 7f6be42ed640 10 monclient: build_initial_monmap 2024-09-11T01:29:04.002 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:03.999+0000 7f6be42ed640 10 monclient: monmap: 2024-09-11T01:29:04.002 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-09-11T01:29:04.002 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:04.002 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:29:04.000804+0000 2024-09-11T01:29:04.002 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:29:04.000804+0000 2024-09-11T01:29:04.002 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:04.002 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.002 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:04.002 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:04.003 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:04.003 INFO:teuthology.orchestra.run.smithi071.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:04.003 INFO:teuthology.orchestra.run.smithi071.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:04.003 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.003 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.000+0000 7f6be42ed640 1 Processor -- start 2024-09-11T01:29:04.003 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.000+0000 7f6be42ed640 1 -- start start 2024-09-11T01:29:04.003 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.000+0000 7f6be42ed640 10 monclient: init 2024-09-11T01:29:04.007 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.000+0000 7f6be42ed640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:04.007 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.000+0000 7f6be42ed640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-11T01:29:04.007 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.000+0000 7f6be42ed640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f6bdc138210 0x7f6bdc1385f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:04.007 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.000+0000 7f6be42ed640 10 monclient(hunting): picked mon.noname-c con 0x7f6bdc138210 addr v2:172.21.15.71:3300/0 2024-09-11T01:29:04.007 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.000+0000 7f6be42ed640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6bdc0a6810 0x7f6bdc0a5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:04.007 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be42ed640 10 monclient(hunting): picked mon.noname-a con 0x7f6bdc0a6810 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:04.007 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be42ed640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6bdc0a55d0 0x7f6bdc0a59e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:04.007 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be42ed640 10 monclient(hunting): picked mon.noname-b con 0x7f6bdc0a55d0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:04.007 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be42ed640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:04.007 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be42ed640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f6bdc0a5f20 con 0x7f6bdc0a6810 2024-09-11T01:29:04.007 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be42ed640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:04.007 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be42ed640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f6bdc09c7a0 con 0x7f6bdc138210 2024-09-11T01:29:04.008 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be42ed640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:04.008 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be42ed640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6bdc09c4c0 con 0x7f6bdc0a55d0 2024-09-11T01:29:04.008 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be42ed640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:04.008 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be42ed640 10 monclient(hunting): authenticate will time out at 732.457059s 2024-09-11T01:29:04.008 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be2089640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f6bdc138210 0x7f6bdc1385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:04.008 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be2089640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f6bdc138210 0x7f6bdc1385f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:53060/0 (socket says 172.21.15.71:53060) 2024-09-11T01:29:04.008 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be2089640 1 -- 172.21.15.71:0/4190769586 learned_addr learned my addr 172.21.15.71:0/4190769586 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-09-11T01:29:04.008 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be2089640 10 monclient(hunting): get_auth_request con 0x7f6bdc138210 auth_method 0 2024-09-11T01:29:04.008 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be2089640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:04.008 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be2089640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:04.008 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be2089640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:04.008 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be2089640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:04.008 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be2089640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:04.009 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be2089640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:04.009 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be288a640 1 --2- 172.21.15.71:0/4190769586 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6bdc0a55d0 0x7f6bdc0a59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:04.009 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be1888640 1 --2- 172.21.15.71:0/4190769586 >> v2:172.21.15.12:3300/0 conn(0x7f6bdc0a6810 0x7f6bdc0a5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:04.009 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be1888640 10 monclient(hunting): get_auth_request con 0x7f6bdc0a6810 auth_method 0 2024-09-11T01:29:04.009 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be1888640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:04.009 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be1888640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:04.009 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be1888640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:04.009 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be288a640 10 monclient(hunting): get_auth_request con 0x7f6bdc0a55d0 auth_method 0 2024-09-11T01:29:04.009 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be288a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:04.009 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be288a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:04.009 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be288a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:04.009 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be288a640 1 -- 172.21.15.71:0/4190769586 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6bdc0a55d0 msgr2=0x7f6bdc0a59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:29:04.009 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be288a640 1 -- 172.21.15.71:0/4190769586 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6bdc0a55d0 msgr2=0x7f6bdc0a59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:29:04.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be288a640 1 --2- 172.21.15.71:0/4190769586 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6bdc0a55d0 0x7f6bdc0a59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:29:04.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be2089640 10 monclient(hunting): handle_auth_done global_id 4102 payload 274 2024-09-11T01:29:04.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be2089640 1 -- 172.21.15.71:0/4190769586 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6bdc0a55d0 msgr2=0x7f6bdc0a59e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:29:04.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be288a640 1 --2- 172.21.15.71:0/4190769586 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6bdc0a55d0 0x7f6bdc0a59e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:29:04.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be2089640 1 --2- 172.21.15.71:0/4190769586 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6bdc0a55d0 0x7f6bdc0a59e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be2089640 1 -- 172.21.15.71:0/4190769586 >> v2:172.21.15.12:3300/0 conn(0x7f6bdc0a6810 msgr2=0x7f6bdc0a5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:04.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be2089640 1 --2- 172.21.15.71:0/4190769586 >> v2:172.21.15.12:3300/0 conn(0x7f6bdc0a6810 0x7f6bdc0a5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be2089640 10 monclient: _finish_hunting 0 2024-09-11T01:29:04.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be2089640 1 monclient: found mon.noname-c 2024-09-11T01:29:04.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be2089640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:04.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be2089640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be2089640 1 -- 172.21.15.71:0/4190769586 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bdc09ba20 con 0x7f6bdc138210 2024-09-11T01:29:04.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.001+0000 7f6be1888640 1 --2- 172.21.15.71:0/4190769586 >> v2:172.21.15.12:3300/0 conn(0x7f6bdc0a6810 0x7f6bdc0a5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:29:04.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.004+0000 7f6be2089640 1 -- 172.21.15.71:0/4190769586 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6bdc138210 msgr2=0x7f6bdc1385f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:04.011 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.004+0000 7f6be2089640 1 --2- 172.21.15.71:0/4190769586 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6bdc138210 0x7f6bdc1385f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc0a8230 tx=0x7f6bd802a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c2605f018092e24 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:04.011 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.004+0000 7f6be1087640 1 -- 172.21.15.71:0/4190769586 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8037070 con 0x7f6bdc138210 2024-09-11T01:29:04.011 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.004+0000 7f6be1087640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:04.011 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.004+0000 7f6be1087640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:04.011 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.004+0000 7f6be1087640 10 monclient: dump: 2024-09-11T01:29:04.011 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-09-11T01:29:04.011 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:04.011 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.011 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.011 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:04.011 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.011 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:04.012 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:04.012 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:04.012 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.004+0000 7f6be1087640 10 monclient: _finish_auth 0 2024-09-11T01:29:04.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.004+0000 7f6be1087640 10 monclient: _check_auth_tickets 2024-09-11T01:29:04.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.004+0000 7f6be1087640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:04.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.004+0000 7f6be42ed640 5 monclient: authenticate success, global_id 4102 2024-09-11T01:29:04.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.004+0000 7f6be1087640 1 -- 172.21.15.71:0/4190769586 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bd8032640 con 0x7f6bdc138210 2024-09-11T01:29:04.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.004+0000 7f6be42ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:04.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.004+0000 7f6be1087640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:04.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.004+0000 7f6be42ed640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:04.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.004+0000 7f6be42ed640 4 set_mon_vals no callback set 2024-09-11T01:29:04.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.004+0000 7f6be1087640 1 -- 172.21.15.71:0/4190769586 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8032a10 con 0x7f6bdc138210 2024-09-11T01:29:04.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.004+0000 7f6be1087640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:04.013 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.004+0000 7f6be1087640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:04.013 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.004+0000 7f6be1087640 10 monclient: dump: 2024-09-11T01:29:04.013 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-09-11T01:29:04.013 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:04.013 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.013 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.013 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:04.013 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.013 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:04.013 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:04.013 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:04.013 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.013 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.005+0000 7f6be42ed640 10 monclient: shutdown 2024-09-11T01:29:04.014 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.005+0000 7f6be42ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:04.014 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.005+0000 7f6be42ed640 1 -- 172.21.15.71:0/4190769586 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6bdc138210 msgr2=0x7f6bdc1385f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:04.014 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.005+0000 7f6be42ed640 1 --2- 172.21.15.71:0/4190769586 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6bdc138210 0x7f6bdc1385f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc0a8230 tx=0x7f6bd802a100 comp rx=0 tx=0).stop 2024-09-11T01:29:04.014 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.005+0000 7f6be42ed640 1 -- 172.21.15.71:0/4190769586 shutdown_connections 2024-09-11T01:29:04.014 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.005+0000 7f6be42ed640 1 --2- 172.21.15.71:0/4190769586 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6bdc0a55d0 0x7f6bdc0a59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.014 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.005+0000 7f6be42ed640 1 --2- 172.21.15.71:0/4190769586 >> v2:172.21.15.12:3300/0 conn(0x7f6bdc0a6810 0x7f6bdc0a5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.014 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.005+0000 7f6be42ed640 1 --2- 172.21.15.71:0/4190769586 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6bdc138210 0x7f6bdc1385f0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.014 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.005+0000 7f6be42ed640 1 -- 172.21.15.71:0/4190769586 >> 172.21.15.71:0/4190769586 conn(0x7f6bdc0aaa80 msgr2=0x7f6bdc0aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:04.014 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.005+0000 7f6be42ed640 1 -- 172.21.15.71:0/4190769586 shutdown_connections 2024-09-11T01:29:04.014 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.005+0000 7f6be42ed640 1 -- 172.21.15.71:0/4190769586 wait complete. 2024-09-11T01:29:04.019 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.005+0000 7f6be42ed640 10 monclient: build_initial_monmap 2024-09-11T01:29:04.019 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.005+0000 7f6be42ed640 10 monclient: monmap: 2024-09-11T01:29:04.019 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-09-11T01:29:04.019 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:04.019 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 0.000000 2024-09-11T01:29:04.019 INFO:teuthology.orchestra.run.smithi071.stderr:created 0.000000 2024-09-11T01:29:04.020 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:04.020 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.020 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:04.020 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:04.020 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:04.020 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.020 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be42ed640 1 Processor -- start 2024-09-11T01:29:04.020 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be42ed640 1 -- start start 2024-09-11T01:29:04.020 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be42ed640 10 monclient: init 2024-09-11T01:29:04.020 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be42ed640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:04.020 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be42ed640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:29:04.020 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be42ed640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bdc0a55d0 0x7f6bdc1367b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:04.020 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be42ed640 10 monclient(hunting): picked mon.noname-a con 0x7f6bdc0a55d0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:04.021 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be42ed640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6bdc0a6810 0x7f6bdc136cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:04.021 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be42ed640 10 monclient(hunting): picked mon.noname-b con 0x7f6bdc0a6810 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:04.021 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be42ed640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6bdc138210 0x7f6bdc120ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:04.021 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be42ed640 10 monclient(hunting): picked mon.noname-c con 0x7f6bdc138210 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:04.021 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be42ed640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:04.021 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be42ed640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6bdc06f4f0 con 0x7f6bdc0a55d0 2024-09-11T01:29:04.021 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be42ed640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:04.021 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be42ed640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6bdc09d620 con 0x7f6bdc0a6810 2024-09-11T01:29:04.021 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be42ed640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:04.021 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be42ed640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6bdc06f670 con 0x7f6bdc138210 2024-09-11T01:29:04.021 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be42ed640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:04.021 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be42ed640 10 monclient(hunting): authenticate will time out at 732.462629s 2024-09-11T01:29:04.021 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be1888640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6bdc0a6810 0x7f6bdc136cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:04.021 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be1888640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6bdc0a6810 0x7f6bdc136cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:53066/0 (socket says 172.21.15.71:53066) 2024-09-11T01:29:04.022 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be1888640 1 -- 172.21.15.71:0/2933158601 learned_addr learned my addr 172.21.15.71:0/2933158601 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-09-11T01:29:04.022 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be1888640 10 monclient(hunting): get_auth_request con 0x7f6bdc0a6810 auth_method 0 2024-09-11T01:29:04.022 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be1888640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:04.022 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be1888640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:04.022 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be1888640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:04.022 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be1888640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:04.022 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be1888640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:04.022 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be1888640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:04.022 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be2089640 1 --2- 172.21.15.71:0/2933158601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bdc0a55d0 0x7f6bdc1367b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:04.022 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be288a640 1 --2- 172.21.15.71:0/2933158601 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6bdc138210 0x7f6bdc120ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:04.022 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be2089640 10 monclient(hunting): get_auth_request con 0x7f6bdc0a55d0 auth_method 0 2024-09-11T01:29:04.022 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be2089640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:04.023 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be2089640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:04.023 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be2089640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:04.023 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be288a640 10 monclient(hunting): get_auth_request con 0x7f6bdc138210 auth_method 0 2024-09-11T01:29:04.023 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be288a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:04.023 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be288a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:04.023 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be288a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:04.023 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be2089640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:04.023 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be2089640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:04.023 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be2089640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:04.023 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be288a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:04.023 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be288a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:04.023 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.006+0000 7f6be288a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:04.023 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.007+0000 7f6be1888640 10 monclient(hunting): handle_auth_done global_id 4105 payload 824 2024-09-11T01:29:04.023 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.007+0000 7f6be1888640 1 -- 172.21.15.71:0/2933158601 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6bdc138210 msgr2=0x7f6bdc120ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:04.024 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.007+0000 7f6be1888640 1 --2- 172.21.15.71:0/2933158601 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6bdc138210 0x7f6bdc120ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.024 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.007+0000 7f6be1888640 1 -- 172.21.15.71:0/2933158601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bdc0a55d0 msgr2=0x7f6bdc1367b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:04.024 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.007+0000 7f6be1888640 1 --2- 172.21.15.71:0/2933158601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bdc0a55d0 0x7f6bdc1367b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.024 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.007+0000 7f6be1888640 10 monclient: _finish_hunting 0 2024-09-11T01:29:04.024 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.007+0000 7f6be1888640 1 monclient: found mon.noname-b 2024-09-11T01:29:04.024 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.007+0000 7f6be1888640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:04.024 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.007+0000 7f6be1888640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.024 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.007+0000 7f6be1888640 1 -- 172.21.15.71:0/2933158601 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bdc124650 con 0x7f6bdc0a6810 2024-09-11T01:29:04.024 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.009+0000 7f6be1888640 1 -- 172.21.15.71:0/2933158601 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6bdc0a6810 msgr2=0x7f6bdc136cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:04.024 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.009+0000 7f6be1888640 1 --2- 172.21.15.71:0/2933158601 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6bdc0a6810 0x7f6bdc136cf0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00e760 tx=0x7f6bcc00ec20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:04.024 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.009+0000 7f6bd2ffd640 1 -- 172.21.15.71:0/2933158601 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc00f040 con 0x7f6bdc0a6810 2024-09-11T01:29:04.024 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.009+0000 7f6bd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:04.024 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.009+0000 7f6bd2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:04.025 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.009+0000 7f6bd2ffd640 10 monclient: dump: 2024-09-11T01:29:04.025 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-09-11T01:29:04.025 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:04.025 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.025 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.025 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:04.025 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.025 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:04.025 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:04.025 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:04.025 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.025 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.009+0000 7f6bd2ffd640 10 monclient: _finish_auth 0 2024-09-11T01:29:04.025 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.009+0000 7f6bd2ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:04.026 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.009+0000 7f6bd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:04.026 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.009+0000 7f6bd2ffd640 1 -- 172.21.15.71:0/2933158601 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bcc004230 con 0x7f6bdc0a6810 2024-09-11T01:29:04.026 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.009+0000 7f6be42ed640 5 monclient: authenticate success, global_id 4105 2024-09-11T01:29:04.026 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.009+0000 7f6be42ed640 10 monclient: _renew_subs 2024-09-11T01:29:04.026 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.009+0000 7f6be42ed640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.026 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.009+0000 7f6be42ed640 1 -- 172.21.15.71:0/2933158601 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bdc0a1880 con 0x7f6bdc0a6810 2024-09-11T01:29:04.026 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.009+0000 7f6bd2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:04.026 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.009+0000 7f6bd2ffd640 1 -- 172.21.15.71:0/2933158601 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc00ee70 con 0x7f6bdc0a6810 2024-09-11T01:29:04.026 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.009+0000 7f6bd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:04.026 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.009+0000 7f6bd2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:04.026 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.009+0000 7f6bd2ffd640 10 monclient: dump: 2024-09-11T01:29:04.026 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-09-11T01:29:04.026 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:04.026 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.027 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.027 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:04.027 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.027 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:04.027 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:04.027 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:04.027 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.027 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.009+0000 7f6be42ed640 10 monclient: _renew_subs 2024-09-11T01:29:04.027 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.009+0000 7f6be0886640 4 set_mon_vals no callback set 2024-09-11T01:29:04.027 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.009+0000 7f6be42ed640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.027 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.009+0000 7f6be42ed640 1 -- 172.21.15.71:0/2933158601 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bdc124370 con 0x7f6bdc0a6810 2024-09-11T01:29:04.027 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.009+0000 7f6bd2ffd640 1 -- 172.21.15.71:0/2933158601 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f6bcc012050 con 0x7f6bdc0a6810 2024-09-11T01:29:04.027 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.009+0000 7f6bd2ffd640 1 -- 172.21.15.71:0/2933158601 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f6bcc013750 con 0x7f6bdc0a6810 2024-09-11T01:29:04.027 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.010+0000 7f6be42ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:04.027 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.010+0000 7f6be42ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:04.028 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.010+0000 7f6be42ed640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.028 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.010+0000 7f6be42ed640 1 -- 172.21.15.71:0/2933158601 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bdc0a5090 con 0x7f6bdc0a6810 2024-09-11T01:29:04.028 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.012+0000 7f6bd2ffd640 1 -- 172.21.15.71:0/2933158601 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f6bcc017020 con 0x7f6bdc0a6810 2024-09-11T01:29:04.028 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.012+0000 7f6bd2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:04.028 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.012+0000 7f6bd2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:04.083 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.081+0000 7f6be42ed640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "616ba1d0-adf6-4fe1-85e3-266e53e21867", "id": 0}] 2024-09-11T01:29:04.084 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.081+0000 7f6be42ed640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "616ba1d0-adf6-4fe1-85e3-266e53e21867", "id": 0}] 2024-09-11T01:29:04.084 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.081+0000 7f6be42ed640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.084 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.081+0000 7f6be42ed640 1 -- 172.21.15.71:0/2933158601 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd new", "uuid": "616ba1d0-adf6-4fe1-85e3-266e53e21867", "id": 0} v 0) -- 0x7f6bdc137590 con 0x7f6bdc0a6810 2024-09-11T01:29:04.089 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.088+0000 7f6bd2ffd640 1 -- 172.21.15.71:0/2933158601 <== mon.1 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "616ba1d0-adf6-4fe1-85e3-266e53e21867", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f6bcc00cda0 con 0x7f6bdc0a6810 2024-09-11T01:29:04.090 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.088+0000 7f6bd2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "616ba1d0-adf6-4fe1-85e3-266e53e21867", "id": 0}] 2024-09-11T01:29:04.090 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.088+0000 7f6bd2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:04.090 INFO:teuthology.orchestra.run.smithi071.stdout:0 2024-09-11T01:29:04.091 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.088+0000 7f6be42ed640 10 monclient: shutdown 2024-09-11T01:29:04.091 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.088+0000 7f6be42ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:04.091 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.088+0000 7f6be42ed640 1 -- 172.21.15.71:0/2933158601 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6bdc0a6810 msgr2=0x7f6bdc136cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:04.091 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.088+0000 7f6be42ed640 1 --2- 172.21.15.71:0/2933158601 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6bdc0a6810 0x7f6bdc136cf0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00e760 tx=0x7f6bcc00ec20 comp rx=0 tx=0).stop 2024-09-11T01:29:04.091 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.089+0000 7f6be42ed640 1 -- 172.21.15.71:0/2933158601 shutdown_connections 2024-09-11T01:29:04.091 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.089+0000 7f6be42ed640 1 --2- 172.21.15.71:0/2933158601 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6bdc138210 0x7f6bdc120ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.091 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.089+0000 7f6be42ed640 1 --2- 172.21.15.71:0/2933158601 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6bdc0a6810 0x7f6bdc136cf0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.091 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.089+0000 7f6be42ed640 1 --2- 172.21.15.71:0/2933158601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6bdc0a55d0 0x7f6bdc1367b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.092 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.089+0000 7f6be42ed640 1 -- 172.21.15.71:0/2933158601 >> 172.21.15.71:0/2933158601 conn(0x7f6bdc0aaa80 msgr2=0x7f6bdc0a3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:04.092 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.089+0000 7f6be42ed640 1 -- 172.21.15.71:0/2933158601 shutdown_connections 2024-09-11T01:29:04.092 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.089+0000 7f6be42ed640 1 -- 172.21.15.71:0/2933158601 wait complete. 2024-09-11T01:29:04.100 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new 9b570874-7a3d-4867-8c44-29084a8bf649 1 2024-09-11T01:29:04.199 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.196+0000 7f037c2be640 10 monclient: get_monmap_and_config 2024-09-11T01:29:04.199 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.196+0000 7f037c2be640 10 monclient: build_initial_monmap 2024-09-11T01:29:04.199 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.196+0000 7f037c2be640 10 monclient: monmap: 2024-09-11T01:29:04.199 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-09-11T01:29:04.199 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:04.199 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:29:04.197898+0000 2024-09-11T01:29:04.199 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:29:04.197898+0000 2024-09-11T01:29:04.199 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:04.199 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.199 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:04.200 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:04.200 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:04.200 INFO:teuthology.orchestra.run.smithi071.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:04.200 INFO:teuthology.orchestra.run.smithi071.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:04.200 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.200 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.197+0000 7f037c2be640 1 Processor -- start 2024-09-11T01:29:04.200 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.197+0000 7f037c2be640 1 -- start start 2024-09-11T01:29:04.200 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.197+0000 7f037c2be640 10 monclient: init 2024-09-11T01:29:04.202 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037c2be640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:04.202 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037c2be640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-11T01:29:04.202 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037c2be640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0374138210 0x7f03741385f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:04.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037c2be640 10 monclient(hunting): picked mon.noname-b con 0x7f0374138210 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:04.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037c2be640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f03740a6810 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:04.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037c2be640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f03740a57a0 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:04.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037c2be640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f037409ba20 con 0x7f03740a6810 2024-09-11T01:29:04.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037c2be640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03740a6590 con 0x7f03740a57a0 2024-09-11T01:29:04.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037c2be640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:04.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037c2be640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f037409c7a0 con 0x7f0374138210 2024-09-11T01:29:04.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037c2be640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:04.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037c2be640 10 monclient(hunting): authenticate will time out at 732.654124s 2024-09-11T01:29:04.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037a85b640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f03740a57a0 0x7f0374140120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:48090/0 (socket says 172.21.15.71:48090) 2024-09-11T01:29:04.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037a85b640 1 -- 172.21.15.71:0/2571515564 learned_addr learned my addr 172.21.15.71:0/2571515564 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-09-11T01:29:04.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037a05a640 1 --2- 172.21.15.71:0/2571515564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0374138210 0x7f03741385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:04.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037a05a640 10 monclient(hunting): get_auth_request con 0x7f0374138210 auth_method 0 2024-09-11T01:29:04.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037a05a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:04.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037a05a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:04.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037a05a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:04.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037a05a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:04.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037a05a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:04.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037a05a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:04.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037a05a640 10 monclient(hunting): handle_auth_done global_id 4112 payload 274 2024-09-11T01:29:04.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037a05a640 1 -- 172.21.15.71:0/2571515564 >> v1:172.21.15.71:6789/0 conn(0x7f03740a57a0 legacy=0x7f0374140120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:04.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037a05a640 1 -- 172.21.15.71:0/2571515564 >> v1:172.21.15.12:6789/0 conn(0x7f03740a6810 legacy=0x7f03740a5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:04.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037a05a640 10 monclient: _finish_hunting 0 2024-09-11T01:29:04.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037a05a640 1 monclient: found mon.noname-b 2024-09-11T01:29:04.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037a05a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:04.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037a05a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:29:04.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.198+0000 7f037a05a640 1 -- 172.21.15.71:0/2571515564 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f037409c1e0 con 0x7f0374138210 2024-09-11T01:29:04.210 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.201+0000 7f037a05a640 1 -- 172.21.15.71:0/2571515564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0374138210 msgr2=0x7f03741385f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:04.210 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.201+0000 7f037a05a640 1 --2- 172.21.15.71:0/2571515564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0374138210 0x7f03741385f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f03700097d0 tx=0x7f0370029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d53ef0fb5985c66d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:04.211 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.201+0000 7f0379058640 1 -- 172.21.15.71:0/2571515564 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0370037070 con 0x7f0374138210 2024-09-11T01:29:04.211 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.201+0000 7f0379058640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:04.211 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.201+0000 7f0379058640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:04.211 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.201+0000 7f0379058640 10 monclient: dump: 2024-09-11T01:29:04.211 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-09-11T01:29:04.211 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:04.211 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.211 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.211 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:04.211 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.211 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:04.211 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:04.211 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:04.211 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.212 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.201+0000 7f0379058640 10 monclient: _finish_auth 0 2024-09-11T01:29:04.212 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.201+0000 7f0379058640 10 monclient: _check_auth_tickets 2024-09-11T01:29:04.212 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.201+0000 7f0379058640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:04.212 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.201+0000 7f0379058640 1 -- 172.21.15.71:0/2571515564 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0370032440 con 0x7f0374138210 2024-09-11T01:29:04.212 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.201+0000 7f037c2be640 5 monclient: authenticate success, global_id 4112 2024-09-11T01:29:04.212 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.201+0000 7f037c2be640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:04.212 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.201+0000 7f0379058640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:04.212 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.201+0000 7f037c2be640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:04.212 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.201+0000 7f0379058640 1 -- 172.21.15.71:0/2571515564 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0370032810 con 0x7f0374138210 2024-09-11T01:29:04.213 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.201+0000 7f0379058640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:04.213 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.201+0000 7f037c2be640 4 set_mon_vals no callback set 2024-09-11T01:29:04.213 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.201+0000 7f0379058640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:04.213 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.201+0000 7f0379058640 10 monclient: dump: 2024-09-11T01:29:04.213 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-09-11T01:29:04.213 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:04.213 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.213 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.213 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:04.213 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.214 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:04.214 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:04.214 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:04.214 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.214 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.202+0000 7f037c2be640 10 monclient: shutdown 2024-09-11T01:29:04.214 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.202+0000 7f037c2be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:04.214 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.202+0000 7f037c2be640 1 -- 172.21.15.71:0/2571515564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0374138210 msgr2=0x7f03741385f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:04.214 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.202+0000 7f037c2be640 1 --2- 172.21.15.71:0/2571515564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0374138210 0x7f03741385f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f03700097d0 tx=0x7f0370029fb0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.214 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.202+0000 7f037c2be640 1 -- 172.21.15.71:0/2571515564 shutdown_connections 2024-09-11T01:29:04.214 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.202+0000 7f037c2be640 1 --2- 172.21.15.71:0/2571515564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0374138210 0x7f03741385f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.214 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.202+0000 7f037c2be640 1 -- 172.21.15.71:0/2571515564 >> 172.21.15.71:0/2571515564 conn(0x7f03740aaa80 msgr2=0x7f03740aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:04.214 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.202+0000 7f037c2be640 1 -- 172.21.15.71:0/2571515564 shutdown_connections 2024-09-11T01:29:04.215 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.202+0000 7f037c2be640 1 -- 172.21.15.71:0/2571515564 wait complete. 2024-09-11T01:29:04.215 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.203+0000 7f037c2be640 10 monclient: build_initial_monmap 2024-09-11T01:29:04.215 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.203+0000 7f037c2be640 10 monclient: monmap: 2024-09-11T01:29:04.215 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-09-11T01:29:04.215 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:04.215 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 0.000000 2024-09-11T01:29:04.215 INFO:teuthology.orchestra.run.smithi071.stderr:created 0.000000 2024-09-11T01:29:04.215 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:04.215 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.215 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:04.215 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:04.215 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:04.215 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.216 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.203+0000 7f037c2be640 1 Processor -- start 2024-09-11T01:29:04.216 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.203+0000 7f037c2be640 1 -- start start 2024-09-11T01:29:04.216 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.203+0000 7f037c2be640 10 monclient: init 2024-09-11T01:29:04.216 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.203+0000 7f037c2be640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:04.216 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.203+0000 7f037c2be640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:29:04.216 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.203+0000 7f037c2be640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03740a57a0 0x7f0374136d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:04.216 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.203+0000 7f037c2be640 10 monclient(hunting): picked mon.noname-a con 0x7f03740a57a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:04.216 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.203+0000 7f037c2be640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03740a6810 0x7f0374120ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:04.216 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.203+0000 7f037c2be640 10 monclient(hunting): picked mon.noname-b con 0x7f03740a6810 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:04.216 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.203+0000 7f037c2be640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0374138210 0x7f0374121220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:04.216 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.203+0000 7f037c2be640 10 monclient(hunting): picked mon.noname-c con 0x7f0374138210 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:04.216 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.203+0000 7f037c2be640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:04.216 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.203+0000 7f037c2be640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f037406f670 con 0x7f03740a57a0 2024-09-11T01:29:04.217 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.203+0000 7f037c2be640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:04.217 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.203+0000 7f037c2be640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f037406f4f0 con 0x7f03740a6810 2024-09-11T01:29:04.217 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.203+0000 7f037c2be640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:04.217 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.203+0000 7f037c2be640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f0374138b30 con 0x7f0374138210 2024-09-11T01:29:04.217 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.203+0000 7f037c2be640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:04.217 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.203+0000 7f037c2be640 10 monclient(hunting): authenticate will time out at 732.659920s 2024-09-11T01:29:04.217 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f0379859640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03740a6810 0x7f0374120ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:04.217 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f0379859640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03740a6810 0x7f0374120ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:53070/0 (socket says 172.21.15.71:53070) 2024-09-11T01:29:04.217 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f0379859640 1 -- 172.21.15.71:0/2048922841 learned_addr learned my addr 172.21.15.71:0/2048922841 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-09-11T01:29:04.217 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f0379859640 10 monclient(hunting): get_auth_request con 0x7f03740a6810 auth_method 0 2024-09-11T01:29:04.217 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f0379859640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:04.217 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f0379859640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:04.217 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f0379859640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:04.217 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f0379859640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:04.218 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f0379859640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:04.218 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f0379859640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:04.218 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f037a05a640 1 --2- 172.21.15.71:0/2048922841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03740a57a0 0x7f0374136d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:04.218 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f037a85b640 1 --2- 172.21.15.71:0/2048922841 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0374138210 0x7f0374121220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:04.218 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f037a05a640 10 monclient(hunting): get_auth_request con 0x7f03740a57a0 auth_method 0 2024-09-11T01:29:04.218 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f037a05a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:04.218 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f037a05a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:04.218 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f037a05a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:04.218 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f037a85b640 10 monclient(hunting): get_auth_request con 0x7f0374138210 auth_method 0 2024-09-11T01:29:04.218 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f037a85b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:04.218 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f037a85b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:04.218 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f037a85b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:04.218 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f0379859640 10 monclient(hunting): handle_auth_done global_id 4108 payload 824 2024-09-11T01:29:04.218 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f0379859640 1 -- 172.21.15.71:0/2048922841 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0374138210 msgr2=0x7f0374121220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:04.219 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f0379859640 1 --2- 172.21.15.71:0/2048922841 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0374138210 0x7f0374121220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.219 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f0379859640 1 -- 172.21.15.71:0/2048922841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03740a57a0 msgr2=0x7f0374136d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:04.219 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f0379859640 1 --2- 172.21.15.71:0/2048922841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03740a57a0 0x7f0374136d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.219 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f0379859640 10 monclient: _finish_hunting 0 2024-09-11T01:29:04.219 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f0379859640 1 monclient: found mon.noname-b 2024-09-11T01:29:04.219 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f0379859640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:04.219 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f0379859640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.219 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f0379859640 1 -- 172.21.15.71:0/2048922841 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03740a10a0 con 0x7f03740a6810 2024-09-11T01:29:04.219 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.204+0000 7f037a05a640 1 --2- 172.21.15.71:0/2048922841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03740a57a0 0x7f0374136d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:29:04.219 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.206+0000 7f0379859640 1 -- 172.21.15.71:0/2048922841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03740a6810 msgr2=0x7f0374120ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:04.219 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.206+0000 7f0379859640 1 --2- 172.21.15.71:0/2048922841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03740a6810 0x7f0374120ce0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f036400c430 tx=0x7f036400c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:04.220 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.206+0000 7f036affd640 1 -- 172.21.15.71:0/2048922841 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03640119a0 con 0x7f03740a6810 2024-09-11T01:29:04.220 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.206+0000 7f036affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:04.220 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.206+0000 7f036affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:04.220 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.206+0000 7f036affd640 10 monclient: dump: 2024-09-11T01:29:04.220 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-09-11T01:29:04.220 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:04.220 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.220 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.220 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:04.220 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.220 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:04.220 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:04.220 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:04.221 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.221 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.206+0000 7f036affd640 10 monclient: _finish_auth 0 2024-09-11T01:29:04.221 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.206+0000 7f036affd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:04.221 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.206+0000 7f036affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:04.221 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.206+0000 7f036affd640 1 -- 172.21.15.71:0/2048922841 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0364017070 con 0x7f03740a6810 2024-09-11T01:29:04.221 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.206+0000 7f037c2be640 5 monclient: authenticate success, global_id 4108 2024-09-11T01:29:04.221 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.206+0000 7f036affd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:04.221 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.206+0000 7f037c2be640 10 monclient: _renew_subs 2024-09-11T01:29:04.221 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.206+0000 7f036affd640 1 -- 172.21.15.71:0/2048922841 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0364011d80 con 0x7f03740a6810 2024-09-11T01:29:04.221 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.206+0000 7f037c2be640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.221 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.206+0000 7f037c2be640 1 -- 172.21.15.71:0/2048922841 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03740a1880 con 0x7f03740a6810 2024-09-11T01:29:04.221 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.206+0000 7f036affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:04.221 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.206+0000 7f036affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:04.222 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.206+0000 7f036affd640 10 monclient: dump: 2024-09-11T01:29:04.222 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-09-11T01:29:04.222 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:04.222 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.222 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.222 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:04.222 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.222 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:04.222 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:04.222 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:04.222 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.222 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.206+0000 7f0378857640 4 set_mon_vals no callback set 2024-09-11T01:29:04.222 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.206+0000 7f037c2be640 10 monclient: _renew_subs 2024-09-11T01:29:04.222 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.207+0000 7f037c2be640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.222 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.207+0000 7f037c2be640 1 -- 172.21.15.71:0/2048922841 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03741247c0 con 0x7f03740a6810 2024-09-11T01:29:04.223 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.207+0000 7f036affd640 1 -- 172.21.15.71:0/2048922841 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f036401c460 con 0x7f03740a6810 2024-09-11T01:29:04.223 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.207+0000 7f036affd640 1 -- 172.21.15.71:0/2048922841 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f0364012a40 con 0x7f03740a6810 2024-09-11T01:29:04.223 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.207+0000 7f037c2be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:04.223 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.207+0000 7f037c2be640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:04.223 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.207+0000 7f037c2be640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.223 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.207+0000 7f037c2be640 1 -- 172.21.15.71:0/2048922841 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03740a5c60 con 0x7f03740a6810 2024-09-11T01:29:04.224 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.209+0000 7f036affd640 1 -- 172.21.15.71:0/2048922841 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f03740a10a0 con 0x7f03740a6810 2024-09-11T01:29:04.224 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.209+0000 7f036affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:04.224 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.209+0000 7f036affd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:04.282 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.280+0000 7f037c2be640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9b570874-7a3d-4867-8c44-29084a8bf649", "id": 1}] 2024-09-11T01:29:04.282 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.280+0000 7f037c2be640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9b570874-7a3d-4867-8c44-29084a8bf649", "id": 1}] 2024-09-11T01:29:04.282 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.280+0000 7f037c2be640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.283 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.280+0000 7f037c2be640 1 -- 172.21.15.71:0/2048922841 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9b570874-7a3d-4867-8c44-29084a8bf649", "id": 1} v 0) -- 0x7f0374137700 con 0x7f03740a6810 2024-09-11T01:29:04.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.288+0000 7f036affd640 1 -- 172.21.15.71:0/2048922841 <== mon.1 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9b570874-7a3d-4867-8c44-29084a8bf649", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f0364012420 con 0x7f03740a6810 2024-09-11T01:29:04.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.288+0000 7f036affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9b570874-7a3d-4867-8c44-29084a8bf649", "id": 1}] 2024-09-11T01:29:04.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.288+0000 7f036affd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:04.290 INFO:teuthology.orchestra.run.smithi071.stdout:1 2024-09-11T01:29:04.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.289+0000 7f037c2be640 10 monclient: shutdown 2024-09-11T01:29:04.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.289+0000 7f037c2be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:04.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.289+0000 7f037c2be640 1 -- 172.21.15.71:0/2048922841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03740a6810 msgr2=0x7f0374120ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:04.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.289+0000 7f037c2be640 1 --2- 172.21.15.71:0/2048922841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03740a6810 0x7f0374120ce0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f036400c430 tx=0x7f036400c8f0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.289+0000 7f037c2be640 1 -- 172.21.15.71:0/2048922841 shutdown_connections 2024-09-11T01:29:04.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.289+0000 7f037c2be640 1 --2- 172.21.15.71:0/2048922841 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0374138210 0x7f0374121220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.289+0000 7f037c2be640 1 --2- 172.21.15.71:0/2048922841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03740a6810 0x7f0374120ce0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.289+0000 7f037c2be640 1 --2- 172.21.15.71:0/2048922841 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03740a57a0 0x7f0374136d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.289+0000 7f037c2be640 1 -- 172.21.15.71:0/2048922841 >> 172.21.15.71:0/2048922841 conn(0x7f03740aaa80 msgr2=0x7f03740a3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:04.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.289+0000 7f037c2be640 1 -- 172.21.15.71:0/2048922841 shutdown_connections 2024-09-11T01:29:04.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.289+0000 7f037c2be640 1 -- 172.21.15.71:0/2048922841 wait complete. 2024-09-11T01:29:04.301 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new b12ee288-ce57-41de-8b8e-6fad509311b6 2 2024-09-11T01:29:04.398 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.396+0000 7fc8239c6640 10 monclient: get_monmap_and_config 2024-09-11T01:29:04.399 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.396+0000 7fc8239c6640 10 monclient: build_initial_monmap 2024-09-11T01:29:04.399 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.396+0000 7fc8239c6640 10 monclient: monmap: 2024-09-11T01:29:04.399 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-09-11T01:29:04.399 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:04.399 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:29:04.397744+0000 2024-09-11T01:29:04.399 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:29:04.397744+0000 2024-09-11T01:29:04.399 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:04.399 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.399 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:04.399 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:04.399 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:04.399 INFO:teuthology.orchestra.run.smithi071.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:04.399 INFO:teuthology.orchestra.run.smithi071.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:04.399 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.400 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.397+0000 7fc8239c6640 1 Processor -- start 2024-09-11T01:29:04.402 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.397+0000 7fc8239c6640 1 -- start start 2024-09-11T01:29:04.402 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.397+0000 7fc8239c6640 10 monclient: init 2024-09-11T01:29:04.402 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.397+0000 7fc8239c6640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:04.402 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.397+0000 7fc8239c6640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-11T01:29:04.402 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.397+0000 7fc8239c6640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc81c138210 0x7fc81c1385f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:04.402 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.397+0000 7fc8239c6640 10 monclient(hunting): picked mon.noname-a con 0x7fc81c138210 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:04.402 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.397+0000 7fc8239c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc81c0a6810 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:04.402 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.397+0000 7fc8239c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc81c0a57a0 0x7fc81c140120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:04.402 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.397+0000 7fc8239c6640 10 monclient(hunting): picked mon.noname-b con 0x7fc81c0a57a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:04.403 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.397+0000 7fc8239c6640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc81c09ba20 con 0x7fc81c0a6810 2024-09-11T01:29:04.403 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.397+0000 7fc8239c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:04.403 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.397+0000 7fc8239c6640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc81c09c7a0 con 0x7fc81c138210 2024-09-11T01:29:04.403 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.397+0000 7fc8239c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:04.403 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.397+0000 7fc8239c6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc81c09c4c0 con 0x7fc81c0a57a0 2024-09-11T01:29:04.403 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.397+0000 7fc8239c6640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:04.403 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.397+0000 7fc8239c6640 10 monclient(hunting): authenticate will time out at 732.853956s 2024-09-11T01:29:04.403 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821f63640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc81c0a57a0 0x7fc81c140120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:04.403 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821762640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc81c138210 0x7fc81c1385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:04.403 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821762640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc81c138210 0x7fc81c1385f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.71:38582/0 (socket says 172.21.15.71:38582) 2024-09-11T01:29:04.403 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821f63640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc81c0a57a0 0x7fc81c140120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.71:54930/0 (socket says 172.21.15.71:54930) 2024-09-11T01:29:04.403 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821762640 1 -- 172.21.15.71:0/3657150666 learned_addr learned my addr 172.21.15.71:0/3657150666 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-09-11T01:29:04.403 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc820f61640 1 --1- 172.21.15.71:0/3657150666 >> v1:172.21.15.12:6789/0 conn(0x7fc81c0a6810 0x7fc81c0a5090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.71:37044/0 (socket says 172.21.15.71:37044) 2024-09-11T01:29:04.403 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821762640 10 monclient(hunting): get_auth_request con 0x7fc81c138210 auth_method 0 2024-09-11T01:29:04.404 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821762640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:04.404 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821762640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:04.404 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821762640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:04.404 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821f63640 10 monclient(hunting): get_auth_request con 0x7fc81c0a57a0 auth_method 0 2024-09-11T01:29:04.404 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821f63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:04.404 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821f63640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:04.404 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821f63640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:04.404 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821762640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:04.404 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821762640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:04.404 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821762640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:04.404 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821f63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:04.404 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821f63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:04.404 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821f63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:04.405 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821762640 10 monclient(hunting): handle_auth_done global_id 4119 payload 274 2024-09-11T01:29:04.405 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821762640 1 -- 172.21.15.71:0/3657150666 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc81c0a57a0 msgr2=0x7fc81c140120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:04.405 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821762640 1 --2- 172.21.15.71:0/3657150666 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc81c0a57a0 0x7fc81c140120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.405 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821762640 1 -- 172.21.15.71:0/3657150666 >> v1:172.21.15.12:6789/0 conn(0x7fc81c0a6810 legacy=0x7fc81c0a5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:04.405 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821762640 10 monclient: _finish_hunting 0 2024-09-11T01:29:04.405 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821762640 1 monclient: found mon.noname-a 2024-09-11T01:29:04.405 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821762640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:04.405 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821762640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:04.405 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.398+0000 7fc821762640 1 -- 172.21.15.71:0/3657150666 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc81c0a6590 con 0x7fc81c138210 2024-09-11T01:29:04.411 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.401+0000 7fc821762640 1 -- 172.21.15.71:0/3657150666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc81c138210 msgr2=0x7fc81c1385f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:04.411 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.401+0000 7fc821762640 1 --2- 172.21.15.71:0/3657150666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc81c138210 0x7fc81c1385f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc8180098a0 tx=0x7fc81802a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fd98d20e81d0da3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:04.411 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.401+0000 7fc813fff640 1 -- 172.21.15.71:0/3657150666 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc818037070 con 0x7fc81c138210 2024-09-11T01:29:04.411 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.401+0000 7fc813fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:04.411 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.401+0000 7fc813fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:04.411 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.401+0000 7fc813fff640 10 monclient: dump: 2024-09-11T01:29:04.411 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-09-11T01:29:04.411 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:04.411 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.411 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.411 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:04.412 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.412 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:04.412 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:04.412 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:04.412 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.412 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.401+0000 7fc813fff640 10 monclient: _finish_auth 0 2024-09-11T01:29:04.412 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.401+0000 7fc813fff640 10 monclient: _check_auth_tickets 2024-09-11T01:29:04.412 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.401+0000 7fc813fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:04.412 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.401+0000 7fc813fff640 1 -- 172.21.15.71:0/3657150666 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc818032440 con 0x7fc81c138210 2024-09-11T01:29:04.412 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.401+0000 7fc8239c6640 5 monclient: authenticate success, global_id 4119 2024-09-11T01:29:04.412 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.401+0000 7fc8239c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:04.412 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.401+0000 7fc813fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:04.412 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.401+0000 7fc8239c6640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:04.413 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.401+0000 7fc813fff640 1 -- 172.21.15.71:0/3657150666 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc818032810 con 0x7fc81c138210 2024-09-11T01:29:04.413 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.401+0000 7fc8239c6640 4 set_mon_vals no callback set 2024-09-11T01:29:04.413 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.401+0000 7fc813fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:04.413 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.401+0000 7fc813fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:04.413 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.401+0000 7fc813fff640 10 monclient: dump: 2024-09-11T01:29:04.413 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-09-11T01:29:04.413 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:04.413 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.413 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.413 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:04.413 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.413 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:04.413 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:04.414 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:04.414 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.414 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.402+0000 7fc8239c6640 10 monclient: shutdown 2024-09-11T01:29:04.414 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.402+0000 7fc8239c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:04.414 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.402+0000 7fc8239c6640 1 -- 172.21.15.71:0/3657150666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc81c138210 msgr2=0x7fc81c1385f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:04.414 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.402+0000 7fc8239c6640 1 --2- 172.21.15.71:0/3657150666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc81c138210 0x7fc81c1385f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc8180098a0 tx=0x7fc81802a0d0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.414 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.402+0000 7fc8239c6640 1 -- 172.21.15.71:0/3657150666 shutdown_connections 2024-09-11T01:29:04.414 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.402+0000 7fc8239c6640 1 --2- 172.21.15.71:0/3657150666 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc81c0a57a0 0x7fc81c140120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.414 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.402+0000 7fc8239c6640 1 --2- 172.21.15.71:0/3657150666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc81c138210 0x7fc81c1385f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.414 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.402+0000 7fc8239c6640 1 -- 172.21.15.71:0/3657150666 >> 172.21.15.71:0/3657150666 conn(0x7fc81c0aaa80 msgr2=0x7fc81c0aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:04.414 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.402+0000 7fc8239c6640 1 -- 172.21.15.71:0/3657150666 shutdown_connections 2024-09-11T01:29:04.414 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.402+0000 7fc8239c6640 1 -- 172.21.15.71:0/3657150666 wait complete. 2024-09-11T01:29:04.414 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.403+0000 7fc8239c6640 10 monclient: build_initial_monmap 2024-09-11T01:29:04.414 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.403+0000 7fc8239c6640 10 monclient: monmap: 2024-09-11T01:29:04.415 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-09-11T01:29:04.415 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:04.415 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 0.000000 2024-09-11T01:29:04.415 INFO:teuthology.orchestra.run.smithi071.stderr:created 0.000000 2024-09-11T01:29:04.415 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:04.415 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.415 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:04.415 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:04.415 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:04.415 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.415 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.403+0000 7fc8239c6640 1 Processor -- start 2024-09-11T01:29:04.415 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.403+0000 7fc8239c6640 1 -- start start 2024-09-11T01:29:04.415 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.403+0000 7fc8239c6640 10 monclient: init 2024-09-11T01:29:04.415 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.403+0000 7fc8239c6640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:04.416 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.403+0000 7fc8239c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:29:04.416 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.403+0000 7fc8239c6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc81c0a57a0 0x7fc81c136df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:04.416 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.403+0000 7fc8239c6640 10 monclient(hunting): picked mon.noname-b con 0x7fc81c0a57a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:04.416 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.403+0000 7fc8239c6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc81c0a6810 0x7fc81c137330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:04.416 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.403+0000 7fc8239c6640 10 monclient(hunting): picked mon.noname-a con 0x7fc81c0a6810 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:04.416 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.403+0000 7fc8239c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc81c138210 0x7fc81c120ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:04.416 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.403+0000 7fc8239c6640 10 monclient(hunting): picked mon.noname-c con 0x7fc81c138210 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:04.416 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.403+0000 7fc8239c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:04.416 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.403+0000 7fc8239c6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc81c06f670 con 0x7fc81c0a6810 2024-09-11T01:29:04.416 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.403+0000 7fc8239c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:04.416 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.403+0000 7fc8239c6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc81c06f4f0 con 0x7fc81c0a57a0 2024-09-11T01:29:04.416 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.403+0000 7fc8239c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:04.416 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.403+0000 7fc8239c6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc81c138b30 con 0x7fc81c138210 2024-09-11T01:29:04.416 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.403+0000 7fc8239c6640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:04.417 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.403+0000 7fc8239c6640 10 monclient(hunting): authenticate will time out at 732.859912s 2024-09-11T01:29:04.417 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.403+0000 7fc821762640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc81c0a57a0 0x7fc81c136df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:04.417 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821762640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc81c0a57a0 0x7fc81c136df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46472/0 (socket says 172.21.15.71:46472) 2024-09-11T01:29:04.417 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821762640 1 -- 172.21.15.71:0/4219011912 learned_addr learned my addr 172.21.15.71:0/4219011912 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-09-11T01:29:04.417 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821762640 10 monclient(hunting): get_auth_request con 0x7fc81c0a57a0 auth_method 0 2024-09-11T01:29:04.417 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821762640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:04.417 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821762640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:04.417 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821762640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:04.417 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc820f61640 1 --2- 172.21.15.71:0/4219011912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc81c0a6810 0x7fc81c137330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:04.417 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821f63640 1 --2- 172.21.15.71:0/4219011912 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc81c138210 0x7fc81c120ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:04.417 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821762640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:04.417 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821762640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:04.417 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821762640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:04.418 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc820f61640 10 monclient(hunting): get_auth_request con 0x7fc81c0a6810 auth_method 0 2024-09-11T01:29:04.418 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc820f61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:04.418 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc820f61640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:04.418 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc820f61640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:04.418 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821f63640 10 monclient(hunting): get_auth_request con 0x7fc81c138210 auth_method 0 2024-09-11T01:29:04.418 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821f63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:04.418 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821f63640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:04.418 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821f63640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:04.418 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc820f61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:04.418 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc820f61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:04.418 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc820f61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:04.418 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821f63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:04.418 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821f63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:04.418 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821f63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:04.419 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821762640 10 monclient(hunting): handle_auth_done global_id 4111 payload 824 2024-09-11T01:29:04.419 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821762640 1 -- 172.21.15.71:0/4219011912 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc81c138210 msgr2=0x7fc81c120ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:04.419 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821762640 1 --2- 172.21.15.71:0/4219011912 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc81c138210 0x7fc81c120ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.419 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821762640 1 -- 172.21.15.71:0/4219011912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc81c0a6810 msgr2=0x7fc81c137330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:04.419 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821762640 1 --2- 172.21.15.71:0/4219011912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc81c0a6810 0x7fc81c137330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.419 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821762640 10 monclient: _finish_hunting 0 2024-09-11T01:29:04.419 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821762640 1 monclient: found mon.noname-b 2024-09-11T01:29:04.419 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821762640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:04.419 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821762640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.419 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.404+0000 7fc821762640 1 -- 172.21.15.71:0/4219011912 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc81c0a1880 con 0x7fc81c0a57a0 2024-09-11T01:29:04.419 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.406+0000 7fc821762640 1 -- 172.21.15.71:0/4219011912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc81c0a57a0 msgr2=0x7fc81c136df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:04.419 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.406+0000 7fc821762640 1 --2- 172.21.15.71:0/4219011912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc81c0a57a0 0x7fc81c136df0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc8180097f0 tx=0x7fc8180046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:04.419 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.406+0000 7fc811ffb640 1 -- 172.21.15.71:0/4219011912 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc818041020 con 0x7fc81c0a57a0 2024-09-11T01:29:04.420 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.406+0000 7fc811ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:04.420 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.406+0000 7fc811ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:04.420 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.406+0000 7fc811ffb640 10 monclient: dump: 2024-09-11T01:29:04.420 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-09-11T01:29:04.420 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:04.420 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.420 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.420 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:04.420 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.420 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:04.420 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:04.420 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:04.420 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.421 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.406+0000 7fc811ffb640 10 monclient: _finish_auth 0 2024-09-11T01:29:04.421 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.406+0000 7fc811ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:29:04.421 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.406+0000 7fc811ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:04.421 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.406+0000 7fc811ffb640 1 -- 172.21.15.71:0/4219011912 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc81803d620 con 0x7fc81c0a57a0 2024-09-11T01:29:04.421 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.406+0000 7fc8239c6640 5 monclient: authenticate success, global_id 4111 2024-09-11T01:29:04.421 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.406+0000 7fc8239c6640 10 monclient: _renew_subs 2024-09-11T01:29:04.421 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.406+0000 7fc8239c6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.421 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.406+0000 7fc8239c6640 1 -- 172.21.15.71:0/4219011912 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc81c122450 con 0x7fc81c0a57a0 2024-09-11T01:29:04.421 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.406+0000 7fc811ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:04.421 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.406+0000 7fc811ffb640 1 -- 172.21.15.71:0/4219011912 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc818037040 con 0x7fc81c0a57a0 2024-09-11T01:29:04.421 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.406+0000 7fc811ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:04.421 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.406+0000 7fc811ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:04.421 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.406+0000 7fc811ffb640 10 monclient: dump: 2024-09-11T01:29:04.421 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-09-11T01:29:04.421 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:04.422 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.422 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.422 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:04.422 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.422 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:04.422 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:04.422 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:04.422 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.422 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.406+0000 7fc8239c6640 10 monclient: _renew_subs 2024-09-11T01:29:04.422 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.406+0000 7fc8239c6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.422 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.406+0000 7fc8239c6640 1 -- 172.21.15.71:0/4219011912 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc81c122fb0 con 0x7fc81c0a57a0 2024-09-11T01:29:04.422 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.406+0000 7fc8137fe640 4 set_mon_vals no callback set 2024-09-11T01:29:04.422 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.407+0000 7fc811ffb640 1 -- 172.21.15.71:0/4219011912 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7fc81803a030 con 0x7fc81c0a57a0 2024-09-11T01:29:04.422 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.407+0000 7fc811ffb640 1 -- 172.21.15.71:0/4219011912 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7fc81803b3f0 con 0x7fc81c0a57a0 2024-09-11T01:29:04.423 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.407+0000 7fc8239c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:04.423 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.407+0000 7fc8239c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:04.423 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.407+0000 7fc8239c6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.423 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.407+0000 7fc8239c6640 1 -- 172.21.15.71:0/4219011912 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc81c0a5bb0 con 0x7fc81c0a57a0 2024-09-11T01:29:04.423 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.409+0000 7fc811ffb640 1 -- 172.21.15.71:0/4219011912 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7fc81804b030 con 0x7fc81c0a57a0 2024-09-11T01:29:04.423 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.409+0000 7fc811ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:04.423 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.409+0000 7fc811ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:04.482 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.480+0000 7fc8239c6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b12ee288-ce57-41de-8b8e-6fad509311b6", "id": 2}] 2024-09-11T01:29:04.482 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.480+0000 7fc8239c6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b12ee288-ce57-41de-8b8e-6fad509311b6", "id": 2}] 2024-09-11T01:29:04.482 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.480+0000 7fc8239c6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.482 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.480+0000 7fc8239c6640 1 -- 172.21.15.71:0/4219011912 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b12ee288-ce57-41de-8b8e-6fad509311b6", "id": 2} v 0) -- 0x7fc81c074b50 con 0x7fc81c0a57a0 2024-09-11T01:29:04.490 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.488+0000 7fc811ffb640 1 -- 172.21.15.71:0/4219011912 <== mon.1 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b12ee288-ce57-41de-8b8e-6fad509311b6", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7fc81c09ba20 con 0x7fc81c0a57a0 2024-09-11T01:29:04.490 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.488+0000 7fc811ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b12ee288-ce57-41de-8b8e-6fad509311b6", "id": 2}] 2024-09-11T01:29:04.490 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.488+0000 7fc811ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:04.490 INFO:teuthology.orchestra.run.smithi071.stdout:2 2024-09-11T01:29:04.491 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.489+0000 7fc8239c6640 10 monclient: shutdown 2024-09-11T01:29:04.491 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.489+0000 7fc8239c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:04.492 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.489+0000 7fc8239c6640 1 -- 172.21.15.71:0/4219011912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc81c0a57a0 msgr2=0x7fc81c136df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:04.492 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.489+0000 7fc8239c6640 1 --2- 172.21.15.71:0/4219011912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc81c0a57a0 0x7fc81c136df0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc8180097f0 tx=0x7fc8180046c0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.492 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.489+0000 7fc8239c6640 1 -- 172.21.15.71:0/4219011912 shutdown_connections 2024-09-11T01:29:04.492 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.489+0000 7fc8239c6640 1 --2- 172.21.15.71:0/4219011912 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc81c138210 0x7fc81c120ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.492 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.489+0000 7fc8239c6640 1 --2- 172.21.15.71:0/4219011912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc81c0a6810 0x7fc81c137330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.492 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.489+0000 7fc8239c6640 1 --2- 172.21.15.71:0/4219011912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc81c0a57a0 0x7fc81c136df0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.492 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.489+0000 7fc8239c6640 1 -- 172.21.15.71:0/4219011912 >> 172.21.15.71:0/4219011912 conn(0x7fc81c0aaa80 msgr2=0x7fc81c1bfbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:04.492 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.489+0000 7fc8239c6640 1 -- 172.21.15.71:0/4219011912 shutdown_connections 2024-09-11T01:29:04.492 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.489+0000 7fc8239c6640 1 -- 172.21.15.71:0/4219011912 wait complete. 2024-09-11T01:29:04.501 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new a8e81aa4-0dee-487d-981d-1f593e363eee 3 2024-09-11T01:29:04.599 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.596+0000 7f25bf15d640 10 monclient: get_monmap_and_config 2024-09-11T01:29:04.599 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.596+0000 7f25bf15d640 10 monclient: build_initial_monmap 2024-09-11T01:29:04.599 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.596+0000 7f25bf15d640 10 monclient: monmap: 2024-09-11T01:29:04.599 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-09-11T01:29:04.599 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:04.599 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:29:04.597956+0000 2024-09-11T01:29:04.599 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:29:04.597956+0000 2024-09-11T01:29:04.599 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:04.599 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.599 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:04.599 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:04.599 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:04.600 INFO:teuthology.orchestra.run.smithi071.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:04.600 INFO:teuthology.orchestra.run.smithi071.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:04.600 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.602 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.597+0000 7f25bf15d640 1 Processor -- start 2024-09-11T01:29:04.602 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.597+0000 7f25bf15d640 1 -- start start 2024-09-11T01:29:04.602 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.597+0000 7f25bf15d640 10 monclient: init 2024-09-11T01:29:04.602 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25bf15d640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:04.602 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25bf15d640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-11T01:29:04.602 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25bf15d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f25b8138210 0x7f25b81385f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:04.602 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25bf15d640 10 monclient(hunting): picked mon.noname-a con 0x7f25b8138210 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:04.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25bf15d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f25b80a6810 0x7f25b80a5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:04.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25bf15d640 10 monclient(hunting): picked mon.noname-c con 0x7f25b80a6810 addr v2:172.21.15.71:3300/0 2024-09-11T01:29:04.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25bf15d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f25b80a55d0 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:04.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25bf15d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25b809ba20 con 0x7f25b80a55d0 2024-09-11T01:29:04.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25bf15d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:04.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25bf15d640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f25b809c4c0 con 0x7f25b8138210 2024-09-11T01:29:04.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25bf15d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:04.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25bf15d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f25b80a6590 con 0x7f25b80a6810 2024-09-11T01:29:04.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25bf15d640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:04.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25bf15d640 10 monclient(hunting): authenticate will time out at 733.054170s 2024-09-11T01:29:04.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25b7fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f25b80a6810 0x7f25b80a5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:04.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25bd6fa640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f25b80a55d0 0x7f25b813f110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:49104/0 (socket says 172.21.15.71:49104) 2024-09-11T01:29:04.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25b7fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f25b80a6810 0x7f25b80a5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46474/0 (socket says 172.21.15.71:46474) 2024-09-11T01:29:04.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25b7fff640 1 -- 172.21.15.71:0/1965823204 learned_addr learned my addr 172.21.15.71:0/1965823204 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-09-11T01:29:04.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25b7fff640 10 monclient(hunting): get_auth_request con 0x7f25b80a6810 auth_method 0 2024-09-11T01:29:04.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:04.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:04.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:04.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:04.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:04.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:04.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25bcef9640 1 --2- 172.21.15.71:0/1965823204 >> v2:172.21.15.12:3300/0 conn(0x7f25b8138210 0x7f25b81385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:04.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25bcef9640 10 monclient(hunting): get_auth_request con 0x7f25b8138210 auth_method 0 2024-09-11T01:29:04.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25bcef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:04.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25bcef9640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:04.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25bcef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:04.605 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25b7fff640 10 monclient(hunting): handle_auth_done global_id 4114 payload 274 2024-09-11T01:29:04.605 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25b7fff640 1 -- 172.21.15.71:0/1965823204 >> v2:172.21.15.12:3300/0 conn(0x7f25b8138210 msgr2=0x7f25b81385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:04.605 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25b7fff640 1 --2- 172.21.15.71:0/1965823204 >> v2:172.21.15.12:3300/0 conn(0x7f25b8138210 0x7f25b81385f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.605 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25b7fff640 1 -- 172.21.15.71:0/1965823204 >> v1:172.21.15.71:6789/0 conn(0x7f25b80a55d0 legacy=0x7f25b813f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:04.605 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25b7fff640 10 monclient: _finish_hunting 0 2024-09-11T01:29:04.605 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25b7fff640 1 monclient: found mon.noname-c 2024-09-11T01:29:04.605 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:04.605 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25b7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.605 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.598+0000 7f25b7fff640 1 -- 172.21.15.71:0/1965823204 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b809c1e0 con 0x7f25b80a6810 2024-09-11T01:29:04.611 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.601+0000 7f25b7fff640 1 -- 172.21.15.71:0/1965823204 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25b80a6810 msgr2=0x7f25b80a5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:04.611 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.601+0000 7f25b7fff640 1 --2- 172.21.15.71:0/1965823204 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25b80a6810 0x7f25b80a5090 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f25b8074ee0 tx=0x7f25a802a0b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3eab01249f050747 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:04.611 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.601+0000 7f25b77fe640 1 -- 172.21.15.71:0/1965823204 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a8037070 con 0x7f25b80a6810 2024-09-11T01:29:04.611 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.601+0000 7f25b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:04.611 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.601+0000 7f25b77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:04.611 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.601+0000 7f25b77fe640 10 monclient: dump: 2024-09-11T01:29:04.611 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-09-11T01:29:04.611 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:04.611 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.611 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.611 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:04.611 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.612 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:04.612 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:04.612 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:04.612 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.612 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.601+0000 7f25b77fe640 10 monclient: _finish_auth 0 2024-09-11T01:29:04.612 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.601+0000 7f25b77fe640 10 monclient: _check_auth_tickets 2024-09-11T01:29:04.612 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.601+0000 7f25b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:04.612 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.601+0000 7f25b77fe640 1 -- 172.21.15.71:0/1965823204 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25a8032640 con 0x7f25b80a6810 2024-09-11T01:29:04.612 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.601+0000 7f25bf15d640 5 monclient: authenticate success, global_id 4114 2024-09-11T01:29:04.613 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.601+0000 7f25bf15d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:04.613 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.601+0000 7f25b77fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:04.613 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.601+0000 7f25bf15d640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:04.613 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.601+0000 7f25b77fe640 1 -- 172.21.15.71:0/1965823204 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a8032a10 con 0x7f25b80a6810 2024-09-11T01:29:04.613 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.601+0000 7f25bf15d640 4 set_mon_vals no callback set 2024-09-11T01:29:04.613 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.601+0000 7f25b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:04.613 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.601+0000 7f25b77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:04.613 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.601+0000 7f25b77fe640 10 monclient: dump: 2024-09-11T01:29:04.613 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-09-11T01:29:04.613 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:04.613 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.613 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.614 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:04.614 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.614 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:04.614 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:04.614 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:04.614 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.614 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.602+0000 7f25bf15d640 10 monclient: shutdown 2024-09-11T01:29:04.614 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.602+0000 7f25bf15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:04.614 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.602+0000 7f25bf15d640 1 -- 172.21.15.71:0/1965823204 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25b80a6810 msgr2=0x7f25b80a5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:04.614 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.602+0000 7f25bf15d640 1 --2- 172.21.15.71:0/1965823204 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25b80a6810 0x7f25b80a5090 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f25b8074ee0 tx=0x7f25a802a0b0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.614 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.602+0000 7f25bf15d640 1 -- 172.21.15.71:0/1965823204 shutdown_connections 2024-09-11T01:29:04.614 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.602+0000 7f25bf15d640 1 --2- 172.21.15.71:0/1965823204 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25b80a6810 0x7f25b80a5090 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.614 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.602+0000 7f25bf15d640 1 --2- 172.21.15.71:0/1965823204 >> v2:172.21.15.12:3300/0 conn(0x7f25b8138210 0x7f25b81385f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.614 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.602+0000 7f25bf15d640 1 -- 172.21.15.71:0/1965823204 >> 172.21.15.71:0/1965823204 conn(0x7f25b80aaa80 msgr2=0x7f25b80aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:04.615 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.602+0000 7f25bf15d640 1 -- 172.21.15.71:0/1965823204 shutdown_connections 2024-09-11T01:29:04.615 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.602+0000 7f25bf15d640 1 -- 172.21.15.71:0/1965823204 wait complete. 2024-09-11T01:29:04.615 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.602+0000 7f25bf15d640 10 monclient: build_initial_monmap 2024-09-11T01:29:04.615 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.602+0000 7f25bf15d640 10 monclient: monmap: 2024-09-11T01:29:04.615 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-09-11T01:29:04.615 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:04.615 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 0.000000 2024-09-11T01:29:04.615 INFO:teuthology.orchestra.run.smithi071.stderr:created 0.000000 2024-09-11T01:29:04.615 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:04.615 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.615 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:04.615 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:04.615 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:04.615 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.616 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.603+0000 7f25bf15d640 1 Processor -- start 2024-09-11T01:29:04.616 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.603+0000 7f25bf15d640 1 -- start start 2024-09-11T01:29:04.616 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.603+0000 7f25bf15d640 10 monclient: init 2024-09-11T01:29:04.616 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.603+0000 7f25bf15d640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:04.616 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.603+0000 7f25bf15d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:29:04.616 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.603+0000 7f25bf15d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25b80a55d0 0x7f25b8137190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:04.616 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.603+0000 7f25bf15d640 10 monclient(hunting): picked mon.noname-b con 0x7f25b80a55d0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:04.616 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.603+0000 7f25bf15d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f25b80a6810 0x7f25b81376d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:04.616 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.603+0000 7f25bf15d640 10 monclient(hunting): picked mon.noname-c con 0x7f25b80a6810 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:04.616 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.603+0000 7f25bf15d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25b8138210 0x7f25b8124e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:04.616 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.603+0000 7f25bf15d640 10 monclient(hunting): picked mon.noname-a con 0x7f25b8138210 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:04.616 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.603+0000 7f25bf15d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:04.616 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.603+0000 7f25bf15d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f25b8138b30 con 0x7f25b8138210 2024-09-11T01:29:04.617 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.603+0000 7f25bf15d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:04.617 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.603+0000 7f25bf15d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f25b809d6e0 con 0x7f25b80a55d0 2024-09-11T01:29:04.617 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.603+0000 7f25bf15d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:04.617 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.603+0000 7f25bf15d640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f25b8138cb0 con 0x7f25b80a6810 2024-09-11T01:29:04.617 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.603+0000 7f25bf15d640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:04.617 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.603+0000 7f25bf15d640 10 monclient(hunting): authenticate will time out at 733.059849s 2024-09-11T01:29:04.617 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.603+0000 7f25bcef9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25b80a55d0 0x7f25b8137190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:04.617 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.603+0000 7f25bcef9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25b80a55d0 0x7f25b8137190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46480/0 (socket says 172.21.15.71:46480) 2024-09-11T01:29:04.617 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.603+0000 7f25bcef9640 1 -- 172.21.15.71:0/1939737374 learned_addr learned my addr 172.21.15.71:0/1939737374 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-09-11T01:29:04.617 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.603+0000 7f25bcef9640 10 monclient(hunting): get_auth_request con 0x7f25b80a55d0 auth_method 0 2024-09-11T01:29:04.617 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.603+0000 7f25bcef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:04.617 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.603+0000 7f25bcef9640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:04.617 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.603+0000 7f25bcef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:04.618 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25b7fff640 1 --2- 172.21.15.71:0/1939737374 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f25b80a6810 0x7f25b81376d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:04.618 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25bd6fa640 1 --2- 172.21.15.71:0/1939737374 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25b8138210 0x7f25b8124e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:04.618 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25bcef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:04.618 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25bcef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:04.618 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25bcef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:04.618 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25bd6fa640 10 monclient(hunting): get_auth_request con 0x7f25b8138210 auth_method 0 2024-09-11T01:29:04.618 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25bd6fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:04.618 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25bd6fa640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:04.618 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25bd6fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:04.618 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25b7fff640 10 monclient(hunting): get_auth_request con 0x7f25b80a6810 auth_method 0 2024-09-11T01:29:04.618 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:04.618 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:04.618 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:04.618 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25bd6fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:04.619 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25bd6fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:04.619 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25bd6fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:04.619 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:04.619 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:04.619 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:04.619 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25bcef9640 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2024-09-11T01:29:04.619 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25bcef9640 1 -- 172.21.15.71:0/1939737374 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f25b80a6810 msgr2=0x7f25b81376d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:04.619 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25bcef9640 1 --2- 172.21.15.71:0/1939737374 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f25b80a6810 0x7f25b81376d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.619 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25bcef9640 1 -- 172.21.15.71:0/1939737374 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25b8138210 msgr2=0x7f25b8124e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:04.619 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25bcef9640 1 --2- 172.21.15.71:0/1939737374 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25b8138210 0x7f25b8124e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.619 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25bcef9640 10 monclient: _finish_hunting 0 2024-09-11T01:29:04.619 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25bcef9640 1 monclient: found mon.noname-b 2024-09-11T01:29:04.619 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25bcef9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:04.619 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25bcef9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.620 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.604+0000 7f25bcef9640 1 -- 172.21.15.71:0/1939737374 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b8128c90 con 0x7f25b80a55d0 2024-09-11T01:29:04.620 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.606+0000 7f25bcef9640 1 -- 172.21.15.71:0/1939737374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25b80a55d0 msgr2=0x7f25b8137190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:04.620 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.606+0000 7f25bcef9640 1 --2- 172.21.15.71:0/1939737374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25b80a55d0 0x7f25b8137190 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f25ac00d830 tx=0x7f25ac00dcf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:04.620 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.606+0000 7f25b57fa640 1 -- 172.21.15.71:0/1939737374 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ac00b8d0 con 0x7f25b80a55d0 2024-09-11T01:29:04.620 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.606+0000 7f25b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:04.620 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.606+0000 7f25b57fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:04.620 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.606+0000 7f25b57fa640 10 monclient: dump: 2024-09-11T01:29:04.620 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-09-11T01:29:04.620 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:04.620 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.620 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.620 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:04.620 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.620 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:04.621 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:04.621 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:04.621 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.621 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.606+0000 7f25b57fa640 10 monclient: _finish_auth 0 2024-09-11T01:29:04.621 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.606+0000 7f25b57fa640 10 monclient: _check_auth_tickets 2024-09-11T01:29:04.621 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.606+0000 7f25b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:04.621 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.606+0000 7f25b57fa640 1 -- 172.21.15.71:0/1939737374 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25ac019070 con 0x7f25b80a55d0 2024-09-11T01:29:04.621 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.606+0000 7f25b57fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:04.621 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.606+0000 7f25b57fa640 1 -- 172.21.15.71:0/1939737374 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ac00bc90 con 0x7f25b80a55d0 2024-09-11T01:29:04.621 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.606+0000 7f25b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:04.621 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.606+0000 7f25b57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:04.621 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.606+0000 7f25b57fa640 10 monclient: dump: 2024-09-11T01:29:04.621 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-09-11T01:29:04.621 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:04.622 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.622 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.622 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:04.622 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.622 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:04.622 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:04.622 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:04.622 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.622 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.606+0000 7f25bf15d640 5 monclient: authenticate success, global_id 4117 2024-09-11T01:29:04.622 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.606+0000 7f25bf15d640 10 monclient: _renew_subs 2024-09-11T01:29:04.622 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.606+0000 7f25bf15d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.622 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.606+0000 7f25bf15d640 1 -- 172.21.15.71:0/1939737374 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25b8137c90 con 0x7f25b80a55d0 2024-09-11T01:29:04.622 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.606+0000 7f25b6ffd640 4 set_mon_vals no callback set 2024-09-11T01:29:04.622 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.606+0000 7f25bf15d640 10 monclient: _renew_subs 2024-09-11T01:29:04.623 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.606+0000 7f25bf15d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.623 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.606+0000 7f25bf15d640 1 -- 172.21.15.71:0/1939737374 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25b80a1880 con 0x7f25b80a55d0 2024-09-11T01:29:04.623 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.606+0000 7f25b57fa640 1 -- 172.21.15.71:0/1939737374 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f25ac013070 con 0x7f25b80a55d0 2024-09-11T01:29:04.623 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.607+0000 7f25b57fa640 1 -- 172.21.15.71:0/1939737374 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f25ac01e8f0 con 0x7f25b80a55d0 2024-09-11T01:29:04.623 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.607+0000 7f25bf15d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:04.623 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.607+0000 7f25bf15d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:04.623 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.607+0000 7f25bf15d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.623 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.607+0000 7f25bf15d640 1 -- 172.21.15.71:0/1939737374 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25b80a1240 con 0x7f25b80a55d0 2024-09-11T01:29:04.623 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.609+0000 7f25b57fa640 1 -- 172.21.15.71:0/1939737374 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f25b80a1240 con 0x7f25b80a55d0 2024-09-11T01:29:04.624 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.609+0000 7f25b57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:04.624 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.609+0000 7f25b57fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:04.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.679+0000 7f25bf15d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a8e81aa4-0dee-487d-981d-1f593e363eee", "id": 3}] 2024-09-11T01:29:04.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.679+0000 7f25bf15d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a8e81aa4-0dee-487d-981d-1f593e363eee", "id": 3}] 2024-09-11T01:29:04.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.679+0000 7f25bf15d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.679+0000 7f25bf15d640 1 -- 172.21.15.71:0/1939737374 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a8e81aa4-0dee-487d-981d-1f593e363eee", "id": 3} v 0) -- 0x7f25b80a5090 con 0x7f25b80a55d0 2024-09-11T01:29:04.687 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.685+0000 7f25b57fa640 1 -- 172.21.15.71:0/1939737374 <== mon.1 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a8e81aa4-0dee-487d-981d-1f593e363eee", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f25ac01ebc0 con 0x7f25b80a55d0 2024-09-11T01:29:04.687 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.685+0000 7f25b57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a8e81aa4-0dee-487d-981d-1f593e363eee", "id": 3}] 2024-09-11T01:29:04.687 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.685+0000 7f25b57fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:04.687 INFO:teuthology.orchestra.run.smithi071.stdout:3 2024-09-11T01:29:04.688 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.686+0000 7f25bf15d640 10 monclient: shutdown 2024-09-11T01:29:04.688 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.686+0000 7f25bf15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:04.688 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.686+0000 7f25bf15d640 1 -- 172.21.15.71:0/1939737374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25b80a55d0 msgr2=0x7f25b8137190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:04.688 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.686+0000 7f25bf15d640 1 --2- 172.21.15.71:0/1939737374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25b80a55d0 0x7f25b8137190 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f25ac00d830 tx=0x7f25ac00dcf0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.688 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.686+0000 7f25bf15d640 1 -- 172.21.15.71:0/1939737374 shutdown_connections 2024-09-11T01:29:04.688 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.686+0000 7f25bf15d640 1 --2- 172.21.15.71:0/1939737374 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25b8138210 0x7f25b8124e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.689 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.686+0000 7f25bf15d640 1 --2- 172.21.15.71:0/1939737374 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f25b80a6810 0x7f25b81376d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.689 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.686+0000 7f25bf15d640 1 --2- 172.21.15.71:0/1939737374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25b80a55d0 0x7f25b8137190 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.689 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.686+0000 7f25bf15d640 1 -- 172.21.15.71:0/1939737374 >> 172.21.15.71:0/1939737374 conn(0x7f25b80aaa80 msgr2=0x7f25b809fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:04.689 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.686+0000 7f25bf15d640 1 -- 172.21.15.71:0/1939737374 shutdown_connections 2024-09-11T01:29:04.689 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.686+0000 7f25bf15d640 1 -- 172.21.15.71:0/1939737374 wait complete. 2024-09-11T01:29:04.698 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new 9db77181-fc18-408f-a35c-87bf4b4ba2d6 4 2024-09-11T01:29:04.795 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.793+0000 7efc1c782640 10 monclient: get_monmap_and_config 2024-09-11T01:29:04.796 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.793+0000 7efc1c782640 10 monclient: build_initial_monmap 2024-09-11T01:29:04.796 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.793+0000 7efc1c782640 10 monclient: monmap: 2024-09-11T01:29:04.796 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-09-11T01:29:04.796 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:04.796 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:29:04.794528+0000 2024-09-11T01:29:04.796 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:29:04.794528+0000 2024-09-11T01:29:04.796 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:04.796 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.796 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:04.796 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:04.796 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:04.796 INFO:teuthology.orchestra.run.smithi071.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:04.796 INFO:teuthology.orchestra.run.smithi071.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:04.797 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.797 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.793+0000 7efc1c782640 1 Processor -- start 2024-09-11T01:29:04.797 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.794+0000 7efc1c782640 1 -- start start 2024-09-11T01:29:04.797 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.794+0000 7efc1c782640 10 monclient: init 2024-09-11T01:29:04.799 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.794+0000 7efc1c782640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:04.799 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.794+0000 7efc1c782640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-11T01:29:04.799 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.794+0000 7efc1c782640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc14138210 0x7efc141385f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:04.799 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.794+0000 7efc1c782640 10 monclient(hunting): picked mon.noname-b con 0x7efc14138210 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:04.799 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.794+0000 7efc1c782640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efc140a6810 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:04.799 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.794+0000 7efc1c782640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7efc140a57a0 0x7efc14140120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:04.800 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.794+0000 7efc1c782640 10 monclient(hunting): picked mon.noname-c con 0x7efc140a57a0 addr v2:172.21.15.71:3300/0 2024-09-11T01:29:04.800 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.794+0000 7efc1c782640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc1409ba20 con 0x7efc140a6810 2024-09-11T01:29:04.800 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.794+0000 7efc1c782640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:04.800 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.794+0000 7efc1c782640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7efc1409c7a0 con 0x7efc140a57a0 2024-09-11T01:29:04.800 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.794+0000 7efc1c782640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:04.800 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.794+0000 7efc1c782640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7efc1409c4c0 con 0x7efc14138210 2024-09-11T01:29:04.800 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.794+0000 7efc1c782640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:04.800 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.794+0000 7efc1c782640 10 monclient(hunting): authenticate will time out at 733.250748s 2024-09-11T01:29:04.800 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.794+0000 7efc1ad1f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7efc140a57a0 0x7efc14140120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:04.800 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.794+0000 7efc1ad1f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7efc140a57a0 0x7efc14140120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46484/0 (socket says 172.21.15.71:46484) 2024-09-11T01:29:04.800 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.794+0000 7efc1ad1f640 1 -- 172.21.15.71:0/2133773625 learned_addr learned my addr 172.21.15.71:0/2133773625 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-09-11T01:29:04.800 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.794+0000 7efc1ad1f640 10 monclient(hunting): get_auth_request con 0x7efc140a57a0 auth_method 0 2024-09-11T01:29:04.800 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.794+0000 7efc1ad1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:04.801 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.794+0000 7efc1ad1f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:04.801 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.794+0000 7efc1ad1f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:04.801 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.795+0000 7efc1ad1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:04.801 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.795+0000 7efc1ad1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:04.801 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.795+0000 7efc1ad1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:04.801 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.795+0000 7efc1a51e640 1 --2- 172.21.15.71:0/2133773625 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc14138210 0x7efc141385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:04.801 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.795+0000 7efc1a51e640 10 monclient(hunting): get_auth_request con 0x7efc14138210 auth_method 0 2024-09-11T01:29:04.801 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.795+0000 7efc1a51e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:04.801 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.795+0000 7efc1a51e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:04.801 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.795+0000 7efc1a51e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:04.801 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.795+0000 7efc1ad1f640 10 monclient(hunting): handle_auth_done global_id 4120 payload 274 2024-09-11T01:29:04.801 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.795+0000 7efc1ad1f640 1 -- 172.21.15.71:0/2133773625 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc14138210 msgr2=0x7efc141385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:04.801 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.795+0000 7efc1ad1f640 1 --2- 172.21.15.71:0/2133773625 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc14138210 0x7efc141385f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.802 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.795+0000 7efc1ad1f640 1 -- 172.21.15.71:0/2133773625 >> v1:172.21.15.12:6789/0 conn(0x7efc140a6810 legacy=0x7efc140a5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:04.802 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.795+0000 7efc1ad1f640 10 monclient: _finish_hunting 0 2024-09-11T01:29:04.802 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.795+0000 7efc1ad1f640 1 monclient: found mon.noname-c 2024-09-11T01:29:04.802 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.795+0000 7efc1ad1f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:04.802 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.795+0000 7efc1ad1f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.802 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.795+0000 7efc1ad1f640 1 -- 172.21.15.71:0/2133773625 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc140a6590 con 0x7efc140a57a0 2024-09-11T01:29:04.802 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.795+0000 7efc1a51e640 1 --2- 172.21.15.71:0/2133773625 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc14138210 0x7efc141385f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:29:04.802 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.797+0000 7efc1ad1f640 1 -- 172.21.15.71:0/2133773625 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc140a57a0 msgr2=0x7efc14140120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:04.802 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.797+0000 7efc1ad1f640 1 --2- 172.21.15.71:0/2133773625 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc140a57a0 0x7efc14140120 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7efc0c009e50 tx=0x7efc0c02a560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=904d8ab6e4681d76 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:04.808 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.798+0000 7efc1951c640 1 -- 172.21.15.71:0/2133773625 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc0c036070 con 0x7efc140a57a0 2024-09-11T01:29:04.808 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.798+0000 7efc1951c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:04.808 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.798+0000 7efc1951c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:04.808 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.798+0000 7efc1951c640 10 monclient: dump: 2024-09-11T01:29:04.808 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-09-11T01:29:04.808 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:04.808 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.808 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.809 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:04.809 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.809 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:04.809 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:04.809 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:04.809 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.809 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.798+0000 7efc1951c640 10 monclient: _finish_auth 0 2024-09-11T01:29:04.809 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.798+0000 7efc1951c640 10 monclient: _check_auth_tickets 2024-09-11T01:29:04.809 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.798+0000 7efc1951c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:04.809 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.798+0000 7efc1c782640 5 monclient: authenticate success, global_id 4120 2024-09-11T01:29:04.809 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.798+0000 7efc1951c640 1 -- 172.21.15.71:0/2133773625 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc0c007800 con 0x7efc140a57a0 2024-09-11T01:29:04.809 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.798+0000 7efc1951c640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:04.810 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.798+0000 7efc1c782640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:04.810 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.798+0000 7efc1951c640 1 -- 172.21.15.71:0/2133773625 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc0c007bd0 con 0x7efc140a57a0 2024-09-11T01:29:04.810 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.798+0000 7efc1c782640 4 set_mon_vals no callback set 2024-09-11T01:29:04.810 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.798+0000 7efc1951c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:04.810 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.798+0000 7efc1951c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:04.810 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.798+0000 7efc1951c640 10 monclient: dump: 2024-09-11T01:29:04.810 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-09-11T01:29:04.810 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:04.810 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.810 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.810 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:04.810 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.810 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:04.810 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:04.811 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:04.811 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.811 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.798+0000 7efc1c782640 10 monclient: shutdown 2024-09-11T01:29:04.811 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.798+0000 7efc1c782640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:04.811 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.798+0000 7efc1c782640 1 -- 172.21.15.71:0/2133773625 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc140a57a0 msgr2=0x7efc14140120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:04.811 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.798+0000 7efc1c782640 1 --2- 172.21.15.71:0/2133773625 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc140a57a0 0x7efc14140120 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7efc0c009e50 tx=0x7efc0c02a560 comp rx=0 tx=0).stop 2024-09-11T01:29:04.811 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.798+0000 7efc1c782640 1 -- 172.21.15.71:0/2133773625 shutdown_connections 2024-09-11T01:29:04.811 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.798+0000 7efc1c782640 1 --2- 172.21.15.71:0/2133773625 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc140a57a0 0x7efc14140120 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.811 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.798+0000 7efc1c782640 1 --2- 172.21.15.71:0/2133773625 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc14138210 0x7efc141385f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.811 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.798+0000 7efc1c782640 1 -- 172.21.15.71:0/2133773625 >> 172.21.15.71:0/2133773625 conn(0x7efc140aaa80 msgr2=0x7efc140aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:04.811 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.798+0000 7efc1c782640 1 -- 172.21.15.71:0/2133773625 shutdown_connections 2024-09-11T01:29:04.812 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.798+0000 7efc1c782640 1 -- 172.21.15.71:0/2133773625 wait complete. 2024-09-11T01:29:04.812 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.799+0000 7efc1c782640 10 monclient: build_initial_monmap 2024-09-11T01:29:04.812 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.799+0000 7efc1c782640 10 monclient: monmap: 2024-09-11T01:29:04.812 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-09-11T01:29:04.812 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:04.812 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 0.000000 2024-09-11T01:29:04.812 INFO:teuthology.orchestra.run.smithi071.stderr:created 0.000000 2024-09-11T01:29:04.812 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:04.812 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.812 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:04.812 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:04.812 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:04.812 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.813 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.799+0000 7efc1c782640 1 Processor -- start 2024-09-11T01:29:04.813 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.799+0000 7efc1c782640 1 -- start start 2024-09-11T01:29:04.813 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.799+0000 7efc1c782640 10 monclient: init 2024-09-11T01:29:04.813 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc1c782640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:04.813 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc1c782640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:29:04.813 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc1c782640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc140a57a0 0x7efc141ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:04.813 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc1c782640 10 monclient(hunting): picked mon.noname-a con 0x7efc140a57a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:04.813 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc1c782640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc140a6810 0x7efc14136f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:04.813 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc1c782640 10 monclient(hunting): picked mon.noname-b con 0x7efc140a6810 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:04.813 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc1c782640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc14138210 0x7efc14120ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:04.813 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc1c782640 10 monclient(hunting): picked mon.noname-c con 0x7efc14138210 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:04.813 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc1c782640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:04.813 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc1c782640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7efc1406f670 con 0x7efc140a57a0 2024-09-11T01:29:04.813 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc1c782640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:04.814 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc1c782640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7efc1406f4f0 con 0x7efc140a6810 2024-09-11T01:29:04.814 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc1c782640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:04.814 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc1c782640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7efc14138b30 con 0x7efc14138210 2024-09-11T01:29:04.814 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc1c782640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:04.814 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc1c782640 10 monclient(hunting): authenticate will time out at 733.256252s 2024-09-11T01:29:04.814 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc19d1d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc140a6810 0x7efc14136f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:04.814 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc19d1d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc140a6810 0x7efc14136f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46494/0 (socket says 172.21.15.71:46494) 2024-09-11T01:29:04.814 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc19d1d640 1 -- 172.21.15.71:0/4076306993 learned_addr learned my addr 172.21.15.71:0/4076306993 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-09-11T01:29:04.814 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc19d1d640 10 monclient(hunting): get_auth_request con 0x7efc140a6810 auth_method 0 2024-09-11T01:29:04.814 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc19d1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:04.814 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc19d1d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:04.814 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc19d1d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:04.814 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc19d1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:04.815 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc19d1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:04.815 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc19d1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:04.815 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc1a51e640 1 --2- 172.21.15.71:0/4076306993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc140a57a0 0x7efc141ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:04.815 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc1ad1f640 1 --2- 172.21.15.71:0/4076306993 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc14138210 0x7efc14120ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:04.815 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc1a51e640 10 monclient(hunting): get_auth_request con 0x7efc140a57a0 auth_method 0 2024-09-11T01:29:04.815 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc1a51e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:04.815 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc1a51e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:04.815 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc1a51e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:04.815 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc1ad1f640 10 monclient(hunting): get_auth_request con 0x7efc14138210 auth_method 0 2024-09-11T01:29:04.815 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc1ad1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:04.815 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc1ad1f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:04.815 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc1ad1f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:04.815 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc19d1d640 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2024-09-11T01:29:04.815 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc19d1d640 1 -- 172.21.15.71:0/4076306993 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc14138210 msgr2=0x7efc14120ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:04.816 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc19d1d640 1 --2- 172.21.15.71:0/4076306993 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc14138210 0x7efc14120ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.816 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc19d1d640 1 -- 172.21.15.71:0/4076306993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc140a57a0 msgr2=0x7efc141ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:04.816 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc19d1d640 1 --2- 172.21.15.71:0/4076306993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc140a57a0 0x7efc141ccfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.816 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc19d1d640 10 monclient: _finish_hunting 0 2024-09-11T01:29:04.816 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc19d1d640 1 monclient: found mon.noname-b 2024-09-11T01:29:04.816 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc19d1d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:04.816 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc19d1d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.816 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.800+0000 7efc19d1d640 1 -- 172.21.15.71:0/4076306993 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc14124670 con 0x7efc140a6810 2024-09-11T01:29:04.816 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.802+0000 7efc19d1d640 1 -- 172.21.15.71:0/4076306993 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc140a6810 msgr2=0x7efc14136f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:04.816 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.802+0000 7efc19d1d640 1 --2- 172.21.15.71:0/4076306993 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc140a6810 0x7efc14136f70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7efc0400c430 tx=0x7efc0400c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:04.816 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.803+0000 7efc0affd640 1 -- 172.21.15.71:0/4076306993 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc040119a0 con 0x7efc140a6810 2024-09-11T01:29:04.816 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.803+0000 7efc0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:04.817 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.803+0000 7efc0affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:04.817 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.803+0000 7efc0affd640 10 monclient: dump: 2024-09-11T01:29:04.817 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-09-11T01:29:04.817 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:04.817 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.817 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.817 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:04.817 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.817 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:04.817 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:04.817 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:04.817 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.817 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.803+0000 7efc0affd640 10 monclient: _finish_auth 0 2024-09-11T01:29:04.817 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.803+0000 7efc0affd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:04.818 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.803+0000 7efc0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:04.818 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.803+0000 7efc0affd640 1 -- 172.21.15.71:0/4076306993 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc04017070 con 0x7efc140a6810 2024-09-11T01:29:04.818 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.803+0000 7efc1c782640 5 monclient: authenticate success, global_id 4123 2024-09-11T01:29:04.818 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.803+0000 7efc1c782640 10 monclient: _renew_subs 2024-09-11T01:29:04.818 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.803+0000 7efc1c782640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.818 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.803+0000 7efc1c782640 1 -- 172.21.15.71:0/4076306993 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc14124950 con 0x7efc140a6810 2024-09-11T01:29:04.818 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.803+0000 7efc0affd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:04.818 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.803+0000 7efc0affd640 1 -- 172.21.15.71:0/4076306993 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc04011dc0 con 0x7efc140a6810 2024-09-11T01:29:04.818 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.803+0000 7efc0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:04.818 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.803+0000 7efc0affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:04.818 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.803+0000 7efc0affd640 10 monclient: dump: 2024-09-11T01:29:04.818 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-09-11T01:29:04.818 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:04.819 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.819 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:04.819 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:04.819 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.819 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:04.819 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:04.819 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:04.819 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.819 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.803+0000 7efc18d1b640 4 set_mon_vals no callback set 2024-09-11T01:29:04.819 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.803+0000 7efc1c782640 10 monclient: _renew_subs 2024-09-11T01:29:04.819 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.803+0000 7efc1c782640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.819 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.803+0000 7efc1c782640 1 -- 172.21.15.71:0/4076306993 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc14124390 con 0x7efc140a6810 2024-09-11T01:29:04.819 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.803+0000 7efc0affd640 1 -- 172.21.15.71:0/4076306993 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7efc0401c460 con 0x7efc140a6810 2024-09-11T01:29:04.819 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.803+0000 7efc0affd640 1 -- 172.21.15.71:0/4076306993 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7efc0401b3c0 con 0x7efc140a6810 2024-09-11T01:29:04.820 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.804+0000 7efc1c782640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:04.820 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.804+0000 7efc1c782640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:04.820 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.804+0000 7efc1c782640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.820 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.804+0000 7efc1c782640 1 -- 172.21.15.71:0/4076306993 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc140ab210 con 0x7efc140a6810 2024-09-11T01:29:04.820 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.806+0000 7efc0affd640 1 -- 172.21.15.71:0/4076306993 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7efc140ab210 con 0x7efc140a6810 2024-09-11T01:29:04.820 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.806+0000 7efc0affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:04.820 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.806+0000 7efc0affd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:04.878 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.876+0000 7efc1c782640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9db77181-fc18-408f-a35c-87bf4b4ba2d6", "id": 4}] 2024-09-11T01:29:04.878 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.876+0000 7efc1c782640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9db77181-fc18-408f-a35c-87bf4b4ba2d6", "id": 4}] 2024-09-11T01:29:04.878 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.876+0000 7efc1c782640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:04.878 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.876+0000 7efc1c782640 1 -- 172.21.15.71:0/4076306993 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9db77181-fc18-408f-a35c-87bf4b4ba2d6", "id": 4} v 0) -- 0x7efc140a5bd0 con 0x7efc140a6810 2024-09-11T01:29:04.885 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.883+0000 7efc0affd640 1 -- 172.21.15.71:0/4076306993 <== mon.1 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9db77181-fc18-408f-a35c-87bf4b4ba2d6", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7efc04012440 con 0x7efc140a6810 2024-09-11T01:29:04.885 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.883+0000 7efc0affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9db77181-fc18-408f-a35c-87bf4b4ba2d6", "id": 4}] 2024-09-11T01:29:04.885 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.883+0000 7efc0affd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:04.885 INFO:teuthology.orchestra.run.smithi071.stdout:4 2024-09-11T01:29:04.886 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.884+0000 7efc1c782640 10 monclient: shutdown 2024-09-11T01:29:04.886 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.884+0000 7efc1c782640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:04.887 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.884+0000 7efc1c782640 1 -- 172.21.15.71:0/4076306993 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc140a6810 msgr2=0x7efc14136f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:04.887 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.884+0000 7efc1c782640 1 --2- 172.21.15.71:0/4076306993 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc140a6810 0x7efc14136f70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7efc0400c430 tx=0x7efc0400c8f0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.887 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.884+0000 7efc1c782640 1 -- 172.21.15.71:0/4076306993 shutdown_connections 2024-09-11T01:29:04.887 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.884+0000 7efc1c782640 1 --2- 172.21.15.71:0/4076306993 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc14138210 0x7efc14120ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.887 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.884+0000 7efc1c782640 1 --2- 172.21.15.71:0/4076306993 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc140a6810 0x7efc14136f70 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.887 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.884+0000 7efc1c782640 1 --2- 172.21.15.71:0/4076306993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc140a57a0 0x7efc141ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:04.887 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.884+0000 7efc1c782640 1 -- 172.21.15.71:0/4076306993 >> 172.21.15.71:0/4076306993 conn(0x7efc140aaa80 msgr2=0x7efc140a0290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:04.887 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.884+0000 7efc1c782640 1 -- 172.21.15.71:0/4076306993 shutdown_connections 2024-09-11T01:29:04.887 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.884+0000 7efc1c782640 1 -- 172.21.15.71:0/4076306993 wait complete. 2024-09-11T01:29:04.896 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new fadf2191-f31a-4880-a529-5b2d1023d236 5 2024-09-11T01:29:04.993 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.991+0000 7f83b93c6640 10 monclient: get_monmap_and_config 2024-09-11T01:29:04.993 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.991+0000 7f83b93c6640 10 monclient: build_initial_monmap 2024-09-11T01:29:04.993 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.991+0000 7f83b93c6640 10 monclient: monmap: 2024-09-11T01:29:04.993 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-09-11T01:29:04.993 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:04.994 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:29:04.992314+0000 2024-09-11T01:29:04.994 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:29:04.992314+0000 2024-09-11T01:29:04.994 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:04.994 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:04.994 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:04.994 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:04.994 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:04.994 INFO:teuthology.orchestra.run.smithi071.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:04.994 INFO:teuthology.orchestra.run.smithi071.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:04.994 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:04.996 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.992+0000 7f83b93c6640 1 Processor -- start 2024-09-11T01:29:04.997 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.992+0000 7f83b93c6640 1 -- start start 2024-09-11T01:29:04.997 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.992+0000 7f83b93c6640 10 monclient: init 2024-09-11T01:29:04.997 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.992+0000 7f83b93c6640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:04.997 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.992+0000 7f83b93c6640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-11T01:29:04.997 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.992+0000 7f83b93c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f83b4138210 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:04.997 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.992+0000 7f83b93c6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f83b40a6810 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:04.997 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.992+0000 7f83b93c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b413e820 0x7f83b4140c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:04.997 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.992+0000 7f83b93c6640 10 monclient(hunting): picked mon.noname-b con 0x7f83b413e820 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:04.997 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.992+0000 7f83b93c6640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83b4138d00 con 0x7f83b4138210 2024-09-11T01:29:04.997 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.992+0000 7f83b93c6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83b409ba20 con 0x7f83b40a6810 2024-09-11T01:29:04.997 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.992+0000 7f83b93c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:04.997 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.992+0000 7f83b93c6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f83b409c4c0 con 0x7f83b413e820 2024-09-11T01:29:04.998 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.992+0000 7f83b93c6640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:04.998 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.992+0000 7f83b93c6640 10 monclient(hunting): authenticate will time out at 733.448644s 2024-09-11T01:29:04.998 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.992+0000 7f83abfff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f83b40a6810 0x7f83b413d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:49116/0 (socket says 172.21.15.71:49116) 2024-09-11T01:29:04.998 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.992+0000 7f83abfff640 1 -- 172.21.15.71:0/1392984083 learned_addr learned my addr 172.21.15.71:0/1392984083 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-09-11T01:29:04.998 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.992+0000 7f83b37fe640 1 --2- 172.21.15.71:0/1392984083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b413e820 0x7f83b4140c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:04.998 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.993+0000 7f83b37fe640 10 monclient(hunting): get_auth_request con 0x7f83b413e820 auth_method 0 2024-09-11T01:29:04.998 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.993+0000 7f83b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:04.998 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.993+0000 7f83b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:04.998 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.993+0000 7f83b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:04.998 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.993+0000 7f83b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:04.998 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.993+0000 7f83b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:04.998 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.993+0000 7f83b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:04.998 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.993+0000 7f83b37fe640 10 monclient(hunting): handle_auth_done global_id 4133 payload 274 2024-09-11T01:29:04.999 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.993+0000 7f83b37fe640 1 -- 172.21.15.71:0/1392984083 >> v1:172.21.15.71:6789/0 conn(0x7f83b40a6810 legacy=0x7f83b413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:04.999 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.993+0000 7f83b37fe640 1 -- 172.21.15.71:0/1392984083 >> v1:172.21.15.12:6789/0 conn(0x7f83b4138210 legacy=0x7f83b41385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:04.999 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.993+0000 7f83b37fe640 10 monclient: _finish_hunting 0 2024-09-11T01:29:04.999 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.993+0000 7f83b37fe640 1 monclient: found mon.noname-b 2024-09-11T01:29:04.999 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.993+0000 7f83b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:04.999 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.993+0000 7f83b37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:29:04.999 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.993+0000 7f83b37fe640 1 -- 172.21.15.71:0/1392984083 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83b40a6590 con 0x7f83b413e820 2024-09-11T01:29:05.005 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.996+0000 7f83b37fe640 1 -- 172.21.15.71:0/1392984083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b413e820 msgr2=0x7f83b4140c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:05.005 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.996+0000 7f83b37fe640 1 --2- 172.21.15.71:0/1392984083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b413e820 0x7f83b4140c10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f83ac009dc0 tx=0x7f83ac02a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3065abac043d58a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:05.005 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.997+0000 7f83b27fc640 1 -- 172.21.15.71:0/1392984083 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ac007a30 con 0x7f83b413e820 2024-09-11T01:29:05.005 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.997+0000 7f83b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:05.005 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.997+0000 7f83b27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:05.005 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.997+0000 7f83b27fc640 10 monclient: dump: 2024-09-11T01:29:05.005 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-09-11T01:29:05.005 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:05.005 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:05.005 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:05.005 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:05.005 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:05.006 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:05.006 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:05.006 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:05.006 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:05.006 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.997+0000 7f83b27fc640 10 monclient: _finish_auth 0 2024-09-11T01:29:05.006 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.997+0000 7f83b27fc640 10 monclient: _check_auth_tickets 2024-09-11T01:29:05.006 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.997+0000 7f83b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:05.006 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.997+0000 7f83b27fc640 1 -- 172.21.15.71:0/1392984083 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83ac007bd0 con 0x7f83b413e820 2024-09-11T01:29:05.006 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.997+0000 7f83b93c6640 5 monclient: authenticate success, global_id 4133 2024-09-11T01:29:05.006 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.997+0000 7f83b93c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:05.006 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.997+0000 7f83b27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:05.006 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.997+0000 7f83b93c6640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:05.006 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.997+0000 7f83b27fc640 1 -- 172.21.15.71:0/1392984083 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ac031400 con 0x7f83b413e820 2024-09-11T01:29:05.006 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.997+0000 7f83b93c6640 4 set_mon_vals no callback set 2024-09-11T01:29:05.007 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.997+0000 7f83b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:05.007 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.997+0000 7f83b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:05.007 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.997+0000 7f83b27fc640 10 monclient: dump: 2024-09-11T01:29:05.007 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-09-11T01:29:05.007 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:05.007 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:05.007 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:05.007 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:05.007 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:05.007 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:05.007 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:05.007 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:05.007 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:05.007 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.997+0000 7f83b93c6640 10 monclient: shutdown 2024-09-11T01:29:05.008 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.997+0000 7f83b93c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:05.008 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.997+0000 7f83b93c6640 1 -- 172.21.15.71:0/1392984083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b413e820 msgr2=0x7f83b4140c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:05.008 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.997+0000 7f83b93c6640 1 --2- 172.21.15.71:0/1392984083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b413e820 0x7f83b4140c10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f83ac009dc0 tx=0x7f83ac02a480 comp rx=0 tx=0).stop 2024-09-11T01:29:05.008 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.997+0000 7f83b93c6640 1 -- 172.21.15.71:0/1392984083 shutdown_connections 2024-09-11T01:29:05.008 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.997+0000 7f83b93c6640 1 --2- 172.21.15.71:0/1392984083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b413e820 0x7f83b4140c10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:05.008 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.997+0000 7f83b93c6640 1 -- 172.21.15.71:0/1392984083 >> 172.21.15.71:0/1392984083 conn(0x7f83b40aaa80 msgr2=0x7f83b40aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:05.008 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.997+0000 7f83b93c6640 1 -- 172.21.15.71:0/1392984083 shutdown_connections 2024-09-11T01:29:05.008 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.997+0000 7f83b93c6640 1 -- 172.21.15.71:0/1392984083 wait complete. 2024-09-11T01:29:05.008 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.998+0000 7f83b93c6640 10 monclient: build_initial_monmap 2024-09-11T01:29:05.008 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.998+0000 7f83b93c6640 10 monclient: monmap: 2024-09-11T01:29:05.008 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-09-11T01:29:05.008 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:05.008 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 0.000000 2024-09-11T01:29:05.009 INFO:teuthology.orchestra.run.smithi071.stderr:created 0.000000 2024-09-11T01:29:05.009 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:05.009 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:05.009 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:05.009 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:05.009 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:05.009 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:05.009 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.998+0000 7f83b93c6640 1 Processor -- start 2024-09-11T01:29:05.009 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.998+0000 7f83b93c6640 1 -- start start 2024-09-11T01:29:05.009 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.998+0000 7f83b93c6640 10 monclient: init 2024-09-11T01:29:05.009 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b93c6640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:05.009 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b93c6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:29:05.009 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b93c6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83b40a6810 0x7f83b413c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:05.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b93c6640 10 monclient(hunting): picked mon.noname-a con 0x7f83b40a6810 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:05.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b93c6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83b4138210 0x7f83b41cd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:05.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b93c6640 10 monclient(hunting): picked mon.noname-b con 0x7f83b4138210 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:05.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b93c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b413e820 0x7f83b4139cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:05.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b93c6640 10 monclient(hunting): picked mon.noname-c con 0x7f83b413e820 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:05.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b93c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:05.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b93c6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f83b406f670 con 0x7f83b40a6810 2024-09-11T01:29:05.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b93c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:05.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b93c6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f83b406f4f0 con 0x7f83b4138210 2024-09-11T01:29:05.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b93c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:05.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b93c6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f83b4141150 con 0x7f83b413e820 2024-09-11T01:29:05.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b93c6640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:05.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b93c6640 10 monclient(hunting): authenticate will time out at 733.455227s 2024-09-11T01:29:05.010 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83abfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83b4138210 0x7f83b41cd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:05.011 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83abfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83b4138210 0x7f83b41cd7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46500/0 (socket says 172.21.15.71:46500) 2024-09-11T01:29:05.011 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83abfff640 1 -- 172.21.15.71:0/1247584803 learned_addr learned my addr 172.21.15.71:0/1247584803 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-09-11T01:29:05.011 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83abfff640 10 monclient(hunting): get_auth_request con 0x7f83b4138210 auth_method 0 2024-09-11T01:29:05.011 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:05.011 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83abfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:05.011 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:05.011 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:05.011 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:05.011 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:05.011 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b37fe640 1 --2- 172.21.15.71:0/1247584803 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b413e820 0x7f83b4139cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:05.011 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b2ffd640 1 --2- 172.21.15.71:0/1247584803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83b40a6810 0x7f83b413c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:05.011 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b37fe640 10 monclient(hunting): get_auth_request con 0x7f83b413e820 auth_method 0 2024-09-11T01:29:05.011 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:05.011 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:05.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:05.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b2ffd640 10 monclient(hunting): get_auth_request con 0x7f83b40a6810 auth_method 0 2024-09-11T01:29:05.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:05.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:05.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:05.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:05.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:05.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:05.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:05.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:05.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:05.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83abfff640 10 monclient(hunting): handle_auth_done global_id 4126 payload 824 2024-09-11T01:29:05.012 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83abfff640 1 -- 172.21.15.71:0/1247584803 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b413e820 msgr2=0x7f83b4139cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:05.013 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83abfff640 1 --2- 172.21.15.71:0/1247584803 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b413e820 0x7f83b4139cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:05.013 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83abfff640 1 -- 172.21.15.71:0/1247584803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83b40a6810 msgr2=0x7f83b413c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:05.013 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83abfff640 1 --2- 172.21.15.71:0/1247584803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83b40a6810 0x7f83b413c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:05.013 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83abfff640 10 monclient: _finish_hunting 0 2024-09-11T01:29:05.013 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83abfff640 1 monclient: found mon.noname-b 2024-09-11T01:29:05.013 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:05.013 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83abfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-09-11T01:29:05.013 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:04.999+0000 7f83abfff640 1 -- 172.21.15.71:0/1247584803 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83b413d700 con 0x7f83b4138210 2024-09-11T01:29:05.013 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.001+0000 7f83abfff640 1 -- 172.21.15.71:0/1247584803 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83b4138210 msgr2=0x7f83b41cd7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:05.013 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.001+0000 7f83abfff640 1 --2- 172.21.15.71:0/1247584803 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83b4138210 0x7f83b41cd7e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f839c00c460 tx=0x7f839c00c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:05.013 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.002+0000 7f83ab7fe640 1 -- 172.21.15.71:0/1247584803 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839c007070 con 0x7f83b4138210 2024-09-11T01:29:05.013 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.002+0000 7f83ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:05.013 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.002+0000 7f83ab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:05.014 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.002+0000 7f83ab7fe640 10 monclient: dump: 2024-09-11T01:29:05.014 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-09-11T01:29:05.014 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:05.014 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:05.014 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:05.014 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:05.014 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:05.014 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:05.014 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:05.014 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:05.014 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:05.014 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.002+0000 7f83ab7fe640 10 monclient: _finish_auth 0 2024-09-11T01:29:05.014 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.002+0000 7f83ab7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:29:05.014 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.002+0000 7f83ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:05.015 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.002+0000 7f83ab7fe640 1 -- 172.21.15.71:0/1247584803 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f839c007230 con 0x7f83b4138210 2024-09-11T01:29:05.015 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.002+0000 7f83b93c6640 5 monclient: authenticate success, global_id 4126 2024-09-11T01:29:05.015 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.002+0000 7f83ab7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:05.015 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.002+0000 7f83b93c6640 10 monclient: _renew_subs 2024-09-11T01:29:05.015 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.002+0000 7f83ab7fe640 1 -- 172.21.15.71:0/1247584803 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839c007620 con 0x7f83b4138210 2024-09-11T01:29:05.015 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.002+0000 7f83b93c6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:05.015 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.002+0000 7f83b93c6640 1 -- 172.21.15.71:0/1247584803 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83b40a1880 con 0x7f83b4138210 2024-09-11T01:29:05.015 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.002+0000 7f83ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:05.015 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.002+0000 7f83ab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:05.015 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.002+0000 7f83b1ffb640 4 set_mon_vals no callback set 2024-09-11T01:29:05.015 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.002+0000 7f83ab7fe640 10 monclient: dump: 2024-09-11T01:29:05.015 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-09-11T01:29:05.015 INFO:teuthology.orchestra.run.smithi071.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:05.016 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:05.016 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:05.016 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:05.016 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-09-11T01:29:05.016 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:05.016 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:05.016 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:05.016 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:29:05.016 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.002+0000 7f83b93c6640 10 monclient: _renew_subs 2024-09-11T01:29:05.016 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.002+0000 7f83b93c6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:05.016 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.002+0000 7f83b93c6640 1 -- 172.21.15.71:0/1247584803 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83b413d420 con 0x7f83b4138210 2024-09-11T01:29:05.016 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.002+0000 7f83ab7fe640 1 -- 172.21.15.71:0/1247584803 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f839c01b400 con 0x7f83b4138210 2024-09-11T01:29:05.016 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.002+0000 7f83ab7fe640 1 -- 172.21.15.71:0/1247584803 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f839c012ce0 con 0x7f83b4138210 2024-09-11T01:29:05.016 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.003+0000 7f83b93c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:05.017 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.003+0000 7f83b93c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:05.017 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.003+0000 7f83b93c6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:05.017 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.003+0000 7f83b93c6640 1 -- 172.21.15.71:0/1247584803 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83b41385f0 con 0x7f83b4138210 2024-09-11T01:29:05.017 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.005+0000 7f83ab7fe640 1 -- 172.21.15.71:0/1247584803 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f839c012720 con 0x7f83b4138210 2024-09-11T01:29:05.017 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.005+0000 7f83ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:05.017 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.005+0000 7f83ab7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:05.075 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.073+0000 7f83b93c6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "fadf2191-f31a-4880-a529-5b2d1023d236", "id": 5}] 2024-09-11T01:29:05.076 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.073+0000 7f83b93c6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "fadf2191-f31a-4880-a529-5b2d1023d236", "id": 5}] 2024-09-11T01:29:05.076 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.073+0000 7f83b93c6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:05.076 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.073+0000 7f83b93c6640 1 -- 172.21.15.71:0/1247584803 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fadf2191-f31a-4880-a529-5b2d1023d236", "id": 5} v 0) -- 0x7f83b4074b50 con 0x7f83b4138210 2024-09-11T01:29:05.083 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.082+0000 7f83ab7fe640 1 -- 172.21.15.71:0/1247584803 <== mon.1 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fadf2191-f31a-4880-a529-5b2d1023d236", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f839c012900 con 0x7f83b4138210 2024-09-11T01:29:05.084 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.082+0000 7f83ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "fadf2191-f31a-4880-a529-5b2d1023d236", "id": 5}] 2024-09-11T01:29:05.084 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.082+0000 7f83ab7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:05.084 INFO:teuthology.orchestra.run.smithi071.stdout:5 2024-09-11T01:29:05.085 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.083+0000 7f83b93c6640 10 monclient: shutdown 2024-09-11T01:29:05.085 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.083+0000 7f83b93c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:05.085 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.083+0000 7f83b93c6640 1 -- 172.21.15.71:0/1247584803 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83b4138210 msgr2=0x7f83b41cd7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:05.085 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.083+0000 7f83b93c6640 1 --2- 172.21.15.71:0/1247584803 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83b4138210 0x7f83b41cd7e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f839c00c460 tx=0x7f839c00c920 comp rx=0 tx=0).stop 2024-09-11T01:29:05.085 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.083+0000 7f83b93c6640 1 -- 172.21.15.71:0/1247584803 shutdown_connections 2024-09-11T01:29:05.085 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.083+0000 7f83b93c6640 1 --2- 172.21.15.71:0/1247584803 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b413e820 0x7f83b4139cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:05.085 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.083+0000 7f83b93c6640 1 --2- 172.21.15.71:0/1247584803 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83b4138210 0x7f83b41cd7e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:05.086 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.083+0000 7f83b93c6640 1 --2- 172.21.15.71:0/1247584803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83b40a6810 0x7f83b413c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:05.086 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.083+0000 7f83b93c6640 1 -- 172.21.15.71:0/1247584803 >> 172.21.15.71:0/1247584803 conn(0x7f83b40aaa80 msgr2=0x7f83b4140020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:05.086 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.083+0000 7f83b93c6640 1 -- 172.21.15.71:0/1247584803 shutdown_connections 2024-09-11T01:29:05.086 INFO:teuthology.orchestra.run.smithi071.stderr:2024-09-11T01:29:05.083+0000 7f83b93c6640 1 -- 172.21.15.71:0/1247584803 wait complete. 2024-09-11T01:29:05.095 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-11T01:29:05.096 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-11T01:29:05.098 INFO:tasks.ceph.osd.0:Started 2024-09-11T01:29:05.099 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-11T01:29:05.099 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-11T01:29:05.101 INFO:tasks.ceph.osd.1:Started 2024-09-11T01:29:05.102 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-11T01:29:05.102 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-11T01:29:05.104 INFO:tasks.ceph.osd.2:Started 2024-09-11T01:29:05.104 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-11T01:29:05.104 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-11T01:29:05.106 INFO:tasks.ceph.osd.3:Started 2024-09-11T01:29:05.106 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-11T01:29:05.107 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-11T01:29:05.109 INFO:tasks.ceph.osd.4:Started 2024-09-11T01:29:05.109 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-11T01:29:05.109 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-11T01:29:05.111 INFO:tasks.ceph.osd.5:Started 2024-09-11T01:29:05.111 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T01:29:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.225+0000 7f57ef59e640 10 monclient: get_monmap_and_config 2024-09-11T01:29:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.226+0000 7f57ef59e640 10 monclient: build_initial_monmap 2024-09-11T01:29:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.226+0000 7f57ef59e640 10 monclient: monmap: 2024-09-11T01:29:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:05.227756+0000 2024-09-11T01:29:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:05.227756+0000 2024-09-11T01:29:05.230 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:05.230 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:05.230 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:05.230 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:05.230 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:05.230 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:05.230 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:05.230 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:05.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.226+0000 7f57ef59e640 1 Processor -- start 2024-09-11T01:29:05.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.226+0000 7f57ef59e640 1 -- start start 2024-09-11T01:29:05.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.226+0000 7f57ef59e640 10 monclient: init 2024-09-11T01:29:05.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.227+0000 7f57ef59e640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:05.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.227+0000 7f57ef59e640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-11T01:29:05.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.227+0000 7f57ef59e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f57f00a4eb0 0x7f57f009cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:05.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.227+0000 7f57ef59e640 10 monclient(hunting): picked mon.noname-a con 0x7f57f00a4eb0 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:05.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.227+0000 7f57ef59e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f57f00a5290 0x7f57f00a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:05.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.227+0000 7f57ef59e640 10 monclient(hunting): picked mon.noname-b con 0x7f57f00a5290 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:05.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.227+0000 7f57ef59e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f57f0137c70 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:05.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.227+0000 7f57ef59e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57f00a6050 con 0x7f57f0137c70 2024-09-11T01:29:05.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.227+0000 7f57ef59e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:05.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.227+0000 7f57ef59e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f57f00a6540 con 0x7f57f00a4eb0 2024-09-11T01:29:05.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.227+0000 7f57ee59c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f57f00a4eb0 0x7f57f009cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:05.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.227+0000 7f57ef59e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:05.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.227+0000 7f57eed9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f57f0137c70 0x7f57f0138090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46936/0 (socket says 172.21.15.12:46936) 2024-09-11T01:29:05.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.227+0000 7f57eed9d640 1 -- 172.21.15.12:0/2090268823 learned_addr learned my addr 172.21.15.12:0/2090268823 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:29:05.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.227+0000 7f57ef59e640 1 -- 172.21.15.12:0/2090268823 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f57f00a6710 con 0x7f57f00a5290 2024-09-11T01:29:05.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.227+0000 7f57edd9b640 1 --2- 172.21.15.12:0/2090268823 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f57f00a5290 0x7f57f00a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:05.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.227+0000 7f57ef59e640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:05.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.227+0000 7f57ef59e640 10 monclient(hunting): authenticate will time out at 751.670958s 2024-09-11T01:29:05.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.227+0000 7f57ee59c640 10 monclient(hunting): get_auth_request con 0x7f57f00a4eb0 auth_method 0 2024-09-11T01:29:05.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.227+0000 7f57ee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:05.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.228+0000 7f57ee59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.228+0000 7f57ee59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.228+0000 7f57edd9b640 10 monclient(hunting): get_auth_request con 0x7f57f00a5290 auth_method 0 2024-09-11T01:29:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.228+0000 7f57edd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.228+0000 7f57edd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.228+0000 7f57edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.228+0000 7f57ee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.228+0000 7f57ee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.228+0000 7f57ee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.228+0000 7f57edd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.228+0000 7f57edd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.228+0000 7f57edd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.228+0000 7f57ee59c640 10 monclient(hunting): handle_auth_done global_id 4149 payload 274 2024-09-11T01:29:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.228+0000 7f57ee59c640 1 -- 172.21.15.12:0/2090268823 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f57f00a5290 msgr2=0x7f57f00a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:05.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.228+0000 7f57ee59c640 1 --2- 172.21.15.12:0/2090268823 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f57f00a5290 0x7f57f00a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:05.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.228+0000 7f57ee59c640 1 -- 172.21.15.12:0/2090268823 >> v1:172.21.15.12:6789/0 conn(0x7f57f0137c70 legacy=0x7f57f0138090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:05.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.228+0000 7f57ee59c640 10 monclient: _finish_hunting 0 2024-09-11T01:29:05.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.228+0000 7f57ee59c640 1 monclient: found mon.noname-a 2024-09-11T01:29:05.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.228+0000 7f57ee59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:05.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.228+0000 7f57ee59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:05.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.228+0000 7f57ee59c640 1 -- 172.21.15.12:0/2090268823 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57f009cb60 con 0x7f57f00a4eb0 2024-09-11T01:29:05.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.228+0000 7f57edd9b640 1 --2- 172.21.15.12:0/2090268823 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f57f00a5290 0x7f57f00a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:05.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ee59c640 1 -- 172.21.15.12:0/2090268823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57f00a4eb0 msgr2=0x7f57f009cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:05.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ee59c640 1 --2- 172.21.15.12:0/2090268823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57f00a4eb0 0x7f57f009cf80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f57e00097a0 tx=0x7f57e0029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b03a879ae457ea1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:05.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ed59a640 1 -- 172.21.15.12:0/2090268823 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e0037070 con 0x7f57f00a4eb0 2024-09-11T01:29:05.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ed59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:05.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ed59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:05.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ed59a640 10 monclient: dump: 2024-09-11T01:29:05.242 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:05.243 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:05.243 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:05.243 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:05.243 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:05.243 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:05.243 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:05.243 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:05.243 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:05.243 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:05.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ed59a640 10 monclient: _finish_auth 0 2024-09-11T01:29:05.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ed59a640 10 monclient: _check_auth_tickets 2024-09-11T01:29:05.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:05.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ed59a640 1 -- 172.21.15.12:0/2090268823 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57e0032440 con 0x7f57f00a4eb0 2024-09-11T01:29:05.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ed59a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:05.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ed59a640 1 -- 172.21.15.12:0/2090268823 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e0032810 con 0x7f57f00a4eb0 2024-09-11T01:29:05.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ed59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:05.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ed59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:05.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ed59a640 10 monclient: dump: 2024-09-11T01:29:05.244 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:05.244 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:05.244 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:05.244 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:05.244 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:05.244 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:05.244 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:05.244 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:05.244 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:05.245 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ef59e640 5 monclient: authenticate success, global_id 4149 2024-09-11T01:29:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ef59e640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ef59e640 4 set_mon_vals no callback set 2024-09-11T01:29:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ef59e640 10 monclient: shutdown 2024-09-11T01:29:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ef59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ef59e640 1 -- 172.21.15.12:0/2090268823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57f00a4eb0 msgr2=0x7f57f009cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ef59e640 1 --2- 172.21.15.12:0/2090268823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57f00a4eb0 0x7f57f009cf80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f57e00097a0 tx=0x7f57e0029fd0 comp rx=0 tx=0).stop 2024-09-11T01:29:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ef59e640 1 -- 172.21.15.12:0/2090268823 shutdown_connections 2024-09-11T01:29:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ef59e640 1 --2- 172.21.15.12:0/2090268823 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f57f00a5290 0x7f57f00a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ef59e640 1 --2- 172.21.15.12:0/2090268823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57f00a4eb0 0x7f57f009cf80 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ef59e640 1 -- 172.21.15.12:0/2090268823 >> 172.21.15.12:0/2090268823 conn(0x7f57f0068b30 msgr2=0x7f57f0068f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ef59e640 1 -- 172.21.15.12:0/2090268823 shutdown_connections 2024-09-11T01:29:05.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.231+0000 7f57ef59e640 1 -- 172.21.15.12:0/2090268823 wait complete. 2024-09-11T01:29:05.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.232+0000 7f57ef59e640 10 monclient: build_initial_monmap 2024-09-11T01:29:05.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.232+0000 7f57ef59e640 10 monclient: monmap: 2024-09-11T01:29:05.246 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:05.246 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:05.246 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:05.246 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:05.246 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:05.246 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:05.246 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:05.246 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:05.246 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:05.246 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:05.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57ef59e640 1 Processor -- start 2024-09-11T01:29:05.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57ef59e640 1 -- start start 2024-09-11T01:29:05.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57ef59e640 10 monclient: init 2024-09-11T01:29:05.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57ef59e640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:05.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57ef59e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:29:05.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57ef59e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f57f00a4eb0 0x7f57f01cd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:05.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57ef59e640 10 monclient(hunting): picked mon.noname-b con 0x7f57f00a4eb0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:05.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57ef59e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f57f00a5290 0x7f57f01cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:05.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57ef59e640 10 monclient(hunting): picked mon.noname-c con 0x7f57f00a5290 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:05.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57ef59e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57f0137c70 0x7f57f01244d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:05.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57ef59e640 10 monclient(hunting): picked mon.noname-a con 0x7f57f0137c70 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:05.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57ef59e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:05.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57ef59e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f57f006f6d0 con 0x7f57f0137c70 2024-09-11T01:29:05.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57ef59e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:05.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57ef59e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f57f00a5c00 con 0x7f57f00a4eb0 2024-09-11T01:29:05.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57ef59e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:05.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57ef59e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f57f006f850 con 0x7f57f00a5290 2024-09-11T01:29:05.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57ef59e640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:05.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57ef59e640 10 monclient(hunting): authenticate will time out at 751.676713s 2024-09-11T01:29:05.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57edd9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f57f00a5290 0x7f57f01cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:05.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57edd9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f57f00a5290 0x7f57f01cd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51128/0 (socket says 172.21.15.12:51128) 2024-09-11T01:29:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57edd9b640 1 -- 172.21.15.12:0/1007280208 learned_addr learned my addr 172.21.15.12:0/1007280208 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57edd9b640 10 monclient(hunting): get_auth_request con 0x7f57f00a5290 auth_method 0 2024-09-11T01:29:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57edd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57edd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57eed9d640 1 --2- 172.21.15.12:0/1007280208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57f0137c70 0x7f57f01244d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57eed9d640 10 monclient(hunting): get_auth_request con 0x7f57f0137c70 auth_method 0 2024-09-11T01:29:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57eed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57eed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57eed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57edd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57edd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57edd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57ee59c640 1 --2- 172.21.15.12:0/1007280208 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f57f00a4eb0 0x7f57f01cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57ee59c640 10 monclient(hunting): get_auth_request con 0x7f57f00a4eb0 auth_method 0 2024-09-11T01:29:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57ee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57ee59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57ee59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57eed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57eed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.233+0000 7f57eed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.234+0000 7f57ee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.234+0000 7f57ee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.234+0000 7f57ee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:05.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.234+0000 7f57edd9b640 10 monclient(hunting): handle_auth_done global_id 4157 payload 824 2024-09-11T01:29:05.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.234+0000 7f57edd9b640 1 -- 172.21.15.12:0/1007280208 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f57f00a4eb0 msgr2=0x7f57f01cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:05.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.234+0000 7f57edd9b640 1 --2- 172.21.15.12:0/1007280208 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f57f00a4eb0 0x7f57f01cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:05.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.234+0000 7f57edd9b640 1 -- 172.21.15.12:0/1007280208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57f0137c70 msgr2=0x7f57f01244d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:05.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.234+0000 7f57edd9b640 1 --2- 172.21.15.12:0/1007280208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57f0137c70 0x7f57f01244d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:05.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.234+0000 7f57edd9b640 10 monclient: _finish_hunting 0 2024-09-11T01:29:05.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.234+0000 7f57edd9b640 1 monclient: found mon.noname-c 2024-09-11T01:29:05.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.234+0000 7f57edd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:05.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.234+0000 7f57edd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:29:05.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.234+0000 7f57edd9b640 1 -- 172.21.15.12:0/1007280208 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57f01378b0 con 0x7f57f00a5290 2024-09-11T01:29:05.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.236+0000 7f57edd9b640 1 -- 172.21.15.12:0/1007280208 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f57f00a5290 msgr2=0x7f57f01cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:05.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.236+0000 7f57edd9b640 1 --2- 172.21.15.12:0/1007280208 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f57f00a5290 0x7f57f01cd7a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f57e400ebe0 tx=0x7f57e400c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:05.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.236+0000 7f57deffd640 1 -- 172.21.15.12:0/1007280208 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e4017070 con 0x7f57f00a5290 2024-09-11T01:29:05.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.236+0000 7f57deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:05.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.236+0000 7f57deffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:05.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.236+0000 7f57deffd640 10 monclient: dump: 2024-09-11T01:29:05.252 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:05.252 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:05.252 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:05.252 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:05.252 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:05.252 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:05.252 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:05.252 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:05.252 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:05.253 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:05.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.236+0000 7f57deffd640 10 monclient: _finish_auth 0 2024-09-11T01:29:05.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.236+0000 7f57deffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:05.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.236+0000 7f57deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:05.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.236+0000 7f57deffd640 1 -- 172.21.15.12:0/1007280208 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57e400ce40 con 0x7f57f00a5290 2024-09-11T01:29:05.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.236+0000 7f57deffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:05.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.236+0000 7f57deffd640 1 -- 172.21.15.12:0/1007280208 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e4013690 con 0x7f57f00a5290 2024-09-11T01:29:05.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.236+0000 7f57deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:05.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.236+0000 7f57deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:05.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.236+0000 7f57deffd640 10 monclient: dump: 2024-09-11T01:29:05.253 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:05.253 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:05.253 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:05.254 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:05.254 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:05.254 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:05.254 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:05.254 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:05.254 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:05.254 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:05.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.236+0000 7f57ef59e640 5 monclient: authenticate success, global_id 4157 2024-09-11T01:29:05.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.236+0000 7f57ef59e640 10 monclient: _renew_subs 2024-09-11T01:29:05.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.236+0000 7f57ef59e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:05.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.236+0000 7f57ef59e640 1 -- 172.21.15.12:0/1007280208 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57f01282f0 con 0x7f57f00a5290 2024-09-11T01:29:05.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.236+0000 7f57dffff640 4 set_mon_vals no callback set 2024-09-11T01:29:05.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.236+0000 7f57ef59e640 10 monclient: _renew_subs 2024-09-11T01:29:05.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.236+0000 7f57ef59e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:05.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.236+0000 7f57ef59e640 1 -- 172.21.15.12:0/1007280208 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f57f0128010 con 0x7f57f00a5290 2024-09-11T01:29:05.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.236+0000 7f57deffd640 1 -- 172.21.15.12:0/1007280208 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f57e4024060 con 0x7f57f00a5290 2024-09-11T01:29:05.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.236+0000 7f57deffd640 1 -- 172.21.15.12:0/1007280208 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f57e4013c20 con 0x7f57f00a5290 2024-09-11T01:29:05.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.237+0000 7f57ef59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:05.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.237+0000 7f57ef59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:05.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.237+0000 7f57ef59e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:05.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.237+0000 7f57ef59e640 1 -- 172.21.15.12:0/1007280208 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57f0138910 con 0x7f57f00a5290 2024-09-11T01:29:05.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.239+0000 7f57deffd640 1 -- 172.21.15.12:0/1007280208 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f57e4026480 con 0x7f57f00a5290 2024-09-11T01:29:05.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.239+0000 7f57deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:05.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.239+0000 7f57deffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:05.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.301+0000 7f57ef59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-11T01:29:05.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.301+0000 7f57ef59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T01:29:05.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.301+0000 7f57ef59e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:05.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.301+0000 7f57ef59e640 1 -- 172.21.15.12:0/1007280208 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f57f0126af0 con 0x7f57f00a5290 2024-09-11T01:29:05.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.301+0000 7f57deffd640 1 -- 172.21.15.12:0/1007280208 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f57e4026660 con 0x7f57f00a5290 2024-09-11T01:29:05.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.301+0000 7f57deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T01:29:05.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.301+0000 7f57deffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:05.305 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:29:05.305 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":8,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","created":"2024-09-11T01:29:02.540751+0000","modified":"2024-09-11T01:29:05.075851+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T01:29:05.075851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"616ba1d0-adf6-4fe1-85e3-266e53e21867","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9b570874-7a3d-4867-8c44-29084a8bf649","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b12ee288-ce57-41de-8b8e-6fad509311b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8e81aa4-0dee-487d-981d-1f593e363eee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9db77181-fc18-408f-a35c-87bf4b4ba2d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fadf2191-f31a-4880-a529-5b2d1023d236","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T01:29:05.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.302+0000 7f57ef59e640 10 monclient: shutdown 2024-09-11T01:29:05.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.302+0000 7f57ef59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:05.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.302+0000 7f57ef59e640 1 -- 172.21.15.12:0/1007280208 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f57f00a5290 msgr2=0x7f57f01cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:05.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.302+0000 7f57ef59e640 1 --2- 172.21.15.12:0/1007280208 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f57f00a5290 0x7f57f01cd7a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f57e400ebe0 tx=0x7f57e400c520 comp rx=0 tx=0).stop 2024-09-11T01:29:05.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.302+0000 7f57ef59e640 1 -- 172.21.15.12:0/1007280208 shutdown_connections 2024-09-11T01:29:05.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.302+0000 7f57ef59e640 1 --2- 172.21.15.12:0/1007280208 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57f0137c70 0x7f57f01244d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:05.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.302+0000 7f57ef59e640 1 --2- 172.21.15.12:0/1007280208 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f57f00a5290 0x7f57f01cd7a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:05.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.302+0000 7f57ef59e640 1 --2- 172.21.15.12:0/1007280208 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f57f00a4eb0 0x7f57f01cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:05.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.302+0000 7f57ef59e640 1 -- 172.21.15.12:0/1007280208 >> 172.21.15.12:0/1007280208 conn(0x7f57f0068b30 msgr2=0x7f57f00a2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:05.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.302+0000 7f57ef59e640 1 -- 172.21.15.12:0/1007280208 shutdown_connections 2024-09-11T01:29:05.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:05.302+0000 7f57ef59e640 1 -- 172.21.15.12:0/1007280208 wait complete. 2024-09-11T01:29:05.315 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-11T01:29:05.315 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-11T01:29:05.920 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T01:29:06.022 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.014+0000 7f518a233640 10 monclient: get_monmap_and_config 2024-09-11T01:29:06.022 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.015+0000 7f518a233640 10 monclient: build_initial_monmap 2024-09-11T01:29:06.022 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.015+0000 7f518a233640 10 monclient: monmap: 2024-09-11T01:29:06.022 INFO:teuthology.misc.health.smithi012.stderr:epoch 0 2024-09-11T01:29:06.023 INFO:teuthology.misc.health.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:06.023 INFO:teuthology.misc.health.smithi012.stderr:last_changed 2024-09-11T01:29:06.016756+0000 2024-09-11T01:29:06.023 INFO:teuthology.misc.health.smithi012.stderr:created 2024-09-11T01:29:06.016756+0000 2024-09-11T01:29:06.023 INFO:teuthology.misc.health.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:06.023 INFO:teuthology.misc.health.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:06.023 INFO:teuthology.misc.health.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:06.023 INFO:teuthology.misc.health.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:06.023 INFO:teuthology.misc.health.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:06.024 INFO:teuthology.misc.health.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:06.024 INFO:teuthology.misc.health.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:06.024 INFO:teuthology.misc.health.smithi012.stderr: 2024-09-11T01:29:06.024 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.015+0000 7f518a233640 1 Processor -- start 2024-09-11T01:29:06.024 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.015+0000 7f518a233640 1 -- start start 2024-09-11T01:29:06.024 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.015+0000 7f518a233640 10 monclient: init 2024-09-11T01:29:06.024 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f518a233640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:06.025 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f518a233640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-11T01:29:06.025 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f518a233640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f51840a1cf0 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:06.025 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f518a233640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f518409e260 0x7f518409bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:06.025 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f518a233640 10 monclient(hunting): picked mon.noname-c con 0x7f518409e260 addr v2:172.21.15.71:3300/0 2024-09-11T01:29:06.025 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f518a233640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f518413a470 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:06.025 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f518a233640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51840a0770 con 0x7f518413a470 2024-09-11T01:29:06.026 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f518a233640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f518409d260 con 0x7f51840a1cf0 2024-09-11T01:29:06.026 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f518a233640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:06.026 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f518a233640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f51840a20d0 con 0x7f518409e260 2024-09-11T01:29:06.026 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f518a233640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:06.027 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f518a233640 10 monclient(hunting): authenticate will time out at 752.459386s 2024-09-11T01:29:06.027 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f5183fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f518413a470 0x7f518413c860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46948/0 (socket says 172.21.15.12:46948) 2024-09-11T01:29:06.027 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f5183fff640 1 -- 172.21.15.12:0/3905582858 learned_addr learned my addr 172.21.15.12:0/3905582858 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:29:06.027 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f5182ffd640 1 --2- 172.21.15.12:0/3905582858 >> v2:172.21.15.71:3300/0 conn(0x7f518409e260 0x7f518409bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:06.027 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f5182ffd640 10 monclient(hunting): get_auth_request con 0x7f518409e260 auth_method 0 2024-09-11T01:29:06.027 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f5182ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:06.028 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f5182ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:06.028 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f5182ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:06.028 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f5182ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:06.028 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f5182ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:06.028 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f5182ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:06.029 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f5182ffd640 10 monclient(hunting): handle_auth_done global_id 4138 payload 274 2024-09-11T01:29:06.029 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f5182ffd640 1 -- 172.21.15.12:0/3905582858 >> v1:172.21.15.71:6789/0 conn(0x7f51840a1cf0 legacy=0x7f518409b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:06.029 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f5182ffd640 1 -- 172.21.15.12:0/3905582858 >> v1:172.21.15.12:6789/0 conn(0x7f518413a470 legacy=0x7f518413c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:06.029 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f5182ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:29:06.029 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f5182ffd640 1 monclient: found mon.noname-c 2024-09-11T01:29:06.030 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f5182ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:06.030 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f5182ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:29:06.030 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.016+0000 7f5182ffd640 1 -- 172.21.15.12:0/3905582858 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f518409d880 con 0x7f518409e260 2024-09-11T01:29:06.030 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.019+0000 7f5182ffd640 1 -- 172.21.15.12:0/3905582858 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f518409e260 msgr2=0x7f518409bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:06.030 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.019+0000 7f5182ffd640 1 --2- 172.21.15.12:0/3905582858 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f518409e260 0x7f518409bb70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f518406d3a0 tx=0x7f516c029fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9de843bc7fcf7781 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:06.036 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.020+0000 7f51827fc640 1 -- 172.21.15.12:0/3905582858 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516c037070 con 0x7f518409e260 2024-09-11T01:29:06.036 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.020+0000 7f51827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:06.036 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.020+0000 7f51827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:06.036 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.020+0000 7f51827fc640 10 monclient: dump: 2024-09-11T01:29:06.036 INFO:teuthology.misc.health.smithi012.stderr:epoch 1 2024-09-11T01:29:06.036 INFO:teuthology.misc.health.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:06.036 INFO:teuthology.misc.health.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:06.036 INFO:teuthology.misc.health.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:06.036 INFO:teuthology.misc.health.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:06.036 INFO:teuthology.misc.health.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:06.036 INFO:teuthology.misc.health.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:06.036 INFO:teuthology.misc.health.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:06.036 INFO:teuthology.misc.health.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:06.037 INFO:teuthology.misc.health.smithi012.stderr: 2024-09-11T01:29:06.037 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.020+0000 7f51827fc640 10 monclient: _finish_auth 0 2024-09-11T01:29:06.037 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.020+0000 7f51827fc640 10 monclient: _check_auth_tickets 2024-09-11T01:29:06.037 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.020+0000 7f51827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:06.037 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.020+0000 7f51827fc640 1 -- 172.21.15.12:0/3905582858 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f516c032440 con 0x7f518409e260 2024-09-11T01:29:06.037 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.020+0000 7f51827fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:06.037 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.020+0000 7f51827fc640 1 -- 172.21.15.12:0/3905582858 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516c032810 con 0x7f518409e260 2024-09-11T01:29:06.037 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.020+0000 7f518a233640 5 monclient: authenticate success, global_id 4138 2024-09-11T01:29:06.037 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.020+0000 7f518a233640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:06.037 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.020+0000 7f518a233640 4 set_mon_vals no callback set 2024-09-11T01:29:06.037 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.020+0000 7f51827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:06.037 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.020+0000 7f51827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:06.038 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.020+0000 7f51827fc640 10 monclient: dump: 2024-09-11T01:29:06.038 INFO:teuthology.misc.health.smithi012.stderr:epoch 1 2024-09-11T01:29:06.038 INFO:teuthology.misc.health.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:06.038 INFO:teuthology.misc.health.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:06.038 INFO:teuthology.misc.health.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:06.038 INFO:teuthology.misc.health.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:06.038 INFO:teuthology.misc.health.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:06.038 INFO:teuthology.misc.health.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:06.038 INFO:teuthology.misc.health.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:06.038 INFO:teuthology.misc.health.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:06.038 INFO:teuthology.misc.health.smithi012.stderr: 2024-09-11T01:29:06.038 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.020+0000 7f518a233640 10 monclient: shutdown 2024-09-11T01:29:06.038 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.020+0000 7f518a233640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:06.038 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.020+0000 7f518a233640 1 -- 172.21.15.12:0/3905582858 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f518409e260 msgr2=0x7f518409bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:06.038 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.020+0000 7f518a233640 1 --2- 172.21.15.12:0/3905582858 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f518409e260 0x7f518409bb70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f518406d3a0 tx=0x7f516c029fb0 comp rx=0 tx=0).stop 2024-09-11T01:29:06.039 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.020+0000 7f518a233640 1 -- 172.21.15.12:0/3905582858 shutdown_connections 2024-09-11T01:29:06.039 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.020+0000 7f518a233640 1 --2- 172.21.15.12:0/3905582858 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f518409e260 0x7f518409bb70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:06.039 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.020+0000 7f518a233640 1 -- 172.21.15.12:0/3905582858 >> 172.21.15.12:0/3905582858 conn(0x7f51840a2fc0 msgr2=0x7f51840a33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:06.039 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.020+0000 7f518a233640 1 -- 172.21.15.12:0/3905582858 shutdown_connections 2024-09-11T01:29:06.039 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.020+0000 7f518a233640 1 -- 172.21.15.12:0/3905582858 wait complete. 2024-09-11T01:29:06.039 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.021+0000 7f518a233640 10 monclient: build_initial_monmap 2024-09-11T01:29:06.039 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.021+0000 7f518a233640 10 monclient: monmap: 2024-09-11T01:29:06.039 INFO:teuthology.misc.health.smithi012.stderr:epoch 0 2024-09-11T01:29:06.039 INFO:teuthology.misc.health.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:06.039 INFO:teuthology.misc.health.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:06.039 INFO:teuthology.misc.health.smithi012.stderr:created 0.000000 2024-09-11T01:29:06.039 INFO:teuthology.misc.health.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:06.039 INFO:teuthology.misc.health.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:06.039 INFO:teuthology.misc.health.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:06.040 INFO:teuthology.misc.health.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:06.040 INFO:teuthology.misc.health.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:06.040 INFO:teuthology.misc.health.smithi012.stderr: 2024-09-11T01:29:06.040 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.021+0000 7f518a233640 1 Processor -- start 2024-09-11T01:29:06.040 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f518a233640 1 -- start start 2024-09-11T01:29:06.040 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f518a233640 10 monclient: init 2024-09-11T01:29:06.040 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f518a233640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:06.040 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f518a233640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:29:06.040 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f518a233640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f518409e260 0x7f51841cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:06.040 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f518a233640 10 monclient(hunting): picked mon.noname-b con 0x7f518409e260 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:06.040 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f518a233640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f51840a1cf0 0x7f51841cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:06.040 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f518a233640 10 monclient(hunting): picked mon.noname-c con 0x7f51840a1cf0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:06.040 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f518a233640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f518413a470 0x7f518412f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:06.040 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f518a233640 10 monclient(hunting): picked mon.noname-a con 0x7f518413a470 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:06.041 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f518a233640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:06.041 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f518a233640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f51840a2c30 con 0x7f518413a470 2024-09-11T01:29:06.041 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f518a233640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:06.041 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f518a233640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f51840a2ab0 con 0x7f518409e260 2024-09-11T01:29:06.041 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f518a233640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:06.041 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f518a233640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f518409c0b0 con 0x7f51840a1cf0 2024-09-11T01:29:06.041 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f518a233640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:06.041 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f518a233640 10 monclient(hunting): authenticate will time out at 752.465553s 2024-09-11T01:29:06.041 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f5183fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f518413a470 0x7f518412f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:06.041 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f5182ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f51840a1cf0 0x7f51841cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:06.042 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f5183fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f518413a470 0x7f518412f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44222/0 (socket says 172.21.15.12:44222) 2024-09-11T01:29:06.042 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f5182ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f51840a1cf0 0x7f51841cddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51138/0 (socket says 172.21.15.12:51138) 2024-09-11T01:29:06.042 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f5183fff640 1 -- 172.21.15.12:0/3101844438 learned_addr learned my addr 172.21.15.12:0/3101844438 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:06.042 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f5183fff640 10 monclient(hunting): get_auth_request con 0x7f518413a470 auth_method 0 2024-09-11T01:29:06.042 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f5183fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:06.042 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f5183fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:06.043 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f5183fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:06.043 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f5182ffd640 10 monclient(hunting): get_auth_request con 0x7f51840a1cf0 auth_method 0 2024-09-11T01:29:06.043 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f5182ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:06.043 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f5182ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:06.043 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f5182ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:06.043 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f51837fe640 1 --2- 172.21.15.12:0/3101844438 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f518409e260 0x7f51841cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:06.044 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f51837fe640 10 monclient(hunting): get_auth_request con 0x7f518409e260 auth_method 0 2024-09-11T01:29:06.044 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f51837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:06.044 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f51837fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:06.044 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f51837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:06.044 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f5182ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:06.044 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f5182ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:06.045 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f5182ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:06.045 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f5183fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:06.045 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f5183fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:06.045 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f5183fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:06.045 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f51837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:06.046 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f51837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:06.046 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f51837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:06.046 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.022+0000 7f5183fff640 10 monclient(hunting): handle_auth_done global_id 4155 payload 824 2024-09-11T01:29:06.046 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.023+0000 7f5183fff640 1 -- 172.21.15.12:0/3101844438 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f51840a1cf0 msgr2=0x7f51841cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:06.046 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.023+0000 7f5183fff640 1 --2- 172.21.15.12:0/3101844438 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f51840a1cf0 0x7f51841cddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:06.046 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.023+0000 7f5183fff640 1 -- 172.21.15.12:0/3101844438 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f518409e260 msgr2=0x7f51841cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:06.046 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.023+0000 7f5183fff640 1 --2- 172.21.15.12:0/3101844438 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f518409e260 0x7f51841cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:06.047 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.023+0000 7f5183fff640 10 monclient: _finish_hunting 0 2024-09-11T01:29:06.047 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.023+0000 7f5183fff640 1 monclient: found mon.noname-a 2024-09-11T01:29:06.047 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.023+0000 7f5183fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:06.047 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.023+0000 7f5183fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:06.047 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.023+0000 7f5183fff640 1 -- 172.21.15.12:0/3101844438 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5184131300 con 0x7f518413a470 2024-09-11T01:29:06.048 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.023+0000 7f51837fe640 1 --2- 172.21.15.12:0/3101844438 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f518409e260 0x7f51841cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:06.048 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.023+0000 7f5182ffd640 1 --2- 172.21.15.12:0/3101844438 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f51840a1cf0 0x7f51841cddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:06.048 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.025+0000 7f5183fff640 1 -- 172.21.15.12:0/3101844438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f518413a470 msgr2=0x7f518412f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:06.048 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.025+0000 7f5183fff640 1 --2- 172.21.15.12:0/3101844438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f518413a470 0x7f518412f7b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f517c00caf0 tx=0x7f517c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:06.048 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.025+0000 7f517bfff640 1 -- 172.21.15.12:0/3101844438 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f517c012400 con 0x7f518413a470 2024-09-11T01:29:06.048 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.025+0000 7f517bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:06.048 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.025+0000 7f517bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:06.049 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.025+0000 7f517bfff640 10 monclient: dump: 2024-09-11T01:29:06.049 INFO:teuthology.misc.health.smithi012.stderr:epoch 1 2024-09-11T01:29:06.049 INFO:teuthology.misc.health.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:06.049 INFO:teuthology.misc.health.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:06.049 INFO:teuthology.misc.health.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:06.049 INFO:teuthology.misc.health.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:06.050 INFO:teuthology.misc.health.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:06.050 INFO:teuthology.misc.health.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:06.050 INFO:teuthology.misc.health.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:06.050 INFO:teuthology.misc.health.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:06.050 INFO:teuthology.misc.health.smithi012.stderr: 2024-09-11T01:29:06.050 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.025+0000 7f517bfff640 10 monclient: _finish_auth 0 2024-09-11T01:29:06.050 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.025+0000 7f517bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:29:06.051 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.025+0000 7f517bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:06.051 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.025+0000 7f517bfff640 1 -- 172.21.15.12:0/3101844438 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f517c0125c0 con 0x7f518413a470 2024-09-11T01:29:06.051 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.025+0000 7f518a233640 5 monclient: authenticate success, global_id 4155 2024-09-11T01:29:06.051 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.025+0000 7f518a233640 10 monclient: _renew_subs 2024-09-11T01:29:06.051 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.025+0000 7f518a233640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:06.051 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.025+0000 7f518a233640 1 -- 172.21.15.12:0/3101844438 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51841329c0 con 0x7f518413a470 2024-09-11T01:29:06.052 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.025+0000 7f517bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:06.052 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.025+0000 7f517bfff640 1 -- 172.21.15.12:0/3101844438 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f517c0129b0 con 0x7f518413a470 2024-09-11T01:29:06.052 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.025+0000 7f517bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:06.052 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.025+0000 7f517bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:06.052 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.025+0000 7f517bfff640 10 monclient: dump: 2024-09-11T01:29:06.052 INFO:teuthology.misc.health.smithi012.stderr:epoch 1 2024-09-11T01:29:06.053 INFO:teuthology.misc.health.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:06.053 INFO:teuthology.misc.health.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:06.053 INFO:teuthology.misc.health.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:06.053 INFO:teuthology.misc.health.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:06.053 INFO:teuthology.misc.health.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:06.054 INFO:teuthology.misc.health.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:06.054 INFO:teuthology.misc.health.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:06.054 INFO:teuthology.misc.health.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:06.054 INFO:teuthology.misc.health.smithi012.stderr: 2024-09-11T01:29:06.054 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.025+0000 7f5181ffb640 4 set_mon_vals no callback set 2024-09-11T01:29:06.054 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.025+0000 7f518a233640 10 monclient: _renew_subs 2024-09-11T01:29:06.054 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.025+0000 7f518a233640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:06.055 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.025+0000 7f518a233640 1 -- 172.21.15.12:0/3101844438 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51840a4510 con 0x7f518413a470 2024-09-11T01:29:06.055 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.025+0000 7f517bfff640 1 -- 172.21.15.12:0/3101844438 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f517c012bd0 con 0x7f518413a470 2024-09-11T01:29:06.055 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.025+0000 7f517bfff640 1 -- 172.21.15.12:0/3101844438 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f517c01b9f0 con 0x7f518413a470 2024-09-11T01:29:06.055 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.026+0000 7f518a233640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:06.055 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.026+0000 7f518a233640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:06.055 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.026+0000 7f518a233640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:06.056 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.026+0000 7f518a233640 1 -- 172.21.15.12:0/3101844438 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f518409a470 con 0x7f518413a470 2024-09-11T01:29:06.056 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.028+0000 7f5183fff640 1 --2- 172.21.15.12:0/3101844438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f518413a470 0x7f518412f7b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f517c00caf0 tx=0x7f517c00a060 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7f517c01b400 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) 2024-09-11T01:29:06.062 INFO:tasks.ceph.osd.2.smithi012.stderr:2024-09-11T01:29:06.059+0000 7f6d2d20d640 -1 Falling back to public interface 2024-09-11T01:29:06.306 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-09-11T01:29:06.303+0000 7f11f0e0c640 -1 Falling back to public interface 2024-09-11T01:29:06.345 INFO:tasks.ceph.osd.4.smithi071.stderr:2024-09-11T01:29:06.343+0000 7f476b00e640 -1 Falling back to public interface 2024-09-11T01:29:06.627 INFO:tasks.ceph.osd.5.smithi071.stderr:2024-09-11T01:29:06.625+0000 7fca7034b640 -1 Falling back to public interface 2024-09-11T01:29:06.805 INFO:tasks.ceph.osd.0.smithi012.stderr:2024-09-11T01:29:06.802+0000 7fc97305e640 -1 Falling back to public interface 2024-09-11T01:29:06.850 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.847+0000 7f517bfff640 1 -- 172.21.15.12:0/3101844438 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f517c01b400 con 0x7f518413a470 2024-09-11T01:29:06.850 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.847+0000 7f517bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:06.850 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.847+0000 7f517bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:06.858 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.856+0000 7f517bfff640 1 -- 172.21.15.12:0/3101844438 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 2) ==== 50324+0+0 (secure 0 0 0) 0x7f517c025480 con 0x7f518413a470 2024-09-11T01:29:06.918 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.915+0000 7f518a233640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-11T01:29:06.918 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.915+0000 7f518a233640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T01:29:06.919 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.915+0000 7f518a233640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:06.919 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.915+0000 7f518a233640 1 -- 172.21.15.12:0/3101844438 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5184131500 con 0x7f518413a470 2024-09-11T01:29:06.919 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.916+0000 7f517bfff640 1 -- 172.21.15.12:0/3101844438 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f517c01f5f0 con 0x7f518413a470 2024-09-11T01:29:06.919 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.916+0000 7f517bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T01:29:06.919 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.916+0000 7f517bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:06.919 INFO:teuthology.misc.health.smithi012.stdout: 2024-09-11T01:29:06.919 INFO:teuthology.misc.health.smithi012.stdout:{"epoch":8,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","created":"2024-09-11T01:29:02.540751+0000","modified":"2024-09-11T01:29:05.075851+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T01:29:05.075851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"616ba1d0-adf6-4fe1-85e3-266e53e21867","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9b570874-7a3d-4867-8c44-29084a8bf649","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b12ee288-ce57-41de-8b8e-6fad509311b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8e81aa4-0dee-487d-981d-1f593e363eee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9db77181-fc18-408f-a35c-87bf4b4ba2d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fadf2191-f31a-4880-a529-5b2d1023d236","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T01:29:06.920 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.917+0000 7f518a233640 10 monclient: shutdown 2024-09-11T01:29:06.920 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.917+0000 7f518a233640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:06.920 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.917+0000 7f518a233640 1 -- 172.21.15.12:0/3101844438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f518413a470 msgr2=0x7f518412f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:06.920 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.917+0000 7f518a233640 1 --2- 172.21.15.12:0/3101844438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f518413a470 0x7f518412f7b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f517c00caf0 tx=0x7f517c00a060 comp rx=0 tx=0).stop 2024-09-11T01:29:06.921 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.917+0000 7f518a233640 1 -- 172.21.15.12:0/3101844438 shutdown_connections 2024-09-11T01:29:06.921 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.917+0000 7f518a233640 1 --2- 172.21.15.12:0/3101844438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f518413a470 0x7f518412f7b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:06.921 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.917+0000 7f518a233640 1 --2- 172.21.15.12:0/3101844438 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f51840a1cf0 0x7f51841cddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:06.921 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.917+0000 7f518a233640 1 --2- 172.21.15.12:0/3101844438 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f518409e260 0x7f51841cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:06.921 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.917+0000 7f518a233640 1 -- 172.21.15.12:0/3101844438 >> 172.21.15.12:0/3101844438 conn(0x7f51840a2fc0 msgr2=0x7f5184139b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:06.921 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.917+0000 7f518a233640 1 -- 172.21.15.12:0/3101844438 shutdown_connections 2024-09-11T01:29:06.921 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:06.917+0000 7f518a233640 1 -- 172.21.15.12:0/3101844438 wait complete. 2024-09-11T01:29:06.930 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-11T01:29:07.051 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-09-11T01:29:07.049+0000 7f0d6d810640 -1 Falling back to public interface 2024-09-11T01:29:07.862 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-11T01:29:07.860+0000 7f64fa64e640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-11T01:29:07.866 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-09-11T01:29:07.864+0000 7f64fa64e640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-11T01:29:09.593 INFO:tasks.ceph.osd.2.smithi012.stderr:2024-09-11T01:29:09.591+0000 7f6d2d20d640 -1 osd.2 0 log_to_monitors true 2024-09-11T01:29:10.114 INFO:tasks.ceph.osd.4.smithi071.stderr:2024-09-11T01:29:10.112+0000 7f476b00e640 -1 osd.4 0 log_to_monitors true 2024-09-11T01:29:10.168 INFO:tasks.ceph.osd.5.smithi071.stderr:2024-09-11T01:29:10.166+0000 7fca7034b640 -1 osd.5 0 log_to_monitors true 2024-09-11T01:29:10.334 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-09-11T01:29:10.331+0000 7f11f0e0c640 -1 osd.1 0 log_to_monitors true 2024-09-11T01:29:10.554 INFO:tasks.ceph.osd.0.smithi012.stderr:2024-09-11T01:29:10.551+0000 7fc97305e640 -1 osd.0 0 log_to_monitors true 2024-09-11T01:29:10.785 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-09-11T01:29:10.783+0000 7f0d6d810640 -1 osd.3 0 log_to_monitors true 2024-09-11T01:29:11.932 INFO:tasks.ceph.osd.2.smithi012.stderr:2024-09-11T01:29:11.930+0000 7f6d293a6640 -1 osd.2 0 waiting for initial osdmap 2024-09-11T01:29:11.945 INFO:tasks.ceph.osd.2.smithi012.stderr:2024-09-11T01:29:11.944+0000 7f6d209b1640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T01:29:12.320 INFO:tasks.ceph.osd.0.smithi012.stderr:2024-09-11T01:29:12.318+0000 7fc96efed640 -1 osd.0 0 waiting for initial osdmap 2024-09-11T01:29:12.321 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-09-11T01:29:12.318+0000 7f11ec6c9640 -1 osd.1 0 waiting for initial osdmap 2024-09-11T01:29:12.325 INFO:tasks.ceph.osd.0.smithi012.stderr:2024-09-11T01:29:12.322+0000 7fc96ae14640 -1 osd.0 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T01:29:12.325 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-09-11T01:29:12.322+0000 7f11e84f0640 -1 osd.1 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T01:29:13.189 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-09-11T01:29:13.187+0000 7f0d699ed640 -1 osd.3 0 waiting for initial osdmap 2024-09-11T01:29:13.205 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-09-11T01:29:13.203+0000 7f0d60ff8640 -1 osd.3 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T01:29:13.217 INFO:tasks.ceph.osd.4.smithi071.stderr:2024-09-11T01:29:13.215+0000 7f47668fe640 -1 osd.4 0 waiting for initial osdmap 2024-09-11T01:29:13.226 INFO:tasks.ceph.osd.4.smithi071.stderr:2024-09-11T01:29:13.225+0000 7f475e71d640 -1 osd.4 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T01:29:13.230 INFO:tasks.ceph.osd.5.smithi071.stderr:2024-09-11T01:29:13.228+0000 7fca6c2da640 -1 osd.5 0 waiting for initial osdmap 2024-09-11T01:29:13.238 INFO:tasks.ceph.osd.5.smithi071.stderr:2024-09-11T01:29:13.237+0000 7fca640f9640 -1 osd.5 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T01:29:13.535 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T01:29:13.638 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.636+0000 7f1b2a6b1640 10 monclient: get_monmap_and_config 2024-09-11T01:29:13.638 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.636+0000 7f1b2a6b1640 10 monclient: build_initial_monmap 2024-09-11T01:29:13.639 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.636+0000 7f1b2a6b1640 10 monclient: monmap: 2024-09-11T01:29:13.639 INFO:teuthology.misc.health.smithi012.stderr:epoch 0 2024-09-11T01:29:13.639 INFO:teuthology.misc.health.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:13.639 INFO:teuthology.misc.health.smithi012.stderr:last_changed 2024-09-11T01:29:13.637921+0000 2024-09-11T01:29:13.639 INFO:teuthology.misc.health.smithi012.stderr:created 2024-09-11T01:29:13.637921+0000 2024-09-11T01:29:13.639 INFO:teuthology.misc.health.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:13.639 INFO:teuthology.misc.health.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:13.639 INFO:teuthology.misc.health.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:13.639 INFO:teuthology.misc.health.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:13.639 INFO:teuthology.misc.health.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:13.639 INFO:teuthology.misc.health.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:13.639 INFO:teuthology.misc.health.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:13.639 INFO:teuthology.misc.health.smithi012.stderr: 2024-09-11T01:29:13.642 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.636+0000 7f1b2a6b1640 1 Processor -- start 2024-09-11T01:29:13.642 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b2a6b1640 1 -- start start 2024-09-11T01:29:13.642 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b2a6b1640 10 monclient: init 2024-09-11T01:29:13.642 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b2a6b1640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:13.642 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b2a6b1640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-11T01:29:13.642 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b2a6b1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1b241380f0 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:13.642 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b2a6b1640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1b24074380 0x7f1b2413cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:13.642 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b2a6b1640 10 monclient(hunting): picked mon.noname-a con 0x7f1b24074380 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:13.642 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b2a6b1640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1b2413d520 0x7f1b2413f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:13.642 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b2a6b1640 10 monclient(hunting): picked mon.noname-c con 0x7f1b2413d520 addr v2:172.21.15.71:3300/0 2024-09-11T01:29:13.642 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b2a6b1640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b24138be0 con 0x7f1b241380f0 2024-09-11T01:29:13.643 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b2a6b1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:13.643 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b2a6b1640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f1b240a6080 con 0x7f1b24074380 2024-09-11T01:29:13.643 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b2a6b1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:13.643 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b2a6b1640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f1b24073120 con 0x7f1b2413d520 2024-09-11T01:29:13.643 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b2a6b1640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:13.643 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b2a6b1640 10 monclient(hunting): authenticate will time out at 760.080636s 2024-09-11T01:29:13.643 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b237fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1b24074380 0x7f1b2413cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:13.643 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b237fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1b24074380 0x7f1b2413cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38216/0 (socket says 172.21.15.12:38216) 2024-09-11T01:29:13.643 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b237fe640 1 -- 172.21.15.12:0/3585783512 learned_addr learned my addr 172.21.15.12:0/3585783512 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:13.643 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b237fe640 10 monclient(hunting): get_auth_request con 0x7f1b24074380 auth_method 0 2024-09-11T01:29:13.643 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:13.643 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b237fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:13.643 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:13.644 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:13.644 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:13.644 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:13.644 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.637+0000 7f1b28c4e640 1 --2- 172.21.15.12:0/3585783512 >> v2:172.21.15.71:3300/0 conn(0x7f1b2413d520 0x7f1b2413f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:13.644 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.638+0000 7f1b28c4e640 10 monclient(hunting): get_auth_request con 0x7f1b2413d520 auth_method 0 2024-09-11T01:29:13.644 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.638+0000 7f1b28c4e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:13.644 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.638+0000 7f1b28c4e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:13.644 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.638+0000 7f1b28c4e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:13.644 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.638+0000 7f1b237fe640 10 monclient(hunting): handle_auth_done global_id 4188 payload 274 2024-09-11T01:29:13.644 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.638+0000 7f1b237fe640 1 -- 172.21.15.12:0/3585783512 >> v2:172.21.15.71:3300/0 conn(0x7f1b2413d520 msgr2=0x7f1b2413f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:13.644 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.638+0000 7f1b237fe640 1 --2- 172.21.15.12:0/3585783512 >> v2:172.21.15.71:3300/0 conn(0x7f1b2413d520 0x7f1b2413f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:13.644 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.638+0000 7f1b237fe640 1 -- 172.21.15.12:0/3585783512 >> v1:172.21.15.12:6789/0 conn(0x7f1b241380f0 legacy=0x7f1b241384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:13.644 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.638+0000 7f1b237fe640 10 monclient: _finish_hunting 0 2024-09-11T01:29:13.645 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.638+0000 7f1b237fe640 1 monclient: found mon.noname-a 2024-09-11T01:29:13.645 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.638+0000 7f1b237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:13.645 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.638+0000 7f1b237fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:13.645 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.638+0000 7f1b237fe640 1 -- 172.21.15.12:0/3585783512 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b240739a0 con 0x7f1b24074380 2024-09-11T01:29:13.645 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.638+0000 7f1b28c4e640 1 --2- 172.21.15.12:0/3585783512 >> v2:172.21.15.71:3300/0 conn(0x7f1b2413d520 0x7f1b2413f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:29:13.651 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.640+0000 7f1b237fe640 1 -- 172.21.15.12:0/3585783512 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b24074380 msgr2=0x7f1b2413cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:13.651 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.640+0000 7f1b237fe640 1 --2- 172.21.15.12:0/3585783512 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b24074380 0x7f1b2413cfe0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1b24077350 tx=0x7f1b1802a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdbc6250bec7b4d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:13.651 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.641+0000 7f1b22ffd640 1 -- 172.21.15.12:0/3585783512 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b18037070 con 0x7f1b24074380 2024-09-11T01:29:13.651 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.641+0000 7f1b22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:13.651 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.641+0000 7f1b22ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:13.651 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.641+0000 7f1b22ffd640 10 monclient: dump: 2024-09-11T01:29:13.651 INFO:teuthology.misc.health.smithi012.stderr:epoch 1 2024-09-11T01:29:13.651 INFO:teuthology.misc.health.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:13.652 INFO:teuthology.misc.health.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:13.652 INFO:teuthology.misc.health.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:13.652 INFO:teuthology.misc.health.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:13.652 INFO:teuthology.misc.health.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:13.652 INFO:teuthology.misc.health.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:13.652 INFO:teuthology.misc.health.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:13.652 INFO:teuthology.misc.health.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:13.652 INFO:teuthology.misc.health.smithi012.stderr: 2024-09-11T01:29:13.652 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.641+0000 7f1b22ffd640 10 monclient: _finish_auth 0 2024-09-11T01:29:13.652 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.641+0000 7f1b22ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:13.652 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.641+0000 7f1b22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:13.652 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.641+0000 7f1b22ffd640 1 -- 172.21.15.12:0/3585783512 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b18032640 con 0x7f1b24074380 2024-09-11T01:29:13.652 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.641+0000 7f1b2a6b1640 5 monclient: authenticate success, global_id 4188 2024-09-11T01:29:13.653 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.641+0000 7f1b22ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:13.653 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.641+0000 7f1b2a6b1640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:13.653 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.641+0000 7f1b22ffd640 1 -- 172.21.15.12:0/3585783512 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b18032a10 con 0x7f1b24074380 2024-09-11T01:29:13.653 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.641+0000 7f1b2a6b1640 4 set_mon_vals no callback set 2024-09-11T01:29:13.653 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.641+0000 7f1b22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:13.653 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.641+0000 7f1b22ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:13.653 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.641+0000 7f1b22ffd640 10 monclient: dump: 2024-09-11T01:29:13.653 INFO:teuthology.misc.health.smithi012.stderr:epoch 1 2024-09-11T01:29:13.653 INFO:teuthology.misc.health.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:13.653 INFO:teuthology.misc.health.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:13.653 INFO:teuthology.misc.health.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:13.653 INFO:teuthology.misc.health.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:13.653 INFO:teuthology.misc.health.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:13.653 INFO:teuthology.misc.health.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:13.654 INFO:teuthology.misc.health.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:13.654 INFO:teuthology.misc.health.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:13.654 INFO:teuthology.misc.health.smithi012.stderr: 2024-09-11T01:29:13.654 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.641+0000 7f1b2a6b1640 10 monclient: shutdown 2024-09-11T01:29:13.654 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.641+0000 7f1b2a6b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:13.654 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.641+0000 7f1b2a6b1640 1 -- 172.21.15.12:0/3585783512 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b24074380 msgr2=0x7f1b2413cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:13.654 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.641+0000 7f1b2a6b1640 1 --2- 172.21.15.12:0/3585783512 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b24074380 0x7f1b2413cfe0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1b24077350 tx=0x7f1b1802a0b0 comp rx=0 tx=0).stop 2024-09-11T01:29:13.654 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.641+0000 7f1b2a6b1640 1 -- 172.21.15.12:0/3585783512 shutdown_connections 2024-09-11T01:29:13.654 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.641+0000 7f1b2a6b1640 1 --2- 172.21.15.12:0/3585783512 >> v2:172.21.15.71:3300/0 conn(0x7f1b2413d520 0x7f1b2413f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:13.654 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.641+0000 7f1b2a6b1640 1 --2- 172.21.15.12:0/3585783512 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b24074380 0x7f1b2413cfe0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:13.654 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.641+0000 7f1b2a6b1640 1 -- 172.21.15.12:0/3585783512 >> 172.21.15.12:0/3585783512 conn(0x7f1b240a70e0 msgr2=0x7f1b240a74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:13.654 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.641+0000 7f1b2a6b1640 1 -- 172.21.15.12:0/3585783512 shutdown_connections 2024-09-11T01:29:13.654 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.641+0000 7f1b2a6b1640 1 -- 172.21.15.12:0/3585783512 wait complete. 2024-09-11T01:29:13.655 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.642+0000 7f1b2a6b1640 10 monclient: build_initial_monmap 2024-09-11T01:29:13.655 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.642+0000 7f1b2a6b1640 10 monclient: monmap: 2024-09-11T01:29:13.655 INFO:teuthology.misc.health.smithi012.stderr:epoch 0 2024-09-11T01:29:13.655 INFO:teuthology.misc.health.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:13.655 INFO:teuthology.misc.health.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:13.655 INFO:teuthology.misc.health.smithi012.stderr:created 0.000000 2024-09-11T01:29:13.655 INFO:teuthology.misc.health.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:13.655 INFO:teuthology.misc.health.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:13.655 INFO:teuthology.misc.health.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:13.655 INFO:teuthology.misc.health.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:13.655 INFO:teuthology.misc.health.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:13.655 INFO:teuthology.misc.health.smithi012.stderr: 2024-09-11T01:29:13.655 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.642+0000 7f1b2a6b1640 1 Processor -- start 2024-09-11T01:29:13.655 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b2a6b1640 1 -- start start 2024-09-11T01:29:13.656 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b2a6b1640 10 monclient: init 2024-09-11T01:29:13.656 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b2a6b1640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:13.656 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b2a6b1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:29:13.656 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b2a6b1640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b24074380 0x7f1b241ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:13.656 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b2a6b1640 10 monclient(hunting): picked mon.noname-c con 0x7f1b24074380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:13.656 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b2a6b1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b241380f0 0x7f1b24137180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:13.656 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b2a6b1640 10 monclient(hunting): picked mon.noname-a con 0x7f1b241380f0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:13.656 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b2a6b1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b2413d520 0x7f1b24139e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:13.656 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b2a6b1640 10 monclient(hunting): picked mon.noname-b con 0x7f1b2413d520 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:13.656 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b2a6b1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:13.657 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b2a6b1640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1b2409dc70 con 0x7f1b241380f0 2024-09-11T01:29:13.657 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b2a6b1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:13.657 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b2a6b1640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1b24074790 con 0x7f1b2413d520 2024-09-11T01:29:13.657 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b2a6b1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:13.657 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b2a6b1640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1b2409ddf0 con 0x7f1b24074380 2024-09-11T01:29:13.657 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b2a6b1640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:13.657 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b2a6b1640 10 monclient(hunting): authenticate will time out at 760.086579s 2024-09-11T01:29:13.657 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b237fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b241380f0 0x7f1b24137180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:13.657 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b23fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b24074380 0x7f1b241ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:13.657 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b237fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b241380f0 0x7f1b24137180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38230/0 (socket says 172.21.15.12:38230) 2024-09-11T01:29:13.657 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b237fe640 1 -- 172.21.15.12:0/3424676467 learned_addr learned my addr 172.21.15.12:0/3424676467 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:13.657 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b237fe640 10 monclient(hunting): get_auth_request con 0x7f1b241380f0 auth_method 0 2024-09-11T01:29:13.657 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:13.658 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b237fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:13.658 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:13.658 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b23fff640 10 monclient(hunting): get_auth_request con 0x7f1b24074380 auth_method 0 2024-09-11T01:29:13.658 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:13.658 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b23fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:13.658 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:13.658 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b28c4e640 1 --2- 172.21.15.12:0/3424676467 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b2413d520 0x7f1b24139e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:13.659 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:13.659 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:13.660 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:13.660 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b28c4e640 10 monclient(hunting): get_auth_request con 0x7f1b2413d520 auth_method 0 2024-09-11T01:29:13.660 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b28c4e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:13.660 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b28c4e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:13.660 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b28c4e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:13.660 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:13.660 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:13.660 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:13.660 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b28c4e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:13.660 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b28c4e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:13.660 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b28c4e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:13.660 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.643+0000 7f1b237fe640 10 monclient(hunting): handle_auth_done global_id 4191 payload 824 2024-09-11T01:29:13.660 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.644+0000 7f1b237fe640 1 -- 172.21.15.12:0/3424676467 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b24074380 msgr2=0x7f1b241ccfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:13.661 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.644+0000 7f1b237fe640 1 --2- 172.21.15.12:0/3424676467 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b24074380 0x7f1b241ccfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:13.661 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.644+0000 7f1b237fe640 1 -- 172.21.15.12:0/3424676467 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b2413d520 msgr2=0x7f1b24139e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:13.661 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.644+0000 7f1b237fe640 1 --2- 172.21.15.12:0/3424676467 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b2413d520 0x7f1b24139e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:13.661 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.644+0000 7f1b237fe640 10 monclient: _finish_hunting 0 2024-09-11T01:29:13.661 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.644+0000 7f1b237fe640 1 monclient: found mon.noname-a 2024-09-11T01:29:13.661 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.644+0000 7f1b237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:13.661 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.644+0000 7f1b237fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:13.661 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.644+0000 7f1b237fe640 1 -- 172.21.15.12:0/3424676467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b24120c50 con 0x7f1b241380f0 2024-09-11T01:29:13.661 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.646+0000 7f1b237fe640 1 -- 172.21.15.12:0/3424676467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b241380f0 msgr2=0x7f1b24137180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:13.661 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.646+0000 7f1b237fe640 1 --2- 172.21.15.12:0/3424676467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b241380f0 0x7f1b24137180 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1b180090b0 tx=0x7f1b180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:13.661 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.646+0000 7f1b20ff9640 1 -- 172.21.15.12:0/3424676467 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b18041020 con 0x7f1b241380f0 2024-09-11T01:29:13.661 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.646+0000 7f1b20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:13.661 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.646+0000 7f1b20ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:13.662 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.646+0000 7f1b20ff9640 10 monclient: dump: 2024-09-11T01:29:13.662 INFO:teuthology.misc.health.smithi012.stderr:epoch 1 2024-09-11T01:29:13.662 INFO:teuthology.misc.health.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:13.662 INFO:teuthology.misc.health.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:13.662 INFO:teuthology.misc.health.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:13.662 INFO:teuthology.misc.health.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:13.662 INFO:teuthology.misc.health.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:13.662 INFO:teuthology.misc.health.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:13.662 INFO:teuthology.misc.health.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:13.662 INFO:teuthology.misc.health.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:13.662 INFO:teuthology.misc.health.smithi012.stderr: 2024-09-11T01:29:13.662 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.646+0000 7f1b20ff9640 10 monclient: _finish_auth 0 2024-09-11T01:29:13.662 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.646+0000 7f1b20ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:29:13.662 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.646+0000 7f1b20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:13.663 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.646+0000 7f1b2a6b1640 5 monclient: authenticate success, global_id 4191 2024-09-11T01:29:13.663 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.646+0000 7f1b20ff9640 1 -- 172.21.15.12:0/3424676467 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b1803d910 con 0x7f1b241380f0 2024-09-11T01:29:13.663 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.646+0000 7f1b20ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:13.663 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.646+0000 7f1b2a6b1640 10 monclient: _renew_subs 2024-09-11T01:29:13.663 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.646+0000 7f1b20ff9640 1 -- 172.21.15.12:0/3424676467 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b18037040 con 0x7f1b241380f0 2024-09-11T01:29:13.663 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.646+0000 7f1b2a6b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:13.663 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.646+0000 7f1b2a6b1640 1 -- 172.21.15.12:0/3424676467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b24121b10 con 0x7f1b241380f0 2024-09-11T01:29:13.663 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.646+0000 7f1b20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:13.663 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.646+0000 7f1b20ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:13.663 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.646+0000 7f1b20ff9640 10 monclient: dump: 2024-09-11T01:29:13.663 INFO:teuthology.misc.health.smithi012.stderr:epoch 1 2024-09-11T01:29:13.663 INFO:teuthology.misc.health.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:13.663 INFO:teuthology.misc.health.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:13.663 INFO:teuthology.misc.health.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:13.664 INFO:teuthology.misc.health.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:13.664 INFO:teuthology.misc.health.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:13.664 INFO:teuthology.misc.health.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:13.664 INFO:teuthology.misc.health.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:13.664 INFO:teuthology.misc.health.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:13.664 INFO:teuthology.misc.health.smithi012.stderr: 2024-09-11T01:29:13.664 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.646+0000 7f1b227fc640 4 set_mon_vals no callback set 2024-09-11T01:29:13.664 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.646+0000 7f1b2a6b1640 10 monclient: _renew_subs 2024-09-11T01:29:13.664 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.646+0000 7f1b2a6b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:13.664 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.646+0000 7f1b2a6b1640 1 -- 172.21.15.12:0/3424676467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b2413b910 con 0x7f1b241380f0 2024-09-11T01:29:13.664 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.647+0000 7f1b20ff9640 1 -- 172.21.15.12:0/3424676467 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 4) ==== 50557+0+0 (secure 0 0 0) 0x7f1b1803a030 con 0x7f1b241380f0 2024-09-11T01:29:13.664 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.647+0000 7f1b2a6b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:13.664 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.647+0000 7f1b2a6b1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:13.664 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.647+0000 7f1b2a6b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:13.665 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.647+0000 7f1b2a6b1640 1 -- 172.21.15.12:0/3424676467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b241384d0 con 0x7f1b241380f0 2024-09-11T01:29:13.665 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.647+0000 7f1b20ff9640 1 --2- 172.21.15.12:0/3424676467 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1b1003e180 0x7f1b10040640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:13.665 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.647+0000 7f1b20ff9640 1 -- 172.21.15.12:0/3424676467 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 3607+0+0 (secure 0 0 0) 0x7f1b18072dc0 con 0x7f1b241380f0 2024-09-11T01:29:13.665 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.648+0000 7f1b23fff640 1 --2- 172.21.15.12:0/3424676467 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1b1003e180 0x7f1b10040640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:13.665 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.648+0000 7f1b23fff640 10 monclient: get_auth_request con 0x7f1b1003e180 auth_method 0 2024-09-11T01:29:13.665 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.650+0000 7f1b23fff640 1 --2- 172.21.15.12:0/3424676467 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1b1003e180 0x7f1b10040640 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1b14004f60 tx=0x7f1b14007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:13.666 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.652+0000 7f1b20ff9640 1 -- 172.21.15.12:0/3424676467 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1b1804b030 con 0x7f1b241380f0 2024-09-11T01:29:13.666 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.652+0000 7f1b20ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:13.666 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.652+0000 7f1b20ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:13.840 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.837+0000 7f1b2a6b1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-11T01:29:13.840 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.837+0000 7f1b2a6b1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T01:29:13.840 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.837+0000 7f1b2a6b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:13.840 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.837+0000 7f1b2a6b1640 1 -- 172.21.15.12:0/3424676467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1b24121630 con 0x7f1b241380f0 2024-09-11T01:29:13.841 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.838+0000 7f1b20ff9640 1 -- 172.21.15.12:0/3424676467 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+8496 (secure 0 0 0) 0x7f1b1803a350 con 0x7f1b241380f0 2024-09-11T01:29:13.841 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.838+0000 7f1b20ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T01:29:13.841 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.838+0000 7f1b20ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:13.841 INFO:teuthology.misc.health.smithi012.stdout: 2024-09-11T01:29:13.841 INFO:teuthology.misc.health.smithi012.stdout:{"epoch":12,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","created":"2024-09-11T01:29:02.540751+0000","modified":"2024-09-11T01:29:13.320573+0000","last_up_change":"2024-09-11T01:29:13.320573+0000","last_in_change":"2024-09-11T01:29:05.075851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"616ba1d0-adf6-4fe1-85e3-266e53e21867","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":4266003406},{"type":"v1","addr":"172.21.15.12:6817","nonce":4266003406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":4266003406},{"type":"v1","addr":"172.21.15.12:6819","nonce":4266003406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":4266003406},{"type":"v1","addr":"172.21.15.12:6823","nonce":4266003406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":4266003406},{"type":"v1","addr":"172.21.15.12:6821","nonce":4266003406}]},"public_addr":"172.21.15.12:6817/4266003406","cluster_addr":"172.21.15.12:6819/4266003406","heartbeat_back_addr":"172.21.15.12:6823/4266003406","heartbeat_front_addr":"172.21.15.12:6821/4266003406","state":["exists","up"]},{"osd":1,"uuid":"9b570874-7a3d-4867-8c44-29084a8bf649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3563357280},{"type":"v1","addr":"172.21.15.12:6809","nonce":3563357280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3563357280},{"type":"v1","addr":"172.21.15.12:6811","nonce":3563357280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3563357280},{"type":"v1","addr":"172.21.15.12:6815","nonce":3563357280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3563357280},{"type":"v1","addr":"172.21.15.12:6813","nonce":3563357280}]},"public_addr":"172.21.15.12:6809/3563357280","cluster_addr":"172.21.15.12:6811/3563357280","heartbeat_back_addr":"172.21.15.12:6815/3563357280","heartbeat_front_addr":"172.21.15.12:6813/3563357280","state":["exists","up"]},{"osd":2,"uuid":"b12ee288-ce57-41de-8b8e-6fad509311b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2893036867},{"type":"v1","addr":"172.21.15.12:6801","nonce":2893036867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2893036867},{"type":"v1","addr":"172.21.15.12:6803","nonce":2893036867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2893036867},{"type":"v1","addr":"172.21.15.12:6807","nonce":2893036867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2893036867},{"type":"v1","addr":"172.21.15.12:6805","nonce":2893036867}]},"public_addr":"172.21.15.12:6801/2893036867","cluster_addr":"172.21.15.12:6803/2893036867","heartbeat_back_addr":"172.21.15.12:6807/2893036867","heartbeat_front_addr":"172.21.15.12:6805/2893036867","state":["exists","up"]},{"osd":3,"uuid":"a8e81aa4-0dee-487d-981d-1f593e363eee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":3091488141},{"type":"v1","addr":"172.21.15.71:6819","nonce":3091488141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":3091488141},{"type":"v1","addr":"172.21.15.71:6821","nonce":3091488141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":3091488141},{"type":"v1","addr":"172.21.15.71:6825","nonce":3091488141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":3091488141},{"type":"v1","addr":"172.21.15.71:6823","nonce":3091488141}]},"public_addr":"172.21.15.71:6819/3091488141","cluster_addr":"172.21.15.71:6821/3091488141","heartbeat_back_addr":"172.21.15.71:6825/3091488141","heartbeat_front_addr":"172.21.15.71:6823/3091488141","state":["exists","up"]},{"osd":4,"uuid":"9db77181-fc18-408f-a35c-87bf4b4ba2d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":1790415427},{"type":"v1","addr":"172.21.15.71:6801","nonce":1790415427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":1790415427},{"type":"v1","addr":"172.21.15.71:6803","nonce":1790415427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":1790415427},{"type":"v1","addr":"172.21.15.71:6807","nonce":1790415427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":1790415427},{"type":"v1","addr":"172.21.15.71:6805","nonce":1790415427}]},"public_addr":"172.21.15.71:6801/1790415427","cluster_addr":"172.21.15.71:6803/1790415427","heartbeat_back_addr":"172.21.15.71:6807/1790415427","heartbeat_front_addr":"172.21.15.71:6805/1790415427","state":["exists","up"]},{"osd":5,"uuid":"fadf2191-f31a-4880-a529-5b2d1023d236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":2032569317},{"type":"v1","addr":"172.21.15.71:6809","nonce":2032569317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":2032569317},{"type":"v1","addr":"172.21.15.71:6811","nonce":2032569317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":2032569317},{"type":"v1","addr":"172.21.15.71:6815","nonce":2032569317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":2032569317},{"type":"v1","addr":"172.21.15.71:6813","nonce":2032569317}]},"public_addr":"172.21.15.71:6809/2032569317","cluster_addr":"172.21.15.71:6811/2032569317","heartbeat_back_addr":"172.21.15.71:6815/2032569317","heartbeat_front_addr":"172.21.15.71:6813/2032569317","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:10.578802+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T01:29:13.842 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.840+0000 7f1b2a6b1640 1 -- 172.21.15.12:0/3424676467 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1b1003e180 msgr2=0x7f1b10040640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:13.843 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.840+0000 7f1b2a6b1640 1 --2- 172.21.15.12:0/3424676467 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1b1003e180 0x7f1b10040640 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1b14004f60 tx=0x7f1b14007e30 comp rx=0 tx=0).stop 2024-09-11T01:29:13.843 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.840+0000 7f1b2a6b1640 10 monclient: shutdown 2024-09-11T01:29:13.843 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.840+0000 7f1b2a6b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:13.843 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.840+0000 7f1b2a6b1640 1 -- 172.21.15.12:0/3424676467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b241380f0 msgr2=0x7f1b24137180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:13.843 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.840+0000 7f1b2a6b1640 1 --2- 172.21.15.12:0/3424676467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b241380f0 0x7f1b24137180 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1b180090b0 tx=0x7f1b180046c0 comp rx=0 tx=0).stop 2024-09-11T01:29:13.843 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.840+0000 7f1b2a6b1640 1 -- 172.21.15.12:0/3424676467 shutdown_connections 2024-09-11T01:29:13.843 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.840+0000 7f1b2a6b1640 1 --2- 172.21.15.12:0/3424676467 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b2413d520 0x7f1b24139e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:13.843 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.840+0000 7f1b2a6b1640 1 --2- 172.21.15.12:0/3424676467 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1b1003e180 0x7f1b10040640 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:13.843 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.840+0000 7f1b2a6b1640 1 --2- 172.21.15.12:0/3424676467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b241380f0 0x7f1b24137180 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:13.843 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.840+0000 7f1b2a6b1640 1 --2- 172.21.15.12:0/3424676467 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b24074380 0x7f1b241ccfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:13.843 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.840+0000 7f1b2a6b1640 1 -- 172.21.15.12:0/3424676467 >> 172.21.15.12:0/3424676467 conn(0x7f1b240a70e0 msgr2=0x7f1b240a39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:13.843 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.840+0000 7f1b2a6b1640 1 -- 172.21.15.12:0/3424676467 shutdown_connections 2024-09-11T01:29:13.844 INFO:teuthology.misc.health.smithi012.stderr:2024-09-11T01:29:13.840+0000 7f1b2a6b1640 1 -- 172.21.15.12:0/3424676467 wait complete. 2024-09-11T01:29:13.852 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-11T01:29:13.852 INFO:tasks.ceph:Creating RBD pool 2024-09-11T01:29:13.852 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-11T01:29:13.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.951+0000 7f3682b4b640 10 monclient: get_monmap_and_config 2024-09-11T01:29:13.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.951+0000 7f3682b4b640 10 monclient: build_initial_monmap 2024-09-11T01:29:13.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.951+0000 7f3682b4b640 10 monclient: monmap: 2024-09-11T01:29:13.953 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:13.953 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:13.953 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:13.952356+0000 2024-09-11T01:29:13.953 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:13.952356+0000 2024-09-11T01:29:13.954 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:13.954 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:13.954 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:13.954 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:13.954 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:13.954 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:13.954 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:13.954 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:13.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.951+0000 7f3682b4b640 1 Processor -- start 2024-09-11T01:29:13.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.951+0000 7f3682b4b640 1 -- start start 2024-09-11T01:29:13.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.951+0000 7f3682b4b640 10 monclient: init 2024-09-11T01:29:13.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.951+0000 7f3682b4b640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:13.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.951+0000 7f3682b4b640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-11T01:29:13.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.951+0000 7f3682b4b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f367c0a5020 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:13.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.951+0000 7f3682b4b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f367c0a5400 0x7f367c0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:13.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.951+0000 7f3682b4b640 10 monclient(hunting): picked mon.noname-b con 0x7f367c0a5400 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:13.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f3682b4b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f367c13d2a0 0x7f367c13f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:13.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f3682b4b640 10 monclient(hunting): picked mon.noname-a con 0x7f367c13d2a0 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:13.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f3682b4b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f367c0a6e60 con 0x7f367c0a5020 2024-09-11T01:29:13.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f3682b4b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:13.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f3682b4b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f367c09ba20 con 0x7f367c13d2a0 2024-09-11T01:29:13.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f3682b4b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:13.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f3682b4b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f367c0a66d0 con 0x7f367c0a5400 2024-09-11T01:29:13.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f3682b4b640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:13.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f3682b4b640 10 monclient(hunting): authenticate will time out at 760.395065s 2024-09-11T01:29:13.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f3673fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f367c0a5400 0x7f367c0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:13.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f36810e8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f367c13d2a0 0x7f367c13f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:13.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f36810e8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f367c13d2a0 0x7f367c13f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38244/0 (socket says 172.21.15.12:38244) 2024-09-11T01:29:13.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f3673fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f367c0a5400 0x7f367c0a5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:33842/0 (socket says 172.21.15.12:33842) 2024-09-11T01:29:13.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f36810e8640 1 -- 172.21.15.12:0/440846301 learned_addr learned my addr 172.21.15.12:0/440846301 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:13.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f36810e8640 10 monclient(hunting): get_auth_request con 0x7f367c13d2a0 auth_method 0 2024-09-11T01:29:13.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f36810e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:13.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f36810e8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:13.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f36810e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:13.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f3673fff640 10 monclient(hunting): get_auth_request con 0x7f367c0a5400 auth_method 0 2024-09-11T01:29:13.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f3673fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:13.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f3673fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:13.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f3673fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:13.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f36810e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:13.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f36810e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:13.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f36810e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:13.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f3673fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:13.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f3673fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:13.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f3673fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:13.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f36810e8640 10 monclient(hunting): handle_auth_done global_id 4194 payload 274 2024-09-11T01:29:13.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f36810e8640 1 -- 172.21.15.12:0/440846301 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f367c0a5400 msgr2=0x7f367c0a5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:13.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f36810e8640 1 --2- 172.21.15.12:0/440846301 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f367c0a5400 0x7f367c0a5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:13.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f36810e8640 1 -- 172.21.15.12:0/440846301 >> v1:172.21.15.71:6789/0 conn(0x7f367c0a5020 legacy=0x7f367c09d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:13.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f36810e8640 10 monclient: _finish_hunting 0 2024-09-11T01:29:13.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f36810e8640 1 monclient: found mon.noname-a 2024-09-11T01:29:13.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f36810e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:13.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f36810e8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:13.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f36810e8640 1 -- 172.21.15.12:0/440846301 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f367c09c370 con 0x7f367c13d2a0 2024-09-11T01:29:13.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.952+0000 7f3673fff640 1 --2- 172.21.15.12:0/440846301 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f367c0a5400 0x7f367c0a5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:13.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f36810e8640 1 -- 172.21.15.12:0/440846301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f367c13d2a0 msgr2=0x7f367c13f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:13.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f36810e8640 1 --2- 172.21.15.12:0/440846301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f367c13d2a0 0x7f367c13f690 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3674009e50 tx=0x7f367402a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22ba6c6def600712 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:13.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f36737fe640 1 -- 172.21.15.12:0/440846301 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3674036070 con 0x7f367c13d2a0 2024-09-11T01:29:13.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f36737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:13.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f36737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:13.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f36737fe640 10 monclient: dump: 2024-09-11T01:29:13.967 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:13.968 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:13.968 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:13.968 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:13.968 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:13.968 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:13.968 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:13.968 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:13.968 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:13.968 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:13.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f36737fe640 10 monclient: _finish_auth 0 2024-09-11T01:29:13.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f36737fe640 10 monclient: _check_auth_tickets 2024-09-11T01:29:13.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f36737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:13.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f36737fe640 1 -- 172.21.15.12:0/440846301 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3674007800 con 0x7f367c13d2a0 2024-09-11T01:29:13.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f3682b4b640 5 monclient: authenticate success, global_id 4194 2024-09-11T01:29:13.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f3682b4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:13.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f36737fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:13.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f3682b4b640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:13.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f3682b4b640 4 set_mon_vals no callback set 2024-09-11T01:29:13.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f36737fe640 1 -- 172.21.15.12:0/440846301 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3674007bd0 con 0x7f367c13d2a0 2024-09-11T01:29:13.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f36737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:13.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f36737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:13.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f36737fe640 10 monclient: dump: 2024-09-11T01:29:13.969 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:13.969 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:13.969 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:13.970 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:13.970 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:13.970 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:13.970 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:13.970 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:13.970 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:13.970 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:13.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f3682b4b640 10 monclient: shutdown 2024-09-11T01:29:13.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f3682b4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:13.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f3682b4b640 1 -- 172.21.15.12:0/440846301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f367c13d2a0 msgr2=0x7f367c13f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:13.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f3682b4b640 1 --2- 172.21.15.12:0/440846301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f367c13d2a0 0x7f367c13f690 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3674009e50 tx=0x7f367402a560 comp rx=0 tx=0).stop 2024-09-11T01:29:13.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f3682b4b640 1 -- 172.21.15.12:0/440846301 shutdown_connections 2024-09-11T01:29:13.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f3682b4b640 1 --2- 172.21.15.12:0/440846301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f367c13d2a0 0x7f367c13f690 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:13.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f3682b4b640 1 --2- 172.21.15.12:0/440846301 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f367c0a5400 0x7f367c0a5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:13.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f3682b4b640 1 -- 172.21.15.12:0/440846301 >> 172.21.15.12:0/440846301 conn(0x7f367c0a7300 msgr2=0x7f367c0a7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:13.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f3682b4b640 1 -- 172.21.15.12:0/440846301 shutdown_connections 2024-09-11T01:29:13.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f3682b4b640 1 -- 172.21.15.12:0/440846301 wait complete. 2024-09-11T01:29:13.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f3682b4b640 10 monclient: build_initial_monmap 2024-09-11T01:29:13.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f3682b4b640 10 monclient: monmap: 2024-09-11T01:29:13.971 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:13.971 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:13.971 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:13.971 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:13.971 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:13.971 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:13.971 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:13.972 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:13.972 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:13.972 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:13.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f3682b4b640 1 Processor -- start 2024-09-11T01:29:13.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f3682b4b640 1 -- start start 2024-09-11T01:29:13.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.956+0000 7f3682b4b640 10 monclient: init 2024-09-11T01:29:13.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f3682b4b640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:13.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f3682b4b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:29:13.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f3682b4b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f367c0a5020 0x7f367c1ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:13.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f3682b4b640 10 monclient(hunting): picked mon.noname-c con 0x7f367c0a5020 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:13.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f3682b4b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f367c0a5400 0x7f367c137ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:13.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f3682b4b640 10 monclient(hunting): picked mon.noname-a con 0x7f367c0a5400 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:13.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f3682b4b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f367c13d2a0 0x7f367c13b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:13.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f3682b4b640 10 monclient(hunting): picked mon.noname-b con 0x7f367c13d2a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:13.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f3682b4b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:13.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f3682b4b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f367c06f970 con 0x7f367c0a5400 2024-09-11T01:29:13.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f3682b4b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:13.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f3682b4b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f367c06f7f0 con 0x7f367c13d2a0 2024-09-11T01:29:13.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f3682b4b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:13.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f3682b4b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f367c13fbd0 con 0x7f367c0a5020 2024-09-11T01:29:13.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f3682b4b640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:13.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f3682b4b640 10 monclient(hunting): authenticate will time out at 760.400527s 2024-09-11T01:29:13.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f36808e7640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f367c0a5020 0x7f367c1ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:13.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f3673fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f367c0a5400 0x7f367c137ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:13.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f36808e7640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f367c0a5020 0x7f367c1ccf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:33852/0 (socket says 172.21.15.12:33852) 2024-09-11T01:29:13.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f36808e7640 1 -- 172.21.15.12:0/35124989 learned_addr learned my addr 172.21.15.12:0/35124989 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:13.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f36808e7640 10 monclient(hunting): get_auth_request con 0x7f367c0a5020 auth_method 0 2024-09-11T01:29:13.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f36808e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:13.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f36808e7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:13.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f36808e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:13.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f3673fff640 10 monclient(hunting): get_auth_request con 0x7f367c0a5400 auth_method 0 2024-09-11T01:29:13.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f3673fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:13.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f3673fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:13.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f3673fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:13.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f36808e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:13.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f36808e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:13.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f36808e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:13.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f3673fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:13.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f3673fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:13.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f3673fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:13.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f36810e8640 1 --2- 172.21.15.12:0/35124989 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f367c13d2a0 0x7f367c13b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:13.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f36810e8640 10 monclient(hunting): get_auth_request con 0x7f367c13d2a0 auth_method 0 2024-09-11T01:29:13.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f36810e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:13.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f36810e8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:13.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f36810e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:13.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f36808e7640 10 monclient(hunting): handle_auth_done global_id 4202 payload 824 2024-09-11T01:29:13.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f36808e7640 1 -- 172.21.15.12:0/35124989 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f367c13d2a0 msgr2=0x7f367c13b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:13.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f36808e7640 1 --2- 172.21.15.12:0/35124989 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f367c13d2a0 0x7f367c13b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:13.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f36808e7640 1 -- 172.21.15.12:0/35124989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f367c0a5400 msgr2=0x7f367c137ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:13.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f36808e7640 1 --2- 172.21.15.12:0/35124989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f367c0a5400 0x7f367c137ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:13.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f36808e7640 10 monclient: _finish_hunting 0 2024-09-11T01:29:13.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f36808e7640 1 monclient: found mon.noname-c 2024-09-11T01:29:13.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f36808e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:13.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f36808e7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:29:13.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f36808e7640 1 -- 172.21.15.12:0/35124989 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f367c1229a0 con 0x7f367c0a5020 2024-09-11T01:29:13.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.957+0000 7f3673fff640 1 --2- 172.21.15.12:0/35124989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f367c0a5400 0x7f367c137ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:13.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.960+0000 7f36808e7640 1 -- 172.21.15.12:0/35124989 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f367c0a5020 msgr2=0x7f367c1ccf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:13.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.960+0000 7f36808e7640 1 --2- 172.21.15.12:0/35124989 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f367c0a5020 0x7f367c1ccf40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f367800c430 tx=0x7f367800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:13.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.960+0000 7f36717fa640 1 -- 172.21.15.12:0/35124989 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36780119a0 con 0x7f367c0a5020 2024-09-11T01:29:13.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.960+0000 7f36717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:13.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.960+0000 7f36717fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:13.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.960+0000 7f36717fa640 10 monclient: dump: 2024-09-11T01:29:13.978 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:13.979 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:13.979 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:13.979 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:13.979 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:13.979 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:13.979 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:13.979 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:13.979 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:13.979 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:13.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.960+0000 7f36717fa640 10 monclient: _finish_auth 0 2024-09-11T01:29:13.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.960+0000 7f36717fa640 10 monclient: _check_auth_tickets 2024-09-11T01:29:13.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.960+0000 7f36717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:13.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.960+0000 7f36717fa640 1 -- 172.21.15.12:0/35124989 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3678017070 con 0x7f367c0a5020 2024-09-11T01:29:13.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.960+0000 7f3682b4b640 5 monclient: authenticate success, global_id 4202 2024-09-11T01:29:13.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.960+0000 7f3682b4b640 10 monclient: _renew_subs 2024-09-11T01:29:13.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.960+0000 7f3682b4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:13.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.960+0000 7f3682b4b640 1 -- 172.21.15.12:0/35124989 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f367c13cc60 con 0x7f367c0a5020 2024-09-11T01:29:13.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.960+0000 7f36717fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:13.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.960+0000 7f36717fa640 1 -- 172.21.15.12:0/35124989 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3678011d80 con 0x7f367c0a5020 2024-09-11T01:29:13.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.960+0000 7f36717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:13.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.960+0000 7f36717fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:13.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.960+0000 7f36717fa640 10 monclient: dump: 2024-09-11T01:29:13.980 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:13.980 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:13.980 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:13.980 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:13.981 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:13.981 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:13.981 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:13.981 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:13.981 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:13.981 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:13.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.960+0000 7f3682b4b640 10 monclient: _renew_subs 2024-09-11T01:29:13.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.960+0000 7f3672ffd640 4 set_mon_vals no callback set 2024-09-11T01:29:13.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.960+0000 7f3682b4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:13.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.960+0000 7f3682b4b640 1 -- 172.21.15.12:0/35124989 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f367c121570 con 0x7f367c0a5020 2024-09-11T01:29:13.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.961+0000 7f36717fa640 1 -- 172.21.15.12:0/35124989 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 4) ==== 50557+0+0 (secure 0 0 0) 0x7f367801c460 con 0x7f367c0a5020 2024-09-11T01:29:13.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.961+0000 7f3682b4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:13.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.961+0000 7f3682b4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:13.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.961+0000 7f3682b4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:13.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.961+0000 7f3682b4b640 1 -- 172.21.15.12:0/35124989 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f367c122210 con 0x7f367c0a5020 2024-09-11T01:29:13.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.961+0000 7f36717fa640 1 --2- 172.21.15.12:0/35124989 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f366803e1b0 0x7f3668040670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:13.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.961+0000 7f36717fa640 1 -- 172.21.15.12:0/35124989 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(12..12 src has 1..12) ==== 3607+0+0 (secure 0 0 0) 0x7f36780525d0 con 0x7f367c0a5020 2024-09-11T01:29:13.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.961+0000 7f3673fff640 1 --2- 172.21.15.12:0/35124989 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f366803e1b0 0x7f3668040670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:13.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.961+0000 7f3673fff640 10 monclient: get_auth_request con 0x7f366803e1b0 auth_method 0 2024-09-11T01:29:13.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.964+0000 7f3673fff640 1 --2- 172.21.15.12:0/35124989 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f366803e1b0 0x7f3668040670 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f367c121080 tx=0x7f366c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:13.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.966+0000 7f36717fa640 1 -- 172.21.15.12:0/35124989 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3678051470 con 0x7f367c0a5020 2024-09-11T01:29:13.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.966+0000 7f36717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:13.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:13.966+0000 7f36717fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:14.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.152+0000 7f3682b4b640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-11T01:29:14.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.152+0000 7f3682b4b640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-11T01:29:14.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.152+0000 7f3682b4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:14.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.152+0000 7f3682b4b640 1 -- 172.21.15.12:0/35124989 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7f367c0a4380 con 0x7f367c0a5020 2024-09-11T01:29:14.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.534+0000 7f36717fa640 1 -- 172.21.15.12:0/35124989 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v13) ==== 109+0+0 (secure 0 0 0) 0x7f367c0a4380 con 0x7f367c0a5020 2024-09-11T01:29:14.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.534+0000 7f36717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-11T01:29:14.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.534+0000 7f36717fa640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-11T01:29:14.538 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'rbd' created 2024-09-11T01:29:14.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.537+0000 7f3682b4b640 1 -- 172.21.15.12:0/35124989 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f366803e1b0 msgr2=0x7f3668040670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:14.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.537+0000 7f3682b4b640 1 --2- 172.21.15.12:0/35124989 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f366803e1b0 0x7f3668040670 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f367c121080 tx=0x7f366c009190 comp rx=0 tx=0).stop 2024-09-11T01:29:14.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.537+0000 7f3682b4b640 10 monclient: shutdown 2024-09-11T01:29:14.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.537+0000 7f3682b4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:14.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.537+0000 7f3682b4b640 1 -- 172.21.15.12:0/35124989 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f367c0a5020 msgr2=0x7f367c1ccf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:14.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.537+0000 7f3682b4b640 1 --2- 172.21.15.12:0/35124989 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f367c0a5020 0x7f367c1ccf40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f367800c430 tx=0x7f367800c8f0 comp rx=0 tx=0).stop 2024-09-11T01:29:14.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.537+0000 7f3682b4b640 1 -- 172.21.15.12:0/35124989 shutdown_connections 2024-09-11T01:29:14.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.537+0000 7f3682b4b640 1 --2- 172.21.15.12:0/35124989 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f367c13d2a0 0x7f367c13b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:14.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.537+0000 7f3682b4b640 1 --2- 172.21.15.12:0/35124989 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f366803e1b0 0x7f3668040670 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:14.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.537+0000 7f3682b4b640 1 --2- 172.21.15.12:0/35124989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f367c0a5400 0x7f367c137ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:14.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.537+0000 7f3682b4b640 1 --2- 172.21.15.12:0/35124989 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f367c0a5020 0x7f367c1ccf40 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:14.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.537+0000 7f3682b4b640 1 -- 172.21.15.12:0/35124989 >> 172.21.15.12:0/35124989 conn(0x7f367c0a7300 msgr2=0x7f367c0a3c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:14.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.537+0000 7f3682b4b640 1 -- 172.21.15.12:0/35124989 shutdown_connections 2024-09-11T01:29:14.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.537+0000 7f3682b4b640 1 -- 172.21.15.12:0/35124989 wait complete. 2024-09-11T01:29:14.551 DEBUG:teuthology.orchestra.run.smithi012:> rbd --cluster ceph pool init rbd 2024-09-11T01:29:14.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.619+0000 7efed1742140 10 monclient: get_monmap_and_config 2024-09-11T01:29:14.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.619+0000 7efed1742140 10 monclient: build_initial_monmap 2024-09-11T01:29:14.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.619+0000 7efed1742140 10 monclient: monmap: 2024-09-11T01:29:14.622 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:14.622 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:14.622 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:14.621453+0000 2024-09-11T01:29:14.622 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:14.621453+0000 2024-09-11T01:29:14.623 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:14.623 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:14.623 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:14.623 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:14.623 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:14.623 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:14.623 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:14.623 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:14.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.620+0000 7efed1742140 1 Processor -- start 2024-09-11T01:29:14.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.620+0000 7efed1742140 1 -- start start 2024-09-11T01:29:14.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.620+0000 7efed1742140 10 monclient: init 2024-09-11T01:29:14.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efed1742140 10 monclient: _reopen_session rank -1 2024-09-11T01:29:14.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efed1742140 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-11T01:29:14.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efed1742140 1 --2- >> v2:172.21.15.12:3300/0 conn(0x564e58048f60 0x564e58049340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:14.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efed1742140 10 monclient(hunting): picked mon.noname-a con 0x564e58048f60 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:14.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efed1742140 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x564e57fb40a0 0x564e58049880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:14.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efed1742140 10 monclient(hunting): picked mon.noname-b con 0x564e57fb40a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:14.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efed1742140 1 --2- >> v2:172.21.15.71:3300/0 conn(0x564e58045d50 0x564e580461e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:14.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efed1742140 10 monclient(hunting): picked mon.noname-c con 0x564e58045d50 addr v2:172.21.15.71:3300/0 2024-09-11T01:29:14.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efed1742140 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:14.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efed1742140 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x564e57e83a80 con 0x564e58048f60 2024-09-11T01:29:14.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efed1742140 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:14.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efed1742140 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x564e57e94650 con 0x564e58045d50 2024-09-11T01:29:14.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efed1742140 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:14.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efed1742140 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x564e57e87890 con 0x564e57fb40a0 2024-09-11T01:29:14.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efed1742140 10 monclient(hunting): _renew_subs 2024-09-11T01:29:14.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efed1742140 10 monclient(hunting): authenticate will time out at 761.064149s 2024-09-11T01:29:14.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecbfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x564e58048f60 0x564e58049340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:14.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecb7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x564e57fb40a0 0x564e58049880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:14.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecbfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x564e58048f60 0x564e58049340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38252/0 (socket says 172.21.15.12:38252) 2024-09-11T01:29:14.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecbfff640 1 -- 172.21.15.12:0/2197292840 learned_addr learned my addr 172.21.15.12:0/2197292840 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:14.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecbfff640 10 monclient(hunting): get_auth_request con 0x564e58048f60 auth_method 0 2024-09-11T01:29:14.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:14.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:14.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:14.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecb7fe640 10 monclient(hunting): get_auth_request con 0x564e57fb40a0 auth_method 0 2024-09-11T01:29:14.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:14.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:14.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:14.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:14.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:14.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:14.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:14.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:14.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:14.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efed1491640 1 --2- 172.21.15.12:0/2197292840 >> v2:172.21.15.71:3300/0 conn(0x564e58045d50 0x564e580461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:14.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efed1491640 10 monclient(hunting): get_auth_request con 0x564e58045d50 auth_method 0 2024-09-11T01:29:14.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efed1491640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:14.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efed1491640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:14.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efed1491640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:14.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecbfff640 10 monclient(hunting): handle_auth_done global_id 4200 payload 274 2024-09-11T01:29:14.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecbfff640 1 -- 172.21.15.12:0/2197292840 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x564e57fb40a0 msgr2=0x564e58049880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:14.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecbfff640 1 --2- 172.21.15.12:0/2197292840 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x564e57fb40a0 0x564e58049880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:14.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecbfff640 1 -- 172.21.15.12:0/2197292840 >> v2:172.21.15.71:3300/0 conn(0x564e58045d50 msgr2=0x564e580461e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:14.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecbfff640 1 --2- 172.21.15.12:0/2197292840 >> v2:172.21.15.71:3300/0 conn(0x564e58045d50 0x564e580461e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:14.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecbfff640 10 monclient: _finish_hunting 0 2024-09-11T01:29:14.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecbfff640 1 monclient: found mon.noname-a 2024-09-11T01:29:14.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:14.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:14.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecbfff640 1 -- 172.21.15.12:0/2197292840 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e57e33d90 con 0x564e58048f60 2024-09-11T01:29:14.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.621+0000 7efecb7fe640 1 --2- 172.21.15.12:0/2197292840 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x564e57fb40a0 0x564e58049880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:14.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.625+0000 7efecbfff640 1 -- 172.21.15.12:0/2197292840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e58048f60 msgr2=0x564e58049340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:14.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.625+0000 7efecbfff640 1 --2- 172.21.15.12:0/2197292840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e58048f60 0x564e58049340 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x564e57e83d60 tx=0x7efebc045960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7f53c57aad9bb server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:14.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.625+0000 7efecaffd640 1 -- 172.21.15.12:0/2197292840 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc05a070 con 0x564e58048f60 2024-09-11T01:29:14.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.625+0000 7efecaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:14.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.625+0000 7efecaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:14.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.625+0000 7efecaffd640 10 monclient: dump: 2024-09-11T01:29:14.635 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:14.635 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:14.635 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:14.635 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:14.635 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:14.635 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:14.635 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:14.636 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:14.636 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:14.636 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:14.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.625+0000 7efecaffd640 10 monclient: _finish_auth 0 2024-09-11T01:29:14.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.625+0000 7efecaffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:14.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.625+0000 7efecaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:14.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.625+0000 7efecaffd640 1 -- 172.21.15.12:0/2197292840 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efebc055680 con 0x564e58048f60 2024-09-11T01:29:14.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.625+0000 7efed1742140 5 monclient: authenticate success, global_id 4200 2024-09-11T01:29:14.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.625+0000 7efed1742140 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:14.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.625+0000 7efecaffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:14.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.625+0000 7efed1742140 10 monclient: get_monmap_and_config success 2024-09-11T01:29:14.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.625+0000 7efecaffd640 1 -- 172.21.15.12:0/2197292840 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc055a70 con 0x564e58048f60 2024-09-11T01:29:14.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.625+0000 7efecaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:14.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.625+0000 7efed1742140 4 set_mon_vals no callback set 2024-09-11T01:29:14.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.625+0000 7efecaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:14.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.625+0000 7efecaffd640 10 monclient: dump: 2024-09-11T01:29:14.637 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:14.637 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:14.637 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:14.637 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:14.637 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:14.637 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:14.637 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:14.637 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:14.637 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:14.637 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:14.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.626+0000 7efed1742140 10 monclient: shutdown 2024-09-11T01:29:14.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.626+0000 7efed1742140 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:14.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.626+0000 7efed1742140 1 -- 172.21.15.12:0/2197292840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e58048f60 msgr2=0x564e58049340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:14.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.626+0000 7efed1742140 1 --2- 172.21.15.12:0/2197292840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e58048f60 0x564e58049340 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x564e57e83d60 tx=0x7efebc045960 comp rx=0 tx=0).stop 2024-09-11T01:29:14.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.626+0000 7efed1742140 1 -- 172.21.15.12:0/2197292840 shutdown_connections 2024-09-11T01:29:14.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.626+0000 7efed1742140 1 --2- 172.21.15.12:0/2197292840 >> v2:172.21.15.71:3300/0 conn(0x564e58045d50 0x564e580461e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:14.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.626+0000 7efed1742140 1 --2- 172.21.15.12:0/2197292840 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x564e57fb40a0 0x564e58049880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:14.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.626+0000 7efed1742140 1 --2- 172.21.15.12:0/2197292840 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e58048f60 0x564e58049340 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:14.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.626+0000 7efed1742140 1 -- 172.21.15.12:0/2197292840 >> 172.21.15.12:0/2197292840 conn(0x564e57f99eb0 msgr2=0x564e57f9a2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:14.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.626+0000 7efed1742140 1 -- 172.21.15.12:0/2197292840 shutdown_connections 2024-09-11T01:29:14.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.626+0000 7efed1742140 1 -- 172.21.15.12:0/2197292840 wait complete. 2024-09-11T01:29:14.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.626+0000 7efed1742140 10 monclient: build_initial_monmap 2024-09-11T01:29:14.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.626+0000 7efed1742140 10 monclient: monmap: 2024-09-11T01:29:14.638 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:14.638 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:14.639 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:14.639 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:14.639 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:14.639 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:14.639 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:14.639 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:14.639 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:14.639 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:14.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.627+0000 7efed1742140 1 Processor -- start 2024-09-11T01:29:14.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.627+0000 7efed1742140 1 -- start start 2024-09-11T01:29:14.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.627+0000 7efed1742140 10 monclient: init 2024-09-11T01:29:14.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.627+0000 7efed1742140 10 monclient: _reopen_session rank -1 2024-09-11T01:29:14.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.627+0000 7efed1742140 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:29:14.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.627+0000 7efed1742140 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564e57fb40a0 0x564e58046210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:14.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.627+0000 7efed1742140 10 monclient(hunting): picked mon.noname-b con 0x564e57fb40a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:14.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.627+0000 7efed1742140 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e58045d50 0x564e58046750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:14.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.627+0000 7efed1742140 10 monclient(hunting): picked mon.noname-a con 0x564e58045d50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:14.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.627+0000 7efed1742140 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x564e58048f60 0x564e58046c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:14.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.627+0000 7efed1742140 10 monclient(hunting): picked mon.noname-c con 0x564e58048f60 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:14.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.627+0000 7efed1742140 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:14.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.627+0000 7efed1742140 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x564e57fb4c10 con 0x564e58045d50 2024-09-11T01:29:14.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.627+0000 7efed1742140 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:14.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.627+0000 7efed1742140 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x564e57e29390 con 0x564e57fb40a0 2024-09-11T01:29:14.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.627+0000 7efed1742140 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:14.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.627+0000 7efed1742140 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x564e57fb4ef0 con 0x564e58048f60 2024-09-11T01:29:14.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.627+0000 7efed1742140 10 monclient(hunting): _renew_subs 2024-09-11T01:29:14.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.627+0000 7efed1742140 10 monclient(hunting): authenticate will time out at 761.070921s 2024-09-11T01:29:14.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efed1491640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x564e58048f60 0x564e58046c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:14.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efecb7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e58045d50 0x564e58046750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:14.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efed1491640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x564e58048f60 0x564e58046c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:33862/0 (socket says 172.21.15.12:33862) 2024-09-11T01:29:14.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efed1491640 1 -- 172.21.15.12:0/28735595 learned_addr learned my addr 172.21.15.12:0/28735595 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:14.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efed1491640 10 monclient(hunting): get_auth_request con 0x564e58048f60 auth_method 0 2024-09-11T01:29:14.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efed1491640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:14.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efed1491640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:14.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efed1491640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:14.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efecb7fe640 10 monclient(hunting): get_auth_request con 0x564e58045d50 auth_method 0 2024-09-11T01:29:14.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efecb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:14.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efecb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:14.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efecb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:14.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efecb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:14.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efecb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:14.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efecb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:14.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efed1491640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:14.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efed1491640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:14.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efecbfff640 1 --2- 172.21.15.12:0/28735595 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564e57fb40a0 0x564e58046210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:14.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efed1491640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:14.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efecbfff640 10 monclient(hunting): get_auth_request con 0x564e57fb40a0 auth_method 0 2024-09-11T01:29:14.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efecbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:14.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efecbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:14.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efecbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:14.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efecbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:14.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efecbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:14.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efecbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:14.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efecb7fe640 10 monclient(hunting): handle_auth_done global_id 4203 payload 824 2024-09-11T01:29:14.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efecb7fe640 1 -- 172.21.15.12:0/28735595 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x564e58048f60 msgr2=0x564e58046c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:14.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efecb7fe640 1 --2- 172.21.15.12:0/28735595 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x564e58048f60 0x564e58046c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:14.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efecb7fe640 1 -- 172.21.15.12:0/28735595 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564e57fb40a0 msgr2=0x564e58046210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:14.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efecb7fe640 1 --2- 172.21.15.12:0/28735595 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564e57fb40a0 0x564e58046210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:14.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efecb7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:29:14.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efecb7fe640 1 monclient: found mon.noname-a 2024-09-11T01:29:14.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efecb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:14.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efecb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:14.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.628+0000 7efecb7fe640 1 -- 172.21.15.12:0/28735595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e57e4b410 con 0x564e58045d50 2024-09-11T01:29:14.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.630+0000 7efecb7fe640 1 -- 172.21.15.12:0/28735595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e58045d50 msgr2=0x564e58046750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:14.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.630+0000 7efecb7fe640 1 --2- 172.21.15.12:0/28735595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e58045d50 0x564e58046750 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efec0005fd0 tx=0x7efec0008880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:14.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.630+0000 7efebbfff640 1 -- 172.21.15.12:0/28735595 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec00129a0 con 0x564e58045d50 2024-09-11T01:29:14.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.630+0000 7efebbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:14.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.630+0000 7efebbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:14.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.630+0000 7efebbfff640 10 monclient: dump: 2024-09-11T01:29:14.644 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:14.644 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:14.644 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:14.644 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:14.644 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:14.644 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:14.644 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:14.645 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:14.645 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:14.645 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:14.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.630+0000 7efebbfff640 10 monclient: _finish_auth 0 2024-09-11T01:29:14.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.630+0000 7efebbfff640 10 monclient: _check_auth_tickets 2024-09-11T01:29:14.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.630+0000 7efebbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:14.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.630+0000 7efebbfff640 1 -- 172.21.15.12:0/28735595 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efec0012b40 con 0x564e58045d50 2024-09-11T01:29:14.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.630+0000 7efebbfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:14.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.630+0000 7efebbfff640 1 -- 172.21.15.12:0/28735595 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec0013420 con 0x564e58045d50 2024-09-11T01:29:14.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.630+0000 7efed1742140 5 monclient: authenticate success, global_id 4203 2024-09-11T01:29:14.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.630+0000 7efed1742140 10 monclient: _renew_subs 2024-09-11T01:29:14.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.630+0000 7efed1742140 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:14.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.630+0000 7efed1742140 1 -- 172.21.15.12:0/28735595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564e57e02720 con 0x564e58045d50 2024-09-11T01:29:14.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.630+0000 7efeca7fc640 4 set_mon_vals no callback set 2024-09-11T01:29:14.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.630+0000 7efebbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:14.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.630+0000 7efebbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:14.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.630+0000 7efebbfff640 10 monclient: dump: 2024-09-11T01:29:14.646 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:14.646 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:14.646 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:14.646 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:14.646 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:14.646 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:14.646 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:14.646 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:14.646 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:14.646 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:14.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.630+0000 7efed1742140 10 monclient: _renew_subs 2024-09-11T01:29:14.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.630+0000 7efed1742140 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:14.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.631+0000 7efed1742140 1 -- 172.21.15.12:0/28735595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e57fb0750 con 0x564e58045d50 2024-09-11T01:29:14.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.631+0000 7efebbfff640 1 -- 172.21.15.12:0/28735595 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 4) ==== 50557+0+0 (secure 0 0 0) 0x7efec00135c0 con 0x564e58045d50 2024-09-11T01:29:14.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.632+0000 7efebbfff640 1 --2- 172.21.15.12:0/28735595 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efeb003ca70 0x7efeb003ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:14.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.632+0000 7efebbfff640 1 -- 172.21.15.12:0/28735595 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 3949+0+0 (secure 0 0 0) 0x7efec00519e0 con 0x564e58045d50 2024-09-11T01:29:14.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.632+0000 7efed1742140 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-11T01:29:14.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.632+0000 7efed1742140 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-11T01:29:14.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.632+0000 7efed1742140 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:14.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.632+0000 7efed1742140 1 -- 172.21.15.12:0/28735595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x564e57fb14f0 con 0x564e58045d50 2024-09-11T01:29:14.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.632+0000 7efecbfff640 1 --2- 172.21.15.12:0/28735595 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efeb003ca70 0x7efeb003ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:14.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.632+0000 7efecbfff640 10 monclient: get_auth_request con 0x7efeb003ca70 auth_method 0 2024-09-11T01:29:14.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:14.635+0000 7efecbfff640 1 --2- 172.21.15.12:0/28735595 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efeb003ca70 0x7efeb003ef30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x564e57f9aa30 tx=0x7efebc052000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:15.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.536+0000 7efebbfff640 1 -- 172.21.15.12:0/28735595 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v14) ==== 141+0+0 (secure 0 0 0) 0x7efec001c400 con 0x564e58045d50 2024-09-11T01:29:15.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.536+0000 7efebbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-11T01:29:15.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.536+0000 7efebbfff640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-11T01:29:15.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.536+0000 7efed1742140 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:15.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.536+0000 7efed1742140 1 -- 172.21.15.12:0/28735595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efebc055680 con 0x564e58045d50 2024-09-11T01:29:15.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.540+0000 7efebbfff640 1 -- 172.21.15.12:0/28735595 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_get_version_reply(handle=1 version=14) ==== 24+0+0 (secure 0 0 0) 0x7efec00138e0 con 0x564e58045d50 2024-09-11T01:29:15.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.540+0000 7efebbfff640 10 monclient: handle_get_version_reply finishing 1 version 14 2024-09-11T01:29:15.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.540+0000 7efeca7fc640 10 monclient: _renew_subs 2024-09-11T01:29:15.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.540+0000 7efeca7fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:15.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.540+0000 7efeca7fc640 1 -- 172.21.15.12:0/28735595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=14}) -- 0x7efea8002ae0 con 0x564e58045d50 2024-09-11T01:29:15.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.540+0000 7efebbfff640 1 -- 172.21.15.12:0/28735595 <== mon.0 v2:172.21.15.12:3300/0 8 ==== osd_map(14..14 src has 1..14) ==== 1033+0+0 (secure 0 0 0) 0x7efec001c780 con 0x564e58045d50 2024-09-11T01:29:15.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.540+0000 7efed1742140 1 --2- 172.21.15.12:0/28735595 >> [v2:172.21.15.12:6808/3563357280,v1:172.21.15.12:6809/3563357280] conn(0x564e58062af0 0x564e58064ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:15.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.540+0000 7efed1742140 1 -- 172.21.15.12:0/28735595 --> [v2:172.21.15.12:6808/3563357280,v1:172.21.15.12:6809/3563357280] -- osd_op(unknown.0.0:1 1.3 1:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14) -- 0x564e58065420 con 0x564e58062af0 2024-09-11T01:29:15.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.541+0000 7efed1491640 1 --2- 172.21.15.12:0/28735595 >> [v2:172.21.15.12:6808/3563357280,v1:172.21.15.12:6809/3563357280] conn(0x564e58062af0 0x564e58064ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:15.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.541+0000 7efed1491640 10 monclient: get_auth_request con 0x564e58062af0 auth_method 0 2024-09-11T01:29:15.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.545+0000 7efed1491640 1 --2- 172.21.15.12:0/28735595 >> [v2:172.21.15.12:6808/3563357280,v1:172.21.15.12:6809/3563357280] conn(0x564e58062af0 0x564e58064ee0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:15.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.548+0000 7efed1491640 1 -- 172.21.15.12:0/28735595 <== osd.1 v2:172.21.15.12:6808/3563357280 1 ==== osd_op_reply(1 rbd_trash [create] v14'1 uv1 ondisk = 0) ==== 153+0+0 (crc 0 0 0) 0x7efec400c9c0 con 0x564e58062af0 2024-09-11T01:29:15.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.551+0000 7efed1742140 1 --2- 172.21.15.12:0/28735595 >> [v2:172.21.15.71:6800/1790415427,v1:172.21.15.71:6801/1790415427] conn(0x564e580b87f0 0x564e580bac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:15.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.551+0000 7efed1742140 1 -- 172.21.15.12:0/28735595 --> [v2:172.21.15.71:6800/1790415427,v1:172.21.15.71:6801/1790415427] -- osd_op(unknown.0.0:2 1.2 1:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e14) -- 0x564e580bb1e0 con 0x564e580b87f0 2024-09-11T01:29:15.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.551+0000 7efecbfff640 1 --2- 172.21.15.12:0/28735595 >> [v2:172.21.15.71:6800/1790415427,v1:172.21.15.71:6801/1790415427] conn(0x564e580b87f0 0x564e580bac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:15.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.551+0000 7efecbfff640 10 monclient: get_auth_request con 0x564e580b87f0 auth_method 0 2024-09-11T01:29:15.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.556+0000 7efecbfff640 1 --2- 172.21.15.12:0/28735595 >> [v2:172.21.15.71:6800/1790415427,v1:172.21.15.71:6801/1790415427] conn(0x564e580b87f0 0x564e580bac80 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:15.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.556+0000 7efecbfff640 1 -- 172.21.15.12:0/28735595 <== osd.4 v2:172.21.15.71:6800/1790415427 1 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7efebc055d70 con 0x564e580b87f0 2024-09-11T01:29:15.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.556+0000 7efed1742140 1 -- 172.21.15.12:0/28735595 --> [v2:172.21.15.71:6800/1790415427,v1:172.21.15.71:6801/1790415427] -- osd_op(unknown.0.0:3 1.2 1:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e14) -- 0x564e580b72d0 con 0x564e580b87f0 2024-09-11T01:29:15.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.556+0000 7efecbfff640 1 -- 172.21.15.12:0/28735595 <== osd.4 v2:172.21.15.71:6800/1790415427 2 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7efebc055d70 con 0x564e580b87f0 2024-09-11T01:29:15.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.556+0000 7efec9ffb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:15.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:15.556+0000 7efec9ffb640 1 -- 172.21.15.12:0/28735595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- pool_op(create unmanaged snap pool 1 tid 4 name v0) -- 0x7efeac001f00 con 0x564e58045d50 2024-09-11T01:29:16.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:16.540+0000 7efebbfff640 1 -- 172.21.15.12:0/28735595 <== mon.0 v2:172.21.15.12:3300/0 9 ==== pool_op_reply(tid 4 (0) Success v15) ==== 55+0+0 (secure 0 0 0) 0x7efec00401f0 con 0x564e58045d50 2024-09-11T01:29:16.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:16.540+0000 7efebbfff640 10 monclient: _renew_subs 2024-09-11T01:29:16.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:16.540+0000 7efebbfff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:16.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:16.540+0000 7efebbfff640 1 -- 172.21.15.12:0/28735595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=15}) -- 0x7efeb0042e70 con 0x564e58045d50 2024-09-11T01:29:16.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:16.541+0000 7efebbfff640 1 -- 172.21.15.12:0/28735595 <== mon.0 v2:172.21.15.12:3300/0 10 ==== osd_map(15..15 src has 1..15) ==== 1004+0+0 (secure 0 0 0) 0x7efec00403d0 con 0x564e58045d50 2024-09-11T01:29:16.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:16.542+0000 7efeca7fc640 1 -- 172.21.15.12:0/28735595 --> [v2:172.21.15.71:6800/1790415427,v1:172.21.15.71:6801/1790415427] -- osd_op(unknown.0.0:5 1.2 1:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15) -- 0x7efea8005790 con 0x564e580b87f0 2024-09-11T01:29:16.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:16.546+0000 7efecbfff640 1 -- 172.21.15.12:0/28735595 <== osd.4 v2:172.21.15.71:6800/1790415427 3 ==== osd_op_reply(5 rbd_info [create,write 0~8] v15'1 uv1 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7efebc055d70 con 0x564e580b87f0 2024-09-11T01:29:16.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:16.546+0000 7efec9ffb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:16.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:16.546+0000 7efec9ffb640 1 -- 172.21.15.12:0/28735595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- pool_op(delete unmanaged snap pool 1 tid 6 name v15) -- 0x7efeac003350 con 0x564e58045d50 2024-09-11T01:29:17.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.572+0000 7efebbfff640 1 -- 172.21.15.12:0/28735595 <== mon.0 v2:172.21.15.12:3300/0 11 ==== pool_op_reply(tid 6 (0) Success v16) ==== 43+0+0 (secure 0 0 0) 0x7efec004f030 con 0x564e58045d50 2024-09-11T01:29:17.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.572+0000 7efebbfff640 10 monclient: _renew_subs 2024-09-11T01:29:17.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.572+0000 7efebbfff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:17.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.572+0000 7efebbfff640 1 -- 172.21.15.12:0/28735595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=16}) -- 0x7efeb0042e70 con 0x564e58045d50 2024-09-11T01:29:17.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.573+0000 7efebbfff640 1 -- 172.21.15.12:0/28735595 <== mon.0 v2:172.21.15.12:3300/0 12 ==== osd_map(16..16 src has 1..16) ==== 1024+0+0 (secure 0 0 0) 0x7efec004f210 con 0x564e58045d50 2024-09-11T01:29:17.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.573+0000 7efeca7fc640 1 -- 172.21.15.12:0/28735595 --> [v2:172.21.15.71:6800/1790415427,v1:172.21.15.71:6801/1790415427] -- osd_op(unknown.0.0:7 1.2 1:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16) -- 0x7efea8005b50 con 0x564e580b87f0 2024-09-11T01:29:17.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.577+0000 7efecbfff640 1 -- 172.21.15.12:0/28735595 <== osd.4 v2:172.21.15.71:6800/1790415427 4 ==== osd_op_reply(7 rbd_info [write 0~19] v16'2 uv2 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efebc055d70 con 0x564e580b87f0 2024-09-11T01:29:17.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.577+0000 7efed1742140 1 -- 172.21.15.12:0/28735595 >> [v2:172.21.15.12:6808/3563357280,v1:172.21.15.12:6809/3563357280] conn(0x564e58062af0 msgr2=0x564e58064ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:17.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.577+0000 7efed1742140 1 --2- 172.21.15.12:0/28735595 >> [v2:172.21.15.12:6808/3563357280,v1:172.21.15.12:6809/3563357280] conn(0x564e58062af0 0x564e58064ee0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:17.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.577+0000 7efed1742140 1 -- 172.21.15.12:0/28735595 >> [v2:172.21.15.71:6800/1790415427,v1:172.21.15.71:6801/1790415427] conn(0x564e580b87f0 msgr2=0x564e580bac80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:17.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.577+0000 7efed1742140 1 --2- 172.21.15.12:0/28735595 >> [v2:172.21.15.71:6800/1790415427,v1:172.21.15.71:6801/1790415427] conn(0x564e580b87f0 0x564e580bac80 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:17.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.577+0000 7efed1742140 1 -- 172.21.15.12:0/28735595 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efeb003ca70 msgr2=0x7efeb003ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:17.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.577+0000 7efed1742140 1 --2- 172.21.15.12:0/28735595 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efeb003ca70 0x7efeb003ef30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x564e57f9aa30 tx=0x7efebc052000 comp rx=0 tx=0).stop 2024-09-11T01:29:17.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.577+0000 7efed1491640 1 -- 172.21.15.12:0/28735595 reap_dead start 2024-09-11T01:29:17.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.577+0000 7efed1742140 10 monclient: shutdown 2024-09-11T01:29:17.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.577+0000 7efed1742140 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:17.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.577+0000 7efed1742140 1 -- 172.21.15.12:0/28735595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e58045d50 msgr2=0x564e58046750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:17.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.577+0000 7efed1742140 1 --2- 172.21.15.12:0/28735595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e58045d50 0x564e58046750 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efec0005fd0 tx=0x7efec0008880 comp rx=0 tx=0).stop 2024-09-11T01:29:17.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.577+0000 7efed1742140 1 -- 172.21.15.12:0/28735595 shutdown_connections 2024-09-11T01:29:17.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.577+0000 7efed1742140 1 --2- 172.21.15.12:0/28735595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564e58045d50 0x564e58046750 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:17.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.577+0000 7efed1742140 1 -- 172.21.15.12:0/28735595 >> 172.21.15.12:0/28735595 conn(0x564e57f99eb0 msgr2=0x564e5804c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:17.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.577+0000 7efed1742140 1 -- 172.21.15.12:0/28735595 shutdown_connections 2024-09-11T01:29:17.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.578+0000 7efed1742140 1 -- 172.21.15.12:0/28735595 wait complete. 2024-09-11T01:29:17.584 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-11T01:29:17.584 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-11T01:29:17.585 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-11T01:29:17.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.686+0000 7f4d7f517640 10 monclient: get_monmap_and_config 2024-09-11T01:29:17.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.686+0000 7f4d7f517640 10 monclient: build_initial_monmap 2024-09-11T01:29:17.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.686+0000 7f4d7f517640 10 monclient: monmap: 2024-09-11T01:29:17.688 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:17.688 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:17.688 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:17.686865+0000 2024-09-11T01:29:17.688 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:17.686865+0000 2024-09-11T01:29:17.688 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:17.688 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:17.688 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:17.688 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:17.688 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:17.689 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:17.689 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:17.689 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:17.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.686+0000 7f4d7f517640 1 Processor -- start 2024-09-11T01:29:17.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.686+0000 7f4d7f517640 1 -- start start 2024-09-11T01:29:17.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.686+0000 7f4d7f517640 10 monclient: init 2024-09-11T01:29:17.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7f517640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:17.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7f517640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-11T01:29:17.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7f517640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4d78137d50 0x7f4d7809d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:17.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7f517640 10 monclient(hunting): picked mon.noname-c con 0x7f4d78137d50 addr v2:172.21.15.71:3300/0 2024-09-11T01:29:17.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7f517640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4d78138130 0x7f4d78138550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:17.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7f517640 10 monclient(hunting): picked mon.noname-b con 0x7f4d78138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:17.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7f517640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d78136cd0 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:17.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7f517640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d7809ba20 con 0x7f4d78136cd0 2024-09-11T01:29:17.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7f517640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:17.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7f517640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f4d7809c650 con 0x7f4d78137d50 2024-09-11T01:29:17.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7f517640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:17.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7f517640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4d7809c370 con 0x7f4d78138130 2024-09-11T01:29:17.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7f517640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:17.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7f517640 10 monclient(hunting): authenticate will time out at 764.129462s 2024-09-11T01:29:17.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7cab2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4d78138130 0x7f4d78138550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:17.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7cab2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4d78138130 0x7f4d78138550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51428/0 (socket says 172.21.15.12:51428) 2024-09-11T01:29:17.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7cab2640 1 -- 172.21.15.12:0/360984011 learned_addr learned my addr 172.21.15.12:0/360984011 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:17.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7cab2640 10 monclient(hunting): get_auth_request con 0x7f4d78138130 auth_method 0 2024-09-11T01:29:17.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7cab2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:17.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7cab2640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:17.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7cab2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:17.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7d2b3640 1 --2- 172.21.15.12:0/360984011 >> v2:172.21.15.71:3300/0 conn(0x7f4d78137d50 0x7f4d7809d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:17.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7d2b3640 10 monclient(hunting): get_auth_request con 0x7f4d78137d50 auth_method 0 2024-09-11T01:29:17.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7d2b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:17.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7d2b3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:17.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7d2b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:17.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7cab2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:17.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7cab2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:17.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7cab2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:17.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7d2b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:17.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7d2b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:17.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7d2b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:17.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7cab2640 10 monclient(hunting): handle_auth_done global_id 4220 payload 274 2024-09-11T01:29:17.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7cab2640 1 -- 172.21.15.12:0/360984011 >> v2:172.21.15.71:3300/0 conn(0x7f4d78137d50 msgr2=0x7f4d7809d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:17.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7cab2640 1 --2- 172.21.15.12:0/360984011 >> v2:172.21.15.71:3300/0 conn(0x7f4d78137d50 0x7f4d7809d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:17.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7cab2640 1 -- 172.21.15.12:0/360984011 >> v1:172.21.15.71:6789/0 conn(0x7f4d78136cd0 legacy=0x7f4d78137120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:17.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7cab2640 10 monclient: _finish_hunting 0 2024-09-11T01:29:17.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7cab2640 1 monclient: found mon.noname-b 2024-09-11T01:29:17.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7cab2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:17.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7cab2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:29:17.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.687+0000 7f4d7cab2640 1 -- 172.21.15.12:0/360984011 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d780a6ce0 con 0x7f4d78138130 2024-09-11T01:29:17.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.690+0000 7f4d7cab2640 1 -- 172.21.15.12:0/360984011 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4d78138130 msgr2=0x7f4d78138550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:17.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.690+0000 7f4d7cab2640 1 --2- 172.21.15.12:0/360984011 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4d78138130 0x7f4d78138550 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4d7809c7d0 tx=0x7f4d6c02a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65dcc54d6ac961b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:17.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.690+0000 7f4d77fff640 1 -- 172.21.15.12:0/360984011 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c037070 con 0x7f4d78138130 2024-09-11T01:29:17.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.690+0000 7f4d77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:17.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.690+0000 7f4d77fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:17.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.690+0000 7f4d77fff640 10 monclient: dump: 2024-09-11T01:29:17.700 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:17.701 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:17.701 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:17.701 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:17.701 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:17.701 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:17.701 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:17.701 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:17.701 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:17.701 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:17.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.691+0000 7f4d77fff640 10 monclient: _finish_auth 0 2024-09-11T01:29:17.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.691+0000 7f4d77fff640 10 monclient: _check_auth_tickets 2024-09-11T01:29:17.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.691+0000 7f4d77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:17.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.691+0000 7f4d7f517640 5 monclient: authenticate success, global_id 4220 2024-09-11T01:29:17.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.691+0000 7f4d77fff640 1 -- 172.21.15.12:0/360984011 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d6c032640 con 0x7f4d78138130 2024-09-11T01:29:17.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.691+0000 7f4d7f517640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:17.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.691+0000 7f4d77fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:17.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.691+0000 7f4d7f517640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:17.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.691+0000 7f4d7f517640 4 set_mon_vals no callback set 2024-09-11T01:29:17.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.691+0000 7f4d77fff640 1 -- 172.21.15.12:0/360984011 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c032a10 con 0x7f4d78138130 2024-09-11T01:29:17.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.691+0000 7f4d77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:17.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.691+0000 7f4d77fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:17.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.691+0000 7f4d77fff640 10 monclient: dump: 2024-09-11T01:29:17.702 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:17.702 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:17.702 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:17.703 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:17.703 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:17.703 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:17.703 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:17.703 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:17.703 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:17.703 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:17.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.691+0000 7f4d7f517640 10 monclient: shutdown 2024-09-11T01:29:17.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.691+0000 7f4d7f517640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:17.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.691+0000 7f4d7f517640 1 -- 172.21.15.12:0/360984011 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4d78138130 msgr2=0x7f4d78138550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:17.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.691+0000 7f4d7f517640 1 --2- 172.21.15.12:0/360984011 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4d78138130 0x7f4d78138550 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4d7809c7d0 tx=0x7f4d6c02a110 comp rx=0 tx=0).stop 2024-09-11T01:29:17.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.691+0000 7f4d7f517640 1 -- 172.21.15.12:0/360984011 shutdown_connections 2024-09-11T01:29:17.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.691+0000 7f4d7f517640 1 --2- 172.21.15.12:0/360984011 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4d78138130 0x7f4d78138550 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:17.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.691+0000 7f4d7f517640 1 --2- 172.21.15.12:0/360984011 >> v2:172.21.15.71:3300/0 conn(0x7f4d78137d50 0x7f4d7809d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:17.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.691+0000 7f4d7f517640 1 -- 172.21.15.12:0/360984011 >> 172.21.15.12:0/360984011 conn(0x7f4d780a7020 msgr2=0x7f4d780a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:17.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.691+0000 7f4d7f517640 1 -- 172.21.15.12:0/360984011 shutdown_connections 2024-09-11T01:29:17.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.691+0000 7f4d7f517640 1 -- 172.21.15.12:0/360984011 wait complete. 2024-09-11T01:29:17.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.692+0000 7f4d7f517640 10 monclient: build_initial_monmap 2024-09-11T01:29:17.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.692+0000 7f4d7f517640 10 monclient: monmap: 2024-09-11T01:29:17.704 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:17.704 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:17.704 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:17.704 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:17.704 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:17.704 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:17.704 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:17.704 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:17.705 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:17.705 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:17.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.692+0000 7f4d7f517640 1 Processor -- start 2024-09-11T01:29:17.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.692+0000 7f4d7f517640 1 -- start start 2024-09-11T01:29:17.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.692+0000 7f4d7f517640 10 monclient: init 2024-09-11T01:29:17.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.692+0000 7f4d7f517640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:17.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.692+0000 7f4d7f517640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:29:17.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7f517640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4d78136cd0 0x7f4d781cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:17.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7f517640 10 monclient(hunting): picked mon.noname-c con 0x7f4d78136cd0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:17.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7f517640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4d78137d50 0x7f4d781cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:17.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7f517640 10 monclient(hunting): picked mon.noname-a con 0x7f4d78137d50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:17.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7f517640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d78138130 0x7f4d78123540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:17.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7f517640 10 monclient(hunting): picked mon.noname-b con 0x7f4d78138130 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:17.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7f517640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:17.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7f517640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7809d650 con 0x7f4d78137d50 2024-09-11T01:29:17.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7f517640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:17.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7f517640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7806f710 con 0x7f4d78138130 2024-09-11T01:29:17.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7f517640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:17.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7f517640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4d7809d7d0 con 0x7f4d78136cd0 2024-09-11T01:29:17.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7f517640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:17.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7f517640 10 monclient(hunting): authenticate will time out at 764.135089s 2024-09-11T01:29:17.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7d2b3640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4d78136cd0 0x7f4d781cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:17.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7d2b3640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4d78136cd0 0x7f4d781cd3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51436/0 (socket says 172.21.15.12:51436) 2024-09-11T01:29:17.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7d2b3640 1 -- 172.21.15.12:0/2921513871 learned_addr learned my addr 172.21.15.12:0/2921513871 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:17.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7d2b3640 10 monclient(hunting): get_auth_request con 0x7f4d78136cd0 auth_method 0 2024-09-11T01:29:17.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7d2b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:17.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7d2b3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:17.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7d2b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:17.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7cab2640 1 --2- 172.21.15.12:0/2921513871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4d78137d50 0x7f4d781cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:17.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7d2b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:17.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7d2b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:17.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7d2b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:17.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7cab2640 10 monclient(hunting): get_auth_request con 0x7f4d78137d50 auth_method 0 2024-09-11T01:29:17.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7cab2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:17.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7cab2640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:17.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7cab2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:17.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7cab2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:17.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7cab2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:17.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7cab2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:17.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7d2b3640 10 monclient(hunting): handle_auth_done global_id 4223 payload 824 2024-09-11T01:29:17.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7d2b3640 1 -- 172.21.15.12:0/2921513871 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d78138130 msgr2=0x7f4d78123540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:17.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7d2b3640 1 --2- 172.21.15.12:0/2921513871 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d78138130 0x7f4d78123540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:17.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7d2b3640 1 -- 172.21.15.12:0/2921513871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4d78137d50 msgr2=0x7f4d781cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:17.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7d2b3640 1 --2- 172.21.15.12:0/2921513871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4d78137d50 0x7f4d781cd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:17.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7d2b3640 10 monclient: _finish_hunting 0 2024-09-11T01:29:17.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7d2b3640 1 monclient: found mon.noname-c 2024-09-11T01:29:17.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7d2b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:17.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7d2b3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:29:17.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.693+0000 7f4d7d2b3640 1 -- 172.21.15.12:0/2921513871 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d78127d40 con 0x7f4d78136cd0 2024-09-11T01:29:17.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.695+0000 7f4d7d2b3640 1 -- 172.21.15.12:0/2921513871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4d78136cd0 msgr2=0x7f4d781cd3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:17.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.695+0000 7f4d7d2b3640 1 --2- 172.21.15.12:0/2921513871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4d78136cd0 0x7f4d781cd3b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4d7000e9b0 tx=0x7f4d7000ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:17.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.695+0000 7f4d75ffb640 1 -- 172.21.15.12:0/2921513871 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d70017070 con 0x7f4d78136cd0 2024-09-11T01:29:17.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.695+0000 7f4d75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:17.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.695+0000 7f4d75ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:17.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.695+0000 7f4d75ffb640 10 monclient: dump: 2024-09-11T01:29:17.709 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:17.709 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:17.709 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:17.709 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:17.709 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:17.709 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:17.709 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:17.709 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:17.709 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:17.709 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:17.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.695+0000 7f4d75ffb640 10 monclient: _finish_auth 0 2024-09-11T01:29:17.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.695+0000 7f4d75ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:29:17.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.695+0000 7f4d75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:17.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.695+0000 7f4d75ffb640 1 -- 172.21.15.12:0/2921513871 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d7000cbb0 con 0x7f4d78136cd0 2024-09-11T01:29:17.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.695+0000 7f4d75ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:17.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.695+0000 7f4d75ffb640 1 -- 172.21.15.12:0/2921513871 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d70013470 con 0x7f4d78136cd0 2024-09-11T01:29:17.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.695+0000 7f4d75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:17.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.695+0000 7f4d75ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:17.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.695+0000 7f4d75ffb640 10 monclient: dump: 2024-09-11T01:29:17.710 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:17.710 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:17.710 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:17.710 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:17.711 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:17.711 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:17.711 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:17.711 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:17.711 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:17.711 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:17.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.695+0000 7f4d7f517640 5 monclient: authenticate success, global_id 4223 2024-09-11T01:29:17.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.695+0000 7f4d7f517640 10 monclient: _renew_subs 2024-09-11T01:29:17.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.695+0000 7f4d7f517640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:17.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.695+0000 7f4d7f517640 1 -- 172.21.15.12:0/2921513871 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d780a1710 con 0x7f4d78136cd0 2024-09-11T01:29:17.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.695+0000 7f4d777fe640 4 set_mon_vals no callback set 2024-09-11T01:29:17.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.696+0000 7f4d7f517640 10 monclient: _renew_subs 2024-09-11T01:29:17.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.696+0000 7f4d7f517640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:17.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.696+0000 7f4d7f517640 1 -- 172.21.15.12:0/2921513871 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d78136920 con 0x7f4d78136cd0 2024-09-11T01:29:17.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.696+0000 7f4d75ffb640 1 -- 172.21.15.12:0/2921513871 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f4d70024060 con 0x7f4d78136cd0 2024-09-11T01:29:17.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.696+0000 7f4d7f517640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:17.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.696+0000 7f4d7f517640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:17.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.696+0000 7f4d7f517640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:17.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.697+0000 7f4d7f517640 1 -- 172.21.15.12:0/2921513871 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d781379d0 con 0x7f4d78136cd0 2024-09-11T01:29:17.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.697+0000 7f4d75ffb640 1 --2- 172.21.15.12:0/2921513871 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4d6403e260 0x7f4d64040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:17.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.697+0000 7f4d75ffb640 1 -- 172.21.15.12:0/2921513871 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4402+0+0 (secure 0 0 0) 0x7f4d70052c10 con 0x7f4d78136cd0 2024-09-11T01:29:17.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.697+0000 7f4d7cab2640 1 --2- 172.21.15.12:0/2921513871 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4d6403e260 0x7f4d64040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:17.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.697+0000 7f4d7cab2640 10 monclient: get_auth_request con 0x7f4d6403e260 auth_method 0 2024-09-11T01:29:17.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.700+0000 7f4d7cab2640 1 --2- 172.21.15.12:0/2921513871 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4d6403e260 0x7f4d64040720 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4d7809c070 tx=0x7f4d6c02e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:17.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.702+0000 7f4d75ffb640 1 -- 172.21.15.12:0/2921513871 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4d780a1710 con 0x7f4d78136cd0 2024-09-11T01:29:17.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.702+0000 7f4d75ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:17.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.702+0000 7f4d75ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:17.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.886+0000 7f4d7f517640 10 start_mon_command cmd=[{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-11T01:29:17.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.886+0000 7f4d7f517640 10 monclient: _send_command 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-11T01:29:17.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.886+0000 7f4d7f517640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:17.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.886+0000 7f4d7f517640 1 -- 172.21.15.12:0/2921513871 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f4d78125000 con 0x7f4d78136cd0 2024-09-11T01:29:17.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.887+0000 7f4d75ffb640 1 -- 172.21.15.12:0/2921513871 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v5) ==== 86+0+159 (secure 0 0 0) 0x7f4d78127d40 con 0x7f4d78136cd0 2024-09-11T01:29:17.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.887+0000 7f4d75ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-11T01:29:17.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.887+0000 7f4d75ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:17.889 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:29:17.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.889+0000 7f4d7f517640 1 -- 172.21.15.12:0/2921513871 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4d6403e260 msgr2=0x7f4d64040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:17.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.889+0000 7f4d7f517640 1 --2- 172.21.15.12:0/2921513871 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4d6403e260 0x7f4d64040720 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4d7809c070 tx=0x7f4d6c02e000 comp rx=0 tx=0).stop 2024-09-11T01:29:17.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.889+0000 7f4d7f517640 10 monclient: shutdown 2024-09-11T01:29:17.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.889+0000 7f4d7f517640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:17.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.889+0000 7f4d7f517640 1 -- 172.21.15.12:0/2921513871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4d78136cd0 msgr2=0x7f4d781cd3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:17.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.889+0000 7f4d7f517640 1 --2- 172.21.15.12:0/2921513871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4d78136cd0 0x7f4d781cd3b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4d7000e9b0 tx=0x7f4d7000ee70 comp rx=0 tx=0).stop 2024-09-11T01:29:17.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.889+0000 7f4d7f517640 1 -- 172.21.15.12:0/2921513871 shutdown_connections 2024-09-11T01:29:17.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.889+0000 7f4d7f517640 1 --2- 172.21.15.12:0/2921513871 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d78138130 0x7f4d78123540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:17.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.889+0000 7f4d7f517640 1 --2- 172.21.15.12:0/2921513871 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4d6403e260 0x7f4d64040720 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:17.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.889+0000 7f4d7f517640 1 --2- 172.21.15.12:0/2921513871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4d78137d50 0x7f4d781cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:17.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.889+0000 7f4d7f517640 1 --2- 172.21.15.12:0/2921513871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4d78136cd0 0x7f4d781cd3b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:17.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.889+0000 7f4d7f517640 1 -- 172.21.15.12:0/2921513871 >> 172.21.15.12:0/2921513871 conn(0x7f4d780a7020 msgr2=0x7f4d7809e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:17.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.889+0000 7f4d7f517640 1 -- 172.21.15.12:0/2921513871 shutdown_connections 2024-09-11T01:29:17.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:17.889+0000 7f4d7f517640 1 -- 172.21.15.12:0/2921513871 wait complete. 2024-09-11T01:29:17.901 INFO:teuthology.orchestra.run.smithi012.stdout:[{"version":5,"timestamp":"2024-09-11T01:29:13.230800+0000","name":"","changes":[{"name":"osd.5/osd_mclock_max_capacity_iops_ssd","new_value":"5837.034764"}]}] 2024-09-11T01:29:17.901 INFO:tasks.ceph_manager:config epoch is 5 2024-09-11T01:29:17.902 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-11T01:29:17.902 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-11T01:29:17.902 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-11T01:29:18.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.000+0000 7f0684240640 10 monclient: get_monmap_and_config 2024-09-11T01:29:18.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.000+0000 7f0684240640 10 monclient: build_initial_monmap 2024-09-11T01:29:18.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.000+0000 7f0684240640 10 monclient: monmap: 2024-09-11T01:29:18.002 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:18.003 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:18.003 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:18.001695+0000 2024-09-11T01:29:18.003 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:18.001695+0000 2024-09-11T01:29:18.003 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:18.003 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:18.003 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:18.003 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:18.003 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:18.003 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:18.003 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:18.003 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:18.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.001+0000 7f0684240640 1 Processor -- start 2024-09-11T01:29:18.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.001+0000 7f0684240640 1 -- start start 2024-09-11T01:29:18.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.001+0000 7f0684240640 10 monclient: init 2024-09-11T01:29:18.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0684240640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:18.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0684240640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-11T01:29:18.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0684240640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f067c137e50 0x7f067c09d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:18.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0684240640 10 monclient(hunting): picked mon.noname-b con 0x7f067c137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:18.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0684240640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f067c138230 0x7f067c138630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:18.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0684240640 10 monclient(hunting): picked mon.noname-a con 0x7f067c138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:18.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0684240640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f067c134cb0 0x7f067c135120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:18.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0684240640 10 monclient(hunting): picked mon.noname-c con 0x7f067c134cb0 addr v2:172.21.15.71:3300/0 2024-09-11T01:29:18.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0684240640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:18.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0684240640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f067c0a65b0 con 0x7f067c138230 2024-09-11T01:29:18.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0684240640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:18.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0684240640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f067c0a62d0 con 0x7f067c134cb0 2024-09-11T01:29:18.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0684240640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:18.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0684240640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f067c09bb40 con 0x7f067c137e50 2024-09-11T01:29:18.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0684240640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:18.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0684240640 10 monclient(hunting): authenticate will time out at 764.444392s 2024-09-11T01:29:18.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0681fdc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f067c137e50 0x7f067c09d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:18.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0681fdc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f067c137e50 0x7f067c09d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51442/0 (socket says 172.21.15.12:51442) 2024-09-11T01:29:18.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0681fdc640 1 -- 172.21.15.12:0/1356107796 learned_addr learned my addr 172.21.15.12:0/1356107796 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:18.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0681fdc640 10 monclient(hunting): get_auth_request con 0x7f067c137e50 auth_method 0 2024-09-11T01:29:18.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0681fdc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:18.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0681fdc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:18.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0681fdc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:18.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f06817db640 1 --2- 172.21.15.12:0/1356107796 >> v2:172.21.15.12:3300/0 conn(0x7f067c138230 0x7f067c138630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:18.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f06817db640 10 monclient(hunting): get_auth_request con 0x7f067c138230 auth_method 0 2024-09-11T01:29:18.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f06817db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:18.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f06817db640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:18.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f06817db640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:18.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f06827dd640 1 --2- 172.21.15.12:0/1356107796 >> v2:172.21.15.71:3300/0 conn(0x7f067c134cb0 0x7f067c135120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:18.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0681fdc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:18.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0681fdc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:18.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0681fdc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:18.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f06827dd640 10 monclient(hunting): get_auth_request con 0x7f067c134cb0 auth_method 0 2024-09-11T01:29:18.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f06827dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:18.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f06827dd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:18.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f06827dd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:18.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f06817db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:18.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f06817db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:18.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f06817db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:18.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f06827dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:18.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f06827dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:18.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f06827dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:18.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0681fdc640 10 monclient(hunting): handle_auth_done global_id 4226 payload 274 2024-09-11T01:29:18.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0681fdc640 1 -- 172.21.15.12:0/1356107796 >> v2:172.21.15.71:3300/0 conn(0x7f067c134cb0 msgr2=0x7f067c135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:18.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0681fdc640 1 --2- 172.21.15.12:0/1356107796 >> v2:172.21.15.71:3300/0 conn(0x7f067c134cb0 0x7f067c135120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:18.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0681fdc640 1 -- 172.21.15.12:0/1356107796 >> v2:172.21.15.12:3300/0 conn(0x7f067c138230 msgr2=0x7f067c138630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:18.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0681fdc640 1 --2- 172.21.15.12:0/1356107796 >> v2:172.21.15.12:3300/0 conn(0x7f067c138230 0x7f067c138630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:18.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0681fdc640 10 monclient: _finish_hunting 0 2024-09-11T01:29:18.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0681fdc640 1 monclient: found mon.noname-b 2024-09-11T01:29:18.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0681fdc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:18.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0681fdc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:29:18.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f0681fdc640 1 -- 172.21.15.12:0/1356107796 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f067c09c4a0 con 0x7f067c137e50 2024-09-11T01:29:18.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.002+0000 7f06817db640 1 --2- 172.21.15.12:0/1356107796 >> v2:172.21.15.12:3300/0 conn(0x7f067c138230 0x7f067c138630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:18.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.005+0000 7f0681fdc640 1 -- 172.21.15.12:0/1356107796 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f067c137e50 msgr2=0x7f067c09d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:18.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.005+0000 7f0681fdc640 1 --2- 172.21.15.12:0/1356107796 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f067c137e50 0x7f067c09d160 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f067c0a8360 tx=0x7f067402a0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bb25e9527f7f0e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:18.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.005+0000 7f0680fda640 1 -- 172.21.15.12:0/1356107796 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0674037070 con 0x7f067c137e50 2024-09-11T01:29:18.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.005+0000 7f0680fda640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:18.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.005+0000 7f0680fda640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:18.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.005+0000 7f0680fda640 10 monclient: dump: 2024-09-11T01:29:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:18.016 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.005+0000 7f0680fda640 10 monclient: _finish_auth 0 2024-09-11T01:29:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.005+0000 7f0680fda640 10 monclient: _check_auth_tickets 2024-09-11T01:29:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.005+0000 7f0680fda640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.005+0000 7f0680fda640 1 -- 172.21.15.12:0/1356107796 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0674032640 con 0x7f067c137e50 2024-09-11T01:29:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.005+0000 7f0684240640 5 monclient: authenticate success, global_id 4226 2024-09-11T01:29:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.005+0000 7f0684240640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.005+0000 7f0680fda640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.005+0000 7f0684240640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.005+0000 7f0680fda640 1 -- 172.21.15.12:0/1356107796 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0674032a10 con 0x7f067c137e50 2024-09-11T01:29:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.005+0000 7f0684240640 4 set_mon_vals no callback set 2024-09-11T01:29:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.005+0000 7f0680fda640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.005+0000 7f0680fda640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.005+0000 7f0680fda640 10 monclient: dump: 2024-09-11T01:29:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:18.018 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:18.018 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:18.018 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:18.018 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:18.018 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:18.018 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:18.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.006+0000 7f0684240640 10 monclient: shutdown 2024-09-11T01:29:18.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.006+0000 7f0684240640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:18.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.006+0000 7f0684240640 1 -- 172.21.15.12:0/1356107796 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f067c137e50 msgr2=0x7f067c09d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:18.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.006+0000 7f0684240640 1 --2- 172.21.15.12:0/1356107796 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f067c137e50 0x7f067c09d160 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f067c0a8360 tx=0x7f067402a0b0 comp rx=0 tx=0).stop 2024-09-11T01:29:18.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.006+0000 7f0684240640 1 -- 172.21.15.12:0/1356107796 shutdown_connections 2024-09-11T01:29:18.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.006+0000 7f0684240640 1 --2- 172.21.15.12:0/1356107796 >> v2:172.21.15.71:3300/0 conn(0x7f067c134cb0 0x7f067c135120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:18.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.006+0000 7f0684240640 1 --2- 172.21.15.12:0/1356107796 >> v2:172.21.15.12:3300/0 conn(0x7f067c138230 0x7f067c138630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:18.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.006+0000 7f0684240640 1 --2- 172.21.15.12:0/1356107796 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f067c137e50 0x7f067c09d160 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:18.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.006+0000 7f0684240640 1 -- 172.21.15.12:0/1356107796 >> 172.21.15.12:0/1356107796 conn(0x7f067c0a7020 msgr2=0x7f067c0a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:18.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.006+0000 7f0684240640 1 -- 172.21.15.12:0/1356107796 shutdown_connections 2024-09-11T01:29:18.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.006+0000 7f0684240640 1 -- 172.21.15.12:0/1356107796 wait complete. 2024-09-11T01:29:18.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.007+0000 7f0684240640 10 monclient: build_initial_monmap 2024-09-11T01:29:18.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.007+0000 7f0684240640 10 monclient: monmap: 2024-09-11T01:29:18.019 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:18.019 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:18.019 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:18.019 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:18.019 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:18.019 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:18.019 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:18.019 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:18.019 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:18.020 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:18.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.007+0000 7f0684240640 1 Processor -- start 2024-09-11T01:29:18.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.007+0000 7f0684240640 1 -- start start 2024-09-11T01:29:18.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.007+0000 7f0684240640 10 monclient: init 2024-09-11T01:29:18.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.007+0000 7f0684240640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:18.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.007+0000 7f0684240640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:29:18.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.007+0000 7f0684240640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f067c134cb0 0x7f067c1cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:18.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.007+0000 7f0684240640 10 monclient(hunting): picked mon.noname-a con 0x7f067c134cb0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:18.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.007+0000 7f0684240640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f067c137e50 0x7f067c1cd910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:18.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.007+0000 7f0684240640 10 monclient(hunting): picked mon.noname-b con 0x7f067c137e50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:18.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.007+0000 7f0684240640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f067c138230 0x7f067c121750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:18.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.007+0000 7f0684240640 10 monclient(hunting): picked mon.noname-c con 0x7f067c138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:18.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.007+0000 7f0684240640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:18.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.007+0000 7f0684240640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f067c06f710 con 0x7f067c134cb0 2024-09-11T01:29:18.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.007+0000 7f0684240640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:18.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.007+0000 7f0684240640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f067c138b70 con 0x7f067c137e50 2024-09-11T01:29:18.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.007+0000 7f0684240640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:18.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.007+0000 7f0684240640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f067c06f890 con 0x7f067c138230 2024-09-11T01:29:18.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.007+0000 7f0684240640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:18.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.007+0000 7f0684240640 10 monclient(hunting): authenticate will time out at 764.449986s 2024-09-11T01:29:18.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f0681fdc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f067c134cb0 0x7f067c1cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:18.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f0681fdc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f067c134cb0 0x7f067c1cd3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54788/0 (socket says 172.21.15.12:54788) 2024-09-11T01:29:18.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f0681fdc640 1 -- 172.21.15.12:0/1179616816 learned_addr learned my addr 172.21.15.12:0/1179616816 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:18.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f0681fdc640 10 monclient(hunting): get_auth_request con 0x7f067c134cb0 auth_method 0 2024-09-11T01:29:18.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f0681fdc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:18.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f0681fdc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:18.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f0681fdc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:18.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f06827dd640 1 --2- 172.21.15.12:0/1179616816 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f067c138230 0x7f067c121750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:18.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f06827dd640 10 monclient(hunting): get_auth_request con 0x7f067c138230 auth_method 0 2024-09-11T01:29:18.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f06827dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:18.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f06827dd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:18.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f06827dd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:18.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f0681fdc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:18.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f0681fdc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:18.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f0681fdc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:18.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f06817db640 1 --2- 172.21.15.12:0/1179616816 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f067c137e50 0x7f067c1cd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:18.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f06827dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:18.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f06827dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:18.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f06827dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:18.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f06817db640 10 monclient(hunting): get_auth_request con 0x7f067c137e50 auth_method 0 2024-09-11T01:29:18.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f06817db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:18.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f06817db640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:18.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f06817db640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:18.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f06817db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:18.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f06817db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:18.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f06817db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:18.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f0681fdc640 10 monclient(hunting): handle_auth_done global_id 4221 payload 824 2024-09-11T01:29:18.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f0681fdc640 1 -- 172.21.15.12:0/1179616816 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f067c138230 msgr2=0x7f067c121750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:18.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f0681fdc640 1 --2- 172.21.15.12:0/1179616816 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f067c138230 0x7f067c121750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:18.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f0681fdc640 1 -- 172.21.15.12:0/1179616816 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f067c137e50 msgr2=0x7f067c1cd910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:18.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f0681fdc640 1 --2- 172.21.15.12:0/1179616816 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f067c137e50 0x7f067c1cd910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:18.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f0681fdc640 10 monclient: _finish_hunting 0 2024-09-11T01:29:18.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f0681fdc640 1 monclient: found mon.noname-a 2024-09-11T01:29:18.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f0681fdc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:18.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f0681fdc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:18.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.008+0000 7f0681fdc640 1 -- 172.21.15.12:0/1179616816 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f067c125250 con 0x7f067c134cb0 2024-09-11T01:29:18.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.010+0000 7f0681fdc640 1 -- 172.21.15.12:0/1179616816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f067c134cb0 msgr2=0x7f067c1cd3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:18.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.010+0000 7f0681fdc640 1 --2- 172.21.15.12:0/1179616816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f067c134cb0 0x7f067c1cd3d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f06740090b0 tx=0x7f06740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:18.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.010+0000 7f067a7fc640 1 -- 172.21.15.12:0/1179616816 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0674041020 con 0x7f067c134cb0 2024-09-11T01:29:18.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.010+0000 7f067a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:18.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.010+0000 7f067a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:18.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.010+0000 7f067a7fc640 10 monclient: dump: 2024-09-11T01:29:18.024 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:18.025 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.010+0000 7f067a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:29:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.010+0000 7f067a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:29:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.010+0000 7f067a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.010+0000 7f0684240640 5 monclient: authenticate success, global_id 4221 2024-09-11T01:29:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.010+0000 7f067a7fc640 1 -- 172.21.15.12:0/1179616816 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f067403d830 con 0x7f067c134cb0 2024-09-11T01:29:18.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.010+0000 7f067a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:18.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.010+0000 7f067a7fc640 1 -- 172.21.15.12:0/1179616816 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0674037040 con 0x7f067c134cb0 2024-09-11T01:29:18.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.010+0000 7f0684240640 10 monclient: _renew_subs 2024-09-11T01:29:18.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.010+0000 7f0684240640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:18.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.010+0000 7f0684240640 1 -- 172.21.15.12:0/1179616816 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f067c0a5c00 con 0x7f067c134cb0 2024-09-11T01:29:18.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.010+0000 7f067a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:18.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.010+0000 7f067a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:18.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.010+0000 7f067a7fc640 10 monclient: dump: 2024-09-11T01:29:18.026 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:18.026 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:18.026 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:18.026 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:18.026 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:18.026 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:18.026 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:18.027 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:18.027 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:18.027 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:18.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.010+0000 7f067bfff640 4 set_mon_vals no callback set 2024-09-11T01:29:18.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.010+0000 7f0684240640 10 monclient: _renew_subs 2024-09-11T01:29:18.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.010+0000 7f0684240640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:18.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.010+0000 7f0684240640 1 -- 172.21.15.12:0/1179616816 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f067c123a80 con 0x7f067c134cb0 2024-09-11T01:29:18.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.011+0000 7f0681fdc640 1 --2- 172.21.15.12:0/1179616816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f067c134cb0 0x7f067c1cd3d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f06740090b0 tx=0x7f06740046c0 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7f067403a030 mgrmap(e 5) 2024-09-11T01:29:18.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.011+0000 7f067a7fc640 1 -- 172.21.15.12:0/1179616816 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f067403a030 con 0x7f067c134cb0 2024-09-11T01:29:18.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.011+0000 7f0684240640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:18.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.011+0000 7f0684240640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:18.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.011+0000 7f0684240640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:18.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.011+0000 7f0684240640 1 -- 172.21.15.12:0/1179616816 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f067c138630 con 0x7f067c134cb0 2024-09-11T01:29:18.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.012+0000 7f067a7fc640 1 --2- 172.21.15.12:0/1179616816 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f066803e210 0x7f06680406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:18.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.012+0000 7f06817db640 1 --2- 172.21.15.12:0/1179616816 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f066803e210 0x7f06680406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:18.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.012+0000 7f06817db640 10 monclient: get_auth_request con 0x7f066803e210 auth_method 0 2024-09-11T01:29:18.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.014+0000 7f06817db640 1 --2- 172.21.15.12:0/1179616816 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f066803e210 0x7f06680406d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0670009560 tx=0x7f0670009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:18.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.017+0000 7f067a7fc640 1 -- 172.21.15.12:0/1179616816 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4402+0+0 (secure 0 0 0) 0x7f06740730a0 con 0x7f067c134cb0 2024-09-11T01:29:18.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.840+0000 7f067a7fc640 1 -- 172.21.15.12:0/1179616816 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f067404b030 con 0x7f067c134cb0 2024-09-11T01:29:18.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.840+0000 7f067a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:18.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:18.840+0000 7f067a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:19.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.077+0000 7f0684240640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-11T01:29:19.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.077+0000 7f0684240640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-11T01:29:19.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.077+0000 7f0684240640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:19.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.077+0000 7f0684240640 1 -- 172.21.15.12:0/1179616816 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f067c0a5820 con 0x7f067c134cb0 2024-09-11T01:29:19.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.079+0000 7f067a7fc640 1 -- 172.21.15.12:0/1179616816 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+97365 (secure 0 0 0) 0x7f067403a350 con 0x7f067c134cb0 2024-09-11T01:29:19.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.079+0000 7f067a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-11T01:29:19.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.079+0000 7f067a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:19.082 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:29:19.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.081+0000 7f0684240640 1 -- 172.21.15.12:0/1179616816 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f066803e210 msgr2=0x7f06680406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.081+0000 7f0684240640 1 --2- 172.21.15.12:0/1179616816 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f066803e210 0x7f06680406d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0670009560 tx=0x7f0670009190 comp rx=0 tx=0).stop 2024-09-11T01:29:19.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.081+0000 7f0684240640 10 monclient: shutdown 2024-09-11T01:29:19.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.081+0000 7f0684240640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:19.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.081+0000 7f0684240640 1 -- 172.21.15.12:0/1179616816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f067c134cb0 msgr2=0x7f067c1cd3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.081+0000 7f0684240640 1 --2- 172.21.15.12:0/1179616816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f067c134cb0 0x7f067c1cd3d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f06740090b0 tx=0x7f06740046c0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.081+0000 7f0684240640 1 -- 172.21.15.12:0/1179616816 shutdown_connections 2024-09-11T01:29:19.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.081+0000 7f0684240640 1 --2- 172.21.15.12:0/1179616816 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f067c138230 0x7f067c121750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.081+0000 7f0684240640 1 --2- 172.21.15.12:0/1179616816 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f067c137e50 0x7f067c1cd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.081+0000 7f0684240640 1 --2- 172.21.15.12:0/1179616816 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f066803e210 0x7f06680406d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.081+0000 7f0684240640 1 --2- 172.21.15.12:0/1179616816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f067c134cb0 0x7f067c1cd3d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.081+0000 7f0684240640 1 -- 172.21.15.12:0/1179616816 >> 172.21.15.12:0/1179616816 conn(0x7f067c0a7020 msgr2=0x7f067c1375d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:19.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.081+0000 7f0684240640 1 -- 172.21.15.12:0/1179616816 shutdown_connections 2024-09-11T01:29:19.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.081+0000 7f0684240640 1 -- 172.21.15.12:0/1179616816 wait complete. 2024-09-11T01:29:19.093 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":3189259519},{"type":"v1","addr":"172.21.15.71:6817","nonce":3189259519}]},"active_addr":"172.21.15.71:6817/3189259519","active_change":"2024-09-11T01:29:06.842428+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":3384958998}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":1315289166}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":1047779171}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":482846230}]}]} 2024-09-11T01:29:19.096 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-11T01:29:19.096 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-11T01:29:19.096 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T01:29:19.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.195+0000 7f1622af6640 10 monclient: get_monmap_and_config 2024-09-11T01:29:19.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.195+0000 7f1622af6640 10 monclient: build_initial_monmap 2024-09-11T01:29:19.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.195+0000 7f1622af6640 10 monclient: monmap: 2024-09-11T01:29:19.197 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:19.197 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.197 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:19.195783+0000 2024-09-11T01:29:19.197 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:19.195783+0000 2024-09-11T01:29:19.197 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:19.197 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.197 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:19.197 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:19.197 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:19.197 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:19.197 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:19.198 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.195+0000 7f1622af6640 1 Processor -- start 2024-09-11T01:29:19.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.195+0000 7f1622af6640 1 -- start start 2024-09-11T01:29:19.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.195+0000 7f1622af6640 10 monclient: init 2024-09-11T01:29:19.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1622af6640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:19.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1622af6640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-11T01:29:19.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1622af6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f161c137d50 0x7f161c09d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1622af6640 10 monclient(hunting): picked mon.noname-b con 0x7f161c137d50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:19.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1622af6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f161c138130 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:19.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1622af6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f161c13d280 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:19.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1622af6640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f161c09ba20 con 0x7f161c138130 2024-09-11T01:29:19.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1622af6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f161c09c370 con 0x7f161c13d280 2024-09-11T01:29:19.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1622af6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1622af6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f161c09c650 con 0x7f161c137d50 2024-09-11T01:29:19.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1622af6640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:19.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1622af6640 10 monclient(hunting): authenticate will time out at 765.638357s 2024-09-11T01:29:19.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1620892640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f161c137d50 0x7f161c09d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f161bfff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f161c138130 0x7f161c138550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55992/0 (socket says 172.21.15.12:55992) 2024-09-11T01:29:19.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1620892640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f161c137d50 0x7f161c09d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51466/0 (socket says 172.21.15.12:51466) 2024-09-11T01:29:19.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f161bfff640 1 -- 172.21.15.12:0/2925937341 learned_addr learned my addr 172.21.15.12:0/2925937341 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:29:19.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1620892640 10 monclient(hunting): get_auth_request con 0x7f161c137d50 auth_method 0 2024-09-11T01:29:19.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1620892640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:19.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1620892640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:19.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1620892640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:19.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1620892640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:19.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1620892640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:19.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1620892640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:19.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1620892640 10 monclient(hunting): handle_auth_done global_id 4232 payload 274 2024-09-11T01:29:19.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1620892640 1 -- 172.21.15.12:0/2925937341 >> v1:172.21.15.71:6789/0 conn(0x7f161c13d280 legacy=0x7f161c13f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:19.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1620892640 1 -- 172.21.15.12:0/2925937341 >> v1:172.21.15.12:6789/0 conn(0x7f161c138130 legacy=0x7f161c138550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:19.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1620892640 10 monclient: _finish_hunting 0 2024-09-11T01:29:19.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1620892640 1 monclient: found mon.noname-b 2024-09-11T01:29:19.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1620892640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:19.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1620892640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:29:19.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.196+0000 7f1620892640 1 -- 172.21.15.12:0/2925937341 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f161c0a6ce0 con 0x7f161c137d50 2024-09-11T01:29:19.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.199+0000 7f1620892640 1 -- 172.21.15.12:0/2925937341 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f161c137d50 msgr2=0x7f161c09d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:19.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.199+0000 7f1620892640 1 --2- 172.21.15.12:0/2925937341 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f161c137d50 0x7f161c09d110 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f16140097d0 tx=0x7f161402a030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3314c29263943c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:19.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.199+0000 7f161b7fe640 1 -- 172.21.15.12:0/2925937341 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1614037070 con 0x7f161c137d50 2024-09-11T01:29:19.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.199+0000 7f161b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:19.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.199+0000 7f161b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:19.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.199+0000 7f161b7fe640 10 monclient: dump: 2024-09-11T01:29:19.205 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:19.205 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.205 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.205 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.205 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:19.205 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.206 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:19.206 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:19.206 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:19.206 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.199+0000 7f161b7fe640 10 monclient: _finish_auth 0 2024-09-11T01:29:19.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.199+0000 7f161b7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:29:19.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.199+0000 7f161b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:19.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.199+0000 7f161b7fe640 1 -- 172.21.15.12:0/2925937341 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1614032640 con 0x7f161c137d50 2024-09-11T01:29:19.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.199+0000 7f161b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:19.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.199+0000 7f161b7fe640 1 -- 172.21.15.12:0/2925937341 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1614032a10 con 0x7f161c137d50 2024-09-11T01:29:19.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.199+0000 7f161b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:19.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.199+0000 7f161b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:19.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.199+0000 7f161b7fe640 10 monclient: dump: 2024-09-11T01:29:19.206 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:19.207 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.207 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.207 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.207 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:19.207 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.207 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:19.207 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:19.207 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:19.207 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.199+0000 7f1622af6640 5 monclient: authenticate success, global_id 4232 2024-09-11T01:29:19.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.199+0000 7f1622af6640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:19.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.199+0000 7f1622af6640 4 set_mon_vals no callback set 2024-09-11T01:29:19.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.200+0000 7f1622af6640 10 monclient: shutdown 2024-09-11T01:29:19.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.200+0000 7f1622af6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:19.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.200+0000 7f1622af6640 1 -- 172.21.15.12:0/2925937341 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f161c137d50 msgr2=0x7f161c09d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.200+0000 7f1622af6640 1 --2- 172.21.15.12:0/2925937341 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f161c137d50 0x7f161c09d110 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f16140097d0 tx=0x7f161402a030 comp rx=0 tx=0).stop 2024-09-11T01:29:19.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.200+0000 7f1622af6640 1 -- 172.21.15.12:0/2925937341 shutdown_connections 2024-09-11T01:29:19.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.200+0000 7f1622af6640 1 --2- 172.21.15.12:0/2925937341 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f161c137d50 0x7f161c09d110 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.200+0000 7f1622af6640 1 -- 172.21.15.12:0/2925937341 >> 172.21.15.12:0/2925937341 conn(0x7f161c0a7020 msgr2=0x7f161c0a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:19.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.200+0000 7f1622af6640 1 -- 172.21.15.12:0/2925937341 shutdown_connections 2024-09-11T01:29:19.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.200+0000 7f1622af6640 1 -- 172.21.15.12:0/2925937341 wait complete. 2024-09-11T01:29:19.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.200+0000 7f1622af6640 10 monclient: build_initial_monmap 2024-09-11T01:29:19.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.200+0000 7f1622af6640 10 monclient: monmap: 2024-09-11T01:29:19.208 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:19.208 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:19.208 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:19.208 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:19.208 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:19.209 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.209 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:19.209 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:19.209 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:19.209 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.201+0000 7f1622af6640 1 Processor -- start 2024-09-11T01:29:19.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.201+0000 7f1622af6640 1 -- start start 2024-09-11T01:29:19.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.201+0000 7f1622af6640 10 monclient: init 2024-09-11T01:29:19.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.201+0000 7f1622af6640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:19.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.201+0000 7f1622af6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:29:19.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.201+0000 7f1622af6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f161c137d50 0x7f161c136b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.201+0000 7f1622af6640 10 monclient(hunting): picked mon.noname-c con 0x7f161c137d50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:19.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.201+0000 7f1622af6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f161c138130 0x7f161c139260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.201+0000 7f1622af6640 10 monclient(hunting): picked mon.noname-b con 0x7f161c138130 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:19.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.201+0000 7f1622af6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f161c13d280 0x7f161c1397a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.201+0000 7f1622af6640 10 monclient(hunting): picked mon.noname-a con 0x7f161c13d280 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:19.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.201+0000 7f1622af6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.201+0000 7f1622af6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f161c09d7d0 con 0x7f161c13d280 2024-09-11T01:29:19.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.201+0000 7f1622af6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.201+0000 7f1622af6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f161c09d650 con 0x7f161c138130 2024-09-11T01:29:19.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.201+0000 7f1622af6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.201+0000 7f1622af6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f161c140d90 con 0x7f161c137d50 2024-09-11T01:29:19.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.201+0000 7f1622af6640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:19.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.201+0000 7f1622af6640 10 monclient(hunting): authenticate will time out at 765.643985s 2024-09-11T01:29:19.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1620892640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f161c137d50 0x7f161c136b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1621093640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f161c13d280 0x7f161c1397a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1620892640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f161c137d50 0x7f161c136b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51482/0 (socket says 172.21.15.12:51482) 2024-09-11T01:29:19.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1620892640 1 -- 172.21.15.12:0/4225912 learned_addr learned my addr 172.21.15.12:0/4225912 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:19.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1620892640 10 monclient(hunting): get_auth_request con 0x7f161c137d50 auth_method 0 2024-09-11T01:29:19.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1620892640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:19.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1620892640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:19.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1620892640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:19.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1621093640 10 monclient(hunting): get_auth_request con 0x7f161c13d280 auth_method 0 2024-09-11T01:29:19.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1621093640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:19.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1621093640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:19.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1621093640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:19.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1620892640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:19.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1620892640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:19.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1620892640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:19.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1621093640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:19.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1621093640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:19.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1621093640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:19.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1620892640 10 monclient(hunting): handle_auth_done global_id 4235 payload 824 2024-09-11T01:29:19.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1620892640 1 -- 172.21.15.12:0/4225912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f161c138130 msgr2=0x7f161c139260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1620892640 1 --2- 172.21.15.12:0/4225912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f161c138130 0x7f161c139260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1620892640 1 -- 172.21.15.12:0/4225912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f161c13d280 msgr2=0x7f161c1397a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1620892640 1 --2- 172.21.15.12:0/4225912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f161c13d280 0x7f161c1397a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1620892640 10 monclient: _finish_hunting 0 2024-09-11T01:29:19.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1620892640 1 monclient: found mon.noname-c 2024-09-11T01:29:19.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1620892640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:19.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1620892640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:29:19.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.202+0000 7f1620892640 1 -- 172.21.15.12:0/4225912 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f161c13cca0 con 0x7f161c137d50 2024-09-11T01:29:19.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.204+0000 7f1620892640 1 -- 172.21.15.12:0/4225912 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f161c137d50 msgr2=0x7f161c136b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:19.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.204+0000 7f1620892640 1 --2- 172.21.15.12:0/4225912 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f161c137d50 0x7f161c136b00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f16140098f0 tx=0x7f16140046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:19.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.204+0000 7f16197fa640 1 -- 172.21.15.12:0/4225912 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1614041070 con 0x7f161c137d50 2024-09-11T01:29:19.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.204+0000 7f16197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:19.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.204+0000 7f16197fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:19.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.204+0000 7f16197fa640 10 monclient: dump: 2024-09-11T01:29:19.215 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:19.215 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.215 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.215 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.215 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:19.215 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.215 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:19.215 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:19.215 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:19.215 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.204+0000 7f16197fa640 10 monclient: _finish_auth 0 2024-09-11T01:29:19.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.204+0000 7f16197fa640 10 monclient: _check_auth_tickets 2024-09-11T01:29:19.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.204+0000 7f16197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:19.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.204+0000 7f16197fa640 1 -- 172.21.15.12:0/4225912 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f161403c880 con 0x7f161c137d50 2024-09-11T01:29:19.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.204+0000 7f16197fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:19.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.204+0000 7f16197fa640 1 -- 172.21.15.12:0/4225912 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1614037040 con 0x7f161c137d50 2024-09-11T01:29:19.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.204+0000 7f1622af6640 5 monclient: authenticate success, global_id 4235 2024-09-11T01:29:19.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.204+0000 7f16197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:19.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.204+0000 7f16197fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:19.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.204+0000 7f16197fa640 10 monclient: dump: 2024-09-11T01:29:19.216 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:19.216 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.216 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.216 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.217 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:19.217 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.217 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:19.217 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:19.217 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:19.217 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.204+0000 7f1622af6640 10 monclient: _renew_subs 2024-09-11T01:29:19.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.204+0000 7f161affd640 4 set_mon_vals no callback set 2024-09-11T01:29:19.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.204+0000 7f1622af6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:19.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.204+0000 7f1622af6640 1 -- 172.21.15.12:0/4225912 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f161c1407b0 con 0x7f161c137d50 2024-09-11T01:29:19.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.204+0000 7f1622af6640 10 monclient: _renew_subs 2024-09-11T01:29:19.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.204+0000 7f1622af6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:19.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.204+0000 7f1622af6640 1 -- 172.21.15.12:0/4225912 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f161c13c6e0 con 0x7f161c137d50 2024-09-11T01:29:19.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.205+0000 7f16197fa640 1 -- 172.21.15.12:0/4225912 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f161403cc70 con 0x7f161c137d50 2024-09-11T01:29:19.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.205+0000 7f1622af6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:19.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.205+0000 7f1622af6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:19.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.205+0000 7f1622af6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:19.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.205+0000 7f1622af6640 1 -- 172.21.15.12:0/4225912 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f161c09d190 con 0x7f161c137d50 2024-09-11T01:29:19.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.206+0000 7f16197fa640 1 --2- 172.21.15.12:0/4225912 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f15fc03e210 0x7f15fc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.206+0000 7f16197fa640 1 -- 172.21.15.12:0/4225912 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4402+0+0 (secure 0 0 0) 0x7f16140738e0 con 0x7f161c137d50 2024-09-11T01:29:19.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.206+0000 7f161bfff640 1 --2- 172.21.15.12:0/4225912 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f15fc03e210 0x7f15fc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.206+0000 7f161bfff640 10 monclient: get_auth_request con 0x7f15fc03e210 auth_method 0 2024-09-11T01:29:19.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.208+0000 7f161bfff640 1 --2- 172.21.15.12:0/4225912 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f15fc03e210 0x7f15fc0406d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f161c13c610 tx=0x7f160c004fb0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:19.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.210+0000 7f16197fa640 1 -- 172.21.15.12:0/4225912 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f161403a360 con 0x7f161c137d50 2024-09-11T01:29:19.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.210+0000 7f16197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:19.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.210+0000 7f16197fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:19.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.394+0000 7f1622af6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-11T01:29:19.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.394+0000 7f1622af6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T01:29:19.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.394+0000 7f1622af6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:19.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.394+0000 7f1622af6640 1 -- 172.21.15.12:0/4225912 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f161c074a40 con 0x7f161c137d50 2024-09-11T01:29:19.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.395+0000 7f16197fa640 1 -- 172.21.15.12:0/4225912 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12373 (secure 0 0 0) 0x7f1614072450 con 0x7f161c137d50 2024-09-11T01:29:19.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.395+0000 7f16197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T01:29:19.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.395+0000 7f16197fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:19.397 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:29:19.397 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":16,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","created":"2024-09-11T01:29:02.540751+0000","modified":"2024-09-11T01:29:17.539307+0000","last_up_change":"2024-09-11T01:29:13.320573+0000","last_in_change":"2024-09-11T01:29:05.075851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-11T01:29:14.157646+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-11T01:29:14.872725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"616ba1d0-adf6-4fe1-85e3-266e53e21867","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":4266003406},{"type":"v1","addr":"172.21.15.12:6817","nonce":4266003406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":4266003406},{"type":"v1","addr":"172.21.15.12:6819","nonce":4266003406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":4266003406},{"type":"v1","addr":"172.21.15.12:6823","nonce":4266003406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":4266003406},{"type":"v1","addr":"172.21.15.12:6821","nonce":4266003406}]},"public_addr":"172.21.15.12:6817/4266003406","cluster_addr":"172.21.15.12:6819/4266003406","heartbeat_back_addr":"172.21.15.12:6823/4266003406","heartbeat_front_addr":"172.21.15.12:6821/4266003406","state":["exists","up"]},{"osd":1,"uuid":"9b570874-7a3d-4867-8c44-29084a8bf649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3563357280},{"type":"v1","addr":"172.21.15.12:6809","nonce":3563357280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3563357280},{"type":"v1","addr":"172.21.15.12:6811","nonce":3563357280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3563357280},{"type":"v1","addr":"172.21.15.12:6815","nonce":3563357280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3563357280},{"type":"v1","addr":"172.21.15.12:6813","nonce":3563357280}]},"public_addr":"172.21.15.12:6809/3563357280","cluster_addr":"172.21.15.12:6811/3563357280","heartbeat_back_addr":"172.21.15.12:6815/3563357280","heartbeat_front_addr":"172.21.15.12:6813/3563357280","state":["exists","up"]},{"osd":2,"uuid":"b12ee288-ce57-41de-8b8e-6fad509311b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2893036867},{"type":"v1","addr":"172.21.15.12:6801","nonce":2893036867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2893036867},{"type":"v1","addr":"172.21.15.12:6803","nonce":2893036867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2893036867},{"type":"v1","addr":"172.21.15.12:6807","nonce":2893036867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2893036867},{"type":"v1","addr":"172.21.15.12:6805","nonce":2893036867}]},"public_addr":"172.21.15.12:6801/2893036867","cluster_addr":"172.21.15.12:6803/2893036867","heartbeat_back_addr":"172.21.15.12:6807/2893036867","heartbeat_front_addr":"172.21.15.12:6805/2893036867","state":["exists","up"]},{"osd":3,"uuid":"a8e81aa4-0dee-487d-981d-1f593e363eee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":3091488141},{"type":"v1","addr":"172.21.15.71:6819","nonce":3091488141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":3091488141},{"type":"v1","addr":"172.21.15.71:6821","nonce":3091488141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":3091488141},{"type":"v1","addr":"172.21.15.71:6825","nonce":3091488141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":3091488141},{"type":"v1","addr":"172.21.15.71:6823","nonce":3091488141}]},"public_addr":"172.21.15.71:6819/3091488141","cluster_addr":"172.21.15.71:6821/3091488141","heartbeat_back_addr":"172.21.15.71:6825/3091488141","heartbeat_front_addr":"172.21.15.71:6823/3091488141","state":["exists","up"]},{"osd":4,"uuid":"9db77181-fc18-408f-a35c-87bf4b4ba2d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":1790415427},{"type":"v1","addr":"172.21.15.71:6801","nonce":1790415427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":1790415427},{"type":"v1","addr":"172.21.15.71:6803","nonce":1790415427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":1790415427},{"type":"v1","addr":"172.21.15.71:6807","nonce":1790415427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":1790415427},{"type":"v1","addr":"172.21.15.71:6805","nonce":1790415427}]},"public_addr":"172.21.15.71:6801/1790415427","cluster_addr":"172.21.15.71:6803/1790415427","heartbeat_back_addr":"172.21.15.71:6807/1790415427","heartbeat_front_addr":"172.21.15.71:6805/1790415427","state":["exists","up"]},{"osd":5,"uuid":"fadf2191-f31a-4880-a529-5b2d1023d236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":2032569317},{"type":"v1","addr":"172.21.15.71:6809","nonce":2032569317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":2032569317},{"type":"v1","addr":"172.21.15.71:6811","nonce":2032569317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":2032569317},{"type":"v1","addr":"172.21.15.71:6815","nonce":2032569317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":2032569317},{"type":"v1","addr":"172.21.15.71:6813","nonce":2032569317}]},"public_addr":"172.21.15.71:6809/2032569317","cluster_addr":"172.21.15.71:6811/2032569317","heartbeat_back_addr":"172.21.15.71:6815/2032569317","heartbeat_front_addr":"172.21.15.71:6813/2032569317","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:11.548818+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:11.310103+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:10.578802+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:11.748432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:11.098916+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:11.150538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T01:29:19.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.397+0000 7f1622af6640 1 -- 172.21.15.12:0/4225912 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f15fc03e210 msgr2=0x7f15fc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.397+0000 7f1622af6640 1 --2- 172.21.15.12:0/4225912 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f15fc03e210 0x7f15fc0406d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f161c13c610 tx=0x7f160c004fb0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.397+0000 7f1622af6640 10 monclient: shutdown 2024-09-11T01:29:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.397+0000 7f1622af6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.397+0000 7f1622af6640 1 -- 172.21.15.12:0/4225912 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f161c137d50 msgr2=0x7f161c136b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.397+0000 7f1622af6640 1 --2- 172.21.15.12:0/4225912 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f161c137d50 0x7f161c136b00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f16140098f0 tx=0x7f16140046c0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.398+0000 7f1622af6640 1 -- 172.21.15.12:0/4225912 shutdown_connections 2024-09-11T01:29:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.398+0000 7f1622af6640 1 --2- 172.21.15.12:0/4225912 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f15fc03e210 0x7f15fc0406d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.398+0000 7f1622af6640 1 --2- 172.21.15.12:0/4225912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f161c13d280 0x7f161c1397a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.398+0000 7f1622af6640 1 --2- 172.21.15.12:0/4225912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f161c138130 0x7f161c139260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.398+0000 7f1622af6640 1 --2- 172.21.15.12:0/4225912 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f161c137d50 0x7f161c136b00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.398+0000 7f1622af6640 1 -- 172.21.15.12:0/4225912 >> 172.21.15.12:0/4225912 conn(0x7f161c0a7020 msgr2=0x7f161c0a2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.398+0000 7f1622af6640 1 -- 172.21.15.12:0/4225912 shutdown_connections 2024-09-11T01:29:19.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.398+0000 7f1622af6640 1 -- 172.21.15.12:0/4225912 wait complete. 2024-09-11T01:29:19.410 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-11T01:29:19.411 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T01:29:19.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.509+0000 7fd389cbd640 10 monclient: get_monmap_and_config 2024-09-11T01:29:19.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.509+0000 7fd389cbd640 10 monclient: build_initial_monmap 2024-09-11T01:29:19.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.509+0000 7fd389cbd640 10 monclient: monmap: 2024-09-11T01:29:19.511 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:19.512 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.512 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:19.510643+0000 2024-09-11T01:29:19.512 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:19.510643+0000 2024-09-11T01:29:19.512 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:19.512 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.512 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:19.512 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:19.512 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:19.512 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:19.512 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:19.513 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.510+0000 7fd389cbd640 1 Processor -- start 2024-09-11T01:29:19.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.510+0000 7fd389cbd640 1 -- start start 2024-09-11T01:29:19.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.510+0000 7fd389cbd640 10 monclient: init 2024-09-11T01:29:19.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd389cbd640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:19.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd389cbd640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-11T01:29:19.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd389cbd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd384137e50 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:19.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd389cbd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd384138230 0x7fd384138640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd389cbd640 10 monclient(hunting): picked mon.noname-a con 0x7fd384138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:19.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd389cbd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd38413d380 0x7fd38413f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd389cbd640 10 monclient(hunting): picked mon.noname-c con 0x7fd38413d380 addr v2:172.21.15.71:3300/0 2024-09-11T01:29:19.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd389cbd640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3840a62d0 con 0x7fd384137e50 2024-09-11T01:29:19.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd389cbd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd389cbd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fd38409bb40 con 0x7fd384138230 2024-09-11T01:29:19.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd389cbd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd389cbd640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fd38409c4a0 con 0x7fd38413d380 2024-09-11T01:29:19.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd389cbd640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:19.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd389cbd640 10 monclient(hunting): authenticate will time out at 765.953335s 2024-09-11T01:29:19.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd382ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd384138230 0x7fd384138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd382ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd384138230 0x7fd384138640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54814/0 (socket says 172.21.15.12:54814) 2024-09-11T01:29:19.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd382ffd640 1 -- 172.21.15.12:0/2423123520 learned_addr learned my addr 172.21.15.12:0/2423123520 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:19.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd382ffd640 10 monclient(hunting): get_auth_request con 0x7fd384138230 auth_method 0 2024-09-11T01:29:19.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd382ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:19.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd382ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:19.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd382ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:19.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd382ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:19.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd382ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:19.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd382ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:19.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd383fff640 1 --2- 172.21.15.12:0/2423123520 >> v2:172.21.15.71:3300/0 conn(0x7fd38413d380 0x7fd38413f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd383fff640 10 monclient(hunting): get_auth_request con 0x7fd38413d380 auth_method 0 2024-09-11T01:29:19.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd383fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:19.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd383fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:19.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd383fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:19.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd382ffd640 10 monclient(hunting): handle_auth_done global_id 4227 payload 274 2024-09-11T01:29:19.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd382ffd640 1 -- 172.21.15.12:0/2423123520 >> v2:172.21.15.71:3300/0 conn(0x7fd38413d380 msgr2=0x7fd38413f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:19.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd382ffd640 1 --2- 172.21.15.12:0/2423123520 >> v2:172.21.15.71:3300/0 conn(0x7fd38413d380 0x7fd38413f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd382ffd640 1 -- 172.21.15.12:0/2423123520 >> v1:172.21.15.71:6789/0 conn(0x7fd384137e50 legacy=0x7fd38409d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:19.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd382ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:29:19.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd382ffd640 1 monclient: found mon.noname-a 2024-09-11T01:29:19.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd382ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:19.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd382ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:19.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd382ffd640 1 -- 172.21.15.12:0/2423123520 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd38409c780 con 0x7fd384138230 2024-09-11T01:29:19.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.511+0000 7fd383fff640 1 --2- 172.21.15.12:0/2423123520 >> v2:172.21.15.71:3300/0 conn(0x7fd38413d380 0x7fd38413f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:29:19.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.514+0000 7fd382ffd640 1 -- 172.21.15.12:0/2423123520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd384138230 msgr2=0x7fd384138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:19.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.514+0000 7fd382ffd640 1 --2- 172.21.15.12:0/2423123520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd384138230 0x7fd384138640 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd38409cf30 tx=0x7fd37402a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=695d87e7d19673bd server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:19.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.514+0000 7fd3827fc640 1 -- 172.21.15.12:0/2423123520 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd374037070 con 0x7fd384138230 2024-09-11T01:29:19.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.514+0000 7fd3827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:19.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.514+0000 7fd3827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:19.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.514+0000 7fd3827fc640 10 monclient: dump: 2024-09-11T01:29:19.521 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:19.521 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.521 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.521 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.521 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:19.521 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.522 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:19.522 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:19.522 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:19.522 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.514+0000 7fd3827fc640 10 monclient: _finish_auth 0 2024-09-11T01:29:19.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.514+0000 7fd3827fc640 10 monclient: _check_auth_tickets 2024-09-11T01:29:19.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.514+0000 7fd3827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:19.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.514+0000 7fd3827fc640 1 -- 172.21.15.12:0/2423123520 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd374032440 con 0x7fd384138230 2024-09-11T01:29:19.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.514+0000 7fd389cbd640 5 monclient: authenticate success, global_id 4227 2024-09-11T01:29:19.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.514+0000 7fd389cbd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:19.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.514+0000 7fd3827fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:19.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.514+0000 7fd389cbd640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:19.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.514+0000 7fd389cbd640 4 set_mon_vals no callback set 2024-09-11T01:29:19.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.514+0000 7fd3827fc640 1 -- 172.21.15.12:0/2423123520 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3740327f0 con 0x7fd384138230 2024-09-11T01:29:19.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.514+0000 7fd3827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:19.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.514+0000 7fd3827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:19.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.514+0000 7fd3827fc640 10 monclient: dump: 2024-09-11T01:29:19.523 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:19.523 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.523 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.523 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.523 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:19.523 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.523 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:19.523 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:19.524 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:19.524 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.515+0000 7fd389cbd640 10 monclient: shutdown 2024-09-11T01:29:19.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.515+0000 7fd389cbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:19.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.515+0000 7fd389cbd640 1 -- 172.21.15.12:0/2423123520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd384138230 msgr2=0x7fd384138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.515+0000 7fd389cbd640 1 --2- 172.21.15.12:0/2423123520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd384138230 0x7fd384138640 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd38409cf30 tx=0x7fd37402a080 comp rx=0 tx=0).stop 2024-09-11T01:29:19.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.515+0000 7fd389cbd640 1 -- 172.21.15.12:0/2423123520 shutdown_connections 2024-09-11T01:29:19.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.515+0000 7fd389cbd640 1 --2- 172.21.15.12:0/2423123520 >> v2:172.21.15.71:3300/0 conn(0x7fd38413d380 0x7fd38413f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.515+0000 7fd389cbd640 1 --2- 172.21.15.12:0/2423123520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd384138230 0x7fd384138640 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.515+0000 7fd389cbd640 1 -- 172.21.15.12:0/2423123520 >> 172.21.15.12:0/2423123520 conn(0x7fd3840a7020 msgr2=0x7fd3840a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:19.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.515+0000 7fd389cbd640 1 -- 172.21.15.12:0/2423123520 shutdown_connections 2024-09-11T01:29:19.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.515+0000 7fd389cbd640 1 -- 172.21.15.12:0/2423123520 wait complete. 2024-09-11T01:29:19.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.515+0000 7fd389cbd640 10 monclient: build_initial_monmap 2024-09-11T01:29:19.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.515+0000 7fd389cbd640 10 monclient: monmap: 2024-09-11T01:29:19.525 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:19.525 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:19.525 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:19.525 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:19.525 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:19.525 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.525 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:19.525 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:19.525 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:19.525 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.516+0000 7fd389cbd640 1 Processor -- start 2024-09-11T01:29:19.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.516+0000 7fd389cbd640 1 -- start start 2024-09-11T01:29:19.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.516+0000 7fd389cbd640 10 monclient: init 2024-09-11T01:29:19.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.516+0000 7fd389cbd640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:19.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.516+0000 7fd389cbd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:29:19.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.516+0000 7fd389cbd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd384137e50 0x7fd3841ccef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.516+0000 7fd389cbd640 10 monclient(hunting): picked mon.noname-c con 0x7fd384137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:19.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.516+0000 7fd389cbd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd384138230 0x7fd384136f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.516+0000 7fd389cbd640 10 monclient(hunting): picked mon.noname-b con 0x7fd384138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:19.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.516+0000 7fd389cbd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd38413d380 0x7fd3840a1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.516+0000 7fd389cbd640 10 monclient(hunting): picked mon.noname-a con 0x7fd38413d380 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:19.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.516+0000 7fd389cbd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.516+0000 7fd389cbd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd38406f890 con 0x7fd38413d380 2024-09-11T01:29:19.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.516+0000 7fd389cbd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.516+0000 7fd389cbd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd38406f710 con 0x7fd384138230 2024-09-11T01:29:19.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.516+0000 7fd389cbd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.516+0000 7fd389cbd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd38413fcb0 con 0x7fd384137e50 2024-09-11T01:29:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.516+0000 7fd389cbd640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.516+0000 7fd389cbd640 10 monclient(hunting): authenticate will time out at 765.958985s 2024-09-11T01:29:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd3837fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd384137e50 0x7fd3841ccef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd383fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd38413d380 0x7fd3840a1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd3837fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd384137e50 0x7fd3841ccef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51486/0 (socket says 172.21.15.12:51486) 2024-09-11T01:29:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd3837fe640 1 -- 172.21.15.12:0/3824984220 learned_addr learned my addr 172.21.15.12:0/3824984220 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd383fff640 10 monclient(hunting): get_auth_request con 0x7fd38413d380 auth_method 0 2024-09-11T01:29:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd383fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd383fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd383fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd3837fe640 10 monclient(hunting): get_auth_request con 0x7fd384137e50 auth_method 0 2024-09-11T01:29:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd3837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd3837fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd3837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd383fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd383fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd383fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd3837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd3837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd3837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd382ffd640 1 --2- 172.21.15.12:0/3824984220 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd384138230 0x7fd384136f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd383fff640 10 monclient(hunting): handle_auth_done global_id 4230 payload 824 2024-09-11T01:29:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd383fff640 1 -- 172.21.15.12:0/3824984220 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd384137e50 msgr2=0x7fd3841ccef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd383fff640 1 --2- 172.21.15.12:0/3824984220 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd384137e50 0x7fd3841ccef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd383fff640 1 -- 172.21.15.12:0/3824984220 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd384138230 msgr2=0x7fd384136f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd383fff640 1 --2- 172.21.15.12:0/3824984220 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd384138230 0x7fd384136f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd383fff640 10 monclient: _finish_hunting 0 2024-09-11T01:29:19.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd383fff640 1 monclient: found mon.noname-a 2024-09-11T01:29:19.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd383fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:19.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd383fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:19.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd383fff640 1 -- 172.21.15.12:0/3824984220 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3841cde10 con 0x7fd38413d380 2024-09-11T01:29:19.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd382ffd640 10 monclient: get_auth_request con 0x7fd384138230 auth_method 0 2024-09-11T01:29:19.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd3837fe640 1 --2- 172.21.15.12:0/3824984220 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd384137e50 0x7fd3841ccef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:19.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.517+0000 7fd382ffd640 1 --2- 172.21.15.12:0/3824984220 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd384138230 0x7fd384136f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:29:19.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.519+0000 7fd383fff640 1 -- 172.21.15.12:0/3824984220 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd38413d380 msgr2=0x7fd3840a1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:19.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.519+0000 7fd383fff640 1 --2- 172.21.15.12:0/3824984220 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd38413d380 0x7fd3840a1ba0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd37c007ef0 tx=0x7fd37c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:19.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.519+0000 7fd37bfff640 1 -- 172.21.15.12:0/3824984220 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37c01a070 con 0x7fd38413d380 2024-09-11T01:29:19.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.519+0000 7fd37bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:19.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.519+0000 7fd37bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:19.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.519+0000 7fd37bfff640 10 monclient: dump: 2024-09-11T01:29:19.534 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:19.534 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.534 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:19.535 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.519+0000 7fd37bfff640 10 monclient: _finish_auth 0 2024-09-11T01:29:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.519+0000 7fd37bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:29:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.519+0000 7fd37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.519+0000 7fd37bfff640 1 -- 172.21.15.12:0/3824984220 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd37c003b90 con 0x7fd38413d380 2024-09-11T01:29:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.519+0000 7fd389cbd640 5 monclient: authenticate success, global_id 4230 2024-09-11T01:29:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.519+0000 7fd389cbd640 10 monclient: _renew_subs 2024-09-11T01:29:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.519+0000 7fd389cbd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.519+0000 7fd389cbd640 1 -- 172.21.15.12:0/3824984220 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3841215d0 con 0x7fd38413d380 2024-09-11T01:29:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.519+0000 7fd37bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.519+0000 7fd37bfff640 1 -- 172.21.15.12:0/3824984220 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37c004030 con 0x7fd38413d380 2024-09-11T01:29:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.519+0000 7fd37bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.519+0000 7fd37bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.519+0000 7fd37bfff640 10 monclient: dump: 2024-09-11T01:29:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:19.537 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.519+0000 7fd389cbd640 10 monclient: _renew_subs 2024-09-11T01:29:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.519+0000 7fd381ffb640 4 set_mon_vals no callback set 2024-09-11T01:29:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.519+0000 7fd389cbd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.519+0000 7fd389cbd640 1 -- 172.21.15.12:0/3824984220 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3840a9190 con 0x7fd38413d380 2024-09-11T01:29:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.521+0000 7fd389cbd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.521+0000 7fd389cbd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.521+0000 7fd389cbd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.521+0000 7fd37bfff640 1 -- 172.21.15.12:0/3824984220 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fd37c004250 con 0x7fd38413d380 2024-09-11T01:29:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.521+0000 7fd389cbd640 1 -- 172.21.15.12:0/3824984220 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd384138640 con 0x7fd38413d380 2024-09-11T01:29:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.521+0000 7fd37bfff640 1 --2- 172.21.15.12:0/3824984220 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd36403e210 0x7fd3640406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.521+0000 7fd37bfff640 1 -- 172.21.15.12:0/3824984220 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4402+0+0 (secure 0 0 0) 0x7fd37c053260 con 0x7fd38413d380 2024-09-11T01:29:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.521+0000 7fd3837fe640 1 --2- 172.21.15.12:0/3824984220 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd36403e210 0x7fd3640406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.521+0000 7fd3837fe640 10 monclient: get_auth_request con 0x7fd36403e210 auth_method 0 2024-09-11T01:29:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.524+0000 7fd3837fe640 1 --2- 172.21.15.12:0/3824984220 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd36403e210 0x7fd3640406d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd36c0043e0 tx=0x7fd36c007ef0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.526+0000 7fd37bfff640 1 -- 172.21.15.12:0/3824984220 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd37c010070 con 0x7fd38413d380 2024-09-11T01:29:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.526+0000 7fd37bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.526+0000 7fd37bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:19.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.713+0000 7fd389cbd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-11T01:29:19.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.713+0000 7fd389cbd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T01:29:19.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.713+0000 7fd389cbd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:19.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.713+0000 7fd389cbd640 1 -- 172.21.15.12:0/3824984220 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd3840a35c0 con 0x7fd38413d380 2024-09-11T01:29:19.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.714+0000 7fd37bfff640 1 -- 172.21.15.12:0/3824984220 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12373 (secure 0 0 0) 0x7fd37c012560 con 0x7fd38413d380 2024-09-11T01:29:19.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.714+0000 7fd37bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T01:29:19.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.714+0000 7fd37bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:19.716 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:29:19.717 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":16,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","created":"2024-09-11T01:29:02.540751+0000","modified":"2024-09-11T01:29:17.539307+0000","last_up_change":"2024-09-11T01:29:13.320573+0000","last_in_change":"2024-09-11T01:29:05.075851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-11T01:29:14.157646+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-11T01:29:14.872725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"616ba1d0-adf6-4fe1-85e3-266e53e21867","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":4266003406},{"type":"v1","addr":"172.21.15.12:6817","nonce":4266003406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":4266003406},{"type":"v1","addr":"172.21.15.12:6819","nonce":4266003406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":4266003406},{"type":"v1","addr":"172.21.15.12:6823","nonce":4266003406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":4266003406},{"type":"v1","addr":"172.21.15.12:6821","nonce":4266003406}]},"public_addr":"172.21.15.12:6817/4266003406","cluster_addr":"172.21.15.12:6819/4266003406","heartbeat_back_addr":"172.21.15.12:6823/4266003406","heartbeat_front_addr":"172.21.15.12:6821/4266003406","state":["exists","up"]},{"osd":1,"uuid":"9b570874-7a3d-4867-8c44-29084a8bf649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3563357280},{"type":"v1","addr":"172.21.15.12:6809","nonce":3563357280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3563357280},{"type":"v1","addr":"172.21.15.12:6811","nonce":3563357280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3563357280},{"type":"v1","addr":"172.21.15.12:6815","nonce":3563357280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3563357280},{"type":"v1","addr":"172.21.15.12:6813","nonce":3563357280}]},"public_addr":"172.21.15.12:6809/3563357280","cluster_addr":"172.21.15.12:6811/3563357280","heartbeat_back_addr":"172.21.15.12:6815/3563357280","heartbeat_front_addr":"172.21.15.12:6813/3563357280","state":["exists","up"]},{"osd":2,"uuid":"b12ee288-ce57-41de-8b8e-6fad509311b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2893036867},{"type":"v1","addr":"172.21.15.12:6801","nonce":2893036867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2893036867},{"type":"v1","addr":"172.21.15.12:6803","nonce":2893036867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2893036867},{"type":"v1","addr":"172.21.15.12:6807","nonce":2893036867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2893036867},{"type":"v1","addr":"172.21.15.12:6805","nonce":2893036867}]},"public_addr":"172.21.15.12:6801/2893036867","cluster_addr":"172.21.15.12:6803/2893036867","heartbeat_back_addr":"172.21.15.12:6807/2893036867","heartbeat_front_addr":"172.21.15.12:6805/2893036867","state":["exists","up"]},{"osd":3,"uuid":"a8e81aa4-0dee-487d-981d-1f593e363eee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":3091488141},{"type":"v1","addr":"172.21.15.71:6819","nonce":3091488141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":3091488141},{"type":"v1","addr":"172.21.15.71:6821","nonce":3091488141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":3091488141},{"type":"v1","addr":"172.21.15.71:6825","nonce":3091488141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":3091488141},{"type":"v1","addr":"172.21.15.71:6823","nonce":3091488141}]},"public_addr":"172.21.15.71:6819/3091488141","cluster_addr":"172.21.15.71:6821/3091488141","heartbeat_back_addr":"172.21.15.71:6825/3091488141","heartbeat_front_addr":"172.21.15.71:6823/3091488141","state":["exists","up"]},{"osd":4,"uuid":"9db77181-fc18-408f-a35c-87bf4b4ba2d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":1790415427},{"type":"v1","addr":"172.21.15.71:6801","nonce":1790415427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":1790415427},{"type":"v1","addr":"172.21.15.71:6803","nonce":1790415427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":1790415427},{"type":"v1","addr":"172.21.15.71:6807","nonce":1790415427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":1790415427},{"type":"v1","addr":"172.21.15.71:6805","nonce":1790415427}]},"public_addr":"172.21.15.71:6801/1790415427","cluster_addr":"172.21.15.71:6803/1790415427","heartbeat_back_addr":"172.21.15.71:6807/1790415427","heartbeat_front_addr":"172.21.15.71:6805/1790415427","state":["exists","up"]},{"osd":5,"uuid":"fadf2191-f31a-4880-a529-5b2d1023d236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":2032569317},{"type":"v1","addr":"172.21.15.71:6809","nonce":2032569317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":2032569317},{"type":"v1","addr":"172.21.15.71:6811","nonce":2032569317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":2032569317},{"type":"v1","addr":"172.21.15.71:6815","nonce":2032569317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":2032569317},{"type":"v1","addr":"172.21.15.71:6813","nonce":2032569317}]},"public_addr":"172.21.15.71:6809/2032569317","cluster_addr":"172.21.15.71:6811/2032569317","heartbeat_back_addr":"172.21.15.71:6815/2032569317","heartbeat_front_addr":"172.21.15.71:6813/2032569317","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:11.548818+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:11.310103+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:10.578802+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:11.748432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:11.098916+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:11.150538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T01:29:19.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.716+0000 7fd389cbd640 1 -- 172.21.15.12:0/3824984220 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd36403e210 msgr2=0x7fd3640406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.716+0000 7fd389cbd640 1 --2- 172.21.15.12:0/3824984220 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd36403e210 0x7fd3640406d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd36c0043e0 tx=0x7fd36c007ef0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.716+0000 7fd389cbd640 10 monclient: shutdown 2024-09-11T01:29:19.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.716+0000 7fd389cbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:19.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.716+0000 7fd389cbd640 1 -- 172.21.15.12:0/3824984220 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd38413d380 msgr2=0x7fd3840a1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.716+0000 7fd389cbd640 1 --2- 172.21.15.12:0/3824984220 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd38413d380 0x7fd3840a1ba0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd37c007ef0 tx=0x7fd37c00d4a0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.717+0000 7fd389cbd640 1 -- 172.21.15.12:0/3824984220 shutdown_connections 2024-09-11T01:29:19.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.717+0000 7fd389cbd640 1 --2- 172.21.15.12:0/3824984220 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd36403e210 0x7fd3640406d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.717+0000 7fd389cbd640 1 --2- 172.21.15.12:0/3824984220 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd38413d380 0x7fd3840a1ba0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.717+0000 7fd389cbd640 1 --2- 172.21.15.12:0/3824984220 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd384138230 0x7fd384136f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.717+0000 7fd389cbd640 1 --2- 172.21.15.12:0/3824984220 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd384137e50 0x7fd3841ccef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.717+0000 7fd389cbd640 1 -- 172.21.15.12:0/3824984220 >> 172.21.15.12:0/3824984220 conn(0x7fd3840a7020 msgr2=0x7fd38413c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:19.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.717+0000 7fd389cbd640 1 -- 172.21.15.12:0/3824984220 shutdown_connections 2024-09-11T01:29:19.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.717+0000 7fd389cbd640 1 -- 172.21.15.12:0/3824984220 wait complete. 2024-09-11T01:29:19.730 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-11T01:29:19.730 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-11T01:29:19.730 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-11T01:29:19.731 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-11T01:29:19.731 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-11T01:29:19.732 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-11T01:29:19.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.846+0000 7f93e23b4640 10 monclient: get_monmap_and_config 2024-09-11T01:29:19.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.846+0000 7f93e23b4640 10 monclient: build_initial_monmap 2024-09-11T01:29:19.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.846+0000 7f93e23b4640 10 monclient: monmap: 2024-09-11T01:29:19.848 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:19.849 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.849 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:19.847379+0000 2024-09-11T01:29:19.849 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:19.847379+0000 2024-09-11T01:29:19.849 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:19.849 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.849 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:19.849 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:19.849 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:19.849 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:19.849 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:19.849 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.847+0000 7f93e23b4640 1 Processor -- start 2024-09-11T01:29:19.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.847+0000 7f93e23b4640 1 -- start start 2024-09-11T01:29:19.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.847+0000 7f93e23b4640 10 monclient: init 2024-09-11T01:29:19.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93e23b4640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:19.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93e23b4640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-11T01:29:19.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93e23b4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f93dc137d50 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:19.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93e23b4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93dc138130 0x7f93dc138550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93e23b4640 10 monclient(hunting): picked mon.noname-b con 0x7f93dc138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:19.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93db7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93dc138130 0x7f93dc138550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93db7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93dc138130 0x7f93dc138550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51494/0 (socket says 172.21.15.12:51494) 2024-09-11T01:29:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93e23b4640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f93dc13d280 0x7f93dc13f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93db7fe640 1 -- 172.21.15.12:0/2819759998 learned_addr learned my addr 172.21.15.12:0/2819759998 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93e23b4640 10 monclient(hunting): picked mon.noname-c con 0x7f93dc13d280 addr v2:172.21.15.71:3300/0 2024-09-11T01:29:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93e23b4640 1 -- 172.21.15.12:0/2819759998 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93dc09d820 con 0x7f93dc137d50 2024-09-11T01:29:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93e23b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93e23b4640 1 -- 172.21.15.12:0/2819759998 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f93dc09c650 con 0x7f93dc13d280 2024-09-11T01:29:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93e23b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93e23b4640 1 -- 172.21.15.12:0/2819759998 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f93dc09ba20 con 0x7f93dc138130 2024-09-11T01:29:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93e23b4640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93e23b4640 10 monclient(hunting): authenticate will time out at 766.290625s 2024-09-11T01:29:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93db7fe640 10 monclient(hunting): get_auth_request con 0x7f93dc138130 auth_method 0 2024-09-11T01:29:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:19.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:19.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:19.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:19.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:19.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:19.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93db7fe640 10 monclient(hunting): handle_auth_done global_id 4241 payload 274 2024-09-11T01:29:19.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93db7fe640 1 -- 172.21.15.12:0/2819759998 >> v2:172.21.15.71:3300/0 conn(0x7f93dc13d280 msgr2=0x7f93dc13f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:19.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93db7fe640 1 --2- 172.21.15.12:0/2819759998 >> v2:172.21.15.71:3300/0 conn(0x7f93dc13d280 0x7f93dc13f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93db7fe640 1 -- 172.21.15.12:0/2819759998 >> v1:172.21.15.71:6789/0 conn(0x7f93dc137d50 legacy=0x7f93dc09d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:19.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93db7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:29:19.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93db7fe640 1 monclient: found mon.noname-b 2024-09-11T01:29:19.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:19.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93db7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:29:19.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.848+0000 7f93db7fe640 1 -- 172.21.15.12:0/2819759998 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93dc09c370 con 0x7f93dc138130 2024-09-11T01:29:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.850+0000 7f1df12bd640 10 monclient: get_monmap_and_config 2024-09-11T01:29:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.850+0000 7f1df12bd640 10 monclient: build_initial_monmap 2024-09-11T01:29:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.850+0000 7f1df12bd640 10 monclient: monmap: 2024-09-11T01:29:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:19.857 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.857 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:19.850974+0000 2024-09-11T01:29:19.857 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:19.850974+0000 2024-09-11T01:29:19.857 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:19.857 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.857 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:19.857 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:19.857 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:19.857 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:19.857 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:19.857 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.850+0000 7f1df12bd640 1 Processor -- start 2024-09-11T01:29:19.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.850+0000 7f1df12bd640 1 -- start start 2024-09-11T01:29:19.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.850+0000 7f1df12bd640 10 monclient: init 2024-09-11T01:29:19.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.851+0000 7f1df12bd640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:19.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.851+0000 7f1df12bd640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-11T01:29:19.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.851+0000 7f1df12bd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1dec137d50 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:19.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.851+0000 7f1df12bd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1dec138130 0x7f1dec138580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.851+0000 7f1df12bd640 10 monclient(hunting): picked mon.noname-a con 0x7f1dec138130 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:19.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.851+0000 7f1df12bd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1dec13d280 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:19.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.851+0000 7f1dea7fc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1dec138130 0x7f1dec138580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.851+0000 7f1dea7fc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1dec138130 0x7f1dec138580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54830/0 (socket says 172.21.15.12:54830) 2024-09-11T01:29:19.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.851+0000 7f1dea7fc640 1 -- 172.21.15.12:0/3025041706 learned_addr learned my addr 172.21.15.12:0/3025041706 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:19.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.851+0000 7f1df12bd640 1 -- 172.21.15.12:0/3025041706 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1dec09d820 con 0x7f1dec13d280 2024-09-11T01:29:19.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.851+0000 7f1df12bd640 1 -- 172.21.15.12:0/3025041706 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1dec09ba20 con 0x7f1dec137d50 2024-09-11T01:29:19.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.851+0000 7f1df12bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.851+0000 7f1df12bd640 1 -- 172.21.15.12:0/3025041706 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f1dec09c650 con 0x7f1dec138130 2024-09-11T01:29:19.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.851+0000 7f1df12bd640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:19.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.851+0000 7f1df12bd640 10 monclient(hunting): authenticate will time out at 766.293928s 2024-09-11T01:29:19.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.851+0000 7f93db7fe640 1 -- 172.21.15.12:0/2819759998 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93dc138130 msgr2=0x7f93dc138550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:19.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.851+0000 7f93db7fe640 1 --2- 172.21.15.12:0/2819759998 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93dc138130 0x7f93dc138550 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f93d0009870 tx=0x7f93d002a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0bacd7aad28ddf9 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:19.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.851+0000 7f1dea7fc640 10 monclient(hunting): get_auth_request con 0x7f1dec138130 auth_method 0 2024-09-11T01:29:19.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.851+0000 7f1dea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:19.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.851+0000 7f1dea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:19.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.851+0000 7f1dea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:19.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f1dea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:19.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f1dea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:19.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f1dea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:19.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f1dea7fc640 10 monclient(hunting): handle_auth_done global_id 4233 payload 274 2024-09-11T01:29:19.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f1dea7fc640 1 -- 172.21.15.12:0/3025041706 >> v1:172.21.15.71:6789/0 conn(0x7f1dec137d50 legacy=0x7f1dec09d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:19.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f1dea7fc640 1 -- 172.21.15.12:0/3025041706 >> v1:172.21.15.12:6789/0 conn(0x7f1dec13d280 legacy=0x7f1dec13f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:19.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f93daffd640 1 -- 172.21.15.12:0/2819759998 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d0037070 con 0x7f93dc138130 2024-09-11T01:29:19.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f93daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:19.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f1dea7fc640 10 monclient: _finish_hunting 0 2024-09-11T01:29:19.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f93daffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:19.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f93daffd640 10 monclient: dump: 2024-09-11T01:29:19.863 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:19.863 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.863 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.863 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.863 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:19.863 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.863 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:19.863 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:19.863 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:19.863 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f93daffd640 10 monclient: _finish_auth 0 2024-09-11T01:29:19.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f93daffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:19.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f93daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:19.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f93e23b4640 5 monclient: authenticate success, global_id 4241 2024-09-11T01:29:19.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f93e23b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:19.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f1dea7fc640 1 monclient: found mon.noname-a 2024-09-11T01:29:19.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f1dea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:19.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f93daffd640 1 -- 172.21.15.12:0/2819759998 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93d0032640 con 0x7f93dc138130 2024-09-11T01:29:19.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f93daffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:19.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f93daffd640 1 -- 172.21.15.12:0/2819759998 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d0032a10 con 0x7f93dc138130 2024-09-11T01:29:19.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f93e23b4640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:19.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f93e23b4640 4 set_mon_vals no callback set 2024-09-11T01:29:19.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f93daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:19.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f93daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:19.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f93daffd640 10 monclient: dump: 2024-09-11T01:29:19.866 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:19.866 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.866 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.866 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.867 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:19.867 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.867 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:19.867 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:19.867 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:19.867 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f1dea7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:19.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.852+0000 7f1dea7fc640 1 -- 172.21.15.12:0/3025041706 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dec09c370 con 0x7f1dec138130 2024-09-11T01:29:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.853+0000 7f93e23b4640 10 monclient: shutdown 2024-09-11T01:29:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.853+0000 7f93e23b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.853+0000 7f93e23b4640 1 -- 172.21.15.12:0/2819759998 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93dc138130 msgr2=0x7f93dc138550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.853+0000 7f93e23b4640 1 --2- 172.21.15.12:0/2819759998 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93dc138130 0x7f93dc138550 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f93d0009870 tx=0x7f93d002a130 comp rx=0 tx=0).stop 2024-09-11T01:29:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.853+0000 7f93e23b4640 1 -- 172.21.15.12:0/2819759998 shutdown_connections 2024-09-11T01:29:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.853+0000 7f93e23b4640 1 --2- 172.21.15.12:0/2819759998 >> v2:172.21.15.71:3300/0 conn(0x7f93dc13d280 0x7f93dc13f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.853+0000 7f93e23b4640 1 --2- 172.21.15.12:0/2819759998 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93dc138130 0x7f93dc138550 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.853+0000 7f93e23b4640 1 -- 172.21.15.12:0/2819759998 >> 172.21.15.12:0/2819759998 conn(0x7f93dc0a7020 msgr2=0x7f93dc0a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:19.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.853+0000 7f93e23b4640 1 -- 172.21.15.12:0/2819759998 shutdown_connections 2024-09-11T01:29:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.853+0000 7f93e23b4640 1 -- 172.21.15.12:0/2819759998 wait complete. 2024-09-11T01:29:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.854+0000 7f93e23b4640 10 monclient: build_initial_monmap 2024-09-11T01:29:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.854+0000 7f93e23b4640 10 monclient: monmap: 2024-09-11T01:29:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:19.870 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:19.871 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:19.871 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.854+0000 7f93e23b4640 1 Processor -- start 2024-09-11T01:29:19.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.854+0000 7f93e23b4640 1 -- start start 2024-09-11T01:29:19.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.854+0000 7f93e23b4640 10 monclient: init 2024-09-11T01:29:19.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f93e23b4640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:19.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f93e23b4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:29:19.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f1dea7fc640 1 -- 172.21.15.12:0/3025041706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1dec138130 msgr2=0x7f1dec138580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:19.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f1dea7fc640 1 --2- 172.21.15.12:0/3025041706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1dec138130 0x7f1dec138580 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1de0009870 tx=0x7f1de002a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a71d7355e08acfd2 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:19.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f93e23b4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93dc137d50 0x7f93dc1ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f93e23b4640 10 monclient(hunting): picked mon.noname-c con 0x7f93dc137d50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:19.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f93e23b4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93dc138130 0x7f93dc136df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f93e23b4640 10 monclient(hunting): picked mon.noname-a con 0x7f93dc138130 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:19.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f1de9ffb640 1 -- 172.21.15.12:0/3025041706 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de0037070 con 0x7f1dec138130 2024-09-11T01:29:19.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f1de9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:19.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f1de9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:19.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f1de9ffb640 10 monclient: dump: 2024-09-11T01:29:19.873 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:19.873 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.873 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.874 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.874 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:19.874 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.874 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:19.874 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:19.874 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:19.874 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f1de9ffb640 10 monclient: _finish_auth 0 2024-09-11T01:29:19.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f1de9ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:29:19.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f1de9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:19.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f1de9ffb640 1 -- 172.21.15.12:0/3025041706 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1de0032640 con 0x7f1dec138130 2024-09-11T01:29:19.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f1de9ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:19.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f1de9ffb640 1 -- 172.21.15.12:0/3025041706 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de0032a10 con 0x7f1dec138130 2024-09-11T01:29:19.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f1de9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:19.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f1de9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:19.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f1de9ffb640 10 monclient: dump: 2024-09-11T01:29:19.875 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:19.875 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.875 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.875 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.875 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:19.875 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.875 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:19.876 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:19.876 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:19.876 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f1df12bd640 5 monclient: authenticate success, global_id 4233 2024-09-11T01:29:19.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f1df12bd640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:19.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f1df12bd640 4 set_mon_vals no callback set 2024-09-11T01:29:19.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f93e23b4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f93dc13d280 0x7f93dc13a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f93db7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93dc138130 0x7f93dc136df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f93db7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93dc138130 0x7f93dc136df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54836/0 (socket says 172.21.15.12:54836) 2024-09-11T01:29:19.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.855+0000 7f93dbfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93dc137d50 0x7f93dc1ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93db7fe640 1 -- 172.21.15.12:0/1003683211 learned_addr learned my addr 172.21.15.12:0/1003683211 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:19.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93e23b4640 10 monclient(hunting): picked mon.noname-b con 0x7f93dc13d280 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:19.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93e23b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93e23b4640 1 -- 172.21.15.12:0/1003683211 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f93dc06f850 con 0x7f93dc138130 2024-09-11T01:29:19.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93e23b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93e23b4640 1 -- 172.21.15.12:0/1003683211 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f93dc06f6d0 con 0x7f93dc13d280 2024-09-11T01:29:19.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93e23b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93e23b4640 1 -- 172.21.15.12:0/1003683211 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f93dc13fbb0 con 0x7f93dc137d50 2024-09-11T01:29:19.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93e23b4640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:19.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93e23b4640 10 monclient(hunting): authenticate will time out at 766.298056s 2024-09-11T01:29:19.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93db7fe640 10 monclient(hunting): get_auth_request con 0x7f93dc138130 auth_method 0 2024-09-11T01:29:19.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:19.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:19.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:19.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93dbfff640 10 monclient(hunting): get_auth_request con 0x7f93dc137d50 auth_method 0 2024-09-11T01:29:19.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:19.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:19.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:19.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:19.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:19.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:19.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:19.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:19.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:19.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93e0951640 1 --2- 172.21.15.12:0/1003683211 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f93dc13d280 0x7f93dc13a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93e0951640 10 monclient(hunting): get_auth_request con 0x7f93dc13d280 auth_method 0 2024-09-11T01:29:19.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93e0951640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:19.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93e0951640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:19.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93e0951640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:19.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93e0951640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:19.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93e0951640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:19.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93e0951640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:19.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93dbfff640 10 monclient(hunting): handle_auth_done global_id 4244 payload 824 2024-09-11T01:29:19.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93dbfff640 1 -- 172.21.15.12:0/1003683211 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f93dc13d280 msgr2=0x7f93dc13a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f1df12bd640 10 monclient: shutdown 2024-09-11T01:29:19.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93dbfff640 1 --2- 172.21.15.12:0/1003683211 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f93dc13d280 0x7f93dc13a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93dbfff640 1 -- 172.21.15.12:0/1003683211 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93dc138130 msgr2=0x7f93dc136df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93dbfff640 1 --2- 172.21.15.12:0/1003683211 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93dc138130 0x7f93dc136df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93dbfff640 10 monclient: _finish_hunting 0 2024-09-11T01:29:19.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93dbfff640 1 monclient: found mon.noname-c 2024-09-11T01:29:19.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:19.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93dbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:29:19.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93dbfff640 1 -- 172.21.15.12:0/1003683211 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93dc121780 con 0x7f93dc137d50 2024-09-11T01:29:19.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f1df12bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:19.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f1df12bd640 1 -- 172.21.15.12:0/3025041706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1dec138130 msgr2=0x7f1dec138580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f1df12bd640 1 --2- 172.21.15.12:0/3025041706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1dec138130 0x7f1dec138580 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1de0009870 tx=0x7f1de002a130 comp rx=0 tx=0).stop 2024-09-11T01:29:19.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.856+0000 7f93db7fe640 1 --2- 172.21.15.12:0/1003683211 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93dc138130 0x7f93dc136df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:19.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.857+0000 7f1df12bd640 1 -- 172.21.15.12:0/3025041706 shutdown_connections 2024-09-11T01:29:19.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.857+0000 7f1df12bd640 1 --2- 172.21.15.12:0/3025041706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1dec138130 0x7f1dec138580 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.857+0000 7f1df12bd640 1 -- 172.21.15.12:0/3025041706 >> 172.21.15.12:0/3025041706 conn(0x7f1dec0a7020 msgr2=0x7f1dec0a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:19.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.857+0000 7f1df12bd640 1 -- 172.21.15.12:0/3025041706 shutdown_connections 2024-09-11T01:29:19.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.857+0000 7f1df12bd640 1 -- 172.21.15.12:0/3025041706 wait complete. 2024-09-11T01:29:19.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.857+0000 7f1df12bd640 10 monclient: build_initial_monmap 2024-09-11T01:29:19.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.857+0000 7f1df12bd640 10 monclient: monmap: 2024-09-11T01:29:19.886 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:19.886 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:19.887 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:19.887 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:19.887 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:19.887 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.887 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:19.887 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:19.887 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:19.887 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.858+0000 7f1df12bd640 1 Processor -- start 2024-09-11T01:29:19.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.858+0000 7f1df12bd640 1 -- start start 2024-09-11T01:29:19.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.858+0000 7f1df12bd640 10 monclient: init 2024-09-11T01:29:19.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.858+0000 7f1df12bd640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:19.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.858+0000 7f1df12bd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:29:19.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.858+0000 7f1df12bd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1dec137d50 0x7f1dec136a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1df12bd640 10 monclient(hunting): picked mon.noname-c con 0x7f1dec137d50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:19.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1df12bd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dec138130 0x7f1dec136fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1df12bd640 10 monclient(hunting): picked mon.noname-b con 0x7f1dec138130 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:19.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1df12bd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1dec13d280 0x7f1dec139260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f93dbfff640 1 -- 172.21.15.12:0/1003683211 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93dc137d50 msgr2=0x7f93dc1ccd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:19.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f93dbfff640 1 --2- 172.21.15.12:0/1003683211 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93dc137d50 0x7f93dc1ccd90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f93cc00c440 tx=0x7f93cc00c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:19.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1df12bd640 10 monclient(hunting): picked mon.noname-a con 0x7f1dec13d280 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:19.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1df12bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1df12bd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1dec06f850 con 0x7f1dec13d280 2024-09-11T01:29:19.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1df12bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1df12bd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1dec06f6d0 con 0x7f1dec138130 2024-09-11T01:29:19.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1df12bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1df12bd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1dec140d90 con 0x7f1dec137d50 2024-09-11T01:29:19.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1df12bd640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:19.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1df12bd640 10 monclient(hunting): authenticate will time out at 766.301173s 2024-09-11T01:29:19.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1deaffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1dec137d50 0x7f1dec136a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1deaffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1dec137d50 0x7f1dec136a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51504/0 (socket says 172.21.15.12:51504) 2024-09-11T01:29:19.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1deaffd640 1 -- 172.21.15.12:0/3444315598 learned_addr learned my addr 172.21.15.12:0/3444315598 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:19.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1deaffd640 10 monclient(hunting): get_auth_request con 0x7f1dec137d50 auth_method 0 2024-09-11T01:29:19.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1deaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:19.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f93d8ff9640 1 -- 172.21.15.12:0/1003683211 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93cc0119a0 con 0x7f93dc137d50 2024-09-11T01:29:19.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f93d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:19.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f93d8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:19.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f93d8ff9640 10 monclient: dump: 2024-09-11T01:29:19.892 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:19.892 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.892 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.892 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.892 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:19.892 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.892 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:19.892 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:19.892 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:19.893 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f93d8ff9640 10 monclient: _finish_auth 0 2024-09-11T01:29:19.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f93d8ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:29:19.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f93d8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:19.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f93d8ff9640 1 -- 172.21.15.12:0/1003683211 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93cc017070 con 0x7f93dc137d50 2024-09-11T01:29:19.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f93d8ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:19.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f93d8ff9640 1 -- 172.21.15.12:0/1003683211 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93cc011d80 con 0x7f93dc137d50 2024-09-11T01:29:19.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f93d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:19.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f93d8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:19.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f93d8ff9640 10 monclient: dump: 2024-09-11T01:29:19.893 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:19.893 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.894 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.894 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.894 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:19.894 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.894 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:19.894 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:19.894 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:19.894 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f93e23b4640 5 monclient: authenticate success, global_id 4244 2024-09-11T01:29:19.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f93e23b4640 10 monclient: _renew_subs 2024-09-11T01:29:19.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f93e23b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:19.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f93e23b4640 1 -- 172.21.15.12:0/1003683211 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93dc0a94c0 con 0x7f93dc137d50 2024-09-11T01:29:19.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1deaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:19.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1deaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:19.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1dea7fc640 1 --2- 172.21.15.12:0/3444315598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dec138130 0x7f1dec136fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1deaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:19.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1deaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:19.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1deaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:19.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1deb7fe640 1 --2- 172.21.15.12:0/3444315598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1dec13d280 0x7f1dec139260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1deaffd640 10 monclient(hunting): handle_auth_done global_id 4247 payload 824 2024-09-11T01:29:19.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1deaffd640 1 -- 172.21.15.12:0/3444315598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dec138130 msgr2=0x7f1dec136fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1deaffd640 1 --2- 172.21.15.12:0/3444315598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dec138130 0x7f1dec136fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f1deaffd640 1 -- 172.21.15.12:0/3444315598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1dec13d280 msgr2=0x7f1dec139260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.859+0000 7f93da7fc640 4 set_mon_vals no callback set 2024-09-11T01:29:19.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.860+0000 7f1deaffd640 1 --2- 172.21.15.12:0/3444315598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1dec13d280 0x7f1dec139260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.860+0000 7f1deaffd640 10 monclient: _finish_hunting 0 2024-09-11T01:29:19.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.860+0000 7f1deaffd640 1 monclient: found mon.noname-c 2024-09-11T01:29:19.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.860+0000 7f1deaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:19.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.860+0000 7f1deaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:29:19.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.860+0000 7f1deaffd640 1 -- 172.21.15.12:0/3444315598 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dec13c640 con 0x7f1dec137d50 2024-09-11T01:29:19.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.860+0000 7f93e23b4640 10 monclient: _renew_subs 2024-09-11T01:29:19.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.860+0000 7f93e23b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:19.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.860+0000 7f1dea7fc640 10 monclient: get_auth_request con 0x7f1dec138130 auth_method 0 2024-09-11T01:29:19.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.860+0000 7f1dea7fc640 1 --2- 172.21.15.12:0/3444315598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dec138130 0x7f1dec136fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:29:19.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.860+0000 7f1deb7fe640 10 monclient: get_auth_request con 0x7f1dec13d280 auth_method 0 2024-09-11T01:29:19.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.860+0000 7f1deb7fe640 1 --2- 172.21.15.12:0/3444315598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1dec13d280 0x7f1dec139260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:29:19.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.860+0000 7f93e23b4640 1 -- 172.21.15.12:0/1003683211 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f93dc1214a0 con 0x7f93dc137d50 2024-09-11T01:29:19.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.861+0000 7f93e23b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:19.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.861+0000 7f93e23b4640 1 -- 172.21.15.12:0/1003683211 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f93dc1385d0 con 0x7f93dc137d50 2024-09-11T01:29:19.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.861+0000 7f93d8ff9640 1 -- 172.21.15.12:0/1003683211 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f93cc01c460 con 0x7f93dc137d50 2024-09-11T01:29:19.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.861+0000 7f93d8ff9640 1 --2- 172.21.15.12:0/1003683211 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f93c803e290 0x7f93c8040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.861+0000 7f93d8ff9640 1 -- 172.21.15.12:0/1003683211 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4402+0+0 (secure 0 0 0) 0x7f93cc052990 con 0x7f93dc137d50 2024-09-11T01:29:19.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.861+0000 7f93d8ff9640 1 --2- 172.21.15.12:0/1003683211 >> [v2:172.21.15.71:6818/3091488141,v1:172.21.15.71:6819/3091488141] conn(0x7f93c8044180 0x7f93c80465c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.861+0000 7f93d8ff9640 1 -- 172.21.15.12:0/1003683211 --> [v2:172.21.15.71:6818/3091488141,v1:172.21.15.71:6819/3091488141] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f93cc01ee70 con 0x7f93c8044180 2024-09-11T01:29:19.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.861+0000 7f93d8ff9640 1 -- 172.21.15.12:0/1003683211 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f93cc0572a0 con 0x7f93dc137d50 2024-09-11T01:29:19.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.861+0000 7f93d8ff9640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-11T01:29:19.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.861+0000 7f93db7fe640 1 --2- 172.21.15.12:0/1003683211 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f93c803e290 0x7f93c8040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.861+0000 7f93db7fe640 10 monclient: get_auth_request con 0x7f93c803e290 auth_method 0 2024-09-11T01:29:19.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.862+0000 7f93e0951640 1 --2- 172.21.15.12:0/1003683211 >> [v2:172.21.15.71:6818/3091488141,v1:172.21.15.71:6819/3091488141] conn(0x7f93c8044180 0x7f93c80465c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.862+0000 7f93e0951640 10 monclient: get_auth_request con 0x7f93c8044180 auth_method 0 2024-09-11T01:29:19.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.863+0000 7f1deaffd640 1 -- 172.21.15.12:0/3444315598 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1dec137d50 msgr2=0x7f1dec136a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:19.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.863+0000 7f1deaffd640 1 --2- 172.21.15.12:0/3444315598 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1dec137d50 0x7f1dec136a70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc00c430 tx=0x7f1ddc00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:19.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.863+0000 7f1dd37fe640 1 -- 172.21.15.12:0/3444315598 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc0119a0 con 0x7f1dec137d50 2024-09-11T01:29:19.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.863+0000 7f1dd37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:19.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.863+0000 7f1dd37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:19.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.863+0000 7f1dd37fe640 10 monclient: dump: 2024-09-11T01:29:19.903 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:19.903 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.903 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.903 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.903 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:19.904 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.904 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:19.904 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:19.904 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:19.904 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.863+0000 7f1dd37fe640 10 monclient: _finish_auth 0 2024-09-11T01:29:19.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.863+0000 7f1dd37fe640 10 monclient: _check_auth_tickets 2024-09-11T01:29:19.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.863+0000 7f1dd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:19.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.863+0000 7f1df12bd640 5 monclient: authenticate success, global_id 4247 2024-09-11T01:29:19.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.863+0000 7f1df12bd640 10 monclient: _renew_subs 2024-09-11T01:29:19.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.863+0000 7f1df12bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:19.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.863+0000 7f1df12bd640 1 -- 172.21.15.12:0/3444315598 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dec13b400 con 0x7f1dec137d50 2024-09-11T01:29:19.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.863+0000 7f1dd37fe640 1 -- 172.21.15.12:0/3444315598 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ddc017070 con 0x7f1dec137d50 2024-09-11T01:29:19.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.863+0000 7f1dd37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:19.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.863+0000 7f1df12bd640 10 monclient: _renew_subs 2024-09-11T01:29:19.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.863+0000 7f1df12bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:19.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.863+0000 7f1de8ff9640 4 set_mon_vals no callback set 2024-09-11T01:29:19.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.863+0000 7f1df12bd640 1 -- 172.21.15.12:0/3444315598 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dec13c360 con 0x7f1dec137d50 2024-09-11T01:29:19.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.864+0000 7f93db7fe640 1 --2- 172.21.15.12:0/1003683211 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f93c803e290 0x7f93c8040750 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f93d0000c00 tx=0x7f93d002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:19.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.864+0000 7f1dd17fa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:19.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.864+0000 7f1dd17fa640 1 -- 172.21.15.12:0/3444315598 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1dc4000f80 con 0x7f1dec137d50 2024-09-11T01:29:19.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.865+0000 7f1dd37fe640 1 -- 172.21.15.12:0/3444315598 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc011d40 con 0x7f1dec137d50 2024-09-11T01:29:19.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.865+0000 7f1dd37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:19.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.865+0000 7f1dd37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:19.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.865+0000 7f1dd37fe640 10 monclient: dump: 2024-09-11T01:29:19.908 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:19.908 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.908 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.908 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.908 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:19.908 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.908 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:19.908 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:19.909 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:19.909 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.865+0000 7f1dd37fe640 1 -- 172.21.15.12:0/3444315598 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f1ddc007020 con 0x7f1dec137d50 2024-09-11T01:29:19.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.865+0000 7f1dd37fe640 1 --2- 172.21.15.12:0/3444315598 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1dd803e1c0 0x7f1dd8040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.865+0000 7f1dd37fe640 1 -- 172.21.15.12:0/3444315598 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4402+0+0 (secure 0 0 0) 0x7f1ddc052ab0 con 0x7f1dec137d50 2024-09-11T01:29:19.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.865+0000 7f1dd37fe640 1 --2- 172.21.15.12:0/3444315598 >> [v2:172.21.15.71:6808/2032569317,v1:172.21.15.71:6809/2032569317] conn(0x7f1dd8044030 0x7f1dd8046470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.865+0000 7f1dd37fe640 1 -- 172.21.15.12:0/3444315598 --> [v2:172.21.15.71:6808/2032569317,v1:172.21.15.71:6809/2032569317] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1ddc01fec0 con 0x7f1dd8044030 2024-09-11T01:29:19.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.865+0000 7f1dd37fe640 1 -- 172.21.15.12:0/3444315598 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f1ddc0572e0 con 0x7f1dec137d50 2024-09-11T01:29:19.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.865+0000 7f1dd37fe640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-11T01:29:19.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.865+0000 7f1dea7fc640 1 --2- 172.21.15.12:0/3444315598 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1dd803e1c0 0x7f1dd8040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.865+0000 7f1dea7fc640 10 monclient: get_auth_request con 0x7f1dd803e1c0 auth_method 0 2024-09-11T01:29:19.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.865+0000 7f1deb7fe640 1 --2- 172.21.15.12:0/3444315598 >> [v2:172.21.15.71:6808/2032569317,v1:172.21.15.71:6809/2032569317] conn(0x7f1dd8044030 0x7f1dd8046470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.865+0000 7f1deb7fe640 10 monclient: get_auth_request con 0x7f1dd8044030 auth_method 0 2024-09-11T01:29:19.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.866+0000 7f93e0951640 1 --2- 172.21.15.12:0/1003683211 >> [v2:172.21.15.71:6818/3091488141,v1:172.21.15.71:6819/3091488141] conn(0x7f93c8044180 0x7f93c80465c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:19.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.867+0000 7f93d8ff9640 1 -- 172.21.15.12:0/1003683211 <== osd.3 v2:172.21.15.71:6818/3091488141 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f93cc01ee70 con 0x7f93c8044180 2024-09-11T01:29:19.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.868+0000 7f1dea7fc640 1 --2- 172.21.15.12:0/3444315598 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1dd803e1c0 0x7f1dd8040680 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1de0031000 tx=0x7f1de002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:19.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.870+0000 7f1deb7fe640 1 --2- 172.21.15.12:0/3444315598 >> [v2:172.21.15.71:6808/2032569317,v1:172.21.15.71:6809/2032569317] conn(0x7f1dd8044030 0x7f1dd8046470 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:19.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.871+0000 7f1dd37fe640 1 -- 172.21.15.12:0/3444315598 <== osd.5 v2:172.21.15.71:6808/2032569317 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1ddc01fec0 con 0x7f1dd8044030 2024-09-11T01:29:19.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.885+0000 7f1dd17fa640 1 -- 172.21.15.12:0/3444315598 --> [v2:172.21.15.71:6808/2032569317,v1:172.21.15.71:6809/2032569317] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1dc4002c70 con 0x7f1dd8044030 2024-09-11T01:29:19.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.885+0000 7f93e23b4640 1 -- 172.21.15.12:0/1003683211 --> [v2:172.21.15.71:6818/3091488141,v1:172.21.15.71:6819/3091488141] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f93dc13c640 con 0x7f93c8044180 2024-09-11T01:29:19.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.885+0000 7f1dd37fe640 1 -- 172.21.15.12:0/3444315598 <== osd.5 v2:172.21.15.71:6808/2032569317 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1dc4002c70 con 0x7f1dd8044030 2024-09-11T01:29:19.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.885+0000 7f93d8ff9640 1 -- 172.21.15.12:0/1003683211 <== osd.3 v2:172.21.15.71:6818/3091488141 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f93dc13c640 con 0x7f93c8044180 2024-09-11T01:29:19.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.886+0000 7f1dd17fa640 1 -- 172.21.15.12:0/3444315598 >> [v2:172.21.15.71:6808/2032569317,v1:172.21.15.71:6809/2032569317] conn(0x7f1dd8044030 msgr2=0x7f1dd8046470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.886+0000 7f1dd17fa640 1 --2- 172.21.15.12:0/3444315598 >> [v2:172.21.15.71:6808/2032569317,v1:172.21.15.71:6809/2032569317] conn(0x7f1dd8044030 0x7f1dd8046470 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.886+0000 7f93e23b4640 1 -- 172.21.15.12:0/1003683211 >> [v2:172.21.15.71:6818/3091488141,v1:172.21.15.71:6819/3091488141] conn(0x7f93c8044180 msgr2=0x7f93c80465c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.886+0000 7f93e23b4640 1 --2- 172.21.15.12:0/1003683211 >> [v2:172.21.15.71:6818/3091488141,v1:172.21.15.71:6819/3091488141] conn(0x7f93c8044180 0x7f93c80465c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.886+0000 7f1dd17fa640 1 -- 172.21.15.12:0/3444315598 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1dd803e1c0 msgr2=0x7f1dd8040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.886+0000 7f1dd17fa640 1 --2- 172.21.15.12:0/3444315598 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1dd803e1c0 0x7f1dd8040680 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1de0031000 tx=0x7f1de002e000 comp rx=0 tx=0).stop 2024-09-11T01:29:19.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.886+0000 7f1dd17fa640 10 monclient: shutdown 2024-09-11T01:29:19.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.886+0000 7f1dd17fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:19.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.886+0000 7f1dd17fa640 1 -- 172.21.15.12:0/3444315598 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1dec137d50 msgr2=0x7f1dec136a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.886+0000 7f1dd17fa640 1 --2- 172.21.15.12:0/3444315598 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1dec137d50 0x7f1dec136a70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc00c430 tx=0x7f1ddc00c8f0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.886+0000 7f1deb7fe640 1 -- 172.21.15.12:0/3444315598 reap_dead start 2024-09-11T01:29:19.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.886+0000 7f1dd17fa640 1 -- 172.21.15.12:0/3444315598 shutdown_connections 2024-09-11T01:29:19.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.886+0000 7f1dd17fa640 1 -- 172.21.15.12:0/3444315598 >> 172.21.15.12:0/3444315598 conn(0x7f1dec0a7020 msgr2=0x7f1dec0a8420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:19.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.886+0000 7f93e23b4640 1 -- 172.21.15.12:0/1003683211 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f93c803e290 msgr2=0x7f93c8040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.886+0000 7f93e23b4640 1 --2- 172.21.15.12:0/1003683211 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f93c803e290 0x7f93c8040750 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f93d0000c00 tx=0x7f93d002e000 comp rx=0 tx=0).stop 2024-09-11T01:29:19.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.886+0000 7f93e23b4640 10 monclient: shutdown 2024-09-11T01:29:19.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.886+0000 7f93e23b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:19.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.886+0000 7f93e23b4640 1 -- 172.21.15.12:0/1003683211 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93dc137d50 msgr2=0x7f93dc1ccd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.886+0000 7f93e23b4640 1 --2- 172.21.15.12:0/1003683211 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93dc137d50 0x7f93dc1ccd90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f93cc00c440 tx=0x7f93cc00c900 comp rx=0 tx=0).stop 2024-09-11T01:29:19.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.886+0000 7f93e0951640 1 -- 172.21.15.12:0/1003683211 reap_dead start 2024-09-11T01:29:19.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.886+0000 7f93e23b4640 1 -- 172.21.15.12:0/1003683211 shutdown_connections 2024-09-11T01:29:19.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.886+0000 7f93e23b4640 1 -- 172.21.15.12:0/1003683211 >> 172.21.15.12:0/1003683211 conn(0x7f93dc0a7020 msgr2=0x7f93dc13e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:19.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.886+0000 7f1dd17fa640 1 -- 172.21.15.12:0/3444315598 shutdown_connections 2024-09-11T01:29:19.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.886+0000 7f1dd17fa640 1 -- 172.21.15.12:0/3444315598 wait complete. 2024-09-11T01:29:19.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.887+0000 7f93e23b4640 1 -- 172.21.15.12:0/1003683211 shutdown_connections 2024-09-11T01:29:19.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.887+0000 7f93e23b4640 1 -- 172.21.15.12:0/1003683211 wait complete. 2024-09-11T01:29:19.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.887+0000 7f04b0ba3640 10 monclient: get_monmap_and_config 2024-09-11T01:29:19.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.887+0000 7f04b0ba3640 10 monclient: build_initial_monmap 2024-09-11T01:29:19.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.887+0000 7f04b0ba3640 10 monclient: monmap: 2024-09-11T01:29:19.923 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:19.924 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.924 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:19.888547+0000 2024-09-11T01:29:19.924 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:19.888547+0000 2024-09-11T01:29:19.924 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:19.924 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.924 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:19.924 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:19.924 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:19.924 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:19.924 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:19.924 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.888+0000 7f04b0ba3640 1 Processor -- start 2024-09-11T01:29:19.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.888+0000 7f04b0ba3640 1 -- start start 2024-09-11T01:29:19.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.888+0000 7f04b0ba3640 10 monclient: init 2024-09-11T01:29:19.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.889+0000 7f04b0ba3640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:19.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.889+0000 7f04b0ba3640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-11T01:29:19.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.889+0000 7f04b0ba3640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f04ac137d50 0x7f04ac09d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.889+0000 7f04b0ba3640 10 monclient(hunting): picked mon.noname-c con 0x7f04ac137d50 addr v2:172.21.15.71:3300/0 2024-09-11T01:29:19.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.889+0000 7f04b0ba3640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f04ac138130 0x7f04ac138550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.889+0000 7f04b0ba3640 10 monclient(hunting): picked mon.noname-b con 0x7f04ac138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:19.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.889+0000 7f04b0ba3640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f04ac136cd0 0x7f04ac137120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04a9d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f04ac138130 0x7f04ac138550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04aad9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f04ac136cd0 0x7f04ac137120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04a9d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f04ac138130 0x7f04ac138550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51518/0 (socket says 172.21.15.12:51518) 2024-09-11T01:29:19.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04aad9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f04ac136cd0 0x7f04ac137120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54852/0 (socket says 172.21.15.12:54852) 2024-09-11T01:29:19.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04b0ba3640 10 monclient(hunting): picked mon.noname-a con 0x7f04ac136cd0 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:19.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04b0ba3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04b0ba3640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f04ac09c650 con 0x7f04ac136cd0 2024-09-11T01:29:19.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04b0ba3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04a9d9b640 1 -- 172.21.15.12:0/1950518544 learned_addr learned my addr 172.21.15.12:0/1950518544 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:19.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04b0ba3640 1 -- 172.21.15.12:0/1950518544 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f04ac09ba20 con 0x7f04ac137d50 2024-09-11T01:29:19.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04b0ba3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04b0ba3640 1 -- 172.21.15.12:0/1950518544 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f04ac09c370 con 0x7f04ac138130 2024-09-11T01:29:19.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04b0ba3640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:19.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04b0ba3640 10 monclient(hunting): authenticate will time out at 766.332130s 2024-09-11T01:29:19.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04a9d9b640 10 monclient(hunting): get_auth_request con 0x7f04ac138130 auth_method 0 2024-09-11T01:29:19.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04a9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:19.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04a9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:19.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:19.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04aad9d640 10 monclient(hunting): get_auth_request con 0x7f04ac136cd0 auth_method 0 2024-09-11T01:29:19.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:19.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04aad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:19.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:19.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04aad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:19.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04aad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:19.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04aad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:19.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04a9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:19.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04a9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:19.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04a9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:19.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04aad9d640 10 monclient(hunting): handle_auth_done global_id 4239 payload 274 2024-09-11T01:29:19.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04aad9d640 1 -- 172.21.15.12:0/1950518544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f04ac138130 msgr2=0x7f04ac138550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:19.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04aad9d640 1 --2- 172.21.15.12:0/1950518544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f04ac138130 0x7f04ac138550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04aad9d640 1 -- 172.21.15.12:0/1950518544 >> v2:172.21.15.71:3300/0 conn(0x7f04ac137d50 msgr2=0x7f04ac09d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:19.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04aa59c640 1 --2- 172.21.15.12:0/1950518544 >> v2:172.21.15.71:3300/0 conn(0x7f04ac137d50 0x7f04ac09d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04aad9d640 1 --2- 172.21.15.12:0/1950518544 >> v2:172.21.15.71:3300/0 conn(0x7f04ac137d50 0x7f04ac09d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04aad9d640 10 monclient: _finish_hunting 0 2024-09-11T01:29:19.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04aad9d640 1 monclient: found mon.noname-a 2024-09-11T01:29:19.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04aad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:19.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04aad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:19.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04aad9d640 1 -- 172.21.15.12:0/1950518544 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ac0a6ce0 con 0x7f04ac136cd0 2024-09-11T01:29:19.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.890+0000 7f04a9d9b640 1 --2- 172.21.15.12:0/1950518544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f04ac138130 0x7f04ac138550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:19.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.892+0000 7f7acb3c2640 10 monclient: get_monmap_and_config 2024-09-11T01:29:19.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.892+0000 7f7acb3c2640 10 monclient: build_initial_monmap 2024-09-11T01:29:19.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.892+0000 7f7acb3c2640 10 monclient: monmap: 2024-09-11T01:29:19.931 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:19.931 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.931 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:19.892829+0000 2024-09-11T01:29:19.931 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:19.892829+0000 2024-09-11T01:29:19.931 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:19.931 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.931 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:19.931 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:19.931 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:19.931 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:19.931 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:19.932 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.892+0000 7f7acb3c2640 1 Processor -- start 2024-09-11T01:29:19.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.893+0000 7f7acb3c2640 1 -- start start 2024-09-11T01:29:19.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.893+0000 7f7acb3c2640 10 monclient: init 2024-09-11T01:29:19.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.893+0000 7f7acb3c2640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:19.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.893+0000 7f7acb3c2640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-11T01:29:19.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.893+0000 7f7acb3c2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7ac4137d50 0x7f7ac409d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.893+0000 7f7acb3c2640 10 monclient(hunting): picked mon.noname-b con 0x7f7ac4137d50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:19.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.893+0000 7f7acb3c2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7ac4138130 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:19.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.893+0000 7f7acb3c2640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7ac413d280 0x7f7ac413f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.893+0000 7f7acb3c2640 10 monclient(hunting): picked mon.noname-a con 0x7f7ac413d280 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:19.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.893+0000 7f7acb3c2640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ac409ba20 con 0x7f7ac4138130 2024-09-11T01:29:19.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.893+0000 7f7acb3c2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.893+0000 7f7acb3c2640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f7ac409c650 con 0x7f7ac413d280 2024-09-11T01:29:19.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.893+0000 7f7acb3c2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.893+0000 7f7acb3c2640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7ac409c370 con 0x7f7ac4137d50 2024-09-11T01:29:19.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.893+0000 7f7acb3c2640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:19.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.893+0000 7f7ac895d640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f7ac4138130 0x7f7ac4138550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.12:45454/0 (socket says 172.21.15.12:45454) 2024-09-11T01:29:19.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.893+0000 7f7ac895d640 1 -- 172.21.15.12:0/3146428084 learned_addr learned my addr 172.21.15.12:0/3146428084 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:29:19.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.893+0000 7f7acb3c2640 10 monclient(hunting): authenticate will time out at 766.335906s 2024-09-11T01:29:19.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f04aad9d640 1 -- 172.21.15.12:0/1950518544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04ac136cd0 msgr2=0x7f04ac137120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:19.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f7ac995f640 1 --2- 172.21.15.12:0/3146428084 >> v2:172.21.15.12:3300/0 conn(0x7f7ac413d280 0x7f7ac413f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f7ac995f640 10 monclient(hunting): get_auth_request con 0x7f7ac413d280 auth_method 0 2024-09-11T01:29:19.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f7ac995f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:19.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f7ac995f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:19.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f7ac995f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:19.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f7ac915e640 1 --2- 172.21.15.12:0/3146428084 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7ac4137d50 0x7f7ac409d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f7ac915e640 10 monclient(hunting): get_auth_request con 0x7f7ac4137d50 auth_method 0 2024-09-11T01:29:19.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f7ac915e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:19.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f7ac915e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:19.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f7ac915e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:19.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f7ac995f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:19.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f7ac995f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:19.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f7ac995f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:19.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f7ac915e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:19.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f7ac915e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:19.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f7ac915e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:19.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f04aad9d640 1 --2- 172.21.15.12:0/1950518544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04ac136cd0 0x7f04ac137120 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0a6f10 tx=0x7f04a4032fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9afba113902bf4a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:19.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f15bf2db640 10 monclient: get_monmap_and_config 2024-09-11T01:29:19.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f15bf2db640 10 monclient: build_initial_monmap 2024-09-11T01:29:19.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f15bf2db640 10 monclient: monmap: 2024-09-11T01:29:19.937 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:19.937 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.937 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:19.895487+0000 2024-09-11T01:29:19.937 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:19.895487+0000 2024-09-11T01:29:19.937 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:19.937 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.937 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:19.937 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:19.937 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:19.937 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:19.937 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:19.938 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f7ac995f640 10 monclient(hunting): handle_auth_done global_id 4242 payload 274 2024-09-11T01:29:19.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f7ac995f640 1 -- 172.21.15.12:0/3146428084 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7ac4137d50 msgr2=0x7f7ac409d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:19.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f7ac995f640 1 --2- 172.21.15.12:0/3146428084 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7ac4137d50 0x7f7ac409d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f7ac995f640 1 -- 172.21.15.12:0/3146428084 >> v1:172.21.15.71:6789/0 conn(0x7f7ac4138130 legacy=0x7f7ac4138550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:19.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f7ac995f640 10 monclient: _finish_hunting 0 2024-09-11T01:29:19.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f7ac995f640 1 monclient: found mon.noname-a 2024-09-11T01:29:19.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f7ac995f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:19.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f7ac995f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:19.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f7ac995f640 1 -- 172.21.15.12:0/3146428084 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ac40a6ce0 con 0x7f7ac413d280 2024-09-11T01:29:19.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.895+0000 7f89ef02a640 10 monclient: get_monmap_and_config 2024-09-11T01:29:19.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.895+0000 7f89ef02a640 10 monclient: build_initial_monmap 2024-09-11T01:29:19.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.895+0000 7f89ef02a640 10 monclient: monmap: 2024-09-11T01:29:19.939 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:19.940 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.940 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:19.895777+0000 2024-09-11T01:29:19.940 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:19.895777+0000 2024-09-11T01:29:19.940 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:19.940 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.940 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:19.940 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:19.940 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:19.940 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:19.940 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:19.940 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.895+0000 7f7ac915e640 1 --2- 172.21.15.12:0/3146428084 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7ac4137d50 0x7f7ac409d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:19.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f04a959a640 1 -- 172.21.15.12:0/1950518544 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a4040070 con 0x7f04ac136cd0 2024-09-11T01:29:19.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f04a959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:19.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f04a959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:19.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f04a959a640 10 monclient: dump: 2024-09-11T01:29:19.941 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:19.942 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.942 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.942 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.942 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:19.942 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.942 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:19.942 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:19.942 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:19.942 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f04a959a640 10 monclient: _finish_auth 0 2024-09-11T01:29:19.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f04a959a640 10 monclient: _check_auth_tickets 2024-09-11T01:29:19.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f04a959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:19.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f04a959a640 1 -- 172.21.15.12:0/1950518544 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04a4033d00 con 0x7f04ac136cd0 2024-09-11T01:29:19.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f04a959a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:19.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f04b0ba3640 5 monclient: authenticate success, global_id 4239 2024-09-11T01:29:19.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f04b0ba3640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:19.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f04b0ba3640 4 set_mon_vals no callback set 2024-09-11T01:29:19.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f04a959a640 1 -- 172.21.15.12:0/1950518544 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a403b660 con 0x7f04ac136cd0 2024-09-11T01:29:19.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f04a959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:19.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f04a959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:19.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.894+0000 7f04a959a640 10 monclient: dump: 2024-09-11T01:29:19.943 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:19.943 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.944 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.944 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.944 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:19.944 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.944 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:19.944 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:19.944 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:19.944 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.895+0000 7f15bf2db640 1 Processor -- start 2024-09-11T01:29:19.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.895+0000 7f15bf2db640 1 -- start start 2024-09-11T01:29:19.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.895+0000 7f15bf2db640 10 monclient: init 2024-09-11T01:29:19.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.895+0000 7f04b0ba3640 10 monclient: shutdown 2024-09-11T01:29:19.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.895+0000 7f04b0ba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:19.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.895+0000 7f04b0ba3640 1 -- 172.21.15.12:0/1950518544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04ac136cd0 msgr2=0x7f04ac137120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.895+0000 7f04b0ba3640 1 --2- 172.21.15.12:0/1950518544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04ac136cd0 0x7f04ac137120 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0a6f10 tx=0x7f04a4032fb0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.896+0000 7f89ef02a640 1 Processor -- start 2024-09-11T01:29:19.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.896+0000 7f04b0ba3640 1 -- 172.21.15.12:0/1950518544 shutdown_connections 2024-09-11T01:29:19.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.896+0000 7f04b0ba3640 1 --2- 172.21.15.12:0/1950518544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04ac136cd0 0x7f04ac137120 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.896+0000 7f04b0ba3640 1 --2- 172.21.15.12:0/1950518544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f04ac138130 0x7f04ac138550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.896+0000 7f04b0ba3640 1 --2- 172.21.15.12:0/1950518544 >> v2:172.21.15.71:3300/0 conn(0x7f04ac137d50 0x7f04ac09d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.896+0000 7f04b0ba3640 1 -- 172.21.15.12:0/1950518544 >> 172.21.15.12:0/1950518544 conn(0x7f04ac0a7020 msgr2=0x7f04ac0a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:19.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.896+0000 7f04b0ba3640 1 -- 172.21.15.12:0/1950518544 shutdown_connections 2024-09-11T01:29:19.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.896+0000 7f89ef02a640 1 -- start start 2024-09-11T01:29:19.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.896+0000 7f89ef02a640 10 monclient: init 2024-09-11T01:29:19.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.896+0000 7f04b0ba3640 1 -- 172.21.15.12:0/1950518544 wait complete. 2024-09-11T01:29:19.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.896+0000 7f15bf2db640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:19.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.896+0000 7f15bf2db640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-11T01:29:19.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.896+0000 7f15bf2db640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f15b00293b0 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:19.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.896+0000 7f15bf2db640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15b0027af0 0x7f15b00c08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.896+0000 7f15bf2db640 10 monclient(hunting): picked mon.noname-b con 0x7f15b0027af0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:19.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.896+0000 7f15bf2db640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f15b00c0e10 0x7f15b00c3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.896+0000 7f15bf2db640 10 monclient(hunting): picked mon.noname-c con 0x7f15b00c0e10 addr v2:172.21.15.71:3300/0 2024-09-11T01:29:19.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.896+0000 7f15be2d9640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f15b00293b0 0x7f15b0026ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56018/0 (socket says 172.21.15.12:56018) 2024-09-11T01:29:19.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.896+0000 7f15be2d9640 1 -- 172.21.15.12:0/2591870182 learned_addr learned my addr 172.21.15.12:0/2591870182 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:29:19.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.896+0000 7f15bf2db640 1 -- 172.21.15.12:0/2591870182 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15b0022d90 con 0x7f15b00293b0 2024-09-11T01:29:19.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.896+0000 7f15bdad8640 1 --2- 172.21.15.12:0/2591870182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15b0027af0 0x7f15b00c08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.896+0000 7f15bf2db640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.896+0000 7f15bf2db640 1 -- 172.21.15.12:0/2591870182 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f15b0029790 con 0x7f15b00c0e10 2024-09-11T01:29:19.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.896+0000 7f15beada640 1 --2- 172.21.15.12:0/2591870182 >> v2:172.21.15.71:3300/0 conn(0x7f15b00c0e10 0x7f15b00c3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.896+0000 7f15bf2db640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.896+0000 7f15bf2db640 1 -- 172.21.15.12:0/2591870182 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f15b0021510 con 0x7f15b0027af0 2024-09-11T01:29:19.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f15bf2db640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:19.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f15bf2db640 10 monclient(hunting): authenticate will time out at 766.339017s 2024-09-11T01:29:19.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f15bdad8640 10 monclient(hunting): get_auth_request con 0x7f15b0027af0 auth_method 0 2024-09-11T01:29:19.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f15bdad8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:19.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f15bdad8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:19.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f15bdad8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:19.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f15beada640 10 monclient(hunting): get_auth_request con 0x7f15b00c0e10 auth_method 0 2024-09-11T01:29:19.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f15beada640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:19.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f15beada640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:19.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f15beada640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:19.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f15beada640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:19.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f15beada640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:19.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f15beada640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:19.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f89ef02a640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:19.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f89ef02a640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-11T01:29:19.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f89ef02a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f89e8137d50 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:19.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f15bdad8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:19.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f15bdad8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:19.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f15bdad8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:19.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f15beada640 10 monclient(hunting): handle_auth_done global_id 4219 payload 274 2024-09-11T01:29:19.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f15beada640 1 -- 172.21.15.12:0/2591870182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15b0027af0 msgr2=0x7f15b00c08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:19.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f15beada640 1 --2- 172.21.15.12:0/2591870182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15b0027af0 0x7f15b00c08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f15beada640 1 -- 172.21.15.12:0/2591870182 >> v1:172.21.15.12:6789/0 conn(0x7f15b00293b0 legacy=0x7f15b0026ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:19.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f15beada640 10 monclient: _finish_hunting 0 2024-09-11T01:29:19.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f15beada640 1 monclient: found mon.noname-c 2024-09-11T01:29:19.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f15beada640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:19.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f15beada640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:29:19.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f15beada640 1 -- 172.21.15.12:0/2591870182 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b0023370 con 0x7f15b00c0e10 2024-09-11T01:29:19.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f89ef02a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f89e8138130 0x7f89e8138550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f89ef02a640 10 monclient(hunting): picked mon.noname-b con 0x7f89e8138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:19.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f89ef02a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f89e813d280 0x7f89e813f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f89ef02a640 10 monclient(hunting): picked mon.noname-a con 0x7f89e813d280 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:19.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f89ef02a640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89e809d820 con 0x7f89e8137d50 2024-09-11T01:29:19.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f89ef02a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f89ef02a640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f89e809c650 con 0x7f89e813d280 2024-09-11T01:29:19.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f89ef02a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f89ef02a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f89e809ba20 con 0x7f89e8138130 2024-09-11T01:29:19.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f89ef02a640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:19.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f89ef02a640 10 monclient(hunting): authenticate will time out at 766.339317s 2024-09-11T01:29:19.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f89ed5c7640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f89e813d280 0x7f89e813f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f89e7fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f89e8138130 0x7f89e8138550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f89e7fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f89e8138130 0x7f89e8138550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51560/0 (socket says 172.21.15.12:51560) 2024-09-11T01:29:19.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f89e7fff640 1 -- 172.21.15.12:0/3587000249 learned_addr learned my addr 172.21.15.12:0/3587000249 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:19.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f89e7fff640 10 monclient(hunting): get_auth_request con 0x7f89e8138130 auth_method 0 2024-09-11T01:29:19.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f89e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:19.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f89e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:19.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f89e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:19.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f89ed5c7640 10 monclient(hunting): get_auth_request con 0x7f89e813d280 auth_method 0 2024-09-11T01:29:19.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f89ed5c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:19.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f89ed5c7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:19.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f7ac995f640 1 -- 172.21.15.12:0/3146428084 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ac413d280 msgr2=0x7f7ac413f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:19.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f7ac995f640 1 --2- 172.21.15.12:0/3146428084 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ac413d280 0x7f7ac413f670 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7ac409c7d0 tx=0x7f7abc02a510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14aa9c1b4733894d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:19.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f89ed5c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:19.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f89e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:19.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f89e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:19.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f89e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:19.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f04b0ba3640 10 monclient: build_initial_monmap 2024-09-11T01:29:19.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.897+0000 7f04b0ba3640 10 monclient: monmap: 2024-09-11T01:29:19.959 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:19.959 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:19.959 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:19.959 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:19.959 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:19.959 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.959 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:19.959 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:19.959 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:19.960 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f7abbfff640 1 -- 172.21.15.12:0/3146428084 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7abc007950 con 0x7f7ac413d280 2024-09-11T01:29:19.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f7abbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:19.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f7abbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:19.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f7abbfff640 10 monclient: dump: 2024-09-11T01:29:19.960 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:19.960 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.960 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.961 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.961 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:19.961 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.961 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:19.961 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:19.961 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:19.961 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f7abbfff640 10 monclient: _finish_auth 0 2024-09-11T01:29:19.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f7abbfff640 10 monclient: _check_auth_tickets 2024-09-11T01:29:19.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f7abbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:19.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f7abbfff640 1 -- 172.21.15.12:0/3146428084 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7abc007af0 con 0x7f7ac413d280 2024-09-11T01:29:19.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f7abbfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:19.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f89e7fff640 10 monclient(hunting): handle_auth_done global_id 4259 payload 274 2024-09-11T01:29:19.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f89e7fff640 1 -- 172.21.15.12:0/3587000249 >> v2:172.21.15.12:3300/0 conn(0x7f89e813d280 msgr2=0x7f89e813f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:19.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f7abbfff640 1 -- 172.21.15.12:0/3146428084 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7abc031400 con 0x7f7ac413d280 2024-09-11T01:29:19.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f7acb3c2640 5 monclient: authenticate success, global_id 4242 2024-09-11T01:29:19.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f7acb3c2640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:19.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f7acb3c2640 4 set_mon_vals no callback set 2024-09-11T01:29:19.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f7abbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:19.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f7abbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:19.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f7abbfff640 10 monclient: dump: 2024-09-11T01:29:19.963 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:19.963 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.963 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.963 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.963 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:19.963 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.963 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:19.963 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:19.963 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:19.964 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f04b0ba3640 1 Processor -- start 2024-09-11T01:29:19.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f89e7fff640 1 --2- 172.21.15.12:0/3587000249 >> v2:172.21.15.12:3300/0 conn(0x7f89e813d280 0x7f89e813f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f89e7fff640 1 -- 172.21.15.12:0/3587000249 >> v1:172.21.15.12:6789/0 conn(0x7f89e8137d50 legacy=0x7f89e809d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:19.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f89e7fff640 10 monclient: _finish_hunting 0 2024-09-11T01:29:19.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f89e7fff640 1 monclient: found mon.noname-b 2024-09-11T01:29:19.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f89e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:19.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f89e7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:29:19.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f89e7fff640 1 -- 172.21.15.12:0/3587000249 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e809c370 con 0x7f89e8138130 2024-09-11T01:29:19.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f89ed5c7640 1 --2- 172.21.15.12:0/3587000249 >> v2:172.21.15.12:3300/0 conn(0x7f89e813d280 0x7f89e813f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:29:19.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f04b0ba3640 1 -- start start 2024-09-11T01:29:19.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.898+0000 7f04b0ba3640 10 monclient: init 2024-09-11T01:29:19.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f7acb3c2640 10 monclient: shutdown 2024-09-11T01:29:19.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f7acb3c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:19.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f7acb3c2640 1 -- 172.21.15.12:0/3146428084 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ac413d280 msgr2=0x7f7ac413f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f7acb3c2640 1 --2- 172.21.15.12:0/3146428084 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ac413d280 0x7f7ac413f670 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7ac409c7d0 tx=0x7f7abc02a510 comp rx=0 tx=0).stop 2024-09-11T01:29:19.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04b0ba3640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:19.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04b0ba3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:29:19.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04b0ba3640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f04ac136cd0 0x7f04ac1214e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04b0ba3640 10 monclient(hunting): picked mon.noname-c con 0x7f04ac136cd0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:19.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f7acb3c2640 1 -- 172.21.15.12:0/3146428084 shutdown_connections 2024-09-11T01:29:19.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f7acb3c2640 1 --2- 172.21.15.12:0/3146428084 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ac413d280 0x7f7ac413f670 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f7acb3c2640 1 --2- 172.21.15.12:0/3146428084 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7ac4137d50 0x7f7ac409d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f7acb3c2640 1 -- 172.21.15.12:0/3146428084 >> 172.21.15.12:0/3146428084 conn(0x7f7ac40a7020 msgr2=0x7f7ac40a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:19.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04b0ba3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04ac137d50 0x7f04ac1c8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04b0ba3640 10 monclient(hunting): picked mon.noname-a con 0x7f04ac137d50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:19.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04b0ba3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04ac138130 0x7f04ac136330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f7acb3c2640 1 -- 172.21.15.12:0/3146428084 shutdown_connections 2024-09-11T01:29:19.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f7acb3c2640 1 -- 172.21.15.12:0/3146428084 wait complete. 2024-09-11T01:29:19.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04aa59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f04ac136cd0 0x7f04ac1214e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04aa59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f04ac136cd0 0x7f04ac1214e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51572/0 (socket says 172.21.15.12:51572) 2024-09-11T01:29:19.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04aa59c640 1 -- 172.21.15.12:0/3333812169 learned_addr learned my addr 172.21.15.12:0/3333812169 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:19.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04a9d9b640 1 --2- 172.21.15.12:0/3333812169 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04ac137d50 0x7f04ac1c8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04aad9d640 1 --2- 172.21.15.12:0/3333812169 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04ac138130 0x7f04ac136330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04b0ba3640 10 monclient(hunting): picked mon.noname-b con 0x7f04ac138130 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:19.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04b0ba3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04b0ba3640 1 -- 172.21.15.12:0/3333812169 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f04ac06f850 con 0x7f04ac137d50 2024-09-11T01:29:19.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04b0ba3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04b0ba3640 1 -- 172.21.15.12:0/3333812169 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f04ac06f6d0 con 0x7f04ac138130 2024-09-11T01:29:19.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04b0ba3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04b0ba3640 1 -- 172.21.15.12:0/3333812169 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f04ac09d650 con 0x7f04ac136cd0 2024-09-11T01:29:19.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04b0ba3640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:19.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04b0ba3640 10 monclient(hunting): authenticate will time out at 766.341782s 2024-09-11T01:29:19.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04aa59c640 10 monclient(hunting): get_auth_request con 0x7f04ac136cd0 auth_method 0 2024-09-11T01:29:19.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:19.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04aa59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:19.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:19.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04a9d9b640 10 monclient(hunting): get_auth_request con 0x7f04ac137d50 auth_method 0 2024-09-11T01:29:19.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04a9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:19.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04a9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:19.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:19.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04aad9d640 10 monclient(hunting): get_auth_request con 0x7f04ac138130 auth_method 0 2024-09-11T01:29:19.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:19.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04aad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:19.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:19.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04aa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:19.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04aa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:19.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f7acb3c2640 10 monclient: build_initial_monmap 2024-09-11T01:29:19.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04aa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:19.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f7acb3c2640 10 monclient: monmap: 2024-09-11T01:29:19.975 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:19.975 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:19.975 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:19.975 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:19.976 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:19.976 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.976 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:19.976 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:19.976 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:19.976 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f15beada640 1 -- 172.21.15.12:0/2591870182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15b00c0e10 msgr2=0x7f15b00c3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:19.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f15beada640 1 --2- 172.21.15.12:0/2591870182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15b00c0e10 0x7f15b00c3200 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f15b8000b60 tx=0x7f15b80a6cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8244f1f2efe759f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:19.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04a9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:19.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04a9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:19.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.899+0000 7f04a9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:19.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f04aad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:19.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f04aad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:19.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f04aad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:19.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f15bd2d7640 1 -- 172.21.15.12:0/2591870182 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b80b4070 con 0x7f15b00c0e10 2024-09-11T01:29:19.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f15bd2d7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:19.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f15bd2d7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:19.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f15bd2d7640 10 monclient: dump: 2024-09-11T01:29:19.979 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:19.979 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.979 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.979 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.979 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:19.979 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.979 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:19.980 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:19.980 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:19.980 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f15bd2d7640 10 monclient: _finish_auth 0 2024-09-11T01:29:19.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f15bd2d7640 10 monclient: _check_auth_tickets 2024-09-11T01:29:19.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f15bd2d7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:19.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f15bd2d7640 1 -- 172.21.15.12:0/2591870182 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15b80a7ce0 con 0x7f15b00c0e10 2024-09-11T01:29:19.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f15bf2db640 5 monclient: authenticate success, global_id 4219 2024-09-11T01:29:19.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f15bd2d7640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:19.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f15bf2db640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:19.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f15bf2db640 4 set_mon_vals no callback set 2024-09-11T01:29:19.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f15bd2d7640 1 -- 172.21.15.12:0/2591870182 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b80af660 con 0x7f15b00c0e10 2024-09-11T01:29:19.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f15bd2d7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:19.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f15bd2d7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:19.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f15bd2d7640 10 monclient: dump: 2024-09-11T01:29:19.981 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:19.982 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.982 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.982 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.982 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:19.982 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.982 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:19.982 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:19.982 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:19.982 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f04aa59c640 10 monclient(hunting): handle_auth_done global_id 4262 payload 824 2024-09-11T01:29:19.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f04aa59c640 1 -- 172.21.15.12:0/3333812169 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04ac138130 msgr2=0x7f04ac136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f04aa59c640 1 --2- 172.21.15.12:0/3333812169 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04ac138130 0x7f04ac136330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f04aa59c640 1 -- 172.21.15.12:0/3333812169 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04ac137d50 msgr2=0x7f04ac1c8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f04aa59c640 1 --2- 172.21.15.12:0/3333812169 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04ac137d50 0x7f04ac1c8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f7acb3c2640 1 Processor -- start 2024-09-11T01:29:19.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f04aa59c640 10 monclient: _finish_hunting 0 2024-09-11T01:29:19.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f04aa59c640 1 monclient: found mon.noname-c 2024-09-11T01:29:19.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f04aa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:19.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.901+0000 7f7acb3c2640 1 -- start start 2024-09-11T01:29:19.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.901+0000 7f7acb3c2640 10 monclient: init 2024-09-11T01:29:19.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f04aa59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:29:19.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f04aa59c640 1 -- 172.21.15.12:0/3333812169 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ac0a85b0 con 0x7f04ac136cd0 2024-09-11T01:29:19.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.900+0000 7f04a9d9b640 1 --2- 172.21.15.12:0/3333812169 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04ac137d50 0x7f04ac1c8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:19.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.901+0000 7f15bf2db640 10 monclient: shutdown 2024-09-11T01:29:19.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.901+0000 7f15bf2db640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:19.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.901+0000 7f15bf2db640 1 -- 172.21.15.12:0/2591870182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15b00c0e10 msgr2=0x7f15b00c3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:19.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.901+0000 7f15bf2db640 1 --2- 172.21.15.12:0/2591870182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15b00c0e10 0x7f15b00c3200 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f15b8000b60 tx=0x7f15b80a6cc0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.901+0000 7f15bf2db640 1 -- 172.21.15.12:0/2591870182 shutdown_connections 2024-09-11T01:29:19.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.901+0000 7f15bf2db640 1 --2- 172.21.15.12:0/2591870182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15b00c0e10 0x7f15b00c3200 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.901+0000 7f15bf2db640 1 --2- 172.21.15.12:0/2591870182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15b0027af0 0x7f15b00c08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:19.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.901+0000 7f15bf2db640 1 -- 172.21.15.12:0/2591870182 >> 172.21.15.12:0/2591870182 conn(0x7f15b0006390 msgr2=0x7f15b00067a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:19.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.901+0000 7f89e7fff640 1 -- 172.21.15.12:0/3587000249 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f89e8138130 msgr2=0x7f89e8138550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:19.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.901+0000 7f89e7fff640 1 --2- 172.21.15.12:0/3587000249 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f89e8138130 0x7f89e8138550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f89d4009870 tx=0x7f89d402a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5269c7b8f3c8c19c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:19.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.901+0000 7f15bf2db640 1 -- 172.21.15.12:0/2591870182 shutdown_connections 2024-09-11T01:29:19.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.901+0000 7f7acb3c2640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:19.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.901+0000 7f15bf2db640 1 -- 172.21.15.12:0/2591870182 wait complete. 2024-09-11T01:29:19.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.901+0000 7f7acb3c2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:29:19.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.901+0000 7f7acb3c2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ac4137d50 0x7f7ac41ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.901+0000 7f7acb3c2640 10 monclient(hunting): picked mon.noname-b con 0x7f7ac4137d50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:19.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.901+0000 7f7acb3c2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7ac4138130 0x7f7ac4136df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.901+0000 7f89e77fe640 1 -- 172.21.15.12:0/3587000249 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d4037070 con 0x7f89e8138130 2024-09-11T01:29:19.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f89e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:19.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f89e77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:19.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7ac915e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ac4137d50 0x7f7ac41ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f89e77fe640 10 monclient: dump: 2024-09-11T01:29:19.993 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:19.993 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.993 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.993 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.993 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:19.993 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.993 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:19.993 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:19.993 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:19.994 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f89e77fe640 10 monclient: _finish_auth 0 2024-09-11T01:29:19.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7ac915e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ac4137d50 0x7f7ac41ccd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:34740/0 (socket says 172.21.15.12:34740) 2024-09-11T01:29:19.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f89e77fe640 10 monclient: _check_auth_tickets 2024-09-11T01:29:19.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f89e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:19.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7acb3c2640 10 monclient(hunting): picked mon.noname-c con 0x7f7ac4138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:19.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7ac915e640 1 -- 172.21.15.12:0/1804404325 learned_addr learned my addr 172.21.15.12:0/1804404325 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:19.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f89e77fe640 1 -- 172.21.15.12:0/3587000249 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89d4032640 con 0x7f89e8138130 2024-09-11T01:29:19.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f89e77fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:19.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7acb3c2640 1 --2- 172.21.15.12:0/1804404325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ac413d280 0x7f7ac413a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:19.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f89e77fe640 1 -- 172.21.15.12:0/3587000249 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d4032a10 con 0x7f89e8138130 2024-09-11T01:29:19.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f89e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:19.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f89e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:19.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f89e77fe640 10 monclient: dump: 2024-09-11T01:29:19.996 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:19.997 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:19.997 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.997 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:19.997 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:19.997 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:19.997 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:19.997 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:19.997 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:19.997 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:19.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7acb3c2640 10 monclient(hunting): picked mon.noname-a con 0x7f7ac413d280 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:19.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7acb3c2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7acb3c2640 1 -- 172.21.15.12:0/1804404325 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7ac409d7d0 con 0x7f7ac413d280 2024-09-11T01:29:19.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f89ef02a640 5 monclient: authenticate success, global_id 4259 2024-09-11T01:29:19.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7ac995f640 1 --2- 172.21.15.12:0/1804404325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ac413d280 0x7f7ac413a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:19.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7acb3c2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:19.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f89ef02a640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:19.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f89ef02a640 4 set_mon_vals no callback set 2024-09-11T01:29:19.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7acb3c2640 1 -- 172.21.15.12:0/1804404325 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f7ac409d650 con 0x7f7ac4137d50 2024-09-11T01:29:20.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7acb3c2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7acb3c2640 1 -- 172.21.15.12:0/1804404325 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7ac413fbb0 con 0x7f7ac4138130 2024-09-11T01:29:20.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7acb3c2640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:20.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7acb3c2640 10 monclient(hunting): authenticate will time out at 766.344587s 2024-09-11T01:29:20.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7ac895d640 1 --2- 172.21.15.12:0/1804404325 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7ac4138130 0x7f7ac4136df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7ac915e640 10 monclient(hunting): get_auth_request con 0x7f7ac4137d50 auth_method 0 2024-09-11T01:29:20.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7ac915e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7ac915e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7ac915e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7ac995f640 10 monclient(hunting): get_auth_request con 0x7f7ac413d280 auth_method 0 2024-09-11T01:29:20.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7ac995f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7ac995f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7ac995f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7ac895d640 10 monclient(hunting): get_auth_request con 0x7f7ac4138130 auth_method 0 2024-09-11T01:29:20.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7ac895d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7ac895d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7ac895d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f15bf2db640 10 monclient: build_initial_monmap 2024-09-11T01:29:20.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f15bf2db640 10 monclient: monmap: 2024-09-11T01:29:20.001 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:20.001 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:20.001 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:20.002 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:20.002 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:20.002 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.002 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:20.002 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:20.002 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:20.002 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f15bf2db640 1 Processor -- start 2024-09-11T01:29:20.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f04aa59c640 1 -- 172.21.15.12:0/3333812169 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f04ac136cd0 msgr2=0x7f04ac1214e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:20.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f04aa59c640 1 --2- 172.21.15.12:0/3333812169 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f04ac136cd0 0x7f04ac1214e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f049c00c7f0 tx=0x7f049c00ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f89ef02a640 10 monclient: shutdown 2024-09-11T01:29:20.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f89ef02a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:20.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f89ef02a640 1 -- 172.21.15.12:0/3587000249 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f89e8138130 msgr2=0x7f89e8138550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f89ef02a640 1 --2- 172.21.15.12:0/3587000249 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f89e8138130 0x7f89e8138550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f89d4009870 tx=0x7f89d402a130 comp rx=0 tx=0).stop 2024-09-11T01:29:20.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f89ef02a640 1 -- 172.21.15.12:0/3587000249 shutdown_connections 2024-09-11T01:29:20.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f15bf2db640 1 -- start start 2024-09-11T01:29:20.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f89ef02a640 1 --2- 172.21.15.12:0/3587000249 >> v2:172.21.15.12:3300/0 conn(0x7f89e813d280 0x7f89e813f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f89ef02a640 1 --2- 172.21.15.12:0/3587000249 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f89e8138130 0x7f89e8138550 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f89ef02a640 1 -- 172.21.15.12:0/3587000249 >> 172.21.15.12:0/3587000249 conn(0x7f89e80a7020 msgr2=0x7f89e80a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:20.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f15bf2db640 10 monclient: init 2024-09-11T01:29:20.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f89ef02a640 1 -- 172.21.15.12:0/3587000249 shutdown_connections 2024-09-11T01:29:20.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7ac995f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7ac995f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.902+0000 7f7ac995f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f7ac895d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f7ac895d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f7ac895d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f7ac915e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f7ac915e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f7ac915e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f89ef02a640 1 -- 172.21.15.12:0/3587000249 wait complete. 2024-09-11T01:29:20.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f7ac915e640 10 monclient(hunting): handle_auth_done global_id 4225 payload 824 2024-09-11T01:29:20.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f7ac915e640 1 -- 172.21.15.12:0/1804404325 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7ac4138130 msgr2=0x7f7ac4136df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f049affd640 1 -- 172.21.15.12:0/3333812169 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049c015070 con 0x7f04ac136cd0 2024-09-11T01:29:20.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f049affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f049affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:20.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f049affd640 10 monclient: dump: 2024-09-11T01:29:20.007 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.007 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.007 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.007 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.007 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.007 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.007 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.008 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.008 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.008 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f049affd640 10 monclient: _finish_auth 0 2024-09-11T01:29:20.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f049affd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:20.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f049affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:20.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f049affd640 1 -- 172.21.15.12:0/3333812169 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f049c004240 con 0x7f04ac136cd0 2024-09-11T01:29:20.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f04b0ba3640 5 monclient: authenticate success, global_id 4262 2024-09-11T01:29:20.009 INFO:teuthology.orchestra.run.smithi012.stdout:51539607556 2024-09-11T01:29:20.009 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-11T01:29:20.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f7ac915e640 1 --2- 172.21.15.12:0/1804404325 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7ac4138130 0x7f7ac4136df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f7ac915e640 1 -- 172.21.15.12:0/1804404325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ac413d280 msgr2=0x7f7ac413a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f7ac915e640 1 --2- 172.21.15.12:0/1804404325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ac413d280 0x7f7ac413a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f7ac915e640 10 monclient: _finish_hunting 0 2024-09-11T01:29:20.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f7ac915e640 1 monclient: found mon.noname-b 2024-09-11T01:29:20.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f7ac915e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:20.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f04b0ba3640 10 monclient: _renew_subs 2024-09-11T01:29:20.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f04b0ba3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:20.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f04b0ba3640 1 -- 172.21.15.12:0/3333812169 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04ac0a1710 con 0x7f04ac136cd0 2024-09-11T01:29:20.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f04b0ba3640 10 monclient: _renew_subs 2024-09-11T01:29:20.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f04b0ba3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:20.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f04b0ba3640 1 -- 172.21.15.12:0/3333812169 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f04ac121e50 con 0x7f04ac136cd0 2024-09-11T01:29:20.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f89ef02a640 10 monclient: build_initial_monmap 2024-09-11T01:29:20.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f89ef02a640 10 monclient: monmap: 2024-09-11T01:29:20.012 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:20.012 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:20.012 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:20.012 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:20.012 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:20.012 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.012 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:20.012 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:20.012 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:20.012 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.903+0000 7f7ac915e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-09-11T01:29:20.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f7ac915e640 1 -- 172.21.15.12:0/1804404325 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ac413f510 con 0x7f7ac4137d50 2024-09-11T01:29:20.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f15bf2db640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:20.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f15bf2db640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:29:20.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f15bf2db640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15b0027af0 0x7f15b00bae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f15bf2db640 10 monclient(hunting): picked mon.noname-a con 0x7f15b0027af0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:20.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f15bf2db640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15b00293b0 0x7f15b000a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f15bf2db640 10 monclient(hunting): picked mon.noname-c con 0x7f15b00293b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:20.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f049affd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:20.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f049affd640 1 -- 172.21.15.12:0/3333812169 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049c007830 con 0x7f04ac136cd0 2024-09-11T01:29:20.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f049affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f049affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:20.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f049affd640 10 monclient: dump: 2024-09-11T01:29:20.014 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.014 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.014 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.014 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.014 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.014 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.015 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.015 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.015 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.015 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f049bfff640 4 set_mon_vals no callback set 2024-09-11T01:29:20.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f15bf2db640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15b00c0e10 0x7f15b000ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f15bf2db640 10 monclient(hunting): picked mon.noname-b con 0x7f15b00c0e10 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:20.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f15bf2db640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f15bf2db640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f15b0027700 con 0x7f15b0027af0 2024-09-11T01:29:20.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f15bf2db640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.016 INFO:teuthology.orchestra.run.smithi012.stdout:51539607556 2024-09-11T01:29:20.016 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T01:29:20.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f15bf2db640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f15b00bba30 con 0x7f15b00c0e10 2024-09-11T01:29:20.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f15bf2db640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f15bf2db640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f15b0027880 con 0x7f15b00293b0 2024-09-11T01:29:20.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f15bf2db640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:20.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f15bf2db640 10 monclient(hunting): authenticate will time out at 766.346833s 2024-09-11T01:29:20.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f89ef02a640 1 Processor -- start 2024-09-11T01:29:20.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f89ef02a640 1 -- start start 2024-09-11T01:29:20.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f89ef02a640 10 monclient: init 2024-09-11T01:29:20.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f0498ff9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:20.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f0498ff9640 1 -- 172.21.15.12:0/3333812169 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f04ac122500 con 0x7f04ac136cd0 2024-09-11T01:29:20.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f7ac995f640 1 --2- 172.21.15.12:0/1804404325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ac413d280 0x7f7ac413a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:20.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f7ac895d640 1 --2- 172.21.15.12:0/1804404325 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7ac4138130 0x7f7ac4136df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:20.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f15beada640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15b00c0e10 0x7f15b000ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f15be2d9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15b0027af0 0x7f15b00bae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.904+0000 7f15beada640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15b00c0e10 0x7f15b000ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:34756/0 (socket says 172.21.15.12:34756) 2024-09-11T01:29:20.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15beada640 1 -- 172.21.15.12:0/3535082076 learned_addr learned my addr 172.21.15.12:0/3535082076 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:20.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15beada640 10 monclient(hunting): get_auth_request con 0x7f15b00c0e10 auth_method 0 2024-09-11T01:29:20.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15beada640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15beada640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15beada640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15be2d9640 10 monclient(hunting): get_auth_request con 0x7f15b0027af0 auth_method 0 2024-09-11T01:29:20.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15be2d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15be2d9640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15be2d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15beada640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15beada640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ef02a640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:20.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ef02a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:29:20.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ef02a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f89e8137d50 0x7f89e8073240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15bdad8640 1 --2- 172.21.15.12:0/3535082076 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15b00293b0 0x7f15b000a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f049affd640 1 -- 172.21.15.12:0/3333812169 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f049c007a50 con 0x7f04ac136cd0 2024-09-11T01:29:20.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15beada640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ef02a640 10 monclient(hunting): picked mon.noname-c con 0x7f89e8137d50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:20.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15bdad8640 10 monclient(hunting): get_auth_request con 0x7f15b00293b0 auth_method 0 2024-09-11T01:29:20.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15bdad8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15bdad8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15bdad8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15be2d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15be2d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15be2d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15bdad8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15bdad8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15bdad8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15beada640 10 monclient(hunting): handle_auth_done global_id 4228 payload 824 2024-09-11T01:29:20.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ef02a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89e8138130 0x7f89e813a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15beada640 1 -- 172.21.15.12:0/3535082076 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15b00293b0 msgr2=0x7f15b000a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15beada640 1 --2- 172.21.15.12:0/3535082076 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15b00293b0 0x7f15b000a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15beada640 1 -- 172.21.15.12:0/3535082076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15b0027af0 msgr2=0x7f15b00bae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15beada640 1 --2- 172.21.15.12:0/3535082076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15b0027af0 0x7f15b00bae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15beada640 10 monclient: _finish_hunting 0 2024-09-11T01:29:20.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15beada640 1 monclient: found mon.noname-b 2024-09-11T01:29:20.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15beada640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:20.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15beada640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-09-11T01:29:20.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f15beada640 1 -- 172.21.15.12:0/3535082076 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b000c340 con 0x7f15b00c0e10 2024-09-11T01:29:20.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ef02a640 10 monclient(hunting): picked mon.noname-b con 0x7f89e8138130 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:20.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ef02a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e813d280 0x7f89e813a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ef02a640 10 monclient(hunting): picked mon.noname-a con 0x7f89e813d280 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:20.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ef02a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ef02a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f89e806f850 con 0x7f89e813d280 2024-09-11T01:29:20.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ef02a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ef02a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f89e806f6d0 con 0x7f89e8138130 2024-09-11T01:29:20.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ef02a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ecdc6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f89e8137d50 0x7f89e8073240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ef02a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f89e813fbb0 con 0x7f89e8137d50 2024-09-11T01:29:20.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ef02a640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:20.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ef02a640 10 monclient(hunting): authenticate will time out at 766.347645s 2024-09-11T01:29:20.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ecdc6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f89e8137d50 0x7f89e8073240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51606/0 (socket says 172.21.15.12:51606) 2024-09-11T01:29:20.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ecdc6640 1 -- 172.21.15.12:0/1176071294 learned_addr learned my addr 172.21.15.12:0/1176071294 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:20.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ecdc6640 10 monclient(hunting): get_auth_request con 0x7f89e8137d50 auth_method 0 2024-09-11T01:29:20.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ecdc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ecdc6640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ecdc6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89e7fff640 1 --2- 172.21.15.12:0/1176071294 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89e8138130 0x7f89e813a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f049affd640 1 --2- 172.21.15.12:0/3333812169 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f048c03e260 0x7f048c040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89e7fff640 10 monclient(hunting): get_auth_request con 0x7f89e8138130 auth_method 0 2024-09-11T01:29:20.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f049affd640 1 -- 172.21.15.12:0/3333812169 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4402+0+0 (secure 0 0 0) 0x7f049c057170 con 0x7f04ac136cd0 2024-09-11T01:29:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ecdc6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ecdc6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ecdc6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ed5c7640 1 --2- 172.21.15.12:0/1176071294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e813d280 0x7f89e813a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ed5c7640 10 monclient(hunting): get_auth_request con 0x7f89e813d280 auth_method 0 2024-09-11T01:29:20.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ed5c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ed5c7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ed5c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f04a9d9b640 1 --2- 172.21.15.12:0/3333812169 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f048c03e260 0x7f048c040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ed5c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ed5c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f89ed5c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.906+0000 7f04a9d9b640 10 monclient: get_auth_request con 0x7f048c03e260 auth_method 0 2024-09-11T01:29:20.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.906+0000 7f89ecdc6640 10 monclient(hunting): handle_auth_done global_id 4271 payload 824 2024-09-11T01:29:20.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.906+0000 7f89ecdc6640 1 -- 172.21.15.12:0/1176071294 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89e8138130 msgr2=0x7f89e813a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.906+0000 7f89ecdc6640 1 --2- 172.21.15.12:0/1176071294 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89e8138130 0x7f89e813a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.906+0000 7f89ecdc6640 1 -- 172.21.15.12:0/1176071294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e813d280 msgr2=0x7f89e813a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.905+0000 7f049affd640 1 --2- 172.21.15.12:0/3333812169 >> [v2:172.21.15.12:6808/3563357280,v1:172.21.15.12:6809/3563357280] conn(0x7f048c0440d0 0x7f048c046510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.906+0000 7f049affd640 1 -- 172.21.15.12:0/3333812169 --> [v2:172.21.15.12:6808/3563357280,v1:172.21.15.12:6809/3563357280] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f048c046be0 con 0x7f048c0440d0 2024-09-11T01:29:20.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.906+0000 7f049affd640 1 -- 172.21.15.12:0/3333812169 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f049c058050 con 0x7f04ac136cd0 2024-09-11T01:29:20.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.906+0000 7f049affd640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-11T01:29:20.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.906+0000 7f04aad9d640 1 --2- 172.21.15.12:0/3333812169 >> [v2:172.21.15.12:6808/3563357280,v1:172.21.15.12:6809/3563357280] conn(0x7f048c0440d0 0x7f048c046510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.906+0000 7f04aad9d640 10 monclient: get_auth_request con 0x7f048c0440d0 auth_method 0 2024-09-11T01:29:20.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.906+0000 7f89ecdc6640 1 --2- 172.21.15.12:0/1176071294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e813d280 0x7f89e813a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.906+0000 7f89ecdc6640 10 monclient: _finish_hunting 0 2024-09-11T01:29:20.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.906+0000 7f89ecdc6640 1 monclient: found mon.noname-c 2024-09-11T01:29:20.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.906+0000 7f89ecdc6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:20.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.906+0000 7f89ecdc6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:29:20.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.906+0000 7f89ecdc6640 1 -- 172.21.15.12:0/1176071294 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e8070410 con 0x7f89e8137d50 2024-09-11T01:29:20.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.906+0000 7f89ed5c7640 1 --2- 172.21.15.12:0/1176071294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e813d280 0x7f89e813a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:20.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f7ac915e640 1 -- 172.21.15.12:0/1804404325 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ac4137d50 msgr2=0x7f7ac41ccd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:20.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f7ac915e640 1 --2- 172.21.15.12:0/1804404325 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ac4137d50 0x7f7ac41ccd40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7ac000e980 tx=0x7f7ac000ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f7ab9ffb640 1 -- 172.21.15.12:0/1804404325 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac0017070 con 0x7f7ac4137d50 2024-09-11T01:29:20.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f7ab9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f7ab9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f7ab9ffb640 10 monclient: dump: 2024-09-11T01:29:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.034 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f7ab9ffb640 10 monclient: _finish_auth 0 2024-09-11T01:29:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f7ab9ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:29:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f7ab9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:20.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f7ab9ffb640 1 -- 172.21.15.12:0/1804404325 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ac000cc10 con 0x7f7ac4137d50 2024-09-11T01:29:20.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f7ab9ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:20.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f7ab9ffb640 1 -- 172.21.15.12:0/1804404325 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac0013470 con 0x7f7ac4137d50 2024-09-11T01:29:20.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f7ab9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f7ab9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:20.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f7ab9ffb640 10 monclient: dump: 2024-09-11T01:29:20.035 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.035 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.035 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.035 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.035 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.035 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.035 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.036 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.036 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.036 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f7abb7fe640 4 set_mon_vals no callback set 2024-09-11T01:29:20.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f15beada640 1 -- 172.21.15.12:0/3535082076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15b00c0e10 msgr2=0x7f15b000ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:20.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f15beada640 1 --2- 172.21.15.12:0/3535082076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15b00c0e10 0x7f15b000ae60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f15b8000b60 tx=0x7f15b806a960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f15aeffd640 1 -- 172.21.15.12:0/3535082076 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b80bc070 con 0x7f15b00c0e10 2024-09-11T01:29:20.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f15aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f15aeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:20.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f15aeffd640 10 monclient: dump: 2024-09-11T01:29:20.037 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.037 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.037 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.038 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f15aeffd640 10 monclient: _finish_auth 0 2024-09-11T01:29:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f15aeffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f15aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f15aeffd640 1 -- 172.21.15.12:0/3535082076 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15b80b8610 con 0x7f15b00c0e10 2024-09-11T01:29:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f15bf2db640 5 monclient: authenticate success, global_id 4228 2024-09-11T01:29:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f15bf2db640 10 monclient: _renew_subs 2024-09-11T01:29:20.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f15bf2db640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.907+0000 7f15bf2db640 1 -- 172.21.15.12:0/3535082076 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15b001e0a0 con 0x7f15b00c0e10 2024-09-11T01:29:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.908+0000 7f7acb3c2640 5 monclient: authenticate success, global_id 4225 2024-09-11T01:29:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.908+0000 7f7acb3c2640 10 monclient: _renew_subs 2024-09-11T01:29:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.908+0000 7f15aeffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.908+0000 7f15aeffd640 1 -- 172.21.15.12:0/3535082076 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b80b4040 con 0x7f15b00c0e10 2024-09-11T01:29:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.908+0000 7f15aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.908+0000 7f15aeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.908+0000 7f7acb3c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.908+0000 7f15aeffd640 10 monclient: dump: 2024-09-11T01:29:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.040 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.041 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.041 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.041 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.041 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.908+0000 7f15bf2db640 10 monclient: _renew_subs 2024-09-11T01:29:20.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.908+0000 7f15bf2db640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:20.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.908+0000 7f15bf2db640 1 -- 172.21.15.12:0/3535082076 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15b00c0680 con 0x7f15b00c0e10 2024-09-11T01:29:20.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.908+0000 7f7acb3c2640 1 -- 172.21.15.12:0/1804404325 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ac40a9430 con 0x7f7ac4137d50 2024-09-11T01:29:20.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.908+0000 7f15bcad6640 4 set_mon_vals no callback set 2024-09-11T01:29:20.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.908+0000 7f7acb3c2640 10 monclient: _renew_subs 2024-09-11T01:29:20.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.908+0000 7f7acb3c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:20.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.908+0000 7f7acb3c2640 1 -- 172.21.15.12:0/1804404325 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ac4121780 con 0x7f7ac4137d50 2024-09-11T01:29:20.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.908+0000 7f89ecdc6640 1 -- 172.21.15.12:0/1176071294 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f89e8137d50 msgr2=0x7f89e8073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:20.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.908+0000 7f89ecdc6640 1 --2- 172.21.15.12:0/1176071294 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f89e8137d50 0x7f89e8073240 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f89dc00c470 tx=0x7f89dc00c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.908+0000 7f04a9d9b640 1 --2- 172.21.15.12:0/3333812169 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f048c03e260 0x7f048c040720 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f04a0009630 tx=0x7f04a0009240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.908+0000 7f15bf2db640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:20.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.908+0000 7f15bf2db640 1 -- 172.21.15.12:0/3535082076 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f15b0026ff0 con 0x7f15b00c0e10 2024-09-11T01:29:20.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f89e57fa640 1 -- 172.21.15.12:0/1176071294 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc017070 con 0x7f89e8137d50 2024-09-11T01:29:20.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f89e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f89e57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:20.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f89e57fa640 10 monclient: dump: 2024-09-11T01:29:20.044 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.044 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.044 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.044 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.044 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.045 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.045 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.045 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.045 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.045 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f7acb3c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:20.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f7acb3c2640 1 -- 172.21.15.12:0/1804404325 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7ac409d110 con 0x7f7ac4137d50 2024-09-11T01:29:20.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f89e57fa640 10 monclient: _finish_auth 0 2024-09-11T01:29:20.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f89e57fa640 10 monclient: _check_auth_tickets 2024-09-11T01:29:20.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f89e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:20.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f89e57fa640 1 -- 172.21.15.12:0/1176071294 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89dc0037f0 con 0x7f89e8137d50 2024-09-11T01:29:20.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f89ef02a640 5 monclient: authenticate success, global_id 4271 2024-09-11T01:29:20.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f89ef02a640 10 monclient: _renew_subs 2024-09-11T01:29:20.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f89ef02a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:20.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f89ef02a640 1 -- 172.21.15.12:0/1176071294 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89e80706f0 con 0x7f89e8137d50 2024-09-11T01:29:20.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f89ef02a640 10 monclient: _renew_subs 2024-09-11T01:29:20.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f89ef02a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:20.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f89ef02a640 1 -- 172.21.15.12:0/1176071294 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f89e8070130 con 0x7f89e8137d50 2024-09-11T01:29:20.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f89e57fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:20.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f7ab9ffb640 1 -- 172.21.15.12:0/1804404325 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f7ac0024060 con 0x7f7ac4137d50 2024-09-11T01:29:20.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f89e57fa640 1 -- 172.21.15.12:0/1176071294 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc003be0 con 0x7f89e8137d50 2024-09-11T01:29:20.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f89e6ffd640 4 set_mon_vals no callback set 2024-09-11T01:29:20.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f89e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f15aeffd640 1 -- 172.21.15.12:0/3535082076 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f15b80b7050 con 0x7f15b00c0e10 2024-09-11T01:29:20.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f89e57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:20.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f89e57fa640 10 monclient: dump: 2024-09-11T01:29:20.048 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.049 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.049 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.049 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.049 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.049 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.049 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.049 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.049 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.049 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f7ab9ffb640 1 --2- 172.21.15.12:0/1804404325 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7aa803e210 0x7f7aa80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.909+0000 7f7ab9ffb640 1 -- 172.21.15.12:0/1804404325 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4402+0+0 (secure 0 0 0) 0x7f7ac0052aa0 con 0x7f7ac4137d50 2024-09-11T01:29:20.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.910+0000 7f7ab9ffb640 1 --2- 172.21.15.12:0/1804404325 >> [v2:172.21.15.12:6816/4266003406,v1:172.21.15.12:6817/4266003406] conn(0x7f7aa8044080 0x7f7aa80464c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.910+0000 7f7ab9ffb640 1 -- 172.21.15.12:0/1804404325 --> [v2:172.21.15.12:6816/4266003406,v1:172.21.15.12:6817/4266003406] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7ac001de50 con 0x7f7aa8044080 2024-09-11T01:29:20.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.910+0000 7f7ab9ffb640 1 -- 172.21.15.12:0/1804404325 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f7ac00572a0 con 0x7f7ac4137d50 2024-09-11T01:29:20.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.910+0000 7f7ab9ffb640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-11T01:29:20.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.910+0000 7f15aeffd640 1 --2- 172.21.15.12:0/3535082076 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f159c03e260 0x7f159c040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.910+0000 7f7ac895d640 1 --2- 172.21.15.12:0/1804404325 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7aa803e210 0x7f7aa80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.910+0000 7f7ac895d640 10 monclient: get_auth_request con 0x7f7aa803e210 auth_method 0 2024-09-11T01:29:20.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.910+0000 7f15aeffd640 1 -- 172.21.15.12:0/3535082076 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4402+0+0 (secure 0 0 0) 0x7f15b80ef900 con 0x7f15b00c0e10 2024-09-11T01:29:20.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.910+0000 7f7ac995f640 1 --2- 172.21.15.12:0/1804404325 >> [v2:172.21.15.12:6816/4266003406,v1:172.21.15.12:6817/4266003406] conn(0x7f7aa8044080 0x7f7aa80464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.910+0000 7f7ac995f640 10 monclient: get_auth_request con 0x7f7aa8044080 auth_method 0 2024-09-11T01:29:20.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.910+0000 7f89ef02a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:20.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.910+0000 7f89ef02a640 1 -- 172.21.15.12:0/1176071294 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f89c8000f80 con 0x7f89e8137d50 2024-09-11T01:29:20.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.910+0000 7f15be2d9640 1 --2- 172.21.15.12:0/3535082076 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f159c03e260 0x7f159c040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.910+0000 7f15be2d9640 10 monclient: get_auth_request con 0x7f159c03e260 auth_method 0 2024-09-11T01:29:20.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.910+0000 7f15aeffd640 1 --2- 172.21.15.12:0/3535082076 >> [v2:172.21.15.12:6800/2893036867,v1:172.21.15.12:6801/2893036867] conn(0x7f159c044150 0x7f159c046590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.910+0000 7f15aeffd640 1 -- 172.21.15.12:0/3535082076 --> [v2:172.21.15.12:6800/2893036867,v1:172.21.15.12:6801/2893036867] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f159c046c60 con 0x7f159c044150 2024-09-11T01:29:20.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.910+0000 7f15aeffd640 1 -- 172.21.15.12:0/3535082076 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f15b80f42a0 con 0x7f15b00c0e10 2024-09-11T01:29:20.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.910+0000 7f15aeffd640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-11T01:29:20.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.910+0000 7f15bdad8640 1 --2- 172.21.15.12:0/3535082076 >> [v2:172.21.15.12:6800/2893036867,v1:172.21.15.12:6801/2893036867] conn(0x7f159c044150 0x7f159c046590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.910+0000 7f04aad9d640 1 --2- 172.21.15.12:0/3333812169 >> [v2:172.21.15.12:6808/3563357280,v1:172.21.15.12:6809/3563357280] conn(0x7f048c0440d0 0x7f048c046510 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.910+0000 7f15bdad8640 10 monclient: get_auth_request con 0x7f159c044150 auth_method 0 2024-09-11T01:29:20.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.910+0000 7f89e57fa640 1 -- 172.21.15.12:0/1176071294 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f89dc01b3b0 con 0x7f89e8137d50 2024-09-11T01:29:20.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.911+0000 7f89e57fa640 1 --2- 172.21.15.12:0/1176071294 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f89cc03e260 0x7f89cc040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.911+0000 7f89e57fa640 1 -- 172.21.15.12:0/1176071294 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4402+0+0 (secure 0 0 0) 0x7f89dc052240 con 0x7f89e8137d50 2024-09-11T01:29:20.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.911+0000 7f89e57fa640 1 --2- 172.21.15.12:0/1176071294 >> [v2:172.21.15.71:6800/1790415427,v1:172.21.15.71:6801/1790415427] conn(0x7f89cc044150 0x7f89cc046590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.911+0000 7f89e57fa640 1 -- 172.21.15.12:0/1176071294 --> [v2:172.21.15.71:6800/1790415427,v1:172.21.15.71:6801/1790415427] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f89cc046c60 con 0x7f89cc044150 2024-09-11T01:29:20.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.911+0000 7f89e57fa640 1 -- 172.21.15.12:0/1176071294 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f89dc059050 con 0x7f89e8137d50 2024-09-11T01:29:20.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.911+0000 7f89e57fa640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-11T01:29:20.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.911+0000 7f89ed5c7640 1 --2- 172.21.15.12:0/1176071294 >> [v2:172.21.15.71:6800/1790415427,v1:172.21.15.71:6801/1790415427] conn(0x7f89cc044150 0x7f89cc046590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.911+0000 7f89ed5c7640 10 monclient: get_auth_request con 0x7f89cc044150 auth_method 0 2024-09-11T01:29:20.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.911+0000 7f049affd640 1 -- 172.21.15.12:0/3333812169 <== osd.1 v2:172.21.15.12:6808/3563357280 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f048c046be0 con 0x7f048c0440d0 2024-09-11T01:29:20.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.913+0000 7f15be2d9640 1 --2- 172.21.15.12:0/3535082076 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f159c03e260 0x7f159c040720 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f15b4007c90 tx=0x7f15b4007be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.914+0000 7f7ac995f640 1 --2- 172.21.15.12:0/1804404325 >> [v2:172.21.15.12:6816/4266003406,v1:172.21.15.12:6817/4266003406] conn(0x7f7aa8044080 0x7f7aa80464c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.915+0000 7f15bdad8640 1 --2- 172.21.15.12:0/3535082076 >> [v2:172.21.15.12:6800/2893036867,v1:172.21.15.12:6801/2893036867] conn(0x7f159c044150 0x7f159c046590 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.915+0000 7f7ab9ffb640 1 -- 172.21.15.12:0/1804404325 <== osd.0 v2:172.21.15.12:6816/4266003406 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7ac001de50 con 0x7f7aa8044080 2024-09-11T01:29:20.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.915+0000 7f15aeffd640 1 -- 172.21.15.12:0/3535082076 <== osd.2 v2:172.21.15.12:6800/2893036867 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f159c046c60 con 0x7f159c044150 2024-09-11T01:29:20.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.916+0000 7f89ed5c7640 1 --2- 172.21.15.12:0/1176071294 >> [v2:172.21.15.71:6800/1790415427,v1:172.21.15.71:6801/1790415427] conn(0x7f89cc044150 0x7f89cc046590 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.916+0000 7f89e57fa640 1 -- 172.21.15.12:0/1176071294 <== osd.4 v2:172.21.15.71:6800/1790415427 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89cc046c60 con 0x7f89cc044150 2024-09-11T01:29:20.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.926+0000 7f0498ff9640 1 -- 172.21.15.12:0/3333812169 --> [v2:172.21.15.12:6808/3563357280,v1:172.21.15.12:6809/3563357280] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f04ac136ac0 con 0x7f048c0440d0 2024-09-11T01:29:20.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.926+0000 7f049affd640 1 -- 172.21.15.12:0/3333812169 <== osd.1 v2:172.21.15.12:6808/3563357280 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f04a4041040 con 0x7f048c0440d0 2024-09-11T01:29:20.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.926+0000 7f89e7fff640 1 --2- 172.21.15.12:0/1176071294 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f89cc03e260 0x7f89cc040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.926+0000 7f89e7fff640 10 monclient: get_auth_request con 0x7f89cc03e260 auth_method 0 2024-09-11T01:29:20.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.926+0000 7f0498ff9640 1 -- 172.21.15.12:0/3333812169 >> [v2:172.21.15.12:6808/3563357280,v1:172.21.15.12:6809/3563357280] conn(0x7f048c0440d0 msgr2=0x7f048c046510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.926+0000 7f0498ff9640 1 --2- 172.21.15.12:0/3333812169 >> [v2:172.21.15.12:6808/3563357280,v1:172.21.15.12:6809/3563357280] conn(0x7f048c0440d0 0x7f048c046510 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.927+0000 7f0498ff9640 1 -- 172.21.15.12:0/3333812169 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f048c03e260 msgr2=0x7f048c040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.927+0000 7f0498ff9640 1 --2- 172.21.15.12:0/3333812169 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f048c03e260 0x7f048c040720 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f04a0009630 tx=0x7f04a0009240 comp rx=0 tx=0).stop 2024-09-11T01:29:20.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.927+0000 7f0498ff9640 10 monclient: shutdown 2024-09-11T01:29:20.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.927+0000 7f0498ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:20.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.927+0000 7f0498ff9640 1 -- 172.21.15.12:0/3333812169 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f04ac136cd0 msgr2=0x7f04ac1214e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.927+0000 7f0498ff9640 1 --2- 172.21.15.12:0/3333812169 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f04ac136cd0 0x7f04ac1214e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f049c00c7f0 tx=0x7f049c00ccb0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.927+0000 7f04aad9d640 1 -- 172.21.15.12:0/3333812169 reap_dead start 2024-09-11T01:29:20.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.927+0000 7f0498ff9640 1 -- 172.21.15.12:0/3333812169 shutdown_connections 2024-09-11T01:29:20.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.927+0000 7f0498ff9640 1 -- 172.21.15.12:0/3333812169 >> 172.21.15.12:0/3333812169 conn(0x7f04ac0a7020 msgr2=0x7f04ac0a0050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:20.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.927+0000 7f0498ff9640 1 -- 172.21.15.12:0/3333812169 shutdown_connections 2024-09-11T01:29:20.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.927+0000 7f0498ff9640 1 -- 172.21.15.12:0/3333812169 wait complete. 2024-09-11T01:29:20.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.929+0000 7f89e7fff640 1 --2- 172.21.15.12:0/1176071294 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f89cc03e260 0x7f89cc040720 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f89d4034000 tx=0x7f89d402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.928+0000 7f7ac895d640 1 --2- 172.21.15.12:0/1804404325 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7aa803e210 0x7f7aa80406d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4007c90 tx=0x7f7ab4007be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.930+0000 7f7acb3c2640 1 -- 172.21.15.12:0/1804404325 --> [v2:172.21.15.12:6816/4266003406,v1:172.21.15.12:6817/4266003406] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7ac4121400 con 0x7f7aa8044080 2024-09-11T01:29:20.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.930+0000 7f7ab9ffb640 1 -- 172.21.15.12:0/1804404325 <== osd.0 v2:172.21.15.12:6816/4266003406 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7ac4121400 con 0x7f7aa8044080 2024-09-11T01:29:20.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.930+0000 7f15bf2db640 1 -- 172.21.15.12:0/3535082076 --> [v2:172.21.15.12:6800/2893036867,v1:172.21.15.12:6801/2893036867] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f15b0020ca0 con 0x7f159c044150 2024-09-11T01:29:20.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.930+0000 7f7acb3c2640 1 -- 172.21.15.12:0/1804404325 >> [v2:172.21.15.12:6816/4266003406,v1:172.21.15.12:6817/4266003406] conn(0x7f7aa8044080 msgr2=0x7f7aa80464c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.930+0000 7f7acb3c2640 1 --2- 172.21.15.12:0/1804404325 >> [v2:172.21.15.12:6816/4266003406,v1:172.21.15.12:6817/4266003406] conn(0x7f7aa8044080 0x7f7aa80464c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f15aeffd640 1 -- 172.21.15.12:0/3535082076 <== osd.2 v2:172.21.15.12:6800/2893036867 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f15b0020ca0 con 0x7f159c044150 2024-09-11T01:29:20.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f7acb3c2640 1 -- 172.21.15.12:0/1804404325 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7aa803e210 msgr2=0x7f7aa80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f7acb3c2640 1 --2- 172.21.15.12:0/1804404325 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7aa803e210 0x7f7aa80406d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4007c90 tx=0x7f7ab4007be0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f7acb3c2640 10 monclient: shutdown 2024-09-11T01:29:20.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f7acb3c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:20.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f7acb3c2640 1 -- 172.21.15.12:0/1804404325 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ac4137d50 msgr2=0x7f7ac41ccd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f7acb3c2640 1 --2- 172.21.15.12:0/1804404325 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ac4137d50 0x7f7ac41ccd40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7ac000e980 tx=0x7f7ac000ee40 comp rx=0 tx=0).stop 2024-09-11T01:29:20.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f7ac995f640 1 -- 172.21.15.12:0/1804404325 reap_dead start 2024-09-11T01:29:20.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f89ef02a640 1 -- 172.21.15.12:0/1176071294 --> [v2:172.21.15.71:6800/1790415427,v1:172.21.15.71:6801/1790415427] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f89c8002c70 con 0x7f89cc044150 2024-09-11T01:29:20.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f7acb3c2640 1 -- 172.21.15.12:0/1804404325 shutdown_connections 2024-09-11T01:29:20.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f7acb3c2640 1 -- 172.21.15.12:0/1804404325 >> 172.21.15.12:0/1804404325 conn(0x7f7ac40a7020 msgr2=0x7f7ac413e880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:20.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f7acb3c2640 1 -- 172.21.15.12:0/1804404325 shutdown_connections 2024-09-11T01:29:20.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f15bf2db640 1 -- 172.21.15.12:0/3535082076 >> [v2:172.21.15.12:6800/2893036867,v1:172.21.15.12:6801/2893036867] conn(0x7f159c044150 msgr2=0x7f159c046590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f15bf2db640 1 --2- 172.21.15.12:0/3535082076 >> [v2:172.21.15.12:6800/2893036867,v1:172.21.15.12:6801/2893036867] conn(0x7f159c044150 0x7f159c046590 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f7acb3c2640 1 -- 172.21.15.12:0/1804404325 wait complete. 2024-09-11T01:29:20.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f15bf2db640 1 -- 172.21.15.12:0/3535082076 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f159c03e260 msgr2=0x7f159c040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f15bf2db640 1 --2- 172.21.15.12:0/3535082076 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f159c03e260 0x7f159c040720 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f15b4007c90 tx=0x7f15b4007be0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f15bf2db640 10 monclient: shutdown 2024-09-11T01:29:20.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f15bf2db640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:20.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f15bf2db640 1 -- 172.21.15.12:0/3535082076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15b00c0e10 msgr2=0x7f15b000ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f15bf2db640 1 --2- 172.21.15.12:0/3535082076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15b00c0e10 0x7f15b000ae60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f15b8000b60 tx=0x7f15b806a960 comp rx=0 tx=0).stop 2024-09-11T01:29:20.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f89e57fa640 1 -- 172.21.15.12:0/1176071294 <== osd.4 v2:172.21.15.71:6800/1790415427 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f89c8002c70 con 0x7f89cc044150 2024-09-11T01:29:20.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f15beada640 1 -- 172.21.15.12:0/3535082076 reap_dead start 2024-09-11T01:29:20.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f15bf2db640 1 -- 172.21.15.12:0/3535082076 shutdown_connections 2024-09-11T01:29:20.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f15bf2db640 1 -- 172.21.15.12:0/3535082076 >> 172.21.15.12:0/3535082076 conn(0x7f15b0006390 msgr2=0x7f15b0025af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:20.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f15bf2db640 1 -- 172.21.15.12:0/3535082076 shutdown_connections 2024-09-11T01:29:20.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.931+0000 7f15bf2db640 1 -- 172.21.15.12:0/3535082076 wait complete. 2024-09-11T01:29:20.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.932+0000 7f89ef02a640 1 -- 172.21.15.12:0/1176071294 >> [v2:172.21.15.71:6800/1790415427,v1:172.21.15.71:6801/1790415427] conn(0x7f89cc044150 msgr2=0x7f89cc046590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.932+0000 7f89ef02a640 1 --2- 172.21.15.12:0/1176071294 >> [v2:172.21.15.71:6800/1790415427,v1:172.21.15.71:6801/1790415427] conn(0x7f89cc044150 0x7f89cc046590 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.932+0000 7f89ef02a640 1 -- 172.21.15.12:0/1176071294 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f89cc03e260 msgr2=0x7f89cc040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.932+0000 7f89ef02a640 1 --2- 172.21.15.12:0/1176071294 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f89cc03e260 0x7f89cc040720 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f89d4034000 tx=0x7f89d402e000 comp rx=0 tx=0).stop 2024-09-11T01:29:20.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.932+0000 7f89ef02a640 10 monclient: shutdown 2024-09-11T01:29:20.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.932+0000 7f89ef02a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:20.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.932+0000 7f89ef02a640 1 -- 172.21.15.12:0/1176071294 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f89e8137d50 msgr2=0x7f89e8073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.932+0000 7f89ef02a640 1 --2- 172.21.15.12:0/1176071294 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f89e8137d50 0x7f89e8073240 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f89dc00c470 tx=0x7f89dc00c930 comp rx=0 tx=0).stop 2024-09-11T01:29:20.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.932+0000 7f89ed5c7640 1 -- 172.21.15.12:0/1176071294 reap_dead start 2024-09-11T01:29:20.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.932+0000 7f89ef02a640 1 -- 172.21.15.12:0/1176071294 shutdown_connections 2024-09-11T01:29:20.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.932+0000 7f89ef02a640 1 -- 172.21.15.12:0/1176071294 >> 172.21.15.12:0/1176071294 conn(0x7f89e80a7020 msgr2=0x7f89e813f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:20.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.932+0000 7f89ef02a640 1 -- 172.21.15.12:0/1176071294 shutdown_connections 2024-09-11T01:29:20.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:19.932+0000 7f89ef02a640 1 -- 172.21.15.12:0/1176071294 wait complete. 2024-09-11T01:29:20.069 INFO:teuthology.orchestra.run.smithi012.stdout:51539607555 2024-09-11T01:29:20.069 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T01:29:20.070 INFO:teuthology.orchestra.run.smithi012.stdout:47244640260 2024-09-11T01:29:20.070 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T01:29:20.071 INFO:teuthology.orchestra.run.smithi012.stdout:51539607556 2024-09-11T01:29:20.071 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-11T01:29:20.072 INFO:teuthology.orchestra.run.smithi012.stdout:51539607556 2024-09-11T01:29:20.073 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T01:29:20.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.225+0000 7f3635791640 10 monclient: get_monmap_and_config 2024-09-11T01:29:20.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.225+0000 7fe8405ec640 10 monclient: get_monmap_and_config 2024-09-11T01:29:20.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.225+0000 7fe8405ec640 10 monclient: build_initial_monmap 2024-09-11T01:29:20.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.226+0000 7fe8405ec640 10 monclient: monmap: 2024-09-11T01:29:20.228 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:20.228 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.228 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:20.226713+0000 2024-09-11T01:29:20.228 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:20.226713+0000 2024-09-11T01:29:20.228 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:20.228 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.228 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:20.228 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:20.229 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:20.229 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:20.229 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:20.229 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.225+0000 7f3635791640 10 monclient: build_initial_monmap 2024-09-11T01:29:20.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.225+0000 7f3635791640 10 monclient: monmap: 2024-09-11T01:29:20.229 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:20.229 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.229 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:20.226668+0000 2024-09-11T01:29:20.230 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:20.226668+0000 2024-09-11T01:29:20.230 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:20.230 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.230 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:20.230 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:20.230 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:20.230 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:20.230 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:20.230 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.226+0000 7fe8405ec640 1 Processor -- start 2024-09-11T01:29:20.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.226+0000 7f3635791640 1 Processor -- start 2024-09-11T01:29:20.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.226+0000 7fe8405ec640 1 -- start start 2024-09-11T01:29:20.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.226+0000 7f3635791640 1 -- start start 2024-09-11T01:29:20.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.226+0000 7f3635791640 10 monclient: init 2024-09-11T01:29:20.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.226+0000 7fe8405ec640 10 monclient: init 2024-09-11T01:29:20.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7f3635791640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:20.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7f3635791640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-11T01:29:20.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7f3635791640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3630137d50 0x7f363009d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7fe8405ec640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:20.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7fe8405ec640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-11T01:29:20.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7f3635791640 10 monclient(hunting): picked mon.noname-c con 0x7f3630137d50 addr v2:172.21.15.71:3300/0 2024-09-11T01:29:20.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7f3635791640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3630138130 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:20.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7f3635791640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f363013d280 0x7f363013f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7f3635791640 10 monclient(hunting): picked mon.noname-a con 0x7f363013d280 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:20.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7f3635791640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f363009ba20 con 0x7f3630138130 2024-09-11T01:29:20.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7f3635791640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7f3635791640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f363009c650 con 0x7f363013d280 2024-09-11T01:29:20.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7f3635791640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7f3635791640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f363009c370 con 0x7f3630137d50 2024-09-11T01:29:20.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7fe8405ec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe838137d50 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:20.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7f3635791640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:20.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7f3635791640 10 monclient(hunting): authenticate will time out at 766.669714s 2024-09-11T01:29:20.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7f362f7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f363013d280 0x7f363013f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7fe8405ec640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe838138130 0x7fe838138550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7fe8405ec640 10 monclient(hunting): picked mon.noname-b con 0x7fe838138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:20.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7fe8405ec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe83813d280 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:20.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7fe8405ec640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe83809d820 con 0x7fe83813d280 2024-09-11T01:29:20.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7fe8405ec640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe83809ba20 con 0x7fe838137d50 2024-09-11T01:29:20.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7fe8405ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7fe8405ec640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe83809c650 con 0x7fe838138130 2024-09-11T01:29:20.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7fe8405ec640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:20.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7fe8405ec640 10 monclient(hunting): authenticate will time out at 766.669658s 2024-09-11T01:29:20.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.227+0000 7fe83db87640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe838138130 0x7fe838138550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7fe83eb89640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe83813d280 0x7fe83813f670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56038/0 (socket says 172.21.15.12:56038) 2024-09-11T01:29:20.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7fe83eb89640 1 -- 172.21.15.12:0/3468861349 learned_addr learned my addr 172.21.15.12:0/3468861349 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:29:20.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7fe83db87640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe838138130 0x7fe838138550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51614/0 (socket says 172.21.15.12:51614) 2024-09-11T01:29:20.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7fe83db87640 10 monclient(hunting): get_auth_request con 0x7fe838138130 auth_method 0 2024-09-11T01:29:20.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7fe83db87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7fe83db87640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7fe83db87640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7f362f7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f363013d280 0x7f363013f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54916/0 (socket says 172.21.15.12:54916) 2024-09-11T01:29:20.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7f362f7fe640 1 -- 172.21.15.12:0/2902379756 learned_addr learned my addr 172.21.15.12:0/2902379756 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:20.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7f362effd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3630137d50 0x7f363009d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7f362f7fe640 10 monclient(hunting): get_auth_request con 0x7f363013d280 auth_method 0 2024-09-11T01:29:20.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7f362f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7fe83db87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7f362f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7f362f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7f362effd640 10 monclient(hunting): get_auth_request con 0x7f3630137d50 auth_method 0 2024-09-11T01:29:20.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7f362effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7f362effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7f362effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7f362f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7f362f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7f362f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7f362effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7f362effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7f362effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7fe83db87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7f362f7fe640 10 monclient(hunting): handle_auth_done global_id 4260 payload 274 2024-09-11T01:29:20.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7fe83db87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7f362f7fe640 1 -- 172.21.15.12:0/2902379756 >> v2:172.21.15.71:3300/0 conn(0x7f3630137d50 msgr2=0x7f363009d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:20.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7f362f7fe640 1 --2- 172.21.15.12:0/2902379756 >> v2:172.21.15.71:3300/0 conn(0x7f3630137d50 0x7f363009d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7fe83db87640 10 monclient(hunting): handle_auth_done global_id 4274 payload 274 2024-09-11T01:29:20.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7fe83db87640 1 -- 172.21.15.12:0/3468861349 >> v1:172.21.15.71:6789/0 conn(0x7fe838137d50 legacy=0x7fe83809d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:20.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7fe83db87640 1 -- 172.21.15.12:0/3468861349 >> v1:172.21.15.12:6789/0 conn(0x7fe83813d280 legacy=0x7fe83813f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:20.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7fe83db87640 10 monclient: _finish_hunting 0 2024-09-11T01:29:20.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7fe83db87640 1 monclient: found mon.noname-b 2024-09-11T01:29:20.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7fe83db87640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:20.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7fe83db87640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:29:20.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7fe83db87640 1 -- 172.21.15.12:0/3468861349 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe83809c370 con 0x7fe838138130 2024-09-11T01:29:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7f362f7fe640 1 -- 172.21.15.12:0/2902379756 >> v1:172.21.15.71:6789/0 conn(0x7f3630138130 legacy=0x7f3630138580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7f362f7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:29:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7f362f7fe640 1 monclient: found mon.noname-a 2024-09-11T01:29:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7f362f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7f362f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.228+0000 7f362f7fe640 1 -- 172.21.15.12:0/2902379756 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36300a6ce0 con 0x7f363013d280 2024-09-11T01:29:20.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7fe83db87640 1 -- 172.21.15.12:0/3468861349 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe838138130 msgr2=0x7fe838138550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:20.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7f362f7fe640 1 -- 172.21.15.12:0/2902379756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f363013d280 msgr2=0x7f363013f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:20.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7f362f7fe640 1 --2- 172.21.15.12:0/2902379756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f363013d280 0x7f363013f670 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f363009c7d0 tx=0x7f362802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23bf7bc9d1b28cb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7fe83db87640 1 --2- 172.21.15.12:0/3468861349 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe838138130 0x7fe838138550 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe834009a30 tx=0x7fe83402a2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86f7305581320f33 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7fe83d386640 1 -- 172.21.15.12:0/3468861349 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe834037070 con 0x7fe838138130 2024-09-11T01:29:20.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7f362dffb640 1 -- 172.21.15.12:0/2902379756 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3628036070 con 0x7f363013d280 2024-09-11T01:29:20.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7f362dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7f362dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:20.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7f362dffb640 10 monclient: dump: 2024-09-11T01:29:20.244 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.245 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.245 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.245 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.245 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.245 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.245 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.245 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.245 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.245 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7f362dffb640 10 monclient: _finish_auth 0 2024-09-11T01:29:20.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7f362dffb640 10 monclient: _check_auth_tickets 2024-09-11T01:29:20.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7f362dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:20.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7f362dffb640 1 -- 172.21.15.12:0/2902379756 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3628007800 con 0x7f363013d280 2024-09-11T01:29:20.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7f3635791640 5 monclient: authenticate success, global_id 4260 2024-09-11T01:29:20.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7f362dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:20.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7f362dffb640 1 -- 172.21.15.12:0/2902379756 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3628007bd0 con 0x7f363013d280 2024-09-11T01:29:20.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7f362dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7f362dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:20.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7f362dffb640 10 monclient: dump: 2024-09-11T01:29:20.246 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.246 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.246 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.246 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.246 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.246 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.246 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.246 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.246 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.246 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7f3635791640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:20.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7f3635791640 4 set_mon_vals no callback set 2024-09-11T01:29:20.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7fe83d386640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7fe83d386640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:20.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7fe83d386640 10 monclient: dump: 2024-09-11T01:29:20.248 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.248 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.248 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.248 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.248 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.248 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.248 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.248 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.249 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.249 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7fe83d386640 10 monclient: _finish_auth 0 2024-09-11T01:29:20.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7fe83d386640 10 monclient: _check_auth_tickets 2024-09-11T01:29:20.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.231+0000 7fe83d386640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:20.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.232+0000 7fe8405ec640 5 monclient: authenticate success, global_id 4274 2024-09-11T01:29:20.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.232+0000 7fe8405ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:20.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.232+0000 7fe83d386640 1 -- 172.21.15.12:0/3468861349 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe834032640 con 0x7fe838138130 2024-09-11T01:29:20.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.232+0000 7fe83d386640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:20.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.232+0000 7fe83d386640 1 -- 172.21.15.12:0/3468861349 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe834032a10 con 0x7fe838138130 2024-09-11T01:29:20.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.232+0000 7fe83d386640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.232+0000 7fe83d386640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:20.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.232+0000 7fe83d386640 10 monclient: dump: 2024-09-11T01:29:20.249 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.250 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.250 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.250 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.250 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.250 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.250 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.250 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.250 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.250 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.232+0000 7fe8405ec640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:20.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.232+0000 7fe8405ec640 4 set_mon_vals no callback set 2024-09-11T01:29:20.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.232+0000 7f3635791640 10 monclient: shutdown 2024-09-11T01:29:20.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.232+0000 7f3635791640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:20.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.232+0000 7f3635791640 1 -- 172.21.15.12:0/2902379756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f363013d280 msgr2=0x7f363013f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.232+0000 7f3635791640 1 --2- 172.21.15.12:0/2902379756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f363013d280 0x7f363013f670 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f363009c7d0 tx=0x7f362802a680 comp rx=0 tx=0).stop 2024-09-11T01:29:20.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.233+0000 7fe8405ec640 10 monclient: shutdown 2024-09-11T01:29:20.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.233+0000 7fe8405ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:20.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.233+0000 7fe8405ec640 1 -- 172.21.15.12:0/3468861349 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe838138130 msgr2=0x7fe838138550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.233+0000 7f3635791640 1 -- 172.21.15.12:0/2902379756 shutdown_connections 2024-09-11T01:29:20.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.233+0000 7fe8405ec640 1 --2- 172.21.15.12:0/3468861349 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe838138130 0x7fe838138550 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe834009a30 tx=0x7fe83402a2e0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.233+0000 7f3635791640 1 --2- 172.21.15.12:0/2902379756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f363013d280 0x7f363013f670 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.233+0000 7f3635791640 1 --2- 172.21.15.12:0/2902379756 >> v2:172.21.15.71:3300/0 conn(0x7f3630137d50 0x7f363009d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.233+0000 7f3635791640 1 -- 172.21.15.12:0/2902379756 >> 172.21.15.12:0/2902379756 conn(0x7f36300a7020 msgr2=0x7f36300a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:20.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.233+0000 7fe8405ec640 1 -- 172.21.15.12:0/3468861349 shutdown_connections 2024-09-11T01:29:20.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.233+0000 7fe8405ec640 1 --2- 172.21.15.12:0/3468861349 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe838138130 0x7fe838138550 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.233+0000 7fe8405ec640 1 -- 172.21.15.12:0/3468861349 >> 172.21.15.12:0/3468861349 conn(0x7fe8380a7020 msgr2=0x7fe8380a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:20.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.233+0000 7f3635791640 1 -- 172.21.15.12:0/2902379756 shutdown_connections 2024-09-11T01:29:20.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.233+0000 7f3635791640 1 -- 172.21.15.12:0/2902379756 wait complete. 2024-09-11T01:29:20.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.233+0000 7fe8405ec640 1 -- 172.21.15.12:0/3468861349 shutdown_connections 2024-09-11T01:29:20.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.233+0000 7fe8405ec640 1 -- 172.21.15.12:0/3468861349 wait complete. 2024-09-11T01:29:20.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.234+0000 7f3635791640 10 monclient: build_initial_monmap 2024-09-11T01:29:20.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.234+0000 7f3635791640 10 monclient: monmap: 2024-09-11T01:29:20.255 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:20.255 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:20.255 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:20.255 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:20.255 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:20.255 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.255 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:20.255 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:20.255 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:20.255 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.234+0000 7fe8405ec640 10 monclient: build_initial_monmap 2024-09-11T01:29:20.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.234+0000 7fe8405ec640 10 monclient: monmap: 2024-09-11T01:29:20.256 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:20.256 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:20.256 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:20.256 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:20.256 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:20.256 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.256 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:20.256 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:20.256 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:20.256 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.234+0000 7fe8405ec640 1 Processor -- start 2024-09-11T01:29:20.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.234+0000 7fe8405ec640 1 -- start start 2024-09-11T01:29:20.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.234+0000 7fe8405ec640 10 monclient: init 2024-09-11T01:29:20.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.234+0000 7f3635791640 1 Processor -- start 2024-09-11T01:29:20.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.234+0000 7f3635791640 1 -- start start 2024-09-11T01:29:20.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.234+0000 7f3635791640 10 monclient: init 2024-09-11T01:29:20.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe8405ec640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:20.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe8405ec640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:29:20.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe8405ec640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe838137d50 0x7fe838136b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe8405ec640 10 monclient(hunting): picked mon.noname-a con 0x7fe838137d50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:20.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe8405ec640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe838138130 0x7fe8381370b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe8405ec640 10 monclient(hunting): picked mon.noname-b con 0x7fe838138130 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:20.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe8405ec640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe83813d280 0x7fe838139260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe8405ec640 10 monclient(hunting): picked mon.noname-c con 0x7fe83813d280 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:20.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe8405ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe8405ec640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe83806f850 con 0x7fe838137d50 2024-09-11T01:29:20.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe8405ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe8405ec640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe83806f6d0 con 0x7fe838138130 2024-09-11T01:29:20.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7f3635791640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:20.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7f3635791640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:29:20.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe8405ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe8405ec640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe838140d90 con 0x7fe83813d280 2024-09-11T01:29:20.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe8405ec640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:20.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe8405ec640 10 monclient(hunting): authenticate will time out at 766.677159s 2024-09-11T01:29:20.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe83e388640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe838137d50 0x7fe838136b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe83e388640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe838137d50 0x7fe838136b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54932/0 (socket says 172.21.15.12:54932) 2024-09-11T01:29:20.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe83e388640 1 -- 172.21.15.12:0/4138121050 learned_addr learned my addr 172.21.15.12:0/4138121050 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:20.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe83e388640 10 monclient(hunting): get_auth_request con 0x7fe838137d50 auth_method 0 2024-09-11T01:29:20.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe83e388640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe83e388640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe83e388640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7f3635791640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3630137d50 0x7f36301cce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7f3635791640 10 monclient(hunting): picked mon.noname-b con 0x7f3630137d50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:20.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe83eb89640 1 --2- 172.21.15.12:0/4138121050 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe83813d280 0x7fe838139260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe83eb89640 10 monclient(hunting): get_auth_request con 0x7fe83813d280 auth_method 0 2024-09-11T01:29:20.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe83eb89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe83eb89640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe83eb89640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe83e388640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe83e388640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe83e388640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7f3635791640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3630138130 0x7f3630136e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe83db87640 1 --2- 172.21.15.12:0/4138121050 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe838138130 0x7fe8381370b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe83db87640 10 monclient(hunting): get_auth_request con 0x7fe838138130 auth_method 0 2024-09-11T01:29:20.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe83db87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe83db87640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7fe83db87640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7f3635791640 10 monclient(hunting): picked mon.noname-c con 0x7f3630138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:20.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7f3635791640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f363013d280 0x7f363013a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7f3635791640 10 monclient(hunting): picked mon.noname-a con 0x7f363013d280 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:20.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7f3635791640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7f3635791640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f363009d7d0 con 0x7f363013d280 2024-09-11T01:29:20.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7fe83eb89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7f3635791640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7f3635791640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f363009d650 con 0x7f3630137d50 2024-09-11T01:29:20.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7f3635791640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7f3635791640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f363013fbb0 con 0x7f3630138130 2024-09-11T01:29:20.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7f3635791640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:20.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7f3635791640 10 monclient(hunting): authenticate will time out at 766.677637s 2024-09-11T01:29:20.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7f362e7fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3630138130 0x7f3630136e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7f362e7fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3630138130 0x7f3630136e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51636/0 (socket says 172.21.15.12:51636) 2024-09-11T01:29:20.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7f362e7fc640 1 -- 172.21.15.12:0/758197832 learned_addr learned my addr 172.21.15.12:0/758197832 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:20.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7f362e7fc640 10 monclient(hunting): get_auth_request con 0x7f3630138130 auth_method 0 2024-09-11T01:29:20.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7f362e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7f362e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7f362e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.235+0000 7f362effd640 1 --2- 172.21.15.12:0/758197832 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3630137d50 0x7f36301cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362effd640 10 monclient(hunting): get_auth_request con 0x7f3630137d50 auth_method 0 2024-09-11T01:29:20.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7fe83eb89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362f7fe640 1 --2- 172.21.15.12:0/758197832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f363013d280 0x7f363013a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362f7fe640 10 monclient(hunting): get_auth_request con 0x7f363013d280 auth_method 0 2024-09-11T01:29:20.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7fe83eb89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7fe83db87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7fe83db87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7fe83db87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7fe83e388640 10 monclient(hunting): handle_auth_done global_id 4263 payload 824 2024-09-11T01:29:20.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7fe83e388640 1 -- 172.21.15.12:0/4138121050 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe83813d280 msgr2=0x7fe838139260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7fe83e388640 1 --2- 172.21.15.12:0/4138121050 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe83813d280 0x7fe838139260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362f7fe640 10 monclient(hunting): handle_auth_done global_id 4266 payload 824 2024-09-11T01:29:20.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362f7fe640 1 -- 172.21.15.12:0/758197832 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3630138130 msgr2=0x7f3630136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362f7fe640 1 --2- 172.21.15.12:0/758197832 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3630138130 0x7f3630136e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362f7fe640 1 -- 172.21.15.12:0/758197832 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3630137d50 msgr2=0x7f36301cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362f7fe640 1 --2- 172.21.15.12:0/758197832 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3630137d50 0x7f36301cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362f7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:29:20.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362f7fe640 1 monclient: found mon.noname-a 2024-09-11T01:29:20.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:20.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:20.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7f362f7fe640 1 -- 172.21.15.12:0/758197832 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36300a9430 con 0x7f363013d280 2024-09-11T01:29:20.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7fe83e388640 1 -- 172.21.15.12:0/4138121050 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe838138130 msgr2=0x7fe8381370b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7fe83e388640 1 --2- 172.21.15.12:0/4138121050 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe838138130 0x7fe8381370b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7fe83e388640 10 monclient: _finish_hunting 0 2024-09-11T01:29:20.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7fe83e388640 1 monclient: found mon.noname-a 2024-09-11T01:29:20.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7fe83e388640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:20.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7fe83e388640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:20.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.236+0000 7fe83e388640 1 -- 172.21.15.12:0/4138121050 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe83813c510 con 0x7fe838137d50 2024-09-11T01:29:20.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.238+0000 7fe83e388640 1 -- 172.21.15.12:0/4138121050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe838137d50 msgr2=0x7fe838136b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:20.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.238+0000 7fe83e388640 1 --2- 172.21.15.12:0/4138121050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe838137d50 0x7fe838136b70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe82c00c460 tx=0x7fe82c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7fe82affd640 1 -- 172.21.15.12:0/4138121050 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82c016020 con 0x7fe838137d50 2024-09-11T01:29:20.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7fe82affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7fe82affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:20.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7fe82affd640 10 monclient: dump: 2024-09-11T01:29:20.273 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.273 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.273 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.273 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.273 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.273 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.273 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.274 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.274 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.274 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7fe82affd640 10 monclient: _finish_auth 0 2024-09-11T01:29:20.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7fe82affd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:20.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7f362f7fe640 1 -- 172.21.15.12:0/758197832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f363013d280 msgr2=0x7f363013a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:20.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7f362f7fe640 1 --2- 172.21.15.12:0/758197832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f363013d280 0x7f363013a200 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3628033000 tx=0x7f3628004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7fe82affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:20.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7fe8405ec640 5 monclient: authenticate success, global_id 4263 2024-09-11T01:29:20.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7fe8405ec640 10 monclient: _renew_subs 2024-09-11T01:29:20.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7fe8405ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:20.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7fe8405ec640 1 -- 172.21.15.12:0/4138121050 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe83813cd90 con 0x7fe838137d50 2024-09-11T01:29:20.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7fe8405ec640 10 monclient: _renew_subs 2024-09-11T01:29:20.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7fe8405ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:20.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7fe8405ec640 1 -- 172.21.15.12:0/4138121050 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe83813cab0 con 0x7fe838137d50 2024-09-11T01:29:20.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7f36177fe640 1 -- 172.21.15.12:0/758197832 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3628041020 con 0x7f363013d280 2024-09-11T01:29:20.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7f36177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7f36177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:20.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7f36177fe640 10 monclient: dump: 2024-09-11T01:29:20.277 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.277 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.277 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.277 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.277 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.277 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.277 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.277 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.277 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.277 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7f36177fe640 10 monclient: _finish_auth 0 2024-09-11T01:29:20.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7f36177fe640 10 monclient: _check_auth_tickets 2024-09-11T01:29:20.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7f36177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:20.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7f36177fe640 1 -- 172.21.15.12:0/758197832 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3628007e50 con 0x7f363013d280 2024-09-11T01:29:20.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7f36177fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:20.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7f36177fe640 1 -- 172.21.15.12:0/758197832 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3628036040 con 0x7f363013d280 2024-09-11T01:29:20.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7f36177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.240+0000 7fe8405ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:20.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7f36177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:20.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7f36177fe640 10 monclient: dump: 2024-09-11T01:29:20.279 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.279 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.279 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.279 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.279 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.279 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.279 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.279 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.279 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.279 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.239+0000 7f362d7fa640 4 set_mon_vals no callback set 2024-09-11T01:29:20.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.240+0000 7f3635791640 5 monclient: authenticate success, global_id 4266 2024-09-11T01:29:20.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.240+0000 7f3635791640 10 monclient: _renew_subs 2024-09-11T01:29:20.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.240+0000 7f3635791640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:20.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.240+0000 7f3635791640 1 -- 172.21.15.12:0/758197832 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f363013bb90 con 0x7f363013d280 2024-09-11T01:29:20.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.240+0000 7fe82affd640 1 -- 172.21.15.12:0/4138121050 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe82c012a80 con 0x7fe838137d50 2024-09-11T01:29:20.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.240+0000 7fe82affd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:20.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.240+0000 7fe82affd640 1 -- 172.21.15.12:0/4138121050 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82c011380 con 0x7fe838137d50 2024-09-11T01:29:20.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.240+0000 7fe82affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.240+0000 7fe82affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:20.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.240+0000 7fe82affd640 10 monclient: dump: 2024-09-11T01:29:20.281 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.281 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.281 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.281 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.281 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.281 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.281 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.281 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.282 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.282 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.241+0000 7f36177fe640 1 -- 172.21.15.12:0/758197832 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f362803c470 con 0x7f363013d280 2024-09-11T01:29:20.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.241+0000 7f3635791640 10 monclient: _renew_subs 2024-09-11T01:29:20.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.241+0000 7f3635791640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:20.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.241+0000 7f3635791640 1 -- 172.21.15.12:0/758197832 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36301cdcd0 con 0x7f363013d280 2024-09-11T01:29:20.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.240+0000 7fe82affd640 1 -- 172.21.15.12:0/4138121050 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fe82c011520 con 0x7fe838137d50 2024-09-11T01:29:20.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.240+0000 7fe83cb85640 4 set_mon_vals no callback set 2024-09-11T01:29:20.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.240+0000 7fe82affd640 1 --2- 172.21.15.12:0/4138121050 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe81c03e1f0 0x7fe81c0406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.241+0000 7fe82affd640 1 -- 172.21.15.12:0/4138121050 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4402+0+0 (secure 0 0 0) 0x7fe82c052f00 con 0x7fe838137d50 2024-09-11T01:29:20.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.241+0000 7fe83db87640 1 --2- 172.21.15.12:0/4138121050 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe81c03e1f0 0x7fe81c0406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.241+0000 7fe8405ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:20.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.241+0000 7fe8405ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:20.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.241+0000 7fe8405ec640 1 -- 172.21.15.12:0/4138121050 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8381385d0 con 0x7fe838137d50 2024-09-11T01:29:20.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.241+0000 7fe83db87640 10 monclient: get_auth_request con 0x7fe81c03e1f0 auth_method 0 2024-09-11T01:29:20.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.242+0000 7f36177fe640 1 --2- 172.21.15.12:0/758197832 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f361c03e260 0x7f361c040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.242+0000 7f362effd640 1 --2- 172.21.15.12:0/758197832 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f361c03e260 0x7f361c040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.242+0000 7f362effd640 10 monclient: get_auth_request con 0x7f361c03e260 auth_method 0 2024-09-11T01:29:20.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.243+0000 7f3635791640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:20.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.243+0000 7f3635791640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:20.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.243+0000 7f3635791640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:20.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.243+0000 7f3635791640 1 -- 172.21.15.12:0/758197832 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36300a76a0 con 0x7f363013d280 2024-09-11T01:29:20.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.244+0000 7fe83db87640 1 --2- 172.21.15.12:0/4138121050 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe81c03e1f0 0x7fe81c0406b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe83809c070 tx=0x7fe83402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.245+0000 7f362effd640 1 --2- 172.21.15.12:0/758197832 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f361c03e260 0x7f361c040720 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3620009ac0 tx=0x7f3620009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.247+0000 7f36177fe640 1 -- 172.21.15.12:0/758197832 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4402+0+0 (secure 0 0 0) 0x7f3628000f60 con 0x7f363013d280 2024-09-11T01:29:20.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.247+0000 7fe82affd640 1 -- 172.21.15.12:0/4138121050 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe82c020020 con 0x7fe838137d50 2024-09-11T01:29:20.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.247+0000 7fe82affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:20.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.247+0000 7fe82affd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:20.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.251+0000 7f36177fe640 1 -- 172.21.15.12:0/758197832 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f36300a76a0 con 0x7f363013d280 2024-09-11T01:29:20.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.251+0000 7f36177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:20.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.251+0000 7f36177fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:20.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.269+0000 7fa2bdaf6640 10 monclient: get_monmap_and_config 2024-09-11T01:29:20.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.269+0000 7fa2bdaf6640 10 monclient: build_initial_monmap 2024-09-11T01:29:20.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.269+0000 7fa2bdaf6640 10 monclient: monmap: 2024-09-11T01:29:20.287 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:20.288 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.288 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:20.270149+0000 2024-09-11T01:29:20.288 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:20.270149+0000 2024-09-11T01:29:20.288 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:20.288 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.288 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:20.288 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:20.288 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:20.288 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:20.288 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:20.288 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.269+0000 7fda64ebe640 10 monclient: get_monmap_and_config 2024-09-11T01:29:20.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.269+0000 7fda64ebe640 10 monclient: build_initial_monmap 2024-09-11T01:29:20.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.269+0000 7fda64ebe640 10 monclient: monmap: 2024-09-11T01:29:20.289 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:20.289 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.289 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:20.270340+0000 2024-09-11T01:29:20.289 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:20.270340+0000 2024-09-11T01:29:20.289 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:20.289 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.290 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:20.290 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:20.290 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:20.290 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:20.290 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:20.290 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.270+0000 7fda64ebe640 1 Processor -- start 2024-09-11T01:29:20.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.270+0000 7fa2bdaf6640 1 Processor -- start 2024-09-11T01:29:20.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.270+0000 7fda64ebe640 1 -- start start 2024-09-11T01:29:20.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.270+0000 7fda64ebe640 10 monclient: init 2024-09-11T01:29:20.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.270+0000 7fa2bdaf6640 1 -- start start 2024-09-11T01:29:20.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.270+0000 7fa2bdaf6640 10 monclient: init 2024-09-11T01:29:20.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.271+0000 7fda64ebe640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:20.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.271+0000 7fda64ebe640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-11T01:29:20.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.271+0000 7fda64ebe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda60074180 0x7fda60074560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.271+0000 7fa2bdaf6640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:20.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.271+0000 7fda64ebe640 10 monclient(hunting): picked mon.noname-b con 0x7fda60074180 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:20.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.271+0000 7fa2bdaf6640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-11T01:29:20.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.271+0000 7fa2bdaf6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa2b8137d50 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:20.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.271+0000 7fda64ebe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fda600a06d0 0x7fda600a25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.271+0000 7fda64ebe640 10 monclient(hunting): picked mon.noname-c con 0x7fda600a06d0 addr v2:172.21.15.71:3300/0 2024-09-11T01:29:20.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.271+0000 7fda64ebe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fda60134ac0 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:20.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.271+0000 7fa2bdaf6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa2b8138130 0x7fa2b8138550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.271+0000 7fda64ebe640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda60074aa0 con 0x7fda60134ac0 2024-09-11T01:29:20.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.271+0000 7fda64ebe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.271+0000 7fda5e59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda60074180 0x7fda60074560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.271+0000 7fda64ebe640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fda60135560 con 0x7fda600a06d0 2024-09-11T01:29:20.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.271+0000 7fa2bdaf6640 10 monclient(hunting): picked mon.noname-b con 0x7fa2b8138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:20.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.271+0000 7fda5e59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda60074180 0x7fda60074560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51650/0 (socket says 172.21.15.12:51650) 2024-09-11T01:29:20.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.271+0000 7fda5e59c640 1 -- 172.21.15.12:0/465259808 learned_addr learned my addr 172.21.15.12:0/465259808 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:20.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.271+0000 7fda64ebe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.271+0000 7fda64ebe640 1 -- 172.21.15.12:0/465259808 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fda600a0490 con 0x7fda60074180 2024-09-11T01:29:20.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.271+0000 7fda64ebe640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:20.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.271+0000 7fda64ebe640 10 monclient(hunting): authenticate will time out at 766.713956s 2024-09-11T01:29:20.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2bdaf6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa2b813d280 0x7fa2b813f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2bdaf6640 10 monclient(hunting): picked mon.noname-c con 0x7fa2b813d280 addr v2:172.21.15.71:3300/0 2024-09-11T01:29:20.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2b6ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa2b8138130 0x7fa2b8138550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2b6ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa2b8138130 0x7fa2b8138550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51664/0 (socket says 172.21.15.12:51664) 2024-09-11T01:29:20.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2b6ffd640 1 -- 172.21.15.12:0/2104466462 learned_addr learned my addr 172.21.15.12:0/2104466462 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:20.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2bdaf6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2b809d820 con 0x7fa2b8137d50 2024-09-11T01:29:20.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fda5e59c640 10 monclient(hunting): get_auth_request con 0x7fda60074180 auth_method 0 2024-09-11T01:29:20.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fda5e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fda5e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fda5e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fda5dd9b640 1 --2- 172.21.15.12:0/465259808 >> v2:172.21.15.71:3300/0 conn(0x7fda600a06d0 0x7fda600a25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fda5dd9b640 10 monclient(hunting): get_auth_request con 0x7fda600a06d0 auth_method 0 2024-09-11T01:29:20.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fda5dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fda5dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fda5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2b7fff640 1 --2- 172.21.15.12:0/2104466462 >> v2:172.21.15.71:3300/0 conn(0x7fa2b813d280 0x7fa2b813f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fda5dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fda5dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fda5dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2bdaf6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fda5e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fda5e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fda5e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2bdaf6640 1 -- 172.21.15.12:0/2104466462 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fa2b809c650 con 0x7fa2b813d280 2024-09-11T01:29:20.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2bdaf6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2bdaf6640 1 -- 172.21.15.12:0/2104466462 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa2b809ba20 con 0x7fa2b8138130 2024-09-11T01:29:20.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2bdaf6640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:20.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2bdaf6640 10 monclient(hunting): authenticate will time out at 766.714587s 2024-09-11T01:29:20.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2b6ffd640 10 monclient(hunting): get_auth_request con 0x7fa2b8138130 auth_method 0 2024-09-11T01:29:20.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2b7fff640 10 monclient(hunting): get_auth_request con 0x7fa2b813d280 auth_method 0 2024-09-11T01:29:20.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2b7fff640 10 monclient(hunting): handle_auth_done global_id 4246 payload 274 2024-09-11T01:29:20.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2b7fff640 1 -- 172.21.15.12:0/2104466462 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa2b8138130 msgr2=0x7fa2b8138550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:20.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2b7fff640 1 --2- 172.21.15.12:0/2104466462 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa2b8138130 0x7fa2b8138550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fa2b7fff640 1 -- 172.21.15.12:0/2104466462 >> v1:172.21.15.71:6789/0 conn(0x7fa2b8137d50 legacy=0x7fa2b809d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:20.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fda5dd9b640 10 monclient(hunting): handle_auth_done global_id 4243 payload 274 2024-09-11T01:29:20.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fda5dd9b640 1 -- 172.21.15.12:0/465259808 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda60074180 msgr2=0x7fda60074560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:20.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fda5dd9b640 1 --2- 172.21.15.12:0/465259808 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda60074180 0x7fda60074560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fda5dd9b640 1 -- 172.21.15.12:0/465259808 >> v1:172.21.15.71:6789/0 conn(0x7fda60134ac0 legacy=0x7fda600a2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:20.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fda5dd9b640 10 monclient: _finish_hunting 0 2024-09-11T01:29:20.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fda5dd9b640 1 monclient: found mon.noname-c 2024-09-11T01:29:20.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fda5dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:20.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fda5dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:29:20.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.272+0000 7fda5dd9b640 1 -- 172.21.15.12:0/465259808 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda60070160 con 0x7fda600a06d0 2024-09-11T01:29:20.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.273+0000 7fa2b7fff640 10 monclient: _finish_hunting 0 2024-09-11T01:29:20.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.273+0000 7fa2b7fff640 1 monclient: found mon.noname-c 2024-09-11T01:29:20.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.273+0000 7fa2b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:20.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.273+0000 7fa2b7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:29:20.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.273+0000 7fa2b7fff640 1 -- 172.21.15.12:0/2104466462 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2b809c370 con 0x7fa2b813d280 2024-09-11T01:29:20.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.275+0000 7fbb241ad640 10 monclient: get_monmap_and_config 2024-09-11T01:29:20.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.275+0000 7fbb241ad640 10 monclient: build_initial_monmap 2024-09-11T01:29:20.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.275+0000 7fbb241ad640 10 monclient: monmap: 2024-09-11T01:29:20.305 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:20.305 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.305 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:20.275960+0000 2024-09-11T01:29:20.305 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:20.275960+0000 2024-09-11T01:29:20.305 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:20.306 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.306 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:20.306 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:20.306 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:20.306 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:20.306 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:20.306 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.275+0000 7fbb241ad640 1 Processor -- start 2024-09-11T01:29:20.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.275+0000 7fda5dd9b640 1 -- 172.21.15.12:0/465259808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda600a06d0 msgr2=0x7fda600a25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:20.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.276+0000 7fda5dd9b640 1 --2- 172.21.15.12:0/465259808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda600a06d0 0x7fda600a25d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fda540097d0 tx=0x7fda54029ff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3beaf7630e83ec60 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.276+0000 7fbb241ad640 1 -- start start 2024-09-11T01:29:20.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.276+0000 7fbb241ad640 10 monclient: init 2024-09-11T01:29:20.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.276+0000 7fbb241ad640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:20.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.276+0000 7fbb241ad640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-11T01:29:20.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.276+0000 7fbb241ad640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbb1c137d50 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:20.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.276+0000 7fbb241ad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbb1c138130 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:20.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.276+0000 7fbb241ad640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbb1c13e290 0x7fbb1c140680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.276+0000 7fbb241ad640 10 monclient(hunting): picked mon.noname-b con 0x7fbb1c13e290 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:20.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.276+0000 7fbb241ad640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb1c09d820 con 0x7fbb1c137d50 2024-09-11T01:29:20.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.276+0000 7fbb241ad640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb1c09ba20 con 0x7fbb1c138130 2024-09-11T01:29:20.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.276+0000 7fbb241ad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.276+0000 7fbb241ad640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fbb1c09c650 con 0x7fbb1c13e290 2024-09-11T01:29:20.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.276+0000 7fbb241ad640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:20.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.276+0000 7fbb241ad640 10 monclient(hunting): authenticate will time out at 766.718818s 2024-09-11T01:29:20.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.276+0000 7fbb21f49640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbb1c137d50 0x7fbb1c09d110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56054/0 (socket says 172.21.15.12:56054) 2024-09-11T01:29:20.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.276+0000 7fbb21f49640 1 -- 172.21.15.12:0/1585547803 learned_addr learned my addr 172.21.15.12:0/1585547803 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:29:20.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.276+0000 7fbb2274a640 1 --2- 172.21.15.12:0/1585547803 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbb1c13e290 0x7fbb1c140680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.276+0000 7fbb2274a640 10 monclient(hunting): get_auth_request con 0x7fbb1c13e290 auth_method 0 2024-09-11T01:29:20.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.276+0000 7fbb2274a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.276+0000 7fbb2274a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.276+0000 7fbb2274a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.277+0000 7fbb2274a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.277+0000 7fbb2274a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.277+0000 7fbb2274a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.277+0000 7fcca759e640 10 monclient: get_monmap_and_config 2024-09-11T01:29:20.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.277+0000 7fcca759e640 10 monclient: build_initial_monmap 2024-09-11T01:29:20.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.277+0000 7fcca759e640 10 monclient: monmap: 2024-09-11T01:29:20.310 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:20.311 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.311 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:20.277886+0000 2024-09-11T01:29:20.311 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:20.277886+0000 2024-09-11T01:29:20.311 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:20.311 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.311 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:20.311 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:20.311 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:20.311 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:20.311 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:20.311 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.277+0000 7fbb2274a640 10 monclient(hunting): handle_auth_done global_id 4289 payload 274 2024-09-11T01:29:20.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.277+0000 7fbb2274a640 1 -- 172.21.15.12:0/1585547803 >> v1:172.21.15.71:6789/0 conn(0x7fbb1c138130 legacy=0x7fbb1c138580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:20.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.277+0000 7fbb2274a640 1 -- 172.21.15.12:0/1585547803 >> v1:172.21.15.12:6789/0 conn(0x7fbb1c137d50 legacy=0x7fbb1c09d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:20.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.277+0000 7fbb2274a640 10 monclient: _finish_hunting 0 2024-09-11T01:29:20.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.277+0000 7fbb2274a640 1 monclient: found mon.noname-b 2024-09-11T01:29:20.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.277+0000 7fbb2274a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:20.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.277+0000 7fbb2274a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:29:20.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.277+0000 7fbb2274a640 1 -- 172.21.15.12:0/1585547803 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb1c09c370 con 0x7fbb1c13e290 2024-09-11T01:29:20.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.277+0000 7fcca759e640 1 Processor -- start 2024-09-11T01:29:20.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.277+0000 7fcca759e640 1 -- start start 2024-09-11T01:29:20.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.277+0000 7fcca759e640 10 monclient: init 2024-09-11T01:29:20.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fa2b7fff640 1 -- 172.21.15.12:0/2104466462 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa2b813d280 msgr2=0x7fa2b813f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:20.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fa2b7fff640 1 --2- 172.21.15.12:0/2104466462 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa2b813d280 0x7fa2b813f670 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00097a0 tx=0x7fa2b002a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36fae6aa4892f76 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fcca759e640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:20.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fda5d59a640 1 -- 172.21.15.12:0/465259808 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda54037070 con 0x7fda600a06d0 2024-09-11T01:29:20.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fda5d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fda5d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:20.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fda5d59a640 10 monclient: dump: 2024-09-11T01:29:20.315 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.315 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.315 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.315 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.315 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.315 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.315 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.315 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.315 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.315 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fda5d59a640 10 monclient: _finish_auth 0 2024-09-11T01:29:20.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fda5d59a640 10 monclient: _check_auth_tickets 2024-09-11T01:29:20.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fda5d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:20.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fcca759e640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-11T01:29:20.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fda64ebe640 5 monclient: authenticate success, global_id 4243 2024-09-11T01:29:20.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fda5d59a640 1 -- 172.21.15.12:0/465259808 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda54032440 con 0x7fda600a06d0 2024-09-11T01:29:20.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fda64ebe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:20.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fda5d59a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:20.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fda64ebe640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:20.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fda5d59a640 1 -- 172.21.15.12:0/465259808 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda54032810 con 0x7fda600a06d0 2024-09-11T01:29:20.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fda64ebe640 4 set_mon_vals no callback set 2024-09-11T01:29:20.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fda5d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fda5d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:20.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fda5d59a640 10 monclient: dump: 2024-09-11T01:29:20.317 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.317 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.317 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.317 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.317 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.317 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.317 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.317 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.317 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.317 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fcca759e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcca806cb50 0x7fcca8068950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fcca759e640 10 monclient(hunting): picked mon.noname-c con 0x7fcca806cb50 addr v2:172.21.15.71:3300/0 2024-09-11T01:29:20.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fcca759e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcca8068e90 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:20.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fcca759e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcca80692a0 0x7fcca813fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fcca759e640 10 monclient(hunting): picked mon.noname-a con 0x7fcca80692a0 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:20.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fcca759e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcca8069ca0 con 0x7fcca8068e90 2024-09-11T01:29:20.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fcca759e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fcca759e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fcca80a14d0 con 0x7fcca80692a0 2024-09-11T01:29:20.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fcca759e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fcca759e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fcca806d490 con 0x7fcca806cb50 2024-09-11T01:29:20.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fcca759e640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:20.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fcca759e640 10 monclient(hunting): authenticate will time out at 766.720606s 2024-09-11T01:29:20.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fda64ebe640 10 monclient: shutdown 2024-09-11T01:29:20.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fda64ebe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:20.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fcca5d9b640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fcca8068e90 0x7fcca813c620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56056/0 (socket says 172.21.15.12:56056) 2024-09-11T01:29:20.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fcca5d9b640 1 -- 172.21.15.12:0/2671796722 learned_addr learned my addr 172.21.15.12:0/2671796722 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:29:20.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fcca6d9d640 1 --2- 172.21.15.12:0/2671796722 >> v2:172.21.15.12:3300/0 conn(0x7fcca80692a0 0x7fcca813fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fcca6d9d640 10 monclient(hunting): get_auth_request con 0x7fcca80692a0 auth_method 0 2024-09-11T01:29:20.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fcca6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fcca6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fcca6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fcca6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fcca6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.278+0000 7fcca6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fcca6d9d640 10 monclient(hunting): handle_auth_done global_id 4269 payload 274 2024-09-11T01:29:20.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fcca6d9d640 1 -- 172.21.15.12:0/2671796722 >> v2:172.21.15.71:3300/0 conn(0x7fcca806cb50 msgr2=0x7fcca8068950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-11T01:29:20.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fcca6d9d640 1 --2- 172.21.15.12:0/2671796722 >> v2:172.21.15.71:3300/0 conn(0x7fcca806cb50 0x7fcca8068950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fcca6d9d640 1 -- 172.21.15.12:0/2671796722 >> v1:172.21.15.12:6789/0 conn(0x7fcca8068e90 legacy=0x7fcca813c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:20.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fcca6d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:29:20.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fcca6d9d640 1 monclient: found mon.noname-a 2024-09-11T01:29:20.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fcca6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:20.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fcca6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:20.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fcca6d9d640 1 -- 172.21.15.12:0/2671796722 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcca809f9d0 con 0x7fcca80692a0 2024-09-11T01:29:20.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fda64ebe640 1 -- 172.21.15.12:0/465259808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda600a06d0 msgr2=0x7fda600a25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fda64ebe640 1 --2- 172.21.15.12:0/465259808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda600a06d0 0x7fda600a25d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fda540097d0 tx=0x7fda54029ff0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fa2b67fc640 1 -- 172.21.15.12:0/2104466462 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b0037070 con 0x7fa2b813d280 2024-09-11T01:29:20.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fa2b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fa2b67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:20.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fa2b67fc640 10 monclient: dump: 2024-09-11T01:29:20.323 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.323 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.323 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.323 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.323 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.323 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.324 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.324 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.324 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.324 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fa2b67fc640 10 monclient: _finish_auth 0 2024-09-11T01:29:20.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fa2b67fc640 10 monclient: _check_auth_tickets 2024-09-11T01:29:20.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fa2b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:20.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fa2b67fc640 1 -- 172.21.15.12:0/2104466462 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2b0032440 con 0x7fa2b813d280 2024-09-11T01:29:20.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fa2b67fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:20.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fa2b67fc640 1 -- 172.21.15.12:0/2104466462 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b0032810 con 0x7fa2b813d280 2024-09-11T01:29:20.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fa2b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fa2b67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:20.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fa2b67fc640 10 monclient: dump: 2024-09-11T01:29:20.324 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.325 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.325 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.325 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.325 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.325 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.325 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.325 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.325 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.325 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fa2bdaf6640 5 monclient: authenticate success, global_id 4246 2024-09-11T01:29:20.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fa2bdaf6640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:20.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fa2bdaf6640 4 set_mon_vals no callback set 2024-09-11T01:29:20.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.279+0000 7fda64ebe640 1 -- 172.21.15.12:0/465259808 shutdown_connections 2024-09-11T01:29:20.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fda64ebe640 1 --2- 172.21.15.12:0/465259808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda600a06d0 0x7fda600a25d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fda64ebe640 1 --2- 172.21.15.12:0/465259808 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda60074180 0x7fda60074560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fa2bdaf6640 10 monclient: shutdown 2024-09-11T01:29:20.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fa2bdaf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:20.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fa2bdaf6640 1 -- 172.21.15.12:0/2104466462 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa2b813d280 msgr2=0x7fa2b813f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fa2bdaf6640 1 --2- 172.21.15.12:0/2104466462 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa2b813d280 0x7fa2b813f670 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00097a0 tx=0x7fa2b002a060 comp rx=0 tx=0).stop 2024-09-11T01:29:20.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fda64ebe640 1 -- 172.21.15.12:0/465259808 >> 172.21.15.12:0/465259808 conn(0x7fda600a36c0 msgr2=0x7fda600a3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:20.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fbb2274a640 1 -- 172.21.15.12:0/1585547803 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbb1c13e290 msgr2=0x7fbb1c140680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:20.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fbb2274a640 1 --2- 172.21.15.12:0/1585547803 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbb1c13e290 0x7fbb1c140680 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbb14009dc0 tx=0x7fbb1402a510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e42b7368b0a191e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fda64ebe640 1 -- 172.21.15.12:0/465259808 shutdown_connections 2024-09-11T01:29:20.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fda64ebe640 1 -- 172.21.15.12:0/465259808 wait complete. 2024-09-11T01:29:20.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fa2bdaf6640 1 -- 172.21.15.12:0/2104466462 shutdown_connections 2024-09-11T01:29:20.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fa2bdaf6640 1 --2- 172.21.15.12:0/2104466462 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa2b813d280 0x7fa2b813f670 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fa2bdaf6640 1 --2- 172.21.15.12:0/2104466462 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa2b8138130 0x7fa2b8138550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fa2bdaf6640 1 -- 172.21.15.12:0/2104466462 >> 172.21.15.12:0/2104466462 conn(0x7fa2b80a7020 msgr2=0x7fa2b80a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:20.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fa2bdaf6640 1 -- 172.21.15.12:0/2104466462 shutdown_connections 2024-09-11T01:29:20.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fa2bdaf6640 1 -- 172.21.15.12:0/2104466462 wait complete. 2024-09-11T01:29:20.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fbb20f47640 1 -- 172.21.15.12:0/1585547803 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb14007a30 con 0x7fbb1c13e290 2024-09-11T01:29:20.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fbb20f47640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fbb20f47640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:20.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fbb20f47640 10 monclient: dump: 2024-09-11T01:29:20.329 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.329 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.329 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.329 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.329 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.329 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.329 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.329 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.329 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.330 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fbb20f47640 10 monclient: _finish_auth 0 2024-09-11T01:29:20.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fbb20f47640 10 monclient: _check_auth_tickets 2024-09-11T01:29:20.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fbb20f47640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:20.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fbb20f47640 1 -- 172.21.15.12:0/1585547803 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb14007bd0 con 0x7fbb1c13e290 2024-09-11T01:29:20.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fbb241ad640 5 monclient: authenticate success, global_id 4289 2024-09-11T01:29:20.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fbb241ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:20.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fbb20f47640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:20.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fbb241ad640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:20.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fbb241ad640 4 set_mon_vals no callback set 2024-09-11T01:29:20.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fbb20f47640 1 -- 172.21.15.12:0/1585547803 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb14031400 con 0x7fbb1c13e290 2024-09-11T01:29:20.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fbb20f47640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fbb20f47640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:20.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.280+0000 7fbb20f47640 10 monclient: dump: 2024-09-11T01:29:20.331 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.331 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.331 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.331 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.331 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.331 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.331 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.331 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.331 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.331 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.281+0000 7fa2bdaf6640 10 monclient: build_initial_monmap 2024-09-11T01:29:20.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.281+0000 7fa2bdaf6640 10 monclient: monmap: 2024-09-11T01:29:20.332 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:20.332 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:20.332 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:20.332 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:20.332 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:20.332 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.332 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:20.332 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:20.332 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:20.332 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.281+0000 7fbb241ad640 10 monclient: shutdown 2024-09-11T01:29:20.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.281+0000 7fbb241ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:20.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.281+0000 7fbb241ad640 1 -- 172.21.15.12:0/1585547803 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbb1c13e290 msgr2=0x7fbb1c140680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.281+0000 7fbb241ad640 1 --2- 172.21.15.12:0/1585547803 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbb1c13e290 0x7fbb1c140680 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbb14009dc0 tx=0x7fbb1402a510 comp rx=0 tx=0).stop 2024-09-11T01:29:20.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.281+0000 7fa2bdaf6640 1 Processor -- start 2024-09-11T01:29:20.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.281+0000 7fbb241ad640 1 -- 172.21.15.12:0/1585547803 shutdown_connections 2024-09-11T01:29:20.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.281+0000 7fbb241ad640 1 --2- 172.21.15.12:0/1585547803 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbb1c13e290 0x7fbb1c140680 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.281+0000 7fbb241ad640 1 -- 172.21.15.12:0/1585547803 >> 172.21.15.12:0/1585547803 conn(0x7fbb1c0a7020 msgr2=0x7fbb1c0a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:20.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.281+0000 7fda64ebe640 10 monclient: build_initial_monmap 2024-09-11T01:29:20.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.281+0000 7fda64ebe640 10 monclient: monmap: 2024-09-11T01:29:20.334 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:20.334 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:20.334 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:20.334 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:20.334 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:20.334 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.335 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:20.335 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:20.335 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:20.335 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.281+0000 7fa2bdaf6640 1 -- start start 2024-09-11T01:29:20.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.281+0000 7fa2bdaf6640 10 monclient: init 2024-09-11T01:29:20.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.281+0000 7fbb241ad640 1 -- 172.21.15.12:0/1585547803 shutdown_connections 2024-09-11T01:29:20.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.281+0000 7fbb241ad640 1 -- 172.21.15.12:0/1585547803 wait complete. 2024-09-11T01:29:20.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2bdaf6640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:20.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2bdaf6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:29:20.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2bdaf6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa2b8137d50 0x7fa2b81ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2bdaf6640 10 monclient(hunting): picked mon.noname-c con 0x7fa2b8137d50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:20.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2bdaf6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa2b8138130 0x7fa2b8136d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2bdaf6640 10 monclient(hunting): picked mon.noname-b con 0x7fa2b8138130 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:20.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2bdaf6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2b813d280 0x7fa2b813a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2bdaf6640 10 monclient(hunting): picked mon.noname-a con 0x7fa2b813d280 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:20.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2bdaf6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b77fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa2b8137d50 0x7fa2b81ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b77fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa2b8137d50 0x7fa2b81ccd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51680/0 (socket says 172.21.15.12:51680) 2024-09-11T01:29:20.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b77fe640 1 -- 172.21.15.12:0/1051573658 learned_addr learned my addr 172.21.15.12:0/1051573658 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:20.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2bdaf6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b806f850 con 0x7fa2b813d280 2024-09-11T01:29:20.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2bdaf6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2bdaf6640 1 -- 172.21.15.12:0/1051573658 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b806f6d0 con 0x7fa2b8138130 2024-09-11T01:29:20.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2bdaf6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2bdaf6640 1 -- 172.21.15.12:0/1051573658 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa2b813fbb0 con 0x7fa2b8137d50 2024-09-11T01:29:20.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2bdaf6640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:20.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2bdaf6640 10 monclient(hunting): authenticate will time out at 766.724351s 2024-09-11T01:29:20.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b77fe640 10 monclient(hunting): get_auth_request con 0x7fa2b8137d50 auth_method 0 2024-09-11T01:29:20.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fbb241ad640 10 monclient: build_initial_monmap 2024-09-11T01:29:20.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fda64ebe640 1 Processor -- start 2024-09-11T01:29:20.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fbb241ad640 10 monclient: monmap: 2024-09-11T01:29:20.338 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:20.339 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:20.339 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:20.339 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:20.339 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:20.339 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.339 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:20.339 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:20.339 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:20.339 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b7fff640 1 --2- 172.21.15.12:0/1051573658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2b813d280 0x7fa2b813a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b7fff640 10 monclient(hunting): get_auth_request con 0x7fa2b813d280 auth_method 0 2024-09-11T01:29:20.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fda64ebe640 1 -- start start 2024-09-11T01:29:20.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fda64ebe640 10 monclient: init 2024-09-11T01:29:20.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b6ffd640 1 --2- 172.21.15.12:0/1051573658 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa2b8138130 0x7fa2b8136d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b6ffd640 10 monclient(hunting): get_auth_request con 0x7fa2b8138130 auth_method 0 2024-09-11T01:29:20.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b77fe640 10 monclient(hunting): handle_auth_done global_id 4292 payload 824 2024-09-11T01:29:20.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b77fe640 1 -- 172.21.15.12:0/1051573658 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa2b8138130 msgr2=0x7fa2b8136d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fbb241ad640 1 Processor -- start 2024-09-11T01:29:20.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b77fe640 1 --2- 172.21.15.12:0/1051573658 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa2b8138130 0x7fa2b8136d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b77fe640 1 -- 172.21.15.12:0/1051573658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2b813d280 msgr2=0x7fa2b813a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b77fe640 1 --2- 172.21.15.12:0/1051573658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2b813d280 0x7fa2b813a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fbb241ad640 1 -- start start 2024-09-11T01:29:20.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fbb241ad640 10 monclient: init 2024-09-11T01:29:20.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b77fe640 10 monclient: _finish_hunting 0 2024-09-11T01:29:20.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b77fe640 1 monclient: found mon.noname-c 2024-09-11T01:29:20.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:20.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:29:20.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fcca6d9d640 1 -- 172.21.15.12:0/2671796722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcca80692a0 msgr2=0x7fcca813fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:20.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fcca6d9d640 1 --2- 172.21.15.12:0/2671796722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcca80692a0 0x7fcca813fd50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcca809fba0 tx=0x7fcca0032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e567e885fc09862f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b77fe640 1 -- 172.21.15.12:0/1051573658 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2b8137ac0 con 0x7fa2b8137d50 2024-09-11T01:29:20.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.282+0000 7fa2b7fff640 1 --2- 172.21.15.12:0/1051573658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2b813d280 0x7fa2b813a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:20.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fda64ebe640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:20.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fda64ebe640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:29:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fcca559a640 1 -- 172.21.15.12:0/2671796722 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca0040070 con 0x7fcca80692a0 2024-09-11T01:29:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fcca559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fcca559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fcca559a640 10 monclient: dump: 2024-09-11T01:29:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.346 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.347 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.347 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.347 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fcca559a640 10 monclient: _finish_auth 0 2024-09-11T01:29:20.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fcca559a640 10 monclient: _check_auth_tickets 2024-09-11T01:29:20.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fcca559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:20.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fcca559a640 1 -- 172.21.15.12:0/2671796722 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcca0033d10 con 0x7fcca80692a0 2024-09-11T01:29:20.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fcca759e640 5 monclient: authenticate success, global_id 4269 2024-09-11T01:29:20.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fda64ebe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda60074180 0x7fda601ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fcca759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fcca559a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fcca759e640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fcca759e640 4 set_mon_vals no callback set 2024-09-11T01:29:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fcca559a640 1 -- 172.21.15.12:0/2671796722 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca003b660 con 0x7fcca80692a0 2024-09-11T01:29:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fcca559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fcca559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fcca559a640 10 monclient: dump: 2024-09-11T01:29:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.348 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.349 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.349 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.349 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.349 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fda64ebe640 10 monclient(hunting): picked mon.noname-c con 0x7fda60074180 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:20.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fda64ebe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda600a06d0 0x7fda601cd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fda64ebe640 10 monclient(hunting): picked mon.noname-b con 0x7fda600a06d0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:20.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fbb241ad640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:20.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fda64ebe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda60134ac0 0x7fda601cda00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fbb241ad640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:29:20.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fbb241ad640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb1c137d50 0x7fbb1c13c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fbb241ad640 10 monclient(hunting): picked mon.noname-a con 0x7fbb1c137d50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:20.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fbb241ad640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb1c138130 0x7fbb1c136cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb241ad640 10 monclient(hunting): picked mon.noname-b con 0x7fbb1c138130 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:20.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb241ad640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbb1c13e290 0x7fbb1c139ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fda64ebe640 10 monclient(hunting): picked mon.noname-a con 0x7fda60134ac0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:20.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.283+0000 7fda5ed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda60134ac0 0x7fda601cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5dd9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda600a06d0 0x7fda601cd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5dd9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda600a06d0 0x7fda601cd4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:34838/0 (socket says 172.21.15.12:34838) 2024-09-11T01:29:20.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5dd9b640 1 -- 172.21.15.12:0/1705444474 learned_addr learned my addr 172.21.15.12:0/1705444474 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:20.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda64ebe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda64ebe640 1 -- 172.21.15.12:0/1705444474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fda600a3220 con 0x7fda60134ac0 2024-09-11T01:29:20.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda64ebe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda64ebe640 1 -- 172.21.15.12:0/1705444474 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fda600a33a0 con 0x7fda600a06d0 2024-09-11T01:29:20.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda64ebe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda64ebe640 1 -- 172.21.15.12:0/1705444474 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fda6013fee0 con 0x7fda60074180 2024-09-11T01:29:20.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda64ebe640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:20.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21748640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb1c138130 0x7fbb1c136cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21748640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb1c138130 0x7fbb1c136cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:34852/0 (socket says 172.21.15.12:34852) 2024-09-11T01:29:20.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21748640 1 -- 172.21.15.12:0/2116106454 learned_addr learned my addr 172.21.15.12:0/2116106454 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:20.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb241ad640 10 monclient(hunting): picked mon.noname-c con 0x7fbb1c13e290 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:20.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda64ebe640 10 monclient(hunting): authenticate will time out at 766.726235s 2024-09-11T01:29:20.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb241ad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5dd9b640 10 monclient(hunting): get_auth_request con 0x7fda600a06d0 auth_method 0 2024-09-11T01:29:20.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb241ad640 1 -- 172.21.15.12:0/2116106454 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1c140bc0 con 0x7fbb1c137d50 2024-09-11T01:29:20.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb241ad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb241ad640 1 -- 172.21.15.12:0/2116106454 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1c0a6da0 con 0x7fbb1c138130 2024-09-11T01:29:20.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5ed9d640 10 monclient(hunting): get_auth_request con 0x7fda60134ac0 auth_method 0 2024-09-11T01:29:20.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb241ad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb241ad640 1 -- 172.21.15.12:0/2116106454 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fbb1c140d40 con 0x7fbb1c13e290 2024-09-11T01:29:20.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb241ad640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:20.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb241ad640 10 monclient(hunting): authenticate will time out at 766.726406s 2024-09-11T01:29:20.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21748640 10 monclient(hunting): get_auth_request con 0x7fbb1c138130 auth_method 0 2024-09-11T01:29:20.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21748640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21748640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21748640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21f49640 1 --2- 172.21.15.12:0/2116106454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb1c137d50 0x7fbb1c13c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21748640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21748640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fcca759e640 10 monclient: shutdown 2024-09-11T01:29:20.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fcca759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:20.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fcca759e640 1 -- 172.21.15.12:0/2671796722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcca80692a0 msgr2=0x7fcca813fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fcca759e640 1 --2- 172.21.15.12:0/2671796722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcca80692a0 0x7fcca813fd50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcca809fba0 tx=0x7fcca0032d10 comp rx=0 tx=0).stop 2024-09-11T01:29:20.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21748640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21f49640 10 monclient(hunting): get_auth_request con 0x7fbb1c137d50 auth_method 0 2024-09-11T01:29:20.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21f49640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21f49640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21f49640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5ed9d640 10 monclient(hunting): handle_auth_done global_id 4275 payload 824 2024-09-11T01:29:20.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5ed9d640 1 -- 172.21.15.12:0/1705444474 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda60074180 msgr2=0x7fda601ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5ed9d640 1 --2- 172.21.15.12:0/1705444474 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda60074180 0x7fda601ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5ed9d640 1 -- 172.21.15.12:0/1705444474 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda600a06d0 msgr2=0x7fda601cd4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5ed9d640 1 --2- 172.21.15.12:0/1705444474 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda600a06d0 0x7fda601cd4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5ed9d640 10 monclient: _finish_hunting 0 2024-09-11T01:29:20.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5ed9d640 1 monclient: found mon.noname-a 2024-09-11T01:29:20.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:20.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:20.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5ed9d640 1 -- 172.21.15.12:0/1705444474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda600a6840 con 0x7fda60134ac0 2024-09-11T01:29:20.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fda5dd9b640 1 --2- 172.21.15.12:0/1705444474 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda600a06d0 0x7fda601cd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:20.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21f49640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21f49640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21f49640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21748640 10 monclient(hunting): handle_auth_done global_id 4255 payload 824 2024-09-11T01:29:20.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fcca759e640 1 -- 172.21.15.12:0/2671796722 shutdown_connections 2024-09-11T01:29:20.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fcca759e640 1 --2- 172.21.15.12:0/2671796722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcca80692a0 0x7fcca813fd50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fcca759e640 1 --2- 172.21.15.12:0/2671796722 >> v2:172.21.15.71:3300/0 conn(0x7fcca806cb50 0x7fcca8068950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fcca759e640 1 -- 172.21.15.12:0/2671796722 >> 172.21.15.12:0/2671796722 conn(0x7fcca80a7660 msgr2=0x7fcca80a7a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:20.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21748640 1 -- 172.21.15.12:0/2116106454 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbb1c13e290 msgr2=0x7fbb1c139ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21748640 1 --2- 172.21.15.12:0/2116106454 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbb1c13e290 0x7fbb1c139ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21748640 1 -- 172.21.15.12:0/2116106454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb1c137d50 msgr2=0x7fbb1c13c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21748640 1 --2- 172.21.15.12:0/2116106454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb1c137d50 0x7fbb1c13c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21748640 10 monclient: _finish_hunting 0 2024-09-11T01:29:20.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21748640 1 monclient: found mon.noname-b 2024-09-11T01:29:20.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21748640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:20.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21748640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-09-11T01:29:20.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fbb21748640 1 -- 172.21.15.12:0/2116106454 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb1c13cf20 con 0x7fbb1c138130 2024-09-11T01:29:20.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fcca759e640 1 -- 172.21.15.12:0/2671796722 shutdown_connections 2024-09-11T01:29:20.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.284+0000 7fcca759e640 1 -- 172.21.15.12:0/2671796722 wait complete. 2024-09-11T01:29:20.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.285+0000 7fa2b77fe640 1 -- 172.21.15.12:0/1051573658 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa2b8137d50 msgr2=0x7fa2b81ccd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:20.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.285+0000 7fa2b77fe640 1 --2- 172.21.15.12:0/1051573658 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa2b8137d50 0x7fa2b81ccd40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa2a800c7c0 tx=0x7fa2a800cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.285+0000 7fa2affff640 1 -- 172.21.15.12:0/1051573658 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8016020 con 0x7fa2b8137d50 2024-09-11T01:29:20.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.285+0000 7fa2affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.285+0000 7fa2affff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:20.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.285+0000 7fa2affff640 10 monclient: dump: 2024-09-11T01:29:20.364 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.364 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.364 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.364 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.364 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.364 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.364 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.365 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.365 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.365 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.285+0000 7fa2affff640 10 monclient: _finish_auth 0 2024-09-11T01:29:20.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.285+0000 7fa2affff640 10 monclient: _check_auth_tickets 2024-09-11T01:29:20.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.285+0000 7fa2affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:20.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.285+0000 7fa2affff640 1 -- 172.21.15.12:0/1051573658 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2a800ac60 con 0x7fa2b8137d50 2024-09-11T01:29:20.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.285+0000 7fa2affff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:20.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.285+0000 7fa2affff640 1 -- 172.21.15.12:0/1051573658 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8012660 con 0x7fa2b8137d50 2024-09-11T01:29:20.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.285+0000 7fa2affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.285+0000 7fa2affff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:20.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.285+0000 7fa2affff640 10 monclient: dump: 2024-09-11T01:29:20.365 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.365 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.366 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.366 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.366 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.366 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.366 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.366 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.366 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.366 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.285+0000 7fa2bdaf6640 5 monclient: authenticate success, global_id 4292 2024-09-11T01:29:20.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.285+0000 7fa2bdaf6640 10 monclient: _renew_subs 2024-09-11T01:29:20.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.285+0000 7fa2bdaf6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:20.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.285+0000 7fa2bdaf6640 1 -- 172.21.15.12:0/1051573658 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2b81211f0 con 0x7fa2b8137d50 2024-09-11T01:29:20.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.285+0000 7fa2b5ffb640 4 set_mon_vals no callback set 2024-09-11T01:29:20.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.285+0000 7fa2bdaf6640 10 monclient: _renew_subs 2024-09-11T01:29:20.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.285+0000 7fa2bdaf6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:20.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.285+0000 7fa2bdaf6640 1 -- 172.21.15.12:0/1051573658 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2b8120f10 con 0x7fa2b8137d50 2024-09-11T01:29:20.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.285+0000 7fcca759e640 10 monclient: build_initial_monmap 2024-09-11T01:29:20.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.285+0000 7fcca759e640 10 monclient: monmap: 2024-09-11T01:29:20.367 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:20.367 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:20.367 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:20.367 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:20.368 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:20.368 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.368 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:20.368 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:20.368 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:20.368 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.286+0000 7fcca759e640 1 Processor -- start 2024-09-11T01:29:20.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.286+0000 7fcca759e640 1 -- start start 2024-09-11T01:29:20.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fda5ed9d640 1 -- 172.21.15.12:0/1705444474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda60134ac0 msgr2=0x7fda601cda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:20.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fda5ed9d640 1 --2- 172.21.15.12:0/1705444474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda60134ac0 0x7fda601cda00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fda5800c780 tx=0x7fda5800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fa2affff640 1 -- 172.21.15.12:0/1051573658 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fa2a80128c0 con 0x7fa2b8137d50 2024-09-11T01:29:20.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.286+0000 7fcca759e640 10 monclient: init 2024-09-11T01:29:20.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca759e640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:20.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fa2bdaf6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:20.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fa2bdaf6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:20.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fbb21748640 1 -- 172.21.15.12:0/2116106454 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb1c138130 msgr2=0x7fbb1c136cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:20.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca759e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:29:20.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca759e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcca8068e90 0x7fcca81b2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca759e640 10 monclient(hunting): picked mon.noname-a con 0x7fcca8068e90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:20.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca759e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca80692a0 0x7fcca81b2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcca8068e90 0x7fcca81b2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca759e640 10 monclient(hunting): picked mon.noname-c con 0x7fcca80692a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:20.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca759e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcca806cb50 0x7fcca81c8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcca8068e90 0x7fcca81b2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54986/0 (socket says 172.21.15.12:54986) 2024-09-11T01:29:20.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca659c640 1 -- 172.21.15.12:0/1951048541 learned_addr learned my addr 172.21.15.12:0/1951048541 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:20.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca759e640 10 monclient(hunting): picked mon.noname-b con 0x7fcca806cb50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:20.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca759e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca759e640 1 -- 172.21.15.12:0/1951048541 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcca809ec80 con 0x7fcca8068e90 2024-09-11T01:29:20.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca759e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca759e640 1 -- 172.21.15.12:0/1951048541 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fcca8069730 con 0x7fcca806cb50 2024-09-11T01:29:20.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca759e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:20.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca759e640 1 -- 172.21.15.12:0/1951048541 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcca809ef60 con 0x7fcca80692a0 2024-09-11T01:29:20.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca759e640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:20.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca759e640 10 monclient(hunting): authenticate will time out at 766.729433s 2024-09-11T01:29:20.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca5d9b640 1 --2- 172.21.15.12:0/1951048541 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca80692a0 0x7fcca81b2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca5d9b640 10 monclient(hunting): get_auth_request con 0x7fcca80692a0 auth_method 0 2024-09-11T01:29:20.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fbb21748640 1 --2- 172.21.15.12:0/2116106454 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb1c138130 0x7fbb1c136cf0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c00c430 tx=0x7fbb0c00c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fbb127fc640 1 -- 172.21.15.12:0/2116106454 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c0119a0 con 0x7fbb1c138130 2024-09-11T01:29:20.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fda4effd640 1 -- 172.21.15.12:0/1705444474 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda58016020 con 0x7fda60134ac0 2024-09-11T01:29:20.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fda4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fda4effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:20.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fda4effd640 10 monclient: dump: 2024-09-11T01:29:20.373 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.374 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.374 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.374 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.374 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.374 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.374 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.374 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.374 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.374 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fda4effd640 10 monclient: _finish_auth 0 2024-09-11T01:29:20.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fda4effd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:20.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fda4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:20.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fda4effd640 1 -- 172.21.15.12:0/1705444474 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda5800ac60 con 0x7fda60134ac0 2024-09-11T01:29:20.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fda4effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:20.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fda5cd99640 4 set_mon_vals no callback set 2024-09-11T01:29:20.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fbb127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca659c640 10 monclient(hunting): get_auth_request con 0x7fcca8068e90 auth_method 0 2024-09-11T01:29:20.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:20.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:20.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:20.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca6d9d640 1 --2- 172.21.15.12:0/1951048541 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcca806cb50 0x7fcca81c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca6d9d640 10 monclient(hunting): get_auth_request con 0x7fcca806cb50 auth_method 0 2024-09-11T01:29:20.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:20.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:20.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:20.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca5d9b640 10 monclient(hunting): handle_auth_done global_id 4295 payload 824 2024-09-11T01:29:20.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca5d9b640 1 -- 172.21.15.12:0/1951048541 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcca806cb50 msgr2=0x7fcca81c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca5d9b640 1 --2- 172.21.15.12:0/1951048541 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcca806cb50 0x7fcca81c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca5d9b640 1 -- 172.21.15.12:0/1951048541 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcca8068e90 msgr2=0x7fcca81b2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca5d9b640 1 --2- 172.21.15.12:0/1951048541 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcca8068e90 0x7fcca81b2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca5d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:29:20.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca5d9b640 1 monclient: found mon.noname-c 2024-09-11T01:29:20.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:20.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:29:20.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca5d9b640 1 -- 172.21.15.12:0/1951048541 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcca81bbea0 con 0x7fcca80692a0 2024-09-11T01:29:20.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fa2bdaf6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:20.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fbb127fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:20.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fbb127fc640 10 monclient: dump: 2024-09-11T01:29:20.378 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.378 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.378 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.378 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.378 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.379 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.379 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.379 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.379 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.379 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fbb127fc640 10 monclient: _finish_auth 0 2024-09-11T01:29:20.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fbb127fc640 10 monclient: _check_auth_tickets 2024-09-11T01:29:20.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fbb127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:20.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fbb127fc640 1 -- 172.21.15.12:0/2116106454 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb0c017070 con 0x7fbb1c138130 2024-09-11T01:29:20.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fbb127fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:20.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fbb127fc640 1 -- 172.21.15.12:0/2116106454 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c011d80 con 0x7fbb1c138130 2024-09-11T01:29:20.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fbb127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fbb127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:20.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fbb127fc640 10 monclient: dump: 2024-09-11T01:29:20.380 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.380 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.380 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.380 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.380 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.380 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.380 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.380 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.380 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.380 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fbb13fff640 4 set_mon_vals no callback set 2024-09-11T01:29:20.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fbb241ad640 5 monclient: authenticate success, global_id 4255 2024-09-11T01:29:20.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fbb241ad640 10 monclient: _renew_subs 2024-09-11T01:29:20.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fbb241ad640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:20.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fbb241ad640 1 -- 172.21.15.12:0/2116106454 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb1c13ba30 con 0x7fbb1c138130 2024-09-11T01:29:20.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.288+0000 7fbb241ad640 10 monclient: _renew_subs 2024-09-11T01:29:20.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.288+0000 7fbb241ad640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:20.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.288+0000 7fbb241ad640 1 -- 172.21.15.12:0/2116106454 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb1c1404e0 con 0x7fbb1c138130 2024-09-11T01:29:20.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fcca6d9d640 1 --2- 172.21.15.12:0/1951048541 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcca806cb50 0x7fcca81c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:29:20.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.288+0000 7fcca659c640 1 --2- 172.21.15.12:0/1951048541 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcca8068e90 0x7fcca81b2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:20.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.288+0000 7fda4effd640 1 -- 172.21.15.12:0/1705444474 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda58012660 con 0x7fda60134ac0 2024-09-11T01:29:20.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.288+0000 7fa2affff640 1 --2- 172.21.15.12:0/1051573658 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa29803e260 0x7fa298040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.288+0000 7fa2affff640 1 -- 172.21.15.12:0/1051573658 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4402+0+0 (secure 0 0 0) 0x7fa2a8052b70 con 0x7fa2b8137d50 2024-09-11T01:29:20.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.287+0000 7fa2bdaf6640 1 -- 172.21.15.12:0/1051573658 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2b8138550 con 0x7fa2b8137d50 2024-09-11T01:29:20.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.288+0000 7fa2b6ffd640 1 --2- 172.21.15.12:0/1051573658 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa29803e260 0x7fa298040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.288+0000 7fda4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.288+0000 7fda4effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:20.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.288+0000 7fda4effd640 10 monclient: dump: 2024-09-11T01:29:20.383 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.383 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.383 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.383 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.383 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.383 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.383 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.383 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.383 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.383 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.288+0000 7fda64ebe640 5 monclient: authenticate success, global_id 4275 2024-09-11T01:29:20.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.288+0000 7fda64ebe640 10 monclient: _renew_subs 2024-09-11T01:29:20.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.288+0000 7fda64ebe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:20.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.288+0000 7fda64ebe640 1 -- 172.21.15.12:0/1705444474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda6013ea40 con 0x7fda60134ac0 2024-09-11T01:29:20.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.288+0000 7fda64ebe640 10 monclient: _renew_subs 2024-09-11T01:29:20.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.288+0000 7fda64ebe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:20.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.288+0000 7fda64ebe640 1 -- 172.21.15.12:0/1705444474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda6013c6e0 con 0x7fda60134ac0 2024-09-11T01:29:20.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.288+0000 7fa2b6ffd640 10 monclient: get_auth_request con 0x7fa29803e260 auth_method 0 2024-09-11T01:29:20.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.289+0000 7fbb127fc640 1 -- 172.21.15.12:0/2116106454 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fbb0c01c460 con 0x7fbb1c138130 2024-09-11T01:29:20.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.289+0000 7fda64ebe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:20.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.289+0000 7fda64ebe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:20.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.289+0000 7fda64ebe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:20.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.289+0000 7fbb127fc640 1 --2- 172.21.15.12:0/2116106454 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbb0003e290 0x7fbb00040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.289+0000 7fbb127fc640 1 -- 172.21.15.12:0/2116106454 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4402+0+0 (secure 0 0 0) 0x7fbb0c052720 con 0x7fbb1c138130 2024-09-11T01:29:20.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.289+0000 7fbb21f49640 1 --2- 172.21.15.12:0/2116106454 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbb0003e290 0x7fbb00040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.289+0000 7fbb21f49640 10 monclient: get_auth_request con 0x7fbb0003e290 auth_method 0 2024-09-11T01:29:20.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.290+0000 7fbb241ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:20.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.290+0000 7fbb241ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:20.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.290+0000 7fbb241ad640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:20.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.290+0000 7fbb241ad640 1 -- 172.21.15.12:0/2116106454 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb1c09d190 con 0x7fbb1c138130 2024-09-11T01:29:20.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.290+0000 7fcca5d9b640 1 -- 172.21.15.12:0/1951048541 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca80692a0 msgr2=0x7fcca81b2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:20.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.290+0000 7fcca5d9b640 1 --2- 172.21.15.12:0/1951048541 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca80692a0 0x7fcca81b2d80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c00c440 tx=0x7fcc9c00c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.290+0000 7fda4effd640 1 -- 172.21.15.12:0/1705444474 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fda580128c0 con 0x7fda60134ac0 2024-09-11T01:29:20.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.290+0000 7fda4effd640 1 --2- 172.21.15.12:0/1705444474 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda4003e260 0x7fda40040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.289+0000 7fda64ebe640 1 -- 172.21.15.12:0/1705444474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda6013e1a0 con 0x7fda60134ac0 2024-09-11T01:29:20.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.290+0000 7fda4effd640 1 -- 172.21.15.12:0/1705444474 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4402+0+0 (secure 0 0 0) 0x7fda58052dd0 con 0x7fda60134ac0 2024-09-11T01:29:20.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.290+0000 7fda5e59c640 1 --2- 172.21.15.12:0/1705444474 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda4003e260 0x7fda40040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.290+0000 7fda5e59c640 10 monclient: get_auth_request con 0x7fda4003e260 auth_method 0 2024-09-11T01:29:20.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.291+0000 7fa2b6ffd640 1 --2- 172.21.15.12:0/1051573658 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa29803e260 0x7fa298040720 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa2b813c1a0 tx=0x7fa2a0009240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.292+0000 7fbb21f49640 1 --2- 172.21.15.12:0/2116106454 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbb0003e290 0x7fbb00040750 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbb18007c90 tx=0x7fbb18007be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.293+0000 7fda5e59c640 1 --2- 172.21.15.12:0/1705444474 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda4003e260 0x7fda40040720 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fda500094a0 tx=0x7fda500093f0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.294+0000 7fa2affff640 1 -- 172.21.15.12:0/1051573658 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa2a8020070 con 0x7fa2b8137d50 2024-09-11T01:29:20.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.294+0000 7fa2affff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:20.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.294+0000 7fa2affff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:20.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.294+0000 7fcc96ffd640 1 -- 172.21.15.12:0/1951048541 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c007070 con 0x7fcca80692a0 2024-09-11T01:29:20.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.294+0000 7fcc96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.294+0000 7fcc96ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:20.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.294+0000 7fcc96ffd640 10 monclient: dump: 2024-09-11T01:29:20.391 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.392 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.392 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.392 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.392 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.392 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.392 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.392 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.392 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.392 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.294+0000 7fcc96ffd640 10 monclient: _finish_auth 0 2024-09-11T01:29:20.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.294+0000 7fcc96ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:20.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.294+0000 7fcc96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:20.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.294+0000 7fcc96ffd640 1 -- 172.21.15.12:0/1951048541 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc9c007210 con 0x7fcca80692a0 2024-09-11T01:29:20.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.294+0000 7fcca759e640 5 monclient: authenticate success, global_id 4295 2024-09-11T01:29:20.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.294+0000 7fcc96ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:20.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.294+0000 7fcc96ffd640 1 -- 172.21.15.12:0/1951048541 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c007600 con 0x7fcca80692a0 2024-09-11T01:29:20.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.294+0000 7fcc96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:20.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.294+0000 7fcc96ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:20.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.294+0000 7fcc96ffd640 10 monclient: dump: 2024-09-11T01:29:20.393 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:20.393 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:20.393 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.393 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:20.393 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:20.393 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:20.393 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:20.394 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:20.394 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:20.394 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:20.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.295+0000 7fcca759e640 10 monclient: _renew_subs 2024-09-11T01:29:20.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.295+0000 7fcca759e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:20.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.295+0000 7fcca759e640 1 -- 172.21.15.12:0/1951048541 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcca80a04c0 con 0x7fcca80692a0 2024-09-11T01:29:20.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.294+0000 7fcca4d99640 4 set_mon_vals no callback set 2024-09-11T01:29:20.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.295+0000 7fcca759e640 10 monclient: _renew_subs 2024-09-11T01:29:20.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.295+0000 7fcca759e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:20.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.295+0000 7fcca759e640 1 -- 172.21.15.12:0/1951048541 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcca81b3d30 con 0x7fcca80692a0 2024-09-11T01:29:20.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.296+0000 7fda4effd640 1 -- 172.21.15.12:0/1705444474 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fda58020030 con 0x7fda60134ac0 2024-09-11T01:29:20.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.296+0000 7fda4effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:20.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.296+0000 7fda4effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:20.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.296+0000 7fcca759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:20.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.296+0000 7fcca759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:20.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.296+0000 7fcca759e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:20.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.296+0000 7fcca759e640 1 -- 172.21.15.12:0/1951048541 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcca80008d0 con 0x7fcca80692a0 2024-09-11T01:29:20.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.296+0000 7fbb127fc640 1 -- 172.21.15.12:0/2116106454 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbb0c021020 con 0x7fbb1c138130 2024-09-11T01:29:20.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.296+0000 7fbb127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:20.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.296+0000 7fbb127fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:20.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.296+0000 7fcc96ffd640 1 -- 172.21.15.12:0/1951048541 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fcc9c01b400 con 0x7fcca80692a0 2024-09-11T01:29:20.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.297+0000 7fcc96ffd640 1 --2- 172.21.15.12:0/1951048541 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcc8803e290 0x7fcc88040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:20.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.297+0000 7fcc96ffd640 1 -- 172.21.15.12:0/1951048541 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4402+0+0 (secure 0 0 0) 0x7fcc9c052770 con 0x7fcca80692a0 2024-09-11T01:29:20.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.297+0000 7fcca659c640 1 --2- 172.21.15.12:0/1951048541 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcc8803e290 0x7fcc88040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:20.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.297+0000 7fcca659c640 10 monclient: get_auth_request con 0x7fcc8803e290 auth_method 0 2024-09-11T01:29:20.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.300+0000 7fcca659c640 1 --2- 172.21.15.12:0/1951048541 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcc8803e290 0x7fcc88040750 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcca81b62a0 tx=0x7fcc98009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:20.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.301+0000 7fcc96ffd640 1 -- 172.21.15.12:0/1951048541 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fcca80a04c0 con 0x7fcca80692a0 2024-09-11T01:29:20.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.301+0000 7fcc96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:20.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.301+0000 7fcc96ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:20.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.455+0000 7f3635791640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-11T01:29:20.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.455+0000 7f3635791640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-11T01:29:20.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.455+0000 7f3635791640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:20.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.455+0000 7f3635791640 1 -- 172.21.15.12:0/758197832 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f3630074a40 con 0x7f363013d280 2024-09-11T01:29:20.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.456+0000 7f36177fe640 1 -- 172.21.15.12:0/758197832 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f362803d420 con 0x7f363013d280 2024-09-11T01:29:20.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.456+0000 7f36177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-11T01:29:20.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.456+0000 7f36177fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:20.458 INFO:teuthology.orchestra.run.smithi012.stdout:51539607554 2024-09-11T01:29:20.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.458+0000 7f3635791640 1 -- 172.21.15.12:0/758197832 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f361c03e260 msgr2=0x7f361c040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.458+0000 7f3635791640 1 --2- 172.21.15.12:0/758197832 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f361c03e260 0x7f361c040720 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3620009ac0 tx=0x7f3620009190 comp rx=0 tx=0).stop 2024-09-11T01:29:20.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.458+0000 7f3635791640 10 monclient: shutdown 2024-09-11T01:29:20.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.458+0000 7f3635791640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:20.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.458+0000 7f3635791640 1 -- 172.21.15.12:0/758197832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f363013d280 msgr2=0x7f363013a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.458+0000 7f3635791640 1 --2- 172.21.15.12:0/758197832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f363013d280 0x7f363013a200 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3628033000 tx=0x7f3628004a40 comp rx=0 tx=0).stop 2024-09-11T01:29:20.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.458+0000 7f3635791640 1 -- 172.21.15.12:0/758197832 shutdown_connections 2024-09-11T01:29:20.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.458+0000 7f3635791640 1 --2- 172.21.15.12:0/758197832 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f361c03e260 0x7f361c040720 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.458+0000 7f3635791640 1 --2- 172.21.15.12:0/758197832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f363013d280 0x7f363013a200 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.458+0000 7f3635791640 1 --2- 172.21.15.12:0/758197832 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3630138130 0x7f3630136e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.458+0000 7f3635791640 1 --2- 172.21.15.12:0/758197832 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3630137d50 0x7f36301cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.458+0000 7f3635791640 1 -- 172.21.15.12:0/758197832 >> 172.21.15.12:0/758197832 conn(0x7f36300a7020 msgr2=0x7f363013e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:20.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.458+0000 7f3635791640 1 -- 172.21.15.12:0/758197832 shutdown_connections 2024-09-11T01:29:20.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.458+0000 7f3635791640 1 -- 172.21.15.12:0/758197832 wait complete. 2024-09-11T01:29:20.471 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607554 for osd.5 2024-09-11T01:29:20.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.490+0000 7fcca759e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-11T01:29:20.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.490+0000 7fcca759e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-11T01:29:20.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.490+0000 7fcca759e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:20.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.490+0000 7fcca759e640 1 -- 172.21.15.12:0/1951048541 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fcca81b4890 con 0x7fcca80692a0 2024-09-11T01:29:20.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.491+0000 7fcc96ffd640 1 -- 172.21.15.12:0/1951048541 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcc9c010d10 con 0x7fcca80692a0 2024-09-11T01:29:20.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.491+0000 7fcc96ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-11T01:29:20.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.491+0000 7fcc96ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:20.493 INFO:teuthology.orchestra.run.smithi012.stdout:47244640259 2024-09-11T01:29:20.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.493+0000 7fcca759e640 1 -- 172.21.15.12:0/1951048541 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcc8803e290 msgr2=0x7fcc88040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.493+0000 7fcca759e640 1 --2- 172.21.15.12:0/1951048541 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcc8803e290 0x7fcc88040750 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcca81b62a0 tx=0x7fcc98009190 comp rx=0 tx=0).stop 2024-09-11T01:29:20.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.493+0000 7fcca759e640 10 monclient: shutdown 2024-09-11T01:29:20.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.493+0000 7fcca759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:20.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.493+0000 7fcca759e640 1 -- 172.21.15.12:0/1951048541 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca80692a0 msgr2=0x7fcca81b2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.493+0000 7fcca759e640 1 --2- 172.21.15.12:0/1951048541 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca80692a0 0x7fcca81b2d80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c00c440 tx=0x7fcc9c00c900 comp rx=0 tx=0).stop 2024-09-11T01:29:20.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.493+0000 7fcca759e640 1 -- 172.21.15.12:0/1951048541 shutdown_connections 2024-09-11T01:29:20.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.493+0000 7fcca759e640 1 --2- 172.21.15.12:0/1951048541 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcca806cb50 0x7fcca81c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.493+0000 7fcca759e640 1 --2- 172.21.15.12:0/1951048541 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca80692a0 0x7fcca81b2d80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.493+0000 7fcca759e640 1 --2- 172.21.15.12:0/1951048541 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcc8803e290 0x7fcc88040750 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.493+0000 7fcca759e640 1 --2- 172.21.15.12:0/1951048541 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcca8068e90 0x7fcca81b2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.493+0000 7fcca759e640 1 -- 172.21.15.12:0/1951048541 >> 172.21.15.12:0/1951048541 conn(0x7fcca80a7660 msgr2=0x7fcca809c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:20.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.493+0000 7fcca759e640 1 -- 172.21.15.12:0/1951048541 shutdown_connections 2024-09-11T01:29:20.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.493+0000 7fcca759e640 1 -- 172.21.15.12:0/1951048541 wait complete. 2024-09-11T01:29:20.505 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.2 2024-09-11T01:29:20.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.597+0000 7fe8405ec640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-11T01:29:20.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.597+0000 7fe8405ec640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-11T01:29:20.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.597+0000 7fe8405ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:20.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.597+0000 7fe8405ec640 1 -- 172.21.15.12:0/4138121050 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fe83813b310 con 0x7fe838137d50 2024-09-11T01:29:20.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.598+0000 7fe82affd640 1 -- 172.21.15.12:0/4138121050 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe82c020020 con 0x7fe838137d50 2024-09-11T01:29:20.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.598+0000 7fe82affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-11T01:29:20.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.598+0000 7fe82affd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:20.600 INFO:teuthology.orchestra.run.smithi012.stdout:51539607554 2024-09-11T01:29:20.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.600+0000 7fe8405ec640 1 -- 172.21.15.12:0/4138121050 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe81c03e1f0 msgr2=0x7fe81c0406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.600+0000 7fe8405ec640 1 --2- 172.21.15.12:0/4138121050 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe81c03e1f0 0x7fe81c0406b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe83809c070 tx=0x7fe83402e000 comp rx=0 tx=0).stop 2024-09-11T01:29:20.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.600+0000 7fe8405ec640 10 monclient: shutdown 2024-09-11T01:29:20.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.600+0000 7fe8405ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:20.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.600+0000 7fe8405ec640 1 -- 172.21.15.12:0/4138121050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe838137d50 msgr2=0x7fe838136b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.600+0000 7fe8405ec640 1 --2- 172.21.15.12:0/4138121050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe838137d50 0x7fe838136b70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe82c00c460 tx=0x7fe82c00c920 comp rx=0 tx=0).stop 2024-09-11T01:29:20.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.600+0000 7fe8405ec640 1 -- 172.21.15.12:0/4138121050 shutdown_connections 2024-09-11T01:29:20.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.600+0000 7fe8405ec640 1 --2- 172.21.15.12:0/4138121050 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe83813d280 0x7fe838139260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.600+0000 7fe8405ec640 1 --2- 172.21.15.12:0/4138121050 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe838138130 0x7fe8381370b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.600+0000 7fe8405ec640 1 --2- 172.21.15.12:0/4138121050 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe81c03e1f0 0x7fe81c0406b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.600+0000 7fe8405ec640 1 --2- 172.21.15.12:0/4138121050 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe838137d50 0x7fe838136b70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.600+0000 7fe8405ec640 1 -- 172.21.15.12:0/4138121050 >> 172.21.15.12:0/4138121050 conn(0x7fe8380a7020 msgr2=0x7fe8380a2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:20.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.600+0000 7fe8405ec640 1 -- 172.21.15.12:0/4138121050 shutdown_connections 2024-09-11T01:29:20.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.600+0000 7fe8405ec640 1 -- 172.21.15.12:0/4138121050 wait complete. 2024-09-11T01:29:20.612 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607554 for osd.4 2024-09-11T01:29:20.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.612+0000 7fda64ebe640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-11T01:29:20.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.612+0000 7fda64ebe640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-11T01:29:20.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.612+0000 7fda64ebe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:20.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.612+0000 7fda64ebe640 1 -- 172.21.15.12:0/1705444474 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fda60074560 con 0x7fda60134ac0 2024-09-11T01:29:20.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.612+0000 7fda4effd640 1 -- 172.21.15.12:0/1705444474 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fda600a6840 con 0x7fda60134ac0 2024-09-11T01:29:20.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.612+0000 7fda4effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-11T01:29:20.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.612+0000 7fda4effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:20.614 INFO:teuthology.orchestra.run.smithi012.stdout:51539607554 2024-09-11T01:29:20.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.614+0000 7fda64ebe640 1 -- 172.21.15.12:0/1705444474 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda4003e260 msgr2=0x7fda40040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.614+0000 7fda64ebe640 1 --2- 172.21.15.12:0/1705444474 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda4003e260 0x7fda40040720 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fda500094a0 tx=0x7fda500093f0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.614+0000 7fda64ebe640 10 monclient: shutdown 2024-09-11T01:29:20.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.614+0000 7fda64ebe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:20.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.614+0000 7fda64ebe640 1 -- 172.21.15.12:0/1705444474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda60134ac0 msgr2=0x7fda601cda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.614+0000 7fda64ebe640 1 --2- 172.21.15.12:0/1705444474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda60134ac0 0x7fda601cda00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fda5800c780 tx=0x7fda5800cc40 comp rx=0 tx=0).stop 2024-09-11T01:29:20.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.615+0000 7fda64ebe640 1 -- 172.21.15.12:0/1705444474 shutdown_connections 2024-09-11T01:29:20.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.615+0000 7fda64ebe640 1 --2- 172.21.15.12:0/1705444474 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda4003e260 0x7fda40040720 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.615+0000 7fda64ebe640 1 --2- 172.21.15.12:0/1705444474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda60134ac0 0x7fda601cda00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.615+0000 7fda64ebe640 1 --2- 172.21.15.12:0/1705444474 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda600a06d0 0x7fda601cd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.615+0000 7fda64ebe640 1 --2- 172.21.15.12:0/1705444474 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda60074180 0x7fda601ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.615+0000 7fda64ebe640 1 -- 172.21.15.12:0/1705444474 >> 172.21.15.12:0/1705444474 conn(0x7fda600a36c0 msgr2=0x7fda600735f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:20.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.615+0000 7fda64ebe640 1 -- 172.21.15.12:0/1705444474 shutdown_connections 2024-09-11T01:29:20.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.615+0000 7fda64ebe640 1 -- 172.21.15.12:0/1705444474 wait complete. 2024-09-11T01:29:20.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.616+0000 7fbb241ad640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-11T01:29:20.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.616+0000 7fbb241ad640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-11T01:29:20.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.616+0000 7fbb241ad640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:20.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.616+0000 7fbb241ad640 1 -- 172.21.15.12:0/2116106454 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fbb1c074a40 con 0x7fbb1c138130 2024-09-11T01:29:20.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.617+0000 7fbb127fc640 1 -- 172.21.15.12:0/2116106454 <== mon.1 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbb0c01ac60 con 0x7fbb1c138130 2024-09-11T01:29:20.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.617+0000 7fbb127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-11T01:29:20.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.617+0000 7fbb127fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:20.619 INFO:teuthology.orchestra.run.smithi012.stdout:51539607554 2024-09-11T01:29:20.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.619+0000 7fbb241ad640 1 -- 172.21.15.12:0/2116106454 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbb0003e290 msgr2=0x7fbb00040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.619+0000 7fbb241ad640 1 --2- 172.21.15.12:0/2116106454 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbb0003e290 0x7fbb00040750 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbb18007c90 tx=0x7fbb18007be0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.619+0000 7fbb241ad640 10 monclient: shutdown 2024-09-11T01:29:20.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.619+0000 7fbb241ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:20.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.619+0000 7fbb241ad640 1 -- 172.21.15.12:0/2116106454 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb1c138130 msgr2=0x7fbb1c136cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.619+0000 7fbb241ad640 1 --2- 172.21.15.12:0/2116106454 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb1c138130 0x7fbb1c136cf0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c00c430 tx=0x7fbb0c00c8f0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.619+0000 7fbb241ad640 1 -- 172.21.15.12:0/2116106454 shutdown_connections 2024-09-11T01:29:20.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.619+0000 7fbb241ad640 1 --2- 172.21.15.12:0/2116106454 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbb1c13e290 0x7fbb1c139ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.619+0000 7fbb241ad640 1 --2- 172.21.15.12:0/2116106454 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb1c138130 0x7fbb1c136cf0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.619+0000 7fbb241ad640 1 --2- 172.21.15.12:0/2116106454 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbb0003e290 0x7fbb00040750 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.619+0000 7fbb241ad640 1 --2- 172.21.15.12:0/2116106454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb1c137d50 0x7fbb1c13c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.619+0000 7fbb241ad640 1 -- 172.21.15.12:0/2116106454 >> 172.21.15.12:0/2116106454 conn(0x7fbb1c0a7020 msgr2=0x7fbb1c13f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:20.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.619+0000 7fbb241ad640 1 -- 172.21.15.12:0/2116106454 shutdown_connections 2024-09-11T01:29:20.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.619+0000 7fbb241ad640 1 -- 172.21.15.12:0/2116106454 wait complete. 2024-09-11T01:29:20.628 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607554 for osd.0 2024-09-11T01:29:20.632 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607554 for osd.3 2024-09-11T01:29:20.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.630+0000 7fa2bdaf6640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-11T01:29:20.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.630+0000 7fa2bdaf6640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-11T01:29:20.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.630+0000 7fa2bdaf6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:20.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.630+0000 7fa2bdaf6640 1 -- 172.21.15.12:0/1051573658 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fa2b80a1710 con 0x7fa2b8137d50 2024-09-11T01:29:20.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.631+0000 7fa2affff640 1 -- 172.21.15.12:0/1051573658 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa2b8137ac0 con 0x7fa2b8137d50 2024-09-11T01:29:20.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.631+0000 7fa2affff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-11T01:29:20.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.631+0000 7fa2affff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:20.633 INFO:teuthology.orchestra.run.smithi012.stdout:51539607553 2024-09-11T01:29:20.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.633+0000 7fa2bdaf6640 1 -- 172.21.15.12:0/1051573658 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa29803e260 msgr2=0x7fa298040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.633+0000 7fa2bdaf6640 1 --2- 172.21.15.12:0/1051573658 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa29803e260 0x7fa298040720 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa2b813c1a0 tx=0x7fa2a0009240 comp rx=0 tx=0).stop 2024-09-11T01:29:20.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.633+0000 7fa2bdaf6640 10 monclient: shutdown 2024-09-11T01:29:20.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.633+0000 7fa2bdaf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:20.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.633+0000 7fa2bdaf6640 1 -- 172.21.15.12:0/1051573658 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa2b8137d50 msgr2=0x7fa2b81ccd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:20.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.633+0000 7fa2bdaf6640 1 --2- 172.21.15.12:0/1051573658 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa2b8137d50 0x7fa2b81ccd40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa2a800c7c0 tx=0x7fa2a800cc80 comp rx=0 tx=0).stop 2024-09-11T01:29:20.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.633+0000 7fa2bdaf6640 1 -- 172.21.15.12:0/1051573658 shutdown_connections 2024-09-11T01:29:20.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.633+0000 7fa2bdaf6640 1 --2- 172.21.15.12:0/1051573658 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa29803e260 0x7fa298040720 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.633+0000 7fa2bdaf6640 1 --2- 172.21.15.12:0/1051573658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa2b813d280 0x7fa2b813a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.633+0000 7fa2bdaf6640 1 --2- 172.21.15.12:0/1051573658 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa2b8138130 0x7fa2b8136d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.633+0000 7fa2bdaf6640 1 --2- 172.21.15.12:0/1051573658 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa2b8137d50 0x7fa2b81ccd40 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:20.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.633+0000 7fa2bdaf6640 1 -- 172.21.15.12:0/1051573658 >> 172.21.15.12:0/1051573658 conn(0x7fa2b80a7020 msgr2=0x7fa2b813e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:20.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.633+0000 7fa2bdaf6640 1 -- 172.21.15.12:0/1051573658 shutdown_connections 2024-09-11T01:29:20.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:20.633+0000 7fa2bdaf6640 1 -- 172.21.15.12:0/1051573658 wait complete. 2024-09-11T01:29:20.645 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607555 got 51539607553 for osd.1 2024-09-11T01:29:21.472 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-11T01:29:21.506 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T01:29:21.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.588+0000 7f6fbca2a640 10 monclient: get_monmap_and_config 2024-09-11T01:29:21.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.588+0000 7f6fbca2a640 10 monclient: build_initial_monmap 2024-09-11T01:29:21.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.588+0000 7f6fbca2a640 10 monclient: monmap: 2024-09-11T01:29:21.590 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:21.590 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.590 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:21.589489+0000 2024-09-11T01:29:21.591 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:21.589489+0000 2024-09-11T01:29:21.591 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:21.591 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.591 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:21.591 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:21.591 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:21.591 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:21.591 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:21.591 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.589+0000 7f6fbca2a640 1 Processor -- start 2024-09-11T01:29:21.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.589+0000 7f6fbca2a640 1 -- start start 2024-09-11T01:29:21.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.589+0000 7f6fbca2a640 10 monclient: init 2024-09-11T01:29:21.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fbca2a640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:21.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fbca2a640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-11T01:29:21.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fbca2a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6fb8137d50 0x7f6fb809d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fbca2a640 10 monclient(hunting): picked mon.noname-b con 0x7f6fb8137d50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:21.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fbca2a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6fb8138130 0x7f6fb8138550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fbca2a640 10 monclient(hunting): picked mon.noname-a con 0x7f6fb8138130 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:21.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fbca2a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6fb8136cd0 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:21.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fbca2a640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fb809ba20 con 0x7f6fb8136cd0 2024-09-11T01:29:21.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fbca2a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fbca2a640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f6fb809c650 con 0x7f6fb8138130 2024-09-11T01:29:21.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fbca2a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fbca2a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6fb809c370 con 0x7f6fb8137d50 2024-09-11T01:29:21.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fbca2a640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:21.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fb659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6fb8137d50 0x7f6fb809d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fbca2a640 10 monclient(hunting): authenticate will time out at 768.032515s 2024-09-11T01:29:21.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6faffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6fb8138130 0x7f6fb8138550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fb659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6fb8137d50 0x7f6fb809d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51710/0 (socket says 172.21.15.12:51710) 2024-09-11T01:29:21.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fb659c640 1 -- 172.21.15.12:0/1969467845 learned_addr learned my addr 172.21.15.12:0/1969467845 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:21.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fb659c640 10 monclient(hunting): get_auth_request con 0x7f6fb8137d50 auth_method 0 2024-09-11T01:29:21.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fb659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fb659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fb659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6faffff640 10 monclient(hunting): get_auth_request con 0x7f6fb8138130 auth_method 0 2024-09-11T01:29:21.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6faffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6faffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6faffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6faffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:21.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6faffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:21.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6faffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:21.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fb659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:21.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fb659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:21.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fb659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:21.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fb659c640 10 monclient(hunting): handle_auth_done global_id 4298 payload 274 2024-09-11T01:29:21.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fb659c640 1 -- 172.21.15.12:0/1969467845 >> v2:172.21.15.12:3300/0 conn(0x7f6fb8138130 msgr2=0x7f6fb8138550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:21.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fb659c640 1 --2- 172.21.15.12:0/1969467845 >> v2:172.21.15.12:3300/0 conn(0x7f6fb8138130 0x7f6fb8138550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fb659c640 1 -- 172.21.15.12:0/1969467845 >> v1:172.21.15.12:6789/0 conn(0x7f6fb8136cd0 legacy=0x7f6fb8137120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:21.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fb659c640 10 monclient: _finish_hunting 0 2024-09-11T01:29:21.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fb659c640 1 monclient: found mon.noname-b 2024-09-11T01:29:21.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fb659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:21.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fb659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:29:21.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6fb659c640 1 -- 172.21.15.12:0/1969467845 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fb80a6ce0 con 0x7f6fb8137d50 2024-09-11T01:29:21.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.590+0000 7f6faffff640 1 --2- 172.21.15.12:0/1969467845 >> v2:172.21.15.12:3300/0 conn(0x7f6fb8138130 0x7f6fb8138550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:21.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.593+0000 7f6fb659c640 1 -- 172.21.15.12:0/1969467845 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6fb8137d50 msgr2=0x7f6fb809d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:21.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.593+0000 7f6fb659c640 1 --2- 172.21.15.12:0/1969467845 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6fb8137d50 0x7f6fb809d110 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6fb809c7d0 tx=0x7f6fa802a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0224541d35c4179 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:21.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.593+0000 7f6fb5d9b640 1 -- 172.21.15.12:0/1969467845 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8037070 con 0x7f6fb8137d50 2024-09-11T01:29:21.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.593+0000 7f6fb5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:21.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.593+0000 7f6fb5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:21.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.593+0000 7f6fb5d9b640 10 monclient: dump: 2024-09-11T01:29:21.601 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:21.601 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.602 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.602 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.602 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:21.602 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.602 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:21.602 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:21.602 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:21.602 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.593+0000 7f6fb5d9b640 10 monclient: _finish_auth 0 2024-09-11T01:29:21.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.593+0000 7f6fb5d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:29:21.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.593+0000 7f6fb5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:21.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.593+0000 7f6fbca2a640 5 monclient: authenticate success, global_id 4298 2024-09-11T01:29:21.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.593+0000 7f6fb5d9b640 1 -- 172.21.15.12:0/1969467845 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fa8032640 con 0x7f6fb8137d50 2024-09-11T01:29:21.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.593+0000 7f6fbca2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:21.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.593+0000 7f6fb5d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:21.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.594+0000 7f6fbca2a640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:21.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.594+0000 7f6fbca2a640 4 set_mon_vals no callback set 2024-09-11T01:29:21.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.594+0000 7f6fb5d9b640 1 -- 172.21.15.12:0/1969467845 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8032a10 con 0x7f6fb8137d50 2024-09-11T01:29:21.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.594+0000 7f6fb5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:21.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.594+0000 7f6fb5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:21.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.594+0000 7f6fb5d9b640 10 monclient: dump: 2024-09-11T01:29:21.603 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:21.603 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.603 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.603 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.603 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:21.603 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.603 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:21.604 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:21.604 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:21.604 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.594+0000 7f6fbca2a640 10 monclient: shutdown 2024-09-11T01:29:21.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.594+0000 7f6fbca2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:21.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.594+0000 7f6fbca2a640 1 -- 172.21.15.12:0/1969467845 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6fb8137d50 msgr2=0x7f6fb809d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:21.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.594+0000 7f6fbca2a640 1 --2- 172.21.15.12:0/1969467845 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6fb8137d50 0x7f6fb809d110 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6fb809c7d0 tx=0x7f6fa802a110 comp rx=0 tx=0).stop 2024-09-11T01:29:21.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.594+0000 7f6fbca2a640 1 -- 172.21.15.12:0/1969467845 shutdown_connections 2024-09-11T01:29:21.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.594+0000 7f6fbca2a640 1 --2- 172.21.15.12:0/1969467845 >> v2:172.21.15.12:3300/0 conn(0x7f6fb8138130 0x7f6fb8138550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.594+0000 7f6fbca2a640 1 --2- 172.21.15.12:0/1969467845 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6fb8137d50 0x7f6fb809d110 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.594+0000 7f6fbca2a640 1 -- 172.21.15.12:0/1969467845 >> 172.21.15.12:0/1969467845 conn(0x7f6fb80a7020 msgr2=0x7f6fb80a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:21.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.594+0000 7f6fbca2a640 1 -- 172.21.15.12:0/1969467845 shutdown_connections 2024-09-11T01:29:21.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.594+0000 7f6fbca2a640 1 -- 172.21.15.12:0/1969467845 wait complete. 2024-09-11T01:29:21.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.595+0000 7f6fbca2a640 10 monclient: build_initial_monmap 2024-09-11T01:29:21.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.595+0000 7f6fbca2a640 10 monclient: monmap: 2024-09-11T01:29:21.605 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:21.605 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:21.605 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:21.605 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:21.605 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:21.605 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.605 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:21.605 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:21.605 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:21.605 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.595+0000 7f6fbca2a640 1 Processor -- start 2024-09-11T01:29:21.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fbca2a640 1 -- start start 2024-09-11T01:29:21.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fbca2a640 10 monclient: init 2024-09-11T01:29:21.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fbca2a640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:21.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fbca2a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:29:21.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fbca2a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6fb8136cd0 0x7f6fb81cd620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fbca2a640 10 monclient(hunting): picked mon.noname-a con 0x7f6fb8136cd0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:21.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fbca2a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6fb8137d50 0x7f6fb8123590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fbca2a640 10 monclient(hunting): picked mon.noname-b con 0x7f6fb8137d50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:21.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fbca2a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6fb8138130 0x7f6fb8123ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fbca2a640 10 monclient(hunting): picked mon.noname-c con 0x7f6fb8138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:21.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fbca2a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fbca2a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6fb809d650 con 0x7f6fb8136cd0 2024-09-11T01:29:21.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fbca2a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fbca2a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6fb806f710 con 0x7f6fb8137d50 2024-09-11T01:29:21.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fbca2a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fbca2a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6fb809d7d0 con 0x7f6fb8138130 2024-09-11T01:29:21.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fbca2a640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:21.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fbca2a640 10 monclient(hunting): authenticate will time out at 768.038515s 2024-09-11T01:29:21.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fb6d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6fb8138130 0x7f6fb8123ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fb659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6fb8136cd0 0x7f6fb81cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fb659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6fb8136cd0 0x7f6fb81cd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54998/0 (socket says 172.21.15.12:54998) 2024-09-11T01:29:21.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fb6d9d640 10 monclient(hunting): get_auth_request con 0x7f6fb8138130 auth_method 0 2024-09-11T01:29:21.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fb659c640 1 -- 172.21.15.12:0/611486321 learned_addr learned my addr 172.21.15.12:0/611486321 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:21.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fb6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fb6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fb6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fb659c640 10 monclient(hunting): get_auth_request con 0x7f6fb8136cd0 auth_method 0 2024-09-11T01:29:21.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fb659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fb659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fb659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6faffff640 1 --2- 172.21.15.12:0/611486321 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6fb8137d50 0x7f6fb8123590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fb6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:21.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fb6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:21.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fb6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:21.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6faffff640 10 monclient(hunting): get_auth_request con 0x7f6fb8137d50 auth_method 0 2024-09-11T01:29:21.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6faffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6faffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6faffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fb659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:21.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fb659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:21.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fb659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:21.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6faffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:21.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6faffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:21.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6faffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:21.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.596+0000 7f6fb6d9d640 10 monclient(hunting): handle_auth_done global_id 4301 payload 824 2024-09-11T01:29:21.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.597+0000 7f6fb6d9d640 1 -- 172.21.15.12:0/611486321 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6fb8137d50 msgr2=0x7f6fb8123590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:21.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.597+0000 7f6fb6d9d640 1 --2- 172.21.15.12:0/611486321 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6fb8137d50 0x7f6fb8123590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.597+0000 7f6fb6d9d640 1 -- 172.21.15.12:0/611486321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6fb8136cd0 msgr2=0x7f6fb81cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:21.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.597+0000 7f6fb6d9d640 1 --2- 172.21.15.12:0/611486321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6fb8136cd0 0x7f6fb81cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.597+0000 7f6fb6d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:29:21.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.597+0000 7f6fb6d9d640 1 monclient: found mon.noname-c 2024-09-11T01:29:21.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.597+0000 7f6fb6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:21.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.597+0000 7f6fb6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:29:21.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.597+0000 7f6fb6d9d640 1 -- 172.21.15.12:0/611486321 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fb8136970 con 0x7f6fb8138130 2024-09-11T01:29:21.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.597+0000 7f6fb659c640 1 --2- 172.21.15.12:0/611486321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6fb8136cd0 0x7f6fb81cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:21.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.599+0000 7f6fb6d9d640 1 -- 172.21.15.12:0/611486321 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6fb8138130 msgr2=0x7f6fb8123ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:21.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.599+0000 7f6fb6d9d640 1 --2- 172.21.15.12:0/611486321 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6fb8138130 0x7f6fb8123ad0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6fb000c780 tx=0x7f6fb000cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:21.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.599+0000 7f6faeffd640 1 -- 172.21.15.12:0/611486321 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0016020 con 0x7f6fb8138130 2024-09-11T01:29:21.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.599+0000 7f6faeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:21.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.599+0000 7f6faeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:21.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.599+0000 7f6faeffd640 10 monclient: dump: 2024-09-11T01:29:21.610 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:21.610 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.610 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.610 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.610 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:21.610 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.610 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:21.610 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:21.610 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:21.610 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.599+0000 7f6faeffd640 10 monclient: _finish_auth 0 2024-09-11T01:29:21.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.599+0000 7f6faeffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:21.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.599+0000 7f6faeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:21.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.599+0000 7f6faeffd640 1 -- 172.21.15.12:0/611486321 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fb000ac00 con 0x7f6fb8138130 2024-09-11T01:29:21.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.599+0000 7f6faeffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:21.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.599+0000 7f6faeffd640 1 -- 172.21.15.12:0/611486321 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb000ce30 con 0x7f6fb8138130 2024-09-11T01:29:21.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.599+0000 7f6faeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:21.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.599+0000 7f6faeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:21.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.599+0000 7f6faeffd640 10 monclient: dump: 2024-09-11T01:29:21.611 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:21.611 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.611 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.611 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.611 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:21.612 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.612 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:21.612 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:21.612 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:21.612 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.599+0000 7f6fbca2a640 5 monclient: authenticate success, global_id 4301 2024-09-11T01:29:21.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.599+0000 7f6fbca2a640 10 monclient: _renew_subs 2024-09-11T01:29:21.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.599+0000 7f6fbca2a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:21.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.599+0000 7f6fbca2a640 1 -- 172.21.15.12:0/611486321 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fb80a1710 con 0x7f6fb8138130 2024-09-11T01:29:21.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.599+0000 7f6fb559a640 4 set_mon_vals no callback set 2024-09-11T01:29:21.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.599+0000 7f6fbca2a640 10 monclient: _renew_subs 2024-09-11T01:29:21.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.599+0000 7f6fbca2a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:21.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.599+0000 7f6fbca2a640 1 -- 172.21.15.12:0/611486321 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fb8126b10 con 0x7f6fb8138130 2024-09-11T01:29:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.600+0000 7f6fbca2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.600+0000 7f6fbca2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.600+0000 7f6fbca2a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.600+0000 7f6faeffd640 1 -- 172.21.15.12:0/611486321 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f6fb0012420 con 0x7f6fb8138130 2024-09-11T01:29:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.600+0000 7f6fbca2a640 1 -- 172.21.15.12:0/611486321 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fb81379d0 con 0x7f6fb8138130 2024-09-11T01:29:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.600+0000 7f6faeffd640 1 --2- 172.21.15.12:0/611486321 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6f9803e200 0x7f6f980406c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.601+0000 7f6faeffd640 1 -- 172.21.15.12:0/611486321 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4402+0+0 (secure 0 0 0) 0x7f6fb0052b20 con 0x7f6fb8138130 2024-09-11T01:29:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.601+0000 7f6fb659c640 1 --2- 172.21.15.12:0/611486321 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6f9803e200 0x7f6f980406c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.601+0000 7f6fb659c640 10 monclient: get_auth_request con 0x7f6f9803e200 auth_method 0 2024-09-11T01:29:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.603+0000 7f6fb659c640 1 --2- 172.21.15.12:0/611486321 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6f9803e200 0x7f6f980406c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6fb80a3670 tx=0x7f6fa802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.605+0000 7f6faeffd640 1 -- 172.21.15.12:0/611486321 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6fb0020070 con 0x7f6fb8138130 2024-09-11T01:29:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.605+0000 7f6faeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:21.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.605+0000 7f6faeffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:21.615 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-11T01:29:21.628 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T01:29:21.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.627+0000 7f464cbc2640 10 monclient: get_monmap_and_config 2024-09-11T01:29:21.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.627+0000 7f464cbc2640 10 monclient: build_initial_monmap 2024-09-11T01:29:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.627+0000 7f464cbc2640 10 monclient: monmap: 2024-09-11T01:29:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:21.628169+0000 2024-09-11T01:29:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:21.628169+0000 2024-09-11T01:29:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:21.635 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.628+0000 7f464cbc2640 1 Processor -- start 2024-09-11T01:29:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.628+0000 7f464cbc2640 1 -- start start 2024-09-11T01:29:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.628+0000 7f464cbc2640 10 monclient: init 2024-09-11T01:29:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.628+0000 7f464cbc2640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.628+0000 7f464cbc2640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-11T01:29:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.628+0000 7f464cbc2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4648137d50 0x7f464809d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.628+0000 7f464cbc2640 10 monclient(hunting): picked mon.noname-b con 0x7f4648137d50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.628+0000 7f464cbc2640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4648138130 0x7f4648138550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.628+0000 7f464cbc2640 10 monclient(hunting): picked mon.noname-a con 0x7f4648138130 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.628+0000 7f464cbc2640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4648136cd0 0x7f4648137120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.628+0000 7f464cbc2640 10 monclient(hunting): picked mon.noname-c con 0x7f4648136cd0 addr v2:172.21.15.71:3300/0 2024-09-11T01:29:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.628+0000 7f464cbc2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.628+0000 7f464cbc2640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f464809c650 con 0x7f4648138130 2024-09-11T01:29:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.628+0000 7f464cbc2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.628+0000 7f464cbc2640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f464809ba20 con 0x7f4648136cd0 2024-09-11T01:29:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.628+0000 7f464cbc2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.628+0000 7f464cbc2640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f464809c370 con 0x7f4648137d50 2024-09-11T01:29:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.628+0000 7f464cbc2640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.628+0000 7f464cbc2640 10 monclient(hunting): authenticate will time out at 768.070999s 2024-09-11T01:29:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f464659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4648137d50 0x7f464809d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f464659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4648137d50 0x7f464809d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51726/0 (socket says 172.21.15.12:51726) 2024-09-11T01:29:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f464659c640 1 -- 172.21.15.12:0/1841506217 learned_addr learned my addr 172.21.15.12:0/1841506217 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f464659c640 10 monclient(hunting): get_auth_request con 0x7f4648137d50 auth_method 0 2024-09-11T01:29:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f464659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f464659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f464659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f4645d9b640 1 --2- 172.21.15.12:0/1841506217 >> v2:172.21.15.12:3300/0 conn(0x7f4648138130 0x7f4648138550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f464659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f464659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f464659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f4645d9b640 10 monclient(hunting): get_auth_request con 0x7f4648138130 auth_method 0 2024-09-11T01:29:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f4645d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f4645d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f4645d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f4646d9d640 1 --2- 172.21.15.12:0/1841506217 >> v2:172.21.15.71:3300/0 conn(0x7f4648136cd0 0x7f4648137120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f4645d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f4645d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f4645d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f4646d9d640 10 monclient(hunting): get_auth_request con 0x7f4648136cd0 auth_method 0 2024-09-11T01:29:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f4646d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f4646d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f4646d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f464659c640 10 monclient(hunting): handle_auth_done global_id 4304 payload 274 2024-09-11T01:29:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f464659c640 1 -- 172.21.15.12:0/1841506217 >> v2:172.21.15.71:3300/0 conn(0x7f4648136cd0 msgr2=0x7f4648137120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f464659c640 1 --2- 172.21.15.12:0/1841506217 >> v2:172.21.15.71:3300/0 conn(0x7f4648136cd0 0x7f4648137120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f464659c640 1 -- 172.21.15.12:0/1841506217 >> v2:172.21.15.12:3300/0 conn(0x7f4648138130 msgr2=0x7f4648138550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f464659c640 1 --2- 172.21.15.12:0/1841506217 >> v2:172.21.15.12:3300/0 conn(0x7f4648138130 0x7f4648138550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f464659c640 10 monclient: _finish_hunting 0 2024-09-11T01:29:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f464659c640 1 monclient: found mon.noname-b 2024-09-11T01:29:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f464659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f464659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:29:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.629+0000 7f464659c640 1 -- 172.21.15.12:0/1841506217 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46480a6ce0 con 0x7f4648137d50 2024-09-11T01:29:21.640 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T01:29:21.647 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T01:29:21.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.632+0000 7f464659c640 1 -- 172.21.15.12:0/1841506217 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4648137d50 msgr2=0x7f464809d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:21.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.632+0000 7f464659c640 1 --2- 172.21.15.12:0/1841506217 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4648137d50 0x7f464809d110 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f46480a6f10 tx=0x7f463802a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5c7140d6ad282a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:21.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.632+0000 7f464559a640 1 -- 172.21.15.12:0/1841506217 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4638036070 con 0x7f4648137d50 2024-09-11T01:29:21.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.632+0000 7f464559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:21.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.632+0000 7f464559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.632+0000 7f464559a640 10 monclient: dump: 2024-09-11T01:29:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:21.649 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.632+0000 7f464559a640 10 monclient: _finish_auth 0 2024-09-11T01:29:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.632+0000 7f464559a640 10 monclient: _check_auth_tickets 2024-09-11T01:29:21.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.632+0000 7f464559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.632+0000 7f464559a640 1 -- 172.21.15.12:0/1841506217 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4638007800 con 0x7f4648137d50 2024-09-11T01:29:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.632+0000 7f464559a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.632+0000 7f464559a640 1 -- 172.21.15.12:0/1841506217 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4638007bd0 con 0x7f4648137d50 2024-09-11T01:29:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.632+0000 7f464cbc2640 5 monclient: authenticate success, global_id 4304 2024-09-11T01:29:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.632+0000 7f464cbc2640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.632+0000 7f464cbc2640 4 set_mon_vals no callback set 2024-09-11T01:29:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.632+0000 7f464559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.632+0000 7f464559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.632+0000 7f464559a640 10 monclient: dump: 2024-09-11T01:29:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:21.651 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.633+0000 7f464cbc2640 10 monclient: shutdown 2024-09-11T01:29:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.633+0000 7f464cbc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.633+0000 7f464cbc2640 1 -- 172.21.15.12:0/1841506217 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4648137d50 msgr2=0x7f464809d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.633+0000 7f464cbc2640 1 --2- 172.21.15.12:0/1841506217 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4648137d50 0x7f464809d110 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f46480a6f10 tx=0x7f463802a5f0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.633+0000 7f464cbc2640 1 -- 172.21.15.12:0/1841506217 shutdown_connections 2024-09-11T01:29:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.633+0000 7f464cbc2640 1 --2- 172.21.15.12:0/1841506217 >> v2:172.21.15.71:3300/0 conn(0x7f4648136cd0 0x7f4648137120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.633+0000 7f464cbc2640 1 --2- 172.21.15.12:0/1841506217 >> v2:172.21.15.12:3300/0 conn(0x7f4648138130 0x7f4648138550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.633+0000 7f464cbc2640 1 --2- 172.21.15.12:0/1841506217 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4648137d50 0x7f464809d110 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.633+0000 7f464cbc2640 1 -- 172.21.15.12:0/1841506217 >> 172.21.15.12:0/1841506217 conn(0x7f46480a7020 msgr2=0x7f46480a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:21.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.633+0000 7f464cbc2640 1 -- 172.21.15.12:0/1841506217 shutdown_connections 2024-09-11T01:29:21.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.633+0000 7f464cbc2640 1 -- 172.21.15.12:0/1841506217 wait complete. 2024-09-11T01:29:21.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.634+0000 7f464cbc2640 10 monclient: build_initial_monmap 2024-09-11T01:29:21.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.634+0000 7f464cbc2640 10 monclient: monmap: 2024-09-11T01:29:21.652 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:21.652 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:21.652 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:21.652 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:21.652 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:21.652 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.652 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:21.652 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:21.652 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:21.653 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.635+0000 7f464cbc2640 1 Processor -- start 2024-09-11T01:29:21.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.635+0000 7f464cbc2640 1 -- start start 2024-09-11T01:29:21.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.635+0000 7f464cbc2640 10 monclient: init 2024-09-11T01:29:21.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.636+0000 7f464cbc2640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:21.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.636+0000 7f464cbc2640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:29:21.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.636+0000 7f464cbc2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4648136cd0 0x7f46480727b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.636+0000 7f464cbc2640 10 monclient(hunting): picked mon.noname-a con 0x7f4648136cd0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:21.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.636+0000 7f464cbc2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4648137d50 0x7f4648072cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.636+0000 7f464cbc2640 10 monclient(hunting): picked mon.noname-b con 0x7f4648137d50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:21.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.636+0000 7f464659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4648136cd0 0x7f46480727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.636+0000 7f464659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4648136cd0 0x7f46480727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55030/0 (socket says 172.21.15.12:55030) 2024-09-11T01:29:21.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.636+0000 7f464659c640 1 -- 172.21.15.12:0/329908919 learned_addr learned my addr 172.21.15.12:0/329908919 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:21.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.636+0000 7f464cbc2640 1 --2- 172.21.15.12:0/329908919 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4648138130 0x7f4648073230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.636+0000 7f4645d9b640 1 --2- 172.21.15.12:0/329908919 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4648137d50 0x7f4648072cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f4646d9d640 1 --2- 172.21.15.12:0/329908919 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4648138130 0x7f4648073230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.636+0000 7f464cbc2640 10 monclient(hunting): picked mon.noname-c con 0x7f4648138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:21.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f464cbc2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f464cbc2640 1 -- 172.21.15.12:0/329908919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f464806f850 con 0x7f4648136cd0 2024-09-11T01:29:21.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f464cbc2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f464cbc2640 1 -- 172.21.15.12:0/329908919 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f464806f6d0 con 0x7f4648137d50 2024-09-11T01:29:21.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f464cbc2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f464cbc2640 1 -- 172.21.15.12:0/329908919 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f464809d650 con 0x7f4648138130 2024-09-11T01:29:21.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f464cbc2640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:21.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f464cbc2640 10 monclient(hunting): authenticate will time out at 768.079744s 2024-09-11T01:29:21.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f464659c640 10 monclient(hunting): get_auth_request con 0x7f4648136cd0 auth_method 0 2024-09-11T01:29:21.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f464659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f464659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f464659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f4645d9b640 10 monclient(hunting): get_auth_request con 0x7f4648137d50 auth_method 0 2024-09-11T01:29:21.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f4645d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f4645d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f4645d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f4646d9d640 10 monclient(hunting): get_auth_request con 0x7f4648138130 auth_method 0 2024-09-11T01:29:21.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f4646d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f4646d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f4646d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f4646d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:21.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f4646d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:21.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f4646d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:21.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f464659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:21.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f464659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:21.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.637+0000 7f464659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:21.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.638+0000 7f4645d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:21.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.638+0000 7f4645d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:21.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.638+0000 7f4645d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:21.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.638+0000 7f4646d9d640 10 monclient(hunting): handle_auth_done global_id 4307 payload 824 2024-09-11T01:29:21.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.638+0000 7f4646d9d640 1 -- 172.21.15.12:0/329908919 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4648137d50 msgr2=0x7f4648072cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:21.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.638+0000 7f4646d9d640 1 --2- 172.21.15.12:0/329908919 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4648137d50 0x7f4648072cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.638+0000 7f4646d9d640 1 -- 172.21.15.12:0/329908919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4648136cd0 msgr2=0x7f46480727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:21.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.638+0000 7f4646d9d640 1 --2- 172.21.15.12:0/329908919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4648136cd0 0x7f46480727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.638+0000 7f4646d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:29:21.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.638+0000 7f4646d9d640 1 monclient: found mon.noname-c 2024-09-11T01:29:21.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.638+0000 7f4646d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:21.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.638+0000 7f4646d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:29:21.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.638+0000 7f4646d9d640 1 -- 172.21.15.12:0/329908919 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f464813b600 con 0x7f4648138130 2024-09-11T01:29:21.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.638+0000 7f464659c640 1 --2- 172.21.15.12:0/329908919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4648136cd0 0x7f46480727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:21.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.640+0000 7f4646d9d640 1 -- 172.21.15.12:0/329908919 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4648138130 msgr2=0x7f4648073230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:21.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.640+0000 7f4646d9d640 1 --2- 172.21.15.12:0/329908919 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4648138130 0x7f4648073230 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f464000d530 tx=0x7f464000d9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:21.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.640+0000 7f463effd640 1 -- 172.21.15.12:0/329908919 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4640012800 con 0x7f4648138130 2024-09-11T01:29:21.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.640+0000 7f463effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:21.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.640+0000 7f463effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:21.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.640+0000 7f463effd640 10 monclient: dump: 2024-09-11T01:29:21.657 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:21.657 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.657 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.657 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.657 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:21.658 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.658 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:21.658 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:21.658 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:21.658 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.640+0000 7f463effd640 10 monclient: _finish_auth 0 2024-09-11T01:29:21.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.640+0000 7f463effd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:21.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.640+0000 7f463effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:21.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.640+0000 7f464cbc2640 5 monclient: authenticate success, global_id 4307 2024-09-11T01:29:21.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.640+0000 7f463effd640 1 -- 172.21.15.12:0/329908919 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4640019070 con 0x7f4648138130 2024-09-11T01:29:21.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.640+0000 7f463effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:21.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.640+0000 7f464cbc2640 10 monclient: _renew_subs 2024-09-11T01:29:21.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.640+0000 7f463effd640 1 -- 172.21.15.12:0/329908919 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4640012bc0 con 0x7f4648138130 2024-09-11T01:29:21.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.640+0000 7f464cbc2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:21.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.640+0000 7f464cbc2640 1 -- 172.21.15.12:0/329908919 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f464813b8e0 con 0x7f4648138130 2024-09-11T01:29:21.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.641+0000 7f463effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:21.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.641+0000 7f463effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:21.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.641+0000 7f4644d99640 4 set_mon_vals no callback set 2024-09-11T01:29:21.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.641+0000 7f463effd640 10 monclient: dump: 2024-09-11T01:29:21.659 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:21.659 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.659 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.659 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.659 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:21.659 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.659 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:21.659 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:21.659 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:21.660 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.641+0000 7f464cbc2640 10 monclient: _renew_subs 2024-09-11T01:29:21.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.641+0000 7f464cbc2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:21.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.641+0000 7f464cbc2640 1 -- 172.21.15.12:0/329908919 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f464813b320 con 0x7f4648138130 2024-09-11T01:29:21.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.642+0000 7f463effd640 1 -- 172.21.15.12:0/329908919 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f4640013070 con 0x7f4648138130 2024-09-11T01:29:21.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.642+0000 7f463effd640 1 --2- 172.21.15.12:0/329908919 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f462803e210 0x7f46280406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.642+0000 7f463effd640 1 -- 172.21.15.12:0/329908919 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4402+0+0 (secure 0 0 0) 0x7f4640052c40 con 0x7f4648138130 2024-09-11T01:29:21.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.642+0000 7f464cbc2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:21.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.642+0000 7f464cbc2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:21.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.642+0000 7f464cbc2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:21.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.642+0000 7f464cbc2640 1 -- 172.21.15.12:0/329908919 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f464809d110 con 0x7f4648138130 2024-09-11T01:29:21.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.642+0000 7f464659c640 1 --2- 172.21.15.12:0/329908919 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f462803e210 0x7f46280406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.642+0000 7f464659c640 10 monclient: get_auth_request con 0x7f462803e210 auth_method 0 2024-09-11T01:29:21.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.645+0000 7f464659c640 1 --2- 172.21.15.12:0/329908919 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f462803e210 0x7f46280406d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4638007e00 tx=0x7f4638009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:21.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.647+0000 7f463effd640 1 -- 172.21.15.12:0/329908919 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4640012d60 con 0x7f4648138130 2024-09-11T01:29:21.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.647+0000 7f463effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:21.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.647+0000 7f463effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.744+0000 7efc069bb640 10 monclient: get_monmap_and_config 2024-09-11T01:29:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.744+0000 7efc069bb640 10 monclient: build_initial_monmap 2024-09-11T01:29:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.744+0000 7efc069bb640 10 monclient: monmap: 2024-09-11T01:29:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:21.744854+0000 2024-09-11T01:29:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:21.744854+0000 2024-09-11T01:29:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:21.747 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:21.747 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:21.747 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.744+0000 7efc069bb640 1 Processor -- start 2024-09-11T01:29:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.745+0000 7efc069bb640 1 -- start start 2024-09-11T01:29:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.745+0000 7efc069bb640 10 monclient: init 2024-09-11T01:29:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.745+0000 7efc069bb640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.745+0000 7efc069bb640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-11T01:29:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.745+0000 7efc069bb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efc00137d50 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.745+0000 7efc069bb640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc00138130 0x7efc00138550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.745+0000 7efc069bb640 10 monclient(hunting): picked mon.noname-b con 0x7efc00138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc069bb640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7efc0013d280 0x7efc0013f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efbfffff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7efc00137d50 0x7efc0009d110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56080/0 (socket says 172.21.15.12:56080) 2024-09-11T01:29:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc04f58640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7efc0013d280 0x7efc0013f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc04f58640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7efc0013d280 0x7efc0013f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55042/0 (socket says 172.21.15.12:55042) 2024-09-11T01:29:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc069bb640 10 monclient(hunting): picked mon.noname-a con 0x7efc0013d280 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efbfffff640 1 -- 172.21.15.12:0/3061349835 learned_addr learned my addr 172.21.15.12:0/3061349835 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:29:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc069bb640 1 -- 172.21.15.12:0/3061349835 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc0009d820 con 0x7efc00137d50 2024-09-11T01:29:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efbff7fe640 1 --2- 172.21.15.12:0/3061349835 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc00138130 0x7efc00138550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc069bb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc069bb640 1 -- 172.21.15.12:0/3061349835 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7efc0009c650 con 0x7efc0013d280 2024-09-11T01:29:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc069bb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc069bb640 1 -- 172.21.15.12:0/3061349835 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7efc0009ba20 con 0x7efc00138130 2024-09-11T01:29:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc069bb640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:21.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc069bb640 10 monclient(hunting): authenticate will time out at 768.188435s 2024-09-11T01:29:21.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc04f58640 10 monclient(hunting): get_auth_request con 0x7efc0013d280 auth_method 0 2024-09-11T01:29:21.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc04f58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc04f58640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc04f58640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efbff7fe640 10 monclient(hunting): get_auth_request con 0x7efc00138130 auth_method 0 2024-09-11T01:29:21.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efbff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efbff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efbff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc04f58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:21.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc04f58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:21.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc04f58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:21.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc04f58640 10 monclient(hunting): handle_auth_done global_id 4296 payload 274 2024-09-11T01:29:21.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc04f58640 1 -- 172.21.15.12:0/3061349835 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc00138130 msgr2=0x7efc00138550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc04f58640 1 --2- 172.21.15.12:0/3061349835 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc00138130 0x7efc00138550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc04f58640 1 -- 172.21.15.12:0/3061349835 >> v1:172.21.15.12:6789/0 conn(0x7efc00137d50 legacy=0x7efc0009d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc04f58640 10 monclient: _finish_hunting 0 2024-09-11T01:29:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc04f58640 1 monclient: found mon.noname-a 2024-09-11T01:29:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc04f58640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc04f58640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efc04f58640 1 -- 172.21.15.12:0/3061349835 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc0009c370 con 0x7efc0013d280 2024-09-11T01:29:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.746+0000 7efbff7fe640 1 --2- 172.21.15.12:0/3061349835 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc00138130 0x7efc00138550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:29:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.749+0000 7efc04f58640 1 -- 172.21.15.12:0/3061349835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc0013d280 msgr2=0x7efc0013f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.749+0000 7efc04f58640 1 --2- 172.21.15.12:0/3061349835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc0013d280 0x7efc0013f670 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efbf8009d90 tx=0x7efbf802a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2e4d0472cbb3dcd server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.750+0000 7efbfeffd640 1 -- 172.21.15.12:0/3061349835 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf8007950 con 0x7efc0013d280 2024-09-11T01:29:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.750+0000 7efbfeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.750+0000 7efbfeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.750+0000 7efbfeffd640 10 monclient: dump: 2024-09-11T01:29:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:21.762 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.762 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.762 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.762 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:21.765 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.765 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:21.765 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:21.765 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:21.765 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.750+0000 7efbfeffd640 10 monclient: _finish_auth 0 2024-09-11T01:29:21.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.750+0000 7efbfeffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:21.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.750+0000 7efbfeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:21.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.750+0000 7efc069bb640 5 monclient: authenticate success, global_id 4296 2024-09-11T01:29:21.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.750+0000 7efc069bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:21.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.750+0000 7efbfeffd640 1 -- 172.21.15.12:0/3061349835 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbf8007af0 con 0x7efc0013d280 2024-09-11T01:29:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.750+0000 7efbfeffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.750+0000 7efbfeffd640 1 -- 172.21.15.12:0/3061349835 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf8031400 con 0x7efc0013d280 2024-09-11T01:29:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.750+0000 7efbfeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.750+0000 7efbfeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.750+0000 7efbfeffd640 10 monclient: dump: 2024-09-11T01:29:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:21.767 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.750+0000 7efc069bb640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.750+0000 7efc069bb640 4 set_mon_vals no callback set 2024-09-11T01:29:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.751+0000 7efc069bb640 10 monclient: shutdown 2024-09-11T01:29:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.751+0000 7efc069bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.751+0000 7efc069bb640 1 -- 172.21.15.12:0/3061349835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc0013d280 msgr2=0x7efc0013f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.751+0000 7efc069bb640 1 --2- 172.21.15.12:0/3061349835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc0013d280 0x7efc0013f670 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efbf8009d90 tx=0x7efbf802a530 comp rx=0 tx=0).stop 2024-09-11T01:29:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.751+0000 7efc069bb640 1 -- 172.21.15.12:0/3061349835 shutdown_connections 2024-09-11T01:29:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.751+0000 7efc069bb640 1 --2- 172.21.15.12:0/3061349835 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc0013d280 0x7efc0013f670 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.751+0000 7efc069bb640 1 --2- 172.21.15.12:0/3061349835 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc00138130 0x7efc00138550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.751+0000 7efc069bb640 1 -- 172.21.15.12:0/3061349835 >> 172.21.15.12:0/3061349835 conn(0x7efc000a7020 msgr2=0x7efc000a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.751+0000 7efc069bb640 1 -- 172.21.15.12:0/3061349835 shutdown_connections 2024-09-11T01:29:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.751+0000 7efc069bb640 1 -- 172.21.15.12:0/3061349835 wait complete. 2024-09-11T01:29:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.752+0000 7efc069bb640 10 monclient: build_initial_monmap 2024-09-11T01:29:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.752+0000 7efc069bb640 10 monclient: monmap: 2024-09-11T01:29:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:21.769 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.752+0000 7efc069bb640 1 Processor -- start 2024-09-11T01:29:21.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.752+0000 7efc069bb640 1 -- start start 2024-09-11T01:29:21.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.752+0000 7efc069bb640 10 monclient: init 2024-09-11T01:29:21.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.753+0000 7efc069bb640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:21.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.753+0000 7efc069bb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:29:21.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.753+0000 7efc069bb640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc00137d50 0x7efc001ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.753+0000 7efc069bb640 10 monclient(hunting): picked mon.noname-c con 0x7efc00137d50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:21.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.753+0000 7efc069bb640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc00138130 0x7efc00136ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.753+0000 7efc069bb640 10 monclient(hunting): picked mon.noname-b con 0x7efc00138130 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:21.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.753+0000 7efc069bb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc0013d280 0x7efc0013a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.753+0000 7efc069bb640 10 monclient(hunting): picked mon.noname-a con 0x7efc0013d280 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.753+0000 7efc069bb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.753+0000 7efc069bb640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7efc0006f850 con 0x7efc0013d280 2024-09-11T01:29:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.753+0000 7efc069bb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.753+0000 7efc069bb640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7efc0006f6d0 con 0x7efc00138130 2024-09-11T01:29:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.753+0000 7efc069bb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.753+0000 7efc069bb640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7efc0013fbb0 con 0x7efc00137d50 2024-09-11T01:29:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.753+0000 7efc069bb640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.753+0000 7efc069bb640 10 monclient(hunting): authenticate will time out at 768.195633s 2024-09-11T01:29:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.753+0000 7efbfffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc00137d50 0x7efc001ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.753+0000 7efbfffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc00137d50 0x7efc001ccee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51756/0 (socket says 172.21.15.12:51756) 2024-09-11T01:29:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.753+0000 7efbfffff640 1 -- 172.21.15.12:0/2607219355 learned_addr learned my addr 172.21.15.12:0/2607219355 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.753+0000 7efbfffff640 10 monclient(hunting): get_auth_request con 0x7efc00137d50 auth_method 0 2024-09-11T01:29:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.753+0000 7efbfffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.753+0000 7efbfffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.753+0000 7efbfffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.754+0000 7efbfffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.754+0000 7efbfffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.754+0000 7efbfffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.754+0000 7efbfffff640 10 monclient(hunting): handle_auth_done global_id 4313 payload 824 2024-09-11T01:29:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.754+0000 7efbfffff640 1 -- 172.21.15.12:0/2607219355 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc00138130 msgr2=0x7efc00136ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.754+0000 7efbfffff640 1 --2- 172.21.15.12:0/2607219355 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc00138130 0x7efc00136ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.754+0000 7efbfffff640 1 -- 172.21.15.12:0/2607219355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc0013d280 msgr2=0x7efc0013a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.754+0000 7efc04f58640 1 --2- 172.21.15.12:0/2607219355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc0013d280 0x7efc0013a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.754+0000 7efbfffff640 1 --2- 172.21.15.12:0/2607219355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc0013d280 0x7efc0013a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.754+0000 7efbfffff640 10 monclient: _finish_hunting 0 2024-09-11T01:29:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.754+0000 7efbfffff640 1 monclient: found mon.noname-c 2024-09-11T01:29:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.754+0000 7efbfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.754+0000 7efbfffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:29:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.754+0000 7efbfffff640 1 -- 172.21.15.12:0/2607219355 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc0013c760 con 0x7efc00137d50 2024-09-11T01:29:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.754+0000 7efc04f58640 10 monclient: get_auth_request con 0x7efc0013d280 auth_method 0 2024-09-11T01:29:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.754+0000 7efc04f58640 1 --2- 172.21.15.12:0/2607219355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc0013d280 0x7efc0013a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:29:21.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.755+0000 7f0c571ca640 10 monclient: get_monmap_and_config 2024-09-11T01:29:21.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.755+0000 7f0c571ca640 10 monclient: build_initial_monmap 2024-09-11T01:29:21.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.755+0000 7f0c571ca640 10 monclient: monmap: 2024-09-11T01:29:21.773 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:21.773 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.774 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:21.755820+0000 2024-09-11T01:29:21.774 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:21.755820+0000 2024-09-11T01:29:21.774 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:21.774 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.774 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:21.774 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:21.774 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:21.774 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:21.774 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:21.774 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.755+0000 7f0c571ca640 1 Processor -- start 2024-09-11T01:29:21.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.755+0000 7f0c571ca640 1 -- start start 2024-09-11T01:29:21.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.756+0000 7f0c571ca640 10 monclient: init 2024-09-11T01:29:21.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.756+0000 7f0c571ca640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:21.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.756+0000 7f0c571ca640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-11T01:29:21.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.756+0000 7f0c571ca640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0c50137d50 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:21.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.756+0000 7f0c571ca640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0c50138130 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:21.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.756+0000 7f0c571ca640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c5013e290 0x7f0c50140680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.756+0000 7f0c571ca640 10 monclient(hunting): picked mon.noname-b con 0x7f0c5013e290 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:21.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.756+0000 7f0c571ca640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c5009d820 con 0x7f0c50137d50 2024-09-11T01:29:21.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.756+0000 7f0c571ca640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c5009ba20 con 0x7f0c50138130 2024-09-11T01:29:21.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.756+0000 7f0c571ca640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.756+0000 7f0c571ca640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f0c5009c650 con 0x7f0c5013e290 2024-09-11T01:29:21.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.756+0000 7f0c571ca640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:21.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.756+0000 7f0c571ca640 10 monclient(hunting): authenticate will time out at 768.198723s 2024-09-11T01:29:21.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.756+0000 7f0c54f66640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0c50137d50 0x7f0c5009d110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56092/0 (socket says 172.21.15.12:56092) 2024-09-11T01:29:21.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.756+0000 7f0c54f66640 1 -- 172.21.15.12:0/793295995 learned_addr learned my addr 172.21.15.12:0/793295995 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:29:21.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.756+0000 7efbfffff640 1 -- 172.21.15.12:0/2607219355 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc00137d50 msgr2=0x7efc001ccee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:21.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.756+0000 7efbfffff640 1 --2- 172.21.15.12:0/2607219355 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc00137d50 0x7efc001ccee0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efbf0005580 tx=0x7efbf0014420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.756+0000 7f0c55767640 1 --2- 172.21.15.12:0/793295995 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c5013e290 0x7f0c50140680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7f0c55767640 10 monclient(hunting): get_auth_request con 0x7f0c5013e290 auth_method 0 2024-09-11T01:29:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7f0c55767640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7f0c55767640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7f0c55767640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7efbfcff9640 1 -- 172.21.15.12:0/2607219355 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf00058d0 con 0x7efc00137d50 2024-09-11T01:29:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7f0c55767640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7efbfcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7efbfcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7efbfcff9640 10 monclient: dump: 2024-09-11T01:29:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:21.779 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7efbfcff9640 10 monclient: _finish_auth 0 2024-09-11T01:29:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7efbfcff9640 10 monclient: _check_auth_tickets 2024-09-11T01:29:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7efbfcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7efc069bb640 5 monclient: authenticate success, global_id 4313 2024-09-11T01:29:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7efc069bb640 10 monclient: _renew_subs 2024-09-11T01:29:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7efc069bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7efc069bb640 1 -- 172.21.15.12:0/2607219355 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc00120fc0 con 0x7efc00137d50 2024-09-11T01:29:21.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7f0c55767640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:21.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7f0c55767640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:21.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7f0c55767640 10 monclient(hunting): handle_auth_done global_id 4316 payload 274 2024-09-11T01:29:21.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7f0c55767640 1 -- 172.21.15.12:0/793295995 >> v1:172.21.15.71:6789/0 conn(0x7f0c50138130 legacy=0x7f0c50138580 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-11T01:29:21.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7f0c55767640 1 -- 172.21.15.12:0/793295995 >> v1:172.21.15.12:6789/0 conn(0x7f0c50137d50 legacy=0x7f0c5009d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:21.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7f0c55767640 10 monclient: _finish_hunting 0 2024-09-11T01:29:21.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7f0c55767640 1 monclient: found mon.noname-b 2024-09-11T01:29:21.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7f0c55767640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:21.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7f0c55767640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:29:21.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7f0c55767640 1 -- 172.21.15.12:0/793295995 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c5009c370 con 0x7f0c5013e290 2024-09-11T01:29:21.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7efc069bb640 10 monclient: _renew_subs 2024-09-11T01:29:21.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7efc069bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:21.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.757+0000 7efc069bb640 1 -- 172.21.15.12:0/2607219355 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efc0013bc00 con 0x7efc00137d50 2024-09-11T01:29:21.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.758+0000 7efc069bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:21.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.758+0000 7efc069bb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:21.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.758+0000 7efc069bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:21.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.758+0000 7efc069bb640 1 -- 172.21.15.12:0/2607219355 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc0006a250 con 0x7efc00137d50 2024-09-11T01:29:21.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.758+0000 7efbfcff9640 1 -- 172.21.15.12:0/2607219355 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbf0005a70 con 0x7efc00137d50 2024-09-11T01:29:21.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.758+0000 7efbfcff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:21.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.758+0000 7efbfe7fc640 4 set_mon_vals no callback set 2024-09-11T01:29:21.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.759+0000 7efbfcff9640 1 -- 172.21.15.12:0/2607219355 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf0005e20 con 0x7efc00137d50 2024-09-11T01:29:21.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.759+0000 7efbfcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:21.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.759+0000 7efbfcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:21.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.759+0000 7efbfcff9640 10 monclient: dump: 2024-09-11T01:29:21.784 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:21.784 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.784 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.784 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.784 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:21.784 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.784 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:21.784 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:21.785 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:21.785 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.759+0000 7efbfcff9640 1 -- 172.21.15.12:0/2607219355 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7efbf0010850 con 0x7efc00137d50 2024-09-11T01:29:21.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.759+0000 7efbfcff9640 1 --2- 172.21.15.12:0/2607219355 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efbec03e1c0 0x7efbec040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.759+0000 7efbfcff9640 1 -- 172.21.15.12:0/2607219355 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4402+0+0 (secure 0 0 0) 0x7efbf005b320 con 0x7efc00137d50 2024-09-11T01:29:21.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.760+0000 7efbff7fe640 1 --2- 172.21.15.12:0/2607219355 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efbec03e1c0 0x7efbec040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.760+0000 7efbff7fe640 10 monclient: get_auth_request con 0x7efbec03e1c0 auth_method 0 2024-09-11T01:29:21.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.760+0000 7f0c55767640 1 -- 172.21.15.12:0/793295995 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c5013e290 msgr2=0x7f0c50140680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:21.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.760+0000 7f0c55767640 1 --2- 172.21.15.12:0/793295995 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c5013e290 0x7f0c50140680 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0c48009dc0 tx=0x7f0c4802a510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93bf87273788ab04 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:21.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.763+0000 7efbff7fe640 1 --2- 172.21.15.12:0/2607219355 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efbec03e1c0 0x7efbec040680 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efbf4006e80 tx=0x7efbf4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:21.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.763+0000 7face77c6640 10 monclient: get_monmap_and_config 2024-09-11T01:29:21.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.763+0000 7face77c6640 10 monclient: build_initial_monmap 2024-09-11T01:29:21.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.763+0000 7face77c6640 10 monclient: monmap: 2024-09-11T01:29:21.788 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:21.788 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.788 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:21.764270+0000 2024-09-11T01:29:21.788 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:21.764270+0000 2024-09-11T01:29:21.788 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:21.788 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.788 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:21.788 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:21.788 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:21.788 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:21.788 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:21.788 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.764+0000 7face77c6640 1 Processor -- start 2024-09-11T01:29:21.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.764+0000 7face77c6640 1 -- start start 2024-09-11T01:29:21.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.764+0000 7face77c6640 10 monclient: init 2024-09-11T01:29:21.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.764+0000 7face77c6640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:21.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.764+0000 7face77c6640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-11T01:29:21.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.764+0000 7face77c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7face0137d50 0x7face009d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.764+0000 7face77c6640 10 monclient(hunting): picked mon.noname-b con 0x7face0137d50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:21.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.764+0000 7face77c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7face0138130 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:21.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.764+0000 7face77c6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7face013d280 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:21.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.764+0000 7face77c6640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7face009ba20 con 0x7face0138130 2024-09-11T01:29:21.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.764+0000 7face77c6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7face009c370 con 0x7face013d280 2024-09-11T01:29:21.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.764+0000 7face77c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.764+0000 7face77c6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7face009c650 con 0x7face0137d50 2024-09-11T01:29:21.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.764+0000 7face77c6640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:21.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.764+0000 7face77c6640 10 monclient(hunting): authenticate will time out at 768.206940s 2024-09-11T01:29:21.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.765+0000 7face5562640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7face0137d50 0x7face009d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.765+0000 7face5562640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7face0137d50 0x7face009d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51762/0 (socket says 172.21.15.12:51762) 2024-09-11T01:29:21.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.765+0000 7face5562640 1 -- 172.21.15.12:0/1582957593 learned_addr learned my addr 172.21.15.12:0/1582957593 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:21.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.765+0000 7face5562640 10 monclient(hunting): get_auth_request con 0x7face0137d50 auth_method 0 2024-09-11T01:29:21.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.765+0000 7face5562640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.765+0000 7face5562640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.765+0000 7face5562640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.765+0000 7face5562640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:21.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.765+0000 7face5562640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:21.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.765+0000 7face5562640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:21.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.765+0000 7face5562640 10 monclient(hunting): handle_auth_done global_id 4319 payload 274 2024-09-11T01:29:21.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.765+0000 7face5562640 1 -- 172.21.15.12:0/1582957593 >> v1:172.21.15.71:6789/0 conn(0x7face013d280 legacy=0x7face013f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:21.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.765+0000 7face5562640 1 -- 172.21.15.12:0/1582957593 >> v1:172.21.15.12:6789/0 conn(0x7face0138130 legacy=0x7face0138550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:21.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.765+0000 7face5562640 10 monclient: _finish_hunting 0 2024-09-11T01:29:21.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.765+0000 7face5562640 1 monclient: found mon.noname-b 2024-09-11T01:29:21.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.765+0000 7face5562640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:21.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.765+0000 7face5562640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:29:21.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.765+0000 7face5562640 1 -- 172.21.15.12:0/1582957593 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7face00a6ce0 con 0x7face0137d50 2024-09-11T01:29:21.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7f0c4f7fe640 1 -- 172.21.15.12:0/793295995 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c48007a30 con 0x7f0c5013e290 2024-09-11T01:29:21.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7f0c4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:21.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7f0c4f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:21.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7efbfcff9640 1 -- 172.21.15.12:0/2607219355 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7efc0006a250 con 0x7efc00137d50 2024-09-11T01:29:21.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7f0c4f7fe640 10 monclient: dump: 2024-09-11T01:29:21.792 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:21.792 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.792 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.792 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.792 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:21.792 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.792 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:21.792 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:21.792 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:21.792 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7f0c4f7fe640 10 monclient: _finish_auth 0 2024-09-11T01:29:21.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7f0c4f7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:29:21.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7f0c4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:21.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7efbfcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:21.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7efbfcff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:21.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7f0c4f7fe640 1 -- 172.21.15.12:0/793295995 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c48007bd0 con 0x7f0c5013e290 2024-09-11T01:29:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7f0c4f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7f0c4f7fe640 1 -- 172.21.15.12:0/793295995 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c48031400 con 0x7f0c5013e290 2024-09-11T01:29:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7f0c4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7f0c4f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7f0c4f7fe640 10 monclient: dump: 2024-09-11T01:29:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:21.795 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:21.795 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7f0c571ca640 5 monclient: authenticate success, global_id 4316 2024-09-11T01:29:21.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7f0c571ca640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:21.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7f0c571ca640 4 set_mon_vals no callback set 2024-09-11T01:29:21.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7face5562640 1 -- 172.21.15.12:0/1582957593 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7face0137d50 msgr2=0x7face009d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:21.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7face5562640 1 --2- 172.21.15.12:0/1582957593 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7face0137d50 0x7face009d110 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7facdc0098a0 tx=0x7facdc02a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=808de70cc6427e85 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:21.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7facd7fff640 1 -- 172.21.15.12:0/1582957593 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facdc037070 con 0x7face0137d50 2024-09-11T01:29:21.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7facd7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:21.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7facd7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:21.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7facd7fff640 10 monclient: dump: 2024-09-11T01:29:21.796 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:21.797 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.797 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.797 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.797 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:21.797 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.797 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:21.797 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:21.797 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:21.797 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7facd7fff640 10 monclient: _finish_auth 0 2024-09-11T01:29:21.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7facd7fff640 10 monclient: _check_auth_tickets 2024-09-11T01:29:21.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7facd7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:21.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7face77c6640 5 monclient: authenticate success, global_id 4319 2024-09-11T01:29:21.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7face77c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:21.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7facd7fff640 1 -- 172.21.15.12:0/1582957593 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facdc032640 con 0x7face0137d50 2024-09-11T01:29:21.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7facd7fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:21.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7face77c6640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:21.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7face77c6640 4 set_mon_vals no callback set 2024-09-11T01:29:21.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7facd7fff640 1 -- 172.21.15.12:0/1582957593 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facdc032a10 con 0x7face0137d50 2024-09-11T01:29:21.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7facd7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:21.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7facd7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:21.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.768+0000 7facd7fff640 10 monclient: dump: 2024-09-11T01:29:21.798 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:21.798 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.798 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.798 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.799 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:21.799 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.799 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:21.799 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:21.799 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:21.799 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.769+0000 7f0c571ca640 10 monclient: shutdown 2024-09-11T01:29:21.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.769+0000 7f0c571ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:21.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.769+0000 7f0c571ca640 1 -- 172.21.15.12:0/793295995 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c5013e290 msgr2=0x7f0c50140680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:21.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.769+0000 7f0c571ca640 1 --2- 172.21.15.12:0/793295995 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c5013e290 0x7f0c50140680 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0c48009dc0 tx=0x7f0c4802a510 comp rx=0 tx=0).stop 2024-09-11T01:29:21.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.769+0000 7f0c571ca640 1 -- 172.21.15.12:0/793295995 shutdown_connections 2024-09-11T01:29:21.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.769+0000 7f0c571ca640 1 --2- 172.21.15.12:0/793295995 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c5013e290 0x7f0c50140680 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.769+0000 7f0c571ca640 1 -- 172.21.15.12:0/793295995 >> 172.21.15.12:0/793295995 conn(0x7f0c500a7020 msgr2=0x7f0c500a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:21.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.769+0000 7f0c571ca640 1 -- 172.21.15.12:0/793295995 shutdown_connections 2024-09-11T01:29:21.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.769+0000 7f0c571ca640 1 -- 172.21.15.12:0/793295995 wait complete. 2024-09-11T01:29:21.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.769+0000 7face77c6640 10 monclient: shutdown 2024-09-11T01:29:21.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.769+0000 7face77c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:21.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.769+0000 7face77c6640 1 -- 172.21.15.12:0/1582957593 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7face0137d50 msgr2=0x7face009d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:21.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.769+0000 7face77c6640 1 --2- 172.21.15.12:0/1582957593 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7face0137d50 0x7face009d110 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7facdc0098a0 tx=0x7facdc02a100 comp rx=0 tx=0).stop 2024-09-11T01:29:21.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.769+0000 7face77c6640 1 -- 172.21.15.12:0/1582957593 shutdown_connections 2024-09-11T01:29:21.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.769+0000 7face77c6640 1 --2- 172.21.15.12:0/1582957593 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7face0137d50 0x7face009d110 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.769+0000 7face77c6640 1 -- 172.21.15.12:0/1582957593 >> 172.21.15.12:0/1582957593 conn(0x7face00a7020 msgr2=0x7face00a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:21.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.769+0000 7face77c6640 1 -- 172.21.15.12:0/1582957593 shutdown_connections 2024-09-11T01:29:21.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.769+0000 7face77c6640 1 -- 172.21.15.12:0/1582957593 wait complete. 2024-09-11T01:29:21.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.770+0000 7f0c571ca640 10 monclient: build_initial_monmap 2024-09-11T01:29:21.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.770+0000 7f0c571ca640 10 monclient: monmap: 2024-09-11T01:29:21.802 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:21.802 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:21.802 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:21.802 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:21.802 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:21.802 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.802 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:21.803 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:21.803 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:21.803 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.770+0000 7face77c6640 10 monclient: build_initial_monmap 2024-09-11T01:29:21.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.770+0000 7face77c6640 10 monclient: monmap: 2024-09-11T01:29:21.803 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:21.803 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:21.803 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:21.803 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:21.804 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:21.804 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.804 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:21.804 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:21.804 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:21.804 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.770+0000 7f0c571ca640 1 Processor -- start 2024-09-11T01:29:21.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.770+0000 7f0c571ca640 1 -- start start 2024-09-11T01:29:21.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.770+0000 7face77c6640 1 Processor -- start 2024-09-11T01:29:21.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.770+0000 7f0c571ca640 10 monclient: init 2024-09-11T01:29:21.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7face77c6640 1 -- start start 2024-09-11T01:29:21.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7face77c6640 10 monclient: init 2024-09-11T01:29:21.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7face77c6640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:21.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7face77c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:29:21.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7face77c6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7face0137d50 0x7face013b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7face77c6640 10 monclient(hunting): picked mon.noname-b con 0x7face0137d50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:21.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7face77c6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7face0138130 0x7face0136d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7face77c6640 10 monclient(hunting): picked mon.noname-a con 0x7face0138130 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:21.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7face77c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7face013d280 0x7face0139260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7face77c6640 10 monclient(hunting): picked mon.noname-c con 0x7face013d280 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:21.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7face77c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7face77c6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7face009d7d0 con 0x7face0138130 2024-09-11T01:29:21.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7f0c571ca640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:21.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7f0c571ca640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:29:21.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7f0c571ca640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c50137d50 0x7f0c50139cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7f0c571ca640 10 monclient(hunting): picked mon.noname-b con 0x7f0c50137d50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:21.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7f0c571ca640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c50138130 0x7f0c5013a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7face77c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7face77c6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7face009d650 con 0x7face0137d50 2024-09-11T01:29:21.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7face77c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7face77c6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7face0140d90 con 0x7face013d280 2024-09-11T01:29:21.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7face77c6640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:21.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7face77c6640 10 monclient(hunting): authenticate will time out at 768.213704s 2024-09-11T01:29:21.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7face5d63640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7face013d280 0x7face0139260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7face5d63640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7face013d280 0x7face0139260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51776/0 (socket says 172.21.15.12:51776) 2024-09-11T01:29:21.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7face5d63640 1 -- 172.21.15.12:0/3368604333 learned_addr learned my addr 172.21.15.12:0/3368604333 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:21.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7face5d63640 10 monclient(hunting): get_auth_request con 0x7face013d280 auth_method 0 2024-09-11T01:29:21.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7face5d63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7face5d63640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7face5d63640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.771+0000 7face5562640 1 --2- 172.21.15.12:0/3368604333 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7face0137d50 0x7face013b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7face5562640 10 monclient(hunting): get_auth_request con 0x7face0137d50 auth_method 0 2024-09-11T01:29:21.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7face5562640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7face5562640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7face5562640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7face4d61640 1 --2- 172.21.15.12:0/3368604333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7face0138130 0x7face0136d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7face5d63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:21.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7face5d63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:21.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7face5d63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:21.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7face5562640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:21.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7face5562640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:21.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7face5562640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:21.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c54f66640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c50137d50 0x7f0c50139cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c54f66640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c50137d50 0x7f0c50139cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:34904/0 (socket says 172.21.15.12:34904) 2024-09-11T01:29:21.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c4ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c50138130 0x7f0c5013a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c4ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c50138130 0x7f0c5013a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51790/0 (socket says 172.21.15.12:51790) 2024-09-11T01:29:21.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c54f66640 1 -- 172.21.15.12:0/305799846 learned_addr learned my addr 172.21.15.12:0/305799846 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:21.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7face4d61640 10 monclient(hunting): get_auth_request con 0x7face0138130 auth_method 0 2024-09-11T01:29:21.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7face4d61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7face4d61640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c571ca640 10 monclient(hunting): picked mon.noname-c con 0x7f0c50138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:21.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7face4d61640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7face5562640 10 monclient(hunting): handle_auth_done global_id 4270 payload 824 2024-09-11T01:29:21.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7face5562640 1 -- 172.21.15.12:0/3368604333 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7face013d280 msgr2=0x7face0139260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:21.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7face5562640 1 --2- 172.21.15.12:0/3368604333 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7face013d280 0x7face0139260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7face5562640 1 -- 172.21.15.12:0/3368604333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7face0138130 msgr2=0x7face0136d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:21.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7face5562640 1 --2- 172.21.15.12:0/3368604333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7face0138130 0x7face0136d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7face5562640 10 monclient: _finish_hunting 0 2024-09-11T01:29:21.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7face5562640 1 monclient: found mon.noname-b 2024-09-11T01:29:21.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7face5562640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:21.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7face5562640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-09-11T01:29:21.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7face5562640 1 -- 172.21.15.12:0/3368604333 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7face01cdce0 con 0x7face0137d50 2024-09-11T01:29:21.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c571ca640 1 --2- 172.21.15.12:0/305799846 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c5013e290 0x7f0c5013a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7face4d61640 1 --2- 172.21.15.12:0/3368604333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7face0138130 0x7face0136d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:29:21.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c571ca640 10 monclient(hunting): picked mon.noname-a con 0x7f0c5013e290 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:21.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c571ca640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c571ca640 1 -- 172.21.15.12:0/305799846 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0c50140bc0 con 0x7f0c5013e290 2024-09-11T01:29:21.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c571ca640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c571ca640 1 -- 172.21.15.12:0/305799846 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0c500a6da0 con 0x7f0c50137d50 2024-09-11T01:29:21.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c571ca640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c571ca640 1 -- 172.21.15.12:0/305799846 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f0c50140d40 con 0x7f0c50138130 2024-09-11T01:29:21.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c571ca640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:21.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c571ca640 10 monclient(hunting): authenticate will time out at 768.214745s 2024-09-11T01:29:21.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c54f66640 10 monclient(hunting): get_auth_request con 0x7f0c50137d50 auth_method 0 2024-09-11T01:29:21.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c54f66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c54f66640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c54f66640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c4ffff640 10 monclient(hunting): get_auth_request con 0x7f0c50138130 auth_method 0 2024-09-11T01:29:21.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c55767640 1 --2- 172.21.15.12:0/305799846 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c5013e290 0x7f0c5013a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c55767640 10 monclient(hunting): get_auth_request con 0x7f0c5013e290 auth_method 0 2024-09-11T01:29:21.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c55767640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c55767640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c55767640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c54f66640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:21.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c54f66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:21.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c54f66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:21.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c55767640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:21.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c55767640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:21.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.772+0000 7f0c55767640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:21.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.773+0000 7f0c4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:21.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.773+0000 7f0c4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:21.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.773+0000 7f0c4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:21.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.773+0000 7f0c55767640 10 monclient(hunting): handle_auth_done global_id 4302 payload 824 2024-09-11T01:29:21.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.773+0000 7f0c55767640 1 -- 172.21.15.12:0/305799846 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c50138130 msgr2=0x7f0c5013a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:21.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.773+0000 7f0c55767640 1 --2- 172.21.15.12:0/305799846 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c50138130 0x7f0c5013a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.773+0000 7f0c55767640 1 -- 172.21.15.12:0/305799846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c50137d50 msgr2=0x7f0c50139cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:21.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.773+0000 7f0c55767640 1 --2- 172.21.15.12:0/305799846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c50137d50 0x7f0c50139cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.773+0000 7f0c55767640 10 monclient: _finish_hunting 0 2024-09-11T01:29:21.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.773+0000 7f0c55767640 1 monclient: found mon.noname-a 2024-09-11T01:29:21.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.773+0000 7f0c55767640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:21.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.773+0000 7f0c55767640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:21.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.773+0000 7f0c55767640 1 -- 172.21.15.12:0/305799846 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c5013cbf0 con 0x7f0c5013e290 2024-09-11T01:29:21.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.773+0000 7f0c54f66640 1 --2- 172.21.15.12:0/305799846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c50137d50 0x7f0c50139cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:21.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.773+0000 7f0c4ffff640 1 --2- 172.21.15.12:0/305799846 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c50138130 0x7f0c5013a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:21.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.774+0000 7face5562640 1 -- 172.21.15.12:0/3368604333 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7face0137d50 msgr2=0x7face013b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:21.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.774+0000 7face5562640 1 --2- 172.21.15.12:0/3368604333 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7face0137d50 0x7face013b5d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7face009c560 tx=0x7facdc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:21.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.775+0000 7facd5ffb640 1 -- 172.21.15.12:0/3368604333 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facdc04f070 con 0x7face0137d50 2024-09-11T01:29:21.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.775+0000 7facd5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:21.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.775+0000 7facd5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:21.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.775+0000 7facd5ffb640 10 monclient: dump: 2024-09-11T01:29:21.821 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:21.821 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.821 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.821 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.821 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:21.821 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.821 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:21.821 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:21.821 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:21.822 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.775+0000 7facd5ffb640 10 monclient: _finish_auth 0 2024-09-11T01:29:21.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.775+0000 7facd5ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:29:21.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.775+0000 7facd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:21.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.775+0000 7facd5ffb640 1 -- 172.21.15.12:0/3368604333 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facdc030550 con 0x7face0137d50 2024-09-11T01:29:21.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.775+0000 7facd5ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:21.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.775+0000 7facd5ffb640 1 -- 172.21.15.12:0/3368604333 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facdc032a10 con 0x7face0137d50 2024-09-11T01:29:21.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.775+0000 7facd77fe640 4 set_mon_vals no callback set 2024-09-11T01:29:21.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.775+0000 7facd5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:21.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.775+0000 7facd5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:21.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.775+0000 7facd5ffb640 10 monclient: dump: 2024-09-11T01:29:21.822 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:21.822 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.822 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.823 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.823 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:21.823 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.823 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:21.823 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:21.823 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:21.823 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.775+0000 7face77c6640 5 monclient: authenticate success, global_id 4270 2024-09-11T01:29:21.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.775+0000 7face77c6640 10 monclient: _renew_subs 2024-09-11T01:29:21.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.775+0000 7face77c6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:21.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.775+0000 7face77c6640 1 -- 172.21.15.12:0/3368604333 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7face013bf60 con 0x7face0137d50 2024-09-11T01:29:21.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.775+0000 7f0c55767640 1 -- 172.21.15.12:0/305799846 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c5013e290 msgr2=0x7f0c5013a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:21.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.775+0000 7f0c55767640 1 --2- 172.21.15.12:0/305799846 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c5013e290 0x7f0c5013a740 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0c48034000 tx=0x7f0c48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:21.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.776+0000 7face77c6640 10 monclient: _renew_subs 2024-09-11T01:29:21.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.776+0000 7face77c6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:21.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.776+0000 7face77c6640 1 -- 172.21.15.12:0/3368604333 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7face013cdd0 con 0x7face0137d50 2024-09-11T01:29:21.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.775+0000 7f0c4d7fa640 1 -- 172.21.15.12:0/305799846 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c48031d50 con 0x7f0c5013e290 2024-09-11T01:29:21.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.776+0000 7f0c4d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:21.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.776+0000 7f0c4d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:21.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.776+0000 7f0c4d7fa640 10 monclient: dump: 2024-09-11T01:29:21.825 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:21.825 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.825 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.825 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.825 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:21.826 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.826 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:21.826 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:21.826 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:21.826 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.776+0000 7f0c4d7fa640 10 monclient: _finish_auth 0 2024-09-11T01:29:21.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.776+0000 7f0c4d7fa640 10 monclient: _check_auth_tickets 2024-09-11T01:29:21.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.776+0000 7f0c4d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:21.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.776+0000 7f0c4d7fa640 1 -- 172.21.15.12:0/305799846 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c4804d450 con 0x7f0c5013e290 2024-09-11T01:29:21.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.776+0000 7f0c4d7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:21.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.776+0000 7f0c4d7fa640 1 -- 172.21.15.12:0/305799846 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c4804d840 con 0x7f0c5013e290 2024-09-11T01:29:21.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.776+0000 7f0c4d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:21.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.776+0000 7f0c4d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:21.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.776+0000 7f0c4d7fa640 10 monclient: dump: 2024-09-11T01:29:21.827 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:21.827 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.827 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.827 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.827 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:21.827 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.827 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:21.827 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:21.827 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:21.827 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.776+0000 7f0c4effd640 4 set_mon_vals no callback set 2024-09-11T01:29:21.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.776+0000 7f0c571ca640 5 monclient: authenticate success, global_id 4302 2024-09-11T01:29:21.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.776+0000 7f0c571ca640 10 monclient: _renew_subs 2024-09-11T01:29:21.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.776+0000 7f0c571ca640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:21.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.776+0000 7f0c571ca640 1 -- 172.21.15.12:0/305799846 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c5013b700 con 0x7f0c5013e290 2024-09-11T01:29:21.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.776+0000 7f0c571ca640 10 monclient: _renew_subs 2024-09-11T01:29:21.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.776+0000 7f0c571ca640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:21.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.776+0000 7f0c571ca640 1 -- 172.21.15.12:0/305799846 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c5006f060 con 0x7f0c5013e290 2024-09-11T01:29:21.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.777+0000 7facd5ffb640 1 -- 172.21.15.12:0/3368604333 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7facdc030740 con 0x7face0137d50 2024-09-11T01:29:21.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.777+0000 7facd5ffb640 1 --2- 172.21.15.12:0/3368604333 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7facc403e260 0x7facc4040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.777+0000 7face77c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:21.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.777+0000 7facd5ffb640 1 -- 172.21.15.12:0/3368604333 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4402+0+0 (secure 0 0 0) 0x7facdc053730 con 0x7face0137d50 2024-09-11T01:29:21.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.777+0000 7face77c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:21.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.777+0000 7face77c6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:21.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.777+0000 7face77c6640 1 -- 172.21.15.12:0/3368604333 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7face009d190 con 0x7face0137d50 2024-09-11T01:29:21.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.778+0000 7f0c3affd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:21.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.778+0000 7f0c3affd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:21.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.778+0000 7face4d61640 1 --2- 172.21.15.12:0/3368604333 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7facc403e260 0x7facc4040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.778+0000 7face4d61640 10 monclient: get_auth_request con 0x7facc403e260 auth_method 0 2024-09-11T01:29:21.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.778+0000 7f0c3affd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:21.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.778+0000 7f0c3affd640 1 -- 172.21.15.12:0/305799846 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c500008d0 con 0x7f0c5013e290 2024-09-11T01:29:21.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.778+0000 7f0c4d7fa640 1 -- 172.21.15.12:0/305799846 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f0c48055030 con 0x7f0c5013e290 2024-09-11T01:29:21.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.778+0000 7f0c4d7fa640 1 --2- 172.21.15.12:0/305799846 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0c3403e260 0x7f0c34040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.778+0000 7f0c4d7fa640 1 -- 172.21.15.12:0/305799846 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4402+0+0 (secure 0 0 0) 0x7f0c4807c790 con 0x7f0c5013e290 2024-09-11T01:29:21.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.779+0000 7f0c54f66640 1 --2- 172.21.15.12:0/305799846 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0c3403e260 0x7f0c34040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.779+0000 7f0c54f66640 10 monclient: get_auth_request con 0x7f0c3403e260 auth_method 0 2024-09-11T01:29:21.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.780+0000 7face4d61640 1 --2- 172.21.15.12:0/3368604333 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7facc403e260 0x7facc4040720 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7face013b3d0 tx=0x7facd0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:21.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.781+0000 7f0c54f66640 1 --2- 172.21.15.12:0/305799846 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0c3403e260 0x7f0c34040720 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0c44007d50 tx=0x7f0c44007ca0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:21.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.783+0000 7facd5ffb640 1 -- 172.21.15.12:0/3368604333 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7facdc045070 con 0x7face0137d50 2024-09-11T01:29:21.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.783+0000 7f0c4d7fa640 1 -- 172.21.15.12:0/305799846 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0c4804ccd0 con 0x7f0c5013e290 2024-09-11T01:29:21.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.783+0000 7f0c4d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:21.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.783+0000 7f0c4d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:21.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.783+0000 7facd5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:21.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.783+0000 7facd5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:21.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.831+0000 7fdca462a640 10 monclient: get_monmap_and_config 2024-09-11T01:29:21.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.831+0000 7fdca462a640 10 monclient: build_initial_monmap 2024-09-11T01:29:21.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.831+0000 7fdca462a640 10 monclient: monmap: 2024-09-11T01:29:21.834 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:21.834 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.834 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:21.831948+0000 2024-09-11T01:29:21.834 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:21.831948+0000 2024-09-11T01:29:21.834 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:21.834 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.834 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:21.834 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:21.835 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:21.835 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:21.835 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:21.835 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.832+0000 7fdca462a640 1 Processor -- start 2024-09-11T01:29:21.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.832+0000 7fdca462a640 1 -- start start 2024-09-11T01:29:21.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.832+0000 7fdca462a640 10 monclient: init 2024-09-11T01:29:21.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.833+0000 7fdca462a640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:21.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.833+0000 7fdca462a640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-11T01:29:21.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.833+0000 7fdca462a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdc9c137d50 0x7fdc9c09d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.833+0000 7fdca462a640 10 monclient(hunting): picked mon.noname-a con 0x7fdc9c137d50 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:21.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.833+0000 7fdca462a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdc9c138130 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:21.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.833+0000 7fdca462a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdc9c13d280 0x7fdc9c13f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.833+0000 7fdca462a640 10 monclient(hunting): picked mon.noname-b con 0x7fdc9c13d280 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:21.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.833+0000 7fdca23c6640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdc9c137d50 0x7fdc9c09d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.833+0000 7fdca23c6640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdc9c137d50 0x7fdc9c09d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55064/0 (socket says 172.21.15.12:55064) 2024-09-11T01:29:21.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.833+0000 7fdca23c6640 1 -- 172.21.15.12:0/2813413851 learned_addr learned my addr 172.21.15.12:0/2813413851 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:21.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.834+0000 7fdca2bc7640 1 --2- 172.21.15.12:0/2813413851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdc9c13d280 0x7fdc9c13f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.834+0000 7fdca462a640 1 -- 172.21.15.12:0/2813413851 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc9c09ba20 con 0x7fdc9c138130 2024-09-11T01:29:21.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.834+0000 7fdca462a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.834+0000 7fdca462a640 1 -- 172.21.15.12:0/2813413851 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fdc9c09c650 con 0x7fdc9c137d50 2024-09-11T01:29:21.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.834+0000 7fdca462a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.834+0000 7fdca462a640 1 -- 172.21.15.12:0/2813413851 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fdc9c09c370 con 0x7fdc9c13d280 2024-09-11T01:29:21.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.834+0000 7fdca462a640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:21.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.834+0000 7fdca462a640 10 monclient(hunting): authenticate will time out at 768.276651s 2024-09-11T01:29:21.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.834+0000 7fdca23c6640 10 monclient(hunting): get_auth_request con 0x7fdc9c137d50 auth_method 0 2024-09-11T01:29:21.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.834+0000 7fdca23c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.834+0000 7fdca23c6640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.834+0000 7fdca23c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.834+0000 7fdca2bc7640 10 monclient(hunting): get_auth_request con 0x7fdc9c13d280 auth_method 0 2024-09-11T01:29:21.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.834+0000 7fdca2bc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.834+0000 7fdca2bc7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.834+0000 7fdca2bc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.834+0000 7fdca23c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:21.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.834+0000 7fdca23c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:21.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.834+0000 7fdca23c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:21.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.834+0000 7fdca2bc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:21.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.834+0000 7fdca2bc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:21.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.834+0000 7fdca2bc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:21.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.835+0000 7fdca2bc7640 10 monclient(hunting): handle_auth_done global_id 4328 payload 274 2024-09-11T01:29:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.835+0000 7fdca2bc7640 1 -- 172.21.15.12:0/2813413851 >> v2:172.21.15.12:3300/0 conn(0x7fdc9c137d50 msgr2=0x7fdc9c09d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.835+0000 7fdca2bc7640 1 --2- 172.21.15.12:0/2813413851 >> v2:172.21.15.12:3300/0 conn(0x7fdc9c137d50 0x7fdc9c09d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.835+0000 7fdca2bc7640 1 -- 172.21.15.12:0/2813413851 >> v1:172.21.15.71:6789/0 conn(0x7fdc9c138130 legacy=0x7fdc9c138580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.835+0000 7fdca2bc7640 10 monclient: _finish_hunting 0 2024-09-11T01:29:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.835+0000 7fdca2bc7640 1 monclient: found mon.noname-b 2024-09-11T01:29:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.835+0000 7fdca2bc7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.835+0000 7fdca2bc7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:29:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.835+0000 7fdca2bc7640 1 -- 172.21.15.12:0/2813413851 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc9c0a6ce0 con 0x7fdc9c13d280 2024-09-11T01:29:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.835+0000 7fdca23c6640 1 --2- 172.21.15.12:0/2813413851 >> v2:172.21.15.12:3300/0 conn(0x7fdc9c137d50 0x7fdc9c09d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.838+0000 7fdca2bc7640 1 -- 172.21.15.12:0/2813413851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdc9c13d280 msgr2=0x7fdc9c13f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.838+0000 7fdca2bc7640 1 --2- 172.21.15.12:0/2813413851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdc9c13d280 0x7fdc9c13f670 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdc9400b050 tx=0x7fdc9402a7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1d1ee3a1399f27a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.838+0000 7fdca13c4640 1 -- 172.21.15.12:0/2813413851 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc94007740 con 0x7fdc9c13d280 2024-09-11T01:29:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.838+0000 7fdca13c4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.838+0000 7fdca13c4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:21.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.838+0000 7fdca13c4640 10 monclient: dump: 2024-09-11T01:29:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:21.848 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.838+0000 7fdca13c4640 10 monclient: _finish_auth 0 2024-09-11T01:29:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.838+0000 7fdca13c4640 10 monclient: _check_auth_tickets 2024-09-11T01:29:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.838+0000 7fdca13c4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:21.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.838+0000 7fdca13c4640 1 -- 172.21.15.12:0/2813413851 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc940078e0 con 0x7fdc9c13d280 2024-09-11T01:29:21.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.838+0000 7fdca462a640 5 monclient: authenticate success, global_id 4328 2024-09-11T01:29:21.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.838+0000 7fdca462a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:21.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.838+0000 7fdca13c4640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:21.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.838+0000 7fdca462a640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:21.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.838+0000 7fdca13c4640 1 -- 172.21.15.12:0/2813413851 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc94007cb0 con 0x7fdc9c13d280 2024-09-11T01:29:21.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.838+0000 7fdca462a640 4 set_mon_vals no callback set 2024-09-11T01:29:21.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.838+0000 7fdca13c4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:21.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.838+0000 7fdca13c4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:21.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.838+0000 7fdca13c4640 10 monclient: dump: 2024-09-11T01:29:21.849 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:21.849 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.849 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.849 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.849 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:21.850 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.850 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:21.850 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:21.850 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:21.850 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.839+0000 7fdca462a640 10 monclient: shutdown 2024-09-11T01:29:21.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.839+0000 7fdca462a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:21.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.839+0000 7fdca462a640 1 -- 172.21.15.12:0/2813413851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdc9c13d280 msgr2=0x7fdc9c13f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:21.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.839+0000 7fdca462a640 1 --2- 172.21.15.12:0/2813413851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdc9c13d280 0x7fdc9c13f670 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdc9400b050 tx=0x7fdc9402a7a0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.840+0000 7fdca462a640 1 -- 172.21.15.12:0/2813413851 shutdown_connections 2024-09-11T01:29:21.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.840+0000 7fdca462a640 1 --2- 172.21.15.12:0/2813413851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdc9c13d280 0x7fdc9c13f670 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.840+0000 7fdca462a640 1 --2- 172.21.15.12:0/2813413851 >> v2:172.21.15.12:3300/0 conn(0x7fdc9c137d50 0x7fdc9c09d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.840+0000 7fdca462a640 1 -- 172.21.15.12:0/2813413851 >> 172.21.15.12:0/2813413851 conn(0x7fdc9c0a7020 msgr2=0x7fdc9c0a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:21.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.840+0000 7fdca462a640 1 -- 172.21.15.12:0/2813413851 shutdown_connections 2024-09-11T01:29:21.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.840+0000 7fdca462a640 1 -- 172.21.15.12:0/2813413851 wait complete. 2024-09-11T01:29:21.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.841+0000 7fdca462a640 10 monclient: build_initial_monmap 2024-09-11T01:29:21.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.841+0000 7fdca462a640 10 monclient: monmap: 2024-09-11T01:29:21.851 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:21.851 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:21.851 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:21.851 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:21.851 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:21.851 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.851 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:21.851 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:21.851 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:21.852 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.842+0000 7fdca462a640 1 Processor -- start 2024-09-11T01:29:21.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.842+0000 7fdca462a640 1 -- start start 2024-09-11T01:29:21.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.842+0000 7fdca462a640 10 monclient: init 2024-09-11T01:29:21.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.843+0000 7fdca462a640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:21.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.843+0000 7fdca462a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:29:21.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.843+0000 7fdca462a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdc9c137d50 0x7fdc9c1ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca462a640 10 monclient(hunting): picked mon.noname-c con 0x7fdc9c137d50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca462a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdc9c138130 0x7fdc9c1cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca23c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdc9c137d50 0x7fdc9c1ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca23c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdc9c137d50 0x7fdc9c1ccfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51808/0 (socket says 172.21.15.12:51808) 2024-09-11T01:29:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca23c6640 1 -- 172.21.15.12:0/2187736890 learned_addr learned my addr 172.21.15.12:0/2187736890 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca462a640 10 monclient(hunting): picked mon.noname-b con 0x7fdc9c138130 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca462a640 1 --2- 172.21.15.12:0/2187736890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc9c13d280 0x7fdc9c139d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca462a640 10 monclient(hunting): picked mon.noname-a con 0x7fdc9c13d280 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca462a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca462a640 1 -- 172.21.15.12:0/2187736890 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdc9c09d7d0 con 0x7fdc9c13d280 2024-09-11T01:29:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca462a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca462a640 1 -- 172.21.15.12:0/2187736890 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fdc9c09d650 con 0x7fdc9c138130 2024-09-11T01:29:21.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca462a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:21.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca462a640 1 -- 172.21.15.12:0/2187736890 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fdc9c13fbb0 con 0x7fdc9c137d50 2024-09-11T01:29:21.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca462a640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:21.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca462a640 10 monclient(hunting): authenticate will time out at 768.286493s 2024-09-11T01:29:21.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca23c6640 10 monclient(hunting): get_auth_request con 0x7fdc9c137d50 auth_method 0 2024-09-11T01:29:21.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca23c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca23c6640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca23c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca1bc5640 1 --2- 172.21.15.12:0/2187736890 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdc9c138130 0x7fdc9c1cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca1bc5640 10 monclient(hunting): get_auth_request con 0x7fdc9c138130 auth_method 0 2024-09-11T01:29:21.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca1bc5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca1bc5640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca1bc5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca1bc5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca1bc5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca1bc5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca2bc7640 1 --2- 172.21.15.12:0/2187736890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc9c13d280 0x7fdc9c139d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca23c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca23c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca23c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca2bc7640 10 monclient(hunting): get_auth_request con 0x7fdc9c13d280 auth_method 0 2024-09-11T01:29:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca2bc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca2bc7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca2bc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca2bc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca2bc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:21.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca2bc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.844+0000 7fdca1bc5640 10 monclient(hunting): handle_auth_done global_id 4276 payload 824 2024-09-11T01:29:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.845+0000 7fdca1bc5640 1 -- 172.21.15.12:0/2187736890 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdc9c137d50 msgr2=0x7fdc9c1ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.845+0000 7fdca1bc5640 1 --2- 172.21.15.12:0/2187736890 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdc9c137d50 0x7fdc9c1ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.845+0000 7fdca1bc5640 1 -- 172.21.15.12:0/2187736890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc9c13d280 msgr2=0x7fdc9c139d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.845+0000 7fdca1bc5640 1 --2- 172.21.15.12:0/2187736890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc9c13d280 0x7fdc9c139d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.845+0000 7fdca1bc5640 10 monclient: _finish_hunting 0 2024-09-11T01:29:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.845+0000 7fdca1bc5640 1 monclient: found mon.noname-b 2024-09-11T01:29:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.845+0000 7fdca1bc5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.845+0000 7fdca1bc5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-09-11T01:29:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.845+0000 7fdca1bc5640 1 -- 172.21.15.12:0/2187736890 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc9c120dc0 con 0x7fdc9c138130 2024-09-11T01:29:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.845+0000 7fdca2bc7640 1 --2- 172.21.15.12:0/2187736890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc9c13d280 0x7fdc9c139d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.847+0000 7fdca1bc5640 1 -- 172.21.15.12:0/2187736890 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdc9c138130 msgr2=0x7fdc9c1cd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:21.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.847+0000 7fdca1bc5640 1 --2- 172.21.15.12:0/2187736890 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdc9c138130 0x7fdc9c1cd4f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c00c410 tx=0x7fdc8c00c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.848+0000 7fdc92ffd640 1 -- 172.21.15.12:0/2187736890 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c017070 con 0x7fdc9c138130 2024-09-11T01:29:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.848+0000 7fdc92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.848+0000 7fdc92ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.848+0000 7fdc92ffd640 10 monclient: dump: 2024-09-11T01:29:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:21.863 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.848+0000 7fdc92ffd640 10 monclient: _finish_auth 0 2024-09-11T01:29:21.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.848+0000 7fdc92ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:21.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.848+0000 7fdc92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:21.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.848+0000 7fdc92ffd640 1 -- 172.21.15.12:0/2187736890 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc8c005230 con 0x7fdc9c138130 2024-09-11T01:29:21.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.848+0000 7fdc92ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:21.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.848+0000 7fdc92ffd640 1 -- 172.21.15.12:0/2187736890 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c0117a0 con 0x7fdc9c138130 2024-09-11T01:29:21.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.848+0000 7fdc92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:21.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.848+0000 7fdc92ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:21.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.848+0000 7fdc92ffd640 10 monclient: dump: 2024-09-11T01:29:21.864 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:21.864 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:21.864 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.864 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:21.864 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:21.864 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:21.865 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:21.865 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:21.865 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:21.865 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:21.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.848+0000 7fdca462a640 5 monclient: authenticate success, global_id 4276 2024-09-11T01:29:21.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.848+0000 7fdca462a640 10 monclient: _renew_subs 2024-09-11T01:29:21.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.848+0000 7fdca462a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:21.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.848+0000 7fdca0bc3640 4 set_mon_vals no callback set 2024-09-11T01:29:21.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.848+0000 7fdca462a640 1 -- 172.21.15.12:0/2187736890 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc9c1210a0 con 0x7fdc9c138130 2024-09-11T01:29:21.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.848+0000 7fdca462a640 10 monclient: _renew_subs 2024-09-11T01:29:21.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.848+0000 7fdca462a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:21.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.848+0000 7fdca462a640 1 -- 172.21.15.12:0/2187736890 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc9c121380 con 0x7fdc9c138130 2024-09-11T01:29:21.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.849+0000 7fdca462a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:21.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.849+0000 7fdca462a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:21.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.849+0000 7fdca462a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:21.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.849+0000 7fdca462a640 1 -- 172.21.15.12:0/2187736890 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc9c13ce30 con 0x7fdc9c138130 2024-09-11T01:29:21.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.849+0000 7fdc92ffd640 1 -- 172.21.15.12:0/2187736890 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fdc8c0119c0 con 0x7fdc9c138130 2024-09-11T01:29:21.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.850+0000 7fdc92ffd640 1 --2- 172.21.15.12:0/2187736890 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdc8003e260 0x7fdc80040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:21.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.850+0000 7fdc92ffd640 1 -- 172.21.15.12:0/2187736890 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4402+0+0 (secure 0 0 0) 0x7fdc8c0522c0 con 0x7fdc9c138130 2024-09-11T01:29:21.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.850+0000 7fdca23c6640 1 --2- 172.21.15.12:0/2187736890 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdc8003e260 0x7fdc80040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:21.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.850+0000 7fdca23c6640 10 monclient: get_auth_request con 0x7fdc8003e260 auth_method 0 2024-09-11T01:29:21.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.853+0000 7fdca23c6640 1 --2- 172.21.15.12:0/2187736890 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdc8003e260 0x7fdc80040720 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdc98006ca0 tx=0x7fdc98009240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:21.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.854+0000 7fdc92ffd640 1 -- 172.21.15.12:0/2187736890 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fdc8c004de0 con 0x7fdc9c138130 2024-09-11T01:29:21.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.854+0000 7fdc92ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:21.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.854+0000 7fdc92ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:21.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.933+0000 7f6fbca2a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-11T01:29:21.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.933+0000 7f6fbca2a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-11T01:29:21.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.933+0000 7f6fbca2a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:21.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.933+0000 7f6fbca2a640 1 -- 172.21.15.12:0/611486321 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f6fb8126ef0 con 0x7f6fb8138130 2024-09-11T01:29:21.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.933+0000 7f6faeffd640 1 -- 172.21.15.12:0/611486321 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6fb0012740 con 0x7f6fb8138130 2024-09-11T01:29:21.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.933+0000 7f6faeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-11T01:29:21.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.933+0000 7f6faeffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:21.936 INFO:teuthology.orchestra.run.smithi012.stdout:51539607556 2024-09-11T01:29:21.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.938+0000 7f6fbca2a640 1 -- 172.21.15.12:0/611486321 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6f9803e200 msgr2=0x7f6f980406c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:21.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.938+0000 7f6fbca2a640 1 --2- 172.21.15.12:0/611486321 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6f9803e200 0x7f6f980406c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6fb80a3670 tx=0x7f6fa802e000 comp rx=0 tx=0).stop 2024-09-11T01:29:21.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.938+0000 7f6fbca2a640 10 monclient: shutdown 2024-09-11T01:29:21.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.938+0000 7f6fbca2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:21.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.938+0000 7f6fbca2a640 1 -- 172.21.15.12:0/611486321 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6fb8138130 msgr2=0x7f6fb8123ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:21.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.938+0000 7f6fbca2a640 1 --2- 172.21.15.12:0/611486321 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6fb8138130 0x7f6fb8123ad0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6fb000c780 tx=0x7f6fb000cc40 comp rx=0 tx=0).stop 2024-09-11T01:29:21.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.938+0000 7f6fbca2a640 1 -- 172.21.15.12:0/611486321 shutdown_connections 2024-09-11T01:29:21.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.938+0000 7f6fbca2a640 1 --2- 172.21.15.12:0/611486321 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6fb8138130 0x7f6fb8123ad0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.938+0000 7f6fbca2a640 1 --2- 172.21.15.12:0/611486321 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6fb8137d50 0x7f6fb8123590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.938+0000 7f6fbca2a640 1 --2- 172.21.15.12:0/611486321 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6f9803e200 0x7f6f980406c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.938+0000 7f6fbca2a640 1 --2- 172.21.15.12:0/611486321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6fb8136cd0 0x7f6fb81cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.939+0000 7f6fbca2a640 1 -- 172.21.15.12:0/611486321 >> 172.21.15.12:0/611486321 conn(0x7f6fb80a7020 msgr2=0x7f6fb80a0050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:21.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.939+0000 7f6fbca2a640 1 -- 172.21.15.12:0/611486321 shutdown_connections 2024-09-11T01:29:21.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.939+0000 7f6fbca2a640 1 -- 172.21.15.12:0/611486321 wait complete. 2024-09-11T01:29:21.956 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607556 for osd.5 2024-09-11T01:29:21.956 DEBUG:teuthology.parallel:result is None 2024-09-11T01:29:21.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.974+0000 7efc069bb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-11T01:29:21.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.974+0000 7efc069bb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-11T01:29:21.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.974+0000 7efc069bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:21.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.974+0000 7efc069bb640 1 -- 172.21.15.12:0/2607219355 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7efc00138550 con 0x7efc00137d50 2024-09-11T01:29:21.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.974+0000 7efbfcff9640 1 -- 172.21.15.12:0/2607219355 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7efc00120fc0 con 0x7efc00137d50 2024-09-11T01:29:21.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.974+0000 7efbfcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-11T01:29:21.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.974+0000 7efbfcff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:21.976 INFO:teuthology.orchestra.run.smithi012.stdout:51539607556 2024-09-11T01:29:21.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.977+0000 7efc069bb640 1 -- 172.21.15.12:0/2607219355 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efbec03e1c0 msgr2=0x7efbec040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:21.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.977+0000 7efc069bb640 1 --2- 172.21.15.12:0/2607219355 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efbec03e1c0 0x7efbec040680 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efbf4006e80 tx=0x7efbf4008040 comp rx=0 tx=0).stop 2024-09-11T01:29:21.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.977+0000 7efc069bb640 10 monclient: shutdown 2024-09-11T01:29:21.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.977+0000 7efc069bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:21.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.977+0000 7efc069bb640 1 -- 172.21.15.12:0/2607219355 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc00137d50 msgr2=0x7efc001ccee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:21.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.977+0000 7efc069bb640 1 --2- 172.21.15.12:0/2607219355 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc00137d50 0x7efc001ccee0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efbf0005580 tx=0x7efbf0014420 comp rx=0 tx=0).stop 2024-09-11T01:29:21.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.977+0000 7efc069bb640 1 -- 172.21.15.12:0/2607219355 shutdown_connections 2024-09-11T01:29:21.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.977+0000 7efc069bb640 1 --2- 172.21.15.12:0/2607219355 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efbec03e1c0 0x7efbec040680 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.977+0000 7efc069bb640 1 --2- 172.21.15.12:0/2607219355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc0013d280 0x7efc0013a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.977+0000 7efc069bb640 1 --2- 172.21.15.12:0/2607219355 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc00138130 0x7efc00136ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.977+0000 7efc069bb640 1 --2- 172.21.15.12:0/2607219355 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc00137d50 0x7efc001ccee0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.977+0000 7efc069bb640 1 -- 172.21.15.12:0/2607219355 >> 172.21.15.12:0/2607219355 conn(0x7efc000a7020 msgr2=0x7efc0013dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:21.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.977+0000 7efc069bb640 1 -- 172.21.15.12:0/2607219355 shutdown_connections 2024-09-11T01:29:21.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.977+0000 7efc069bb640 1 -- 172.21.15.12:0/2607219355 wait complete. 2024-09-11T01:29:21.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.980+0000 7f0c3affd640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-11T01:29:21.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.980+0000 7f0c3affd640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-11T01:29:21.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.980+0000 7f0c3affd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:21.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.980+0000 7f0c3affd640 1 -- 172.21.15.12:0/305799846 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f0c5013d4f0 con 0x7f0c5013e290 2024-09-11T01:29:21.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.980+0000 7f0c4d7fa640 1 -- 172.21.15.12:0/305799846 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0c48000f60 con 0x7f0c5013e290 2024-09-11T01:29:21.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.980+0000 7f0c4d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-11T01:29:21.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.980+0000 7f0c4d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:21.982 INFO:teuthology.orchestra.run.smithi012.stdout:51539607556 2024-09-11T01:29:21.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.983+0000 7f0c3affd640 1 -- 172.21.15.12:0/305799846 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0c3403e260 msgr2=0x7f0c34040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:21.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.983+0000 7f0c3affd640 1 --2- 172.21.15.12:0/305799846 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0c3403e260 0x7f0c34040720 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0c44007d50 tx=0x7f0c44007ca0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.983+0000 7f0c3affd640 10 monclient: shutdown 2024-09-11T01:29:21.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.983+0000 7f0c3affd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:21.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.983+0000 7f0c3affd640 1 -- 172.21.15.12:0/305799846 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c5013e290 msgr2=0x7f0c5013a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:21.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.983+0000 7f0c3affd640 1 --2- 172.21.15.12:0/305799846 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c5013e290 0x7f0c5013a740 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0c48034000 tx=0x7f0c48004a40 comp rx=0 tx=0).stop 2024-09-11T01:29:21.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.983+0000 7f0c3affd640 1 -- 172.21.15.12:0/305799846 shutdown_connections 2024-09-11T01:29:21.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.983+0000 7f0c3affd640 1 --2- 172.21.15.12:0/305799846 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0c3403e260 0x7f0c34040720 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.983+0000 7f0c3affd640 1 --2- 172.21.15.12:0/305799846 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c5013e290 0x7f0c5013a740 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.983+0000 7f0c3affd640 1 --2- 172.21.15.12:0/305799846 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c50138130 0x7f0c5013a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.983+0000 7f0c3affd640 1 --2- 172.21.15.12:0/305799846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c50137d50 0x7f0c50139cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:21.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.983+0000 7f0c3affd640 1 -- 172.21.15.12:0/305799846 >> 172.21.15.12:0/305799846 conn(0x7f0c500a7020 msgr2=0x7f0c50140420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:21.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.983+0000 7f0c3affd640 1 -- 172.21.15.12:0/305799846 shutdown_connections 2024-09-11T01:29:21.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:21.983+0000 7f0c3affd640 1 -- 172.21.15.12:0/305799846 wait complete. 2024-09-11T01:29:21.990 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607556 for osd.0 2024-09-11T01:29:21.990 DEBUG:teuthology.parallel:result is None 2024-09-11T01:29:21.996 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607556 for osd.3 2024-09-11T01:29:21.996 DEBUG:teuthology.parallel:result is None 2024-09-11T01:29:22.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.000+0000 7f464cbc2640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-11T01:29:22.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.000+0000 7f464cbc2640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-11T01:29:22.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.000+0000 7f464cbc2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:22.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.000+0000 7f464cbc2640 1 -- 172.21.15.12:0/329908919 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f46480a2890 con 0x7f4648138130 2024-09-11T01:29:22.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.001+0000 7f463effd640 1 -- 172.21.15.12:0/329908919 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4640013390 con 0x7f4648138130 2024-09-11T01:29:22.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.001+0000 7f463effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-11T01:29:22.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.001+0000 7f463effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:22.003 INFO:teuthology.orchestra.run.smithi012.stdout:47244640260 2024-09-11T01:29:22.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.004+0000 7f464cbc2640 1 -- 172.21.15.12:0/329908919 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f462803e210 msgr2=0x7f46280406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:22.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.004+0000 7f464cbc2640 1 --2- 172.21.15.12:0/329908919 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f462803e210 0x7f46280406d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4638007e00 tx=0x7f4638009000 comp rx=0 tx=0).stop 2024-09-11T01:29:22.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.004+0000 7f464cbc2640 10 monclient: shutdown 2024-09-11T01:29:22.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.004+0000 7f464cbc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:22.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.004+0000 7f464cbc2640 1 -- 172.21.15.12:0/329908919 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4648138130 msgr2=0x7f4648073230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:22.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.004+0000 7f464cbc2640 1 --2- 172.21.15.12:0/329908919 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4648138130 0x7f4648073230 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f464000d530 tx=0x7f464000d9f0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.004+0000 7f464cbc2640 1 -- 172.21.15.12:0/329908919 shutdown_connections 2024-09-11T01:29:22.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.004+0000 7f464cbc2640 1 --2- 172.21.15.12:0/329908919 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4648138130 0x7f4648073230 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.004+0000 7f464cbc2640 1 --2- 172.21.15.12:0/329908919 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4648137d50 0x7f4648072cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.004+0000 7f464cbc2640 1 --2- 172.21.15.12:0/329908919 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f462803e210 0x7f46280406d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.004+0000 7f464cbc2640 1 --2- 172.21.15.12:0/329908919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4648136cd0 0x7f46480727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.004+0000 7f464cbc2640 1 -- 172.21.15.12:0/329908919 >> 172.21.15.12:0/329908919 conn(0x7f46480a7020 msgr2=0x7f46480a3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:22.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.004+0000 7f464cbc2640 1 -- 172.21.15.12:0/329908919 shutdown_connections 2024-09-11T01:29:22.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.004+0000 7f464cbc2640 1 -- 172.21.15.12:0/329908919 wait complete. 2024-09-11T01:29:22.020 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.2 2024-09-11T01:29:22.020 DEBUG:teuthology.parallel:result is None 2024-09-11T01:29:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.058+0000 7face77c6640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-11T01:29:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.058+0000 7face77c6640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-11T01:29:22.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.058+0000 7face77c6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:22.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.058+0000 7face77c6640 1 -- 172.21.15.12:0/3368604333 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7face0074a40 con 0x7face0137d50 2024-09-11T01:29:22.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.059+0000 7facd5ffb640 1 -- 172.21.15.12:0/3368604333 <== mon.1 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7facdc0533b0 con 0x7face0137d50 2024-09-11T01:29:22.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.059+0000 7facd5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-11T01:29:22.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.059+0000 7facd5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:22.061 INFO:teuthology.orchestra.run.smithi012.stdout:51539607556 2024-09-11T01:29:22.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.061+0000 7face77c6640 1 -- 172.21.15.12:0/3368604333 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7facc403e260 msgr2=0x7facc4040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:22.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.061+0000 7face77c6640 1 --2- 172.21.15.12:0/3368604333 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7facc403e260 0x7facc4040720 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7face013b3d0 tx=0x7facd0009040 comp rx=0 tx=0).stop 2024-09-11T01:29:22.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.061+0000 7face77c6640 10 monclient: shutdown 2024-09-11T01:29:22.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.061+0000 7face77c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:22.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.061+0000 7face77c6640 1 -- 172.21.15.12:0/3368604333 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7face0137d50 msgr2=0x7face013b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:22.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.061+0000 7face77c6640 1 --2- 172.21.15.12:0/3368604333 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7face0137d50 0x7face013b5d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7face009c560 tx=0x7facdc0046c0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.061+0000 7face77c6640 1 -- 172.21.15.12:0/3368604333 shutdown_connections 2024-09-11T01:29:22.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.061+0000 7face77c6640 1 --2- 172.21.15.12:0/3368604333 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7face013d280 0x7face0139260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.061+0000 7face77c6640 1 --2- 172.21.15.12:0/3368604333 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7facc403e260 0x7facc4040720 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.061+0000 7face77c6640 1 --2- 172.21.15.12:0/3368604333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7face0138130 0x7face0136d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.061+0000 7face77c6640 1 --2- 172.21.15.12:0/3368604333 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7face0137d50 0x7face013b5d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.061+0000 7face77c6640 1 -- 172.21.15.12:0/3368604333 >> 172.21.15.12:0/3368604333 conn(0x7face00a7020 msgr2=0x7face00a25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:22.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.061+0000 7face77c6640 1 -- 172.21.15.12:0/3368604333 shutdown_connections 2024-09-11T01:29:22.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.061+0000 7face77c6640 1 -- 172.21.15.12:0/3368604333 wait complete. 2024-09-11T01:29:22.073 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607556 for osd.4 2024-09-11T01:29:22.073 DEBUG:teuthology.parallel:result is None 2024-09-11T01:29:22.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.124+0000 7fdca462a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-11T01:29:22.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.124+0000 7fdca462a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-11T01:29:22.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.124+0000 7fdca462a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:22.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.124+0000 7fdca462a640 1 -- 172.21.15.12:0/2187736890 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fdc9c13c6a0 con 0x7fdc9c138130 2024-09-11T01:29:22.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.124+0000 7fdc92ffd640 1 -- 172.21.15.12:0/2187736890 <== mon.1 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdc8c0192c0 con 0x7fdc9c138130 2024-09-11T01:29:22.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.124+0000 7fdc92ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-11T01:29:22.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.124+0000 7fdc92ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:22.127 INFO:teuthology.orchestra.run.smithi012.stdout:51539607555 2024-09-11T01:29:22.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.127+0000 7fdca462a640 1 -- 172.21.15.12:0/2187736890 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdc8003e260 msgr2=0x7fdc80040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:22.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.127+0000 7fdca462a640 1 --2- 172.21.15.12:0/2187736890 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdc8003e260 0x7fdc80040720 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdc98006ca0 tx=0x7fdc98009240 comp rx=0 tx=0).stop 2024-09-11T01:29:22.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.127+0000 7fdca462a640 10 monclient: shutdown 2024-09-11T01:29:22.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.127+0000 7fdca462a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:22.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.127+0000 7fdca462a640 1 -- 172.21.15.12:0/2187736890 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdc9c138130 msgr2=0x7fdc9c1cd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:22.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.127+0000 7fdca462a640 1 --2- 172.21.15.12:0/2187736890 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdc9c138130 0x7fdc9c1cd4f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c00c410 tx=0x7fdc8c00c8d0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.127+0000 7fdca462a640 1 -- 172.21.15.12:0/2187736890 shutdown_connections 2024-09-11T01:29:22.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.127+0000 7fdca462a640 1 --2- 172.21.15.12:0/2187736890 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdc8003e260 0x7fdc80040720 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.127+0000 7fdca462a640 1 --2- 172.21.15.12:0/2187736890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc9c13d280 0x7fdc9c139d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.127+0000 7fdca462a640 1 --2- 172.21.15.12:0/2187736890 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdc9c138130 0x7fdc9c1cd4f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.127+0000 7fdca462a640 1 --2- 172.21.15.12:0/2187736890 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdc9c137d50 0x7fdc9c1ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.127+0000 7fdca462a640 1 -- 172.21.15.12:0/2187736890 >> 172.21.15.12:0/2187736890 conn(0x7fdc9c0a7020 msgr2=0x7fdc9c0a3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:22.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.127+0000 7fdca462a640 1 -- 172.21.15.12:0/2187736890 shutdown_connections 2024-09-11T01:29:22.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.127+0000 7fdca462a640 1 -- 172.21.15.12:0/2187736890 wait complete. 2024-09-11T01:29:22.139 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607555 got 51539607555 for osd.1 2024-09-11T01:29:22.139 DEBUG:teuthology.parallel:result is None 2024-09-11T01:29:22.139 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T01:29:22.139 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T01:29:22.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.238+0000 7f501619b640 10 monclient: get_monmap_and_config 2024-09-11T01:29:22.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.238+0000 7f501619b640 10 monclient: build_initial_monmap 2024-09-11T01:29:22.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.238+0000 7f501619b640 10 monclient: monmap: 2024-09-11T01:29:22.240 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:22.240 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:22.240 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:22.239399+0000 2024-09-11T01:29:22.241 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:22.239399+0000 2024-09-11T01:29:22.241 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:22.241 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:22.241 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:22.241 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:22.241 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:22.241 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:22.241 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:22.241 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:22.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.239+0000 7f501619b640 1 Processor -- start 2024-09-11T01:29:22.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.239+0000 7f501619b640 1 -- start start 2024-09-11T01:29:22.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.239+0000 7f501619b640 10 monclient: init 2024-09-11T01:29:22.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.239+0000 7f501619b640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:22.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f501619b640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-11T01:29:22.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f501619b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f50101381c0 0x7f50101385a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:22.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f501619b640 10 monclient(hunting): picked mon.noname-b con 0x7f50101381c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:22.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f501619b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f50100a6aa0 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:22.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f501619b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f501013d000 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:22.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f501619b640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f501009c310 con 0x7f501013d000 2024-09-11T01:29:22.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f501619b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f501009c8d0 con 0x7f50100a6aa0 2024-09-11T01:29:22.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f501619b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:22.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f501619b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f501006fc90 con 0x7f50101381c0 2024-09-11T01:29:22.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f501619b640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:22.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f501619b640 10 monclient(hunting): authenticate will time out at 768.682198s 2024-09-11T01:29:22.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f500f7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f50101381c0 0x7f50101385a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:22.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f500f7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f50101381c0 0x7f50101385a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51818/0 (socket says 172.21.15.12:51818) 2024-09-11T01:29:22.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f500f7fe640 1 -- 172.21.15.12:0/3345543635 learned_addr learned my addr 172.21.15.12:0/3345543635 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:22.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f500f7fe640 10 monclient(hunting): get_auth_request con 0x7f50101381c0 auth_method 0 2024-09-11T01:29:22.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f500f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:22.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f500f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:22.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f500f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:22.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f500ffff640 1 --1- 172.21.15.12:0/3345543635 >> v1:172.21.15.12:6789/0 conn(0x7f501013d000 0x7f501013f3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56106/0 (socket says 172.21.15.12:56106) 2024-09-11T01:29:22.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f500f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:22.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f500f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:22.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f500f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:22.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f500f7fe640 10 monclient(hunting): handle_auth_done global_id 4334 payload 274 2024-09-11T01:29:22.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f500f7fe640 1 -- 172.21.15.12:0/3345543635 >> v1:172.21.15.71:6789/0 conn(0x7f50100a6aa0 legacy=0x7f501009cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:22.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f500f7fe640 1 -- 172.21.15.12:0/3345543635 >> v1:172.21.15.12:6789/0 conn(0x7f501013d000 legacy=0x7f501013f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:22.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f500f7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:29:22.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f500f7fe640 1 monclient: found mon.noname-b 2024-09-11T01:29:22.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f500f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:22.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f500f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:29:22.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.240+0000 7f500f7fe640 1 -- 172.21.15.12:0/3345543635 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5010138ae0 con 0x7f50101381c0 2024-09-11T01:29:22.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.243+0000 7f500f7fe640 1 -- 172.21.15.12:0/3345543635 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f50101381c0 msgr2=0x7f50101385a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:22.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.243+0000 7f500f7fe640 1 --2- 172.21.15.12:0/3345543635 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f50101381c0 0x7f50101385a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f50000098a0 tx=0x7f500002a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24e7c4a644a07323 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:22.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.243+0000 7f500e7fc640 1 -- 172.21.15.12:0/3345543635 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5000037070 con 0x7f50101381c0 2024-09-11T01:29:22.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.243+0000 7f500e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:22.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.243+0000 7f500e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:22.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.243+0000 7f500e7fc640 10 monclient: dump: 2024-09-11T01:29:22.249 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:22.249 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:22.249 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:22.249 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:22.249 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:22.249 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:22.249 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:22.249 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:22.250 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.243+0000 7f500e7fc640 10 monclient: _finish_auth 0 2024-09-11T01:29:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.243+0000 7f500e7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:29:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.243+0000 7f500e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.243+0000 7f500e7fc640 1 -- 172.21.15.12:0/3345543635 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5000032640 con 0x7f50101381c0 2024-09-11T01:29:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.243+0000 7f500e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.243+0000 7f500e7fc640 1 -- 172.21.15.12:0/3345543635 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5000032a10 con 0x7f50101381c0 2024-09-11T01:29:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.243+0000 7f501619b640 5 monclient: authenticate success, global_id 4334 2024-09-11T01:29:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.243+0000 7f501619b640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.243+0000 7f501619b640 4 set_mon_vals no callback set 2024-09-11T01:29:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.243+0000 7f500e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.243+0000 7f500e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:22.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.243+0000 7f500e7fc640 10 monclient: dump: 2024-09-11T01:29:22.251 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:22.251 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:22.251 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:22.251 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:22.251 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:22.251 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:22.251 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:22.251 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:22.251 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:22.251 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:22.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.243+0000 7f501619b640 10 monclient: shutdown 2024-09-11T01:29:22.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.243+0000 7f501619b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:22.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.243+0000 7f501619b640 1 -- 172.21.15.12:0/3345543635 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f50101381c0 msgr2=0x7f50101385a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:22.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.243+0000 7f501619b640 1 --2- 172.21.15.12:0/3345543635 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f50101381c0 0x7f50101385a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f50000098a0 tx=0x7f500002a100 comp rx=0 tx=0).stop 2024-09-11T01:29:22.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.244+0000 7f501619b640 1 -- 172.21.15.12:0/3345543635 shutdown_connections 2024-09-11T01:29:22.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.244+0000 7f501619b640 1 --2- 172.21.15.12:0/3345543635 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f50101381c0 0x7f50101385a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.244+0000 7f501619b640 1 -- 172.21.15.12:0/3345543635 >> 172.21.15.12:0/3345543635 conn(0x7f50100a7440 msgr2=0x7f50100a7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:22.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.244+0000 7f501619b640 1 -- 172.21.15.12:0/3345543635 shutdown_connections 2024-09-11T01:29:22.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.244+0000 7f501619b640 1 -- 172.21.15.12:0/3345543635 wait complete. 2024-09-11T01:29:22.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.244+0000 7f501619b640 10 monclient: build_initial_monmap 2024-09-11T01:29:22.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.244+0000 7f501619b640 10 monclient: monmap: 2024-09-11T01:29:22.252 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:22.252 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:22.252 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:22.252 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:22.252 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:22.252 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:22.253 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:22.253 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:22.253 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:22.253 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:22.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.244+0000 7f501619b640 1 Processor -- start 2024-09-11T01:29:22.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f501619b640 1 -- start start 2024-09-11T01:29:22.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f501619b640 10 monclient: init 2024-09-11T01:29:22.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f501619b640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:22.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f501619b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:29:22.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f501619b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f50100a6aa0 0x7f5010131fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:22.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f501619b640 10 monclient(hunting): picked mon.noname-b con 0x7f50100a6aa0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:22.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f501619b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f50101381c0 0x7f5010132510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:22.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f501619b640 10 monclient(hunting): picked mon.noname-c con 0x7f50101381c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:22.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f501619b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f501013d000 0x7f5010132ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:22.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f501619b640 10 monclient(hunting): picked mon.noname-a con 0x7f501013d000 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:22.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f501619b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:22.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f501619b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5010140d20 con 0x7f501013d000 2024-09-11T01:29:22.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f501619b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:22.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f501619b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f501009c5f0 con 0x7f50100a6aa0 2024-09-11T01:29:22.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f501619b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:22.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f501619b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5010140ea0 con 0x7f50101381c0 2024-09-11T01:29:22.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f501619b640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:22.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f501619b640 10 monclient(hunting): authenticate will time out at 768.687652s 2024-09-11T01:29:22.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f500effd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f50101381c0 0x7f5010132510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:22.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f500ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f501013d000 0x7f5010132ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:22.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f500effd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f50101381c0 0x7f5010132510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51822/0 (socket says 172.21.15.12:51822) 2024-09-11T01:29:22.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f500effd640 1 -- 172.21.15.12:0/4096621014 learned_addr learned my addr 172.21.15.12:0/4096621014 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:22.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f500effd640 10 monclient(hunting): get_auth_request con 0x7f50101381c0 auth_method 0 2024-09-11T01:29:22.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f500effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:22.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f500effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:22.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f500effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:22.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f500ffff640 10 monclient(hunting): get_auth_request con 0x7f501013d000 auth_method 0 2024-09-11T01:29:22.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f500ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:22.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f500ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:22.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f500ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:22.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f500effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:22.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f500effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:22.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f500effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:22.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f500ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:22.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f500ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:22.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f500ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:22.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f500f7fe640 1 --2- 172.21.15.12:0/4096621014 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f50100a6aa0 0x7f5010131fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:22.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f500f7fe640 10 monclient(hunting): get_auth_request con 0x7f50100a6aa0 auth_method 0 2024-09-11T01:29:22.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f500f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:22.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f500f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:22.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.245+0000 7f500f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:22.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.246+0000 7f500f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:22.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.246+0000 7f500f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:22.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.246+0000 7f500f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:22.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.246+0000 7f500effd640 10 monclient(hunting): handle_auth_done global_id 4337 payload 824 2024-09-11T01:29:22.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.246+0000 7f500effd640 1 -- 172.21.15.12:0/4096621014 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f50100a6aa0 msgr2=0x7f5010131fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:22.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.246+0000 7f500effd640 1 --2- 172.21.15.12:0/4096621014 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f50100a6aa0 0x7f5010131fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.246+0000 7f500effd640 1 -- 172.21.15.12:0/4096621014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f501013d000 msgr2=0x7f5010132ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:22.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.246+0000 7f500effd640 1 --2- 172.21.15.12:0/4096621014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f501013d000 0x7f5010132ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.246+0000 7f500effd640 10 monclient: _finish_hunting 0 2024-09-11T01:29:22.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.246+0000 7f500effd640 1 monclient: found mon.noname-c 2024-09-11T01:29:22.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.246+0000 7f500effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:22.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.246+0000 7f500effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:29:22.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.246+0000 7f500effd640 1 -- 172.21.15.12:0/4096621014 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5010135b30 con 0x7f50101381c0 2024-09-11T01:29:22.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.246+0000 7f500ffff640 1 --2- 172.21.15.12:0/4096621014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f501013d000 0x7f5010132ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:22.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.248+0000 7f500effd640 1 -- 172.21.15.12:0/4096621014 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f50101381c0 msgr2=0x7f5010132510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:22.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.248+0000 7f500effd640 1 --2- 172.21.15.12:0/4096621014 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f50101381c0 0x7f5010132510 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f500400c430 tx=0x7f500400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:22.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.248+0000 7f4ff7fff640 1 -- 172.21.15.12:0/4096621014 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50040119a0 con 0x7f50101381c0 2024-09-11T01:29:22.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.248+0000 7f4ff7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:22.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.248+0000 7f4ff7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:22.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.248+0000 7f4ff7fff640 10 monclient: dump: 2024-09-11T01:29:22.262 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:22.262 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:22.262 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:22.262 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:22.262 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:22.262 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:22.262 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:22.262 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:22.262 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:22.262 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:22.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.248+0000 7f4ff7fff640 10 monclient: _finish_auth 0 2024-09-11T01:29:22.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.248+0000 7f4ff7fff640 10 monclient: _check_auth_tickets 2024-09-11T01:29:22.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.248+0000 7f4ff7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:22.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.248+0000 7f4ff7fff640 1 -- 172.21.15.12:0/4096621014 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5004017070 con 0x7f50101381c0 2024-09-11T01:29:22.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.248+0000 7f501619b640 5 monclient: authenticate success, global_id 4337 2024-09-11T01:29:22.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.248+0000 7f501619b640 10 monclient: _renew_subs 2024-09-11T01:29:22.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.248+0000 7f501619b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:22.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.248+0000 7f501619b640 1 -- 172.21.15.12:0/4096621014 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5010140410 con 0x7f50101381c0 2024-09-11T01:29:22.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.248+0000 7f4ff7fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:22.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.248+0000 7f4ff7fff640 1 -- 172.21.15.12:0/4096621014 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5004011d80 con 0x7f50101381c0 2024-09-11T01:29:22.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.248+0000 7f4ff7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:22.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.248+0000 7f4ff7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:22.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.248+0000 7f4ff7fff640 10 monclient: dump: 2024-09-11T01:29:22.263 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:22.264 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:22.264 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:22.264 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:22.264 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:22.264 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:22.264 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:22.264 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:22.264 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:22.264 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:22.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.248+0000 7f500dffb640 4 set_mon_vals no callback set 2024-09-11T01:29:22.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.248+0000 7f501619b640 10 monclient: _renew_subs 2024-09-11T01:29:22.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.248+0000 7f501619b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:22.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.248+0000 7f501619b640 1 -- 172.21.15.12:0/4096621014 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f50100a09e0 con 0x7f50101381c0 2024-09-11T01:29:22.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.249+0000 7f501619b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:22.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.249+0000 7f501619b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:22.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.249+0000 7f501619b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:22.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.249+0000 7f501619b640 1 -- 172.21.15.12:0/4096621014 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50101385a0 con 0x7f50101381c0 2024-09-11T01:29:22.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.249+0000 7f4ff7fff640 1 -- 172.21.15.12:0/4096621014 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f500401c460 con 0x7f50101381c0 2024-09-11T01:29:22.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.250+0000 7f4ff7fff640 1 --2- 172.21.15.12:0/4096621014 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4ffc03e260 0x7f4ffc040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:22.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.250+0000 7f4ff7fff640 1 -- 172.21.15.12:0/4096621014 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4402+0+0 (secure 0 0 0) 0x7f5004052970 con 0x7f50101381c0 2024-09-11T01:29:22.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.250+0000 7f500f7fe640 1 --2- 172.21.15.12:0/4096621014 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4ffc03e260 0x7f4ffc040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:22.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.250+0000 7f500f7fe640 10 monclient: get_auth_request con 0x7f4ffc03e260 auth_method 0 2024-09-11T01:29:22.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.252+0000 7f500f7fe640 1 --2- 172.21.15.12:0/4096621014 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4ffc03e260 0x7f4ffc040720 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f501009bb60 tx=0x7f500002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:22.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.254+0000 7f4ff7fff640 1 -- 172.21.15.12:0/4096621014 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5004012420 con 0x7f50101381c0 2024-09-11T01:29:22.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.254+0000 7f4ff7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:22.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.254+0000 7f4ff7fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:22.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.442+0000 7f501619b640 1 -- 172.21.15.12:0/4096621014 --> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5010069840 con 0x7f4ffc03e260 2024-09-11T01:29:22.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.444+0000 7f4ff7fff640 1 -- 172.21.15.12:0/4096621014 <== mgr.4101 v2:172.21.15.71:6816/3189259519 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34179 (secure 0 0 0) 0x7f5010069840 con 0x7f4ffc03e260 2024-09-11T01:29:22.446 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:29:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-09-11T01:29:22.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.446+0000 7f501619b640 1 -- 172.21.15.12:0/4096621014 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4ffc03e260 msgr2=0x7f4ffc040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:22.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.446+0000 7f501619b640 1 --2- 172.21.15.12:0/4096621014 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4ffc03e260 0x7f4ffc040720 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f501009bb60 tx=0x7f500002e000 comp rx=0 tx=0).stop 2024-09-11T01:29:22.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.446+0000 7f501619b640 10 monclient: shutdown 2024-09-11T01:29:22.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.446+0000 7f501619b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:22.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.446+0000 7f501619b640 1 -- 172.21.15.12:0/4096621014 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f50101381c0 msgr2=0x7f5010132510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:22.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.446+0000 7f501619b640 1 --2- 172.21.15.12:0/4096621014 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f50101381c0 0x7f5010132510 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f500400c430 tx=0x7f500400c8f0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.446+0000 7f501619b640 1 -- 172.21.15.12:0/4096621014 shutdown_connections 2024-09-11T01:29:22.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.446+0000 7f501619b640 1 --2- 172.21.15.12:0/4096621014 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4ffc03e260 0x7f4ffc040720 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.446+0000 7f501619b640 1 --2- 172.21.15.12:0/4096621014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f501013d000 0x7f5010132ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.446+0000 7f501619b640 1 --2- 172.21.15.12:0/4096621014 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f50101381c0 0x7f5010132510 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.446+0000 7f501619b640 1 --2- 172.21.15.12:0/4096621014 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f50100a6aa0 0x7f5010131fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.446+0000 7f501619b640 1 -- 172.21.15.12:0/4096621014 >> 172.21.15.12:0/4096621014 conn(0x7f50100a7440 msgr2=0x7f501013f970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:22.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.446+0000 7f501619b640 1 -- 172.21.15.12:0/4096621014 shutdown_connections 2024-09-11T01:29:22.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.446+0000 7f501619b640 1 -- 172.21.15.12:0/4096621014 wait complete. 2024-09-11T01:29:22.458 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-09-11T01:29:22.317516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163236,"kb_used_data":1016,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067804,"statfs":{"total":579820584960,"available":579653431296,"internally_reserved":0,"allocated":1040384,"data_stored":1336972,"data_compressed":59136,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.465675"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.597492+0000","last_change":"2024-09-11T01:29:17.597492+0000","last_active":"2024-09-11T01:29:17.597492+0000","last_peered":"2024-09-11T01:29:17.597492+0000","last_clean":"2024-09-11T01:29:17.597492+0000","last_became_active":"2024-09-11T01:29:15.546444+0000","last_became_peered":"2024-09-11T01:29:15.546444+0000","last_unstale":"2024-09-11T01:29:17.597492+0000","last_undegraded":"2024-09-11T01:29:17.597492+0000","last_fullsized":"2024-09-11T01:29:17.597492+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:52:43.624650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000396053,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.579590+0000","last_change":"2024-09-11T01:29:17.579590+0000","last_active":"2024-09-11T01:29:17.579590+0000","last_peered":"2024-09-11T01:29:17.579590+0000","last_clean":"2024-09-11T01:29:17.579590+0000","last_became_active":"2024-09-11T01:29:15.542800+0000","last_became_peered":"2024-09-11T01:29:15.542800+0000","last_unstale":"2024-09-11T01:29:17.579590+0000","last_undegraded":"2024-09-11T01:29:17.579590+0000","last_fullsized":"2024-09-11T01:29:17.579590+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:23:55.773396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040701099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.576853+0000","last_change":"2024-09-11T01:29:17.576853+0000","last_active":"2024-09-11T01:29:17.576853+0000","last_peered":"2024-09-11T01:29:17.576853+0000","last_clean":"2024-09-11T01:29:17.576853+0000","last_became_active":"2024-09-11T01:29:15.546476+0000","last_became_peered":"2024-09-11T01:29:15.546476+0000","last_unstale":"2024-09-11T01:29:17.576853+0000","last_undegraded":"2024-09-11T01:29:17.576853+0000","last_fullsized":"2024-09-11T01:29:17.576853+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:11:54.706642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037031899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.583012+0000","last_change":"2024-09-11T01:29:17.583012+0000","last_active":"2024-09-11T01:29:17.583012+0000","last_peered":"2024-09-11T01:29:17.583012+0000","last_clean":"2024-09-11T01:29:17.583012+0000","last_became_active":"2024-09-11T01:29:15.547447+0000","last_became_peered":"2024-09-11T01:29:15.547447+0000","last_unstale":"2024-09-11T01:29:17.583012+0000","last_undegraded":"2024-09-11T01:29:17.583012+0000","last_fullsized":"2024-09-11T01:29:17.583012+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:17:48.616008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037448800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"16'72","reported_seq":103,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.599361+0000","last_change":"2024-09-11T01:29:16.691671+0000","last_active":"2024-09-11T01:29:17.599361+0000","last_peered":"2024-09-11T01:29:17.599361+0000","last_clean":"2024-09-11T01:29:17.599361+0000","last_became_active":"2024-09-11T01:29:16.690647+0000","last_became_peered":"2024-09-11T01:29:16.690647+0000","last_unstale":"2024-09-11T01:29:17.599361+0000","last_undegraded":"2024-09-11T01:29:17.599361+0000","last_fullsized":"2024-09-11T01:29:17.599361+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:15.533298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:15.533298+0000","last_clean_scrub_stamp":"2024-09-11T01:29:15.533298+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:41:07.416896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.583117+0000","last_change":"2024-09-11T01:29:17.583117+0000","last_active":"2024-09-11T01:29:17.583117+0000","last_peered":"2024-09-11T01:29:17.583117+0000","last_clean":"2024-09-11T01:29:17.583117+0000","last_became_active":"2024-09-11T01:29:15.546142+0000","last_became_peered":"2024-09-11T01:29:15.546142+0000","last_unstale":"2024-09-11T01:29:17.583117+0000","last_undegraded":"2024-09-11T01:29:17.583117+0000","last_fullsized":"2024-09-11T01:29:17.583117+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:16:49.024029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044411199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"16'2","reported_seq":22,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.578002+0000","last_change":"2024-09-11T01:29:17.577463+0000","last_active":"2024-09-11T01:29:17.578002+0000","last_peered":"2024-09-11T01:29:17.578002+0000","last_clean":"2024-09-11T01:29:17.578002+0000","last_became_active":"2024-09-11T01:29:15.548884+0000","last_became_peered":"2024-09-11T01:29:15.548884+0000","last_unstale":"2024-09-11T01:29:17.578002+0000","last_undegraded":"2024-09-11T01:29:17.578002+0000","last_fullsized":"2024-09-11T01:29:17.578002+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:09.120675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094832499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.597459+0000","last_change":"2024-09-11T01:29:17.597459+0000","last_active":"2024-09-11T01:29:17.597459+0000","last_peered":"2024-09-11T01:29:17.597459+0000","last_clean":"2024-09-11T01:29:17.597459+0000","last_became_active":"2024-09-11T01:29:15.549013+0000","last_became_peered":"2024-09-11T01:29:15.549013+0000","last_unstale":"2024-09-11T01:29:17.597459+0000","last_undegraded":"2024-09-11T01:29:17.597459+0000","last_fullsized":"2024-09-11T01:29:17.597459+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:46:02.130174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041667199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.579548+0000","last_change":"2024-09-11T01:29:17.579548+0000","last_active":"2024-09-11T01:29:17.579548+0000","last_peered":"2024-09-11T01:29:17.579548+0000","last_clean":"2024-09-11T01:29:17.579548+0000","last_became_active":"2024-09-11T01:29:15.546952+0000","last_became_peered":"2024-09-11T01:29:15.546952+0000","last_unstale":"2024-09-11T01:29:17.579548+0000","last_undegraded":"2024-09-11T01:29:17.579548+0000","last_fullsized":"2024-09-11T01:29:17.579548+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:45:28.460820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038245800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59136,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607556,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":151552,"data_stored":46513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607556,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":155648,"data_stored":46532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607556,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":212992,"data_stored":575441,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607555,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":212992,"data_stored":575441,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607556,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":155648,"data_stored":46532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":151552,"data_stored":46513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T01:29:22.459 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T01:29:22.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.558+0000 7fea4a252640 10 monclient: get_monmap_and_config 2024-09-11T01:29:22.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.558+0000 7fea4a252640 10 monclient: build_initial_monmap 2024-09-11T01:29:22.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.558+0000 7fea4a252640 10 monclient: monmap: 2024-09-11T01:29:22.560 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:22.561 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:22.561 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:22.559664+0000 2024-09-11T01:29:22.561 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:22.559664+0000 2024-09-11T01:29:22.561 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:22.561 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:22.561 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:22.561 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:22.561 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:22.561 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:22.561 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:22.561 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:22.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.559+0000 7fea4a252640 1 Processor -- start 2024-09-11T01:29:22.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.559+0000 7fea4a252640 1 -- start start 2024-09-11T01:29:22.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.559+0000 7fea4a252640 10 monclient: init 2024-09-11T01:29:22.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea4a252640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:22.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea4a252640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-11T01:29:22.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea4a252640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fea441381c0 0x7fea441385a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:22.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea4a252640 10 monclient(hunting): picked mon.noname-a con 0x7fea441381c0 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:22.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea4a252640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fea440a6aa0 0x7fea4409cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:22.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea4a252640 10 monclient(hunting): picked mon.noname-b con 0x7fea440a6aa0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:22.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea4a252640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fea440a4f20 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:22.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea4a252640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea4409c310 con 0x7fea440a4f20 2024-09-11T01:29:22.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea4a252640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:22.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea4a252640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fea4406fc90 con 0x7fea441381c0 2024-09-11T01:29:22.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea4a252640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:22.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea4a252640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fea4409c5f0 con 0x7fea440a6aa0 2024-09-11T01:29:22.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea4a252640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:22.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea4a252640 10 monclient(hunting): authenticate will time out at 769.002357s 2024-09-11T01:29:22.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea437fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fea441381c0 0x7fea441385a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:22.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea42ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fea440a6aa0 0x7fea4409cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:22.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea437fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fea441381c0 0x7fea441385a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55096/0 (socket says 172.21.15.12:55096) 2024-09-11T01:29:22.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea43fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fea440a4f20 0x7fea440a5390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56116/0 (socket says 172.21.15.12:56116) 2024-09-11T01:29:22.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea42ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fea440a6aa0 0x7fea4409cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51832/0 (socket says 172.21.15.12:51832) 2024-09-11T01:29:22.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea437fe640 1 -- 172.21.15.12:0/3794060478 learned_addr learned my addr 172.21.15.12:0/3794060478 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:22.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea437fe640 10 monclient(hunting): get_auth_request con 0x7fea441381c0 auth_method 0 2024-09-11T01:29:22.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:22.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea437fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:22.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:22.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea42ffd640 10 monclient(hunting): get_auth_request con 0x7fea440a6aa0 auth_method 0 2024-09-11T01:29:22.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea42ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:22.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea42ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:22.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:22.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea42ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:22.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea42ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:22.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea42ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:22.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:22.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:22.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:22.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea42ffd640 10 monclient(hunting): handle_auth_done global_id 4340 payload 274 2024-09-11T01:29:22.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea42ffd640 1 -- 172.21.15.12:0/3794060478 >> v2:172.21.15.12:3300/0 conn(0x7fea441381c0 msgr2=0x7fea441385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:22.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea42ffd640 1 --2- 172.21.15.12:0/3794060478 >> v2:172.21.15.12:3300/0 conn(0x7fea441381c0 0x7fea441385a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea42ffd640 1 -- 172.21.15.12:0/3794060478 >> v1:172.21.15.12:6789/0 conn(0x7fea440a4f20 legacy=0x7fea440a5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:22.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea42ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:29:22.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea42ffd640 1 monclient: found mon.noname-b 2024-09-11T01:29:22.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:22.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea42ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:29:22.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea42ffd640 1 -- 172.21.15.12:0/3794060478 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea4409c8d0 con 0x7fea440a6aa0 2024-09-11T01:29:22.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.560+0000 7fea437fe640 1 --2- 172.21.15.12:0/3794060478 >> v2:172.21.15.12:3300/0 conn(0x7fea441381c0 0x7fea441385a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:22.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.563+0000 7fea42ffd640 1 -- 172.21.15.12:0/3794060478 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fea440a6aa0 msgr2=0x7fea4409cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:22.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.563+0000 7fea42ffd640 1 --2- 172.21.15.12:0/3794060478 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fea440a6aa0 0x7fea4409cfd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fea380097a0 tx=0x7fea3802a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d73cd4c4d879bd37 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:22.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea427fc640 1 -- 172.21.15.12:0/3794060478 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea38037070 con 0x7fea440a6aa0 2024-09-11T01:29:22.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:22.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:22.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea427fc640 10 monclient: dump: 2024-09-11T01:29:22.572 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:22.572 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:22.573 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:22.573 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:22.573 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:22.573 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:22.573 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:22.573 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:22.573 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:22.573 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:22.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea427fc640 10 monclient: _finish_auth 0 2024-09-11T01:29:22.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea427fc640 10 monclient: _check_auth_tickets 2024-09-11T01:29:22.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:22.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea4a252640 5 monclient: authenticate success, global_id 4340 2024-09-11T01:29:22.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea427fc640 1 -- 172.21.15.12:0/3794060478 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea38032440 con 0x7fea440a6aa0 2024-09-11T01:29:22.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea4a252640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:22.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea427fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:22.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea4a252640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:22.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea4a252640 4 set_mon_vals no callback set 2024-09-11T01:29:22.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea427fc640 1 -- 172.21.15.12:0/3794060478 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea38032810 con 0x7fea440a6aa0 2024-09-11T01:29:22.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:22.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:22.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea427fc640 10 monclient: dump: 2024-09-11T01:29:22.574 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:22.574 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:22.574 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:22.574 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:22.574 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:22.575 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:22.575 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:22.575 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:22.575 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:22.575 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:22.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea4a252640 10 monclient: shutdown 2024-09-11T01:29:22.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea4a252640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:22.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea4a252640 1 -- 172.21.15.12:0/3794060478 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fea440a6aa0 msgr2=0x7fea4409cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:22.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea4a252640 1 --2- 172.21.15.12:0/3794060478 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fea440a6aa0 0x7fea4409cfd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fea380097a0 tx=0x7fea3802a040 comp rx=0 tx=0).stop 2024-09-11T01:29:22.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea4a252640 1 -- 172.21.15.12:0/3794060478 shutdown_connections 2024-09-11T01:29:22.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea4a252640 1 --2- 172.21.15.12:0/3794060478 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fea440a6aa0 0x7fea4409cfd0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea4a252640 1 --2- 172.21.15.12:0/3794060478 >> v2:172.21.15.12:3300/0 conn(0x7fea441381c0 0x7fea441385a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea4a252640 1 -- 172.21.15.12:0/3794060478 >> 172.21.15.12:0/3794060478 conn(0x7fea440a7440 msgr2=0x7fea440a7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:22.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea4a252640 1 -- 172.21.15.12:0/3794060478 shutdown_connections 2024-09-11T01:29:22.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.564+0000 7fea4a252640 1 -- 172.21.15.12:0/3794060478 wait complete. 2024-09-11T01:29:22.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.565+0000 7fea4a252640 10 monclient: build_initial_monmap 2024-09-11T01:29:22.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.565+0000 7fea4a252640 10 monclient: monmap: 2024-09-11T01:29:22.576 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:22.576 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:22.576 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:22.576 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:22.576 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:22.576 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:22.576 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:22.576 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:22.576 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:22.577 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:22.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.565+0000 7fea4a252640 1 Processor -- start 2024-09-11T01:29:22.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.565+0000 7fea4a252640 1 -- start start 2024-09-11T01:29:22.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.565+0000 7fea4a252640 10 monclient: init 2024-09-11T01:29:22.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea4a252640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:22.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea4a252640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:29:22.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea4a252640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea440a4f20 0x7fea44135f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:22.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea4a252640 10 monclient(hunting): picked mon.noname-a con 0x7fea440a4f20 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:22.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea4a252640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fea440a6aa0 0x7fea441364a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:22.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea4a252640 10 monclient(hunting): picked mon.noname-c con 0x7fea440a6aa0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:22.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea4a252640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fea441381c0 0x7fea44123af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:22.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea4a252640 10 monclient(hunting): picked mon.noname-b con 0x7fea441381c0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:22.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea4a252640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:22.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea4a252640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fea4409d690 con 0x7fea440a4f20 2024-09-11T01:29:22.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea4a252640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:22.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea4a252640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fea4409d510 con 0x7fea441381c0 2024-09-11T01:29:22.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea4a252640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:22.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea4a252640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fea440a5aa0 con 0x7fea440a6aa0 2024-09-11T01:29:22.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea4a252640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:22.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea4a252640 10 monclient(hunting): authenticate will time out at 769.008368s 2024-09-11T01:29:22.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea42ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fea440a6aa0 0x7fea441364a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:22.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea437fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea440a4f20 0x7fea44135f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:22.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea42ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fea440a6aa0 0x7fea441364a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51848/0 (socket says 172.21.15.12:51848) 2024-09-11T01:29:22.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea42ffd640 1 -- 172.21.15.12:0/1325844645 learned_addr learned my addr 172.21.15.12:0/1325844645 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:22.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea42ffd640 10 monclient(hunting): get_auth_request con 0x7fea440a6aa0 auth_method 0 2024-09-11T01:29:22.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea42ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:22.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea42ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:22.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:22.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea437fe640 10 monclient(hunting): get_auth_request con 0x7fea440a4f20 auth_method 0 2024-09-11T01:29:22.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:22.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea437fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:22.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:22.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea42ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:22.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea42ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:22.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea42ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:22.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:22.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:22.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:22.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea42ffd640 10 monclient(hunting): handle_auth_done global_id 4343 payload 824 2024-09-11T01:29:22.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea42ffd640 1 -- 172.21.15.12:0/1325844645 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fea441381c0 msgr2=0x7fea44123af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:22.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea42ffd640 1 --2- 172.21.15.12:0/1325844645 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fea441381c0 0x7fea44123af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea42ffd640 1 -- 172.21.15.12:0/1325844645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea440a4f20 msgr2=0x7fea44135f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:22.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea42ffd640 1 --2- 172.21.15.12:0/1325844645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea440a4f20 0x7fea44135f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea42ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:29:22.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea42ffd640 1 monclient: found mon.noname-c 2024-09-11T01:29:22.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:22.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea42ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:29:22.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.566+0000 7fea42ffd640 1 -- 172.21.15.12:0/1325844645 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea440a8420 con 0x7fea440a6aa0 2024-09-11T01:29:22.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.568+0000 7fea42ffd640 1 -- 172.21.15.12:0/1325844645 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fea440a6aa0 msgr2=0x7fea441364a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:22.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.568+0000 7fea42ffd640 1 --2- 172.21.15.12:0/1325844645 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fea440a6aa0 0x7fea441364a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fea380098c0 tx=0x7fea380046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:22.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.569+0000 7fea2bfff640 1 -- 172.21.15.12:0/1325844645 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea38037040 con 0x7fea440a6aa0 2024-09-11T01:29:22.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.569+0000 7fea2bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:22.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.569+0000 7fea2bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:22.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.569+0000 7fea2bfff640 10 monclient: dump: 2024-09-11T01:29:22.582 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:22.582 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:22.582 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:22.582 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:22.582 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:22.582 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:22.582 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:22.582 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:22.582 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:22.582 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:22.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.569+0000 7fea2bfff640 10 monclient: _finish_auth 0 2024-09-11T01:29:22.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.569+0000 7fea2bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:29:22.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.569+0000 7fea2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:22.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.569+0000 7fea2bfff640 1 -- 172.21.15.12:0/1325844645 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea38042070 con 0x7fea440a6aa0 2024-09-11T01:29:22.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.569+0000 7fea2bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:22.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.569+0000 7fea2bfff640 1 -- 172.21.15.12:0/1325844645 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3803bb00 con 0x7fea440a6aa0 2024-09-11T01:29:22.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.569+0000 7fea4a252640 5 monclient: authenticate success, global_id 4343 2024-09-11T01:29:22.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.569+0000 7fea2bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:22.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.569+0000 7fea2bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:22.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.569+0000 7fea2bfff640 10 monclient: dump: 2024-09-11T01:29:22.583 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:22.583 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:22.583 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:22.583 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:22.584 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.569+0000 7fea4a252640 10 monclient: _renew_subs 2024-09-11T01:29:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.569+0000 7fea4a252640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.569+0000 7fea4a252640 1 -- 172.21.15.12:0/1325844645 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea441369e0 con 0x7fea440a6aa0 2024-09-11T01:29:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.569+0000 7fea41ffb640 4 set_mon_vals no callback set 2024-09-11T01:29:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.569+0000 7fea4a252640 10 monclient: _renew_subs 2024-09-11T01:29:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.569+0000 7fea4a252640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.569+0000 7fea4a252640 1 -- 172.21.15.12:0/1325844645 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fea44127680 con 0x7fea440a6aa0 2024-09-11T01:29:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.570+0000 7fea2bfff640 1 -- 172.21.15.12:0/1325844645 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fea3803a030 con 0x7fea440a6aa0 2024-09-11T01:29:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.570+0000 7fea4a252640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.570+0000 7fea4a252640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.570+0000 7fea4a252640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.570+0000 7fea4a252640 1 -- 172.21.15.12:0/1325844645 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea4409cfd0 con 0x7fea440a6aa0 2024-09-11T01:29:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.570+0000 7fea2bfff640 1 --2- 172.21.15.12:0/1325844645 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fea3003e290 0x7fea30040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.570+0000 7fea2bfff640 1 -- 172.21.15.12:0/1325844645 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 4374+0+0 (secure 0 0 0) 0x7fea38072920 con 0x7fea440a6aa0 2024-09-11T01:29:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.570+0000 7fea437fe640 1 --2- 172.21.15.12:0/1325844645 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fea3003e290 0x7fea30040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.570+0000 7fea437fe640 10 monclient: get_auth_request con 0x7fea3003e290 auth_method 0 2024-09-11T01:29:22.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.573+0000 7fea437fe640 1 --2- 172.21.15.12:0/1325844645 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fea3003e290 0x7fea30040750 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fea34006ca0 tx=0x7fea34009240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:22.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.574+0000 7fea2bfff640 1 -- 172.21.15.12:0/1325844645 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fea3804b020 con 0x7fea440a6aa0 2024-09-11T01:29:22.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.574+0000 7fea2bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:22.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.574+0000 7fea2bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:22.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.755+0000 7fea4a252640 1 -- 172.21.15.12:0/1325844645 --> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fea440a1710 con 0x7fea3003e290 2024-09-11T01:29:22.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.756+0000 7fea2bfff640 1 -- 172.21.15.12:0/1325844645 <== mgr.4101 v2:172.21.15.71:6816/3189259519 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34179 (secure 0 0 0) 0x7fea440a1710 con 0x7fea3003e290 2024-09-11T01:29:22.758 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:29:22.759 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-09-11T01:29:22.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.758+0000 7fea4a252640 1 -- 172.21.15.12:0/1325844645 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fea3003e290 msgr2=0x7fea30040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:22.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.758+0000 7fea4a252640 1 --2- 172.21.15.12:0/1325844645 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fea3003e290 0x7fea30040750 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fea34006ca0 tx=0x7fea34009240 comp rx=0 tx=0).stop 2024-09-11T01:29:22.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.758+0000 7fea4a252640 10 monclient: shutdown 2024-09-11T01:29:22.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.758+0000 7fea4a252640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:22.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.758+0000 7fea4a252640 1 -- 172.21.15.12:0/1325844645 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fea440a6aa0 msgr2=0x7fea441364a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:22.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.758+0000 7fea4a252640 1 --2- 172.21.15.12:0/1325844645 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fea440a6aa0 0x7fea441364a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fea380098c0 tx=0x7fea380046c0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.758+0000 7fea4a252640 1 -- 172.21.15.12:0/1325844645 shutdown_connections 2024-09-11T01:29:22.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.758+0000 7fea4a252640 1 --2- 172.21.15.12:0/1325844645 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fea441381c0 0x7fea44123af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.758+0000 7fea4a252640 1 --2- 172.21.15.12:0/1325844645 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fea440a6aa0 0x7fea441364a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.758+0000 7fea4a252640 1 --2- 172.21.15.12:0/1325844645 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fea3003e290 0x7fea30040750 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.758+0000 7fea4a252640 1 --2- 172.21.15.12:0/1325844645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fea440a4f20 0x7fea44135f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.758+0000 7fea4a252640 1 -- 172.21.15.12:0/1325844645 >> 172.21.15.12:0/1325844645 conn(0x7fea440a7440 msgr2=0x7fea4409ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:22.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.758+0000 7fea4a252640 1 -- 172.21.15.12:0/1325844645 shutdown_connections 2024-09-11T01:29:22.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.758+0000 7fea4a252640 1 -- 172.21.15.12:0/1325844645 wait complete. 2024-09-11T01:29:22.770 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-09-11T01:29:22.317516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163236,"kb_used_data":1016,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067804,"statfs":{"total":579820584960,"available":579653431296,"internally_reserved":0,"allocated":1040384,"data_stored":1336972,"data_compressed":59136,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.465675"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.597492+0000","last_change":"2024-09-11T01:29:17.597492+0000","last_active":"2024-09-11T01:29:17.597492+0000","last_peered":"2024-09-11T01:29:17.597492+0000","last_clean":"2024-09-11T01:29:17.597492+0000","last_became_active":"2024-09-11T01:29:15.546444+0000","last_became_peered":"2024-09-11T01:29:15.546444+0000","last_unstale":"2024-09-11T01:29:17.597492+0000","last_undegraded":"2024-09-11T01:29:17.597492+0000","last_fullsized":"2024-09-11T01:29:17.597492+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:52:43.624650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000396053,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.579590+0000","last_change":"2024-09-11T01:29:17.579590+0000","last_active":"2024-09-11T01:29:17.579590+0000","last_peered":"2024-09-11T01:29:17.579590+0000","last_clean":"2024-09-11T01:29:17.579590+0000","last_became_active":"2024-09-11T01:29:15.542800+0000","last_became_peered":"2024-09-11T01:29:15.542800+0000","last_unstale":"2024-09-11T01:29:17.579590+0000","last_undegraded":"2024-09-11T01:29:17.579590+0000","last_fullsized":"2024-09-11T01:29:17.579590+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:23:55.773396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040701099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.576853+0000","last_change":"2024-09-11T01:29:17.576853+0000","last_active":"2024-09-11T01:29:17.576853+0000","last_peered":"2024-09-11T01:29:17.576853+0000","last_clean":"2024-09-11T01:29:17.576853+0000","last_became_active":"2024-09-11T01:29:15.546476+0000","last_became_peered":"2024-09-11T01:29:15.546476+0000","last_unstale":"2024-09-11T01:29:17.576853+0000","last_undegraded":"2024-09-11T01:29:17.576853+0000","last_fullsized":"2024-09-11T01:29:17.576853+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:11:54.706642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037031899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.583012+0000","last_change":"2024-09-11T01:29:17.583012+0000","last_active":"2024-09-11T01:29:17.583012+0000","last_peered":"2024-09-11T01:29:17.583012+0000","last_clean":"2024-09-11T01:29:17.583012+0000","last_became_active":"2024-09-11T01:29:15.547447+0000","last_became_peered":"2024-09-11T01:29:15.547447+0000","last_unstale":"2024-09-11T01:29:17.583012+0000","last_undegraded":"2024-09-11T01:29:17.583012+0000","last_fullsized":"2024-09-11T01:29:17.583012+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:17:48.616008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037448800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"16'72","reported_seq":103,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.599361+0000","last_change":"2024-09-11T01:29:16.691671+0000","last_active":"2024-09-11T01:29:17.599361+0000","last_peered":"2024-09-11T01:29:17.599361+0000","last_clean":"2024-09-11T01:29:17.599361+0000","last_became_active":"2024-09-11T01:29:16.690647+0000","last_became_peered":"2024-09-11T01:29:16.690647+0000","last_unstale":"2024-09-11T01:29:17.599361+0000","last_undegraded":"2024-09-11T01:29:17.599361+0000","last_fullsized":"2024-09-11T01:29:17.599361+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:15.533298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:15.533298+0000","last_clean_scrub_stamp":"2024-09-11T01:29:15.533298+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:41:07.416896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.583117+0000","last_change":"2024-09-11T01:29:17.583117+0000","last_active":"2024-09-11T01:29:17.583117+0000","last_peered":"2024-09-11T01:29:17.583117+0000","last_clean":"2024-09-11T01:29:17.583117+0000","last_became_active":"2024-09-11T01:29:15.546142+0000","last_became_peered":"2024-09-11T01:29:15.546142+0000","last_unstale":"2024-09-11T01:29:17.583117+0000","last_undegraded":"2024-09-11T01:29:17.583117+0000","last_fullsized":"2024-09-11T01:29:17.583117+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:16:49.024029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044411199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"16'2","reported_seq":22,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.578002+0000","last_change":"2024-09-11T01:29:17.577463+0000","last_active":"2024-09-11T01:29:17.578002+0000","last_peered":"2024-09-11T01:29:17.578002+0000","last_clean":"2024-09-11T01:29:17.578002+0000","last_became_active":"2024-09-11T01:29:15.548884+0000","last_became_peered":"2024-09-11T01:29:15.548884+0000","last_unstale":"2024-09-11T01:29:17.578002+0000","last_undegraded":"2024-09-11T01:29:17.578002+0000","last_fullsized":"2024-09-11T01:29:17.578002+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:09.120675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094832499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.597459+0000","last_change":"2024-09-11T01:29:17.597459+0000","last_active":"2024-09-11T01:29:17.597459+0000","last_peered":"2024-09-11T01:29:17.597459+0000","last_clean":"2024-09-11T01:29:17.597459+0000","last_became_active":"2024-09-11T01:29:15.549013+0000","last_became_peered":"2024-09-11T01:29:15.549013+0000","last_unstale":"2024-09-11T01:29:17.597459+0000","last_undegraded":"2024-09-11T01:29:17.597459+0000","last_fullsized":"2024-09-11T01:29:17.597459+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:46:02.130174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041667199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.579548+0000","last_change":"2024-09-11T01:29:17.579548+0000","last_active":"2024-09-11T01:29:17.579548+0000","last_peered":"2024-09-11T01:29:17.579548+0000","last_clean":"2024-09-11T01:29:17.579548+0000","last_became_active":"2024-09-11T01:29:15.546952+0000","last_became_peered":"2024-09-11T01:29:15.546952+0000","last_unstale":"2024-09-11T01:29:17.579548+0000","last_undegraded":"2024-09-11T01:29:17.579548+0000","last_fullsized":"2024-09-11T01:29:17.579548+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:45:28.460820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038245800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59136,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607556,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":151552,"data_stored":46513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607556,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":155648,"data_stored":46532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607556,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":212992,"data_stored":575441,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607555,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":212992,"data_stored":575441,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607556,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":155648,"data_stored":46532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":151552,"data_stored":46513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T01:29:22.771 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T01:29:22.771 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-11T01:29:22.771 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-11T01:29:22.771 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-11T01:29:22.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.869+0000 7f84f2ebe640 10 monclient: get_monmap_and_config 2024-09-11T01:29:22.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.869+0000 7f84f2ebe640 10 monclient: build_initial_monmap 2024-09-11T01:29:22.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.869+0000 7f84f2ebe640 10 monclient: monmap: 2024-09-11T01:29:22.871 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:22.871 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:22.871 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:22.870534+0000 2024-09-11T01:29:22.872 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:22.870534+0000 2024-09-11T01:29:22.872 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:22.872 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:22.872 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:22.872 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:22.872 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:22.872 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:22.872 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:22.872 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:22.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.870+0000 7f84f2ebe640 1 Processor -- start 2024-09-11T01:29:22.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.870+0000 7f84f2ebe640 1 -- start start 2024-09-11T01:29:22.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.870+0000 7f84f2ebe640 10 monclient: init 2024-09-11T01:29:22.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.870+0000 7f84f2ebe640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:22.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.870+0000 7f84f2ebe640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-11T01:29:22.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f2ebe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84ec138230 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:22.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f2ebe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f84ec09d030 0x7f84ec13d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:22.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f2ebe640 10 monclient(hunting): picked mon.noname-a con 0x7f84ec09d030 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:22.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f2ebe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f84ec09d460 0x7f84ec13f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:22.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f2ebe640 10 monclient(hunting): picked mon.noname-b con 0x7f84ec09d460 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:22.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f2ebe640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84ec0a6e30 con 0x7f84ec138230 2024-09-11T01:29:22.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f2ebe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:22.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f2ebe640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f84ec09c9c0 con 0x7f84ec09d030 2024-09-11T01:29:22.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f2ebe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:22.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f2ebe640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f84ec09ba20 con 0x7f84ec09d460 2024-09-11T01:29:22.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f2ebe640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:22.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f2ebe640 10 monclient(hunting): authenticate will time out at 769.313094s 2024-09-11T01:29:22.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f0c5a640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f84ec138230 0x7f84ec138610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56120/0 (socket says 172.21.15.12:56120) 2024-09-11T01:29:22.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f145b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f84ec09d460 0x7f84ec13f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:22.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f0c5a640 1 -- 172.21.15.12:0/2321108246 learned_addr learned my addr 172.21.15.12:0/2321108246 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:29:22.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f145b640 10 monclient(hunting): get_auth_request con 0x7f84ec09d460 auth_method 0 2024-09-11T01:29:22.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f145b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:22.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f145b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:22.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f145b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:22.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84e3fff640 1 --2- 172.21.15.12:0/2321108246 >> v2:172.21.15.12:3300/0 conn(0x7f84ec09d030 0x7f84ec13d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:22.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84e3fff640 10 monclient(hunting): get_auth_request con 0x7f84ec09d030 auth_method 0 2024-09-11T01:29:22.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:22.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:22.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:22.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f145b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:22.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f145b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:22.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f145b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:22.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:22.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:22.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:22.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f145b640 10 monclient(hunting): handle_auth_done global_id 4346 payload 274 2024-09-11T01:29:22.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f145b640 1 -- 172.21.15.12:0/2321108246 >> v2:172.21.15.12:3300/0 conn(0x7f84ec09d030 msgr2=0x7f84ec13d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:22.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f145b640 1 --2- 172.21.15.12:0/2321108246 >> v2:172.21.15.12:3300/0 conn(0x7f84ec09d030 0x7f84ec13d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f145b640 1 -- 172.21.15.12:0/2321108246 >> v1:172.21.15.12:6789/0 conn(0x7f84ec138230 legacy=0x7f84ec138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:22.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f145b640 10 monclient: _finish_hunting 0 2024-09-11T01:29:22.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f145b640 1 monclient: found mon.noname-b 2024-09-11T01:29:22.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f145b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:22.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f145b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:29:22.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.871+0000 7f84f145b640 1 -- 172.21.15.12:0/2321108246 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84ec0a66a0 con 0x7f84ec09d460 2024-09-11T01:29:22.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.874+0000 7f84f145b640 1 -- 172.21.15.12:0/2321108246 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f84ec09d460 msgr2=0x7f84ec13f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:22.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.874+0000 7f84f145b640 1 --2- 172.21.15.12:0/2321108246 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f84ec09d460 0x7f84ec13f5e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f84ec09dae0 tx=0x7f84e402a460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c57237e1059fa50f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:22.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.874+0000 7f84e37fe640 1 -- 172.21.15.12:0/2321108246 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e4007950 con 0x7f84ec09d460 2024-09-11T01:29:22.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.874+0000 7f84e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:22.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.874+0000 7f84e37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:22.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.874+0000 7f84e37fe640 10 monclient: dump: 2024-09-11T01:29:22.884 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:22.884 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:22.884 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:22.884 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:22.884 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:22.884 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:22.884 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:22.884 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:22.884 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:22.884 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:22.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.874+0000 7f84e37fe640 10 monclient: _finish_auth 0 2024-09-11T01:29:22.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.874+0000 7f84e37fe640 10 monclient: _check_auth_tickets 2024-09-11T01:29:22.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.874+0000 7f84e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:22.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.874+0000 7f84e37fe640 1 -- 172.21.15.12:0/2321108246 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84e4007af0 con 0x7f84ec09d460 2024-09-11T01:29:22.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.874+0000 7f84f2ebe640 5 monclient: authenticate success, global_id 4346 2024-09-11T01:29:22.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.874+0000 7f84f2ebe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:22.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.874+0000 7f84e37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:22.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.874+0000 7f84f2ebe640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:22.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.874+0000 7f84e37fe640 1 -- 172.21.15.12:0/2321108246 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e4031400 con 0x7f84ec09d460 2024-09-11T01:29:22.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.874+0000 7f84f2ebe640 4 set_mon_vals no callback set 2024-09-11T01:29:22.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.874+0000 7f84e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:22.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.874+0000 7f84e37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:22.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.874+0000 7f84e37fe640 10 monclient: dump: 2024-09-11T01:29:22.885 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:22.885 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:22.886 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:22.886 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:22.886 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:22.886 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:22.886 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:22.886 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:22.886 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:22.886 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:22.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.874+0000 7f84f2ebe640 10 monclient: shutdown 2024-09-11T01:29:22.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.874+0000 7f84f2ebe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:22.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.874+0000 7f84f2ebe640 1 -- 172.21.15.12:0/2321108246 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f84ec09d460 msgr2=0x7f84ec13f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:22.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.874+0000 7f84f2ebe640 1 --2- 172.21.15.12:0/2321108246 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f84ec09d460 0x7f84ec13f5e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f84ec09dae0 tx=0x7f84e402a460 comp rx=0 tx=0).stop 2024-09-11T01:29:22.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.875+0000 7f84f2ebe640 1 -- 172.21.15.12:0/2321108246 shutdown_connections 2024-09-11T01:29:22.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.875+0000 7f84f2ebe640 1 --2- 172.21.15.12:0/2321108246 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f84ec09d460 0x7f84ec13f5e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.875+0000 7f84f2ebe640 1 --2- 172.21.15.12:0/2321108246 >> v2:172.21.15.12:3300/0 conn(0x7f84ec09d030 0x7f84ec13d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.875+0000 7f84f2ebe640 1 -- 172.21.15.12:0/2321108246 >> 172.21.15.12:0/2321108246 conn(0x7f84ec0a7120 msgr2=0x7f84ec0a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:22.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.875+0000 7f84f2ebe640 1 -- 172.21.15.12:0/2321108246 shutdown_connections 2024-09-11T01:29:22.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.875+0000 7f84f2ebe640 1 -- 172.21.15.12:0/2321108246 wait complete. 2024-09-11T01:29:22.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.875+0000 7f84f2ebe640 10 monclient: build_initial_monmap 2024-09-11T01:29:22.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.875+0000 7f84f2ebe640 10 monclient: monmap: 2024-09-11T01:29:22.887 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:22.887 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:22.887 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:22.887 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:22.887 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:22.887 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:22.887 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:22.888 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:22.888 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:22.888 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:22.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.875+0000 7f84f2ebe640 1 Processor -- start 2024-09-11T01:29:22.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f2ebe640 1 -- start start 2024-09-11T01:29:22.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f2ebe640 10 monclient: init 2024-09-11T01:29:22.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f2ebe640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:22.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f2ebe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:29:22.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f2ebe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84ec09d030 0x7f84ec1cd140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:22.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f2ebe640 10 monclient(hunting): picked mon.noname-a con 0x7f84ec09d030 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:22.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f2ebe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f84ec09d460 0x7f84ec1371e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:22.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f2ebe640 10 monclient(hunting): picked mon.noname-c con 0x7f84ec09d460 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:22.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f2ebe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84ec138230 0x7f84ec13cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:22.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f2ebe640 10 monclient(hunting): picked mon.noname-b con 0x7f84ec138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:22.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f2ebe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:22.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f2ebe640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec13fca0 con 0x7f84ec09d030 2024-09-11T01:29:22.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f2ebe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:22.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f2ebe640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec13fb20 con 0x7f84ec138230 2024-09-11T01:29:22.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f2ebe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:22.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f2ebe640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f84ec13fe20 con 0x7f84ec09d460 2024-09-11T01:29:22.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f2ebe640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:22.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f2ebe640 10 monclient(hunting): authenticate will time out at 769.318622s 2024-09-11T01:29:22.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f0c5a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84ec09d030 0x7f84ec1cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:22.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84e3fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f84ec09d460 0x7f84ec1371e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:22.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84e3fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f84ec09d460 0x7f84ec1371e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51866/0 (socket says 172.21.15.12:51866) 2024-09-11T01:29:22.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f0c5a640 10 monclient(hunting): get_auth_request con 0x7f84ec09d030 auth_method 0 2024-09-11T01:29:22.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84e3fff640 1 -- 172.21.15.12:0/3868825975 learned_addr learned my addr 172.21.15.12:0/3868825975 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:22.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f0c5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:22.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f0c5a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:22.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f0c5a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:22.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84e3fff640 10 monclient(hunting): get_auth_request con 0x7f84ec09d460 auth_method 0 2024-09-11T01:29:22.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:22.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:22.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:22.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:22.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:22.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:22.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f0c5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:22.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f0c5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:22.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f0c5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:22.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84f145b640 1 --2- 172.21.15.12:0/3868825975 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84ec138230 0x7f84ec13cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:22.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.876+0000 7f84e3fff640 10 monclient(hunting): handle_auth_done global_id 4349 payload 824 2024-09-11T01:29:22.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.877+0000 7f84e3fff640 1 -- 172.21.15.12:0/3868825975 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84ec138230 msgr2=0x7f84ec13cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:22.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.877+0000 7f84e3fff640 1 --2- 172.21.15.12:0/3868825975 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84ec138230 0x7f84ec13cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.877+0000 7f84e3fff640 1 -- 172.21.15.12:0/3868825975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84ec09d030 msgr2=0x7f84ec1cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:22.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.877+0000 7f84e3fff640 1 --2- 172.21.15.12:0/3868825975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84ec09d030 0x7f84ec1cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:22.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.877+0000 7f84e3fff640 10 monclient: _finish_hunting 0 2024-09-11T01:29:22.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.877+0000 7f84e3fff640 1 monclient: found mon.noname-c 2024-09-11T01:29:22.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.877+0000 7f84e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:22.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.877+0000 7f84e3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:29:22.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.877+0000 7f84e3fff640 1 -- 172.21.15.12:0/3868825975 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84ec13e6c0 con 0x7f84ec09d460 2024-09-11T01:29:22.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.877+0000 7f84f145b640 10 monclient: get_auth_request con 0x7f84ec138230 auth_method 0 2024-09-11T01:29:22.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.877+0000 7f84f145b640 1 --2- 172.21.15.12:0/3868825975 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84ec138230 0x7f84ec13cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:29:22.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.879+0000 7f84e3fff640 1 -- 172.21.15.12:0/3868825975 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f84ec09d460 msgr2=0x7f84ec1371e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:22.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.879+0000 7f84e3fff640 1 --2- 172.21.15.12:0/3868825975 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f84ec09d460 0x7f84ec1371e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00e890 tx=0x7f84dc00ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:22.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.879+0000 7f84e17fa640 1 -- 172.21.15.12:0/3868825975 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84dc017070 con 0x7f84ec09d460 2024-09-11T01:29:22.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.879+0000 7f84e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:22.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.879+0000 7f84e17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:22.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.879+0000 7f84e17fa640 10 monclient: dump: 2024-09-11T01:29:22.893 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:22.893 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:22.893 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:22.893 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:22.893 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:22.893 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:22.893 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:22.893 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:22.893 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:22.893 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:22.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.879+0000 7f84e17fa640 10 monclient: _finish_auth 0 2024-09-11T01:29:22.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.879+0000 7f84e17fa640 10 monclient: _check_auth_tickets 2024-09-11T01:29:22.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.879+0000 7f84e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:22.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.879+0000 7f84e17fa640 1 -- 172.21.15.12:0/3868825975 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84dc00caf0 con 0x7f84ec09d460 2024-09-11T01:29:22.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.879+0000 7f84e17fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:22.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.879+0000 7f84e17fa640 1 -- 172.21.15.12:0/3868825975 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84dc013470 con 0x7f84ec09d460 2024-09-11T01:29:22.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.879+0000 7f84f2ebe640 5 monclient: authenticate success, global_id 4349 2024-09-11T01:29:22.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.879+0000 7f84f2ebe640 10 monclient: _renew_subs 2024-09-11T01:29:22.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.879+0000 7f84f2ebe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:22.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.879+0000 7f84f2ebe640 1 -- 172.21.15.12:0/3868825975 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84ec0a1810 con 0x7f84ec09d460 2024-09-11T01:29:22.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.879+0000 7f84e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:22.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.879+0000 7f84e17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:22.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.879+0000 7f84e17fa640 10 monclient: dump: 2024-09-11T01:29:22.894 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:22.894 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:22.894 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:22.894 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:22.895 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:22.895 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:22.895 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:22.895 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:22.895 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:22.895 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:22.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.879+0000 7f84e2ffd640 4 set_mon_vals no callback set 2024-09-11T01:29:22.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.879+0000 7f84f2ebe640 10 monclient: _renew_subs 2024-09-11T01:29:22.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.879+0000 7f84f2ebe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:22.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.879+0000 7f84f2ebe640 1 -- 172.21.15.12:0/3868825975 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f84ec120c70 con 0x7f84ec09d460 2024-09-11T01:29:22.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.880+0000 7f84f2ebe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:22.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.880+0000 7f84f2ebe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:22.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.880+0000 7f84f2ebe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:22.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.880+0000 7f84f2ebe640 1 -- 172.21.15.12:0/3868825975 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84ec138610 con 0x7f84ec09d460 2024-09-11T01:29:22.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.880+0000 7f84e17fa640 1 -- 172.21.15.12:0/3868825975 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f84dc024060 con 0x7f84ec09d460 2024-09-11T01:29:22.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.880+0000 7f84e17fa640 1 --2- 172.21.15.12:0/3868825975 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f84d803e290 0x7f84d8040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:22.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.880+0000 7f84e17fa640 1 -- 172.21.15.12:0/3868825975 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 4374+0+0 (secure 0 0 0) 0x7f84dc052b70 con 0x7f84ec09d460 2024-09-11T01:29:22.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.881+0000 7f84f0c5a640 1 --2- 172.21.15.12:0/3868825975 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f84d803e290 0x7f84d8040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:22.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.881+0000 7f84f0c5a640 10 monclient: get_auth_request con 0x7f84d803e290 auth_method 0 2024-09-11T01:29:22.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.883+0000 7f84f0c5a640 1 --2- 172.21.15.12:0/3868825975 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f84d803e290 0x7f84d8040750 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f84e8007c90 tx=0x7f84e8007be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:22.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.885+0000 7f84e17fa640 1 -- 172.21.15.12:0/3868825975 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f84dc023d00 con 0x7f84ec09d460 2024-09-11T01:29:22.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.885+0000 7f84e17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:22.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:22.885+0000 7f84e17fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:23.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.106+0000 7f84f2ebe640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-11T01:29:23.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.106+0000 7f84f2ebe640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-11T01:29:23.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.106+0000 7f84f2ebe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:23.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.106+0000 7f84f2ebe640 1 -- 172.21.15.12:0/3868825975 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f84ec0a1ab0 con 0x7f84ec09d460 2024-09-11T01:29:23.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.107+0000 7f84e17fa640 1 -- 172.21.15.12:0/3868825975 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f84dc0123d0 con 0x7f84ec09d460 2024-09-11T01:29:23.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.107+0000 7f84e17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-11T01:29:23.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.107+0000 7f84e17fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:23.109 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:29:23.109 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-11T01:29:23.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.109+0000 7f84f2ebe640 1 -- 172.21.15.12:0/3868825975 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f84d803e290 msgr2=0x7f84d8040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:23.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.109+0000 7f84f2ebe640 1 --2- 172.21.15.12:0/3868825975 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f84d803e290 0x7f84d8040750 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f84e8007c90 tx=0x7f84e8007be0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.109+0000 7f84f2ebe640 10 monclient: shutdown 2024-09-11T01:29:23.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.109+0000 7f84f2ebe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:23.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.109+0000 7f84f2ebe640 1 -- 172.21.15.12:0/3868825975 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f84ec09d460 msgr2=0x7f84ec1371e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:23.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.109+0000 7f84f2ebe640 1 --2- 172.21.15.12:0/3868825975 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f84ec09d460 0x7f84ec1371e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00e890 tx=0x7f84dc00ed50 comp rx=0 tx=0).stop 2024-09-11T01:29:23.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.109+0000 7f84f2ebe640 1 -- 172.21.15.12:0/3868825975 shutdown_connections 2024-09-11T01:29:23.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.109+0000 7f84f2ebe640 1 --2- 172.21.15.12:0/3868825975 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84ec138230 0x7f84ec13cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.109+0000 7f84f2ebe640 1 --2- 172.21.15.12:0/3868825975 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f84ec09d460 0x7f84ec1371e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.109+0000 7f84f2ebe640 1 --2- 172.21.15.12:0/3868825975 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f84d803e290 0x7f84d8040750 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.109+0000 7f84f2ebe640 1 --2- 172.21.15.12:0/3868825975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84ec09d030 0x7f84ec1cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.109+0000 7f84f2ebe640 1 -- 172.21.15.12:0/3868825975 >> 172.21.15.12:0/3868825975 conn(0x7f84ec0a7120 msgr2=0x7f84ec09e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:23.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.109+0000 7f84f2ebe640 1 -- 172.21.15.12:0/3868825975 shutdown_connections 2024-09-11T01:29:23.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.109+0000 7f84f2ebe640 1 -- 172.21.15.12:0/3868825975 wait complete. 2024-09-11T01:29:23.121 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-11T01:29:23.121 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-11T01:29:23.176 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-11T01:29:23.177 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T01:29:23.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.276+0000 7f1b4a833640 10 monclient: get_monmap_and_config 2024-09-11T01:29:23.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.276+0000 7f1b4a833640 10 monclient: build_initial_monmap 2024-09-11T01:29:23.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.276+0000 7f1b4a833640 10 monclient: monmap: 2024-09-11T01:29:23.279 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:23.279 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:23.279 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:23.277687+0000 2024-09-11T01:29:23.279 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:23.277687+0000 2024-09-11T01:29:23.279 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:23.279 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:23.279 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:23.279 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:23.279 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:23.279 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:23.279 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:23.279 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:23.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.277+0000 7f1b4a833640 1 Processor -- start 2024-09-11T01:29:23.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.277+0000 7f1b4a833640 1 -- start start 2024-09-11T01:29:23.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.277+0000 7f1b4a833640 10 monclient: init 2024-09-11T01:29:23.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b4a833640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:23.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b4a833640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-11T01:29:23.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b4a833640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1b44135fc0 0x7f1b441363a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:23.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b4a833640 10 monclient(hunting): picked mon.noname-a con 0x7f1b44135fc0 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:23.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b4a833640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1b44073fe0 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:23.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b4a833640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1b440743f0 0x7f1b440a34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:23.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b4a833640 10 monclient(hunting): picked mon.noname-c con 0x7f1b440743f0 addr v2:172.21.15.71:3300/0 2024-09-11T01:29:23.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b4a833640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b44072fb0 con 0x7f1b44073fe0 2024-09-11T01:29:23.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b4a833640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:23.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b4a833640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f1b44073850 con 0x7f1b44135fc0 2024-09-11T01:29:23.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b4a833640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:23.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b4a833640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f1b44073290 con 0x7f1b440743f0 2024-09-11T01:29:23.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b4a833640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:23.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b4a833640 10 monclient(hunting): authenticate will time out at 769.720389s 2024-09-11T01:29:23.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b437fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1b44073fe0 0x7f1b440a2db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56130/0 (socket says 172.21.15.12:56130) 2024-09-11T01:29:23.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b437fe640 1 -- 172.21.15.12:0/300481975 learned_addr learned my addr 172.21.15.12:0/300481975 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:29:23.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b43fff640 1 --2- 172.21.15.12:0/300481975 >> v2:172.21.15.12:3300/0 conn(0x7f1b44135fc0 0x7f1b441363a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:23.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b43fff640 10 monclient(hunting): get_auth_request con 0x7f1b44135fc0 auth_method 0 2024-09-11T01:29:23.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:23.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b43fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:23.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:23.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:23.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:23.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:23.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b43fff640 10 monclient(hunting): handle_auth_done global_id 4326 payload 274 2024-09-11T01:29:23.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b43fff640 1 -- 172.21.15.12:0/300481975 >> v2:172.21.15.71:3300/0 conn(0x7f1b440743f0 msgr2=0x7f1b440a34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:23.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b43fff640 1 --2- 172.21.15.12:0/300481975 >> v2:172.21.15.71:3300/0 conn(0x7f1b440743f0 0x7f1b440a34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b43fff640 1 -- 172.21.15.12:0/300481975 >> v1:172.21.15.12:6789/0 conn(0x7f1b44073fe0 legacy=0x7f1b440a2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:23.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b43fff640 10 monclient: _finish_hunting 0 2024-09-11T01:29:23.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b43fff640 1 monclient: found mon.noname-a 2024-09-11T01:29:23.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:23.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b43fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:23.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.278+0000 7f1b43fff640 1 -- 172.21.15.12:0/300481975 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b44073570 con 0x7f1b44135fc0 2024-09-11T01:29:23.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.281+0000 7f1b43fff640 1 -- 172.21.15.12:0/300481975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b44135fc0 msgr2=0x7f1b441363a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:23.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.281+0000 7f1b43fff640 1 --2- 172.21.15.12:0/300481975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b44135fc0 0x7f1b441363a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1b440a4790 tx=0x7f1b3402a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6147c32d57a01e09 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:23.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.281+0000 7f1b42ffd640 1 -- 172.21.15.12:0/300481975 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b34037070 con 0x7f1b44135fc0 2024-09-11T01:29:23.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.281+0000 7f1b42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:23.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.281+0000 7f1b42ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:23.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.281+0000 7f1b42ffd640 10 monclient: dump: 2024-09-11T01:29:23.289 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:23.289 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:23.289 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:23.289 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:23.290 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:23.290 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:23.290 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:23.290 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:23.290 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:23.290 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:23.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.281+0000 7f1b42ffd640 10 monclient: _finish_auth 0 2024-09-11T01:29:23.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.281+0000 7f1b42ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:23.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.281+0000 7f1b42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:23.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.281+0000 7f1b42ffd640 1 -- 172.21.15.12:0/300481975 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b34032640 con 0x7f1b44135fc0 2024-09-11T01:29:23.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.281+0000 7f1b42ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:23.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.281+0000 7f1b42ffd640 1 -- 172.21.15.12:0/300481975 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b34032a10 con 0x7f1b44135fc0 2024-09-11T01:29:23.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.281+0000 7f1b4a833640 5 monclient: authenticate success, global_id 4326 2024-09-11T01:29:23.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.281+0000 7f1b4a833640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:23.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.281+0000 7f1b4a833640 4 set_mon_vals no callback set 2024-09-11T01:29:23.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.281+0000 7f1b42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:23.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.281+0000 7f1b42ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:23.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.281+0000 7f1b42ffd640 10 monclient: dump: 2024-09-11T01:29:23.291 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:23.291 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:23.291 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:23.291 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:23.291 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:23.291 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:23.291 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:23.291 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:23.291 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:23.292 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:23.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.282+0000 7f1b4a833640 10 monclient: shutdown 2024-09-11T01:29:23.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.282+0000 7f1b4a833640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:23.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.282+0000 7f1b4a833640 1 -- 172.21.15.12:0/300481975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b44135fc0 msgr2=0x7f1b441363a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:23.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.282+0000 7f1b4a833640 1 --2- 172.21.15.12:0/300481975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b44135fc0 0x7f1b441363a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1b440a4790 tx=0x7f1b3402a000 comp rx=0 tx=0).stop 2024-09-11T01:29:23.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.282+0000 7f1b4a833640 1 -- 172.21.15.12:0/300481975 shutdown_connections 2024-09-11T01:29:23.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.282+0000 7f1b4a833640 1 --2- 172.21.15.12:0/300481975 >> v2:172.21.15.71:3300/0 conn(0x7f1b440743f0 0x7f1b440a34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.282+0000 7f1b4a833640 1 --2- 172.21.15.12:0/300481975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b44135fc0 0x7f1b441363a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.282+0000 7f1b4a833640 1 -- 172.21.15.12:0/300481975 >> 172.21.15.12:0/300481975 conn(0x7f1b440a83b0 msgr2=0x7f1b440a87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:23.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.282+0000 7f1b4a833640 1 -- 172.21.15.12:0/300481975 shutdown_connections 2024-09-11T01:29:23.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.282+0000 7f1b4a833640 1 -- 172.21.15.12:0/300481975 wait complete. 2024-09-11T01:29:23.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.283+0000 7f1b4a833640 10 monclient: build_initial_monmap 2024-09-11T01:29:23.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.283+0000 7f1b4a833640 10 monclient: monmap: 2024-09-11T01:29:23.293 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:23.293 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:23.293 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:23.293 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:23.293 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:23.293 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:23.293 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:23.293 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:23.293 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:23.293 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:23.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.283+0000 7f1b4a833640 1 Processor -- start 2024-09-11T01:29:23.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.283+0000 7f1b4a833640 1 -- start start 2024-09-11T01:29:23.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.283+0000 7f1b4a833640 10 monclient: init 2024-09-11T01:29:23.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b4a833640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:23.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b4a833640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:29:23.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b4a833640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b44073fe0 0x7f1b441cd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:23.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b4a833640 10 monclient(hunting): picked mon.noname-b con 0x7f1b44073fe0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:23.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b4a833640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b440743f0 0x7f1b441320b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:23.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b4a833640 10 monclient(hunting): picked mon.noname-c con 0x7f1b440743f0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:23.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b4a833640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b44135fc0 0x7f1b441c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:23.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b4a833640 10 monclient(hunting): picked mon.noname-a con 0x7f1b44135fc0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:23.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b4a833640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:23.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b4a833640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1b44136a60 con 0x7f1b44135fc0 2024-09-11T01:29:23.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b4a833640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:23.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b4a833640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1b441368e0 con 0x7f1b44073fe0 2024-09-11T01:29:23.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b4a833640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:23.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b4a833640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1b440a3a00 con 0x7f1b440743f0 2024-09-11T01:29:23.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b4a833640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:23.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b4a833640 10 monclient(hunting): authenticate will time out at 769.726124s 2024-09-11T01:29:23.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b48dd0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b44135fc0 0x7f1b441c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:23.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b437fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b440743f0 0x7f1b441320b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:23.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b437fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b440743f0 0x7f1b441320b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51870/0 (socket says 172.21.15.12:51870) 2024-09-11T01:29:23.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b48dd0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b44135fc0 0x7f1b441c9d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55148/0 (socket says 172.21.15.12:55148) 2024-09-11T01:29:23.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b437fe640 1 -- 172.21.15.12:0/1971384328 learned_addr learned my addr 172.21.15.12:0/1971384328 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:23.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b437fe640 10 monclient(hunting): get_auth_request con 0x7f1b440743f0 auth_method 0 2024-09-11T01:29:23.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b43fff640 1 --2- 172.21.15.12:0/1971384328 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b44073fe0 0x7f1b441cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:23.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:23.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b437fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:23.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:23.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b48dd0640 10 monclient(hunting): get_auth_request con 0x7f1b44135fc0 auth_method 0 2024-09-11T01:29:23.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b48dd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:23.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b48dd0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:23.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b48dd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:23.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b43fff640 10 monclient(hunting): get_auth_request con 0x7f1b44073fe0 auth_method 0 2024-09-11T01:29:23.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:23.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b43fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:23.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:23.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b48dd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:23.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b48dd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:23.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b48dd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:23.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:23.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:23.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:23.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:23.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:23.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:23.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b48dd0640 10 monclient(hunting): handle_auth_done global_id 4329 payload 824 2024-09-11T01:29:23.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b48dd0640 1 -- 172.21.15.12:0/1971384328 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b440743f0 msgr2=0x7f1b441320b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:23.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b48dd0640 1 --2- 172.21.15.12:0/1971384328 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b440743f0 0x7f1b441320b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b48dd0640 1 -- 172.21.15.12:0/1971384328 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b44073fe0 msgr2=0x7f1b441cd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:23.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b48dd0640 1 --2- 172.21.15.12:0/1971384328 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b44073fe0 0x7f1b441cd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b48dd0640 10 monclient: _finish_hunting 0 2024-09-11T01:29:23.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b48dd0640 1 monclient: found mon.noname-a 2024-09-11T01:29:23.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b48dd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:23.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b48dd0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:23.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b48dd0640 1 -- 172.21.15.12:0/1971384328 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b4411eaa0 con 0x7f1b44135fc0 2024-09-11T01:29:23.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.284+0000 7f1b437fe640 1 --2- 172.21.15.12:0/1971384328 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b440743f0 0x7f1b441320b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:23.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.286+0000 7f1b48dd0640 1 -- 172.21.15.12:0/1971384328 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b44135fc0 msgr2=0x7f1b441c9d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:23.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.286+0000 7f1b48dd0640 1 --2- 172.21.15.12:0/1971384328 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b44135fc0 0x7f1b441c9d30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c00be50 tx=0x7f1b3c00bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:23.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.286+0000 7f1b40ff9640 1 -- 172.21.15.12:0/1971384328 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c019070 con 0x7f1b44135fc0 2024-09-11T01:29:23.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.286+0000 7f1b40ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:23.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.286+0000 7f1b40ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:23.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.286+0000 7f1b40ff9640 10 monclient: dump: 2024-09-11T01:29:23.298 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:23.298 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:23.298 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:23.298 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:23.298 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:23.298 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:23.299 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:23.299 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:23.299 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:23.299 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:23.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.286+0000 7f1b40ff9640 10 monclient: _finish_auth 0 2024-09-11T01:29:23.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.286+0000 7f1b40ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:29:23.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.286+0000 7f1b40ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:23.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.286+0000 7f1b40ff9640 1 -- 172.21.15.12:0/1971384328 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b3c014660 con 0x7f1b44135fc0 2024-09-11T01:29:23.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.286+0000 7f1b40ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:23.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.286+0000 7f1b40ff9640 1 -- 172.21.15.12:0/1971384328 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c014a50 con 0x7f1b44135fc0 2024-09-11T01:29:23.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.286+0000 7f1b4a833640 5 monclient: authenticate success, global_id 4329 2024-09-11T01:29:23.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.286+0000 7f1b4a833640 10 monclient: _renew_subs 2024-09-11T01:29:23.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.287+0000 7f1b4a833640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:23.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.287+0000 7f1b4a833640 1 -- 172.21.15.12:0/1971384328 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b4409f5a0 con 0x7f1b44135fc0 2024-09-11T01:29:23.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.287+0000 7f1b40ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:23.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.287+0000 7f1b40ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:23.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.287+0000 7f1b40ff9640 10 monclient: dump: 2024-09-11T01:29:23.300 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:23.300 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:23.300 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:23.300 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:23.300 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:23.300 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:23.300 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:23.301 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:23.301 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:23.301 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:23.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.286+0000 7f1b427fc640 4 set_mon_vals no callback set 2024-09-11T01:29:23.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.287+0000 7f1b4a833640 10 monclient: _renew_subs 2024-09-11T01:29:23.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.287+0000 7f1b4a833640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:23.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.287+0000 7f1b4a833640 1 -- 172.21.15.12:0/1971384328 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b441cdd20 con 0x7f1b44135fc0 2024-09-11T01:29:23.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.287+0000 7f1b40ff9640 1 -- 172.21.15.12:0/1971384328 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f1b3c01d3b0 con 0x7f1b44135fc0 2024-09-11T01:29:23.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.288+0000 7f1b4a833640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:23.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.288+0000 7f1b4a833640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:23.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.288+0000 7f1b4a833640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:23.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.288+0000 7f1b4a833640 1 -- 172.21.15.12:0/1971384328 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b441363a0 con 0x7f1b44135fc0 2024-09-11T01:29:23.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.288+0000 7f1b40ff9640 1 --2- 172.21.15.12:0/1971384328 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1b3003e260 0x7f1b30040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:23.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.288+0000 7f1b40ff9640 1 -- 172.21.15.12:0/1971384328 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4374+0+0 (secure 0 0 0) 0x7f1b3c053280 con 0x7f1b44135fc0 2024-09-11T01:29:23.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.288+0000 7f1b43fff640 1 --2- 172.21.15.12:0/1971384328 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1b3003e260 0x7f1b30040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:23.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.288+0000 7f1b43fff640 10 monclient: get_auth_request con 0x7f1b3003e260 auth_method 0 2024-09-11T01:29:23.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.291+0000 7f1b43fff640 1 --2- 172.21.15.12:0/1971384328 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1b3003e260 0x7f1b30040720 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1b44073d00 tx=0x7f1b3402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:23.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.292+0000 7f1b40ff9640 1 -- 172.21.15.12:0/1971384328 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1b3c051e10 con 0x7f1b44135fc0 2024-09-11T01:29:23.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.292+0000 7f1b40ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:23.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.292+0000 7f1b40ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:23.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.477+0000 7f1b4a833640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-11T01:29:23.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.477+0000 7f1b4a833640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T01:29:23.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.477+0000 7f1b4a833640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:23.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.477+0000 7f1b4a833640 1 -- 172.21.15.12:0/1971384328 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1b440a7a30 con 0x7f1b44135fc0 2024-09-11T01:29:23.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.478+0000 7f1b40ff9640 1 -- 172.21.15.12:0/1971384328 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12330 (secure 0 0 0) 0x7f1b3c01c360 con 0x7f1b44135fc0 2024-09-11T01:29:23.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.478+0000 7f1b40ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T01:29:23.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.478+0000 7f1b40ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:23.480 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:29:23.480 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":17,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","created":"2024-09-11T01:29:02.540751+0000","modified":"2024-09-11T01:29:22.315249+0000","last_up_change":"2024-09-11T01:29:13.320573+0000","last_in_change":"2024-09-11T01:29:05.075851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-11T01:29:14.157646+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-11T01:29:14.872725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"616ba1d0-adf6-4fe1-85e3-266e53e21867","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":4266003406},{"type":"v1","addr":"172.21.15.12:6817","nonce":4266003406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":4266003406},{"type":"v1","addr":"172.21.15.12:6819","nonce":4266003406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":4266003406},{"type":"v1","addr":"172.21.15.12:6823","nonce":4266003406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":4266003406},{"type":"v1","addr":"172.21.15.12:6821","nonce":4266003406}]},"public_addr":"172.21.15.12:6817/4266003406","cluster_addr":"172.21.15.12:6819/4266003406","heartbeat_back_addr":"172.21.15.12:6823/4266003406","heartbeat_front_addr":"172.21.15.12:6821/4266003406","state":["exists","up"]},{"osd":1,"uuid":"9b570874-7a3d-4867-8c44-29084a8bf649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3563357280},{"type":"v1","addr":"172.21.15.12:6809","nonce":3563357280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3563357280},{"type":"v1","addr":"172.21.15.12:6811","nonce":3563357280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3563357280},{"type":"v1","addr":"172.21.15.12:6815","nonce":3563357280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3563357280},{"type":"v1","addr":"172.21.15.12:6813","nonce":3563357280}]},"public_addr":"172.21.15.12:6809/3563357280","cluster_addr":"172.21.15.12:6811/3563357280","heartbeat_back_addr":"172.21.15.12:6815/3563357280","heartbeat_front_addr":"172.21.15.12:6813/3563357280","state":["exists","up"]},{"osd":2,"uuid":"b12ee288-ce57-41de-8b8e-6fad509311b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2893036867},{"type":"v1","addr":"172.21.15.12:6801","nonce":2893036867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2893036867},{"type":"v1","addr":"172.21.15.12:6803","nonce":2893036867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2893036867},{"type":"v1","addr":"172.21.15.12:6807","nonce":2893036867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2893036867},{"type":"v1","addr":"172.21.15.12:6805","nonce":2893036867}]},"public_addr":"172.21.15.12:6801/2893036867","cluster_addr":"172.21.15.12:6803/2893036867","heartbeat_back_addr":"172.21.15.12:6807/2893036867","heartbeat_front_addr":"172.21.15.12:6805/2893036867","state":["exists","up"]},{"osd":3,"uuid":"a8e81aa4-0dee-487d-981d-1f593e363eee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":3091488141},{"type":"v1","addr":"172.21.15.71:6819","nonce":3091488141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":3091488141},{"type":"v1","addr":"172.21.15.71:6821","nonce":3091488141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":3091488141},{"type":"v1","addr":"172.21.15.71:6825","nonce":3091488141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":3091488141},{"type":"v1","addr":"172.21.15.71:6823","nonce":3091488141}]},"public_addr":"172.21.15.71:6819/3091488141","cluster_addr":"172.21.15.71:6821/3091488141","heartbeat_back_addr":"172.21.15.71:6825/3091488141","heartbeat_front_addr":"172.21.15.71:6823/3091488141","state":["exists","up"]},{"osd":4,"uuid":"9db77181-fc18-408f-a35c-87bf4b4ba2d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":1790415427},{"type":"v1","addr":"172.21.15.71:6801","nonce":1790415427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":1790415427},{"type":"v1","addr":"172.21.15.71:6803","nonce":1790415427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":1790415427},{"type":"v1","addr":"172.21.15.71:6807","nonce":1790415427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":1790415427},{"type":"v1","addr":"172.21.15.71:6805","nonce":1790415427}]},"public_addr":"172.21.15.71:6801/1790415427","cluster_addr":"172.21.15.71:6803/1790415427","heartbeat_back_addr":"172.21.15.71:6807/1790415427","heartbeat_front_addr":"172.21.15.71:6805/1790415427","state":["exists","up"]},{"osd":5,"uuid":"fadf2191-f31a-4880-a529-5b2d1023d236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":2032569317},{"type":"v1","addr":"172.21.15.71:6809","nonce":2032569317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":2032569317},{"type":"v1","addr":"172.21.15.71:6811","nonce":2032569317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":2032569317},{"type":"v1","addr":"172.21.15.71:6815","nonce":2032569317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":2032569317},{"type":"v1","addr":"172.21.15.71:6813","nonce":2032569317}]},"public_addr":"172.21.15.71:6809/2032569317","cluster_addr":"172.21.15.71:6811/2032569317","heartbeat_back_addr":"172.21.15.71:6815/2032569317","heartbeat_front_addr":"172.21.15.71:6813/2032569317","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:11.548818+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:11.310103+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:10.578802+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:11.748432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:11.098916+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:11.150538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T01:29:23.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.480+0000 7f1b4a833640 1 -- 172.21.15.12:0/1971384328 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1b3003e260 msgr2=0x7f1b30040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:23.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.480+0000 7f1b4a833640 1 --2- 172.21.15.12:0/1971384328 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1b3003e260 0x7f1b30040720 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1b44073d00 tx=0x7f1b3402e000 comp rx=0 tx=0).stop 2024-09-11T01:29:23.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.480+0000 7f1b4a833640 10 monclient: shutdown 2024-09-11T01:29:23.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.480+0000 7f1b4a833640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:23.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.480+0000 7f1b4a833640 1 -- 172.21.15.12:0/1971384328 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b44135fc0 msgr2=0x7f1b441c9d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:23.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.480+0000 7f1b4a833640 1 --2- 172.21.15.12:0/1971384328 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b44135fc0 0x7f1b441c9d30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c00be50 tx=0x7f1b3c00bf40 comp rx=0 tx=0).stop 2024-09-11T01:29:23.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.480+0000 7f1b4a833640 1 -- 172.21.15.12:0/1971384328 shutdown_connections 2024-09-11T01:29:23.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.480+0000 7f1b4a833640 1 --2- 172.21.15.12:0/1971384328 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1b3003e260 0x7f1b30040720 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.480+0000 7f1b4a833640 1 --2- 172.21.15.12:0/1971384328 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b44135fc0 0x7f1b441c9d30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.480+0000 7f1b4a833640 1 --2- 172.21.15.12:0/1971384328 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b440743f0 0x7f1b441320b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.480+0000 7f1b4a833640 1 --2- 172.21.15.12:0/1971384328 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b44073fe0 0x7f1b441cd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.480+0000 7f1b4a833640 1 -- 172.21.15.12:0/1971384328 >> 172.21.15.12:0/1971384328 conn(0x7f1b440a83b0 msgr2=0x7f1b440a6b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:23.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.480+0000 7f1b4a833640 1 -- 172.21.15.12:0/1971384328 shutdown_connections 2024-09-11T01:29:23.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.480+0000 7f1b4a833640 1 -- 172.21.15.12:0/1971384328 wait complete. 2024-09-11T01:29:23.493 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-09-11T01:29:14.157646+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 16, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-09-11T01:29:14.872725+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-11T01:29:23.493 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-11T01:29:23.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.597+0000 7f149d2c6640 10 monclient: get_monmap_and_config 2024-09-11T01:29:23.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.597+0000 7f149d2c6640 10 monclient: build_initial_monmap 2024-09-11T01:29:23.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.597+0000 7f149d2c6640 10 monclient: monmap: 2024-09-11T01:29:23.599 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:23.599 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:23.599 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:23.598331+0000 2024-09-11T01:29:23.599 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:23.598331+0000 2024-09-11T01:29:23.599 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:23.600 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:23.600 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:23.600 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:23.600 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:23.600 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:23.600 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:23.600 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:23.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.598+0000 7f149d2c6640 1 Processor -- start 2024-09-11T01:29:23.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.598+0000 7f149d2c6640 1 -- start start 2024-09-11T01:29:23.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.598+0000 7f149d2c6640 10 monclient: init 2024-09-11T01:29:23.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.598+0000 7f149d2c6640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:23.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.598+0000 7f149d2c6640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-11T01:29:23.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.598+0000 7f149d2c6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f14980a5d00 0x7f14980a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:23.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.598+0000 7f149d2c6640 10 monclient(hunting): picked mon.noname-c con 0x7f14980a5d00 addr v2:172.21.15.71:3300/0 2024-09-11T01:29:23.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.598+0000 7f149d2c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14980a60e0 0x7f1498137f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:23.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.598+0000 7f149d2c6640 10 monclient(hunting): picked mon.noname-b con 0x7f14980a60e0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:23.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.598+0000 7f149d2c6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1498138470 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:23.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.598+0000 7f149d2c6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f149809d890 con 0x7f1498138470 2024-09-11T01:29:23.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.598+0000 7f149d2c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:23.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.598+0000 7f149d2c6640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f14980a53d0 con 0x7f14980a5d00 2024-09-11T01:29:23.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.598+0000 7f149d2c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:23.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.598+0000 7f149d2c6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f149809c560 con 0x7f14980a60e0 2024-09-11T01:29:23.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.598+0000 7f149d2c6640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:23.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.598+0000 7f149d2c6640 10 monclient(hunting): authenticate will time out at 770.040979s 2024-09-11T01:29:23.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f14967fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14980a60e0 0x7f1498137f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:23.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f14967fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14980a60e0 0x7f1498137f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51876/0 (socket says 172.21.15.12:51876) 2024-09-11T01:29:23.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f14967fc640 1 -- 172.21.15.12:0/3049702629 learned_addr learned my addr 172.21.15.12:0/3049702629 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:23.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f14967fc640 10 monclient(hunting): get_auth_request con 0x7f14980a60e0 auth_method 0 2024-09-11T01:29:23.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f14967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:23.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f14967fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:23.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f14967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:23.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f1496ffd640 1 --2- 172.21.15.12:0/3049702629 >> v2:172.21.15.71:3300/0 conn(0x7f14980a5d00 0x7f14980a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:23.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f1496ffd640 10 monclient(hunting): get_auth_request con 0x7f14980a5d00 auth_method 0 2024-09-11T01:29:23.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f1496ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:23.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f1496ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:23.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f1496ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:23.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f14967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:23.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f14967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:23.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f14967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:23.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f1496ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:23.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f1496ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:23.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f1496ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:23.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f14967fc640 10 monclient(hunting): handle_auth_done global_id 4355 payload 274 2024-09-11T01:29:23.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f14967fc640 1 -- 172.21.15.12:0/3049702629 >> v2:172.21.15.71:3300/0 conn(0x7f14980a5d00 msgr2=0x7f14980a4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:23.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f14967fc640 1 --2- 172.21.15.12:0/3049702629 >> v2:172.21.15.71:3300/0 conn(0x7f14980a5d00 0x7f14980a4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f14967fc640 1 -- 172.21.15.12:0/3049702629 >> v1:172.21.15.71:6789/0 conn(0x7f1498138470 legacy=0x7f149813f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:23.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f14967fc640 10 monclient: _finish_hunting 0 2024-09-11T01:29:23.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f14967fc640 1 monclient: found mon.noname-b 2024-09-11T01:29:23.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f14967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:23.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f14967fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:29:23.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.599+0000 7f14967fc640 1 -- 172.21.15.12:0/3049702629 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f149809c280 con 0x7f14980a60e0 2024-09-11T01:29:23.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.602+0000 7f14967fc640 1 -- 172.21.15.12:0/3049702629 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14980a60e0 msgr2=0x7f1498137f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:23.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.602+0000 7f14967fc640 1 --2- 172.21.15.12:0/3049702629 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14980a60e0 0x7f1498137f30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f148c009870 tx=0x7f148c02a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a8ba2bcc0a42eed server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:23.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.602+0000 7f1495ffb640 1 -- 172.21.15.12:0/3049702629 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f148c037070 con 0x7f14980a60e0 2024-09-11T01:29:23.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.602+0000 7f1495ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:23.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.602+0000 7f1495ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:23.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.602+0000 7f1495ffb640 10 monclient: dump: 2024-09-11T01:29:23.611 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:23.611 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:23.611 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:23.611 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:23.611 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:23.612 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:23.612 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:23.612 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:23.612 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:23.612 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:23.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.602+0000 7f1495ffb640 10 monclient: _finish_auth 0 2024-09-11T01:29:23.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.602+0000 7f1495ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:29:23.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.602+0000 7f1495ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:23.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.602+0000 7f1495ffb640 1 -- 172.21.15.12:0/3049702629 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f148c032440 con 0x7f14980a60e0 2024-09-11T01:29:23.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.602+0000 7f149d2c6640 5 monclient: authenticate success, global_id 4355 2024-09-11T01:29:23.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.602+0000 7f149d2c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:23.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.602+0000 7f1495ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:23.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.602+0000 7f149d2c6640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.602+0000 7f1495ffb640 1 -- 172.21.15.12:0/3049702629 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f148c032810 con 0x7f14980a60e0 2024-09-11T01:29:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.602+0000 7f149d2c6640 4 set_mon_vals no callback set 2024-09-11T01:29:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.602+0000 7f1495ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.602+0000 7f1495ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.602+0000 7f1495ffb640 10 monclient: dump: 2024-09-11T01:29:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:23.614 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:23.614 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:23.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.603+0000 7f149d2c6640 10 monclient: shutdown 2024-09-11T01:29:23.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.603+0000 7f149d2c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:23.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.603+0000 7f149d2c6640 1 -- 172.21.15.12:0/3049702629 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14980a60e0 msgr2=0x7f1498137f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:23.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.603+0000 7f149d2c6640 1 --2- 172.21.15.12:0/3049702629 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14980a60e0 0x7f1498137f30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f148c009870 tx=0x7f148c02a0a0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.603+0000 7f149d2c6640 1 -- 172.21.15.12:0/3049702629 shutdown_connections 2024-09-11T01:29:23.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.603+0000 7f149d2c6640 1 --2- 172.21.15.12:0/3049702629 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14980a60e0 0x7f1498137f30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.603+0000 7f149d2c6640 1 --2- 172.21.15.12:0/3049702629 >> v2:172.21.15.71:3300/0 conn(0x7f14980a5d00 0x7f14980a4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.603+0000 7f149d2c6640 1 -- 172.21.15.12:0/3049702629 >> 172.21.15.12:0/3049702629 conn(0x7f149806ee50 msgr2=0x7f149806f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:23.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.603+0000 7f149d2c6640 1 -- 172.21.15.12:0/3049702629 shutdown_connections 2024-09-11T01:29:23.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.603+0000 7f149d2c6640 1 -- 172.21.15.12:0/3049702629 wait complete. 2024-09-11T01:29:23.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.603+0000 7f149d2c6640 10 monclient: build_initial_monmap 2024-09-11T01:29:23.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.603+0000 7f149d2c6640 10 monclient: monmap: 2024-09-11T01:29:23.615 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:23.615 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:23.615 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:23.615 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:23.615 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:23.615 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:23.616 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:23.616 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:23.616 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:23.616 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:23.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f149d2c6640 1 Processor -- start 2024-09-11T01:29:23.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f149d2c6640 1 -- start start 2024-09-11T01:29:23.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f149d2c6640 10 monclient: init 2024-09-11T01:29:23.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f149d2c6640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:23.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f149d2c6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:29:23.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f149d2c6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14980a5d00 0x7f14981cb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:23.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f149d2c6640 10 monclient(hunting): picked mon.noname-a con 0x7f14980a5d00 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:23.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f149d2c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14980a60e0 0x7f1498136d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:23.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f149d2c6640 10 monclient(hunting): picked mon.noname-c con 0x7f14980a60e0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:23.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f149d2c6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1498138470 0x7f1498137290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:23.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f149d2c6640 10 monclient(hunting): picked mon.noname-b con 0x7f1498138470 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:23.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f149d2c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:23.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f149d2c6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f14980a6bd0 con 0x7f14980a5d00 2024-09-11T01:29:23.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f149d2c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:23.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f149d2c6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f14980a6a50 con 0x7f1498138470 2024-09-11T01:29:23.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f149d2c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:23.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f149d2c6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1498138900 con 0x7f14980a60e0 2024-09-11T01:29:23.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f149d2c6640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:23.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f149d2c6640 10 monclient(hunting): authenticate will time out at 770.046716s 2024-09-11T01:29:23.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f14967fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14980a60e0 0x7f1498136d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:23.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f1496ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14980a5d00 0x7f14981cb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:23.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f1496ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14980a5d00 0x7f14981cb2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55158/0 (socket says 172.21.15.12:55158) 2024-09-11T01:29:23.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f14967fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14980a60e0 0x7f1498136d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51878/0 (socket says 172.21.15.12:51878) 2024-09-11T01:29:23.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f1496ffd640 1 -- 172.21.15.12:0/2733588502 learned_addr learned my addr 172.21.15.12:0/2733588502 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:23.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f1496ffd640 10 monclient(hunting): get_auth_request con 0x7f14980a5d00 auth_method 0 2024-09-11T01:29:23.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f1496ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:23.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f1496ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:23.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f1496ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:23.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f14967fc640 10 monclient(hunting): get_auth_request con 0x7f14980a60e0 auth_method 0 2024-09-11T01:29:23.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f14967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:23.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f14967fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:23.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f14967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:23.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f1496ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f1496ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f1496ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f14967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f14967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f14967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f14977fe640 1 --2- 172.21.15.12:0/2733588502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1498138470 0x7f1498137290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.604+0000 7f14977fe640 10 monclient(hunting): get_auth_request con 0x7f1498138470 auth_method 0 2024-09-11T01:29:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.605+0000 7f14977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.605+0000 7f14977fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.605+0000 7f14977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.605+0000 7f14977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.605+0000 7f14977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.605+0000 7f14977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.605+0000 7f1496ffd640 10 monclient(hunting): handle_auth_done global_id 4332 payload 824 2024-09-11T01:29:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.605+0000 7f1496ffd640 1 -- 172.21.15.12:0/2733588502 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14980a60e0 msgr2=0x7f1498136d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.605+0000 7f1496ffd640 1 --2- 172.21.15.12:0/2733588502 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14980a60e0 0x7f1498136d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.605+0000 7f1496ffd640 1 -- 172.21.15.12:0/2733588502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1498138470 msgr2=0x7f1498137290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.605+0000 7f1496ffd640 1 --2- 172.21.15.12:0/2733588502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1498138470 0x7f1498137290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.605+0000 7f1496ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:29:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.605+0000 7f1496ffd640 1 monclient: found mon.noname-a 2024-09-11T01:29:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.605+0000 7f1496ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.605+0000 7f1496ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.605+0000 7f1496ffd640 1 -- 172.21.15.12:0/2733588502 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1498124810 con 0x7f14980a5d00 2024-09-11T01:29:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.605+0000 7f14967fc640 1 --2- 172.21.15.12:0/2733588502 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14980a60e0 0x7f1498136d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.607+0000 7f1496ffd640 1 -- 172.21.15.12:0/2733588502 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14980a5d00 msgr2=0x7f14981cb2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.607+0000 7f1496ffd640 1 --2- 172.21.15.12:0/2733588502 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14980a5d00 0x7f14981cb2a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f148800e9e0 tx=0x7f148800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.607+0000 7f147f7fe640 1 -- 172.21.15.12:0/2733588502 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1488017070 con 0x7f14980a5d00 2024-09-11T01:29:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.607+0000 7f147f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.607+0000 7f147f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.607+0000 7f147f7fe640 10 monclient: dump: 2024-09-11T01:29:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:23.621 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.607+0000 7f147f7fe640 10 monclient: _finish_auth 0 2024-09-11T01:29:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.607+0000 7f147f7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:29:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.607+0000 7f147f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.607+0000 7f149d2c6640 5 monclient: authenticate success, global_id 4332 2024-09-11T01:29:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.607+0000 7f147f7fe640 1 -- 172.21.15.12:0/2733588502 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f148800cbe0 con 0x7f14980a5d00 2024-09-11T01:29:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.607+0000 7f149d2c6640 10 monclient: _renew_subs 2024-09-11T01:29:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.607+0000 7f149d2c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.607+0000 7f149d2c6640 1 -- 172.21.15.12:0/2733588502 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14981377d0 con 0x7f14980a5d00 2024-09-11T01:29:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.607+0000 7f147f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.607+0000 7f147f7fe640 1 -- 172.21.15.12:0/2733588502 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1488013470 con 0x7f14980a5d00 2024-09-11T01:29:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.607+0000 7f147f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.607+0000 7f147f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.607+0000 7f147f7fe640 10 monclient: dump: 2024-09-11T01:29:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:23.623 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.607+0000 7f149d2c6640 10 monclient: _renew_subs 2024-09-11T01:29:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.607+0000 7f149d2c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.607+0000 7f149d2c6640 1 -- 172.21.15.12:0/2733588502 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1498123f70 con 0x7f14980a5d00 2024-09-11T01:29:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.607+0000 7f14957fa640 4 set_mon_vals no callback set 2024-09-11T01:29:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.608+0000 7f149d2c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:23.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.608+0000 7f147f7fe640 1 -- 172.21.15.12:0/2733588502 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f1488024060 con 0x7f14980a5d00 2024-09-11T01:29:23.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.608+0000 7f149d2c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:23.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.608+0000 7f149d2c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:23.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.608+0000 7f149d2c6640 1 -- 172.21.15.12:0/2733588502 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1498138d30 con 0x7f14980a5d00 2024-09-11T01:29:23.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.608+0000 7f147f7fe640 1 --2- 172.21.15.12:0/2733588502 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f148403e240 0x7f1484040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:23.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.609+0000 7f147f7fe640 1 -- 172.21.15.12:0/2733588502 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4374+0+0 (secure 0 0 0) 0x7f1488052c90 con 0x7f14980a5d00 2024-09-11T01:29:23.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.609+0000 7f14967fc640 1 --2- 172.21.15.12:0/2733588502 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f148403e240 0x7f1484040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:23.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.609+0000 7f14967fc640 10 monclient: get_auth_request con 0x7f148403e240 auth_method 0 2024-09-11T01:29:23.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.611+0000 7f14967fc640 1 --2- 172.21.15.12:0/2733588502 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f148403e240 0x7f1484040700 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f148c034000 tx=0x7f148c02e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:23.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.613+0000 7f147f7fe640 1 -- 172.21.15.12:0/2733588502 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1488059080 con 0x7f14980a5d00 2024-09-11T01:29:23.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.613+0000 7f147f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:23.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.613+0000 7f147f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:23.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.797+0000 7f149d2c6640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-11T01:29:23.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.797+0000 7f149d2c6640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-11T01:29:23.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.797+0000 7f149d2c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:23.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.797+0000 7f149d2c6640 1 -- 172.21.15.12:0/2733588502 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7f14980a6580 con 0x7f14980a5d00 2024-09-11T01:29:23.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.798+0000 7f147f7fe640 1 -- 172.21.15.12:0/2733588502 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v17) ==== 92+0+10 (secure 0 0 0) 0x7f1488051820 con 0x7f14980a5d00 2024-09-11T01:29:23.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.798+0000 7f147f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-11T01:29:23.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.798+0000 7f147f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:23.800 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 8 2024-09-11T01:29:23.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.800+0000 7f149d2c6640 1 -- 172.21.15.12:0/2733588502 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f148403e240 msgr2=0x7f1484040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:23.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.800+0000 7f149d2c6640 1 --2- 172.21.15.12:0/2733588502 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f148403e240 0x7f1484040700 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f148c034000 tx=0x7f148c02e000 comp rx=0 tx=0).stop 2024-09-11T01:29:23.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.800+0000 7f149d2c6640 10 monclient: shutdown 2024-09-11T01:29:23.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.800+0000 7f149d2c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:23.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.800+0000 7f149d2c6640 1 -- 172.21.15.12:0/2733588502 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14980a5d00 msgr2=0x7f14981cb2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:23.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.800+0000 7f149d2c6640 1 --2- 172.21.15.12:0/2733588502 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14980a5d00 0x7f14981cb2a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f148800e9e0 tx=0x7f148800eea0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.800+0000 7f149d2c6640 1 -- 172.21.15.12:0/2733588502 shutdown_connections 2024-09-11T01:29:23.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.800+0000 7f149d2c6640 1 --2- 172.21.15.12:0/2733588502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1498138470 0x7f1498137290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.800+0000 7f149d2c6640 1 --2- 172.21.15.12:0/2733588502 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14980a60e0 0x7f1498136d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.800+0000 7f149d2c6640 1 --2- 172.21.15.12:0/2733588502 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f148403e240 0x7f1484040700 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.800+0000 7f149d2c6640 1 --2- 172.21.15.12:0/2733588502 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14980a5d00 0x7f14981cb2a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.800+0000 7f149d2c6640 1 -- 172.21.15.12:0/2733588502 >> 172.21.15.12:0/2733588502 conn(0x7f149806ee50 msgr2=0x7f14980a4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:23.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.800+0000 7f149d2c6640 1 -- 172.21.15.12:0/2733588502 shutdown_connections 2024-09-11T01:29:23.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.800+0000 7f149d2c6640 1 -- 172.21.15.12:0/2733588502 wait complete. 2024-09-11T01:29:23.812 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-11T01:29:23.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.914+0000 7fcfaff7d640 10 monclient: get_monmap_and_config 2024-09-11T01:29:23.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.914+0000 7fcfaff7d640 10 monclient: build_initial_monmap 2024-09-11T01:29:23.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.914+0000 7fcfaff7d640 10 monclient: monmap: 2024-09-11T01:29:23.916 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:23.916 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:23.916 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:23.915227+0000 2024-09-11T01:29:23.916 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:23.915227+0000 2024-09-11T01:29:23.916 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:23.916 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:23.917 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:23.917 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:23.917 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:23.917 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:23.917 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:23.917 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:23.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfaff7d640 1 Processor -- start 2024-09-11T01:29:23.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfaff7d640 1 -- start start 2024-09-11T01:29:23.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfaff7d640 10 monclient: init 2024-09-11T01:29:23.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfaff7d640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:23.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfaff7d640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-11T01:29:23.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfaff7d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcfa80a5020 0x7fcfa809d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:23.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfaff7d640 10 monclient(hunting): picked mon.noname-b con 0x7fcfa80a5020 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:23.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfaff7d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcfa80a5400 0x7fcfa80a57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:23.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfaff7d640 10 monclient(hunting): picked mon.noname-a con 0x7fcfa80a5400 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:23.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfaff7d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcfa8137de0 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:23.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfaff7d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfa80a6e60 con 0x7fcfa8137de0 2024-09-11T01:29:23.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfaff7d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:23.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfaff7d640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fcfa809ba20 con 0x7fcfa80a5400 2024-09-11T01:29:23.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfaff7d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:23.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfaff7d640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcfa80a66d0 con 0x7fcfa80a5020 2024-09-11T01:29:23.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfaff7d640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:23.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfaff7d640 10 monclient(hunting): authenticate will time out at 770.357800s 2024-09-11T01:29:23.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfad518640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcfa80a5400 0x7fcfa80a57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:23.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfadd19640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcfa80a5020 0x7fcfa809d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:23.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfad518640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcfa80a5400 0x7fcfa80a57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55160/0 (socket says 172.21.15.12:55160) 2024-09-11T01:29:23.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfadd19640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcfa80a5020 0x7fcfa809d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51892/0 (socket says 172.21.15.12:51892) 2024-09-11T01:29:23.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfad518640 1 -- 172.21.15.12:0/4270248891 learned_addr learned my addr 172.21.15.12:0/4270248891 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:23.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfad518640 10 monclient(hunting): get_auth_request con 0x7fcfa80a5400 auth_method 0 2024-09-11T01:29:23.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfad518640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:23.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfad518640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:23.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfad518640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:23.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfadd19640 10 monclient(hunting): get_auth_request con 0x7fcfa80a5020 auth_method 0 2024-09-11T01:29:23.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfadd19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:23.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfadd19640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:23.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.915+0000 7fcfadd19640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:23.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.916+0000 7fcfadd19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:23.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.916+0000 7fcfadd19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:23.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.916+0000 7fcfadd19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:23.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.916+0000 7fcfad518640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:23.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.916+0000 7fcfad518640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:23.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.916+0000 7fcfad518640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:23.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.916+0000 7fcfad518640 10 monclient(hunting): handle_auth_done global_id 4335 payload 274 2024-09-11T01:29:23.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.916+0000 7fcfad518640 1 -- 172.21.15.12:0/4270248891 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcfa80a5020 msgr2=0x7fcfa809d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:23.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.916+0000 7fcfad518640 1 --2- 172.21.15.12:0/4270248891 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcfa80a5020 0x7fcfa809d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.916+0000 7fcfad518640 1 -- 172.21.15.12:0/4270248891 >> v1:172.21.15.71:6789/0 conn(0x7fcfa8137de0 legacy=0x7fcfa8138250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:23.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.916+0000 7fcfad518640 10 monclient: _finish_hunting 0 2024-09-11T01:29:23.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.916+0000 7fcfad518640 1 monclient: found mon.noname-a 2024-09-11T01:29:23.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.916+0000 7fcfad518640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:23.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.916+0000 7fcfad518640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:23.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.916+0000 7fcfad518640 1 -- 172.21.15.12:0/4270248891 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa809c370 con 0x7fcfa80a5400 2024-09-11T01:29:23.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.916+0000 7fcfadd19640 1 --2- 172.21.15.12:0/4270248891 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcfa80a5020 0x7fcfa809d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:23.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfad518640 1 -- 172.21.15.12:0/4270248891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfa80a5400 msgr2=0x7fcfa80a57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:23.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfad518640 1 --2- 172.21.15.12:0/4270248891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfa80a5400 0x7fcfa80a57e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c009870 tx=0x7fcf9c02a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7904736a4292881 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:23.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfacd17640 1 -- 172.21.15.12:0/4270248891 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c037070 con 0x7fcfa80a5400 2024-09-11T01:29:23.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfacd17640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:23.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfacd17640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:23.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfacd17640 10 monclient: dump: 2024-09-11T01:29:23.928 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:23.928 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:23.928 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:23.928 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:23.928 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:23.928 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:23.928 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:23.929 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:23.929 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:23.929 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:23.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfacd17640 10 monclient: _finish_auth 0 2024-09-11T01:29:23.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfacd17640 10 monclient: _check_auth_tickets 2024-09-11T01:29:23.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfacd17640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:23.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfacd17640 1 -- 172.21.15.12:0/4270248891 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf9c032440 con 0x7fcfa80a5400 2024-09-11T01:29:23.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfaff7d640 5 monclient: authenticate success, global_id 4335 2024-09-11T01:29:23.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfaff7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:23.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfacd17640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:23.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfacd17640 1 -- 172.21.15.12:0/4270248891 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c032810 con 0x7fcfa80a5400 2024-09-11T01:29:23.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfaff7d640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:23.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfaff7d640 4 set_mon_vals no callback set 2024-09-11T01:29:23.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfacd17640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:23.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfacd17640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:23.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfacd17640 10 monclient: dump: 2024-09-11T01:29:23.930 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:23.930 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:23.930 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:23.930 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:23.930 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:23.930 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:23.930 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:23.930 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:23.930 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:23.930 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:23.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfaff7d640 10 monclient: shutdown 2024-09-11T01:29:23.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfaff7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:23.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfaff7d640 1 -- 172.21.15.12:0/4270248891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfa80a5400 msgr2=0x7fcfa80a57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:23.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfaff7d640 1 --2- 172.21.15.12:0/4270248891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfa80a5400 0x7fcfa80a57e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c009870 tx=0x7fcf9c02a0a0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfaff7d640 1 -- 172.21.15.12:0/4270248891 shutdown_connections 2024-09-11T01:29:23.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfaff7d640 1 --2- 172.21.15.12:0/4270248891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfa80a5400 0x7fcfa80a57e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfaff7d640 1 --2- 172.21.15.12:0/4270248891 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcfa80a5020 0x7fcfa809d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfaff7d640 1 -- 172.21.15.12:0/4270248891 >> 172.21.15.12:0/4270248891 conn(0x7fcfa80a7300 msgr2=0x7fcfa80a7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:23.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfaff7d640 1 -- 172.21.15.12:0/4270248891 shutdown_connections 2024-09-11T01:29:23.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.919+0000 7fcfaff7d640 1 -- 172.21.15.12:0/4270248891 wait complete. 2024-09-11T01:29:23.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.920+0000 7fcfaff7d640 10 monclient: build_initial_monmap 2024-09-11T01:29:23.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.920+0000 7fcfaff7d640 10 monclient: monmap: 2024-09-11T01:29:23.931 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:23.931 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:23.931 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:23.932 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:23.932 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:23.932 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:23.932 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:23.932 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:23.932 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:23.932 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:23.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.920+0000 7fcfaff7d640 1 Processor -- start 2024-09-11T01:29:23.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.920+0000 7fcfaff7d640 1 -- start start 2024-09-11T01:29:23.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.920+0000 7fcfaff7d640 10 monclient: init 2024-09-11T01:29:23.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfaff7d640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:23.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfaff7d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:29:23.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfaff7d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfa80a5020 0x7fcfa81cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:23.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfaff7d640 10 monclient(hunting): picked mon.noname-a con 0x7fcfa80a5020 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:23.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfaff7d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcfa80a5400 0x7fcfa81249d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:23.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfaff7d640 10 monclient(hunting): picked mon.noname-c con 0x7fcfa80a5400 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:23.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfaff7d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcfa8137de0 0x7fcfa8124f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:23.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfaff7d640 10 monclient(hunting): picked mon.noname-b con 0x7fcfa8137de0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:23.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfaff7d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:23.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfaff7d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa806f7f0 con 0x7fcfa80a5020 2024-09-11T01:29:23.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfaff7d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:23.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfaff7d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa806f970 con 0x7fcfa8137de0 2024-09-11T01:29:23.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfaff7d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:23.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfaff7d640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcfa8138960 con 0x7fcfa80a5400 2024-09-11T01:29:23.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfaff7d640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:23.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfaff7d640 10 monclient(hunting): authenticate will time out at 770.363435s 2024-09-11T01:29:23.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfadd19640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfa80a5020 0x7fcfa81cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:23.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfad518640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcfa80a5400 0x7fcfa81249d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:23.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfadd19640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfa80a5020 0x7fcfa81cd6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55164/0 (socket says 172.21.15.12:55164) 2024-09-11T01:29:23.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfad518640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcfa80a5400 0x7fcfa81249d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51906/0 (socket says 172.21.15.12:51906) 2024-09-11T01:29:23.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfadd19640 1 -- 172.21.15.12:0/2568763087 learned_addr learned my addr 172.21.15.12:0/2568763087 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:23.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfadd19640 10 monclient(hunting): get_auth_request con 0x7fcfa80a5020 auth_method 0 2024-09-11T01:29:23.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfadd19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:23.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfadd19640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:23.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfadd19640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:23.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfad518640 10 monclient(hunting): get_auth_request con 0x7fcfa80a5400 auth_method 0 2024-09-11T01:29:23.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfad518640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:23.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfad518640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:23.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfad518640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:23.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfad518640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:23.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfad518640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:23.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfad518640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:23.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfadd19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:23.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfadd19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:23.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfadd19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:23.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfae51a640 1 --2- 172.21.15.12:0/2568763087 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcfa8137de0 0x7fcfa8124f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:23.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfae51a640 10 monclient(hunting): get_auth_request con 0x7fcfa8137de0 auth_method 0 2024-09-11T01:29:23.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfae51a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:23.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfae51a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:23.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfae51a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:23.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfad518640 10 monclient(hunting): handle_auth_done global_id 4364 payload 824 2024-09-11T01:29:23.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfad518640 1 -- 172.21.15.12:0/2568763087 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcfa8137de0 msgr2=0x7fcfa8124f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:23.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfad518640 1 --2- 172.21.15.12:0/2568763087 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcfa8137de0 0x7fcfa8124f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfad518640 1 -- 172.21.15.12:0/2568763087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfa80a5020 msgr2=0x7fcfa81cd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:23.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfad518640 1 --2- 172.21.15.12:0/2568763087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfa80a5020 0x7fcfa81cd6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:23.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfad518640 10 monclient: _finish_hunting 0 2024-09-11T01:29:23.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfad518640 1 monclient: found mon.noname-c 2024-09-11T01:29:23.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfad518640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:23.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfad518640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:29:23.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfad518640 1 -- 172.21.15.12:0/2568763087 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa8128750 con 0x7fcfa80a5400 2024-09-11T01:29:23.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.921+0000 7fcfadd19640 1 --2- 172.21.15.12:0/2568763087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfa80a5020 0x7fcfa81cd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:23.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.924+0000 7fcfad518640 1 -- 172.21.15.12:0/2568763087 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcfa80a5400 msgr2=0x7fcfa81249d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:23.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.924+0000 7fcfad518640 1 --2- 172.21.15.12:0/2568763087 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcfa80a5400 0x7fcfa81249d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c031000 tx=0x7fcf9c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:23.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.924+0000 7fcfa67fc640 1 -- 172.21.15.12:0/2568763087 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c041020 con 0x7fcfa80a5400 2024-09-11T01:29:23.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.924+0000 7fcfa67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:23.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.924+0000 7fcfa67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:23.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.924+0000 7fcfa67fc640 10 monclient: dump: 2024-09-11T01:29:23.937 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:23.937 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:23.937 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:23.937 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:23.937 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:23.937 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:23.937 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:23.937 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:23.937 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:23.937 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:23.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.924+0000 7fcfa67fc640 10 monclient: _finish_auth 0 2024-09-11T01:29:23.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.924+0000 7fcfa67fc640 10 monclient: _check_auth_tickets 2024-09-11T01:29:23.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.924+0000 7fcfa67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:23.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.924+0000 7fcfa67fc640 1 -- 172.21.15.12:0/2568763087 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf9c03d630 con 0x7fcfa80a5400 2024-09-11T01:29:23.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.924+0000 7fcfaff7d640 5 monclient: authenticate success, global_id 4364 2024-09-11T01:29:23.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.924+0000 7fcfa67fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:23.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.924+0000 7fcfa67fc640 1 -- 172.21.15.12:0/2568763087 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c037040 con 0x7fcfa80a5400 2024-09-11T01:29:23.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.924+0000 7fcfaff7d640 10 monclient: _renew_subs 2024-09-11T01:29:23.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.924+0000 7fcfaff7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:23.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.924+0000 7fcfaff7d640 1 -- 172.21.15.12:0/2568763087 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfa80a1810 con 0x7fcfa80a5400 2024-09-11T01:29:23.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.924+0000 7fcfa67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:23.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.924+0000 7fcfa67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:23.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.924+0000 7fcfa67fc640 10 monclient: dump: 2024-09-11T01:29:23.939 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:23.939 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:23.939 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:23.939 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:23.939 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:23.939 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:23.939 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:23.939 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:23.939 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:23.939 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:23.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.924+0000 7fcfa77fe640 4 set_mon_vals no callback set 2024-09-11T01:29:23.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.924+0000 7fcfaff7d640 10 monclient: _renew_subs 2024-09-11T01:29:23.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.924+0000 7fcfaff7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:23.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.924+0000 7fcfaff7d640 1 -- 172.21.15.12:0/2568763087 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfa8129130 con 0x7fcfa80a5400 2024-09-11T01:29:23.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.925+0000 7fcfaff7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:23.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.925+0000 7fcfaff7d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:23.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.925+0000 7fcfaff7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:23.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.925+0000 7fcfaff7d640 1 -- 172.21.15.12:0/2568763087 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfa809d110 con 0x7fcfa80a5400 2024-09-11T01:29:23.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.925+0000 7fcfa67fc640 1 -- 172.21.15.12:0/2568763087 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fcf9c03a030 con 0x7fcfa80a5400 2024-09-11T01:29:23.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.925+0000 7fcfa67fc640 1 --2- 172.21.15.12:0/2568763087 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcf9403e240 0x7fcf94040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:23.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.925+0000 7fcfa67fc640 1 -- 172.21.15.12:0/2568763087 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 4374+0+0 (secure 0 0 0) 0x7fcf9c072b30 con 0x7fcfa80a5400 2024-09-11T01:29:23.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.926+0000 7fcfadd19640 1 --2- 172.21.15.12:0/2568763087 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcf9403e240 0x7fcf94040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:23.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.926+0000 7fcfadd19640 10 monclient: get_auth_request con 0x7fcf9403e240 auth_method 0 2024-09-11T01:29:23.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.928+0000 7fcfadd19640 1 --2- 172.21.15.12:0/2568763087 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcf9403e240 0x7fcf94040700 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcf98009bd0 tx=0x7fcf9800b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:23.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.930+0000 7fcfa67fc640 1 -- 172.21.15.12:0/2568763087 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fcf9c04b070 con 0x7fcfa80a5400 2024-09-11T01:29:23.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.930+0000 7fcfa67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:23.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:23.930+0000 7fcfa67fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:24.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.111+0000 7fcfaff7d640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-11T01:29:24.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.111+0000 7fcfaff7d640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-11T01:29:24.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.111+0000 7fcfaff7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:24.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.111+0000 7fcfaff7d640 1 -- 172.21.15.12:0/2568763087 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcfa80698e0 con 0x7fcfa80a5400 2024-09-11T01:29:24.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.112+0000 7fcfa67fc640 1 -- 172.21.15.12:0/2568763087 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7fcf9c03b3f0 con 0x7fcfa80a5400 2024-09-11T01:29:24.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.112+0000 7fcfa67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-11T01:29:24.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.112+0000 7fcfa67fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:24.114 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-09-11T01:29:24.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.114+0000 7fcfaff7d640 1 -- 172.21.15.12:0/2568763087 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcf9403e240 msgr2=0x7fcf94040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:24.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.114+0000 7fcfaff7d640 1 --2- 172.21.15.12:0/2568763087 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcf9403e240 0x7fcf94040700 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcf98009bd0 tx=0x7fcf9800b040 comp rx=0 tx=0).stop 2024-09-11T01:29:24.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.114+0000 7fcfaff7d640 10 monclient: shutdown 2024-09-11T01:29:24.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.114+0000 7fcfaff7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:24.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.114+0000 7fcfaff7d640 1 -- 172.21.15.12:0/2568763087 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcfa80a5400 msgr2=0x7fcfa81249d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:24.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.114+0000 7fcfaff7d640 1 --2- 172.21.15.12:0/2568763087 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcfa80a5400 0x7fcfa81249d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c031000 tx=0x7fcf9c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.114+0000 7fcfaff7d640 1 -- 172.21.15.12:0/2568763087 shutdown_connections 2024-09-11T01:29:24.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.114+0000 7fcfaff7d640 1 --2- 172.21.15.12:0/2568763087 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcfa8137de0 0x7fcfa8124f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.114+0000 7fcfaff7d640 1 --2- 172.21.15.12:0/2568763087 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcfa80a5400 0x7fcfa81249d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.114+0000 7fcfaff7d640 1 --2- 172.21.15.12:0/2568763087 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcf9403e240 0x7fcf94040700 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.114+0000 7fcfaff7d640 1 --2- 172.21.15.12:0/2568763087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcfa80a5020 0x7fcfa81cd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.114+0000 7fcfaff7d640 1 -- 172.21.15.12:0/2568763087 >> 172.21.15.12:0/2568763087 conn(0x7fcfa80a7300 msgr2=0x7fcfa80a8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:24.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.114+0000 7fcfaff7d640 1 -- 172.21.15.12:0/2568763087 shutdown_connections 2024-09-11T01:29:24.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.114+0000 7fcfaff7d640 1 -- 172.21.15.12:0/2568763087 wait complete. 2024-09-11T01:29:24.126 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-11T01:29:24.126 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T01:29:24.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.225+0000 7fb23d717640 10 monclient: get_monmap_and_config 2024-09-11T01:29:24.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.225+0000 7fb23d717640 10 monclient: build_initial_monmap 2024-09-11T01:29:24.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.225+0000 7fb23d717640 10 monclient: monmap: 2024-09-11T01:29:24.227 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:24.227 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:24.227 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:24.225984+0000 2024-09-11T01:29:24.227 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:24.225984+0000 2024-09-11T01:29:24.227 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:24.227 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:24.227 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:24.227 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:24.227 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:24.228 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:24.228 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:24.228 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:24.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.225+0000 7fb23d717640 1 Processor -- start 2024-09-11T01:29:24.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.225+0000 7fb23d717640 1 -- start start 2024-09-11T01:29:24.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.225+0000 7fb23d717640 10 monclient: init 2024-09-11T01:29:24.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb23d717640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:24.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb23d717640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-11T01:29:24.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb23d717640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb238137d50 0x7fb23809d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb23d717640 10 monclient(hunting): picked mon.noname-b con 0x7fb238137d50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb23d717640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb238138130 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb23d717640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb23813d280 0x7fb23813f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb23d717640 10 monclient(hunting): picked mon.noname-a con 0x7fb23813d280 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb23d717640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb23809ba20 con 0x7fb238138130 2024-09-11T01:29:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb23d717640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb23d717640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fb23809c650 con 0x7fb23813d280 2024-09-11T01:29:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb23d717640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb23d717640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb23809c370 con 0x7fb238137d50 2024-09-11T01:29:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb23d717640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb23d717640 10 monclient(hunting): authenticate will time out at 770.668527s 2024-09-11T01:29:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb236ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb238137d50 0x7fb23809d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb2377fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb23813d280 0x7fb23813f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb236ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb238137d50 0x7fb23809d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51916/0 (socket says 172.21.15.12:51916) 2024-09-11T01:29:24.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb236ffd640 1 -- 172.21.15.12:0/3049945941 learned_addr learned my addr 172.21.15.12:0/3049945941 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:24.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb236ffd640 10 monclient(hunting): get_auth_request con 0x7fb238137d50 auth_method 0 2024-09-11T01:29:24.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb236ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:24.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb236ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:24.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb236ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:24.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb2367fc640 1 --1- 172.21.15.12:0/3049945941 >> v1:172.21.15.12:6789/0 conn(0x7fb238138130 0x7fb238138550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56140/0 (socket says 172.21.15.12:56140) 2024-09-11T01:29:24.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb2377fe640 10 monclient(hunting): get_auth_request con 0x7fb23813d280 auth_method 0 2024-09-11T01:29:24.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb2377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:24.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb2377fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:24.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb2377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:24.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb236ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:24.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb236ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:24.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb236ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:24.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb2377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:24.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb2377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:24.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb2377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:24.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb236ffd640 10 monclient(hunting): handle_auth_done global_id 4367 payload 274 2024-09-11T01:29:24.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb236ffd640 1 -- 172.21.15.12:0/3049945941 >> v2:172.21.15.12:3300/0 conn(0x7fb23813d280 msgr2=0x7fb23813f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:24.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb236ffd640 1 --2- 172.21.15.12:0/3049945941 >> v2:172.21.15.12:3300/0 conn(0x7fb23813d280 0x7fb23813f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb236ffd640 1 -- 172.21.15.12:0/3049945941 >> v1:172.21.15.12:6789/0 conn(0x7fb238138130 legacy=0x7fb238138550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:24.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb236ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:29:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb236ffd640 1 monclient: found mon.noname-b 2024-09-11T01:29:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb236ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb236ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:29:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.226+0000 7fb236ffd640 1 -- 172.21.15.12:0/3049945941 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2380a6ce0 con 0x7fb238137d50 2024-09-11T01:29:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.229+0000 7fb236ffd640 1 -- 172.21.15.12:0/3049945941 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb238137d50 msgr2=0x7fb23809d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.229+0000 7fb236ffd640 1 --2- 172.21.15.12:0/3049945941 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb238137d50 0x7fb23809d110 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb23809c7d0 tx=0x7fb22802a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c224105192c9c3b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.229+0000 7fb235ffb640 1 -- 172.21.15.12:0/3049945941 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb228037070 con 0x7fb238137d50 2024-09-11T01:29:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.229+0000 7fb235ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.229+0000 7fb235ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.229+0000 7fb235ffb640 10 monclient: dump: 2024-09-11T01:29:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:24.241 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:24.241 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.241 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.241 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:24.241 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:24.241 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:24.241 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:24.241 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:24.241 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:24.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.229+0000 7fb235ffb640 10 monclient: _finish_auth 0 2024-09-11T01:29:24.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.229+0000 7fb235ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:29:24.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.229+0000 7fb235ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:24.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.229+0000 7fb235ffb640 1 -- 172.21.15.12:0/3049945941 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb228032640 con 0x7fb238137d50 2024-09-11T01:29:24.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.229+0000 7fb23d717640 5 monclient: authenticate success, global_id 4367 2024-09-11T01:29:24.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.229+0000 7fb23d717640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:24.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.229+0000 7fb235ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:24.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.230+0000 7fb23d717640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:24.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.230+0000 7fb23d717640 4 set_mon_vals no callback set 2024-09-11T01:29:24.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.230+0000 7fb235ffb640 1 -- 172.21.15.12:0/3049945941 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb228032a10 con 0x7fb238137d50 2024-09-11T01:29:24.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.230+0000 7fb235ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:24.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.230+0000 7fb235ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:24.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.230+0000 7fb235ffb640 10 monclient: dump: 2024-09-11T01:29:24.242 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:24.242 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:24.242 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.242 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.243 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:24.243 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:24.243 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:24.243 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:24.243 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:24.243 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:24.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.230+0000 7fb23d717640 10 monclient: shutdown 2024-09-11T01:29:24.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.230+0000 7fb23d717640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:24.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.230+0000 7fb23d717640 1 -- 172.21.15.12:0/3049945941 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb238137d50 msgr2=0x7fb23809d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:24.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.230+0000 7fb23d717640 1 --2- 172.21.15.12:0/3049945941 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb238137d50 0x7fb23809d110 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb23809c7d0 tx=0x7fb22802a110 comp rx=0 tx=0).stop 2024-09-11T01:29:24.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.230+0000 7fb23d717640 1 -- 172.21.15.12:0/3049945941 shutdown_connections 2024-09-11T01:29:24.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.230+0000 7fb23d717640 1 --2- 172.21.15.12:0/3049945941 >> v2:172.21.15.12:3300/0 conn(0x7fb23813d280 0x7fb23813f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.230+0000 7fb23d717640 1 --2- 172.21.15.12:0/3049945941 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb238137d50 0x7fb23809d110 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.230+0000 7fb23d717640 1 -- 172.21.15.12:0/3049945941 >> 172.21.15.12:0/3049945941 conn(0x7fb2380a7020 msgr2=0x7fb2380a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.230+0000 7fb23d717640 1 -- 172.21.15.12:0/3049945941 shutdown_connections 2024-09-11T01:29:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.230+0000 7fb23d717640 1 -- 172.21.15.12:0/3049945941 wait complete. 2024-09-11T01:29:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.231+0000 7fb23d717640 10 monclient: build_initial_monmap 2024-09-11T01:29:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.231+0000 7fb23d717640 10 monclient: monmap: 2024-09-11T01:29:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:24.245 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:24.245 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:24.245 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:24.245 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:24.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.231+0000 7fb23d717640 1 Processor -- start 2024-09-11T01:29:24.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.231+0000 7fb23d717640 1 -- start start 2024-09-11T01:29:24.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.231+0000 7fb23d717640 10 monclient: init 2024-09-11T01:29:24.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb23d717640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:24.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb23d717640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:29:24.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb23d717640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb238137d50 0x7fb2381cce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:24.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb23d717640 10 monclient(hunting): picked mon.noname-a con 0x7fb238137d50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:24.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb23d717640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb238138130 0x7fb238136f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:24.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb23d717640 10 monclient(hunting): picked mon.noname-c con 0x7fb238138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:24.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb23d717640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb23813d280 0x7fb2380a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:24.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb23d717640 10 monclient(hunting): picked mon.noname-b con 0x7fb23813d280 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:24.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb23d717640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:24.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb23d717640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb23809d7d0 con 0x7fb238137d50 2024-09-11T01:29:24.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb23d717640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:24.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb23d717640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb23809d650 con 0x7fb23813d280 2024-09-11T01:29:24.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb23d717640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:24.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb23d717640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb23813fbb0 con 0x7fb238138130 2024-09-11T01:29:24.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb23d717640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:24.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb23d717640 10 monclient(hunting): authenticate will time out at 770.674222s 2024-09-11T01:29:24.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb236ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb238137d50 0x7fb2381cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:24.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb236ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb238137d50 0x7fb2381cce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55172/0 (socket says 172.21.15.12:55172) 2024-09-11T01:29:24.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb236ffd640 1 -- 172.21.15.12:0/3532024657 learned_addr learned my addr 172.21.15.12:0/3532024657 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:24.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb236ffd640 10 monclient(hunting): get_auth_request con 0x7fb238137d50 auth_method 0 2024-09-11T01:29:24.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb2367fc640 1 --2- 172.21.15.12:0/3532024657 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb238138130 0x7fb238136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:24.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb236ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:24.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb236ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:24.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb236ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:24.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb2367fc640 10 monclient(hunting): get_auth_request con 0x7fb238138130 auth_method 0 2024-09-11T01:29:24.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb2367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:24.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb2367fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:24.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb2367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:24.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb236ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:24.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb236ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:24.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb236ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:24.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb2367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:24.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb2367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:24.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb2367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:24.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb2377fe640 1 --2- 172.21.15.12:0/3532024657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb23813d280 0x7fb2380a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:24.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb2367fc640 10 monclient(hunting): handle_auth_done global_id 4370 payload 824 2024-09-11T01:29:24.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb2367fc640 1 -- 172.21.15.12:0/3532024657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb23813d280 msgr2=0x7fb2380a19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:24.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb2367fc640 1 --2- 172.21.15.12:0/3532024657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb23813d280 0x7fb2380a19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb2367fc640 1 -- 172.21.15.12:0/3532024657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb238137d50 msgr2=0x7fb2381cce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:24.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb2367fc640 1 --2- 172.21.15.12:0/3532024657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb238137d50 0x7fb2381cce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb2367fc640 10 monclient: _finish_hunting 0 2024-09-11T01:29:24.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb2367fc640 1 monclient: found mon.noname-c 2024-09-11T01:29:24.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb2367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:24.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb2367fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:29:24.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb2367fc640 1 -- 172.21.15.12:0/3532024657 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2380a34a0 con 0x7fb238138130 2024-09-11T01:29:24.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb2377fe640 10 monclient: get_auth_request con 0x7fb23813d280 auth_method 0 2024-09-11T01:29:24.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb2377fe640 1 --2- 172.21.15.12:0/3532024657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb23813d280 0x7fb2380a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:29:24.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.232+0000 7fb236ffd640 1 --2- 172.21.15.12:0/3532024657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb238137d50 0x7fb2381cce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:24.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.234+0000 7fb2367fc640 1 -- 172.21.15.12:0/3532024657 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb238138130 msgr2=0x7fb238136f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:24.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.234+0000 7fb2367fc640 1 --2- 172.21.15.12:0/3532024657 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb238138130 0x7fb238136f40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb22000c3c0 tx=0x7fb22000c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:24.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.235+0000 7fb22f7fe640 1 -- 172.21.15.12:0/3532024657 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb220017070 con 0x7fb238138130 2024-09-11T01:29:24.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.235+0000 7fb22f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:24.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.235+0000 7fb22f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:24.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.235+0000 7fb22f7fe640 10 monclient: dump: 2024-09-11T01:29:24.249 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:24.249 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:24.250 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.250 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.250 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:24.250 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:24.250 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:24.250 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:24.250 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:24.250 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:24.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.235+0000 7fb22f7fe640 10 monclient: _finish_auth 0 2024-09-11T01:29:24.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.235+0000 7fb22f7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:29:24.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.235+0000 7fb22f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:24.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.235+0000 7fb22f7fe640 1 -- 172.21.15.12:0/3532024657 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb220005810 con 0x7fb238138130 2024-09-11T01:29:24.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.235+0000 7fb23d717640 5 monclient: authenticate success, global_id 4370 2024-09-11T01:29:24.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.235+0000 7fb23d717640 10 monclient: _renew_subs 2024-09-11T01:29:24.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.235+0000 7fb23d717640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:24.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.235+0000 7fb23d717640 1 -- 172.21.15.12:0/3532024657 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2381218d0 con 0x7fb238138130 2024-09-11T01:29:24.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.235+0000 7fb22f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:24.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.235+0000 7fb22f7fe640 1 -- 172.21.15.12:0/3532024657 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb220005c00 con 0x7fb238138130 2024-09-11T01:29:24.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.235+0000 7fb22f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:24.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.235+0000 7fb22f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:24.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.235+0000 7fb22f7fe640 10 monclient: dump: 2024-09-11T01:29:24.251 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:24.251 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:24.251 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.252 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.252 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:24.252 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:24.252 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:24.252 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:24.252 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:24.252 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:24.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.235+0000 7fb23d717640 10 monclient: _renew_subs 2024-09-11T01:29:24.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.235+0000 7fb2357fa640 4 set_mon_vals no callback set 2024-09-11T01:29:24.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.235+0000 7fb23d717640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:24.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.235+0000 7fb23d717640 1 -- 172.21.15.12:0/3532024657 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2381cdd20 con 0x7fb238138130 2024-09-11T01:29:24.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.236+0000 7fb23d717640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:24.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.236+0000 7fb22f7fe640 1 -- 172.21.15.12:0/3532024657 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fb22001b3b0 con 0x7fb238138130 2024-09-11T01:29:24.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.236+0000 7fb23d717640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:24.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.236+0000 7fb23d717640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:24.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.236+0000 7fb23d717640 1 -- 172.21.15.12:0/3532024657 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb23809d110 con 0x7fb238138130 2024-09-11T01:29:24.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.236+0000 7fb22f7fe640 1 --2- 172.21.15.12:0/3532024657 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb21803e210 0x7fb2180406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:24.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.236+0000 7fb22f7fe640 1 -- 172.21.15.12:0/3532024657 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 4374+0+0 (secure 0 0 0) 0x7fb220052310 con 0x7fb238138130 2024-09-11T01:29:24.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.236+0000 7fb236ffd640 1 --2- 172.21.15.12:0/3532024657 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb21803e210 0x7fb2180406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:24.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.236+0000 7fb236ffd640 10 monclient: get_auth_request con 0x7fb21803e210 auth_method 0 2024-09-11T01:29:24.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.239+0000 7fb236ffd640 1 --2- 172.21.15.12:0/3532024657 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb21803e210 0x7fb2180406d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb22802a150 tx=0x7fb22802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:24.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.240+0000 7fb22f7fe640 1 -- 172.21.15.12:0/3532024657 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb220050ea0 con 0x7fb238138130 2024-09-11T01:29:24.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.240+0000 7fb22f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:24.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.240+0000 7fb22f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:24.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.421+0000 7fb23d717640 1 -- 172.21.15.12:0/3532024657 --> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb23813f610 con 0x7fb21803e210 2024-09-11T01:29:24.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.423+0000 7fb22f7fe640 1 -- 172.21.15.12:0/3532024657 <== mgr.4101 v2:172.21.15.71:6816/3189259519 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34179 (secure 0 0 0) 0x7fb23813f610 con 0x7fb21803e210 2024-09-11T01:29:24.425 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:29:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-09-11T01:29:24.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.425+0000 7fb23d717640 1 -- 172.21.15.12:0/3532024657 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb21803e210 msgr2=0x7fb2180406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:24.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.425+0000 7fb23d717640 1 --2- 172.21.15.12:0/3532024657 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb21803e210 0x7fb2180406d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb22802a150 tx=0x7fb22802e000 comp rx=0 tx=0).stop 2024-09-11T01:29:24.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.425+0000 7fb23d717640 10 monclient: shutdown 2024-09-11T01:29:24.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.425+0000 7fb23d717640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:24.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.425+0000 7fb23d717640 1 -- 172.21.15.12:0/3532024657 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb238138130 msgr2=0x7fb238136f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:24.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.425+0000 7fb23d717640 1 --2- 172.21.15.12:0/3532024657 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb238138130 0x7fb238136f40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb22000c3c0 tx=0x7fb22000c880 comp rx=0 tx=0).stop 2024-09-11T01:29:24.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.425+0000 7fb23d717640 1 -- 172.21.15.12:0/3532024657 shutdown_connections 2024-09-11T01:29:24.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.425+0000 7fb23d717640 1 --2- 172.21.15.12:0/3532024657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb23813d280 0x7fb2380a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.425+0000 7fb23d717640 1 --2- 172.21.15.12:0/3532024657 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb238138130 0x7fb238136f40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.425+0000 7fb23d717640 1 --2- 172.21.15.12:0/3532024657 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb21803e210 0x7fb2180406d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.425+0000 7fb23d717640 1 --2- 172.21.15.12:0/3532024657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb238137d50 0x7fb2381cce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.425+0000 7fb23d717640 1 -- 172.21.15.12:0/3532024657 >> 172.21.15.12:0/3532024657 conn(0x7fb2380a7020 msgr2=0x7fb23813e7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:24.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.425+0000 7fb23d717640 1 -- 172.21.15.12:0/3532024657 shutdown_connections 2024-09-11T01:29:24.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.425+0000 7fb23d717640 1 -- 172.21.15.12:0/3532024657 wait complete. 2024-09-11T01:29:24.437 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-11T01:29:22.852280+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163236,"kb_used_data":1016,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067804,"statfs":{"total":579820584960,"available":579653431296,"internally_reserved":0,"allocated":1040384,"data_stored":1336972,"data_compressed":59136,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000439"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.597492+0000","last_change":"2024-09-11T01:29:17.597492+0000","last_active":"2024-09-11T01:29:17.597492+0000","last_peered":"2024-09-11T01:29:17.597492+0000","last_clean":"2024-09-11T01:29:17.597492+0000","last_became_active":"2024-09-11T01:29:15.546444+0000","last_became_peered":"2024-09-11T01:29:15.546444+0000","last_unstale":"2024-09-11T01:29:17.597492+0000","last_undegraded":"2024-09-11T01:29:17.597492+0000","last_fullsized":"2024-09-11T01:29:17.597492+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:52:43.624650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000396053,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.579590+0000","last_change":"2024-09-11T01:29:17.579590+0000","last_active":"2024-09-11T01:29:17.579590+0000","last_peered":"2024-09-11T01:29:17.579590+0000","last_clean":"2024-09-11T01:29:17.579590+0000","last_became_active":"2024-09-11T01:29:15.542800+0000","last_became_peered":"2024-09-11T01:29:15.542800+0000","last_unstale":"2024-09-11T01:29:17.579590+0000","last_undegraded":"2024-09-11T01:29:17.579590+0000","last_fullsized":"2024-09-11T01:29:17.579590+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:23:55.773396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040701099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.576853+0000","last_change":"2024-09-11T01:29:17.576853+0000","last_active":"2024-09-11T01:29:17.576853+0000","last_peered":"2024-09-11T01:29:17.576853+0000","last_clean":"2024-09-11T01:29:17.576853+0000","last_became_active":"2024-09-11T01:29:15.546476+0000","last_became_peered":"2024-09-11T01:29:15.546476+0000","last_unstale":"2024-09-11T01:29:17.576853+0000","last_undegraded":"2024-09-11T01:29:17.576853+0000","last_fullsized":"2024-09-11T01:29:17.576853+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:11:54.706642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037031899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.583012+0000","last_change":"2024-09-11T01:29:17.583012+0000","last_active":"2024-09-11T01:29:17.583012+0000","last_peered":"2024-09-11T01:29:17.583012+0000","last_clean":"2024-09-11T01:29:17.583012+0000","last_became_active":"2024-09-11T01:29:15.547447+0000","last_became_peered":"2024-09-11T01:29:15.547447+0000","last_unstale":"2024-09-11T01:29:17.583012+0000","last_undegraded":"2024-09-11T01:29:17.583012+0000","last_fullsized":"2024-09-11T01:29:17.583012+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:17:48.616008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037448800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"16'72","reported_seq":103,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.599361+0000","last_change":"2024-09-11T01:29:16.691671+0000","last_active":"2024-09-11T01:29:17.599361+0000","last_peered":"2024-09-11T01:29:17.599361+0000","last_clean":"2024-09-11T01:29:17.599361+0000","last_became_active":"2024-09-11T01:29:16.690647+0000","last_became_peered":"2024-09-11T01:29:16.690647+0000","last_unstale":"2024-09-11T01:29:17.599361+0000","last_undegraded":"2024-09-11T01:29:17.599361+0000","last_fullsized":"2024-09-11T01:29:17.599361+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:15.533298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:15.533298+0000","last_clean_scrub_stamp":"2024-09-11T01:29:15.533298+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:41:07.416896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.583117+0000","last_change":"2024-09-11T01:29:17.583117+0000","last_active":"2024-09-11T01:29:17.583117+0000","last_peered":"2024-09-11T01:29:17.583117+0000","last_clean":"2024-09-11T01:29:17.583117+0000","last_became_active":"2024-09-11T01:29:15.546142+0000","last_became_peered":"2024-09-11T01:29:15.546142+0000","last_unstale":"2024-09-11T01:29:17.583117+0000","last_undegraded":"2024-09-11T01:29:17.583117+0000","last_fullsized":"2024-09-11T01:29:17.583117+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:16:49.024029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044411199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"16'2","reported_seq":22,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.578002+0000","last_change":"2024-09-11T01:29:17.577463+0000","last_active":"2024-09-11T01:29:17.578002+0000","last_peered":"2024-09-11T01:29:17.578002+0000","last_clean":"2024-09-11T01:29:17.578002+0000","last_became_active":"2024-09-11T01:29:15.548884+0000","last_became_peered":"2024-09-11T01:29:15.548884+0000","last_unstale":"2024-09-11T01:29:17.578002+0000","last_undegraded":"2024-09-11T01:29:17.578002+0000","last_fullsized":"2024-09-11T01:29:17.578002+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:09.120675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094832499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.597459+0000","last_change":"2024-09-11T01:29:17.597459+0000","last_active":"2024-09-11T01:29:17.597459+0000","last_peered":"2024-09-11T01:29:17.597459+0000","last_clean":"2024-09-11T01:29:17.597459+0000","last_became_active":"2024-09-11T01:29:15.549013+0000","last_became_peered":"2024-09-11T01:29:15.549013+0000","last_unstale":"2024-09-11T01:29:17.597459+0000","last_undegraded":"2024-09-11T01:29:17.597459+0000","last_fullsized":"2024-09-11T01:29:17.597459+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:46:02.130174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041667199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.579548+0000","last_change":"2024-09-11T01:29:17.579548+0000","last_active":"2024-09-11T01:29:17.579548+0000","last_peered":"2024-09-11T01:29:17.579548+0000","last_clean":"2024-09-11T01:29:17.579548+0000","last_became_active":"2024-09-11T01:29:15.546952+0000","last_became_peered":"2024-09-11T01:29:15.546952+0000","last_unstale":"2024-09-11T01:29:17.579548+0000","last_undegraded":"2024-09-11T01:29:17.579548+0000","last_fullsized":"2024-09-11T01:29:17.579548+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:45:28.460820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038245800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59136,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607556,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":151552,"data_stored":46513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607556,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":155648,"data_stored":46532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607556,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":212992,"data_stored":575441,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607555,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":212992,"data_stored":575441,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607556,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":155648,"data_stored":46532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":151552,"data_stored":46513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T01:29:24.438 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T01:29:24.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.536+0000 7fc747462640 10 monclient: get_monmap_and_config 2024-09-11T01:29:24.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.536+0000 7fc747462640 10 monclient: build_initial_monmap 2024-09-11T01:29:24.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.536+0000 7fc747462640 10 monclient: monmap: 2024-09-11T01:29:24.539 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:24.539 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:24.539 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:24.537598+0000 2024-09-11T01:29:24.539 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:24.537598+0000 2024-09-11T01:29:24.539 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:24.539 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:24.539 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:24.539 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:24.539 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:24.539 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:24.539 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:24.539 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:24.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.537+0000 7fc747462640 1 Processor -- start 2024-09-11T01:29:24.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.537+0000 7fc747462640 1 -- start start 2024-09-11T01:29:24.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.537+0000 7fc747462640 10 monclient: init 2024-09-11T01:29:24.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc747462640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:24.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc747462640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-11T01:29:24.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc747462640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc740134ad0 0x7fc740134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:24.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc747462640 10 monclient(hunting): picked mon.noname-c con 0x7fc740134ad0 addr v2:172.21.15.71:3300/0 2024-09-11T01:29:24.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc747462640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc7400a2200 0x7fc7401353f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:24.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc747462640 10 monclient(hunting): picked mon.noname-a con 0x7fc7400a2200 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:24.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc747462640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc740133a50 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:24.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc747462640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc74009ed30 con 0x7fc740133a50 2024-09-11T01:29:24.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc747462640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:24.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc747462640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc74009f5d0 con 0x7fc7400a2200 2024-09-11T01:29:24.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc747462640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:24.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc747462640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fc740135930 con 0x7fc740134ad0 2024-09-11T01:29:24.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc747462640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:24.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc747462640 10 monclient(hunting): authenticate will time out at 770.980392s 2024-09-11T01:29:24.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc7449fd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc7400a2200 0x7fc7401353f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:24.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc7459ff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc740133a50 0x7fc740133ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56152/0 (socket says 172.21.15.12:56152) 2024-09-11T01:29:24.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc7459ff640 1 -- 172.21.15.12:0/4076658286 learned_addr learned my addr 172.21.15.12:0/4076658286 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:29:24.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc7449fd640 10 monclient(hunting): get_auth_request con 0x7fc7400a2200 auth_method 0 2024-09-11T01:29:24.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc7449fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:24.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc7449fd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:24.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc7449fd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:24.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc7449fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:24.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc7449fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:24.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc7449fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:24.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc7451fe640 1 --2- 172.21.15.12:0/4076658286 >> v2:172.21.15.71:3300/0 conn(0x7fc740134ad0 0x7fc740134eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:24.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc7451fe640 10 monclient(hunting): get_auth_request con 0x7fc740134ad0 auth_method 0 2024-09-11T01:29:24.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc7451fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:24.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc7451fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:24.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc7451fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:24.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc7449fd640 10 monclient(hunting): handle_auth_done global_id 4347 payload 274 2024-09-11T01:29:24.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc7449fd640 1 -- 172.21.15.12:0/4076658286 >> v2:172.21.15.71:3300/0 conn(0x7fc740134ad0 msgr2=0x7fc740134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:24.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc7449fd640 1 --2- 172.21.15.12:0/4076658286 >> v2:172.21.15.71:3300/0 conn(0x7fc740134ad0 0x7fc740134eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc7449fd640 1 -- 172.21.15.12:0/4076658286 >> v1:172.21.15.12:6789/0 conn(0x7fc740133a50 legacy=0x7fc740133ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:24.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc7449fd640 10 monclient: _finish_hunting 0 2024-09-11T01:29:24.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc7449fd640 1 monclient: found mon.noname-a 2024-09-11T01:29:24.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc7449fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:24.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc7449fd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:24.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.538+0000 7fc7449fd640 1 -- 172.21.15.12:0/4076658286 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc74009f010 con 0x7fc7400a2200 2024-09-11T01:29:24.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.541+0000 7fc7449fd640 1 -- 172.21.15.12:0/4076658286 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7400a2200 msgr2=0x7fc7401353f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:24.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.541+0000 7fc7449fd640 1 --2- 172.21.15.12:0/4076658286 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7400a2200 0x7fc7401353f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc7300097a0 tx=0x7fc73002a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f76b91a190a18d1c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:24.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.541+0000 7fc737fff640 1 -- 172.21.15.12:0/4076658286 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc730037070 con 0x7fc7400a2200 2024-09-11T01:29:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.541+0000 7fc737fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:24.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.541+0000 7fc737fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:24.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.541+0000 7fc737fff640 10 monclient: dump: 2024-09-11T01:29:24.554 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:24.554 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:24.554 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.554 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.554 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:24.554 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:24.554 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:24.554 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:24.554 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:24.554 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:24.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.541+0000 7fc737fff640 10 monclient: _finish_auth 0 2024-09-11T01:29:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.541+0000 7fc737fff640 10 monclient: _check_auth_tickets 2024-09-11T01:29:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.541+0000 7fc737fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.541+0000 7fc737fff640 1 -- 172.21.15.12:0/4076658286 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc730032440 con 0x7fc7400a2200 2024-09-11T01:29:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.541+0000 7fc747462640 5 monclient: authenticate success, global_id 4347 2024-09-11T01:29:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.541+0000 7fc747462640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.541+0000 7fc737fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.541+0000 7fc747462640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.541+0000 7fc747462640 4 set_mon_vals no callback set 2024-09-11T01:29:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.541+0000 7fc737fff640 1 -- 172.21.15.12:0/4076658286 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc730032810 con 0x7fc7400a2200 2024-09-11T01:29:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.541+0000 7fc737fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:24.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.541+0000 7fc737fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.541+0000 7fc737fff640 10 monclient: dump: 2024-09-11T01:29:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:24.556 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.542+0000 7fc747462640 10 monclient: shutdown 2024-09-11T01:29:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.542+0000 7fc747462640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.542+0000 7fc747462640 1 -- 172.21.15.12:0/4076658286 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7400a2200 msgr2=0x7fc7401353f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.542+0000 7fc747462640 1 --2- 172.21.15.12:0/4076658286 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7400a2200 0x7fc7401353f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc7300097a0 tx=0x7fc73002a0a0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.542+0000 7fc747462640 1 -- 172.21.15.12:0/4076658286 shutdown_connections 2024-09-11T01:29:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.542+0000 7fc747462640 1 --2- 172.21.15.12:0/4076658286 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7400a2200 0x7fc7401353f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.542+0000 7fc747462640 1 --2- 172.21.15.12:0/4076658286 >> v2:172.21.15.71:3300/0 conn(0x7fc740134ad0 0x7fc740134eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.542+0000 7fc747462640 1 -- 172.21.15.12:0/4076658286 >> 172.21.15.12:0/4076658286 conn(0x7fc7400a2f70 msgr2=0x7fc7400a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.542+0000 7fc747462640 1 -- 172.21.15.12:0/4076658286 shutdown_connections 2024-09-11T01:29:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.542+0000 7fc747462640 1 -- 172.21.15.12:0/4076658286 wait complete. 2024-09-11T01:29:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.543+0000 7fc747462640 10 monclient: build_initial_monmap 2024-09-11T01:29:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.543+0000 7fc747462640 10 monclient: monmap: 2024-09-11T01:29:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:24.558 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.543+0000 7fc747462640 1 Processor -- start 2024-09-11T01:29:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.543+0000 7fc747462640 1 -- start start 2024-09-11T01:29:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.543+0000 7fc747462640 10 monclient: init 2024-09-11T01:29:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc747462640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc747462640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:29:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc747462640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7400a2200 0x7fc7401cd140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc747462640 10 monclient(hunting): picked mon.noname-c con 0x7fc7400a2200 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc747462640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc740133a50 0x7fc740132b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc747462640 10 monclient(hunting): picked mon.noname-a con 0x7fc740133a50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc747462640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc740134ad0 0x7fc74013cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc747462640 10 monclient(hunting): picked mon.noname-b con 0x7fc740134ad0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc747462640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc747462640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc7400a0150 con 0x7fc740133a50 2024-09-11T01:29:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc747462640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc747462640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc74009ffd0 con 0x7fc740134ad0 2024-09-11T01:29:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc747462640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc747462640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc7401345d0 con 0x7fc7400a2200 2024-09-11T01:29:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc747462640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc747462640 10 monclient(hunting): authenticate will time out at 770.986119s 2024-09-11T01:29:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7449fd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc740133a50 0x7fc740132b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7451fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7400a2200 0x7fc7401cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7449fd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc740133a50 0x7fc740132b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55198/0 (socket says 172.21.15.12:55198) 2024-09-11T01:29:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7449fd640 1 -- 172.21.15.12:0/725853942 learned_addr learned my addr 172.21.15.12:0/725853942 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7451fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7400a2200 0x7fc7401cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51938/0 (socket says 172.21.15.12:51938) 2024-09-11T01:29:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7449fd640 10 monclient(hunting): get_auth_request con 0x7fc740133a50 auth_method 0 2024-09-11T01:29:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7449fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7449fd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7449fd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7451fe640 10 monclient(hunting): get_auth_request con 0x7fc7400a2200 auth_method 0 2024-09-11T01:29:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7451fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7451fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:24.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7451fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:24.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7449fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:24.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7449fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:24.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7449fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:24.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7451fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:24.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7451fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:24.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7451fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:24.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7449fd640 10 monclient(hunting): handle_auth_done global_id 4350 payload 824 2024-09-11T01:29:24.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7459ff640 1 --2- 172.21.15.12:0/725853942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc740134ad0 0x7fc74013cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:24.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7449fd640 1 -- 172.21.15.12:0/725853942 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7400a2200 msgr2=0x7fc7401cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:24.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7449fd640 1 --2- 172.21.15.12:0/725853942 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7400a2200 0x7fc7401cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7449fd640 1 -- 172.21.15.12:0/725853942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc740134ad0 msgr2=0x7fc74013cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:24.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7449fd640 1 --2- 172.21.15.12:0/725853942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc740134ad0 0x7fc74013cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7449fd640 10 monclient: _finish_hunting 0 2024-09-11T01:29:24.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7449fd640 1 monclient: found mon.noname-a 2024-09-11T01:29:24.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7449fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:24.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7449fd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:24.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.544+0000 7fc7449fd640 1 -- 172.21.15.12:0/725853942 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7400a5850 con 0x7fc740133a50 2024-09-11T01:29:24.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.546+0000 7fc7449fd640 1 -- 172.21.15.12:0/725853942 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc740133a50 msgr2=0x7fc740132b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:24.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.546+0000 7fc7449fd640 1 --2- 172.21.15.12:0/725853942 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc740133a50 0x7fc740132b80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc730000c00 tx=0x7fc7300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:24.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.546+0000 7fc735ffb640 1 -- 172.21.15.12:0/725853942 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc730041020 con 0x7fc740133a50 2024-09-11T01:29:24.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.546+0000 7fc735ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:24.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.546+0000 7fc735ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:24.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.546+0000 7fc735ffb640 10 monclient: dump: 2024-09-11T01:29:24.563 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:24.563 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:24.563 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.563 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.563 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:24.563 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:24.563 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:24.563 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:24.564 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:24.564 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:24.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.546+0000 7fc735ffb640 10 monclient: _finish_auth 0 2024-09-11T01:29:24.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.546+0000 7fc735ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:29:24.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.546+0000 7fc735ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:24.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.546+0000 7fc747462640 5 monclient: authenticate success, global_id 4350 2024-09-11T01:29:24.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.546+0000 7fc735ffb640 1 -- 172.21.15.12:0/725853942 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc73003d8f0 con 0x7fc740133a50 2024-09-11T01:29:24.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.546+0000 7fc747462640 10 monclient: _renew_subs 2024-09-11T01:29:24.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.547+0000 7fc747462640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:24.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.547+0000 7fc747462640 1 -- 172.21.15.12:0/725853942 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc74009d480 con 0x7fc740133a50 2024-09-11T01:29:24.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.547+0000 7fc735ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:24.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.547+0000 7fc735ffb640 1 -- 172.21.15.12:0/725853942 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc730037040 con 0x7fc740133a50 2024-09-11T01:29:24.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.547+0000 7fc735ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.547+0000 7fc735ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.547+0000 7fc735ffb640 10 monclient: dump: 2024-09-11T01:29:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:24.565 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.547+0000 7fc747462640 10 monclient: _renew_subs 2024-09-11T01:29:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.547+0000 7fc747462640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.547+0000 7fc747462640 1 -- 172.21.15.12:0/725853942 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc74011e240 con 0x7fc740133a50 2024-09-11T01:29:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.547+0000 7fc7377fe640 4 set_mon_vals no callback set 2024-09-11T01:29:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.548+0000 7fc735ffb640 1 -- 172.21.15.12:0/725853942 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fc73003a030 con 0x7fc740133a50 2024-09-11T01:29:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.548+0000 7fc735ffb640 1 --2- 172.21.15.12:0/725853942 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc72403e260 0x7fc724040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.548+0000 7fc747462640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.548+0000 7fc735ffb640 1 -- 172.21.15.12:0/725853942 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4374+0+0 (secure 0 0 0) 0x7fc7300731a0 con 0x7fc740133a50 2024-09-11T01:29:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.548+0000 7fc747462640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.548+0000 7fc747462640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.548+0000 7fc747462640 1 -- 172.21.15.12:0/725853942 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc74011cd20 con 0x7fc740133a50 2024-09-11T01:29:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.548+0000 7fc7451fe640 1 --2- 172.21.15.12:0/725853942 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc72403e260 0x7fc724040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.548+0000 7fc7451fe640 10 monclient: get_auth_request con 0x7fc72403e260 auth_method 0 2024-09-11T01:29:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.550+0000 7fc7451fe640 1 --2- 172.21.15.12:0/725853942 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc72403e260 0x7fc724040720 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc73c006fd0 tx=0x7fc73c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.553+0000 7fc735ffb640 1 -- 172.21.15.12:0/725853942 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc73004b030 con 0x7fc740133a50 2024-09-11T01:29:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.553+0000 7fc735ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.553+0000 7fc735ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:24.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.736+0000 7fc747462640 1 -- 172.21.15.12:0/725853942 --> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc740074020 con 0x7fc72403e260 2024-09-11T01:29:24.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.738+0000 7fc735ffb640 1 -- 172.21.15.12:0/725853942 <== mgr.4101 v2:172.21.15.71:6816/3189259519 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34179 (secure 0 0 0) 0x7fc740074020 con 0x7fc72403e260 2024-09-11T01:29:24.740 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:29:24.740 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-09-11T01:29:24.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.741+0000 7fc747462640 1 -- 172.21.15.12:0/725853942 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc72403e260 msgr2=0x7fc724040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:24.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.741+0000 7fc747462640 1 --2- 172.21.15.12:0/725853942 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc72403e260 0x7fc724040720 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc73c006fd0 tx=0x7fc73c008040 comp rx=0 tx=0).stop 2024-09-11T01:29:24.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.741+0000 7fc747462640 10 monclient: shutdown 2024-09-11T01:29:24.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.741+0000 7fc747462640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:24.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.741+0000 7fc747462640 1 -- 172.21.15.12:0/725853942 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc740133a50 msgr2=0x7fc740132b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:24.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.741+0000 7fc747462640 1 --2- 172.21.15.12:0/725853942 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc740133a50 0x7fc740132b80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc730000c00 tx=0x7fc7300046c0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.741+0000 7fc747462640 1 -- 172.21.15.12:0/725853942 shutdown_connections 2024-09-11T01:29:24.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.741+0000 7fc747462640 1 --2- 172.21.15.12:0/725853942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc740134ad0 0x7fc74013cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.741+0000 7fc747462640 1 --2- 172.21.15.12:0/725853942 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc72403e260 0x7fc724040720 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.741+0000 7fc747462640 1 --2- 172.21.15.12:0/725853942 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc740133a50 0x7fc740132b80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.741+0000 7fc747462640 1 --2- 172.21.15.12:0/725853942 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7400a2200 0x7fc7401cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.741+0000 7fc747462640 1 -- 172.21.15.12:0/725853942 >> 172.21.15.12:0/725853942 conn(0x7fc7400a2f70 msgr2=0x7fc7400a45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:24.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.741+0000 7fc747462640 1 -- 172.21.15.12:0/725853942 shutdown_connections 2024-09-11T01:29:24.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.741+0000 7fc747462640 1 -- 172.21.15.12:0/725853942 wait complete. 2024-09-11T01:29:24.753 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-11T01:29:22.852280+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163236,"kb_used_data":1016,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067804,"statfs":{"total":579820584960,"available":579653431296,"internally_reserved":0,"allocated":1040384,"data_stored":1336972,"data_compressed":59136,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000439"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.597492+0000","last_change":"2024-09-11T01:29:17.597492+0000","last_active":"2024-09-11T01:29:17.597492+0000","last_peered":"2024-09-11T01:29:17.597492+0000","last_clean":"2024-09-11T01:29:17.597492+0000","last_became_active":"2024-09-11T01:29:15.546444+0000","last_became_peered":"2024-09-11T01:29:15.546444+0000","last_unstale":"2024-09-11T01:29:17.597492+0000","last_undegraded":"2024-09-11T01:29:17.597492+0000","last_fullsized":"2024-09-11T01:29:17.597492+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:52:43.624650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000396053,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.579590+0000","last_change":"2024-09-11T01:29:17.579590+0000","last_active":"2024-09-11T01:29:17.579590+0000","last_peered":"2024-09-11T01:29:17.579590+0000","last_clean":"2024-09-11T01:29:17.579590+0000","last_became_active":"2024-09-11T01:29:15.542800+0000","last_became_peered":"2024-09-11T01:29:15.542800+0000","last_unstale":"2024-09-11T01:29:17.579590+0000","last_undegraded":"2024-09-11T01:29:17.579590+0000","last_fullsized":"2024-09-11T01:29:17.579590+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:23:55.773396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040701099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.576853+0000","last_change":"2024-09-11T01:29:17.576853+0000","last_active":"2024-09-11T01:29:17.576853+0000","last_peered":"2024-09-11T01:29:17.576853+0000","last_clean":"2024-09-11T01:29:17.576853+0000","last_became_active":"2024-09-11T01:29:15.546476+0000","last_became_peered":"2024-09-11T01:29:15.546476+0000","last_unstale":"2024-09-11T01:29:17.576853+0000","last_undegraded":"2024-09-11T01:29:17.576853+0000","last_fullsized":"2024-09-11T01:29:17.576853+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:11:54.706642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037031899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.583012+0000","last_change":"2024-09-11T01:29:17.583012+0000","last_active":"2024-09-11T01:29:17.583012+0000","last_peered":"2024-09-11T01:29:17.583012+0000","last_clean":"2024-09-11T01:29:17.583012+0000","last_became_active":"2024-09-11T01:29:15.547447+0000","last_became_peered":"2024-09-11T01:29:15.547447+0000","last_unstale":"2024-09-11T01:29:17.583012+0000","last_undegraded":"2024-09-11T01:29:17.583012+0000","last_fullsized":"2024-09-11T01:29:17.583012+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:17:48.616008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037448800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"16'72","reported_seq":103,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.599361+0000","last_change":"2024-09-11T01:29:16.691671+0000","last_active":"2024-09-11T01:29:17.599361+0000","last_peered":"2024-09-11T01:29:17.599361+0000","last_clean":"2024-09-11T01:29:17.599361+0000","last_became_active":"2024-09-11T01:29:16.690647+0000","last_became_peered":"2024-09-11T01:29:16.690647+0000","last_unstale":"2024-09-11T01:29:17.599361+0000","last_undegraded":"2024-09-11T01:29:17.599361+0000","last_fullsized":"2024-09-11T01:29:17.599361+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:15.533298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:15.533298+0000","last_clean_scrub_stamp":"2024-09-11T01:29:15.533298+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:41:07.416896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.583117+0000","last_change":"2024-09-11T01:29:17.583117+0000","last_active":"2024-09-11T01:29:17.583117+0000","last_peered":"2024-09-11T01:29:17.583117+0000","last_clean":"2024-09-11T01:29:17.583117+0000","last_became_active":"2024-09-11T01:29:15.546142+0000","last_became_peered":"2024-09-11T01:29:15.546142+0000","last_unstale":"2024-09-11T01:29:17.583117+0000","last_undegraded":"2024-09-11T01:29:17.583117+0000","last_fullsized":"2024-09-11T01:29:17.583117+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:16:49.024029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044411199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"16'2","reported_seq":22,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.578002+0000","last_change":"2024-09-11T01:29:17.577463+0000","last_active":"2024-09-11T01:29:17.578002+0000","last_peered":"2024-09-11T01:29:17.578002+0000","last_clean":"2024-09-11T01:29:17.578002+0000","last_became_active":"2024-09-11T01:29:15.548884+0000","last_became_peered":"2024-09-11T01:29:15.548884+0000","last_unstale":"2024-09-11T01:29:17.578002+0000","last_undegraded":"2024-09-11T01:29:17.578002+0000","last_fullsized":"2024-09-11T01:29:17.578002+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:09.120675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094832499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.597459+0000","last_change":"2024-09-11T01:29:17.597459+0000","last_active":"2024-09-11T01:29:17.597459+0000","last_peered":"2024-09-11T01:29:17.597459+0000","last_clean":"2024-09-11T01:29:17.597459+0000","last_became_active":"2024-09-11T01:29:15.549013+0000","last_became_peered":"2024-09-11T01:29:15.549013+0000","last_unstale":"2024-09-11T01:29:17.597459+0000","last_undegraded":"2024-09-11T01:29:17.597459+0000","last_fullsized":"2024-09-11T01:29:17.597459+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:46:02.130174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041667199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T01:29:17.579548+0000","last_change":"2024-09-11T01:29:17.579548+0000","last_active":"2024-09-11T01:29:17.579548+0000","last_peered":"2024-09-11T01:29:17.579548+0000","last_clean":"2024-09-11T01:29:17.579548+0000","last_became_active":"2024-09-11T01:29:15.546952+0000","last_became_peered":"2024-09-11T01:29:15.546952+0000","last_unstale":"2024-09-11T01:29:17.579548+0000","last_undegraded":"2024-09-11T01:29:17.579548+0000","last_fullsized":"2024-09-11T01:29:17.579548+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:45:28.460820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038245800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59136,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607556,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":151552,"data_stored":46513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607556,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":155648,"data_stored":46532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607556,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":212992,"data_stored":575441,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607555,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":212992,"data_stored":575441,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607556,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":155648,"data_stored":46532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":151552,"data_stored":46513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T01:29:24.754 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-11T01:29:24.754 DEBUG:tasks.mon_thrash:Yielding 2024-09-11T01:29:24.755 INFO:teuthology.run_tasks:Running task rados... 2024-09-11T01:29:24.768 INFO:tasks.rados:Beginning rados... 2024-09-11T01:29:24.768 INFO:tasks.rados:config is {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 50, 'delete': 50, 'read': 100, 'rollback': 50, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000} 2024-09-11T01:29:24.768 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'wip-hemanth-testing-2024-09-10-0723'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-hemanth-testing-2024-09-10-0723', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}} 2024-09-11T01:29:24.769 INFO:tasks.rados:config is {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 50, 'delete': 50, 'read': 100, 'rollback': 50, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000} 2024-09-11T01:29:24.769 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-09-11T01:29:24.777 INFO:tasks.rados:joining rados 2024-09-11T01:29:24.777 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-11T01:29:24.777 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726018164, revive delay: 20.0, thrash delay: 1.0 thrash many: False, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-11T01:29:24.777 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:29:24.778 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:29:24.778 INFO:tasks.rados:clients are ['client.0'] 2024-09-11T01:29:24.778 INFO:tasks.rados:starting run 0 out of 1 2024-09-11T01:29:24.778 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-11T01:29:24.778 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2024-09-11T01:29:24.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.878+0000 7f141a965640 10 monclient: get_monmap_and_config 2024-09-11T01:29:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.878+0000 7f141a965640 10 monclient: build_initial_monmap 2024-09-11T01:29:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.878+0000 7f141a965640 10 monclient: monmap: 2024-09-11T01:29:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:24.878866+0000 2024-09-11T01:29:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:24.878866+0000 2024-09-11T01:29:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:24.881 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.878+0000 7f141a965640 1 Processor -- start 2024-09-11T01:29:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.879+0000 7f141a965640 1 -- start start 2024-09-11T01:29:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.879+0000 7f141a965640 10 monclient: init 2024-09-11T01:29:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.879+0000 7f141a965640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.879+0000 7f141a965640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-11T01:29:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.879+0000 7f141a965640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1414138230 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.879+0000 7f141a965640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f141409d030 0x7f141413d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.879+0000 7f141a965640 10 monclient(hunting): picked mon.noname-a con 0x7f141409d030 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.879+0000 7f141a965640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f141409d460 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.879+0000 7f141a965640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14140a6e30 con 0x7f1414138230 2024-09-11T01:29:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.879+0000 7f141a965640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f141409ba20 con 0x7f141409d460 2024-09-11T01:29:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.879+0000 7f141a965640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.879+0000 7f141a965640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f141409c9c0 con 0x7f141409d030 2024-09-11T01:29:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.879+0000 7f141a965640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.879+0000 7f141a965640 10 monclient(hunting): authenticate will time out at 771.321690s 2024-09-11T01:29:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.879+0000 7f1413fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1414138230 0x7f1414138610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56168/0 (socket says 172.21.15.12:56168) 2024-09-11T01:29:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.879+0000 7f1413fff640 1 -- 172.21.15.12:0/658808310 learned_addr learned my addr 172.21.15.12:0/658808310 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:29:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.879+0000 7f140bfff640 1 --2- 172.21.15.12:0/658808310 >> v2:172.21.15.12:3300/0 conn(0x7f141409d030 0x7f141413d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.879+0000 7f140bfff640 10 monclient(hunting): get_auth_request con 0x7f141409d030 auth_method 0 2024-09-11T01:29:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.879+0000 7f140bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.879+0000 7f140bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.879+0000 7f140bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.879+0000 7f140bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.879+0000 7f140bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.879+0000 7f140bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.880+0000 7f140bfff640 10 monclient(hunting): handle_auth_done global_id 4353 payload 274 2024-09-11T01:29:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.880+0000 7f140bfff640 1 -- 172.21.15.12:0/658808310 >> v1:172.21.15.71:6789/0 conn(0x7f141409d460 legacy=0x7f141413f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.880+0000 7f140bfff640 1 -- 172.21.15.12:0/658808310 >> v1:172.21.15.12:6789/0 conn(0x7f1414138230 legacy=0x7f1414138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.880+0000 7f140bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:29:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.880+0000 7f140bfff640 1 monclient: found mon.noname-a 2024-09-11T01:29:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.880+0000 7f140bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.880+0000 7f140bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.880+0000 7f140bfff640 1 -- 172.21.15.12:0/658808310 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14140a66a0 con 0x7f141409d030 2024-09-11T01:29:24.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.882+0000 7ff05a4e6640 10 monclient: get_monmap_and_config 2024-09-11T01:29:24.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.882+0000 7ff05a4e6640 10 monclient: build_initial_monmap 2024-09-11T01:29:24.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.882+0000 7ff05a4e6640 10 monclient: monmap: 2024-09-11T01:29:24.886 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:24.886 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:24.886 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:24.883366+0000 2024-09-11T01:29:24.886 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:24.883366+0000 2024-09-11T01:29:24.886 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:24.886 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:24.886 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:24.886 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:24.887 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.882+0000 7f140bfff640 1 -- 172.21.15.12:0/658808310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f141409d030 msgr2=0x7f141413d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.882+0000 7f140bfff640 1 --2- 172.21.15.12:0/658808310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f141409d030 0x7f141413d090 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f13fc0097a0 tx=0x7f13fc029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd969c097f655a9e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7f14137fe640 1 -- 172.21.15.12:0/658808310 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13fc037070 con 0x7f141409d030 2024-09-11T01:29:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7f14137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7f14137fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7f14137fe640 10 monclient: dump: 2024-09-11T01:29:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:24.889 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7f14137fe640 10 monclient: _finish_auth 0 2024-09-11T01:29:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7f14137fe640 10 monclient: _check_auth_tickets 2024-09-11T01:29:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7f14137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7f141a965640 5 monclient: authenticate success, global_id 4353 2024-09-11T01:29:24.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7f141a965640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:24.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7f14137fe640 1 -- 172.21.15.12:0/658808310 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13fc032640 con 0x7f141409d030 2024-09-11T01:29:24.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7f14137fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:24.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7f141a965640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:24.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7f141a965640 4 set_mon_vals no callback set 2024-09-11T01:29:24.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7f14137fe640 1 -- 172.21.15.12:0/658808310 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13fc032a10 con 0x7f141409d030 2024-09-11T01:29:24.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7f14137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:24.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7f14137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:24.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7f14137fe640 10 monclient: dump: 2024-09-11T01:29:24.890 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:24.890 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:24.890 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.890 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.890 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:24.891 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:24.891 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:24.891 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:24.891 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:24.891 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:24.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7ff05a4e6640 1 Processor -- start 2024-09-11T01:29:24.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7ff05a4e6640 1 -- start start 2024-09-11T01:29:24.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7ff05a4e6640 10 monclient: init 2024-09-11T01:29:24.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7f141a965640 10 monclient: shutdown 2024-09-11T01:29:24.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7f141a965640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:24.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7f141a965640 1 -- 172.21.15.12:0/658808310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f141409d030 msgr2=0x7f141413d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:24.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7f141a965640 1 --2- 172.21.15.12:0/658808310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f141409d030 0x7f141413d090 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f13fc0097a0 tx=0x7f13fc029fd0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7f141a965640 1 -- 172.21.15.12:0/658808310 shutdown_connections 2024-09-11T01:29:24.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7f141a965640 1 --2- 172.21.15.12:0/658808310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f141409d030 0x7f141413d090 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7f141a965640 1 -- 172.21.15.12:0/658808310 >> 172.21.15.12:0/658808310 conn(0x7f14140a7120 msgr2=0x7f14140a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:24.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7f141a965640 1 -- 172.21.15.12:0/658808310 shutdown_connections 2024-09-11T01:29:24.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.883+0000 7f141a965640 1 -- 172.21.15.12:0/658808310 wait complete. 2024-09-11T01:29:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff05a4e6640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff05a4e6640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-11T01:29:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff05a4e6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff054137ec0 0x7ff0540a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff05a4e6640 10 monclient(hunting): picked mon.noname-b con 0x7ff054137ec0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff05a4e6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff0541382a0 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff05a4e6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff05413d460 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff05a4e6640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff05409c2c0 con 0x7ff0541382a0 2024-09-11T01:29:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff05a4e6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff05409c530 con 0x7ff05413d460 2024-09-11T01:29:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff05a4e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff05a4e6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff0540a68f0 con 0x7ff054137ec0 2024-09-11T01:29:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff05a4e6640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff05a4e6640 10 monclient(hunting): authenticate will time out at 771.326250s 2024-09-11T01:29:24.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff053fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff054137ec0 0x7ff0540a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:24.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff053fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff054137ec0 0x7ff0540a4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51950/0 (socket says 172.21.15.12:51950) 2024-09-11T01:29:24.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff053fff640 1 -- 172.21.15.12:0/1700099481 learned_addr learned my addr 172.21.15.12:0/1700099481 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:24.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff053fff640 10 monclient(hunting): get_auth_request con 0x7ff054137ec0 auth_method 0 2024-09-11T01:29:24.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff053fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:24.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff053fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:24.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff053fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:24.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff0537fe640 1 --1- 172.21.15.12:0/1700099481 >> v1:172.21.15.12:6789/0 conn(0x7ff0541382a0 0x7ff0541386c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56176/0 (socket says 172.21.15.12:56176) 2024-09-11T01:29:24.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff053fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:24.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff053fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:24.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff053fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:24.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7f141a965640 10 monclient: build_initial_monmap 2024-09-11T01:29:24.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7f141a965640 10 monclient: monmap: 2024-09-11T01:29:24.897 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:24.897 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:24.897 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:24.897 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:24.897 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:24.898 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:24.898 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:24.898 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:24.898 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:24.898 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:24.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff053fff640 10 monclient(hunting): handle_auth_done global_id 4376 payload 274 2024-09-11T01:29:24.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff053fff640 1 -- 172.21.15.12:0/1700099481 >> v1:172.21.15.71:6789/0 conn(0x7ff05413d460 legacy=0x7ff05413f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:24.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff053fff640 1 -- 172.21.15.12:0/1700099481 >> v1:172.21.15.12:6789/0 conn(0x7ff0541382a0 legacy=0x7ff0541386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:24.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff053fff640 10 monclient: _finish_hunting 0 2024-09-11T01:29:24.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff053fff640 1 monclient: found mon.noname-b 2024-09-11T01:29:24.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff053fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:24.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff053fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:29:24.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7ff053fff640 1 -- 172.21.15.12:0/1700099481 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff05409c810 con 0x7ff054137ec0 2024-09-11T01:29:24.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7f141a965640 1 Processor -- start 2024-09-11T01:29:24.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7f141a965640 1 -- start start 2024-09-11T01:29:24.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.884+0000 7f141a965640 10 monclient: init 2024-09-11T01:29:24.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f141a965640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:24.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f141a965640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:29:24.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f141a965640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f141409d030 0x7f14141cd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:24.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f141a965640 10 monclient(hunting): picked mon.noname-b con 0x7f141409d030 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:24.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f141a965640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f141409d460 0x7f1414136e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:24.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f141a965640 10 monclient(hunting): picked mon.noname-a con 0x7f141409d460 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:24.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f141a965640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1414138230 0x7f141413cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:24.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f141a965640 10 monclient(hunting): picked mon.noname-c con 0x7f1414138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:24.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f141a965640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:24.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f141a965640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1414140e80 con 0x7f141409d460 2024-09-11T01:29:24.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f141a965640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:24.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f141a965640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1414140d00 con 0x7f141409d030 2024-09-11T01:29:24.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f141a965640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:24.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f141a965640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1414141000 con 0x7f1414138230 2024-09-11T01:29:24.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f141a965640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:24.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f141a965640 10 monclient(hunting): authenticate will time out at 771.327416s 2024-09-11T01:29:24.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f1413fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f141409d030 0x7f14141cd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:24.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f1413fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f141409d030 0x7f14141cd0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:35038/0 (socket says 172.21.15.12:35038) 2024-09-11T01:29:24.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f1413fff640 1 -- 172.21.15.12:0/2213318598 learned_addr learned my addr 172.21.15.12:0/2213318598 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:24.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f1413fff640 10 monclient(hunting): get_auth_request con 0x7f141409d030 auth_method 0 2024-09-11T01:29:24.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f1413fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:24.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f1413fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:24.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f1413fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:24.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f1413fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:24.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f1413fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:24.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f1413fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:24.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f1413fff640 10 monclient(hunting): handle_auth_done global_id 4297 payload 824 2024-09-11T01:29:24.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f1413fff640 1 -- 172.21.15.12:0/2213318598 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1414138230 msgr2=0x7f141413cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:24.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f1413fff640 1 --2- 172.21.15.12:0/2213318598 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1414138230 0x7f141413cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f1413fff640 1 -- 172.21.15.12:0/2213318598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f141409d460 msgr2=0x7f1414136e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:24.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f1413fff640 1 --2- 172.21.15.12:0/2213318598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f141409d460 0x7f1414136e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f1413fff640 10 monclient: _finish_hunting 0 2024-09-11T01:29:24.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f1413fff640 1 monclient: found mon.noname-b 2024-09-11T01:29:24.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f1413fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:24.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f1413fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-09-11T01:29:24.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.885+0000 7f1413fff640 1 -- 172.21.15.12:0/2213318598 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1414140650 con 0x7f141409d030 2024-09-11T01:29:24.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.887+0000 7ff053fff640 1 -- 172.21.15.12:0/1700099481 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff054137ec0 msgr2=0x7ff0540a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:24.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.887+0000 7ff053fff640 1 --2- 172.21.15.12:0/1700099481 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff054137ec0 0x7ff0540a4e90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff0440098a0 tx=0x7ff04402a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40199c44225f4001 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:24.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.887+0000 7ff052ffd640 1 -- 172.21.15.12:0/1700099481 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff044037070 con 0x7ff054137ec0 2024-09-11T01:29:24.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.887+0000 7ff052ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:24.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.887+0000 7ff052ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:24.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.887+0000 7ff052ffd640 10 monclient: dump: 2024-09-11T01:29:24.910 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:24.910 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:24.910 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.910 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.910 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:24.910 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:24.910 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:24.910 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:24.910 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:24.910 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:24.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.887+0000 7ff052ffd640 10 monclient: _finish_auth 0 2024-09-11T01:29:24.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.887+0000 7ff052ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:24.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.887+0000 7ff052ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:24.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.887+0000 7ff05a4e6640 5 monclient: authenticate success, global_id 4376 2024-09-11T01:29:24.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.887+0000 7ff052ffd640 1 -- 172.21.15.12:0/1700099481 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff044032640 con 0x7ff054137ec0 2024-09-11T01:29:24.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.887+0000 7ff05a4e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:24.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.887+0000 7ff052ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:24.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.887+0000 7ff05a4e6640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:24.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.887+0000 7ff05a4e6640 4 set_mon_vals no callback set 2024-09-11T01:29:24.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.887+0000 7ff052ffd640 1 -- 172.21.15.12:0/1700099481 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff044032a10 con 0x7ff054137ec0 2024-09-11T01:29:24.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.887+0000 7ff052ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:24.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.887+0000 7ff052ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:24.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.887+0000 7ff052ffd640 10 monclient: dump: 2024-09-11T01:29:24.911 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:24.911 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:24.911 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.911 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.912 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:24.912 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:24.912 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:24.912 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:24.912 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:24.912 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:24.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7ff05a4e6640 10 monclient: shutdown 2024-09-11T01:29:24.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7ff05a4e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:24.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7ff05a4e6640 1 -- 172.21.15.12:0/1700099481 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff054137ec0 msgr2=0x7ff0540a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:24.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7ff05a4e6640 1 --2- 172.21.15.12:0/1700099481 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff054137ec0 0x7ff0540a4e90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff0440098a0 tx=0x7ff04402a080 comp rx=0 tx=0).stop 2024-09-11T01:29:24.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7ff05a4e6640 1 -- 172.21.15.12:0/1700099481 shutdown_connections 2024-09-11T01:29:24.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7ff05a4e6640 1 --2- 172.21.15.12:0/1700099481 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff054137ec0 0x7ff0540a4e90 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7ff05a4e6640 1 -- 172.21.15.12:0/1700099481 >> 172.21.15.12:0/1700099481 conn(0x7ff05406ec70 msgr2=0x7ff05406f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:24.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7f1413fff640 1 -- 172.21.15.12:0/2213318598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f141409d030 msgr2=0x7f14141cd0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:24.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7f1413fff640 1 --2- 172.21.15.12:0/2213318598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f141409d030 0x7f14141cd0e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f140400c490 tx=0x7f140400c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:24.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7ff05a4e6640 1 -- 172.21.15.12:0/1700099481 shutdown_connections 2024-09-11T01:29:24.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7ff05a4e6640 1 -- 172.21.15.12:0/1700099481 wait complete. 2024-09-11T01:29:24.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7f14117fa640 1 -- 172.21.15.12:0/2213318598 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1404007070 con 0x7f141409d030 2024-09-11T01:29:24.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7f14117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:24.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7f14117fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:24.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7f14117fa640 10 monclient: dump: 2024-09-11T01:29:24.915 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:24.915 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:24.915 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.915 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.915 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:24.915 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:24.915 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:24.915 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:24.915 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:24.916 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:24.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7f14117fa640 10 monclient: _finish_auth 0 2024-09-11T01:29:24.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7f14117fa640 10 monclient: _check_auth_tickets 2024-09-11T01:29:24.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7f14117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:24.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7f14117fa640 1 -- 172.21.15.12:0/2213318598 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1404007230 con 0x7f141409d030 2024-09-11T01:29:24.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7f14117fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:24.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7f14117fa640 1 -- 172.21.15.12:0/2213318598 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1404007620 con 0x7f141409d030 2024-09-11T01:29:24.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7f141a965640 5 monclient: authenticate success, global_id 4297 2024-09-11T01:29:24.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7f14117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:24.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7f14117fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:24.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7f14117fa640 10 monclient: dump: 2024-09-11T01:29:24.916 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:24.916 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:24.917 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.917 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.917 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:24.917 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:24.917 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:24.917 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:24.917 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:24.917 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:24.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7f141a965640 10 monclient: _renew_subs 2024-09-11T01:29:24.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7f141a965640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:24.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7f1412ffd640 4 set_mon_vals no callback set 2024-09-11T01:29:24.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7f141a965640 1 -- 172.21.15.12:0/2213318598 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1414140930 con 0x7f141409d030 2024-09-11T01:29:24.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7f141a965640 10 monclient: _renew_subs 2024-09-11T01:29:24.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7f141a965640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:24.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7f141a965640 1 -- 172.21.15.12:0/2213318598 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14140a1810 con 0x7f141409d030 2024-09-11T01:29:24.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7ff05a4e6640 10 monclient: build_initial_monmap 2024-09-11T01:29:24.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.888+0000 7ff05a4e6640 10 monclient: monmap: 2024-09-11T01:29:24.918 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:24.918 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:24.918 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:24.919 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:24.919 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:24.919 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:24.919 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:24.919 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:24.919 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:24.919 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:24.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.889+0000 7ff05a4e6640 1 Processor -- start 2024-09-11T01:29:24.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.889+0000 7ff05a4e6640 1 -- start start 2024-09-11T01:29:24.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.889+0000 7ff05a4e6640 10 monclient: init 2024-09-11T01:29:24.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.889+0000 7f141a965640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:24.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.889+0000 7f141a965640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:24.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.889+0000 7f141a965640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:24.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.889+0000 7f14117fa640 1 -- 172.21.15.12:0/2213318598 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f140401b400 con 0x7f141409d030 2024-09-11T01:29:24.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.889+0000 7f141a965640 1 -- 172.21.15.12:0/2213318598 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1414138610 con 0x7f141409d030 2024-09-11T01:29:24.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.889+0000 7f14117fa640 1 --2- 172.21.15.12:0/2213318598 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f13f403e260 0x7f13f4040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:24.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.889+0000 7ff05a4e6640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:24.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.889+0000 7ff05a4e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:29:24.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.889+0000 7ff05a4e6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff054137ec0 0x7ff05413b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:24.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.889+0000 7ff05a4e6640 10 monclient(hunting): picked mon.noname-c con 0x7ff054137ec0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:24.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.889+0000 7ff05a4e6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0541382a0 0x7ff054136d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:24.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.889+0000 7ff05a4e6640 10 monclient(hunting): picked mon.noname-a con 0x7ff0541382a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:24.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.889+0000 7ff05a4e6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff05413d460 0x7ff054139440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:24.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.889+0000 7ff05a4e6640 10 monclient(hunting): picked mon.noname-b con 0x7ff05413d460 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:24.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.889+0000 7f14117fa640 1 -- 172.21.15.12:0/2213318598 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4374+0+0 (secure 0 0 0) 0x7f14040529e0 con 0x7f141409d030 2024-09-11T01:29:24.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.889+0000 7ff05a4e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:24.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.889+0000 7ff05a4e6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff05409d810 con 0x7ff0541382a0 2024-09-11T01:29:24.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.889+0000 7ff05a4e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:24.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.889+0000 7ff05a4e6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff05409d690 con 0x7ff05413d460 2024-09-11T01:29:24.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.889+0000 7ff05a4e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:24.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.889+0000 7ff05a4e6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff054140f70 con 0x7ff054137ec0 2024-09-11T01:29:24.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.889+0000 7ff05a4e6640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:24.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.889+0000 7ff05a4e6640 10 monclient(hunting): authenticate will time out at 771.331986s 2024-09-11T01:29:24.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff0537fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0541382a0 0x7ff054136d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:24.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff053fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff054137ec0 0x7ff05413b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:24.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff0537fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0541382a0 0x7ff054136d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55222/0 (socket says 172.21.15.12:55222) 2024-09-11T01:29:24.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff053fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff054137ec0 0x7ff05413b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51968/0 (socket says 172.21.15.12:51968) 2024-09-11T01:29:24.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff0537fe640 1 -- 172.21.15.12:0/3421807171 learned_addr learned my addr 172.21.15.12:0/3421807171 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:24.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff0537fe640 10 monclient(hunting): get_auth_request con 0x7ff0541382a0 auth_method 0 2024-09-11T01:29:24.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7f140bfff640 1 --2- 172.21.15.12:0/2213318598 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f13f403e260 0x7f13f4040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:24.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff0537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:24.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff0537fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:24.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff0537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:24.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff053fff640 10 monclient(hunting): get_auth_request con 0x7ff054137ec0 auth_method 0 2024-09-11T01:29:24.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff053fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:24.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7f140bfff640 10 monclient: get_auth_request con 0x7f13f403e260 auth_method 0 2024-09-11T01:29:24.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff053fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:24.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff053fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:24.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff058a83640 1 --2- 172.21.15.12:0/3421807171 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff05413d460 0x7ff054139440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:24.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff058a83640 10 monclient(hunting): get_auth_request con 0x7ff05413d460 auth_method 0 2024-09-11T01:29:24.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff058a83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:24.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff058a83640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:24.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff058a83640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:24.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff0537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:24.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff0537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:24.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff0537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:24.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff053fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:24.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff053fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:24.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff053fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:24.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff058a83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:24.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff058a83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:24.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff058a83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:24.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff0537fe640 10 monclient(hunting): handle_auth_done global_id 4356 payload 824 2024-09-11T01:29:24.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff0537fe640 1 -- 172.21.15.12:0/3421807171 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff054137ec0 msgr2=0x7ff05413b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:24.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff0537fe640 1 --2- 172.21.15.12:0/3421807171 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff054137ec0 0x7ff05413b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff0537fe640 1 -- 172.21.15.12:0/3421807171 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff05413d460 msgr2=0x7ff054139440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:24.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff0537fe640 1 --2- 172.21.15.12:0/3421807171 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff05413d460 0x7ff054139440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:24.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff0537fe640 10 monclient: _finish_hunting 0 2024-09-11T01:29:24.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff0537fe640 1 monclient: found mon.noname-a 2024-09-11T01:29:24.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff0537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:24.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff0537fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:24.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff0537fe640 1 -- 172.21.15.12:0/3421807171 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff05413c0b0 con 0x7ff0541382a0 2024-09-11T01:29:24.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.890+0000 7ff053fff640 1 --2- 172.21.15.12:0/3421807171 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff054137ec0 0x7ff05413b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:24.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.892+0000 7f140bfff640 1 --2- 172.21.15.12:0/2213318598 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f13f403e260 0x7f13f4040720 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f14140a17e0 tx=0x7f13fc02c000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:24.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.892+0000 7ff0537fe640 1 -- 172.21.15.12:0/3421807171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0541382a0 msgr2=0x7ff054136d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:24.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.892+0000 7ff0537fe640 1 --2- 172.21.15.12:0/3421807171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0541382a0 0x7ff054136d40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff04800c480 tx=0x7ff04800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:24.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.892+0000 7ff050ff9640 1 -- 172.21.15.12:0/3421807171 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff048017070 con 0x7ff0541382a0 2024-09-11T01:29:24.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.892+0000 7ff050ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:24.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.892+0000 7ff050ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:24.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.892+0000 7ff050ff9640 10 monclient: dump: 2024-09-11T01:29:24.930 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:24.930 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:24.930 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.930 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.930 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:24.930 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:24.931 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:24.931 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:24.931 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:24.931 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:24.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.893+0000 7ff050ff9640 10 monclient: _finish_auth 0 2024-09-11T01:29:24.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.893+0000 7ff050ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:29:24.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.893+0000 7ff050ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:24.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.893+0000 7ff050ff9640 1 -- 172.21.15.12:0/3421807171 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff048005810 con 0x7ff0541382a0 2024-09-11T01:29:24.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.893+0000 7ff05a4e6640 5 monclient: authenticate success, global_id 4356 2024-09-11T01:29:24.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.893+0000 7ff050ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:24.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.893+0000 7ff05a4e6640 10 monclient: _renew_subs 2024-09-11T01:29:24.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.893+0000 7ff050ff9640 1 -- 172.21.15.12:0/3421807171 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff048007690 con 0x7ff0541382a0 2024-09-11T01:29:24.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.893+0000 7ff05a4e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:24.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.893+0000 7ff05a4e6640 1 -- 172.21.15.12:0/3421807171 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0540a95c0 con 0x7ff0541382a0 2024-09-11T01:29:24.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.893+0000 7ff050ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:24.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.893+0000 7ff050ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:24.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.893+0000 7ff050ff9640 10 monclient: dump: 2024-09-11T01:29:24.932 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:24.932 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:24.932 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.932 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:24.932 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:24.932 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:24.932 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:24.932 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:24.932 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:24.932 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:24.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.893+0000 7ff0527fc640 4 set_mon_vals no callback set 2024-09-11T01:29:24.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.893+0000 7ff05a4e6640 10 monclient: _renew_subs 2024-09-11T01:29:24.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.893+0000 7ff05a4e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:24.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.893+0000 7ff05a4e6640 1 -- 172.21.15.12:0/3421807171 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff05413cf20 con 0x7ff0541382a0 2024-09-11T01:29:24.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.894+0000 7ff05a4e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:24.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.894+0000 7ff05a4e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:24.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.894+0000 7ff05a4e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:24.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.894+0000 7ff050ff9640 1 -- 172.21.15.12:0/3421807171 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff048005cc0 con 0x7ff0541382a0 2024-09-11T01:29:24.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.894+0000 7ff05a4e6640 1 -- 172.21.15.12:0/3421807171 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0540a4e90 con 0x7ff0541382a0 2024-09-11T01:29:24.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.894+0000 7ff050ff9640 1 --2- 172.21.15.12:0/3421807171 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff04003e210 0x7ff0400406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:24.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.894+0000 7ff050ff9640 1 -- 172.21.15.12:0/3421807171 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4374+0+0 (secure 0 0 0) 0x7ff0480522f0 con 0x7ff0541382a0 2024-09-11T01:29:24.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.894+0000 7ff053fff640 1 --2- 172.21.15.12:0/3421807171 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff04003e210 0x7ff0400406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:24.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.894+0000 7ff053fff640 10 monclient: get_auth_request con 0x7ff04003e210 auth_method 0 2024-09-11T01:29:24.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.894+0000 7f14117fa640 1 -- 172.21.15.12:0/2213318598 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1404051570 con 0x7f141409d030 2024-09-11T01:29:24.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.895+0000 7f14117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:24.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.895+0000 7f14117fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:24.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.897+0000 7ff053fff640 1 --2- 172.21.15.12:0/3421807171 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff04003e210 0x7ff0400406d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff0540a7010 tx=0x7ff04402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:24.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.899+0000 7ff050ff9640 1 -- 172.21.15.12:0/3421807171 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff0540a95c0 con 0x7ff0541382a0 2024-09-11T01:29:24.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.899+0000 7ff050ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:24.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:24.899+0000 7ff050ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:25.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.085+0000 7ff05a4e6640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}] 2024-09-11T01:29:25.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.085+0000 7ff05a4e6640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}] 2024-09-11T01:29:25.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.085+0000 7ff05a4e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:25.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.085+0000 7ff05a4e6640 1 -- 172.21.15.12:0/3421807171 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} v 0) -- 0x7ff0540698c0 con 0x7ff0541382a0 2024-09-11T01:29:25.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.151+0000 7f141a965640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:29:25.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.151+0000 7f141a965640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:29:25.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.151+0000 7f141a965640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:25.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.151+0000 7f141a965640 1 -- 172.21.15.12:0/2213318598 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f14140698e0 con 0x7f141409d030 2024-09-11T01:29:25.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.152+0000 7f14117fa640 1 -- 172.21.15.12:0/2213318598 <== mon.1 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f1404012400 con 0x7f141409d030 2024-09-11T01:29:25.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.152+0000 7f14117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:29:25.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.152+0000 7f14117fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:25.154 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:29:25.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.154+0000 7f141a965640 1 -- 172.21.15.12:0/2213318598 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f13f403e260 msgr2=0x7f13f4040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.154+0000 7f141a965640 1 --2- 172.21.15.12:0/2213318598 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f13f403e260 0x7f13f4040720 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f14140a17e0 tx=0x7f13fc02c000 comp rx=0 tx=0).stop 2024-09-11T01:29:25.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.154+0000 7f141a965640 10 monclient: shutdown 2024-09-11T01:29:25.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.154+0000 7f141a965640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:25.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.154+0000 7f141a965640 1 -- 172.21.15.12:0/2213318598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f141409d030 msgr2=0x7f14141cd0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.154+0000 7f141a965640 1 --2- 172.21.15.12:0/2213318598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f141409d030 0x7f14141cd0e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f140400c490 tx=0x7f140400c950 comp rx=0 tx=0).stop 2024-09-11T01:29:25.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.154+0000 7f141a965640 1 -- 172.21.15.12:0/2213318598 shutdown_connections 2024-09-11T01:29:25.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.154+0000 7f141a965640 1 --2- 172.21.15.12:0/2213318598 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1414138230 0x7f141413cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.154+0000 7f141a965640 1 --2- 172.21.15.12:0/2213318598 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f13f403e260 0x7f13f4040720 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.154+0000 7f141a965640 1 --2- 172.21.15.12:0/2213318598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f141409d460 0x7f1414136e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.154+0000 7f141a965640 1 --2- 172.21.15.12:0/2213318598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f141409d030 0x7f14141cd0e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.154+0000 7f141a965640 1 -- 172.21.15.12:0/2213318598 >> 172.21.15.12:0/2213318598 conn(0x7f14140a7120 msgr2=0x7f14140a3400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:25.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.154+0000 7f141a965640 1 -- 172.21.15.12:0/2213318598 shutdown_connections 2024-09-11T01:29:25.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.154+0000 7f141a965640 1 -- 172.21.15.12:0/2213318598 wait complete. 2024-09-11T01:29:25.167 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:29:25.167 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:29:25.167 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:29:25.167 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:29:25.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.268+0000 7f932a12b640 10 monclient: get_monmap_and_config 2024-09-11T01:29:25.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.268+0000 7f932a12b640 10 monclient: build_initial_monmap 2024-09-11T01:29:25.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.268+0000 7f932a12b640 10 monclient: monmap: 2024-09-11T01:29:25.270 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:25.270 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.270 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:25.268792+0000 2024-09-11T01:29:25.270 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:25.268792+0000 2024-09-11T01:29:25.270 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:25.270 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.270 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:25.270 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:25.270 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:25.270 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:25.271 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:25.271 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.268+0000 7f932a12b640 1 Processor -- start 2024-09-11T01:29:25.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.268+0000 7f932a12b640 1 -- start start 2024-09-11T01:29:25.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.268+0000 7f932a12b640 10 monclient: init 2024-09-11T01:29:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.269+0000 7f932a12b640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.269+0000 7f932a12b640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-11T01:29:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.269+0000 7f932a12b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9324137d50 0x7f932409d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.269+0000 7f932a12b640 10 monclient(hunting): picked mon.noname-c con 0x7f9324137d50 addr v2:172.21.15.71:3300/0 2024-09-11T01:29:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.269+0000 7f932a12b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9324138130 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.269+0000 7f932a12b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f932413d280 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.269+0000 7f932a12b640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f932409ba20 con 0x7f932413d280 2024-09-11T01:29:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.269+0000 7f932a12b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f932409c370 con 0x7f9324138130 2024-09-11T01:29:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.269+0000 7f932a12b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.269+0000 7f932a12b640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f932409c650 con 0x7f9324137d50 2024-09-11T01:29:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.269+0000 7f932a12b640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.269+0000 7f932a12b640 10 monclient(hunting): authenticate will time out at 771.711407s 2024-09-11T01:29:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.269+0000 7f9323fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f932413d280 0x7f932413f670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56182/0 (socket says 172.21.15.12:56182) 2024-09-11T01:29:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.269+0000 7f9323fff640 1 -- 172.21.15.12:0/3967052414 learned_addr learned my addr 172.21.15.12:0/3967052414 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:29:25.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.269+0000 7f93237fe640 1 --2- 172.21.15.12:0/3967052414 >> v2:172.21.15.71:3300/0 conn(0x7f9324137d50 0x7f932409d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.269+0000 7f93237fe640 1 -- 172.21.15.12:0/3967052414 >> v2:172.21.15.71:3300/0 conn(0x7f9324137d50 msgr2=0x7f932409d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:29:25.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.269+0000 7f93237fe640 1 -- 172.21.15.12:0/3967052414 >> v2:172.21.15.71:3300/0 conn(0x7f9324137d50 msgr2=0x7f932409d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:29:25.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.269+0000 7f93237fe640 1 --2- 172.21.15.12:0/3967052414 >> v2:172.21.15.71:3300/0 conn(0x7f9324137d50 0x7f932409d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:29:25.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.269+0000 7f93237fe640 1 --2- 172.21.15.12:0/3967052414 >> v2:172.21.15.71:3300/0 conn(0x7f9324137d50 0x7f932409d110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:29:25.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.271+0000 7f9323fff640 1 -- 172.21.15.12:0/3967052414 >> v1:172.21.15.12:6789/0 conn(0x7f932413d280 legacy=0x7f932413f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:25.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.271+0000 7f9322ffd640 1 -- 172.21.15.12:0/3967052414 >> v1:172.21.15.71:6789/0 conn(0x7f9324138130 legacy=0x7f9324138580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:25.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.271+0000 7f93227fc640 1 -- 172.21.15.12:0/3967052414 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 861073783 0 0) 0x7f932409ba20 con 0x7f932413d280 2024-09-11T01:29:25.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.271+0000 7f93227fc640 10 monclient(hunting): _negotiate 2024-09-11T01:29:25.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.271+0000 7f93227fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.271+0000 7f93227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:25.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.271+0000 7f93227fc640 10 monclient(hunting): my global_id is 4359 2024-09-11T01:29:25.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.271+0000 7f93227fc640 1 -- 172.21.15.12:0/3967052414 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f930c003520 con 0x7f932413d280 2024-09-11T01:29:25.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 1 -- 172.21.15.12:0/3967052414 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2401137962 0 0) 0x7f932409c370 con 0x7f9324138130 2024-09-11T01:29:25.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 10 monclient(hunting): _negotiate 2024-09-11T01:29:25.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:25.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 10 monclient(hunting): my global_id is 4303 2024-09-11T01:29:25.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 1 -- 172.21.15.12:0/3967052414 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f932409ba20 con 0x7f9324138130 2024-09-11T01:29:25.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 1 -- 172.21.15.12:0/3967052414 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4031622733 0 0) 0x7f930c003520 con 0x7f932413d280 2024-09-11T01:29:25.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 1 -- 172.21.15.12:0/3967052414 >> v2:172.21.15.71:3300/0 conn(0x7f9324137d50 msgr2=0x7f932409d110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:29:25.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 1 --2- 172.21.15.12:0/3967052414 >> v2:172.21.15.71:3300/0 conn(0x7f9324137d50 0x7f932409d110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 1 -- 172.21.15.12:0/3967052414 >> v1:172.21.15.71:6789/0 conn(0x7f9324138130 legacy=0x7f9324138580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 10 monclient: _finish_hunting 0 2024-09-11T01:29:25.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:29:25.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:25.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:29:25.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 1 -- 172.21.15.12:0/3967052414 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93240a6ce0 con 0x7f932413d280 2024-09-11T01:29:25.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 10 monclient: _finish_auth 0 2024-09-11T01:29:25.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 10 monclient: _check_auth_tickets 2024-09-11T01:29:25.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:25.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 1 -- 172.21.15.12:0/3967052414 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f931c001d20 con 0x7f932413d280 2024-09-11T01:29:25.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f932a12b640 5 monclient: authenticate success, global_id 4359 2024-09-11T01:29:25.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f932a12b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:25.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:25.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:29:25.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 10 monclient: dump: 2024-09-11T01:29:25.281 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:25.281 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.281 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.281 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.281 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:25.281 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.281 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:25.281 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:25.281 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:25.281 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:29:25.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:25.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 1 -- 172.21.15.12:0/3967052414 >> v1:172.21.15.12:6789/0 conn(0x7f932413d280 legacy=0x7f932413f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:29:25.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 1 --2- 172.21.15.12:0/3967052414 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f930c003c80 0x7f930c004080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 10 monclient(hunting): picked mon.b con 0x7f930c003c80 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:25.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 1 --2- 172.21.15.12:0/3967052414 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f930c004800 0x7f930c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 10 monclient(hunting): picked mon.c con 0x7f930c004800 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:25.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 1 --2- 172.21.15.12:0/3967052414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f930c0053c0 0x7f930c00ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 10 monclient(hunting): picked mon.a con 0x7f930c0053c0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:25.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 1 -- 172.21.15.12:0/3967052414 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f932409c650 con 0x7f930c0053c0 2024-09-11T01:29:25.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 1 -- 172.21.15.12:0/3967052414 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f930c00c300 con 0x7f930c003c80 2024-09-11T01:29:25.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 1 -- 172.21.15.12:0/3967052414 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f930c00c500 con 0x7f930c004800 2024-09-11T01:29:25.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:25.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93227fc640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:29:25.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f932a12b640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:25.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9322ffd640 1 --2- 172.21.15.12:0/3967052414 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f930c004800 0x7f930c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9322ffd640 10 monclient(hunting): get_auth_request con 0x7f930c004800 auth_method 0 2024-09-11T01:29:25.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9322ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:25.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9322ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9322ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:29:25.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9323fff640 1 --2- 172.21.15.12:0/3967052414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f930c0053c0 0x7f930c00ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93237fe640 1 --2- 172.21.15.12:0/3967052414 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f930c003c80 0x7f930c004080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9323fff640 10 monclient(hunting): get_auth_request con 0x7f930c0053c0 auth_method 0 2024-09-11T01:29:25.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9323fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:25.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9323fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9323fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:29:25.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93237fe640 10 monclient(hunting): get_auth_request con 0x7f930c003c80 auth_method 0 2024-09-11T01:29:25.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:25.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93237fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93237fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:29:25.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9322ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:25.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9322ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:25.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9322ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:29:25.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9323fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:25.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9323fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:25.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9323fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:29:25.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:25.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:25.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f93237fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:29:25.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9322ffd640 10 monclient(hunting): handle_auth_done global_id 4359 payload 293 2024-09-11T01:29:25.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9322ffd640 1 -- 172.21.15.12:0/3967052414 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f930c003c80 msgr2=0x7f930c004080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:25.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9322ffd640 1 --2- 172.21.15.12:0/3967052414 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f930c003c80 0x7f930c004080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9322ffd640 1 -- 172.21.15.12:0/3967052414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f930c0053c0 msgr2=0x7f930c00ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:25.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9322ffd640 1 --2- 172.21.15.12:0/3967052414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f930c0053c0 0x7f930c00ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9322ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:29:25.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9322ffd640 1 monclient: found mon.c 2024-09-11T01:29:25.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9322ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:25.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9322ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:25.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9322ffd640 1 -- 172.21.15.12:0/3967052414 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f930c003520 con 0x7f930c004800 2024-09-11T01:29:25.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9322ffd640 10 monclient: _finish_auth 0 2024-09-11T01:29:25.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9322ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:25.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9323fff640 1 -- 172.21.15.12:0/3967052414 reap_dead start 2024-09-11T01:29:25.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.272+0000 7f9322ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:25.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.275+0000 7f9322ffd640 1 -- 172.21.15.12:0/3967052414 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f930c004800 msgr2=0x7f930c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:25.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.275+0000 7f9322ffd640 1 --2- 172.21.15.12:0/3967052414 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f930c004800 0x7f930c004c80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9318001520 tx=0x7f930c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d47f154074a8070 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:25.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.275+0000 7f93227fc640 1 -- 172.21.15.12:0/3967052414 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9318007960 con 0x7f930c004800 2024-09-11T01:29:25.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.275+0000 7f93227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:25.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.275+0000 7f93227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:25.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.275+0000 7f93227fc640 10 monclient: dump: 2024-09-11T01:29:25.286 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:25.286 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.286 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.287 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.287 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:25.287 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.287 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:25.287 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:25.287 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:25.287 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.275+0000 7f932a12b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:25.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.275+0000 7f93227fc640 1 -- 172.21.15.12:0/3967052414 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f931802e070 con 0x7f930c004800 2024-09-11T01:29:25.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.275+0000 7f93227fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:25.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.275+0000 7f932a12b640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:25.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.275+0000 7f93227fc640 1 -- 172.21.15.12:0/3967052414 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9318007070 con 0x7f930c004800 2024-09-11T01:29:25.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.275+0000 7f93227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:25.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.275+0000 7f932a12b640 4 set_mon_vals no callback set 2024-09-11T01:29:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.275+0000 7f93227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.275+0000 7f93227fc640 10 monclient: dump: 2024-09-11T01:29:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:25.288 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.276+0000 7f932a12b640 10 monclient: shutdown 2024-09-11T01:29:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.276+0000 7f932a12b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:25.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.276+0000 7f932a12b640 1 -- 172.21.15.12:0/3967052414 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f930c004800 msgr2=0x7f930c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.276+0000 7f932a12b640 1 --2- 172.21.15.12:0/3967052414 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f930c004800 0x7f930c004c80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9318001520 tx=0x7f930c005830 comp rx=0 tx=0).stop 2024-09-11T01:29:25.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.276+0000 7f932a12b640 1 -- 172.21.15.12:0/3967052414 shutdown_connections 2024-09-11T01:29:25.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.276+0000 7f932a12b640 1 --2- 172.21.15.12:0/3967052414 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f930c004800 0x7f930c004c80 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.276+0000 7f932a12b640 1 -- 172.21.15.12:0/3967052414 >> 172.21.15.12:0/3967052414 conn(0x7f93240a7020 msgr2=0x7f93240a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:25.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.276+0000 7f932a12b640 1 -- 172.21.15.12:0/3967052414 shutdown_connections 2024-09-11T01:29:25.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.276+0000 7f932a12b640 1 -- 172.21.15.12:0/3967052414 wait complete. 2024-09-11T01:29:25.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.277+0000 7f932a12b640 10 monclient: build_initial_monmap 2024-09-11T01:29:25.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.277+0000 7f932a12b640 10 monclient: monmap: 2024-09-11T01:29:25.290 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:25.290 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:25.290 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:25.290 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:25.290 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:25.290 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.290 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:25.290 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:25.290 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:25.290 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.277+0000 7f932a12b640 1 Processor -- start 2024-09-11T01:29:25.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.277+0000 7f932a12b640 1 -- start start 2024-09-11T01:29:25.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.277+0000 7f932a12b640 10 monclient: init 2024-09-11T01:29:25.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.277+0000 7f932a12b640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:25.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f932a12b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:29:25.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f932a12b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f930c004800 0x7f9324120b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f932a12b640 10 monclient(hunting): picked mon.noname-a con 0x7f930c004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:25.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f932a12b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f93241210b0 0x7f93241214f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f932a12b640 10 monclient(hunting): picked mon.noname-b con 0x7f93241210b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:25.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f932a12b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93241d1440 0x7f93241d3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f932a12b640 10 monclient(hunting): picked mon.noname-c con 0x7f93241d1440 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:25.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f932a12b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f932a12b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9324138700 con 0x7f930c004800 2024-09-11T01:29:25.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f932a12b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f932a12b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f9324138580 con 0x7f93241210b0 2024-09-11T01:29:25.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f932a12b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f932a12b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f9324138880 con 0x7f93241d1440 2024-09-11T01:29:25.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f932a12b640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:25.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f932a12b640 10 monclient(hunting): authenticate will time out at 771.720139s 2024-09-11T01:29:25.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f93237fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f930c004800 0x7f9324120b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f93237fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f930c004800 0x7f9324120b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55240/0 (socket says 172.21.15.12:55240) 2024-09-11T01:29:25.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f93237fe640 1 -- 172.21.15.12:0/286465303 learned_addr learned my addr 172.21.15.12:0/286465303 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:25.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f93237fe640 10 monclient(hunting): get_auth_request con 0x7f930c004800 auth_method 0 2024-09-11T01:29:25.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f93237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:25.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f93237fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f93237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:25.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f9323fff640 1 --2- 172.21.15.12:0/286465303 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93241d1440 0x7f93241d3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f9323fff640 10 monclient(hunting): get_auth_request con 0x7f93241d1440 auth_method 0 2024-09-11T01:29:25.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f9323fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:25.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f9323fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f9323fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:25.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f93237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:25.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f93237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:25.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f93237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:25.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f9323fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:25.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f9323fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:25.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f9323fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:25.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f9322ffd640 1 --2- 172.21.15.12:0/286465303 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f93241210b0 0x7f93241214f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f9322ffd640 10 monclient(hunting): get_auth_request con 0x7f93241210b0 auth_method 0 2024-09-11T01:29:25.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f9322ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:25.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f9322ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f9322ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:25.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f93237fe640 10 monclient(hunting): handle_auth_done global_id 4362 payload 824 2024-09-11T01:29:25.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f93237fe640 1 -- 172.21.15.12:0/286465303 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93241d1440 msgr2=0x7f93241d3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f93237fe640 1 --2- 172.21.15.12:0/286465303 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93241d1440 0x7f93241d3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f93237fe640 1 -- 172.21.15.12:0/286465303 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f93241210b0 msgr2=0x7f93241214f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f93237fe640 1 --2- 172.21.15.12:0/286465303 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f93241210b0 0x7f93241214f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f93237fe640 10 monclient: _finish_hunting 0 2024-09-11T01:29:25.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f93237fe640 1 monclient: found mon.noname-a 2024-09-11T01:29:25.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f93237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:25.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f93237fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:25.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f93237fe640 1 -- 172.21.15.12:0/286465303 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9324123d30 con 0x7f930c004800 2024-09-11T01:29:25.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.278+0000 7f9323fff640 1 --2- 172.21.15.12:0/286465303 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93241d1440 0x7f93241d3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:25.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.280+0000 7f93237fe640 1 -- 172.21.15.12:0/286465303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f930c004800 msgr2=0x7f9324120b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:25.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.280+0000 7f93237fe640 1 --2- 172.21.15.12:0/286465303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f930c004800 0x7f9324120b70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f931400f8e0 tx=0x7f931400fda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:25.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.280+0000 7f930bfff640 1 -- 172.21.15.12:0/286465303 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931400ddc0 con 0x7f930c004800 2024-09-11T01:29:25.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.280+0000 7f930bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:25.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.280+0000 7f930bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:25.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.280+0000 7f930bfff640 10 monclient: dump: 2024-09-11T01:29:25.304 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:25.304 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.304 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.304 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.305 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:25.305 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.305 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:25.305 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:25.305 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:25.305 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.280+0000 7f930bfff640 10 monclient: _finish_auth 0 2024-09-11T01:29:25.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.280+0000 7f930bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:29:25.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.280+0000 7f930bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:25.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.280+0000 7f930bfff640 1 -- 172.21.15.12:0/286465303 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9314004020 con 0x7f930c004800 2024-09-11T01:29:25.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.280+0000 7f932a12b640 5 monclient: authenticate success, global_id 4362 2024-09-11T01:29:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.280+0000 7f932a12b640 10 monclient: _renew_subs 2024-09-11T01:29:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.280+0000 7f932a12b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.280+0000 7f932a12b640 1 -- 172.21.15.12:0/286465303 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9324124010 con 0x7f930c004800 2024-09-11T01:29:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.280+0000 7f930bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.280+0000 7f930bfff640 1 -- 172.21.15.12:0/286465303 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9314004410 con 0x7f930c004800 2024-09-11T01:29:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.280+0000 7f930bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.281+0000 7f930bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.281+0000 7f930bfff640 10 monclient: dump: 2024-09-11T01:29:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:25.306 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.307 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.307 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.307 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:25.307 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.307 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:25.307 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:25.307 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:25.307 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.281+0000 7f932a12b640 10 monclient: _renew_subs 2024-09-11T01:29:25.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.281+0000 7f932a12b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:25.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.280+0000 7f9321ffb640 4 set_mon_vals no callback set 2024-09-11T01:29:25.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.281+0000 7f932a12b640 1 -- 172.21.15.12:0/286465303 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9324121cb0 con 0x7f930c004800 2024-09-11T01:29:25.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.281+0000 7f932a12b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:25.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.281+0000 7f932a12b640 1 --2- 172.21.15.12:0/286465303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93240a23b0 0x7f93240a2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.281+0000 7f932a12b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.281+0000 7f932a12b640 1 -- 172.21.15.12:0/286465303 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f932409d110 con 0x7f93240a23b0 2024-09-11T01:29:25.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.282+0000 7f9322ffd640 1 --2- 172.21.15.12:0/286465303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93240a23b0 0x7f93240a2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.282+0000 7f930bfff640 1 -- 172.21.15.12:0/286465303 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f9314003610 con 0x7f930c004800 2024-09-11T01:29:25.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.282+0000 7f9322ffd640 10 monclient: get_auth_request con 0x7f93240a23b0 auth_method 0 2024-09-11T01:29:25.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.282+0000 7f9322ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:25.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.282+0000 7f9322ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.282+0000 7f9322ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:25.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.282+0000 7f9322ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:25.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.282+0000 7f9322ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:25.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.282+0000 7f9322ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:25.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.282+0000 7f9322ffd640 10 monclient(hunting): handle_auth_done global_id 4365 payload 824 2024-09-11T01:29:25.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.282+0000 7f9322ffd640 1 -- 172.21.15.12:0/286465303 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f93240008d0 con 0x7f93240a23b0 2024-09-11T01:29:25.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.282+0000 7f930bfff640 1 --2- 172.21.15.12:0/286465303 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f931003e210 0x7f93100406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.282+0000 7f930bfff640 1 -- 172.21.15.12:0/286465303 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4374+0+0 (secure 0 0 0) 0x7f9314053340 con 0x7f930c004800 2024-09-11T01:29:25.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.282+0000 7f9323fff640 1 --2- 172.21.15.12:0/286465303 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f931003e210 0x7f93100406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.282+0000 7f9323fff640 10 monclient: get_auth_request con 0x7f931003e210 auth_method 0 2024-09-11T01:29:25.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.284+0000 7f9322ffd640 1 -- 172.21.15.12:0/286465303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93240a23b0 msgr2=0x7f93240a2790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:25.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.284+0000 7f9322ffd640 1 --2- 172.21.15.12:0/286465303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93240a23b0 0x7f93240a2790 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9318032000 tx=0x7f9318001720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:25.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.284+0000 7f930bfff640 1 -- 172.21.15.12:0/286465303 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9318044070 con 0x7f93240a23b0 2024-09-11T01:29:25.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.284+0000 7f930bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:25.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.284+0000 7f930bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:25.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.284+0000 7f930bfff640 10 monclient: dump: 2024-09-11T01:29:25.310 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:25.310 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.310 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.310 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.310 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:25.310 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.310 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:25.311 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:25.311 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:25.311 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.285+0000 7f930bfff640 1 -- 172.21.15.12:0/286465303 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f931802e040 con 0x7f93240a23b0 2024-09-11T01:29:25.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.285+0000 7f930bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:25.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.285+0000 7f930bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:25.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.285+0000 7f930bfff640 1 -- 172.21.15.12:0/286465303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93240a23b0 msgr2=0x7f93240a2790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.285+0000 7f930bfff640 1 --2- 172.21.15.12:0/286465303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93240a23b0 0x7f93240a2790 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9318032000 tx=0x7f9318001720 comp rx=0 tx=0).stop 2024-09-11T01:29:25.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.285+0000 7f930bfff640 1 -- 172.21.15.12:0/286465303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93240a23b0 msgr2=0x7f93240a2790 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:29:25.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.285+0000 7f930bfff640 1 --2- 172.21.15.12:0/286465303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93240a23b0 0x7f93240a2790 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.285+0000 7f9323fff640 1 --2- 172.21.15.12:0/286465303 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f931003e210 0x7f93100406d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9324138c90 tx=0x7f931c006040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.291+0000 7f932a12b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:29:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.291+0000 7f932a12b640 1 --2- 172.21.15.12:0/286465303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93240a2cd0 0x7f93241d7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.291+0000 7f932a12b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.291+0000 7f932a12b640 1 -- 172.21.15.12:0/286465303 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9324137990 con 0x7f93240a2cd0 2024-09-11T01:29:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.291+0000 7f9322ffd640 1 --2- 172.21.15.12:0/286465303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93240a2cd0 0x7f93241d7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.291+0000 7f9322ffd640 10 monclient: get_auth_request con 0x7f93240a2cd0 auth_method 0 2024-09-11T01:29:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.291+0000 7f9322ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.291+0000 7f9322ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.291+0000 7f9322ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:25.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.291+0000 7f9322ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:25.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.291+0000 7f9322ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:25.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.291+0000 7f9322ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:25.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.291+0000 7f9322ffd640 10 monclient(hunting): handle_auth_done global_id 4368 payload 824 2024-09-11T01:29:25.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.291+0000 7f9322ffd640 1 -- 172.21.15.12:0/286465303 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f932413be60 con 0x7f93240a2cd0 2024-09-11T01:29:25.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.293+0000 7f9322ffd640 1 -- 172.21.15.12:0/286465303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93240a2cd0 msgr2=0x7f93241d7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:25.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.293+0000 7f9322ffd640 1 --2- 172.21.15.12:0/286465303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93240a2cd0 0x7f93241d7d90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9318001070 tx=0x7f93180016b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:25.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.293+0000 7f930bfff640 1 -- 172.21.15.12:0/286465303 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9318044050 con 0x7f93240a2cd0 2024-09-11T01:29:25.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.293+0000 7f930bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:25.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.293+0000 7f930bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:25.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.293+0000 7f930bfff640 10 monclient: dump: 2024-09-11T01:29:25.314 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:25.314 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.314 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.314 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.314 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:25.314 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.314 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:25.314 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:25.314 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:25.315 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.294+0000 7f930bfff640 1 -- 172.21.15.12:0/286465303 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7f9318045040 con 0x7f93240a2cd0 2024-09-11T01:29:25.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.294+0000 7f930bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:29:25.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.294+0000 7f930bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:25.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.294+0000 7f930bfff640 1 -- 172.21.15.12:0/286465303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93240a2cd0 msgr2=0x7f93241d7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.294+0000 7f930bfff640 1 --2- 172.21.15.12:0/286465303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93240a2cd0 0x7f93241d7d90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9318001070 tx=0x7f93180016b0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.294+0000 7f930bfff640 1 -- 172.21.15.12:0/286465303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93240a2cd0 msgr2=0x7f93241d7d90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:29:25.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.294+0000 7f930bfff640 1 --2- 172.21.15.12:0/286465303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93240a2cd0 0x7f93241d7d90 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9318001070 tx=0x7f93180016b0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.316 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:29:25.316 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:29:25.316 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:29:25.316 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:29:25.316 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 8, 2024-09-11T01:29:25.316 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 27984, 2024-09-11T01:29:25.316 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:29:25.317 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:29:25.317 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:29:25.317 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:29:25.317 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:25.317 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 22, 2024-09-11T01:29:25.317 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:29:25.317 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:29:25.317 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:29:25.317 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:29:25.317 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:29:25.318 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:29:25.318 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:29:25.318 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:29:25.318 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:29:25.318 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:29:25.318 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:29:25.318 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:29:25.318 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:29:25.318 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:29:25.318 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:25.319 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:29:25.319 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:29:25.319 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:29:25.319 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:29:25.319 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:29:25.319 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:29:25.319 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:29:25.319 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:29:25.319 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:29:25.319 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:29:25.320 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:29:25.320 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:29:25.320 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:29:25.320 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:25.320 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.320 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:29:25.320 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:29:25.320 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:29:25.320 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:29:25.320 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:29:25.320 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:29:25.320 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:29:25.321 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:29:25.321 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:29:25.321 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:29:25.321 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:29:25.321 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:29:25.321 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:29:25.321 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:29:25.321 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:29:25.321 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:29:25.321 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:29:25.321 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:29:25.322 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:29:25.322 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:29:25.322 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:29:25.322 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:29:25.322 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:29:25.322 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:29:25.322 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:29:25.322 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:29:25.322 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:29:25.322 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:29:25.322 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:25.322 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:29:25.323 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.323 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:29:25.323 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.323 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:29:25.323 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:29:25.323 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:29:25.323 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:29:25.323 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.323 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:29:25.323 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:29:25.323 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:25.323 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.324 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.324 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:29:25.324 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:29:25.324 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:25.324 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:25.324 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:25.324 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.324 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:29:25.324 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:29:25.324 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:29:25.324 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:29:25.324 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:29:25.325 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.325 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.325 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:29:25.325 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:29:25.325 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:29:25.325 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:29:25.325 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.325 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:29:25.325 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:29:25.325 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:25.325 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.326 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.326 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:29:25.326 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:29:25.326 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:25.326 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:25.326 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:25.326 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.326 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:29:25.326 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:29:25.326 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:29:25.326 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:29:25.326 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:29:25.327 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.327 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.327 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:29:25.327 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:29:25.327 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:29:25.327 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:29:25.327 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.327 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:29:25.327 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:29:25.327 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:25.327 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.327 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.328 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:29:25.328 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:29:25.328 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:25.328 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:25.328 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:25.328 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.328 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:29:25.328 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:29:25.328 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:29:25.328 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:29:25.328 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:29:25.328 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:25.329 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:25.329 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.329 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:29:25.329 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:29:25.329 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.329 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:29:25.329 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:29:25.329 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:29:25.329 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:25.329 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:25.329 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:29:25.330 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.330 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:29:25.330 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:29:25.330 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:29:25.330 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:25.330 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:25.330 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:29:25.330 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.330 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:29:25.330 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:29:25.330 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:29:25.331 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:25.331 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:25.331 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.331 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:29:25.331 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:29:25.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.294+0000 7f932a12b640 1 -- 172.21.15.12:0/286465303 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f931003e210 msgr2=0x7f93100406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.294+0000 7f932a12b640 1 --2- 172.21.15.12:0/286465303 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f931003e210 0x7f93100406d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9324138c90 tx=0x7f931c006040 comp rx=0 tx=0).stop 2024-09-11T01:29:25.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.294+0000 7f932a12b640 10 monclient: shutdown 2024-09-11T01:29:25.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.294+0000 7f932a12b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:25.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.294+0000 7f932a12b640 1 -- 172.21.15.12:0/286465303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f930c004800 msgr2=0x7f9324120b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.294+0000 7f932a12b640 1 --2- 172.21.15.12:0/286465303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f930c004800 0x7f9324120b70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f931400f8e0 tx=0x7f931400fda0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.294+0000 7f9323fff640 1 -- 172.21.15.12:0/286465303 reap_dead start 2024-09-11T01:29:25.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.294+0000 7f932a12b640 1 -- 172.21.15.12:0/286465303 shutdown_connections 2024-09-11T01:29:25.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.294+0000 7f932a12b640 1 --2- 172.21.15.12:0/286465303 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f930c004800 0x7f9324120b70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.294+0000 7f932a12b640 1 -- 172.21.15.12:0/286465303 >> 172.21.15.12:0/286465303 conn(0x7f93240a7020 msgr2=0x7f93240a3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:25.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.294+0000 7f932a12b640 1 -- 172.21.15.12:0/286465303 shutdown_connections 2024-09-11T01:29:25.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.294+0000 7f932a12b640 1 -- 172.21.15.12:0/286465303 wait complete. 2024-09-11T01:29:25.334 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:29:25.334 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:29:25.334 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:29:25.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.435+0000 7f7e28447640 10 monclient: get_monmap_and_config 2024-09-11T01:29:25.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.435+0000 7f7e28447640 10 monclient: build_initial_monmap 2024-09-11T01:29:25.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.435+0000 7f7e28447640 10 monclient: monmap: 2024-09-11T01:29:25.437 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:25.437 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.437 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:25.435908+0000 2024-09-11T01:29:25.437 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:25.435908+0000 2024-09-11T01:29:25.437 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:25.437 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.437 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:25.438 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:25.438 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:25.438 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:25.438 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:25.438 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.435+0000 7f7e28447640 1 Processor -- start 2024-09-11T01:29:25.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.435+0000 7f7e28447640 1 -- start start 2024-09-11T01:29:25.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.435+0000 7f7e28447640 10 monclient: init 2024-09-11T01:29:25.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e28447640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:25.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e28447640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-11T01:29:25.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e28447640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7e20137d50 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:25.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e28447640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7e20138130 0x7f7e20138580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e28447640 10 monclient(hunting): picked mon.noname-a con 0x7f7e20138130 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:25.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e28447640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7e2013d280 0x7f7e2013f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e28447640 10 monclient(hunting): picked mon.noname-c con 0x7f7e2013d280 addr v2:172.21.15.71:3300/0 2024-09-11T01:29:25.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e28447640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e2009d820 con 0x7f7e20137d50 2024-09-11T01:29:25.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e28447640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e28447640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f7e2009c650 con 0x7f7e20138130 2024-09-11T01:29:25.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e28447640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e28447640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f7e2009ba20 con 0x7f7e2013d280 2024-09-11T01:29:25.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e28447640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:25.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e28447640 10 monclient(hunting): authenticate will time out at 771.878499s 2024-09-11T01:29:25.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e259e2640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7e20138130 0x7f7e20138580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e259e2640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7e20138130 0x7f7e20138580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55280/0 (socket says 172.21.15.12:55280) 2024-09-11T01:29:25.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e259e2640 1 -- 172.21.15.12:0/3138922489 learned_addr learned my addr 172.21.15.12:0/3138922489 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:25.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e259e2640 10 monclient(hunting): get_auth_request con 0x7f7e20138130 auth_method 0 2024-09-11T01:29:25.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e259e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:25.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e259e2640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e259e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:25.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e259e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:25.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e259e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:25.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e259e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:25.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e269e4640 1 --2- 172.21.15.12:0/3138922489 >> v2:172.21.15.71:3300/0 conn(0x7f7e2013d280 0x7f7e2013f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e269e4640 10 monclient(hunting): get_auth_request con 0x7f7e2013d280 auth_method 0 2024-09-11T01:29:25.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e269e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:25.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e269e4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e269e4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:25.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e259e2640 10 monclient(hunting): handle_auth_done global_id 4371 payload 274 2024-09-11T01:29:25.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e259e2640 1 -- 172.21.15.12:0/3138922489 >> v2:172.21.15.71:3300/0 conn(0x7f7e2013d280 msgr2=0x7f7e2013f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:25.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e259e2640 1 --2- 172.21.15.12:0/3138922489 >> v2:172.21.15.71:3300/0 conn(0x7f7e2013d280 0x7f7e2013f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e259e2640 1 -- 172.21.15.12:0/3138922489 >> v1:172.21.15.71:6789/0 conn(0x7f7e20137d50 legacy=0x7f7e2009d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:25.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e259e2640 10 monclient: _finish_hunting 0 2024-09-11T01:29:25.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e259e2640 1 monclient: found mon.noname-a 2024-09-11T01:29:25.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e259e2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:25.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e259e2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:25.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e259e2640 1 -- 172.21.15.12:0/3138922489 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e2009c370 con 0x7f7e20138130 2024-09-11T01:29:25.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.436+0000 7f7e269e4640 1 --2- 172.21.15.12:0/3138922489 >> v2:172.21.15.71:3300/0 conn(0x7f7e2013d280 0x7f7e2013f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:29:25.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.439+0000 7f7e259e2640 1 -- 172.21.15.12:0/3138922489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e20138130 msgr2=0x7f7e20138580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:25.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.439+0000 7f7e259e2640 1 --2- 172.21.15.12:0/3138922489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e20138130 0x7f7e20138580 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7e2006fb70 tx=0x7f7e1002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9244a70f69918f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:25.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.439+0000 7f7e251e1640 1 -- 172.21.15.12:0/3138922489 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e10037070 con 0x7f7e20138130 2024-09-11T01:29:25.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.439+0000 7f7e251e1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:25.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.439+0000 7f7e251e1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:25.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.440+0000 7f7e251e1640 10 monclient: dump: 2024-09-11T01:29:25.449 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:25.449 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.450 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.450 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.450 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:25.450 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.450 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:25.450 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:25.450 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:25.450 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.440+0000 7f7e251e1640 10 monclient: _finish_auth 0 2024-09-11T01:29:25.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.440+0000 7f7e251e1640 10 monclient: _check_auth_tickets 2024-09-11T01:29:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.440+0000 7f7e251e1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.440+0000 7f7e28447640 5 monclient: authenticate success, global_id 4371 2024-09-11T01:29:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.440+0000 7f7e251e1640 1 -- 172.21.15.12:0/3138922489 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e10032640 con 0x7f7e20138130 2024-09-11T01:29:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.440+0000 7f7e28447640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.440+0000 7f7e251e1640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.440+0000 7f7e28447640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.440+0000 7f7e251e1640 1 -- 172.21.15.12:0/3138922489 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e10032a10 con 0x7f7e20138130 2024-09-11T01:29:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.440+0000 7f7e251e1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.440+0000 7f7e28447640 4 set_mon_vals no callback set 2024-09-11T01:29:25.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.440+0000 7f7e251e1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.440+0000 7f7e251e1640 10 monclient: dump: 2024-09-11T01:29:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:25.452 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.440+0000 7f7e28447640 10 monclient: shutdown 2024-09-11T01:29:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.440+0000 7f7e28447640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.440+0000 7f7e28447640 1 -- 172.21.15.12:0/3138922489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e20138130 msgr2=0x7f7e20138580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.440+0000 7f7e28447640 1 --2- 172.21.15.12:0/3138922489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e20138130 0x7f7e20138580 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7e2006fb70 tx=0x7f7e1002a140 comp rx=0 tx=0).stop 2024-09-11T01:29:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.440+0000 7f7e28447640 1 -- 172.21.15.12:0/3138922489 shutdown_connections 2024-09-11T01:29:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.440+0000 7f7e28447640 1 --2- 172.21.15.12:0/3138922489 >> v2:172.21.15.71:3300/0 conn(0x7f7e2013d280 0x7f7e2013f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.440+0000 7f7e28447640 1 --2- 172.21.15.12:0/3138922489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e20138130 0x7f7e20138580 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.440+0000 7f7e28447640 1 -- 172.21.15.12:0/3138922489 >> 172.21.15.12:0/3138922489 conn(0x7f7e200a7020 msgr2=0x7f7e200a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.440+0000 7f7e28447640 1 -- 172.21.15.12:0/3138922489 shutdown_connections 2024-09-11T01:29:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.440+0000 7f7e28447640 1 -- 172.21.15.12:0/3138922489 wait complete. 2024-09-11T01:29:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.441+0000 7f7e28447640 10 monclient: build_initial_monmap 2024-09-11T01:29:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.441+0000 7f7e28447640 10 monclient: monmap: 2024-09-11T01:29:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:25.453 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:25.454 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:25.454 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:25.454 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:25.454 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.454 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:25.454 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:25.454 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:25.454 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.441+0000 7f7e28447640 1 Processor -- start 2024-09-11T01:29:25.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.441+0000 7f7e28447640 1 -- start start 2024-09-11T01:29:25.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.441+0000 7f7e28447640 10 monclient: init 2024-09-11T01:29:25.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e28447640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:25.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e28447640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:29:25.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e28447640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e20137d50 0x7f7e201cce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e28447640 10 monclient(hunting): picked mon.noname-c con 0x7f7e20137d50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:25.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e28447640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e20138130 0x7f7e20136f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e28447640 10 monclient(hunting): picked mon.noname-b con 0x7f7e20138130 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:25.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e28447640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e2013d280 0x7f7e2009e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e28447640 10 monclient(hunting): picked mon.noname-a con 0x7f7e2013d280 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:25.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e28447640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e28447640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7e2006f850 con 0x7f7e2013d280 2024-09-11T01:29:25.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e28447640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e28447640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f7e2006f6d0 con 0x7f7e20138130 2024-09-11T01:29:25.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e28447640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e28447640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7e2013fbb0 con 0x7f7e20137d50 2024-09-11T01:29:25.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e28447640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:25.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e28447640 10 monclient(hunting): authenticate will time out at 771.884162s 2024-09-11T01:29:25.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e269e4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e2013d280 0x7f7e2009e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e261e3640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e20137d50 0x7f7e201cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e269e4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e2013d280 0x7f7e2009e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55288/0 (socket says 172.21.15.12:55288) 2024-09-11T01:29:25.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e269e4640 1 -- 172.21.15.12:0/289631118 learned_addr learned my addr 172.21.15.12:0/289631118 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:25.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e269e4640 10 monclient(hunting): get_auth_request con 0x7f7e2013d280 auth_method 0 2024-09-11T01:29:25.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e269e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:25.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e269e4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e269e4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:25.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e261e3640 10 monclient(hunting): get_auth_request con 0x7f7e20137d50 auth_method 0 2024-09-11T01:29:25.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e261e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:25.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e261e3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e261e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e269e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e269e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e269e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e261e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e261e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e261e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e259e2640 1 --2- 172.21.15.12:0/289631118 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e20138130 0x7f7e20136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e259e2640 10 monclient(hunting): get_auth_request con 0x7f7e20138130 auth_method 0 2024-09-11T01:29:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e259e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e259e2640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e259e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e269e4640 10 monclient(hunting): handle_auth_done global_id 4374 payload 824 2024-09-11T01:29:25.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e269e4640 1 -- 172.21.15.12:0/289631118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e20137d50 msgr2=0x7f7e201cce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e269e4640 1 --2- 172.21.15.12:0/289631118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e20137d50 0x7f7e201cce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e269e4640 1 -- 172.21.15.12:0/289631118 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e20138130 msgr2=0x7f7e20136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e269e4640 1 --2- 172.21.15.12:0/289631118 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e20138130 0x7f7e20136f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e269e4640 10 monclient: _finish_hunting 0 2024-09-11T01:29:25.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e269e4640 1 monclient: found mon.noname-a 2024-09-11T01:29:25.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e269e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:25.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e269e4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:25.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.442+0000 7f7e269e4640 1 -- 172.21.15.12:0/289631118 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e2013f830 con 0x7f7e2013d280 2024-09-11T01:29:25.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.444+0000 7f7e269e4640 1 -- 172.21.15.12:0/289631118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e2013d280 msgr2=0x7f7e2009e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:25.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.444+0000 7f7e269e4640 1 --2- 172.21.15.12:0/289631118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e2013d280 0x7f7e2009e2e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7e18007ef0 tx=0x7f7e1800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:25.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.444+0000 7f7e16ffd640 1 -- 172.21.15.12:0/289631118 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1801a070 con 0x7f7e2013d280 2024-09-11T01:29:25.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.444+0000 7f7e16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:25.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.444+0000 7f7e16ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:25.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.444+0000 7f7e16ffd640 10 monclient: dump: 2024-09-11T01:29:25.459 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:25.459 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.459 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.459 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.459 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:25.459 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.459 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:25.459 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:25.459 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:25.459 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.444+0000 7f7e16ffd640 10 monclient: _finish_auth 0 2024-09-11T01:29:25.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.444+0000 7f7e16ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:25.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.444+0000 7f7e16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:25.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.444+0000 7f7e16ffd640 1 -- 172.21.15.12:0/289631118 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e18012760 con 0x7f7e2013d280 2024-09-11T01:29:25.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.444+0000 7f7e28447640 5 monclient: authenticate success, global_id 4374 2024-09-11T01:29:25.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.444+0000 7f7e28447640 10 monclient: _renew_subs 2024-09-11T01:29:25.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.444+0000 7f7e28447640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:25.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.444+0000 7f7e28447640 1 -- 172.21.15.12:0/289631118 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e200a1710 con 0x7f7e2013d280 2024-09-11T01:29:25.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.444+0000 7f7e16ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:25.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.444+0000 7f7e16ffd640 1 -- 172.21.15.12:0/289631118 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e18012b50 con 0x7f7e2013d280 2024-09-11T01:29:25.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.444+0000 7f7e16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:25.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.445+0000 7f7e16ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:25.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.445+0000 7f7e16ffd640 10 monclient: dump: 2024-09-11T01:29:25.460 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:25.460 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.460 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.461 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.461 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:25.461 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.461 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:25.461 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:25.461 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:25.461 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.444+0000 7f7e249e0640 4 set_mon_vals no callback set 2024-09-11T01:29:25.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.445+0000 7f7e28447640 10 monclient: _renew_subs 2024-09-11T01:29:25.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.445+0000 7f7e28447640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:25.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.445+0000 7f7e28447640 1 -- 172.21.15.12:0/289631118 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e20121030 con 0x7f7e2013d280 2024-09-11T01:29:25.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.445+0000 7f7e28447640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:25.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.445+0000 7f7e28447640 1 --2- 172.21.15.12:0/289631118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e20137480 0x7f7e2009e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.445+0000 7f7e28447640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.445+0000 7f7e28447640 1 -- 172.21.15.12:0/289631118 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7e20121550 con 0x7f7e20137480 2024-09-11T01:29:25.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.445+0000 7f7e261e3640 1 --2- 172.21.15.12:0/289631118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e20137480 0x7f7e2009e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.445+0000 7f7e261e3640 10 monclient: get_auth_request con 0x7f7e20137480 auth_method 0 2024-09-11T01:29:25.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.445+0000 7f7e261e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:25.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.445+0000 7f7e261e3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.445+0000 7f7e261e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:25.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.445+0000 7f7e261e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:25.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.445+0000 7f7e261e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:25.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.445+0000 7f7e261e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:25.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.446+0000 7f7e16ffd640 1 -- 172.21.15.12:0/289631118 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f7e1801e3b0 con 0x7f7e2013d280 2024-09-11T01:29:25.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.446+0000 7f7e261e3640 10 monclient(hunting): handle_auth_done global_id 4388 payload 824 2024-09-11T01:29:25.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.446+0000 7f7e261e3640 1 -- 172.21.15.12:0/289631118 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7e20138580 con 0x7f7e20137480 2024-09-11T01:29:25.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.446+0000 7f7e16ffd640 1 --2- 172.21.15.12:0/289631118 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7e0403e240 0x7f7e04040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.446+0000 7f7e16ffd640 1 -- 172.21.15.12:0/289631118 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4374+0+0 (secure 0 0 0) 0x7f7e18053350 con 0x7f7e2013d280 2024-09-11T01:29:25.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.446+0000 7f7e259e2640 1 --2- 172.21.15.12:0/289631118 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7e0403e240 0x7f7e04040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.446+0000 7f7e259e2640 10 monclient: get_auth_request con 0x7f7e0403e240 auth_method 0 2024-09-11T01:29:25.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.448+0000 7f7e261e3640 1 -- 172.21.15.12:0/289631118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e20137480 msgr2=0x7f7e2009e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:25.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.448+0000 7f7e261e3640 1 --2- 172.21.15.12:0/289631118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e20137480 0x7f7e2009e820 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c00f7a0 tx=0x7f7e1c00fc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:25.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.448+0000 7f7e16ffd640 1 -- 172.21.15.12:0/289631118 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c018030 con 0x7f7e20137480 2024-09-11T01:29:25.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.448+0000 7f7e16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:25.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.448+0000 7f7e16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:25.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.448+0000 7f7e16ffd640 10 monclient: dump: 2024-09-11T01:29:25.466 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:25.466 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.466 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.466 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.466 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:25.466 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.466 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:25.466 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:25.466 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:25.466 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.448+0000 7f7e16ffd640 1 -- 172.21.15.12:0/289631118 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7e1c00d9b0 con 0x7f7e20137480 2024-09-11T01:29:25.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.448+0000 7f7e16ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:25.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.448+0000 7f7e16ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:25.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.448+0000 7f7e16ffd640 1 -- 172.21.15.12:0/289631118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e20137480 msgr2=0x7f7e2009e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.448+0000 7f7e16ffd640 1 --2- 172.21.15.12:0/289631118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e20137480 0x7f7e2009e820 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c00f7a0 tx=0x7f7e1c00fc60 comp rx=0 tx=0).stop 2024-09-11T01:29:25.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.448+0000 7f7e16ffd640 1 -- 172.21.15.12:0/289631118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e20137480 msgr2=0x7f7e2009e820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:29:25.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.448+0000 7f7e16ffd640 1 --2- 172.21.15.12:0/289631118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e20137480 0x7f7e2009e820 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c00f7a0 tx=0x7f7e1c00fc60 comp rx=0 tx=0).stop 2024-09-11T01:29:25.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.449+0000 7f7e259e2640 1 --2- 172.21.15.12:0/289631118 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7e0403e240 0x7f7e04040700 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7e2006d4f0 tx=0x7f7e1002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:25.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.454+0000 7f7e28447640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:29:25.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.454+0000 7f7e28447640 1 --2- 172.21.15.12:0/289631118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e2009ed60 0x7f7e200a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.454+0000 7f7e28447640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.454+0000 7f7e28447640 1 -- 172.21.15.12:0/289631118 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7e2009f170 con 0x7f7e2009ed60 2024-09-11T01:29:25.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.455+0000 7f7e261e3640 1 --2- 172.21.15.12:0/289631118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e2009ed60 0x7f7e200a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.455+0000 7f7e261e3640 10 monclient: get_auth_request con 0x7f7e2009ed60 auth_method 0 2024-09-11T01:29:25.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.455+0000 7f7e261e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:25.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.455+0000 7f7e261e3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.455+0000 7f7e261e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:25.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.455+0000 7f7e261e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:25.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.455+0000 7f7e261e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:25.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.455+0000 7f7e261e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:25.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.455+0000 7f7e261e3640 10 monclient(hunting): handle_auth_done global_id 4391 payload 824 2024-09-11T01:29:25.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.455+0000 7f7e261e3640 1 -- 172.21.15.12:0/289631118 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7e20074a40 con 0x7f7e2009ed60 2024-09-11T01:29:25.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.457+0000 7f7e261e3640 1 -- 172.21.15.12:0/289631118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e2009ed60 msgr2=0x7f7e200a18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:25.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.457+0000 7f7e261e3640 1 --2- 172.21.15.12:0/289631118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e2009ed60 0x7f7e200a18e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c00f7a0 tx=0x7f7e1c003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:25.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.457+0000 7f7e16ffd640 1 -- 172.21.15.12:0/289631118 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c01dbe0 con 0x7f7e2009ed60 2024-09-11T01:29:25.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.457+0000 7f7e16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:25.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.457+0000 7f7e16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:25.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.457+0000 7f7e16ffd640 10 monclient: dump: 2024-09-11T01:29:25.469 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:25.469 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.469 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.469 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.469 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:25.469 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.469 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:25.469 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:25.469 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:25.469 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.457+0000 7f7e16ffd640 1 -- 172.21.15.12:0/289631118 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4707 (secure 0 0 0) 0x7f7e1c01dbe0 con 0x7f7e2009ed60 2024-09-11T01:29:25.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.457+0000 7f7e16ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:29:25.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.457+0000 7f7e16ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:25.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.457+0000 7f7e16ffd640 1 -- 172.21.15.12:0/289631118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e2009ed60 msgr2=0x7f7e200a18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.457+0000 7f7e16ffd640 1 --2- 172.21.15.12:0/289631118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e2009ed60 0x7f7e200a18e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c00f7a0 tx=0x7f7e1c003760 comp rx=0 tx=0).stop 2024-09-11T01:29:25.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.457+0000 7f7e16ffd640 1 -- 172.21.15.12:0/289631118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e2009ed60 msgr2=0x7f7e200a18e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:29:25.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.457+0000 7f7e16ffd640 1 --2- 172.21.15.12:0/289631118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e2009ed60 0x7f7e200a18e0 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c00f7a0 tx=0x7f7e1c003760 comp rx=0 tx=0).stop 2024-09-11T01:29:25.470 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:29:25.470 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:29:25.470 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:29:25.470 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:29:25.470 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 8, 2024-09-11T01:29:25.470 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 28146, 2024-09-11T01:29:25.470 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:29:25.471 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:29:25.471 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:29:25.471 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:29:25.471 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:25.471 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 22, 2024-09-11T01:29:25.471 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:29:25.471 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:29:25.471 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:29:25.471 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:29:25.471 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:29:25.471 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:29:25.471 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:29:25.471 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:29:25.472 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:29:25.472 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:29:25.472 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:29:25.472 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:29:25.472 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:29:25.472 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:29:25.472 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:25.472 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:29:25.472 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:29:25.472 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:29:25.472 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:29:25.472 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:29:25.472 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:29:25.473 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:29:25.473 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:29:25.473 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:29:25.473 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:29:25.473 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:29:25.473 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:29:25.473 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:29:25.473 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:25.473 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.473 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:29:25.473 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:29:25.473 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:29:25.474 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:29:25.474 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:29:25.474 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:29:25.474 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:29:25.474 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:29:25.474 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:29:25.474 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:29:25.474 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:29:25.474 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:29:25.474 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:29:25.474 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:29:25.474 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:29:25.474 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:29:25.475 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:29:25.475 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:29:25.475 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:29:25.475 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:29:25.475 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:29:25.475 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:29:25.475 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:29:25.475 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:29:25.475 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:29:25.475 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:29:25.475 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:29:25.475 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:29:25.475 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:25.476 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:29:25.476 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.476 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:29:25.476 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.476 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:29:25.476 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:29:25.476 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:29:25.476 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:29:25.476 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.476 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:29:25.476 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:29:25.476 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:25.476 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.477 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.477 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:29:25.477 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:29:25.477 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:25.477 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:25.477 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:25.477 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.477 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:29:25.477 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:29:25.477 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:29:25.477 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:29:25.477 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:29:25.477 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.478 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.478 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:29:25.478 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:29:25.478 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:29:25.478 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:29:25.478 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.478 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:29:25.478 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:29:25.478 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:25.478 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.478 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.478 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:29:25.478 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:29:25.478 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:25.479 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:25.479 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:25.479 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.479 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:29:25.479 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:29:25.479 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:29:25.479 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:29:25.479 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:29:25.479 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.479 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.479 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:29:25.479 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:29:25.479 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:29:25.480 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:29:25.480 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.480 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:29:25.480 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:29:25.480 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:25.480 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.480 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.480 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:29:25.480 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:29:25.480 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:25.480 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:25.480 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:25.480 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.480 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:29:25.481 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:29:25.481 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:29:25.481 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:29:25.481 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:29:25.481 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:25.481 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:25.481 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.481 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:29:25.481 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:29:25.481 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.481 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:29:25.481 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:29:25.481 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:29:25.482 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:25.482 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:25.482 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:29:25.482 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.482 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:29:25.482 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:29:25.482 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:29:25.482 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:25.482 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:25.482 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:29:25.482 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.482 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:29:25.482 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:29:25.482 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:29:25.483 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:25.483 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:25.483 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.483 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:29:25.483 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:29:25.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.458+0000 7f7e28447640 1 -- 172.21.15.12:0/289631118 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7e0403e240 msgr2=0x7f7e04040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.458+0000 7f7e28447640 1 --2- 172.21.15.12:0/289631118 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7e0403e240 0x7f7e04040700 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7e2006d4f0 tx=0x7f7e1002e000 comp rx=0 tx=0).stop 2024-09-11T01:29:25.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.458+0000 7f7e28447640 10 monclient: shutdown 2024-09-11T01:29:25.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.458+0000 7f7e28447640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:25.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.458+0000 7f7e28447640 1 -- 172.21.15.12:0/289631118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e2013d280 msgr2=0x7f7e2009e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.458+0000 7f7e28447640 1 --2- 172.21.15.12:0/289631118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e2013d280 0x7f7e2009e2e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7e18007ef0 tx=0x7f7e1800d4a0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.458+0000 7f7e269e4640 1 -- 172.21.15.12:0/289631118 reap_dead start 2024-09-11T01:29:25.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.458+0000 7f7e28447640 1 -- 172.21.15.12:0/289631118 shutdown_connections 2024-09-11T01:29:25.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.458+0000 7f7e28447640 1 --2- 172.21.15.12:0/289631118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e2013d280 0x7f7e2009e2e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.458+0000 7f7e28447640 1 -- 172.21.15.12:0/289631118 >> 172.21.15.12:0/289631118 conn(0x7f7e200a7020 msgr2=0x7f7e200a8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:25.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.458+0000 7f7e28447640 1 -- 172.21.15.12:0/289631118 shutdown_connections 2024-09-11T01:29:25.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.458+0000 7f7e28447640 1 -- 172.21.15.12:0/289631118 wait complete. 2024-09-11T01:29:25.485 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:29:25.486 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:29:25.486 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:29:25.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.586+0000 7ff3b8d32640 10 monclient: get_monmap_and_config 2024-09-11T01:29:25.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.586+0000 7ff3b8d32640 10 monclient: build_initial_monmap 2024-09-11T01:29:25.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.586+0000 7ff3b8d32640 10 monclient: monmap: 2024-09-11T01:29:25.588 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:25.588 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.588 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:25.586989+0000 2024-09-11T01:29:25.588 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:25.586989+0000 2024-09-11T01:29:25.588 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:25.588 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.588 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:25.588 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:25.588 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:25.589 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:25.589 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:25.589 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.586+0000 7ff3b8d32640 1 Processor -- start 2024-09-11T01:29:25.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.586+0000 7ff3b8d32640 1 -- start start 2024-09-11T01:29:25.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.586+0000 7ff3b8d32640 10 monclient: init 2024-09-11T01:29:25.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b8d32640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:25.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b8d32640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-11T01:29:25.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b8d32640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff3b4137e50 0x7ff3b409d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b8d32640 10 monclient(hunting): picked mon.noname-a con 0x7ff3b4137e50 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:25.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b8d32640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3b4138230 0x7ff3b4138660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b8d32640 10 monclient(hunting): picked mon.noname-b con 0x7ff3b4138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:25.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b8d32640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff3b4134cb0 0x7ff3b4135120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b8d32640 10 monclient(hunting): picked mon.noname-c con 0x7ff3b4134cb0 addr v2:172.21.15.71:3300/0 2024-09-11T01:29:25.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b8d32640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b8d32640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff3b40a6580 con 0x7ff3b4137e50 2024-09-11T01:29:25.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b8d32640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b8d32640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7ff3b40a62d0 con 0x7ff3b4134cb0 2024-09-11T01:29:25.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b8d32640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b8d32640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff3b409bb40 con 0x7ff3b4138230 2024-09-11T01:29:25.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b8d32640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:25.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b8d32640 10 monclient(hunting): authenticate will time out at 772.029553s 2024-09-11T01:29:25.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b1d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3b4138230 0x7ff3b4138660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b259c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff3b4137e50 0x7ff3b409d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b259c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff3b4137e50 0x7ff3b409d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55304/0 (socket says 172.21.15.12:55304) 2024-09-11T01:29:25.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b1d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3b4138230 0x7ff3b4138660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52006/0 (socket says 172.21.15.12:52006) 2024-09-11T01:29:25.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b259c640 1 -- 172.21.15.12:0/2025624097 learned_addr learned my addr 172.21.15.12:0/2025624097 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:25.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b259c640 10 monclient(hunting): get_auth_request con 0x7ff3b4137e50 auth_method 0 2024-09-11T01:29:25.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:25.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:25.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b1d9b640 10 monclient(hunting): get_auth_request con 0x7ff3b4138230 auth_method 0 2024-09-11T01:29:25.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:25.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:25.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:25.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:25.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:25.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b2d9d640 1 --2- 172.21.15.12:0/2025624097 >> v2:172.21.15.71:3300/0 conn(0x7ff3b4134cb0 0x7ff3b4135120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:25.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:25.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:25.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b2d9d640 10 monclient(hunting): get_auth_request con 0x7ff3b4134cb0 auth_method 0 2024-09-11T01:29:25.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:25.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:25.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:25.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:25.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:25.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b1d9b640 10 monclient(hunting): handle_auth_done global_id 4394 payload 274 2024-09-11T01:29:25.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b1d9b640 1 -- 172.21.15.12:0/2025624097 >> v2:172.21.15.71:3300/0 conn(0x7ff3b4134cb0 msgr2=0x7ff3b4135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:25.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.587+0000 7ff3b1d9b640 1 --2- 172.21.15.12:0/2025624097 >> v2:172.21.15.71:3300/0 conn(0x7ff3b4134cb0 0x7ff3b4135120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.588+0000 7ff3b1d9b640 1 -- 172.21.15.12:0/2025624097 >> v2:172.21.15.12:3300/0 conn(0x7ff3b4137e50 msgr2=0x7ff3b409d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:25.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.588+0000 7ff3b1d9b640 1 --2- 172.21.15.12:0/2025624097 >> v2:172.21.15.12:3300/0 conn(0x7ff3b4137e50 0x7ff3b409d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.588+0000 7ff3b1d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:29:25.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.588+0000 7ff3b1d9b640 1 monclient: found mon.noname-b 2024-09-11T01:29:25.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.588+0000 7ff3b1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:25.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.588+0000 7ff3b1d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:29:25.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.588+0000 7ff3b1d9b640 1 -- 172.21.15.12:0/2025624097 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3b409c4a0 con 0x7ff3b4138230 2024-09-11T01:29:25.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.588+0000 7ff3b259c640 1 --2- 172.21.15.12:0/2025624097 >> v2:172.21.15.12:3300/0 conn(0x7ff3b4137e50 0x7ff3b409d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.590+0000 7ff3b1d9b640 1 -- 172.21.15.12:0/2025624097 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3b4138230 msgr2=0x7ff3b4138660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.590+0000 7ff3b1d9b640 1 --2- 172.21.15.12:0/2025624097 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3b4138230 0x7ff3b4138660 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff3b409db80 tx=0x7ff3a402a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4b4597485eb3fb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.590+0000 7ff3b159a640 1 -- 172.21.15.12:0/2025624097 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a4037070 con 0x7ff3b4138230 2024-09-11T01:29:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b159a640 10 monclient: dump: 2024-09-11T01:29:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:25.598 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:25.598 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b159a640 10 monclient: _finish_auth 0 2024-09-11T01:29:25.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b159a640 10 monclient: _check_auth_tickets 2024-09-11T01:29:25.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:25.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b159a640 1 -- 172.21.15.12:0/2025624097 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3a4032440 con 0x7ff3b4138230 2024-09-11T01:29:25.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b8d32640 5 monclient: authenticate success, global_id 4394 2024-09-11T01:29:25.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b8d32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:25.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b159a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:25.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b8d32640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:25.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b159a640 1 -- 172.21.15.12:0/2025624097 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a4032810 con 0x7ff3b4138230 2024-09-11T01:29:25.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:25.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b8d32640 4 set_mon_vals no callback set 2024-09-11T01:29:25.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:25.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b159a640 10 monclient: dump: 2024-09-11T01:29:25.599 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:25.599 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.599 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.599 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.599 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:25.599 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.599 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:25.599 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:25.599 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:25.600 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b8d32640 10 monclient: shutdown 2024-09-11T01:29:25.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b8d32640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:25.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b8d32640 1 -- 172.21.15.12:0/2025624097 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3b4138230 msgr2=0x7ff3b4138660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b8d32640 1 --2- 172.21.15.12:0/2025624097 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3b4138230 0x7ff3b4138660 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff3b409db80 tx=0x7ff3a402a000 comp rx=0 tx=0).stop 2024-09-11T01:29:25.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b8d32640 1 -- 172.21.15.12:0/2025624097 shutdown_connections 2024-09-11T01:29:25.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b8d32640 1 --2- 172.21.15.12:0/2025624097 >> v2:172.21.15.71:3300/0 conn(0x7ff3b4134cb0 0x7ff3b4135120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b8d32640 1 --2- 172.21.15.12:0/2025624097 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3b4138230 0x7ff3b4138660 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b8d32640 1 --2- 172.21.15.12:0/2025624097 >> v2:172.21.15.12:3300/0 conn(0x7ff3b4137e50 0x7ff3b409d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b8d32640 1 -- 172.21.15.12:0/2025624097 >> 172.21.15.12:0/2025624097 conn(0x7ff3b40a7020 msgr2=0x7ff3b40a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:25.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b8d32640 1 -- 172.21.15.12:0/2025624097 shutdown_connections 2024-09-11T01:29:25.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.591+0000 7ff3b8d32640 1 -- 172.21.15.12:0/2025624097 wait complete. 2024-09-11T01:29:25.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.592+0000 7ff3b8d32640 10 monclient: build_initial_monmap 2024-09-11T01:29:25.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.592+0000 7ff3b8d32640 10 monclient: monmap: 2024-09-11T01:29:25.606 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:25.606 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:25.606 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:25.606 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:25.606 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:25.606 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.606 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:25.606 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:25.607 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:25.607 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.592+0000 7ff3b8d32640 1 Processor -- start 2024-09-11T01:29:25.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.592+0000 7ff3b8d32640 1 -- start start 2024-09-11T01:29:25.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.592+0000 7ff3b8d32640 10 monclient: init 2024-09-11T01:29:25.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b8d32640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:25.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b8d32640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:29:25.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b8d32640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3b4134cb0 0x7ff3b4133cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b8d32640 10 monclient(hunting): picked mon.noname-b con 0x7ff3b4134cb0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:25.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b8d32640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3b4137e50 0x7ff3b4134210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b8d32640 10 monclient(hunting): picked mon.noname-a con 0x7ff3b4137e50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:25.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b8d32640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3b4138230 0x7ff3b41218b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b8d32640 10 monclient(hunting): picked mon.noname-c con 0x7ff3b4138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:25.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b8d32640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b8d32640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b406f710 con 0x7ff3b4137e50 2024-09-11T01:29:25.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b8d32640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b8d32640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b4138ba0 con 0x7ff3b4134cb0 2024-09-11T01:29:25.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b8d32640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b8d32640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff3b406f890 con 0x7ff3b4138230 2024-09-11T01:29:25.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b8d32640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:25.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b8d32640 10 monclient(hunting): authenticate will time out at 772.035395s 2024-09-11T01:29:25.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b2d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3b4138230 0x7ff3b41218b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b1d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3b4137e50 0x7ff3b4134210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b1d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3b4137e50 0x7ff3b4134210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55320/0 (socket says 172.21.15.12:55320) 2024-09-11T01:29:25.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b2d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3b4138230 0x7ff3b41218b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52022/0 (socket says 172.21.15.12:52022) 2024-09-11T01:29:25.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b1d9b640 1 -- 172.21.15.12:0/2513567311 learned_addr learned my addr 172.21.15.12:0/2513567311 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:25.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b1d9b640 10 monclient(hunting): get_auth_request con 0x7ff3b4137e50 auth_method 0 2024-09-11T01:29:25.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:25.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:25.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b2d9d640 10 monclient(hunting): get_auth_request con 0x7ff3b4138230 auth_method 0 2024-09-11T01:29:25.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:25.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:25.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:25.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:25.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:25.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:25.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:25.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:25.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b259c640 1 --2- 172.21.15.12:0/2513567311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3b4134cb0 0x7ff3b4133cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b259c640 10 monclient(hunting): get_auth_request con 0x7ff3b4134cb0 auth_method 0 2024-09-11T01:29:25.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:25.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:25.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b1d9b640 10 monclient(hunting): handle_auth_done global_id 4380 payload 824 2024-09-11T01:29:25.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b1d9b640 1 -- 172.21.15.12:0/2513567311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3b4138230 msgr2=0x7ff3b41218b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b1d9b640 1 --2- 172.21.15.12:0/2513567311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3b4138230 0x7ff3b41218b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b1d9b640 1 -- 172.21.15.12:0/2513567311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3b4134cb0 msgr2=0x7ff3b4133cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b1d9b640 1 --2- 172.21.15.12:0/2513567311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3b4134cb0 0x7ff3b4133cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b1d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:29:25.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b1d9b640 1 monclient: found mon.noname-a 2024-09-11T01:29:25.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:25.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:29:25.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b1d9b640 1 -- 172.21.15.12:0/2513567311 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3b41ce0f0 con 0x7ff3b4137e50 2024-09-11T01:29:25.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b259c640 1 --2- 172.21.15.12:0/2513567311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3b4134cb0 0x7ff3b4133cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:29:25.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.593+0000 7ff3b2d9d640 1 --2- 172.21.15.12:0/2513567311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3b4138230 0x7ff3b41218b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:25.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.596+0000 7ff3b1d9b640 1 -- 172.21.15.12:0/2513567311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3b4137e50 msgr2=0x7ff3b4134210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:25.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.596+0000 7ff3b1d9b640 1 --2- 172.21.15.12:0/2513567311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3b4137e50 0x7ff3b4134210 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4000c00 tx=0x7ff3a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:25.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.596+0000 7ff3aaffd640 1 -- 172.21.15.12:0/2513567311 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a4041070 con 0x7ff3b4137e50 2024-09-11T01:29:25.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.596+0000 7ff3aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:25.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.596+0000 7ff3aaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:29:25.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.596+0000 7ff3aaffd640 10 monclient: dump: 2024-09-11T01:29:25.612 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:25.612 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.612 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.612 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.612 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:25.612 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.612 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:25.612 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:25.612 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:25.612 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.596+0000 7ff3aaffd640 10 monclient: _finish_auth 0 2024-09-11T01:29:25.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.596+0000 7ff3aaffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:25.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.596+0000 7ff3aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:25.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.596+0000 7ff3aaffd640 1 -- 172.21.15.12:0/2513567311 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3a403c890 con 0x7ff3b4137e50 2024-09-11T01:29:25.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.596+0000 7ff3aaffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:25.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.596+0000 7ff3aaffd640 1 -- 172.21.15.12:0/2513567311 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a4037040 con 0x7ff3b4137e50 2024-09-11T01:29:25.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.596+0000 7ff3b8d32640 5 monclient: authenticate success, global_id 4380 2024-09-11T01:29:25.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.596+0000 7ff3b8d32640 10 monclient: _renew_subs 2024-09-11T01:29:25.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.596+0000 7ff3b8d32640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:25.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.596+0000 7ff3b8d32640 1 -- 172.21.15.12:0/2513567311 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3b4134750 con 0x7ff3b4137e50 2024-09-11T01:29:25.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.596+0000 7ff3b0d99640 4 set_mon_vals no callback set 2024-09-11T01:29:25.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.596+0000 7ff3aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:25.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.596+0000 7ff3aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:29:25.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.596+0000 7ff3aaffd640 10 monclient: dump: 2024-09-11T01:29:25.613 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:25.614 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.614 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.614 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.614 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:25.614 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.614 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:25.614 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:25.614 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:25.614 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.596+0000 7ff3b8d32640 10 monclient: _renew_subs 2024-09-11T01:29:25.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.596+0000 7ff3b8d32640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:29:25.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.596+0000 7ff3b8d32640 1 -- 172.21.15.12:0/2513567311 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3b41233c0 con 0x7ff3b4137e50 2024-09-11T01:29:25.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.597+0000 7ff3b8d32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:25.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.597+0000 7ff3b8d32640 1 --2- 172.21.15.12:0/2513567311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3b41cd440 0x7ff3b4121df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.597+0000 7ff3b8d32640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.597+0000 7ff3b8d32640 1 -- 172.21.15.12:0/2513567311 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b406a060 con 0x7ff3b41cd440 2024-09-11T01:29:25.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.597+0000 7ff3aaffd640 1 -- 172.21.15.12:0/2513567311 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff3a403ccc0 con 0x7ff3b4137e50 2024-09-11T01:29:25.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.597+0000 7ff3b259c640 1 --2- 172.21.15.12:0/2513567311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3b41cd440 0x7ff3b4121df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.597+0000 7ff3b259c640 10 monclient: get_auth_request con 0x7ff3b41cd440 auth_method 0 2024-09-11T01:29:25.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.597+0000 7ff3b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:25.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.597+0000 7ff3b259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.597+0000 7ff3b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:25.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.597+0000 7ff3aaffd640 1 --2- 172.21.15.12:0/2513567311 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff39403e210 0x7ff3940406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.597+0000 7ff3b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:25.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.597+0000 7ff3b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:25.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.597+0000 7ff3b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:25.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.597+0000 7ff3aaffd640 1 -- 172.21.15.12:0/2513567311 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4374+0+0 (secure 0 0 0) 0x7ff3a40734c0 con 0x7ff3b4137e50 2024-09-11T01:29:25.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.597+0000 7ff3b259c640 10 monclient(hunting): handle_auth_done global_id 4321 payload 824 2024-09-11T01:29:25.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.597+0000 7ff3b259c640 1 -- 172.21.15.12:0/2513567311 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff3b41357d0 con 0x7ff3b41cd440 2024-09-11T01:29:25.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.597+0000 7ff3b2d9d640 1 --2- 172.21.15.12:0/2513567311 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff39403e210 0x7ff3940406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.597+0000 7ff3b2d9d640 10 monclient: get_auth_request con 0x7ff39403e210 auth_method 0 2024-09-11T01:29:25.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.600+0000 7ff3b259c640 1 -- 172.21.15.12:0/2513567311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3b41cd440 msgr2=0x7ff3b4121df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:25.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.600+0000 7ff3b259c640 1 --2- 172.21.15.12:0/2513567311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3b41cd440 0x7ff3b4121df0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff39c0115d0 tx=0x7ff39c011a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:25.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.600+0000 7ff3aaffd640 1 -- 172.21.15.12:0/2513567311 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff39c00fc50 con 0x7ff3b41cd440 2024-09-11T01:29:25.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.600+0000 7ff3aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:25.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.600+0000 7ff3aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:25.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.600+0000 7ff3aaffd640 10 monclient: dump: 2024-09-11T01:29:25.616 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:25.616 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.616 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.616 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.617 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:25.617 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.617 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:25.617 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:25.617 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:25.617 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.600+0000 7ff3b2d9d640 1 --2- 172.21.15.12:0/2513567311 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff39403e210 0x7ff3940406d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac009fd0 tx=0x7ff3ac00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:25.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.605+0000 7ff3aaffd640 1 -- 172.21.15.12:0/2513567311 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff39c00fc50 con 0x7ff3b41cd440 2024-09-11T01:29:25.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.605+0000 7ff3aaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:25.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.605+0000 7ff3aaffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:25.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.605+0000 7ff3aaffd640 1 -- 172.21.15.12:0/2513567311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3b41cd440 msgr2=0x7ff3b4121df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.605+0000 7ff3aaffd640 1 --2- 172.21.15.12:0/2513567311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3b41cd440 0x7ff3b4121df0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff39c0115d0 tx=0x7ff39c011a90 comp rx=0 tx=0).stop 2024-09-11T01:29:25.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.605+0000 7ff3aaffd640 1 -- 172.21.15.12:0/2513567311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3b41cd440 msgr2=0x7ff3b4121df0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:29:25.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.605+0000 7ff3aaffd640 1 --2- 172.21.15.12:0/2513567311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3b41cd440 0x7ff3b4121df0 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff39c0115d0 tx=0x7ff39c011a90 comp rx=0 tx=0).stop 2024-09-11T01:29:25.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.605+0000 7ff050ff9640 1 -- 172.21.15.12:0/3421807171 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]=0 pool 'unique_pool_0' created v18) ==== 130+0+0 (secure 0 0 0) 0x7ff048050e80 con 0x7ff0541382a0 2024-09-11T01:29:25.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.605+0000 7ff050ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}] 2024-09-11T01:29:25.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.605+0000 7ff050ff9640 10 monclient: _finish_command 2 = system:0 pool 'unique_pool_0' created 2024-09-11T01:29:25.619 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'unique_pool_0' created 2024-09-11T01:29:25.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.609+0000 7ff05a4e6640 1 -- 172.21.15.12:0/3421807171 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff04003e210 msgr2=0x7ff0400406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.609+0000 7ff05a4e6640 1 --2- 172.21.15.12:0/3421807171 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff04003e210 0x7ff0400406d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff0540a7010 tx=0x7ff04402e000 comp rx=0 tx=0).stop 2024-09-11T01:29:25.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.609+0000 7ff05a4e6640 10 monclient: shutdown 2024-09-11T01:29:25.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.609+0000 7ff05a4e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:25.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.609+0000 7ff05a4e6640 1 -- 172.21.15.12:0/3421807171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0541382a0 msgr2=0x7ff054136d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.609+0000 7ff05a4e6640 1 --2- 172.21.15.12:0/3421807171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0541382a0 0x7ff054136d40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff04800c480 tx=0x7ff04800c940 comp rx=0 tx=0).stop 2024-09-11T01:29:25.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.609+0000 7ff05a4e6640 1 -- 172.21.15.12:0/3421807171 shutdown_connections 2024-09-11T01:29:25.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.609+0000 7ff05a4e6640 1 --2- 172.21.15.12:0/3421807171 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff05413d460 0x7ff054139440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.609+0000 7ff05a4e6640 1 --2- 172.21.15.12:0/3421807171 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff04003e210 0x7ff0400406d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.609+0000 7ff05a4e6640 1 --2- 172.21.15.12:0/3421807171 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0541382a0 0x7ff054136d40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.609+0000 7ff05a4e6640 1 --2- 172.21.15.12:0/3421807171 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff054137ec0 0x7ff05413b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.609+0000 7ff05a4e6640 1 -- 172.21.15.12:0/3421807171 >> 172.21.15.12:0/3421807171 conn(0x7ff05406ec70 msgr2=0x7ff0540a8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:25.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.609+0000 7ff05a4e6640 1 -- 172.21.15.12:0/3421807171 shutdown_connections 2024-09-11T01:29:25.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.609+0000 7ff05a4e6640 1 -- 172.21.15.12:0/3421807171 wait complete. 2024-09-11T01:29:25.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.615+0000 7ff3b8d32640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:29:25.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.615+0000 7ff3b8d32640 1 --2- 172.21.15.12:0/2513567311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3b4123ed0 0x7ff3b4122330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.615+0000 7ff3b8d32640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.615+0000 7ff3b8d32640 1 -- 172.21.15.12:0/2513567311 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b40a28f0 con 0x7ff3b4123ed0 2024-09-11T01:29:25.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.615+0000 7ff3b259c640 1 --2- 172.21.15.12:0/2513567311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3b4123ed0 0x7ff3b4122330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.615+0000 7ff3b259c640 10 monclient: get_auth_request con 0x7ff3b4123ed0 auth_method 0 2024-09-11T01:29:25.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.615+0000 7ff3b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:25.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.615+0000 7ff3b259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.615+0000 7ff3b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:25.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.616+0000 7ff3b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:25.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.616+0000 7ff3b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:25.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.616+0000 7ff3b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:25.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.616+0000 7ff3b259c640 10 monclient(hunting): handle_auth_done global_id 4324 payload 824 2024-09-11T01:29:25.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.616+0000 7ff3b259c640 1 -- 172.21.15.12:0/2513567311 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff3b4069950 con 0x7ff3b4123ed0 2024-09-11T01:29:25.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.618+0000 7ff3b259c640 1 -- 172.21.15.12:0/2513567311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3b4123ed0 msgr2=0x7ff3b4122330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:25.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.618+0000 7ff3b259c640 1 --2- 172.21.15.12:0/2513567311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3b4123ed0 0x7ff3b4122330 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff39c0115d0 tx=0x7ff39c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:25.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.619+0000 7ff3aaffd640 1 -- 172.21.15.12:0/2513567311 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff39c02b020 con 0x7ff3b4123ed0 2024-09-11T01:29:25.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.619+0000 7ff3aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:25.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.619+0000 7ff3aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:25.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.619+0000 7ff3aaffd640 10 monclient: dump: 2024-09-11T01:29:25.623 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:25.624 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.619+0000 7ff3aaffd640 1 -- 172.21.15.12:0/2513567311 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4874 (secure 0 0 0) 0x7ff39c015460 con 0x7ff3b4123ed0 2024-09-11T01:29:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.619+0000 7ff3aaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:29:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.619+0000 7ff3aaffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:25.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.619+0000 7ff3aaffd640 1 -- 172.21.15.12:0/2513567311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3b4123ed0 msgr2=0x7ff3b4122330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.619+0000 7ff3aaffd640 1 --2- 172.21.15.12:0/2513567311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3b4123ed0 0x7ff3b4122330 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff39c0115d0 tx=0x7ff39c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.619+0000 7ff3aaffd640 1 -- 172.21.15.12:0/2513567311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3b4123ed0 msgr2=0x7ff3b4122330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:29:25.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.619+0000 7ff3aaffd640 1 --2- 172.21.15.12:0/2513567311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3b4123ed0 0x7ff3b4122330 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff39c0115d0 tx=0x7ff39c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.625 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:29:25.625 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:29:25.625 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:29:25.625 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:29:25.625 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 8, 2024-09-11T01:29:25.625 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 28273, 2024-09-11T01:29:25.625 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:29:25.626 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:29:25.626 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:29:25.626 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:29:25.626 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:25.626 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 23, 2024-09-11T01:29:25.626 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:29:25.626 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:29:25.626 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:29:25.626 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:29:25.626 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:29:25.626 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:29:25.626 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:29:25.626 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:29:25.626 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:29:25.627 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:29:25.627 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:29:25.627 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:29:25.627 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:29:25.627 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:29:25.627 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:25.627 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:29:25.627 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:29:25.627 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:29:25.627 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:29:25.627 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:29:25.627 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:29:25.627 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:29:25.627 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:29:25.628 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:29:25.628 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:29:25.628 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:29:25.628 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:29:25.628 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:29:25.628 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:25.628 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.628 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:29:25.628 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:29:25.628 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:29:25.628 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:29:25.628 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:29:25.628 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:29:25.629 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:29:25.629 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:29:25.629 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:29:25.629 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:29:25.629 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:29:25.629 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:29:25.629 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:29:25.629 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:29:25.629 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:29:25.629 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:29:25.629 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:29:25.629 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:29:25.629 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:29:25.629 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:29:25.630 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:29:25.630 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:29:25.630 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:29:25.630 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:29:25.630 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:29:25.630 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:29:25.630 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:29:25.630 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:29:25.630 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:25.630 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:29:25.630 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.630 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:29:25.630 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.630 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:29:25.631 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:29:25.631 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:29:25.631 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:29:25.631 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.631 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:29:25.631 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:29:25.631 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:25.631 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.631 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.631 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:29:25.631 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:29:25.631 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:25.631 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:25.632 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:25.632 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.632 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:29:25.632 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:29:25.632 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:29:25.632 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:29:25.632 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:29:25.632 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.632 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.632 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:29:25.632 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:29:25.632 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:29:25.633 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:29:25.633 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.633 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:29:25.633 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:29:25.633 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:25.633 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.633 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.633 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:29:25.633 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:29:25.633 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:25.633 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:25.633 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:25.633 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.634 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:29:25.634 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:29:25.634 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:29:25.634 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:29:25.634 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:29:25.634 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.634 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.634 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:29:25.634 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:29:25.634 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:29:25.634 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:29:25.634 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.634 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:29:25.634 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:29:25.634 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:25.635 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.635 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.635 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:29:25.635 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:29:25.635 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:25.635 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:25.635 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:25.635 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.635 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:29:25.635 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:29:25.635 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:29:25.635 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:29:25.635 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:29:25.636 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:25.636 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:25.636 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.636 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:29:25.636 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:29:25.636 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.636 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:29:25.636 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:29:25.636 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:29:25.636 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:25.636 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:25.636 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:29:25.636 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.636 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:29:25.637 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:29:25.637 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:29:25.637 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:25.637 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:25.637 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:29:25.637 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.637 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:29:25.637 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:29:25.637 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:29:25.637 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:25.637 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:25.637 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:29:25.637 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:25.637 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:29:25.638 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:29:25.638 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:29:25.638 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:25.638 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:25.638 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:25.638 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:29:25.638 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:29:25.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.620+0000 7ff3b8d32640 1 -- 172.21.15.12:0/2513567311 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff39403e210 msgr2=0x7ff3940406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.620+0000 7ff3b8d32640 1 --2- 172.21.15.12:0/2513567311 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff39403e210 0x7ff3940406d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac009fd0 tx=0x7ff3ac00b040 comp rx=0 tx=0).stop 2024-09-11T01:29:25.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.620+0000 7ff3b8d32640 10 monclient: shutdown 2024-09-11T01:29:25.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.620+0000 7ff3b8d32640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:25.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.620+0000 7ff3b8d32640 1 -- 172.21.15.12:0/2513567311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3b4137e50 msgr2=0x7ff3b4134210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.620+0000 7ff3b8d32640 1 --2- 172.21.15.12:0/2513567311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3b4137e50 0x7ff3b4134210 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4000c00 tx=0x7ff3a40046c0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.620+0000 7ff3b2d9d640 1 -- 172.21.15.12:0/2513567311 reap_dead start 2024-09-11T01:29:25.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.620+0000 7ff3b2d9d640 1 -- 172.21.15.12:0/2513567311 reap_dead start 2024-09-11T01:29:25.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.620+0000 7ff3b8d32640 1 -- 172.21.15.12:0/2513567311 shutdown_connections 2024-09-11T01:29:25.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.620+0000 7ff3b8d32640 1 -- 172.21.15.12:0/2513567311 >> 172.21.15.12:0/2513567311 conn(0x7ff3b40a7020 msgr2=0x7ff3b40a0430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:25.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.620+0000 7ff3b8d32640 1 -- 172.21.15.12:0/2513567311 shutdown_connections 2024-09-11T01:29:25.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.620+0000 7ff3b8d32640 1 -- 172.21.15.12:0/2513567311 wait complete. 2024-09-11T01:29:25.641 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-11T01:29:25.642 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:29:25.642 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-11T01:29:25.642 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-11T01:29:25.642 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-11T01:29:25.642 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-11T01:29:25.643 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T01:29:25.643 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:29:25.695 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T01:29:25.695 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-11T01:29:25.695 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-11T01:29:25.695 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T01:29:25.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.787+0000 7f10611a0640 10 monclient: get_monmap_and_config 2024-09-11T01:29:25.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.787+0000 7f10611a0640 10 monclient: build_initial_monmap 2024-09-11T01:29:25.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.787+0000 7f10611a0640 10 monclient: monmap: 2024-09-11T01:29:25.789 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:25.789 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.789 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:25.788256+0000 2024-09-11T01:29:25.789 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:25.788256+0000 2024-09-11T01:29:25.789 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:25.790 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.790 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:25.790 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:25.790 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:25.790 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:25.790 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:25.790 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.788+0000 7f10611a0640 1 Processor -- start 2024-09-11T01:29:25.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.788+0000 7f10611a0640 1 -- start start 2024-09-11T01:29:25.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.788+0000 7f10611a0640 10 monclient: init 2024-09-11T01:29:25.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.788+0000 7f10611a0640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:25.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.788+0000 7f10611a0640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-11T01:29:25.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.788+0000 7f10611a0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f105c0a5020 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:25.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.788+0000 7f10611a0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f105c0a5b10 0x7f105c09d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.788+0000 7f10611a0640 10 monclient(hunting): picked mon.noname-b con 0x7f105c0a5b10 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:25.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.788+0000 7f10611a0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f105c13d230 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:25.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.788+0000 7f10611a0640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f105c0a6b50 con 0x7f105c13d230 2024-09-11T01:29:25.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.788+0000 7f10611a0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f105c0a6910 con 0x7f105c0a5020 2024-09-11T01:29:25.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.788+0000 7f10611a0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.788+0000 7f10611a0640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f105c09d8e0 con 0x7f105c0a5b10 2024-09-11T01:29:25.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.788+0000 7f10611a0640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:25.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.788+0000 7f10611a0640 10 monclient(hunting): authenticate will time out at 772.230750s 2024-09-11T01:29:25.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.788+0000 7f105b59e640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7f105c13d230 legacy=0x7f105c13f620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:29:25.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.789+0000 7f105ad9d640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f105c0a5020 0x7f105c0a5400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.12:45604/0 (socket says 172.21.15.12:45604) 2024-09-11T01:29:25.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.789+0000 7f105ad9d640 1 -- 172.21.15.12:0/119862313 learned_addr learned my addr 172.21.15.12:0/119862313 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:29:25.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105ad9d640 1 -- 172.21.15.12:0/119862313 >> v1:172.21.15.71:6789/0 conn(0x7f105c0a5020 legacy=0x7f105c0a5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:25.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 1 -- 172.21.15.12:0/119862313 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3954303852 0 0) 0x7f105c0a6910 con 0x7f105c0a5020 2024-09-11T01:29:25.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 10 monclient(hunting): _negotiate 2024-09-11T01:29:25.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:25.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 10 monclient(hunting): my global_id is 4327 2024-09-11T01:29:25.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 1 -- 172.21.15.12:0/119862313 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1048003520 con 0x7f105c0a5020 2024-09-11T01:29:25.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 1 -- 172.21.15.12:0/119862313 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 729243631 0 0) 0x7f1048003520 con 0x7f105c0a5020 2024-09-11T01:29:25.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 1 -- 172.21.15.12:0/119862313 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f105c0a5b10 msgr2=0x7f105c09d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:25.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 1 --2- 172.21.15.12:0/119862313 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f105c0a5b10 0x7f105c09d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 1 -- 172.21.15.12:0/119862313 >> v1:172.21.15.12:6789/0 conn(0x7f105c13d230 legacy=0x7f105c13f620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:29:25.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 10 monclient: _finish_hunting 0 2024-09-11T01:29:25.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 1 monclient: found mon.noname-c-legacy 2024-09-11T01:29:25.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:25.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-09-11T01:29:25.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 1 -- 172.21.15.12:0/119862313 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f105c09cc10 con 0x7f105c0a5020 2024-09-11T01:29:25.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 10 monclient: _finish_auth 0 2024-09-11T01:29:25.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 10 monclient: _check_auth_tickets 2024-09-11T01:29:25.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:25.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 1 -- 172.21.15.12:0/119862313 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f104c001a60 con 0x7f105c0a5020 2024-09-11T01:29:25.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:25.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-11T01:29:25.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 10 monclient: dump: 2024-09-11T01:29:25.797 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:25.797 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.797 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.797 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.797 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:25.797 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.797 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:25.797 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:25.797 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:25.797 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-09-11T01:29:25.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:25.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 1 -- 172.21.15.12:0/119862313 >> v1:172.21.15.71:6789/0 conn(0x7f105c0a5020 legacy=0x7f105c0a5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:29:25.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 1 --2- 172.21.15.12:0/119862313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10480039b0 0x7f1048003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 10 monclient(hunting): picked mon.b con 0x7f10480039b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:25.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 1 --2- 172.21.15.12:0/119862313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1048004800 0x7f1048004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 10 monclient(hunting): picked mon.a con 0x7f1048004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:25.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.791+0000 7f105a59c640 1 --2- 172.21.15.12:0/119862313 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f105c0a5b10 0x7f104800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105a59c640 10 monclient(hunting): picked mon.c con 0x7f105c0a5b10 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:25.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105a59c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105a59c640 1 -- 172.21.15.12:0/119862313 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f105c09d8e0 con 0x7f1048004800 2024-09-11T01:29:25.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105a59c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105a59c640 1 -- 172.21.15.12:0/119862313 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f10480055d0 con 0x7f10480039b0 2024-09-11T01:29:25.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105a59c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105a59c640 1 -- 172.21.15.12:0/119862313 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f104800bf00 con 0x7f105c0a5b10 2024-09-11T01:29:25.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105a59c640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:25.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105a59c640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:29:25.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105259c640 1 -- 172.21.15.12:0/119862313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1048004800 msgr2=0x7f1048004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:29:25.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105259c640 1 --2- 172.21.15.12:0/119862313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1048004800 0x7f1048004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:29:25.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105ad9d640 1 --2- 172.21.15.12:0/119862313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10480039b0 0x7f1048003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105ad9d640 10 monclient(hunting): get_auth_request con 0x7f10480039b0 auth_method 0 2024-09-11T01:29:25.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:25.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:29:25.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:25.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:25.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:29:25.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105ad9d640 10 monclient(hunting): handle_auth_done global_id 4327 payload 293 2024-09-11T01:29:25.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105ad9d640 1 -- 172.21.15.12:0/119862313 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f105c0a5b10 msgr2=0x7f104800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:25.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105ad9d640 1 --2- 172.21.15.12:0/119862313 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f105c0a5b10 0x7f104800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105ad9d640 1 -- 172.21.15.12:0/119862313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1048004800 msgr2=0x7f1048004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:29:25.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105ad9d640 1 --2- 172.21.15.12:0/119862313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1048004800 0x7f1048004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105ad9d640 10 monclient: _finish_hunting 0 2024-09-11T01:29:25.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105ad9d640 1 monclient: found mon.b 2024-09-11T01:29:25.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:25.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105ad9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:25.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105ad9d640 1 -- 172.21.15.12:0/119862313 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1048003520 con 0x7f10480039b0 2024-09-11T01:29:25.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105ad9d640 10 monclient: _finish_auth 0 2024-09-11T01:29:25.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105ad9d640 10 monclient: _check_auth_tickets 2024-09-11T01:29:25.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f105ad9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:25.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f10611a0640 5 monclient: authenticate success, global_id 4327 2024-09-11T01:29:25.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.792+0000 7f10611a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:25.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.795+0000 7f105ad9d640 1 -- 172.21.15.12:0/119862313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10480039b0 msgr2=0x7f1048003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:25.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.795+0000 7f105ad9d640 1 --2- 172.21.15.12:0/119862313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10480039b0 0x7f1048003db0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f104c00c420 tx=0x7f1048005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee502577665ac1a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:25.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.795+0000 7f105a59c640 1 -- 172.21.15.12:0/119862313 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c00c120 con 0x7f10480039b0 2024-09-11T01:29:25.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.795+0000 7f105a59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:25.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.795+0000 7f105a59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:25.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.795+0000 7f105a59c640 10 monclient: dump: 2024-09-11T01:29:25.806 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:25.806 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.806 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.806 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.806 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:25.806 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.806 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:25.807 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:25.807 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:25.807 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.795+0000 7f105a59c640 1 -- 172.21.15.12:0/119862313 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f104c034070 con 0x7f10480039b0 2024-09-11T01:29:25.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.795+0000 7f105a59c640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:25.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.795+0000 7f105a59c640 1 -- 172.21.15.12:0/119862313 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c005d30 con 0x7f10480039b0 2024-09-11T01:29:25.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.795+0000 7f10611a0640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:25.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.795+0000 7f10611a0640 4 set_mon_vals no callback set 2024-09-11T01:29:25.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.795+0000 7f105a59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:25.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.795+0000 7f105a59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:25.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.795+0000 7f105a59c640 10 monclient: dump: 2024-09-11T01:29:25.807 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:25.807 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.807 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.808 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.808 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:25.808 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.808 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:25.808 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:25.808 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:25.808 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.796+0000 7f10611a0640 10 monclient: shutdown 2024-09-11T01:29:25.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.796+0000 7f10611a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:25.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.796+0000 7f10611a0640 1 -- 172.21.15.12:0/119862313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10480039b0 msgr2=0x7f1048003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.796+0000 7f10611a0640 1 --2- 172.21.15.12:0/119862313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10480039b0 0x7f1048003db0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f104c00c420 tx=0x7f1048005830 comp rx=0 tx=0).stop 2024-09-11T01:29:25.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.796+0000 7f105b59e640 1 -- 172.21.15.12:0/119862313 reap_dead start 2024-09-11T01:29:25.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.796+0000 7f10611a0640 1 -- 172.21.15.12:0/119862313 shutdown_connections 2024-09-11T01:29:25.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.796+0000 7f10611a0640 1 -- 172.21.15.12:0/119862313 >> 172.21.15.12:0/119862313 conn(0x7f105c0a7540 msgr2=0x7f105c0a7950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:25.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.796+0000 7f10611a0640 1 -- 172.21.15.12:0/119862313 shutdown_connections 2024-09-11T01:29:25.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.796+0000 7f10611a0640 1 -- 172.21.15.12:0/119862313 wait complete. 2024-09-11T01:29:25.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.797+0000 7f10611a0640 10 monclient: build_initial_monmap 2024-09-11T01:29:25.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.797+0000 7f10611a0640 10 monclient: monmap: 2024-09-11T01:29:25.809 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:25.809 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:25.809 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:25.809 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:25.809 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:25.809 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.809 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:25.810 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:25.810 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:25.810 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.797+0000 7f10611a0640 1 Processor -- start 2024-09-11T01:29:25.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.797+0000 7f10611a0640 1 -- start start 2024-09-11T01:29:25.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.797+0000 7f10611a0640 10 monclient: init 2024-09-11T01:29:25.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f10611a0640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:25.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f10611a0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:29:25.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f10611a0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10480039b0 0x7f105c121300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f10611a0640 10 monclient(hunting): picked mon.noname-b con 0x7f10480039b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:25.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f10611a0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f105c121840 0x7f105c121cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f10611a0640 10 monclient(hunting): picked mon.noname-c con 0x7f105c121840 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:25.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f10611a0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c1d13f0 0x7f105c1d37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f10611a0640 10 monclient(hunting): picked mon.noname-a con 0x7f105c1d13f0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f10611a0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f10611a0640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f105c09d3a0 con 0x7f105c1d13f0 2024-09-11T01:29:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f10611a0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f10611a0640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f105c06fa80 con 0x7f10480039b0 2024-09-11T01:29:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f10611a0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f10611a0640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f105c09d520 con 0x7f105c121840 2024-09-11T01:29:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f10611a0640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f10611a0640 10 monclient(hunting): authenticate will time out at 772.240230s 2024-09-11T01:29:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f105b59e640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c1d13f0 msgr2=0x7f105c1d37e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:29:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f105b59e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c1d13f0 0x7f105c1d37e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:29:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f105ad9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10480039b0 0x7f105c121300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f105ad9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10480039b0 0x7f105c121300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:35140/0 (socket says 172.21.15.12:35140) 2024-09-11T01:29:25.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f105ad9d640 1 -- 172.21.15.12:0/1788765889 learned_addr learned my addr 172.21.15.12:0/1788765889 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:25.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f105ad9d640 10 monclient(hunting): get_auth_request con 0x7f10480039b0 auth_method 0 2024-09-11T01:29:25.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f105ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:25.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f105ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:25.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f105ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:25.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f105ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:25.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f105ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:25.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f105ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:25.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f105ad9d640 10 monclient(hunting): handle_auth_done global_id 4330 payload 824 2024-09-11T01:29:25.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f105ad9d640 1 -- 172.21.15.12:0/1788765889 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f105c121840 msgr2=0x7f105c121cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:25.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f105ad9d640 1 --2- 172.21.15.12:0/1788765889 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f105c121840 0x7f105c121cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f105ad9d640 1 -- 172.21.15.12:0/1788765889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c1d13f0 msgr2=0x7f105c1d37e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:29:25.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f105ad9d640 1 --2- 172.21.15.12:0/1788765889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c1d13f0 0x7f105c1d37e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:25.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f105ad9d640 10 monclient: _finish_hunting 0 2024-09-11T01:29:25.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f105ad9d640 1 monclient: found mon.noname-b 2024-09-11T01:29:25.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f105ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:25.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f105ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-09-11T01:29:25.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.798+0000 7f105ad9d640 1 -- 172.21.15.12:0/1788765889 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f105c125120 con 0x7f10480039b0 2024-09-11T01:29:25.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.801+0000 7f105ad9d640 1 -- 172.21.15.12:0/1788765889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10480039b0 msgr2=0x7f105c121300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:25.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.801+0000 7f105ad9d640 1 --2- 172.21.15.12:0/1788765889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10480039b0 0x7f105c121300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f104c031040 tx=0x7f105c09cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:25.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.801+0000 7f1053fff640 1 -- 172.21.15.12:0/1788765889 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c00c120 con 0x7f10480039b0 2024-09-11T01:29:25.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.801+0000 7f1053fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:25.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.801+0000 7f1053fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:25.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.801+0000 7f1053fff640 10 monclient: dump: 2024-09-11T01:29:25.813 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:25.814 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.801+0000 7f1053fff640 10 monclient: _finish_auth 0 2024-09-11T01:29:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.801+0000 7f1053fff640 10 monclient: _check_auth_tickets 2024-09-11T01:29:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.801+0000 7f1053fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.801+0000 7f1053fff640 1 -- 172.21.15.12:0/1788765889 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f104c03f070 con 0x7f10480039b0 2024-09-11T01:29:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.801+0000 7f10611a0640 5 monclient: authenticate success, global_id 4330 2024-09-11T01:29:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.801+0000 7f10611a0640 10 monclient: _renew_subs 2024-09-11T01:29:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.801+0000 7f10611a0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.801+0000 7f10611a0640 1 -- 172.21.15.12:0/1788765889 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f105c0a1f60 con 0x7f10480039b0 2024-09-11T01:29:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.801+0000 7f1053fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.801+0000 7f1053fff640 1 -- 172.21.15.12:0/1788765889 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c007470 con 0x7f10480039b0 2024-09-11T01:29:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.801+0000 7f1053fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.801+0000 7f1053fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.801+0000 7f1053fff640 10 monclient: dump: 2024-09-11T01:29:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:25.816 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.801+0000 7f10611a0640 10 monclient: _renew_subs 2024-09-11T01:29:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.801+0000 7f10611a0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.801+0000 7f10611a0640 1 -- 172.21.15.12:0/1788765889 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f105c1242b0 con 0x7f10480039b0 2024-09-11T01:29:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.801+0000 7f1059d9b640 4 set_mon_vals no callback set 2024-09-11T01:29:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.802+0000 7f1053fff640 1 -- 172.21.15.12:0/1788765889 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f104c03b420 con 0x7f10480039b0 2024-09-11T01:29:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.802+0000 7f10611a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.802+0000 7f10611a0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.802+0000 7f10611a0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.802+0000 7f10611a0640 1 -- 172.21.15.12:0/1788765889 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f105c0a5400 con 0x7f10480039b0 2024-09-11T01:29:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.802+0000 7f1053fff640 1 --2- 172.21.15.12:0/1788765889 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f103c03e210 0x7f103c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.802+0000 7f1053fff640 1 -- 172.21.15.12:0/1788765889 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4698+0+0 (secure 0 0 0) 0x7f104c073930 con 0x7f10480039b0 2024-09-11T01:29:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.803+0000 7f105259c640 1 --2- 172.21.15.12:0/1788765889 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f103c03e210 0x7f103c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.803+0000 7f105259c640 10 monclient: get_auth_request con 0x7f103c03e210 auth_method 0 2024-09-11T01:29:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.805+0000 7f105259c640 1 --2- 172.21.15.12:0/1788765889 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f103c03e210 0x7f103c0406d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f105c125330 tx=0x7f1044007b20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.808+0000 7f1053fff640 1 -- 172.21.15.12:0/1788765889 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f104c02c040 con 0x7f10480039b0 2024-09-11T01:29:25.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.808+0000 7f1053fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:25.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.808+0000 7f1053fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.994+0000 7f10611a0640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}] 2024-09-11T01:29:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.994+0000 7f10611a0640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}] 2024-09-11T01:29:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.994+0000 7f10611a0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:29:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:25.994+0000 7f10611a0640 1 -- 172.21.15.12:0/1788765889 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} v 0) -- 0x7f105c069910 con 0x7f10480039b0 2024-09-11T01:29:28.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:28.798+0000 7f1052ffd640 10 monclient: tick 2024-09-11T01:29:28.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:28.798+0000 7f1052ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:28.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:28.798+0000 7f1052ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:28.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:28.798+0000 7f1052ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:29.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:29.798+0000 7f1052ffd640 10 monclient: tick 2024-09-11T01:29:29.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:29.798+0000 7f1052ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:29.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:29.798+0000 7f1052ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:30.798+0000 7f1052ffd640 10 monclient: tick 2024-09-11T01:29:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:30.798+0000 7f1052ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:30.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:30.798+0000 7f1052ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:31.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:31.798+0000 7f1052ffd640 10 monclient: tick 2024-09-11T01:29:31.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:31.798+0000 7f1052ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:31.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:31.798+0000 7f1052ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:32.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:32.799+0000 7f1052ffd640 10 monclient: tick 2024-09-11T01:29:32.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:32.799+0000 7f1052ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:32.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:32.799+0000 7f1052ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:33.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:33.799+0000 7f1052ffd640 10 monclient: tick 2024-09-11T01:29:33.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:33.799+0000 7f1052ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:33.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:33.799+0000 7f1052ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:34.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:34.799+0000 7f1052ffd640 10 monclient: tick 2024-09-11T01:29:34.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:34.799+0000 7f1052ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:34.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:34.799+0000 7f1052ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:35.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:35.799+0000 7f1052ffd640 10 monclient: tick 2024-09-11T01:29:35.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:35.799+0000 7f1052ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:35.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:35.799+0000 7f1052ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:36.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:36.799+0000 7f1052ffd640 10 monclient: tick 2024-09-11T01:29:36.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:36.799+0000 7f1052ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:36.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:36.799+0000 7f1052ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:37.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:37.799+0000 7f1052ffd640 10 monclient: tick 2024-09-11T01:29:37.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:37.799+0000 7f1052ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:37.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:37.799+0000 7f1052ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:38.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:38.799+0000 7f1052ffd640 10 monclient: tick 2024-09-11T01:29:38.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:38.799+0000 7f1052ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:38.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:38.799+0000 7f1052ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:38.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:38.799+0000 7f1052ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:39.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:39.800+0000 7f1052ffd640 10 monclient: tick 2024-09-11T01:29:39.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:39.800+0000 7f1052ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:39.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:39.800+0000 7f1052ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:40.697 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-11T01:29:40.697 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-11T01:29:40.697 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:29:40.698 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:29:40.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.800+0000 7f1052ffd640 10 monclient: tick 2024-09-11T01:29:40.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.800+0000 7f1052ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:40.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.800+0000 7f1052ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.808+0000 7f5b0b384640 10 monclient: get_monmap_and_config 2024-09-11T01:29:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.808+0000 7f5b0b384640 10 monclient: build_initial_monmap 2024-09-11T01:29:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.808+0000 7f5b0b384640 10 monclient: monmap: 2024-09-11T01:29:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:40.810 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:40.809508+0000 2024-09-11T01:29:40.811 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:40.809508+0000 2024-09-11T01:29:40.811 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:40.811 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:40.811 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:40.811 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:40.811 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:40.811 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:40.811 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:40.811 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:40.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.809+0000 7f5b0b384640 1 Processor -- start 2024-09-11T01:29:40.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.809+0000 7f5b0b384640 1 -- start start 2024-09-11T01:29:40.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.809+0000 7f5b0b384640 10 monclient: init 2024-09-11T01:29:40.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b0b384640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:40.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b0b384640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-11T01:29:40.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b0b384640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5b040a1380 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:40.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b0b384640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5b040a0c90 0x7f5b0413ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:40.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b0b384640 10 monclient(hunting): picked mon.noname-c con 0x7f5b040a0c90 addr v2:172.21.15.71:3300/0 2024-09-11T01:29:40.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b0b384640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5b0413b340 0x7f5b0413d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:40.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b0b384640 10 monclient(hunting): picked mon.noname-a con 0x7f5b0413b340 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:40.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b09120640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7f5b040a1380 legacy=0x7f5b040a1760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:29:40.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b0b384640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b040a2b40 con 0x7f5b040a1380 2024-09-11T01:29:40.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b0b384640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:40.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b0b384640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f5b0409f9a0 con 0x7f5b0413b340 2024-09-11T01:29:40.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b0b384640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:40.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b0b384640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f5b040a2230 con 0x7f5b040a0c90 2024-09-11T01:29:40.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b0b384640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:40.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b0b384640 10 monclient(hunting): authenticate will time out at 787.252270s 2024-09-11T01:29:40.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b09921640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7f5b0413b340 msgr2=0x7f5b0413d730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:29:40.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b09921640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5b0413b340 0x7f5b0413d730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:29:40.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b0891f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5b040a0c90 0x7f5b0413ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:40.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b0891f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5b040a0c90 0x7f5b0413ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:44074/0 (socket says 172.21.15.12:44074) 2024-09-11T01:29:40.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b0891f640 1 -- 172.21.15.12:0/1459468685 learned_addr learned my addr 172.21.15.12:0/1459468685 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:40.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b0891f640 10 monclient(hunting): get_auth_request con 0x7f5b040a0c90 auth_method 0 2024-09-11T01:29:40.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b0891f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:40.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b0891f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:40.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b0891f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:40.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b0891f640 1 -- 172.21.15.12:0/1459468685 >> v2:172.21.15.71:3300/0 conn(0x7f5b040a0c90 msgr2=0x7f5b0413ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:29:40.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b0891f640 1 -- 172.21.15.12:0/1459468685 >> v2:172.21.15.71:3300/0 conn(0x7f5b040a0c90 msgr2=0x7f5b0413ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:29:40.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b0891f640 1 --2- 172.21.15.12:0/1459468685 >> v2:172.21.15.71:3300/0 conn(0x7f5b040a0c90 0x7f5b0413ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:29:40.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:40.810+0000 7f5b0891f640 1 --2- 172.21.15.12:0/1459468685 >> v2:172.21.15.71:3300/0 conn(0x7f5b040a0c90 0x7f5b0413ae00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:29:41.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.011+0000 7f5b09120640 1 -- 172.21.15.12:0/1459468685 >> v1:172.21.15.12:6789/0 conn(0x7f5b040a1380 legacy=0x7f5b040a1760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:29:41.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.011+0000 7f5b09921640 1 -- 172.21.15.12:0/1459468685 >> v2:172.21.15.12:3300/0 conn(0x7f5b0413b340 msgr2=0x7f5b0413d730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:29:41.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.011+0000 7f5b09921640 1 --2- 172.21.15.12:0/1459468685 >> v2:172.21.15.12:3300/0 conn(0x7f5b0413b340 0x7f5b0413d730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:29:41.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.012+0000 7f5b0891f640 1 --2- 172.21.15.12:0/1459468685 >> v2:172.21.15.71:3300/0 conn(0x7f5b040a0c90 0x7f5b0413ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:41.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.012+0000 7f5b0891f640 10 monclient(hunting): get_auth_request con 0x7f5b040a0c90 auth_method 0 2024-09-11T01:29:41.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.012+0000 7f5b0891f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:41.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.012+0000 7f5b0891f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:41.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.012+0000 7f5b0891f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:29:41.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.012+0000 7f5b0891f640 1 -- 172.21.15.12:0/1459468685 >> v2:172.21.15.71:3300/0 conn(0x7f5b040a0c90 msgr2=0x7f5b0413ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:29:41.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.012+0000 7f5b0891f640 1 -- 172.21.15.12:0/1459468685 >> v2:172.21.15.71:3300/0 conn(0x7f5b040a0c90 msgr2=0x7f5b0413ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:29:41.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.012+0000 7f5b0891f640 1 --2- 172.21.15.12:0/1459468685 >> v2:172.21.15.71:3300/0 conn(0x7f5b040a0c90 0x7f5b0413ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:29:41.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.012+0000 7f5b0891f640 1 --2- 172.21.15.12:0/1459468685 >> v2:172.21.15.71:3300/0 conn(0x7f5b040a0c90 0x7f5b0413ae00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:29:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.412+0000 7f5b09120640 1 -- 172.21.15.12:0/1459468685 >> v1:172.21.15.12:6789/0 conn(0x7f5b040a1380 legacy=0x7f5b040a1760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:29:41.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.412+0000 7f5b09921640 1 -- 172.21.15.12:0/1459468685 >> v2:172.21.15.12:3300/0 conn(0x7f5b0413b340 msgr2=0x7f5b0413d730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:29:41.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.412+0000 7f5b09921640 1 --2- 172.21.15.12:0/1459468685 >> v2:172.21.15.12:3300/0 conn(0x7f5b0413b340 0x7f5b0413d730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:29:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.413+0000 7f5b0891f640 1 --2- 172.21.15.12:0/1459468685 >> v2:172.21.15.71:3300/0 conn(0x7f5b040a0c90 0x7f5b0413ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.413+0000 7f5b0891f640 10 monclient(hunting): get_auth_request con 0x7f5b040a0c90 auth_method 0 2024-09-11T01:29:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.413+0000 7f5b0891f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.413+0000 7f5b0891f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.413+0000 7f5b0891f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:29:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.413+0000 7f5b0891f640 1 -- 172.21.15.12:0/1459468685 >> v2:172.21.15.71:3300/0 conn(0x7f5b040a0c90 msgr2=0x7f5b0413ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:29:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.413+0000 7f5b0891f640 1 -- 172.21.15.12:0/1459468685 >> v2:172.21.15.71:3300/0 conn(0x7f5b040a0c90 msgr2=0x7f5b0413ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:29:41.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.413+0000 7f5b0891f640 1 --2- 172.21.15.12:0/1459468685 >> v2:172.21.15.71:3300/0 conn(0x7f5b040a0c90 0x7f5b0413ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:29:41.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.413+0000 7f5b0891f640 1 --2- 172.21.15.12:0/1459468685 >> v2:172.21.15.71:3300/0 conn(0x7f5b040a0c90 0x7f5b0413ae00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:29:41.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.800+0000 7f1052ffd640 10 monclient: tick 2024-09-11T01:29:41.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.800+0000 7f1052ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:41.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:41.800+0000 7f1052ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:42.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:42.213+0000 7f5b09921640 1 -- 172.21.15.12:0/1459468685 >> v2:172.21.15.12:3300/0 conn(0x7f5b0413b340 msgr2=0x7f5b0413d730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:29:42.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:42.213+0000 7f5b09120640 1 -- 172.21.15.12:0/1459468685 >> v1:172.21.15.12:6789/0 conn(0x7f5b040a1380 legacy=0x7f5b040a1760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:29:42.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:42.213+0000 7f5b09921640 1 --2- 172.21.15.12:0/1459468685 >> v2:172.21.15.12:3300/0 conn(0x7f5b0413b340 0x7f5b0413d730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:29:42.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:42.214+0000 7f5b0891f640 1 --2- 172.21.15.12:0/1459468685 >> v2:172.21.15.71:3300/0 conn(0x7f5b040a0c90 0x7f5b0413ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:42.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:42.214+0000 7f5b0891f640 10 monclient(hunting): get_auth_request con 0x7f5b040a0c90 auth_method 0 2024-09-11T01:29:42.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:42.215+0000 7f5b0891f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:42.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:42.215+0000 7f5b0891f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:42.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:42.215+0000 7f5b0891f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:29:42.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:42.215+0000 7f5b0891f640 1 -- 172.21.15.12:0/1459468685 >> v2:172.21.15.71:3300/0 conn(0x7f5b040a0c90 msgr2=0x7f5b0413ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:29:42.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:42.215+0000 7f5b0891f640 1 -- 172.21.15.12:0/1459468685 >> v2:172.21.15.71:3300/0 conn(0x7f5b040a0c90 msgr2=0x7f5b0413ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:29:42.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:42.215+0000 7f5b0891f640 1 --2- 172.21.15.12:0/1459468685 >> v2:172.21.15.71:3300/0 conn(0x7f5b040a0c90 0x7f5b0413ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:29:42.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:42.215+0000 7f5b0891f640 1 --2- 172.21.15.12:0/1459468685 >> v2:172.21.15.71:3300/0 conn(0x7f5b040a0c90 0x7f5b0413ae00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:29:42.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:42.800+0000 7f1052ffd640 10 monclient: tick 2024-09-11T01:29:42.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:42.800+0000 7f1052ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:42.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:42.800+0000 7f1052ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:43.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.800+0000 7f1052ffd640 10 monclient: tick 2024-09-11T01:29:43.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.800+0000 7f1052ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:43.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.800+0000 7f1052ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:43.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.810+0000 7f5b02ffd640 10 monclient(hunting): tick 2024-09-11T01:29:43.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.810+0000 7f5b02ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:29:43.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.810+0000 7f5b02ffd640 1 monclient(hunting): continuing hunt 2024-09-11T01:29:43.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.810+0000 7f5b02ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:29:43.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.810+0000 7f5b02ffd640 1 -- 172.21.15.12:0/1459468685 >> v2:172.21.15.71:3300/0 conn(0x7f5b040a0c90 msgr2=0x7f5b0413ae00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:29:43.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.810+0000 7f5b02ffd640 1 --2- 172.21.15.12:0/1459468685 >> v2:172.21.15.71:3300/0 conn(0x7f5b040a0c90 0x7f5b0413ae00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:43.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.810+0000 7f5b02ffd640 1 -- 172.21.15.12:0/1459468685 >> v2:172.21.15.12:3300/0 conn(0x7f5b0413b340 msgr2=0x7f5b0413d730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:29:43.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.810+0000 7f5b02ffd640 1 --2- 172.21.15.12:0/1459468685 >> v2:172.21.15.12:3300/0 conn(0x7f5b0413b340 0x7f5b0413d730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:43.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.810+0000 7f5b02ffd640 1 -- 172.21.15.12:0/1459468685 >> v1:172.21.15.12:6789/0 conn(0x7f5b040a1380 legacy=0x7f5b040a1760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:29:43.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.810+0000 7f5b02ffd640 10 monclient: _add_conns ranks=[2,1] 2024-09-11T01:29:43.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.810+0000 7f5b02ffd640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 0x7f5af0005ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:43.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.810+0000 7f5b02ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f5af0003690 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:43.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.810+0000 7f5b02ffd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5af0006150 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:43.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.810+0000 7f5b02ffd640 1 -- 172.21.15.12:0/1459468685 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b040a2510 con 0x7f5af0006150 2024-09-11T01:29:43.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.810+0000 7f5b02ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:43.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.810+0000 7f5b02ffd640 1 -- 172.21.15.12:0/1459468685 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5b0409f9a0 con 0x7f5af0003690 2024-09-11T01:29:43.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.810+0000 7f5b02ffd640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:43.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.810+0000 7f5b09120640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 0x7f5af0005ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:43.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.810+0000 7f5b09120640 10 monclient(hunting): get_auth_request con 0x7f5af0003690 auth_method 0 2024-09-11T01:29:43.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.810+0000 7f5b09120640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:43.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.810+0000 7f5b09120640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:43.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.810+0000 7f5b09120640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:43.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.810+0000 7f5b09120640 1 -- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 msgr2=0x7f5af0005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:29:43.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.810+0000 7f5b09120640 1 -- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 msgr2=0x7f5af0005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:29:43.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.810+0000 7f5b09120640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 0x7f5af0005ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:29:43.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.810+0000 7f5b09120640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 0x7f5af0005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:29:43.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:43.813+0000 7f5b0891f640 1 -- 172.21.15.12:0/1459468685 >> v1:172.21.15.71:6789/0 conn(0x7f5af0006150 legacy=0x7f5af0008610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:44.011+0000 7f5b09120640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 0x7f5af0005ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:44.011+0000 7f5b09120640 10 monclient(hunting): get_auth_request con 0x7f5af0003690 auth_method 0 2024-09-11T01:29:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:44.011+0000 7f5b09120640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:44.011+0000 7f5b09120640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:44.011+0000 7f5b09120640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:29:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:44.011+0000 7f5b09120640 1 -- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 msgr2=0x7f5af0005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:29:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:44.011+0000 7f5b09120640 1 -- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 msgr2=0x7f5af0005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:29:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:44.011+0000 7f5b09120640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 0x7f5af0005ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:29:44.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:44.011+0000 7f5b09120640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 0x7f5af0005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:29:44.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:44.412+0000 7f5b09120640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 0x7f5af0005ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:44.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:44.412+0000 7f5b09120640 10 monclient(hunting): get_auth_request con 0x7f5af0003690 auth_method 0 2024-09-11T01:29:44.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:44.412+0000 7f5b09120640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:44.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:44.412+0000 7f5b09120640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:44.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:44.412+0000 7f5b09120640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:29:44.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:44.412+0000 7f5b09120640 1 -- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 msgr2=0x7f5af0005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:29:44.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:44.412+0000 7f5b09120640 1 -- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 msgr2=0x7f5af0005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:29:44.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:44.412+0000 7f5b09120640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 0x7f5af0005ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:29:44.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:44.412+0000 7f5b09120640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 0x7f5af0005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:29:44.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:44.800+0000 7f1052ffd640 10 monclient: tick 2024-09-11T01:29:44.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:44.800+0000 7f1052ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:29:44.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:44.800+0000 7f1052ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:45.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.213+0000 7f5b09120640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 0x7f5af0005ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:45.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.213+0000 7f5b09120640 10 monclient(hunting): get_auth_request con 0x7f5af0003690 auth_method 0 2024-09-11T01:29:45.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.213+0000 7f5b09120640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:45.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.213+0000 7f5b09120640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:45.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.213+0000 7f5b09120640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:29:45.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.213+0000 7f5b09120640 1 -- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 msgr2=0x7f5af0005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:29:45.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.213+0000 7f5b09120640 1 -- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 msgr2=0x7f5af0005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:29:45.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.213+0000 7f5b09120640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 0x7f5af0005ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:29:45.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.213+0000 7f5b09120640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 0x7f5af0005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:29:45.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.705+0000 7f5b03fff640 1 -- 172.21.15.12:0/1459468685 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3313898497 0 0) 0x7f5b040a2510 con 0x7f5af0006150 2024-09-11T01:29:45.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.705+0000 7f5b03fff640 10 monclient(hunting): _negotiate 2024-09-11T01:29:45.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.705+0000 7f5b03fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:45.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.705+0000 7f5b03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:45.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.705+0000 7f5b03fff640 10 monclient(hunting): my global_id is 4333 2024-09-11T01:29:45.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.705+0000 7f5b03fff640 1 -- 172.21.15.12:0/1459468685 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5aec003520 con 0x7f5af0006150 2024-09-11T01:29:45.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.706+0000 7f5b03fff640 1 -- 172.21.15.12:0/1459468685 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2425543650 0 0) 0x7f5aec003520 con 0x7f5af0006150 2024-09-11T01:29:45.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.706+0000 7f5b03fff640 1 -- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 msgr2=0x7f5af0005ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:29:45.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.706+0000 7f5b03fff640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 0x7f5af0005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:45.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.706+0000 7f5b03fff640 10 monclient: _finish_hunting 0 2024-09-11T01:29:45.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.706+0000 7f5b03fff640 1 monclient: found mon.noname-c-legacy 2024-09-11T01:29:45.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.706+0000 7f5b03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:45.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.706+0000 7f5b03fff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-09-11T01:29:45.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.706+0000 7f5b03fff640 1 -- 172.21.15.12:0/1459468685 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b040a2b40 con 0x7f5af0006150 2024-09-11T01:29:45.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.706+0000 7f5b03fff640 10 monclient: _finish_auth 0 2024-09-11T01:29:45.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.706+0000 7f5b03fff640 10 monclient: _check_auth_tickets 2024-09-11T01:29:45.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.706+0000 7f5b03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:45.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.706+0000 7f5b03fff640 1 -- 172.21.15.12:0/1459468685 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f5af80075f0 con 0x7f5af0006150 2024-09-11T01:29:45.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.706+0000 7f5b0b384640 5 monclient: authenticate success, global_id 4333 2024-09-11T01:29:45.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.706+0000 7f5b0b384640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:45.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.706+0000 7f5b03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:45.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b03fff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-11T01:29:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b03fff640 10 monclient: dump: 2024-09-11T01:29:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:45.712 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b03fff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-09-11T01:29:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b03fff640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b03fff640 1 -- 172.21.15.12:0/1459468685 >> v1:172.21.15.71:6789/0 conn(0x7f5af0006150 legacy=0x7f5af0008610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b03fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:29:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b09921640 1 -- 172.21.15.12:0/1459468685 reap_dead start 2024-09-11T01:29:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b03fff640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aec0039b0 0x7f5aec003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b03fff640 10 monclient(hunting): picked mon.a con 0x7f5aec0039b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b03fff640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 0x7f5aec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b03fff640 10 monclient(hunting): picked mon.c con 0x7f5af0003690 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b03fff640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5aec004d40 0x7f5aec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b03fff640 10 monclient(hunting): picked mon.b con 0x7f5aec004d40 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b03fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b03fff640 1 -- 172.21.15.12:0/1459468685 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5b0409f9a0 con 0x7f5aec0039b0 2024-09-11T01:29:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b03fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b03fff640 1 -- 172.21.15.12:0/1459468685 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5aec00ffb0 con 0x7f5aec004d40 2024-09-11T01:29:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b03fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b03fff640 1 -- 172.21.15.12:0/1459468685 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5aec0101b0 con 0x7f5af0003690 2024-09-11T01:29:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b03fff640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b03fff640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:29:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b09120640 1 -- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aec0039b0 msgr2=0x7f5aec003db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:29:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b09120640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aec0039b0 0x7f5aec003db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:29:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b0b384640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b0891f640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 0x7f5aec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b0891f640 10 monclient(hunting): get_auth_request con 0x7f5af0003690 auth_method 0 2024-09-11T01:29:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b0891f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b0891f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b0891f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:29:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b09921640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5aec004d40 0x7f5aec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b09921640 10 monclient(hunting): get_auth_request con 0x7f5aec004d40 auth_method 0 2024-09-11T01:29:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b09921640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b09921640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b09921640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:29:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b0891f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b0891f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b0891f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:29:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b09921640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b09921640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b09921640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:29:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b0891f640 10 monclient(hunting): handle_auth_done global_id 4333 payload 293 2024-09-11T01:29:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b0891f640 1 -- 172.21.15.12:0/1459468685 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5aec004d40 msgr2=0x7f5aec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b0891f640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5aec004d40 0x7f5aec0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b0891f640 1 -- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aec0039b0 msgr2=0x7f5aec003db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:29:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b0891f640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aec0039b0 0x7f5aec003db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b0891f640 10 monclient: _finish_hunting 0 2024-09-11T01:29:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b0891f640 1 monclient: found mon.c 2024-09-11T01:29:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b0891f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b0891f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b0891f640 1 -- 172.21.15.12:0/1459468685 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aec003520 con 0x7f5af0003690 2024-09-11T01:29:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b0891f640 10 monclient: _finish_auth 0 2024-09-11T01:29:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b0891f640 10 monclient: _check_auth_tickets 2024-09-11T01:29:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b0891f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.707+0000 7f5b09921640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5aec004d40 0x7f5aec0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:45.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.710+0000 7f5b0891f640 1 -- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 msgr2=0x7f5aec004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:45.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.710+0000 7f5b0891f640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 0x7f5aec004800 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5af80096d0 tx=0x7f5aec005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e01ede6694d81a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:45.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.711+0000 7f5b03fff640 1 -- 172.21.15.12:0/1459468685 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af8007ca0 con 0x7f5af0003690 2024-09-11T01:29:45.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.711+0000 7f5b03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:45.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.711+0000 7f5b03fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:45.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.711+0000 7f5b03fff640 10 monclient: dump: 2024-09-11T01:29:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:45.723 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.711+0000 7f5b03fff640 1 -- 172.21.15.12:0/1459468685 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5af800b0d0 con 0x7f5af0003690 2024-09-11T01:29:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.711+0000 7f5b03fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.711+0000 7f5b03fff640 1 -- 172.21.15.12:0/1459468685 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af800b4c0 con 0x7f5af0003690 2024-09-11T01:29:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.711+0000 7f5b0b384640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.711+0000 7f5b0b384640 4 set_mon_vals no callback set 2024-09-11T01:29:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.711+0000 7f5b03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.711+0000 7f5b03fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.711+0000 7f5b03fff640 10 monclient: dump: 2024-09-11T01:29:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:45.724 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.711+0000 7f5b0b384640 10 monclient: shutdown 2024-09-11T01:29:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.711+0000 7f5b0b384640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.711+0000 7f5b0b384640 1 -- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 msgr2=0x7f5aec004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.711+0000 7f5b0b384640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 0x7f5aec004800 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5af80096d0 tx=0x7f5aec005830 comp rx=0 tx=0).stop 2024-09-11T01:29:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.712+0000 7f5b0b384640 1 -- 172.21.15.12:0/1459468685 shutdown_connections 2024-09-11T01:29:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.712+0000 7f5b0b384640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5aec004d40 0x7f5aec0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.712+0000 7f5b0b384640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aec0039b0 0x7f5aec003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.712+0000 7f5b0b384640 1 --2- 172.21.15.12:0/1459468685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5af0003690 0x7f5aec004800 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.712+0000 7f5b0b384640 1 -- 172.21.15.12:0/1459468685 >> 172.21.15.12:0/1459468685 conn(0x7f5b040a2d90 msgr2=0x7f5b040a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.712+0000 7f5b0b384640 1 -- 172.21.15.12:0/1459468685 shutdown_connections 2024-09-11T01:29:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.712+0000 7f5b0b384640 1 -- 172.21.15.12:0/1459468685 wait complete. 2024-09-11T01:29:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.713+0000 7f5b0b384640 10 monclient: build_initial_monmap 2024-09-11T01:29:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.713+0000 7f5b0b384640 10 monclient: monmap: 2024-09-11T01:29:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:45.726 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.713+0000 7f5b0b384640 1 Processor -- start 2024-09-11T01:29:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.713+0000 7f5b0b384640 1 -- start start 2024-09-11T01:29:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.713+0000 7f5b0b384640 10 monclient: init 2024-09-11T01:29:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b0b384640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b0b384640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:29:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b0b384640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5aec0039b0 0x7f5b0411ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b0b384640 10 monclient(hunting): picked mon.noname-c con 0x7f5aec0039b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b0b384640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5aec004d40 0x7f5b041c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b0b384640 10 monclient(hunting): picked mon.noname-b con 0x7f5aec004d40 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b0b384640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5af0003690 0x7f5b0411c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b0b384640 10 monclient(hunting): picked mon.noname-a con 0x7f5af0003690 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b0b384640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b0b384640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5b040a0510 con 0x7f5af0003690 2024-09-11T01:29:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b0b384640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b0b384640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5b040a0390 con 0x7f5aec004d40 2024-09-11T01:29:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b0b384640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b0b384640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5b040a1760 con 0x7f5aec0039b0 2024-09-11T01:29:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b0b384640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b0b384640 10 monclient(hunting): authenticate will time out at 792.156171s 2024-09-11T01:29:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b09921640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5af0003690 msgr2=0x7f5b0411c8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:29:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b09921640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5af0003690 0x7f5b0411c8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:29:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b09120640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5aec0039b0 0x7f5b0411ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b09120640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5aec0039b0 0x7f5b0411ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:53460/0 (socket says 172.21.15.12:53460) 2024-09-11T01:29:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b09120640 1 -- 172.21.15.12:0/3642498407 learned_addr learned my addr 172.21.15.12:0/3642498407 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b09120640 10 monclient(hunting): get_auth_request con 0x7f5aec0039b0 auth_method 0 2024-09-11T01:29:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b09120640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b09120640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b09120640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b09120640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b09120640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b09120640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b0891f640 1 --2- 172.21.15.12:0/3642498407 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5aec004d40 0x7f5b041c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b09120640 10 monclient(hunting): handle_auth_done global_id 4400 payload 824 2024-09-11T01:29:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b09120640 1 -- 172.21.15.12:0/3642498407 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5aec004d40 msgr2=0x7f5b041c8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b09120640 1 --2- 172.21.15.12:0/3642498407 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5aec004d40 0x7f5b041c8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b09120640 1 -- 172.21.15.12:0/3642498407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5af0003690 msgr2=0x7f5b0411c8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:29:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b09120640 1 --2- 172.21.15.12:0/3642498407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5af0003690 0x7f5b0411c8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b09120640 10 monclient: _finish_hunting 0 2024-09-11T01:29:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b09120640 1 monclient: found mon.noname-c 2024-09-11T01:29:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b09120640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b09120640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:29:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b09120640 1 -- 172.21.15.12:0/3642498407 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b041c0430 con 0x7f5aec0039b0 2024-09-11T01:29:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b0891f640 10 monclient: get_auth_request con 0x7f5aec004d40 auth_method 0 2024-09-11T01:29:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.714+0000 7f5b0891f640 1 --2- 172.21.15.12:0/3642498407 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5aec004d40 0x7f5b041c8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:29:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.716+0000 7f5b09120640 1 -- 172.21.15.12:0/3642498407 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5aec0039b0 msgr2=0x7f5b0411ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.717+0000 7f5b09120640 1 --2- 172.21.15.12:0/3642498407 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5aec0039b0 0x7f5b0411ea10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5af400bbb0 tx=0x7f5af4005420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.717+0000 7f5b01ffb640 1 -- 172.21.15.12:0/3642498407 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af4017600 con 0x7f5aec0039b0 2024-09-11T01:29:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.717+0000 7f5b01ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.717+0000 7f5b01ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.717+0000 7f5b01ffb640 10 monclient: dump: 2024-09-11T01:29:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:45.731 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.717+0000 7f5b01ffb640 10 monclient: _finish_auth 0 2024-09-11T01:29:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.717+0000 7f5b01ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:29:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.717+0000 7f5b01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.717+0000 7f5b0b384640 5 monclient: authenticate success, global_id 4400 2024-09-11T01:29:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.717+0000 7f5b0b384640 10 monclient: _renew_subs 2024-09-11T01:29:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.717+0000 7f5b01ffb640 1 -- 172.21.15.12:0/3642498407 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5af40177a0 con 0x7f5aec0039b0 2024-09-11T01:29:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.717+0000 7f5b0b384640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.717+0000 7f5b0b384640 1 -- 172.21.15.12:0/3642498407 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b0411e5d0 con 0x7f5aec0039b0 2024-09-11T01:29:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.717+0000 7f5b01ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.717+0000 7f5b01ffb640 1 -- 172.21.15.12:0/3642498407 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af4017b90 con 0x7f5aec0039b0 2024-09-11T01:29:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.717+0000 7f5b01ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.717+0000 7f5b01ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.717+0000 7f5b01ffb640 10 monclient: dump: 2024-09-11T01:29:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:45.732 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.717+0000 7f5b0b384640 10 monclient: _renew_subs 2024-09-11T01:29:45.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.717+0000 7f5b0b384640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:45.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.717+0000 7f5b0b384640 1 -- 172.21.15.12:0/3642498407 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b0411f970 con 0x7f5aec0039b0 2024-09-11T01:29:45.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.717+0000 7f5b02ffd640 4 set_mon_vals no callback set 2024-09-11T01:29:45.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.719+0000 7f5b0b384640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:45.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.719+0000 7f5b0b384640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:45.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.719+0000 7f5b01ffb640 1 -- 172.21.15.12:0/3642498407 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f5af4028060 con 0x7f5aec0039b0 2024-09-11T01:29:45.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.719+0000 7f5b0b384640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:45.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.719+0000 7f5b0b384640 1 -- 172.21.15.12:0/3642498407 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b040a2230 con 0x7f5aec0039b0 2024-09-11T01:29:45.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.719+0000 7f5b01ffb640 1 --2- 172.21.15.12:0/3642498407 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5af003ea40 0x7f5af0040f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:45.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.719+0000 7f5b01ffb640 1 -- 172.21.15.12:0/3642498407 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4698+0+0 (secure 0 0 0) 0x7f5af4054220 con 0x7f5aec0039b0 2024-09-11T01:29:45.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.719+0000 7f5b0891f640 1 --2- 172.21.15.12:0/3642498407 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5af003ea40 0x7f5af0040f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:45.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.719+0000 7f5b0891f640 10 monclient: get_auth_request con 0x7f5af003ea40 auth_method 0 2024-09-11T01:29:45.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.722+0000 7f5b0891f640 1 --2- 172.21.15.12:0/3642498407 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5af003ea40 0x7f5af0040f00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5af8031000 tx=0x7f5af802f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:45.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.724+0000 7f5b01ffb640 1 -- 172.21.15.12:0/3642498407 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5af4052c70 con 0x7f5aec0039b0 2024-09-11T01:29:45.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.724+0000 7f5b01ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:45.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.724+0000 7f5b01ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:45.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.760+0000 7f1053fff640 1 -- 172.21.15.12:0/1788765889 <== mon.1 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]=0 enabled application 'rados' on pool 'unique_pool_0' v19) ==== 197+0+0 (secure 0 0 0) 0x7f104c072380 con 0x7f10480039b0 2024-09-11T01:29:45.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.760+0000 7f1053fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}] 2024-09-11T01:29:45.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.760+0000 7f1053fff640 10 monclient: _finish_command 2 = system:0 enabled application 'rados' on pool 'unique_pool_0' 2024-09-11T01:29:45.762 INFO:teuthology.orchestra.run.smithi012.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-11T01:29:45.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.764+0000 7f10611a0640 1 -- 172.21.15.12:0/1788765889 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f103c03e210 msgr2=0x7f103c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:45.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.765+0000 7f10611a0640 1 --2- 172.21.15.12:0/1788765889 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f103c03e210 0x7f103c0406d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f105c125330 tx=0x7f1044007b20 comp rx=0 tx=0).stop 2024-09-11T01:29:45.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.765+0000 7f10611a0640 10 monclient: shutdown 2024-09-11T01:29:45.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.765+0000 7f10611a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:45.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.765+0000 7f10611a0640 1 -- 172.21.15.12:0/1788765889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10480039b0 msgr2=0x7f105c121300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:45.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.765+0000 7f10611a0640 1 --2- 172.21.15.12:0/1788765889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10480039b0 0x7f105c121300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f104c031040 tx=0x7f105c09cde0 comp rx=0 tx=0).stop 2024-09-11T01:29:45.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.765+0000 7f10611a0640 1 -- 172.21.15.12:0/1788765889 shutdown_connections 2024-09-11T01:29:45.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.765+0000 7f10611a0640 1 --2- 172.21.15.12:0/1788765889 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f103c03e210 0x7f103c0406d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:45.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.765+0000 7f10611a0640 1 --2- 172.21.15.12:0/1788765889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f105c1d13f0 0x7f105c1d37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:45.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.765+0000 7f10611a0640 1 --2- 172.21.15.12:0/1788765889 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f105c121840 0x7f105c121cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:45.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.765+0000 7f10611a0640 1 --2- 172.21.15.12:0/1788765889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10480039b0 0x7f105c121300 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:45.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.765+0000 7f10611a0640 1 -- 172.21.15.12:0/1788765889 >> 172.21.15.12:0/1788765889 conn(0x7f105c0a7540 msgr2=0x7f105c13c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:45.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.765+0000 7f10611a0640 1 -- 172.21.15.12:0/1788765889 shutdown_connections 2024-09-11T01:29:45.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.765+0000 7f10611a0640 1 -- 172.21.15.12:0/1788765889 wait complete. 2024-09-11T01:29:45.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.994+0000 7f5b0b384640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:29:45.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.994+0000 7f5b0b384640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:29:45.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.994+0000 7f5b0b384640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:45.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.994+0000 7f5b0b384640 1 -- 172.21.15.12:0/3642498407 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5b04135f10 con 0x7f5aec0039b0 2024-09-11T01:29:45.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.994+0000 7f5b01ffb640 1 -- 172.21.15.12:0/3642498407 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f5af4020c50 con 0x7f5aec0039b0 2024-09-11T01:29:45.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.994+0000 7f5b01ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:29:45.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.994+0000 7f5b01ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:45.997 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":10,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:29:45.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.996+0000 7f5b0b384640 1 -- 172.21.15.12:0/3642498407 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5af003ea40 msgr2=0x7f5af0040f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:45.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.996+0000 7f5b0b384640 1 --2- 172.21.15.12:0/3642498407 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5af003ea40 0x7f5af0040f00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5af8031000 tx=0x7f5af802f000 comp rx=0 tx=0).stop 2024-09-11T01:29:45.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.996+0000 7f5b0b384640 10 monclient: shutdown 2024-09-11T01:29:45.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.996+0000 7f5b0b384640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:45.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.996+0000 7f5b0b384640 1 -- 172.21.15.12:0/3642498407 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5aec0039b0 msgr2=0x7f5b0411ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:45.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.996+0000 7f5b0b384640 1 --2- 172.21.15.12:0/3642498407 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5aec0039b0 0x7f5b0411ea10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5af400bbb0 tx=0x7f5af4005420 comp rx=0 tx=0).stop 2024-09-11T01:29:45.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.996+0000 7f5b0b384640 1 -- 172.21.15.12:0/3642498407 shutdown_connections 2024-09-11T01:29:45.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.996+0000 7f5b0b384640 1 --2- 172.21.15.12:0/3642498407 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5af003ea40 0x7f5af0040f00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:45.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.996+0000 7f5b0b384640 1 --2- 172.21.15.12:0/3642498407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5af0003690 0x7f5b0411c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:45.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.996+0000 7f5b0b384640 1 --2- 172.21.15.12:0/3642498407 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5aec004d40 0x7f5b041c8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:45.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.996+0000 7f5b0b384640 1 --2- 172.21.15.12:0/3642498407 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5aec0039b0 0x7f5b0411ea10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:45.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.996+0000 7f5b0b384640 1 -- 172.21.15.12:0/3642498407 >> 172.21.15.12:0/3642498407 conn(0x7f5b040a2d90 msgr2=0x7f5b0409de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:45.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.996+0000 7f5b0b384640 1 -- 172.21.15.12:0/3642498407 shutdown_connections 2024-09-11T01:29:45.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:45.996+0000 7f5b0b384640 1 -- 172.21.15.12:0/3642498407 wait complete. 2024-09-11T01:29:46.009 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:29:46.009 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T01:29:46.010 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:29:46.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.110+0000 7f7398a2a640 10 monclient: get_monmap_and_config 2024-09-11T01:29:46.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.110+0000 7f7398a2a640 10 monclient: build_initial_monmap 2024-09-11T01:29:46.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.110+0000 7f7398a2a640 10 monclient: monmap: 2024-09-11T01:29:46.112 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:46.112 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:46.112 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:46.111327+0000 2024-09-11T01:29:46.112 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:46.111327+0000 2024-09-11T01:29:46.112 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:46.113 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:46.113 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:46.113 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:46.113 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:46.113 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:46.113 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:46.113 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:46.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.111+0000 7f7398a2a640 1 Processor -- start 2024-09-11T01:29:46.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.111+0000 7f7398a2a640 1 -- start start 2024-09-11T01:29:46.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.111+0000 7f7398a2a640 10 monclient: init 2024-09-11T01:29:46.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.111+0000 7f7398a2a640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:46.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.111+0000 7f7398a2a640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-11T01:29:46.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.111+0000 7f7398a2a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7394134ad0 0x7f7394134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:46.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.111+0000 7f7398a2a640 10 monclient(hunting): picked mon.noname-a con 0x7f7394134ad0 addr v2:172.21.15.12:3300/0 2024-09-11T01:29:46.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.111+0000 7f7398a2a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f73940a2200 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:46.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.111+0000 7f7398a2a640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f73941353f0 0x7f7394134160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:46.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.111+0000 7f7398a2a640 10 monclient(hunting): picked mon.noname-c con 0x7f73941353f0 addr v2:172.21.15.71:3300/0 2024-09-11T01:29:46.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.111+0000 7f7398a2a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f739409ed30 con 0x7f73940a2200 2024-09-11T01:29:46.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.111+0000 7f7398a2a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:46.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.111+0000 7f7398a2a640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f739409f5d0 con 0x7f7394134ad0 2024-09-11T01:29:46.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.111+0000 7f7398a2a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:46.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.111+0000 7f7398a2a640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f739409f010 con 0x7f73941353f0 2024-09-11T01:29:46.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.111+0000 7f7398a2a640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:46.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.111+0000 7f7398a2a640 10 monclient(hunting): authenticate will time out at 792.553764s 2024-09-11T01:29:46.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.111+0000 7f739259c640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7f7394134ad0 msgr2=0x7f7394134eb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:29:46.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.111+0000 7f739259c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7394134ad0 0x7f7394134eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:29:46.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.112+0000 7f7392d9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f73941353f0 0x7f7394134160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:46.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.112+0000 7f7392d9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f73941353f0 0x7f7394134160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:44122/0 (socket says 172.21.15.12:44122) 2024-09-11T01:29:46.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.112+0000 7f7392d9d640 1 -- 172.21.15.12:0/2043794547 learned_addr learned my addr 172.21.15.12:0/2043794547 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:46.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.112+0000 7f7392d9d640 10 monclient(hunting): get_auth_request con 0x7f73941353f0 auth_method 0 2024-09-11T01:29:46.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.112+0000 7f7392d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:46.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.112+0000 7f7392d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:46.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.112+0000 7f7392d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:46.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.112+0000 7f7392d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:46.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.112+0000 7f7392d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:46.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.112+0000 7f7392d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:46.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.112+0000 7f7392d9d640 10 monclient(hunting): handle_auth_done global_id 4336 payload 274 2024-09-11T01:29:46.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.112+0000 7f7392d9d640 1 -- 172.21.15.12:0/2043794547 >> v2:172.21.15.12:3300/0 conn(0x7f7394134ad0 msgr2=0x7f7394134eb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:29:46.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.112+0000 7f7392d9d640 1 --2- 172.21.15.12:0/2043794547 >> v2:172.21.15.12:3300/0 conn(0x7f7394134ad0 0x7f7394134eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:46.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.112+0000 7f7392d9d640 1 -- 172.21.15.12:0/2043794547 >> v1:172.21.15.71:6789/0 conn(0x7f73940a2200 legacy=0x7f7394133a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:46.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.112+0000 7f7392d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:29:46.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.112+0000 7f7392d9d640 1 monclient: found mon.noname-c 2024-09-11T01:29:46.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.112+0000 7f7392d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:46.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.112+0000 7f7392d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:29:46.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.112+0000 7f7392d9d640 1 -- 172.21.15.12:0/2043794547 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f739409f2f0 con 0x7f73941353f0 2024-09-11T01:29:46.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.115+0000 7f7392d9d640 1 -- 172.21.15.12:0/2043794547 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73941353f0 msgr2=0x7f7394134160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:46.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.115+0000 7f7392d9d640 1 --2- 172.21.15.12:0/2043794547 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73941353f0 0x7f7394134160 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f738c009e50 tx=0x7f738c02a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0d685a9e46c6a51 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:46.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.115+0000 7f7391d9b640 1 -- 172.21.15.12:0/2043794547 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738c036070 con 0x7f73941353f0 2024-09-11T01:29:46.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.115+0000 7f7391d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:46.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.115+0000 7f7391d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:46.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.115+0000 7f7391d9b640 10 monclient: dump: 2024-09-11T01:29:46.125 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:46.125 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:46.126 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:46.126 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:46.126 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:46.126 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:46.126 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:46.126 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:46.126 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:46.126 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:46.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.116+0000 7f7391d9b640 10 monclient: _finish_auth 0 2024-09-11T01:29:46.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.116+0000 7f7391d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:29:46.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.116+0000 7f7391d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:46.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.116+0000 7f7391d9b640 1 -- 172.21.15.12:0/2043794547 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f738c007800 con 0x7f73941353f0 2024-09-11T01:29:46.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.116+0000 7f7391d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:46.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.116+0000 7f7391d9b640 1 -- 172.21.15.12:0/2043794547 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738c007bd0 con 0x7f73941353f0 2024-09-11T01:29:46.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.116+0000 7f7398a2a640 5 monclient: authenticate success, global_id 4336 2024-09-11T01:29:46.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.116+0000 7f7398a2a640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:46.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.116+0000 7f7398a2a640 4 set_mon_vals no callback set 2024-09-11T01:29:46.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.116+0000 7f7391d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:46.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.116+0000 7f7391d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:46.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.116+0000 7f7391d9b640 10 monclient: dump: 2024-09-11T01:29:46.127 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:46.127 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:46.127 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:46.128 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:46.128 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:46.128 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:46.128 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:46.128 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:46.128 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:46.128 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:46.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.116+0000 7f7398a2a640 10 monclient: shutdown 2024-09-11T01:29:46.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.116+0000 7f7398a2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:46.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.116+0000 7f7398a2a640 1 -- 172.21.15.12:0/2043794547 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73941353f0 msgr2=0x7f7394134160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:46.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.116+0000 7f7398a2a640 1 --2- 172.21.15.12:0/2043794547 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73941353f0 0x7f7394134160 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f738c009e50 tx=0x7f738c02a660 comp rx=0 tx=0).stop 2024-09-11T01:29:46.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.116+0000 7f7398a2a640 1 -- 172.21.15.12:0/2043794547 shutdown_connections 2024-09-11T01:29:46.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.116+0000 7f7398a2a640 1 --2- 172.21.15.12:0/2043794547 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73941353f0 0x7f7394134160 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:46.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.116+0000 7f7398a2a640 1 --2- 172.21.15.12:0/2043794547 >> v2:172.21.15.12:3300/0 conn(0x7f7394134ad0 0x7f7394134eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:46.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.116+0000 7f7398a2a640 1 -- 172.21.15.12:0/2043794547 >> 172.21.15.12:0/2043794547 conn(0x7f73940a2f70 msgr2=0x7f73940a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:46.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.116+0000 7f7398a2a640 1 -- 172.21.15.12:0/2043794547 shutdown_connections 2024-09-11T01:29:46.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.116+0000 7f7398a2a640 1 -- 172.21.15.12:0/2043794547 wait complete. 2024-09-11T01:29:46.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.117+0000 7f7398a2a640 10 monclient: build_initial_monmap 2024-09-11T01:29:46.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.117+0000 7f7398a2a640 10 monclient: monmap: 2024-09-11T01:29:46.129 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:46.129 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:46.129 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:46.129 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:46.129 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:46.129 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:46.129 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:46.129 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:46.129 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:46.130 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:46.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.117+0000 7f7398a2a640 1 Processor -- start 2024-09-11T01:29:46.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.117+0000 7f7398a2a640 1 -- start start 2024-09-11T01:29:46.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.117+0000 7f7398a2a640 10 monclient: init 2024-09-11T01:29:46.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7398a2a640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:46.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7398a2a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:29:46.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7398a2a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f73940a2200 0x7f7394072960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:46.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7398a2a640 10 monclient(hunting): picked mon.noname-c con 0x7f73940a2200 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:46.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7398a2a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7394134ad0 0x7f7394072ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:46.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7398a2a640 10 monclient(hunting): picked mon.noname-a con 0x7f7394134ad0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:46.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7398a2a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73941353f0 0x7f73940733e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:46.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7398a2a640 10 monclient(hunting): picked mon.noname-b con 0x7f73941353f0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:46.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7398a2a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:46.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7398a2a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f73941346a0 con 0x7f7394134ad0 2024-09-11T01:29:46.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7398a2a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:46.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7398a2a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f7394135920 con 0x7f73941353f0 2024-09-11T01:29:46.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7398a2a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:46.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7398a2a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7394134820 con 0x7f73940a2200 2024-09-11T01:29:46.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7398a2a640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:46.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7398a2a640 10 monclient(hunting): authenticate will time out at 792.560398s 2024-09-11T01:29:46.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7389d9b640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7394134ad0 msgr2=0x7f7394072ea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:29:46.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7389d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7394134ad0 0x7f7394072ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:29:46.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f739259c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f73940a2200 0x7f7394072960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:46.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f739259c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f73940a2200 0x7f7394072960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:53464/0 (socket says 172.21.15.12:53464) 2024-09-11T01:29:46.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f739259c640 1 -- 172.21.15.12:0/4074098013 learned_addr learned my addr 172.21.15.12:0/4074098013 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:46.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f739259c640 10 monclient(hunting): get_auth_request con 0x7f73940a2200 auth_method 0 2024-09-11T01:29:46.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f739259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:46.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f739259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:46.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f739259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:46.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7392d9d640 1 --2- 172.21.15.12:0/4074098013 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73941353f0 0x7f73940733e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:46.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7392d9d640 10 monclient(hunting): get_auth_request con 0x7f73941353f0 auth_method 0 2024-09-11T01:29:46.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7392d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:46.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7392d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:46.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7392d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:46.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f739259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f739259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f739259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7392d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7392d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7392d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f739259c640 10 monclient(hunting): handle_auth_done global_id 4403 payload 824 2024-09-11T01:29:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f739259c640 1 -- 172.21.15.12:0/4074098013 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73941353f0 msgr2=0x7f73940733e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f739259c640 1 --2- 172.21.15.12:0/4074098013 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73941353f0 0x7f73940733e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f739259c640 1 -- 172.21.15.12:0/4074098013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7394134ad0 msgr2=0x7f7394072ea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:29:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f739259c640 1 --2- 172.21.15.12:0/4074098013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7394134ad0 0x7f7394072ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f739259c640 10 monclient: _finish_hunting 0 2024-09-11T01:29:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f739259c640 1 monclient: found mon.noname-c 2024-09-11T01:29:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f739259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:46.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f739259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:29:46.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f739259c640 1 -- 172.21.15.12:0/4074098013 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7394133f00 con 0x7f73940a2200 2024-09-11T01:29:46.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.118+0000 7f7392d9d640 1 --2- 172.21.15.12:0/4074098013 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73941353f0 0x7f73940733e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:29:46.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.121+0000 7f739259c640 1 -- 172.21.15.12:0/4074098013 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f73940a2200 msgr2=0x7f7394072960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:46.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.121+0000 7f739259c640 1 --2- 172.21.15.12:0/4074098013 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f73940a2200 0x7f7394072960 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f738400b550 tx=0x7f738400ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:46.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.121+0000 7f738b7fe640 1 -- 172.21.15.12:0/4074098013 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7384005070 con 0x7f73940a2200 2024-09-11T01:29:46.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.121+0000 7f738b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:46.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.121+0000 7f738b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:46.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.121+0000 7f738b7fe640 10 monclient: dump: 2024-09-11T01:29:46.134 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:46.134 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:46.134 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:46.134 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:46.134 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:46.134 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:46.135 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:46.135 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:46.135 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:46.135 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:46.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.121+0000 7f738b7fe640 10 monclient: _finish_auth 0 2024-09-11T01:29:46.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.121+0000 7f738b7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:29:46.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.121+0000 7f738b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:46.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.121+0000 7f738b7fe640 1 -- 172.21.15.12:0/4074098013 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7384015070 con 0x7f73940a2200 2024-09-11T01:29:46.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.121+0000 7f7398a2a640 5 monclient: authenticate success, global_id 4403 2024-09-11T01:29:46.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.121+0000 7f7398a2a640 10 monclient: _renew_subs 2024-09-11T01:29:46.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.121+0000 7f7398a2a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:46.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.121+0000 7f7398a2a640 1 -- 172.21.15.12:0/4074098013 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73940a4460 con 0x7f73940a2200 2024-09-11T01:29:46.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.121+0000 7f738b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:46.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.121+0000 7f738b7fe640 1 -- 172.21.15.12:0/4074098013 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7384005450 con 0x7f73940a2200 2024-09-11T01:29:46.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.121+0000 7f738b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:46.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.121+0000 7f738b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:46.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.121+0000 7f738b7fe640 10 monclient: dump: 2024-09-11T01:29:46.136 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:46.136 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:46.136 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:46.137 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:46.137 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:46.137 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:46.137 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:46.137 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:46.137 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:46.137 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:46.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.121+0000 7f7398a2a640 10 monclient: _renew_subs 2024-09-11T01:29:46.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.121+0000 7f739159a640 4 set_mon_vals no callback set 2024-09-11T01:29:46.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.121+0000 7f7398a2a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:46.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.121+0000 7f7398a2a640 1 -- 172.21.15.12:0/4074098013 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f73941311d0 con 0x7f73940a2200 2024-09-11T01:29:46.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.122+0000 7f7398a2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:46.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.122+0000 7f7398a2a640 1 --2- 172.21.15.12:0/4074098013 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f73941cb020 0x7f73941cb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.122+0000 7f7398a2a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.122+0000 7f7398a2a640 1 -- 172.21.15.12:0/4074098013 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f739409c3e0 con 0x7f73941cb020 2024-09-11T01:29:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.122+0000 7f7389d9b640 1 --2- 172.21.15.12:0/4074098013 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f73941cb020 0x7f73941cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.122+0000 7f7389d9b640 10 monclient: get_auth_request con 0x7f73941cb020 auth_method 0 2024-09-11T01:29:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.122+0000 7f7389d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.122+0000 7f7389d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.122+0000 7f7389d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.122+0000 7f7389d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.122+0000 7f7389d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.122+0000 7f738b7fe640 1 -- 172.21.15.12:0/4074098013 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f7384005630 con 0x7f73940a2200 2024-09-11T01:29:46.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.122+0000 7f7389d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:46.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.122+0000 7f7389d9b640 10 monclient(hunting): handle_auth_done global_id 4406 payload 824 2024-09-11T01:29:46.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.122+0000 7f7389d9b640 1 -- 172.21.15.12:0/4074098013 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f73940a00f0 con 0x7f73941cb020 2024-09-11T01:29:46.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.122+0000 7f738b7fe640 1 --2- 172.21.15.12:0/4074098013 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f737403e1f0 0x7f73740406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:46.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.122+0000 7f738b7fe640 1 -- 172.21.15.12:0/4074098013 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4711+0+0 (secure 0 0 0) 0x7f7384052b10 con 0x7f73940a2200 2024-09-11T01:29:46.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.123+0000 7f7392d9d640 1 --2- 172.21.15.12:0/4074098013 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f737403e1f0 0x7f73740406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:46.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.123+0000 7f7392d9d640 10 monclient: get_auth_request con 0x7f737403e1f0 auth_method 0 2024-09-11T01:29:46.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.124+0000 7f7389d9b640 1 -- 172.21.15.12:0/4074098013 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f73941cb020 msgr2=0x7f73941cb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:46.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.124+0000 7f7389d9b640 1 --2- 172.21.15.12:0/4074098013 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f73941cb020 0x7f73941cb400 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f739412f460 tx=0x7f737c00c7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:46.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.124+0000 7f738b7fe640 1 -- 172.21.15.12:0/4074098013 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737c017070 con 0x7f73941cb020 2024-09-11T01:29:46.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.124+0000 7f738b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:46.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.124+0000 7f738b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:46.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.124+0000 7f738b7fe640 10 monclient: dump: 2024-09-11T01:29:46.142 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:46.142 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:46.142 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:46.142 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:46.142 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:46.142 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:46.142 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:46.142 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:46.142 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:46.142 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:46.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.125+0000 7f738b7fe640 1 -- 172.21.15.12:0/4074098013 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f737c0057c0 con 0x7f73941cb020 2024-09-11T01:29:46.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.125+0000 7f738b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:46.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.125+0000 7f738b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:46.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.125+0000 7f738b7fe640 1 -- 172.21.15.12:0/4074098013 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f73941cb020 msgr2=0x7f73941cb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:46.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.125+0000 7f738b7fe640 1 --2- 172.21.15.12:0/4074098013 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f73941cb020 0x7f73941cb400 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f739412f460 tx=0x7f737c00c7a0 comp rx=0 tx=0).stop 2024-09-11T01:29:46.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.125+0000 7f738b7fe640 1 -- 172.21.15.12:0/4074098013 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f73941cb020 msgr2=0x7f73941cb400 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:29:46.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.125+0000 7f738b7fe640 1 --2- 172.21.15.12:0/4074098013 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f73941cb020 0x7f73941cb400 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f739412f460 tx=0x7f737c00c7a0 comp rx=0 tx=0).stop 2024-09-11T01:29:46.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.125+0000 7f7392d9d640 1 --2- 172.21.15.12:0/4074098013 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f737403e1f0 0x7f73740406b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f73940a2110 tx=0x7f738c009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:46.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.131+0000 7f7398a2a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:29:46.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.131+0000 7f7398a2a640 1 --2- 172.21.15.12:0/4074098013 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f73941cb940 0x7f739406f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:46.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.131+0000 7f7398a2a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:46.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.131+0000 7f7398a2a640 1 -- 172.21.15.12:0/4074098013 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7394130cf0 con 0x7f73941cb940 2024-09-11T01:29:46.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.131+0000 7f7389d9b640 1 --2- 172.21.15.12:0/4074098013 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f73941cb940 0x7f739406f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:46.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.131+0000 7f7389d9b640 10 monclient: get_auth_request con 0x7f73941cb940 auth_method 0 2024-09-11T01:29:46.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.131+0000 7f7389d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:46.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.131+0000 7f7389d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:46.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.131+0000 7f7389d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:46.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.131+0000 7f7389d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:46.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.131+0000 7f7389d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:46.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.131+0000 7f7389d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:46.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.131+0000 7f7389d9b640 10 monclient(hunting): handle_auth_done global_id 4409 payload 824 2024-09-11T01:29:46.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.131+0000 7f7389d9b640 1 -- 172.21.15.12:0/4074098013 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f739406d480 con 0x7f73941cb940 2024-09-11T01:29:46.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.134+0000 7f7389d9b640 1 -- 172.21.15.12:0/4074098013 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f73941cb940 msgr2=0x7f739406f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:46.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.134+0000 7f7389d9b640 1 --2- 172.21.15.12:0/4074098013 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f73941cb940 0x7f739406f740 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f737c014000 tx=0x7f737c0037b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:46.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.134+0000 7f738b7fe640 1 -- 172.21.15.12:0/4074098013 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737c017040 con 0x7f73941cb940 2024-09-11T01:29:46.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.134+0000 7f738b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:46.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.134+0000 7f738b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:46.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.134+0000 7f738b7fe640 10 monclient: dump: 2024-09-11T01:29:46.145 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:46.145 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:46.145 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:46.145 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:46.145 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:46.145 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:46.145 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:46.145 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:46.145 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:46.145 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:46.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.134+0000 7f738b7fe640 1 -- 172.21.15.12:0/4074098013 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4696 (secure 0 0 0) 0x7f737c012400 con 0x7f73941cb940 2024-09-11T01:29:46.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.134+0000 7f738b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:29:46.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.134+0000 7f738b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:46.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.134+0000 7f738b7fe640 1 -- 172.21.15.12:0/4074098013 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f73941cb940 msgr2=0x7f739406f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:46.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.134+0000 7f738b7fe640 1 --2- 172.21.15.12:0/4074098013 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f73941cb940 0x7f739406f740 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f737c014000 tx=0x7f737c0037b0 comp rx=0 tx=0).stop 2024-09-11T01:29:46.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.134+0000 7f738b7fe640 1 -- 172.21.15.12:0/4074098013 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f73941cb940 msgr2=0x7f739406f740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:29:46.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.134+0000 7f738b7fe640 1 --2- 172.21.15.12:0/4074098013 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f73941cb940 0x7f739406f740 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f737c014000 tx=0x7f737c0037b0 comp rx=0 tx=0).stop 2024-09-11T01:29:46.146 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:29:46.147 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:29:46.147 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:29:46.147 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:29:46.147 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 10, 2024-09-11T01:29:46.147 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 48823, 2024-09-11T01:29:46.147 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:29:46.147 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:29:46.147 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:29:46.147 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:46.147 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:29:46.147 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:29:46.147 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:29:46.147 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:29:46.148 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:29:46.148 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:29:46.148 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:29:46.148 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:29:46.148 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:29:46.148 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:29:46.148 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:29:46.148 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:29:46.148 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:29:46.148 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:29:46.148 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:29:46.148 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:46.148 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:29:46.149 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:29:46.149 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:29:46.149 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:29:46.149 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:29:46.149 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:29:46.149 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:29:46.149 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:29:46.149 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:29:46.149 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:29:46.149 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:29:46.149 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:29:46.149 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:29:46.149 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:46.149 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:46.150 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:29:46.150 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:29:46.150 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:29:46.150 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:29:46.150 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:29:46.150 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:29:46.150 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:29:46.150 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:29:46.150 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:29:46.150 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:29:46.150 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:29:46.150 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:29:46.150 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:29:46.150 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:29:46.151 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:29:46.151 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:29:46.151 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:29:46.151 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:29:46.151 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:29:46.151 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:29:46.151 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:29:46.151 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:29:46.151 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:29:46.151 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:29:46.151 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:29:46.151 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:29:46.151 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:29:46.151 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:29:46.152 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:46.152 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:29:46.152 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:46.152 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:29:46.152 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:46.152 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:29:46.152 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:29:46.152 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:29:46.152 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:29:46.152 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:46.152 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:29:46.152 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:29:46.152 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:46.153 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:46.153 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:46.153 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:29:46.153 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:29:46.153 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:46.153 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:46.153 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:46.153 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:46.153 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:29:46.153 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:29:46.153 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:29:46.153 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:29:46.153 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:29:46.153 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:46.154 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:46.154 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:29:46.154 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:29:46.154 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:29:46.154 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:29:46.154 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:46.154 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:29:46.154 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:29:46.154 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:46.154 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:46.154 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:46.154 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:29:46.154 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:29:46.155 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:46.155 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:46.155 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:46.155 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:46.155 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:29:46.155 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:29:46.155 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:29:46.155 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:29:46.155 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:29:46.155 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:46.155 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:46.155 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:29:46.155 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:29:46.155 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:29:46.156 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:29:46.156 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:46.156 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:29:46.156 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:29:46.156 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:46.156 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:46.156 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:46.156 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:29:46.156 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:29:46.156 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:46.156 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:46.156 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:46.156 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:46.157 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:29:46.157 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:29:46.157 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:29:46.157 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:29:46.157 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:29:46.157 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:46.157 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:46.157 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:46.157 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:29:46.157 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:29:46.158 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:46.158 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:29:46.158 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:29:46.158 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:29:46.158 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:46.158 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:46.158 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:29:46.158 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:46.158 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:29:46.158 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:29:46.158 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:29:46.159 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:46.159 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:46.159 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:29:46.159 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:46.159 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:29:46.159 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:29:46.159 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:29:46.159 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:46.159 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:46.159 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:46.159 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:29:46.159 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:29:46.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.134+0000 7f7398a2a640 1 -- 172.21.15.12:0/4074098013 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f737403e1f0 msgr2=0x7f73740406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:46.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.134+0000 7f7398a2a640 1 --2- 172.21.15.12:0/4074098013 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f737403e1f0 0x7f73740406b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f73940a2110 tx=0x7f738c009000 comp rx=0 tx=0).stop 2024-09-11T01:29:46.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.134+0000 7f7398a2a640 10 monclient: shutdown 2024-09-11T01:29:46.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.134+0000 7f7398a2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:46.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.134+0000 7f7398a2a640 1 -- 172.21.15.12:0/4074098013 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f73940a2200 msgr2=0x7f7394072960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:46.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.134+0000 7f7398a2a640 1 --2- 172.21.15.12:0/4074098013 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f73940a2200 0x7f7394072960 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f738400b550 tx=0x7f738400ba10 comp rx=0 tx=0).stop 2024-09-11T01:29:46.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.134+0000 7f7392d9d640 1 -- 172.21.15.12:0/4074098013 reap_dead start 2024-09-11T01:29:46.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.134+0000 7f7392d9d640 1 -- 172.21.15.12:0/4074098013 reap_dead start 2024-09-11T01:29:46.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.134+0000 7f7398a2a640 1 -- 172.21.15.12:0/4074098013 shutdown_connections 2024-09-11T01:29:46.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.134+0000 7f7398a2a640 1 -- 172.21.15.12:0/4074098013 >> 172.21.15.12:0/4074098013 conn(0x7f73940a2f70 msgr2=0x7f739412df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:46.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.134+0000 7f7398a2a640 1 -- 172.21.15.12:0/4074098013 shutdown_connections 2024-09-11T01:29:46.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.134+0000 7f7398a2a640 1 -- 172.21.15.12:0/4074098013 wait complete. 2024-09-11T01:29:46.162 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:29:46.162 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T01:29:46.162 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:29:46.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.262+0000 7f1d26b94640 10 monclient: get_monmap_and_config 2024-09-11T01:29:46.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.262+0000 7f1d26b94640 10 monclient: build_initial_monmap 2024-09-11T01:29:46.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.262+0000 7f1d26b94640 10 monclient: monmap: 2024-09-11T01:29:46.264 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:46.264 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:46.264 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:29:46.263253+0000 2024-09-11T01:29:46.264 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:29:46.263253+0000 2024-09-11T01:29:46.264 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:46.265 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:46.265 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:29:46.265 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:29:46.265 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:29:46.265 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:29:46.265 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:29:46.265 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:46.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.263+0000 7f1d26b94640 1 Processor -- start 2024-09-11T01:29:46.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.263+0000 7f1d26b94640 1 -- start start 2024-09-11T01:29:46.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.263+0000 7f1d26b94640 10 monclient: init 2024-09-11T01:29:46.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.263+0000 7f1d26b94640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:46.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.263+0000 7f1d26b94640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-11T01:29:46.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.263+0000 7f1d26b94640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1d2006f540 addr v1:172.21.15.12:6789/0 2024-09-11T01:29:46.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.263+0000 7f1d26b94640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1d2009cff0 addr v1:172.21.15.71:6789/0 2024-09-11T01:29:46.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.263+0000 7f1d26b94640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1d2013e240 0x7f1d20140630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:46.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.263+0000 7f1d26b94640 10 monclient(hunting): picked mon.noname-b con 0x7f1d2013e240 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:46.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.263+0000 7f1d24930640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7f1d2006f540 legacy=0x7f1d200a5020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:29:46.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.263+0000 7f1d26b94640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d200a6ea0 con 0x7f1d2006f540 2024-09-11T01:29:46.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.263+0000 7f1d26b94640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d200a5e40 con 0x7f1d2009cff0 2024-09-11T01:29:46.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.263+0000 7f1d26b94640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:46.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.263+0000 7f1d26b94640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1d2006f920 con 0x7f1d2013e240 2024-09-11T01:29:46.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.264+0000 7f1d26b94640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:46.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.264+0000 7f1d26b94640 10 monclient(hunting): authenticate will time out at 792.706040s 2024-09-11T01:29:46.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.264+0000 7f1d25131640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1d2013e240 0x7f1d20140630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:46.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.264+0000 7f1d25131640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1d2013e240 0x7f1d20140630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:53488/0 (socket says 172.21.15.12:53488) 2024-09-11T01:29:46.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.264+0000 7f1d25131640 1 -- 172.21.15.12:0/2712402313 learned_addr learned my addr 172.21.15.12:0/2712402313 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:46.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.264+0000 7f1d25131640 10 monclient(hunting): get_auth_request con 0x7f1d2013e240 auth_method 0 2024-09-11T01:29:46.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.264+0000 7f1d25131640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:46.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.264+0000 7f1d25131640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:46.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.264+0000 7f1d25131640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:46.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.264+0000 7f1d25131640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:46.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.264+0000 7f1d25131640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:46.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.264+0000 7f1d25131640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:46.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.264+0000 7f1d25131640 10 monclient(hunting): handle_auth_done global_id 4412 payload 274 2024-09-11T01:29:46.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.264+0000 7f1d25131640 1 -- 172.21.15.12:0/2712402313 >> v1:172.21.15.71:6789/0 conn(0x7f1d2009cff0 legacy=0x7f1d200a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:29:46.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.264+0000 7f1d25131640 1 -- 172.21.15.12:0/2712402313 >> v1:172.21.15.12:6789/0 conn(0x7f1d2006f540 legacy=0x7f1d200a5020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:29:46.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.264+0000 7f1d25131640 10 monclient: _finish_hunting 0 2024-09-11T01:29:46.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.264+0000 7f1d25131640 1 monclient: found mon.noname-b 2024-09-11T01:29:46.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.264+0000 7f1d25131640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:46.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.264+0000 7f1d25131640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:29:46.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.264+0000 7f1d25131640 1 -- 172.21.15.12:0/2712402313 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d2009d4d0 con 0x7f1d2013e240 2024-09-11T01:29:46.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.267+0000 7f1d25131640 1 -- 172.21.15.12:0/2712402313 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1d2013e240 msgr2=0x7f1d20140630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:46.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.267+0000 7f1d25131640 1 --2- 172.21.15.12:0/2712402313 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1d2013e240 0x7f1d20140630 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1d18009e80 tx=0x7f1d1802a530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96400631b1afc61e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:46.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.267+0000 7f1d1f7fe640 1 -- 172.21.15.12:0/2712402313 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d18036070 con 0x7f1d2013e240 2024-09-11T01:29:46.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.267+0000 7f1d1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:46.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.267+0000 7f1d1f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:29:46.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.267+0000 7f1d1f7fe640 10 monclient: dump: 2024-09-11T01:29:46.277 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:46.277 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:46.277 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:46.277 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:46.277 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:46.277 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:46.277 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:46.277 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:46.278 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:46.278 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:46.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.267+0000 7f1d1f7fe640 10 monclient: _finish_auth 0 2024-09-11T01:29:46.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.267+0000 7f1d1f7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:29:46.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.267+0000 7f1d1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:46.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.267+0000 7f1d26b94640 5 monclient: authenticate success, global_id 4412 2024-09-11T01:29:46.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.267+0000 7f1d26b94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:29:46.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.267+0000 7f1d1f7fe640 1 -- 172.21.15.12:0/2712402313 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d18007800 con 0x7f1d2013e240 2024-09-11T01:29:46.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.267+0000 7f1d1f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:46.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.267+0000 7f1d26b94640 10 monclient: get_monmap_and_config success 2024-09-11T01:29:46.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.267+0000 7f1d26b94640 4 set_mon_vals no callback set 2024-09-11T01:29:46.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.267+0000 7f1d1f7fe640 1 -- 172.21.15.12:0/2712402313 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d18007bd0 con 0x7f1d2013e240 2024-09-11T01:29:46.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.267+0000 7f1d1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:46.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.267+0000 7f1d1f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:46.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.267+0000 7f1d1f7fe640 10 monclient: dump: 2024-09-11T01:29:46.279 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:46.279 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:46.279 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:46.279 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:46.279 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:46.279 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:46.279 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:46.279 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:46.279 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:46.279 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:46.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.267+0000 7f1d26b94640 10 monclient: shutdown 2024-09-11T01:29:46.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.267+0000 7f1d26b94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:46.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.267+0000 7f1d26b94640 1 -- 172.21.15.12:0/2712402313 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1d2013e240 msgr2=0x7f1d20140630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:46.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.267+0000 7f1d26b94640 1 --2- 172.21.15.12:0/2712402313 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1d2013e240 0x7f1d20140630 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1d18009e80 tx=0x7f1d1802a530 comp rx=0 tx=0).stop 2024-09-11T01:29:46.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.268+0000 7f1d26b94640 1 -- 172.21.15.12:0/2712402313 shutdown_connections 2024-09-11T01:29:46.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.268+0000 7f1d26b94640 1 --2- 172.21.15.12:0/2712402313 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1d2013e240 0x7f1d20140630 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:46.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.268+0000 7f1d26b94640 1 -- 172.21.15.12:0/2712402313 >> 172.21.15.12:0/2712402313 conn(0x7f1d200a7300 msgr2=0x7f1d200a7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:46.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.268+0000 7f1d26b94640 1 -- 172.21.15.12:0/2712402313 shutdown_connections 2024-09-11T01:29:46.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.268+0000 7f1d26b94640 1 -- 172.21.15.12:0/2712402313 wait complete. 2024-09-11T01:29:46.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.268+0000 7f1d26b94640 10 monclient: build_initial_monmap 2024-09-11T01:29:46.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.268+0000 7f1d26b94640 10 monclient: monmap: 2024-09-11T01:29:46.280 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:29:46.280 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:29:46.280 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:29:46.281 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:29:46.281 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:29:46.281 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:46.281 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:29:46.281 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:29:46.281 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:29:46.281 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:46.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.268+0000 7f1d26b94640 1 Processor -- start 2024-09-11T01:29:46.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.268+0000 7f1d26b94640 1 -- start start 2024-09-11T01:29:46.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.268+0000 7f1d26b94640 10 monclient: init 2024-09-11T01:29:46.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d26b94640 10 monclient: _reopen_session rank -1 2024-09-11T01:29:46.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d26b94640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:29:46.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d26b94640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d2006f540 0x7f1d201cd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:46.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d26b94640 10 monclient(hunting): picked mon.noname-a con 0x7f1d2006f540 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:29:46.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d26b94640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1d2009cff0 0x7f1d20138210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:46.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d26b94640 10 monclient(hunting): picked mon.noname-c con 0x7f1d2009cff0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:29:46.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d26b94640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d2013e240 0x7f1d2013a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:46.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d26b94640 10 monclient(hunting): picked mon.noname-b con 0x7f1d2013e240 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:29:46.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d26b94640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:46.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d26b94640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1d20140cf0 con 0x7f1d2006f540 2024-09-11T01:29:46.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d26b94640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:46.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d26b94640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1d20140b70 con 0x7f1d2013e240 2024-09-11T01:29:46.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d26b94640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:46.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d26b94640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1d20140e70 con 0x7f1d2009cff0 2024-09-11T01:29:46.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d26b94640 10 monclient(hunting): _renew_subs 2024-09-11T01:29:46.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d26b94640 10 monclient(hunting): authenticate will time out at 792.711548s 2024-09-11T01:29:46.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d24930640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d2006f540 msgr2=0x7f1d201cd240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:29:46.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d24930640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d2006f540 0x7f1d201cd240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:29:46.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d1ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1d2009cff0 0x7f1d20138210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:46.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d1ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1d2009cff0 0x7f1d20138210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:53492/0 (socket says 172.21.15.12:53492) 2024-09-11T01:29:46.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d1ffff640 1 -- 172.21.15.12:0/1175174374 learned_addr learned my addr 172.21.15.12:0/1175174374 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:29:46.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d1ffff640 10 monclient(hunting): get_auth_request con 0x7f1d2009cff0 auth_method 0 2024-09-11T01:29:46.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:46.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:46.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:46.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:46.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:46.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d25131640 1 --2- 172.21.15.12:0/1175174374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d2013e240 0x7f1d2013a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:46.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:46.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d25131640 10 monclient(hunting): get_auth_request con 0x7f1d2013e240 auth_method 0 2024-09-11T01:29:46.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d25131640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:46.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d25131640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:46.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d25131640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:46.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d25131640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:46.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d25131640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:46.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.269+0000 7f1d25131640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:46.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.270+0000 7f1d1ffff640 10 monclient(hunting): handle_auth_done global_id 4415 payload 824 2024-09-11T01:29:46.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.270+0000 7f1d1ffff640 1 -- 172.21.15.12:0/1175174374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d2013e240 msgr2=0x7f1d2013a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:46.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.270+0000 7f1d1ffff640 1 --2- 172.21.15.12:0/1175174374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d2013e240 0x7f1d2013a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:46.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.270+0000 7f1d1ffff640 1 -- 172.21.15.12:0/1175174374 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d2006f540 msgr2=0x7f1d201cd240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:29:46.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.270+0000 7f1d1ffff640 1 --2- 172.21.15.12:0/1175174374 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d2006f540 0x7f1d201cd240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:29:46.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.270+0000 7f1d1ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:29:46.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.270+0000 7f1d1ffff640 1 monclient: found mon.noname-c 2024-09-11T01:29:46.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.270+0000 7f1d1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:29:46.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.270+0000 7f1d1ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:29:46.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.270+0000 7f1d1ffff640 1 -- 172.21.15.12:0/1175174374 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d2013be00 con 0x7f1d2009cff0 2024-09-11T01:29:46.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.272+0000 7f1d1ffff640 1 -- 172.21.15.12:0/1175174374 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1d2009cff0 msgr2=0x7f1d20138210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:46.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.272+0000 7f1d1ffff640 1 --2- 172.21.15.12:0/1175174374 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1d2009cff0 0x7f1d20138210 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c00c430 tx=0x7f1d0c00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:46.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.272+0000 7f1d1d7fa640 1 -- 172.21.15.12:0/1175174374 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c0119a0 con 0x7f1d2009cff0 2024-09-11T01:29:46.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.272+0000 7f1d1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:46.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.272+0000 7f1d1d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:29:46.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.272+0000 7f1d1d7fa640 10 monclient: dump: 2024-09-11T01:29:46.285 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:46.285 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:46.285 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:46.285 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:46.285 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:46.286 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:46.286 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:46.286 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:46.286 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:46.286 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:46.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.272+0000 7f1d1d7fa640 10 monclient: _finish_auth 0 2024-09-11T01:29:46.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.272+0000 7f1d1d7fa640 10 monclient: _check_auth_tickets 2024-09-11T01:29:46.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.272+0000 7f1d1d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:29:46.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.272+0000 7f1d1d7fa640 1 -- 172.21.15.12:0/1175174374 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d0c017070 con 0x7f1d2009cff0 2024-09-11T01:29:46.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.272+0000 7f1d26b94640 5 monclient: authenticate success, global_id 4415 2024-09-11T01:29:46.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.272+0000 7f1d1d7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:29:46.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.272+0000 7f1d1d7fa640 1 -- 172.21.15.12:0/1175174374 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c011d80 con 0x7f1d2009cff0 2024-09-11T01:29:46.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.272+0000 7f1d26b94640 10 monclient: _renew_subs 2024-09-11T01:29:46.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.272+0000 7f1d26b94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:46.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.272+0000 7f1d26b94640 1 -- 172.21.15.12:0/1175174374 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d200a1810 con 0x7f1d2009cff0 2024-09-11T01:29:46.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.272+0000 7f1d1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:46.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.272+0000 7f1d1d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:29:46.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.272+0000 7f1d1d7fa640 10 monclient: dump: 2024-09-11T01:29:46.287 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:46.287 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:46.287 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:46.287 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:46.287 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:46.287 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:46.287 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:46.287 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:46.288 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:46.288 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:46.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.272+0000 7f1d1effd640 4 set_mon_vals no callback set 2024-09-11T01:29:46.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.272+0000 7f1d26b94640 10 monclient: _renew_subs 2024-09-11T01:29:46.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.272+0000 7f1d26b94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:29:46.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.272+0000 7f1d26b94640 1 -- 172.21.15.12:0/1175174374 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d20140320 con 0x7f1d2009cff0 2024-09-11T01:29:46.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.273+0000 7f1d26b94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:46.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.273+0000 7f1d26b94640 1 --2- 172.21.15.12:0/1175174374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d2013a880 0x7f1d2013ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:46.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.273+0000 7f1d26b94640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:46.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.273+0000 7f1d26b94640 1 -- 172.21.15.12:0/1175174374 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2006a060 con 0x7f1d2013a880 2024-09-11T01:29:46.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.273+0000 7f1d1d7fa640 1 -- 172.21.15.12:0/1175174374 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f1d0c01c460 con 0x7f1d2009cff0 2024-09-11T01:29:46.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.273+0000 7f1d24930640 1 --2- 172.21.15.12:0/1175174374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d2013a880 0x7f1d2013ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:46.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.273+0000 7f1d1d7fa640 1 --2- 172.21.15.12:0/1175174374 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1d0403e1c0 0x7f1d04040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:46.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.273+0000 7f1d24930640 10 monclient: get_auth_request con 0x7f1d2013a880 auth_method 0 2024-09-11T01:29:46.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.273+0000 7f1d24930640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:46.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.273+0000 7f1d24930640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:46.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.273+0000 7f1d24930640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:46.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.273+0000 7f1d1d7fa640 1 -- 172.21.15.12:0/1175174374 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4711+0+0 (secure 0 0 0) 0x7f1d0c052b10 con 0x7f1d2009cff0 2024-09-11T01:29:46.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.273+0000 7f1d24930640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:46.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.273+0000 7f1d24930640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:46.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.273+0000 7f1d24930640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:46.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.274+0000 7f1d25131640 1 --2- 172.21.15.12:0/1175174374 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1d0403e1c0 0x7f1d04040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:46.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.274+0000 7f1d25131640 10 monclient: get_auth_request con 0x7f1d0403e1c0 auth_method 0 2024-09-11T01:29:46.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.274+0000 7f1d24930640 10 monclient(hunting): handle_auth_done global_id 4345 payload 824 2024-09-11T01:29:46.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.274+0000 7f1d24930640 1 -- 172.21.15.12:0/1175174374 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1d2013b1a0 con 0x7f1d2013a880 2024-09-11T01:29:46.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.276+0000 7f1d24930640 1 -- 172.21.15.12:0/1175174374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d2013a880 msgr2=0x7f1d2013ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:46.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.276+0000 7f1d24930640 1 --2- 172.21.15.12:0/1175174374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d2013a880 0x7f1d2013ac60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1d1400b450 tx=0x7f1d1400b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:46.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.276+0000 7f1d25131640 1 --2- 172.21.15.12:0/1175174374 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1d0403e1c0 0x7f1d04040680 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1d2006cc40 tx=0x7f1d18009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:46.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.276+0000 7f1d1d7fa640 1 -- 172.21.15.12:0/1175174374 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d14005270 con 0x7f1d2013a880 2024-09-11T01:29:46.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.276+0000 7f1d1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:46.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.276+0000 7f1d1d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:46.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.276+0000 7f1d1d7fa640 10 monclient: dump: 2024-09-11T01:29:46.293 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:46.293 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:46.293 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:46.293 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:46.293 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:46.293 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:46.293 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:46.293 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:46.293 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:46.293 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:46.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.277+0000 7f1d1d7fa640 1 -- 172.21.15.12:0/1175174374 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1d14015070 con 0x7f1d2013a880 2024-09-11T01:29:46.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.277+0000 7f1d1d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:29:46.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.277+0000 7f1d1d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:29:46.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.277+0000 7f1d1d7fa640 1 -- 172.21.15.12:0/1175174374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d2013a880 msgr2=0x7f1d2013ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:46.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.277+0000 7f1d1d7fa640 1 --2- 172.21.15.12:0/1175174374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d2013a880 0x7f1d2013ac60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1d1400b450 tx=0x7f1d1400b910 comp rx=0 tx=0).stop 2024-09-11T01:29:46.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.277+0000 7f1d1d7fa640 1 -- 172.21.15.12:0/1175174374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d2013a880 msgr2=0x7f1d2013ac60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:29:46.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.277+0000 7f1d1d7fa640 1 --2- 172.21.15.12:0/1175174374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d2013a880 0x7f1d2013ac60 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1d1400b450 tx=0x7f1d1400b910 comp rx=0 tx=0).stop 2024-09-11T01:29:46.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.283+0000 7f1d26b94640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:29:46.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.283+0000 7f1d26b94640 1 --2- 172.21.15.12:0/1175174374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d2009e4e0 0x7f1d2009e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:29:46.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.283+0000 7f1d26b94640 10 monclient(hunting): start opening mon connection 2024-09-11T01:29:46.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.283+0000 7f1d26b94640 1 -- 172.21.15.12:0/1175174374 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2013ce20 con 0x7f1d2009e4e0 2024-09-11T01:29:46.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.283+0000 7f1d24930640 1 --2- 172.21.15.12:0/1175174374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d2009e4e0 0x7f1d2009e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:29:46.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.283+0000 7f1d24930640 10 monclient: get_auth_request con 0x7f1d2009e4e0 auth_method 0 2024-09-11T01:29:46.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.283+0000 7f1d24930640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:29:46.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.283+0000 7f1d24930640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:29:46.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.283+0000 7f1d24930640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:29:46.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.283+0000 7f1d24930640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:29:46.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.283+0000 7f1d24930640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:29:46.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.283+0000 7f1d24930640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:29:46.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.284+0000 7f1d24930640 10 monclient(hunting): handle_auth_done global_id 4348 payload 824 2024-09-11T01:29:46.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.284+0000 7f1d24930640 1 -- 172.21.15.12:0/1175174374 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1d20121400 con 0x7f1d2009e4e0 2024-09-11T01:29:46.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.286+0000 7f1d24930640 1 -- 172.21.15.12:0/1175174374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d2009e4e0 msgr2=0x7f1d2009e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:29:46.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.286+0000 7f1d24930640 1 --2- 172.21.15.12:0/1175174374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d2009e4e0 0x7f1d2009e8f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1d1400b450 tx=0x7f1d140059d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:29:46.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.286+0000 7f1d1d7fa640 1 -- 172.21.15.12:0/1175174374 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d14023410 con 0x7f1d2009e4e0 2024-09-11T01:29:46.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.286+0000 7f1d1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:29:46.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.286+0000 7f1d1d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:29:46.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.286+0000 7f1d1d7fa640 10 monclient: dump: 2024-09-11T01:29:46.295 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:29:46.295 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:29:46.296 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:46.296 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:29:46.296 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:29:46.296 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:29:46.296 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:29:46.296 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:29:46.296 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:29:46.296 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:29:46.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.286+0000 7f1d1d7fa640 1 -- 172.21.15.12:0/1175174374 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4865 (secure 0 0 0) 0x7f1d14023410 con 0x7f1d2009e4e0 2024-09-11T01:29:46.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.286+0000 7f1d1d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:29:46.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.286+0000 7f1d1d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:29:46.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.286+0000 7f1d1d7fa640 1 -- 172.21.15.12:0/1175174374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d2009e4e0 msgr2=0x7f1d2009e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:46.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.286+0000 7f1d1d7fa640 1 --2- 172.21.15.12:0/1175174374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d2009e4e0 0x7f1d2009e8f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1d1400b450 tx=0x7f1d140059d0 comp rx=0 tx=0).stop 2024-09-11T01:29:46.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.286+0000 7f1d1d7fa640 1 -- 172.21.15.12:0/1175174374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d2009e4e0 msgr2=0x7f1d2009e8f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:29:46.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.286+0000 7f1d1d7fa640 1 --2- 172.21.15.12:0/1175174374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d2009e4e0 0x7f1d2009e8f0 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1d1400b450 tx=0x7f1d140059d0 comp rx=0 tx=0).stop 2024-09-11T01:29:46.297 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:29:46.297 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:29:46.297 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:29:46.297 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:29:46.297 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 10, 2024-09-11T01:29:46.297 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 48940, 2024-09-11T01:29:46.297 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:29:46.298 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:29:46.298 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:29:46.298 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:46.298 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:29:46.298 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:29:46.298 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:29:46.298 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:29:46.298 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:29:46.298 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:29:46.298 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:29:46.298 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:29:46.298 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:29:46.298 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:29:46.299 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:29:46.299 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:29:46.299 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:29:46.299 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:29:46.299 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:29:46.299 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:46.299 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:29:46.299 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:29:46.299 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:29:46.299 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:29:46.299 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:29:46.299 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:29:46.299 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:29:46.300 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:29:46.300 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:29:46.300 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:29:46.300 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:29:46.300 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:29:46.300 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:29:46.300 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:46.300 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:46.300 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:29:46.300 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:29:46.300 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:29:46.300 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:29:46.300 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:29:46.301 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:29:46.301 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:29:46.301 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:29:46.301 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:29:46.301 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:29:46.301 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:29:46.301 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:29:46.301 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:29:46.301 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:29:46.301 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:29:46.301 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:29:46.301 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:29:46.301 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:29:46.301 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:29:46.302 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:29:46.302 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:29:46.302 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:29:46.302 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:29:46.302 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:29:46.302 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:29:46.302 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:29:46.302 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:29:46.302 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:29:46.302 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:46.302 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:29:46.302 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:46.302 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:29:46.302 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:46.303 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:29:46.303 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:29:46.303 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:29:46.303 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:29:46.303 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:46.303 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:29:46.303 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:29:46.303 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:46.303 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:46.303 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:46.303 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:29:46.303 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:29:46.303 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:46.303 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:46.304 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:46.304 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:46.304 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:29:46.304 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:29:46.304 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:29:46.304 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:29:46.304 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:29:46.304 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:46.304 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:46.304 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:29:46.304 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:29:46.304 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:29:46.304 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:29:46.305 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:46.305 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:29:46.305 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:29:46.305 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:46.305 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:46.305 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:46.305 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:29:46.305 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:29:46.305 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:46.305 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:46.305 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:46.305 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:46.305 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:29:46.305 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:29:46.305 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:29:46.306 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:29:46.306 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:29:46.306 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:46.306 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:46.306 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:29:46.306 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:29:46.306 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:29:46.306 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:29:46.306 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:46.306 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:29:46.306 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:29:46.306 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:46.306 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:46.306 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:46.307 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:29:46.307 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:29:46.307 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:29:46.307 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:46.307 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:46.307 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:46.307 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:29:46.307 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:29:46.307 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:29:46.307 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:29:46.307 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:29:46.307 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:46.307 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:46.308 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:46.308 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:29:46.308 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:29:46.308 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:46.308 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:29:46.308 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:29:46.308 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:29:46.308 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:46.308 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:46.308 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:29:46.308 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:46.308 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:29:46.308 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:29:46.309 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:29:46.309 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:46.309 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:46.309 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:29:46.309 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:46.309 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:29:46.309 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:29:46.309 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:29:46.309 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:46.309 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:29:46.309 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:29:46.309 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:29:46.309 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:29:46.309 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:29:46.310 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:29:46.310 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:29:46.310 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:29:46.310 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:29:46.310 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:29:46.310 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:29:46.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.287+0000 7f1d26b94640 1 -- 172.21.15.12:0/1175174374 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1d0403e1c0 msgr2=0x7f1d04040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:46.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.287+0000 7f1d26b94640 1 --2- 172.21.15.12:0/1175174374 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1d0403e1c0 0x7f1d04040680 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1d2006cc40 tx=0x7f1d18009000 comp rx=0 tx=0).stop 2024-09-11T01:29:46.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.287+0000 7f1d26b94640 10 monclient: shutdown 2024-09-11T01:29:46.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.287+0000 7f1d26b94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:29:46.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.287+0000 7f1d26b94640 1 -- 172.21.15.12:0/1175174374 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1d2009cff0 msgr2=0x7f1d20138210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:29:46.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.287+0000 7f1d26b94640 1 --2- 172.21.15.12:0/1175174374 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1d2009cff0 0x7f1d20138210 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c00c430 tx=0x7f1d0c00c8f0 comp rx=0 tx=0).stop 2024-09-11T01:29:46.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.287+0000 7f1d25131640 1 -- 172.21.15.12:0/1175174374 reap_dead start 2024-09-11T01:29:46.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.287+0000 7f1d25131640 1 -- 172.21.15.12:0/1175174374 reap_dead start 2024-09-11T01:29:46.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.287+0000 7f1d26b94640 1 -- 172.21.15.12:0/1175174374 shutdown_connections 2024-09-11T01:29:46.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.287+0000 7f1d26b94640 1 -- 172.21.15.12:0/1175174374 >> 172.21.15.12:0/1175174374 conn(0x7f1d200a7300 msgr2=0x7f1d200a3c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:29:46.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.287+0000 7f1d26b94640 1 -- 172.21.15.12:0/1175174374 shutdown_connections 2024-09-11T01:29:46.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:29:46.287+0000 7f1d26b94640 1 -- 172.21.15.12:0/1175174374 wait complete. 2024-09-11T01:29:46.312 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:29:46.313 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:29:46.785 DEBUG:teuthology.orchestra.run.smithi071:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0 2024-09-11T01:29:46.818 INFO:tasks.rados.rados.0.smithi071.stdout:adding op weight read -> 100 2024-09-11T01:29:46.819 INFO:tasks.rados.rados.0.smithi071.stdout:adding op weight write -> 50 2024-09-11T01:29:46.819 INFO:tasks.rados.rados.0.smithi071.stdout:adding op weight delete -> 50 2024-09-11T01:29:46.819 INFO:tasks.rados.rados.0.smithi071.stdout:adding op weight snap_create -> 50 2024-09-11T01:29:46.819 INFO:tasks.rados.rados.0.smithi071.stdout:adding op weight snap_remove -> 50 2024-09-11T01:29:46.819 INFO:tasks.rados.rados.0.smithi071.stdout:adding op weight rollback -> 50 2024-09-11T01:29:46.819 INFO:tasks.rados.rados.0.smithi071.stdout:adding op weight copy_from -> 50 2024-09-11T01:29:46.819 INFO:tasks.rados.rados.0.smithi071.stdout:adding op weight write_excl -> 50 2024-09-11T01:29:46.819 INFO:tasks.rados.rados.0.smithi071.stdout:ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev) 2024-09-11T01:29:46.819 INFO:tasks.rados.rados.0.smithi071.stdout:Configuration: 2024-09-11T01:29:46.819 INFO:tasks.rados.rados.0.smithi071.stdout: Number of operations: 4000 2024-09-11T01:29:46.819 INFO:tasks.rados.rados.0.smithi071.stdout: Number of objects: 50 2024-09-11T01:29:46.821 INFO:tasks.rados.rados.0.smithi071.stdout: Max in flight operations: 16 2024-09-11T01:29:46.821 INFO:tasks.rados.rados.0.smithi071.stdout: Object size (in bytes): 4000000 2024-09-11T01:29:46.821 INFO:tasks.rados.rados.0.smithi071.stdout: Write stride min: 400000 2024-09-11T01:29:46.821 INFO:tasks.rados.rados.0.smithi071.stdout: Write stride max: 800000 2024-09-11T01:29:47.771 INFO:tasks.rados.rados.0.smithi071.stdout:1: write initial oid 1 2024-09-11T01:29:47.771 INFO:tasks.rados.rados.0.smithi071.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2024-09-11T01:29:47.776 INFO:tasks.rados.rados.0.smithi071.stdout:1: writing smithi07134777-1 from 416807 to 892056 tid 1 2024-09-11T01:29:47.783 INFO:tasks.rados.rados.0.smithi071.stdout:1: writing smithi07134777-1 from 1542129 to 2016807 tid 2 2024-09-11T01:29:47.783 INFO:tasks.rados.rados.0.smithi071.stdout:2: write initial oid 2 2024-09-11T01:29:47.783 INFO:tasks.rados.rados.0.smithi071.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2024-09-11T01:29:47.788 INFO:tasks.rados.rados.0.smithi071.stdout:2: writing smithi07134777-2 from 416807 to 892056 tid 1 2024-09-11T01:29:47.795 INFO:tasks.rados.rados.0.smithi071.stdout:2: writing smithi07134777-2 from 1542129 to 2016807 tid 2 2024-09-11T01:29:47.795 INFO:tasks.rados.rados.0.smithi071.stdout:3: write initial oid 3 2024-09-11T01:29:47.795 INFO:tasks.rados.rados.0.smithi071.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2024-09-11T01:29:47.800 INFO:tasks.rados.rados.0.smithi071.stdout:3: writing smithi07134777-3 from 433614 to 984112 tid 1 2024-09-11T01:29:47.806 INFO:tasks.rados.rados.0.smithi071.stdout:3: writing smithi07134777-3 from 1600611 to 2033614 tid 2 2024-09-11T01:29:47.806 INFO:tasks.rados.rados.0.smithi071.stdout:4: write initial oid 4 2024-09-11T01:29:47.806 INFO:tasks.rados.rados.0.smithi071.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2024-09-11T01:29:47.813 INFO:tasks.rados.rados.0.smithi071.stdout:4: writing smithi07134777-4 from 450421 to 1076168 tid 1 2024-09-11T01:29:47.818 INFO:tasks.rados.rados.0.smithi071.stdout:4: writing smithi07134777-4 from 1659093 to 2050421 tid 2 2024-09-11T01:29:47.818 INFO:tasks.rados.rados.0.smithi071.stdout:5: write initial oid 5 2024-09-11T01:29:47.818 INFO:tasks.rados.rados.0.smithi071.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2024-09-11T01:29:47.825 INFO:tasks.rados.rados.0.smithi071.stdout:5: writing smithi07134777-5 from 467228 to 1168224 tid 1 2024-09-11T01:29:47.830 INFO:tasks.rados.rados.0.smithi071.stdout:5: writing smithi07134777-5 from 1717575 to 2067228 tid 2 2024-09-11T01:29:47.830 INFO:tasks.rados.rados.0.smithi071.stdout:6: write initial oid 6 2024-09-11T01:29:47.830 INFO:tasks.rados.rados.0.smithi071.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2024-09-11T01:29:47.837 INFO:tasks.rados.rados.0.smithi071.stdout:6: writing smithi07134777-6 from 484035 to 1260280 tid 1 2024-09-11T01:29:47.841 INFO:tasks.rados.rados.0.smithi071.stdout:6: writing smithi07134777-6 from 2059704 to 2084035 tid 2 2024-09-11T01:29:47.841 INFO:tasks.rados.rados.0.smithi071.stdout:7: write initial oid 7 2024-09-11T01:29:47.841 INFO:tasks.rados.rados.0.smithi071.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2024-09-11T01:29:47.846 INFO:tasks.rados.rados.0.smithi071.stdout:7: writing smithi07134777-7 from 500842 to 952336 tid 1 2024-09-11T01:29:47.853 INFO:tasks.rados.rados.0.smithi071.stdout:7: writing smithi07134777-7 from 1718186 to 2100842 tid 2 2024-09-11T01:29:47.854 INFO:tasks.rados.rados.0.smithi071.stdout:8: write initial oid 8 2024-09-11T01:29:47.854 INFO:tasks.rados.rados.0.smithi071.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2024-09-11T01:29:47.859 INFO:tasks.rados.rados.0.smithi071.stdout:8: writing smithi07134777-8 from 517649 to 1044392 tid 1 2024-09-11T01:29:47.865 INFO:tasks.rados.rados.0.smithi071.stdout:8: writing smithi07134777-8 from 1776668 to 2117649 tid 2 2024-09-11T01:29:47.865 INFO:tasks.rados.rados.0.smithi071.stdout:9: write initial oid 9 2024-09-11T01:29:47.865 INFO:tasks.rados.rados.0.smithi071.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2024-09-11T01:29:47.871 INFO:tasks.rados.rados.0.smithi071.stdout:9: writing smithi07134777-9 from 534456 to 1252801 tid 1 2024-09-11T01:29:47.876 INFO:tasks.rados.rados.0.smithi071.stdout:9: writing smithi07134777-9 from 1951503 to 2134456 tid 2 2024-09-11T01:29:47.876 INFO:tasks.rados.rados.0.smithi071.stdout:10: write initial oid 10 2024-09-11T01:29:47.876 INFO:tasks.rados.rados.0.smithi071.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2024-09-11T01:29:47.883 INFO:tasks.rados.rados.0.smithi071.stdout:10: writing smithi07134777-10 from 551263 to 1344857 tid 1 2024-09-11T01:29:47.888 INFO:tasks.rados.rados.0.smithi071.stdout:10: writing smithi07134777-10 from 1893632 to 2151263 tid 2 2024-09-11T01:29:47.888 INFO:tasks.rados.rados.0.smithi071.stdout:11: write initial oid 11 2024-09-11T01:29:47.888 INFO:tasks.rados.rados.0.smithi071.stdout:1: finishing write tid 1 to smithi07134777-1 2024-09-11T01:29:47.888 INFO:tasks.rados.rados.0.smithi071.stdout:1: finishing write tid 2 to smithi07134777-1 2024-09-11T01:29:47.889 INFO:tasks.rados.rados.0.smithi071.stdout:1: finishing write tid 3 to smithi07134777-1 2024-09-11T01:29:47.889 INFO:tasks.rados.rados.0.smithi071.stdout:1: finishing write tid 4 to smithi07134777-1 2024-09-11T01:29:47.889 INFO:tasks.rados.rados.0.smithi071.stdout:1: oid 1 updating version 0 to 2 2024-09-11T01:29:47.889 INFO:tasks.rados.rados.0.smithi071.stdout:1: oid 1 version 2 is already newer than 1 2024-09-11T01:29:47.889 INFO:tasks.rados.rados.0.smithi071.stdout:1: oid 1 updating version 2 to 3 2024-09-11T01:29:47.889 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-11T01:29:47.889 INFO:tasks.rados.rados.0.smithi071.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-09-11T01:29:47.889 INFO:tasks.rados.rados.0.smithi071.stdout:2: finishing write tid 1 to smithi07134777-2 2024-09-11T01:29:47.889 INFO:tasks.rados.rados.0.smithi071.stdout:2: finishing write tid 2 to smithi07134777-2 2024-09-11T01:29:47.889 INFO:tasks.rados.rados.0.smithi071.stdout:2: finishing write tid 3 to smithi07134777-2 2024-09-11T01:29:47.890 INFO:tasks.rados.rados.0.smithi071.stdout:2: finishing write tid 4 to smithi07134777-2 2024-09-11T01:29:47.890 INFO:tasks.rados.rados.0.smithi071.stdout:2: oid 2 updating version 0 to 3 2024-09-11T01:29:47.890 INFO:tasks.rados.rados.0.smithi071.stdout:2: oid 2 version 3 is already newer than 1 2024-09-11T01:29:47.890 INFO:tasks.rados.rados.0.smithi071.stdout:2: oid 2 version 3 is already newer than 2 2024-09-11T01:29:47.890 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-11T01:29:47.890 INFO:tasks.rados.rados.0.smithi071.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-09-11T01:29:47.890 INFO:tasks.rados.rados.0.smithi071.stdout:3: finishing write tid 1 to smithi07134777-3 2024-09-11T01:29:47.890 INFO:tasks.rados.rados.0.smithi071.stdout:3: finishing write tid 2 to smithi07134777-3 2024-09-11T01:29:47.890 INFO:tasks.rados.rados.0.smithi071.stdout:3: finishing write tid 3 to smithi07134777-3 2024-09-11T01:29:47.890 INFO:tasks.rados.rados.0.smithi071.stdout:3: finishing write tid 4 to smithi07134777-3 2024-09-11T01:29:47.890 INFO:tasks.rados.rados.0.smithi071.stdout:3: oid 3 updating version 0 to 2 2024-09-11T01:29:47.890 INFO:tasks.rados.rados.0.smithi071.stdout:3: oid 3 updating version 2 to 3 2024-09-11T01:29:47.890 INFO:tasks.rados.rados.0.smithi071.stdout:3: oid 3 version 3 is already newer than 1 2024-09-11T01:29:47.890 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-11T01:29:47.891 INFO:tasks.rados.rados.0.smithi071.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-09-11T01:29:47.891 INFO:tasks.rados.rados.0.smithi071.stdout:4: finishing write tid 1 to smithi07134777-4 2024-09-11T01:29:47.891 INFO:tasks.rados.rados.0.smithi071.stdout:4: finishing write tid 2 to smithi07134777-4 2024-09-11T01:29:47.891 INFO:tasks.rados.rados.0.smithi071.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2024-09-11T01:29:47.894 INFO:tasks.rados.rados.0.smithi071.stdout:11: writing smithi07134777-11 from 568070 to 1036913 tid 1 2024-09-11T01:29:47.900 INFO:tasks.rados.rados.0.smithi071.stdout:11: writing smithi07134777-11 from 1552114 to 2168070 tid 2 2024-09-11T01:29:47.900 INFO:tasks.rados.rados.0.smithi071.stdout:1: done (10 left) 2024-09-11T01:29:47.900 INFO:tasks.rados.rados.0.smithi071.stdout:2: done (9 left) 2024-09-11T01:29:47.900 INFO:tasks.rados.rados.0.smithi071.stdout:3: done (8 left) 2024-09-11T01:29:47.900 INFO:tasks.rados.rados.0.smithi071.stdout:12: write initial oid 12 2024-09-11T01:29:47.900 INFO:tasks.rados.rados.0.smithi071.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2024-09-11T01:29:47.906 INFO:tasks.rados.rados.0.smithi071.stdout:12: writing smithi07134777-12 from 584877 to 1128969 tid 1 2024-09-11T01:29:47.912 INFO:tasks.rados.rados.0.smithi071.stdout:12: writing smithi07134777-12 from 1610596 to 2172599 tid 2 2024-09-11T01:29:47.912 INFO:tasks.rados.rados.0.smithi071.stdout:13: write initial oid 13 2024-09-11T01:29:47.912 INFO:tasks.rados.rados.0.smithi071.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2024-09-11T01:29:47.918 INFO:tasks.rados.rados.0.smithi071.stdout:13: writing smithi07134777-13 from 601684 to 1221025 tid 1 2024-09-11T01:29:47.924 INFO:tasks.rados.rados.0.smithi071.stdout:13: writing smithi07134777-13 from 1669078 to 2201684 tid 2 2024-09-11T01:29:47.924 INFO:tasks.rados.rados.0.smithi071.stdout:14: write initial oid 14 2024-09-11T01:29:47.924 INFO:tasks.rados.rados.0.smithi071.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2024-09-11T01:29:47.931 INFO:tasks.rados.rados.0.smithi071.stdout:14: writing smithi07134777-14 from 618491 to 1313081 tid 1 2024-09-11T01:29:47.936 INFO:tasks.rados.rados.0.smithi071.stdout:14: writing smithi07134777-14 from 2011207 to 2218491 tid 2 2024-09-11T01:29:47.936 INFO:tasks.rados.rados.0.smithi071.stdout:15: write initial oid 15 2024-09-11T01:29:47.937 INFO:tasks.rados.rados.0.smithi071.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2024-09-11T01:29:47.944 INFO:tasks.rados.rados.0.smithi071.stdout:15: writing smithi07134777-15 from 635298 to 1405137 tid 1 2024-09-11T01:29:47.948 INFO:tasks.rados.rados.0.smithi071.stdout:15: writing smithi07134777-15 from 2069689 to 2235298 tid 2 2024-09-11T01:29:47.949 INFO:tasks.rados.rados.0.smithi071.stdout:16: write initial oid 16 2024-09-11T01:29:47.949 INFO:tasks.rados.rados.0.smithi071.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2024-09-11T01:29:47.954 INFO:tasks.rados.rados.0.smithi071.stdout:16: writing smithi07134777-16 from 652105 to 1097193 tid 1 2024-09-11T01:29:47.961 INFO:tasks.rados.rados.0.smithi071.stdout:16: writing smithi07134777-16 from 1728171 to 2181159 tid 2 2024-09-11T01:29:47.961 INFO:tasks.rados.rados.0.smithi071.stdout:17: write initial oid 17 2024-09-11T01:29:47.961 INFO:tasks.rados.rados.0.smithi071.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2024-09-11T01:29:47.967 INFO:tasks.rados.rados.0.smithi071.stdout:17: writing smithi07134777-17 from 668912 to 1305602 tid 1 2024-09-11T01:29:47.973 INFO:tasks.rados.rados.0.smithi071.stdout:17: writing smithi07134777-17 from 1903006 to 2268912 tid 2 2024-09-11T01:29:47.973 INFO:tasks.rados.rados.0.smithi071.stdout:18: write initial oid 18 2024-09-11T01:29:47.973 INFO:tasks.rados.rados.0.smithi071.stdout:4: finishing write tid 3 to smithi07134777-4 2024-09-11T01:29:47.973 INFO:tasks.rados.rados.0.smithi071.stdout:4: finishing write tid 4 to smithi07134777-4 2024-09-11T01:29:47.974 INFO:tasks.rados.rados.0.smithi071.stdout:4: oid 4 updating version 0 to 3 2024-09-11T01:29:47.974 INFO:tasks.rados.rados.0.smithi071.stdout:4: oid 4 version 3 is already newer than 2 2024-09-11T01:29:47.974 INFO:tasks.rados.rados.0.smithi071.stdout:4: oid 4 version 3 is already newer than 1 2024-09-11T01:29:47.974 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-11T01:29:47.974 INFO:tasks.rados.rados.0.smithi071.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-09-11T01:29:47.974 INFO:tasks.rados.rados.0.smithi071.stdout:5: finishing write tid 1 to smithi07134777-5 2024-09-11T01:29:47.974 INFO:tasks.rados.rados.0.smithi071.stdout:5: finishing write tid 2 to smithi07134777-5 2024-09-11T01:29:47.974 INFO:tasks.rados.rados.0.smithi071.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2024-09-11T01:29:47.981 INFO:tasks.rados.rados.0.smithi071.stdout:18: writing smithi07134777-18 from 685719 to 1397658 tid 1 2024-09-11T01:29:47.986 INFO:tasks.rados.rados.0.smithi071.stdout:18: writing smithi07134777-18 from 1845135 to 2285719 tid 2 2024-09-11T01:29:47.986 INFO:tasks.rados.rados.0.smithi071.stdout:4: done (14 left) 2024-09-11T01:29:47.986 INFO:tasks.rados.rados.0.smithi071.stdout:19: write initial oid 19 2024-09-11T01:29:47.986 INFO:tasks.rados.rados.0.smithi071.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2024-09-11T01:29:47.993 INFO:tasks.rados.rados.0.smithi071.stdout:19: writing smithi07134777-19 from 702526 to 1489714 tid 1 2024-09-11T01:29:47.998 INFO:tasks.rados.rados.0.smithi071.stdout:19: writing smithi07134777-19 from 1903617 to 2302526 tid 2 2024-09-11T01:29:47.998 INFO:tasks.rados.rados.0.smithi071.stdout:20: write initial oid 20 2024-09-11T01:29:47.998 INFO:tasks.rados.rados.0.smithi071.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2024-09-11T01:29:48.004 INFO:tasks.rados.rados.0.smithi071.stdout:20: writing smithi07134777-20 from 719333 to 1181770 tid 1 2024-09-11T01:29:48.011 INFO:tasks.rados.rados.0.smithi071.stdout:20: writing smithi07134777-20 from 1962099 to 2319333 tid 2 2024-09-11T01:29:48.011 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2024-09-11T01:29:48.011 INFO:tasks.rados.rados.0.smithi071.stdout:5: finishing write tid 3 to smithi07134777-5 2024-09-11T01:29:48.011 INFO:tasks.rados.rados.0.smithi071.stdout:5: finishing write tid 4 to smithi07134777-5 2024-09-11T01:29:48.011 INFO:tasks.rados.rados.0.smithi071.stdout:5: oid 5 updating version 0 to 1 2024-09-11T01:29:48.011 INFO:tasks.rados.rados.0.smithi071.stdout:5: oid 5 updating version 1 to 3 2024-09-11T01:29:48.011 INFO:tasks.rados.rados.0.smithi071.stdout:5: oid 5 version 3 is already newer than 2 2024-09-11T01:29:48.011 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-11T01:29:48.012 INFO:tasks.rados.rados.0.smithi071.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-09-11T01:29:48.012 INFO:tasks.rados.rados.0.smithi071.stdout:6: finishing write tid 1 to smithi07134777-6 2024-09-11T01:29:48.012 INFO:tasks.rados.rados.0.smithi071.stdout:5: done (15 left) 2024-09-11T01:29:48.012 INFO:tasks.rados.rados.0.smithi071.stdout:21: write initial oid 21 2024-09-11T01:29:48.012 INFO:tasks.rados.rados.0.smithi071.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2024-09-11T01:29:48.017 INFO:tasks.rados.rados.0.smithi071.stdout:21: writing smithi07134777-21 from 736140 to 1273826 tid 1 2024-09-11T01:29:48.024 INFO:tasks.rados.rados.0.smithi071.stdout:21: writing smithi07134777-21 from 2020581 to 2336140 tid 2 2024-09-11T01:29:48.024 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2024-09-11T01:29:48.024 INFO:tasks.rados.rados.0.smithi071.stdout:6: finishing write tid 2 to smithi07134777-6 2024-09-11T01:29:48.024 INFO:tasks.rados.rados.0.smithi071.stdout:6: finishing write tid 3 to smithi07134777-6 2024-09-11T01:29:48.024 INFO:tasks.rados.rados.0.smithi071.stdout:6: finishing write tid 4 to smithi07134777-6 2024-09-11T01:29:48.024 INFO:tasks.rados.rados.0.smithi071.stdout:6: oid 6 updating version 0 to 2 2024-09-11T01:29:48.024 INFO:tasks.rados.rados.0.smithi071.stdout:6: oid 6 updating version 2 to 3 2024-09-11T01:29:48.024 INFO:tasks.rados.rados.0.smithi071.stdout:6: oid 6 version 3 is already newer than 1 2024-09-11T01:29:48.024 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-11T01:29:48.024 INFO:tasks.rados.rados.0.smithi071.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-09-11T01:29:48.025 INFO:tasks.rados.rados.0.smithi071.stdout:7: finishing write tid 1 to smithi07134777-7 2024-09-11T01:29:48.025 INFO:tasks.rados.rados.0.smithi071.stdout:6: done (15 left) 2024-09-11T01:29:48.025 INFO:tasks.rados.rados.0.smithi071.stdout:22: write initial oid 22 2024-09-11T01:29:48.025 INFO:tasks.rados.rados.0.smithi071.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2024-09-11T01:29:48.031 INFO:tasks.rados.rados.0.smithi071.stdout:22: writing smithi07134777-22 from 752947 to 1365882 tid 1 2024-09-11T01:29:48.037 INFO:tasks.rados.rados.0.smithi071.stdout:22: writing smithi07134777-22 from 1962710 to 2352947 tid 2 2024-09-11T01:29:48.037 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2024-09-11T01:29:48.037 INFO:tasks.rados.rados.0.smithi071.stdout:7: finishing write tid 2 to smithi07134777-7 2024-09-11T01:29:48.037 INFO:tasks.rados.rados.0.smithi071.stdout:7: finishing write tid 3 to smithi07134777-7 2024-09-11T01:29:48.037 INFO:tasks.rados.rados.0.smithi071.stdout:7: finishing write tid 4 to smithi07134777-7 2024-09-11T01:29:48.037 INFO:tasks.rados.rados.0.smithi071.stdout:7: oid 7 updating version 0 to 2 2024-09-11T01:29:48.037 INFO:tasks.rados.rados.0.smithi071.stdout:7: oid 7 updating version 2 to 3 2024-09-11T01:29:48.037 INFO:tasks.rados.rados.0.smithi071.stdout:7: oid 7 version 3 is already newer than 1 2024-09-11T01:29:48.037 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-11T01:29:48.038 INFO:tasks.rados.rados.0.smithi071.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-09-11T01:29:48.038 INFO:tasks.rados.rados.0.smithi071.stdout:8: finishing write tid 1 to smithi07134777-8 2024-09-11T01:29:48.038 INFO:tasks.rados.rados.0.smithi071.stdout:8: finishing write tid 2 to smithi07134777-8 2024-09-11T01:29:48.038 INFO:tasks.rados.rados.0.smithi071.stdout:7: done (15 left) 2024-09-11T01:29:48.038 INFO:tasks.rados.rados.0.smithi071.stdout:23: write initial oid 23 2024-09-11T01:29:48.038 INFO:tasks.rados.rados.0.smithi071.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2024-09-11T01:29:48.044 INFO:tasks.rados.rados.0.smithi071.stdout:23: writing smithi07134777-23 from 769754 to 1457938 tid 1 2024-09-11T01:29:48.050 INFO:tasks.rados.rados.0.smithi071.stdout:23: writing smithi07134777-23 from 2021192 to 2369754 tid 2 2024-09-11T01:29:48.050 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2024-09-11T01:29:48.050 INFO:tasks.rados.rados.0.smithi071.stdout:8: finishing write tid 3 to smithi07134777-8 2024-09-11T01:29:48.050 INFO:tasks.rados.rados.0.smithi071.stdout:8: finishing write tid 4 to smithi07134777-8 2024-09-11T01:29:48.050 INFO:tasks.rados.rados.0.smithi071.stdout:8: oid 8 updating version 0 to 2 2024-09-11T01:29:48.050 INFO:tasks.rados.rados.0.smithi071.stdout:8: oid 8 updating version 2 to 3 2024-09-11T01:29:48.050 INFO:tasks.rados.rados.0.smithi071.stdout:8: oid 8 version 3 is already newer than 1 2024-09-11T01:29:48.051 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-11T01:29:48.051 INFO:tasks.rados.rados.0.smithi071.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-09-11T01:29:48.051 INFO:tasks.rados.rados.0.smithi071.stdout:9: finishing write tid 1 to smithi07134777-9 2024-09-11T01:29:48.051 INFO:tasks.rados.rados.0.smithi071.stdout:8: done (15 left) 2024-09-11T01:29:48.051 INFO:tasks.rados.rados.0.smithi071.stdout:24: write initial oid 24 2024-09-11T01:29:48.051 INFO:tasks.rados.rados.0.smithi071.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2024-09-11T01:29:48.056 INFO:tasks.rados.rados.0.smithi071.stdout:24: writing smithi07134777-24 from 786561 to 1266347 tid 1 2024-09-11T01:29:48.063 INFO:tasks.rados.rados.0.smithi071.stdout:24: writing smithi07134777-24 from 1796027 to 2263691 tid 2 2024-09-11T01:29:48.063 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2024-09-11T01:29:48.063 INFO:tasks.rados.rados.0.smithi071.stdout:9: finishing write tid 2 to smithi07134777-9 2024-09-11T01:29:48.063 INFO:tasks.rados.rados.0.smithi071.stdout:9: finishing write tid 3 to smithi07134777-9 2024-09-11T01:29:48.063 INFO:tasks.rados.rados.0.smithi071.stdout:9: finishing write tid 4 to smithi07134777-9 2024-09-11T01:29:48.063 INFO:tasks.rados.rados.0.smithi071.stdout:9: oid 9 updating version 0 to 1 2024-09-11T01:29:48.063 INFO:tasks.rados.rados.0.smithi071.stdout:9: oid 9 updating version 1 to 3 2024-09-11T01:29:48.063 INFO:tasks.rados.rados.0.smithi071.stdout:9: oid 9 version 3 is already newer than 2 2024-09-11T01:29:48.063 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-11T01:29:48.064 INFO:tasks.rados.rados.0.smithi071.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-09-11T01:29:48.064 INFO:tasks.rados.rados.0.smithi071.stdout:10: finishing write tid 1 to smithi07134777-10 2024-09-11T01:29:48.064 INFO:tasks.rados.rados.0.smithi071.stdout:10: finishing write tid 2 to smithi07134777-10 2024-09-11T01:29:48.064 INFO:tasks.rados.rados.0.smithi071.stdout:10: finishing write tid 3 to smithi07134777-10 2024-09-11T01:29:48.064 INFO:tasks.rados.rados.0.smithi071.stdout:10: finishing write tid 4 to smithi07134777-10 2024-09-11T01:29:48.064 INFO:tasks.rados.rados.0.smithi071.stdout:10: oid 10 updating version 0 to 3 2024-09-11T01:29:48.064 INFO:tasks.rados.rados.0.smithi071.stdout:10: oid 10 version 3 is already newer than 1 2024-09-11T01:29:48.064 INFO:tasks.rados.rados.0.smithi071.stdout:10: oid 10 version 3 is already newer than 2 2024-09-11T01:29:48.064 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-11T01:29:48.064 INFO:tasks.rados.rados.0.smithi071.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-09-11T01:29:48.065 INFO:tasks.rados.rados.0.smithi071.stdout:11: finishing write tid 1 to smithi07134777-11 2024-09-11T01:29:48.065 INFO:tasks.rados.rados.0.smithi071.stdout:9: done (15 left) 2024-09-11T01:29:48.065 INFO:tasks.rados.rados.0.smithi071.stdout:10: done (14 left) 2024-09-11T01:29:48.065 INFO:tasks.rados.rados.0.smithi071.stdout:25: write initial oid 25 2024-09-11T01:29:48.065 INFO:tasks.rados.rados.0.smithi071.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2024-09-11T01:29:48.067 INFO:tasks.rados.rados.0.smithi071.stdout:25: writing smithi07134777-25 from 403368 to 958403 tid 1 2024-09-11T01:29:48.074 INFO:tasks.rados.rados.0.smithi071.stdout:25: writing smithi07134777-25 from 1454509 to 2182184 tid 2 2024-09-11T01:29:48.074 INFO:tasks.rados.rados.0.smithi071.stdout:26: write initial oid 26 2024-09-11T01:29:48.074 INFO:tasks.rados.rados.0.smithi071.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2024-09-11T01:29:48.080 INFO:tasks.rados.rados.0.smithi071.stdout:26: writing smithi07134777-26 from 420175 to 1050459 tid 1 2024-09-11T01:29:48.088 INFO:tasks.rados.rados.0.smithi071.stdout:26: writing smithi07134777-26 from 1796638 to 2267971 tid 2 2024-09-11T01:29:48.088 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2024-09-11T01:29:48.088 INFO:tasks.rados.rados.0.smithi071.stdout:11: finishing write tid 2 to smithi07134777-11 2024-09-11T01:29:48.088 INFO:tasks.rados.rados.0.smithi071.stdout:11: finishing write tid 3 to smithi07134777-11 2024-09-11T01:29:48.088 INFO:tasks.rados.rados.0.smithi071.stdout:11: finishing write tid 4 to smithi07134777-11 2024-09-11T01:29:48.088 INFO:tasks.rados.rados.0.smithi071.stdout:11: oid 11 updating version 0 to 4 2024-09-11T01:29:48.088 INFO:tasks.rados.rados.0.smithi071.stdout:11: oid 11 updating version 4 to 5 2024-09-11T01:29:48.088 INFO:tasks.rados.rados.0.smithi071.stdout:11: oid 11 updating version 5 to 6 2024-09-11T01:29:48.088 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 6 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-11T01:29:48.088 INFO:tasks.rados.rados.0.smithi071.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-09-11T01:29:48.088 INFO:tasks.rados.rados.0.smithi071.stdout:12: finishing write tid 1 to smithi07134777-12 2024-09-11T01:29:48.088 INFO:tasks.rados.rados.0.smithi071.stdout:12: finishing write tid 2 to smithi07134777-12 2024-09-11T01:29:48.088 INFO:tasks.rados.rados.0.smithi071.stdout:12: finishing write tid 3 to smithi07134777-12 2024-09-11T01:29:48.089 INFO:tasks.rados.rados.0.smithi071.stdout:12: finishing write tid 4 to smithi07134777-12 2024-09-11T01:29:48.089 INFO:tasks.rados.rados.0.smithi071.stdout:12: oid 12 updating version 0 to 5 2024-09-11T01:29:48.089 INFO:tasks.rados.rados.0.smithi071.stdout:12: oid 12 version 5 is already newer than 4 2024-09-11T01:29:48.089 INFO:tasks.rados.rados.0.smithi071.stdout:12: oid 12 updating version 5 to 6 2024-09-11T01:29:48.089 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 6 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-11T01:29:48.089 INFO:tasks.rados.rados.0.smithi071.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-09-11T01:29:48.089 INFO:tasks.rados.rados.0.smithi071.stdout:13: finishing write tid 1 to smithi07134777-13 2024-09-11T01:29:48.089 INFO:tasks.rados.rados.0.smithi071.stdout:13: finishing write tid 2 to smithi07134777-13 2024-09-11T01:29:48.089 INFO:tasks.rados.rados.0.smithi071.stdout:13: finishing write tid 3 to smithi07134777-13 2024-09-11T01:29:48.089 INFO:tasks.rados.rados.0.smithi071.stdout:13: finishing write tid 4 to smithi07134777-13 2024-09-11T01:29:48.089 INFO:tasks.rados.rados.0.smithi071.stdout:13: oid 13 updating version 0 to 6 2024-09-11T01:29:48.089 INFO:tasks.rados.rados.0.smithi071.stdout:13: oid 13 version 6 is already newer than 4 2024-09-11T01:29:48.089 INFO:tasks.rados.rados.0.smithi071.stdout:13: oid 13 version 6 is already newer than 5 2024-09-11T01:29:48.090 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-11T01:29:48.090 INFO:tasks.rados.rados.0.smithi071.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-09-11T01:29:48.090 INFO:tasks.rados.rados.0.smithi071.stdout:14: finishing write tid 1 to smithi07134777-14 2024-09-11T01:29:48.090 INFO:tasks.rados.rados.0.smithi071.stdout:14: finishing write tid 2 to smithi07134777-14 2024-09-11T01:29:48.090 INFO:tasks.rados.rados.0.smithi071.stdout:14: finishing write tid 3 to smithi07134777-14 2024-09-11T01:29:48.090 INFO:tasks.rados.rados.0.smithi071.stdout:14: finishing write tid 4 to smithi07134777-14 2024-09-11T01:29:48.090 INFO:tasks.rados.rados.0.smithi071.stdout:14: oid 14 updating version 0 to 4 2024-09-11T01:29:48.090 INFO:tasks.rados.rados.0.smithi071.stdout:14: oid 14 updating version 4 to 5 2024-09-11T01:29:48.090 INFO:tasks.rados.rados.0.smithi071.stdout:14: oid 14 updating version 5 to 6 2024-09-11T01:29:48.090 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-11T01:29:48.090 INFO:tasks.rados.rados.0.smithi071.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-09-11T01:29:48.090 INFO:tasks.rados.rados.0.smithi071.stdout:15: finishing write tid 1 to smithi07134777-15 2024-09-11T01:29:48.090 INFO:tasks.rados.rados.0.smithi071.stdout:15: finishing write tid 2 to smithi07134777-15 2024-09-11T01:29:48.090 INFO:tasks.rados.rados.0.smithi071.stdout:15: finishing write tid 3 to smithi07134777-15 2024-09-11T01:29:48.090 INFO:tasks.rados.rados.0.smithi071.stdout:15: finishing write tid 4 to smithi07134777-15 2024-09-11T01:29:48.091 INFO:tasks.rados.rados.0.smithi071.stdout:15: oid 15 updating version 0 to 9 2024-09-11T01:29:48.091 INFO:tasks.rados.rados.0.smithi071.stdout:15: oid 15 version 9 is already newer than 8 2024-09-11T01:29:48.091 INFO:tasks.rados.rados.0.smithi071.stdout:15: oid 15 version 9 is already newer than 7 2024-09-11T01:29:48.091 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 9 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-11T01:29:48.091 INFO:tasks.rados.rados.0.smithi071.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-09-11T01:29:48.091 INFO:tasks.rados.rados.0.smithi071.stdout:16: finishing write tid 1 to smithi07134777-16 2024-09-11T01:29:48.091 INFO:tasks.rados.rados.0.smithi071.stdout:11: done (15 left) 2024-09-11T01:29:48.091 INFO:tasks.rados.rados.0.smithi071.stdout:12: done (14 left) 2024-09-11T01:29:48.091 INFO:tasks.rados.rados.0.smithi071.stdout:13: done (13 left) 2024-09-11T01:29:48.091 INFO:tasks.rados.rados.0.smithi071.stdout:14: done (12 left) 2024-09-11T01:29:48.091 INFO:tasks.rados.rados.0.smithi071.stdout:15: done (11 left) 2024-09-11T01:29:48.091 INFO:tasks.rados.rados.0.smithi071.stdout:27: write initial oid 27 2024-09-11T01:29:48.091 INFO:tasks.rados.rados.0.smithi071.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2024-09-11T01:29:48.092 INFO:tasks.rados.rados.0.smithi071.stdout:27: writing smithi07134777-27 from 436982 to 1142515 tid 1 2024-09-11T01:29:48.100 INFO:tasks.rados.rados.0.smithi071.stdout:27: writing smithi07134777-27 from 1855120 to 2436982 tid 2 2024-09-11T01:29:48.100 INFO:tasks.rados.rados.0.smithi071.stdout:28: write initial oid 28 2024-09-11T01:29:48.100 INFO:tasks.rados.rados.0.smithi071.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2024-09-11T01:29:48.106 INFO:tasks.rados.rados.0.smithi071.stdout:28: writing smithi07134777-28 from 453789 to 1234571 tid 1 2024-09-11T01:29:48.113 INFO:tasks.rados.rados.0.smithi071.stdout:28: writing smithi07134777-28 from 1913602 to 2388604 tid 2 2024-09-11T01:29:48.113 INFO:tasks.rados.rados.0.smithi071.stdout:29: write initial oid 29 2024-09-11T01:29:48.113 INFO:tasks.rados.rados.0.smithi071.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2024-09-11T01:29:48.118 INFO:tasks.rados.rados.0.smithi071.stdout:29: writing smithi07134777-29 from 470596 to 926627 tid 1 2024-09-11T01:29:48.125 INFO:tasks.rados.rados.0.smithi071.stdout:29: writing smithi07134777-29 from 1572084 to 2190744 tid 2 2024-09-11T01:29:48.125 INFO:tasks.rados.rados.0.smithi071.stdout:30: write initial oid 30 2024-09-11T01:29:48.125 INFO:tasks.rados.rados.0.smithi071.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2024-09-11T01:29:48.131 INFO:tasks.rados.rados.0.smithi071.stdout:30: writing smithi07134777-30 from 487403 to 1018683 tid 1 2024-09-11T01:29:48.136 INFO:tasks.rados.rados.0.smithi071.stdout:30: writing smithi07134777-30 from 1514213 to 1992884 tid 2 2024-09-11T01:29:48.136 INFO:tasks.rados.rados.0.smithi071.stdout:31: write initial oid 31 2024-09-11T01:29:48.136 INFO:tasks.rados.rados.0.smithi071.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2024-09-11T01:29:48.142 INFO:tasks.rados.rados.0.smithi071.stdout:31: writing smithi07134777-31 from 504210 to 1110739 tid 1 2024-09-11T01:29:48.149 INFO:tasks.rados.rados.0.smithi071.stdout:31: writing smithi07134777-31 from 1572695 to 2195024 tid 2 2024-09-11T01:29:48.149 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2024-09-11T01:29:48.149 INFO:tasks.rados.rados.0.smithi071.stdout:16: finishing write tid 2 to smithi07134777-16 2024-09-11T01:29:48.149 INFO:tasks.rados.rados.0.smithi071.stdout:16: finishing write tid 3 to smithi07134777-16 2024-09-11T01:29:48.150 INFO:tasks.rados.rados.0.smithi071.stdout:16: finishing write tid 4 to smithi07134777-16 2024-09-11T01:29:48.150 INFO:tasks.rados.rados.0.smithi071.stdout:16: oid 16 updating version 0 to 3 2024-09-11T01:29:48.150 INFO:tasks.rados.rados.0.smithi071.stdout:16: oid 16 version 3 is already newer than 1 2024-09-11T01:29:48.150 INFO:tasks.rados.rados.0.smithi071.stdout:16: oid 16 version 3 is already newer than 2 2024-09-11T01:29:48.150 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 3 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-11T01:29:48.150 INFO:tasks.rados.rados.0.smithi071.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-09-11T01:29:48.150 INFO:tasks.rados.rados.0.smithi071.stdout:17: finishing write tid 1 to smithi07134777-17 2024-09-11T01:29:48.150 INFO:tasks.rados.rados.0.smithi071.stdout:17: finishing write tid 2 to smithi07134777-17 2024-09-11T01:29:48.150 INFO:tasks.rados.rados.0.smithi071.stdout:16: done (15 left) 2024-09-11T01:29:48.150 INFO:tasks.rados.rados.0.smithi071.stdout:32: write initial oid 32 2024-09-11T01:29:48.150 INFO:tasks.rados.rados.0.smithi071.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2024-09-11T01:29:48.155 INFO:tasks.rados.rados.0.smithi071.stdout:32: writing smithi07134777-32 from 521017 to 1319148 tid 1 2024-09-11T01:29:48.161 INFO:tasks.rados.rados.0.smithi071.stdout:32: writing smithi07134777-32 from 1747530 to 2229870 tid 2 2024-09-11T01:29:48.161 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2024-09-11T01:29:48.161 INFO:tasks.rados.rados.0.smithi071.stdout:17: finishing write tid 3 to smithi07134777-17 2024-09-11T01:29:48.161 INFO:tasks.rados.rados.0.smithi071.stdout:17: finishing write tid 4 to smithi07134777-17 2024-09-11T01:29:48.161 INFO:tasks.rados.rados.0.smithi071.stdout:17: oid 17 updating version 0 to 7 2024-09-11T01:29:48.161 INFO:tasks.rados.rados.0.smithi071.stdout:17: oid 17 updating version 7 to 9 2024-09-11T01:29:48.161 INFO:tasks.rados.rados.0.smithi071.stdout:17: oid 17 version 9 is already newer than 8 2024-09-11T01:29:48.161 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 9 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-11T01:29:48.161 INFO:tasks.rados.rados.0.smithi071.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-09-11T01:29:48.161 INFO:tasks.rados.rados.0.smithi071.stdout:18: finishing write tid 1 to smithi07134777-18 2024-09-11T01:29:48.161 INFO:tasks.rados.rados.0.smithi071.stdout:18: finishing write tid 2 to smithi07134777-18 2024-09-11T01:29:48.162 INFO:tasks.rados.rados.0.smithi071.stdout:17: done (15 left) 2024-09-11T01:29:48.162 INFO:tasks.rados.rados.0.smithi071.stdout:33: write initial oid 33 2024-09-11T01:29:48.162 INFO:tasks.rados.rados.0.smithi071.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2024-09-11T01:29:48.165 INFO:tasks.rados.rados.0.smithi071.stdout:33: writing smithi07134777-33 from 537824 to 1011204 tid 1 2024-09-11T01:29:48.174 INFO:tasks.rados.rados.0.smithi071.stdout:33: writing smithi07134777-33 from 1806012 to 2432010 tid 2 2024-09-11T01:29:48.174 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2024-09-11T01:29:48.174 INFO:tasks.rados.rados.0.smithi071.stdout:18: finishing write tid 3 to smithi07134777-18 2024-09-11T01:29:48.174 INFO:tasks.rados.rados.0.smithi071.stdout:18: finishing write tid 4 to smithi07134777-18 2024-09-11T01:29:48.174 INFO:tasks.rados.rados.0.smithi071.stdout:18: oid 18 updating version 0 to 9 2024-09-11T01:29:48.174 INFO:tasks.rados.rados.0.smithi071.stdout:18: oid 18 version 9 is already newer than 7 2024-09-11T01:29:48.174 INFO:tasks.rados.rados.0.smithi071.stdout:18: oid 18 version 9 is already newer than 8 2024-09-11T01:29:48.174 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 9 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-11T01:29:48.174 INFO:tasks.rados.rados.0.smithi071.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-09-11T01:29:48.175 INFO:tasks.rados.rados.0.smithi071.stdout:19: finishing write tid 1 to smithi07134777-19 2024-09-11T01:29:48.175 INFO:tasks.rados.rados.0.smithi071.stdout:19: finishing write tid 2 to smithi07134777-19 2024-09-11T01:29:48.175 INFO:tasks.rados.rados.0.smithi071.stdout:18: done (15 left) 2024-09-11T01:29:48.175 INFO:tasks.rados.rados.0.smithi071.stdout:34: write initial oid 34 2024-09-11T01:29:48.175 INFO:tasks.rados.rados.0.smithi071.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2024-09-11T01:29:48.179 INFO:tasks.rados.rados.0.smithi071.stdout:34: writing smithi07134777-34 from 554631 to 1103260 tid 1 2024-09-11T01:29:48.186 INFO:tasks.rados.rados.0.smithi071.stdout:34: writing smithi07134777-34 from 1748141 to 2234150 tid 2 2024-09-11T01:29:48.186 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2024-09-11T01:29:48.186 INFO:tasks.rados.rados.0.smithi071.stdout:19: finishing write tid 3 to smithi07134777-19 2024-09-11T01:29:48.186 INFO:tasks.rados.rados.0.smithi071.stdout:19: finishing write tid 4 to smithi07134777-19 2024-09-11T01:29:48.186 INFO:tasks.rados.rados.0.smithi071.stdout:19: oid 19 updating version 0 to 5 2024-09-11T01:29:48.186 INFO:tasks.rados.rados.0.smithi071.stdout:19: oid 19 updating version 5 to 6 2024-09-11T01:29:48.186 INFO:tasks.rados.rados.0.smithi071.stdout:19: oid 19 version 6 is already newer than 4 2024-09-11T01:29:48.186 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-11T01:29:48.187 INFO:tasks.rados.rados.0.smithi071.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-09-11T01:29:48.187 INFO:tasks.rados.rados.0.smithi071.stdout:20: finishing write tid 1 to smithi07134777-20 2024-09-11T01:29:48.187 INFO:tasks.rados.rados.0.smithi071.stdout:19: done (15 left) 2024-09-11T01:29:48.187 INFO:tasks.rados.rados.0.smithi071.stdout:35: write initial oid 35 2024-09-11T01:29:48.187 INFO:tasks.rados.rados.0.smithi071.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2024-09-11T01:29:48.192 INFO:tasks.rados.rados.0.smithi071.stdout:35: writing smithi07134777-35 from 571438 to 1195316 tid 1 2024-09-11T01:29:48.199 INFO:tasks.rados.rados.0.smithi071.stdout:35: writing smithi07134777-35 from 1806623 to 2436290 tid 2 2024-09-11T01:29:48.199 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2024-09-11T01:29:48.199 INFO:tasks.rados.rados.0.smithi071.stdout:20: finishing write tid 2 to smithi07134777-20 2024-09-11T01:29:48.200 INFO:tasks.rados.rados.0.smithi071.stdout:20: finishing write tid 3 to smithi07134777-20 2024-09-11T01:29:48.200 INFO:tasks.rados.rados.0.smithi071.stdout:20: finishing write tid 4 to smithi07134777-20 2024-09-11T01:29:48.200 INFO:tasks.rados.rados.0.smithi071.stdout:20: oid 20 updating version 0 to 1 2024-09-11T01:29:48.200 INFO:tasks.rados.rados.0.smithi071.stdout:20: oid 20 updating version 1 to 2 2024-09-11T01:29:48.200 INFO:tasks.rados.rados.0.smithi071.stdout:20: oid 20 updating version 2 to 3 2024-09-11T01:29:48.200 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 3 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-11T01:29:48.200 INFO:tasks.rados.rados.0.smithi071.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-09-11T01:29:48.200 INFO:tasks.rados.rados.0.smithi071.stdout:21: finishing write tid 1 to smithi07134777-21 2024-09-11T01:29:48.200 INFO:tasks.rados.rados.0.smithi071.stdout:21: finishing write tid 2 to smithi07134777-21 2024-09-11T01:29:48.200 INFO:tasks.rados.rados.0.smithi071.stdout:20: done (15 left) 2024-09-11T01:29:48.200 INFO:tasks.rados.rados.0.smithi071.stdout:36: write initial oid 36 2024-09-11T01:29:48.200 INFO:tasks.rados.rados.0.smithi071.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2024-09-11T01:29:48.205 INFO:tasks.rados.rados.0.smithi071.stdout:36: writing smithi07134777-36 from 588245 to 1287372 tid 1 2024-09-11T01:29:48.212 INFO:tasks.rados.rados.0.smithi071.stdout:36: writing smithi07134777-36 from 1865105 to 2354783 tid 2 2024-09-11T01:29:48.212 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2024-09-11T01:29:48.212 INFO:tasks.rados.rados.0.smithi071.stdout:21: finishing write tid 3 to smithi07134777-21 2024-09-11T01:29:48.212 INFO:tasks.rados.rados.0.smithi071.stdout:21: finishing write tid 4 to smithi07134777-21 2024-09-11T01:29:48.212 INFO:tasks.rados.rados.0.smithi071.stdout:21: oid 21 updating version 0 to 4 2024-09-11T01:29:48.212 INFO:tasks.rados.rados.0.smithi071.stdout:21: oid 21 updating version 4 to 5 2024-09-11T01:29:48.212 INFO:tasks.rados.rados.0.smithi071.stdout:21: oid 21 updating version 5 to 6 2024-09-11T01:29:48.212 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 6 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-11T01:29:48.212 INFO:tasks.rados.rados.0.smithi071.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-09-11T01:29:48.213 INFO:tasks.rados.rados.0.smithi071.stdout:22: finishing write tid 1 to smithi07134777-22 2024-09-11T01:29:48.213 INFO:tasks.rados.rados.0.smithi071.stdout:22: finishing write tid 2 to smithi07134777-22 2024-09-11T01:29:48.213 INFO:tasks.rados.rados.0.smithi071.stdout:22: finishing write tid 3 to smithi07134777-22 2024-09-11T01:29:48.213 INFO:tasks.rados.rados.0.smithi071.stdout:22: finishing write tid 4 to smithi07134777-22 2024-09-11T01:29:48.213 INFO:tasks.rados.rados.0.smithi071.stdout:22: oid 22 updating version 0 to 5 2024-09-11T01:29:48.213 INFO:tasks.rados.rados.0.smithi071.stdout:22: oid 22 updating version 5 to 6 2024-09-11T01:29:48.213 INFO:tasks.rados.rados.0.smithi071.stdout:22: oid 22 version 6 is already newer than 4 2024-09-11T01:29:48.213 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 6 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-11T01:29:48.213 INFO:tasks.rados.rados.0.smithi071.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-09-11T01:29:48.213 INFO:tasks.rados.rados.0.smithi071.stdout:23: finishing write tid 1 to smithi07134777-23 2024-09-11T01:29:48.213 INFO:tasks.rados.rados.0.smithi071.stdout:21: done (15 left) 2024-09-11T01:29:48.213 INFO:tasks.rados.rados.0.smithi071.stdout:22: done (14 left) 2024-09-11T01:29:48.213 INFO:tasks.rados.rados.0.smithi071.stdout:37: write initial oid 37 2024-09-11T01:29:48.213 INFO:tasks.rados.rados.0.smithi071.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2024-09-11T01:29:48.219 INFO:tasks.rados.rados.0.smithi071.stdout:37: writing smithi07134777-37 from 605052 to 1379428 tid 1 2024-09-11T01:29:48.225 INFO:tasks.rados.rados.0.smithi071.stdout:37: writing smithi07134777-37 from 1923587 to 2556923 tid 2 2024-09-11T01:29:48.225 INFO:tasks.rados.rados.0.smithi071.stdout:38: write initial oid 38 2024-09-11T01:29:48.225 INFO:tasks.rados.rados.0.smithi071.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2024-09-11T01:29:48.231 INFO:tasks.rados.rados.0.smithi071.stdout:38: writing smithi07134777-38 from 621859 to 1071484 tid 1 2024-09-11T01:29:48.240 INFO:tasks.rados.rados.0.smithi071.stdout:38: writing smithi07134777-38 from 1865716 to 2621859 tid 2 2024-09-11T01:29:48.240 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2024-09-11T01:29:48.240 INFO:tasks.rados.rados.0.smithi071.stdout:23: finishing write tid 2 to smithi07134777-23 2024-09-11T01:29:48.240 INFO:tasks.rados.rados.0.smithi071.stdout:23: finishing write tid 3 to smithi07134777-23 2024-09-11T01:29:48.240 INFO:tasks.rados.rados.0.smithi071.stdout:23: finishing write tid 4 to smithi07134777-23 2024-09-11T01:29:48.240 INFO:tasks.rados.rados.0.smithi071.stdout:23: oid 23 updating version 0 to 6 2024-09-11T01:29:48.240 INFO:tasks.rados.rados.0.smithi071.stdout:23: oid 23 version 6 is already newer than 5 2024-09-11T01:29:48.240 INFO:tasks.rados.rados.0.smithi071.stdout:23: oid 23 version 6 is already newer than 4 2024-09-11T01:29:48.241 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-11T01:29:48.241 INFO:tasks.rados.rados.0.smithi071.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-09-11T01:29:48.241 INFO:tasks.rados.rados.0.smithi071.stdout:24: finishing write tid 1 to smithi07134777-24 2024-09-11T01:29:48.241 INFO:tasks.rados.rados.0.smithi071.stdout:23: done (15 left) 2024-09-11T01:29:48.241 INFO:tasks.rados.rados.0.smithi071.stdout:39: write initial oid 39 2024-09-11T01:29:48.241 INFO:tasks.rados.rados.0.smithi071.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2024-09-11T01:29:48.246 INFO:tasks.rados.rados.0.smithi071.stdout:39: writing smithi07134777-39 from 638666 to 1163540 tid 1 2024-09-11T01:29:48.254 INFO:tasks.rados.rados.0.smithi071.stdout:39: writing smithi07134777-39 from 1924198 to 2561203 tid 2 2024-09-11T01:29:48.254 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2024-09-11T01:29:48.254 INFO:tasks.rados.rados.0.smithi071.stdout:24: finishing write tid 2 to smithi07134777-24 2024-09-11T01:29:48.254 INFO:tasks.rados.rados.0.smithi071.stdout:24: finishing write tid 3 to smithi07134777-24 2024-09-11T01:29:48.254 INFO:tasks.rados.rados.0.smithi071.stdout:24: finishing write tid 4 to smithi07134777-24 2024-09-11T01:29:48.254 INFO:tasks.rados.rados.0.smithi071.stdout:24: oid 24 updating version 0 to 7 2024-09-11T01:29:48.255 INFO:tasks.rados.rados.0.smithi071.stdout:24: oid 24 updating version 7 to 8 2024-09-11T01:29:48.255 INFO:tasks.rados.rados.0.smithi071.stdout:24: oid 24 updating version 8 to 9 2024-09-11T01:29:48.255 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 9 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-11T01:29:48.255 INFO:tasks.rados.rados.0.smithi071.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-09-11T01:29:48.255 INFO:tasks.rados.rados.0.smithi071.stdout:25: finishing write tid 1 to smithi07134777-25 2024-09-11T01:29:48.255 INFO:tasks.rados.rados.0.smithi071.stdout:24: done (15 left) 2024-09-11T01:29:48.255 INFO:tasks.rados.rados.0.smithi071.stdout:40: write initial oid 40 2024-09-11T01:29:48.255 INFO:tasks.rados.rados.0.smithi071.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2024-09-11T01:29:48.261 INFO:tasks.rados.rados.0.smithi071.stdout:40: writing smithi07134777-40 from 655473 to 1371949 tid 1 2024-09-11T01:29:48.268 INFO:tasks.rados.rados.0.smithi071.stdout:40: writing smithi07134777-40 from 2099033 to 2655473 tid 2 2024-09-11T01:29:48.268 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2024-09-11T01:29:48.269 INFO:tasks.rados.rados.0.smithi071.stdout:25: finishing write tid 2 to smithi07134777-25 2024-09-11T01:29:48.269 INFO:tasks.rados.rados.0.smithi071.stdout:25: finishing write tid 3 to smithi07134777-25 2024-09-11T01:29:48.269 INFO:tasks.rados.rados.0.smithi071.stdout:25: finishing write tid 4 to smithi07134777-25 2024-09-11T01:29:48.269 INFO:tasks.rados.rados.0.smithi071.stdout:25: oid 25 updating version 0 to 5 2024-09-11T01:29:48.269 INFO:tasks.rados.rados.0.smithi071.stdout:25: oid 25 updating version 5 to 6 2024-09-11T01:29:48.269 INFO:tasks.rados.rados.0.smithi071.stdout:25: oid 25 version 6 is already newer than 4 2024-09-11T01:29:48.269 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 6 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-11T01:29:48.269 INFO:tasks.rados.rados.0.smithi071.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-09-11T01:29:48.269 INFO:tasks.rados.rados.0.smithi071.stdout:26: finishing write tid 1 to smithi07134777-26 2024-09-11T01:29:48.270 INFO:tasks.rados.rados.0.smithi071.stdout:26: finishing write tid 2 to smithi07134777-26 2024-09-11T01:29:48.270 INFO:tasks.rados.rados.0.smithi071.stdout:26: finishing write tid 3 to smithi07134777-26 2024-09-11T01:29:48.270 INFO:tasks.rados.rados.0.smithi071.stdout:26: finishing write tid 4 to smithi07134777-26 2024-09-11T01:29:48.270 INFO:tasks.rados.rados.0.smithi071.stdout:26: oid 26 updating version 0 to 9 2024-09-11T01:29:48.270 INFO:tasks.rados.rados.0.smithi071.stdout:26: oid 26 version 9 is already newer than 7 2024-09-11T01:29:48.270 INFO:tasks.rados.rados.0.smithi071.stdout:26: oid 26 version 9 is already newer than 8 2024-09-11T01:29:48.270 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 9 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-11T01:29:48.270 INFO:tasks.rados.rados.0.smithi071.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-09-11T01:29:48.270 INFO:tasks.rados.rados.0.smithi071.stdout:27: finishing write tid 1 to smithi07134777-27 2024-09-11T01:29:48.270 INFO:tasks.rados.rados.0.smithi071.stdout:27: finishing write tid 2 to smithi07134777-27 2024-09-11T01:29:48.270 INFO:tasks.rados.rados.0.smithi071.stdout:27: finishing write tid 3 to smithi07134777-27 2024-09-11T01:29:48.270 INFO:tasks.rados.rados.0.smithi071.stdout:25: done (15 left) 2024-09-11T01:29:48.270 INFO:tasks.rados.rados.0.smithi071.stdout:26: done (14 left) 2024-09-11T01:29:48.271 INFO:tasks.rados.rados.0.smithi071.stdout:41: write initial oid 41 2024-09-11T01:29:48.271 INFO:tasks.rados.rados.0.smithi071.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2024-09-11T01:29:48.276 INFO:tasks.rados.rados.0.smithi071.stdout:41: writing smithi07134777-41 from 672280 to 1464005 tid 1 2024-09-11T01:29:48.283 INFO:tasks.rados.rados.0.smithi071.stdout:41: writing smithi07134777-41 from 2157515 to 2672280 tid 2 2024-09-11T01:29:48.283 INFO:tasks.rados.rados.0.smithi071.stdout:42: write initial oid 42 2024-09-11T01:29:48.283 INFO:tasks.rados.rados.0.smithi071.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2024-09-11T01:29:48.289 INFO:tasks.rados.rados.0.smithi071.stdout:42: writing smithi07134777-42 from 689087 to 1156061 tid 1 2024-09-11T01:29:48.298 INFO:tasks.rados.rados.0.smithi071.stdout:42: writing smithi07134777-42 from 1699644 to 2483976 tid 2 2024-09-11T01:29:48.298 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2024-09-11T01:29:48.298 INFO:tasks.rados.rados.0.smithi071.stdout:27: finishing write tid 4 to smithi07134777-27 2024-09-11T01:29:48.298 INFO:tasks.rados.rados.0.smithi071.stdout:27: oid 27 updating version 0 to 10 2024-09-11T01:29:48.298 INFO:tasks.rados.rados.0.smithi071.stdout:27: oid 27 updating version 10 to 11 2024-09-11T01:29:48.298 INFO:tasks.rados.rados.0.smithi071.stdout:27: oid 27 updating version 11 to 12 2024-09-11T01:29:48.298 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 12 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-11T01:29:48.298 INFO:tasks.rados.rados.0.smithi071.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-09-11T01:29:48.298 INFO:tasks.rados.rados.0.smithi071.stdout:28: finishing write tid 1 to smithi07134777-28 2024-09-11T01:29:48.298 INFO:tasks.rados.rados.0.smithi071.stdout:28: finishing write tid 2 to smithi07134777-28 2024-09-11T01:29:48.299 INFO:tasks.rados.rados.0.smithi071.stdout:28: finishing write tid 3 to smithi07134777-28 2024-09-11T01:29:48.299 INFO:tasks.rados.rados.0.smithi071.stdout:28: finishing write tid 4 to smithi07134777-28 2024-09-11T01:29:48.299 INFO:tasks.rados.rados.0.smithi071.stdout:28: oid 28 updating version 0 to 11 2024-09-11T01:29:48.299 INFO:tasks.rados.rados.0.smithi071.stdout:28: oid 28 version 11 is already newer than 10 2024-09-11T01:29:48.299 INFO:tasks.rados.rados.0.smithi071.stdout:28: oid 28 updating version 11 to 12 2024-09-11T01:29:48.299 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 12 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-11T01:29:48.299 INFO:tasks.rados.rados.0.smithi071.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-09-11T01:29:48.299 INFO:tasks.rados.rados.0.smithi071.stdout:29: finishing write tid 1 to smithi07134777-29 2024-09-11T01:29:48.299 INFO:tasks.rados.rados.0.smithi071.stdout:29: finishing write tid 2 to smithi07134777-29 2024-09-11T01:29:48.299 INFO:tasks.rados.rados.0.smithi071.stdout:29: finishing write tid 3 to smithi07134777-29 2024-09-11T01:29:48.299 INFO:tasks.rados.rados.0.smithi071.stdout:29: finishing write tid 4 to smithi07134777-29 2024-09-11T01:29:48.299 INFO:tasks.rados.rados.0.smithi071.stdout:29: oid 29 updating version 0 to 10 2024-09-11T01:29:48.299 INFO:tasks.rados.rados.0.smithi071.stdout:29: oid 29 updating version 10 to 12 2024-09-11T01:29:48.300 INFO:tasks.rados.rados.0.smithi071.stdout:29: oid 29 version 12 is already newer than 11 2024-09-11T01:29:48.300 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 12 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-11T01:29:48.300 INFO:tasks.rados.rados.0.smithi071.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-09-11T01:29:48.300 INFO:tasks.rados.rados.0.smithi071.stdout:30: finishing write tid 1 to smithi07134777-30 2024-09-11T01:29:48.300 INFO:tasks.rados.rados.0.smithi071.stdout:30: finishing write tid 2 to smithi07134777-30 2024-09-11T01:29:48.300 INFO:tasks.rados.rados.0.smithi071.stdout:30: finishing write tid 3 to smithi07134777-30 2024-09-11T01:29:48.300 INFO:tasks.rados.rados.0.smithi071.stdout:30: finishing write tid 4 to smithi07134777-30 2024-09-11T01:29:48.300 INFO:tasks.rados.rados.0.smithi071.stdout:30: oid 30 updating version 0 to 12 2024-09-11T01:29:48.300 INFO:tasks.rados.rados.0.smithi071.stdout:30: oid 30 version 12 is already newer than 11 2024-09-11T01:29:48.300 INFO:tasks.rados.rados.0.smithi071.stdout:30: oid 30 version 12 is already newer than 10 2024-09-11T01:29:48.300 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 12 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-11T01:29:48.300 INFO:tasks.rados.rados.0.smithi071.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-09-11T01:29:48.300 INFO:tasks.rados.rados.0.smithi071.stdout:31: finishing write tid 1 to smithi07134777-31 2024-09-11T01:29:48.300 INFO:tasks.rados.rados.0.smithi071.stdout:27: done (15 left) 2024-09-11T01:29:48.301 INFO:tasks.rados.rados.0.smithi071.stdout:28: done (14 left) 2024-09-11T01:29:48.301 INFO:tasks.rados.rados.0.smithi071.stdout:29: done (13 left) 2024-09-11T01:29:48.301 INFO:tasks.rados.rados.0.smithi071.stdout:30: done (12 left) 2024-09-11T01:29:48.301 INFO:tasks.rados.rados.0.smithi071.stdout:43: write initial oid 43 2024-09-11T01:29:48.301 INFO:tasks.rados.rados.0.smithi071.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2024-09-11T01:29:48.303 INFO:tasks.rados.rados.0.smithi071.stdout:43: writing smithi07134777-43 from 705894 to 1248117 tid 1 2024-09-11T01:29:48.309 INFO:tasks.rados.rados.0.smithi071.stdout:43: writing smithi07134777-43 from 1758126 to 2402469 tid 2 2024-09-11T01:29:48.310 INFO:tasks.rados.rados.0.smithi071.stdout:44: write initial oid 44 2024-09-11T01:29:48.310 INFO:tasks.rados.rados.0.smithi071.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2024-09-11T01:29:48.316 INFO:tasks.rados.rados.0.smithi071.stdout:44: writing smithi07134777-44 from 722701 to 1340173 tid 1 2024-09-11T01:29:48.323 INFO:tasks.rados.rados.0.smithi071.stdout:44: writing smithi07134777-44 from 1816608 to 2604609 tid 2 2024-09-11T01:29:48.324 INFO:tasks.rados.rados.0.smithi071.stdout:45: write initial oid 45 2024-09-11T01:29:48.324 INFO:tasks.rados.rados.0.smithi071.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2024-09-11T01:29:48.331 INFO:tasks.rados.rados.0.smithi071.stdout:45: writing smithi07134777-45 from 739508 to 1432229 tid 1 2024-09-11T01:29:48.338 INFO:tasks.rados.rados.0.smithi071.stdout:45: writing smithi07134777-45 from 1875090 to 2523102 tid 2 2024-09-11T01:29:48.338 INFO:tasks.rados.rados.0.smithi071.stdout:46: write initial oid 46 2024-09-11T01:29:48.338 INFO:tasks.rados.rados.0.smithi071.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2024-09-11T01:29:48.346 INFO:tasks.rados.rados.0.smithi071.stdout:46: writing smithi07134777-46 from 756315 to 1524285 tid 1 2024-09-11T01:29:48.353 INFO:tasks.rados.rados.0.smithi071.stdout:46: writing smithi07134777-46 from 1933572 to 2725242 tid 2 2024-09-11T01:29:48.353 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2024-09-11T01:29:48.353 INFO:tasks.rados.rados.0.smithi071.stdout:31: finishing write tid 2 to smithi07134777-31 2024-09-11T01:29:48.353 INFO:tasks.rados.rados.0.smithi071.stdout:31: finishing write tid 3 to smithi07134777-31 2024-09-11T01:29:48.353 INFO:tasks.rados.rados.0.smithi071.stdout:31: finishing write tid 4 to smithi07134777-31 2024-09-11T01:29:48.353 INFO:tasks.rados.rados.0.smithi071.stdout:31: oid 31 updating version 0 to 11 2024-09-11T01:29:48.353 INFO:tasks.rados.rados.0.smithi071.stdout:31: oid 31 version 11 is already newer than 10 2024-09-11T01:29:48.354 INFO:tasks.rados.rados.0.smithi071.stdout:31: oid 31 updating version 11 to 12 2024-09-11T01:29:48.354 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 12 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-11T01:29:48.354 INFO:tasks.rados.rados.0.smithi071.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-09-11T01:29:48.354 INFO:tasks.rados.rados.0.smithi071.stdout:32: finishing write tid 1 to smithi07134777-32 2024-09-11T01:29:48.354 INFO:tasks.rados.rados.0.smithi071.stdout:31: done (15 left) 2024-09-11T01:29:48.354 INFO:tasks.rados.rados.0.smithi071.stdout:47: write initial oid 47 2024-09-11T01:29:48.354 INFO:tasks.rados.rados.0.smithi071.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2024-09-11T01:29:48.360 INFO:tasks.rados.rados.0.smithi071.stdout:47: writing smithi07134777-47 from 773122 to 1332694 tid 1 2024-09-11T01:29:48.368 INFO:tasks.rados.rados.0.smithi071.stdout:47: writing smithi07134777-47 from 1992054 to 2643735 tid 2 2024-09-11T01:29:48.368 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2024-09-11T01:29:48.368 INFO:tasks.rados.rados.0.smithi071.stdout:32: finishing write tid 2 to smithi07134777-32 2024-09-11T01:29:48.368 INFO:tasks.rados.rados.0.smithi071.stdout:32: finishing write tid 3 to smithi07134777-32 2024-09-11T01:29:48.368 INFO:tasks.rados.rados.0.smithi071.stdout:32: finishing write tid 4 to smithi07134777-32 2024-09-11T01:29:48.368 INFO:tasks.rados.rados.0.smithi071.stdout:32: oid 32 updating version 0 to 1 2024-09-11T01:29:48.368 INFO:tasks.rados.rados.0.smithi071.stdout:32: oid 32 updating version 1 to 3 2024-09-11T01:29:48.368 INFO:tasks.rados.rados.0.smithi071.stdout:32: oid 32 version 3 is already newer than 2 2024-09-11T01:29:48.368 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 3 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-11T01:29:48.368 INFO:tasks.rados.rados.0.smithi071.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-09-11T01:29:48.369 INFO:tasks.rados.rados.0.smithi071.stdout:33: finishing write tid 1 to smithi07134777-33 2024-09-11T01:29:48.369 INFO:tasks.rados.rados.0.smithi071.stdout:32: done (15 left) 2024-09-11T01:29:48.369 INFO:tasks.rados.rados.0.smithi071.stdout:48: write initial oid 48 2024-09-11T01:29:48.369 INFO:tasks.rados.rados.0.smithi071.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2024-09-11T01:29:48.375 INFO:tasks.rados.rados.0.smithi071.stdout:48: writing smithi07134777-48 from 789929 to 1424750 tid 1 2024-09-11T01:29:48.383 INFO:tasks.rados.rados.0.smithi071.stdout:48: writing smithi07134777-48 from 2050536 to 2789929 tid 2 2024-09-11T01:29:48.383 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2024-09-11T01:29:48.383 INFO:tasks.rados.rados.0.smithi071.stdout:33: finishing write tid 2 to smithi07134777-33 2024-09-11T01:29:48.383 INFO:tasks.rados.rados.0.smithi071.stdout:33: finishing write tid 3 to smithi07134777-33 2024-09-11T01:29:48.383 INFO:tasks.rados.rados.0.smithi071.stdout:33: finishing write tid 4 to smithi07134777-33 2024-09-11T01:29:48.383 INFO:tasks.rados.rados.0.smithi071.stdout:33: oid 33 updating version 0 to 13 2024-09-11T01:29:48.383 INFO:tasks.rados.rados.0.smithi071.stdout:33: oid 33 updating version 13 to 15 2024-09-11T01:29:48.383 INFO:tasks.rados.rados.0.smithi071.stdout:33: oid 33 version 15 is already newer than 14 2024-09-11T01:29:48.384 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 15 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-11T01:29:48.384 INFO:tasks.rados.rados.0.smithi071.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-09-11T01:29:48.384 INFO:tasks.rados.rados.0.smithi071.stdout:34: finishing write tid 1 to smithi07134777-34 2024-09-11T01:29:48.384 INFO:tasks.rados.rados.0.smithi071.stdout:34: finishing write tid 2 to smithi07134777-34 2024-09-11T01:29:48.384 INFO:tasks.rados.rados.0.smithi071.stdout:34: finishing write tid 3 to smithi07134777-34 2024-09-11T01:29:48.384 INFO:tasks.rados.rados.0.smithi071.stdout:34: finishing write tid 4 to smithi07134777-34 2024-09-11T01:29:48.384 INFO:tasks.rados.rados.0.smithi071.stdout:34: oid 34 updating version 0 to 17 2024-09-11T01:29:48.385 INFO:tasks.rados.rados.0.smithi071.stdout:34: oid 34 updating version 17 to 18 2024-09-11T01:29:48.385 INFO:tasks.rados.rados.0.smithi071.stdout:34: oid 34 version 18 is already newer than 16 2024-09-11T01:29:48.385 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 18 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-11T01:29:48.385 INFO:tasks.rados.rados.0.smithi071.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-09-11T01:29:48.385 INFO:tasks.rados.rados.0.smithi071.stdout:35: finishing write tid 1 to smithi07134777-35 2024-09-11T01:29:48.385 INFO:tasks.rados.rados.0.smithi071.stdout:33: done (15 left) 2024-09-11T01:29:48.385 INFO:tasks.rados.rados.0.smithi071.stdout:34: done (14 left) 2024-09-11T01:29:48.385 INFO:tasks.rados.rados.0.smithi071.stdout:49: write initial oid 49 2024-09-11T01:29:48.385 INFO:tasks.rados.rados.0.smithi071.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2024-09-11T01:29:48.389 INFO:tasks.rados.rados.0.smithi071.stdout:49: writing smithi07134777-49 from 406736 to 1116806 tid 1 2024-09-11T01:29:48.395 INFO:tasks.rados.rados.0.smithi071.stdout:49: writing smithi07134777-49 from 1709018 to 2364368 tid 2 2024-09-11T01:29:48.395 INFO:tasks.rados.rados.0.smithi071.stdout:50: write initial oid 50 2024-09-11T01:29:48.395 INFO:tasks.rados.rados.0.smithi071.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2024-09-11T01:29:48.402 INFO:tasks.rados.rados.0.smithi071.stdout:50: writing smithi07134777-50 from 423543 to 1208862 tid 1 2024-09-11T01:29:48.413 INFO:tasks.rados.rados.0.smithi071.stdout:50: writing smithi07134777-50 from 1767500 to 2566508 tid 2 2024-09-11T01:29:48.413 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2024-09-11T01:29:48.413 INFO:tasks.rados.rados.0.smithi071.stdout:35: finishing write tid 2 to smithi07134777-35 2024-09-11T01:29:48.413 INFO:tasks.rados.rados.0.smithi071.stdout:35: finishing write tid 3 to smithi07134777-35 2024-09-11T01:29:48.413 INFO:tasks.rados.rados.0.smithi071.stdout:35: finishing write tid 4 to smithi07134777-35 2024-09-11T01:29:48.414 INFO:tasks.rados.rados.0.smithi071.stdout:35: oid 35 updating version 0 to 4 2024-09-11T01:29:48.414 INFO:tasks.rados.rados.0.smithi071.stdout:35: oid 35 updating version 4 to 6 2024-09-11T01:29:48.414 INFO:tasks.rados.rados.0.smithi071.stdout:35: oid 35 version 6 is already newer than 5 2024-09-11T01:29:48.414 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 6 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-11T01:29:48.414 INFO:tasks.rados.rados.0.smithi071.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-09-11T01:29:48.414 INFO:tasks.rados.rados.0.smithi071.stdout:36: finishing write tid 1 to smithi07134777-36 2024-09-11T01:29:48.414 INFO:tasks.rados.rados.0.smithi071.stdout:36: finishing write tid 2 to smithi07134777-36 2024-09-11T01:29:48.414 INFO:tasks.rados.rados.0.smithi071.stdout:35: done (15 left) 2024-09-11T01:29:48.414 INFO:tasks.rados.rados.0.smithi071.stdout:51: delete oid 25 current snap is 0 2024-09-11T01:29:48.414 INFO:tasks.rados.rados.0.smithi071.stdout:36: finishing write tid 3 to smithi07134777-36 2024-09-11T01:29:48.414 INFO:tasks.rados.rados.0.smithi071.stdout:36: finishing write tid 4 to smithi07134777-36 2024-09-11T01:29:48.414 INFO:tasks.rados.rados.0.smithi071.stdout:36: oid 36 updating version 0 to 9 2024-09-11T01:29:48.415 INFO:tasks.rados.rados.0.smithi071.stdout:36: oid 36 version 9 is already newer than 8 2024-09-11T01:29:48.415 INFO:tasks.rados.rados.0.smithi071.stdout:36: oid 36 version 9 is already newer than 7 2024-09-11T01:29:48.415 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 9 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-11T01:29:48.415 INFO:tasks.rados.rados.0.smithi071.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-09-11T01:29:48.415 INFO:tasks.rados.rados.0.smithi071.stdout:37: finishing write tid 1 to smithi07134777-37 2024-09-11T01:29:48.415 INFO:tasks.rados.rados.0.smithi071.stdout:37: finishing write tid 2 to smithi07134777-37 2024-09-11T01:29:48.415 INFO:tasks.rados.rados.0.smithi071.stdout:37: finishing write tid 3 to smithi07134777-37 2024-09-11T01:29:48.415 INFO:tasks.rados.rados.0.smithi071.stdout:37: finishing write tid 4 to smithi07134777-37 2024-09-11T01:29:48.415 INFO:tasks.rados.rados.0.smithi071.stdout:37: oid 37 updating version 0 to 9 2024-09-11T01:29:48.415 INFO:tasks.rados.rados.0.smithi071.stdout:37: oid 37 version 9 is already newer than 8 2024-09-11T01:29:48.415 INFO:tasks.rados.rados.0.smithi071.stdout:37: oid 37 version 9 is already newer than 7 2024-09-11T01:29:48.415 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 9 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-11T01:29:48.415 INFO:tasks.rados.rados.0.smithi071.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-09-11T01:29:48.415 INFO:tasks.rados.rados.0.smithi071.stdout:38: finishing write tid 1 to smithi07134777-38 2024-09-11T01:29:48.416 INFO:tasks.rados.rados.0.smithi071.stdout:38: finishing write tid 2 to smithi07134777-38 2024-09-11T01:29:48.416 INFO:tasks.rados.rados.0.smithi071.stdout:38: finishing write tid 3 to smithi07134777-38 2024-09-11T01:29:48.416 INFO:tasks.rados.rados.0.smithi071.stdout:38: finishing write tid 4 to smithi07134777-38 2024-09-11T01:29:48.416 INFO:tasks.rados.rados.0.smithi071.stdout:38: oid 38 updating version 0 to 14 2024-09-11T01:29:48.416 INFO:tasks.rados.rados.0.smithi071.stdout:38: oid 38 version 14 is already newer than 13 2024-09-11T01:29:48.416 INFO:tasks.rados.rados.0.smithi071.stdout:38: oid 38 updating version 14 to 15 2024-09-11T01:29:48.416 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 15 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-11T01:29:48.416 INFO:tasks.rados.rados.0.smithi071.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-09-11T01:29:48.416 INFO:tasks.rados.rados.0.smithi071.stdout:39: finishing write tid 1 to smithi07134777-39 2024-09-11T01:29:48.416 INFO:tasks.rados.rados.0.smithi071.stdout:39: finishing write tid 2 to smithi07134777-39 2024-09-11T01:29:48.416 INFO:tasks.rados.rados.0.smithi071.stdout:39: finishing write tid 3 to smithi07134777-39 2024-09-11T01:29:48.416 INFO:tasks.rados.rados.0.smithi071.stdout:39: finishing write tid 4 to smithi07134777-39 2024-09-11T01:29:48.416 INFO:tasks.rados.rados.0.smithi071.stdout:39: oid 39 updating version 0 to 12 2024-09-11T01:29:48.417 INFO:tasks.rados.rados.0.smithi071.stdout:39: oid 39 version 12 is already newer than 11 2024-09-11T01:29:48.417 INFO:tasks.rados.rados.0.smithi071.stdout:39: oid 39 version 12 is already newer than 10 2024-09-11T01:29:48.417 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 12 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-11T01:29:48.417 INFO:tasks.rados.rados.0.smithi071.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-09-11T01:29:48.417 INFO:tasks.rados.rados.0.smithi071.stdout:40: finishing write tid 1 to smithi07134777-40 2024-09-11T01:29:48.417 INFO:tasks.rados.rados.0.smithi071.stdout:40: finishing write tid 2 to smithi07134777-40 2024-09-11T01:29:48.417 INFO:tasks.rados.rados.0.smithi071.stdout:40: finishing write tid 3 to smithi07134777-40 2024-09-11T01:29:48.417 INFO:tasks.rados.rados.0.smithi071.stdout:40: finishing write tid 4 to smithi07134777-40 2024-09-11T01:29:48.417 INFO:tasks.rados.rados.0.smithi071.stdout:40: oid 40 updating version 0 to 2 2024-09-11T01:29:48.417 INFO:tasks.rados.rados.0.smithi071.stdout:40: oid 40 updating version 2 to 3 2024-09-11T01:29:48.417 INFO:tasks.rados.rados.0.smithi071.stdout:40: oid 40 version 3 is already newer than 1 2024-09-11T01:29:48.417 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 3 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-11T01:29:48.418 INFO:tasks.rados.rados.0.smithi071.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-09-11T01:29:48.418 INFO:tasks.rados.rados.0.smithi071.stdout:41: finishing write tid 1 to smithi07134777-41 2024-09-11T01:29:48.418 INFO:tasks.rados.rados.0.smithi071.stdout:41: finishing write tid 2 to smithi07134777-41 2024-09-11T01:29:48.418 INFO:tasks.rados.rados.0.smithi071.stdout:41: finishing write tid 3 to smithi07134777-41 2024-09-11T01:29:48.418 INFO:tasks.rados.rados.0.smithi071.stdout:41: finishing write tid 4 to smithi07134777-41 2024-09-11T01:29:48.418 INFO:tasks.rados.rados.0.smithi071.stdout:41: oid 41 updating version 0 to 4 2024-09-11T01:29:48.418 INFO:tasks.rados.rados.0.smithi071.stdout:41: oid 41 updating version 4 to 5 2024-09-11T01:29:48.418 INFO:tasks.rados.rados.0.smithi071.stdout:41: oid 41 updating version 5 to 6 2024-09-11T01:29:48.418 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 6 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-11T01:29:48.418 INFO:tasks.rados.rados.0.smithi071.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-09-11T01:29:48.418 INFO:tasks.rados.rados.0.smithi071.stdout:42: finishing write tid 1 to smithi07134777-42 2024-09-11T01:29:48.418 INFO:tasks.rados.rados.0.smithi071.stdout:42: finishing write tid 2 to smithi07134777-42 2024-09-11T01:29:48.418 INFO:tasks.rados.rados.0.smithi071.stdout:42: finishing write tid 3 to smithi07134777-42 2024-09-11T01:29:48.419 INFO:tasks.rados.rados.0.smithi071.stdout:42: finishing write tid 4 to smithi07134777-42 2024-09-11T01:29:48.419 INFO:tasks.rados.rados.0.smithi071.stdout:42: oid 42 updating version 0 to 4 2024-09-11T01:29:48.419 INFO:tasks.rados.rados.0.smithi071.stdout:42: oid 42 updating version 4 to 6 2024-09-11T01:29:48.419 INFO:tasks.rados.rados.0.smithi071.stdout:42: oid 42 version 6 is already newer than 5 2024-09-11T01:29:48.419 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 6 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-11T01:29:48.419 INFO:tasks.rados.rados.0.smithi071.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-09-11T01:29:48.419 INFO:tasks.rados.rados.0.smithi071.stdout:43: finishing write tid 1 to smithi07134777-43 2024-09-11T01:29:48.419 INFO:tasks.rados.rados.0.smithi071.stdout:43: finishing write tid 2 to smithi07134777-43 2024-09-11T01:29:48.419 INFO:tasks.rados.rados.0.smithi071.stdout:43: finishing write tid 3 to smithi07134777-43 2024-09-11T01:29:48.419 INFO:tasks.rados.rados.0.smithi071.stdout:43: finishing write tid 4 to smithi07134777-43 2024-09-11T01:29:48.419 INFO:tasks.rados.rados.0.smithi071.stdout:43: oid 43 updating version 0 to 20 2024-09-11T01:29:48.419 INFO:tasks.rados.rados.0.smithi071.stdout:43: oid 43 updating version 20 to 21 2024-09-11T01:29:48.419 INFO:tasks.rados.rados.0.smithi071.stdout:43: oid 43 version 21 is already newer than 19 2024-09-11T01:29:48.419 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 21 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-11T01:29:48.420 INFO:tasks.rados.rados.0.smithi071.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-09-11T01:29:48.420 INFO:tasks.rados.rados.0.smithi071.stdout:44: finishing write tid 1 to smithi07134777-44 2024-09-11T01:29:48.420 INFO:tasks.rados.rados.0.smithi071.stdout:44: finishing write tid 2 to smithi07134777-44 2024-09-11T01:29:48.420 INFO:tasks.rados.rados.0.smithi071.stdout:44: finishing write tid 3 to smithi07134777-44 2024-09-11T01:29:48.420 INFO:tasks.rados.rados.0.smithi071.stdout:44: finishing write tid 4 to smithi07134777-44 2024-09-11T01:29:48.420 INFO:tasks.rados.rados.0.smithi071.stdout:44: oid 44 updating version 0 to 8 2024-09-11T01:29:48.420 INFO:tasks.rados.rados.0.smithi071.stdout:44: oid 44 updating version 8 to 9 2024-09-11T01:29:48.420 INFO:tasks.rados.rados.0.smithi071.stdout:44: oid 44 version 9 is already newer than 7 2024-09-11T01:29:48.420 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 9 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-11T01:29:48.420 INFO:tasks.rados.rados.0.smithi071.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-09-11T01:29:48.420 INFO:tasks.rados.rados.0.smithi071.stdout:45: finishing write tid 1 to smithi07134777-45 2024-09-11T01:29:48.420 INFO:tasks.rados.rados.0.smithi071.stdout:45: finishing write tid 2 to smithi07134777-45 2024-09-11T01:29:48.420 INFO:tasks.rados.rados.0.smithi071.stdout:45: finishing write tid 3 to smithi07134777-45 2024-09-11T01:29:48.420 INFO:tasks.rados.rados.0.smithi071.stdout:45: finishing write tid 4 to smithi07134777-45 2024-09-11T01:29:48.421 INFO:tasks.rados.rados.0.smithi071.stdout:45: oid 45 updating version 0 to 14 2024-09-11T01:29:48.421 INFO:tasks.rados.rados.0.smithi071.stdout:45: oid 45 version 14 is already newer than 13 2024-09-11T01:29:48.421 INFO:tasks.rados.rados.0.smithi071.stdout:45: oid 45 updating version 14 to 15 2024-09-11T01:29:48.421 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 15 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-11T01:29:48.421 INFO:tasks.rados.rados.0.smithi071.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-09-11T01:29:48.421 INFO:tasks.rados.rados.0.smithi071.stdout:46: finishing write tid 1 to smithi07134777-46 2024-09-11T01:29:48.421 INFO:tasks.rados.rados.0.smithi071.stdout:46: finishing write tid 2 to smithi07134777-46 2024-09-11T01:29:48.421 INFO:tasks.rados.rados.0.smithi071.stdout:46: finishing write tid 3 to smithi07134777-46 2024-09-11T01:29:48.421 INFO:tasks.rados.rados.0.smithi071.stdout:46: finishing write tid 4 to smithi07134777-46 2024-09-11T01:29:48.421 INFO:tasks.rados.rados.0.smithi071.stdout:46: oid 46 updating version 0 to 5 2024-09-11T01:29:48.421 INFO:tasks.rados.rados.0.smithi071.stdout:46: oid 46 updating version 5 to 6 2024-09-11T01:29:48.421 INFO:tasks.rados.rados.0.smithi071.stdout:46: oid 46 version 6 is already newer than 4 2024-09-11T01:29:48.421 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-11T01:29:48.421 INFO:tasks.rados.rados.0.smithi071.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:29:48.422 INFO:tasks.rados.rados.0.smithi071.stdout:47: finishing write tid 1 to smithi07134777-47 2024-09-11T01:29:48.422 INFO:tasks.rados.rados.0.smithi071.stdout:47: finishing write tid 2 to smithi07134777-47 2024-09-11T01:29:48.422 INFO:tasks.rados.rados.0.smithi071.stdout:47: finishing write tid 3 to smithi07134777-47 2024-09-11T01:29:48.422 INFO:tasks.rados.rados.0.smithi071.stdout:47: finishing write tid 4 to smithi07134777-47 2024-09-11T01:29:48.422 INFO:tasks.rados.rados.0.smithi071.stdout:47: oid 47 updating version 0 to 13 2024-09-11T01:29:48.422 INFO:tasks.rados.rados.0.smithi071.stdout:47: oid 47 updating version 13 to 14 2024-09-11T01:29:48.422 INFO:tasks.rados.rados.0.smithi071.stdout:47: oid 47 updating version 14 to 15 2024-09-11T01:29:48.422 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 15 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-11T01:29:48.422 INFO:tasks.rados.rados.0.smithi071.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-09-11T01:29:48.422 INFO:tasks.rados.rados.0.smithi071.stdout:48: finishing write tid 1 to smithi07134777-48 2024-09-11T01:29:48.422 INFO:tasks.rados.rados.0.smithi071.stdout:48: finishing write tid 2 to smithi07134777-48 2024-09-11T01:29:48.422 INFO:tasks.rados.rados.0.smithi071.stdout:48: finishing write tid 3 to smithi07134777-48 2024-09-11T01:29:48.422 INFO:tasks.rados.rados.0.smithi071.stdout:48: finishing write tid 4 to smithi07134777-48 2024-09-11T01:29:48.422 INFO:tasks.rados.rados.0.smithi071.stdout:48: oid 48 updating version 0 to 17 2024-09-11T01:29:48.423 INFO:tasks.rados.rados.0.smithi071.stdout:48: oid 48 updating version 17 to 18 2024-09-11T01:29:48.423 INFO:tasks.rados.rados.0.smithi071.stdout:48: oid 48 version 18 is already newer than 16 2024-09-11T01:29:48.423 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 18 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-11T01:29:48.423 INFO:tasks.rados.rados.0.smithi071.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-09-11T01:29:48.423 INFO:tasks.rados.rados.0.smithi071.stdout:49: finishing write tid 1 to smithi07134777-49 2024-09-11T01:29:48.423 INFO:tasks.rados.rados.0.smithi071.stdout:49: finishing write tid 2 to smithi07134777-49 2024-09-11T01:29:48.423 INFO:tasks.rados.rados.0.smithi071.stdout:49: finishing write tid 3 to smithi07134777-49 2024-09-11T01:29:48.423 INFO:tasks.rados.rados.0.smithi071.stdout:49: finishing write tid 4 to smithi07134777-49 2024-09-11T01:29:48.423 INFO:tasks.rados.rados.0.smithi071.stdout:49: oid 49 updating version 0 to 3 2024-09-11T01:29:48.423 INFO:tasks.rados.rados.0.smithi071.stdout:49: oid 49 version 3 is already newer than 1 2024-09-11T01:29:48.423 INFO:tasks.rados.rados.0.smithi071.stdout:49: oid 49 version 3 is already newer than 2 2024-09-11T01:29:48.423 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 3 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-11T01:29:48.423 INFO:tasks.rados.rados.0.smithi071.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-09-11T01:29:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:50: finishing write tid 1 to smithi07134777-50 2024-09-11T01:29:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:36: done (15 left) 2024-09-11T01:29:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:37: done (14 left) 2024-09-11T01:29:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:38: done (13 left) 2024-09-11T01:29:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:39: done (12 left) 2024-09-11T01:29:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:40: done (11 left) 2024-09-11T01:29:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:41: done (10 left) 2024-09-11T01:29:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:42: done (9 left) 2024-09-11T01:29:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:43: done (8 left) 2024-09-11T01:29:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:44: done (7 left) 2024-09-11T01:29:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:45: done (6 left) 2024-09-11T01:29:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:46: done (5 left) 2024-09-11T01:29:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:47: done (4 left) 2024-09-11T01:29:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:48: done (3 left) 2024-09-11T01:29:48.425 INFO:tasks.rados.rados.0.smithi071.stdout:49: done (2 left) 2024-09-11T01:29:48.425 INFO:tasks.rados.rados.0.smithi071.stdout:51: done (1 left) 2024-09-11T01:29:48.425 INFO:tasks.rados.rados.0.smithi071.stdout:52: read oid 27 snap -1 2024-09-11T01:29:48.425 INFO:tasks.rados.rados.0.smithi071.stdout:52: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-11T01:29:48.425 INFO:tasks.rados.rados.0.smithi071.stdout:53: snap_create 2024-09-11T01:29:48.426 INFO:tasks.rados.rados.0.smithi071.stdout:50: finishing write tid 2 to smithi07134777-50 2024-09-11T01:29:48.426 INFO:tasks.rados.rados.0.smithi071.stdout:50: finishing write tid 3 to smithi07134777-50 2024-09-11T01:29:48.426 INFO:tasks.rados.rados.0.smithi071.stdout:50: finishing write tid 4 to smithi07134777-50 2024-09-11T01:29:48.426 INFO:tasks.rados.rados.0.smithi071.stdout:50: oid 50 updating version 0 to 21 2024-09-11T01:29:48.426 INFO:tasks.rados.rados.0.smithi071.stdout:50: oid 50 version 21 is already newer than 19 2024-09-11T01:29:48.426 INFO:tasks.rados.rados.0.smithi071.stdout:50: oid 50 version 21 is already newer than 20 2024-09-11T01:29:48.426 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 21 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-11T01:29:48.426 INFO:tasks.rados.rados.0.smithi071.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-09-11T01:29:48.426 INFO:tasks.rados.rados.0.smithi071.stdout:52: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-11T01:29:48.426 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:29:48.777 INFO:tasks.rados.rados.0.smithi071.stdout:50: done (2 left) 2024-09-11T01:29:48.777 INFO:tasks.rados.rados.0.smithi071.stdout:52: done (1 left) 2024-09-11T01:29:48.777 INFO:tasks.rados.rados.0.smithi071.stdout:53: done (0 left) 2024-09-11T01:29:48.777 INFO:tasks.rados.rados.0.smithi071.stdout:54: read oid 13 snap -1 2024-09-11T01:29:48.777 INFO:tasks.rados.rados.0.smithi071.stdout:54: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-11T01:29:48.777 INFO:tasks.rados.rados.0.smithi071.stdout:55: read oid 8 snap -1 2024-09-11T01:29:48.778 INFO:tasks.rados.rados.0.smithi071.stdout:55: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-11T01:29:48.778 INFO:tasks.rados.rados.0.smithi071.stdout:56: snap_create 2024-09-11T01:29:48.786 INFO:tasks.rados.rados.0.smithi071.stdout:54: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-11T01:29:48.810 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3202 2024-09-11T01:29:48.856 INFO:tasks.rados.rados.0.smithi071.stdout:55: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-11T01:29:48.882 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 528478 2024-09-11T01:29:50.572 INFO:tasks.rados.rados.0.smithi071.stdout:54: done (2 left) 2024-09-11T01:29:50.573 INFO:tasks.rados.rados.0.smithi071.stdout:55: done (1 left) 2024-09-11T01:29:50.573 INFO:tasks.rados.rados.0.smithi071.stdout:56: done (0 left) 2024-09-11T01:29:50.573 INFO:tasks.rados.rados.0.smithi071.stdout:57: write (excl) oid 31 current snap is 2 2024-09-11T01:29:50.573 INFO:tasks.rados.rados.0.smithi071.stdout:57: seq_num 51 ranges {457157=535817,1768111=402677} 2024-09-11T01:29:50.577 INFO:tasks.rados.rados.0.smithi071.stdout:57: writing smithi07134777-31 from 457157 to 992974 tid 1 2024-09-11T01:29:50.585 INFO:tasks.rados.rados.0.smithi071.stdout:57: writing smithi07134777-31 from 1768111 to 2170788 tid 2 2024-09-11T01:29:50.585 INFO:tasks.rados.rados.0.smithi071.stdout:58: write (excl) oid 29 current snap is 2 2024-09-11T01:29:50.585 INFO:tasks.rados.rados.0.smithi071.stdout:58: seq_num 52 ranges {473964=611066,1826593=546335,2778819=95145} 2024-09-11T01:29:50.590 INFO:tasks.rados.rados.0.smithi071.stdout:58: writing smithi07134777-29 from 473964 to 1085030 tid 1 2024-09-11T01:29:50.597 INFO:tasks.rados.rados.0.smithi071.stdout:58: writing smithi07134777-29 from 1826593 to 2372928 tid 2 2024-09-11T01:29:50.601 INFO:tasks.rados.rados.0.smithi071.stdout:58: writing smithi07134777-29 from 2778819 to 2873964 tid 3 2024-09-11T01:29:50.601 INFO:tasks.rados.rados.0.smithi071.stdout:57: finishing write tid 1 to smithi07134777-31 2024-09-11T01:29:50.601 INFO:tasks.rados.rados.0.smithi071.stdout:59: read oid 25 snap -1 2024-09-11T01:29:50.601 INFO:tasks.rados.rados.0.smithi071.stdout:59: expect deleted 2024-09-11T01:29:50.601 INFO:tasks.rados.rados.0.smithi071.stdout:60: read oid 14 snap -1 2024-09-11T01:29:50.601 INFO:tasks.rados.rados.0.smithi071.stdout:60: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-11T01:29:50.601 INFO:tasks.rados.rados.0.smithi071.stdout:61: snap_remove snap 1 2024-09-11T01:29:51.714 INFO:tasks.rados.rados.0.smithi071.stdout:61: done (4 left) 2024-09-11T01:29:51.715 INFO:tasks.rados.rados.0.smithi071.stdout:62: snap_create 2024-09-11T01:29:51.715 INFO:tasks.rados.rados.0.smithi071.stdout:57: finishing write tid 2 to smithi07134777-31 2024-09-11T01:29:51.715 INFO:tasks.rados.rados.0.smithi071.stdout:57: finishing write tid 3 to smithi07134777-31 2024-09-11T01:29:51.715 INFO:tasks.rados.rados.0.smithi071.stdout:57: finishing write tid 4 to smithi07134777-31 2024-09-11T01:29:51.715 INFO:tasks.rados.rados.0.smithi071.stdout:57: oid 31 updating version 0 to 19 2024-09-11T01:29:51.715 INFO:tasks.rados.rados.0.smithi071.stdout:57: oid 31 version 19 is already newer than 18 2024-09-11T01:29:51.715 INFO:tasks.rados.rados.0.smithi071.stdout:57: oid 31 version 19 is already newer than 17 2024-09-11T01:29:51.715 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 19 (ObjNum 51 snap 2 seq_num 51) dirty exists 2024-09-11T01:29:51.716 INFO:tasks.rados.rados.0.smithi071.stdout:57: left oid 31 (ObjNum 51 snap 2 seq_num 51) 2024-09-11T01:29:51.716 INFO:tasks.rados.rados.0.smithi071.stdout:58: finishing write tid 1 to smithi07134777-29 2024-09-11T01:29:51.716 INFO:tasks.rados.rados.0.smithi071.stdout:58: finishing write tid 2 to smithi07134777-29 2024-09-11T01:29:51.716 INFO:tasks.rados.rados.0.smithi071.stdout:58: finishing write tid 3 to smithi07134777-29 2024-09-11T01:29:51.716 INFO:tasks.rados.rados.0.smithi071.stdout:58: finishing write tid 4 to smithi07134777-29 2024-09-11T01:29:51.716 INFO:tasks.rados.rados.0.smithi071.stdout:58: finishing write tid 5 to smithi07134777-29 2024-09-11T01:29:51.716 INFO:tasks.rados.rados.0.smithi071.stdout:58: oid 29 updating version 0 to 26 2024-09-11T01:29:51.716 INFO:tasks.rados.rados.0.smithi071.stdout:58: oid 29 version 26 is already newer than 23 2024-09-11T01:29:51.716 INFO:tasks.rados.rados.0.smithi071.stdout:58: oid 29 version 26 is already newer than 24 2024-09-11T01:29:51.716 INFO:tasks.rados.rados.0.smithi071.stdout:58: oid 29 version 26 is already newer than 25 2024-09-11T01:29:51.716 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 26 (ObjNum 52 snap 2 seq_num 52) dirty exists 2024-09-11T01:29:51.717 INFO:tasks.rados.rados.0.smithi071.stdout:58: left oid 29 (ObjNum 52 snap 2 seq_num 52) 2024-09-11T01:29:51.717 INFO:tasks.rados.rados.0.smithi071.stdout:60: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-11T01:29:51.717 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:29:51.767 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1828421 2024-09-11T01:29:52.720 INFO:tasks.rados.rados.0.smithi071.stdout:57: done (4 left) 2024-09-11T01:29:52.720 INFO:tasks.rados.rados.0.smithi071.stdout:58: done (3 left) 2024-09-11T01:29:52.720 INFO:tasks.rados.rados.0.smithi071.stdout:59: done (2 left) 2024-09-11T01:29:52.720 INFO:tasks.rados.rados.0.smithi071.stdout:60: done (1 left) 2024-09-11T01:29:52.720 INFO:tasks.rados.rados.0.smithi071.stdout:62: done (0 left) 2024-09-11T01:29:52.720 INFO:tasks.rados.rados.0.smithi071.stdout:63: copy_from oid 45 from oid 46 current snap is 3 2024-09-11T01:29:52.720 INFO:tasks.rados.rados.0.smithi071.stdout:64: read oid 38 snap -1 2024-09-11T01:29:52.720 INFO:tasks.rados.rados.0.smithi071.stdout:64: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-11T01:29:52.720 INFO:tasks.rados.rados.0.smithi071.stdout:65: copy_from oid 43 from oid 22 current snap is 3 2024-09-11T01:29:52.721 INFO:tasks.rados.rados.0.smithi071.stdout:66: read oid 6 snap -1 2024-09-11T01:29:52.721 INFO:tasks.rados.rados.0.smithi071.stdout:66: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-11T01:29:52.721 INFO:tasks.rados.rados.0.smithi071.stdout:67: delete oid 9 current snap is 3 2024-09-11T01:29:52.722 INFO:tasks.rados.rados.0.smithi071.stdout:67: done (4 left) 2024-09-11T01:29:52.723 INFO:tasks.rados.rados.0.smithi071.stdout:68: snap_remove snap 0 2024-09-11T01:29:53.726 INFO:tasks.rados.rados.0.smithi071.stdout:68: done (4 left) 2024-09-11T01:29:53.726 INFO:tasks.rados.rados.0.smithi071.stdout:69: rollback oid 28 current snap is 3 2024-09-11T01:29:53.726 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 28 to 2 2024-09-11T01:29:53.726 INFO:tasks.rados.rados.0.smithi071.stdout:64: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-11T01:29:53.798 INFO:tasks.rados.rados.0.smithi071.stdout:66: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-11T01:29:53.798 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:29:53.868 INFO:tasks.rados.rados.0.smithi071.stdout:63: finishing copy_from racing read to smithi07134777-45 2024-09-11T01:29:53.868 INFO:tasks.rados.rados.0.smithi071.stdout:63: finishing copy_from to smithi07134777-45 2024-09-11T01:29:53.868 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-11T01:29:53.869 INFO:tasks.rados.rados.0.smithi071.stdout:65: finishing copy_from racing read to smithi07134777-43 2024-09-11T01:29:53.869 INFO:tasks.rados.rados.0.smithi071.stdout:65: finishing copy_from to smithi07134777-43 2024-09-11T01:29:53.869 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 6 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-11T01:29:53.869 INFO:tasks.rados.rados.0.smithi071.stdout:69: finishing rollback tid 0 to smithi07134777-28 2024-09-11T01:29:53.869 INFO:tasks.rados.rados.0.smithi071.stdout:63: done (4 left) 2024-09-11T01:29:53.869 INFO:tasks.rados.rados.0.smithi071.stdout:64: done (3 left) 2024-09-11T01:29:53.869 INFO:tasks.rados.rados.0.smithi071.stdout:65: done (2 left) 2024-09-11T01:29:53.869 INFO:tasks.rados.rados.0.smithi071.stdout:66: done (1 left) 2024-09-11T01:29:53.869 INFO:tasks.rados.rados.0.smithi071.stdout:70: read oid 36 snap -1 2024-09-11T01:29:53.869 INFO:tasks.rados.rados.0.smithi071.stdout:70: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-11T01:29:53.869 INFO:tasks.rados.rados.0.smithi071.stdout:71: read oid 6 snap 2 2024-09-11T01:29:53.869 INFO:tasks.rados.rados.0.smithi071.stdout:71: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-11T01:29:53.870 INFO:tasks.rados.rados.0.smithi071.stdout:72: write oid 27 current snap is 3 2024-09-11T01:29:53.870 INFO:tasks.rados.rados.0.smithi071.stdout:69: finishing rollback tid 1 to smithi07134777-28 2024-09-11T01:29:53.870 INFO:tasks.rados.rados.0.smithi071.stdout:69: finishing rollback tid 2 to smithi07134777-28 2024-09-11T01:29:53.870 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 21 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-11T01:29:53.870 INFO:tasks.rados.rados.0.smithi071.stdout:72: seq_num 54 ranges {507578=477917,1543557=550004,2833665=73913} 2024-09-11T01:29:53.873 INFO:tasks.rados.rados.0.smithi071.stdout:72: writing smithi07134777-27 from 507578 to 985495 tid 1 2024-09-11T01:29:53.879 INFO:tasks.rados.rados.0.smithi071.stdout:72: writing smithi07134777-27 from 1543557 to 2093561 tid 2 2024-09-11T01:29:53.883 INFO:tasks.rados.rados.0.smithi071.stdout:72: writing smithi07134777-27 from 2833665 to 2907578 tid 3 2024-09-11T01:29:53.884 INFO:tasks.rados.rados.0.smithi071.stdout:69: done (3 left) 2024-09-11T01:29:53.884 INFO:tasks.rados.rados.0.smithi071.stdout:73: read oid 31 snap 2 2024-09-11T01:29:53.884 INFO:tasks.rados.rados.0.smithi071.stdout:73: expect (ObjNum 51 snap 2 seq_num 51) 2024-09-11T01:29:53.884 INFO:tasks.rados.rados.0.smithi071.stdout:74: write (excl) oid 35 current snap is 3 2024-09-11T01:29:53.884 INFO:tasks.rados.rados.0.smithi071.stdout:74: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2024-09-11T01:29:53.889 INFO:tasks.rados.rados.0.smithi071.stdout:74: writing smithi07134777-35 from 524385 to 1077551 tid 1 2024-09-11T01:29:53.894 INFO:tasks.rados.rados.0.smithi071.stdout:74: writing smithi07134777-35 from 1602039 to 2012054 tid 2 2024-09-11T01:29:53.899 INFO:tasks.rados.rados.0.smithi071.stdout:74: writing smithi07134777-35 from 2577441 to 2924385 tid 3 2024-09-11T01:29:53.899 INFO:tasks.rados.rados.0.smithi071.stdout:75: snap_create 2024-09-11T01:29:53.899 INFO:tasks.rados.rados.0.smithi071.stdout:71: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-11T01:29:53.920 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 214812 2024-09-11T01:29:53.964 INFO:tasks.rados.rados.0.smithi071.stdout:70: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-11T01:29:54.038 INFO:tasks.rados.rados.0.smithi071.stdout:72: finishing write tid 1 to smithi07134777-27 2024-09-11T01:29:54.038 INFO:tasks.rados.rados.0.smithi071.stdout:72: finishing write tid 2 to smithi07134777-27 2024-09-11T01:29:54.038 INFO:tasks.rados.rados.0.smithi071.stdout:72: finishing write tid 3 to smithi07134777-27 2024-09-11T01:29:54.038 INFO:tasks.rados.rados.0.smithi071.stdout:72: finishing write tid 4 to smithi07134777-27 2024-09-11T01:29:54.038 INFO:tasks.rados.rados.0.smithi071.stdout:73: expect (ObjNum 51 snap 2 seq_num 51) 2024-09-11T01:29:54.139 INFO:tasks.rados.rados.0.smithi071.stdout:72: finishing write tid 5 to smithi07134777-27 2024-09-11T01:29:54.139 INFO:tasks.rados.rados.0.smithi071.stdout:72: oid 27 updating version 0 to 27 2024-09-11T01:29:54.139 INFO:tasks.rados.rados.0.smithi071.stdout:72: oid 27 updating version 27 to 29 2024-09-11T01:29:54.139 INFO:tasks.rados.rados.0.smithi071.stdout:72: oid 27 version 29 is already newer than 28 2024-09-11T01:29:54.139 INFO:tasks.rados.rados.0.smithi071.stdout:72: oid 27 updating version 29 to 30 2024-09-11T01:29:54.139 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 30 (ObjNum 54 snap 3 seq_num 54) dirty exists 2024-09-11T01:29:54.139 INFO:tasks.rados.rados.0.smithi071.stdout:72: left oid 27 (ObjNum 54 snap 3 seq_num 54) 2024-09-11T01:29:54.139 INFO:tasks.rados.rados.0.smithi071.stdout:74: finishing write tid 1 to smithi07134777-35 2024-09-11T01:29:54.139 INFO:tasks.rados.rados.0.smithi071.stdout:74: finishing write tid 2 to smithi07134777-35 2024-09-11T01:29:54.139 INFO:tasks.rados.rados.0.smithi071.stdout:74: finishing write tid 3 to smithi07134777-35 2024-09-11T01:29:54.139 INFO:tasks.rados.rados.0.smithi071.stdout:74: finishing write tid 4 to smithi07134777-35 2024-09-11T01:29:54.140 INFO:tasks.rados.rados.0.smithi071.stdout:74: finishing write tid 5 to smithi07134777-35 2024-09-11T01:29:54.140 INFO:tasks.rados.rados.0.smithi071.stdout:74: oid 35 updating version 0 to 16 2024-09-11T01:29:54.140 INFO:tasks.rados.rados.0.smithi071.stdout:74: oid 35 version 16 is already newer than 14 2024-09-11T01:29:54.140 INFO:tasks.rados.rados.0.smithi071.stdout:74: oid 35 updating version 16 to 17 2024-09-11T01:29:54.140 INFO:tasks.rados.rados.0.smithi071.stdout:74: oid 35 version 17 is already newer than 15 2024-09-11T01:29:54.140 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 17 (ObjNum 55 snap 3 seq_num 55) dirty exists 2024-09-11T01:29:54.140 INFO:tasks.rados.rados.0.smithi071.stdout:74: left oid 35 (ObjNum 55 snap 3 seq_num 55) 2024-09-11T01:29:54.734 INFO:tasks.rados.rados.0.smithi071.stdout:70: done (5 left) 2024-09-11T01:29:54.734 INFO:tasks.rados.rados.0.smithi071.stdout:71: done (4 left) 2024-09-11T01:29:54.734 INFO:tasks.rados.rados.0.smithi071.stdout:72: done (3 left) 2024-09-11T01:29:54.735 INFO:tasks.rados.rados.0.smithi071.stdout:73: done (2 left) 2024-09-11T01:29:54.735 INFO:tasks.rados.rados.0.smithi071.stdout:74: done (1 left) 2024-09-11T01:29:54.735 INFO:tasks.rados.rados.0.smithi071.stdout:75: done (0 left) 2024-09-11T01:29:54.735 INFO:tasks.rados.rados.0.smithi071.stdout:76: rollback oid 33 current snap is 4 2024-09-11T01:29:54.735 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 33 to 2 2024-09-11T01:29:54.735 INFO:tasks.rados.rados.0.smithi071.stdout:77: read oid 19 snap 2 2024-09-11T01:29:54.735 INFO:tasks.rados.rados.0.smithi071.stdout:77: expect (ObjNum 18 snap 0 seq_num 18) 2024-09-11T01:29:54.735 INFO:tasks.rados.rados.0.smithi071.stdout:78: snap_remove snap 3 2024-09-11T01:29:55.743 INFO:tasks.rados.rados.0.smithi071.stdout:78: done (2 left) 2024-09-11T01:29:55.743 INFO:tasks.rados.rados.0.smithi071.stdout:79: write oid 16 current snap is 4 2024-09-11T01:29:55.743 INFO:tasks.rados.rados.0.smithi071.stdout:79: seq_num 56 ranges {541192=628415,1660521=553673,2888511=52681} 2024-09-11T01:29:55.749 INFO:tasks.rados.rados.0.smithi071.stdout:79: writing smithi07134777-16 from 541192 to 1169607 tid 1 2024-09-11T01:29:55.755 INFO:tasks.rados.rados.0.smithi071.stdout:79: writing smithi07134777-16 from 1660521 to 2214194 tid 2 2024-09-11T01:29:55.759 INFO:tasks.rados.rados.0.smithi071.stdout:79: writing smithi07134777-16 from 2888511 to 2941192 tid 3 2024-09-11T01:29:55.759 INFO:tasks.rados.rados.0.smithi071.stdout:80: write oid 29 current snap is 4 2024-09-11T01:29:55.759 INFO:tasks.rados.rados.0.smithi071.stdout:80: seq_num 57 ranges {557999=703664,1719003=413684,2632287=325712} 2024-09-11T01:29:55.766 INFO:tasks.rados.rados.0.smithi071.stdout:80: writing smithi07134777-29 from 557999 to 1261663 tid 1 2024-09-11T01:29:55.771 INFO:tasks.rados.rados.0.smithi071.stdout:80: writing smithi07134777-29 from 1719003 to 2132687 tid 2 2024-09-11T01:29:55.776 INFO:tasks.rados.rados.0.smithi071.stdout:80: writing smithi07134777-29 from 2632287 to 2957999 tid 3 2024-09-11T01:29:55.776 INFO:tasks.rados.rados.0.smithi071.stdout:81: read oid 30 snap 2 2024-09-11T01:29:55.776 INFO:tasks.rados.rados.0.smithi071.stdout:81: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-11T01:29:55.776 INFO:tasks.rados.rados.0.smithi071.stdout:82: snap_create 2024-09-11T01:29:55.776 INFO:tasks.rados.rados.0.smithi071.stdout:76: finishing rollback tid 0 to smithi07134777-33 2024-09-11T01:29:55.777 INFO:tasks.rados.rados.0.smithi071.stdout:77: expect (ObjNum 18 snap 0 seq_num 18) 2024-09-11T01:29:55.806 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 905749 2024-09-11T01:29:55.844 INFO:tasks.rados.rados.0.smithi071.stdout:76: finishing rollback tid 1 to smithi07134777-33 2024-09-11T01:29:55.845 INFO:tasks.rados.rados.0.smithi071.stdout:76: finishing rollback tid 2 to smithi07134777-33 2024-09-11T01:29:55.845 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 34 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-11T01:29:55.845 INFO:tasks.rados.rados.0.smithi071.stdout:79: finishing write tid 1 to smithi07134777-16 2024-09-11T01:29:55.845 INFO:tasks.rados.rados.0.smithi071.stdout:79: finishing write tid 2 to smithi07134777-16 2024-09-11T01:29:55.845 INFO:tasks.rados.rados.0.smithi071.stdout:79: finishing write tid 3 to smithi07134777-16 2024-09-11T01:29:55.845 INFO:tasks.rados.rados.0.smithi071.stdout:79: finishing write tid 4 to smithi07134777-16 2024-09-11T01:29:55.845 INFO:tasks.rados.rados.0.smithi071.stdout:79: finishing write tid 5 to smithi07134777-16 2024-09-11T01:29:55.845 INFO:tasks.rados.rados.0.smithi071.stdout:79: oid 16 updating version 0 to 28 2024-09-11T01:29:55.846 INFO:tasks.rados.rados.0.smithi071.stdout:79: oid 16 version 28 is already newer than 27 2024-09-11T01:29:55.846 INFO:tasks.rados.rados.0.smithi071.stdout:79: oid 16 version 28 is already newer than 25 2024-09-11T01:29:55.846 INFO:tasks.rados.rados.0.smithi071.stdout:79: oid 16 version 28 is already newer than 26 2024-09-11T01:29:55.846 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 28 (ObjNum 56 snap 4 seq_num 56) dirty exists 2024-09-11T01:29:55.846 INFO:tasks.rados.rados.0.smithi071.stdout:79: left oid 16 (ObjNum 56 snap 4 seq_num 56) 2024-09-11T01:29:55.846 INFO:tasks.rados.rados.0.smithi071.stdout:80: finishing write tid 1 to smithi07134777-29 2024-09-11T01:29:55.846 INFO:tasks.rados.rados.0.smithi071.stdout:80: finishing write tid 2 to smithi07134777-29 2024-09-11T01:29:55.846 INFO:tasks.rados.rados.0.smithi071.stdout:80: finishing write tid 3 to smithi07134777-29 2024-09-11T01:29:55.846 INFO:tasks.rados.rados.0.smithi071.stdout:80: finishing write tid 4 to smithi07134777-29 2024-09-11T01:29:55.846 INFO:tasks.rados.rados.0.smithi071.stdout:80: finishing write tid 5 to smithi07134777-29 2024-09-11T01:29:55.846 INFO:tasks.rados.rados.0.smithi071.stdout:80: oid 29 updating version 0 to 33 2024-09-11T01:29:55.846 INFO:tasks.rados.rados.0.smithi071.stdout:80: oid 29 updating version 33 to 35 2024-09-11T01:29:55.846 INFO:tasks.rados.rados.0.smithi071.stdout:80: oid 29 version 35 is already newer than 34 2024-09-11T01:29:55.846 INFO:tasks.rados.rados.0.smithi071.stdout:80: oid 29 version 35 is already newer than 32 2024-09-11T01:29:55.846 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 35 (ObjNum 57 snap 4 seq_num 57) dirty exists 2024-09-11T01:29:55.847 INFO:tasks.rados.rados.0.smithi071.stdout:80: left oid 29 (ObjNum 57 snap 4 seq_num 57) 2024-09-11T01:29:55.847 INFO:tasks.rados.rados.0.smithi071.stdout:81: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-11T01:29:56.742 INFO:tasks.rados.rados.0.smithi071.stdout:76: done (5 left) 2024-09-11T01:29:56.742 INFO:tasks.rados.rados.0.smithi071.stdout:77: done (4 left) 2024-09-11T01:29:56.742 INFO:tasks.rados.rados.0.smithi071.stdout:79: done (3 left) 2024-09-11T01:29:56.742 INFO:tasks.rados.rados.0.smithi071.stdout:80: done (2 left) 2024-09-11T01:29:56.742 INFO:tasks.rados.rados.0.smithi071.stdout:81: done (1 left) 2024-09-11T01:29:56.742 INFO:tasks.rados.rados.0.smithi071.stdout:82: done (0 left) 2024-09-11T01:29:56.742 INFO:tasks.rados.rados.0.smithi071.stdout:83: rollback oid 43 current snap is 5 2024-09-11T01:29:56.742 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 43 to 2 2024-09-11T01:29:56.742 INFO:tasks.rados.rados.0.smithi071.stdout:84: delete oid 13 current snap is 5 2024-09-11T01:29:56.745 INFO:tasks.rados.rados.0.smithi071.stdout:84: done (1 left) 2024-09-11T01:29:56.745 INFO:tasks.rados.rados.0.smithi071.stdout:85: read oid 38 snap -1 2024-09-11T01:29:56.745 INFO:tasks.rados.rados.0.smithi071.stdout:85: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-11T01:29:56.745 INFO:tasks.rados.rados.0.smithi071.stdout:86: write (excl) oid 31 current snap is 5 2024-09-11T01:29:56.746 INFO:tasks.rados.rados.0.smithi071.stdout:86: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2024-09-11T01:29:56.750 INFO:tasks.rados.rados.0.smithi071.stdout:86: writing smithi07134777-31 from 591613 to 1045775 tid 1 2024-09-11T01:29:56.756 INFO:tasks.rados.rados.0.smithi071.stdout:86: writing smithi07134777-31 from 1719614 to 2136967 tid 2 2024-09-11T01:29:56.762 INFO:tasks.rados.rados.0.smithi071.stdout:86: writing smithi07134777-31 from 2570780 to 2991613 tid 3 2024-09-11T01:29:56.762 INFO:tasks.rados.rados.0.smithi071.stdout:87: read oid 45 snap 4 2024-09-11T01:29:56.762 INFO:tasks.rados.rados.0.smithi071.stdout:87: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:29:56.762 INFO:tasks.rados.rados.0.smithi071.stdout:88: write (excl) oid 1 current snap is 5 2024-09-11T01:29:56.762 INFO:tasks.rados.rados.0.smithi071.stdout:88: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2024-09-11T01:29:56.768 INFO:tasks.rados.rados.0.smithi071.stdout:88: writing smithi07134777-1 from 608420 to 1137831 tid 1 2024-09-11T01:29:56.775 INFO:tasks.rados.rados.0.smithi071.stdout:88: writing smithi07134777-1 from 1778096 to 2339107 tid 2 2024-09-11T01:29:56.779 INFO:tasks.rados.rados.0.smithi071.stdout:88: writing smithi07134777-1 from 2881850 to 3008420 tid 3 2024-09-11T01:29:56.779 INFO:tasks.rados.rados.0.smithi071.stdout:89: write (excl) oid 37 current snap is 5 2024-09-11T01:29:56.779 INFO:tasks.rados.rados.0.smithi071.stdout:89: seq_num 61 ranges {625227=721013,1952931=704669} 2024-09-11T01:29:56.786 INFO:tasks.rados.rados.0.smithi071.stdout:89: writing smithi07134777-37 from 625227 to 1346240 tid 1 2024-09-11T01:29:56.793 INFO:tasks.rados.rados.0.smithi071.stdout:89: writing smithi07134777-37 from 1952931 to 2657600 tid 2 2024-09-11T01:29:56.793 INFO:tasks.rados.rados.0.smithi071.stdout:90: read oid 9 snap -1 2024-09-11T01:29:56.793 INFO:tasks.rados.rados.0.smithi071.stdout:90: expect deleted 2024-09-11T01:29:56.793 INFO:tasks.rados.rados.0.smithi071.stdout:91: copy_from oid 11 from oid 46 current snap is 5 2024-09-11T01:29:56.793 INFO:tasks.rados.rados.0.smithi071.stdout:83: finishing rollback tid 0 to smithi07134777-43 2024-09-11T01:29:56.793 INFO:tasks.rados.rados.0.smithi071.stdout:83: finishing rollback tid 1 to smithi07134777-43 2024-09-11T01:29:56.793 INFO:tasks.rados.rados.0.smithi071.stdout:83: finishing rollback tid 2 to smithi07134777-43 2024-09-11T01:29:56.794 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 40 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-11T01:29:56.794 INFO:tasks.rados.rados.0.smithi071.stdout:85: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-11T01:29:56.794 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:29:56.846 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 252128 2024-09-11T01:29:56.873 INFO:tasks.rados.rados.0.smithi071.stdout:86: finishing write tid 1 to smithi07134777-31 2024-09-11T01:29:56.873 INFO:tasks.rados.rados.0.smithi071.stdout:86: finishing write tid 2 to smithi07134777-31 2024-09-11T01:29:56.874 INFO:tasks.rados.rados.0.smithi071.stdout:86: finishing write tid 3 to smithi07134777-31 2024-09-11T01:29:56.874 INFO:tasks.rados.rados.0.smithi071.stdout:87: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:29:56.906 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2333689 2024-09-11T01:29:56.938 INFO:tasks.rados.rados.0.smithi071.stdout:86: finishing write tid 4 to smithi07134777-31 2024-09-11T01:29:56.939 INFO:tasks.rados.rados.0.smithi071.stdout:86: finishing write tid 5 to smithi07134777-31 2024-09-11T01:29:56.939 INFO:tasks.rados.rados.0.smithi071.stdout:86: oid 31 updating version 0 to 27 2024-09-11T01:29:56.939 INFO:tasks.rados.rados.0.smithi071.stdout:86: oid 31 updating version 27 to 29 2024-09-11T01:29:56.939 INFO:tasks.rados.rados.0.smithi071.stdout:86: oid 31 version 29 is already newer than 28 2024-09-11T01:29:56.939 INFO:tasks.rados.rados.0.smithi071.stdout:86: oid 31 updating version 29 to 30 2024-09-11T01:29:56.939 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 30 (ObjNum 59 snap 5 seq_num 59) dirty exists 2024-09-11T01:29:56.939 INFO:tasks.rados.rados.0.smithi071.stdout:86: left oid 31 (ObjNum 59 snap 5 seq_num 59) 2024-09-11T01:29:56.939 INFO:tasks.rados.rados.0.smithi071.stdout:88: finishing write tid 1 to smithi07134777-1 2024-09-11T01:29:56.939 INFO:tasks.rados.rados.0.smithi071.stdout:88: finishing write tid 2 to smithi07134777-1 2024-09-11T01:29:56.939 INFO:tasks.rados.rados.0.smithi071.stdout:83: done (7 left) 2024-09-11T01:29:56.939 INFO:tasks.rados.rados.0.smithi071.stdout:85: done (6 left) 2024-09-11T01:29:56.939 INFO:tasks.rados.rados.0.smithi071.stdout:86: done (5 left) 2024-09-11T01:29:56.939 INFO:tasks.rados.rados.0.smithi071.stdout:87: done (4 left) 2024-09-11T01:29:56.939 INFO:tasks.rados.rados.0.smithi071.stdout:92: snap_remove snap 4 2024-09-11T01:29:57.359 INFO:tasks.rados.rados.0.smithi071.stdout:92: done (4 left) 2024-09-11T01:29:57.360 INFO:tasks.rados.rados.0.smithi071.stdout:93: read oid 19 snap -1 2024-09-11T01:29:57.360 INFO:tasks.rados.rados.0.smithi071.stdout:93: expect (ObjNum 18 snap 0 seq_num 18) 2024-09-11T01:29:57.360 INFO:tasks.rados.rados.0.smithi071.stdout:94: read oid 41 snap -1 2024-09-11T01:29:57.360 INFO:tasks.rados.rados.0.smithi071.stdout:94: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-11T01:29:57.360 INFO:tasks.rados.rados.0.smithi071.stdout:95: read oid 30 snap -1 2024-09-11T01:29:57.360 INFO:tasks.rados.rados.0.smithi071.stdout:95: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-11T01:29:57.360 INFO:tasks.rados.rados.0.smithi071.stdout:96: snap_create 2024-09-11T01:29:57.360 INFO:tasks.rados.rados.0.smithi071.stdout:88: finishing write tid 3 to smithi07134777-1 2024-09-11T01:29:57.360 INFO:tasks.rados.rados.0.smithi071.stdout:88: finishing write tid 4 to smithi07134777-1 2024-09-11T01:29:57.360 INFO:tasks.rados.rados.0.smithi071.stdout:88: finishing write tid 5 to smithi07134777-1 2024-09-11T01:29:57.360 INFO:tasks.rados.rados.0.smithi071.stdout:88: oid 1 updating version 0 to 33 2024-09-11T01:29:57.360 INFO:tasks.rados.rados.0.smithi071.stdout:88: oid 1 updating version 33 to 34 2024-09-11T01:29:57.361 INFO:tasks.rados.rados.0.smithi071.stdout:88: oid 1 updating version 34 to 35 2024-09-11T01:29:57.361 INFO:tasks.rados.rados.0.smithi071.stdout:88: oid 1 version 35 is already newer than 32 2024-09-11T01:29:57.361 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 35 (ObjNum 60 snap 5 seq_num 60) dirty exists 2024-09-11T01:29:57.361 INFO:tasks.rados.rados.0.smithi071.stdout:88: left oid 1 (ObjNum 60 snap 5 seq_num 60) 2024-09-11T01:29:57.361 INFO:tasks.rados.rados.0.smithi071.stdout:89: finishing write tid 1 to smithi07134777-37 2024-09-11T01:29:57.361 INFO:tasks.rados.rados.0.smithi071.stdout:89: finishing write tid 2 to smithi07134777-37 2024-09-11T01:29:57.361 INFO:tasks.rados.rados.0.smithi071.stdout:89: finishing write tid 3 to smithi07134777-37 2024-09-11T01:29:57.361 INFO:tasks.rados.rados.0.smithi071.stdout:89: finishing write tid 4 to smithi07134777-37 2024-09-11T01:29:57.361 INFO:tasks.rados.rados.0.smithi071.stdout:89: oid 37 updating version 0 to 20 2024-09-11T01:29:57.361 INFO:tasks.rados.rados.0.smithi071.stdout:89: oid 37 updating version 20 to 21 2024-09-11T01:29:57.361 INFO:tasks.rados.rados.0.smithi071.stdout:89: oid 37 version 21 is already newer than 19 2024-09-11T01:29:57.361 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 21 (ObjNum 61 snap 5 seq_num 61) dirty exists 2024-09-11T01:29:57.361 INFO:tasks.rados.rados.0.smithi071.stdout:89: left oid 37 (ObjNum 61 snap 5 seq_num 61) 2024-09-11T01:29:57.361 INFO:tasks.rados.rados.0.smithi071.stdout:91: finishing copy_from racing read to smithi07134777-11 2024-09-11T01:29:57.362 INFO:tasks.rados.rados.0.smithi071.stdout:91: finishing copy_from to smithi07134777-11 2024-09-11T01:29:57.362 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-11T01:29:57.369 INFO:tasks.rados.rados.0.smithi071.stdout:95: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-11T01:29:57.393 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:29:57.445 INFO:tasks.rados.rados.0.smithi071.stdout:93: expect (ObjNum 18 snap 0 seq_num 18) 2024-09-11T01:29:57.453 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1413010 2024-09-11T01:29:57.486 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:29:57.702 INFO:tasks.rados.rados.0.smithi071.stdout:94: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-11T01:29:57.705 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2124165 2024-09-11T01:29:57.717 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:29:57.751 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 713565 2024-09-11T01:29:58.361 INFO:tasks.rados.rados.0.smithi071.stdout:88: done (7 left) 2024-09-11T01:29:58.361 INFO:tasks.rados.rados.0.smithi071.stdout:89: done (6 left) 2024-09-11T01:29:58.361 INFO:tasks.rados.rados.0.smithi071.stdout:90: done (5 left) 2024-09-11T01:29:58.361 INFO:tasks.rados.rados.0.smithi071.stdout:91: done (4 left) 2024-09-11T01:29:58.361 INFO:tasks.rados.rados.0.smithi071.stdout:93: done (3 left) 2024-09-11T01:29:58.361 INFO:tasks.rados.rados.0.smithi071.stdout:94: done (2 left) 2024-09-11T01:29:58.361 INFO:tasks.rados.rados.0.smithi071.stdout:95: done (1 left) 2024-09-11T01:29:58.361 INFO:tasks.rados.rados.0.smithi071.stdout:96: done (0 left) 2024-09-11T01:29:58.362 INFO:tasks.rados.rados.0.smithi071.stdout:97: read oid 18 snap -1 2024-09-11T01:29:58.362 INFO:tasks.rados.rados.0.smithi071.stdout:97: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-11T01:29:58.362 INFO:tasks.rados.rados.0.smithi071.stdout:98: snap_create 2024-09-11T01:29:58.375 INFO:tasks.rados.rados.0.smithi071.stdout:97: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-11T01:29:58.407 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:29:59.365 INFO:tasks.rados.rados.0.smithi071.stdout:97: done (1 left) 2024-09-11T01:29:59.365 INFO:tasks.rados.rados.0.smithi071.stdout:98: done (0 left) 2024-09-11T01:29:59.365 INFO:tasks.rados.rados.0.smithi071.stdout:99: read oid 39 snap -1 2024-09-11T01:29:59.365 INFO:tasks.rados.rados.0.smithi071.stdout:99: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-11T01:29:59.365 INFO:tasks.rados.rados.0.smithi071.stdout:100: read oid 14 snap -1 2024-09-11T01:29:59.365 INFO:tasks.rados.rados.0.smithi071.stdout:100: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-11T01:29:59.365 INFO:tasks.rados.rados.0.smithi071.stdout:101: snap_create 2024-09-11T01:29:59.371 INFO:tasks.rados.rados.0.smithi071.stdout:100: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-11T01:29:59.372 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:29:59.396 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:29:59.445 INFO:tasks.rados.rados.0.smithi071.stdout:99: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-11T01:29:59.446 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:29:59.478 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 840488 2024-09-11T01:30:00.369 INFO:tasks.rados.rados.0.smithi071.stdout:99: done (2 left) 2024-09-11T01:30:00.370 INFO:tasks.rados.rados.0.smithi071.stdout:100: done (1 left) 2024-09-11T01:30:00.370 INFO:tasks.rados.rados.0.smithi071.stdout:101: done (0 left) 2024-09-11T01:30:00.370 INFO:tasks.rados.rados.0.smithi071.stdout:102: snap_remove snap 5 2024-09-11T01:30:01.374 INFO:tasks.rados.rados.0.smithi071.stdout:102: done (0 left) 2024-09-11T01:30:01.375 INFO:tasks.rados.rados.0.smithi071.stdout:103: delete oid 34 current snap is 8 2024-09-11T01:30:01.381 INFO:tasks.rados.rados.0.smithi071.stdout:103: done (0 left) 2024-09-11T01:30:01.381 INFO:tasks.rados.rados.0.smithi071.stdout:104: write (excl) oid 6 current snap is 8 2024-09-11T01:30:01.382 INFO:tasks.rados.rados.0.smithi071.stdout:104: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2024-09-11T01:30:01.387 INFO:tasks.rados.rados.0.smithi071.stdout:104: writing smithi07134777-6 from 658841 to 1130352 tid 1 2024-09-11T01:30:01.393 INFO:tasks.rados.rados.0.smithi071.stdout:104: writing smithi07134777-6 from 1553542 to 2261880 tid 2 2024-09-11T01:30:01.398 INFO:tasks.rados.rados.0.smithi071.stdout:104: writing smithi07134777-6 from 2964119 to 3058841 tid 3 2024-09-11T01:30:01.398 INFO:tasks.rados.rados.0.smithi071.stdout:105: copy_from oid 20 from oid 2 current snap is 8 2024-09-11T01:30:01.399 INFO:tasks.rados.rados.0.smithi071.stdout:104: finishing write tid 1 to smithi07134777-6 2024-09-11T01:30:01.399 INFO:tasks.rados.rados.0.smithi071.stdout:106: read oid 37 snap -1 2024-09-11T01:30:01.399 INFO:tasks.rados.rados.0.smithi071.stdout:106: expect (ObjNum 61 snap 5 seq_num 61) 2024-09-11T01:30:01.399 INFO:tasks.rados.rados.0.smithi071.stdout:107: snap_remove snap 6 2024-09-11T01:30:02.360 INFO:tasks.rados.rados.0.smithi071.stdout:107: done (3 left) 2024-09-11T01:30:02.361 INFO:tasks.rados.rados.0.smithi071.stdout:108: write (excl) oid 5 current snap is 8 2024-09-11T01:30:02.361 INFO:tasks.rados.rados.0.smithi071.stdout:108: seq_num 64 ranges {675648=546760,2012024=568349} 2024-09-11T01:30:02.367 INFO:tasks.rados.rados.0.smithi071.stdout:108: writing smithi07134777-5 from 675648 to 1222408 tid 1 2024-09-11T01:30:02.375 INFO:tasks.rados.rados.0.smithi071.stdout:108: writing smithi07134777-5 from 2012024 to 2580373 tid 2 2024-09-11T01:30:02.375 INFO:tasks.rados.rados.0.smithi071.stdout:109: snap_create 2024-09-11T01:30:02.375 INFO:tasks.rados.rados.0.smithi071.stdout:104: finishing write tid 2 to smithi07134777-6 2024-09-11T01:30:02.375 INFO:tasks.rados.rados.0.smithi071.stdout:104: finishing write tid 3 to smithi07134777-6 2024-09-11T01:30:02.375 INFO:tasks.rados.rados.0.smithi071.stdout:104: finishing write tid 4 to smithi07134777-6 2024-09-11T01:30:02.376 INFO:tasks.rados.rados.0.smithi071.stdout:104: finishing write tid 5 to smithi07134777-6 2024-09-11T01:30:02.376 INFO:tasks.rados.rados.0.smithi071.stdout:104: oid 6 updating version 0 to 48 2024-09-11T01:30:02.376 INFO:tasks.rados.rados.0.smithi071.stdout:104: oid 6 version 48 is already newer than 47 2024-09-11T01:30:02.376 INFO:tasks.rados.rados.0.smithi071.stdout:104: oid 6 updating version 48 to 49 2024-09-11T01:30:02.376 INFO:tasks.rados.rados.0.smithi071.stdout:104: oid 6 version 49 is already newer than 46 2024-09-11T01:30:02.376 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 49 (ObjNum 63 snap 8 seq_num 63) dirty exists 2024-09-11T01:30:02.376 INFO:tasks.rados.rados.0.smithi071.stdout:104: left oid 6 (ObjNum 63 snap 8 seq_num 63) 2024-09-11T01:30:02.376 INFO:tasks.rados.rados.0.smithi071.stdout:105: finishing copy_from racing read to smithi07134777-20 2024-09-11T01:30:02.376 INFO:tasks.rados.rados.0.smithi071.stdout:106: expect (ObjNum 61 snap 5 seq_num 61) 2024-09-11T01:30:02.376 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:30:02.478 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1118822 2024-09-11T01:30:02.509 INFO:tasks.rados.rados.0.smithi071.stdout:105: finishing copy_from to smithi07134777-20 2024-09-11T01:30:02.509 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-11T01:30:02.510 INFO:tasks.rados.rados.0.smithi071.stdout:108: finishing write tid 1 to smithi07134777-5 2024-09-11T01:30:02.510 INFO:tasks.rados.rados.0.smithi071.stdout:108: finishing write tid 2 to smithi07134777-5 2024-09-11T01:30:02.510 INFO:tasks.rados.rados.0.smithi071.stdout:108: finishing write tid 3 to smithi07134777-5 2024-09-11T01:30:02.510 INFO:tasks.rados.rados.0.smithi071.stdout:108: finishing write tid 4 to smithi07134777-5 2024-09-11T01:30:02.510 INFO:tasks.rados.rados.0.smithi071.stdout:108: oid 5 updating version 0 to 6 2024-09-11T01:30:02.510 INFO:tasks.rados.rados.0.smithi071.stdout:108: oid 5 updating version 6 to 7 2024-09-11T01:30:02.510 INFO:tasks.rados.rados.0.smithi071.stdout:108: oid 5 version 7 is already newer than 5 2024-09-11T01:30:02.510 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 7 (ObjNum 64 snap 8 seq_num 64) dirty exists 2024-09-11T01:30:02.511 INFO:tasks.rados.rados.0.smithi071.stdout:108: left oid 5 (ObjNum 64 snap 8 seq_num 64) 2024-09-11T01:30:03.380 INFO:tasks.rados.rados.0.smithi071.stdout:104: done (4 left) 2024-09-11T01:30:03.380 INFO:tasks.rados.rados.0.smithi071.stdout:105: done (3 left) 2024-09-11T01:30:03.380 INFO:tasks.rados.rados.0.smithi071.stdout:106: done (2 left) 2024-09-11T01:30:03.380 INFO:tasks.rados.rados.0.smithi071.stdout:108: done (1 left) 2024-09-11T01:30:03.380 INFO:tasks.rados.rados.0.smithi071.stdout:109: done (0 left) 2024-09-11T01:30:03.380 INFO:tasks.rados.rados.0.smithi071.stdout:110: snap_remove snap 8 2024-09-11T01:30:04.383 INFO:tasks.rados.rados.0.smithi071.stdout:110: done (0 left) 2024-09-11T01:30:04.383 INFO:tasks.rados.rados.0.smithi071.stdout:111: copy_from oid 44 from oid 12 current snap is 9 2024-09-11T01:30:04.383 INFO:tasks.rados.rados.0.smithi071.stdout:112: copy_from oid 48 from oid 8 current snap is 9 2024-09-11T01:30:04.383 INFO:tasks.rados.rados.0.smithi071.stdout:113: copy_from oid 18 from oid 9 current snap is 9 2024-09-11T01:30:04.383 INFO:tasks.rados.rados.0.smithi071.stdout:114: snap_create 2024-09-11T01:30:04.393 INFO:tasks.rados.rados.0.smithi071.stdout:113: finishing copy_from to smithi07134777-18 2024-09-11T01:30:04.393 INFO:tasks.rados.rados.0.smithi071.stdout:113: got expected ENOENT (src dne) 2024-09-11T01:30:04.393 INFO:tasks.rados.rados.0.smithi071.stdout:113: finishing copy_from racing read to smithi07134777-18 2024-09-11T01:30:04.397 INFO:tasks.rados.rados.0.smithi071.stdout:112: finishing copy_from racing read to smithi07134777-48 2024-09-11T01:30:04.399 INFO:tasks.rados.rados.0.smithi071.stdout:112: finishing copy_from to smithi07134777-48 2024-09-11T01:30:04.399 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-11T01:30:04.403 INFO:tasks.rados.rados.0.smithi071.stdout:111: finishing copy_from racing read to smithi07134777-44 2024-09-11T01:30:04.405 INFO:tasks.rados.rados.0.smithi071.stdout:111: finishing copy_from to smithi07134777-44 2024-09-11T01:30:04.405 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 6 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-11T01:30:05.384 INFO:tasks.rados.rados.0.smithi071.stdout:111: done (3 left) 2024-09-11T01:30:05.384 INFO:tasks.rados.rados.0.smithi071.stdout:112: done (2 left) 2024-09-11T01:30:05.384 INFO:tasks.rados.rados.0.smithi071.stdout:113: done (1 left) 2024-09-11T01:30:05.384 INFO:tasks.rados.rados.0.smithi071.stdout:114: done (0 left) 2024-09-11T01:30:05.384 INFO:tasks.rados.rados.0.smithi071.stdout:115: write (excl) oid 35 current snap is 10 2024-09-11T01:30:05.384 INFO:tasks.rados.rados.0.smithi071.stdout:115: seq_num 65 ranges {692455=622009,2070506=712007} 2024-09-11T01:30:05.391 INFO:tasks.rados.rados.0.smithi071.stdout:115: writing smithi07134777-35 from 692455 to 1314464 tid 1 2024-09-11T01:30:05.400 INFO:tasks.rados.rados.0.smithi071.stdout:115: writing smithi07134777-35 from 2070506 to 2782513 tid 2 2024-09-11T01:30:05.400 INFO:tasks.rados.rados.0.smithi071.stdout:116: read oid 18 snap -1 2024-09-11T01:30:05.400 INFO:tasks.rados.rados.0.smithi071.stdout:116: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-11T01:30:05.401 INFO:tasks.rados.rados.0.smithi071.stdout:117: read oid 11 snap 7 2024-09-11T01:30:05.401 INFO:tasks.rados.rados.0.smithi071.stdout:117: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:30:05.401 INFO:tasks.rados.rados.0.smithi071.stdout:118: snap_create 2024-09-11T01:30:05.401 INFO:tasks.rados.rados.0.smithi071.stdout:115: finishing write tid 1 to smithi07134777-35 2024-09-11T01:30:05.406 INFO:tasks.rados.rados.0.smithi071.stdout:115: finishing write tid 2 to smithi07134777-35 2024-09-11T01:30:05.407 INFO:tasks.rados.rados.0.smithi071.stdout:115: finishing write tid 3 to smithi07134777-35 2024-09-11T01:30:05.407 INFO:tasks.rados.rados.0.smithi071.stdout:115: finishing write tid 4 to smithi07134777-35 2024-09-11T01:30:05.407 INFO:tasks.rados.rados.0.smithi071.stdout:115: oid 35 updating version 0 to 25 2024-09-11T01:30:05.407 INFO:tasks.rados.rados.0.smithi071.stdout:115: oid 35 updating version 25 to 26 2024-09-11T01:30:05.407 INFO:tasks.rados.rados.0.smithi071.stdout:115: oid 35 updating version 26 to 27 2024-09-11T01:30:05.407 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 27 (ObjNum 65 snap 10 seq_num 65) dirty exists 2024-09-11T01:30:05.407 INFO:tasks.rados.rados.0.smithi071.stdout:115: left oid 35 (ObjNum 65 snap 10 seq_num 65) 2024-09-11T01:30:05.408 INFO:tasks.rados.rados.0.smithi071.stdout:116: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-11T01:30:05.437 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1717509 2024-09-11T01:30:05.447 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 98664 2024-09-11T01:30:05.472 INFO:tasks.rados.rados.0.smithi071.stdout:117: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:30:05.472 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:30:05.534 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2101079 2024-09-11T01:30:06.313 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-11T01:30:06.313 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T01:30:06.313 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T01:30:06.316 INFO:tasks.ceph.mon.a:Started 2024-09-11T01:30:06.317 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-11T01:30:06.317 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-11T01:30:06.317 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T01:30:21.317 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-11T01:30:21.318 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-11T01:30:21.319 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:30:21.319 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:30:21.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.430+0000 7f10476dd640 10 monclient: get_monmap_and_config 2024-09-11T01:30:21.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.430+0000 7f10476dd640 10 monclient: build_initial_monmap 2024-09-11T01:30:21.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.430+0000 7f10476dd640 10 monclient: monmap: 2024-09-11T01:30:21.432 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:21.432 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:21.433 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:30:21.431335+0000 2024-09-11T01:30:21.433 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:30:21.431335+0000 2024-09-11T01:30:21.433 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:21.433 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:21.433 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:30:21.433 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:30:21.433 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:30:21.433 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:30:21.433 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:30:21.433 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:21.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.431+0000 7f10476dd640 1 Processor -- start 2024-09-11T01:30:21.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.431+0000 7f10476dd640 1 -- start start 2024-09-11T01:30:21.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.431+0000 7f10476dd640 10 monclient: init 2024-09-11T01:30:21.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.431+0000 7f10476dd640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:21.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.431+0000 7f10476dd640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-11T01:30:21.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.431+0000 7f10476dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f10400a2e90 addr v1:172.21.15.71:6789/0 2024-09-11T01:30:21.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.431+0000 7f10476dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f10400a3270 addr v1:172.21.15.12:6789/0 2024-09-11T01:30:21.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.431+0000 7f10476dd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f104013e4a0 0x7f1040140890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:21.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.431+0000 7f10476dd640 10 monclient(hunting): picked mon.noname-a con 0x7f104013e4a0 addr v2:172.21.15.12:3300/0 2024-09-11T01:30:21.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.431+0000 7f10476dd640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10400a46c0 con 0x7f10400a3270 2024-09-11T01:30:21.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.432+0000 7f10476dd640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f104006ee50 con 0x7f10400a2e90 2024-09-11T01:30:21.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.432+0000 7f10476dd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:21.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.432+0000 7f10476dd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f10400a4900 con 0x7f104013e4a0 2024-09-11T01:30:21.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.432+0000 7f10476dd640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:21.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.432+0000 7f10476dd640 10 monclient(hunting): authenticate will time out at 827.874035s 2024-09-11T01:30:21.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.432+0000 7f1045c7a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f104013e4a0 0x7f1040140890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:21.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.432+0000 7f1044c78640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f10400a3270 0x7f10400a3680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43986/0 (socket says 172.21.15.12:43986) 2024-09-11T01:30:21.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.432+0000 7f1044c78640 1 -- 172.21.15.12:0/2769584471 learned_addr learned my addr 172.21.15.12:0/2769584471 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:30:21.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.432+0000 7f1045c7a640 10 monclient(hunting): get_auth_request con 0x7f104013e4a0 auth_method 0 2024-09-11T01:30:21.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.432+0000 7f1045c7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:21.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.432+0000 7f1045c7a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:21.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.432+0000 7f1045c7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:21.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.432+0000 7f1045c7a640 1 -- 172.21.15.12:0/2769584471 >> v2:172.21.15.12:3300/0 conn(0x7f104013e4a0 msgr2=0x7f1040140890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:30:21.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.432+0000 7f1045c7a640 1 -- 172.21.15.12:0/2769584471 >> v2:172.21.15.12:3300/0 conn(0x7f104013e4a0 msgr2=0x7f1040140890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:30:21.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.432+0000 7f1045c7a640 1 --2- 172.21.15.12:0/2769584471 >> v2:172.21.15.12:3300/0 conn(0x7f104013e4a0 0x7f1040140890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:30:21.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.432+0000 7f1045c7a640 1 --2- 172.21.15.12:0/2769584471 >> v2:172.21.15.12:3300/0 conn(0x7f104013e4a0 0x7f1040140890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:30:21.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.434+0000 7f1044c78640 1 -- 172.21.15.12:0/2769584471 >> v1:172.21.15.12:6789/0 conn(0x7f10400a3270 legacy=0x7f10400a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:21.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.434+0000 7f1045479640 1 -- 172.21.15.12:0/2769584471 >> v1:172.21.15.71:6789/0 conn(0x7f10400a2e90 legacy=0x7f104006dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.634+0000 7f1045c7a640 1 --2- 172.21.15.12:0/2769584471 >> v2:172.21.15.12:3300/0 conn(0x7f104013e4a0 0x7f1040140890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.634+0000 7f1045c7a640 10 monclient(hunting): get_auth_request con 0x7f104013e4a0 auth_method 0 2024-09-11T01:30:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.634+0000 7f1045c7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.634+0000 7f1045c7a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.634+0000 7f1045c7a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:30:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.634+0000 7f1045c7a640 1 -- 172.21.15.12:0/2769584471 >> v2:172.21.15.12:3300/0 conn(0x7f104013e4a0 msgr2=0x7f1040140890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:30:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.634+0000 7f1045c7a640 1 -- 172.21.15.12:0/2769584471 >> v2:172.21.15.12:3300/0 conn(0x7f104013e4a0 msgr2=0x7f1040140890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:30:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.634+0000 7f1045c7a640 1 --2- 172.21.15.12:0/2769584471 >> v2:172.21.15.12:3300/0 conn(0x7f104013e4a0 0x7f1040140890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:30:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.634+0000 7f1045c7a640 1 --2- 172.21.15.12:0/2769584471 >> v2:172.21.15.12:3300/0 conn(0x7f104013e4a0 0x7f1040140890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:30:21.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.738+0000 7f1037fff640 1 -- 172.21.15.12:0/2769584471 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2577177927 0 0) 0x7f104006ee50 con 0x7f10400a2e90 2024-09-11T01:30:21.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.738+0000 7f1037fff640 10 monclient(hunting): _negotiate 2024-09-11T01:30:21.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.738+0000 7f1037fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:21.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.738+0000 7f1037fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:21.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.738+0000 7f1037fff640 10 monclient(hunting): my global_id is 4357 2024-09-11T01:30:21.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.738+0000 7f1037fff640 1 -- 172.21.15.12:0/2769584471 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f102c003520 con 0x7f10400a2e90 2024-09-11T01:30:21.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.738+0000 7f1037fff640 1 -- 172.21.15.12:0/2769584471 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1505540226 0 0) 0x7f102c003520 con 0x7f10400a2e90 2024-09-11T01:30:21.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.738+0000 7f1037fff640 1 -- 172.21.15.12:0/2769584471 >> v2:172.21.15.12:3300/0 conn(0x7f104013e4a0 msgr2=0x7f1040140890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:30:21.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.738+0000 7f1037fff640 1 --2- 172.21.15.12:0/2769584471 >> v2:172.21.15.12:3300/0 conn(0x7f104013e4a0 0x7f1040140890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:21.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 1 -- 172.21.15.12:0/2769584471 >> v1:172.21.15.12:6789/0 conn(0x7f10400a3270 legacy=0x7f10400a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:21.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 10 monclient: _finish_hunting 0 2024-09-11T01:30:21.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 1 monclient: found mon.noname-c-legacy 2024-09-11T01:30:21.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:21.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-09-11T01:30:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 1 -- 172.21.15.12:0/2769584471 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f104006eb70 con 0x7f10400a2e90 2024-09-11T01:30:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 10 monclient: _finish_auth 0 2024-09-11T01:30:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 10 monclient: _check_auth_tickets 2024-09-11T01:30:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 1 -- 172.21.15.12:0/2769584471 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f103c0019f0 con 0x7f10400a2e90 2024-09-11T01:30:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-11T01:30:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 10 monclient: dump: 2024-09-11T01:30:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:21.745 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-09-11T01:30:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 1 -- 172.21.15.12:0/2769584471 >> v1:172.21.15.71:6789/0 conn(0x7f10400a2e90 legacy=0x7f104006dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:21.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:30:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 1 --2- 172.21.15.12:0/2769584471 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f102c003b20 0x7f102c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 10 monclient(hunting): picked mon.c con 0x7f102c003b20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 1 --2- 172.21.15.12:0/2769584471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f102c004800 0x7f102c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 10 monclient(hunting): picked mon.a con 0x7f102c004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:30:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 1 --2- 172.21.15.12:0/2769584471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f102c0099b0 0x7f102c00bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 10 monclient(hunting): picked mon.b con 0x7f102c0099b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:30:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 1 -- 172.21.15.12:0/2769584471 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f10400a4900 con 0x7f102c004800 2024-09-11T01:30:21.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:21.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 1 -- 172.21.15.12:0/2769584471 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f102c00c3c0 con 0x7f102c0099b0 2024-09-11T01:30:21.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:21.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 1 -- 172.21.15.12:0/2769584471 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f102c00c5c0 con 0x7f102c003b20 2024-09-11T01:30:21.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:21.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1037fff640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:30:21.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045479640 1 --2- 172.21.15.12:0/2769584471 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f102c003b20 0x7f102c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:21.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045c7a640 1 --2- 172.21.15.12:0/2769584471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f102c0099b0 0x7f102c00bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:21.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045479640 10 monclient(hunting): get_auth_request con 0x7f102c003b20 auth_method 0 2024-09-11T01:30:21.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045479640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:21.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045479640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:21.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045479640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:30:21.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045c7a640 10 monclient(hunting): get_auth_request con 0x7f102c0099b0 auth_method 0 2024-09-11T01:30:21.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045c7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:21.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045c7a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:21.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045c7a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:30:21.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045479640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:21.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045479640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:21.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045479640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:30:21.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045c7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:21.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045c7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:21.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045c7a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:30:21.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045479640 10 monclient(hunting): handle_auth_done global_id 4357 payload 293 2024-09-11T01:30:21.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045479640 1 -- 172.21.15.12:0/2769584471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f102c0099b0 msgr2=0x7f102c00bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:30:21.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045479640 1 --2- 172.21.15.12:0/2769584471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f102c0099b0 0x7f102c00bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:21.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045479640 1 -- 172.21.15.12:0/2769584471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f102c004800 msgr2=0x7f102c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:30:21.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045479640 1 --2- 172.21.15.12:0/2769584471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f102c004800 0x7f102c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:21.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045479640 10 monclient: _finish_hunting 0 2024-09-11T01:30:21.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045479640 1 monclient: found mon.c 2024-09-11T01:30:21.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045479640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:21.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045479640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:21.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045479640 1 -- 172.21.15.12:0/2769584471 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f102c003520 con 0x7f102c003b20 2024-09-11T01:30:21.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045479640 10 monclient: _finish_auth 0 2024-09-11T01:30:21.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045479640 10 monclient: _check_auth_tickets 2024-09-11T01:30:21.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045479640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:21.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f1045c7a640 1 -- 172.21.15.12:0/2769584471 reap_dead start 2024-09-11T01:30:21.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f10476dd640 5 monclient: authenticate success, global_id 4357 2024-09-11T01:30:21.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.739+0000 7f10476dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:30:21.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.742+0000 7f1045479640 1 -- 172.21.15.12:0/2769584471 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f102c003b20 msgr2=0x7f102c003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:21.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.742+0000 7f1045479640 1 --2- 172.21.15.12:0/2769584471 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f102c003b20 0x7f102c003f40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f103c0013e0 tx=0x7f102c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2e301a65f797250 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:21.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.744+0000 7f1037fff640 1 -- 172.21.15.12:0/2769584471 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103c00c0e0 con 0x7f102c003b20 2024-09-11T01:30:21.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.744+0000 7f1037fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:21.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.744+0000 7f1037fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:21.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.744+0000 7f1037fff640 10 monclient: dump: 2024-09-11T01:30:21.760 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:21.760 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:21.760 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:21.760 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:21.760 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:21.761 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.744+0000 7f1037fff640 1 -- 172.21.15.12:0/2769584471 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f103c02e070 con 0x7f102c003b20 2024-09-11T01:30:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.744+0000 7f1037fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.744+0000 7f1037fff640 1 -- 172.21.15.12:0/2769584471 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103c0074b0 con 0x7f102c003b20 2024-09-11T01:30:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.744+0000 7f10476dd640 10 monclient: get_monmap_and_config success 2024-09-11T01:30:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.744+0000 7f10476dd640 4 set_mon_vals no callback set 2024-09-11T01:30:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.744+0000 7f1037fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.744+0000 7f1037fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:21.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.744+0000 7f1037fff640 10 monclient: dump: 2024-09-11T01:30:21.762 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:21.762 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:21.762 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:21.762 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:21.762 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:21.762 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:21.762 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:21.762 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:21.762 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:21.762 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:21.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.745+0000 7f10476dd640 10 monclient: shutdown 2024-09-11T01:30:21.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.745+0000 7f10476dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:21.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.745+0000 7f10476dd640 1 -- 172.21.15.12:0/2769584471 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f102c003b20 msgr2=0x7f102c003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:21.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.745+0000 7f10476dd640 1 --2- 172.21.15.12:0/2769584471 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f102c003b20 0x7f102c003f40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f103c0013e0 tx=0x7f102c005830 comp rx=0 tx=0).stop 2024-09-11T01:30:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.745+0000 7f10476dd640 1 -- 172.21.15.12:0/2769584471 shutdown_connections 2024-09-11T01:30:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.745+0000 7f10476dd640 1 --2- 172.21.15.12:0/2769584471 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f102c003b20 0x7f102c003f40 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.745+0000 7f10476dd640 1 -- 172.21.15.12:0/2769584471 >> 172.21.15.12:0/2769584471 conn(0x7f10400a8490 msgr2=0x7f10400a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.745+0000 7f10476dd640 1 -- 172.21.15.12:0/2769584471 shutdown_connections 2024-09-11T01:30:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.745+0000 7f10476dd640 1 -- 172.21.15.12:0/2769584471 wait complete. 2024-09-11T01:30:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.746+0000 7f10476dd640 10 monclient: build_initial_monmap 2024-09-11T01:30:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.746+0000 7f10476dd640 10 monclient: monmap: 2024-09-11T01:30:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:30:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:30:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:30:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:30:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:30:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:30:21.764 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.746+0000 7f10476dd640 1 Processor -- start 2024-09-11T01:30:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f10476dd640 1 -- start start 2024-09-11T01:30:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f10476dd640 10 monclient: init 2024-09-11T01:30:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f10476dd640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f10476dd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:30:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f10476dd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f102c003b20 0x7f1040138830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f10476dd640 10 monclient(hunting): picked mon.noname-a con 0x7f102c003b20 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:30:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f10476dd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1040138d70 0x7f10401c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f10476dd640 10 monclient(hunting): picked mon.noname-c con 0x7f1040138d70 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f10476dd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10401c9c10 0x7f10401ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f10476dd640 10 monclient(hunting): picked mon.noname-b con 0x7f10401c9c10 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:30:21.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f10476dd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:21.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f10476dd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f104006dde0 con 0x7f102c003b20 2024-09-11T01:30:21.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f10476dd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:21.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f10476dd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f104006dc60 con 0x7f10401c9c10 2024-09-11T01:30:21.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f10476dd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:21.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f10476dd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f104006df60 con 0x7f1040138d70 2024-09-11T01:30:21.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f10476dd640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:21.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f10476dd640 10 monclient(hunting): authenticate will time out at 828.189648s 2024-09-11T01:30:21.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f1045479640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f102c003b20 0x7f1040138830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f1044c78640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1040138d70 0x7f10401c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f1045479640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f102c003b20 0x7f1040138830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40976/0 (socket says 172.21.15.12:40976) 2024-09-11T01:30:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f1045479640 1 -- 172.21.15.12:0/2148047237 learned_addr learned my addr 172.21.15.12:0/2148047237 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:30:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f1045479640 10 monclient(hunting): get_auth_request con 0x7f102c003b20 auth_method 0 2024-09-11T01:30:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f1045479640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f1045479640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f1045479640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f1044c78640 10 monclient(hunting): get_auth_request con 0x7f1040138d70 auth_method 0 2024-09-11T01:30:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f1044c78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f1044c78640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f1044c78640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f1045479640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f1045479640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f1045c7a640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10401c9c10 0x7f10401ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f1045479640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f1044c78640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f1044c78640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f1044c78640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.747+0000 7f1045c7a640 10 monclient(hunting): get_auth_request con 0x7f10401c9c10 auth_method 0 2024-09-11T01:30:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.748+0000 7f1045c7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.748+0000 7f1045c7a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.748+0000 7f1045c7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.748+0000 7f1045c7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.748+0000 7f1045c7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.748+0000 7f1045c7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.748+0000 7f1045479640 10 monclient(hunting): handle_auth_done global_id 14097 payload 824 2024-09-11T01:30:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.748+0000 7f1045479640 1 -- 172.21.15.12:0/2148047237 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1040138d70 msgr2=0x7f10401c96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.748+0000 7f1045479640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1040138d70 0x7f10401c96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.748+0000 7f1045479640 1 -- 172.21.15.12:0/2148047237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10401c9c10 msgr2=0x7f10401ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.748+0000 7f1045479640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10401c9c10 0x7f10401ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.748+0000 7f1045479640 10 monclient: _finish_hunting 0 2024-09-11T01:30:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.748+0000 7f1045479640 1 monclient: found mon.noname-a 2024-09-11T01:30:21.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.748+0000 7f1045479640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:21.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.748+0000 7f1045479640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:30:21.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.748+0000 7f1045479640 1 -- 172.21.15.12:0/2148047237 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f104013bf70 con 0x7f102c003b20 2024-09-11T01:30:21.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.748+0000 7f1044c78640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1040138d70 0x7f10401c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:30:21.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.750+0000 7f1045479640 1 -- 172.21.15.12:0/2148047237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f102c003b20 msgr2=0x7f1040138830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:21.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.750+0000 7f1045479640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f102c003b20 0x7f1040138830 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f103c001520 tx=0x7f104006ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:21.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.750+0000 7f1035ffb640 1 -- 172.21.15.12:0/2148047237 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103c0059d0 con 0x7f102c003b20 2024-09-11T01:30:21.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.750+0000 7f1035ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.750+0000 7f1035ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:30:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.750+0000 7f1035ffb640 10 monclient: dump: 2024-09-11T01:30:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:21.770 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.750+0000 7f1035ffb640 10 monclient: _finish_auth 0 2024-09-11T01:30:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.750+0000 7f1035ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:30:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.750+0000 7f1035ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.750+0000 7f1035ffb640 1 -- 172.21.15.12:0/2148047237 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f103c005b70 con 0x7f102c003b20 2024-09-11T01:30:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.750+0000 7f1035ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.750+0000 7f1035ffb640 1 -- 172.21.15.12:0/2148047237 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103c005d40 con 0x7f102c003b20 2024-09-11T01:30:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.750+0000 7f1035ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.750+0000 7f1035ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:30:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.750+0000 7f1035ffb640 10 monclient: dump: 2024-09-11T01:30:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:21.772 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.750+0000 7f10476dd640 5 monclient: authenticate success, global_id 14097 2024-09-11T01:30:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.750+0000 7f10476dd640 10 monclient: _renew_subs 2024-09-11T01:30:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.750+0000 7f10476dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:30:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.750+0000 7f10476dd640 1 -- 172.21.15.12:0/2148047237 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10401ca050 con 0x7f102c003b20 2024-09-11T01:30:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.750+0000 7f10377fe640 4 set_mon_vals no callback set 2024-09-11T01:30:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f10476dd640 10 monclient: _renew_subs 2024-09-11T01:30:21.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f10476dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:30:21.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f10476dd640 1 -- 172.21.15.12:0/2148047237 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10401cd110 con 0x7f102c003b20 2024-09-11T01:30:21.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1045479640 1 -- 172.21.15.12:0/2148047237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f102c003b20 msgr2=0x7f1040138830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:30:21.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1045479640 1 -- 172.21.15.12:0/2148047237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f102c003b20 msgr2=0x7f1040138830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:30:21.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1045479640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f102c003b20 0x7f1040138830 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f103c001520 tx=0x7f104006ed40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:30:21.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1045479640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f102c003b20 0x7f1040138830 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f103c001520 tx=0x7f104006ed40 comp rx=0 tx=0).stop 2024-09-11T01:30:21.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1035ffb640 10 monclient: ms_handle_reset current mon [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:30:21.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1035ffb640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:21.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1035ffb640 1 -- 172.21.15.12:0/2148047237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f102c003b20 msgr2=0x7f1040138830 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:30:21.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1035ffb640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f102c003b20 0x7f1040138830 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:21.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1035ffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:30:21.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1035ffb640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1040138d70 0x7f10240058a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:21.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1035ffb640 10 monclient(hunting): picked mon.c con 0x7f1040138d70 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:21.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1035ffb640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10401c9c10 0x7f10240082e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:21.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1035ffb640 10 monclient(hunting): picked mon.b con 0x7f10401c9c10 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:30:21.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1035ffb640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1024008cf0 0x7f102400b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:21.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1035ffb640 10 monclient(hunting): picked mon.a con 0x7f1024008cf0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:30:21.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1035ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:21.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1035ffb640 1 -- 172.21.15.12:0/2148047237 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f102400bc30 con 0x7f1024008cf0 2024-09-11T01:30:21.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1035ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:21.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1035ffb640 1 -- 172.21.15.12:0/2148047237 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f102400bdb0 con 0x7f10401c9c10 2024-09-11T01:30:21.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1035ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:21.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1035ffb640 1 -- 172.21.15.12:0/2148047237 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f102400bf60 con 0x7f1040138d70 2024-09-11T01:30:21.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1035ffb640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:21.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1045479640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1040138d70 0x7f10240058a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:21.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1045479640 10 monclient(hunting): get_auth_request con 0x7f1040138d70 auth_method 0 2024-09-11T01:30:21.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1045479640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:21.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1045479640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:21.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1045479640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:30:21.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1045c7a640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1024008cf0 0x7f102400b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:21.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1045c7a640 10 monclient(hunting): get_auth_request con 0x7f1024008cf0 auth_method 0 2024-09-11T01:30:21.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1045c7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:21.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1045c7a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:21.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1045c7a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:30:21.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1045479640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:21.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1045479640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:21.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1045479640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:30:21.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1045c7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:21.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1045c7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:21.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1045c7a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:30:21.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.751+0000 7f1044c78640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10401c9c10 0x7f10240082e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:21.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.752+0000 7f1044c78640 10 monclient(hunting): get_auth_request con 0x7f10401c9c10 auth_method 0 2024-09-11T01:30:21.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.752+0000 7f1044c78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:21.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.752+0000 7f1044c78640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:21.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.752+0000 7f1044c78640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:30:21.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.752+0000 7f1045479640 10 monclient(hunting): handle_auth_done global_id 14097 payload 293 2024-09-11T01:30:21.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.752+0000 7f1045479640 1 -- 172.21.15.12:0/2148047237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10401c9c10 msgr2=0x7f10240082e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:21.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.752+0000 7f1045479640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10401c9c10 0x7f10240082e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:21.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.752+0000 7f1045479640 1 -- 172.21.15.12:0/2148047237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1024008cf0 msgr2=0x7f102400b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.752+0000 7f1045479640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1024008cf0 0x7f102400b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.752+0000 7f1045479640 10 monclient: _finish_hunting 0 2024-09-11T01:30:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.752+0000 7f1045479640 1 monclient: found mon.c 2024-09-11T01:30:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.752+0000 7f1045479640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.752+0000 7f1045479640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.752+0000 7f1045479640 1 -- 172.21.15.12:0/2148047237 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f102400c280 con 0x7f1040138d70 2024-09-11T01:30:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.752+0000 7f1045479640 10 monclient: _finish_auth 0 2024-09-11T01:30:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.752+0000 7f1045479640 10 monclient: _check_auth_tickets 2024-09-11T01:30:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.752+0000 7f1045479640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.752+0000 7f1045c7a640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1024008cf0 0x7f102400b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:30:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.752+0000 7f1044c78640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10401c9c10 0x7f10240082e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:30:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.752+0000 7f10476dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.752+0000 7f10476dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.752+0000 7f10476dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.752+0000 7f10476dd640 1 -- 172.21.15.12:0/2148047237 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10400a3680 con 0x7f1040138d70 2024-09-11T01:30:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.754+0000 7f1045479640 1 -- 172.21.15.12:0/2148047237 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1040138d70 msgr2=0x7f10240058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.754+0000 7f1045479640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1040138d70 0x7f10240058a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f103c03f000 tx=0x7f103c007000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.754+0000 7f1035ffb640 1 -- 172.21.15.12:0/2148047237 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103c033470 con 0x7f1040138d70 2024-09-11T01:30:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.754+0000 7f1035ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.754+0000 7f1035ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.754+0000 7f1035ffb640 10 monclient: dump: 2024-09-11T01:30:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:21.780 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:21.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.754+0000 7f1035ffb640 1 -- 172.21.15.12:0/2148047237 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f103c058070 con 0x7f1040138d70 2024-09-11T01:30:21.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.754+0000 7f1035ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:21.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.754+0000 7f1036ffd640 4 set_mon_vals no callback set 2024-09-11T01:30:21.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.755+0000 7f1035ffb640 1 -- 172.21.15.12:0/2148047237 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f103c053460 con 0x7f1040138d70 2024-09-11T01:30:21.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.755+0000 7f1035ffb640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f102c003b20 0x7f10240484a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:21.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.755+0000 7f1035ffb640 1 -- 172.21.15.12:0/2148047237 <== mon.2 v2:172.21.15.12:3301/0 4 ==== osd_map(38..38 src has 1..38) ==== 4755+0+0 (secure 0 0 0) 0x7f103c075ad0 con 0x7f1040138d70 2024-09-11T01:30:21.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.756+0000 7f1044c78640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f102c003b20 0x7f10240484a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:21.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.756+0000 7f1044c78640 10 monclient: get_auth_request con 0x7f102c003b20 auth_method 0 2024-09-11T01:30:21.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.758+0000 7f1044c78640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f102c003b20 0x7f10240484a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1030005880 tx=0x7f1030009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:21.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.760+0000 7f1035ffb640 1 -- 172.21.15.12:0/2148047237 <== mon.2 v2:172.21.15.12:3301/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f103c04c060 con 0x7f1040138d70 2024-09-11T01:30:21.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.760+0000 7f1035ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:21.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:21.760+0000 7f1035ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:30:22.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.021+0000 7f10476dd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:30:22.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.021+0000 7f10476dd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:30:22.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.021+0000 7f10476dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:22.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.021+0000 7f10476dd640 1 -- 172.21.15.12:0/2148047237 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f10400a0800 con 0x7f1040138d70 2024-09-11T01:30:22.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.022+0000 7f1035ffb640 1 -- 172.21.15.12:0/2148047237 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f103c047d10 con 0x7f1040138d70 2024-09-11T01:30:22.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.022+0000 7f1035ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:30:22.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.022+0000 7f1035ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:30:22.024 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:30:22.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.024+0000 7f10476dd640 1 -- 172.21.15.12:0/2148047237 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f102c003b20 msgr2=0x7f10240484a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:22.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.024+0000 7f10476dd640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f102c003b20 0x7f10240484a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1030005880 tx=0x7f1030009040 comp rx=0 tx=0).stop 2024-09-11T01:30:22.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.024+0000 7f10476dd640 10 monclient: shutdown 2024-09-11T01:30:22.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.024+0000 7f10476dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:22.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.024+0000 7f10476dd640 1 -- 172.21.15.12:0/2148047237 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1040138d70 msgr2=0x7f10240058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:22.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.024+0000 7f10476dd640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1040138d70 0x7f10240058a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f103c03f000 tx=0x7f103c007000 comp rx=0 tx=0).stop 2024-09-11T01:30:22.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.024+0000 7f10476dd640 1 -- 172.21.15.12:0/2148047237 shutdown_connections 2024-09-11T01:30:22.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.024+0000 7f10476dd640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f102c003b20 0x7f10240484a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.024+0000 7f10476dd640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1024008cf0 0x7f102400b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.024+0000 7f10476dd640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10401c9c10 0x7f10240082e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.024+0000 7f10476dd640 1 --2- 172.21.15.12:0/2148047237 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1040138d70 0x7f10240058a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.024+0000 7f10476dd640 1 -- 172.21.15.12:0/2148047237 >> 172.21.15.12:0/2148047237 conn(0x7f10400a8490 msgr2=0x7f10400a7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:22.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.024+0000 7f10476dd640 1 -- 172.21.15.12:0/2148047237 shutdown_connections 2024-09-11T01:30:22.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.024+0000 7f10476dd640 1 -- 172.21.15.12:0/2148047237 wait complete. 2024-09-11T01:30:22.037 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:30:22.037 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:30:22.037 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:30:22.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.137+0000 7ff15636b640 10 monclient: get_monmap_and_config 2024-09-11T01:30:22.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.137+0000 7ff15636b640 10 monclient: build_initial_monmap 2024-09-11T01:30:22.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.137+0000 7ff15636b640 10 monclient: monmap: 2024-09-11T01:30:22.139 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:22.139 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:22.140 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:30:22.138502+0000 2024-09-11T01:30:22.140 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:30:22.138502+0000 2024-09-11T01:30:22.140 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:22.140 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.140 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:30:22.140 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:30:22.140 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:30:22.140 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:30:22.140 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:30:22.140 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.138+0000 7ff15636b640 1 Processor -- start 2024-09-11T01:30:22.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.138+0000 7ff15636b640 1 -- start start 2024-09-11T01:30:22.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.138+0000 7ff15636b640 10 monclient: init 2024-09-11T01:30:22.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff15636b640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:22.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff15636b640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-11T01:30:22.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff15636b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff15006f540 addr v1:172.21.15.12:6789/0 2024-09-11T01:30:22.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff15636b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff15009cff0 0x7ff1500a5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff15636b640 10 monclient(hunting): picked mon.noname-b con 0x7ff15009cff0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:22.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff15636b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff15013d230 addr v1:172.21.15.71:6789/0 2024-09-11T01:30:22.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff15636b640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1500a6ea0 con 0x7ff15006f540 2024-09-11T01:30:22.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff15636b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff15009d4d0 con 0x7ff15013d230 2024-09-11T01:30:22.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff15636b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:22.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff15636b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff15006f920 con 0x7ff15009cff0 2024-09-11T01:30:22.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff15636b640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:22.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff15636b640 10 monclient(hunting): authenticate will time out at 828.581272s 2024-09-11T01:30:22.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff14ffff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff15006f540 0x7ff1500a5020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43998/0 (socket says 172.21.15.12:43998) 2024-09-11T01:30:22.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff14ffff640 1 -- 172.21.15.12:0/559719181 learned_addr learned my addr 172.21.15.12:0/559719181 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:30:22.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff14f7fe640 1 --2- 172.21.15.12:0/559719181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff15009cff0 0x7ff1500a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff14f7fe640 10 monclient(hunting): get_auth_request con 0x7ff15009cff0 auth_method 0 2024-09-11T01:30:22.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff14f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:22.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff14f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:22.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff14f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:22.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff14f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:22.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff14f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:22.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff14f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:22.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff14f7fe640 10 monclient(hunting): handle_auth_done global_id 4427 payload 274 2024-09-11T01:30:22.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff14f7fe640 1 -- 172.21.15.12:0/559719181 >> v1:172.21.15.71:6789/0 conn(0x7ff15013d230 legacy=0x7ff15013f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:30:22.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff14f7fe640 1 -- 172.21.15.12:0/559719181 >> v1:172.21.15.12:6789/0 conn(0x7ff15006f540 legacy=0x7ff1500a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:30:22.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff14f7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:30:22.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff14f7fe640 1 monclient: found mon.noname-b 2024-09-11T01:30:22.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff14f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:22.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff14f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:30:22.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.139+0000 7ff14f7fe640 1 -- 172.21.15.12:0/559719181 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff15009c330 con 0x7ff15009cff0 2024-09-11T01:30:22.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.142+0000 7ff14f7fe640 1 -- 172.21.15.12:0/559719181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff15009cff0 msgr2=0x7ff1500a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:22.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.142+0000 7ff14f7fe640 1 --2- 172.21.15.12:0/559719181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff15009cff0 0x7ff1500a5730 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff1440097d0 tx=0x7ff144029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f877ac198fac9cdf server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:22.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.142+0000 7ff14effd640 1 -- 172.21.15.12:0/559719181 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff144037070 con 0x7ff15009cff0 2024-09-11T01:30:22.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.142+0000 7ff14effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:22.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.142+0000 7ff14effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:30:22.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.142+0000 7ff14effd640 10 monclient: dump: 2024-09-11T01:30:22.151 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:22.151 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:22.152 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.152 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.152 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:22.152 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.152 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:22.152 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:22.152 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:22.152 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.142+0000 7ff14effd640 10 monclient: _finish_auth 0 2024-09-11T01:30:22.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.142+0000 7ff14effd640 10 monclient: _check_auth_tickets 2024-09-11T01:30:22.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.142+0000 7ff14effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:22.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.142+0000 7ff14effd640 1 -- 172.21.15.12:0/559719181 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff144032440 con 0x7ff15009cff0 2024-09-11T01:30:22.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.142+0000 7ff15636b640 5 monclient: authenticate success, global_id 4427 2024-09-11T01:30:22.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.142+0000 7ff15636b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:30:22.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.142+0000 7ff14effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:22.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.142+0000 7ff15636b640 10 monclient: get_monmap_and_config success 2024-09-11T01:30:22.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.142+0000 7ff14effd640 1 -- 172.21.15.12:0/559719181 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff144032810 con 0x7ff15009cff0 2024-09-11T01:30:22.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.142+0000 7ff14effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:22.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.142+0000 7ff15636b640 4 set_mon_vals no callback set 2024-09-11T01:30:22.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.142+0000 7ff14effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:22.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.142+0000 7ff14effd640 10 monclient: dump: 2024-09-11T01:30:22.153 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:22.153 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:22.154 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.154 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.154 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:22.154 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.154 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:22.154 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:22.154 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:22.154 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.143+0000 7ff15636b640 10 monclient: shutdown 2024-09-11T01:30:22.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.143+0000 7ff15636b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:22.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.143+0000 7ff15636b640 1 -- 172.21.15.12:0/559719181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff15009cff0 msgr2=0x7ff1500a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:22.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.143+0000 7ff15636b640 1 --2- 172.21.15.12:0/559719181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff15009cff0 0x7ff1500a5730 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff1440097d0 tx=0x7ff144029fb0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.143+0000 7ff15636b640 1 -- 172.21.15.12:0/559719181 shutdown_connections 2024-09-11T01:30:22.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.143+0000 7ff15636b640 1 --2- 172.21.15.12:0/559719181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff15009cff0 0x7ff1500a5730 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.143+0000 7ff15636b640 1 -- 172.21.15.12:0/559719181 >> 172.21.15.12:0/559719181 conn(0x7ff1500a7300 msgr2=0x7ff1500a7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:22.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.143+0000 7ff15636b640 1 -- 172.21.15.12:0/559719181 shutdown_connections 2024-09-11T01:30:22.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.143+0000 7ff15636b640 1 -- 172.21.15.12:0/559719181 wait complete. 2024-09-11T01:30:22.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.143+0000 7ff15636b640 10 monclient: build_initial_monmap 2024-09-11T01:30:22.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.143+0000 7ff15636b640 10 monclient: monmap: 2024-09-11T01:30:22.155 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:22.155 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:30:22.155 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:30:22.155 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:30:22.155 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:22.155 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.156 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:30:22.156 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:30:22.156 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:30:22.156 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff15636b640 1 Processor -- start 2024-09-11T01:30:22.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff15636b640 1 -- start start 2024-09-11T01:30:22.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff15636b640 10 monclient: init 2024-09-11T01:30:22.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff15636b640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:22.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff15636b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:30:22.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff15636b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15006f540 0x7ff1501cd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff15636b640 10 monclient(hunting): picked mon.noname-a con 0x7ff15006f540 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:30:22.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff15636b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff15009cff0 0x7ff150138120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff15636b640 10 monclient(hunting): picked mon.noname-b con 0x7ff15009cff0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:30:22.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff15636b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff15013d230 0x7ff15013a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff15636b640 10 monclient(hunting): picked mon.noname-c con 0x7ff15013d230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:22.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff15636b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:22.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff15636b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff150140d40 con 0x7ff15006f540 2024-09-11T01:30:22.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff15636b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:22.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff15636b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff1500a5e80 con 0x7ff15009cff0 2024-09-11T01:30:22.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff15636b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:22.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff15636b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff150140ec0 con 0x7ff15013d230 2024-09-11T01:30:22.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff15636b640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:22.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff15636b640 10 monclient(hunting): authenticate will time out at 828.586775s 2024-09-11T01:30:22.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff14ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15006f540 0x7ff1501cd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff154908640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff15013d230 0x7ff15013a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff14ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15006f540 0x7ff1501cd0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41006/0 (socket says 172.21.15.12:41006) 2024-09-11T01:30:22.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff14ffff640 1 -- 172.21.15.12:0/1164348246 learned_addr learned my addr 172.21.15.12:0/1164348246 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:30:22.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff14ffff640 10 monclient(hunting): get_auth_request con 0x7ff15006f540 auth_method 0 2024-09-11T01:30:22.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff14ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:22.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff154908640 1 --2- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff15013d230 0x7ff15013a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:40590/0 (socket says 172.21.15.12:40590) 2024-09-11T01:30:22.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff14ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:22.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff14ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:22.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff154908640 10 monclient(hunting): get_auth_request con 0x7ff15013d230 auth_method 0 2024-09-11T01:30:22.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff154908640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:22.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff154908640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:22.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.144+0000 7ff154908640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:22.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.145+0000 7ff14ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:22.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.145+0000 7ff14ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:22.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.145+0000 7ff14f7fe640 1 --2- 172.21.15.12:0/1164348246 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff15009cff0 0x7ff150138120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.145+0000 7ff14ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:22.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.145+0000 7ff14f7fe640 10 monclient(hunting): get_auth_request con 0x7ff15009cff0 auth_method 0 2024-09-11T01:30:22.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.145+0000 7ff14f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:22.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.145+0000 7ff14f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:22.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.145+0000 7ff14f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:22.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.145+0000 7ff154908640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:22.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.145+0000 7ff154908640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:22.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.145+0000 7ff154908640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:22.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.145+0000 7ff14f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:22.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.145+0000 7ff14f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:22.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.145+0000 7ff14f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:22.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.145+0000 7ff14ffff640 10 monclient(hunting): handle_auth_done global_id 14100 payload 824 2024-09-11T01:30:22.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.145+0000 7ff14ffff640 1 -- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff15013d230 msgr2=0x7ff15013a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:22.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.145+0000 7ff14ffff640 1 --2- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff15013d230 0x7ff15013a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.145+0000 7ff14ffff640 1 -- 172.21.15.12:0/1164348246 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff15009cff0 msgr2=0x7ff150138120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:22.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.145+0000 7ff14ffff640 1 --2- 172.21.15.12:0/1164348246 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff15009cff0 0x7ff150138120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.145+0000 7ff14ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:30:22.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.145+0000 7ff14ffff640 1 monclient: found mon.noname-a 2024-09-11T01:30:22.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.145+0000 7ff14ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:22.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.145+0000 7ff14ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:30:22.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.145+0000 7ff14ffff640 1 -- 172.21.15.12:0/1164348246 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff15013c3b0 con 0x7ff15006f540 2024-09-11T01:30:22.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.145+0000 7ff154908640 1 --2- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff15013d230 0x7ff15013a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:30:22.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.147+0000 7ff14ffff640 1 -- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15006f540 msgr2=0x7ff1501cd0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:22.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.147+0000 7ff14ffff640 1 --2- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15006f540 0x7ff1501cd0f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff14000b430 tx=0x7ff14000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:22.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.147+0000 7ff14cff9640 1 -- 172.21.15.12:0/1164348246 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff140015020 con 0x7ff15006f540 2024-09-11T01:30:22.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.147+0000 7ff14cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:22.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.147+0000 7ff14cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:30:22.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.147+0000 7ff14cff9640 10 monclient: dump: 2024-09-11T01:30:22.163 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:22.163 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:22.163 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.163 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.163 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:22.163 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.163 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:22.163 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:22.164 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:22.164 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.147+0000 7ff14cff9640 10 monclient: _finish_auth 0 2024-09-11T01:30:22.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.147+0000 7ff14cff9640 10 monclient: _check_auth_tickets 2024-09-11T01:30:22.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.147+0000 7ff14cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:22.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.147+0000 7ff14cff9640 1 -- 172.21.15.12:0/1164348246 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff140011a60 con 0x7ff15006f540 2024-09-11T01:30:22.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.147+0000 7ff15636b640 5 monclient: authenticate success, global_id 14100 2024-09-11T01:30:22.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.147+0000 7ff15636b640 10 monclient: _renew_subs 2024-09-11T01:30:22.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.147+0000 7ff15636b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:30:22.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.147+0000 7ff15636b640 1 -- 172.21.15.12:0/1164348246 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff150120d70 con 0x7ff15006f540 2024-09-11T01:30:22.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.147+0000 7ff14cff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:22.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.147+0000 7ff14cff9640 1 -- 172.21.15.12:0/1164348246 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff140011e50 con 0x7ff15006f540 2024-09-11T01:30:22.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.147+0000 7ff14cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:22.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.147+0000 7ff14cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:30:22.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.147+0000 7ff14cff9640 10 monclient: dump: 2024-09-11T01:30:22.165 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:22.165 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:22.165 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.165 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.165 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:22.165 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.165 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:22.165 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:22.165 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:22.165 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.147+0000 7ff15636b640 10 monclient: _renew_subs 2024-09-11T01:30:22.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.147+0000 7ff14e7fc640 4 set_mon_vals no callback set 2024-09-11T01:30:22.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.147+0000 7ff15636b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:30:22.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.147+0000 7ff15636b640 1 -- 172.21.15.12:0/1164348246 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff150121ac0 con 0x7ff15006f540 2024-09-11T01:30:22.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.148+0000 7ff15636b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:22.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.148+0000 7ff15636b640 1 --2- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15013a750 0x7ff15013ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.148+0000 7ff15636b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:22.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.148+0000 7ff15636b640 1 -- 172.21.15.12:0/1164348246 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff15006a060 con 0x7ff15013a750 2024-09-11T01:30:22.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.148+0000 7ff14cff9640 1 -- 172.21.15.12:0/1164348246 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff14001a400 con 0x7ff15006f540 2024-09-11T01:30:22.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.148+0000 7ff14f7fe640 1 --2- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15013a750 0x7ff15013ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.148+0000 7ff14f7fe640 10 monclient: get_auth_request con 0x7ff15013a750 auth_method 0 2024-09-11T01:30:22.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.148+0000 7ff14f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:22.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.148+0000 7ff14f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:22.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.148+0000 7ff14f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:22.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.148+0000 7ff14f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:22.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.148+0000 7ff14f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:22.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.148+0000 7ff14f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:22.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.148+0000 7ff14f7fe640 10 monclient(hunting): handle_auth_done global_id 14103 payload 824 2024-09-11T01:30:22.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.148+0000 7ff14f7fe640 1 -- 172.21.15.12:0/1164348246 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff15013b070 con 0x7ff15013a750 2024-09-11T01:30:22.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.148+0000 7ff14cff9640 1 --2- 172.21.15.12:0/1164348246 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff13c03e3e0 0x7ff13c0408a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.148+0000 7ff14cff9640 1 -- 172.21.15.12:0/1164348246 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 4755+0+0 (secure 0 0 0) 0x7ff140052f80 con 0x7ff15006f540 2024-09-11T01:30:22.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.149+0000 7ff154908640 1 --2- 172.21.15.12:0/1164348246 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff13c03e3e0 0x7ff13c0408a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.149+0000 7ff154908640 10 monclient: get_auth_request con 0x7ff13c03e3e0 auth_method 0 2024-09-11T01:30:22.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.151+0000 7ff14f7fe640 1 -- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15013a750 msgr2=0x7ff15013ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:22.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.151+0000 7ff14f7fe640 1 --2- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15013a750 0x7ff15013ab30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff144034000 tx=0x7ff1440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:22.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.151+0000 7ff14cff9640 1 -- 172.21.15.12:0/1164348246 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff144037040 con 0x7ff15013a750 2024-09-11T01:30:22.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.151+0000 7ff14cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:22.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.151+0000 7ff14cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:30:22.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.151+0000 7ff14cff9640 10 monclient: dump: 2024-09-11T01:30:22.170 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:22.170 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:22.170 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.170 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.170 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:22.170 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.170 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:22.170 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:22.170 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:22.171 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.151+0000 7ff14cff9640 1 -- 172.21.15.12:0/1164348246 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff14403c610 con 0x7ff15013a750 2024-09-11T01:30:22.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.151+0000 7ff14cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:22.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.151+0000 7ff14cff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:30:22.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.151+0000 7ff14cff9640 1 -- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15013a750 msgr2=0x7ff15013ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:22.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.151+0000 7ff14cff9640 1 --2- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15013a750 0x7ff15013ab30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff144034000 tx=0x7ff1440046c0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.151+0000 7ff14cff9640 1 -- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15013a750 msgr2=0x7ff15013ab30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:30:22.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.151+0000 7ff14cff9640 1 --2- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15013a750 0x7ff15013ab30 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff144034000 tx=0x7ff1440046c0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.151+0000 7ff154908640 1 --2- 172.21.15.12:0/1164348246 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff13c03e3e0 0x7ff13c0408a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff15013bf70 tx=0x7ff148003d10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:22.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.157+0000 7ff15636b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:30:22.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.157+0000 7ff15636b640 1 --2- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15009e4e0 0x7ff15009e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.157+0000 7ff15636b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:22.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.157+0000 7ff15636b640 1 -- 172.21.15.12:0/1164348246 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff1500a5020 con 0x7ff15009e4e0 2024-09-11T01:30:22.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.157+0000 7ff14f7fe640 1 --2- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15009e4e0 0x7ff15009e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.157+0000 7ff14f7fe640 10 monclient: get_auth_request con 0x7ff15009e4e0 auth_method 0 2024-09-11T01:30:22.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.157+0000 7ff14f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:22.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.157+0000 7ff14f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:22.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.157+0000 7ff14f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:22.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.157+0000 7ff14f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:22.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.157+0000 7ff14f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:22.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.157+0000 7ff14f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:22.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.157+0000 7ff14f7fe640 10 monclient(hunting): handle_auth_done global_id 14106 payload 824 2024-09-11T01:30:22.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.157+0000 7ff14f7fe640 1 -- 172.21.15.12:0/1164348246 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff1501cdf80 con 0x7ff15009e4e0 2024-09-11T01:30:22.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.158+0000 7ff14f7fe640 1 -- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15009e4e0 msgr2=0x7ff15009e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:30:22.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.158+0000 7ff14f7fe640 1 -- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15009e4e0 msgr2=0x7ff15009e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:30:22.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.158+0000 7ff14f7fe640 1 --2- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15009e4e0 0x7ff15009e8f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7ff144029ff0 tx=0x7ff144008030 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:30:22.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.158+0000 7ff14f7fe640 1 --2- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15009e4e0 0x7ff15009e8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:30:22.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.359+0000 7ff14f7fe640 1 --2- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15009e4e0 0x7ff15009e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.359+0000 7ff14f7fe640 10 monclient: get_auth_request con 0x7ff15009e4e0 auth_method 0 2024-09-11T01:30:22.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.359+0000 7ff14f7fe640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:22.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.359+0000 7ff14f7fe640 10 monclient: _init_auth method 2 2024-09-11T01:30:22.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.359+0000 7ff14f7fe640 10 monclient: _init_auth already have auth, reseting 2024-09-11T01:30:22.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.359+0000 7ff14f7fe640 10 monclient: handle_auth_reply_more payload 9 2024-09-11T01:30:22.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.359+0000 7ff14f7fe640 10 monclient: handle_auth_reply_more payload_len 9 2024-09-11T01:30:22.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.359+0000 7ff14f7fe640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-09-11T01:30:22.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.359+0000 7ff14f7fe640 10 monclient: handle_auth_done global_id 14106 payload 293 2024-09-11T01:30:22.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.362+0000 7ff14f7fe640 1 -- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15009e4e0 msgr2=0x7ff15009e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:22.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.362+0000 7ff14f7fe640 1 --2- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15009e4e0 0x7ff15009e8f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff144004700 tx=0x7ff14403f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:22.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.362+0000 7ff14cff9640 1 -- 172.21.15.12:0/1164348246 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff14405a070 con 0x7ff15009e4e0 2024-09-11T01:30:22.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.362+0000 7ff14cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:22.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.362+0000 7ff14cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:30:22.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.362+0000 7ff14cff9640 10 monclient: dump: 2024-09-11T01:30:22.364 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:22.364 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:22.364 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.364 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.365 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:22.365 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.365 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:22.365 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:22.365 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:22.365 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.362+0000 7ff14cff9640 1 -- 172.21.15.12:0/1164348246 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4542 (secure 0 0 0) 0x7ff14404c610 con 0x7ff15009e4e0 2024-09-11T01:30:22.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.362+0000 7ff14cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:30:22.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.362+0000 7ff14cff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:30:22.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.362+0000 7ff14cff9640 1 -- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15009e4e0 msgr2=0x7ff15009e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:22.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.362+0000 7ff14cff9640 1 --2- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15009e4e0 0x7ff15009e8f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff144004700 tx=0x7ff14403f000 comp rx=0 tx=0).stop 2024-09-11T01:30:22.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.362+0000 7ff14cff9640 1 -- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15009e4e0 msgr2=0x7ff15009e8f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:30:22.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.362+0000 7ff14cff9640 1 --2- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15009e4e0 0x7ff15009e8f0 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff144004700 tx=0x7ff14403f000 comp rx=0 tx=0).stop 2024-09-11T01:30:22.366 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:30:22.366 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:30:22.366 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:30:22.366 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:30:22.366 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 14, 2024-09-11T01:30:22.366 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 15949, 2024-09-11T01:30:22.366 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:30:22.366 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:30:22.367 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:30:22.367 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:30:22.367 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:22.367 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:30:22.367 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:30:22.367 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:30:22.367 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:30:22.367 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:30:22.367 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:30:22.367 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:30:22.367 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:30:22.367 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:30:22.367 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:30:22.368 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:30:22.368 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:30:22.368 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:30:22.368 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:30:22.368 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:30:22.368 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:22.368 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:30:22.369 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:30:22.369 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:30:22.369 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:30:22.369 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:30:22.369 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:30:22.369 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:30:22.369 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:30:22.370 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:30:22.370 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:30:22.370 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:30:22.370 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:30:22.370 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:30:22.370 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:22.370 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.370 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:30:22.371 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:30:22.371 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:30:22.371 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:30:22.371 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:30:22.371 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:30:22.371 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:30:22.371 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:30:22.371 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:30:22.371 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:30:22.371 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:30:22.371 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:30:22.371 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:30:22.372 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:30:22.372 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:30:22.372 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:30:22.372 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:30:22.372 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:30:22.372 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:30:22.372 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:30:22.372 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:30:22.372 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:30:22.372 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:30:22.372 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:30:22.372 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:30:22.372 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:30:22.373 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:30:22.373 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:30:22.373 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:22.373 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:30:22.373 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.373 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:30:22.373 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.373 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:30:22.374 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:30:22.374 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:30:22.374 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:30:22.374 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.374 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:30:22.374 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:30:22.374 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:22.374 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.374 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.374 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:30:22.374 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:30:22.375 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:22.375 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:22.375 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:22.375 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.375 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:30:22.375 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:30:22.375 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:30:22.375 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:30:22.375 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:30:22.375 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.375 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.376 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:30:22.376 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:30:22.376 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:30:22.376 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:30:22.376 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.376 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:30:22.376 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:30:22.376 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:22.376 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.376 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.377 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:30:22.377 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:30:22.377 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:22.377 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:22.377 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:22.377 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.377 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:30:22.377 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:30:22.377 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:30:22.377 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:30:22.377 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:30:22.377 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.377 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.377 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:30:22.378 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:30:22.378 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:30:22.378 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:30:22.378 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.378 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:30:22.378 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:30:22.378 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:22.378 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.378 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.378 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:30:22.378 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:30:22.378 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:22.378 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:22.379 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:22.379 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.379 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:30:22.379 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:30:22.379 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:30:22.379 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:30:22.379 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:30:22.379 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:22.379 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:22.379 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.379 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:30:22.379 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:30:22.379 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.379 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:30:22.380 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:30:22.380 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:30:22.380 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:22.380 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:22.380 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:30:22.380 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.380 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:30:22.380 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:30:22.380 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:30:22.380 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:22.380 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:22.381 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.381 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:30:22.381 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:30:22.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.363+0000 7ff15636b640 1 -- 172.21.15.12:0/1164348246 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff13c03e3e0 msgr2=0x7ff13c0408a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:22.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.363+0000 7ff15636b640 1 --2- 172.21.15.12:0/1164348246 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff13c03e3e0 0x7ff13c0408a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff15013bf70 tx=0x7ff148003d10 comp rx=0 tx=0).stop 2024-09-11T01:30:22.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.363+0000 7ff15636b640 10 monclient: shutdown 2024-09-11T01:30:22.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.363+0000 7ff15636b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:22.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.363+0000 7ff15636b640 1 -- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15006f540 msgr2=0x7ff1501cd0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:22.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.363+0000 7ff15636b640 1 --2- 172.21.15.12:0/1164348246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff15006f540 0x7ff1501cd0f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff14000b430 tx=0x7ff14000b8f0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.363+0000 7ff154908640 1 -- 172.21.15.12:0/1164348246 reap_dead start 2024-09-11T01:30:22.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.363+0000 7ff154908640 1 -- 172.21.15.12:0/1164348246 reap_dead start 2024-09-11T01:30:22.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.363+0000 7ff15636b640 1 -- 172.21.15.12:0/1164348246 shutdown_connections 2024-09-11T01:30:22.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.363+0000 7ff15636b640 1 -- 172.21.15.12:0/1164348246 >> 172.21.15.12:0/1164348246 conn(0x7ff1500a7300 msgr2=0x7ff1500a3140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:22.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.363+0000 7ff15636b640 1 -- 172.21.15.12:0/1164348246 shutdown_connections 2024-09-11T01:30:22.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.363+0000 7ff15636b640 1 -- 172.21.15.12:0/1164348246 wait complete. 2024-09-11T01:30:22.384 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:30:22.384 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:30:22.384 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:30:22.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.485+0000 7f3df9964640 10 monclient: get_monmap_and_config 2024-09-11T01:30:22.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.486+0000 7f3df9964640 10 monclient: build_initial_monmap 2024-09-11T01:30:22.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.486+0000 7f3df9964640 10 monclient: monmap: 2024-09-11T01:30:22.488 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:22.488 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:22.488 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:30:22.486755+0000 2024-09-11T01:30:22.488 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:30:22.486755+0000 2024-09-11T01:30:22.488 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:22.488 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.489 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:30:22.489 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:30:22.489 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:30:22.489 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:30:22.489 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:30:22.489 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.486+0000 7f3df9964640 1 Processor -- start 2024-09-11T01:30:22.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.486+0000 7f3df9964640 1 -- start start 2024-09-11T01:30:22.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.486+0000 7f3df9964640 10 monclient: init 2024-09-11T01:30:22.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df9964640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:22.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df9964640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-11T01:30:22.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df9964640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df4137e50 0x7f3df409d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df9964640 10 monclient(hunting): picked mon.noname-b con 0x7f3df4137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:22.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df9964640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3df4138230 addr v1:172.21.15.71:6789/0 2024-09-11T01:30:22.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df9964640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3df413d380 0x7f3df413f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df9964640 10 monclient(hunting): picked mon.noname-c con 0x7f3df413d380 addr v2:172.21.15.71:3300/0 2024-09-11T01:30:22.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df9964640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3df40a6580 con 0x7f3df4138230 2024-09-11T01:30:22.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df9964640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:22.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df9964640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f3df40a62d0 con 0x7f3df413d380 2024-09-11T01:30:22.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df9964640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:22.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df9964640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f3df409bb40 con 0x7f3df4137e50 2024-09-11T01:30:22.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df9964640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:22.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df9964640 10 monclient(hunting): authenticate will time out at 828.929307s 2024-09-11T01:30:22.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df2ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df4137e50 0x7f3df409d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df2ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df4137e50 0x7f3df409d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:40594/0 (socket says 172.21.15.12:40594) 2024-09-11T01:30:22.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df2ffd640 1 -- 172.21.15.12:0/504385548 learned_addr learned my addr 172.21.15.12:0/504385548 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:30:22.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df2ffd640 10 monclient(hunting): get_auth_request con 0x7f3df4137e50 auth_method 0 2024-09-11T01:30:22.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:22.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:22.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:22.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:22.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:22.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:22.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df2ffd640 10 monclient(hunting): handle_auth_done global_id 4433 payload 274 2024-09-11T01:30:22.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df2ffd640 1 -- 172.21.15.12:0/504385548 >> v2:172.21.15.71:3300/0 conn(0x7f3df413d380 msgr2=0x7f3df413f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:30:22.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df37fe640 1 --2- 172.21.15.12:0/504385548 >> v2:172.21.15.71:3300/0 conn(0x7f3df413d380 0x7f3df413f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df2ffd640 1 --2- 172.21.15.12:0/504385548 >> v2:172.21.15.71:3300/0 conn(0x7f3df413d380 0x7f3df413f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df2ffd640 1 -- 172.21.15.12:0/504385548 >> v1:172.21.15.71:6789/0 conn(0x7f3df4138230 legacy=0x7f3df4138630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:30:22.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df2ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:30:22.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df2ffd640 1 monclient: found mon.noname-b 2024-09-11T01:30:22.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:22.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:30:22.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.487+0000 7f3df2ffd640 1 -- 172.21.15.12:0/504385548 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3df409c4a0 con 0x7f3df4137e50 2024-09-11T01:30:22.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.490+0000 7f3df2ffd640 1 -- 172.21.15.12:0/504385548 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df4137e50 msgr2=0x7f3df409d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:22.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.490+0000 7f3df2ffd640 1 --2- 172.21.15.12:0/504385548 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df4137e50 0x7f3df409d160 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc009870 tx=0x7f3ddc02a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a966cdd49313ddd server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:22.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.490+0000 7f3df1ffb640 1 -- 172.21.15.12:0/504385548 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc037070 con 0x7f3df4137e50 2024-09-11T01:30:22.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.490+0000 7f3df1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:22.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.490+0000 7f3df1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:30:22.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.490+0000 7f3df1ffb640 10 monclient: dump: 2024-09-11T01:30:22.504 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:22.504 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:22.504 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.504 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.504 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:22.504 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.504 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:22.504 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:22.504 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:22.504 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.490+0000 7f3df1ffb640 10 monclient: _finish_auth 0 2024-09-11T01:30:22.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.490+0000 7f3df1ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:30:22.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.490+0000 7f3df1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:22.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.490+0000 7f3df1ffb640 1 -- 172.21.15.12:0/504385548 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ddc032640 con 0x7f3df4137e50 2024-09-11T01:30:22.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.490+0000 7f3df1ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:22.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.490+0000 7f3df1ffb640 1 -- 172.21.15.12:0/504385548 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc032a10 con 0x7f3df4137e50 2024-09-11T01:30:22.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.490+0000 7f3df9964640 5 monclient: authenticate success, global_id 4433 2024-09-11T01:30:22.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.490+0000 7f3df9964640 10 monclient: get_monmap_and_config success 2024-09-11T01:30:22.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.490+0000 7f3df9964640 4 set_mon_vals no callback set 2024-09-11T01:30:22.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.490+0000 7f3df1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:22.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.490+0000 7f3df1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:22.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.490+0000 7f3df1ffb640 10 monclient: dump: 2024-09-11T01:30:22.506 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:22.506 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:22.506 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.506 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.507 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:22.507 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.507 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:22.507 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:22.507 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:22.507 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.491+0000 7f3df9964640 10 monclient: shutdown 2024-09-11T01:30:22.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.491+0000 7f3df9964640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:22.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.491+0000 7f3df9964640 1 -- 172.21.15.12:0/504385548 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df4137e50 msgr2=0x7f3df409d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:22.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.491+0000 7f3df9964640 1 --2- 172.21.15.12:0/504385548 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df4137e50 0x7f3df409d160 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc009870 tx=0x7f3ddc02a0a0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.491+0000 7f3df9964640 1 -- 172.21.15.12:0/504385548 shutdown_connections 2024-09-11T01:30:22.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.491+0000 7f3df9964640 1 --2- 172.21.15.12:0/504385548 >> v2:172.21.15.71:3300/0 conn(0x7f3df413d380 0x7f3df413f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.491+0000 7f3df9964640 1 --2- 172.21.15.12:0/504385548 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df4137e50 0x7f3df409d160 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.491+0000 7f3df9964640 1 -- 172.21.15.12:0/504385548 >> 172.21.15.12:0/504385548 conn(0x7f3df40a7020 msgr2=0x7f3df40a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:22.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.491+0000 7f3df9964640 1 -- 172.21.15.12:0/504385548 shutdown_connections 2024-09-11T01:30:22.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.491+0000 7f3df9964640 1 -- 172.21.15.12:0/504385548 wait complete. 2024-09-11T01:30:22.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.491+0000 7f3df9964640 10 monclient: build_initial_monmap 2024-09-11T01:30:22.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.491+0000 7f3df9964640 10 monclient: monmap: 2024-09-11T01:30:22.508 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:22.508 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:30:22.508 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:30:22.508 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:30:22.508 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:22.508 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.508 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:30:22.509 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:30:22.509 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:30:22.509 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.492+0000 7f3df9964640 1 Processor -- start 2024-09-11T01:30:22.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.492+0000 7f3df9964640 1 -- start start 2024-09-11T01:30:22.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.492+0000 7f3df9964640 10 monclient: init 2024-09-11T01:30:22.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.492+0000 7f3df9964640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:22.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.492+0000 7f3df9964640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:30:22.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.492+0000 7f3df9964640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3df4137e50 0x7f3df41ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.492+0000 7f3df9964640 10 monclient(hunting): picked mon.noname-b con 0x7f3df4137e50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:30:22.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.492+0000 7f3df9964640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df4138230 0x7f3df4136f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df9964640 10 monclient(hunting): picked mon.noname-c con 0x7f3df4138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:22.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df9964640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3df413d380 0x7f3df40a1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df9964640 10 monclient(hunting): picked mon.noname-a con 0x7f3df413d380 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:30:22.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df9964640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:22.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df9964640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3df406f890 con 0x7f3df413d380 2024-09-11T01:30:22.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df9964640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:22.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df9964640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3df406f710 con 0x7f3df4137e50 2024-09-11T01:30:22.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df9964640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:22.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df9964640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f3df413fcb0 con 0x7f3df4138230 2024-09-11T01:30:22.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df9964640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:22.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df9964640 10 monclient(hunting): authenticate will time out at 828.935058s 2024-09-11T01:30:22.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df27fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df4138230 0x7f3df4136f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3df413d380 0x7f3df40a1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df27fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df4138230 0x7f3df4136f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:40604/0 (socket says 172.21.15.12:40604) 2024-09-11T01:30:22.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df27fc640 1 -- 172.21.15.12:0/4237901067 learned_addr learned my addr 172.21.15.12:0/4237901067 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:30:22.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df27fc640 10 monclient(hunting): get_auth_request con 0x7f3df4138230 auth_method 0 2024-09-11T01:30:22.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:22.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df27fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:22.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:22.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df37fe640 10 monclient(hunting): get_auth_request con 0x7f3df413d380 auth_method 0 2024-09-11T01:30:22.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:22.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:22.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:22.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:22.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:22.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:22.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:22.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:22.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:22.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df2ffd640 1 --2- 172.21.15.12:0/4237901067 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3df4137e50 0x7f3df41ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df2ffd640 10 monclient(hunting): get_auth_request con 0x7f3df4137e50 auth_method 0 2024-09-11T01:30:22.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:22.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:22.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:22.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df37fe640 10 monclient(hunting): handle_auth_done global_id 14109 payload 824 2024-09-11T01:30:22.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df37fe640 1 -- 172.21.15.12:0/4237901067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df4138230 msgr2=0x7f3df4136f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:22.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df37fe640 1 --2- 172.21.15.12:0/4237901067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df4138230 0x7f3df4136f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df37fe640 1 -- 172.21.15.12:0/4237901067 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3df4137e50 msgr2=0x7f3df41ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:22.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df37fe640 1 --2- 172.21.15.12:0/4237901067 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3df4137e50 0x7f3df41ccf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df37fe640 10 monclient: _finish_hunting 0 2024-09-11T01:30:22.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df37fe640 1 monclient: found mon.noname-a 2024-09-11T01:30:22.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:22.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:30:22.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df37fe640 1 -- 172.21.15.12:0/4237901067 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3df4121940 con 0x7f3df413d380 2024-09-11T01:30:22.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.493+0000 7f3df27fc640 1 --2- 172.21.15.12:0/4237901067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df4138230 0x7f3df4136f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:30:22.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.495+0000 7f3df37fe640 1 -- 172.21.15.12:0/4237901067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3df413d380 msgr2=0x7f3df40a1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:22.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.495+0000 7f3df37fe640 1 --2- 172.21.15.12:0/4237901067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3df413d380 0x7f3df40a1b90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3dec00ccb0 tx=0x7f3dec00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:22.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.495+0000 7f3debfff640 1 -- 172.21.15.12:0/4237901067 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dec003b90 con 0x7f3df413d380 2024-09-11T01:30:22.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.495+0000 7f3debfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:22.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.495+0000 7f3debfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:30:22.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.495+0000 7f3debfff640 10 monclient: dump: 2024-09-11T01:30:22.515 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:22.515 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:22.515 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.515 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.515 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:22.515 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.515 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:22.515 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:22.515 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:22.515 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.495+0000 7f3debfff640 10 monclient: _finish_auth 0 2024-09-11T01:30:22.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.495+0000 7f3debfff640 10 monclient: _check_auth_tickets 2024-09-11T01:30:22.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.495+0000 7f3debfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:22.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.495+0000 7f3debfff640 1 -- 172.21.15.12:0/4237901067 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dec003d30 con 0x7f3df413d380 2024-09-11T01:30:22.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.495+0000 7f3df9964640 5 monclient: authenticate success, global_id 14109 2024-09-11T01:30:22.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.495+0000 7f3df9964640 10 monclient: _renew_subs 2024-09-11T01:30:22.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.495+0000 7f3df9964640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:30:22.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.495+0000 7f3df9964640 1 -- 172.21.15.12:0/4237901067 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3df413f930 con 0x7f3df413d380 2024-09-11T01:30:22.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.495+0000 7f3debfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:22.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.495+0000 7f3debfff640 1 -- 172.21.15.12:0/4237901067 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dec004240 con 0x7f3df413d380 2024-09-11T01:30:22.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.495+0000 7f3debfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:22.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.495+0000 7f3debfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:30:22.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.495+0000 7f3debfff640 10 monclient: dump: 2024-09-11T01:30:22.517 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:22.517 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:22.517 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.517 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.517 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:22.518 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.518 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:22.518 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:22.518 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:22.518 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.496+0000 7f3df9964640 10 monclient: _renew_subs 2024-09-11T01:30:22.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.495+0000 7f3df17fa640 4 set_mon_vals no callback set 2024-09-11T01:30:22.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.496+0000 7f3df9964640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:30:22.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.496+0000 7f3df9964640 1 -- 172.21.15.12:0/4237901067 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3df40a4160 con 0x7f3df413d380 2024-09-11T01:30:22.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.496+0000 7f3df9964640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:22.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.496+0000 7f3df9964640 1 --2- 172.21.15.12:0/4237901067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df40a20d0 0x7f3df40a24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.496+0000 7f3df9964640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:22.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.496+0000 7f3df9964640 1 -- 172.21.15.12:0/4237901067 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f3df406a060 con 0x7f3df40a20d0 2024-09-11T01:30:22.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.496+0000 7f3df2ffd640 1 --2- 172.21.15.12:0/4237901067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df40a20d0 0x7f3df40a24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.496+0000 7f3debfff640 1 -- 172.21.15.12:0/4237901067 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f3dec004460 con 0x7f3df413d380 2024-09-11T01:30:22.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.496+0000 7f3df2ffd640 10 monclient: get_auth_request con 0x7f3df40a20d0 auth_method 0 2024-09-11T01:30:22.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.496+0000 7f3df2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:22.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.496+0000 7f3df2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:22.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.496+0000 7f3df2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:22.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.497+0000 7f3df2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:22.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.497+0000 7f3df2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:22.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.497+0000 7f3df2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:22.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.497+0000 7f3df2ffd640 10 monclient(hunting): handle_auth_done global_id 4439 payload 824 2024-09-11T01:30:22.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.497+0000 7f3df2ffd640 1 -- 172.21.15.12:0/4237901067 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3df40a29f0 con 0x7f3df40a20d0 2024-09-11T01:30:22.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.497+0000 7f3debfff640 1 --2- 172.21.15.12:0/4237901067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3dd403e290 0x7f3dd4040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.497+0000 7f3debfff640 1 -- 172.21.15.12:0/4237901067 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 4755+0+0 (secure 0 0 0) 0x7f3dec053830 con 0x7f3df413d380 2024-09-11T01:30:22.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.497+0000 7f3df27fc640 1 --2- 172.21.15.12:0/4237901067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3dd403e290 0x7f3dd4040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.497+0000 7f3df27fc640 10 monclient: get_auth_request con 0x7f3dd403e290 auth_method 0 2024-09-11T01:30:22.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.499+0000 7f3df2ffd640 1 -- 172.21.15.12:0/4237901067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df40a20d0 msgr2=0x7f3df40a24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:22.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.499+0000 7f3df2ffd640 1 --2- 172.21.15.12:0/4237901067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df40a20d0 0x7f3df40a24b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3df406fc00 tx=0x7f3ddc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:22.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.499+0000 7f3debfff640 1 -- 172.21.15.12:0/4237901067 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc037040 con 0x7f3df40a20d0 2024-09-11T01:30:22.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.499+0000 7f3debfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:22.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.499+0000 7f3debfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:22.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.499+0000 7f3debfff640 10 monclient: dump: 2024-09-11T01:30:22.520 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:22.521 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:22.521 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.521 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.521 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:22.521 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.521 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:22.521 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:22.522 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:22.522 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.499+0000 7f3debfff640 1 -- 172.21.15.12:0/4237901067 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3ddc03e660 con 0x7f3df40a20d0 2024-09-11T01:30:22.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.499+0000 7f3debfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:22.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.499+0000 7f3debfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:30:22.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.499+0000 7f3debfff640 1 -- 172.21.15.12:0/4237901067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df40a20d0 msgr2=0x7f3df40a24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:22.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.499+0000 7f3debfff640 1 --2- 172.21.15.12:0/4237901067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df40a20d0 0x7f3df40a24b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3df406fc00 tx=0x7f3ddc0046c0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.499+0000 7f3debfff640 1 -- 172.21.15.12:0/4237901067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df40a20d0 msgr2=0x7f3df40a24b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:30:22.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.499+0000 7f3debfff640 1 --2- 172.21.15.12:0/4237901067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df40a20d0 0x7f3df40a24b0 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3df406fc00 tx=0x7f3ddc0046c0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.499+0000 7f3df27fc640 1 --2- 172.21.15.12:0/4237901067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3dd403e290 0x7f3dd4040750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3df4120c20 tx=0x7f3de4007ef0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:22.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.505+0000 7f3df9964640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:30:22.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.505+0000 7f3df9964640 1 --2- 172.21.15.12:0/4237901067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df409e420 0x7f3df409e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.506+0000 7f3df9964640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:22.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.506+0000 7f3df9964640 1 -- 172.21.15.12:0/4237901067 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f3df409d160 con 0x7f3df409e420 2024-09-11T01:30:22.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.506+0000 7f3df2ffd640 1 --2- 172.21.15.12:0/4237901067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df409e420 0x7f3df409e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.506+0000 7f3df2ffd640 10 monclient: get_auth_request con 0x7f3df409e420 auth_method 0 2024-09-11T01:30:22.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.506+0000 7f3df2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:22.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.506+0000 7f3df2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:22.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.506+0000 7f3df2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:22.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.506+0000 7f3df2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:22.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.506+0000 7f3df2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:22.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.506+0000 7f3df2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:22.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.506+0000 7f3df2ffd640 10 monclient(hunting): handle_auth_done global_id 4442 payload 824 2024-09-11T01:30:22.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.506+0000 7f3df2ffd640 1 -- 172.21.15.12:0/4237901067 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3df41cdda0 con 0x7f3df409e420 2024-09-11T01:30:22.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.508+0000 7f3df2ffd640 1 -- 172.21.15.12:0/4237901067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df409e420 msgr2=0x7f3df409e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:22.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.508+0000 7f3df2ffd640 1 --2- 172.21.15.12:0/4237901067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df409e420 0x7f3df409e830 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc000c00 tx=0x7f3ddc009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:22.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.508+0000 7f3debfff640 1 -- 172.21.15.12:0/4237901067 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc037040 con 0x7f3df409e420 2024-09-11T01:30:22.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.508+0000 7f3debfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:22.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.508+0000 7f3debfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:22.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.508+0000 7f3debfff640 10 monclient: dump: 2024-09-11T01:30:22.528 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:22.528 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:22.528 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.528 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.528 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:22.528 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.529 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:22.529 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:22.529 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:22.529 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.508+0000 7f3debfff640 1 -- 172.21.15.12:0/4237901067 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4707 (secure 0 0 0) 0x7f3ddc03c930 con 0x7f3df409e420 2024-09-11T01:30:22.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.508+0000 7f3debfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:30:22.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.508+0000 7f3debfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:30:22.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.508+0000 7f3debfff640 1 -- 172.21.15.12:0/4237901067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df409e420 msgr2=0x7f3df409e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:22.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.508+0000 7f3debfff640 1 --2- 172.21.15.12:0/4237901067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df409e420 0x7f3df409e830 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc000c00 tx=0x7f3ddc009240 comp rx=0 tx=0).stop 2024-09-11T01:30:22.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.508+0000 7f3debfff640 1 -- 172.21.15.12:0/4237901067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df409e420 msgr2=0x7f3df409e830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:30:22.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.508+0000 7f3debfff640 1 --2- 172.21.15.12:0/4237901067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3df409e420 0x7f3df409e830 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc000c00 tx=0x7f3ddc009240 comp rx=0 tx=0).stop 2024-09-11T01:30:22.531 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:30:22.531 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:30:22.531 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:30:22.531 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:30:22.531 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 14, 2024-09-11T01:30:22.532 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 85197, 2024-09-11T01:30:22.532 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:30:22.532 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:30:22.532 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:30:22.532 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:30:22.532 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:22.533 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:30:22.533 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:30:22.533 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:30:22.533 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:30:22.533 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:30:22.533 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:30:22.534 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:30:22.534 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:30:22.534 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:30:22.534 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:30:22.534 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:30:22.534 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:30:22.535 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:30:22.535 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:30:22.535 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:30:22.535 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:22.535 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:30:22.536 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:30:22.536 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:30:22.536 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:30:22.536 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:30:22.536 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:30:22.536 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:30:22.537 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:30:22.537 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:30:22.537 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:30:22.537 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:30:22.537 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:30:22.537 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:30:22.538 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:22.538 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.538 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:30:22.538 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:30:22.538 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:30:22.539 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:30:22.539 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:30:22.539 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:30:22.539 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:30:22.539 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:30:22.539 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:30:22.539 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:30:22.539 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:30:22.540 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:30:22.540 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:30:22.540 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:30:22.540 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:30:22.540 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:30:22.540 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:30:22.541 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:30:22.541 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:30:22.541 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:30:22.541 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:30:22.541 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:30:22.541 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:30:22.542 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:30:22.542 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:30:22.542 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:30:22.542 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:30:22.542 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:30:22.542 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:22.542 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:30:22.542 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.542 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:30:22.542 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.542 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:30:22.542 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:30:22.542 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:30:22.543 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:30:22.543 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.543 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:30:22.543 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:30:22.543 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:22.543 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.543 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.543 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:30:22.543 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:30:22.543 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:22.543 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:22.543 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:22.543 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.544 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:30:22.544 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:30:22.544 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:30:22.544 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:30:22.544 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:30:22.544 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.544 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.544 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:30:22.544 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:30:22.545 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:30:22.545 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:30:22.545 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.545 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:30:22.545 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:30:22.545 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:22.545 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.545 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.545 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:30:22.545 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:30:22.545 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:22.545 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:22.546 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:22.546 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.546 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:30:22.546 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:30:22.546 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:30:22.546 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:30:22.546 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:30:22.546 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.546 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.546 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:30:22.546 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:30:22.546 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:30:22.546 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:30:22.546 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.546 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:30:22.547 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:30:22.547 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:22.547 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.547 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.547 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:30:22.547 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:30:22.547 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:22.547 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:22.547 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:22.547 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.547 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:30:22.547 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:30:22.547 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:30:22.547 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:30:22.548 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:30:22.548 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:22.548 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:22.548 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.548 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:30:22.548 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:30:22.548 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.548 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:30:22.548 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:30:22.548 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:30:22.548 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:22.548 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:22.548 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:30:22.549 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.549 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:30:22.549 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:30:22.549 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:30:22.549 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:22.549 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:22.549 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:30:22.549 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.549 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:30:22.549 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:30:22.549 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:30:22.549 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:22.549 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:22.549 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.549 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:30:22.550 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:30:22.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.509+0000 7f3df9964640 1 -- 172.21.15.12:0/4237901067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3dd403e290 msgr2=0x7f3dd4040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:22.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.509+0000 7f3df9964640 1 --2- 172.21.15.12:0/4237901067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3dd403e290 0x7f3dd4040750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3df4120c20 tx=0x7f3de4007ef0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.509+0000 7f3df9964640 10 monclient: shutdown 2024-09-11T01:30:22.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.509+0000 7f3df9964640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:22.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.509+0000 7f3df9964640 1 -- 172.21.15.12:0/4237901067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3df413d380 msgr2=0x7f3df40a1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:22.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.509+0000 7f3df9964640 1 --2- 172.21.15.12:0/4237901067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3df413d380 0x7f3df40a1b90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3dec00ccb0 tx=0x7f3dec00a660 comp rx=0 tx=0).stop 2024-09-11T01:30:22.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.509+0000 7f3df37fe640 1 -- 172.21.15.12:0/4237901067 reap_dead start 2024-09-11T01:30:22.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.509+0000 7f3df37fe640 1 -- 172.21.15.12:0/4237901067 reap_dead start 2024-09-11T01:30:22.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.509+0000 7f3df9964640 1 -- 172.21.15.12:0/4237901067 shutdown_connections 2024-09-11T01:30:22.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.509+0000 7f3df9964640 1 -- 172.21.15.12:0/4237901067 >> 172.21.15.12:0/4237901067 conn(0x7f3df40a7020 msgr2=0x7f3df413c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:22.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.509+0000 7f3df9964640 1 -- 172.21.15.12:0/4237901067 shutdown_connections 2024-09-11T01:30:22.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.509+0000 7f3df9964640 1 -- 172.21.15.12:0/4237901067 wait complete. 2024-09-11T01:30:22.553 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:30:22.553 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:30:22.553 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:30:22.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.653+0000 7fd28ed4b640 10 monclient: get_monmap_and_config 2024-09-11T01:30:22.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.653+0000 7fd28ed4b640 10 monclient: build_initial_monmap 2024-09-11T01:30:22.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.653+0000 7fd28ed4b640 10 monclient: monmap: 2024-09-11T01:30:22.655 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:22.655 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:22.656 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:30:22.654413+0000 2024-09-11T01:30:22.656 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:30:22.654413+0000 2024-09-11T01:30:22.656 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:22.656 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.656 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:30:22.656 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:30:22.656 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:30:22.656 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:30:22.656 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:30:22.656 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.654+0000 7fd28ed4b640 1 Processor -- start 2024-09-11T01:30:22.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.654+0000 7fd28ed4b640 1 -- start start 2024-09-11T01:30:22.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.654+0000 7fd28ed4b640 10 monclient: init 2024-09-11T01:30:22.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.654+0000 7fd28ed4b640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:22.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.654+0000 7fd28ed4b640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-11T01:30:22.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.654+0000 7fd28ed4b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd288137d50 addr v1:172.21.15.12:6789/0 2024-09-11T01:30:22.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.654+0000 7fd28ed4b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd288138130 0x7fd288138550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.654+0000 7fd28ed4b640 10 monclient(hunting): picked mon.noname-b con 0x7fd288138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:22.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.654+0000 7fd28ed4b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd28813d280 0x7fd28813f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.654+0000 7fd28ed4b640 10 monclient(hunting): picked mon.noname-a con 0x7fd28813d280 addr v2:172.21.15.12:3300/0 2024-09-11T01:30:22.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.654+0000 7fd28ed4b640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd28809d820 con 0x7fd288137d50 2024-09-11T01:30:22.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.654+0000 7fd28ed4b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:22.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.654+0000 7fd28ed4b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fd28809c650 con 0x7fd28813d280 2024-09-11T01:30:22.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.654+0000 7fd28ed4b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:22.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.654+0000 7fd28ed4b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd28809ba20 con 0x7fd288138130 2024-09-11T01:30:22.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.654+0000 7fd28ed4b640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:22.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.654+0000 7fd28ed4b640 10 monclient(hunting): authenticate will time out at 829.096978s 2024-09-11T01:30:22.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd28d2e8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd28813d280 0x7fd28813f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd28cae7640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd288137d50 0x7fd28809d110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44006/0 (socket says 172.21.15.12:44006) 2024-09-11T01:30:22.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd287fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd288138130 0x7fd288138550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd28cae7640 1 -- 172.21.15.12:0/1825251895 learned_addr learned my addr 172.21.15.12:0/1825251895 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:30:22.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd28d2e8640 10 monclient(hunting): get_auth_request con 0x7fd28813d280 auth_method 0 2024-09-11T01:30:22.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd28d2e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:22.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd28d2e8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:22.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd28d2e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:22.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd287fff640 10 monclient(hunting): get_auth_request con 0x7fd288138130 auth_method 0 2024-09-11T01:30:22.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd287fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:22.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd287fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:22.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd287fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:22.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd28d2e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:22.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd28d2e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:22.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd28d2e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:22.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd287fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:22.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd287fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:22.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd287fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:22.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd28d2e8640 10 monclient(hunting): handle_auth_done global_id 14112 payload 274 2024-09-11T01:30:22.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd28d2e8640 1 -- 172.21.15.12:0/1825251895 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd288138130 msgr2=0x7fd288138550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:30:22.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd28d2e8640 1 --2- 172.21.15.12:0/1825251895 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd288138130 0x7fd288138550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd28d2e8640 1 -- 172.21.15.12:0/1825251895 >> v1:172.21.15.12:6789/0 conn(0x7fd288137d50 legacy=0x7fd28809d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:30:22.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd28d2e8640 10 monclient: _finish_hunting 0 2024-09-11T01:30:22.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd28d2e8640 1 monclient: found mon.noname-a 2024-09-11T01:30:22.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd28d2e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:22.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd28d2e8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:30:22.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd28d2e8640 1 -- 172.21.15.12:0/1825251895 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd28809c370 con 0x7fd28813d280 2024-09-11T01:30:22.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.655+0000 7fd287fff640 1 --2- 172.21.15.12:0/1825251895 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd288138130 0x7fd288138550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:30:22.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd28d2e8640 1 -- 172.21.15.12:0/1825251895 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd28813d280 msgr2=0x7fd28813f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:22.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd28d2e8640 1 --2- 172.21.15.12:0/1825251895 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd28813d280 0x7fd28813f670 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd280009d90 tx=0x7fd28002a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a6401d26dd7ed21 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:22.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd2877fe640 1 -- 172.21.15.12:0/1825251895 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd280007950 con 0x7fd28813d280 2024-09-11T01:30:22.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd2877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:22.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd2877fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:30:22.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd2877fe640 10 monclient: dump: 2024-09-11T01:30:22.665 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:22.665 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:22.665 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.665 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.665 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:22.665 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.665 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:22.665 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:22.665 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:22.665 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd2877fe640 10 monclient: _finish_auth 0 2024-09-11T01:30:22.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd2877fe640 10 monclient: _check_auth_tickets 2024-09-11T01:30:22.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd2877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:22.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd2877fe640 1 -- 172.21.15.12:0/1825251895 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd280007af0 con 0x7fd28813d280 2024-09-11T01:30:22.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd28ed4b640 5 monclient: authenticate success, global_id 14112 2024-09-11T01:30:22.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd28ed4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:30:22.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd2877fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:22.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd28ed4b640 10 monclient: get_monmap_and_config success 2024-09-11T01:30:22.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd2877fe640 1 -- 172.21.15.12:0/1825251895 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd280031400 con 0x7fd28813d280 2024-09-11T01:30:22.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd28ed4b640 4 set_mon_vals no callback set 2024-09-11T01:30:22.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd2877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:22.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd2877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:30:22.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd2877fe640 10 monclient: dump: 2024-09-11T01:30:22.667 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:22.667 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:22.667 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.667 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.667 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:22.667 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.667 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:22.667 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:22.667 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:22.667 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd28ed4b640 10 monclient: shutdown 2024-09-11T01:30:22.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd28ed4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:22.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd28ed4b640 1 -- 172.21.15.12:0/1825251895 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd28813d280 msgr2=0x7fd28813f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:22.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd28ed4b640 1 --2- 172.21.15.12:0/1825251895 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd28813d280 0x7fd28813f670 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd280009d90 tx=0x7fd28002a530 comp rx=0 tx=0).stop 2024-09-11T01:30:22.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd28ed4b640 1 -- 172.21.15.12:0/1825251895 shutdown_connections 2024-09-11T01:30:22.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd28ed4b640 1 --2- 172.21.15.12:0/1825251895 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd28813d280 0x7fd28813f670 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd28ed4b640 1 --2- 172.21.15.12:0/1825251895 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd288138130 0x7fd288138550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd28ed4b640 1 -- 172.21.15.12:0/1825251895 >> 172.21.15.12:0/1825251895 conn(0x7fd2880a7020 msgr2=0x7fd2880a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:22.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.658+0000 7fd28ed4b640 1 -- 172.21.15.12:0/1825251895 shutdown_connections 2024-09-11T01:30:22.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.659+0000 7fd28ed4b640 1 -- 172.21.15.12:0/1825251895 wait complete. 2024-09-11T01:30:22.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.659+0000 7fd28ed4b640 10 monclient: build_initial_monmap 2024-09-11T01:30:22.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.659+0000 7fd28ed4b640 10 monclient: monmap: 2024-09-11T01:30:22.674 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:22.674 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:30:22.674 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:30:22.674 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:30:22.674 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:22.674 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.675 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:30:22.675 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:30:22.675 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:30:22.675 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.659+0000 7fd28ed4b640 1 Processor -- start 2024-09-11T01:30:22.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.659+0000 7fd28ed4b640 1 -- start start 2024-09-11T01:30:22.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.659+0000 7fd28ed4b640 10 monclient: init 2024-09-11T01:30:22.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28ed4b640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:22.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28ed4b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:30:22.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28ed4b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd288137d50 0x7fd2881ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28ed4b640 10 monclient(hunting): picked mon.noname-b con 0x7fd288137d50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:30:22.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28ed4b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd288138130 0x7fd288136ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28ed4b640 10 monclient(hunting): picked mon.noname-c con 0x7fd288138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:22.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28ed4b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd28813d280 0x7fd28813a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28ed4b640 10 monclient(hunting): picked mon.noname-a con 0x7fd28813d280 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:30:22.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28ed4b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:22.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28ed4b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd28806f850 con 0x7fd28813d280 2024-09-11T01:30:22.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28ed4b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:22.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28ed4b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd28806f6d0 con 0x7fd288137d50 2024-09-11T01:30:22.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28ed4b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:22.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28ed4b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd28813fbb0 con 0x7fd288138130 2024-09-11T01:30:22.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28ed4b640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:22.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28ed4b640 10 monclient(hunting): authenticate will time out at 829.102494s 2024-09-11T01:30:22.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd287fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd288138130 0x7fd288136ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28d2e8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd28813d280 0x7fd28813a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd287fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd288138130 0x7fd288136ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:40648/0 (socket says 172.21.15.12:40648) 2024-09-11T01:30:22.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd287fff640 1 -- 172.21.15.12:0/4073586252 learned_addr learned my addr 172.21.15.12:0/4073586252 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:30:22.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd287fff640 10 monclient(hunting): get_auth_request con 0x7fd288138130 auth_method 0 2024-09-11T01:30:22.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd287fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:22.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd287fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:22.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd287fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:22.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28d2e8640 10 monclient(hunting): get_auth_request con 0x7fd28813d280 auth_method 0 2024-09-11T01:30:22.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28d2e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:22.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28d2e8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:22.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28d2e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:22.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd287fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:22.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd287fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:22.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd287fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:22.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28d2e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:22.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28d2e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:22.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28d2e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:22.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28cae7640 1 --2- 172.21.15.12:0/4073586252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd288137d50 0x7fd2881ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28cae7640 10 monclient(hunting): get_auth_request con 0x7fd288137d50 auth_method 0 2024-09-11T01:30:22.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28cae7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:22.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28cae7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:22.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28cae7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:22.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd287fff640 10 monclient(hunting): handle_auth_done global_id 4448 payload 824 2024-09-11T01:30:22.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd287fff640 1 -- 172.21.15.12:0/4073586252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd288137d50 msgr2=0x7fd2881ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:22.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd287fff640 1 --2- 172.21.15.12:0/4073586252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd288137d50 0x7fd2881ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd287fff640 1 -- 172.21.15.12:0/4073586252 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd28813d280 msgr2=0x7fd28813a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:22.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd287fff640 1 --2- 172.21.15.12:0/4073586252 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd28813d280 0x7fd28813a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd287fff640 10 monclient: _finish_hunting 0 2024-09-11T01:30:22.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd287fff640 1 monclient: found mon.noname-c 2024-09-11T01:30:22.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd287fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:22.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd287fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:30:22.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd287fff640 1 -- 172.21.15.12:0/4073586252 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd28813c760 con 0x7fd288138130 2024-09-11T01:30:22.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.660+0000 7fd28d2e8640 1 --2- 172.21.15.12:0/4073586252 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd28813d280 0x7fd28813a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:30:22.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.663+0000 7fd287fff640 1 -- 172.21.15.12:0/4073586252 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd288138130 msgr2=0x7fd288136ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:22.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.663+0000 7fd287fff640 1 --2- 172.21.15.12:0/4073586252 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd288138130 0x7fd288136ed0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd27400ec10 tx=0x7fd27400c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:22.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.663+0000 7fd2857fa640 1 -- 172.21.15.12:0/4073586252 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd274017070 con 0x7fd288138130 2024-09-11T01:30:22.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.663+0000 7fd2857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:22.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.663+0000 7fd2857fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:30:22.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.663+0000 7fd2857fa640 10 monclient: dump: 2024-09-11T01:30:22.679 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:22.680 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:22.680 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.680 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.680 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:22.680 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.680 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:22.680 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:22.680 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:22.680 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.663+0000 7fd2857fa640 10 monclient: _finish_auth 0 2024-09-11T01:30:22.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.663+0000 7fd2857fa640 10 monclient: _check_auth_tickets 2024-09-11T01:30:22.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.663+0000 7fd2857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:22.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.663+0000 7fd2857fa640 1 -- 172.21.15.12:0/4073586252 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd27400ce40 con 0x7fd288138130 2024-09-11T01:30:22.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.663+0000 7fd28ed4b640 5 monclient: authenticate success, global_id 4448 2024-09-11T01:30:22.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.663+0000 7fd28ed4b640 10 monclient: _renew_subs 2024-09-11T01:30:22.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.663+0000 7fd28ed4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:22.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.663+0000 7fd28ed4b640 1 -- 172.21.15.12:0/4073586252 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd288120fc0 con 0x7fd288138130 2024-09-11T01:30:22.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.663+0000 7fd2857fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:22.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.663+0000 7fd2857fa640 1 -- 172.21.15.12:0/4073586252 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd274013690 con 0x7fd288138130 2024-09-11T01:30:22.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.663+0000 7fd2857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:22.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.663+0000 7fd2857fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:22.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.663+0000 7fd2857fa640 10 monclient: dump: 2024-09-11T01:30:22.681 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:22.681 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:22.681 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.682 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.682 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:22.682 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.682 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:22.682 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:22.682 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:22.682 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.663+0000 7fd28ed4b640 10 monclient: _renew_subs 2024-09-11T01:30:22.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.663+0000 7fd286ffd640 4 set_mon_vals no callback set 2024-09-11T01:30:22.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.663+0000 7fd28ed4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:22.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.663+0000 7fd28ed4b640 1 -- 172.21.15.12:0/4073586252 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd28813bc00 con 0x7fd288138130 2024-09-11T01:30:22.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.664+0000 7fd28ed4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:22.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.664+0000 7fd28ed4b640 1 --2- 172.21.15.12:0/4073586252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd288137410 0x7fd28813a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.664+0000 7fd28ed4b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:22.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.664+0000 7fd28ed4b640 1 -- 172.21.15.12:0/4073586252 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd288121410 con 0x7fd288137410 2024-09-11T01:30:22.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.664+0000 7fd2857fa640 1 -- 172.21.15.12:0/4073586252 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fd274024060 con 0x7fd288138130 2024-09-11T01:30:22.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.664+0000 7fd2857fa640 1 --2- 172.21.15.12:0/4073586252 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd26c03e210 0x7fd26c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.664+0000 7fd28cae7640 1 --2- 172.21.15.12:0/4073586252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd288137410 0x7fd28813a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.664+0000 7fd28cae7640 10 monclient: get_auth_request con 0x7fd288137410 auth_method 0 2024-09-11T01:30:22.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.664+0000 7fd28cae7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:22.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.664+0000 7fd28cae7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:22.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.664+0000 7fd28cae7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:22.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.664+0000 7fd2857fa640 1 -- 172.21.15.12:0/4073586252 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 4755+0+0 (secure 0 0 0) 0x7fd274052d20 con 0x7fd288138130 2024-09-11T01:30:22.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.664+0000 7fd28cae7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:22.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.664+0000 7fd28cae7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:22.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.664+0000 7fd28cae7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:22.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.664+0000 7fd28d2e8640 1 --2- 172.21.15.12:0/4073586252 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd26c03e210 0x7fd26c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.664+0000 7fd28d2e8640 10 monclient: get_auth_request con 0x7fd26c03e210 auth_method 0 2024-09-11T01:30:22.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.664+0000 7fd28cae7640 10 monclient(hunting): handle_auth_done global_id 4372 payload 824 2024-09-11T01:30:22.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.664+0000 7fd28cae7640 1 -- 172.21.15.12:0/4073586252 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd288138550 con 0x7fd288137410 2024-09-11T01:30:22.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.667+0000 7fd28cae7640 1 -- 172.21.15.12:0/4073586252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd288137410 msgr2=0x7fd28813a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:22.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.667+0000 7fd28cae7640 1 --2- 172.21.15.12:0/4073586252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd288137410 0x7fd28813a6c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd28813d1d0 tx=0x7fd27c00e960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:22.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.667+0000 7fd28d2e8640 1 --2- 172.21.15.12:0/4073586252 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd26c03e210 0x7fd26c0406d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd28806d4f0 tx=0x7fd28002f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:22.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.667+0000 7fd2857fa640 1 -- 172.21.15.12:0/4073586252 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27c005cf0 con 0x7fd288137410 2024-09-11T01:30:22.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.667+0000 7fd2857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:22.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.667+0000 7fd2857fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:30:22.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.667+0000 7fd2857fa640 10 monclient: dump: 2024-09-11T01:30:22.684 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:22.684 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:22.684 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.684 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.684 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:22.684 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.685 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:22.685 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:22.685 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:22.685 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.667+0000 7fd2857fa640 1 -- 172.21.15.12:0/4073586252 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd27c018070 con 0x7fd288137410 2024-09-11T01:30:22.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.667+0000 7fd2857fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:22.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.667+0000 7fd2857fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:30:22.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.667+0000 7fd2857fa640 1 -- 172.21.15.12:0/4073586252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd288137410 msgr2=0x7fd28813a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:22.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.667+0000 7fd2857fa640 1 --2- 172.21.15.12:0/4073586252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd288137410 0x7fd28813a6c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd28813d1d0 tx=0x7fd27c00e960 comp rx=0 tx=0).stop 2024-09-11T01:30:22.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.667+0000 7fd2857fa640 1 -- 172.21.15.12:0/4073586252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd288137410 msgr2=0x7fd28813a6c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:30:22.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.667+0000 7fd2857fa640 1 --2- 172.21.15.12:0/4073586252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd288137410 0x7fd28813a6c0 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd28813d1d0 tx=0x7fd27c00e960 comp rx=0 tx=0).stop 2024-09-11T01:30:22.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.673+0000 7fd28ed4b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:30:22.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.673+0000 7fd28ed4b640 1 --2- 172.21.15.12:0/4073586252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd28813ac00 0x7fd28809e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.674+0000 7fd28ed4b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:22.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.674+0000 7fd28ed4b640 1 -- 172.21.15.12:0/4073586252 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd28813afe0 con 0x7fd28813ac00 2024-09-11T01:30:22.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.674+0000 7fd28cae7640 1 --2- 172.21.15.12:0/4073586252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd28813ac00 0x7fd28809e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.674+0000 7fd28cae7640 10 monclient: get_auth_request con 0x7fd28813ac00 auth_method 0 2024-09-11T01:30:22.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.674+0000 7fd28cae7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:22.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.674+0000 7fd28cae7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.674+0000 7fd28cae7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.674+0000 7fd28cae7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.674+0000 7fd28cae7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.674+0000 7fd28cae7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.674+0000 7fd28cae7640 10 monclient(hunting): handle_auth_done global_id 4375 payload 824 2024-09-11T01:30:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.674+0000 7fd28cae7640 1 -- 172.21.15.12:0/4073586252 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd28806a250 con 0x7fd28813ac00 2024-09-11T01:30:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.676+0000 7fd28cae7640 1 -- 172.21.15.12:0/4073586252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd28813ac00 msgr2=0x7fd28809e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.676+0000 7fd28cae7640 1 --2- 172.21.15.12:0/4073586252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd28813ac00 0x7fd28809e040 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd27c017000 tx=0x7fd27c004940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.677+0000 7fd2857fa640 1 -- 172.21.15.12:0/4073586252 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27c0074e0 con 0x7fd28813ac00 2024-09-11T01:30:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.677+0000 7fd2857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.677+0000 7fd2857fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:30:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.677+0000 7fd2857fa640 10 monclient: dump: 2024-09-11T01:30:22.688 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:22.689 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:22.689 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.689 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.689 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:22.689 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.689 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:22.689 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:22.689 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:22.689 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.677+0000 7fd2857fa640 1 -- 172.21.15.12:0/4073586252 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4874 (secure 0 0 0) 0x7fd27c0074e0 con 0x7fd28813ac00 2024-09-11T01:30:22.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.677+0000 7fd2857fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:30:22.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.677+0000 7fd2857fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:30:22.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.677+0000 7fd2857fa640 1 -- 172.21.15.12:0/4073586252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd28813ac00 msgr2=0x7fd28809e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:22.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.677+0000 7fd2857fa640 1 --2- 172.21.15.12:0/4073586252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd28813ac00 0x7fd28809e040 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd27c017000 tx=0x7fd27c004940 comp rx=0 tx=0).stop 2024-09-11T01:30:22.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.677+0000 7fd2857fa640 1 -- 172.21.15.12:0/4073586252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd28813ac00 msgr2=0x7fd28809e040 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:30:22.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.677+0000 7fd2857fa640 1 --2- 172.21.15.12:0/4073586252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd28813ac00 0x7fd28809e040 secure :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd27c017000 tx=0x7fd27c004940 comp rx=0 tx=0).stop 2024-09-11T01:30:22.690 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:30:22.690 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:30:22.690 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:30:22.691 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:30:22.691 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 14, 2024-09-11T01:30:22.691 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 85330, 2024-09-11T01:30:22.691 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:30:22.691 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:30:22.691 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:30:22.691 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:30:22.691 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:22.691 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:30:22.691 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:30:22.691 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:30:22.691 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:30:22.691 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:30:22.692 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:30:22.692 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:30:22.692 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:30:22.692 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:30:22.692 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:30:22.692 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:30:22.692 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:30:22.692 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:30:22.692 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:30:22.692 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:30:22.692 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:22.692 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:30:22.692 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:30:22.692 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:30:22.693 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:30:22.693 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:30:22.693 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:30:22.693 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:30:22.693 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:30:22.693 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:30:22.693 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:30:22.693 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:30:22.693 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:30:22.693 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:30:22.693 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:22.694 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.694 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:30:22.694 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:30:22.694 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:30:22.694 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:30:22.694 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:30:22.694 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:30:22.694 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:30:22.694 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:30:22.694 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:30:22.694 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:30:22.694 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:30:22.694 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:30:22.695 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:30:22.695 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:30:22.695 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:30:22.695 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:30:22.695 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:30:22.695 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:30:22.695 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:30:22.695 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:30:22.695 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:30:22.695 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:30:22.695 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:30:22.695 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:30:22.695 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:30:22.695 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:30:22.696 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:30:22.696 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:30:22.696 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:22.696 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:30:22.696 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.696 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:30:22.696 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.696 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:30:22.696 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:30:22.696 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:30:22.696 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:30:22.696 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.696 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:30:22.696 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:30:22.697 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:22.697 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.697 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.697 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:30:22.697 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:30:22.697 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:22.697 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:22.697 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:22.697 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.697 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:30:22.697 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:30:22.698 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:30:22.698 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:30:22.698 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:30:22.698 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.698 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.698 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:30:22.698 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:30:22.698 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:30:22.698 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:30:22.698 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.698 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:30:22.698 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:30:22.699 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:22.699 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.699 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.699 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:30:22.699 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:30:22.699 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:22.699 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:22.699 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:22.699 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.699 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:30:22.699 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:30:22.699 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:30:22.699 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:30:22.699 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:30:22.699 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.700 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.700 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:30:22.700 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:30:22.700 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:30:22.700 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:30:22.700 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.700 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:30:22.700 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:30:22.700 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:22.700 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.700 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.700 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:30:22.700 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:30:22.701 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:22.701 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:22.701 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:22.701 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.701 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:30:22.701 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:30:22.701 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:30:22.701 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:30:22.701 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:30:22.701 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:22.701 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:22.701 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.701 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:30:22.701 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:30:22.702 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.702 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:30:22.702 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:30:22.702 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:30:22.702 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:22.702 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:22.702 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:30:22.702 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.702 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:30:22.702 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:30:22.702 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:30:22.702 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:22.702 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:22.702 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:30:22.703 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.703 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:30:22.703 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:30:22.703 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:30:22.703 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:22.703 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:22.703 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:30:22.703 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:22.703 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:30:22.703 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:30:22.703 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:30:22.703 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:22.703 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:22.703 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:22.703 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:30:22.704 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:30:22.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.678+0000 7fd28ed4b640 1 -- 172.21.15.12:0/4073586252 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd26c03e210 msgr2=0x7fd26c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:22.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.678+0000 7fd28ed4b640 1 --2- 172.21.15.12:0/4073586252 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd26c03e210 0x7fd26c0406d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd28806d4f0 tx=0x7fd28002f000 comp rx=0 tx=0).stop 2024-09-11T01:30:22.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.678+0000 7fd28ed4b640 10 monclient: shutdown 2024-09-11T01:30:22.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.678+0000 7fd28ed4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:22.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.678+0000 7fd28ed4b640 1 -- 172.21.15.12:0/4073586252 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd288138130 msgr2=0x7fd288136ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:22.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.678+0000 7fd28ed4b640 1 --2- 172.21.15.12:0/4073586252 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd288138130 0x7fd288136ed0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd27400ec10 tx=0x7fd27400c520 comp rx=0 tx=0).stop 2024-09-11T01:30:22.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.678+0000 7fd28d2e8640 1 -- 172.21.15.12:0/4073586252 reap_dead start 2024-09-11T01:30:22.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.678+0000 7fd28d2e8640 1 -- 172.21.15.12:0/4073586252 reap_dead start 2024-09-11T01:30:22.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.678+0000 7fd28ed4b640 1 -- 172.21.15.12:0/4073586252 shutdown_connections 2024-09-11T01:30:22.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.678+0000 7fd28ed4b640 1 -- 172.21.15.12:0/4073586252 >> 172.21.15.12:0/4073586252 conn(0x7fd2880a7020 msgr2=0x7fd28813dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:22.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.678+0000 7fd28ed4b640 1 -- 172.21.15.12:0/4073586252 shutdown_connections 2024-09-11T01:30:22.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.678+0000 7fd28ed4b640 1 -- 172.21.15.12:0/4073586252 wait complete. 2024-09-11T01:30:22.707 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:30:22.707 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:30:22.708 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:30:22.741 INFO:tasks.rados.rados.0.smithi071.stdout:115: done (3 left) 2024-09-11T01:30:22.742 INFO:tasks.rados.rados.0.smithi071.stdout:116: done (2 left) 2024-09-11T01:30:22.742 INFO:tasks.rados.rados.0.smithi071.stdout:117: done (1 left) 2024-09-11T01:30:22.742 INFO:tasks.rados.rados.0.smithi071.stdout:118: done (0 left) 2024-09-11T01:30:22.742 INFO:tasks.rados.rados.0.smithi071.stdout:119: read oid 18 snap -1 2024-09-11T01:30:22.742 INFO:tasks.rados.rados.0.smithi071.stdout:119: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-11T01:30:22.742 INFO:tasks.rados.rados.0.smithi071.stdout:120: copy_from oid 31 from oid 6 current snap is 11 2024-09-11T01:30:22.743 INFO:tasks.rados.rados.0.smithi071.stdout:121: write oid 14 current snap is 11 2024-09-11T01:30:22.743 INFO:tasks.rados.rados.0.smithi071.stdout:121: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2024-09-11T01:30:22.749 INFO:tasks.rados.rados.0.smithi071.stdout:121: writing smithi07134777-14 from 709262 to 1406520 tid 1 2024-09-11T01:30:22.755 INFO:tasks.rados.rados.0.smithi071.stdout:121: writing smithi07134777-14 from 2012635 to 2584653 tid 2 2024-09-11T01:30:22.758 INFO:tasks.rados.rados.0.smithi071.stdout:121: writing smithi07134777-14 from 3046388 to 3109262 tid 3 2024-09-11T01:30:22.758 INFO:tasks.rados.rados.0.smithi071.stdout:122: snap_remove snap 7 2024-09-11T01:30:22.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.813+0000 7fe10393d640 10 monclient: get_monmap_and_config 2024-09-11T01:30:22.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.813+0000 7fe10393d640 10 monclient: build_initial_monmap 2024-09-11T01:30:22.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.813+0000 7fe10393d640 10 monclient: monmap: 2024-09-11T01:30:22.815 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:22.815 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:22.815 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:30:22.814082+0000 2024-09-11T01:30:22.815 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:30:22.814082+0000 2024-09-11T01:30:22.815 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:22.815 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.815 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:30:22.815 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:30:22.815 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:30:22.815 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:30:22.816 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:30:22.816 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.813+0000 7fe10393d640 1 Processor -- start 2024-09-11T01:30:22.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe10393d640 1 -- start start 2024-09-11T01:30:22.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe10393d640 10 monclient: init 2024-09-11T01:30:22.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe10393d640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:22.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe10393d640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-11T01:30:22.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe10393d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe0fc0a2530 0x7fe0fc0a03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe10393d640 10 monclient(hunting): picked mon.noname-b con 0x7fe0fc0a2530 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:22.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe10393d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe0fc09f1d0 addr v1:172.21.15.12:6789/0 2024-09-11T01:30:22.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe10393d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe0fc139b90 0x7fe0fc13bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe10393d640 10 monclient(hunting): picked mon.noname-a con 0x7fe0fc139b90 addr v2:172.21.15.12:3300/0 2024-09-11T01:30:22.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe10393d640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0fc09e510 con 0x7fe0fc09f1d0 2024-09-11T01:30:22.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe10393d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:22.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe10393d640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fe0fc09d140 con 0x7fe0fc139b90 2024-09-11T01:30:22.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe10393d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:22.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe10393d640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe0fc0a29b0 con 0x7fe0fc0a2530 2024-09-11T01:30:22.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe10393d640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:22.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe10393d640 10 monclient(hunting): authenticate will time out at 829.256700s 2024-09-11T01:30:22.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe101eda640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe0fc139b90 0x7fe0fc13bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe1016d9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe0fc0a2530 0x7fe0fc0a03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe100ed8640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe0fc09f1d0 0x7fe0fc09ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44010/0 (socket says 172.21.15.12:44010) 2024-09-11T01:30:22.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe100ed8640 1 -- 172.21.15.12:0/4207805385 learned_addr learned my addr 172.21.15.12:0/4207805385 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:30:22.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe101eda640 10 monclient(hunting): get_auth_request con 0x7fe0fc139b90 auth_method 0 2024-09-11T01:30:22.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe101eda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:22.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe101eda640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:22.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe101eda640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:22.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe1016d9640 10 monclient(hunting): get_auth_request con 0x7fe0fc0a2530 auth_method 0 2024-09-11T01:30:22.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe1016d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:22.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe1016d9640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:22.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe1016d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:22.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe101eda640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:22.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe101eda640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:22.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe101eda640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:22.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe1016d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:22.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.814+0000 7fe1016d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:22.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.815+0000 7fe1016d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:22.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.815+0000 7fe101eda640 10 monclient(hunting): handle_auth_done global_id 14118 payload 274 2024-09-11T01:30:22.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.815+0000 7fe101eda640 1 -- 172.21.15.12:0/4207805385 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe0fc0a2530 msgr2=0x7fe0fc0a03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:30:22.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.815+0000 7fe101eda640 1 --2- 172.21.15.12:0/4207805385 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe0fc0a2530 0x7fe0fc0a03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.815+0000 7fe101eda640 1 -- 172.21.15.12:0/4207805385 >> v1:172.21.15.12:6789/0 conn(0x7fe0fc09f1d0 legacy=0x7fe0fc09ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:30:22.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.815+0000 7fe101eda640 10 monclient: _finish_hunting 0 2024-09-11T01:30:22.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.815+0000 7fe101eda640 1 monclient: found mon.noname-a 2024-09-11T01:30:22.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.815+0000 7fe101eda640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:22.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.815+0000 7fe101eda640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:30:22.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.815+0000 7fe101eda640 1 -- 172.21.15.12:0/4207805385 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0fc09edb0 con 0x7fe0fc139b90 2024-09-11T01:30:22.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.815+0000 7fe1016d9640 1 --2- 172.21.15.12:0/4207805385 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe0fc0a2530 0x7fe0fc0a03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:30:22.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.817+0000 7fe101eda640 1 -- 172.21.15.12:0/4207805385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0fc139b90 msgr2=0x7fe0fc13bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:22.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.817+0000 7fe101eda640 1 --2- 172.21.15.12:0/4207805385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0fc139b90 0x7fe0fc13bf80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec0097a0 tx=0x7fe0ec029fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=228fcd8213bb9454 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:22.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.818+0000 7fe0fbfff640 1 -- 172.21.15.12:0/4207805385 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec037070 con 0x7fe0fc139b90 2024-09-11T01:30:22.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.818+0000 7fe0fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:22.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.818+0000 7fe0fbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:30:22.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.818+0000 7fe0fbfff640 10 monclient: dump: 2024-09-11T01:30:22.827 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:22.827 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:22.828 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.828 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.828 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:22.828 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.828 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:22.829 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:22.829 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:22.829 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.818+0000 7fe0fbfff640 10 monclient: _finish_auth 0 2024-09-11T01:30:22.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.818+0000 7fe0fbfff640 10 monclient: _check_auth_tickets 2024-09-11T01:30:22.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.818+0000 7fe0fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:22.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.818+0000 7fe10393d640 5 monclient: authenticate success, global_id 14118 2024-09-11T01:30:22.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.818+0000 7fe0fbfff640 1 -- 172.21.15.12:0/4207805385 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0ec032440 con 0x7fe0fc139b90 2024-09-11T01:30:22.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.818+0000 7fe0fbfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:22.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.818+0000 7fe10393d640 10 monclient: get_monmap_and_config success 2024-09-11T01:30:22.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.818+0000 7fe10393d640 4 set_mon_vals no callback set 2024-09-11T01:30:22.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.818+0000 7fe0fbfff640 1 -- 172.21.15.12:0/4207805385 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec032810 con 0x7fe0fc139b90 2024-09-11T01:30:22.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.818+0000 7fe0fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:22.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.818+0000 7fe0fbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:30:22.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.818+0000 7fe0fbfff640 10 monclient: dump: 2024-09-11T01:30:22.831 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:22.831 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:22.831 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.831 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.831 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:22.831 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.831 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:22.831 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:22.832 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:22.832 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.818+0000 7fe10393d640 10 monclient: shutdown 2024-09-11T01:30:22.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.818+0000 7fe10393d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:22.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.818+0000 7fe10393d640 1 -- 172.21.15.12:0/4207805385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0fc139b90 msgr2=0x7fe0fc13bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:22.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.818+0000 7fe10393d640 1 --2- 172.21.15.12:0/4207805385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0fc139b90 0x7fe0fc13bf80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec0097a0 tx=0x7fe0ec029fb0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.818+0000 7fe10393d640 1 -- 172.21.15.12:0/4207805385 shutdown_connections 2024-09-11T01:30:22.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.818+0000 7fe10393d640 1 --2- 172.21.15.12:0/4207805385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0fc139b90 0x7fe0fc13bf80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.818+0000 7fe10393d640 1 --2- 172.21.15.12:0/4207805385 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe0fc0a2530 0x7fe0fc0a03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.818+0000 7fe10393d640 1 -- 172.21.15.12:0/4207805385 >> 172.21.15.12:0/4207805385 conn(0x7fe0fc0a2f70 msgr2=0x7fe0fc0a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:22.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.818+0000 7fe10393d640 1 -- 172.21.15.12:0/4207805385 shutdown_connections 2024-09-11T01:30:22.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.818+0000 7fe10393d640 1 -- 172.21.15.12:0/4207805385 wait complete. 2024-09-11T01:30:22.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.819+0000 7fe10393d640 10 monclient: build_initial_monmap 2024-09-11T01:30:22.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.819+0000 7fe10393d640 10 monclient: monmap: 2024-09-11T01:30:22.834 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:22.834 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:30:22.834 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:30:22.834 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:30:22.834 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:22.834 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.834 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:30:22.834 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:30:22.834 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:30:22.834 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.819+0000 7fe10393d640 1 Processor -- start 2024-09-11T01:30:22.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.819+0000 7fe10393d640 1 -- start start 2024-09-11T01:30:22.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.819+0000 7fe10393d640 10 monclient: init 2024-09-11T01:30:22.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe10393d640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:22.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe10393d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:30:22.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe10393d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0fc09f1d0 0x7fe0fc136270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe10393d640 10 monclient(hunting): picked mon.noname-b con 0x7fe0fc09f1d0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:30:22.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe10393d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe0fc0a2530 0x7fe0fc1ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe10393d640 10 monclient(hunting): picked mon.noname-c con 0x7fe0fc0a2530 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:22.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe10393d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0fc139b90 0x7fe0fc1cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe10393d640 10 monclient(hunting): picked mon.noname-a con 0x7fe0fc139b90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:30:22.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe10393d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:22.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe10393d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe0fc13c640 con 0x7fe0fc139b90 2024-09-11T01:30:22.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe10393d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:22.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe10393d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe0fc13c4c0 con 0x7fe0fc09f1d0 2024-09-11T01:30:22.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe10393d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:22.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe10393d640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe0fc13c7c0 con 0x7fe0fc0a2530 2024-09-11T01:30:22.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe10393d640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:22.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe10393d640 10 monclient(hunting): authenticate will time out at 829.262275s 2024-09-11T01:30:22.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe100ed8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe0fc0a2530 0x7fe0fc1ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe101eda640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0fc139b90 0x7fe0fc1cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe100ed8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe0fc0a2530 0x7fe0fc1ccff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:40660/0 (socket says 172.21.15.12:40660) 2024-09-11T01:30:22.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe100ed8640 1 -- 172.21.15.12:0/1729000181 learned_addr learned my addr 172.21.15.12:0/1729000181 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:30:22.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe100ed8640 10 monclient(hunting): get_auth_request con 0x7fe0fc0a2530 auth_method 0 2024-09-11T01:30:22.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe100ed8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:22.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe100ed8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:22.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe100ed8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:22.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe101eda640 10 monclient(hunting): get_auth_request con 0x7fe0fc139b90 auth_method 0 2024-09-11T01:30:22.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe101eda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:22.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe101eda640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:22.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe101eda640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:22.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe100ed8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:22.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe100ed8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:22.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe100ed8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:22.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe101eda640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:22.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe101eda640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:22.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe101eda640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:22.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe1016d9640 1 --2- 172.21.15.12:0/1729000181 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0fc09f1d0 0x7fe0fc136270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe100ed8640 10 monclient(hunting): handle_auth_done global_id 4454 payload 824 2024-09-11T01:30:22.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe100ed8640 1 -- 172.21.15.12:0/1729000181 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0fc09f1d0 msgr2=0x7fe0fc136270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:22.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe100ed8640 1 --2- 172.21.15.12:0/1729000181 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0fc09f1d0 0x7fe0fc136270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe100ed8640 1 -- 172.21.15.12:0/1729000181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0fc139b90 msgr2=0x7fe0fc1cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:22.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe100ed8640 1 --2- 172.21.15.12:0/1729000181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0fc139b90 0x7fe0fc1cd530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:22.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe100ed8640 10 monclient: _finish_hunting 0 2024-09-11T01:30:22.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe100ed8640 1 monclient: found mon.noname-c 2024-09-11T01:30:22.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe100ed8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:22.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe100ed8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:30:22.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe100ed8640 1 -- 172.21.15.12:0/1729000181 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0fc13ba10 con 0x7fe0fc0a2530 2024-09-11T01:30:22.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe1016d9640 10 monclient: get_auth_request con 0x7fe0fc09f1d0 auth_method 0 2024-09-11T01:30:22.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe101eda640 1 --2- 172.21.15.12:0/1729000181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0fc139b90 0x7fe0fc1cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:30:22.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.820+0000 7fe1016d9640 1 --2- 172.21.15.12:0/1729000181 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0fc09f1d0 0x7fe0fc136270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:30:22.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.822+0000 7fe100ed8640 1 -- 172.21.15.12:0/1729000181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe0fc0a2530 msgr2=0x7fe0fc1ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:22.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.822+0000 7fe100ed8640 1 --2- 172.21.15.12:0/1729000181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe0fc0a2530 0x7fe0fc1ccff0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe0f000c460 tx=0x7fe0f000c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:22.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.823+0000 7fe0f9ffb640 1 -- 172.21.15.12:0/1729000181 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f0007070 con 0x7fe0fc0a2530 2024-09-11T01:30:22.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.823+0000 7fe0f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:22.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.823+0000 7fe0f9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:30:22.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.823+0000 7fe0f9ffb640 10 monclient: dump: 2024-09-11T01:30:22.840 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:22.840 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:22.840 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.840 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.840 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:22.840 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.840 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:22.841 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:22.841 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:22.841 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.823+0000 7fe0f9ffb640 10 monclient: _finish_auth 0 2024-09-11T01:30:22.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.823+0000 7fe0f9ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:30:22.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.823+0000 7fe0f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:22.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.823+0000 7fe0f9ffb640 1 -- 172.21.15.12:0/1729000181 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0f0007210 con 0x7fe0fc0a2530 2024-09-11T01:30:22.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.823+0000 7fe10393d640 5 monclient: authenticate success, global_id 4454 2024-09-11T01:30:22.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.823+0000 7fe10393d640 10 monclient: _renew_subs 2024-09-11T01:30:22.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.823+0000 7fe10393d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:22.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.823+0000 7fe10393d640 1 -- 172.21.15.12:0/1729000181 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0fc123d30 con 0x7fe0fc0a2530 2024-09-11T01:30:22.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.823+0000 7fe0f9ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:22.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.823+0000 7fe0f9ffb640 1 -- 172.21.15.12:0/1729000181 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f0007600 con 0x7fe0fc0a2530 2024-09-11T01:30:22.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.823+0000 7fe0f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:22.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.823+0000 7fe0f9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:22.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.823+0000 7fe0f9ffb640 10 monclient: dump: 2024-09-11T01:30:22.842 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:22.842 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:22.842 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.842 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:22.842 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:22.842 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:22.842 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:22.842 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:22.842 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:22.842 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:22.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.823+0000 7fe10393d640 10 monclient: _renew_subs 2024-09-11T01:30:22.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.823+0000 7fe10393d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:22.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.823+0000 7fe10393d640 1 -- 172.21.15.12:0/1729000181 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0fc1367b0 con 0x7fe0fc0a2530 2024-09-11T01:30:22.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.823+0000 7fe0fb7fe640 4 set_mon_vals no callback set 2024-09-11T01:30:22.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.824+0000 7fe0f9ffb640 1 -- 172.21.15.12:0/1729000181 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fe0f001b400 con 0x7fe0fc0a2530 2024-09-11T01:30:22.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.824+0000 7fe10393d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:22.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.824+0000 7fe10393d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:22.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.824+0000 7fe10393d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:22.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.824+0000 7fe10393d640 1 -- 172.21.15.12:0/1729000181 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0fc09dd70 con 0x7fe0fc0a2530 2024-09-11T01:30:22.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.824+0000 7fe0f9ffb640 1 --2- 172.21.15.12:0/1729000181 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe0e803e210 0x7fe0e80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:22.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.824+0000 7fe0f9ffb640 1 -- 172.21.15.12:0/1729000181 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(39..39 src has 1..39) ==== 4755+0+0 (secure 0 0 0) 0x7fe0f0052a40 con 0x7fe0fc0a2530 2024-09-11T01:30:22.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.824+0000 7fe1016d9640 1 --2- 172.21.15.12:0/1729000181 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe0e803e210 0x7fe0e80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:22.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.824+0000 7fe1016d9640 10 monclient: get_auth_request con 0x7fe0e803e210 auth_method 0 2024-09-11T01:30:22.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.827+0000 7fe1016d9640 1 --2- 172.21.15.12:0/1729000181 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe0e803e210 0x7fe0e80406d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe0f40074d0 tx=0x7fe0f400b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:22.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.829+0000 7fe0f9ffb640 1 -- 172.21.15.12:0/1729000181 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe0fc09dd70 con 0x7fe0fc0a2530 2024-09-11T01:30:22.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.829+0000 7fe0f9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:22.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:22.829+0000 7fe0f9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:30:23.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:23.084+0000 7fe10393d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:30:23.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:23.084+0000 7fe10393d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:30:23.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:23.084+0000 7fe10393d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:23.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:23.084+0000 7fe10393d640 1 -- 172.21.15.12:0/1729000181 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fe0fc0a0350 con 0x7fe0fc0a2530 2024-09-11T01:30:23.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:23.085+0000 7fe0f9ffb640 1 -- 172.21.15.12:0/1729000181 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fe0f0051450 con 0x7fe0fc0a2530 2024-09-11T01:30:23.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:23.085+0000 7fe0f9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:30:23.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:23.085+0000 7fe0f9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:30:23.090 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.085+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:30:23.091 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.085+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=8,config=2,health=12,kv=12,logm=66} crc {auth=2536123749,config=2586702516,health=861809967,kv=3513807615,logm=1749438400}) 2024-09-11T01:30:23.091 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.085+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=7,config=2,health=12,kv=12,logm=67} crc {auth=1707420980,config=2586702516,health=861809967,kv=3263697420,logm=1508941290}) 2024-09-11T01:30:23.091 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.085+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:30:23.091 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.085+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=8,config=2,health=12,kv=12,logm=66} crc {auth=2536123749,config=2586702516,health=861809967,kv=3513807615,logm=1749438400}) 2024-09-11T01:30:23.091 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.085+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=7,config=1,health=9,kv=12,logm=71} crc {auth=1078389514,config=1747058506,health=3538735643,kv=3263697420,logm=224685147}) 2024-09-11T01:30:23.091 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.086+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:30:23.091 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.086+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2789621318}) 2024-09-11T01:30:23.091 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.086+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=280264090}) 2024-09-11T01:30:23.091 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.086+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:30:23.091 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.086+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2789621318}) 2024-09-11T01:30:23.091 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.086+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=193769821}) 2024-09-11T01:30:23.091 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.087+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:30:23.092 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.087+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3564602534}) 2024-09-11T01:30:23.092 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.087+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1991551550}) 2024-09-11T01:30:23.092 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.087+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:30:23.092 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.087+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3564602534}) 2024-09-11T01:30:23.092 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.087+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3489446360}) 2024-09-11T01:30:23.092 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.087+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:30:23.092 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.087+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=100253280}) 2024-09-11T01:30:23.092 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.087+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=183039871}) 2024-09-11T01:30:23.092 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.087+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:30:23.092 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.087+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=100253280}) 2024-09-11T01:30:23.092 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.087+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2909595207}) 2024-09-11T01:30:23.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:23.088+0000 7fe10393d640 1 -- 172.21.15.12:0/1729000181 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe0e803e210 msgr2=0x7fe0e80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:23.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:23.088+0000 7fe10393d640 1 --2- 172.21.15.12:0/1729000181 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe0e803e210 0x7fe0e80406d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe0f40074d0 tx=0x7fe0f400b040 comp rx=0 tx=0).stop 2024-09-11T01:30:23.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:23.088+0000 7fe10393d640 10 monclient: shutdown 2024-09-11T01:30:23.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:23.088+0000 7fe10393d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:23.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:23.088+0000 7fe10393d640 1 -- 172.21.15.12:0/1729000181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe0fc0a2530 msgr2=0x7fe0fc1ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:23.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:23.088+0000 7fe10393d640 1 --2- 172.21.15.12:0/1729000181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe0fc0a2530 0x7fe0fc1ccff0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe0f000c460 tx=0x7fe0f000c920 comp rx=0 tx=0).stop 2024-09-11T01:30:23.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:23.088+0000 7fe10393d640 1 -- 172.21.15.12:0/1729000181 shutdown_connections 2024-09-11T01:30:23.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:23.088+0000 7fe10393d640 1 --2- 172.21.15.12:0/1729000181 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe0e803e210 0x7fe0e80406d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:23.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:23.088+0000 7fe10393d640 1 --2- 172.21.15.12:0/1729000181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0fc139b90 0x7fe0fc1cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:23.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:23.088+0000 7fe10393d640 1 --2- 172.21.15.12:0/1729000181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe0fc0a2530 0x7fe0fc1ccff0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:23.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:23.088+0000 7fe10393d640 1 --2- 172.21.15.12:0/1729000181 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0fc09f1d0 0x7fe0fc136270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:23.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:23.088+0000 7fe10393d640 1 -- 172.21.15.12:0/1729000181 >> 172.21.15.12:0/1729000181 conn(0x7fe0fc0a2f70 msgr2=0x7fe0fc139360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:23.095 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.088+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:30:23.095 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.088+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=27,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=6,osdmap=17} crc {logm=798158767,mdsmap=3760429986,mgr=2161719361,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=3071392264,mon_config_key=3330371543,monmap=2933275274,nvmeofgw=598337686,osd_metadata=642414809,osd_pg_creating=1238027124,osd_snap=2722643615,osdmap=2120955072}) 2024-09-11T01:30:23.095 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.088+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=27,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=6,osdmap=15} crc {logm=798158767,mdsmap=3760429986,mgr=2387707879,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1944120778,mon_config_key=3330371543,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=1238027124,osd_snap=2722643615,osdmap=579124412}) 2024-09-11T01:30:23.095 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.088+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:30:23.096 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.088+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=27,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=6,osdmap=17} crc {logm=798158767,mdsmap=3760429986,mgr=2161719361,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=3071392264,mon_config_key=3330371543,monmap=2933275274,nvmeofgw=598337686,osd_metadata=642414809,osd_pg_creating=1238027124,osd_snap=2722643615,osdmap=2120955072}) 2024-09-11T01:30:23.096 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.088+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=25,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=10,mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=6,osdmap=19} crc {logm=302798129,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgrstat=619913040,mon_config_key=4175047702,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=1238027124,osd_snap=2722643615,osdmap=394868372}) 2024-09-11T01:30:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:23.088+0000 7fe10393d640 1 -- 172.21.15.12:0/1729000181 shutdown_connections 2024-09-11T01:30:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:23.088+0000 7fe10393d640 1 -- 172.21.15.12:0/1729000181 wait complete. 2024-09-11T01:30:23.097 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.090+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:30:23.098 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.090+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=62} crc {osdmap=271675264}) 2024-09-11T01:30:23.098 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.090+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=61} crc {osdmap=445332301}) 2024-09-11T01:30:23.098 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.090+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:30:23.098 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.090+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=62} crc {osdmap=271675264}) 2024-09-11T01:30:23.098 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:30:23.090+0000 7f6b040ef640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=57} crc {osdmap=307466159}) 2024-09-11T01:30:23.101 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:30:23.762 INFO:tasks.rados.rados.0.smithi071.stdout:122: done (3 left) 2024-09-11T01:30:23.762 INFO:tasks.rados.rados.0.smithi071.stdout:123: read oid 38 snap -1 2024-09-11T01:30:23.762 INFO:tasks.rados.rados.0.smithi071.stdout:123: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-11T01:30:23.762 INFO:tasks.rados.rados.0.smithi071.stdout:124: write oid 48 current snap is 11 2024-09-11T01:30:23.762 INFO:tasks.rados.rados.0.smithi071.stdout:124: seq_num 67 ranges {726069=772507,2071117=715676} 2024-09-11T01:30:23.770 INFO:tasks.rados.rados.0.smithi071.stdout:124: writing smithi07134777-48 from 726069 to 1498576 tid 1 2024-09-11T01:30:23.777 INFO:tasks.rados.rados.0.smithi071.stdout:124: writing smithi07134777-48 from 2071117 to 2786793 tid 2 2024-09-11T01:30:23.777 INFO:tasks.rados.rados.0.smithi071.stdout:125: rollback oid 24 current snap is 11 2024-09-11T01:30:23.777 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 24 to 10 2024-09-11T01:30:23.778 INFO:tasks.rados.rados.0.smithi071.stdout:119: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-11T01:30:23.796 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:30:23.846 INFO:tasks.rados.rados.0.smithi071.stdout:121: finishing write tid 1 to smithi07134777-14 2024-09-11T01:30:23.846 INFO:tasks.rados.rados.0.smithi071.stdout:121: finishing write tid 2 to smithi07134777-14 2024-09-11T01:30:23.846 INFO:tasks.rados.rados.0.smithi071.stdout:121: finishing write tid 3 to smithi07134777-14 2024-09-11T01:30:23.846 INFO:tasks.rados.rados.0.smithi071.stdout:121: finishing write tid 4 to smithi07134777-14 2024-09-11T01:30:23.846 INFO:tasks.rados.rados.0.smithi071.stdout:121: finishing write tid 5 to smithi07134777-14 2024-09-11T01:30:23.846 INFO:tasks.rados.rados.0.smithi071.stdout:121: oid 14 updating version 0 to 40 2024-09-11T01:30:23.846 INFO:tasks.rados.rados.0.smithi071.stdout:121: oid 14 updating version 40 to 42 2024-09-11T01:30:23.846 INFO:tasks.rados.rados.0.smithi071.stdout:121: oid 14 version 42 is already newer than 39 2024-09-11T01:30:23.846 INFO:tasks.rados.rados.0.smithi071.stdout:121: oid 14 version 42 is already newer than 41 2024-09-11T01:30:23.846 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 42 (ObjNum 66 snap 11 seq_num 66) dirty exists 2024-09-11T01:30:23.847 INFO:tasks.rados.rados.0.smithi071.stdout:121: left oid 14 (ObjNum 66 snap 11 seq_num 66) 2024-09-11T01:30:23.847 INFO:tasks.rados.rados.0.smithi071.stdout:120: finishing copy_from racing read to smithi07134777-31 2024-09-11T01:30:23.847 INFO:tasks.rados.rados.0.smithi071.stdout:120: finishing copy_from to smithi07134777-31 2024-09-11T01:30:23.847 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 49 (ObjNum 63 snap 8 seq_num 63) dirty exists 2024-09-11T01:30:23.847 INFO:tasks.rados.rados.0.smithi071.stdout:119: done (5 left) 2024-09-11T01:30:23.847 INFO:tasks.rados.rados.0.smithi071.stdout:120: done (4 left) 2024-09-11T01:30:23.847 INFO:tasks.rados.rados.0.smithi071.stdout:121: done (3 left) 2024-09-11T01:30:23.847 INFO:tasks.rados.rados.0.smithi071.stdout:126: copy_from oid 34 from oid 10 current snap is 11 2024-09-11T01:30:23.847 INFO:tasks.rados.rados.0.smithi071.stdout:125: finishing rollback tid 0 to smithi07134777-24 2024-09-11T01:30:23.847 INFO:tasks.rados.rados.0.smithi071.stdout:125: finishing rollback tid 1 to smithi07134777-24 2024-09-11T01:30:23.847 INFO:tasks.rados.rados.0.smithi071.stdout:125: finishing rollback tid 2 to smithi07134777-24 2024-09-11T01:30:23.847 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 32 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-11T01:30:23.847 INFO:tasks.rados.rados.0.smithi071.stdout:123: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-11T01:30:23.859 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:30:23.916 INFO:tasks.rados.rados.0.smithi071.stdout:124: finishing write tid 1 to smithi07134777-48 2024-09-11T01:30:23.916 INFO:tasks.rados.rados.0.smithi071.stdout:124: finishing write tid 2 to smithi07134777-48 2024-09-11T01:30:23.917 INFO:tasks.rados.rados.0.smithi071.stdout:123: done (3 left) 2024-09-11T01:30:23.917 INFO:tasks.rados.rados.0.smithi071.stdout:125: done (2 left) 2024-09-11T01:30:23.917 INFO:tasks.rados.rados.0.smithi071.stdout:127: write (excl) oid 19 current snap is 11 2024-09-11T01:30:23.917 INFO:tasks.rados.rados.0.smithi071.stdout:124: finishing write tid 3 to smithi07134777-48 2024-09-11T01:30:23.917 INFO:tasks.rados.rados.0.smithi071.stdout:124: finishing write tid 4 to smithi07134777-48 2024-09-11T01:30:23.917 INFO:tasks.rados.rados.0.smithi071.stdout:124: oid 48 updating version 0 to 45 2024-09-11T01:30:23.917 INFO:tasks.rados.rados.0.smithi071.stdout:124: oid 48 updating version 45 to 46 2024-09-11T01:30:23.917 INFO:tasks.rados.rados.0.smithi071.stdout:124: oid 48 version 46 is already newer than 44 2024-09-11T01:30:23.917 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 46 (ObjNum 67 snap 11 seq_num 67) dirty exists 2024-09-11T01:30:23.917 INFO:tasks.rados.rados.0.smithi071.stdout:124: left oid 48 (ObjNum 67 snap 11 seq_num 67) 2024-09-11T01:30:23.917 INFO:tasks.rados.rados.0.smithi071.stdout:127: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2024-09-11T01:30:23.922 INFO:tasks.rados.rados.0.smithi071.stdout:127: writing smithi07134777-19 from 742876 to 1190632 tid 1 2024-09-11T01:30:23.928 INFO:tasks.rados.rados.0.smithi071.stdout:127: writing smithi07134777-19 from 1729599 to 2305286 tid 2 2024-09-11T01:30:23.933 INFO:tasks.rados.rados.0.smithi071.stdout:127: writing smithi07134777-19 from 3101234 to 3142876 tid 3 2024-09-11T01:30:23.933 INFO:tasks.rados.rados.0.smithi071.stdout:124: done (2 left) 2024-09-11T01:30:23.933 INFO:tasks.rados.rados.0.smithi071.stdout:128: snap_create 2024-09-11T01:30:23.933 INFO:tasks.rados.rados.0.smithi071.stdout:126: finishing copy_from racing read to smithi07134777-34 2024-09-11T01:30:23.933 INFO:tasks.rados.rados.0.smithi071.stdout:126: finishing copy_from to smithi07134777-34 2024-09-11T01:30:23.933 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-11T01:30:23.933 INFO:tasks.rados.rados.0.smithi071.stdout:127: finishing write tid 1 to smithi07134777-19 2024-09-11T01:30:23.933 INFO:tasks.rados.rados.0.smithi071.stdout:127: finishing write tid 2 to smithi07134777-19 2024-09-11T01:30:23.934 INFO:tasks.rados.rados.0.smithi071.stdout:127: finishing write tid 3 to smithi07134777-19 2024-09-11T01:30:23.935 INFO:tasks.rados.rados.0.smithi071.stdout:127: finishing write tid 4 to smithi07134777-19 2024-09-11T01:30:23.935 INFO:tasks.rados.rados.0.smithi071.stdout:127: finishing write tid 5 to smithi07134777-19 2024-09-11T01:30:23.935 INFO:tasks.rados.rados.0.smithi071.stdout:127: oid 19 updating version 0 to 12 2024-09-11T01:30:23.935 INFO:tasks.rados.rados.0.smithi071.stdout:127: oid 19 version 12 is already newer than 11 2024-09-11T01:30:23.936 INFO:tasks.rados.rados.0.smithi071.stdout:127: oid 19 updating version 12 to 13 2024-09-11T01:30:23.936 INFO:tasks.rados.rados.0.smithi071.stdout:127: oid 19 updating version 13 to 14 2024-09-11T01:30:23.936 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 14 (ObjNum 68 snap 11 seq_num 68) dirty exists 2024-09-11T01:30:23.936 INFO:tasks.rados.rados.0.smithi071.stdout:127: left oid 19 (ObjNum 68 snap 11 seq_num 68) 2024-09-11T01:30:24.102 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:30:24.102 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:30:24.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.213+0000 7f7a7c732640 10 monclient: get_monmap_and_config 2024-09-11T01:30:24.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.213+0000 7f7a7c732640 10 monclient: build_initial_monmap 2024-09-11T01:30:24.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.213+0000 7f7a7c732640 10 monclient: monmap: 2024-09-11T01:30:24.215 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:24.215 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:24.215 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:30:24.214363+0000 2024-09-11T01:30:24.215 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:30:24.214363+0000 2024-09-11T01:30:24.216 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:24.216 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.216 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:30:24.216 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:30:24.216 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:30:24.216 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:30:24.216 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:30:24.216 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.214+0000 7f7a7c732640 1 Processor -- start 2024-09-11T01:30:24.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.214+0000 7f7a7c732640 1 -- start start 2024-09-11T01:30:24.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.214+0000 7f7a7c732640 10 monclient: init 2024-09-11T01:30:24.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7c732640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:24.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7c732640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-11T01:30:24.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7c732640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a740a2e90 0x7f7a74073d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7c732640 10 monclient(hunting): picked mon.noname-b con 0x7f7a740a2e90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:24.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7c732640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7a740a3270 addr v1:172.21.15.12:6789/0 2024-09-11T01:30:24.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7c732640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7a7413b150 0x7f7a7413d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7c732640 10 monclient(hunting): picked mon.noname-c con 0x7f7a7413b150 addr v2:172.21.15.71:3300/0 2024-09-11T01:30:24.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7c732640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a740a46c0 con 0x7f7a740a3270 2024-09-11T01:30:24.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7c732640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:24.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7c732640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f7a74072730 con 0x7f7a7413b150 2024-09-11T01:30:24.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7c732640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:24.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7c732640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7a74072fa0 con 0x7f7a740a2e90 2024-09-11T01:30:24.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7c732640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:24.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7c732640 10 monclient(hunting): authenticate will time out at 830.657187s 2024-09-11T01:30:24.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7a4ce640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a740a2e90 0x7f7a74073d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a79ccd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7a740a3270 0x7f7a740a3670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44012/0 (socket says 172.21.15.12:44012) 2024-09-11T01:30:24.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a79ccd640 1 -- 172.21.15.12:0/725945906 learned_addr learned my addr 172.21.15.12:0/725945906 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:30:24.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7a4ce640 10 monclient(hunting): get_auth_request con 0x7f7a740a2e90 auth_method 0 2024-09-11T01:30:24.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7a4ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:24.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7a4ce640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:24.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7a4ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:24.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7a4ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:24.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7a4ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:24.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7a4ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:24.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7accf640 1 --2- 172.21.15.12:0/725945906 >> v2:172.21.15.71:3300/0 conn(0x7f7a7413b150 0x7f7a7413d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7accf640 10 monclient(hunting): get_auth_request con 0x7f7a7413b150 auth_method 0 2024-09-11T01:30:24.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7accf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:24.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7accf640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:24.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7accf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:24.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7a4ce640 10 monclient(hunting): handle_auth_done global_id 4457 payload 274 2024-09-11T01:30:24.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7a4ce640 1 -- 172.21.15.12:0/725945906 >> v2:172.21.15.71:3300/0 conn(0x7f7a7413b150 msgr2=0x7f7a7413d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:30:24.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7a4ce640 1 --2- 172.21.15.12:0/725945906 >> v2:172.21.15.71:3300/0 conn(0x7f7a7413b150 0x7f7a7413d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7a4ce640 1 -- 172.21.15.12:0/725945906 >> v1:172.21.15.12:6789/0 conn(0x7f7a740a3270 legacy=0x7f7a740a3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:30:24.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7a4ce640 10 monclient: _finish_hunting 0 2024-09-11T01:30:24.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7a4ce640 1 monclient: found mon.noname-b 2024-09-11T01:30:24.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7a4ce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:24.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7a4ce640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:30:24.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.215+0000 7f7a7a4ce640 1 -- 172.21.15.12:0/725945906 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a74073280 con 0x7f7a740a2e90 2024-09-11T01:30:24.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.218+0000 7f7a7a4ce640 1 -- 172.21.15.12:0/725945906 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a740a2e90 msgr2=0x7f7a74073d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:24.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.218+0000 7f7a7a4ce640 1 --2- 172.21.15.12:0/725945906 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a740a2e90 0x7f7a74073d40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7a740735f0 tx=0x7f7a70029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25c7b8701f20e19 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:24.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.218+0000 7f7a794cc640 1 -- 172.21.15.12:0/725945906 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a70037070 con 0x7f7a740a2e90 2024-09-11T01:30:24.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.219+0000 7f7a794cc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:24.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.219+0000 7f7a794cc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:30:24.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.219+0000 7f7a794cc640 10 monclient: dump: 2024-09-11T01:30:24.228 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:24.228 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:24.228 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.228 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.229 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:24.229 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.229 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:24.229 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:24.229 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:24.229 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.219+0000 7f7a794cc640 10 monclient: _finish_auth 0 2024-09-11T01:30:24.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.219+0000 7f7a794cc640 10 monclient: _check_auth_tickets 2024-09-11T01:30:24.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.219+0000 7f7a794cc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:24.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.219+0000 7f7a794cc640 1 -- 172.21.15.12:0/725945906 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a70032440 con 0x7f7a740a2e90 2024-09-11T01:30:24.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.219+0000 7f7a7c732640 5 monclient: authenticate success, global_id 4457 2024-09-11T01:30:24.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.219+0000 7f7a7c732640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:30:24.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.219+0000 7f7a794cc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:24.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.219+0000 7f7a7c732640 10 monclient: get_monmap_and_config success 2024-09-11T01:30:24.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.219+0000 7f7a7c732640 4 set_mon_vals no callback set 2024-09-11T01:30:24.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.219+0000 7f7a794cc640 1 -- 172.21.15.12:0/725945906 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a70032810 con 0x7f7a740a2e90 2024-09-11T01:30:24.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.219+0000 7f7a794cc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:24.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.219+0000 7f7a794cc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:24.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.219+0000 7f7a794cc640 10 monclient: dump: 2024-09-11T01:30:24.230 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:24.230 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:24.230 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.230 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.230 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:24.230 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.230 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:24.230 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:24.231 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.219+0000 7f7a7c732640 10 monclient: shutdown 2024-09-11T01:30:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.219+0000 7f7a7c732640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.219+0000 7f7a7c732640 1 -- 172.21.15.12:0/725945906 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a740a2e90 msgr2=0x7f7a74073d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.219+0000 7f7a7c732640 1 --2- 172.21.15.12:0/725945906 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a740a2e90 0x7f7a74073d40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7a740735f0 tx=0x7f7a70029fb0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.219+0000 7f7a7c732640 1 -- 172.21.15.12:0/725945906 shutdown_connections 2024-09-11T01:30:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.219+0000 7f7a7c732640 1 --2- 172.21.15.12:0/725945906 >> v2:172.21.15.71:3300/0 conn(0x7f7a7413b150 0x7f7a7413d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.219+0000 7f7a7c732640 1 --2- 172.21.15.12:0/725945906 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a740a2e90 0x7f7a74073d40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.219+0000 7f7a7c732640 1 -- 172.21.15.12:0/725945906 >> 172.21.15.12:0/725945906 conn(0x7f7a740a8490 msgr2=0x7f7a740a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.219+0000 7f7a7c732640 1 -- 172.21.15.12:0/725945906 shutdown_connections 2024-09-11T01:30:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.219+0000 7f7a7c732640 1 -- 172.21.15.12:0/725945906 wait complete. 2024-09-11T01:30:24.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.220+0000 7f7a7c732640 10 monclient: build_initial_monmap 2024-09-11T01:30:24.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.220+0000 7f7a7c732640 10 monclient: monmap: 2024-09-11T01:30:24.232 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:24.232 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:30:24.232 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:30:24.232 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:30:24.232 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:24.232 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.232 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:30:24.232 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:30:24.232 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:30:24.232 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.220+0000 7f7a7c732640 1 Processor -- start 2024-09-11T01:30:24.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.220+0000 7f7a7c732640 1 -- start start 2024-09-11T01:30:24.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.220+0000 7f7a7c732640 10 monclient: init 2024-09-11T01:30:24.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7c732640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:24.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7c732640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:30:24.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7c732640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a740a2e90 0x7f7a74137110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7c732640 10 monclient(hunting): picked mon.noname-c con 0x7f7a740a2e90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:24.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7c732640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a740a3270 0x7f7a74137650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7c732640 10 monclient(hunting): picked mon.noname-a con 0x7f7a740a3270 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:30:24.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7c732640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a7413b150 0x7f7a74137b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7c732640 10 monclient(hunting): picked mon.noname-b con 0x7f7a7413b150 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:30:24.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7c732640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:24.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7c732640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7a74072480 con 0x7f7a740a3270 2024-09-11T01:30:24.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7c732640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7c732640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f7a74072300 con 0x7f7a7413b150 2024-09-11T01:30:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7c732640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7c732640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7a7413da80 con 0x7f7a740a2e90 2024-09-11T01:30:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7c732640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7c732640 10 monclient(hunting): authenticate will time out at 830.663164s 2024-09-11T01:30:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7a4ce640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a740a2e90 0x7f7a74137110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a79ccd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a740a3270 0x7f7a74137650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7a4ce640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a740a2e90 0x7f7a74137110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:40682/0 (socket says 172.21.15.12:40682) 2024-09-11T01:30:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7a4ce640 1 -- 172.21.15.12:0/3480686153 learned_addr learned my addr 172.21.15.12:0/3480686153 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:30:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7a4ce640 10 monclient(hunting): get_auth_request con 0x7f7a740a2e90 auth_method 0 2024-09-11T01:30:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7a4ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7a4ce640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7a4ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:24.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a79ccd640 10 monclient(hunting): get_auth_request con 0x7f7a740a3270 auth_method 0 2024-09-11T01:30:24.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a79ccd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:24.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a79ccd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:24.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a79ccd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:24.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7a4ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:24.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7a4ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:24.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7a4ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:24.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7accf640 1 --2- 172.21.15.12:0/3480686153 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a7413b150 0x7f7a74137b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a79ccd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:24.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a79ccd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:24.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a79ccd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:24.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7accf640 10 monclient(hunting): get_auth_request con 0x7f7a7413b150 auth_method 0 2024-09-11T01:30:24.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7accf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:24.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7accf640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:24.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7accf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:24.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7accf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:24.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7accf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:24.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7accf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:24.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7a4ce640 10 monclient(hunting): handle_auth_done global_id 4460 payload 824 2024-09-11T01:30:24.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7a4ce640 1 -- 172.21.15.12:0/3480686153 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a7413b150 msgr2=0x7f7a74137b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:24.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7a4ce640 1 --2- 172.21.15.12:0/3480686153 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a7413b150 0x7f7a74137b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7a4ce640 1 -- 172.21.15.12:0/3480686153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a740a3270 msgr2=0x7f7a74137650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:24.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7a4ce640 1 --2- 172.21.15.12:0/3480686153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a740a3270 0x7f7a74137650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7a4ce640 10 monclient: _finish_hunting 0 2024-09-11T01:30:24.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7a4ce640 1 monclient: found mon.noname-c 2024-09-11T01:30:24.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7a4ce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:24.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7a4ce640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:30:24.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a7a4ce640 1 -- 172.21.15.12:0/3480686153 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a74139be0 con 0x7f7a740a2e90 2024-09-11T01:30:24.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.221+0000 7f7a79ccd640 1 --2- 172.21.15.12:0/3480686153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a740a3270 0x7f7a74137650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:30:24.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.223+0000 7f7a7a4ce640 1 -- 172.21.15.12:0/3480686153 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a740a2e90 msgr2=0x7f7a74137110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:24.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.223+0000 7f7a7a4ce640 1 --2- 172.21.15.12:0/3480686153 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a740a2e90 0x7f7a74137110 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7a7002a5a0 tx=0x7f7a700046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:24.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.224+0000 7f7a6affd640 1 -- 172.21.15.12:0/3480686153 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a70037070 con 0x7f7a740a2e90 2024-09-11T01:30:24.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.224+0000 7f7a6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:24.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.224+0000 7f7a6affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:30:24.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.224+0000 7f7a6affd640 10 monclient: dump: 2024-09-11T01:30:24.237 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:24.237 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:24.237 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.237 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.237 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:24.237 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.237 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:24.238 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:24.238 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:24.238 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.224+0000 7f7a6affd640 10 monclient: _finish_auth 0 2024-09-11T01:30:24.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.224+0000 7f7a6affd640 10 monclient: _check_auth_tickets 2024-09-11T01:30:24.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.224+0000 7f7a6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:24.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.224+0000 7f7a6affd640 1 -- 172.21.15.12:0/3480686153 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a70041070 con 0x7f7a740a2e90 2024-09-11T01:30:24.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.224+0000 7f7a7c732640 5 monclient: authenticate success, global_id 4460 2024-09-11T01:30:24.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.224+0000 7f7a7c732640 10 monclient: _renew_subs 2024-09-11T01:30:24.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.224+0000 7f7a7c732640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:24.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.224+0000 7f7a7c732640 1 -- 172.21.15.12:0/3480686153 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a74139ec0 con 0x7f7a740a2e90 2024-09-11T01:30:24.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.224+0000 7f7a6affd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:24.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.224+0000 7f7a6affd640 1 -- 172.21.15.12:0/3480686153 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a7003bcc0 con 0x7f7a740a2e90 2024-09-11T01:30:24.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.224+0000 7f7a6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:24.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.224+0000 7f7a6affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:24.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.224+0000 7f7a6affd640 10 monclient: dump: 2024-09-11T01:30:24.239 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:24.239 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:24.239 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.239 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.239 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:24.239 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.239 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:24.239 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:24.239 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:24.239 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.224+0000 7f7a7c732640 10 monclient: _renew_subs 2024-09-11T01:30:24.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.224+0000 7f7a7c732640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.224+0000 7f7a7c732640 1 -- 172.21.15.12:0/3480686153 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a7413b880 con 0x7f7a740a2e90 2024-09-11T01:30:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.224+0000 7f7a78ccb640 4 set_mon_vals no callback set 2024-09-11T01:30:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.225+0000 7f7a7c732640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.225+0000 7f7a7c732640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.225+0000 7f7a7c732640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.225+0000 7f7a7c732640 1 -- 172.21.15.12:0/3480686153 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a7409ecf0 con 0x7f7a740a2e90 2024-09-11T01:30:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.225+0000 7f7a6affd640 1 -- 172.21.15.12:0/3480686153 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f7a70046460 con 0x7f7a740a2e90 2024-09-11T01:30:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.225+0000 7f7a6affd640 1 --2- 172.21.15.12:0/3480686153 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7a5803e260 0x7f7a58040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.225+0000 7f7a6affd640 1 -- 172.21.15.12:0/3480686153 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(40..40 src has 1..40) ==== 4767+0+0 (secure 0 0 0) 0x7f7a700731d0 con 0x7f7a740a2e90 2024-09-11T01:30:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.225+0000 7f7a79ccd640 1 --2- 172.21.15.12:0/3480686153 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7a5803e260 0x7f7a58040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.225+0000 7f7a79ccd640 10 monclient: get_auth_request con 0x7f7a5803e260 auth_method 0 2024-09-11T01:30:24.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.228+0000 7f7a79ccd640 1 --2- 172.21.15.12:0/3480686153 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7a5803e260 0x7f7a58040720 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7a64007c90 tx=0x7f7a64007be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:24.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.230+0000 7f7a6affd640 1 -- 172.21.15.12:0/3480686153 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7a7409ecf0 con 0x7f7a740a2e90 2024-09-11T01:30:24.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.230+0000 7f7a6affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:24.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.230+0000 7f7a6affd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:30:24.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.483+0000 7f7a7c732640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:30:24.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.484+0000 7f7a7c732640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:30:24.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.484+0000 7f7a7c732640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:24.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.484+0000 7f7a7c732640 1 -- 172.21.15.12:0/3480686153 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7a74073d40 con 0x7f7a740a2e90 2024-09-11T01:30:24.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.484+0000 7f7a6affd640 1 -- 172.21.15.12:0/3480686153 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f7a70046780 con 0x7f7a740a2e90 2024-09-11T01:30:24.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.484+0000 7f7a6affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:30:24.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.484+0000 7f7a6affd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:30:24.486 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:30:24.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.486+0000 7f7a7c732640 1 -- 172.21.15.12:0/3480686153 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7a5803e260 msgr2=0x7f7a58040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:24.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.486+0000 7f7a7c732640 1 --2- 172.21.15.12:0/3480686153 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7a5803e260 0x7f7a58040720 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7a64007c90 tx=0x7f7a64007be0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.486+0000 7f7a7c732640 10 monclient: shutdown 2024-09-11T01:30:24.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.486+0000 7f7a7c732640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:24.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.486+0000 7f7a7c732640 1 -- 172.21.15.12:0/3480686153 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a740a2e90 msgr2=0x7f7a74137110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:24.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.486+0000 7f7a7c732640 1 --2- 172.21.15.12:0/3480686153 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a740a2e90 0x7f7a74137110 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7a7002a5a0 tx=0x7f7a700046c0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.486+0000 7f7a7c732640 1 -- 172.21.15.12:0/3480686153 shutdown_connections 2024-09-11T01:30:24.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.486+0000 7f7a7c732640 1 --2- 172.21.15.12:0/3480686153 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a7413b150 0x7f7a74137b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.486+0000 7f7a7c732640 1 --2- 172.21.15.12:0/3480686153 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7a5803e260 0x7f7a58040720 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.486+0000 7f7a7c732640 1 --2- 172.21.15.12:0/3480686153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a740a3270 0x7f7a74137650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.486+0000 7f7a7c732640 1 --2- 172.21.15.12:0/3480686153 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a740a2e90 0x7f7a74137110 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.486+0000 7f7a7c732640 1 -- 172.21.15.12:0/3480686153 >> 172.21.15.12:0/3480686153 conn(0x7f7a740a8490 msgr2=0x7f7a7409e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:24.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.486+0000 7f7a7c732640 1 -- 172.21.15.12:0/3480686153 shutdown_connections 2024-09-11T01:30:24.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.486+0000 7f7a7c732640 1 -- 172.21.15.12:0/3480686153 wait complete. 2024-09-11T01:30:24.498 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:30:24.498 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:30:24.498 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:30:24.498 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:30:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.598+0000 7f008f2c6640 10 monclient: get_monmap_and_config 2024-09-11T01:30:24.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.598+0000 7f008f2c6640 10 monclient: build_initial_monmap 2024-09-11T01:30:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.598+0000 7f008f2c6640 10 monclient: monmap: 2024-09-11T01:30:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:30:24.599587+0000 2024-09-11T01:30:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:30:24.599587+0000 2024-09-11T01:30:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:30:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:30:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:30:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:30:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:30:24.601 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.599+0000 7f008f2c6640 1 Processor -- start 2024-09-11T01:30:24.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.599+0000 7f008f2c6640 1 -- start start 2024-09-11T01:30:24.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.599+0000 7f008f2c6640 10 monclient: init 2024-09-11T01:30:24.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008f2c6640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:24.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008f2c6640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-11T01:30:24.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008f2c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f00880a2d40 addr v1:172.21.15.12:6789/0 2024-09-11T01:30:24.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008f2c6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f008806d260 addr v1:172.21.15.71:6789/0 2024-09-11T01:30:24.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008f2c6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f008806d670 0x7f0088140c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008f2c6640 10 monclient(hunting): picked mon.noname-c con 0x7f008806d670 addr v2:172.21.15.71:3300/0 2024-09-11T01:30:24.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008f2c6640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f008809aed0 con 0x7f00880a2d40 2024-09-11T01:30:24.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008f2c6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00880a3920 con 0x7f008806d260 2024-09-11T01:30:24.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008f2c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:24.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008f2c6640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f008806dd10 con 0x7f008806d670 2024-09-11T01:30:24.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008f2c6640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:24.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008f2c6640 10 monclient(hunting): authenticate will time out at 831.042168s 2024-09-11T01:30:24.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008d062640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f00880a2d40 0x7f008806cb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44026/0 (socket says 172.21.15.12:44026) 2024-09-11T01:30:24.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008d062640 1 -- 172.21.15.12:0/2367807240 learned_addr learned my addr 172.21.15.12:0/2367807240 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:30:24.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008d863640 1 --2- 172.21.15.12:0/2367807240 >> v2:172.21.15.71:3300/0 conn(0x7f008806d670 0x7f0088140c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008d863640 10 monclient(hunting): get_auth_request con 0x7f008806d670 auth_method 0 2024-09-11T01:30:24.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008d863640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:24.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008d863640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:24.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008d863640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:24.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008d863640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:24.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008d863640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:24.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008d863640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:24.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008d863640 10 monclient(hunting): handle_auth_done global_id 4384 payload 274 2024-09-11T01:30:24.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008d863640 1 -- 172.21.15.12:0/2367807240 >> v1:172.21.15.71:6789/0 conn(0x7f008806d260 legacy=0x7f008813d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:30:24.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008d863640 1 -- 172.21.15.12:0/2367807240 >> v1:172.21.15.12:6789/0 conn(0x7f00880a2d40 legacy=0x7f008806cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:30:24.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008d863640 10 monclient: _finish_hunting 0 2024-09-11T01:30:24.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008d863640 1 monclient: found mon.noname-c 2024-09-11T01:30:24.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008d863640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:24.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008d863640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:30:24.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.600+0000 7f008d863640 1 -- 172.21.15.12:0/2367807240 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00880a3320 con 0x7f008806d670 2024-09-11T01:30:24.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.603+0000 7f008d863640 1 -- 172.21.15.12:0/2367807240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f008806d670 msgr2=0x7f0088140c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:24.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.603+0000 7f008d863640 1 --2- 172.21.15.12:0/2367807240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f008806d670 0x7f0088140c10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0080009d90 tx=0x7f008002a490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2758dc4bd898e31 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:24.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.603+0000 7f007bfff640 1 -- 172.21.15.12:0/2367807240 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0080007a30 con 0x7f008806d670 2024-09-11T01:30:24.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.603+0000 7f007bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:24.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.603+0000 7f007bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:30:24.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.603+0000 7f007bfff640 10 monclient: dump: 2024-09-11T01:30:24.611 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:24.611 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:24.611 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.611 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.611 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:24.611 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.611 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:24.611 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:24.611 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:24.611 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.603+0000 7f007bfff640 10 monclient: _finish_auth 0 2024-09-11T01:30:24.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.603+0000 7f007bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:30:24.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.603+0000 7f007bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:24.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.603+0000 7f007bfff640 1 -- 172.21.15.12:0/2367807240 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0080007bd0 con 0x7f008806d670 2024-09-11T01:30:24.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.603+0000 7f007bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:24.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.604+0000 7f008f2c6640 5 monclient: authenticate success, global_id 4384 2024-09-11T01:30:24.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.604+0000 7f008f2c6640 10 monclient: get_monmap_and_config success 2024-09-11T01:30:24.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.604+0000 7f008f2c6640 4 set_mon_vals no callback set 2024-09-11T01:30:24.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.604+0000 7f007bfff640 1 -- 172.21.15.12:0/2367807240 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0080031400 con 0x7f008806d670 2024-09-11T01:30:24.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.604+0000 7f007bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:24.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.604+0000 7f007bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:30:24.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.604+0000 7f007bfff640 10 monclient: dump: 2024-09-11T01:30:24.612 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:24.612 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:24.613 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.613 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.613 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:24.613 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.613 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:24.613 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:24.613 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:24.613 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.604+0000 7f008f2c6640 10 monclient: shutdown 2024-09-11T01:30:24.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.604+0000 7f008f2c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:24.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.604+0000 7f008f2c6640 1 -- 172.21.15.12:0/2367807240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f008806d670 msgr2=0x7f0088140c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:24.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.604+0000 7f008f2c6640 1 --2- 172.21.15.12:0/2367807240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f008806d670 0x7f0088140c10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0080009d90 tx=0x7f008002a490 comp rx=0 tx=0).stop 2024-09-11T01:30:24.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.604+0000 7f008f2c6640 1 -- 172.21.15.12:0/2367807240 shutdown_connections 2024-09-11T01:30:24.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.604+0000 7f008f2c6640 1 --2- 172.21.15.12:0/2367807240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f008806d670 0x7f0088140c10 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.604+0000 7f008f2c6640 1 -- 172.21.15.12:0/2367807240 >> 172.21.15.12:0/2367807240 conn(0x7f00880a3e30 msgr2=0x7f00880a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.604+0000 7f008f2c6640 1 -- 172.21.15.12:0/2367807240 shutdown_connections 2024-09-11T01:30:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.604+0000 7f008f2c6640 1 -- 172.21.15.12:0/2367807240 wait complete. 2024-09-11T01:30:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.605+0000 7f008f2c6640 10 monclient: build_initial_monmap 2024-09-11T01:30:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.605+0000 7f008f2c6640 10 monclient: monmap: 2024-09-11T01:30:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:30:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:30:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:30:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:30:24.614 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:30:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:30:24.615 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.605+0000 7f008f2c6640 1 Processor -- start 2024-09-11T01:30:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.605+0000 7f008f2c6640 1 -- start start 2024-09-11T01:30:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.605+0000 7f008f2c6640 10 monclient: init 2024-09-11T01:30:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.605+0000 7f008f2c6640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.605+0000 7f008f2c6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:30:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.605+0000 7f008f2c6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f008806d260 0x7f008813bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.605+0000 7f008f2c6640 10 monclient(hunting): picked mon.noname-a con 0x7f008806d260 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:30:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.605+0000 7f008f2c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f008806d670 0x7f008813c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.605+0000 7f008f2c6640 10 monclient(hunting): picked mon.noname-c con 0x7f008806d670 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.605+0000 7f008f2c6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00880a2d40 0x7f008813c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.605+0000 7f008f2c6640 10 monclient(hunting): picked mon.noname-b con 0x7f00880a2d40 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:30:24.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.605+0000 7f008f2c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:24.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.605+0000 7f008f2c6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f00880a3120 con 0x7f008806d260 2024-09-11T01:30:24.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.605+0000 7f008f2c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:24.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.605+0000 7f008f2c6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f008809bcf0 con 0x7f00880a2d40 2024-09-11T01:30:24.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.605+0000 7f008f2c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:24.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.605+0000 7f008f2c6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f008809ba10 con 0x7f008806d670 2024-09-11T01:30:24.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.605+0000 7f008f2c6640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:24.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.605+0000 7f008f2c6640 10 monclient(hunting): authenticate will time out at 831.047973s 2024-09-11T01:30:24.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008d062640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f008806d260 0x7f008813bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008c861640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f008806d670 0x7f008813c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008d062640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f008806d260 0x7f008813bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41086/0 (socket says 172.21.15.12:41086) 2024-09-11T01:30:24.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008d062640 1 -- 172.21.15.12:0/536790347 learned_addr learned my addr 172.21.15.12:0/536790347 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:30:24.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008d062640 10 monclient(hunting): get_auth_request con 0x7f008806d260 auth_method 0 2024-09-11T01:30:24.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008d062640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:24.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008d062640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:24.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008d062640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:24.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008c861640 10 monclient(hunting): get_auth_request con 0x7f008806d670 auth_method 0 2024-09-11T01:30:24.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008c861640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:24.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008c861640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:24.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008c861640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:24.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008d062640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:24.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008d062640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:24.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008d062640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:24.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008c861640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:24.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008c861640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:24.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008c861640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:24.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008d863640 1 --2- 172.21.15.12:0/536790347 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00880a2d40 0x7f008813c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008d863640 10 monclient(hunting): get_auth_request con 0x7f00880a2d40 auth_method 0 2024-09-11T01:30:24.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008d863640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:24.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008d863640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:24.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008d863640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:24.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008d863640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:24.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008d863640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:24.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008d863640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:24.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008c861640 10 monclient(hunting): handle_auth_done global_id 4463 payload 824 2024-09-11T01:30:24.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008c861640 1 -- 172.21.15.12:0/536790347 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00880a2d40 msgr2=0x7f008813c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:24.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008c861640 1 --2- 172.21.15.12:0/536790347 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00880a2d40 0x7f008813c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008c861640 1 -- 172.21.15.12:0/536790347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f008806d260 msgr2=0x7f008813bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:24.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008c861640 1 --2- 172.21.15.12:0/536790347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f008806d260 0x7f008813bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008c861640 10 monclient: _finish_hunting 0 2024-09-11T01:30:24.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008c861640 1 monclient: found mon.noname-c 2024-09-11T01:30:24.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008c861640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:24.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008c861640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:30:24.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008c861640 1 -- 172.21.15.12:0/536790347 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f008813e7a0 con 0x7f008806d670 2024-09-11T01:30:24.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.606+0000 7f008d062640 1 --2- 172.21.15.12:0/536790347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f008806d260 0x7f008813bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:30:24.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.608+0000 7f008c861640 1 -- 172.21.15.12:0/536790347 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f008806d670 msgr2=0x7f008813c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:24.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.608+0000 7f008c861640 1 --2- 172.21.15.12:0/536790347 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f008806d670 0x7f008813c090 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f008400c440 tx=0x7f008400c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:24.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.608+0000 7f0079ffb640 1 -- 172.21.15.12:0/536790347 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00840119a0 con 0x7f008806d670 2024-09-11T01:30:24.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.608+0000 7f0079ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:24.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.608+0000 7f0079ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:30:24.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.608+0000 7f0079ffb640 10 monclient: dump: 2024-09-11T01:30:24.619 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:24.619 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:24.619 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.619 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.619 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:24.619 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:24.620 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.608+0000 7f0079ffb640 10 monclient: _finish_auth 0 2024-09-11T01:30:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.608+0000 7f0079ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:30:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.608+0000 7f0079ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.608+0000 7f0079ffb640 1 -- 172.21.15.12:0/536790347 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0084017070 con 0x7f008806d670 2024-09-11T01:30:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.608+0000 7f0079ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.608+0000 7f0079ffb640 1 -- 172.21.15.12:0/536790347 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0084011d80 con 0x7f008806d670 2024-09-11T01:30:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.608+0000 7f008f2c6640 5 monclient: authenticate success, global_id 4463 2024-09-11T01:30:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.608+0000 7f008f2c6640 10 monclient: _renew_subs 2024-09-11T01:30:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.608+0000 7f008f2c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:24.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.608+0000 7f008f2c6640 1 -- 172.21.15.12:0/536790347 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f008813dbf0 con 0x7f008806d670 2024-09-11T01:30:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.608+0000 7f0079ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.608+0000 7f0079ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.608+0000 7f0079ffb640 10 monclient: dump: 2024-09-11T01:30:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:24.621 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.608+0000 7f007b7fe640 4 set_mon_vals no callback set 2024-09-11T01:30:24.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.609+0000 7f008f2c6640 10 monclient: _renew_subs 2024-09-11T01:30:24.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.609+0000 7f008f2c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:24.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.609+0000 7f008f2c6640 1 -- 172.21.15.12:0/536790347 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f008813df00 con 0x7f008806d670 2024-09-11T01:30:24.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.609+0000 7f0079ffb640 1 -- 172.21.15.12:0/536790347 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f008401c460 con 0x7f008806d670 2024-09-11T01:30:24.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.609+0000 7f008f2c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:24.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.609+0000 7f008f2c6640 1 --2- 172.21.15.12:0/536790347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00881374a0 0x7f0088137880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.609+0000 7f008f2c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:24.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.609+0000 7f008f2c6640 1 -- 172.21.15.12:0/536790347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f008813e0d0 con 0x7f00881374a0 2024-09-11T01:30:24.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.610+0000 7f008d062640 1 --2- 172.21.15.12:0/536790347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00881374a0 0x7f0088137880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.610+0000 7f008d062640 10 monclient: get_auth_request con 0x7f00881374a0 auth_method 0 2024-09-11T01:30:24.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.610+0000 7f008d062640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:24.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.610+0000 7f008d062640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:24.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.610+0000 7f008d062640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:24.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.610+0000 7f0079ffb640 1 --2- 172.21.15.12:0/536790347 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f006c03e210 0x7f006c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.610+0000 7f008d062640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:24.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.610+0000 7f008d062640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:24.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.610+0000 7f008d062640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:24.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.610+0000 7f0079ffb640 1 -- 172.21.15.12:0/536790347 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(40..40 src has 1..40) ==== 4767+0+0 (secure 0 0 0) 0x7f0084052af0 con 0x7f008806d670 2024-09-11T01:30:24.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.610+0000 7f008d062640 10 monclient(hunting): handle_auth_done global_id 14130 payload 824 2024-09-11T01:30:24.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.610+0000 7f008d062640 1 -- 172.21.15.12:0/536790347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f008806cbd0 con 0x7f00881374a0 2024-09-11T01:30:24.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.610+0000 7f008d863640 1 --2- 172.21.15.12:0/536790347 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f006c03e210 0x7f006c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.610+0000 7f008d863640 10 monclient: get_auth_request con 0x7f006c03e210 auth_method 0 2024-09-11T01:30:24.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.612+0000 7f008d062640 1 -- 172.21.15.12:0/536790347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00881374a0 msgr2=0x7f0088137880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:24.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.612+0000 7f008d062640 1 --2- 172.21.15.12:0/536790347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00881374a0 0x7f0088137880 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f007c0072e0 tx=0x7f007c0077a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:24.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.612+0000 7f0079ffb640 1 -- 172.21.15.12:0/536790347 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c00fd40 con 0x7f00881374a0 2024-09-11T01:30:24.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.612+0000 7f0079ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:24.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.612+0000 7f0079ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:30:24.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.612+0000 7f0079ffb640 10 monclient: dump: 2024-09-11T01:30:24.627 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:24.627 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:24.627 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.627 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.628 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:24.628 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.628 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:24.628 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:24.628 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:24.628 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.612+0000 7f0079ffb640 1 -- 172.21.15.12:0/536790347 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f007c00fd40 con 0x7f00881374a0 2024-09-11T01:30:24.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.612+0000 7f0079ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:24.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.612+0000 7f0079ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:30:24.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.612+0000 7f0079ffb640 1 -- 172.21.15.12:0/536790347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00881374a0 msgr2=0x7f0088137880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:24.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.612+0000 7f0079ffb640 1 --2- 172.21.15.12:0/536790347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00881374a0 0x7f0088137880 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f007c0072e0 tx=0x7f007c0077a0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.612+0000 7f0079ffb640 1 -- 172.21.15.12:0/536790347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00881374a0 msgr2=0x7f0088137880 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:30:24.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.612+0000 7f0079ffb640 1 --2- 172.21.15.12:0/536790347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f00881374a0 0x7f0088137880 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f007c0072e0 tx=0x7f007c0077a0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.612+0000 7f008d863640 1 --2- 172.21.15.12:0/536790347 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f006c03e210 0x7f006c0406d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0080034000 tx=0x7f008002f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:24.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.619+0000 7f008f2c6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:30:24.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.619+0000 7f008f2c6640 1 --2- 172.21.15.12:0/536790347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0088137dc0 0x7f0088139810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.619+0000 7f008f2c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:24.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.619+0000 7f008f2c6640 1 -- 172.21.15.12:0/536790347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f008813e390 con 0x7f0088137dc0 2024-09-11T01:30:24.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.619+0000 7f008d062640 1 --2- 172.21.15.12:0/536790347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0088137dc0 0x7f0088139810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.619+0000 7f008d062640 10 monclient: get_auth_request con 0x7f0088137dc0 auth_method 0 2024-09-11T01:30:24.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.619+0000 7f008d062640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:24.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.619+0000 7f008d062640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:24.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.619+0000 7f008d062640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:24.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.619+0000 7f008d062640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:24.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.619+0000 7f008d062640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:24.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.619+0000 7f008d062640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:24.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.619+0000 7f008d062640 10 monclient(hunting): handle_auth_done global_id 14133 payload 824 2024-09-11T01:30:24.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.619+0000 7f008d062640 1 -- 172.21.15.12:0/536790347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0088074320 con 0x7f0088137dc0 2024-09-11T01:30:24.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.621+0000 7f008d062640 1 -- 172.21.15.12:0/536790347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0088137dc0 msgr2=0x7f0088139810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:24.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.621+0000 7f008d062640 1 --2- 172.21.15.12:0/536790347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0088137dc0 0x7f0088139810 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f007c0072e0 tx=0x7f007c004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:24.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.621+0000 7f0079ffb640 1 -- 172.21.15.12:0/536790347 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c01ee40 con 0x7f0088137dc0 2024-09-11T01:30:24.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.621+0000 7f0079ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:24.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.621+0000 7f0079ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:30:24.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.621+0000 7f0079ffb640 10 monclient: dump: 2024-09-11T01:30:24.630 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:24.630 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:24.630 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.630 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.630 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:24.630 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.630 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:24.630 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:24.631 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:24.631 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.622+0000 7f0079ffb640 1 -- 172.21.15.12:0/536790347 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4542 (secure 0 0 0) 0x7f007c006240 con 0x7f0088137dc0 2024-09-11T01:30:24.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.622+0000 7f0079ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:30:24.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.622+0000 7f0079ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:30:24.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.622+0000 7f0079ffb640 1 -- 172.21.15.12:0/536790347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0088137dc0 msgr2=0x7f0088139810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:24.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.622+0000 7f0079ffb640 1 --2- 172.21.15.12:0/536790347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0088137dc0 0x7f0088139810 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f007c0072e0 tx=0x7f007c004940 comp rx=0 tx=0).stop 2024-09-11T01:30:24.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.622+0000 7f0079ffb640 1 -- 172.21.15.12:0/536790347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0088137dc0 msgr2=0x7f0088139810 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:30:24.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.622+0000 7f0079ffb640 1 --2- 172.21.15.12:0/536790347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0088137dc0 0x7f0088139810 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f007c0072e0 tx=0x7f007c004940 comp rx=0 tx=0).stop 2024-09-11T01:30:24.631 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:30:24.632 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:30:24.632 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:30:24.632 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:30:24.632 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 14, 2024-09-11T01:30:24.632 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 18208, 2024-09-11T01:30:24.632 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:30:24.632 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:30:24.632 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:30:24.632 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:30:24.632 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:24.632 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:30:24.632 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:30:24.632 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:30:24.633 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:30:24.633 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:30:24.633 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:30:24.633 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:30:24.633 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:30:24.633 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:30:24.633 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:30:24.633 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:30:24.633 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:30:24.633 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:30:24.633 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:30:24.633 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:30:24.633 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:24.633 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:30:24.634 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:30:24.634 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:30:24.634 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:30:24.634 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:30:24.634 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:30:24.634 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:30:24.634 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:30:24.634 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:30:24.634 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:30:24.634 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:30:24.634 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:30:24.634 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:30:24.634 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:24.634 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.635 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:30:24.635 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:30:24.635 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:30:24.635 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:30:24.635 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:30:24.635 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:30:24.635 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:30:24.635 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:30:24.635 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:30:24.635 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:30:24.635 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:30:24.635 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:30:24.635 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:30:24.635 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:30:24.636 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:30:24.636 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:30:24.636 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:30:24.636 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:30:24.636 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:30:24.636 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:30:24.636 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:30:24.636 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:30:24.636 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:30:24.636 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:30:24.636 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:30:24.636 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:30:24.636 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:30:24.636 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:30:24.637 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:24.637 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:30:24.637 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.637 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:30:24.637 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.637 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:30:24.637 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:30:24.637 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:30:24.637 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:30:24.637 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.637 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:30:24.637 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:30:24.637 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:24.638 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.638 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.638 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:30:24.638 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:30:24.638 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:24.638 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:24.638 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:24.638 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.638 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:30:24.638 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:30:24.638 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:30:24.638 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:30:24.638 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:30:24.638 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.638 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.639 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:30:24.639 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:30:24.639 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:30:24.639 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:30:24.639 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.639 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:30:24.639 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:30:24.639 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:24.639 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.639 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.639 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:30:24.639 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:30:24.639 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:24.639 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:24.640 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:24.640 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.640 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:30:24.640 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:30:24.640 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:30:24.640 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:30:24.640 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:30:24.640 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.640 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.640 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:30:24.641 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:30:24.641 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:30:24.641 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:30:24.641 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.641 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:30:24.641 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:30:24.641 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:24.641 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.641 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.641 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:30:24.641 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:30:24.641 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:24.641 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:24.641 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:24.641 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.642 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:30:24.642 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:30:24.642 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:30:24.642 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:30:24.642 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:30:24.642 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:24.642 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:24.642 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.642 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:30:24.642 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:30:24.642 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.643 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:30:24.643 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:30:24.643 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:30:24.643 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:24.643 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:24.643 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:30:24.643 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.643 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:30:24.643 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:30:24.643 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:30:24.643 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:24.643 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:24.643 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.644 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:30:24.644 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:30:24.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.622+0000 7f008f2c6640 1 -- 172.21.15.12:0/536790347 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f006c03e210 msgr2=0x7f006c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:24.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.622+0000 7f008f2c6640 1 --2- 172.21.15.12:0/536790347 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f006c03e210 0x7f006c0406d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0080034000 tx=0x7f008002f000 comp rx=0 tx=0).stop 2024-09-11T01:30:24.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.622+0000 7f008f2c6640 10 monclient: shutdown 2024-09-11T01:30:24.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.622+0000 7f008f2c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:24.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.622+0000 7f008f2c6640 1 -- 172.21.15.12:0/536790347 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f008806d670 msgr2=0x7f008813c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:24.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.622+0000 7f008f2c6640 1 --2- 172.21.15.12:0/536790347 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f008806d670 0x7f008813c090 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f008400c440 tx=0x7f008400c900 comp rx=0 tx=0).stop 2024-09-11T01:30:24.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.622+0000 7f008d863640 1 -- 172.21.15.12:0/536790347 reap_dead start 2024-09-11T01:30:24.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.622+0000 7f008d863640 1 -- 172.21.15.12:0/536790347 reap_dead start 2024-09-11T01:30:24.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.622+0000 7f008f2c6640 1 -- 172.21.15.12:0/536790347 shutdown_connections 2024-09-11T01:30:24.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.622+0000 7f008f2c6640 1 -- 172.21.15.12:0/536790347 >> 172.21.15.12:0/536790347 conn(0x7f00880a3e30 msgr2=0x7f00880a5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:24.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.622+0000 7f008f2c6640 1 -- 172.21.15.12:0/536790347 shutdown_connections 2024-09-11T01:30:24.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.622+0000 7f008f2c6640 1 -- 172.21.15.12:0/536790347 wait complete. 2024-09-11T01:30:24.646 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:30:24.646 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:30:24.646 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:30:24.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.747+0000 7f9c36d16640 10 monclient: get_monmap_and_config 2024-09-11T01:30:24.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.747+0000 7f9c36d16640 10 monclient: build_initial_monmap 2024-09-11T01:30:24.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.747+0000 7f9c36d16640 10 monclient: monmap: 2024-09-11T01:30:24.749 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:24.749 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:24.749 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:30:24.747871+0000 2024-09-11T01:30:24.749 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:30:24.747871+0000 2024-09-11T01:30:24.749 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:24.749 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.749 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:30:24.749 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:30:24.749 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:30:24.750 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:30:24.750 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:30:24.750 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.747+0000 7f9c36d16640 1 Processor -- start 2024-09-11T01:30:24.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.747+0000 7f9c36d16640 1 -- start start 2024-09-11T01:30:24.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.747+0000 7f9c36d16640 10 monclient: init 2024-09-11T01:30:24.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c36d16640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:24.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c36d16640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-11T01:30:24.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c36d16640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9c30137d50 addr v1:172.21.15.12:6789/0 2024-09-11T01:30:24.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c36d16640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c30138130 0x7f9c30138550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c36d16640 10 monclient(hunting): picked mon.noname-b con 0x7f9c30138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:24.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c36d16640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9c3013d280 0x7f9c3013f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c36d16640 10 monclient(hunting): picked mon.noname-a con 0x7f9c3013d280 addr v2:172.21.15.12:3300/0 2024-09-11T01:30:24.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c36d16640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c3009d820 con 0x7f9c30137d50 2024-09-11T01:30:24.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c36d16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:24.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c36d16640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f9c3009c650 con 0x7f9c3013d280 2024-09-11T01:30:24.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c36d16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c36d16640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f9c3009ba20 con 0x7f9c30138130 2024-09-11T01:30:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c36d16640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c36d16640 10 monclient(hunting): authenticate will time out at 831.190484s 2024-09-11T01:30:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c2ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c30138130 0x7f9c30138550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c34ab2640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9c30137d50 0x7f9c3009d110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44034/0 (socket says 172.21.15.12:44034) 2024-09-11T01:30:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c34ab2640 1 -- 172.21.15.12:0/612344950 learned_addr learned my addr 172.21.15.12:0/612344950 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:30:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c352b3640 1 --2- 172.21.15.12:0/612344950 >> v2:172.21.15.12:3300/0 conn(0x7f9c3013d280 0x7f9c3013f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c2ffff640 10 monclient(hunting): get_auth_request con 0x7f9c30138130 auth_method 0 2024-09-11T01:30:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c352b3640 10 monclient(hunting): get_auth_request con 0x7f9c3013d280 auth_method 0 2024-09-11T01:30:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c352b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c352b3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c352b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c352b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c352b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c352b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c352b3640 10 monclient(hunting): handle_auth_done global_id 14136 payload 274 2024-09-11T01:30:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c352b3640 1 -- 172.21.15.12:0/612344950 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c30138130 msgr2=0x7f9c30138550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:30:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c352b3640 1 --2- 172.21.15.12:0/612344950 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c30138130 0x7f9c30138550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c352b3640 1 -- 172.21.15.12:0/612344950 >> v1:172.21.15.12:6789/0 conn(0x7f9c30137d50 legacy=0x7f9c3009d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:30:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c352b3640 10 monclient: _finish_hunting 0 2024-09-11T01:30:24.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c352b3640 1 monclient: found mon.noname-a 2024-09-11T01:30:24.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c352b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:24.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c352b3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:30:24.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c352b3640 1 -- 172.21.15.12:0/612344950 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c3009c370 con 0x7f9c3013d280 2024-09-11T01:30:24.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.748+0000 7f9c2ffff640 1 --2- 172.21.15.12:0/612344950 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c30138130 0x7f9c30138550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:30:24.767 INFO:tasks.rados.rados.0.smithi071.stdout:126: done (2 left) 2024-09-11T01:30:24.767 INFO:tasks.rados.rados.0.smithi071.stdout:127: done (1 left) 2024-09-11T01:30:24.767 INFO:tasks.rados.rados.0.smithi071.stdout:128: done (0 left) 2024-09-11T01:30:24.768 INFO:tasks.rados.rados.0.smithi071.stdout:129: read oid 24 snap -1 2024-09-11T01:30:24.768 INFO:tasks.rados.rados.0.smithi071.stdout:129: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-11T01:30:24.768 INFO:tasks.rados.rados.0.smithi071.stdout:130: rollback oid 43 current snap is 12 2024-09-11T01:30:24.768 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 43 to 9 2024-09-11T01:30:24.768 INFO:tasks.rados.rados.0.smithi071.stdout:131: delete oid 10 current snap is 12 2024-09-11T01:30:24.768 INFO:tasks.rados.rados.0.smithi071.stdout:131: done (2 left) 2024-09-11T01:30:24.768 INFO:tasks.rados.rados.0.smithi071.stdout:132: read oid 7 snap -1 2024-09-11T01:30:24.768 INFO:tasks.rados.rados.0.smithi071.stdout:132: expect (ObjNum 6 snap 0 seq_num 6) 2024-09-11T01:30:24.768 INFO:tasks.rados.rados.0.smithi071.stdout:133: copy_from oid 37 from oid 28 current snap is 12 2024-09-11T01:30:24.768 INFO:tasks.rados.rados.0.smithi071.stdout:134: snap_remove snap 11 2024-09-11T01:30:24.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.751+0000 7f9c352b3640 1 -- 172.21.15.12:0/612344950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c3013d280 msgr2=0x7f9c3013f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:24.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.751+0000 7f9c352b3640 1 --2- 172.21.15.12:0/612344950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c3013d280 0x7f9c3013f670 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9c28009d90 tx=0x7f9c2802a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19dd15261bbb2cd0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:24.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.751+0000 7f9c2f7fe640 1 -- 172.21.15.12:0/612344950 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c28007950 con 0x7f9c3013d280 2024-09-11T01:30:24.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.751+0000 7f9c2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:24.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.751+0000 7f9c2f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:30:24.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.751+0000 7f9c2f7fe640 10 monclient: dump: 2024-09-11T01:30:24.770 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:24.770 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:24.770 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.770 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.770 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:24.771 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.771 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:24.771 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:24.771 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:24.771 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.751+0000 7f9c2f7fe640 10 monclient: _finish_auth 0 2024-09-11T01:30:24.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.751+0000 7f9c2f7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:30:24.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.751+0000 7f9c2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:24.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.751+0000 7f9c2f7fe640 1 -- 172.21.15.12:0/612344950 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c28007af0 con 0x7f9c3013d280 2024-09-11T01:30:24.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.751+0000 7f9c36d16640 5 monclient: authenticate success, global_id 14136 2024-09-11T01:30:24.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.751+0000 7f9c36d16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:30:24.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.751+0000 7f9c2f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:24.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.751+0000 7f9c36d16640 10 monclient: get_monmap_and_config success 2024-09-11T01:30:24.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.751+0000 7f9c2f7fe640 1 -- 172.21.15.12:0/612344950 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c28031400 con 0x7f9c3013d280 2024-09-11T01:30:24.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.751+0000 7f9c2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:24.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.751+0000 7f9c36d16640 4 set_mon_vals no callback set 2024-09-11T01:30:24.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.751+0000 7f9c2f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:30:24.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.751+0000 7f9c2f7fe640 10 monclient: dump: 2024-09-11T01:30:24.772 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:24.773 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:24.773 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.773 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.773 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:24.773 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.773 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:24.773 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:24.774 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:24.774 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.752+0000 7f9c36d16640 10 monclient: shutdown 2024-09-11T01:30:24.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.752+0000 7f9c36d16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:24.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.752+0000 7f9c36d16640 1 -- 172.21.15.12:0/612344950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c3013d280 msgr2=0x7f9c3013f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:24.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.752+0000 7f9c36d16640 1 --2- 172.21.15.12:0/612344950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c3013d280 0x7f9c3013f670 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9c28009d90 tx=0x7f9c2802a530 comp rx=0 tx=0).stop 2024-09-11T01:30:24.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.752+0000 7f9c36d16640 1 -- 172.21.15.12:0/612344950 shutdown_connections 2024-09-11T01:30:24.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.752+0000 7f9c36d16640 1 --2- 172.21.15.12:0/612344950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c3013d280 0x7f9c3013f670 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.752+0000 7f9c36d16640 1 --2- 172.21.15.12:0/612344950 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c30138130 0x7f9c30138550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.752+0000 7f9c36d16640 1 -- 172.21.15.12:0/612344950 >> 172.21.15.12:0/612344950 conn(0x7f9c300a7020 msgr2=0x7f9c300a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:24.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.752+0000 7f9c36d16640 1 -- 172.21.15.12:0/612344950 shutdown_connections 2024-09-11T01:30:24.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.752+0000 7f9c36d16640 1 -- 172.21.15.12:0/612344950 wait complete. 2024-09-11T01:30:24.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.753+0000 7f9c36d16640 10 monclient: build_initial_monmap 2024-09-11T01:30:24.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.753+0000 7f9c36d16640 10 monclient: monmap: 2024-09-11T01:30:24.776 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:24.776 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:30:24.776 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:30:24.776 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:30:24.776 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:30:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:30:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:30:24.777 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.753+0000 7f9c36d16640 1 Processor -- start 2024-09-11T01:30:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.753+0000 7f9c36d16640 1 -- start start 2024-09-11T01:30:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.753+0000 7f9c36d16640 10 monclient: init 2024-09-11T01:30:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c36d16640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c36d16640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:30:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c36d16640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c30137d50 0x7f9c301ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c36d16640 10 monclient(hunting): picked mon.noname-a con 0x7f9c30137d50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:30:24.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c36d16640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c30138130 0x7f9c30136ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c36d16640 10 monclient(hunting): picked mon.noname-c con 0x7f9c30138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:24.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c36d16640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c3013d280 0x7f9c3013a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c36d16640 10 monclient(hunting): picked mon.noname-b con 0x7f9c3013d280 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:30:24.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c36d16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:24.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c36d16640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3006f850 con 0x7f9c30137d50 2024-09-11T01:30:24.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c36d16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:24.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c36d16640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3006f6d0 con 0x7f9c3013d280 2024-09-11T01:30:24.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c36d16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:24.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c36d16640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f9c3013fbb0 con 0x7f9c30138130 2024-09-11T01:30:24.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c36d16640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:24.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c36d16640 10 monclient(hunting): authenticate will time out at 831.196197s 2024-09-11T01:30:24.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c34ab2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c30137d50 0x7f9c301ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c34ab2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c30137d50 0x7f9c301ccee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41128/0 (socket says 172.21.15.12:41128) 2024-09-11T01:30:24.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c2ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c30138130 0x7f9c30136ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c34ab2640 1 -- 172.21.15.12:0/1515306519 learned_addr learned my addr 172.21.15.12:0/1515306519 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:30:24.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c34ab2640 10 monclient(hunting): get_auth_request con 0x7f9c30137d50 auth_method 0 2024-09-11T01:30:24.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c34ab2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:24.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c34ab2640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:24.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c34ab2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:24.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c2ffff640 10 monclient(hunting): get_auth_request con 0x7f9c30138130 auth_method 0 2024-09-11T01:30:24.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:24.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:24.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:24.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c34ab2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:24.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c34ab2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:24.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c34ab2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:24.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:24.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:24.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c352b3640 1 --2- 172.21.15.12:0/1515306519 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c3013d280 0x7f9c3013a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:24.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c352b3640 10 monclient(hunting): get_auth_request con 0x7f9c3013d280 auth_method 0 2024-09-11T01:30:24.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c352b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:24.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c352b3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:24.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c352b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:24.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c352b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:24.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c352b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:24.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c352b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:24.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c34ab2640 10 monclient(hunting): handle_auth_done global_id 14139 payload 824 2024-09-11T01:30:24.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c34ab2640 1 -- 172.21.15.12:0/1515306519 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c30138130 msgr2=0x7f9c30136ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:24.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c34ab2640 1 --2- 172.21.15.12:0/1515306519 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c30138130 0x7f9c30136ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c34ab2640 1 -- 172.21.15.12:0/1515306519 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c3013d280 msgr2=0x7f9c3013a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:24.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c34ab2640 1 --2- 172.21.15.12:0/1515306519 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c3013d280 0x7f9c3013a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c34ab2640 10 monclient: _finish_hunting 0 2024-09-11T01:30:24.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c34ab2640 1 monclient: found mon.noname-a 2024-09-11T01:30:24.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c34ab2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:24.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c34ab2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:30:24.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c34ab2640 1 -- 172.21.15.12:0/1515306519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c3013c760 con 0x7f9c30137d50 2024-09-11T01:30:24.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.754+0000 7f9c2ffff640 1 --2- 172.21.15.12:0/1515306519 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c30138130 0x7f9c30136ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:30:24.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.756+0000 7f9c34ab2640 1 -- 172.21.15.12:0/1515306519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c30137d50 msgr2=0x7f9c301ccee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:24.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.756+0000 7f9c34ab2640 1 --2- 172.21.15.12:0/1515306519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c30137d50 0x7f9c301ccee0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9c2000c490 tx=0x7f9c2000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:24.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.757+0000 7f9c2d7fa640 1 -- 172.21.15.12:0/1515306519 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c20007070 con 0x7f9c30137d50 2024-09-11T01:30:24.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.757+0000 7f9c2d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:24.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.757+0000 7f9c2d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:30:24.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.757+0000 7f9c2d7fa640 10 monclient: dump: 2024-09-11T01:30:24.786 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:24.786 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:24.786 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.787 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.787 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:24.787 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.787 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:24.787 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:24.787 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:24.788 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.757+0000 7f9c2d7fa640 10 monclient: _finish_auth 0 2024-09-11T01:30:24.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.757+0000 7f9c2d7fa640 10 monclient: _check_auth_tickets 2024-09-11T01:30:24.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.757+0000 7f9c2d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:24.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.757+0000 7f9c2d7fa640 1 -- 172.21.15.12:0/1515306519 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c20007230 con 0x7f9c30137d50 2024-09-11T01:30:24.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.757+0000 7f9c36d16640 5 monclient: authenticate success, global_id 14139 2024-09-11T01:30:24.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.757+0000 7f9c36d16640 10 monclient: _renew_subs 2024-09-11T01:30:24.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.757+0000 7f9c36d16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:30:24.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.757+0000 7f9c36d16640 1 -- 172.21.15.12:0/1515306519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c30120fc0 con 0x7f9c30137d50 2024-09-11T01:30:24.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.757+0000 7f9c2d7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:24.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.757+0000 7f9c2d7fa640 1 -- 172.21.15.12:0/1515306519 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c20007620 con 0x7f9c30137d50 2024-09-11T01:30:24.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.757+0000 7f9c2d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:24.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.757+0000 7f9c2d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:30:24.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.757+0000 7f9c2d7fa640 10 monclient: dump: 2024-09-11T01:30:24.790 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:24.790 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:24.790 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.790 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.790 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:24.791 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.791 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:24.791 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:24.791 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:24.791 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.757+0000 7f9c2effd640 4 set_mon_vals no callback set 2024-09-11T01:30:24.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.757+0000 7f9c36d16640 10 monclient: _renew_subs 2024-09-11T01:30:24.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.757+0000 7f9c36d16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:30:24.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.757+0000 7f9c36d16640 1 -- 172.21.15.12:0/1515306519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c3013bc00 con 0x7f9c30137d50 2024-09-11T01:30:24.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.758+0000 7f9c36d16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:24.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.758+0000 7f9c36d16640 1 --2- 172.21.15.12:0/1515306519 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c30137410 0x7f9c3013a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.758+0000 7f9c2d7fa640 1 -- 172.21.15.12:0/1515306519 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f9c2001b400 con 0x7f9c30137d50 2024-09-11T01:30:24.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.758+0000 7f9c36d16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:24.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.758+0000 7f9c36d16640 1 -- 172.21.15.12:0/1515306519 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f9c30138550 con 0x7f9c30137410 2024-09-11T01:30:24.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.758+0000 7f9c2ffff640 1 --2- 172.21.15.12:0/1515306519 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c30137410 0x7f9c3013a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.758+0000 7f9c2ffff640 10 monclient: get_auth_request con 0x7f9c30137410 auth_method 0 2024-09-11T01:30:24.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.758+0000 7f9c2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:24.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.758+0000 7f9c2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:24.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.758+0000 7f9c2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:24.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.758+0000 7f9c2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:24.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.758+0000 7f9c2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:24.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.758+0000 7f9c2d7fa640 1 --2- 172.21.15.12:0/1515306519 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9c1c03e260 0x7f9c1c040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.758+0000 7f9c2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:24.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.758+0000 7f9c2d7fa640 1 -- 172.21.15.12:0/1515306519 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4739+0+0 (secure 0 0 0) 0x7f9c200529f0 con 0x7f9c30137d50 2024-09-11T01:30:24.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.758+0000 7f9c352b3640 1 --2- 172.21.15.12:0/1515306519 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9c1c03e260 0x7f9c1c040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.758+0000 7f9c2ffff640 10 monclient(hunting): handle_auth_done global_id 4472 payload 824 2024-09-11T01:30:24.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.758+0000 7f9c2ffff640 1 -- 172.21.15.12:0/1515306519 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9c3006a250 con 0x7f9c30137410 2024-09-11T01:30:24.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.758+0000 7f9c352b3640 10 monclient: get_auth_request con 0x7f9c1c03e260 auth_method 0 2024-09-11T01:30:24.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.761+0000 7f9c2ffff640 1 -- 172.21.15.12:0/1515306519 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c30137410 msgr2=0x7f9c3013a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:24.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.761+0000 7f9c2ffff640 1 --2- 172.21.15.12:0/1515306519 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c30137410 0x7f9c3013a6c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9c24011d20 tx=0x7f9c2400f610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:24.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.761+0000 7f9c2d7fa640 1 -- 172.21.15.12:0/1515306519 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c24019070 con 0x7f9c30137410 2024-09-11T01:30:24.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.761+0000 7f9c2d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:24.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.761+0000 7f9c2d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:24.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.761+0000 7f9c2d7fa640 10 monclient: dump: 2024-09-11T01:30:24.796 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:24.796 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:24.796 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.796 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.797 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:24.797 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.797 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:24.797 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:24.797 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:24.797 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.761+0000 7f9c352b3640 1 --2- 172.21.15.12:0/1515306519 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9c1c03e260 0x7f9c1c040720 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9c3006d4f0 tx=0x7f9c2802f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:24.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.761+0000 7f9c2d7fa640 1 -- 172.21.15.12:0/1515306519 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9c24004240 con 0x7f9c30137410 2024-09-11T01:30:24.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.761+0000 7f9c2d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:24.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.761+0000 7f9c2d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:30:24.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.761+0000 7f9c2d7fa640 1 -- 172.21.15.12:0/1515306519 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c30137410 msgr2=0x7f9c3013a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:24.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.761+0000 7f9c2d7fa640 1 --2- 172.21.15.12:0/1515306519 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c30137410 0x7f9c3013a6c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9c24011d20 tx=0x7f9c2400f610 comp rx=0 tx=0).stop 2024-09-11T01:30:24.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.761+0000 7f9c2d7fa640 1 -- 172.21.15.12:0/1515306519 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c30137410 msgr2=0x7f9c3013a6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:30:24.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.761+0000 7f9c2d7fa640 1 --2- 172.21.15.12:0/1515306519 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c30137410 0x7f9c3013a6c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.770+0000 7f9c36d16640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:30:24.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.770+0000 7f9c36d16640 1 --2- 172.21.15.12:0/1515306519 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c301211c0 0x7f9c3013ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.770+0000 7f9c36d16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:24.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.770+0000 7f9c36d16640 1 -- 172.21.15.12:0/1515306519 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f9c30120b70 con 0x7f9c301211c0 2024-09-11T01:30:24.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.770+0000 7f9c2ffff640 1 --2- 172.21.15.12:0/1515306519 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c301211c0 0x7f9c3013ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.770+0000 7f9c2ffff640 10 monclient: get_auth_request con 0x7f9c301211c0 auth_method 0 2024-09-11T01:30:24.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.770+0000 7f9c2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:24.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.770+0000 7f9c2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:24.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.770+0000 7f9c2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:24.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.770+0000 7f9c2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:24.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.770+0000 7f9c2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:24.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.770+0000 7f9c2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:24.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.770+0000 7f9c2ffff640 10 monclient(hunting): handle_auth_done global_id 4475 payload 824 2024-09-11T01:30:24.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.770+0000 7f9c2ffff640 1 -- 172.21.15.12:0/1515306519 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9c301217c0 con 0x7f9c301211c0 2024-09-11T01:30:24.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.773+0000 7f9c2ffff640 1 -- 172.21.15.12:0/1515306519 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c301211c0 msgr2=0x7f9c3013ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:24.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.773+0000 7f9c2ffff640 1 --2- 172.21.15.12:0/1515306519 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c301211c0 0x7f9c3013ac00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9c2401b000 tx=0x7f9c240046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:24.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.773+0000 7f9c2d7fa640 1 -- 172.21.15.12:0/1515306519 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c24019040 con 0x7f9c301211c0 2024-09-11T01:30:24.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.773+0000 7f9c2d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:24.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.773+0000 7f9c2d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:24.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.773+0000 7f9c2d7fa640 10 monclient: dump: 2024-09-11T01:30:24.806 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:24.806 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:24.806 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.806 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.807 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:24.807 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.807 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:24.807 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:24.807 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:24.807 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.773+0000 7f9c2d7fa640 1 -- 172.21.15.12:0/1515306519 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4707 (secure 0 0 0) 0x7f9c2401e450 con 0x7f9c301211c0 2024-09-11T01:30:24.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.773+0000 7f9c2d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:30:24.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.773+0000 7f9c2d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:30:24.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.773+0000 7f9c2d7fa640 1 -- 172.21.15.12:0/1515306519 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c301211c0 msgr2=0x7f9c3013ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:24.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.773+0000 7f9c2d7fa640 1 --2- 172.21.15.12:0/1515306519 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c301211c0 0x7f9c3013ac00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9c2401b000 tx=0x7f9c240046c0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.773+0000 7f9c2d7fa640 1 -- 172.21.15.12:0/1515306519 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c301211c0 msgr2=0x7f9c3013ac00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:30:24.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.773+0000 7f9c2d7fa640 1 --2- 172.21.15.12:0/1515306519 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9c301211c0 0x7f9c3013ac00 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9c2401b000 tx=0x7f9c240046c0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.808 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:30:24.809 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:30:24.809 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:30:24.809 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:30:24.809 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 14, 2024-09-11T01:30:24.809 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 87462, 2024-09-11T01:30:24.809 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:30:24.809 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:30:24.809 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:30:24.810 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:30:24.810 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:24.810 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:30:24.810 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:30:24.810 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:30:24.810 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:30:24.810 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:30:24.811 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:30:24.811 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:30:24.811 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:30:24.811 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:30:24.812 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:30:24.812 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:30:24.812 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:30:24.812 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:30:24.812 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:30:24.812 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:30:24.812 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:24.813 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:30:24.813 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:30:24.813 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:30:24.813 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:30:24.813 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:30:24.813 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:30:24.813 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:30:24.813 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:30:24.813 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:30:24.813 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:30:24.814 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:30:24.814 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:30:24.814 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:30:24.814 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:24.814 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.814 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:30:24.814 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:30:24.814 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:30:24.815 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:30:24.815 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:30:24.815 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:30:24.815 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:30:24.815 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:30:24.815 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:30:24.815 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:30:24.815 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:30:24.815 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:30:24.815 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:30:24.815 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:30:24.816 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:30:24.816 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:30:24.816 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:30:24.816 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:30:24.816 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:30:24.816 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:30:24.816 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:30:24.816 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:30:24.816 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:30:24.816 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:30:24.816 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:30:24.817 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:30:24.817 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:30:24.817 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:30:24.817 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:24.817 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:30:24.817 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.817 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:30:24.817 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.817 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:30:24.817 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:30:24.817 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:30:24.817 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:30:24.818 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.818 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:30:24.818 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:30:24.818 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:24.818 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.818 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.818 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:30:24.818 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:30:24.818 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:24.818 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:24.818 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:24.818 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.818 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:30:24.819 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:30:24.819 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:30:24.819 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:30:24.819 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:30:24.819 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.819 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.819 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:30:24.820 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:30:24.820 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:30:24.820 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:30:24.820 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.820 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:30:24.820 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:30:24.821 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:24.821 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.821 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.821 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:30:24.821 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:30:24.821 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:24.821 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:24.822 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:24.822 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.822 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:30:24.822 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:30:24.822 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:30:24.822 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:30:24.823 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:30:24.823 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.823 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.823 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:30:24.824 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:30:24.824 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:30:24.824 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:30:24.824 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.824 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:30:24.824 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:30:24.825 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:24.825 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.825 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.825 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:30:24.825 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:30:24.825 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:24.826 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:24.826 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:24.826 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.826 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:30:24.826 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:30:24.827 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:30:24.827 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:30:24.827 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:30:24.827 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:24.827 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:24.827 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.828 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:30:24.828 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:30:24.828 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.828 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:30:24.828 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:30:24.828 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:30:24.829 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:24.829 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:24.829 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:30:24.829 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.829 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:30:24.829 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:30:24.830 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:30:24.830 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:24.830 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:24.830 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:30:24.830 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.830 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:30:24.830 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:30:24.831 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:30:24.831 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:24.831 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:24.831 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.831 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:30:24.831 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:30:24.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.774+0000 7f9c36d16640 1 -- 172.21.15.12:0/1515306519 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9c1c03e260 msgr2=0x7f9c1c040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:24.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.774+0000 7f9c36d16640 1 --2- 172.21.15.12:0/1515306519 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9c1c03e260 0x7f9c1c040720 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9c3006d4f0 tx=0x7f9c2802f000 comp rx=0 tx=0).stop 2024-09-11T01:30:24.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.774+0000 7f9c36d16640 10 monclient: shutdown 2024-09-11T01:30:24.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.774+0000 7f9c36d16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:24.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.774+0000 7f9c36d16640 1 -- 172.21.15.12:0/1515306519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c30137d50 msgr2=0x7f9c301ccee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:24.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.774+0000 7f9c36d16640 1 --2- 172.21.15.12:0/1515306519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c30137d50 0x7f9c301ccee0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9c2000c490 tx=0x7f9c2000c950 comp rx=0 tx=0).stop 2024-09-11T01:30:24.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.774+0000 7f9c352b3640 1 -- 172.21.15.12:0/1515306519 reap_dead start 2024-09-11T01:30:24.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.774+0000 7f9c352b3640 1 -- 172.21.15.12:0/1515306519 reap_dead start 2024-09-11T01:30:24.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.774+0000 7f9c36d16640 1 -- 172.21.15.12:0/1515306519 shutdown_connections 2024-09-11T01:30:24.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.774+0000 7f9c36d16640 1 -- 172.21.15.12:0/1515306519 >> 172.21.15.12:0/1515306519 conn(0x7f9c300a7020 msgr2=0x7f9c3013dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.774+0000 7f9c36d16640 1 -- 172.21.15.12:0/1515306519 shutdown_connections 2024-09-11T01:30:24.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.774+0000 7f9c36d16640 1 -- 172.21.15.12:0/1515306519 wait complete. 2024-09-11T01:30:24.836 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:30:24.837 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:30:24.837 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:30:24.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.938+0000 7f6e7f9a3640 10 monclient: get_monmap_and_config 2024-09-11T01:30:24.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.938+0000 7f6e7f9a3640 10 monclient: build_initial_monmap 2024-09-11T01:30:24.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.938+0000 7f6e7f9a3640 10 monclient: monmap: 2024-09-11T01:30:24.940 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:24.940 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:24.940 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:30:24.938831+0000 2024-09-11T01:30:24.940 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:30:24.938831+0000 2024-09-11T01:30:24.940 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:24.940 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.940 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:30:24.940 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:30:24.940 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:30:24.940 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:30:24.940 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:30:24.940 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.938+0000 7f6e7f9a3640 1 Processor -- start 2024-09-11T01:30:24.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.938+0000 7f6e7f9a3640 1 -- start start 2024-09-11T01:30:24.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.938+0000 7f6e7f9a3640 10 monclient: init 2024-09-11T01:30:24.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7f9a3640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:24.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7f9a3640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-11T01:30:24.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7f9a3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6e78137e50 addr v1:172.21.15.12:6789/0 2024-09-11T01:30:24.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7f9a3640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6e78138230 0x7f6e78138640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7f9a3640 10 monclient(hunting): picked mon.noname-a con 0x7f6e78138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:30:24.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7f9a3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6e7813d380 addr v1:172.21.15.71:6789/0 2024-09-11T01:30:24.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7f9a3640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e780a62d0 con 0x7f6e78137e50 2024-09-11T01:30:24.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7f9a3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e7809bb40 con 0x7f6e7813d380 2024-09-11T01:30:24.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7f9a3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:24.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7f9a3640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f6e7809c4a0 con 0x7f6e78138230 2024-09-11T01:30:24.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7f9a3640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:24.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7f9a3640 10 monclient(hunting): authenticate will time out at 831.381443s 2024-09-11T01:30:24.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7cf3e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6e78138230 0x7f6e78138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7d73f640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6e78137e50 0x7f6e7809d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44046/0 (socket says 172.21.15.12:44046) 2024-09-11T01:30:24.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7d73f640 1 -- 172.21.15.12:0/3362344032 learned_addr learned my addr 172.21.15.12:0/3362344032 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:30:24.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7cf3e640 10 monclient(hunting): get_auth_request con 0x7f6e78138230 auth_method 0 2024-09-11T01:30:24.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7cf3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:24.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7cf3e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:24.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7cf3e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:24.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7cf3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:24.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7cf3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:24.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7cf3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:24.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7cf3e640 10 monclient(hunting): handle_auth_done global_id 14142 payload 274 2024-09-11T01:30:24.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7cf3e640 1 -- 172.21.15.12:0/3362344032 >> v1:172.21.15.71:6789/0 conn(0x7f6e7813d380 legacy=0x7f6e7813f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:30:24.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7cf3e640 1 -- 172.21.15.12:0/3362344032 >> v1:172.21.15.12:6789/0 conn(0x7f6e78137e50 legacy=0x7f6e7809d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:30:24.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7cf3e640 10 monclient: _finish_hunting 0 2024-09-11T01:30:24.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7cf3e640 1 monclient: found mon.noname-a 2024-09-11T01:30:24.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7cf3e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:24.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7cf3e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:30:24.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.939+0000 7f6e7cf3e640 1 -- 172.21.15.12:0/3362344032 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e7809c780 con 0x7f6e78138230 2024-09-11T01:30:24.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.942+0000 7f6e7cf3e640 1 -- 172.21.15.12:0/3362344032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e78138230 msgr2=0x7f6e78138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:24.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.942+0000 7f6e7cf3e640 1 --2- 172.21.15.12:0/3362344032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e78138230 0x7f6e78138640 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6e780a6f80 tx=0x7f6e68029fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41813c35112bd01c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:24.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.943+0000 7f6e6ffff640 1 -- 172.21.15.12:0/3362344032 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e68037070 con 0x7f6e78138230 2024-09-11T01:30:24.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.943+0000 7f6e6ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:24.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.943+0000 7f6e6ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:30:24.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.943+0000 7f6e6ffff640 10 monclient: dump: 2024-09-11T01:30:24.949 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:24.949 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:24.949 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.949 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.952 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:24.952 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.952 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:24.952 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:24.952 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:24.952 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.943+0000 7f6e6ffff640 10 monclient: _finish_auth 0 2024-09-11T01:30:24.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.943+0000 7f6e6ffff640 10 monclient: _check_auth_tickets 2024-09-11T01:30:24.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.943+0000 7f6e6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:24.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.943+0000 7f6e6ffff640 1 -- 172.21.15.12:0/3362344032 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e68032440 con 0x7f6e78138230 2024-09-11T01:30:24.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.943+0000 7f6e7f9a3640 5 monclient: authenticate success, global_id 14142 2024-09-11T01:30:24.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.943+0000 7f6e7f9a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:30:24.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.943+0000 7f6e6ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:24.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.943+0000 7f6e6ffff640 1 -- 172.21.15.12:0/3362344032 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e68032810 con 0x7f6e78138230 2024-09-11T01:30:24.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.943+0000 7f6e7f9a3640 10 monclient: get_monmap_and_config success 2024-09-11T01:30:24.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.943+0000 7f6e7f9a3640 4 set_mon_vals no callback set 2024-09-11T01:30:24.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.943+0000 7f6e6ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:24.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.943+0000 7f6e6ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:30:24.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.943+0000 7f6e6ffff640 10 monclient: dump: 2024-09-11T01:30:24.953 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:24.953 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:24.953 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.953 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.953 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:24.954 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.954 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:24.954 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:24.954 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:24.954 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.943+0000 7f6e7f9a3640 10 monclient: shutdown 2024-09-11T01:30:24.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.943+0000 7f6e7f9a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:24.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.943+0000 7f6e7f9a3640 1 -- 172.21.15.12:0/3362344032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e78138230 msgr2=0x7f6e78138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:24.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.943+0000 7f6e7f9a3640 1 --2- 172.21.15.12:0/3362344032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e78138230 0x7f6e78138640 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6e780a6f80 tx=0x7f6e68029fb0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.943+0000 7f6e7f9a3640 1 -- 172.21.15.12:0/3362344032 shutdown_connections 2024-09-11T01:30:24.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.943+0000 7f6e7f9a3640 1 --2- 172.21.15.12:0/3362344032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e78138230 0x7f6e78138640 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.943+0000 7f6e7f9a3640 1 -- 172.21.15.12:0/3362344032 >> 172.21.15.12:0/3362344032 conn(0x7f6e780a7020 msgr2=0x7f6e780a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:24.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.943+0000 7f6e7f9a3640 1 -- 172.21.15.12:0/3362344032 shutdown_connections 2024-09-11T01:30:24.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.943+0000 7f6e7f9a3640 1 -- 172.21.15.12:0/3362344032 wait complete. 2024-09-11T01:30:24.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.944+0000 7f6e7f9a3640 10 monclient: build_initial_monmap 2024-09-11T01:30:24.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.944+0000 7f6e7f9a3640 10 monclient: monmap: 2024-09-11T01:30:24.955 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:24.955 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:30:24.955 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:30:24.955 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:30:24.955 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:24.955 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.955 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:30:24.955 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:30:24.955 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:30:24.955 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.944+0000 7f6e7f9a3640 1 Processor -- start 2024-09-11T01:30:24.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.944+0000 7f6e7f9a3640 1 -- start start 2024-09-11T01:30:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.944+0000 7f6e7f9a3640 10 monclient: init 2024-09-11T01:30:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7f9a3640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7f9a3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:30:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7f9a3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e78137e50 0x7f6e7813fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7f9a3640 10 monclient(hunting): picked mon.noname-b con 0x7f6e78137e50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:30:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7f9a3640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6e78138230 0x7f6e780a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7f9a3640 10 monclient(hunting): picked mon.noname-c con 0x7f6e78138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7f9a3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e7813d380 0x7f6e780a1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7f9a3640 10 monclient(hunting): picked mon.noname-a con 0x7f6e7813d380 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:30:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7f9a3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7f9a3640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7806f890 con 0x7f6e7813d380 2024-09-11T01:30:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7f9a3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7f9a3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7806f710 con 0x7f6e78137e50 2024-09-11T01:30:24.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7f9a3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:24.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7f9a3640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6e78140e90 con 0x7f6e78138230 2024-09-11T01:30:24.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7f9a3640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:24.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7f9a3640 10 monclient(hunting): authenticate will time out at 831.387156s 2024-09-11T01:30:24.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7df40640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e7813d380 0x7f6e780a1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7cf3e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6e78138230 0x7f6e780a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7df40640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e7813d380 0x7f6e780a1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41138/0 (socket says 172.21.15.12:41138) 2024-09-11T01:30:24.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7df40640 1 -- 172.21.15.12:0/3355750825 learned_addr learned my addr 172.21.15.12:0/3355750825 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:30:24.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7cf3e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6e78138230 0x7f6e780a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:40726/0 (socket says 172.21.15.12:40726) 2024-09-11T01:30:24.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7df40640 10 monclient(hunting): get_auth_request con 0x7f6e7813d380 auth_method 0 2024-09-11T01:30:24.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7df40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:24.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7df40640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:24.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7df40640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:24.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7cf3e640 10 monclient(hunting): get_auth_request con 0x7f6e78138230 auth_method 0 2024-09-11T01:30:24.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7cf3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:24.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7cf3e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:24.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7cf3e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:24.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7df40640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:24.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7df40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:24.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7df40640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:24.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7cf3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:24.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7cf3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:24.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7cf3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:24.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7d73f640 1 --2- 172.21.15.12:0/3355750825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e78137e50 0x7f6e7813fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7d73f640 10 monclient(hunting): get_auth_request con 0x7f6e78137e50 auth_method 0 2024-09-11T01:30:24.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7d73f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:24.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7d73f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:24.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7d73f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:24.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7df40640 10 monclient(hunting): handle_auth_done global_id 14145 payload 824 2024-09-11T01:30:24.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7df40640 1 -- 172.21.15.12:0/3355750825 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6e78138230 msgr2=0x7f6e780a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:24.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7df40640 1 --2- 172.21.15.12:0/3355750825 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6e78138230 0x7f6e780a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7df40640 1 -- 172.21.15.12:0/3355750825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e78137e50 msgr2=0x7f6e7813fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:24.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7df40640 1 --2- 172.21.15.12:0/3355750825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e78137e50 0x7f6e7813fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7df40640 10 monclient: _finish_hunting 0 2024-09-11T01:30:24.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7df40640 1 monclient: found mon.noname-a 2024-09-11T01:30:24.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7df40640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:24.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7df40640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:30:24.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7df40640 1 -- 172.21.15.12:0/3355750825 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e78140af0 con 0x7f6e7813d380 2024-09-11T01:30:24.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.945+0000 7f6e7d73f640 1 --2- 172.21.15.12:0/3355750825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e78137e50 0x7f6e7813fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:30:24.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.947+0000 7f6e7df40640 1 -- 172.21.15.12:0/3355750825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e7813d380 msgr2=0x7f6e780a1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:24.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.947+0000 7f6e7df40640 1 --2- 172.21.15.12:0/3355750825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e7813d380 0x7f6e780a1fa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6e7000c780 tx=0x7f6e7000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:24.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.947+0000 7f6e6dffb640 1 -- 172.21.15.12:0/3355750825 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7000ac60 con 0x7f6e7813d380 2024-09-11T01:30:24.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.947+0000 7f6e6dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:24.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.947+0000 7f6e6dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:30:24.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.947+0000 7f6e6dffb640 10 monclient: dump: 2024-09-11T01:30:24.964 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:24.965 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:24.965 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.965 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.965 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:24.965 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.965 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:24.965 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:24.965 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:24.965 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.947+0000 7f6e6dffb640 10 monclient: _finish_auth 0 2024-09-11T01:30:24.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.947+0000 7f6e6dffb640 10 monclient: _check_auth_tickets 2024-09-11T01:30:24.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.947+0000 7f6e6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:24.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.947+0000 7f6e7f9a3640 5 monclient: authenticate success, global_id 14145 2024-09-11T01:30:24.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.947+0000 7f6e6dffb640 1 -- 172.21.15.12:0/3355750825 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e7000ae00 con 0x7f6e7813d380 2024-09-11T01:30:24.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.947+0000 7f6e6dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:24.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.948+0000 7f6e7f9a3640 10 monclient: _renew_subs 2024-09-11T01:30:24.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.948+0000 7f6e6dffb640 1 -- 172.21.15.12:0/3355750825 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e70004ba0 con 0x7f6e7813d380 2024-09-11T01:30:24.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.948+0000 7f6e7f9a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:30:24.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.948+0000 7f6e7f9a3640 1 -- 172.21.15.12:0/3355750825 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e781403e0 con 0x7f6e7813d380 2024-09-11T01:30:24.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.948+0000 7f6e6dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:24.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.948+0000 7f6e6dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:30:24.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.948+0000 7f6e6dffb640 10 monclient: dump: 2024-09-11T01:30:24.966 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:24.966 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:24.967 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.967 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.967 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:24.967 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.967 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:24.967 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:24.967 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:24.967 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.948+0000 7f6e6f7fe640 4 set_mon_vals no callback set 2024-09-11T01:30:24.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.948+0000 7f6e7f9a3640 10 monclient: _renew_subs 2024-09-11T01:30:24.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.948+0000 7f6e7f9a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:30:24.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.948+0000 7f6e7f9a3640 1 -- 172.21.15.12:0/3355750825 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e78137670 con 0x7f6e7813d380 2024-09-11T01:30:24.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.949+0000 7f6e6dffb640 1 -- 172.21.15.12:0/3355750825 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f6e70005030 con 0x7f6e7813d380 2024-09-11T01:30:24.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.949+0000 7f6e7f9a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:24.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.949+0000 7f6e7f9a3640 1 --2- 172.21.15.12:0/3355750825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e780a24e0 0x7f6e781cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.949+0000 7f6e7f9a3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:24.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.949+0000 7f6e7f9a3640 1 -- 172.21.15.12:0/3355750825 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6e78138640 con 0x7f6e780a24e0 2024-09-11T01:30:24.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.949+0000 7f6e6dffb640 1 --2- 172.21.15.12:0/3355750825 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6e5c03e210 0x7f6e5c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.949+0000 7f6e6dffb640 1 -- 172.21.15.12:0/3355750825 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4739+0+0 (secure 0 0 0) 0x7f6e70052f70 con 0x7f6e7813d380 2024-09-11T01:30:24.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.949+0000 7f6e7d73f640 1 --2- 172.21.15.12:0/3355750825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e780a24e0 0x7f6e781cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.949+0000 7f6e7d73f640 10 monclient: get_auth_request con 0x7f6e780a24e0 auth_method 0 2024-09-11T01:30:24.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.949+0000 7f6e7d73f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:24.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.949+0000 7f6e7d73f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:24.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.949+0000 7f6e7d73f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:24.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.949+0000 7f6e7d73f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:24.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.949+0000 7f6e7d73f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:24.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.949+0000 7f6e7cf3e640 1 --2- 172.21.15.12:0/3355750825 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6e5c03e210 0x7f6e5c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.949+0000 7f6e7d73f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:24.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.949+0000 7f6e7cf3e640 10 monclient: get_auth_request con 0x7f6e5c03e210 auth_method 0 2024-09-11T01:30:24.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.949+0000 7f6e7d73f640 10 monclient(hunting): handle_auth_done global_id 4396 payload 824 2024-09-11T01:30:24.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.949+0000 7f6e7d73f640 1 -- 172.21.15.12:0/3355750825 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6e780a43e0 con 0x7f6e780a24e0 2024-09-11T01:30:24.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.952+0000 7f6e7d73f640 1 -- 172.21.15.12:0/3355750825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e780a24e0 msgr2=0x7f6e781cd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:24.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.952+0000 7f6e7d73f640 1 --2- 172.21.15.12:0/3355750825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e780a24e0 0x7f6e781cd1d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6e7400e520 tx=0x7f6e7400e9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:24.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.952+0000 7f6e7cf3e640 1 --2- 172.21.15.12:0/3355750825 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6e5c03e210 0x7f6e5c0406d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6e68031000 tx=0x7f6e6802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:24.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.952+0000 7f6e6dffb640 1 -- 172.21.15.12:0/3355750825 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e74005cf0 con 0x7f6e780a24e0 2024-09-11T01:30:24.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.952+0000 7f6e6dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:24.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.952+0000 7f6e6dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:30:24.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.952+0000 7f6e6dffb640 10 monclient: dump: 2024-09-11T01:30:24.969 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:24.969 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:24.969 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.970 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.970 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:24.970 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.970 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:24.970 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:24.970 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:24.970 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.952+0000 7f6e6dffb640 1 -- 172.21.15.12:0/3355750825 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6e74018070 con 0x7f6e780a24e0 2024-09-11T01:30:24.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.952+0000 7f6e6dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:24.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.952+0000 7f6e6dffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:30:24.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.952+0000 7f6e6dffb640 1 -- 172.21.15.12:0/3355750825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e780a24e0 msgr2=0x7f6e781cd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:24.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.952+0000 7f6e6dffb640 1 --2- 172.21.15.12:0/3355750825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e780a24e0 0x7f6e781cd1d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6e7400e520 tx=0x7f6e7400e9e0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.952+0000 7f6e6dffb640 1 -- 172.21.15.12:0/3355750825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e780a24e0 msgr2=0x7f6e781cd1d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:30:24.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.952+0000 7f6e6dffb640 1 --2- 172.21.15.12:0/3355750825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e780a24e0 0x7f6e781cd1d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.958+0000 7f6e7f9a3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:30:24.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.958+0000 7f6e7f9a3640 1 --2- 172.21.15.12:0/3355750825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e781cd710 0x7f6e781cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:24.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.958+0000 7f6e7f9a3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:24.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.958+0000 7f6e7f9a3640 1 -- 172.21.15.12:0/3355750825 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6e781ce060 con 0x7f6e781cd710 2024-09-11T01:30:24.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.959+0000 7f6e7d73f640 1 --2- 172.21.15.12:0/3355750825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e781cd710 0x7f6e781cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:24.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.959+0000 7f6e7d73f640 10 monclient: get_auth_request con 0x7f6e781cd710 auth_method 0 2024-09-11T01:30:24.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.959+0000 7f6e7d73f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:24.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.959+0000 7f6e7d73f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:24.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.959+0000 7f6e7d73f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:24.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.959+0000 7f6e7d73f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:24.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.959+0000 7f6e7d73f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:24.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.959+0000 7f6e7d73f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:24.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.959+0000 7f6e7d73f640 10 monclient(hunting): handle_auth_done global_id 4399 payload 824 2024-09-11T01:30:24.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.959+0000 7f6e7d73f640 1 -- 172.21.15.12:0/3355750825 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6e78069950 con 0x7f6e781cd710 2024-09-11T01:30:24.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.961+0000 7f6e7d73f640 1 -- 172.21.15.12:0/3355750825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e781cd710 msgr2=0x7f6e781cdb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:24.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.961+0000 7f6e7d73f640 1 --2- 172.21.15.12:0/3355750825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e781cd710 0x7f6e781cdb20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6e74017000 tx=0x7f6e74004940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:24.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.962+0000 7f6e6dffb640 1 -- 172.21.15.12:0/3355750825 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e740074f0 con 0x7f6e781cd710 2024-09-11T01:30:24.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.962+0000 7f6e6dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:24.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.962+0000 7f6e6dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:30:24.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.962+0000 7f6e6dffb640 10 monclient: dump: 2024-09-11T01:30:24.972 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:24.972 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:24.972 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.972 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:24.972 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:24.972 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:24.972 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:24.973 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:24.973 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:24.973 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:24.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.962+0000 7f6e6dffb640 1 -- 172.21.15.12:0/3355750825 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4874 (secure 0 0 0) 0x7f6e740074f0 con 0x7f6e781cd710 2024-09-11T01:30:24.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.962+0000 7f6e6dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:30:24.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.962+0000 7f6e6dffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:30:24.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.962+0000 7f6e6dffb640 1 -- 172.21.15.12:0/3355750825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e781cd710 msgr2=0x7f6e781cdb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:24.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.962+0000 7f6e6dffb640 1 --2- 172.21.15.12:0/3355750825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e781cd710 0x7f6e781cdb20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6e74017000 tx=0x7f6e74004940 comp rx=0 tx=0).stop 2024-09-11T01:30:24.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.962+0000 7f6e6dffb640 1 -- 172.21.15.12:0/3355750825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e781cd710 msgr2=0x7f6e781cdb20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:30:24.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.962+0000 7f6e6dffb640 1 --2- 172.21.15.12:0/3355750825 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e781cd710 0x7f6e781cdb20 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:24.974 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:30:24.974 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:30:24.974 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:30:24.974 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:30:24.974 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 14, 2024-09-11T01:30:24.974 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 87615, 2024-09-11T01:30:24.974 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:30:24.974 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:30:24.974 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:30:24.974 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:30:24.974 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:24.974 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:30:24.975 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:30:24.975 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:30:24.975 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:30:24.975 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:30:24.975 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:30:24.975 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:30:24.975 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:30:24.975 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:30:24.975 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:30:24.975 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:30:24.975 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:30:24.975 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:30:24.975 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:30:24.975 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:30:24.976 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:24.976 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:30:24.976 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:30:24.976 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:30:24.976 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:30:24.976 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:30:24.976 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:30:24.976 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:30:24.976 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:30:24.976 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:30:24.976 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:30:24.976 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:30:24.976 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:30:24.977 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:30:24.977 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:24.977 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.977 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:30:24.977 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:30:24.977 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:30:24.977 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:30:24.977 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:30:24.977 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:30:24.977 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:30:24.977 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:30:24.977 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:30:24.977 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:30:24.977 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:30:24.978 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:30:24.978 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:30:24.978 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:30:24.978 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:30:24.978 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:30:24.978 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:30:24.978 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:30:24.978 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:30:24.978 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:30:24.978 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:30:24.978 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:30:24.978 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:30:24.978 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:30:24.978 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:30:24.979 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:30:24.979 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:30:24.979 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:30:24.979 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:24.979 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:30:24.979 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.979 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:30:24.979 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.979 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:30:24.979 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:30:24.979 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:30:24.979 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:30:24.979 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.979 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:30:24.980 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:30:24.980 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:24.980 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.980 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.980 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:30:24.980 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:30:24.980 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:24.980 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:24.980 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:24.980 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.980 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:30:24.980 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:30:24.980 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:30:24.980 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:30:24.981 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:30:24.981 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.981 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.981 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:30:24.981 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:30:24.981 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:30:24.981 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:30:24.981 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.981 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:30:24.981 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:30:24.981 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:24.981 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.981 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.982 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:30:24.982 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:30:24.982 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:24.982 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:24.982 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:24.982 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.982 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:30:24.982 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:30:24.982 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:30:24.982 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:30:24.982 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:30:24.982 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.982 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.983 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:30:24.983 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:30:24.983 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:30:24.983 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:30:24.983 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.983 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:30:24.983 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:30:24.983 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:24.983 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.983 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.983 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:30:24.983 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:30:24.983 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:24.983 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:24.983 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:24.984 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.984 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:30:24.984 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:30:24.984 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:30:24.984 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:30:24.984 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:30:24.984 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:24.984 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:24.984 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.984 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:30:24.984 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:30:24.984 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.984 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:30:24.985 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:30:24.985 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:30:24.985 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:24.985 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:24.985 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:30:24.985 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.985 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:30:24.985 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:30:24.985 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:30:24.985 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:24.985 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:24.985 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:30:24.985 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.985 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:30:24.986 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:30:24.986 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:30:24.986 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:24.986 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:24.986 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:30:24.986 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:24.986 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:30:24.986 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:30:24.986 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:30:24.986 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:24.986 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:24.986 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:24.986 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:30:24.986 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:30:24.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.962+0000 7f6e7f9a3640 1 -- 172.21.15.12:0/3355750825 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6e5c03e210 msgr2=0x7f6e5c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:24.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.962+0000 7f6e7f9a3640 1 --2- 172.21.15.12:0/3355750825 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6e5c03e210 0x7f6e5c0406d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6e68031000 tx=0x7f6e6802e000 comp rx=0 tx=0).stop 2024-09-11T01:30:24.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.962+0000 7f6e7f9a3640 10 monclient: shutdown 2024-09-11T01:30:24.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.962+0000 7f6e7f9a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:24.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.962+0000 7f6e7f9a3640 1 -- 172.21.15.12:0/3355750825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e7813d380 msgr2=0x7f6e780a1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:24.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.962+0000 7f6e7f9a3640 1 --2- 172.21.15.12:0/3355750825 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e7813d380 0x7f6e780a1fa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6e7000c780 tx=0x7f6e7000cc40 comp rx=0 tx=0).stop 2024-09-11T01:30:24.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.962+0000 7f6e7df40640 1 -- 172.21.15.12:0/3355750825 reap_dead start 2024-09-11T01:30:24.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.962+0000 7f6e7df40640 1 -- 172.21.15.12:0/3355750825 reap_dead start 2024-09-11T01:30:24.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.962+0000 7f6e7f9a3640 1 -- 172.21.15.12:0/3355750825 shutdown_connections 2024-09-11T01:30:24.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.962+0000 7f6e7f9a3640 1 -- 172.21.15.12:0/3355750825 >> 172.21.15.12:0/3355750825 conn(0x7f6e780a7020 msgr2=0x7f6e7809eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:24.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.962+0000 7f6e7f9a3640 1 -- 172.21.15.12:0/3355750825 shutdown_connections 2024-09-11T01:30:24.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:24.962+0000 7f6e7f9a3640 1 -- 172.21.15.12:0/3355750825 wait complete. 2024-09-11T01:30:24.989 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:30:24.989 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-11T01:30:24.989 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T01:30:24.989 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-11T01:30:24.990 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-11T01:30:24.990 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T01:30:24.990 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:30:25.000 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T01:30:25.000 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:30:25.000 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:30:25.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.139+0000 7fb81b553640 10 monclient: get_monmap_and_config 2024-09-11T01:30:25.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.139+0000 7fb81b553640 10 monclient: build_initial_monmap 2024-09-11T01:30:25.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.139+0000 7fb81b553640 10 monclient: monmap: 2024-09-11T01:30:25.141 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:25.141 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:25.142 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:30:25.140077+0000 2024-09-11T01:30:25.142 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:30:25.140077+0000 2024-09-11T01:30:25.142 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:25.142 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:25.142 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:30:25.142 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:30:25.142 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:30:25.142 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:30:25.142 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:30:25.142 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:25.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.139+0000 7fb81b553640 1 Processor -- start 2024-09-11T01:30:25.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb81b553640 1 -- start start 2024-09-11T01:30:25.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb81b553640 10 monclient: init 2024-09-11T01:30:25.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb81b553640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:25.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb81b553640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-11T01:30:25.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb81b553640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb81409d790 0x7fb81409db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:25.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb81b553640 10 monclient(hunting): picked mon.noname-c con 0x7fb81409d790 addr v2:172.21.15.71:3300/0 2024-09-11T01:30:25.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb81b553640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb81409e0b0 addr v1:172.21.15.12:6789/0 2024-09-11T01:30:25.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb81b553640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb81409e500 0x7fb814135170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:25.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb81b553640 10 monclient(hunting): picked mon.noname-b con 0x7fb81409e500 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:25.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb81b553640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8140a1da0 con 0x7fb81409e0b0 2024-09-11T01:30:25.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb81b553640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:25.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb81b553640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fb81409e960 con 0x7fb81409d790 2024-09-11T01:30:25.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb81b553640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:25.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb81b553640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb8140a0760 con 0x7fb81409e500 2024-09-11T01:30:25.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb81b553640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:25.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb81b553640 10 monclient(hunting): authenticate will time out at 831.582746s 2024-09-11T01:30:25.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb818aee640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7fb81409e0b0 legacy=0x7fb814134a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:30:25.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb819af0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb81409e500 0x7fb814135170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:25.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb819af0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb81409e500 0x7fb814135170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:40736/0 (socket says 172.21.15.12:40736) 2024-09-11T01:30:25.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb819af0640 1 -- 172.21.15.12:0/1031570288 learned_addr learned my addr 172.21.15.12:0/1031570288 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:30:25.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb819af0640 10 monclient(hunting): get_auth_request con 0x7fb81409e500 auth_method 0 2024-09-11T01:30:25.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb819af0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:25.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb819af0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:25.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb819af0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:25.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb819af0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:25.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb819af0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:25.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.140+0000 7fb819af0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:25.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.141+0000 7fb8192ef640 1 --2- 172.21.15.12:0/1031570288 >> v2:172.21.15.71:3300/0 conn(0x7fb81409d790 0x7fb81409db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:25.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.141+0000 7fb8192ef640 10 monclient(hunting): get_auth_request con 0x7fb81409d790 auth_method 0 2024-09-11T01:30:25.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.141+0000 7fb8192ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:25.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.141+0000 7fb8192ef640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:25.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.141+0000 7fb8192ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:25.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.141+0000 7fb819af0640 10 monclient(hunting): handle_auth_done global_id 4481 payload 274 2024-09-11T01:30:25.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.141+0000 7fb819af0640 1 -- 172.21.15.12:0/1031570288 >> v2:172.21.15.71:3300/0 conn(0x7fb81409d790 msgr2=0x7fb81409db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:30:25.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.141+0000 7fb819af0640 1 --2- 172.21.15.12:0/1031570288 >> v2:172.21.15.71:3300/0 conn(0x7fb81409d790 0x7fb81409db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:25.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.141+0000 7fb819af0640 1 -- 172.21.15.12:0/1031570288 >> v1:172.21.15.12:6789/0 conn(0x7fb81409e0b0 legacy=0x7fb814134a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:30:25.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.141+0000 7fb819af0640 10 monclient: _finish_hunting 0 2024-09-11T01:30:25.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.141+0000 7fb819af0640 1 monclient: found mon.noname-b 2024-09-11T01:30:25.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.141+0000 7fb819af0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:25.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.141+0000 7fb819af0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:30:25.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.141+0000 7fb819af0640 1 -- 172.21.15.12:0/1031570288 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8140a1fa0 con 0x7fb81409e500 2024-09-11T01:30:25.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.143+0000 7fb819af0640 1 -- 172.21.15.12:0/1031570288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb81409e500 msgr2=0x7fb814135170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:25.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.143+0000 7fb819af0640 1 --2- 172.21.15.12:0/1031570288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb81409e500 0x7fb814135170 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb80c009e50 tx=0x7fb80c02a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b90aa8f5f19097f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:25.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb80bfff640 1 -- 172.21.15.12:0/1031570288 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80c036070 con 0x7fb81409e500 2024-09-11T01:30:25.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb80bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:25.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb80bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:30:25.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb80bfff640 10 monclient: dump: 2024-09-11T01:30:25.153 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:25.153 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:25.153 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:25.153 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:25.154 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:25.154 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:25.154 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:25.154 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:25.154 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:25.154 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:25.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb80bfff640 10 monclient: _finish_auth 0 2024-09-11T01:30:25.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb80bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:30:25.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb80bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:25.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb80bfff640 1 -- 172.21.15.12:0/1031570288 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb80c007800 con 0x7fb81409e500 2024-09-11T01:30:25.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb81b553640 5 monclient: authenticate success, global_id 4481 2024-09-11T01:30:25.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb81b553640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:30:25.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb80bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:25.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb81b553640 10 monclient: get_monmap_and_config success 2024-09-11T01:30:25.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb81b553640 4 set_mon_vals no callback set 2024-09-11T01:30:25.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb80bfff640 1 -- 172.21.15.12:0/1031570288 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80c007bd0 con 0x7fb81409e500 2024-09-11T01:30:25.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb80bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:25.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb80bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:25.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb80bfff640 10 monclient: dump: 2024-09-11T01:30:25.156 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:25.156 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:25.156 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:25.157 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:25.157 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:25.157 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:25.157 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:25.157 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:25.157 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:25.157 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:25.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb81b553640 10 monclient: shutdown 2024-09-11T01:30:25.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb81b553640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:25.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb81b553640 1 -- 172.21.15.12:0/1031570288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb81409e500 msgr2=0x7fb814135170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:25.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb81b553640 1 --2- 172.21.15.12:0/1031570288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb81409e500 0x7fb814135170 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb80c009e50 tx=0x7fb80c02a560 comp rx=0 tx=0).stop 2024-09-11T01:30:25.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb81b553640 1 -- 172.21.15.12:0/1031570288 shutdown_connections 2024-09-11T01:30:25.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb81b553640 1 --2- 172.21.15.12:0/1031570288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb81409e500 0x7fb814135170 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:25.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb81b553640 1 --2- 172.21.15.12:0/1031570288 >> v2:172.21.15.71:3300/0 conn(0x7fb81409d790 0x7fb81409db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:25.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb81b553640 1 -- 172.21.15.12:0/1031570288 >> 172.21.15.12:0/1031570288 conn(0x7fb8140a2d90 msgr2=0x7fb8140a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:25.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb81b553640 1 -- 172.21.15.12:0/1031570288 shutdown_connections 2024-09-11T01:30:25.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.144+0000 7fb81b553640 1 -- 172.21.15.12:0/1031570288 wait complete. 2024-09-11T01:30:25.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.145+0000 7fb81b553640 10 monclient: build_initial_monmap 2024-09-11T01:30:25.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.145+0000 7fb81b553640 10 monclient: monmap: 2024-09-11T01:30:25.158 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:25.158 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:30:25.158 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:30:25.158 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:30:25.158 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:25.158 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:25.158 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:30:25.159 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:30:25.159 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:30:25.159 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:25.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.145+0000 7fb81b553640 1 Processor -- start 2024-09-11T01:30:25.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.145+0000 7fb81b553640 1 -- start start 2024-09-11T01:30:25.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.145+0000 7fb81b553640 10 monclient: init 2024-09-11T01:30:25.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb81b553640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:25.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb81b553640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:30:25.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb81b553640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb81409d790 0x7fb8141cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:25.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb81b553640 10 monclient(hunting): picked mon.noname-a con 0x7fb81409d790 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:30:25.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb81b553640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb81409e0b0 0x7fb81406f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:25.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb81b553640 10 monclient(hunting): picked mon.noname-b con 0x7fb81409e0b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:30:25.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb81b553640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb81409e500 0x7fb81406fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:25.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb81b553640 10 monclient(hunting): picked mon.noname-c con 0x7fb81409e500 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:25.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb81b553640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:25.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb81b553640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb8140a23a0 con 0x7fb81409d790 2024-09-11T01:30:25.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb81b553640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:25.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb81b553640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb8140a2220 con 0x7fb81409e0b0 2024-09-11T01:30:25.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb81b553640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:25.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb81b553640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb81409cca0 con 0x7fb81409e500 2024-09-11T01:30:25.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb81b553640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:25.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb81b553640 10 monclient(hunting): authenticate will time out at 831.588242s 2024-09-11T01:30:25.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb8192ef640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb81409d790 msgr2=0x7fb8141cd3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:30:25.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb8192ef640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb81409d790 0x7fb8141cd3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:30:25.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb819af0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb81409e500 0x7fb81406fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:25.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb819af0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb81409e500 0x7fb81406fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:40740/0 (socket says 172.21.15.12:40740) 2024-09-11T01:30:25.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb819af0640 1 -- 172.21.15.12:0/2834361821 learned_addr learned my addr 172.21.15.12:0/2834361821 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:30:25.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb819af0640 10 monclient(hunting): get_auth_request con 0x7fb81409e500 auth_method 0 2024-09-11T01:30:25.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb819af0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:25.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb819af0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:25.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb819af0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:25.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb819af0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:25.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb819af0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:25.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb819af0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:25.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb818aee640 1 --2- 172.21.15.12:0/2834361821 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb81409e0b0 0x7fb81406f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:25.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb819af0640 10 monclient(hunting): handle_auth_done global_id 4484 payload 824 2024-09-11T01:30:25.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb819af0640 1 -- 172.21.15.12:0/2834361821 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb81409e0b0 msgr2=0x7fb81406f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:25.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb819af0640 1 --2- 172.21.15.12:0/2834361821 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb81409e0b0 0x7fb81406f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:25.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb819af0640 1 -- 172.21.15.12:0/2834361821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb81409d790 msgr2=0x7fb8141cd3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:30:25.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb819af0640 1 --2- 172.21.15.12:0/2834361821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb81409d790 0x7fb8141cd3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:25.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb819af0640 10 monclient: _finish_hunting 0 2024-09-11T01:30:25.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb819af0640 1 monclient: found mon.noname-c 2024-09-11T01:30:25.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb819af0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:25.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb819af0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:30:25.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb819af0640 1 -- 172.21.15.12:0/2834361821 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8141344b0 con 0x7fb81409e500 2024-09-11T01:30:25.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb818aee640 10 monclient: get_auth_request con 0x7fb81409e0b0 auth_method 0 2024-09-11T01:30:25.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.146+0000 7fb818aee640 1 --2- 172.21.15.12:0/2834361821 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb81409e0b0 0x7fb81406f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:30:25.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.148+0000 7fb819af0640 1 -- 172.21.15.12:0/2834361821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb81409e500 msgr2=0x7fb81406fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:25.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.148+0000 7fb819af0640 1 --2- 172.21.15.12:0/2834361821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb81409e500 0x7fb81406fae0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb80c009f70 tx=0x7fb80c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:25.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.149+0000 7fb809ffb640 1 -- 172.21.15.12:0/2834361821 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80c041020 con 0x7fb81409e500 2024-09-11T01:30:25.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.149+0000 7fb809ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:25.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.149+0000 7fb809ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:30:25.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.149+0000 7fb809ffb640 10 monclient: dump: 2024-09-11T01:30:25.163 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:25.163 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:25.163 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:25.163 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:25.163 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:25.163 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:25.163 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:25.163 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:25.163 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:25.163 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:25.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.149+0000 7fb809ffb640 10 monclient: _finish_auth 0 2024-09-11T01:30:25.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.149+0000 7fb809ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:30:25.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.149+0000 7fb809ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:25.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.149+0000 7fb809ffb640 1 -- 172.21.15.12:0/2834361821 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb80c031940 con 0x7fb81409e500 2024-09-11T01:30:25.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.149+0000 7fb81b553640 5 monclient: authenticate success, global_id 4484 2024-09-11T01:30:25.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.149+0000 7fb809ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:25.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.149+0000 7fb81b553640 10 monclient: _renew_subs 2024-09-11T01:30:25.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.149+0000 7fb809ffb640 1 -- 172.21.15.12:0/2834361821 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80c036040 con 0x7fb81409e500 2024-09-11T01:30:25.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.149+0000 7fb81b553640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:25.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.149+0000 7fb81b553640 1 -- 172.21.15.12:0/2834361821 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb81409d480 con 0x7fb81409e500 2024-09-11T01:30:25.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.149+0000 7fb809ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:25.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.149+0000 7fb809ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:25.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.149+0000 7fb809ffb640 10 monclient: dump: 2024-09-11T01:30:25.164 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:25.164 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:25.164 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:25.164 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:25.165 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:25.165 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:25.165 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:25.165 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:25.165 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:25.165 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:25.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.149+0000 7fb80b7fe640 4 set_mon_vals no callback set 2024-09-11T01:30:25.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.149+0000 7fb81b553640 10 monclient: _renew_subs 2024-09-11T01:30:25.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.149+0000 7fb81b553640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:25.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.149+0000 7fb81b553640 1 -- 172.21.15.12:0/2834361821 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb814128190 con 0x7fb81409e500 2024-09-11T01:30:25.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.150+0000 7fb81b553640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:25.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.150+0000 7fb81b553640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:25.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.150+0000 7fb81b553640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:25.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.150+0000 7fb809ffb640 1 -- 172.21.15.12:0/2834361821 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fb80c03c470 con 0x7fb81409e500 2024-09-11T01:30:25.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.150+0000 7fb81b553640 1 -- 172.21.15.12:0/2834361821 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb814135730 con 0x7fb81409e500 2024-09-11T01:30:25.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.150+0000 7fb809ffb640 1 --2- 172.21.15.12:0/2834361821 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb7f803e1f0 0x7fb7f80406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:25.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.150+0000 7fb809ffb640 1 -- 172.21.15.12:0/2834361821 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(41..41 src has 1..41) ==== 4739+0+0 (secure 0 0 0) 0x7fb80c073e50 con 0x7fb81409e500 2024-09-11T01:30:25.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.150+0000 7fb8192ef640 1 --2- 172.21.15.12:0/2834361821 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb7f803e1f0 0x7fb7f80406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:25.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.150+0000 7fb8192ef640 10 monclient: get_auth_request con 0x7fb7f803e1f0 auth_method 0 2024-09-11T01:30:25.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.153+0000 7fb8192ef640 1 --2- 172.21.15.12:0/2834361821 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb7f803e1f0 0x7fb7f80406b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb810006fd0 tx=0x7fb8100092e0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:25.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.155+0000 7fb809ffb640 1 -- 172.21.15.12:0/2834361821 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb80c04b070 con 0x7fb81409e500 2024-09-11T01:30:25.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.155+0000 7fb809ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:25.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.155+0000 7fb809ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:30:25.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.410+0000 7fb81b553640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:30:25.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.410+0000 7fb81b553640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:30:25.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.410+0000 7fb81b553640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:25.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.410+0000 7fb81b553640 1 -- 172.21.15.12:0/2834361821 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb814126c70 con 0x7fb81409e500 2024-09-11T01:30:25.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.410+0000 7fb809ffb640 1 -- 172.21.15.12:0/2834361821 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fb80c031db0 con 0x7fb81409e500 2024-09-11T01:30:25.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.410+0000 7fb809ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:30:25.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.410+0000 7fb809ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:30:25.413 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:30:25.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.412+0000 7fb81b553640 1 -- 172.21.15.12:0/2834361821 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb7f803e1f0 msgr2=0x7fb7f80406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:25.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.412+0000 7fb81b553640 1 --2- 172.21.15.12:0/2834361821 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb7f803e1f0 0x7fb7f80406b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb810006fd0 tx=0x7fb8100092e0 comp rx=0 tx=0).stop 2024-09-11T01:30:25.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.412+0000 7fb81b553640 10 monclient: shutdown 2024-09-11T01:30:25.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.412+0000 7fb81b553640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:25.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.412+0000 7fb81b553640 1 -- 172.21.15.12:0/2834361821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb81409e500 msgr2=0x7fb81406fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:25.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.412+0000 7fb81b553640 1 --2- 172.21.15.12:0/2834361821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb81409e500 0x7fb81406fae0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb80c009f70 tx=0x7fb80c004a40 comp rx=0 tx=0).stop 2024-09-11T01:30:25.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.412+0000 7fb81b553640 1 -- 172.21.15.12:0/2834361821 shutdown_connections 2024-09-11T01:30:25.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.412+0000 7fb81b553640 1 --2- 172.21.15.12:0/2834361821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb81409e500 0x7fb81406fae0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:25.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.412+0000 7fb81b553640 1 --2- 172.21.15.12:0/2834361821 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb81409e0b0 0x7fb81406f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:25.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.412+0000 7fb81b553640 1 --2- 172.21.15.12:0/2834361821 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb7f803e1f0 0x7fb7f80406b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:25.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.412+0000 7fb81b553640 1 --2- 172.21.15.12:0/2834361821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb81409d790 0x7fb8141cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:25.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.412+0000 7fb81b553640 1 -- 172.21.15.12:0/2834361821 >> 172.21.15.12:0/2834361821 conn(0x7fb8140a2d90 msgr2=0x7fb81411c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:25.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.412+0000 7fb81b553640 1 -- 172.21.15.12:0/2834361821 shutdown_connections 2024-09-11T01:30:25.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:25.412+0000 7fb81b553640 1 -- 172.21.15.12:0/2834361821 wait complete. 2024-09-11T01:30:28.426 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:30:28.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.537+0000 7f37d04a9640 10 monclient: get_monmap_and_config 2024-09-11T01:30:28.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.537+0000 7f37d04a9640 10 monclient: build_initial_monmap 2024-09-11T01:30:28.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.537+0000 7f37d04a9640 10 monclient: monmap: 2024-09-11T01:30:28.539 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:28.539 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:28.540 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:30:28.538496+0000 2024-09-11T01:30:28.540 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:30:28.538496+0000 2024-09-11T01:30:28.540 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:28.540 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:28.540 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:30:28.540 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:30:28.540 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:30:28.540 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:30:28.540 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:30:28.540 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:28.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.538+0000 7f37d04a9640 1 Processor -- start 2024-09-11T01:30:28.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.538+0000 7f37d04a9640 1 -- start start 2024-09-11T01:30:28.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.538+0000 7f37d04a9640 10 monclient: init 2024-09-11T01:30:28.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37d04a9640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:28.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37d04a9640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-11T01:30:28.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37d04a9640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f37c8138230 0x7f37c8138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:28.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37d04a9640 10 monclient(hunting): picked mon.noname-c con 0x7f37c8138230 addr v2:172.21.15.71:3300/0 2024-09-11T01:30:28.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37d04a9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f37c809d030 0x7f37c80a5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:28.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37d04a9640 10 monclient(hunting): picked mon.noname-b con 0x7f37c809d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:28.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37d04a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f37c809d460 addr v1:172.21.15.71:6789/0 2024-09-11T01:30:28.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37d04a9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37c80a6e30 con 0x7f37c809d460 2024-09-11T01:30:28.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37d04a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:28.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37d04a9640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f37c809c9c0 con 0x7f37c8138230 2024-09-11T01:30:28.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37d04a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:28.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37d04a9640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f37c809ba20 con 0x7f37c809d030 2024-09-11T01:30:28.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37d04a9640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:28.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37d04a9640 10 monclient(hunting): authenticate will time out at 834.981195s 2024-09-11T01:30:28.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37cda44640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f37c809d030 0x7f37c80a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:28.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37cda44640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f37c809d030 0x7f37c80a5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45390/0 (socket says 172.21.15.12:45390) 2024-09-11T01:30:28.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37cda44640 1 -- 172.21.15.12:0/3124780994 learned_addr learned my addr 172.21.15.12:0/3124780994 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:30:28.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37cda44640 10 monclient(hunting): get_auth_request con 0x7f37c809d030 auth_method 0 2024-09-11T01:30:28.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37cda44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:28.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37cda44640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:28.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37cda44640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:28.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37cda44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:28.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37cda44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:28.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37cda44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:28.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37ce245640 1 --2- 172.21.15.12:0/3124780994 >> v2:172.21.15.71:3300/0 conn(0x7f37c8138230 0x7f37c8138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:28.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37ce245640 10 monclient(hunting): get_auth_request con 0x7f37c8138230 auth_method 0 2024-09-11T01:30:28.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37ce245640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:28.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37ce245640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:28.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37ce245640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:28.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37cda44640 10 monclient(hunting): handle_auth_done global_id 4487 payload 274 2024-09-11T01:30:28.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37cda44640 1 -- 172.21.15.12:0/3124780994 >> v2:172.21.15.71:3300/0 conn(0x7f37c8138230 msgr2=0x7f37c8138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:30:28.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37cda44640 1 --2- 172.21.15.12:0/3124780994 >> v2:172.21.15.71:3300/0 conn(0x7f37c8138230 0x7f37c8138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:28.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37cda44640 1 -- 172.21.15.12:0/3124780994 >> v1:172.21.15.71:6789/0 conn(0x7f37c809d460 legacy=0x7f37c80a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:30:28.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37cda44640 10 monclient: _finish_hunting 0 2024-09-11T01:30:28.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37cda44640 1 monclient: found mon.noname-b 2024-09-11T01:30:28.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37cda44640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:28.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37cda44640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:30:28.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.539+0000 7f37cda44640 1 -- 172.21.15.12:0/3124780994 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37c80a66a0 con 0x7f37c809d030 2024-09-11T01:30:28.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.542+0000 7f37cda44640 1 -- 172.21.15.12:0/3124780994 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f37c809d030 msgr2=0x7f37c80a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:28.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.542+0000 7f37cda44640 1 --2- 172.21.15.12:0/3124780994 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f37c809d030 0x7f37c80a5020 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f37c80a17e0 tx=0x7f37b802a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3368c7857a647a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:28.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.542+0000 7f37cd243640 1 -- 172.21.15.12:0/3124780994 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b8037070 con 0x7f37c809d030 2024-09-11T01:30:28.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.542+0000 7f37cd243640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:28.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.542+0000 7f37cd243640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:30:28.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.542+0000 7f37cd243640 10 monclient: dump: 2024-09-11T01:30:28.552 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:28.552 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:28.553 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:28.553 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:28.553 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:28.553 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:28.553 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:28.553 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:28.553 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:28.553 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:28.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.542+0000 7f37cd243640 10 monclient: _finish_auth 0 2024-09-11T01:30:28.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.542+0000 7f37cd243640 10 monclient: _check_auth_tickets 2024-09-11T01:30:28.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.542+0000 7f37cd243640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:28.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.542+0000 7f37cd243640 1 -- 172.21.15.12:0/3124780994 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37b8032440 con 0x7f37c809d030 2024-09-11T01:30:28.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.542+0000 7f37d04a9640 5 monclient: authenticate success, global_id 4487 2024-09-11T01:30:28.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.542+0000 7f37d04a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:30:28.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.542+0000 7f37cd243640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:28.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.542+0000 7f37d04a9640 10 monclient: get_monmap_and_config success 2024-09-11T01:30:28.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.542+0000 7f37cd243640 1 -- 172.21.15.12:0/3124780994 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b80327f0 con 0x7f37c809d030 2024-09-11T01:30:28.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.542+0000 7f37d04a9640 4 set_mon_vals no callback set 2024-09-11T01:30:28.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.542+0000 7f37cd243640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:28.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.542+0000 7f37cd243640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:28.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.542+0000 7f37cd243640 10 monclient: dump: 2024-09-11T01:30:28.554 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:28.555 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:28.555 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:28.555 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:28.555 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:28.555 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:28.555 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:28.555 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:28.555 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:28.555 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:28.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.543+0000 7f37d04a9640 10 monclient: shutdown 2024-09-11T01:30:28.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.543+0000 7f37d04a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:28.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.543+0000 7f37d04a9640 1 -- 172.21.15.12:0/3124780994 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f37c809d030 msgr2=0x7f37c80a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:28.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.543+0000 7f37d04a9640 1 --2- 172.21.15.12:0/3124780994 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f37c809d030 0x7f37c80a5020 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f37c80a17e0 tx=0x7f37b802a080 comp rx=0 tx=0).stop 2024-09-11T01:30:28.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.543+0000 7f37d04a9640 1 -- 172.21.15.12:0/3124780994 shutdown_connections 2024-09-11T01:30:28.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.543+0000 7f37d04a9640 1 --2- 172.21.15.12:0/3124780994 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f37c809d030 0x7f37c80a5020 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:28.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.543+0000 7f37d04a9640 1 --2- 172.21.15.12:0/3124780994 >> v2:172.21.15.71:3300/0 conn(0x7f37c8138230 0x7f37c8138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:28.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.543+0000 7f37d04a9640 1 -- 172.21.15.12:0/3124780994 >> 172.21.15.12:0/3124780994 conn(0x7f37c80a7120 msgr2=0x7f37c80a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:28.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.543+0000 7f37d04a9640 1 -- 172.21.15.12:0/3124780994 shutdown_connections 2024-09-11T01:30:28.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.543+0000 7f37d04a9640 1 -- 172.21.15.12:0/3124780994 wait complete. 2024-09-11T01:30:28.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.544+0000 7f37d04a9640 10 monclient: build_initial_monmap 2024-09-11T01:30:28.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.544+0000 7f37d04a9640 10 monclient: monmap: 2024-09-11T01:30:28.556 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:28.556 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:30:28.556 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:30:28.556 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:30:28.556 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:28.557 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:28.557 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:30:28.557 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:30:28.557 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:30:28.557 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:28.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.544+0000 7f37d04a9640 1 Processor -- start 2024-09-11T01:30:28.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.544+0000 7f37d04a9640 1 -- start start 2024-09-11T01:30:28.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.544+0000 7f37d04a9640 10 monclient: init 2024-09-11T01:30:28.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37d04a9640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:28.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37d04a9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:30:28.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37d04a9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f37c809d030 0x7f37c8136d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:28.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37d04a9640 10 monclient(hunting): picked mon.noname-c con 0x7f37c809d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:28.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37d04a9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37c809d460 0x7f37c81372c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:28.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37d04a9640 10 monclient(hunting): picked mon.noname-a con 0x7f37c809d460 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:30:28.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37d04a9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37c8138230 0x7f37c8120c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:28.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37d04a9640 10 monclient(hunting): picked mon.noname-b con 0x7f37c8138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:30:28.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37d04a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:28.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37d04a9640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f37c806f970 con 0x7f37c809d460 2024-09-11T01:30:28.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37d04a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:28.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37d04a9640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f37c806f7f0 con 0x7f37c8138230 2024-09-11T01:30:28.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37d04a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:28.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37d04a9640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f37c80a5c70 con 0x7f37c809d030 2024-09-11T01:30:28.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37d04a9640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:28.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37d04a9640 10 monclient(hunting): authenticate will time out at 834.987196s 2024-09-11T01:30:28.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37cda44640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37c809d460 msgr2=0x7f37c81372c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:30:28.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37cda44640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37c809d460 0x7f37c81372c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:30:28.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37ce245640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f37c809d030 0x7f37c8136d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:28.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37ce245640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f37c809d030 0x7f37c8136d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45394/0 (socket says 172.21.15.12:45394) 2024-09-11T01:30:28.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37ce245640 1 -- 172.21.15.12:0/3110904181 learned_addr learned my addr 172.21.15.12:0/3110904181 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:30:28.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37ce245640 10 monclient(hunting): get_auth_request con 0x7f37c809d030 auth_method 0 2024-09-11T01:30:28.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37ce245640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:28.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37ce245640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:28.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37ce245640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:28.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37ce245640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:28.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37ce245640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:28.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37ce245640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:28.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37ce245640 10 monclient(hunting): handle_auth_done global_id 4490 payload 824 2024-09-11T01:30:28.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37cea46640 1 --2- 172.21.15.12:0/3110904181 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37c8138230 0x7f37c8120c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:28.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37ce245640 1 -- 172.21.15.12:0/3110904181 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37c8138230 msgr2=0x7f37c8120c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:28.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37ce245640 1 --2- 172.21.15.12:0/3110904181 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37c8138230 0x7f37c8120c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:28.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37ce245640 1 -- 172.21.15.12:0/3110904181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37c809d460 msgr2=0x7f37c81372c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:30:28.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37ce245640 1 --2- 172.21.15.12:0/3110904181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37c809d460 0x7f37c81372c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:28.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37ce245640 10 monclient: _finish_hunting 0 2024-09-11T01:30:28.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37ce245640 1 monclient: found mon.noname-c 2024-09-11T01:30:28.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37ce245640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:28.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37ce245640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:30:28.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.545+0000 7f37ce245640 1 -- 172.21.15.12:0/3110904181 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37c81223e0 con 0x7f37c809d030 2024-09-11T01:30:28.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.547+0000 7f37ce245640 1 -- 172.21.15.12:0/3110904181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f37c809d030 msgr2=0x7f37c8136d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:28.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.547+0000 7f37ce245640 1 --2- 172.21.15.12:0/3110904181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f37c809d030 0x7f37c8136d80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f37c400d800 tx=0x7f37c400dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:28.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.548+0000 7f37beffd640 1 -- 172.21.15.12:0/3110904181 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c400b8d0 con 0x7f37c809d030 2024-09-11T01:30:28.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.548+0000 7f37beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:28.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.548+0000 7f37beffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:30:28.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.548+0000 7f37beffd640 10 monclient: dump: 2024-09-11T01:30:28.561 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:28.561 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:28.561 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:28.561 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:28.561 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:28.561 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:28.561 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:28.561 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:28.561 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:28.561 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:28.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.548+0000 7f37beffd640 10 monclient: _finish_auth 0 2024-09-11T01:30:28.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.548+0000 7f37beffd640 10 monclient: _check_auth_tickets 2024-09-11T01:30:28.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.548+0000 7f37beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:28.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.548+0000 7f37d04a9640 5 monclient: authenticate success, global_id 4490 2024-09-11T01:30:28.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.548+0000 7f37beffd640 1 -- 172.21.15.12:0/3110904181 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37c4019070 con 0x7f37c809d030 2024-09-11T01:30:28.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.548+0000 7f37d04a9640 10 monclient: _renew_subs 2024-09-11T01:30:28.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.548+0000 7f37d04a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:28.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.548+0000 7f37d04a9640 1 -- 172.21.15.12:0/3110904181 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37c8123ea0 con 0x7f37c809d030 2024-09-11T01:30:28.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.548+0000 7f37beffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:28.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.548+0000 7f37beffd640 1 -- 172.21.15.12:0/3110904181 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c400bc90 con 0x7f37c809d030 2024-09-11T01:30:28.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.548+0000 7f37beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:28.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.548+0000 7f37beffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:28.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.548+0000 7f37beffd640 10 monclient: dump: 2024-09-11T01:30:28.562 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:28.562 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:28.562 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:28.562 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:28.562 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:28.562 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:28.563 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:28.563 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:28.563 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:28.563 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:28.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.548+0000 7f37d04a9640 10 monclient: _renew_subs 2024-09-11T01:30:28.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.548+0000 7f37cca42640 4 set_mon_vals no callback set 2024-09-11T01:30:28.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.548+0000 7f37d04a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:28.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.548+0000 7f37d04a9640 1 -- 172.21.15.12:0/3110904181 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f37c8122f40 con 0x7f37c809d030 2024-09-11T01:30:28.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.548+0000 7f37d04a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:28.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.548+0000 7f37d04a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:28.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.549+0000 7f37d04a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:28.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.549+0000 7f37beffd640 1 -- 172.21.15.12:0/3110904181 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f37c4013070 con 0x7f37c809d030 2024-09-11T01:30:28.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.549+0000 7f37d04a9640 1 -- 172.21.15.12:0/3110904181 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37c8138610 con 0x7f37c809d030 2024-09-11T01:30:28.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.549+0000 7f37beffd640 1 --2- 172.21.15.12:0/3110904181 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f37ac03e210 0x7f37ac0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:28.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.549+0000 7f37beffd640 1 -- 172.21.15.12:0/3110904181 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(41..41 src has 1..41) ==== 4739+0+0 (secure 0 0 0) 0x7f37c4052f80 con 0x7f37c809d030 2024-09-11T01:30:28.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.549+0000 7f37cda44640 1 --2- 172.21.15.12:0/3110904181 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f37ac03e210 0x7f37ac0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:28.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.549+0000 7f37cda44640 10 monclient: get_auth_request con 0x7f37ac03e210 auth_method 0 2024-09-11T01:30:28.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.552+0000 7f37cda44640 1 --2- 172.21.15.12:0/3110904181 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f37ac03e210 0x7f37ac0406d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f37c806e500 tx=0x7f37b802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:28.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.552+0000 7f37beffd640 1 -- 172.21.15.12:0/3110904181 DispatchQueue::entry inject delay of 0.002000 2024-09-11T01:30:28.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.554+0000 7f37beffd640 1 -- 172.21.15.12:0/3110904181 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f37c40519a0 con 0x7f37c809d030 2024-09-11T01:30:28.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.554+0000 7f37beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:28.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.554+0000 7f37beffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:30:28.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.811+0000 7f37d04a9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:30:28.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.811+0000 7f37d04a9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:30:28.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.811+0000 7f37d04a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:28.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.811+0000 7f37d04a9640 1 -- 172.21.15.12:0/3110904181 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f37c80698e0 con 0x7f37c809d030 2024-09-11T01:30:28.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.811+0000 7f37beffd640 1 -- 172.21.15.12:0/3110904181 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f37c401ebc0 con 0x7f37c809d030 2024-09-11T01:30:28.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.811+0000 7f37beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:30:28.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.811+0000 7f37beffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:30:28.814 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:30:28.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.813+0000 7f37d04a9640 1 -- 172.21.15.12:0/3110904181 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f37ac03e210 msgr2=0x7f37ac0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:28.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.813+0000 7f37d04a9640 1 --2- 172.21.15.12:0/3110904181 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f37ac03e210 0x7f37ac0406d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f37c806e500 tx=0x7f37b802e000 comp rx=0 tx=0).stop 2024-09-11T01:30:28.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.813+0000 7f37d04a9640 10 monclient: shutdown 2024-09-11T01:30:28.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.813+0000 7f37d04a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:28.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.813+0000 7f37d04a9640 1 -- 172.21.15.12:0/3110904181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f37c809d030 msgr2=0x7f37c8136d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:28.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.813+0000 7f37d04a9640 1 --2- 172.21.15.12:0/3110904181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f37c809d030 0x7f37c8136d80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f37c400d800 tx=0x7f37c400dcc0 comp rx=0 tx=0).stop 2024-09-11T01:30:28.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.813+0000 7f37d04a9640 1 -- 172.21.15.12:0/3110904181 shutdown_connections 2024-09-11T01:30:28.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.813+0000 7f37d04a9640 1 --2- 172.21.15.12:0/3110904181 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37c8138230 0x7f37c8120c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:28.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.813+0000 7f37d04a9640 1 --2- 172.21.15.12:0/3110904181 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f37ac03e210 0x7f37ac0406d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:28.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.813+0000 7f37d04a9640 1 --2- 172.21.15.12:0/3110904181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37c809d460 0x7f37c81372c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:28.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.813+0000 7f37d04a9640 1 --2- 172.21.15.12:0/3110904181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f37c809d030 0x7f37c8136d80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:28.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.813+0000 7f37d04a9640 1 -- 172.21.15.12:0/3110904181 >> 172.21.15.12:0/3110904181 conn(0x7f37c80a7120 msgr2=0x7f37c80a3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:28.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.813+0000 7f37d04a9640 1 -- 172.21.15.12:0/3110904181 shutdown_connections 2024-09-11T01:30:28.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:28.813+0000 7f37d04a9640 1 -- 172.21.15.12:0/3110904181 wait complete. 2024-09-11T01:30:31.826 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:30:31.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.938+0000 7fe36d33d640 10 monclient: get_monmap_and_config 2024-09-11T01:30:31.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.938+0000 7fe36d33d640 10 monclient: build_initial_monmap 2024-09-11T01:30:31.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.938+0000 7fe36d33d640 10 monclient: monmap: 2024-09-11T01:30:31.940 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:31.940 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:31.940 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:30:31.939197+0000 2024-09-11T01:30:31.940 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:30:31.939197+0000 2024-09-11T01:30:31.941 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:31.941 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:31.941 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:30:31.941 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:30:31.941 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:30:31.941 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:30:31.941 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:30:31.941 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:31.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.939+0000 7fe36d33d640 1 Processor -- start 2024-09-11T01:30:31.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.939+0000 7fe36d33d640 1 -- start start 2024-09-11T01:30:31.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.939+0000 7fe36d33d640 10 monclient: init 2024-09-11T01:30:31.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.939+0000 7fe36d33d640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:31.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.939+0000 7fe36d33d640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-11T01:30:31.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.939+0000 7fe36d33d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe3680a2e90 0x7fe368073d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:31.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.939+0000 7fe36d33d640 10 monclient(hunting): picked mon.noname-a con 0x7fe3680a2e90 addr v2:172.21.15.12:3300/0 2024-09-11T01:30:31.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.939+0000 7fe36d33d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe3680a3270 0x7fe3680a36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:31.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.939+0000 7fe36d33d640 10 monclient(hunting): picked mon.noname-c con 0x7fe3680a3270 addr v2:172.21.15.71:3300/0 2024-09-11T01:30:31.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.939+0000 7fe36d33d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe368135c50 addr v1:172.21.15.12:6789/0 2024-09-11T01:30:31.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.939+0000 7fe366ffd640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7fe3680a2e90 msgr2=0x7fe368073d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:30:31.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.939+0000 7fe36d33d640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3680a46c0 con 0x7fe368135c50 2024-09-11T01:30:31.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.939+0000 7fe36d33d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:31.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.939+0000 7fe36d33d640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fe368072730 con 0x7fe3680a2e90 2024-09-11T01:30:31.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.939+0000 7fe366ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe3680a2e90 0x7fe368073d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:30:31.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.939+0000 7fe36d33d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:31.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.939+0000 7fe36d33d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fe368072fa0 con 0x7fe3680a3270 2024-09-11T01:30:31.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.939+0000 7fe36d33d640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:31.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.939+0000 7fe36d33d640 10 monclient(hunting): authenticate will time out at 838.381874s 2024-09-11T01:30:31.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.939+0000 7fe3677fe640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7fe368135c50 legacy=0x7fe3681360c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:30:31.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.940+0000 7fe35e7fc640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe3680a3270 0x7fe3680a36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:31.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.940+0000 7fe35e7fc640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe3680a3270 0x7fe3680a36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:55370/0 (socket says 172.21.15.12:55370) 2024-09-11T01:30:31.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.940+0000 7fe35e7fc640 1 -- 172.21.15.12:0/2997000844 learned_addr learned my addr 172.21.15.12:0/2997000844 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:30:31.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.940+0000 7fe35e7fc640 10 monclient(hunting): get_auth_request con 0x7fe3680a3270 auth_method 0 2024-09-11T01:30:31.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.940+0000 7fe35e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:31.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.940+0000 7fe35e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:31.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.940+0000 7fe35e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:31.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.940+0000 7fe35e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:31.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.940+0000 7fe35e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:31.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.940+0000 7fe35e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:31.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.940+0000 7fe35e7fc640 10 monclient(hunting): handle_auth_done global_id 4408 payload 274 2024-09-11T01:30:31.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.940+0000 7fe35e7fc640 1 -- 172.21.15.12:0/2997000844 >> v2:172.21.15.12:3300/0 conn(0x7fe3680a2e90 msgr2=0x7fe368073d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:30:31.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.940+0000 7fe35e7fc640 1 --2- 172.21.15.12:0/2997000844 >> v2:172.21.15.12:3300/0 conn(0x7fe3680a2e90 0x7fe368073d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:31.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.940+0000 7fe35e7fc640 1 -- 172.21.15.12:0/2997000844 >> v1:172.21.15.12:6789/0 conn(0x7fe368135c50 legacy=0x7fe3681360c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:30:31.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.940+0000 7fe35e7fc640 10 monclient: _finish_hunting 0 2024-09-11T01:30:31.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.940+0000 7fe35e7fc640 1 monclient: found mon.noname-c 2024-09-11T01:30:31.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.940+0000 7fe35e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:31.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.940+0000 7fe35e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:30:31.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.940+0000 7fe35e7fc640 1 -- 172.21.15.12:0/2997000844 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe368073280 con 0x7fe3680a3270 2024-09-11T01:30:31.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.943+0000 7fe35e7fc640 1 -- 172.21.15.12:0/2997000844 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3680a3270 msgr2=0x7fe3680a36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:31.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.943+0000 7fe35e7fc640 1 --2- 172.21.15.12:0/2997000844 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3680a3270 0x7fe3680a36a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe3680a4a20 tx=0x7fe35002a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f045674af7ef0546 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:31.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.944+0000 7fe3667fc640 1 -- 172.21.15.12:0/2997000844 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe350037070 con 0x7fe3680a3270 2024-09-11T01:30:31.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.944+0000 7fe3667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:31.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.944+0000 7fe3667fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:30:31.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.944+0000 7fe3667fc640 10 monclient: dump: 2024-09-11T01:30:31.950 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:31.950 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:31.950 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:31.951 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:31.951 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:31.951 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:31.951 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:31.951 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:31.951 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:31.951 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:31.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.944+0000 7fe3667fc640 10 monclient: _finish_auth 0 2024-09-11T01:30:31.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.944+0000 7fe3667fc640 10 monclient: _check_auth_tickets 2024-09-11T01:30:31.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.944+0000 7fe3667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:31.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.944+0000 7fe3667fc640 1 -- 172.21.15.12:0/2997000844 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe350032640 con 0x7fe3680a3270 2024-09-11T01:30:31.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.944+0000 7fe3667fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:31.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.944+0000 7fe3667fc640 1 -- 172.21.15.12:0/2997000844 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe350032a10 con 0x7fe3680a3270 2024-09-11T01:30:31.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.944+0000 7fe36d33d640 5 monclient: authenticate success, global_id 4408 2024-09-11T01:30:31.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.944+0000 7fe36d33d640 10 monclient: get_monmap_and_config success 2024-09-11T01:30:31.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.944+0000 7fe36d33d640 4 set_mon_vals no callback set 2024-09-11T01:30:31.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.944+0000 7fe3667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:31.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.944+0000 7fe3667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:30:31.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.944+0000 7fe3667fc640 10 monclient: dump: 2024-09-11T01:30:31.952 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:31.952 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:31.952 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:31.952 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:31.952 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:31.952 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:31.952 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:31.952 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:31.953 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:31.953 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:31.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.944+0000 7fe36d33d640 10 monclient: shutdown 2024-09-11T01:30:31.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.944+0000 7fe36d33d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:31.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.944+0000 7fe36d33d640 1 -- 172.21.15.12:0/2997000844 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3680a3270 msgr2=0x7fe3680a36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:31.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.944+0000 7fe36d33d640 1 --2- 172.21.15.12:0/2997000844 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3680a3270 0x7fe3680a36a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe3680a4a20 tx=0x7fe35002a100 comp rx=0 tx=0).stop 2024-09-11T01:30:31.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.944+0000 7fe36d33d640 1 -- 172.21.15.12:0/2997000844 shutdown_connections 2024-09-11T01:30:31.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.944+0000 7fe36d33d640 1 --2- 172.21.15.12:0/2997000844 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3680a3270 0x7fe3680a36a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:31.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.944+0000 7fe36d33d640 1 --2- 172.21.15.12:0/2997000844 >> v2:172.21.15.12:3300/0 conn(0x7fe3680a2e90 0x7fe368073d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:31.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.944+0000 7fe36d33d640 1 -- 172.21.15.12:0/2997000844 >> 172.21.15.12:0/2997000844 conn(0x7fe3680a8490 msgr2=0x7fe3680a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:31.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.944+0000 7fe36d33d640 1 -- 172.21.15.12:0/2997000844 shutdown_connections 2024-09-11T01:30:31.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.944+0000 7fe36d33d640 1 -- 172.21.15.12:0/2997000844 wait complete. 2024-09-11T01:30:31.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.945+0000 7fe36d33d640 10 monclient: build_initial_monmap 2024-09-11T01:30:31.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.945+0000 7fe36d33d640 10 monclient: monmap: 2024-09-11T01:30:31.954 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:31.954 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:30:31.955 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:30:31.955 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:30:31.955 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:31.955 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:31.955 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:30:31.955 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:30:31.955 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:30:31.955 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:31.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.945+0000 7fe36d33d640 1 Processor -- start 2024-09-11T01:30:31.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.945+0000 7fe36d33d640 1 -- start start 2024-09-11T01:30:31.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.945+0000 7fe36d33d640 10 monclient: init 2024-09-11T01:30:31.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe36d33d640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:31.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe36d33d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:30:31.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe36d33d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3680a2e90 0x7fe3681cb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:31.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe36d33d640 10 monclient(hunting): picked mon.noname-a con 0x7fe3680a2e90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:30:31.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe36d33d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3680a3270 0x7fe3681cb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:31.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe36d33d640 10 monclient(hunting): picked mon.noname-c con 0x7fe3680a3270 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:31.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe36d33d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe368135c50 0x7fe368122790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:31.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe366ffd640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3680a2e90 msgr2=0x7fe3681cb180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:30:31.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe36d33d640 10 monclient(hunting): picked mon.noname-b con 0x7fe368135c50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:30:31.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe36d33d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:31.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe36d33d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe368072300 con 0x7fe3680a2e90 2024-09-11T01:30:31.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe366ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3680a2e90 0x7fe3681cb180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:30:31.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe36d33d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:31.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe36d33d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe368074310 con 0x7fe368135c50 2024-09-11T01:30:31.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe36d33d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:31.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe36d33d640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe368072480 con 0x7fe3680a3270 2024-09-11T01:30:31.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe36d33d640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:31.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe36d33d640 10 monclient(hunting): authenticate will time out at 838.388491s 2024-09-11T01:30:31.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe35e7fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3680a3270 0x7fe3681cb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:31.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe35e7fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3680a3270 0x7fe3681cb6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45400/0 (socket says 172.21.15.12:45400) 2024-09-11T01:30:31.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe35e7fc640 1 -- 172.21.15.12:0/3803047529 learned_addr learned my addr 172.21.15.12:0/3803047529 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:30:31.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe35e7fc640 10 monclient(hunting): get_auth_request con 0x7fe3680a3270 auth_method 0 2024-09-11T01:30:31.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe35e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:31.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe35e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:31.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe35e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:31.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe3677fe640 1 --2- 172.21.15.12:0/3803047529 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe368135c50 0x7fe368122790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:31.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe35e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:31.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe35e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:31.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe35e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:31.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe3677fe640 10 monclient(hunting): get_auth_request con 0x7fe368135c50 auth_method 0 2024-09-11T01:30:31.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe3677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:31.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe3677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:31.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe3677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:31.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe3677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:31.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe3677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:31.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe3677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:31.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe35e7fc640 10 monclient(hunting): handle_auth_done global_id 4493 payload 824 2024-09-11T01:30:31.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe35e7fc640 1 -- 172.21.15.12:0/3803047529 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe368135c50 msgr2=0x7fe368122790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:31.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe35e7fc640 1 --2- 172.21.15.12:0/3803047529 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe368135c50 0x7fe368122790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:31.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe35e7fc640 1 -- 172.21.15.12:0/3803047529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3680a2e90 msgr2=0x7fe3681cb180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:30:31.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe35e7fc640 1 --2- 172.21.15.12:0/3803047529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3680a2e90 0x7fe3681cb180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:31.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe35e7fc640 10 monclient: _finish_hunting 0 2024-09-11T01:30:31.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe35e7fc640 1 monclient: found mon.noname-c 2024-09-11T01:30:31.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe35e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:31.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe35e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:30:31.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.946+0000 7fe35e7fc640 1 -- 172.21.15.12:0/3803047529 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3680a5800 con 0x7fe3680a3270 2024-09-11T01:30:31.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.949+0000 7fe35e7fc640 1 -- 172.21.15.12:0/3803047529 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3680a3270 msgr2=0x7fe3681cb6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:31.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.949+0000 7fe35e7fc640 1 --2- 172.21.15.12:0/3803047529 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3680a3270 0x7fe3681cb6c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe350009960 tx=0x7fe3500046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:31.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.949+0000 7fe35ffff640 1 -- 172.21.15.12:0/3803047529 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe350041070 con 0x7fe3680a3270 2024-09-11T01:30:31.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.949+0000 7fe35ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:31.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.949+0000 7fe35ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:30:31.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.949+0000 7fe35ffff640 10 monclient: dump: 2024-09-11T01:30:31.962 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:31.962 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:31.962 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:31.962 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:31.962 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:31.962 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:31.962 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:31.963 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:31.963 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:31.963 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:31.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.949+0000 7fe35ffff640 10 monclient: _finish_auth 0 2024-09-11T01:30:31.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.949+0000 7fe35ffff640 10 monclient: _check_auth_tickets 2024-09-11T01:30:31.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.949+0000 7fe35ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:31.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.949+0000 7fe35ffff640 1 -- 172.21.15.12:0/3803047529 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe35003c880 con 0x7fe3680a3270 2024-09-11T01:30:31.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.949+0000 7fe35ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:31.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.949+0000 7fe35ffff640 1 -- 172.21.15.12:0/3803047529 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe350037040 con 0x7fe3680a3270 2024-09-11T01:30:31.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.949+0000 7fe36d33d640 5 monclient: authenticate success, global_id 4493 2024-09-11T01:30:31.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.949+0000 7fe36d33d640 10 monclient: _renew_subs 2024-09-11T01:30:31.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.949+0000 7fe36d33d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:31.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.949+0000 7fe36d33d640 1 -- 172.21.15.12:0/3803047529 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe368124200 con 0x7fe3680a3270 2024-09-11T01:30:31.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.949+0000 7fe365ffb640 4 set_mon_vals no callback set 2024-09-11T01:30:31.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.949+0000 7fe35ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:31.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.949+0000 7fe35ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:31.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.949+0000 7fe35ffff640 10 monclient: dump: 2024-09-11T01:30:31.964 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:31.964 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:31.964 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:31.964 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:31.964 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:31.964 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:31.964 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:31.964 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:31.964 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:31.964 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:31.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.949+0000 7fe36d33d640 10 monclient: _renew_subs 2024-09-11T01:30:31.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.949+0000 7fe36d33d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:31.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.949+0000 7fe36d33d640 1 -- 172.21.15.12:0/3803047529 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe368125fa0 con 0x7fe3680a3270 2024-09-11T01:30:31.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.950+0000 7fe35ffff640 1 -- 172.21.15.12:0/3803047529 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fe35003cc90 con 0x7fe3680a3270 2024-09-11T01:30:31.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.950+0000 7fe36d33d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:31.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.950+0000 7fe36d33d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:31.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.950+0000 7fe36d33d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:31.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.950+0000 7fe36d33d640 1 -- 172.21.15.12:0/3803047529 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3681369e0 con 0x7fe3680a3270 2024-09-11T01:30:31.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.950+0000 7fe35ffff640 1 --2- 172.21.15.12:0/3803047529 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe34803e1f0 0x7fe3480406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:31.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.950+0000 7fe35ffff640 1 -- 172.21.15.12:0/3803047529 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(41..41 src has 1..41) ==== 4739+0+0 (secure 0 0 0) 0x7fe3500734a0 con 0x7fe3680a3270 2024-09-11T01:30:31.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.950+0000 7fe366ffd640 1 --2- 172.21.15.12:0/3803047529 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe34803e1f0 0x7fe3480406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:31.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.950+0000 7fe366ffd640 10 monclient: get_auth_request con 0x7fe34803e1f0 auth_method 0 2024-09-11T01:30:31.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.953+0000 7fe366ffd640 1 --2- 172.21.15.12:0/3803047529 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe34803e1f0 0x7fe3480406b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe368124cf0 tx=0x7fe358006d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:31.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.955+0000 7fe35ffff640 1 -- 172.21.15.12:0/3803047529 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe35004b070 con 0x7fe3680a3270 2024-09-11T01:30:31.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.955+0000 7fe35ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:31.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:31.955+0000 7fe35ffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:30:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:32.212+0000 7fe36d33d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:30:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:32.212+0000 7fe36d33d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:30:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:32.212+0000 7fe36d33d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:32.212+0000 7fe36d33d640 1 -- 172.21.15.12:0/3803047529 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe3680698e0 con 0x7fe3680a3270 2024-09-11T01:30:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:32.213+0000 7fe35ffff640 1 -- 172.21.15.12:0/3803047529 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fe350071ec0 con 0x7fe3680a3270 2024-09-11T01:30:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:32.213+0000 7fe35ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:30:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:32.213+0000 7fe35ffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:30:32.215 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:30:32.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:32.215+0000 7fe36d33d640 1 -- 172.21.15.12:0/3803047529 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe34803e1f0 msgr2=0x7fe3480406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:32.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:32.215+0000 7fe36d33d640 1 --2- 172.21.15.12:0/3803047529 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe34803e1f0 0x7fe3480406b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe368124cf0 tx=0x7fe358006d30 comp rx=0 tx=0).stop 2024-09-11T01:30:32.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:32.215+0000 7fe36d33d640 10 monclient: shutdown 2024-09-11T01:30:32.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:32.215+0000 7fe36d33d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:32.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:32.215+0000 7fe36d33d640 1 -- 172.21.15.12:0/3803047529 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3680a3270 msgr2=0x7fe3681cb6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:32.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:32.215+0000 7fe36d33d640 1 --2- 172.21.15.12:0/3803047529 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3680a3270 0x7fe3681cb6c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe350009960 tx=0x7fe3500046c0 comp rx=0 tx=0).stop 2024-09-11T01:30:32.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:32.215+0000 7fe36d33d640 1 -- 172.21.15.12:0/3803047529 shutdown_connections 2024-09-11T01:30:32.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:32.215+0000 7fe36d33d640 1 --2- 172.21.15.12:0/3803047529 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe368135c50 0x7fe368122790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:32.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:32.215+0000 7fe36d33d640 1 --2- 172.21.15.12:0/3803047529 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3680a3270 0x7fe3681cb6c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:32.215+0000 7fe36d33d640 1 --2- 172.21.15.12:0/3803047529 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe34803e1f0 0x7fe3480406b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:32.215+0000 7fe36d33d640 1 --2- 172.21.15.12:0/3803047529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3680a2e90 0x7fe3681cb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:32.215+0000 7fe36d33d640 1 -- 172.21.15.12:0/3803047529 >> 172.21.15.12:0/3803047529 conn(0x7fe3680a8490 msgr2=0x7fe3680a7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:32.215+0000 7fe36d33d640 1 -- 172.21.15.12:0/3803047529 shutdown_connections 2024-09-11T01:30:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:32.215+0000 7fe36d33d640 1 -- 172.21.15.12:0/3803047529 wait complete. 2024-09-11T01:30:35.228 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:30:35.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.340+0000 7f1eb6f24640 10 monclient: get_monmap_and_config 2024-09-11T01:30:35.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.340+0000 7f1eb6f24640 10 monclient: build_initial_monmap 2024-09-11T01:30:35.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.340+0000 7f1eb6f24640 10 monclient: monmap: 2024-09-11T01:30:35.342 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:35.342 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:35.342 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:30:35.341003+0000 2024-09-11T01:30:35.342 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:30:35.341003+0000 2024-09-11T01:30:35.342 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:35.342 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:35.342 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:30:35.342 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:30:35.343 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:30:35.343 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:30:35.343 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:30:35.343 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:35.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.341+0000 7f1eb6f24640 1 Processor -- start 2024-09-11T01:30:35.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.341+0000 7f1eb6f24640 1 -- start start 2024-09-11T01:30:35.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.341+0000 7f1eb6f24640 10 monclient: init 2024-09-11T01:30:35.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.341+0000 7f1eb6f24640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:35.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.341+0000 7f1eb6f24640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-11T01:30:35.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.341+0000 7f1eb6f24640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1eb0138230 addr v1:172.21.15.71:6789/0 2024-09-11T01:30:35.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.341+0000 7f1eb6f24640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1eb009d030 addr v1:172.21.15.12:6789/0 2024-09-11T01:30:35.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.341+0000 7f1eb6f24640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1eb009d460 0x7f1eb01407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:35.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.341+0000 7f1eb6f24640 10 monclient(hunting): picked mon.noname-a con 0x7f1eb009d460 addr v2:172.21.15.12:3300/0 2024-09-11T01:30:35.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.341+0000 7f1eb6f24640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1eb00a6e30 con 0x7f1eb009d030 2024-09-11T01:30:35.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.341+0000 7f1eb6f24640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1eb009ba20 con 0x7f1eb0138230 2024-09-11T01:30:35.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.341+0000 7f1eb6f24640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:35.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.341+0000 7f1eb6f24640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f1eb009c9c0 con 0x7f1eb009d460 2024-09-11T01:30:35.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.341+0000 7f1eb6f24640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:35.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.341+0000 7f1eb6f24640 10 monclient(hunting): authenticate will time out at 841.783760s 2024-09-11T01:30:35.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.341+0000 7f1eb54c1640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7f1eb009d460 msgr2=0x7f1eb01407c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:30:35.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.341+0000 7f1eaffff640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7f1eb009d030 legacy=0x7f1eb013d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:30:35.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.341+0000 7f1eb54c1640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1eb009d460 0x7f1eb01407c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:30:35.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.341+0000 7f1eb4cc0640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f1eb0138230 0x7f1eb0138610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.12:56432/0 (socket says 172.21.15.12:56432) 2024-09-11T01:30:35.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.342+0000 7f1eb4cc0640 1 -- 172.21.15.12:0/4014972898 learned_addr learned my addr 172.21.15.12:0/4014972898 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:30:35.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.344+0000 7f1eb4cc0640 1 -- 172.21.15.12:0/4014972898 >> v1:172.21.15.71:6789/0 conn(0x7f1eb0138230 legacy=0x7f1eb0138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:35.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.542+0000 7f1eb54c1640 1 -- 172.21.15.12:0/4014972898 >> v2:172.21.15.12:3300/0 conn(0x7f1eb009d460 msgr2=0x7f1eb01407c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:30:35.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.542+0000 7f1eaffff640 1 -- 172.21.15.12:0/4014972898 >> v1:172.21.15.12:6789/0 conn(0x7f1eb009d030 legacy=0x7f1eb013d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:30:35.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.542+0000 7f1eb54c1640 1 --2- 172.21.15.12:0/4014972898 >> v2:172.21.15.12:3300/0 conn(0x7f1eb009d460 0x7f1eb01407c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:30:35.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.943+0000 7f1eb54c1640 1 -- 172.21.15.12:0/4014972898 >> v2:172.21.15.12:3300/0 conn(0x7f1eb009d460 msgr2=0x7f1eb01407c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:30:35.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.943+0000 7f1eaffff640 1 -- 172.21.15.12:0/4014972898 >> v1:172.21.15.12:6789/0 conn(0x7f1eb009d030 legacy=0x7f1eb013d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:30:35.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:35.943+0000 7f1eb54c1640 1 --2- 172.21.15.12:0/4014972898 >> v2:172.21.15.12:3300/0 conn(0x7f1eb009d460 0x7f1eb01407c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:30:36.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:36.744+0000 7f1eaffff640 1 -- 172.21.15.12:0/4014972898 >> v1:172.21.15.12:6789/0 conn(0x7f1eb009d030 legacy=0x7f1eb013d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:30:36.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:36.745+0000 7f1eb54c1640 1 -- 172.21.15.12:0/4014972898 >> v2:172.21.15.12:3300/0 conn(0x7f1eb009d460 msgr2=0x7f1eb01407c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:30:36.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:36.745+0000 7f1eb54c1640 1 --2- 172.21.15.12:0/4014972898 >> v2:172.21.15.12:3300/0 conn(0x7f1eb009d460 0x7f1eb01407c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:30:38.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eae7fc640 10 monclient(hunting): tick 2024-09-11T01:30:38.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eae7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:30:38.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eae7fc640 1 monclient(hunting): continuing hunt 2024-09-11T01:30:38.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eae7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:30:38.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eae7fc640 1 -- 172.21.15.12:0/4014972898 >> v2:172.21.15.12:3300/0 conn(0x7f1eb009d460 msgr2=0x7f1eb01407c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:30:38.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eae7fc640 1 --2- 172.21.15.12:0/4014972898 >> v2:172.21.15.12:3300/0 conn(0x7f1eb009d460 0x7f1eb01407c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:38.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eae7fc640 1 -- 172.21.15.12:0/4014972898 >> v1:172.21.15.71:6789/0 conn(0x7f1eb0138230 legacy=0x7f1eb0138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:38.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eae7fc640 1 -- 172.21.15.12:0/4014972898 >> v1:172.21.15.12:6789/0 conn(0x7f1eb009d030 legacy=0x7f1eb013d090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:30:38.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eae7fc640 10 monclient: _add_conns ranks=[2,4] 2024-09-11T01:30:38.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eae7fc640 1 --2- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c003740 0x7f1e9c005b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:38.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eae7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f1e9c003740 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:38.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eae7fc640 1 --2- 172.21.15.12:0/4014972898 >> v2:172.21.15.71:3300/0 conn(0x7f1e9c006270 0x7f1e9c008730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:38.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eae7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f1e9c006270 addr v2:172.21.15.71:3300/0 2024-09-11T01:30:38.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eae7fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eae7fc640 1 -- 172.21.15.12:0/4014972898 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f1eb009c9c0 con 0x7f1e9c006270 2024-09-11T01:30:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eae7fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eae7fc640 1 -- 172.21.15.12:0/4014972898 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1e9c008df0 con 0x7f1e9c003740 2024-09-11T01:30:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eae7fc640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eb4cc0640 1 --2- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c003740 0x7f1e9c005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eb4cc0640 10 monclient(hunting): get_auth_request con 0x7f1e9c003740 auth_method 0 2024-09-11T01:30:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eb4cc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eb4cc0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eb4cc0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eaffff640 1 --2- 172.21.15.12:0/4014972898 >> v2:172.21.15.71:3300/0 conn(0x7f1e9c006270 0x7f1e9c008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eb4cc0640 1 -- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c003740 msgr2=0x7f1e9c005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:30:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eb4cc0640 1 -- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c003740 msgr2=0x7f1e9c005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:30:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eb4cc0640 1 --2- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c003740 0x7f1e9c005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:30:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eaffff640 10 monclient(hunting): get_auth_request con 0x7f1e9c006270 auth_method 0 2024-09-11T01:30:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eaffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eaffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eb4cc0640 1 --2- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c003740 0x7f1e9c005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:30:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eaffff640 1 -- 172.21.15.12:0/4014972898 >> v2:172.21.15.71:3300/0 conn(0x7f1e9c006270 msgr2=0x7f1e9c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:30:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eaffff640 1 -- 172.21.15.12:0/4014972898 >> v2:172.21.15.71:3300/0 conn(0x7f1e9c006270 msgr2=0x7f1e9c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:30:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eaffff640 1 --2- 172.21.15.12:0/4014972898 >> v2:172.21.15.71:3300/0 conn(0x7f1e9c006270 0x7f1e9c008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:30:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.342+0000 7f1eaffff640 1 --2- 172.21.15.12:0/4014972898 >> v2:172.21.15.71:3300/0 conn(0x7f1e9c006270 0x7f1e9c008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:30:38.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.543+0000 7f1eb4cc0640 1 --2- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c003740 0x7f1e9c005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:38.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.543+0000 7f1eb4cc0640 10 monclient(hunting): get_auth_request con 0x7f1e9c003740 auth_method 0 2024-09-11T01:30:38.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.543+0000 7f1eb4cc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:38.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.543+0000 7f1eb4cc0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:38.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.543+0000 7f1eb4cc0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:30:38.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.543+0000 7f1eb4cc0640 1 -- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c003740 msgr2=0x7f1e9c005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:30:38.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.543+0000 7f1eb4cc0640 1 -- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c003740 msgr2=0x7f1e9c005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:30:38.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.543+0000 7f1eb4cc0640 1 --2- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c003740 0x7f1e9c005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:30:38.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.543+0000 7f1eaffff640 1 --2- 172.21.15.12:0/4014972898 >> v2:172.21.15.71:3300/0 conn(0x7f1e9c006270 0x7f1e9c008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:38.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.543+0000 7f1eaffff640 10 monclient(hunting): get_auth_request con 0x7f1e9c006270 auth_method 0 2024-09-11T01:30:38.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.543+0000 7f1eaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:38.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.543+0000 7f1eaffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:38.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.543+0000 7f1eaffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:30:38.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.543+0000 7f1eb4cc0640 1 --2- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c003740 0x7f1e9c005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:30:38.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.543+0000 7f1eaffff640 1 -- 172.21.15.12:0/4014972898 >> v2:172.21.15.71:3300/0 conn(0x7f1e9c006270 msgr2=0x7f1e9c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:30:38.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.543+0000 7f1eaffff640 1 -- 172.21.15.12:0/4014972898 >> v2:172.21.15.71:3300/0 conn(0x7f1e9c006270 msgr2=0x7f1e9c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:30:38.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.543+0000 7f1eaffff640 1 --2- 172.21.15.12:0/4014972898 >> v2:172.21.15.71:3300/0 conn(0x7f1e9c006270 0x7f1e9c008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:30:38.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.543+0000 7f1eaffff640 1 --2- 172.21.15.12:0/4014972898 >> v2:172.21.15.71:3300/0 conn(0x7f1e9c006270 0x7f1e9c008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:30:38.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.945+0000 7f1eb4cc0640 1 --2- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c003740 0x7f1e9c005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:38.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.945+0000 7f1eb4cc0640 10 monclient(hunting): get_auth_request con 0x7f1e9c003740 auth_method 0 2024-09-11T01:30:38.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.945+0000 7f1eb4cc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:38.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.945+0000 7f1eb4cc0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:38.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.945+0000 7f1eb4cc0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:30:38.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.945+0000 7f1eaffff640 1 --2- 172.21.15.12:0/4014972898 >> v2:172.21.15.71:3300/0 conn(0x7f1e9c006270 0x7f1e9c008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:38.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.945+0000 7f1eaffff640 10 monclient(hunting): get_auth_request con 0x7f1e9c006270 auth_method 0 2024-09-11T01:30:38.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.945+0000 7f1eaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:38.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.945+0000 7f1eaffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:38.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.945+0000 7f1eaffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:30:38.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.945+0000 7f1eb4cc0640 1 -- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c003740 msgr2=0x7f1e9c005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:30:38.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.945+0000 7f1eb4cc0640 1 -- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c003740 msgr2=0x7f1e9c005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:30:38.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.945+0000 7f1eb4cc0640 1 --2- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c003740 0x7f1e9c005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:30:38.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.945+0000 7f1eb4cc0640 1 --2- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c003740 0x7f1e9c005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:30:38.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.945+0000 7f1eaffff640 1 -- 172.21.15.12:0/4014972898 >> v2:172.21.15.71:3300/0 conn(0x7f1e9c006270 msgr2=0x7f1e9c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:30:38.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.945+0000 7f1eaffff640 1 -- 172.21.15.12:0/4014972898 >> v2:172.21.15.71:3300/0 conn(0x7f1e9c006270 msgr2=0x7f1e9c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:30:38.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.945+0000 7f1eaffff640 1 --2- 172.21.15.12:0/4014972898 >> v2:172.21.15.71:3300/0 conn(0x7f1e9c006270 0x7f1e9c008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:30:38.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:38.945+0000 7f1eaffff640 1 --2- 172.21.15.12:0/4014972898 >> v2:172.21.15.71:3300/0 conn(0x7f1e9c006270 0x7f1e9c008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:30:39.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:39.746+0000 7f1eb4cc0640 1 --2- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c003740 0x7f1e9c005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:39.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:39.746+0000 7f1eb4cc0640 10 monclient(hunting): get_auth_request con 0x7f1e9c003740 auth_method 0 2024-09-11T01:30:39.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:39.746+0000 7f1eb4cc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:39.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:39.746+0000 7f1eb4cc0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:39.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:39.746+0000 7f1eb4cc0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:30:39.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:39.746+0000 7f1eb4cc0640 1 -- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c003740 msgr2=0x7f1e9c005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:30:39.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:39.746+0000 7f1eb4cc0640 1 -- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c003740 msgr2=0x7f1e9c005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:30:39.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:39.746+0000 7f1eb4cc0640 1 --2- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c003740 0x7f1e9c005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:30:39.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:39.746+0000 7f1eb4cc0640 1 --2- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c003740 0x7f1e9c005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:30:39.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:39.746+0000 7f1eaffff640 1 --2- 172.21.15.12:0/4014972898 >> v2:172.21.15.71:3300/0 conn(0x7f1e9c006270 0x7f1e9c008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:39.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:39.746+0000 7f1eaffff640 10 monclient(hunting): get_auth_request con 0x7f1e9c006270 auth_method 0 2024-09-11T01:30:39.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:39.746+0000 7f1eaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:39.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:39.746+0000 7f1eaffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:39.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:39.746+0000 7f1eaffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:30:39.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:39.746+0000 7f1eaffff640 1 -- 172.21.15.12:0/4014972898 >> v2:172.21.15.71:3300/0 conn(0x7f1e9c006270 msgr2=0x7f1e9c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:30:39.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:39.746+0000 7f1eaffff640 1 -- 172.21.15.12:0/4014972898 >> v2:172.21.15.71:3300/0 conn(0x7f1e9c006270 msgr2=0x7f1e9c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:30:39.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:39.746+0000 7f1eaffff640 1 --2- 172.21.15.12:0/4014972898 >> v2:172.21.15.71:3300/0 conn(0x7f1e9c006270 0x7f1e9c008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:30:39.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:39.746+0000 7f1eaffff640 1 --2- 172.21.15.12:0/4014972898 >> v2:172.21.15.71:3300/0 conn(0x7f1e9c006270 0x7f1e9c008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:30:40.051 INFO:tasks.rados.rados.0.smithi071.stdout:134: done (4 left) 2024-09-11T01:30:40.051 INFO:tasks.rados.rados.0.smithi071.stdout:135: snap_remove snap 2 2024-09-11T01:30:41.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eae7fc640 10 monclient(hunting): tick 2024-09-11T01:30:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eae7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:30:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eae7fc640 1 monclient(hunting): continuing hunt 2024-09-11T01:30:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eae7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:30:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eae7fc640 1 -- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c003740 msgr2=0x7f1e9c005b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:30:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eae7fc640 1 --2- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c003740 0x7f1e9c005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eae7fc640 1 -- 172.21.15.12:0/4014972898 >> v2:172.21.15.71:3300/0 conn(0x7f1e9c006270 msgr2=0x7f1e9c008730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:30:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eae7fc640 1 --2- 172.21.15.12:0/4014972898 >> v2:172.21.15.71:3300/0 conn(0x7f1e9c006270 0x7f1e9c008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eae7fc640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-11T01:30:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eb54c1640 1 -- 172.21.15.12:0/4014972898 reap_dead start 2024-09-11T01:30:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eae7fc640 1 --2- 172.21.15.12:0/4014972898 >> v2:172.21.15.12:3300/0 conn(0x7f1eb009d460 0x7f1e9c00b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eae7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f1eb009d460 addr v2:172.21.15.12:3300/0 2024-09-11T01:30:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eae7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1e9c003b80 addr v1:172.21.15.12:6789/0 2024-09-11T01:30:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eae7fc640 1 --2- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c007e40 0x7f1e9c0082f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eae7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f1e9c007e40 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eae7fc640 1 -- 172.21.15.12:0/4014972898 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1eb00a66a0 con 0x7f1e9c003b80 2024-09-11T01:30:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eae7fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eb4cc0640 1 -- 172.21.15.12:0/4014972898 >> v2:172.21.15.12:3300/0 conn(0x7f1eb009d460 msgr2=0x7f1e9c00b240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:30:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eae7fc640 1 -- 172.21.15.12:0/4014972898 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f1eb009c9c0 con 0x7f1eb009d460 2024-09-11T01:30:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eae7fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eae7fc640 1 -- 172.21.15.12:0/4014972898 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1e9c008df0 con 0x7f1e9c007e40 2024-09-11T01:30:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eaffff640 1 -- 172.21.15.12:0/4014972898 >> v1:172.21.15.12:6789/0 conn(0x7f1e9c003b80 legacy=0x7f1e9c006720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:30:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eae7fc640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eb4cc0640 1 --2- 172.21.15.12:0/4014972898 >> v2:172.21.15.12:3300/0 conn(0x7f1eb009d460 0x7f1e9c00b240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:30:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eb54c1640 1 --2- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c007e40 0x7f1e9c0082f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eb54c1640 10 monclient(hunting): get_auth_request con 0x7f1e9c007e40 auth_method 0 2024-09-11T01:30:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eb54c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eb54c1640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eb54c1640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eb54c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eb54c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eb54c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eb54c1640 10 monclient(hunting): handle_auth_done global_id 4496 payload 274 2024-09-11T01:30:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eb54c1640 1 -- 172.21.15.12:0/4014972898 >> v2:172.21.15.12:3300/0 conn(0x7f1eb009d460 msgr2=0x7f1e9c00b240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:30:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eb54c1640 1 --2- 172.21.15.12:0/4014972898 >> v2:172.21.15.12:3300/0 conn(0x7f1eb009d460 0x7f1e9c00b240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eb54c1640 1 -- 172.21.15.12:0/4014972898 >> v1:172.21.15.12:6789/0 conn(0x7f1e9c003b80 legacy=0x7f1e9c006720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:30:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eb54c1640 10 monclient: _finish_hunting 0 2024-09-11T01:30:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eb54c1640 1 monclient: found mon.noname-b 2024-09-11T01:30:41.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eb54c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:41.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eb54c1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:30:41.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.342+0000 7f1eb54c1640 1 -- 172.21.15.12:0/4014972898 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eb00a6e30 con 0x7f1e9c007e40 2024-09-11T01:30:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.345+0000 7f1eb54c1640 1 -- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c007e40 msgr2=0x7f1e9c0082f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.345+0000 7f1eb54c1640 1 --2- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c007e40 0x7f1e9c0082f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0069f20 tx=0x7f1ea802a970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c8c0ec8cc0e3492 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.346+0000 7f1eaf7fe640 1 -- 172.21.15.12:0/4014972898 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea8008040 con 0x7f1e9c007e40 2024-09-11T01:30:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.346+0000 7f1eaf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.346+0000 7f1eaf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:30:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.346+0000 7f1eaf7fe640 10 monclient: dump: 2024-09-11T01:30:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:41.356 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.346+0000 7f1eaf7fe640 10 monclient: _finish_auth 0 2024-09-11T01:30:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.346+0000 7f1eaf7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:30:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.346+0000 7f1eaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.346+0000 7f1eaf7fe640 1 -- 172.21.15.12:0/4014972898 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ea802e7f0 con 0x7f1e9c007e40 2024-09-11T01:30:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.346+0000 7f1eb6f24640 5 monclient: authenticate success, global_id 4496 2024-09-11T01:30:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.346+0000 7f1eb6f24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:30:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.346+0000 7f1eaf7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.346+0000 7f1eaf7fe640 1 -- 172.21.15.12:0/4014972898 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea802ebe0 con 0x7f1e9c007e40 2024-09-11T01:30:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.346+0000 7f1eb6f24640 10 monclient: get_monmap_and_config success 2024-09-11T01:30:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.346+0000 7f1eaf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.346+0000 7f1eb6f24640 4 set_mon_vals no callback set 2024-09-11T01:30:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.346+0000 7f1eaf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.346+0000 7f1eaf7fe640 10 monclient: dump: 2024-09-11T01:30:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:41.358 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:41.358 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:41.358 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:41.358 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:41.358 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:41.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.347+0000 7f1eb6f24640 10 monclient: shutdown 2024-09-11T01:30:41.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.347+0000 7f1eb6f24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:41.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.347+0000 7f1eb6f24640 1 -- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c007e40 msgr2=0x7f1e9c0082f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:41.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.347+0000 7f1eb6f24640 1 --2- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c007e40 0x7f1e9c0082f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0069f20 tx=0x7f1ea802a970 comp rx=0 tx=0).stop 2024-09-11T01:30:41.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.347+0000 7f1eb6f24640 1 -- 172.21.15.12:0/4014972898 shutdown_connections 2024-09-11T01:30:41.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.347+0000 7f1eb6f24640 1 --2- 172.21.15.12:0/4014972898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e9c007e40 0x7f1e9c0082f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.347+0000 7f1eb6f24640 1 --2- 172.21.15.12:0/4014972898 >> v2:172.21.15.12:3300/0 conn(0x7f1eb009d460 0x7f1e9c00b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.347+0000 7f1eb6f24640 1 -- 172.21.15.12:0/4014972898 >> 172.21.15.12:0/4014972898 conn(0x7f1eb00a7120 msgr2=0x7f1eb00a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:41.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.347+0000 7f1eb6f24640 1 -- 172.21.15.12:0/4014972898 shutdown_connections 2024-09-11T01:30:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.347+0000 7f1eb6f24640 1 -- 172.21.15.12:0/4014972898 wait complete. 2024-09-11T01:30:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.348+0000 7f1eb6f24640 10 monclient: build_initial_monmap 2024-09-11T01:30:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.348+0000 7f1eb6f24640 10 monclient: monmap: 2024-09-11T01:30:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:30:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:30:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:30:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:30:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:30:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:30:41.359 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:41.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.348+0000 7f1eb6f24640 1 Processor -- start 2024-09-11T01:30:41.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.348+0000 7f1eb6f24640 1 -- start start 2024-09-11T01:30:41.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.348+0000 7f1eb6f24640 10 monclient: init 2024-09-11T01:30:41.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb6f24640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:41.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb6f24640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:30:41.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb6f24640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e9c003b80 0x7f1eb01cd210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:41.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb6f24640 10 monclient(hunting): picked mon.noname-a con 0x7f1e9c003b80 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:30:41.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb6f24640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1e9c007e40 0x7f1eb0137200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:41.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb6f24640 10 monclient(hunting): picked mon.noname-b con 0x7f1e9c007e40 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:30:41.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb6f24640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1eb009d460 0x7f1eb01211e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:41.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb6f24640 10 monclient(hunting): picked mon.noname-c con 0x7f1eb009d460 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:41.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb6f24640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:41.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb6f24640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb0138790 con 0x7f1e9c003b80 2024-09-11T01:30:41.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb6f24640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:41.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb6f24640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb0138610 con 0x7f1e9c007e40 2024-09-11T01:30:41.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb6f24640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:41.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb6f24640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1eb0138910 con 0x7f1eb009d460 2024-09-11T01:30:41.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb6f24640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:41.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb6f24640 10 monclient(hunting): authenticate will time out at 847.791451s 2024-09-11T01:30:41.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb4cc0640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e9c003b80 msgr2=0x7f1eb01cd210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:30:41.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb4cc0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e9c003b80 0x7f1eb01cd210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:30:41.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb54c1640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1eb009d460 0x7f1eb01211e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:41.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb54c1640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1eb009d460 0x7f1eb01211e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:38808/0 (socket says 172.21.15.12:38808) 2024-09-11T01:30:41.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb54c1640 1 -- 172.21.15.12:0/4001914056 learned_addr learned my addr 172.21.15.12:0/4001914056 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:30:41.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb54c1640 10 monclient(hunting): get_auth_request con 0x7f1eb009d460 auth_method 0 2024-09-11T01:30:41.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb54c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:41.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb54c1640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb54c1640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb54c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb54c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb54c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eaffff640 1 --2- 172.21.15.12:0/4001914056 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1e9c007e40 0x7f1eb0137200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eaffff640 10 monclient(hunting): get_auth_request con 0x7f1e9c007e40 auth_method 0 2024-09-11T01:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eaffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eaffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eaffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eaffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eaffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb54c1640 10 monclient(hunting): handle_auth_done global_id 4499 payload 824 2024-09-11T01:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb54c1640 1 -- 172.21.15.12:0/4001914056 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1e9c007e40 msgr2=0x7f1eb0137200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.349+0000 7f1eb54c1640 1 --2- 172.21.15.12:0/4001914056 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1e9c007e40 0x7f1eb0137200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.350+0000 7f1eb54c1640 1 -- 172.21.15.12:0/4001914056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e9c003b80 msgr2=0x7f1eb01cd210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:30:41.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.350+0000 7f1eb54c1640 1 --2- 172.21.15.12:0/4001914056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e9c003b80 0x7f1eb01cd210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.350+0000 7f1eb54c1640 10 monclient: _finish_hunting 0 2024-09-11T01:30:41.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.350+0000 7f1eb54c1640 1 monclient: found mon.noname-c 2024-09-11T01:30:41.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.350+0000 7f1eb54c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:41.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.350+0000 7f1eb54c1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:30:41.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.350+0000 7f1eb54c1640 1 -- 172.21.15.12:0/4001914056 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eb0122960 con 0x7f1eb009d460 2024-09-11T01:30:41.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.352+0000 7f1eb54c1640 1 -- 172.21.15.12:0/4001914056 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1eb009d460 msgr2=0x7f1eb01211e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:41.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.352+0000 7f1eb54c1640 1 --2- 172.21.15.12:0/4001914056 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1eb009d460 0x7f1eb01211e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8001220 tx=0x7f1ea8006020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:41.364 INFO:tasks.rados.rados.0.smithi071.stdout:135: done (4 left) 2024-09-11T01:30:41.364 INFO:tasks.rados.rados.0.smithi071.stdout:136: snap_create 2024-09-11T01:30:41.364 INFO:tasks.rados.rados.0.smithi071.stdout:130: finishing rollback tid 0 to smithi07134777-43 2024-09-11T01:30:41.364 INFO:tasks.rados.rados.0.smithi071.stdout:130: finishing rollback tid 1 to smithi07134777-43 2024-09-11T01:30:41.365 INFO:tasks.rados.rados.0.smithi071.stdout:130: finishing rollback tid 2 to smithi07134777-43 2024-09-11T01:30:41.365 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 62 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-11T01:30:41.365 INFO:tasks.rados.rados.0.smithi071.stdout:132: expect (ObjNum 6 snap 0 seq_num 6) 2024-09-11T01:30:41.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.360+0000 7f1ead7fa640 1 -- 172.21.15.12:0/4001914056 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea8040070 con 0x7f1eb009d460 2024-09-11T01:30:41.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.361+0000 7f1ead7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:41.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.361+0000 7f1ead7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:30:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.361+0000 7f1ead7fa640 10 monclient: dump: 2024-09-11T01:30:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:41.367 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.361+0000 7f1ead7fa640 10 monclient: _finish_auth 0 2024-09-11T01:30:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.361+0000 7f1ead7fa640 10 monclient: _check_auth_tickets 2024-09-11T01:30:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.361+0000 7f1ead7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.361+0000 7f1ead7fa640 1 -- 172.21.15.12:0/4001914056 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ea8030b30 con 0x7f1eb009d460 2024-09-11T01:30:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.361+0000 7f1ead7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.361+0000 7f1ead7fa640 1 -- 172.21.15.12:0/4001914056 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea803c430 con 0x7f1eb009d460 2024-09-11T01:30:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.361+0000 7f1eb6f24640 5 monclient: authenticate success, global_id 4499 2024-09-11T01:30:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.361+0000 7f1ead7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.361+0000 7f1ead7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.361+0000 7f1ead7fa640 10 monclient: dump: 2024-09-11T01:30:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:41.369 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:41.369 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:41.369 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:41.369 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:41.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.361+0000 7f1eb6f24640 10 monclient: _renew_subs 2024-09-11T01:30:41.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.361+0000 7f1eb6f24640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:41.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.361+0000 7f1eb6f24640 1 -- 172.21.15.12:0/4001914056 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1eb00a12a0 con 0x7f1eb009d460 2024-09-11T01:30:41.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.361+0000 7f1eb6f24640 10 monclient: _renew_subs 2024-09-11T01:30:41.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.361+0000 7f1eaeffd640 4 set_mon_vals no callback set 2024-09-11T01:30:41.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.361+0000 7f1eb6f24640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:41.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.361+0000 7f1eb6f24640 1 -- 172.21.15.12:0/4001914056 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1eb0124420 con 0x7f1eb009d460 2024-09-11T01:30:41.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.362+0000 7f1ead7fa640 1 -- 172.21.15.12:0/4001914056 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f1ea803c610 con 0x7f1eb009d460 2024-09-11T01:30:41.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.362+0000 7f1ead7fa640 1 --2- 172.21.15.12:0/4001914056 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1e9c03fa90 0x7f1e9c041f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:41.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.362+0000 7f1ead7fa640 1 -- 172.21.15.12:0/4001914056 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(43..43 src has 1..43) ==== 4799+0+0 (secure 0 0 0) 0x7f1ea8074420 con 0x7f1eb009d460 2024-09-11T01:30:41.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.362+0000 7f1eb6f24640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:41.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.362+0000 7f1eb6f24640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:41.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.362+0000 7f1eb6f24640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:41.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.362+0000 7f1eb6f24640 1 -- 172.21.15.12:0/4001914056 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1eb00a5980 con 0x7f1eb009d460 2024-09-11T01:30:41.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.362+0000 7f1eb4cc0640 1 --2- 172.21.15.12:0/4001914056 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1e9c03fa90 0x7f1e9c041f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:41.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.362+0000 7f1eb4cc0640 10 monclient: get_auth_request con 0x7f1e9c03fa90 auth_method 0 2024-09-11T01:30:41.370 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1904695 2024-09-11T01:30:41.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.365+0000 7f1eb4cc0640 1 --2- 172.21.15.12:0/4001914056 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1e9c03fa90 0x7f1e9c041f50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0000c00 tx=0x7f1ea0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:41.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.369+0000 7f1ead7fa640 1 -- 172.21.15.12:0/4001914056 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1eb00a5980 con 0x7f1eb009d460 2024-09-11T01:30:41.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.369+0000 7f1ead7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:41.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.369+0000 7f1ead7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:30:41.398 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1051013 2024-09-11T01:30:41.412 INFO:tasks.rados.rados.0.smithi071.stdout:129: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-11T01:30:41.413 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:30:41.439 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:30:41.465 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:30:41.492 INFO:tasks.rados.rados.0.smithi071.stdout:133: finishing copy_from racing read to smithi07134777-37 2024-09-11T01:30:41.492 INFO:tasks.rados.rados.0.smithi071.stdout:133: finishing copy_from to smithi07134777-37 2024-09-11T01:30:41.492 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 21 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-11T01:30:41.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.632+0000 7f1eb6f24640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:30:41.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.632+0000 7f1eb6f24640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:30:41.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.632+0000 7f1eb6f24640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:41.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.632+0000 7f1eb6f24640 1 -- 172.21.15.12:0/4001914056 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1eb0076fc0 con 0x7f1eb009d460 2024-09-11T01:30:41.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.633+0000 7f1ead7fa640 1 -- 172.21.15.12:0/4001914056 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f1eb0076fc0 con 0x7f1eb009d460 2024-09-11T01:30:41.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.633+0000 7f1ead7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:30:41.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.633+0000 7f1ead7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:30:41.635 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":18,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:30:41.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.635+0000 7f1eb6f24640 1 -- 172.21.15.12:0/4001914056 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1e9c03fa90 msgr2=0x7f1e9c041f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:41.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.635+0000 7f1eb6f24640 1 --2- 172.21.15.12:0/4001914056 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1e9c03fa90 0x7f1e9c041f50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0000c00 tx=0x7f1ea0009040 comp rx=0 tx=0).stop 2024-09-11T01:30:41.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.635+0000 7f1eb6f24640 10 monclient: shutdown 2024-09-11T01:30:41.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.635+0000 7f1eb6f24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:41.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.635+0000 7f1eb6f24640 1 -- 172.21.15.12:0/4001914056 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1eb009d460 msgr2=0x7f1eb01211e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:41.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.635+0000 7f1eb6f24640 1 --2- 172.21.15.12:0/4001914056 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1eb009d460 0x7f1eb01211e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8001220 tx=0x7f1ea8006020 comp rx=0 tx=0).stop 2024-09-11T01:30:41.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.635+0000 7f1eb6f24640 1 -- 172.21.15.12:0/4001914056 shutdown_connections 2024-09-11T01:30:41.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.635+0000 7f1eb6f24640 1 --2- 172.21.15.12:0/4001914056 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1eb009d460 0x7f1eb01211e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.635+0000 7f1eb6f24640 1 --2- 172.21.15.12:0/4001914056 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1e9c007e40 0x7f1eb0137200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.635+0000 7f1eb6f24640 1 --2- 172.21.15.12:0/4001914056 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1e9c03fa90 0x7f1e9c041f50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.635+0000 7f1eb6f24640 1 --2- 172.21.15.12:0/4001914056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e9c003b80 0x7f1eb01cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.635+0000 7f1eb6f24640 1 -- 172.21.15.12:0/4001914056 >> 172.21.15.12:0/4001914056 conn(0x7f1eb00a7120 msgr2=0x7f1eb00a3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:41.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.635+0000 7f1eb6f24640 1 -- 172.21.15.12:0/4001914056 shutdown_connections 2024-09-11T01:30:41.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.635+0000 7f1eb6f24640 1 -- 172.21.15.12:0/4001914056 wait complete. 2024-09-11T01:30:41.648 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:30:41.648 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T01:30:41.648 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:30:41.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.748+0000 7fa7d81f8640 10 monclient: get_monmap_and_config 2024-09-11T01:30:41.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.748+0000 7fa7d81f8640 10 monclient: build_initial_monmap 2024-09-11T01:30:41.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.748+0000 7fa7d81f8640 10 monclient: monmap: 2024-09-11T01:30:41.750 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:41.750 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:41.750 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:30:41.748885+0000 2024-09-11T01:30:41.750 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:30:41.748885+0000 2024-09-11T01:30:41.750 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:41.750 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:41.750 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:30:41.750 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:30:41.750 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:30:41.750 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:30:41.750 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:30:41.751 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:41.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.748+0000 7fa7d81f8640 1 Processor -- start 2024-09-11T01:30:41.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.748+0000 7fa7d81f8640 1 -- start start 2024-09-11T01:30:41.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.748+0000 7fa7d81f8640 10 monclient: init 2024-09-11T01:30:41.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d81f8640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:41.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d81f8640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-11T01:30:41.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d81f8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa7d0137e50 addr v1:172.21.15.71:6789/0 2024-09-11T01:30:41.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d81f8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d0138230 0x7fa7d0138660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:41.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d81f8640 10 monclient(hunting): picked mon.noname-b con 0x7fa7d0138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:41.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d81f8640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa7d013d380 0x7fa7d013f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:41.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d81f8640 10 monclient(hunting): picked mon.noname-c con 0x7fa7d013d380 addr v2:172.21.15.71:3300/0 2024-09-11T01:30:41.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d81f8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7d00a6580 con 0x7fa7d0137e50 2024-09-11T01:30:41.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d81f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:41.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d81f8640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fa7d00a62d0 con 0x7fa7d013d380 2024-09-11T01:30:41.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d81f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:41.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d81f8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa7d009bb40 con 0x7fa7d0138230 2024-09-11T01:30:41.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d81f8640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:41.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d81f8640 10 monclient(hunting): authenticate will time out at 848.191486s 2024-09-11T01:30:41.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d5793640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d0138230 0x7fa7d0138660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:41.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d5793640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d0138230 0x7fa7d0138660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:38824/0 (socket says 172.21.15.12:38824) 2024-09-11T01:30:41.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d5793640 1 -- 172.21.15.12:0/3997268620 learned_addr learned my addr 172.21.15.12:0/3997268620 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:30:41.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d5793640 10 monclient(hunting): get_auth_request con 0x7fa7d0138230 auth_method 0 2024-09-11T01:30:41.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d5793640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:41.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d5793640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:41.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d5793640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:41.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d5793640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:41.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d5793640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:41.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d5793640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:41.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d6795640 1 --2- 172.21.15.12:0/3997268620 >> v2:172.21.15.71:3300/0 conn(0x7fa7d013d380 0x7fa7d013f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:41.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d6795640 10 monclient(hunting): get_auth_request con 0x7fa7d013d380 auth_method 0 2024-09-11T01:30:41.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d6795640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:41.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d6795640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:41.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d6795640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:41.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d6795640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:41.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d6795640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:41.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d6795640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:41.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d5793640 10 monclient(hunting): handle_auth_done global_id 4502 payload 274 2024-09-11T01:30:41.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d5793640 1 -- 172.21.15.12:0/3997268620 >> v2:172.21.15.71:3300/0 conn(0x7fa7d013d380 msgr2=0x7fa7d013f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:30:41.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d5793640 1 --2- 172.21.15.12:0/3997268620 >> v2:172.21.15.71:3300/0 conn(0x7fa7d013d380 0x7fa7d013f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d5793640 1 -- 172.21.15.12:0/3997268620 >> v1:172.21.15.71:6789/0 conn(0x7fa7d0137e50 legacy=0x7fa7d009d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:30:41.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d5793640 10 monclient: _finish_hunting 0 2024-09-11T01:30:41.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d5793640 1 monclient: found mon.noname-b 2024-09-11T01:30:41.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d5793640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:41.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d5793640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:30:41.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.749+0000 7fa7d5793640 1 -- 172.21.15.12:0/3997268620 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7d009c4a0 con 0x7fa7d0138230 2024-09-11T01:30:41.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.752+0000 7fa7d5793640 1 -- 172.21.15.12:0/3997268620 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d0138230 msgr2=0x7fa7d0138660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:41.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.752+0000 7fa7d5793640 1 --2- 172.21.15.12:0/3997268620 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d0138230 0x7fa7d0138660 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0009870 tx=0x7fa7c002a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae4a66a2f409d416 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:41.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.752+0000 7fa7d4f92640 1 -- 172.21.15.12:0/3997268620 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c0037070 con 0x7fa7d0138230 2024-09-11T01:30:41.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.752+0000 7fa7d4f92640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:41.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.752+0000 7fa7d4f92640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:30:41.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.752+0000 7fa7d4f92640 10 monclient: dump: 2024-09-11T01:30:41.767 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:41.767 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:41.767 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.768 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.768 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:41.768 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:41.768 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:41.768 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:41.768 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:41.768 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:41.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.752+0000 7fa7d4f92640 10 monclient: _finish_auth 0 2024-09-11T01:30:41.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.752+0000 7fa7d4f92640 10 monclient: _check_auth_tickets 2024-09-11T01:30:41.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.752+0000 7fa7d4f92640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:41.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.752+0000 7fa7d81f8640 5 monclient: authenticate success, global_id 4502 2024-09-11T01:30:41.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.752+0000 7fa7d81f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:30:41.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.752+0000 7fa7d4f92640 1 -- 172.21.15.12:0/3997268620 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7c0032440 con 0x7fa7d0138230 2024-09-11T01:30:41.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.752+0000 7fa7d4f92640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:41.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.752+0000 7fa7d81f8640 10 monclient: get_monmap_and_config success 2024-09-11T01:30:41.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.752+0000 7fa7d4f92640 1 -- 172.21.15.12:0/3997268620 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c0032810 con 0x7fa7d0138230 2024-09-11T01:30:41.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.752+0000 7fa7d4f92640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:41.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.752+0000 7fa7d81f8640 4 set_mon_vals no callback set 2024-09-11T01:30:41.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.752+0000 7fa7d4f92640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:41.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.752+0000 7fa7d4f92640 10 monclient: dump: 2024-09-11T01:30:41.769 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:41.769 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:41.769 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.769 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.769 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:41.770 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:41.770 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:41.770 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:41.770 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:41.770 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:41.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.753+0000 7fa7d81f8640 10 monclient: shutdown 2024-09-11T01:30:41.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.753+0000 7fa7d81f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:41.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.753+0000 7fa7d81f8640 1 -- 172.21.15.12:0/3997268620 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d0138230 msgr2=0x7fa7d0138660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:41.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.753+0000 7fa7d81f8640 1 --2- 172.21.15.12:0/3997268620 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d0138230 0x7fa7d0138660 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0009870 tx=0x7fa7c002a0a0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.753+0000 7fa7d81f8640 1 -- 172.21.15.12:0/3997268620 shutdown_connections 2024-09-11T01:30:41.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.753+0000 7fa7d81f8640 1 --2- 172.21.15.12:0/3997268620 >> v2:172.21.15.71:3300/0 conn(0x7fa7d013d380 0x7fa7d013f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.753+0000 7fa7d81f8640 1 --2- 172.21.15.12:0/3997268620 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d0138230 0x7fa7d0138660 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.753+0000 7fa7d81f8640 1 -- 172.21.15.12:0/3997268620 >> 172.21.15.12:0/3997268620 conn(0x7fa7d00a7020 msgr2=0x7fa7d00a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:41.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.753+0000 7fa7d81f8640 1 -- 172.21.15.12:0/3997268620 shutdown_connections 2024-09-11T01:30:41.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.753+0000 7fa7d81f8640 1 -- 172.21.15.12:0/3997268620 wait complete. 2024-09-11T01:30:41.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.754+0000 7fa7d81f8640 10 monclient: build_initial_monmap 2024-09-11T01:30:41.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.754+0000 7fa7d81f8640 10 monclient: monmap: 2024-09-11T01:30:41.771 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:41.771 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:30:41.771 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:30:41.771 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:30:41.771 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:41.771 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:41.771 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:30:41.771 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:30:41.771 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:30:41.771 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:41.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.754+0000 7fa7d81f8640 1 Processor -- start 2024-09-11T01:30:41.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.754+0000 7fa7d81f8640 1 -- start start 2024-09-11T01:30:41.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.754+0000 7fa7d81f8640 10 monclient: init 2024-09-11T01:30:41.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d81f8640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:41.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d81f8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:30:41.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d81f8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7d0137e50 0x7fa7d01c9e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:41.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d81f8640 10 monclient(hunting): picked mon.noname-b con 0x7fa7d0137e50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:30:41.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d81f8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7d0138230 0x7fa7d00a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:41.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d81f8640 10 monclient(hunting): picked mon.noname-a con 0x7fa7d0138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:30:41.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d81f8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d013d380 0x7fa7d01cd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:41.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d81f8640 10 monclient(hunting): picked mon.noname-c con 0x7fa7d013d380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:41.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d81f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:41.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d81f8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d006f890 con 0x7fa7d0138230 2024-09-11T01:30:41.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d81f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:41.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d81f8640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d006f710 con 0x7fa7d0137e50 2024-09-11T01:30:41.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d81f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:41.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d81f8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa7d013fcb0 con 0x7fa7d013d380 2024-09-11T01:30:41.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d81f8640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:41.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d81f8640 10 monclient(hunting): authenticate will time out at 848.197111s 2024-09-11T01:30:41.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d5793640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7d0138230 msgr2=0x7fa7d00a42c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:30:41.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d5793640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7d0138230 0x7fa7d00a42c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:30:41.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d6795640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d013d380 0x7fa7d01cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:41.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d6795640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d013d380 0x7fa7d01cd390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:38834/0 (socket says 172.21.15.12:38834) 2024-09-11T01:30:41.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d6795640 1 -- 172.21.15.12:0/3109410243 learned_addr learned my addr 172.21.15.12:0/3109410243 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:30:41.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d6795640 10 monclient(hunting): get_auth_request con 0x7fa7d013d380 auth_method 0 2024-09-11T01:30:41.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d6795640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:41.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d6795640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:41.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d6795640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:41.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d5f94640 1 --2- 172.21.15.12:0/3109410243 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7d0137e50 0x7fa7d01c9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:41.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d6795640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:41.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d6795640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:41.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d6795640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:41.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d5f94640 10 monclient(hunting): get_auth_request con 0x7fa7d0137e50 auth_method 0 2024-09-11T01:30:41.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d5f94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:41.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d5f94640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:41.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d5f94640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:41.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d5f94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:41.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d5f94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:41.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d5f94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:41.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d6795640 10 monclient(hunting): handle_auth_done global_id 4505 payload 824 2024-09-11T01:30:41.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d6795640 1 -- 172.21.15.12:0/3109410243 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7d0137e50 msgr2=0x7fa7d01c9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:41.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d6795640 1 --2- 172.21.15.12:0/3109410243 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7d0137e50 0x7fa7d01c9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d6795640 1 -- 172.21.15.12:0/3109410243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7d0138230 msgr2=0x7fa7d00a42c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:30:41.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d6795640 1 --2- 172.21.15.12:0/3109410243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7d0138230 0x7fa7d00a42c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d6795640 10 monclient: _finish_hunting 0 2024-09-11T01:30:41.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d6795640 1 monclient: found mon.noname-c 2024-09-11T01:30:41.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d6795640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:41.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d6795640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:30:41.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.755+0000 7fa7d6795640 1 -- 172.21.15.12:0/3109410243 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7d013b6d0 con 0x7fa7d013d380 2024-09-11T01:30:41.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.757+0000 7fa7d6795640 1 -- 172.21.15.12:0/3109410243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d013d380 msgr2=0x7fa7d01cd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:41.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.757+0000 7fa7d6795640 1 --2- 172.21.15.12:0/3109410243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d013d380 0x7fa7d01cd390 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa7c8009dd0 tx=0x7fa7c800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:41.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.757+0000 7fa7c67fc640 1 -- 172.21.15.12:0/3109410243 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c8018020 con 0x7fa7d013d380 2024-09-11T01:30:41.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.757+0000 7fa7c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:41.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.757+0000 7fa7c67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:30:41.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.757+0000 7fa7c67fc640 10 monclient: dump: 2024-09-11T01:30:41.776 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:41.776 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:41.776 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.776 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.776 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:41.776 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:41.776 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:41.776 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:41.776 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:41.776 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:41.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.757+0000 7fa7c67fc640 10 monclient: _finish_auth 0 2024-09-11T01:30:41.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.757+0000 7fa7c67fc640 10 monclient: _check_auth_tickets 2024-09-11T01:30:41.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.757+0000 7fa7c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:41.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.757+0000 7fa7d81f8640 5 monclient: authenticate success, global_id 4505 2024-09-11T01:30:41.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.757+0000 7fa7c67fc640 1 -- 172.21.15.12:0/3109410243 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7c80146a0 con 0x7fa7d013d380 2024-09-11T01:30:41.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.757+0000 7fa7d81f8640 10 monclient: _renew_subs 2024-09-11T01:30:41.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.757+0000 7fa7d81f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:41.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.757+0000 7fa7d81f8640 1 -- 172.21.15.12:0/3109410243 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7d013faa0 con 0x7fa7d013d380 2024-09-11T01:30:41.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.757+0000 7fa7c67fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:41.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.757+0000 7fa7c67fc640 1 -- 172.21.15.12:0/3109410243 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c8014a90 con 0x7fa7d013d380 2024-09-11T01:30:41.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.757+0000 7fa7c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:41.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.757+0000 7fa7c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:41.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.757+0000 7fa7c67fc640 10 monclient: dump: 2024-09-11T01:30:41.777 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:41.778 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:41.778 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.778 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.778 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:41.778 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:41.778 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:41.778 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:41.778 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:41.778 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:41.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.757+0000 7fa7c7fff640 4 set_mon_vals no callback set 2024-09-11T01:30:41.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.758+0000 7fa7d81f8640 10 monclient: _renew_subs 2024-09-11T01:30:41.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.758+0000 7fa7d81f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:41.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.758+0000 7fa7d81f8640 1 -- 172.21.15.12:0/3109410243 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7d00a27d0 con 0x7fa7d013d380 2024-09-11T01:30:41.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.758+0000 7fa7d81f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:41.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.758+0000 7fa7d81f8640 1 --2- 172.21.15.12:0/3109410243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d00a4800 0x7fa7d01cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:41.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.758+0000 7fa7d81f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:41.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.758+0000 7fa7d81f8640 1 -- 172.21.15.12:0/3109410243 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa7d006a060 con 0x7fa7d00a4800 2024-09-11T01:30:41.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.758+0000 7fa7d5f94640 1 --2- 172.21.15.12:0/3109410243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d00a4800 0x7fa7d01cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:41.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.758+0000 7fa7d5f94640 10 monclient: get_auth_request con 0x7fa7d00a4800 auth_method 0 2024-09-11T01:30:41.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.758+0000 7fa7d5f94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:41.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.758+0000 7fa7d5f94640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:41.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.758+0000 7fa7d5f94640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:41.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.758+0000 7fa7d5f94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:41.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.758+0000 7fa7d5f94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:41.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.758+0000 7fa7d5f94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:41.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.759+0000 7fa7c67fc640 1 -- 172.21.15.12:0/3109410243 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fa7c8014c30 con 0x7fa7d013d380 2024-09-11T01:30:41.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.759+0000 7fa7d5f94640 10 monclient(hunting): handle_auth_done global_id 4508 payload 824 2024-09-11T01:30:41.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.759+0000 7fa7d5f94640 1 -- 172.21.15.12:0/3109410243 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa7d0138660 con 0x7fa7d00a4800 2024-09-11T01:30:41.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.759+0000 7fa7c67fc640 1 --2- 172.21.15.12:0/3109410243 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa7b403e1c0 0x7fa7b4040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:41.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.759+0000 7fa7c67fc640 1 -- 172.21.15.12:0/3109410243 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(43..43 src has 1..43) ==== 4799+0+0 (secure 0 0 0) 0x7fa7c8053bf0 con 0x7fa7d013d380 2024-09-11T01:30:41.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.759+0000 7fa7d5793640 1 --2- 172.21.15.12:0/3109410243 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa7b403e1c0 0x7fa7b4040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:41.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.759+0000 7fa7d5793640 10 monclient: get_auth_request con 0x7fa7b403e1c0 auth_method 0 2024-09-11T01:30:41.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.761+0000 7fa7d5f94640 1 -- 172.21.15.12:0/3109410243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d00a4800 msgr2=0x7fa7d01cd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:41.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.761+0000 7fa7d5f94640 1 --2- 172.21.15.12:0/3109410243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d00a4800 0x7fa7d01cd8d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc00e770 tx=0x7fa7cc00ec30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:41.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.761+0000 7fa7c67fc640 1 -- 172.21.15.12:0/3109410243 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc017070 con 0x7fa7d00a4800 2024-09-11T01:30:41.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.761+0000 7fa7c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:41.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.761+0000 7fa7c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:41.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.761+0000 7fa7c67fc640 10 monclient: dump: 2024-09-11T01:30:41.780 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:41.781 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:41.781 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.781 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.781 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:41.781 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:41.781 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:41.781 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:41.781 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:41.781 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:41.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.761+0000 7fa7c67fc640 1 -- 172.21.15.12:0/3109410243 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa7cc00c9b0 con 0x7fa7d00a4800 2024-09-11T01:30:41.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.761+0000 7fa7c67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:41.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.761+0000 7fa7c67fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:30:41.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.761+0000 7fa7c67fc640 1 -- 172.21.15.12:0/3109410243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d00a4800 msgr2=0x7fa7d01cd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:41.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.761+0000 7fa7c67fc640 1 --2- 172.21.15.12:0/3109410243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d00a4800 0x7fa7d01cd8d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc00e770 tx=0x7fa7cc00ec30 comp rx=0 tx=0).stop 2024-09-11T01:30:41.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.761+0000 7fa7c67fc640 1 -- 172.21.15.12:0/3109410243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d00a4800 msgr2=0x7fa7d01cd8d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:30:41.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.761+0000 7fa7c67fc640 1 --2- 172.21.15.12:0/3109410243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d00a4800 0x7fa7d01cd8d0 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc00e770 tx=0x7fa7cc00ec30 comp rx=0 tx=0).stop 2024-09-11T01:30:41.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.762+0000 7fa7d5793640 1 --2- 172.21.15.12:0/3109410243 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa7b403e1c0 0x7fa7b4040680 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0009870 tx=0x7fa7c002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:41.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.767+0000 7fa7d81f8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:30:41.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.767+0000 7fa7d81f8640 1 --2- 172.21.15.12:0/3109410243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d00a1600 0x7fa7d01cde10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:41.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.767+0000 7fa7d81f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:41.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.767+0000 7fa7d81f8640 1 -- 172.21.15.12:0/3109410243 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa7d0069950 con 0x7fa7d00a1600 2024-09-11T01:30:41.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.768+0000 7fa7d5f94640 1 --2- 172.21.15.12:0/3109410243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d00a1600 0x7fa7d01cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:41.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.768+0000 7fa7d5f94640 10 monclient: get_auth_request con 0x7fa7d00a1600 auth_method 0 2024-09-11T01:30:41.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.768+0000 7fa7d5f94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:41.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.768+0000 7fa7d5f94640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:41.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.768+0000 7fa7d5f94640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:41.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.768+0000 7fa7d5f94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:41.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.768+0000 7fa7d5f94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:41.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.768+0000 7fa7d5f94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:41.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.768+0000 7fa7d5f94640 10 monclient(hunting): handle_auth_done global_id 4511 payload 824 2024-09-11T01:30:41.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.768+0000 7fa7d5f94640 1 -- 172.21.15.12:0/3109410243 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa7d013c230 con 0x7fa7d00a1600 2024-09-11T01:30:41.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.770+0000 7fa7d5f94640 1 -- 172.21.15.12:0/3109410243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d00a1600 msgr2=0x7fa7d01cde10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:41.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.770+0000 7fa7d5f94640 1 --2- 172.21.15.12:0/3109410243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d00a1600 0x7fa7d01cde10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc019000 tx=0x7fa7cc0046f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:41.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.770+0000 7fa7c67fc640 1 -- 172.21.15.12:0/3109410243 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc017070 con 0x7fa7d00a1600 2024-09-11T01:30:41.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.770+0000 7fa7c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:41.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.770+0000 7fa7c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:41.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.770+0000 7fa7c67fc640 10 monclient: dump: 2024-09-11T01:30:41.785 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:41.785 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:41.785 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.785 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.786 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:41.786 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:41.786 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:41.786 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:41.786 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:41.786 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:41.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.770+0000 7fa7c67fc640 1 -- 172.21.15.12:0/3109410243 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4697 (secure 0 0 0) 0x7fa7cc006540 con 0x7fa7d00a1600 2024-09-11T01:30:41.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.770+0000 7fa7c67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:30:41.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.770+0000 7fa7c67fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:30:41.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.770+0000 7fa7c67fc640 1 -- 172.21.15.12:0/3109410243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d00a1600 msgr2=0x7fa7d01cde10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:41.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.770+0000 7fa7c67fc640 1 --2- 172.21.15.12:0/3109410243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d00a1600 0x7fa7d01cde10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc019000 tx=0x7fa7cc0046f0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.770+0000 7fa7c67fc640 1 -- 172.21.15.12:0/3109410243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d00a1600 msgr2=0x7fa7d01cde10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:30:41.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.770+0000 7fa7c67fc640 1 --2- 172.21.15.12:0/3109410243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d00a1600 0x7fa7d01cde10 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc019000 tx=0x7fa7cc0046f0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.787 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:30:41.787 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:30:41.787 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:30:41.787 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:30:41.787 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 18, 2024-09-11T01:30:41.787 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 104459, 2024-09-11T01:30:41.787 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:30:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:30:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:30:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:30:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:30:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:30:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:30:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:30:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:30:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:30:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:30:41.788 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:30:41.789 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:30:41.789 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:30:41.789 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:30:41.789 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:30:41.789 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:30:41.789 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:30:41.789 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:41.789 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:30:41.789 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:30:41.789 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:30:41.789 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:30:41.790 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:30:41.790 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:30:41.790 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:30:41.790 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:30:41.790 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:30:41.790 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:30:41.790 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:30:41.790 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:30:41.790 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:30:41.790 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:41.790 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:41.790 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:30:41.790 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:30:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:30:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:30:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:30:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:30:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:30:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:30:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:30:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:30:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:30:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:30:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:30:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:30:41.791 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:30:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:30:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:30:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:30:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:30:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:30:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:30:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:30:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:30:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:30:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:30:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:30:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:30:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:30:41.792 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:41.793 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:30:41.793 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:41.793 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:30:41.793 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:41.793 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:30:41.793 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:30:41.793 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:30:41.793 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:30:41.793 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:41.793 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:30:41.793 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:30:41.793 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:41.793 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:30:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:30:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:30:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:30:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:30:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:30:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:30:41.794 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:30:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:30:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:30:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:30:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:30:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:30:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:30:41.795 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:30:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:30:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:30:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:30:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:30:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:30:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:30:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:30:41.796 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:30:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:30:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:30:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:30:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:30:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:30:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:41.797 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:30:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:30:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:30:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:30:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:30:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:30:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:30:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:30:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:30:41.798 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:30:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:30:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:30:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:30:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:30:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:30:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:30:41.799 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:30:41.800 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:30:41.800 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:41.800 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:41.800 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:41.800 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:30:41.800 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:30:41.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.771+0000 7fa7d81f8640 1 -- 172.21.15.12:0/3109410243 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa7b403e1c0 msgr2=0x7fa7b4040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:41.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.771+0000 7fa7d81f8640 1 --2- 172.21.15.12:0/3109410243 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa7b403e1c0 0x7fa7b4040680 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0009870 tx=0x7fa7c002e000 comp rx=0 tx=0).stop 2024-09-11T01:30:41.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.771+0000 7fa7d81f8640 10 monclient: shutdown 2024-09-11T01:30:41.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.771+0000 7fa7d81f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:41.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.771+0000 7fa7d81f8640 1 -- 172.21.15.12:0/3109410243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d013d380 msgr2=0x7fa7d01cd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:41.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.771+0000 7fa7d81f8640 1 --2- 172.21.15.12:0/3109410243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d013d380 0x7fa7d01cd390 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa7c8009dd0 tx=0x7fa7c800c490 comp rx=0 tx=0).stop 2024-09-11T01:30:41.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.771+0000 7fa7d6795640 1 -- 172.21.15.12:0/3109410243 reap_dead start 2024-09-11T01:30:41.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.771+0000 7fa7d81f8640 1 -- 172.21.15.12:0/3109410243 shutdown_connections 2024-09-11T01:30:41.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.771+0000 7fa7d81f8640 1 --2- 172.21.15.12:0/3109410243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa7d013d380 0x7fa7d01cd390 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.771+0000 7fa7d81f8640 1 -- 172.21.15.12:0/3109410243 >> 172.21.15.12:0/3109410243 conn(0x7fa7d00a7020 msgr2=0x7fa7d0136ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:41.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.771+0000 7fa7d81f8640 1 -- 172.21.15.12:0/3109410243 shutdown_connections 2024-09-11T01:30:41.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.771+0000 7fa7d81f8640 1 -- 172.21.15.12:0/3109410243 wait complete. 2024-09-11T01:30:41.803 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:30:41.803 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T01:30:41.803 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:30:41.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.904+0000 7fc659008640 10 monclient: get_monmap_and_config 2024-09-11T01:30:41.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.904+0000 7fc659008640 10 monclient: build_initial_monmap 2024-09-11T01:30:41.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.904+0000 7fc659008640 10 monclient: monmap: 2024-09-11T01:30:41.906 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:41.906 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:41.906 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:30:41.905017+0000 2024-09-11T01:30:41.906 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:30:41.905017+0000 2024-09-11T01:30:41.906 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:41.906 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:41.906 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:30:41.906 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:30:41.906 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:30:41.907 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:30:41.907 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:30:41.907 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.904+0000 7fc659008640 1 Processor -- start 2024-09-11T01:30:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.904+0000 7fc659008640 1 -- start start 2024-09-11T01:30:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.904+0000 7fc659008640 10 monclient: init 2024-09-11T01:30:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc659008640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc659008640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-11T01:30:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc659008640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc6540a2db0 0x7fc65406f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc659008640 10 monclient(hunting): picked mon.noname-a con 0x7fc6540a2db0 addr v2:172.21.15.12:3300/0 2024-09-11T01:30:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc659008640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc65406ded0 addr v1:172.21.15.71:6789/0 2024-09-11T01:30:41.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc659008640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6540a38a0 0x7fc65413f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:41.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc659008640 10 monclient(hunting): picked mon.noname-b con 0x7fc6540a38a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:41.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc659008640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6540a4650 con 0x7fc65406ded0 2024-09-11T01:30:41.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc659008640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:41.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc659008640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc65406f1a0 con 0x7fc6540a2db0 2024-09-11T01:30:41.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc659008640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:41.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc659008640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc65406eec0 con 0x7fc6540a38a0 2024-09-11T01:30:41.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc659008640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:41.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc659008640 10 monclient(hunting): authenticate will time out at 848.347467s 2024-09-11T01:30:41.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc652d9d640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7fc6540a2db0 msgr2=0x7fc65406f8a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:30:41.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc652d9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc6540a2db0 0x7fc65406f8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:30:41.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc65359e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6540a38a0 0x7fc65413f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:41.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc65359e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6540a38a0 0x7fc65413f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:38860/0 (socket says 172.21.15.12:38860) 2024-09-11T01:30:41.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc65359e640 1 -- 172.21.15.12:0/1146324451 learned_addr learned my addr 172.21.15.12:0/1146324451 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:30:41.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc65359e640 10 monclient(hunting): get_auth_request con 0x7fc6540a38a0 auth_method 0 2024-09-11T01:30:41.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc65359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:41.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc65359e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:41.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc65359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:41.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc65359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:41.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc65359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:41.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc65359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:41.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc65359e640 10 monclient(hunting): handle_auth_done global_id 4514 payload 274 2024-09-11T01:30:41.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc65359e640 1 -- 172.21.15.12:0/1146324451 >> v2:172.21.15.12:3300/0 conn(0x7fc6540a2db0 msgr2=0x7fc65406f8a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:30:41.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc65359e640 1 --2- 172.21.15.12:0/1146324451 >> v2:172.21.15.12:3300/0 conn(0x7fc6540a2db0 0x7fc65406f8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc65359e640 1 -- 172.21.15.12:0/1146324451 >> v1:172.21.15.71:6789/0 conn(0x7fc65406ded0 legacy=0x7fc6540a3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:30:41.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc65359e640 10 monclient: _finish_hunting 0 2024-09-11T01:30:41.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc65359e640 1 monclient: found mon.noname-b 2024-09-11T01:30:41.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc65359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:41.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc65359e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:30:41.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.905+0000 7fc65359e640 1 -- 172.21.15.12:0/1146324451 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc65406e2e0 con 0x7fc6540a38a0 2024-09-11T01:30:41.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.908+0000 7fc65359e640 1 -- 172.21.15.12:0/1146324451 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6540a38a0 msgr2=0x7fc65413f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:41.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.908+0000 7fc65359e640 1 --2- 172.21.15.12:0/1146324451 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6540a38a0 0x7fc65413f3c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc64c009e50 tx=0x7fc64c02a610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=826773c92229a468 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:41.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.908+0000 7fc651d9b640 1 -- 172.21.15.12:0/1146324451 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64c036070 con 0x7fc6540a38a0 2024-09-11T01:30:41.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.908+0000 7fc651d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:41.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.908+0000 7fc651d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:30:41.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.908+0000 7fc651d9b640 10 monclient: dump: 2024-09-11T01:30:41.918 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:41.918 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:41.918 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.918 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.918 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:41.918 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:41.918 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:41.918 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:41.918 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:41.919 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:41.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.908+0000 7fc651d9b640 10 monclient: _finish_auth 0 2024-09-11T01:30:41.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.908+0000 7fc651d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:30:41.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.908+0000 7fc651d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:41.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.908+0000 7fc651d9b640 1 -- 172.21.15.12:0/1146324451 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc64c007800 con 0x7fc6540a38a0 2024-09-11T01:30:41.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.908+0000 7fc651d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:41.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.908+0000 7fc651d9b640 1 -- 172.21.15.12:0/1146324451 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64c007bd0 con 0x7fc6540a38a0 2024-09-11T01:30:41.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.908+0000 7fc659008640 5 monclient: authenticate success, global_id 4514 2024-09-11T01:30:41.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.908+0000 7fc659008640 10 monclient: get_monmap_and_config success 2024-09-11T01:30:41.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.908+0000 7fc651d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:41.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.908+0000 7fc659008640 4 set_mon_vals no callback set 2024-09-11T01:30:41.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.908+0000 7fc651d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:41.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.908+0000 7fc651d9b640 10 monclient: dump: 2024-09-11T01:30:41.920 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:41.920 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:41.920 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.920 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.920 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:41.920 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:41.920 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:41.920 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:41.921 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:41.921 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:41.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.909+0000 7fc659008640 10 monclient: shutdown 2024-09-11T01:30:41.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.909+0000 7fc659008640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:41.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.909+0000 7fc659008640 1 -- 172.21.15.12:0/1146324451 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6540a38a0 msgr2=0x7fc65413f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:41.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.909+0000 7fc659008640 1 --2- 172.21.15.12:0/1146324451 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6540a38a0 0x7fc65413f3c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc64c009e50 tx=0x7fc64c02a610 comp rx=0 tx=0).stop 2024-09-11T01:30:41.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.909+0000 7fc659008640 1 -- 172.21.15.12:0/1146324451 shutdown_connections 2024-09-11T01:30:41.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.909+0000 7fc659008640 1 --2- 172.21.15.12:0/1146324451 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6540a38a0 0x7fc65413f3c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.909+0000 7fc659008640 1 --2- 172.21.15.12:0/1146324451 >> v2:172.21.15.12:3300/0 conn(0x7fc6540a2db0 0x7fc65406f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.909+0000 7fc659008640 1 -- 172.21.15.12:0/1146324451 >> 172.21.15.12:0/1146324451 conn(0x7fc6540a83b0 msgr2=0x7fc6540a87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:41.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.909+0000 7fc659008640 1 -- 172.21.15.12:0/1146324451 shutdown_connections 2024-09-11T01:30:41.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.909+0000 7fc659008640 1 -- 172.21.15.12:0/1146324451 wait complete. 2024-09-11T01:30:41.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.910+0000 7fc659008640 10 monclient: build_initial_monmap 2024-09-11T01:30:41.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.910+0000 7fc659008640 10 monclient: monmap: 2024-09-11T01:30:41.922 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:30:41.922 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:30:41.922 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:30:41.922 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:30:41.922 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:30:41.922 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:41.922 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:30:41.922 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:30:41.922 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:30:41.922 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:41.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.910+0000 7fc659008640 1 Processor -- start 2024-09-11T01:30:41.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.910+0000 7fc659008640 1 -- start start 2024-09-11T01:30:41.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.910+0000 7fc659008640 10 monclient: init 2024-09-11T01:30:41.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc659008640 10 monclient: _reopen_session rank -1 2024-09-11T01:30:41.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc659008640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:30:41.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc659008640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65406ded0 0x7fc65409d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:41.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc659008640 10 monclient(hunting): picked mon.noname-a con 0x7fc65406ded0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:30:41.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc659008640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6540a2db0 0x7fc65409d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:41.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc659008640 10 monclient(hunting): picked mon.noname-c con 0x7fc6540a2db0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:30:41.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc659008640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6540a38a0 0x7fc65409dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:41.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc652d9d640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65406ded0 msgr2=0x7fc65409d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:30:41.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc659008640 10 monclient(hunting): picked mon.noname-b con 0x7fc6540a38a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:30:41.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc659008640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:41.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc659008640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc65413fc90 con 0x7fc65406ded0 2024-09-11T01:30:41.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc652d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65406ded0 0x7fc65409d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:30:41.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc659008640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:41.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc659008640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc65413fb10 con 0x7fc6540a38a0 2024-09-11T01:30:41.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc659008640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:41.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc659008640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc65413fe10 con 0x7fc6540a2db0 2024-09-11T01:30:41.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc659008640 10 monclient(hunting): _renew_subs 2024-09-11T01:30:41.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc659008640 10 monclient(hunting): authenticate will time out at 848.353280s 2024-09-11T01:30:41.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc65259c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6540a2db0 0x7fc65409d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:41.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc65259c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6540a2db0 0x7fc65409d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:38862/0 (socket says 172.21.15.12:38862) 2024-09-11T01:30:41.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc65259c640 1 -- 172.21.15.12:0/1249505539 learned_addr learned my addr 172.21.15.12:0/1249505539 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:30:41.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc65259c640 10 monclient(hunting): get_auth_request con 0x7fc6540a2db0 auth_method 0 2024-09-11T01:30:41.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc65259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:41.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc65259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:41.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc65259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:41.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc65259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:41.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc65259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:41.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc65259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:41.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc65359e640 1 --2- 172.21.15.12:0/1249505539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6540a38a0 0x7fc65409dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:41.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc65259c640 10 monclient(hunting): handle_auth_done global_id 4517 payload 824 2024-09-11T01:30:41.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc65259c640 1 -- 172.21.15.12:0/1249505539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6540a38a0 msgr2=0x7fc65409dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:41.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc65259c640 1 --2- 172.21.15.12:0/1249505539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6540a38a0 0x7fc65409dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc65259c640 1 -- 172.21.15.12:0/1249505539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65406ded0 msgr2=0x7fc65409d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:30:41.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc65259c640 1 --2- 172.21.15.12:0/1249505539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc65406ded0 0x7fc65409d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc65259c640 10 monclient: _finish_hunting 0 2024-09-11T01:30:41.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc65259c640 1 monclient: found mon.noname-c 2024-09-11T01:30:41.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc65259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:30:41.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc65259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:30:41.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc65259c640 1 -- 172.21.15.12:0/1249505539 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc65413f510 con 0x7fc6540a2db0 2024-09-11T01:30:41.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc65359e640 10 monclient: get_auth_request con 0x7fc6540a38a0 auth_method 0 2024-09-11T01:30:41.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.911+0000 7fc65359e640 1 --2- 172.21.15.12:0/1249505539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6540a38a0 0x7fc65409dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:30:41.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.913+0000 7fc65259c640 1 -- 172.21.15.12:0/1249505539 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6540a2db0 msgr2=0x7fc65409d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:41.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.913+0000 7fc65259c640 1 --2- 172.21.15.12:0/1249505539 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6540a2db0 0x7fc65409d960 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc64400c460 tx=0x7fc64400c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:41.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.914+0000 7fc64b7fe640 1 -- 172.21.15.12:0/1249505539 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc644016020 con 0x7fc6540a2db0 2024-09-11T01:30:41.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.914+0000 7fc64b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:41.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.914+0000 7fc64b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:30:41.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.914+0000 7fc64b7fe640 10 monclient: dump: 2024-09-11T01:30:41.926 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:41.927 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:41.927 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.927 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.927 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:41.927 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:41.927 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:41.927 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:41.927 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:41.927 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:41.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.914+0000 7fc64b7fe640 10 monclient: _finish_auth 0 2024-09-11T01:30:41.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.914+0000 7fc64b7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:30:41.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.914+0000 7fc64b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:30:41.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.914+0000 7fc659008640 5 monclient: authenticate success, global_id 4517 2024-09-11T01:30:41.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.914+0000 7fc64b7fe640 1 -- 172.21.15.12:0/1249505539 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc644012a80 con 0x7fc6540a2db0 2024-09-11T01:30:41.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.914+0000 7fc64b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:30:41.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.914+0000 7fc659008640 10 monclient: _renew_subs 2024-09-11T01:30:41.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.914+0000 7fc64b7fe640 1 -- 172.21.15.12:0/1249505539 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc644011380 con 0x7fc6540a2db0 2024-09-11T01:30:41.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.914+0000 7fc659008640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:41.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.914+0000 7fc659008640 1 -- 172.21.15.12:0/1249505539 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc65409f5a0 con 0x7fc6540a2db0 2024-09-11T01:30:41.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.914+0000 7fc64b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:41.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.914+0000 7fc64b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:30:41.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.914+0000 7fc65159a640 4 set_mon_vals no callback set 2024-09-11T01:30:41.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.914+0000 7fc64b7fe640 10 monclient: dump: 2024-09-11T01:30:41.928 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:41.928 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:41.928 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.928 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.929 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:41.929 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:41.929 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:41.929 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:41.929 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:41.929 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:41.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.914+0000 7fc659008640 10 monclient: _renew_subs 2024-09-11T01:30:41.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.914+0000 7fc659008640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:30:41.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.914+0000 7fc659008640 1 -- 172.21.15.12:0/1249505539 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc65413ade0 con 0x7fc6540a2db0 2024-09-11T01:30:41.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.914+0000 7fc659008640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:41.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.915+0000 7fc659008640 1 --2- 172.21.15.12:0/1249505539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6541cd0e0 0x7fc6541cd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:41.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.915+0000 7fc659008640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:41.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.915+0000 7fc659008640 1 -- 172.21.15.12:0/1249505539 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc65406a060 con 0x7fc6541cd0e0 2024-09-11T01:30:41.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.915+0000 7fc64b7fe640 1 -- 172.21.15.12:0/1249505539 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fc644011520 con 0x7fc6540a2db0 2024-09-11T01:30:41.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.915+0000 7fc64b7fe640 1 --2- 172.21.15.12:0/1249505539 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc63403e1c0 0x7fc634040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:41.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.915+0000 7fc652d9d640 1 --2- 172.21.15.12:0/1249505539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6541cd0e0 0x7fc6541cd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:41.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.915+0000 7fc652d9d640 10 monclient: get_auth_request con 0x7fc6541cd0e0 auth_method 0 2024-09-11T01:30:41.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.915+0000 7fc652d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:41.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.915+0000 7fc652d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:41.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.915+0000 7fc652d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:41.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.915+0000 7fc64b7fe640 1 -- 172.21.15.12:0/1249505539 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(43..43 src has 1..43) ==== 4799+0+0 (secure 0 0 0) 0x7fc644052f20 con 0x7fc6540a2db0 2024-09-11T01:30:41.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.915+0000 7fc652d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:41.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.915+0000 7fc652d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:41.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.915+0000 7fc652d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:41.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.915+0000 7fc65359e640 1 --2- 172.21.15.12:0/1249505539 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc63403e1c0 0x7fc634040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:41.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.915+0000 7fc65359e640 10 monclient: get_auth_request con 0x7fc63403e1c0 auth_method 0 2024-09-11T01:30:41.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.915+0000 7fc652d9d640 10 monclient(hunting): handle_auth_done global_id 4423 payload 824 2024-09-11T01:30:41.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.915+0000 7fc652d9d640 1 -- 172.21.15.12:0/1249505539 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc6540a2450 con 0x7fc6541cd0e0 2024-09-11T01:30:41.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.917+0000 7fc652d9d640 1 -- 172.21.15.12:0/1249505539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6541cd0e0 msgr2=0x7fc6541cd4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:41.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.917+0000 7fc652d9d640 1 --2- 172.21.15.12:0/1249505539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6541cd0e0 0x7fc6541cd4c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc63c003980 tx=0x7fc63c00b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:41.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.918+0000 7fc65359e640 1 --2- 172.21.15.12:0/1249505539 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc63403e1c0 0x7fc634040680 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc654072030 tx=0x7fc64c009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:41.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.918+0000 7fc64b7fe640 1 -- 172.21.15.12:0/1249505539 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63c016070 con 0x7fc6541cd0e0 2024-09-11T01:30:41.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.918+0000 7fc64b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:41.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.918+0000 7fc64b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:30:41.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.918+0000 7fc64b7fe640 10 monclient: dump: 2024-09-11T01:30:41.934 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:41.934 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:41.934 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.934 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.934 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:41.934 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:41.934 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:41.934 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:41.934 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:41.934 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:41.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.918+0000 7fc64b7fe640 1 -- 172.21.15.12:0/1249505539 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc63c005270 con 0x7fc6541cd0e0 2024-09-11T01:30:41.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.918+0000 7fc64b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:30:41.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.918+0000 7fc64b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:30:41.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.918+0000 7fc64b7fe640 1 -- 172.21.15.12:0/1249505539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6541cd0e0 msgr2=0x7fc6541cd4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:41.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.918+0000 7fc64b7fe640 1 --2- 172.21.15.12:0/1249505539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6541cd0e0 0x7fc6541cd4c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc63c003980 tx=0x7fc63c00b8e0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.918+0000 7fc64b7fe640 1 -- 172.21.15.12:0/1249505539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6541cd0e0 msgr2=0x7fc6541cd4c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:30:41.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.918+0000 7fc64b7fe640 1 --2- 172.21.15.12:0/1249505539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6541cd0e0 0x7fc6541cd4c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.924+0000 7fc659008640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:30:41.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.924+0000 7fc659008640 1 --2- 172.21.15.12:0/1249505539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc654134f30 0x7fc6541362e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:30:41.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.924+0000 7fc659008640 10 monclient(hunting): start opening mon connection 2024-09-11T01:30:41.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.924+0000 7fc659008640 1 -- 172.21.15.12:0/1249505539 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc65413b180 con 0x7fc654134f30 2024-09-11T01:30:41.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.924+0000 7fc652d9d640 1 --2- 172.21.15.12:0/1249505539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc654134f30 0x7fc6541362e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:30:41.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.924+0000 7fc652d9d640 10 monclient: get_auth_request con 0x7fc654134f30 auth_method 0 2024-09-11T01:30:41.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.924+0000 7fc652d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:30:41.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.924+0000 7fc652d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:30:41.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.924+0000 7fc652d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:30:41.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.924+0000 7fc652d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:30:41.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.924+0000 7fc652d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:30:41.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.924+0000 7fc652d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:30:41.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.925+0000 7fc652d9d640 10 monclient(hunting): handle_auth_done global_id 4426 payload 824 2024-09-11T01:30:41.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.925+0000 7fc652d9d640 1 -- 172.21.15.12:0/1249505539 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc6541398c0 con 0x7fc654134f30 2024-09-11T01:30:41.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.927+0000 7fc652d9d640 1 -- 172.21.15.12:0/1249505539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc654134f30 msgr2=0x7fc6541362e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:30:41.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.927+0000 7fc652d9d640 1 --2- 172.21.15.12:0/1249505539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc654134f30 0x7fc6541362e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc63c003980 tx=0x7fc63c0059d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:30:41.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.927+0000 7fc64b7fe640 1 -- 172.21.15.12:0/1249505539 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63c016040 con 0x7fc654134f30 2024-09-11T01:30:41.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.927+0000 7fc64b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:30:41.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.927+0000 7fc64b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:30:41.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.927+0000 7fc64b7fe640 10 monclient: dump: 2024-09-11T01:30:41.937 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:30:41.937 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:30:41.937 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.937 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:30:41.937 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:30:41.937 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:30:41.937 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:30:41.937 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:30:41.937 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:30:41.937 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:30:41.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.927+0000 7fc64b7fe640 1 -- 172.21.15.12:0/1249505539 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4866 (secure 0 0 0) 0x7fc63c018c10 con 0x7fc654134f30 2024-09-11T01:30:41.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.927+0000 7fc64b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:30:41.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.927+0000 7fc64b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:30:41.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.927+0000 7fc64b7fe640 1 -- 172.21.15.12:0/1249505539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc654134f30 msgr2=0x7fc6541362e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:41.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.927+0000 7fc64b7fe640 1 --2- 172.21.15.12:0/1249505539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc654134f30 0x7fc6541362e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc63c003980 tx=0x7fc63c0059d0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.927+0000 7fc64b7fe640 1 -- 172.21.15.12:0/1249505539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc654134f30 msgr2=0x7fc6541362e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:30:41.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.927+0000 7fc64b7fe640 1 --2- 172.21.15.12:0/1249505539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc654134f30 0x7fc6541362e0 secure :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc63c003980 tx=0x7fc63c0059d0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.938 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:30:41.938 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:30:41.938 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:30:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:30:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 18, 2024-09-11T01:30:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 104581, 2024-09-11T01:30:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:30:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:30:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:30:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:30:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:30:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:30:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:30:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:30:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:30:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:30:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:30:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:30:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:30:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:30:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:30:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:30:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:30:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:30:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:30:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:30:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:30:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:30:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:30:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:30:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:30:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:30:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:30:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:30:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:30:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:30:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:30:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:30:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:30:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:30:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:30:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:30:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:30:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:30:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:30:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:30:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:30:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:30:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:30:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:30:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:30:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:30:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:30:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:30:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:30:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:30:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:30:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:30:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:30:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:30:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:30:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:30:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:30:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:30:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:30:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:30:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:30:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:30:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:30:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:30:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:30:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:30:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:30:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:30:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:30:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:30:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:30:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:30:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:30:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:30:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:30:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:30:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:30:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:30:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:30:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:30:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:30:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:30:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:30:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:30:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:30:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:30:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:30:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:30:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:30:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:30:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:30:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:30:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:30:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:30:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:30:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:30:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:30:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:30:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:30:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:30:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:30:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:30:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:30:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:30:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:30:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:30:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:30:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:30:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:30:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:30:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:30:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:30:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:30:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:30:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:30:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:30:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:30:41.952 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:30:41.952 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:30:41.952 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:30:41.952 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:30:41.952 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:30:41.952 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:30:41.952 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:30:41.952 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:30:41.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.928+0000 7fc659008640 1 -- 172.21.15.12:0/1249505539 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc63403e1c0 msgr2=0x7fc634040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:41.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.928+0000 7fc659008640 1 --2- 172.21.15.12:0/1249505539 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc63403e1c0 0x7fc634040680 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc654072030 tx=0x7fc64c009000 comp rx=0 tx=0).stop 2024-09-11T01:30:41.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.928+0000 7fc659008640 10 monclient: shutdown 2024-09-11T01:30:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.928+0000 7fc659008640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:30:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.928+0000 7fc659008640 1 -- 172.21.15.12:0/1249505539 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6540a2db0 msgr2=0x7fc65409d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:30:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.928+0000 7fc65359e640 1 -- 172.21.15.12:0/1249505539 reap_dead start 2024-09-11T01:30:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.928+0000 7fc659008640 1 --2- 172.21.15.12:0/1249505539 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6540a2db0 0x7fc65409d960 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc64400c460 tx=0x7fc64400c920 comp rx=0 tx=0).stop 2024-09-11T01:30:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.928+0000 7fc659008640 1 -- 172.21.15.12:0/1249505539 shutdown_connections 2024-09-11T01:30:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.928+0000 7fc659008640 1 --2- 172.21.15.12:0/1249505539 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6540a2db0 0x7fc65409d960 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:30:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.928+0000 7fc659008640 1 -- 172.21.15.12:0/1249505539 >> 172.21.15.12:0/1249505539 conn(0x7fc6540a83b0 msgr2=0x7fc6540a1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:30:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.928+0000 7fc659008640 1 -- 172.21.15.12:0/1249505539 shutdown_connections 2024-09-11T01:30:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:30:41.928+0000 7fc659008640 1 -- 172.21.15.12:0/1249505539 wait complete. 2024-09-11T01:30:41.958 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:30:41.959 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:30:42.357 INFO:tasks.rados.rados.0.smithi071.stdout:129: done (4 left) 2024-09-11T01:30:42.357 INFO:tasks.rados.rados.0.smithi071.stdout:130: done (3 left) 2024-09-11T01:30:42.357 INFO:tasks.rados.rados.0.smithi071.stdout:132: done (2 left) 2024-09-11T01:30:42.358 INFO:tasks.rados.rados.0.smithi071.stdout:133: done (1 left) 2024-09-11T01:30:42.358 INFO:tasks.rados.rados.0.smithi071.stdout:136: done (0 left) 2024-09-11T01:30:42.358 INFO:tasks.rados.rados.0.smithi071.stdout:137: delete oid 4 current snap is 13 2024-09-11T01:30:42.360 INFO:tasks.rados.rados.0.smithi071.stdout:137: done (0 left) 2024-09-11T01:30:42.360 INFO:tasks.rados.rados.0.smithi071.stdout:138: rollback oid 37 current snap is 13 2024-09-11T01:30:42.360 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 37 to 10 2024-09-11T01:30:42.360 INFO:tasks.rados.rados.0.smithi071.stdout:139: write (excl) oid 44 current snap is 13 2024-09-11T01:30:42.361 INFO:tasks.rados.rados.0.smithi071.stdout:139: seq_num 71 ranges {793297=789856,2305045=723014} 2024-09-11T01:30:42.369 INFO:tasks.rados.rados.0.smithi071.stdout:139: writing smithi07134777-44 from 793297 to 1583153 tid 1 2024-09-11T01:30:42.378 INFO:tasks.rados.rados.0.smithi071.stdout:139: writing smithi07134777-44 from 2305045 to 3028059 tid 2 2024-09-11T01:30:42.378 INFO:tasks.rados.rados.0.smithi071.stdout:140: delete oid 34 current snap is 13 2024-09-11T01:30:42.378 INFO:tasks.rados.rados.0.smithi071.stdout:138: finishing rollback tid 0 to smithi07134777-37 2024-09-11T01:30:42.378 INFO:tasks.rados.rados.0.smithi071.stdout:138: finishing rollback tid 1 to smithi07134777-37 2024-09-11T01:30:42.378 INFO:tasks.rados.rados.0.smithi071.stdout:138: finishing rollback tid 2 to smithi07134777-37 2024-09-11T01:30:42.378 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 41 (ObjNum 61 snap 5 seq_num 61) dirty exists 2024-09-11T01:30:42.379 INFO:tasks.rados.rados.0.smithi071.stdout:139: finishing write tid 1 to smithi07134777-44 2024-09-11T01:30:42.381 INFO:tasks.rados.rados.0.smithi071.stdout:139: finishing write tid 2 to smithi07134777-44 2024-09-11T01:30:42.381 INFO:tasks.rados.rados.0.smithi071.stdout:139: finishing write tid 3 to smithi07134777-44 2024-09-11T01:30:42.382 INFO:tasks.rados.rados.0.smithi071.stdout:139: finishing write tid 4 to smithi07134777-44 2024-09-11T01:30:42.382 INFO:tasks.rados.rados.0.smithi071.stdout:139: oid 44 updating version 0 to 26 2024-09-11T01:30:42.382 INFO:tasks.rados.rados.0.smithi071.stdout:139: oid 44 version 26 is already newer than 25 2024-09-11T01:30:42.382 INFO:tasks.rados.rados.0.smithi071.stdout:139: oid 44 updating version 26 to 27 2024-09-11T01:30:42.382 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 27 (ObjNum 71 snap 13 seq_num 71) dirty exists 2024-09-11T01:30:42.382 INFO:tasks.rados.rados.0.smithi071.stdout:139: left oid 44 (ObjNum 71 snap 13 seq_num 71) 2024-09-11T01:30:42.383 INFO:tasks.rados.rados.0.smithi071.stdout:138: done (2 left) 2024-09-11T01:30:42.383 INFO:tasks.rados.rados.0.smithi071.stdout:139: done (1 left) 2024-09-11T01:30:42.383 INFO:tasks.rados.rados.0.smithi071.stdout:140: done (0 left) 2024-09-11T01:30:42.383 INFO:tasks.rados.rados.0.smithi071.stdout:141: read oid 30 snap 9 2024-09-11T01:30:42.383 INFO:tasks.rados.rados.0.smithi071.stdout:141: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-11T01:30:42.383 INFO:tasks.rados.rados.0.smithi071.stdout:142: write oid 8 current snap is 13 2024-09-11T01:30:42.383 INFO:tasks.rados.rados.0.smithi071.stdout:142: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2024-09-11T01:30:42.388 INFO:tasks.rados.rados.0.smithi071.stdout:142: writing smithi07134777-8 from 426911 to 967265 tid 1 2024-09-11T01:30:42.395 INFO:tasks.rados.rados.0.smithi071.stdout:142: writing smithi07134777-8 from 1622009 to 2348692 tid 2 2024-09-11T01:30:42.401 INFO:tasks.rados.rados.0.smithi071.stdout:142: writing smithi07134777-8 from 3121996 to 3226911 tid 3 2024-09-11T01:30:42.401 INFO:tasks.rados.rados.0.smithi071.stdout:143: copy_from oid 14 from oid 23 current snap is 13 2024-09-11T01:30:42.401 INFO:tasks.rados.rados.0.smithi071.stdout:141: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-11T01:30:42.409 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2462363 2024-09-11T01:30:42.414 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 261303 2024-09-11T01:30:42.440 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:30:42.468 INFO:tasks.rados.rados.0.smithi071.stdout:142: finishing write tid 1 to smithi07134777-8 2024-09-11T01:30:42.468 INFO:tasks.rados.rados.0.smithi071.stdout:142: finishing write tid 2 to smithi07134777-8 2024-09-11T01:30:42.468 INFO:tasks.rados.rados.0.smithi071.stdout:142: finishing write tid 3 to smithi07134777-8 2024-09-11T01:30:42.469 INFO:tasks.rados.rados.0.smithi071.stdout:142: finishing write tid 4 to smithi07134777-8 2024-09-11T01:30:42.469 INFO:tasks.rados.rados.0.smithi071.stdout:142: finishing write tid 5 to smithi07134777-8 2024-09-11T01:30:42.469 INFO:tasks.rados.rados.0.smithi071.stdout:142: oid 8 updating version 0 to 19 2024-09-11T01:30:42.469 INFO:tasks.rados.rados.0.smithi071.stdout:142: oid 8 version 19 is already newer than 18 2024-09-11T01:30:42.469 INFO:tasks.rados.rados.0.smithi071.stdout:142: oid 8 updating version 19 to 20 2024-09-11T01:30:42.469 INFO:tasks.rados.rados.0.smithi071.stdout:142: oid 8 updating version 20 to 21 2024-09-11T01:30:42.469 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 21 (ObjNum 73 snap 13 seq_num 73) dirty exists 2024-09-11T01:30:42.469 INFO:tasks.rados.rados.0.smithi071.stdout:142: left oid 8 (ObjNum 73 snap 13 seq_num 73) 2024-09-11T01:30:42.469 INFO:tasks.rados.rados.0.smithi071.stdout:143: finishing copy_from racing read to smithi07134777-14 2024-09-11T01:30:42.469 INFO:tasks.rados.rados.0.smithi071.stdout:141: done (2 left) 2024-09-11T01:30:42.469 INFO:tasks.rados.rados.0.smithi071.stdout:142: done (1 left) 2024-09-11T01:30:42.469 INFO:tasks.rados.rados.0.smithi071.stdout:144: copy_from oid 30 from oid 37 current snap is 13 2024-09-11T01:30:42.470 INFO:tasks.rados.rados.0.smithi071.stdout:143: finishing copy_from to smithi07134777-14 2024-09-11T01:30:42.470 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-11T01:30:42.470 INFO:tasks.rados.rados.0.smithi071.stdout:143: done (1 left) 2024-09-11T01:30:42.470 INFO:tasks.rados.rados.0.smithi071.stdout:145: snap_remove snap 9 2024-09-11T01:30:43.385 INFO:tasks.rados.rados.0.smithi071.stdout:145: done (1 left) 2024-09-11T01:30:43.385 INFO:tasks.rados.rados.0.smithi071.stdout:146: write (excl) oid 8 current snap is 13 2024-09-11T01:30:43.385 INFO:tasks.rados.rados.0.smithi071.stdout:144: finishing copy_from racing read to smithi07134777-30 2024-09-11T01:30:43.385 INFO:tasks.rados.rados.0.smithi071.stdout:146: seq_num 74 ranges {443718=615603,1564138=470341,2633066=610652} 2024-09-11T01:30:43.390 INFO:tasks.rados.rados.0.smithi071.stdout:146: writing smithi07134777-8 from 443718 to 1059321 tid 1 2024-09-11T01:30:43.396 INFO:tasks.rados.rados.0.smithi071.stdout:146: writing smithi07134777-8 from 1564138 to 2034479 tid 2 2024-09-11T01:30:43.403 INFO:tasks.rados.rados.0.smithi071.stdout:146: writing smithi07134777-8 from 2633066 to 3243718 tid 3 2024-09-11T01:30:43.403 INFO:tasks.rados.rados.0.smithi071.stdout:147: snap_remove snap 12 2024-09-11T01:30:45.150 INFO:tasks.rados.rados.0.smithi071.stdout:147: done (2 left) 2024-09-11T01:30:45.150 INFO:tasks.rados.rados.0.smithi071.stdout:148: snap_create 2024-09-11T01:30:45.150 INFO:tasks.rados.rados.0.smithi071.stdout:144: finishing copy_from to smithi07134777-30 2024-09-11T01:30:45.151 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 41 (ObjNum 61 snap 5 seq_num 61) dirty exists 2024-09-11T01:30:45.151 INFO:tasks.rados.rados.0.smithi071.stdout:146: finishing write tid 1 to smithi07134777-8 2024-09-11T01:30:45.151 INFO:tasks.rados.rados.0.smithi071.stdout:146: finishing write tid 2 to smithi07134777-8 2024-09-11T01:30:45.151 INFO:tasks.rados.rados.0.smithi071.stdout:146: finishing write tid 3 to smithi07134777-8 2024-09-11T01:30:45.151 INFO:tasks.rados.rados.0.smithi071.stdout:146: finishing write tid 4 to smithi07134777-8 2024-09-11T01:30:45.151 INFO:tasks.rados.rados.0.smithi071.stdout:146: finishing write tid 5 to smithi07134777-8 2024-09-11T01:30:45.151 INFO:tasks.rados.rados.0.smithi071.stdout:146: oid 8 updating version 0 to 29 2024-09-11T01:30:45.151 INFO:tasks.rados.rados.0.smithi071.stdout:146: oid 8 version 29 is already newer than 26 2024-09-11T01:30:45.151 INFO:tasks.rados.rados.0.smithi071.stdout:146: oid 8 version 29 is already newer than 28 2024-09-11T01:30:45.151 INFO:tasks.rados.rados.0.smithi071.stdout:146: oid 8 version 29 is already newer than 27 2024-09-11T01:30:45.151 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 29 (ObjNum 74 snap 13 seq_num 74) dirty exists 2024-09-11T01:30:45.151 INFO:tasks.rados.rados.0.smithi071.stdout:146: left oid 8 (ObjNum 74 snap 13 seq_num 74) 2024-09-11T01:30:46.155 INFO:tasks.rados.rados.0.smithi071.stdout:144: done (2 left) 2024-09-11T01:30:46.155 INFO:tasks.rados.rados.0.smithi071.stdout:146: done (1 left) 2024-09-11T01:30:46.155 INFO:tasks.rados.rados.0.smithi071.stdout:148: done (0 left) 2024-09-11T01:30:46.155 INFO:tasks.rados.rados.0.smithi071.stdout:149: snap_remove snap 10 2024-09-11T01:30:47.160 INFO:tasks.rados.rados.0.smithi071.stdout:149: done (0 left) 2024-09-11T01:30:47.160 INFO:tasks.rados.rados.0.smithi071.stdout:150: read oid 34 snap -1 2024-09-11T01:30:47.161 INFO:tasks.rados.rados.0.smithi071.stdout:150: expect deleted 2024-09-11T01:30:47.161 INFO:tasks.rados.rados.0.smithi071.stdout:151: copy_from oid 26 from oid 11 current snap is 14 2024-09-11T01:30:47.161 INFO:tasks.rados.rados.0.smithi071.stdout:152: copy_from oid 30 from oid 21 current snap is 14 2024-09-11T01:30:47.161 INFO:tasks.rados.rados.0.smithi071.stdout:153: snap_remove snap 13 2024-09-11T01:30:47.367 INFO:tasks.rados.rados.0.smithi071.stdout:153: done (3 left) 2024-09-11T01:30:47.367 INFO:tasks.rados.rados.0.smithi071.stdout:154: write (excl) oid 33 current snap is 14 2024-09-11T01:30:47.368 INFO:tasks.rados.rados.0.smithi071.stdout:154: seq_num 75 ranges {460525=690852,1622620=730352,3060489=200036} 2024-09-11T01:30:47.373 INFO:tasks.rados.rados.0.smithi071.stdout:154: writing smithi07134777-33 from 460525 to 1151377 tid 1 2024-09-11T01:30:47.380 INFO:tasks.rados.rados.0.smithi071.stdout:154: writing smithi07134777-33 from 1622620 to 2352972 tid 2 2024-09-11T01:30:47.386 INFO:tasks.rados.rados.0.smithi071.stdout:154: writing smithi07134777-33 from 3060489 to 3260525 tid 3 2024-09-11T01:30:47.386 INFO:tasks.rados.rados.0.smithi071.stdout:155: delete oid 29 current snap is 14 2024-09-11T01:30:47.386 INFO:tasks.rados.rados.0.smithi071.stdout:152: finishing copy_from racing read to smithi07134777-30 2024-09-11T01:30:47.386 INFO:tasks.rados.rados.0.smithi071.stdout:152: finishing copy_from to smithi07134777-30 2024-09-11T01:30:47.386 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 6 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-11T01:30:47.386 INFO:tasks.rados.rados.0.smithi071.stdout:151: finishing copy_from racing read to smithi07134777-26 2024-09-11T01:30:47.386 INFO:tasks.rados.rados.0.smithi071.stdout:151: finishing copy_from to smithi07134777-26 2024-09-11T01:30:47.386 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-11T01:30:47.386 INFO:tasks.rados.rados.0.smithi071.stdout:154: finishing write tid 1 to smithi07134777-33 2024-09-11T01:30:47.388 INFO:tasks.rados.rados.0.smithi071.stdout:154: finishing write tid 2 to smithi07134777-33 2024-09-11T01:30:47.388 INFO:tasks.rados.rados.0.smithi071.stdout:154: finishing write tid 3 to smithi07134777-33 2024-09-11T01:30:47.388 INFO:tasks.rados.rados.0.smithi071.stdout:154: finishing write tid 4 to smithi07134777-33 2024-09-11T01:30:47.388 INFO:tasks.rados.rados.0.smithi071.stdout:154: finishing write tid 5 to smithi07134777-33 2024-09-11T01:30:47.388 INFO:tasks.rados.rados.0.smithi071.stdout:154: oid 33 updating version 0 to 90 2024-09-11T01:30:47.388 INFO:tasks.rados.rados.0.smithi071.stdout:154: oid 33 version 90 is already newer than 87 2024-09-11T01:30:47.388 INFO:tasks.rados.rados.0.smithi071.stdout:154: oid 33 version 90 is already newer than 89 2024-09-11T01:30:47.388 INFO:tasks.rados.rados.0.smithi071.stdout:154: oid 33 version 90 is already newer than 88 2024-09-11T01:30:47.389 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 90 (ObjNum 75 snap 14 seq_num 75) dirty exists 2024-09-11T01:30:47.389 INFO:tasks.rados.rados.0.smithi071.stdout:154: left oid 33 (ObjNum 75 snap 14 seq_num 75) 2024-09-11T01:30:47.390 INFO:tasks.rados.rados.0.smithi071.stdout:150: done (4 left) 2024-09-11T01:30:47.391 INFO:tasks.rados.rados.0.smithi071.stdout:151: done (3 left) 2024-09-11T01:30:47.391 INFO:tasks.rados.rados.0.smithi071.stdout:152: done (2 left) 2024-09-11T01:30:47.391 INFO:tasks.rados.rados.0.smithi071.stdout:154: done (1 left) 2024-09-11T01:30:47.391 INFO:tasks.rados.rados.0.smithi071.stdout:155: done (0 left) 2024-09-11T01:30:47.391 INFO:tasks.rados.rados.0.smithi071.stdout:156: read oid 31 snap -1 2024-09-11T01:30:47.391 INFO:tasks.rados.rados.0.smithi071.stdout:156: expect (ObjNum 63 snap 8 seq_num 63) 2024-09-11T01:30:47.391 INFO:tasks.rados.rados.0.smithi071.stdout:157: snap_create 2024-09-11T01:30:47.395 INFO:tasks.rados.rados.0.smithi071.stdout:156: expect (ObjNum 63 snap 8 seq_num 63) 2024-09-11T01:30:47.443 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1551141 2024-09-11T01:30:48.368 INFO:tasks.rados.rados.0.smithi071.stdout:156: done (1 left) 2024-09-11T01:30:48.369 INFO:tasks.rados.rados.0.smithi071.stdout:157: done (0 left) 2024-09-11T01:30:48.369 INFO:tasks.rados.rados.0.smithi071.stdout:158: write oid 33 current snap is 15 2024-09-11T01:30:48.369 INFO:tasks.rados.rados.0.smithi071.stdout:158: seq_num 77 ranges {494139=557703,1455937=734021,2948041=346098} 2024-09-11T01:30:48.374 INFO:tasks.rados.rados.0.smithi071.stdout:158: writing smithi07134777-33 from 494139 to 1051842 tid 1 2024-09-11T01:30:48.380 INFO:tasks.rados.rados.0.smithi071.stdout:158: writing smithi07134777-33 from 1455937 to 2189958 tid 2 2024-09-11T01:30:48.386 INFO:tasks.rados.rados.0.smithi071.stdout:158: writing smithi07134777-33 from 2948041 to 3294139 tid 3 2024-09-11T01:30:48.386 INFO:tasks.rados.rados.0.smithi071.stdout:159: write oid 39 current snap is 15 2024-09-11T01:30:48.386 INFO:tasks.rados.rados.0.smithi071.stdout:159: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2024-09-11T01:30:48.392 INFO:tasks.rados.rados.0.smithi071.stdout:159: writing smithi07134777-39 from 510946 to 1143898 tid 1 2024-09-11T01:30:48.398 INFO:tasks.rados.rados.0.smithi071.stdout:159: writing smithi07134777-39 from 1798066 to 2275745 tid 2 2024-09-11T01:30:48.404 INFO:tasks.rados.rados.0.smithi071.stdout:159: writing smithi07134777-39 from 2742758 to 3199975 tid 3 2024-09-11T01:30:48.404 INFO:tasks.rados.rados.0.smithi071.stdout:160: snap_create 2024-09-11T01:30:48.404 INFO:tasks.rados.rados.0.smithi071.stdout:158: finishing write tid 1 to smithi07134777-33 2024-09-11T01:30:48.404 INFO:tasks.rados.rados.0.smithi071.stdout:158: finishing write tid 2 to smithi07134777-33 2024-09-11T01:30:48.404 INFO:tasks.rados.rados.0.smithi071.stdout:158: finishing write tid 3 to smithi07134777-33 2024-09-11T01:30:48.404 INFO:tasks.rados.rados.0.smithi071.stdout:158: finishing write tid 4 to smithi07134777-33 2024-09-11T01:30:48.404 INFO:tasks.rados.rados.0.smithi071.stdout:158: finishing write tid 5 to smithi07134777-33 2024-09-11T01:30:48.404 INFO:tasks.rados.rados.0.smithi071.stdout:158: oid 33 updating version 0 to 95 2024-09-11T01:30:48.404 INFO:tasks.rados.rados.0.smithi071.stdout:158: oid 33 version 95 is already newer than 92 2024-09-11T01:30:48.404 INFO:tasks.rados.rados.0.smithi071.stdout:158: oid 33 version 95 is already newer than 93 2024-09-11T01:30:48.405 INFO:tasks.rados.rados.0.smithi071.stdout:158: oid 33 version 95 is already newer than 94 2024-09-11T01:30:48.405 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 95 (ObjNum 77 snap 15 seq_num 77) dirty exists 2024-09-11T01:30:48.405 INFO:tasks.rados.rados.0.smithi071.stdout:158: left oid 33 (ObjNum 77 snap 15 seq_num 77) 2024-09-11T01:30:48.405 INFO:tasks.rados.rados.0.smithi071.stdout:159: finishing write tid 1 to smithi07134777-39 2024-09-11T01:30:48.405 INFO:tasks.rados.rados.0.smithi071.stdout:159: finishing write tid 2 to smithi07134777-39 2024-09-11T01:30:48.406 INFO:tasks.rados.rados.0.smithi071.stdout:159: finishing write tid 3 to smithi07134777-39 2024-09-11T01:30:48.408 INFO:tasks.rados.rados.0.smithi071.stdout:159: finishing write tid 4 to smithi07134777-39 2024-09-11T01:30:48.408 INFO:tasks.rados.rados.0.smithi071.stdout:159: finishing write tid 5 to smithi07134777-39 2024-09-11T01:30:48.408 INFO:tasks.rados.rados.0.smithi071.stdout:159: oid 39 updating version 0 to 51 2024-09-11T01:30:48.408 INFO:tasks.rados.rados.0.smithi071.stdout:159: oid 39 updating version 51 to 53 2024-09-11T01:30:48.408 INFO:tasks.rados.rados.0.smithi071.stdout:159: oid 39 version 53 is already newer than 52 2024-09-11T01:30:48.408 INFO:tasks.rados.rados.0.smithi071.stdout:159: oid 39 updating version 53 to 54 2024-09-11T01:30:48.408 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 54 (ObjNum 78 snap 15 seq_num 78) dirty exists 2024-09-11T01:30:48.408 INFO:tasks.rados.rados.0.smithi071.stdout:159: left oid 39 (ObjNum 78 snap 15 seq_num 78) 2024-09-11T01:30:49.374 INFO:tasks.rados.rados.0.smithi071.stdout:158: done (2 left) 2024-09-11T01:30:49.374 INFO:tasks.rados.rados.0.smithi071.stdout:159: done (1 left) 2024-09-11T01:30:49.374 INFO:tasks.rados.rados.0.smithi071.stdout:160: done (0 left) 2024-09-11T01:30:49.374 INFO:tasks.rados.rados.0.smithi071.stdout:161: snap_remove snap 14 2024-09-11T01:30:50.381 INFO:tasks.rados.rados.0.smithi071.stdout:161: done (0 left) 2024-09-11T01:30:50.381 INFO:tasks.rados.rados.0.smithi071.stdout:162: rollback oid 50 current snap is 16 2024-09-11T01:30:50.381 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 50 to 15 2024-09-11T01:30:50.381 INFO:tasks.rados.rados.0.smithi071.stdout:163: delete oid 28 current snap is 16 2024-09-11T01:30:50.386 INFO:tasks.rados.rados.0.smithi071.stdout:162: finishing rollback tid 0 to smithi07134777-50 2024-09-11T01:30:50.386 INFO:tasks.rados.rados.0.smithi071.stdout:162: finishing rollback tid 1 to smithi07134777-50 2024-09-11T01:30:50.388 INFO:tasks.rados.rados.0.smithi071.stdout:162: finishing rollback tid 2 to smithi07134777-50 2024-09-11T01:30:50.388 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 73 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-11T01:30:50.389 INFO:tasks.rados.rados.0.smithi071.stdout:162: done (1 left) 2024-09-11T01:30:50.389 INFO:tasks.rados.rados.0.smithi071.stdout:163: done (0 left) 2024-09-11T01:30:50.389 INFO:tasks.rados.rados.0.smithi071.stdout:164: snap_create 2024-09-11T01:30:51.388 INFO:tasks.rados.rados.0.smithi071.stdout:164: done (0 left) 2024-09-11T01:30:51.388 INFO:tasks.rados.rados.0.smithi071.stdout:165: rollback oid 18 current snap is 17 2024-09-11T01:30:51.388 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 18 to 15 2024-09-11T01:30:51.388 INFO:tasks.rados.rados.0.smithi071.stdout:166: rollback oid 43 current snap is 17 2024-09-11T01:30:51.388 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 43 to 15 2024-09-11T01:30:51.388 INFO:tasks.rados.rados.0.smithi071.stdout:167: copy_from oid 14 from oid 24 current snap is 17 2024-09-11T01:30:51.389 INFO:tasks.rados.rados.0.smithi071.stdout:168: write oid 50 current snap is 17 2024-09-11T01:30:51.389 INFO:tasks.rados.rados.0.smithi071.stdout:168: seq_num 80 ranges {544560=783450,1915030=481348,2797604=479761} 2024-09-11T01:30:51.393 INFO:tasks.rados.rados.0.smithi071.stdout:168: writing smithi07134777-50 from 544560 to 1328010 tid 1 2024-09-11T01:30:51.399 INFO:tasks.rados.rados.0.smithi071.stdout:168: writing smithi07134777-50 from 1915030 to 2396378 tid 2 2024-09-11T01:30:51.405 INFO:tasks.rados.rados.0.smithi071.stdout:168: writing smithi07134777-50 from 2797604 to 3277365 tid 3 2024-09-11T01:30:51.405 INFO:tasks.rados.rados.0.smithi071.stdout:169: snap_create 2024-09-11T01:30:51.405 INFO:tasks.rados.rados.0.smithi071.stdout:165: finishing rollback tid 0 to smithi07134777-18 2024-09-11T01:30:51.405 INFO:tasks.rados.rados.0.smithi071.stdout:168: finishing write tid 1 to smithi07134777-50 2024-09-11T01:30:51.408 INFO:tasks.rados.rados.0.smithi071.stdout:167: finishing copy_from racing read to smithi07134777-14 2024-09-11T01:30:51.408 INFO:tasks.rados.rados.0.smithi071.stdout:168: finishing write tid 2 to smithi07134777-50 2024-09-11T01:30:51.409 INFO:tasks.rados.rados.0.smithi071.stdout:165: finishing rollback tid 1 to smithi07134777-18 2024-09-11T01:30:51.410 INFO:tasks.rados.rados.0.smithi071.stdout:165: finishing rollback tid 2 to smithi07134777-18 2024-09-11T01:30:51.410 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 44 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-11T01:30:51.410 INFO:tasks.rados.rados.0.smithi071.stdout:166: finishing rollback tid 0 to smithi07134777-43 2024-09-11T01:30:51.410 INFO:tasks.rados.rados.0.smithi071.stdout:166: finishing rollback tid 1 to smithi07134777-43 2024-09-11T01:30:51.411 INFO:tasks.rados.rados.0.smithi071.stdout:166: finishing rollback tid 2 to smithi07134777-43 2024-09-11T01:30:51.411 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 101 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-11T01:30:51.411 INFO:tasks.rados.rados.0.smithi071.stdout:167: finishing copy_from to smithi07134777-14 2024-09-11T01:30:51.411 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 32 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-11T01:30:51.411 INFO:tasks.rados.rados.0.smithi071.stdout:168: finishing write tid 3 to smithi07134777-50 2024-09-11T01:30:51.412 INFO:tasks.rados.rados.0.smithi071.stdout:168: finishing write tid 4 to smithi07134777-50 2024-09-11T01:30:51.412 INFO:tasks.rados.rados.0.smithi071.stdout:168: finishing write tid 5 to smithi07134777-50 2024-09-11T01:30:51.412 INFO:tasks.rados.rados.0.smithi071.stdout:168: oid 50 updating version 0 to 75 2024-09-11T01:30:51.412 INFO:tasks.rados.rados.0.smithi071.stdout:168: oid 50 updating version 75 to 79 2024-09-11T01:30:51.412 INFO:tasks.rados.rados.0.smithi071.stdout:168: oid 50 version 79 is already newer than 76 2024-09-11T01:30:51.412 INFO:tasks.rados.rados.0.smithi071.stdout:168: oid 50 updating version 79 to 80 2024-09-11T01:30:51.412 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 80 (ObjNum 80 snap 17 seq_num 80) dirty exists 2024-09-11T01:30:51.412 INFO:tasks.rados.rados.0.smithi071.stdout:168: left oid 50 (ObjNum 80 snap 17 seq_num 80) 2024-09-11T01:30:52.369 INFO:tasks.rados.rados.0.smithi071.stdout:165: done (4 left) 2024-09-11T01:30:52.369 INFO:tasks.rados.rados.0.smithi071.stdout:166: done (3 left) 2024-09-11T01:30:52.369 INFO:tasks.rados.rados.0.smithi071.stdout:167: done (2 left) 2024-09-11T01:30:52.369 INFO:tasks.rados.rados.0.smithi071.stdout:168: done (1 left) 2024-09-11T01:30:52.369 INFO:tasks.rados.rados.0.smithi071.stdout:169: done (0 left) 2024-09-11T01:30:52.369 INFO:tasks.rados.rados.0.smithi071.stdout:170: write oid 10 current snap is 18 2024-09-11T01:30:52.369 INFO:tasks.rados.rados.0.smithi071.stdout:170: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2024-09-11T01:30:52.374 INFO:tasks.rados.rados.0.smithi071.stdout:170: writing smithi07134777-10 from 561367 to 1020066 tid 1 2024-09-11T01:30:52.381 INFO:tasks.rados.rados.0.smithi071.stdout:170: writing smithi07134777-10 from 1573512 to 2314871 tid 2 2024-09-11T01:30:52.387 INFO:tasks.rados.rados.0.smithi071.stdout:170: writing smithi07134777-10 from 2941380 to 3361367 tid 3 2024-09-11T01:30:52.387 INFO:tasks.rados.rados.0.smithi071.stdout:171: write oid 23 current snap is 18 2024-09-11T01:30:52.387 INFO:tasks.rados.rados.0.smithi071.stdout:170: finishing write tid 1 to smithi07134777-10 2024-09-11T01:30:52.387 INFO:tasks.rados.rados.0.smithi071.stdout:171: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2024-09-11T01:30:52.393 INFO:tasks.rados.rados.0.smithi071.stdout:171: writing smithi07134777-23 from 578174 to 1112122 tid 1 2024-09-11T01:30:52.398 INFO:tasks.rados.rados.0.smithi071.stdout:171: writing smithi07134777-23 from 1515641 to 2000658 tid 2 2024-09-11T01:30:52.405 INFO:tasks.rados.rados.0.smithi071.stdout:171: writing smithi07134777-23 from 2736097 to 3238402 tid 3 2024-09-11T01:30:52.405 INFO:tasks.rados.rados.0.smithi071.stdout:172: snap_remove snap 17 2024-09-11T01:30:53.975 INFO:tasks.rados.rados.0.smithi071.stdout:172: done (2 left) 2024-09-11T01:30:53.975 INFO:tasks.rados.rados.0.smithi071.stdout:173: write (excl) oid 33 current snap is 18 2024-09-11T01:30:53.975 INFO:tasks.rados.rados.0.smithi071.stdout:173: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2024-09-11T01:30:53.981 INFO:tasks.rados.rados.0.smithi071.stdout:173: writing smithi07134777-33 from 594981 to 1204178 tid 1 2024-09-11T01:30:53.990 INFO:tasks.rados.rados.0.smithi071.stdout:173: writing smithi07134777-33 from 1974123 to 2719151 tid 2 2024-09-11T01:30:53.994 INFO:tasks.rados.rados.0.smithi071.stdout:173: writing smithi07134777-33 from 3279873 to 3394981 tid 3 2024-09-11T01:30:53.994 INFO:tasks.rados.rados.0.smithi071.stdout:174: write oid 9 current snap is 18 2024-09-11T01:30:53.994 INFO:tasks.rados.rados.0.smithi071.stdout:170: finishing write tid 2 to smithi07134777-10 2024-09-11T01:30:53.994 INFO:tasks.rados.rados.0.smithi071.stdout:174: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2024-09-11T01:30:53.999 INFO:tasks.rados.rados.0.smithi071.stdout:174: writing smithi07134777-9 from 611788 to 1012587 tid 1 2024-09-11T01:30:54.006 INFO:tasks.rados.rados.0.smithi071.stdout:174: writing smithi07134777-9 from 1748958 to 2237644 tid 2 2024-09-11T01:30:54.013 INFO:tasks.rados.rados.0.smithi071.stdout:174: writing smithi07134777-9 from 2907296 to 3411788 tid 3 2024-09-11T01:30:54.013 INFO:tasks.rados.rados.0.smithi071.stdout:175: write oid 5 current snap is 18 2024-09-11T01:30:54.013 INFO:tasks.rados.rados.0.smithi071.stdout:170: finishing write tid 3 to smithi07134777-10 2024-09-11T01:30:54.013 INFO:tasks.rados.rados.0.smithi071.stdout:170: finishing write tid 4 to smithi07134777-10 2024-09-11T01:30:54.013 INFO:tasks.rados.rados.0.smithi071.stdout:170: finishing write tid 5 to smithi07134777-10 2024-09-11T01:30:54.013 INFO:tasks.rados.rados.0.smithi071.stdout:170: oid 10 updating version 0 to 18 2024-09-11T01:30:54.013 INFO:tasks.rados.rados.0.smithi071.stdout:170: oid 10 updating version 18 to 19 2024-09-11T01:30:54.013 INFO:tasks.rados.rados.0.smithi071.stdout:170: oid 10 updating version 19 to 20 2024-09-11T01:30:54.013 INFO:tasks.rados.rados.0.smithi071.stdout:170: oid 10 version 20 is already newer than 17 2024-09-11T01:30:54.013 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 20 (ObjNum 81 snap 18 seq_num 81) dirty exists 2024-09-11T01:30:54.014 INFO:tasks.rados.rados.0.smithi071.stdout:170: left oid 10 (ObjNum 81 snap 18 seq_num 81) 2024-09-11T01:30:54.014 INFO:tasks.rados.rados.0.smithi071.stdout:171: finishing write tid 1 to smithi07134777-23 2024-09-11T01:30:54.014 INFO:tasks.rados.rados.0.smithi071.stdout:171: finishing write tid 2 to smithi07134777-23 2024-09-11T01:30:54.014 INFO:tasks.rados.rados.0.smithi071.stdout:171: finishing write tid 3 to smithi07134777-23 2024-09-11T01:30:54.014 INFO:tasks.rados.rados.0.smithi071.stdout:171: finishing write tid 4 to smithi07134777-23 2024-09-11T01:30:54.014 INFO:tasks.rados.rados.0.smithi071.stdout:175: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2024-09-11T01:30:54.018 INFO:tasks.rados.rados.0.smithi071.stdout:175: writing smithi07134777-5 from 628595 to 1104643 tid 1 2024-09-11T01:30:54.026 INFO:tasks.rados.rados.0.smithi071.stdout:175: writing smithi07134777-5 from 1807440 to 2439784 tid 2 2024-09-11T01:30:54.033 INFO:tasks.rados.rados.0.smithi071.stdout:175: writing smithi07134777-5 from 2934719 to 3387193 tid 3 2024-09-11T01:30:54.033 INFO:tasks.rados.rados.0.smithi071.stdout:170: done (4 left) 2024-09-11T01:30:54.033 INFO:tasks.rados.rados.0.smithi071.stdout:176: read oid 13 snap -1 2024-09-11T01:30:54.033 INFO:tasks.rados.rados.0.smithi071.stdout:176: expect deleted 2024-09-11T01:30:54.033 INFO:tasks.rados.rados.0.smithi071.stdout:177: copy_from oid 4 from oid 2 current snap is 18 2024-09-11T01:30:54.034 INFO:tasks.rados.rados.0.smithi071.stdout:171: finishing write tid 5 to smithi07134777-23 2024-09-11T01:30:54.034 INFO:tasks.rados.rados.0.smithi071.stdout:171: oid 23 updating version 0 to 35 2024-09-11T01:30:54.034 INFO:tasks.rados.rados.0.smithi071.stdout:171: oid 23 updating version 35 to 36 2024-09-11T01:30:54.034 INFO:tasks.rados.rados.0.smithi071.stdout:171: oid 23 updating version 36 to 37 2024-09-11T01:30:54.034 INFO:tasks.rados.rados.0.smithi071.stdout:171: oid 23 updating version 37 to 38 2024-09-11T01:30:54.034 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 38 (ObjNum 82 snap 18 seq_num 82) dirty exists 2024-09-11T01:30:54.034 INFO:tasks.rados.rados.0.smithi071.stdout:171: left oid 23 (ObjNum 82 snap 18 seq_num 82) 2024-09-11T01:30:54.034 INFO:tasks.rados.rados.0.smithi071.stdout:171: done (5 left) 2024-09-11T01:30:54.034 INFO:tasks.rados.rados.0.smithi071.stdout:178: read oid 28 snap -1 2024-09-11T01:30:54.034 INFO:tasks.rados.rados.0.smithi071.stdout:178: expect deleted 2024-09-11T01:30:54.035 INFO:tasks.rados.rados.0.smithi071.stdout:179: rollback oid 37 current snap is 18 2024-09-11T01:30:54.035 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 37 to 16 2024-09-11T01:30:54.035 INFO:tasks.rados.rados.0.smithi071.stdout:173: finishing write tid 1 to smithi07134777-33 2024-09-11T01:30:54.035 INFO:tasks.rados.rados.0.smithi071.stdout:173: finishing write tid 2 to smithi07134777-33 2024-09-11T01:30:54.035 INFO:tasks.rados.rados.0.smithi071.stdout:173: finishing write tid 3 to smithi07134777-33 2024-09-11T01:30:54.035 INFO:tasks.rados.rados.0.smithi071.stdout:173: finishing write tid 4 to smithi07134777-33 2024-09-11T01:30:54.035 INFO:tasks.rados.rados.0.smithi071.stdout:173: finishing write tid 5 to smithi07134777-33 2024-09-11T01:30:54.035 INFO:tasks.rados.rados.0.smithi071.stdout:173: oid 33 updating version 0 to 103 2024-09-11T01:30:54.035 INFO:tasks.rados.rados.0.smithi071.stdout:173: oid 33 updating version 103 to 104 2024-09-11T01:30:54.035 INFO:tasks.rados.rados.0.smithi071.stdout:173: oid 33 updating version 104 to 106 2024-09-11T01:30:54.035 INFO:tasks.rados.rados.0.smithi071.stdout:173: oid 33 version 106 is already newer than 105 2024-09-11T01:30:54.035 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 106 (ObjNum 83 snap 18 seq_num 83) dirty exists 2024-09-11T01:30:54.035 INFO:tasks.rados.rados.0.smithi071.stdout:173: left oid 33 (ObjNum 83 snap 18 seq_num 83) 2024-09-11T01:30:54.036 INFO:tasks.rados.rados.0.smithi071.stdout:174: finishing write tid 1 to smithi07134777-9 2024-09-11T01:30:54.036 INFO:tasks.rados.rados.0.smithi071.stdout:174: finishing write tid 2 to smithi07134777-9 2024-09-11T01:30:54.036 INFO:tasks.rados.rados.0.smithi071.stdout:174: finishing write tid 3 to smithi07134777-9 2024-09-11T01:30:54.036 INFO:tasks.rados.rados.0.smithi071.stdout:174: finishing write tid 4 to smithi07134777-9 2024-09-11T01:30:54.036 INFO:tasks.rados.rados.0.smithi071.stdout:174: finishing write tid 5 to smithi07134777-9 2024-09-11T01:30:54.036 INFO:tasks.rados.rados.0.smithi071.stdout:174: oid 9 updating version 0 to 16 2024-09-11T01:30:54.036 INFO:tasks.rados.rados.0.smithi071.stdout:174: oid 9 version 16 is already newer than 14 2024-09-11T01:30:54.036 INFO:tasks.rados.rados.0.smithi071.stdout:174: oid 9 version 16 is already newer than 15 2024-09-11T01:30:54.036 INFO:tasks.rados.rados.0.smithi071.stdout:174: oid 9 updating version 16 to 17 2024-09-11T01:30:54.036 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 17 (ObjNum 84 snap 18 seq_num 84) dirty exists 2024-09-11T01:30:54.036 INFO:tasks.rados.rados.0.smithi071.stdout:174: left oid 9 (ObjNum 84 snap 18 seq_num 84) 2024-09-11T01:30:54.036 INFO:tasks.rados.rados.0.smithi071.stdout:175: finishing write tid 1 to smithi07134777-5 2024-09-11T01:30:54.036 INFO:tasks.rados.rados.0.smithi071.stdout:175: finishing write tid 2 to smithi07134777-5 2024-09-11T01:30:54.036 INFO:tasks.rados.rados.0.smithi071.stdout:173: done (6 left) 2024-09-11T01:30:54.037 INFO:tasks.rados.rados.0.smithi071.stdout:174: done (5 left) 2024-09-11T01:30:54.037 INFO:tasks.rados.rados.0.smithi071.stdout:180: read oid 47 snap -1 2024-09-11T01:30:54.037 INFO:tasks.rados.rados.0.smithi071.stdout:180: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-11T01:30:54.037 INFO:tasks.rados.rados.0.smithi071.stdout:181: write (excl) oid 3 current snap is 18 2024-09-11T01:30:54.037 INFO:tasks.rados.rados.0.smithi071.stdout:181: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2024-09-11T01:30:54.038 INFO:tasks.rados.rados.0.smithi071.stdout:181: writing smithi07134777-3 from 645402 to 1196699 tid 1 2024-09-11T01:30:54.045 INFO:tasks.rados.rados.0.smithi071.stdout:181: writing smithi07134777-3 from 1749569 to 2241924 tid 2 2024-09-11T01:30:54.052 INFO:tasks.rados.rados.0.smithi071.stdout:181: writing smithi07134777-3 from 2845789 to 3445402 tid 3 2024-09-11T01:30:54.052 INFO:tasks.rados.rados.0.smithi071.stdout:182: rollback oid 46 current snap is 18 2024-09-11T01:30:54.052 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 46 to 15 2024-09-11T01:30:54.052 INFO:tasks.rados.rados.0.smithi071.stdout:179: finishing rollback tid 0 to smithi07134777-37 2024-09-11T01:30:54.052 INFO:tasks.rados.rados.0.smithi071.stdout:179: finishing rollback tid 1 to smithi07134777-37 2024-09-11T01:30:54.052 INFO:tasks.rados.rados.0.smithi071.stdout:179: finishing rollback tid 2 to smithi07134777-37 2024-09-11T01:30:54.053 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 60 (ObjNum 61 snap 5 seq_num 61) dirty exists 2024-09-11T01:30:54.053 INFO:tasks.rados.rados.0.smithi071.stdout:180: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-11T01:30:54.083 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 703501 2024-09-11T01:30:54.131 INFO:tasks.rados.rados.0.smithi071.stdout:175: finishing write tid 3 to smithi07134777-5 2024-09-11T01:30:54.131 INFO:tasks.rados.rados.0.smithi071.stdout:175: finishing write tid 4 to smithi07134777-5 2024-09-11T01:30:54.131 INFO:tasks.rados.rados.0.smithi071.stdout:175: finishing write tid 5 to smithi07134777-5 2024-09-11T01:30:54.132 INFO:tasks.rados.rados.0.smithi071.stdout:175: oid 5 updating version 0 to 15 2024-09-11T01:30:54.132 INFO:tasks.rados.rados.0.smithi071.stdout:175: oid 5 version 15 is already newer than 13 2024-09-11T01:30:54.132 INFO:tasks.rados.rados.0.smithi071.stdout:175: oid 5 updating version 15 to 16 2024-09-11T01:30:54.132 INFO:tasks.rados.rados.0.smithi071.stdout:175: oid 5 version 16 is already newer than 14 2024-09-11T01:30:54.132 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 16 (ObjNum 85 snap 18 seq_num 85) dirty exists 2024-09-11T01:30:54.132 INFO:tasks.rados.rados.0.smithi071.stdout:175: left oid 5 (ObjNum 85 snap 18 seq_num 85) 2024-09-11T01:30:54.132 INFO:tasks.rados.rados.0.smithi071.stdout:175: done (7 left) 2024-09-11T01:30:54.132 INFO:tasks.rados.rados.0.smithi071.stdout:176: done (6 left) 2024-09-11T01:30:54.133 INFO:tasks.rados.rados.0.smithi071.stdout:178: done (5 left) 2024-09-11T01:30:54.133 INFO:tasks.rados.rados.0.smithi071.stdout:179: done (4 left) 2024-09-11T01:30:54.133 INFO:tasks.rados.rados.0.smithi071.stdout:180: done (3 left) 2024-09-11T01:30:54.133 INFO:tasks.rados.rados.0.smithi071.stdout:183: write oid 10 current snap is 18 2024-09-11T01:30:54.133 INFO:tasks.rados.rados.0.smithi071.stdout:183: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2024-09-11T01:30:54.137 INFO:tasks.rados.rados.0.smithi071.stdout:183: writing smithi07134777-10 from 662209 to 1288755 tid 1 2024-09-11T01:30:54.143 INFO:tasks.rados.rados.0.smithi071.stdout:183: writing smithi07134777-10 from 1808051 to 2444064 tid 2 2024-09-11T01:30:54.150 INFO:tasks.rados.rados.0.smithi071.stdout:183: writing smithi07134777-10 from 2873212 to 3462209 tid 3 2024-09-11T01:30:54.150 INFO:tasks.rados.rados.0.smithi071.stdout:184: snap_create 2024-09-11T01:30:54.150 INFO:tasks.rados.rados.0.smithi071.stdout:181: finishing write tid 1 to smithi07134777-3 2024-09-11T01:30:54.150 INFO:tasks.rados.rados.0.smithi071.stdout:177: finishing copy_from racing read to smithi07134777-4 2024-09-11T01:30:54.150 INFO:tasks.rados.rados.0.smithi071.stdout:177: finishing copy_from to smithi07134777-4 2024-09-11T01:30:54.150 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-11T01:30:54.150 INFO:tasks.rados.rados.0.smithi071.stdout:181: finishing write tid 2 to smithi07134777-3 2024-09-11T01:30:54.151 INFO:tasks.rados.rados.0.smithi071.stdout:181: finishing write tid 3 to smithi07134777-3 2024-09-11T01:30:54.151 INFO:tasks.rados.rados.0.smithi071.stdout:181: finishing write tid 4 to smithi07134777-3 2024-09-11T01:30:54.151 INFO:tasks.rados.rados.0.smithi071.stdout:181: finishing write tid 5 to smithi07134777-3 2024-09-11T01:30:54.151 INFO:tasks.rados.rados.0.smithi071.stdout:181: oid 3 updating version 0 to 8 2024-09-11T01:30:54.151 INFO:tasks.rados.rados.0.smithi071.stdout:181: oid 3 updating version 8 to 9 2024-09-11T01:30:54.151 INFO:tasks.rados.rados.0.smithi071.stdout:181: oid 3 updating version 9 to 10 2024-09-11T01:30:54.151 INFO:tasks.rados.rados.0.smithi071.stdout:181: oid 3 updating version 10 to 11 2024-09-11T01:30:54.151 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 11 (ObjNum 86 snap 18 seq_num 86) dirty exists 2024-09-11T01:30:54.151 INFO:tasks.rados.rados.0.smithi071.stdout:181: left oid 3 (ObjNum 86 snap 18 seq_num 86) 2024-09-11T01:30:54.151 INFO:tasks.rados.rados.0.smithi071.stdout:182: finishing rollback tid 0 to smithi07134777-46 2024-09-11T01:30:54.151 INFO:tasks.rados.rados.0.smithi071.stdout:182: finishing rollback tid 1 to smithi07134777-46 2024-09-11T01:30:54.151 INFO:tasks.rados.rados.0.smithi071.stdout:182: finishing rollback tid 2 to smithi07134777-46 2024-09-11T01:30:54.151 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 10 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-11T01:30:54.151 INFO:tasks.rados.rados.0.smithi071.stdout:183: finishing write tid 1 to smithi07134777-10 2024-09-11T01:30:54.152 INFO:tasks.rados.rados.0.smithi071.stdout:183: finishing write tid 2 to smithi07134777-10 2024-09-11T01:30:54.152 INFO:tasks.rados.rados.0.smithi071.stdout:183: finishing write tid 3 to smithi07134777-10 2024-09-11T01:30:54.152 INFO:tasks.rados.rados.0.smithi071.stdout:183: finishing write tid 4 to smithi07134777-10 2024-09-11T01:30:54.153 INFO:tasks.rados.rados.0.smithi071.stdout:183: finishing write tid 5 to smithi07134777-10 2024-09-11T01:30:54.153 INFO:tasks.rados.rados.0.smithi071.stdout:183: oid 10 updating version 0 to 21 2024-09-11T01:30:54.153 INFO:tasks.rados.rados.0.smithi071.stdout:183: oid 10 updating version 21 to 23 2024-09-11T01:30:54.153 INFO:tasks.rados.rados.0.smithi071.stdout:183: oid 10 version 23 is already newer than 22 2024-09-11T01:30:54.153 INFO:tasks.rados.rados.0.smithi071.stdout:183: oid 10 updating version 23 to 24 2024-09-11T01:30:54.153 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 24 (ObjNum 87 snap 18 seq_num 87) dirty exists 2024-09-11T01:30:54.153 INFO:tasks.rados.rados.0.smithi071.stdout:183: left oid 10 (ObjNum 87 snap 18 seq_num 87) 2024-09-11T01:30:54.976 INFO:tasks.rados.rados.0.smithi071.stdout:177: done (4 left) 2024-09-11T01:30:54.977 INFO:tasks.rados.rados.0.smithi071.stdout:181: done (3 left) 2024-09-11T01:30:54.977 INFO:tasks.rados.rados.0.smithi071.stdout:182: done (2 left) 2024-09-11T01:30:54.977 INFO:tasks.rados.rados.0.smithi071.stdout:183: done (1 left) 2024-09-11T01:30:54.977 INFO:tasks.rados.rados.0.smithi071.stdout:184: done (0 left) 2024-09-11T01:30:54.977 INFO:tasks.rados.rados.0.smithi071.stdout:185: write (excl) oid 10 current snap is 19 2024-09-11T01:30:54.977 INFO:tasks.rados.rados.0.smithi071.stdout:185: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2024-09-11T01:30:54.983 INFO:tasks.rados.rados.0.smithi071.stdout:185: writing smithi07134777-10 from 679016 to 1380811 tid 1 2024-09-11T01:30:54.989 INFO:tasks.rados.rados.0.smithi071.stdout:185: writing smithi07134777-10 from 1866533 to 2362557 tid 2 2024-09-11T01:30:54.994 INFO:tasks.rados.rados.0.smithi071.stdout:185: writing smithi07134777-10 from 2900635 to 3303278 tid 3 2024-09-11T01:30:54.994 INFO:tasks.rados.rados.0.smithi071.stdout:186: rollback oid 42 current snap is 19 2024-09-11T01:30:54.995 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 42 to 15 2024-09-11T01:30:54.995 INFO:tasks.rados.rados.0.smithi071.stdout:185: finishing write tid 1 to smithi07134777-10 2024-09-11T01:30:54.995 INFO:tasks.rados.rados.0.smithi071.stdout:187: delete oid 50 current snap is 19 2024-09-11T01:30:54.997 INFO:tasks.rados.rados.0.smithi071.stdout:185: finishing write tid 2 to smithi07134777-10 2024-09-11T01:30:54.997 INFO:tasks.rados.rados.0.smithi071.stdout:185: finishing write tid 3 to smithi07134777-10 2024-09-11T01:30:54.997 INFO:tasks.rados.rados.0.smithi071.stdout:185: finishing write tid 4 to smithi07134777-10 2024-09-11T01:30:54.998 INFO:tasks.rados.rados.0.smithi071.stdout:185: finishing write tid 5 to smithi07134777-10 2024-09-11T01:30:54.998 INFO:tasks.rados.rados.0.smithi071.stdout:185: oid 10 updating version 0 to 28 2024-09-11T01:30:54.998 INFO:tasks.rados.rados.0.smithi071.stdout:185: oid 10 updating version 28 to 29 2024-09-11T01:30:54.998 INFO:tasks.rados.rados.0.smithi071.stdout:185: oid 10 version 29 is already newer than 26 2024-09-11T01:30:54.998 INFO:tasks.rados.rados.0.smithi071.stdout:185: oid 10 version 29 is already newer than 27 2024-09-11T01:30:54.998 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 29 (ObjNum 88 snap 19 seq_num 88) dirty exists 2024-09-11T01:30:54.998 INFO:tasks.rados.rados.0.smithi071.stdout:185: left oid 10 (ObjNum 88 snap 19 seq_num 88) 2024-09-11T01:30:55.000 INFO:tasks.rados.rados.0.smithi071.stdout:186: finishing rollback tid 0 to smithi07134777-42 2024-09-11T01:30:55.000 INFO:tasks.rados.rados.0.smithi071.stdout:186: finishing rollback tid 1 to smithi07134777-42 2024-09-11T01:30:55.000 INFO:tasks.rados.rados.0.smithi071.stdout:185: done (2 left) 2024-09-11T01:30:55.000 INFO:tasks.rados.rados.0.smithi071.stdout:187: done (1 left) 2024-09-11T01:30:55.000 INFO:tasks.rados.rados.0.smithi071.stdout:188: write oid 38 current snap is 19 2024-09-11T01:30:55.000 INFO:tasks.rados.rados.0.smithi071.stdout:188: seq_num 90 ranges {712630=452293,1583497=499693,2555481=425187} 2024-09-11T01:30:55.006 INFO:tasks.rados.rados.0.smithi071.stdout:188: writing smithi07134777-38 from 712630 to 1164923 tid 1 2024-09-11T01:30:55.012 INFO:tasks.rados.rados.0.smithi071.stdout:188: writing smithi07134777-38 from 1583497 to 2083190 tid 2 2024-09-11T01:30:55.017 INFO:tasks.rados.rados.0.smithi071.stdout:188: writing smithi07134777-38 from 2555481 to 2980668 tid 3 2024-09-11T01:30:55.017 INFO:tasks.rados.rados.0.smithi071.stdout:189: copy_from oid 21 from oid 5 current snap is 19 2024-09-11T01:30:55.017 INFO:tasks.rados.rados.0.smithi071.stdout:186: finishing rollback tid 2 to smithi07134777-42 2024-09-11T01:30:55.017 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 10 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-11T01:30:55.017 INFO:tasks.rados.rados.0.smithi071.stdout:188: finishing write tid 1 to smithi07134777-38 2024-09-11T01:30:55.017 INFO:tasks.rados.rados.0.smithi071.stdout:188: finishing write tid 2 to smithi07134777-38 2024-09-11T01:30:55.018 INFO:tasks.rados.rados.0.smithi071.stdout:186: done (2 left) 2024-09-11T01:30:55.018 INFO:tasks.rados.rados.0.smithi071.stdout:190: delete oid 7 current snap is 19 2024-09-11T01:30:55.022 INFO:tasks.rados.rados.0.smithi071.stdout:188: finishing write tid 3 to smithi07134777-38 2024-09-11T01:30:55.022 INFO:tasks.rados.rados.0.smithi071.stdout:188: finishing write tid 4 to smithi07134777-38 2024-09-11T01:30:55.022 INFO:tasks.rados.rados.0.smithi071.stdout:188: finishing write tid 5 to smithi07134777-38 2024-09-11T01:30:55.022 INFO:tasks.rados.rados.0.smithi071.stdout:188: oid 38 updating version 0 to 85 2024-09-11T01:30:55.022 INFO:tasks.rados.rados.0.smithi071.stdout:188: oid 38 updating version 85 to 87 2024-09-11T01:30:55.022 INFO:tasks.rados.rados.0.smithi071.stdout:188: oid 38 version 87 is already newer than 86 2024-09-11T01:30:55.022 INFO:tasks.rados.rados.0.smithi071.stdout:188: oid 38 version 87 is already newer than 84 2024-09-11T01:30:55.022 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 87 (ObjNum 90 snap 19 seq_num 90) dirty exists 2024-09-11T01:30:55.022 INFO:tasks.rados.rados.0.smithi071.stdout:188: left oid 38 (ObjNum 90 snap 19 seq_num 90) 2024-09-11T01:30:55.022 INFO:tasks.rados.rados.0.smithi071.stdout:188: done (2 left) 2024-09-11T01:30:55.022 INFO:tasks.rados.rados.0.smithi071.stdout:190: done (1 left) 2024-09-11T01:30:55.022 INFO:tasks.rados.rados.0.smithi071.stdout:191: write oid 11 current snap is 19 2024-09-11T01:30:55.022 INFO:tasks.rados.rados.0.smithi071.stdout:191: seq_num 92 ranges {746244=719144,2100461=503362,3126680=419564} 2024-09-11T01:30:55.030 INFO:tasks.rados.rados.0.smithi071.stdout:191: writing smithi07134777-11 from 746244 to 1465388 tid 1 2024-09-11T01:30:55.036 INFO:tasks.rados.rados.0.smithi071.stdout:191: writing smithi07134777-11 from 2100461 to 2603823 tid 2 2024-09-11T01:30:55.041 INFO:tasks.rados.rados.0.smithi071.stdout:191: writing smithi07134777-11 from 3126680 to 3546244 tid 3 2024-09-11T01:30:55.042 INFO:tasks.rados.rados.0.smithi071.stdout:192: snap_remove snap 18 2024-09-11T01:30:55.979 INFO:tasks.rados.rados.0.smithi071.stdout:192: done (2 left) 2024-09-11T01:30:55.979 INFO:tasks.rados.rados.0.smithi071.stdout:193: read oid 19 snap -1 2024-09-11T01:30:55.979 INFO:tasks.rados.rados.0.smithi071.stdout:193: expect (ObjNum 68 snap 11 seq_num 68) 2024-09-11T01:30:55.979 INFO:tasks.rados.rados.0.smithi071.stdout:194: write (excl) oid 17 current snap is 19 2024-09-11T01:30:55.979 INFO:tasks.rados.rados.0.smithi071.stdout:189: finishing copy_from racing read to smithi07134777-21 2024-09-11T01:30:55.980 INFO:tasks.rados.rados.0.smithi071.stdout:194: seq_num 93 ranges {763051=794393,2158943=647020,3437750=125301} 2024-09-11T01:30:55.988 INFO:tasks.rados.rados.0.smithi071.stdout:194: writing smithi07134777-17 from 763051 to 1557444 tid 1 2024-09-11T01:30:55.995 INFO:tasks.rados.rados.0.smithi071.stdout:194: writing smithi07134777-17 from 2158943 to 2805963 tid 2 2024-09-11T01:30:56.000 INFO:tasks.rados.rados.0.smithi071.stdout:194: writing smithi07134777-17 from 3437750 to 3563051 tid 3 2024-09-11T01:30:56.000 INFO:tasks.rados.rados.0.smithi071.stdout:195: read oid 30 snap -1 2024-09-11T01:30:56.000 INFO:tasks.rados.rados.0.smithi071.stdout:195: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-11T01:30:56.000 INFO:tasks.rados.rados.0.smithi071.stdout:196: snap_remove snap 16 2024-09-11T01:30:56.989 INFO:tasks.rados.rados.0.smithi071.stdout:196: done (5 left) 2024-09-11T01:30:56.989 INFO:tasks.rados.rados.0.smithi071.stdout:197: snap_create 2024-09-11T01:30:56.989 INFO:tasks.rados.rados.0.smithi071.stdout:189: finishing copy_from to smithi07134777-21 2024-09-11T01:30:56.989 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 16 (ObjNum 85 snap 18 seq_num 85) dirty exists 2024-09-11T01:30:56.990 INFO:tasks.rados.rados.0.smithi071.stdout:191: finishing write tid 1 to smithi07134777-11 2024-09-11T01:30:56.990 INFO:tasks.rados.rados.0.smithi071.stdout:191: finishing write tid 2 to smithi07134777-11 2024-09-11T01:30:56.990 INFO:tasks.rados.rados.0.smithi071.stdout:191: finishing write tid 3 to smithi07134777-11 2024-09-11T01:30:56.990 INFO:tasks.rados.rados.0.smithi071.stdout:191: finishing write tid 4 to smithi07134777-11 2024-09-11T01:30:56.990 INFO:tasks.rados.rados.0.smithi071.stdout:191: finishing write tid 5 to smithi07134777-11 2024-09-11T01:30:56.990 INFO:tasks.rados.rados.0.smithi071.stdout:191: oid 11 updating version 0 to 46 2024-09-11T01:30:56.990 INFO:tasks.rados.rados.0.smithi071.stdout:191: oid 11 updating version 46 to 48 2024-09-11T01:30:56.990 INFO:tasks.rados.rados.0.smithi071.stdout:191: oid 11 version 48 is already newer than 45 2024-09-11T01:30:56.990 INFO:tasks.rados.rados.0.smithi071.stdout:191: oid 11 version 48 is already newer than 47 2024-09-11T01:30:56.990 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 48 (ObjNum 92 snap 19 seq_num 92) dirty exists 2024-09-11T01:30:56.990 INFO:tasks.rados.rados.0.smithi071.stdout:191: left oid 11 (ObjNum 92 snap 19 seq_num 92) 2024-09-11T01:30:56.990 INFO:tasks.rados.rados.0.smithi071.stdout:193: expect (ObjNum 68 snap 11 seq_num 68) 2024-09-11T01:30:57.038 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:30:57.084 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 664266 2024-09-11T01:30:57.125 INFO:tasks.rados.rados.0.smithi071.stdout:194: finishing write tid 1 to smithi07134777-17 2024-09-11T01:30:57.125 INFO:tasks.rados.rados.0.smithi071.stdout:194: finishing write tid 2 to smithi07134777-17 2024-09-11T01:30:57.125 INFO:tasks.rados.rados.0.smithi071.stdout:194: finishing write tid 3 to smithi07134777-17 2024-09-11T01:30:57.125 INFO:tasks.rados.rados.0.smithi071.stdout:194: finishing write tid 4 to smithi07134777-17 2024-09-11T01:30:57.125 INFO:tasks.rados.rados.0.smithi071.stdout:194: finishing write tid 5 to smithi07134777-17 2024-09-11T01:30:57.125 INFO:tasks.rados.rados.0.smithi071.stdout:194: oid 17 updating version 0 to 60 2024-09-11T01:30:57.125 INFO:tasks.rados.rados.0.smithi071.stdout:194: oid 17 version 60 is already newer than 58 2024-09-11T01:30:57.125 INFO:tasks.rados.rados.0.smithi071.stdout:194: oid 17 version 60 is already newer than 59 2024-09-11T01:30:57.125 INFO:tasks.rados.rados.0.smithi071.stdout:194: oid 17 version 60 is already newer than 57 2024-09-11T01:30:57.125 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 60 (ObjNum 93 snap 19 seq_num 93) dirty exists 2024-09-11T01:30:57.126 INFO:tasks.rados.rados.0.smithi071.stdout:194: left oid 17 (ObjNum 93 snap 19 seq_num 93) 2024-09-11T01:30:57.126 INFO:tasks.rados.rados.0.smithi071.stdout:195: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-11T01:30:57.126 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 249892 2024-09-11T01:30:57.175 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:30:57.371 INFO:tasks.rados.rados.0.smithi071.stdout:189: done (5 left) 2024-09-11T01:30:57.372 INFO:tasks.rados.rados.0.smithi071.stdout:191: done (4 left) 2024-09-11T01:30:57.372 INFO:tasks.rados.rados.0.smithi071.stdout:193: done (3 left) 2024-09-11T01:30:57.372 INFO:tasks.rados.rados.0.smithi071.stdout:194: done (2 left) 2024-09-11T01:30:57.372 INFO:tasks.rados.rados.0.smithi071.stdout:195: done (1 left) 2024-09-11T01:30:57.372 INFO:tasks.rados.rados.0.smithi071.stdout:197: done (0 left) 2024-09-11T01:30:57.372 INFO:tasks.rados.rados.0.smithi071.stdout:198: delete oid 19 current snap is 20 2024-09-11T01:30:57.377 INFO:tasks.rados.rados.0.smithi071.stdout:198: done (0 left) 2024-09-11T01:30:57.377 INFO:tasks.rados.rados.0.smithi071.stdout:199: snap_remove snap 19 2024-09-11T01:30:58.374 INFO:tasks.rados.rados.0.smithi071.stdout:199: done (0 left) 2024-09-11T01:30:58.374 INFO:tasks.rados.rados.0.smithi071.stdout:200: read oid 12 snap 15 2024-09-11T01:30:58.374 INFO:tasks.rados.rados.0.smithi071.stdout:200: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-11T01:30:58.374 INFO:tasks.rados.rados.0.smithi071.stdout:201: read oid 48 snap -1 2024-09-11T01:30:58.374 INFO:tasks.rados.rados.0.smithi071.stdout:201: expect (ObjNum 67 snap 11 seq_num 67) 2024-09-11T01:30:58.374 INFO:tasks.rados.rados.0.smithi071.stdout:202: rollback oid 36 current snap is 20 2024-09-11T01:30:58.374 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 36 to 15 2024-09-11T01:30:58.375 INFO:tasks.rados.rados.0.smithi071.stdout:203: write (excl) oid 50 current snap is 20 2024-09-11T01:30:58.375 INFO:tasks.rados.rados.0.smithi071.stdout:203: seq_num 95 ranges {796665=544891,1759554=650689,2976243=620422} 2024-09-11T01:30:58.381 INFO:tasks.rados.rados.0.smithi071.stdout:203: writing smithi07134777-50 from 796665 to 1341556 tid 1 2024-09-11T01:30:58.387 INFO:tasks.rados.rados.0.smithi071.stdout:203: writing smithi07134777-50 from 1759554 to 2410243 tid 2 2024-09-11T01:30:58.394 INFO:tasks.rados.rados.0.smithi071.stdout:203: writing smithi07134777-50 from 2976243 to 3596665 tid 3 2024-09-11T01:30:58.394 INFO:tasks.rados.rados.0.smithi071.stdout:204: snap_create 2024-09-11T01:30:58.394 INFO:tasks.rados.rados.0.smithi071.stdout:200: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-11T01:30:58.453 INFO:tasks.rados.rados.0.smithi071.stdout:202: finishing rollback tid 0 to smithi07134777-36 2024-09-11T01:30:58.453 INFO:tasks.rados.rados.0.smithi071.stdout:202: finishing rollback tid 1 to smithi07134777-36 2024-09-11T01:30:58.453 INFO:tasks.rados.rados.0.smithi071.stdout:202: finishing rollback tid 2 to smithi07134777-36 2024-09-11T01:30:58.454 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 43 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-11T01:30:58.454 INFO:tasks.rados.rados.0.smithi071.stdout:201: expect (ObjNum 67 snap 11 seq_num 67) 2024-09-11T01:30:58.538 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:30:58.584 INFO:tasks.rados.rados.0.smithi071.stdout:203: finishing write tid 1 to smithi07134777-50 2024-09-11T01:30:58.584 INFO:tasks.rados.rados.0.smithi071.stdout:203: finishing write tid 2 to smithi07134777-50 2024-09-11T01:30:58.584 INFO:tasks.rados.rados.0.smithi071.stdout:203: finishing write tid 3 to smithi07134777-50 2024-09-11T01:30:58.584 INFO:tasks.rados.rados.0.smithi071.stdout:203: finishing write tid 4 to smithi07134777-50 2024-09-11T01:30:58.584 INFO:tasks.rados.rados.0.smithi071.stdout:203: finishing write tid 5 to smithi07134777-50 2024-09-11T01:30:58.584 INFO:tasks.rados.rados.0.smithi071.stdout:203: oid 50 updating version 0 to 104 2024-09-11T01:30:58.584 INFO:tasks.rados.rados.0.smithi071.stdout:203: oid 50 updating version 104 to 105 2024-09-11T01:30:58.584 INFO:tasks.rados.rados.0.smithi071.stdout:203: oid 50 updating version 105 to 107 2024-09-11T01:30:58.584 INFO:tasks.rados.rados.0.smithi071.stdout:203: oid 50 version 107 is already newer than 106 2024-09-11T01:30:58.584 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 107 (ObjNum 95 snap 20 seq_num 95) dirty exists 2024-09-11T01:30:58.585 INFO:tasks.rados.rados.0.smithi071.stdout:203: left oid 50 (ObjNum 95 snap 20 seq_num 95) 2024-09-11T01:30:59.377 INFO:tasks.rados.rados.0.smithi071.stdout:200: done (4 left) 2024-09-11T01:30:59.377 INFO:tasks.rados.rados.0.smithi071.stdout:201: done (3 left) 2024-09-11T01:30:59.377 INFO:tasks.rados.rados.0.smithi071.stdout:202: done (2 left) 2024-09-11T01:30:59.377 INFO:tasks.rados.rados.0.smithi071.stdout:203: done (1 left) 2024-09-11T01:30:59.377 INFO:tasks.rados.rados.0.smithi071.stdout:204: done (0 left) 2024-09-11T01:30:59.377 INFO:tasks.rados.rados.0.smithi071.stdout:205: read oid 37 snap -1 2024-09-11T01:30:59.377 INFO:tasks.rados.rados.0.smithi071.stdout:205: expect (ObjNum 61 snap 5 seq_num 61) 2024-09-11T01:30:59.377 INFO:tasks.rados.rados.0.smithi071.stdout:206: read oid 42 snap 15 2024-09-11T01:30:59.377 INFO:tasks.rados.rados.0.smithi071.stdout:206: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-11T01:30:59.378 INFO:tasks.rados.rados.0.smithi071.stdout:207: delete oid 19 current snap is 21 2024-09-11T01:30:59.381 INFO:tasks.rados.rados.0.smithi071.stdout:206: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-11T01:30:59.415 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1545106 2024-09-11T01:30:59.459 INFO:tasks.rados.rados.0.smithi071.stdout:205: expect (ObjNum 61 snap 5 seq_num 61) 2024-09-11T01:30:59.592 INFO:tasks.rados.rados.0.smithi071.stdout:205: done (2 left) 2024-09-11T01:30:59.592 INFO:tasks.rados.rados.0.smithi071.stdout:206: done (1 left) 2024-09-11T01:30:59.592 INFO:tasks.rados.rados.0.smithi071.stdout:207: done (0 left) 2024-09-11T01:30:59.592 INFO:tasks.rados.rados.0.smithi071.stdout:208: snap_create 2024-09-11T01:31:00.379 INFO:tasks.rados.rados.0.smithi071.stdout:208: done (0 left) 2024-09-11T01:31:00.379 INFO:tasks.rados.rados.0.smithi071.stdout:209: write (excl) oid 16 current snap is 22 2024-09-11T01:31:00.379 INFO:tasks.rados.rados.0.smithi071.stdout:209: seq_num 97 ranges {430279=695389,1876518=654358,3031089=536797} 2024-09-11T01:31:00.384 INFO:tasks.rados.rados.0.smithi071.stdout:209: writing smithi07134777-16 from 430279 to 1125668 tid 1 2024-09-11T01:31:00.392 INFO:tasks.rados.rados.0.smithi071.stdout:209: writing smithi07134777-16 from 1876518 to 2530876 tid 2 2024-09-11T01:31:00.398 INFO:tasks.rados.rados.0.smithi071.stdout:209: writing smithi07134777-16 from 3031089 to 3567886 tid 3 2024-09-11T01:31:00.398 INFO:tasks.rados.rados.0.smithi071.stdout:210: write oid 48 current snap is 22 2024-09-11T01:31:00.399 INFO:tasks.rados.rados.0.smithi071.stdout:209: finishing write tid 1 to smithi07134777-16 2024-09-11T01:31:00.399 INFO:tasks.rados.rados.0.smithi071.stdout:210: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2024-09-11T01:31:00.405 INFO:tasks.rados.rados.0.smithi071.stdout:210: writing smithi07134777-48 from 447086 to 1217724 tid 1 2024-09-11T01:31:00.413 INFO:tasks.rados.rados.0.smithi071.stdout:210: writing smithi07134777-48 from 1935000 to 2733016 tid 2 2024-09-11T01:31:00.419 INFO:tasks.rados.rados.0.smithi071.stdout:210: writing smithi07134777-48 from 3458512 to 3647086 tid 3 2024-09-11T01:31:00.419 INFO:tasks.rados.rados.0.smithi071.stdout:211: snap_create 2024-09-11T01:31:00.419 INFO:tasks.rados.rados.0.smithi071.stdout:209: finishing write tid 2 to smithi07134777-16 2024-09-11T01:31:00.420 INFO:tasks.rados.rados.0.smithi071.stdout:209: finishing write tid 3 to smithi07134777-16 2024-09-11T01:31:00.420 INFO:tasks.rados.rados.0.smithi071.stdout:209: finishing write tid 4 to smithi07134777-16 2024-09-11T01:31:00.420 INFO:tasks.rados.rados.0.smithi071.stdout:209: finishing write tid 5 to smithi07134777-16 2024-09-11T01:31:00.420 INFO:tasks.rados.rados.0.smithi071.stdout:209: oid 16 updating version 0 to 54 2024-09-11T01:31:00.420 INFO:tasks.rados.rados.0.smithi071.stdout:209: oid 16 updating version 54 to 57 2024-09-11T01:31:00.420 INFO:tasks.rados.rados.0.smithi071.stdout:209: oid 16 version 57 is already newer than 55 2024-09-11T01:31:00.420 INFO:tasks.rados.rados.0.smithi071.stdout:209: oid 16 version 57 is already newer than 56 2024-09-11T01:31:00.420 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 57 (ObjNum 97 snap 22 seq_num 97) dirty exists 2024-09-11T01:31:00.420 INFO:tasks.rados.rados.0.smithi071.stdout:209: left oid 16 (ObjNum 97 snap 22 seq_num 97) 2024-09-11T01:31:00.420 INFO:tasks.rados.rados.0.smithi071.stdout:210: finishing write tid 1 to smithi07134777-48 2024-09-11T01:31:00.420 INFO:tasks.rados.rados.0.smithi071.stdout:210: finishing write tid 2 to smithi07134777-48 2024-09-11T01:31:00.421 INFO:tasks.rados.rados.0.smithi071.stdout:210: finishing write tid 3 to smithi07134777-48 2024-09-11T01:31:00.421 INFO:tasks.rados.rados.0.smithi071.stdout:210: finishing write tid 4 to smithi07134777-48 2024-09-11T01:31:00.421 INFO:tasks.rados.rados.0.smithi071.stdout:210: finishing write tid 5 to smithi07134777-48 2024-09-11T01:31:00.421 INFO:tasks.rados.rados.0.smithi071.stdout:210: oid 48 updating version 0 to 112 2024-09-11T01:31:00.421 INFO:tasks.rados.rados.0.smithi071.stdout:210: oid 48 version 112 is already newer than 111 2024-09-11T01:31:00.421 INFO:tasks.rados.rados.0.smithi071.stdout:210: oid 48 version 112 is already newer than 109 2024-09-11T01:31:00.422 INFO:tasks.rados.rados.0.smithi071.stdout:210: oid 48 version 112 is already newer than 110 2024-09-11T01:31:00.422 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 112 (ObjNum 98 snap 22 seq_num 98) dirty exists 2024-09-11T01:31:00.422 INFO:tasks.rados.rados.0.smithi071.stdout:210: left oid 48 (ObjNum 98 snap 22 seq_num 98) 2024-09-11T01:31:01.387 INFO:tasks.rados.rados.0.smithi071.stdout:209: done (2 left) 2024-09-11T01:31:01.387 INFO:tasks.rados.rados.0.smithi071.stdout:210: done (1 left) 2024-09-11T01:31:01.387 INFO:tasks.rados.rados.0.smithi071.stdout:211: done (0 left) 2024-09-11T01:31:01.387 INFO:tasks.rados.rados.0.smithi071.stdout:212: snap_remove snap 15 2024-09-11T01:31:01.959 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-11T01:31:01.959 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T01:31:01.960 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T01:31:01.963 INFO:tasks.ceph.mon.a:Started 2024-09-11T01:31:01.963 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:31:01.963 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:31:02.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.073+0000 7f0810857640 10 monclient: get_monmap_and_config 2024-09-11T01:31:02.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.073+0000 7f0810857640 10 monclient: build_initial_monmap 2024-09-11T01:31:02.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.073+0000 7f0810857640 10 monclient: monmap: 2024-09-11T01:31:02.076 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:02.076 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:02.076 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:31:02.074910+0000 2024-09-11T01:31:02.076 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:31:02.074910+0000 2024-09-11T01:31:02.076 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:02.076 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:02.076 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:31:02.076 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:31:02.076 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:31:02.076 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:31:02.077 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:31:02.077 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.073+0000 7f0810857640 1 Processor -- start 2024-09-11T01:31:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.073+0000 7f0810857640 1 -- start start 2024-09-11T01:31:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.073+0000 7f0810857640 10 monclient: init 2024-09-11T01:31:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f0810857640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f0810857640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-11T01:31:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f0810857640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f080c138230 0x7f080c138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f0810857640 10 monclient(hunting): picked mon.noname-b con 0x7f080c138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f0810857640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f080c09d030 addr v1:172.21.15.12:6789/0 2024-09-11T01:31:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f0810857640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f080c09d430 0x7f080c0a5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f0810857640 10 monclient(hunting): picked mon.noname-a con 0x7f080c09d430 addr v2:172.21.15.12:3300/0 2024-09-11T01:31:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f0810857640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f080c0a6e30 con 0x7f080c09d030 2024-09-11T01:31:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f0810857640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f0810857640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f080c09c9c0 con 0x7f080c09d430 2024-09-11T01:31:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f0810857640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f0810857640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f080c09ba20 con 0x7f080c138230 2024-09-11T01:31:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f0810857640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f0810857640 10 monclient(hunting): authenticate will time out at 868.517568s 2024-09-11T01:31:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f080ad9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f080c09d430 0x7f080c0a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f080a59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f080c138230 0x7f080c138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f080ad9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f080c09d430 0x7f080c0a5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54822/0 (socket says 172.21.15.12:54822) 2024-09-11T01:31:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f0809d9b640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f080c09d030 0x7f080c0a5020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53286/0 (socket says 172.21.15.12:53286) 2024-09-11T01:31:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f080ad9d640 1 -- 172.21.15.12:0/4242647159 learned_addr learned my addr 172.21.15.12:0/4242647159 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f080ad9d640 10 monclient(hunting): get_auth_request con 0x7f080c09d430 auth_method 0 2024-09-11T01:31:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f080ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f080ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f080ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f080a59c640 1 --2- 172.21.15.12:0/4242647159 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f080c138230 0x7f080c138610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46860/0 (socket says 172.21.15.12:46860) 2024-09-11T01:31:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f080a59c640 10 monclient(hunting): get_auth_request con 0x7f080c138230 auth_method 0 2024-09-11T01:31:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f080a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f080a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f080a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f080ad9d640 1 -- 172.21.15.12:0/4242647159 >> v2:172.21.15.12:3300/0 conn(0x7f080c09d430 msgr2=0x7f080c0a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:31:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f080a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f080ad9d640 1 -- 172.21.15.12:0/4242647159 >> v2:172.21.15.12:3300/0 conn(0x7f080c09d430 msgr2=0x7f080c0a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:31:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f080a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f080ad9d640 1 --2- 172.21.15.12:0/4242647159 >> v2:172.21.15.12:3300/0 conn(0x7f080c09d430 0x7f080c0a5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:31:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f080a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f080ad9d640 1 --2- 172.21.15.12:0/4242647159 >> v2:172.21.15.12:3300/0 conn(0x7f080c09d430 0x7f080c0a5730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:31:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.074+0000 7f080a59c640 10 monclient(hunting): handle_auth_done global_id 4520 payload 274 2024-09-11T01:31:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.075+0000 7f080a59c640 1 -- 172.21.15.12:0/4242647159 >> v2:172.21.15.12:3300/0 conn(0x7f080c09d430 msgr2=0x7f080c0a5730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:31:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.075+0000 7f080a59c640 1 --2- 172.21.15.12:0/4242647159 >> v2:172.21.15.12:3300/0 conn(0x7f080c09d430 0x7f080c0a5730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.075+0000 7f080a59c640 1 -- 172.21.15.12:0/4242647159 >> v1:172.21.15.12:6789/0 conn(0x7f080c09d030 legacy=0x7f080c0a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.075+0000 7f080a59c640 10 monclient: _finish_hunting 0 2024-09-11T01:31:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.075+0000 7f080a59c640 1 monclient: found mon.noname-b 2024-09-11T01:31:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.075+0000 7f080a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.075+0000 7f080a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:31:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.075+0000 7f080a59c640 1 -- 172.21.15.12:0/4242647159 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f080c0a66a0 con 0x7f080c138230 2024-09-11T01:31:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.077+0000 7f080a59c640 1 -- 172.21.15.12:0/4242647159 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f080c138230 msgr2=0x7f080c138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.077+0000 7f080a59c640 1 --2- 172.21.15.12:0/4242647159 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f080c138230 0x7f080c138610 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f080c09cf80 tx=0x7f080402a460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be07d8c6f2354ea3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f080959a640 1 -- 172.21.15.12:0/4242647159 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0804007950 con 0x7f080c138230 2024-09-11T01:31:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f080959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f080959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:31:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f080959a640 10 monclient: dump: 2024-09-11T01:31:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:02.088 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f080959a640 10 monclient: _finish_auth 0 2024-09-11T01:31:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f080959a640 10 monclient: _check_auth_tickets 2024-09-11T01:31:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f080959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f0810857640 5 monclient: authenticate success, global_id 4520 2024-09-11T01:31:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f080959a640 1 -- 172.21.15.12:0/4242647159 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0804007af0 con 0x7f080c138230 2024-09-11T01:31:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f0810857640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:31:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f080959a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f080959a640 1 -- 172.21.15.12:0/4242647159 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0804031400 con 0x7f080c138230 2024-09-11T01:31:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f0810857640 10 monclient: get_monmap_and_config success 2024-09-11T01:31:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f0810857640 4 set_mon_vals no callback set 2024-09-11T01:31:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f080959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f080959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f080959a640 10 monclient: dump: 2024-09-11T01:31:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:02.090 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f0810857640 10 monclient: shutdown 2024-09-11T01:31:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f0810857640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f0810857640 1 -- 172.21.15.12:0/4242647159 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f080c138230 msgr2=0x7f080c138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f0810857640 1 --2- 172.21.15.12:0/4242647159 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f080c138230 0x7f080c138610 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f080c09cf80 tx=0x7f080402a460 comp rx=0 tx=0).stop 2024-09-11T01:31:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f0810857640 1 -- 172.21.15.12:0/4242647159 shutdown_connections 2024-09-11T01:31:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f0810857640 1 --2- 172.21.15.12:0/4242647159 >> v2:172.21.15.12:3300/0 conn(0x7f080c09d430 0x7f080c0a5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f0810857640 1 --2- 172.21.15.12:0/4242647159 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f080c138230 0x7f080c138610 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f0810857640 1 -- 172.21.15.12:0/4242647159 >> 172.21.15.12:0/4242647159 conn(0x7f080c0a7120 msgr2=0x7f080c0a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f0810857640 1 -- 172.21.15.12:0/4242647159 shutdown_connections 2024-09-11T01:31:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.078+0000 7f0810857640 1 -- 172.21.15.12:0/4242647159 wait complete. 2024-09-11T01:31:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.079+0000 7f0810857640 10 monclient: build_initial_monmap 2024-09-11T01:31:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.079+0000 7f0810857640 10 monclient: monmap: 2024-09-11T01:31:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:31:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:31:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:31:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:31:02.092 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:31:02.092 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:31:02.092 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:02.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.079+0000 7f0810857640 1 Processor -- start 2024-09-11T01:31:02.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.079+0000 7f0810857640 1 -- start start 2024-09-11T01:31:02.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.079+0000 7f0810857640 10 monclient: init 2024-09-11T01:31:02.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f0810857640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:02.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f0810857640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:31:02.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f0810857640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f080c09d030 0x7f080c1ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:02.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f0810857640 10 monclient(hunting): picked mon.noname-a con 0x7f080c09d030 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:02.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f0810857640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f080c09d430 0x7f080c136e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:02.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f0810857640 10 monclient(hunting): picked mon.noname-b con 0x7f080c09d430 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:31:02.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f0810857640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f080c138230 0x7f080c137370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:02.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f0810857640 10 monclient(hunting): picked mon.noname-c con 0x7f080c138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:02.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f0810857640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:02.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f0810857640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f080c06f970 con 0x7f080c09d030 2024-09-11T01:31:02.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f0810857640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:02.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f0810857640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f080c06f7f0 con 0x7f080c09d430 2024-09-11T01:31:02.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f0810857640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:02.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f0810857640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f080c0a5c70 con 0x7f080c138230 2024-09-11T01:31:02.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f0810857640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:02.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f0810857640 10 monclient(hunting): authenticate will time out at 868.523402s 2024-09-11T01:31:02.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080a59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f080c09d030 0x7f080c1ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:02.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080ad9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f080c138230 0x7f080c137370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:02.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080a59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f080c09d030 0x7f080c1ccf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54824/0 (socket says 172.21.15.12:54824) 2024-09-11T01:31:02.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080a59c640 1 -- 172.21.15.12:0/2581184152 learned_addr learned my addr 172.21.15.12:0/2581184152 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080a59c640 10 monclient(hunting): get_auth_request con 0x7f080c09d030 auth_method 0 2024-09-11T01:31:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080ad9d640 10 monclient(hunting): get_auth_request con 0x7f080c138230 auth_method 0 2024-09-11T01:31:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080a59c640 1 -- 172.21.15.12:0/2581184152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f080c09d030 msgr2=0x7f080c1ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:31:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080a59c640 1 -- 172.21.15.12:0/2581184152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f080c09d030 msgr2=0x7f080c1ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:31:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080a59c640 1 --2- 172.21.15.12:0/2581184152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f080c09d030 0x7f080c1ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:31:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080a59c640 1 --2- 172.21.15.12:0/2581184152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f080c09d030 0x7f080c1ccf10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:31:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f0809d9b640 1 --2- 172.21.15.12:0/2581184152 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f080c09d430 0x7f080c136e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080ad9d640 10 monclient(hunting): handle_auth_done global_id 4523 payload 824 2024-09-11T01:31:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080ad9d640 1 -- 172.21.15.12:0/2581184152 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f080c09d430 msgr2=0x7f080c136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080ad9d640 1 --2- 172.21.15.12:0/2581184152 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f080c09d430 0x7f080c136e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080ad9d640 1 -- 172.21.15.12:0/2581184152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f080c09d030 msgr2=0x7f080c1ccf10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:31:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080ad9d640 1 --2- 172.21.15.12:0/2581184152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f080c09d030 0x7f080c1ccf10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080ad9d640 10 monclient: _finish_hunting 0 2024-09-11T01:31:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080ad9d640 1 monclient: found mon.noname-c 2024-09-11T01:31:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:31:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.080+0000 7f080ad9d640 1 -- 172.21.15.12:0/2581184152 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f080c0a13a0 con 0x7f080c138230 2024-09-11T01:31:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.083+0000 7f080ad9d640 1 -- 172.21.15.12:0/2581184152 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f080c138230 msgr2=0x7f080c137370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:02.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.083+0000 7f080ad9d640 1 --2- 172.21.15.12:0/2581184152 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f080c138230 0x7f080c137370 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f080c09d860 tx=0x7f07fc00da30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:02.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.083+0000 7f0802ffd640 1 -- 172.21.15.12:0/2581184152 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07fc004030 con 0x7f080c138230 2024-09-11T01:31:02.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.083+0000 7f0802ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:02.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.083+0000 7f0802ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:31:02.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.083+0000 7f0802ffd640 10 monclient: dump: 2024-09-11T01:31:02.096 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:02.096 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:02.096 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:02.096 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:02.096 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:02.096 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:02.096 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:02.096 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:02.097 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.083+0000 7f0802ffd640 10 monclient: _finish_auth 0 2024-09-11T01:31:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.083+0000 7f0802ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:31:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.083+0000 7f0802ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.083+0000 7f0810857640 5 monclient: authenticate success, global_id 4523 2024-09-11T01:31:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.083+0000 7f0802ffd640 1 -- 172.21.15.12:0/2581184152 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07fc0041d0 con 0x7f080c138230 2024-09-11T01:31:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.083+0000 7f0810857640 10 monclient: _renew_subs 2024-09-11T01:31:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.083+0000 7f0810857640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.083+0000 7f0810857640 1 -- 172.21.15.12:0/2581184152 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f080c0a1ab0 con 0x7f080c138230 2024-09-11T01:31:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.083+0000 7f0802ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.083+0000 7f0802ffd640 1 -- 172.21.15.12:0/2581184152 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07fc014830 con 0x7f080c138230 2024-09-11T01:31:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.083+0000 7f0802ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.083+0000 7f0802ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.083+0000 7f0802ffd640 10 monclient: dump: 2024-09-11T01:31:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:02.098 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.083+0000 7f0808d99640 4 set_mon_vals no callback set 2024-09-11T01:31:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.083+0000 7f0810857640 10 monclient: _renew_subs 2024-09-11T01:31:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.083+0000 7f0810857640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.083+0000 7f0810857640 1 -- 172.21.15.12:0/2581184152 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f080c1247f0 con 0x7f080c138230 2024-09-11T01:31:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.084+0000 7f0810857640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.084+0000 7f0810857640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.084+0000 7f0810857640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.084+0000 7f0810857640 1 -- 172.21.15.12:0/2581184152 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f080c138610 con 0x7f080c138230 2024-09-11T01:31:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.084+0000 7f0802ffd640 1 -- 172.21.15.12:0/2581184152 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f07fc025060 con 0x7f080c138230 2024-09-11T01:31:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.084+0000 7f0802ffd640 1 --2- 172.21.15.12:0/2581184152 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f07ec03e210 0x7f07ec0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.084+0000 7f0802ffd640 1 -- 172.21.15.12:0/2581184152 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 4739+0+0 (secure 0 0 0) 0x7f07fc0530f0 con 0x7f080c138230 2024-09-11T01:31:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.085+0000 7f080a59c640 1 --2- 172.21.15.12:0/2581184152 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f07ec03e210 0x7f07ec0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.085+0000 7f080a59c640 10 monclient: get_auth_request con 0x7f07ec03e210 auth_method 0 2024-09-11T01:31:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.087+0000 7f080a59c640 1 --2- 172.21.15.12:0/2581184152 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f07ec03e210 0x7f07ec0406d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0804009bd0 tx=0x7f080402f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.089+0000 7f0802ffd640 1 -- 172.21.15.12:0/2581184152 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f07fc003b30 con 0x7f080c138230 2024-09-11T01:31:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.089+0000 7f0802ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.089+0000 7f0802ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:31:02.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.349+0000 7f0810857640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:31:02.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.349+0000 7f0810857640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:31:02.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.349+0000 7f0810857640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:02.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.349+0000 7f0810857640 1 -- 172.21.15.12:0/2581184152 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f080c0698e0 con 0x7f080c138230 2024-09-11T01:31:02.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.350+0000 7f0802ffd640 1 -- 172.21.15.12:0/2581184152 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f07fc003610 con 0x7f080c138230 2024-09-11T01:31:02.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.350+0000 7f0802ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:31:02.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.350+0000 7f0802ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:31:02.353 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":18,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":22,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:31:02.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.352+0000 7f0810857640 1 -- 172.21.15.12:0/2581184152 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f07ec03e210 msgr2=0x7f07ec0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:02.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.352+0000 7f0810857640 1 --2- 172.21.15.12:0/2581184152 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f07ec03e210 0x7f07ec0406d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0804009bd0 tx=0x7f080402f000 comp rx=0 tx=0).stop 2024-09-11T01:31:02.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.352+0000 7f0810857640 10 monclient: shutdown 2024-09-11T01:31:02.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.352+0000 7f0810857640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:02.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.352+0000 7f0810857640 1 -- 172.21.15.12:0/2581184152 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f080c138230 msgr2=0x7f080c137370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:02.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.352+0000 7f0810857640 1 --2- 172.21.15.12:0/2581184152 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f080c138230 0x7f080c137370 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f080c09d860 tx=0x7f07fc00da30 comp rx=0 tx=0).stop 2024-09-11T01:31:02.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.352+0000 7f0810857640 1 -- 172.21.15.12:0/2581184152 shutdown_connections 2024-09-11T01:31:02.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.352+0000 7f0810857640 1 --2- 172.21.15.12:0/2581184152 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f080c138230 0x7f080c137370 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:02.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.352+0000 7f0810857640 1 --2- 172.21.15.12:0/2581184152 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f080c09d430 0x7f080c136e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:02.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.352+0000 7f0810857640 1 --2- 172.21.15.12:0/2581184152 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f07ec03e210 0x7f07ec0406d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:02.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.352+0000 7f0810857640 1 --2- 172.21.15.12:0/2581184152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f080c09d030 0x7f080c1ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:02.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.352+0000 7f0810857640 1 -- 172.21.15.12:0/2581184152 >> 172.21.15.12:0/2581184152 conn(0x7f080c0a7120 msgr2=0x7f080c0a8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:02.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.352+0000 7f0810857640 1 -- 172.21.15.12:0/2581184152 shutdown_connections 2024-09-11T01:31:02.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:02.352+0000 7f0810857640 1 -- 172.21.15.12:0/2581184152 wait complete. 2024-09-11T01:31:02.371 INFO:tasks.rados.rados.0.smithi071.stdout:212: done (0 left) 2024-09-11T01:31:02.371 INFO:tasks.rados.rados.0.smithi071.stdout:213: snap_remove snap 20 2024-09-11T01:31:03.374 INFO:tasks.rados.rados.0.smithi071.stdout:213: done (0 left) 2024-09-11T01:31:03.374 INFO:tasks.rados.rados.0.smithi071.stdout:214: read oid 48 snap -1 2024-09-11T01:31:03.374 INFO:tasks.rados.rados.0.smithi071.stdout:214: expect (ObjNum 98 snap 22 seq_num 98) 2024-09-11T01:31:03.374 INFO:tasks.rados.rados.0.smithi071.stdout:215: read oid 34 snap 22 2024-09-11T01:31:03.374 INFO:tasks.rados.rados.0.smithi071.stdout:215: expect deleted 2024-09-11T01:31:03.374 INFO:tasks.rados.rados.0.smithi071.stdout:216: copy_from oid 39 from oid 10 current snap is 23 2024-09-11T01:31:03.374 INFO:tasks.rados.rados.0.smithi071.stdout:217: snap_remove snap 21 2024-09-11T01:31:04.377 INFO:tasks.rados.rados.0.smithi071.stdout:217: done (3 left) 2024-09-11T01:31:04.377 INFO:tasks.rados.rados.0.smithi071.stdout:218: copy_from oid 30 from oid 1 current snap is 23 2024-09-11T01:31:04.377 INFO:tasks.rados.rados.0.smithi071.stdout:214: expect (ObjNum 98 snap 22 seq_num 98) 2024-09-11T01:31:04.383 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1040769 2024-09-11T01:31:04.563 INFO:tasks.rados.rados.0.smithi071.stdout:216: finishing copy_from racing read to smithi07134777-39 2024-09-11T01:31:04.563 INFO:tasks.rados.rados.0.smithi071.stdout:216: finishing copy_from to smithi07134777-39 2024-09-11T01:31:04.563 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 29 (ObjNum 88 snap 19 seq_num 88) dirty exists 2024-09-11T01:31:04.563 INFO:tasks.rados.rados.0.smithi071.stdout:218: finishing copy_from racing read to smithi07134777-30 2024-09-11T01:31:04.563 INFO:tasks.rados.rados.0.smithi071.stdout:218: finishing copy_from to smithi07134777-30 2024-09-11T01:31:04.563 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 35 (ObjNum 60 snap 5 seq_num 60) dirty exists 2024-09-11T01:31:04.563 INFO:tasks.rados.rados.0.smithi071.stdout:214: done (3 left) 2024-09-11T01:31:04.563 INFO:tasks.rados.rados.0.smithi071.stdout:215: done (2 left) 2024-09-11T01:31:04.563 INFO:tasks.rados.rados.0.smithi071.stdout:216: done (1 left) 2024-09-11T01:31:04.564 INFO:tasks.rados.rados.0.smithi071.stdout:218: done (0 left) 2024-09-11T01:31:04.564 INFO:tasks.rados.rados.0.smithi071.stdout:219: snap_remove snap 22 2024-09-11T01:31:05.366 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:31:05.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.476+0000 7f44e1847640 10 monclient: get_monmap_and_config 2024-09-11T01:31:05.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.476+0000 7f44e1847640 10 monclient: build_initial_monmap 2024-09-11T01:31:05.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.476+0000 7f44e1847640 10 monclient: monmap: 2024-09-11T01:31:05.479 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:05.479 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:05.479 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:31:05.478494+0000 2024-09-11T01:31:05.480 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:31:05.478494+0000 2024-09-11T01:31:05.480 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:05.480 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:05.480 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:31:05.480 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:31:05.480 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:31:05.480 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:31:05.480 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:31:05.480 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:05.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.477+0000 7f44e1847640 1 Processor -- start 2024-09-11T01:31:05.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.477+0000 7f44e1847640 1 -- start start 2024-09-11T01:31:05.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.477+0000 7f44e1847640 10 monclient: init 2024-09-11T01:31:05.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.478+0000 7f44e1847640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:05.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.478+0000 7f44e1847640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-11T01:31:05.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.478+0000 7f44e1847640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 0x7f44dc138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:05.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.478+0000 7f44e1847640 10 monclient(hunting): picked mon.noname-b con 0x7f44dc138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:05.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.478+0000 7f44e1847640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f44dc09d030 addr v1:172.21.15.12:6789/0 2024-09-11T01:31:05.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.478+0000 7f44e1847640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f44dc09d430 addr v1:172.21.15.71:6789/0 2024-09-11T01:31:05.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.478+0000 7f44e1847640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44dc0a5e40 con 0x7f44dc09d030 2024-09-11T01:31:05.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.478+0000 7f44e1847640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44dc0a6e30 con 0x7f44dc09d430 2024-09-11T01:31:05.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.478+0000 7f44e1847640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:05.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.478+0000 7f44e1847640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f44dc09c9c0 con 0x7f44dc138230 2024-09-11T01:31:05.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.478+0000 7f44e1847640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:05.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.478+0000 7f44e1847640 10 monclient(hunting): authenticate will time out at 871.921152s 2024-09-11T01:31:05.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.478+0000 7f44daffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 0x7f44dc138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:05.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.478+0000 7f44daffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 0x7f44dc138610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46914/0 (socket says 172.21.15.12:46914) 2024-09-11T01:31:05.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.478+0000 7f44daffd640 1 -- 172.21.15.12:0/1663548296 learned_addr learned my addr 172.21.15.12:0/1663548296 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:05.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.478+0000 7f44d3fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f44dc09d030 0x7f44dc0a5020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53296/0 (socket says 172.21.15.12:53296) 2024-09-11T01:31:05.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.478+0000 7f44daffd640 10 monclient(hunting): get_auth_request con 0x7f44dc138230 auth_method 0 2024-09-11T01:31:05.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.478+0000 7f44daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.478+0000 7f44daffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.478+0000 7f44daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.478+0000 7f44daffd640 1 -- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 msgr2=0x7f44dc138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:31:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.478+0000 7f44daffd640 1 -- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 msgr2=0x7f44dc138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:31:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.478+0000 7f44daffd640 1 --2- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 0x7f44dc138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:31:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.478+0000 7f44daffd640 1 --2- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 0x7f44dc138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:31:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.480+0000 7f44d3fff640 1 -- 172.21.15.12:0/1663548296 >> v1:172.21.15.12:6789/0 conn(0x7f44dc09d030 legacy=0x7f44dc0a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:05.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.480+0000 7f44db7fe640 1 -- 172.21.15.12:0/1663548296 >> v1:172.21.15.71:6789/0 conn(0x7f44dc09d430 legacy=0x7f44dc0a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:05.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.680+0000 7f44daffd640 1 --2- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 0x7f44dc138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:05.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.680+0000 7f44daffd640 10 monclient(hunting): get_auth_request con 0x7f44dc138230 auth_method 0 2024-09-11T01:31:05.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.680+0000 7f44daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:05.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.680+0000 7f44daffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:05.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.680+0000 7f44daffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:31:05.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.680+0000 7f44daffd640 1 -- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 msgr2=0x7f44dc138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:31:05.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.680+0000 7f44daffd640 1 -- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 msgr2=0x7f44dc138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:31:05.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.680+0000 7f44daffd640 1 --2- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 0x7f44dc138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:31:05.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:05.680+0000 7f44daffd640 1 --2- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 0x7f44dc138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:31:06.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:06.081+0000 7f44daffd640 1 --2- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 0x7f44dc138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:06.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:06.081+0000 7f44daffd640 10 monclient(hunting): get_auth_request con 0x7f44dc138230 auth_method 0 2024-09-11T01:31:06.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:06.081+0000 7f44daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:06.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:06.081+0000 7f44daffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:06.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:06.081+0000 7f44daffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:31:06.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:06.081+0000 7f44daffd640 1 -- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 msgr2=0x7f44dc138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:31:06.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:06.081+0000 7f44daffd640 1 -- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 msgr2=0x7f44dc138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:31:06.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:06.081+0000 7f44daffd640 1 --2- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 0x7f44dc138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:31:06.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:06.081+0000 7f44daffd640 1 --2- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 0x7f44dc138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:31:06.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:06.883+0000 7f44daffd640 1 --2- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 0x7f44dc138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:06.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:06.883+0000 7f44daffd640 10 monclient(hunting): get_auth_request con 0x7f44dc138230 auth_method 0 2024-09-11T01:31:06.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:06.883+0000 7f44daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:06.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:06.883+0000 7f44daffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:06.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:06.883+0000 7f44daffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:31:06.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:06.883+0000 7f44daffd640 1 -- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 msgr2=0x7f44dc138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:31:06.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:06.883+0000 7f44daffd640 1 -- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 msgr2=0x7f44dc138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:31:06.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:06.883+0000 7f44daffd640 1 --2- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 0x7f44dc138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:31:06.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:06.883+0000 7f44daffd640 1 --2- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 0x7f44dc138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:31:08.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 1 -- 172.21.15.12:0/1663548296 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1392835021 0 0) 0x7f44dc0a6e30 con 0x7f44dc09d430 2024-09-11T01:31:08.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 10 monclient(hunting): _negotiate 2024-09-11T01:31:08.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:08.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:08.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 10 monclient(hunting): my global_id is 4429 2024-09-11T01:31:08.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 1 -- 172.21.15.12:0/1663548296 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44c8003520 con 0x7f44dc09d430 2024-09-11T01:31:08.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 1 -- 172.21.15.12:0/1663548296 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3607155822 0 0) 0x7f44c8003520 con 0x7f44dc09d430 2024-09-11T01:31:08.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 1 -- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 msgr2=0x7f44dc138610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:31:08.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 1 --2- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 0x7f44dc138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:08.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 1 -- 172.21.15.12:0/1663548296 >> v1:172.21.15.12:6789/0 conn(0x7f44dc09d030 legacy=0x7f44dc0a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:08.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 10 monclient: _finish_hunting 0 2024-09-11T01:31:08.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 1 monclient: found mon.noname-c-legacy 2024-09-11T01:31:08.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:08.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-09-11T01:31:08.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 1 -- 172.21.15.12:0/1663548296 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44dc09ba20 con 0x7f44dc09d430 2024-09-11T01:31:08.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 10 monclient: _finish_auth 0 2024-09-11T01:31:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:31:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 1 -- 172.21.15.12:0/1663548296 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f44d4001d70 con 0x7f44dc09d430 2024-09-11T01:31:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44e1847640 5 monclient: authenticate success, global_id 4429 2024-09-11T01:31:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44e1847640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:31:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-11T01:31:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 10 monclient: dump: 2024-09-11T01:31:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.302 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.303 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:08.303 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:08.303 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:08.303 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:08.303 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:08.303 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:08.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-09-11T01:31:08.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:08.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 1 -- 172.21.15.12:0/1663548296 >> v1:172.21.15.71:6789/0 conn(0x7f44dc09d430 legacy=0x7f44dc0a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:08.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:31:08.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 1 --2- 172.21.15.12:0/1663548296 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44c8003ad0 0x7f44c8003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:08.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 10 monclient(hunting): picked mon.b con 0x7f44c8003ad0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:31:08.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 1 --2- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44c8004800 0x7f44c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:08.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 10 monclient(hunting): picked mon.a con 0x7f44c8004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:08.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 1 --2- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 0x7f44c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:08.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 10 monclient(hunting): picked mon.c con 0x7f44dc138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:08.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:08.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 1 -- 172.21.15.12:0/1663548296 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f44dc09c9c0 con 0x7f44c8004800 2024-09-11T01:31:08.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:08.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 1 -- 172.21.15.12:0/1663548296 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f44c8005680 con 0x7f44c8003ad0 2024-09-11T01:31:08.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:08.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 1 -- 172.21.15.12:0/1663548296 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f44c800bf80 con 0x7f44dc138230 2024-09-11T01:31:08.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:08.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44da7fc640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:31:08.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44e1847640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:31:08.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44db7fe640 1 --2- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 0x7f44c800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:08.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44db7fe640 10 monclient(hunting): get_auth_request con 0x7f44dc138230 auth_method 0 2024-09-11T01:31:08.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:08.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:08.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44db7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:31:08.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44d3fff640 1 --2- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44c8004800 0x7f44c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:08.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.295+0000 7f44daffd640 1 --2- 172.21.15.12:0/1663548296 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44c8003ad0 0x7f44c8003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:08.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44daffd640 10 monclient(hunting): get_auth_request con 0x7f44c8003ad0 auth_method 0 2024-09-11T01:31:08.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:08.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44daffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:08.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44daffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:31:08.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44d3fff640 10 monclient(hunting): get_auth_request con 0x7f44c8004800 auth_method 0 2024-09-11T01:31:08.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:08.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:08.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44d3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:31:08.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:08.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:08.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:31:08.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:08.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:08.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44daffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:31:08.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:08.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:08.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:31:08.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44db7fe640 10 monclient(hunting): handle_auth_done global_id 4429 payload 293 2024-09-11T01:31:08.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44db7fe640 1 -- 172.21.15.12:0/1663548296 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44c8003ad0 msgr2=0x7f44c8003ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:08.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44db7fe640 1 --2- 172.21.15.12:0/1663548296 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44c8003ad0 0x7f44c8003ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:08.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44db7fe640 1 -- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44c8004800 msgr2=0x7f44c8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:08.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44db7fe640 1 --2- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44c8004800 0x7f44c8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:08.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44db7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:31:08.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44db7fe640 1 monclient: found mon.c 2024-09-11T01:31:08.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:08.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44db7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:08.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44db7fe640 1 -- 172.21.15.12:0/1663548296 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44c8003520 con 0x7f44dc138230 2024-09-11T01:31:08.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44db7fe640 10 monclient: _finish_auth 0 2024-09-11T01:31:08.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44db7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:31:08.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.296+0000 7f44db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:08.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.299+0000 7f44db7fe640 1 -- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 msgr2=0x7f44c800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.299+0000 7f44db7fe640 1 --2- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 0x7f44c800b9c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f44dc09d860 tx=0x7f44c8005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9730cc582caec1b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.302+0000 7f44da7fc640 1 -- 172.21.15.12:0/1663548296 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d4031070 con 0x7f44dc138230 2024-09-11T01:31:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.302+0000 7f44da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.302+0000 7f44da7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.302+0000 7f44da7fc640 10 monclient: dump: 2024-09-11T01:31:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:08.315 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:08.316 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:08.316 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:08.316 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:08.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.302+0000 7f44da7fc640 1 -- 172.21.15.12:0/1663548296 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44d4005e00 con 0x7f44dc138230 2024-09-11T01:31:08.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.302+0000 7f44da7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:08.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.302+0000 7f44da7fc640 1 -- 172.21.15.12:0/1663548296 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d400a2d0 con 0x7f44dc138230 2024-09-11T01:31:08.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.302+0000 7f44e1847640 10 monclient: get_monmap_and_config success 2024-09-11T01:31:08.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.302+0000 7f44e1847640 4 set_mon_vals no callback set 2024-09-11T01:31:08.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.302+0000 7f44da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:08.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.302+0000 7f44da7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:08.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.302+0000 7f44da7fc640 10 monclient: dump: 2024-09-11T01:31:08.316 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:08.316 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:08.317 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.317 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.317 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:08.317 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:08.317 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:08.317 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:08.317 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:08.317 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:08.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.302+0000 7f44e1847640 10 monclient: shutdown 2024-09-11T01:31:08.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.302+0000 7f44e1847640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:08.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.302+0000 7f44e1847640 1 -- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 msgr2=0x7f44c800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:08.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.302+0000 7f44e1847640 1 --2- 172.21.15.12:0/1663548296 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc138230 0x7f44c800b9c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f44dc09d860 tx=0x7f44c8005830 comp rx=0 tx=0).stop 2024-09-11T01:31:08.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.302+0000 7f44db7fe640 1 -- 172.21.15.12:0/1663548296 reap_dead start 2024-09-11T01:31:08.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.303+0000 7f44e1847640 1 -- 172.21.15.12:0/1663548296 shutdown_connections 2024-09-11T01:31:08.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.303+0000 7f44e1847640 1 -- 172.21.15.12:0/1663548296 >> 172.21.15.12:0/1663548296 conn(0x7f44dc0a7120 msgr2=0x7f44dc0a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:08.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.303+0000 7f44e1847640 1 -- 172.21.15.12:0/1663548296 shutdown_connections 2024-09-11T01:31:08.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.303+0000 7f44e1847640 1 -- 172.21.15.12:0/1663548296 wait complete. 2024-09-11T01:31:08.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.303+0000 7f44e1847640 10 monclient: build_initial_monmap 2024-09-11T01:31:08.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.304+0000 7f44e1847640 10 monclient: monmap: 2024-09-11T01:31:08.318 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:08.318 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:31:08.318 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:31:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:31:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:31:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:31:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:31:08.319 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.304+0000 7f44e1847640 1 Processor -- start 2024-09-11T01:31:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.304+0000 7f44e1847640 1 -- start start 2024-09-11T01:31:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.304+0000 7f44e1847640 10 monclient: init 2024-09-11T01:31:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.304+0000 7f44e1847640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.304+0000 7f44e1847640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:31:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44e1847640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44dc138230 0x7f44dc136f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:08.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44e1847640 10 monclient(hunting): picked mon.noname-a con 0x7f44dc138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44e1847640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc0a1810 0x7f44dc137440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44e1847640 10 monclient(hunting): picked mon.noname-c con 0x7f44dc0a1810 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44e1847640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44dc1ccf50 0x7f44dc124c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44e1847640 10 monclient(hunting): picked mon.noname-b con 0x7f44dc1ccf50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:31:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44e1847640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44e1847640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f44dc138610 con 0x7f44dc138230 2024-09-11T01:31:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44e1847640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44e1847640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f44dc0a6730 con 0x7f44dc1ccf50 2024-09-11T01:31:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44e1847640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44e1847640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f44dc138790 con 0x7f44dc0a1810 2024-09-11T01:31:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44e1847640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44e1847640 10 monclient(hunting): authenticate will time out at 874.748102s 2024-09-11T01:31:08.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44daffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44dc138230 0x7f44dc136f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:08.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44daffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44dc138230 0x7f44dc136f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45646/0 (socket says 172.21.15.12:45646) 2024-09-11T01:31:08.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44daffd640 1 -- 172.21.15.12:0/779986072 learned_addr learned my addr 172.21.15.12:0/779986072 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:08.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44d3fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc0a1810 0x7f44dc137440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:08.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44daffd640 10 monclient(hunting): get_auth_request con 0x7f44dc138230 auth_method 0 2024-09-11T01:31:08.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:08.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44daffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:08.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:08.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44d3fff640 10 monclient(hunting): get_auth_request con 0x7f44dc0a1810 auth_method 0 2024-09-11T01:31:08.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:08.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:08.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:08.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44db7fe640 1 --2- 172.21.15.12:0/779986072 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44dc1ccf50 0x7f44dc124c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:08.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44db7fe640 10 monclient(hunting): get_auth_request con 0x7f44dc1ccf50 auth_method 0 2024-09-11T01:31:08.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:08.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:08.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:08.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:08.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:08.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:08.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:08.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:08.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44d3fff640 10 monclient(hunting): handle_auth_done global_id 4526 payload 824 2024-09-11T01:31:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44d3fff640 1 -- 172.21.15.12:0/779986072 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44dc1ccf50 msgr2=0x7f44dc124c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44d3fff640 1 --2- 172.21.15.12:0/779986072 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44dc1ccf50 0x7f44dc124c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44d3fff640 1 -- 172.21.15.12:0/779986072 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44dc138230 msgr2=0x7f44dc136f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44d3fff640 1 --2- 172.21.15.12:0/779986072 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44dc138230 0x7f44dc136f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44d3fff640 10 monclient: _finish_hunting 0 2024-09-11T01:31:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44d3fff640 1 monclient: found mon.noname-c 2024-09-11T01:31:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44d3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:31:08.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44d3fff640 1 -- 172.21.15.12:0/779986072 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44dc137980 con 0x7f44dc0a1810 2024-09-11T01:31:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.305+0000 7f44daffd640 1 --2- 172.21.15.12:0/779986072 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44dc138230 0x7f44dc136f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:31:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.308+0000 7f44d3fff640 1 -- 172.21.15.12:0/779986072 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc0a1810 msgr2=0x7f44dc137440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.308+0000 7f44d3fff640 1 --2- 172.21.15.12:0/779986072 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc0a1810 0x7f44dc137440 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f44c4000c00 tx=0x7f44c4001970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.308+0000 7f44d8ff9640 1 -- 172.21.15.12:0/779986072 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c4011790 con 0x7f44dc0a1810 2024-09-11T01:31:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.308+0000 7f44d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.308+0000 7f44d8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:31:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.308+0000 7f44d8ff9640 10 monclient: dump: 2024-09-11T01:31:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.324 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:08.325 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:08.325 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:08.325 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:08.325 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:08.325 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:08.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.308+0000 7f44d8ff9640 10 monclient: _finish_auth 0 2024-09-11T01:31:08.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.308+0000 7f44d8ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:31:08.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.308+0000 7f44d8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:08.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.308+0000 7f44d8ff9640 1 -- 172.21.15.12:0/779986072 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44c4018070 con 0x7f44dc0a1810 2024-09-11T01:31:08.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.308+0000 7f44e1847640 5 monclient: authenticate success, global_id 4526 2024-09-11T01:31:08.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.308+0000 7f44e1847640 10 monclient: _renew_subs 2024-09-11T01:31:08.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.308+0000 7f44e1847640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:08.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.308+0000 7f44e1847640 1 -- 172.21.15.12:0/779986072 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44dc1265a0 con 0x7f44dc0a1810 2024-09-11T01:31:08.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.308+0000 7f44d8ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:08.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.308+0000 7f44d8ff9640 1 -- 172.21.15.12:0/779986072 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c4011b70 con 0x7f44dc0a1810 2024-09-11T01:31:08.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.308+0000 7f44d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:08.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.308+0000 7f44d8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:08.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.308+0000 7f44d8ff9640 10 monclient: dump: 2024-09-11T01:31:08.327 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:08.327 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:08.327 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.327 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.327 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:08.327 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:08.327 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:08.327 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:08.327 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:08.327 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:08.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.308+0000 7f44d9ffb640 4 set_mon_vals no callback set 2024-09-11T01:31:08.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.308+0000 7f44e1847640 10 monclient: _renew_subs 2024-09-11T01:31:08.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.308+0000 7f44e1847640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:08.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.308+0000 7f44e1847640 1 -- 172.21.15.12:0/779986072 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f44dc128060 con 0x7f44dc0a1810 2024-09-11T01:31:08.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.309+0000 7f44d8ff9640 1 -- 172.21.15.12:0/779986072 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f44c401d460 con 0x7f44dc0a1810 2024-09-11T01:31:08.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.309+0000 7f44d8ff9640 1 --2- 172.21.15.12:0/779986072 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f44bc03e260 0x7f44bc040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:08.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.309+0000 7f44d8ff9640 1 -- 172.21.15.12:0/779986072 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(67..67 src has 1..67) ==== 4783+0+0 (secure 0 0 0) 0x7f44c4052a10 con 0x7f44dc0a1810 2024-09-11T01:31:08.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.309+0000 7f44e1847640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:08.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.309+0000 7f44e1847640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:08.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.309+0000 7f44e1847640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:08.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.309+0000 7f44e1847640 1 -- 172.21.15.12:0/779986072 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44dc138b60 con 0x7f44dc0a1810 2024-09-11T01:31:08.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.309+0000 7f44daffd640 1 --2- 172.21.15.12:0/779986072 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f44bc03e260 0x7f44bc040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:08.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.309+0000 7f44daffd640 10 monclient: get_auth_request con 0x7f44bc03e260 auth_method 0 2024-09-11T01:31:08.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.312+0000 7f44daffd640 1 --2- 172.21.15.12:0/779986072 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f44bc03e260 0x7f44bc040720 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f44dc127e70 tx=0x7f44cc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:08.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.314+0000 7f44d8ff9640 1 -- 172.21.15.12:0/779986072 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f44c4011d10 con 0x7f44dc0a1810 2024-09-11T01:31:08.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.314+0000 7f44d8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:08.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.314+0000 7f44d8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:31:08.360 INFO:tasks.rados.rados.0.smithi071.stdout:219: done (0 left) 2024-09-11T01:31:08.360 INFO:tasks.rados.rados.0.smithi071.stdout:220: read oid 26 snap -1 2024-09-11T01:31:08.360 INFO:tasks.rados.rados.0.smithi071.stdout:220: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:31:08.361 INFO:tasks.rados.rados.0.smithi071.stdout:221: write (excl) oid 5 current snap is 23 2024-09-11T01:31:08.361 INFO:tasks.rados.rados.0.smithi071.stdout:221: seq_num 99 ranges {463893=562240,1593482=658027,2685935=559341} 2024-09-11T01:31:08.366 INFO:tasks.rados.rados.0.smithi071.stdout:221: writing smithi07134777-5 from 463893 to 1026133 tid 1 2024-09-11T01:31:08.373 INFO:tasks.rados.rados.0.smithi071.stdout:221: writing smithi07134777-5 from 1593482 to 2251509 tid 2 2024-09-11T01:31:08.379 INFO:tasks.rados.rados.0.smithi071.stdout:221: writing smithi07134777-5 from 2685935 to 3245276 tid 3 2024-09-11T01:31:08.379 INFO:tasks.rados.rados.0.smithi071.stdout:222: copy_from oid 23 from oid 6 current snap is 23 2024-09-11T01:31:08.379 INFO:tasks.rados.rados.0.smithi071.stdout:220: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:31:08.426 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:31:08.454 INFO:tasks.rados.rados.0.smithi071.stdout:221: finishing write tid 1 to smithi07134777-5 2024-09-11T01:31:08.454 INFO:tasks.rados.rados.0.smithi071.stdout:221: finishing write tid 2 to smithi07134777-5 2024-09-11T01:31:08.454 INFO:tasks.rados.rados.0.smithi071.stdout:221: finishing write tid 3 to smithi07134777-5 2024-09-11T01:31:08.454 INFO:tasks.rados.rados.0.smithi071.stdout:221: finishing write tid 4 to smithi07134777-5 2024-09-11T01:31:08.454 INFO:tasks.rados.rados.0.smithi071.stdout:221: finishing write tid 5 to smithi07134777-5 2024-09-11T01:31:08.455 INFO:tasks.rados.rados.0.smithi071.stdout:221: oid 5 updating version 0 to 23 2024-09-11T01:31:08.455 INFO:tasks.rados.rados.0.smithi071.stdout:221: oid 5 version 23 is already newer than 22 2024-09-11T01:31:08.455 INFO:tasks.rados.rados.0.smithi071.stdout:221: oid 5 version 23 is already newer than 21 2024-09-11T01:31:08.455 INFO:tasks.rados.rados.0.smithi071.stdout:221: oid 5 updating version 23 to 24 2024-09-11T01:31:08.455 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 24 (ObjNum 99 snap 23 seq_num 99) dirty exists 2024-09-11T01:31:08.455 INFO:tasks.rados.rados.0.smithi071.stdout:221: left oid 5 (ObjNum 99 snap 23 seq_num 99) 2024-09-11T01:31:08.455 INFO:tasks.rados.rados.0.smithi071.stdout:222: finishing copy_from racing read to smithi07134777-23 2024-09-11T01:31:08.455 INFO:tasks.rados.rados.0.smithi071.stdout:222: finishing copy_from to smithi07134777-23 2024-09-11T01:31:08.455 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 49 (ObjNum 63 snap 8 seq_num 63) dirty exists 2024-09-11T01:31:08.455 INFO:tasks.rados.rados.0.smithi071.stdout:220: done (2 left) 2024-09-11T01:31:08.455 INFO:tasks.rados.rados.0.smithi071.stdout:221: done (1 left) 2024-09-11T01:31:08.455 INFO:tasks.rados.rados.0.smithi071.stdout:222: done (0 left) 2024-09-11T01:31:08.455 INFO:tasks.rados.rados.0.smithi071.stdout:223: read oid 7 snap -1 2024-09-11T01:31:08.455 INFO:tasks.rados.rados.0.smithi071.stdout:223: expect deleted 2024-09-11T01:31:08.456 INFO:tasks.rados.rados.0.smithi071.stdout:224: snap_create 2024-09-11T01:31:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.591+0000 7f44e1847640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:31:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.591+0000 7f44e1847640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:31:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.591+0000 7f44e1847640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.591+0000 7f44e1847640 1 -- 172.21.15.12:0/779986072 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f44dc0698e0 con 0x7f44dc0a1810 2024-09-11T01:31:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.591+0000 7f44d8ff9640 1 -- 172.21.15.12:0/779986072 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f44c40133f0 con 0x7f44dc0a1810 2024-09-11T01:31:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.591+0000 7f44d8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:31:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.591+0000 7f44d8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:31:08.595 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:31:08.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.594+0000 7f44e1847640 1 -- 172.21.15.12:0/779986072 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f44bc03e260 msgr2=0x7f44bc040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:08.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.594+0000 7f44e1847640 1 --2- 172.21.15.12:0/779986072 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f44bc03e260 0x7f44bc040720 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f44dc127e70 tx=0x7f44cc008040 comp rx=0 tx=0).stop 2024-09-11T01:31:08.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.594+0000 7f44e1847640 10 monclient: shutdown 2024-09-11T01:31:08.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.594+0000 7f44e1847640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:08.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.594+0000 7f44e1847640 1 -- 172.21.15.12:0/779986072 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc0a1810 msgr2=0x7f44dc137440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:08.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.594+0000 7f44e1847640 1 --2- 172.21.15.12:0/779986072 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc0a1810 0x7f44dc137440 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f44c4000c00 tx=0x7f44c4001970 comp rx=0 tx=0).stop 2024-09-11T01:31:08.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.594+0000 7f44e1847640 1 -- 172.21.15.12:0/779986072 shutdown_connections 2024-09-11T01:31:08.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.594+0000 7f44e1847640 1 --2- 172.21.15.12:0/779986072 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44dc1ccf50 0x7f44dc124c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:08.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.594+0000 7f44e1847640 1 --2- 172.21.15.12:0/779986072 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44dc0a1810 0x7f44dc137440 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:08.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.594+0000 7f44e1847640 1 --2- 172.21.15.12:0/779986072 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f44bc03e260 0x7f44bc040720 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:08.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.594+0000 7f44e1847640 1 --2- 172.21.15.12:0/779986072 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44dc138230 0x7f44dc136f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:08.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.594+0000 7f44e1847640 1 -- 172.21.15.12:0/779986072 >> 172.21.15.12:0/779986072 conn(0x7f44dc0a7120 msgr2=0x7f44dc0a3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:08.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.594+0000 7f44e1847640 1 -- 172.21.15.12:0/779986072 shutdown_connections 2024-09-11T01:31:08.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.594+0000 7f44e1847640 1 -- 172.21.15.12:0/779986072 wait complete. 2024-09-11T01:31:08.607 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:31:08.607 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:31:08.607 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:31:08.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.706+0000 7f30203ec640 10 monclient: get_monmap_and_config 2024-09-11T01:31:08.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.706+0000 7f30203ec640 10 monclient: build_initial_monmap 2024-09-11T01:31:08.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.706+0000 7f30203ec640 10 monclient: monmap: 2024-09-11T01:31:08.709 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:08.709 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:08.709 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:31:08.708511+0000 2024-09-11T01:31:08.709 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:31:08.708511+0000 2024-09-11T01:31:08.710 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:08.710 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:08.710 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:31:08.710 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:31:08.710 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:31:08.710 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:31:08.710 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:31:08.710 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:08.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.707+0000 7f30203ec640 1 Processor -- start 2024-09-11T01:31:08.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.707+0000 7f30203ec640 1 -- start start 2024-09-11T01:31:08.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.707+0000 7f30203ec640 10 monclient: init 2024-09-11T01:31:08.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f30203ec640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f30203ec640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-11T01:31:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f30203ec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f30180a2e20 addr v1:172.21.15.71:6789/0 2024-09-11T01:31:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f30203ec640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a4970 0x7f301806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f30203ec640 10 monclient(hunting): picked mon.noname-b con 0x7f30180a4970 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f30203ec640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f301806f910 0x7f301813f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f30203ec640 10 monclient(hunting): picked mon.noname-a con 0x7f301806f910 addr v2:172.21.15.12:3300/0 2024-09-11T01:31:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f30203ec640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f301806f4f0 con 0x7f30180a2e20 2024-09-11T01:31:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f30203ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f30203ec640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f301806ef30 con 0x7f301806f910 2024-09-11T01:31:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f30203ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f30203ec640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f301806ec50 con 0x7f30180a4970 2024-09-11T01:31:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f30203ec640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:08.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f30203ec640 10 monclient(hunting): authenticate will time out at 875.151124s 2024-09-11T01:31:08.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301d987640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a4970 0x7f301806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:08.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301e989640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f301806f910 0x7f301813f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:08.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301d987640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a4970 0x7f301806ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55854/0 (socket says 172.21.15.12:55854) 2024-09-11T01:31:08.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301e989640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f301806f910 0x7f301813f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45658/0 (socket says 172.21.15.12:45658) 2024-09-11T01:31:08.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301d987640 1 -- 172.21.15.12:0/905960238 learned_addr learned my addr 172.21.15.12:0/905960238 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:08.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301d987640 10 monclient(hunting): get_auth_request con 0x7f30180a4970 auth_method 0 2024-09-11T01:31:08.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301d987640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:08.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301d987640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:08.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301d987640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:08.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301e989640 10 monclient(hunting): get_auth_request con 0x7f301806f910 auth_method 0 2024-09-11T01:31:08.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301e989640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:08.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301e989640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:08.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301e989640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:08.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301d987640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:08.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301d987640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:08.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301d987640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:08.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301e989640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:08.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301e989640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:08.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301e989640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:08.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301d987640 10 monclient(hunting): handle_auth_done global_id 4529 payload 274 2024-09-11T01:31:08.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301d987640 1 -- 172.21.15.12:0/905960238 >> v2:172.21.15.12:3300/0 conn(0x7f301806f910 msgr2=0x7f301813f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:08.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301d987640 1 --2- 172.21.15.12:0/905960238 >> v2:172.21.15.12:3300/0 conn(0x7f301806f910 0x7f301813f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:08.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301d987640 1 -- 172.21.15.12:0/905960238 >> v1:172.21.15.71:6789/0 conn(0x7f30180a2e20 legacy=0x7f30180a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:08.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301d987640 10 monclient: _finish_hunting 0 2024-09-11T01:31:08.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301d987640 1 monclient: found mon.noname-b 2024-09-11T01:31:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301d987640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301d987640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:31:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301d987640 1 -- 172.21.15.12:0/905960238 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30180a3910 con 0x7f30180a4970 2024-09-11T01:31:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.708+0000 7f301e989640 1 --2- 172.21.15.12:0/905960238 >> v2:172.21.15.12:3300/0 conn(0x7f301806f910 0x7f301813f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:31:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.711+0000 7f301d987640 1 -- 172.21.15.12:0/905960238 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a4970 msgr2=0x7f301806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.711+0000 7f301d987640 1 --2- 172.21.15.12:0/905960238 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a4970 0x7f301806ded0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3008009870 tx=0x7f300802a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58996dbe9f83d7b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.711+0000 7f301d186640 1 -- 172.21.15.12:0/905960238 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3008037070 con 0x7f30180a4970 2024-09-11T01:31:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.711+0000 7f301d186640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.711+0000 7f301d186640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:31:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.711+0000 7f301d186640 10 monclient: dump: 2024-09-11T01:31:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:08.724 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.711+0000 7f301d186640 10 monclient: _finish_auth 0 2024-09-11T01:31:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.711+0000 7f301d186640 10 monclient: _check_auth_tickets 2024-09-11T01:31:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.711+0000 7f301d186640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.711+0000 7f30203ec640 5 monclient: authenticate success, global_id 4529 2024-09-11T01:31:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.711+0000 7f30203ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:31:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.711+0000 7f301d186640 1 -- 172.21.15.12:0/905960238 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3008032440 con 0x7f30180a4970 2024-09-11T01:31:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.711+0000 7f301d186640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.711+0000 7f30203ec640 10 monclient: get_monmap_and_config success 2024-09-11T01:31:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.711+0000 7f30203ec640 4 set_mon_vals no callback set 2024-09-11T01:31:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.711+0000 7f301d186640 1 -- 172.21.15.12:0/905960238 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3008032810 con 0x7f30180a4970 2024-09-11T01:31:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.711+0000 7f301d186640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.711+0000 7f301d186640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.711+0000 7f301d186640 10 monclient: dump: 2024-09-11T01:31:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:08.726 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.712+0000 7f30203ec640 10 monclient: shutdown 2024-09-11T01:31:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.712+0000 7f30203ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.712+0000 7f30203ec640 1 -- 172.21.15.12:0/905960238 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a4970 msgr2=0x7f301806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.712+0000 7f30203ec640 1 --2- 172.21.15.12:0/905960238 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a4970 0x7f301806ded0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3008009870 tx=0x7f300802a0a0 comp rx=0 tx=0).stop 2024-09-11T01:31:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.712+0000 7f30203ec640 1 -- 172.21.15.12:0/905960238 shutdown_connections 2024-09-11T01:31:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.712+0000 7f30203ec640 1 --2- 172.21.15.12:0/905960238 >> v2:172.21.15.12:3300/0 conn(0x7f301806f910 0x7f301813f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.712+0000 7f30203ec640 1 --2- 172.21.15.12:0/905960238 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a4970 0x7f301806ded0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.712+0000 7f30203ec640 1 -- 172.21.15.12:0/905960238 >> 172.21.15.12:0/905960238 conn(0x7f30180a8600 msgr2=0x7f30180a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.712+0000 7f30203ec640 1 -- 172.21.15.12:0/905960238 shutdown_connections 2024-09-11T01:31:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.712+0000 7f30203ec640 1 -- 172.21.15.12:0/905960238 wait complete. 2024-09-11T01:31:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.712+0000 7f30203ec640 10 monclient: build_initial_monmap 2024-09-11T01:31:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.712+0000 7f30203ec640 10 monclient: monmap: 2024-09-11T01:31:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:31:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:31:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:31:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:31:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:31:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:31:08.728 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f30203ec640 1 Processor -- start 2024-09-11T01:31:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f30203ec640 1 -- start start 2024-09-11T01:31:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f30203ec640 10 monclient: init 2024-09-11T01:31:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f30203ec640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f30203ec640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:31:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f30203ec640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301806f910 0x7f30181398c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f30203ec640 10 monclient(hunting): picked mon.noname-a con 0x7f301806f910 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f30203ec640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a2e20 0x7f3018139e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f30203ec640 10 monclient(hunting): picked mon.noname-c con 0x7f30180a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f30203ec640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30180a4970 0x7f30181ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f30203ec640 10 monclient(hunting): picked mon.noname-b con 0x7f30180a4970 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:31:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f30203ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f30203ec640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f301813fd00 con 0x7f301806f910 2024-09-11T01:31:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f30203ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f30203ec640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f301813fb80 con 0x7f30180a4970 2024-09-11T01:31:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f30203ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f30203ec640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f301813fe80 con 0x7f30180a2e20 2024-09-11T01:31:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f30203ec640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f30203ec640 10 monclient(hunting): authenticate will time out at 875.156580s 2024-09-11T01:31:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301e188640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301806f910 0x7f30181398c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301d987640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a2e20 0x7f3018139e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301d987640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a2e20 0x7f3018139e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55866/0 (socket says 172.21.15.12:55866) 2024-09-11T01:31:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301e188640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301806f910 0x7f30181398c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45664/0 (socket says 172.21.15.12:45664) 2024-09-11T01:31:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301d987640 1 -- 172.21.15.12:0/3762074336 learned_addr learned my addr 172.21.15.12:0/3762074336 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301d987640 10 monclient(hunting): get_auth_request con 0x7f30180a2e20 auth_method 0 2024-09-11T01:31:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301d987640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301d987640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301d987640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301e188640 10 monclient(hunting): get_auth_request con 0x7f301806f910 auth_method 0 2024-09-11T01:31:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301e188640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301e188640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301e188640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301e188640 1 -- 172.21.15.12:0/3762074336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301806f910 msgr2=0x7f30181398c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:31:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301e188640 1 -- 172.21.15.12:0/3762074336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301806f910 msgr2=0x7f30181398c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:31:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301e188640 1 --2- 172.21.15.12:0/3762074336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301806f910 0x7f30181398c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:31:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301e188640 1 --2- 172.21.15.12:0/3762074336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301806f910 0x7f30181398c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:31:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301d987640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301d987640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301d987640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301e989640 1 --2- 172.21.15.12:0/3762074336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30180a4970 0x7f30181ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301e989640 10 monclient(hunting): get_auth_request con 0x7f30180a4970 auth_method 0 2024-09-11T01:31:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301e989640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301e989640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301e989640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301e989640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301e989640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301e989640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.713+0000 7f301d987640 10 monclient(hunting): handle_auth_done global_id 4532 payload 824 2024-09-11T01:31:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.714+0000 7f301d987640 1 -- 172.21.15.12:0/3762074336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30180a4970 msgr2=0x7f30181ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.714+0000 7f301d987640 1 --2- 172.21.15.12:0/3762074336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30180a4970 0x7f30181ccfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.714+0000 7f301d987640 1 -- 172.21.15.12:0/3762074336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301806f910 msgr2=0x7f30181398c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:31:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.714+0000 7f301d987640 1 --2- 172.21.15.12:0/3762074336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301806f910 0x7f30181398c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.714+0000 7f301d987640 10 monclient: _finish_hunting 0 2024-09-11T01:31:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.714+0000 7f301d987640 1 monclient: found mon.noname-c 2024-09-11T01:31:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.714+0000 7f301d987640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.714+0000 7f301d987640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:31:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.714+0000 7f301d987640 1 -- 172.21.15.12:0/3762074336 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30180a2310 con 0x7f30180a2e20 2024-09-11T01:31:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.716+0000 7f301d987640 1 -- 172.21.15.12:0/3762074336 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a2e20 msgr2=0x7f3018139e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.716+0000 7f301d987640 1 --2- 172.21.15.12:0/3762074336 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a2e20 0x7f3018139e00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f30080090b0 tx=0x7f30080046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.716+0000 7f300effd640 1 -- 172.21.15.12:0/3762074336 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3008037040 con 0x7f30180a2e20 2024-09-11T01:31:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.716+0000 7f300effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.716+0000 7f300effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:31:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.716+0000 7f300effd640 10 monclient: dump: 2024-09-11T01:31:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:08.733 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.716+0000 7f300effd640 10 monclient: _finish_auth 0 2024-09-11T01:31:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.716+0000 7f300effd640 10 monclient: _check_auth_tickets 2024-09-11T01:31:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.716+0000 7f300effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.716+0000 7f300effd640 1 -- 172.21.15.12:0/3762074336 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3008042070 con 0x7f30180a2e20 2024-09-11T01:31:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.716+0000 7f30203ec640 5 monclient: authenticate success, global_id 4532 2024-09-11T01:31:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.716+0000 7f30203ec640 10 monclient: _renew_subs 2024-09-11T01:31:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.716+0000 7f30203ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.716+0000 7f30203ec640 1 -- 172.21.15.12:0/3762074336 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30180a5d90 con 0x7f30180a2e20 2024-09-11T01:31:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.716+0000 7f300effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.716+0000 7f300effd640 1 -- 172.21.15.12:0/3762074336 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300803b860 con 0x7f30180a2e20 2024-09-11T01:31:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.716+0000 7f300effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.716+0000 7f300effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.716+0000 7f300effd640 10 monclient: dump: 2024-09-11T01:31:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:08.735 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.716+0000 7f301c985640 4 set_mon_vals no callback set 2024-09-11T01:31:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.716+0000 7f30203ec640 10 monclient: _renew_subs 2024-09-11T01:31:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.716+0000 7f30203ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.716+0000 7f30203ec640 1 -- 172.21.15.12:0/3762074336 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f301813b790 con 0x7f30180a2e20 2024-09-11T01:31:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.717+0000 7f30203ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.717+0000 7f30203ec640 1 --2- 172.21.15.12:0/3762074336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301802d440 0x7f301802d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.717+0000 7f30203ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.717+0000 7f30203ec640 1 -- 172.21.15.12:0/3762074336 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f301806a060 con 0x7f301802d440 2024-09-11T01:31:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.717+0000 7f301e188640 1 --2- 172.21.15.12:0/3762074336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301802d440 0x7f301802d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.717+0000 7f301e188640 10 monclient: get_auth_request con 0x7f301802d440 auth_method 0 2024-09-11T01:31:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.717+0000 7f301e188640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.717+0000 7f301e188640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.717+0000 7f301e188640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.717+0000 7f300effd640 1 -- 172.21.15.12:0/3762074336 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f300803a030 con 0x7f30180a2e20 2024-09-11T01:31:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.717+0000 7f301e188640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.717+0000 7f301e188640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.717+0000 7f301e188640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.717+0000 7f301e188640 10 monclient(hunting): handle_auth_done global_id 24105 payload 824 2024-09-11T01:31:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.717+0000 7f301e188640 1 -- 172.21.15.12:0/3762074336 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f301806ded0 con 0x7f301802d440 2024-09-11T01:31:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.717+0000 7f300effd640 1 --2- 172.21.15.12:0/3762074336 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2ffc03e210 0x7f2ffc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.717+0000 7f300effd640 1 -- 172.21.15.12:0/3762074336 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(68..68 src has 1..68) ==== 4783+0+0 (secure 0 0 0) 0x7f3008072f30 con 0x7f30180a2e20 2024-09-11T01:31:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.718+0000 7f301e989640 1 --2- 172.21.15.12:0/3762074336 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2ffc03e210 0x7f2ffc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.718+0000 7f301e989640 10 monclient: get_auth_request con 0x7f2ffc03e210 auth_method 0 2024-09-11T01:31:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.719+0000 7f301e188640 1 -- 172.21.15.12:0/3762074336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301802d440 msgr2=0x7f301802d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.719+0000 7f301e188640 1 --2- 172.21.15.12:0/3762074336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301802d440 0x7f301802d820 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f301400e420 tx=0x7f301400e8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.720+0000 7f300effd640 1 -- 172.21.15.12:0/3762074336 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3014007450 con 0x7f301802d440 2024-09-11T01:31:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.720+0000 7f300effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.720+0000 7f300effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.720+0000 7f300effd640 10 monclient: dump: 2024-09-11T01:31:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:08.738 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:08.738 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.738 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.738 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:08.738 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:08.738 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:08.738 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:08.738 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:08.738 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:08.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.720+0000 7f300effd640 1 -- 172.21.15.12:0/3762074336 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3014007450 con 0x7f301802d440 2024-09-11T01:31:08.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.720+0000 7f300effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:08.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.720+0000 7f300effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:31:08.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.720+0000 7f300effd640 1 -- 172.21.15.12:0/3762074336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301802d440 msgr2=0x7f301802d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:08.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.720+0000 7f300effd640 1 --2- 172.21.15.12:0/3762074336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301802d440 0x7f301802d820 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f301400e420 tx=0x7f301400e8e0 comp rx=0 tx=0).stop 2024-09-11T01:31:08.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.720+0000 7f300effd640 1 -- 172.21.15.12:0/3762074336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301802d440 msgr2=0x7f301802d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:31:08.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.720+0000 7f300effd640 1 --2- 172.21.15.12:0/3762074336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301802d440 0x7f301802d820 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f301400e420 tx=0x7f301400e8e0 comp rx=0 tx=0).stop 2024-09-11T01:31:08.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.720+0000 7f301e989640 1 --2- 172.21.15.12:0/3762074336 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2ffc03e210 0x7f2ffc0406d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3010009fd0 tx=0x7f301000b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:08.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.726+0000 7f30203ec640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:31:08.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.726+0000 7f30203ec640 1 --2- 172.21.15.12:0/3762074336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301802dd60 0x7f301802e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:08.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.726+0000 7f30203ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:08.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.726+0000 7f30203ec640 1 -- 172.21.15.12:0/3762074336 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f30180a66a0 con 0x7f301802dd60 2024-09-11T01:31:08.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.726+0000 7f301e188640 1 --2- 172.21.15.12:0/3762074336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301802dd60 0x7f301802e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:08.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.726+0000 7f301e188640 10 monclient: get_auth_request con 0x7f301802dd60 auth_method 0 2024-09-11T01:31:08.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.726+0000 7f301e188640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:08.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.726+0000 7f301e188640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:08.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.726+0000 7f301e188640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:08.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.726+0000 7f301e188640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:08.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.726+0000 7f301e188640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:08.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.726+0000 7f301e188640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.726+0000 7f301e188640 10 monclient(hunting): handle_auth_done global_id 24108 payload 824 2024-09-11T01:31:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.726+0000 7f301e188640 1 -- 172.21.15.12:0/3762074336 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3018137960 con 0x7f301802dd60 2024-09-11T01:31:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.729+0000 7f301e188640 1 -- 172.21.15.12:0/3762074336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301802dd60 msgr2=0x7f301802e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.729+0000 7f301e188640 1 --2- 172.21.15.12:0/3762074336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301802dd60 0x7f301802e170 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3014018000 tx=0x7f3014003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.729+0000 7f300effd640 1 -- 172.21.15.12:0/3762074336 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f301401eb70 con 0x7f301802dd60 2024-09-11T01:31:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.729+0000 7f300effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.729+0000 7f300effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.729+0000 7f300effd640 10 monclient: dump: 2024-09-11T01:31:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:08.743 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.729+0000 7f300effd640 1 -- 172.21.15.12:0/3762074336 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7f3014006e50 con 0x7f301802dd60 2024-09-11T01:31:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.729+0000 7f300effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:31:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.729+0000 7f300effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:31:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.729+0000 7f300effd640 1 -- 172.21.15.12:0/3762074336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301802dd60 msgr2=0x7f301802e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.729+0000 7f300effd640 1 --2- 172.21.15.12:0/3762074336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301802dd60 0x7f301802e170 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3014018000 tx=0x7f3014003760 comp rx=0 tx=0).stop 2024-09-11T01:31:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.729+0000 7f300effd640 1 -- 172.21.15.12:0/3762074336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301802dd60 msgr2=0x7f301802e170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:31:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.729+0000 7f300effd640 1 --2- 172.21.15.12:0/3762074336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f301802dd60 0x7f301802e170 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3014018000 tx=0x7f3014003760 comp rx=0 tx=0).stop 2024-09-11T01:31:08.744 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:31:08.744 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:31:08.744 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:31:08.744 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:31:08.744 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 22, 2024-09-11T01:31:08.744 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 6676, 2024-09-11T01:31:08.744 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:31:08.744 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:31:08.744 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:31:08.744 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:31:08.744 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:08.744 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:31:08.745 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:31:08.745 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:31:08.745 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:31:08.745 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:08.745 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:08.745 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:08.745 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:08.745 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:08.745 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:08.745 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:08.745 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:08.745 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:08.745 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:08.746 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:08.746 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:08.746 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:31:08.746 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:31:08.746 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:08.746 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:08.746 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:08.746 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:08.746 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:08.746 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:08.746 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:08.746 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:08.746 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:31:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:31:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:31:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:31:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:31:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:31:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:31:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:31:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:31:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:31:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:31:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:31:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:31:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:31:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:31:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:31:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:31:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:31:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:31:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:31:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:31:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:31:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:31:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:31:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:31:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:31:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:31:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:31:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:31:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:31:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:31:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:08.753 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:08.753 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:08.753 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:08.753 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:31:08.753 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:31:08.753 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:08.753 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:08.753 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:08.753 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:08.753 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:31:08.753 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:08.753 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:08.753 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:08.754 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:08.754 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:31:08.754 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:08.754 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:08.754 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:08.754 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:08.754 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:31:08.754 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:31:08.754 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:08.754 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:08.754 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:08.754 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:08.754 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:08.755 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:08.755 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:31:08.755 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:31:08.755 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:08.755 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:08.755 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:08.755 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:31:08.755 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:08.755 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:08.755 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:31:08.755 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:08.755 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:08.755 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:08.756 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:31:08.756 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:08.756 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:08.756 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:08.756 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:31:08.756 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:31:08.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.729+0000 7f30203ec640 1 -- 172.21.15.12:0/3762074336 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2ffc03e210 msgr2=0x7f2ffc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:08.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.729+0000 7f30203ec640 1 --2- 172.21.15.12:0/3762074336 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2ffc03e210 0x7f2ffc0406d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3010009fd0 tx=0x7f301000b040 comp rx=0 tx=0).stop 2024-09-11T01:31:08.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.729+0000 7f30203ec640 10 monclient: shutdown 2024-09-11T01:31:08.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.729+0000 7f30203ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:08.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.729+0000 7f30203ec640 1 -- 172.21.15.12:0/3762074336 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a2e20 msgr2=0x7f3018139e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:08.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.729+0000 7f30203ec640 1 --2- 172.21.15.12:0/3762074336 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a2e20 0x7f3018139e00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f30080090b0 tx=0x7f30080046c0 comp rx=0 tx=0).stop 2024-09-11T01:31:08.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.729+0000 7f301e989640 1 -- 172.21.15.12:0/3762074336 reap_dead start 2024-09-11T01:31:08.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.729+0000 7f301e989640 1 -- 172.21.15.12:0/3762074336 reap_dead start 2024-09-11T01:31:08.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.729+0000 7f30203ec640 1 -- 172.21.15.12:0/3762074336 shutdown_connections 2024-09-11T01:31:08.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.729+0000 7f30203ec640 1 -- 172.21.15.12:0/3762074336 >> 172.21.15.12:0/3762074336 conn(0x7f30180a8600 msgr2=0x7f301809eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:08.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.729+0000 7f30203ec640 1 -- 172.21.15.12:0/3762074336 shutdown_connections 2024-09-11T01:31:08.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.729+0000 7f30203ec640 1 -- 172.21.15.12:0/3762074336 wait complete. 2024-09-11T01:31:08.758 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:31:08.759 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:31:08.759 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:31:08.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.858+0000 7fdb691ec640 10 monclient: get_monmap_and_config 2024-09-11T01:31:08.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.858+0000 7fdb691ec640 10 monclient: build_initial_monmap 2024-09-11T01:31:08.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.858+0000 7fdb691ec640 10 monclient: monmap: 2024-09-11T01:31:08.861 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:08.861 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:08.861 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:31:08.859923+0000 2024-09-11T01:31:08.861 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:31:08.859923+0000 2024-09-11T01:31:08.861 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:08.861 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:08.861 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:31:08.861 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:31:08.861 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:31:08.861 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:31:08.862 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:31:08.862 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:08.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.858+0000 7fdb691ec640 1 Processor -- start 2024-09-11T01:31:08.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.858+0000 7fdb691ec640 1 -- start start 2024-09-11T01:31:08.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.858+0000 7fdb691ec640 10 monclient: init 2024-09-11T01:31:08.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb691ec640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:08.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb691ec640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-11T01:31:08.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb691ec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdb64137d50 addr v1:172.21.15.12:6789/0 2024-09-11T01:31:08.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb691ec640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdb64138130 0x7fdb64138580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:08.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb691ec640 10 monclient(hunting): picked mon.noname-a con 0x7fdb64138130 addr v2:172.21.15.12:3300/0 2024-09-11T01:31:08.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb691ec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdb6413d280 addr v1:172.21.15.71:6789/0 2024-09-11T01:31:08.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb691ec640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb6409d820 con 0x7fdb64137d50 2024-09-11T01:31:08.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb691ec640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb6409ba20 con 0x7fdb6413d280 2024-09-11T01:31:08.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb691ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:08.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb691ec640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fdb6409c650 con 0x7fdb64138130 2024-09-11T01:31:08.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb691ec640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:08.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb691ec640 10 monclient(hunting): authenticate will time out at 875.302503s 2024-09-11T01:31:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb5bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdb64138130 0x7fdb64138580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb5bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdb64138130 0x7fdb64138580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45682/0 (socket says 172.21.15.12:45682) 2024-09-11T01:31:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb62d9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fdb64137d50 0x7fdb6409d110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34822/0 (socket says 172.21.15.12:34822) 2024-09-11T01:31:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb5bfff640 1 -- 172.21.15.12:0/77137610 learned_addr learned my addr 172.21.15.12:0/77137610 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb5bfff640 10 monclient(hunting): get_auth_request con 0x7fdb64138130 auth_method 0 2024-09-11T01:31:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb5bfff640 10 monclient(hunting): handle_auth_done global_id 24111 payload 274 2024-09-11T01:31:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb5bfff640 1 -- 172.21.15.12:0/77137610 >> v1:172.21.15.71:6789/0 conn(0x7fdb6413d280 legacy=0x7fdb6413f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:08.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb5bfff640 1 -- 172.21.15.12:0/77137610 >> v1:172.21.15.12:6789/0 conn(0x7fdb64137d50 legacy=0x7fdb6409d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:08.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb5bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:31:08.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb5bfff640 1 monclient: found mon.noname-a 2024-09-11T01:31:08.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:08.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb5bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:31:08.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.859+0000 7fdb5bfff640 1 -- 172.21.15.12:0/77137610 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb6409c370 con 0x7fdb64138130 2024-09-11T01:31:08.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.862+0000 7fdb5bfff640 1 -- 172.21.15.12:0/77137610 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb64138130 msgr2=0x7fdb64138580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:08.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.862+0000 7fdb5bfff640 1 --2- 172.21.15.12:0/77137610 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb64138130 0x7fdb64138580 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009870 tx=0x7fdb4c02a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=945ff943004dba1b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:08.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.863+0000 7fdb6259c640 1 -- 172.21.15.12:0/77137610 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c037070 con 0x7fdb64138130 2024-09-11T01:31:08.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.863+0000 7fdb6259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:08.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.863+0000 7fdb6259c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:31:08.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.863+0000 7fdb6259c640 10 monclient: dump: 2024-09-11T01:31:08.870 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:08.870 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:08.870 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.870 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.870 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:08.871 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:08.871 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:08.871 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:08.871 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:08.871 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:08.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.863+0000 7fdb6259c640 10 monclient: _finish_auth 0 2024-09-11T01:31:08.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.863+0000 7fdb6259c640 10 monclient: _check_auth_tickets 2024-09-11T01:31:08.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.863+0000 7fdb6259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:08.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.863+0000 7fdb691ec640 5 monclient: authenticate success, global_id 24111 2024-09-11T01:31:08.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.863+0000 7fdb691ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:31:08.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.863+0000 7fdb6259c640 1 -- 172.21.15.12:0/77137610 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb4c032640 con 0x7fdb64138130 2024-09-11T01:31:08.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.863+0000 7fdb6259c640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:08.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.863+0000 7fdb6259c640 1 -- 172.21.15.12:0/77137610 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c032a10 con 0x7fdb64138130 2024-09-11T01:31:08.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.863+0000 7fdb691ec640 10 monclient: get_monmap_and_config success 2024-09-11T01:31:08.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.863+0000 7fdb691ec640 4 set_mon_vals no callback set 2024-09-11T01:31:08.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.863+0000 7fdb6259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:08.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.863+0000 7fdb6259c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:08.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.863+0000 7fdb6259c640 10 monclient: dump: 2024-09-11T01:31:08.872 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:08.872 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:08.872 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.872 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.872 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:08.872 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:08.872 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:08.873 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:08.873 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:08.873 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:08.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.863+0000 7fdb691ec640 10 monclient: shutdown 2024-09-11T01:31:08.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.863+0000 7fdb691ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:08.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.863+0000 7fdb691ec640 1 -- 172.21.15.12:0/77137610 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb64138130 msgr2=0x7fdb64138580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:08.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.863+0000 7fdb691ec640 1 --2- 172.21.15.12:0/77137610 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb64138130 0x7fdb64138580 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009870 tx=0x7fdb4c02a130 comp rx=0 tx=0).stop 2024-09-11T01:31:08.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.863+0000 7fdb691ec640 1 -- 172.21.15.12:0/77137610 shutdown_connections 2024-09-11T01:31:08.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.863+0000 7fdb691ec640 1 --2- 172.21.15.12:0/77137610 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb64138130 0x7fdb64138580 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:08.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.863+0000 7fdb691ec640 1 -- 172.21.15.12:0/77137610 >> 172.21.15.12:0/77137610 conn(0x7fdb640a7020 msgr2=0x7fdb640a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:08.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.863+0000 7fdb691ec640 1 -- 172.21.15.12:0/77137610 shutdown_connections 2024-09-11T01:31:08.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.863+0000 7fdb691ec640 1 -- 172.21.15.12:0/77137610 wait complete. 2024-09-11T01:31:08.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.864+0000 7fdb691ec640 10 monclient: build_initial_monmap 2024-09-11T01:31:08.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.864+0000 7fdb691ec640 10 monclient: monmap: 2024-09-11T01:31:08.874 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:08.874 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:31:08.874 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:31:08.874 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:31:08.874 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:08.874 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:08.874 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:31:08.874 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:31:08.874 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:31:08.875 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:08.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.864+0000 7fdb691ec640 1 Processor -- start 2024-09-11T01:31:08.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.864+0000 7fdb691ec640 1 -- start start 2024-09-11T01:31:08.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.864+0000 7fdb691ec640 10 monclient: init 2024-09-11T01:31:08.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb691ec640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:08.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb691ec640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:31:08.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb691ec640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb64137d50 0x7fdb641ccef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:08.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb691ec640 10 monclient(hunting): picked mon.noname-b con 0x7fdb64137d50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:31:08.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb691ec640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb64138130 0x7fdb64136dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:08.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb691ec640 10 monclient(hunting): picked mon.noname-c con 0x7fdb64138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:08.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb691ec640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb6413d280 0x7fdb64139260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:08.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb691ec640 10 monclient(hunting): picked mon.noname-a con 0x7fdb6413d280 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:08.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb691ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:08.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb691ec640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6406f850 con 0x7fdb6413d280 2024-09-11T01:31:08.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb691ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:08.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb691ec640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6406f6d0 con 0x7fdb64137d50 2024-09-11T01:31:08.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb691ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:08.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb691ec640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fdb64140d90 con 0x7fdb64138130 2024-09-11T01:31:08.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb691ec640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:08.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb691ec640 10 monclient(hunting): authenticate will time out at 875.308342s 2024-09-11T01:31:08.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb5bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb64138130 0x7fdb64136dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:08.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb6359e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb6413d280 0x7fdb64139260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:08.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb5bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb64138130 0x7fdb64136dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55876/0 (socket says 172.21.15.12:55876) 2024-09-11T01:31:08.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb5bfff640 1 -- 172.21.15.12:0/2889968129 learned_addr learned my addr 172.21.15.12:0/2889968129 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:08.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb6359e640 10 monclient(hunting): get_auth_request con 0x7fdb6413d280 auth_method 0 2024-09-11T01:31:08.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb6359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:08.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb6359e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:08.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb6359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:08.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb5bfff640 10 monclient(hunting): get_auth_request con 0x7fdb64138130 auth_method 0 2024-09-11T01:31:08.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:08.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:08.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:08.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb62d9d640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb64137d50 0x7fdb641ccef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:08.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb62d9d640 10 monclient(hunting): get_auth_request con 0x7fdb64137d50 auth_method 0 2024-09-11T01:31:08.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb62d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:08.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb62d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:08.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb62d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:08.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb6359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:08.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb6359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:08.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb6359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:08.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:08.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:08.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:08.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb62d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:08.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb62d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:08.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb62d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:08.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb6359e640 10 monclient(hunting): handle_auth_done global_id 24114 payload 824 2024-09-11T01:31:08.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb6359e640 1 -- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb64138130 msgr2=0x7fdb64136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:08.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb6359e640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb64138130 0x7fdb64136dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:08.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb6359e640 1 -- 172.21.15.12:0/2889968129 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb64137d50 msgr2=0x7fdb641ccef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:08.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb6359e640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb64137d50 0x7fdb641ccef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:08.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb6359e640 10 monclient: _finish_hunting 0 2024-09-11T01:31:08.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb6359e640 1 monclient: found mon.noname-a 2024-09-11T01:31:08.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb6359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:08.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb6359e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:31:08.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb6359e640 1 -- 172.21.15.12:0/2889968129 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb640a1710 con 0x7fdb6413d280 2024-09-11T01:31:08.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.865+0000 7fdb5bfff640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb64138130 0x7fdb64136dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:31:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb6359e640 1 -- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb6413d280 msgr2=0x7fdb64139260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb6359e640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb6413d280 0x7fdb64139260 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c00c750 tx=0x7fdb5c00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb6359e640 1 -- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb6413d280 msgr2=0x7fdb64139260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:31:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb6359e640 1 -- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb6413d280 msgr2=0x7fdb64139260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:31:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb6359e640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb6413d280 0x7fdb64139260 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c00c750 tx=0x7fdb5c00cc10 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:31:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb6359e640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb6413d280 0x7fdb64139260 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c00c750 tx=0x7fdb5c00cc10 comp rx=0 tx=0).stop 2024-09-11T01:31:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5b7fe640 10 monclient: ms_handle_reset current mon [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5b7fe640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5b7fe640 1 -- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb6413d280 msgr2=0x7fdb64139260 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:31:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5b7fe640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb6413d280 0x7fdb64139260 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5b7fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:31:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5b7fe640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb440034a0 0x7fdb440058f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5b7fe640 10 monclient(hunting): picked mon.noname-a con 0x7fdb440034a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5b7fe640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb64138130 0x7fdb44008420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5b7fe640 10 monclient(hunting): picked mon.noname-c con 0x7fdb64138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5b7fe640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb64137d50 0x7fdb4400aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5b7fe640 10 monclient(hunting): picked mon.noname-b con 0x7fdb64137d50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:31:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5b7fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5b7fe640 1 -- 172.21.15.12:0/2889968129 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4400b900 con 0x7fdb440034a0 2024-09-11T01:31:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5b7fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5b7fe640 1 -- 172.21.15.12:0/2889968129 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4400ba80 con 0x7fdb64137d50 2024-09-11T01:31:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5b7fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5b7fe640 1 -- 172.21.15.12:0/2889968129 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fdb4400bc80 con 0x7fdb64138130 2024-09-11T01:31:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5b7fe640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb62d9d640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb440034a0 0x7fdb440058f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb62d9d640 10 monclient(hunting): get_auth_request con 0x7fdb440034a0 auth_method 0 2024-09-11T01:31:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb62d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb62d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb62d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:31:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5bfff640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb64138130 0x7fdb44008420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5bfff640 10 monclient(hunting): get_auth_request con 0x7fdb64138130 auth_method 0 2024-09-11T01:31:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:31:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb62d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb62d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb62d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:31:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb6359e640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb64137d50 0x7fdb4400aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:31:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb6359e640 10 monclient(hunting): get_auth_request con 0x7fdb64137d50 auth_method 0 2024-09-11T01:31:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb6359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb6359e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb6359e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:31:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb6359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb6359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb6359e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:31:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb62d9d640 10 monclient(hunting): handle_auth_done global_id 24114 payload 293 2024-09-11T01:31:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb62d9d640 1 -- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb64138130 msgr2=0x7fdb44008420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb62d9d640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb64138130 0x7fdb44008420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb62d9d640 1 -- 172.21.15.12:0/2889968129 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb64137d50 msgr2=0x7fdb4400aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb62d9d640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb64137d50 0x7fdb4400aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb62d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:31:08.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb62d9d640 1 monclient: found mon.noname-a 2024-09-11T01:31:08.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb62d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:08.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb62d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:31:08.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb62d9d640 1 -- 172.21.15.12:0/2889968129 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb4400bf40 con 0x7fdb440034a0 2024-09-11T01:31:08.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.868+0000 7fdb5bfff640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb64138130 0x7fdb44008420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:31:08.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.870+0000 7fdb62d9d640 1 -- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb440034a0 msgr2=0x7fdb440058f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:08.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.870+0000 7fdb62d9d640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb440034a0 0x7fdb440058f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdb54007fd0 tx=0x7fdb5c003d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:08.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.871+0000 7fdb5b7fe640 1 -- 172.21.15.12:0/2889968129 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb54006b10 con 0x7fdb440034a0 2024-09-11T01:31:08.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.871+0000 7fdb5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:08.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.871+0000 7fdb5b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:31:08.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.871+0000 7fdb5b7fe640 10 monclient: dump: 2024-09-11T01:31:08.890 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:08.891 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.871+0000 7fdb5b7fe640 10 monclient: _finish_auth 0 2024-09-11T01:31:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.871+0000 7fdb5b7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:31:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.871+0000 7fdb5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.871+0000 7fdb5b7fe640 1 -- 172.21.15.12:0/2889968129 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb54018070 con 0x7fdb440034a0 2024-09-11T01:31:08.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.871+0000 7fdb5b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:08.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.871+0000 7fdb5b7fe640 1 -- 172.21.15.12:0/2889968129 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb54005310 con 0x7fdb440034a0 2024-09-11T01:31:08.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.871+0000 7fdb691ec640 5 monclient: authenticate success, global_id 24114 2024-09-11T01:31:08.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.871+0000 7fdb691ec640 10 monclient: _renew_subs 2024-09-11T01:31:08.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.871+0000 7fdb691ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:08.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.871+0000 7fdb691ec640 1 -- 172.21.15.12:0/2889968129 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb6413cb30 con 0x7fdb440034a0 2024-09-11T01:31:08.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.871+0000 7fdb5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:08.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.871+0000 7fdb5b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:08.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.871+0000 7fdb5b7fe640 10 monclient: dump: 2024-09-11T01:31:08.892 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:08.892 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:08.892 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.892 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:08.893 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.871+0000 7fdb61d9b640 4 set_mon_vals no callback set 2024-09-11T01:31:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.871+0000 7fdb691ec640 10 monclient: _renew_subs 2024-09-11T01:31:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.871+0000 7fdb691ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.871+0000 7fdb691ec640 1 -- 172.21.15.12:0/2889968129 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb6413c850 con 0x7fdb440034a0 2024-09-11T01:31:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.872+0000 7fdb691ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.872+0000 7fdb691ec640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb641ccef0 0x7fdb64136dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.872+0000 7fdb691ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:08.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.872+0000 7fdb691ec640 1 -- 172.21.15.12:0/2889968129 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fdb640a94e0 con 0x7fdb641ccef0 2024-09-11T01:31:08.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.872+0000 7fdb5bfff640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb641ccef0 0x7fdb64136dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:08.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.872+0000 7fdb5b7fe640 1 -- 172.21.15.12:0/2889968129 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fdb540054b0 con 0x7fdb440034a0 2024-09-11T01:31:08.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.872+0000 7fdb5bfff640 10 monclient: get_auth_request con 0x7fdb641ccef0 auth_method 0 2024-09-11T01:31:08.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.872+0000 7fdb5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:08.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.872+0000 7fdb5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:08.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.872+0000 7fdb5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:08.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.872+0000 7fdb5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:08.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.872+0000 7fdb5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:08.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.872+0000 7fdb5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:08.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.872+0000 7fdb5bfff640 10 monclient(hunting): handle_auth_done global_id 4538 payload 824 2024-09-11T01:31:08.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.872+0000 7fdb5bfff640 1 -- 172.21.15.12:0/2889968129 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdb641cd2d0 con 0x7fdb641ccef0 2024-09-11T01:31:08.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.872+0000 7fdb5b7fe640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdb6413d280 0x7fdb44048a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.872+0000 7fdb5b7fe640 1 -- 172.21.15.12:0/2889968129 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 4783+0+0 (secure 0 0 0) 0x7fdb54051be0 con 0x7fdb440034a0 2024-09-11T01:31:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.872+0000 7fdb6359e640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdb6413d280 0x7fdb44048a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.872+0000 7fdb6359e640 10 monclient: get_auth_request con 0x7fdb6413d280 auth_method 0 2024-09-11T01:31:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.874+0000 7fdb5bfff640 1 -- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb641ccef0 msgr2=0x7fdb64136dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.874+0000 7fdb5bfff640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb641ccef0 0x7fdb64136dd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c035040 tx=0x7fdb4c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.874+0000 7fdb5b7fe640 1 -- 172.21.15.12:0/2889968129 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c04b070 con 0x7fdb641ccef0 2024-09-11T01:31:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.875+0000 7fdb5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.875+0000 7fdb5b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.875+0000 7fdb5b7fe640 10 monclient: dump: 2024-09-11T01:31:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.896 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:08.896 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:08.896 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:08.896 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:08.896 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:08.896 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:08.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.875+0000 7fdb5b7fe640 1 -- 172.21.15.12:0/2889968129 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdb4c03c830 con 0x7fdb641ccef0 2024-09-11T01:31:08.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.875+0000 7fdb5b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:08.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.875+0000 7fdb5b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:31:08.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.875+0000 7fdb5b7fe640 1 -- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb641ccef0 msgr2=0x7fdb64136dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:08.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.875+0000 7fdb5b7fe640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb641ccef0 0x7fdb64136dd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c035040 tx=0x7fdb4c004730 comp rx=0 tx=0).stop 2024-09-11T01:31:08.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.875+0000 7fdb5b7fe640 1 -- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb641ccef0 msgr2=0x7fdb64136dd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:31:08.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.875+0000 7fdb6359e640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdb6413d280 0x7fdb44048a10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c00c6a0 tx=0x7fdb5c020040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:08.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.875+0000 7fdb5b7fe640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb641ccef0 0x7fdb64136dd0 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c035040 tx=0x7fdb4c004730 comp rx=0 tx=0).stop 2024-09-11T01:31:08.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.881+0000 7fdb691ec640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:31:08.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.881+0000 7fdb691ec640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb64139260 0x7fdb64139640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:08.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.881+0000 7fdb691ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:08.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.881+0000 7fdb691ec640 1 -- 172.21.15.12:0/2889968129 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fdb64138600 con 0x7fdb64139260 2024-09-11T01:31:08.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.881+0000 7fdb5bfff640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb64139260 0x7fdb64139640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:08.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.881+0000 7fdb5bfff640 10 monclient: get_auth_request con 0x7fdb64139260 auth_method 0 2024-09-11T01:31:08.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.881+0000 7fdb5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:08.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.881+0000 7fdb5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:08.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.881+0000 7fdb5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:08.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.881+0000 7fdb5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:08.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.881+0000 7fdb5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:08.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.881+0000 7fdb5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:08.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.881+0000 7fdb5bfff640 10 monclient(hunting): handle_auth_done global_id 4541 payload 824 2024-09-11T01:31:08.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.881+0000 7fdb5bfff640 1 -- 172.21.15.12:0/2889968129 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdb64140900 con 0x7fdb64139260 2024-09-11T01:31:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.884+0000 7fdb5bfff640 1 -- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb64139260 msgr2=0x7fdb64139640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.884+0000 7fdb5bfff640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb64139260 0x7fdb64139640 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c02a170 tx=0x7fdb4c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.884+0000 7fdb5b7fe640 1 -- 172.21.15.12:0/2889968129 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c04b040 con 0x7fdb64139260 2024-09-11T01:31:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.884+0000 7fdb5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.884+0000 7fdb5b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.884+0000 7fdb5b7fe640 10 monclient: dump: 2024-09-11T01:31:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:08.900 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.884+0000 7fdb5b7fe640 1 -- 172.21.15.12:0/2889968129 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7fdb4c003610 con 0x7fdb64139260 2024-09-11T01:31:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.884+0000 7fdb5b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:31:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.884+0000 7fdb5b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:31:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.884+0000 7fdb5b7fe640 1 -- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb64139260 msgr2=0x7fdb64139640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.884+0000 7fdb5b7fe640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb64139260 0x7fdb64139640 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c02a170 tx=0x7fdb4c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:31:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.884+0000 7fdb5b7fe640 1 -- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb64139260 msgr2=0x7fdb64139640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:31:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.884+0000 7fdb5b7fe640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdb64139260 0x7fdb64139640 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c02a170 tx=0x7fdb4c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:31:08.901 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:31:08.901 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:31:08.901 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:31:08.901 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:31:08.901 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 22, 2024-09-11T01:31:08.901 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 131574, 2024-09-11T01:31:08.901 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:31:08.901 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:31:08.901 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:31:08.901 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:31:08.901 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:08.901 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:31:08.902 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:31:08.902 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:31:08.902 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:31:08.902 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:08.902 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:08.902 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:08.902 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:08.902 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:08.902 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:08.902 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:08.902 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:08.902 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:08.902 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:08.902 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:08.903 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:08.903 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:31:08.903 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:31:08.903 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:08.903 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:08.903 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:08.903 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:08.903 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:08.903 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:08.903 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:08.903 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:08.903 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:08.903 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:08.904 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:08.904 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:08.904 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:08.904 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:31:08.904 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:31:08.904 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:31:08.904 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:31:08.904 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:31:08.904 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:31:08.904 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:31:08.904 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:31:08.904 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:31:08.904 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:31:08.905 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:31:08.905 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:31:08.905 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:31:08.905 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:31:08.905 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:31:08.905 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:31:08.905 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:31:08.905 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:08.905 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:08.905 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:08.905 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:08.905 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:08.905 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:08.905 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:08.906 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:08.906 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:08.906 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:08.906 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:08.906 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:08.906 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:31:08.906 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:08.906 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:31:08.906 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:08.906 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:31:08.906 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:31:08.906 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:08.906 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:08.907 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:08.907 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:08.907 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:31:08.907 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:08.907 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:08.907 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:08.907 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:08.907 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:31:08.907 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:08.907 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:08.907 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:08.907 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:08.907 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:31:08.907 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:31:08.908 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:08.908 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:08.908 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:08.908 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:08.908 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:08.908 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:31:08.908 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:31:08.908 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:08.908 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:08.908 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:08.908 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:08.908 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:31:08.908 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:08.908 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:08.909 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:08.909 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:08.909 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:31:08.909 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:08.909 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:08.909 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:08.909 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:08.909 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:31:08.909 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:31:08.909 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:08.909 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:08.909 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:08.909 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:08.910 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:08.910 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:31:08.910 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:31:08.910 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:08.910 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:08.910 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:08.910 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:08.910 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:31:08.910 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:08.910 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:08.910 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:08.910 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:08.910 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:31:08.910 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:08.911 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:08.911 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:08.911 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:08.911 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:31:08.911 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:31:08.911 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:08.911 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:08.911 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:08.911 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:08.911 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:08.911 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:08.911 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:31:08.911 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:31:08.911 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:08.912 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:08.912 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:08.912 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:31:08.912 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:08.912 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:08.912 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:31:08.912 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:08.912 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:08.912 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:08.912 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:31:08.912 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:08.912 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:08.912 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:31:08.913 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:08.913 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:08.913 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:08.913 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:31:08.913 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:08.913 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:08.913 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:08.913 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:31:08.913 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:31:08.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.884+0000 7fdb691ec640 1 -- 172.21.15.12:0/2889968129 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdb6413d280 msgr2=0x7fdb44048a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:08.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.884+0000 7fdb691ec640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdb6413d280 0x7fdb44048a10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c00c6a0 tx=0x7fdb5c020040 comp rx=0 tx=0).stop 2024-09-11T01:31:08.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.884+0000 7fdb691ec640 10 monclient: shutdown 2024-09-11T01:31:08.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.884+0000 7fdb691ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:08.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.884+0000 7fdb691ec640 1 -- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb440034a0 msgr2=0x7fdb440058f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:08.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.884+0000 7fdb691ec640 1 --2- 172.21.15.12:0/2889968129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb440034a0 0x7fdb440058f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdb54007fd0 tx=0x7fdb5c003d10 comp rx=0 tx=0).stop 2024-09-11T01:31:08.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.884+0000 7fdb6359e640 1 -- 172.21.15.12:0/2889968129 reap_dead start 2024-09-11T01:31:08.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.884+0000 7fdb6359e640 1 -- 172.21.15.12:0/2889968129 reap_dead start 2024-09-11T01:31:08.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.884+0000 7fdb691ec640 1 -- 172.21.15.12:0/2889968129 shutdown_connections 2024-09-11T01:31:08.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.884+0000 7fdb691ec640 1 -- 172.21.15.12:0/2889968129 >> 172.21.15.12:0/2889968129 conn(0x7fdb640a7020 msgr2=0x7fdb640a8320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:08.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.884+0000 7fdb691ec640 1 -- 172.21.15.12:0/2889968129 shutdown_connections 2024-09-11T01:31:08.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:08.884+0000 7fdb691ec640 1 -- 172.21.15.12:0/2889968129 wait complete. 2024-09-11T01:31:08.916 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:31:08.916 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:31:08.916 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:31:09.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.016+0000 7f1eac1ec640 10 monclient: get_monmap_and_config 2024-09-11T01:31:09.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.016+0000 7f1eac1ec640 10 monclient: build_initial_monmap 2024-09-11T01:31:09.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.016+0000 7f1eac1ec640 10 monclient: monmap: 2024-09-11T01:31:09.019 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:09.019 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:09.019 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:31:09.018139+0000 2024-09-11T01:31:09.020 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:31:09.018139+0000 2024-09-11T01:31:09.020 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:09.020 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:09.020 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:31:09.020 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:31:09.020 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:31:09.020 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:31:09.020 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:31:09.021 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:09.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.017+0000 7f1eac1ec640 1 Processor -- start 2024-09-11T01:31:09.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.017+0000 7f1eac1ec640 1 -- start start 2024-09-11T01:31:09.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.017+0000 7f1eac1ec640 10 monclient: init 2024-09-11T01:31:09.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.017+0000 7f1eac1ec640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:09.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.017+0000 7f1eac1ec640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-11T01:31:09.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.017+0000 7f1eac1ec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1ea41381c0 addr v1:172.21.15.71:6789/0 2024-09-11T01:31:09.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.017+0000 7f1eac1ec640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1ea40a6aa0 0x7f1ea409cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:09.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.017+0000 7f1eac1ec640 10 monclient(hunting): picked mon.noname-a con 0x7f1ea40a6aa0 addr v2:172.21.15.12:3300/0 2024-09-11T01:31:09.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.017+0000 7f1eac1ec640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ea413d000 0x7f1ea413f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:09.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.017+0000 7f1eac1ec640 10 monclient(hunting): picked mon.noname-b con 0x7f1ea413d000 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:09.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.017+0000 7f1eac1ec640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ea409c310 con 0x7f1ea41381c0 2024-09-11T01:31:09.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.017+0000 7f1eac1ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:09.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.017+0000 7f1eac1ec640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f1ea406fc90 con 0x7f1ea40a6aa0 2024-09-11T01:31:09.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.017+0000 7f1eac1ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:09.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.017+0000 7f1eac1ec640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1ea409c5f0 con 0x7f1ea413d000 2024-09-11T01:31:09.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.017+0000 7f1eac1ec640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:09.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.017+0000 7f1eac1ec640 10 monclient(hunting): authenticate will time out at 875.460868s 2024-09-11T01:31:09.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.017+0000 7f1ea9787640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1ea40a6aa0 0x7f1ea409cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:09.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.017+0000 7f1eaa789640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ea413d000 0x7f1ea413f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:09.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1eaa789640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ea413d000 0x7f1ea413f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55928/0 (socket says 172.21.15.12:55928) 2024-09-11T01:31:09.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1ea9787640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1ea40a6aa0 0x7f1ea409cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45704/0 (socket says 172.21.15.12:45704) 2024-09-11T01:31:09.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1eaa789640 1 -- 172.21.15.12:0/3902119926 learned_addr learned my addr 172.21.15.12:0/3902119926 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:09.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1eaa789640 10 monclient(hunting): get_auth_request con 0x7f1ea413d000 auth_method 0 2024-09-11T01:31:09.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1eaa789640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:09.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1eaa789640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:09.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1eaa789640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:09.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1ea9787640 10 monclient(hunting): get_auth_request con 0x7f1ea40a6aa0 auth_method 0 2024-09-11T01:31:09.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1ea9787640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:09.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1ea9787640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:09.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1ea9787640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:09.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1ea9787640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:09.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1ea9787640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:09.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1ea9787640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:09.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1eaa789640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:09.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1eaa789640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:09.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1eaa789640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:09.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1ea9787640 10 monclient(hunting): handle_auth_done global_id 24117 payload 274 2024-09-11T01:31:09.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1ea9787640 1 -- 172.21.15.12:0/3902119926 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ea413d000 msgr2=0x7f1ea413f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:09.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1ea9787640 1 --2- 172.21.15.12:0/3902119926 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ea413d000 0x7f1ea413f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:09.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1ea9787640 1 -- 172.21.15.12:0/3902119926 >> v1:172.21.15.71:6789/0 conn(0x7f1ea41381c0 legacy=0x7f1ea41385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:09.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1ea9787640 10 monclient: _finish_hunting 0 2024-09-11T01:31:09.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1ea9787640 1 monclient: found mon.noname-a 2024-09-11T01:31:09.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1ea9787640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:09.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1ea9787640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:31:09.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1ea9787640 1 -- 172.21.15.12:0/3902119926 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea409c8d0 con 0x7f1ea40a6aa0 2024-09-11T01:31:09.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.018+0000 7f1eaa789640 1 --2- 172.21.15.12:0/3902119926 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ea413d000 0x7f1ea413f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:31:09.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1ea9787640 1 -- 172.21.15.12:0/3902119926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ea40a6aa0 msgr2=0x7f1ea409cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:09.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1ea9787640 1 --2- 172.21.15.12:0/3902119926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ea40a6aa0 0x7f1ea409cfd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1e940097d0 tx=0x7f1e94029fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab0a1e631e7c2e40 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:09.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1ea8f86640 1 -- 172.21.15.12:0/3902119926 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e94037070 con 0x7f1ea40a6aa0 2024-09-11T01:31:09.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1ea8f86640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:09.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1ea8f86640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:31:09.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1ea8f86640 10 monclient: dump: 2024-09-11T01:31:09.040 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:09.040 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:09.040 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:09.040 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:09.040 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:09.040 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:09.041 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:09.041 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:09.041 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:09.041 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:09.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1ea8f86640 10 monclient: _finish_auth 0 2024-09-11T01:31:09.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1ea8f86640 10 monclient: _check_auth_tickets 2024-09-11T01:31:09.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1ea8f86640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:09.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1eac1ec640 5 monclient: authenticate success, global_id 24117 2024-09-11T01:31:09.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1eac1ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:31:09.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1ea8f86640 1 -- 172.21.15.12:0/3902119926 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e94032440 con 0x7f1ea40a6aa0 2024-09-11T01:31:09.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1ea8f86640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:09.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1eac1ec640 10 monclient: get_monmap_and_config success 2024-09-11T01:31:09.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1eac1ec640 4 set_mon_vals no callback set 2024-09-11T01:31:09.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1ea8f86640 1 -- 172.21.15.12:0/3902119926 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e94032810 con 0x7f1ea40a6aa0 2024-09-11T01:31:09.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1ea8f86640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:09.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1ea8f86640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:09.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1ea8f86640 10 monclient: dump: 2024-09-11T01:31:09.042 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:09.042 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:09.042 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:09.042 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:09.042 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:09.042 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:09.042 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:09.042 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:09.043 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:09.043 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:09.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1eac1ec640 10 monclient: shutdown 2024-09-11T01:31:09.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1eac1ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:09.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1eac1ec640 1 -- 172.21.15.12:0/3902119926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ea40a6aa0 msgr2=0x7f1ea409cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:09.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1eac1ec640 1 --2- 172.21.15.12:0/3902119926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ea40a6aa0 0x7f1ea409cfd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1e940097d0 tx=0x7f1e94029fe0 comp rx=0 tx=0).stop 2024-09-11T01:31:09.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1eac1ec640 1 -- 172.21.15.12:0/3902119926 shutdown_connections 2024-09-11T01:31:09.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1eac1ec640 1 --2- 172.21.15.12:0/3902119926 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ea413d000 0x7f1ea413f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:09.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1eac1ec640 1 --2- 172.21.15.12:0/3902119926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ea40a6aa0 0x7f1ea409cfd0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1eac1ec640 1 -- 172.21.15.12:0/3902119926 >> 172.21.15.12:0/3902119926 conn(0x7f1ea40a7440 msgr2=0x7f1ea40a7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1eac1ec640 1 -- 172.21.15.12:0/3902119926 shutdown_connections 2024-09-11T01:31:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.021+0000 7f1eac1ec640 1 -- 172.21.15.12:0/3902119926 wait complete. 2024-09-11T01:31:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.022+0000 7f1eac1ec640 10 monclient: build_initial_monmap 2024-09-11T01:31:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.022+0000 7f1eac1ec640 10 monclient: monmap: 2024-09-11T01:31:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:31:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:31:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:31:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:31:09.044 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:31:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:31:09.045 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.022+0000 7f1eac1ec640 1 Processor -- start 2024-09-11T01:31:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.022+0000 7f1eac1ec640 1 -- start start 2024-09-11T01:31:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.022+0000 7f1eac1ec640 10 monclient: init 2024-09-11T01:31:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1eac1ec640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1eac1ec640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:31:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1eac1ec640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ea40a6aa0 0x7f1ea413b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1eac1ec640 10 monclient(hunting): picked mon.noname-a con 0x7f1ea40a6aa0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1eac1ec640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ea41381c0 0x7f1ea41396e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1eac1ec640 10 monclient(hunting): picked mon.noname-c con 0x7f1ea41381c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1eac1ec640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea413d000 0x7f1ea4136170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:09.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1eac1ec640 10 monclient(hunting): picked mon.noname-b con 0x7f1ea413d000 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:31:09.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1eac1ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:09.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1eac1ec640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea413fab0 con 0x7f1ea40a6aa0 2024-09-11T01:31:09.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1eac1ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:09.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1eac1ec640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea413f930 con 0x7f1ea413d000 2024-09-11T01:31:09.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1eac1ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:09.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1eac1ec640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1ea413fc30 con 0x7f1ea41381c0 2024-09-11T01:31:09.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1eac1ec640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:09.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1eac1ec640 10 monclient(hunting): authenticate will time out at 875.466415s 2024-09-11T01:31:09.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9f88640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ea40a6aa0 0x7f1ea413b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:09.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9787640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ea41381c0 0x7f1ea41396e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:09.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9f88640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ea40a6aa0 0x7f1ea413b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45710/0 (socket says 172.21.15.12:45710) 2024-09-11T01:31:09.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9f88640 1 -- 172.21.15.12:0/3424827076 learned_addr learned my addr 172.21.15.12:0/3424827076 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:09.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9f88640 10 monclient(hunting): get_auth_request con 0x7f1ea40a6aa0 auth_method 0 2024-09-11T01:31:09.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9f88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:09.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9f88640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:09.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9f88640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:09.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9787640 10 monclient(hunting): get_auth_request con 0x7f1ea41381c0 auth_method 0 2024-09-11T01:31:09.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9787640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:09.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9787640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:09.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9787640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:09.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9f88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:09.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9f88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:09.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1eaa789640 1 --2- 172.21.15.12:0/3424827076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea413d000 0x7f1ea4136170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9f88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9787640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9787640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9787640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1eaa789640 10 monclient(hunting): get_auth_request con 0x7f1ea413d000 auth_method 0 2024-09-11T01:31:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1eaa789640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1eaa789640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1eaa789640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1eaa789640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1eaa789640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1eaa789640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9f88640 10 monclient(hunting): handle_auth_done global_id 24120 payload 824 2024-09-11T01:31:09.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9f88640 1 -- 172.21.15.12:0/3424827076 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ea41381c0 msgr2=0x7f1ea41396e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:09.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9f88640 1 --2- 172.21.15.12:0/3424827076 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ea41381c0 0x7f1ea41396e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:09.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9f88640 1 -- 172.21.15.12:0/3424827076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea413d000 msgr2=0x7f1ea4136170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:09.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9f88640 1 --2- 172.21.15.12:0/3424827076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea413d000 0x7f1ea4136170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:09.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9f88640 10 monclient: _finish_hunting 0 2024-09-11T01:31:09.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9f88640 1 monclient: found mon.noname-a 2024-09-11T01:31:09.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9f88640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:09.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9f88640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:31:09.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9f88640 1 -- 172.21.15.12:0/3424827076 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea413abf0 con 0x7f1ea40a6aa0 2024-09-11T01:31:09.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.023+0000 7f1ea9787640 1 --2- 172.21.15.12:0/3424827076 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ea41381c0 0x7f1ea41396e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:31:09.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.026+0000 7f1ea9f88640 1 -- 172.21.15.12:0/3424827076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ea40a6aa0 msgr2=0x7f1ea413b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:09.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.026+0000 7f1ea9f88640 1 --2- 172.21.15.12:0/3424827076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ea40a6aa0 0x7f1ea413b750 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1ea000c430 tx=0x7f1ea000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:09.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.026+0000 7f1e9a7fc640 1 -- 172.21.15.12:0/3424827076 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea00119a0 con 0x7f1ea40a6aa0 2024-09-11T01:31:09.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.026+0000 7f1e9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:09.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.026+0000 7f1e9a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:31:09.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.026+0000 7f1e9a7fc640 10 monclient: dump: 2024-09-11T01:31:09.050 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:09.050 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:09.050 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:09.050 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:09.050 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:09.050 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:09.050 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:09.050 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:09.050 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:09.050 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:09.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.026+0000 7f1e9a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:31:09.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.026+0000 7f1e9a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:31:09.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.026+0000 7f1e9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:09.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.026+0000 7f1e9a7fc640 1 -- 172.21.15.12:0/3424827076 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ea0017070 con 0x7f1ea40a6aa0 2024-09-11T01:31:09.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.026+0000 7f1eac1ec640 5 monclient: authenticate success, global_id 24120 2024-09-11T01:31:09.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.026+0000 7f1e9a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:09.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.026+0000 7f1e9a7fc640 1 -- 172.21.15.12:0/3424827076 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea0011d80 con 0x7f1ea40a6aa0 2024-09-11T01:31:09.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.026+0000 7f1eac1ec640 10 monclient: _renew_subs 2024-09-11T01:31:09.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.026+0000 7f1eac1ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:09.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.026+0000 7f1eac1ec640 1 -- 172.21.15.12:0/3424827076 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ea40a1710 con 0x7f1ea40a6aa0 2024-09-11T01:31:09.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.026+0000 7f1e9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:09.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.026+0000 7f1e9a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:09.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.026+0000 7f1e9a7fc640 10 monclient: dump: 2024-09-11T01:31:09.052 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:09.052 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:09.052 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:09.052 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:09.052 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:09.052 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:09.052 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:09.052 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:09.052 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:09.052 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:09.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.026+0000 7f1e9bfff640 4 set_mon_vals no callback set 2024-09-11T01:31:09.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.026+0000 7f1eac1ec640 10 monclient: _renew_subs 2024-09-11T01:31:09.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.026+0000 7f1eac1ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:09.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.026+0000 7f1eac1ec640 1 -- 172.21.15.12:0/3424827076 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ea4136f10 con 0x7f1ea40a6aa0 2024-09-11T01:31:09.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.027+0000 7f1eac1ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:09.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.027+0000 7f1eac1ec640 1 --2- 172.21.15.12:0/3424827076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea41366b0 0x7f1ea41cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:09.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.027+0000 7f1eac1ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:09.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.027+0000 7f1eac1ec640 1 -- 172.21.15.12:0/3424827076 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea413c890 con 0x7f1ea41366b0 2024-09-11T01:31:09.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.027+0000 7f1e9a7fc640 1 -- 172.21.15.12:0/3424827076 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f1ea001c460 con 0x7f1ea40a6aa0 2024-09-11T01:31:09.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.027+0000 7f1ea9787640 1 --2- 172.21.15.12:0/3424827076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea41366b0 0x7f1ea41cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:09.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.027+0000 7f1ea9787640 10 monclient: get_auth_request con 0x7f1ea41366b0 auth_method 0 2024-09-11T01:31:09.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.027+0000 7f1ea9787640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:09.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.027+0000 7f1ea9787640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:09.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.027+0000 7f1ea9787640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:09.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.027+0000 7f1ea9787640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:09.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.027+0000 7f1ea9787640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:09.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.027+0000 7f1ea9787640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:09.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.027+0000 7f1e9a7fc640 1 --2- 172.21.15.12:0/3424827076 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1e8803e210 0x7f1e880406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:09.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.027+0000 7f1e9a7fc640 1 -- 172.21.15.12:0/3424827076 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 4783+0+0 (secure 0 0 0) 0x7f1ea0052bd0 con 0x7f1ea40a6aa0 2024-09-11T01:31:09.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.027+0000 7f1ea9787640 10 monclient(hunting): handle_auth_done global_id 4444 payload 824 2024-09-11T01:31:09.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.027+0000 7f1ea9787640 1 -- 172.21.15.12:0/3424827076 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1ea409cfd0 con 0x7f1ea41366b0 2024-09-11T01:31:09.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.027+0000 7f1eaa789640 1 --2- 172.21.15.12:0/3424827076 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1e8803e210 0x7f1e880406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:09.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.027+0000 7f1eaa789640 10 monclient: get_auth_request con 0x7f1e8803e210 auth_method 0 2024-09-11T01:31:09.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.030+0000 7f1ea9787640 1 -- 172.21.15.12:0/3424827076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea41366b0 msgr2=0x7f1ea41cceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:09.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.030+0000 7f1ea9787640 1 --2- 172.21.15.12:0/3424827076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea41366b0 0x7f1ea41cceb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1e94034000 tx=0x7f1e940046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:09.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.030+0000 7f1e9a7fc640 1 -- 172.21.15.12:0/3424827076 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e94043070 con 0x7f1ea41366b0 2024-09-11T01:31:09.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.030+0000 7f1e9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:09.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.030+0000 7f1e9a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:31:09.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.030+0000 7f1e9a7fc640 10 monclient: dump: 2024-09-11T01:31:09.055 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:09.055 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:09.055 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:09.055 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:09.055 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:09.055 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:09.055 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:09.055 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:09.055 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:09.055 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:09.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.030+0000 7f1eaa789640 1 --2- 172.21.15.12:0/3424827076 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1e8803e210 0x7f1e880406d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4069fe0 tx=0x7f1e9c00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:09.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.030+0000 7f1e9a7fc640 1 -- 172.21.15.12:0/3424827076 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1e9403ca60 con 0x7f1ea41366b0 2024-09-11T01:31:09.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.030+0000 7f1e9a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:09.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.030+0000 7f1e9a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:31:09.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.030+0000 7f1e9a7fc640 1 -- 172.21.15.12:0/3424827076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea41366b0 msgr2=0x7f1ea41cceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:09.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.030+0000 7f1e9a7fc640 1 --2- 172.21.15.12:0/3424827076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea41366b0 0x7f1ea41cceb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1e94034000 tx=0x7f1e940046c0 comp rx=0 tx=0).stop 2024-09-11T01:31:09.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.030+0000 7f1e9a7fc640 1 -- 172.21.15.12:0/3424827076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea41366b0 msgr2=0x7f1ea41cceb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:31:09.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.030+0000 7f1e9a7fc640 1 --2- 172.21.15.12:0/3424827076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea41366b0 0x7f1ea41cceb0 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1e94034000 tx=0x7f1e940046c0 comp rx=0 tx=0).stop 2024-09-11T01:31:09.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.036+0000 7f1eac1ec640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:31:09.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.036+0000 7f1eac1ec640 1 --2- 172.21.15.12:0/3424827076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea41cd3f0 0x7f1ea41cd800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:09.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.036+0000 7f1eac1ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:09.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.036+0000 7f1eac1ec640 1 -- 172.21.15.12:0/3424827076 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea41cdd40 con 0x7f1ea41cd3f0 2024-09-11T01:31:09.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.036+0000 7f1ea9787640 1 --2- 172.21.15.12:0/3424827076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea41cd3f0 0x7f1ea41cd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:09.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.036+0000 7f1ea9787640 10 monclient: get_auth_request con 0x7f1ea41cd3f0 auth_method 0 2024-09-11T01:31:09.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.036+0000 7f1ea9787640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:09.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.036+0000 7f1ea9787640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:09.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.036+0000 7f1ea9787640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:09.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.036+0000 7f1ea9787640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:09.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.036+0000 7f1ea9787640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:09.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.036+0000 7f1ea9787640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:09.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.037+0000 7f1ea9787640 10 monclient(hunting): handle_auth_done global_id 4447 payload 824 2024-09-11T01:31:09.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.037+0000 7f1ea9787640 1 -- 172.21.15.12:0/3424827076 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1ea4069840 con 0x7f1ea41cd3f0 2024-09-11T01:31:09.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.039+0000 7f1ea9787640 1 -- 172.21.15.12:0/3424827076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea41cd3f0 msgr2=0x7f1ea41cd800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:09.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.039+0000 7f1ea9787640 1 --2- 172.21.15.12:0/3424827076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea41cd3f0 0x7f1ea41cd800 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1e9402a020 tx=0x7f1e94009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:09.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.039+0000 7f1e9a7fc640 1 -- 172.21.15.12:0/3424827076 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e94037040 con 0x7f1ea41cd3f0 2024-09-11T01:31:09.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.039+0000 7f1e9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:09.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.039+0000 7f1e9a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:31:09.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.039+0000 7f1e9a7fc640 10 monclient: dump: 2024-09-11T01:31:09.059 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:09.059 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:09.059 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:09.059 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:09.060 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:09.060 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:09.060 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:09.060 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:09.060 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:09.060 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:09.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.039+0000 7f1e9a7fc640 1 -- 172.21.15.12:0/3424827076 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7f1e9403ce00 con 0x7f1ea41cd3f0 2024-09-11T01:31:09.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.039+0000 7f1e9a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:31:09.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.039+0000 7f1e9a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:31:09.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.039+0000 7f1e9a7fc640 1 -- 172.21.15.12:0/3424827076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea41cd3f0 msgr2=0x7f1ea41cd800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:09.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.039+0000 7f1e9a7fc640 1 --2- 172.21.15.12:0/3424827076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea41cd3f0 0x7f1ea41cd800 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1e9402a020 tx=0x7f1e94009190 comp rx=0 tx=0).stop 2024-09-11T01:31:09.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.039+0000 7f1e9a7fc640 1 -- 172.21.15.12:0/3424827076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea41cd3f0 msgr2=0x7f1ea41cd800 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:31:09.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.039+0000 7f1e9a7fc640 1 --2- 172.21.15.12:0/3424827076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea41cd3f0 0x7f1ea41cd800 secure :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1e9402a020 tx=0x7f1e94009190 comp rx=0 tx=0).stop 2024-09-11T01:31:09.061 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:31:09.061 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:31:09.061 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:31:09.061 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:31:09.061 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 22, 2024-09-11T01:31:09.061 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 131694, 2024-09-11T01:31:09.061 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:31:09.061 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:31:09.061 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:31:09.061 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:31:09.062 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:09.062 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:31:09.062 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:31:09.062 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:31:09.062 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:31:09.062 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:09.062 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:09.062 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:09.062 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:09.062 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:09.062 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:09.062 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:09.062 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:09.063 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:09.063 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:09.063 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:09.063 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:09.063 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:31:09.063 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:31:09.063 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:09.063 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:09.063 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:09.063 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:09.063 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:09.063 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:09.063 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:09.064 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:09.064 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:09.064 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:09.064 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:09.064 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:09.064 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:09.064 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:31:09.064 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:31:09.064 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:31:09.064 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:31:09.064 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:31:09.064 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:31:09.065 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:31:09.065 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:31:09.065 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:31:09.065 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:31:09.065 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:31:09.065 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:31:09.065 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:31:09.065 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:31:09.065 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:31:09.065 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:31:09.065 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:31:09.065 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:09.065 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:09.066 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:09.066 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:09.066 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:09.066 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:09.066 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:09.066 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:09.066 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:09.066 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:09.066 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:09.066 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:09.066 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:31:09.067 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:09.067 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:31:09.067 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:09.067 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:31:09.067 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:31:09.067 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:09.067 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:09.067 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:09.067 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:09.067 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:31:09.067 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:09.067 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:09.068 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:09.068 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:09.068 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:31:09.068 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:09.068 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:09.068 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:09.068 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:09.068 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:31:09.068 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:31:09.068 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:09.068 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:09.068 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:09.069 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:09.069 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:09.069 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:31:09.069 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:31:09.069 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:09.069 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:09.069 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:09.069 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:09.069 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:31:09.069 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:09.069 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:09.069 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:09.070 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:09.070 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:31:09.070 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:09.070 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:09.070 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:09.070 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:09.070 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:31:09.070 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:31:09.070 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:09.070 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:09.070 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:09.070 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:09.070 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:09.071 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:31:09.071 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:31:09.071 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:09.071 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:09.071 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:09.071 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:09.071 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:31:09.071 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:09.071 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:09.071 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:09.071 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:09.071 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:31:09.071 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:09.072 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:09.072 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:09.072 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:09.072 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:31:09.072 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:31:09.072 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:09.072 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:09.072 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:09.072 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:09.072 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:09.072 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:09.072 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:31:09.072 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:31:09.072 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:09.073 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:09.073 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:09.073 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:31:09.073 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:09.073 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:09.073 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:31:09.073 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:09.073 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:09.073 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:09.073 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:31:09.073 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:09.073 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:09.073 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:31:09.073 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:09.074 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:09.074 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:09.074 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:31:09.074 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:09.074 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:09.074 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:31:09.074 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:09.074 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:09.074 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:09.074 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:31:09.074 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:09.074 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:09.074 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:09.075 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:31:09.075 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:31:09.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.040+0000 7f1eac1ec640 1 -- 172.21.15.12:0/3424827076 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1e8803e210 msgr2=0x7f1e880406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:09.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.040+0000 7f1eac1ec640 1 --2- 172.21.15.12:0/3424827076 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1e8803e210 0x7f1e880406d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4069fe0 tx=0x7f1e9c00b040 comp rx=0 tx=0).stop 2024-09-11T01:31:09.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.040+0000 7f1eac1ec640 10 monclient: shutdown 2024-09-11T01:31:09.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.040+0000 7f1eac1ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:09.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.040+0000 7f1eac1ec640 1 -- 172.21.15.12:0/3424827076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ea40a6aa0 msgr2=0x7f1ea413b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:09.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.040+0000 7f1eac1ec640 1 --2- 172.21.15.12:0/3424827076 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ea40a6aa0 0x7f1ea413b750 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1ea000c430 tx=0x7f1ea000c8f0 comp rx=0 tx=0).stop 2024-09-11T01:31:09.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.040+0000 7f1eaa789640 1 -- 172.21.15.12:0/3424827076 reap_dead start 2024-09-11T01:31:09.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.040+0000 7f1eaa789640 1 -- 172.21.15.12:0/3424827076 reap_dead start 2024-09-11T01:31:09.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.040+0000 7f1eac1ec640 1 -- 172.21.15.12:0/3424827076 shutdown_connections 2024-09-11T01:31:09.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.040+0000 7f1eac1ec640 1 -- 172.21.15.12:0/3424827076 >> 172.21.15.12:0/3424827076 conn(0x7f1ea40a7440 msgr2=0x7f1ea4137420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:09.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.040+0000 7f1eac1ec640 1 -- 172.21.15.12:0/3424827076 shutdown_connections 2024-09-11T01:31:09.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.040+0000 7f1eac1ec640 1 -- 172.21.15.12:0/3424827076 wait complete. 2024-09-11T01:31:09.077 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:31:09.077 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:31:09.077 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:31:09.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.175+0000 7f1ea638a640 10 monclient: get_monmap_and_config 2024-09-11T01:31:09.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.175+0000 7f1ea638a640 10 monclient: build_initial_monmap 2024-09-11T01:31:09.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.176+0000 7f1ea638a640 10 monclient: monmap: 2024-09-11T01:31:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:31:09.177721+0000 2024-09-11T01:31:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:31:09.177721+0000 2024-09-11T01:31:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:31:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:31:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:31:09.180 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:31:09.180 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:31:09.180 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:09.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.176+0000 7f1ea638a640 1 Processor -- start 2024-09-11T01:31:09.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.176+0000 7f1ea638a640 1 -- start start 2024-09-11T01:31:09.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.176+0000 7f1ea638a640 10 monclient: init 2024-09-11T01:31:09.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea638a640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:09.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea638a640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-11T01:31:09.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea638a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1ea0074180 addr v1:172.21.15.12:6789/0 2024-09-11T01:31:09.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea638a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1ea00a06d0 addr v1:172.21.15.71:6789/0 2024-09-11T01:31:09.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea638a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1ea0134ac0 0x7f1ea01405f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:09.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea638a640 10 monclient(hunting): picked mon.noname-a con 0x7f1ea0134ac0 addr v2:172.21.15.12:3300/0 2024-09-11T01:31:09.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea638a640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ea0070160 con 0x7f1ea0074180 2024-09-11T01:31:09.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea638a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ea0070440 con 0x7f1ea00a06d0 2024-09-11T01:31:09.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea638a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:09.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea638a640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f1ea0135560 con 0x7f1ea0134ac0 2024-09-11T01:31:09.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea638a640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:09.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea638a640 10 monclient(hunting): authenticate will time out at 875.620476s 2024-09-11T01:31:09.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1e9ffff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1ea0074180 0x7f1ea0074560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34838/0 (socket says 172.21.15.12:34838) 2024-09-11T01:31:09.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1e9ffff640 1 -- 172.21.15.12:0/727198648 learned_addr learned my addr 172.21.15.12:0/727198648 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:31:09.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea4927640 1 --2- 172.21.15.12:0/727198648 >> v2:172.21.15.12:3300/0 conn(0x7f1ea0134ac0 0x7f1ea01405f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:09.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea4927640 10 monclient(hunting): get_auth_request con 0x7f1ea0134ac0 auth_method 0 2024-09-11T01:31:09.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea4927640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:09.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea4927640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:09.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea4927640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:09.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea4927640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:09.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea4927640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:09.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea4927640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:09.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea4927640 10 monclient(hunting): handle_auth_done global_id 24123 payload 274 2024-09-11T01:31:09.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea4927640 1 -- 172.21.15.12:0/727198648 >> v1:172.21.15.71:6789/0 conn(0x7f1ea00a06d0 legacy=0x7f1ea013cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:09.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea4927640 1 -- 172.21.15.12:0/727198648 >> v1:172.21.15.12:6789/0 conn(0x7f1ea0074180 legacy=0x7f1ea0074560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:09.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea4927640 10 monclient: _finish_hunting 0 2024-09-11T01:31:09.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea4927640 1 monclient: found mon.noname-a 2024-09-11T01:31:09.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea4927640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:09.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea4927640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:31:09.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.177+0000 7f1ea4927640 1 -- 172.21.15.12:0/727198648 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea0068bc0 con 0x7f1ea0134ac0 2024-09-11T01:31:09.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.180+0000 7f1ea4927640 1 -- 172.21.15.12:0/727198648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ea0134ac0 msgr2=0x7f1ea01405f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:09.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.180+0000 7f1ea4927640 1 --2- 172.21.15.12:0/727198648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ea0134ac0 0x7f1ea01405f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1e98009d90 tx=0x7f1e9802a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=590b076461f9f27c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:09.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.180+0000 7f1e9effd640 1 -- 172.21.15.12:0/727198648 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e98007a30 con 0x7f1ea0134ac0 2024-09-11T01:31:09.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.180+0000 7f1e9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:09.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.180+0000 7f1e9effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:31:09.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.180+0000 7f1e9effd640 10 monclient: dump: 2024-09-11T01:31:09.192 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:09.192 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:09.192 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:09.192 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:09.192 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:09.193 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.180+0000 7f1e9effd640 10 monclient: _finish_auth 0 2024-09-11T01:31:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.180+0000 7f1e9effd640 10 monclient: _check_auth_tickets 2024-09-11T01:31:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.180+0000 7f1e9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.180+0000 7f1e9effd640 1 -- 172.21.15.12:0/727198648 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e98007bd0 con 0x7f1ea0134ac0 2024-09-11T01:31:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.180+0000 7f1ea638a640 5 monclient: authenticate success, global_id 24123 2024-09-11T01:31:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.180+0000 7f1ea638a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:31:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.180+0000 7f1e9effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:09.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.180+0000 7f1ea638a640 10 monclient: get_monmap_and_config success 2024-09-11T01:31:09.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.180+0000 7f1e9effd640 1 -- 172.21.15.12:0/727198648 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e98031400 con 0x7f1ea0134ac0 2024-09-11T01:31:09.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.180+0000 7f1e9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:09.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.180+0000 7f1ea638a640 4 set_mon_vals no callback set 2024-09-11T01:31:09.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.180+0000 7f1e9effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:09.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.180+0000 7f1e9effd640 10 monclient: dump: 2024-09-11T01:31:09.194 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:09.194 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:09.194 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:09.194 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:09.194 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:09.194 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:09.194 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:09.194 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:09.195 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.181+0000 7f1ea638a640 10 monclient: shutdown 2024-09-11T01:31:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.181+0000 7f1ea638a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.181+0000 7f1ea638a640 1 -- 172.21.15.12:0/727198648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ea0134ac0 msgr2=0x7f1ea01405f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.181+0000 7f1ea638a640 1 --2- 172.21.15.12:0/727198648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ea0134ac0 0x7f1ea01405f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1e98009d90 tx=0x7f1e9802a490 comp rx=0 tx=0).stop 2024-09-11T01:31:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.181+0000 7f1ea638a640 1 -- 172.21.15.12:0/727198648 shutdown_connections 2024-09-11T01:31:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.181+0000 7f1ea638a640 1 --2- 172.21.15.12:0/727198648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ea0134ac0 0x7f1ea01405f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.181+0000 7f1ea638a640 1 -- 172.21.15.12:0/727198648 >> 172.21.15.12:0/727198648 conn(0x7f1ea00a36c0 msgr2=0x7f1ea00a3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.181+0000 7f1ea638a640 1 -- 172.21.15.12:0/727198648 shutdown_connections 2024-09-11T01:31:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.181+0000 7f1ea638a640 1 -- 172.21.15.12:0/727198648 wait complete. 2024-09-11T01:31:09.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.182+0000 7f1ea638a640 10 monclient: build_initial_monmap 2024-09-11T01:31:09.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.182+0000 7f1ea638a640 10 monclient: monmap: 2024-09-11T01:31:09.196 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:09.196 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:31:09.196 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:31:09.196 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:31:09.196 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:09.196 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:09.196 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:31:09.196 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:31:09.196 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:31:09.196 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:09.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.182+0000 7f1ea638a640 1 Processor -- start 2024-09-11T01:31:09.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.182+0000 7f1ea638a640 1 -- start start 2024-09-11T01:31:09.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.182+0000 7f1ea638a640 10 monclient: init 2024-09-11T01:31:09.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea638a640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:09.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea638a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:31:09.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea638a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea0074180 0x7f1ea0132740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:09.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea638a640 10 monclient(hunting): picked mon.noname-b con 0x7f1ea0074180 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:31:09.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea638a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ea00a06d0 0x7f1ea0132c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:09.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea638a640 10 monclient(hunting): picked mon.noname-c con 0x7f1ea00a06d0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:09.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea638a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ea0134ac0 0x7f1ea01ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:09.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea638a640 10 monclient(hunting): picked mon.noname-a con 0x7f1ea0134ac0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:09.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea638a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:09.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea638a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea0140d40 con 0x7f1ea0134ac0 2024-09-11T01:31:09.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea638a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:09.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea638a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea0070740 con 0x7f1ea0074180 2024-09-11T01:31:09.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea638a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea638a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1ea0140ec0 con 0x7f1ea00a06d0 2024-09-11T01:31:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea638a640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea638a640 10 monclient(hunting): authenticate will time out at 875.626122s 2024-09-11T01:31:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea4927640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ea0134ac0 0x7f1ea01ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1e9f7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ea00a06d0 0x7f1ea0132c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea4927640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ea0134ac0 0x7f1ea01ccfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45730/0 (socket says 172.21.15.12:45730) 2024-09-11T01:31:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea4927640 1 -- 172.21.15.12:0/452307119 learned_addr learned my addr 172.21.15.12:0/452307119 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea4927640 10 monclient(hunting): get_auth_request con 0x7f1ea0134ac0 auth_method 0 2024-09-11T01:31:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea4927640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea4927640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea4927640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1e9f7fe640 10 monclient(hunting): get_auth_request con 0x7f1ea00a06d0 auth_method 0 2024-09-11T01:31:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1e9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:09.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1e9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:09.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1e9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:09.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea4927640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:09.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea4927640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:09.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea4927640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:09.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1e9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:09.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1e9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:09.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1e9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:09.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1e9ffff640 1 --2- 172.21.15.12:0/452307119 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea0074180 0x7f1ea0132740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:09.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1e9ffff640 10 monclient(hunting): get_auth_request con 0x7f1ea0074180 auth_method 0 2024-09-11T01:31:09.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1e9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:09.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1e9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:09.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1e9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:09.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea4927640 10 monclient(hunting): handle_auth_done global_id 24126 payload 824 2024-09-11T01:31:09.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea4927640 1 -- 172.21.15.12:0/452307119 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ea00a06d0 msgr2=0x7f1ea0132c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:09.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea4927640 1 --2- 172.21.15.12:0/452307119 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ea00a06d0 0x7f1ea0132c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:09.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea4927640 1 -- 172.21.15.12:0/452307119 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea0074180 msgr2=0x7f1ea0132740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:09.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea4927640 1 --2- 172.21.15.12:0/452307119 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea0074180 0x7f1ea0132740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:09.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea4927640 10 monclient: _finish_hunting 0 2024-09-11T01:31:09.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea4927640 1 monclient: found mon.noname-a 2024-09-11T01:31:09.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea4927640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:09.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea4927640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:31:09.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1ea4927640 1 -- 172.21.15.12:0/452307119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea013e0b0 con 0x7f1ea0134ac0 2024-09-11T01:31:09.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1e9ffff640 1 --2- 172.21.15.12:0/452307119 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea0074180 0x7f1ea0132740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:31:09.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.183+0000 7f1e9f7fe640 1 --2- 172.21.15.12:0/452307119 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ea00a06d0 0x7f1ea0132c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:31:09.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.185+0000 7f1ea4927640 1 -- 172.21.15.12:0/452307119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ea0134ac0 msgr2=0x7f1ea01ccfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:09.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.185+0000 7f1ea4927640 1 --2- 172.21.15.12:0/452307119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ea0134ac0 0x7f1ea01ccfe0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1e98009d90 tx=0x7f1e98004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:09.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.185+0000 7f1e9cff9640 1 -- 172.21.15.12:0/452307119 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e98040070 con 0x7f1ea0134ac0 2024-09-11T01:31:09.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.186+0000 7f1e9cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:09.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.186+0000 7f1e9cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:31:09.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.186+0000 7f1e9cff9640 10 monclient: dump: 2024-09-11T01:31:09.204 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:09.204 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:09.204 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:09.204 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:09.204 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:09.204 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:09.204 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:09.204 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:09.205 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:09.205 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:09.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.186+0000 7f1e9cff9640 10 monclient: _finish_auth 0 2024-09-11T01:31:09.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.186+0000 7f1e9cff9640 10 monclient: _check_auth_tickets 2024-09-11T01:31:09.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.186+0000 7f1e9cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:09.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.186+0000 7f1e9cff9640 1 -- 172.21.15.12:0/452307119 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e9803c620 con 0x7f1ea0134ac0 2024-09-11T01:31:09.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.186+0000 7f1ea638a640 5 monclient: authenticate success, global_id 24126 2024-09-11T01:31:09.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.186+0000 7f1ea638a640 10 monclient: _renew_subs 2024-09-11T01:31:09.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.186+0000 7f1ea638a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:09.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.186+0000 7f1ea638a640 1 -- 172.21.15.12:0/452307119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ea013cbc0 con 0x7f1ea0134ac0 2024-09-11T01:31:09.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.186+0000 7f1e9cff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:09.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.186+0000 7f1e9cff9640 1 -- 172.21.15.12:0/452307119 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e980314c0 con 0x7f1ea0134ac0 2024-09-11T01:31:09.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.186+0000 7f1e9cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.186+0000 7f1e9cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.186+0000 7f1e9cff9640 10 monclient: dump: 2024-09-11T01:31:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:09.206 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.186+0000 7f1ea638a640 10 monclient: _renew_subs 2024-09-11T01:31:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.186+0000 7f1ea638a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.186+0000 7f1ea638a640 1 -- 172.21.15.12:0/452307119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ea013e680 con 0x7f1ea0134ac0 2024-09-11T01:31:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.186+0000 7f1e9e7fc640 4 set_mon_vals no callback set 2024-09-11T01:31:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.187+0000 7f1ea638a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.187+0000 7f1ea638a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.187+0000 7f1ea638a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.187+0000 7f1e9cff9640 1 -- 172.21.15.12:0/452307119 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f1e98007a40 con 0x7f1ea0134ac0 2024-09-11T01:31:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.187+0000 7f1ea638a640 1 -- 172.21.15.12:0/452307119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ea0074560 con 0x7f1ea0134ac0 2024-09-11T01:31:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.187+0000 7f1e9cff9640 1 --2- 172.21.15.12:0/452307119 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1e8c03e210 0x7f1e8c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.187+0000 7f1e9cff9640 1 -- 172.21.15.12:0/452307119 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 4783+0+0 (secure 0 0 0) 0x7f1e98074750 con 0x7f1ea0134ac0 2024-09-11T01:31:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.187+0000 7f1e9ffff640 1 --2- 172.21.15.12:0/452307119 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1e8c03e210 0x7f1e8c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.187+0000 7f1e9ffff640 10 monclient: get_auth_request con 0x7f1e8c03e210 auth_method 0 2024-09-11T01:31:09.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.190+0000 7f1e9ffff640 1 --2- 172.21.15.12:0/452307119 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1e8c03e210 0x7f1e8c0406d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1e90005160 tx=0x7f1e90009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:09.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.192+0000 7f1e9cff9640 1 -- 172.21.15.12:0/452307119 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1e9804d3f0 con 0x7f1ea0134ac0 2024-09-11T01:31:09.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.192+0000 7f1e9cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:09.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.192+0000 7f1e9cff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:31:09.361 INFO:tasks.rados.rados.0.smithi071.stdout:223: done (1 left) 2024-09-11T01:31:09.362 INFO:tasks.rados.rados.0.smithi071.stdout:224: done (0 left) 2024-09-11T01:31:09.362 INFO:tasks.rados.rados.0.smithi071.stdout:225: rollback oid 3 current snap is 24 2024-09-11T01:31:09.362 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 3 to 23 2024-09-11T01:31:09.362 INFO:tasks.rados.rados.0.smithi071.stdout:226: write oid 35 current snap is 24 2024-09-11T01:31:09.362 INFO:tasks.rados.rados.0.smithi071.stdout:226: seq_num 100 ranges {480700=637489,1651964=401685,2713358=770613} 2024-09-11T01:31:09.368 INFO:tasks.rados.rados.0.smithi071.stdout:226: writing smithi07134777-35 from 480700 to 1118189 tid 1 2024-09-11T01:31:09.373 INFO:tasks.rados.rados.0.smithi071.stdout:226: writing smithi07134777-35 from 1651964 to 2053649 tid 2 2024-09-11T01:31:09.381 INFO:tasks.rados.rados.0.smithi071.stdout:226: writing smithi07134777-35 from 2713358 to 3483971 tid 3 2024-09-11T01:31:09.381 INFO:tasks.rados.rados.0.smithi071.stdout:227: delete oid 39 current snap is 24 2024-09-11T01:31:09.381 INFO:tasks.rados.rados.0.smithi071.stdout:225: finishing rollback tid 0 to smithi07134777-3 2024-09-11T01:31:09.382 INFO:tasks.rados.rados.0.smithi071.stdout:225: finishing rollback tid 1 to smithi07134777-3 2024-09-11T01:31:09.382 INFO:tasks.rados.rados.0.smithi071.stdout:225: finishing rollback tid 2 to smithi07134777-3 2024-09-11T01:31:09.382 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 19 (ObjNum 86 snap 18 seq_num 86) dirty exists 2024-09-11T01:31:09.382 INFO:tasks.rados.rados.0.smithi071.stdout:226: finishing write tid 1 to smithi07134777-35 2024-09-11T01:31:09.382 INFO:tasks.rados.rados.0.smithi071.stdout:226: finishing write tid 2 to smithi07134777-35 2024-09-11T01:31:09.387 INFO:tasks.rados.rados.0.smithi071.stdout:226: finishing write tid 3 to smithi07134777-35 2024-09-11T01:31:09.387 INFO:tasks.rados.rados.0.smithi071.stdout:226: finishing write tid 4 to smithi07134777-35 2024-09-11T01:31:09.387 INFO:tasks.rados.rados.0.smithi071.stdout:226: finishing write tid 5 to smithi07134777-35 2024-09-11T01:31:09.387 INFO:tasks.rados.rados.0.smithi071.stdout:226: oid 35 updating version 0 to 75 2024-09-11T01:31:09.387 INFO:tasks.rados.rados.0.smithi071.stdout:226: oid 35 version 75 is already newer than 72 2024-09-11T01:31:09.387 INFO:tasks.rados.rados.0.smithi071.stdout:226: oid 35 version 75 is already newer than 73 2024-09-11T01:31:09.387 INFO:tasks.rados.rados.0.smithi071.stdout:226: oid 35 version 75 is already newer than 74 2024-09-11T01:31:09.387 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 75 (ObjNum 100 snap 24 seq_num 100) dirty exists 2024-09-11T01:31:09.388 INFO:tasks.rados.rados.0.smithi071.stdout:226: left oid 35 (ObjNum 100 snap 24 seq_num 100) 2024-09-11T01:31:09.388 INFO:tasks.rados.rados.0.smithi071.stdout:225: done (2 left) 2024-09-11T01:31:09.388 INFO:tasks.rados.rados.0.smithi071.stdout:226: done (1 left) 2024-09-11T01:31:09.388 INFO:tasks.rados.rados.0.smithi071.stdout:227: done (0 left) 2024-09-11T01:31:09.388 INFO:tasks.rados.rados.0.smithi071.stdout:228: rollback oid 26 current snap is 24 2024-09-11T01:31:09.388 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 26 to 23 2024-09-11T01:31:09.388 INFO:tasks.rados.rados.0.smithi071.stdout:229: read oid 19 snap -1 2024-09-11T01:31:09.388 INFO:tasks.rados.rados.0.smithi071.stdout:229: expect deleted 2024-09-11T01:31:09.388 INFO:tasks.rados.rados.0.smithi071.stdout:230: copy_from oid 33 from oid 10 current snap is 24 2024-09-11T01:31:09.388 INFO:tasks.rados.rados.0.smithi071.stdout:231: copy_from oid 28 from oid 13 current snap is 24 2024-09-11T01:31:09.388 INFO:tasks.rados.rados.0.smithi071.stdout:232: rollback oid 37 current snap is 24 2024-09-11T01:31:09.388 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 37 to 23 2024-09-11T01:31:09.388 INFO:tasks.rados.rados.0.smithi071.stdout:233: delete oid 48 current snap is 24 2024-09-11T01:31:09.389 INFO:tasks.rados.rados.0.smithi071.stdout:231: finishing copy_from to smithi07134777-28 2024-09-11T01:31:09.389 INFO:tasks.rados.rados.0.smithi071.stdout:231: got expected ENOENT (src dne) 2024-09-11T01:31:09.389 INFO:tasks.rados.rados.0.smithi071.stdout:231: finishing copy_from racing read to smithi07134777-28 2024-09-11T01:31:09.391 INFO:tasks.rados.rados.0.smithi071.stdout:232: finishing rollback tid 0 to smithi07134777-37 2024-09-11T01:31:09.391 INFO:tasks.rados.rados.0.smithi071.stdout:232: finishing rollback tid 1 to smithi07134777-37 2024-09-11T01:31:09.391 INFO:tasks.rados.rados.0.smithi071.stdout:228: finishing rollback tid 0 to smithi07134777-26 2024-09-11T01:31:09.391 INFO:tasks.rados.rados.0.smithi071.stdout:229: done (5 left) 2024-09-11T01:31:09.392 INFO:tasks.rados.rados.0.smithi071.stdout:231: done (4 left) 2024-09-11T01:31:09.392 INFO:tasks.rados.rados.0.smithi071.stdout:233: done (3 left) 2024-09-11T01:31:09.392 INFO:tasks.rados.rados.0.smithi071.stdout:234: delete oid 48 current snap is 24 2024-09-11T01:31:09.393 INFO:tasks.rados.rados.0.smithi071.stdout:232: finishing rollback tid 2 to smithi07134777-37 2024-09-11T01:31:09.393 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 81 (ObjNum 61 snap 5 seq_num 61) dirty exists 2024-09-11T01:31:09.394 INFO:tasks.rados.rados.0.smithi071.stdout:232: done (3 left) 2024-09-11T01:31:09.394 INFO:tasks.rados.rados.0.smithi071.stdout:234: done (2 left) 2024-09-11T01:31:09.394 INFO:tasks.rados.rados.0.smithi071.stdout:235: read oid 48 snap 23 2024-09-11T01:31:09.394 INFO:tasks.rados.rados.0.smithi071.stdout:235: expect (ObjNum 98 snap 22 seq_num 98) 2024-09-11T01:31:09.395 INFO:tasks.rados.rados.0.smithi071.stdout:236: read oid 25 snap -1 2024-09-11T01:31:09.395 INFO:tasks.rados.rados.0.smithi071.stdout:236: expect deleted 2024-09-11T01:31:09.395 INFO:tasks.rados.rados.0.smithi071.stdout:237: copy_from oid 1 from oid 44 current snap is 24 2024-09-11T01:31:09.395 INFO:tasks.rados.rados.0.smithi071.stdout:238: write (excl) oid 14 current snap is 24 2024-09-11T01:31:09.395 INFO:tasks.rados.rados.0.smithi071.stdout:238: seq_num 104 ranges {547928=538485,1769539=409023,2706697=532054} 2024-09-11T01:31:09.400 INFO:tasks.rados.rados.0.smithi071.stdout:238: writing smithi07134777-14 from 547928 to 1086413 tid 1 2024-09-11T01:31:09.407 INFO:tasks.rados.rados.0.smithi071.stdout:238: writing smithi07134777-14 from 1769539 to 2178562 tid 2 2024-09-11T01:31:09.413 INFO:tasks.rados.rados.0.smithi071.stdout:238: writing smithi07134777-14 from 2706697 to 3238751 tid 3 2024-09-11T01:31:09.413 INFO:tasks.rados.rados.0.smithi071.stdout:239: snap_create 2024-09-11T01:31:09.413 INFO:tasks.rados.rados.0.smithi071.stdout:228: finishing rollback tid 1 to smithi07134777-26 2024-09-11T01:31:09.413 INFO:tasks.rados.rados.0.smithi071.stdout:228: finishing rollback tid 2 to smithi07134777-26 2024-09-11T01:31:09.413 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 118 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-11T01:31:09.414 INFO:tasks.rados.rados.0.smithi071.stdout:235: expect (ObjNum 98 snap 22 seq_num 98) 2024-09-11T01:31:09.414 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:31:09.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.463+0000 7f1ea638a640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:31:09.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.463+0000 7f1ea638a640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:31:09.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.463+0000 7f1ea638a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:09.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.463+0000 7f1ea638a640 1 -- 172.21.15.12:0/452307119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f1ea013ea20 con 0x7f1ea0134ac0 2024-09-11T01:31:09.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.464+0000 7f1e9cff9640 1 -- 172.21.15.12:0/452307119 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f1e98073150 con 0x7f1ea0134ac0 2024-09-11T01:31:09.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.464+0000 7f1e9cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:31:09.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.464+0000 7f1e9cff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:31:09.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.464+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:09.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.464+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=8,config=2,health=12,kv=12,logm=66} crc {auth=1930248783,config=2586702516,health=3897446751,kv=3263697420,logm=3254800704}) 2024-09-11T01:31:09.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.464+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=12,kv=11,logm=66} crc {auth=2729599117,config=2586702516,health=3897446751,kv=58696370,logm=1118020541}) 2024-09-11T01:31:09.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.464+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:09.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.464+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=8,config=2,health=12,kv=12,logm=66} crc {auth=1930248783,config=2586702516,health=3897446751,kv=3263697420,logm=3254800704}) 2024-09-11T01:31:09.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.464+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=9,config=2,health=11,kv=12,logm=66} crc {auth=2729599117,config=2586702516,health=1997762793,kv=3263697420,logm=3541217957}) 2024-09-11T01:31:09.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.465+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:09.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.465+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2243191951}) 2024-09-11T01:31:09.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.465+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=851615755}) 2024-09-11T01:31:09.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.465+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:09.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.465+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2243191951}) 2024-09-11T01:31:09.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.465+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2912843763}) 2024-09-11T01:31:09.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.466+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:09.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.466+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=280453612}) 2024-09-11T01:31:09.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.466+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=982524574}) 2024-09-11T01:31:09.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.466+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:09.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.466+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=280453612}) 2024-09-11T01:31:09.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.466+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=86172861}) 2024-09-11T01:31:09.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.466+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:09.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.466+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=540613245}) 2024-09-11T01:31:09.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.466+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=988791334}) 2024-09-11T01:31:09.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.466+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:09.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.466+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=540613245}) 2024-09-11T01:31:09.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.466+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4234036119}) 2024-09-11T01:31:09.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.467+0000 7f1ea638a640 1 -- 172.21.15.12:0/452307119 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1e8c03e210 msgr2=0x7f1e8c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:09.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.467+0000 7f1ea638a640 1 --2- 172.21.15.12:0/452307119 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1e8c03e210 0x7f1e8c0406d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1e90005160 tx=0x7f1e90009040 comp rx=0 tx=0).stop 2024-09-11T01:31:09.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.467+0000 7f1ea638a640 10 monclient: shutdown 2024-09-11T01:31:09.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.467+0000 7f1ea638a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:09.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.467+0000 7f1ea638a640 1 -- 172.21.15.12:0/452307119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ea0134ac0 msgr2=0x7f1ea01ccfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:09.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.467+0000 7f1ea638a640 1 --2- 172.21.15.12:0/452307119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ea0134ac0 0x7f1ea01ccfe0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1e98009d90 tx=0x7f1e98004a40 comp rx=0 tx=0).stop 2024-09-11T01:31:09.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.467+0000 7f1ea638a640 1 -- 172.21.15.12:0/452307119 shutdown_connections 2024-09-11T01:31:09.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.467+0000 7f1ea638a640 1 --2- 172.21.15.12:0/452307119 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1e8c03e210 0x7f1e8c0406d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:09.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.467+0000 7f1ea638a640 1 --2- 172.21.15.12:0/452307119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ea0134ac0 0x7f1ea01ccfe0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:09.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.467+0000 7f1ea638a640 1 --2- 172.21.15.12:0/452307119 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ea00a06d0 0x7f1ea0132c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:09.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.467+0000 7f1ea638a640 1 --2- 172.21.15.12:0/452307119 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea0074180 0x7f1ea0132740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:09.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.467+0000 7f1ea638a640 1 -- 172.21.15.12:0/452307119 >> 172.21.15.12:0/452307119 conn(0x7f1ea00a36c0 msgr2=0x7f1ea00a6170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:09.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.467+0000 7f1ea638a640 1 -- 172.21.15.12:0/452307119 shutdown_connections 2024-09-11T01:31:09.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:09.467+0000 7f1ea638a640 1 -- 172.21.15.12:0/452307119 wait complete. 2024-09-11T01:31:09.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.467+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:09.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.467+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=748842353}) 2024-09-11T01:31:09.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.467+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2591577101}) 2024-09-11T01:31:09.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.467+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:09.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.467+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=748842353}) 2024-09-11T01:31:09.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.467+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3032995884}) 2024-09-11T01:31:09.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.468+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:09.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.468+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=1246075256,mdsmap=2682192099}) 2024-09-11T01:31:09.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.468+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=3316762071,mdsmap=3760429986}) 2024-09-11T01:31:09.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.468+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:09.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.468+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=1246075256,mdsmap=2682192099}) 2024-09-11T01:31:09.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.468+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=4286327896,mdsmap=3760429986,mgr=4133961934}) 2024-09-11T01:31:09.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.469+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:09.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.469+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=11,osdmap=38} crc {mdsmap=3140514009,mgr=2431795800,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1112062367,mon_config_key=3330371543,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=3137194527,osd_snap=922559302,osdmap=706176723}) 2024-09-11T01:31:09.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.469+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=14,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=8,osdmap=46} crc {mdsmap=3140514009,mgr=4080174641,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=697929348,mon_config_key=3752130409,monmap=1324671969,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=3137194527,osd_snap=3620543820,osdmap=966641682}) 2024-09-11T01:31:09.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.469+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:09.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.469+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=11,osdmap=38} crc {mdsmap=3140514009,mgr=2431795800,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1112062367,mon_config_key=3330371543,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=3137194527,osd_snap=922559302,osdmap=706176723}) 2024-09-11T01:31:09.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.469+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=11,osdmap=38} crc {mdsmap=3140514009,mgr=454318940,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=3384279874,mon_config_key=3330371543,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=3137194527,osd_snap=922559302,osdmap=4746933}) 2024-09-11T01:31:09.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.470+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:09.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.470+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=79} crc {osdmap=2963186486}) 2024-09-11T01:31:09.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.470+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=78} crc {osdmap=19647851}) 2024-09-11T01:31:09.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.470+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:09.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.470+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=79} crc {osdmap=2963186486}) 2024-09-11T01:31:09.476 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:09.470+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=77} crc {osdmap=2530017920}) 2024-09-11T01:31:09.481 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:31:09.611 INFO:tasks.rados.rados.0.smithi071.stdout:230: finishing copy_from racing read to smithi07134777-33 2024-09-11T01:31:09.611 INFO:tasks.rados.rados.0.smithi071.stdout:238: finishing write tid 1 to smithi07134777-14 2024-09-11T01:31:09.611 INFO:tasks.rados.rados.0.smithi071.stdout:230: finishing copy_from to smithi07134777-33 2024-09-11T01:31:09.611 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 29 (ObjNum 88 snap 19 seq_num 88) dirty exists 2024-09-11T01:31:09.611 INFO:tasks.rados.rados.0.smithi071.stdout:237: finishing copy_from racing read to smithi07134777-1 2024-09-11T01:31:09.611 INFO:tasks.rados.rados.0.smithi071.stdout:237: finishing copy_from to smithi07134777-1 2024-09-11T01:31:09.611 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 27 (ObjNum 71 snap 13 seq_num 71) dirty exists 2024-09-11T01:31:09.611 INFO:tasks.rados.rados.0.smithi071.stdout:238: finishing write tid 2 to smithi07134777-14 2024-09-11T01:31:09.611 INFO:tasks.rados.rados.0.smithi071.stdout:238: finishing write tid 3 to smithi07134777-14 2024-09-11T01:31:09.611 INFO:tasks.rados.rados.0.smithi071.stdout:238: finishing write tid 4 to smithi07134777-14 2024-09-11T01:31:09.612 INFO:tasks.rados.rados.0.smithi071.stdout:238: finishing write tid 5 to smithi07134777-14 2024-09-11T01:31:09.612 INFO:tasks.rados.rados.0.smithi071.stdout:238: oid 14 updating version 0 to 133 2024-09-11T01:31:09.612 INFO:tasks.rados.rados.0.smithi071.stdout:238: oid 14 version 133 is already newer than 131 2024-09-11T01:31:09.612 INFO:tasks.rados.rados.0.smithi071.stdout:238: oid 14 updating version 133 to 134 2024-09-11T01:31:09.612 INFO:tasks.rados.rados.0.smithi071.stdout:238: oid 14 version 134 is already newer than 132 2024-09-11T01:31:09.612 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 134 (ObjNum 104 snap 24 seq_num 104) dirty exists 2024-09-11T01:31:09.612 INFO:tasks.rados.rados.0.smithi071.stdout:238: left oid 14 (ObjNum 104 snap 24 seq_num 104) 2024-09-11T01:31:10.482 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:31:10.482 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:31:10.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.592+0000 7fda3f2f7640 10 monclient: get_monmap_and_config 2024-09-11T01:31:10.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.592+0000 7fda3f2f7640 10 monclient: build_initial_monmap 2024-09-11T01:31:10.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.592+0000 7fda3f2f7640 10 monclient: monmap: 2024-09-11T01:31:10.595 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:10.595 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:10.595 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:31:10.594414+0000 2024-09-11T01:31:10.595 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:31:10.594414+0000 2024-09-11T01:31:10.596 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:10.596 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:10.596 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:31:10.596 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:31:10.596 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:31:10.596 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:31:10.596 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:31:10.596 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:10.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.593+0000 7fda3f2f7640 1 Processor -- start 2024-09-11T01:31:10.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.593+0000 7fda3f2f7640 1 -- start start 2024-09-11T01:31:10.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.593+0000 7fda3f2f7640 10 monclient: init 2024-09-11T01:31:10.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3f2f7640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:10.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3f2f7640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-11T01:31:10.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3f2f7640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fda38138230 0x7fda38138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:10.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3f2f7640 10 monclient(hunting): picked mon.noname-a con 0x7fda38138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:31:10.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3f2f7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fda3809d030 addr v1:172.21.15.71:6789/0 2024-09-11T01:31:10.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3f2f7640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda3809d460 0x7fda380a5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:10.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3f2f7640 10 monclient(hunting): picked mon.noname-b con 0x7fda3809d460 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:10.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3f2f7640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda380a6e30 con 0x7fda3809d030 2024-09-11T01:31:10.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3f2f7640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:10.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3f2f7640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fda3809c9c0 con 0x7fda38138230 2024-09-11T01:31:10.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3f2f7640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:10.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3f2f7640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fda3809ba20 con 0x7fda3809d460 2024-09-11T01:31:10.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3f2f7640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:10.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3f2f7640 10 monclient(hunting): authenticate will time out at 877.037189s 2024-09-11T01:31:10.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d093640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fda38138230 0x7fda38138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:10.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d894640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda3809d460 0x7fda380a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:10.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d894640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda3809d460 0x7fda380a5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55948/0 (socket says 172.21.15.12:55948) 2024-09-11T01:31:10.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d093640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fda38138230 0x7fda38138610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45740/0 (socket says 172.21.15.12:45740) 2024-09-11T01:31:10.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d894640 1 -- 172.21.15.12:0/4288980242 learned_addr learned my addr 172.21.15.12:0/4288980242 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:10.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d894640 10 monclient(hunting): get_auth_request con 0x7fda3809d460 auth_method 0 2024-09-11T01:31:10.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d894640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:10.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d894640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:10.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d894640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:10.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d093640 10 monclient(hunting): get_auth_request con 0x7fda38138230 auth_method 0 2024-09-11T01:31:10.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d093640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:10.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d093640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:10.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d093640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:10.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d093640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:10.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d093640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:10.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d093640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:10.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d894640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:10.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d894640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:10.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d894640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:10.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d894640 10 monclient(hunting): handle_auth_done global_id 4553 payload 274 2024-09-11T01:31:10.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d894640 1 -- 172.21.15.12:0/4288980242 >> v2:172.21.15.12:3300/0 conn(0x7fda38138230 msgr2=0x7fda38138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:10.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d894640 1 --2- 172.21.15.12:0/4288980242 >> v2:172.21.15.12:3300/0 conn(0x7fda38138230 0x7fda38138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:10.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d894640 1 -- 172.21.15.12:0/4288980242 >> v1:172.21.15.71:6789/0 conn(0x7fda3809d030 legacy=0x7fda380a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:10.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d894640 10 monclient: _finish_hunting 0 2024-09-11T01:31:10.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d894640 1 monclient: found mon.noname-b 2024-09-11T01:31:10.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d894640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:10.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d894640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:31:10.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d894640 1 -- 172.21.15.12:0/4288980242 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda380a66a0 con 0x7fda3809d460 2024-09-11T01:31:10.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.594+0000 7fda3d093640 1 --2- 172.21.15.12:0/4288980242 >> v2:172.21.15.12:3300/0 conn(0x7fda38138230 0x7fda38138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:31:10.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.597+0000 7fda3d894640 1 -- 172.21.15.12:0/4288980242 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda3809d460 msgr2=0x7fda380a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:10.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.597+0000 7fda3d894640 1 --2- 172.21.15.12:0/4288980242 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda3809d460 0x7fda380a5730 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fda380a17e0 tx=0x7fda3002a530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64411a854b656886 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:10.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.597+0000 7fda2ffff640 1 -- 172.21.15.12:0/4288980242 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda30036070 con 0x7fda3809d460 2024-09-11T01:31:10.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.597+0000 7fda2ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:10.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.597+0000 7fda2ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:31:10.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.597+0000 7fda2ffff640 10 monclient: dump: 2024-09-11T01:31:10.608 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:10.608 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:10.608 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:10.608 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:10.608 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:10.608 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:10.608 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:10.609 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:10.609 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:10.609 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:10.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.597+0000 7fda2ffff640 10 monclient: _finish_auth 0 2024-09-11T01:31:10.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.597+0000 7fda2ffff640 10 monclient: _check_auth_tickets 2024-09-11T01:31:10.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.597+0000 7fda2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:10.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.597+0000 7fda2ffff640 1 -- 172.21.15.12:0/4288980242 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda30007800 con 0x7fda3809d460 2024-09-11T01:31:10.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.597+0000 7fda3f2f7640 5 monclient: authenticate success, global_id 4553 2024-09-11T01:31:10.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.597+0000 7fda3f2f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:31:10.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.597+0000 7fda2ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:10.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.597+0000 7fda3f2f7640 10 monclient: get_monmap_and_config success 2024-09-11T01:31:10.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.597+0000 7fda2ffff640 1 -- 172.21.15.12:0/4288980242 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda30007bd0 con 0x7fda3809d460 2024-09-11T01:31:10.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.597+0000 7fda2ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:10.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.597+0000 7fda3f2f7640 4 set_mon_vals no callback set 2024-09-11T01:31:10.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.597+0000 7fda2ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:10.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.597+0000 7fda2ffff640 10 monclient: dump: 2024-09-11T01:31:10.610 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:10.610 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:10.610 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:10.610 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:10.610 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:10.610 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:10.610 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:10.610 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:10.610 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:10.610 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:10.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.598+0000 7fda3f2f7640 10 monclient: shutdown 2024-09-11T01:31:10.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.598+0000 7fda3f2f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:10.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.598+0000 7fda3f2f7640 1 -- 172.21.15.12:0/4288980242 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda3809d460 msgr2=0x7fda380a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:10.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.598+0000 7fda3f2f7640 1 --2- 172.21.15.12:0/4288980242 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda3809d460 0x7fda380a5730 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fda380a17e0 tx=0x7fda3002a530 comp rx=0 tx=0).stop 2024-09-11T01:31:10.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.598+0000 7fda3f2f7640 1 -- 172.21.15.12:0/4288980242 shutdown_connections 2024-09-11T01:31:10.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.598+0000 7fda3f2f7640 1 --2- 172.21.15.12:0/4288980242 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda3809d460 0x7fda380a5730 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:10.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.598+0000 7fda3f2f7640 1 --2- 172.21.15.12:0/4288980242 >> v2:172.21.15.12:3300/0 conn(0x7fda38138230 0x7fda38138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:10.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.598+0000 7fda3f2f7640 1 -- 172.21.15.12:0/4288980242 >> 172.21.15.12:0/4288980242 conn(0x7fda380a7120 msgr2=0x7fda380a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:10.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.598+0000 7fda3f2f7640 1 -- 172.21.15.12:0/4288980242 shutdown_connections 2024-09-11T01:31:10.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.598+0000 7fda3f2f7640 1 -- 172.21.15.12:0/4288980242 wait complete. 2024-09-11T01:31:10.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.599+0000 7fda3f2f7640 10 monclient: build_initial_monmap 2024-09-11T01:31:10.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.599+0000 7fda3f2f7640 10 monclient: monmap: 2024-09-11T01:31:10.611 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:10.611 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:31:10.611 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:31:10.612 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:31:10.612 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:10.612 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:10.612 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:31:10.612 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:31:10.612 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:31:10.612 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:10.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.599+0000 7fda3f2f7640 1 Processor -- start 2024-09-11T01:31:10.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.599+0000 7fda3f2f7640 1 -- start start 2024-09-11T01:31:10.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.599+0000 7fda3f2f7640 10 monclient: init 2024-09-11T01:31:10.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.599+0000 7fda3f2f7640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:10.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.599+0000 7fda3f2f7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:31:10.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3f2f7640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda3809d030 0x7fda38136bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:10.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3f2f7640 10 monclient(hunting): picked mon.noname-b con 0x7fda3809d030 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:31:10.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3f2f7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda3809d460 0x7fda381370f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:10.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3f2f7640 10 monclient(hunting): picked mon.noname-a con 0x7fda3809d460 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:10.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3f2f7640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda38138230 0x7fda38120c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:10.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3f2f7640 10 monclient(hunting): picked mon.noname-c con 0x7fda38138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:10.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3f2f7640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:10.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3f2f7640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fda3806f970 con 0x7fda3809d460 2024-09-11T01:31:10.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3f2f7640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:10.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3f2f7640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fda3806f7f0 con 0x7fda3809d030 2024-09-11T01:31:10.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3f2f7640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:10.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3f2f7640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fda380a5c70 con 0x7fda38138230 2024-09-11T01:31:10.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3f2f7640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:10.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3f2f7640 10 monclient(hunting): authenticate will time out at 877.043090s 2024-09-11T01:31:10.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3c892640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda3809d460 0x7fda381370f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:10.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3c892640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda3809d460 0x7fda381370f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45750/0 (socket says 172.21.15.12:45750) 2024-09-11T01:31:10.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3d894640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda38138230 0x7fda38120c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:10.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3c892640 1 -- 172.21.15.12:0/1239694468 learned_addr learned my addr 172.21.15.12:0/1239694468 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:10.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3c892640 10 monclient(hunting): get_auth_request con 0x7fda3809d460 auth_method 0 2024-09-11T01:31:10.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3c892640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:10.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3c892640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:10.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3c892640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:10.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3d894640 10 monclient(hunting): get_auth_request con 0x7fda38138230 auth_method 0 2024-09-11T01:31:10.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3d894640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:10.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3d894640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:10.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3d894640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:10.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3c892640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:10.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3c892640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:10.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3c892640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:10.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3d894640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:10.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3d894640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:10.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3d894640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:10.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3d093640 1 --2- 172.21.15.12:0/1239694468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda3809d030 0x7fda38136bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:10.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3d093640 10 monclient(hunting): get_auth_request con 0x7fda3809d030 auth_method 0 2024-09-11T01:31:10.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3d093640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:10.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3d093640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:10.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3d093640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:10.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3c892640 10 monclient(hunting): handle_auth_done global_id 24132 payload 824 2024-09-11T01:31:10.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3c892640 1 -- 172.21.15.12:0/1239694468 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda38138230 msgr2=0x7fda38120c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:10.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3c892640 1 --2- 172.21.15.12:0/1239694468 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda38138230 0x7fda38120c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:10.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3c892640 1 -- 172.21.15.12:0/1239694468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda3809d030 msgr2=0x7fda38136bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:10.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3c892640 1 --2- 172.21.15.12:0/1239694468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda3809d030 0x7fda38136bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:10.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3c892640 10 monclient: _finish_hunting 0 2024-09-11T01:31:10.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3c892640 1 monclient: found mon.noname-a 2024-09-11T01:31:10.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3c892640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:10.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3c892640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:31:10.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.600+0000 7fda3c892640 1 -- 172.21.15.12:0/1239694468 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda38137660 con 0x7fda3809d460 2024-09-11T01:31:10.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.602+0000 7fda3c892640 1 -- 172.21.15.12:0/1239694468 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda3809d460 msgr2=0x7fda381370f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:10.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.602+0000 7fda3c892640 1 --2- 172.21.15.12:0/1239694468 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda3809d460 0x7fda381370f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fda2800c430 tx=0x7fda2800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:10.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.602+0000 7fda2dffb640 1 -- 172.21.15.12:0/1239694468 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda280119a0 con 0x7fda3809d460 2024-09-11T01:31:10.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.602+0000 7fda2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:10.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.602+0000 7fda2dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:31:10.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.602+0000 7fda2dffb640 10 monclient: dump: 2024-09-11T01:31:10.616 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:10.616 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:10.617 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:10.617 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:10.617 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:10.617 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:10.617 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:10.617 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:10.617 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:10.617 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:10.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.602+0000 7fda2dffb640 10 monclient: _finish_auth 0 2024-09-11T01:31:10.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.602+0000 7fda2dffb640 10 monclient: _check_auth_tickets 2024-09-11T01:31:10.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.602+0000 7fda2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:10.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.602+0000 7fda3f2f7640 5 monclient: authenticate success, global_id 24132 2024-09-11T01:31:10.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.602+0000 7fda3f2f7640 10 monclient: _renew_subs 2024-09-11T01:31:10.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.602+0000 7fda3f2f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:10.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.602+0000 7fda3f2f7640 1 -- 172.21.15.12:0/1239694468 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda38123b70 con 0x7fda3809d460 2024-09-11T01:31:10.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.602+0000 7fda2dffb640 1 -- 172.21.15.12:0/1239694468 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda28017070 con 0x7fda3809d460 2024-09-11T01:31:10.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.603+0000 7fda2dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:10.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.603+0000 7fda2dffb640 1 -- 172.21.15.12:0/1239694468 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda28011d80 con 0x7fda3809d460 2024-09-11T01:31:10.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.603+0000 7fda2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:10.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.603+0000 7fda2dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:10.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.603+0000 7fda2dffb640 10 monclient: dump: 2024-09-11T01:31:10.618 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:10.618 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:10.618 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:10.618 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:10.618 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:10.619 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:10.619 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:10.619 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:10.619 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:10.619 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:10.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.603+0000 7fda2f7fe640 4 set_mon_vals no callback set 2024-09-11T01:31:10.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.603+0000 7fda3f2f7640 10 monclient: _renew_subs 2024-09-11T01:31:10.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.603+0000 7fda3f2f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:10.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.603+0000 7fda3f2f7640 1 -- 172.21.15.12:0/1239694468 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda380a1810 con 0x7fda3809d460 2024-09-11T01:31:10.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.604+0000 7fda3f2f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:10.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.604+0000 7fda3f2f7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:10.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.604+0000 7fda3f2f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:10.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.604+0000 7fda3f2f7640 1 -- 172.21.15.12:0/1239694468 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda38138610 con 0x7fda3809d460 2024-09-11T01:31:10.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.604+0000 7fda2dffb640 1 -- 172.21.15.12:0/1239694468 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fda2801c460 con 0x7fda3809d460 2024-09-11T01:31:10.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.604+0000 7fda2dffb640 1 --2- 172.21.15.12:0/1239694468 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda1c03e210 0x7fda1c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:10.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.604+0000 7fda2dffb640 1 -- 172.21.15.12:0/1239694468 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 4755+0+0 (secure 0 0 0) 0x7fda28052c10 con 0x7fda3809d460 2024-09-11T01:31:10.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.604+0000 7fda3d093640 1 --2- 172.21.15.12:0/1239694468 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda1c03e210 0x7fda1c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:10.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.605+0000 7fda3d093640 10 monclient: get_auth_request con 0x7fda1c03e210 auth_method 0 2024-09-11T01:31:10.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.607+0000 7fda3d093640 1 --2- 172.21.15.12:0/1239694468 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda1c03e210 0x7fda1c0406d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fda34009a60 tx=0x7fda34008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:10.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.609+0000 7fda2dffb640 1 -- 172.21.15.12:0/1239694468 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fda28012470 con 0x7fda3809d460 2024-09-11T01:31:10.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.609+0000 7fda2dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:10.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.609+0000 7fda2dffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:31:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.863+0000 7fda3f2f7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:31:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.863+0000 7fda3f2f7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:31:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.863+0000 7fda3f2f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.863+0000 7fda3f2f7640 1 -- 172.21.15.12:0/1239694468 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fda380698e0 con 0x7fda3809d460 2024-09-11T01:31:10.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.864+0000 7fda2dffb640 1 -- 172.21.15.12:0/1239694468 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fda2801ac80 con 0x7fda3809d460 2024-09-11T01:31:10.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.864+0000 7fda2dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:31:10.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.864+0000 7fda2dffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:31:10.867 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:31:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.866+0000 7fda3f2f7640 1 -- 172.21.15.12:0/1239694468 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda1c03e210 msgr2=0x7fda1c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.866+0000 7fda3f2f7640 1 --2- 172.21.15.12:0/1239694468 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda1c03e210 0x7fda1c0406d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fda34009a60 tx=0x7fda34008040 comp rx=0 tx=0).stop 2024-09-11T01:31:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.866+0000 7fda3f2f7640 10 monclient: shutdown 2024-09-11T01:31:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.866+0000 7fda3f2f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.866+0000 7fda3f2f7640 1 -- 172.21.15.12:0/1239694468 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda3809d460 msgr2=0x7fda381370f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.866+0000 7fda3f2f7640 1 --2- 172.21.15.12:0/1239694468 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda3809d460 0x7fda381370f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fda2800c430 tx=0x7fda2800c8f0 comp rx=0 tx=0).stop 2024-09-11T01:31:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.866+0000 7fda3f2f7640 1 -- 172.21.15.12:0/1239694468 shutdown_connections 2024-09-11T01:31:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.866+0000 7fda3f2f7640 1 --2- 172.21.15.12:0/1239694468 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda38138230 0x7fda38120c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.866+0000 7fda3f2f7640 1 --2- 172.21.15.12:0/1239694468 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda1c03e210 0x7fda1c0406d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.866+0000 7fda3f2f7640 1 --2- 172.21.15.12:0/1239694468 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda3809d460 0x7fda381370f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.866+0000 7fda3f2f7640 1 --2- 172.21.15.12:0/1239694468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda3809d030 0x7fda38136bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.866+0000 7fda3f2f7640 1 -- 172.21.15.12:0/1239694468 >> 172.21.15.12:0/1239694468 conn(0x7fda380a7120 msgr2=0x7fda380a33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.866+0000 7fda3f2f7640 1 -- 172.21.15.12:0/1239694468 shutdown_connections 2024-09-11T01:31:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.866+0000 7fda3f2f7640 1 -- 172.21.15.12:0/1239694468 wait complete. 2024-09-11T01:31:10.879 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:31:10.879 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:31:10.879 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:31:10.880 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:31:10.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.979+0000 7fb25e9f8640 10 monclient: get_monmap_and_config 2024-09-11T01:31:10.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.979+0000 7fb25e9f8640 10 monclient: build_initial_monmap 2024-09-11T01:31:10.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.979+0000 7fb25e9f8640 10 monclient: monmap: 2024-09-11T01:31:10.982 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:10.983 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:10.983 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:31:10.981465+0000 2024-09-11T01:31:10.983 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:31:10.981465+0000 2024-09-11T01:31:10.983 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:10.983 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:10.983 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:31:10.983 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:31:10.983 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:31:10.983 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:31:10.983 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:31:10.983 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:10.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.980+0000 7fb25e9f8640 1 Processor -- start 2024-09-11T01:31:10.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.980+0000 7fb25e9f8640 1 -- start start 2024-09-11T01:31:10.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.980+0000 7fb25e9f8640 10 monclient: init 2024-09-11T01:31:10.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25e9f8640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:10.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25e9f8640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-11T01:31:10.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25e9f8640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb2580a2e20 0x7fb25806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:10.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25e9f8640 10 monclient(hunting): picked mon.noname-c con 0x7fb2580a2e20 addr v2:172.21.15.71:3300/0 2024-09-11T01:31:10.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25e9f8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb2580a4970 addr v1:172.21.15.12:6789/0 2024-09-11T01:31:10.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25e9f8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb25806f910 0x7fb25813f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:10.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25e9f8640 10 monclient(hunting): picked mon.noname-a con 0x7fb25806f910 addr v2:172.21.15.12:3300/0 2024-09-11T01:31:10.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25e9f8640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb25806f4f0 con 0x7fb2580a4970 2024-09-11T01:31:10.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25e9f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:10.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25e9f8640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fb25806ef30 con 0x7fb25806f910 2024-09-11T01:31:10.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25e9f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:10.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25e9f8640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fb25806ec50 con 0x7fb2580a2e20 2024-09-11T01:31:10.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25e9f8640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:10.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25e9f8640 10 monclient(hunting): authenticate will time out at 877.424192s 2024-09-11T01:31:10.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25cf95640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb25806f910 0x7fb25813f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:10.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb24ffff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb2580a4970 0x7fb2580a3200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34850/0 (socket says 172.21.15.12:34850) 2024-09-11T01:31:10.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb24ffff640 1 -- 172.21.15.12:0/926435908 learned_addr learned my addr 172.21.15.12:0/926435908 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:31:10.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25cf95640 10 monclient(hunting): get_auth_request con 0x7fb25806f910 auth_method 0 2024-09-11T01:31:10.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25cf95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:10.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25cf95640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:10.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25cf95640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:10.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25cf95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:10.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25cf95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:10.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25cf95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:10.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb257fff640 1 --2- 172.21.15.12:0/926435908 >> v2:172.21.15.71:3300/0 conn(0x7fb2580a2e20 0x7fb25806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:10.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb257fff640 10 monclient(hunting): get_auth_request con 0x7fb2580a2e20 auth_method 0 2024-09-11T01:31:10.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb257fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:10.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb257fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:10.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb257fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:10.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25cf95640 10 monclient(hunting): handle_auth_done global_id 24135 payload 274 2024-09-11T01:31:10.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25cf95640 1 -- 172.21.15.12:0/926435908 >> v2:172.21.15.71:3300/0 conn(0x7fb2580a2e20 msgr2=0x7fb25806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:10.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25cf95640 1 --2- 172.21.15.12:0/926435908 >> v2:172.21.15.71:3300/0 conn(0x7fb2580a2e20 0x7fb25806ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:10.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25cf95640 1 -- 172.21.15.12:0/926435908 >> v1:172.21.15.12:6789/0 conn(0x7fb2580a4970 legacy=0x7fb2580a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:10.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25cf95640 10 monclient: _finish_hunting 0 2024-09-11T01:31:10.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25cf95640 1 monclient: found mon.noname-a 2024-09-11T01:31:10.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25cf95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:10.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25cf95640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:31:10.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.981+0000 7fb25cf95640 1 -- 172.21.15.12:0/926435908 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2580a3910 con 0x7fb25806f910 2024-09-11T01:31:10.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.984+0000 7fb25cf95640 1 -- 172.21.15.12:0/926435908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb25806f910 msgr2=0x7fb25813f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:10.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.984+0000 7fb25cf95640 1 --2- 172.21.15.12:0/926435908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb25806f910 0x7fb25813f430 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb258070ba0 tx=0x7fb25002a4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d653c23ad0be957 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:10.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.984+0000 7fb2577fe640 1 -- 172.21.15.12:0/926435908 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb250007950 con 0x7fb25806f910 2024-09-11T01:31:10.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.984+0000 7fb2577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:10.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.984+0000 7fb2577fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:31:10.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.984+0000 7fb2577fe640 10 monclient: dump: 2024-09-11T01:31:10.998 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:10.998 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:10.998 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:10.998 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:10.999 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:10.999 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:10.999 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:10.999 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:10.999 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:10.999 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:10.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.984+0000 7fb2577fe640 10 monclient: _finish_auth 0 2024-09-11T01:31:10.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.984+0000 7fb2577fe640 10 monclient: _check_auth_tickets 2024-09-11T01:31:10.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.984+0000 7fb2577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:10.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.984+0000 7fb25e9f8640 5 monclient: authenticate success, global_id 24135 2024-09-11T01:31:11.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.984+0000 7fb25e9f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:31:11.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.984+0000 7fb2577fe640 1 -- 172.21.15.12:0/926435908 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb250007af0 con 0x7fb25806f910 2024-09-11T01:31:11.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.984+0000 7fb2577fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:11.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.984+0000 7fb25e9f8640 10 monclient: get_monmap_and_config success 2024-09-11T01:31:11.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.984+0000 7fb25e9f8640 4 set_mon_vals no callback set 2024-09-11T01:31:11.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.984+0000 7fb2577fe640 1 -- 172.21.15.12:0/926435908 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb250031400 con 0x7fb25806f910 2024-09-11T01:31:11.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.984+0000 7fb2577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:11.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.984+0000 7fb2577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:11.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.984+0000 7fb2577fe640 10 monclient: dump: 2024-09-11T01:31:11.000 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:11.000 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:11.001 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.985+0000 7fb25e9f8640 10 monclient: shutdown 2024-09-11T01:31:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.985+0000 7fb25e9f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.985+0000 7fb25e9f8640 1 -- 172.21.15.12:0/926435908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb25806f910 msgr2=0x7fb25813f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.985+0000 7fb25e9f8640 1 --2- 172.21.15.12:0/926435908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb25806f910 0x7fb25813f430 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb258070ba0 tx=0x7fb25002a4b0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.985+0000 7fb25e9f8640 1 -- 172.21.15.12:0/926435908 shutdown_connections 2024-09-11T01:31:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.985+0000 7fb25e9f8640 1 --2- 172.21.15.12:0/926435908 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb25806f910 0x7fb25813f430 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.985+0000 7fb25e9f8640 1 --2- 172.21.15.12:0/926435908 >> v2:172.21.15.71:3300/0 conn(0x7fb2580a2e20 0x7fb25806ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.985+0000 7fb25e9f8640 1 -- 172.21.15.12:0/926435908 >> 172.21.15.12:0/926435908 conn(0x7fb2580a8600 msgr2=0x7fb2580a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.985+0000 7fb25e9f8640 1 -- 172.21.15.12:0/926435908 shutdown_connections 2024-09-11T01:31:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.985+0000 7fb25e9f8640 1 -- 172.21.15.12:0/926435908 wait complete. 2024-09-11T01:31:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.986+0000 7fb25e9f8640 10 monclient: build_initial_monmap 2024-09-11T01:31:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.986+0000 7fb25e9f8640 10 monclient: monmap: 2024-09-11T01:31:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:31:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:31:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:31:11.003 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:11.003 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.003 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:31:11.003 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:31:11.003 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:31:11.003 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.986+0000 7fb25e9f8640 1 Processor -- start 2024-09-11T01:31:11.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.986+0000 7fb25e9f8640 1 -- start start 2024-09-11T01:31:11.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.986+0000 7fb25e9f8640 10 monclient: init 2024-09-11T01:31:11.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.986+0000 7fb25e9f8640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:11.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.986+0000 7fb25e9f8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:31:11.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.986+0000 7fb25e9f8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb25806f910 0x7fb258139400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.986+0000 7fb25e9f8640 10 monclient(hunting): picked mon.noname-a con 0x7fb25806f910 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:11.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.986+0000 7fb25e9f8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb2580a2e20 0x7fb258139940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.986+0000 7fb25e9f8640 10 monclient(hunting): picked mon.noname-b con 0x7fb2580a2e20 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:31:11.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.986+0000 7fb25e9f8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb2580a4970 0x7fb258139e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.986+0000 7fb25e9f8640 10 monclient(hunting): picked mon.noname-c con 0x7fb2580a4970 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:11.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.986+0000 7fb25e9f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:11.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.986+0000 7fb25e9f8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb25813fd00 con 0x7fb25806f910 2024-09-11T01:31:11.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.986+0000 7fb25e9f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:11.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.986+0000 7fb25e9f8640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb25813fb80 con 0x7fb2580a2e20 2024-09-11T01:31:11.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.986+0000 7fb25e9f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:11.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.986+0000 7fb25e9f8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb25813fe80 con 0x7fb2580a4970 2024-09-11T01:31:11.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.986+0000 7fb25e9f8640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:11.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.986+0000 7fb25e9f8640 10 monclient(hunting): authenticate will time out at 877.429942s 2024-09-11T01:31:11.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb257fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb25806f910 0x7fb258139400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:11.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb25cf95640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb2580a4970 0x7fb258139e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:11.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb257fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb25806f910 0x7fb258139400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45780/0 (socket says 172.21.15.12:45780) 2024-09-11T01:31:11.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb257fff640 1 -- 172.21.15.12:0/2151312278 learned_addr learned my addr 172.21.15.12:0/2151312278 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:11.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb257fff640 10 monclient(hunting): get_auth_request con 0x7fb25806f910 auth_method 0 2024-09-11T01:31:11.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb257fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:11.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb257fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:11.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb257fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:11.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb25cf95640 10 monclient(hunting): get_auth_request con 0x7fb2580a4970 auth_method 0 2024-09-11T01:31:11.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb25cf95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:11.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb25cf95640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:11.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb25cf95640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:11.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb257fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:11.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb257fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:11.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb257fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:11.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb25cf95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:11.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb25cf95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:11.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb24ffff640 1 --2- 172.21.15.12:0/2151312278 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb2580a2e20 0x7fb258139940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:11.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb25cf95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:11.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb24ffff640 10 monclient(hunting): get_auth_request con 0x7fb2580a2e20 auth_method 0 2024-09-11T01:31:11.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb24ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:11.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb24ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:11.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb24ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:11.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb24ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:11.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb24ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:11.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb24ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:11.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb257fff640 10 monclient(hunting): handle_auth_done global_id 24138 payload 824 2024-09-11T01:31:11.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb257fff640 1 -- 172.21.15.12:0/2151312278 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb2580a4970 msgr2=0x7fb258139e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:11.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb257fff640 1 --2- 172.21.15.12:0/2151312278 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb2580a4970 0x7fb258139e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb257fff640 1 -- 172.21.15.12:0/2151312278 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb2580a2e20 msgr2=0x7fb258139940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:11.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb257fff640 1 --2- 172.21.15.12:0/2151312278 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb2580a2e20 0x7fb258139940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb257fff640 10 monclient: _finish_hunting 0 2024-09-11T01:31:11.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb257fff640 1 monclient: found mon.noname-a 2024-09-11T01:31:11.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb257fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:11.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb257fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:31:11.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.987+0000 7fb257fff640 1 -- 172.21.15.12:0/2151312278 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb25813bc90 con 0x7fb25806f910 2024-09-11T01:31:11.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.989+0000 7fb257fff640 1 -- 172.21.15.12:0/2151312278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb25806f910 msgr2=0x7fb258139400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:11.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.989+0000 7fb257fff640 1 --2- 172.21.15.12:0/2151312278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb25806f910 0x7fb258139400 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb24800d830 tx=0x7fb24800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:11.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.989+0000 7fb2557fa640 1 -- 172.21.15.12:0/2151312278 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb24800b870 con 0x7fb25806f910 2024-09-11T01:31:11.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.989+0000 7fb2557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:11.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.989+0000 7fb2557fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:31:11.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.989+0000 7fb2557fa640 10 monclient: dump: 2024-09-11T01:31:11.009 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:11.009 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:11.009 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.009 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.009 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:11.009 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.009 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:11.009 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:11.009 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:11.009 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.989+0000 7fb2557fa640 10 monclient: _finish_auth 0 2024-09-11T01:31:11.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.989+0000 7fb2557fa640 10 monclient: _check_auth_tickets 2024-09-11T01:31:11.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.989+0000 7fb2557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:11.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.989+0000 7fb2557fa640 1 -- 172.21.15.12:0/2151312278 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb248019070 con 0x7fb25806f910 2024-09-11T01:31:11.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.989+0000 7fb2557fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:11.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.989+0000 7fb2557fa640 1 -- 172.21.15.12:0/2151312278 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb24800bc30 con 0x7fb25806f910 2024-09-11T01:31:11.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.989+0000 7fb25e9f8640 5 monclient: authenticate success, global_id 24138 2024-09-11T01:31:11.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.989+0000 7fb2557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:11.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.989+0000 7fb2557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:11.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.989+0000 7fb2557fa640 10 monclient: dump: 2024-09-11T01:31:11.010 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:11.010 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:11.010 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.011 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.011 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:11.011 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.011 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:11.011 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:11.011 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:11.011 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.989+0000 7fb25e9f8640 10 monclient: _renew_subs 2024-09-11T01:31:11.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.989+0000 7fb256ffd640 4 set_mon_vals no callback set 2024-09-11T01:31:11.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.989+0000 7fb25e9f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:11.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.989+0000 7fb25e9f8640 1 -- 172.21.15.12:0/2151312278 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb25813bf70 con 0x7fb25806f910 2024-09-11T01:31:11.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.989+0000 7fb25e9f8640 10 monclient: _renew_subs 2024-09-11T01:31:11.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.989+0000 7fb25e9f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:11.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.989+0000 7fb25e9f8640 1 -- 172.21.15.12:0/2151312278 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2580a6700 con 0x7fb25806f910 2024-09-11T01:31:11.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.990+0000 7fb25e9f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:11.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.990+0000 7fb25e9f8640 1 --2- 172.21.15.12:0/2151312278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb258135760 0x7fb25802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.990+0000 7fb25e9f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:11.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.990+0000 7fb25e9f8640 1 -- 172.21.15.12:0/2151312278 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb25806a060 con 0x7fb258135760 2024-09-11T01:31:11.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.990+0000 7fb24ffff640 1 --2- 172.21.15.12:0/2151312278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb258135760 0x7fb25802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:11.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.990+0000 7fb24ffff640 10 monclient: get_auth_request con 0x7fb258135760 auth_method 0 2024-09-11T01:31:11.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.990+0000 7fb24ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:11.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.990+0000 7fb24ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:11.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.990+0000 7fb24ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:11.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.990+0000 7fb257fff640 1 --2- 172.21.15.12:0/2151312278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb25806f910 0x7fb258139400 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb24800d830 tx=0x7fb24800dcf0 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7fb248052e90 osd_map(69..69 src has 1..69) 2024-09-11T01:31:11.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.990+0000 7fb2557fa640 1 -- 172.21.15.12:0/2151312278 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fb248013070 con 0x7fb25806f910 2024-09-11T01:31:11.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.990+0000 7fb24ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:11.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.990+0000 7fb24ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:11.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.990+0000 7fb24ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:11.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.990+0000 7fb24ffff640 10 monclient(hunting): handle_auth_done global_id 24141 payload 824 2024-09-11T01:31:11.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.990+0000 7fb24ffff640 1 -- 172.21.15.12:0/2151312278 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb25806ded0 con 0x7fb258135760 2024-09-11T01:31:11.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.991+0000 7fb2557fa640 1 --2- 172.21.15.12:0/2151312278 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb23803e260 0x7fb238040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.991+0000 7fb25cf95640 1 --2- 172.21.15.12:0/2151312278 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb23803e260 0x7fb238040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:11.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.991+0000 7fb25cf95640 10 monclient: get_auth_request con 0x7fb23803e260 auth_method 0 2024-09-11T01:31:11.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.993+0000 7fb24ffff640 1 -- 172.21.15.12:0/2151312278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb258135760 msgr2=0x7fb25802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:11.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.993+0000 7fb24ffff640 1 --2- 172.21.15.12:0/2151312278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb258135760 0x7fb25802d440 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb2580a22c0 tx=0x7fb2400052d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:11.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.993+0000 7fb2557fa640 1 -- 172.21.15.12:0/2151312278 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2400138f0 con 0x7fb258135760 2024-09-11T01:31:11.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.993+0000 7fb2557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:11.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.993+0000 7fb2557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:11.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.993+0000 7fb2557fa640 10 monclient: dump: 2024-09-11T01:31:11.014 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:11.014 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:11.014 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.014 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.014 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:11.015 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.015 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:11.015 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:11.015 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:11.015 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.993+0000 7fb2557fa640 1 -- 172.21.15.12:0/2151312278 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb2400138f0 con 0x7fb258135760 2024-09-11T01:31:11.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.993+0000 7fb2557fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:11.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.993+0000 7fb2557fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:31:11.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.993+0000 7fb2557fa640 1 -- 172.21.15.12:0/2151312278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb258135760 msgr2=0x7fb25802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:11.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.993+0000 7fb2557fa640 1 --2- 172.21.15.12:0/2151312278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb258135760 0x7fb25802d440 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb2580a22c0 tx=0x7fb2400052d0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.993+0000 7fb2557fa640 1 -- 172.21.15.12:0/2151312278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb258135760 msgr2=0x7fb25802d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:31:11.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.993+0000 7fb2557fa640 1 --2- 172.21.15.12:0/2151312278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb258135760 0x7fb25802d440 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb2580a22c0 tx=0x7fb2400052d0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.994+0000 7fb25cf95640 1 --2- 172.21.15.12:0/2151312278 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb23803e260 0x7fb238040720 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb258074f40 tx=0x7fb25002f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:11.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.999+0000 7fb25e9f8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:31:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.999+0000 7fb25e9f8640 1 --2- 172.21.15.12:0/2151312278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb258135b40 0x7fb25802d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.999+0000 7fb25e9f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:10.999+0000 7fb25e9f8640 1 -- 172.21.15.12:0/2151312278 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb25809da30 con 0x7fb258135b40 2024-09-11T01:31:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.000+0000 7fb24ffff640 1 --2- 172.21.15.12:0/2151312278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb258135b40 0x7fb25802d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.000+0000 7fb24ffff640 10 monclient: get_auth_request con 0x7fb258135b40 auth_method 0 2024-09-11T01:31:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.000+0000 7fb24ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.000+0000 7fb24ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.000+0000 7fb24ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.000+0000 7fb24ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.000+0000 7fb24ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.000+0000 7fb24ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.000+0000 7fb24ffff640 10 monclient(hunting): handle_auth_done global_id 24144 payload 824 2024-09-11T01:31:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.000+0000 7fb24ffff640 1 -- 172.21.15.12:0/2151312278 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb258069950 con 0x7fb258135b40 2024-09-11T01:31:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.002+0000 7fb24ffff640 1 -- 172.21.15.12:0/2151312278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb258135b40 msgr2=0x7fb25802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.002+0000 7fb24ffff640 1 --2- 172.21.15.12:0/2151312278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb258135b40 0x7fb25802d980 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb240017000 tx=0x7fb240004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.002+0000 7fb2557fa640 1 -- 172.21.15.12:0/2151312278 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb24001dc20 con 0x7fb258135b40 2024-09-11T01:31:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.002+0000 7fb2557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.002+0000 7fb2557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.002+0000 7fb2557fa640 10 monclient: dump: 2024-09-11T01:31:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:11.020 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.020 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.020 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:11.020 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.020 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:11.020 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:11.020 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:11.020 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.002+0000 7fb2557fa640 1 -- 172.21.15.12:0/2151312278 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7fb24001dc20 con 0x7fb258135b40 2024-09-11T01:31:11.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.002+0000 7fb2557fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:31:11.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.002+0000 7fb2557fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:31:11.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.002+0000 7fb2557fa640 1 -- 172.21.15.12:0/2151312278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb258135b40 msgr2=0x7fb25802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.002+0000 7fb2557fa640 1 --2- 172.21.15.12:0/2151312278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb258135b40 0x7fb25802d980 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb240017000 tx=0x7fb240004940 comp rx=0 tx=0).stop 2024-09-11T01:31:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.002+0000 7fb2557fa640 1 -- 172.21.15.12:0/2151312278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb258135b40 msgr2=0x7fb25802d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:31:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.002+0000 7fb2557fa640 1 --2- 172.21.15.12:0/2151312278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb258135b40 0x7fb25802d980 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb240017000 tx=0x7fb240004940 comp rx=0 tx=0).stop 2024-09-11T01:31:11.021 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:31:11.021 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:31:11.021 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:31:11.021 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:31:11.021 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 22, 2024-09-11T01:31:11.022 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 8949, 2024-09-11T01:31:11.022 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:31:11.022 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:31:11.022 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:31:11.022 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:31:11.022 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:11.022 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:31:11.022 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:31:11.022 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:31:11.022 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:31:11.022 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:11.022 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:11.023 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:11.023 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:11.023 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:11.023 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:11.023 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:11.023 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:11.023 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:11.023 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:11.023 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:11.023 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:11.023 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:31:11.024 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:31:11.024 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:11.024 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:11.024 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:11.024 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:11.024 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:11.024 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:11.024 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:11.024 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:11.024 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:11.024 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:11.025 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:11.025 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:11.025 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.025 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:31:11.025 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:31:11.025 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:31:11.025 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:31:11.025 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:31:11.025 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:31:11.025 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:31:11.025 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:31:11.025 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:31:11.026 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:31:11.026 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:31:11.026 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:31:11.026 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:31:11.026 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:31:11.026 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:31:11.026 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:31:11.026 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:31:11.026 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:11.026 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:11.026 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:11.027 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:11.027 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:11.027 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:11.027 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:11.027 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:11.027 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:11.027 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:11.027 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:11.027 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:11.027 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:31:11.027 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.028 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:31:11.028 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.028 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:31:11.028 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:31:11.028 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:11.028 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:11.028 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.028 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:11.028 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:31:11.028 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:11.028 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.028 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.029 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:11.029 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:31:11.029 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:11.029 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:11.030 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:11.030 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.030 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:31:11.030 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:31:11.030 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:11.030 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:11.030 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:11.031 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.031 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.031 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:31:11.031 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:31:11.031 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:11.031 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:11.031 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.031 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:11.031 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:31:11.031 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:11.031 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.032 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.032 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:11.032 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:31:11.032 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:11.032 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:11.032 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:11.032 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.032 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:31:11.032 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:31:11.033 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:11.033 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:11.033 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:11.033 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.033 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.033 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:31:11.033 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:31:11.034 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:11.034 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:11.034 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.034 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:11.034 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:31:11.034 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:11.034 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.034 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.034 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:11.035 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:31:11.035 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:11.035 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:11.035 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:11.035 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.035 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:31:11.035 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:31:11.035 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:11.035 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:11.036 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:11.036 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:11.036 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:11.036 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.036 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:31:11.036 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:31:11.036 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.036 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:11.036 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:11.036 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:31:11.036 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:11.036 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:11.037 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:31:11.037 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.037 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:11.037 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:11.037 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:31:11.037 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:11.037 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:11.037 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.037 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:31:11.037 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:31:11.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.003+0000 7fb25e9f8640 1 -- 172.21.15.12:0/2151312278 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb23803e260 msgr2=0x7fb238040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:11.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.003+0000 7fb25e9f8640 1 --2- 172.21.15.12:0/2151312278 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb23803e260 0x7fb238040720 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb258074f40 tx=0x7fb25002f000 comp rx=0 tx=0).stop 2024-09-11T01:31:11.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.003+0000 7fb25e9f8640 10 monclient: shutdown 2024-09-11T01:31:11.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.003+0000 7fb25e9f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:11.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.003+0000 7fb25e9f8640 1 -- 172.21.15.12:0/2151312278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb25806f910 msgr2=0x7fb258139400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:11.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.003+0000 7fb25e9f8640 1 --2- 172.21.15.12:0/2151312278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb25806f910 0x7fb258139400 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb24800d830 tx=0x7fb24800dcf0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.003+0000 7fb25cf95640 1 -- 172.21.15.12:0/2151312278 reap_dead start 2024-09-11T01:31:11.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.003+0000 7fb2557fa640 1 -- 172.21.15.12:0/2151312278 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 4755+0+0 (unknown 0 0 0) 0x7fb248052e90 con 0x7fb25806f910 2024-09-11T01:31:11.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.003+0000 7fb25cf95640 1 -- 172.21.15.12:0/2151312278 reap_dead start 2024-09-11T01:31:11.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.003+0000 7fb25e9f8640 1 -- 172.21.15.12:0/2151312278 shutdown_connections 2024-09-11T01:31:11.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.003+0000 7fb25e9f8640 1 -- 172.21.15.12:0/2151312278 >> 172.21.15.12:0/2151312278 conn(0x7fb2580a8600 msgr2=0x7fb2580a6c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:11.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.003+0000 7fb25e9f8640 1 -- 172.21.15.12:0/2151312278 shutdown_connections 2024-09-11T01:31:11.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.003+0000 7fb25e9f8640 1 -- 172.21.15.12:0/2151312278 wait complete. 2024-09-11T01:31:11.042 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:31:11.043 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:31:11.043 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:31:11.079 INFO:tasks.rados.rados.0.smithi071.stdout:228: done (6 left) 2024-09-11T01:31:11.079 INFO:tasks.rados.rados.0.smithi071.stdout:230: done (5 left) 2024-09-11T01:31:11.079 INFO:tasks.rados.rados.0.smithi071.stdout:235: done (4 left) 2024-09-11T01:31:11.079 INFO:tasks.rados.rados.0.smithi071.stdout:236: done (3 left) 2024-09-11T01:31:11.079 INFO:tasks.rados.rados.0.smithi071.stdout:237: done (2 left) 2024-09-11T01:31:11.079 INFO:tasks.rados.rados.0.smithi071.stdout:238: done (1 left) 2024-09-11T01:31:11.079 INFO:tasks.rados.rados.0.smithi071.stdout:239: done (0 left) 2024-09-11T01:31:11.079 INFO:tasks.rados.rados.0.smithi071.stdout:240: copy_from oid 43 from oid 8 current snap is 25 2024-09-11T01:31:11.079 INFO:tasks.rados.rados.0.smithi071.stdout:241: copy_from oid 37 from oid 10 current snap is 25 2024-09-11T01:31:11.079 INFO:tasks.rados.rados.0.smithi071.stdout:242: write oid 4 current snap is 25 2024-09-11T01:31:11.080 INFO:tasks.rados.rados.0.smithi071.stdout:242: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2024-09-11T01:31:11.085 INFO:tasks.rados.rados.0.smithi071.stdout:242: writing smithi07134777-4 from 564735 to 1178469 tid 1 2024-09-11T01:31:11.092 INFO:tasks.rados.rados.0.smithi071.stdout:242: writing smithi07134777-4 from 1828021 to 2497055 tid 2 2024-09-11T01:31:11.099 INFO:tasks.rados.rados.0.smithi071.stdout:242: writing smithi07134777-4 from 3134120 to 3764735 tid 3 2024-09-11T01:31:11.099 INFO:tasks.rados.rados.0.smithi071.stdout:243: rollback oid 49 current snap is 25 2024-09-11T01:31:11.099 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 49 to 24 2024-09-11T01:31:11.100 INFO:tasks.rados.rados.0.smithi071.stdout:242: finishing write tid 1 to smithi07134777-4 2024-09-11T01:31:11.100 INFO:tasks.rados.rados.0.smithi071.stdout:241: finishing copy_from racing read to smithi07134777-37 2024-09-11T01:31:11.100 INFO:tasks.rados.rados.0.smithi071.stdout:244: write (excl) oid 1 current snap is 25 2024-09-11T01:31:11.100 INFO:tasks.rados.rados.0.smithi071.stdout:244: seq_num 106 ranges {581542=688983,1886503=412692,2761543=670951} 2024-09-11T01:31:11.106 INFO:tasks.rados.rados.0.smithi071.stdout:244: writing smithi07134777-1 from 581542 to 1270525 tid 1 2024-09-11T01:31:11.112 INFO:tasks.rados.rados.0.smithi071.stdout:244: writing smithi07134777-1 from 1886503 to 2299195 tid 2 2024-09-11T01:31:11.120 INFO:tasks.rados.rados.0.smithi071.stdout:244: writing smithi07134777-1 from 2761543 to 3432494 tid 3 2024-09-11T01:31:11.120 INFO:tasks.rados.rados.0.smithi071.stdout:245: rollback oid 35 current snap is 25 2024-09-11T01:31:11.120 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 35 to 24 2024-09-11T01:31:11.120 INFO:tasks.rados.rados.0.smithi071.stdout:240: finishing copy_from racing read to smithi07134777-43 2024-09-11T01:31:11.120 INFO:tasks.rados.rados.0.smithi071.stdout:242: finishing write tid 2 to smithi07134777-4 2024-09-11T01:31:11.120 INFO:tasks.rados.rados.0.smithi071.stdout:241: finishing copy_from to smithi07134777-37 2024-09-11T01:31:11.120 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 29 (ObjNum 88 snap 19 seq_num 88) dirty exists 2024-09-11T01:31:11.120 INFO:tasks.rados.rados.0.smithi071.stdout:240: finishing copy_from to smithi07134777-43 2024-09-11T01:31:11.120 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 29 (ObjNum 74 snap 13 seq_num 74) dirty exists 2024-09-11T01:31:11.120 INFO:tasks.rados.rados.0.smithi071.stdout:242: finishing write tid 3 to smithi07134777-4 2024-09-11T01:31:11.121 INFO:tasks.rados.rados.0.smithi071.stdout:242: finishing write tid 4 to smithi07134777-4 2024-09-11T01:31:11.121 INFO:tasks.rados.rados.0.smithi071.stdout:242: finishing write tid 5 to smithi07134777-4 2024-09-11T01:31:11.121 INFO:tasks.rados.rados.0.smithi071.stdout:242: oid 4 updating version 0 to 66 2024-09-11T01:31:11.121 INFO:tasks.rados.rados.0.smithi071.stdout:242: oid 4 updating version 66 to 67 2024-09-11T01:31:11.121 INFO:tasks.rados.rados.0.smithi071.stdout:242: oid 4 version 67 is already newer than 64 2024-09-11T01:31:11.121 INFO:tasks.rados.rados.0.smithi071.stdout:242: oid 4 version 67 is already newer than 65 2024-09-11T01:31:11.121 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 67 (ObjNum 105 snap 25 seq_num 105) dirty exists 2024-09-11T01:31:11.121 INFO:tasks.rados.rados.0.smithi071.stdout:242: left oid 4 (ObjNum 105 snap 25 seq_num 105) 2024-09-11T01:31:11.121 INFO:tasks.rados.rados.0.smithi071.stdout:243: finishing rollback tid 0 to smithi07134777-49 2024-09-11T01:31:11.121 INFO:tasks.rados.rados.0.smithi071.stdout:243: finishing rollback tid 1 to smithi07134777-49 2024-09-11T01:31:11.121 INFO:tasks.rados.rados.0.smithi071.stdout:243: finishing rollback tid 2 to smithi07134777-49 2024-09-11T01:31:11.121 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 7 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-11T01:31:11.121 INFO:tasks.rados.rados.0.smithi071.stdout:244: finishing write tid 1 to smithi07134777-1 2024-09-11T01:31:11.121 INFO:tasks.rados.rados.0.smithi071.stdout:240: done (5 left) 2024-09-11T01:31:11.122 INFO:tasks.rados.rados.0.smithi071.stdout:241: done (4 left) 2024-09-11T01:31:11.122 INFO:tasks.rados.rados.0.smithi071.stdout:242: done (3 left) 2024-09-11T01:31:11.122 INFO:tasks.rados.rados.0.smithi071.stdout:243: done (2 left) 2024-09-11T01:31:11.122 INFO:tasks.rados.rados.0.smithi071.stdout:246: copy_from oid 18 from oid 33 current snap is 25 2024-09-11T01:31:11.122 INFO:tasks.rados.rados.0.smithi071.stdout:247: write (excl) oid 30 current snap is 25 2024-09-11T01:31:11.122 INFO:tasks.rados.rados.0.smithi071.stdout:247: seq_num 107 ranges {598349=480585,1544985=672703,2905319=482223} 2024-09-11T01:31:11.125 INFO:tasks.rados.rados.0.smithi071.stdout:247: writing smithi07134777-30 from 598349 to 1078934 tid 1 2024-09-11T01:31:11.131 INFO:tasks.rados.rados.0.smithi071.stdout:247: writing smithi07134777-30 from 1544985 to 2217688 tid 2 2024-09-11T01:31:11.138 INFO:tasks.rados.rados.0.smithi071.stdout:247: writing smithi07134777-30 from 2905319 to 3387542 tid 3 2024-09-11T01:31:11.138 INFO:tasks.rados.rados.0.smithi071.stdout:248: read oid 23 snap -1 2024-09-11T01:31:11.138 INFO:tasks.rados.rados.0.smithi071.stdout:248: expect (ObjNum 63 snap 8 seq_num 63) 2024-09-11T01:31:11.138 INFO:tasks.rados.rados.0.smithi071.stdout:249: snap_create 2024-09-11T01:31:11.138 INFO:tasks.rados.rados.0.smithi071.stdout:244: finishing write tid 2 to smithi07134777-1 2024-09-11T01:31:11.138 INFO:tasks.rados.rados.0.smithi071.stdout:245: finishing rollback tid 0 to smithi07134777-35 2024-09-11T01:31:11.139 INFO:tasks.rados.rados.0.smithi071.stdout:244: finishing write tid 3 to smithi07134777-1 2024-09-11T01:31:11.139 INFO:tasks.rados.rados.0.smithi071.stdout:245: finishing rollback tid 1 to smithi07134777-35 2024-09-11T01:31:11.139 INFO:tasks.rados.rados.0.smithi071.stdout:245: finishing rollback tid 2 to smithi07134777-35 2024-09-11T01:31:11.139 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 87 (ObjNum 100 snap 24 seq_num 100) dirty exists 2024-09-11T01:31:11.139 INFO:tasks.rados.rados.0.smithi071.stdout:244: finishing write tid 4 to smithi07134777-1 2024-09-11T01:31:11.139 INFO:tasks.rados.rados.0.smithi071.stdout:244: finishing write tid 5 to smithi07134777-1 2024-09-11T01:31:11.139 INFO:tasks.rados.rados.0.smithi071.stdout:244: oid 1 updating version 0 to 68 2024-09-11T01:31:11.139 INFO:tasks.rados.rados.0.smithi071.stdout:244: oid 1 updating version 68 to 70 2024-09-11T01:31:11.139 INFO:tasks.rados.rados.0.smithi071.stdout:244: oid 1 updating version 70 to 71 2024-09-11T01:31:11.139 INFO:tasks.rados.rados.0.smithi071.stdout:244: oid 1 version 71 is already newer than 69 2024-09-11T01:31:11.139 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 71 (ObjNum 106 snap 25 seq_num 106) dirty exists 2024-09-11T01:31:11.139 INFO:tasks.rados.rados.0.smithi071.stdout:244: left oid 1 (ObjNum 106 snap 25 seq_num 106) 2024-09-11T01:31:11.139 INFO:tasks.rados.rados.0.smithi071.stdout:246: finishing copy_from racing read to smithi07134777-18 2024-09-11T01:31:11.140 INFO:tasks.rados.rados.0.smithi071.stdout:246: finishing copy_from to smithi07134777-18 2024-09-11T01:31:11.140 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 29 (ObjNum 88 snap 19 seq_num 88) dirty exists 2024-09-11T01:31:11.143 INFO:tasks.rados.rados.0.smithi071.stdout:247: finishing write tid 1 to smithi07134777-30 2024-09-11T01:31:11.143 INFO:tasks.rados.rados.0.smithi071.stdout:247: finishing write tid 2 to smithi07134777-30 2024-09-11T01:31:11.143 INFO:tasks.rados.rados.0.smithi071.stdout:247: finishing write tid 3 to smithi07134777-30 2024-09-11T01:31:11.143 INFO:tasks.rados.rados.0.smithi071.stdout:248: expect (ObjNum 63 snap 8 seq_num 63) 2024-09-11T01:31:11.153 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1248722 2024-09-11T01:31:11.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.159+0000 7f35baa85640 10 monclient: get_monmap_and_config 2024-09-11T01:31:11.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.159+0000 7f35baa85640 10 monclient: build_initial_monmap 2024-09-11T01:31:11.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.159+0000 7f35baa85640 10 monclient: monmap: 2024-09-11T01:31:11.162 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:11.162 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:11.162 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:31:11.161410+0000 2024-09-11T01:31:11.162 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:31:11.161410+0000 2024-09-11T01:31:11.162 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:11.162 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.163 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:31:11.163 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:31:11.163 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:31:11.163 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:31:11.163 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:31:11.163 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.160+0000 7f35baa85640 1 Processor -- start 2024-09-11T01:31:11.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.160+0000 7f35baa85640 1 -- start start 2024-09-11T01:31:11.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.160+0000 7f35baa85640 10 monclient: init 2024-09-11T01:31:11.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.160+0000 7f35baa85640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:11.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.160+0000 7f35baa85640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-11T01:31:11.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.160+0000 7f35baa85640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f35b4137d50 addr v1:172.21.15.12:6789/0 2024-09-11T01:31:11.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.160+0000 7f35baa85640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f35b4138130 addr v1:172.21.15.71:6789/0 2024-09-11T01:31:11.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.160+0000 7f35baa85640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b413e290 0x7f35b4140680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.160+0000 7f35baa85640 10 monclient(hunting): picked mon.noname-b con 0x7f35b413e290 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:11.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.160+0000 7f35baa85640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35b409d820 con 0x7f35b4137d50 2024-09-11T01:31:11.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.160+0000 7f35baa85640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35b409ba20 con 0x7f35b4138130 2024-09-11T01:31:11.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.160+0000 7f35baa85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:11.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.160+0000 7f35baa85640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f35b409c650 con 0x7f35b413e290 2024-09-11T01:31:11.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.160+0000 7f35baa85640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:11.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.160+0000 7f35baa85640 10 monclient(hunting): authenticate will time out at 877.603982s 2024-09-11T01:31:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.161+0000 7f35b9022640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b413e290 0x7f35b4140680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.161+0000 7f35b8821640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f35b4137d50 0x7f35b409d110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34860/0 (socket says 172.21.15.12:34860) 2024-09-11T01:31:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.161+0000 7f35b8821640 1 -- 172.21.15.12:0/651672498 learned_addr learned my addr 172.21.15.12:0/651672498 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:31:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.161+0000 7f35b9022640 10 monclient(hunting): get_auth_request con 0x7f35b413e290 auth_method 0 2024-09-11T01:31:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.161+0000 7f35b9022640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.161+0000 7f35b9022640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.161+0000 7f35b9022640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.161+0000 7f35b9022640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.161+0000 7f35b9022640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.161+0000 7f35b9022640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.161+0000 7f35b9022640 10 monclient(hunting): handle_auth_done global_id 4562 payload 274 2024-09-11T01:31:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.161+0000 7f35b9022640 1 -- 172.21.15.12:0/651672498 >> v1:172.21.15.71:6789/0 conn(0x7f35b4138130 legacy=0x7f35b4138580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.161+0000 7f35b9022640 1 -- 172.21.15.12:0/651672498 >> v1:172.21.15.12:6789/0 conn(0x7f35b4137d50 legacy=0x7f35b409d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:11.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.161+0000 7f35b9022640 10 monclient: _finish_hunting 0 2024-09-11T01:31:11.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.161+0000 7f35b9022640 1 monclient: found mon.noname-b 2024-09-11T01:31:11.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.161+0000 7f35b9022640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:11.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.161+0000 7f35b9022640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:31:11.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.161+0000 7f35b9022640 1 -- 172.21.15.12:0/651672498 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35b409c370 con 0x7f35b413e290 2024-09-11T01:31:11.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.164+0000 7f35b9022640 1 -- 172.21.15.12:0/651672498 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b413e290 msgr2=0x7f35b4140680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:11.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.164+0000 7f35b9022640 1 --2- 172.21.15.12:0/651672498 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b413e290 0x7f35b4140680 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f35ac009dc0 tx=0x7f35ac02a510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65b9a819e8b15475 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:11.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.164+0000 7f35b37fe640 1 -- 172.21.15.12:0/651672498 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ac007a30 con 0x7f35b413e290 2024-09-11T01:31:11.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.164+0000 7f35b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:11.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.164+0000 7f35b37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:31:11.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.164+0000 7f35b37fe640 10 monclient: dump: 2024-09-11T01:31:11.174 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:11.174 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:11.175 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.164+0000 7f35b37fe640 10 monclient: _finish_auth 0 2024-09-11T01:31:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.164+0000 7f35b37fe640 10 monclient: _check_auth_tickets 2024-09-11T01:31:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.164+0000 7f35b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.164+0000 7f35baa85640 5 monclient: authenticate success, global_id 4562 2024-09-11T01:31:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.164+0000 7f35baa85640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:31:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.164+0000 7f35b37fe640 1 -- 172.21.15.12:0/651672498 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35ac007bd0 con 0x7f35b413e290 2024-09-11T01:31:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.164+0000 7f35b37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.164+0000 7f35b37fe640 1 -- 172.21.15.12:0/651672498 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ac031400 con 0x7f35b413e290 2024-09-11T01:31:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.164+0000 7f35baa85640 10 monclient: get_monmap_and_config success 2024-09-11T01:31:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.164+0000 7f35baa85640 4 set_mon_vals no callback set 2024-09-11T01:31:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.164+0000 7f35b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.164+0000 7f35b37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.164+0000 7f35b37fe640 10 monclient: dump: 2024-09-11T01:31:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:11.177 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.164+0000 7f35baa85640 10 monclient: shutdown 2024-09-11T01:31:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.164+0000 7f35baa85640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.164+0000 7f35baa85640 1 -- 172.21.15.12:0/651672498 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b413e290 msgr2=0x7f35b4140680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.164+0000 7f35baa85640 1 --2- 172.21.15.12:0/651672498 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b413e290 0x7f35b4140680 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f35ac009dc0 tx=0x7f35ac02a510 comp rx=0 tx=0).stop 2024-09-11T01:31:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.165+0000 7f35baa85640 1 -- 172.21.15.12:0/651672498 shutdown_connections 2024-09-11T01:31:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.165+0000 7f35baa85640 1 --2- 172.21.15.12:0/651672498 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b413e290 0x7f35b4140680 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.165+0000 7f35baa85640 1 -- 172.21.15.12:0/651672498 >> 172.21.15.12:0/651672498 conn(0x7f35b40a7020 msgr2=0x7f35b40a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.165+0000 7f35baa85640 1 -- 172.21.15.12:0/651672498 shutdown_connections 2024-09-11T01:31:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.165+0000 7f35baa85640 1 -- 172.21.15.12:0/651672498 wait complete. 2024-09-11T01:31:11.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.165+0000 7f35baa85640 10 monclient: build_initial_monmap 2024-09-11T01:31:11.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.165+0000 7f35baa85640 10 monclient: monmap: 2024-09-11T01:31:11.178 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:11.178 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:31:11.178 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:31:11.178 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:31:11.178 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:11.178 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.178 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:31:11.178 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:31:11.178 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:31:11.178 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.165+0000 7f35baa85640 1 Processor -- start 2024-09-11T01:31:11.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35baa85640 1 -- start start 2024-09-11T01:31:11.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35baa85640 10 monclient: init 2024-09-11T01:31:11.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35baa85640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:11.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35baa85640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:31:11.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35baa85640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35b4137d50 0x7f35b413c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35baa85640 10 monclient(hunting): picked mon.noname-a con 0x7f35b4137d50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:11.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35baa85640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b4138130 0x7f35b413a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35baa85640 10 monclient(hunting): picked mon.noname-c con 0x7f35b4138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:11.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35baa85640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f35b413e290 0x7f35b4136db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35baa85640 10 monclient(hunting): picked mon.noname-b con 0x7f35b413e290 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:31:11.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35baa85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:11.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35baa85640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f35b4140bc0 con 0x7f35b4137d50 2024-09-11T01:31:11.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35baa85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:11.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35baa85640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f35b40a6da0 con 0x7f35b413e290 2024-09-11T01:31:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35baa85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35baa85640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f35b4140d40 con 0x7f35b4138130 2024-09-11T01:31:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35baa85640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35baa85640 10 monclient(hunting): authenticate will time out at 877.609635s 2024-09-11T01:31:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35b8821640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35b4137d50 0x7f35b413c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35b3fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b4138130 0x7f35b413a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35b3fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b4138130 0x7f35b413a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55970/0 (socket says 172.21.15.12:55970) 2024-09-11T01:31:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35b8821640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35b4137d50 0x7f35b413c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45810/0 (socket says 172.21.15.12:45810) 2024-09-11T01:31:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35b3fff640 1 -- 172.21.15.12:0/2425099601 learned_addr learned my addr 172.21.15.12:0/2425099601 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35b3fff640 10 monclient(hunting): get_auth_request con 0x7f35b4138130 auth_method 0 2024-09-11T01:31:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35b8821640 10 monclient(hunting): get_auth_request con 0x7f35b4137d50 auth_method 0 2024-09-11T01:31:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35b8821640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35b8821640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35b8821640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35b9022640 1 --2- 172.21.15.12:0/2425099601 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f35b413e290 0x7f35b4136db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35b8821640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:11.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35b8821640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:11.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35b8821640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:11.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35b9022640 10 monclient(hunting): get_auth_request con 0x7f35b413e290 auth_method 0 2024-09-11T01:31:11.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35b9022640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:11.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35b9022640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:11.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35b9022640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:11.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35b9022640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:11.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.166+0000 7f35b9022640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:11.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.167+0000 7f35b9022640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:11.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.167+0000 7f35b3fff640 10 monclient(hunting): handle_auth_done global_id 4565 payload 824 2024-09-11T01:31:11.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.167+0000 7f35b3fff640 1 -- 172.21.15.12:0/2425099601 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f35b413e290 msgr2=0x7f35b4136db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:11.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.167+0000 7f35b3fff640 1 --2- 172.21.15.12:0/2425099601 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f35b413e290 0x7f35b4136db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.167+0000 7f35b3fff640 1 -- 172.21.15.12:0/2425099601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35b4137d50 msgr2=0x7f35b413c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:11.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.167+0000 7f35b3fff640 1 --2- 172.21.15.12:0/2425099601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35b4137d50 0x7f35b413c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.167+0000 7f35b3fff640 10 monclient: _finish_hunting 0 2024-09-11T01:31:11.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.167+0000 7f35b3fff640 1 monclient: found mon.noname-c 2024-09-11T01:31:11.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.167+0000 7f35b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:11.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.167+0000 7f35b3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:31:11.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.167+0000 7f35b3fff640 1 -- 172.21.15.12:0/2425099601 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35b413cfe0 con 0x7f35b4138130 2024-09-11T01:31:11.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.167+0000 7f35b8821640 1 --2- 172.21.15.12:0/2425099601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35b4137d50 0x7f35b413c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:31:11.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.169+0000 7f35b3fff640 1 -- 172.21.15.12:0/2425099601 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b4138130 msgr2=0x7f35b413a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:11.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.169+0000 7f35b3fff640 1 --2- 172.21.15.12:0/2425099601 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b4138130 0x7f35b413a510 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f35a000c430 tx=0x7f35a000c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:11.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.169+0000 7f35b17fa640 1 -- 172.21.15.12:0/2425099601 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a00119a0 con 0x7f35b4138130 2024-09-11T01:31:11.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.169+0000 7f35b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:11.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.169+0000 7f35b17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:31:11.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.169+0000 7f35b17fa640 10 monclient: dump: 2024-09-11T01:31:11.183 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:11.183 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:11.184 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.184 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.184 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:11.184 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.184 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:11.184 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:11.184 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:11.184 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.169+0000 7f35b17fa640 10 monclient: _finish_auth 0 2024-09-11T01:31:11.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.169+0000 7f35b17fa640 10 monclient: _check_auth_tickets 2024-09-11T01:31:11.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.169+0000 7f35b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:11.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.169+0000 7f35b17fa640 1 -- 172.21.15.12:0/2425099601 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35a0017070 con 0x7f35b4138130 2024-09-11T01:31:11.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.169+0000 7f35baa85640 5 monclient: authenticate success, global_id 4565 2024-09-11T01:31:11.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.169+0000 7f35baa85640 10 monclient: _renew_subs 2024-09-11T01:31:11.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.169+0000 7f35baa85640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:11.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.169+0000 7f35baa85640 1 -- 172.21.15.12:0/2425099601 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35b413b4c0 con 0x7f35b4138130 2024-09-11T01:31:11.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.169+0000 7f35b17fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:11.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.169+0000 7f35b17fa640 1 -- 172.21.15.12:0/2425099601 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a0011d80 con 0x7f35b4138130 2024-09-11T01:31:11.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.169+0000 7f35b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:11.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.169+0000 7f35b17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:11.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.169+0000 7f35b17fa640 10 monclient: dump: 2024-09-11T01:31:11.185 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:11.185 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:11.185 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.186 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.186 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:11.186 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.186 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:11.186 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:11.186 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:11.186 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.169+0000 7f35b2ffd640 4 set_mon_vals no callback set 2024-09-11T01:31:11.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.169+0000 7f35baa85640 10 monclient: _renew_subs 2024-09-11T01:31:11.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.169+0000 7f35baa85640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:11.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.169+0000 7f35baa85640 1 -- 172.21.15.12:0/2425099601 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f35b413bb40 con 0x7f35b4138130 2024-09-11T01:31:11.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.170+0000 7f35baa85640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:11.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.170+0000 7f35baa85640 1 --2- 172.21.15.12:0/2425099601 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b41ccea0 0x7f35b41cd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.170+0000 7f35baa85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:11.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.170+0000 7f35baa85640 1 -- 172.21.15.12:0/2425099601 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f35b406a390 con 0x7f35b41ccea0 2024-09-11T01:31:11.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.170+0000 7f35b8821640 1 --2- 172.21.15.12:0/2425099601 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b41ccea0 0x7f35b41cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:11.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.170+0000 7f35b8821640 10 monclient: get_auth_request con 0x7f35b41ccea0 auth_method 0 2024-09-11T01:31:11.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.170+0000 7f35b8821640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:11.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.170+0000 7f35b8821640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:11.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.170+0000 7f35b8821640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:11.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.170+0000 7f35b8821640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:11.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.170+0000 7f35b8821640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:11.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.170+0000 7f35b8821640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:11.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.170+0000 7f35b17fa640 1 -- 172.21.15.12:0/2425099601 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f35a001c460 con 0x7f35b4138130 2024-09-11T01:31:11.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.170+0000 7f35b8821640 10 monclient(hunting): handle_auth_done global_id 4568 payload 824 2024-09-11T01:31:11.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.170+0000 7f35b8821640 1 -- 172.21.15.12:0/2425099601 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f35b409d190 con 0x7f35b41ccea0 2024-09-11T01:31:11.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.170+0000 7f35b17fa640 1 --2- 172.21.15.12:0/2425099601 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f359803e210 0x7f35980406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.170+0000 7f35b17fa640 1 -- 172.21.15.12:0/2425099601 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(70..70 src has 1..70) ==== 4755+0+0 (secure 0 0 0) 0x7f35a0052be0 con 0x7f35b4138130 2024-09-11T01:31:11.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.171+0000 7f35b9022640 1 --2- 172.21.15.12:0/2425099601 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f359803e210 0x7f35980406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:11.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.171+0000 7f35b9022640 10 monclient: get_auth_request con 0x7f359803e210 auth_method 0 2024-09-11T01:31:11.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.172+0000 7f35b8821640 1 -- 172.21.15.12:0/2425099601 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b41ccea0 msgr2=0x7f35b41cd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:11.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.172+0000 7f35b8821640 1 --2- 172.21.15.12:0/2425099601 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b41ccea0 0x7f35b41cd280 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f35a8007270 tx=0x7f35a8007730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:11.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.173+0000 7f35b17fa640 1 -- 172.21.15.12:0/2425099601 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a8017020 con 0x7f35b41ccea0 2024-09-11T01:31:11.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.173+0000 7f35b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:11.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.173+0000 7f35b17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:11.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.173+0000 7f35b17fa640 10 monclient: dump: 2024-09-11T01:31:11.190 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:11.190 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:11.191 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.191 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.191 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:11.191 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.191 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:11.191 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:11.191 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:11.191 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.173+0000 7f35b17fa640 1 -- 172.21.15.12:0/2425099601 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f35a800fd00 con 0x7f35b41ccea0 2024-09-11T01:31:11.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.173+0000 7f35b17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:11.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.173+0000 7f35b17fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:31:11.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.173+0000 7f35b17fa640 1 -- 172.21.15.12:0/2425099601 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b41ccea0 msgr2=0x7f35b41cd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:11.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.173+0000 7f35b17fa640 1 --2- 172.21.15.12:0/2425099601 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b41ccea0 0x7f35b41cd280 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f35a8007270 tx=0x7f35a8007730 comp rx=0 tx=0).stop 2024-09-11T01:31:11.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.173+0000 7f35b17fa640 1 -- 172.21.15.12:0/2425099601 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b41ccea0 msgr2=0x7f35b41cd280 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:31:11.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.173+0000 7f35b17fa640 1 --2- 172.21.15.12:0/2425099601 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b41ccea0 0x7f35b41cd280 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f35a8007270 tx=0x7f35a8007730 comp rx=0 tx=0).stop 2024-09-11T01:31:11.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.173+0000 7f35b9022640 1 --2- 172.21.15.12:0/2425099601 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f359803e210 0x7f35980406d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f35b406d4f0 tx=0x7f35ac02f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:11.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.179+0000 7f35baa85640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:31:11.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.179+0000 7f35baa85640 1 --2- 172.21.15.12:0/2425099601 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b41cd7c0 0x7f35b40a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.179+0000 7f35baa85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:11.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.179+0000 7f35baa85640 1 -- 172.21.15.12:0/2425099601 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f35b4074a40 con 0x7f35b41cd7c0 2024-09-11T01:31:11.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.179+0000 7f35b8821640 1 --2- 172.21.15.12:0/2425099601 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b41cd7c0 0x7f35b40a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:11.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.179+0000 7f35b8821640 10 monclient: get_auth_request con 0x7f35b41cd7c0 auth_method 0 2024-09-11T01:31:11.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.179+0000 7f35b8821640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:11.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.179+0000 7f35b8821640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:11.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.179+0000 7f35b8821640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:11.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.179+0000 7f35b8821640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:11.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.179+0000 7f35b8821640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:11.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.179+0000 7f35b8821640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:11.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.179+0000 7f35b8821640 10 monclient(hunting): handle_auth_done global_id 4571 payload 824 2024-09-11T01:31:11.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.179+0000 7f35b8821640 1 -- 172.21.15.12:0/2425099601 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f35b413d5b0 con 0x7f35b41cd7c0 2024-09-11T01:31:11.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.182+0000 7f35b8821640 1 -- 172.21.15.12:0/2425099601 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b41cd7c0 msgr2=0x7f35b40a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:11.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.182+0000 7f35b8821640 1 --2- 172.21.15.12:0/2425099601 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b41cd7c0 0x7f35b40a1920 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f35a8007270 tx=0x7f35a8004940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:11.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.182+0000 7f35b17fa640 1 -- 172.21.15.12:0/2425099601 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a8013880 con 0x7f35b41cd7c0 2024-09-11T01:31:11.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.182+0000 7f35b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:11.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.182+0000 7f35b17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:11.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.182+0000 7f35b17fa640 10 monclient: dump: 2024-09-11T01:31:11.193 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:11.193 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:11.193 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.193 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.194 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:11.194 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.194 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:11.194 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:11.194 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:11.194 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.182+0000 7f35b17fa640 1 -- 172.21.15.12:0/2425099601 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7f35a8013880 con 0x7f35b41cd7c0 2024-09-11T01:31:11.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.182+0000 7f35b17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:31:11.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.182+0000 7f35b17fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:31:11.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.182+0000 7f35b17fa640 1 -- 172.21.15.12:0/2425099601 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b41cd7c0 msgr2=0x7f35b40a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:11.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.182+0000 7f35b17fa640 1 --2- 172.21.15.12:0/2425099601 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b41cd7c0 0x7f35b40a1920 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f35a8007270 tx=0x7f35a8004940 comp rx=0 tx=0).stop 2024-09-11T01:31:11.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.182+0000 7f35b17fa640 1 -- 172.21.15.12:0/2425099601 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b41cd7c0 msgr2=0x7f35b40a1920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:31:11.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.182+0000 7f35b17fa640 1 --2- 172.21.15.12:0/2425099601 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b41cd7c0 0x7f35b40a1920 secure :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f35a8007270 tx=0x7f35a8004940 comp rx=0 tx=0).stop 2024-09-11T01:31:11.195 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:31:11.195 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:31:11.195 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:31:11.195 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:31:11.195 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 22, 2024-09-11T01:31:11.195 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 133871, 2024-09-11T01:31:11.195 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:31:11.195 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:31:11.195 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:31:11.195 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:31:11.196 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:11.196 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:31:11.196 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:31:11.196 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:31:11.196 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:31:11.196 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:11.196 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:11.196 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:11.196 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:11.196 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:11.196 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:11.196 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:11.196 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:11.197 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:11.197 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:11.197 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:11.197 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:11.197 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:31:11.197 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:31:11.197 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:11.197 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:11.197 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:11.197 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:11.197 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:11.197 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:11.197 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:11.198 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:11.198 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:11.198 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:11.198 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:11.198 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:11.198 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.198 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:31:11.198 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:31:11.198 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:31:11.198 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:31:11.198 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:31:11.198 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:31:11.198 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:31:11.199 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:31:11.199 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:31:11.199 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:31:11.199 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:31:11.199 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:31:11.199 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:31:11.199 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:31:11.199 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:31:11.199 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:31:11.199 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:31:11.199 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:11.199 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:11.199 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:11.199 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:11.200 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:11.200 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:11.200 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:11.200 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:11.200 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:11.200 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:11.200 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:11.200 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:11.200 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:31:11.200 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.200 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:31:11.200 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.200 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:31:11.200 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:31:11.201 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:11.201 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:11.201 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.201 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:11.201 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:31:11.201 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:11.201 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.201 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.201 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:11.201 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:31:11.201 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:11.201 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:11.201 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:11.201 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.202 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:31:11.202 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:31:11.202 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:11.202 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:11.202 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:11.202 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.202 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.202 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:31:11.202 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:31:11.202 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:11.202 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:11.202 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.202 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:11.202 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:31:11.203 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:11.203 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.203 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.203 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:11.203 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:31:11.203 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:11.203 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:11.203 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:11.203 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.203 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:31:11.203 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:31:11.203 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:11.203 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:11.203 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:11.204 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.204 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.204 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:31:11.204 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:31:11.204 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:11.204 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:11.204 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.204 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:11.204 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:31:11.204 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:11.204 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.204 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.204 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:11.204 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:31:11.205 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:11.205 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:11.205 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:11.205 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.205 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:31:11.205 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:31:11.205 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:11.205 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:11.205 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:11.205 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:11.205 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:11.205 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.205 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:31:11.206 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:31:11.206 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.206 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:11.206 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:11.206 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:31:11.206 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:11.206 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:11.206 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:31:11.206 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.206 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:11.206 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:11.206 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:31:11.206 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:11.207 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:11.207 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:31:11.207 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.207 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:11.207 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:11.207 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:31:11.207 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:11.207 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:11.207 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.207 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:31:11.207 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:31:11.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.182+0000 7f35baa85640 1 -- 172.21.15.12:0/2425099601 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f359803e210 msgr2=0x7f35980406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:11.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.182+0000 7f35baa85640 1 --2- 172.21.15.12:0/2425099601 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f359803e210 0x7f35980406d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f35b406d4f0 tx=0x7f35ac02f000 comp rx=0 tx=0).stop 2024-09-11T01:31:11.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.182+0000 7f35baa85640 10 monclient: shutdown 2024-09-11T01:31:11.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.182+0000 7f35baa85640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:11.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.182+0000 7f35baa85640 1 -- 172.21.15.12:0/2425099601 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b4138130 msgr2=0x7f35b413a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:11.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.182+0000 7f35baa85640 1 --2- 172.21.15.12:0/2425099601 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f35b4138130 0x7f35b413a510 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f35a000c430 tx=0x7f35a000c8f0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.182+0000 7f35b9022640 1 -- 172.21.15.12:0/2425099601 reap_dead start 2024-09-11T01:31:11.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.182+0000 7f35b9022640 1 -- 172.21.15.12:0/2425099601 reap_dead start 2024-09-11T01:31:11.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.182+0000 7f35baa85640 1 -- 172.21.15.12:0/2425099601 shutdown_connections 2024-09-11T01:31:11.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.182+0000 7f35baa85640 1 -- 172.21.15.12:0/2425099601 >> 172.21.15.12:0/2425099601 conn(0x7f35b40a7020 msgr2=0x7f35b413f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:11.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.182+0000 7f35baa85640 1 -- 172.21.15.12:0/2425099601 shutdown_connections 2024-09-11T01:31:11.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.182+0000 7f35baa85640 1 -- 172.21.15.12:0/2425099601 wait complete. 2024-09-11T01:31:11.210 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:31:11.210 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:31:11.210 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:31:11.231 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 886454 2024-09-11T01:31:11.263 INFO:tasks.rados.rados.0.smithi071.stdout:247: finishing write tid 4 to smithi07134777-30 2024-09-11T01:31:11.263 INFO:tasks.rados.rados.0.smithi071.stdout:247: finishing write tid 5 to smithi07134777-30 2024-09-11T01:31:11.263 INFO:tasks.rados.rados.0.smithi071.stdout:247: oid 30 updating version 0 to 73 2024-09-11T01:31:11.263 INFO:tasks.rados.rados.0.smithi071.stdout:247: oid 30 updating version 73 to 74 2024-09-11T01:31:11.263 INFO:tasks.rados.rados.0.smithi071.stdout:247: oid 30 version 74 is already newer than 71 2024-09-11T01:31:11.264 INFO:tasks.rados.rados.0.smithi071.stdout:247: oid 30 version 74 is already newer than 72 2024-09-11T01:31:11.264 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 74 (ObjNum 107 snap 25 seq_num 107) dirty exists 2024-09-11T01:31:11.264 INFO:tasks.rados.rados.0.smithi071.stdout:247: left oid 30 (ObjNum 107 snap 25 seq_num 107) 2024-09-11T01:31:11.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.309+0000 7f29eb332640 10 monclient: get_monmap_and_config 2024-09-11T01:31:11.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.309+0000 7f29eb332640 10 monclient: build_initial_monmap 2024-09-11T01:31:11.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.309+0000 7f29eb332640 10 monclient: monmap: 2024-09-11T01:31:11.312 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:11.312 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:31:11.311613+0000 2024-09-11T01:31:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:31:11.311613+0000 2024-09-11T01:31:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:31:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:31:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:31:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:31:11.313 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:31:11.313 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.310+0000 7f29eb332640 1 Processor -- start 2024-09-11T01:31:11.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.310+0000 7f29eb332640 1 -- start start 2024-09-11T01:31:11.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.310+0000 7f29eb332640 10 monclient: init 2024-09-11T01:31:11.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29eb332640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:11.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29eb332640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-11T01:31:11.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29eb332640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f29e4131710 0x7f29e4131af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29eb332640 10 monclient(hunting): picked mon.noname-a con 0x7f29e4131710 addr v2:172.21.15.12:3300/0 2024-09-11T01:31:11.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29eb332640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f29e4132030 addr v1:172.21.15.71:6789/0 2024-09-11T01:31:11.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29eb332640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f29e40a0d70 0x7f29e40a2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29eb332640 10 monclient(hunting): picked mon.noname-b con 0x7f29e40a0d70 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:11.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29eb332640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29e4074a70 con 0x7f29e4132030 2024-09-11T01:31:11.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29eb332640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:11.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29eb332640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f29e4073ab0 con 0x7f29e4131710 2024-09-11T01:31:11.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29eb332640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:11.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29eb332640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f29e4073d90 con 0x7f29e40a0d70 2024-09-11T01:31:11.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29eb332640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:11.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29eb332640 10 monclient(hunting): authenticate will time out at 877.754435s 2024-09-11T01:31:11.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e90ce640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f29e4131710 0x7f29e4131af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:11.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e90ce640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f29e4131710 0x7f29e4131af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45820/0 (socket says 172.21.15.12:45820) 2024-09-11T01:31:11.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e90ce640 1 -- 172.21.15.12:0/3105721115 learned_addr learned my addr 172.21.15.12:0/3105721115 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:11.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e90ce640 10 monclient(hunting): get_auth_request con 0x7f29e4131710 auth_method 0 2024-09-11T01:31:11.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e90ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:11.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e90ce640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:11.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e90ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:11.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e98cf640 1 --2- 172.21.15.12:0/3105721115 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f29e40a0d70 0x7f29e40a2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:11.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e98cf640 10 monclient(hunting): get_auth_request con 0x7f29e40a0d70 auth_method 0 2024-09-11T01:31:11.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e98cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:11.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e98cf640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:11.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e98cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:11.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e90ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:11.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e90ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:11.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e90ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:11.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e98cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:11.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e98cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:11.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e98cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:11.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e90ce640 10 monclient(hunting): handle_auth_done global_id 24150 payload 274 2024-09-11T01:31:11.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e90ce640 1 -- 172.21.15.12:0/3105721115 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f29e40a0d70 msgr2=0x7f29e40a2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:11.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e90ce640 1 --2- 172.21.15.12:0/3105721115 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f29e40a0d70 0x7f29e40a2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e90ce640 1 -- 172.21.15.12:0/3105721115 >> v1:172.21.15.71:6789/0 conn(0x7f29e4132030 legacy=0x7f29e40a25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:11.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e90ce640 10 monclient: _finish_hunting 0 2024-09-11T01:31:11.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e90ce640 1 monclient: found mon.noname-a 2024-09-11T01:31:11.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e90ce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:11.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e90ce640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:31:11.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.311+0000 7f29e90ce640 1 -- 172.21.15.12:0/3105721115 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29e4074070 con 0x7f29e4131710 2024-09-11T01:31:11.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.314+0000 7f29e90ce640 1 -- 172.21.15.12:0/3105721115 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29e4131710 msgr2=0x7f29e4131af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:11.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.314+0000 7f29e90ce640 1 --2- 172.21.15.12:0/3105721115 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29e4131710 0x7f29e4131af0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f29e0009870 tx=0x7f29e002a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10b3a01cbedb722b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:11.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.314+0000 7f29dbfff640 1 -- 172.21.15.12:0/3105721115 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e0037070 con 0x7f29e4131710 2024-09-11T01:31:11.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.314+0000 7f29dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:11.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.314+0000 7f29dbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:31:11.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.314+0000 7f29dbfff640 10 monclient: dump: 2024-09-11T01:31:11.325 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:11.325 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:11.325 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.326 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.326 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:11.326 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.326 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:11.326 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:11.326 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:11.326 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.314+0000 7f29dbfff640 10 monclient: _finish_auth 0 2024-09-11T01:31:11.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.314+0000 7f29dbfff640 10 monclient: _check_auth_tickets 2024-09-11T01:31:11.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.314+0000 7f29dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:11.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.314+0000 7f29dbfff640 1 -- 172.21.15.12:0/3105721115 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29e0032640 con 0x7f29e4131710 2024-09-11T01:31:11.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.314+0000 7f29dbfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:11.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.314+0000 7f29dbfff640 1 -- 172.21.15.12:0/3105721115 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e0032a10 con 0x7f29e4131710 2024-09-11T01:31:11.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.314+0000 7f29dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.314+0000 7f29dbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.314+0000 7f29dbfff640 10 monclient: dump: 2024-09-11T01:31:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:11.327 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.314+0000 7f29eb332640 5 monclient: authenticate success, global_id 24150 2024-09-11T01:31:11.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.314+0000 7f29eb332640 10 monclient: get_monmap_and_config success 2024-09-11T01:31:11.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.314+0000 7f29eb332640 4 set_mon_vals no callback set 2024-09-11T01:31:11.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.315+0000 7f29eb332640 10 monclient: shutdown 2024-09-11T01:31:11.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.315+0000 7f29eb332640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:11.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.315+0000 7f29eb332640 1 -- 172.21.15.12:0/3105721115 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29e4131710 msgr2=0x7f29e4131af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:11.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.315+0000 7f29eb332640 1 --2- 172.21.15.12:0/3105721115 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29e4131710 0x7f29e4131af0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f29e0009870 tx=0x7f29e002a130 comp rx=0 tx=0).stop 2024-09-11T01:31:11.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.315+0000 7f29eb332640 1 -- 172.21.15.12:0/3105721115 shutdown_connections 2024-09-11T01:31:11.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.315+0000 7f29eb332640 1 --2- 172.21.15.12:0/3105721115 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f29e40a0d70 0x7f29e40a2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.315+0000 7f29eb332640 1 --2- 172.21.15.12:0/3105721115 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29e4131710 0x7f29e4131af0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.315+0000 7f29eb332640 1 -- 172.21.15.12:0/3105721115 >> 172.21.15.12:0/3105721115 conn(0x7f29e40a36c0 msgr2=0x7f29e40a3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:11.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.315+0000 7f29eb332640 1 -- 172.21.15.12:0/3105721115 shutdown_connections 2024-09-11T01:31:11.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.315+0000 7f29eb332640 1 -- 172.21.15.12:0/3105721115 wait complete. 2024-09-11T01:31:11.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.316+0000 7f29eb332640 10 monclient: build_initial_monmap 2024-09-11T01:31:11.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.316+0000 7f29eb332640 10 monclient: monmap: 2024-09-11T01:31:11.329 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:11.329 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:31:11.329 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:31:11.329 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:31:11.329 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:11.329 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.329 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:31:11.329 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:31:11.329 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:31:11.329 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.316+0000 7f29eb332640 1 Processor -- start 2024-09-11T01:31:11.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.316+0000 7f29eb332640 1 -- start start 2024-09-11T01:31:11.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.316+0000 7f29eb332640 10 monclient: init 2024-09-11T01:31:11.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.316+0000 7f29eb332640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:11.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.316+0000 7f29eb332640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:31:11.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.316+0000 7f29eb332640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e40a0d70 0x7f29e41cad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.316+0000 7f29eb332640 10 monclient(hunting): picked mon.noname-b con 0x7f29e40a0d70 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:31:11.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.316+0000 7f29eb332640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29e4131710 0x7f29e413c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.316+0000 7f29eb332640 10 monclient(hunting): picked mon.noname-a con 0x7f29e4131710 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:11.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.316+0000 7f29eb332640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f29e4132030 0x7f29e413cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.316+0000 7f29eb332640 10 monclient(hunting): picked mon.noname-c con 0x7f29e4132030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:11.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.316+0000 7f29eb332640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:11.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.316+0000 7f29eb332640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f29e4073350 con 0x7f29e4131710 2024-09-11T01:31:11.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.316+0000 7f29eb332640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:11.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.316+0000 7f29eb332640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f29e40731d0 con 0x7f29e40a0d70 2024-09-11T01:31:11.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.316+0000 7f29eb332640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.316+0000 7f29eb332640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f29e40a3220 con 0x7f29e4132030 2024-09-11T01:31:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29eb332640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29eb332640 10 monclient(hunting): authenticate will time out at 877.760020s 2024-09-11T01:31:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e88cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29e4131710 0x7f29e413c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e98cf640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f29e4132030 0x7f29e413cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e98cf640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f29e4132030 0x7f29e413cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55998/0 (socket says 172.21.15.12:55998) 2024-09-11T01:31:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e88cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29e4131710 0x7f29e413c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45830/0 (socket says 172.21.15.12:45830) 2024-09-11T01:31:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e98cf640 1 -- 172.21.15.12:0/887797678 learned_addr learned my addr 172.21.15.12:0/887797678 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e98cf640 10 monclient(hunting): get_auth_request con 0x7f29e4132030 auth_method 0 2024-09-11T01:31:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e98cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e98cf640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e98cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:11.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e88cd640 10 monclient(hunting): get_auth_request con 0x7f29e4131710 auth_method 0 2024-09-11T01:31:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e88cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e88cd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e88cd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e98cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e98cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e98cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e88cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e88cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e88cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e90ce640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e40a0d70 0x7f29e41cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e90ce640 10 monclient(hunting): get_auth_request con 0x7f29e40a0d70 auth_method 0 2024-09-11T01:31:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e90ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e90ce640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:11.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e90ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:11.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e98cf640 10 monclient(hunting): handle_auth_done global_id 4577 payload 824 2024-09-11T01:31:11.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e98cf640 1 -- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e40a0d70 msgr2=0x7f29e41cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:11.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e98cf640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e40a0d70 0x7f29e41cad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e98cf640 1 -- 172.21.15.12:0/887797678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29e4131710 msgr2=0x7f29e413c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:11.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e98cf640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29e4131710 0x7f29e413c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e98cf640 10 monclient: _finish_hunting 0 2024-09-11T01:31:11.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e98cf640 1 monclient: found mon.noname-c 2024-09-11T01:31:11.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e98cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:11.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e98cf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:31:11.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e98cf640 1 -- 172.21.15.12:0/887797678 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29e413b2e0 con 0x7f29e4132030 2024-09-11T01:31:11.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e90ce640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e40a0d70 0x7f29e41cad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:31:11.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.317+0000 7f29e88cd640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29e4131710 0x7f29e413c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:31:11.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.319+0000 7f29e98cf640 1 -- 172.21.15.12:0/887797678 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f29e4132030 msgr2=0x7f29e413cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:11.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.319+0000 7f29e98cf640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f29e4132030 0x7f29e413cd50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f29dc00efc0 tx=0x7f29dc00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:11.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.319+0000 7f29d9ffb640 1 -- 172.21.15.12:0/887797678 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29dc018020 con 0x7f29e4132030 2024-09-11T01:31:11.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.319+0000 7f29d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:11.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.319+0000 7f29d9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:31:11.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.319+0000 7f29d9ffb640 10 monclient: dump: 2024-09-11T01:31:11.334 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:11.334 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:11.334 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.334 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.334 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:11.334 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.335 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:11.335 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:11.335 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:11.335 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.319+0000 7f29d9ffb640 10 monclient: _finish_auth 0 2024-09-11T01:31:11.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.319+0000 7f29d9ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:31:11.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.319+0000 7f29d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:11.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.319+0000 7f29d9ffb640 1 -- 172.21.15.12:0/887797678 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29dc014680 con 0x7f29e4132030 2024-09-11T01:31:11.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.319+0000 7f29eb332640 5 monclient: authenticate success, global_id 4577 2024-09-11T01:31:11.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.319+0000 7f29eb332640 10 monclient: _renew_subs 2024-09-11T01:31:11.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.319+0000 7f29eb332640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:11.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.319+0000 7f29eb332640 1 -- 172.21.15.12:0/887797678 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29e40a5c30 con 0x7f29e4132030 2024-09-11T01:31:11.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.319+0000 7f29d9ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:11.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.319+0000 7f29d9ffb640 1 -- 172.21.15.12:0/887797678 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29dc014a70 con 0x7f29e4132030 2024-09-11T01:31:11.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.319+0000 7f29d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:11.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.319+0000 7f29d9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:11.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.319+0000 7f29d9ffb640 10 monclient: dump: 2024-09-11T01:31:11.336 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:11.336 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:11.336 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.336 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.336 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:11.336 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.336 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:11.336 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:11.336 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:11.337 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.319+0000 7f29daffd640 4 set_mon_vals no callback set 2024-09-11T01:31:11.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.320+0000 7f29eb332640 10 monclient: _renew_subs 2024-09-11T01:31:11.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.320+0000 7f29eb332640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:11.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.320+0000 7f29eb332640 1 -- 172.21.15.12:0/887797678 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f29e4139ae0 con 0x7f29e4132030 2024-09-11T01:31:11.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.320+0000 7f29eb332640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:11.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.320+0000 7f29eb332640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e41374c0 0x7f29e41378a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.320+0000 7f29eb332640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:11.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.320+0000 7f29eb332640 1 -- 172.21.15.12:0/887797678 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f29e413d290 con 0x7f29e41374c0 2024-09-11T01:31:11.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.320+0000 7f29d9ffb640 1 -- 172.21.15.12:0/887797678 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f29dc014c10 con 0x7f29e4132030 2024-09-11T01:31:11.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.321+0000 7f29e90ce640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e41374c0 0x7f29e41378a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:11.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.321+0000 7f29e90ce640 10 monclient: get_auth_request con 0x7f29e41374c0 auth_method 0 2024-09-11T01:31:11.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.321+0000 7f29e90ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:11.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.321+0000 7f29e90ce640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:11.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.321+0000 7f29e90ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:11.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.321+0000 7f29e90ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:11.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.321+0000 7f29e90ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:11.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.321+0000 7f29e90ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:11.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.321+0000 7f29d9ffb640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f29c803e210 0x7f29c80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.321+0000 7f29d9ffb640 1 -- 172.21.15.12:0/887797678 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(70..70 src has 1..70) ==== 4755+0+0 (secure 0 0 0) 0x7f29dc054030 con 0x7f29e4132030 2024-09-11T01:31:11.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.321+0000 7f29e90ce640 10 monclient(hunting): handle_auth_done global_id 4468 payload 824 2024-09-11T01:31:11.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.321+0000 7f29e90ce640 1 -- 172.21.15.12:0/887797678 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f29e4131af0 con 0x7f29e41374c0 2024-09-11T01:31:11.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.321+0000 7f29e88cd640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f29c803e210 0x7f29c80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:11.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.321+0000 7f29e88cd640 10 monclient: get_auth_request con 0x7f29c803e210 auth_method 0 2024-09-11T01:31:11.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.323+0000 7f29e90ce640 1 -- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e41374c0 msgr2=0x7f29e41378a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:11.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.323+0000 7f29e90ce640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e41374c0 0x7f29e41378a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f29e0034000 tx=0x7f29e00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:11.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.323+0000 7f29e88cd640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f29c803e210 0x7f29c80406d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f29e4070f10 tx=0x7f29d4007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:11.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.324+0000 7f29d9ffb640 1 -- 172.21.15.12:0/887797678 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e0037040 con 0x7f29e41374c0 2024-09-11T01:31:11.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.324+0000 7f29d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:11.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.324+0000 7f29d9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:31:11.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.324+0000 7f29d9ffb640 10 monclient: dump: 2024-09-11T01:31:11.343 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:11.343 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:11.343 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.343 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.343 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:11.343 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.343 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:11.343 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:11.343 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:11.343 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.324+0000 7f29e90ce640 1 -- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e41374c0 msgr2=0x7f29e41378a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:31:11.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.324+0000 7f29e90ce640 1 -- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e41374c0 msgr2=0x7f29e41378a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:31:11.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.324+0000 7f29e90ce640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e41374c0 0x7f29e41378a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f29e0034000 tx=0x7f29e00046c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:31:11.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.324+0000 7f29e90ce640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e41374c0 0x7f29e41378a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f29e0034000 tx=0x7f29e00046c0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.324+0000 7f29d9ffb640 1 -- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e41374c0 msgr2=0x7f29e41378a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:31:11.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.324+0000 7f29d9ffb640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e41374c0 0x7f29e41378a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.324+0000 7f29d9ffb640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29c80443b0 0x7f29c80467a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.324+0000 7f29d9ffb640 1 -- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e41374c0 msgr2=0x7f29e41378a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:31:11.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.324+0000 7f29d9ffb640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e41374c0 0x7f29e41378a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.324+0000 7f29d9ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:11.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.324+0000 7f29d9ffb640 1 -- 172.21.15.12:0/887797678 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f29c8046de0 con 0x7f29c80443b0 2024-09-11T01:31:11.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.324+0000 7f29e90ce640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29c80443b0 0x7f29c80467a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:11.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.324+0000 7f29e90ce640 10 monclient: get_auth_request con 0x7f29c80443b0 auth_method 0 2024-09-11T01:31:11.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.324+0000 7f29e90ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:11.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.324+0000 7f29e90ce640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:11.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.324+0000 7f29e90ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:11.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.324+0000 7f29e90ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:11.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.324+0000 7f29e90ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:11.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.324+0000 7f29e90ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:11.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.324+0000 7f29e90ce640 10 monclient(hunting): handle_auth_done global_id 4471 payload 824 2024-09-11T01:31:11.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.324+0000 7f29e90ce640 1 -- 172.21.15.12:0/887797678 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f29dc01bec0 con 0x7f29c80443b0 2024-09-11T01:31:11.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.327+0000 7f29e90ce640 1 -- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29c80443b0 msgr2=0x7f29c80467a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:11.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.327+0000 7f29e90ce640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29c80443b0 0x7f29c80467a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f29e0000c00 tx=0x7f29e0009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:11.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.327+0000 7f29d9ffb640 1 -- 172.21.15.12:0/887797678 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e004c070 con 0x7f29c80443b0 2024-09-11T01:31:11.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.327+0000 7f29d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:11.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.327+0000 7f29d9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:31:11.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.327+0000 7f29d9ffb640 10 monclient: dump: 2024-09-11T01:31:11.346 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:11.346 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:11.346 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.346 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.346 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:11.346 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.346 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:11.346 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:11.346 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:11.346 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.327+0000 7f29d9ffb640 1 -- 172.21.15.12:0/887797678 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f29e003b3d0 con 0x7f29c80443b0 2024-09-11T01:31:11.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.327+0000 7f29d9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:11.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.327+0000 7f29d9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:31:11.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.327+0000 7f29d9ffb640 1 -- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29c80443b0 msgr2=0x7f29c80467a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:11.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.327+0000 7f29d9ffb640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29c80443b0 0x7f29c80467a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f29e0000c00 tx=0x7f29e0009240 comp rx=0 tx=0).stop 2024-09-11T01:31:11.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.327+0000 7f29d9ffb640 1 -- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29c80443b0 msgr2=0x7f29c80467a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:31:11.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.327+0000 7f29d9ffb640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29c80443b0 0x7f29c80467a0 secure :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f29e0000c00 tx=0x7f29e0009240 comp rx=0 tx=0).stop 2024-09-11T01:31:11.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.333+0000 7f29eb332640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:31:11.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.333+0000 7f29eb332640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e4137de0 0x7f29e409e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.334+0000 7f29eb332640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:11.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.334+0000 7f29eb332640 1 -- 172.21.15.12:0/887797678 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f29e40a0640 con 0x7f29e4137de0 2024-09-11T01:31:11.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.334+0000 7f29e90ce640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e4137de0 0x7f29e409e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:11.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.334+0000 7f29e90ce640 10 monclient: get_auth_request con 0x7f29e4137de0 auth_method 0 2024-09-11T01:31:11.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.334+0000 7f29e90ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:11.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.334+0000 7f29e90ce640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:11.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.334+0000 7f29e90ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:11.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.334+0000 7f29e90ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:11.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.334+0000 7f29e90ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:11.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.334+0000 7f29e90ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:11.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.334+0000 7f29e90ce640 10 monclient(hunting): handle_auth_done global_id 4474 payload 824 2024-09-11T01:31:11.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.334+0000 7f29e90ce640 1 -- 172.21.15.12:0/887797678 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f29e4075000 con 0x7f29e4137de0 2024-09-11T01:31:11.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.336+0000 7f29e90ce640 1 -- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e4137de0 msgr2=0x7f29e409e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:11.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.336+0000 7f29e90ce640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e4137de0 0x7f29e409e4f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f29e0009870 tx=0x7f29e0008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:11.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.337+0000 7f29d9ffb640 1 -- 172.21.15.12:0/887797678 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e0037040 con 0x7f29e4137de0 2024-09-11T01:31:11.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.337+0000 7f29d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:11.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.337+0000 7f29d9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:31:11.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.337+0000 7f29d9ffb640 10 monclient: dump: 2024-09-11T01:31:11.349 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:11.349 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:11.349 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.349 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.349 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:11.349 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.349 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:11.349 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:11.349 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:11.349 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.337+0000 7f29d9ffb640 1 -- 172.21.15.12:0/887797678 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7f29e00463f0 con 0x7f29e4137de0 2024-09-11T01:31:11.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.337+0000 7f29d9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:31:11.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.337+0000 7f29d9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:31:11.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.337+0000 7f29d9ffb640 1 -- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e4137de0 msgr2=0x7f29e409e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:11.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.337+0000 7f29d9ffb640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e4137de0 0x7f29e409e4f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f29e0009870 tx=0x7f29e0008030 comp rx=0 tx=0).stop 2024-09-11T01:31:11.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.337+0000 7f29d9ffb640 1 -- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e4137de0 msgr2=0x7f29e409e4f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:31:11.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.337+0000 7f29d9ffb640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e4137de0 0x7f29e409e4f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.337+0000 7f29e98cf640 1 -- 172.21.15.12:0/887797678 reap_dead start 2024-09-11T01:31:11.350 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:31:11.350 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:31:11.350 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:31:11.350 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:31:11.350 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 22, 2024-09-11T01:31:11.351 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 133992, 2024-09-11T01:31:11.351 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:31:11.351 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:31:11.351 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:31:11.351 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:31:11.351 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:11.351 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:31:11.351 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:31:11.351 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:31:11.351 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:31:11.351 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:11.351 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:11.351 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:11.352 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:11.352 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:11.352 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:11.352 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:11.352 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:11.352 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:11.352 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:11.352 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:11.352 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:11.352 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:31:11.352 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:31:11.352 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:11.352 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:11.353 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:11.353 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:11.353 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:11.353 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:11.353 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:11.353 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:11.353 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:11.353 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:11.353 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:11.353 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:11.353 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.353 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:31:11.353 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:31:11.354 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:31:11.354 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:31:11.354 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:31:11.354 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:31:11.354 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:31:11.354 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:31:11.354 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:31:11.354 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:31:11.354 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:31:11.354 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:31:11.354 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:31:11.354 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:31:11.354 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:31:11.355 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:31:11.355 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:31:11.355 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:11.355 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:11.355 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:11.355 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:11.355 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:11.355 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:11.355 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:11.355 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:11.355 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:11.355 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:11.355 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:11.355 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:11.356 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:31:11.356 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.356 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:31:11.356 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.356 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:31:11.356 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:31:11.356 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:11.356 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:11.356 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.356 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:11.356 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:31:11.356 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:11.356 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.357 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.357 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:11.357 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:31:11.357 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:11.357 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:11.357 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:11.357 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.357 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:31:11.357 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:31:11.357 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:11.357 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:11.357 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:11.357 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.358 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.358 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:31:11.358 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:31:11.358 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:11.358 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:11.358 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.358 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:11.358 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:31:11.358 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:11.358 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.358 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.358 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:11.358 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:31:11.359 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:11.359 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:11.359 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:11.359 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.359 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:31:11.359 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:31:11.359 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:11.359 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:11.359 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:11.359 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.359 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.359 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:31:11.359 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:31:11.359 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:11.359 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:11.360 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.360 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:11.360 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:31:11.360 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:11.360 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.360 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.360 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:11.360 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:31:11.360 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:11.360 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:11.360 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:11.360 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.360 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:31:11.361 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:31:11.361 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:11.361 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:11.361 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:11.361 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:11.361 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:11.361 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.361 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:31:11.361 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:31:11.361 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.361 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:11.361 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:11.361 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:31:11.361 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:11.362 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:11.362 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:31:11.362 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.362 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:11.362 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:11.362 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:31:11.362 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:11.362 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:11.362 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:31:11.362 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.362 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:11.362 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:11.362 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:31:11.362 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:11.362 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:11.363 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:31:11.363 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:11.363 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:11.363 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:11.363 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:31:11.363 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:11.363 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:11.363 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:11.363 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:31:11.363 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:31:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.338+0000 7f29eb332640 1 -- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f29c803e210 msgr2=0x7f29c80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.338+0000 7f29eb332640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f29c803e210 0x7f29c80406d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f29e4070f10 tx=0x7f29d4007e30 comp rx=0 tx=0).stop 2024-09-11T01:31:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.338+0000 7f29eb332640 10 monclient: shutdown 2024-09-11T01:31:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.338+0000 7f29eb332640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.338+0000 7f29eb332640 1 -- 172.21.15.12:0/887797678 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f29e4132030 msgr2=0x7f29e413cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.338+0000 7f29eb332640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f29e4132030 0x7f29e413cd50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f29dc00efc0 tx=0x7f29dc00c490 comp rx=0 tx=0).stop 2024-09-11T01:31:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.338+0000 7f29eb332640 1 -- 172.21.15.12:0/887797678 shutdown_connections 2024-09-11T01:31:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.338+0000 7f29eb332640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f29e4132030 0x7f29e413cd50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.338+0000 7f29eb332640 1 --2- 172.21.15.12:0/887797678 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f29c803e210 0x7f29c80406d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.338+0000 7f29eb332640 1 -- 172.21.15.12:0/887797678 >> 172.21.15.12:0/887797678 conn(0x7f29e40a36c0 msgr2=0x7f29e409bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.338+0000 7f29eb332640 1 -- 172.21.15.12:0/887797678 shutdown_connections 2024-09-11T01:31:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.338+0000 7f29eb332640 1 -- 172.21.15.12:0/887797678 wait complete. 2024-09-11T01:31:11.366 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:31:11.366 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-11T01:31:11.366 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T01:31:11.366 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-11T01:31:11.366 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-11T01:31:11.366 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-11T01:31:11.367 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:31:11.379 INFO:tasks.ceph.mon.b:Stopped 2024-09-11T01:31:11.379 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:31:11.379 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:31:11.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.476+0000 7fed1cad4640 10 monclient: get_monmap_and_config 2024-09-11T01:31:11.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.476+0000 7fed1cad4640 10 monclient: build_initial_monmap 2024-09-11T01:31:11.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.476+0000 7fed1cad4640 10 monclient: monmap: 2024-09-11T01:31:11.479 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:11.479 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:11.479 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:31:11.477921+0000 2024-09-11T01:31:11.479 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:31:11.477921+0000 2024-09-11T01:31:11.479 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:11.479 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.479 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:31:11.479 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:31:11.479 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:31:11.479 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:31:11.479 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:31:11.479 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.476+0000 7fed1cad4640 1 Processor -- start 2024-09-11T01:31:11.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1cad4640 1 -- start start 2024-09-11T01:31:11.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1cad4640 10 monclient: init 2024-09-11T01:31:11.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1cad4640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:11.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1cad4640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-11T01:31:11.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1cad4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fed18138230 0x7fed18138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1cad4640 10 monclient(hunting): picked mon.noname-b con 0x7fed18138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:11.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1cad4640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fed1809d030 0x7fed180a5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1cad4640 10 monclient(hunting): picked mon.noname-c con 0x7fed1809d030 addr v2:172.21.15.71:3300/0 2024-09-11T01:31:11.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1cad4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fed1809d430 addr v1:172.21.15.12:6789/0 2024-09-11T01:31:11.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1cad4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed180a6e30 con 0x7fed1809d430 2024-09-11T01:31:11.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1cad4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:11.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1cad4640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fed1809c9c0 con 0x7fed1809d030 2024-09-11T01:31:11.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1cad4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:11.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1cad4640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fed1809ba20 con 0x7fed18138230 2024-09-11T01:31:11.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1cad4640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:11.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1cad4640 10 monclient(hunting): authenticate will time out at 877.920580s 2024-09-11T01:31:11.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed0ffff640 1 -- >> v2:172.21.15.71:3300/0 conn(0x7fed1809d030 msgr2=0x7fed180a5020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:31:11.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed0ffff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fed1809d030 0x7fed180a5020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:31:11.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed16d9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fed1809d430 0x7fed180a5560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34876/0 (socket says 172.21.15.12:34876) 2024-09-11T01:31:11.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fed18138230 0x7fed18138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:11.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed16d9d640 1 -- 172.21.15.12:0/1129754736 learned_addr learned my addr 172.21.15.12:0/1129754736 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:31:11.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1659c640 10 monclient(hunting): get_auth_request con 0x7fed18138230 auth_method 0 2024-09-11T01:31:11.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:11.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:11.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:11.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:11.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:11.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:11.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1659c640 10 monclient(hunting): handle_auth_done global_id 4580 payload 274 2024-09-11T01:31:11.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1659c640 1 -- 172.21.15.12:0/1129754736 >> v2:172.21.15.71:3300/0 conn(0x7fed1809d030 msgr2=0x7fed180a5020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:31:11.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1659c640 1 --2- 172.21.15.12:0/1129754736 >> v2:172.21.15.71:3300/0 conn(0x7fed1809d030 0x7fed180a5020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1659c640 1 -- 172.21.15.12:0/1129754736 >> v1:172.21.15.12:6789/0 conn(0x7fed1809d430 legacy=0x7fed180a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:11.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1659c640 10 monclient: _finish_hunting 0 2024-09-11T01:31:11.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1659c640 1 monclient: found mon.noname-b 2024-09-11T01:31:11.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:11.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.477+0000 7fed1659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:31:11.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.478+0000 7fed1659c640 1 -- 172.21.15.12:0/1129754736 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed180a66a0 con 0x7fed18138230 2024-09-11T01:31:11.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.480+0000 7fed1659c640 1 -- 172.21.15.12:0/1129754736 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fed18138230 msgr2=0x7fed18138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:11.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.480+0000 7fed1659c640 1 --2- 172.21.15.12:0/1129754736 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fed18138230 0x7fed18138610 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fed1806d650 tx=0x7fed08029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1defd5ba61b5285 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:11.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.481+0000 7fed15d9b640 1 -- 172.21.15.12:0/1129754736 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed08037070 con 0x7fed18138230 2024-09-11T01:31:11.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.481+0000 7fed15d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:11.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.481+0000 7fed15d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:31:11.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.481+0000 7fed15d9b640 10 monclient: dump: 2024-09-11T01:31:11.489 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:11.489 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:11.489 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.489 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.489 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:11.489 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.489 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:11.489 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:11.489 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:11.489 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.481+0000 7fed15d9b640 10 monclient: _finish_auth 0 2024-09-11T01:31:11.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.481+0000 7fed15d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:31:11.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.481+0000 7fed15d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:11.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.481+0000 7fed15d9b640 1 -- 172.21.15.12:0/1129754736 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed08032640 con 0x7fed18138230 2024-09-11T01:31:11.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.481+0000 7fed15d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:11.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.481+0000 7fed15d9b640 1 -- 172.21.15.12:0/1129754736 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed08032a10 con 0x7fed18138230 2024-09-11T01:31:11.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.481+0000 7fed1cad4640 5 monclient: authenticate success, global_id 4580 2024-09-11T01:31:11.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.481+0000 7fed1cad4640 10 monclient: get_monmap_and_config success 2024-09-11T01:31:11.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.481+0000 7fed1cad4640 4 set_mon_vals no callback set 2024-09-11T01:31:11.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.481+0000 7fed15d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:11.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.481+0000 7fed15d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:11.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.481+0000 7fed15d9b640 10 monclient: dump: 2024-09-11T01:31:11.490 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:11.490 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:11.490 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.490 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.490 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:11.491 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.491 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:11.491 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:11.491 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:11.491 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.481+0000 7fed1cad4640 10 monclient: shutdown 2024-09-11T01:31:11.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.481+0000 7fed1cad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:11.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.481+0000 7fed1cad4640 1 -- 172.21.15.12:0/1129754736 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fed18138230 msgr2=0x7fed18138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:11.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.481+0000 7fed1cad4640 1 --2- 172.21.15.12:0/1129754736 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fed18138230 0x7fed18138610 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fed1806d650 tx=0x7fed08029fd0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.481+0000 7fed1cad4640 1 -- 172.21.15.12:0/1129754736 shutdown_connections 2024-09-11T01:31:11.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.481+0000 7fed1cad4640 1 --2- 172.21.15.12:0/1129754736 >> v2:172.21.15.71:3300/0 conn(0x7fed1809d030 0x7fed180a5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.481+0000 7fed1cad4640 1 --2- 172.21.15.12:0/1129754736 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fed18138230 0x7fed18138610 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.481+0000 7fed1cad4640 1 -- 172.21.15.12:0/1129754736 >> 172.21.15.12:0/1129754736 conn(0x7fed180a7120 msgr2=0x7fed180a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:11.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.481+0000 7fed1cad4640 1 -- 172.21.15.12:0/1129754736 shutdown_connections 2024-09-11T01:31:11.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.481+0000 7fed1cad4640 1 -- 172.21.15.12:0/1129754736 wait complete. 2024-09-11T01:31:11.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.482+0000 7fed1cad4640 10 monclient: build_initial_monmap 2024-09-11T01:31:11.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.482+0000 7fed1cad4640 10 monclient: monmap: 2024-09-11T01:31:11.492 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:11.492 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:31:11.492 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:31:11.492 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:31:11.492 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:11.492 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.492 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:31:11.492 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:31:11.492 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:31:11.492 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.482+0000 7fed1cad4640 1 Processor -- start 2024-09-11T01:31:11.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.482+0000 7fed1cad4640 1 -- start start 2024-09-11T01:31:11.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.482+0000 7fed1cad4640 10 monclient: init 2024-09-11T01:31:11.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1cad4640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:11.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1cad4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:31:11.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1cad4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed1809d030 0x7fed1813eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1cad4640 10 monclient(hunting): picked mon.noname-a con 0x7fed1809d030 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:11.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1cad4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed1809d430 0x7fed1813f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1cad4640 10 monclient(hunting): picked mon.noname-b con 0x7fed1809d430 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:31:11.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1cad4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fed18138230 0x7fed18139cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1cad4640 10 monclient(hunting): picked mon.noname-c con 0x7fed18138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:11.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1cad4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:11.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1cad4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fed1806f970 con 0x7fed1809d030 2024-09-11T01:31:11.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1cad4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1cad4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fed1806f7f0 con 0x7fed1809d430 2024-09-11T01:31:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1cad4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1cad4640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fed180a5c70 con 0x7fed18138230 2024-09-11T01:31:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1cad4640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1cad4640 10 monclient(hunting): authenticate will time out at 877.926339s 2024-09-11T01:31:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed1809d030 0x7fed1813eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed1809d030 0x7fed1813eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45840/0 (socket says 172.21.15.12:45840) 2024-09-11T01:31:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1659c640 1 -- 172.21.15.12:0/2067092335 learned_addr learned my addr 172.21.15.12:0/2067092335 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1659c640 10 monclient(hunting): get_auth_request con 0x7fed1809d030 auth_method 0 2024-09-11T01:31:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed0ffff640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed1809d430 msgr2=0x7fed1813f530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:31:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed16d9d640 1 --2- 172.21.15.12:0/2067092335 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fed18138230 0x7fed18139cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:11.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed0ffff640 1 --2- 172.21.15.12:0/2067092335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed1809d430 0x7fed1813f530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:31:11.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed16d9d640 10 monclient(hunting): get_auth_request con 0x7fed18138230 auth_method 0 2024-09-11T01:31:11.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed16d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:11.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed16d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:11.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed16d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:11.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:11.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:11.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:11.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed16d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:11.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed16d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:11.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed16d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:11.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1659c640 10 monclient(hunting): handle_auth_done global_id 24156 payload 824 2024-09-11T01:31:11.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1659c640 1 -- 172.21.15.12:0/2067092335 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fed18138230 msgr2=0x7fed18139cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:11.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1659c640 1 --2- 172.21.15.12:0/2067092335 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fed18138230 0x7fed18139cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1659c640 1 -- 172.21.15.12:0/2067092335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed1809d430 msgr2=0x7fed1813f530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:31:11.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1659c640 1 --2- 172.21.15.12:0/2067092335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed1809d430 0x7fed1813f530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1659c640 10 monclient: _finish_hunting 0 2024-09-11T01:31:11.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1659c640 1 monclient: found mon.noname-a 2024-09-11T01:31:11.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:11.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:31:11.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed1659c640 1 -- 172.21.15.12:0/2067092335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed180a1810 con 0x7fed1809d030 2024-09-11T01:31:11.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.483+0000 7fed16d9d640 1 --2- 172.21.15.12:0/2067092335 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fed18138230 0x7fed18139cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:31:11.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.485+0000 7fed1659c640 1 -- 172.21.15.12:0/2067092335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed1809d030 msgr2=0x7fed1813eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:11.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.485+0000 7fed1659c640 1 --2- 172.21.15.12:0/2067092335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed1809d030 0x7fed1813eff0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fed08009890 tx=0x7fed080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:11.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.485+0000 7fed0effd640 1 -- 172.21.15.12:0/2067092335 DispatchQueue::entry inject delay of 0.002000 2024-09-11T01:31:11.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.488+0000 7fed0effd640 1 -- 172.21.15.12:0/2067092335 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed08042020 con 0x7fed1809d030 2024-09-11T01:31:11.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.488+0000 7fed0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:11.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.488+0000 7fed0effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:31:11.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.488+0000 7fed0effd640 10 monclient: dump: 2024-09-11T01:31:11.499 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:11.499 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:11.499 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.499 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.499 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:11.499 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.499 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:11.499 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:11.499 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:11.499 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.488+0000 7fed0effd640 10 monclient: _finish_auth 0 2024-09-11T01:31:11.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.488+0000 7fed0effd640 10 monclient: _check_auth_tickets 2024-09-11T01:31:11.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.488+0000 7fed0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:11.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.488+0000 7fed0effd640 1 -- 172.21.15.12:0/2067092335 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed08030880 con 0x7fed1809d030 2024-09-11T01:31:11.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.488+0000 7fed0effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:11.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.488+0000 7fed0effd640 1 -- 172.21.15.12:0/2067092335 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed08037040 con 0x7fed1809d030 2024-09-11T01:31:11.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.488+0000 7fed1cad4640 5 monclient: authenticate success, global_id 24156 2024-09-11T01:31:11.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.488+0000 7fed1cad4640 10 monclient: _renew_subs 2024-09-11T01:31:11.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.488+0000 7fed1cad4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:11.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.488+0000 7fed1cad4640 1 -- 172.21.15.12:0/2067092335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed1813d7e0 con 0x7fed1809d030 2024-09-11T01:31:11.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.488+0000 7fed1559a640 4 set_mon_vals no callback set 2024-09-11T01:31:11.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.488+0000 7fed0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:11.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.488+0000 7fed0effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:11.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.488+0000 7fed0effd640 10 monclient: dump: 2024-09-11T01:31:11.501 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:11.501 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:11.501 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.501 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:11.501 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:11.501 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:11.501 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:11.501 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:11.501 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:11.501 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:11.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.488+0000 7fed1cad4640 10 monclient: _renew_subs 2024-09-11T01:31:11.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.488+0000 7fed1cad4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:11.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.488+0000 7fed1cad4640 1 -- 172.21.15.12:0/2067092335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed180a1550 con 0x7fed1809d030 2024-09-11T01:31:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.489+0000 7fed1cad4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.489+0000 7fed1cad4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.489+0000 7fed0effd640 1 -- 172.21.15.12:0/2067092335 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fed08030c50 con 0x7fed1809d030 2024-09-11T01:31:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.489+0000 7fed1cad4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.489+0000 7fed1cad4640 1 -- 172.21.15.12:0/2067092335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed1809bc50 con 0x7fed1809d030 2024-09-11T01:31:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.489+0000 7fed0effd640 1 --2- 172.21.15.12:0/2067092335 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fecf803e210 0x7fecf80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.489+0000 7fed0effd640 1 -- 172.21.15.12:0/2067092335 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 4755+0+0 (secure 0 0 0) 0x7fed08073770 con 0x7fed1809d030 2024-09-11T01:31:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.490+0000 7fed0ffff640 1 --2- 172.21.15.12:0/2067092335 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fecf803e210 0x7fecf80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.490+0000 7fed0ffff640 10 monclient: get_auth_request con 0x7fecf803e210 auth_method 0 2024-09-11T01:31:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.492+0000 7fed0ffff640 1 --2- 172.21.15.12:0/2067092335 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fecf803e210 0x7fecf80406d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fed00006fd0 tx=0x7fed00006d80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.494+0000 7fed0effd640 1 -- 172.21.15.12:0/2067092335 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fed1809bc50 con 0x7fed1809d030 2024-09-11T01:31:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.494+0000 7fed0effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.494+0000 7fed0effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:31:11.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.754+0000 7fed1cad4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:31:11.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.754+0000 7fed1cad4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:31:11.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.754+0000 7fed1cad4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:11.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.754+0000 7fed1cad4640 1 -- 172.21.15.12:0/2067092335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fed1813fa70 con 0x7fed1809d030 2024-09-11T01:31:11.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.755+0000 7fed0effd640 1 -- 172.21.15.12:0/2067092335 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fed0804c070 con 0x7fed1809d030 2024-09-11T01:31:11.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.755+0000 7fed0effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:31:11.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.755+0000 7fed0effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:31:11.758 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:31:11.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.757+0000 7fed1cad4640 1 -- 172.21.15.12:0/2067092335 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fecf803e210 msgr2=0x7fecf80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:11.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.757+0000 7fed1cad4640 1 --2- 172.21.15.12:0/2067092335 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fecf803e210 0x7fecf80406d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fed00006fd0 tx=0x7fed00006d80 comp rx=0 tx=0).stop 2024-09-11T01:31:11.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.757+0000 7fed1cad4640 10 monclient: shutdown 2024-09-11T01:31:11.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.757+0000 7fed1cad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:11.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.757+0000 7fed1cad4640 1 -- 172.21.15.12:0/2067092335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed1809d030 msgr2=0x7fed1813eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:11.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.757+0000 7fed1cad4640 1 --2- 172.21.15.12:0/2067092335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed1809d030 0x7fed1813eff0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fed08009890 tx=0x7fed080046c0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.757+0000 7fed1cad4640 1 -- 172.21.15.12:0/2067092335 shutdown_connections 2024-09-11T01:31:11.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.757+0000 7fed1cad4640 1 --2- 172.21.15.12:0/2067092335 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fed18138230 0x7fed18139cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.757+0000 7fed1cad4640 1 --2- 172.21.15.12:0/2067092335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed1809d430 0x7fed1813f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.757+0000 7fed1cad4640 1 --2- 172.21.15.12:0/2067092335 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fecf803e210 0x7fecf80406d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.757+0000 7fed1cad4640 1 --2- 172.21.15.12:0/2067092335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed1809d030 0x7fed1813eff0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:11.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.757+0000 7fed1cad4640 1 -- 172.21.15.12:0/2067092335 >> 172.21.15.12:0/2067092335 conn(0x7fed180a7120 msgr2=0x7fed180a3320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:11.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.757+0000 7fed1cad4640 1 -- 172.21.15.12:0/2067092335 shutdown_connections 2024-09-11T01:31:11.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:11.757+0000 7fed1cad4640 1 -- 172.21.15.12:0/2067092335 wait complete. 2024-09-11T01:31:14.771 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:31:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.882+0000 7f74789ec640 10 monclient: get_monmap_and_config 2024-09-11T01:31:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.882+0000 7f74789ec640 10 monclient: build_initial_monmap 2024-09-11T01:31:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.882+0000 7f74789ec640 10 monclient: monmap: 2024-09-11T01:31:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:31:14.883858+0000 2024-09-11T01:31:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:31:14.883858+0000 2024-09-11T01:31:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:31:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:31:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:31:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:31:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:31:14.886 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:14.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.882+0000 7f74789ec640 1 Processor -- start 2024-09-11T01:31:14.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.882+0000 7f74789ec640 1 -- start start 2024-09-11T01:31:14.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.882+0000 7f74789ec640 10 monclient: init 2024-09-11T01:31:14.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f74789ec640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:14.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f74789ec640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-11T01:31:14.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f74789ec640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f74740a2e90 0x7f747406dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:14.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f74789ec640 10 monclient(hunting): picked mon.noname-c con 0x7f74740a2e90 addr v2:172.21.15.71:3300/0 2024-09-11T01:31:14.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f74789ec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f74740a3270 addr v1:172.21.15.71:6789/0 2024-09-11T01:31:14.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f74789ec640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f747413d490 0x7f747413f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:14.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f74789ec640 10 monclient(hunting): picked mon.noname-a con 0x7f747413d490 addr v2:172.21.15.12:3300/0 2024-09-11T01:31:14.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f74789ec640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74740a46c0 con 0x7f74740a3270 2024-09-11T01:31:14.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f74789ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:14.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f74789ec640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f74740a4900 con 0x7f747413d490 2024-09-11T01:31:14.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f74789ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:14.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f74789ec640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f747406ee50 con 0x7f74740a2e90 2024-09-11T01:31:14.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f74789ec640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:14.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f74789ec640 10 monclient(hunting): authenticate will time out at 881.326477s 2024-09-11T01:31:14.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f7469d9b640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f74740a3270 legacy=0x7f74740a3680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:31:14.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f747259c640 1 -- >> v2:172.21.15.71:3300/0 conn(0x7f74740a2e90 msgr2=0x7f747406dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:31:14.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f7472d9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f747413d490 0x7f747413f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:14.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f747259c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f74740a2e90 0x7f747406dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:31:14.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f7472d9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f747413d490 0x7f747413f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45852/0 (socket says 172.21.15.12:45852) 2024-09-11T01:31:14.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f7472d9d640 1 -- 172.21.15.12:0/2372627695 learned_addr learned my addr 172.21.15.12:0/2372627695 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:14.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f7472d9d640 10 monclient(hunting): get_auth_request con 0x7f747413d490 auth_method 0 2024-09-11T01:31:14.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f7472d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:14.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f7472d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:14.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f7472d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:14.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f7472d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:14.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f7472d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:14.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f7472d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:14.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f7472d9d640 10 monclient(hunting): handle_auth_done global_id 24159 payload 274 2024-09-11T01:31:14.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f7472d9d640 1 -- 172.21.15.12:0/2372627695 >> v2:172.21.15.71:3300/0 conn(0x7f74740a2e90 msgr2=0x7f747406dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:31:14.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f7472d9d640 1 --2- 172.21.15.12:0/2372627695 >> v2:172.21.15.71:3300/0 conn(0x7f74740a2e90 0x7f747406dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:14.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f7472d9d640 1 -- 172.21.15.12:0/2372627695 >> v1:172.21.15.71:6789/0 conn(0x7f74740a3270 legacy=0x7f74740a3680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:31:14.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f7472d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:31:14.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f7472d9d640 1 monclient: found mon.noname-a 2024-09-11T01:31:14.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f7472d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:14.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f7472d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:31:14.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.883+0000 7f7472d9d640 1 -- 172.21.15.12:0/2372627695 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f747406eb70 con 0x7f747413d490 2024-09-11T01:31:14.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.886+0000 7f7472d9d640 1 -- 172.21.15.12:0/2372627695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f747413d490 msgr2=0x7f747413f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:14.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.886+0000 7f7472d9d640 1 --2- 172.21.15.12:0/2372627695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f747413d490 0x7f747413f880 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f74740a94a0 tx=0x7f746c02a580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e38c8f1233db4cbf server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:14.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f7471d9b640 1 -- 172.21.15.12:0/2372627695 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746c036070 con 0x7f747413d490 2024-09-11T01:31:14.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f7471d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:14.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f7471d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:31:14.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f7471d9b640 10 monclient: dump: 2024-09-11T01:31:14.894 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:14.895 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:14.895 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:14.895 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:14.895 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:14.895 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:14.895 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:14.895 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:14.895 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:14.895 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:14.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f7471d9b640 10 monclient: _finish_auth 0 2024-09-11T01:31:14.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f7471d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:31:14.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f7471d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:14.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f7471d9b640 1 -- 172.21.15.12:0/2372627695 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f746c007800 con 0x7f747413d490 2024-09-11T01:31:14.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f74789ec640 5 monclient: authenticate success, global_id 24159 2024-09-11T01:31:14.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f74789ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:31:14.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f7471d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:14.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f7471d9b640 1 -- 172.21.15.12:0/2372627695 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746c007bd0 con 0x7f747413d490 2024-09-11T01:31:14.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f74789ec640 10 monclient: get_monmap_and_config success 2024-09-11T01:31:14.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f74789ec640 4 set_mon_vals no callback set 2024-09-11T01:31:14.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f7471d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:14.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f7471d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:14.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f7471d9b640 10 monclient: dump: 2024-09-11T01:31:14.896 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:14.896 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:14.896 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:14.896 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:14.896 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:14.897 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:14.897 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:14.897 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:14.897 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:14.897 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:14.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f74789ec640 10 monclient: shutdown 2024-09-11T01:31:14.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f74789ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:14.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f74789ec640 1 -- 172.21.15.12:0/2372627695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f747413d490 msgr2=0x7f747413f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:14.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f74789ec640 1 --2- 172.21.15.12:0/2372627695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f747413d490 0x7f747413f880 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f74740a94a0 tx=0x7f746c02a580 comp rx=0 tx=0).stop 2024-09-11T01:31:14.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f74789ec640 1 -- 172.21.15.12:0/2372627695 shutdown_connections 2024-09-11T01:31:14.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f74789ec640 1 --2- 172.21.15.12:0/2372627695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f747413d490 0x7f747413f880 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:14.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f74789ec640 1 --2- 172.21.15.12:0/2372627695 >> v2:172.21.15.71:3300/0 conn(0x7f74740a2e90 0x7f747406dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:14.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f74789ec640 1 -- 172.21.15.12:0/2372627695 >> 172.21.15.12:0/2372627695 conn(0x7f74740a8490 msgr2=0x7f74740a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:14.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f74789ec640 1 -- 172.21.15.12:0/2372627695 shutdown_connections 2024-09-11T01:31:14.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.887+0000 7f74789ec640 1 -- 172.21.15.12:0/2372627695 wait complete. 2024-09-11T01:31:14.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.888+0000 7f74789ec640 10 monclient: build_initial_monmap 2024-09-11T01:31:14.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.888+0000 7f74789ec640 10 monclient: monmap: 2024-09-11T01:31:14.898 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:14.898 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:31:14.898 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:31:14.898 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:31:14.898 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:14.898 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:14.898 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:31:14.898 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:31:14.898 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:31:14.898 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f74789ec640 1 Processor -- start 2024-09-11T01:31:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f74789ec640 1 -- start start 2024-09-11T01:31:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f74789ec640 10 monclient: init 2024-09-11T01:31:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f74789ec640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f74789ec640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:31:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f74789ec640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f74740a2e90 0x7f7474139c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f74789ec640 10 monclient(hunting): picked mon.noname-c con 0x7f74740a2e90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f74789ec640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74740a3270 0x7f747413bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f74789ec640 10 monclient(hunting): picked mon.noname-a con 0x7f74740a3270 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f74789ec640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f747413d490 0x7f7474138520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f74789ec640 10 monclient(hunting): picked mon.noname-b con 0x7f747413d490 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:31:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f74789ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f74789ec640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f747406fd20 con 0x7f74740a3270 2024-09-11T01:31:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f74789ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:14.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f74789ec640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f747406fba0 con 0x7f747413d490 2024-09-11T01:31:14.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f74789ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:14.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f74789ec640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f747413fdc0 con 0x7f74740a2e90 2024-09-11T01:31:14.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f74789ec640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:14.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f74789ec640 10 monclient(hunting): authenticate will time out at 881.332794s 2024-09-11T01:31:14.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f747259c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f74740a2e90 0x7f7474139c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:14.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f7469d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74740a3270 0x7f747413bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:14.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f7469d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74740a3270 0x7f747413bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45860/0 (socket says 172.21.15.12:45860) 2024-09-11T01:31:14.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f747259c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f74740a2e90 0x7f7474139c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:56026/0 (socket says 172.21.15.12:56026) 2024-09-11T01:31:14.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f7469d9b640 1 -- 172.21.15.12:0/4046914576 learned_addr learned my addr 172.21.15.12:0/4046914576 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:14.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f7472d9d640 1 -- 172.21.15.12:0/4046914576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f747413d490 msgr2=0x7f7474138520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:31:14.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f7469d9b640 10 monclient(hunting): get_auth_request con 0x7f74740a3270 auth_method 0 2024-09-11T01:31:14.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f7469d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f7469d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f7469d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f747259c640 10 monclient(hunting): get_auth_request con 0x7f74740a2e90 auth_method 0 2024-09-11T01:31:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f747259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f747259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f747259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.889+0000 7f7472d9d640 1 --2- 172.21.15.12:0/4046914576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f747413d490 0x7f7474138520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:31:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.890+0000 7f7469d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.890+0000 7f7469d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.890+0000 7f7469d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.890+0000 7f747259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.890+0000 7f747259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.890+0000 7f747259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.890+0000 7f7469d9b640 10 monclient(hunting): handle_auth_done global_id 24162 payload 824 2024-09-11T01:31:14.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.890+0000 7f7469d9b640 1 -- 172.21.15.12:0/4046914576 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f74740a2e90 msgr2=0x7f7474139c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:14.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.890+0000 7f7469d9b640 1 --2- 172.21.15.12:0/4046914576 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f74740a2e90 0x7f7474139c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:14.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.890+0000 7f7469d9b640 1 -- 172.21.15.12:0/4046914576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f747413d490 msgr2=0x7f7474138520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:31:14.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.890+0000 7f7469d9b640 1 --2- 172.21.15.12:0/4046914576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f747413d490 0x7f7474138520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:14.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.890+0000 7f7469d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:31:14.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.890+0000 7f7469d9b640 1 monclient: found mon.noname-a 2024-09-11T01:31:14.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.890+0000 7f7469d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:14.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.890+0000 7f7469d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:31:14.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.890+0000 7f7469d9b640 1 -- 172.21.15.12:0/4046914576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7474122ef0 con 0x7f74740a3270 2024-09-11T01:31:14.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.892+0000 7f7469d9b640 1 -- 172.21.15.12:0/4046914576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74740a3270 msgr2=0x7f747413bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:14.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.892+0000 7f7469d9b640 1 --2- 172.21.15.12:0/4046914576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74740a3270 0x7f747413bc50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f745c00b550 tx=0x7f745c00ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:14.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.892+0000 7f746b7fe640 1 -- 172.21.15.12:0/4046914576 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745c013020 con 0x7f74740a3270 2024-09-11T01:31:14.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.892+0000 7f746b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:14.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.892+0000 7f746b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:31:14.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.892+0000 7f746b7fe640 10 monclient: dump: 2024-09-11T01:31:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:14.905 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.892+0000 7f746b7fe640 10 monclient: _finish_auth 0 2024-09-11T01:31:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.892+0000 7f746b7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:31:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.892+0000 7f746b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.892+0000 7f746b7fe640 1 -- 172.21.15.12:0/4046914576 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f745c005070 con 0x7f74740a3270 2024-09-11T01:31:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.892+0000 7f746b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.892+0000 7f746b7fe640 1 -- 172.21.15.12:0/4046914576 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745c005480 con 0x7f74740a3270 2024-09-11T01:31:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.892+0000 7f74789ec640 5 monclient: authenticate success, global_id 24162 2024-09-11T01:31:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.892+0000 7f74789ec640 10 monclient: _renew_subs 2024-09-11T01:31:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.892+0000 7f74789ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.892+0000 7f74789ec640 1 -- 172.21.15.12:0/4046914576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f747413b110 con 0x7f74740a3270 2024-09-11T01:31:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.892+0000 7f746b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.892+0000 7f746b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.892+0000 7f746b7fe640 10 monclient: dump: 2024-09-11T01:31:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:14.907 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:14.907 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:14.907 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:14.907 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:14.907 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:14.907 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:14.907 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:14.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.892+0000 7f7470d99640 4 set_mon_vals no callback set 2024-09-11T01:31:14.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.892+0000 7f74789ec640 10 monclient: _renew_subs 2024-09-11T01:31:14.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.892+0000 7f74789ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:14.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.892+0000 7f74789ec640 1 -- 172.21.15.12:0/4046914576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f747409f680 con 0x7f74740a3270 2024-09-11T01:31:14.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.893+0000 7f74789ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:14.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.893+0000 7f74789ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:14.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.893+0000 7f74789ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:14.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.893+0000 7f746b7fe640 1 -- 172.21.15.12:0/4046914576 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f745c005620 con 0x7f74740a3270 2024-09-11T01:31:14.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.893+0000 7f74789ec640 1 -- 172.21.15.12:0/4046914576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74740a7590 con 0x7f74740a3270 2024-09-11T01:31:14.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.894+0000 7f746b7fe640 1 --2- 172.21.15.12:0/4046914576 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f745403e1f0 0x7f74540406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:14.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.894+0000 7f746b7fe640 1 -- 172.21.15.12:0/4046914576 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 4755+0+0 (secure 0 0 0) 0x7f745c052d00 con 0x7f74740a3270 2024-09-11T01:31:14.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.894+0000 7f747259c640 1 --2- 172.21.15.12:0/4046914576 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f745403e1f0 0x7f74540406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:14.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.894+0000 7f747259c640 10 monclient: get_auth_request con 0x7f745403e1f0 auth_method 0 2024-09-11T01:31:14.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.897+0000 7f747259c640 1 --2- 172.21.15.12:0/4046914576 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f745403e1f0 0x7f74540406b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f74640093d0 tx=0x7f7464009360 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:14.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.898+0000 7f746b7fe640 1 -- 172.21.15.12:0/4046914576 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f74740a7590 con 0x7f74740a3270 2024-09-11T01:31:14.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.898+0000 7f746b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:14.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:14.898+0000 7f746b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:31:15.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:15.155+0000 7f74789ec640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:31:15.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:15.155+0000 7f74789ec640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:31:15.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:15.155+0000 7f74789ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:15.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:15.155+0000 7f74789ec640 1 -- 172.21.15.12:0/4046914576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f747406dc60 con 0x7f74740a3270 2024-09-11T01:31:15.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:15.155+0000 7f746b7fe640 1 -- 172.21.15.12:0/4046914576 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f745c01d020 con 0x7f74740a3270 2024-09-11T01:31:15.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:15.155+0000 7f746b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:31:15.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:15.155+0000 7f746b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:31:15.159 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:31:15.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:15.158+0000 7f74789ec640 1 -- 172.21.15.12:0/4046914576 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f745403e1f0 msgr2=0x7f74540406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:15.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:15.158+0000 7f74789ec640 1 --2- 172.21.15.12:0/4046914576 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f745403e1f0 0x7f74540406b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f74640093d0 tx=0x7f7464009360 comp rx=0 tx=0).stop 2024-09-11T01:31:15.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:15.158+0000 7f74789ec640 10 monclient: shutdown 2024-09-11T01:31:15.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:15.158+0000 7f74789ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:15.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:15.158+0000 7f74789ec640 1 -- 172.21.15.12:0/4046914576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74740a3270 msgr2=0x7f747413bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:15.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:15.158+0000 7f74789ec640 1 --2- 172.21.15.12:0/4046914576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74740a3270 0x7f747413bc50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f745c00b550 tx=0x7f745c00ba10 comp rx=0 tx=0).stop 2024-09-11T01:31:15.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:15.158+0000 7f74789ec640 1 -- 172.21.15.12:0/4046914576 shutdown_connections 2024-09-11T01:31:15.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:15.158+0000 7f74789ec640 1 --2- 172.21.15.12:0/4046914576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f747413d490 0x7f7474138520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:15.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:15.158+0000 7f74789ec640 1 --2- 172.21.15.12:0/4046914576 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f745403e1f0 0x7f74540406b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:15.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:15.158+0000 7f74789ec640 1 --2- 172.21.15.12:0/4046914576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f74740a3270 0x7f747413bc50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:15.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:15.158+0000 7f74789ec640 1 --2- 172.21.15.12:0/4046914576 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f74740a2e90 0x7f7474139c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:15.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:15.158+0000 7f74789ec640 1 -- 172.21.15.12:0/4046914576 >> 172.21.15.12:0/4046914576 conn(0x7f74740a8490 msgr2=0x7f74740a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:15.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:15.158+0000 7f74789ec640 1 -- 172.21.15.12:0/4046914576 shutdown_connections 2024-09-11T01:31:15.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:15.158+0000 7f74789ec640 1 -- 172.21.15.12:0/4046914576 wait complete. 2024-09-11T01:31:18.172 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:31:18.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.282+0000 7fc7b5da3640 10 monclient: get_monmap_and_config 2024-09-11T01:31:18.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.282+0000 7fc7b5da3640 10 monclient: build_initial_monmap 2024-09-11T01:31:18.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.282+0000 7fc7b5da3640 10 monclient: monmap: 2024-09-11T01:31:18.285 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:18.285 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:18.285 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:31:18.283986+0000 2024-09-11T01:31:18.285 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:31:18.283986+0000 2024-09-11T01:31:18.285 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:18.285 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:18.285 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:31:18.285 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:31:18.285 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:31:18.285 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:31:18.286 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:31:18.286 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:18.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.283+0000 7fc7b5da3640 1 Processor -- start 2024-09-11T01:31:18.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.283+0000 7fc7b5da3640 1 -- start start 2024-09-11T01:31:18.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.283+0000 7fc7b5da3640 10 monclient: init 2024-09-11T01:31:18.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.283+0000 7fc7b5da3640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:18.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.283+0000 7fc7b5da3640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-11T01:31:18.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.283+0000 7fc7b5da3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc7b0138210 addr v1:172.21.15.12:6789/0 2024-09-11T01:31:18.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.283+0000 7fc7b5da3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc7b00a6810 addr v1:172.21.15.71:6789/0 2024-09-11T01:31:18.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.283+0000 7fc7b5da3640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7b013e820 0x7fc7b0140c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:18.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.283+0000 7fc7b5da3640 10 monclient(hunting): picked mon.noname-b con 0x7fc7b013e820 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:18.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.283+0000 7fc7b5da3640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7b0138d00 con 0x7fc7b0138210 2024-09-11T01:31:18.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.283+0000 7fc7b5da3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7b009ba20 con 0x7fc7b00a6810 2024-09-11T01:31:18.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.283+0000 7fc7b5da3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:18.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.283+0000 7fc7b5da3640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc7b009c4c0 con 0x7fc7b013e820 2024-09-11T01:31:18.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.283+0000 7fc7b5da3640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:18.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.283+0000 7fc7b5da3640 10 monclient(hunting): authenticate will time out at 884.726808s 2024-09-11T01:31:18.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.283+0000 7fc7a6ffd640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7fc7b00a6810 legacy=0x7fc7b013d100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:31:18.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.283+0000 7fc7af7fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc7b0138210 0x7fc7b01385f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50004/0 (socket says 172.21.15.12:50004) 2024-09-11T01:31:18.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.283+0000 7fc7af7fe640 1 -- 172.21.15.12:0/3380439144 learned_addr learned my addr 172.21.15.12:0/3380439144 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:31:18.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.283+0000 7fc7affff640 1 --2- 172.21.15.12:0/3380439144 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7b013e820 0x7fc7b0140c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:18.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.283+0000 7fc7affff640 10 monclient(hunting): get_auth_request con 0x7fc7b013e820 auth_method 0 2024-09-11T01:31:18.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.283+0000 7fc7affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:18.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.283+0000 7fc7affff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:18.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.283+0000 7fc7affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:18.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.284+0000 7fc7affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:18.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.284+0000 7fc7affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:18.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.284+0000 7fc7affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:18.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.284+0000 7fc7affff640 10 monclient(hunting): handle_auth_done global_id 4589 payload 274 2024-09-11T01:31:18.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.284+0000 7fc7affff640 1 -- 172.21.15.12:0/3380439144 >> v1:172.21.15.71:6789/0 conn(0x7fc7b00a6810 legacy=0x7fc7b013d100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:31:18.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.284+0000 7fc7affff640 1 -- 172.21.15.12:0/3380439144 >> v1:172.21.15.12:6789/0 conn(0x7fc7b0138210 legacy=0x7fc7b01385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:18.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.284+0000 7fc7affff640 10 monclient: _finish_hunting 0 2024-09-11T01:31:18.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.284+0000 7fc7affff640 1 monclient: found mon.noname-b 2024-09-11T01:31:18.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.284+0000 7fc7affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:18.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.284+0000 7fc7affff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:31:18.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.284+0000 7fc7affff640 1 -- 172.21.15.12:0/3380439144 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7b00a6590 con 0x7fc7b013e820 2024-09-11T01:31:18.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7affff640 1 -- 172.21.15.12:0/3380439144 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7b013e820 msgr2=0x7fc7b0140c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:18.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7affff640 1 --2- 172.21.15.12:0/3380439144 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7b013e820 0x7fc7b0140c10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8009dc0 tx=0x7fc7a802a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba1e7b1ee5d34010 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:18.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7aeffd640 1 -- 172.21.15.12:0/3380439144 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a8007a30 con 0x7fc7b013e820 2024-09-11T01:31:18.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:18.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7aeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:31:18.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7aeffd640 10 monclient: dump: 2024-09-11T01:31:18.294 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:18.294 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:18.294 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:18.295 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:18.295 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:18.295 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:18.295 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:18.295 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:18.295 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:18.295 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:18.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7aeffd640 10 monclient: _finish_auth 0 2024-09-11T01:31:18.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7aeffd640 10 monclient: _check_auth_tickets 2024-09-11T01:31:18.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:18.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7aeffd640 1 -- 172.21.15.12:0/3380439144 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7a8007bd0 con 0x7fc7b013e820 2024-09-11T01:31:18.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7b5da3640 5 monclient: authenticate success, global_id 4589 2024-09-11T01:31:18.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7b5da3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:31:18.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7aeffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:18.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7b5da3640 10 monclient: get_monmap_and_config success 2024-09-11T01:31:18.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7b5da3640 4 set_mon_vals no callback set 2024-09-11T01:31:18.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7aeffd640 1 -- 172.21.15.12:0/3380439144 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a8031400 con 0x7fc7b013e820 2024-09-11T01:31:18.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:18.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:18.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7aeffd640 10 monclient: dump: 2024-09-11T01:31:18.296 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:18.296 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:18.296 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:18.296 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:18.296 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:18.296 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:18.296 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:18.297 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:18.297 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:18.297 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:18.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7b5da3640 10 monclient: shutdown 2024-09-11T01:31:18.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7b5da3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:18.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7b5da3640 1 -- 172.21.15.12:0/3380439144 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7b013e820 msgr2=0x7fc7b0140c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:18.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7b5da3640 1 --2- 172.21.15.12:0/3380439144 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7b013e820 0x7fc7b0140c10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8009dc0 tx=0x7fc7a802a480 comp rx=0 tx=0).stop 2024-09-11T01:31:18.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7b5da3640 1 -- 172.21.15.12:0/3380439144 shutdown_connections 2024-09-11T01:31:18.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7b5da3640 1 --2- 172.21.15.12:0/3380439144 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7b013e820 0x7fc7b0140c10 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:18.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7b5da3640 1 -- 172.21.15.12:0/3380439144 >> 172.21.15.12:0/3380439144 conn(0x7fc7b00aaa80 msgr2=0x7fc7b00aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:18.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7b5da3640 1 -- 172.21.15.12:0/3380439144 shutdown_connections 2024-09-11T01:31:18.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.287+0000 7fc7b5da3640 1 -- 172.21.15.12:0/3380439144 wait complete. 2024-09-11T01:31:18.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.288+0000 7fc7b5da3640 10 monclient: build_initial_monmap 2024-09-11T01:31:18.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.288+0000 7fc7b5da3640 10 monclient: monmap: 2024-09-11T01:31:18.298 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:18.298 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:31:18.298 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:31:18.298 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:31:18.298 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:18.298 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:18.298 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:31:18.298 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:31:18.298 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:31:18.298 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:18.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.288+0000 7fc7b5da3640 1 Processor -- start 2024-09-11T01:31:18.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7b5da3640 1 -- start start 2024-09-11T01:31:18.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7b5da3640 10 monclient: init 2024-09-11T01:31:18.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7b5da3640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:18.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7b5da3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:31:18.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7b5da3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7b00a6810 0x7fc7b0139b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:18.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7b5da3640 10 monclient(hunting): picked mon.noname-b con 0x7fc7b00a6810 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:31:18.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7b5da3640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7b0138210 0x7fc7b013a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:18.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7b5da3640 10 monclient(hunting): picked mon.noname-c con 0x7fc7b0138210 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:18.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7b5da3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7b013e820 0x7fc7b013a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:18.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7b5da3640 10 monclient(hunting): picked mon.noname-a con 0x7fc7b013e820 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:18.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7b5da3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:18.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7b5da3640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b006f670 con 0x7fc7b013e820 2024-09-11T01:31:18.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7b5da3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:18.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7b5da3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b006f4f0 con 0x7fc7b00a6810 2024-09-11T01:31:18.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7b5da3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:18.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7b5da3640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc7b0141150 con 0x7fc7b0138210 2024-09-11T01:31:18.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7b5da3640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:18.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7b5da3640 10 monclient(hunting): authenticate will time out at 884.732558s 2024-09-11T01:31:18.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7a6ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7b0138210 0x7fc7b013a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:18.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7affff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7b013e820 0x7fc7b013a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:18.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7a6ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7b0138210 0x7fc7b013a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:38966/0 (socket says 172.21.15.12:38966) 2024-09-11T01:31:18.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7a6ffd640 1 -- 172.21.15.12:0/1016054345 learned_addr learned my addr 172.21.15.12:0/1016054345 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:18.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7a6ffd640 10 monclient(hunting): get_auth_request con 0x7fc7b0138210 auth_method 0 2024-09-11T01:31:18.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:18.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:18.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:18.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7affff640 10 monclient(hunting): get_auth_request con 0x7fc7b013e820 auth_method 0 2024-09-11T01:31:18.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:18.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7affff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:18.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:18.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7af7fe640 1 -- 172.21.15.12:0/1016054345 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7b00a6810 msgr2=0x7fc7b0139b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:31:18.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7af7fe640 1 --2- 172.21.15.12:0/1016054345 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7b00a6810 0x7fc7b0139b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:31:18.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:18.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:18.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:18.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:18.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:18.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:18.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7a6ffd640 10 monclient(hunting): handle_auth_done global_id 4592 payload 824 2024-09-11T01:31:18.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7a6ffd640 1 -- 172.21.15.12:0/1016054345 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7b00a6810 msgr2=0x7fc7b0139b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:31:18.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7a6ffd640 1 --2- 172.21.15.12:0/1016054345 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7b00a6810 0x7fc7b0139b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:18.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.289+0000 7fc7a6ffd640 1 -- 172.21.15.12:0/1016054345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7b013e820 msgr2=0x7fc7b013a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:18.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.290+0000 7fc7a6ffd640 1 --2- 172.21.15.12:0/1016054345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7b013e820 0x7fc7b013a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:18.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.290+0000 7fc7a6ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:31:18.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.290+0000 7fc7a6ffd640 1 monclient: found mon.noname-c 2024-09-11T01:31:18.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.290+0000 7fc7a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:18.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.290+0000 7fc7a6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:31:18.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.290+0000 7fc7a6ffd640 1 -- 172.21.15.12:0/1016054345 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7b013d9b0 con 0x7fc7b0138210 2024-09-11T01:31:18.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.290+0000 7fc7affff640 1 --2- 172.21.15.12:0/1016054345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7b013e820 0x7fc7b013a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:31:18.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.292+0000 7fc7a6ffd640 1 -- 172.21.15.12:0/1016054345 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7b0138210 msgr2=0x7fc7b013a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:18.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.292+0000 7fc7a6ffd640 1 --2- 172.21.15.12:0/1016054345 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7b0138210 0x7fc7b013a0d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc79800b550 tx=0x7fc79800ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:18.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.292+0000 7fc7acff9640 1 -- 172.21.15.12:0/1016054345 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc798013020 con 0x7fc7b0138210 2024-09-11T01:31:18.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.292+0000 7fc7acff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:18.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.292+0000 7fc7acff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:31:18.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.292+0000 7fc7acff9640 10 monclient: dump: 2024-09-11T01:31:18.305 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:18.305 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:18.305 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:18.305 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:18.305 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:18.306 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:18.306 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:18.306 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:18.306 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:18.306 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:18.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.292+0000 7fc7acff9640 10 monclient: _finish_auth 0 2024-09-11T01:31:18.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.292+0000 7fc7acff9640 10 monclient: _check_auth_tickets 2024-09-11T01:31:18.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.292+0000 7fc7acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:18.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.292+0000 7fc7acff9640 1 -- 172.21.15.12:0/1016054345 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc798005070 con 0x7fc7b0138210 2024-09-11T01:31:18.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.292+0000 7fc7acff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:18.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.292+0000 7fc7acff9640 1 -- 172.21.15.12:0/1016054345 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc798005480 con 0x7fc7b0138210 2024-09-11T01:31:18.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.292+0000 7fc7b5da3640 5 monclient: authenticate success, global_id 4592 2024-09-11T01:31:18.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.292+0000 7fc7b5da3640 10 monclient: _renew_subs 2024-09-11T01:31:18.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.292+0000 7fc7b5da3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:18.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.292+0000 7fc7b5da3640 1 -- 172.21.15.12:0/1016054345 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7b013b650 con 0x7fc7b0138210 2024-09-11T01:31:18.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.292+0000 7fc7acff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:18.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.292+0000 7fc7ae7fc640 4 set_mon_vals no callback set 2024-09-11T01:31:18.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.292+0000 7fc7acff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:18.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.292+0000 7fc7acff9640 10 monclient: dump: 2024-09-11T01:31:18.307 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:18.307 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:18.307 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:18.307 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:18.307 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:18.307 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:18.307 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:18.307 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:18.308 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:18.308 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:18.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.292+0000 7fc7b5da3640 10 monclient: _renew_subs 2024-09-11T01:31:18.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.292+0000 7fc7b5da3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:18.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.292+0000 7fc7b5da3640 1 -- 172.21.15.12:0/1016054345 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7b013d6d0 con 0x7fc7b0138210 2024-09-11T01:31:18.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.293+0000 7fc7acff9640 1 -- 172.21.15.12:0/1016054345 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fc798005620 con 0x7fc7b0138210 2024-09-11T01:31:18.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.293+0000 7fc7b5da3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:18.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.293+0000 7fc7b5da3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:18.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.293+0000 7fc7b5da3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:18.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.293+0000 7fc7b5da3640 1 -- 172.21.15.12:0/1016054345 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7b01385f0 con 0x7fc7b0138210 2024-09-11T01:31:18.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.293+0000 7fc7acff9640 1 --2- 172.21.15.12:0/1016054345 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc79003e1c0 0x7fc790040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:18.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.293+0000 7fc7acff9640 1 -- 172.21.15.12:0/1016054345 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(70..70 src has 1..70) ==== 4755+0+0 (secure 0 0 0) 0x7fc798052a30 con 0x7fc7b0138210 2024-09-11T01:31:18.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.294+0000 7fc7af7fe640 1 --2- 172.21.15.12:0/1016054345 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc79003e1c0 0x7fc790040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:18.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.294+0000 7fc7af7fe640 10 monclient: get_auth_request con 0x7fc79003e1c0 auth_method 0 2024-09-11T01:31:18.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.296+0000 7fc7af7fe640 1 --2- 172.21.15.12:0/1016054345 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc79003e1c0 0x7fc790040680 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0007fd0 tx=0x7fc7a0007b20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:18.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.298+0000 7fc7acff9640 1 -- 172.21.15.12:0/1016054345 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc79801d030 con 0x7fc7b0138210 2024-09-11T01:31:18.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.298+0000 7fc7acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:18.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.298+0000 7fc7acff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:31:18.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.554+0000 7fc7b5da3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:31:18.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.554+0000 7fc7b5da3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:31:18.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.554+0000 7fc7b5da3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:18.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.554+0000 7fc7b5da3640 1 -- 172.21.15.12:0/1016054345 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc7b0074b50 con 0x7fc7b0138210 2024-09-11T01:31:18.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.554+0000 7fc7acff9640 1 -- 172.21.15.12:0/1016054345 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fc798051440 con 0x7fc7b0138210 2024-09-11T01:31:18.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.554+0000 7fc7acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:31:18.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.554+0000 7fc7acff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:31:18.557 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:31:18.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.556+0000 7fc7b5da3640 1 -- 172.21.15.12:0/1016054345 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc79003e1c0 msgr2=0x7fc790040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:18.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.556+0000 7fc7b5da3640 1 --2- 172.21.15.12:0/1016054345 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc79003e1c0 0x7fc790040680 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0007fd0 tx=0x7fc7a0007b20 comp rx=0 tx=0).stop 2024-09-11T01:31:18.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.556+0000 7fc7b5da3640 10 monclient: shutdown 2024-09-11T01:31:18.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.556+0000 7fc7b5da3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:18.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.556+0000 7fc7b5da3640 1 -- 172.21.15.12:0/1016054345 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7b0138210 msgr2=0x7fc7b013a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:18.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.556+0000 7fc7b5da3640 1 --2- 172.21.15.12:0/1016054345 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7b0138210 0x7fc7b013a0d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc79800b550 tx=0x7fc79800ba10 comp rx=0 tx=0).stop 2024-09-11T01:31:18.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.556+0000 7fc7b5da3640 1 -- 172.21.15.12:0/1016054345 shutdown_connections 2024-09-11T01:31:18.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.556+0000 7fc7b5da3640 1 --2- 172.21.15.12:0/1016054345 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc79003e1c0 0x7fc790040680 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:18.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.556+0000 7fc7b5da3640 1 --2- 172.21.15.12:0/1016054345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7b013e820 0x7fc7b013a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:18.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.556+0000 7fc7b5da3640 1 --2- 172.21.15.12:0/1016054345 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7b0138210 0x7fc7b013a0d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:18.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.556+0000 7fc7b5da3640 1 --2- 172.21.15.12:0/1016054345 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7b00a6810 0x7fc7b0139b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:18.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.556+0000 7fc7b5da3640 1 -- 172.21.15.12:0/1016054345 >> 172.21.15.12:0/1016054345 conn(0x7fc7b00aaa80 msgr2=0x7fc7b0140260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:18.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.556+0000 7fc7b5da3640 1 -- 172.21.15.12:0/1016054345 shutdown_connections 2024-09-11T01:31:18.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:18.556+0000 7fc7b5da3640 1 -- 172.21.15.12:0/1016054345 wait complete. 2024-09-11T01:31:21.570 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:31:21.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.680+0000 7fd141041640 10 monclient: get_monmap_and_config 2024-09-11T01:31:21.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.680+0000 7fd141041640 10 monclient: build_initial_monmap 2024-09-11T01:31:21.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.680+0000 7fd141041640 10 monclient: monmap: 2024-09-11T01:31:21.683 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:21.683 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:21.683 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:31:21.682463+0000 2024-09-11T01:31:21.684 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:31:21.682463+0000 2024-09-11T01:31:21.684 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:21.684 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:21.684 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:31:21.684 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:31:21.684 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:31:21.684 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:31:21.684 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:31:21.684 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:21.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.681+0000 7fd141041640 1 Processor -- start 2024-09-11T01:31:21.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.681+0000 7fd141041640 1 -- start start 2024-09-11T01:31:21.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.681+0000 7fd141041640 10 monclient: init 2024-09-11T01:31:21.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.681+0000 7fd141041640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:21.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.681+0000 7fd141041640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-11T01:31:21.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.681+0000 7fd141041640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd13c0a2e90 addr v1:172.21.15.12:6789/0 2024-09-11T01:31:21.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.681+0000 7fd141041640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd13c0a3270 0x7fd13c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:21.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.681+0000 7fd141041640 10 monclient(hunting): picked mon.noname-c con 0x7fd13c0a3270 addr v2:172.21.15.71:3300/0 2024-09-11T01:31:21.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.681+0000 7fd141041640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd13c13d490 0x7fd13c13f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:21.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.681+0000 7fd141041640 10 monclient(hunting): picked mon.noname-a con 0x7fd13c13d490 addr v2:172.21.15.12:3300/0 2024-09-11T01:31:21.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.681+0000 7fd141041640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd13c0a46c0 con 0x7fd13c0a2e90 2024-09-11T01:31:21.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.682+0000 7fd141041640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:21.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.682+0000 7fd141041640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fd13c0a4900 con 0x7fd13c13d490 2024-09-11T01:31:21.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.682+0000 7fd141041640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:21.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.682+0000 7fd141041640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fd13c06ee50 con 0x7fd13c0a3270 2024-09-11T01:31:21.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.682+0000 7fd141041640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:21.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.682+0000 7fd141041640 10 monclient(hunting): authenticate will time out at 888.125022s 2024-09-11T01:31:21.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.682+0000 7fd13ad9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd13c0a2e90 0x7fd13c06dc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50010/0 (socket says 172.21.15.12:50010) 2024-09-11T01:31:21.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.682+0000 7fd13ad9d640 1 -- 172.21.15.12:0/4213447872 learned_addr learned my addr 172.21.15.12:0/4213447872 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:31:21.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.682+0000 7fd13a59c640 1 -- 172.21.15.12:0/4213447872 >> v2:172.21.15.71:3300/0 conn(0x7fd13c0a3270 msgr2=0x7fd13c0a3680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:31:21.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.682+0000 7fd13b59e640 1 --2- 172.21.15.12:0/4213447872 >> v2:172.21.15.12:3300/0 conn(0x7fd13c13d490 0x7fd13c13f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:21.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.682+0000 7fd13a59c640 1 --2- 172.21.15.12:0/4213447872 >> v2:172.21.15.71:3300/0 conn(0x7fd13c0a3270 0x7fd13c0a3680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:31:21.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.682+0000 7fd13b59e640 10 monclient(hunting): get_auth_request con 0x7fd13c13d490 auth_method 0 2024-09-11T01:31:21.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.682+0000 7fd13b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:21.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.682+0000 7fd13b59e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:21.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.682+0000 7fd13b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:21.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.684+0000 7fd13ad9d640 1 -- 172.21.15.12:0/4213447872 >> v1:172.21.15.12:6789/0 conn(0x7fd13c0a2e90 legacy=0x7fd13c06dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:21.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.684+0000 7fd13b59e640 1 -- 172.21.15.12:0/4213447872 >> v2:172.21.15.12:3300/0 conn(0x7fd13c13d490 msgr2=0x7fd13c13f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:31:21.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.684+0000 7fd13b59e640 1 -- 172.21.15.12:0/4213447872 >> v2:172.21.15.12:3300/0 conn(0x7fd13c13d490 msgr2=0x7fd13c13f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:31:21.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.684+0000 7fd13b59e640 1 --2- 172.21.15.12:0/4213447872 >> v2:172.21.15.12:3300/0 conn(0x7fd13c13d490 0x7fd13c13f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:31:21.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.684+0000 7fd13b59e640 1 --2- 172.21.15.12:0/4213447872 >> v2:172.21.15.12:3300/0 conn(0x7fd13c13d490 0x7fd13c13f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:31:21.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.883+0000 7fd13a59c640 1 -- 172.21.15.12:0/4213447872 >> v2:172.21.15.71:3300/0 conn(0x7fd13c0a3270 msgr2=0x7fd13c0a3680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:31:21.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.883+0000 7fd13a59c640 1 --2- 172.21.15.12:0/4213447872 >> v2:172.21.15.71:3300/0 conn(0x7fd13c0a3270 0x7fd13c0a3680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:31:21.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.886+0000 7fd13b59e640 1 --2- 172.21.15.12:0/4213447872 >> v2:172.21.15.12:3300/0 conn(0x7fd13c13d490 0x7fd13c13f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:21.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.886+0000 7fd13b59e640 10 monclient(hunting): get_auth_request con 0x7fd13c13d490 auth_method 0 2024-09-11T01:31:21.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.886+0000 7fd13b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:21.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.886+0000 7fd13b59e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:21.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.886+0000 7fd13b59e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:31:21.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.886+0000 7fd13b59e640 1 -- 172.21.15.12:0/4213447872 >> v2:172.21.15.12:3300/0 conn(0x7fd13c13d490 msgr2=0x7fd13c13f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:31:21.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.886+0000 7fd13b59e640 1 -- 172.21.15.12:0/4213447872 >> v2:172.21.15.12:3300/0 conn(0x7fd13c13d490 msgr2=0x7fd13c13f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:31:21.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.886+0000 7fd13b59e640 1 --2- 172.21.15.12:0/4213447872 >> v2:172.21.15.12:3300/0 conn(0x7fd13c13d490 0x7fd13c13f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:31:21.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:21.886+0000 7fd13b59e640 1 --2- 172.21.15.12:0/4213447872 >> v2:172.21.15.12:3300/0 conn(0x7fd13c13d490 0x7fd13c13f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:31:22.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:22.284+0000 7fd13a59c640 1 -- 172.21.15.12:0/4213447872 >> v2:172.21.15.71:3300/0 conn(0x7fd13c0a3270 msgr2=0x7fd13c0a3680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:31:22.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:22.284+0000 7fd13a59c640 1 --2- 172.21.15.12:0/4213447872 >> v2:172.21.15.71:3300/0 conn(0x7fd13c0a3270 0x7fd13c0a3680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:31:22.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:22.288+0000 7fd13b59e640 1 --2- 172.21.15.12:0/4213447872 >> v2:172.21.15.12:3300/0 conn(0x7fd13c13d490 0x7fd13c13f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:22.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:22.288+0000 7fd13b59e640 10 monclient(hunting): get_auth_request con 0x7fd13c13d490 auth_method 0 2024-09-11T01:31:22.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:22.288+0000 7fd13b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:22.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:22.288+0000 7fd13b59e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:22.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:22.288+0000 7fd13b59e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:31:22.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:22.288+0000 7fd13b59e640 1 -- 172.21.15.12:0/4213447872 >> v2:172.21.15.12:3300/0 conn(0x7fd13c13d490 msgr2=0x7fd13c13f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:31:22.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:22.288+0000 7fd13b59e640 1 -- 172.21.15.12:0/4213447872 >> v2:172.21.15.12:3300/0 conn(0x7fd13c13d490 msgr2=0x7fd13c13f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:31:22.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:22.288+0000 7fd13b59e640 1 --2- 172.21.15.12:0/4213447872 >> v2:172.21.15.12:3300/0 conn(0x7fd13c13d490 0x7fd13c13f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:31:22.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:22.288+0000 7fd13b59e640 1 --2- 172.21.15.12:0/4213447872 >> v2:172.21.15.12:3300/0 conn(0x7fd13c13d490 0x7fd13c13f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:31:23.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:23.085+0000 7fd13a59c640 1 -- 172.21.15.12:0/4213447872 >> v2:172.21.15.71:3300/0 conn(0x7fd13c0a3270 msgr2=0x7fd13c0a3680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:31:23.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:23.085+0000 7fd13a59c640 1 --2- 172.21.15.12:0/4213447872 >> v2:172.21.15.71:3300/0 conn(0x7fd13c0a3270 0x7fd13c0a3680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:31:23.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:23.090+0000 7fd13b59e640 1 --2- 172.21.15.12:0/4213447872 >> v2:172.21.15.12:3300/0 conn(0x7fd13c13d490 0x7fd13c13f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:23.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:23.090+0000 7fd13b59e640 10 monclient(hunting): get_auth_request con 0x7fd13c13d490 auth_method 0 2024-09-11T01:31:23.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:23.090+0000 7fd13b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:23.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:23.090+0000 7fd13b59e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:23.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:23.090+0000 7fd13b59e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:31:23.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:23.090+0000 7fd13b59e640 1 -- 172.21.15.12:0/4213447872 >> v2:172.21.15.12:3300/0 conn(0x7fd13c13d490 msgr2=0x7fd13c13f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:31:23.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:23.090+0000 7fd13b59e640 1 -- 172.21.15.12:0/4213447872 >> v2:172.21.15.12:3300/0 conn(0x7fd13c13d490 msgr2=0x7fd13c13f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:31:23.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:23.090+0000 7fd13b59e640 1 --2- 172.21.15.12:0/4213447872 >> v2:172.21.15.12:3300/0 conn(0x7fd13c13d490 0x7fd13c13f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:31:23.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:23.090+0000 7fd13b59e640 1 --2- 172.21.15.12:0/4213447872 >> v2:172.21.15.12:3300/0 conn(0x7fd13c13d490 0x7fd13c13f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:31:24.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd138d99640 10 monclient(hunting): tick 2024-09-11T01:31:24.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd138d99640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:31:24.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd138d99640 1 monclient(hunting): continuing hunt 2024-09-11T01:31:24.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd138d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:31:24.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd138d99640 1 -- 172.21.15.12:0/4213447872 >> v2:172.21.15.71:3300/0 conn(0x7fd13c0a3270 msgr2=0x7fd13c0a3680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:31:24.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd138d99640 1 --2- 172.21.15.12:0/4213447872 >> v2:172.21.15.71:3300/0 conn(0x7fd13c0a3270 0x7fd13c0a3680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:24.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd138d99640 1 -- 172.21.15.12:0/4213447872 >> v2:172.21.15.12:3300/0 conn(0x7fd13c13d490 msgr2=0x7fd13c13f880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:31:24.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd138d99640 1 --2- 172.21.15.12:0/4213447872 >> v2:172.21.15.12:3300/0 conn(0x7fd13c13d490 0x7fd13c13f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:24.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd138d99640 1 -- 172.21.15.12:0/4213447872 >> v1:172.21.15.12:6789/0 conn(0x7fd13c0a2e90 legacy=0x7fd13c06dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:24.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd138d99640 10 monclient: _add_conns ranks=[2,1] 2024-09-11T01:31:24.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd138d99640 1 --2- 172.21.15.12:0/4213447872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd128003730 0x7fd128005b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:24.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd138d99640 10 monclient(hunting): picked mon.noname-b con 0x7fd128003730 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:24.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd138d99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd128006290 addr v1:172.21.15.71:6789/0 2024-09-11T01:31:24.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd138d99640 1 -- 172.21.15.12:0/4213447872 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd13c06eb70 con 0x7fd128006290 2024-09-11T01:31:24.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd138d99640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:24.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd138d99640 1 -- 172.21.15.12:0/4213447872 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd13c0a4900 con 0x7fd128003730 2024-09-11T01:31:24.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd138d99640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:24.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd13ad9d640 1 --2- 172.21.15.12:0/4213447872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd128003730 0x7fd128005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:24.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd13a59c640 1 -- 172.21.15.12:0/4213447872 >> v1:172.21.15.71:6789/0 conn(0x7fd128006290 legacy=0x7fd128008750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:31:24.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd13ad9d640 10 monclient(hunting): get_auth_request con 0x7fd128003730 auth_method 0 2024-09-11T01:31:24.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd13ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:24.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd13ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:24.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd13ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:24.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd13ad9d640 1 -- 172.21.15.12:0/4213447872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd128003730 msgr2=0x7fd128005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:31:24.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd13ad9d640 1 -- 172.21.15.12:0/4213447872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd128003730 msgr2=0x7fd128005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:31:24.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd13ad9d640 1 --2- 172.21.15.12:0/4213447872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd128003730 0x7fd128005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:31:24.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.682+0000 7fd13ad9d640 1 --2- 172.21.15.12:0/4213447872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd128003730 0x7fd128005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:31:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.884+0000 7fd13a59c640 1 -- 172.21.15.12:0/4213447872 >> v1:172.21.15.71:6789/0 conn(0x7fd128006290 legacy=0x7fd128008750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:31:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.884+0000 7fd13ad9d640 1 --2- 172.21.15.12:0/4213447872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd128003730 0x7fd128005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.884+0000 7fd13ad9d640 10 monclient(hunting): get_auth_request con 0x7fd128003730 auth_method 0 2024-09-11T01:31:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.884+0000 7fd13ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.884+0000 7fd13ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.884+0000 7fd13ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:31:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.884+0000 7fd13ad9d640 1 -- 172.21.15.12:0/4213447872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd128003730 msgr2=0x7fd128005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:31:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.884+0000 7fd13ad9d640 1 -- 172.21.15.12:0/4213447872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd128003730 msgr2=0x7fd128005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:31:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.884+0000 7fd13ad9d640 1 --2- 172.21.15.12:0/4213447872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd128003730 0x7fd128005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:31:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:24.884+0000 7fd13ad9d640 1 --2- 172.21.15.12:0/4213447872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd128003730 0x7fd128005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:31:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:25.285+0000 7fd13a59c640 1 -- 172.21.15.12:0/4213447872 >> v1:172.21.15.71:6789/0 conn(0x7fd128006290 legacy=0x7fd128008750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:31:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:25.285+0000 7fd13ad9d640 1 --2- 172.21.15.12:0/4213447872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd128003730 0x7fd128005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:25.285+0000 7fd13ad9d640 10 monclient(hunting): get_auth_request con 0x7fd128003730 auth_method 0 2024-09-11T01:31:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:25.285+0000 7fd13ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:25.285+0000 7fd13ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:25.285+0000 7fd13ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:31:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:25.285+0000 7fd13ad9d640 1 -- 172.21.15.12:0/4213447872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd128003730 msgr2=0x7fd128005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:31:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:25.285+0000 7fd13ad9d640 1 -- 172.21.15.12:0/4213447872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd128003730 msgr2=0x7fd128005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:31:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:25.285+0000 7fd13ad9d640 1 --2- 172.21.15.12:0/4213447872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd128003730 0x7fd128005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:31:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:25.285+0000 7fd13ad9d640 1 --2- 172.21.15.12:0/4213447872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd128003730 0x7fd128005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:31:26.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.086+0000 7fd13ad9d640 1 --2- 172.21.15.12:0/4213447872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd128003730 0x7fd128005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:26.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.086+0000 7fd13ad9d640 10 monclient(hunting): get_auth_request con 0x7fd128003730 auth_method 0 2024-09-11T01:31:26.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.086+0000 7fd13ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:26.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.086+0000 7fd13ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:26.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.086+0000 7fd13ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:31:26.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.086+0000 7fd13ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:26.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.086+0000 7fd13ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:26.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.086+0000 7fd13ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:26.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.086+0000 7fd13ad9d640 10 monclient(hunting): handle_auth_done global_id 4595 payload 274 2024-09-11T01:31:26.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.086+0000 7fd13ad9d640 1 -- 172.21.15.12:0/4213447872 >> v1:172.21.15.71:6789/0 conn(0x7fd128006290 legacy=0x7fd128008750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:31:26.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.086+0000 7fd13ad9d640 10 monclient: _finish_hunting 0 2024-09-11T01:31:26.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.086+0000 7fd13ad9d640 1 monclient: found mon.noname-b 2024-09-11T01:31:26.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.086+0000 7fd13ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:26.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.086+0000 7fd13ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:31:26.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.086+0000 7fd13ad9d640 1 -- 172.21.15.12:0/4213447872 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd12800b0d0 con 0x7fd128003730 2024-09-11T01:31:26.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.089+0000 7fd13ad9d640 1 -- 172.21.15.12:0/4213447872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd128003730 msgr2=0x7fd128005b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:26.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.089+0000 7fd13ad9d640 1 --2- 172.21.15.12:0/4213447872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd128003730 0x7fd128005b60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd12c005ab0 tx=0x7fd12c02a910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a40b2e9ca4184432 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:26.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.093+0000 7fd139d9b640 1 -- 172.21.15.12:0/4213447872 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd12c038070 con 0x7fd128003730 2024-09-11T01:31:26.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.093+0000 7fd139d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:26.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.093+0000 7fd139d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:31:26.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.093+0000 7fd139d9b640 10 monclient: dump: 2024-09-11T01:31:26.096 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:26.096 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:26.096 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.096 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.097 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:26.097 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:26.097 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:26.097 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:26.097 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:26.097 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:26.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.093+0000 7fd139d9b640 10 monclient: _finish_auth 0 2024-09-11T01:31:26.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.093+0000 7fd139d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:31:26.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.093+0000 7fd139d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:26.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.093+0000 7fd139d9b640 1 -- 172.21.15.12:0/4213447872 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd12c004c40 con 0x7fd128003730 2024-09-11T01:31:26.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.093+0000 7fd139d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:26.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.093+0000 7fd139d9b640 1 -- 172.21.15.12:0/4213447872 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd12c005050 con 0x7fd128003730 2024-09-11T01:31:26.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.093+0000 7fd139d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:26.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.093+0000 7fd139d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:26.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.093+0000 7fd139d9b640 10 monclient: dump: 2024-09-11T01:31:26.098 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:26.098 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:26.099 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.099 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.099 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:26.099 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:26.099 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:26.099 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:26.099 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:26.099 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:26.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.093+0000 7fd141041640 5 monclient: authenticate success, global_id 4595 2024-09-11T01:31:26.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.093+0000 7fd141041640 10 monclient: get_monmap_and_config success 2024-09-11T01:31:26.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.093+0000 7fd141041640 4 set_mon_vals no callback set 2024-09-11T01:31:26.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.094+0000 7fd141041640 10 monclient: shutdown 2024-09-11T01:31:26.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.094+0000 7fd141041640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:26.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.094+0000 7fd141041640 1 -- 172.21.15.12:0/4213447872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd128003730 msgr2=0x7fd128005b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:26.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.094+0000 7fd141041640 1 --2- 172.21.15.12:0/4213447872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd128003730 0x7fd128005b60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd12c005ab0 tx=0x7fd12c02a910 comp rx=0 tx=0).stop 2024-09-11T01:31:26.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.094+0000 7fd13b59e640 1 -- 172.21.15.12:0/4213447872 reap_dead start 2024-09-11T01:31:26.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.094+0000 7fd141041640 1 -- 172.21.15.12:0/4213447872 shutdown_connections 2024-09-11T01:31:26.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.094+0000 7fd141041640 1 -- 172.21.15.12:0/4213447872 >> 172.21.15.12:0/4213447872 conn(0x7fd13c0a8490 msgr2=0x7fd13c0a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:26.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.094+0000 7fd141041640 1 -- 172.21.15.12:0/4213447872 shutdown_connections 2024-09-11T01:31:26.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.094+0000 7fd141041640 1 -- 172.21.15.12:0/4213447872 wait complete. 2024-09-11T01:31:26.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.096+0000 7fd141041640 10 monclient: build_initial_monmap 2024-09-11T01:31:26.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.096+0000 7fd141041640 10 monclient: monmap: 2024-09-11T01:31:26.106 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:26.106 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:31:26.106 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:31:26.106 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:31:26.107 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:26.107 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:26.107 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:31:26.107 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:31:26.107 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:31:26.107 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:26.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.096+0000 7fd141041640 1 Processor -- start 2024-09-11T01:31:26.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.096+0000 7fd141041640 1 -- start start 2024-09-11T01:31:26.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.096+0000 7fd141041640 10 monclient: init 2024-09-11T01:31:26.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd141041640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:26.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd141041640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:31:26.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd141041640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd128003730 0x7fd13c137ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:26.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd141041640 10 monclient(hunting): picked mon.noname-b con 0x7fd128003730 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:31:26.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd141041640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd13c138420 0x7fd13c139470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:26.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd141041640 10 monclient(hunting): picked mon.noname-a con 0x7fd13c138420 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:26.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd141041640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd13c1399b0 0x7fd13c139e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:26.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd141041640 10 monclient(hunting): picked mon.noname-c con 0x7fd13c1399b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:26.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd141041640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:26.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd141041640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd13c06fd20 con 0x7fd13c138420 2024-09-11T01:31:26.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd141041640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:26.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd141041640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd13c06fba0 con 0x7fd128003730 2024-09-11T01:31:26.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd141041640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:26.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd141041640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd13c13fdc0 con 0x7fd13c1399b0 2024-09-11T01:31:26.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd141041640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:26.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd141041640 10 monclient(hunting): authenticate will time out at 892.540541s 2024-09-11T01:31:26.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd13ad9d640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd128003730 msgr2=0x7fd13c137ee0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:31:26.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd13a59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd13c138420 0x7fd13c139470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:26.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd13ad9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd128003730 0x7fd13c137ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:31:26.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd13b59e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd13c1399b0 0x7fd13c139e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:26.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd13a59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd13c138420 0x7fd13c139470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48042/0 (socket says 172.21.15.12:48042) 2024-09-11T01:31:26.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd13a59c640 1 -- 172.21.15.12:0/3452554898 learned_addr learned my addr 172.21.15.12:0/3452554898 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:26.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd13a59c640 10 monclient(hunting): get_auth_request con 0x7fd13c138420 auth_method 0 2024-09-11T01:31:26.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd13a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:26.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd13a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:26.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd13a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:26.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd13b59e640 10 monclient(hunting): get_auth_request con 0x7fd13c1399b0 auth_method 0 2024-09-11T01:31:26.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd13b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:26.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd13b59e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:26.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd13b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:26.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd13a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:26.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd13a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:26.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd13a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:26.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd13b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:26.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd13b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:26.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.097+0000 7fd13b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:26.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.098+0000 7fd13b59e640 10 monclient(hunting): handle_auth_done global_id 4598 payload 824 2024-09-11T01:31:26.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.098+0000 7fd13b59e640 1 -- 172.21.15.12:0/3452554898 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd128003730 msgr2=0x7fd13c137ee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:31:26.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.098+0000 7fd13b59e640 1 --2- 172.21.15.12:0/3452554898 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd128003730 0x7fd13c137ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:26.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.098+0000 7fd13b59e640 1 -- 172.21.15.12:0/3452554898 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd13c138420 msgr2=0x7fd13c139470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:26.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.098+0000 7fd13b59e640 1 --2- 172.21.15.12:0/3452554898 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd13c138420 0x7fd13c139470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:26.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.098+0000 7fd13b59e640 10 monclient: _finish_hunting 0 2024-09-11T01:31:26.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.098+0000 7fd13b59e640 1 monclient: found mon.noname-c 2024-09-11T01:31:26.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.098+0000 7fd13b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:26.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.098+0000 7fd13b59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:31:26.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.098+0000 7fd13b59e640 1 -- 172.21.15.12:0/3452554898 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd13c13b310 con 0x7fd13c1399b0 2024-09-11T01:31:26.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.098+0000 7fd13a59c640 1 --2- 172.21.15.12:0/3452554898 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd13c138420 0x7fd13c139470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:31:26.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.100+0000 7fd13b59e640 1 -- 172.21.15.12:0/3452554898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd13c1399b0 msgr2=0x7fd13c139e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:26.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.100+0000 7fd13b59e640 1 --2- 172.21.15.12:0/3452554898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd13c1399b0 0x7fd13c139e40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd13c06e3d0 tx=0x7fd134009da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:26.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.101+0000 7fd1237fe640 1 -- 172.21.15.12:0/3452554898 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13401b070 con 0x7fd13c1399b0 2024-09-11T01:31:26.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.101+0000 7fd1237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:26.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.101+0000 7fd1237fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:31:26.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.101+0000 7fd1237fe640 10 monclient: dump: 2024-09-11T01:31:26.112 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:26.112 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:26.112 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.112 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.112 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:26.112 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:26.112 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:26.112 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:26.112 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:26.112 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:26.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.101+0000 7fd1237fe640 10 monclient: _finish_auth 0 2024-09-11T01:31:26.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.101+0000 7fd1237fe640 10 monclient: _check_auth_tickets 2024-09-11T01:31:26.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.101+0000 7fd1237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:26.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.101+0000 7fd1237fe640 1 -- 172.21.15.12:0/3452554898 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd134013c20 con 0x7fd13c1399b0 2024-09-11T01:31:26.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.101+0000 7fd141041640 5 monclient: authenticate success, global_id 4598 2024-09-11T01:31:26.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.101+0000 7fd1237fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:26.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.101+0000 7fd1237fe640 1 -- 172.21.15.12:0/3452554898 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd134017430 con 0x7fd13c1399b0 2024-09-11T01:31:26.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.101+0000 7fd141041640 10 monclient: _renew_subs 2024-09-11T01:31:26.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.101+0000 7fd141041640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:26.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.101+0000 7fd141041640 1 -- 172.21.15.12:0/3452554898 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd13c13cc40 con 0x7fd13c1399b0 2024-09-11T01:31:26.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.101+0000 7fd1237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:26.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.101+0000 7fd1237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:26.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.101+0000 7fd1237fe640 10 monclient: dump: 2024-09-11T01:31:26.114 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:26.114 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:26.114 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.114 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.114 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:26.115 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:26.115 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:26.115 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:26.115 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:26.115 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:26.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.101+0000 7fd13959a640 4 set_mon_vals no callback set 2024-09-11T01:31:26.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.101+0000 7fd141041640 10 monclient: _renew_subs 2024-09-11T01:31:26.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.101+0000 7fd141041640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:26.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.101+0000 7fd141041640 1 -- 172.21.15.12:0/3452554898 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd13c13fb90 con 0x7fd13c1399b0 2024-09-11T01:31:26.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.102+0000 7fd1237fe640 1 -- 172.21.15.12:0/3452554898 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fd134017610 con 0x7fd13c1399b0 2024-09-11T01:31:26.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.102+0000 7fd141041640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:26.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.102+0000 7fd141041640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:26.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.102+0000 7fd141041640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:26.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.102+0000 7fd1237fe640 1 --2- 172.21.15.12:0/3452554898 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd12803f1d0 0x7fd128041690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:26.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.102+0000 7fd141041640 1 -- 172.21.15.12:0/3452554898 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd13c09f680 con 0x7fd13c1399b0 2024-09-11T01:31:26.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.102+0000 7fd1237fe640 1 -- 172.21.15.12:0/3452554898 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(71..71 src has 1..71) ==== 4783+0+0 (secure 0 0 0) 0x7fd134053b60 con 0x7fd13c1399b0 2024-09-11T01:31:26.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.103+0000 7fd13ad9d640 1 --2- 172.21.15.12:0/3452554898 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd12803f1d0 0x7fd128041690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:26.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.103+0000 7fd13ad9d640 10 monclient: get_auth_request con 0x7fd12803f1d0 auth_method 0 2024-09-11T01:31:26.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.105+0000 7fd13ad9d640 1 --2- 172.21.15.12:0/3452554898 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd12803f1d0 0x7fd128041690 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd12c032000 tx=0x7fd12c010040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:26.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.108+0000 7fd1237fe640 1 -- 172.21.15.12:0/3452554898 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd134025030 con 0x7fd13c1399b0 2024-09-11T01:31:26.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.108+0000 7fd1237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:26.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.108+0000 7fd1237fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:31:26.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.369+0000 7fd141041640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:31:26.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.369+0000 7fd141041640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:31:26.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.369+0000 7fd141041640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:26.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.369+0000 7fd141041640 1 -- 172.21.15.12:0/3452554898 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd13c0a3680 con 0x7fd13c1399b0 2024-09-11T01:31:26.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.369+0000 7fd1237fe640 1 -- 172.21.15.12:0/3452554898 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7fd1340163d0 con 0x7fd13c1399b0 2024-09-11T01:31:26.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.369+0000 7fd1237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:31:26.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.369+0000 7fd1237fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:31:26.373 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":26,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:31:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.372+0000 7fd141041640 1 -- 172.21.15.12:0/3452554898 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd12803f1d0 msgr2=0x7fd128041690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.372+0000 7fd141041640 1 --2- 172.21.15.12:0/3452554898 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd12803f1d0 0x7fd128041690 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd12c032000 tx=0x7fd12c010040 comp rx=0 tx=0).stop 2024-09-11T01:31:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.372+0000 7fd141041640 10 monclient: shutdown 2024-09-11T01:31:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.372+0000 7fd141041640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.372+0000 7fd141041640 1 -- 172.21.15.12:0/3452554898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd13c1399b0 msgr2=0x7fd13c139e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.372+0000 7fd141041640 1 --2- 172.21.15.12:0/3452554898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd13c1399b0 0x7fd13c139e40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd13c06e3d0 tx=0x7fd134009da0 comp rx=0 tx=0).stop 2024-09-11T01:31:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.372+0000 7fd141041640 1 -- 172.21.15.12:0/3452554898 shutdown_connections 2024-09-11T01:31:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.372+0000 7fd141041640 1 --2- 172.21.15.12:0/3452554898 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd13c1399b0 0x7fd13c139e40 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.372+0000 7fd141041640 1 --2- 172.21.15.12:0/3452554898 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd12803f1d0 0x7fd128041690 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:26.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.372+0000 7fd141041640 1 --2- 172.21.15.12:0/3452554898 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd13c138420 0x7fd13c139470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:26.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.372+0000 7fd141041640 1 --2- 172.21.15.12:0/3452554898 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd128003730 0x7fd13c137ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:26.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.372+0000 7fd141041640 1 -- 172.21.15.12:0/3452554898 >> 172.21.15.12:0/3452554898 conn(0x7fd13c0a8490 msgr2=0x7fd13c09e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:26.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.372+0000 7fd141041640 1 -- 172.21.15.12:0/3452554898 shutdown_connections 2024-09-11T01:31:26.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.372+0000 7fd141041640 1 -- 172.21.15.12:0/3452554898 wait complete. 2024-09-11T01:31:26.385 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:31:26.385 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-11T01:31:26.386 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:31:26.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.485+0000 7f2e69364640 10 monclient: get_monmap_and_config 2024-09-11T01:31:26.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.485+0000 7f2e69364640 10 monclient: build_initial_monmap 2024-09-11T01:31:26.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.485+0000 7f2e69364640 10 monclient: monmap: 2024-09-11T01:31:26.488 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:26.488 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:26.488 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:31:26.486923+0000 2024-09-11T01:31:26.488 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:31:26.486923+0000 2024-09-11T01:31:26.488 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:26.488 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:26.488 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:31:26.488 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:31:26.488 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:31:26.488 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:31:26.488 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:31:26.489 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:26.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.485+0000 7f2e69364640 1 Processor -- start 2024-09-11T01:31:26.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.485+0000 7f2e69364640 1 -- start start 2024-09-11T01:31:26.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.485+0000 7f2e69364640 10 monclient: init 2024-09-11T01:31:26.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e69364640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:26.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e69364640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-11T01:31:26.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e69364640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2e64137d50 addr v1:172.21.15.71:6789/0 2024-09-11T01:31:26.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e69364640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2e64138130 0x7f2e64138550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:26.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e69364640 10 monclient(hunting): picked mon.noname-b con 0x7f2e64138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:26.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e69364640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e6413d280 addr v1:172.21.15.12:6789/0 2024-09-11T01:31:26.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e69364640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e6409d820 con 0x7f2e6413d280 2024-09-11T01:31:26.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e69364640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e6409ba20 con 0x7f2e64137d50 2024-09-11T01:31:26.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e69364640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:26.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e69364640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2e6409c650 con 0x7f2e64138130 2024-09-11T01:31:26.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e69364640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:26.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e69364640 10 monclient(hunting): authenticate will time out at 892.929388s 2024-09-11T01:31:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e62ffd640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f2e64137d50 legacy=0x7f2e6409d110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:31:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e637fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2e6413d280 0x7f2e6413f670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60800/0 (socket says 172.21.15.12:60800) 2024-09-11T01:31:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e637fe640 1 -- 172.21.15.12:0/2826999349 learned_addr learned my addr 172.21.15.12:0/2826999349 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:31:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e627fc640 1 --2- 172.21.15.12:0/2826999349 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2e64138130 0x7f2e64138550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e627fc640 10 monclient(hunting): get_auth_request con 0x7f2e64138130 auth_method 0 2024-09-11T01:31:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e627fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e627fc640 10 monclient(hunting): handle_auth_done global_id 4601 payload 274 2024-09-11T01:31:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e627fc640 1 -- 172.21.15.12:0/2826999349 >> v1:172.21.15.71:6789/0 conn(0x7f2e64137d50 legacy=0x7f2e6409d110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:31:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e627fc640 1 -- 172.21.15.12:0/2826999349 >> v1:172.21.15.12:6789/0 conn(0x7f2e6413d280 legacy=0x7f2e6413f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e627fc640 10 monclient: _finish_hunting 0 2024-09-11T01:31:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e627fc640 1 monclient: found mon.noname-b 2024-09-11T01:31:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e627fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:31:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.486+0000 7f2e627fc640 1 -- 172.21.15.12:0/2826999349 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e6409c370 con 0x7f2e64138130 2024-09-11T01:31:26.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.489+0000 7f2e627fc640 1 -- 172.21.15.12:0/2826999349 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2e64138130 msgr2=0x7f2e64138550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:26.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.489+0000 7f2e627fc640 1 --2- 172.21.15.12:0/2826999349 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2e64138130 0x7f2e64138550 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c0097d0 tx=0x7f2e4c02a030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0627c78622c6673 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:26.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.490+0000 7f2e61ffb640 1 -- 172.21.15.12:0/2826999349 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c037070 con 0x7f2e64138130 2024-09-11T01:31:26.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.490+0000 7f2e61ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:26.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.490+0000 7f2e61ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:31:26.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.490+0000 7f2e61ffb640 10 monclient: dump: 2024-09-11T01:31:26.503 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:26.503 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:26.503 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.503 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.503 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:26.503 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:26.503 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:26.503 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:26.504 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:26.504 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:26.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.490+0000 7f2e61ffb640 10 monclient: _finish_auth 0 2024-09-11T01:31:26.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.490+0000 7f2e61ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:31:26.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.490+0000 7f2e61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:26.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.490+0000 7f2e69364640 5 monclient: authenticate success, global_id 4601 2024-09-11T01:31:26.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.490+0000 7f2e61ffb640 1 -- 172.21.15.12:0/2826999349 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e4c032640 con 0x7f2e64138130 2024-09-11T01:31:26.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.490+0000 7f2e69364640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:31:26.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.490+0000 7f2e61ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:26.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.490+0000 7f2e69364640 10 monclient: get_monmap_and_config success 2024-09-11T01:31:26.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.490+0000 7f2e69364640 4 set_mon_vals no callback set 2024-09-11T01:31:26.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.490+0000 7f2e61ffb640 1 -- 172.21.15.12:0/2826999349 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c032a10 con 0x7f2e64138130 2024-09-11T01:31:26.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.490+0000 7f2e61ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:26.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.490+0000 7f2e61ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:26.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.490+0000 7f2e61ffb640 10 monclient: dump: 2024-09-11T01:31:26.505 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:26.505 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:26.505 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.505 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.505 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:26.505 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:26.505 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:26.505 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:26.505 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:26.505 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:26.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.490+0000 7f2e69364640 10 monclient: shutdown 2024-09-11T01:31:26.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.490+0000 7f2e69364640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:26.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.490+0000 7f2e69364640 1 -- 172.21.15.12:0/2826999349 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2e64138130 msgr2=0x7f2e64138550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:26.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.490+0000 7f2e69364640 1 --2- 172.21.15.12:0/2826999349 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2e64138130 0x7f2e64138550 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c0097d0 tx=0x7f2e4c02a030 comp rx=0 tx=0).stop 2024-09-11T01:31:26.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.490+0000 7f2e69364640 1 -- 172.21.15.12:0/2826999349 shutdown_connections 2024-09-11T01:31:26.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.490+0000 7f2e69364640 1 --2- 172.21.15.12:0/2826999349 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2e64138130 0x7f2e64138550 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:26.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.490+0000 7f2e69364640 1 -- 172.21.15.12:0/2826999349 >> 172.21.15.12:0/2826999349 conn(0x7f2e640a7020 msgr2=0x7f2e640a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:26.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.490+0000 7f2e69364640 1 -- 172.21.15.12:0/2826999349 shutdown_connections 2024-09-11T01:31:26.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.490+0000 7f2e69364640 1 -- 172.21.15.12:0/2826999349 wait complete. 2024-09-11T01:31:26.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.491+0000 7f2e69364640 10 monclient: build_initial_monmap 2024-09-11T01:31:26.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.491+0000 7f2e69364640 10 monclient: monmap: 2024-09-11T01:31:26.506 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:26.506 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:31:26.506 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:31:26.507 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:31:26.507 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:26.507 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:26.507 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:31:26.507 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:31:26.507 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:31:26.507 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:26.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.491+0000 7f2e69364640 1 Processor -- start 2024-09-11T01:31:26.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.491+0000 7f2e69364640 1 -- start start 2024-09-11T01:31:26.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.491+0000 7f2e69364640 10 monclient: init 2024-09-11T01:31:26.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e69364640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:26.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e69364640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:31:26.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e69364640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2e64137d50 0x7f2e6413b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:26.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e69364640 10 monclient(hunting): picked mon.noname-b con 0x7f2e64137d50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:31:26.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e69364640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2e64138130 0x7f2e64136cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:26.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e69364640 10 monclient(hunting): picked mon.noname-c con 0x7f2e64138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:26.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e69364640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e6413d280 0x7f2e64139260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:26.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e69364640 10 monclient(hunting): picked mon.noname-a con 0x7f2e6413d280 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:26.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e69364640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:26.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e69364640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6406f850 con 0x7f2e6413d280 2024-09-11T01:31:26.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e69364640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:26.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e69364640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6406f6d0 con 0x7f2e64137d50 2024-09-11T01:31:26.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e69364640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:26.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e69364640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2e64140d90 con 0x7f2e64138130 2024-09-11T01:31:26.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e69364640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:26.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e69364640 10 monclient(hunting): authenticate will time out at 892.935230s 2024-09-11T01:31:26.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e637fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e6413d280 0x7f2e64139260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:26.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e627fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2e64138130 0x7f2e64136cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:26.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e62ffd640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2e64137d50 msgr2=0x7f2e6413b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:31:26.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e637fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e6413d280 0x7f2e64139260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56346/0 (socket says 172.21.15.12:56346) 2024-09-11T01:31:26.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e637fe640 1 -- 172.21.15.12:0/4083336247 learned_addr learned my addr 172.21.15.12:0/4083336247 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:26.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e637fe640 10 monclient(hunting): get_auth_request con 0x7f2e6413d280 auth_method 0 2024-09-11T01:31:26.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:26.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e637fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:26.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:26.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e62ffd640 1 --2- 172.21.15.12:0/4083336247 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2e64137d50 0x7f2e6413b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:31:26.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e627fc640 10 monclient(hunting): get_auth_request con 0x7f2e64138130 auth_method 0 2024-09-11T01:31:26.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:26.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e627fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:26.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:26.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:26.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:26.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:26.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:26.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:26.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:26.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e637fe640 10 monclient(hunting): handle_auth_done global_id 24171 payload 824 2024-09-11T01:31:26.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e637fe640 1 -- 172.21.15.12:0/4083336247 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2e64138130 msgr2=0x7f2e64136cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:26.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e637fe640 1 --2- 172.21.15.12:0/4083336247 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2e64138130 0x7f2e64136cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:26.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e637fe640 1 -- 172.21.15.12:0/4083336247 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2e64137d50 msgr2=0x7f2e6413b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:31:26.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e637fe640 1 --2- 172.21.15.12:0/4083336247 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2e64137d50 0x7f2e6413b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:26.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e637fe640 10 monclient: _finish_hunting 0 2024-09-11T01:31:26.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e637fe640 1 monclient: found mon.noname-a 2024-09-11T01:31:26.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:26.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e637fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:31:26.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e637fe640 1 -- 172.21.15.12:0/4083336247 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e6413ccc0 con 0x7f2e6413d280 2024-09-11T01:31:26.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.492+0000 7f2e627fc640 1 --2- 172.21.15.12:0/4083336247 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2e64138130 0x7f2e64136cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:31:26.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.494+0000 7f2e637fe640 1 -- 172.21.15.12:0/4083336247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e6413d280 msgr2=0x7f2e64139260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:26.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.494+0000 7f2e637fe640 1 --2- 172.21.15.12:0/4083336247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e6413d280 0x7f2e64139260 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c00caf0 tx=0x7f2e5c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:26.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.495+0000 7f2e5b7fe640 1 -- 172.21.15.12:0/4083336247 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c012400 con 0x7f2e6413d280 2024-09-11T01:31:26.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.495+0000 7f2e5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:26.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.495+0000 7f2e5b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:31:26.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.495+0000 7f2e5b7fe640 10 monclient: dump: 2024-09-11T01:31:26.512 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:26.512 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:26.512 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.512 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.512 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:26.512 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:26.512 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:26.512 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:26.512 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:26.512 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:26.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.495+0000 7f2e5b7fe640 10 monclient: _finish_auth 0 2024-09-11T01:31:26.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.495+0000 7f2e5b7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:31:26.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.495+0000 7f2e5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:26.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.495+0000 7f2e69364640 5 monclient: authenticate success, global_id 24171 2024-09-11T01:31:26.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.495+0000 7f2e5b7fe640 1 -- 172.21.15.12:0/4083336247 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e5c0125c0 con 0x7f2e6413d280 2024-09-11T01:31:26.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.495+0000 7f2e69364640 10 monclient: _renew_subs 2024-09-11T01:31:26.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.495+0000 7f2e69364640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:26.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.495+0000 7f2e69364640 1 -- 172.21.15.12:0/4083336247 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e64140890 con 0x7f2e6413d280 2024-09-11T01:31:26.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.495+0000 7f2e5b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:26.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.495+0000 7f2e5b7fe640 1 -- 172.21.15.12:0/4083336247 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c0129b0 con 0x7f2e6413d280 2024-09-11T01:31:26.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.495+0000 7f2e5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:26.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.495+0000 7f2e5b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:26.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.495+0000 7f2e5b7fe640 10 monclient: dump: 2024-09-11T01:31:26.513 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:26.514 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:26.514 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.514 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.514 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:26.514 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:26.514 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:26.514 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:26.514 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:26.514 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:26.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.495+0000 7f2e69364640 10 monclient: _renew_subs 2024-09-11T01:31:26.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.495+0000 7f2e617fa640 4 set_mon_vals no callback set 2024-09-11T01:31:26.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.495+0000 7f2e69364640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:26.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.495+0000 7f2e69364640 1 -- 172.21.15.12:0/4083336247 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e6413be50 con 0x7f2e6413d280 2024-09-11T01:31:26.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.495+0000 7f2e69364640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:26.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.495+0000 7f2e69364640 1 --2- 172.21.15.12:0/4083336247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e641397a0 0x7f2e64139b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:26.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.495+0000 7f2e69364640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:26.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.495+0000 7f2e69364640 1 -- 172.21.15.12:0/4083336247 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6413a0c0 con 0x7f2e641397a0 2024-09-11T01:31:26.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.496+0000 7f2e62ffd640 1 --2- 172.21.15.12:0/4083336247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e641397a0 0x7f2e64139b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:26.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.496+0000 7f2e5b7fe640 1 -- 172.21.15.12:0/4083336247 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f2e5c012b90 con 0x7f2e6413d280 2024-09-11T01:31:26.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.496+0000 7f2e62ffd640 10 monclient: get_auth_request con 0x7f2e641397a0 auth_method 0 2024-09-11T01:31:26.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.496+0000 7f2e62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:26.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.496+0000 7f2e62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:26.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.496+0000 7f2e62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:26.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.496+0000 7f2e62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:26.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.496+0000 7f2e62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:26.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.496+0000 7f2e62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:26.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.496+0000 7f2e62ffd640 10 monclient(hunting): handle_auth_done global_id 24174 payload 824 2024-09-11T01:31:26.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.496+0000 7f2e62ffd640 1 -- 172.21.15.12:0/4083336247 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2e641385d0 con 0x7f2e641397a0 2024-09-11T01:31:26.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.496+0000 7f2e5b7fe640 1 --2- 172.21.15.12:0/4083336247 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2e4403e1f0 0x7f2e440406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:26.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.496+0000 7f2e5b7fe640 1 -- 172.21.15.12:0/4083336247 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 4783+0+0 (secure 0 0 0) 0x7f2e5c0533c0 con 0x7f2e6413d280 2024-09-11T01:31:26.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.496+0000 7f2e627fc640 1 --2- 172.21.15.12:0/4083336247 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2e4403e1f0 0x7f2e440406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:26.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.496+0000 7f2e627fc640 10 monclient: get_auth_request con 0x7f2e4403e1f0 auth_method 0 2024-09-11T01:31:26.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.498+0000 7f2e62ffd640 1 -- 172.21.15.12:0/4083336247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e641397a0 msgr2=0x7f2e64139b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:26.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.498+0000 7f2e62ffd640 1 --2- 172.21.15.12:0/4083336247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e641397a0 0x7f2e64139b80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2e6413d250 tx=0x7f2e5400b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:26.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.498+0000 7f2e5b7fe640 1 -- 172.21.15.12:0/4083336247 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e54005270 con 0x7f2e641397a0 2024-09-11T01:31:26.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.498+0000 7f2e5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:26.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.498+0000 7f2e5b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:26.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.498+0000 7f2e5b7fe640 10 monclient: dump: 2024-09-11T01:31:26.516 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:26.517 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:26.517 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.517 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.517 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:26.517 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:26.517 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:26.517 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:26.517 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:26.517 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:26.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.498+0000 7f2e5b7fe640 1 -- 172.21.15.12:0/4083336247 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2e54015070 con 0x7f2e641397a0 2024-09-11T01:31:26.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.498+0000 7f2e5b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:26.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.498+0000 7f2e5b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:31:26.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.498+0000 7f2e5b7fe640 1 -- 172.21.15.12:0/4083336247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e641397a0 msgr2=0x7f2e64139b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:26.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.498+0000 7f2e5b7fe640 1 --2- 172.21.15.12:0/4083336247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e641397a0 0x7f2e64139b80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2e6413d250 tx=0x7f2e5400b8c0 comp rx=0 tx=0).stop 2024-09-11T01:31:26.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.498+0000 7f2e5b7fe640 1 -- 172.21.15.12:0/4083336247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e641397a0 msgr2=0x7f2e64139b80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:31:26.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.498+0000 7f2e5b7fe640 1 --2- 172.21.15.12:0/4083336247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e641397a0 0x7f2e64139b80 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2e6413d250 tx=0x7f2e5400b8c0 comp rx=0 tx=0).stop 2024-09-11T01:31:26.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.499+0000 7f2e627fc640 1 --2- 172.21.15.12:0/4083336247 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2e4403e1f0 0x7f2e440406b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2e6409c070 tx=0x7f2e4c02e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:26.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.505+0000 7f2e69364640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:31:26.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.505+0000 7f2e69364640 1 --2- 172.21.15.12:0/4083336247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e640a3040 0x7f2e640a3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:26.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.505+0000 7f2e69364640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:26.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.505+0000 7f2e69364640 1 -- 172.21.15.12:0/4083336247 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2e64074a40 con 0x7f2e640a3040 2024-09-11T01:31:26.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.505+0000 7f2e62ffd640 1 --2- 172.21.15.12:0/4083336247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e640a3040 0x7f2e640a3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:26.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.505+0000 7f2e62ffd640 10 monclient: get_auth_request con 0x7f2e640a3040 auth_method 0 2024-09-11T01:31:26.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.505+0000 7f2e62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:26.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.505+0000 7f2e62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:26.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.505+0000 7f2e62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:26.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.505+0000 7f2e62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:26.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.505+0000 7f2e62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:26.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.505+0000 7f2e62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:26.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.505+0000 7f2e62ffd640 10 monclient(hunting): handle_auth_done global_id 24177 payload 824 2024-09-11T01:31:26.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.505+0000 7f2e62ffd640 1 -- 172.21.15.12:0/4083336247 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2e64137210 con 0x7f2e640a3040 2024-09-11T01:31:26.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.507+0000 7f2e62ffd640 1 -- 172.21.15.12:0/4083336247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e640a3040 msgr2=0x7f2e640a3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:26.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.507+0000 7f2e62ffd640 1 --2- 172.21.15.12:0/4083336247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e640a3040 0x7f2e640a3420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2e6413d250 tx=0x7f2e54005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:26.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.507+0000 7f2e5b7fe640 1 -- 172.21.15.12:0/4083336247 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5401ae60 con 0x7f2e640a3040 2024-09-11T01:31:26.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.507+0000 7f2e5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:26.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.507+0000 7f2e5b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:26.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.507+0000 7f2e5b7fe640 10 monclient: dump: 2024-09-11T01:31:26.524 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:26.524 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:26.524 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.524 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.524 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:26.524 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:26.525 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:26.525 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:26.525 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:26.525 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:26.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.507+0000 7f2e5b7fe640 1 -- 172.21.15.12:0/4083336247 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4865 (secure 0 0 0) 0x7f2e54010400 con 0x7f2e640a3040 2024-09-11T01:31:26.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.507+0000 7f2e5b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:31:26.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.507+0000 7f2e5b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:31:26.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.507+0000 7f2e5b7fe640 1 -- 172.21.15.12:0/4083336247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e640a3040 msgr2=0x7f2e640a3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:26.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.507+0000 7f2e5b7fe640 1 --2- 172.21.15.12:0/4083336247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e640a3040 0x7f2e640a3420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2e6413d250 tx=0x7f2e54005a20 comp rx=0 tx=0).stop 2024-09-11T01:31:26.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.508+0000 7f2e5b7fe640 1 -- 172.21.15.12:0/4083336247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e640a3040 msgr2=0x7f2e640a3420 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:31:26.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.508+0000 7f2e5b7fe640 1 --2- 172.21.15.12:0/4083336247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e640a3040 0x7f2e640a3420 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2e6413d250 tx=0x7f2e54005a20 comp rx=0 tx=0).stop 2024-09-11T01:31:26.527 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:31:26.527 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:31:26.527 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:31:26.527 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:31:26.527 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 26, 2024-09-11T01:31:26.527 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 24454, 2024-09-11T01:31:26.528 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:31:26.528 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:31:26.528 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:31:26.528 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:26.528 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:31:26.528 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:31:26.529 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:31:26.529 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:31:26.529 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:26.529 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:26.529 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:26.529 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:26.529 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:26.529 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:26.530 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:26.530 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:26.530 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:26.530 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:26.530 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:26.530 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:26.530 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:31:26.530 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:31:26.530 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:26.530 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:26.530 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:26.531 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:26.531 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:26.531 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:26.531 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:26.531 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:26.531 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:26.531 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:26.531 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:26.531 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:26.531 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:26.531 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:31:26.531 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:31:26.531 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:31:26.531 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:31:26.531 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:31:26.532 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:31:26.532 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:31:26.532 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:31:26.532 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:31:26.532 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:31:26.532 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:31:26.532 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:31:26.532 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:31:26.532 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:31:26.532 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:31:26.532 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:31:26.532 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:31:26.532 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:26.533 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:26.533 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:26.533 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:26.533 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:26.533 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:26.533 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:26.533 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:26.533 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:26.533 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:26.533 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:26.533 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:26.533 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:31:26.533 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:26.534 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:31:26.534 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:26.534 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:31:26.534 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:31:26.534 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:26.534 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:26.534 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:26.534 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:26.534 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:31:26.534 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:26.534 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:26.534 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:26.534 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:26.534 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:31:26.535 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:26.535 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:26.535 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:26.535 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:26.535 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:31:26.535 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:31:26.535 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:26.535 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:26.535 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:26.535 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:26.535 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:26.535 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:31:26.535 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:31:26.535 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:26.536 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:26.536 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:26.536 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:26.536 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:31:26.536 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:26.536 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:26.536 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:26.536 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:26.536 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:31:26.536 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:26.536 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:26.536 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:26.536 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:26.537 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:31:26.537 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:31:26.537 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:26.537 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:26.537 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:26.537 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:26.537 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:26.537 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:31:26.537 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:31:26.537 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:26.538 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:26.538 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:26.538 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:26.538 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:31:26.538 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:26.538 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:26.538 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:26.539 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:26.539 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:31:26.539 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:26.539 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:26.539 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:26.539 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:26.539 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:31:26.539 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:31:26.539 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:26.540 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:26.540 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:26.540 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:26.540 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:26.540 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:26.540 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:31:26.540 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:31:26.540 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:26.540 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:26.540 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:26.540 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:31:26.541 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:26.541 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:26.541 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:31:26.541 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:26.541 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:26.541 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:26.541 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:31:26.541 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:26.541 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:26.541 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:31:26.541 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:26.541 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:26.542 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:26.542 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:31:26.542 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:26.542 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:26.542 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:31:26.542 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:26.542 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:26.542 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:26.543 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:31:26.543 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:26.543 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:26.543 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:26.543 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:31:26.543 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:31:26.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.508+0000 7f2e69364640 1 -- 172.21.15.12:0/4083336247 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2e4403e1f0 msgr2=0x7f2e440406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:26.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.508+0000 7f2e69364640 1 --2- 172.21.15.12:0/4083336247 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2e4403e1f0 0x7f2e440406b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2e6409c070 tx=0x7f2e4c02e000 comp rx=0 tx=0).stop 2024-09-11T01:31:26.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.508+0000 7f2e69364640 10 monclient: shutdown 2024-09-11T01:31:26.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.508+0000 7f2e69364640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:26.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.508+0000 7f2e69364640 1 -- 172.21.15.12:0/4083336247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e6413d280 msgr2=0x7f2e64139260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:26.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.508+0000 7f2e69364640 1 --2- 172.21.15.12:0/4083336247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e6413d280 0x7f2e64139260 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c00caf0 tx=0x7f2e5c00a060 comp rx=0 tx=0).stop 2024-09-11T01:31:26.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.508+0000 7f2e637fe640 1 -- 172.21.15.12:0/4083336247 reap_dead start 2024-09-11T01:31:26.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.508+0000 7f2e637fe640 1 -- 172.21.15.12:0/4083336247 reap_dead start 2024-09-11T01:31:26.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.508+0000 7f2e69364640 1 -- 172.21.15.12:0/4083336247 shutdown_connections 2024-09-11T01:31:26.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.508+0000 7f2e69364640 1 -- 172.21.15.12:0/4083336247 >> 172.21.15.12:0/4083336247 conn(0x7f2e640a7020 msgr2=0x7f2e640a2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:26.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.508+0000 7f2e69364640 1 -- 172.21.15.12:0/4083336247 shutdown_connections 2024-09-11T01:31:26.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.508+0000 7f2e69364640 1 -- 172.21.15.12:0/4083336247 wait complete. 2024-09-11T01:31:26.546 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:31:26.546 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T01:31:26.546 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:31:26.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.645+0000 7faa94840640 10 monclient: get_monmap_and_config 2024-09-11T01:31:26.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.645+0000 7faa94840640 10 monclient: build_initial_monmap 2024-09-11T01:31:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.645+0000 7faa94840640 10 monclient: monmap: 2024-09-11T01:31:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:31:26.646935+0000 2024-09-11T01:31:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:31:26.646935+0000 2024-09-11T01:31:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:26.649 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:31:26.649 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:31:26.649 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:31:26.649 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:31:26.649 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:31:26.649 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:26.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.645+0000 7faa94840640 1 Processor -- start 2024-09-11T01:31:26.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.645+0000 7faa94840640 1 -- start start 2024-09-11T01:31:26.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.645+0000 7faa94840640 10 monclient: init 2024-09-11T01:31:26.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa94840640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:26.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa94840640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-11T01:31:26.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa94840640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faa90137e50 addr v1:172.21.15.71:6789/0 2024-09-11T01:31:26.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa94840640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7faa90138230 0x7faa90138640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:26.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa94840640 10 monclient(hunting): picked mon.noname-a con 0x7faa90138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:31:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa94840640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7faa9013d380 0x7faa9013f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa94840640 10 monclient(hunting): picked mon.noname-c con 0x7faa9013d380 addr v2:172.21.15.71:3300/0 2024-09-11T01:31:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa94840640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa900a62d0 con 0x7faa90137e50 2024-09-11T01:31:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa94840640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa94840640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7faa9009bb40 con 0x7faa90138230 2024-09-11T01:31:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa94840640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa94840640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7faa9009c4a0 con 0x7faa9013d380 2024-09-11T01:31:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa94840640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa94840640 10 monclient(hunting): authenticate will time out at 893.089515s 2024-09-11T01:31:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa8e59c640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7faa90137e50 legacy=0x7faa9009d160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:31:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa8ed9d640 1 -- >> v2:172.21.15.71:3300/0 conn(0x7faa9013d380 msgr2=0x7faa9013f770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:31:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa8ed9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7faa9013d380 0x7faa9013f770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:31:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa87fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7faa90138230 0x7faa90138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:26.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa87fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7faa90138230 0x7faa90138640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56386/0 (socket says 172.21.15.12:56386) 2024-09-11T01:31:26.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa87fff640 1 -- 172.21.15.12:0/3416774519 learned_addr learned my addr 172.21.15.12:0/3416774519 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:26.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa87fff640 10 monclient(hunting): get_auth_request con 0x7faa90138230 auth_method 0 2024-09-11T01:31:26.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:26.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa87fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:26.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:26.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:26.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:26.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:26.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa87fff640 10 monclient(hunting): handle_auth_done global_id 24180 payload 274 2024-09-11T01:31:26.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa87fff640 1 -- 172.21.15.12:0/3416774519 >> v2:172.21.15.71:3300/0 conn(0x7faa9013d380 msgr2=0x7faa9013f770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:31:26.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa87fff640 1 --2- 172.21.15.12:0/3416774519 >> v2:172.21.15.71:3300/0 conn(0x7faa9013d380 0x7faa9013f770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:26.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa87fff640 1 -- 172.21.15.12:0/3416774519 >> v1:172.21.15.71:6789/0 conn(0x7faa90137e50 legacy=0x7faa9009d160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:31:26.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa87fff640 10 monclient: _finish_hunting 0 2024-09-11T01:31:26.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa87fff640 1 monclient: found mon.noname-a 2024-09-11T01:31:26.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:26.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa87fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:31:26.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.646+0000 7faa87fff640 1 -- 172.21.15.12:0/3416774519 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa9009c780 con 0x7faa90138230 2024-09-11T01:31:26.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa87fff640 1 -- 172.21.15.12:0/3416774519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa90138230 msgr2=0x7faa90138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:26.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa87fff640 1 --2- 172.21.15.12:0/3416774519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa90138230 0x7faa90138640 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7faa9009cf30 tx=0x7faa7802a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8584d2ed907d4cf server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:26.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa8dd9b640 1 -- 172.21.15.12:0/3416774519 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa78037070 con 0x7faa90138230 2024-09-11T01:31:26.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa8dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:26.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa8dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:31:26.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa8dd9b640 10 monclient: dump: 2024-09-11T01:31:26.664 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:26.664 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:26.664 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.664 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.664 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:26.664 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:26.664 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:26.664 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:26.665 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:26.665 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:26.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa8dd9b640 10 monclient: _finish_auth 0 2024-09-11T01:31:26.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa8dd9b640 10 monclient: _check_auth_tickets 2024-09-11T01:31:26.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa8dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:26.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa94840640 5 monclient: authenticate success, global_id 24180 2024-09-11T01:31:26.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa94840640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:31:26.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa8dd9b640 1 -- 172.21.15.12:0/3416774519 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa78032640 con 0x7faa90138230 2024-09-11T01:31:26.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa8dd9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:26.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa94840640 10 monclient: get_monmap_and_config success 2024-09-11T01:31:26.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa94840640 4 set_mon_vals no callback set 2024-09-11T01:31:26.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa8dd9b640 1 -- 172.21.15.12:0/3416774519 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa78032a10 con 0x7faa90138230 2024-09-11T01:31:26.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa8dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:26.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa8dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:26.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa8dd9b640 10 monclient: dump: 2024-09-11T01:31:26.666 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:26.666 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:26.666 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.666 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.666 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:26.666 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:26.666 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:26.666 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:26.666 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:26.666 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:26.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa94840640 10 monclient: shutdown 2024-09-11T01:31:26.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa94840640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:26.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa94840640 1 -- 172.21.15.12:0/3416774519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa90138230 msgr2=0x7faa90138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:26.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa94840640 1 --2- 172.21.15.12:0/3416774519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa90138230 0x7faa90138640 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7faa9009cf30 tx=0x7faa7802a0d0 comp rx=0 tx=0).stop 2024-09-11T01:31:26.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa94840640 1 -- 172.21.15.12:0/3416774519 shutdown_connections 2024-09-11T01:31:26.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa94840640 1 --2- 172.21.15.12:0/3416774519 >> v2:172.21.15.71:3300/0 conn(0x7faa9013d380 0x7faa9013f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:26.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa94840640 1 --2- 172.21.15.12:0/3416774519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa90138230 0x7faa90138640 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:26.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa94840640 1 -- 172.21.15.12:0/3416774519 >> 172.21.15.12:0/3416774519 conn(0x7faa900a7020 msgr2=0x7faa900a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:26.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa94840640 1 -- 172.21.15.12:0/3416774519 shutdown_connections 2024-09-11T01:31:26.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.650+0000 7faa94840640 1 -- 172.21.15.12:0/3416774519 wait complete. 2024-09-11T01:31:26.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.651+0000 7faa94840640 10 monclient: build_initial_monmap 2024-09-11T01:31:26.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.651+0000 7faa94840640 10 monclient: monmap: 2024-09-11T01:31:26.667 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:26.667 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:31:26.668 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:31:26.668 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:31:26.668 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:26.668 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:26.668 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:31:26.668 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:31:26.668 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:31:26.668 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:26.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.651+0000 7faa94840640 1 Processor -- start 2024-09-11T01:31:26.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.651+0000 7faa94840640 1 -- start start 2024-09-11T01:31:26.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.651+0000 7faa94840640 10 monclient: init 2024-09-11T01:31:26.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa94840640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:26.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa94840640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:31:26.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa94840640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa90137e50 0x7faa9013be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:26.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa94840640 10 monclient(hunting): picked mon.noname-c con 0x7faa90137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:26.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa94840640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa90138230 0x7faa9013c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:26.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa94840640 10 monclient(hunting): picked mon.noname-b con 0x7faa90138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:31:26.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa94840640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa9013d380 0x7faa900a2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:26.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa94840640 10 monclient(hunting): picked mon.noname-a con 0x7faa9013d380 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:26.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa94840640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:26.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa94840640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7faa9006f890 con 0x7faa9013d380 2024-09-11T01:31:26.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa94840640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:26.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa94840640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7faa9006f710 con 0x7faa90138230 2024-09-11T01:31:26.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa94840640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:26.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa94840640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7faa9013fcb0 con 0x7faa90137e50 2024-09-11T01:31:26.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa94840640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:26.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa94840640 10 monclient(hunting): authenticate will time out at 893.095243s 2024-09-11T01:31:26.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8ed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa9013d380 0x7faa900a2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:26.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8e59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa90137e50 0x7faa9013be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:26.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa87fff640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa90138230 msgr2=0x7faa9013c340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:31:26.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8e59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa90137e50 0x7faa9013be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:58012/0 (socket says 172.21.15.12:58012) 2024-09-11T01:31:26.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8ed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa9013d380 0x7faa900a2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56398/0 (socket says 172.21.15.12:56398) 2024-09-11T01:31:26.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8e59c640 1 -- 172.21.15.12:0/64335851 learned_addr learned my addr 172.21.15.12:0/64335851 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:26.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8e59c640 10 monclient(hunting): get_auth_request con 0x7faa90137e50 auth_method 0 2024-09-11T01:31:26.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:26.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:26.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:26.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8ed9d640 10 monclient(hunting): get_auth_request con 0x7faa9013d380 auth_method 0 2024-09-11T01:31:26.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:26.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:26.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:26.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa87fff640 1 --2- 172.21.15.12:0/64335851 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa90138230 0x7faa9013c340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:31:26.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:26.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:26.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:26.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:26.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:26.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:26.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8ed9d640 10 monclient(hunting): handle_auth_done global_id 24183 payload 824 2024-09-11T01:31:26.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8ed9d640 1 -- 172.21.15.12:0/64335851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa90137e50 msgr2=0x7faa9013be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:26.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8ed9d640 1 --2- 172.21.15.12:0/64335851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa90137e50 0x7faa9013be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:26.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8ed9d640 1 -- 172.21.15.12:0/64335851 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa90138230 msgr2=0x7faa9013c340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:31:26.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8ed9d640 1 --2- 172.21.15.12:0/64335851 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa90138230 0x7faa9013c340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:26.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8ed9d640 10 monclient: _finish_hunting 0 2024-09-11T01:31:26.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8ed9d640 1 monclient: found mon.noname-a 2024-09-11T01:31:26.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:26.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:31:26.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8ed9d640 1 -- 172.21.15.12:0/64335851 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa9013a6f0 con 0x7faa9013d380 2024-09-11T01:31:26.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.652+0000 7faa8e59c640 1 --2- 172.21.15.12:0/64335851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa90137e50 0x7faa9013be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:31:26.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.654+0000 7faa8ed9d640 1 -- 172.21.15.12:0/64335851 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa9013d380 msgr2=0x7faa900a2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:26.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.654+0000 7faa8ed9d640 1 --2- 172.21.15.12:0/64335851 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa9013d380 0x7faa900a2520 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7faa8800bd90 tx=0x7faa88009eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:26.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.655+0000 7faa86ffd640 1 -- 172.21.15.12:0/64335851 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa880066f0 con 0x7faa9013d380 2024-09-11T01:31:26.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.655+0000 7faa86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:26.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.655+0000 7faa86ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:31:26.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.655+0000 7faa86ffd640 10 monclient: dump: 2024-09-11T01:31:26.675 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:26.675 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:26.675 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.675 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.676 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:26.676 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:26.676 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:26.676 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:26.676 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:26.676 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:26.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.655+0000 7faa86ffd640 10 monclient: _finish_auth 0 2024-09-11T01:31:26.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.655+0000 7faa86ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:31:26.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.655+0000 7faa86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:26.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.655+0000 7faa94840640 5 monclient: authenticate success, global_id 24183 2024-09-11T01:31:26.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.655+0000 7faa86ffd640 1 -- 172.21.15.12:0/64335851 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa88006890 con 0x7faa9013d380 2024-09-11T01:31:26.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.655+0000 7faa94840640 10 monclient: _renew_subs 2024-09-11T01:31:26.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.655+0000 7faa94840640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:26.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.655+0000 7faa94840640 1 -- 172.21.15.12:0/64335851 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa900a9630 con 0x7faa9013d380 2024-09-11T01:31:26.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.655+0000 7faa86ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:26.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.655+0000 7faa86ffd640 1 -- 172.21.15.12:0/64335851 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa88015440 con 0x7faa9013d380 2024-09-11T01:31:26.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.655+0000 7faa86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:26.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.655+0000 7faa86ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.655+0000 7faa86ffd640 10 monclient: dump: 2024-09-11T01:31:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:26.679 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.655+0000 7faa94840640 10 monclient: _renew_subs 2024-09-11T01:31:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.655+0000 7faa8d59a640 4 set_mon_vals no callback set 2024-09-11T01:31:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.655+0000 7faa94840640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.655+0000 7faa94840640 1 -- 172.21.15.12:0/64335851 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa9013a9d0 con 0x7faa9013d380 2024-09-11T01:31:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.656+0000 7faa94840640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.656+0000 7faa94840640 1 --2- 172.21.15.12:0/64335851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa901cd210 0x7faa900a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.656+0000 7faa94840640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.656+0000 7faa94840640 1 -- 172.21.15.12:0/64335851 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7faa900a4390 con 0x7faa901cd210 2024-09-11T01:31:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.656+0000 7faa86ffd640 1 -- 172.21.15.12:0/64335851 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7faa8801d030 con 0x7faa9013d380 2024-09-11T01:31:26.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.656+0000 7faa8e59c640 1 --2- 172.21.15.12:0/64335851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa901cd210 0x7faa900a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:26.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.656+0000 7faa8e59c640 10 monclient: get_auth_request con 0x7faa901cd210 auth_method 0 2024-09-11T01:31:26.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.656+0000 7faa8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:26.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.656+0000 7faa8e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:26.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.656+0000 7faa8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:26.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.656+0000 7faa8e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:26.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.656+0000 7faa8e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:26.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.656+0000 7faa8e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:26.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.656+0000 7faa8e59c640 10 monclient(hunting): handle_auth_done global_id 4610 payload 824 2024-09-11T01:31:26.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.656+0000 7faa8e59c640 1 -- 172.21.15.12:0/64335851 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faa90138640 con 0x7faa901cd210 2024-09-11T01:31:26.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.656+0000 7faa86ffd640 1 --2- 172.21.15.12:0/64335851 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7faa7003e1c0 0x7faa70040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:26.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.656+0000 7faa86ffd640 1 -- 172.21.15.12:0/64335851 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 4783+0+0 (secure 0 0 0) 0x7faa880539a0 con 0x7faa9013d380 2024-09-11T01:31:26.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.656+0000 7faa87fff640 1 --2- 172.21.15.12:0/64335851 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7faa7003e1c0 0x7faa70040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:26.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.656+0000 7faa87fff640 10 monclient: get_auth_request con 0x7faa7003e1c0 auth_method 0 2024-09-11T01:31:26.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.658+0000 7faa8e59c640 1 -- 172.21.15.12:0/64335851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa901cd210 msgr2=0x7faa900a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:26.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.658+0000 7faa8e59c640 1 --2- 172.21.15.12:0/64335851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa901cd210 0x7faa900a2a60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7faa800053f0 tx=0x7faa800058b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:26.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.658+0000 7faa86ffd640 1 -- 172.21.15.12:0/64335851 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa80016070 con 0x7faa901cd210 2024-09-11T01:31:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.658+0000 7faa86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.658+0000 7faa86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.658+0000 7faa86ffd640 10 monclient: dump: 2024-09-11T01:31:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:26.683 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.659+0000 7faa86ffd640 1 -- 172.21.15.12:0/64335851 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7faa8000ee20 con 0x7faa901cd210 2024-09-11T01:31:26.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.659+0000 7faa86ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:26.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.659+0000 7faa86ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:31:26.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.659+0000 7faa86ffd640 1 -- 172.21.15.12:0/64335851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa901cd210 msgr2=0x7faa900a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:26.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.659+0000 7faa86ffd640 1 --2- 172.21.15.12:0/64335851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa901cd210 0x7faa900a2a60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7faa800053f0 tx=0x7faa800058b0 comp rx=0 tx=0).stop 2024-09-11T01:31:26.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.659+0000 7faa86ffd640 1 -- 172.21.15.12:0/64335851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa901cd210 msgr2=0x7faa900a2a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:31:26.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.659+0000 7faa86ffd640 1 --2- 172.21.15.12:0/64335851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa901cd210 0x7faa900a2a60 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7faa800053f0 tx=0x7faa800058b0 comp rx=0 tx=0).stop 2024-09-11T01:31:26.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.659+0000 7faa87fff640 1 --2- 172.21.15.12:0/64335851 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7faa7003e1c0 0x7faa70040680 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7faa900ab580 tx=0x7faa7802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:26.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.665+0000 7faa94840640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:31:26.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.665+0000 7faa94840640 1 --2- 172.21.15.12:0/64335851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa900a2fa0 0x7faa9013dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:26.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.665+0000 7faa94840640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:26.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.665+0000 7faa94840640 1 -- 172.21.15.12:0/64335851 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7faa900a3380 con 0x7faa900a2fa0 2024-09-11T01:31:26.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.665+0000 7faa8e59c640 1 --2- 172.21.15.12:0/64335851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa900a2fa0 0x7faa9013dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:26.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.665+0000 7faa8e59c640 10 monclient: get_auth_request con 0x7faa900a2fa0 auth_method 0 2024-09-11T01:31:26.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.665+0000 7faa8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:26.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.665+0000 7faa8e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:26.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.665+0000 7faa8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:26.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.665+0000 7faa8e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:26.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.665+0000 7faa8e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:26.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.665+0000 7faa8e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:26.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.665+0000 7faa8e59c640 10 monclient(hunting): handle_auth_done global_id 4613 payload 824 2024-09-11T01:31:26.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.665+0000 7faa8e59c640 1 -- 172.21.15.12:0/64335851 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faa90069950 con 0x7faa900a2fa0 2024-09-11T01:31:26.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.668+0000 7faa8e59c640 1 -- 172.21.15.12:0/64335851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa900a2fa0 msgr2=0x7faa9013dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:26.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.668+0000 7faa8e59c640 1 --2- 172.21.15.12:0/64335851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa900a2fa0 0x7faa9013dd70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7faa80018000 tx=0x7faa800059d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:26.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.668+0000 7faa86ffd640 1 -- 172.21.15.12:0/64335851 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa80016040 con 0x7faa900a2fa0 2024-09-11T01:31:26.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.668+0000 7faa86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:26.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.668+0000 7faa86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:26.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.668+0000 7faa86ffd640 10 monclient: dump: 2024-09-11T01:31:26.689 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:26.689 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:26.689 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.689 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:26.689 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:26.689 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:26.689 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:26.689 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:26.689 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:26.689 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:26.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.668+0000 7faa86ffd640 1 -- 172.21.15.12:0/64335851 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4697 (secure 0 0 0) 0x7faa80005c50 con 0x7faa900a2fa0 2024-09-11T01:31:26.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.668+0000 7faa86ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:31:26.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.668+0000 7faa86ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:31:26.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.668+0000 7faa86ffd640 1 -- 172.21.15.12:0/64335851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa900a2fa0 msgr2=0x7faa9013dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:26.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.668+0000 7faa86ffd640 1 --2- 172.21.15.12:0/64335851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa900a2fa0 0x7faa9013dd70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7faa80018000 tx=0x7faa800059d0 comp rx=0 tx=0).stop 2024-09-11T01:31:26.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.668+0000 7faa86ffd640 1 -- 172.21.15.12:0/64335851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa900a2fa0 msgr2=0x7faa9013dd70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:31:26.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.668+0000 7faa86ffd640 1 --2- 172.21.15.12:0/64335851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa900a2fa0 0x7faa9013dd70 secure :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7faa80018000 tx=0x7faa800059d0 comp rx=0 tx=0).stop 2024-09-11T01:31:26.690 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:31:26.691 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:31:26.691 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:31:26.691 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:31:26.691 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 26, 2024-09-11T01:31:26.691 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 149357, 2024-09-11T01:31:26.691 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:31:26.691 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:31:26.691 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:31:26.691 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:26.691 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:31:26.691 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:31:26.691 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:31:26.692 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:31:26.692 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:26.692 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:26.692 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:26.692 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:26.692 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:26.692 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:26.692 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:26.693 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:26.693 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:26.693 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:26.693 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:26.693 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:26.694 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:31:26.694 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:31:26.694 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:26.694 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:26.694 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:26.694 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:26.694 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:26.694 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:26.694 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:26.694 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:26.694 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:26.695 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:26.695 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:26.695 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:26.695 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:26.695 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:31:26.695 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:31:26.695 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:31:26.695 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:31:26.695 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:31:26.695 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:31:26.695 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:31:26.695 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:31:26.695 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:31:26.695 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:31:26.696 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:31:26.696 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:31:26.696 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:31:26.696 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:31:26.696 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:31:26.696 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:31:26.696 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:31:26.696 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:26.696 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:26.696 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:26.696 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:26.696 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:26.696 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:26.697 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:26.697 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:26.697 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:26.697 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:26.697 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:26.697 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:26.697 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:31:26.697 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:26.697 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:31:26.697 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:26.697 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:31:26.697 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:31:26.698 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:26.698 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:26.698 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:26.698 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:26.698 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:31:26.698 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:26.698 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:26.698 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:26.698 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:26.698 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:31:26.698 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:26.698 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:26.699 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:26.699 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:26.699 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:31:26.699 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:31:26.699 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:26.699 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:26.699 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:26.699 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:26.699 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:26.699 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:31:26.699 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:31:26.699 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:26.699 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:26.700 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:26.700 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:26.700 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:31:26.700 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:26.700 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:26.700 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:26.700 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:26.700 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:31:26.700 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:26.700 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:26.700 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:26.700 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:26.700 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:31:26.701 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:31:26.701 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:26.701 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:26.701 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:26.701 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:26.701 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:26.701 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:31:26.701 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:31:26.701 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:26.701 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:26.701 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:26.701 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:26.701 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:31:26.702 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:26.702 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:26.702 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:26.702 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:26.702 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:31:26.702 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:26.702 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:26.702 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:26.702 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:26.702 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:31:26.702 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:31:26.702 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:26.702 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:26.702 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:26.703 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:26.703 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:26.703 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:26.703 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:31:26.703 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:31:26.703 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:26.703 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:26.703 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:26.703 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:31:26.703 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:26.703 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:26.703 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:31:26.703 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:26.704 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:26.704 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:26.704 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:31:26.704 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:26.704 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:26.704 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:31:26.704 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:26.704 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:26.704 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:26.704 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:31:26.704 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:26.704 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:26.704 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:26.704 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:31:26.705 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:31:26.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.668+0000 7faa94840640 1 -- 172.21.15.12:0/64335851 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7faa7003e1c0 msgr2=0x7faa70040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:26.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.668+0000 7faa94840640 1 --2- 172.21.15.12:0/64335851 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7faa7003e1c0 0x7faa70040680 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7faa900ab580 tx=0x7faa7802e000 comp rx=0 tx=0).stop 2024-09-11T01:31:26.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.668+0000 7faa94840640 10 monclient: shutdown 2024-09-11T01:31:26.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.668+0000 7faa94840640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:26.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.668+0000 7faa94840640 1 -- 172.21.15.12:0/64335851 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa9013d380 msgr2=0x7faa900a2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:26.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.668+0000 7faa94840640 1 --2- 172.21.15.12:0/64335851 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa9013d380 0x7faa900a2520 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7faa8800bd90 tx=0x7faa88009eb0 comp rx=0 tx=0).stop 2024-09-11T01:31:26.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.668+0000 7faa8ed9d640 1 -- 172.21.15.12:0/64335851 reap_dead start 2024-09-11T01:31:26.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.668+0000 7faa94840640 1 -- 172.21.15.12:0/64335851 shutdown_connections 2024-09-11T01:31:26.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.668+0000 7faa94840640 1 --2- 172.21.15.12:0/64335851 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa9013d380 0x7faa900a2520 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:26.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.668+0000 7faa94840640 1 -- 172.21.15.12:0/64335851 >> 172.21.15.12:0/64335851 conn(0x7faa900a7020 msgr2=0x7faa90136ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:26.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.668+0000 7faa94840640 1 -- 172.21.15.12:0/64335851 shutdown_connections 2024-09-11T01:31:26.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:26.668+0000 7faa94840640 1 -- 172.21.15.12:0/64335851 wait complete. 2024-09-11T01:31:26.707 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:31:26.707 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:31:27.097 INFO:tasks.rados.rados.0.smithi071.stdout:244: done (5 left) 2024-09-11T01:31:27.097 INFO:tasks.rados.rados.0.smithi071.stdout:245: done (4 left) 2024-09-11T01:31:27.097 INFO:tasks.rados.rados.0.smithi071.stdout:246: done (3 left) 2024-09-11T01:31:27.098 INFO:tasks.rados.rados.0.smithi071.stdout:247: done (2 left) 2024-09-11T01:31:27.098 INFO:tasks.rados.rados.0.smithi071.stdout:248: done (1 left) 2024-09-11T01:31:27.098 INFO:tasks.rados.rados.0.smithi071.stdout:249: done (0 left) 2024-09-11T01:31:27.098 INFO:tasks.rados.rados.0.smithi071.stdout:250: delete oid 18 current snap is 26 2024-09-11T01:31:27.102 INFO:tasks.rados.rados.0.smithi071.stdout:250: done (0 left) 2024-09-11T01:31:27.102 INFO:tasks.rados.rados.0.smithi071.stdout:251: write oid 47 current snap is 26 2024-09-11T01:31:27.102 INFO:tasks.rados.rados.0.smithi071.stdout:251: seq_num 109 ranges {631963=631083,2061949=560019,3243812=504767} 2024-09-11T01:31:27.108 INFO:tasks.rados.rados.0.smithi071.stdout:251: writing smithi07134777-47 from 631963 to 1263046 tid 1 2024-09-11T01:31:27.116 INFO:tasks.rados.rados.0.smithi071.stdout:251: writing smithi07134777-47 from 2061949 to 2621968 tid 2 2024-09-11T01:31:27.123 INFO:tasks.rados.rados.0.smithi071.stdout:251: writing smithi07134777-47 from 3243812 to 3748579 tid 3 2024-09-11T01:31:27.123 INFO:tasks.rados.rados.0.smithi071.stdout:252: rollback oid 16 current snap is 26 2024-09-11T01:31:27.123 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 16 to 23 2024-09-11T01:31:27.123 INFO:tasks.rados.rados.0.smithi071.stdout:251: finishing write tid 1 to smithi07134777-47 2024-09-11T01:31:27.123 INFO:tasks.rados.rados.0.smithi071.stdout:251: finishing write tid 2 to smithi07134777-47 2024-09-11T01:31:27.123 INFO:tasks.rados.rados.0.smithi071.stdout:253: delete oid 13 current snap is 26 2024-09-11T01:31:27.127 INFO:tasks.rados.rados.0.smithi071.stdout:251: finishing write tid 3 to smithi07134777-47 2024-09-11T01:31:27.127 INFO:tasks.rados.rados.0.smithi071.stdout:251: finishing write tid 4 to smithi07134777-47 2024-09-11T01:31:27.128 INFO:tasks.rados.rados.0.smithi071.stdout:251: finishing write tid 5 to smithi07134777-47 2024-09-11T01:31:27.128 INFO:tasks.rados.rados.0.smithi071.stdout:251: oid 47 updating version 0 to 76 2024-09-11T01:31:27.128 INFO:tasks.rados.rados.0.smithi071.stdout:251: oid 47 version 76 is already newer than 75 2024-09-11T01:31:27.128 INFO:tasks.rados.rados.0.smithi071.stdout:251: oid 47 version 76 is already newer than 73 2024-09-11T01:31:27.128 INFO:tasks.rados.rados.0.smithi071.stdout:251: oid 47 version 76 is already newer than 74 2024-09-11T01:31:27.129 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 76 (ObjNum 109 snap 26 seq_num 109) dirty exists 2024-09-11T01:31:27.129 INFO:tasks.rados.rados.0.smithi071.stdout:251: left oid 47 (ObjNum 109 snap 26 seq_num 109) 2024-09-11T01:31:27.129 INFO:tasks.rados.rados.0.smithi071.stdout:251: done (2 left) 2024-09-11T01:31:27.129 INFO:tasks.rados.rados.0.smithi071.stdout:253: done (1 left) 2024-09-11T01:31:27.129 INFO:tasks.rados.rados.0.smithi071.stdout:254: snap_remove snap 24 2024-09-11T01:31:28.100 INFO:tasks.rados.rados.0.smithi071.stdout:254: done (1 left) 2024-09-11T01:31:28.100 INFO:tasks.rados.rados.0.smithi071.stdout:255: snap_remove snap 25 2024-09-11T01:31:29.102 INFO:tasks.rados.rados.0.smithi071.stdout:255: done (1 left) 2024-09-11T01:31:29.103 INFO:tasks.rados.rados.0.smithi071.stdout:256: read oid 20 snap -1 2024-09-11T01:31:29.103 INFO:tasks.rados.rados.0.smithi071.stdout:256: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-11T01:31:29.103 INFO:tasks.rados.rados.0.smithi071.stdout:257: copy_from oid 1 from oid 11 current snap is 26 2024-09-11T01:31:29.103 INFO:tasks.rados.rados.0.smithi071.stdout:252: finishing rollback tid 0 to smithi07134777-16 2024-09-11T01:31:29.103 INFO:tasks.rados.rados.0.smithi071.stdout:252: finishing rollback tid 1 to smithi07134777-16 2024-09-11T01:31:29.103 INFO:tasks.rados.rados.0.smithi071.stdout:252: finishing rollback tid 2 to smithi07134777-16 2024-09-11T01:31:29.103 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 71 (ObjNum 97 snap 22 seq_num 97) dirty exists 2024-09-11T01:31:29.103 INFO:tasks.rados.rados.0.smithi071.stdout:252: done (2 left) 2024-09-11T01:31:29.103 INFO:tasks.rados.rados.0.smithi071.stdout:258: write oid 7 current snap is 26 2024-09-11T01:31:29.103 INFO:tasks.rados.rados.0.smithi071.stdout:258: seq_num 111 ranges {665577=781581,2062560=563688,3182305=643664} 2024-09-11T01:31:29.110 INFO:tasks.rados.rados.0.smithi071.stdout:258: writing smithi07134777-7 from 665577 to 1447158 tid 1 2024-09-11T01:31:29.117 INFO:tasks.rados.rados.0.smithi071.stdout:258: writing smithi07134777-7 from 2062560 to 2626248 tid 2 2024-09-11T01:31:29.124 INFO:tasks.rados.rados.0.smithi071.stdout:258: writing smithi07134777-7 from 3182305 to 3825969 tid 3 2024-09-11T01:31:29.124 INFO:tasks.rados.rados.0.smithi071.stdout:259: read oid 38 snap -1 2024-09-11T01:31:29.124 INFO:tasks.rados.rados.0.smithi071.stdout:259: expect (ObjNum 90 snap 19 seq_num 90) 2024-09-11T01:31:29.124 INFO:tasks.rados.rados.0.smithi071.stdout:260: snap_remove snap 23 2024-09-11T01:31:30.109 INFO:tasks.rados.rados.0.smithi071.stdout:260: done (4 left) 2024-09-11T01:31:30.109 INFO:tasks.rados.rados.0.smithi071.stdout:261: snap_create 2024-09-11T01:31:30.109 INFO:tasks.rados.rados.0.smithi071.stdout:256: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-11T01:31:30.146 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:31:30.168 INFO:tasks.rados.rados.0.smithi071.stdout:258: finishing write tid 1 to smithi07134777-7 2024-09-11T01:31:30.168 INFO:tasks.rados.rados.0.smithi071.stdout:258: finishing write tid 2 to smithi07134777-7 2024-09-11T01:31:30.168 INFO:tasks.rados.rados.0.smithi071.stdout:257: finishing copy_from racing read to smithi07134777-1 2024-09-11T01:31:30.168 INFO:tasks.rados.rados.0.smithi071.stdout:257: finishing copy_from to smithi07134777-1 2024-09-11T01:31:30.169 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 48 (ObjNum 92 snap 19 seq_num 92) dirty exists 2024-09-11T01:31:30.169 INFO:tasks.rados.rados.0.smithi071.stdout:258: finishing write tid 3 to smithi07134777-7 2024-09-11T01:31:30.169 INFO:tasks.rados.rados.0.smithi071.stdout:258: finishing write tid 4 to smithi07134777-7 2024-09-11T01:31:30.169 INFO:tasks.rados.rados.0.smithi071.stdout:258: finishing write tid 5 to smithi07134777-7 2024-09-11T01:31:30.169 INFO:tasks.rados.rados.0.smithi071.stdout:258: oid 7 updating version 0 to 138 2024-09-11T01:31:30.169 INFO:tasks.rados.rados.0.smithi071.stdout:258: oid 7 version 138 is already newer than 137 2024-09-11T01:31:30.169 INFO:tasks.rados.rados.0.smithi071.stdout:258: oid 7 version 138 is already newer than 135 2024-09-11T01:31:30.169 INFO:tasks.rados.rados.0.smithi071.stdout:258: oid 7 version 138 is already newer than 136 2024-09-11T01:31:30.169 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 138 (ObjNum 111 snap 26 seq_num 111) dirty exists 2024-09-11T01:31:30.169 INFO:tasks.rados.rados.0.smithi071.stdout:258: left oid 7 (ObjNum 111 snap 26 seq_num 111) 2024-09-11T01:31:30.169 INFO:tasks.rados.rados.0.smithi071.stdout:259: expect (ObjNum 90 snap 19 seq_num 90) 2024-09-11T01:31:30.193 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2623637 2024-09-11T01:31:31.109 INFO:tasks.rados.rados.0.smithi071.stdout:256: done (4 left) 2024-09-11T01:31:31.110 INFO:tasks.rados.rados.0.smithi071.stdout:257: done (3 left) 2024-09-11T01:31:31.110 INFO:tasks.rados.rados.0.smithi071.stdout:258: done (2 left) 2024-09-11T01:31:31.110 INFO:tasks.rados.rados.0.smithi071.stdout:259: done (1 left) 2024-09-11T01:31:31.110 INFO:tasks.rados.rados.0.smithi071.stdout:261: done (0 left) 2024-09-11T01:31:31.110 INFO:tasks.rados.rados.0.smithi071.stdout:262: write (excl) oid 23 current snap is 27 2024-09-11T01:31:31.110 INFO:tasks.rados.rados.0.smithi071.stdout:262: seq_num 112 ranges {682384=456830,1721042=423699,2809728=454936,3731357=151027} 2024-09-11T01:31:31.115 INFO:tasks.rados.rados.0.smithi071.stdout:262: writing smithi07134777-23 from 682384 to 1139214 tid 1 2024-09-11T01:31:31.121 INFO:tasks.rados.rados.0.smithi071.stdout:262: writing smithi07134777-23 from 1721042 to 2144741 tid 2 2024-09-11T01:31:31.127 INFO:tasks.rados.rados.0.smithi071.stdout:262: writing smithi07134777-23 from 2809728 to 3264664 tid 3 2024-09-11T01:31:31.130 INFO:tasks.rados.rados.0.smithi071.stdout:262: writing smithi07134777-23 from 3731357 to 3882384 tid 4 2024-09-11T01:31:31.130 INFO:tasks.rados.rados.0.smithi071.stdout:263: snap_remove snap 26 2024-09-11T01:31:32.070 INFO:tasks.rados.rados.0.smithi071.stdout:263: done (1 left) 2024-09-11T01:31:32.070 INFO:tasks.rados.rados.0.smithi071.stdout:264: copy_from oid 18 from oid 47 current snap is 27 2024-09-11T01:31:32.070 INFO:tasks.rados.rados.0.smithi071.stdout:262: finishing write tid 1 to smithi07134777-23 2024-09-11T01:31:32.071 INFO:tasks.rados.rados.0.smithi071.stdout:262: finishing write tid 2 to smithi07134777-23 2024-09-11T01:31:32.071 INFO:tasks.rados.rados.0.smithi071.stdout:262: finishing write tid 3 to smithi07134777-23 2024-09-11T01:31:32.071 INFO:tasks.rados.rados.0.smithi071.stdout:262: finishing write tid 4 to smithi07134777-23 2024-09-11T01:31:32.071 INFO:tasks.rados.rados.0.smithi071.stdout:262: finishing write tid 5 to smithi07134777-23 2024-09-11T01:31:32.071 INFO:tasks.rados.rados.0.smithi071.stdout:262: finishing write tid 6 to smithi07134777-23 2024-09-11T01:31:32.071 INFO:tasks.rados.rados.0.smithi071.stdout:262: oid 23 updating version 0 to 52 2024-09-11T01:31:32.071 INFO:tasks.rados.rados.0.smithi071.stdout:262: oid 23 version 52 is already newer than 51 2024-09-11T01:31:32.071 INFO:tasks.rados.rados.0.smithi071.stdout:262: oid 23 version 52 is already newer than 50 2024-09-11T01:31:32.071 INFO:tasks.rados.rados.0.smithi071.stdout:262: oid 23 updating version 52 to 54 2024-09-11T01:31:32.071 INFO:tasks.rados.rados.0.smithi071.stdout:262: oid 23 version 54 is already newer than 53 2024-09-11T01:31:32.071 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 54 (ObjNum 112 snap 27 seq_num 112) dirty exists 2024-09-11T01:31:32.071 INFO:tasks.rados.rados.0.smithi071.stdout:262: left oid 23 (ObjNum 112 snap 27 seq_num 112) 2024-09-11T01:31:32.071 INFO:tasks.rados.rados.0.smithi071.stdout:262: done (1 left) 2024-09-11T01:31:32.071 INFO:tasks.rados.rados.0.smithi071.stdout:265: delete oid 13 current snap is 27 2024-09-11T01:31:32.075 INFO:tasks.rados.rados.0.smithi071.stdout:265: done (1 left) 2024-09-11T01:31:32.075 INFO:tasks.rados.rados.0.smithi071.stdout:266: write oid 12 current snap is 27 2024-09-11T01:31:32.075 INFO:tasks.rados.rados.0.smithi071.stdout:266: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2024-09-11T01:31:32.082 INFO:tasks.rados.rados.0.smithi071.stdout:266: writing smithi07134777-12 from 715998 to 1323326 tid 1 2024-09-11T01:31:32.087 INFO:tasks.rados.rados.0.smithi071.stdout:266: writing smithi07134777-12 from 1838006 to 2265374 tid 2 2024-09-11T01:31:32.093 INFO:tasks.rados.rados.0.smithi071.stdout:266: writing smithi07134777-12 from 2864574 to 3342054 tid 3 2024-09-11T01:31:32.096 INFO:tasks.rados.rados.0.smithi071.stdout:266: writing smithi07134777-12 from 3863835 to 3915998 tid 4 2024-09-11T01:31:32.096 INFO:tasks.rados.rados.0.smithi071.stdout:267: read oid 35 snap -1 2024-09-11T01:31:32.096 INFO:tasks.rados.rados.0.smithi071.stdout:267: expect (ObjNum 100 snap 24 seq_num 100) 2024-09-11T01:31:32.097 INFO:tasks.rados.rados.0.smithi071.stdout:268: write (excl) oid 7 current snap is 27 2024-09-11T01:31:32.097 INFO:tasks.rados.rados.0.smithi071.stdout:268: seq_num 115 ranges {732805=798930,2296488=571026,3291997=405105} 2024-09-11T01:31:32.104 INFO:tasks.rados.rados.0.smithi071.stdout:268: writing smithi07134777-7 from 732805 to 1531735 tid 1 2024-09-11T01:31:32.112 INFO:tasks.rados.rados.0.smithi071.stdout:268: writing smithi07134777-7 from 2296488 to 2867514 tid 2 2024-09-11T01:31:32.118 INFO:tasks.rados.rados.0.smithi071.stdout:268: writing smithi07134777-7 from 3291997 to 3697102 tid 3 2024-09-11T01:31:32.118 INFO:tasks.rados.rados.0.smithi071.stdout:269: copy_from oid 1 from oid 48 current snap is 27 2024-09-11T01:31:32.118 INFO:tasks.rados.rados.0.smithi071.stdout:266: finishing write tid 1 to smithi07134777-12 2024-09-11T01:31:32.118 INFO:tasks.rados.rados.0.smithi071.stdout:264: finishing copy_from racing read to smithi07134777-18 2024-09-11T01:31:32.118 INFO:tasks.rados.rados.0.smithi071.stdout:270: write oid 30 current snap is 27 2024-09-11T01:31:32.118 INFO:tasks.rados.rados.0.smithi071.stdout:266: finishing write tid 2 to smithi07134777-12 2024-09-11T01:31:32.118 INFO:tasks.rados.rados.0.smithi071.stdout:270: seq_num 116 ranges {749612=474179,1954970=431037,2919420=616377} 2024-09-11T01:31:32.124 INFO:tasks.rados.rados.0.smithi071.stdout:270: writing smithi07134777-30 from 749612 to 1223791 tid 1 2024-09-11T01:31:32.131 INFO:tasks.rados.rados.0.smithi071.stdout:270: writing smithi07134777-30 from 1954970 to 2386007 tid 2 2024-09-11T01:31:32.138 INFO:tasks.rados.rados.0.smithi071.stdout:270: writing smithi07134777-30 from 2919420 to 3535797 tid 3 2024-09-11T01:31:32.138 INFO:tasks.rados.rados.0.smithi071.stdout:271: read oid 43 snap -1 2024-09-11T01:31:32.138 INFO:tasks.rados.rados.0.smithi071.stdout:271: expect (ObjNum 74 snap 13 seq_num 74) 2024-09-11T01:31:32.138 INFO:tasks.rados.rados.0.smithi071.stdout:272: delete oid 46 current snap is 27 2024-09-11T01:31:32.138 INFO:tasks.rados.rados.0.smithi071.stdout:264: finishing copy_from to smithi07134777-18 2024-09-11T01:31:32.138 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 76 (ObjNum 109 snap 26 seq_num 109) dirty exists 2024-09-11T01:31:32.138 INFO:tasks.rados.rados.0.smithi071.stdout:266: finishing write tid 3 to smithi07134777-12 2024-09-11T01:31:32.138 INFO:tasks.rados.rados.0.smithi071.stdout:266: finishing write tid 4 to smithi07134777-12 2024-09-11T01:31:32.138 INFO:tasks.rados.rados.0.smithi071.stdout:266: finishing write tid 5 to smithi07134777-12 2024-09-11T01:31:32.138 INFO:tasks.rados.rados.0.smithi071.stdout:266: finishing write tid 6 to smithi07134777-12 2024-09-11T01:31:32.138 INFO:tasks.rados.rados.0.smithi071.stdout:266: oid 12 updating version 0 to 134 2024-09-11T01:31:32.138 INFO:tasks.rados.rados.0.smithi071.stdout:266: oid 12 version 134 is already newer than 131 2024-09-11T01:31:32.138 INFO:tasks.rados.rados.0.smithi071.stdout:266: oid 12 version 134 is already newer than 132 2024-09-11T01:31:32.139 INFO:tasks.rados.rados.0.smithi071.stdout:266: oid 12 version 134 is already newer than 133 2024-09-11T01:31:32.139 INFO:tasks.rados.rados.0.smithi071.stdout:266: oid 12 updating version 134 to 135 2024-09-11T01:31:32.139 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 135 (ObjNum 114 snap 27 seq_num 114) dirty exists 2024-09-11T01:31:32.139 INFO:tasks.rados.rados.0.smithi071.stdout:266: left oid 12 (ObjNum 114 snap 27 seq_num 114) 2024-09-11T01:31:32.139 INFO:tasks.rados.rados.0.smithi071.stdout:268: finishing write tid 1 to smithi07134777-7 2024-09-11T01:31:32.139 INFO:tasks.rados.rados.0.smithi071.stdout:267: expect (ObjNum 100 snap 24 seq_num 100) 2024-09-11T01:31:32.215 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 887371 2024-09-11T01:31:32.279 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:31:32.356 INFO:tasks.rados.rados.0.smithi071.stdout:268: finishing write tid 2 to smithi07134777-7 2024-09-11T01:31:32.356 INFO:tasks.rados.rados.0.smithi071.stdout:268: finishing write tid 3 to smithi07134777-7 2024-09-11T01:31:32.356 INFO:tasks.rados.rados.0.smithi071.stdout:269: finishing copy_from to smithi07134777-1 2024-09-11T01:31:32.356 INFO:tasks.rados.rados.0.smithi071.stdout:269: got expected ENOENT (src dne) 2024-09-11T01:31:32.357 INFO:tasks.rados.rados.0.smithi071.stdout:269: finishing copy_from racing read to smithi07134777-1 2024-09-11T01:31:32.357 INFO:tasks.rados.rados.0.smithi071.stdout:268: finishing write tid 4 to smithi07134777-7 2024-09-11T01:31:32.357 INFO:tasks.rados.rados.0.smithi071.stdout:268: finishing write tid 5 to smithi07134777-7 2024-09-11T01:31:32.357 INFO:tasks.rados.rados.0.smithi071.stdout:268: oid 7 updating version 0 to 144 2024-09-11T01:31:32.357 INFO:tasks.rados.rados.0.smithi071.stdout:268: oid 7 version 144 is already newer than 143 2024-09-11T01:31:32.357 INFO:tasks.rados.rados.0.smithi071.stdout:268: oid 7 updating version 144 to 145 2024-09-11T01:31:32.357 INFO:tasks.rados.rados.0.smithi071.stdout:268: oid 7 updating version 145 to 146 2024-09-11T01:31:32.357 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 146 (ObjNum 115 snap 27 seq_num 115) dirty exists 2024-09-11T01:31:32.357 INFO:tasks.rados.rados.0.smithi071.stdout:268: left oid 7 (ObjNum 115 snap 27 seq_num 115) 2024-09-11T01:31:32.357 INFO:tasks.rados.rados.0.smithi071.stdout:270: finishing write tid 1 to smithi07134777-30 2024-09-11T01:31:32.357 INFO:tasks.rados.rados.0.smithi071.stdout:264: done (7 left) 2024-09-11T01:31:32.357 INFO:tasks.rados.rados.0.smithi071.stdout:266: done (6 left) 2024-09-11T01:31:32.357 INFO:tasks.rados.rados.0.smithi071.stdout:267: done (5 left) 2024-09-11T01:31:32.357 INFO:tasks.rados.rados.0.smithi071.stdout:268: done (4 left) 2024-09-11T01:31:32.358 INFO:tasks.rados.rados.0.smithi071.stdout:269: done (3 left) 2024-09-11T01:31:32.358 INFO:tasks.rados.rados.0.smithi071.stdout:272: done (2 left) 2024-09-11T01:31:32.358 INFO:tasks.rados.rados.0.smithi071.stdout:273: rollback oid 38 current snap is 27 2024-09-11T01:31:32.358 INFO:tasks.rados.rados.0.smithi071.stdout:273: done (2 left) 2024-09-11T01:31:32.358 INFO:tasks.rados.rados.0.smithi071.stdout:274: rollback oid 23 current snap is 27 2024-09-11T01:31:32.358 INFO:tasks.rados.rados.0.smithi071.stdout:274: done (2 left) 2024-09-11T01:31:32.358 INFO:tasks.rados.rados.0.smithi071.stdout:275: read oid 41 snap -1 2024-09-11T01:31:32.358 INFO:tasks.rados.rados.0.smithi071.stdout:275: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-11T01:31:32.358 INFO:tasks.rados.rados.0.smithi071.stdout:276: snap_create 2024-09-11T01:31:32.358 INFO:tasks.rados.rados.0.smithi071.stdout:270: finishing write tid 2 to smithi07134777-30 2024-09-11T01:31:32.358 INFO:tasks.rados.rados.0.smithi071.stdout:270: finishing write tid 3 to smithi07134777-30 2024-09-11T01:31:32.358 INFO:tasks.rados.rados.0.smithi071.stdout:270: finishing write tid 4 to smithi07134777-30 2024-09-11T01:31:32.358 INFO:tasks.rados.rados.0.smithi071.stdout:270: finishing write tid 5 to smithi07134777-30 2024-09-11T01:31:32.358 INFO:tasks.rados.rados.0.smithi071.stdout:270: oid 30 updating version 0 to 95 2024-09-11T01:31:32.359 INFO:tasks.rados.rados.0.smithi071.stdout:270: oid 30 version 95 is already newer than 94 2024-09-11T01:31:32.359 INFO:tasks.rados.rados.0.smithi071.stdout:270: oid 30 version 95 is already newer than 93 2024-09-11T01:31:32.359 INFO:tasks.rados.rados.0.smithi071.stdout:270: oid 30 version 95 is already newer than 92 2024-09-11T01:31:32.359 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 95 (ObjNum 116 snap 27 seq_num 116) dirty exists 2024-09-11T01:31:32.359 INFO:tasks.rados.rados.0.smithi071.stdout:270: left oid 30 (ObjNum 116 snap 27 seq_num 116) 2024-09-11T01:31:32.359 INFO:tasks.rados.rados.0.smithi071.stdout:271: expect (ObjNum 74 snap 13 seq_num 74) 2024-09-11T01:31:32.359 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:31:32.441 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2590874 2024-09-11T01:31:32.472 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2251470 2024-09-11T01:31:32.498 INFO:tasks.rados.rados.0.smithi071.stdout:275: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-11T01:31:32.498 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:31:32.553 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 9058 2024-09-11T01:31:33.129 INFO:tasks.rados.rados.0.smithi071.stdout:270: done (3 left) 2024-09-11T01:31:33.129 INFO:tasks.rados.rados.0.smithi071.stdout:271: done (2 left) 2024-09-11T01:31:33.129 INFO:tasks.rados.rados.0.smithi071.stdout:275: done (1 left) 2024-09-11T01:31:33.129 INFO:tasks.rados.rados.0.smithi071.stdout:276: done (0 left) 2024-09-11T01:31:33.130 INFO:tasks.rados.rados.0.smithi071.stdout:277: read oid 32 snap -1 2024-09-11T01:31:33.130 INFO:tasks.rados.rados.0.smithi071.stdout:277: expect (ObjNum 31 snap 0 seq_num 31) 2024-09-11T01:31:33.130 INFO:tasks.rados.rados.0.smithi071.stdout:278: snap_remove snap 27 2024-09-11T01:31:34.139 INFO:tasks.rados.rados.0.smithi071.stdout:278: done (1 left) 2024-09-11T01:31:34.139 INFO:tasks.rados.rados.0.smithi071.stdout:279: write (excl) oid 39 current snap is 28 2024-09-11T01:31:34.139 INFO:tasks.rados.rados.0.smithi071.stdout:279: seq_num 118 ranges {783226=624677,2071934=434706,2974266=638921} 2024-09-11T01:31:34.146 INFO:tasks.rados.rados.0.smithi071.stdout:279: writing smithi07134777-39 from 783226 to 1407903 tid 1 2024-09-11T01:31:34.152 INFO:tasks.rados.rados.0.smithi071.stdout:279: writing smithi07134777-39 from 2071934 to 2506640 tid 2 2024-09-11T01:31:34.160 INFO:tasks.rados.rados.0.smithi071.stdout:279: writing smithi07134777-39 from 2974266 to 3613187 tid 3 2024-09-11T01:31:34.160 INFO:tasks.rados.rados.0.smithi071.stdout:280: write (excl) oid 11 current snap is 28 2024-09-11T01:31:34.160 INFO:tasks.rados.rados.0.smithi071.stdout:280: seq_num 119 ranges {400033=699926,1614063=385970} 2024-09-11T01:31:34.164 INFO:tasks.rados.rados.0.smithi071.stdout:280: writing smithi07134777-11 from 400033 to 1099959 tid 1 2024-09-11T01:31:34.169 INFO:tasks.rados.rados.0.smithi071.stdout:280: writing smithi07134777-11 from 1614063 to 2000033 tid 2 2024-09-11T01:31:34.169 INFO:tasks.rados.rados.0.smithi071.stdout:281: copy_from oid 42 from oid 25 current snap is 28 2024-09-11T01:31:34.169 INFO:tasks.rados.rados.0.smithi071.stdout:277: expect (ObjNum 31 snap 0 seq_num 31) 2024-09-11T01:31:34.222 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 746957 2024-09-11T01:31:34.242 INFO:tasks.rados.rados.0.smithi071.stdout:279: finishing write tid 1 to smithi07134777-39 2024-09-11T01:31:34.242 INFO:tasks.rados.rados.0.smithi071.stdout:279: finishing write tid 2 to smithi07134777-39 2024-09-11T01:31:34.242 INFO:tasks.rados.rados.0.smithi071.stdout:279: finishing write tid 3 to smithi07134777-39 2024-09-11T01:31:34.242 INFO:tasks.rados.rados.0.smithi071.stdout:279: finishing write tid 4 to smithi07134777-39 2024-09-11T01:31:34.242 INFO:tasks.rados.rados.0.smithi071.stdout:279: finishing write tid 5 to smithi07134777-39 2024-09-11T01:31:34.242 INFO:tasks.rados.rados.0.smithi071.stdout:279: oid 39 updating version 0 to 99 2024-09-11T01:31:34.242 INFO:tasks.rados.rados.0.smithi071.stdout:279: oid 39 version 99 is already newer than 98 2024-09-11T01:31:34.242 INFO:tasks.rados.rados.0.smithi071.stdout:279: oid 39 updating version 99 to 100 2024-09-11T01:31:34.242 INFO:tasks.rados.rados.0.smithi071.stdout:279: oid 39 updating version 100 to 101 2024-09-11T01:31:34.243 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 101 (ObjNum 118 snap 28 seq_num 118) dirty exists 2024-09-11T01:31:34.243 INFO:tasks.rados.rados.0.smithi071.stdout:279: left oid 39 (ObjNum 118 snap 28 seq_num 118) 2024-09-11T01:31:34.243 INFO:tasks.rados.rados.0.smithi071.stdout:281: finishing copy_from to smithi07134777-42 2024-09-11T01:31:34.243 INFO:tasks.rados.rados.0.smithi071.stdout:281: got expected ENOENT (src dne) 2024-09-11T01:31:34.243 INFO:tasks.rados.rados.0.smithi071.stdout:281: finishing copy_from racing read to smithi07134777-42 2024-09-11T01:31:34.243 INFO:tasks.rados.rados.0.smithi071.stdout:280: finishing write tid 1 to smithi07134777-11 2024-09-11T01:31:34.243 INFO:tasks.rados.rados.0.smithi071.stdout:277: done (3 left) 2024-09-11T01:31:34.243 INFO:tasks.rados.rados.0.smithi071.stdout:279: done (2 left) 2024-09-11T01:31:34.243 INFO:tasks.rados.rados.0.smithi071.stdout:281: done (1 left) 2024-09-11T01:31:34.243 INFO:tasks.rados.rados.0.smithi071.stdout:282: read oid 31 snap -1 2024-09-11T01:31:34.243 INFO:tasks.rados.rados.0.smithi071.stdout:282: expect (ObjNum 63 snap 8 seq_num 63) 2024-09-11T01:31:34.243 INFO:tasks.rados.rados.0.smithi071.stdout:283: rollback oid 9 current snap is 28 2024-09-11T01:31:34.243 INFO:tasks.rados.rados.0.smithi071.stdout:283: done (2 left) 2024-09-11T01:31:34.243 INFO:tasks.rados.rados.0.smithi071.stdout:284: read oid 38 snap -1 2024-09-11T01:31:34.243 INFO:tasks.rados.rados.0.smithi071.stdout:284: expect (ObjNum 90 snap 19 seq_num 90) 2024-09-11T01:31:34.244 INFO:tasks.rados.rados.0.smithi071.stdout:285: snap_create 2024-09-11T01:31:34.244 INFO:tasks.rados.rados.0.smithi071.stdout:280: finishing write tid 2 to smithi07134777-11 2024-09-11T01:31:34.244 INFO:tasks.rados.rados.0.smithi071.stdout:280: finishing write tid 3 to smithi07134777-11 2024-09-11T01:31:34.244 INFO:tasks.rados.rados.0.smithi071.stdout:280: finishing write tid 4 to smithi07134777-11 2024-09-11T01:31:34.244 INFO:tasks.rados.rados.0.smithi071.stdout:280: oid 11 updating version 0 to 98 2024-09-11T01:31:34.244 INFO:tasks.rados.rados.0.smithi071.stdout:280: oid 11 version 98 is already newer than 97 2024-09-11T01:31:34.244 INFO:tasks.rados.rados.0.smithi071.stdout:280: oid 11 version 98 is already newer than 96 2024-09-11T01:31:34.244 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 98 (ObjNum 119 snap 28 seq_num 119) dirty exists 2024-09-11T01:31:34.244 INFO:tasks.rados.rados.0.smithi071.stdout:280: left oid 11 (ObjNum 119 snap 28 seq_num 119) 2024-09-11T01:31:34.244 INFO:tasks.rados.rados.0.smithi071.stdout:282: expect (ObjNum 63 snap 8 seq_num 63) 2024-09-11T01:31:34.314 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2822563 2024-09-11T01:31:34.358 INFO:tasks.rados.rados.0.smithi071.stdout:284: expect (ObjNum 90 snap 19 seq_num 90) 2024-09-11T01:31:34.440 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:31:35.142 INFO:tasks.rados.rados.0.smithi071.stdout:280: done (3 left) 2024-09-11T01:31:35.142 INFO:tasks.rados.rados.0.smithi071.stdout:282: done (2 left) 2024-09-11T01:31:35.142 INFO:tasks.rados.rados.0.smithi071.stdout:284: done (1 left) 2024-09-11T01:31:35.142 INFO:tasks.rados.rados.0.smithi071.stdout:285: done (0 left) 2024-09-11T01:31:35.142 INFO:tasks.rados.rados.0.smithi071.stdout:286: copy_from oid 10 from oid 28 current snap is 29 2024-09-11T01:31:35.142 INFO:tasks.rados.rados.0.smithi071.stdout:287: snap_remove snap 28 2024-09-11T01:31:36.145 INFO:tasks.rados.rados.0.smithi071.stdout:287: done (1 left) 2024-09-11T01:31:36.145 INFO:tasks.rados.rados.0.smithi071.stdout:288: rollback oid 45 current snap is 29 2024-09-11T01:31:36.145 INFO:tasks.rados.rados.0.smithi071.stdout:288: done (1 left) 2024-09-11T01:31:36.145 INFO:tasks.rados.rados.0.smithi071.stdout:289: write (excl) oid 15 current snap is 29 2024-09-11T01:31:36.145 INFO:tasks.rados.rados.0.smithi071.stdout:289: seq_num 120 ranges {416840=775175,1672545=344295} 2024-09-11T01:31:36.151 INFO:tasks.rados.rados.0.smithi071.stdout:289: writing smithi07134777-15 from 416840 to 1192015 tid 1 2024-09-11T01:31:36.157 INFO:tasks.rados.rados.0.smithi071.stdout:289: writing smithi07134777-15 from 1672545 to 2016840 tid 2 2024-09-11T01:31:36.157 INFO:tasks.rados.rados.0.smithi071.stdout:290: read oid 49 snap -1 2024-09-11T01:31:36.157 INFO:tasks.rados.rados.0.smithi071.stdout:290: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-11T01:31:36.157 INFO:tasks.rados.rados.0.smithi071.stdout:291: copy_from oid 37 from oid 39 current snap is 29 2024-09-11T01:31:36.157 INFO:tasks.rados.rados.0.smithi071.stdout:286: finishing copy_from to smithi07134777-10 2024-09-11T01:31:36.157 INFO:tasks.rados.rados.0.smithi071.stdout:286: got expected ENOENT (src dne) 2024-09-11T01:31:36.158 INFO:tasks.rados.rados.0.smithi071.stdout:286: finishing copy_from racing read to smithi07134777-10 2024-09-11T01:31:36.158 INFO:tasks.rados.rados.0.smithi071.stdout:286: done (3 left) 2024-09-11T01:31:36.158 INFO:tasks.rados.rados.0.smithi071.stdout:292: read oid 46 snap -1 2024-09-11T01:31:36.158 INFO:tasks.rados.rados.0.smithi071.stdout:292: expect deleted 2024-09-11T01:31:36.158 INFO:tasks.rados.rados.0.smithi071.stdout:293: copy_from oid 43 from oid 7 current snap is 29 2024-09-11T01:31:36.158 INFO:tasks.rados.rados.0.smithi071.stdout:294: read oid 9 snap -1 2024-09-11T01:31:36.158 INFO:tasks.rados.rados.0.smithi071.stdout:294: expect (ObjNum 84 snap 18 seq_num 84) 2024-09-11T01:31:36.158 INFO:tasks.rados.rados.0.smithi071.stdout:295: write oid 28 current snap is 29 2024-09-11T01:31:36.158 INFO:tasks.rados.rados.0.smithi071.stdout:295: seq_num 121 ranges {433647=450424,1331027=582033} 2024-09-11T01:31:36.162 INFO:tasks.rados.rados.0.smithi071.stdout:295: writing smithi07134777-28 from 433647 to 884071 tid 1 2024-09-11T01:31:36.169 INFO:tasks.rados.rados.0.smithi071.stdout:295: writing smithi07134777-28 from 1331027 to 1913060 tid 2 2024-09-11T01:31:36.169 INFO:tasks.rados.rados.0.smithi071.stdout:296: rollback oid 27 current snap is 29 2024-09-11T01:31:36.169 INFO:tasks.rados.rados.0.smithi071.stdout:296: done (7 left) 2024-09-11T01:31:36.169 INFO:tasks.rados.rados.0.smithi071.stdout:297: delete oid 30 current snap is 29 2024-09-11T01:31:36.169 INFO:tasks.rados.rados.0.smithi071.stdout:289: finishing write tid 1 to smithi07134777-15 2024-09-11T01:31:36.169 INFO:tasks.rados.rados.0.smithi071.stdout:289: finishing write tid 2 to smithi07134777-15 2024-09-11T01:31:36.169 INFO:tasks.rados.rados.0.smithi071.stdout:289: finishing write tid 3 to smithi07134777-15 2024-09-11T01:31:36.169 INFO:tasks.rados.rados.0.smithi071.stdout:289: finishing write tid 4 to smithi07134777-15 2024-09-11T01:31:36.169 INFO:tasks.rados.rados.0.smithi071.stdout:289: oid 15 updating version 0 to 147 2024-09-11T01:31:36.169 INFO:tasks.rados.rados.0.smithi071.stdout:289: oid 15 updating version 147 to 149 2024-09-11T01:31:36.169 INFO:tasks.rados.rados.0.smithi071.stdout:289: oid 15 version 149 is already newer than 148 2024-09-11T01:31:36.170 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 149 (ObjNum 120 snap 29 seq_num 120) dirty exists 2024-09-11T01:31:36.170 INFO:tasks.rados.rados.0.smithi071.stdout:289: left oid 15 (ObjNum 120 snap 29 seq_num 120) 2024-09-11T01:31:36.170 INFO:tasks.rados.rados.0.smithi071.stdout:290: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-11T01:31:36.251 INFO:tasks.rados.rados.0.smithi071.stdout:294: expect (ObjNum 84 snap 18 seq_num 84) 2024-09-11T01:31:36.342 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3132329 2024-09-11T01:31:36.344 INFO:tasks.rados.rados.0.smithi071.stdout:295: finishing write tid 1 to smithi07134777-28 2024-09-11T01:31:36.345 INFO:tasks.rados.rados.0.smithi071.stdout:295: finishing write tid 2 to smithi07134777-28 2024-09-11T01:31:36.345 INFO:tasks.rados.rados.0.smithi071.stdout:295: finishing write tid 3 to smithi07134777-28 2024-09-11T01:31:36.345 INFO:tasks.rados.rados.0.smithi071.stdout:295: finishing write tid 4 to smithi07134777-28 2024-09-11T01:31:36.345 INFO:tasks.rados.rados.0.smithi071.stdout:295: oid 28 updating version 0 to 80 2024-09-11T01:31:36.345 INFO:tasks.rados.rados.0.smithi071.stdout:295: oid 28 version 80 is already newer than 78 2024-09-11T01:31:36.345 INFO:tasks.rados.rados.0.smithi071.stdout:295: oid 28 version 80 is already newer than 79 2024-09-11T01:31:36.345 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 80 (ObjNum 121 snap 29 seq_num 121) dirty exists 2024-09-11T01:31:36.345 INFO:tasks.rados.rados.0.smithi071.stdout:295: left oid 28 (ObjNum 121 snap 29 seq_num 121) 2024-09-11T01:31:36.345 INFO:tasks.rados.rados.0.smithi071.stdout:293: finishing copy_from racing read to smithi07134777-43 2024-09-11T01:31:36.345 INFO:tasks.rados.rados.0.smithi071.stdout:291: finishing copy_from racing read to smithi07134777-37 2024-09-11T01:31:36.345 INFO:tasks.rados.rados.0.smithi071.stdout:289: done (7 left) 2024-09-11T01:31:36.345 INFO:tasks.rados.rados.0.smithi071.stdout:290: done (6 left) 2024-09-11T01:31:36.345 INFO:tasks.rados.rados.0.smithi071.stdout:292: done (5 left) 2024-09-11T01:31:36.345 INFO:tasks.rados.rados.0.smithi071.stdout:294: done (4 left) 2024-09-11T01:31:36.346 INFO:tasks.rados.rados.0.smithi071.stdout:295: done (3 left) 2024-09-11T01:31:36.346 INFO:tasks.rados.rados.0.smithi071.stdout:297: done (2 left) 2024-09-11T01:31:36.346 INFO:tasks.rados.rados.0.smithi071.stdout:298: write oid 31 current snap is 29 2024-09-11T01:31:36.346 INFO:tasks.rados.rados.0.smithi071.stdout:293: finishing copy_from to smithi07134777-43 2024-09-11T01:31:36.346 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 146 (ObjNum 115 snap 27 seq_num 115) dirty exists 2024-09-11T01:31:36.346 INFO:tasks.rados.rados.0.smithi071.stdout:291: finishing copy_from to smithi07134777-37 2024-09-11T01:31:36.346 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 101 (ObjNum 118 snap 28 seq_num 118) dirty exists 2024-09-11T01:31:36.346 INFO:tasks.rados.rados.0.smithi071.stdout:298: seq_num 123 ranges {467261=717275,1847991=219270} 2024-09-11T01:31:36.350 INFO:tasks.rados.rados.0.smithi071.stdout:298: writing smithi07134777-31 from 467261 to 1184536 tid 1 2024-09-11T01:31:36.356 INFO:tasks.rados.rados.0.smithi071.stdout:298: writing smithi07134777-31 from 1847991 to 2067261 tid 2 2024-09-11T01:31:36.356 INFO:tasks.rados.rados.0.smithi071.stdout:291: done (2 left) 2024-09-11T01:31:36.356 INFO:tasks.rados.rados.0.smithi071.stdout:293: done (1 left) 2024-09-11T01:31:36.356 INFO:tasks.rados.rados.0.smithi071.stdout:299: read oid 23 snap -1 2024-09-11T01:31:36.356 INFO:tasks.rados.rados.0.smithi071.stdout:299: expect (ObjNum 112 snap 27 seq_num 112) 2024-09-11T01:31:36.356 INFO:tasks.rados.rados.0.smithi071.stdout:300: write (excl) oid 46 current snap is 29 2024-09-11T01:31:36.356 INFO:tasks.rados.rados.0.smithi071.stdout:300: seq_num 124 ranges {484068=792524,1906473=177595} 2024-09-11T01:31:36.363 INFO:tasks.rados.rados.0.smithi071.stdout:300: writing smithi07134777-46 from 484068 to 1276592 tid 1 2024-09-11T01:31:36.368 INFO:tasks.rados.rados.0.smithi071.stdout:300: writing smithi07134777-46 from 1906473 to 2084068 tid 2 2024-09-11T01:31:36.368 INFO:tasks.rados.rados.0.smithi071.stdout:301: write oid 3 current snap is 29 2024-09-11T01:31:36.368 INFO:tasks.rados.rados.0.smithi071.stdout:298: finishing write tid 1 to smithi07134777-31 2024-09-11T01:31:36.368 INFO:tasks.rados.rados.0.smithi071.stdout:301: seq_num 125 ranges {500875=467773,1564955=535920} 2024-09-11T01:31:36.373 INFO:tasks.rados.rados.0.smithi071.stdout:301: writing smithi07134777-3 from 500875 to 968648 tid 1 2024-09-11T01:31:36.380 INFO:tasks.rados.rados.0.smithi071.stdout:301: writing smithi07134777-3 from 1564955 to 2100875 tid 2 2024-09-11T01:31:36.380 INFO:tasks.rados.rados.0.smithi071.stdout:302: write oid 36 current snap is 29 2024-09-11T01:31:36.380 INFO:tasks.rados.rados.0.smithi071.stdout:298: finishing write tid 2 to smithi07134777-31 2024-09-11T01:31:36.380 INFO:tasks.rados.rados.0.smithi071.stdout:298: finishing write tid 3 to smithi07134777-31 2024-09-11T01:31:36.380 INFO:tasks.rados.rados.0.smithi071.stdout:298: finishing write tid 4 to smithi07134777-31 2024-09-11T01:31:36.380 INFO:tasks.rados.rados.0.smithi071.stdout:298: oid 31 updating version 0 to 90 2024-09-11T01:31:36.380 INFO:tasks.rados.rados.0.smithi071.stdout:298: oid 31 updating version 90 to 92 2024-09-11T01:31:36.380 INFO:tasks.rados.rados.0.smithi071.stdout:298: oid 31 version 92 is already newer than 91 2024-09-11T01:31:36.380 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 92 (ObjNum 123 snap 29 seq_num 123) dirty exists 2024-09-11T01:31:36.380 INFO:tasks.rados.rados.0.smithi071.stdout:298: left oid 31 (ObjNum 123 snap 29 seq_num 123) 2024-09-11T01:31:36.380 INFO:tasks.rados.rados.0.smithi071.stdout:299: expect (ObjNum 112 snap 27 seq_num 112) 2024-09-11T01:31:36.381 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:31:36.467 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1674368 2024-09-11T01:31:36.509 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:31:36.577 INFO:tasks.rados.rados.0.smithi071.stdout:300: finishing write tid 1 to smithi07134777-46 2024-09-11T01:31:36.577 INFO:tasks.rados.rados.0.smithi071.stdout:300: finishing write tid 2 to smithi07134777-46 2024-09-11T01:31:36.577 INFO:tasks.rados.rados.0.smithi071.stdout:302: seq_num 126 ranges {517682=543022,1623437=494245} 2024-09-11T01:31:36.583 INFO:tasks.rados.rados.0.smithi071.stdout:302: writing smithi07134777-36 from 517682 to 1060704 tid 1 2024-09-11T01:31:36.589 INFO:tasks.rados.rados.0.smithi071.stdout:302: writing smithi07134777-36 from 1623437 to 2117682 tid 2 2024-09-11T01:31:36.589 INFO:tasks.rados.rados.0.smithi071.stdout:298: done (4 left) 2024-09-11T01:31:36.589 INFO:tasks.rados.rados.0.smithi071.stdout:299: done (3 left) 2024-09-11T01:31:36.589 INFO:tasks.rados.rados.0.smithi071.stdout:303: write (excl) oid 37 current snap is 29 2024-09-11T01:31:36.589 INFO:tasks.rados.rados.0.smithi071.stdout:301: finishing write tid 1 to smithi07134777-3 2024-09-11T01:31:36.589 INFO:tasks.rados.rados.0.smithi071.stdout:303: seq_num 127 ranges {534489=618271,1565566=568923} 2024-09-11T01:31:36.594 INFO:tasks.rados.rados.0.smithi071.stdout:303: writing smithi07134777-37 from 534489 to 1152760 tid 1 2024-09-11T01:31:36.600 INFO:tasks.rados.rados.0.smithi071.stdout:303: writing smithi07134777-37 from 1565566 to 2134489 tid 2 2024-09-11T01:31:36.601 INFO:tasks.rados.rados.0.smithi071.stdout:304: read oid 7 snap -1 2024-09-11T01:31:36.601 INFO:tasks.rados.rados.0.smithi071.stdout:304: expect (ObjNum 115 snap 27 seq_num 115) 2024-09-11T01:31:36.601 INFO:tasks.rados.rados.0.smithi071.stdout:305: write (excl) oid 33 current snap is 29 2024-09-11T01:31:36.601 INFO:tasks.rados.rados.0.smithi071.stdout:300: finishing write tid 3 to smithi07134777-46 2024-09-11T01:31:36.601 INFO:tasks.rados.rados.0.smithi071.stdout:300: finishing write tid 4 to smithi07134777-46 2024-09-11T01:31:36.601 INFO:tasks.rados.rados.0.smithi071.stdout:300: oid 46 updating version 0 to 17 2024-09-11T01:31:36.601 INFO:tasks.rados.rados.0.smithi071.stdout:300: oid 46 version 17 is already newer than 16 2024-09-11T01:31:36.601 INFO:tasks.rados.rados.0.smithi071.stdout:300: oid 46 updating version 17 to 18 2024-09-11T01:31:36.601 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 18 (ObjNum 124 snap 29 seq_num 124) dirty exists 2024-09-11T01:31:36.601 INFO:tasks.rados.rados.0.smithi071.stdout:300: left oid 46 (ObjNum 124 snap 29 seq_num 124) 2024-09-11T01:31:36.601 INFO:tasks.rados.rados.0.smithi071.stdout:301: finishing write tid 2 to smithi07134777-3 2024-09-11T01:31:36.601 INFO:tasks.rados.rados.0.smithi071.stdout:301: finishing write tid 3 to smithi07134777-3 2024-09-11T01:31:36.602 INFO:tasks.rados.rados.0.smithi071.stdout:305: seq_num 128 ranges {551296=693520,2024048=127248} 2024-09-11T01:31:36.607 INFO:tasks.rados.rados.0.smithi071.stdout:305: writing smithi07134777-33 from 551296 to 1244816 tid 1 2024-09-11T01:31:36.613 INFO:tasks.rados.rados.0.smithi071.stdout:305: writing smithi07134777-33 from 2024048 to 2151296 tid 2 2024-09-11T01:31:36.613 INFO:tasks.rados.rados.0.smithi071.stdout:300: done (5 left) 2024-09-11T01:31:36.613 INFO:tasks.rados.rados.0.smithi071.stdout:306: write (excl) oid 42 current snap is 29 2024-09-11T01:31:36.613 INFO:tasks.rados.rados.0.smithi071.stdout:301: finishing write tid 4 to smithi07134777-3 2024-09-11T01:31:36.613 INFO:tasks.rados.rados.0.smithi071.stdout:301: oid 3 updating version 0 to 24 2024-09-11T01:31:36.613 INFO:tasks.rados.rados.0.smithi071.stdout:301: oid 3 version 24 is already newer than 22 2024-09-11T01:31:36.613 INFO:tasks.rados.rados.0.smithi071.stdout:301: oid 3 version 24 is already newer than 23 2024-09-11T01:31:36.613 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 24 (ObjNum 125 snap 29 seq_num 125) dirty exists 2024-09-11T01:31:36.613 INFO:tasks.rados.rados.0.smithi071.stdout:301: left oid 3 (ObjNum 125 snap 29 seq_num 125) 2024-09-11T01:31:36.613 INFO:tasks.rados.rados.0.smithi071.stdout:302: finishing write tid 1 to smithi07134777-36 2024-09-11T01:31:36.613 INFO:tasks.rados.rados.0.smithi071.stdout:302: finishing write tid 2 to smithi07134777-36 2024-09-11T01:31:36.614 INFO:tasks.rados.rados.0.smithi071.stdout:306: seq_num 129 ranges {568103=768769,2082530=85573} 2024-09-11T01:31:36.619 INFO:tasks.rados.rados.0.smithi071.stdout:306: writing smithi07134777-42 from 568103 to 1336872 tid 1 2024-09-11T01:31:36.624 INFO:tasks.rados.rados.0.smithi071.stdout:306: writing smithi07134777-42 from 2082530 to 2168103 tid 2 2024-09-11T01:31:36.625 INFO:tasks.rados.rados.0.smithi071.stdout:301: done (5 left) 2024-09-11T01:31:36.625 INFO:tasks.rados.rados.0.smithi071.stdout:307: snap_create 2024-09-11T01:31:36.625 INFO:tasks.rados.rados.0.smithi071.stdout:302: finishing write tid 3 to smithi07134777-36 2024-09-11T01:31:36.625 INFO:tasks.rados.rados.0.smithi071.stdout:302: finishing write tid 4 to smithi07134777-36 2024-09-11T01:31:36.625 INFO:tasks.rados.rados.0.smithi071.stdout:302: oid 36 updating version 0 to 51 2024-09-11T01:31:36.625 INFO:tasks.rados.rados.0.smithi071.stdout:302: oid 36 version 51 is already newer than 50 2024-09-11T01:31:36.625 INFO:tasks.rados.rados.0.smithi071.stdout:302: oid 36 version 51 is already newer than 49 2024-09-11T01:31:36.625 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 51 (ObjNum 126 snap 29 seq_num 126) dirty exists 2024-09-11T01:31:36.625 INFO:tasks.rados.rados.0.smithi071.stdout:302: left oid 36 (ObjNum 126 snap 29 seq_num 126) 2024-09-11T01:31:36.625 INFO:tasks.rados.rados.0.smithi071.stdout:303: finishing write tid 1 to smithi07134777-37 2024-09-11T01:31:36.625 INFO:tasks.rados.rados.0.smithi071.stdout:304: expect (ObjNum 115 snap 27 seq_num 115) 2024-09-11T01:31:36.743 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:31:36.803 INFO:tasks.rados.rados.0.smithi071.stdout:303: finishing write tid 2 to smithi07134777-37 2024-09-11T01:31:36.803 INFO:tasks.rados.rados.0.smithi071.stdout:303: finishing write tid 3 to smithi07134777-37 2024-09-11T01:31:36.803 INFO:tasks.rados.rados.0.smithi071.stdout:303: finishing write tid 4 to smithi07134777-37 2024-09-11T01:31:36.803 INFO:tasks.rados.rados.0.smithi071.stdout:303: oid 37 updating version 0 to 104 2024-09-11T01:31:36.804 INFO:tasks.rados.rados.0.smithi071.stdout:303: oid 37 updating version 104 to 105 2024-09-11T01:31:36.804 INFO:tasks.rados.rados.0.smithi071.stdout:303: oid 37 version 105 is already newer than 103 2024-09-11T01:31:36.804 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 105 (ObjNum 127 snap 29 seq_num 127) dirty exists 2024-09-11T01:31:36.804 INFO:tasks.rados.rados.0.smithi071.stdout:303: left oid 37 (ObjNum 127 snap 29 seq_num 127) 2024-09-11T01:31:36.804 INFO:tasks.rados.rados.0.smithi071.stdout:305: finishing write tid 1 to smithi07134777-33 2024-09-11T01:31:36.804 INFO:tasks.rados.rados.0.smithi071.stdout:305: finishing write tid 2 to smithi07134777-33 2024-09-11T01:31:36.804 INFO:tasks.rados.rados.0.smithi071.stdout:305: finishing write tid 3 to smithi07134777-33 2024-09-11T01:31:36.804 INFO:tasks.rados.rados.0.smithi071.stdout:305: finishing write tid 4 to smithi07134777-33 2024-09-11T01:31:36.804 INFO:tasks.rados.rados.0.smithi071.stdout:305: oid 33 updating version 0 to 147 2024-09-11T01:31:36.804 INFO:tasks.rados.rados.0.smithi071.stdout:305: oid 33 updating version 147 to 148 2024-09-11T01:31:36.804 INFO:tasks.rados.rados.0.smithi071.stdout:305: oid 33 updating version 148 to 149 2024-09-11T01:31:36.804 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 149 (ObjNum 128 snap 29 seq_num 128) dirty exists 2024-09-11T01:31:36.804 INFO:tasks.rados.rados.0.smithi071.stdout:305: left oid 33 (ObjNum 128 snap 29 seq_num 128) 2024-09-11T01:31:36.804 INFO:tasks.rados.rados.0.smithi071.stdout:306: finishing write tid 1 to smithi07134777-42 2024-09-11T01:31:36.805 INFO:tasks.rados.rados.0.smithi071.stdout:306: finishing write tid 2 to smithi07134777-42 2024-09-11T01:31:36.805 INFO:tasks.rados.rados.0.smithi071.stdout:306: finishing write tid 3 to smithi07134777-42 2024-09-11T01:31:36.805 INFO:tasks.rados.rados.0.smithi071.stdout:306: finishing write tid 4 to smithi07134777-42 2024-09-11T01:31:36.805 INFO:tasks.rados.rados.0.smithi071.stdout:306: oid 42 updating version 0 to 19 2024-09-11T01:31:36.805 INFO:tasks.rados.rados.0.smithi071.stdout:306: oid 42 version 19 is already newer than 17 2024-09-11T01:31:36.805 INFO:tasks.rados.rados.0.smithi071.stdout:306: oid 42 version 19 is already newer than 18 2024-09-11T01:31:36.805 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 19 (ObjNum 129 snap 29 seq_num 129) dirty exists 2024-09-11T01:31:36.805 INFO:tasks.rados.rados.0.smithi071.stdout:306: left oid 42 (ObjNum 129 snap 29 seq_num 129) 2024-09-11T01:31:37.073 INFO:tasks.rados.rados.0.smithi071.stdout:302: done (5 left) 2024-09-11T01:31:37.073 INFO:tasks.rados.rados.0.smithi071.stdout:303: done (4 left) 2024-09-11T01:31:37.073 INFO:tasks.rados.rados.0.smithi071.stdout:304: done (3 left) 2024-09-11T01:31:37.073 INFO:tasks.rados.rados.0.smithi071.stdout:305: done (2 left) 2024-09-11T01:31:37.073 INFO:tasks.rados.rados.0.smithi071.stdout:306: done (1 left) 2024-09-11T01:31:37.073 INFO:tasks.rados.rados.0.smithi071.stdout:307: done (0 left) 2024-09-11T01:31:37.073 INFO:tasks.rados.rados.0.smithi071.stdout:308: write (excl) oid 34 current snap is 30 2024-09-11T01:31:37.073 INFO:tasks.rados.rados.0.smithi071.stdout:308: seq_num 130 ranges {584910=560371,1857365=327545} 2024-09-11T01:31:37.076 INFO:tasks.rados.rados.0.smithi071.stdout:308: writing smithi07134777-34 from 584910 to 1145281 tid 1 2024-09-11T01:31:37.082 INFO:tasks.rados.rados.0.smithi071.stdout:308: writing smithi07134777-34 from 1857365 to 2184910 tid 2 2024-09-11T01:31:37.082 INFO:tasks.rados.rados.0.smithi071.stdout:309: read oid 50 snap 29 2024-09-11T01:31:37.082 INFO:tasks.rados.rados.0.smithi071.stdout:309: expect (ObjNum 95 snap 20 seq_num 95) 2024-09-11T01:31:37.082 INFO:tasks.rados.rados.0.smithi071.stdout:310: copy_from oid 41 from oid 37 current snap is 30 2024-09-11T01:31:37.082 INFO:tasks.rados.rados.0.smithi071.stdout:311: read oid 45 snap -1 2024-09-11T01:31:37.082 INFO:tasks.rados.rados.0.smithi071.stdout:311: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:31:37.083 INFO:tasks.rados.rados.0.smithi071.stdout:312: read oid 10 snap 29 2024-09-11T01:31:37.083 INFO:tasks.rados.rados.0.smithi071.stdout:312: expect (ObjNum 88 snap 19 seq_num 88) 2024-09-11T01:31:37.083 INFO:tasks.rados.rados.0.smithi071.stdout:313: write oid 16 current snap is 30 2024-09-11T01:31:37.083 INFO:tasks.rados.rados.0.smithi071.stdout:313: seq_num 131 ranges {601717=635620,1799494=402223} 2024-09-11T01:31:37.089 INFO:tasks.rados.rados.0.smithi071.stdout:313: writing smithi07134777-16 from 601717 to 1237337 tid 1 2024-09-11T01:31:37.095 INFO:tasks.rados.rados.0.smithi071.stdout:313: writing smithi07134777-16 from 1799494 to 2201717 tid 2 2024-09-11T01:31:37.095 INFO:tasks.rados.rados.0.smithi071.stdout:314: write oid 15 current snap is 30 2024-09-11T01:31:37.095 INFO:tasks.rados.rados.0.smithi071.stdout:314: seq_num 132 ranges {618524=710869,1857976=360548} 2024-09-11T01:31:37.103 INFO:tasks.rados.rados.0.smithi071.stdout:314: writing smithi07134777-15 from 618524 to 1329393 tid 1 2024-09-11T01:31:37.108 INFO:tasks.rados.rados.0.smithi071.stdout:314: writing smithi07134777-15 from 1857976 to 2218524 tid 2 2024-09-11T01:31:37.108 INFO:tasks.rados.rados.0.smithi071.stdout:315: write oid 4 current snap is 30 2024-09-11T01:31:37.108 INFO:tasks.rados.rados.0.smithi071.stdout:308: finishing write tid 1 to smithi07134777-34 2024-09-11T01:31:37.108 INFO:tasks.rados.rados.0.smithi071.stdout:312: expect (ObjNum 88 snap 19 seq_num 88) 2024-09-11T01:31:37.244 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2577925 2024-09-11T01:31:37.262 INFO:tasks.rados.rados.0.smithi071.stdout:309: expect (ObjNum 95 snap 20 seq_num 95) 2024-09-11T01:31:37.262 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 69743 2024-09-11T01:31:37.299 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 771390 2024-09-11T01:31:37.369 INFO:tasks.rados.rados.0.smithi071.stdout:311: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:31:37.397 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:31:37.453 INFO:tasks.rados.rados.0.smithi071.stdout:308: finishing write tid 2 to smithi07134777-34 2024-09-11T01:31:37.453 INFO:tasks.rados.rados.0.smithi071.stdout:308: finishing write tid 3 to smithi07134777-34 2024-09-11T01:31:37.453 INFO:tasks.rados.rados.0.smithi071.stdout:308: finishing write tid 4 to smithi07134777-34 2024-09-11T01:31:37.453 INFO:tasks.rados.rados.0.smithi071.stdout:308: oid 34 updating version 0 to 152 2024-09-11T01:31:37.453 INFO:tasks.rados.rados.0.smithi071.stdout:308: oid 34 version 152 is already newer than 150 2024-09-11T01:31:37.453 INFO:tasks.rados.rados.0.smithi071.stdout:308: oid 34 version 152 is already newer than 151 2024-09-11T01:31:37.453 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 152 (ObjNum 130 snap 30 seq_num 130) dirty exists 2024-09-11T01:31:37.454 INFO:tasks.rados.rados.0.smithi071.stdout:308: left oid 34 (ObjNum 130 snap 30 seq_num 130) 2024-09-11T01:31:37.454 INFO:tasks.rados.rados.0.smithi071.stdout:313: finishing write tid 1 to smithi07134777-16 2024-09-11T01:31:37.454 INFO:tasks.rados.rados.0.smithi071.stdout:310: finishing copy_from racing read to smithi07134777-41 2024-09-11T01:31:37.454 INFO:tasks.rados.rados.0.smithi071.stdout:313: finishing write tid 2 to smithi07134777-16 2024-09-11T01:31:37.454 INFO:tasks.rados.rados.0.smithi071.stdout:310: finishing copy_from to smithi07134777-41 2024-09-11T01:31:37.454 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 105 (ObjNum 127 snap 29 seq_num 127) dirty exists 2024-09-11T01:31:37.454 INFO:tasks.rados.rados.0.smithi071.stdout:313: finishing write tid 3 to smithi07134777-16 2024-09-11T01:31:37.454 INFO:tasks.rados.rados.0.smithi071.stdout:313: finishing write tid 4 to smithi07134777-16 2024-09-11T01:31:37.454 INFO:tasks.rados.rados.0.smithi071.stdout:313: oid 16 updating version 0 to 84 2024-09-11T01:31:37.454 INFO:tasks.rados.rados.0.smithi071.stdout:313: oid 16 version 84 is already newer than 82 2024-09-11T01:31:37.454 INFO:tasks.rados.rados.0.smithi071.stdout:313: oid 16 version 84 is already newer than 83 2024-09-11T01:31:37.454 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 84 (ObjNum 131 snap 30 seq_num 131) dirty exists 2024-09-11T01:31:37.454 INFO:tasks.rados.rados.0.smithi071.stdout:313: left oid 16 (ObjNum 131 snap 30 seq_num 131) 2024-09-11T01:31:37.454 INFO:tasks.rados.rados.0.smithi071.stdout:314: finishing write tid 1 to smithi07134777-15 2024-09-11T01:31:37.455 INFO:tasks.rados.rados.0.smithi071.stdout:314: finishing write tid 2 to smithi07134777-15 2024-09-11T01:31:37.455 INFO:tasks.rados.rados.0.smithi071.stdout:315: seq_num 133 ranges {635331=786118,1916458=318873} 2024-09-11T01:31:37.460 INFO:tasks.rados.rados.0.smithi071.stdout:315: writing smithi07134777-4 from 635331 to 1421449 tid 1 2024-09-11T01:31:37.465 INFO:tasks.rados.rados.0.smithi071.stdout:315: writing smithi07134777-4 from 1916458 to 2235331 tid 2 2024-09-11T01:31:37.466 INFO:tasks.rados.rados.0.smithi071.stdout:308: done (7 left) 2024-09-11T01:31:37.466 INFO:tasks.rados.rados.0.smithi071.stdout:309: done (6 left) 2024-09-11T01:31:37.466 INFO:tasks.rados.rados.0.smithi071.stdout:310: done (5 left) 2024-09-11T01:31:37.466 INFO:tasks.rados.rados.0.smithi071.stdout:311: done (4 left) 2024-09-11T01:31:37.466 INFO:tasks.rados.rados.0.smithi071.stdout:312: done (3 left) 2024-09-11T01:31:37.466 INFO:tasks.rados.rados.0.smithi071.stdout:313: done (2 left) 2024-09-11T01:31:37.466 INFO:tasks.rados.rados.0.smithi071.stdout:316: delete oid 9 current snap is 30 2024-09-11T01:31:37.466 INFO:tasks.rados.rados.0.smithi071.stdout:314: finishing write tid 3 to smithi07134777-15 2024-09-11T01:31:37.466 INFO:tasks.rados.rados.0.smithi071.stdout:314: finishing write tid 4 to smithi07134777-15 2024-09-11T01:31:37.466 INFO:tasks.rados.rados.0.smithi071.stdout:314: oid 15 updating version 0 to 153 2024-09-11T01:31:37.466 INFO:tasks.rados.rados.0.smithi071.stdout:314: oid 15 version 153 is already newer than 152 2024-09-11T01:31:37.466 INFO:tasks.rados.rados.0.smithi071.stdout:314: oid 15 version 153 is already newer than 151 2024-09-11T01:31:37.466 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 153 (ObjNum 132 snap 30 seq_num 132) dirty exists 2024-09-11T01:31:37.467 INFO:tasks.rados.rados.0.smithi071.stdout:314: left oid 15 (ObjNum 132 snap 30 seq_num 132) 2024-09-11T01:31:37.471 INFO:tasks.rados.rados.0.smithi071.stdout:315: finishing write tid 1 to smithi07134777-4 2024-09-11T01:31:37.475 INFO:tasks.rados.rados.0.smithi071.stdout:315: finishing write tid 2 to smithi07134777-4 2024-09-11T01:31:37.475 INFO:tasks.rados.rados.0.smithi071.stdout:315: finishing write tid 3 to smithi07134777-4 2024-09-11T01:31:37.475 INFO:tasks.rados.rados.0.smithi071.stdout:315: finishing write tid 4 to smithi07134777-4 2024-09-11T01:31:37.475 INFO:tasks.rados.rados.0.smithi071.stdout:315: oid 4 updating version 0 to 103 2024-09-11T01:31:37.475 INFO:tasks.rados.rados.0.smithi071.stdout:315: oid 4 version 103 is already newer than 101 2024-09-11T01:31:37.475 INFO:tasks.rados.rados.0.smithi071.stdout:315: oid 4 version 103 is already newer than 102 2024-09-11T01:31:37.476 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 103 (ObjNum 133 snap 30 seq_num 133) dirty exists 2024-09-11T01:31:37.476 INFO:tasks.rados.rados.0.smithi071.stdout:315: left oid 4 (ObjNum 133 snap 30 seq_num 133) 2024-09-11T01:31:37.477 INFO:tasks.rados.rados.0.smithi071.stdout:314: done (2 left) 2024-09-11T01:31:37.477 INFO:tasks.rados.rados.0.smithi071.stdout:315: done (1 left) 2024-09-11T01:31:37.477 INFO:tasks.rados.rados.0.smithi071.stdout:316: done (0 left) 2024-09-11T01:31:37.477 INFO:tasks.rados.rados.0.smithi071.stdout:317: write (excl) oid 46 current snap is 30 2024-09-11T01:31:37.477 INFO:tasks.rados.rados.0.smithi071.stdout:317: seq_num 135 ranges {668945=536616,1633422=491363} 2024-09-11T01:31:37.486 INFO:tasks.rados.rados.0.smithi071.stdout:317: writing smithi07134777-46 from 668945 to 1205561 tid 1 2024-09-11T01:31:37.490 INFO:tasks.rados.rados.0.smithi071.stdout:317: writing smithi07134777-46 from 1633422 to 2124785 tid 2 2024-09-11T01:31:37.490 INFO:tasks.rados.rados.0.smithi071.stdout:318: copy_from oid 44 from oid 19 current snap is 30 2024-09-11T01:31:37.490 INFO:tasks.rados.rados.0.smithi071.stdout:317: finishing write tid 1 to smithi07134777-46 2024-09-11T01:31:37.490 INFO:tasks.rados.rados.0.smithi071.stdout:319: read oid 5 snap -1 2024-09-11T01:31:37.490 INFO:tasks.rados.rados.0.smithi071.stdout:319: expect (ObjNum 99 snap 23 seq_num 99) 2024-09-11T01:31:37.490 INFO:tasks.rados.rados.0.smithi071.stdout:320: read oid 47 snap -1 2024-09-11T01:31:37.490 INFO:tasks.rados.rados.0.smithi071.stdout:320: expect (ObjNum 109 snap 26 seq_num 109) 2024-09-11T01:31:37.490 INFO:tasks.rados.rados.0.smithi071.stdout:321: rollback oid 37 current snap is 30 2024-09-11T01:31:37.490 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 37 to 29 2024-09-11T01:31:37.490 INFO:tasks.rados.rados.0.smithi071.stdout:322: copy_from oid 21 from oid 15 current snap is 30 2024-09-11T01:31:37.490 INFO:tasks.rados.rados.0.smithi071.stdout:323: copy_from oid 24 from oid 6 current snap is 30 2024-09-11T01:31:37.490 INFO:tasks.rados.rados.0.smithi071.stdout:324: read oid 14 snap -1 2024-09-11T01:31:37.491 INFO:tasks.rados.rados.0.smithi071.stdout:324: expect (ObjNum 104 snap 24 seq_num 104) 2024-09-11T01:31:37.491 INFO:tasks.rados.rados.0.smithi071.stdout:325: snap_create 2024-09-11T01:31:37.492 INFO:tasks.rados.rados.0.smithi071.stdout:320: expect (ObjNum 109 snap 26 seq_num 109) 2024-09-11T01:31:37.621 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1740254 2024-09-11T01:31:37.661 INFO:tasks.rados.rados.0.smithi071.stdout:317: finishing write tid 2 to smithi07134777-46 2024-09-11T01:31:37.661 INFO:tasks.rados.rados.0.smithi071.stdout:317: finishing write tid 3 to smithi07134777-46 2024-09-11T01:31:37.661 INFO:tasks.rados.rados.0.smithi071.stdout:317: finishing write tid 4 to smithi07134777-46 2024-09-11T01:31:37.661 INFO:tasks.rados.rados.0.smithi071.stdout:317: oid 46 updating version 0 to 20 2024-09-11T01:31:37.661 INFO:tasks.rados.rados.0.smithi071.stdout:317: oid 46 updating version 20 to 22 2024-09-11T01:31:37.661 INFO:tasks.rados.rados.0.smithi071.stdout:317: oid 46 version 22 is already newer than 21 2024-09-11T01:31:37.661 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 22 (ObjNum 135 snap 30 seq_num 135) dirty exists 2024-09-11T01:31:37.662 INFO:tasks.rados.rados.0.smithi071.stdout:317: left oid 46 (ObjNum 135 snap 30 seq_num 135) 2024-09-11T01:31:37.662 INFO:tasks.rados.rados.0.smithi071.stdout:319: expect (ObjNum 99 snap 23 seq_num 99) 2024-09-11T01:31:37.662 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:31:37.893 INFO:tasks.rados.rados.0.smithi071.stdout:321: finishing rollback tid 0 to smithi07134777-37 2024-09-11T01:31:37.894 INFO:tasks.rados.rados.0.smithi071.stdout:321: finishing rollback tid 1 to smithi07134777-37 2024-09-11T01:31:37.894 INFO:tasks.rados.rados.0.smithi071.stdout:321: finishing rollback tid 2 to smithi07134777-37 2024-09-11T01:31:37.894 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 109 (ObjNum 127 snap 29 seq_num 127) dirty exists 2024-09-11T01:31:37.894 INFO:tasks.rados.rados.0.smithi071.stdout:318: finishing copy_from to smithi07134777-44 2024-09-11T01:31:37.894 INFO:tasks.rados.rados.0.smithi071.stdout:318: got expected ENOENT (src dne) 2024-09-11T01:31:37.894 INFO:tasks.rados.rados.0.smithi071.stdout:318: finishing copy_from racing read to smithi07134777-44 2024-09-11T01:31:37.894 INFO:tasks.rados.rados.0.smithi071.stdout:324: expect (ObjNum 104 snap 24 seq_num 104) 2024-09-11T01:31:37.902 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1334114 2024-09-11T01:31:37.947 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 591455 2024-09-11T01:31:38.024 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2988431 2024-09-11T01:31:38.037 INFO:tasks.rados.rados.0.smithi071.stdout:323: finishing copy_from racing read to smithi07134777-24 2024-09-11T01:31:38.037 INFO:tasks.rados.rados.0.smithi071.stdout:322: finishing copy_from racing read to smithi07134777-21 2024-09-11T01:31:38.037 INFO:tasks.rados.rados.0.smithi071.stdout:323: finishing copy_from to smithi07134777-24 2024-09-11T01:31:38.037 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 49 (ObjNum 63 snap 8 seq_num 63) dirty exists 2024-09-11T01:31:38.037 INFO:tasks.rados.rados.0.smithi071.stdout:322: finishing copy_from to smithi07134777-21 2024-09-11T01:31:38.037 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 153 (ObjNum 132 snap 30 seq_num 132) dirty exists 2024-09-11T01:31:38.161 INFO:tasks.rados.rados.0.smithi071.stdout:317: done (8 left) 2024-09-11T01:31:38.161 INFO:tasks.rados.rados.0.smithi071.stdout:318: done (7 left) 2024-09-11T01:31:38.161 INFO:tasks.rados.rados.0.smithi071.stdout:319: done (6 left) 2024-09-11T01:31:38.161 INFO:tasks.rados.rados.0.smithi071.stdout:320: done (5 left) 2024-09-11T01:31:38.161 INFO:tasks.rados.rados.0.smithi071.stdout:321: done (4 left) 2024-09-11T01:31:38.162 INFO:tasks.rados.rados.0.smithi071.stdout:322: done (3 left) 2024-09-11T01:31:38.162 INFO:tasks.rados.rados.0.smithi071.stdout:323: done (2 left) 2024-09-11T01:31:38.162 INFO:tasks.rados.rados.0.smithi071.stdout:324: done (1 left) 2024-09-11T01:31:38.162 INFO:tasks.rados.rados.0.smithi071.stdout:325: done (0 left) 2024-09-11T01:31:38.162 INFO:tasks.rados.rados.0.smithi071.stdout:326: read oid 33 snap -1 2024-09-11T01:31:38.162 INFO:tasks.rados.rados.0.smithi071.stdout:326: expect (ObjNum 128 snap 29 seq_num 128) 2024-09-11T01:31:38.162 INFO:tasks.rados.rados.0.smithi071.stdout:327: write (excl) oid 41 current snap is 31 2024-09-11T01:31:38.162 INFO:tasks.rados.rados.0.smithi071.stdout:327: seq_num 136 ranges {685752=611865,1975551=310201} 2024-09-11T01:31:38.167 INFO:tasks.rados.rados.0.smithi071.stdout:327: writing smithi07134777-41 from 685752 to 1297617 tid 1 2024-09-11T01:31:38.173 INFO:tasks.rados.rados.0.smithi071.stdout:327: writing smithi07134777-41 from 1975551 to 2285752 tid 2 2024-09-11T01:31:38.173 INFO:tasks.rados.rados.0.smithi071.stdout:328: write (excl) oid 35 current snap is 31 2024-09-11T01:31:38.173 INFO:tasks.rados.rados.0.smithi071.stdout:328: seq_num 137 ranges {702559=403467,1750386=495032} 2024-09-11T01:31:38.179 INFO:tasks.rados.rados.0.smithi071.stdout:328: writing smithi07134777-35 from 702559 to 1106026 tid 1 2024-09-11T01:31:38.185 INFO:tasks.rados.rados.0.smithi071.stdout:328: writing smithi07134777-35 from 1750386 to 2245418 tid 2 2024-09-11T01:31:38.186 INFO:tasks.rados.rados.0.smithi071.stdout:329: rollback oid 8 current snap is 31 2024-09-11T01:31:38.186 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 8 to 30 2024-09-11T01:31:38.186 INFO:tasks.rados.rados.0.smithi071.stdout:326: expect (ObjNum 128 snap 29 seq_num 128) 2024-09-11T01:31:38.242 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1105526 2024-09-11T01:31:38.281 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 43746 2024-09-11T01:31:38.332 INFO:tasks.rados.rados.0.smithi071.stdout:327: finishing write tid 1 to smithi07134777-41 2024-09-11T01:31:38.332 INFO:tasks.rados.rados.0.smithi071.stdout:327: finishing write tid 2 to smithi07134777-41 2024-09-11T01:31:38.332 INFO:tasks.rados.rados.0.smithi071.stdout:327: finishing write tid 3 to smithi07134777-41 2024-09-11T01:31:38.333 INFO:tasks.rados.rados.0.smithi071.stdout:327: finishing write tid 4 to smithi07134777-41 2024-09-11T01:31:38.333 INFO:tasks.rados.rados.0.smithi071.stdout:327: oid 41 updating version 0 to 108 2024-09-11T01:31:38.333 INFO:tasks.rados.rados.0.smithi071.stdout:327: oid 41 version 108 is already newer than 107 2024-09-11T01:31:38.333 INFO:tasks.rados.rados.0.smithi071.stdout:327: oid 41 version 108 is already newer than 106 2024-09-11T01:31:38.333 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 108 (ObjNum 136 snap 31 seq_num 136) dirty exists 2024-09-11T01:31:38.333 INFO:tasks.rados.rados.0.smithi071.stdout:327: left oid 41 (ObjNum 136 snap 31 seq_num 136) 2024-09-11T01:31:38.333 INFO:tasks.rados.rados.0.smithi071.stdout:328: finishing write tid 1 to smithi07134777-35 2024-09-11T01:31:38.333 INFO:tasks.rados.rados.0.smithi071.stdout:328: finishing write tid 2 to smithi07134777-35 2024-09-11T01:31:38.333 INFO:tasks.rados.rados.0.smithi071.stdout:329: finishing rollback tid 0 to smithi07134777-8 2024-09-11T01:31:38.333 INFO:tasks.rados.rados.0.smithi071.stdout:329: finishing rollback tid 1 to smithi07134777-8 2024-09-11T01:31:38.333 INFO:tasks.rados.rados.0.smithi071.stdout:328: finishing write tid 3 to smithi07134777-35 2024-09-11T01:31:38.333 INFO:tasks.rados.rados.0.smithi071.stdout:328: finishing write tid 4 to smithi07134777-35 2024-09-11T01:31:38.333 INFO:tasks.rados.rados.0.smithi071.stdout:328: oid 35 updating version 0 to 112 2024-09-11T01:31:38.334 INFO:tasks.rados.rados.0.smithi071.stdout:328: oid 35 version 112 is already newer than 111 2024-09-11T01:31:38.334 INFO:tasks.rados.rados.0.smithi071.stdout:328: oid 35 updating version 112 to 113 2024-09-11T01:31:38.334 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 113 (ObjNum 137 snap 31 seq_num 137) dirty exists 2024-09-11T01:31:38.334 INFO:tasks.rados.rados.0.smithi071.stdout:328: left oid 35 (ObjNum 137 snap 31 seq_num 137) 2024-09-11T01:31:38.334 INFO:tasks.rados.rados.0.smithi071.stdout:329: finishing rollback tid 2 to smithi07134777-8 2024-09-11T01:31:38.334 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 55 (ObjNum 74 snap 13 seq_num 74) dirty exists 2024-09-11T01:31:38.334 INFO:tasks.rados.rados.0.smithi071.stdout:326: done (3 left) 2024-09-11T01:31:38.334 INFO:tasks.rados.rados.0.smithi071.stdout:327: done (2 left) 2024-09-11T01:31:38.334 INFO:tasks.rados.rados.0.smithi071.stdout:328: done (1 left) 2024-09-11T01:31:38.334 INFO:tasks.rados.rados.0.smithi071.stdout:329: done (0 left) 2024-09-11T01:31:38.334 INFO:tasks.rados.rados.0.smithi071.stdout:330: copy_from oid 2 from oid 26 current snap is 31 2024-09-11T01:31:38.335 INFO:tasks.rados.rados.0.smithi071.stdout:331: rollback oid 24 current snap is 31 2024-09-11T01:31:38.335 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 24 to 30 2024-09-11T01:31:38.335 INFO:tasks.rados.rados.0.smithi071.stdout:332: write (excl) oid 46 current snap is 31 2024-09-11T01:31:38.335 INFO:tasks.rados.rados.0.smithi071.stdout:332: seq_num 138 ranges {719366=478716,1808868=510498} 2024-09-11T01:31:38.338 INFO:tasks.rados.rados.0.smithi071.stdout:332: writing smithi07134777-46 from 719366 to 1198082 tid 1 2024-09-11T01:31:38.345 INFO:tasks.rados.rados.0.smithi071.stdout:332: writing smithi07134777-46 from 1808868 to 2319366 tid 2 2024-09-11T01:31:38.345 INFO:tasks.rados.rados.0.smithi071.stdout:333: write (excl) oid 21 current snap is 31 2024-09-11T01:31:38.345 INFO:tasks.rados.rados.0.smithi071.stdout:331: finishing rollback tid 0 to smithi07134777-24 2024-09-11T01:31:38.345 INFO:tasks.rados.rados.0.smithi071.stdout:331: finishing rollback tid 1 to smithi07134777-24 2024-09-11T01:31:38.346 INFO:tasks.rados.rados.0.smithi071.stdout:331: finishing rollback tid 2 to smithi07134777-24 2024-09-11T01:31:38.346 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 156 (ObjNum 63 snap 8 seq_num 63) dirty exists 2024-09-11T01:31:38.346 INFO:tasks.rados.rados.0.smithi071.stdout:330: finishing copy_from racing read to smithi07134777-2 2024-09-11T01:31:38.346 INFO:tasks.rados.rados.0.smithi071.stdout:333: seq_num 139 ranges {736173=553965,1867350=468823} 2024-09-11T01:31:38.352 INFO:tasks.rados.rados.0.smithi071.stdout:333: writing smithi07134777-21 from 736173 to 1290138 tid 1 2024-09-11T01:31:38.359 INFO:tasks.rados.rados.0.smithi071.stdout:333: writing smithi07134777-21 from 1867350 to 2336173 tid 2 2024-09-11T01:31:38.359 INFO:tasks.rados.rados.0.smithi071.stdout:331: done (3 left) 2024-09-11T01:31:38.359 INFO:tasks.rados.rados.0.smithi071.stdout:334: snap_create 2024-09-11T01:31:38.359 INFO:tasks.rados.rados.0.smithi071.stdout:332: finishing write tid 1 to smithi07134777-46 2024-09-11T01:31:38.359 INFO:tasks.rados.rados.0.smithi071.stdout:330: finishing copy_from to smithi07134777-2 2024-09-11T01:31:38.359 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 118 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-11T01:31:38.359 INFO:tasks.rados.rados.0.smithi071.stdout:332: finishing write tid 2 to smithi07134777-46 2024-09-11T01:31:38.359 INFO:tasks.rados.rados.0.smithi071.stdout:332: finishing write tid 3 to smithi07134777-46 2024-09-11T01:31:38.359 INFO:tasks.rados.rados.0.smithi071.stdout:332: finishing write tid 4 to smithi07134777-46 2024-09-11T01:31:38.359 INFO:tasks.rados.rados.0.smithi071.stdout:332: oid 46 updating version 0 to 24 2024-09-11T01:31:38.359 INFO:tasks.rados.rados.0.smithi071.stdout:332: oid 46 updating version 24 to 25 2024-09-11T01:31:38.360 INFO:tasks.rados.rados.0.smithi071.stdout:332: oid 46 updating version 25 to 26 2024-09-11T01:31:38.360 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 26 (ObjNum 138 snap 31 seq_num 138) dirty exists 2024-09-11T01:31:38.360 INFO:tasks.rados.rados.0.smithi071.stdout:332: left oid 46 (ObjNum 138 snap 31 seq_num 138) 2024-09-11T01:31:38.360 INFO:tasks.rados.rados.0.smithi071.stdout:333: finishing write tid 1 to smithi07134777-21 2024-09-11T01:31:38.360 INFO:tasks.rados.rados.0.smithi071.stdout:333: finishing write tid 2 to smithi07134777-21 2024-09-11T01:31:38.363 INFO:tasks.rados.rados.0.smithi071.stdout:333: finishing write tid 3 to smithi07134777-21 2024-09-11T01:31:38.363 INFO:tasks.rados.rados.0.smithi071.stdout:333: finishing write tid 4 to smithi07134777-21 2024-09-11T01:31:38.363 INFO:tasks.rados.rados.0.smithi071.stdout:333: oid 21 updating version 0 to 157 2024-09-11T01:31:38.363 INFO:tasks.rados.rados.0.smithi071.stdout:333: oid 21 updating version 157 to 159 2024-09-11T01:31:38.363 INFO:tasks.rados.rados.0.smithi071.stdout:333: oid 21 version 159 is already newer than 158 2024-09-11T01:31:38.363 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 159 (ObjNum 139 snap 31 seq_num 139) dirty exists 2024-09-11T01:31:38.363 INFO:tasks.rados.rados.0.smithi071.stdout:333: left oid 21 (ObjNum 139 snap 31 seq_num 139) 2024-09-11T01:31:39.165 INFO:tasks.rados.rados.0.smithi071.stdout:330: done (3 left) 2024-09-11T01:31:39.165 INFO:tasks.rados.rados.0.smithi071.stdout:332: done (2 left) 2024-09-11T01:31:39.165 INFO:tasks.rados.rados.0.smithi071.stdout:333: done (1 left) 2024-09-11T01:31:39.165 INFO:tasks.rados.rados.0.smithi071.stdout:334: done (0 left) 2024-09-11T01:31:39.165 INFO:tasks.rados.rados.0.smithi071.stdout:335: write (excl) oid 7 current snap is 32 2024-09-11T01:31:39.165 INFO:tasks.rados.rados.0.smithi071.stdout:335: seq_num 140 ranges {752980=629214,1809479=543501} 2024-09-11T01:31:39.172 INFO:tasks.rados.rados.0.smithi071.stdout:335: writing smithi07134777-7 from 752980 to 1382194 tid 1 2024-09-11T01:31:39.178 INFO:tasks.rados.rados.0.smithi071.stdout:335: writing smithi07134777-7 from 1809479 to 2352980 tid 2 2024-09-11T01:31:39.178 INFO:tasks.rados.rados.0.smithi071.stdout:336: write (excl) oid 18 current snap is 32 2024-09-11T01:31:39.178 INFO:tasks.rados.rados.0.smithi071.stdout:335: finishing write tid 1 to smithi07134777-7 2024-09-11T01:31:39.178 INFO:tasks.rados.rados.0.smithi071.stdout:336: seq_num 141 ranges {769787=704463,2267961=101826} 2024-09-11T01:31:39.186 INFO:tasks.rados.rados.0.smithi071.stdout:336: writing smithi07134777-18 from 769787 to 1474250 tid 1 2024-09-11T01:31:39.192 INFO:tasks.rados.rados.0.smithi071.stdout:336: writing smithi07134777-18 from 2267961 to 2369787 tid 2 2024-09-11T01:31:39.192 INFO:tasks.rados.rados.0.smithi071.stdout:337: read oid 39 snap -1 2024-09-11T01:31:39.192 INFO:tasks.rados.rados.0.smithi071.stdout:337: expect (ObjNum 118 snap 28 seq_num 118) 2024-09-11T01:31:39.192 INFO:tasks.rados.rados.0.smithi071.stdout:338: rollback oid 26 current snap is 32 2024-09-11T01:31:39.192 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 26 to 29 2024-09-11T01:31:39.192 INFO:tasks.rados.rados.0.smithi071.stdout:335: finishing write tid 2 to smithi07134777-7 2024-09-11T01:31:39.192 INFO:tasks.rados.rados.0.smithi071.stdout:335: finishing write tid 3 to smithi07134777-7 2024-09-11T01:31:39.193 INFO:tasks.rados.rados.0.smithi071.stdout:335: finishing write tid 4 to smithi07134777-7 2024-09-11T01:31:39.193 INFO:tasks.rados.rados.0.smithi071.stdout:335: oid 7 updating version 0 to 155 2024-09-11T01:31:39.193 INFO:tasks.rados.rados.0.smithi071.stdout:335: oid 7 updating version 155 to 157 2024-09-11T01:31:39.193 INFO:tasks.rados.rados.0.smithi071.stdout:335: oid 7 version 157 is already newer than 156 2024-09-11T01:31:39.193 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 157 (ObjNum 140 snap 32 seq_num 140) dirty exists 2024-09-11T01:31:39.193 INFO:tasks.rados.rados.0.smithi071.stdout:335: left oid 7 (ObjNum 140 snap 32 seq_num 140) 2024-09-11T01:31:39.193 INFO:tasks.rados.rados.0.smithi071.stdout:335: done (3 left) 2024-09-11T01:31:39.193 INFO:tasks.rados.rados.0.smithi071.stdout:339: read oid 37 snap 29 2024-09-11T01:31:39.193 INFO:tasks.rados.rados.0.smithi071.stdout:339: expect (ObjNum 127 snap 29 seq_num 127) 2024-09-11T01:31:39.193 INFO:tasks.rados.rados.0.smithi071.stdout:340: copy_from oid 17 from oid 45 current snap is 32 2024-09-11T01:31:39.193 INFO:tasks.rados.rados.0.smithi071.stdout:341: snap_remove snap 30 2024-09-11T01:31:40.170 INFO:tasks.rados.rados.0.smithi071.stdout:341: done (5 left) 2024-09-11T01:31:40.170 INFO:tasks.rados.rados.0.smithi071.stdout:342: delete oid 24 current snap is 32 2024-09-11T01:31:40.170 INFO:tasks.rados.rados.0.smithi071.stdout:338: finishing rollback tid 0 to smithi07134777-26 2024-09-11T01:31:40.170 INFO:tasks.rados.rados.0.smithi071.stdout:338: finishing rollback tid 1 to smithi07134777-26 2024-09-11T01:31:40.170 INFO:tasks.rados.rados.0.smithi071.stdout:338: finishing rollback tid 2 to smithi07134777-26 2024-09-11T01:31:40.170 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 155 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-11T01:31:40.170 INFO:tasks.rados.rados.0.smithi071.stdout:336: finishing write tid 1 to smithi07134777-18 2024-09-11T01:31:40.170 INFO:tasks.rados.rados.0.smithi071.stdout:336: finishing write tid 2 to smithi07134777-18 2024-09-11T01:31:40.170 INFO:tasks.rados.rados.0.smithi071.stdout:336: finishing write tid 3 to smithi07134777-18 2024-09-11T01:31:40.170 INFO:tasks.rados.rados.0.smithi071.stdout:336: finishing write tid 4 to smithi07134777-18 2024-09-11T01:31:40.171 INFO:tasks.rados.rados.0.smithi071.stdout:336: oid 18 updating version 0 to 106 2024-09-11T01:31:40.171 INFO:tasks.rados.rados.0.smithi071.stdout:336: oid 18 updating version 106 to 107 2024-09-11T01:31:40.171 INFO:tasks.rados.rados.0.smithi071.stdout:336: oid 18 version 107 is already newer than 105 2024-09-11T01:31:40.171 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 107 (ObjNum 141 snap 32 seq_num 141) dirty exists 2024-09-11T01:31:40.171 INFO:tasks.rados.rados.0.smithi071.stdout:336: left oid 18 (ObjNum 141 snap 32 seq_num 141) 2024-09-11T01:31:40.171 INFO:tasks.rados.rados.0.smithi071.stdout:337: expect (ObjNum 118 snap 28 seq_num 118) 2024-09-11T01:31:40.186 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3628853 2024-09-11T01:31:40.194 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:31:40.249 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2598863 2024-09-11T01:31:40.267 INFO:tasks.rados.rados.0.smithi071.stdout:339: expect (ObjNum 127 snap 29 seq_num 127) 2024-09-11T01:31:40.284 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:31:40.312 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:31:40.341 INFO:tasks.rados.rados.0.smithi071.stdout:340: finishing copy_from racing read to smithi07134777-17 2024-09-11T01:31:40.341 INFO:tasks.rados.rados.0.smithi071.stdout:340: finishing copy_from to smithi07134777-17 2024-09-11T01:31:40.341 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-11T01:31:40.341 INFO:tasks.rados.rados.0.smithi071.stdout:336: done (5 left) 2024-09-11T01:31:40.341 INFO:tasks.rados.rados.0.smithi071.stdout:337: done (4 left) 2024-09-11T01:31:40.341 INFO:tasks.rados.rados.0.smithi071.stdout:338: done (3 left) 2024-09-11T01:31:40.341 INFO:tasks.rados.rados.0.smithi071.stdout:339: done (2 left) 2024-09-11T01:31:40.341 INFO:tasks.rados.rados.0.smithi071.stdout:340: done (1 left) 2024-09-11T01:31:40.342 INFO:tasks.rados.rados.0.smithi071.stdout:342: done (0 left) 2024-09-11T01:31:40.342 INFO:tasks.rados.rados.0.smithi071.stdout:343: write (excl) oid 40 current snap is 32 2024-09-11T01:31:40.342 INFO:tasks.rados.rados.0.smithi071.stdout:343: seq_num 143 ranges {403401=454961,1584925=506039} 2024-09-11T01:31:40.345 INFO:tasks.rados.rados.0.smithi071.stdout:343: writing smithi07134777-40 from 403401 to 858362 tid 1 2024-09-11T01:31:40.353 INFO:tasks.rados.rados.0.smithi071.stdout:343: writing smithi07134777-40 from 1584925 to 2090964 tid 2 2024-09-11T01:31:40.353 INFO:tasks.rados.rados.0.smithi071.stdout:344: read oid 23 snap -1 2024-09-11T01:31:40.353 INFO:tasks.rados.rados.0.smithi071.stdout:344: expect (ObjNum 112 snap 27 seq_num 112) 2024-09-11T01:31:40.353 INFO:tasks.rados.rados.0.smithi071.stdout:345: write oid 50 current snap is 32 2024-09-11T01:31:40.353 INFO:tasks.rados.rados.0.smithi071.stdout:345: seq_num 144 ranges {420208=530210,1527054=766050} 2024-09-11T01:31:40.357 INFO:tasks.rados.rados.0.smithi071.stdout:345: writing smithi07134777-50 from 420208 to 950418 tid 1 2024-09-11T01:31:40.365 INFO:tasks.rados.rados.0.smithi071.stdout:345: writing smithi07134777-50 from 1527054 to 2293104 tid 2 2024-09-11T01:31:40.365 INFO:tasks.rados.rados.0.smithi071.stdout:346: snap_create 2024-09-11T01:31:40.365 INFO:tasks.rados.rados.0.smithi071.stdout:343: finishing write tid 1 to smithi07134777-40 2024-09-11T01:31:40.365 INFO:tasks.rados.rados.0.smithi071.stdout:344: expect (ObjNum 112 snap 27 seq_num 112) 2024-09-11T01:31:40.556 INFO:tasks.rados.rados.0.smithi071.stdout:343: finishing write tid 2 to smithi07134777-40 2024-09-11T01:31:40.556 INFO:tasks.rados.rados.0.smithi071.stdout:343: finishing write tid 3 to smithi07134777-40 2024-09-11T01:31:40.556 INFO:tasks.rados.rados.0.smithi071.stdout:343: finishing write tid 4 to smithi07134777-40 2024-09-11T01:31:40.556 INFO:tasks.rados.rados.0.smithi071.stdout:343: oid 40 updating version 0 to 30 2024-09-11T01:31:40.556 INFO:tasks.rados.rados.0.smithi071.stdout:343: oid 40 updating version 30 to 31 2024-09-11T01:31:40.556 INFO:tasks.rados.rados.0.smithi071.stdout:343: oid 40 updating version 31 to 32 2024-09-11T01:31:40.556 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 32 (ObjNum 143 snap 32 seq_num 143) dirty exists 2024-09-11T01:31:40.557 INFO:tasks.rados.rados.0.smithi071.stdout:343: left oid 40 (ObjNum 143 snap 32 seq_num 143) 2024-09-11T01:31:40.557 INFO:tasks.rados.rados.0.smithi071.stdout:345: finishing write tid 1 to smithi07134777-50 2024-09-11T01:31:40.557 INFO:tasks.rados.rados.0.smithi071.stdout:345: finishing write tid 2 to smithi07134777-50 2024-09-11T01:31:40.557 INFO:tasks.rados.rados.0.smithi071.stdout:345: finishing write tid 3 to smithi07134777-50 2024-09-11T01:31:40.557 INFO:tasks.rados.rados.0.smithi071.stdout:345: finishing write tid 4 to smithi07134777-50 2024-09-11T01:31:40.557 INFO:tasks.rados.rados.0.smithi071.stdout:345: oid 50 updating version 0 to 163 2024-09-11T01:31:40.557 INFO:tasks.rados.rados.0.smithi071.stdout:345: oid 50 version 163 is already newer than 162 2024-09-11T01:31:40.557 INFO:tasks.rados.rados.0.smithi071.stdout:345: oid 50 version 163 is already newer than 161 2024-09-11T01:31:40.557 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 163 (ObjNum 144 snap 32 seq_num 144) dirty exists 2024-09-11T01:31:40.557 INFO:tasks.rados.rados.0.smithi071.stdout:345: left oid 50 (ObjNum 144 snap 32 seq_num 144) 2024-09-11T01:31:41.244 INFO:tasks.rados.rados.0.smithi071.stdout:343: done (3 left) 2024-09-11T01:31:41.244 INFO:tasks.rados.rados.0.smithi071.stdout:344: done (2 left) 2024-09-11T01:31:41.245 INFO:tasks.rados.rados.0.smithi071.stdout:345: done (1 left) 2024-09-11T01:31:41.245 INFO:tasks.rados.rados.0.smithi071.stdout:346: done (0 left) 2024-09-11T01:31:41.245 INFO:tasks.rados.rados.0.smithi071.stdout:347: delete oid 35 current snap is 33 2024-09-11T01:31:41.250 INFO:tasks.rados.rados.0.smithi071.stdout:347: done (0 left) 2024-09-11T01:31:41.250 INFO:tasks.rados.rados.0.smithi071.stdout:348: snap_remove snap 29 2024-09-11T01:31:42.069 INFO:tasks.rados.rados.0.smithi071.stdout:348: done (0 left) 2024-09-11T01:31:42.069 INFO:tasks.rados.rados.0.smithi071.stdout:349: write oid 37 current snap is 33 2024-09-11T01:31:42.069 INFO:tasks.rados.rados.0.smithi071.stdout:349: seq_num 146 ranges {453822=797061,1760371=653366} 2024-09-11T01:31:42.076 INFO:tasks.rados.rados.0.smithi071.stdout:349: writing smithi07134777-37 from 453822 to 1250883 tid 1 2024-09-11T01:31:42.082 INFO:tasks.rados.rados.0.smithi071.stdout:349: writing smithi07134777-37 from 1760371 to 2413737 tid 2 2024-09-11T01:31:42.082 INFO:tasks.rados.rados.0.smithi071.stdout:350: read oid 33 snap 32 2024-09-11T01:31:42.083 INFO:tasks.rados.rados.0.smithi071.stdout:350: expect (ObjNum 128 snap 29 seq_num 128) 2024-09-11T01:31:42.083 INFO:tasks.rados.rados.0.smithi071.stdout:351: copy_from oid 21 from oid 36 current snap is 33 2024-09-11T01:31:42.083 INFO:tasks.rados.rados.0.smithi071.stdout:352: delete oid 23 current snap is 33 2024-09-11T01:31:42.086 INFO:tasks.rados.rados.0.smithi071.stdout:352: done (3 left) 2024-09-11T01:31:42.086 INFO:tasks.rados.rados.0.smithi071.stdout:353: snap_remove snap 31 2024-09-11T01:31:43.075 INFO:tasks.rados.rados.0.smithi071.stdout:353: done (3 left) 2024-09-11T01:31:43.075 INFO:tasks.rados.rados.0.smithi071.stdout:354: read oid 47 snap -1 2024-09-11T01:31:43.075 INFO:tasks.rados.rados.0.smithi071.stdout:354: expect (ObjNum 109 snap 26 seq_num 109) 2024-09-11T01:31:43.075 INFO:tasks.rados.rados.0.smithi071.stdout:355: read oid 31 snap -1 2024-09-11T01:31:43.075 INFO:tasks.rados.rados.0.smithi071.stdout:355: expect (ObjNum 123 snap 29 seq_num 123) 2024-09-11T01:31:43.075 INFO:tasks.rados.rados.0.smithi071.stdout:356: write (excl) oid 13 current snap is 33 2024-09-11T01:31:43.076 INFO:tasks.rados.rados.0.smithi071.stdout:356: seq_num 148 ranges {487436=547559,1760982=657035} 2024-09-11T01:31:43.081 INFO:tasks.rados.rados.0.smithi071.stdout:356: writing smithi07134777-13 from 487436 to 1034995 tid 1 2024-09-11T01:31:43.089 INFO:tasks.rados.rados.0.smithi071.stdout:356: writing smithi07134777-13 from 1760982 to 2418017 tid 2 2024-09-11T01:31:43.090 INFO:tasks.rados.rados.0.smithi071.stdout:357: copy_from oid 6 from oid 19 current snap is 33 2024-09-11T01:31:43.090 INFO:tasks.rados.rados.0.smithi071.stdout:350: expect (ObjNum 128 snap 29 seq_num 128) 2024-09-11T01:31:43.188 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 65475 2024-09-11T01:31:43.239 INFO:tasks.rados.rados.0.smithi071.stdout:349: finishing write tid 1 to smithi07134777-37 2024-09-11T01:31:43.239 INFO:tasks.rados.rados.0.smithi071.stdout:349: finishing write tid 2 to smithi07134777-37 2024-09-11T01:31:43.239 INFO:tasks.rados.rados.0.smithi071.stdout:349: finishing write tid 3 to smithi07134777-37 2024-09-11T01:31:43.239 INFO:tasks.rados.rados.0.smithi071.stdout:349: finishing write tid 4 to smithi07134777-37 2024-09-11T01:31:43.239 INFO:tasks.rados.rados.0.smithi071.stdout:349: oid 37 updating version 0 to 119 2024-09-11T01:31:43.239 INFO:tasks.rados.rados.0.smithi071.stdout:349: oid 37 updating version 119 to 121 2024-09-11T01:31:43.239 INFO:tasks.rados.rados.0.smithi071.stdout:349: oid 37 version 121 is already newer than 120 2024-09-11T01:31:43.239 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 121 (ObjNum 146 snap 33 seq_num 146) dirty exists 2024-09-11T01:31:43.239 INFO:tasks.rados.rados.0.smithi071.stdout:349: left oid 37 (ObjNum 146 snap 33 seq_num 146) 2024-09-11T01:31:43.240 INFO:tasks.rados.rados.0.smithi071.stdout:351: finishing copy_from racing read to smithi07134777-21 2024-09-11T01:31:43.240 INFO:tasks.rados.rados.0.smithi071.stdout:351: finishing copy_from to smithi07134777-21 2024-09-11T01:31:43.240 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 51 (ObjNum 126 snap 29 seq_num 126) dirty exists 2024-09-11T01:31:43.240 INFO:tasks.rados.rados.0.smithi071.stdout:354: expect (ObjNum 109 snap 26 seq_num 109) 2024-09-11T01:31:43.261 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2292938 2024-09-11T01:31:43.317 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3315027 2024-09-11T01:31:43.363 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3780627 2024-09-11T01:31:43.367 INFO:tasks.rados.rados.0.smithi071.stdout:355: expect (ObjNum 123 snap 29 seq_num 123) 2024-09-11T01:31:43.403 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:31:43.440 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:31:43.477 INFO:tasks.rados.rados.0.smithi071.stdout:356: finishing write tid 1 to smithi07134777-13 2024-09-11T01:31:43.477 INFO:tasks.rados.rados.0.smithi071.stdout:357: finishing copy_from to smithi07134777-6 2024-09-11T01:31:43.477 INFO:tasks.rados.rados.0.smithi071.stdout:357: got expected ENOENT (src dne) 2024-09-11T01:31:43.477 INFO:tasks.rados.rados.0.smithi071.stdout:357: finishing copy_from racing read to smithi07134777-6 2024-09-11T01:31:43.477 INFO:tasks.rados.rados.0.smithi071.stdout:356: finishing write tid 2 to smithi07134777-13 2024-09-11T01:31:43.477 INFO:tasks.rados.rados.0.smithi071.stdout:356: finishing write tid 3 to smithi07134777-13 2024-09-11T01:31:43.477 INFO:tasks.rados.rados.0.smithi071.stdout:356: finishing write tid 4 to smithi07134777-13 2024-09-11T01:31:43.477 INFO:tasks.rados.rados.0.smithi071.stdout:356: oid 13 updating version 0 to 101 2024-09-11T01:31:43.477 INFO:tasks.rados.rados.0.smithi071.stdout:356: oid 13 updating version 101 to 103 2024-09-11T01:31:43.478 INFO:tasks.rados.rados.0.smithi071.stdout:356: oid 13 version 103 is already newer than 102 2024-09-11T01:31:43.478 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 103 (ObjNum 148 snap 33 seq_num 148) dirty exists 2024-09-11T01:31:43.478 INFO:tasks.rados.rados.0.smithi071.stdout:356: left oid 13 (ObjNum 148 snap 33 seq_num 148) 2024-09-11T01:31:43.478 INFO:tasks.rados.rados.0.smithi071.stdout:349: done (6 left) 2024-09-11T01:31:43.478 INFO:tasks.rados.rados.0.smithi071.stdout:350: done (5 left) 2024-09-11T01:31:43.478 INFO:tasks.rados.rados.0.smithi071.stdout:351: done (4 left) 2024-09-11T01:31:43.478 INFO:tasks.rados.rados.0.smithi071.stdout:354: done (3 left) 2024-09-11T01:31:43.478 INFO:tasks.rados.rados.0.smithi071.stdout:355: done (2 left) 2024-09-11T01:31:43.478 INFO:tasks.rados.rados.0.smithi071.stdout:356: done (1 left) 2024-09-11T01:31:43.478 INFO:tasks.rados.rados.0.smithi071.stdout:357: done (0 left) 2024-09-11T01:31:43.478 INFO:tasks.rados.rados.0.smithi071.stdout:358: delete oid 13 current snap is 33 2024-09-11T01:31:43.479 INFO:tasks.rados.rados.0.smithi071.stdout:358: done (0 left) 2024-09-11T01:31:43.479 INFO:tasks.rados.rados.0.smithi071.stdout:359: snap_remove snap 32 2024-09-11T01:31:44.248 INFO:tasks.rados.rados.0.smithi071.stdout:359: done (0 left) 2024-09-11T01:31:44.248 INFO:tasks.rados.rados.0.smithi071.stdout:360: write oid 39 current snap is 33 2024-09-11T01:31:44.248 INFO:tasks.rados.rados.0.smithi071.stdout:360: seq_num 150 ranges {521050=698057,1877946=643104} 2024-09-11T01:31:44.254 INFO:tasks.rados.rados.0.smithi071.stdout:360: writing smithi07134777-39 from 521050 to 1219107 tid 1 2024-09-11T01:31:44.262 INFO:tasks.rados.rados.0.smithi071.stdout:360: writing smithi07134777-39 from 1877946 to 2521050 tid 2 2024-09-11T01:31:44.262 INFO:tasks.rados.rados.0.smithi071.stdout:361: write oid 23 current snap is 33 2024-09-11T01:31:44.262 INFO:tasks.rados.rados.0.smithi071.stdout:361: seq_num 151 ranges {537857=773306,1936428=520715} 2024-09-11T01:31:44.269 INFO:tasks.rados.rados.0.smithi071.stdout:361: writing smithi07134777-23 from 537857 to 1311163 tid 1 2024-09-11T01:31:44.275 INFO:tasks.rados.rados.0.smithi071.stdout:361: writing smithi07134777-23 from 1936428 to 2457143 tid 2 2024-09-11T01:31:44.276 INFO:tasks.rados.rados.0.smithi071.stdout:362: write oid 6 current snap is 33 2024-09-11T01:31:44.276 INFO:tasks.rados.rados.0.smithi071.stdout:360: finishing write tid 1 to smithi07134777-39 2024-09-11T01:31:44.276 INFO:tasks.rados.rados.0.smithi071.stdout:362: seq_num 152 ranges {554664=448555,1478557=664373} 2024-09-11T01:31:44.281 INFO:tasks.rados.rados.0.smithi071.stdout:362: writing smithi07134777-6 from 554664 to 1003219 tid 1 2024-09-11T01:31:44.288 INFO:tasks.rados.rados.0.smithi071.stdout:362: writing smithi07134777-6 from 1478557 to 2142930 tid 2 2024-09-11T01:31:44.288 INFO:tasks.rados.rados.0.smithi071.stdout:363: snap_create 2024-09-11T01:31:44.288 INFO:tasks.rados.rados.0.smithi071.stdout:360: finishing write tid 2 to smithi07134777-39 2024-09-11T01:31:44.288 INFO:tasks.rados.rados.0.smithi071.stdout:360: finishing write tid 3 to smithi07134777-39 2024-09-11T01:31:44.288 INFO:tasks.rados.rados.0.smithi071.stdout:360: finishing write tid 4 to smithi07134777-39 2024-09-11T01:31:44.288 INFO:tasks.rados.rados.0.smithi071.stdout:360: oid 39 updating version 0 to 136 2024-09-11T01:31:44.288 INFO:tasks.rados.rados.0.smithi071.stdout:360: oid 39 version 136 is already newer than 135 2024-09-11T01:31:44.288 INFO:tasks.rados.rados.0.smithi071.stdout:360: oid 39 version 136 is already newer than 134 2024-09-11T01:31:44.288 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 136 (ObjNum 150 snap 33 seq_num 150) dirty exists 2024-09-11T01:31:44.288 INFO:tasks.rados.rados.0.smithi071.stdout:360: left oid 39 (ObjNum 150 snap 33 seq_num 150) 2024-09-11T01:31:44.289 INFO:tasks.rados.rados.0.smithi071.stdout:361: finishing write tid 1 to smithi07134777-23 2024-09-11T01:31:44.289 INFO:tasks.rados.rados.0.smithi071.stdout:361: finishing write tid 2 to smithi07134777-23 2024-09-11T01:31:44.289 INFO:tasks.rados.rados.0.smithi071.stdout:361: finishing write tid 3 to smithi07134777-23 2024-09-11T01:31:44.289 INFO:tasks.rados.rados.0.smithi071.stdout:361: finishing write tid 4 to smithi07134777-23 2024-09-11T01:31:44.289 INFO:tasks.rados.rados.0.smithi071.stdout:361: oid 23 updating version 0 to 62 2024-09-11T01:31:44.289 INFO:tasks.rados.rados.0.smithi071.stdout:361: oid 23 version 62 is already newer than 61 2024-09-11T01:31:44.289 INFO:tasks.rados.rados.0.smithi071.stdout:361: oid 23 version 62 is already newer than 60 2024-09-11T01:31:44.289 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 62 (ObjNum 151 snap 33 seq_num 151) dirty exists 2024-09-11T01:31:44.289 INFO:tasks.rados.rados.0.smithi071.stdout:361: left oid 23 (ObjNum 151 snap 33 seq_num 151) 2024-09-11T01:31:44.289 INFO:tasks.rados.rados.0.smithi071.stdout:362: finishing write tid 1 to smithi07134777-6 2024-09-11T01:31:44.294 INFO:tasks.rados.rados.0.smithi071.stdout:362: finishing write tid 2 to smithi07134777-6 2024-09-11T01:31:44.294 INFO:tasks.rados.rados.0.smithi071.stdout:362: finishing write tid 3 to smithi07134777-6 2024-09-11T01:31:44.294 INFO:tasks.rados.rados.0.smithi071.stdout:362: finishing write tid 4 to smithi07134777-6 2024-09-11T01:31:44.294 INFO:tasks.rados.rados.0.smithi071.stdout:362: oid 6 updating version 0 to 158 2024-09-11T01:31:44.294 INFO:tasks.rados.rados.0.smithi071.stdout:362: oid 6 version 158 is already newer than 156 2024-09-11T01:31:44.294 INFO:tasks.rados.rados.0.smithi071.stdout:362: oid 6 version 158 is already newer than 157 2024-09-11T01:31:44.294 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 158 (ObjNum 152 snap 33 seq_num 152) dirty exists 2024-09-11T01:31:44.294 INFO:tasks.rados.rados.0.smithi071.stdout:362: left oid 6 (ObjNum 152 snap 33 seq_num 152) 2024-09-11T01:31:46.069 INFO:tasks.rados.rados.0.smithi071.stdout:360: done (3 left) 2024-09-11T01:31:46.069 INFO:tasks.rados.rados.0.smithi071.stdout:361: done (2 left) 2024-09-11T01:31:46.069 INFO:tasks.rados.rados.0.smithi071.stdout:362: done (1 left) 2024-09-11T01:31:46.069 INFO:tasks.rados.rados.0.smithi071.stdout:363: done (0 left) 2024-09-11T01:31:46.069 INFO:tasks.rados.rados.0.smithi071.stdout:364: read oid 5 snap -1 2024-09-11T01:31:46.069 INFO:tasks.rados.rados.0.smithi071.stdout:364: expect (ObjNum 99 snap 23 seq_num 99) 2024-09-11T01:31:46.069 INFO:tasks.rados.rados.0.smithi071.stdout:365: read oid 9 snap -1 2024-09-11T01:31:46.069 INFO:tasks.rados.rados.0.smithi071.stdout:365: expect deleted 2024-09-11T01:31:46.069 INFO:tasks.rados.rados.0.smithi071.stdout:366: read oid 8 snap -1 2024-09-11T01:31:46.069 INFO:tasks.rados.rados.0.smithi071.stdout:366: expect (ObjNum 74 snap 13 seq_num 74) 2024-09-11T01:31:46.070 INFO:tasks.rados.rados.0.smithi071.stdout:367: write oid 2 current snap is 34 2024-09-11T01:31:46.070 INFO:tasks.rados.rados.0.smithi071.stdout:367: seq_num 153 ranges {571471=640157,1653392=524384} 2024-09-11T01:31:46.076 INFO:tasks.rados.rados.0.smithi071.stdout:367: writing smithi07134777-2 from 571471 to 1211628 tid 1 2024-09-11T01:31:46.085 INFO:tasks.rados.rados.0.smithi071.stdout:367: writing smithi07134777-2 from 1653392 to 2177776 tid 2 2024-09-11T01:31:46.085 INFO:tasks.rados.rados.0.smithi071.stdout:368: read oid 10 snap -1 2024-09-11T01:31:46.085 INFO:tasks.rados.rados.0.smithi071.stdout:368: expect (ObjNum 88 snap 19 seq_num 88) 2024-09-11T01:31:46.085 INFO:tasks.rados.rados.0.smithi071.stdout:369: write (excl) oid 7 current snap is 34 2024-09-11T01:31:46.085 INFO:tasks.rados.rados.0.smithi071.stdout:369: seq_num 154 ranges {588278=715406,1711874=668042} 2024-09-11T01:31:46.089 INFO:tasks.rados.rados.0.smithi071.stdout:369: writing smithi07134777-7 from 588278 to 1303684 tid 1 2024-09-11T01:31:46.096 INFO:tasks.rados.rados.0.smithi071.stdout:369: writing smithi07134777-7 from 1711874 to 2379916 tid 2 2024-09-11T01:31:46.096 INFO:tasks.rados.rados.0.smithi071.stdout:370: rollback oid 50 current snap is 34 2024-09-11T01:31:46.097 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 50 to 33 2024-09-11T01:31:46.097 INFO:tasks.rados.rados.0.smithi071.stdout:364: expect (ObjNum 99 snap 23 seq_num 99) 2024-09-11T01:31:46.349 INFO:tasks.rados.rados.0.smithi071.stdout:366: expect (ObjNum 74 snap 13 seq_num 74) 2024-09-11T01:31:46.405 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 324802 2024-09-11T01:31:46.516 INFO:tasks.rados.rados.0.smithi071.stdout:367: finishing write tid 1 to smithi07134777-2 2024-09-11T01:31:46.516 INFO:tasks.rados.rados.0.smithi071.stdout:367: finishing write tid 2 to smithi07134777-2 2024-09-11T01:31:46.516 INFO:tasks.rados.rados.0.smithi071.stdout:367: finishing write tid 3 to smithi07134777-2 2024-09-11T01:31:46.516 INFO:tasks.rados.rados.0.smithi071.stdout:367: finishing write tid 4 to smithi07134777-2 2024-09-11T01:31:46.516 INFO:tasks.rados.rados.0.smithi071.stdout:367: oid 2 updating version 0 to 120 2024-09-11T01:31:46.516 INFO:tasks.rados.rados.0.smithi071.stdout:367: oid 2 version 120 is already newer than 119 2024-09-11T01:31:46.516 INFO:tasks.rados.rados.0.smithi071.stdout:367: oid 2 updating version 120 to 121 2024-09-11T01:31:46.516 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 121 (ObjNum 153 snap 34 seq_num 153) dirty exists 2024-09-11T01:31:46.516 INFO:tasks.rados.rados.0.smithi071.stdout:367: left oid 2 (ObjNum 153 snap 34 seq_num 153) 2024-09-11T01:31:46.516 INFO:tasks.rados.rados.0.smithi071.stdout:368: expect (ObjNum 88 snap 19 seq_num 88) 2024-09-11T01:31:46.566 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:31:46.679 INFO:tasks.rados.rados.0.smithi071.stdout:369: finishing write tid 1 to smithi07134777-7 2024-09-11T01:31:46.679 INFO:tasks.rados.rados.0.smithi071.stdout:369: finishing write tid 2 to smithi07134777-7 2024-09-11T01:31:46.679 INFO:tasks.rados.rados.0.smithi071.stdout:369: finishing write tid 3 to smithi07134777-7 2024-09-11T01:31:46.679 INFO:tasks.rados.rados.0.smithi071.stdout:369: finishing write tid 4 to smithi07134777-7 2024-09-11T01:31:46.679 INFO:tasks.rados.rados.0.smithi071.stdout:369: oid 7 updating version 0 to 177 2024-09-11T01:31:46.679 INFO:tasks.rados.rados.0.smithi071.stdout:369: oid 7 version 177 is already newer than 176 2024-09-11T01:31:46.679 INFO:tasks.rados.rados.0.smithi071.stdout:369: oid 7 version 177 is already newer than 175 2024-09-11T01:31:46.680 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 177 (ObjNum 154 snap 34 seq_num 154) dirty exists 2024-09-11T01:31:46.680 INFO:tasks.rados.rados.0.smithi071.stdout:369: left oid 7 (ObjNum 154 snap 34 seq_num 154) 2024-09-11T01:31:46.680 INFO:tasks.rados.rados.0.smithi071.stdout:370: finishing rollback tid 0 to smithi07134777-50 2024-09-11T01:31:46.680 INFO:tasks.rados.rados.0.smithi071.stdout:370: finishing rollback tid 1 to smithi07134777-50 2024-09-11T01:31:46.680 INFO:tasks.rados.rados.0.smithi071.stdout:364: done (6 left) 2024-09-11T01:31:46.680 INFO:tasks.rados.rados.0.smithi071.stdout:365: done (5 left) 2024-09-11T01:31:46.680 INFO:tasks.rados.rados.0.smithi071.stdout:366: done (4 left) 2024-09-11T01:31:46.680 INFO:tasks.rados.rados.0.smithi071.stdout:367: done (3 left) 2024-09-11T01:31:46.680 INFO:tasks.rados.rados.0.smithi071.stdout:368: done (2 left) 2024-09-11T01:31:46.680 INFO:tasks.rados.rados.0.smithi071.stdout:369: done (1 left) 2024-09-11T01:31:46.680 INFO:tasks.rados.rados.0.smithi071.stdout:371: delete oid 5 current snap is 34 2024-09-11T01:31:46.680 INFO:tasks.rados.rados.0.smithi071.stdout:370: finishing rollback tid 2 to smithi07134777-50 2024-09-11T01:31:46.680 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 181 (ObjNum 144 snap 32 seq_num 144) dirty exists 2024-09-11T01:31:46.683 INFO:tasks.rados.rados.0.smithi071.stdout:370: done (1 left) 2024-09-11T01:31:46.683 INFO:tasks.rados.rados.0.smithi071.stdout:371: done (0 left) 2024-09-11T01:31:46.683 INFO:tasks.rados.rados.0.smithi071.stdout:372: snap_remove snap 33 2024-09-11T01:31:46.708 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-11T01:31:46.709 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-11T01:31:46.709 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-11T01:31:46.711 INFO:tasks.ceph.mon.b:Started 2024-09-11T01:31:46.712 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:31:46.712 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:31:46.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.822+0000 7f97bfff8640 10 monclient: get_monmap_and_config 2024-09-11T01:31:46.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.822+0000 7f97bfff8640 10 monclient: build_initial_monmap 2024-09-11T01:31:46.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.822+0000 7f97bfff8640 10 monclient: monmap: 2024-09-11T01:31:46.824 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:46.825 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:46.825 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:31:46.823599+0000 2024-09-11T01:31:46.825 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:31:46.823599+0000 2024-09-11T01:31:46.825 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:46.825 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:46.825 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:31:46.825 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:31:46.825 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:31:46.825 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:31:46.825 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:31:46.825 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:46.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.823+0000 7f97bfff8640 1 Processor -- start 2024-09-11T01:31:46.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.823+0000 7f97bfff8640 1 -- start start 2024-09-11T01:31:46.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.823+0000 7f97bfff8640 10 monclient: init 2024-09-11T01:31:46.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97bfff8640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:46.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97bfff8640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-11T01:31:46.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97bfff8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f97b80a2e90 addr v1:172.21.15.12:6789/0 2024-09-11T01:31:46.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97bfff8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f97b80a3270 0x7f97b80a36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:46.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97bfff8640 10 monclient(hunting): picked mon.noname-a con 0x7f97b80a3270 addr v2:172.21.15.12:3300/0 2024-09-11T01:31:46.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97bfff8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97b813b150 0x7f97b813d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:46.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97bfff8640 10 monclient(hunting): picked mon.noname-b con 0x7f97b813b150 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:46.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97bfff8640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97b80a46c0 con 0x7f97b80a2e90 2024-09-11T01:31:46.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97bfff8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:46.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97bfff8640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f97b8072730 con 0x7f97b80a3270 2024-09-11T01:31:46.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97bfff8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:46.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97bfff8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f97b8072fa0 con 0x7f97b813b150 2024-09-11T01:31:46.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97bfff8640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:46.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97bfff8640 10 monclient(hunting): authenticate will time out at 913.266339s 2024-09-11T01:31:46.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97bdd94640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f97b80a2e90 0x7f97b8073d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34316/0 (socket says 172.21.15.12:34316) 2024-09-11T01:31:46.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97bdd94640 1 -- 172.21.15.12:0/173426769 learned_addr learned my addr 172.21.15.12:0/173426769 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:31:46.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97be595640 1 --2- 172.21.15.12:0/173426769 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97b813b150 0x7f97b813d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:46.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97bd593640 1 --2- 172.21.15.12:0/173426769 >> v2:172.21.15.12:3300/0 conn(0x7f97b80a3270 0x7f97b80a36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:46.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97be595640 10 monclient(hunting): get_auth_request con 0x7f97b813b150 auth_method 0 2024-09-11T01:31:46.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97be595640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:46.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97be595640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:46.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97be595640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:46.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97bd593640 10 monclient(hunting): get_auth_request con 0x7f97b80a3270 auth_method 0 2024-09-11T01:31:46.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97bd593640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:46.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97bd593640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:46.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97bd593640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:46.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97be595640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:46.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97be595640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:46.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97be595640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:46.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97bd593640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:46.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97bd593640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:46.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97bd593640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:46.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97be595640 10 monclient(hunting): handle_auth_done global_id 4616 payload 274 2024-09-11T01:31:46.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97be595640 1 -- 172.21.15.12:0/173426769 >> v2:172.21.15.12:3300/0 conn(0x7f97b80a3270 msgr2=0x7f97b80a36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:46.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97be595640 1 --2- 172.21.15.12:0/173426769 >> v2:172.21.15.12:3300/0 conn(0x7f97b80a3270 0x7f97b80a36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:46.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97be595640 1 -- 172.21.15.12:0/173426769 >> v1:172.21.15.12:6789/0 conn(0x7f97b80a2e90 legacy=0x7f97b8073d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:46.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97be595640 10 monclient: _finish_hunting 0 2024-09-11T01:31:46.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97be595640 1 monclient: found mon.noname-b 2024-09-11T01:31:46.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97be595640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:46.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97be595640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:31:46.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97be595640 1 -- 172.21.15.12:0/173426769 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b8073280 con 0x7f97b813b150 2024-09-11T01:31:46.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.824+0000 7f97bd593640 1 --2- 172.21.15.12:0/173426769 >> v2:172.21.15.12:3300/0 conn(0x7f97b80a3270 0x7f97b80a36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:31:46.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.827+0000 7f97be595640 1 -- 172.21.15.12:0/173426769 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97b813b150 msgr2=0x7f97b813d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:46.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.827+0000 7f97be595640 1 --2- 172.21.15.12:0/173426769 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97b813b150 0x7f97b813d540 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f97b0009dc0 tx=0x7f97b002a4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=162d342f8ac5bcb server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:46.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.827+0000 7f97bcd92640 1 -- 172.21.15.12:0/173426769 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b0007950 con 0x7f97b813b150 2024-09-11T01:31:46.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.827+0000 7f97bcd92640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:46.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.827+0000 7f97bcd92640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:31:46.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.827+0000 7f97bcd92640 10 monclient: dump: 2024-09-11T01:31:46.836 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:46.836 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:46.836 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:46.836 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:46.837 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:46.837 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:46.837 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:46.837 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:46.837 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:46.837 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:46.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.827+0000 7f97bcd92640 10 monclient: _finish_auth 0 2024-09-11T01:31:46.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.827+0000 7f97bcd92640 10 monclient: _check_auth_tickets 2024-09-11T01:31:46.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.827+0000 7f97bcd92640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:46.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.827+0000 7f97bfff8640 5 monclient: authenticate success, global_id 4616 2024-09-11T01:31:46.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.827+0000 7f97bcd92640 1 -- 172.21.15.12:0/173426769 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97b0007af0 con 0x7f97b813b150 2024-09-11T01:31:46.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.827+0000 7f97bfff8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:31:46.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.827+0000 7f97bcd92640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:46.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.827+0000 7f97bfff8640 10 monclient: get_monmap_and_config success 2024-09-11T01:31:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.827+0000 7f97bfff8640 4 set_mon_vals no callback set 2024-09-11T01:31:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.827+0000 7f97bcd92640 1 -- 172.21.15.12:0/173426769 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b0031400 con 0x7f97b813b150 2024-09-11T01:31:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.827+0000 7f97bcd92640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.827+0000 7f97bcd92640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.827+0000 7f97bcd92640 10 monclient: dump: 2024-09-11T01:31:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:46.838 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:46.839 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:46.839 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:46.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.828+0000 7f97bfff8640 10 monclient: shutdown 2024-09-11T01:31:46.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.828+0000 7f97bfff8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:46.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.828+0000 7f97bfff8640 1 -- 172.21.15.12:0/173426769 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97b813b150 msgr2=0x7f97b813d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:46.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.828+0000 7f97bfff8640 1 --2- 172.21.15.12:0/173426769 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97b813b150 0x7f97b813d540 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f97b0009dc0 tx=0x7f97b002a4d0 comp rx=0 tx=0).stop 2024-09-11T01:31:46.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.828+0000 7f97bfff8640 1 -- 172.21.15.12:0/173426769 shutdown_connections 2024-09-11T01:31:46.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.828+0000 7f97bfff8640 1 --2- 172.21.15.12:0/173426769 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97b813b150 0x7f97b813d540 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:46.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.828+0000 7f97bfff8640 1 --2- 172.21.15.12:0/173426769 >> v2:172.21.15.12:3300/0 conn(0x7f97b80a3270 0x7f97b80a36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:46.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.828+0000 7f97bfff8640 1 -- 172.21.15.12:0/173426769 >> 172.21.15.12:0/173426769 conn(0x7f97b80a8490 msgr2=0x7f97b80a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:46.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.828+0000 7f97bfff8640 1 -- 172.21.15.12:0/173426769 shutdown_connections 2024-09-11T01:31:46.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.828+0000 7f97bfff8640 1 -- 172.21.15.12:0/173426769 wait complete. 2024-09-11T01:31:46.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.829+0000 7f97bfff8640 10 monclient: build_initial_monmap 2024-09-11T01:31:46.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.829+0000 7f97bfff8640 10 monclient: monmap: 2024-09-11T01:31:46.840 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:46.840 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:31:46.840 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:31:46.840 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:31:46.840 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:46.840 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:46.840 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:31:46.840 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:31:46.840 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:31:46.840 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:46.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.829+0000 7f97bfff8640 1 Processor -- start 2024-09-11T01:31:46.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.829+0000 7f97bfff8640 1 -- start start 2024-09-11T01:31:46.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.829+0000 7f97bfff8640 10 monclient: init 2024-09-11T01:31:46.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.829+0000 7f97bfff8640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:46.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.829+0000 7f97bfff8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:31:46.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.829+0000 7f97bfff8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97b80a2e90 0x7f97b81cad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:46.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.829+0000 7f97bfff8640 10 monclient(hunting): picked mon.noname-a con 0x7f97b80a2e90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:46.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bfff8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97b80a3270 0x7f97b81cb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:46.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bfff8640 10 monclient(hunting): picked mon.noname-b con 0x7f97b80a3270 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:31:46.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bfff8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97b813b150 0x7f97b81cb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:46.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bfff8640 10 monclient(hunting): picked mon.noname-c con 0x7f97b813b150 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:46.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bfff8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:46.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bfff8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f97b813da80 con 0x7f97b80a2e90 2024-09-11T01:31:46.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bfff8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:46.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bfff8640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f97b80a3cf0 con 0x7f97b80a3270 2024-09-11T01:31:46.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bfff8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:46.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bfff8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f97b813dc00 con 0x7f97b813b150 2024-09-11T01:31:46.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bfff8640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:46.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bfff8640 10 monclient(hunting): authenticate will time out at 913.272073s 2024-09-11T01:31:46.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bdd94640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97b80a2e90 0x7f97b81cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:46.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bdd94640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97b80a2e90 0x7f97b81cad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40754/0 (socket says 172.21.15.12:40754) 2024-09-11T01:31:46.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bdd94640 1 -- 172.21.15.12:0/2483088010 learned_addr learned my addr 172.21.15.12:0/2483088010 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:46.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bdd94640 10 monclient(hunting): get_auth_request con 0x7f97b80a2e90 auth_method 0 2024-09-11T01:31:46.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bdd94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:46.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bdd94640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:46.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bdd94640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:46.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97be595640 1 --2- 172.21.15.12:0/2483088010 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97b813b150 0x7f97b81cb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:46.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97be595640 10 monclient(hunting): get_auth_request con 0x7f97b813b150 auth_method 0 2024-09-11T01:31:46.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97be595640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:46.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97be595640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:46.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97be595640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:46.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bdd94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:46.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bdd94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:46.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bdd94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:46.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bd593640 1 --2- 172.21.15.12:0/2483088010 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97b80a3270 0x7f97b81cb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:46.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97be595640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:46.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97be595640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:46.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97be595640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:46.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bd593640 10 monclient(hunting): get_auth_request con 0x7f97b80a3270 auth_method 0 2024-09-11T01:31:46.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bd593640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:46.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bd593640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:46.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bd593640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:46.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bdd94640 10 monclient(hunting): handle_auth_done global_id 24189 payload 824 2024-09-11T01:31:46.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bd593640 1 -- 172.21.15.12:0/2483088010 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97b80a3270 msgr2=0x7f97b81cb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:31:46.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bd593640 1 -- 172.21.15.12:0/2483088010 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97b80a3270 msgr2=0x7f97b81cb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:31:46.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bd593640 1 --2- 172.21.15.12:0/2483088010 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97b80a3270 0x7f97b81cb2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:31:46.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bdd94640 1 -- 172.21.15.12:0/2483088010 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97b813b150 msgr2=0x7f97b81cb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:46.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bdd94640 1 --2- 172.21.15.12:0/2483088010 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97b813b150 0x7f97b81cb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:46.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bdd94640 1 -- 172.21.15.12:0/2483088010 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97b80a3270 msgr2=0x7f97b81cb2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:31:46.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bd593640 1 --2- 172.21.15.12:0/2483088010 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97b80a3270 0x7f97b81cb2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:31:46.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bdd94640 1 --2- 172.21.15.12:0/2483088010 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97b80a3270 0x7f97b81cb2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:46.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bdd94640 10 monclient: _finish_hunting 0 2024-09-11T01:31:46.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bdd94640 1 monclient: found mon.noname-a 2024-09-11T01:31:46.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bdd94640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:46.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bdd94640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:31:46.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.830+0000 7f97bdd94640 1 -- 172.21.15.12:0/2483088010 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b813d5e0 con 0x7f97b80a2e90 2024-09-11T01:31:46.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.832+0000 7f97bdd94640 1 -- 172.21.15.12:0/2483088010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97b80a2e90 msgr2=0x7f97b81cad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:46.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.832+0000 7f97bdd94640 1 --2- 172.21.15.12:0/2483088010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97b80a2e90 0x7f97b81cad60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f97b400c460 tx=0x7f97b400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:46.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.832+0000 7f97ae7fc640 1 -- 172.21.15.12:0/2483088010 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b4007070 con 0x7f97b80a2e90 2024-09-11T01:31:46.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.832+0000 7f97ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:46.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.832+0000 7f97ae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:31:46.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.832+0000 7f97ae7fc640 10 monclient: dump: 2024-09-11T01:31:46.845 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:46.845 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:46.845 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:46.845 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:46.845 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:46.846 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:46.846 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:46.846 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:46.846 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:46.846 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:46.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.832+0000 7f97ae7fc640 10 monclient: _finish_auth 0 2024-09-11T01:31:46.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.832+0000 7f97ae7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:31:46.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.832+0000 7f97ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:46.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.832+0000 7f97ae7fc640 1 -- 172.21.15.12:0/2483088010 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97b4007230 con 0x7f97b80a2e90 2024-09-11T01:31:46.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.832+0000 7f97ae7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:46.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.832+0000 7f97ae7fc640 1 -- 172.21.15.12:0/2483088010 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b4007620 con 0x7f97b80a2e90 2024-09-11T01:31:46.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.832+0000 7f97bfff8640 5 monclient: authenticate success, global_id 24189 2024-09-11T01:31:46.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.832+0000 7f97bfff8640 10 monclient: _renew_subs 2024-09-11T01:31:46.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.832+0000 7f97bfff8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:46.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.832+0000 7f97bfff8640 1 -- 172.21.15.12:0/2483088010 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97b8139730 con 0x7f97b80a2e90 2024-09-11T01:31:46.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.832+0000 7f97affff640 4 set_mon_vals no callback set 2024-09-11T01:31:46.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.832+0000 7f97ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:46.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.832+0000 7f97ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:46.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.832+0000 7f97ae7fc640 10 monclient: dump: 2024-09-11T01:31:46.847 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:46.847 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:46.847 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:46.847 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:46.847 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:46.847 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:46.847 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:46.847 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:46.848 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:46.848 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:46.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.833+0000 7f97bfff8640 10 monclient: _renew_subs 2024-09-11T01:31:46.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.833+0000 7f97bfff8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:46.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.833+0000 7f97bfff8640 1 -- 172.21.15.12:0/2483088010 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97b81384f0 con 0x7f97b80a2e90 2024-09-11T01:31:46.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.834+0000 7f97ae7fc640 1 -- 172.21.15.12:0/2483088010 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f97b401b400 con 0x7f97b80a2e90 2024-09-11T01:31:46.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.834+0000 7f97bfff8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:46.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.834+0000 7f97bfff8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:46.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.834+0000 7f97bfff8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:46.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.834+0000 7f97bfff8640 1 -- 172.21.15.12:0/2483088010 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97b80a36a0 con 0x7f97b80a2e90 2024-09-11T01:31:46.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.834+0000 7f97ae7fc640 1 --2- 172.21.15.12:0/2483088010 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f979c03e260 0x7f979c040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:46.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.834+0000 7f97ae7fc640 1 -- 172.21.15.12:0/2483088010 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 4739+0+0 (secure 0 0 0) 0x7f97b4052960 con 0x7f97b80a2e90 2024-09-11T01:31:46.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.834+0000 7f97bd593640 1 --2- 172.21.15.12:0/2483088010 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f979c03e260 0x7f979c040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:46.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.834+0000 7f97bd593640 10 monclient: get_auth_request con 0x7f979c03e260 auth_method 0 2024-09-11T01:31:46.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.837+0000 7f97bd593640 1 --2- 172.21.15.12:0/2483088010 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f979c03e260 0x7f979c040720 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f97b809f380 tx=0x7f97a8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:46.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.838+0000 7f97ae7fc640 1 -- 172.21.15.12:0/2483088010 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f97b813d5e0 con 0x7f97b80a2e90 2024-09-11T01:31:46.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.838+0000 7f97ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:46.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:46.838+0000 7f97ae7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:31:47.070 INFO:tasks.rados.rados.0.smithi071.stdout:372: done (0 left) 2024-09-11T01:31:47.071 INFO:tasks.rados.rados.0.smithi071.stdout:373: rollback oid 44 current snap is 34 2024-09-11T01:31:47.071 INFO:tasks.rados.rados.0.smithi071.stdout:373: done (0 left) 2024-09-11T01:31:47.071 INFO:tasks.rados.rados.0.smithi071.stdout:374: delete oid 15 current snap is 34 2024-09-11T01:31:47.074 INFO:tasks.rados.rados.0.smithi071.stdout:374: done (0 left) 2024-09-11T01:31:47.074 INFO:tasks.rados.rados.0.smithi071.stdout:375: write oid 18 current snap is 34 2024-09-11T01:31:47.074 INFO:tasks.rados.rados.0.smithi071.stdout:375: seq_num 157 ranges {638699=541153,1770967=531722} 2024-09-11T01:31:47.080 INFO:tasks.rados.rados.0.smithi071.stdout:375: writing smithi07134777-18 from 638699 to 1179852 tid 1 2024-09-11T01:31:47.087 INFO:tasks.rados.rados.0.smithi071.stdout:375: writing smithi07134777-18 from 1770967 to 2302689 tid 2 2024-09-11T01:31:47.087 INFO:tasks.rados.rados.0.smithi071.stdout:376: delete oid 32 current snap is 34 2024-09-11T01:31:47.087 INFO:tasks.rados.rados.0.smithi071.stdout:375: finishing write tid 1 to smithi07134777-18 2024-09-11T01:31:47.089 INFO:tasks.rados.rados.0.smithi071.stdout:376: done (1 left) 2024-09-11T01:31:47.089 INFO:tasks.rados.rados.0.smithi071.stdout:377: snap_create 2024-09-11T01:31:47.090 INFO:tasks.rados.rados.0.smithi071.stdout:375: finishing write tid 2 to smithi07134777-18 2024-09-11T01:31:47.090 INFO:tasks.rados.rados.0.smithi071.stdout:375: finishing write tid 3 to smithi07134777-18 2024-09-11T01:31:47.090 INFO:tasks.rados.rados.0.smithi071.stdout:375: finishing write tid 4 to smithi07134777-18 2024-09-11T01:31:47.091 INFO:tasks.rados.rados.0.smithi071.stdout:375: oid 18 updating version 0 to 116 2024-09-11T01:31:47.091 INFO:tasks.rados.rados.0.smithi071.stdout:375: oid 18 updating version 116 to 117 2024-09-11T01:31:47.091 INFO:tasks.rados.rados.0.smithi071.stdout:375: oid 18 updating version 117 to 118 2024-09-11T01:31:47.091 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 118 (ObjNum 157 snap 34 seq_num 157) dirty exists 2024-09-11T01:31:47.091 INFO:tasks.rados.rados.0.smithi071.stdout:375: left oid 18 (ObjNum 157 snap 34 seq_num 157) 2024-09-11T01:31:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:47.100+0000 7f97bfff8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:31:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:47.100+0000 7f97bfff8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:31:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:47.100+0000 7f97bfff8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:47.100+0000 7f97bfff8640 1 -- 172.21.15.12:0/2483088010 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f97b809f600 con 0x7f97b80a2e90 2024-09-11T01:31:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:47.101+0000 7f97ae7fc640 1 -- 172.21.15.12:0/2483088010 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f97b4051380 con 0x7f97b80a2e90 2024-09-11T01:31:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:47.101+0000 7f97ae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:31:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:47.101+0000 7f97ae7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:31:47.103 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":26,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:31:47.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:47.103+0000 7f97bfff8640 1 -- 172.21.15.12:0/2483088010 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f979c03e260 msgr2=0x7f979c040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:47.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:47.103+0000 7f97bfff8640 1 --2- 172.21.15.12:0/2483088010 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f979c03e260 0x7f979c040720 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f97b809f380 tx=0x7f97a8008040 comp rx=0 tx=0).stop 2024-09-11T01:31:47.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:47.103+0000 7f97bfff8640 10 monclient: shutdown 2024-09-11T01:31:47.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:47.103+0000 7f97bfff8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:47.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:47.103+0000 7f97bfff8640 1 -- 172.21.15.12:0/2483088010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97b80a2e90 msgr2=0x7f97b81cad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:47.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:47.103+0000 7f97bfff8640 1 --2- 172.21.15.12:0/2483088010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97b80a2e90 0x7f97b81cad60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f97b400c460 tx=0x7f97b400c920 comp rx=0 tx=0).stop 2024-09-11T01:31:47.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:47.103+0000 7f97bfff8640 1 -- 172.21.15.12:0/2483088010 shutdown_connections 2024-09-11T01:31:47.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:47.103+0000 7f97bfff8640 1 --2- 172.21.15.12:0/2483088010 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97b813b150 0x7f97b81cb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:47.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:47.103+0000 7f97bfff8640 1 --2- 172.21.15.12:0/2483088010 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97b80a3270 0x7f97b81cb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:47.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:47.103+0000 7f97bfff8640 1 --2- 172.21.15.12:0/2483088010 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f979c03e260 0x7f979c040720 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:47.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:47.103+0000 7f97bfff8640 1 --2- 172.21.15.12:0/2483088010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97b80a2e90 0x7f97b81cad60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:47.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:47.103+0000 7f97bfff8640 1 -- 172.21.15.12:0/2483088010 >> 172.21.15.12:0/2483088010 conn(0x7f97b80a8490 msgr2=0x7f97b813cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:47.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:47.103+0000 7f97bfff8640 1 -- 172.21.15.12:0/2483088010 shutdown_connections 2024-09-11T01:31:47.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:47.103+0000 7f97bfff8640 1 -- 172.21.15.12:0/2483088010 wait complete. 2024-09-11T01:31:48.076 INFO:tasks.rados.rados.0.smithi071.stdout:375: done (1 left) 2024-09-11T01:31:48.076 INFO:tasks.rados.rados.0.smithi071.stdout:377: done (0 left) 2024-09-11T01:31:48.076 INFO:tasks.rados.rados.0.smithi071.stdout:378: copy_from oid 24 from oid 42 current snap is 35 2024-09-11T01:31:48.076 INFO:tasks.rados.rados.0.smithi071.stdout:379: read oid 37 snap 34 2024-09-11T01:31:48.076 INFO:tasks.rados.rados.0.smithi071.stdout:379: expect (ObjNum 146 snap 33 seq_num 146) 2024-09-11T01:31:48.076 INFO:tasks.rados.rados.0.smithi071.stdout:380: read oid 38 snap -1 2024-09-11T01:31:48.076 INFO:tasks.rados.rados.0.smithi071.stdout:380: expect (ObjNum 90 snap 19 seq_num 90) 2024-09-11T01:31:48.076 INFO:tasks.rados.rados.0.smithi071.stdout:381: write (excl) oid 44 current snap is 35 2024-09-11T01:31:48.077 INFO:tasks.rados.rados.0.smithi071.stdout:381: seq_num 159 ranges {672313=691651,1887931=419038} 2024-09-11T01:31:48.083 INFO:tasks.rados.rados.0.smithi071.stdout:381: writing smithi07134777-44 from 672313 to 1363964 tid 1 2024-09-11T01:31:48.090 INFO:tasks.rados.rados.0.smithi071.stdout:381: writing smithi07134777-44 from 1887931 to 2306969 tid 2 2024-09-11T01:31:48.090 INFO:tasks.rados.rados.0.smithi071.stdout:382: rollback oid 2 current snap is 35 2024-09-11T01:31:48.090 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 2 to 34 2024-09-11T01:31:48.090 INFO:tasks.rados.rados.0.smithi071.stdout:383: copy_from oid 10 from oid 47 current snap is 35 2024-09-11T01:31:48.090 INFO:tasks.rados.rados.0.smithi071.stdout:380: expect (ObjNum 90 snap 19 seq_num 90) 2024-09-11T01:31:48.247 INFO:tasks.rados.rados.0.smithi071.stdout:378: finishing copy_from racing read to smithi07134777-24 2024-09-11T01:31:48.247 INFO:tasks.rados.rados.0.smithi071.stdout:379: expect (ObjNum 146 snap 33 seq_num 146) 2024-09-11T01:31:48.329 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:31:48.374 INFO:tasks.rados.rados.0.smithi071.stdout:382: finishing rollback tid 0 to smithi07134777-2 2024-09-11T01:31:48.374 INFO:tasks.rados.rados.0.smithi071.stdout:381: finishing write tid 1 to smithi07134777-44 2024-09-11T01:31:48.374 INFO:tasks.rados.rados.0.smithi071.stdout:381: finishing write tid 2 to smithi07134777-44 2024-09-11T01:31:48.374 INFO:tasks.rados.rados.0.smithi071.stdout:382: finishing rollback tid 1 to smithi07134777-2 2024-09-11T01:31:48.374 INFO:tasks.rados.rados.0.smithi071.stdout:379: done (5 left) 2024-09-11T01:31:48.374 INFO:tasks.rados.rados.0.smithi071.stdout:380: done (4 left) 2024-09-11T01:31:48.374 INFO:tasks.rados.rados.0.smithi071.stdout:384: copy_from oid 50 from oid 7 current snap is 35 2024-09-11T01:31:48.374 INFO:tasks.rados.rados.0.smithi071.stdout:381: finishing write tid 3 to smithi07134777-44 2024-09-11T01:31:48.375 INFO:tasks.rados.rados.0.smithi071.stdout:378: finishing copy_from to smithi07134777-24 2024-09-11T01:31:48.375 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 19 (ObjNum 129 snap 29 seq_num 129) dirty exists 2024-09-11T01:31:48.375 INFO:tasks.rados.rados.0.smithi071.stdout:381: finishing write tid 4 to smithi07134777-44 2024-09-11T01:31:48.375 INFO:tasks.rados.rados.0.smithi071.stdout:381: oid 44 updating version 0 to 65 2024-09-11T01:31:48.375 INFO:tasks.rados.rados.0.smithi071.stdout:381: oid 44 version 65 is already newer than 64 2024-09-11T01:31:48.375 INFO:tasks.rados.rados.0.smithi071.stdout:381: oid 44 updating version 65 to 66 2024-09-11T01:31:48.375 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 66 (ObjNum 159 snap 35 seq_num 159) dirty exists 2024-09-11T01:31:48.375 INFO:tasks.rados.rados.0.smithi071.stdout:381: left oid 44 (ObjNum 159 snap 35 seq_num 159) 2024-09-11T01:31:48.375 INFO:tasks.rados.rados.0.smithi071.stdout:382: finishing rollback tid 2 to smithi07134777-2 2024-09-11T01:31:48.375 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 124 (ObjNum 153 snap 34 seq_num 153) dirty exists 2024-09-11T01:31:48.375 INFO:tasks.rados.rados.0.smithi071.stdout:383: finishing copy_from racing read to smithi07134777-10 2024-09-11T01:31:48.375 INFO:tasks.rados.rados.0.smithi071.stdout:383: finishing copy_from to smithi07134777-10 2024-09-11T01:31:48.375 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 76 (ObjNum 109 snap 26 seq_num 109) dirty exists 2024-09-11T01:31:48.375 INFO:tasks.rados.rados.0.smithi071.stdout:378: done (4 left) 2024-09-11T01:31:48.376 INFO:tasks.rados.rados.0.smithi071.stdout:381: done (3 left) 2024-09-11T01:31:48.376 INFO:tasks.rados.rados.0.smithi071.stdout:382: done (2 left) 2024-09-11T01:31:48.376 INFO:tasks.rados.rados.0.smithi071.stdout:383: done (1 left) 2024-09-11T01:31:48.376 INFO:tasks.rados.rados.0.smithi071.stdout:385: copy_from oid 46 from oid 2 current snap is 35 2024-09-11T01:31:48.376 INFO:tasks.rados.rados.0.smithi071.stdout:386: snap_remove snap 34 2024-09-11T01:31:49.086 INFO:tasks.rados.rados.0.smithi071.stdout:386: done (2 left) 2024-09-11T01:31:49.086 INFO:tasks.rados.rados.0.smithi071.stdout:387: snap_create 2024-09-11T01:31:49.086 INFO:tasks.rados.rados.0.smithi071.stdout:385: finishing copy_from racing read to smithi07134777-46 2024-09-11T01:31:49.086 INFO:tasks.rados.rados.0.smithi071.stdout:384: finishing copy_from racing read to smithi07134777-50 2024-09-11T01:31:49.086 INFO:tasks.rados.rados.0.smithi071.stdout:385: finishing copy_from to smithi07134777-46 2024-09-11T01:31:49.086 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 124 (ObjNum 153 snap 34 seq_num 153) dirty exists 2024-09-11T01:31:49.086 INFO:tasks.rados.rados.0.smithi071.stdout:384: finishing copy_from to smithi07134777-50 2024-09-11T01:31:49.087 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 177 (ObjNum 154 snap 34 seq_num 154) dirty exists 2024-09-11T01:31:50.084 INFO:tasks.rados.rados.0.smithi071.stdout:384: done (2 left) 2024-09-11T01:31:50.085 INFO:tasks.rados.rados.0.smithi071.stdout:385: done (1 left) 2024-09-11T01:31:50.085 INFO:tasks.rados.rados.0.smithi071.stdout:387: done (0 left) 2024-09-11T01:31:50.085 INFO:tasks.rados.rados.0.smithi071.stdout:388: delete oid 47 current snap is 36 2024-09-11T01:31:50.091 INFO:tasks.rados.rados.0.smithi071.stdout:388: done (0 left) 2024-09-11T01:31:50.091 INFO:tasks.rados.rados.0.smithi071.stdout:389: write (excl) oid 46 current snap is 36 2024-09-11T01:31:50.091 INFO:tasks.rados.rados.0.smithi071.stdout:389: seq_num 161 ranges {705927=558502,2004895=422707} 2024-09-11T01:31:50.097 INFO:tasks.rados.rados.0.smithi071.stdout:389: writing smithi07134777-46 from 705927 to 1264429 tid 1 2024-09-11T01:31:50.104 INFO:tasks.rados.rados.0.smithi071.stdout:389: writing smithi07134777-46 from 2004895 to 2427602 tid 2 2024-09-11T01:31:50.104 INFO:tasks.rados.rados.0.smithi071.stdout:390: write oid 50 current snap is 36 2024-09-11T01:31:50.104 INFO:tasks.rados.rados.0.smithi071.stdout:390: seq_num 162 ranges {722734=633751,2063377=659357} 2024-09-11T01:31:50.111 INFO:tasks.rados.rados.0.smithi071.stdout:390: writing smithi07134777-50 from 722734 to 1356485 tid 1 2024-09-11T01:31:50.116 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:31:50.119 INFO:tasks.rados.rados.0.smithi071.stdout:390: writing smithi07134777-50 from 2063377 to 2722734 tid 2 2024-09-11T01:31:50.119 INFO:tasks.rados.rados.0.smithi071.stdout:391: read oid 33 snap -1 2024-09-11T01:31:50.119 INFO:tasks.rados.rados.0.smithi071.stdout:391: expect (ObjNum 128 snap 29 seq_num 128) 2024-09-11T01:31:50.119 INFO:tasks.rados.rados.0.smithi071.stdout:392: write (excl) oid 49 current snap is 36 2024-09-11T01:31:50.119 INFO:tasks.rados.rados.0.smithi071.stdout:389: finishing write tid 1 to smithi07134777-46 2024-09-11T01:31:50.120 INFO:tasks.rados.rados.0.smithi071.stdout:392: seq_num 163 ranges {739541=709000,2121859=426376} 2024-09-11T01:31:50.127 INFO:tasks.rados.rados.0.smithi071.stdout:392: writing smithi07134777-49 from 739541 to 1448541 tid 1 2024-09-11T01:31:50.134 INFO:tasks.rados.rados.0.smithi071.stdout:392: writing smithi07134777-49 from 2121859 to 2548235 tid 2 2024-09-11T01:31:50.134 INFO:tasks.rados.rados.0.smithi071.stdout:393: copy_from oid 45 from oid 36 current snap is 36 2024-09-11T01:31:50.134 INFO:tasks.rados.rados.0.smithi071.stdout:389: finishing write tid 2 to smithi07134777-46 2024-09-11T01:31:50.134 INFO:tasks.rados.rados.0.smithi071.stdout:389: finishing write tid 3 to smithi07134777-46 2024-09-11T01:31:50.134 INFO:tasks.rados.rados.0.smithi071.stdout:389: finishing write tid 4 to smithi07134777-46 2024-09-11T01:31:50.135 INFO:tasks.rados.rados.0.smithi071.stdout:389: oid 46 updating version 0 to 126 2024-09-11T01:31:50.135 INFO:tasks.rados.rados.0.smithi071.stdout:389: oid 46 updating version 126 to 127 2024-09-11T01:31:50.135 INFO:tasks.rados.rados.0.smithi071.stdout:389: oid 46 version 127 is already newer than 125 2024-09-11T01:31:50.135 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 127 (ObjNum 161 snap 36 seq_num 161) dirty exists 2024-09-11T01:31:50.135 INFO:tasks.rados.rados.0.smithi071.stdout:389: left oid 46 (ObjNum 161 snap 36 seq_num 161) 2024-09-11T01:31:50.135 INFO:tasks.rados.rados.0.smithi071.stdout:390: finishing write tid 1 to smithi07134777-50 2024-09-11T01:31:50.135 INFO:tasks.rados.rados.0.smithi071.stdout:390: finishing write tid 2 to smithi07134777-50 2024-09-11T01:31:50.135 INFO:tasks.rados.rados.0.smithi071.stdout:390: finishing write tid 3 to smithi07134777-50 2024-09-11T01:31:50.136 INFO:tasks.rados.rados.0.smithi071.stdout:390: finishing write tid 4 to smithi07134777-50 2024-09-11T01:31:50.136 INFO:tasks.rados.rados.0.smithi071.stdout:390: oid 50 updating version 0 to 193 2024-09-11T01:31:50.136 INFO:tasks.rados.rados.0.smithi071.stdout:390: oid 50 version 193 is already newer than 192 2024-09-11T01:31:50.136 INFO:tasks.rados.rados.0.smithi071.stdout:390: oid 50 updating version 193 to 194 2024-09-11T01:31:50.136 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 194 (ObjNum 162 snap 36 seq_num 162) dirty exists 2024-09-11T01:31:50.136 INFO:tasks.rados.rados.0.smithi071.stdout:390: left oid 50 (ObjNum 162 snap 36 seq_num 162) 2024-09-11T01:31:50.136 INFO:tasks.rados.rados.0.smithi071.stdout:389: done (4 left) 2024-09-11T01:31:50.136 INFO:tasks.rados.rados.0.smithi071.stdout:390: done (3 left) 2024-09-11T01:31:50.136 INFO:tasks.rados.rados.0.smithi071.stdout:394: read oid 1 snap -1 2024-09-11T01:31:50.136 INFO:tasks.rados.rados.0.smithi071.stdout:394: expect (ObjNum 92 snap 19 seq_num 92) 2024-09-11T01:31:50.136 INFO:tasks.rados.rados.0.smithi071.stdout:395: read oid 32 snap -1 2024-09-11T01:31:50.137 INFO:tasks.rados.rados.0.smithi071.stdout:395: expect deleted 2024-09-11T01:31:50.137 INFO:tasks.rados.rados.0.smithi071.stdout:396: snap_remove snap 35 2024-09-11T01:31:50.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.228+0000 7fcc16a85640 10 monclient: get_monmap_and_config 2024-09-11T01:31:50.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.228+0000 7fcc16a85640 10 monclient: build_initial_monmap 2024-09-11T01:31:50.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.228+0000 7fcc16a85640 10 monclient: monmap: 2024-09-11T01:31:50.230 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:50.231 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:50.231 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:31:50.229421+0000 2024-09-11T01:31:50.231 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:31:50.229421+0000 2024-09-11T01:31:50.231 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:50.231 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:50.231 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:31:50.231 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:31:50.231 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:31:50.231 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:31:50.231 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:31:50.231 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:50.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.229+0000 7fcc16a85640 1 Processor -- start 2024-09-11T01:31:50.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.229+0000 7fcc16a85640 1 -- start start 2024-09-11T01:31:50.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.229+0000 7fcc16a85640 10 monclient: init 2024-09-11T01:31:50.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.229+0000 7fcc16a85640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:50.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc16a85640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-11T01:31:50.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc16a85640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc1009da80 0x7fcc10136010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:50.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc16a85640 10 monclient(hunting): picked mon.noname-b con 0x7fcc1009da80 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:50.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc16a85640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcc1009de60 addr v1:172.21.15.71:6789/0 2024-09-11T01:31:50.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc16a85640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcc100a34c0 0x7fcc1013de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:50.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc16a85640 10 monclient(hunting): picked mon.noname-a con 0x7fcc100a34c0 addr v2:172.21.15.12:3300/0 2024-09-11T01:31:50.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc16a85640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc1006d5b0 con 0x7fcc1009de60 2024-09-11T01:31:50.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc16a85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:50.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc16a85640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fcc100a1590 con 0x7fcc100a34c0 2024-09-11T01:31:50.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc16a85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:50.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc16a85640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcc1006d890 con 0x7fcc1009da80 2024-09-11T01:31:50.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc16a85640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:50.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc16a85640 10 monclient(hunting): authenticate will time out at 916.672199s 2024-09-11T01:31:50.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc15022640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcc100a34c0 0x7fcc1013de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:50.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc14821640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc1009da80 0x7fcc10136010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:50.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc15022640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcc100a34c0 0x7fcc1013de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40764/0 (socket says 172.21.15.12:40764) 2024-09-11T01:31:50.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc15022640 1 -- 172.21.15.12:0/1179721877 learned_addr learned my addr 172.21.15.12:0/1179721877 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:50.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc14821640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc1009da80 0x7fcc10136010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41278/0 (socket says 172.21.15.12:41278) 2024-09-11T01:31:50.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc15022640 10 monclient(hunting): get_auth_request con 0x7fcc100a34c0 auth_method 0 2024-09-11T01:31:50.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc15022640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:50.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc15022640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:50.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc15022640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:50.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc14821640 10 monclient(hunting): get_auth_request con 0x7fcc1009da80 auth_method 0 2024-09-11T01:31:50.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc14821640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:50.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc14821640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:50.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc14821640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:50.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc14821640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:50.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc14821640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:50.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc14821640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:50.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc15022640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:50.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc15022640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:50.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc15022640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:50.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc14821640 10 monclient(hunting): handle_auth_done global_id 4622 payload 274 2024-09-11T01:31:50.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc14821640 1 -- 172.21.15.12:0/1179721877 >> v2:172.21.15.12:3300/0 conn(0x7fcc100a34c0 msgr2=0x7fcc1013de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:50.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc14821640 1 --2- 172.21.15.12:0/1179721877 >> v2:172.21.15.12:3300/0 conn(0x7fcc100a34c0 0x7fcc1013de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:50.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc14821640 1 -- 172.21.15.12:0/1179721877 >> v1:172.21.15.71:6789/0 conn(0x7fcc1009de60 legacy=0x7fcc100a2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:50.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc14821640 10 monclient: _finish_hunting 0 2024-09-11T01:31:50.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc14821640 1 monclient: found mon.noname-b 2024-09-11T01:31:50.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc14821640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:50.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc14821640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:31:50.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc14821640 1 -- 172.21.15.12:0/1179721877 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc1006d2d0 con 0x7fcc1009da80 2024-09-11T01:31:50.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.230+0000 7fcc15022640 1 --2- 172.21.15.12:0/1179721877 >> v2:172.21.15.12:3300/0 conn(0x7fcc100a34c0 0x7fcc1013de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:31:50.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.233+0000 7fcc14821640 1 -- 172.21.15.12:0/1179721877 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc1009da80 msgr2=0x7fcc10136010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:50.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.233+0000 7fcc14821640 1 --2- 172.21.15.12:0/1179721877 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc1009da80 0x7fcc10136010 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c00dbc0 tx=0x7fcc0c02e2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0299f2a587dd5bd server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:50.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.233+0000 7fcc077fe640 1 -- 172.21.15.12:0/1179721877 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c00b650 con 0x7fcc1009da80 2024-09-11T01:31:50.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.233+0000 7fcc077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:50.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.233+0000 7fcc077fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:31:50.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.233+0000 7fcc077fe640 10 monclient: dump: 2024-09-11T01:31:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:50.245 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.233+0000 7fcc077fe640 10 monclient: _finish_auth 0 2024-09-11T01:31:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.233+0000 7fcc077fe640 10 monclient: _check_auth_tickets 2024-09-11T01:31:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.233+0000 7fcc077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.233+0000 7fcc16a85640 5 monclient: authenticate success, global_id 4622 2024-09-11T01:31:50.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.233+0000 7fcc16a85640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:31:50.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.233+0000 7fcc077fe640 1 -- 172.21.15.12:0/1179721877 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc0c00b7f0 con 0x7fcc1009da80 2024-09-11T01:31:50.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.233+0000 7fcc077fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:50.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.233+0000 7fcc16a85640 10 monclient: get_monmap_and_config success 2024-09-11T01:31:50.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.233+0000 7fcc16a85640 4 set_mon_vals no callback set 2024-09-11T01:31:50.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.233+0000 7fcc077fe640 1 -- 172.21.15.12:0/1179721877 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c00bbc0 con 0x7fcc1009da80 2024-09-11T01:31:50.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.233+0000 7fcc077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:50.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.233+0000 7fcc077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:50.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.233+0000 7fcc077fe640 10 monclient: dump: 2024-09-11T01:31:50.246 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:50.246 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:50.246 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:50.246 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:50.246 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:50.247 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.234+0000 7fcc16a85640 10 monclient: shutdown 2024-09-11T01:31:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.234+0000 7fcc16a85640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.234+0000 7fcc16a85640 1 -- 172.21.15.12:0/1179721877 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc1009da80 msgr2=0x7fcc10136010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.234+0000 7fcc16a85640 1 --2- 172.21.15.12:0/1179721877 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc1009da80 0x7fcc10136010 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c00dbc0 tx=0x7fcc0c02e2c0 comp rx=0 tx=0).stop 2024-09-11T01:31:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.234+0000 7fcc16a85640 1 -- 172.21.15.12:0/1179721877 shutdown_connections 2024-09-11T01:31:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.234+0000 7fcc16a85640 1 --2- 172.21.15.12:0/1179721877 >> v2:172.21.15.12:3300/0 conn(0x7fcc100a34c0 0x7fcc1013de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.234+0000 7fcc16a85640 1 --2- 172.21.15.12:0/1179721877 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc1009da80 0x7fcc10136010 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.234+0000 7fcc16a85640 1 -- 172.21.15.12:0/1179721877 >> 172.21.15.12:0/1179721877 conn(0x7fcc100a3ea0 msgr2=0x7fcc100a42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.234+0000 7fcc16a85640 1 -- 172.21.15.12:0/1179721877 shutdown_connections 2024-09-11T01:31:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.234+0000 7fcc16a85640 1 -- 172.21.15.12:0/1179721877 wait complete. 2024-09-11T01:31:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.234+0000 7fcc16a85640 10 monclient: build_initial_monmap 2024-09-11T01:31:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.234+0000 7fcc16a85640 10 monclient: monmap: 2024-09-11T01:31:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:31:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:31:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:31:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:31:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:31:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:31:50.248 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.235+0000 7fcc16a85640 1 Processor -- start 2024-09-11T01:31:50.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.235+0000 7fcc16a85640 1 -- start start 2024-09-11T01:31:50.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.235+0000 7fcc16a85640 10 monclient: init 2024-09-11T01:31:50.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.235+0000 7fcc16a85640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:50.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.235+0000 7fcc16a85640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:31:50.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.235+0000 7fcc16a85640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc1009da80 0x7fcc10134950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:50.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.235+0000 7fcc16a85640 10 monclient(hunting): picked mon.noname-c con 0x7fcc1009da80 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:50.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.235+0000 7fcc16a85640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc1009de60 0x7fcc10134e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:50.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.235+0000 7fcc16a85640 10 monclient(hunting): picked mon.noname-b con 0x7fcc1009de60 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:31:50.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.235+0000 7fcc16a85640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc100a34c0 0x7fcc101c8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.235+0000 7fcc16a85640 10 monclient(hunting): picked mon.noname-a con 0x7fcc100a34c0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.235+0000 7fcc16a85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.235+0000 7fcc16a85640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcc100a0ea0 con 0x7fcc100a34c0 2024-09-11T01:31:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.235+0000 7fcc16a85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.235+0000 7fcc16a85640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fcc100a0d20 con 0x7fcc1009de60 2024-09-11T01:31:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.235+0000 7fcc16a85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.235+0000 7fcc16a85640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcc1009e240 con 0x7fcc1009da80 2024-09-11T01:31:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.235+0000 7fcc16a85640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.235+0000 7fcc16a85640 10 monclient(hunting): authenticate will time out at 916.677998s 2024-09-11T01:31:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc14821640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc1009da80 0x7fcc10134950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc15022640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc100a34c0 0x7fcc101c8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc14821640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc1009da80 0x7fcc10134950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41288/0 (socket says 172.21.15.12:41288) 2024-09-11T01:31:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc14821640 1 -- 172.21.15.12:0/555026652 learned_addr learned my addr 172.21.15.12:0/555026652 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc14821640 10 monclient(hunting): get_auth_request con 0x7fcc1009da80 auth_method 0 2024-09-11T01:31:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc14821640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc14821640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc14821640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc15022640 10 monclient(hunting): get_auth_request con 0x7fcc100a34c0 auth_method 0 2024-09-11T01:31:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc15022640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc15022640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc15022640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc14821640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc14821640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc14821640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc15022640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc15022640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc15022640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc07fff640 1 --2- 172.21.15.12:0/555026652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc1009de60 0x7fcc10134e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc07fff640 10 monclient(hunting): get_auth_request con 0x7fcc1009de60 auth_method 0 2024-09-11T01:31:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc07fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc14821640 10 monclient(hunting): handle_auth_done global_id 4625 payload 824 2024-09-11T01:31:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc14821640 1 -- 172.21.15.12:0/555026652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc1009de60 msgr2=0x7fcc10134e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc14821640 1 --2- 172.21.15.12:0/555026652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc1009de60 0x7fcc10134e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc14821640 1 -- 172.21.15.12:0/555026652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc100a34c0 msgr2=0x7fcc101c8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:50.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc14821640 1 --2- 172.21.15.12:0/555026652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc100a34c0 0x7fcc101c8da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:50.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc14821640 10 monclient: _finish_hunting 0 2024-09-11T01:31:50.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc14821640 1 monclient: found mon.noname-c 2024-09-11T01:31:50.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc14821640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:50.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc14821640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:31:50.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc14821640 1 -- 172.21.15.12:0/555026652 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc1009d4a0 con 0x7fcc1009da80 2024-09-11T01:31:50.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.236+0000 7fcc15022640 1 --2- 172.21.15.12:0/555026652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc100a34c0 0x7fcc101c8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:31:50.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.238+0000 7fcc14821640 1 -- 172.21.15.12:0/555026652 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc1009da80 msgr2=0x7fcc10134950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:50.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.238+0000 7fcc14821640 1 --2- 172.21.15.12:0/555026652 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc1009da80 0x7fcc10134950 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c038040 tx=0x7fcc0c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:50.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.238+0000 7fcc057fa640 1 -- 172.21.15.12:0/555026652 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c045020 con 0x7fcc1009da80 2024-09-11T01:31:50.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.238+0000 7fcc057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:50.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.238+0000 7fcc057fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:31:50.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.238+0000 7fcc057fa640 10 monclient: dump: 2024-09-11T01:31:50.255 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:50.255 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:50.255 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:50.255 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:50.255 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:50.256 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:50.256 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:50.256 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:50.256 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:50.256 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:50.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.238+0000 7fcc057fa640 10 monclient: _finish_auth 0 2024-09-11T01:31:50.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.238+0000 7fcc057fa640 10 monclient: _check_auth_tickets 2024-09-11T01:31:50.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.238+0000 7fcc057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:50.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.238+0000 7fcc16a85640 5 monclient: authenticate success, global_id 4625 2024-09-11T01:31:50.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.238+0000 7fcc057fa640 1 -- 172.21.15.12:0/555026652 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc0c040870 con 0x7fcc1009da80 2024-09-11T01:31:50.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.238+0000 7fcc16a85640 10 monclient: _renew_subs 2024-09-11T01:31:50.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.238+0000 7fcc16a85640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:50.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.238+0000 7fcc16a85640 1 -- 172.21.15.12:0/555026652 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc10130000 con 0x7fcc1009da80 2024-09-11T01:31:50.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.238+0000 7fcc057fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:50.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.238+0000 7fcc057fa640 1 -- 172.21.15.12:0/555026652 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c040c60 con 0x7fcc1009da80 2024-09-11T01:31:50.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.238+0000 7fcc057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:50.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.238+0000 7fcc057fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:50.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.238+0000 7fcc057fa640 10 monclient: dump: 2024-09-11T01:31:50.259 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:50.259 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:50.259 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:50.259 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:50.259 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:50.260 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:50.260 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:50.260 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:50.260 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:50.260 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:50.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.238+0000 7fcc16a85640 10 monclient: _renew_subs 2024-09-11T01:31:50.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.238+0000 7fcc16a85640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:50.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.238+0000 7fcc16a85640 1 -- 172.21.15.12:0/555026652 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc10074600 con 0x7fcc1009da80 2024-09-11T01:31:50.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.238+0000 7fcc06ffd640 4 set_mon_vals no callback set 2024-09-11T01:31:50.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.239+0000 7fcc16a85640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:50.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.239+0000 7fcc16a85640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:50.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.239+0000 7fcc16a85640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:50.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.239+0000 7fcc16a85640 1 -- 172.21.15.12:0/555026652 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc10074cc0 con 0x7fcc1009da80 2024-09-11T01:31:50.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.239+0000 7fcc057fa640 1 -- 172.21.15.12:0/555026652 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fcc0c041400 con 0x7fcc1009da80 2024-09-11T01:31:50.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.240+0000 7fcc057fa640 1 --2- 172.21.15.12:0/555026652 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcbfc03e260 0x7fcbfc040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:50.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.240+0000 7fcc057fa640 1 -- 172.21.15.12:0/555026652 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(94..94 src has 1..94) ==== 4739+0+0 (secure 0 0 0) 0x7fcc0c077b40 con 0x7fcc1009da80 2024-09-11T01:31:50.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.240+0000 7fcc07fff640 1 --2- 172.21.15.12:0/555026652 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcbfc03e260 0x7fcbfc040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:50.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.240+0000 7fcc07fff640 10 monclient: get_auth_request con 0x7fcbfc03e260 auth_method 0 2024-09-11T01:31:50.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.242+0000 7fcc07fff640 1 --2- 172.21.15.12:0/555026652 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcbfc03e260 0x7fcbfc040720 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcc00004ea0 tx=0x7fcc00009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:50.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.245+0000 7fcc057fa640 1 -- 172.21.15.12:0/555026652 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fcc10074cc0 con 0x7fcc1009da80 2024-09-11T01:31:50.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.245+0000 7fcc057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:50.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.245+0000 7fcc057fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:31:50.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.503+0000 7fcc16a85640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:31:50.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.503+0000 7fcc16a85640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:31:50.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.503+0000 7fcc16a85640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:50.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.503+0000 7fcc16a85640 1 -- 172.21.15.12:0/555026652 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcc100a38f0 con 0x7fcc1009da80 2024-09-11T01:31:50.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.503+0000 7fcc057fa640 1 -- 172.21.15.12:0/555026652 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fcc0c04f030 con 0x7fcc1009da80 2024-09-11T01:31:50.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.504+0000 7fcc057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:31:50.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.504+0000 7fcc057fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:31:50.506 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":26,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:31:50.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.506+0000 7fcc16a85640 1 -- 172.21.15.12:0/555026652 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcbfc03e260 msgr2=0x7fcbfc040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:50.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.506+0000 7fcc16a85640 1 --2- 172.21.15.12:0/555026652 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcbfc03e260 0x7fcbfc040720 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcc00004ea0 tx=0x7fcc00009040 comp rx=0 tx=0).stop 2024-09-11T01:31:50.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.506+0000 7fcc16a85640 10 monclient: shutdown 2024-09-11T01:31:50.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.506+0000 7fcc16a85640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:50.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.506+0000 7fcc16a85640 1 -- 172.21.15.12:0/555026652 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc1009da80 msgr2=0x7fcc10134950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:50.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.506+0000 7fcc16a85640 1 --2- 172.21.15.12:0/555026652 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc1009da80 0x7fcc10134950 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c038040 tx=0x7fcc0c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:31:50.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.506+0000 7fcc16a85640 1 -- 172.21.15.12:0/555026652 shutdown_connections 2024-09-11T01:31:50.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.506+0000 7fcc16a85640 1 --2- 172.21.15.12:0/555026652 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcbfc03e260 0x7fcbfc040720 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:50.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.506+0000 7fcc16a85640 1 --2- 172.21.15.12:0/555026652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc100a34c0 0x7fcc101c8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:50.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.506+0000 7fcc16a85640 1 --2- 172.21.15.12:0/555026652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc1009de60 0x7fcc10134e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:50.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.506+0000 7fcc16a85640 1 --2- 172.21.15.12:0/555026652 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc1009da80 0x7fcc10134950 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:50.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.506+0000 7fcc16a85640 1 -- 172.21.15.12:0/555026652 >> 172.21.15.12:0/555026652 conn(0x7fcc100a3ea0 msgr2=0x7fcc1009e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:50.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.506+0000 7fcc16a85640 1 -- 172.21.15.12:0/555026652 shutdown_connections 2024-09-11T01:31:50.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:50.506+0000 7fcc16a85640 1 -- 172.21.15.12:0/555026652 wait complete. 2024-09-11T01:31:51.097 INFO:tasks.rados.rados.0.smithi071.stdout:396: done (5 left) 2024-09-11T01:31:51.097 INFO:tasks.rados.rados.0.smithi071.stdout:397: write oid 43 current snap is 36 2024-09-11T01:31:51.097 INFO:tasks.rados.rados.0.smithi071.stdout:391: expect (ObjNum 128 snap 29 seq_num 128) 2024-09-11T01:31:51.101 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 728821 2024-09-11T01:31:51.145 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:31:51.246 INFO:tasks.rados.rados.0.smithi071.stdout:392: finishing write tid 1 to smithi07134777-49 2024-09-11T01:31:51.246 INFO:tasks.rados.rados.0.smithi071.stdout:392: finishing write tid 2 to smithi07134777-49 2024-09-11T01:31:51.246 INFO:tasks.rados.rados.0.smithi071.stdout:392: finishing write tid 3 to smithi07134777-49 2024-09-11T01:31:51.246 INFO:tasks.rados.rados.0.smithi071.stdout:392: finishing write tid 4 to smithi07134777-49 2024-09-11T01:31:51.246 INFO:tasks.rados.rados.0.smithi071.stdout:392: oid 49 updating version 0 to 13 2024-09-11T01:31:51.246 INFO:tasks.rados.rados.0.smithi071.stdout:392: oid 49 updating version 13 to 15 2024-09-11T01:31:51.246 INFO:tasks.rados.rados.0.smithi071.stdout:392: oid 49 version 15 is already newer than 14 2024-09-11T01:31:51.246 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 15 (ObjNum 163 snap 36 seq_num 163) dirty exists 2024-09-11T01:31:51.247 INFO:tasks.rados.rados.0.smithi071.stdout:392: left oid 49 (ObjNum 163 snap 36 seq_num 163) 2024-09-11T01:31:51.247 INFO:tasks.rados.rados.0.smithi071.stdout:394: expect (ObjNum 92 snap 19 seq_num 92) 2024-09-11T01:31:51.247 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 80314 2024-09-11T01:31:51.340 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 902643 2024-09-11T01:31:51.380 INFO:tasks.rados.rados.0.smithi071.stdout:393: finishing copy_from racing read to smithi07134777-45 2024-09-11T01:31:51.381 INFO:tasks.rados.rados.0.smithi071.stdout:393: finishing copy_from to smithi07134777-45 2024-09-11T01:31:51.381 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 51 (ObjNum 126 snap 29 seq_num 126) dirty exists 2024-09-11T01:31:51.381 INFO:tasks.rados.rados.0.smithi071.stdout:397: seq_num 164 ranges {756348=784249,2063988=686387} 2024-09-11T01:31:51.388 INFO:tasks.rados.rados.0.smithi071.stdout:397: writing smithi07134777-43 from 756348 to 1540597 tid 1 2024-09-11T01:31:51.396 INFO:tasks.rados.rados.0.smithi071.stdout:397: writing smithi07134777-43 from 2063988 to 2750375 tid 2 2024-09-11T01:31:51.396 INFO:tasks.rados.rados.0.smithi071.stdout:391: done (5 left) 2024-09-11T01:31:51.396 INFO:tasks.rados.rados.0.smithi071.stdout:392: done (4 left) 2024-09-11T01:31:51.396 INFO:tasks.rados.rados.0.smithi071.stdout:393: done (3 left) 2024-09-11T01:31:51.396 INFO:tasks.rados.rados.0.smithi071.stdout:394: done (2 left) 2024-09-11T01:31:51.396 INFO:tasks.rados.rados.0.smithi071.stdout:395: done (1 left) 2024-09-11T01:31:51.396 INFO:tasks.rados.rados.0.smithi071.stdout:398: copy_from oid 50 from oid 34 current snap is 36 2024-09-11T01:31:51.396 INFO:tasks.rados.rados.0.smithi071.stdout:397: finishing write tid 1 to smithi07134777-43 2024-09-11T01:31:51.397 INFO:tasks.rados.rados.0.smithi071.stdout:399: read oid 3 snap -1 2024-09-11T01:31:51.397 INFO:tasks.rados.rados.0.smithi071.stdout:399: expect (ObjNum 125 snap 29 seq_num 125) 2024-09-11T01:31:51.397 INFO:tasks.rados.rados.0.smithi071.stdout:400: write oid 48 current snap is 36 2024-09-11T01:31:51.397 INFO:tasks.rados.rados.0.smithi071.stdout:400: seq_num 165 ranges {773155=459498,1722470=430045,2648676=124479} 2024-09-11T01:31:51.402 INFO:tasks.rados.rados.0.smithi071.stdout:400: writing smithi07134777-48 from 773155 to 1232653 tid 1 2024-09-11T01:31:51.407 INFO:tasks.rados.rados.0.smithi071.stdout:400: writing smithi07134777-48 from 1722470 to 2152515 tid 2 2024-09-11T01:31:51.411 INFO:tasks.rados.rados.0.smithi071.stdout:400: writing smithi07134777-48 from 2648676 to 2773155 tid 3 2024-09-11T01:31:51.411 INFO:tasks.rados.rados.0.smithi071.stdout:401: delete oid 37 current snap is 36 2024-09-11T01:31:51.411 INFO:tasks.rados.rados.0.smithi071.stdout:399: expect (ObjNum 125 snap 29 seq_num 125) 2024-09-11T01:31:51.430 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 263428 2024-09-11T01:31:51.486 INFO:tasks.rados.rados.0.smithi071.stdout:398: finishing copy_from racing read to smithi07134777-50 2024-09-11T01:31:51.486 INFO:tasks.rados.rados.0.smithi071.stdout:397: finishing write tid 2 to smithi07134777-43 2024-09-11T01:31:51.486 INFO:tasks.rados.rados.0.smithi071.stdout:398: finishing copy_from to smithi07134777-50 2024-09-11T01:31:51.486 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 152 (ObjNum 130 snap 30 seq_num 130) dirty exists 2024-09-11T01:31:51.486 INFO:tasks.rados.rados.0.smithi071.stdout:397: finishing write tid 3 to smithi07134777-43 2024-09-11T01:31:51.486 INFO:tasks.rados.rados.0.smithi071.stdout:397: finishing write tid 4 to smithi07134777-43 2024-09-11T01:31:51.486 INFO:tasks.rados.rados.0.smithi071.stdout:397: oid 43 updating version 0 to 159 2024-09-11T01:31:51.486 INFO:tasks.rados.rados.0.smithi071.stdout:397: oid 43 updating version 159 to 161 2024-09-11T01:31:51.486 INFO:tasks.rados.rados.0.smithi071.stdout:397: oid 43 version 161 is already newer than 160 2024-09-11T01:31:51.486 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 161 (ObjNum 164 snap 36 seq_num 164) dirty exists 2024-09-11T01:31:51.487 INFO:tasks.rados.rados.0.smithi071.stdout:397: left oid 43 (ObjNum 164 snap 36 seq_num 164) 2024-09-11T01:31:51.487 INFO:tasks.rados.rados.0.smithi071.stdout:400: finishing write tid 1 to smithi07134777-48 2024-09-11T01:31:51.487 INFO:tasks.rados.rados.0.smithi071.stdout:397: done (4 left) 2024-09-11T01:31:51.487 INFO:tasks.rados.rados.0.smithi071.stdout:398: done (3 left) 2024-09-11T01:31:51.487 INFO:tasks.rados.rados.0.smithi071.stdout:399: done (2 left) 2024-09-11T01:31:51.487 INFO:tasks.rados.rados.0.smithi071.stdout:401: done (1 left) 2024-09-11T01:31:51.487 INFO:tasks.rados.rados.0.smithi071.stdout:402: write (excl) oid 47 current snap is 36 2024-09-11T01:31:51.487 INFO:tasks.rados.rados.0.smithi071.stdout:400: finishing write tid 2 to smithi07134777-48 2024-09-11T01:31:51.487 INFO:tasks.rados.rados.0.smithi071.stdout:400: finishing write tid 3 to smithi07134777-48 2024-09-11T01:31:51.487 INFO:tasks.rados.rados.0.smithi071.stdout:400: finishing write tid 4 to smithi07134777-48 2024-09-11T01:31:51.487 INFO:tasks.rados.rados.0.smithi071.stdout:400: finishing write tid 5 to smithi07134777-48 2024-09-11T01:31:51.487 INFO:tasks.rados.rados.0.smithi071.stdout:400: oid 48 updating version 0 to 200 2024-09-11T01:31:51.487 INFO:tasks.rados.rados.0.smithi071.stdout:400: oid 48 updating version 200 to 201 2024-09-11T01:31:51.488 INFO:tasks.rados.rados.0.smithi071.stdout:400: oid 48 version 201 is already newer than 199 2024-09-11T01:31:51.488 INFO:tasks.rados.rados.0.smithi071.stdout:400: oid 48 version 201 is already newer than 198 2024-09-11T01:31:51.488 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 201 (ObjNum 165 snap 36 seq_num 165) dirty exists 2024-09-11T01:31:51.488 INFO:tasks.rados.rados.0.smithi071.stdout:400: left oid 48 (ObjNum 165 snap 36 seq_num 165) 2024-09-11T01:31:51.488 INFO:tasks.rados.rados.0.smithi071.stdout:402: seq_num 167 ranges {406769=609996,1439434=433714,2303522=503247} 2024-09-11T01:31:51.490 INFO:tasks.rados.rados.0.smithi071.stdout:402: writing smithi07134777-47 from 406769 to 1016765 tid 1 2024-09-11T01:31:51.495 INFO:tasks.rados.rados.0.smithi071.stdout:402: writing smithi07134777-47 from 1439434 to 1873148 tid 2 2024-09-11T01:31:51.501 INFO:tasks.rados.rados.0.smithi071.stdout:402: writing smithi07134777-47 from 2303522 to 2806769 tid 3 2024-09-11T01:31:51.501 INFO:tasks.rados.rados.0.smithi071.stdout:400: done (1 left) 2024-09-11T01:31:51.502 INFO:tasks.rados.rados.0.smithi071.stdout:403: read oid 5 snap -1 2024-09-11T01:31:51.502 INFO:tasks.rados.rados.0.smithi071.stdout:403: expect deleted 2024-09-11T01:31:51.502 INFO:tasks.rados.rados.0.smithi071.stdout:404: read oid 26 snap -1 2024-09-11T01:31:51.502 INFO:tasks.rados.rados.0.smithi071.stdout:404: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:31:51.502 INFO:tasks.rados.rados.0.smithi071.stdout:405: delete oid 42 current snap is 36 2024-09-11T01:31:51.502 INFO:tasks.rados.rados.0.smithi071.stdout:402: finishing write tid 1 to smithi07134777-47 2024-09-11T01:31:51.503 INFO:tasks.rados.rados.0.smithi071.stdout:402: finishing write tid 2 to smithi07134777-47 2024-09-11T01:31:51.506 INFO:tasks.rados.rados.0.smithi071.stdout:403: done (3 left) 2024-09-11T01:31:51.507 INFO:tasks.rados.rados.0.smithi071.stdout:405: done (2 left) 2024-09-11T01:31:51.507 INFO:tasks.rados.rados.0.smithi071.stdout:406: read oid 6 snap -1 2024-09-11T01:31:51.507 INFO:tasks.rados.rados.0.smithi071.stdout:406: expect (ObjNum 152 snap 33 seq_num 152) 2024-09-11T01:31:51.507 INFO:tasks.rados.rados.0.smithi071.stdout:407: copy_from oid 13 from oid 41 current snap is 36 2024-09-11T01:31:51.507 INFO:tasks.rados.rados.0.smithi071.stdout:408: delete oid 38 current snap is 36 2024-09-11T01:31:51.507 INFO:tasks.rados.rados.0.smithi071.stdout:402: finishing write tid 3 to smithi07134777-47 2024-09-11T01:31:51.508 INFO:tasks.rados.rados.0.smithi071.stdout:402: finishing write tid 4 to smithi07134777-47 2024-09-11T01:31:51.508 INFO:tasks.rados.rados.0.smithi071.stdout:402: finishing write tid 5 to smithi07134777-47 2024-09-11T01:31:51.508 INFO:tasks.rados.rados.0.smithi071.stdout:402: oid 47 updating version 0 to 112 2024-09-11T01:31:51.508 INFO:tasks.rados.rados.0.smithi071.stdout:402: oid 47 version 112 is already newer than 111 2024-09-11T01:31:51.508 INFO:tasks.rados.rados.0.smithi071.stdout:402: oid 47 version 112 is already newer than 110 2024-09-11T01:31:51.508 INFO:tasks.rados.rados.0.smithi071.stdout:402: oid 47 version 112 is already newer than 109 2024-09-11T01:31:51.509 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 112 (ObjNum 167 snap 36 seq_num 167) dirty exists 2024-09-11T01:31:51.509 INFO:tasks.rados.rados.0.smithi071.stdout:402: left oid 47 (ObjNum 167 snap 36 seq_num 167) 2024-09-11T01:31:51.509 INFO:tasks.rados.rados.0.smithi071.stdout:402: done (4 left) 2024-09-11T01:31:51.509 INFO:tasks.rados.rados.0.smithi071.stdout:408: done (3 left) 2024-09-11T01:31:51.509 INFO:tasks.rados.rados.0.smithi071.stdout:409: copy_from oid 3 from oid 27 current snap is 36 2024-09-11T01:31:51.509 INFO:tasks.rados.rados.0.smithi071.stdout:410: rollback oid 30 current snap is 36 2024-09-11T01:31:51.510 INFO:tasks.rados.rados.0.smithi071.stdout:410: done (4 left) 2024-09-11T01:31:51.510 INFO:tasks.rados.rados.0.smithi071.stdout:411: read oid 15 snap -1 2024-09-11T01:31:51.510 INFO:tasks.rados.rados.0.smithi071.stdout:411: expect deleted 2024-09-11T01:31:51.510 INFO:tasks.rados.rados.0.smithi071.stdout:412: rollback oid 20 current snap is 36 2024-09-11T01:31:51.510 INFO:tasks.rados.rados.0.smithi071.stdout:412: done (5 left) 2024-09-11T01:31:51.510 INFO:tasks.rados.rados.0.smithi071.stdout:413: copy_from oid 33 from oid 44 current snap is 36 2024-09-11T01:31:51.511 INFO:tasks.rados.rados.0.smithi071.stdout:414: rollback oid 42 current snap is 36 2024-09-11T01:31:51.511 INFO:tasks.rados.rados.0.smithi071.stdout:414: done (6 left) 2024-09-11T01:31:51.511 INFO:tasks.rados.rados.0.smithi071.stdout:415: read oid 46 snap -1 2024-09-11T01:31:51.511 INFO:tasks.rados.rados.0.smithi071.stdout:415: expect (ObjNum 161 snap 36 seq_num 161) 2024-09-11T01:31:51.511 INFO:tasks.rados.rados.0.smithi071.stdout:416: rollback oid 42 current snap is 36 2024-09-11T01:31:51.511 INFO:tasks.rados.rados.0.smithi071.stdout:416: done (7 left) 2024-09-11T01:31:51.512 INFO:tasks.rados.rados.0.smithi071.stdout:417: read oid 50 snap -1 2024-09-11T01:31:51.512 INFO:tasks.rados.rados.0.smithi071.stdout:417: expect (ObjNum 130 snap 30 seq_num 130) 2024-09-11T01:31:51.512 INFO:tasks.rados.rados.0.smithi071.stdout:418: snap_create 2024-09-11T01:31:51.513 INFO:tasks.rados.rados.0.smithi071.stdout:404: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:31:51.561 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1097612 2024-09-11T01:31:51.577 INFO:tasks.rados.rados.0.smithi071.stdout:417: expect (ObjNum 130 snap 30 seq_num 130) 2024-09-11T01:31:51.580 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1326845 2024-09-11T01:31:51.593 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 73713 2024-09-11T01:31:51.617 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 530845 2024-09-11T01:31:51.637 INFO:tasks.rados.rados.0.smithi071.stdout:415: expect (ObjNum 161 snap 36 seq_num 161) 2024-09-11T01:31:51.664 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2397183 2024-09-11T01:31:51.679 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:31:51.735 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 698213 2024-09-11T01:31:51.779 INFO:tasks.rados.rados.0.smithi071.stdout:406: expect (ObjNum 152 snap 33 seq_num 152) 2024-09-11T01:31:51.780 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:31:51.922 INFO:tasks.rados.rados.0.smithi071.stdout:407: finishing copy_from racing read to smithi07134777-13 2024-09-11T01:31:51.922 INFO:tasks.rados.rados.0.smithi071.stdout:409: finishing copy_from racing read to smithi07134777-3 2024-09-11T01:31:51.922 INFO:tasks.rados.rados.0.smithi071.stdout:407: finishing copy_from to smithi07134777-13 2024-09-11T01:31:51.922 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 108 (ObjNum 136 snap 31 seq_num 136) dirty exists 2024-09-11T01:31:51.923 INFO:tasks.rados.rados.0.smithi071.stdout:409: finishing copy_from to smithi07134777-3 2024-09-11T01:31:51.923 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 30 (ObjNum 54 snap 3 seq_num 54) dirty exists 2024-09-11T01:31:51.923 INFO:tasks.rados.rados.0.smithi071.stdout:413: finishing copy_from racing read to smithi07134777-33 2024-09-11T01:31:51.923 INFO:tasks.rados.rados.0.smithi071.stdout:413: finishing copy_from to smithi07134777-33 2024-09-11T01:31:51.923 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 66 (ObjNum 159 snap 35 seq_num 159) dirty exists 2024-09-11T01:31:52.072 INFO:tasks.rados.rados.0.smithi071.stdout:404: done (8 left) 2024-09-11T01:31:52.072 INFO:tasks.rados.rados.0.smithi071.stdout:406: done (7 left) 2024-09-11T01:31:52.072 INFO:tasks.rados.rados.0.smithi071.stdout:407: done (6 left) 2024-09-11T01:31:52.072 INFO:tasks.rados.rados.0.smithi071.stdout:409: done (5 left) 2024-09-11T01:31:52.072 INFO:tasks.rados.rados.0.smithi071.stdout:411: done (4 left) 2024-09-11T01:31:52.072 INFO:tasks.rados.rados.0.smithi071.stdout:413: done (3 left) 2024-09-11T01:31:52.072 INFO:tasks.rados.rados.0.smithi071.stdout:415: done (2 left) 2024-09-11T01:31:52.072 INFO:tasks.rados.rados.0.smithi071.stdout:417: done (1 left) 2024-09-11T01:31:52.072 INFO:tasks.rados.rados.0.smithi071.stdout:418: done (0 left) 2024-09-11T01:31:52.072 INFO:tasks.rados.rados.0.smithi071.stdout:419: snap_remove snap 36 2024-09-11T01:31:53.316 INFO:tasks.rados.rados.0.smithi071.stdout:419: done (0 left) 2024-09-11T01:31:53.316 INFO:tasks.rados.rados.0.smithi071.stdout:420: read oid 33 snap -1 2024-09-11T01:31:53.317 INFO:tasks.rados.rados.0.smithi071.stdout:420: expect (ObjNum 159 snap 35 seq_num 159) 2024-09-11T01:31:53.317 INFO:tasks.rados.rados.0.smithi071.stdout:421: copy_from oid 44 from oid 27 current snap is 37 2024-09-11T01:31:53.317 INFO:tasks.rados.rados.0.smithi071.stdout:422: read oid 43 snap -1 2024-09-11T01:31:53.317 INFO:tasks.rados.rados.0.smithi071.stdout:422: expect (ObjNum 164 snap 36 seq_num 164) 2024-09-11T01:31:53.317 INFO:tasks.rados.rados.0.smithi071.stdout:423: write oid 37 current snap is 37 2024-09-11T01:31:53.317 INFO:tasks.rados.rados.0.smithi071.stdout:423: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2024-09-11T01:31:53.322 INFO:tasks.rados.rados.0.smithi071.stdout:423: writing smithi07134777-37 from 457190 to 1009286 tid 1 2024-09-11T01:31:53.329 INFO:tasks.rados.rados.0.smithi071.stdout:423: writing smithi07134777-37 from 1614880 to 2195921 tid 2 2024-09-11T01:31:53.334 INFO:tasks.rados.rados.0.smithi071.stdout:423: writing smithi07134777-37 from 2785791 to 2857190 tid 3 2024-09-11T01:31:53.334 INFO:tasks.rados.rados.0.smithi071.stdout:424: write (excl) oid 50 current snap is 37 2024-09-11T01:31:53.334 INFO:tasks.rados.rados.0.smithi071.stdout:424: seq_num 171 ranges {473997=627345,1673362=441052,2529567=344430} 2024-09-11T01:31:53.339 INFO:tasks.rados.rados.0.smithi071.stdout:424: writing smithi07134777-50 from 473997 to 1101342 tid 1 2024-09-11T01:31:53.345 INFO:tasks.rados.rados.0.smithi071.stdout:424: writing smithi07134777-50 from 1673362 to 2114414 tid 2 2024-09-11T01:31:53.350 INFO:tasks.rados.rados.0.smithi071.stdout:424: writing smithi07134777-50 from 2529567 to 2873997 tid 3 2024-09-11T01:31:53.350 INFO:tasks.rados.rados.0.smithi071.stdout:425: read oid 26 snap -1 2024-09-11T01:31:53.350 INFO:tasks.rados.rados.0.smithi071.stdout:425: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:31:53.350 INFO:tasks.rados.rados.0.smithi071.stdout:426: read oid 28 snap -1 2024-09-11T01:31:53.350 INFO:tasks.rados.rados.0.smithi071.stdout:426: expect (ObjNum 121 snap 29 seq_num 121) 2024-09-11T01:31:53.350 INFO:tasks.rados.rados.0.smithi071.stdout:427: write (excl) oid 9 current snap is 37 2024-09-11T01:31:53.350 INFO:tasks.rados.rados.0.smithi071.stdout:427: seq_num 172 ranges {490804=702594,1615491=584710,2724284=166520} 2024-09-11T01:31:53.357 INFO:tasks.rados.rados.0.smithi071.stdout:427: writing smithi07134777-9 from 490804 to 1193398 tid 1 2024-09-11T01:31:53.362 INFO:tasks.rados.rados.0.smithi071.stdout:427: writing smithi07134777-9 from 1615491 to 2200201 tid 2 2024-09-11T01:31:53.366 INFO:tasks.rados.rados.0.smithi071.stdout:427: writing smithi07134777-9 from 2724284 to 2890804 tid 3 2024-09-11T01:31:53.366 INFO:tasks.rados.rados.0.smithi071.stdout:428: read oid 21 snap -1 2024-09-11T01:31:53.366 INFO:tasks.rados.rados.0.smithi071.stdout:428: expect (ObjNum 126 snap 29 seq_num 126) 2024-09-11T01:31:53.367 INFO:tasks.rados.rados.0.smithi071.stdout:429: write (excl) oid 25 current snap is 37 2024-09-11T01:31:53.367 INFO:tasks.rados.rados.0.smithi071.stdout:420: expect (ObjNum 159 snap 35 seq_num 159) 2024-09-11T01:31:53.447 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:31:53.495 INFO:tasks.rados.rados.0.smithi071.stdout:423: finishing write tid 1 to smithi07134777-37 2024-09-11T01:31:53.495 INFO:tasks.rados.rados.0.smithi071.stdout:422: expect (ObjNum 164 snap 36 seq_num 164) 2024-09-11T01:31:53.519 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:31:53.551 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 836348 2024-09-11T01:31:53.587 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:31:53.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.623+0000 7fa4ac89b640 10 monclient: get_monmap_and_config 2024-09-11T01:31:53.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.623+0000 7fa4ac89b640 10 monclient: build_initial_monmap 2024-09-11T01:31:53.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.623+0000 7fa4ac89b640 10 monclient: monmap: 2024-09-11T01:31:53.625 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:53.625 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:53.625 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:31:53.624201+0000 2024-09-11T01:31:53.625 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:31:53.624201+0000 2024-09-11T01:31:53.625 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:53.625 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:53.625 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:31:53.625 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:31:53.625 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:31:53.626 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:31:53.626 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:31:53.626 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:53.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa4ac89b640 1 Processor -- start 2024-09-11T01:31:53.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa4ac89b640 1 -- start start 2024-09-11T01:31:53.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa4ac89b640 10 monclient: init 2024-09-11T01:31:53.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa4ac89b640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:53.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa4ac89b640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-11T01:31:53.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa4ac89b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa4a80a2e90 0x7fa4a806dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:53.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa4ac89b640 10 monclient(hunting): picked mon.noname-a con 0x7fa4a80a2e90 addr v2:172.21.15.12:3300/0 2024-09-11T01:31:53.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa4ac89b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4a80a3270 0x7fa4a80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:53.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa4ac89b640 10 monclient(hunting): picked mon.noname-b con 0x7fa4a80a3270 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:53.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa4ac89b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa4a8138040 addr v1:172.21.15.12:6789/0 2024-09-11T01:31:53.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa4ac89b640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4a80a46c0 con 0x7fa4a8138040 2024-09-11T01:31:53.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa4ac89b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:53.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa4ac89b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fa4a80a4900 con 0x7fa4a80a2e90 2024-09-11T01:31:53.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa4ac89b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:53.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa4ac89b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa4a806ee50 con 0x7fa4a80a3270 2024-09-11T01:31:53.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa4ac89b640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:53.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa4ac89b640 10 monclient(hunting): authenticate will time out at 920.066774s 2024-09-11T01:31:53.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa4a659c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa4a80a2e90 0x7fa4a806dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:53.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa49ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4a80a3270 0x7fa4a80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:53.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa4a6d9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa4a8138040 0x7fa4a8138490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34332/0 (socket says 172.21.15.12:34332) 2024-09-11T01:31:53.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa4a6d9d640 1 -- 172.21.15.12:0/1367676763 learned_addr learned my addr 172.21.15.12:0/1367676763 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:31:53.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa4a659c640 10 monclient(hunting): get_auth_request con 0x7fa4a80a2e90 auth_method 0 2024-09-11T01:31:53.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa4a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:53.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa4a659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:53.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa4a659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:53.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa49ffff640 10 monclient(hunting): get_auth_request con 0x7fa4a80a3270 auth_method 0 2024-09-11T01:31:53.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa49ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:53.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa49ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:53.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.624+0000 7fa49ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:53.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.625+0000 7fa4a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:53.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.625+0000 7fa4a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:53.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.625+0000 7fa4a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:53.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.625+0000 7fa49ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:53.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.625+0000 7fa49ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:53.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.625+0000 7fa49ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:53.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.625+0000 7fa4a659c640 10 monclient(hunting): handle_auth_done global_id 24198 payload 274 2024-09-11T01:31:53.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.625+0000 7fa4a659c640 1 -- 172.21.15.12:0/1367676763 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4a80a3270 msgr2=0x7fa4a80a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:53.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.625+0000 7fa4a659c640 1 --2- 172.21.15.12:0/1367676763 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4a80a3270 0x7fa4a80a3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:53.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.625+0000 7fa4a659c640 1 -- 172.21.15.12:0/1367676763 >> v1:172.21.15.12:6789/0 conn(0x7fa4a8138040 legacy=0x7fa4a8138490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:53.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.625+0000 7fa4a659c640 10 monclient: _finish_hunting 0 2024-09-11T01:31:53.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.625+0000 7fa4a659c640 1 monclient: found mon.noname-a 2024-09-11T01:31:53.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.625+0000 7fa4a659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:53.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.625+0000 7fa4a659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:31:53.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.625+0000 7fa4a659c640 1 -- 172.21.15.12:0/1367676763 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4a806eb70 con 0x7fa4a80a2e90 2024-09-11T01:31:53.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.625+0000 7fa49ffff640 1 --2- 172.21.15.12:0/1367676763 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4a80a3270 0x7fa4a80a3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:31:53.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4a659c640 1 -- 172.21.15.12:0/1367676763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4a80a2e90 msgr2=0x7fa4a806dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:53.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4a659c640 1 --2- 172.21.15.12:0/1367676763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4a80a2e90 0x7fa4a806dc60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa4980097a0 tx=0x7fa498029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7633ac2e9c42261 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:53.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4a5d9b640 1 -- 172.21.15.12:0/1367676763 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa498037070 con 0x7fa4a80a2e90 2024-09-11T01:31:53.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4a5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:53.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4a5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:31:53.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4a5d9b640 10 monclient: dump: 2024-09-11T01:31:53.637 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:53.637 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:53.637 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:53.637 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:53.637 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:53.637 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:53.637 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:53.637 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:53.637 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:53.637 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:53.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4a5d9b640 10 monclient: _finish_auth 0 2024-09-11T01:31:53.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4a5d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:31:53.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4a5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:53.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4a5d9b640 1 -- 172.21.15.12:0/1367676763 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa498032440 con 0x7fa4a80a2e90 2024-09-11T01:31:53.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4ac89b640 5 monclient: authenticate success, global_id 24198 2024-09-11T01:31:53.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4ac89b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:31:53.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4a5d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:53.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4ac89b640 10 monclient: get_monmap_and_config success 2024-09-11T01:31:53.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4ac89b640 4 set_mon_vals no callback set 2024-09-11T01:31:53.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4a5d9b640 1 -- 172.21.15.12:0/1367676763 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa498032810 con 0x7fa4a80a2e90 2024-09-11T01:31:53.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4a5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:53.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4a5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:53.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4a5d9b640 10 monclient: dump: 2024-09-11T01:31:53.638 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:53.638 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:53.639 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:53.639 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:53.639 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:53.639 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:53.639 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:53.639 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:53.639 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:53.639 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:53.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4ac89b640 10 monclient: shutdown 2024-09-11T01:31:53.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4ac89b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:53.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4ac89b640 1 -- 172.21.15.12:0/1367676763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4a80a2e90 msgr2=0x7fa4a806dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:53.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4ac89b640 1 --2- 172.21.15.12:0/1367676763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4a80a2e90 0x7fa4a806dc60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa4980097a0 tx=0x7fa498029fd0 comp rx=0 tx=0).stop 2024-09-11T01:31:53.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4ac89b640 1 -- 172.21.15.12:0/1367676763 shutdown_connections 2024-09-11T01:31:53.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4ac89b640 1 --2- 172.21.15.12:0/1367676763 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4a80a3270 0x7fa4a80a3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:53.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4ac89b640 1 --2- 172.21.15.12:0/1367676763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4a80a2e90 0x7fa4a806dc60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:53.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4ac89b640 1 -- 172.21.15.12:0/1367676763 >> 172.21.15.12:0/1367676763 conn(0x7fa4a80a8490 msgr2=0x7fa4a80a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:53.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4ac89b640 1 -- 172.21.15.12:0/1367676763 shutdown_connections 2024-09-11T01:31:53.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.628+0000 7fa4ac89b640 1 -- 172.21.15.12:0/1367676763 wait complete. 2024-09-11T01:31:53.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.629+0000 7fa4ac89b640 10 monclient: build_initial_monmap 2024-09-11T01:31:53.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.629+0000 7fa4ac89b640 10 monclient: monmap: 2024-09-11T01:31:53.640 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:53.640 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:31:53.640 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:31:53.640 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:31:53.640 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:53.640 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:53.640 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:31:53.641 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:31:53.641 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:31:53.641 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:53.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.629+0000 7fa4ac89b640 1 Processor -- start 2024-09-11T01:31:53.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4ac89b640 1 -- start start 2024-09-11T01:31:53.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4ac89b640 10 monclient: init 2024-09-11T01:31:53.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4ac89b640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:53.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4ac89b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:31:53.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4ac89b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4a80a2e90 0x7fa4a81cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:53.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4ac89b640 10 monclient(hunting): picked mon.noname-c con 0x7fa4a80a2e90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:53.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4ac89b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4a80a3270 0x7fa4a81cda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:53.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4ac89b640 10 monclient(hunting): picked mon.noname-a con 0x7fa4a80a3270 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:53.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4ac89b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4a8138040 0x7fa4a8124910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:53.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4ac89b640 10 monclient(hunting): picked mon.noname-b con 0x7fa4a8138040 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:31:53.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4ac89b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:53.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4ac89b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a806fba0 con 0x7fa4a80a3270 2024-09-11T01:31:53.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4ac89b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:53.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4ac89b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a80a3bc0 con 0x7fa4a8138040 2024-09-11T01:31:53.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4ac89b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:53.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4ac89b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa4a806fd20 con 0x7fa4a80a2e90 2024-09-11T01:31:53.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4ac89b640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:53.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4ac89b640 10 monclient(hunting): authenticate will time out at 920.072526s 2024-09-11T01:31:53.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4a80a2e90 0x7fa4a81cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:53.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa49ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4a80a3270 0x7fa4a81cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:53.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4a80a2e90 0x7fa4a81cd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41298/0 (socket says 172.21.15.12:41298) 2024-09-11T01:31:53.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a659c640 1 -- 172.21.15.12:0/1833127606 learned_addr learned my addr 172.21.15.12:0/1833127606 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:53.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa49ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4a80a3270 0x7fa4a81cda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40792/0 (socket says 172.21.15.12:40792) 2024-09-11T01:31:53.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a659c640 10 monclient(hunting): get_auth_request con 0x7fa4a80a2e90 auth_method 0 2024-09-11T01:31:53.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:53.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:53.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:53.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa49ffff640 10 monclient(hunting): get_auth_request con 0x7fa4a80a3270 auth_method 0 2024-09-11T01:31:53.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa49ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:53.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa49ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:53.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa49ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:53.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:53.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:53.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:53.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa49ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:53.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa49ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:53.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa49ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:53.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a6d9d640 1 --2- 172.21.15.12:0/1833127606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4a8138040 0x7fa4a8124910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:53.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a6d9d640 10 monclient(hunting): get_auth_request con 0x7fa4a8138040 auth_method 0 2024-09-11T01:31:53.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:53.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:53.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:53.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a6d9d640 1 -- 172.21.15.12:0/1833127606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4a8138040 msgr2=0x7fa4a8124910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:31:53.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a6d9d640 1 -- 172.21.15.12:0/1833127606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4a8138040 msgr2=0x7fa4a8124910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:31:53.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a659c640 10 monclient(hunting): handle_auth_done global_id 4631 payload 824 2024-09-11T01:31:53.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a6d9d640 1 --2- 172.21.15.12:0/1833127606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4a8138040 0x7fa4a8124910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:31:53.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a659c640 1 -- 172.21.15.12:0/1833127606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4a8138040 msgr2=0x7fa4a8124910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:31:53.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a6d9d640 1 --2- 172.21.15.12:0/1833127606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4a8138040 0x7fa4a8124910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:31:53.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a659c640 1 --2- 172.21.15.12:0/1833127606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4a8138040 0x7fa4a8124910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:53.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a659c640 1 -- 172.21.15.12:0/1833127606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4a80a3270 msgr2=0x7fa4a81cda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:53.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a659c640 1 --2- 172.21.15.12:0/1833127606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4a80a3270 0x7fa4a81cda20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:53.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a659c640 10 monclient: _finish_hunting 0 2024-09-11T01:31:53.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a659c640 1 monclient: found mon.noname-c 2024-09-11T01:31:53.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:53.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:31:53.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa4a659c640 1 -- 172.21.15.12:0/1833127606 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4a8137ca0 con 0x7fa4a80a2e90 2024-09-11T01:31:53.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.630+0000 7fa49ffff640 1 --2- 172.21.15.12:0/1833127606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4a80a3270 0x7fa4a81cda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:31:53.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.633+0000 7fa4a659c640 1 -- 172.21.15.12:0/1833127606 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4a80a2e90 msgr2=0x7fa4a81cd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:53.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.633+0000 7fa4a659c640 1 --2- 172.21.15.12:0/1833127606 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4a80a2e90 0x7fa4a81cd4e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa498000c00 tx=0x7fa4980046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:53.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.633+0000 7fa49effd640 1 -- 172.21.15.12:0/1833127606 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa498037040 con 0x7fa4a80a2e90 2024-09-11T01:31:53.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.633+0000 7fa49effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:53.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.633+0000 7fa49effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:31:53.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.633+0000 7fa49effd640 10 monclient: dump: 2024-09-11T01:31:53.646 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:53.646 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:53.646 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:53.646 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:53.646 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:53.646 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:53.646 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:53.646 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:53.646 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:53.646 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:53.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.633+0000 7fa49effd640 10 monclient: _finish_auth 0 2024-09-11T01:31:53.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.633+0000 7fa49effd640 10 monclient: _check_auth_tickets 2024-09-11T01:31:53.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.633+0000 7fa49effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:53.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.633+0000 7fa49effd640 1 -- 172.21.15.12:0/1833127606 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa498042070 con 0x7fa4a80a2e90 2024-09-11T01:31:53.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.633+0000 7fa49effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:53.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.633+0000 7fa49effd640 1 -- 172.21.15.12:0/1833127606 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49803bb00 con 0x7fa4a80a2e90 2024-09-11T01:31:53.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.633+0000 7fa4ac89b640 5 monclient: authenticate success, global_id 4631 2024-09-11T01:31:53.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.633+0000 7fa4ac89b640 10 monclient: _renew_subs 2024-09-11T01:31:53.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.633+0000 7fa4ac89b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:53.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.633+0000 7fa4ac89b640 1 -- 172.21.15.12:0/1833127606 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4a81286e0 con 0x7fa4a80a2e90 2024-09-11T01:31:53.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.633+0000 7fa4a559a640 4 set_mon_vals no callback set 2024-09-11T01:31:53.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.633+0000 7fa49effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:53.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.633+0000 7fa49effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:53.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.633+0000 7fa49effd640 10 monclient: dump: 2024-09-11T01:31:53.648 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:53.648 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:53.648 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:53.648 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:53.648 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:53.648 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:53.648 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:53.648 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:53.648 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:53.648 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:53.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.633+0000 7fa4ac89b640 10 monclient: _renew_subs 2024-09-11T01:31:53.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.633+0000 7fa4ac89b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:53.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.633+0000 7fa4ac89b640 1 -- 172.21.15.12:0/1833127606 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4a80a7b70 con 0x7fa4a80a2e90 2024-09-11T01:31:53.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.634+0000 7fa4a659c640 1 --2- 172.21.15.12:0/1833127606 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4a80a2e90 0x7fa4a81cd4e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa498000c00 tx=0x7fa4980046c0 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7fa498073290 osd_map(97..97 src has 1..97) 2024-09-11T01:31:53.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.634+0000 7fa49effd640 1 -- 172.21.15.12:0/1833127606 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fa49803a030 con 0x7fa4a80a2e90 2024-09-11T01:31:53.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.634+0000 7fa4ac89b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:53.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.634+0000 7fa4ac89b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:53.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.634+0000 7fa4ac89b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:53.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.634+0000 7fa4ac89b640 1 -- 172.21.15.12:0/1833127606 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4a8138c30 con 0x7fa4a80a2e90 2024-09-11T01:31:53.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.634+0000 7fa49effd640 1 --2- 172.21.15.12:0/1833127606 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa48803e210 0x7fa4880406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:53.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.634+0000 7fa49ffff640 1 --2- 172.21.15.12:0/1833127606 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa48803e210 0x7fa4880406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:53.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.634+0000 7fa49ffff640 10 monclient: get_auth_request con 0x7fa48803e210 auth_method 0 2024-09-11T01:31:53.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.637+0000 7fa49ffff640 1 --2- 172.21.15.12:0/1833127606 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa48803e210 0x7fa4880406d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8127a50 tx=0x7fa490009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:53.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:53.639+0000 7fa49effd640 1 -- 172.21.15.12:0/1833127606 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(97..97 src has 1..97) ==== 4767+0+0 (secure 0 0 0) 0x7fa498073290 con 0x7fa4a80a2e90 2024-09-11T01:31:53.650 INFO:tasks.rados.rados.0.smithi071.stdout:421: finishing copy_from racing read to smithi07134777-44 2024-09-11T01:31:53.650 INFO:tasks.rados.rados.0.smithi071.stdout:423: finishing write tid 2 to smithi07134777-37 2024-09-11T01:31:53.650 INFO:tasks.rados.rados.0.smithi071.stdout:421: finishing copy_from to smithi07134777-44 2024-09-11T01:31:53.650 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 30 (ObjNum 54 snap 3 seq_num 54) dirty exists 2024-09-11T01:31:53.651 INFO:tasks.rados.rados.0.smithi071.stdout:423: finishing write tid 3 to smithi07134777-37 2024-09-11T01:31:53.651 INFO:tasks.rados.rados.0.smithi071.stdout:423: finishing write tid 4 to smithi07134777-37 2024-09-11T01:31:53.651 INFO:tasks.rados.rados.0.smithi071.stdout:423: finishing write tid 5 to smithi07134777-37 2024-09-11T01:31:53.651 INFO:tasks.rados.rados.0.smithi071.stdout:423: oid 37 updating version 0 to 139 2024-09-11T01:31:53.651 INFO:tasks.rados.rados.0.smithi071.stdout:423: oid 37 updating version 139 to 140 2024-09-11T01:31:53.651 INFO:tasks.rados.rados.0.smithi071.stdout:423: oid 37 updating version 140 to 141 2024-09-11T01:31:53.651 INFO:tasks.rados.rados.0.smithi071.stdout:423: oid 37 updating version 141 to 142 2024-09-11T01:31:53.651 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 142 (ObjNum 170 snap 37 seq_num 170) dirty exists 2024-09-11T01:31:53.651 INFO:tasks.rados.rados.0.smithi071.stdout:423: left oid 37 (ObjNum 170 snap 37 seq_num 170) 2024-09-11T01:31:53.651 INFO:tasks.rados.rados.0.smithi071.stdout:424: finishing write tid 1 to smithi07134777-50 2024-09-11T01:31:53.651 INFO:tasks.rados.rados.0.smithi071.stdout:429: seq_num 173 ranges {507611=777843,2073973=444721} 2024-09-11T01:31:53.651 INFO:tasks.rados.rados.0.smithi071.stdout:429: writing smithi07134777-25 from 507611 to 1285454 tid 1 2024-09-11T01:31:53.655 INFO:tasks.rados.rados.0.smithi071.stdout:429: writing smithi07134777-25 from 2073973 to 2518694 tid 2 2024-09-11T01:31:53.655 INFO:tasks.rados.rados.0.smithi071.stdout:420: done (9 left) 2024-09-11T01:31:53.656 INFO:tasks.rados.rados.0.smithi071.stdout:421: done (8 left) 2024-09-11T01:31:53.656 INFO:tasks.rados.rados.0.smithi071.stdout:422: done (7 left) 2024-09-11T01:31:53.656 INFO:tasks.rados.rados.0.smithi071.stdout:423: done (6 left) 2024-09-11T01:31:53.656 INFO:tasks.rados.rados.0.smithi071.stdout:430: copy_from oid 44 from oid 17 current snap is 37 2024-09-11T01:31:53.656 INFO:tasks.rados.rados.0.smithi071.stdout:424: finishing write tid 2 to smithi07134777-50 2024-09-11T01:31:53.656 INFO:tasks.rados.rados.0.smithi071.stdout:424: finishing write tid 3 to smithi07134777-50 2024-09-11T01:31:53.656 INFO:tasks.rados.rados.0.smithi071.stdout:424: finishing write tid 4 to smithi07134777-50 2024-09-11T01:31:53.656 INFO:tasks.rados.rados.0.smithi071.stdout:424: finishing write tid 5 to smithi07134777-50 2024-09-11T01:31:53.656 INFO:tasks.rados.rados.0.smithi071.stdout:424: oid 50 updating version 0 to 206 2024-09-11T01:31:53.656 INFO:tasks.rados.rados.0.smithi071.stdout:424: oid 50 version 206 is already newer than 205 2024-09-11T01:31:53.656 INFO:tasks.rados.rados.0.smithi071.stdout:424: oid 50 version 206 is already newer than 204 2024-09-11T01:31:53.656 INFO:tasks.rados.rados.0.smithi071.stdout:424: oid 50 version 206 is already newer than 203 2024-09-11T01:31:53.656 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 206 (ObjNum 171 snap 37 seq_num 171) dirty exists 2024-09-11T01:31:53.657 INFO:tasks.rados.rados.0.smithi071.stdout:424: left oid 50 (ObjNum 171 snap 37 seq_num 171) 2024-09-11T01:31:53.657 INFO:tasks.rados.rados.0.smithi071.stdout:426: expect (ObjNum 121 snap 29 seq_num 121) 2024-09-11T01:31:53.701 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1007850 2024-09-11T01:31:53.714 INFO:tasks.rados.rados.0.smithi071.stdout:425: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:31:53.754 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2159017 2024-09-11T01:31:53.788 INFO:tasks.rados.rados.0.smithi071.stdout:427: finishing write tid 1 to smithi07134777-9 2024-09-11T01:31:53.789 INFO:tasks.rados.rados.0.smithi071.stdout:427: finishing write tid 2 to smithi07134777-9 2024-09-11T01:31:53.789 INFO:tasks.rados.rados.0.smithi071.stdout:427: finishing write tid 3 to smithi07134777-9 2024-09-11T01:31:53.789 INFO:tasks.rados.rados.0.smithi071.stdout:427: finishing write tid 4 to smithi07134777-9 2024-09-11T01:31:53.789 INFO:tasks.rados.rados.0.smithi071.stdout:427: finishing write tid 5 to smithi07134777-9 2024-09-11T01:31:53.789 INFO:tasks.rados.rados.0.smithi071.stdout:427: oid 9 updating version 0 to 24 2024-09-11T01:31:53.789 INFO:tasks.rados.rados.0.smithi071.stdout:427: oid 9 version 24 is already newer than 23 2024-09-11T01:31:53.789 INFO:tasks.rados.rados.0.smithi071.stdout:427: oid 9 updating version 24 to 25 2024-09-11T01:31:53.789 INFO:tasks.rados.rados.0.smithi071.stdout:427: oid 9 version 25 is already newer than 22 2024-09-11T01:31:53.789 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 25 (ObjNum 172 snap 37 seq_num 172) dirty exists 2024-09-11T01:31:53.789 INFO:tasks.rados.rados.0.smithi071.stdout:427: left oid 9 (ObjNum 172 snap 37 seq_num 172) 2024-09-11T01:31:53.789 INFO:tasks.rados.rados.0.smithi071.stdout:428: expect (ObjNum 126 snap 29 seq_num 126) 2024-09-11T01:31:53.873 INFO:tasks.rados.rados.0.smithi071.stdout:429: finishing write tid 1 to smithi07134777-25 2024-09-11T01:31:53.874 INFO:tasks.rados.rados.0.smithi071.stdout:429: finishing write tid 2 to smithi07134777-25 2024-09-11T01:31:53.874 INFO:tasks.rados.rados.0.smithi071.stdout:429: finishing write tid 3 to smithi07134777-25 2024-09-11T01:31:53.874 INFO:tasks.rados.rados.0.smithi071.stdout:429: finishing write tid 4 to smithi07134777-25 2024-09-11T01:31:53.874 INFO:tasks.rados.rados.0.smithi071.stdout:429: oid 25 updating version 0 to 28 2024-09-11T01:31:53.874 INFO:tasks.rados.rados.0.smithi071.stdout:429: oid 25 version 28 is already newer than 26 2024-09-11T01:31:53.874 INFO:tasks.rados.rados.0.smithi071.stdout:429: oid 25 version 28 is already newer than 27 2024-09-11T01:31:53.874 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 28 (ObjNum 173 snap 37 seq_num 173) dirty exists 2024-09-11T01:31:53.874 INFO:tasks.rados.rados.0.smithi071.stdout:429: left oid 25 (ObjNum 173 snap 37 seq_num 173) 2024-09-11T01:31:53.874 INFO:tasks.rados.rados.0.smithi071.stdout:430: finishing copy_from racing read to smithi07134777-44 2024-09-11T01:31:53.874 INFO:tasks.rados.rados.0.smithi071.stdout:430: finishing copy_from to smithi07134777-44 2024-09-11T01:31:53.874 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-11T01:31:53.874 INFO:tasks.rados.rados.0.smithi071.stdout:424: done (6 left) 2024-09-11T01:31:53.874 INFO:tasks.rados.rados.0.smithi071.stdout:425: done (5 left) 2024-09-11T01:31:53.875 INFO:tasks.rados.rados.0.smithi071.stdout:426: done (4 left) 2024-09-11T01:31:53.875 INFO:tasks.rados.rados.0.smithi071.stdout:427: done (3 left) 2024-09-11T01:31:53.875 INFO:tasks.rados.rados.0.smithi071.stdout:428: done (2 left) 2024-09-11T01:31:53.875 INFO:tasks.rados.rados.0.smithi071.stdout:429: done (1 left) 2024-09-11T01:31:53.875 INFO:tasks.rados.rados.0.smithi071.stdout:430: done (0 left) 2024-09-11T01:31:53.875 INFO:tasks.rados.rados.0.smithi071.stdout:431: write oid 44 current snap is 37 2024-09-11T01:31:53.875 INFO:tasks.rados.rados.0.smithi071.stdout:431: seq_num 174 ranges {524418=453092,1732455=588379,2779130=145288} 2024-09-11T01:31:53.878 INFO:tasks.rados.rados.0.smithi071.stdout:431: writing smithi07134777-44 from 524418 to 977510 tid 1 2024-09-11T01:31:53.885 INFO:tasks.rados.rados.0.smithi071.stdout:431: writing smithi07134777-44 from 1732455 to 2320834 tid 2 2024-09-11T01:31:53.889 INFO:tasks.rados.rados.0.smithi071.stdout:431: writing smithi07134777-44 from 2779130 to 2924418 tid 3 2024-09-11T01:31:53.889 INFO:tasks.rados.rados.0.smithi071.stdout:432: read oid 1 snap -1 2024-09-11T01:31:53.889 INFO:tasks.rados.rados.0.smithi071.stdout:432: expect (ObjNum 92 snap 19 seq_num 92) 2024-09-11T01:31:53.890 INFO:tasks.rados.rados.0.smithi071.stdout:433: read oid 11 snap -1 2024-09-11T01:31:53.890 INFO:tasks.rados.rados.0.smithi071.stdout:433: expect (ObjNum 119 snap 28 seq_num 119) 2024-09-11T01:31:53.890 INFO:tasks.rados.rados.0.smithi071.stdout:434: delete oid 41 current snap is 37 2024-09-11T01:31:53.890 INFO:tasks.rados.rados.0.smithi071.stdout:431: finishing write tid 1 to smithi07134777-44 2024-09-11T01:31:53.890 INFO:tasks.rados.rados.0.smithi071.stdout:431: finishing write tid 2 to smithi07134777-44 2024-09-11T01:31:53.890 INFO:tasks.rados.rados.0.smithi071.stdout:431: finishing write tid 3 to smithi07134777-44 2024-09-11T01:31:53.892 INFO:tasks.rados.rados.0.smithi071.stdout:434: done (3 left) 2024-09-11T01:31:53.893 INFO:tasks.rados.rados.0.smithi071.stdout:435: write oid 24 current snap is 37 2024-09-11T01:31:53.893 INFO:tasks.rados.rados.0.smithi071.stdout:435: seq_num 176 ranges {558032=719943,1849419=592048} 2024-09-11T01:31:53.899 INFO:tasks.rados.rados.0.smithi071.stdout:435: writing smithi07134777-24 from 558032 to 1277975 tid 1 2024-09-11T01:31:53.907 INFO:tasks.rados.rados.0.smithi071.stdout:435: writing smithi07134777-24 from 1849419 to 2441467 tid 2 2024-09-11T01:31:53.907 INFO:tasks.rados.rados.0.smithi071.stdout:436: read oid 3 snap -1 2024-09-11T01:31:53.907 INFO:tasks.rados.rados.0.smithi071.stdout:436: expect (ObjNum 54 snap 3 seq_num 54) 2024-09-11T01:31:53.907 INFO:tasks.rados.rados.0.smithi071.stdout:437: read oid 5 snap -1 2024-09-11T01:31:53.907 INFO:tasks.rados.rados.0.smithi071.stdout:437: expect deleted 2024-09-11T01:31:53.907 INFO:tasks.rados.rados.0.smithi071.stdout:438: delete oid 39 current snap is 37 2024-09-11T01:31:53.907 INFO:tasks.rados.rados.0.smithi071.stdout:432: expect (ObjNum 92 snap 19 seq_num 92) 2024-09-11T01:31:54.014 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1020655 2024-09-11T01:31:54.053 INFO:tasks.rados.rados.0.smithi071.stdout:433: expect (ObjNum 119 snap 28 seq_num 119) 2024-09-11T01:31:54.117 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 688302 2024-09-11T01:31:54.145 INFO:tasks.rados.rados.0.smithi071.stdout:431: finishing write tid 4 to smithi07134777-44 2024-09-11T01:31:54.145 INFO:tasks.rados.rados.0.smithi071.stdout:431: finishing write tid 5 to smithi07134777-44 2024-09-11T01:31:54.145 INFO:tasks.rados.rados.0.smithi071.stdout:431: oid 44 updating version 0 to 74 2024-09-11T01:31:54.145 INFO:tasks.rados.rados.0.smithi071.stdout:431: oid 44 version 74 is already newer than 73 2024-09-11T01:31:54.145 INFO:tasks.rados.rados.0.smithi071.stdout:431: oid 44 version 74 is already newer than 71 2024-09-11T01:31:54.145 INFO:tasks.rados.rados.0.smithi071.stdout:431: oid 44 version 74 is already newer than 72 2024-09-11T01:31:54.145 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 74 (ObjNum 174 snap 37 seq_num 174) dirty exists 2024-09-11T01:31:54.145 INFO:tasks.rados.rados.0.smithi071.stdout:431: left oid 44 (ObjNum 174 snap 37 seq_num 174) 2024-09-11T01:31:54.145 INFO:tasks.rados.rados.0.smithi071.stdout:435: finishing write tid 1 to smithi07134777-24 2024-09-11T01:31:54.146 INFO:tasks.rados.rados.0.smithi071.stdout:435: finishing write tid 2 to smithi07134777-24 2024-09-11T01:31:54.146 INFO:tasks.rados.rados.0.smithi071.stdout:431: done (6 left) 2024-09-11T01:31:54.146 INFO:tasks.rados.rados.0.smithi071.stdout:432: done (5 left) 2024-09-11T01:31:54.146 INFO:tasks.rados.rados.0.smithi071.stdout:433: done (4 left) 2024-09-11T01:31:54.146 INFO:tasks.rados.rados.0.smithi071.stdout:437: done (3 left) 2024-09-11T01:31:54.146 INFO:tasks.rados.rados.0.smithi071.stdout:438: done (2 left) 2024-09-11T01:31:54.146 INFO:tasks.rados.rados.0.smithi071.stdout:439: rollback oid 30 current snap is 37 2024-09-11T01:31:54.146 INFO:tasks.rados.rados.0.smithi071.stdout:439: done (2 left) 2024-09-11T01:31:54.146 INFO:tasks.rados.rados.0.smithi071.stdout:440: snap_create 2024-09-11T01:31:54.146 INFO:tasks.rados.rados.0.smithi071.stdout:435: finishing write tid 3 to smithi07134777-24 2024-09-11T01:31:54.146 INFO:tasks.rados.rados.0.smithi071.stdout:435: finishing write tid 4 to smithi07134777-24 2024-09-11T01:31:54.146 INFO:tasks.rados.rados.0.smithi071.stdout:435: oid 24 updating version 0 to 162 2024-09-11T01:31:54.146 INFO:tasks.rados.rados.0.smithi071.stdout:435: oid 24 updating version 162 to 163 2024-09-11T01:31:54.146 INFO:tasks.rados.rados.0.smithi071.stdout:435: oid 24 version 163 is already newer than 161 2024-09-11T01:31:54.147 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 163 (ObjNum 176 snap 37 seq_num 176) dirty exists 2024-09-11T01:31:54.147 INFO:tasks.rados.rados.0.smithi071.stdout:435: left oid 24 (ObjNum 176 snap 37 seq_num 176) 2024-09-11T01:31:54.147 INFO:tasks.rados.rados.0.smithi071.stdout:436: expect (ObjNum 54 snap 3 seq_num 54) 2024-09-11T01:31:54.150 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 701862 2024-09-11T01:31:54.210 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:31:54.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.462+0000 7fa49effd640 1 -- 172.21.15.12:0/1833127606 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa49803adc0 con 0x7fa4a80a2e90 2024-09-11T01:31:54.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.462+0000 7fa49effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:54.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.462+0000 7fa49effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:31:54.521 INFO:tasks.rados.rados.0.smithi071.stdout:435: done (2 left) 2024-09-11T01:31:54.521 INFO:tasks.rados.rados.0.smithi071.stdout:436: done (1 left) 2024-09-11T01:31:54.521 INFO:tasks.rados.rados.0.smithi071.stdout:440: done (0 left) 2024-09-11T01:31:54.522 INFO:tasks.rados.rados.0.smithi071.stdout:441: write (excl) oid 23 current snap is 38 2024-09-11T01:31:54.522 INFO:tasks.rados.rados.0.smithi071.stdout:441: seq_num 178 ranges {591646=470441,1566383=595717,2888822=102824} 2024-09-11T01:31:54.527 INFO:tasks.rados.rados.0.smithi071.stdout:441: writing smithi07134777-23 from 591646 to 1062087 tid 1 2024-09-11T01:31:54.533 INFO:tasks.rados.rados.0.smithi071.stdout:441: writing smithi07134777-23 from 1566383 to 2162100 tid 2 2024-09-11T01:31:54.538 INFO:tasks.rados.rados.0.smithi071.stdout:441: writing smithi07134777-23 from 2888822 to 2991646 tid 3 2024-09-11T01:31:54.538 INFO:tasks.rados.rados.0.smithi071.stdout:442: write (excl) oid 29 current snap is 38 2024-09-11T01:31:54.538 INFO:tasks.rados.rados.0.smithi071.stdout:442: seq_num 179 ranges {608453=545690,1624865=455728,2632598=375855} 2024-09-11T01:31:54.544 INFO:tasks.rados.rados.0.smithi071.stdout:442: writing smithi07134777-29 from 608453 to 1154143 tid 1 2024-09-11T01:31:54.549 INFO:tasks.rados.rados.0.smithi071.stdout:442: writing smithi07134777-29 from 1624865 to 2080593 tid 2 2024-09-11T01:31:54.555 INFO:tasks.rados.rados.0.smithi071.stdout:442: writing smithi07134777-29 from 2632598 to 3008453 tid 3 2024-09-11T01:31:54.555 INFO:tasks.rados.rados.0.smithi071.stdout:443: copy_from oid 45 from oid 9 current snap is 38 2024-09-11T01:31:54.555 INFO:tasks.rados.rados.0.smithi071.stdout:441: finishing write tid 1 to smithi07134777-23 2024-09-11T01:31:54.555 INFO:tasks.rados.rados.0.smithi071.stdout:441: finishing write tid 2 to smithi07134777-23 2024-09-11T01:31:54.555 INFO:tasks.rados.rados.0.smithi071.stdout:441: finishing write tid 3 to smithi07134777-23 2024-09-11T01:31:54.556 INFO:tasks.rados.rados.0.smithi071.stdout:441: finishing write tid 4 to smithi07134777-23 2024-09-11T01:31:54.556 INFO:tasks.rados.rados.0.smithi071.stdout:441: finishing write tid 5 to smithi07134777-23 2024-09-11T01:31:54.556 INFO:tasks.rados.rados.0.smithi071.stdout:441: oid 23 updating version 0 to 79 2024-09-11T01:31:54.556 INFO:tasks.rados.rados.0.smithi071.stdout:441: oid 23 version 79 is already newer than 78 2024-09-11T01:31:54.556 INFO:tasks.rados.rados.0.smithi071.stdout:441: oid 23 version 79 is already newer than 76 2024-09-11T01:31:54.556 INFO:tasks.rados.rados.0.smithi071.stdout:441: oid 23 version 79 is already newer than 77 2024-09-11T01:31:54.556 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 79 (ObjNum 178 snap 38 seq_num 178) dirty exists 2024-09-11T01:31:54.556 INFO:tasks.rados.rados.0.smithi071.stdout:441: left oid 23 (ObjNum 178 snap 38 seq_num 178) 2024-09-11T01:31:54.556 INFO:tasks.rados.rados.0.smithi071.stdout:442: finishing write tid 1 to smithi07134777-29 2024-09-11T01:31:54.556 INFO:tasks.rados.rados.0.smithi071.stdout:442: finishing write tid 2 to smithi07134777-29 2024-09-11T01:31:54.556 INFO:tasks.rados.rados.0.smithi071.stdout:441: done (2 left) 2024-09-11T01:31:54.556 INFO:tasks.rados.rados.0.smithi071.stdout:444: snap_create 2024-09-11T01:31:54.559 INFO:tasks.rados.rados.0.smithi071.stdout:442: finishing write tid 3 to smithi07134777-29 2024-09-11T01:31:54.559 INFO:tasks.rados.rados.0.smithi071.stdout:442: finishing write tid 4 to smithi07134777-29 2024-09-11T01:31:54.559 INFO:tasks.rados.rados.0.smithi071.stdout:442: finishing write tid 5 to smithi07134777-29 2024-09-11T01:31:54.559 INFO:tasks.rados.rados.0.smithi071.stdout:442: oid 29 updating version 0 to 210 2024-09-11T01:31:54.559 INFO:tasks.rados.rados.0.smithi071.stdout:442: oid 29 version 210 is already newer than 208 2024-09-11T01:31:54.559 INFO:tasks.rados.rados.0.smithi071.stdout:442: oid 29 version 210 is already newer than 209 2024-09-11T01:31:54.559 INFO:tasks.rados.rados.0.smithi071.stdout:442: oid 29 version 210 is already newer than 207 2024-09-11T01:31:54.559 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 210 (ObjNum 179 snap 38 seq_num 179) dirty exists 2024-09-11T01:31:54.560 INFO:tasks.rados.rados.0.smithi071.stdout:442: left oid 29 (ObjNum 179 snap 38 seq_num 179) 2024-09-11T01:31:54.570 INFO:tasks.rados.rados.0.smithi071.stdout:443: finishing copy_from racing read to smithi07134777-45 2024-09-11T01:31:54.571 INFO:tasks.rados.rados.0.smithi071.stdout:443: finishing copy_from to smithi07134777-45 2024-09-11T01:31:54.571 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 25 (ObjNum 172 snap 37 seq_num 172) dirty exists 2024-09-11T01:31:54.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.729+0000 7fa4ac89b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:31:54.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.729+0000 7fa4ac89b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:31:54.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.729+0000 7fa4ac89b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:54.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.729+0000 7fa4ac89b640 1 -- 172.21.15.12:0/1833127606 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa4a80698e0 con 0x7fa4a80a2e90 2024-09-11T01:31:54.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.729+0000 7fa49effd640 1 -- 172.21.15.12:0/1833127606 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fa49803c3a0 con 0x7fa4a80a2e90 2024-09-11T01:31:54.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.729+0000 7fa49effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:31:54.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.729+0000 7fa49effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:31:54.732 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:31:54.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.732+0000 7fa4ac89b640 1 -- 172.21.15.12:0/1833127606 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa48803e210 msgr2=0x7fa4880406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:54.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.732+0000 7fa4ac89b640 1 --2- 172.21.15.12:0/1833127606 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa48803e210 0x7fa4880406d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8127a50 tx=0x7fa490009190 comp rx=0 tx=0).stop 2024-09-11T01:31:54.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.732+0000 7fa4ac89b640 10 monclient: shutdown 2024-09-11T01:31:54.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.732+0000 7fa4ac89b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:54.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.732+0000 7fa4ac89b640 1 -- 172.21.15.12:0/1833127606 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4a80a2e90 msgr2=0x7fa4a81cd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:54.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.732+0000 7fa4ac89b640 1 --2- 172.21.15.12:0/1833127606 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4a80a2e90 0x7fa4a81cd4e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa498000c00 tx=0x7fa4980046c0 comp rx=0 tx=0).stop 2024-09-11T01:31:54.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.732+0000 7fa4ac89b640 1 -- 172.21.15.12:0/1833127606 shutdown_connections 2024-09-11T01:31:54.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.732+0000 7fa4ac89b640 1 --2- 172.21.15.12:0/1833127606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4a8138040 0x7fa4a8124910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:54.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.732+0000 7fa4ac89b640 1 --2- 172.21.15.12:0/1833127606 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa48803e210 0x7fa4880406d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:54.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.732+0000 7fa4ac89b640 1 --2- 172.21.15.12:0/1833127606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4a80a3270 0x7fa4a81cda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:54.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.732+0000 7fa4ac89b640 1 --2- 172.21.15.12:0/1833127606 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4a80a2e90 0x7fa4a81cd4e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:54.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.732+0000 7fa4ac89b640 1 -- 172.21.15.12:0/1833127606 >> 172.21.15.12:0/1833127606 conn(0x7fa4a80a8490 msgr2=0x7fa4a8072090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:54.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.732+0000 7fa4ac89b640 1 -- 172.21.15.12:0/1833127606 shutdown_connections 2024-09-11T01:31:54.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.732+0000 7fa4ac89b640 1 -- 172.21.15.12:0/1833127606 wait complete. 2024-09-11T01:31:54.744 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:31:54.744 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:31:54.744 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:31:54.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.844+0000 7f9e6eb9f640 10 monclient: get_monmap_and_config 2024-09-11T01:31:54.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.844+0000 7f9e6eb9f640 10 monclient: build_initial_monmap 2024-09-11T01:31:54.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.844+0000 7f9e6eb9f640 10 monclient: monmap: 2024-09-11T01:31:54.847 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:54.847 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:54.847 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:31:54.845616+0000 2024-09-11T01:31:54.847 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:31:54.845616+0000 2024-09-11T01:31:54.847 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:54.847 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:54.847 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:31:54.847 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:31:54.847 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:31:54.847 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:31:54.847 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:31:54.847 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:54.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.845+0000 7f9e6eb9f640 1 Processor -- start 2024-09-11T01:31:54.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.845+0000 7f9e6eb9f640 1 -- start start 2024-09-11T01:31:54.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.845+0000 7f9e6eb9f640 10 monclient: init 2024-09-11T01:31:54.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6eb9f640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:54.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6eb9f640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-11T01:31:54.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6eb9f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e680a4dd0 addr v1:172.21.15.12:6789/0 2024-09-11T01:31:54.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6eb9f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9e680a51b0 addr v1:172.21.15.71:6789/0 2024-09-11T01:31:54.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6eb9f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9e6813e1b0 0x7f9e681405a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:54.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6eb9f640 10 monclient(hunting): picked mon.noname-b con 0x7f9e6813e1b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:54.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6eb9f640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e680a6ca0 con 0x7f9e680a4dd0 2024-09-11T01:31:54.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6eb9f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e6809c530 con 0x7f9e680a51b0 2024-09-11T01:31:54.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6eb9f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:54.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6eb9f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f9e680a6770 con 0x7f9e6813e1b0 2024-09-11T01:31:54.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6eb9f640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:54.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6eb9f640 10 monclient(hunting): authenticate will time out at 921.288206s 2024-09-11T01:31:54.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6db9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9e680a4dd0 0x7f9e6809cf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34342/0 (socket says 172.21.15.12:34342) 2024-09-11T01:31:54.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6db9d640 1 -- 172.21.15.12:0/1871431617 learned_addr learned my addr 172.21.15.12:0/1871431617 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:31:54.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6e39e640 1 --2- 172.21.15.12:0/1871431617 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9e6813e1b0 0x7f9e681405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:54.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6e39e640 10 monclient(hunting): get_auth_request con 0x7f9e6813e1b0 auth_method 0 2024-09-11T01:31:54.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6e39e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:54.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6e39e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:54.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6e39e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:54.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6e39e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:54.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6e39e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:54.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6e39e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:54.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6e39e640 10 monclient(hunting): handle_auth_done global_id 4634 payload 274 2024-09-11T01:31:54.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6e39e640 1 -- 172.21.15.12:0/1871431617 >> v1:172.21.15.71:6789/0 conn(0x7f9e680a51b0 legacy=0x7f9e680a5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:54.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6e39e640 1 -- 172.21.15.12:0/1871431617 >> v1:172.21.15.12:6789/0 conn(0x7f9e680a4dd0 legacy=0x7f9e6809cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:54.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6e39e640 10 monclient: _finish_hunting 0 2024-09-11T01:31:54.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6e39e640 1 monclient: found mon.noname-b 2024-09-11T01:31:54.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6e39e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:54.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6e39e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:31:54.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.846+0000 7f9e6e39e640 1 -- 172.21.15.12:0/1871431617 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e6809c250 con 0x7f9e6813e1b0 2024-09-11T01:31:54.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.849+0000 7f9e6e39e640 1 -- 172.21.15.12:0/1871431617 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9e6813e1b0 msgr2=0x7f9e681405a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:54.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.849+0000 7f9e6e39e640 1 --2- 172.21.15.12:0/1871431617 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9e6813e1b0 0x7f9e681405a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9e6809bf70 tx=0x7f9e6002a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=188533793fd8686d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:54.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.849+0000 7f9e6cb9b640 1 -- 172.21.15.12:0/1871431617 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e60037070 con 0x7f9e6813e1b0 2024-09-11T01:31:54.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.849+0000 7f9e6cb9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:54.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.849+0000 7f9e6cb9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:31:54.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.849+0000 7f9e6cb9b640 10 monclient: dump: 2024-09-11T01:31:54.857 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:54.857 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:54.857 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:54.857 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:54.857 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:54.857 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:54.857 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:54.857 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:54.857 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:54.857 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:54.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.849+0000 7f9e6cb9b640 10 monclient: _finish_auth 0 2024-09-11T01:31:54.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.849+0000 7f9e6cb9b640 10 monclient: _check_auth_tickets 2024-09-11T01:31:54.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.849+0000 7f9e6cb9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:54.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.849+0000 7f9e6cb9b640 1 -- 172.21.15.12:0/1871431617 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e60032640 con 0x7f9e6813e1b0 2024-09-11T01:31:54.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.849+0000 7f9e6cb9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:54.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.849+0000 7f9e6cb9b640 1 -- 172.21.15.12:0/1871431617 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e60032a10 con 0x7f9e6813e1b0 2024-09-11T01:31:54.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.849+0000 7f9e6eb9f640 5 monclient: authenticate success, global_id 4634 2024-09-11T01:31:54.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.849+0000 7f9e6eb9f640 10 monclient: get_monmap_and_config success 2024-09-11T01:31:54.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.849+0000 7f9e6eb9f640 4 set_mon_vals no callback set 2024-09-11T01:31:54.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.849+0000 7f9e6cb9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:54.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.849+0000 7f9e6cb9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:54.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.849+0000 7f9e6cb9b640 10 monclient: dump: 2024-09-11T01:31:54.858 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:54.858 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:54.858 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:54.858 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:54.858 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:54.859 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.850+0000 7f9e6eb9f640 10 monclient: shutdown 2024-09-11T01:31:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.850+0000 7f9e6eb9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.850+0000 7f9e6eb9f640 1 -- 172.21.15.12:0/1871431617 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9e6813e1b0 msgr2=0x7f9e681405a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.850+0000 7f9e6eb9f640 1 --2- 172.21.15.12:0/1871431617 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9e6813e1b0 0x7f9e681405a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9e6809bf70 tx=0x7f9e6002a000 comp rx=0 tx=0).stop 2024-09-11T01:31:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.850+0000 7f9e6eb9f640 1 -- 172.21.15.12:0/1871431617 shutdown_connections 2024-09-11T01:31:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.850+0000 7f9e6eb9f640 1 --2- 172.21.15.12:0/1871431617 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9e6813e1b0 0x7f9e681405a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.850+0000 7f9e6eb9f640 1 -- 172.21.15.12:0/1871431617 >> 172.21.15.12:0/1871431617 conn(0x7f9e680aaf90 msgr2=0x7f9e680a6ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.850+0000 7f9e6eb9f640 1 -- 172.21.15.12:0/1871431617 shutdown_connections 2024-09-11T01:31:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.850+0000 7f9e6eb9f640 1 -- 172.21.15.12:0/1871431617 wait complete. 2024-09-11T01:31:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.850+0000 7f9e6eb9f640 10 monclient: build_initial_monmap 2024-09-11T01:31:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.850+0000 7f9e6eb9f640 10 monclient: monmap: 2024-09-11T01:31:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:31:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:31:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:31:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:31:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:31:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:31:54.860 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6eb9f640 1 Processor -- start 2024-09-11T01:31:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6eb9f640 1 -- start start 2024-09-11T01:31:54.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6eb9f640 10 monclient: init 2024-09-11T01:31:54.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6eb9f640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:54.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6eb9f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:31:54.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6eb9f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e680a4dd0 0x7f9e681cd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:54.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6eb9f640 10 monclient(hunting): picked mon.noname-b con 0x7f9e680a4dd0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:31:54.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6eb9f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e680a51b0 0x7f9e68137f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:54.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6eb9f640 10 monclient(hunting): picked mon.noname-a con 0x7f9e680a51b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:54.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6eb9f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9e6813e1b0 0x7f9e6813a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:54.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6eb9f640 10 monclient(hunting): picked mon.noname-c con 0x7f9e6813e1b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:54.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6eb9f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:54.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6eb9f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9e68140c60 con 0x7f9e680a51b0 2024-09-11T01:31:54.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6eb9f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:54.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6eb9f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f9e68140ae0 con 0x7f9e680a4dd0 2024-09-11T01:31:54.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6eb9f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6eb9f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f9e68140de0 con 0x7f9e6813e1b0 2024-09-11T01:31:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6eb9f640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6eb9f640 10 monclient(hunting): authenticate will time out at 921.293846s 2024-09-11T01:31:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6d39c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e680a51b0 0x7f9e68137f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6e39e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9e6813e1b0 0x7f9e6813a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6e39e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9e6813e1b0 0x7f9e6813a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41308/0 (socket says 172.21.15.12:41308) 2024-09-11T01:31:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6e39e640 1 -- 172.21.15.12:0/3570064983 learned_addr learned my addr 172.21.15.12:0/3570064983 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6e39e640 10 monclient(hunting): get_auth_request con 0x7f9e6813e1b0 auth_method 0 2024-09-11T01:31:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6e39e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6e39e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.851+0000 7f9e6e39e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6d39c640 10 monclient(hunting): get_auth_request con 0x7f9e680a51b0 auth_method 0 2024-09-11T01:31:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6d39c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:54.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6d39c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:54.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6d39c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:54.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6db9d640 1 --2- 172.21.15.12:0/3570064983 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e680a4dd0 0x7f9e681cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:54.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6db9d640 10 monclient(hunting): get_auth_request con 0x7f9e680a4dd0 auth_method 0 2024-09-11T01:31:54.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6db9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:54.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6db9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:54.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6db9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:54.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6e39e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:54.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6e39e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:54.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6e39e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:54.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6d39c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:54.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6d39c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:54.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6d39c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:54.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6db9d640 1 -- 172.21.15.12:0/3570064983 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e680a4dd0 msgr2=0x7f9e681cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:31:54.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6db9d640 1 -- 172.21.15.12:0/3570064983 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e680a4dd0 msgr2=0x7f9e681cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:31:54.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6db9d640 1 --2- 172.21.15.12:0/3570064983 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e680a4dd0 0x7f9e681cd160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:31:54.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6db9d640 1 --2- 172.21.15.12:0/3570064983 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e680a4dd0 0x7f9e681cd160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:31:54.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6e39e640 10 monclient(hunting): handle_auth_done global_id 4637 payload 824 2024-09-11T01:31:54.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6e39e640 1 -- 172.21.15.12:0/3570064983 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e680a4dd0 msgr2=0x7f9e681cd160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:31:54.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6e39e640 1 --2- 172.21.15.12:0/3570064983 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e680a4dd0 0x7f9e681cd160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:54.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6e39e640 1 -- 172.21.15.12:0/3570064983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e680a51b0 msgr2=0x7f9e68137f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:54.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6e39e640 1 --2- 172.21.15.12:0/3570064983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e680a51b0 0x7f9e68137f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:54.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6e39e640 10 monclient: _finish_hunting 0 2024-09-11T01:31:54.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6e39e640 1 monclient: found mon.noname-c 2024-09-11T01:31:54.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6e39e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:54.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6e39e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:31:54.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6e39e640 1 -- 172.21.15.12:0/3570064983 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e681215b0 con 0x7f9e6813e1b0 2024-09-11T01:31:54.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.852+0000 7f9e6d39c640 1 --2- 172.21.15.12:0/3570064983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e680a51b0 0x7f9e68137f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:31:54.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.854+0000 7f9e6e39e640 1 -- 172.21.15.12:0/3570064983 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9e6813e1b0 msgr2=0x7f9e6813a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:54.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.854+0000 7f9e6e39e640 1 --2- 172.21.15.12:0/3570064983 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9e6813e1b0 0x7f9e6813a200 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9e60009720 tx=0x7f9e600046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:54.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.854+0000 7f9e667fc640 1 -- 172.21.15.12:0/3570064983 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e60041070 con 0x7f9e6813e1b0 2024-09-11T01:31:54.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.854+0000 7f9e667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:54.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.854+0000 7f9e667fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:31:54.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.854+0000 7f9e667fc640 10 monclient: dump: 2024-09-11T01:31:54.865 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:54.865 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:54.865 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:54.865 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:54.866 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:54.866 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:54.866 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:54.866 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:54.866 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:54.866 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:54.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.854+0000 7f9e667fc640 10 monclient: _finish_auth 0 2024-09-11T01:31:54.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.854+0000 7f9e667fc640 10 monclient: _check_auth_tickets 2024-09-11T01:31:54.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.854+0000 7f9e667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:54.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.854+0000 7f9e667fc640 1 -- 172.21.15.12:0/3570064983 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e6003c880 con 0x7f9e6813e1b0 2024-09-11T01:31:54.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.854+0000 7f9e6eb9f640 5 monclient: authenticate success, global_id 4637 2024-09-11T01:31:54.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.854+0000 7f9e6eb9f640 10 monclient: _renew_subs 2024-09-11T01:31:54.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.854+0000 7f9e6eb9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:54.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.854+0000 7f9e6eb9f640 1 -- 172.21.15.12:0/3570064983 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e68121890 con 0x7f9e6813e1b0 2024-09-11T01:31:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.854+0000 7f9e667fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.854+0000 7f9e667fc640 1 -- 172.21.15.12:0/3570064983 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e60037040 con 0x7f9e6813e1b0 2024-09-11T01:31:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.854+0000 7f9e667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.854+0000 7f9e667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.854+0000 7f9e667fc640 10 monclient: dump: 2024-09-11T01:31:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:54.867 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:54.868 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:54.868 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:54.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.854+0000 7f9e6eb9f640 10 monclient: _renew_subs 2024-09-11T01:31:54.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.854+0000 7f9e6eb9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:54.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.854+0000 7f9e6eb9f640 1 -- 172.21.15.12:0/3570064983 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e68120a20 con 0x7f9e6813e1b0 2024-09-11T01:31:54.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.854+0000 7f9e67fff640 4 set_mon_vals no callback set 2024-09-11T01:31:54.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.855+0000 7f9e6eb9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:54.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.855+0000 7f9e6eb9f640 1 --2- 172.21.15.12:0/3570064983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e68120ff0 0x7f9e6813a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:54.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.855+0000 7f9e6eb9f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:54.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.855+0000 7f9e6eb9f640 1 -- 172.21.15.12:0/3570064983 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6809cf10 con 0x7f9e68120ff0 2024-09-11T01:31:54.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.855+0000 7f9e6db9d640 1 --2- 172.21.15.12:0/3570064983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e68120ff0 0x7f9e6813a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:54.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.855+0000 7f9e6db9d640 10 monclient: get_auth_request con 0x7f9e68120ff0 auth_method 0 2024-09-11T01:31:54.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.855+0000 7f9e6db9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:54.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.855+0000 7f9e6db9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:54.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.855+0000 7f9e6db9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:54.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.855+0000 7f9e667fc640 1 -- 172.21.15.12:0/3570064983 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f9e6003cc70 con 0x7f9e6813e1b0 2024-09-11T01:31:54.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.855+0000 7f9e6db9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:54.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.855+0000 7f9e6db9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:54.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.855+0000 7f9e6db9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:54.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.855+0000 7f9e6db9d640 10 monclient(hunting): handle_auth_done global_id 24207 payload 824 2024-09-11T01:31:54.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.856+0000 7f9e6db9d640 1 -- 172.21.15.12:0/3570064983 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9e6813cce0 con 0x7f9e68120ff0 2024-09-11T01:31:54.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.856+0000 7f9e667fc640 1 --2- 172.21.15.12:0/3570064983 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9e5403e210 0x7f9e540406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:54.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.856+0000 7f9e667fc640 1 -- 172.21.15.12:0/3570064983 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(98..98 src has 1..98) ==== 4739+0+0 (secure 0 0 0) 0x7f9e60073aa0 con 0x7f9e6813e1b0 2024-09-11T01:31:54.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.856+0000 7f9e6d39c640 1 --2- 172.21.15.12:0/3570064983 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9e5403e210 0x7f9e540406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:54.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.856+0000 7f9e6d39c640 10 monclient: get_auth_request con 0x7f9e5403e210 auth_method 0 2024-09-11T01:31:54.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.858+0000 7f9e6db9d640 1 -- 172.21.15.12:0/3570064983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e68120ff0 msgr2=0x7f9e6813a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:54.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.858+0000 7f9e6db9d640 1 --2- 172.21.15.12:0/3570064983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e68120ff0 0x7f9e6813a740 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9e5800eb90 tx=0x7f9e5800c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:54.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.858+0000 7f9e667fc640 1 -- 172.21.15.12:0/3570064983 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e58017020 con 0x7f9e68120ff0 2024-09-11T01:31:54.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.858+0000 7f9e667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:54.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.858+0000 7f9e667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:54.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.858+0000 7f9e667fc640 10 monclient: dump: 2024-09-11T01:31:54.873 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:54.873 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:54.873 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:54.873 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:54.874 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:54.874 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:54.874 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:54.874 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:54.874 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:54.874 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:54.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.858+0000 7f9e667fc640 1 -- 172.21.15.12:0/3570064983 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9e5800ce40 con 0x7f9e68120ff0 2024-09-11T01:31:54.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.858+0000 7f9e667fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:54.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.858+0000 7f9e667fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:31:54.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.858+0000 7f9e667fc640 1 -- 172.21.15.12:0/3570064983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e68120ff0 msgr2=0x7f9e6813a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:54.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.858+0000 7f9e667fc640 1 --2- 172.21.15.12:0/3570064983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e68120ff0 0x7f9e6813a740 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9e5800eb90 tx=0x7f9e5800c4b0 comp rx=0 tx=0).stop 2024-09-11T01:31:54.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.858+0000 7f9e667fc640 1 -- 172.21.15.12:0/3570064983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e68120ff0 msgr2=0x7f9e6813a740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:31:54.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.858+0000 7f9e667fc640 1 --2- 172.21.15.12:0/3570064983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e68120ff0 0x7f9e6813a740 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9e5800eb90 tx=0x7f9e5800c4b0 comp rx=0 tx=0).stop 2024-09-11T01:31:54.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.858+0000 7f9e6d39c640 1 --2- 172.21.15.12:0/3570064983 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9e5403e210 0x7f9e540406d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9e6809dc60 tx=0x7f9e5c007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:54.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.864+0000 7f9e6eb9f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:31:54.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.864+0000 7f9e6eb9f640 1 --2- 172.21.15.12:0/3570064983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e6813ac80 0x7f9e6809e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:54.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.864+0000 7f9e6eb9f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:54.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.864+0000 7f9e6eb9f640 1 -- 172.21.15.12:0/3570064983 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6813b090 con 0x7f9e6813ac80 2024-09-11T01:31:54.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.864+0000 7f9e6db9d640 1 --2- 172.21.15.12:0/3570064983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e6813ac80 0x7f9e6809e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:54.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.864+0000 7f9e6db9d640 10 monclient: get_auth_request con 0x7f9e6813ac80 auth_method 0 2024-09-11T01:31:54.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.864+0000 7f9e6db9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:54.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.864+0000 7f9e6db9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:54.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.864+0000 7f9e6db9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:54.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.864+0000 7f9e6db9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:54.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.864+0000 7f9e6db9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:54.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.865+0000 7f9e6db9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:54.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.865+0000 7f9e6db9d640 10 monclient(hunting): handle_auth_done global_id 24210 payload 824 2024-09-11T01:31:54.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.865+0000 7f9e6db9d640 1 -- 172.21.15.12:0/3570064983 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9e680a5e50 con 0x7f9e6813ac80 2024-09-11T01:31:54.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.867+0000 7f9e6db9d640 1 -- 172.21.15.12:0/3570064983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e6813ac80 msgr2=0x7f9e6809e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:54.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.867+0000 7f9e6db9d640 1 --2- 172.21.15.12:0/3570064983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e6813ac80 0x7f9e6809e290 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9e58018000 tx=0x7f9e58004cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:54.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.867+0000 7f9e667fc640 1 -- 172.21.15.12:0/3570064983 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e58017040 con 0x7f9e6813ac80 2024-09-11T01:31:54.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.867+0000 7f9e667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:54.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.867+0000 7f9e667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:54.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.867+0000 7f9e667fc640 10 monclient: dump: 2024-09-11T01:31:54.876 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:54.876 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:54.876 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:54.876 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:54.876 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:54.876 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:54.876 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:54.877 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:54.877 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:54.877 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:54.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.867+0000 7f9e667fc640 1 -- 172.21.15.12:0/3570064983 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f9e5801f030 con 0x7f9e6813ac80 2024-09-11T01:31:54.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.867+0000 7f9e667fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:31:54.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.867+0000 7f9e667fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:31:54.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.867+0000 7f9e667fc640 1 -- 172.21.15.12:0/3570064983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e6813ac80 msgr2=0x7f9e6809e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:54.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.867+0000 7f9e667fc640 1 --2- 172.21.15.12:0/3570064983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e6813ac80 0x7f9e6809e290 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9e58018000 tx=0x7f9e58004cc0 comp rx=0 tx=0).stop 2024-09-11T01:31:54.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.867+0000 7f9e667fc640 1 -- 172.21.15.12:0/3570064983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e6813ac80 msgr2=0x7f9e6809e290 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:31:54.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.867+0000 7f9e667fc640 1 --2- 172.21.15.12:0/3570064983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e6813ac80 0x7f9e6809e290 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:54.878 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:31:54.878 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:31:54.878 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:31:54.878 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:31:54.878 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 28, 2024-09-11T01:31:54.878 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 52814, 2024-09-11T01:31:54.878 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:31:54.878 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:31:54.878 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:31:54.878 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:31:54.878 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:54.878 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:31:54.879 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:31:54.879 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:31:54.879 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:31:54.879 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:54.879 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:54.879 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:54.879 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:54.879 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:54.879 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:54.879 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:54.879 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:54.879 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:54.879 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:54.880 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:54.880 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:54.880 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:31:54.880 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:31:54.880 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:54.880 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:54.880 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:54.880 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:54.880 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:54.880 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:54.880 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:54.880 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:54.880 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:54.880 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:54.881 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:54.881 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:54.881 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:54.881 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:31:54.881 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:31:54.881 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:31:54.881 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:31:54.881 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:31:54.881 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:31:54.881 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:31:54.881 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:31:54.881 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:31:54.881 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:31:54.882 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:31:54.882 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:31:54.882 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:31:54.882 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:31:54.882 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:31:54.882 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:31:54.882 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:31:54.882 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:54.882 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:54.882 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:54.882 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:54.882 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:54.882 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:54.882 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:54.883 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:54.883 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:54.883 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:54.883 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:54.883 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:54.883 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:31:54.883 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:54.883 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:31:54.883 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:54.883 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:31:54.883 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:31:54.883 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:54.883 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:54.883 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:54.884 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:54.884 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:31:54.884 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:54.884 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:54.884 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:54.884 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:54.884 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:31:54.884 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:54.884 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:54.884 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:54.884 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:54.884 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:31:54.884 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:31:54.884 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:54.885 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:54.885 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:54.885 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:54.885 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:54.885 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:31:54.885 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:31:54.885 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:54.885 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:54.885 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:54.885 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:54.885 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:31:54.885 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:54.885 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:54.886 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:54.886 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:54.886 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:31:54.886 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:54.886 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:54.886 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:54.886 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:54.886 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:31:54.886 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:31:54.886 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:54.886 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:54.886 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:54.886 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:54.886 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:54.887 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:31:54.887 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:31:54.887 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:54.887 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:54.887 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:54.887 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:54.887 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:31:54.887 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:54.887 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:54.887 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:54.887 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:54.887 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:31:54.887 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:54.887 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:54.887 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:54.888 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:54.888 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:31:54.888 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:31:54.888 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:54.888 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:54.888 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:54.888 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:54.888 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:54.888 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:54.888 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:31:54.888 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:31:54.888 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:54.888 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:54.888 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:54.889 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:31:54.889 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:54.889 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:54.889 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:31:54.889 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:54.889 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:54.889 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:54.889 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:31:54.889 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:54.889 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:54.889 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:31:54.889 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:54.889 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:54.889 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:54.890 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:31:54.890 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:54.890 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:54.890 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:31:54.890 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:54.890 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:54.890 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:54.890 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:31:54.890 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:54.890 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:54.890 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:54.890 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:31:54.890 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:31:54.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.868+0000 7f9e6eb9f640 1 -- 172.21.15.12:0/3570064983 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9e5403e210 msgr2=0x7f9e540406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:54.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.868+0000 7f9e6eb9f640 1 --2- 172.21.15.12:0/3570064983 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9e5403e210 0x7f9e540406d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9e6809dc60 tx=0x7f9e5c007e30 comp rx=0 tx=0).stop 2024-09-11T01:31:54.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.868+0000 7f9e6eb9f640 10 monclient: shutdown 2024-09-11T01:31:54.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.868+0000 7f9e6eb9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:54.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.868+0000 7f9e6eb9f640 1 -- 172.21.15.12:0/3570064983 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9e6813e1b0 msgr2=0x7f9e6813a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:54.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.868+0000 7f9e6eb9f640 1 --2- 172.21.15.12:0/3570064983 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9e6813e1b0 0x7f9e6813a200 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9e60009720 tx=0x7f9e600046c0 comp rx=0 tx=0).stop 2024-09-11T01:31:54.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.868+0000 7f9e6e39e640 1 -- 172.21.15.12:0/3570064983 reap_dead start 2024-09-11T01:31:54.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.868+0000 7f9e6e39e640 1 -- 172.21.15.12:0/3570064983 reap_dead start 2024-09-11T01:31:54.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.868+0000 7f9e6eb9f640 1 -- 172.21.15.12:0/3570064983 shutdown_connections 2024-09-11T01:31:54.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.868+0000 7f9e6eb9f640 1 -- 172.21.15.12:0/3570064983 >> 172.21.15.12:0/3570064983 conn(0x7f9e680aaf90 msgr2=0x7f9e680a3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:54.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.868+0000 7f9e6eb9f640 1 -- 172.21.15.12:0/3570064983 shutdown_connections 2024-09-11T01:31:54.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.868+0000 7f9e6eb9f640 1 -- 172.21.15.12:0/3570064983 wait complete. 2024-09-11T01:31:54.893 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:31:54.893 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:31:54.893 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:31:54.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.994+0000 7f8df967c640 10 monclient: get_monmap_and_config 2024-09-11T01:31:54.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.994+0000 7f8df967c640 10 monclient: build_initial_monmap 2024-09-11T01:31:54.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.994+0000 7f8df967c640 10 monclient: monmap: 2024-09-11T01:31:54.996 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:54.996 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:54.996 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:31:54.995001+0000 2024-09-11T01:31:54.996 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:31:54.995001+0000 2024-09-11T01:31:54.996 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:54.997 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:54.997 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:31:54.997 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:31:54.997 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:31:54.997 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:31:54.997 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:31:54.997 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:55.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.994+0000 7f8df967c640 1 Processor -- start 2024-09-11T01:31:55.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8df967c640 1 -- start start 2024-09-11T01:31:55.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8df967c640 10 monclient: init 2024-09-11T01:31:55.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8df967c640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:55.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8df967c640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-11T01:31:55.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8df967c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8df4137d50 0x7f8df409d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:55.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8df967c640 10 monclient(hunting): picked mon.noname-a con 0x7f8df4137d50 addr v2:172.21.15.12:3300/0 2024-09-11T01:31:55.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8df967c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df4138130 0x7f8df4138550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:55.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8df967c640 10 monclient(hunting): picked mon.noname-b con 0x7f8df4138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:55.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8df967c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8df4136cd0 addr v1:172.21.15.71:6789/0 2024-09-11T01:31:55.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8df967c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df409ba20 con 0x7f8df4136cd0 2024-09-11T01:31:55.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8df967c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:55.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8df967c640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f8df409c650 con 0x7f8df4137d50 2024-09-11T01:31:55.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8df967c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:55.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8df967c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8df409c370 con 0x7f8df4138130 2024-09-11T01:31:55.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8df967c640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:55.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8df967c640 10 monclient(hunting): authenticate will time out at 921.437558s 2024-09-11T01:31:55.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8debfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df4138130 0x7f8df4138550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:55.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8debfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df4138130 0x7f8df4138550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41322/0 (socket says 172.21.15.12:41322) 2024-09-11T01:31:55.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8debfff640 1 -- 172.21.15.12:0/737595483 learned_addr learned my addr 172.21.15.12:0/737595483 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:55.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8debfff640 10 monclient(hunting): get_auth_request con 0x7f8df4138130 auth_method 0 2024-09-11T01:31:55.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8debfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:55.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8debfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:55.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8debfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:55.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8df2ffd640 1 --2- 172.21.15.12:0/737595483 >> v2:172.21.15.12:3300/0 conn(0x7f8df4137d50 0x7f8df409d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:55.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8df2ffd640 10 monclient(hunting): get_auth_request con 0x7f8df4137d50 auth_method 0 2024-09-11T01:31:55.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8df2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:55.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8df2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:55.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8df2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:55.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8debfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:55.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8debfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:55.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8debfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:55.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8df2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:55.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8df2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:55.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8df2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:55.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8debfff640 10 monclient(hunting): handle_auth_done global_id 4640 payload 274 2024-09-11T01:31:55.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.995+0000 7f8debfff640 1 -- 172.21.15.12:0/737595483 >> v2:172.21.15.12:3300/0 conn(0x7f8df4137d50 msgr2=0x7f8df409d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:55.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.996+0000 7f8debfff640 1 --2- 172.21.15.12:0/737595483 >> v2:172.21.15.12:3300/0 conn(0x7f8df4137d50 0x7f8df409d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:55.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.996+0000 7f8debfff640 1 -- 172.21.15.12:0/737595483 >> v1:172.21.15.71:6789/0 conn(0x7f8df4136cd0 legacy=0x7f8df4137120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:55.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.996+0000 7f8debfff640 10 monclient: _finish_hunting 0 2024-09-11T01:31:55.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.996+0000 7f8debfff640 1 monclient: found mon.noname-b 2024-09-11T01:31:55.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.996+0000 7f8debfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:55.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.996+0000 7f8debfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:31:55.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.996+0000 7f8debfff640 1 -- 172.21.15.12:0/737595483 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df40a6ce0 con 0x7f8df4138130 2024-09-11T01:31:55.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.996+0000 7f8df2ffd640 1 --2- 172.21.15.12:0/737595483 >> v2:172.21.15.12:3300/0 conn(0x7f8df4137d50 0x7f8df409d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:31:55.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.998+0000 7f8debfff640 1 -- 172.21.15.12:0/737595483 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df4138130 msgr2=0x7f8df4138550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:55.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.998+0000 7f8debfff640 1 --2- 172.21.15.12:0/737595483 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df4138130 0x7f8df4138550 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8df409c7d0 tx=0x7f8ddc02a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f76e8f6c195d8928 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:55.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.998+0000 7f8df27fc640 1 -- 172.21.15.12:0/737595483 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc037070 con 0x7f8df4138130 2024-09-11T01:31:55.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.999+0000 7f8df27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:55.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.999+0000 7f8df27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:31:55.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.999+0000 7f8df27fc640 10 monclient: dump: 2024-09-11T01:31:55.004 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:55.005 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:55.005 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:55.005 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:55.005 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:55.005 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:55.005 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:55.005 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:55.005 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:55.005 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:55.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.999+0000 7f8df27fc640 10 monclient: _finish_auth 0 2024-09-11T01:31:55.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.999+0000 7f8df27fc640 10 monclient: _check_auth_tickets 2024-09-11T01:31:55.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.999+0000 7f8df27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:55.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.999+0000 7f8df27fc640 1 -- 172.21.15.12:0/737595483 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ddc032640 con 0x7f8df4138130 2024-09-11T01:31:55.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.999+0000 7f8df967c640 5 monclient: authenticate success, global_id 4640 2024-09-11T01:31:55.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.999+0000 7f8df967c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:31:55.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.999+0000 7f8df27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:55.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.999+0000 7f8df967c640 10 monclient: get_monmap_and_config success 2024-09-11T01:31:55.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.999+0000 7f8df27fc640 1 -- 172.21.15.12:0/737595483 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc032a10 con 0x7f8df4138130 2024-09-11T01:31:55.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.999+0000 7f8df27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:55.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.999+0000 7f8df967c640 4 set_mon_vals no callback set 2024-09-11T01:31:55.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.999+0000 7f8df27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:55.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.999+0000 7f8df27fc640 10 monclient: dump: 2024-09-11T01:31:55.007 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:55.007 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:55.007 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:55.007 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:55.007 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:55.007 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:55.007 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:55.007 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:55.007 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:55.007 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:55.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.999+0000 7f8df967c640 10 monclient: shutdown 2024-09-11T01:31:55.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.999+0000 7f8df967c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:55.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.999+0000 7f8df967c640 1 -- 172.21.15.12:0/737595483 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df4138130 msgr2=0x7f8df4138550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:55.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.999+0000 7f8df967c640 1 --2- 172.21.15.12:0/737595483 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df4138130 0x7f8df4138550 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8df409c7d0 tx=0x7f8ddc02a110 comp rx=0 tx=0).stop 2024-09-11T01:31:55.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.999+0000 7f8df967c640 1 -- 172.21.15.12:0/737595483 shutdown_connections 2024-09-11T01:31:55.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.999+0000 7f8df967c640 1 --2- 172.21.15.12:0/737595483 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df4138130 0x7f8df4138550 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:55.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.999+0000 7f8df967c640 1 --2- 172.21.15.12:0/737595483 >> v2:172.21.15.12:3300/0 conn(0x7f8df4137d50 0x7f8df409d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:55.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.999+0000 7f8df967c640 1 -- 172.21.15.12:0/737595483 >> 172.21.15.12:0/737595483 conn(0x7f8df40a7020 msgr2=0x7f8df40a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:55.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.999+0000 7f8df967c640 1 -- 172.21.15.12:0/737595483 shutdown_connections 2024-09-11T01:31:55.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:54.999+0000 7f8df967c640 1 -- 172.21.15.12:0/737595483 wait complete. 2024-09-11T01:31:55.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.000+0000 7f8df967c640 10 monclient: build_initial_monmap 2024-09-11T01:31:55.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.000+0000 7f8df967c640 10 monclient: monmap: 2024-09-11T01:31:55.015 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:55.015 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:31:55.016 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:31:55.016 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:31:55.016 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:55.016 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:55.016 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:31:55.016 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:31:55.016 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:31:55.016 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:55.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.000+0000 7f8df967c640 1 Processor -- start 2024-09-11T01:31:55.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.000+0000 7f8df967c640 1 -- start start 2024-09-11T01:31:55.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.000+0000 7f8df967c640 10 monclient: init 2024-09-11T01:31:55.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df967c640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:55.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df967c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:31:55.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df967c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8df4136cd0 0x7f8df41cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:55.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df967c640 10 monclient(hunting): picked mon.noname-b con 0x7f8df4136cd0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:31:55.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df967c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8df4137d50 0x7f8df41cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:55.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df967c640 10 monclient(hunting): picked mon.noname-a con 0x7f8df4137d50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:55.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df967c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df4138130 0x7f8df41234e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:55.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df967c640 10 monclient(hunting): picked mon.noname-c con 0x7f8df4138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:55.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df967c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:55.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df967c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8df409d650 con 0x7f8df4137d50 2024-09-11T01:31:55.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df967c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:55.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df967c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8df406f710 con 0x7f8df4136cd0 2024-09-11T01:31:55.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df967c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:55.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df967c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8df409d7d0 con 0x7f8df4138130 2024-09-11T01:31:55.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df967c640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:55.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df967c640 10 monclient(hunting): authenticate will time out at 921.443083s 2024-09-11T01:31:55.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8debfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8df4137d50 0x7f8df41cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:55.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df37fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df4138130 0x7f8df41234e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:55.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8debfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8df4137d50 0x7f8df41cd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40828/0 (socket says 172.21.15.12:40828) 2024-09-11T01:31:55.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8debfff640 1 -- 172.21.15.12:0/676457756 learned_addr learned my addr 172.21.15.12:0/676457756 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:55.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df37fe640 10 monclient(hunting): get_auth_request con 0x7f8df4138130 auth_method 0 2024-09-11T01:31:55.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:55.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:55.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:55.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8debfff640 10 monclient(hunting): get_auth_request con 0x7f8df4137d50 auth_method 0 2024-09-11T01:31:55.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8debfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:55.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8debfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:55.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8debfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:55.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:55.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:55.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:55.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8debfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:55.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8debfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:55.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8debfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:55.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df37fe640 10 monclient(hunting): handle_auth_done global_id 4643 payload 824 2024-09-11T01:31:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df2ffd640 1 --2- 172.21.15.12:0/676457756 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8df4136cd0 0x7f8df41cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df37fe640 1 -- 172.21.15.12:0/676457756 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8df4136cd0 msgr2=0x7f8df41cd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df37fe640 1 --2- 172.21.15.12:0/676457756 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8df4136cd0 0x7f8df41cd3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df37fe640 1 -- 172.21.15.12:0/676457756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8df4137d50 msgr2=0x7f8df41cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df37fe640 1 --2- 172.21.15.12:0/676457756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8df4137d50 0x7f8df41cd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df37fe640 10 monclient: _finish_hunting 0 2024-09-11T01:31:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df37fe640 1 monclient: found mon.noname-c 2024-09-11T01:31:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:31:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8df37fe640 1 -- 172.21.15.12:0/676457756 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df41368c0 con 0x7f8df4138130 2024-09-11T01:31:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.001+0000 7f8debfff640 1 --2- 172.21.15.12:0/676457756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8df4137d50 0x7f8df41cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:31:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.003+0000 7f8df37fe640 1 -- 172.21.15.12:0/676457756 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df4138130 msgr2=0x7f8df41234e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:55.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.003+0000 7f8df37fe640 1 --2- 172.21.15.12:0/676457756 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df4138130 0x7f8df41234e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8dec00c780 tx=0x7f8dec00cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.003+0000 7f8deb7fe640 1 -- 172.21.15.12:0/676457756 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dec016020 con 0x7f8df4138130 2024-09-11T01:31:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.003+0000 7f8deb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.003+0000 7f8deb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:31:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.003+0000 7f8deb7fe640 10 monclient: dump: 2024-09-11T01:31:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:55.021 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:55.021 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:55.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.003+0000 7f8deb7fe640 10 monclient: _finish_auth 0 2024-09-11T01:31:55.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.003+0000 7f8deb7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:31:55.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.003+0000 7f8deb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:55.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.003+0000 7f8deb7fe640 1 -- 172.21.15.12:0/676457756 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dec00ac60 con 0x7f8df4138130 2024-09-11T01:31:55.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.003+0000 7f8df967c640 5 monclient: authenticate success, global_id 4643 2024-09-11T01:31:55.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.003+0000 7f8df967c640 10 monclient: _renew_subs 2024-09-11T01:31:55.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.003+0000 7f8df967c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:55.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.003+0000 7f8df967c640 1 -- 172.21.15.12:0/676457756 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8df4127300 con 0x7f8df4138130 2024-09-11T01:31:55.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.003+0000 7f8deb7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:55.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.003+0000 7f8deb7fe640 1 -- 172.21.15.12:0/676457756 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dec012660 con 0x7f8df4138130 2024-09-11T01:31:55.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.003+0000 7f8deb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:55.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.003+0000 7f8deb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:55.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.003+0000 7f8deb7fe640 10 monclient: dump: 2024-09-11T01:31:55.023 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:55.023 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:55.023 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:55.023 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:55.023 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:55.023 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:55.023 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:55.023 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:55.023 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:55.023 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:55.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.003+0000 7f8df967c640 10 monclient: _renew_subs 2024-09-11T01:31:55.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.003+0000 7f8df967c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:31:55.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.003+0000 7f8df1ffb640 4 set_mon_vals no callback set 2024-09-11T01:31:55.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.003+0000 7f8df967c640 1 -- 172.21.15.12:0/676457756 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8df4127020 con 0x7f8df4138130 2024-09-11T01:31:55.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.004+0000 7f8df967c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:55.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.004+0000 7f8df967c640 1 --2- 172.21.15.12:0/676457756 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df41241e0 0x7f8df402d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:55.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.004+0000 7f8df967c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:55.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.004+0000 7f8df967c640 1 -- 172.21.15.12:0/676457756 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8df40a8bf0 con 0x7f8df41241e0 2024-09-11T01:31:55.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.004+0000 7f8deb7fe640 1 -- 172.21.15.12:0/676457756 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f8dec0128c0 con 0x7f8df4138130 2024-09-11T01:31:55.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.004+0000 7f8df2ffd640 1 --2- 172.21.15.12:0/676457756 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df41241e0 0x7f8df402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:55.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.004+0000 7f8df2ffd640 10 monclient: get_auth_request con 0x7f8df41241e0 auth_method 0 2024-09-11T01:31:55.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.004+0000 7f8df2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:55.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.004+0000 7f8df2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:55.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.004+0000 7f8df2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:55.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.005+0000 7f8df2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:55.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.005+0000 7f8df2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:55.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.005+0000 7f8df2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:55.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.005+0000 7f8deb7fe640 1 --2- 172.21.15.12:0/676457756 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8dd403e260 0x7f8dd4040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:55.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.005+0000 7f8deb7fe640 1 -- 172.21.15.12:0/676457756 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(98..98 src has 1..98) ==== 4739+0+0 (secure 0 0 0) 0x7f8dec053140 con 0x7f8df4138130 2024-09-11T01:31:55.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.005+0000 7f8df2ffd640 10 monclient(hunting): handle_auth_done global_id 4646 payload 824 2024-09-11T01:31:55.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.005+0000 7f8df2ffd640 1 -- 172.21.15.12:0/676457756 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8df41379d0 con 0x7f8df41241e0 2024-09-11T01:31:55.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.005+0000 7f8debfff640 1 --2- 172.21.15.12:0/676457756 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8dd403e260 0x7f8dd4040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:55.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.005+0000 7f8debfff640 10 monclient: get_auth_request con 0x7f8dd403e260 auth_method 0 2024-09-11T01:31:55.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.007+0000 7f8df2ffd640 1 -- 172.21.15.12:0/676457756 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df41241e0 msgr2=0x7f8df402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:55.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.007+0000 7f8df2ffd640 1 --2- 172.21.15.12:0/676457756 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df41241e0 0x7f8df402d440 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8de40107c0 tx=0x7f8de4010c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:55.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.007+0000 7f8deb7fe640 1 -- 172.21.15.12:0/676457756 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de400f040 con 0x7f8df41241e0 2024-09-11T01:31:55.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.007+0000 7f8deb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:55.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.007+0000 7f8deb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:55.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.007+0000 7f8deb7fe640 10 monclient: dump: 2024-09-11T01:31:55.026 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:55.026 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:55.026 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:55.026 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:55.026 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:55.026 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:55.026 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:55.026 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:55.026 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:55.026 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:55.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.007+0000 7f8deb7fe640 1 -- 172.21.15.12:0/676457756 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8de4010e90 con 0x7f8df41241e0 2024-09-11T01:31:55.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.007+0000 7f8deb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:55.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.007+0000 7f8deb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:31:55.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.007+0000 7f8deb7fe640 1 -- 172.21.15.12:0/676457756 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df41241e0 msgr2=0x7f8df402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:55.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.007+0000 7f8deb7fe640 1 --2- 172.21.15.12:0/676457756 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df41241e0 0x7f8df402d440 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8de40107c0 tx=0x7f8de4010c80 comp rx=0 tx=0).stop 2024-09-11T01:31:55.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.007+0000 7f8deb7fe640 1 -- 172.21.15.12:0/676457756 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df41241e0 msgr2=0x7f8df402d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:31:55.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.007+0000 7f8deb7fe640 1 --2- 172.21.15.12:0/676457756 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df41241e0 0x7f8df402d440 secure :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8de40107c0 tx=0x7f8de4010c80 comp rx=0 tx=0).stop 2024-09-11T01:31:55.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.007+0000 7f8debfff640 1 --2- 172.21.15.12:0/676457756 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8dd403e260 0x7f8dd4040720 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc02a150 tx=0x7f8ddc02e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:55.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.014+0000 7f8df967c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:31:55.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.014+0000 7f8df967c640 1 --2- 172.21.15.12:0/676457756 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df41245c0 0x7f8df402d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:55.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.014+0000 7f8df967c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:55.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.014+0000 7f8df967c640 1 -- 172.21.15.12:0/676457756 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8df40a0780 con 0x7f8df41245c0 2024-09-11T01:31:55.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.014+0000 7f8df2ffd640 1 --2- 172.21.15.12:0/676457756 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df41245c0 0x7f8df402d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:55.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.014+0000 7f8df2ffd640 10 monclient: get_auth_request con 0x7f8df41245c0 auth_method 0 2024-09-11T01:31:55.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.014+0000 7f8df2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:55.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.014+0000 7f8df2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:55.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.014+0000 7f8df2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:55.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.014+0000 7f8df2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:55.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.014+0000 7f8df2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:55.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.014+0000 7f8df2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:55.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.014+0000 7f8df2ffd640 10 monclient(hunting): handle_auth_done global_id 4649 payload 824 2024-09-11T01:31:55.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.014+0000 7f8df2ffd640 1 -- 172.21.15.12:0/676457756 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8df4074a40 con 0x7f8df41245c0 2024-09-11T01:31:55.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.016+0000 7f8df2ffd640 1 -- 172.21.15.12:0/676457756 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df41245c0 msgr2=0x7f8df402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:55.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.016+0000 7f8df2ffd640 1 --2- 172.21.15.12:0/676457756 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df41245c0 0x7f8df402d980 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8de40107c0 tx=0x7f8de40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:55.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.016+0000 7f8deb7fe640 1 -- 172.21.15.12:0/676457756 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de4021790 con 0x7f8df41245c0 2024-09-11T01:31:55.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.016+0000 7f8deb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:55.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.016+0000 7f8deb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:31:55.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.016+0000 7f8deb7fe640 10 monclient: dump: 2024-09-11T01:31:55.031 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:55.031 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:55.031 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:55.031 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:55.031 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:55.031 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:55.031 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:55.031 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:55.031 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:55.031 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:55.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.017+0000 7f8deb7fe640 1 -- 172.21.15.12:0/676457756 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7f8de4021770 con 0x7f8df41245c0 2024-09-11T01:31:55.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.017+0000 7f8deb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:31:55.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.017+0000 7f8deb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:31:55.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.017+0000 7f8deb7fe640 1 -- 172.21.15.12:0/676457756 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df41245c0 msgr2=0x7f8df402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:55.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.017+0000 7f8deb7fe640 1 --2- 172.21.15.12:0/676457756 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df41245c0 0x7f8df402d980 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8de40107c0 tx=0x7f8de40046c0 comp rx=0 tx=0).stop 2024-09-11T01:31:55.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.017+0000 7f8deb7fe640 1 -- 172.21.15.12:0/676457756 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df41245c0 msgr2=0x7f8df402d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:31:55.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.017+0000 7f8deb7fe640 1 --2- 172.21.15.12:0/676457756 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df41245c0 0x7f8df402d980 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8de40107c0 tx=0x7f8de40046c0 comp rx=0 tx=0).stop 2024-09-11T01:31:55.032 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:31:55.032 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:31:55.033 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:31:55.033 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:31:55.033 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 28, 2024-09-11T01:31:55.033 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 177705, 2024-09-11T01:31:55.033 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:31:55.033 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:31:55.033 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:31:55.033 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:31:55.033 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:55.033 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:31:55.033 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:31:55.033 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:31:55.034 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:31:55.034 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:55.034 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:55.034 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:55.034 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:55.034 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:55.034 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:55.034 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:55.034 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:55.034 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:55.034 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:55.034 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:55.034 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:55.034 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:31:55.035 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:31:55.035 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:55.035 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:55.035 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:55.035 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:55.035 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:55.035 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:55.035 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:55.035 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:55.035 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:55.035 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:55.035 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:55.035 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:55.036 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:55.036 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:31:55.036 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:31:55.036 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:31:55.036 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:31:55.036 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:31:55.036 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:31:55.036 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:31:55.036 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:31:55.036 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:31:55.036 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:31:55.036 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:31:55.036 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:31:55.037 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:31:55.037 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:31:55.037 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:31:55.037 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:31:55.037 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:31:55.037 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:55.037 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:55.037 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:55.037 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:55.037 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:55.037 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:55.037 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:55.037 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:55.038 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:55.038 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:55.038 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:55.038 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:55.038 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:31:55.038 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:55.038 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:31:55.038 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:55.038 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:31:55.038 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:31:55.038 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:55.038 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:55.038 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:55.038 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:55.039 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:31:55.039 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:55.039 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:55.039 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:55.039 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:55.039 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:31:55.039 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:55.039 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:55.039 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:55.039 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:55.039 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:31:55.039 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:31:55.039 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:55.039 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:55.040 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:55.040 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:55.040 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:55.040 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:31:55.040 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:31:55.040 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:55.040 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:55.040 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:55.040 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:55.040 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:31:55.040 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:55.040 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:55.040 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:55.041 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:55.041 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:31:55.041 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:55.041 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:55.041 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:55.041 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:55.041 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:31:55.041 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:31:55.041 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:55.041 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:55.041 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:55.041 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:55.041 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:55.041 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:31:55.042 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:31:55.042 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:55.042 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:55.042 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:55.042 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:55.042 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:31:55.042 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:55.042 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:55.042 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:55.042 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:55.042 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:31:55.042 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:55.042 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:55.043 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:55.043 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:55.043 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:31:55.043 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:31:55.043 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:55.043 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:55.043 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:55.043 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:55.043 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:55.043 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:55.044 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:31:55.044 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:31:55.044 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:55.044 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:55.044 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:55.044 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:31:55.044 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:55.044 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:55.044 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:31:55.044 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:55.044 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:55.044 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:55.045 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:31:55.045 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:55.045 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:55.045 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:31:55.045 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:55.045 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:55.045 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:55.045 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:31:55.045 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:55.045 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:55.045 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:55.045 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:31:55.045 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:31:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.017+0000 7f8df967c640 1 -- 172.21.15.12:0/676457756 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8dd403e260 msgr2=0x7f8dd4040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.017+0000 7f8df967c640 1 --2- 172.21.15.12:0/676457756 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8dd403e260 0x7f8dd4040720 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc02a150 tx=0x7f8ddc02e000 comp rx=0 tx=0).stop 2024-09-11T01:31:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.017+0000 7f8df967c640 10 monclient: shutdown 2024-09-11T01:31:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.017+0000 7f8df967c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.017+0000 7f8df967c640 1 -- 172.21.15.12:0/676457756 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df4138130 msgr2=0x7f8df41234e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.017+0000 7f8df967c640 1 --2- 172.21.15.12:0/676457756 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df4138130 0x7f8df41234e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8dec00c780 tx=0x7f8dec00cc40 comp rx=0 tx=0).stop 2024-09-11T01:31:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.017+0000 7f8df37fe640 1 -- 172.21.15.12:0/676457756 reap_dead start 2024-09-11T01:31:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.017+0000 7f8df967c640 1 -- 172.21.15.12:0/676457756 shutdown_connections 2024-09-11T01:31:55.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.017+0000 7f8df967c640 1 --2- 172.21.15.12:0/676457756 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8df4138130 0x7f8df41234e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:55.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.017+0000 7f8df967c640 1 -- 172.21.15.12:0/676457756 >> 172.21.15.12:0/676457756 conn(0x7f8df40a7020 msgr2=0x7f8df40a4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:55.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.017+0000 7f8df967c640 1 -- 172.21.15.12:0/676457756 shutdown_connections 2024-09-11T01:31:55.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.017+0000 7f8df967c640 1 -- 172.21.15.12:0/676457756 wait complete. 2024-09-11T01:31:55.048 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:31:55.048 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:31:55.048 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:31:55.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.148+0000 7fc5a467f640 10 monclient: get_monmap_and_config 2024-09-11T01:31:55.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.148+0000 7fc5a467f640 10 monclient: build_initial_monmap 2024-09-11T01:31:55.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.148+0000 7fc5a467f640 10 monclient: monmap: 2024-09-11T01:31:55.150 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:55.150 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:55.150 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:31:55.149319+0000 2024-09-11T01:31:55.150 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:31:55.149319+0000 2024-09-11T01:31:55.150 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:55.151 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:55.151 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:31:55.151 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:31:55.151 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:31:55.151 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:31:55.151 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:31:55.151 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:55.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.149+0000 7fc5a467f640 1 Processor -- start 2024-09-11T01:31:55.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.149+0000 7fc5a467f640 1 -- start start 2024-09-11T01:31:55.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.149+0000 7fc5a467f640 10 monclient: init 2024-09-11T01:31:55.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.149+0000 7fc5a467f640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:55.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.149+0000 7fc5a467f640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-11T01:31:55.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.149+0000 7fc5a467f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc59c135fc0 0x7fc59c1363a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:55.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.149+0000 7fc5a467f640 10 monclient(hunting): picked mon.noname-a con 0x7fc59c135fc0 addr v2:172.21.15.12:3300/0 2024-09-11T01:31:55.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.149+0000 7fc5a467f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc59c074000 addr v1:172.21.15.71:6789/0 2024-09-11T01:31:55.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.149+0000 7fc5a467f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc59c074410 addr v1:172.21.15.12:6789/0 2024-09-11T01:31:55.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.149+0000 7fc5a467f640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc59c0732d0 con 0x7fc59c074410 2024-09-11T01:31:55.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.149+0000 7fc5a467f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc59c073890 con 0x7fc59c074000 2024-09-11T01:31:55.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.149+0000 7fc5a467f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:55.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.149+0000 7fc5a467f640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc59c074820 con 0x7fc59c135fc0 2024-09-11T01:31:55.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.149+0000 7fc5a467f640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:55.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.149+0000 7fc5a467f640 10 monclient(hunting): authenticate will time out at 921.591897s 2024-09-11T01:31:55.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.150+0000 7fc5a241b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc59c135fc0 0x7fc59c1363a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:55.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.150+0000 7fc5a2c1c640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc59c074410 0x7fc59c0a3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34344/0 (socket says 172.21.15.12:34344) 2024-09-11T01:31:55.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.150+0000 7fc5a241b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc59c135fc0 0x7fc59c1363a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40834/0 (socket says 172.21.15.12:40834) 2024-09-11T01:31:55.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.150+0000 7fc5a2c1c640 1 -- 172.21.15.12:0/2334668397 learned_addr learned my addr 172.21.15.12:0/2334668397 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:31:55.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.150+0000 7fc5a241b640 10 monclient(hunting): get_auth_request con 0x7fc59c135fc0 auth_method 0 2024-09-11T01:31:55.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.150+0000 7fc5a241b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:55.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.150+0000 7fc5a241b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:55.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.150+0000 7fc5a241b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:55.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.150+0000 7fc5a241b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:55.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.150+0000 7fc5a241b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:55.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.150+0000 7fc5a241b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:55.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.150+0000 7fc5a241b640 10 monclient(hunting): handle_auth_done global_id 24219 payload 274 2024-09-11T01:31:55.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.150+0000 7fc5a241b640 1 -- 172.21.15.12:0/2334668397 >> v1:172.21.15.71:6789/0 conn(0x7fc59c074000 legacy=0x7fc59c0a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:55.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.150+0000 7fc5a241b640 1 -- 172.21.15.12:0/2334668397 >> v1:172.21.15.12:6789/0 conn(0x7fc59c074410 legacy=0x7fc59c0a3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:55.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.150+0000 7fc5a241b640 10 monclient: _finish_hunting 0 2024-09-11T01:31:55.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.150+0000 7fc5a241b640 1 monclient: found mon.noname-a 2024-09-11T01:31:55.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.150+0000 7fc5a241b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:55.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.150+0000 7fc5a241b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:31:55.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.150+0000 7fc5a241b640 1 -- 172.21.15.12:0/2334668397 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc59c0a3c40 con 0x7fc59c135fc0 2024-09-11T01:31:55.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a241b640 1 -- 172.21.15.12:0/2334668397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc59c135fc0 msgr2=0x7fc59c1363a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:55.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a241b640 1 --2- 172.21.15.12:0/2334668397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc59c135fc0 0x7fc59c1363a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc5980097a0 tx=0x7fc598029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49ca789f7041cee5 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:55.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a1419640 1 -- 172.21.15.12:0/2334668397 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc598037070 con 0x7fc59c135fc0 2024-09-11T01:31:55.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a1419640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:55.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a1419640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:31:55.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a1419640 10 monclient: dump: 2024-09-11T01:31:55.161 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:55.161 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:55.162 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:55.162 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:55.162 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:55.162 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:55.162 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:55.162 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:55.162 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:55.162 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:55.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a1419640 10 monclient: _finish_auth 0 2024-09-11T01:31:55.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a1419640 10 monclient: _check_auth_tickets 2024-09-11T01:31:55.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a1419640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:55.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a467f640 5 monclient: authenticate success, global_id 24219 2024-09-11T01:31:55.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a1419640 1 -- 172.21.15.12:0/2334668397 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc598032640 con 0x7fc59c135fc0 2024-09-11T01:31:55.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a467f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:31:55.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a1419640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:55.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a467f640 10 monclient: get_monmap_and_config success 2024-09-11T01:31:55.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a467f640 4 set_mon_vals no callback set 2024-09-11T01:31:55.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a1419640 1 -- 172.21.15.12:0/2334668397 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc598032a10 con 0x7fc59c135fc0 2024-09-11T01:31:55.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a1419640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:55.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a1419640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:55.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a1419640 10 monclient: dump: 2024-09-11T01:31:55.163 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:55.163 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:55.163 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:55.163 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:55.163 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:55.163 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:55.164 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:55.164 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:55.164 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:55.164 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:55.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a467f640 10 monclient: shutdown 2024-09-11T01:31:55.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a467f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:55.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a467f640 1 -- 172.21.15.12:0/2334668397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc59c135fc0 msgr2=0x7fc59c1363a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:55.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a467f640 1 --2- 172.21.15.12:0/2334668397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc59c135fc0 0x7fc59c1363a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc5980097a0 tx=0x7fc598029fd0 comp rx=0 tx=0).stop 2024-09-11T01:31:55.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a467f640 1 -- 172.21.15.12:0/2334668397 shutdown_connections 2024-09-11T01:31:55.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a467f640 1 --2- 172.21.15.12:0/2334668397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc59c135fc0 0x7fc59c1363a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:55.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a467f640 1 -- 172.21.15.12:0/2334668397 >> 172.21.15.12:0/2334668397 conn(0x7fc59c0a8600 msgr2=0x7fc59c0a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:55.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a467f640 1 -- 172.21.15.12:0/2334668397 shutdown_connections 2024-09-11T01:31:55.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.153+0000 7fc5a467f640 1 -- 172.21.15.12:0/2334668397 wait complete. 2024-09-11T01:31:55.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.154+0000 7fc5a467f640 10 monclient: build_initial_monmap 2024-09-11T01:31:55.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.154+0000 7fc5a467f640 10 monclient: monmap: 2024-09-11T01:31:55.165 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:55.165 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:31:55.165 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:31:55.165 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:31:55.165 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:55.165 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:55.165 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:31:55.165 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:31:55.166 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:31:55.166 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:55.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.154+0000 7fc5a467f640 1 Processor -- start 2024-09-11T01:31:55.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.154+0000 7fc5a467f640 1 -- start start 2024-09-11T01:31:55.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.154+0000 7fc5a467f640 10 monclient: init 2024-09-11T01:31:55.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a467f640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:55.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a467f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:31:55.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a467f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc59c074000 0x7fc59c134b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:55.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a467f640 10 monclient(hunting): picked mon.noname-a con 0x7fc59c074000 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:55.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a467f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc59c074410 0x7fc59c13aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:55.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a467f640 10 monclient(hunting): picked mon.noname-c con 0x7fc59c074410 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:55.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a467f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c135fc0 0x7fc59c13afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:55.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a467f640 10 monclient(hunting): picked mon.noname-b con 0x7fc59c135fc0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:31:55.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a467f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:55.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a467f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc59c1368e0 con 0x7fc59c074000 2024-09-11T01:31:55.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a467f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:55.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a467f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc59c072ff0 con 0x7fc59c135fc0 2024-09-11T01:31:55.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a467f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:55.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a467f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc59c0a7930 con 0x7fc59c074410 2024-09-11T01:31:55.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a467f640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:55.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a467f640 10 monclient(hunting): authenticate will time out at 921.597239s 2024-09-11T01:31:55.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a241b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc59c074000 0x7fc59c134b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:55.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a1c1a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc59c074410 0x7fc59c13aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:55.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a241b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc59c074000 0x7fc59c134b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40836/0 (socket says 172.21.15.12:40836) 2024-09-11T01:31:55.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a241b640 1 -- 172.21.15.12:0/3769418429 learned_addr learned my addr 172.21.15.12:0/3769418429 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:55.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a241b640 10 monclient(hunting): get_auth_request con 0x7fc59c074000 auth_method 0 2024-09-11T01:31:55.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a241b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:55.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a241b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:55.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a241b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:55.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a1c1a640 10 monclient(hunting): get_auth_request con 0x7fc59c074410 auth_method 0 2024-09-11T01:31:55.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a1c1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:55.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a1c1a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:55.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a1c1a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:55.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a241b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:55.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a241b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:55.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a241b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:55.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a1c1a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:55.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a1c1a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:55.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a1c1a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:55.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a2c1c640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c135fc0 0x7fc59c13afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:55.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a2c1c640 10 monclient(hunting): get_auth_request con 0x7fc59c135fc0 auth_method 0 2024-09-11T01:31:55.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a2c1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:55.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a2c1c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:55.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a2c1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:55.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a241b640 10 monclient(hunting): handle_auth_done global_id 24222 payload 824 2024-09-11T01:31:55.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a2c1c640 1 -- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c135fc0 msgr2=0x7fc59c13afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:31:55.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a2c1c640 1 -- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c135fc0 msgr2=0x7fc59c13afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:31:55.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a2c1c640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c135fc0 0x7fc59c13afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:31:55.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a241b640 1 -- 172.21.15.12:0/3769418429 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc59c074410 msgr2=0x7fc59c13aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:55.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a241b640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc59c074410 0x7fc59c13aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:55.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a241b640 1 -- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c135fc0 msgr2=0x7fc59c13afa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:31:55.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a2c1c640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c135fc0 0x7fc59c13afa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:31:55.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a241b640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c135fc0 0x7fc59c13afa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:55.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a241b640 10 monclient: _finish_hunting 0 2024-09-11T01:31:55.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a241b640 1 monclient: found mon.noname-a 2024-09-11T01:31:55.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a241b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:55.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a241b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:31:55.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.155+0000 7fc5a241b640 1 -- 172.21.15.12:0/3769418429 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc59c13e880 con 0x7fc59c074000 2024-09-11T01:31:55.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.157+0000 7fc5a241b640 1 -- 172.21.15.12:0/3769418429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc59c074000 msgr2=0x7fc59c134b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:55.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.157+0000 7fc5a241b640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc59c074000 0x7fc59c134b00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc5980097a0 tx=0x7fc5980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:55.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.158+0000 7fc592ffd640 1 -- 172.21.15.12:0/3769418429 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc598041070 con 0x7fc59c074000 2024-09-11T01:31:55.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.158+0000 7fc592ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:55.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.158+0000 7fc592ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:31:55.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.158+0000 7fc592ffd640 10 monclient: dump: 2024-09-11T01:31:55.170 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:55.170 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:55.171 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:55.171 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:55.171 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:55.171 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:55.171 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:55.171 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:55.171 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:55.171 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:55.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.158+0000 7fc592ffd640 10 monclient: _finish_auth 0 2024-09-11T01:31:55.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.158+0000 7fc592ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:31:55.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.158+0000 7fc592ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:55.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.158+0000 7fc5a467f640 5 monclient: authenticate success, global_id 24222 2024-09-11T01:31:55.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.158+0000 7fc592ffd640 1 -- 172.21.15.12:0/3769418429 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc59803c880 con 0x7fc59c074000 2024-09-11T01:31:55.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.158+0000 7fc5a467f640 10 monclient: _renew_subs 2024-09-11T01:31:55.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.158+0000 7fc5a467f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.158+0000 7fc5a467f640 1 -- 172.21.15.12:0/3769418429 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc59c09f610 con 0x7fc59c074000 2024-09-11T01:31:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.158+0000 7fc592ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.158+0000 7fc592ffd640 1 -- 172.21.15.12:0/3769418429 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc598037040 con 0x7fc59c074000 2024-09-11T01:31:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.158+0000 7fc592ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.158+0000 7fc592ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.158+0000 7fc592ffd640 10 monclient: dump: 2024-09-11T01:31:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:55.172 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:55.173 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.158+0000 7fc5a0c18640 4 set_mon_vals no callback set 2024-09-11T01:31:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.158+0000 7fc5a467f640 10 monclient: _renew_subs 2024-09-11T01:31:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.158+0000 7fc5a467f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.158+0000 7fc5a467f640 1 -- 172.21.15.12:0/3769418429 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc59c13d080 con 0x7fc59c074000 2024-09-11T01:31:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.158+0000 7fc5a467f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.158+0000 7fc5a467f640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 0x7fc59c13b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.158+0000 7fc5a467f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.158+0000 7fc5a467f640 1 -- 172.21.15.12:0/3769418429 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc59c06a060 con 0x7fc59c13b4e0 2024-09-11T01:31:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.159+0000 7fc592ffd640 1 -- 172.21.15.12:0/3769418429 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fc59803cc70 con 0x7fc59c074000 2024-09-11T01:31:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.159+0000 7fc5a1c1a640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 0x7fc59c13b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:55.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.159+0000 7fc592ffd640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc58003e210 0x7fc5800406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:55.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.159+0000 7fc5a1c1a640 10 monclient: get_auth_request con 0x7fc59c13b4e0 auth_method 0 2024-09-11T01:31:55.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.159+0000 7fc5a1c1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:55.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.159+0000 7fc5a1c1a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:55.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.159+0000 7fc5a1c1a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:55.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.159+0000 7fc592ffd640 1 -- 172.21.15.12:0/3769418429 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 4739+0+0 (secure 0 0 0) 0x7fc598073aa0 con 0x7fc59c074000 2024-09-11T01:31:55.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.159+0000 7fc5a2c1c640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc58003e210 0x7fc5800406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:55.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.159+0000 7fc5a1c1a640 1 -- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 msgr2=0x7fc59c13b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:31:55.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.159+0000 7fc5a1c1a640 1 -- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 msgr2=0x7fc59c13b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:31:55.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.159+0000 7fc5a1c1a640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 0x7fc59c13b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:31:55.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.159+0000 7fc5a2c1c640 10 monclient: get_auth_request con 0x7fc58003e210 auth_method 0 2024-09-11T01:31:55.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.159+0000 7fc5a1c1a640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 0x7fc59c13b8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:31:55.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.162+0000 7fc5a2c1c640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc58003e210 0x7fc5800406d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc594005160 tx=0x7fc594009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:55.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.361+0000 7fc5a1c1a640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 0x7fc59c13b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:55.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.361+0000 7fc5a1c1a640 10 monclient: get_auth_request con 0x7fc59c13b4e0 auth_method 0 2024-09-11T01:31:55.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.361+0000 7fc5a1c1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:55.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.361+0000 7fc5a1c1a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:55.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.361+0000 7fc5a1c1a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:31:55.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.361+0000 7fc5a1c1a640 1 -- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 msgr2=0x7fc59c13b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:31:55.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.361+0000 7fc5a1c1a640 1 -- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 msgr2=0x7fc59c13b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:31:55.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.361+0000 7fc5a1c1a640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 0x7fc59c13b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:31:55.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.361+0000 7fc5a1c1a640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 0x7fc59c13b8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:31:55.525 INFO:tasks.rados.rados.0.smithi071.stdout:442: done (2 left) 2024-09-11T01:31:55.526 INFO:tasks.rados.rados.0.smithi071.stdout:443: done (1 left) 2024-09-11T01:31:55.526 INFO:tasks.rados.rados.0.smithi071.stdout:444: done (0 left) 2024-09-11T01:31:55.526 INFO:tasks.rados.rados.0.smithi071.stdout:445: rollback oid 18 current snap is 39 2024-09-11T01:31:55.526 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 18 to 38 2024-09-11T01:31:55.526 INFO:tasks.rados.rados.0.smithi071.stdout:446: rollback oid 5 current snap is 39 2024-09-11T01:31:55.526 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 5 to 38 2024-09-11T01:31:55.526 INFO:tasks.rados.rados.0.smithi071.stdout:447: snap_remove snap 37 2024-09-11T01:31:55.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.762+0000 7fc5a1c1a640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 0x7fc59c13b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:55.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.762+0000 7fc5a1c1a640 10 monclient: get_auth_request con 0x7fc59c13b4e0 auth_method 0 2024-09-11T01:31:55.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.762+0000 7fc5a1c1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:55.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.762+0000 7fc5a1c1a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:55.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.762+0000 7fc5a1c1a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:31:55.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.762+0000 7fc5a1c1a640 1 -- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 msgr2=0x7fc59c13b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:31:55.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.762+0000 7fc5a1c1a640 1 -- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 msgr2=0x7fc59c13b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:31:55.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.762+0000 7fc5a1c1a640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 0x7fc59c13b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:31:55.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:55.762+0000 7fc5a1c1a640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 0x7fc59c13b8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:31:56.530 INFO:tasks.rados.rados.0.smithi071.stdout:447: done (2 left) 2024-09-11T01:31:56.530 INFO:tasks.rados.rados.0.smithi071.stdout:448: rollback oid 37 current snap is 39 2024-09-11T01:31:56.530 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 37 to 38 2024-09-11T01:31:56.530 INFO:tasks.rados.rados.0.smithi071.stdout:446: finishing rollback tid 1 to smithi07134777-5 2024-09-11T01:31:56.531 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 0 (ObjNum 81 snap 0 seq_num 0) dirty dne 2024-09-11T01:31:56.531 INFO:tasks.rados.rados.0.smithi071.stdout:445: finishing rollback tid 0 to smithi07134777-18 2024-09-11T01:31:56.531 INFO:tasks.rados.rados.0.smithi071.stdout:445: finishing rollback tid 1 to smithi07134777-18 2024-09-11T01:31:56.531 INFO:tasks.rados.rados.0.smithi071.stdout:445: finishing rollback tid 2 to smithi07134777-18 2024-09-11T01:31:56.531 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 122 (ObjNum 157 snap 34 seq_num 157) dirty exists 2024-09-11T01:31:56.531 INFO:tasks.rados.rados.0.smithi071.stdout:445: done (2 left) 2024-09-11T01:31:56.531 INFO:tasks.rados.rados.0.smithi071.stdout:446: done (1 left) 2024-09-11T01:31:56.531 INFO:tasks.rados.rados.0.smithi071.stdout:449: read oid 49 snap 38 2024-09-11T01:31:56.531 INFO:tasks.rados.rados.0.smithi071.stdout:449: expect (ObjNum 163 snap 36 seq_num 163) 2024-09-11T01:31:56.531 INFO:tasks.rados.rados.0.smithi071.stdout:450: snap_create 2024-09-11T01:31:56.534 INFO:tasks.rados.rados.0.smithi071.stdout:448: finishing rollback tid 0 to smithi07134777-37 2024-09-11T01:31:56.535 INFO:tasks.rados.rados.0.smithi071.stdout:448: finishing rollback tid 1 to smithi07134777-37 2024-09-11T01:31:56.536 INFO:tasks.rados.rados.0.smithi071.stdout:448: finishing rollback tid 2 to smithi07134777-37 2024-09-11T01:31:56.537 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 147 (ObjNum 170 snap 37 seq_num 170) dirty exists 2024-09-11T01:31:56.538 INFO:tasks.rados.rados.0.smithi071.stdout:449: expect (ObjNum 163 snap 36 seq_num 163) 2024-09-11T01:31:56.541 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 561980 2024-09-11T01:31:56.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:56.563+0000 7fc5a1c1a640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 0x7fc59c13b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:56.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:56.563+0000 7fc5a1c1a640 10 monclient: get_auth_request con 0x7fc59c13b4e0 auth_method 0 2024-09-11T01:31:56.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:56.563+0000 7fc5a1c1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:56.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:56.563+0000 7fc5a1c1a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:56.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:56.563+0000 7fc5a1c1a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:31:56.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:56.564+0000 7fc5a1c1a640 1 -- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 msgr2=0x7fc59c13b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:31:56.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:56.564+0000 7fc5a1c1a640 1 -- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 msgr2=0x7fc59c13b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:31:56.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:56.564+0000 7fc5a1c1a640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 0x7fc59c13b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:31:56.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:56.564+0000 7fc5a1c1a640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 0x7fc59c13b8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:31:57.074 INFO:tasks.rados.rados.0.smithi071.stdout:448: done (2 left) 2024-09-11T01:31:57.074 INFO:tasks.rados.rados.0.smithi071.stdout:449: done (1 left) 2024-09-11T01:31:57.074 INFO:tasks.rados.rados.0.smithi071.stdout:450: done (0 left) 2024-09-11T01:31:57.074 INFO:tasks.rados.rados.0.smithi071.stdout:451: write oid 28 current snap is 40 2024-09-11T01:31:57.074 INFO:tasks.rados.rados.0.smithi071.stdout:451: seq_num 180 ranges {625260=620939,1683347=599386,2943668=81592} 2024-09-11T01:31:57.080 INFO:tasks.rados.rados.0.smithi071.stdout:451: writing smithi07134777-28 from 625260 to 1246199 tid 1 2024-09-11T01:31:57.087 INFO:tasks.rados.rados.0.smithi071.stdout:451: writing smithi07134777-28 from 1683347 to 2282733 tid 2 2024-09-11T01:31:57.091 INFO:tasks.rados.rados.0.smithi071.stdout:451: writing smithi07134777-28 from 2943668 to 3025260 tid 3 2024-09-11T01:31:57.091 INFO:tasks.rados.rados.0.smithi071.stdout:452: snap_remove snap 38 2024-09-11T01:31:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.155+0000 7fc591ffb640 10 monclient: tick 2024-09-11T01:31:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.155+0000 7fc591ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:31:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.155+0000 7fc591ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.155+0000 7fc591ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:58.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.166+0000 7fc5a1c1a640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 0x7fc59c13b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:58.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.166+0000 7fc5a1c1a640 10 monclient: get_auth_request con 0x7fc59c13b4e0 auth_method 0 2024-09-11T01:31:58.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.166+0000 7fc5a1c1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:58.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.166+0000 7fc5a1c1a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:58.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.166+0000 7fc5a1c1a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:31:58.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.166+0000 7fc5a1c1a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:58.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.166+0000 7fc5a1c1a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:58.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.166+0000 7fc5a1c1a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:58.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.166+0000 7fc5a1c1a640 10 monclient(hunting): handle_auth_done global_id 34099 payload 824 2024-09-11T01:31:58.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.166+0000 7fc5a1c1a640 1 -- 172.21.15.12:0/3769418429 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc59c1363a0 con 0x7fc59c13b4e0 2024-09-11T01:31:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.169+0000 7fc5a1c1a640 1 -- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 msgr2=0x7fc59c13b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.169+0000 7fc5a1c1a640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 0x7fc59c13b8c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc59c13ea90 tx=0x7fc58c017e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.169+0000 7fc592ffd640 1 -- 172.21.15.12:0/3769418429 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58c0051b0 con 0x7fc59c13b4e0 2024-09-11T01:31:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.169+0000 7fc592ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.169+0000 7fc592ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:31:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.169+0000 7fc592ffd640 10 monclient: dump: 2024-09-11T01:31:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:58.171 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.169+0000 7fc592ffd640 1 -- 172.21.15.12:0/3769418429 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc58c0051b0 con 0x7fc59c13b4e0 2024-09-11T01:31:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.169+0000 7fc592ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.169+0000 7fc592ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:31:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.169+0000 7fc592ffd640 1 -- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 msgr2=0x7fc59c13b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.169+0000 7fc592ffd640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 0x7fc59c13b8c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc59c13ea90 tx=0x7fc58c017e00 comp rx=0 tx=0).stop 2024-09-11T01:31:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.169+0000 7fc592ffd640 1 -- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 msgr2=0x7fc59c13b8c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:31:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.169+0000 7fc592ffd640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c13b4e0 0x7fc59c13b8c0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc59c13ea90 tx=0x7fc58c017e00 comp rx=0 tx=0).stop 2024-09-11T01:31:58.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.176+0000 7fc5a467f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:31:58.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.176+0000 7fc5a467f640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c136a60 0x7fc59c136e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:58.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.176+0000 7fc5a467f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:58.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.176+0000 7fc5a467f640 1 -- 172.21.15.12:0/3769418429 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc59c13e170 con 0x7fc59c136a60 2024-09-11T01:31:58.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.177+0000 7fc5a1c1a640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c136a60 0x7fc59c136e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:58.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.177+0000 7fc5a1c1a640 10 monclient: get_auth_request con 0x7fc59c136a60 auth_method 0 2024-09-11T01:31:58.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.177+0000 7fc5a1c1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:58.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.177+0000 7fc5a1c1a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:58.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.177+0000 7fc5a1c1a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:58.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.177+0000 7fc5a1c1a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:58.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.177+0000 7fc5a1c1a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:58.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.177+0000 7fc5a1c1a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:58.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.177+0000 7fc5a1c1a640 10 monclient(hunting): handle_auth_done global_id 34102 payload 824 2024-09-11T01:31:58.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.177+0000 7fc5a1c1a640 1 -- 172.21.15.12:0/3769418429 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc59c13be00 con 0x7fc59c136a60 2024-09-11T01:31:58.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.179+0000 7fc5a1c1a640 1 -- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c136a60 msgr2=0x7fc59c136e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:58.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.180+0000 7fc5a1c1a640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c136a60 0x7fc59c136e70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc59c13ea90 tx=0x7fc58c003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:58.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.180+0000 7fc592ffd640 1 -- 172.21.15.12:0/3769418429 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58c027b20 con 0x7fc59c136a60 2024-09-11T01:31:58.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.180+0000 7fc592ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:58.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.180+0000 7fc592ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:31:58.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.180+0000 7fc592ffd640 10 monclient: dump: 2024-09-11T01:31:58.183 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:58.183 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:58.183 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:58.183 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:58.183 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:58.183 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:58.183 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:58.184 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:58.184 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:58.184 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:58.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.180+0000 7fc592ffd640 1 -- 172.21.15.12:0/3769418429 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4540 (secure 0 0 0) 0x7fc58c007030 con 0x7fc59c136a60 2024-09-11T01:31:58.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.180+0000 7fc592ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:31:58.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.180+0000 7fc592ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:31:58.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.180+0000 7fc592ffd640 1 -- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c136a60 msgr2=0x7fc59c136e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:58.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.180+0000 7fc592ffd640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c136a60 0x7fc59c136e70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc59c13ea90 tx=0x7fc58c003760 comp rx=0 tx=0).stop 2024-09-11T01:31:58.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.180+0000 7fc592ffd640 1 -- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c136a60 msgr2=0x7fc59c136e70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:31:58.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.180+0000 7fc592ffd640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc59c136a60 0x7fc59c136e70 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:58.185 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:31:58.185 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:31:58.185 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:31:58.185 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:31:58.185 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 28, 2024-09-11T01:31:58.185 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 11393, 2024-09-11T01:31:58.185 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:31:58.185 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:31:58.185 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:31:58.185 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:31:58.185 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:58.186 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 5, 2024-09-11T01:31:58.186 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:31:58.186 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:31:58.186 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:31:58.186 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:58.186 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:58.186 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:58.186 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:58.186 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:58.186 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:58.186 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:58.186 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:58.186 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:58.186 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:58.187 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:58.187 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:58.187 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:31:58.187 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:31:58.187 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:58.187 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:58.187 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:58.187 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:58.187 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:58.187 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:58.187 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:58.187 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:58.187 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:58.187 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:58.188 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:58.188 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:58.188 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:58.188 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:31:58.188 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:31:58.188 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:31:58.188 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:31:58.188 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:31:58.188 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:31:58.188 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:31:58.188 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:31:58.188 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:31:58.189 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:31:58.189 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:31:58.189 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:31:58.189 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:31:58.189 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:31:58.189 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:31:58.189 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:31:58.189 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:31:58.189 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:31:58.189 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:31:58.189 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:31:58.189 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:31:58.189 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:31:58.189 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:31:58.190 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:31:58.190 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:31:58.190 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:31:58.190 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:31:58.190 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:31:58.190 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:58.190 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:31:58.190 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:58.190 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:31:58.190 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:58.191 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:31:58.191 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:31:58.191 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:58.191 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:58.191 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:58.191 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:58.191 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:31:58.191 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:58.191 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:58.191 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:58.191 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:58.191 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:31:58.191 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:58.191 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:58.192 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:58.192 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:58.192 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:31:58.192 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:31:58.192 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:58.192 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:58.192 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:58.192 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:58.192 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:58.192 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:31:58.192 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:31:58.192 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:58.192 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:58.193 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:58.193 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:58.193 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:31:58.193 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:58.193 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:58.193 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:58.193 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:58.193 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:31:58.193 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:58.193 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:58.194 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:58.194 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:58.194 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:31:58.194 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:31:58.194 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:58.194 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:58.194 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:58.194 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:58.194 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:58.194 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:31:58.194 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:31:58.194 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:31:58.194 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:31:58.195 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:58.195 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:31:58.195 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:31:58.195 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:58.195 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:58.195 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:58.195 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:31:58.195 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:31:58.195 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:31:58.196 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:58.196 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:58.196 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:58.196 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:31:58.196 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:31:58.196 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:31:58.196 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:31:58.196 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:31:58.197 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:58.197 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:58.197 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:58.197 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:31:58.197 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:31:58.197 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:58.197 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:58.197 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:58.197 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:31:58.197 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:58.198 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:31:58.198 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:31:58.198 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:31:58.198 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:31:58.198 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:31:58.198 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:31:58.198 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:31:58.198 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:31:58.198 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:31:58.198 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:31:58.198 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:31:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.181+0000 7fc5a467f640 1 -- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc58003e210 msgr2=0x7fc5800406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.181+0000 7fc5a467f640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc58003e210 0x7fc5800406d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc594005160 tx=0x7fc594009040 comp rx=0 tx=0).stop 2024-09-11T01:31:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.181+0000 7fc5a467f640 10 monclient: shutdown 2024-09-11T01:31:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.181+0000 7fc5a467f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.181+0000 7fc5a467f640 1 -- 172.21.15.12:0/3769418429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc59c074000 msgr2=0x7fc59c134b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.181+0000 7fc5a467f640 1 --2- 172.21.15.12:0/3769418429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc59c074000 0x7fc59c134b00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc5980097a0 tx=0x7fc5980046c0 comp rx=0 tx=0).stop 2024-09-11T01:31:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.181+0000 7fc5a2c1c640 1 -- 172.21.15.12:0/3769418429 reap_dead start 2024-09-11T01:31:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.181+0000 7fc5a2c1c640 1 -- 172.21.15.12:0/3769418429 reap_dead start 2024-09-11T01:31:58.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.181+0000 7fc5a467f640 1 -- 172.21.15.12:0/3769418429 shutdown_connections 2024-09-11T01:31:58.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.181+0000 7fc5a467f640 1 -- 172.21.15.12:0/3769418429 >> 172.21.15.12:0/3769418429 conn(0x7fc59c0a8600 msgr2=0x7fc59c0a6c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:58.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.181+0000 7fc5a467f640 1 -- 172.21.15.12:0/3769418429 shutdown_connections 2024-09-11T01:31:58.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.181+0000 7fc5a467f640 1 -- 172.21.15.12:0/3769418429 wait complete. 2024-09-11T01:31:58.201 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:31:58.201 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:31:58.202 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:31:58.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.310+0000 7f1c171b4640 10 monclient: get_monmap_and_config 2024-09-11T01:31:58.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.310+0000 7f1c171b4640 10 monclient: build_initial_monmap 2024-09-11T01:31:58.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.310+0000 7f1c171b4640 10 monclient: monmap: 2024-09-11T01:31:58.311 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:58.312 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:58.312 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:31:58.310844+0000 2024-09-11T01:31:58.312 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:31:58.310844+0000 2024-09-11T01:31:58.312 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:58.312 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:58.312 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:31:58.312 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:31:58.312 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:31:58.312 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:31:58.312 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:31:58.312 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:58.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.310+0000 7f1c171b4640 1 Processor -- start 2024-09-11T01:31:58.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.310+0000 7f1c171b4640 1 -- start start 2024-09-11T01:31:58.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.310+0000 7f1c171b4640 10 monclient: init 2024-09-11T01:31:58.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c171b4640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:58.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c171b4640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-11T01:31:58.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c171b4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1c10137e50 addr v1:172.21.15.71:6789/0 2024-09-11T01:31:58.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c171b4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1c10138230 0x7f1c10138640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:58.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c171b4640 10 monclient(hunting): picked mon.noname-a con 0x7f1c10138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:31:58.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c171b4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c1013d380 0x7f1c1013f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:58.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c171b4640 10 monclient(hunting): picked mon.noname-b con 0x7f1c1013d380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:58.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c171b4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c100a6580 con 0x7f1c10137e50 2024-09-11T01:31:58.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c171b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:58.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c171b4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f1c100a6310 con 0x7f1c10138230 2024-09-11T01:31:58.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c171b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:58.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c171b4640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1c1009bb90 con 0x7f1c1013d380 2024-09-11T01:31:58.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c171b4640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:58.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c171b4640 10 monclient(hunting): authenticate will time out at 924.753518s 2024-09-11T01:31:58.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c07fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1c10138230 0x7f1c10138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:58.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c15751640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c1013d380 0x7f1c1013f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:58.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c07fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1c10138230 0x7f1c10138640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50474/0 (socket says 172.21.15.12:50474) 2024-09-11T01:31:58.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c07fff640 1 -- 172.21.15.12:0/546240961 learned_addr learned my addr 172.21.15.12:0/546240961 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:58.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c07fff640 10 monclient(hunting): get_auth_request con 0x7f1c10138230 auth_method 0 2024-09-11T01:31:58.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c15751640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c1013d380 0x7f1c1013f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:48256/0 (socket says 172.21.15.12:48256) 2024-09-11T01:31:58.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:58.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c07fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:58.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:58.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c15751640 10 monclient(hunting): get_auth_request con 0x7f1c1013d380 auth_method 0 2024-09-11T01:31:58.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c15751640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:58.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c15751640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:58.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c15751640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:58.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:58.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:58.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:58.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c15751640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:58.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c15751640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:58.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c15751640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:58.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c07fff640 10 monclient(hunting): handle_auth_done global_id 24225 payload 274 2024-09-11T01:31:58.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c07fff640 1 -- 172.21.15.12:0/546240961 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c1013d380 msgr2=0x7f1c1013f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:58.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c07fff640 1 --2- 172.21.15.12:0/546240961 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c1013d380 0x7f1c1013f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:58.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c07fff640 1 -- 172.21.15.12:0/546240961 >> v1:172.21.15.71:6789/0 conn(0x7f1c10137e50 legacy=0x7f1c1009d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:58.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c07fff640 10 monclient: _finish_hunting 0 2024-09-11T01:31:58.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c07fff640 1 monclient: found mon.noname-a 2024-09-11T01:31:58.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:58.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c07fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:31:58.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.311+0000 7f1c07fff640 1 -- 172.21.15.12:0/546240961 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c1009c780 con 0x7f1c10138230 2024-09-11T01:31:58.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.312+0000 7f1c15751640 1 --2- 172.21.15.12:0/546240961 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c1013d380 0x7f1c1013f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:31:58.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.314+0000 7f1c07fff640 1 -- 172.21.15.12:0/546240961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c10138230 msgr2=0x7f1c10138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:58.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.314+0000 7f1c07fff640 1 --2- 172.21.15.12:0/546240961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c10138230 0x7f1c10138640 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1c00009870 tx=0x7f1c0002a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62a5f7f2f1dc7107 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:58.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.314+0000 7f1c077fe640 1 -- 172.21.15.12:0/546240961 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c00037070 con 0x7f1c10138230 2024-09-11T01:31:58.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.315+0000 7f1c077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:58.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.315+0000 7f1c077fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:31:58.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.315+0000 7f1c077fe640 10 monclient: dump: 2024-09-11T01:31:58.328 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:58.328 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:58.328 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:58.328 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:58.329 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:58.329 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:58.329 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:58.329 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:58.329 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:58.329 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:58.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.315+0000 7f1c077fe640 10 monclient: _finish_auth 0 2024-09-11T01:31:58.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.315+0000 7f1c077fe640 10 monclient: _check_auth_tickets 2024-09-11T01:31:58.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.315+0000 7f1c077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:58.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.315+0000 7f1c077fe640 1 -- 172.21.15.12:0/546240961 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c00032440 con 0x7f1c10138230 2024-09-11T01:31:58.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.315+0000 7f1c171b4640 5 monclient: authenticate success, global_id 24225 2024-09-11T01:31:58.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.315+0000 7f1c171b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:31:58.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.315+0000 7f1c077fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:58.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.315+0000 7f1c171b4640 10 monclient: get_monmap_and_config success 2024-09-11T01:31:58.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.315+0000 7f1c077fe640 1 -- 172.21.15.12:0/546240961 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c00032810 con 0x7f1c10138230 2024-09-11T01:31:58.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.315+0000 7f1c171b4640 4 set_mon_vals no callback set 2024-09-11T01:31:58.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.315+0000 7f1c077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:58.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.315+0000 7f1c077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:58.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.315+0000 7f1c077fe640 10 monclient: dump: 2024-09-11T01:31:58.332 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:58.332 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:58.332 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:58.332 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:58.332 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:58.332 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:58.333 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:58.333 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:58.333 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:58.333 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:58.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.315+0000 7f1c171b4640 10 monclient: shutdown 2024-09-11T01:31:58.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.315+0000 7f1c171b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:58.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.315+0000 7f1c171b4640 1 -- 172.21.15.12:0/546240961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c10138230 msgr2=0x7f1c10138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:58.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.315+0000 7f1c171b4640 1 --2- 172.21.15.12:0/546240961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c10138230 0x7f1c10138640 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1c00009870 tx=0x7f1c0002a0a0 comp rx=0 tx=0).stop 2024-09-11T01:31:58.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.315+0000 7f1c171b4640 1 -- 172.21.15.12:0/546240961 shutdown_connections 2024-09-11T01:31:58.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.315+0000 7f1c171b4640 1 --2- 172.21.15.12:0/546240961 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c1013d380 0x7f1c1013f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:58.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.315+0000 7f1c171b4640 1 --2- 172.21.15.12:0/546240961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c10138230 0x7f1c10138640 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:58.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.315+0000 7f1c171b4640 1 -- 172.21.15.12:0/546240961 >> 172.21.15.12:0/546240961 conn(0x7f1c100a7020 msgr2=0x7f1c100a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:58.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.315+0000 7f1c171b4640 1 -- 172.21.15.12:0/546240961 shutdown_connections 2024-09-11T01:31:58.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.315+0000 7f1c171b4640 1 -- 172.21.15.12:0/546240961 wait complete. 2024-09-11T01:31:58.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.316+0000 7f1c171b4640 10 monclient: build_initial_monmap 2024-09-11T01:31:58.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.316+0000 7f1c171b4640 10 monclient: monmap: 2024-09-11T01:31:58.335 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:58.336 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:31:58.336 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:31:58.336 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:31:58.336 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:58.336 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:58.336 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:31:58.337 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:31:58.337 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:31:58.337 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:58.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.316+0000 7f1c171b4640 1 Processor -- start 2024-09-11T01:31:58.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.316+0000 7f1c171b4640 1 -- start start 2024-09-11T01:31:58.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.316+0000 7f1c171b4640 10 monclient: init 2024-09-11T01:31:58.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c171b4640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:58.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c171b4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:31:58.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c171b4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c10137e50 0x7f1c101ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:58.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c171b4640 10 monclient(hunting): picked mon.noname-a con 0x7f1c10137e50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:58.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c171b4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c10138230 0x7f1c101cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:58.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c171b4640 10 monclient(hunting): picked mon.noname-b con 0x7f1c10138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:31:58.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c171b4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c1013d380 0x7f1c100a1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:58.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c171b4640 10 monclient(hunting): picked mon.noname-c con 0x7f1c1013d380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:58.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c171b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:58.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c171b4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1006f890 con 0x7f1c10137e50 2024-09-11T01:31:58.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c171b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:58.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c171b4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1006f710 con 0x7f1c10138230 2024-09-11T01:31:58.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c171b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:58.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c171b4640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1c1013fcb0 con 0x7f1c1013d380 2024-09-11T01:31:58.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c171b4640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:58.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c171b4640 10 monclient(hunting): authenticate will time out at 924.759411s 2024-09-11T01:31:58.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c14f50640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c10137e50 0x7f1c101ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:58.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c14f50640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c10137e50 0x7f1c101ccf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50482/0 (socket says 172.21.15.12:50482) 2024-09-11T01:31:58.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c15751640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c1013d380 0x7f1c100a1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:58.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c14f50640 1 -- 172.21.15.12:0/3386419067 learned_addr learned my addr 172.21.15.12:0/3386419067 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:58.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c14f50640 10 monclient(hunting): get_auth_request con 0x7f1c10137e50 auth_method 0 2024-09-11T01:31:58.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c14f50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:58.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c14f50640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:58.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c14f50640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:58.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c15751640 10 monclient(hunting): get_auth_request con 0x7f1c1013d380 auth_method 0 2024-09-11T01:31:58.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c15751640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:58.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c15751640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:58.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c15751640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:58.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c07fff640 1 --2- 172.21.15.12:0/3386419067 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c10138230 0x7f1c101cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:58.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c14f50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:58.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c14f50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:58.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c14f50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:58.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c07fff640 10 monclient(hunting): get_auth_request con 0x7f1c10138230 auth_method 0 2024-09-11T01:31:58.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:58.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c07fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:58.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:58.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c15751640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:58.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c15751640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:58.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c15751640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:58.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:58.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:58.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:58.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c14f50640 10 monclient(hunting): handle_auth_done global_id 24228 payload 824 2024-09-11T01:31:58.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c14f50640 1 -- 172.21.15.12:0/3386419067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c1013d380 msgr2=0x7f1c100a1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:58.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c14f50640 1 --2- 172.21.15.12:0/3386419067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c1013d380 0x7f1c100a1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:58.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c14f50640 1 -- 172.21.15.12:0/3386419067 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c10138230 msgr2=0x7f1c101cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:58.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c14f50640 1 --2- 172.21.15.12:0/3386419067 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c10138230 0x7f1c101cd6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:58.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c14f50640 10 monclient: _finish_hunting 0 2024-09-11T01:31:58.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c14f50640 1 monclient: found mon.noname-a 2024-09-11T01:31:58.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c14f50640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:58.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c14f50640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:31:58.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.317+0000 7f1c14f50640 1 -- 172.21.15.12:0/3386419067 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c10121940 con 0x7f1c10137e50 2024-09-11T01:31:58.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.320+0000 7f1c14f50640 1 -- 172.21.15.12:0/3386419067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c10137e50 msgr2=0x7f1c101ccf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:58.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.320+0000 7f1c14f50640 1 --2- 172.21.15.12:0/3386419067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c10137e50 0x7f1c101ccf80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c00c430 tx=0x7f1c0c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:58.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.320+0000 7f1c057fa640 1 -- 172.21.15.12:0/3386419067 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c0119a0 con 0x7f1c10137e50 2024-09-11T01:31:58.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.320+0000 7f1c057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:58.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.320+0000 7f1c057fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:31:58.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.320+0000 7f1c057fa640 10 monclient: dump: 2024-09-11T01:31:58.343 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:58.343 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:58.343 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:58.343 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:58.343 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:58.343 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:58.343 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:58.344 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:58.344 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:58.344 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:58.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.320+0000 7f1c057fa640 10 monclient: _finish_auth 0 2024-09-11T01:31:58.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.320+0000 7f1c057fa640 10 monclient: _check_auth_tickets 2024-09-11T01:31:58.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.320+0000 7f1c057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:58.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.320+0000 7f1c057fa640 1 -- 172.21.15.12:0/3386419067 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c0c017070 con 0x7f1c10137e50 2024-09-11T01:31:58.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.320+0000 7f1c171b4640 5 monclient: authenticate success, global_id 24228 2024-09-11T01:31:58.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.320+0000 7f1c171b4640 10 monclient: _renew_subs 2024-09-11T01:31:58.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.320+0000 7f1c171b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:58.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.320+0000 7f1c171b4640 1 -- 172.21.15.12:0/3386419067 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c1013fa00 con 0x7f1c10137e50 2024-09-11T01:31:58.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.320+0000 7f1c057fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:58.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.320+0000 7f1c057fa640 1 -- 172.21.15.12:0/3386419067 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c011d80 con 0x7f1c10137e50 2024-09-11T01:31:58.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.320+0000 7f1c057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:58.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.320+0000 7f1c057fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:58.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.320+0000 7f1c057fa640 10 monclient: dump: 2024-09-11T01:31:58.345 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:58.345 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:58.345 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:58.345 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:58.345 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:58.345 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:58.345 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:58.345 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:58.345 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:58.346 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:58.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.320+0000 7f1c171b4640 10 monclient: _renew_subs 2024-09-11T01:31:58.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.320+0000 7f1c171b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:58.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.320+0000 7f1c171b4640 1 -- 172.21.15.12:0/3386419067 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c100a41b0 con 0x7f1c10137e50 2024-09-11T01:31:58.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.320+0000 7f1c06ffd640 4 set_mon_vals no callback set 2024-09-11T01:31:58.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.321+0000 7f1c057fa640 1 -- 172.21.15.12:0/3386419067 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f1c0c01c460 con 0x7f1c10137e50 2024-09-11T01:31:58.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.321+0000 7f1c171b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:58.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.321+0000 7f1c171b4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:58.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.321+0000 7f1c171b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:58.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.321+0000 7f1c057fa640 1 --2- 172.21.15.12:0/3386419067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1bfc03e210 0x7f1bfc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:58.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.321+0000 7f1c171b4640 1 -- 172.21.15.12:0/3386419067 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c10138640 con 0x7f1c10137e50 2024-09-11T01:31:58.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.321+0000 7f1c057fa640 1 -- 172.21.15.12:0/3386419067 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 4767+0+0 (secure 0 0 0) 0x7f1c0c052b40 con 0x7f1c10137e50 2024-09-11T01:31:58.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.321+0000 7f1c07fff640 1 --2- 172.21.15.12:0/3386419067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1bfc03e210 0x7f1bfc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:58.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.321+0000 7f1c07fff640 10 monclient: get_auth_request con 0x7f1bfc03e210 auth_method 0 2024-09-11T01:31:58.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.324+0000 7f1c07fff640 1 --2- 172.21.15.12:0/3386419067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1bfc03e210 0x7f1bfc0406d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1c100ab580 tx=0x7f1c0002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:58.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.326+0000 7f1c057fa640 1 -- 172.21.15.12:0/3386419067 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1c0c0224a0 con 0x7f1c10137e50 2024-09-11T01:31:58.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.326+0000 7f1c057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:58.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.326+0000 7f1c057fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:31:58.352 INFO:tasks.rados.rados.0.smithi071.stdout:452: done (1 left) 2024-09-11T01:31:58.352 INFO:tasks.rados.rados.0.smithi071.stdout:453: read oid 32 snap 39 2024-09-11T01:31:58.352 INFO:tasks.rados.rados.0.smithi071.stdout:453: expect deleted 2024-09-11T01:31:58.352 INFO:tasks.rados.rados.0.smithi071.stdout:454: copy_from oid 50 from oid 5 current snap is 40 2024-09-11T01:31:58.353 INFO:tasks.rados.rados.0.smithi071.stdout:451: finishing write tid 1 to smithi07134777-28 2024-09-11T01:31:58.353 INFO:tasks.rados.rados.0.smithi071.stdout:451: finishing write tid 2 to smithi07134777-28 2024-09-11T01:31:58.353 INFO:tasks.rados.rados.0.smithi071.stdout:451: finishing write tid 3 to smithi07134777-28 2024-09-11T01:31:58.353 INFO:tasks.rados.rados.0.smithi071.stdout:451: finishing write tid 4 to smithi07134777-28 2024-09-11T01:31:58.353 INFO:tasks.rados.rados.0.smithi071.stdout:451: finishing write tid 5 to smithi07134777-28 2024-09-11T01:31:58.353 INFO:tasks.rados.rados.0.smithi071.stdout:451: oid 28 updating version 0 to 165 2024-09-11T01:31:58.354 INFO:tasks.rados.rados.0.smithi071.stdout:451: oid 28 updating version 165 to 167 2024-09-11T01:31:58.354 INFO:tasks.rados.rados.0.smithi071.stdout:451: oid 28 version 167 is already newer than 164 2024-09-11T01:31:58.354 INFO:tasks.rados.rados.0.smithi071.stdout:451: oid 28 version 167 is already newer than 166 2024-09-11T01:31:58.354 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 167 (ObjNum 180 snap 40 seq_num 180) dirty exists 2024-09-11T01:31:58.354 INFO:tasks.rados.rados.0.smithi071.stdout:451: left oid 28 (ObjNum 180 snap 40 seq_num 180) 2024-09-11T01:31:58.355 INFO:tasks.rados.rados.0.smithi071.stdout:451: done (2 left) 2024-09-11T01:31:58.355 INFO:tasks.rados.rados.0.smithi071.stdout:455: delete oid 45 current snap is 40 2024-09-11T01:31:58.355 INFO:tasks.rados.rados.0.smithi071.stdout:454: finishing copy_from to smithi07134777-50 2024-09-11T01:31:58.355 INFO:tasks.rados.rados.0.smithi071.stdout:454: got expected ENOENT (src dne) 2024-09-11T01:31:58.355 INFO:tasks.rados.rados.0.smithi071.stdout:454: finishing copy_from racing read to smithi07134777-50 2024-09-11T01:31:58.356 INFO:tasks.rados.rados.0.smithi071.stdout:453: done (2 left) 2024-09-11T01:31:58.356 INFO:tasks.rados.rados.0.smithi071.stdout:454: done (1 left) 2024-09-11T01:31:58.356 INFO:tasks.rados.rados.0.smithi071.stdout:455: done (0 left) 2024-09-11T01:31:58.356 INFO:tasks.rados.rados.0.smithi071.stdout:456: snap_create 2024-09-11T01:31:58.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.597+0000 7f1c171b4640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:31:58.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.597+0000 7f1c171b4640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:31:58.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.597+0000 7f1c171b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:58.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.597+0000 7f1c171b4640 1 -- 172.21.15.12:0/3386419067 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f1c101215c0 con 0x7f1c10137e50 2024-09-11T01:31:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.598+0000 7f1c057fa640 1 -- 172.21.15.12:0/3386419067 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f1c0c012420 con 0x7f1c10137e50 2024-09-11T01:31:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.598+0000 7f1c057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:31:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.598+0000 7f1c057fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:31:58.601 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.598+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:58.602 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.598+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=11,kv=12,logm=65} crc {auth=3118786053,config=2586702516,health=548468587,kv=3263697421,logm=3810800558}) 2024-09-11T01:31:58.602 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.598+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=12,kv=12,logm=64} crc {auth=3118786053,config=575665002,health=3897446751,kv=3263697420,logm=3989975897}) 2024-09-11T01:31:58.602 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.598+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:58.602 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.598+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=11,kv=12,logm=65} crc {auth=3118786053,config=2586702516,health=548468587,kv=3263697421,logm=3810800558}) 2024-09-11T01:31:58.602 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.598+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=12,kv=11,logm=65} crc {auth=3118786053,config=2586702516,health=3897446751,kv=2372998920,logm=2471451895}) 2024-09-11T01:31:58.602 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.599+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:58.602 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.599+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=866236779}) 2024-09-11T01:31:58.602 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.599+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=26795516}) 2024-09-11T01:31:58.602 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.599+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:58.603 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.599+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=866236779}) 2024-09-11T01:31:58.603 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.599+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1386941659}) 2024-09-11T01:31:58.604 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.600+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:58.604 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.600+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1044403370}) 2024-09-11T01:31:58.604 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.600+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=248834829}) 2024-09-11T01:31:58.605 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.600+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:58.605 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.600+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1044403370}) 2024-09-11T01:31:58.605 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.600+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=787580863}) 2024-09-11T01:31:58.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.600+0000 7f1c171b4640 1 -- 172.21.15.12:0/3386419067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1bfc03e210 msgr2=0x7f1bfc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:58.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.600+0000 7f1c171b4640 1 --2- 172.21.15.12:0/3386419067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1bfc03e210 0x7f1bfc0406d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1c100ab580 tx=0x7f1c0002e000 comp rx=0 tx=0).stop 2024-09-11T01:31:58.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.600+0000 7f1c171b4640 10 monclient: shutdown 2024-09-11T01:31:58.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.601+0000 7f1c171b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:58.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.601+0000 7f1c171b4640 1 -- 172.21.15.12:0/3386419067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c10137e50 msgr2=0x7f1c101ccf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:58.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.601+0000 7f1c171b4640 1 --2- 172.21.15.12:0/3386419067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c10137e50 0x7f1c101ccf80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c00c430 tx=0x7f1c0c00c830 comp rx=0 tx=0).stop 2024-09-11T01:31:58.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.601+0000 7f1c171b4640 1 -- 172.21.15.12:0/3386419067 shutdown_connections 2024-09-11T01:31:58.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.601+0000 7f1c171b4640 1 --2- 172.21.15.12:0/3386419067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c1013d380 0x7f1c100a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:58.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.601+0000 7f1c171b4640 1 --2- 172.21.15.12:0/3386419067 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c10138230 0x7f1c101cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:58.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.601+0000 7f1c171b4640 1 --2- 172.21.15.12:0/3386419067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1bfc03e210 0x7f1bfc0406d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:58.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.601+0000 7f1c171b4640 1 --2- 172.21.15.12:0/3386419067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c10137e50 0x7f1c101ccf80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:58.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.601+0000 7f1c171b4640 1 -- 172.21.15.12:0/3386419067 >> 172.21.15.12:0/3386419067 conn(0x7f1c100a7020 msgr2=0x7f1c1013e9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:58.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.601+0000 7f1c171b4640 1 -- 172.21.15.12:0/3386419067 shutdown_connections 2024-09-11T01:31:58.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:58.601+0000 7f1c171b4640 1 -- 172.21.15.12:0/3386419067 wait complete. 2024-09-11T01:31:58.609 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.601+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:58.610 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.601+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077749561}) 2024-09-11T01:31:58.610 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.602+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=481229452}) 2024-09-11T01:31:58.610 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.602+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:58.610 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.602+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077749561}) 2024-09-11T01:31:58.610 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.602+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=308808133}) 2024-09-11T01:31:58.610 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.602+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:58.610 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.602+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3402291417}) 2024-09-11T01:31:58.610 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.602+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3296043134}) 2024-09-11T01:31:58.610 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.602+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:58.610 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.602+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3402291417}) 2024-09-11T01:31:58.610 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.602+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1974338444}) 2024-09-11T01:31:58.610 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.603+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:58.610 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.603+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1910853640}) 2024-09-11T01:31:58.611 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.603+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2583620496}) 2024-09-11T01:31:58.611 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.603+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:58.611 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.603+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1910853640}) 2024-09-11T01:31:58.611 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.603+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=722516838}) 2024-09-11T01:31:58.611 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.604+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:58.611 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.604+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2923841231}) 2024-09-11T01:31:58.611 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.604+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2515001700}) 2024-09-11T01:31:58.611 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.604+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:58.611 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.604+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2923841231}) 2024-09-11T01:31:58.611 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.604+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2052559753}) 2024-09-11T01:31:58.611 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.605+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:58.612 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.605+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=50,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=1} crc {logm=1805356864,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1624919405,mon_config_key=2028089358,monmap=2933275274,nvmeofgw=598337686,osd_metadata=642414809,osd_pg_creating=2695838175,osd_snap=3718248}) 2024-09-11T01:31:58.612 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.605+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=51,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=3} crc {logm=428580608,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1857512878,mon_config_key=309786242,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2609798099}) 2024-09-11T01:31:58.612 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.605+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:58.612 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.605+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=50,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=1} crc {logm=1805356864,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1624919405,mon_config_key=2028089358,monmap=2933275274,nvmeofgw=598337686,osd_metadata=642414809,osd_pg_creating=2695838175,osd_snap=3718248}) 2024-09-11T01:31:58.612 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.605+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=47,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=6} crc {logm=3072682487,mdsmap=3760429986,mgr=49723171,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1189456763,mon_config_key=1992046156,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2406434821,osd_pg_creating=2695838175,osd_snap=1703336224}) 2024-09-11T01:31:58.612 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.606+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:58.612 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.606+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=16,osdmap=84} crc {osd_snap=641912761,osdmap=2969486770}) 2024-09-11T01:31:58.613 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.606+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=16,osdmap=84} crc {osd_snap=641912761,osdmap=3292582844}) 2024-09-11T01:31:58.613 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.606+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:58.613 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.606+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=16,osdmap=84} crc {osd_snap=641912761,osdmap=2969486770}) 2024-09-11T01:31:58.613 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.606+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=16,osdmap=84} crc {osd_snap=641912761,osdmap=2612118587}) 2024-09-11T01:31:58.613 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.609+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:58.613 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.609+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=76} crc {osdmap=2910325535}) 2024-09-11T01:31:58.613 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.609+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=80} crc {osdmap=849807879}) 2024-09-11T01:31:58.613 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.609+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:31:58.614 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.609+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=76} crc {osdmap=2910325535}) 2024-09-11T01:31:58.614 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:31:58.609+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=78} crc {osdmap=916808456}) 2024-09-11T01:31:58.614 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:31:59.343 INFO:tasks.rados.rados.0.smithi071.stdout:456: done (0 left) 2024-09-11T01:31:59.343 INFO:tasks.rados.rados.0.smithi071.stdout:457: write (excl) oid 25 current snap is 41 2024-09-11T01:31:59.343 INFO:tasks.rados.rados.0.smithi071.stdout:457: seq_num 182 ranges {658874=771437,2083958=603055} 2024-09-11T01:31:59.350 INFO:tasks.rados.rados.0.smithi071.stdout:457: writing smithi07134777-25 from 658874 to 1430311 tid 1 2024-09-11T01:31:59.357 INFO:tasks.rados.rados.0.smithi071.stdout:457: writing smithi07134777-25 from 2083958 to 2687013 tid 2 2024-09-11T01:31:59.357 INFO:tasks.rados.rados.0.smithi071.stdout:458: write (excl) oid 4 current snap is 41 2024-09-11T01:31:59.357 INFO:tasks.rados.rados.0.smithi071.stdout:458: seq_num 183 ranges {675681=563039,1858793=746713,3025937=49744} 2024-09-11T01:31:59.364 INFO:tasks.rados.rados.0.smithi071.stdout:458: writing smithi07134777-4 from 675681 to 1238720 tid 1 2024-09-11T01:31:59.371 INFO:tasks.rados.rados.0.smithi071.stdout:458: writing smithi07134777-4 from 1858793 to 2605506 tid 2 2024-09-11T01:31:59.374 INFO:tasks.rados.rados.0.smithi071.stdout:458: writing smithi07134777-4 from 3025937 to 3075681 tid 3 2024-09-11T01:31:59.374 INFO:tasks.rados.rados.0.smithi071.stdout:459: snap_remove snap 39 2024-09-11T01:31:59.615 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:31:59.616 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:31:59.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.727+0000 7f7173b6b640 10 monclient: get_monmap_and_config 2024-09-11T01:31:59.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.727+0000 7f7173b6b640 10 monclient: build_initial_monmap 2024-09-11T01:31:59.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.727+0000 7f7173b6b640 10 monclient: monmap: 2024-09-11T01:31:59.766 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:59.766 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:59.766 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:31:59.728030+0000 2024-09-11T01:31:59.766 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:31:59.728030+0000 2024-09-11T01:31:59.767 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:59.767 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:59.767 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:31:59.767 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:31:59.767 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:31:59.767 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:31:59.767 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:31:59.767 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:59.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.727+0000 7f7173b6b640 1 Processor -- start 2024-09-11T01:31:59.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.728+0000 7f7173b6b640 1 -- start start 2024-09-11T01:31:59.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.728+0000 7f7173b6b640 10 monclient: init 2024-09-11T01:31:59.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.728+0000 7f7173b6b640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:59.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.728+0000 7f7173b6b640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-11T01:31:59.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.728+0000 7f7173b6b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f716c138230 0x7f716c138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:59.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.728+0000 7f7173b6b640 10 monclient(hunting): picked mon.noname-a con 0x7f716c138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:31:59.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.728+0000 7f7173b6b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f716c09d030 addr v1:172.21.15.71:6789/0 2024-09-11T01:31:59.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.728+0000 7f7173b6b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f716c09d460 addr v1:172.21.15.12:6789/0 2024-09-11T01:31:59.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.728+0000 7f7173b6b640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f716c0a5e40 con 0x7f716c09d460 2024-09-11T01:31:59.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.728+0000 7f7173b6b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f716c0a6e30 con 0x7f716c09d030 2024-09-11T01:31:59.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.728+0000 7f7173b6b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:59.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.728+0000 7f7173b6b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f716c09c9c0 con 0x7f716c138230 2024-09-11T01:31:59.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.728+0000 7f7173b6b640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:59.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.728+0000 7f7173b6b640 10 monclient(hunting): authenticate will time out at 926.170804s 2024-09-11T01:31:59.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.728+0000 7f7171907640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f716c138230 0x7f716c138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:59.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.728+0000 7f7171907640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f716c138230 0x7f716c138610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50498/0 (socket says 172.21.15.12:50498) 2024-09-11T01:31:59.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.728+0000 7f7171907640 1 -- 172.21.15.12:0/497648545 learned_addr learned my addr 172.21.15.12:0/497648545 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:59.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.728+0000 7f7171907640 10 monclient(hunting): get_auth_request con 0x7f716c138230 auth_method 0 2024-09-11T01:31:59.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.728+0000 7f7171907640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:59.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.728+0000 7f7171907640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:59.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.728+0000 7f7171907640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:59.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.728+0000 7f7172108640 1 --1- 172.21.15.12:0/497648545 >> v1:172.21.15.12:6789/0 conn(0x7f716c09d460 0x7f716c0a5730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50712/0 (socket says 172.21.15.12:50712) 2024-09-11T01:31:59.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.729+0000 7f7171907640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:59.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.729+0000 7f7171907640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:59.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.729+0000 7f7171907640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.731+0000 7f7172108640 1 -- 172.21.15.12:0/497648545 >> v1:172.21.15.12:6789/0 conn(0x7f716c09d460 legacy=0x7f716c0a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.731+0000 7f7171907640 10 monclient(hunting): handle_auth_done global_id 24231 payload 274 2024-09-11T01:31:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.731+0000 7f7171907640 1 -- 172.21.15.12:0/497648545 >> v1:172.21.15.71:6789/0 conn(0x7f716c09d030 legacy=0x7f716c0a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:31:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.731+0000 7f7171907640 1 -- 172.21.15.12:0/497648545 >> v1:172.21.15.12:6789/0 conn(0x7f716c09d460 legacy=0x7f716c0a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.731+0000 7f7171907640 10 monclient: _finish_hunting 0 2024-09-11T01:31:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.731+0000 7f7171907640 1 monclient: found mon.noname-a 2024-09-11T01:31:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.731+0000 7f7171907640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.731+0000 7f7171907640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:31:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.731+0000 7f7171907640 1 -- 172.21.15.12:0/497648545 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f716c09ba20 con 0x7f716c138230 2024-09-11T01:31:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.734+0000 7f7171907640 1 -- 172.21.15.12:0/497648545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f716c138230 msgr2=0x7f716c138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.734+0000 7f7171907640 1 --2- 172.21.15.12:0/497648545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f716c138230 0x7f716c138610 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f71680098f0 tx=0x7f716802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67a2503083f5b211 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.734+0000 7f7170905640 1 -- 172.21.15.12:0/497648545 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7168037070 con 0x7f716c138230 2024-09-11T01:31:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.734+0000 7f7170905640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:59.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.734+0000 7f7170905640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:31:59.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.734+0000 7f7170905640 10 monclient: dump: 2024-09-11T01:31:59.778 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:59.778 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:59.778 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:59.778 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:59.778 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:59.778 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:59.778 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:59.778 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:59.778 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:59.778 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.734+0000 7f7170905640 10 monclient: _finish_auth 0 2024-09-11T01:31:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.734+0000 7f7170905640 10 monclient: _check_auth_tickets 2024-09-11T01:31:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.734+0000 7f7170905640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.734+0000 7f7170905640 1 -- 172.21.15.12:0/497648545 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7168032640 con 0x7f716c138230 2024-09-11T01:31:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.734+0000 7f7170905640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.734+0000 7f7170905640 1 -- 172.21.15.12:0/497648545 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7168032a10 con 0x7f716c138230 2024-09-11T01:31:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.734+0000 7f7173b6b640 5 monclient: authenticate success, global_id 24231 2024-09-11T01:31:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.734+0000 7f7173b6b640 10 monclient: get_monmap_and_config success 2024-09-11T01:31:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.734+0000 7f7173b6b640 4 set_mon_vals no callback set 2024-09-11T01:31:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.734+0000 7f7170905640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.734+0000 7f7170905640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.734+0000 7f7170905640 10 monclient: dump: 2024-09-11T01:31:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:59.780 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.734+0000 7f7173b6b640 10 monclient: shutdown 2024-09-11T01:31:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.734+0000 7f7173b6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:31:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.734+0000 7f7173b6b640 1 -- 172.21.15.12:0/497648545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f716c138230 msgr2=0x7f716c138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.734+0000 7f7173b6b640 1 --2- 172.21.15.12:0/497648545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f716c138230 0x7f716c138610 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f71680098f0 tx=0x7f716802a1c0 comp rx=0 tx=0).stop 2024-09-11T01:31:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.735+0000 7f7173b6b640 1 -- 172.21.15.12:0/497648545 shutdown_connections 2024-09-11T01:31:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.735+0000 7f7173b6b640 1 --2- 172.21.15.12:0/497648545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f716c138230 0x7f716c138610 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.735+0000 7f7173b6b640 1 -- 172.21.15.12:0/497648545 >> 172.21.15.12:0/497648545 conn(0x7f716c0a7120 msgr2=0x7f716c0a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:31:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.735+0000 7f7173b6b640 1 -- 172.21.15.12:0/497648545 shutdown_connections 2024-09-11T01:31:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.735+0000 7f7173b6b640 1 -- 172.21.15.12:0/497648545 wait complete. 2024-09-11T01:31:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.735+0000 7f7173b6b640 10 monclient: build_initial_monmap 2024-09-11T01:31:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.735+0000 7f7173b6b640 10 monclient: monmap: 2024-09-11T01:31:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:31:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:31:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:31:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:31:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:31:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:31:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:31:59.782 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:31:59.782 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:59.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.735+0000 7f7173b6b640 1 Processor -- start 2024-09-11T01:31:59.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7173b6b640 1 -- start start 2024-09-11T01:31:59.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7173b6b640 10 monclient: init 2024-09-11T01:31:59.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7173b6b640 10 monclient: _reopen_session rank -1 2024-09-11T01:31:59.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7173b6b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:31:59.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7173b6b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f716c09d030 0x7f716c133000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:59.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7173b6b640 10 monclient(hunting): picked mon.noname-b con 0x7f716c09d030 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:31:59.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7173b6b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f716c09d460 0x7f716c133540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:59.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7173b6b640 10 monclient(hunting): picked mon.noname-a con 0x7f716c09d460 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:31:59.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7173b6b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f716c138230 0x7f716c138e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:59.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7173b6b640 10 monclient(hunting): picked mon.noname-c con 0x7f716c138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:31:59.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7173b6b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:59.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7173b6b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f716c06f7f0 con 0x7f716c09d460 2024-09-11T01:31:59.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7173b6b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:59.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7173b6b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f716c0a6730 con 0x7f716c09d030 2024-09-11T01:31:59.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7173b6b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:31:59.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7173b6b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f716c06f970 con 0x7f716c138230 2024-09-11T01:31:59.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7173b6b640 10 monclient(hunting): _renew_subs 2024-09-11T01:31:59.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7173b6b640 10 monclient(hunting): authenticate will time out at 926.178534s 2024-09-11T01:31:59.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7171106640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f716c09d460 0x7f716c133540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:59.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7171106640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f716c09d460 0x7f716c133540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50510/0 (socket says 172.21.15.12:50510) 2024-09-11T01:31:59.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7171106640 1 -- 172.21.15.12:0/873140831 learned_addr learned my addr 172.21.15.12:0/873140831 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:31:59.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7171106640 10 monclient(hunting): get_auth_request con 0x7f716c09d460 auth_method 0 2024-09-11T01:31:59.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7171106640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:59.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7171106640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7171106640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7172108640 1 --2- 172.21.15.12:0/873140831 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f716c138230 0x7f716c138e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7171106640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7171106640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7171106640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7172108640 10 monclient(hunting): get_auth_request con 0x7f716c138230 auth_method 0 2024-09-11T01:31:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7172108640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7172108640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7172108640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7171907640 1 --2- 172.21.15.12:0/873140831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f716c09d030 0x7f716c133000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7171907640 10 monclient(hunting): get_auth_request con 0x7f716c09d030 auth_method 0 2024-09-11T01:31:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7171907640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:31:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7171907640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:31:59.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7171907640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:31:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7172108640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:31:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7172108640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:31:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7172108640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:31:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7171106640 10 monclient(hunting): handle_auth_done global_id 24234 payload 824 2024-09-11T01:31:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7171106640 1 -- 172.21.15.12:0/873140831 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f716c138230 msgr2=0x7f716c138e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7171106640 1 --2- 172.21.15.12:0/873140831 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f716c138230 0x7f716c138e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7171106640 1 -- 172.21.15.12:0/873140831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f716c09d030 msgr2=0x7f716c133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:31:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.736+0000 7f7171106640 1 --2- 172.21.15.12:0/873140831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f716c09d030 0x7f716c133000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:31:59.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.737+0000 7f7171106640 10 monclient: _finish_hunting 0 2024-09-11T01:31:59.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.737+0000 7f7171106640 1 monclient: found mon.noname-a 2024-09-11T01:31:59.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.737+0000 7f7171106640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:31:59.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.737+0000 7f7171106640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:31:59.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.737+0000 7f7171106640 1 -- 172.21.15.12:0/873140831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f716c13a6d0 con 0x7f716c09d460 2024-09-11T01:31:59.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.737+0000 7f7171907640 1 --2- 172.21.15.12:0/873140831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f716c09d030 0x7f716c133000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:31:59.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.739+0000 7f7171106640 1 -- 172.21.15.12:0/873140831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f716c09d460 msgr2=0x7f716c133540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:31:59.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.739+0000 7f7171106640 1 --2- 172.21.15.12:0/873140831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f716c09d460 0x7f716c133540 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f715c00c430 tx=0x7f715c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:59.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.739+0000 7f71627fc640 1 -- 172.21.15.12:0/873140831 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c0119a0 con 0x7f716c09d460 2024-09-11T01:31:59.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.739+0000 7f71627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:59.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.739+0000 7f71627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:31:59.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.739+0000 7f71627fc640 10 monclient: dump: 2024-09-11T01:31:59.787 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:59.787 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:59.787 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:59.787 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:59.787 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:59.787 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:59.787 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:59.787 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:59.787 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:59.787 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:59.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.739+0000 7f71627fc640 10 monclient: _finish_auth 0 2024-09-11T01:31:59.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.739+0000 7f71627fc640 10 monclient: _check_auth_tickets 2024-09-11T01:31:59.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.739+0000 7f71627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:31:59.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.739+0000 7f71627fc640 1 -- 172.21.15.12:0/873140831 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f715c017070 con 0x7f716c09d460 2024-09-11T01:31:59.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.739+0000 7f7173b6b640 5 monclient: authenticate success, global_id 24234 2024-09-11T01:31:59.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.739+0000 7f71627fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:31:59.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.739+0000 7f71627fc640 1 -- 172.21.15.12:0/873140831 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c011d80 con 0x7f716c09d460 2024-09-11T01:31:59.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.739+0000 7f7173b6b640 10 monclient: _renew_subs 2024-09-11T01:31:59.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.739+0000 7f7173b6b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:59.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.739+0000 7f7173b6b640 1 -- 172.21.15.12:0/873140831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f716c13c190 con 0x7f716c09d460 2024-09-11T01:31:59.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.739+0000 7f71627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:31:59.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.739+0000 7f71627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:31:59.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.739+0000 7f7163fff640 4 set_mon_vals no callback set 2024-09-11T01:31:59.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.739+0000 7f71627fc640 10 monclient: dump: 2024-09-11T01:31:59.789 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:31:59.789 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:31:59.789 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:59.789 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:31:59.789 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:31:59.789 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:31:59.789 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:31:59.789 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:31:59.789 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:31:59.789 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:31:59.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.739+0000 7f7173b6b640 10 monclient: _renew_subs 2024-09-11T01:31:59.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.739+0000 7f7173b6b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:59.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.739+0000 7f7173b6b640 1 -- 172.21.15.12:0/873140831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f716c13b230 con 0x7f716c09d460 2024-09-11T01:31:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.740+0000 7f7173b6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.740+0000 7f7173b6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.740+0000 7f7173b6b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:31:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.740+0000 7f7173b6b640 1 -- 172.21.15.12:0/873140831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f716c138610 con 0x7f716c09d460 2024-09-11T01:31:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.740+0000 7f71627fc640 1 -- 172.21.15.12:0/873140831 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f715c01c460 con 0x7f716c09d460 2024-09-11T01:31:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.740+0000 7f71627fc640 1 --2- 172.21.15.12:0/873140831 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f715003e210 0x7f71500406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:31:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.740+0000 7f71627fc640 1 -- 172.21.15.12:0/873140831 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 4739+0+0 (secure 0 0 0) 0x7f715c052bb0 con 0x7f716c09d460 2024-09-11T01:31:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.741+0000 7f7171907640 1 --2- 172.21.15.12:0/873140831 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f715003e210 0x7f71500406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:31:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.741+0000 7f7171907640 10 monclient: get_auth_request con 0x7f715003e210 auth_method 0 2024-09-11T01:31:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.743+0000 7f7171907640 1 --2- 172.21.15.12:0/873140831 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f715003e210 0x7f71500406d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f716c09bff0 tx=0x7f716802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:31:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.745+0000 7f71627fc640 1 -- 172.21.15.12:0/873140831 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f715c010cd0 con 0x7f716c09d460 2024-09-11T01:31:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.745+0000 7f71627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:31:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.745+0000 7f71627fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:32:00.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.999+0000 7f7173b6b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:32:00.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.999+0000 7f7173b6b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:32:00.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.999+0000 7f7173b6b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:00.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.999+0000 7f7173b6b640 1 -- 172.21.15.12:0/873140831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f716c13c990 con 0x7f716c09d460 2024-09-11T01:32:00.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.999+0000 7f71627fc640 1 -- 172.21.15.12:0/873140831 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f715c019cd0 con 0x7f716c09d460 2024-09-11T01:32:00.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.999+0000 7f71627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:32:00.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:31:59.999+0000 7f71627fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:32:00.002 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:32:00.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.002+0000 7f7173b6b640 1 -- 172.21.15.12:0/873140831 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f715003e210 msgr2=0x7f71500406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.002+0000 7f7173b6b640 1 --2- 172.21.15.12:0/873140831 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f715003e210 0x7f71500406d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f716c09bff0 tx=0x7f716802e000 comp rx=0 tx=0).stop 2024-09-11T01:32:00.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.002+0000 7f7173b6b640 10 monclient: shutdown 2024-09-11T01:32:00.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.002+0000 7f7173b6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:00.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.002+0000 7f7173b6b640 1 -- 172.21.15.12:0/873140831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f716c09d460 msgr2=0x7f716c133540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.002+0000 7f7173b6b640 1 --2- 172.21.15.12:0/873140831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f716c09d460 0x7f716c133540 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f715c00c430 tx=0x7f715c00c8f0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.002+0000 7f7173b6b640 1 -- 172.21.15.12:0/873140831 shutdown_connections 2024-09-11T01:32:00.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.002+0000 7f7173b6b640 1 --2- 172.21.15.12:0/873140831 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f716c138230 0x7f716c138e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.002+0000 7f7173b6b640 1 --2- 172.21.15.12:0/873140831 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f715003e210 0x7f71500406d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.002+0000 7f7173b6b640 1 --2- 172.21.15.12:0/873140831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f716c09d460 0x7f716c133540 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.002+0000 7f7173b6b640 1 --2- 172.21.15.12:0/873140831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f716c09d030 0x7f716c133000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.002+0000 7f7173b6b640 1 -- 172.21.15.12:0/873140831 >> 172.21.15.12:0/873140831 conn(0x7f716c0a7120 msgr2=0x7f716c0a3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:00.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.002+0000 7f7173b6b640 1 -- 172.21.15.12:0/873140831 shutdown_connections 2024-09-11T01:32:00.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.002+0000 7f7173b6b640 1 -- 172.21.15.12:0/873140831 wait complete. 2024-09-11T01:32:00.017 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:32:00.017 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:32:00.017 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:32:00.017 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:32:00.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.118+0000 7f6847672640 10 monclient: get_monmap_and_config 2024-09-11T01:32:00.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.118+0000 7f6847672640 10 monclient: build_initial_monmap 2024-09-11T01:32:00.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.118+0000 7f6847672640 10 monclient: monmap: 2024-09-11T01:32:00.120 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:00.120 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.120 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:32:00.119266+0000 2024-09-11T01:32:00.120 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:32:00.119266+0000 2024-09-11T01:32:00.121 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:00.121 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.121 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:32:00.121 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:32:00.121 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:32:00.121 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:32:00.121 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:32:00.121 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.119+0000 7f6847672640 1 Processor -- start 2024-09-11T01:32:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.119+0000 7f6847672640 1 -- start start 2024-09-11T01:32:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.119+0000 7f6847672640 10 monclient: init 2024-09-11T01:32:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.119+0000 7f6847672640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.119+0000 7f6847672640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-11T01:32:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.119+0000 7f6847672640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68400a1380 0x7f68400a1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.119+0000 7f6847672640 10 monclient(hunting): picked mon.noname-b con 0x7f68400a1380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.119+0000 7f6847672640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f68400a0370 0x7f6840135ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.119+0000 7f6847672640 10 monclient(hunting): picked mon.noname-a con 0x7f68400a0370 addr v2:172.21.15.12:3300/0 2024-09-11T01:32:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.119+0000 7f6847672640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f68400a0c90 0x7f6840136020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.119+0000 7f6847672640 10 monclient(hunting): picked mon.noname-c con 0x7f68400a0c90 addr v2:172.21.15.71:3300/0 2024-09-11T01:32:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.119+0000 7f6847672640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.119+0000 7f6847672640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f68400a1ca0 con 0x7f68400a0370 2024-09-11T01:32:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.119+0000 7f6847672640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.119+0000 7f6847672640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f68400a21c0 con 0x7f68400a0c90 2024-09-11T01:32:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.119+0000 7f6847672640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.119+0000 7f6847672640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f68400a24a0 con 0x7f68400a1380 2024-09-11T01:32:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.119+0000 7f6847672640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.119+0000 7f6847672640 10 monclient(hunting): authenticate will time out at 926.561839s 2024-09-11T01:32:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.119+0000 7f684540e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68400a1380 0x7f68400a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.119+0000 7f6844c0d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f68400a0370 0x7f6840135ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.119+0000 7f684540e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68400a1380 0x7f68400a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:48276/0 (socket says 172.21.15.12:48276) 2024-09-11T01:32:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f684540e640 1 -- 172.21.15.12:0/2832206977 learned_addr learned my addr 172.21.15.12:0/2832206977 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f684540e640 10 monclient(hunting): get_auth_request con 0x7f68400a1380 auth_method 0 2024-09-11T01:32:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f684540e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f684540e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f684540e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f6844c0d640 10 monclient(hunting): get_auth_request con 0x7f68400a0370 auth_method 0 2024-09-11T01:32:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f6844c0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f6844c0d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f6844c0d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f684540e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f684540e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f684540e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f6844c0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f6844c0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f6845c0f640 1 --2- 172.21.15.12:0/2832206977 >> v2:172.21.15.71:3300/0 conn(0x7f68400a0c90 0x7f6840136020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f6844c0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f6845c0f640 10 monclient(hunting): get_auth_request con 0x7f68400a0c90 auth_method 0 2024-09-11T01:32:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f6845c0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f6845c0f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f6845c0f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f6845c0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f6845c0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f6845c0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f6844c0d640 10 monclient(hunting): handle_auth_done global_id 24237 payload 274 2024-09-11T01:32:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f6844c0d640 1 -- 172.21.15.12:0/2832206977 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68400a1380 msgr2=0x7f68400a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f6844c0d640 1 --2- 172.21.15.12:0/2832206977 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68400a1380 0x7f68400a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f6844c0d640 1 -- 172.21.15.12:0/2832206977 >> v2:172.21.15.71:3300/0 conn(0x7f68400a0c90 msgr2=0x7f6840136020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f6844c0d640 1 --2- 172.21.15.12:0/2832206977 >> v2:172.21.15.71:3300/0 conn(0x7f68400a0c90 0x7f6840136020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f6844c0d640 10 monclient: _finish_hunting 0 2024-09-11T01:32:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f6844c0d640 1 monclient: found mon.noname-a 2024-09-11T01:32:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f6844c0d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f6844c0d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:32:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f6844c0d640 1 -- 172.21.15.12:0/2832206977 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68400a2780 con 0x7f68400a0370 2024-09-11T01:32:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.120+0000 7f684540e640 1 --2- 172.21.15.12:0/2832206977 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68400a1380 0x7f68400a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:32:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f6844c0d640 1 -- 172.21.15.12:0/2832206977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68400a0370 msgr2=0x7f6840135ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f6844c0d640 1 --2- 172.21.15.12:0/2832206977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68400a0370 0x7f6840135ae0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f68400a23d0 tx=0x7f683402a4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5929fb8741da6dba server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f683ffff640 1 -- 172.21.15.12:0/2832206977 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6834007950 con 0x7f68400a0370 2024-09-11T01:32:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f683ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f683ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:32:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f683ffff640 10 monclient: dump: 2024-09-11T01:32:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:00.138 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f683ffff640 10 monclient: _finish_auth 0 2024-09-11T01:32:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f683ffff640 10 monclient: _check_auth_tickets 2024-09-11T01:32:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f683ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f6847672640 5 monclient: authenticate success, global_id 24237 2024-09-11T01:32:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f683ffff640 1 -- 172.21.15.12:0/2832206977 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6834007af0 con 0x7f68400a0370 2024-09-11T01:32:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f6847672640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:32:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f683ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f6847672640 10 monclient: get_monmap_and_config success 2024-09-11T01:32:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f6847672640 4 set_mon_vals no callback set 2024-09-11T01:32:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f683ffff640 1 -- 172.21.15.12:0/2832206977 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6834031400 con 0x7f68400a0370 2024-09-11T01:32:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f683ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f683ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f683ffff640 10 monclient: dump: 2024-09-11T01:32:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:00.140 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f6847672640 10 monclient: shutdown 2024-09-11T01:32:00.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f6847672640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:00.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f6847672640 1 -- 172.21.15.12:0/2832206977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68400a0370 msgr2=0x7f6840135ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f6847672640 1 --2- 172.21.15.12:0/2832206977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68400a0370 0x7f6840135ae0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f68400a23d0 tx=0x7f683402a4b0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f6847672640 1 -- 172.21.15.12:0/2832206977 shutdown_connections 2024-09-11T01:32:00.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f6847672640 1 --2- 172.21.15.12:0/2832206977 >> v2:172.21.15.71:3300/0 conn(0x7f68400a0c90 0x7f6840136020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f6847672640 1 --2- 172.21.15.12:0/2832206977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68400a0370 0x7f6840135ae0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f6847672640 1 --2- 172.21.15.12:0/2832206977 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68400a1380 0x7f68400a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f6847672640 1 -- 172.21.15.12:0/2832206977 >> 172.21.15.12:0/2832206977 conn(0x7f68400a2f70 msgr2=0x7f68400a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:00.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.123+0000 7f6847672640 1 -- 172.21.15.12:0/2832206977 shutdown_connections 2024-09-11T01:32:00.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.124+0000 7f6847672640 1 -- 172.21.15.12:0/2832206977 wait complete. 2024-09-11T01:32:00.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.124+0000 7f6847672640 10 monclient: build_initial_monmap 2024-09-11T01:32:00.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.124+0000 7f6847672640 10 monclient: monmap: 2024-09-11T01:32:00.141 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:00.141 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:32:00.141 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:32:00.141 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:32:00.141 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:00.141 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.141 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:32:00.141 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:32:00.141 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:32:00.141 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.124+0000 7f6847672640 1 Processor -- start 2024-09-11T01:32:00.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.124+0000 7f6847672640 1 -- start start 2024-09-11T01:32:00.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.124+0000 7f6847672640 10 monclient: init 2024-09-11T01:32:00.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6847672640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:00.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6847672640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:32:00.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6847672640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68400a0370 0x7f68401ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6847672640 10 monclient(hunting): picked mon.noname-c con 0x7f68400a0370 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:00.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6847672640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68400a0c90 0x7f6840135080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6847672640 10 monclient(hunting): picked mon.noname-b con 0x7f68400a0c90 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:32:00.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6847672640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68400a1380 0x7f68401c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6847672640 10 monclient(hunting): picked mon.noname-a con 0x7f68400a1380 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:32:00.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6847672640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6847672640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f684009ce20 con 0x7f68400a1380 2024-09-11T01:32:00.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6847672640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6847672640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f684009cca0 con 0x7f68400a0c90 2024-09-11T01:32:00.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6847672640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6847672640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6840136560 con 0x7f68400a0370 2024-09-11T01:32:00.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6847672640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:00.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6847672640 10 monclient(hunting): authenticate will time out at 926.567517s 2024-09-11T01:32:00.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f684540e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68400a0370 0x7f68401ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6845c0f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68400a1380 0x7f68401c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f684540e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68400a0370 0x7f68401ccfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:48282/0 (socket says 172.21.15.12:48282) 2024-09-11T01:32:00.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f684540e640 1 -- 172.21.15.12:0/1863791216 learned_addr learned my addr 172.21.15.12:0/1863791216 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:00.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f684540e640 10 monclient(hunting): get_auth_request con 0x7f68400a0370 auth_method 0 2024-09-11T01:32:00.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f684540e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:00.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f684540e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f684540e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:00.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6845c0f640 10 monclient(hunting): get_auth_request con 0x7f68400a1380 auth_method 0 2024-09-11T01:32:00.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6845c0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:00.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6845c0f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6845c0f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:00.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f684540e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:00.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f684540e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:00.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f684540e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:00.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6845c0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:00.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6845c0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:00.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6845c0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:00.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6844c0d640 1 --2- 172.21.15.12:0/1863791216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68400a0c90 0x7f6840135080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6844c0d640 10 monclient(hunting): get_auth_request con 0x7f68400a0c90 auth_method 0 2024-09-11T01:32:00.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6844c0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:00.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6844c0d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6844c0d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:00.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6845c0f640 10 monclient(hunting): handle_auth_done global_id 24240 payload 824 2024-09-11T01:32:00.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6845c0f640 1 -- 172.21.15.12:0/1863791216 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68400a0370 msgr2=0x7f68401ccfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6845c0f640 1 --2- 172.21.15.12:0/1863791216 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68400a0370 0x7f68401ccfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6845c0f640 1 -- 172.21.15.12:0/1863791216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68400a0c90 msgr2=0x7f6840135080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6845c0f640 1 --2- 172.21.15.12:0/1863791216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68400a0c90 0x7f6840135080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6845c0f640 10 monclient: _finish_hunting 0 2024-09-11T01:32:00.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6845c0f640 1 monclient: found mon.noname-a 2024-09-11T01:32:00.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6845c0f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:00.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6845c0f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:32:00.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6845c0f640 1 -- 172.21.15.12:0/1863791216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f684009f140 con 0x7f68400a1380 2024-09-11T01:32:00.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f684540e640 1 --2- 172.21.15.12:0/1863791216 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68400a0370 0x7f68401ccfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:32:00.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.125+0000 7f6844c0d640 1 --2- 172.21.15.12:0/1863791216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68400a0c90 0x7f6840135080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:32:00.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.128+0000 7f6845c0f640 1 -- 172.21.15.12:0/1863791216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68400a1380 msgr2=0x7f68401c9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.128+0000 7f6845c0f640 1 --2- 172.21.15.12:0/1863791216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68400a1380 0x7f68401c9ab0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f683000e7c0 tx=0x7f683000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.128+0000 7f683dffb640 1 -- 172.21.15.12:0/1863791216 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6830017070 con 0x7f68400a1380 2024-09-11T01:32:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.128+0000 7f683dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.128+0000 7f683dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:32:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.128+0000 7f683dffb640 10 monclient: dump: 2024-09-11T01:32:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:00.147 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.128+0000 7f683dffb640 10 monclient: _finish_auth 0 2024-09-11T01:32:00.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.128+0000 7f683dffb640 10 monclient: _check_auth_tickets 2024-09-11T01:32:00.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.128+0000 7f683dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:00.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.128+0000 7f683dffb640 1 -- 172.21.15.12:0/1863791216 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f683000cb50 con 0x7f68400a1380 2024-09-11T01:32:00.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.128+0000 7f6847672640 5 monclient: authenticate success, global_id 24240 2024-09-11T01:32:00.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.128+0000 7f6847672640 10 monclient: _renew_subs 2024-09-11T01:32:00.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.128+0000 7f6847672640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:00.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.128+0000 7f6847672640 1 -- 172.21.15.12:0/1863791216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f684009ec70 con 0x7f68400a1380 2024-09-11T01:32:00.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.128+0000 7f683dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:00.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.128+0000 7f683dffb640 1 -- 172.21.15.12:0/1863791216 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6830013470 con 0x7f68400a1380 2024-09-11T01:32:00.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.128+0000 7f683dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:00.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.128+0000 7f683dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:00.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.128+0000 7f683dffb640 10 monclient: dump: 2024-09-11T01:32:00.148 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:00.148 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.148 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.148 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.148 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:00.148 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.148 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:00.148 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:00.148 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:00.148 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.128+0000 7f6847672640 10 monclient: _renew_subs 2024-09-11T01:32:00.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.128+0000 7f683f7fe640 4 set_mon_vals no callback set 2024-09-11T01:32:00.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.128+0000 7f6847672640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.128+0000 7f6847672640 1 -- 172.21.15.12:0/1863791216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f684009d480 con 0x7f68400a1380 2024-09-11T01:32:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.129+0000 7f6847672640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.129+0000 7f6847672640 1 --2- 172.21.15.12:0/1863791216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68401355c0 0x7f684002d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.129+0000 7f6847672640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.129+0000 7f6847672640 1 -- 172.21.15.12:0/1863791216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6840077300 con 0x7f68401355c0 2024-09-11T01:32:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.129+0000 7f684540e640 1 --2- 172.21.15.12:0/1863791216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68401355c0 0x7f684002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.129+0000 7f684540e640 10 monclient: get_auth_request con 0x7f68401355c0 auth_method 0 2024-09-11T01:32:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.129+0000 7f684540e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.129+0000 7f684540e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.129+0000 7f684540e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.129+0000 7f684540e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.129+0000 7f684540e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.129+0000 7f684540e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.129+0000 7f683dffb640 1 -- 172.21.15.12:0/1863791216 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f6830024060 con 0x7f68400a1380 2024-09-11T01:32:00.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.129+0000 7f684540e640 10 monclient(hunting): handle_auth_done global_id 24243 payload 824 2024-09-11T01:32:00.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.129+0000 7f684540e640 1 -- 172.21.15.12:0/1863791216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f68400a1760 con 0x7f68401355c0 2024-09-11T01:32:00.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.129+0000 7f683dffb640 1 --2- 172.21.15.12:0/1863791216 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f682c03e210 0x7f682c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.129+0000 7f683dffb640 1 -- 172.21.15.12:0/1863791216 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 4739+0+0 (secure 0 0 0) 0x7f6830052d10 con 0x7f68400a1380 2024-09-11T01:32:00.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.130+0000 7f6844c0d640 1 --2- 172.21.15.12:0/1863791216 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f682c03e210 0x7f682c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.130+0000 7f6844c0d640 10 monclient: get_auth_request con 0x7f682c03e210 auth_method 0 2024-09-11T01:32:00.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.131+0000 7f684540e640 1 -- 172.21.15.12:0/1863791216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68401355c0 msgr2=0x7f684002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:00.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.131+0000 7f684540e640 1 --2- 172.21.15.12:0/1863791216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68401355c0 0x7f684002d440 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6838009f20 tx=0x7f683800f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:00.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.132+0000 7f683dffb640 1 -- 172.21.15.12:0/1863791216 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6838004030 con 0x7f68401355c0 2024-09-11T01:32:00.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.132+0000 7f683dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:00.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.132+0000 7f683dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:00.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.132+0000 7f683dffb640 10 monclient: dump: 2024-09-11T01:32:00.150 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:00.151 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.151 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.151 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.151 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:00.151 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.151 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:00.151 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:00.151 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:00.151 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.132+0000 7f683dffb640 1 -- 172.21.15.12:0/1863791216 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6838004030 con 0x7f68401355c0 2024-09-11T01:32:00.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.132+0000 7f683dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:00.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.132+0000 7f683dffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:32:00.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.132+0000 7f683dffb640 1 -- 172.21.15.12:0/1863791216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68401355c0 msgr2=0x7f684002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.132+0000 7f683dffb640 1 --2- 172.21.15.12:0/1863791216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68401355c0 0x7f684002d440 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6838009f20 tx=0x7f683800f610 comp rx=0 tx=0).stop 2024-09-11T01:32:00.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.132+0000 7f683dffb640 1 -- 172.21.15.12:0/1863791216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68401355c0 msgr2=0x7f684002d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:32:00.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.132+0000 7f683dffb640 1 --2- 172.21.15.12:0/1863791216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68401355c0 0x7f684002d440 secure :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6838009f20 tx=0x7f683800f610 comp rx=0 tx=0).stop 2024-09-11T01:32:00.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.132+0000 7f6844c0d640 1 --2- 172.21.15.12:0/1863791216 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f682c03e210 0x7f682c0406d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6834031000 tx=0x7f683402f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.138+0000 7f6847672640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:32:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.138+0000 7f6847672640 1 --2- 172.21.15.12:0/1863791216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f684002d980 0x7f684002dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.138+0000 7f6847672640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.138+0000 7f6847672640 1 -- 172.21.15.12:0/1863791216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6840070310 con 0x7f684002d980 2024-09-11T01:32:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.138+0000 7f684540e640 1 --2- 172.21.15.12:0/1863791216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f684002d980 0x7f684002dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.138+0000 7f684540e640 10 monclient: get_auth_request con 0x7f684002d980 auth_method 0 2024-09-11T01:32:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.138+0000 7f684540e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.138+0000 7f684540e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.138+0000 7f684540e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.138+0000 7f684540e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.138+0000 7f684540e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.138+0000 7f684540e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.138+0000 7f684540e640 10 monclient(hunting): handle_auth_done global_id 24246 payload 824 2024-09-11T01:32:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.138+0000 7f684540e640 1 -- 172.21.15.12:0/1863791216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f68401bdd70 con 0x7f684002d980 2024-09-11T01:32:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.141+0000 7f684540e640 1 -- 172.21.15.12:0/1863791216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f684002d980 msgr2=0x7f684002dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.141+0000 7f684540e640 1 --2- 172.21.15.12:0/1863791216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f684002d980 0x7f684002dd90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6838009f20 tx=0x7f68380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.141+0000 7f683dffb640 1 -- 172.21.15.12:0/1863791216 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6838003c30 con 0x7f684002d980 2024-09-11T01:32:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.141+0000 7f683dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.141+0000 7f683dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.141+0000 7f683dffb640 10 monclient: dump: 2024-09-11T01:32:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:00.156 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.141+0000 7f683dffb640 1 -- 172.21.15.12:0/1863791216 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f6838003c30 con 0x7f684002d980 2024-09-11T01:32:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.141+0000 7f683dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:32:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.141+0000 7f683dffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:32:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.141+0000 7f683dffb640 1 -- 172.21.15.12:0/1863791216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f684002d980 msgr2=0x7f684002dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.141+0000 7f683dffb640 1 --2- 172.21.15.12:0/1863791216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f684002d980 0x7f684002dd90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6838009f20 tx=0x7f68380046c0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.141+0000 7f683dffb640 1 -- 172.21.15.12:0/1863791216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f684002d980 msgr2=0x7f684002dd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:32:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.141+0000 7f683dffb640 1 --2- 172.21.15.12:0/1863791216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f684002d980 0x7f684002dd90 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6838009f20 tx=0x7f68380046c0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.157 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:32:00.157 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:32:00.157 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:32:00.157 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:32:00.157 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 28, 2024-09-11T01:32:00.157 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 58088, 2024-09-11T01:32:00.157 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:32:00.157 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:32:00.157 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:32:00.157 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:32:00.157 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:00.158 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 7, 2024-09-11T01:32:00.158 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:32:00.158 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:32:00.158 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:32:00.158 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:00.158 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:00.158 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:00.158 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:00.158 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:00.158 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:00.158 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:00.158 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:00.158 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:00.158 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:00.159 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:00.159 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:00.159 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:32:00.159 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:32:00.159 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:00.159 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:00.159 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:00.159 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:00.159 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:00.159 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:00.159 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:00.159 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:00.159 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:00.159 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:00.160 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:00.160 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:00.160 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.160 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:32:00.160 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:32:00.160 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:32:00.160 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:32:00.160 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:32:00.160 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:32:00.160 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:32:00.160 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:32:00.160 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:32:00.160 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:32:00.160 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:32:00.161 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:32:00.161 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:32:00.161 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:32:00.161 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:32:00.161 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:32:00.161 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:32:00.161 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:00.161 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:00.161 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:00.161 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:00.161 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:00.161 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:00.161 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:00.162 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:00.162 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:00.162 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:00.162 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:00.162 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:00.162 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:32:00.162 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.162 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:32:00.162 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.162 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:32:00.162 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:32:00.162 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:00.162 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:00.162 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.163 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:00.163 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:32:00.163 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:00.163 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.163 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.163 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:00.163 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:32:00.163 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:00.163 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:00.163 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:00.163 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.163 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:32:00.163 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:32:00.163 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:00.164 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:00.164 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:00.164 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.164 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.164 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:32:00.164 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:32:00.164 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:00.164 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:00.164 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.164 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:00.164 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:32:00.164 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:00.164 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.165 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.165 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:00.165 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:32:00.165 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:00.165 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:00.165 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:00.165 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.165 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:32:00.165 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:32:00.165 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:00.165 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:00.165 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:00.165 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.165 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.166 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:32:00.166 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:32:00.166 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:00.166 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:00.166 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.166 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:00.166 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:32:00.166 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:00.166 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.166 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.166 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:00.166 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:32:00.166 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:00.166 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:00.167 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:00.167 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.167 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:32:00.167 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:32:00.167 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:00.167 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:00.167 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:00.167 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:00.167 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:00.167 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.167 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:32:00.167 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:32:00.167 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.167 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:00.168 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:00.168 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:32:00.168 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:00.168 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:00.168 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:32:00.168 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.168 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:00.168 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:00.168 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:32:00.168 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:00.168 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:00.168 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:32:00.168 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.168 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:00.169 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:00.169 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:32:00.169 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:00.169 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:00.169 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:32:00.169 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.169 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:00.169 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:00.169 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:32:00.169 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:00.169 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:00.169 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.169 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:32:00.169 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:32:00.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.141+0000 7f6847672640 1 -- 172.21.15.12:0/1863791216 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f682c03e210 msgr2=0x7f682c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.141+0000 7f6847672640 1 --2- 172.21.15.12:0/1863791216 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f682c03e210 0x7f682c0406d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6834031000 tx=0x7f683402f000 comp rx=0 tx=0).stop 2024-09-11T01:32:00.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.141+0000 7f6847672640 10 monclient: shutdown 2024-09-11T01:32:00.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.141+0000 7f6847672640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:00.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.141+0000 7f6847672640 1 -- 172.21.15.12:0/1863791216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68400a1380 msgr2=0x7f68401c9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.141+0000 7f6847672640 1 --2- 172.21.15.12:0/1863791216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68400a1380 0x7f68401c9ab0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f683000e7c0 tx=0x7f683000ec80 comp rx=0 tx=0).stop 2024-09-11T01:32:00.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.141+0000 7f6845c0f640 1 -- 172.21.15.12:0/1863791216 reap_dead start 2024-09-11T01:32:00.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.141+0000 7f6845c0f640 1 -- 172.21.15.12:0/1863791216 reap_dead start 2024-09-11T01:32:00.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.141+0000 7f6847672640 1 -- 172.21.15.12:0/1863791216 shutdown_connections 2024-09-11T01:32:00.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.141+0000 7f6847672640 1 -- 172.21.15.12:0/1863791216 >> 172.21.15.12:0/1863791216 conn(0x7f68400a2f70 msgr2=0x7f6840070720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:00.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.142+0000 7f6847672640 1 -- 172.21.15.12:0/1863791216 shutdown_connections 2024-09-11T01:32:00.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.142+0000 7f6847672640 1 -- 172.21.15.12:0/1863791216 wait complete. 2024-09-11T01:32:00.172 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:32:00.172 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:32:00.172 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:32:00.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.272+0000 7f866008c640 10 monclient: get_monmap_and_config 2024-09-11T01:32:00.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.272+0000 7f866008c640 10 monclient: build_initial_monmap 2024-09-11T01:32:00.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.272+0000 7f866008c640 10 monclient: monmap: 2024-09-11T01:32:00.274 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:00.274 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.274 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:32:00.272932+0000 2024-09-11T01:32:00.274 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:32:00.272932+0000 2024-09-11T01:32:00.274 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:00.274 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.274 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:32:00.274 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:32:00.274 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:32:00.274 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:32:00.275 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:32:00.275 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.272+0000 7f866008c640 1 Processor -- start 2024-09-11T01:32:00.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.273+0000 7f866008c640 1 -- start start 2024-09-11T01:32:00.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.273+0000 7f866008c640 10 monclient: init 2024-09-11T01:32:00.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.273+0000 7f866008c640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:00.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.273+0000 7f866008c640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-11T01:32:00.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.273+0000 7f866008c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8658137e50 addr v1:172.21.15.12:6789/0 2024-09-11T01:32:00.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.273+0000 7f866008c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f8658138230 0x7f8658138640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.273+0000 7f866008c640 10 monclient(hunting): picked mon.noname-c con 0x7f8658138230 addr v2:172.21.15.71:3300/0 2024-09-11T01:32:00.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.273+0000 7f866008c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f865813d380 addr v1:172.21.15.71:6789/0 2024-09-11T01:32:00.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.273+0000 7f866008c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86580a62d0 con 0x7f8658137e50 2024-09-11T01:32:00.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.273+0000 7f866008c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f865809bb40 con 0x7f865813d380 2024-09-11T01:32:00.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.273+0000 7f866008c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.273+0000 7f866008c640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f865809c4a0 con 0x7f8658138230 2024-09-11T01:32:00.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.273+0000 7f866008c640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:00.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.273+0000 7f866008c640 10 monclient(hunting): authenticate will time out at 926.715660s 2024-09-11T01:32:00.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.273+0000 7f865de28640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f8658137e50 0x7f865809d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50724/0 (socket says 172.21.15.12:50724) 2024-09-11T01:32:00.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.273+0000 7f865de28640 1 -- 172.21.15.12:0/3902321161 learned_addr learned my addr 172.21.15.12:0/3902321161 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:32:00.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.273+0000 7f865d627640 1 --2- 172.21.15.12:0/3902321161 >> v2:172.21.15.71:3300/0 conn(0x7f8658138230 0x7f8658138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.273+0000 7f865d627640 10 monclient(hunting): get_auth_request con 0x7f8658138230 auth_method 0 2024-09-11T01:32:00.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.273+0000 7f865d627640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:00.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.273+0000 7f865d627640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.273+0000 7f865d627640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:00.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.274+0000 7f865d627640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:00.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.274+0000 7f865d627640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:00.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.274+0000 7f865d627640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.275+0000 7f865de28640 1 -- 172.21.15.12:0/3902321161 >> v1:172.21.15.12:6789/0 conn(0x7f8658137e50 legacy=0x7f865809d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 1 -- 172.21.15.12:0/3902321161 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2267464228 0 0) 0x7f86580a62d0 con 0x7f8658137e50 2024-09-11T01:32:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 10 monclient(hunting): _negotiate 2024-09-11T01:32:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 10 monclient(hunting): my global_id is 24249 2024-09-11T01:32:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 1 -- 172.21.15.12:0/3902321161 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8644003520 con 0x7f8658137e50 2024-09-11T01:32:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865e629640 1 -- 172.21.15.12:0/3902321161 >> v1:172.21.15.71:6789/0 conn(0x7f865813d380 legacy=0x7f865813f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 1 -- 172.21.15.12:0/3902321161 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2962672996 0 0) 0x7f8644003520 con 0x7f8658137e50 2024-09-11T01:32:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 1 -- 172.21.15.12:0/3902321161 >> v2:172.21.15.71:3300/0 conn(0x7f8658138230 msgr2=0x7f8658138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 1 --2- 172.21.15.12:0/3902321161 >> v2:172.21.15.71:3300/0 conn(0x7f8658138230 0x7f8658138640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 1 -- 172.21.15.12:0/3902321161 >> v1:172.21.15.71:6789/0 conn(0x7f865813d380 legacy=0x7f865813f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 10 monclient: _finish_hunting 0 2024-09-11T01:32:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:32:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:32:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 1 -- 172.21.15.12:0/3902321161 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f865809c780 con 0x7f8658137e50 2024-09-11T01:32:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 10 monclient: _finish_auth 0 2024-09-11T01:32:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 10 monclient: _check_auth_tickets 2024-09-11T01:32:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 1 -- 172.21.15.12:0/3902321161 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f8654001990 con 0x7f8658137e50 2024-09-11T01:32:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:32:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 10 monclient: dump: 2024-09-11T01:32:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:00.291 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:32:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 1 -- 172.21.15.12:0/3902321161 >> v1:172.21.15.12:6789/0 conn(0x7f8658137e50 legacy=0x7f865809d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:32:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 1 --2- 172.21.15.12:0/3902321161 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8644003b20 0x7f8644003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 10 monclient(hunting): picked mon.c con 0x7f8644003b20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 1 --2- 172.21.15.12:0/3902321161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8644004800 0x7f8644004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 10 monclient(hunting): picked mon.a con 0x7f8644004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:32:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 1 --2- 172.21.15.12:0/3902321161 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86440053c0 0x7f864400ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 10 monclient(hunting): picked mon.b con 0x7f86440053c0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:32:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 1 -- 172.21.15.12:0/3902321161 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f865809c4a0 con 0x7f8644004800 2024-09-11T01:32:00.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 1 -- 172.21.15.12:0/3902321161 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f864400c300 con 0x7f86440053c0 2024-09-11T01:32:00.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 1 -- 172.21.15.12:0/3902321161 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f864400c500 con 0x7f8644003b20 2024-09-11T01:32:00.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:00.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865ce26640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:32:00.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865de28640 1 --2- 172.21.15.12:0/3902321161 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8644003b20 0x7f8644003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865d627640 1 --2- 172.21.15.12:0/3902321161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8644004800 0x7f8644004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865de28640 10 monclient(hunting): get_auth_request con 0x7f8644003b20 auth_method 0 2024-09-11T01:32:00.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865de28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:00.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865de28640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865de28640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:32:00.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865d627640 10 monclient(hunting): get_auth_request con 0x7f8644004800 auth_method 0 2024-09-11T01:32:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865d627640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865d627640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865d627640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:32:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865e629640 1 --2- 172.21.15.12:0/3902321161 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86440053c0 0x7f864400ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865e629640 10 monclient(hunting): get_auth_request con 0x7f86440053c0 auth_method 0 2024-09-11T01:32:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865e629640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865e629640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865e629640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:32:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865d627640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865d627640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865d627640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:32:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865de28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:00.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865de28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865de28640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:32:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865e629640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865e629640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865e629640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:32:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865d627640 10 monclient(hunting): handle_auth_done global_id 24249 payload 293 2024-09-11T01:32:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865d627640 1 -- 172.21.15.12:0/3902321161 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8644003b20 msgr2=0x7f8644003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865d627640 1 --2- 172.21.15.12:0/3902321161 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8644003b20 0x7f8644003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865d627640 1 -- 172.21.15.12:0/3902321161 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86440053c0 msgr2=0x7f864400ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865d627640 1 --2- 172.21.15.12:0/3902321161 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86440053c0 0x7f864400ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865d627640 10 monclient: _finish_hunting 0 2024-09-11T01:32:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865d627640 1 monclient: found mon.a 2024-09-11T01:32:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865d627640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:00.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865d627640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865e629640 1 -- 172.21.15.12:0/3902321161 reap_dead start 2024-09-11T01:32:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865d627640 1 -- 172.21.15.12:0/3902321161 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8644003520 con 0x7f8644004800 2024-09-11T01:32:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865d627640 10 monclient: _finish_auth 0 2024-09-11T01:32:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865d627640 10 monclient: _check_auth_tickets 2024-09-11T01:32:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f865d627640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f866008c640 5 monclient: authenticate success, global_id 24249 2024-09-11T01:32:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.276+0000 7f866008c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:32:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.279+0000 7f865d627640 1 -- 172.21.15.12:0/3902321161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8644004800 msgr2=0x7f8644004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.279+0000 7f865d627640 1 --2- 172.21.15.12:0/3902321161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8644004800 0x7f8644004c80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f864c009ac0 tx=0x7f8644005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd09c8f9c6af276d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.279+0000 7f865ce26640 1 -- 172.21.15.12:0/3902321161 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864c036070 con 0x7f8644004800 2024-09-11T01:32:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.279+0000 7f865ce26640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.279+0000 7f865ce26640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.279+0000 7f865ce26640 10 monclient: dump: 2024-09-11T01:32:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:00.296 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.279+0000 7f866008c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:32:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.279+0000 7f865ce26640 1 -- 172.21.15.12:0/3902321161 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f864c02f510 con 0x7f8644004800 2024-09-11T01:32:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.279+0000 7f865ce26640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.279+0000 7f866008c640 10 monclient: get_monmap_and_config success 2024-09-11T01:32:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.279+0000 7f865ce26640 1 -- 172.21.15.12:0/3902321161 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864c009690 con 0x7f8644004800 2024-09-11T01:32:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.279+0000 7f866008c640 4 set_mon_vals no callback set 2024-09-11T01:32:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.279+0000 7f865ce26640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.279+0000 7f865ce26640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.279+0000 7f865ce26640 10 monclient: dump: 2024-09-11T01:32:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:00.298 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:00.298 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.280+0000 7f866008c640 10 monclient: shutdown 2024-09-11T01:32:00.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.280+0000 7f866008c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:00.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.280+0000 7f866008c640 1 -- 172.21.15.12:0/3902321161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8644004800 msgr2=0x7f8644004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.280+0000 7f866008c640 1 --2- 172.21.15.12:0/3902321161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8644004800 0x7f8644004c80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f864c009ac0 tx=0x7f8644005830 comp rx=0 tx=0).stop 2024-09-11T01:32:00.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.280+0000 7f866008c640 1 -- 172.21.15.12:0/3902321161 shutdown_connections 2024-09-11T01:32:00.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.280+0000 7f866008c640 1 --2- 172.21.15.12:0/3902321161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8644004800 0x7f8644004c80 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.280+0000 7f866008c640 1 -- 172.21.15.12:0/3902321161 >> 172.21.15.12:0/3902321161 conn(0x7f86580a7020 msgr2=0x7f86580a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:00.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.280+0000 7f866008c640 1 -- 172.21.15.12:0/3902321161 shutdown_connections 2024-09-11T01:32:00.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.280+0000 7f866008c640 1 -- 172.21.15.12:0/3902321161 wait complete. 2024-09-11T01:32:00.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.281+0000 7f866008c640 10 monclient: build_initial_monmap 2024-09-11T01:32:00.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.281+0000 7f866008c640 10 monclient: monmap: 2024-09-11T01:32:00.298 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:00.299 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:32:00.299 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:32:00.299 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:32:00.299 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:00.299 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.299 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:32:00.299 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:32:00.299 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:32:00.299 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.281+0000 7f866008c640 1 Processor -- start 2024-09-11T01:32:00.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.281+0000 7f866008c640 1 -- start start 2024-09-11T01:32:00.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.281+0000 7f866008c640 10 monclient: init 2024-09-11T01:32:00.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.281+0000 7f866008c640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:00.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.281+0000 7f866008c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:32:00.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.281+0000 7f866008c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8644004800 0x7f8658120b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.281+0000 7f866008c640 10 monclient(hunting): picked mon.noname-c con 0x7f8644004800 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:00.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.281+0000 7f866008c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86581210b0 0x7f8658121530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.281+0000 7f866008c640 10 monclient(hunting): picked mon.noname-b con 0x7f86581210b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:32:00.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.281+0000 7f866008c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86581cd520 0x7f86581cd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.281+0000 7f866008c640 10 monclient(hunting): picked mon.noname-a con 0x7f86581cd520 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:32:00.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.281+0000 7f866008c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.281+0000 7f866008c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f865809d2e0 con 0x7f86581cd520 2024-09-11T01:32:00.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.281+0000 7f866008c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.281+0000 7f866008c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f865809d160 con 0x7f86581210b0 2024-09-11T01:32:00.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.281+0000 7f866008c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.281+0000 7f866008c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f865809d460 con 0x7f8644004800 2024-09-11T01:32:00.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.281+0000 7f866008c640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:00.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.281+0000 7f866008c640 10 monclient(hunting): authenticate will time out at 926.723929s 2024-09-11T01:32:00.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865de28640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8644004800 0x7f8658120b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865e629640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86581cd520 0x7f86581cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865de28640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8644004800 0x7f8658120b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:48302/0 (socket says 172.21.15.12:48302) 2024-09-11T01:32:00.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865de28640 1 -- 172.21.15.12:0/1930770695 learned_addr learned my addr 172.21.15.12:0/1930770695 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:00.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865de28640 10 monclient(hunting): get_auth_request con 0x7f8644004800 auth_method 0 2024-09-11T01:32:00.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865de28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:00.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865e629640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86581cd520 0x7f86581cd990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50546/0 (socket says 172.21.15.12:50546) 2024-09-11T01:32:00.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865de28640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865de28640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:00.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865e629640 10 monclient(hunting): get_auth_request con 0x7f86581cd520 auth_method 0 2024-09-11T01:32:00.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865e629640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:00.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865e629640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865e629640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:00.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865de28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:00.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865de28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:00.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865de28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:00.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865e629640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:00.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865e629640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:00.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865e629640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:00.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865d627640 1 --2- 172.21.15.12:0/1930770695 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86581210b0 0x7f8658121530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865d627640 10 monclient(hunting): get_auth_request con 0x7f86581210b0 auth_method 0 2024-09-11T01:32:00.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865d627640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:00.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865d627640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865d627640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:00.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865de28640 10 monclient(hunting): handle_auth_done global_id 4670 payload 824 2024-09-11T01:32:00.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865de28640 1 -- 172.21.15.12:0/1930770695 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86581210b0 msgr2=0x7f8658121530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865de28640 1 --2- 172.21.15.12:0/1930770695 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86581210b0 0x7f8658121530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865de28640 1 -- 172.21.15.12:0/1930770695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86581cd520 msgr2=0x7f86581cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865de28640 1 --2- 172.21.15.12:0/1930770695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86581cd520 0x7f86581cd990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865de28640 10 monclient: _finish_hunting 0 2024-09-11T01:32:00.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865de28640 1 monclient: found mon.noname-c 2024-09-11T01:32:00.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865de28640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:00.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865de28640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:32:00.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865de28640 1 -- 172.21.15.12:0/1930770695 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86580a1710 con 0x7f8644004800 2024-09-11T01:32:00.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.282+0000 7f865e629640 1 --2- 172.21.15.12:0/1930770695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86581cd520 0x7f86581cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:32:00.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.284+0000 7f865de28640 1 -- 172.21.15.12:0/1930770695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8644004800 msgr2=0x7f8658120b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:00.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.284+0000 7f865de28640 1 --2- 172.21.15.12:0/1930770695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8644004800 0x7f8658120b70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8654001b00 tx=0x7f8654007660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:00.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.284+0000 7f864a7fc640 1 -- 172.21.15.12:0/1930770695 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865400fbd0 con 0x7f8644004800 2024-09-11T01:32:00.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.284+0000 7f864a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:00.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.284+0000 7f864a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:32:00.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.284+0000 7f864a7fc640 10 monclient: dump: 2024-09-11T01:32:00.304 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:00.304 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.304 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.304 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.304 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:00.304 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.304 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:00.304 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:00.304 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:00.304 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.284+0000 7f864a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:32:00.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.284+0000 7f864a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:32:00.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.284+0000 7f864a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:00.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.284+0000 7f866008c640 5 monclient: authenticate success, global_id 4670 2024-09-11T01:32:00.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.284+0000 7f864a7fc640 1 -- 172.21.15.12:0/1930770695 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8654018070 con 0x7f8644004800 2024-09-11T01:32:00.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.284+0000 7f866008c640 10 monclient: _renew_subs 2024-09-11T01:32:00.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.284+0000 7f866008c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:32:00.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.284+0000 7f866008c640 1 -- 172.21.15.12:0/1930770695 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8658123f20 con 0x7f8644004800 2024-09-11T01:32:00.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.284+0000 7f864a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:00.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.284+0000 7f864a7fc640 1 -- 172.21.15.12:0/1930770695 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86540133f0 con 0x7f8644004800 2024-09-11T01:32:00.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.284+0000 7f864a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:00.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.284+0000 7f864a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:32:00.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.284+0000 7f864a7fc640 10 monclient: dump: 2024-09-11T01:32:00.305 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:00.306 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.306 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.306 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.306 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:00.306 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.306 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:00.306 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:00.306 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:00.306 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.284+0000 7f866008c640 10 monclient: _renew_subs 2024-09-11T01:32:00.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.284+0000 7f866008c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:32:00.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.284+0000 7f866008c640 1 -- 172.21.15.12:0/1930770695 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f86580a06d0 con 0x7f8644004800 2024-09-11T01:32:00.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.284+0000 7f864bfff640 4 set_mon_vals no callback set 2024-09-11T01:32:00.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.285+0000 7f866008c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:00.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.285+0000 7f866008c640 1 --2- 172.21.15.12:0/1930770695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86581cded0 0x7f865809ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.285+0000 7f866008c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.285+0000 7f866008c640 1 -- 172.21.15.12:0/1930770695 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f86580a9a40 con 0x7f86581cded0 2024-09-11T01:32:00.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.285+0000 7f865d627640 1 --2- 172.21.15.12:0/1930770695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86581cded0 0x7f865809ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.285+0000 7f865d627640 10 monclient: get_auth_request con 0x7f86581cded0 auth_method 0 2024-09-11T01:32:00.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.285+0000 7f865d627640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:00.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.285+0000 7f865d627640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.285+0000 7f865d627640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:00.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.285+0000 7f865d627640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:00.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.285+0000 7f865d627640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:00.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.285+0000 7f865d627640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:00.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.285+0000 7f864a7fc640 1 -- 172.21.15.12:0/1930770695 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f8654013590 con 0x7f8644004800 2024-09-11T01:32:00.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.285+0000 7f865d627640 10 monclient(hunting): handle_auth_done global_id 4673 payload 824 2024-09-11T01:32:00.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.285+0000 7f865d627640 1 -- 172.21.15.12:0/1930770695 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8658138640 con 0x7f86581cded0 2024-09-11T01:32:00.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.286+0000 7f864a7fc640 1 --2- 172.21.15.12:0/1930770695 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f863c03e260 0x7f863c040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.286+0000 7f864a7fc640 1 -- 172.21.15.12:0/1930770695 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(103..103 src has 1..103) ==== 4739+0+0 (secure 0 0 0) 0x7f8654052d00 con 0x7f8644004800 2024-09-11T01:32:00.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.286+0000 7f865e629640 1 --2- 172.21.15.12:0/1930770695 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f863c03e260 0x7f863c040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.286+0000 7f865e629640 10 monclient: get_auth_request con 0x7f863c03e260 auth_method 0 2024-09-11T01:32:00.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.288+0000 7f865d627640 1 -- 172.21.15.12:0/1930770695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86581cded0 msgr2=0x7f865809ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:00.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.288+0000 7f865d627640 1 --2- 172.21.15.12:0/1930770695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86581cded0 0x7f865809ede0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f864c009a90 tx=0x7f864c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:00.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.288+0000 7f864a7fc640 1 -- 172.21.15.12:0/1930770695 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864c040030 con 0x7f86581cded0 2024-09-11T01:32:00.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.288+0000 7f864a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:00.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.288+0000 7f864a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:32:00.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.288+0000 7f864a7fc640 10 monclient: dump: 2024-09-11T01:32:00.311 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:00.311 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.312 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.312 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.312 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:00.312 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.312 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:00.312 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:00.312 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:00.312 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.288+0000 7f864a7fc640 1 -- 172.21.15.12:0/1930770695 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f864c003610 con 0x7f86581cded0 2024-09-11T01:32:00.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.288+0000 7f864a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:00.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.288+0000 7f864a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:32:00.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.288+0000 7f864a7fc640 1 -- 172.21.15.12:0/1930770695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86581cded0 msgr2=0x7f865809ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.288+0000 7f864a7fc640 1 --2- 172.21.15.12:0/1930770695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86581cded0 0x7f865809ede0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f864c009a90 tx=0x7f864c004730 comp rx=0 tx=0).stop 2024-09-11T01:32:00.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.288+0000 7f864a7fc640 1 -- 172.21.15.12:0/1930770695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86581cded0 msgr2=0x7f865809ede0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:32:00.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.288+0000 7f864a7fc640 1 --2- 172.21.15.12:0/1930770695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86581cded0 0x7f865809ede0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.288+0000 7f865e629640 1 --2- 172.21.15.12:0/1930770695 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f863c03e260 0x7f863c040720 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f86580a8360 tx=0x7f8650001970 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:00.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.294+0000 7f866008c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:32:00.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.294+0000 7f866008c640 1 --2- 172.21.15.12:0/1930770695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f865809f320 0x7f865809f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.294+0000 7f866008c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.294+0000 7f866008c640 1 -- 172.21.15.12:0/1930770695 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8658123390 con 0x7f865809f320 2024-09-11T01:32:00.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.295+0000 7f865d627640 1 --2- 172.21.15.12:0/1930770695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f865809f320 0x7f865809f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.295+0000 7f865d627640 10 monclient: get_auth_request con 0x7f865809f320 auth_method 0 2024-09-11T01:32:00.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.295+0000 7f865d627640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:00.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.295+0000 7f865d627640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.295+0000 7f865d627640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:00.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.295+0000 7f865d627640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:00.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.295+0000 7f865d627640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:00.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.295+0000 7f865d627640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:00.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.295+0000 7f865d627640 10 monclient(hunting): handle_auth_done global_id 4676 payload 824 2024-09-11T01:32:00.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.295+0000 7f865d627640 1 -- 172.21.15.12:0/1930770695 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f86581ce2b0 con 0x7f865809f320 2024-09-11T01:32:00.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.297+0000 7f865d627640 1 -- 172.21.15.12:0/1930770695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f865809f320 msgr2=0x7f865809f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:00.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.297+0000 7f865d627640 1 --2- 172.21.15.12:0/1930770695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f865809f320 0x7f865809f720 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f864c009ac0 tx=0x7f864c02f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:00.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.297+0000 7f864a7fc640 1 -- 172.21.15.12:0/1930770695 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864c040070 con 0x7f865809f320 2024-09-11T01:32:00.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.297+0000 7f864a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:00.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.297+0000 7f864a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:32:00.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.297+0000 7f864a7fc640 10 monclient: dump: 2024-09-11T01:32:00.314 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:00.314 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.314 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.314 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.315 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:00.315 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.315 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:00.315 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:00.315 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:00.315 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.297+0000 7f864a7fc640 1 -- 172.21.15.12:0/1930770695 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7f864c003bd0 con 0x7f865809f320 2024-09-11T01:32:00.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.297+0000 7f864a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:32:00.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.297+0000 7f864a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:32:00.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.297+0000 7f864a7fc640 1 -- 172.21.15.12:0/1930770695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f865809f320 msgr2=0x7f865809f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.297+0000 7f864a7fc640 1 --2- 172.21.15.12:0/1930770695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f865809f320 0x7f865809f720 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f864c009ac0 tx=0x7f864c02f000 comp rx=0 tx=0).stop 2024-09-11T01:32:00.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.297+0000 7f864a7fc640 1 -- 172.21.15.12:0/1930770695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f865809f320 msgr2=0x7f865809f720 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:32:00.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.297+0000 7f864a7fc640 1 --2- 172.21.15.12:0/1930770695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f865809f320 0x7f865809f720 secure :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f864c009ac0 tx=0x7f864c02f000 comp rx=0 tx=0).stop 2024-09-11T01:32:00.316 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:32:00.316 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:32:00.316 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:32:00.316 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:32:00.316 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 28, 2024-09-11T01:32:00.316 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 182986, 2024-09-11T01:32:00.316 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:32:00.316 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:32:00.316 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:32:00.316 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:32:00.317 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:00.317 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 7, 2024-09-11T01:32:00.317 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:32:00.317 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:32:00.317 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:32:00.317 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:00.317 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:00.317 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:00.317 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:00.317 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:00.317 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:00.317 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:00.317 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:00.318 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:00.318 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:00.318 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:00.318 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:00.318 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:32:00.318 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:32:00.318 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:00.318 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:00.318 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:00.318 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:00.318 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:00.318 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:00.318 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:00.319 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:00.319 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:00.319 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:00.319 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:00.319 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:00.319 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.319 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:32:00.319 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:32:00.319 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:32:00.319 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:32:00.319 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:32:00.319 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:32:00.319 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:32:00.319 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:32:00.320 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:32:00.320 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:32:00.320 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:32:00.320 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:32:00.320 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:32:00.320 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:32:00.320 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:32:00.320 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:32:00.320 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:32:00.320 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:00.320 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:00.320 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:00.320 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:00.321 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:00.321 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:00.321 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:00.321 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:00.321 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:00.321 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:00.321 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:00.321 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:00.321 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:32:00.321 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.321 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:32:00.321 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.321 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:32:00.322 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:32:00.322 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:00.322 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:00.322 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.322 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:00.322 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:32:00.322 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:00.322 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.322 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.322 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:00.322 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:32:00.322 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:00.322 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:00.322 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:00.323 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.323 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:32:00.323 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:32:00.323 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:00.323 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:00.323 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:00.323 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.323 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.323 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:32:00.323 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:32:00.323 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:00.323 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:00.323 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.323 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:00.324 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:32:00.324 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:00.324 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.324 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.324 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:00.324 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:32:00.324 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:00.325 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:00.325 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:00.325 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.325 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:32:00.325 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:32:00.325 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:00.325 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:00.325 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:00.325 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.326 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.326 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:32:00.326 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:32:00.326 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:00.326 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:00.326 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.326 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:00.326 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:32:00.326 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:00.326 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.326 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.326 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:00.326 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:32:00.327 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:00.327 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:00.327 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:00.327 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.327 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:32:00.327 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:32:00.327 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:00.327 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:00.327 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:00.327 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:00.327 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:00.327 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.327 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:32:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:32:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:32:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:32:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:32:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:00.328 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:00.329 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:32:00.329 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.329 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:00.329 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:00.329 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:32:00.329 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:00.329 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:00.329 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.329 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:32:00.329 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:32:00.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.298+0000 7f866008c640 1 -- 172.21.15.12:0/1930770695 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f863c03e260 msgr2=0x7f863c040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.298+0000 7f866008c640 1 --2- 172.21.15.12:0/1930770695 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f863c03e260 0x7f863c040720 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f86580a8360 tx=0x7f8650001970 comp rx=0 tx=0).stop 2024-09-11T01:32:00.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.298+0000 7f866008c640 10 monclient: shutdown 2024-09-11T01:32:00.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.298+0000 7f866008c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:00.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.298+0000 7f866008c640 1 -- 172.21.15.12:0/1930770695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8644004800 msgr2=0x7f8658120b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.298+0000 7f866008c640 1 --2- 172.21.15.12:0/1930770695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8644004800 0x7f8658120b70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8654001b00 tx=0x7f8654007660 comp rx=0 tx=0).stop 2024-09-11T01:32:00.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.298+0000 7f865e629640 1 -- 172.21.15.12:0/1930770695 reap_dead start 2024-09-11T01:32:00.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.298+0000 7f865e629640 1 -- 172.21.15.12:0/1930770695 reap_dead start 2024-09-11T01:32:00.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.298+0000 7f866008c640 1 -- 172.21.15.12:0/1930770695 shutdown_connections 2024-09-11T01:32:00.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.298+0000 7f866008c640 1 -- 172.21.15.12:0/1930770695 >> 172.21.15.12:0/1930770695 conn(0x7f86580a7020 msgr2=0x7f865813cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:00.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.298+0000 7f866008c640 1 -- 172.21.15.12:0/1930770695 shutdown_connections 2024-09-11T01:32:00.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.298+0000 7f866008c640 1 -- 172.21.15.12:0/1930770695 wait complete. 2024-09-11T01:32:00.332 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:32:00.332 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:32:00.332 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:32:00.351 INFO:tasks.rados.rados.0.smithi071.stdout:459: done (2 left) 2024-09-11T01:32:00.351 INFO:tasks.rados.rados.0.smithi071.stdout:460: copy_from oid 16 from oid 50 current snap is 41 2024-09-11T01:32:00.351 INFO:tasks.rados.rados.0.smithi071.stdout:457: finishing write tid 1 to smithi07134777-25 2024-09-11T01:32:00.351 INFO:tasks.rados.rados.0.smithi071.stdout:457: finishing write tid 2 to smithi07134777-25 2024-09-11T01:32:00.351 INFO:tasks.rados.rados.0.smithi071.stdout:457: finishing write tid 3 to smithi07134777-25 2024-09-11T01:32:00.351 INFO:tasks.rados.rados.0.smithi071.stdout:457: finishing write tid 4 to smithi07134777-25 2024-09-11T01:32:00.351 INFO:tasks.rados.rados.0.smithi071.stdout:457: oid 25 updating version 0 to 31 2024-09-11T01:32:00.352 INFO:tasks.rados.rados.0.smithi071.stdout:457: oid 25 updating version 31 to 32 2024-09-11T01:32:00.352 INFO:tasks.rados.rados.0.smithi071.stdout:457: oid 25 version 32 is already newer than 30 2024-09-11T01:32:00.352 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 32 (ObjNum 182 snap 41 seq_num 182) dirty exists 2024-09-11T01:32:00.352 INFO:tasks.rados.rados.0.smithi071.stdout:457: left oid 25 (ObjNum 182 snap 41 seq_num 182) 2024-09-11T01:32:00.352 INFO:tasks.rados.rados.0.smithi071.stdout:458: finishing write tid 1 to smithi07134777-4 2024-09-11T01:32:00.352 INFO:tasks.rados.rados.0.smithi071.stdout:458: finishing write tid 2 to smithi07134777-4 2024-09-11T01:32:00.352 INFO:tasks.rados.rados.0.smithi071.stdout:458: finishing write tid 3 to smithi07134777-4 2024-09-11T01:32:00.352 INFO:tasks.rados.rados.0.smithi071.stdout:458: finishing write tid 4 to smithi07134777-4 2024-09-11T01:32:00.352 INFO:tasks.rados.rados.0.smithi071.stdout:458: finishing write tid 5 to smithi07134777-4 2024-09-11T01:32:00.353 INFO:tasks.rados.rados.0.smithi071.stdout:458: oid 4 updating version 0 to 131 2024-09-11T01:32:00.353 INFO:tasks.rados.rados.0.smithi071.stdout:458: oid 4 version 131 is already newer than 128 2024-09-11T01:32:00.353 INFO:tasks.rados.rados.0.smithi071.stdout:458: oid 4 version 131 is already newer than 129 2024-09-11T01:32:00.353 INFO:tasks.rados.rados.0.smithi071.stdout:458: oid 4 version 131 is already newer than 130 2024-09-11T01:32:00.353 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 131 (ObjNum 183 snap 41 seq_num 183) dirty exists 2024-09-11T01:32:00.353 INFO:tasks.rados.rados.0.smithi071.stdout:458: left oid 4 (ObjNum 183 snap 41 seq_num 183) 2024-09-11T01:32:00.353 INFO:tasks.rados.rados.0.smithi071.stdout:457: done (2 left) 2024-09-11T01:32:00.353 INFO:tasks.rados.rados.0.smithi071.stdout:458: done (1 left) 2024-09-11T01:32:00.353 INFO:tasks.rados.rados.0.smithi071.stdout:461: write oid 10 current snap is 41 2024-09-11T01:32:00.353 INFO:tasks.rados.rados.0.smithi071.stdout:461: seq_num 184 ranges {692488=638288,1917275=606724} 2024-09-11T01:32:00.358 INFO:tasks.rados.rados.0.smithi071.stdout:461: writing smithi07134777-10 from 692488 to 1330776 tid 1 2024-09-11T01:32:00.365 INFO:tasks.rados.rados.0.smithi071.stdout:461: writing smithi07134777-10 from 1917275 to 2523999 tid 2 2024-09-11T01:32:00.365 INFO:tasks.rados.rados.0.smithi071.stdout:462: read oid 34 snap 40 2024-09-11T01:32:00.365 INFO:tasks.rados.rados.0.smithi071.stdout:462: expect (ObjNum 130 snap 30 seq_num 130) 2024-09-11T01:32:00.365 INFO:tasks.rados.rados.0.smithi071.stdout:463: write oid 30 current snap is 41 2024-09-11T01:32:00.365 INFO:tasks.rados.rados.0.smithi071.stdout:463: seq_num 185 ranges {709295=713537,1859404=750382} 2024-09-11T01:32:00.373 INFO:tasks.rados.rados.0.smithi071.stdout:463: writing smithi07134777-30 from 709295 to 1422832 tid 1 2024-09-11T01:32:00.380 INFO:tasks.rados.rados.0.smithi071.stdout:463: writing smithi07134777-30 from 1859404 to 2609786 tid 2 2024-09-11T01:32:00.380 INFO:tasks.rados.rados.0.smithi071.stdout:464: write oid 48 current snap is 41 2024-09-11T01:32:00.380 INFO:tasks.rados.rados.0.smithi071.stdout:460: finishing copy_from racing read to smithi07134777-16 2024-09-11T01:32:00.380 INFO:tasks.rados.rados.0.smithi071.stdout:461: finishing write tid 1 to smithi07134777-10 2024-09-11T01:32:00.380 INFO:tasks.rados.rados.0.smithi071.stdout:461: finishing write tid 2 to smithi07134777-10 2024-09-11T01:32:00.380 INFO:tasks.rados.rados.0.smithi071.stdout:464: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2024-09-11T01:32:00.387 INFO:tasks.rados.rados.0.smithi071.stdout:464: writing smithi07134777-48 from 726102 to 1514888 tid 1 2024-09-11T01:32:00.393 INFO:tasks.rados.rados.0.smithi071.stdout:464: writing smithi07134777-48 from 1917886 to 2528279 tid 2 2024-09-11T01:32:00.397 INFO:tasks.rados.rados.0.smithi071.stdout:464: writing smithi07134777-48 from 3108206 to 3126102 tid 3 2024-09-11T01:32:00.397 INFO:tasks.rados.rados.0.smithi071.stdout:465: write (excl) oid 47 current snap is 41 2024-09-11T01:32:00.397 INFO:tasks.rados.rados.0.smithi071.stdout:465: seq_num 187 ranges {742909=464035,1976368=754051} 2024-09-11T01:32:00.403 INFO:tasks.rados.rados.0.smithi071.stdout:465: writing smithi07134777-47 from 742909 to 1206944 tid 1 2024-09-11T01:32:00.412 INFO:tasks.rados.rados.0.smithi071.stdout:465: writing smithi07134777-47 from 1976368 to 2730419 tid 2 2024-09-11T01:32:00.412 INFO:tasks.rados.rados.0.smithi071.stdout:466: snap_create 2024-09-11T01:32:00.412 INFO:tasks.rados.rados.0.smithi071.stdout:461: finishing write tid 3 to smithi07134777-10 2024-09-11T01:32:00.412 INFO:tasks.rados.rados.0.smithi071.stdout:461: finishing write tid 4 to smithi07134777-10 2024-09-11T01:32:00.413 INFO:tasks.rados.rados.0.smithi071.stdout:461: oid 10 updating version 0 to 112 2024-09-11T01:32:00.413 INFO:tasks.rados.rados.0.smithi071.stdout:461: oid 10 version 112 is already newer than 111 2024-09-11T01:32:00.413 INFO:tasks.rados.rados.0.smithi071.stdout:461: oid 10 version 112 is already newer than 110 2024-09-11T01:32:00.413 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 112 (ObjNum 184 snap 41 seq_num 184) dirty exists 2024-09-11T01:32:00.413 INFO:tasks.rados.rados.0.smithi071.stdout:461: left oid 10 (ObjNum 184 snap 41 seq_num 184) 2024-09-11T01:32:00.413 INFO:tasks.rados.rados.0.smithi071.stdout:462: expect (ObjNum 130 snap 30 seq_num 130) 2024-09-11T01:32:00.415 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 622966 2024-09-11T01:32:00.432 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 300537 2024-09-11T01:32:00.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.441+0000 7f9b19fc6640 10 monclient: get_monmap_and_config 2024-09-11T01:32:00.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.441+0000 7f9b19fc6640 10 monclient: build_initial_monmap 2024-09-11T01:32:00.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.441+0000 7f9b19fc6640 10 monclient: monmap: 2024-09-11T01:32:00.443 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:00.443 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.443 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:32:00.442482+0000 2024-09-11T01:32:00.443 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:32:00.442482+0000 2024-09-11T01:32:00.443 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:00.443 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.443 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:32:00.444 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:32:00.444 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:32:00.444 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:32:00.444 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:32:00.444 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.442+0000 7f9b19fc6640 1 Processor -- start 2024-09-11T01:32:00.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.442+0000 7f9b19fc6640 1 -- start start 2024-09-11T01:32:00.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.442+0000 7f9b19fc6640 10 monclient: init 2024-09-11T01:32:00.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b19fc6640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:00.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b19fc6640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-11T01:32:00.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b19fc6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9b141381c0 0x7f9b141385a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b19fc6640 10 monclient(hunting): picked mon.noname-b con 0x7f9b141381c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:00.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b19fc6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9b140a6aa0 addr v1:172.21.15.71:6789/0 2024-09-11T01:32:00.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b19fc6640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9b1413d000 0x7f9b1413f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b19fc6640 10 monclient(hunting): picked mon.noname-a con 0x7f9b1413d000 addr v2:172.21.15.12:3300/0 2024-09-11T01:32:00.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b19fc6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b1409c310 con 0x7f9b140a6aa0 2024-09-11T01:32:00.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b19fc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b19fc6640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f9b1406fc90 con 0x7f9b1413d000 2024-09-11T01:32:00.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b19fc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b19fc6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f9b1409c5f0 con 0x7f9b141381c0 2024-09-11T01:32:00.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b19fc6640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:00.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b19fc6640 10 monclient(hunting): authenticate will time out at 926.885214s 2024-09-11T01:32:00.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b137fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9b141381c0 0x7f9b141385a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b137fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9b141381c0 0x7f9b141385a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:48330/0 (socket says 172.21.15.12:48330) 2024-09-11T01:32:00.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b13fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9b1413d000 0x7f9b1413f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b137fe640 1 -- 172.21.15.12:0/185732023 learned_addr learned my addr 172.21.15.12:0/185732023 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:00.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b137fe640 10 monclient(hunting): get_auth_request con 0x7f9b141381c0 auth_method 0 2024-09-11T01:32:00.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:00.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b137fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:00.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b13fff640 10 monclient(hunting): get_auth_request con 0x7f9b1413d000 auth_method 0 2024-09-11T01:32:00.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:00.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b13fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b13fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:00.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:00.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:00.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:00.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:00.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:00.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:00.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b137fe640 10 monclient(hunting): handle_auth_done global_id 4679 payload 274 2024-09-11T01:32:00.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b137fe640 1 -- 172.21.15.12:0/185732023 >> v2:172.21.15.12:3300/0 conn(0x7f9b1413d000 msgr2=0x7f9b1413f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:00.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b137fe640 1 --2- 172.21.15.12:0/185732023 >> v2:172.21.15.12:3300/0 conn(0x7f9b1413d000 0x7f9b1413f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b137fe640 1 -- 172.21.15.12:0/185732023 >> v1:172.21.15.71:6789/0 conn(0x7f9b140a6aa0 legacy=0x7f9b1409cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:00.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b137fe640 10 monclient: _finish_hunting 0 2024-09-11T01:32:00.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b137fe640 1 monclient: found mon.noname-b 2024-09-11T01:32:00.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:00.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b137fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:32:00.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b137fe640 1 -- 172.21.15.12:0/185732023 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b1409c8d0 con 0x7f9b141381c0 2024-09-11T01:32:00.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.443+0000 7f9b13fff640 1 --2- 172.21.15.12:0/185732023 >> v2:172.21.15.12:3300/0 conn(0x7f9b1413d000 0x7f9b1413f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:32:00.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.446+0000 7f9b137fe640 1 -- 172.21.15.12:0/185732023 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9b141381c0 msgr2=0x7f9b141385a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:00.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.446+0000 7f9b137fe640 1 --2- 172.21.15.12:0/185732023 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9b141381c0 0x7f9b141385a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9b04009870 tx=0x7f9b0402a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=272556ebec468fac server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:00.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.446+0000 7f9b127fc640 1 -- 172.21.15.12:0/185732023 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b04037070 con 0x7f9b141381c0 2024-09-11T01:32:00.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.446+0000 7f9b127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:00.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.446+0000 7f9b127fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:32:00.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.446+0000 7f9b127fc640 10 monclient: dump: 2024-09-11T01:32:00.458 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:00.458 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.458 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.458 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.458 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:00.458 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.458 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:00.458 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:00.458 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:00.458 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.446+0000 7f9b127fc640 10 monclient: _finish_auth 0 2024-09-11T01:32:00.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.446+0000 7f9b127fc640 10 monclient: _check_auth_tickets 2024-09-11T01:32:00.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.446+0000 7f9b127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:00.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.446+0000 7f9b127fc640 1 -- 172.21.15.12:0/185732023 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b04032640 con 0x7f9b141381c0 2024-09-11T01:32:00.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.446+0000 7f9b19fc6640 5 monclient: authenticate success, global_id 4679 2024-09-11T01:32:00.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.446+0000 7f9b19fc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:32:00.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.446+0000 7f9b127fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:00.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.446+0000 7f9b19fc6640 10 monclient: get_monmap_and_config success 2024-09-11T01:32:00.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.446+0000 7f9b127fc640 1 -- 172.21.15.12:0/185732023 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b04032a10 con 0x7f9b141381c0 2024-09-11T01:32:00.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.446+0000 7f9b19fc6640 4 set_mon_vals no callback set 2024-09-11T01:32:00.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.446+0000 7f9b127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:00.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.446+0000 7f9b127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:32:00.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.446+0000 7f9b127fc640 10 monclient: dump: 2024-09-11T01:32:00.459 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:00.460 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.460 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.460 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.460 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:00.460 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.460 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:00.460 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:00.460 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:00.460 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.447+0000 7f9b19fc6640 10 monclient: shutdown 2024-09-11T01:32:00.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.447+0000 7f9b19fc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:00.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.447+0000 7f9b19fc6640 1 -- 172.21.15.12:0/185732023 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9b141381c0 msgr2=0x7f9b141385a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.447+0000 7f9b19fc6640 1 --2- 172.21.15.12:0/185732023 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9b141381c0 0x7f9b141385a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9b04009870 tx=0x7f9b0402a130 comp rx=0 tx=0).stop 2024-09-11T01:32:00.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.447+0000 7f9b19fc6640 1 -- 172.21.15.12:0/185732023 shutdown_connections 2024-09-11T01:32:00.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.447+0000 7f9b19fc6640 1 --2- 172.21.15.12:0/185732023 >> v2:172.21.15.12:3300/0 conn(0x7f9b1413d000 0x7f9b1413f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.447+0000 7f9b19fc6640 1 --2- 172.21.15.12:0/185732023 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9b141381c0 0x7f9b141385a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.447+0000 7f9b19fc6640 1 -- 172.21.15.12:0/185732023 >> 172.21.15.12:0/185732023 conn(0x7f9b140a7440 msgr2=0x7f9b140a7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:00.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.447+0000 7f9b19fc6640 1 -- 172.21.15.12:0/185732023 shutdown_connections 2024-09-11T01:32:00.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.447+0000 7f9b19fc6640 1 -- 172.21.15.12:0/185732023 wait complete. 2024-09-11T01:32:00.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.447+0000 7f9b19fc6640 10 monclient: build_initial_monmap 2024-09-11T01:32:00.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.447+0000 7f9b19fc6640 10 monclient: monmap: 2024-09-11T01:32:00.461 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:00.461 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:32:00.461 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:32:00.461 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:32:00.461 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:00.462 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.462 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:32:00.462 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:32:00.462 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:32:00.462 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b19fc6640 1 Processor -- start 2024-09-11T01:32:00.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b19fc6640 1 -- start start 2024-09-11T01:32:00.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b19fc6640 10 monclient: init 2024-09-11T01:32:00.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b19fc6640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:00.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b19fc6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:32:00.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b19fc6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b140a6aa0 0x7f9b141cd130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b19fc6640 10 monclient(hunting): picked mon.noname-a con 0x7f9b140a6aa0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:32:00.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b19fc6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9b141381c0 0x7f9b1413c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b19fc6640 10 monclient(hunting): picked mon.noname-b con 0x7f9b141381c0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:32:00.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b19fc6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9b1413d000 0x7f9b1413c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b19fc6640 10 monclient(hunting): picked mon.noname-c con 0x7f9b1413d000 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:00.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b19fc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b19fc6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1413fab0 con 0x7f9b140a6aa0 2024-09-11T01:32:00.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b19fc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b19fc6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1413f930 con 0x7f9b141381c0 2024-09-11T01:32:00.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b19fc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b19fc6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f9b1413fc30 con 0x7f9b1413d000 2024-09-11T01:32:00.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b19fc6640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:00.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b19fc6640 10 monclient(hunting): authenticate will time out at 926.890858s 2024-09-11T01:32:00.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b137fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b140a6aa0 0x7f9b141cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b137fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b140a6aa0 0x7f9b141cd130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50552/0 (socket says 172.21.15.12:50552) 2024-09-11T01:32:00.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b13fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9b1413d000 0x7f9b1413c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b137fe640 1 -- 172.21.15.12:0/2142712674 learned_addr learned my addr 172.21.15.12:0/2142712674 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:00.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b137fe640 10 monclient(hunting): get_auth_request con 0x7f9b140a6aa0 auth_method 0 2024-09-11T01:32:00.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:00.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b137fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:00.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b13fff640 10 monclient(hunting): get_auth_request con 0x7f9b1413d000 auth_method 0 2024-09-11T01:32:00.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:00.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b13fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.448+0000 7f9b13fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:00.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.449+0000 7f9b12ffd640 1 --2- 172.21.15.12:0/2142712674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9b141381c0 0x7f9b1413c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.449+0000 7f9b137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:00.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.449+0000 7f9b137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:00.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.449+0000 7f9b137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:00.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.449+0000 7f9b12ffd640 10 monclient(hunting): get_auth_request con 0x7f9b141381c0 auth_method 0 2024-09-11T01:32:00.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.449+0000 7f9b12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:00.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.449+0000 7f9b12ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.449+0000 7f9b12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:00.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.449+0000 7f9b13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:00.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.449+0000 7f9b13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:00.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.449+0000 7f9b13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:00.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.449+0000 7f9b12ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:00.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.449+0000 7f9b12ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:00.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.449+0000 7f9b12ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:00.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.449+0000 7f9b137fe640 10 monclient(hunting): handle_auth_done global_id 24258 payload 824 2024-09-11T01:32:00.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.449+0000 7f9b137fe640 1 -- 172.21.15.12:0/2142712674 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9b1413d000 msgr2=0x7f9b1413c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.449+0000 7f9b137fe640 1 --2- 172.21.15.12:0/2142712674 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9b1413d000 0x7f9b1413c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.449+0000 7f9b137fe640 1 -- 172.21.15.12:0/2142712674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9b141381c0 msgr2=0x7f9b1413c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.449+0000 7f9b137fe640 1 --2- 172.21.15.12:0/2142712674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9b141381c0 0x7f9b1413c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.449+0000 7f9b137fe640 10 monclient: _finish_hunting 0 2024-09-11T01:32:00.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.449+0000 7f9b137fe640 1 monclient: found mon.noname-a 2024-09-11T01:32:00.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.449+0000 7f9b137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:00.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.449+0000 7f9b137fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:32:00.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.449+0000 7f9b137fe640 1 -- 172.21.15.12:0/2142712674 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b141394e0 con 0x7f9b140a6aa0 2024-09-11T01:32:00.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.451+0000 7f9b137fe640 1 -- 172.21.15.12:0/2142712674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b140a6aa0 msgr2=0x7f9b141cd130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:00.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.451+0000 7f9b137fe640 1 --2- 172.21.15.12:0/2142712674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b140a6aa0 0x7f9b141cd130 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9b040090b0 tx=0x7f9b040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:00.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.451+0000 7f9afbfff640 1 -- 172.21.15.12:0/2142712674 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b04041020 con 0x7f9b140a6aa0 2024-09-11T01:32:00.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.451+0000 7f9afbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:00.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.451+0000 7f9afbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:32:00.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.451+0000 7f9afbfff640 10 monclient: dump: 2024-09-11T01:32:00.467 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:00.467 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.467 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.467 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.467 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:00.467 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.467 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:00.467 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:00.467 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:00.467 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.451+0000 7f9afbfff640 10 monclient: _finish_auth 0 2024-09-11T01:32:00.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.451+0000 7f9afbfff640 10 monclient: _check_auth_tickets 2024-09-11T01:32:00.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.451+0000 7f9afbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:00.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.451+0000 7f9afbfff640 1 -- 172.21.15.12:0/2142712674 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b0403d830 con 0x7f9b140a6aa0 2024-09-11T01:32:00.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.451+0000 7f9b19fc6640 5 monclient: authenticate success, global_id 24258 2024-09-11T01:32:00.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.451+0000 7f9b19fc6640 10 monclient: _renew_subs 2024-09-11T01:32:00.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.451+0000 7f9b19fc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:00.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.451+0000 7f9b19fc6640 1 -- 172.21.15.12:0/2142712674 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b140a1710 con 0x7f9b140a6aa0 2024-09-11T01:32:00.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.451+0000 7f9afbfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:00.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.451+0000 7f9afbfff640 1 -- 172.21.15.12:0/2142712674 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b04037040 con 0x7f9b140a6aa0 2024-09-11T01:32:00.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.451+0000 7f9afbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:00.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.451+0000 7f9afbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:00.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.451+0000 7f9afbfff640 10 monclient: dump: 2024-09-11T01:32:00.468 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:00.468 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.468 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.469 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.469 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:00.469 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.469 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:00.469 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:00.469 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:00.469 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.451+0000 7f9b19fc6640 10 monclient: _renew_subs 2024-09-11T01:32:00.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.451+0000 7f9b19fc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:00.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.451+0000 7f9b19fc6640 1 -- 172.21.15.12:0/2142712674 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b1409fe00 con 0x7f9b140a6aa0 2024-09-11T01:32:00.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.451+0000 7f9b11ffb640 4 set_mon_vals no callback set 2024-09-11T01:32:00.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.452+0000 7f9b19fc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:00.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.452+0000 7f9b19fc6640 1 --2- 172.21.15.12:0/2142712674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9b1409e3d0 0x7f9b1409e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.452+0000 7f9b19fc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.452+0000 7f9b19fc6640 1 -- 172.21.15.12:0/2142712674 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1413f400 con 0x7f9b1409e3d0 2024-09-11T01:32:00.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.452+0000 7f9afbfff640 1 -- 172.21.15.12:0/2142712674 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f9b0403a030 con 0x7f9b140a6aa0 2024-09-11T01:32:00.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.452+0000 7f9b12ffd640 1 --2- 172.21.15.12:0/2142712674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9b1409e3d0 0x7f9b1409e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.452+0000 7f9b12ffd640 10 monclient: get_auth_request con 0x7f9b1409e3d0 auth_method 0 2024-09-11T01:32:00.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.452+0000 7f9b12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:00.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.452+0000 7f9b12ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.452+0000 7f9b12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:00.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.452+0000 7f9b12ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:00.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.452+0000 7f9b12ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:00.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.452+0000 7f9afbfff640 1 --2- 172.21.15.12:0/2142712674 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9b0003e240 0x7f9b00040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.452+0000 7f9b12ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:00.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.453+0000 7f9afbfff640 1 -- 172.21.15.12:0/2142712674 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4767+0+0 (secure 0 0 0) 0x7f9b04072f40 con 0x7f9b140a6aa0 2024-09-11T01:32:00.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.453+0000 7f9b12ffd640 10 monclient(hunting): handle_auth_done global_id 34126 payload 824 2024-09-11T01:32:00.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.453+0000 7f9b12ffd640 1 -- 172.21.15.12:0/2142712674 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9b140a8aa0 con 0x7f9b1409e3d0 2024-09-11T01:32:00.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.453+0000 7f9b13fff640 1 --2- 172.21.15.12:0/2142712674 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9b0003e240 0x7f9b00040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.453+0000 7f9b13fff640 10 monclient: get_auth_request con 0x7f9b0003e240 auth_method 0 2024-09-11T01:32:00.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.455+0000 7f9b12ffd640 1 -- 172.21.15.12:0/2142712674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9b1409e3d0 msgr2=0x7f9b1409e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:00.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.455+0000 7f9b12ffd640 1 --2- 172.21.15.12:0/2142712674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9b1409e3d0 0x7f9b1409e7b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9b141ce1f0 tx=0x7f9b0800fc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:00.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.455+0000 7f9afbfff640 1 -- 172.21.15.12:0/2142712674 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b08018030 con 0x7f9b1409e3d0 2024-09-11T01:32:00.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.455+0000 7f9afbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:00.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.455+0000 7f9afbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:32:00.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.455+0000 7f9afbfff640 10 monclient: dump: 2024-09-11T01:32:00.471 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:00.471 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.472 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.472 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.472 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:00.472 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.472 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:00.472 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:00.472 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:00.472 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.455+0000 7f9b13fff640 1 --2- 172.21.15.12:0/2142712674 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9b0003e240 0x7f9b00040700 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9b1409dba0 tx=0x7f9b0c00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:00.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.455+0000 7f9afbfff640 1 -- 172.21.15.12:0/2142712674 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9b08003940 con 0x7f9b1409e3d0 2024-09-11T01:32:00.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.455+0000 7f9afbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:00.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.455+0000 7f9afbfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:32:00.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.455+0000 7f9afbfff640 1 -- 172.21.15.12:0/2142712674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9b1409e3d0 msgr2=0x7f9b1409e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.455+0000 7f9afbfff640 1 --2- 172.21.15.12:0/2142712674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9b1409e3d0 0x7f9b1409e7b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9b141ce1f0 tx=0x7f9b0800fc00 comp rx=0 tx=0).stop 2024-09-11T01:32:00.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.455+0000 7f9afbfff640 1 -- 172.21.15.12:0/2142712674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9b1409e3d0 msgr2=0x7f9b1409e7b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:32:00.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.455+0000 7f9afbfff640 1 --2- 172.21.15.12:0/2142712674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9b1409e3d0 0x7f9b1409e7b0 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9b141ce1f0 tx=0x7f9b0800fc00 comp rx=0 tx=0).stop 2024-09-11T01:32:00.474 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1331548 2024-09-11T01:32:00.474 INFO:tasks.rados.rados.0.smithi071.stdout:460: finishing copy_from to smithi07134777-16 2024-09-11T01:32:00.474 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 206 (ObjNum 171 snap 37 seq_num 171) dirty exists 2024-09-11T01:32:00.474 INFO:tasks.rados.rados.0.smithi071.stdout:463: finishing write tid 1 to smithi07134777-30 2024-09-11T01:32:00.474 INFO:tasks.rados.rados.0.smithi071.stdout:463: finishing write tid 2 to smithi07134777-30 2024-09-11T01:32:00.474 INFO:tasks.rados.rados.0.smithi071.stdout:463: finishing write tid 3 to smithi07134777-30 2024-09-11T01:32:00.474 INFO:tasks.rados.rados.0.smithi071.stdout:463: finishing write tid 4 to smithi07134777-30 2024-09-11T01:32:00.474 INFO:tasks.rados.rados.0.smithi071.stdout:463: oid 30 updating version 0 to 134 2024-09-11T01:32:00.475 INFO:tasks.rados.rados.0.smithi071.stdout:463: oid 30 updating version 134 to 135 2024-09-11T01:32:00.475 INFO:tasks.rados.rados.0.smithi071.stdout:463: oid 30 updating version 135 to 136 2024-09-11T01:32:00.475 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 136 (ObjNum 185 snap 41 seq_num 185) dirty exists 2024-09-11T01:32:00.475 INFO:tasks.rados.rados.0.smithi071.stdout:463: left oid 30 (ObjNum 185 snap 41 seq_num 185) 2024-09-11T01:32:00.475 INFO:tasks.rados.rados.0.smithi071.stdout:464: finishing write tid 1 to smithi07134777-48 2024-09-11T01:32:00.475 INFO:tasks.rados.rados.0.smithi071.stdout:464: finishing write tid 2 to smithi07134777-48 2024-09-11T01:32:00.475 INFO:tasks.rados.rados.0.smithi071.stdout:464: finishing write tid 3 to smithi07134777-48 2024-09-11T01:32:00.475 INFO:tasks.rados.rados.0.smithi071.stdout:464: finishing write tid 4 to smithi07134777-48 2024-09-11T01:32:00.475 INFO:tasks.rados.rados.0.smithi071.stdout:464: finishing write tid 5 to smithi07134777-48 2024-09-11T01:32:00.475 INFO:tasks.rados.rados.0.smithi071.stdout:464: oid 48 updating version 0 to 214 2024-09-11T01:32:00.475 INFO:tasks.rados.rados.0.smithi071.stdout:464: oid 48 version 214 is already newer than 212 2024-09-11T01:32:00.475 INFO:tasks.rados.rados.0.smithi071.stdout:464: oid 48 updating version 214 to 215 2024-09-11T01:32:00.475 INFO:tasks.rados.rados.0.smithi071.stdout:464: oid 48 version 215 is already newer than 213 2024-09-11T01:32:00.475 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 215 (ObjNum 186 snap 41 seq_num 186) dirty exists 2024-09-11T01:32:00.476 INFO:tasks.rados.rados.0.smithi071.stdout:464: left oid 48 (ObjNum 186 snap 41 seq_num 186) 2024-09-11T01:32:00.476 INFO:tasks.rados.rados.0.smithi071.stdout:465: finishing write tid 1 to smithi07134777-47 2024-09-11T01:32:00.476 INFO:tasks.rados.rados.0.smithi071.stdout:465: finishing write tid 2 to smithi07134777-47 2024-09-11T01:32:00.476 INFO:tasks.rados.rados.0.smithi071.stdout:465: finishing write tid 3 to smithi07134777-47 2024-09-11T01:32:00.476 INFO:tasks.rados.rados.0.smithi071.stdout:465: finishing write tid 4 to smithi07134777-47 2024-09-11T01:32:00.476 INFO:tasks.rados.rados.0.smithi071.stdout:465: oid 47 updating version 0 to 115 2024-09-11T01:32:00.476 INFO:tasks.rados.rados.0.smithi071.stdout:465: oid 47 updating version 115 to 117 2024-09-11T01:32:00.476 INFO:tasks.rados.rados.0.smithi071.stdout:465: oid 47 version 117 is already newer than 116 2024-09-11T01:32:00.476 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 117 (ObjNum 187 snap 41 seq_num 187) dirty exists 2024-09-11T01:32:00.476 INFO:tasks.rados.rados.0.smithi071.stdout:465: left oid 47 (ObjNum 187 snap 41 seq_num 187) 2024-09-11T01:32:00.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.461+0000 7f9b19fc6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:32:00.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.461+0000 7f9b19fc6640 1 --2- 172.21.15.12:0/2142712674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9b1409ecf0 0x7f9b1413d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.461+0000 7f9b19fc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.461+0000 7f9b19fc6640 1 -- 172.21.15.12:0/2142712674 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f9b140a09d0 con 0x7f9b1409ecf0 2024-09-11T01:32:00.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.462+0000 7f9b12ffd640 1 --2- 172.21.15.12:0/2142712674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9b1409ecf0 0x7f9b1413d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.462+0000 7f9b12ffd640 10 monclient: get_auth_request con 0x7f9b1409ecf0 auth_method 0 2024-09-11T01:32:00.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.462+0000 7f9b12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:00.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.462+0000 7f9b12ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.462+0000 7f9b12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:00.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.462+0000 7f9b12ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:00.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.462+0000 7f9b12ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:00.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.462+0000 7f9b12ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:00.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.462+0000 7f9b12ffd640 10 monclient(hunting): handle_auth_done global_id 34129 payload 824 2024-09-11T01:32:00.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.462+0000 7f9b12ffd640 1 -- 172.21.15.12:0/2142712674 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9b141385a0 con 0x7f9b1409ecf0 2024-09-11T01:32:00.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.464+0000 7f9b12ffd640 1 -- 172.21.15.12:0/2142712674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9b1409ecf0 msgr2=0x7f9b1413d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:00.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.464+0000 7f9b12ffd640 1 --2- 172.21.15.12:0/2142712674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9b1409ecf0 0x7f9b1413d860 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9b141ce1f0 tx=0x7f9b08003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:00.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.465+0000 7f9afbfff640 1 -- 172.21.15.12:0/2142712674 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b0801dbe0 con 0x7f9b1409ecf0 2024-09-11T01:32:00.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.465+0000 7f9afbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:00.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.465+0000 7f9afbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:32:00.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.465+0000 7f9afbfff640 10 monclient: dump: 2024-09-11T01:32:00.479 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:00.479 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.479 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.479 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.479 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:00.480 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.480 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:00.480 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:00.480 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:00.480 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.465+0000 7f9afbfff640 1 -- 172.21.15.12:0/2142712674 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4540 (secure 0 0 0) 0x7f9b0801dbe0 con 0x7f9b1409ecf0 2024-09-11T01:32:00.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.465+0000 7f9afbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:32:00.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.465+0000 7f9afbfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:32:00.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.465+0000 7f9afbfff640 1 -- 172.21.15.12:0/2142712674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9b1409ecf0 msgr2=0x7f9b1413d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.465+0000 7f9afbfff640 1 --2- 172.21.15.12:0/2142712674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9b1409ecf0 0x7f9b1413d860 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9b141ce1f0 tx=0x7f9b08003760 comp rx=0 tx=0).stop 2024-09-11T01:32:00.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.465+0000 7f9afbfff640 1 -- 172.21.15.12:0/2142712674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9b1409ecf0 msgr2=0x7f9b1413d860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:32:00.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.465+0000 7f9afbfff640 1 --2- 172.21.15.12:0/2142712674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9b1409ecf0 0x7f9b1413d860 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9b141ce1f0 tx=0x7f9b08003760 comp rx=0 tx=0).stop 2024-09-11T01:32:00.481 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:32:00.481 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:32:00.481 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:32:00.481 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:32:00.481 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 28, 2024-09-11T01:32:00.481 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 13678, 2024-09-11T01:32:00.481 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:32:00.481 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:32:00.482 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:32:00.482 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:32:00.482 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:00.482 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 7, 2024-09-11T01:32:00.482 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:32:00.482 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:32:00.482 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:32:00.482 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:00.482 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:00.482 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:00.482 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:00.482 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:00.482 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:00.482 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:00.483 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:00.483 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:00.483 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:00.483 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:00.483 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:00.483 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:32:00.483 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:32:00.483 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:00.483 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:00.483 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:00.483 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:00.483 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:00.483 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:00.484 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:00.484 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:00.484 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:00.484 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:00.484 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:00.484 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:00.484 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.484 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:32:00.484 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:32:00.484 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:32:00.485 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:32:00.485 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:32:00.485 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:32:00.485 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:32:00.485 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:32:00.485 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:32:00.485 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:32:00.485 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:32:00.485 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:32:00.485 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:32:00.485 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:32:00.485 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:32:00.485 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:32:00.486 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:32:00.486 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:00.486 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:00.486 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:00.486 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:00.486 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:00.486 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:00.486 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:00.486 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:00.486 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:00.486 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:00.486 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:00.486 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:00.486 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:32:00.487 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.487 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:32:00.487 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.487 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:32:00.487 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:32:00.487 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:00.487 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:00.487 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.487 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:00.487 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:32:00.487 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:00.487 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.487 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.487 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:00.488 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:32:00.488 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:00.488 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:00.488 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:00.488 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.488 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:32:00.488 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:32:00.488 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:00.488 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:00.489 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:00.489 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.489 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.489 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:32:00.489 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:32:00.489 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:00.489 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:00.489 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.489 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:00.489 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:32:00.489 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:00.489 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.490 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.490 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:00.490 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:32:00.490 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:00.490 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:00.490 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:00.490 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.490 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:32:00.490 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:32:00.490 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:00.490 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:00.490 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:00.490 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.491 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.491 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:32:00.491 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:32:00.491 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:00.491 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:00.491 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.491 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:00.491 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:32:00.491 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:00.491 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.491 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.491 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:00.491 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:32:00.491 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:00.492 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:00.492 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:00.492 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.492 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:32:00.492 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:32:00.492 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:00.492 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:00.492 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:00.492 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:00.492 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:00.492 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.492 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:32:00.492 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:32:00.493 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.493 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:00.493 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:00.493 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:32:00.493 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:00.493 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:00.493 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:32:00.493 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:00.493 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:00.493 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:00.493 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:32:00.493 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:00.493 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:00.493 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:00.494 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:32:00.494 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:32:00.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.465+0000 7f9b19fc6640 1 -- 172.21.15.12:0/2142712674 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9b0003e240 msgr2=0x7f9b00040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.465+0000 7f9b19fc6640 1 --2- 172.21.15.12:0/2142712674 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9b0003e240 0x7f9b00040700 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9b1409dba0 tx=0x7f9b0c00b040 comp rx=0 tx=0).stop 2024-09-11T01:32:00.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.465+0000 7f9b19fc6640 10 monclient: shutdown 2024-09-11T01:32:00.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.465+0000 7f9b19fc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:00.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.465+0000 7f9b19fc6640 1 -- 172.21.15.12:0/2142712674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b140a6aa0 msgr2=0x7f9b141cd130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.465+0000 7f9b19fc6640 1 --2- 172.21.15.12:0/2142712674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9b140a6aa0 0x7f9b141cd130 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9b040090b0 tx=0x7f9b040046c0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.465+0000 7f9b13fff640 1 -- 172.21.15.12:0/2142712674 reap_dead start 2024-09-11T01:32:00.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.465+0000 7f9b13fff640 1 -- 172.21.15.12:0/2142712674 reap_dead start 2024-09-11T01:32:00.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.465+0000 7f9b19fc6640 1 -- 172.21.15.12:0/2142712674 shutdown_connections 2024-09-11T01:32:00.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.465+0000 7f9b19fc6640 1 -- 172.21.15.12:0/2142712674 >> 172.21.15.12:0/2142712674 conn(0x7f9b140a7440 msgr2=0x7f9b14137520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:00.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.465+0000 7f9b19fc6640 1 -- 172.21.15.12:0/2142712674 shutdown_connections 2024-09-11T01:32:00.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.465+0000 7f9b19fc6640 1 -- 172.21.15.12:0/2142712674 wait complete. 2024-09-11T01:32:00.496 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:32:00.496 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-11T01:32:00.496 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T01:32:00.497 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-11T01:32:00.497 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-11T01:32:00.497 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-11T01:32:00.497 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:32:00.509 INFO:tasks.ceph.mon.c:Stopped 2024-09-11T01:32:00.509 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:32:00.509 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:32:00.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.649+0000 7f90540cf640 10 monclient: get_monmap_and_config 2024-09-11T01:32:00.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.649+0000 7f90540cf640 10 monclient: build_initial_monmap 2024-09-11T01:32:00.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.649+0000 7f90540cf640 10 monclient: monmap: 2024-09-11T01:32:00.651 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:00.651 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.651 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:32:00.649825+0000 2024-09-11T01:32:00.651 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:32:00.649825+0000 2024-09-11T01:32:00.651 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:00.651 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.651 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:32:00.651 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:32:00.651 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:32:00.651 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:32:00.651 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:32:00.651 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.649+0000 7f90540cf640 1 Processor -- start 2024-09-11T01:32:00.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.649+0000 7f90540cf640 1 -- start start 2024-09-11T01:32:00.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.649+0000 7f90540cf640 10 monclient: init 2024-09-11T01:32:00.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f90540cf640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:00.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f90540cf640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-11T01:32:00.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f90540cf640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f904c138230 0x7f904c138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f90540cf640 10 monclient(hunting): picked mon.noname-a con 0x7f904c138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:32:00.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f90540cf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f904c09d030 addr v1:172.21.15.71:6789/0 2024-09-11T01:32:00.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f90540cf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f904c09d460 addr v1:172.21.15.12:6789/0 2024-09-11T01:32:00.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f90540cf640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f904c0a5e40 con 0x7f904c09d460 2024-09-11T01:32:00.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f90540cf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f904c0a6e30 con 0x7f904c09d030 2024-09-11T01:32:00.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f90540cf640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f90540cf640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f904c09c9c0 con 0x7f904c138230 2024-09-11T01:32:00.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f90540cf640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:00.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f90540cf640 10 monclient(hunting): authenticate will time out at 927.092540s 2024-09-11T01:32:00.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f9051e6b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f904c138230 0x7f904c138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f905266c640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f904c09d460 0x7f904c0a5730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50732/0 (socket says 172.21.15.12:50732) 2024-09-11T01:32:00.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f9051e6b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f904c138230 0x7f904c138610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50588/0 (socket says 172.21.15.12:50588) 2024-09-11T01:32:00.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f905266c640 1 -- 172.21.15.12:0/624127781 learned_addr learned my addr 172.21.15.12:0/624127781 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:32:00.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f9051e6b640 10 monclient(hunting): get_auth_request con 0x7f904c138230 auth_method 0 2024-09-11T01:32:00.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f9051e6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:00.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f9051e6b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f9051e6b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:00.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f9051e6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:00.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f9051e6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:00.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f9051e6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:00.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f9051e6b640 10 monclient(hunting): handle_auth_done global_id 24261 payload 274 2024-09-11T01:32:00.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f9051e6b640 1 -- 172.21.15.12:0/624127781 >> v1:172.21.15.71:6789/0 conn(0x7f904c09d030 legacy=0x7f904c0a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:00.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f9051e6b640 1 -- 172.21.15.12:0/624127781 >> v1:172.21.15.12:6789/0 conn(0x7f904c09d460 legacy=0x7f904c0a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:00.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f9051e6b640 10 monclient: _finish_hunting 0 2024-09-11T01:32:00.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f9051e6b640 1 monclient: found mon.noname-a 2024-09-11T01:32:00.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f9051e6b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:00.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f9051e6b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:32:00.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.650+0000 7f9051e6b640 1 -- 172.21.15.12:0/624127781 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f904c09ba20 con 0x7f904c138230 2024-09-11T01:32:00.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.653+0000 7f9051e6b640 1 -- 172.21.15.12:0/624127781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f904c138230 msgr2=0x7f904c138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:00.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.653+0000 7f9051e6b640 1 --2- 172.21.15.12:0/624127781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f904c138230 0x7f904c138610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f90480097a0 tx=0x7f9048029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6afea1403a20127e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:00.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.653+0000 7f9050e69640 1 -- 172.21.15.12:0/624127781 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9048037070 con 0x7f904c138230 2024-09-11T01:32:00.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.653+0000 7f9050e69640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:00.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.653+0000 7f9050e69640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:32:00.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.653+0000 7f9050e69640 10 monclient: dump: 2024-09-11T01:32:00.662 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:00.662 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.662 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.662 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.663 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:00.663 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.663 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:00.663 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:00.663 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:00.663 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.653+0000 7f9050e69640 10 monclient: _finish_auth 0 2024-09-11T01:32:00.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.653+0000 7f9050e69640 10 monclient: _check_auth_tickets 2024-09-11T01:32:00.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.653+0000 7f9050e69640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:00.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.653+0000 7f90540cf640 5 monclient: authenticate success, global_id 24261 2024-09-11T01:32:00.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.653+0000 7f90540cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:32:00.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.653+0000 7f9050e69640 1 -- 172.21.15.12:0/624127781 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9048032640 con 0x7f904c138230 2024-09-11T01:32:00.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.653+0000 7f9050e69640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:00.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.653+0000 7f90540cf640 10 monclient: get_monmap_and_config success 2024-09-11T01:32:00.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.653+0000 7f90540cf640 4 set_mon_vals no callback set 2024-09-11T01:32:00.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.653+0000 7f9050e69640 1 -- 172.21.15.12:0/624127781 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9048032a10 con 0x7f904c138230 2024-09-11T01:32:00.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.653+0000 7f9050e69640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:00.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.653+0000 7f9050e69640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:00.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.653+0000 7f9050e69640 10 monclient: dump: 2024-09-11T01:32:00.664 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:00.664 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.664 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.665 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.665 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:00.665 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.665 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:00.665 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:00.665 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:00.665 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.654+0000 7f90540cf640 10 monclient: shutdown 2024-09-11T01:32:00.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.654+0000 7f90540cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:00.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.654+0000 7f90540cf640 1 -- 172.21.15.12:0/624127781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f904c138230 msgr2=0x7f904c138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.654+0000 7f90540cf640 1 --2- 172.21.15.12:0/624127781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f904c138230 0x7f904c138610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f90480097a0 tx=0x7f9048029fd0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.654+0000 7f90540cf640 1 -- 172.21.15.12:0/624127781 shutdown_connections 2024-09-11T01:32:00.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.654+0000 7f90540cf640 1 --2- 172.21.15.12:0/624127781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f904c138230 0x7f904c138610 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.654+0000 7f90540cf640 1 -- 172.21.15.12:0/624127781 >> 172.21.15.12:0/624127781 conn(0x7f904c0a7120 msgr2=0x7f904c0a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:00.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.654+0000 7f90540cf640 1 -- 172.21.15.12:0/624127781 shutdown_connections 2024-09-11T01:32:00.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.654+0000 7f90540cf640 1 -- 172.21.15.12:0/624127781 wait complete. 2024-09-11T01:32:00.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.655+0000 7f90540cf640 10 monclient: build_initial_monmap 2024-09-11T01:32:00.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.655+0000 7f90540cf640 10 monclient: monmap: 2024-09-11T01:32:00.666 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:00.666 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:32:00.666 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:32:00.666 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:32:00.666 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:00.666 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.666 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:32:00.666 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:32:00.666 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:32:00.666 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.655+0000 7f90540cf640 1 Processor -- start 2024-09-11T01:32:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.655+0000 7f90540cf640 1 -- start start 2024-09-11T01:32:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.655+0000 7f90540cf640 10 monclient: init 2024-09-11T01:32:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.655+0000 7f90540cf640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.655+0000 7f90540cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:32:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.655+0000 7f90540cf640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f904c09d030 0x7f904c133000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.655+0000 7f90540cf640 10 monclient(hunting): picked mon.noname-c con 0x7f904c09d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.655+0000 7f90540cf640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f904c09d460 0x7f904c133540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.655+0000 7f90540cf640 10 monclient(hunting): picked mon.noname-a con 0x7f904c09d460 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:32:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.655+0000 7f90540cf640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f904c138230 0x7f904c138e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.655+0000 7f90540cf640 10 monclient(hunting): picked mon.noname-b con 0x7f904c138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:32:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.655+0000 7f90540cf640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.655+0000 7f90540cf640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f904c06f7f0 con 0x7f904c09d460 2024-09-11T01:32:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.655+0000 7f90540cf640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.655+0000 7f90540cf640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f904c0a6730 con 0x7f904c138230 2024-09-11T01:32:00.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.655+0000 7f90540cf640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:00.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.655+0000 7f90540cf640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f904c06f970 con 0x7f904c09d030 2024-09-11T01:32:00.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.655+0000 7f90540cf640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:00.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.655+0000 7f90540cf640 10 monclient(hunting): authenticate will time out at 927.097907s 2024-09-11T01:32:00.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.655+0000 7f9051e6b640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f904c09d030 msgr2=0x7f904c133000 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:32:00.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.655+0000 7f9051e6b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f904c09d030 0x7f904c133000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:00.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.655+0000 7f905166a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f904c09d460 0x7f904c133540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.656+0000 7f905166a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f904c09d460 0x7f904c133540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50598/0 (socket says 172.21.15.12:50598) 2024-09-11T01:32:00.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.656+0000 7f905166a640 1 -- 172.21.15.12:0/2786941605 learned_addr learned my addr 172.21.15.12:0/2786941605 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:00.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.656+0000 7f905166a640 10 monclient(hunting): get_auth_request con 0x7f904c09d460 auth_method 0 2024-09-11T01:32:00.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.656+0000 7f905166a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:00.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.656+0000 7f905166a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:00.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.656+0000 7f905166a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:00.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.656+0000 7f905166a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:00.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.656+0000 7f905166a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:00.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.656+0000 7f905166a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:00.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.656+0000 7f905166a640 10 monclient(hunting): handle_auth_done global_id 24264 payload 824 2024-09-11T01:32:00.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.656+0000 7f905166a640 1 -- 172.21.15.12:0/2786941605 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f904c09d030 msgr2=0x7f904c133000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:32:00.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.656+0000 7f905166a640 1 --2- 172.21.15.12:0/2786941605 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f904c09d030 0x7f904c133000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.656+0000 7f905166a640 1 -- 172.21.15.12:0/2786941605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f904c138230 msgr2=0x7f904c138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.656+0000 7f905266c640 1 --2- 172.21.15.12:0/2786941605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f904c138230 0x7f904c138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.656+0000 7f905166a640 1 --2- 172.21.15.12:0/2786941605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f904c138230 0x7f904c138e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.656+0000 7f905166a640 10 monclient: _finish_hunting 0 2024-09-11T01:32:00.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.656+0000 7f905166a640 1 monclient: found mon.noname-a 2024-09-11T01:32:00.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.656+0000 7f905166a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:00.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.656+0000 7f905166a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:32:00.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.656+0000 7f905166a640 1 -- 172.21.15.12:0/2786941605 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f904c13c7b0 con 0x7f904c09d460 2024-09-11T01:32:00.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.656+0000 7f905266c640 10 monclient: get_auth_request con 0x7f904c138230 auth_method 0 2024-09-11T01:32:00.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.656+0000 7f905266c640 1 --2- 172.21.15.12:0/2786941605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f904c138230 0x7f904c138e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:32:00.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.658+0000 7f905166a640 1 -- 172.21.15.12:0/2786941605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f904c09d460 msgr2=0x7f904c133540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:00.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.658+0000 7f905166a640 1 --2- 172.21.15.12:0/2786941605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f904c09d460 0x7f904c133540 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f903c00b3f0 tx=0x7f903c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:00.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.658+0000 7f90427fc640 1 -- 172.21.15.12:0/2786941605 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903c015020 con 0x7f904c09d460 2024-09-11T01:32:00.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.658+0000 7f90427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:00.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.658+0000 7f90427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:32:00.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.658+0000 7f90427fc640 10 monclient: dump: 2024-09-11T01:32:00.670 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:00.670 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.670 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.670 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.671 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:00.671 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.671 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:00.671 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:00.671 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:00.671 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.658+0000 7f90427fc640 10 monclient: _finish_auth 0 2024-09-11T01:32:00.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.658+0000 7f90427fc640 10 monclient: _check_auth_tickets 2024-09-11T01:32:00.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.658+0000 7f90427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:00.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.658+0000 7f90427fc640 1 -- 172.21.15.12:0/2786941605 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f903c011a40 con 0x7f904c09d460 2024-09-11T01:32:00.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.658+0000 7f90540cf640 5 monclient: authenticate success, global_id 24264 2024-09-11T01:32:00.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.658+0000 7f90540cf640 10 monclient: _renew_subs 2024-09-11T01:32:00.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.658+0000 7f90540cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:00.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.658+0000 7f90540cf640 1 -- 172.21.15.12:0/2786941605 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f904c13ca90 con 0x7f904c09d460 2024-09-11T01:32:00.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.658+0000 7f90427fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:00.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.658+0000 7f90427fc640 1 -- 172.21.15.12:0/2786941605 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903c011e30 con 0x7f904c09d460 2024-09-11T01:32:00.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.658+0000 7f90427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:00.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.658+0000 7f90427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:00.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.658+0000 7f90427fc640 10 monclient: dump: 2024-09-11T01:32:00.672 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:00.672 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:00.672 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.672 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:00.672 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:00.672 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:00.672 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:00.672 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:00.673 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:00.673 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:00.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.658+0000 7f90540cf640 10 monclient: _renew_subs 2024-09-11T01:32:00.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.658+0000 7f9043fff640 4 set_mon_vals no callback set 2024-09-11T01:32:00.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.658+0000 7f90540cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:00.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.658+0000 7f90540cf640 1 -- 172.21.15.12:0/2786941605 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f904c1408e0 con 0x7f904c09d460 2024-09-11T01:32:00.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.659+0000 7f90427fc640 1 -- 172.21.15.12:0/2786941605 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f903c01a400 con 0x7f904c09d460 2024-09-11T01:32:00.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.659+0000 7f90540cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:00.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.659+0000 7f90540cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:00.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.659+0000 7f90540cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:00.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.659+0000 7f90540cf640 1 -- 172.21.15.12:0/2786941605 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f904c138610 con 0x7f904c09d460 2024-09-11T01:32:00.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.659+0000 7f90427fc640 1 --2- 172.21.15.12:0/2786941605 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f903003e1c0 0x7f9030040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:00.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.659+0000 7f90427fc640 1 -- 172.21.15.12:0/2786941605 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4767+0+0 (secure 0 0 0) 0x7f903c052df0 con 0x7f904c09d460 2024-09-11T01:32:00.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.660+0000 7f9051e6b640 1 --2- 172.21.15.12:0/2786941605 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f903003e1c0 0x7f9030040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:00.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.660+0000 7f9051e6b640 10 monclient: get_auth_request con 0x7f903003e1c0 auth_method 0 2024-09-11T01:32:00.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.662+0000 7f9051e6b640 1 --2- 172.21.15.12:0/2786941605 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f903003e1c0 0x7f9030040680 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f904c09bff0 tx=0x7f904802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:00.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.664+0000 7f90427fc640 1 -- 172.21.15.12:0/2786941605 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f903c056030 con 0x7f904c09d460 2024-09-11T01:32:00.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.664+0000 7f90427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:00.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.664+0000 7f90427fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:32:00.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.920+0000 7f90540cf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:32:00.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.920+0000 7f90540cf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:32:00.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.920+0000 7f90540cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:00.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.920+0000 7f90540cf640 1 -- 172.21.15.12:0/2786941605 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f904c0698e0 con 0x7f904c09d460 2024-09-11T01:32:00.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.920+0000 7f90427fc640 1 -- 172.21.15.12:0/2786941605 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f903c01f030 con 0x7f904c09d460 2024-09-11T01:32:00.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.920+0000 7f90427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:32:00.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.920+0000 7f90427fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:32:00.922 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:32:00.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.922+0000 7f90540cf640 1 -- 172.21.15.12:0/2786941605 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f903003e1c0 msgr2=0x7f9030040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.922+0000 7f90540cf640 1 --2- 172.21.15.12:0/2786941605 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f903003e1c0 0x7f9030040680 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f904c09bff0 tx=0x7f904802e000 comp rx=0 tx=0).stop 2024-09-11T01:32:00.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.922+0000 7f90540cf640 10 monclient: shutdown 2024-09-11T01:32:00.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.922+0000 7f90540cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:00.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.922+0000 7f90540cf640 1 -- 172.21.15.12:0/2786941605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f904c09d460 msgr2=0x7f904c133540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:00.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.922+0000 7f90540cf640 1 --2- 172.21.15.12:0/2786941605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f904c09d460 0x7f904c133540 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f903c00b3f0 tx=0x7f903c00b8b0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.922+0000 7f90540cf640 1 -- 172.21.15.12:0/2786941605 shutdown_connections 2024-09-11T01:32:00.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.922+0000 7f90540cf640 1 --2- 172.21.15.12:0/2786941605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f904c138230 0x7f904c138e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.922+0000 7f90540cf640 1 --2- 172.21.15.12:0/2786941605 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f903003e1c0 0x7f9030040680 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.922+0000 7f90540cf640 1 --2- 172.21.15.12:0/2786941605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f904c09d460 0x7f904c133540 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.922+0000 7f90540cf640 1 --2- 172.21.15.12:0/2786941605 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f904c09d030 0x7f904c133000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:00.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.922+0000 7f90540cf640 1 -- 172.21.15.12:0/2786941605 >> 172.21.15.12:0/2786941605 conn(0x7f904c0a7120 msgr2=0x7f904c0a3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:00.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.923+0000 7f90540cf640 1 -- 172.21.15.12:0/2786941605 shutdown_connections 2024-09-11T01:32:00.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:00.923+0000 7f90540cf640 1 -- 172.21.15.12:0/2786941605 wait complete. 2024-09-11T01:32:03.935 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:32:04.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.047+0000 7fd5e868c640 10 monclient: get_monmap_and_config 2024-09-11T01:32:04.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.047+0000 7fd5e868c640 10 monclient: build_initial_monmap 2024-09-11T01:32:04.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.047+0000 7fd5e868c640 10 monclient: monmap: 2024-09-11T01:32:04.049 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:04.049 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:04.049 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:32:04.047827+0000 2024-09-11T01:32:04.049 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:32:04.047827+0000 2024-09-11T01:32:04.049 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:04.049 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:04.049 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:32:04.049 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:32:04.049 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:32:04.049 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:32:04.049 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:32:04.050 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:04.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.047+0000 7fd5e868c640 1 Processor -- start 2024-09-11T01:32:04.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.047+0000 7fd5e868c640 1 -- start start 2024-09-11T01:32:04.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.047+0000 7fd5e868c640 10 monclient: init 2024-09-11T01:32:04.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.048+0000 7fd5e868c640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:04.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.048+0000 7fd5e868c640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-11T01:32:04.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.048+0000 7fd5e868c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd5e0138230 addr v1:172.21.15.12:6789/0 2024-09-11T01:32:04.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.048+0000 7fd5e868c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd5e009d030 addr v1:172.21.15.71:6789/0 2024-09-11T01:32:04.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.048+0000 7fd5e868c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd5e009d460 0x7fd5e01407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:04.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.048+0000 7fd5e868c640 10 monclient(hunting): picked mon.noname-c con 0x7fd5e009d460 addr v2:172.21.15.71:3300/0 2024-09-11T01:32:04.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.048+0000 7fd5e868c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5e00a6e30 con 0x7fd5e0138230 2024-09-11T01:32:04.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.048+0000 7fd5e868c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5e009ba20 con 0x7fd5e009d030 2024-09-11T01:32:04.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.048+0000 7fd5e868c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:04.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.048+0000 7fd5e868c640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fd5e009c9c0 con 0x7fd5e009d460 2024-09-11T01:32:04.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.048+0000 7fd5e868c640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:04.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.048+0000 7fd5e868c640 10 monclient(hunting): authenticate will time out at 930.490559s 2024-09-11T01:32:04.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.048+0000 7fd5e6428640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd5e0138230 0x7fd5e0138610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50748/0 (socket says 172.21.15.12:50748) 2024-09-11T01:32:04.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.048+0000 7fd5e6428640 1 -- 172.21.15.12:0/829036046 learned_addr learned my addr 172.21.15.12:0/829036046 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:32:04.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.048+0000 7fd5e6c29640 1 --2- 172.21.15.12:0/829036046 >> v2:172.21.15.71:3300/0 conn(0x7fd5e009d460 0x7fd5e01407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:04.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.048+0000 7fd5e6c29640 10 monclient(hunting): get_auth_request con 0x7fd5e009d460 auth_method 0 2024-09-11T01:32:04.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.048+0000 7fd5e6c29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:04.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.048+0000 7fd5e6c29640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:04.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.048+0000 7fd5e6c29640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:04.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.050+0000 7fd5e6428640 1 -- 172.21.15.12:0/829036046 >> v1:172.21.15.12:6789/0 conn(0x7fd5e0138230 legacy=0x7fd5e0138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:04.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 1 -- 172.21.15.12:0/829036046 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2417747142 0 0) 0x7fd5e00a6e30 con 0x7fd5e0138230 2024-09-11T01:32:04.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 10 monclient(hunting): _negotiate 2024-09-11T01:32:04.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:04.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:04.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 10 monclient(hunting): my global_id is 24267 2024-09-11T01:32:04.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 1 -- 172.21.15.12:0/829036046 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5cc003520 con 0x7fd5e0138230 2024-09-11T01:32:04.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5c27640 1 -- 172.21.15.12:0/829036046 >> v1:172.21.15.71:6789/0 conn(0x7fd5e009d030 legacy=0x7fd5e013d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:04.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e6c29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:04.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e6c29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:04.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e6c29640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:04.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 1 -- 172.21.15.12:0/829036046 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1940106550 0 0) 0x7fd5cc003520 con 0x7fd5e0138230 2024-09-11T01:32:04.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 1 -- 172.21.15.12:0/829036046 >> v2:172.21.15.71:3300/0 conn(0x7fd5e009d460 msgr2=0x7fd5e01407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:04.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 1 --2- 172.21.15.12:0/829036046 >> v2:172.21.15.71:3300/0 conn(0x7fd5e009d460 0x7fd5e01407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:04.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 1 -- 172.21.15.12:0/829036046 >> v1:172.21.15.71:6789/0 conn(0x7fd5e009d030 legacy=0x7fd5e013d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:04.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 10 monclient: _finish_hunting 0 2024-09-11T01:32:04.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:32:04.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:04.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:32:04.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 1 -- 172.21.15.12:0/829036046 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5e00a66a0 con 0x7fd5e0138230 2024-09-11T01:32:04.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 10 monclient: _finish_auth 0 2024-09-11T01:32:04.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 10 monclient: _check_auth_tickets 2024-09-11T01:32:04.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:04.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 1 -- 172.21.15.12:0/829036046 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7fd5dc0019a0 con 0x7fd5e0138230 2024-09-11T01:32:04.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:04.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:32:04.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 10 monclient: dump: 2024-09-11T01:32:04.065 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:04.066 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:04.066 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:04.066 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:04.066 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:04.066 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:04.066 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:04.066 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:04.066 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:04.066 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:04.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:32:04.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:04.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 1 -- 172.21.15.12:0/829036046 >> v1:172.21.15.12:6789/0 conn(0x7fd5e0138230 legacy=0x7fd5e0138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:04.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:32:04.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 1 --2- 172.21.15.12:0/829036046 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5cc003b20 0x7fd5cc003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:04.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 10 monclient(hunting): picked mon.b con 0x7fd5cc003b20 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:32:04.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 1 --2- 172.21.15.12:0/829036046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5cc004800 0x7fd5cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:04.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 10 monclient(hunting): picked mon.a con 0x7fd5cc004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:32:04.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 1 --2- 172.21.15.12:0/829036046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5cc0053c0 0x7fd5cc00ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:04.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 10 monclient(hunting): picked mon.c con 0x7fd5cc0053c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:04.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:04.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 1 -- 172.21.15.12:0/829036046 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e009c9c0 con 0x7fd5cc004800 2024-09-11T01:32:04.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:04.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 1 -- 172.21.15.12:0/829036046 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd5cc00c300 con 0x7fd5cc003b20 2024-09-11T01:32:04.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:04.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 1 -- 172.21.15.12:0/829036046 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd5cc00c500 con 0x7fd5cc0053c0 2024-09-11T01:32:04.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:04.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5426640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:32:04.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e6c29640 1 -- 172.21.15.12:0/829036046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5cc0053c0 msgr2=0x7fd5cc00ba20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:32:04.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e6c29640 1 --2- 172.21.15.12:0/829036046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5cc0053c0 0x7fd5cc00ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:04.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5c27640 1 --2- 172.21.15.12:0/829036046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5cc004800 0x7fd5cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:04.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e6428640 1 --2- 172.21.15.12:0/829036046 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5cc003b20 0x7fd5cc003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:04.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5c27640 10 monclient(hunting): get_auth_request con 0x7fd5cc004800 auth_method 0 2024-09-11T01:32:04.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5c27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:04.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5c27640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:04.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5c27640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:32:04.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e6428640 10 monclient(hunting): get_auth_request con 0x7fd5cc003b20 auth_method 0 2024-09-11T01:32:04.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e6428640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:04.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e6428640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:04.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e6428640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:32:04.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5c27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:04.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5c27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:04.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5c27640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:32:04.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e6428640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:04.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e6428640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:04.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e6428640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:32:04.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5c27640 10 monclient(hunting): handle_auth_done global_id 24267 payload 293 2024-09-11T01:32:04.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5c27640 1 -- 172.21.15.12:0/829036046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5cc0053c0 msgr2=0x7fd5cc00ba20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:32:04.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5c27640 1 --2- 172.21.15.12:0/829036046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5cc0053c0 0x7fd5cc00ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:04.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5c27640 1 -- 172.21.15.12:0/829036046 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5cc003b20 msgr2=0x7fd5cc003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:04.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5c27640 1 --2- 172.21.15.12:0/829036046 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5cc003b20 0x7fd5cc003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:04.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5c27640 10 monclient: _finish_hunting 0 2024-09-11T01:32:04.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5c27640 1 monclient: found mon.a 2024-09-11T01:32:04.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e6c29640 1 -- 172.21.15.12:0/829036046 reap_dead start 2024-09-11T01:32:04.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5c27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:04.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5c27640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:04.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5c27640 1 -- 172.21.15.12:0/829036046 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5cc003520 con 0x7fd5cc004800 2024-09-11T01:32:04.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5c27640 10 monclient: _finish_auth 0 2024-09-11T01:32:04.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5c27640 10 monclient: _check_auth_tickets 2024-09-11T01:32:04.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e5c27640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:04.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e868c640 5 monclient: authenticate success, global_id 24267 2024-09-11T01:32:04.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e868c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:32:04.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.051+0000 7fd5e6428640 1 --2- 172.21.15.12:0/829036046 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5cc003b20 0x7fd5cc003f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:32:04.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.054+0000 7fd5e5c27640 1 -- 172.21.15.12:0/829036046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5cc004800 msgr2=0x7fd5cc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:04.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.054+0000 7fd5e5c27640 1 --2- 172.21.15.12:0/829036046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5cc004800 0x7fd5cc004c80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0001990 tx=0x7fd5cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=857df6388b6a6a4e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:04.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.054+0000 7fd5e5426640 1 -- 172.21.15.12:0/829036046 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d0001690 con 0x7fd5cc004800 2024-09-11T01:32:04.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.054+0000 7fd5e5426640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:04.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.054+0000 7fd5e5426640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:04.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.054+0000 7fd5e5426640 10 monclient: dump: 2024-09-11T01:32:04.071 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:04.071 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:04.071 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:04.071 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:04.071 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:04.071 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:04.072 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:04.072 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:04.072 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:04.072 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:04.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.054+0000 7fd5e5426640 1 -- 172.21.15.12:0/829036046 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5d002e070 con 0x7fd5cc004800 2024-09-11T01:32:04.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.054+0000 7fd5e868c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:32:04.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.054+0000 7fd5e5426640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:04.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.054+0000 7fd5e5426640 1 -- 172.21.15.12:0/829036046 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d0007070 con 0x7fd5cc004800 2024-09-11T01:32:04.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.054+0000 7fd5e868c640 10 monclient: get_monmap_and_config success 2024-09-11T01:32:04.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.054+0000 7fd5e868c640 4 set_mon_vals no callback set 2024-09-11T01:32:04.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.054+0000 7fd5e5426640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:04.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.054+0000 7fd5e5426640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:04.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.054+0000 7fd5e5426640 10 monclient: dump: 2024-09-11T01:32:04.072 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:04.073 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:04.073 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:04.073 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:04.073 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:04.073 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:04.074 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:04.074 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:04.074 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:04.074 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:04.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.055+0000 7fd5e868c640 10 monclient: shutdown 2024-09-11T01:32:04.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.055+0000 7fd5e868c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:04.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.055+0000 7fd5e868c640 1 -- 172.21.15.12:0/829036046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5cc004800 msgr2=0x7fd5cc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:04.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.055+0000 7fd5e868c640 1 --2- 172.21.15.12:0/829036046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5cc004800 0x7fd5cc004c80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0001990 tx=0x7fd5cc005830 comp rx=0 tx=0).stop 2024-09-11T01:32:04.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.055+0000 7fd5e868c640 1 -- 172.21.15.12:0/829036046 shutdown_connections 2024-09-11T01:32:04.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.055+0000 7fd5e868c640 1 --2- 172.21.15.12:0/829036046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5cc004800 0x7fd5cc004c80 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:04.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.055+0000 7fd5e868c640 1 -- 172.21.15.12:0/829036046 >> 172.21.15.12:0/829036046 conn(0x7fd5e00a7120 msgr2=0x7fd5e00a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:04.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.055+0000 7fd5e868c640 1 -- 172.21.15.12:0/829036046 shutdown_connections 2024-09-11T01:32:04.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.055+0000 7fd5e868c640 1 -- 172.21.15.12:0/829036046 wait complete. 2024-09-11T01:32:04.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.056+0000 7fd5e868c640 10 monclient: build_initial_monmap 2024-09-11T01:32:04.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.056+0000 7fd5e868c640 10 monclient: monmap: 2024-09-11T01:32:04.075 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:04.075 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:32:04.075 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:32:04.075 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:32:04.075 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:04.075 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:04.076 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:32:04.076 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:32:04.076 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:32:04.076 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:04.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.056+0000 7fd5e868c640 1 Processor -- start 2024-09-11T01:32:04.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.056+0000 7fd5e868c640 1 -- start start 2024-09-11T01:32:04.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.056+0000 7fd5e868c640 10 monclient: init 2024-09-11T01:32:04.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e868c640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:04.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e868c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:32:04.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e868c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5cc004800 0x7fd5e0136fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:04.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e868c640 10 monclient(hunting): picked mon.noname-b con 0x7fd5cc004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:32:04.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e868c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5e01cd170 0x7fd5e01374f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:04.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e868c640 10 monclient(hunting): picked mon.noname-a con 0x7fd5e01cd170 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:32:04.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e868c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5e0120c70 0x7fd5e0121120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:04.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e868c640 10 monclient(hunting): picked mon.noname-c con 0x7fd5e0120c70 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:04.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e868c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:04.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e868c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e0138790 con 0x7fd5e01cd170 2024-09-11T01:32:04.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e868c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:04.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e868c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e0138610 con 0x7fd5cc004800 2024-09-11T01:32:04.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e868c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:04.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e868c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd5e0138910 con 0x7fd5e0120c70 2024-09-11T01:32:04.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e868c640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:04.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e868c640 10 monclient(hunting): authenticate will time out at 930.499311s 2024-09-11T01:32:04.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e6c29640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5e0120c70 msgr2=0x7fd5e0121120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:32:04.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e6c29640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5e0120c70 0x7fd5e0121120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:04.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e5c27640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5e01cd170 0x7fd5e01374f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:04.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e5c27640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5e01cd170 0x7fd5e01374f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50620/0 (socket says 172.21.15.12:50620) 2024-09-11T01:32:04.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e5c27640 1 -- 172.21.15.12:0/2968489035 learned_addr learned my addr 172.21.15.12:0/2968489035 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:04.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e5c27640 10 monclient(hunting): get_auth_request con 0x7fd5e01cd170 auth_method 0 2024-09-11T01:32:04.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e5c27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:04.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e5c27640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:04.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e5c27640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:04.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e6428640 1 --2- 172.21.15.12:0/2968489035 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5cc004800 0x7fd5e0136fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:04.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e6428640 10 monclient(hunting): get_auth_request con 0x7fd5cc004800 auth_method 0 2024-09-11T01:32:04.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e6428640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:04.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e6428640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:04.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e6428640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:04.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e5c27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:04.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e5c27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:04.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e5c27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:04.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e6428640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:04.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e6428640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:04.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e6428640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:04.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e5c27640 10 monclient(hunting): handle_auth_done global_id 24270 payload 824 2024-09-11T01:32:04.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e5c27640 1 -- 172.21.15.12:0/2968489035 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5e0120c70 msgr2=0x7fd5e0121120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:32:04.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e5c27640 1 --2- 172.21.15.12:0/2968489035 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5e0120c70 0x7fd5e0121120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:04.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e5c27640 1 -- 172.21.15.12:0/2968489035 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5cc004800 msgr2=0x7fd5e0136fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:04.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e5c27640 1 --2- 172.21.15.12:0/2968489035 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5cc004800 0x7fd5e0136fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:04.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e5c27640 10 monclient: _finish_hunting 0 2024-09-11T01:32:04.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e5c27640 1 monclient: found mon.noname-a 2024-09-11T01:32:04.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e5c27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:04.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e5c27640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:32:04.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.057+0000 7fd5e5c27640 1 -- 172.21.15.12:0/2968489035 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5e0122610 con 0x7fd5e01cd170 2024-09-11T01:32:04.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.059+0000 7fd5e5c27640 1 -- 172.21.15.12:0/2968489035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5e01cd170 msgr2=0x7fd5e01374f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:04.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.059+0000 7fd5e5c27640 1 --2- 172.21.15.12:0/2968489035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5e01cd170 0x7fd5e01374f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0001110 tx=0x7fd5d0001520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:04.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.060+0000 7fd5d6ffd640 1 -- 172.21.15.12:0/2968489035 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d0001690 con 0x7fd5e01cd170 2024-09-11T01:32:04.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.060+0000 7fd5d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:04.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.060+0000 7fd5d6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:32:04.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.060+0000 7fd5d6ffd640 10 monclient: dump: 2024-09-11T01:32:04.081 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:04.081 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:04.081 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:04.081 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:04.081 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:04.081 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:04.081 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:04.081 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:04.081 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:04.081 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:04.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.060+0000 7fd5d6ffd640 10 monclient: _finish_auth 0 2024-09-11T01:32:04.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.060+0000 7fd5d6ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:32:04.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.060+0000 7fd5d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:04.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.060+0000 7fd5d6ffd640 1 -- 172.21.15.12:0/2968489035 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5d00059c0 con 0x7fd5e01cd170 2024-09-11T01:32:04.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.060+0000 7fd5e868c640 5 monclient: authenticate success, global_id 24270 2024-09-11T01:32:04.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.060+0000 7fd5e868c640 10 monclient: _renew_subs 2024-09-11T01:32:04.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.060+0000 7fd5e868c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:04.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.060+0000 7fd5e868c640 1 -- 172.21.15.12:0/2968489035 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5e00a1810 con 0x7fd5e01cd170 2024-09-11T01:32:04.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.060+0000 7fd5d6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:04.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.060+0000 7fd5d6ffd640 1 -- 172.21.15.12:0/2968489035 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d0005d90 con 0x7fd5e01cd170 2024-09-11T01:32:04.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.060+0000 7fd5d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:04.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.060+0000 7fd5d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:04.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.060+0000 7fd5d6ffd640 10 monclient: dump: 2024-09-11T01:32:04.082 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:04.082 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:04.083 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:04.083 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:04.083 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:04.083 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:04.083 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:04.083 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:04.083 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:04.083 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:04.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.060+0000 7fd5e4c25640 4 set_mon_vals no callback set 2024-09-11T01:32:04.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.060+0000 7fd5e868c640 10 monclient: _renew_subs 2024-09-11T01:32:04.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.060+0000 7fd5e868c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:04.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.060+0000 7fd5e868c640 1 -- 172.21.15.12:0/2968489035 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5e00a13c0 con 0x7fd5e01cd170 2024-09-11T01:32:04.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.061+0000 7fd5d6ffd640 1 -- 172.21.15.12:0/2968489035 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fd5d0033470 con 0x7fd5e01cd170 2024-09-11T01:32:04.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.061+0000 7fd5e868c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:04.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.061+0000 7fd5e868c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:04.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.061+0000 7fd5e868c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:04.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.061+0000 7fd5e868c640 1 -- 172.21.15.12:0/2968489035 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5e0076fc0 con 0x7fd5e01cd170 2024-09-11T01:32:04.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.061+0000 7fd5d6ffd640 1 --2- 172.21.15.12:0/2968489035 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd5c403e1c0 0x7fd5c4040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:04.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.061+0000 7fd5d6ffd640 1 -- 172.21.15.12:0/2968489035 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4767+0+0 (secure 0 0 0) 0x7fd5d00727b0 con 0x7fd5e01cd170 2024-09-11T01:32:04.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.061+0000 7fd5e6428640 1 --2- 172.21.15.12:0/2968489035 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd5c403e1c0 0x7fd5c4040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:04.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.061+0000 7fd5e6428640 10 monclient: get_auth_request con 0x7fd5c403e1c0 auth_method 0 2024-09-11T01:32:04.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.064+0000 7fd5e6428640 1 --2- 172.21.15.12:0/2968489035 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd5c403e1c0 0x7fd5c4040680 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc007f30 tx=0x7fd5dc007ec0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:04.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.065+0000 7fd5d6ffd640 1 -- 172.21.15.12:0/2968489035 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd5e0076fc0 con 0x7fd5e01cd170 2024-09-11T01:32:04.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.065+0000 7fd5d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:04.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.065+0000 7fd5d6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:32:04.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.324+0000 7fd5e868c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:32:04.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.324+0000 7fd5e868c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:32:04.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.324+0000 7fd5e868c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:04.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.324+0000 7fd5e868c640 1 -- 172.21.15.12:0/2968489035 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd5e00a5020 con 0x7fd5e01cd170 2024-09-11T01:32:04.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.325+0000 7fd5d6ffd640 1 -- 172.21.15.12:0/2968489035 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fd5e00a1810 con 0x7fd5e01cd170 2024-09-11T01:32:04.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.325+0000 7fd5d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:32:04.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.325+0000 7fd5d6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:32:04.327 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:32:04.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.327+0000 7fd5e868c640 1 -- 172.21.15.12:0/2968489035 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd5c403e1c0 msgr2=0x7fd5c4040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:04.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.327+0000 7fd5e868c640 1 --2- 172.21.15.12:0/2968489035 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd5c403e1c0 0x7fd5c4040680 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc007f30 tx=0x7fd5dc007ec0 comp rx=0 tx=0).stop 2024-09-11T01:32:04.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.327+0000 7fd5e868c640 10 monclient: shutdown 2024-09-11T01:32:04.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.327+0000 7fd5e868c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:04.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.327+0000 7fd5e868c640 1 -- 172.21.15.12:0/2968489035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5e01cd170 msgr2=0x7fd5e01374f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:04.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.327+0000 7fd5e868c640 1 --2- 172.21.15.12:0/2968489035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5e01cd170 0x7fd5e01374f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0001110 tx=0x7fd5d0001520 comp rx=0 tx=0).stop 2024-09-11T01:32:04.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.327+0000 7fd5e868c640 1 -- 172.21.15.12:0/2968489035 shutdown_connections 2024-09-11T01:32:04.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.327+0000 7fd5e868c640 1 --2- 172.21.15.12:0/2968489035 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5e0120c70 0x7fd5e0121120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:04.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.327+0000 7fd5e868c640 1 --2- 172.21.15.12:0/2968489035 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd5c403e1c0 0x7fd5c4040680 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:04.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.327+0000 7fd5e868c640 1 --2- 172.21.15.12:0/2968489035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5e01cd170 0x7fd5e01374f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:04.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.327+0000 7fd5e868c640 1 --2- 172.21.15.12:0/2968489035 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5cc004800 0x7fd5e0136fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:04.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.327+0000 7fd5e868c640 1 -- 172.21.15.12:0/2968489035 >> 172.21.15.12:0/2968489035 conn(0x7fd5e00a7120 msgr2=0x7fd5e00a3cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:04.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.327+0000 7fd5e868c640 1 -- 172.21.15.12:0/2968489035 shutdown_connections 2024-09-11T01:32:04.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:04.327+0000 7fd5e868c640 1 -- 172.21.15.12:0/2968489035 wait complete. 2024-09-11T01:32:07.341 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:32:07.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.452+0000 7fb74140c640 10 monclient: get_monmap_and_config 2024-09-11T01:32:07.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.452+0000 7fb74140c640 10 monclient: build_initial_monmap 2024-09-11T01:32:07.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.453+0000 7fb74140c640 10 monclient: monmap: 2024-09-11T01:32:07.455 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:07.455 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:07.455 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:32:07.453728+0000 2024-09-11T01:32:07.455 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:32:07.453728+0000 2024-09-11T01:32:07.455 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:07.455 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:07.455 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:32:07.455 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:32:07.455 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:32:07.455 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:32:07.455 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:32:07.455 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:07.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.453+0000 7fb74140c640 1 Processor -- start 2024-09-11T01:32:07.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.453+0000 7fb74140c640 1 -- start start 2024-09-11T01:32:07.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.453+0000 7fb74140c640 10 monclient: init 2024-09-11T01:32:07.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb74140c640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:07.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb74140c640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-11T01:32:07.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb74140c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb73c138230 addr v1:172.21.15.12:6789/0 2024-09-11T01:32:07.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb74140c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb73c09d030 0x7fb73c13d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:07.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb74140c640 10 monclient(hunting): picked mon.noname-b con 0x7fb73c09d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:07.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb74140c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb73c09d460 0x7fb73c13f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:07.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb74140c640 10 monclient(hunting): picked mon.noname-c con 0x7fb73c09d460 addr v2:172.21.15.71:3300/0 2024-09-11T01:32:07.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb74140c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb73c0a6e30 con 0x7fb73c138230 2024-09-11T01:32:07.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb74140c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:07.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb74140c640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fb73c09c9c0 con 0x7fb73c09d460 2024-09-11T01:32:07.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb74140c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:07.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb74140c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb73c09ba20 con 0x7fb73c09d030 2024-09-11T01:32:07.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb74140c640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:07.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb74140c640 10 monclient(hunting): authenticate will time out at 933.896227s 2024-09-11T01:32:07.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb73a7fc640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb73c09d030 msgr2=0x7fb73c13d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:32:07.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb73a7fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb73c09d030 0x7fb73c13d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:07.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb73affd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb73c138230 0x7fb73c138610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36092/0 (socket says 172.21.15.12:36092) 2024-09-11T01:32:07.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb73affd640 1 -- 172.21.15.12:0/919949346 learned_addr learned my addr 172.21.15.12:0/919949346 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:32:07.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb73b7fe640 1 --2- 172.21.15.12:0/919949346 >> v2:172.21.15.71:3300/0 conn(0x7fb73c09d460 0x7fb73c13f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:07.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb73b7fe640 10 monclient(hunting): get_auth_request con 0x7fb73c09d460 auth_method 0 2024-09-11T01:32:07.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb73b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:07.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb73b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:07.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb73b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:07.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb73b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:07.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb73b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:07.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb73b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:07.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb73b7fe640 10 monclient(hunting): handle_auth_done global_id 34138 payload 274 2024-09-11T01:32:07.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb73b7fe640 1 -- 172.21.15.12:0/919949346 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb73c09d030 msgr2=0x7fb73c13d090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:32:07.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb73b7fe640 1 --2- 172.21.15.12:0/919949346 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb73c09d030 0x7fb73c13d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:07.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb73b7fe640 1 -- 172.21.15.12:0/919949346 >> v1:172.21.15.12:6789/0 conn(0x7fb73c138230 legacy=0x7fb73c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:07.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb73b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:32:07.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb73b7fe640 1 monclient: found mon.noname-c 2024-09-11T01:32:07.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb73b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:07.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb73b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:32:07.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.454+0000 7fb73b7fe640 1 -- 172.21.15.12:0/919949346 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb73c0a66a0 con 0x7fb73c09d460 2024-09-11T01:32:07.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.457+0000 7fb73b7fe640 1 -- 172.21.15.12:0/919949346 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb73c09d460 msgr2=0x7fb73c13f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:07.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.457+0000 7fb73b7fe640 1 --2- 172.21.15.12:0/919949346 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb73c09d460 0x7fb73c13f5e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb73c06e500 tx=0x7fb73402a480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5c4fd8b58b27477 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:07.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.458+0000 7fb739ffb640 1 -- 172.21.15.12:0/919949346 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb734007a30 con 0x7fb73c09d460 2024-09-11T01:32:07.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.458+0000 7fb739ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:07.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.458+0000 7fb739ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:32:07.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.458+0000 7fb739ffb640 10 monclient: dump: 2024-09-11T01:32:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:07.467 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.458+0000 7fb739ffb640 10 monclient: _finish_auth 0 2024-09-11T01:32:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.458+0000 7fb739ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:32:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.458+0000 7fb739ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.458+0000 7fb739ffb640 1 -- 172.21.15.12:0/919949346 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb734007bd0 con 0x7fb73c09d460 2024-09-11T01:32:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.458+0000 7fb74140c640 5 monclient: authenticate success, global_id 34138 2024-09-11T01:32:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.458+0000 7fb74140c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:32:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.458+0000 7fb739ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.458+0000 7fb74140c640 10 monclient: get_monmap_and_config success 2024-09-11T01:32:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.458+0000 7fb74140c640 4 set_mon_vals no callback set 2024-09-11T01:32:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.458+0000 7fb739ffb640 1 -- 172.21.15.12:0/919949346 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb734031400 con 0x7fb73c09d460 2024-09-11T01:32:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.458+0000 7fb739ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.458+0000 7fb739ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:32:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.458+0000 7fb739ffb640 10 monclient: dump: 2024-09-11T01:32:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:07.469 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.458+0000 7fb74140c640 10 monclient: shutdown 2024-09-11T01:32:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.458+0000 7fb74140c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.458+0000 7fb74140c640 1 -- 172.21.15.12:0/919949346 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb73c09d460 msgr2=0x7fb73c13f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.458+0000 7fb74140c640 1 --2- 172.21.15.12:0/919949346 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb73c09d460 0x7fb73c13f5e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb73c06e500 tx=0x7fb73402a480 comp rx=0 tx=0).stop 2024-09-11T01:32:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.458+0000 7fb74140c640 1 -- 172.21.15.12:0/919949346 shutdown_connections 2024-09-11T01:32:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.458+0000 7fb74140c640 1 --2- 172.21.15.12:0/919949346 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb73c09d460 0x7fb73c13f5e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.458+0000 7fb74140c640 1 --2- 172.21.15.12:0/919949346 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb73c09d030 0x7fb73c13d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.458+0000 7fb74140c640 1 -- 172.21.15.12:0/919949346 >> 172.21.15.12:0/919949346 conn(0x7fb73c0a7120 msgr2=0x7fb73c0a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.459+0000 7fb74140c640 1 -- 172.21.15.12:0/919949346 shutdown_connections 2024-09-11T01:32:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.459+0000 7fb74140c640 1 -- 172.21.15.12:0/919949346 wait complete. 2024-09-11T01:32:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.459+0000 7fb74140c640 10 monclient: build_initial_monmap 2024-09-11T01:32:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.459+0000 7fb74140c640 10 monclient: monmap: 2024-09-11T01:32:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:32:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:32:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:32:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:32:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:32:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:32:07.471 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb74140c640 1 Processor -- start 2024-09-11T01:32:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb74140c640 1 -- start start 2024-09-11T01:32:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb74140c640 10 monclient: init 2024-09-11T01:32:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb74140c640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb74140c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:32:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb74140c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb73c09d030 0x7fb73c1ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb74140c640 10 monclient(hunting): picked mon.noname-b con 0x7fb73c09d030 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:32:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb74140c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb73c09d460 0x7fb73c136fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb74140c640 10 monclient(hunting): picked mon.noname-a con 0x7fb73c09d460 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:32:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb74140c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb73c138230 0x7fb73c13c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb74140c640 10 monclient(hunting): picked mon.noname-c con 0x7fb73c138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb74140c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:07.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb74140c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb73c13fca0 con 0x7fb73c09d460 2024-09-11T01:32:07.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb74140c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:07.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb74140c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb73c13fb20 con 0x7fb73c09d030 2024-09-11T01:32:07.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb74140c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:07.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb74140c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb73c13fe20 con 0x7fb73c138230 2024-09-11T01:32:07.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb74140c640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:07.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb74140c640 10 monclient(hunting): authenticate will time out at 933.902714s 2024-09-11T01:32:07.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb73b7fe640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb73c138230 msgr2=0x7fb73c13c9c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:32:07.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb73b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb73c138230 0x7fb73c13c9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:07.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb73a7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb73c09d460 0x7fb73c136fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:07.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb73a7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb73c09d460 0x7fb73c136fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40998/0 (socket says 172.21.15.12:40998) 2024-09-11T01:32:07.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb73a7fc640 1 -- 172.21.15.12:0/974545631 learned_addr learned my addr 172.21.15.12:0/974545631 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:07.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb73a7fc640 10 monclient(hunting): get_auth_request con 0x7fb73c09d460 auth_method 0 2024-09-11T01:32:07.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb73a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb73a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb73a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb73affd640 1 --2- 172.21.15.12:0/974545631 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb73c09d030 0x7fb73c1ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb73a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb73a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb73a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb73affd640 10 monclient(hunting): get_auth_request con 0x7fb73c09d030 auth_method 0 2024-09-11T01:32:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb73affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb73affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.460+0000 7fb73affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.461+0000 7fb73affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.461+0000 7fb73affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.461+0000 7fb73affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.461+0000 7fb73a7fc640 10 monclient(hunting): handle_auth_done global_id 24273 payload 824 2024-09-11T01:32:07.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.461+0000 7fb73a7fc640 1 -- 172.21.15.12:0/974545631 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb73c138230 msgr2=0x7fb73c13c9c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:32:07.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.461+0000 7fb73a7fc640 1 --2- 172.21.15.12:0/974545631 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb73c138230 0x7fb73c13c9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:07.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.461+0000 7fb73a7fc640 1 -- 172.21.15.12:0/974545631 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb73c09d030 msgr2=0x7fb73c1ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:07.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.461+0000 7fb73a7fc640 1 --2- 172.21.15.12:0/974545631 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb73c09d030 0x7fb73c1ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:07.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.461+0000 7fb73a7fc640 10 monclient: _finish_hunting 0 2024-09-11T01:32:07.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.461+0000 7fb73a7fc640 1 monclient: found mon.noname-a 2024-09-11T01:32:07.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.461+0000 7fb73a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:07.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.461+0000 7fb73a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:32:07.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.461+0000 7fb73a7fc640 1 -- 172.21.15.12:0/974545631 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb73c0a1810 con 0x7fb73c09d460 2024-09-11T01:32:07.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.463+0000 7fb73a7fc640 1 -- 172.21.15.12:0/974545631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb73c09d460 msgr2=0x7fb73c136fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:07.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.463+0000 7fb73a7fc640 1 --2- 172.21.15.12:0/974545631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb73c09d460 0x7fb73c136fd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb73000b550 tx=0x7fb73000ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:07.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.463+0000 7fb7237fe640 1 -- 172.21.15.12:0/974545631 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb730005070 con 0x7fb73c09d460 2024-09-11T01:32:07.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.463+0000 7fb7237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:07.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.463+0000 7fb7237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:32:07.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.463+0000 7fb7237fe640 10 monclient: dump: 2024-09-11T01:32:07.475 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:07.475 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:07.475 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:07.475 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:07.475 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:07.475 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:07.475 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:07.475 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:07.475 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:07.475 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:07.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.463+0000 7fb7237fe640 10 monclient: _finish_auth 0 2024-09-11T01:32:07.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.463+0000 7fb7237fe640 10 monclient: _check_auth_tickets 2024-09-11T01:32:07.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.463+0000 7fb7237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:07.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.463+0000 7fb7237fe640 1 -- 172.21.15.12:0/974545631 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb730015070 con 0x7fb73c09d460 2024-09-11T01:32:07.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.463+0000 7fb74140c640 5 monclient: authenticate success, global_id 24273 2024-09-11T01:32:07.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.463+0000 7fb74140c640 10 monclient: _renew_subs 2024-09-11T01:32:07.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.463+0000 7fb74140c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:07.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.463+0000 7fb74140c640 1 -- 172.21.15.12:0/974545631 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb73c0a14e0 con 0x7fb73c09d460 2024-09-11T01:32:07.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.463+0000 7fb7237fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:07.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.463+0000 7fb7237fe640 1 -- 172.21.15.12:0/974545631 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb730005450 con 0x7fb73c09d460 2024-09-11T01:32:07.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.463+0000 7fb7237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:07.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.463+0000 7fb7237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:07.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.463+0000 7fb7237fe640 10 monclient: dump: 2024-09-11T01:32:07.476 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:07.476 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:07.476 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:07.477 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.463+0000 7fb74140c640 10 monclient: _renew_subs 2024-09-11T01:32:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.463+0000 7fb7397fa640 4 set_mon_vals no callback set 2024-09-11T01:32:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.463+0000 7fb74140c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.463+0000 7fb74140c640 1 -- 172.21.15.12:0/974545631 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb73c137c70 con 0x7fb73c09d460 2024-09-11T01:32:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.464+0000 7fb74140c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.464+0000 7fb74140c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.464+0000 7fb74140c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.464+0000 7fb7237fe640 1 -- 172.21.15.12:0/974545631 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fb730005630 con 0x7fb73c09d460 2024-09-11T01:32:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.464+0000 7fb74140c640 1 -- 172.21.15.12:0/974545631 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb73c138610 con 0x7fb73c09d460 2024-09-11T01:32:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.465+0000 7fb7237fe640 1 --2- 172.21.15.12:0/974545631 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb72803e1f0 0x7fb7280406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.465+0000 7fb7237fe640 1 -- 172.21.15.12:0/974545631 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4767+0+0 (secure 0 0 0) 0x7fb730052ac0 con 0x7fb73c09d460 2024-09-11T01:32:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.465+0000 7fb73affd640 1 --2- 172.21.15.12:0/974545631 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb72803e1f0 0x7fb7280406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:07.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.465+0000 7fb73affd640 10 monclient: get_auth_request con 0x7fb72803e1f0 auth_method 0 2024-09-11T01:32:07.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.468+0000 7fb7237fe640 1 -- 172.21.15.12:0/974545631 DispatchQueue::entry inject delay of 0.002000 2024-09-11T01:32:07.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.468+0000 7fb73affd640 1 --2- 172.21.15.12:0/974545631 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb72803e1f0 0x7fb7280406b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb72c007c90 tx=0x7fb72c007be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:07.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.470+0000 7fb7237fe640 1 -- 172.21.15.12:0/974545631 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb7300514d0 con 0x7fb73c09d460 2024-09-11T01:32:07.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.470+0000 7fb7237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:07.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.470+0000 7fb7237fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:32:07.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.724+0000 7fb74140c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:32:07.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.724+0000 7fb74140c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:32:07.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.724+0000 7fb74140c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:07.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.724+0000 7fb74140c640 1 -- 172.21.15.12:0/974545631 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb73c0698e0 con 0x7fb73c09d460 2024-09-11T01:32:07.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.725+0000 7fb7237fe640 1 -- 172.21.15.12:0/974545631 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fb73001a460 con 0x7fb73c09d460 2024-09-11T01:32:07.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.725+0000 7fb7237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:32:07.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.725+0000 7fb7237fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:32:07.727 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:32:07.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.727+0000 7fb74140c640 1 -- 172.21.15.12:0/974545631 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb72803e1f0 msgr2=0x7fb7280406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:07.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.727+0000 7fb74140c640 1 --2- 172.21.15.12:0/974545631 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb72803e1f0 0x7fb7280406b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb72c007c90 tx=0x7fb72c007be0 comp rx=0 tx=0).stop 2024-09-11T01:32:07.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.727+0000 7fb74140c640 10 monclient: shutdown 2024-09-11T01:32:07.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.727+0000 7fb74140c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:07.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.727+0000 7fb74140c640 1 -- 172.21.15.12:0/974545631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb73c09d460 msgr2=0x7fb73c136fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:07.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.727+0000 7fb74140c640 1 --2- 172.21.15.12:0/974545631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb73c09d460 0x7fb73c136fd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb73000b550 tx=0x7fb73000ba10 comp rx=0 tx=0).stop 2024-09-11T01:32:07.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.727+0000 7fb74140c640 1 -- 172.21.15.12:0/974545631 shutdown_connections 2024-09-11T01:32:07.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.727+0000 7fb74140c640 1 --2- 172.21.15.12:0/974545631 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb73c138230 0x7fb73c13c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:07.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.727+0000 7fb74140c640 1 --2- 172.21.15.12:0/974545631 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb72803e1f0 0x7fb7280406b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:07.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.727+0000 7fb74140c640 1 --2- 172.21.15.12:0/974545631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb73c09d460 0x7fb73c136fd0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:07.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.727+0000 7fb74140c640 1 --2- 172.21.15.12:0/974545631 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb73c09d030 0x7fb73c1ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:07.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.727+0000 7fb74140c640 1 -- 172.21.15.12:0/974545631 >> 172.21.15.12:0/974545631 conn(0x7fb73c0a7120 msgr2=0x7fb73c0a8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:07.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.727+0000 7fb74140c640 1 -- 172.21.15.12:0/974545631 shutdown_connections 2024-09-11T01:32:07.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:07.727+0000 7fb74140c640 1 -- 172.21.15.12:0/974545631 wait complete. 2024-09-11T01:32:10.739 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:32:10.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.850+0000 7f8bee860640 10 monclient: get_monmap_and_config 2024-09-11T01:32:10.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.850+0000 7f8bee860640 10 monclient: build_initial_monmap 2024-09-11T01:32:10.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.850+0000 7f8bee860640 10 monclient: monmap: 2024-09-11T01:32:10.853 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:10.853 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:10.853 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:32:10.851646+0000 2024-09-11T01:32:10.853 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:32:10.851646+0000 2024-09-11T01:32:10.853 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:10.853 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:10.853 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:32:10.853 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:32:10.853 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:32:10.853 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:32:10.853 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:32:10.853 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:10.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.851+0000 7f8bee860640 1 Processor -- start 2024-09-11T01:32:10.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.851+0000 7f8bee860640 1 -- start start 2024-09-11T01:32:10.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.851+0000 7f8bee860640 10 monclient: init 2024-09-11T01:32:10.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.852+0000 7f8bee860640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:10.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.852+0000 7f8bee860640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-11T01:32:10.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.852+0000 7f8bee860640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8be80a03a0 0x7f8be80a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:10.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.852+0000 7f8bee860640 10 monclient(hunting): picked mon.noname-b con 0x7f8be80a03a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:10.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.852+0000 7f8bee860640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8be80a11d0 addr v1:172.21.15.71:6789/0 2024-09-11T01:32:10.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.852+0000 7f8bee860640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f8be80a15d0 0x7f8be813c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:10.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.852+0000 7f8bee860640 10 monclient(hunting): picked mon.noname-c con 0x7f8be80a15d0 addr v2:172.21.15.71:3300/0 2024-09-11T01:32:10.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.852+0000 7f8bee860640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8be809fa00 con 0x7f8be80a11d0 2024-09-11T01:32:10.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.852+0000 7f8bee860640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:10.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.852+0000 7f8bee860640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f8be809ead0 con 0x7f8be80a15d0 2024-09-11T01:32:10.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.852+0000 7f8bee860640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:10.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.852+0000 7f8bee860640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8be80a29b0 con 0x7f8be80a03a0 2024-09-11T01:32:10.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.852+0000 7f8bee860640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:10.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.852+0000 7f8bee860640 10 monclient(hunting): authenticate will time out at 937.294437s 2024-09-11T01:32:10.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.852+0000 7f8be7fff640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8be80a03a0 msgr2=0x7f8be80a0c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:32:10.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.852+0000 7f8be7fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8be80a03a0 0x7f8be80a0c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:10.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.852+0000 7f8becdfd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f8be80a15d0 0x7f8be813c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:10.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.852+0000 7f8becdfd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f8be80a15d0 0x7f8be813c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:37656/0 (socket says 172.21.15.12:37656) 2024-09-11T01:32:10.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.852+0000 7f8becdfd640 1 -- 172.21.15.12:0/1858219494 learned_addr learned my addr 172.21.15.12:0/1858219494 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:10.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.852+0000 7f8becdfd640 10 monclient(hunting): get_auth_request con 0x7f8be80a15d0 auth_method 0 2024-09-11T01:32:10.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.852+0000 7f8becdfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:10.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.852+0000 7f8becdfd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:10.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.852+0000 7f8becdfd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:10.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.852+0000 7f8be77fe640 1 --1- 172.21.15.12:0/1858219494 >> v1:172.21.15.71:6789/0 conn(0x7f8be80a11d0 0x7f8be8138a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.12:45036/0 (socket says 172.21.15.12:45036) 2024-09-11T01:32:10.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.855+0000 7f8becdfd640 1 -- 172.21.15.12:0/1858219494 >> v2:172.21.15.71:3300/0 conn(0x7f8be80a15d0 msgr2=0x7f8be813c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:32:10.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.855+0000 7f8becdfd640 1 -- 172.21.15.12:0/1858219494 >> v2:172.21.15.71:3300/0 conn(0x7f8be80a15d0 msgr2=0x7f8be813c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:32:10.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.855+0000 7f8becdfd640 1 --2- 172.21.15.12:0/1858219494 >> v2:172.21.15.71:3300/0 conn(0x7f8be80a15d0 0x7f8be813c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:10.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.855+0000 7f8be77fe640 1 -- 172.21.15.12:0/1858219494 >> v1:172.21.15.71:6789/0 conn(0x7f8be80a11d0 legacy=0x7f8be8138a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:10.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:10.855+0000 7f8becdfd640 1 --2- 172.21.15.12:0/1858219494 >> v2:172.21.15.71:3300/0 conn(0x7f8be80a15d0 0x7f8be813c1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:11.053+0000 7f8be7fff640 1 -- 172.21.15.12:0/1858219494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8be80a03a0 msgr2=0x7f8be80a0c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:32:11.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:11.054+0000 7f8be7fff640 1 --2- 172.21.15.12:0/1858219494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8be80a03a0 0x7f8be80a0c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:32:11.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:11.056+0000 7f8becdfd640 1 --2- 172.21.15.12:0/1858219494 >> v2:172.21.15.71:3300/0 conn(0x7f8be80a15d0 0x7f8be813c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:11.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:11.056+0000 7f8becdfd640 10 monclient(hunting): get_auth_request con 0x7f8be80a15d0 auth_method 0 2024-09-11T01:32:11.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:11.056+0000 7f8becdfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:11.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:11.056+0000 7f8becdfd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:11.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:11.056+0000 7f8becdfd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:32:11.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:11.056+0000 7f8becdfd640 1 -- 172.21.15.12:0/1858219494 >> v2:172.21.15.71:3300/0 conn(0x7f8be80a15d0 msgr2=0x7f8be813c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:32:11.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:11.056+0000 7f8becdfd640 1 -- 172.21.15.12:0/1858219494 >> v2:172.21.15.71:3300/0 conn(0x7f8be80a15d0 msgr2=0x7f8be813c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:32:11.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:11.056+0000 7f8becdfd640 1 --2- 172.21.15.12:0/1858219494 >> v2:172.21.15.71:3300/0 conn(0x7f8be80a15d0 0x7f8be813c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:11.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:11.056+0000 7f8becdfd640 1 --2- 172.21.15.12:0/1858219494 >> v2:172.21.15.71:3300/0 conn(0x7f8be80a15d0 0x7f8be813c1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:32:11.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:11.455+0000 7f8be7fff640 1 -- 172.21.15.12:0/1858219494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8be80a03a0 msgr2=0x7f8be80a0c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:32:11.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:11.455+0000 7f8be7fff640 1 --2- 172.21.15.12:0/1858219494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8be80a03a0 0x7f8be80a0c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:32:11.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:11.457+0000 7f8becdfd640 1 --2- 172.21.15.12:0/1858219494 >> v2:172.21.15.71:3300/0 conn(0x7f8be80a15d0 0x7f8be813c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:11.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:11.457+0000 7f8becdfd640 10 monclient(hunting): get_auth_request con 0x7f8be80a15d0 auth_method 0 2024-09-11T01:32:11.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:11.457+0000 7f8becdfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:11.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:11.457+0000 7f8becdfd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:11.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:11.457+0000 7f8becdfd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:32:11.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:11.458+0000 7f8becdfd640 1 -- 172.21.15.12:0/1858219494 >> v2:172.21.15.71:3300/0 conn(0x7f8be80a15d0 msgr2=0x7f8be813c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:32:11.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:11.458+0000 7f8becdfd640 1 -- 172.21.15.12:0/1858219494 >> v2:172.21.15.71:3300/0 conn(0x7f8be80a15d0 msgr2=0x7f8be813c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:32:11.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:11.458+0000 7f8becdfd640 1 --2- 172.21.15.12:0/1858219494 >> v2:172.21.15.71:3300/0 conn(0x7f8be80a15d0 0x7f8be813c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:11.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:11.458+0000 7f8becdfd640 1 --2- 172.21.15.12:0/1858219494 >> v2:172.21.15.71:3300/0 conn(0x7f8be80a15d0 0x7f8be813c1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:32:12.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:12.256+0000 7f8be7fff640 1 -- 172.21.15.12:0/1858219494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8be80a03a0 msgr2=0x7f8be80a0c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:32:12.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:12.256+0000 7f8be7fff640 1 --2- 172.21.15.12:0/1858219494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8be80a03a0 0x7f8be80a0c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:32:12.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:12.260+0000 7f8becdfd640 1 --2- 172.21.15.12:0/1858219494 >> v2:172.21.15.71:3300/0 conn(0x7f8be80a15d0 0x7f8be813c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:12.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:12.260+0000 7f8becdfd640 10 monclient(hunting): get_auth_request con 0x7f8be80a15d0 auth_method 0 2024-09-11T01:32:12.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:12.260+0000 7f8becdfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:12.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:12.260+0000 7f8becdfd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:12.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:12.260+0000 7f8becdfd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:32:12.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:12.260+0000 7f8becdfd640 1 -- 172.21.15.12:0/1858219494 >> v2:172.21.15.71:3300/0 conn(0x7f8be80a15d0 msgr2=0x7f8be813c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:32:12.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:12.260+0000 7f8becdfd640 1 -- 172.21.15.12:0/1858219494 >> v2:172.21.15.71:3300/0 conn(0x7f8be80a15d0 msgr2=0x7f8be813c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:32:12.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:12.260+0000 7f8becdfd640 1 --2- 172.21.15.12:0/1858219494 >> v2:172.21.15.71:3300/0 conn(0x7f8be80a15d0 0x7f8be813c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:12.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:12.260+0000 7f8becdfd640 1 --2- 172.21.15.12:0/1858219494 >> v2:172.21.15.71:3300/0 conn(0x7f8be80a15d0 0x7f8be813c1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:32:13.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.852+0000 7f8be5ffb640 10 monclient(hunting): tick 2024-09-11T01:32:13.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.852+0000 7f8be5ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:32:13.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.852+0000 7f8be5ffb640 1 monclient(hunting): continuing hunt 2024-09-11T01:32:13.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.852+0000 7f8be5ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:32:13.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.852+0000 7f8be5ffb640 1 -- 172.21.15.12:0/1858219494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8be80a03a0 msgr2=0x7f8be80a0c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:32:13.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.852+0000 7f8be5ffb640 1 --2- 172.21.15.12:0/1858219494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8be80a03a0 0x7f8be80a0c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:13.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.852+0000 7f8be5ffb640 1 -- 172.21.15.12:0/1858219494 >> v2:172.21.15.71:3300/0 conn(0x7f8be80a15d0 msgr2=0x7f8be813c1b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:32:13.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.852+0000 7f8be5ffb640 1 --2- 172.21.15.12:0/1858219494 >> v2:172.21.15.71:3300/0 conn(0x7f8be80a15d0 0x7f8be813c1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:13.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.852+0000 7f8be5ffb640 1 -- 172.21.15.12:0/1858219494 >> v1:172.21.15.71:6789/0 conn(0x7f8be80a11d0 legacy=0x7f8be8138a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:13.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.852+0000 7f8be5ffb640 10 monclient: _add_conns ranks=[0,3] 2024-09-11T01:32:13.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.852+0000 7f8be5ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8bd4003740 addr v1:172.21.15.12:6789/0 2024-09-11T01:32:13.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.852+0000 7f8be5ffb640 1 --2- 172.21.15.12:0/1858219494 >> v2:172.21.15.12:3300/0 conn(0x7f8bd4007490 0x7f8bd4009930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:13.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.852+0000 7f8be5ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f8bd4007490 addr v2:172.21.15.12:3300/0 2024-09-11T01:32:13.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.852+0000 7f8be5ffb640 1 -- 172.21.15.12:0/1858219494 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8be809e510 con 0x7f8bd4003740 2024-09-11T01:32:13.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.852+0000 7f8be5ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:13.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.852+0000 7f8be5ffb640 1 -- 172.21.15.12:0/1858219494 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f8be809ead0 con 0x7f8bd4007490 2024-09-11T01:32:13.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.852+0000 7f8be5ffb640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:13.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.852+0000 7f8be77fe640 1 --2- 172.21.15.12:0/1858219494 >> v2:172.21.15.12:3300/0 conn(0x7f8bd4007490 0x7f8bd4009930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:13.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.853+0000 7f8be77fe640 10 monclient(hunting): get_auth_request con 0x7f8bd4007490 auth_method 0 2024-09-11T01:32:13.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.853+0000 7f8be77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:13.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.853+0000 7f8be77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:13.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.853+0000 7f8be77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:13.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.853+0000 7f8be77fe640 1 -- 172.21.15.12:0/1858219494 >> v2:172.21.15.12:3300/0 conn(0x7f8bd4007490 msgr2=0x7f8bd4009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:32:13.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.853+0000 7f8be77fe640 1 -- 172.21.15.12:0/1858219494 >> v2:172.21.15.12:3300/0 conn(0x7f8bd4007490 msgr2=0x7f8bd4009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:32:13.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.853+0000 7f8be77fe640 1 --2- 172.21.15.12:0/1858219494 >> v2:172.21.15.12:3300/0 conn(0x7f8bd4007490 0x7f8bd4009930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:13.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.853+0000 7f8be77fe640 1 --2- 172.21.15.12:0/1858219494 >> v2:172.21.15.12:3300/0 conn(0x7f8bd4007490 0x7f8bd4009930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:13.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:13.855+0000 7f8be7fff640 1 -- 172.21.15.12:0/1858219494 >> v1:172.21.15.12:6789/0 conn(0x7f8bd4003740 legacy=0x7f8bd4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:14.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:14.054+0000 7f8be77fe640 1 --2- 172.21.15.12:0/1858219494 >> v2:172.21.15.12:3300/0 conn(0x7f8bd4007490 0x7f8bd4009930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:14.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:14.054+0000 7f8be77fe640 10 monclient(hunting): get_auth_request con 0x7f8bd4007490 auth_method 0 2024-09-11T01:32:14.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:14.054+0000 7f8be77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:14.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:14.054+0000 7f8be77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:14.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:14.054+0000 7f8be77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:32:14.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:14.055+0000 7f8be77fe640 1 -- 172.21.15.12:0/1858219494 >> v2:172.21.15.12:3300/0 conn(0x7f8bd4007490 msgr2=0x7f8bd4009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:32:14.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:14.055+0000 7f8be77fe640 1 -- 172.21.15.12:0/1858219494 >> v2:172.21.15.12:3300/0 conn(0x7f8bd4007490 msgr2=0x7f8bd4009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:32:14.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:14.055+0000 7f8be77fe640 1 --2- 172.21.15.12:0/1858219494 >> v2:172.21.15.12:3300/0 conn(0x7f8bd4007490 0x7f8bd4009930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:14.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:14.055+0000 7f8be77fe640 1 --2- 172.21.15.12:0/1858219494 >> v2:172.21.15.12:3300/0 conn(0x7f8bd4007490 0x7f8bd4009930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:32:14.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:14.456+0000 7f8be77fe640 1 --2- 172.21.15.12:0/1858219494 >> v2:172.21.15.12:3300/0 conn(0x7f8bd4007490 0x7f8bd4009930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:14.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:14.456+0000 7f8be77fe640 10 monclient(hunting): get_auth_request con 0x7f8bd4007490 auth_method 0 2024-09-11T01:32:14.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:14.456+0000 7f8be77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:14.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:14.456+0000 7f8be77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:14.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:14.456+0000 7f8be77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:32:14.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:14.456+0000 7f8be77fe640 1 -- 172.21.15.12:0/1858219494 >> v2:172.21.15.12:3300/0 conn(0x7f8bd4007490 msgr2=0x7f8bd4009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:32:14.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:14.456+0000 7f8be77fe640 1 -- 172.21.15.12:0/1858219494 >> v2:172.21.15.12:3300/0 conn(0x7f8bd4007490 msgr2=0x7f8bd4009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:32:14.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:14.456+0000 7f8be77fe640 1 --2- 172.21.15.12:0/1858219494 >> v2:172.21.15.12:3300/0 conn(0x7f8bd4007490 0x7f8bd4009930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:14.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:14.457+0000 7f8be77fe640 1 --2- 172.21.15.12:0/1858219494 >> v2:172.21.15.12:3300/0 conn(0x7f8bd4007490 0x7f8bd4009930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:32:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.258+0000 7f8be77fe640 1 --2- 172.21.15.12:0/1858219494 >> v2:172.21.15.12:3300/0 conn(0x7f8bd4007490 0x7f8bd4009930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.258+0000 7f8be77fe640 10 monclient(hunting): get_auth_request con 0x7f8bd4007490 auth_method 0 2024-09-11T01:32:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.258+0000 7f8be77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.258+0000 7f8be77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.258+0000 7f8be77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:32:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.258+0000 7f8be77fe640 1 -- 172.21.15.12:0/1858219494 >> v2:172.21.15.12:3300/0 conn(0x7f8bd4007490 msgr2=0x7f8bd4009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:32:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.258+0000 7f8be77fe640 1 -- 172.21.15.12:0/1858219494 >> v2:172.21.15.12:3300/0 conn(0x7f8bd4007490 msgr2=0x7f8bd4009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:32:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.258+0000 7f8be77fe640 1 --2- 172.21.15.12:0/1858219494 >> v2:172.21.15.12:3300/0 conn(0x7f8bd4007490 0x7f8bd4009930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:15.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.258+0000 7f8be77fe640 1 --2- 172.21.15.12:0/1858219494 >> v2:172.21.15.12:3300/0 conn(0x7f8bd4007490 0x7f8bd4009930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:32:15.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.362+0000 7f8be6ffd640 1 -- 172.21.15.12:0/1858219494 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1072318076 0 0) 0x7f8be809e510 con 0x7f8bd4003740 2024-09-11T01:32:15.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.362+0000 7f8be6ffd640 10 monclient(hunting): _negotiate 2024-09-11T01:32:15.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.362+0000 7f8be6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:15.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.362+0000 7f8be6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:15.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.362+0000 7f8be6ffd640 10 monclient(hunting): my global_id is 24276 2024-09-11T01:32:15.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.362+0000 7f8be6ffd640 1 -- 172.21.15.12:0/1858219494 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bd0003520 con 0x7f8bd4003740 2024-09-11T01:32:15.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 1 -- 172.21.15.12:0/1858219494 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2212250845 0 0) 0x7f8bd0003520 con 0x7f8bd4003740 2024-09-11T01:32:15.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 1 -- 172.21.15.12:0/1858219494 >> v2:172.21.15.12:3300/0 conn(0x7f8bd4007490 msgr2=0x7f8bd4009930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:32:15.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 1 --2- 172.21.15.12:0/1858219494 >> v2:172.21.15.12:3300/0 conn(0x7f8bd4007490 0x7f8bd4009930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:32:15.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:32:15.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:15.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:32:15.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 1 -- 172.21.15.12:0/1858219494 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd400b050 con 0x7f8bd4003740 2024-09-11T01:32:15.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 10 monclient: _finish_auth 0 2024-09-11T01:32:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:32:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 1 -- 172.21.15.12:0/1858219494 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f8bd8004600 con 0x7f8bd4003740 2024-09-11T01:32:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8bee860640 5 monclient: authenticate success, global_id 24276 2024-09-11T01:32:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8bee860640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:32:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:32:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 10 monclient: dump: 2024-09-11T01:32:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.376 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:15.377 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:15.377 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:15.377 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:15.377 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:15.377 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:15.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:32:15.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:15.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 1 -- 172.21.15.12:0/1858219494 >> v1:172.21.15.12:6789/0 conn(0x7f8bd4003740 legacy=0x7f8bd4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:15.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8becdfd640 1 -- 172.21.15.12:0/1858219494 reap_dead start 2024-09-11T01:32:15.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:32:15.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 1 --2- 172.21.15.12:0/1858219494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8bd00039b0 0x7f8bd0003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:15.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 10 monclient(hunting): picked mon.c con 0x7f8bd00039b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:15.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 1 --2- 172.21.15.12:0/1858219494 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bd0004800 0x7f8bd0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:15.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 10 monclient(hunting): picked mon.a con 0x7f8bd0004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:32:15.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 1 --2- 172.21.15.12:0/1858219494 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8bd0005290 0x7f8bd000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:15.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 10 monclient(hunting): picked mon.b con 0x7f8bd0005290 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:32:15.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:15.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 1 -- 172.21.15.12:0/1858219494 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8be809ead0 con 0x7f8bd0004800 2024-09-11T01:32:15.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:15.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 1 -- 172.21.15.12:0/1858219494 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd000c1e0 con 0x7f8bd0005290 2024-09-11T01:32:15.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:15.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be7fff640 1 -- 172.21.15.12:0/1858219494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8bd00039b0 msgr2=0x7f8bd0003db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:32:15.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 1 -- 172.21.15.12:0/1858219494 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8bd000c3e0 con 0x7f8bd00039b0 2024-09-11T01:32:15.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:15.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:32:15.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be7fff640 1 --2- 172.21.15.12:0/1858219494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8bd00039b0 0x7f8bd0003db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:15.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8bee860640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:32:15.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be77fe640 1 --2- 172.21.15.12:0/1858219494 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bd0004800 0x7f8bd0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:15.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be77fe640 10 monclient(hunting): get_auth_request con 0x7f8bd0004800 auth_method 0 2024-09-11T01:32:15.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:15.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:15.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.364+0000 7f8be77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:32:15.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.365+0000 7f8becdfd640 1 --2- 172.21.15.12:0/1858219494 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8bd0005290 0x7f8bd000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:15.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.365+0000 7f8becdfd640 10 monclient(hunting): get_auth_request con 0x7f8bd0005290 auth_method 0 2024-09-11T01:32:15.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.365+0000 7f8becdfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:15.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.365+0000 7f8becdfd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:15.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.365+0000 7f8becdfd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:32:15.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.365+0000 7f8be77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:15.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.365+0000 7f8be77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:15.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.365+0000 7f8be77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:32:15.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.365+0000 7f8becdfd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:15.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.365+0000 7f8becdfd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:15.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.365+0000 7f8becdfd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:32:15.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.365+0000 7f8be77fe640 10 monclient(hunting): handle_auth_done global_id 24276 payload 293 2024-09-11T01:32:15.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.365+0000 7f8be77fe640 1 -- 172.21.15.12:0/1858219494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8bd00039b0 msgr2=0x7f8bd0003db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:32:15.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.365+0000 7f8be77fe640 1 --2- 172.21.15.12:0/1858219494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8bd00039b0 0x7f8bd0003db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.365+0000 7f8be77fe640 1 -- 172.21.15.12:0/1858219494 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8bd0005290 msgr2=0x7f8bd000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:15.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.365+0000 7f8be77fe640 1 --2- 172.21.15.12:0/1858219494 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8bd0005290 0x7f8bd000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.365+0000 7f8be77fe640 10 monclient: _finish_hunting 0 2024-09-11T01:32:15.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.365+0000 7f8be77fe640 1 monclient: found mon.a 2024-09-11T01:32:15.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.365+0000 7f8be77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:15.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.365+0000 7f8be77fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:15.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.365+0000 7f8be77fe640 1 -- 172.21.15.12:0/1858219494 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd0003520 con 0x7f8bd0004800 2024-09-11T01:32:15.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.365+0000 7f8be77fe640 10 monclient: _finish_auth 0 2024-09-11T01:32:15.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.365+0000 7f8be77fe640 10 monclient: _check_auth_tickets 2024-09-11T01:32:15.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.365+0000 7f8be77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:15.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.368+0000 7f8be77fe640 1 -- 172.21.15.12:0/1858219494 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bd0004800 msgr2=0x7f8bd0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:15.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.368+0000 7f8be77fe640 1 --2- 172.21.15.12:0/1858219494 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bd0004800 0x7f8bd0004c80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc005ce0 tx=0x7f8bd0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e022ae5c8957f83 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:15.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.368+0000 7f8be6ffd640 1 -- 172.21.15.12:0/1858219494 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bdc005930 con 0x7f8bd0004800 2024-09-11T01:32:15.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.368+0000 7f8be6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:15.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.368+0000 7f8be6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:15.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.368+0000 7f8be6ffd640 10 monclient: dump: 2024-09-11T01:32:15.381 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:15.381 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:15.382 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.382 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.382 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:15.382 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:15.382 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:15.382 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:15.382 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:15.382 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:15.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.368+0000 7f8bee860640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:32:15.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.368+0000 7f8be6ffd640 1 -- 172.21.15.12:0/1858219494 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bdc004ac0 con 0x7f8bd0004800 2024-09-11T01:32:15.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.368+0000 7f8be6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:15.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.368+0000 7f8bee860640 10 monclient: get_monmap_and_config success 2024-09-11T01:32:15.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.368+0000 7f8be6ffd640 1 -- 172.21.15.12:0/1858219494 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bdc005050 con 0x7f8bd0004800 2024-09-11T01:32:15.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.368+0000 7f8bee860640 4 set_mon_vals no callback set 2024-09-11T01:32:15.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.368+0000 7f8be6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:15.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.368+0000 7f8be6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:15.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.368+0000 7f8be6ffd640 10 monclient: dump: 2024-09-11T01:32:15.383 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:15.383 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:15.383 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.383 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.383 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:15.383 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:15.383 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:15.383 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:15.383 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:15.383 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:15.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.369+0000 7f8bee860640 10 monclient: shutdown 2024-09-11T01:32:15.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.369+0000 7f8bee860640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:15.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.369+0000 7f8bee860640 1 -- 172.21.15.12:0/1858219494 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bd0004800 msgr2=0x7f8bd0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:15.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.369+0000 7f8bee860640 1 --2- 172.21.15.12:0/1858219494 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bd0004800 0x7f8bd0004c80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc005ce0 tx=0x7f8bd0005830 comp rx=0 tx=0).stop 2024-09-11T01:32:15.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.369+0000 7f8bee860640 1 -- 172.21.15.12:0/1858219494 shutdown_connections 2024-09-11T01:32:15.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.369+0000 7f8bee860640 1 --2- 172.21.15.12:0/1858219494 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8bd0005290 0x7f8bd000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.369+0000 7f8bee860640 1 --2- 172.21.15.12:0/1858219494 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bd0004800 0x7f8bd0004c80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.369+0000 7f8bee860640 1 --2- 172.21.15.12:0/1858219494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8bd00039b0 0x7f8bd0003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.369+0000 7f8bee860640 1 -- 172.21.15.12:0/1858219494 >> 172.21.15.12:0/1858219494 conn(0x7f8be80a2d90 msgr2=0x7f8be80a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:15.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.369+0000 7f8bee860640 1 -- 172.21.15.12:0/1858219494 shutdown_connections 2024-09-11T01:32:15.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.369+0000 7f8bee860640 1 -- 172.21.15.12:0/1858219494 wait complete. 2024-09-11T01:32:15.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.370+0000 7f8bee860640 10 monclient: build_initial_monmap 2024-09-11T01:32:15.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.370+0000 7f8bee860640 10 monclient: monmap: 2024-09-11T01:32:15.384 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:15.384 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:32:15.385 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:32:15.385 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:32:15.385 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:15.385 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:15.385 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:32:15.385 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:32:15.385 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:32:15.385 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:15.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.370+0000 7f8bee860640 1 Processor -- start 2024-09-11T01:32:15.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.370+0000 7f8bee860640 1 -- start start 2024-09-11T01:32:15.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.370+0000 7f8bee860640 10 monclient: init 2024-09-11T01:32:15.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8bee860640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:15.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8bee860640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:32:15.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8bee860640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bd00039b0 0x7f8be8133090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:15.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8bee860640 10 monclient(hunting): picked mon.noname-a con 0x7f8bd00039b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:32:15.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8bee860640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8bd0004800 0x7f8be81335d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:15.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8bee860640 10 monclient(hunting): picked mon.noname-b con 0x7f8bd0004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:32:15.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8bee860640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8bd0005290 0x7f8be8138140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:15.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8bee860640 10 monclient(hunting): picked mon.noname-c con 0x7f8bd0005290 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:15.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8bee860640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:15.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8bee860640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8be80a1a20 con 0x7f8bd00039b0 2024-09-11T01:32:15.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8bee860640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:15.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8bee860640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8be80a26f0 con 0x7f8bd0004800 2024-09-11T01:32:15.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8bee860640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:15.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8bee860640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8be809f1d0 con 0x7f8bd0005290 2024-09-11T01:32:15.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8bee860640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:15.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8bee860640 10 monclient(hunting): authenticate will time out at 941.813479s 2024-09-11T01:32:15.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8becdfd640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8bd0005290 msgr2=0x7f8be8138140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:32:15.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8be7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bd00039b0 0x7f8be8133090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:15.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8becdfd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8bd0005290 0x7f8be8138140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:15.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8be7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bd00039b0 0x7f8be8133090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41066/0 (socket says 172.21.15.12:41066) 2024-09-11T01:32:15.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8be7fff640 1 -- 172.21.15.12:0/1969045786 learned_addr learned my addr 172.21.15.12:0/1969045786 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:15.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8be7fff640 10 monclient(hunting): get_auth_request con 0x7f8bd00039b0 auth_method 0 2024-09-11T01:32:15.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8be7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:15.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8be7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:15.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8be7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:15.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8be7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:15.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8be7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8be7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8be77fe640 1 --2- 172.21.15.12:0/1969045786 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8bd0004800 0x7f8be81335d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8be77fe640 10 monclient(hunting): get_auth_request con 0x7f8bd0004800 auth_method 0 2024-09-11T01:32:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8be77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8be77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8be77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8be7fff640 10 monclient(hunting): handle_auth_done global_id 24279 payload 824 2024-09-11T01:32:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8be7fff640 1 -- 172.21.15.12:0/1969045786 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8bd0005290 msgr2=0x7f8be8138140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:32:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8be7fff640 1 --2- 172.21.15.12:0/1969045786 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8bd0005290 0x7f8be8138140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8be7fff640 1 -- 172.21.15.12:0/1969045786 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8bd0004800 msgr2=0x7f8be81335d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8be7fff640 1 --2- 172.21.15.12:0/1969045786 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8bd0004800 0x7f8be81335d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8be7fff640 10 monclient: _finish_hunting 0 2024-09-11T01:32:15.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8be7fff640 1 monclient: found mon.noname-a 2024-09-11T01:32:15.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8be7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:15.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8be7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:32:15.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8be7fff640 1 -- 172.21.15.12:0/1969045786 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8be8139c00 con 0x7f8bd00039b0 2024-09-11T01:32:15.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.371+0000 7f8be77fe640 1 --2- 172.21.15.12:0/1969045786 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8bd0004800 0x7f8be81335d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:32:15.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.374+0000 7f8be7fff640 1 -- 172.21.15.12:0/1969045786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bd00039b0 msgr2=0x7f8be8133090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:15.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.374+0000 7f8be7fff640 1 --2- 172.21.15.12:0/1969045786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bd00039b0 0x7f8be8133090 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8bd80047f0 tx=0x7f8bd800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:15.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.374+0000 7f8be4ff9640 1 -- 172.21.15.12:0/1969045786 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd8004ce0 con 0x7f8bd00039b0 2024-09-11T01:32:15.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.374+0000 7f8be4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:15.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.374+0000 7f8be4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:32:15.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.374+0000 7f8be4ff9640 10 monclient: dump: 2024-09-11T01:32:15.391 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:15.391 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:15.392 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.392 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.392 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:15.392 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:15.392 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:15.392 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:15.392 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:15.392 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:15.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.374+0000 7f8be4ff9640 10 monclient: _finish_auth 0 2024-09-11T01:32:15.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.374+0000 7f8be4ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:32:15.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.374+0000 7f8be4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:15.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.374+0000 7f8be4ff9640 1 -- 172.21.15.12:0/1969045786 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bd8007110 con 0x7f8bd00039b0 2024-09-11T01:32:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.374+0000 7f8bee860640 5 monclient: authenticate success, global_id 24279 2024-09-11T01:32:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.374+0000 7f8bee860640 10 monclient: _renew_subs 2024-09-11T01:32:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.374+0000 7f8bee860640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.374+0000 7f8bee860640 1 -- 172.21.15.12:0/1969045786 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8be809d480 con 0x7f8bd00039b0 2024-09-11T01:32:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.374+0000 7f8be4ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.374+0000 7f8be4ff9640 1 -- 172.21.15.12:0/1969045786 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd80074e0 con 0x7f8bd00039b0 2024-09-11T01:32:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.374+0000 7f8be4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.374+0000 7f8be4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.374+0000 7f8be4ff9640 10 monclient: dump: 2024-09-11T01:32:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.393 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.394 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:15.394 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:15.394 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:15.394 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:15.394 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:15.394 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:15.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.374+0000 7f8be67fc640 4 set_mon_vals no callback set 2024-09-11T01:32:15.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.374+0000 7f8bee860640 10 monclient: _renew_subs 2024-09-11T01:32:15.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.374+0000 7f8bee860640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:15.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.374+0000 7f8bee860640 1 -- 172.21.15.12:0/1969045786 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8be813b6c0 con 0x7f8bd00039b0 2024-09-11T01:32:15.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.376+0000 7f8be4ff9640 1 -- 172.21.15.12:0/1969045786 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f8bd800a5e0 con 0x7f8bd00039b0 2024-09-11T01:32:15.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.376+0000 7f8bee860640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:15.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.376+0000 7f8bee860640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:15.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.376+0000 7f8bee860640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:15.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.376+0000 7f8bee860640 1 -- 172.21.15.12:0/1969045786 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8be8074ba0 con 0x7f8bd00039b0 2024-09-11T01:32:15.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.376+0000 7f8be4ff9640 1 --2- 172.21.15.12:0/1969045786 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8bd403eb00 0x7f8bd4040fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:15.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.376+0000 7f8be4ff9640 1 -- 172.21.15.12:0/1969045786 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4767+0+0 (secure 0 0 0) 0x7f8bd8052d60 con 0x7f8bd00039b0 2024-09-11T01:32:15.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.376+0000 7f8be77fe640 1 --2- 172.21.15.12:0/1969045786 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8bd403eb00 0x7f8bd4040fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:15.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.376+0000 7f8be77fe640 10 monclient: get_auth_request con 0x7f8bd403eb00 auth_method 0 2024-09-11T01:32:15.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.379+0000 7f8be77fe640 1 --2- 172.21.15.12:0/1969045786 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8bd403eb00 0x7f8bd4040fc0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc007b40 tx=0x7f8bdc030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:15.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.381+0000 7f8be4ff9640 1 -- 172.21.15.12:0/1969045786 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8bd8051770 con 0x7f8bd00039b0 2024-09-11T01:32:15.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.381+0000 7f8be4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:15.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.381+0000 7f8be4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:32:15.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.639+0000 7f8bee860640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:32:15.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.639+0000 7f8bee860640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:32:15.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.639+0000 7f8bee860640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:15.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.639+0000 7f8bee860640 1 -- 172.21.15.12:0/1969045786 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8be806d460 con 0x7f8bd00039b0 2024-09-11T01:32:15.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.640+0000 7f8be4ff9640 1 -- 172.21.15.12:0/1969045786 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f8bd800abe0 con 0x7f8bd00039b0 2024-09-11T01:32:15.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.640+0000 7f8be4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:32:15.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.640+0000 7f8be4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:32:15.642 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":32,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:32:15.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.642+0000 7f8bee860640 1 -- 172.21.15.12:0/1969045786 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8bd403eb00 msgr2=0x7f8bd4040fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:15.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.642+0000 7f8bee860640 1 --2- 172.21.15.12:0/1969045786 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8bd403eb00 0x7f8bd4040fc0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc007b40 tx=0x7f8bdc030000 comp rx=0 tx=0).stop 2024-09-11T01:32:15.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.642+0000 7f8bee860640 10 monclient: shutdown 2024-09-11T01:32:15.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.642+0000 7f8bee860640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:15.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.642+0000 7f8bee860640 1 -- 172.21.15.12:0/1969045786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bd00039b0 msgr2=0x7f8be8133090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:15.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.642+0000 7f8bee860640 1 --2- 172.21.15.12:0/1969045786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bd00039b0 0x7f8be8133090 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8bd80047f0 tx=0x7f8bd800cb90 comp rx=0 tx=0).stop 2024-09-11T01:32:15.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.642+0000 7f8bee860640 1 -- 172.21.15.12:0/1969045786 shutdown_connections 2024-09-11T01:32:15.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.642+0000 7f8bee860640 1 --2- 172.21.15.12:0/1969045786 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8bd0005290 0x7f8be8138140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.642+0000 7f8bee860640 1 --2- 172.21.15.12:0/1969045786 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8bd0004800 0x7f8be81335d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.642+0000 7f8bee860640 1 --2- 172.21.15.12:0/1969045786 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8bd403eb00 0x7f8bd4040fc0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.642+0000 7f8bee860640 1 --2- 172.21.15.12:0/1969045786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bd00039b0 0x7f8be8133090 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.642+0000 7f8bee860640 1 -- 172.21.15.12:0/1969045786 >> 172.21.15.12:0/1969045786 conn(0x7f8be80a2d90 msgr2=0x7f8be80a4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:15.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.642+0000 7f8bee860640 1 -- 172.21.15.12:0/1969045786 shutdown_connections 2024-09-11T01:32:15.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.642+0000 7f8bee860640 1 -- 172.21.15.12:0/1969045786 wait complete. 2024-09-11T01:32:15.654 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:32:15.655 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-11T01:32:15.655 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:32:15.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.755+0000 7f12ad55d640 10 monclient: get_monmap_and_config 2024-09-11T01:32:15.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.755+0000 7f12ad55d640 10 monclient: build_initial_monmap 2024-09-11T01:32:15.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.755+0000 7f12ad55d640 10 monclient: monmap: 2024-09-11T01:32:15.757 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:15.757 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:15.757 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:32:15.755899+0000 2024-09-11T01:32:15.757 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:32:15.755899+0000 2024-09-11T01:32:15.757 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:15.757 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:15.757 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:32:15.757 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:32:15.757 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:32:15.757 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:32:15.758 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:32:15.758 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:15.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.755+0000 7f12ad55d640 1 Processor -- start 2024-09-11T01:32:15.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.755+0000 7f12ad55d640 1 -- start start 2024-09-11T01:32:15.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.755+0000 7f12ad55d640 10 monclient: init 2024-09-11T01:32:15.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.756+0000 7f12ad55d640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:15.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.756+0000 7f12ad55d640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-11T01:32:15.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.756+0000 7f12ad55d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f12a8137e50 0x7f12a809d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:15.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.756+0000 7f12ad55d640 10 monclient(hunting): picked mon.noname-c con 0x7f12a8137e50 addr v2:172.21.15.71:3300/0 2024-09-11T01:32:15.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.756+0000 7f12ad55d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f12a8138230 0x7f12a8138660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:15.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.756+0000 7f12ad55d640 10 monclient(hunting): picked mon.noname-b con 0x7f12a8138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:15.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.756+0000 7f12ad55d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f12a8134cb0 addr v1:172.21.15.71:6789/0 2024-09-11T01:32:15.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.756+0000 7f12ad55d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12a80a6580 con 0x7f12a8134cb0 2024-09-11T01:32:15.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.756+0000 7f12ad55d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:15.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.756+0000 7f12ad55d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f12a80a62d0 con 0x7f12a8137e50 2024-09-11T01:32:15.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.756+0000 7f12ad55d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:15.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.756+0000 7f12ad55d640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f12a809bb40 con 0x7f12a8138230 2024-09-11T01:32:15.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.756+0000 7f12ad55d640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:15.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.756+0000 7f12ad55d640 10 monclient(hunting): authenticate will time out at 942.198343s 2024-09-11T01:32:15.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.756+0000 7f12a67fc640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f12a8138230 msgr2=0x7f12a8138660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:32:15.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.756+0000 7f12a67fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f12a8138230 0x7f12a8138660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:15.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.756+0000 7f12a6ffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f12a8137e50 0x7f12a809d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:15.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.756+0000 7f12a6ffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f12a8137e50 0x7f12a809d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:37702/0 (socket says 172.21.15.12:37702) 2024-09-11T01:32:15.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.756+0000 7f12a6ffd640 1 -- 172.21.15.12:0/196076051 learned_addr learned my addr 172.21.15.12:0/196076051 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:15.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.756+0000 7f12a6ffd640 10 monclient(hunting): get_auth_request con 0x7f12a8137e50 auth_method 0 2024-09-11T01:32:15.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.756+0000 7f12a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:15.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.756+0000 7f12a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:15.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.756+0000 7f12a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:15.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.756+0000 7f12a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:15.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.756+0000 7f12a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:15.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.756+0000 7f12a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:15.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.759+0000 7f12a77fe640 1 -- 172.21.15.12:0/196076051 >> v1:172.21.15.71:6789/0 conn(0x7f12a8134cb0 legacy=0x7f12a8135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:15.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.759+0000 7f12a6ffd640 10 monclient(hunting): handle_auth_done global_id 34147 payload 274 2024-09-11T01:32:15.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.759+0000 7f12a6ffd640 1 -- 172.21.15.12:0/196076051 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f12a8138230 msgr2=0x7f12a8138660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:32:15.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.759+0000 7f12a6ffd640 1 --2- 172.21.15.12:0/196076051 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f12a8138230 0x7f12a8138660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.759+0000 7f12a6ffd640 1 -- 172.21.15.12:0/196076051 >> v1:172.21.15.71:6789/0 conn(0x7f12a8134cb0 legacy=0x7f12a8135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:15.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.759+0000 7f12a6ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:32:15.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.759+0000 7f12a6ffd640 1 monclient: found mon.noname-c 2024-09-11T01:32:15.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.759+0000 7f12a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:15.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.759+0000 7f12a6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:32:15.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.759+0000 7f12a6ffd640 1 -- 172.21.15.12:0/196076051 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12a809c4a0 con 0x7f12a8137e50 2024-09-11T01:32:15.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.762+0000 7f12a6ffd640 1 -- 172.21.15.12:0/196076051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f12a8137e50 msgr2=0x7f12a809d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:15.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.762+0000 7f12a6ffd640 1 --2- 172.21.15.12:0/196076051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f12a8137e50 0x7f12a809d160 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f12980098f0 tx=0x7f129802a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e817ff855ebeabf server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:15.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.762+0000 7f12a5ffb640 1 -- 172.21.15.12:0/196076051 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1298037070 con 0x7f12a8137e50 2024-09-11T01:32:15.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.762+0000 7f12a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:15.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.762+0000 7f12a5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:32:15.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.762+0000 7f12a5ffb640 10 monclient: dump: 2024-09-11T01:32:15.765 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:15.765 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:15.765 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.765 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.765 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:15.765 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:15.765 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:15.765 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:15.765 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:15.766 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:15.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.762+0000 7f12a5ffb640 10 monclient: _finish_auth 0 2024-09-11T01:32:15.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.762+0000 7f12a5ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:32:15.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.762+0000 7f12a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:15.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.762+0000 7f12a5ffb640 1 -- 172.21.15.12:0/196076051 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1298032640 con 0x7f12a8137e50 2024-09-11T01:32:15.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.762+0000 7f12a5ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:15.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.762+0000 7f12a5ffb640 1 -- 172.21.15.12:0/196076051 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1298032a10 con 0x7f12a8137e50 2024-09-11T01:32:15.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.762+0000 7f12a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:15.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.762+0000 7f12a5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:32:15.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.762+0000 7f12a5ffb640 10 monclient: dump: 2024-09-11T01:32:15.766 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:15.766 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:15.766 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.766 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.767 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:15.767 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:15.767 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:15.767 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:15.767 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:15.767 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:15.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.762+0000 7f12ad55d640 5 monclient: authenticate success, global_id 34147 2024-09-11T01:32:15.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.762+0000 7f12ad55d640 10 monclient: get_monmap_and_config success 2024-09-11T01:32:15.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.762+0000 7f12ad55d640 4 set_mon_vals no callback set 2024-09-11T01:32:15.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.763+0000 7f12ad55d640 10 monclient: shutdown 2024-09-11T01:32:15.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.763+0000 7f12ad55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:15.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.763+0000 7f12ad55d640 1 -- 172.21.15.12:0/196076051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f12a8137e50 msgr2=0x7f12a809d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:15.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.763+0000 7f12ad55d640 1 --2- 172.21.15.12:0/196076051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f12a8137e50 0x7f12a809d160 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f12980098f0 tx=0x7f129802a1e0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.763+0000 7f12ad55d640 1 -- 172.21.15.12:0/196076051 shutdown_connections 2024-09-11T01:32:15.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.763+0000 7f12ad55d640 1 --2- 172.21.15.12:0/196076051 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f12a8138230 0x7f12a8138660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.763+0000 7f12ad55d640 1 --2- 172.21.15.12:0/196076051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f12a8137e50 0x7f12a809d160 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.763+0000 7f12ad55d640 1 -- 172.21.15.12:0/196076051 >> 172.21.15.12:0/196076051 conn(0x7f12a80a7020 msgr2=0x7f12a80a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:15.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.763+0000 7f12ad55d640 1 -- 172.21.15.12:0/196076051 shutdown_connections 2024-09-11T01:32:15.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.763+0000 7f12ad55d640 1 -- 172.21.15.12:0/196076051 wait complete. 2024-09-11T01:32:15.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.763+0000 7f12ad55d640 10 monclient: build_initial_monmap 2024-09-11T01:32:15.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.763+0000 7f12ad55d640 10 monclient: monmap: 2024-09-11T01:32:15.777 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:15.777 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:32:15.777 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:32:15.777 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:32:15.777 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:15.777 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:15.777 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:32:15.777 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:32:15.778 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:32:15.778 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:15.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.764+0000 7f12ad55d640 1 Processor -- start 2024-09-11T01:32:15.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.764+0000 7f12ad55d640 1 -- start start 2024-09-11T01:32:15.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.764+0000 7f12ad55d640 10 monclient: init 2024-09-11T01:32:15.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.764+0000 7f12ad55d640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:15.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.764+0000 7f12ad55d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:32:15.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.764+0000 7f12ad55d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f12a8134cb0 0x7f12a81cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:15.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.764+0000 7f12ad55d640 10 monclient(hunting): picked mon.noname-b con 0x7f12a8134cb0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:32:15.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.764+0000 7f12ad55d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f12a8137e50 0x7f12a8133cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:15.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.764+0000 7f12ad55d640 10 monclient(hunting): picked mon.noname-c con 0x7f12a8137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:15.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.764+0000 7f12ad55d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12a8138230 0x7f12a8134230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:15.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.764+0000 7f12ad55d640 10 monclient(hunting): picked mon.noname-a con 0x7f12a8138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:32:15.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.764+0000 7f12ad55d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:15.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.764+0000 7f12ad55d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f12a806f710 con 0x7f12a8138230 2024-09-11T01:32:15.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.764+0000 7f12ad55d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:15.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.764+0000 7f12ad55d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f12a8138ba0 con 0x7f12a8134cb0 2024-09-11T01:32:15.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.764+0000 7f12ad55d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:15.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.764+0000 7f12ad55d640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f12a806f890 con 0x7f12a8137e50 2024-09-11T01:32:15.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.764+0000 7f12ad55d640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:15.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.764+0000 7f12ad55d640 10 monclient(hunting): authenticate will time out at 942.206937s 2024-09-11T01:32:15.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.764+0000 7f12a67fc640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f12a8137e50 msgr2=0x7f12a8133cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:32:15.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.764+0000 7f12a67fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f12a8137e50 0x7f12a8133cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:15.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12a8138230 0x7f12a8134230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:15.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a6ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f12a8134cb0 0x7f12a81cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:15.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12a8138230 0x7f12a8134230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41076/0 (socket says 172.21.15.12:41076) 2024-09-11T01:32:15.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a77fe640 1 -- 172.21.15.12:0/3057520228 learned_addr learned my addr 172.21.15.12:0/3057520228 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:15.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a77fe640 10 monclient(hunting): get_auth_request con 0x7f12a8138230 auth_method 0 2024-09-11T01:32:15.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:15.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:15.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:15.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a6ffd640 10 monclient(hunting): get_auth_request con 0x7f12a8134cb0 auth_method 0 2024-09-11T01:32:15.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:15.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:15.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:15.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:15.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:15.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:15.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:15.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:15.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a77fe640 10 monclient(hunting): handle_auth_done global_id 24282 payload 824 2024-09-11T01:32:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a77fe640 1 -- 172.21.15.12:0/3057520228 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f12a8137e50 msgr2=0x7f12a8133cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:32:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a77fe640 1 --2- 172.21.15.12:0/3057520228 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f12a8137e50 0x7f12a8133cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a77fe640 1 -- 172.21.15.12:0/3057520228 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f12a8134cb0 msgr2=0x7f12a81cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a77fe640 1 --2- 172.21.15.12:0/3057520228 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f12a8134cb0 0x7f12a81cd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a77fe640 10 monclient: _finish_hunting 0 2024-09-11T01:32:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a77fe640 1 monclient: found mon.noname-a 2024-09-11T01:32:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:32:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a77fe640 1 -- 172.21.15.12:0/3057520228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12a80a1710 con 0x7f12a8138230 2024-09-11T01:32:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.765+0000 7f12a6ffd640 1 --2- 172.21.15.12:0/3057520228 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f12a8134cb0 0x7f12a81cd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:32:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.767+0000 7f12a77fe640 1 -- 172.21.15.12:0/3057520228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12a8138230 msgr2=0x7f12a8134230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:15.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.767+0000 7f12a77fe640 1 --2- 172.21.15.12:0/3057520228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12a8138230 0x7f12a8134230 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f12a0002230 tx=0x7f12a000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.767+0000 7f128f7fe640 1 -- 172.21.15.12:0/3057520228 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a0017070 con 0x7f12a8138230 2024-09-11T01:32:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.767+0000 7f128f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.767+0000 7f128f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:32:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.767+0000 7f128f7fe640 10 monclient: dump: 2024-09-11T01:32:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:15.782 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:15.782 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.767+0000 7f128f7fe640 10 monclient: _finish_auth 0 2024-09-11T01:32:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.767+0000 7f128f7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:32:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.767+0000 7f128f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.767+0000 7f128f7fe640 1 -- 172.21.15.12:0/3057520228 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12a000ce50 con 0x7f12a8138230 2024-09-11T01:32:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.767+0000 7f128f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.767+0000 7f128f7fe640 1 -- 172.21.15.12:0/3057520228 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a000a9e0 con 0x7f12a8138230 2024-09-11T01:32:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.767+0000 7f12ad55d640 5 monclient: authenticate success, global_id 24282 2024-09-11T01:32:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.767+0000 7f128f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.767+0000 7f128f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.767+0000 7f128f7fe640 10 monclient: dump: 2024-09-11T01:32:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:15.783 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.784 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.784 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:15.784 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:15.784 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:15.784 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:15.784 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:15.784 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:15.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.767+0000 7f12ad55d640 10 monclient: _renew_subs 2024-09-11T01:32:15.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.767+0000 7f12ad55d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:15.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.767+0000 7f12ad55d640 1 -- 172.21.15.12:0/3057520228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12a81256f0 con 0x7f12a8138230 2024-09-11T01:32:15.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.767+0000 7f12a57fa640 4 set_mon_vals no callback set 2024-09-11T01:32:15.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.767+0000 7f12ad55d640 10 monclient: _renew_subs 2024-09-11T01:32:15.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.767+0000 7f12ad55d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:15.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.767+0000 7f12ad55d640 1 -- 172.21.15.12:0/3057520228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12a8123390 con 0x7f12a8138230 2024-09-11T01:32:15.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.768+0000 7f128f7fe640 1 -- 172.21.15.12:0/3057520228 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f12a000ac00 con 0x7f12a8138230 2024-09-11T01:32:15.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.768+0000 7f12ad55d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:15.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.768+0000 7f12ad55d640 1 --2- 172.21.15.12:0/3057520228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12a81218d0 0x7f12a8121cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:15.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.768+0000 7f12ad55d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:15.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.768+0000 7f12ad55d640 1 -- 172.21.15.12:0/3057520228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f12a81252e0 con 0x7f12a81218d0 2024-09-11T01:32:15.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.768+0000 7f12a6ffd640 1 --2- 172.21.15.12:0/3057520228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12a81218d0 0x7f12a8121cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:15.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.769+0000 7f128f7fe640 1 --2- 172.21.15.12:0/3057520228 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f129403e1c0 0x7f1294040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:15.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.769+0000 7f12a6ffd640 10 monclient: get_auth_request con 0x7f12a81218d0 auth_method 0 2024-09-11T01:32:15.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.769+0000 7f12a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:15.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.769+0000 7f12a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:15.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.769+0000 7f12a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:15.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.769+0000 7f128f7fe640 1 -- 172.21.15.12:0/3057520228 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4767+0+0 (secure 0 0 0) 0x7f12a0052500 con 0x7f12a8138230 2024-09-11T01:32:15.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.769+0000 7f12a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:15.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.769+0000 7f12a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:15.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.769+0000 7f12a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:15.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.769+0000 7f12a67fc640 1 --2- 172.21.15.12:0/3057520228 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f129403e1c0 0x7f1294040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:15.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.769+0000 7f12a67fc640 10 monclient: get_auth_request con 0x7f129403e1c0 auth_method 0 2024-09-11T01:32:15.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.769+0000 7f12a6ffd640 10 monclient(hunting): handle_auth_done global_id 24285 payload 824 2024-09-11T01:32:15.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.769+0000 7f12a6ffd640 1 -- 172.21.15.12:0/3057520228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f12a8136110 con 0x7f12a81218d0 2024-09-11T01:32:15.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.771+0000 7f12a6ffd640 1 -- 172.21.15.12:0/3057520228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12a81218d0 msgr2=0x7f12a8121cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:15.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.771+0000 7f12a6ffd640 1 --2- 172.21.15.12:0/3057520228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12a81218d0 0x7f12a8121cb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1298034000 tx=0x7f12980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:15.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.771+0000 7f128f7fe640 1 -- 172.21.15.12:0/3057520228 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1298037040 con 0x7f12a81218d0 2024-09-11T01:32:15.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.771+0000 7f128f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:15.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.771+0000 7f128f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:15.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.771+0000 7f128f7fe640 10 monclient: dump: 2024-09-11T01:32:15.786 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:15.786 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:15.787 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.771+0000 7f12a67fc640 1 --2- 172.21.15.12:0/3057520228 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f129403e1c0 0x7f1294040680 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f129c006fd0 tx=0x7f129c006d80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.771+0000 7f128f7fe640 1 -- 172.21.15.12:0/3057520228 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f129802fad0 con 0x7f12a81218d0 2024-09-11T01:32:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.772+0000 7f128f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.772+0000 7f128f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:32:15.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.772+0000 7f128f7fe640 1 -- 172.21.15.12:0/3057520228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12a81218d0 msgr2=0x7f12a8121cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.772+0000 7f128f7fe640 1 --2- 172.21.15.12:0/3057520228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12a81218d0 0x7f12a8121cb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1298034000 tx=0x7f12980046c0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.772+0000 7f128f7fe640 1 -- 172.21.15.12:0/3057520228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12a81218d0 msgr2=0x7f12a8121cb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:32:15.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.772+0000 7f128f7fe640 1 --2- 172.21.15.12:0/3057520228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12a81218d0 0x7f12a8121cb0 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1298034000 tx=0x7f12980046c0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.778+0000 7f12ad55d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:32:15.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.778+0000 7f12ad55d640 1 --2- 172.21.15.12:0/3057520228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12a81221f0 0x7f12a80a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:15.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.778+0000 7f12ad55d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:15.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.778+0000 7f12ad55d640 1 -- 172.21.15.12:0/3057520228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f12a8069950 con 0x7f12a81221f0 2024-09-11T01:32:15.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.778+0000 7f12a6ffd640 1 --2- 172.21.15.12:0/3057520228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12a81221f0 0x7f12a80a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:15.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.778+0000 7f12a6ffd640 10 monclient: get_auth_request con 0x7f12a81221f0 auth_method 0 2024-09-11T01:32:15.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.778+0000 7f12a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:15.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.778+0000 7f12a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:15.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.778+0000 7f12a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:15.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.778+0000 7f12a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:15.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.778+0000 7f12a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:15.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.778+0000 7f12a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:15.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.778+0000 7f12a6ffd640 10 monclient(hunting): handle_auth_done global_id 24288 payload 824 2024-09-11T01:32:15.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.778+0000 7f12a6ffd640 1 -- 172.21.15.12:0/3057520228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f12a8124e50 con 0x7f12a81221f0 2024-09-11T01:32:15.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.780+0000 7f12a6ffd640 1 -- 172.21.15.12:0/3057520228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12a81221f0 msgr2=0x7f12a80a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:15.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.780+0000 7f12a6ffd640 1 --2- 172.21.15.12:0/3057520228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12a81221f0 0x7f12a80a1920 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1298000c00 tx=0x7f1298008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:15.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.780+0000 7f128f7fe640 1 -- 172.21.15.12:0/3057520228 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1298037040 con 0x7f12a81221f0 2024-09-11T01:32:15.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.780+0000 7f128f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:15.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.780+0000 7f128f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:15.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.780+0000 7f128f7fe640 10 monclient: dump: 2024-09-11T01:32:15.791 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:15.791 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:15.791 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.791 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.791 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:15.792 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:15.792 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:15.792 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:15.792 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:15.792 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:15.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.780+0000 7f128f7fe640 1 -- 172.21.15.12:0/3057520228 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4865 (secure 0 0 0) 0x7f12980463f0 con 0x7f12a81221f0 2024-09-11T01:32:15.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.780+0000 7f128f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:32:15.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.780+0000 7f128f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:32:15.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.780+0000 7f128f7fe640 1 -- 172.21.15.12:0/3057520228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12a81221f0 msgr2=0x7f12a80a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:15.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.780+0000 7f128f7fe640 1 --2- 172.21.15.12:0/3057520228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12a81221f0 0x7f12a80a1920 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1298000c00 tx=0x7f1298008030 comp rx=0 tx=0).stop 2024-09-11T01:32:15.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.780+0000 7f128f7fe640 1 -- 172.21.15.12:0/3057520228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12a81221f0 msgr2=0x7f12a80a1920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:32:15.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.780+0000 7f128f7fe640 1 --2- 172.21.15.12:0/3057520228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12a81221f0 0x7f12a80a1920 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1298000c00 tx=0x7f1298008030 comp rx=0 tx=0).stop 2024-09-11T01:32:15.793 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:32:15.793 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:32:15.793 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:32:15.793 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:32:15.793 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 32, 2024-09-11T01:32:15.793 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 73727, 2024-09-11T01:32:15.793 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:32:15.793 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:32:15.793 INFO:teuthology.orchestra.run.smithi012.stdout: 1 2024-09-11T01:32:15.793 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:15.793 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:32:15.793 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:32:15.794 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:32:15.794 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:32:15.794 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:15.794 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:15.794 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:15.794 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:15.794 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:15.794 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:15.794 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:15.794 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:15.794 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:15.794 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:15.794 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:15.795 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:15.795 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:32:15.795 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:32:15.795 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:15.795 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:15.795 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:15.795 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:15.795 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:15.795 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:15.795 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:15.795 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:15.795 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:15.795 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:15.796 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:15.796 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:15.796 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:15.796 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:32:15.796 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:32:15.796 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:32:15.796 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:32:15.796 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:32:15.796 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:32:15.796 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:32:15.796 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:32:15.796 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:32:15.796 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:32:15.797 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:32:15.797 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:32:15.797 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:32:15.797 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:32:15.797 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:32:15.797 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:32:15.797 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:32:15.797 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:15.797 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:15.797 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:15.797 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:15.797 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:15.797 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:15.798 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:15.798 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:15.798 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:15.798 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:15.798 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:15.798 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:15.798 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:32:15.798 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:15.798 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:32:15.798 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:15.798 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:32:15.798 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:32:15.798 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:15.799 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:15.799 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:15.799 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:15.799 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:32:15.799 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:15.799 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:15.799 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:15.799 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:15.799 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:32:15.799 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:15.799 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:15.799 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:15.799 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:15.799 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:32:15.800 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:32:15.800 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:15.800 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:15.800 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:15.800 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:15.800 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:15.800 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:32:15.800 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:32:15.800 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:15.800 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:15.800 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:15.800 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:15.800 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:32:15.801 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:15.801 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:15.801 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:15.801 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:15.801 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:32:15.801 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:15.801 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:15.801 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:15.801 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:15.801 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:32:15.801 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:32:15.801 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:15.801 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:15.801 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:15.802 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:15.802 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:15.802 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:32:15.802 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:32:15.802 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:15.802 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:15.802 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:15.802 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:15.802 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:32:15.802 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:15.802 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:15.802 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:15.802 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:15.802 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:32:15.803 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:15.803 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:15.803 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:15.803 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:15.803 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:32:15.803 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:32:15.803 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:15.803 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:15.803 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:15.803 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:15.803 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:15.803 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:15.803 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:32:15.803 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:32:15.804 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:15.804 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:15.804 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:15.804 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:32:15.804 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:15.804 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:15.804 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:32:15.804 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:15.804 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:15.804 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:15.804 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:32:15.804 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:15.804 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:15.805 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:32:15.805 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:15.805 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:15.805 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:15.805 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:32:15.805 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:15.805 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:15.805 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:32:15.805 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:15.805 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:15.805 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:15.805 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:32:15.805 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:15.806 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:15.806 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:15.806 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:32:15.806 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:32:15.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.781+0000 7f12ad55d640 1 -- 172.21.15.12:0/3057520228 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f129403e1c0 msgr2=0x7f1294040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:15.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.781+0000 7f12ad55d640 1 --2- 172.21.15.12:0/3057520228 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f129403e1c0 0x7f1294040680 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f129c006fd0 tx=0x7f129c006d80 comp rx=0 tx=0).stop 2024-09-11T01:32:15.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.781+0000 7f12ad55d640 10 monclient: shutdown 2024-09-11T01:32:15.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.781+0000 7f12ad55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:15.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.781+0000 7f12ad55d640 1 -- 172.21.15.12:0/3057520228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12a8138230 msgr2=0x7f12a8134230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:15.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.781+0000 7f12a77fe640 1 -- 172.21.15.12:0/3057520228 reap_dead start 2024-09-11T01:32:15.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.781+0000 7f12ad55d640 1 --2- 172.21.15.12:0/3057520228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12a8138230 0x7f12a8134230 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f12a0002230 tx=0x7f12a000cc00 comp rx=0 tx=0).stop 2024-09-11T01:32:15.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.781+0000 7f12ad55d640 1 -- 172.21.15.12:0/3057520228 shutdown_connections 2024-09-11T01:32:15.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.781+0000 7f12ad55d640 1 --2- 172.21.15.12:0/3057520228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12a8138230 0x7f12a8134230 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.781+0000 7f12ad55d640 1 -- 172.21.15.12:0/3057520228 >> 172.21.15.12:0/3057520228 conn(0x7f12a80a7020 msgr2=0x7f12a80a9630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:15.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.781+0000 7f12ad55d640 1 -- 172.21.15.12:0/3057520228 shutdown_connections 2024-09-11T01:32:15.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.781+0000 7f12ad55d640 1 -- 172.21.15.12:0/3057520228 wait complete. 2024-09-11T01:32:15.808 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:32:15.808 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T01:32:15.808 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:32:15.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.909+0000 7ff436a7f640 10 monclient: get_monmap_and_config 2024-09-11T01:32:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.909+0000 7ff436a7f640 10 monclient: build_initial_monmap 2024-09-11T01:32:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.909+0000 7ff436a7f640 10 monclient: monmap: 2024-09-11T01:32:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:32:15.909839+0000 2024-09-11T01:32:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:32:15.909839+0000 2024-09-11T01:32:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:32:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:32:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:32:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:32:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:32:15.911 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:15.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.909+0000 7ff436a7f640 1 Processor -- start 2024-09-11T01:32:15.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.909+0000 7ff436a7f640 1 -- start start 2024-09-11T01:32:15.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.909+0000 7ff436a7f640 10 monclient: init 2024-09-11T01:32:15.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff436a7f640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:15.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff436a7f640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-11T01:32:15.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff436a7f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff430137d50 0x7ff43009d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:15.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff436a7f640 10 monclient(hunting): picked mon.noname-c con 0x7ff430137d50 addr v2:172.21.15.71:3300/0 2024-09-11T01:32:15.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff436a7f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff430138130 0x7ff430138580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:15.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff436a7f640 10 monclient(hunting): picked mon.noname-a con 0x7ff430138130 addr v2:172.21.15.12:3300/0 2024-09-11T01:32:15.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff436a7f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff430136cd0 addr v1:172.21.15.71:6789/0 2024-09-11T01:32:15.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff436a7f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff43009ba20 con 0x7ff430136cd0 2024-09-11T01:32:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff436a7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff436a7f640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff43009c650 con 0x7ff430138130 2024-09-11T01:32:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff436a7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff436a7f640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7ff43009c370 con 0x7ff430137d50 2024-09-11T01:32:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff436a7f640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff436a7f640 10 monclient(hunting): authenticate will time out at 942.352413s 2024-09-11T01:32:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff42ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff430138130 0x7ff430138580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff42ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff430138130 0x7ff430138580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41108/0 (socket says 172.21.15.12:41108) 2024-09-11T01:32:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff42ffff640 1 -- 172.21.15.12:0/380095173 learned_addr learned my addr 172.21.15.12:0/380095173 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff42ffff640 10 monclient(hunting): get_auth_request con 0x7ff430138130 auth_method 0 2024-09-11T01:32:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff42ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff42ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff42ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff43481b640 1 --2- 172.21.15.12:0/380095173 >> v2:172.21.15.71:3300/0 conn(0x7ff430137d50 0x7ff43009d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff43481b640 10 monclient(hunting): get_auth_request con 0x7ff430137d50 auth_method 0 2024-09-11T01:32:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff43481b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff43481b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff43481b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff42ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff42ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff42ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff43481b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff43481b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff43481b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff42ffff640 10 monclient(hunting): handle_auth_done global_id 24291 payload 274 2024-09-11T01:32:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff42ffff640 1 -- 172.21.15.12:0/380095173 >> v2:172.21.15.71:3300/0 conn(0x7ff430137d50 msgr2=0x7ff43009d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff42ffff640 1 --2- 172.21.15.12:0/380095173 >> v2:172.21.15.71:3300/0 conn(0x7ff430137d50 0x7ff43009d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff42ffff640 1 -- 172.21.15.12:0/380095173 >> v1:172.21.15.71:6789/0 conn(0x7ff430136cd0 legacy=0x7ff430137120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:15.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff42ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:32:15.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff42ffff640 1 monclient: found mon.noname-a 2024-09-11T01:32:15.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff42ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:15.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff42ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:32:15.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.910+0000 7ff42ffff640 1 -- 172.21.15.12:0/380095173 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4300a6ce0 con 0x7ff430138130 2024-09-11T01:32:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.913+0000 7ff42ffff640 1 -- 172.21.15.12:0/380095173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff430138130 msgr2=0x7ff430138580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.913+0000 7ff42ffff640 1 --2- 172.21.15.12:0/380095173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff430138130 0x7ff430138580 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff43006d4f0 tx=0x7ff41802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e480551f38ab347b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.913+0000 7ff42f7fe640 1 -- 172.21.15.12:0/380095173 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff418037070 con 0x7ff430138130 2024-09-11T01:32:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.913+0000 7ff42f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.913+0000 7ff42f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:32:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.913+0000 7ff42f7fe640 10 monclient: dump: 2024-09-11T01:32:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:15.924 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.913+0000 7ff42f7fe640 10 monclient: _finish_auth 0 2024-09-11T01:32:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.913+0000 7ff42f7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:32:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.913+0000 7ff42f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.913+0000 7ff42f7fe640 1 -- 172.21.15.12:0/380095173 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff418032640 con 0x7ff430138130 2024-09-11T01:32:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.913+0000 7ff436a7f640 5 monclient: authenticate success, global_id 24291 2024-09-11T01:32:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.913+0000 7ff436a7f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:32:15.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.913+0000 7ff42f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:15.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.913+0000 7ff436a7f640 10 monclient: get_monmap_and_config success 2024-09-11T01:32:15.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.913+0000 7ff42f7fe640 1 -- 172.21.15.12:0/380095173 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff418032a10 con 0x7ff430138130 2024-09-11T01:32:15.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.913+0000 7ff436a7f640 4 set_mon_vals no callback set 2024-09-11T01:32:15.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.913+0000 7ff42f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:15.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.913+0000 7ff42f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:15.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.913+0000 7ff42f7fe640 10 monclient: dump: 2024-09-11T01:32:15.925 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:15.925 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:15.925 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.925 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.925 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:15.925 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:15.926 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:15.926 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:15.926 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:15.926 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:15.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.914+0000 7ff436a7f640 10 monclient: shutdown 2024-09-11T01:32:15.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.914+0000 7ff436a7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:15.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.914+0000 7ff436a7f640 1 -- 172.21.15.12:0/380095173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff430138130 msgr2=0x7ff430138580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:15.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.914+0000 7ff436a7f640 1 --2- 172.21.15.12:0/380095173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff430138130 0x7ff430138580 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff43006d4f0 tx=0x7ff41802a140 comp rx=0 tx=0).stop 2024-09-11T01:32:15.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.914+0000 7ff436a7f640 1 -- 172.21.15.12:0/380095173 shutdown_connections 2024-09-11T01:32:15.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.914+0000 7ff436a7f640 1 --2- 172.21.15.12:0/380095173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff430138130 0x7ff430138580 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.914+0000 7ff436a7f640 1 --2- 172.21.15.12:0/380095173 >> v2:172.21.15.71:3300/0 conn(0x7ff430137d50 0x7ff43009d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.914+0000 7ff436a7f640 1 -- 172.21.15.12:0/380095173 >> 172.21.15.12:0/380095173 conn(0x7ff4300a7020 msgr2=0x7ff4300a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:15.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.914+0000 7ff436a7f640 1 -- 172.21.15.12:0/380095173 shutdown_connections 2024-09-11T01:32:15.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.914+0000 7ff436a7f640 1 -- 172.21.15.12:0/380095173 wait complete. 2024-09-11T01:32:15.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.915+0000 7ff436a7f640 10 monclient: build_initial_monmap 2024-09-11T01:32:15.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.915+0000 7ff436a7f640 10 monclient: monmap: 2024-09-11T01:32:15.927 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:15.927 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:32:15.927 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:32:15.927 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:32:15.927 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:15.927 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:15.927 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:32:15.927 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:32:15.928 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:32:15.928 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:15.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.915+0000 7ff436a7f640 1 Processor -- start 2024-09-11T01:32:15.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.915+0000 7ff436a7f640 1 -- start start 2024-09-11T01:32:15.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.915+0000 7ff436a7f640 10 monclient: init 2024-09-11T01:32:15.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.915+0000 7ff436a7f640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:15.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.915+0000 7ff436a7f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:32:15.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.915+0000 7ff436a7f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff430136cd0 0x7ff4301cd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:15.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.915+0000 7ff436a7f640 10 monclient(hunting): picked mon.noname-a con 0x7ff430136cd0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:32:15.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.915+0000 7ff436a7f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff430137d50 0x7ff4301cd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:15.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.915+0000 7ff436a7f640 10 monclient(hunting): picked mon.noname-c con 0x7ff430137d50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:15.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.915+0000 7ff436a7f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff430138130 0x7ff4301234c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:15.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.915+0000 7ff436a7f640 10 monclient(hunting): picked mon.noname-b con 0x7ff430138130 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:32:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.915+0000 7ff436a7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff436a7f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff43009d650 con 0x7ff430136cd0 2024-09-11T01:32:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff436a7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff436a7f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff430138b80 con 0x7ff430138130 2024-09-11T01:32:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff436a7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff436a7f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff43009d7d0 con 0x7ff430137d50 2024-09-11T01:32:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff436a7f640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff436a7f640 10 monclient(hunting): authenticate will time out at 942.358022s 2024-09-11T01:32:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff42ffff640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff430137d50 msgr2=0x7ff4301cd810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:32:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff42ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff430137d50 0x7ff4301cd810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff43481b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff430136cd0 0x7ff4301cd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff43481b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff430136cd0 0x7ff4301cd2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41124/0 (socket says 172.21.15.12:41124) 2024-09-11T01:32:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff43481b640 1 -- 172.21.15.12:0/222025489 learned_addr learned my addr 172.21.15.12:0/222025489 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:15.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff43481b640 10 monclient(hunting): get_auth_request con 0x7ff430136cd0 auth_method 0 2024-09-11T01:32:15.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff43481b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:15.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff43481b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:15.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff43481b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:15.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff43481b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:15.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff43481b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:15.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff43481b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:15.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff43501c640 1 --2- 172.21.15.12:0/222025489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff430138130 0x7ff4301234c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:15.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff43481b640 10 monclient(hunting): handle_auth_done global_id 24294 payload 824 2024-09-11T01:32:15.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff43481b640 1 -- 172.21.15.12:0/222025489 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff430137d50 msgr2=0x7ff4301cd810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:32:15.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff43481b640 1 --2- 172.21.15.12:0/222025489 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff430137d50 0x7ff4301cd810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff43481b640 1 -- 172.21.15.12:0/222025489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff430138130 msgr2=0x7ff4301234c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:15.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff43481b640 1 --2- 172.21.15.12:0/222025489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff430138130 0x7ff4301234c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff43481b640 10 monclient: _finish_hunting 0 2024-09-11T01:32:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff43481b640 1 monclient: found mon.noname-a 2024-09-11T01:32:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff43481b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff43481b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:32:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff43481b640 1 -- 172.21.15.12:0/222025489 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4301272e0 con 0x7ff430136cd0 2024-09-11T01:32:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff43501c640 10 monclient: get_auth_request con 0x7ff430138130 auth_method 0 2024-09-11T01:32:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.916+0000 7ff43501c640 1 --2- 172.21.15.12:0/222025489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff430138130 0x7ff4301234c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:32:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.918+0000 7ff43481b640 1 -- 172.21.15.12:0/222025489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff430136cd0 msgr2=0x7ff4301cd2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.918+0000 7ff43481b640 1 --2- 172.21.15.12:0/222025489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff430136cd0 0x7ff4301cd2d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff42000e9b0 tx=0x7ff42000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.918+0000 7ff42d7fa640 1 -- 172.21.15.12:0/222025489 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff420017070 con 0x7ff430136cd0 2024-09-11T01:32:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.918+0000 7ff42d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.918+0000 7ff42d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:32:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.918+0000 7ff42d7fa640 10 monclient: dump: 2024-09-11T01:32:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:15.932 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:15.932 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.932 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.932 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:15.932 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:15.932 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:15.932 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:15.932 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:15.932 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:15.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.918+0000 7ff42d7fa640 10 monclient: _finish_auth 0 2024-09-11T01:32:15.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.918+0000 7ff42d7fa640 10 monclient: _check_auth_tickets 2024-09-11T01:32:15.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.918+0000 7ff42d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:15.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.918+0000 7ff42d7fa640 1 -- 172.21.15.12:0/222025489 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff42000cbb0 con 0x7ff430136cd0 2024-09-11T01:32:15.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.918+0000 7ff436a7f640 5 monclient: authenticate success, global_id 24294 2024-09-11T01:32:15.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.918+0000 7ff436a7f640 10 monclient: _renew_subs 2024-09-11T01:32:15.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.918+0000 7ff436a7f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:15.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.918+0000 7ff436a7f640 1 -- 172.21.15.12:0/222025489 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4300a1710 con 0x7ff430136cd0 2024-09-11T01:32:15.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.918+0000 7ff42d7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:15.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.918+0000 7ff42d7fa640 1 -- 172.21.15.12:0/222025489 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff420013470 con 0x7ff430136cd0 2024-09-11T01:32:15.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.918+0000 7ff42d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:15.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.918+0000 7ff42d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:15.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.918+0000 7ff42d7fa640 10 monclient: dump: 2024-09-11T01:32:15.933 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:15.933 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:15.933 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.933 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.933 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:15.934 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:15.934 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:15.934 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:15.934 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:15.934 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:15.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.918+0000 7ff436a7f640 10 monclient: _renew_subs 2024-09-11T01:32:15.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.918+0000 7ff436a7f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:15.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.918+0000 7ff436a7f640 1 -- 172.21.15.12:0/222025489 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff430127000 con 0x7ff430136cd0 2024-09-11T01:32:15.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.918+0000 7ff42effd640 4 set_mon_vals no callback set 2024-09-11T01:32:15.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.919+0000 7ff436a7f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:15.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.919+0000 7ff436a7f640 1 --2- 172.21.15.12:0/222025489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4301241c0 0x7ff43002d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:15.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.919+0000 7ff436a7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:15.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.919+0000 7ff436a7f640 1 -- 172.21.15.12:0/222025489 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff430126bd0 con 0x7ff4301241c0 2024-09-11T01:32:15.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.919+0000 7ff42d7fa640 1 -- 172.21.15.12:0/222025489 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff420024060 con 0x7ff430136cd0 2024-09-11T01:32:15.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.920+0000 7ff42ffff640 1 --2- 172.21.15.12:0/222025489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4301241c0 0x7ff43002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:15.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.920+0000 7ff42ffff640 10 monclient: get_auth_request con 0x7ff4301241c0 auth_method 0 2024-09-11T01:32:15.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.920+0000 7ff42ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:15.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.920+0000 7ff42ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:15.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.920+0000 7ff42ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:15.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.920+0000 7ff42ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:15.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.920+0000 7ff42ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:15.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.920+0000 7ff42ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:15.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.920+0000 7ff42d7fa640 1 --2- 172.21.15.12:0/222025489 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff41003e1c0 0x7ff410040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:15.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.920+0000 7ff42d7fa640 1 -- 172.21.15.12:0/222025489 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4767+0+0 (secure 0 0 0) 0x7ff420052d50 con 0x7ff430136cd0 2024-09-11T01:32:15.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.920+0000 7ff42ffff640 10 monclient(hunting): handle_auth_done global_id 34159 payload 824 2024-09-11T01:32:15.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.920+0000 7ff42ffff640 1 -- 172.21.15.12:0/222025489 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff4301378b0 con 0x7ff4301241c0 2024-09-11T01:32:15.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.920+0000 7ff43501c640 1 --2- 172.21.15.12:0/222025489 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff41003e1c0 0x7ff410040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:15.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.920+0000 7ff43501c640 10 monclient: get_auth_request con 0x7ff41003e1c0 auth_method 0 2024-09-11T01:32:15.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.922+0000 7ff42ffff640 1 -- 172.21.15.12:0/222025489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4301241c0 msgr2=0x7ff43002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:15.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.922+0000 7ff42ffff640 1 --2- 172.21.15.12:0/222025489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4301241c0 0x7ff43002d440 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff418034000 tx=0x7ff4180046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:15.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.922+0000 7ff42d7fa640 1 -- 172.21.15.12:0/222025489 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff418042020 con 0x7ff4301241c0 2024-09-11T01:32:15.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.922+0000 7ff42d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:15.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.922+0000 7ff42d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:32:15.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.922+0000 7ff42d7fa640 10 monclient: dump: 2024-09-11T01:32:15.938 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:15.939 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:15.939 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.939 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.939 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:15.939 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:15.939 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:15.939 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:15.939 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:15.939 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:15.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.922+0000 7ff43501c640 1 --2- 172.21.15.12:0/222025489 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff41003e1c0 0x7ff410040680 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff4280044a0 tx=0x7ff428009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:15.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.923+0000 7ff42d7fa640 1 -- 172.21.15.12:0/222025489 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff41803d880 con 0x7ff4301241c0 2024-09-11T01:32:15.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.923+0000 7ff42d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:15.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.923+0000 7ff42d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:32:15.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.923+0000 7ff42d7fa640 1 -- 172.21.15.12:0/222025489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4301241c0 msgr2=0x7ff43002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:15.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.923+0000 7ff42d7fa640 1 --2- 172.21.15.12:0/222025489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4301241c0 0x7ff43002d440 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff418034000 tx=0x7ff4180046c0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.923+0000 7ff42d7fa640 1 -- 172.21.15.12:0/222025489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4301241c0 msgr2=0x7ff43002d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:32:15.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.923+0000 7ff42d7fa640 1 --2- 172.21.15.12:0/222025489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4301241c0 0x7ff43002d440 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff418034000 tx=0x7ff4180046c0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.929+0000 7ff436a7f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:32:15.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.929+0000 7ff436a7f640 1 --2- 172.21.15.12:0/222025489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4301245a0 0x7ff43002d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:15.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.929+0000 7ff436a7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:15.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.929+0000 7ff436a7f640 1 -- 172.21.15.12:0/222025489 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff4300a0760 con 0x7ff4301245a0 2024-09-11T01:32:15.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.930+0000 7ff42ffff640 1 --2- 172.21.15.12:0/222025489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4301245a0 0x7ff43002d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:15.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.930+0000 7ff42ffff640 10 monclient: get_auth_request con 0x7ff4301245a0 auth_method 0 2024-09-11T01:32:15.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.930+0000 7ff42ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:15.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.930+0000 7ff42ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:15.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.930+0000 7ff42ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:15.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.930+0000 7ff42ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:15.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.930+0000 7ff42ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:15.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.930+0000 7ff42ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:15.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.930+0000 7ff42ffff640 10 monclient(hunting): handle_auth_done global_id 34162 payload 824 2024-09-11T01:32:15.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.930+0000 7ff42ffff640 1 -- 172.21.15.12:0/222025489 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff430125ae0 con 0x7ff4301245a0 2024-09-11T01:32:15.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.932+0000 7ff42ffff640 1 -- 172.21.15.12:0/222025489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4301245a0 msgr2=0x7ff43002d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:15.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.932+0000 7ff42ffff640 1 --2- 172.21.15.12:0/222025489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4301245a0 0x7ff43002d980 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff418000c00 tx=0x7ff418009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:15.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.933+0000 7ff42d7fa640 1 -- 172.21.15.12:0/222025489 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff418042040 con 0x7ff4301245a0 2024-09-11T01:32:15.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.933+0000 7ff42d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:15.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.933+0000 7ff42d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:32:15.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.933+0000 7ff42d7fa640 10 monclient: dump: 2024-09-11T01:32:15.941 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:15.941 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:15.942 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.942 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:15.942 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:15.942 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:15.942 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:15.942 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:15.942 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:15.942 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:15.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.933+0000 7ff42d7fa640 1 -- 172.21.15.12:0/222025489 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4696 (secure 0 0 0) 0x7ff418004460 con 0x7ff4301245a0 2024-09-11T01:32:15.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.933+0000 7ff42d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:32:15.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.933+0000 7ff42d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:32:15.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.933+0000 7ff42d7fa640 1 -- 172.21.15.12:0/222025489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4301245a0 msgr2=0x7ff43002d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:15.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.933+0000 7ff42d7fa640 1 --2- 172.21.15.12:0/222025489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4301245a0 0x7ff43002d980 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff418000c00 tx=0x7ff418009240 comp rx=0 tx=0).stop 2024-09-11T01:32:15.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.933+0000 7ff42d7fa640 1 -- 172.21.15.12:0/222025489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4301245a0 msgr2=0x7ff43002d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:32:15.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.933+0000 7ff42d7fa640 1 --2- 172.21.15.12:0/222025489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4301245a0 0x7ff43002d980 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff418000c00 tx=0x7ff418009240 comp rx=0 tx=0).stop 2024-09-11T01:32:15.943 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:32:15.943 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:32:15.943 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:32:15.943 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:32:15.943 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 32, 2024-09-11T01:32:15.943 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 29146, 2024-09-11T01:32:15.943 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:32:15.944 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:32:15.944 INFO:teuthology.orchestra.run.smithi012.stdout: 1 2024-09-11T01:32:15.944 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:15.944 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:32:15.944 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:32:15.944 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:32:15.944 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:32:15.944 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:15.944 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:15.944 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:15.944 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:15.944 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:15.944 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:15.944 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:15.945 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:15.945 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:15.945 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:15.945 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:15.945 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:15.945 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:32:15.945 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:32:15.945 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:15.945 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:15.945 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:15.945 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:15.945 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:15.945 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:15.945 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:15.946 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:15.946 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:15.946 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:15.946 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:15.946 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:15.946 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:15.946 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:32:15.946 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:32:15.946 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:32:15.946 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:32:15.946 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:32:15.946 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:32:15.946 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:32:15.946 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:32:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:32:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:32:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:32:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:32:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:32:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:32:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:32:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:32:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:32:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:32:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:32:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:32:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:32:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:32:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:32:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:32:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:32:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:32:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:32:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:32:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:32:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:32:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:32:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:15.952 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:15.952 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:15.952 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:32:15.952 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:32:15.952 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:15.952 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:15.952 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:15.952 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:15.952 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:32:15.952 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:15.952 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:15.952 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:15.952 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:15.952 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:32:15.953 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:15.953 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:15.953 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:15.953 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:15.953 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:32:15.953 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:32:15.953 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:15.953 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:15.953 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:15.953 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:15.953 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:15.953 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:15.953 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:32:15.953 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:32:15.954 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:15.954 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:15.954 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:15.954 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:32:15.954 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:15.954 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:15.954 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:32:15.954 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:15.954 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:15.954 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:15.954 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:32:15.954 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:15.954 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:15.954 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:32:15.955 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:15.955 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:15.955 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:15.955 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:32:15.955 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:15.955 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:15.955 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:15.955 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:32:15.955 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:32:15.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.933+0000 7ff436a7f640 1 -- 172.21.15.12:0/222025489 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff41003e1c0 msgr2=0x7ff410040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:15.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.933+0000 7ff436a7f640 1 --2- 172.21.15.12:0/222025489 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff41003e1c0 0x7ff410040680 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff4280044a0 tx=0x7ff428009040 comp rx=0 tx=0).stop 2024-09-11T01:32:15.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.933+0000 7ff436a7f640 10 monclient: shutdown 2024-09-11T01:32:15.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.933+0000 7ff436a7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:15.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.933+0000 7ff436a7f640 1 -- 172.21.15.12:0/222025489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff430136cd0 msgr2=0x7ff4301cd2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:15.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.933+0000 7ff436a7f640 1 --2- 172.21.15.12:0/222025489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff430136cd0 0x7ff4301cd2d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff42000e9b0 tx=0x7ff42000ee70 comp rx=0 tx=0).stop 2024-09-11T01:32:15.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.933+0000 7ff43501c640 1 -- 172.21.15.12:0/222025489 reap_dead start 2024-09-11T01:32:15.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.933+0000 7ff436a7f640 1 -- 172.21.15.12:0/222025489 shutdown_connections 2024-09-11T01:32:15.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.933+0000 7ff436a7f640 1 --2- 172.21.15.12:0/222025489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff430136cd0 0x7ff4301cd2d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:15.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.933+0000 7ff436a7f640 1 -- 172.21.15.12:0/222025489 >> 172.21.15.12:0/222025489 conn(0x7ff4300a7020 msgr2=0x7ff4300a5740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:15.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.933+0000 7ff436a7f640 1 -- 172.21.15.12:0/222025489 shutdown_connections 2024-09-11T01:32:15.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:15.933+0000 7ff436a7f640 1 -- 172.21.15.12:0/222025489 wait complete. 2024-09-11T01:32:15.958 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:32:15.958 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:32:16.366 INFO:tasks.rados.rados.0.smithi071.stdout:460: done (6 left) 2024-09-11T01:32:16.367 INFO:tasks.rados.rados.0.smithi071.stdout:461: done (5 left) 2024-09-11T01:32:16.367 INFO:tasks.rados.rados.0.smithi071.stdout:462: done (4 left) 2024-09-11T01:32:16.367 INFO:tasks.rados.rados.0.smithi071.stdout:463: done (3 left) 2024-09-11T01:32:16.367 INFO:tasks.rados.rados.0.smithi071.stdout:464: done (2 left) 2024-09-11T01:32:16.367 INFO:tasks.rados.rados.0.smithi071.stdout:465: done (1 left) 2024-09-11T01:32:16.367 INFO:tasks.rados.rados.0.smithi071.stdout:466: done (0 left) 2024-09-11T01:32:16.367 INFO:tasks.rados.rados.0.smithi071.stdout:467: delete oid 21 current snap is 42 2024-09-11T01:32:16.370 INFO:tasks.rados.rados.0.smithi071.stdout:467: done (0 left) 2024-09-11T01:32:16.370 INFO:tasks.rados.rados.0.smithi071.stdout:468: rollback oid 18 current snap is 42 2024-09-11T01:32:16.370 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 18 to 41 2024-09-11T01:32:16.370 INFO:tasks.rados.rados.0.smithi071.stdout:469: read oid 38 snap -1 2024-09-11T01:32:16.370 INFO:tasks.rados.rados.0.smithi071.stdout:469: expect deleted 2024-09-11T01:32:16.370 INFO:tasks.rados.rados.0.smithi071.stdout:470: write (excl) oid 14 current snap is 42 2024-09-11T01:32:16.370 INFO:tasks.rados.rados.0.smithi071.stdout:470: seq_num 189 ranges {776523=614533,1976979=757720} 2024-09-11T01:32:16.378 INFO:tasks.rados.rados.0.smithi071.stdout:470: writing smithi07134777-14 from 776523 to 1391056 tid 1 2024-09-11T01:32:16.386 INFO:tasks.rados.rados.0.smithi071.stdout:470: writing smithi07134777-14 from 1976979 to 2734699 tid 2 2024-09-11T01:32:16.386 INFO:tasks.rados.rados.0.smithi071.stdout:471: rollback oid 25 current snap is 42 2024-09-11T01:32:16.386 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 25 to 40 2024-09-11T01:32:16.386 INFO:tasks.rados.rados.0.smithi071.stdout:468: finishing rollback tid 0 to smithi07134777-18 2024-09-11T01:32:16.386 INFO:tasks.rados.rados.0.smithi071.stdout:468: finishing rollback tid 1 to smithi07134777-18 2024-09-11T01:32:16.387 INFO:tasks.rados.rados.0.smithi071.stdout:468: finishing rollback tid 2 to smithi07134777-18 2024-09-11T01:32:16.387 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 140 (ObjNum 157 snap 34 seq_num 157) dirty exists 2024-09-11T01:32:16.387 INFO:tasks.rados.rados.0.smithi071.stdout:470: finishing write tid 1 to smithi07134777-14 2024-09-11T01:32:16.387 INFO:tasks.rados.rados.0.smithi071.stdout:468: done (3 left) 2024-09-11T01:32:16.387 INFO:tasks.rados.rados.0.smithi071.stdout:469: done (2 left) 2024-09-11T01:32:16.387 INFO:tasks.rados.rados.0.smithi071.stdout:472: read oid 30 snap -1 2024-09-11T01:32:16.387 INFO:tasks.rados.rados.0.smithi071.stdout:472: expect (ObjNum 185 snap 41 seq_num 185) 2024-09-11T01:32:16.387 INFO:tasks.rados.rados.0.smithi071.stdout:473: snap_remove snap 41 2024-09-11T01:32:17.079 INFO:tasks.rados.rados.0.smithi071.stdout:473: done (3 left) 2024-09-11T01:32:17.079 INFO:tasks.rados.rados.0.smithi071.stdout:474: copy_from oid 44 from oid 50 current snap is 42 2024-09-11T01:32:17.079 INFO:tasks.rados.rados.0.smithi071.stdout:470: finishing write tid 2 to smithi07134777-14 2024-09-11T01:32:17.079 INFO:tasks.rados.rados.0.smithi071.stdout:471: finishing rollback tid 0 to smithi07134777-25 2024-09-11T01:32:17.079 INFO:tasks.rados.rados.0.smithi071.stdout:470: finishing write tid 3 to smithi07134777-14 2024-09-11T01:32:17.079 INFO:tasks.rados.rados.0.smithi071.stdout:471: finishing rollback tid 1 to smithi07134777-25 2024-09-11T01:32:17.079 INFO:tasks.rados.rados.0.smithi071.stdout:470: finishing write tid 4 to smithi07134777-14 2024-09-11T01:32:17.079 INFO:tasks.rados.rados.0.smithi071.stdout:470: oid 14 updating version 0 to 218 2024-09-11T01:32:17.079 INFO:tasks.rados.rados.0.smithi071.stdout:470: oid 14 updating version 218 to 219 2024-09-11T01:32:17.079 INFO:tasks.rados.rados.0.smithi071.stdout:470: oid 14 version 219 is already newer than 217 2024-09-11T01:32:17.079 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 219 (ObjNum 189 snap 42 seq_num 189) dirty exists 2024-09-11T01:32:17.080 INFO:tasks.rados.rados.0.smithi071.stdout:470: left oid 14 (ObjNum 189 snap 42 seq_num 189) 2024-09-11T01:32:17.080 INFO:tasks.rados.rados.0.smithi071.stdout:471: finishing rollback tid 2 to smithi07134777-25 2024-09-11T01:32:17.080 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 38 (ObjNum 173 snap 37 seq_num 173) dirty exists 2024-09-11T01:32:17.080 INFO:tasks.rados.rados.0.smithi071.stdout:472: expect (ObjNum 185 snap 41 seq_num 185) 2024-09-11T01:32:17.080 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:32:17.120 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 861483 2024-09-11T01:32:17.154 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2219414 2024-09-11T01:32:17.165 INFO:tasks.rados.rados.0.smithi071.stdout:474: finishing copy_from racing read to smithi07134777-44 2024-09-11T01:32:17.165 INFO:tasks.rados.rados.0.smithi071.stdout:474: finishing copy_from to smithi07134777-44 2024-09-11T01:32:17.165 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 206 (ObjNum 171 snap 37 seq_num 171) dirty exists 2024-09-11T01:32:17.165 INFO:tasks.rados.rados.0.smithi071.stdout:470: done (3 left) 2024-09-11T01:32:17.165 INFO:tasks.rados.rados.0.smithi071.stdout:471: done (2 left) 2024-09-11T01:32:17.165 INFO:tasks.rados.rados.0.smithi071.stdout:472: done (1 left) 2024-09-11T01:32:17.165 INFO:tasks.rados.rados.0.smithi071.stdout:474: done (0 left) 2024-09-11T01:32:17.165 INFO:tasks.rados.rados.0.smithi071.stdout:475: copy_from oid 30 from oid 39 current snap is 42 2024-09-11T01:32:17.165 INFO:tasks.rados.rados.0.smithi071.stdout:476: read oid 36 snap -1 2024-09-11T01:32:17.165 INFO:tasks.rados.rados.0.smithi071.stdout:476: expect (ObjNum 126 snap 29 seq_num 126) 2024-09-11T01:32:17.165 INFO:tasks.rados.rados.0.smithi071.stdout:477: snap_remove snap 40 2024-09-11T01:32:18.086 INFO:tasks.rados.rados.0.smithi071.stdout:477: done (2 left) 2024-09-11T01:32:18.086 INFO:tasks.rados.rados.0.smithi071.stdout:478: write (excl) oid 35 current snap is 42 2024-09-11T01:32:18.087 INFO:tasks.rados.rados.0.smithi071.stdout:478: seq_num 190 ranges {793330=689782,2035461=617731,3101545=91785} 2024-09-11T01:32:18.095 INFO:tasks.rados.rados.0.smithi071.stdout:478: writing smithi07134777-35 from 793330 to 1483112 tid 1 2024-09-11T01:32:18.101 INFO:tasks.rados.rados.0.smithi071.stdout:478: writing smithi07134777-35 from 2035461 to 2653192 tid 2 2024-09-11T01:32:18.105 INFO:tasks.rados.rados.0.smithi071.stdout:478: writing smithi07134777-35 from 3101545 to 3193330 tid 3 2024-09-11T01:32:18.105 INFO:tasks.rados.rados.0.smithi071.stdout:479: copy_from oid 22 from oid 44 current snap is 42 2024-09-11T01:32:18.105 INFO:tasks.rados.rados.0.smithi071.stdout:475: finishing copy_from to smithi07134777-30 2024-09-11T01:32:18.105 INFO:tasks.rados.rados.0.smithi071.stdout:475: got expected ENOENT (src dne) 2024-09-11T01:32:18.105 INFO:tasks.rados.rados.0.smithi071.stdout:475: finishing copy_from racing read to smithi07134777-30 2024-09-11T01:32:18.105 INFO:tasks.rados.rados.0.smithi071.stdout:476: expect (ObjNum 126 snap 29 seq_num 126) 2024-09-11T01:32:18.134 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1254884 2024-09-11T01:32:18.175 INFO:tasks.rados.rados.0.smithi071.stdout:478: finishing write tid 1 to smithi07134777-35 2024-09-11T01:32:18.175 INFO:tasks.rados.rados.0.smithi071.stdout:478: finishing write tid 2 to smithi07134777-35 2024-09-11T01:32:18.176 INFO:tasks.rados.rados.0.smithi071.stdout:478: finishing write tid 3 to smithi07134777-35 2024-09-11T01:32:18.176 INFO:tasks.rados.rados.0.smithi071.stdout:478: finishing write tid 4 to smithi07134777-35 2024-09-11T01:32:18.176 INFO:tasks.rados.rados.0.smithi071.stdout:478: finishing write tid 5 to smithi07134777-35 2024-09-11T01:32:18.176 INFO:tasks.rados.rados.0.smithi071.stdout:478: oid 35 updating version 0 to 151 2024-09-11T01:32:18.176 INFO:tasks.rados.rados.0.smithi071.stdout:478: oid 35 updating version 151 to 152 2024-09-11T01:32:18.176 INFO:tasks.rados.rados.0.smithi071.stdout:478: oid 35 updating version 152 to 153 2024-09-11T01:32:18.176 INFO:tasks.rados.rados.0.smithi071.stdout:478: oid 35 version 153 is already newer than 150 2024-09-11T01:32:18.176 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 153 (ObjNum 190 snap 42 seq_num 190) dirty exists 2024-09-11T01:32:18.176 INFO:tasks.rados.rados.0.smithi071.stdout:478: left oid 35 (ObjNum 190 snap 42 seq_num 190) 2024-09-11T01:32:18.176 INFO:tasks.rados.rados.0.smithi071.stdout:479: finishing copy_from racing read to smithi07134777-22 2024-09-11T01:32:18.176 INFO:tasks.rados.rados.0.smithi071.stdout:479: finishing copy_from to smithi07134777-22 2024-09-11T01:32:18.176 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 206 (ObjNum 171 snap 37 seq_num 171) dirty exists 2024-09-11T01:32:18.177 INFO:tasks.rados.rados.0.smithi071.stdout:475: done (3 left) 2024-09-11T01:32:18.177 INFO:tasks.rados.rados.0.smithi071.stdout:476: done (2 left) 2024-09-11T01:32:18.177 INFO:tasks.rados.rados.0.smithi071.stdout:478: done (1 left) 2024-09-11T01:32:18.177 INFO:tasks.rados.rados.0.smithi071.stdout:479: done (0 left) 2024-09-11T01:32:18.177 INFO:tasks.rados.rados.0.smithi071.stdout:480: read oid 26 snap -1 2024-09-11T01:32:18.177 INFO:tasks.rados.rados.0.smithi071.stdout:480: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:32:18.177 INFO:tasks.rados.rados.0.smithi071.stdout:481: write (excl) oid 29 current snap is 42 2024-09-11T01:32:18.177 INFO:tasks.rados.rados.0.smithi071.stdout:481: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2024-09-11T01:32:18.180 INFO:tasks.rados.rados.0.smithi071.stdout:481: writing smithi07134777-29 from 410137 to 891521 tid 1 2024-09-11T01:32:18.187 INFO:tasks.rados.rados.0.smithi071.stdout:481: writing smithi07134777-29 from 1410296 to 2171685 tid 2 2024-09-11T01:32:18.193 INFO:tasks.rados.rados.0.smithi071.stdout:481: writing smithi07134777-29 from 2728968 to 3210137 tid 3 2024-09-11T01:32:18.193 INFO:tasks.rados.rados.0.smithi071.stdout:482: delete oid 13 current snap is 42 2024-09-11T01:32:18.193 INFO:tasks.rados.rados.0.smithi071.stdout:480: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:32:18.205 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2363987 2024-09-11T01:32:18.249 INFO:tasks.rados.rados.0.smithi071.stdout:481: finishing write tid 1 to smithi07134777-29 2024-09-11T01:32:18.249 INFO:tasks.rados.rados.0.smithi071.stdout:481: finishing write tid 2 to smithi07134777-29 2024-09-11T01:32:18.249 INFO:tasks.rados.rados.0.smithi071.stdout:481: finishing write tid 3 to smithi07134777-29 2024-09-11T01:32:18.249 INFO:tasks.rados.rados.0.smithi071.stdout:481: finishing write tid 4 to smithi07134777-29 2024-09-11T01:32:18.249 INFO:tasks.rados.rados.0.smithi071.stdout:481: finishing write tid 5 to smithi07134777-29 2024-09-11T01:32:18.249 INFO:tasks.rados.rados.0.smithi071.stdout:481: oid 29 updating version 0 to 229 2024-09-11T01:32:18.249 INFO:tasks.rados.rados.0.smithi071.stdout:481: oid 29 version 229 is already newer than 228 2024-09-11T01:32:18.249 INFO:tasks.rados.rados.0.smithi071.stdout:481: oid 29 version 229 is already newer than 227 2024-09-11T01:32:18.250 INFO:tasks.rados.rados.0.smithi071.stdout:481: oid 29 version 229 is already newer than 226 2024-09-11T01:32:18.250 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 229 (ObjNum 191 snap 42 seq_num 191) dirty exists 2024-09-11T01:32:18.250 INFO:tasks.rados.rados.0.smithi071.stdout:481: left oid 29 (ObjNum 191 snap 42 seq_num 191) 2024-09-11T01:32:18.250 INFO:tasks.rados.rados.0.smithi071.stdout:480: done (2 left) 2024-09-11T01:32:18.250 INFO:tasks.rados.rados.0.smithi071.stdout:481: done (1 left) 2024-09-11T01:32:18.250 INFO:tasks.rados.rados.0.smithi071.stdout:482: done (0 left) 2024-09-11T01:32:18.250 INFO:tasks.rados.rados.0.smithi071.stdout:483: write (excl) oid 18 current snap is 42 2024-09-11T01:32:18.250 INFO:tasks.rados.rados.0.smithi071.stdout:483: seq_num 193 ranges {443751=631882,1810907=765058,3067461=176290} 2024-09-11T01:32:18.253 INFO:tasks.rados.rados.0.smithi071.stdout:483: writing smithi07134777-18 from 443751 to 1075633 tid 1 2024-09-11T01:32:18.261 INFO:tasks.rados.rados.0.smithi071.stdout:483: writing smithi07134777-18 from 1810907 to 2575965 tid 2 2024-09-11T01:32:18.265 INFO:tasks.rados.rados.0.smithi071.stdout:483: writing smithi07134777-18 from 3067461 to 3243751 tid 3 2024-09-11T01:32:18.265 INFO:tasks.rados.rados.0.smithi071.stdout:484: snap_create 2024-09-11T01:32:18.265 INFO:tasks.rados.rados.0.smithi071.stdout:483: finishing write tid 1 to smithi07134777-18 2024-09-11T01:32:18.267 INFO:tasks.rados.rados.0.smithi071.stdout:483: finishing write tid 2 to smithi07134777-18 2024-09-11T01:32:18.270 INFO:tasks.rados.rados.0.smithi071.stdout:483: finishing write tid 3 to smithi07134777-18 2024-09-11T01:32:18.273 INFO:tasks.rados.rados.0.smithi071.stdout:483: finishing write tid 4 to smithi07134777-18 2024-09-11T01:32:18.273 INFO:tasks.rados.rados.0.smithi071.stdout:483: finishing write tid 5 to smithi07134777-18 2024-09-11T01:32:18.273 INFO:tasks.rados.rados.0.smithi071.stdout:483: oid 18 updating version 0 to 150 2024-09-11T01:32:18.273 INFO:tasks.rados.rados.0.smithi071.stdout:483: oid 18 version 150 is already newer than 148 2024-09-11T01:32:18.273 INFO:tasks.rados.rados.0.smithi071.stdout:483: oid 18 version 150 is already newer than 149 2024-09-11T01:32:18.273 INFO:tasks.rados.rados.0.smithi071.stdout:483: oid 18 version 150 is already newer than 147 2024-09-11T01:32:18.273 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 150 (ObjNum 193 snap 42 seq_num 193) dirty exists 2024-09-11T01:32:18.273 INFO:tasks.rados.rados.0.smithi071.stdout:483: left oid 18 (ObjNum 193 snap 42 seq_num 193) 2024-09-11T01:32:19.089 INFO:tasks.rados.rados.0.smithi071.stdout:483: done (1 left) 2024-09-11T01:32:19.089 INFO:tasks.rados.rados.0.smithi071.stdout:484: done (0 left) 2024-09-11T01:32:19.089 INFO:tasks.rados.rados.0.smithi071.stdout:485: read oid 20 snap 42 2024-09-11T01:32:19.090 INFO:tasks.rados.rados.0.smithi071.stdout:485: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-11T01:32:19.090 INFO:tasks.rados.rados.0.smithi071.stdout:486: write (excl) oid 28 current snap is 43 2024-09-11T01:32:19.090 INFO:tasks.rados.rados.0.smithi071.stdout:486: seq_num 194 ranges {460558=707131,1869389=508716,3094884=165674} 2024-09-11T01:32:19.095 INFO:tasks.rados.rados.0.smithi071.stdout:486: writing smithi07134777-28 from 460558 to 1167689 tid 1 2024-09-11T01:32:19.102 INFO:tasks.rados.rados.0.smithi071.stdout:486: writing smithi07134777-28 from 1869389 to 2378105 tid 2 2024-09-11T01:32:19.107 INFO:tasks.rados.rados.0.smithi071.stdout:486: writing smithi07134777-28 from 3094884 to 3260558 tid 3 2024-09-11T01:32:19.108 INFO:tasks.rados.rados.0.smithi071.stdout:487: copy_from oid 18 from oid 3 current snap is 43 2024-09-11T01:32:19.108 INFO:tasks.rados.rados.0.smithi071.stdout:485: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-11T01:32:19.129 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:32:19.155 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 764970 2024-09-11T01:32:19.169 INFO:tasks.rados.rados.0.smithi071.stdout:486: finishing write tid 1 to smithi07134777-28 2024-09-11T01:32:19.169 INFO:tasks.rados.rados.0.smithi071.stdout:486: finishing write tid 2 to smithi07134777-28 2024-09-11T01:32:19.169 INFO:tasks.rados.rados.0.smithi071.stdout:486: finishing write tid 3 to smithi07134777-28 2024-09-11T01:32:19.169 INFO:tasks.rados.rados.0.smithi071.stdout:486: finishing write tid 4 to smithi07134777-28 2024-09-11T01:32:19.169 INFO:tasks.rados.rados.0.smithi071.stdout:486: finishing write tid 5 to smithi07134777-28 2024-09-11T01:32:19.169 INFO:tasks.rados.rados.0.smithi071.stdout:486: oid 28 updating version 0 to 209 2024-09-11T01:32:19.169 INFO:tasks.rados.rados.0.smithi071.stdout:486: oid 28 version 209 is already newer than 208 2024-09-11T01:32:19.169 INFO:tasks.rados.rados.0.smithi071.stdout:486: oid 28 updating version 209 to 210 2024-09-11T01:32:19.170 INFO:tasks.rados.rados.0.smithi071.stdout:486: oid 28 updating version 210 to 211 2024-09-11T01:32:19.170 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 211 (ObjNum 194 snap 43 seq_num 194) dirty exists 2024-09-11T01:32:19.170 INFO:tasks.rados.rados.0.smithi071.stdout:486: left oid 28 (ObjNum 194 snap 43 seq_num 194) 2024-09-11T01:32:19.170 INFO:tasks.rados.rados.0.smithi071.stdout:485: done (2 left) 2024-09-11T01:32:19.170 INFO:tasks.rados.rados.0.smithi071.stdout:486: done (1 left) 2024-09-11T01:32:19.170 INFO:tasks.rados.rados.0.smithi071.stdout:488: delete oid 8 current snap is 43 2024-09-11T01:32:19.170 INFO:tasks.rados.rados.0.smithi071.stdout:487: finishing copy_from racing read to smithi07134777-18 2024-09-11T01:32:19.170 INFO:tasks.rados.rados.0.smithi071.stdout:487: finishing copy_from to smithi07134777-18 2024-09-11T01:32:19.170 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 30 (ObjNum 54 snap 3 seq_num 54) dirty exists 2024-09-11T01:32:19.171 INFO:tasks.rados.rados.0.smithi071.stdout:487: done (1 left) 2024-09-11T01:32:19.171 INFO:tasks.rados.rados.0.smithi071.stdout:488: done (0 left) 2024-09-11T01:32:19.171 INFO:tasks.rados.rados.0.smithi071.stdout:489: rollback oid 9 current snap is 43 2024-09-11T01:32:19.171 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 9 to 42 2024-09-11T01:32:19.171 INFO:tasks.rados.rados.0.smithi071.stdout:490: delete oid 18 current snap is 43 2024-09-11T01:32:19.173 INFO:tasks.rados.rados.0.smithi071.stdout:490: done (1 left) 2024-09-11T01:32:19.173 INFO:tasks.rados.rados.0.smithi071.stdout:491: copy_from oid 45 from oid 11 current snap is 43 2024-09-11T01:32:19.173 INFO:tasks.rados.rados.0.smithi071.stdout:492: write (excl) oid 48 current snap is 43 2024-09-11T01:32:19.173 INFO:tasks.rados.rados.0.smithi071.stdout:492: seq_num 197 ranges {510979=532878,1528482=772396,3060800=250179} 2024-09-11T01:32:19.179 INFO:tasks.rados.rados.0.smithi071.stdout:492: writing smithi07134777-48 from 510979 to 1043857 tid 1 2024-09-11T01:32:19.186 INFO:tasks.rados.rados.0.smithi071.stdout:492: writing smithi07134777-48 from 1528482 to 2300878 tid 2 2024-09-11T01:32:19.192 INFO:tasks.rados.rados.0.smithi071.stdout:492: writing smithi07134777-48 from 3060800 to 3310979 tid 3 2024-09-11T01:32:19.192 INFO:tasks.rados.rados.0.smithi071.stdout:493: read oid 6 snap 42 2024-09-11T01:32:19.192 INFO:tasks.rados.rados.0.smithi071.stdout:493: expect (ObjNum 152 snap 33 seq_num 152) 2024-09-11T01:32:19.192 INFO:tasks.rados.rados.0.smithi071.stdout:494: snap_create 2024-09-11T01:32:19.192 INFO:tasks.rados.rados.0.smithi071.stdout:489: finishing rollback tid 0 to smithi07134777-9 2024-09-11T01:32:19.192 INFO:tasks.rados.rados.0.smithi071.stdout:489: finishing rollback tid 1 to smithi07134777-9 2024-09-11T01:32:19.192 INFO:tasks.rados.rados.0.smithi071.stdout:489: finishing rollback tid 2 to smithi07134777-9 2024-09-11T01:32:19.192 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 46 (ObjNum 172 snap 37 seq_num 172) dirty exists 2024-09-11T01:32:19.193 INFO:tasks.rados.rados.0.smithi071.stdout:491: finishing copy_from racing read to smithi07134777-45 2024-09-11T01:32:19.193 INFO:tasks.rados.rados.0.smithi071.stdout:491: finishing copy_from to smithi07134777-45 2024-09-11T01:32:19.193 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 98 (ObjNum 119 snap 28 seq_num 119) dirty exists 2024-09-11T01:32:19.193 INFO:tasks.rados.rados.0.smithi071.stdout:492: finishing write tid 1 to smithi07134777-48 2024-09-11T01:32:19.193 INFO:tasks.rados.rados.0.smithi071.stdout:492: finishing write tid 2 to smithi07134777-48 2024-09-11T01:32:19.193 INFO:tasks.rados.rados.0.smithi071.stdout:492: finishing write tid 3 to smithi07134777-48 2024-09-11T01:32:19.194 INFO:tasks.rados.rados.0.smithi071.stdout:492: finishing write tid 4 to smithi07134777-48 2024-09-11T01:32:19.194 INFO:tasks.rados.rados.0.smithi071.stdout:492: finishing write tid 5 to smithi07134777-48 2024-09-11T01:32:19.194 INFO:tasks.rados.rados.0.smithi071.stdout:492: oid 48 updating version 0 to 233 2024-09-11T01:32:19.194 INFO:tasks.rados.rados.0.smithi071.stdout:492: oid 48 version 233 is already newer than 231 2024-09-11T01:32:19.194 INFO:tasks.rados.rados.0.smithi071.stdout:492: oid 48 version 233 is already newer than 232 2024-09-11T01:32:19.194 INFO:tasks.rados.rados.0.smithi071.stdout:492: oid 48 updating version 233 to 234 2024-09-11T01:32:19.194 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 234 (ObjNum 197 snap 43 seq_num 197) dirty exists 2024-09-11T01:32:19.195 INFO:tasks.rados.rados.0.smithi071.stdout:492: left oid 48 (ObjNum 197 snap 43 seq_num 197) 2024-09-11T01:32:19.201 INFO:tasks.rados.rados.0.smithi071.stdout:493: expect (ObjNum 152 snap 33 seq_num 152) 2024-09-11T01:32:19.201 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 436452 2024-09-11T01:32:19.247 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:32:19.298 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 407888 2024-09-11T01:32:20.093 INFO:tasks.rados.rados.0.smithi071.stdout:489: done (4 left) 2024-09-11T01:32:20.093 INFO:tasks.rados.rados.0.smithi071.stdout:491: done (3 left) 2024-09-11T01:32:20.093 INFO:tasks.rados.rados.0.smithi071.stdout:492: done (2 left) 2024-09-11T01:32:20.093 INFO:tasks.rados.rados.0.smithi071.stdout:493: done (1 left) 2024-09-11T01:32:20.093 INFO:tasks.rados.rados.0.smithi071.stdout:494: done (0 left) 2024-09-11T01:32:20.093 INFO:tasks.rados.rados.0.smithi071.stdout:495: read oid 18 snap -1 2024-09-11T01:32:20.093 INFO:tasks.rados.rados.0.smithi071.stdout:495: expect deleted 2024-09-11T01:32:20.093 INFO:tasks.rados.rados.0.smithi071.stdout:496: copy_from oid 17 from oid 25 current snap is 44 2024-09-11T01:32:20.093 INFO:tasks.rados.rados.0.smithi071.stdout:497: snap_create 2024-09-11T01:32:20.234 INFO:tasks.rados.rados.0.smithi071.stdout:496: finishing copy_from racing read to smithi07134777-17 2024-09-11T01:32:20.237 INFO:tasks.rados.rados.0.smithi071.stdout:496: finishing copy_from to smithi07134777-17 2024-09-11T01:32:20.237 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 38 (ObjNum 173 snap 37 seq_num 173) dirty exists 2024-09-11T01:32:21.097 INFO:tasks.rados.rados.0.smithi071.stdout:495: done (2 left) 2024-09-11T01:32:21.098 INFO:tasks.rados.rados.0.smithi071.stdout:496: done (1 left) 2024-09-11T01:32:21.098 INFO:tasks.rados.rados.0.smithi071.stdout:497: done (0 left) 2024-09-11T01:32:21.098 INFO:tasks.rados.rados.0.smithi071.stdout:498: delete oid 5 current snap is 45 2024-09-11T01:32:21.101 INFO:tasks.rados.rados.0.smithi071.stdout:498: done (0 left) 2024-09-11T01:32:21.101 INFO:tasks.rados.rados.0.smithi071.stdout:499: copy_from oid 16 from oid 21 current snap is 45 2024-09-11T01:32:21.101 INFO:tasks.rados.rados.0.smithi071.stdout:500: snap_create 2024-09-11T01:32:21.104 INFO:tasks.rados.rados.0.smithi071.stdout:499: finishing copy_from to smithi07134777-16 2024-09-11T01:32:21.104 INFO:tasks.rados.rados.0.smithi071.stdout:499: got expected ENOENT (src dne) 2024-09-11T01:32:21.104 INFO:tasks.rados.rados.0.smithi071.stdout:499: finishing copy_from racing read to smithi07134777-16 2024-09-11T01:32:22.077 INFO:tasks.rados.rados.0.smithi071.stdout:499: done (1 left) 2024-09-11T01:32:22.077 INFO:tasks.rados.rados.0.smithi071.stdout:500: done (0 left) 2024-09-11T01:32:22.077 INFO:tasks.rados.rados.0.smithi071.stdout:501: copy_from oid 34 from oid 20 current snap is 46 2024-09-11T01:32:22.077 INFO:tasks.rados.rados.0.smithi071.stdout:502: rollback oid 7 current snap is 46 2024-09-11T01:32:22.077 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 7 to 43 2024-09-11T01:32:22.077 INFO:tasks.rados.rados.0.smithi071.stdout:503: delete oid 9 current snap is 46 2024-09-11T01:32:22.080 INFO:tasks.rados.rados.0.smithi071.stdout:503: done (2 left) 2024-09-11T01:32:22.080 INFO:tasks.rados.rados.0.smithi071.stdout:504: write oid 28 current snap is 46 2024-09-11T01:32:22.080 INFO:tasks.rados.rados.0.smithi071.stdout:504: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2024-09-11T01:32:22.085 INFO:tasks.rados.rados.0.smithi071.stdout:504: writing smithi07134777-28 from 561400 to 1036378 tid 1 2024-09-11T01:32:22.093 INFO:tasks.rados.rados.0.smithi071.stdout:504: writing smithi07134777-28 from 1820281 to 2340004 tid 2 2024-09-11T01:32:22.099 INFO:tasks.rados.rados.0.smithi071.stdout:504: writing smithi07134777-28 from 2859422 to 3317001 tid 3 2024-09-11T01:32:22.099 INFO:tasks.rados.rados.0.smithi071.stdout:505: write oid 5 current snap is 46 2024-09-11T01:32:22.099 INFO:tasks.rados.rados.0.smithi071.stdout:502: finishing rollback tid 0 to smithi07134777-7 2024-09-11T01:32:22.099 INFO:tasks.rados.rados.0.smithi071.stdout:502: finishing rollback tid 1 to smithi07134777-7 2024-09-11T01:32:22.100 INFO:tasks.rados.rados.0.smithi071.stdout:502: finishing rollback tid 2 to smithi07134777-7 2024-09-11T01:32:22.100 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 238 (ObjNum 154 snap 34 seq_num 154) dirty exists 2024-09-11T01:32:22.100 INFO:tasks.rados.rados.0.smithi071.stdout:501: finishing copy_from racing read to smithi07134777-34 2024-09-11T01:32:22.100 INFO:tasks.rados.rados.0.smithi071.stdout:501: finishing copy_from to smithi07134777-34 2024-09-11T01:32:22.100 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-11T01:32:22.100 INFO:tasks.rados.rados.0.smithi071.stdout:504: finishing write tid 1 to smithi07134777-28 2024-09-11T01:32:22.100 INFO:tasks.rados.rados.0.smithi071.stdout:504: finishing write tid 2 to smithi07134777-28 2024-09-11T01:32:22.100 INFO:tasks.rados.rados.0.smithi071.stdout:505: seq_num 201 ranges {578207=550227,1762410=779734,3286845=91362} 2024-09-11T01:32:22.105 INFO:tasks.rados.rados.0.smithi071.stdout:505: writing smithi07134777-5 from 578207 to 1128434 tid 1 2024-09-11T01:32:22.113 INFO:tasks.rados.rados.0.smithi071.stdout:505: writing smithi07134777-5 from 1762410 to 2542144 tid 2 2024-09-11T01:32:22.118 INFO:tasks.rados.rados.0.smithi071.stdout:505: writing smithi07134777-5 from 3286845 to 3378207 tid 3 2024-09-11T01:32:22.118 INFO:tasks.rados.rados.0.smithi071.stdout:501: done (3 left) 2024-09-11T01:32:22.118 INFO:tasks.rados.rados.0.smithi071.stdout:502: done (2 left) 2024-09-11T01:32:22.118 INFO:tasks.rados.rados.0.smithi071.stdout:506: delete oid 19 current snap is 46 2024-09-11T01:32:22.118 INFO:tasks.rados.rados.0.smithi071.stdout:504: finishing write tid 3 to smithi07134777-28 2024-09-11T01:32:22.118 INFO:tasks.rados.rados.0.smithi071.stdout:504: finishing write tid 4 to smithi07134777-28 2024-09-11T01:32:22.118 INFO:tasks.rados.rados.0.smithi071.stdout:504: finishing write tid 5 to smithi07134777-28 2024-09-11T01:32:22.118 INFO:tasks.rados.rados.0.smithi071.stdout:504: oid 28 updating version 0 to 215 2024-09-11T01:32:22.118 INFO:tasks.rados.rados.0.smithi071.stdout:504: oid 28 version 215 is already newer than 214 2024-09-11T01:32:22.119 INFO:tasks.rados.rados.0.smithi071.stdout:504: oid 28 version 215 is already newer than 213 2024-09-11T01:32:22.119 INFO:tasks.rados.rados.0.smithi071.stdout:504: oid 28 version 215 is already newer than 212 2024-09-11T01:32:22.119 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 215 (ObjNum 200 snap 46 seq_num 200) dirty exists 2024-09-11T01:32:22.119 INFO:tasks.rados.rados.0.smithi071.stdout:504: left oid 28 (ObjNum 200 snap 46 seq_num 200) 2024-09-11T01:32:22.119 INFO:tasks.rados.rados.0.smithi071.stdout:505: finishing write tid 1 to smithi07134777-5 2024-09-11T01:32:22.119 INFO:tasks.rados.rados.0.smithi071.stdout:505: finishing write tid 2 to smithi07134777-5 2024-09-11T01:32:22.120 INFO:tasks.rados.rados.0.smithi071.stdout:504: done (2 left) 2024-09-11T01:32:22.120 INFO:tasks.rados.rados.0.smithi071.stdout:506: done (1 left) 2024-09-11T01:32:22.120 INFO:tasks.rados.rados.0.smithi071.stdout:507: rollback oid 32 current snap is 46 2024-09-11T01:32:22.120 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 32 to 43 2024-09-11T01:32:22.120 INFO:tasks.rados.rados.0.smithi071.stdout:508: snap_create 2024-09-11T01:32:22.120 INFO:tasks.rados.rados.0.smithi071.stdout:505: finishing write tid 3 to smithi07134777-5 2024-09-11T01:32:22.121 INFO:tasks.rados.rados.0.smithi071.stdout:505: finishing write tid 4 to smithi07134777-5 2024-09-11T01:32:22.121 INFO:tasks.rados.rados.0.smithi071.stdout:505: finishing write tid 5 to smithi07134777-5 2024-09-11T01:32:22.121 INFO:tasks.rados.rados.0.smithi071.stdout:505: oid 5 updating version 0 to 32 2024-09-11T01:32:22.121 INFO:tasks.rados.rados.0.smithi071.stdout:505: oid 5 version 32 is already newer than 30 2024-09-11T01:32:22.121 INFO:tasks.rados.rados.0.smithi071.stdout:505: oid 5 updating version 32 to 33 2024-09-11T01:32:22.121 INFO:tasks.rados.rados.0.smithi071.stdout:505: oid 5 version 33 is already newer than 31 2024-09-11T01:32:22.121 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 33 (ObjNum 201 snap 46 seq_num 201) dirty exists 2024-09-11T01:32:22.121 INFO:tasks.rados.rados.0.smithi071.stdout:505: left oid 5 (ObjNum 201 snap 46 seq_num 201) 2024-09-11T01:32:22.121 INFO:tasks.rados.rados.0.smithi071.stdout:507: finishing rollback tid 1 to smithi07134777-32 2024-09-11T01:32:22.121 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 0 (ObjNum 2065669344 snap 32661 seq_num 304) dirty dne 2024-09-11T01:32:23.103 INFO:tasks.rados.rados.0.smithi071.stdout:505: done (2 left) 2024-09-11T01:32:23.103 INFO:tasks.rados.rados.0.smithi071.stdout:507: done (1 left) 2024-09-11T01:32:23.103 INFO:tasks.rados.rados.0.smithi071.stdout:508: done (0 left) 2024-09-11T01:32:23.103 INFO:tasks.rados.rados.0.smithi071.stdout:509: copy_from oid 47 from oid 42 current snap is 47 2024-09-11T01:32:23.104 INFO:tasks.rados.rados.0.smithi071.stdout:510: delete oid 44 current snap is 47 2024-09-11T01:32:23.106 INFO:tasks.rados.rados.0.smithi071.stdout:509: finishing copy_from to smithi07134777-47 2024-09-11T01:32:23.106 INFO:tasks.rados.rados.0.smithi071.stdout:509: got expected ENOENT (src dne) 2024-09-11T01:32:23.106 INFO:tasks.rados.rados.0.smithi071.stdout:509: finishing copy_from racing read to smithi07134777-47 2024-09-11T01:32:23.108 INFO:tasks.rados.rados.0.smithi071.stdout:509: done (1 left) 2024-09-11T01:32:23.108 INFO:tasks.rados.rados.0.smithi071.stdout:510: done (0 left) 2024-09-11T01:32:23.108 INFO:tasks.rados.rados.0.smithi071.stdout:511: write (excl) oid 35 current snap is 47 2024-09-11T01:32:23.109 INFO:tasks.rados.rados.0.smithi071.stdout:511: seq_num 204 ranges {628628=775974,1937856=527061,3252761=175867} 2024-09-11T01:32:23.115 INFO:tasks.rados.rados.0.smithi071.stdout:511: writing smithi07134777-35 from 628628 to 1404602 tid 1 2024-09-11T01:32:23.121 INFO:tasks.rados.rados.0.smithi071.stdout:511: writing smithi07134777-35 from 1937856 to 2464917 tid 2 2024-09-11T01:32:23.126 INFO:tasks.rados.rados.0.smithi071.stdout:511: writing smithi07134777-35 from 3252761 to 3428628 tid 3 2024-09-11T01:32:23.126 INFO:tasks.rados.rados.0.smithi071.stdout:512: write oid 13 current snap is 47 2024-09-11T01:32:23.127 INFO:tasks.rados.rados.0.smithi071.stdout:511: finishing write tid 1 to smithi07134777-35 2024-09-11T01:32:23.127 INFO:tasks.rados.rados.0.smithi071.stdout:512: seq_num 205 ranges {645435=451223,1879985=787072,3280184=165251} 2024-09-11T01:32:23.132 INFO:tasks.rados.rados.0.smithi071.stdout:512: writing smithi07134777-13 from 645435 to 1096658 tid 1 2024-09-11T01:32:23.141 INFO:tasks.rados.rados.0.smithi071.stdout:512: writing smithi07134777-13 from 1879985 to 2667057 tid 2 2024-09-11T01:32:23.146 INFO:tasks.rados.rados.0.smithi071.stdout:512: writing smithi07134777-13 from 3280184 to 3445435 tid 3 2024-09-11T01:32:23.146 INFO:tasks.rados.rados.0.smithi071.stdout:513: snap_create 2024-09-11T01:32:23.146 INFO:tasks.rados.rados.0.smithi071.stdout:511: finishing write tid 2 to smithi07134777-35 2024-09-11T01:32:23.146 INFO:tasks.rados.rados.0.smithi071.stdout:511: finishing write tid 3 to smithi07134777-35 2024-09-11T01:32:23.146 INFO:tasks.rados.rados.0.smithi071.stdout:511: finishing write tid 4 to smithi07134777-35 2024-09-11T01:32:23.146 INFO:tasks.rados.rados.0.smithi071.stdout:511: finishing write tid 5 to smithi07134777-35 2024-09-11T01:32:23.146 INFO:tasks.rados.rados.0.smithi071.stdout:511: oid 35 updating version 0 to 159 2024-09-11T01:32:23.146 INFO:tasks.rados.rados.0.smithi071.stdout:511: oid 35 version 159 is already newer than 157 2024-09-11T01:32:23.146 INFO:tasks.rados.rados.0.smithi071.stdout:511: oid 35 version 159 is already newer than 156 2024-09-11T01:32:23.146 INFO:tasks.rados.rados.0.smithi071.stdout:511: oid 35 version 159 is already newer than 158 2024-09-11T01:32:23.146 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 159 (ObjNum 204 snap 47 seq_num 204) dirty exists 2024-09-11T01:32:23.147 INFO:tasks.rados.rados.0.smithi071.stdout:511: left oid 35 (ObjNum 204 snap 47 seq_num 204) 2024-09-11T01:32:23.147 INFO:tasks.rados.rados.0.smithi071.stdout:512: finishing write tid 1 to smithi07134777-13 2024-09-11T01:32:23.147 INFO:tasks.rados.rados.0.smithi071.stdout:512: finishing write tid 2 to smithi07134777-13 2024-09-11T01:32:23.148 INFO:tasks.rados.rados.0.smithi071.stdout:512: finishing write tid 3 to smithi07134777-13 2024-09-11T01:32:23.148 INFO:tasks.rados.rados.0.smithi071.stdout:512: finishing write tid 4 to smithi07134777-13 2024-09-11T01:32:23.149 INFO:tasks.rados.rados.0.smithi071.stdout:512: finishing write tid 5 to smithi07134777-13 2024-09-11T01:32:23.149 INFO:tasks.rados.rados.0.smithi071.stdout:512: oid 13 updating version 0 to 126 2024-09-11T01:32:23.149 INFO:tasks.rados.rados.0.smithi071.stdout:512: oid 13 version 126 is already newer than 123 2024-09-11T01:32:23.149 INFO:tasks.rados.rados.0.smithi071.stdout:512: oid 13 version 126 is already newer than 125 2024-09-11T01:32:23.149 INFO:tasks.rados.rados.0.smithi071.stdout:512: oid 13 version 126 is already newer than 124 2024-09-11T01:32:23.149 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 126 (ObjNum 205 snap 47 seq_num 205) dirty exists 2024-09-11T01:32:23.149 INFO:tasks.rados.rados.0.smithi071.stdout:512: left oid 13 (ObjNum 205 snap 47 seq_num 205) 2024-09-11T01:32:24.107 INFO:tasks.rados.rados.0.smithi071.stdout:511: done (2 left) 2024-09-11T01:32:24.107 INFO:tasks.rados.rados.0.smithi071.stdout:512: done (1 left) 2024-09-11T01:32:24.108 INFO:tasks.rados.rados.0.smithi071.stdout:513: done (0 left) 2024-09-11T01:32:24.108 INFO:tasks.rados.rados.0.smithi071.stdout:514: write oid 49 current snap is 48 2024-09-11T01:32:24.108 INFO:tasks.rados.rados.0.smithi071.stdout:514: seq_num 206 ranges {662242=642825,2054820=530730,3307607=154635} 2024-09-11T01:32:24.114 INFO:tasks.rados.rados.0.smithi071.stdout:514: writing smithi07134777-49 from 662242 to 1305067 tid 1 2024-09-11T01:32:24.121 INFO:tasks.rados.rados.0.smithi071.stdout:514: writing smithi07134777-49 from 2054820 to 2585550 tid 2 2024-09-11T01:32:24.126 INFO:tasks.rados.rados.0.smithi071.stdout:514: writing smithi07134777-49 from 3307607 to 3462242 tid 3 2024-09-11T01:32:24.126 INFO:tasks.rados.rados.0.smithi071.stdout:515: snap_remove snap 42 2024-09-11T01:32:25.118 INFO:tasks.rados.rados.0.smithi071.stdout:515: done (1 left) 2024-09-11T01:32:25.119 INFO:tasks.rados.rados.0.smithi071.stdout:516: copy_from oid 3 from oid 34 current snap is 48 2024-09-11T01:32:25.119 INFO:tasks.rados.rados.0.smithi071.stdout:514: finishing write tid 1 to smithi07134777-49 2024-09-11T01:32:25.119 INFO:tasks.rados.rados.0.smithi071.stdout:514: finishing write tid 2 to smithi07134777-49 2024-09-11T01:32:25.119 INFO:tasks.rados.rados.0.smithi071.stdout:514: finishing write tid 3 to smithi07134777-49 2024-09-11T01:32:25.119 INFO:tasks.rados.rados.0.smithi071.stdout:514: finishing write tid 4 to smithi07134777-49 2024-09-11T01:32:25.119 INFO:tasks.rados.rados.0.smithi071.stdout:514: finishing write tid 5 to smithi07134777-49 2024-09-11T01:32:25.119 INFO:tasks.rados.rados.0.smithi071.stdout:514: oid 49 updating version 0 to 20 2024-09-11T01:32:25.119 INFO:tasks.rados.rados.0.smithi071.stdout:514: oid 49 updating version 20 to 21 2024-09-11T01:32:25.119 INFO:tasks.rados.rados.0.smithi071.stdout:514: oid 49 updating version 21 to 22 2024-09-11T01:32:25.119 INFO:tasks.rados.rados.0.smithi071.stdout:514: oid 49 version 22 is already newer than 19 2024-09-11T01:32:25.119 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 22 (ObjNum 206 snap 48 seq_num 206) dirty exists 2024-09-11T01:32:25.119 INFO:tasks.rados.rados.0.smithi071.stdout:514: left oid 49 (ObjNum 206 snap 48 seq_num 206) 2024-09-11T01:32:25.119 INFO:tasks.rados.rados.0.smithi071.stdout:514: done (1 left) 2024-09-11T01:32:25.120 INFO:tasks.rados.rados.0.smithi071.stdout:517: delete oid 49 current snap is 48 2024-09-11T01:32:25.124 INFO:tasks.rados.rados.0.smithi071.stdout:517: done (1 left) 2024-09-11T01:32:25.124 INFO:tasks.rados.rados.0.smithi071.stdout:518: snap_create 2024-09-11T01:32:25.132 INFO:tasks.rados.rados.0.smithi071.stdout:516: finishing copy_from racing read to smithi07134777-3 2024-09-11T01:32:25.134 INFO:tasks.rados.rados.0.smithi071.stdout:516: finishing copy_from to smithi07134777-3 2024-09-11T01:32:25.134 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-11T01:32:26.123 INFO:tasks.rados.rados.0.smithi071.stdout:516: done (1 left) 2024-09-11T01:32:26.123 INFO:tasks.rados.rados.0.smithi071.stdout:518: done (0 left) 2024-09-11T01:32:26.123 INFO:tasks.rados.rados.0.smithi071.stdout:519: read oid 46 snap 48 2024-09-11T01:32:26.124 INFO:tasks.rados.rados.0.smithi071.stdout:519: expect (ObjNum 161 snap 36 seq_num 161) 2024-09-11T01:32:26.124 INFO:tasks.rados.rados.0.smithi071.stdout:520: snap_remove snap 47 2024-09-11T01:32:27.078 INFO:tasks.rados.rados.0.smithi071.stdout:520: done (1 left) 2024-09-11T01:32:27.078 INFO:tasks.rados.rados.0.smithi071.stdout:521: copy_from oid 18 from oid 50 current snap is 49 2024-09-11T01:32:27.078 INFO:tasks.rados.rados.0.smithi071.stdout:519: expect (ObjNum 161 snap 36 seq_num 161) 2024-09-11T01:32:27.078 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 500793 2024-09-11T01:32:27.178 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 808537 2024-09-11T01:32:27.219 INFO:tasks.rados.rados.0.smithi071.stdout:521: finishing copy_from racing read to smithi07134777-18 2024-09-11T01:32:27.219 INFO:tasks.rados.rados.0.smithi071.stdout:521: finishing copy_from to smithi07134777-18 2024-09-11T01:32:27.219 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 206 (ObjNum 171 snap 37 seq_num 171) dirty exists 2024-09-11T01:32:27.219 INFO:tasks.rados.rados.0.smithi071.stdout:519: done (1 left) 2024-09-11T01:32:27.219 INFO:tasks.rados.rados.0.smithi071.stdout:521: done (0 left) 2024-09-11T01:32:27.219 INFO:tasks.rados.rados.0.smithi071.stdout:522: copy_from oid 4 from oid 32 current snap is 49 2024-09-11T01:32:27.219 INFO:tasks.rados.rados.0.smithi071.stdout:523: snap_remove snap 44 2024-09-11T01:32:28.133 INFO:tasks.rados.rados.0.smithi071.stdout:523: done (1 left) 2024-09-11T01:32:28.133 INFO:tasks.rados.rados.0.smithi071.stdout:524: write oid 44 current snap is 49 2024-09-11T01:32:28.133 INFO:tasks.rados.rados.0.smithi071.stdout:524: seq_num 208 ranges {695856=793323,2171784=534399,3362453=133403} 2024-09-11T01:32:28.141 INFO:tasks.rados.rados.0.smithi071.stdout:524: writing smithi07134777-44 from 695856 to 1489179 tid 1 2024-09-11T01:32:28.148 INFO:tasks.rados.rados.0.smithi071.stdout:524: writing smithi07134777-44 from 2171784 to 2706183 tid 2 2024-09-11T01:32:28.154 INFO:tasks.rados.rados.0.smithi071.stdout:524: writing smithi07134777-44 from 3362453 to 3495856 tid 3 2024-09-11T01:32:28.154 INFO:tasks.rados.rados.0.smithi071.stdout:525: read oid 27 snap -1 2024-09-11T01:32:28.154 INFO:tasks.rados.rados.0.smithi071.stdout:525: expect (ObjNum 54 snap 3 seq_num 54) 2024-09-11T01:32:28.154 INFO:tasks.rados.rados.0.smithi071.stdout:526: read oid 40 snap -1 2024-09-11T01:32:28.154 INFO:tasks.rados.rados.0.smithi071.stdout:526: expect (ObjNum 143 snap 32 seq_num 143) 2024-09-11T01:32:28.154 INFO:tasks.rados.rados.0.smithi071.stdout:527: rollback oid 50 current snap is 49 2024-09-11T01:32:28.154 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 50 to 48 2024-09-11T01:32:28.154 INFO:tasks.rados.rados.0.smithi071.stdout:522: finishing copy_from to smithi07134777-4 2024-09-11T01:32:28.154 INFO:tasks.rados.rados.0.smithi071.stdout:522: got expected ENOENT (src dne) 2024-09-11T01:32:28.154 INFO:tasks.rados.rados.0.smithi071.stdout:522: finishing copy_from racing read to smithi07134777-4 2024-09-11T01:32:28.154 INFO:tasks.rados.rados.0.smithi071.stdout:524: finishing write tid 1 to smithi07134777-44 2024-09-11T01:32:28.154 INFO:tasks.rados.rados.0.smithi071.stdout:524: finishing write tid 2 to smithi07134777-44 2024-09-11T01:32:28.154 INFO:tasks.rados.rados.0.smithi071.stdout:522: done (4 left) 2024-09-11T01:32:28.155 INFO:tasks.rados.rados.0.smithi071.stdout:528: snap_remove snap 43 2024-09-11T01:32:29.494 INFO:tasks.rados.rados.0.smithi071.stdout:528: done (4 left) 2024-09-11T01:32:29.494 INFO:tasks.rados.rados.0.smithi071.stdout:529: copy_from oid 48 from oid 26 current snap is 49 2024-09-11T01:32:29.494 INFO:tasks.rados.rados.0.smithi071.stdout:524: finishing write tid 3 to smithi07134777-44 2024-09-11T01:32:29.494 INFO:tasks.rados.rados.0.smithi071.stdout:524: finishing write tid 4 to smithi07134777-44 2024-09-11T01:32:29.494 INFO:tasks.rados.rados.0.smithi071.stdout:524: finishing write tid 5 to smithi07134777-44 2024-09-11T01:32:29.495 INFO:tasks.rados.rados.0.smithi071.stdout:524: oid 44 updating version 0 to 211 2024-09-11T01:32:29.495 INFO:tasks.rados.rados.0.smithi071.stdout:524: oid 44 version 211 is already newer than 208 2024-09-11T01:32:29.495 INFO:tasks.rados.rados.0.smithi071.stdout:524: oid 44 version 211 is already newer than 210 2024-09-11T01:32:29.495 INFO:tasks.rados.rados.0.smithi071.stdout:524: oid 44 version 211 is already newer than 209 2024-09-11T01:32:29.495 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 211 (ObjNum 208 snap 49 seq_num 208) dirty exists 2024-09-11T01:32:29.495 INFO:tasks.rados.rados.0.smithi071.stdout:524: left oid 44 (ObjNum 208 snap 49 seq_num 208) 2024-09-11T01:32:29.495 INFO:tasks.rados.rados.0.smithi071.stdout:526: expect (ObjNum 143 snap 32 seq_num 143) 2024-09-11T01:32:29.495 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 170550 2024-09-11T01:32:29.538 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 898639 2024-09-11T01:32:29.596 INFO:tasks.rados.rados.0.smithi071.stdout:525: expect (ObjNum 54 snap 3 seq_num 54) 2024-09-11T01:32:29.679 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2071015 2024-09-11T01:32:29.695 INFO:tasks.rados.rados.0.smithi071.stdout:527: finishing rollback tid 0 to smithi07134777-50 2024-09-11T01:32:29.695 INFO:tasks.rados.rados.0.smithi071.stdout:527: finishing rollback tid 1 to smithi07134777-50 2024-09-11T01:32:29.695 INFO:tasks.rados.rados.0.smithi071.stdout:527: finishing rollback tid 2 to smithi07134777-50 2024-09-11T01:32:29.695 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 246 (ObjNum 171 snap 37 seq_num 171) dirty exists 2024-09-11T01:32:29.695 INFO:tasks.rados.rados.0.smithi071.stdout:529: finishing copy_from racing read to smithi07134777-48 2024-09-11T01:32:29.695 INFO:tasks.rados.rados.0.smithi071.stdout:529: finishing copy_from to smithi07134777-48 2024-09-11T01:32:29.695 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 155 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-11T01:32:29.695 INFO:tasks.rados.rados.0.smithi071.stdout:524: done (4 left) 2024-09-11T01:32:29.695 INFO:tasks.rados.rados.0.smithi071.stdout:525: done (3 left) 2024-09-11T01:32:29.696 INFO:tasks.rados.rados.0.smithi071.stdout:526: done (2 left) 2024-09-11T01:32:29.696 INFO:tasks.rados.rados.0.smithi071.stdout:527: done (1 left) 2024-09-11T01:32:29.696 INFO:tasks.rados.rados.0.smithi071.stdout:529: done (0 left) 2024-09-11T01:32:29.696 INFO:tasks.rados.rados.0.smithi071.stdout:530: snap_remove snap 48 2024-09-11T01:32:30.498 INFO:tasks.rados.rados.0.smithi071.stdout:530: done (0 left) 2024-09-11T01:32:30.498 INFO:tasks.rados.rados.0.smithi071.stdout:531: read oid 2 snap -1 2024-09-11T01:32:30.498 INFO:tasks.rados.rados.0.smithi071.stdout:531: expect (ObjNum 153 snap 34 seq_num 153) 2024-09-11T01:32:30.498 INFO:tasks.rados.rados.0.smithi071.stdout:532: snap_remove snap 45 2024-09-11T01:32:31.507 INFO:tasks.rados.rados.0.smithi071.stdout:532: done (1 left) 2024-09-11T01:32:31.507 INFO:tasks.rados.rados.0.smithi071.stdout:533: write (excl) oid 44 current snap is 49 2024-09-11T01:32:31.507 INFO:tasks.rados.rados.0.smithi071.stdout:533: seq_num 209 ranges {712663=468572,1713913=678057,2873523=591733} 2024-09-11T01:32:31.513 INFO:tasks.rados.rados.0.smithi071.stdout:533: writing smithi07134777-44 from 712663 to 1181235 tid 1 2024-09-11T01:32:31.520 INFO:tasks.rados.rados.0.smithi071.stdout:533: writing smithi07134777-44 from 1713913 to 2391970 tid 2 2024-09-11T01:32:31.526 INFO:tasks.rados.rados.0.smithi071.stdout:533: writing smithi07134777-44 from 2873523 to 3465256 tid 3 2024-09-11T01:32:31.527 INFO:tasks.rados.rados.0.smithi071.stdout:534: delete oid 34 current snap is 49 2024-09-11T01:32:31.527 INFO:tasks.rados.rados.0.smithi071.stdout:531: expect (ObjNum 153 snap 34 seq_num 153) 2024-09-11T01:32:31.578 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1770212 2024-09-11T01:32:31.601 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1553541 2024-09-11T01:32:31.622 INFO:tasks.rados.rados.0.smithi071.stdout:533: finishing write tid 1 to smithi07134777-44 2024-09-11T01:32:31.623 INFO:tasks.rados.rados.0.smithi071.stdout:533: finishing write tid 2 to smithi07134777-44 2024-09-11T01:32:31.623 INFO:tasks.rados.rados.0.smithi071.stdout:533: finishing write tid 3 to smithi07134777-44 2024-09-11T01:32:31.623 INFO:tasks.rados.rados.0.smithi071.stdout:533: finishing write tid 4 to smithi07134777-44 2024-09-11T01:32:31.623 INFO:tasks.rados.rados.0.smithi071.stdout:533: finishing write tid 5 to smithi07134777-44 2024-09-11T01:32:31.623 INFO:tasks.rados.rados.0.smithi071.stdout:533: oid 44 updating version 0 to 212 2024-09-11T01:32:31.623 INFO:tasks.rados.rados.0.smithi071.stdout:533: oid 44 updating version 212 to 213 2024-09-11T01:32:31.623 INFO:tasks.rados.rados.0.smithi071.stdout:533: oid 44 updating version 213 to 214 2024-09-11T01:32:31.623 INFO:tasks.rados.rados.0.smithi071.stdout:533: oid 44 updating version 214 to 215 2024-09-11T01:32:31.623 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 215 (ObjNum 209 snap 49 seq_num 209) dirty exists 2024-09-11T01:32:31.623 INFO:tasks.rados.rados.0.smithi071.stdout:533: left oid 44 (ObjNum 209 snap 49 seq_num 209) 2024-09-11T01:32:31.623 INFO:tasks.rados.rados.0.smithi071.stdout:531: done (2 left) 2024-09-11T01:32:31.624 INFO:tasks.rados.rados.0.smithi071.stdout:533: done (1 left) 2024-09-11T01:32:31.624 INFO:tasks.rados.rados.0.smithi071.stdout:534: done (0 left) 2024-09-11T01:32:31.624 INFO:tasks.rados.rados.0.smithi071.stdout:535: write oid 46 current snap is 49 2024-09-11T01:32:31.624 INFO:tasks.rados.rados.0.smithi071.stdout:535: seq_num 211 ranges {746277=619070,1830877=681726,2928369=617908} 2024-09-11T01:32:31.629 INFO:tasks.rados.rados.0.smithi071.stdout:535: writing smithi07134777-46 from 746277 to 1365347 tid 1 2024-09-11T01:32:31.636 INFO:tasks.rados.rados.0.smithi071.stdout:535: writing smithi07134777-46 from 1830877 to 2512603 tid 2 2024-09-11T01:32:31.641 INFO:tasks.rados.rados.0.smithi071.stdout:535: writing smithi07134777-46 from 2928369 to 3546277 tid 3 2024-09-11T01:32:31.642 INFO:tasks.rados.rados.0.smithi071.stdout:536: snap_remove snap 46 2024-09-11T01:32:32.082 INFO:tasks.rados.rados.0.smithi071.stdout:536: done (1 left) 2024-09-11T01:32:32.082 INFO:tasks.rados.rados.0.smithi071.stdout:537: copy_from oid 43 from oid 48 current snap is 49 2024-09-11T01:32:32.082 INFO:tasks.rados.rados.0.smithi071.stdout:535: finishing write tid 1 to smithi07134777-46 2024-09-11T01:32:32.083 INFO:tasks.rados.rados.0.smithi071.stdout:535: finishing write tid 2 to smithi07134777-46 2024-09-11T01:32:32.083 INFO:tasks.rados.rados.0.smithi071.stdout:535: finishing write tid 3 to smithi07134777-46 2024-09-11T01:32:32.083 INFO:tasks.rados.rados.0.smithi071.stdout:535: finishing write tid 4 to smithi07134777-46 2024-09-11T01:32:32.083 INFO:tasks.rados.rados.0.smithi071.stdout:535: finishing write tid 5 to smithi07134777-46 2024-09-11T01:32:32.083 INFO:tasks.rados.rados.0.smithi071.stdout:535: oid 46 updating version 0 to 128 2024-09-11T01:32:32.083 INFO:tasks.rados.rados.0.smithi071.stdout:535: oid 46 updating version 128 to 131 2024-09-11T01:32:32.083 INFO:tasks.rados.rados.0.smithi071.stdout:535: oid 46 version 131 is already newer than 129 2024-09-11T01:32:32.083 INFO:tasks.rados.rados.0.smithi071.stdout:535: oid 46 version 131 is already newer than 130 2024-09-11T01:32:32.083 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 131 (ObjNum 211 snap 49 seq_num 211) dirty exists 2024-09-11T01:32:32.083 INFO:tasks.rados.rados.0.smithi071.stdout:535: left oid 46 (ObjNum 211 snap 49 seq_num 211) 2024-09-11T01:32:32.083 INFO:tasks.rados.rados.0.smithi071.stdout:535: done (1 left) 2024-09-11T01:32:32.083 INFO:tasks.rados.rados.0.smithi071.stdout:538: read oid 42 snap -1 2024-09-11T01:32:32.083 INFO:tasks.rados.rados.0.smithi071.stdout:538: expect deleted 2024-09-11T01:32:32.084 INFO:tasks.rados.rados.0.smithi071.stdout:539: delete oid 14 current snap is 49 2024-09-11T01:32:32.094 INFO:tasks.rados.rados.0.smithi071.stdout:538: done (2 left) 2024-09-11T01:32:32.094 INFO:tasks.rados.rados.0.smithi071.stdout:539: done (1 left) 2024-09-11T01:32:32.094 INFO:tasks.rados.rados.0.smithi071.stdout:540: snap_create 2024-09-11T01:32:32.098 INFO:tasks.rados.rados.0.smithi071.stdout:537: finishing copy_from racing read to smithi07134777-43 2024-09-11T01:32:32.102 INFO:tasks.rados.rados.0.smithi071.stdout:537: finishing copy_from to smithi07134777-43 2024-09-11T01:32:32.102 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 155 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-11T01:32:33.082 INFO:tasks.rados.rados.0.smithi071.stdout:537: done (1 left) 2024-09-11T01:32:33.082 INFO:tasks.rados.rados.0.smithi071.stdout:540: done (0 left) 2024-09-11T01:32:33.082 INFO:tasks.rados.rados.0.smithi071.stdout:541: write oid 23 current snap is 50 2024-09-11T01:32:33.082 INFO:tasks.rados.rados.0.smithi071.stdout:541: seq_num 213 ranges {779891=485921,1947841=685395,3383215=196676} 2024-09-11T01:32:33.088 INFO:tasks.rados.rados.0.smithi071.stdout:541: writing smithi07134777-23 from 779891 to 1265812 tid 1 2024-09-11T01:32:33.096 INFO:tasks.rados.rados.0.smithi071.stdout:541: writing smithi07134777-23 from 1947841 to 2633236 tid 2 2024-09-11T01:32:33.102 INFO:tasks.rados.rados.0.smithi071.stdout:541: writing smithi07134777-23 from 3383215 to 3579891 tid 3 2024-09-11T01:32:33.102 INFO:tasks.rados.rados.0.smithi071.stdout:542: copy_from oid 25 from oid 39 current snap is 50 2024-09-11T01:32:33.102 INFO:tasks.rados.rados.0.smithi071.stdout:541: finishing write tid 1 to smithi07134777-23 2024-09-11T01:32:33.102 INFO:tasks.rados.rados.0.smithi071.stdout:541: finishing write tid 2 to smithi07134777-23 2024-09-11T01:32:33.102 INFO:tasks.rados.rados.0.smithi071.stdout:543: snap_create 2024-09-11T01:32:33.104 INFO:tasks.rados.rados.0.smithi071.stdout:541: finishing write tid 3 to smithi07134777-23 2024-09-11T01:32:33.104 INFO:tasks.rados.rados.0.smithi071.stdout:541: finishing write tid 4 to smithi07134777-23 2024-09-11T01:32:33.105 INFO:tasks.rados.rados.0.smithi071.stdout:541: finishing write tid 5 to smithi07134777-23 2024-09-11T01:32:33.105 INFO:tasks.rados.rados.0.smithi071.stdout:541: oid 23 updating version 0 to 218 2024-09-11T01:32:33.105 INFO:tasks.rados.rados.0.smithi071.stdout:541: oid 23 version 218 is already newer than 217 2024-09-11T01:32:33.105 INFO:tasks.rados.rados.0.smithi071.stdout:541: oid 23 version 218 is already newer than 216 2024-09-11T01:32:33.105 INFO:tasks.rados.rados.0.smithi071.stdout:541: oid 23 updating version 218 to 219 2024-09-11T01:32:33.105 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 219 (ObjNum 213 snap 50 seq_num 213) dirty exists 2024-09-11T01:32:33.105 INFO:tasks.rados.rados.0.smithi071.stdout:541: left oid 23 (ObjNum 213 snap 50 seq_num 213) 2024-09-11T01:32:33.112 INFO:tasks.rados.rados.0.smithi071.stdout:542: finishing copy_from to smithi07134777-25 2024-09-11T01:32:33.112 INFO:tasks.rados.rados.0.smithi071.stdout:542: got expected ENOENT (src dne) 2024-09-11T01:32:33.112 INFO:tasks.rados.rados.0.smithi071.stdout:542: finishing copy_from racing read to smithi07134777-25 2024-09-11T01:32:34.082 INFO:tasks.rados.rados.0.smithi071.stdout:541: done (2 left) 2024-09-11T01:32:34.082 INFO:tasks.rados.rados.0.smithi071.stdout:542: done (1 left) 2024-09-11T01:32:34.082 INFO:tasks.rados.rados.0.smithi071.stdout:543: done (0 left) 2024-09-11T01:32:34.082 INFO:tasks.rados.rados.0.smithi071.stdout:544: read oid 24 snap -1 2024-09-11T01:32:34.082 INFO:tasks.rados.rados.0.smithi071.stdout:544: expect (ObjNum 176 snap 37 seq_num 176) 2024-09-11T01:32:34.082 INFO:tasks.rados.rados.0.smithi071.stdout:545: read oid 41 snap -1 2024-09-11T01:32:34.082 INFO:tasks.rados.rados.0.smithi071.stdout:545: expect deleted 2024-09-11T01:32:34.082 INFO:tasks.rados.rados.0.smithi071.stdout:546: snap_create 2024-09-11T01:32:34.089 INFO:tasks.rados.rados.0.smithi071.stdout:544: expect (ObjNum 176 snap 37 seq_num 176) 2024-09-11T01:32:34.165 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2170438 2024-09-11T01:32:34.212 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2673849 2024-09-11T01:32:35.083 INFO:tasks.rados.rados.0.smithi071.stdout:544: done (2 left) 2024-09-11T01:32:35.083 INFO:tasks.rados.rados.0.smithi071.stdout:545: done (1 left) 2024-09-11T01:32:35.084 INFO:tasks.rados.rados.0.smithi071.stdout:546: done (0 left) 2024-09-11T01:32:35.084 INFO:tasks.rados.rados.0.smithi071.stdout:547: write (excl) oid 36 current snap is 52 2024-09-11T01:32:35.084 INFO:tasks.rados.rados.0.smithi071.stdout:547: seq_num 214 ranges {796698=561170,2006323=545406,3126991=469707} 2024-09-11T01:32:35.090 INFO:tasks.rados.rados.0.smithi071.stdout:547: writing smithi07134777-36 from 796698 to 1357868 tid 1 2024-09-11T01:32:35.097 INFO:tasks.rados.rados.0.smithi071.stdout:547: writing smithi07134777-36 from 2006323 to 2551729 tid 2 2024-09-11T01:32:35.103 INFO:tasks.rados.rados.0.smithi071.stdout:547: writing smithi07134777-36 from 3126991 to 3596698 tid 3 2024-09-11T01:32:35.103 INFO:tasks.rados.rados.0.smithi071.stdout:548: read oid 1 snap 50 2024-09-11T01:32:35.103 INFO:tasks.rados.rados.0.smithi071.stdout:548: expect (ObjNum 92 snap 19 seq_num 92) 2024-09-11T01:32:35.103 INFO:tasks.rados.rados.0.smithi071.stdout:549: rollback oid 38 current snap is 52 2024-09-11T01:32:35.103 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 38 to 51 2024-09-11T01:32:35.103 INFO:tasks.rados.rados.0.smithi071.stdout:547: finishing write tid 1 to smithi07134777-36 2024-09-11T01:32:35.103 INFO:tasks.rados.rados.0.smithi071.stdout:550: copy_from oid 29 from oid 32 current snap is 52 2024-09-11T01:32:35.103 INFO:tasks.rados.rados.0.smithi071.stdout:551: delete oid 33 current snap is 52 2024-09-11T01:32:35.104 INFO:tasks.rados.rados.0.smithi071.stdout:547: finishing write tid 2 to smithi07134777-36 2024-09-11T01:32:35.107 INFO:tasks.rados.rados.0.smithi071.stdout:549: finishing rollback tid 1 to smithi07134777-38 2024-09-11T01:32:35.107 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 0 (ObjNum 33 snap 0 seq_num 2371439776) dirty dne 2024-09-11T01:32:35.107 INFO:tasks.rados.rados.0.smithi071.stdout:549: done (4 left) 2024-09-11T01:32:35.107 INFO:tasks.rados.rados.0.smithi071.stdout:551: done (3 left) 2024-09-11T01:32:35.107 INFO:tasks.rados.rados.0.smithi071.stdout:552: write oid 2 current snap is 52 2024-09-11T01:32:35.107 INFO:tasks.rados.rados.0.smithi071.stdout:552: seq_num 216 ranges {430312=711668,1723287=549075,2781837=703343} 2024-09-11T01:32:35.113 INFO:tasks.rados.rados.0.smithi071.stdout:552: writing smithi07134777-2 from 430312 to 1141980 tid 1 2024-09-11T01:32:35.119 INFO:tasks.rados.rados.0.smithi071.stdout:552: writing smithi07134777-2 from 1723287 to 2272362 tid 2 2024-09-11T01:32:35.126 INFO:tasks.rados.rados.0.smithi071.stdout:552: writing smithi07134777-2 from 2781837 to 3485180 tid 3 2024-09-11T01:32:35.127 INFO:tasks.rados.rados.0.smithi071.stdout:553: write (excl) oid 30 current snap is 52 2024-09-11T01:32:35.127 INFO:tasks.rados.rados.0.smithi071.stdout:553: seq_num 217 ranges {447119=786917,1665416=692733,2976554=514615} 2024-09-11T01:32:35.133 INFO:tasks.rados.rados.0.smithi071.stdout:553: writing smithi07134777-30 from 447119 to 1234036 tid 1 2024-09-11T01:32:35.139 INFO:tasks.rados.rados.0.smithi071.stdout:553: writing smithi07134777-30 from 1665416 to 2358149 tid 2 2024-09-11T01:32:35.145 INFO:tasks.rados.rados.0.smithi071.stdout:553: writing smithi07134777-30 from 2976554 to 3491169 tid 3 2024-09-11T01:32:35.145 INFO:tasks.rados.rados.0.smithi071.stdout:554: delete oid 3 current snap is 52 2024-09-11T01:32:35.146 INFO:tasks.rados.rados.0.smithi071.stdout:548: expect (ObjNum 92 snap 19 seq_num 92) 2024-09-11T01:32:35.186 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 247772 2024-09-11T01:32:35.265 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3264164 2024-09-11T01:32:35.269 INFO:tasks.rados.rados.0.smithi071.stdout:547: finishing write tid 3 to smithi07134777-36 2024-09-11T01:32:35.269 INFO:tasks.rados.rados.0.smithi071.stdout:547: finishing write tid 4 to smithi07134777-36 2024-09-11T01:32:35.269 INFO:tasks.rados.rados.0.smithi071.stdout:547: finishing write tid 5 to smithi07134777-36 2024-09-11T01:32:35.269 INFO:tasks.rados.rados.0.smithi071.stdout:547: oid 36 updating version 0 to 68 2024-09-11T01:32:35.269 INFO:tasks.rados.rados.0.smithi071.stdout:547: oid 36 version 68 is already newer than 66 2024-09-11T01:32:35.270 INFO:tasks.rados.rados.0.smithi071.stdout:547: oid 36 version 68 is already newer than 67 2024-09-11T01:32:35.270 INFO:tasks.rados.rados.0.smithi071.stdout:547: oid 36 updating version 68 to 69 2024-09-11T01:32:35.270 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 69 (ObjNum 214 snap 52 seq_num 214) dirty exists 2024-09-11T01:32:35.270 INFO:tasks.rados.rados.0.smithi071.stdout:547: left oid 36 (ObjNum 214 snap 52 seq_num 214) 2024-09-11T01:32:35.270 INFO:tasks.rados.rados.0.smithi071.stdout:550: finishing copy_from to smithi07134777-29 2024-09-11T01:32:35.270 INFO:tasks.rados.rados.0.smithi071.stdout:550: got expected ENOENT (src dne) 2024-09-11T01:32:35.270 INFO:tasks.rados.rados.0.smithi071.stdout:550: finishing copy_from racing read to smithi07134777-29 2024-09-11T01:32:35.270 INFO:tasks.rados.rados.0.smithi071.stdout:552: finishing write tid 1 to smithi07134777-2 2024-09-11T01:32:35.270 INFO:tasks.rados.rados.0.smithi071.stdout:552: finishing write tid 2 to smithi07134777-2 2024-09-11T01:32:35.270 INFO:tasks.rados.rados.0.smithi071.stdout:547: done (5 left) 2024-09-11T01:32:35.270 INFO:tasks.rados.rados.0.smithi071.stdout:548: done (4 left) 2024-09-11T01:32:35.270 INFO:tasks.rados.rados.0.smithi071.stdout:550: done (3 left) 2024-09-11T01:32:35.270 INFO:tasks.rados.rados.0.smithi071.stdout:554: done (2 left) 2024-09-11T01:32:35.271 INFO:tasks.rados.rados.0.smithi071.stdout:555: snap_create 2024-09-11T01:32:35.271 INFO:tasks.rados.rados.0.smithi071.stdout:552: finishing write tid 3 to smithi07134777-2 2024-09-11T01:32:35.271 INFO:tasks.rados.rados.0.smithi071.stdout:552: finishing write tid 4 to smithi07134777-2 2024-09-11T01:32:35.271 INFO:tasks.rados.rados.0.smithi071.stdout:552: finishing write tid 5 to smithi07134777-2 2024-09-11T01:32:35.271 INFO:tasks.rados.rados.0.smithi071.stdout:552: oid 2 updating version 0 to 129 2024-09-11T01:32:35.271 INFO:tasks.rados.rados.0.smithi071.stdout:552: oid 2 version 129 is already newer than 127 2024-09-11T01:32:35.271 INFO:tasks.rados.rados.0.smithi071.stdout:552: oid 2 version 129 is already newer than 128 2024-09-11T01:32:35.271 INFO:tasks.rados.rados.0.smithi071.stdout:552: oid 2 version 129 is already newer than 126 2024-09-11T01:32:35.271 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 129 (ObjNum 216 snap 52 seq_num 216) dirty exists 2024-09-11T01:32:35.271 INFO:tasks.rados.rados.0.smithi071.stdout:552: left oid 2 (ObjNum 216 snap 52 seq_num 216) 2024-09-11T01:32:35.271 INFO:tasks.rados.rados.0.smithi071.stdout:553: finishing write tid 1 to smithi07134777-30 2024-09-11T01:32:35.271 INFO:tasks.rados.rados.0.smithi071.stdout:553: finishing write tid 2 to smithi07134777-30 2024-09-11T01:32:35.271 INFO:tasks.rados.rados.0.smithi071.stdout:553: finishing write tid 3 to smithi07134777-30 2024-09-11T01:32:35.272 INFO:tasks.rados.rados.0.smithi071.stdout:553: finishing write tid 4 to smithi07134777-30 2024-09-11T01:32:35.272 INFO:tasks.rados.rados.0.smithi071.stdout:553: finishing write tid 5 to smithi07134777-30 2024-09-11T01:32:35.272 INFO:tasks.rados.rados.0.smithi071.stdout:553: oid 30 updating version 0 to 208 2024-09-11T01:32:35.272 INFO:tasks.rados.rados.0.smithi071.stdout:553: oid 30 updating version 208 to 210 2024-09-11T01:32:35.272 INFO:tasks.rados.rados.0.smithi071.stdout:553: oid 30 version 210 is already newer than 207 2024-09-11T01:32:35.272 INFO:tasks.rados.rados.0.smithi071.stdout:553: oid 30 version 210 is already newer than 209 2024-09-11T01:32:35.272 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 210 (ObjNum 217 snap 52 seq_num 217) dirty exists 2024-09-11T01:32:35.272 INFO:tasks.rados.rados.0.smithi071.stdout:553: left oid 30 (ObjNum 217 snap 52 seq_num 217) 2024-09-11T01:32:35.959 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-11T01:32:35.959 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-11T01:32:35.959 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-11T01:32:35.962 INFO:tasks.ceph.mon.c:Started 2024-09-11T01:32:35.963 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:32:35.963 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:32:36.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.074+0000 7f46bcee6640 10 monclient: get_monmap_and_config 2024-09-11T01:32:36.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.074+0000 7f46bcee6640 10 monclient: build_initial_monmap 2024-09-11T01:32:36.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.074+0000 7f46bcee6640 10 monclient: monmap: 2024-09-11T01:32:36.076 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:36.076 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:36.076 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:32:36.075218+0000 2024-09-11T01:32:36.076 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:32:36.075218+0000 2024-09-11T01:32:36.076 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:36.077 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:36.077 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:32:36.077 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:32:36.077 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:32:36.077 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:32:36.077 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:32:36.077 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:36.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.075+0000 7f46bcee6640 1 Processor -- start 2024-09-11T01:32:36.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.075+0000 7f46bcee6640 1 -- start start 2024-09-11T01:32:36.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.075+0000 7f46bcee6640 10 monclient: init 2024-09-11T01:32:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.075+0000 7f46bcee6640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.075+0000 7f46bcee6640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-11T01:32:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.075+0000 7f46bcee6640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f46b8138210 0x7f46b81385f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.075+0000 7f46bcee6640 10 monclient(hunting): picked mon.noname-a con 0x7f46b8138210 addr v2:172.21.15.12:3300/0 2024-09-11T01:32:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.075+0000 7f46bcee6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f46b80a6810 addr v1:172.21.15.71:6789/0 2024-09-11T01:32:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.075+0000 7f46bcee6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46b80a57a0 0x7f46b8140120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.075+0000 7f46bcee6640 10 monclient(hunting): picked mon.noname-b con 0x7f46b80a57a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.075+0000 7f46bcee6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46b809ba20 con 0x7f46b80a6810 2024-09-11T01:32:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.075+0000 7f46bcee6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.075+0000 7f46bcee6640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f46b809c7a0 con 0x7f46b8138210 2024-09-11T01:32:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.075+0000 7f46bcee6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.075+0000 7f46bcee6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f46b809c4c0 con 0x7f46b80a57a0 2024-09-11T01:32:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.075+0000 7f46bcee6640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.075+0000 7f46bcee6640 10 monclient(hunting): authenticate will time out at 962.517875s 2024-09-11T01:32:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.075+0000 7f46b659c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f46b8138210 0x7f46b81385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.075+0000 7f46b6d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46b80a57a0 0x7f46b8140120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b659c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f46b8138210 0x7f46b81385f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58728/0 (socket says 172.21.15.12:58728) 2024-09-11T01:32:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b659c640 1 -- 172.21.15.12:0/811409197 learned_addr learned my addr 172.21.15.12:0/811409197 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b659c640 10 monclient(hunting): get_auth_request con 0x7f46b8138210 auth_method 0 2024-09-11T01:32:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b6d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46b80a57a0 0x7f46b8140120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:50112/0 (socket says 172.21.15.12:50112) 2024-09-11T01:32:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b6d9d640 10 monclient(hunting): get_auth_request con 0x7f46b80a57a0 auth_method 0 2024-09-11T01:32:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b6d9d640 1 -- 172.21.15.12:0/811409197 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46b80a57a0 msgr2=0x7f46b8140120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:32:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b6d9d640 1 -- 172.21.15.12:0/811409197 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46b80a57a0 msgr2=0x7f46b8140120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:32:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b6d9d640 1 --2- 172.21.15.12:0/811409197 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46b80a57a0 0x7f46b8140120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b6d9d640 1 --2- 172.21.15.12:0/811409197 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46b80a57a0 0x7f46b8140120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b659c640 10 monclient(hunting): handle_auth_done global_id 24297 payload 274 2024-09-11T01:32:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b659c640 1 -- 172.21.15.12:0/811409197 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46b80a57a0 msgr2=0x7f46b8140120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:32:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b659c640 1 --2- 172.21.15.12:0/811409197 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46b80a57a0 0x7f46b8140120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:36.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b659c640 1 -- 172.21.15.12:0/811409197 >> v1:172.21.15.71:6789/0 conn(0x7f46b80a6810 legacy=0x7f46b80a5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:36.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b659c640 10 monclient: _finish_hunting 0 2024-09-11T01:32:36.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b659c640 1 monclient: found mon.noname-a 2024-09-11T01:32:36.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:36.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:32:36.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.076+0000 7f46b659c640 1 -- 172.21.15.12:0/811409197 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46b80a6590 con 0x7f46b8138210 2024-09-11T01:32:36.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46b659c640 1 -- 172.21.15.12:0/811409197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46b8138210 msgr2=0x7f46b81385f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:36.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46b659c640 1 --2- 172.21.15.12:0/811409197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46b8138210 0x7f46b81385f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f46a80098a0 tx=0x7f46a802a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ace8fa8b33ac811 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:36.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46b5d9b640 1 -- 172.21.15.12:0/811409197 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a8037070 con 0x7f46b8138210 2024-09-11T01:32:36.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46b5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:36.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46b5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:32:36.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46b5d9b640 10 monclient: dump: 2024-09-11T01:32:36.089 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:36.089 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:36.089 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:36.089 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:36.089 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:36.089 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:36.089 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:36.089 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:36.089 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:36.089 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:36.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46b5d9b640 10 monclient: _finish_auth 0 2024-09-11T01:32:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46b5d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:32:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46b5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46bcee6640 5 monclient: authenticate success, global_id 24297 2024-09-11T01:32:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46bcee6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:32:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46b5d9b640 1 -- 172.21.15.12:0/811409197 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46a8032440 con 0x7f46b8138210 2024-09-11T01:32:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46b5d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46bcee6640 10 monclient: get_monmap_and_config success 2024-09-11T01:32:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46b5d9b640 1 -- 172.21.15.12:0/811409197 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a8032810 con 0x7f46b8138210 2024-09-11T01:32:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46bcee6640 4 set_mon_vals no callback set 2024-09-11T01:32:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46b5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46b5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46b5d9b640 10 monclient: dump: 2024-09-11T01:32:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:36.091 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:36.091 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:36.091 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:36.091 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:36.091 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:36.091 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:36.091 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:36.091 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:36.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46bcee6640 10 monclient: shutdown 2024-09-11T01:32:36.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46bcee6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:36.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46bcee6640 1 -- 172.21.15.12:0/811409197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46b8138210 msgr2=0x7f46b81385f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:36.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46bcee6640 1 --2- 172.21.15.12:0/811409197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46b8138210 0x7f46b81385f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f46a80098a0 tx=0x7f46a802a0d0 comp rx=0 tx=0).stop 2024-09-11T01:32:36.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46bcee6640 1 -- 172.21.15.12:0/811409197 shutdown_connections 2024-09-11T01:32:36.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46bcee6640 1 --2- 172.21.15.12:0/811409197 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46b80a57a0 0x7f46b8140120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:36.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46bcee6640 1 --2- 172.21.15.12:0/811409197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46b8138210 0x7f46b81385f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:36.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46bcee6640 1 -- 172.21.15.12:0/811409197 >> 172.21.15.12:0/811409197 conn(0x7f46b80aaa80 msgr2=0x7f46b80aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:36.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.079+0000 7f46bcee6640 1 -- 172.21.15.12:0/811409197 shutdown_connections 2024-09-11T01:32:36.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.080+0000 7f46bcee6640 1 -- 172.21.15.12:0/811409197 wait complete. 2024-09-11T01:32:36.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.080+0000 7f46bcee6640 10 monclient: build_initial_monmap 2024-09-11T01:32:36.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.080+0000 7f46bcee6640 10 monclient: monmap: 2024-09-11T01:32:36.092 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:36.092 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:32:36.092 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:32:36.092 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:32:36.092 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:36.092 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:36.092 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:32:36.093 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:32:36.093 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:32:36.093 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:36.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46bcee6640 1 Processor -- start 2024-09-11T01:32:36.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46bcee6640 1 -- start start 2024-09-11T01:32:36.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46bcee6640 10 monclient: init 2024-09-11T01:32:36.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46bcee6640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:36.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46bcee6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:32:36.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46bcee6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46b80a57a0 0x7f46b8125270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:36.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46bcee6640 10 monclient(hunting): picked mon.noname-a con 0x7f46b80a57a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:32:36.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46bcee6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46b80a6810 0x7f46b81257b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:36.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46bcee6640 10 monclient(hunting): picked mon.noname-b con 0x7f46b80a6810 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:32:36.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46bcee6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46b8138210 0x7f46b8125cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:36.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46bcee6640 10 monclient(hunting): picked mon.noname-c con 0x7f46b8138210 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:36.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46bcee6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:36.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46bcee6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f46b806f670 con 0x7f46b80a57a0 2024-09-11T01:32:36.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46bcee6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:36.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46bcee6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f46b806f4f0 con 0x7f46b80a6810 2024-09-11T01:32:36.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46bcee6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:36.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46bcee6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f46b8138b30 con 0x7f46b8138210 2024-09-11T01:32:36.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46bcee6640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:36.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46bcee6640 10 monclient(hunting): authenticate will time out at 962.523629s 2024-09-11T01:32:36.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46b659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46b80a57a0 0x7f46b8125270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:36.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46b6d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46b8138210 0x7f46b8125cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:36.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46b659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46b80a57a0 0x7f46b8125270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58730/0 (socket says 172.21.15.12:58730) 2024-09-11T01:32:36.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46b659c640 1 -- 172.21.15.12:0/3096587538 learned_addr learned my addr 172.21.15.12:0/3096587538 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:36.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46b659c640 10 monclient(hunting): get_auth_request con 0x7f46b80a57a0 auth_method 0 2024-09-11T01:32:36.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:36.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46b659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:36.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46b659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:36.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46b6d9d640 10 monclient(hunting): get_auth_request con 0x7f46b8138210 auth_method 0 2024-09-11T01:32:36.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46b6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:36.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46b6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:36.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46b6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:36.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:36.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:36.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:36.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46b6d9d640 1 -- 172.21.15.12:0/3096587538 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46b8138210 msgr2=0x7f46b8125cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:32:36.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46b6d9d640 1 -- 172.21.15.12:0/3096587538 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46b8138210 msgr2=0x7f46b8125cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:32:36.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46b6d9d640 1 --2- 172.21.15.12:0/3096587538 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46b8138210 0x7f46b8125cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:36.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46affff640 1 --2- 172.21.15.12:0/3096587538 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46b80a6810 0x7f46b81257b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:36.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46b6d9d640 1 --2- 172.21.15.12:0/3096587538 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46b8138210 0x7f46b8125cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:36.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46affff640 10 monclient(hunting): get_auth_request con 0x7f46b80a6810 auth_method 0 2024-09-11T01:32:36.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:36.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46affff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:36.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:36.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.081+0000 7f46b659c640 10 monclient(hunting): handle_auth_done global_id 24300 payload 824 2024-09-11T01:32:36.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.082+0000 7f46b659c640 1 -- 172.21.15.12:0/3096587538 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46b8138210 msgr2=0x7f46b8125cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:32:36.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.082+0000 7f46b659c640 1 --2- 172.21.15.12:0/3096587538 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46b8138210 0x7f46b8125cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:36.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.082+0000 7f46b659c640 1 -- 172.21.15.12:0/3096587538 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46b80a6810 msgr2=0x7f46b81257b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:36.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.082+0000 7f46b659c640 1 --2- 172.21.15.12:0/3096587538 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46b80a6810 0x7f46b81257b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:36.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.082+0000 7f46b659c640 10 monclient: _finish_hunting 0 2024-09-11T01:32:36.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.082+0000 7f46b659c640 1 monclient: found mon.noname-a 2024-09-11T01:32:36.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.082+0000 7f46b659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:36.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.082+0000 7f46b659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:32:36.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.082+0000 7f46b659c640 1 -- 172.21.15.12:0/3096587538 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46b8128db0 con 0x7f46b80a57a0 2024-09-11T01:32:36.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.082+0000 7f46affff640 1 --2- 172.21.15.12:0/3096587538 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46b80a6810 0x7f46b81257b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:32:36.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.084+0000 7f46b659c640 1 -- 172.21.15.12:0/3096587538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46b80a57a0 msgr2=0x7f46b8125270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:36.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.084+0000 7f46b659c640 1 --2- 172.21.15.12:0/3096587538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46b80a57a0 0x7f46b8125270 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f46a80097f0 tx=0x7f46a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:36.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.084+0000 7f46aeffd640 1 -- 172.21.15.12:0/3096587538 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a8041020 con 0x7f46b80a57a0 2024-09-11T01:32:36.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.084+0000 7f46aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:36.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.084+0000 7f46aeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:32:36.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.084+0000 7f46aeffd640 10 monclient: dump: 2024-09-11T01:32:36.097 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:36.097 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:36.097 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:36.098 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:36.098 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:36.098 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:36.098 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:36.098 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:36.098 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:36.098 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:36.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.084+0000 7f46aeffd640 10 monclient: _finish_auth 0 2024-09-11T01:32:36.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.084+0000 7f46aeffd640 10 monclient: _check_auth_tickets 2024-09-11T01:32:36.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.084+0000 7f46aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:36.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.084+0000 7f46bcee6640 5 monclient: authenticate success, global_id 24300 2024-09-11T01:32:36.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.084+0000 7f46aeffd640 1 -- 172.21.15.12:0/3096587538 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46a803d620 con 0x7f46b80a57a0 2024-09-11T01:32:36.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.084+0000 7f46aeffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:36.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.084+0000 7f46aeffd640 1 -- 172.21.15.12:0/3096587538 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a8037040 con 0x7f46b80a57a0 2024-09-11T01:32:36.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.084+0000 7f46aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:36.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.084+0000 7f46aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:36.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.084+0000 7f46aeffd640 10 monclient: dump: 2024-09-11T01:32:36.099 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:36.099 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:36.099 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:36.099 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:36.099 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:36.099 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:36.099 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:36.099 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:36.099 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:36.099 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:36.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.084+0000 7f46bcee6640 10 monclient: _renew_subs 2024-09-11T01:32:36.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.084+0000 7f46b559a640 4 set_mon_vals no callback set 2024-09-11T01:32:36.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.084+0000 7f46bcee6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:36.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.084+0000 7f46bcee6640 1 -- 172.21.15.12:0/3096587538 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46b8129090 con 0x7f46b80a57a0 2024-09-11T01:32:36.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.084+0000 7f46bcee6640 10 monclient: _renew_subs 2024-09-11T01:32:36.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.084+0000 7f46bcee6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:36.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.084+0000 7f46bcee6640 1 -- 172.21.15.12:0/3096587538 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46b80a1010 con 0x7f46b80a57a0 2024-09-11T01:32:36.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.085+0000 7f46bcee6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:36.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.085+0000 7f46bcee6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:36.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.085+0000 7f46bcee6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:36.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.085+0000 7f46aeffd640 1 -- 172.21.15.12:0/3096587538 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f46a803a030 con 0x7f46b80a57a0 2024-09-11T01:32:36.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.085+0000 7f46bcee6640 1 -- 172.21.15.12:0/3096587538 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46b80a6c70 con 0x7f46b80a57a0 2024-09-11T01:32:36.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.085+0000 7f46aeffd640 1 --2- 172.21.15.12:0/3096587538 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f469803e210 0x7f46980406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:36.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.085+0000 7f46aeffd640 1 -- 172.21.15.12:0/3096587538 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 4755+0+0 (secure 0 0 0) 0x7f46a8073180 con 0x7f46b80a57a0 2024-09-11T01:32:36.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.086+0000 7f46affff640 1 --2- 172.21.15.12:0/3096587538 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f469803e210 0x7f46980406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:36.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.086+0000 7f46affff640 10 monclient: get_auth_request con 0x7f469803e210 auth_method 0 2024-09-11T01:32:36.101 INFO:tasks.rados.rados.0.smithi071.stdout:552: done (2 left) 2024-09-11T01:32:36.101 INFO:tasks.rados.rados.0.smithi071.stdout:553: done (1 left) 2024-09-11T01:32:36.101 INFO:tasks.rados.rados.0.smithi071.stdout:555: done (0 left) 2024-09-11T01:32:36.102 INFO:tasks.rados.rados.0.smithi071.stdout:556: snap_remove snap 51 2024-09-11T01:32:36.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.088+0000 7f46affff640 1 --2- 172.21.15.12:0/3096587538 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f469803e210 0x7f46980406d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f46a0003eb0 tx=0x7f46a0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:36.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.090+0000 7f46aeffd640 1 -- 172.21.15.12:0/3096587538 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f46b80a6c70 con 0x7f46b80a57a0 2024-09-11T01:32:36.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.090+0000 7f46aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:36.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.090+0000 7f46aeffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:32:36.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.347+0000 7f46bcee6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:32:36.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.347+0000 7f46bcee6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:32:36.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.347+0000 7f46bcee6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:36.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.347+0000 7f46bcee6640 1 -- 172.21.15.12:0/3096587538 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f46b80a7b10 con 0x7f46b80a57a0 2024-09-11T01:32:36.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.348+0000 7f46aeffd640 1 -- 172.21.15.12:0/3096587538 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f46a8071e20 con 0x7f46b80a57a0 2024-09-11T01:32:36.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.348+0000 7f46aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:32:36.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.348+0000 7f46aeffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:32:36.350 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":32,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:32:36.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.349+0000 7f46bcee6640 1 -- 172.21.15.12:0/3096587538 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f469803e210 msgr2=0x7f46980406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:36.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.349+0000 7f46bcee6640 1 --2- 172.21.15.12:0/3096587538 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f469803e210 0x7f46980406d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f46a0003eb0 tx=0x7f46a0009040 comp rx=0 tx=0).stop 2024-09-11T01:32:36.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.349+0000 7f46bcee6640 10 monclient: shutdown 2024-09-11T01:32:36.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.349+0000 7f46bcee6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.349+0000 7f46bcee6640 1 -- 172.21.15.12:0/3096587538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46b80a57a0 msgr2=0x7f46b8125270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.349+0000 7f46bcee6640 1 --2- 172.21.15.12:0/3096587538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46b80a57a0 0x7f46b8125270 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f46a80097f0 tx=0x7f46a80046c0 comp rx=0 tx=0).stop 2024-09-11T01:32:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.350+0000 7f46bcee6640 1 -- 172.21.15.12:0/3096587538 shutdown_connections 2024-09-11T01:32:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.350+0000 7f46bcee6640 1 --2- 172.21.15.12:0/3096587538 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46b8138210 0x7f46b8125cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.350+0000 7f46bcee6640 1 --2- 172.21.15.12:0/3096587538 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46b80a6810 0x7f46b81257b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.350+0000 7f46bcee6640 1 --2- 172.21.15.12:0/3096587538 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f469803e210 0x7f46980406d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.350+0000 7f46bcee6640 1 --2- 172.21.15.12:0/3096587538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46b80a57a0 0x7f46b8125270 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.350+0000 7f46bcee6640 1 -- 172.21.15.12:0/3096587538 >> 172.21.15.12:0/3096587538 conn(0x7f46b80aaa80 msgr2=0x7f46b80a9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.350+0000 7f46bcee6640 1 -- 172.21.15.12:0/3096587538 shutdown_connections 2024-09-11T01:32:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:36.350+0000 7f46bcee6640 1 -- 172.21.15.12:0/3096587538 wait complete. 2024-09-11T01:32:37.080 INFO:tasks.rados.rados.0.smithi071.stdout:556: done (0 left) 2024-09-11T01:32:37.080 INFO:tasks.rados.rados.0.smithi071.stdout:557: copy_from oid 31 from oid 5 current snap is 53 2024-09-11T01:32:37.080 INFO:tasks.rados.rados.0.smithi071.stdout:558: rollback oid 43 current snap is 53 2024-09-11T01:32:37.080 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 43 to 50 2024-09-11T01:32:37.080 INFO:tasks.rados.rados.0.smithi071.stdout:559: snap_create 2024-09-11T01:32:37.090 INFO:tasks.rados.rados.0.smithi071.stdout:558: finishing rollback tid 0 to smithi07134777-43 2024-09-11T01:32:37.091 INFO:tasks.rados.rados.0.smithi071.stdout:558: finishing rollback tid 1 to smithi07134777-43 2024-09-11T01:32:37.091 INFO:tasks.rados.rados.0.smithi071.stdout:558: finishing rollback tid 2 to smithi07134777-43 2024-09-11T01:32:37.091 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 180 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-11T01:32:37.103 INFO:tasks.rados.rados.0.smithi071.stdout:557: finishing copy_from racing read to smithi07134777-31 2024-09-11T01:32:37.107 INFO:tasks.rados.rados.0.smithi071.stdout:557: finishing copy_from to smithi07134777-31 2024-09-11T01:32:37.107 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 33 (ObjNum 201 snap 46 seq_num 201) dirty exists 2024-09-11T01:32:38.081 INFO:tasks.rados.rados.0.smithi071.stdout:557: done (2 left) 2024-09-11T01:32:38.082 INFO:tasks.rados.rados.0.smithi071.stdout:558: done (1 left) 2024-09-11T01:32:38.082 INFO:tasks.rados.rados.0.smithi071.stdout:559: done (0 left) 2024-09-11T01:32:38.082 INFO:tasks.rados.rados.0.smithi071.stdout:560: read oid 31 snap 49 2024-09-11T01:32:38.082 INFO:tasks.rados.rados.0.smithi071.stdout:560: expect (ObjNum 123 snap 29 seq_num 123) 2024-09-11T01:32:38.082 INFO:tasks.rados.rados.0.smithi071.stdout:561: read oid 42 snap 50 2024-09-11T01:32:38.082 INFO:tasks.rados.rados.0.smithi071.stdout:561: expect deleted 2024-09-11T01:32:38.082 INFO:tasks.rados.rados.0.smithi071.stdout:562: read oid 35 snap -1 2024-09-11T01:32:38.082 INFO:tasks.rados.rados.0.smithi071.stdout:562: expect (ObjNum 204 snap 47 seq_num 204) 2024-09-11T01:32:38.082 INFO:tasks.rados.rados.0.smithi071.stdout:563: delete oid 34 current snap is 54 2024-09-11T01:32:38.086 INFO:tasks.rados.rados.0.smithi071.stdout:561: done (3 left) 2024-09-11T01:32:38.086 INFO:tasks.rados.rados.0.smithi071.stdout:563: done (2 left) 2024-09-11T01:32:38.086 INFO:tasks.rados.rados.0.smithi071.stdout:564: snap_remove snap 53 2024-09-11T01:32:39.084 INFO:tasks.rados.rados.0.smithi071.stdout:564: done (2 left) 2024-09-11T01:32:39.084 INFO:tasks.rados.rados.0.smithi071.stdout:565: read oid 46 snap -1 2024-09-11T01:32:39.084 INFO:tasks.rados.rados.0.smithi071.stdout:565: expect (ObjNum 211 snap 49 seq_num 211) 2024-09-11T01:32:39.084 INFO:tasks.rados.rados.0.smithi071.stdout:566: write oid 27 current snap is 54 2024-09-11T01:32:39.084 INFO:tasks.rados.rados.0.smithi071.stdout:566: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2024-09-11T01:32:39.089 INFO:tasks.rados.rados.0.smithi071.stdout:566: writing smithi07134777-27 from 497540 to 1110204 tid 1 2024-09-11T01:32:39.096 INFO:tasks.rados.rados.0.smithi071.stdout:566: writing smithi07134777-27 from 1840862 to 2280922 tid 2 2024-09-11T01:32:39.103 INFO:tasks.rados.rados.0.smithi071.stdout:566: writing smithi07134777-27 from 3058823 to 3523607 tid 3 2024-09-11T01:32:39.103 INFO:tasks.rados.rados.0.smithi071.stdout:567: snap_create 2024-09-11T01:32:39.103 INFO:tasks.rados.rados.0.smithi071.stdout:560: expect (ObjNum 123 snap 29 seq_num 123) 2024-09-11T01:32:39.140 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:32:39.211 INFO:tasks.rados.rados.0.smithi071.stdout:562: expect (ObjNum 204 snap 47 seq_num 204) 2024-09-11T01:32:39.211 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 32362 2024-09-11T01:32:39.262 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:32:39.357 INFO:tasks.rados.rados.0.smithi071.stdout:565: expect (ObjNum 211 snap 49 seq_num 211) 2024-09-11T01:32:39.363 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:32:39.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.475+0000 7f10165f8640 10 monclient: get_monmap_and_config 2024-09-11T01:32:39.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.475+0000 7f10165f8640 10 monclient: build_initial_monmap 2024-09-11T01:32:39.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.475+0000 7f10165f8640 10 monclient: monmap: 2024-09-11T01:32:39.477 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:39.477 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:39.477 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:32:39.475924+0000 2024-09-11T01:32:39.477 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:32:39.475924+0000 2024-09-11T01:32:39.477 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:39.477 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:39.477 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:32:39.477 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:32:39.477 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:32:39.477 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:32:39.477 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:32:39.477 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:39.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.475+0000 7f10165f8640 1 Processor -- start 2024-09-11T01:32:39.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f10165f8640 1 -- start start 2024-09-11T01:32:39.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f10165f8640 10 monclient: init 2024-09-11T01:32:39.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f10165f8640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:39.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f10165f8640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-11T01:32:39.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f10165f8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1010138230 0x7f1010138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:39.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f10165f8640 10 monclient(hunting): picked mon.noname-b con 0x7f1010138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:39.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f10165f8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f101009d030 0x7f10100a5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:39.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f10165f8640 10 monclient(hunting): picked mon.noname-a con 0x7f101009d030 addr v2:172.21.15.12:3300/0 2024-09-11T01:32:39.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f10165f8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f101009d430 addr v1:172.21.15.12:6789/0 2024-09-11T01:32:39.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f10165f8640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10100a6e30 con 0x7f101009d430 2024-09-11T01:32:39.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f10165f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:39.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f10165f8640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f101009c9c0 con 0x7f101009d030 2024-09-11T01:32:39.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f10165f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:39.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f10165f8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f101009ba20 con 0x7f1010138230 2024-09-11T01:32:39.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f10165f8640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:39.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f10165f8640 10 monclient(hunting): authenticate will time out at 965.918720s 2024-09-11T01:32:39.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f100ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1010138230 0x7f1010138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:39.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f100f7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f101009d030 0x7f10100a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:39.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f1014b95640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f101009d430 0x7f10100a5560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47632/0 (socket says 172.21.15.12:47632) 2024-09-11T01:32:39.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f1014b95640 1 -- 172.21.15.12:0/468936265 learned_addr learned my addr 172.21.15.12:0/468936265 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:32:39.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f100ffff640 10 monclient(hunting): get_auth_request con 0x7f1010138230 auth_method 0 2024-09-11T01:32:39.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f100ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:39.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f100ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:39.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f100ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:39.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f100f7fe640 10 monclient(hunting): get_auth_request con 0x7f101009d030 auth_method 0 2024-09-11T01:32:39.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f100f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:39.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f100f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:39.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f100f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:39.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f100ffff640 1 -- 172.21.15.12:0/468936265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1010138230 msgr2=0x7f1010138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:32:39.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f100f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:39.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f100ffff640 1 -- 172.21.15.12:0/468936265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1010138230 msgr2=0x7f1010138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:32:39.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f100f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:39.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.476+0000 7f100ffff640 1 --2- 172.21.15.12:0/468936265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1010138230 0x7f1010138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:39.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.477+0000 7f100f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:39.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.477+0000 7f100ffff640 1 --2- 172.21.15.12:0/468936265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1010138230 0x7f1010138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:39.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.477+0000 7f100f7fe640 10 monclient(hunting): handle_auth_done global_id 24303 payload 274 2024-09-11T01:32:39.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.477+0000 7f100f7fe640 1 -- 172.21.15.12:0/468936265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1010138230 msgr2=0x7f1010138610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:32:39.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.477+0000 7f100f7fe640 1 --2- 172.21.15.12:0/468936265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1010138230 0x7f1010138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:39.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.477+0000 7f100f7fe640 1 -- 172.21.15.12:0/468936265 >> v1:172.21.15.12:6789/0 conn(0x7f101009d430 legacy=0x7f10100a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:39.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.477+0000 7f100f7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:32:39.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.477+0000 7f100f7fe640 1 monclient: found mon.noname-a 2024-09-11T01:32:39.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.477+0000 7f100f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:39.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.477+0000 7f100f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:32:39.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.477+0000 7f100f7fe640 1 -- 172.21.15.12:0/468936265 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10100a66a0 con 0x7f101009d030 2024-09-11T01:32:39.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.479+0000 7f100f7fe640 1 -- 172.21.15.12:0/468936265 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f101009d030 msgr2=0x7f10100a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:39.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.479+0000 7f100f7fe640 1 --2- 172.21.15.12:0/468936265 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f101009d030 0x7f10100a5020 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f101006d650 tx=0x7f1004029fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cbb425341135cc4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:39.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f100effd640 1 -- 172.21.15.12:0/468936265 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1004037070 con 0x7f101009d030 2024-09-11T01:32:39.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f100effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:39.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f100effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:32:39.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f100effd640 10 monclient: dump: 2024-09-11T01:32:39.489 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:39.489 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:39.489 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:39.489 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:39.489 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:39.489 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:39.489 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:39.489 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:39.489 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:39.489 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:39.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f100effd640 10 monclient: _finish_auth 0 2024-09-11T01:32:39.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f100effd640 10 monclient: _check_auth_tickets 2024-09-11T01:32:39.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f100effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:39.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f100effd640 1 -- 172.21.15.12:0/468936265 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1004032440 con 0x7f101009d030 2024-09-11T01:32:39.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f10165f8640 5 monclient: authenticate success, global_id 24303 2024-09-11T01:32:39.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f10165f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:32:39.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f100effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:39.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f10165f8640 10 monclient: get_monmap_and_config success 2024-09-11T01:32:39.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f100effd640 1 -- 172.21.15.12:0/468936265 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1004032810 con 0x7f101009d030 2024-09-11T01:32:39.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f10165f8640 4 set_mon_vals no callback set 2024-09-11T01:32:39.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f100effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:39.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f100effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:39.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f100effd640 10 monclient: dump: 2024-09-11T01:32:39.490 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:39.491 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:39.491 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:39.491 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:39.491 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:39.491 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:39.491 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:39.491 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:39.491 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:39.491 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:39.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f10165f8640 10 monclient: shutdown 2024-09-11T01:32:39.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f10165f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:39.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f10165f8640 1 -- 172.21.15.12:0/468936265 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f101009d030 msgr2=0x7f10100a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:39.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f10165f8640 1 --2- 172.21.15.12:0/468936265 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f101009d030 0x7f10100a5020 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f101006d650 tx=0x7f1004029fb0 comp rx=0 tx=0).stop 2024-09-11T01:32:39.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f10165f8640 1 -- 172.21.15.12:0/468936265 shutdown_connections 2024-09-11T01:32:39.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f10165f8640 1 --2- 172.21.15.12:0/468936265 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f101009d030 0x7f10100a5020 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:39.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f10165f8640 1 --2- 172.21.15.12:0/468936265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1010138230 0x7f1010138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:39.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f10165f8640 1 -- 172.21.15.12:0/468936265 >> 172.21.15.12:0/468936265 conn(0x7f10100a7120 msgr2=0x7f10100a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:39.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f10165f8640 1 -- 172.21.15.12:0/468936265 shutdown_connections 2024-09-11T01:32:39.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.480+0000 7f10165f8640 1 -- 172.21.15.12:0/468936265 wait complete. 2024-09-11T01:32:39.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.481+0000 7f10165f8640 10 monclient: build_initial_monmap 2024-09-11T01:32:39.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.481+0000 7f10165f8640 10 monclient: monmap: 2024-09-11T01:32:39.492 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:39.492 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:32:39.492 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:32:39.492 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:32:39.492 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:39.492 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:39.493 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:32:39.493 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:32:39.493 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:32:39.493 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:39.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.481+0000 7f10165f8640 1 Processor -- start 2024-09-11T01:32:39.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.481+0000 7f10165f8640 1 -- start start 2024-09-11T01:32:39.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.481+0000 7f10165f8640 10 monclient: init 2024-09-11T01:32:39.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f10165f8640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:39.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f10165f8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:32:39.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f10165f8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f101009d030 0x7f10101cd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:39.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f10165f8640 10 monclient(hunting): picked mon.noname-b con 0x7f101009d030 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:32:39.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f10165f8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f101009d430 0x7f101013f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:39.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f10165f8640 10 monclient(hunting): picked mon.noname-c con 0x7f101009d430 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:39.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f10165f8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1010138230 0x7f101013f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:39.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f10165f8640 10 monclient(hunting): picked mon.noname-a con 0x7f1010138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:32:39.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f10165f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:39.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f10165f8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f101006f970 con 0x7f1010138230 2024-09-11T01:32:39.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f10165f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:39.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f10165f8640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f101006f7f0 con 0x7f101009d030 2024-09-11T01:32:39.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f10165f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:39.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f10165f8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f10100a5c70 con 0x7f101009d430 2024-09-11T01:32:39.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f10165f8640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:39.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f10165f8640 10 monclient(hunting): authenticate will time out at 965.924422s 2024-09-11T01:32:39.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f1014b95640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1010138230 0x7f101013f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:39.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f100f7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f101009d430 0x7f101013f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:39.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f1014b95640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1010138230 0x7f101013f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50302/0 (socket says 172.21.15.12:50302) 2024-09-11T01:32:39.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f1014b95640 1 -- 172.21.15.12:0/2727692185 learned_addr learned my addr 172.21.15.12:0/2727692185 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:39.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f100f7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f101009d430 0x7f101013f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:58648/0 (socket says 172.21.15.12:58648) 2024-09-11T01:32:39.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f1014b95640 10 monclient(hunting): get_auth_request con 0x7f1010138230 auth_method 0 2024-09-11T01:32:39.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f1014b95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:39.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f1014b95640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:39.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f1014b95640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:39.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f100f7fe640 10 monclient(hunting): get_auth_request con 0x7f101009d430 auth_method 0 2024-09-11T01:32:39.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f100f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:39.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f100f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:39.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f100f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:39.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f100ffff640 1 --2- 172.21.15.12:0/2727692185 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f101009d030 0x7f10101cd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:39.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f100ffff640 10 monclient(hunting): get_auth_request con 0x7f101009d030 auth_method 0 2024-09-11T01:32:39.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f100ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:39.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f100ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:39.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f100ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:39.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f1014b95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:39.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f1014b95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:39.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f1014b95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:39.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f100f7fe640 1 -- 172.21.15.12:0/2727692185 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f101009d430 msgr2=0x7f101013f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:32:39.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f100f7fe640 1 -- 172.21.15.12:0/2727692185 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f101009d430 msgr2=0x7f101013f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:32:39.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f100f7fe640 1 --2- 172.21.15.12:0/2727692185 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f101009d430 0x7f101013f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:39.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f100f7fe640 1 --2- 172.21.15.12:0/2727692185 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f101009d430 0x7f101013f360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:39.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f100ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:39.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f100ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:39.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f100ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:39.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f1014b95640 10 monclient(hunting): handle_auth_done global_id 24306 payload 824 2024-09-11T01:32:39.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f1014b95640 1 -- 172.21.15.12:0/2727692185 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f101009d430 msgr2=0x7f101013f360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:32:39.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f1014b95640 1 --2- 172.21.15.12:0/2727692185 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f101009d430 0x7f101013f360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:39.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f1014b95640 1 -- 172.21.15.12:0/2727692185 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f101009d030 msgr2=0x7f10101cd4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:39.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f1014b95640 1 --2- 172.21.15.12:0/2727692185 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f101009d030 0x7f10101cd4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:39.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f1014b95640 10 monclient: _finish_hunting 0 2024-09-11T01:32:39.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f1014b95640 1 monclient: found mon.noname-a 2024-09-11T01:32:39.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f1014b95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:39.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f1014b95640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:32:39.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.482+0000 7f1014b95640 1 -- 172.21.15.12:0/2727692185 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10100a1810 con 0x7f1010138230 2024-09-11T01:32:39.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.485+0000 7f1014b95640 1 -- 172.21.15.12:0/2727692185 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1010138230 msgr2=0x7f101013f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:39.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.485+0000 7f1014b95640 1 --2- 172.21.15.12:0/2727692185 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1010138230 0x7f101013f8a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f100800caf0 tx=0x7f100800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:39.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.485+0000 7f100cff9640 1 -- 172.21.15.12:0/2727692185 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100800ae50 con 0x7f1010138230 2024-09-11T01:32:39.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.485+0000 7f100cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:39.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.485+0000 7f100cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:32:39.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.485+0000 7f100cff9640 10 monclient: dump: 2024-09-11T01:32:39.498 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:39.498 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:39.498 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:39.498 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:39.498 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:39.498 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:39.498 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:39.498 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:39.498 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:39.498 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:39.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.485+0000 7f100cff9640 10 monclient: _finish_auth 0 2024-09-11T01:32:39.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.485+0000 7f100cff9640 10 monclient: _check_auth_tickets 2024-09-11T01:32:39.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.485+0000 7f100cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:39.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.485+0000 7f100cff9640 1 -- 172.21.15.12:0/2727692185 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1008012400 con 0x7f1010138230 2024-09-11T01:32:39.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.485+0000 7f100cff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:39.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.485+0000 7f100cff9640 1 -- 172.21.15.12:0/2727692185 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10080127d0 con 0x7f1010138230 2024-09-11T01:32:39.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.485+0000 7f10165f8640 5 monclient: authenticate success, global_id 24306 2024-09-11T01:32:39.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.485+0000 7f10165f8640 10 monclient: _renew_subs 2024-09-11T01:32:39.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.485+0000 7f10165f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:39.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.485+0000 7f10165f8640 1 -- 172.21.15.12:0/2727692185 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f101013bad0 con 0x7f1010138230 2024-09-11T01:32:39.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.485+0000 7f100cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:39.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.485+0000 7f100cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:39.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.485+0000 7f100cff9640 10 monclient: dump: 2024-09-11T01:32:39.499 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:39.499 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:39.499 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:39.499 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:39.499 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:39.500 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:39.500 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:39.500 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:39.500 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:39.500 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:39.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.485+0000 7f100e7fc640 4 set_mon_vals no callback set 2024-09-11T01:32:39.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.485+0000 7f10165f8640 10 monclient: _renew_subs 2024-09-11T01:32:39.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.485+0000 7f10165f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:39.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.485+0000 7f10165f8640 1 -- 172.21.15.12:0/2727692185 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f101013d590 con 0x7f1010138230 2024-09-11T01:32:39.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.486+0000 7f100cff9640 1 -- 172.21.15.12:0/2727692185 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f10080129f0 con 0x7f1010138230 2024-09-11T01:32:39.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.486+0000 7f10165f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:39.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.486+0000 7f10165f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:39.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.486+0000 7f10165f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:39.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.486+0000 7f10165f8640 1 -- 172.21.15.12:0/2727692185 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1010138610 con 0x7f1010138230 2024-09-11T01:32:39.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.486+0000 7f100cff9640 1 --2- 172.21.15.12:0/2727692185 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0ffc03e430 0x7f0ffc0408f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:39.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.486+0000 7f100cff9640 1 -- 172.21.15.12:0/2727692185 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 4783+0+0 (secure 0 0 0) 0x7f1008053350 con 0x7f1010138230 2024-09-11T01:32:39.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.486+0000 7f100ffff640 1 --2- 172.21.15.12:0/2727692185 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0ffc03e430 0x7f0ffc0408f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:39.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.486+0000 7f100ffff640 10 monclient: get_auth_request con 0x7f0ffc03e430 auth_method 0 2024-09-11T01:32:39.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.489+0000 7f100ffff640 1 --2- 172.21.15.12:0/2727692185 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0ffc03e430 0x7f0ffc0408f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1000006fd0 tx=0x7f1000008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:39.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.491+0000 7f100cff9640 1 -- 172.21.15.12:0/2727692185 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1008051d50 con 0x7f1010138230 2024-09-11T01:32:39.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.491+0000 7f100cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:39.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.491+0000 7f100cff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:32:39.570 INFO:tasks.rados.rados.0.smithi071.stdout:566: finishing write tid 1 to smithi07134777-27 2024-09-11T01:32:39.570 INFO:tasks.rados.rados.0.smithi071.stdout:566: finishing write tid 2 to smithi07134777-27 2024-09-11T01:32:39.570 INFO:tasks.rados.rados.0.smithi071.stdout:566: finishing write tid 3 to smithi07134777-27 2024-09-11T01:32:39.570 INFO:tasks.rados.rados.0.smithi071.stdout:566: finishing write tid 4 to smithi07134777-27 2024-09-11T01:32:39.570 INFO:tasks.rados.rados.0.smithi071.stdout:566: finishing write tid 5 to smithi07134777-27 2024-09-11T01:32:39.571 INFO:tasks.rados.rados.0.smithi071.stdout:566: oid 27 updating version 0 to 185 2024-09-11T01:32:39.571 INFO:tasks.rados.rados.0.smithi071.stdout:566: oid 27 updating version 185 to 188 2024-09-11T01:32:39.571 INFO:tasks.rados.rados.0.smithi071.stdout:566: oid 27 version 188 is already newer than 187 2024-09-11T01:32:39.571 INFO:tasks.rados.rados.0.smithi071.stdout:566: oid 27 version 188 is already newer than 186 2024-09-11T01:32:39.571 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 188 (ObjNum 220 snap 54 seq_num 220) dirty exists 2024-09-11T01:32:39.571 INFO:tasks.rados.rados.0.smithi071.stdout:566: left oid 27 (ObjNum 220 snap 54 seq_num 220) 2024-09-11T01:32:39.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.746+0000 7f10165f8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:32:39.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.746+0000 7f10165f8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:32:39.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.746+0000 7f10165f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:39.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.746+0000 7f10165f8640 1 -- 172.21.15.12:0/2727692185 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f10100aa090 con 0x7f1010138230 2024-09-11T01:32:39.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.746+0000 7f100cff9640 1 -- 172.21.15.12:0/2727692185 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f1008053080 con 0x7f1010138230 2024-09-11T01:32:39.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.746+0000 7f100cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:32:39.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.746+0000 7f100cff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:32:39.749 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":32,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:32:39.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.748+0000 7f10165f8640 1 -- 172.21.15.12:0/2727692185 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0ffc03e430 msgr2=0x7f0ffc0408f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:39.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.748+0000 7f10165f8640 1 --2- 172.21.15.12:0/2727692185 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0ffc03e430 0x7f0ffc0408f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1000006fd0 tx=0x7f1000008040 comp rx=0 tx=0).stop 2024-09-11T01:32:39.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.748+0000 7f10165f8640 10 monclient: shutdown 2024-09-11T01:32:39.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.748+0000 7f10165f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:39.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.748+0000 7f10165f8640 1 -- 172.21.15.12:0/2727692185 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1010138230 msgr2=0x7f101013f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:39.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.748+0000 7f10165f8640 1 --2- 172.21.15.12:0/2727692185 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1010138230 0x7f101013f8a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f100800caf0 tx=0x7f100800a060 comp rx=0 tx=0).stop 2024-09-11T01:32:39.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.749+0000 7f10165f8640 1 -- 172.21.15.12:0/2727692185 shutdown_connections 2024-09-11T01:32:39.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.749+0000 7f10165f8640 1 --2- 172.21.15.12:0/2727692185 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0ffc03e430 0x7f0ffc0408f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:39.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.749+0000 7f10165f8640 1 --2- 172.21.15.12:0/2727692185 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1010138230 0x7f101013f8a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:39.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.749+0000 7f10165f8640 1 --2- 172.21.15.12:0/2727692185 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f101009d430 0x7f101013f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:39.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.749+0000 7f10165f8640 1 --2- 172.21.15.12:0/2727692185 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f101009d030 0x7f10101cd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:39.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.749+0000 7f10165f8640 1 -- 172.21.15.12:0/2727692185 >> 172.21.15.12:0/2727692185 conn(0x7f10100a7120 msgr2=0x7f10100a3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:39.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.749+0000 7f10165f8640 1 -- 172.21.15.12:0/2727692185 shutdown_connections 2024-09-11T01:32:39.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:39.749+0000 7f10165f8640 1 -- 172.21.15.12:0/2727692185 wait complete. 2024-09-11T01:32:40.087 INFO:tasks.rados.rados.0.smithi071.stdout:560: done (4 left) 2024-09-11T01:32:40.087 INFO:tasks.rados.rados.0.smithi071.stdout:562: done (3 left) 2024-09-11T01:32:40.087 INFO:tasks.rados.rados.0.smithi071.stdout:565: done (2 left) 2024-09-11T01:32:40.087 INFO:tasks.rados.rados.0.smithi071.stdout:566: done (1 left) 2024-09-11T01:32:40.087 INFO:tasks.rados.rados.0.smithi071.stdout:567: done (0 left) 2024-09-11T01:32:40.087 INFO:tasks.rados.rados.0.smithi071.stdout:568: read oid 36 snap -1 2024-09-11T01:32:40.087 INFO:tasks.rados.rados.0.smithi071.stdout:568: expect (ObjNum 214 snap 52 seq_num 214) 2024-09-11T01:32:40.088 INFO:tasks.rados.rados.0.smithi071.stdout:569: snap_remove snap 52 2024-09-11T01:32:41.093 INFO:tasks.rados.rados.0.smithi071.stdout:569: done (1 left) 2024-09-11T01:32:41.093 INFO:tasks.rados.rados.0.smithi071.stdout:570: read oid 45 snap -1 2024-09-11T01:32:41.093 INFO:tasks.rados.rados.0.smithi071.stdout:570: expect (ObjNum 119 snap 28 seq_num 119) 2024-09-11T01:32:41.093 INFO:tasks.rados.rados.0.smithi071.stdout:571: snap_remove snap 50 2024-09-11T01:32:42.762 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:32:42.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.873+0000 7f956d8a9640 10 monclient: get_monmap_and_config 2024-09-11T01:32:42.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.873+0000 7f956d8a9640 10 monclient: build_initial_monmap 2024-09-11T01:32:42.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.873+0000 7f956d8a9640 10 monclient: monmap: 2024-09-11T01:32:42.875 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:42.875 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:42.875 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:32:42.874269+0000 2024-09-11T01:32:42.875 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:32:42.874269+0000 2024-09-11T01:32:42.876 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:42.876 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:42.876 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:32:42.876 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:32:42.876 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:32:42.876 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:32:42.876 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:32:42.876 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:42.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.874+0000 7f956d8a9640 1 Processor -- start 2024-09-11T01:32:42.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.874+0000 7f956d8a9640 1 -- start start 2024-09-11T01:32:42.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.874+0000 7f956d8a9640 10 monclient: init 2024-09-11T01:32:42.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.874+0000 7f956d8a9640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:42.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.874+0000 7f956d8a9640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-11T01:32:42.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.874+0000 7f956d8a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f95680a2e90 addr v1:172.21.15.71:6789/0 2024-09-11T01:32:42.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.874+0000 7f956d8a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f95680a3270 addr v1:172.21.15.12:6789/0 2024-09-11T01:32:42.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.874+0000 7f956d8a9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f956813e4a0 0x7f9568140890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:42.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.874+0000 7f956d8a9640 10 monclient(hunting): picked mon.noname-b con 0x7f956813e4a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:42.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.874+0000 7f956d8a9640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95680a46c0 con 0x7f95680a3270 2024-09-11T01:32:42.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.874+0000 7f956d8a9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f956806ee50 con 0x7f95680a2e90 2024-09-11T01:32:42.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.874+0000 7f956d8a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:42.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.874+0000 7f956d8a9640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f95680a4900 con 0x7f956813e4a0 2024-09-11T01:32:42.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.874+0000 7f956d8a9640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:42.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.874+0000 7f956d8a9640 10 monclient(hunting): authenticate will time out at 969.316988s 2024-09-11T01:32:42.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.875+0000 7f955ffff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f95680a3270 0x7f95680a3680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47638/0 (socket says 172.21.15.12:47638) 2024-09-11T01:32:42.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.875+0000 7f955ffff640 1 -- 172.21.15.12:0/13941063 learned_addr learned my addr 172.21.15.12:0/13941063 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:32:42.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.875+0000 7f95677fe640 1 --2- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f956813e4a0 0x7f9568140890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:42.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.875+0000 7f95677fe640 10 monclient(hunting): get_auth_request con 0x7f956813e4a0 auth_method 0 2024-09-11T01:32:42.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.875+0000 7f95677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:42.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.875+0000 7f95677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:42.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.875+0000 7f95677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:42.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.875+0000 7f95677fe640 1 -- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f956813e4a0 msgr2=0x7f9568140890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:32:42.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.875+0000 7f95677fe640 1 -- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f956813e4a0 msgr2=0x7f9568140890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:32:42.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.875+0000 7f95677fe640 1 --2- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f956813e4a0 0x7f9568140890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:42.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.875+0000 7f95677fe640 1 --2- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f956813e4a0 0x7f9568140890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:42.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.877+0000 7f955ffff640 1 -- 172.21.15.12:0/13941063 >> v1:172.21.15.12:6789/0 conn(0x7f95680a3270 legacy=0x7f95680a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:42.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:42.877+0000 7f9566ffd640 1 -- 172.21.15.12:0/13941063 >> v1:172.21.15.71:6789/0 conn(0x7f95680a2e90 legacy=0x7f956806dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:43.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:43.076+0000 7f95677fe640 1 --2- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f956813e4a0 0x7f9568140890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:43.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:43.076+0000 7f95677fe640 10 monclient(hunting): get_auth_request con 0x7f956813e4a0 auth_method 0 2024-09-11T01:32:43.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:43.076+0000 7f95677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:43.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:43.076+0000 7f95677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:43.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:43.076+0000 7f95677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:32:43.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:43.076+0000 7f95677fe640 1 -- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f956813e4a0 msgr2=0x7f9568140890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:32:43.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:43.076+0000 7f95677fe640 1 -- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f956813e4a0 msgr2=0x7f9568140890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:32:43.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:43.077+0000 7f95677fe640 1 --2- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f956813e4a0 0x7f9568140890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:43.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:43.077+0000 7f95677fe640 1 --2- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f956813e4a0 0x7f9568140890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:32:43.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:43.478+0000 7f95677fe640 1 --2- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f956813e4a0 0x7f9568140890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:43.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:43.478+0000 7f95677fe640 10 monclient(hunting): get_auth_request con 0x7f956813e4a0 auth_method 0 2024-09-11T01:32:43.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:43.478+0000 7f95677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:43.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:43.478+0000 7f95677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:43.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:43.478+0000 7f95677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:32:43.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:43.478+0000 7f95677fe640 1 -- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f956813e4a0 msgr2=0x7f9568140890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:32:43.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:43.478+0000 7f95677fe640 1 -- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f956813e4a0 msgr2=0x7f9568140890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:32:43.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:43.478+0000 7f95677fe640 1 --2- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f956813e4a0 0x7f9568140890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:43.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:43.478+0000 7f95677fe640 1 --2- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f956813e4a0 0x7f9568140890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:32:44.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:44.279+0000 7f95677fe640 1 --2- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f956813e4a0 0x7f9568140890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:44.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:44.279+0000 7f95677fe640 10 monclient(hunting): get_auth_request con 0x7f956813e4a0 auth_method 0 2024-09-11T01:32:44.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:44.279+0000 7f95677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:44.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:44.279+0000 7f95677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:44.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:44.279+0000 7f95677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:32:44.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:44.279+0000 7f95677fe640 1 -- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f956813e4a0 msgr2=0x7f9568140890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:32:44.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:44.279+0000 7f95677fe640 1 -- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f956813e4a0 msgr2=0x7f9568140890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:32:44.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:44.279+0000 7f95677fe640 1 --2- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f956813e4a0 0x7f9568140890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:44.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:44.279+0000 7f95677fe640 1 --2- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f956813e4a0 0x7f9568140890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:32:45.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f95657fa640 10 monclient(hunting): tick 2024-09-11T01:32:45.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f95657fa640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:32:45.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f95657fa640 1 monclient(hunting): continuing hunt 2024-09-11T01:32:45.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f95657fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:32:45.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f95657fa640 1 -- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f956813e4a0 msgr2=0x7f9568140890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:32:45.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f95657fa640 1 --2- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f956813e4a0 0x7f9568140890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:45.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f95657fa640 1 -- 172.21.15.12:0/13941063 >> v1:172.21.15.71:6789/0 conn(0x7f95680a2e90 legacy=0x7f956806dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:45.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f95657fa640 1 -- 172.21.15.12:0/13941063 >> v1:172.21.15.12:6789/0 conn(0x7f95680a3270 legacy=0x7f95680a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:45.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f95657fa640 10 monclient: _add_conns ranks=[4,3] 2024-09-11T01:32:45.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f95657fa640 1 --2- 172.21.15.12:0/13941063 >> v2:172.21.15.71:3300/0 conn(0x7f9548003810 0x7f9548005c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:45.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f95657fa640 10 monclient(hunting): picked mon.noname-c con 0x7f9548003810 addr v2:172.21.15.71:3300/0 2024-09-11T01:32:45.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f95657fa640 1 --2- 172.21.15.12:0/13941063 >> v2:172.21.15.12:3300/0 conn(0x7f9548006360 0x7f9548008820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:45.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f95657fa640 10 monclient(hunting): picked mon.noname-a con 0x7f9548006360 addr v2:172.21.15.12:3300/0 2024-09-11T01:32:45.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f95657fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:45.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f95657fa640 1 -- 172.21.15.12:0/13941063 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f95680a4900 con 0x7f9548006360 2024-09-11T01:32:45.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f95657fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:45.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f95657fa640 1 -- 172.21.15.12:0/13941063 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f9548008ee0 con 0x7f9548003810 2024-09-11T01:32:45.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f95657fa640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:45.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f955ffff640 1 --2- 172.21.15.12:0/13941063 >> v2:172.21.15.12:3300/0 conn(0x7f9548006360 0x7f9548008820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:45.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f955ffff640 10 monclient(hunting): get_auth_request con 0x7f9548006360 auth_method 0 2024-09-11T01:32:45.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f955ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:45.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f955ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:45.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f955ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:45.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f9566ffd640 1 --2- 172.21.15.12:0/13941063 >> v2:172.21.15.71:3300/0 conn(0x7f9548003810 0x7f9548005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:45.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f955ffff640 1 -- 172.21.15.12:0/13941063 >> v2:172.21.15.12:3300/0 conn(0x7f9548006360 msgr2=0x7f9548008820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:32:45.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f955ffff640 1 -- 172.21.15.12:0/13941063 >> v2:172.21.15.12:3300/0 conn(0x7f9548006360 msgr2=0x7f9548008820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:32:45.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f9566ffd640 10 monclient(hunting): get_auth_request con 0x7f9548003810 auth_method 0 2024-09-11T01:32:45.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f955ffff640 1 --2- 172.21.15.12:0/13941063 >> v2:172.21.15.12:3300/0 conn(0x7f9548006360 0x7f9548008820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:45.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f9566ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:45.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f9566ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:45.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f9566ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:45.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f955ffff640 1 --2- 172.21.15.12:0/13941063 >> v2:172.21.15.12:3300/0 conn(0x7f9548006360 0x7f9548008820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:45.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f9566ffd640 1 -- 172.21.15.12:0/13941063 >> v2:172.21.15.71:3300/0 conn(0x7f9548003810 msgr2=0x7f9548005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:32:45.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f9566ffd640 1 -- 172.21.15.12:0/13941063 >> v2:172.21.15.71:3300/0 conn(0x7f9548003810 msgr2=0x7f9548005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:32:45.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f9566ffd640 1 --2- 172.21.15.12:0/13941063 >> v2:172.21.15.71:3300/0 conn(0x7f9548003810 0x7f9548005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:45.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:45.875+0000 7f9566ffd640 1 --2- 172.21.15.12:0/13941063 >> v2:172.21.15.71:3300/0 conn(0x7f9548003810 0x7f9548005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:46.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.076+0000 7f955ffff640 1 --2- 172.21.15.12:0/13941063 >> v2:172.21.15.12:3300/0 conn(0x7f9548006360 0x7f9548008820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:46.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.076+0000 7f955ffff640 10 monclient(hunting): get_auth_request con 0x7f9548006360 auth_method 0 2024-09-11T01:32:46.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.076+0000 7f955ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:46.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.076+0000 7f955ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:46.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.076+0000 7f955ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:32:46.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.076+0000 7f955ffff640 1 -- 172.21.15.12:0/13941063 >> v2:172.21.15.12:3300/0 conn(0x7f9548006360 msgr2=0x7f9548008820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:32:46.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.076+0000 7f955ffff640 1 -- 172.21.15.12:0/13941063 >> v2:172.21.15.12:3300/0 conn(0x7f9548006360 msgr2=0x7f9548008820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:32:46.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.076+0000 7f955ffff640 1 --2- 172.21.15.12:0/13941063 >> v2:172.21.15.12:3300/0 conn(0x7f9548006360 0x7f9548008820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:46.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.076+0000 7f955ffff640 1 --2- 172.21.15.12:0/13941063 >> v2:172.21.15.12:3300/0 conn(0x7f9548006360 0x7f9548008820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:32:46.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.076+0000 7f9566ffd640 1 --2- 172.21.15.12:0/13941063 >> v2:172.21.15.71:3300/0 conn(0x7f9548003810 0x7f9548005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:46.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.076+0000 7f9566ffd640 10 monclient(hunting): get_auth_request con 0x7f9548003810 auth_method 0 2024-09-11T01:32:46.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.076+0000 7f9566ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:46.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.076+0000 7f9566ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:46.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.076+0000 7f9566ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:32:46.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.076+0000 7f9566ffd640 1 -- 172.21.15.12:0/13941063 >> v2:172.21.15.71:3300/0 conn(0x7f9548003810 msgr2=0x7f9548005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:32:46.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.076+0000 7f9566ffd640 1 -- 172.21.15.12:0/13941063 >> v2:172.21.15.71:3300/0 conn(0x7f9548003810 msgr2=0x7f9548005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:32:46.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.076+0000 7f9566ffd640 1 --2- 172.21.15.12:0/13941063 >> v2:172.21.15.71:3300/0 conn(0x7f9548003810 0x7f9548005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:46.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.076+0000 7f9566ffd640 1 --2- 172.21.15.12:0/13941063 >> v2:172.21.15.71:3300/0 conn(0x7f9548003810 0x7f9548005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:32:46.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.477+0000 7f955ffff640 1 --2- 172.21.15.12:0/13941063 >> v2:172.21.15.12:3300/0 conn(0x7f9548006360 0x7f9548008820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:46.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.477+0000 7f955ffff640 10 monclient(hunting): get_auth_request con 0x7f9548006360 auth_method 0 2024-09-11T01:32:46.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.477+0000 7f955ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:46.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.477+0000 7f955ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:46.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.477+0000 7f955ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:32:46.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.477+0000 7f955ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:46.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.477+0000 7f955ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:46.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.477+0000 7f955ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:46.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.477+0000 7f9566ffd640 1 --2- 172.21.15.12:0/13941063 >> v2:172.21.15.71:3300/0 conn(0x7f9548003810 0x7f9548005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:46.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.477+0000 7f955ffff640 10 monclient(hunting): handle_auth_done global_id 24309 payload 274 2024-09-11T01:32:46.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.477+0000 7f955ffff640 1 -- 172.21.15.12:0/13941063 >> v2:172.21.15.71:3300/0 conn(0x7f9548003810 msgr2=0x7f9548005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:46.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.477+0000 7f955ffff640 1 --2- 172.21.15.12:0/13941063 >> v2:172.21.15.71:3300/0 conn(0x7f9548003810 0x7f9548005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:46.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.477+0000 7f955ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:32:46.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.477+0000 7f955ffff640 1 monclient: found mon.noname-a 2024-09-11T01:32:46.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.477+0000 7f955ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:46.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.477+0000 7f955ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:32:46.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.477+0000 7f955ffff640 1 -- 172.21.15.12:0/13941063 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f956806eb70 con 0x7f9548006360 2024-09-11T01:32:46.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.477+0000 7f9566ffd640 10 monclient: get_auth_request con 0x7f9548003810 auth_method 0 2024-09-11T01:32:46.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.477+0000 7f9566ffd640 1 --2- 172.21.15.12:0/13941063 >> v2:172.21.15.71:3300/0 conn(0x7f9548003810 0x7f9548005c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:32:46.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.480+0000 7f955ffff640 1 -- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9548006360 msgr2=0x7f9548008820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:46.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.480+0000 7f955ffff640 1 --2- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9548006360 0x7f9548008820 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f95480037a0 tx=0x7f955002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fc32f4e2fa14ad6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:46.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.480+0000 7f95667fc640 1 -- 172.21.15.12:0/13941063 DispatchQueue::entry inject delay of 0.002000 2024-09-11T01:32:46.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.482+0000 7f95667fc640 1 -- 172.21.15.12:0/13941063 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9550004b60 con 0x7f9548006360 2024-09-11T01:32:46.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.482+0000 7f95667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:46.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.482+0000 7f95667fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:32:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.482+0000 7f95667fc640 10 monclient: dump: 2024-09-11T01:32:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:46.485 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.482+0000 7f95667fc640 10 monclient: _finish_auth 0 2024-09-11T01:32:46.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.482+0000 7f95667fc640 10 monclient: _check_auth_tickets 2024-09-11T01:32:46.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.482+0000 7f95667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:46.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.482+0000 7f95667fc640 1 -- 172.21.15.12:0/13941063 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9550004d00 con 0x7f9548006360 2024-09-11T01:32:46.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.482+0000 7f95667fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:46.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.482+0000 7f95667fc640 1 -- 172.21.15.12:0/13941063 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9550005290 con 0x7f9548006360 2024-09-11T01:32:46.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.482+0000 7f95667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:46.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.482+0000 7f95667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:46.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.482+0000 7f95667fc640 10 monclient: dump: 2024-09-11T01:32:46.486 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:46.486 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:46.486 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:46.486 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:46.486 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:46.486 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:46.486 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:46.487 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:46.487 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:46.487 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:46.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.482+0000 7f956d8a9640 5 monclient: authenticate success, global_id 24309 2024-09-11T01:32:46.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.482+0000 7f956d8a9640 10 monclient: get_monmap_and_config success 2024-09-11T01:32:46.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.482+0000 7f956d8a9640 4 set_mon_vals no callback set 2024-09-11T01:32:46.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.483+0000 7f956d8a9640 10 monclient: shutdown 2024-09-11T01:32:46.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.483+0000 7f956d8a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:46.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.483+0000 7f956d8a9640 1 -- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9548006360 msgr2=0x7f9548008820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:46.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.483+0000 7f956d8a9640 1 --2- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9548006360 0x7f9548008820 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f95480037a0 tx=0x7f955002a6a0 comp rx=0 tx=0).stop 2024-09-11T01:32:46.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.483+0000 7f95677fe640 1 -- 172.21.15.12:0/13941063 reap_dead start 2024-09-11T01:32:46.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.483+0000 7f956d8a9640 1 -- 172.21.15.12:0/13941063 shutdown_connections 2024-09-11T01:32:46.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.483+0000 7f956d8a9640 1 --2- 172.21.15.12:0/13941063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9548006360 0x7f9548008820 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:46.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.483+0000 7f956d8a9640 1 -- 172.21.15.12:0/13941063 >> 172.21.15.12:0/13941063 conn(0x7f95680a8490 msgr2=0x7f95680a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:46.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.483+0000 7f956d8a9640 1 -- 172.21.15.12:0/13941063 shutdown_connections 2024-09-11T01:32:46.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.483+0000 7f956d8a9640 1 -- 172.21.15.12:0/13941063 wait complete. 2024-09-11T01:32:46.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.484+0000 7f956d8a9640 10 monclient: build_initial_monmap 2024-09-11T01:32:46.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.484+0000 7f956d8a9640 10 monclient: monmap: 2024-09-11T01:32:46.493 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:46.493 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:32:46.493 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:32:46.493 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:32:46.493 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:46.493 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:46.494 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:32:46.494 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:32:46.494 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:32:46.494 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:46.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.484+0000 7f956d8a9640 1 Processor -- start 2024-09-11T01:32:46.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.484+0000 7f956d8a9640 1 -- start start 2024-09-11T01:32:46.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.484+0000 7f956d8a9640 10 monclient: init 2024-09-11T01:32:46.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f956d8a9640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:46.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f956d8a9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:32:46.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f956d8a9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9548006360 0x7f95681cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:46.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f956d8a9640 10 monclient(hunting): picked mon.noname-b con 0x7f9548006360 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:32:46.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f956d8a9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95681cd7e0 0x7f95681cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:46.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f956d8a9640 10 monclient(hunting): picked mon.noname-a con 0x7f95681cd7e0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:32:46.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f956d8a9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9568138160 0x7f95681385f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:46.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f956d8a9640 10 monclient(hunting): picked mon.noname-c con 0x7f9568138160 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:46.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f956d8a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:46.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f956d8a9640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9568140f50 con 0x7f95681cd7e0 2024-09-11T01:32:46.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f956d8a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:46.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f956d8a9640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f9568140dd0 con 0x7f9548006360 2024-09-11T01:32:46.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f956d8a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:46.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f956d8a9640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f95681410d0 con 0x7f9568138160 2024-09-11T01:32:46.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f956d8a9640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:46.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f956d8a9640 10 monclient(hunting): authenticate will time out at 972.927528s 2024-09-11T01:32:46.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f955ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95681cd7e0 0x7f95681cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:46.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f95677fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9568138160 0x7f95681385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:46.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f95677fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9568138160 0x7f95681385f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:39298/0 (socket says 172.21.15.12:39298) 2024-09-11T01:32:46.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f955ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95681cd7e0 0x7f95681cdc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42716/0 (socket says 172.21.15.12:42716) 2024-09-11T01:32:46.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f95677fe640 1 -- 172.21.15.12:0/1239070307 learned_addr learned my addr 172.21.15.12:0/1239070307 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:46.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f95677fe640 10 monclient(hunting): get_auth_request con 0x7f9568138160 auth_method 0 2024-09-11T01:32:46.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f95677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:46.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f95677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:46.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f95677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:46.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f955ffff640 10 monclient(hunting): get_auth_request con 0x7f95681cd7e0 auth_method 0 2024-09-11T01:32:46.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f955ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:46.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f955ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:46.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f955ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:46.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f9566ffd640 1 --2- 172.21.15.12:0/1239070307 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9548006360 0x7f95681cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:46.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f9566ffd640 10 monclient(hunting): get_auth_request con 0x7f9548006360 auth_method 0 2024-09-11T01:32:46.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f9566ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:46.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f9566ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:46.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f9566ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:46.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f955ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:46.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f955ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:46.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f95677fe640 1 -- 172.21.15.12:0/1239070307 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9568138160 msgr2=0x7f95681385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:32:46.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f955ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:46.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f95677fe640 1 -- 172.21.15.12:0/1239070307 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9568138160 msgr2=0x7f95681385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:32:46.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f95677fe640 1 --2- 172.21.15.12:0/1239070307 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9568138160 0x7f95681385f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:46.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f9566ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:46.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f9566ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:46.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f9566ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:46.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.485+0000 7f95677fe640 1 --2- 172.21.15.12:0/1239070307 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9568138160 0x7f95681385f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:46.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.486+0000 7f955ffff640 10 monclient(hunting): handle_auth_done global_id 24312 payload 824 2024-09-11T01:32:46.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.486+0000 7f955ffff640 1 -- 172.21.15.12:0/1239070307 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9568138160 msgr2=0x7f95681385f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:32:46.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.486+0000 7f955ffff640 1 --2- 172.21.15.12:0/1239070307 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9568138160 0x7f95681385f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:46.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.486+0000 7f955ffff640 1 -- 172.21.15.12:0/1239070307 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9548006360 msgr2=0x7f95681cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:46.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.486+0000 7f955ffff640 1 --2- 172.21.15.12:0/1239070307 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9548006360 0x7f95681cd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:46.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.486+0000 7f955ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:32:46.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.486+0000 7f955ffff640 1 monclient: found mon.noname-a 2024-09-11T01:32:46.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.486+0000 7f955ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:46.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.486+0000 7f955ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:32:46.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.486+0000 7f955ffff640 1 -- 172.21.15.12:0/1239070307 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9568139710 con 0x7f95681cd7e0 2024-09-11T01:32:46.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.488+0000 7f955ffff640 1 -- 172.21.15.12:0/1239070307 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95681cd7e0 msgr2=0x7f95681cdc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:46.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.488+0000 7f955ffff640 1 --2- 172.21.15.12:0/1239070307 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95681cd7e0 0x7f95681cdc60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9550034000 tx=0x7f95500014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:46.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.488+0000 7f9564ff9640 1 -- 172.21.15.12:0/1239070307 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9550042030 con 0x7f95681cd7e0 2024-09-11T01:32:46.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.488+0000 7f9564ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:46.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.488+0000 7f9564ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:32:46.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.488+0000 7f9564ff9640 10 monclient: dump: 2024-09-11T01:32:46.499 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:46.499 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:46.499 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:46.499 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:46.499 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:46.499 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:46.499 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:46.499 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:46.499 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:46.500 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:46.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.488+0000 7f9564ff9640 10 monclient: _finish_auth 0 2024-09-11T01:32:46.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.488+0000 7f9564ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:32:46.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.488+0000 7f9564ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:46.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.488+0000 7f9564ff9640 1 -- 172.21.15.12:0/1239070307 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f955003e670 con 0x7f95681cd7e0 2024-09-11T01:32:46.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.488+0000 7f9564ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:46.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.488+0000 7f9564ff9640 1 -- 172.21.15.12:0/1239070307 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955003e880 con 0x7f95681cd7e0 2024-09-11T01:32:46.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.488+0000 7f9564ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:46.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.488+0000 7f9564ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:46.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.488+0000 7f9564ff9640 10 monclient: dump: 2024-09-11T01:32:46.500 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:46.500 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:46.500 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:46.501 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:46.501 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:46.501 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:46.501 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:46.501 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:46.501 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:46.501 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:46.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.488+0000 7f956d8a9640 5 monclient: authenticate success, global_id 24312 2024-09-11T01:32:46.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.488+0000 7f956d8a9640 10 monclient: _renew_subs 2024-09-11T01:32:46.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.488+0000 7f956d8a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:46.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.488+0000 7f956d8a9640 1 -- 172.21.15.12:0/1239070307 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f956813b050 con 0x7f95681cd7e0 2024-09-11T01:32:46.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.488+0000 7f9565ffb640 4 set_mon_vals no callback set 2024-09-11T01:32:46.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.488+0000 7f956d8a9640 10 monclient: _renew_subs 2024-09-11T01:32:46.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.488+0000 7f956d8a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:46.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.488+0000 7f956d8a9640 1 -- 172.21.15.12:0/1239070307 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f956813db40 con 0x7f95681cd7e0 2024-09-11T01:32:46.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.489+0000 7f9564ff9640 1 -- 172.21.15.12:0/1239070307 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f9550032070 con 0x7f95681cd7e0 2024-09-11T01:32:46.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.490+0000 7f956d8a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:46.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.490+0000 7f956d8a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:46.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.490+0000 7f956d8a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:46.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.490+0000 7f956d8a9640 1 -- 172.21.15.12:0/1239070307 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95680a3680 con 0x7f95681cd7e0 2024-09-11T01:32:46.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.490+0000 7f9564ff9640 1 --2- 172.21.15.12:0/1239070307 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f954803f140 0x7f9548041600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:46.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.490+0000 7f9564ff9640 1 -- 172.21.15.12:0/1239070307 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 4767+0+0 (secure 0 0 0) 0x7f9550072dd0 con 0x7f95681cd7e0 2024-09-11T01:32:46.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.490+0000 7f9566ffd640 1 --2- 172.21.15.12:0/1239070307 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f954803f140 0x7f9548041600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:46.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.490+0000 7f9566ffd640 10 monclient: get_auth_request con 0x7f954803f140 auth_method 0 2024-09-11T01:32:46.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.493+0000 7f9566ffd640 1 --2- 172.21.15.12:0/1239070307 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f954803f140 0x7f9548041600 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9568073cc0 tx=0x7f9558009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:46.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.495+0000 7f9564ff9640 1 -- 172.21.15.12:0/1239070307 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9568139710 con 0x7f95681cd7e0 2024-09-11T01:32:46.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.495+0000 7f9564ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:46.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.495+0000 7f9564ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:32:46.515 INFO:tasks.rados.rados.0.smithi071.stdout:571: done (2 left) 2024-09-11T01:32:46.515 INFO:tasks.rados.rados.0.smithi071.stdout:572: read oid 18 snap 54 2024-09-11T01:32:46.515 INFO:tasks.rados.rados.0.smithi071.stdout:572: expect (ObjNum 171 snap 37 seq_num 171) 2024-09-11T01:32:46.515 INFO:tasks.rados.rados.0.smithi071.stdout:573: write oid 10 current snap is 55 2024-09-11T01:32:46.516 INFO:tasks.rados.rados.0.smithi071.stdout:573: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2024-09-11T01:32:46.520 INFO:tasks.rados.rados.0.smithi071.stdout:573: writing smithi07134777-10 from 514347 to 918613 tid 1 2024-09-11T01:32:46.527 INFO:tasks.rados.rados.0.smithi071.stdout:573: writing smithi07134777-10 from 1499344 to 2199415 tid 2 2024-09-11T01:32:46.534 INFO:tasks.rados.rados.0.smithi071.stdout:573: writing smithi07134777-10 from 2686246 to 3362302 tid 3 2024-09-11T01:32:46.534 INFO:tasks.rados.rados.0.smithi071.stdout:574: write (excl) oid 44 current snap is 55 2024-09-11T01:32:46.534 INFO:tasks.rados.rados.0.smithi071.stdout:574: seq_num 222 ranges {531154=479515,1557826=443729,2713669=487328,3679295=51859} 2024-09-11T01:32:46.539 INFO:tasks.rados.rados.0.smithi071.stdout:574: writing smithi07134777-44 from 531154 to 1010669 tid 1 2024-09-11T01:32:46.545 INFO:tasks.rados.rados.0.smithi071.stdout:574: writing smithi07134777-44 from 1557826 to 2001555 tid 2 2024-09-11T01:32:46.552 INFO:tasks.rados.rados.0.smithi071.stdout:574: writing smithi07134777-44 from 2713669 to 3200997 tid 3 2024-09-11T01:32:46.555 INFO:tasks.rados.rados.0.smithi071.stdout:574: writing smithi07134777-44 from 3679295 to 3731154 tid 4 2024-09-11T01:32:46.555 INFO:tasks.rados.rados.0.smithi071.stdout:575: rollback oid 25 current snap is 55 2024-09-11T01:32:46.555 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 25 to 49 2024-09-11T01:32:46.555 INFO:tasks.rados.rados.0.smithi071.stdout:568: expect (ObjNum 214 snap 52 seq_num 214) 2024-09-11T01:32:46.555 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:32:46.605 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:32:46.654 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:32:46.704 INFO:tasks.rados.rados.0.smithi071.stdout:570: expect (ObjNum 119 snap 28 seq_num 119) 2024-09-11T01:32:46.704 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:32:46.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.763+0000 7f956d8a9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:32:46.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.763+0000 7f956d8a9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:32:46.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.763+0000 7f956d8a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:46.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.763+0000 7f956d8a9640 1 -- 172.21.15.12:0/1239070307 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f95680698e0 con 0x7f95681cd7e0 2024-09-11T01:32:46.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.764+0000 7f9564ff9640 1 -- 172.21.15.12:0/1239070307 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f955004f620 con 0x7f95681cd7e0 2024-09-11T01:32:46.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.764+0000 7f9564ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:32:46.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.764+0000 7f9564ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:32:46.766 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":36,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:32:46.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.766+0000 7f956d8a9640 1 -- 172.21.15.12:0/1239070307 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f954803f140 msgr2=0x7f9548041600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:46.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.766+0000 7f956d8a9640 1 --2- 172.21.15.12:0/1239070307 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f954803f140 0x7f9548041600 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9568073cc0 tx=0x7f9558009040 comp rx=0 tx=0).stop 2024-09-11T01:32:46.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.766+0000 7f956d8a9640 10 monclient: shutdown 2024-09-11T01:32:46.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.766+0000 7f956d8a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:46.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.766+0000 7f956d8a9640 1 -- 172.21.15.12:0/1239070307 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95681cd7e0 msgr2=0x7f95681cdc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:46.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.766+0000 7f956d8a9640 1 --2- 172.21.15.12:0/1239070307 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95681cd7e0 0x7f95681cdc60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9550034000 tx=0x7f95500014a0 comp rx=0 tx=0).stop 2024-09-11T01:32:46.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.766+0000 7f956d8a9640 1 -- 172.21.15.12:0/1239070307 shutdown_connections 2024-09-11T01:32:46.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.766+0000 7f956d8a9640 1 --2- 172.21.15.12:0/1239070307 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9568138160 0x7f95681385f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:46.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.766+0000 7f956d8a9640 1 --2- 172.21.15.12:0/1239070307 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f954803f140 0x7f9548041600 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:46.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.766+0000 7f956d8a9640 1 --2- 172.21.15.12:0/1239070307 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95681cd7e0 0x7f95681cdc60 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:46.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.766+0000 7f956d8a9640 1 --2- 172.21.15.12:0/1239070307 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9548006360 0x7f95681cd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:46.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.766+0000 7f956d8a9640 1 -- 172.21.15.12:0/1239070307 >> 172.21.15.12:0/1239070307 conn(0x7f95680a8490 msgr2=0x7f95680a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:46.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.766+0000 7f956d8a9640 1 -- 172.21.15.12:0/1239070307 shutdown_connections 2024-09-11T01:32:46.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:46.766+0000 7f956d8a9640 1 -- 172.21.15.12:0/1239070307 wait complete. 2024-09-11T01:32:46.773 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:32:46.807 INFO:tasks.rados.rados.0.smithi071.stdout:573: finishing write tid 1 to smithi07134777-10 2024-09-11T01:32:46.808 INFO:tasks.rados.rados.0.smithi071.stdout:572: expect (ObjNum 171 snap 37 seq_num 171) 2024-09-11T01:32:46.808 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:32:46.863 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1673881 2024-09-11T01:32:46.925 INFO:tasks.rados.rados.0.smithi071.stdout:573: finishing write tid 2 to smithi07134777-10 2024-09-11T01:32:46.925 INFO:tasks.rados.rados.0.smithi071.stdout:573: finishing write tid 3 to smithi07134777-10 2024-09-11T01:32:46.925 INFO:tasks.rados.rados.0.smithi071.stdout:573: finishing write tid 4 to smithi07134777-10 2024-09-11T01:32:46.925 INFO:tasks.rados.rados.0.smithi071.stdout:573: finishing write tid 5 to smithi07134777-10 2024-09-11T01:32:46.925 INFO:tasks.rados.rados.0.smithi071.stdout:573: oid 10 updating version 0 to 116 2024-09-11T01:32:46.925 INFO:tasks.rados.rados.0.smithi071.stdout:573: oid 10 version 116 is already newer than 113 2024-09-11T01:32:46.926 INFO:tasks.rados.rados.0.smithi071.stdout:573: oid 10 version 116 is already newer than 115 2024-09-11T01:32:46.926 INFO:tasks.rados.rados.0.smithi071.stdout:573: oid 10 version 116 is already newer than 114 2024-09-11T01:32:46.926 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 116 (ObjNum 221 snap 55 seq_num 221) dirty exists 2024-09-11T01:32:46.926 INFO:tasks.rados.rados.0.smithi071.stdout:573: left oid 10 (ObjNum 221 snap 55 seq_num 221) 2024-09-11T01:32:46.926 INFO:tasks.rados.rados.0.smithi071.stdout:574: finishing write tid 1 to smithi07134777-44 2024-09-11T01:32:46.926 INFO:tasks.rados.rados.0.smithi071.stdout:568: done (5 left) 2024-09-11T01:32:46.926 INFO:tasks.rados.rados.0.smithi071.stdout:570: done (4 left) 2024-09-11T01:32:46.926 INFO:tasks.rados.rados.0.smithi071.stdout:572: done (3 left) 2024-09-11T01:32:46.926 INFO:tasks.rados.rados.0.smithi071.stdout:573: done (2 left) 2024-09-11T01:32:46.926 INFO:tasks.rados.rados.0.smithi071.stdout:576: delete oid 6 current snap is 55 2024-09-11T01:32:46.926 INFO:tasks.rados.rados.0.smithi071.stdout:574: finishing write tid 2 to smithi07134777-44 2024-09-11T01:32:46.926 INFO:tasks.rados.rados.0.smithi071.stdout:574: finishing write tid 3 to smithi07134777-44 2024-09-11T01:32:46.926 INFO:tasks.rados.rados.0.smithi071.stdout:574: finishing write tid 4 to smithi07134777-44 2024-09-11T01:32:46.927 INFO:tasks.rados.rados.0.smithi071.stdout:575: finishing rollback tid 0 to smithi07134777-25 2024-09-11T01:32:46.927 INFO:tasks.rados.rados.0.smithi071.stdout:574: finishing write tid 5 to smithi07134777-44 2024-09-11T01:32:46.927 INFO:tasks.rados.rados.0.smithi071.stdout:574: finishing write tid 6 to smithi07134777-44 2024-09-11T01:32:46.927 INFO:tasks.rados.rados.0.smithi071.stdout:574: oid 44 updating version 0 to 222 2024-09-11T01:32:46.927 INFO:tasks.rados.rados.0.smithi071.stdout:574: oid 44 updating version 222 to 223 2024-09-11T01:32:46.927 INFO:tasks.rados.rados.0.smithi071.stdout:574: oid 44 updating version 223 to 224 2024-09-11T01:32:46.927 INFO:tasks.rados.rados.0.smithi071.stdout:574: oid 44 version 224 is already newer than 220 2024-09-11T01:32:46.927 INFO:tasks.rados.rados.0.smithi071.stdout:574: oid 44 version 224 is already newer than 221 2024-09-11T01:32:46.927 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 224 (ObjNum 222 snap 55 seq_num 222) dirty exists 2024-09-11T01:32:46.927 INFO:tasks.rados.rados.0.smithi071.stdout:574: left oid 44 (ObjNum 222 snap 55 seq_num 222) 2024-09-11T01:32:46.927 INFO:tasks.rados.rados.0.smithi071.stdout:575: finishing rollback tid 1 to smithi07134777-25 2024-09-11T01:32:46.927 INFO:tasks.rados.rados.0.smithi071.stdout:575: finishing rollback tid 2 to smithi07134777-25 2024-09-11T01:32:46.927 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 60 (ObjNum 173 snap 37 seq_num 173) dirty exists 2024-09-11T01:32:46.928 INFO:tasks.rados.rados.0.smithi071.stdout:574: done (2 left) 2024-09-11T01:32:46.928 INFO:tasks.rados.rados.0.smithi071.stdout:575: done (1 left) 2024-09-11T01:32:46.928 INFO:tasks.rados.rados.0.smithi071.stdout:576: done (0 left) 2024-09-11T01:32:46.928 INFO:tasks.rados.rados.0.smithi071.stdout:577: delete oid 19 current snap is 55 2024-09-11T01:32:46.929 INFO:tasks.rados.rados.0.smithi071.stdout:577: done (0 left) 2024-09-11T01:32:46.929 INFO:tasks.rados.rados.0.smithi071.stdout:578: write (excl) oid 25 current snap is 55 2024-09-11T01:32:46.929 INFO:tasks.rados.rados.0.smithi071.stdout:578: seq_num 225 ranges {581575=705262,1733272=707409,3195938=437497} 2024-09-11T01:32:46.935 INFO:tasks.rados.rados.0.smithi071.stdout:578: writing smithi07134777-25 from 581575 to 1286837 tid 1 2024-09-11T01:32:46.942 INFO:tasks.rados.rados.0.smithi071.stdout:578: writing smithi07134777-25 from 1733272 to 2440681 tid 2 2024-09-11T01:32:46.948 INFO:tasks.rados.rados.0.smithi071.stdout:578: writing smithi07134777-25 from 3195938 to 3633435 tid 3 2024-09-11T01:32:46.948 INFO:tasks.rados.rados.0.smithi071.stdout:579: write (excl) oid 8 current snap is 55 2024-09-11T01:32:46.948 INFO:tasks.rados.rados.0.smithi071.stdout:579: seq_num 226 ranges {598382=780511,2075401=451067,3107008=648769} 2024-09-11T01:32:46.955 INFO:tasks.rados.rados.0.smithi071.stdout:579: writing smithi07134777-8 from 598382 to 1378893 tid 1 2024-09-11T01:32:46.961 INFO:tasks.rados.rados.0.smithi071.stdout:579: writing smithi07134777-8 from 2075401 to 2526468 tid 2 2024-09-11T01:32:46.968 INFO:tasks.rados.rados.0.smithi071.stdout:579: writing smithi07134777-8 from 3107008 to 3755777 tid 3 2024-09-11T01:32:46.968 INFO:tasks.rados.rados.0.smithi071.stdout:580: write (excl) oid 50 current snap is 55 2024-09-11T01:32:46.968 INFO:tasks.rados.rados.0.smithi071.stdout:580: seq_num 227 ranges {615189=455760,1733883=711078,3134431=460041} 2024-09-11T01:32:46.974 INFO:tasks.rados.rados.0.smithi071.stdout:580: writing smithi07134777-50 from 615189 to 1070949 tid 1 2024-09-11T01:32:46.981 INFO:tasks.rados.rados.0.smithi071.stdout:580: writing smithi07134777-50 from 1733883 to 2444961 tid 2 2024-09-11T01:32:46.988 INFO:tasks.rados.rados.0.smithi071.stdout:580: writing smithi07134777-50 from 3134431 to 3594472 tid 3 2024-09-11T01:32:46.988 INFO:tasks.rados.rados.0.smithi071.stdout:581: write oid 17 current snap is 55 2024-09-11T01:32:46.988 INFO:tasks.rados.rados.0.smithi071.stdout:578: finishing write tid 1 to smithi07134777-25 2024-09-11T01:32:46.988 INFO:tasks.rados.rados.0.smithi071.stdout:581: seq_num 228 ranges {631996=531009,1792365=454736,2761854=671313} 2024-09-11T01:32:46.994 INFO:tasks.rados.rados.0.smithi071.stdout:581: writing smithi07134777-17 from 631996 to 1163005 tid 1 2024-09-11T01:32:47.000 INFO:tasks.rados.rados.0.smithi071.stdout:581: writing smithi07134777-17 from 1792365 to 2247101 tid 2 2024-09-11T01:32:47.007 INFO:tasks.rados.rados.0.smithi071.stdout:581: writing smithi07134777-17 from 2761854 to 3433167 tid 3 2024-09-11T01:32:47.007 INFO:tasks.rados.rados.0.smithi071.stdout:582: write oid 39 current snap is 55 2024-09-11T01:32:47.007 INFO:tasks.rados.rados.0.smithi071.stdout:578: finishing write tid 2 to smithi07134777-25 2024-09-11T01:32:47.007 INFO:tasks.rados.rados.0.smithi071.stdout:578: finishing write tid 3 to smithi07134777-25 2024-09-11T01:32:47.007 INFO:tasks.rados.rados.0.smithi071.stdout:582: seq_num 229 ranges {648803=722611,1967200=714747,3421983=426820} 2024-09-11T01:32:47.014 INFO:tasks.rados.rados.0.smithi071.stdout:582: writing smithi07134777-39 from 648803 to 1371414 tid 1 2024-09-11T01:32:47.021 INFO:tasks.rados.rados.0.smithi071.stdout:582: writing smithi07134777-39 from 1967200 to 2681947 tid 2 2024-09-11T01:32:47.029 INFO:tasks.rados.rados.0.smithi071.stdout:582: writing smithi07134777-39 from 3421983 to 3848803 tid 3 2024-09-11T01:32:47.029 INFO:tasks.rados.rados.0.smithi071.stdout:583: snap_create 2024-09-11T01:32:47.029 INFO:tasks.rados.rados.0.smithi071.stdout:578: finishing write tid 4 to smithi07134777-25 2024-09-11T01:32:47.030 INFO:tasks.rados.rados.0.smithi071.stdout:578: finishing write tid 5 to smithi07134777-25 2024-09-11T01:32:47.030 INFO:tasks.rados.rados.0.smithi071.stdout:578: oid 25 updating version 0 to 61 2024-09-11T01:32:47.030 INFO:tasks.rados.rados.0.smithi071.stdout:578: oid 25 updating version 61 to 62 2024-09-11T01:32:47.030 INFO:tasks.rados.rados.0.smithi071.stdout:578: oid 25 updating version 62 to 64 2024-09-11T01:32:47.030 INFO:tasks.rados.rados.0.smithi071.stdout:578: oid 25 version 64 is already newer than 63 2024-09-11T01:32:47.030 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 64 (ObjNum 225 snap 55 seq_num 225) dirty exists 2024-09-11T01:32:47.030 INFO:tasks.rados.rados.0.smithi071.stdout:578: left oid 25 (ObjNum 225 snap 55 seq_num 225) 2024-09-11T01:32:47.030 INFO:tasks.rados.rados.0.smithi071.stdout:579: finishing write tid 1 to smithi07134777-8 2024-09-11T01:32:47.030 INFO:tasks.rados.rados.0.smithi071.stdout:579: finishing write tid 2 to smithi07134777-8 2024-09-11T01:32:47.030 INFO:tasks.rados.rados.0.smithi071.stdout:579: finishing write tid 3 to smithi07134777-8 2024-09-11T01:32:47.030 INFO:tasks.rados.rados.0.smithi071.stdout:579: finishing write tid 4 to smithi07134777-8 2024-09-11T01:32:47.030 INFO:tasks.rados.rados.0.smithi071.stdout:579: finishing write tid 5 to smithi07134777-8 2024-09-11T01:32:47.030 INFO:tasks.rados.rados.0.smithi071.stdout:579: oid 8 updating version 0 to 75 2024-09-11T01:32:47.030 INFO:tasks.rados.rados.0.smithi071.stdout:579: oid 8 updating version 75 to 77 2024-09-11T01:32:47.031 INFO:tasks.rados.rados.0.smithi071.stdout:579: oid 8 updating version 77 to 78 2024-09-11T01:32:47.031 INFO:tasks.rados.rados.0.smithi071.stdout:579: oid 8 version 78 is already newer than 76 2024-09-11T01:32:47.031 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 78 (ObjNum 226 snap 55 seq_num 226) dirty exists 2024-09-11T01:32:47.031 INFO:tasks.rados.rados.0.smithi071.stdout:579: left oid 8 (ObjNum 226 snap 55 seq_num 226) 2024-09-11T01:32:47.031 INFO:tasks.rados.rados.0.smithi071.stdout:580: finishing write tid 1 to smithi07134777-50 2024-09-11T01:32:47.031 INFO:tasks.rados.rados.0.smithi071.stdout:580: finishing write tid 2 to smithi07134777-50 2024-09-11T01:32:47.031 INFO:tasks.rados.rados.0.smithi071.stdout:580: finishing write tid 3 to smithi07134777-50 2024-09-11T01:32:47.031 INFO:tasks.rados.rados.0.smithi071.stdout:580: finishing write tid 4 to smithi07134777-50 2024-09-11T01:32:47.031 INFO:tasks.rados.rados.0.smithi071.stdout:580: finishing write tid 5 to smithi07134777-50 2024-09-11T01:32:47.031 INFO:tasks.rados.rados.0.smithi071.stdout:580: oid 50 updating version 0 to 274 2024-09-11T01:32:47.031 INFO:tasks.rados.rados.0.smithi071.stdout:580: oid 50 version 274 is already newer than 273 2024-09-11T01:32:47.031 INFO:tasks.rados.rados.0.smithi071.stdout:580: oid 50 updating version 274 to 275 2024-09-11T01:32:47.031 INFO:tasks.rados.rados.0.smithi071.stdout:580: oid 50 version 275 is already newer than 272 2024-09-11T01:32:47.032 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 275 (ObjNum 227 snap 55 seq_num 227) dirty exists 2024-09-11T01:32:47.032 INFO:tasks.rados.rados.0.smithi071.stdout:580: left oid 50 (ObjNum 227 snap 55 seq_num 227) 2024-09-11T01:32:47.032 INFO:tasks.rados.rados.0.smithi071.stdout:581: finishing write tid 1 to smithi07134777-17 2024-09-11T01:32:47.032 INFO:tasks.rados.rados.0.smithi071.stdout:581: finishing write tid 2 to smithi07134777-17 2024-09-11T01:32:47.032 INFO:tasks.rados.rados.0.smithi071.stdout:581: finishing write tid 3 to smithi07134777-17 2024-09-11T01:32:47.032 INFO:tasks.rados.rados.0.smithi071.stdout:581: finishing write tid 4 to smithi07134777-17 2024-09-11T01:32:47.032 INFO:tasks.rados.rados.0.smithi071.stdout:581: finishing write tid 5 to smithi07134777-17 2024-09-11T01:32:47.032 INFO:tasks.rados.rados.0.smithi071.stdout:581: oid 17 updating version 0 to 140 2024-09-11T01:32:47.032 INFO:tasks.rados.rados.0.smithi071.stdout:581: oid 17 updating version 140 to 141 2024-09-11T01:32:47.032 INFO:tasks.rados.rados.0.smithi071.stdout:581: oid 17 version 141 is already newer than 139 2024-09-11T01:32:47.032 INFO:tasks.rados.rados.0.smithi071.stdout:581: oid 17 version 141 is already newer than 138 2024-09-11T01:32:47.032 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 141 (ObjNum 228 snap 55 seq_num 228) dirty exists 2024-09-11T01:32:47.032 INFO:tasks.rados.rados.0.smithi071.stdout:581: left oid 17 (ObjNum 228 snap 55 seq_num 228) 2024-09-11T01:32:47.032 INFO:tasks.rados.rados.0.smithi071.stdout:582: finishing write tid 1 to smithi07134777-39 2024-09-11T01:32:47.033 INFO:tasks.rados.rados.0.smithi071.stdout:582: finishing write tid 2 to smithi07134777-39 2024-09-11T01:32:47.033 INFO:tasks.rados.rados.0.smithi071.stdout:582: finishing write tid 3 to smithi07134777-39 2024-09-11T01:32:47.033 INFO:tasks.rados.rados.0.smithi071.stdout:582: finishing write tid 4 to smithi07134777-39 2024-09-11T01:32:47.033 INFO:tasks.rados.rados.0.smithi071.stdout:582: finishing write tid 5 to smithi07134777-39 2024-09-11T01:32:47.033 INFO:tasks.rados.rados.0.smithi071.stdout:582: oid 39 updating version 0 to 170 2024-09-11T01:32:47.033 INFO:tasks.rados.rados.0.smithi071.stdout:582: oid 39 updating version 170 to 171 2024-09-11T01:32:47.034 INFO:tasks.rados.rados.0.smithi071.stdout:582: oid 39 updating version 171 to 173 2024-09-11T01:32:47.034 INFO:tasks.rados.rados.0.smithi071.stdout:582: oid 39 version 173 is already newer than 172 2024-09-11T01:32:47.034 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 173 (ObjNum 229 snap 55 seq_num 229) dirty exists 2024-09-11T01:32:47.034 INFO:tasks.rados.rados.0.smithi071.stdout:582: left oid 39 (ObjNum 229 snap 55 seq_num 229) 2024-09-11T01:32:47.080 INFO:tasks.rados.rados.0.smithi071.stdout:578: done (5 left) 2024-09-11T01:32:47.080 INFO:tasks.rados.rados.0.smithi071.stdout:579: done (4 left) 2024-09-11T01:32:47.080 INFO:tasks.rados.rados.0.smithi071.stdout:580: done (3 left) 2024-09-11T01:32:47.080 INFO:tasks.rados.rados.0.smithi071.stdout:581: done (2 left) 2024-09-11T01:32:47.080 INFO:tasks.rados.rados.0.smithi071.stdout:582: done (1 left) 2024-09-11T01:32:47.080 INFO:tasks.rados.rados.0.smithi071.stdout:583: done (0 left) 2024-09-11T01:32:47.080 INFO:tasks.rados.rados.0.smithi071.stdout:584: copy_from oid 36 from oid 47 current snap is 56 2024-09-11T01:32:47.080 INFO:tasks.rados.rados.0.smithi071.stdout:585: snap_remove snap 55 2024-09-11T01:32:48.090 INFO:tasks.rados.rados.0.smithi071.stdout:585: done (1 left) 2024-09-11T01:32:48.091 INFO:tasks.rados.rados.0.smithi071.stdout:586: delete oid 50 current snap is 56 2024-09-11T01:32:48.091 INFO:tasks.rados.rados.0.smithi071.stdout:584: finishing copy_from racing read to smithi07134777-36 2024-09-11T01:32:48.091 INFO:tasks.rados.rados.0.smithi071.stdout:584: finishing copy_from to smithi07134777-36 2024-09-11T01:32:48.091 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 117 (ObjNum 187 snap 41 seq_num 187) dirty exists 2024-09-11T01:32:48.095 INFO:tasks.rados.rados.0.smithi071.stdout:584: done (1 left) 2024-09-11T01:32:48.095 INFO:tasks.rados.rados.0.smithi071.stdout:586: done (0 left) 2024-09-11T01:32:48.095 INFO:tasks.rados.rados.0.smithi071.stdout:587: read oid 29 snap 49 2024-09-11T01:32:48.095 INFO:tasks.rados.rados.0.smithi071.stdout:587: expect (ObjNum 191 snap 42 seq_num 191) 2024-09-11T01:32:48.096 INFO:tasks.rados.rados.0.smithi071.stdout:588: delete oid 6 current snap is 56 2024-09-11T01:32:48.097 INFO:tasks.rados.rados.0.smithi071.stdout:588: done (1 left) 2024-09-11T01:32:48.097 INFO:tasks.rados.rados.0.smithi071.stdout:589: snap_create 2024-09-11T01:32:48.102 INFO:tasks.rados.rados.0.smithi071.stdout:587: expect (ObjNum 191 snap 42 seq_num 191) 2024-09-11T01:32:48.144 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:32:49.108 INFO:tasks.rados.rados.0.smithi071.stdout:587: done (1 left) 2024-09-11T01:32:49.108 INFO:tasks.rados.rados.0.smithi071.stdout:589: done (0 left) 2024-09-11T01:32:49.108 INFO:tasks.rados.rados.0.smithi071.stdout:590: write oid 1 current snap is 57 2024-09-11T01:32:49.108 INFO:tasks.rados.rados.0.smithi071.stdout:590: seq_num 232 ranges {699224=548358,2026293=462074,3271546=432754} 2024-09-11T01:32:49.113 INFO:tasks.rados.rados.0.smithi071.stdout:590: writing smithi07134777-1 from 699224 to 1247582 tid 1 2024-09-11T01:32:49.121 INFO:tasks.rados.rados.0.smithi071.stdout:590: writing smithi07134777-1 from 2026293 to 2488367 tid 2 2024-09-11T01:32:49.128 INFO:tasks.rados.rados.0.smithi071.stdout:590: writing smithi07134777-1 from 3271546 to 3704300 tid 3 2024-09-11T01:32:49.128 INFO:tasks.rados.rados.0.smithi071.stdout:591: write oid 32 current snap is 57 2024-09-11T01:32:49.128 INFO:tasks.rados.rados.0.smithi071.stdout:590: finishing write tid 1 to smithi07134777-1 2024-09-11T01:32:49.128 INFO:tasks.rados.rados.0.smithi071.stdout:590: finishing write tid 2 to smithi07134777-1 2024-09-11T01:32:49.128 INFO:tasks.rados.rados.0.smithi071.stdout:591: seq_num 233 ranges {716031=623607,2084775=605732,3298969=617062} 2024-09-11T01:32:49.135 INFO:tasks.rados.rados.0.smithi071.stdout:591: writing smithi07134777-32 from 716031 to 1339638 tid 1 2024-09-11T01:32:49.142 INFO:tasks.rados.rados.0.smithi071.stdout:591: writing smithi07134777-32 from 2084775 to 2690507 tid 2 2024-09-11T01:32:49.149 INFO:tasks.rados.rados.0.smithi071.stdout:591: writing smithi07134777-32 from 3298969 to 3916031 tid 3 2024-09-11T01:32:49.149 INFO:tasks.rados.rados.0.smithi071.stdout:592: write (excl) oid 11 current snap is 57 2024-09-11T01:32:49.149 INFO:tasks.rados.rados.0.smithi071.stdout:590: finishing write tid 3 to smithi07134777-1 2024-09-11T01:32:49.149 INFO:tasks.rados.rados.0.smithi071.stdout:590: finishing write tid 4 to smithi07134777-1 2024-09-11T01:32:49.149 INFO:tasks.rados.rados.0.smithi071.stdout:590: finishing write tid 5 to smithi07134777-1 2024-09-11T01:32:49.149 INFO:tasks.rados.rados.0.smithi071.stdout:590: oid 1 updating version 0 to 146 2024-09-11T01:32:49.149 INFO:tasks.rados.rados.0.smithi071.stdout:590: oid 1 version 146 is already newer than 145 2024-09-11T01:32:49.150 INFO:tasks.rados.rados.0.smithi071.stdout:590: oid 1 version 146 is already newer than 143 2024-09-11T01:32:49.150 INFO:tasks.rados.rados.0.smithi071.stdout:590: oid 1 version 146 is already newer than 144 2024-09-11T01:32:49.150 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 146 (ObjNum 232 snap 57 seq_num 232) dirty exists 2024-09-11T01:32:49.150 INFO:tasks.rados.rados.0.smithi071.stdout:590: left oid 1 (ObjNum 232 snap 57 seq_num 232) 2024-09-11T01:32:49.150 INFO:tasks.rados.rados.0.smithi071.stdout:591: finishing write tid 1 to smithi07134777-32 2024-09-11T01:32:49.150 INFO:tasks.rados.rados.0.smithi071.stdout:592: seq_num 234 ranges {732838=698856,2026904=465743,3210039=571651} 2024-09-11T01:32:49.156 INFO:tasks.rados.rados.0.smithi071.stdout:592: writing smithi07134777-11 from 732838 to 1431694 tid 1 2024-09-11T01:32:49.162 INFO:tasks.rados.rados.0.smithi071.stdout:592: writing smithi07134777-11 from 2026904 to 2492647 tid 2 2024-09-11T01:32:49.169 INFO:tasks.rados.rados.0.smithi071.stdout:592: writing smithi07134777-11 from 3210039 to 3781690 tid 3 2024-09-11T01:32:49.169 INFO:tasks.rados.rados.0.smithi071.stdout:590: done (2 left) 2024-09-11T01:32:49.169 INFO:tasks.rados.rados.0.smithi071.stdout:593: read oid 48 snap -1 2024-09-11T01:32:49.169 INFO:tasks.rados.rados.0.smithi071.stdout:593: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:32:49.169 INFO:tasks.rados.rados.0.smithi071.stdout:594: write oid 25 current snap is 57 2024-09-11T01:32:49.169 INFO:tasks.rados.rados.0.smithi071.stdout:591: finishing write tid 2 to smithi07134777-32 2024-09-11T01:32:49.170 INFO:tasks.rados.rados.0.smithi071.stdout:591: finishing write tid 3 to smithi07134777-32 2024-09-11T01:32:49.170 INFO:tasks.rados.rados.0.smithi071.stdout:591: finishing write tid 4 to smithi07134777-32 2024-09-11T01:32:49.170 INFO:tasks.rados.rados.0.smithi071.stdout:591: finishing write tid 5 to smithi07134777-32 2024-09-11T01:32:49.170 INFO:tasks.rados.rados.0.smithi071.stdout:591: oid 32 updating version 0 to 174 2024-09-11T01:32:49.170 INFO:tasks.rados.rados.0.smithi071.stdout:591: oid 32 updating version 174 to 176 2024-09-11T01:32:49.170 INFO:tasks.rados.rados.0.smithi071.stdout:591: oid 32 updating version 176 to 177 2024-09-11T01:32:49.170 INFO:tasks.rados.rados.0.smithi071.stdout:591: oid 32 version 177 is already newer than 175 2024-09-11T01:32:49.170 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 177 (ObjNum 233 snap 57 seq_num 233) dirty exists 2024-09-11T01:32:49.170 INFO:tasks.rados.rados.0.smithi071.stdout:591: left oid 32 (ObjNum 233 snap 57 seq_num 233) 2024-09-11T01:32:49.170 INFO:tasks.rados.rados.0.smithi071.stdout:592: finishing write tid 1 to smithi07134777-11 2024-09-11T01:32:49.170 INFO:tasks.rados.rados.0.smithi071.stdout:592: finishing write tid 2 to smithi07134777-11 2024-09-11T01:32:49.170 INFO:tasks.rados.rados.0.smithi071.stdout:594: seq_num 235 ranges {749645=774105,2085386=609401,3237462=712183} 2024-09-11T01:32:49.177 INFO:tasks.rados.rados.0.smithi071.stdout:594: writing smithi07134777-25 from 749645 to 1523750 tid 1 2024-09-11T01:32:49.183 INFO:tasks.rados.rados.0.smithi071.stdout:594: writing smithi07134777-25 from 2085386 to 2694787 tid 2 2024-09-11T01:32:49.190 INFO:tasks.rados.rados.0.smithi071.stdout:594: writing smithi07134777-25 from 3237462 to 3949645 tid 3 2024-09-11T01:32:49.190 INFO:tasks.rados.rados.0.smithi071.stdout:591: done (3 left) 2024-09-11T01:32:49.190 INFO:tasks.rados.rados.0.smithi071.stdout:595: write oid 10 current snap is 57 2024-09-11T01:32:49.190 INFO:tasks.rados.rados.0.smithi071.stdout:595: seq_num 236 ranges {766452=565707,1860221=469412,2981238=594195} 2024-09-11T01:32:49.197 INFO:tasks.rados.rados.0.smithi071.stdout:595: writing smithi07134777-10 from 766452 to 1332159 tid 1 2024-09-11T01:32:49.203 INFO:tasks.rados.rados.0.smithi071.stdout:595: writing smithi07134777-10 from 1860221 to 2329633 tid 2 2024-09-11T01:32:49.210 INFO:tasks.rados.rados.0.smithi071.stdout:595: writing smithi07134777-10 from 2981238 to 3575433 tid 3 2024-09-11T01:32:49.211 INFO:tasks.rados.rados.0.smithi071.stdout:596: read oid 33 snap -1 2024-09-11T01:32:49.211 INFO:tasks.rados.rados.0.smithi071.stdout:596: expect deleted 2024-09-11T01:32:49.211 INFO:tasks.rados.rados.0.smithi071.stdout:597: rollback oid 32 current snap is 57 2024-09-11T01:32:49.211 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 32 to 56 2024-09-11T01:32:49.211 INFO:tasks.rados.rados.0.smithi071.stdout:593: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:32:49.239 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:32:49.267 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:32:49.295 INFO:tasks.rados.rados.0.smithi071.stdout:592: finishing write tid 3 to smithi07134777-11 2024-09-11T01:32:49.295 INFO:tasks.rados.rados.0.smithi071.stdout:593: done (5 left) 2024-09-11T01:32:49.295 INFO:tasks.rados.rados.0.smithi071.stdout:598: snap_remove snap 54 2024-09-11T01:32:49.779 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:32:49.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.891+0000 7f2550bb4640 10 monclient: get_monmap_and_config 2024-09-11T01:32:49.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.891+0000 7f2550bb4640 10 monclient: build_initial_monmap 2024-09-11T01:32:49.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.891+0000 7f2550bb4640 10 monclient: monmap: 2024-09-11T01:32:49.893 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:49.893 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:49.893 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:32:49.891908+0000 2024-09-11T01:32:49.893 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:32:49.891908+0000 2024-09-11T01:32:49.893 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:49.893 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:49.893 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:32:49.893 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:32:49.893 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:32:49.894 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:32:49.894 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:32:49.894 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:49.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.891+0000 7f2550bb4640 1 Processor -- start 2024-09-11T01:32:49.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.891+0000 7f2550bb4640 1 -- start start 2024-09-11T01:32:49.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.891+0000 7f2550bb4640 10 monclient: init 2024-09-11T01:32:49.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f2550bb4640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:49.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f2550bb4640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-11T01:32:49.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f2550bb4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f254c138230 addr v1:172.21.15.71:6789/0 2024-09-11T01:32:49.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f2550bb4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f254c09d030 0x7f254c13d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:49.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f2550bb4640 10 monclient(hunting): picked mon.noname-b con 0x7f254c09d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:49.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f2550bb4640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f254c09d460 0x7f254c13f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:49.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f2550bb4640 10 monclient(hunting): picked mon.noname-c con 0x7f254c09d460 addr v2:172.21.15.71:3300/0 2024-09-11T01:32:49.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f2550bb4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f254c0a6e30 con 0x7f254c138230 2024-09-11T01:32:49.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f2550bb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:49.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f2550bb4640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f254c09c9c0 con 0x7f254c09d460 2024-09-11T01:32:49.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f2550bb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:49.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f2550bb4640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f254c09ba20 con 0x7f254c09d030 2024-09-11T01:32:49.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f2550bb4640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:49.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f2550bb4640 10 monclient(hunting): authenticate will time out at 976.334619s 2024-09-11T01:32:49.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f2549d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f254c09d030 0x7f254c13d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:49.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f2549d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f254c09d030 0x7f254c13d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:39308/0 (socket says 172.21.15.12:39308) 2024-09-11T01:32:49.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f254a59c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f254c138230 0x7f254c138610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.12:50880/0 (socket says 172.21.15.12:50880) 2024-09-11T01:32:49.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f2549d9b640 1 -- 172.21.15.12:0/2626188288 learned_addr learned my addr 172.21.15.12:0/2626188288 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:49.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f2549d9b640 10 monclient(hunting): get_auth_request con 0x7f254c09d030 auth_method 0 2024-09-11T01:32:49.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f2549d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:49.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f2549d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:49.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f2549d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:49.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f254ad9d640 1 --2- 172.21.15.12:0/2626188288 >> v2:172.21.15.71:3300/0 conn(0x7f254c09d460 0x7f254c13f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:49.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f254ad9d640 10 monclient(hunting): get_auth_request con 0x7f254c09d460 auth_method 0 2024-09-11T01:32:49.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f254ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:49.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f254ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:49.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f254ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:49.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f2549d9b640 1 -- 172.21.15.12:0/2626188288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f254c09d030 msgr2=0x7f254c13d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:32:49.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f2549d9b640 1 -- 172.21.15.12:0/2626188288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f254c09d030 msgr2=0x7f254c13d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:32:49.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f2549d9b640 1 --2- 172.21.15.12:0/2626188288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f254c09d030 0x7f254c13d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:49.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.892+0000 7f2549d9b640 1 --2- 172.21.15.12:0/2626188288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f254c09d030 0x7f254c13d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.894+0000 7f254a59c640 1 -- 172.21.15.12:0/2626188288 >> v1:172.21.15.71:6789/0 conn(0x7f254c138230 legacy=0x7f254c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.895+0000 7f254ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.895+0000 7f254ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.895+0000 7f254ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.895+0000 7f254ad9d640 10 monclient(hunting): handle_auth_done global_id 34174 payload 274 2024-09-11T01:32:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.895+0000 7f254ad9d640 1 -- 172.21.15.12:0/2626188288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f254c09d030 msgr2=0x7f254c13d090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:32:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.895+0000 7f254ad9d640 1 --2- 172.21.15.12:0/2626188288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f254c09d030 0x7f254c13d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.895+0000 7f254ad9d640 1 -- 172.21.15.12:0/2626188288 >> v1:172.21.15.71:6789/0 conn(0x7f254c138230 legacy=0x7f254c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.895+0000 7f254ad9d640 10 monclient: _finish_hunting 0 2024-09-11T01:32:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.895+0000 7f254ad9d640 1 monclient: found mon.noname-c 2024-09-11T01:32:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.895+0000 7f254ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.895+0000 7f254959a640 1 -- 172.21.15.12:0/2626188288 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3979490382 0 0) 0x7f254c0a6e30 con 0x7f254c138230 2024-09-11T01:32:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.895+0000 7f254ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:32:49.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.895+0000 7f254ad9d640 1 -- 172.21.15.12:0/2626188288 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f254c0a66a0 con 0x7f254c09d460 2024-09-11T01:32:49.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.895+0000 7f254959a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T01:32:49.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.898+0000 7f254ad9d640 1 -- 172.21.15.12:0/2626188288 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f254c09d460 msgr2=0x7f254c13f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:49.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.898+0000 7f254ad9d640 1 --2- 172.21.15.12:0/2626188288 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f254c09d460 0x7f254c13f5e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2544009e10 tx=0x7f254402a5b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36efeec3a88a0c4f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:49.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.898+0000 7f254959a640 1 -- 172.21.15.12:0/2626188288 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2544007950 con 0x7f254c09d460 2024-09-11T01:32:49.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.898+0000 7f254959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:49.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.898+0000 7f254959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:32:49.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.898+0000 7f254959a640 10 monclient: dump: 2024-09-11T01:32:49.908 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:49.908 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:49.908 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:49.908 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:49.908 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:49.908 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:49.908 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:49.908 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:49.908 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:49.908 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:49.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.898+0000 7f254959a640 10 monclient: _finish_auth 0 2024-09-11T01:32:49.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.898+0000 7f254959a640 10 monclient: _check_auth_tickets 2024-09-11T01:32:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.898+0000 7f254959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.898+0000 7f2550bb4640 5 monclient: authenticate success, global_id 34174 2024-09-11T01:32:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.898+0000 7f2550bb4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:32:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.898+0000 7f254959a640 1 -- 172.21.15.12:0/2626188288 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2544007af0 con 0x7f254c09d460 2024-09-11T01:32:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.898+0000 7f254959a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.898+0000 7f254959a640 1 -- 172.21.15.12:0/2626188288 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2544031400 con 0x7f254c09d460 2024-09-11T01:32:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.898+0000 7f254959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.898+0000 7f254959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:32:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.898+0000 7f254959a640 10 monclient: dump: 2024-09-11T01:32:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:49.910 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:49.910 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:49.910 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:49.910 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:49.910 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:49.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.898+0000 7f2550bb4640 10 monclient: get_monmap_and_config success 2024-09-11T01:32:49.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.898+0000 7f2550bb4640 4 set_mon_vals no callback set 2024-09-11T01:32:49.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.898+0000 7f2550bb4640 10 monclient: shutdown 2024-09-11T01:32:49.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.898+0000 7f2550bb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:49.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.898+0000 7f2550bb4640 1 -- 172.21.15.12:0/2626188288 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f254c09d460 msgr2=0x7f254c13f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:49.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.898+0000 7f2550bb4640 1 --2- 172.21.15.12:0/2626188288 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f254c09d460 0x7f254c13f5e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2544009e10 tx=0x7f254402a5b0 comp rx=0 tx=0).stop 2024-09-11T01:32:49.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.899+0000 7f2550bb4640 1 -- 172.21.15.12:0/2626188288 shutdown_connections 2024-09-11T01:32:49.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.899+0000 7f2550bb4640 1 --2- 172.21.15.12:0/2626188288 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f254c09d460 0x7f254c13f5e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:49.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.899+0000 7f2550bb4640 1 --2- 172.21.15.12:0/2626188288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f254c09d030 0x7f254c13d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:49.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.899+0000 7f2550bb4640 1 -- 172.21.15.12:0/2626188288 >> 172.21.15.12:0/2626188288 conn(0x7f254c0a7120 msgr2=0x7f254c0a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:49.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.899+0000 7f2550bb4640 1 -- 172.21.15.12:0/2626188288 shutdown_connections 2024-09-11T01:32:49.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.899+0000 7f2550bb4640 1 -- 172.21.15.12:0/2626188288 wait complete. 2024-09-11T01:32:49.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.899+0000 7f2550bb4640 10 monclient: build_initial_monmap 2024-09-11T01:32:49.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.899+0000 7f2550bb4640 10 monclient: monmap: 2024-09-11T01:32:49.911 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:49.911 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:32:49.911 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:32:49.911 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:32:49.911 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:49.912 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:49.912 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:32:49.912 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:32:49.912 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:32:49.912 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:49.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.900+0000 7f2550bb4640 1 Processor -- start 2024-09-11T01:32:49.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.900+0000 7f2550bb4640 1 -- start start 2024-09-11T01:32:49.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.900+0000 7f2550bb4640 10 monclient: init 2024-09-11T01:32:49.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.900+0000 7f2550bb4640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:49.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.900+0000 7f2550bb4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:32:49.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.900+0000 7f2550bb4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f254c09d030 0x7f254c13e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:49.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.900+0000 7f2550bb4640 10 monclient(hunting): picked mon.noname-c con 0x7f254c09d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:49.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.900+0000 7f2550bb4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f254c09d460 0x7f254c1cd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:49.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.900+0000 7f2550bb4640 10 monclient(hunting): picked mon.noname-a con 0x7f254c09d460 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:32:49.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.900+0000 7f2550bb4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f254c138230 0x7f254c1cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:49.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.900+0000 7f2550bb4640 10 monclient(hunting): picked mon.noname-b con 0x7f254c138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:32:49.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.900+0000 7f2550bb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:49.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.900+0000 7f2550bb4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f254c13fca0 con 0x7f254c09d460 2024-09-11T01:32:49.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.900+0000 7f2550bb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:49.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.900+0000 7f2550bb4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f254c13fb20 con 0x7f254c138230 2024-09-11T01:32:49.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.900+0000 7f2550bb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:49.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.900+0000 7f2550bb4640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f254c13fe20 con 0x7f254c09d030 2024-09-11T01:32:49.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.900+0000 7f2550bb4640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:49.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.900+0000 7f2550bb4640 10 monclient(hunting): authenticate will time out at 976.342907s 2024-09-11T01:32:49.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.900+0000 7f254a59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f254c09d030 0x7f254c13e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:49.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.900+0000 7f254a59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f254c09d030 0x7f254c13e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:39310/0 (socket says 172.21.15.12:39310) 2024-09-11T01:32:49.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.900+0000 7f254a59c640 1 -- 172.21.15.12:0/561598385 learned_addr learned my addr 172.21.15.12:0/561598385 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:49.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f254a59c640 10 monclient(hunting): get_auth_request con 0x7f254c09d030 auth_method 0 2024-09-11T01:32:49.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f254a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:49.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f254a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:49.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f254a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:49.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f2549d9b640 1 --2- 172.21.15.12:0/561598385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f254c09d460 0x7f254c1cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:49.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f2549d9b640 10 monclient(hunting): get_auth_request con 0x7f254c09d460 auth_method 0 2024-09-11T01:32:49.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f2549d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:49.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f2549d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:49.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f2549d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:49.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f254a59c640 1 -- 172.21.15.12:0/561598385 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f254c09d030 msgr2=0x7f254c13e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:32:49.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f254a59c640 1 -- 172.21.15.12:0/561598385 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f254c09d030 msgr2=0x7f254c13e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:32:49.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f254a59c640 1 --2- 172.21.15.12:0/561598385 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f254c09d030 0x7f254c13e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:49.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f254a59c640 1 --2- 172.21.15.12:0/561598385 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f254c09d030 0x7f254c13e0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:49.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f2549d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:49.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f2549d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:49.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f2549d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:49.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f254ad9d640 1 --2- 172.21.15.12:0/561598385 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f254c138230 0x7f254c1cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:49.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f254ad9d640 10 monclient(hunting): get_auth_request con 0x7f254c138230 auth_method 0 2024-09-11T01:32:49.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f254ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:49.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f254ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:49.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f254ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:49.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f254ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:49.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f254ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:49.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f254ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:49.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f2549d9b640 10 monclient(hunting): handle_auth_done global_id 24315 payload 824 2024-09-11T01:32:49.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f2549d9b640 1 -- 172.21.15.12:0/561598385 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f254c09d030 msgr2=0x7f254c13e0c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:32:49.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f2549d9b640 1 --2- 172.21.15.12:0/561598385 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f254c09d030 0x7f254c13e0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:49.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f2549d9b640 1 -- 172.21.15.12:0/561598385 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f254c138230 msgr2=0x7f254c1cd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:49.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f2549d9b640 1 --2- 172.21.15.12:0/561598385 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f254c138230 0x7f254c1cd8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:49.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f2549d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:32:49.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f2549d9b640 1 monclient: found mon.noname-a 2024-09-11T01:32:49.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f2549d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:49.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f2549d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:32:49.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.901+0000 7f2549d9b640 1 -- 172.21.15.12:0/561598385 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f254c121290 con 0x7f254c09d460 2024-09-11T01:32:49.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.903+0000 7f2549d9b640 1 -- 172.21.15.12:0/561598385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f254c09d460 msgr2=0x7f254c1cd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:49.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.903+0000 7f2549d9b640 1 --2- 172.21.15.12:0/561598385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f254c09d460 0x7f254c1cd380 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f253400e810 tx=0x7f253400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:49.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.903+0000 7f2542ffd640 1 -- 172.21.15.12:0/561598385 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2534017070 con 0x7f254c09d460 2024-09-11T01:32:49.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.903+0000 7f2542ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:49.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.903+0000 7f2542ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:32:49.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.903+0000 7f2542ffd640 10 monclient: dump: 2024-09-11T01:32:49.917 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:49.917 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:49.917 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:49.917 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:49.917 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:49.917 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:49.917 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:49.917 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:49.917 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:49.917 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:49.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.903+0000 7f2542ffd640 10 monclient: _finish_auth 0 2024-09-11T01:32:49.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.903+0000 7f2542ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:32:49.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.903+0000 7f2542ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:49.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.903+0000 7f2550bb4640 5 monclient: authenticate success, global_id 24315 2024-09-11T01:32:49.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.903+0000 7f2550bb4640 10 monclient: _renew_subs 2024-09-11T01:32:49.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.903+0000 7f2550bb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:49.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.903+0000 7f2542ffd640 1 -- 172.21.15.12:0/561598385 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2534004260 con 0x7f254c09d460 2024-09-11T01:32:49.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.903+0000 7f2550bb4640 1 -- 172.21.15.12:0/561598385 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f254c121570 con 0x7f254c09d460 2024-09-11T01:32:49.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.903+0000 7f2542ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:49.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.903+0000 7f2542ffd640 1 -- 172.21.15.12:0/561598385 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253400f040 con 0x7f254c09d460 2024-09-11T01:32:49.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.903+0000 7f2542ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:49.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.903+0000 7f2542ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:49.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.903+0000 7f2542ffd640 10 monclient: dump: 2024-09-11T01:32:49.918 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:49.919 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:49.919 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:49.919 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:49.919 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:49.919 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:49.919 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:49.919 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:49.919 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:49.919 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:49.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.904+0000 7f2550bb4640 10 monclient: _renew_subs 2024-09-11T01:32:49.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.903+0000 7f2548d99640 4 set_mon_vals no callback set 2024-09-11T01:32:49.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.904+0000 7f2550bb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:49.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.904+0000 7f2550bb4640 1 -- 172.21.15.12:0/561598385 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f254c120fb0 con 0x7f254c09d460 2024-09-11T01:32:49.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.904+0000 7f2550bb4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:49.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.904+0000 7f2550bb4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:49.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.904+0000 7f2550bb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:49.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.904+0000 7f2542ffd640 1 -- 172.21.15.12:0/561598385 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f2534012050 con 0x7f254c09d460 2024-09-11T01:32:49.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.904+0000 7f2550bb4640 1 -- 172.21.15.12:0/561598385 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f254c138610 con 0x7f254c09d460 2024-09-11T01:32:49.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.905+0000 7f2542ffd640 1 --2- 172.21.15.12:0/561598385 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f252c03e210 0x7f252c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:49.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.905+0000 7f2542ffd640 1 -- 172.21.15.12:0/561598385 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 4755+0+0 (secure 0 0 0) 0x7f25340526b0 con 0x7f254c09d460 2024-09-11T01:32:49.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.905+0000 7f254a59c640 1 --2- 172.21.15.12:0/561598385 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f252c03e210 0x7f252c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:49.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.905+0000 7f254a59c640 10 monclient: get_auth_request con 0x7f252c03e210 auth_method 0 2024-09-11T01:32:49.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.908+0000 7f254a59c640 1 --2- 172.21.15.12:0/561598385 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f252c03e210 0x7f252c0406d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f253c000f60 tx=0x7f253c007a30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:49.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.909+0000 7f2542ffd640 1 -- 172.21.15.12:0/561598385 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f25340510c0 con 0x7f254c09d460 2024-09-11T01:32:49.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.909+0000 7f2542ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:49.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:49.909+0000 7f2542ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:32:50.106 INFO:tasks.rados.rados.0.smithi071.stdout:598: done (5 left) 2024-09-11T01:32:50.106 INFO:tasks.rados.rados.0.smithi071.stdout:599: snap_remove snap 49 2024-09-11T01:32:50.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.172+0000 7f2550bb4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:32:50.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.172+0000 7f2550bb4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:32:50.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.172+0000 7f2550bb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:50.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.172+0000 7f2550bb4640 1 -- 172.21.15.12:0/561598385 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f254c0a21d0 con 0x7f254c09d460 2024-09-11T01:32:50.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.172+0000 7f2542ffd640 1 -- 172.21.15.12:0/561598385 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f253400cd30 con 0x7f254c09d460 2024-09-11T01:32:50.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.172+0000 7f2542ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:32:50.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.172+0000 7f2542ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:32:50.174 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:32:50.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.174+0000 7f2550bb4640 1 -- 172.21.15.12:0/561598385 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f252c03e210 msgr2=0x7f252c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:50.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.174+0000 7f2550bb4640 1 --2- 172.21.15.12:0/561598385 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f252c03e210 0x7f252c0406d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f253c000f60 tx=0x7f253c007a30 comp rx=0 tx=0).stop 2024-09-11T01:32:50.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.175+0000 7f2550bb4640 10 monclient: shutdown 2024-09-11T01:32:50.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.175+0000 7f2550bb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:50.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.175+0000 7f2550bb4640 1 -- 172.21.15.12:0/561598385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f254c09d460 msgr2=0x7f254c1cd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:50.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.175+0000 7f2550bb4640 1 --2- 172.21.15.12:0/561598385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f254c09d460 0x7f254c1cd380 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f253400e810 tx=0x7f253400ecd0 comp rx=0 tx=0).stop 2024-09-11T01:32:50.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.175+0000 7f2550bb4640 1 -- 172.21.15.12:0/561598385 shutdown_connections 2024-09-11T01:32:50.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.175+0000 7f2550bb4640 1 --2- 172.21.15.12:0/561598385 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f254c138230 0x7f254c1cd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:50.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.175+0000 7f2550bb4640 1 --2- 172.21.15.12:0/561598385 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f252c03e210 0x7f252c0406d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:50.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.175+0000 7f2550bb4640 1 --2- 172.21.15.12:0/561598385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f254c09d460 0x7f254c1cd380 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:50.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.175+0000 7f2550bb4640 1 --2- 172.21.15.12:0/561598385 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f254c09d030 0x7f254c13e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:50.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.175+0000 7f2550bb4640 1 -- 172.21.15.12:0/561598385 >> 172.21.15.12:0/561598385 conn(0x7f254c0a7120 msgr2=0x7f254c0a0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:50.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.175+0000 7f2550bb4640 1 -- 172.21.15.12:0/561598385 shutdown_connections 2024-09-11T01:32:50.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.175+0000 7f2550bb4640 1 -- 172.21.15.12:0/561598385 wait complete. 2024-09-11T01:32:50.188 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:32:50.188 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:32:50.188 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:32:50.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.290+0000 7ff824f9d640 10 monclient: get_monmap_and_config 2024-09-11T01:32:50.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.290+0000 7ff824f9d640 10 monclient: build_initial_monmap 2024-09-11T01:32:50.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.290+0000 7ff824f9d640 10 monclient: monmap: 2024-09-11T01:32:50.292 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:50.292 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:50.292 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:32:50.290784+0000 2024-09-11T01:32:50.292 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:32:50.290784+0000 2024-09-11T01:32:50.292 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:50.292 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:50.292 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:32:50.292 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:32:50.292 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:32:50.293 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:32:50.293 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:32:50.293 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:50.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.290+0000 7ff824f9d640 1 Processor -- start 2024-09-11T01:32:50.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.290+0000 7ff824f9d640 1 -- start start 2024-09-11T01:32:50.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.290+0000 7ff824f9d640 10 monclient: init 2024-09-11T01:32:50.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff824f9d640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:50.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff824f9d640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-11T01:32:50.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff824f9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff8200a4eb0 0x7ff82009cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:50.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff824f9d640 10 monclient(hunting): picked mon.noname-c con 0x7ff8200a4eb0 addr v2:172.21.15.71:3300/0 2024-09-11T01:32:50.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff824f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff8200a5290 addr v1:172.21.15.71:6789/0 2024-09-11T01:32:50.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff824f9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff82013d1a0 0x7ff82013f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:50.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff824f9d640 10 monclient(hunting): picked mon.noname-a con 0x7ff82013d1a0 addr v2:172.21.15.12:3300/0 2024-09-11T01:32:50.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff824f9d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8200a6050 con 0x7ff8200a5290 2024-09-11T01:32:50.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff824f9d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:50.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff824f9d640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff8200a6540 con 0x7ff82013d1a0 2024-09-11T01:32:50.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff824f9d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:50.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff824f9d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7ff8200a6710 con 0x7ff8200a4eb0 2024-09-11T01:32:50.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff824f9d640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:50.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff824f9d640 10 monclient(hunting): authenticate will time out at 976.733372s 2024-09-11T01:32:50.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff81ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff82013d1a0 0x7ff82013f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:50.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff81ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff82013d1a0 0x7ff82013f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42748/0 (socket says 172.21.15.12:42748) 2024-09-11T01:32:50.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff81ffff640 1 -- 172.21.15.12:0/1081120195 learned_addr learned my addr 172.21.15.12:0/1081120195 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:50.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff81ffff640 10 monclient(hunting): get_auth_request con 0x7ff82013d1a0 auth_method 0 2024-09-11T01:32:50.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff81ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:50.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff81ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:50.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff81ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:50.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff81ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:50.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff81ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:50.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff81ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:50.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff81f7fe640 1 --2- 172.21.15.12:0/1081120195 >> v2:172.21.15.71:3300/0 conn(0x7ff8200a4eb0 0x7ff82009cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:50.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff81f7fe640 10 monclient(hunting): get_auth_request con 0x7ff8200a4eb0 auth_method 0 2024-09-11T01:32:50.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff81f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:50.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff81f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:50.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff81f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:50.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff81ffff640 10 monclient(hunting): handle_auth_done global_id 24318 payload 274 2024-09-11T01:32:50.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff81ffff640 1 -- 172.21.15.12:0/1081120195 >> v2:172.21.15.71:3300/0 conn(0x7ff8200a4eb0 msgr2=0x7ff82009cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:50.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff81ffff640 1 --2- 172.21.15.12:0/1081120195 >> v2:172.21.15.71:3300/0 conn(0x7ff8200a4eb0 0x7ff82009cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:50.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff81ffff640 1 -- 172.21.15.12:0/1081120195 >> v1:172.21.15.71:6789/0 conn(0x7ff8200a5290 legacy=0x7ff8200a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:50.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff81ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:32:50.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff81ffff640 1 monclient: found mon.noname-a 2024-09-11T01:32:50.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff81ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:50.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff81ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:32:50.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.291+0000 7ff81ffff640 1 -- 172.21.15.12:0/1081120195 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff82009cb60 con 0x7ff82013d1a0 2024-09-11T01:32:50.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.294+0000 7ff81ffff640 1 -- 172.21.15.12:0/1081120195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff82013d1a0 msgr2=0x7ff82013f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:50.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.294+0000 7ff81ffff640 1 --2- 172.21.15.12:0/1081120195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff82013d1a0 0x7ff82013f590 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff8200a63e0 tx=0x7ff81802a540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8cf05309b8408b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:50.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.294+0000 7ff81e7fc640 1 -- 172.21.15.12:0/1081120195 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff818036070 con 0x7ff82013d1a0 2024-09-11T01:32:50.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.294+0000 7ff81e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:50.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.294+0000 7ff81e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:32:50.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.294+0000 7ff81e7fc640 10 monclient: dump: 2024-09-11T01:32:50.306 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:50.306 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:50.306 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:50.307 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:50.307 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:50.307 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:50.307 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:50.307 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:50.307 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:50.307 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:50.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.294+0000 7ff81e7fc640 10 monclient: _finish_auth 0 2024-09-11T01:32:50.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.294+0000 7ff81e7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:32:50.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.294+0000 7ff81e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:50.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.294+0000 7ff81e7fc640 1 -- 172.21.15.12:0/1081120195 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff818007800 con 0x7ff82013d1a0 2024-09-11T01:32:50.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.294+0000 7ff824f9d640 5 monclient: authenticate success, global_id 24318 2024-09-11T01:32:50.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.294+0000 7ff824f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:32:50.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.294+0000 7ff81e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:50.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.294+0000 7ff81e7fc640 1 -- 172.21.15.12:0/1081120195 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff818007bd0 con 0x7ff82013d1a0 2024-09-11T01:32:50.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.294+0000 7ff824f9d640 10 monclient: get_monmap_and_config success 2024-09-11T01:32:50.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.294+0000 7ff824f9d640 4 set_mon_vals no callback set 2024-09-11T01:32:50.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.294+0000 7ff81e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:50.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.294+0000 7ff81e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:50.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.294+0000 7ff81e7fc640 10 monclient: dump: 2024-09-11T01:32:50.308 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:50.308 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:50.308 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:50.308 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:50.308 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:50.308 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:50.308 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:50.309 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:50.309 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:50.309 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:50.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.295+0000 7ff824f9d640 10 monclient: shutdown 2024-09-11T01:32:50.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.295+0000 7ff824f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:50.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.295+0000 7ff824f9d640 1 -- 172.21.15.12:0/1081120195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff82013d1a0 msgr2=0x7ff82013f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:50.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.295+0000 7ff824f9d640 1 --2- 172.21.15.12:0/1081120195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff82013d1a0 0x7ff82013f590 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff8200a63e0 tx=0x7ff81802a540 comp rx=0 tx=0).stop 2024-09-11T01:32:50.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.295+0000 7ff824f9d640 1 -- 172.21.15.12:0/1081120195 shutdown_connections 2024-09-11T01:32:50.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.295+0000 7ff824f9d640 1 --2- 172.21.15.12:0/1081120195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff82013d1a0 0x7ff82013f590 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:50.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.295+0000 7ff824f9d640 1 --2- 172.21.15.12:0/1081120195 >> v2:172.21.15.71:3300/0 conn(0x7ff8200a4eb0 0x7ff82009cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:50.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.295+0000 7ff824f9d640 1 -- 172.21.15.12:0/1081120195 >> 172.21.15.12:0/1081120195 conn(0x7ff820068b30 msgr2=0x7ff820068f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:50.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.295+0000 7ff824f9d640 1 -- 172.21.15.12:0/1081120195 shutdown_connections 2024-09-11T01:32:50.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.295+0000 7ff824f9d640 1 -- 172.21.15.12:0/1081120195 wait complete. 2024-09-11T01:32:50.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.295+0000 7ff824f9d640 10 monclient: build_initial_monmap 2024-09-11T01:32:50.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.295+0000 7ff824f9d640 10 monclient: monmap: 2024-09-11T01:32:50.310 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:50.310 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:32:50.310 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:32:50.310 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:32:50.310 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:50.310 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:50.310 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:32:50.310 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:32:50.310 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:32:50.310 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:50.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.296+0000 7ff824f9d640 1 Processor -- start 2024-09-11T01:32:50.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.296+0000 7ff824f9d640 1 -- start start 2024-09-11T01:32:50.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.296+0000 7ff824f9d640 10 monclient: init 2024-09-11T01:32:50.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.296+0000 7ff824f9d640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:50.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.296+0000 7ff824f9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:32:50.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.296+0000 7ff824f9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff8200a4eb0 0x7ff8201ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:50.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.296+0000 7ff824f9d640 10 monclient(hunting): picked mon.noname-c con 0x7ff8200a4eb0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:50.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff824f9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8200a5290 0x7ff820137d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:50.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff824f9d640 10 monclient(hunting): picked mon.noname-b con 0x7ff8200a5290 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:32:50.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff824f9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff82013d1a0 0x7ff82013b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:50.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff824f9d640 10 monclient(hunting): picked mon.noname-a con 0x7ff82013d1a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:32:50.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff824f9d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:50.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff824f9d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff82006f850 con 0x7ff82013d1a0 2024-09-11T01:32:50.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff824f9d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:50.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff824f9d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff82006f6d0 con 0x7ff8200a5290 2024-09-11T01:32:50.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff824f9d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:50.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff824f9d640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff82013fad0 con 0x7ff8200a4eb0 2024-09-11T01:32:50.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff824f9d640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:50.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff824f9d640 10 monclient(hunting): authenticate will time out at 976.739056s 2024-09-11T01:32:50.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81f7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff8200a4eb0 0x7ff8201ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:50.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81f7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff8200a4eb0 0x7ff8201ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:39316/0 (socket says 172.21.15.12:39316) 2024-09-11T01:32:50.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff82013d1a0 0x7ff82013b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:50.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81f7fe640 1 -- 172.21.15.12:0/3261870528 learned_addr learned my addr 172.21.15.12:0/3261870528 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:50.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81f7fe640 10 monclient(hunting): get_auth_request con 0x7ff8200a4eb0 auth_method 0 2024-09-11T01:32:50.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:50.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:50.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:50.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81ffff640 10 monclient(hunting): get_auth_request con 0x7ff82013d1a0 auth_method 0 2024-09-11T01:32:50.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:50.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:50.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:50.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:50.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81f7fe640 1 -- 172.21.15.12:0/3261870528 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff8200a4eb0 msgr2=0x7ff8201ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:32:50.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:50.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81f7fe640 1 -- 172.21.15.12:0/3261870528 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff8200a4eb0 msgr2=0x7ff8201ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:32:50.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81f7fe640 1 --2- 172.21.15.12:0/3261870528 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff8200a4eb0 0x7ff8201ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:50.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:50.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81f7fe640 1 --2- 172.21.15.12:0/3261870528 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff8200a4eb0 0x7ff8201ccd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:50.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81effd640 1 --2- 172.21.15.12:0/3261870528 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8200a5290 0x7ff820137d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:50.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81effd640 10 monclient(hunting): get_auth_request con 0x7ff8200a5290 auth_method 0 2024-09-11T01:32:50.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:50.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:50.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:50.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81ffff640 10 monclient(hunting): handle_auth_done global_id 24321 payload 824 2024-09-11T01:32:50.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81ffff640 1 -- 172.21.15.12:0/3261870528 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff8200a4eb0 msgr2=0x7ff8201ccd10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:32:50.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81ffff640 1 --2- 172.21.15.12:0/3261870528 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff8200a4eb0 0x7ff8201ccd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:50.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81ffff640 1 -- 172.21.15.12:0/3261870528 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8200a5290 msgr2=0x7ff820137d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:50.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81ffff640 1 --2- 172.21.15.12:0/3261870528 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8200a5290 0x7ff820137d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:50.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:32:50.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81ffff640 1 monclient: found mon.noname-a 2024-09-11T01:32:50.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:50.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:32:50.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.297+0000 7ff81ffff640 1 -- 172.21.15.12:0/3261870528 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff820121280 con 0x7ff82013d1a0 2024-09-11T01:32:50.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.299+0000 7ff81ffff640 1 -- 172.21.15.12:0/3261870528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff82013d1a0 msgr2=0x7ff82013b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:50.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.299+0000 7ff81ffff640 1 --2- 172.21.15.12:0/3261870528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff82013d1a0 0x7ff82013b170 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff818030000 tx=0x7ff818004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:50.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.299+0000 7ff807fff640 1 -- 172.21.15.12:0/3261870528 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff818036050 con 0x7ff82013d1a0 2024-09-11T01:32:50.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.299+0000 7ff807fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:50.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.299+0000 7ff807fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:32:50.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.299+0000 7ff807fff640 10 monclient: dump: 2024-09-11T01:32:50.315 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:50.315 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:50.315 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:50.315 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:50.315 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:50.315 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:50.315 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:50.315 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:50.315 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:50.315 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:50.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.299+0000 7ff807fff640 10 monclient: _finish_auth 0 2024-09-11T01:32:50.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.299+0000 7ff807fff640 10 monclient: _check_auth_tickets 2024-09-11T01:32:50.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.299+0000 7ff807fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:50.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.299+0000 7ff807fff640 1 -- 172.21.15.12:0/3261870528 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff818042070 con 0x7ff82013d1a0 2024-09-11T01:32:50.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.299+0000 7ff807fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:50.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.299+0000 7ff807fff640 1 -- 172.21.15.12:0/3261870528 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff818031d90 con 0x7ff82013d1a0 2024-09-11T01:32:50.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.299+0000 7ff807fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:50.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.299+0000 7ff807fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:50.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.299+0000 7ff807fff640 10 monclient: dump: 2024-09-11T01:32:50.316 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:50.316 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:50.316 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:50.316 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:50.316 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:50.316 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:50.316 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:50.316 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:50.316 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:50.317 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:50.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.299+0000 7ff824f9d640 5 monclient: authenticate success, global_id 24321 2024-09-11T01:32:50.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.299+0000 7ff824f9d640 10 monclient: _renew_subs 2024-09-11T01:32:50.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.299+0000 7ff824f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:50.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.299+0000 7ff824f9d640 1 -- 172.21.15.12:0/3261870528 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff820122720 con 0x7ff82013d1a0 2024-09-11T01:32:50.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.299+0000 7ff81dffb640 4 set_mon_vals no callback set 2024-09-11T01:32:50.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.300+0000 7ff824f9d640 10 monclient: _renew_subs 2024-09-11T01:32:50.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.300+0000 7ff824f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:50.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.300+0000 7ff824f9d640 1 -- 172.21.15.12:0/3261870528 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff820122440 con 0x7ff82013d1a0 2024-09-11T01:32:50.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.300+0000 7ff824f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:50.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.300+0000 7ff824f9d640 1 --2- 172.21.15.12:0/3261870528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8201382b0 0x7ff82013b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:50.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.300+0000 7ff824f9d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:50.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.300+0000 7ff824f9d640 1 -- 172.21.15.12:0/3261870528 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff8200a9d20 con 0x7ff8201382b0 2024-09-11T01:32:50.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.300+0000 7ff807fff640 1 -- 172.21.15.12:0/3261870528 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff818047460 con 0x7ff82013d1a0 2024-09-11T01:32:50.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.301+0000 7ff81f7fe640 1 --2- 172.21.15.12:0/3261870528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8201382b0 0x7ff82013b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:50.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.301+0000 7ff81f7fe640 10 monclient: get_auth_request con 0x7ff8201382b0 auth_method 0 2024-09-11T01:32:50.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.301+0000 7ff81f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:50.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.301+0000 7ff81f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:50.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.301+0000 7ff81f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:50.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.301+0000 7ff81f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:50.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.301+0000 7ff81f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:50.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.301+0000 7ff81f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:50.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.301+0000 7ff807fff640 1 --2- 172.21.15.12:0/3261870528 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff80c03e210 0x7ff80c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:50.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.301+0000 7ff81f7fe640 10 monclient(hunting): handle_auth_done global_id 24324 payload 824 2024-09-11T01:32:50.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.301+0000 7ff81f7fe640 1 -- 172.21.15.12:0/3261870528 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff82009d230 con 0x7ff8201382b0 2024-09-11T01:32:50.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.301+0000 7ff807fff640 1 -- 172.21.15.12:0/3261870528 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 4783+0+0 (secure 0 0 0) 0x7ff818073e50 con 0x7ff82013d1a0 2024-09-11T01:32:50.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.301+0000 7ff81effd640 1 --2- 172.21.15.12:0/3261870528 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff80c03e210 0x7ff80c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:50.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.301+0000 7ff81effd640 10 monclient: get_auth_request con 0x7ff80c03e210 auth_method 0 2024-09-11T01:32:50.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.303+0000 7ff81f7fe640 1 -- 172.21.15.12:0/3261870528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8201382b0 msgr2=0x7ff82013b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:50.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.303+0000 7ff81f7fe640 1 --2- 172.21.15.12:0/3261870528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8201382b0 0x7ff82013b6b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff81000f530 tx=0x7ff81000f9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:50.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.303+0000 7ff807fff640 1 -- 172.21.15.12:0/3261870528 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff810019030 con 0x7ff8201382b0 2024-09-11T01:32:50.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.303+0000 7ff807fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:50.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.303+0000 7ff807fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:50.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.303+0000 7ff807fff640 10 monclient: dump: 2024-09-11T01:32:50.319 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:50.319 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:50.319 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:50.319 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:50.319 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:50.319 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:50.319 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:50.319 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:50.320 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:50.320 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:50.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.303+0000 7ff807fff640 1 -- 172.21.15.12:0/3261870528 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff810004020 con 0x7ff8201382b0 2024-09-11T01:32:50.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.304+0000 7ff807fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:50.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.304+0000 7ff807fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:32:50.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.304+0000 7ff807fff640 1 -- 172.21.15.12:0/3261870528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8201382b0 msgr2=0x7ff82013b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:50.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.304+0000 7ff807fff640 1 --2- 172.21.15.12:0/3261870528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8201382b0 0x7ff82013b6b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff81000f530 tx=0x7ff81000f9f0 comp rx=0 tx=0).stop 2024-09-11T01:32:50.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.304+0000 7ff807fff640 1 -- 172.21.15.12:0/3261870528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8201382b0 msgr2=0x7ff82013b6b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:32:50.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.304+0000 7ff807fff640 1 --2- 172.21.15.12:0/3261870528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8201382b0 0x7ff82013b6b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:50.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.304+0000 7ff81effd640 1 --2- 172.21.15.12:0/3261870528 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff80c03e210 0x7ff80c0406d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff820138b00 tx=0x7ff814009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:50.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.310+0000 7ff824f9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:32:50.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.310+0000 7ff824f9d640 1 --2- 172.21.15.12:0/3261870528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff82013bbf0 0x7ff82009df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:50.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.310+0000 7ff824f9d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:50.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.310+0000 7ff824f9d640 1 -- 172.21.15.12:0/3261870528 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff820121f00 con 0x7ff82013bbf0 2024-09-11T01:32:50.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.310+0000 7ff81f7fe640 1 --2- 172.21.15.12:0/3261870528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff82013bbf0 0x7ff82009df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:50.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.310+0000 7ff81f7fe640 10 monclient: get_auth_request con 0x7ff82013bbf0 auth_method 0 2024-09-11T01:32:50.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.310+0000 7ff81f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:50.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.310+0000 7ff81f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:50.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.310+0000 7ff81f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:50.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.310+0000 7ff81f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:50.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.310+0000 7ff81f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:50.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.310+0000 7ff81f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:50.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.310+0000 7ff81f7fe640 10 monclient(hunting): handle_auth_done global_id 24327 payload 824 2024-09-11T01:32:50.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.310+0000 7ff81f7fe640 1 -- 172.21.15.12:0/3261870528 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff82013bfd0 con 0x7ff82013bbf0 2024-09-11T01:32:50.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.312+0000 7ff81f7fe640 1 -- 172.21.15.12:0/3261870528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff82013bbf0 msgr2=0x7ff82009df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:50.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.312+0000 7ff81f7fe640 1 --2- 172.21.15.12:0/3261870528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff82013bbf0 0x7ff82009df40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff81000f530 tx=0x7ff8100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:50.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.312+0000 7ff807fff640 1 -- 172.21.15.12:0/3261870528 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff810003810 con 0x7ff82013bbf0 2024-09-11T01:32:50.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.312+0000 7ff807fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:50.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.312+0000 7ff807fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:50.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.312+0000 7ff807fff640 10 monclient: dump: 2024-09-11T01:32:50.324 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:50.324 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:50.324 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:50.324 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:50.324 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:50.324 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:50.324 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:50.324 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:50.324 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:50.324 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:50.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.313+0000 7ff807fff640 1 -- 172.21.15.12:0/3261870528 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4877 (secure 0 0 0) 0x7ff810003810 con 0x7ff82013bbf0 2024-09-11T01:32:50.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.313+0000 7ff807fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:32:50.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.313+0000 7ff807fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:32:50.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.313+0000 7ff807fff640 1 -- 172.21.15.12:0/3261870528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff82013bbf0 msgr2=0x7ff82009df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:50.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.313+0000 7ff807fff640 1 --2- 172.21.15.12:0/3261870528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff82013bbf0 0x7ff82009df40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff81000f530 tx=0x7ff8100046c0 comp rx=0 tx=0).stop 2024-09-11T01:32:50.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.313+0000 7ff807fff640 1 -- 172.21.15.12:0/3261870528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff82013bbf0 msgr2=0x7ff82009df40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:32:50.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.313+0000 7ff807fff640 1 --2- 172.21.15.12:0/3261870528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff82013bbf0 0x7ff82009df40 secure :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff81000f530 tx=0x7ff8100046c0 comp rx=0 tx=0).stop 2024-09-11T01:32:50.325 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:32:50.325 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:32:50.325 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:32:50.325 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:32:50.326 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 38, 2024-09-11T01:32:50.326 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 108259, 2024-09-11T01:32:50.326 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:32:50.326 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:32:50.326 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:32:50.326 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:32:50.326 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:50.326 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:32:50.326 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:32:50.326 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:32:50.326 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:32:50.326 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:50.326 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:50.326 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:50.327 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:50.327 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:50.327 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:50.327 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:50.327 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:50.327 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:50.327 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:50.327 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:50.327 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:50.327 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:32:50.327 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:32:50.327 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:50.327 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:50.327 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:50.328 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:50.328 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:50.328 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:50.328 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:50.328 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:50.328 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:50.328 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:50.328 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:50.328 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:50.328 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:50.328 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:32:50.328 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:32:50.328 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:32:50.329 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:32:50.329 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:32:50.329 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:32:50.329 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:32:50.329 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:32:50.329 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:32:50.329 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:32:50.329 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:32:50.329 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:32:50.329 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:32:50.329 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:32:50.329 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:32:50.329 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:32:50.330 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:32:50.330 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:50.330 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:50.330 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:50.330 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:50.330 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:50.330 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:50.330 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:50.330 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:50.330 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:50.330 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:50.330 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:50.331 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:50.331 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:32:50.331 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:50.331 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:32:50.331 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:50.331 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:32:50.331 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:32:50.331 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:50.331 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:50.331 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:50.331 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:50.331 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:32:50.331 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:50.332 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:50.332 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:50.332 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:50.332 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:32:50.332 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:50.332 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:50.332 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:50.332 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:50.332 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:32:50.332 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:32:50.332 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:50.332 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:50.332 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:50.332 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:50.333 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:50.333 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:32:50.333 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:32:50.333 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:50.333 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:50.333 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:50.333 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:50.333 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:32:50.333 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:50.333 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:50.333 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:50.333 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:50.333 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:32:50.334 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:50.334 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:50.334 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:50.334 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:50.334 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:32:50.334 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:32:50.334 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:50.334 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:50.334 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:50.334 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:50.334 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:50.334 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:32:50.334 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:32:50.334 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:50.334 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:50.335 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:50.335 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:50.335 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:32:50.335 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:50.335 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:50.335 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:50.335 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:50.335 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:32:50.335 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:50.335 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:50.335 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:50.335 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:50.335 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:32:50.335 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:32:50.336 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:50.336 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:50.336 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:50.336 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:50.336 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:50.336 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:50.336 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:32:50.336 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:32:50.336 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:50.336 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:50.336 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:50.336 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:32:50.336 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:50.336 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:50.337 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:32:50.337 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:50.337 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:50.337 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:50.337 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:32:50.337 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:50.337 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:50.337 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:32:50.337 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:50.337 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:50.337 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:50.337 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:32:50.337 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:50.337 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:50.338 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:32:50.338 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:50.338 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:50.338 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:50.338 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:32:50.338 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:50.338 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:50.338 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:50.338 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:32:50.338 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:32:50.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.313+0000 7ff824f9d640 1 -- 172.21.15.12:0/3261870528 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff80c03e210 msgr2=0x7ff80c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:50.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.313+0000 7ff824f9d640 1 --2- 172.21.15.12:0/3261870528 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff80c03e210 0x7ff80c0406d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff820138b00 tx=0x7ff814009040 comp rx=0 tx=0).stop 2024-09-11T01:32:50.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.313+0000 7ff824f9d640 10 monclient: shutdown 2024-09-11T01:32:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.313+0000 7ff824f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.313+0000 7ff824f9d640 1 -- 172.21.15.12:0/3261870528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff82013d1a0 msgr2=0x7ff82013b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.313+0000 7ff824f9d640 1 --2- 172.21.15.12:0/3261870528 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff82013d1a0 0x7ff82013b170 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff818030000 tx=0x7ff818004a40 comp rx=0 tx=0).stop 2024-09-11T01:32:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.313+0000 7ff81ffff640 1 -- 172.21.15.12:0/3261870528 reap_dead start 2024-09-11T01:32:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.313+0000 7ff81ffff640 1 -- 172.21.15.12:0/3261870528 reap_dead start 2024-09-11T01:32:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.313+0000 7ff824f9d640 1 -- 172.21.15.12:0/3261870528 shutdown_connections 2024-09-11T01:32:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.313+0000 7ff824f9d640 1 -- 172.21.15.12:0/3261870528 >> 172.21.15.12:0/3261870528 conn(0x7ff820068b30 msgr2=0x7ff8200a3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.313+0000 7ff824f9d640 1 -- 172.21.15.12:0/3261870528 shutdown_connections 2024-09-11T01:32:50.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.313+0000 7ff824f9d640 1 -- 172.21.15.12:0/3261870528 wait complete. 2024-09-11T01:32:50.340 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:32:50.341 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:32:50.341 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:32:50.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.441+0000 7f13da32c640 10 monclient: get_monmap_and_config 2024-09-11T01:32:50.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.441+0000 7f13da32c640 10 monclient: build_initial_monmap 2024-09-11T01:32:50.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.441+0000 7f13da32c640 10 monclient: monmap: 2024-09-11T01:32:50.443 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:50.443 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:50.443 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:32:50.442194+0000 2024-09-11T01:32:50.443 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:32:50.442194+0000 2024-09-11T01:32:50.443 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:50.443 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:50.443 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:32:50.444 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:32:50.444 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:32:50.444 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:32:50.444 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:32:50.444 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:50.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.442+0000 7f13da32c640 1 Processor -- start 2024-09-11T01:32:50.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.442+0000 7f13da32c640 1 -- start start 2024-09-11T01:32:50.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.442+0000 7f13da32c640 10 monclient: init 2024-09-11T01:32:50.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.442+0000 7f13da32c640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:50.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.442+0000 7f13da32c640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-11T01:32:50.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.442+0000 7f13da32c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f13d40a2e20 addr v1:172.21.15.71:6789/0 2024-09-11T01:32:50.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.442+0000 7f13da32c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f13d40a4970 0x7f13d406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:50.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.442+0000 7f13da32c640 10 monclient(hunting): picked mon.noname-a con 0x7f13d40a4970 addr v2:172.21.15.12:3300/0 2024-09-11T01:32:50.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.442+0000 7f13da32c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f13d406f910 addr v1:172.21.15.12:6789/0 2024-09-11T01:32:50.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.442+0000 7f13da32c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d406f4f0 con 0x7f13d406f910 2024-09-11T01:32:50.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.442+0000 7f13da32c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d40a3910 con 0x7f13d40a2e20 2024-09-11T01:32:50.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.442+0000 7f13da32c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:50.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.442+0000 7f13da32c640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f13d406ef30 con 0x7f13d40a4970 2024-09-11T01:32:50.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.442+0000 7f13da32c640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:50.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.442+0000 7f13da32c640 10 monclient(hunting): authenticate will time out at 976.884973s 2024-09-11T01:32:50.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.443+0000 7f13d3fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f13d40a2e20 0x7f13d40a3200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.12:50884/0 (socket says 172.21.15.12:50884) 2024-09-11T01:32:50.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.443+0000 7f13d3fff640 1 -- 172.21.15.12:0/2916946334 learned_addr learned my addr 172.21.15.12:0/2916946334 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:32:50.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.443+0000 7f13d37fe640 1 --2- 172.21.15.12:0/2916946334 >> v2:172.21.15.12:3300/0 conn(0x7f13d40a4970 0x7f13d406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:50.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.443+0000 7f13d37fe640 10 monclient(hunting): get_auth_request con 0x7f13d40a4970 auth_method 0 2024-09-11T01:32:50.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.443+0000 7f13d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:50.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.443+0000 7f13d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:50.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.443+0000 7f13d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:50.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.443+0000 7f13d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:50.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.443+0000 7f13d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:50.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.443+0000 7f13d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:50.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.443+0000 7f13d37fe640 10 monclient(hunting): handle_auth_done global_id 24330 payload 274 2024-09-11T01:32:50.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.443+0000 7f13d37fe640 1 -- 172.21.15.12:0/2916946334 >> v1:172.21.15.71:6789/0 conn(0x7f13d40a2e20 legacy=0x7f13d40a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:50.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.443+0000 7f13d37fe640 1 -- 172.21.15.12:0/2916946334 >> v1:172.21.15.12:6789/0 conn(0x7f13d406f910 legacy=0x7f13d413f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:50.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.443+0000 7f13d37fe640 10 monclient: _finish_hunting 0 2024-09-11T01:32:50.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.443+0000 7f13d37fe640 1 monclient: found mon.noname-a 2024-09-11T01:32:50.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.443+0000 7f13d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:50.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.443+0000 7f13d37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:32:50.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.443+0000 7f13d37fe640 1 -- 172.21.15.12:0/2916946334 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d40a3ae0 con 0x7f13d40a4970 2024-09-11T01:32:50.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.446+0000 7f13d37fe640 1 -- 172.21.15.12:0/2916946334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13d40a4970 msgr2=0x7f13d406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:50.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.446+0000 7f13d37fe640 1 --2- 172.21.15.12:0/2916946334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13d40a4970 0x7f13d406ded0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f13c80097a0 tx=0x7f13c8029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=524298a3db5d6662 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:50.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.446+0000 7f13d2ffd640 1 -- 172.21.15.12:0/2916946334 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c8037070 con 0x7f13d40a4970 2024-09-11T01:32:50.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.446+0000 7f13d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:50.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.446+0000 7f13d2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:32:50.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.446+0000 7f13d2ffd640 10 monclient: dump: 2024-09-11T01:32:50.456 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:50.456 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:50.456 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:50.456 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:50.456 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:50.456 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:50.456 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:50.456 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:50.457 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:50.457 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:50.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.446+0000 7f13d2ffd640 10 monclient: _finish_auth 0 2024-09-11T01:32:50.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.446+0000 7f13d2ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:32:50.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.446+0000 7f13d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:50.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.446+0000 7f13d2ffd640 1 -- 172.21.15.12:0/2916946334 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13c8032640 con 0x7f13d40a4970 2024-09-11T01:32:50.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.446+0000 7f13da32c640 5 monclient: authenticate success, global_id 24330 2024-09-11T01:32:50.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.446+0000 7f13da32c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:32:50.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.446+0000 7f13d2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:50.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.446+0000 7f13da32c640 10 monclient: get_monmap_and_config success 2024-09-11T01:32:50.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.446+0000 7f13da32c640 4 set_mon_vals no callback set 2024-09-11T01:32:50.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.446+0000 7f13d2ffd640 1 -- 172.21.15.12:0/2916946334 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c8032a10 con 0x7f13d40a4970 2024-09-11T01:32:50.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.446+0000 7f13d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:50.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.446+0000 7f13d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.446+0000 7f13d2ffd640 10 monclient: dump: 2024-09-11T01:32:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:50.458 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.446+0000 7f13da32c640 10 monclient: shutdown 2024-09-11T01:32:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.446+0000 7f13da32c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:50.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.446+0000 7f13da32c640 1 -- 172.21.15.12:0/2916946334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13d40a4970 msgr2=0x7f13d406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.446+0000 7f13da32c640 1 --2- 172.21.15.12:0/2916946334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13d40a4970 0x7f13d406ded0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f13c80097a0 tx=0x7f13c8029fd0 comp rx=0 tx=0).stop 2024-09-11T01:32:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.446+0000 7f13da32c640 1 -- 172.21.15.12:0/2916946334 shutdown_connections 2024-09-11T01:32:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.446+0000 7f13da32c640 1 --2- 172.21.15.12:0/2916946334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13d40a4970 0x7f13d406ded0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.446+0000 7f13da32c640 1 -- 172.21.15.12:0/2916946334 >> 172.21.15.12:0/2916946334 conn(0x7f13d40a8600 msgr2=0x7f13d40a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.447+0000 7f13da32c640 1 -- 172.21.15.12:0/2916946334 shutdown_connections 2024-09-11T01:32:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.447+0000 7f13da32c640 1 -- 172.21.15.12:0/2916946334 wait complete. 2024-09-11T01:32:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.447+0000 7f13da32c640 10 monclient: build_initial_monmap 2024-09-11T01:32:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.447+0000 7f13da32c640 10 monclient: monmap: 2024-09-11T01:32:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:32:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:32:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:32:50.459 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:50.460 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:50.460 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:32:50.460 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:32:50.460 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:32:50.460 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:50.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.447+0000 7f13da32c640 1 Processor -- start 2024-09-11T01:32:50.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13da32c640 1 -- start start 2024-09-11T01:32:50.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13da32c640 10 monclient: init 2024-09-11T01:32:50.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13da32c640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:50.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13da32c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:32:50.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13da32c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13d406f910 0x7f13d4137140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:50.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13da32c640 10 monclient(hunting): picked mon.noname-a con 0x7f13d406f910 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:32:50.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13da32c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13d40a2e20 0x7f13d4137680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:50.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13da32c640 10 monclient(hunting): picked mon.noname-b con 0x7f13d40a2e20 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:32:50.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13da32c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d40a4970 0x7f13d4137bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:50.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13da32c640 10 monclient(hunting): picked mon.noname-c con 0x7f13d40a4970 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:50.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13da32c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:50.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13da32c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f13d4140d60 con 0x7f13d406f910 2024-09-11T01:32:50.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13da32c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:50.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13da32c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f13d406ec50 con 0x7f13d40a2e20 2024-09-11T01:32:50.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13da32c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:50.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13da32c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f13d4140ee0 con 0x7f13d40a4970 2024-09-11T01:32:50.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13da32c640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:50.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13da32c640 10 monclient(hunting): authenticate will time out at 976.890626s 2024-09-11T01:32:50.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13d3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13d406f910 0x7f13d4137140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:50.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13d88c9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d40a4970 0x7f13d4137bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:50.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13d3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13d406f910 0x7f13d4137140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42800/0 (socket says 172.21.15.12:42800) 2024-09-11T01:32:50.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13d3fff640 1 -- 172.21.15.12:0/3088094054 learned_addr learned my addr 172.21.15.12:0/3088094054 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:50.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13d3fff640 10 monclient(hunting): get_auth_request con 0x7f13d406f910 auth_method 0 2024-09-11T01:32:50.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:50.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:50.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:50.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13d88c9640 10 monclient(hunting): get_auth_request con 0x7f13d40a4970 auth_method 0 2024-09-11T01:32:50.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13d88c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:50.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13d88c9640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:50.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13d88c9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:50.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:50.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:50.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13d88c9640 1 -- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d40a4970 msgr2=0x7f13d4137bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:32:50.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13d88c9640 1 -- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d40a4970 msgr2=0x7f13d4137bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:32:50.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13d88c9640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d40a4970 0x7f13d4137bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:50.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:50.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13d88c9640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d40a4970 0x7f13d4137bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:50.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13d37fe640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13d40a2e20 0x7f13d4137680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:50.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13d37fe640 10 monclient(hunting): get_auth_request con 0x7f13d40a2e20 auth_method 0 2024-09-11T01:32:50.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:50.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:50.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.448+0000 7f13d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:50.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.449+0000 7f13d3fff640 10 monclient(hunting): handle_auth_done global_id 24333 payload 824 2024-09-11T01:32:50.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.449+0000 7f13d3fff640 1 -- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d40a4970 msgr2=0x7f13d4137bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:32:50.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.449+0000 7f13d3fff640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d40a4970 0x7f13d4137bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:50.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.449+0000 7f13d3fff640 1 -- 172.21.15.12:0/3088094054 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13d40a2e20 msgr2=0x7f13d4137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:50.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.449+0000 7f13d3fff640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13d40a2e20 0x7f13d4137680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:50.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.449+0000 7f13d3fff640 10 monclient: _finish_hunting 0 2024-09-11T01:32:50.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.449+0000 7f13d3fff640 1 monclient: found mon.noname-a 2024-09-11T01:32:50.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.449+0000 7f13d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:50.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.449+0000 7f13d3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:32:50.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.449+0000 7f13d3fff640 1 -- 172.21.15.12:0/3088094054 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d413abe0 con 0x7f13d406f910 2024-09-11T01:32:50.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.449+0000 7f13d37fe640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13d40a2e20 0x7f13d4137680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:32:50.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.451+0000 7f13d3fff640 1 -- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13d406f910 msgr2=0x7f13d4137140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:50.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.451+0000 7f13d3fff640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13d406f910 0x7f13d4137140 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f13c400c460 tx=0x7f13c400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:50.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.451+0000 7f13d0ff9640 1 -- 172.21.15.12:0/3088094054 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c4007070 con 0x7f13d406f910 2024-09-11T01:32:50.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.451+0000 7f13d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:50.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.451+0000 7f13d0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:32:50.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.451+0000 7f13d0ff9640 10 monclient: dump: 2024-09-11T01:32:50.465 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:50.465 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:50.465 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:50.465 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:50.465 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:50.465 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:50.465 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:50.465 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:50.465 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:50.465 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:50.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.451+0000 7f13d0ff9640 10 monclient: _finish_auth 0 2024-09-11T01:32:50.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.451+0000 7f13d0ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:32:50.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.451+0000 7f13d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:50.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.451+0000 7f13d0ff9640 1 -- 172.21.15.12:0/3088094054 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13c4007230 con 0x7f13d406f910 2024-09-11T01:32:50.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.451+0000 7f13d0ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:50.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.451+0000 7f13d0ff9640 1 -- 172.21.15.12:0/3088094054 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c4007620 con 0x7f13d406f910 2024-09-11T01:32:50.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.451+0000 7f13d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:50.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.451+0000 7f13d0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:50.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.451+0000 7f13d0ff9640 10 monclient: dump: 2024-09-11T01:32:50.466 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:50.466 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:50.466 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:50.466 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:50.466 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:50.466 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:50.466 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:50.466 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:50.466 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:50.467 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:50.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.451+0000 7f13da32c640 5 monclient: authenticate success, global_id 24333 2024-09-11T01:32:50.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.451+0000 7f13da32c640 10 monclient: _renew_subs 2024-09-11T01:32:50.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.451+0000 7f13da32c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:50.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.451+0000 7f13da32c640 1 -- 172.21.15.12:0/3088094054 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13d413aec0 con 0x7f13d406f910 2024-09-11T01:32:50.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.451+0000 7f13d27fc640 4 set_mon_vals no callback set 2024-09-11T01:32:50.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.451+0000 7f13da32c640 10 monclient: _renew_subs 2024-09-11T01:32:50.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.451+0000 7f13da32c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:50.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.451+0000 7f13da32c640 1 -- 172.21.15.12:0/3088094054 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13d4140800 con 0x7f13d406f910 2024-09-11T01:32:50.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.452+0000 7f13da32c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:50.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.452+0000 7f13da32c640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cda30 0x7f13d402d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:50.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.452+0000 7f13da32c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:50.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.452+0000 7f13da32c640 1 -- 172.21.15.12:0/3088094054 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f13d406a060 con 0x7f13d41cda30 2024-09-11T01:32:50.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.452+0000 7f13d37fe640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cda30 0x7f13d402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:50.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.452+0000 7f13d37fe640 10 monclient: get_auth_request con 0x7f13d41cda30 auth_method 0 2024-09-11T01:32:50.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.452+0000 7f13d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:50.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.452+0000 7f13d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:50.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.452+0000 7f13d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:50.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.452+0000 7f13d0ff9640 1 -- 172.21.15.12:0/3088094054 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f13c401b400 con 0x7f13d406f910 2024-09-11T01:32:50.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.452+0000 7f13d37fe640 1 -- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cda30 msgr2=0x7f13d402d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:32:50.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.452+0000 7f13d37fe640 1 -- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cda30 msgr2=0x7f13d402d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:32:50.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.452+0000 7f13d37fe640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cda30 0x7f13d402d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:50.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.452+0000 7f13d37fe640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cda30 0x7f13d402d440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:50.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.452+0000 7f13d0ff9640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f13c003e290 0x7f13c0040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:50.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.452+0000 7f13d0ff9640 1 -- 172.21.15.12:0/3088094054 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 4783+0+0 (secure 0 0 0) 0x7f13c4052970 con 0x7f13d406f910 2024-09-11T01:32:50.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.453+0000 7f13d88c9640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f13c003e290 0x7f13c0040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:50.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.453+0000 7f13d88c9640 10 monclient: get_auth_request con 0x7f13c003e290 auth_method 0 2024-09-11T01:32:50.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.455+0000 7f13d88c9640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f13c003e290 0x7f13c0040750 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f13cc007f50 tx=0x7f13cc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.654+0000 7f13d37fe640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cda30 0x7f13d402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.654+0000 7f13d37fe640 10 monclient: get_auth_request con 0x7f13d41cda30 auth_method 0 2024-09-11T01:32:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.654+0000 7f13d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.654+0000 7f13d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.654+0000 7f13d37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:32:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.654+0000 7f13d37fe640 1 -- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cda30 msgr2=0x7f13d402d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:32:50.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.654+0000 7f13d37fe640 1 -- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cda30 msgr2=0x7f13d402d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:32:50.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.654+0000 7f13d37fe640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cda30 0x7f13d402d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:50.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:50.654+0000 7f13d37fe640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cda30 0x7f13d402d440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:32:51.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.055+0000 7f13d37fe640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cda30 0x7f13d402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:51.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.055+0000 7f13d37fe640 10 monclient: get_auth_request con 0x7f13d41cda30 auth_method 0 2024-09-11T01:32:51.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.055+0000 7f13d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:51.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.055+0000 7f13d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:51.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.055+0000 7f13d37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:32:51.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.055+0000 7f13d37fe640 1 -- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cda30 msgr2=0x7f13d402d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:32:51.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.055+0000 7f13d37fe640 1 -- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cda30 msgr2=0x7f13d402d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:32:51.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.055+0000 7f13d37fe640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cda30 0x7f13d402d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:32:51.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.055+0000 7f13d37fe640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cda30 0x7f13d402d440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:32:51.109 INFO:tasks.rados.rados.0.smithi071.stdout:599: done (5 left) 2024-09-11T01:32:51.109 INFO:tasks.rados.rados.0.smithi071.stdout:600: write oid 7 current snap is 57 2024-09-11T01:32:51.109 INFO:tasks.rados.rados.0.smithi071.stdout:600: seq_num 237 ranges {783259=640956,1918703=613070,3008661=405467,3821939=161320} 2024-09-11T01:32:51.117 INFO:tasks.rados.rados.0.smithi071.stdout:600: writing smithi07134777-7 from 783259 to 1424215 tid 1 2024-09-11T01:32:51.123 INFO:tasks.rados.rados.0.smithi071.stdout:600: writing smithi07134777-7 from 1918703 to 2531773 tid 2 2024-09-11T01:32:51.128 INFO:tasks.rados.rados.0.smithi071.stdout:600: writing smithi07134777-7 from 3008661 to 3414128 tid 3 2024-09-11T01:32:51.133 INFO:tasks.rados.rados.0.smithi071.stdout:600: writing smithi07134777-7 from 3821939 to 3983259 tid 4 2024-09-11T01:32:51.133 INFO:tasks.rados.rados.0.smithi071.stdout:601: read oid 21 snap -1 2024-09-11T01:32:51.133 INFO:tasks.rados.rados.0.smithi071.stdout:601: expect deleted 2024-09-11T01:32:51.133 INFO:tasks.rados.rados.0.smithi071.stdout:602: snap_create 2024-09-11T01:32:51.133 INFO:tasks.rados.rados.0.smithi071.stdout:592: finishing write tid 4 to smithi07134777-11 2024-09-11T01:32:51.133 INFO:tasks.rados.rados.0.smithi071.stdout:592: finishing write tid 5 to smithi07134777-11 2024-09-11T01:32:51.133 INFO:tasks.rados.rados.0.smithi071.stdout:592: oid 11 updating version 0 to 213 2024-09-11T01:32:51.133 INFO:tasks.rados.rados.0.smithi071.stdout:592: oid 11 version 213 is already newer than 211 2024-09-11T01:32:51.133 INFO:tasks.rados.rados.0.smithi071.stdout:592: oid 11 version 213 is already newer than 212 2024-09-11T01:32:51.134 INFO:tasks.rados.rados.0.smithi071.stdout:592: oid 11 updating version 213 to 214 2024-09-11T01:32:51.134 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 214 (ObjNum 234 snap 57 seq_num 234) dirty exists 2024-09-11T01:32:51.134 INFO:tasks.rados.rados.0.smithi071.stdout:592: left oid 11 (ObjNum 234 snap 57 seq_num 234) 2024-09-11T01:32:51.134 INFO:tasks.rados.rados.0.smithi071.stdout:594: finishing write tid 1 to smithi07134777-25 2024-09-11T01:32:51.134 INFO:tasks.rados.rados.0.smithi071.stdout:594: finishing write tid 2 to smithi07134777-25 2024-09-11T01:32:51.134 INFO:tasks.rados.rados.0.smithi071.stdout:594: finishing write tid 3 to smithi07134777-25 2024-09-11T01:32:51.134 INFO:tasks.rados.rados.0.smithi071.stdout:594: finishing write tid 4 to smithi07134777-25 2024-09-11T01:32:51.134 INFO:tasks.rados.rados.0.smithi071.stdout:594: finishing write tid 5 to smithi07134777-25 2024-09-11T01:32:51.134 INFO:tasks.rados.rados.0.smithi071.stdout:594: oid 25 updating version 0 to 66 2024-09-11T01:32:51.134 INFO:tasks.rados.rados.0.smithi071.stdout:594: oid 25 updating version 66 to 68 2024-09-11T01:32:51.134 INFO:tasks.rados.rados.0.smithi071.stdout:594: oid 25 updating version 68 to 69 2024-09-11T01:32:51.134 INFO:tasks.rados.rados.0.smithi071.stdout:594: oid 25 version 69 is already newer than 67 2024-09-11T01:32:51.134 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 69 (ObjNum 235 snap 57 seq_num 235) dirty exists 2024-09-11T01:32:51.134 INFO:tasks.rados.rados.0.smithi071.stdout:594: left oid 25 (ObjNum 235 snap 57 seq_num 235) 2024-09-11T01:32:51.135 INFO:tasks.rados.rados.0.smithi071.stdout:595: finishing write tid 1 to smithi07134777-10 2024-09-11T01:32:51.135 INFO:tasks.rados.rados.0.smithi071.stdout:595: finishing write tid 2 to smithi07134777-10 2024-09-11T01:32:51.135 INFO:tasks.rados.rados.0.smithi071.stdout:597: finishing rollback tid 0 to smithi07134777-32 2024-09-11T01:32:51.135 INFO:tasks.rados.rados.0.smithi071.stdout:597: finishing rollback tid 1 to smithi07134777-32 2024-09-11T01:32:51.135 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 179 (ObjNum 33 snap 0 seq_num 2370935408) dirty dne 2024-09-11T01:32:51.135 INFO:tasks.rados.rados.0.smithi071.stdout:595: finishing write tid 3 to smithi07134777-10 2024-09-11T01:32:51.135 INFO:tasks.rados.rados.0.smithi071.stdout:595: finishing write tid 4 to smithi07134777-10 2024-09-11T01:32:51.135 INFO:tasks.rados.rados.0.smithi071.stdout:595: finishing write tid 5 to smithi07134777-10 2024-09-11T01:32:51.135 INFO:tasks.rados.rados.0.smithi071.stdout:595: oid 10 updating version 0 to 118 2024-09-11T01:32:51.135 INFO:tasks.rados.rados.0.smithi071.stdout:595: oid 10 version 118 is already newer than 117 2024-09-11T01:32:51.135 INFO:tasks.rados.rados.0.smithi071.stdout:595: oid 10 updating version 118 to 120 2024-09-11T01:32:51.135 INFO:tasks.rados.rados.0.smithi071.stdout:595: oid 10 version 120 is already newer than 119 2024-09-11T01:32:51.135 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 120 (ObjNum 236 snap 57 seq_num 236) dirty exists 2024-09-11T01:32:51.136 INFO:tasks.rados.rados.0.smithi071.stdout:595: left oid 10 (ObjNum 236 snap 57 seq_num 236) 2024-09-11T01:32:51.136 INFO:tasks.rados.rados.0.smithi071.stdout:600: finishing write tid 1 to smithi07134777-7 2024-09-11T01:32:51.136 INFO:tasks.rados.rados.0.smithi071.stdout:600: finishing write tid 2 to smithi07134777-7 2024-09-11T01:32:51.136 INFO:tasks.rados.rados.0.smithi071.stdout:600: finishing write tid 3 to smithi07134777-7 2024-09-11T01:32:51.137 INFO:tasks.rados.rados.0.smithi071.stdout:600: finishing write tid 4 to smithi07134777-7 2024-09-11T01:32:51.137 INFO:tasks.rados.rados.0.smithi071.stdout:600: finishing write tid 5 to smithi07134777-7 2024-09-11T01:32:51.137 INFO:tasks.rados.rados.0.smithi071.stdout:600: finishing write tid 6 to smithi07134777-7 2024-09-11T01:32:51.137 INFO:tasks.rados.rados.0.smithi071.stdout:600: oid 7 updating version 0 to 281 2024-09-11T01:32:51.137 INFO:tasks.rados.rados.0.smithi071.stdout:600: oid 7 updating version 281 to 286 2024-09-11T01:32:51.137 INFO:tasks.rados.rados.0.smithi071.stdout:600: oid 7 version 286 is already newer than 280 2024-09-11T01:32:51.137 INFO:tasks.rados.rados.0.smithi071.stdout:600: oid 7 version 286 is already newer than 285 2024-09-11T01:32:51.137 INFO:tasks.rados.rados.0.smithi071.stdout:600: oid 7 version 286 is already newer than 282 2024-09-11T01:32:51.137 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 286 (ObjNum 237 snap 57 seq_num 237) dirty exists 2024-09-11T01:32:51.137 INFO:tasks.rados.rados.0.smithi071.stdout:600: left oid 7 (ObjNum 237 snap 57 seq_num 237) 2024-09-11T01:32:51.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.857+0000 7f13d37fe640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cda30 0x7f13d402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:51.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.857+0000 7f13d37fe640 10 monclient: get_auth_request con 0x7f13d41cda30 auth_method 0 2024-09-11T01:32:51.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.857+0000 7f13d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:51.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.857+0000 7f13d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:51.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.857+0000 7f13d37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:32:51.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.857+0000 7f13d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:51.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.857+0000 7f13d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:51.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.857+0000 7f13d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:51.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.857+0000 7f13d37fe640 10 monclient(hunting): handle_auth_done global_id 44099 payload 824 2024-09-11T01:32:51.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.857+0000 7f13d37fe640 1 -- 172.21.15.12:0/3088094054 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f13d413b2c0 con 0x7f13d41cda30 2024-09-11T01:32:51.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.859+0000 7f13d37fe640 1 -- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cda30 msgr2=0x7f13d402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:51.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.859+0000 7f13d37fe640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cda30 0x7f13d402d440 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f13c8034000 tx=0x7f13c8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:51.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.860+0000 7f13d0ff9640 1 -- 172.21.15.12:0/3088094054 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c804c070 con 0x7f13d41cda30 2024-09-11T01:32:51.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.860+0000 7f13d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:51.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.860+0000 7f13d0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:32:51.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.860+0000 7f13d0ff9640 10 monclient: dump: 2024-09-11T01:32:51.861 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:51.862 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:51.862 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:51.862 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:51.862 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:51.862 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:51.862 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:51.862 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:51.862 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:51.862 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:51.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.860+0000 7f13d0ff9640 1 -- 172.21.15.12:0/3088094054 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f13c8047630 con 0x7f13d41cda30 2024-09-11T01:32:51.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.860+0000 7f13d0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:51.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.860+0000 7f13d0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:32:51.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.860+0000 7f13d0ff9640 1 -- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cda30 msgr2=0x7f13d402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:51.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.860+0000 7f13d0ff9640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cda30 0x7f13d402d440 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f13c8034000 tx=0x7f13c8004730 comp rx=0 tx=0).stop 2024-09-11T01:32:51.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.860+0000 7f13d0ff9640 1 -- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cda30 msgr2=0x7f13d402d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:32:51.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.860+0000 7f13d0ff9640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cda30 0x7f13d402d440 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f13c8034000 tx=0x7f13c8004730 comp rx=0 tx=0).stop 2024-09-11T01:32:51.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.867+0000 7f13da32c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:32:51.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.867+0000 7f13da32c640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cde10 0x7f13d402d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:51.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.867+0000 7f13da32c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:51.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.867+0000 7f13da32c640 1 -- 172.21.15.12:0/3088094054 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f13d4072350 con 0x7f13d41cde10 2024-09-11T01:32:51.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.867+0000 7f13d37fe640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cde10 0x7f13d402d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:51.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.867+0000 7f13d37fe640 10 monclient: get_auth_request con 0x7f13d41cde10 auth_method 0 2024-09-11T01:32:51.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.867+0000 7f13d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:51.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.867+0000 7f13d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:51.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.867+0000 7f13d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:51.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.867+0000 7f13d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:51.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.867+0000 7f13d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:51.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.867+0000 7f13d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:51.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.868+0000 7f13d37fe640 10 monclient(hunting): handle_auth_done global_id 44102 payload 824 2024-09-11T01:32:51.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.868+0000 7f13d37fe640 1 -- 172.21.15.12:0/3088094054 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f13d406ded0 con 0x7f13d41cde10 2024-09-11T01:32:51.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.870+0000 7f13d37fe640 1 -- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cde10 msgr2=0x7f13d402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:51.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.870+0000 7f13d37fe640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cde10 0x7f13d402d980 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f13d4070aa0 tx=0x7f13c8009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:51.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.870+0000 7f13d0ff9640 1 -- 172.21.15.12:0/3088094054 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c804c040 con 0x7f13d41cde10 2024-09-11T01:32:51.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.870+0000 7f13d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:51.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.870+0000 7f13d0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:32:51.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.870+0000 7f13d0ff9640 10 monclient: dump: 2024-09-11T01:32:51.873 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:51.873 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:51.873 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:51.873 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:51.873 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:51.873 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:51.873 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:51.873 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:51.873 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:51.873 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:51.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.870+0000 7f13d0ff9640 1 -- 172.21.15.12:0/3088094054 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4540 (secure 0 0 0) 0x7f13c8003610 con 0x7f13d41cde10 2024-09-11T01:32:51.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.870+0000 7f13d0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:32:51.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.870+0000 7f13d0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:32:51.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.870+0000 7f13d0ff9640 1 -- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cde10 msgr2=0x7f13d402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:51.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.870+0000 7f13d0ff9640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cde10 0x7f13d402d980 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f13d4070aa0 tx=0x7f13c8009190 comp rx=0 tx=0).stop 2024-09-11T01:32:51.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.870+0000 7f13d0ff9640 1 -- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cde10 msgr2=0x7f13d402d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:32:51.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.870+0000 7f13d0ff9640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13d41cde10 0x7f13d402d980 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f13d4070aa0 tx=0x7f13c8009190 comp rx=0 tx=0).stop 2024-09-11T01:32:51.874 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:32:51.874 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:32:51.874 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:32:51.874 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:32:51.874 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 38, 2024-09-11T01:32:51.874 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 15814, 2024-09-11T01:32:51.874 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:32:51.875 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:32:51.875 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:32:51.875 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:32:51.875 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:51.875 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 4, 2024-09-11T01:32:51.875 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:32:51.875 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:32:51.875 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:32:51.875 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:51.875 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:51.875 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:51.875 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:51.875 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:51.875 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:51.876 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:51.876 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:51.876 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:51.876 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:51.876 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:51.876 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:51.876 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:32:51.876 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:32:51.876 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:51.876 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:51.876 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:51.876 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:51.876 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:51.877 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:51.877 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:51.877 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:51.877 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:51.877 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:51.877 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:51.877 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:51.877 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:51.877 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:32:51.877 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:32:51.877 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:32:51.877 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:32:51.877 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:32:51.877 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:32:51.878 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:32:51.878 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:32:51.878 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:32:51.878 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:32:51.878 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:32:51.878 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:32:51.878 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:32:51.878 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:32:51.878 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:32:51.878 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:32:51.878 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:32:51.878 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:51.878 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:51.878 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:51.879 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:51.879 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:51.879 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:51.879 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:51.879 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:51.879 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:51.879 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:51.879 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:51.879 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:51.879 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:32:51.879 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:51.879 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:32:51.879 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:51.880 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:32:51.880 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:32:51.880 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:51.880 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:51.880 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:51.880 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:51.880 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:32:51.880 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:51.880 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:51.881 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:51.881 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:51.881 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:32:51.881 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:51.881 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:51.881 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:51.881 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:51.881 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:32:51.881 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:32:51.881 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:51.881 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:51.881 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:51.881 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:51.881 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:51.882 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:32:51.882 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:32:51.882 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:51.882 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:51.882 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:51.882 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:51.882 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:32:51.882 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:51.882 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:51.882 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:51.882 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:51.882 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:32:51.882 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:51.882 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:51.883 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:51.883 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:51.883 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:32:51.883 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:32:51.883 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:51.883 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:51.883 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:51.883 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:51.883 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:51.883 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:32:51.883 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:32:51.883 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:51.883 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:51.883 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:51.884 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:51.884 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:32:51.884 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:51.884 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:51.884 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:51.884 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:51.884 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:32:51.884 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:51.884 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:51.884 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:51.884 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:51.884 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:32:51.884 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:32:51.884 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:51.885 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:51.885 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:51.885 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:51.885 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:51.885 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:51.885 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:32:51.885 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:32:51.885 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:51.885 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:51.885 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:51.885 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:32:51.885 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:51.885 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:51.885 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:32:51.886 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:51.886 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:51.886 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:51.886 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:32:51.886 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:51.886 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:51.886 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:51.886 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:32:51.886 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:32:51.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.871+0000 7f13da32c640 1 -- 172.21.15.12:0/3088094054 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f13c003e290 msgr2=0x7f13c0040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:51.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.871+0000 7f13da32c640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f13c003e290 0x7f13c0040750 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f13cc007f50 tx=0x7f13cc009040 comp rx=0 tx=0).stop 2024-09-11T01:32:51.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.871+0000 7f13da32c640 10 monclient: shutdown 2024-09-11T01:32:51.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.871+0000 7f13da32c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:51.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.871+0000 7f13da32c640 1 -- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13d406f910 msgr2=0x7f13d4137140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:51.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.871+0000 7f13da32c640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13d406f910 0x7f13d4137140 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f13c400c460 tx=0x7f13c400c920 comp rx=0 tx=0).stop 2024-09-11T01:32:51.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.871+0000 7f13d88c9640 1 -- 172.21.15.12:0/3088094054 reap_dead start 2024-09-11T01:32:51.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.871+0000 7f13da32c640 1 -- 172.21.15.12:0/3088094054 shutdown_connections 2024-09-11T01:32:51.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.871+0000 7f13da32c640 1 --2- 172.21.15.12:0/3088094054 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13d406f910 0x7f13d4137140 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:51.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.871+0000 7f13da32c640 1 -- 172.21.15.12:0/3088094054 >> 172.21.15.12:0/3088094054 conn(0x7f13d40a8600 msgr2=0x7f13d409e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:51.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.871+0000 7f13da32c640 1 -- 172.21.15.12:0/3088094054 shutdown_connections 2024-09-11T01:32:51.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.871+0000 7f13da32c640 1 -- 172.21.15.12:0/3088094054 wait complete. 2024-09-11T01:32:51.889 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:32:51.889 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:32:51.889 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:32:52.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.998+0000 7f41f6265640 10 monclient: get_monmap_and_config 2024-09-11T01:32:52.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.998+0000 7f41f6265640 10 monclient: build_initial_monmap 2024-09-11T01:32:52.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.998+0000 7f41f6265640 10 monclient: monmap: 2024-09-11T01:32:52.000 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:52.000 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:52.000 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:32:51.999444+0000 2024-09-11T01:32:52.000 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:32:51.999444+0000 2024-09-11T01:32:52.001 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:52.001 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:52.001 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:32:52.001 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:32:52.001 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:32:52.001 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:32:52.001 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:32:52.001 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:52.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.999+0000 7f41f6265640 1 Processor -- start 2024-09-11T01:32:52.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.999+0000 7f41f6265640 1 -- start start 2024-09-11T01:32:52.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:51.999+0000 7f41f6265640 10 monclient: init 2024-09-11T01:32:52.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41f6265640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:52.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41f6265640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-11T01:32:52.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41f6265640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f41f00a2e20 0x7f41f006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:52.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41f6265640 10 monclient(hunting): picked mon.noname-b con 0x7f41f00a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:52.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41f6265640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f41f00a4970 0x7f41f00a3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:52.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41f6265640 10 monclient(hunting): picked mon.noname-a con 0x7f41f00a4970 addr v2:172.21.15.12:3300/0 2024-09-11T01:32:52.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41f6265640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f41f006f910 addr v1:172.21.15.12:6789/0 2024-09-11T01:32:52.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41f6265640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41f006f4f0 con 0x7f41f006f910 2024-09-11T01:32:52.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41f6265640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:52.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41f6265640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f41f006ef30 con 0x7f41f00a4970 2024-09-11T01:32:52.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41f6265640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:52.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41f6265640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f41f006ec50 con 0x7f41f00a2e20 2024-09-11T01:32:52.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41f6265640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:52.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41f6265640 10 monclient(hunting): authenticate will time out at 978.442254s 2024-09-11T01:32:52.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41e7fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f41f00a4970 0x7f41f00a3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:52.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41effff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f41f00a2e20 0x7f41f006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:52.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41e7fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f41f00a4970 0x7f41f00a3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42816/0 (socket says 172.21.15.12:42816) 2024-09-11T01:32:52.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41effff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f41f00a2e20 0x7f41f006ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:39362/0 (socket says 172.21.15.12:39362) 2024-09-11T01:32:52.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41e7fff640 1 -- 172.21.15.12:0/3619732324 learned_addr learned my addr 172.21.15.12:0/3619732324 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:52.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41e7fff640 10 monclient(hunting): get_auth_request con 0x7f41f00a4970 auth_method 0 2024-09-11T01:32:52.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:52.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:52.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:52.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41effff640 10 monclient(hunting): get_auth_request con 0x7f41f00a2e20 auth_method 0 2024-09-11T01:32:52.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:52.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41effff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:52.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:52.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:52.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:52.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:52.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:52.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:52.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:52.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41effff640 10 monclient(hunting): handle_auth_done global_id 44105 payload 274 2024-09-11T01:32:52.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41effff640 1 -- 172.21.15.12:0/3619732324 >> v2:172.21.15.12:3300/0 conn(0x7f41f00a4970 msgr2=0x7f41f00a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:52.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41effff640 1 --2- 172.21.15.12:0/3619732324 >> v2:172.21.15.12:3300/0 conn(0x7f41f00a4970 0x7f41f00a3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:52.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41effff640 1 -- 172.21.15.12:0/3619732324 >> v1:172.21.15.12:6789/0 conn(0x7f41f006f910 legacy=0x7f41f0137fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:52.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41effff640 10 monclient: _finish_hunting 0 2024-09-11T01:32:52.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41effff640 1 monclient: found mon.noname-b 2024-09-11T01:32:52.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:52.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41effff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:32:52.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.000+0000 7f41effff640 1 -- 172.21.15.12:0/3619732324 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41f00a3740 con 0x7f41f00a2e20 2024-09-11T01:32:52.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.003+0000 7f41effff640 1 -- 172.21.15.12:0/3619732324 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f41f00a2e20 msgr2=0x7f41f006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:52.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.003+0000 7f41effff640 1 --2- 172.21.15.12:0/3619732324 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f41f00a2e20 0x7f41f006ded0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f41e00097a0 tx=0x7f41e0029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=210fa0400d85b7b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:52.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.003+0000 7f41ef7fe640 1 -- 172.21.15.12:0/3619732324 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e0037070 con 0x7f41f00a2e20 2024-09-11T01:32:52.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.003+0000 7f41ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:52.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.003+0000 7f41ef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:32:52.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.003+0000 7f41ef7fe640 10 monclient: dump: 2024-09-11T01:32:52.014 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:52.014 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:52.014 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:52.014 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:52.015 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:52.015 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:52.015 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:52.015 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:52.015 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:52.015 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:52.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.003+0000 7f41ef7fe640 10 monclient: _finish_auth 0 2024-09-11T01:32:52.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.003+0000 7f41ef7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:32:52.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.003+0000 7f41ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:52.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.003+0000 7f41ef7fe640 1 -- 172.21.15.12:0/3619732324 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41e0032440 con 0x7f41f00a2e20 2024-09-11T01:32:52.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.003+0000 7f41ef7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:52.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.003+0000 7f41ef7fe640 1 -- 172.21.15.12:0/3619732324 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e0032810 con 0x7f41f00a2e20 2024-09-11T01:32:52.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.003+0000 7f41ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:52.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.003+0000 7f41ef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:32:52.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.003+0000 7f41ef7fe640 10 monclient: dump: 2024-09-11T01:32:52.016 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:52.016 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:52.016 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:52.016 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:52.016 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:52.016 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:52.016 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:52.016 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:52.016 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:52.016 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:52.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.003+0000 7f41f6265640 5 monclient: authenticate success, global_id 44105 2024-09-11T01:32:52.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.003+0000 7f41f6265640 10 monclient: get_monmap_and_config success 2024-09-11T01:32:52.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.003+0000 7f41f6265640 4 set_mon_vals no callback set 2024-09-11T01:32:52.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.004+0000 7f41f6265640 10 monclient: shutdown 2024-09-11T01:32:52.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.004+0000 7f41f6265640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:52.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.004+0000 7f41f6265640 1 -- 172.21.15.12:0/3619732324 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f41f00a2e20 msgr2=0x7f41f006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:52.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.004+0000 7f41f6265640 1 --2- 172.21.15.12:0/3619732324 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f41f00a2e20 0x7f41f006ded0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f41e00097a0 tx=0x7f41e0029fd0 comp rx=0 tx=0).stop 2024-09-11T01:32:52.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.004+0000 7f41f6265640 1 -- 172.21.15.12:0/3619732324 shutdown_connections 2024-09-11T01:32:52.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.004+0000 7f41f6265640 1 --2- 172.21.15.12:0/3619732324 >> v2:172.21.15.12:3300/0 conn(0x7f41f00a4970 0x7f41f00a3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:52.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.004+0000 7f41f6265640 1 --2- 172.21.15.12:0/3619732324 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f41f00a2e20 0x7f41f006ded0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:52.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.004+0000 7f41f6265640 1 -- 172.21.15.12:0/3619732324 >> 172.21.15.12:0/3619732324 conn(0x7f41f00a8600 msgr2=0x7f41f00a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:52.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.004+0000 7f41f6265640 1 -- 172.21.15.12:0/3619732324 shutdown_connections 2024-09-11T01:32:52.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.004+0000 7f41f6265640 1 -- 172.21.15.12:0/3619732324 wait complete. 2024-09-11T01:32:52.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.005+0000 7f41f6265640 10 monclient: build_initial_monmap 2024-09-11T01:32:52.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.005+0000 7f41f6265640 10 monclient: monmap: 2024-09-11T01:32:52.017 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:52.017 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:32:52.018 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:32:52.018 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:32:52.018 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:52.018 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:52.018 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:32:52.018 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:32:52.018 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:32:52.018 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:52.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.005+0000 7f41f6265640 1 Processor -- start 2024-09-11T01:32:52.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.005+0000 7f41f6265640 1 -- start start 2024-09-11T01:32:52.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.005+0000 7f41f6265640 10 monclient: init 2024-09-11T01:32:52.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.005+0000 7f41f6265640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:52.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.005+0000 7f41f6265640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:32:52.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.005+0000 7f41f6265640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41f006f910 0x7f41f01cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:52.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.005+0000 7f41f6265640 10 monclient(hunting): picked mon.noname-a con 0x7f41f006f910 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:32:52.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.005+0000 7f41f6265640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f41f00a2e20 0x7f41f01cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:52.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.005+0000 7f41f6265640 10 monclient(hunting): picked mon.noname-c con 0x7f41f00a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:52.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.005+0000 7f41f6265640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41f00a4970 0x7f41f0124f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:52.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41f6265640 10 monclient(hunting): picked mon.noname-b con 0x7f41f00a4970 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:32:52.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41f6265640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:52.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41f6265640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f41f0138860 con 0x7f41f006f910 2024-09-11T01:32:52.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41f6265640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:52.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41f6265640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f41f01386e0 con 0x7f41f00a4970 2024-09-11T01:32:52.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41f6265640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:52.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41f6265640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f41f01389e0 con 0x7f41f00a2e20 2024-09-11T01:32:52.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41f6265640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:52.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41f6265640 10 monclient(hunting): authenticate will time out at 978.448020s 2024-09-11T01:32:52.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41effff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41f006f910 0x7f41f01cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:52.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41e7fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f41f00a2e20 0x7f41f01cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:52.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41effff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41f006f910 0x7f41f01cd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42818/0 (socket says 172.21.15.12:42818) 2024-09-11T01:32:52.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41e7fff640 10 monclient(hunting): get_auth_request con 0x7f41f00a2e20 auth_method 0 2024-09-11T01:32:52.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41effff640 1 -- 172.21.15.12:0/4108941775 learned_addr learned my addr 172.21.15.12:0/4108941775 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:52.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:52.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:52.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:52.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41effff640 10 monclient(hunting): get_auth_request con 0x7f41f006f910 auth_method 0 2024-09-11T01:32:52.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:52.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41effff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:52.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:52.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:52.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:52.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:52.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:52.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:52.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:52.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41f4802640 1 --2- 172.21.15.12:0/4108941775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41f00a4970 0x7f41f0124f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:52.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41f4802640 10 monclient(hunting): get_auth_request con 0x7f41f00a4970 auth_method 0 2024-09-11T01:32:52.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41f4802640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:52.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41f4802640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:52.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41f4802640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:52.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41e7fff640 10 monclient(hunting): handle_auth_done global_id 44108 payload 824 2024-09-11T01:32:52.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41e7fff640 1 -- 172.21.15.12:0/4108941775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41f00a4970 msgr2=0x7f41f0124f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:52.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41e7fff640 1 --2- 172.21.15.12:0/4108941775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41f00a4970 0x7f41f0124f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:52.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41e7fff640 1 -- 172.21.15.12:0/4108941775 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41f006f910 msgr2=0x7f41f01cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:52.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41e7fff640 1 --2- 172.21.15.12:0/4108941775 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41f006f910 0x7f41f01cd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:52.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41e7fff640 10 monclient: _finish_hunting 0 2024-09-11T01:32:52.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41e7fff640 1 monclient: found mon.noname-c 2024-09-11T01:32:52.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:52.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41e7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:32:52.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41e7fff640 1 -- 172.21.15.12:0/4108941775 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41f0137d70 con 0x7f41f00a2e20 2024-09-11T01:32:52.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.006+0000 7f41effff640 1 --2- 172.21.15.12:0/4108941775 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41f006f910 0x7f41f01cd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:32:52.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.008+0000 7f41e7fff640 1 -- 172.21.15.12:0/4108941775 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f41f00a2e20 msgr2=0x7f41f01cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:52.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.008+0000 7f41e7fff640 1 --2- 172.21.15.12:0/4108941775 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f41f00a2e20 0x7f41f01cd9b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f41d800e850 tx=0x7f41d800ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:52.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.008+0000 7f41ed7fa640 1 -- 172.21.15.12:0/4108941775 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d8017070 con 0x7f41f00a2e20 2024-09-11T01:32:52.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.008+0000 7f41ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:52.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.008+0000 7f41ed7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:32:52.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.008+0000 7f41ed7fa640 10 monclient: dump: 2024-09-11T01:32:52.023 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:52.023 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:52.023 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:52.023 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:52.023 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:52.023 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:52.023 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:52.023 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:52.023 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:52.023 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:52.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.008+0000 7f41ed7fa640 10 monclient: _finish_auth 0 2024-09-11T01:32:52.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.008+0000 7f41ed7fa640 10 monclient: _check_auth_tickets 2024-09-11T01:32:52.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.008+0000 7f41ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:52.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.008+0000 7f41ed7fa640 1 -- 172.21.15.12:0/4108941775 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41d800cb50 con 0x7f41f00a2e20 2024-09-11T01:32:52.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.008+0000 7f41f6265640 5 monclient: authenticate success, global_id 44108 2024-09-11T01:32:52.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.008+0000 7f41f6265640 10 monclient: _renew_subs 2024-09-11T01:32:52.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.008+0000 7f41f6265640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:32:52.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.008+0000 7f41f6265640 1 -- 172.21.15.12:0/4108941775 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41f0126970 con 0x7f41f00a2e20 2024-09-11T01:32:52.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.008+0000 7f41ed7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:52.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.008+0000 7f41ed7fa640 1 -- 172.21.15.12:0/4108941775 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d8013470 con 0x7f41f00a2e20 2024-09-11T01:32:52.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.008+0000 7f41ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:52.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.008+0000 7f41ed7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:32:52.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.008+0000 7f41ed7fa640 10 monclient: dump: 2024-09-11T01:32:52.024 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:52.024 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:52.024 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:52.024 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:52.025 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:52.025 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:52.025 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:52.025 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:52.025 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:52.025 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:52.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.008+0000 7f41f6265640 10 monclient: _renew_subs 2024-09-11T01:32:52.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.008+0000 7f41eeffd640 4 set_mon_vals no callback set 2024-09-11T01:32:52.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.008+0000 7f41f6265640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:32:52.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.008+0000 7f41f6265640 1 -- 172.21.15.12:0/4108941775 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f41f01274d0 con 0x7f41f00a2e20 2024-09-11T01:32:52.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.009+0000 7f41f6265640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:52.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.009+0000 7f41f6265640 1 --2- 172.21.15.12:0/4108941775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41f0125440 0x7f41f0125820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:52.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.009+0000 7f41f6265640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:52.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.009+0000 7f41f6265640 1 -- 172.21.15.12:0/4108941775 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f41f006a060 con 0x7f41f0125440 2024-09-11T01:32:52.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.009+0000 7f41ed7fa640 1 -- 172.21.15.12:0/4108941775 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f41d8024060 con 0x7f41f00a2e20 2024-09-11T01:32:52.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.010+0000 7f41effff640 1 --2- 172.21.15.12:0/4108941775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41f0125440 0x7f41f0125820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:52.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.010+0000 7f41ed7fa640 1 --2- 172.21.15.12:0/4108941775 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f41d003e210 0x7f41d00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:52.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.010+0000 7f41effff640 10 monclient: get_auth_request con 0x7f41f0125440 auth_method 0 2024-09-11T01:32:52.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.010+0000 7f41effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:52.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.010+0000 7f41effff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:52.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.010+0000 7f41effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:52.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.010+0000 7f41ed7fa640 1 -- 172.21.15.12:0/4108941775 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(136..136 src has 1..136) ==== 4783+0+0 (secure 0 0 0) 0x7f41d8052d00 con 0x7f41f00a2e20 2024-09-11T01:32:52.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.010+0000 7f41effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:52.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.010+0000 7f41effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:52.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.010+0000 7f41effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:52.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.010+0000 7f41f4802640 1 --2- 172.21.15.12:0/4108941775 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f41d003e210 0x7f41d00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:52.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.010+0000 7f41f4802640 10 monclient: get_auth_request con 0x7f41d003e210 auth_method 0 2024-09-11T01:32:52.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.010+0000 7f41effff640 10 monclient(hunting): handle_auth_done global_id 34195 payload 824 2024-09-11T01:32:52.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.010+0000 7f41effff640 1 -- 172.21.15.12:0/4108941775 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f41f0128550 con 0x7f41f0125440 2024-09-11T01:32:52.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.013+0000 7f41effff640 1 -- 172.21.15.12:0/4108941775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41f0125440 msgr2=0x7f41f0125820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:52.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.013+0000 7f41effff640 1 --2- 172.21.15.12:0/4108941775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41f0125440 0x7f41f0125820 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f41e00097a0 tx=0x7f41e00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:52.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.013+0000 7f41f4802640 1 --2- 172.21.15.12:0/4108941775 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f41d003e210 0x7f41d00406d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f41f006fd60 tx=0x7f41e8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:52.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.013+0000 7f41ed7fa640 1 -- 172.21.15.12:0/4108941775 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e0043070 con 0x7f41f0125440 2024-09-11T01:32:52.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.013+0000 7f41ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:52.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.013+0000 7f41ed7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:32:52.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.013+0000 7f41ed7fa640 10 monclient: dump: 2024-09-11T01:32:52.029 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:52.029 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:52.029 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:52.030 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:52.030 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:52.030 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:52.030 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:52.030 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:52.030 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:52.030 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:52.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.013+0000 7f41ed7fa640 1 -- 172.21.15.12:0/4108941775 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f41e0038040 con 0x7f41f0125440 2024-09-11T01:32:52.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.013+0000 7f41ed7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:52.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.013+0000 7f41ed7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:32:52.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.013+0000 7f41ed7fa640 1 -- 172.21.15.12:0/4108941775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41f0125440 msgr2=0x7f41f0125820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:52.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.013+0000 7f41ed7fa640 1 --2- 172.21.15.12:0/4108941775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41f0125440 0x7f41f0125820 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f41e00097a0 tx=0x7f41e00046c0 comp rx=0 tx=0).stop 2024-09-11T01:32:52.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.013+0000 7f41ed7fa640 1 -- 172.21.15.12:0/4108941775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41f0125440 msgr2=0x7f41f0125820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:32:52.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.013+0000 7f41ed7fa640 1 --2- 172.21.15.12:0/4108941775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41f0125440 0x7f41f0125820 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f41e00097a0 tx=0x7f41e00046c0 comp rx=0 tx=0).stop 2024-09-11T01:32:52.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.020+0000 7f41f6265640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:32:52.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.020+0000 7f41f6265640 1 --2- 172.21.15.12:0/4108941775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41f0125d60 0x7f41f00a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:52.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.020+0000 7f41f6265640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:52.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.020+0000 7f41f6265640 1 -- 172.21.15.12:0/4108941775 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f41f006ded0 con 0x7f41f0125d60 2024-09-11T01:32:52.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.020+0000 7f41effff640 1 --2- 172.21.15.12:0/4108941775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41f0125d60 0x7f41f00a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:52.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.020+0000 7f41effff640 10 monclient: get_auth_request con 0x7f41f0125d60 auth_method 0 2024-09-11T01:32:52.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.020+0000 7f41effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:52.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.020+0000 7f41effff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:52.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.020+0000 7f41effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:52.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.020+0000 7f41effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:52.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.020+0000 7f41effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:52.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.020+0000 7f41effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:52.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.020+0000 7f41effff640 10 monclient(hunting): handle_auth_done global_id 34198 payload 824 2024-09-11T01:32:52.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.020+0000 7f41effff640 1 -- 172.21.15.12:0/4108941775 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f41f0138290 con 0x7f41f0125d60 2024-09-11T01:32:52.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.023+0000 7f41effff640 1 -- 172.21.15.12:0/4108941775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41f0125d60 msgr2=0x7f41f00a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:52.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.023+0000 7f41effff640 1 --2- 172.21.15.12:0/4108941775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41f0125d60 0x7f41f00a0830 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f41e0042000 tx=0x7f41e0009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:52.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.023+0000 7f41ed7fa640 1 -- 172.21.15.12:0/4108941775 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e0037040 con 0x7f41f0125d60 2024-09-11T01:32:52.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.023+0000 7f41ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:52.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.023+0000 7f41ed7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:32:52.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.023+0000 7f41ed7fa640 10 monclient: dump: 2024-09-11T01:32:52.032 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:52.032 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:52.032 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:52.032 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:52.032 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:52.032 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:52.033 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:52.033 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:52.033 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:52.033 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:52.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.023+0000 7f41ed7fa640 1 -- 172.21.15.12:0/4108941775 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4707 (secure 0 0 0) 0x7f41e003d3f0 con 0x7f41f0125d60 2024-09-11T01:32:52.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.023+0000 7f41ed7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:32:52.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.023+0000 7f41ed7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:32:52.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.023+0000 7f41ed7fa640 1 -- 172.21.15.12:0/4108941775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41f0125d60 msgr2=0x7f41f00a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:52.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.023+0000 7f41ed7fa640 1 --2- 172.21.15.12:0/4108941775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41f0125d60 0x7f41f00a0830 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f41e0042000 tx=0x7f41e0009190 comp rx=0 tx=0).stop 2024-09-11T01:32:52.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.023+0000 7f41ed7fa640 1 -- 172.21.15.12:0/4108941775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41f0125d60 msgr2=0x7f41f00a0830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:32:52.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.023+0000 7f41ed7fa640 1 --2- 172.21.15.12:0/4108941775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41f0125d60 0x7f41f00a0830 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f41e0042000 tx=0x7f41e0009190 comp rx=0 tx=0).stop 2024-09-11T01:32:52.034 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:32:52.034 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:32:52.034 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:32:52.034 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:32:52.035 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 38, 2024-09-11T01:32:52.036 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 65236, 2024-09-11T01:32:52.036 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:32:52.036 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:32:52.036 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:32:52.036 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:32:52.036 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:52.036 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 4, 2024-09-11T01:32:52.036 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:32:52.036 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:32:52.036 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:32:52.036 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:52.036 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:52.036 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:52.037 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:52.037 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:52.037 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:52.037 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:52.037 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:52.037 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:52.037 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:52.037 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:52.037 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:52.037 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:32:52.037 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:32:52.037 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:52.037 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:52.037 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:52.038 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:52.038 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:52.038 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:52.038 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:52.038 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:52.038 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:52.038 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:52.038 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:52.038 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:52.038 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:52.038 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:32:52.038 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:32:52.038 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:32:52.038 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:32:52.039 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:32:52.039 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:32:52.039 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:32:52.039 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:32:52.039 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:32:52.039 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:32:52.039 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:32:52.039 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:32:52.039 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:32:52.039 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:32:52.039 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:32:52.039 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:32:52.039 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:32:52.040 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:52.040 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:52.040 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:52.040 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:52.040 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:52.040 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:52.040 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:52.040 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:52.040 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:52.040 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:52.040 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:52.040 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:52.040 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:32:52.040 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:52.041 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:32:52.041 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:52.041 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:32:52.041 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:32:52.041 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:52.041 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:52.041 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:52.041 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:52.041 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:32:52.041 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:52.041 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:52.041 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:52.041 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:52.042 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:32:52.042 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:52.042 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:52.042 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:52.042 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:52.042 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:32:52.042 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:32:52.042 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:52.042 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:52.042 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:52.042 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:52.042 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:52.042 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:32:52.042 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:32:52.043 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:52.043 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:52.043 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:52.043 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:52.043 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:32:52.043 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:52.043 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:52.043 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:52.043 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:52.043 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:32:52.043 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:52.043 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:52.043 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:52.043 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:52.044 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:32:52.044 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:32:52.044 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:52.044 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:52.044 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:52.044 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:52.044 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:52.044 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:32:52.044 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:32:52.044 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:52.044 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:52.044 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:52.044 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:52.045 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:32:52.045 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:52.045 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:52.045 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:52.045 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:52.045 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:32:52.045 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:52.045 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:52.045 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:52.045 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:52.045 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:32:52.045 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:32:52.045 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:52.045 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:52.046 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:52.046 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:52.046 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:52.046 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:52.046 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:32:52.046 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:32:52.046 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:52.046 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:52.046 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:52.046 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:32:52.046 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:52.047 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:52.047 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:32:52.047 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:52.047 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:52.047 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:52.047 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:32:52.047 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:52.047 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:52.047 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:32:52.047 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:52.047 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:52.047 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:52.047 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:32:52.047 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:52.047 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:52.048 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:52.048 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:32:52.048 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:32:52.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.024+0000 7f41f6265640 1 -- 172.21.15.12:0/4108941775 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f41d003e210 msgr2=0x7f41d00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:52.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.024+0000 7f41f6265640 1 --2- 172.21.15.12:0/4108941775 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f41d003e210 0x7f41d00406d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f41f006fd60 tx=0x7f41e8009040 comp rx=0 tx=0).stop 2024-09-11T01:32:52.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.024+0000 7f41f6265640 10 monclient: shutdown 2024-09-11T01:32:52.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.024+0000 7f41f6265640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:52.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.024+0000 7f41f6265640 1 -- 172.21.15.12:0/4108941775 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f41f00a2e20 msgr2=0x7f41f01cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:52.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.024+0000 7f41f6265640 1 --2- 172.21.15.12:0/4108941775 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f41f00a2e20 0x7f41f01cd9b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f41d800e850 tx=0x7f41d800ed10 comp rx=0 tx=0).stop 2024-09-11T01:32:52.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.024+0000 7f41f4802640 1 -- 172.21.15.12:0/4108941775 reap_dead start 2024-09-11T01:32:52.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.024+0000 7f41f6265640 1 -- 172.21.15.12:0/4108941775 shutdown_connections 2024-09-11T01:32:52.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.024+0000 7f41f6265640 1 --2- 172.21.15.12:0/4108941775 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f41f00a2e20 0x7f41f01cd9b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:52.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.024+0000 7f41f6265640 1 -- 172.21.15.12:0/4108941775 >> 172.21.15.12:0/4108941775 conn(0x7f41f00a8600 msgr2=0x7f41f0072e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:52.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.024+0000 7f41f6265640 1 -- 172.21.15.12:0/4108941775 shutdown_connections 2024-09-11T01:32:52.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.024+0000 7f41f6265640 1 -- 172.21.15.12:0/4108941775 wait complete. 2024-09-11T01:32:52.050 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:32:52.050 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:32:52.051 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:32:52.081 INFO:tasks.rados.rados.0.smithi071.stdout:592: done (7 left) 2024-09-11T01:32:52.081 INFO:tasks.rados.rados.0.smithi071.stdout:594: done (6 left) 2024-09-11T01:32:52.081 INFO:tasks.rados.rados.0.smithi071.stdout:595: done (5 left) 2024-09-11T01:32:52.081 INFO:tasks.rados.rados.0.smithi071.stdout:596: done (4 left) 2024-09-11T01:32:52.082 INFO:tasks.rados.rados.0.smithi071.stdout:597: done (3 left) 2024-09-11T01:32:52.082 INFO:tasks.rados.rados.0.smithi071.stdout:600: done (2 left) 2024-09-11T01:32:52.082 INFO:tasks.rados.rados.0.smithi071.stdout:601: done (1 left) 2024-09-11T01:32:52.082 INFO:tasks.rados.rados.0.smithi071.stdout:602: done (0 left) 2024-09-11T01:32:52.082 INFO:tasks.rados.rados.0.smithi071.stdout:603: delete oid 3 current snap is 58 2024-09-11T01:32:52.084 INFO:tasks.rados.rados.0.smithi071.stdout:603: done (0 left) 2024-09-11T01:32:52.084 INFO:tasks.rados.rados.0.smithi071.stdout:604: write (excl) oid 10 current snap is 58 2024-09-11T01:32:52.084 INFO:tasks.rados.rados.0.smithi071.stdout:604: seq_num 239 ranges {416873=791454,1919314=97559} 2024-09-11T01:32:52.091 INFO:tasks.rados.rados.0.smithi071.stdout:604: writing smithi07134777-10 from 416873 to 1208327 tid 1 2024-09-11T01:32:52.096 INFO:tasks.rados.rados.0.smithi071.stdout:604: writing smithi07134777-10 from 1919314 to 2016873 tid 2 2024-09-11T01:32:52.096 INFO:tasks.rados.rados.0.smithi071.stdout:605: copy_from oid 3 from oid 20 current snap is 58 2024-09-11T01:32:52.096 INFO:tasks.rados.rados.0.smithi071.stdout:606: write oid 22 current snap is 58 2024-09-11T01:32:52.096 INFO:tasks.rados.rados.0.smithi071.stdout:606: seq_num 240 ranges {433680=466703,1577796=455884} 2024-09-11T01:32:52.101 INFO:tasks.rados.rados.0.smithi071.stdout:606: writing smithi07134777-22 from 433680 to 900383 tid 1 2024-09-11T01:32:52.109 INFO:tasks.rados.rados.0.smithi071.stdout:606: writing smithi07134777-22 from 1577796 to 2033680 tid 2 2024-09-11T01:32:52.109 INFO:tasks.rados.rados.0.smithi071.stdout:607: read oid 41 snap -1 2024-09-11T01:32:52.109 INFO:tasks.rados.rados.0.smithi071.stdout:607: expect deleted 2024-09-11T01:32:52.109 INFO:tasks.rados.rados.0.smithi071.stdout:608: rollback oid 48 current snap is 58 2024-09-11T01:32:52.109 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 48 to 57 2024-09-11T01:32:52.109 INFO:tasks.rados.rados.0.smithi071.stdout:605: finishing copy_from racing read to smithi07134777-3 2024-09-11T01:32:52.109 INFO:tasks.rados.rados.0.smithi071.stdout:604: finishing write tid 1 to smithi07134777-10 2024-09-11T01:32:52.109 INFO:tasks.rados.rados.0.smithi071.stdout:604: finishing write tid 2 to smithi07134777-10 2024-09-11T01:32:52.109 INFO:tasks.rados.rados.0.smithi071.stdout:605: finishing copy_from to smithi07134777-3 2024-09-11T01:32:52.109 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-11T01:32:52.110 INFO:tasks.rados.rados.0.smithi071.stdout:604: finishing write tid 3 to smithi07134777-10 2024-09-11T01:32:52.110 INFO:tasks.rados.rados.0.smithi071.stdout:604: finishing write tid 4 to smithi07134777-10 2024-09-11T01:32:52.110 INFO:tasks.rados.rados.0.smithi071.stdout:604: oid 10 updating version 0 to 121 2024-09-11T01:32:52.110 INFO:tasks.rados.rados.0.smithi071.stdout:604: oid 10 updating version 121 to 123 2024-09-11T01:32:52.110 INFO:tasks.rados.rados.0.smithi071.stdout:604: oid 10 version 123 is already newer than 122 2024-09-11T01:32:52.110 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 123 (ObjNum 239 snap 58 seq_num 239) dirty exists 2024-09-11T01:32:52.110 INFO:tasks.rados.rados.0.smithi071.stdout:604: left oid 10 (ObjNum 239 snap 58 seq_num 239) 2024-09-11T01:32:52.110 INFO:tasks.rados.rados.0.smithi071.stdout:606: finishing write tid 1 to smithi07134777-22 2024-09-11T01:32:52.110 INFO:tasks.rados.rados.0.smithi071.stdout:604: done (4 left) 2024-09-11T01:32:52.110 INFO:tasks.rados.rados.0.smithi071.stdout:605: done (3 left) 2024-09-11T01:32:52.110 INFO:tasks.rados.rados.0.smithi071.stdout:609: read oid 5 snap 56 2024-09-11T01:32:52.110 INFO:tasks.rados.rados.0.smithi071.stdout:609: expect (ObjNum 201 snap 46 seq_num 201) 2024-09-11T01:32:52.110 INFO:tasks.rados.rados.0.smithi071.stdout:610: snap_create 2024-09-11T01:32:52.111 INFO:tasks.rados.rados.0.smithi071.stdout:606: finishing write tid 2 to smithi07134777-22 2024-09-11T01:32:52.111 INFO:tasks.rados.rados.0.smithi071.stdout:606: finishing write tid 3 to smithi07134777-22 2024-09-11T01:32:52.111 INFO:tasks.rados.rados.0.smithi071.stdout:606: finishing write tid 4 to smithi07134777-22 2024-09-11T01:32:52.111 INFO:tasks.rados.rados.0.smithi071.stdout:606: oid 22 updating version 0 to 208 2024-09-11T01:32:52.111 INFO:tasks.rados.rados.0.smithi071.stdout:606: oid 22 updating version 208 to 209 2024-09-11T01:32:52.111 INFO:tasks.rados.rados.0.smithi071.stdout:606: oid 22 updating version 209 to 210 2024-09-11T01:32:52.111 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 210 (ObjNum 240 snap 58 seq_num 240) dirty exists 2024-09-11T01:32:52.112 INFO:tasks.rados.rados.0.smithi071.stdout:606: left oid 22 (ObjNum 240 snap 58 seq_num 240) 2024-09-11T01:32:52.116 INFO:tasks.rados.rados.0.smithi071.stdout:609: expect (ObjNum 201 snap 46 seq_num 201) 2024-09-11T01:32:52.120 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 807273 2024-09-11T01:32:52.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.158+0000 7fb498671640 10 monclient: get_monmap_and_config 2024-09-11T01:32:52.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.158+0000 7fb498671640 10 monclient: build_initial_monmap 2024-09-11T01:32:52.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.158+0000 7fb498671640 10 monclient: monmap: 2024-09-11T01:32:52.160 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:52.160 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:52.160 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:32:52.159579+0000 2024-09-11T01:32:52.160 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:32:52.159579+0000 2024-09-11T01:32:52.161 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:52.161 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:52.161 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:32:52.161 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:32:52.161 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:32:52.161 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:32:52.161 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:32:52.161 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:52.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.159+0000 7fb498671640 1 Processor -- start 2024-09-11T01:32:52.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.159+0000 7fb498671640 1 -- start start 2024-09-11T01:32:52.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.159+0000 7fb498671640 10 monclient: init 2024-09-11T01:32:52.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb498671640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:52.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb498671640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-11T01:32:52.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb498671640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb49006f540 addr v1:172.21.15.12:6789/0 2024-09-11T01:32:52.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb498671640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb49009cff0 addr v1:172.21.15.71:6789/0 2024-09-11T01:32:52.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb498671640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb49013e240 0x7fb490140630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:52.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb498671640 10 monclient(hunting): picked mon.noname-b con 0x7fb49013e240 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:52.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb498671640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4900a6ea0 con 0x7fb49006f540 2024-09-11T01:32:52.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb498671640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4900a5e40 con 0x7fb49009cff0 2024-09-11T01:32:52.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb498671640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:52.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb498671640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb49006f920 con 0x7fb49013e240 2024-09-11T01:32:52.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb498671640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:52.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb498671640 10 monclient(hunting): authenticate will time out at 978.602350s 2024-09-11T01:32:52.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb49640d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb49006f540 0x7fb4900a5020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:39802/0 (socket says 172.21.15.12:39802) 2024-09-11T01:32:52.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb49640d640 1 -- 172.21.15.12:0/136373472 learned_addr learned my addr 172.21.15.12:0/136373472 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:32:52.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb496c0e640 1 --2- 172.21.15.12:0/136373472 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb49013e240 0x7fb490140630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:52.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb496c0e640 10 monclient(hunting): get_auth_request con 0x7fb49013e240 auth_method 0 2024-09-11T01:32:52.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb496c0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:52.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb496c0e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:52.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb496c0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:52.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb496c0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:52.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb496c0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:52.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb496c0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:52.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb496c0e640 10 monclient(hunting): handle_auth_done global_id 44111 payload 274 2024-09-11T01:32:52.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb496c0e640 1 -- 172.21.15.12:0/136373472 >> v1:172.21.15.71:6789/0 conn(0x7fb49009cff0 legacy=0x7fb4900a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:52.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb496c0e640 1 -- 172.21.15.12:0/136373472 >> v1:172.21.15.12:6789/0 conn(0x7fb49006f540 legacy=0x7fb4900a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:52.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb496c0e640 10 monclient: _finish_hunting 0 2024-09-11T01:32:52.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb496c0e640 1 monclient: found mon.noname-b 2024-09-11T01:32:52.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb496c0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:52.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb496c0e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:32:52.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.160+0000 7fb496c0e640 1 -- 172.21.15.12:0/136373472 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb49009d4d0 con 0x7fb49013e240 2024-09-11T01:32:52.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.163+0000 7fb496c0e640 1 -- 172.21.15.12:0/136373472 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb49013e240 msgr2=0x7fb490140630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:52.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.163+0000 7fb496c0e640 1 --2- 172.21.15.12:0/136373472 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb49013e240 0x7fb490140630 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb4880097d0 tx=0x7fb488029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44f6b7be2f58a45 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:52.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.163+0000 7fb49540b640 1 -- 172.21.15.12:0/136373472 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb488037070 con 0x7fb49013e240 2024-09-11T01:32:52.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.163+0000 7fb49540b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:52.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.163+0000 7fb49540b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:32:52.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.163+0000 7fb49540b640 10 monclient: dump: 2024-09-11T01:32:52.173 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:52.173 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:52.173 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:52.173 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:52.173 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:52.173 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:52.173 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:52.173 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:52.173 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:52.173 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:52.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.163+0000 7fb49540b640 10 monclient: _finish_auth 0 2024-09-11T01:32:52.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.163+0000 7fb49540b640 10 monclient: _check_auth_tickets 2024-09-11T01:32:52.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.163+0000 7fb49540b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:52.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.163+0000 7fb498671640 5 monclient: authenticate success, global_id 44111 2024-09-11T01:32:52.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.163+0000 7fb49540b640 1 -- 172.21.15.12:0/136373472 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb488032440 con 0x7fb49013e240 2024-09-11T01:32:52.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.163+0000 7fb49540b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:52.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.163+0000 7fb498671640 10 monclient: get_monmap_and_config success 2024-09-11T01:32:52.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.163+0000 7fb498671640 4 set_mon_vals no callback set 2024-09-11T01:32:52.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.163+0000 7fb49540b640 1 -- 172.21.15.12:0/136373472 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb488032810 con 0x7fb49013e240 2024-09-11T01:32:52.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.163+0000 7fb49540b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:52.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.163+0000 7fb49540b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:32:52.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.163+0000 7fb49540b640 10 monclient: dump: 2024-09-11T01:32:52.174 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:52.175 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:52.175 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:52.175 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:52.175 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:52.175 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:52.175 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:52.175 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:52.175 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:52.175 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:52.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.164+0000 7fb498671640 10 monclient: shutdown 2024-09-11T01:32:52.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.164+0000 7fb498671640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:52.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.164+0000 7fb498671640 1 -- 172.21.15.12:0/136373472 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb49013e240 msgr2=0x7fb490140630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:52.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.164+0000 7fb498671640 1 --2- 172.21.15.12:0/136373472 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb49013e240 0x7fb490140630 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb4880097d0 tx=0x7fb488029fb0 comp rx=0 tx=0).stop 2024-09-11T01:32:52.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.164+0000 7fb498671640 1 -- 172.21.15.12:0/136373472 shutdown_connections 2024-09-11T01:32:52.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.164+0000 7fb498671640 1 --2- 172.21.15.12:0/136373472 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb49013e240 0x7fb490140630 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:52.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.164+0000 7fb498671640 1 -- 172.21.15.12:0/136373472 >> 172.21.15.12:0/136373472 conn(0x7fb4900a7300 msgr2=0x7fb4900a7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:52.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.164+0000 7fb498671640 1 -- 172.21.15.12:0/136373472 shutdown_connections 2024-09-11T01:32:52.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.164+0000 7fb498671640 1 -- 172.21.15.12:0/136373472 wait complete. 2024-09-11T01:32:52.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.165+0000 7fb498671640 10 monclient: build_initial_monmap 2024-09-11T01:32:52.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.165+0000 7fb498671640 10 monclient: monmap: 2024-09-11T01:32:52.176 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:52.176 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:32:52.176 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:32:52.176 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:32:52.176 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:52.176 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:52.177 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:32:52.177 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:32:52.177 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:32:52.177 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:52.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.165+0000 7fb498671640 1 Processor -- start 2024-09-11T01:32:52.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.165+0000 7fb498671640 1 -- start start 2024-09-11T01:32:52.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.165+0000 7fb498671640 10 monclient: init 2024-09-11T01:32:52.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.165+0000 7fb498671640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:52.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.165+0000 7fb498671640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:32:52.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.165+0000 7fb498671640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb49006f540 0x7fb4901cd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:52.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.165+0000 7fb498671640 10 monclient(hunting): picked mon.noname-b con 0x7fb49006f540 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:32:52.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.165+0000 7fb498671640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb49009cff0 0x7fb490138060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:52.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.165+0000 7fb498671640 10 monclient(hunting): picked mon.noname-c con 0x7fb49009cff0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:52.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.165+0000 7fb498671640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb49013e240 0x7fb49013a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:52.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.165+0000 7fb498671640 10 monclient(hunting): picked mon.noname-a con 0x7fb49013e240 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:32:52.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.165+0000 7fb498671640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:52.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.165+0000 7fb498671640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb490140cf0 con 0x7fb49013e240 2024-09-11T01:32:52.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.165+0000 7fb498671640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:52.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.165+0000 7fb498671640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb490140b70 con 0x7fb49006f540 2024-09-11T01:32:52.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.165+0000 7fb498671640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:52.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.165+0000 7fb498671640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb490140e70 con 0x7fb49009cff0 2024-09-11T01:32:52.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.165+0000 7fb498671640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:52.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.165+0000 7fb498671640 10 monclient(hunting): authenticate will time out at 978.607880s 2024-09-11T01:32:52.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.165+0000 7fb496c0e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb49013e240 0x7fb49013a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:52.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.165+0000 7fb495c0c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb49009cff0 0x7fb490138060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:52.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb496c0e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb49013e240 0x7fb49013a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42826/0 (socket says 172.21.15.12:42826) 2024-09-11T01:32:52.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb496c0e640 1 -- 172.21.15.12:0/2445641740 learned_addr learned my addr 172.21.15.12:0/2445641740 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:52.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb496c0e640 10 monclient(hunting): get_auth_request con 0x7fb49013e240 auth_method 0 2024-09-11T01:32:52.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb496c0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:52.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb496c0e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:52.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb496c0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:52.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb49640d640 1 --2- 172.21.15.12:0/2445641740 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb49006f540 0x7fb4901cd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:52.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb495c0c640 10 monclient(hunting): get_auth_request con 0x7fb49009cff0 auth_method 0 2024-09-11T01:32:52.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb495c0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:52.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb495c0c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:52.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb495c0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:52.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb49640d640 10 monclient(hunting): get_auth_request con 0x7fb49006f540 auth_method 0 2024-09-11T01:32:52.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb49640d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:52.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb49640d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:52.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb49640d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:52.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb495c0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:52.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb495c0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:52.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb495c0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:52.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb496c0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:52.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb496c0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:52.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb496c0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:52.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb49640d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:52.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb49640d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:52.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb49640d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:52.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb495c0c640 10 monclient(hunting): handle_auth_done global_id 44114 payload 824 2024-09-11T01:32:52.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb495c0c640 1 -- 172.21.15.12:0/2445641740 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb49006f540 msgr2=0x7fb4901cd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:52.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb495c0c640 1 --2- 172.21.15.12:0/2445641740 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb49006f540 0x7fb4901cd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:52.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb495c0c640 1 -- 172.21.15.12:0/2445641740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb49013e240 msgr2=0x7fb49013a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:52.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb495c0c640 1 --2- 172.21.15.12:0/2445641740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb49013e240 0x7fb49013a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:52.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb495c0c640 10 monclient: _finish_hunting 0 2024-09-11T01:32:52.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb495c0c640 1 monclient: found mon.noname-c 2024-09-11T01:32:52.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb495c0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:52.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb495c0c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:32:52.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb495c0c640 1 -- 172.21.15.12:0/2445641740 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb49013ce50 con 0x7fb49009cff0 2024-09-11T01:32:52.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.166+0000 7fb496c0e640 1 --2- 172.21.15.12:0/2445641740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb49013e240 0x7fb49013a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:32:52.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.168+0000 7fb495c0c640 1 -- 172.21.15.12:0/2445641740 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb49009cff0 msgr2=0x7fb490138060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:52.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.168+0000 7fb495c0c640 1 --2- 172.21.15.12:0/2445641740 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb49009cff0 0x7fb490138060 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb48400c410 tx=0x7fb48400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:52.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.168+0000 7fb48effd640 1 -- 172.21.15.12:0/2445641740 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb484017070 con 0x7fb49009cff0 2024-09-11T01:32:52.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.168+0000 7fb48effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:52.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.168+0000 7fb48effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:32:52.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.168+0000 7fb48effd640 10 monclient: dump: 2024-09-11T01:32:52.182 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:52.182 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:52.182 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:52.182 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:52.182 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:52.182 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:52.182 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:52.182 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:52.182 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:52.183 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:52.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.168+0000 7fb48effd640 10 monclient: _finish_auth 0 2024-09-11T01:32:52.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.168+0000 7fb48effd640 10 monclient: _check_auth_tickets 2024-09-11T01:32:52.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.168+0000 7fb48effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:52.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.168+0000 7fb48effd640 1 -- 172.21.15.12:0/2445641740 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb484005230 con 0x7fb49009cff0 2024-09-11T01:32:52.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.168+0000 7fb498671640 5 monclient: authenticate success, global_id 44114 2024-09-11T01:32:52.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.168+0000 7fb498671640 10 monclient: _renew_subs 2024-09-11T01:32:52.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.168+0000 7fb498671640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:32:52.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.168+0000 7fb498671640 1 -- 172.21.15.12:0/2445641740 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb49013bc80 con 0x7fb49009cff0 2024-09-11T01:32:52.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.168+0000 7fb48effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:52.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.168+0000 7fb48effd640 1 -- 172.21.15.12:0/2445641740 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4840117a0 con 0x7fb49009cff0 2024-09-11T01:32:52.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.168+0000 7fb48effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:52.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.168+0000 7fb48effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:32:52.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.168+0000 7fb48effd640 10 monclient: dump: 2024-09-11T01:32:52.184 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:52.184 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:52.184 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:52.184 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:52.184 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:52.184 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:52.184 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:52.184 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:52.184 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:52.184 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:52.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.168+0000 7fb498671640 10 monclient: _renew_subs 2024-09-11T01:32:52.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.168+0000 7fb498671640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:32:52.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.168+0000 7fb498671640 1 -- 172.21.15.12:0/2445641740 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4901cdee0 con 0x7fb49009cff0 2024-09-11T01:32:52.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.168+0000 7fb494c0a640 4 set_mon_vals no callback set 2024-09-11T01:32:52.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.169+0000 7fb498671640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:52.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.169+0000 7fb498671640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:52.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.169+0000 7fb498671640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:32:52.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.169+0000 7fb498671640 1 -- 172.21.15.12:0/2445641740 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4900a5020 con 0x7fb49009cff0 2024-09-11T01:32:52.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.169+0000 7fb48effd640 1 -- 172.21.15.12:0/2445641740 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fb4840119c0 con 0x7fb49009cff0 2024-09-11T01:32:52.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.170+0000 7fb48effd640 1 --2- 172.21.15.12:0/2445641740 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb47c03e260 0x7fb47c040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:52.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.170+0000 7fb48effd640 1 -- 172.21.15.12:0/2445641740 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(137..137 src has 1..137) ==== 4783+0+0 (secure 0 0 0) 0x7fb4840523e0 con 0x7fb49009cff0 2024-09-11T01:32:52.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.170+0000 7fb49640d640 1 --2- 172.21.15.12:0/2445641740 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb47c03e260 0x7fb47c040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:52.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.170+0000 7fb49640d640 10 monclient: get_auth_request con 0x7fb47c03e260 auth_method 0 2024-09-11T01:32:52.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.172+0000 7fb49640d640 1 --2- 172.21.15.12:0/2445641740 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb47c03e260 0x7fb47c040720 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb48000c550 tx=0x7fb480009b90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:52.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.174+0000 7fb48effd640 1 -- 172.21.15.12:0/2445641740 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb484004d90 con 0x7fb49009cff0 2024-09-11T01:32:52.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.174+0000 7fb48effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:52.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.174+0000 7fb48effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:32:52.188 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:32:52.224 INFO:tasks.rados.rados.0.smithi071.stdout:608: finishing rollback tid 0 to smithi07134777-48 2024-09-11T01:32:52.224 INFO:tasks.rados.rados.0.smithi071.stdout:608: finishing rollback tid 1 to smithi07134777-48 2024-09-11T01:32:52.225 INFO:tasks.rados.rados.0.smithi071.stdout:608: finishing rollback tid 2 to smithi07134777-48 2024-09-11T01:32:52.225 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 290 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-11T01:32:52.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.430+0000 7fb498671640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:32:52.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.430+0000 7fb498671640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:32:52.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.430+0000 7fb498671640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:32:52.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.430+0000 7fb498671640 1 -- 172.21.15.12:0/2445641740 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fb490069990 con 0x7fb49009cff0 2024-09-11T01:32:52.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.431+0000 7fb48effd640 1 -- 172.21.15.12:0/2445641740 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fb4840520d0 con 0x7fb49009cff0 2024-09-11T01:32:52.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.431+0000 7fb48effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:32:52.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.431+0000 7fb48effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:32:52.434 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.431+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:32:52.434 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.431+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=12,logm=64} crc {auth=123548275,config=2586702516,health=2157093694,kv=3263697420,logm=1465266110}) 2024-09-11T01:32:52.434 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.431+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=11,config=2,health=10,kv=11,logm=66} crc {auth=245074880,config=2586702516,health=4289373631,kv=2545089054,logm=4002328426}) 2024-09-11T01:32:52.434 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.431+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:32:52.434 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.431+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=12,logm=64} crc {auth=123548275,config=2586702516,health=2157093694,kv=3263697420,logm=1465266110}) 2024-09-11T01:32:52.434 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.431+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=11,config=2,health=12,kv=11,logm=64} crc {auth=245074880,config=2586702516,health=2157093694,kv=1918975338,logm=2654515692}) 2024-09-11T01:32:52.434 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.432+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:32:52.434 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.432+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3008020816}) 2024-09-11T01:32:52.435 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.432+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4047131571}) 2024-09-11T01:32:52.435 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.432+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:32:52.435 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.432+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3008020816}) 2024-09-11T01:32:52.435 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.432+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=983977573}) 2024-09-11T01:32:52.436 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.433+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:32:52.436 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.433+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1283418006}) 2024-09-11T01:32:52.436 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.433+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1085863784}) 2024-09-11T01:32:52.436 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.433+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:32:52.436 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.433+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1283418006}) 2024-09-11T01:32:52.436 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.433+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2410312351}) 2024-09-11T01:32:52.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.434+0000 7fb498671640 1 -- 172.21.15.12:0/2445641740 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb47c03e260 msgr2=0x7fb47c040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:52.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.434+0000 7fb498671640 1 --2- 172.21.15.12:0/2445641740 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb47c03e260 0x7fb47c040720 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb48000c550 tx=0x7fb480009b90 comp rx=0 tx=0).stop 2024-09-11T01:32:52.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.434+0000 7fb498671640 10 monclient: shutdown 2024-09-11T01:32:52.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.434+0000 7fb498671640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:52.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.434+0000 7fb498671640 1 -- 172.21.15.12:0/2445641740 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb49009cff0 msgr2=0x7fb490138060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:52.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.434+0000 7fb498671640 1 --2- 172.21.15.12:0/2445641740 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb49009cff0 0x7fb490138060 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb48400c410 tx=0x7fb48400c8d0 comp rx=0 tx=0).stop 2024-09-11T01:32:52.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.434+0000 7fb498671640 1 -- 172.21.15.12:0/2445641740 shutdown_connections 2024-09-11T01:32:52.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.434+0000 7fb498671640 1 --2- 172.21.15.12:0/2445641740 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb47c03e260 0x7fb47c040720 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:52.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.434+0000 7fb498671640 1 --2- 172.21.15.12:0/2445641740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb49013e240 0x7fb49013a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:52.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.434+0000 7fb498671640 1 --2- 172.21.15.12:0/2445641740 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb49009cff0 0x7fb490138060 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:52.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.434+0000 7fb498671640 1 --2- 172.21.15.12:0/2445641740 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb49006f540 0x7fb4901cd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:52.438 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.434+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:32:52.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.434+0000 7fb498671640 1 -- 172.21.15.12:0/2445641740 >> 172.21.15.12:0/2445641740 conn(0x7fb4900a7300 msgr2=0x7fb4900a3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:52.439 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.434+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=540861215}) 2024-09-11T01:32:52.439 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.434+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=771392100}) 2024-09-11T01:32:52.439 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.434+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:32:52.439 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.434+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=540861215}) 2024-09-11T01:32:52.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.434+0000 7fb498671640 1 -- 172.21.15.12:0/2445641740 shutdown_connections 2024-09-11T01:32:52.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:52.434+0000 7fb498671640 1 -- 172.21.15.12:0/2445641740 wait complete. 2024-09-11T01:32:52.443 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.434+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4242594743}) 2024-09-11T01:32:52.443 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.434+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:32:52.443 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.434+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1120136405}) 2024-09-11T01:32:52.443 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.435+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3801332458}) 2024-09-11T01:32:52.443 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.435+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:32:52.444 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.435+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1120136405}) 2024-09-11T01:32:52.444 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.435+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2681657364}) 2024-09-11T01:32:52.444 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.435+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:32:52.444 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.435+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1780467352}) 2024-09-11T01:32:52.444 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.436+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1273368609}) 2024-09-11T01:32:52.444 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.436+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:32:52.444 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.436+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1780467352}) 2024-09-11T01:32:52.444 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.436+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=487424284}) 2024-09-11T01:32:52.444 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.436+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:32:52.444 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.436+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1098260717}) 2024-09-11T01:32:52.444 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.436+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2857657085}) 2024-09-11T01:32:52.444 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.436+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:32:52.445 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.436+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1098260717}) 2024-09-11T01:32:52.445 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.436+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=395221543}) 2024-09-11T01:32:52.445 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.437+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:32:52.445 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.437+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2852644026}) 2024-09-11T01:32:52.445 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.437+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3767503388}) 2024-09-11T01:32:52.445 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.437+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:32:52.445 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.437+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2852644026}) 2024-09-11T01:32:52.445 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.437+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1726720826}) 2024-09-11T01:32:52.445 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.438+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:32:52.445 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.438+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3655749875}) 2024-09-11T01:32:52.445 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.438+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4066405233}) 2024-09-11T01:32:52.445 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.438+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:32:52.445 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.438+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3655749875}) 2024-09-11T01:32:52.446 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.438+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2851870780}) 2024-09-11T01:32:52.446 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.439+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:32:52.446 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.439+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=67,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=13} crc {logm=2566510072,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2896208888,mon_config_key=287496917}) 2024-09-11T01:32:52.446 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.439+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=66,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=15} crc {logm=635768869,mdsmap=3760429986,mgr=1816177914,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2706747733,mon_config_key=2226403993}) 2024-09-11T01:32:52.446 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.439+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:32:52.446 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.439+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=67,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=13} crc {logm=2566510072,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2896208888,mon_config_key=287496917}) 2024-09-11T01:32:52.446 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.439+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=11} crc {logm=2147276458,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=321061422,mon_config_key=2792997069}) 2024-09-11T01:32:52.446 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.440+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:32:52.446 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.440+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=4,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=24,osdmap=60} crc {mon_config_key=2571357278,monmap=1324671969,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=2870354246,osd_snap=2981951665,osdmap=3540953362}) 2024-09-11T01:32:52.446 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.440+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=3,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=61} crc {mon_config_key=2943408303,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=2870354246,osd_snap=3212592440,osdmap=4115402560}) 2024-09-11T01:32:52.446 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.440+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:32:52.446 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.440+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=4,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=24,osdmap=60} crc {mon_config_key=2571357278,monmap=1324671969,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=2870354246,osd_snap=2981951665,osdmap=3540953362}) 2024-09-11T01:32:52.446 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.440+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=4,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=23,osdmap=61} crc {mon_config_key=2571357278,monmap=2933275274,nvmeofgw=598337686,osd_metadata=1404192910,osd_pg_creating=2870354246,osd_snap=182557628,osdmap=2871221382}) 2024-09-11T01:32:52.447 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.441+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:32:52.447 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.441+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1389000036}) 2024-09-11T01:32:52.447 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.441+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2843451705}) 2024-09-11T01:32:52.447 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.441+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:32:52.447 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.441+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1389000036}) 2024-09-11T01:32:52.447 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.441+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1888140994}) 2024-09-11T01:32:52.448 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.444+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:32:52.448 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.444+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=51} crc {osdmap=2037557153}) 2024-09-11T01:32:52.448 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.444+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=53} crc {osdmap=1689063181}) 2024-09-11T01:32:52.448 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.444+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:32:52.448 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.444+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=51} crc {osdmap=2037557153}) 2024-09-11T01:32:52.448 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:32:52.444+0000 7f86ad63a640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=50} crc {osdmap=637339330}) 2024-09-11T01:32:52.449 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:32:53.088 INFO:tasks.rados.rados.0.smithi071.stdout:606: done (4 left) 2024-09-11T01:32:53.088 INFO:tasks.rados.rados.0.smithi071.stdout:607: done (3 left) 2024-09-11T01:32:53.088 INFO:tasks.rados.rados.0.smithi071.stdout:608: done (2 left) 2024-09-11T01:32:53.088 INFO:tasks.rados.rados.0.smithi071.stdout:609: done (1 left) 2024-09-11T01:32:53.088 INFO:tasks.rados.rados.0.smithi071.stdout:610: done (0 left) 2024-09-11T01:32:53.088 INFO:tasks.rados.rados.0.smithi071.stdout:611: write (excl) oid 29 current snap is 59 2024-09-11T01:32:53.088 INFO:tasks.rados.rados.0.smithi071.stdout:611: seq_num 241 ranges {450487=541952,1636278=414209} 2024-09-11T01:32:53.093 INFO:tasks.rados.rados.0.smithi071.stdout:611: writing smithi07134777-29 from 450487 to 992439 tid 1 2024-09-11T01:32:53.099 INFO:tasks.rados.rados.0.smithi071.stdout:611: writing smithi07134777-29 from 1636278 to 2050487 tid 2 2024-09-11T01:32:53.099 INFO:tasks.rados.rados.0.smithi071.stdout:612: write (excl) oid 7 current snap is 59 2024-09-11T01:32:53.099 INFO:tasks.rados.rados.0.smithi071.stdout:611: finishing write tid 1 to smithi07134777-29 2024-09-11T01:32:53.099 INFO:tasks.rados.rados.0.smithi071.stdout:612: seq_num 242 ranges {467294=617201,1578407=488887} 2024-09-11T01:32:53.105 INFO:tasks.rados.rados.0.smithi071.stdout:612: writing smithi07134777-7 from 467294 to 1084495 tid 1 2024-09-11T01:32:53.111 INFO:tasks.rados.rados.0.smithi071.stdout:612: writing smithi07134777-7 from 1578407 to 2067294 tid 2 2024-09-11T01:32:53.111 INFO:tasks.rados.rados.0.smithi071.stdout:613: read oid 34 snap -1 2024-09-11T01:32:53.112 INFO:tasks.rados.rados.0.smithi071.stdout:613: expect deleted 2024-09-11T01:32:53.112 INFO:tasks.rados.rados.0.smithi071.stdout:614: read oid 2 snap -1 2024-09-11T01:32:53.112 INFO:tasks.rados.rados.0.smithi071.stdout:614: expect (ObjNum 216 snap 52 seq_num 216) 2024-09-11T01:32:53.112 INFO:tasks.rados.rados.0.smithi071.stdout:615: snap_create 2024-09-11T01:32:53.112 INFO:tasks.rados.rados.0.smithi071.stdout:611: finishing write tid 2 to smithi07134777-29 2024-09-11T01:32:53.112 INFO:tasks.rados.rados.0.smithi071.stdout:611: finishing write tid 3 to smithi07134777-29 2024-09-11T01:32:53.112 INFO:tasks.rados.rados.0.smithi071.stdout:611: finishing write tid 4 to smithi07134777-29 2024-09-11T01:32:53.112 INFO:tasks.rados.rados.0.smithi071.stdout:611: oid 29 updating version 0 to 294 2024-09-11T01:32:53.112 INFO:tasks.rados.rados.0.smithi071.stdout:611: oid 29 version 294 is already newer than 292 2024-09-11T01:32:53.112 INFO:tasks.rados.rados.0.smithi071.stdout:611: oid 29 version 294 is already newer than 293 2024-09-11T01:32:53.112 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 294 (ObjNum 241 snap 59 seq_num 241) dirty exists 2024-09-11T01:32:53.112 INFO:tasks.rados.rados.0.smithi071.stdout:611: left oid 29 (ObjNum 241 snap 59 seq_num 241) 2024-09-11T01:32:53.112 INFO:tasks.rados.rados.0.smithi071.stdout:612: finishing write tid 1 to smithi07134777-7 2024-09-11T01:32:53.115 INFO:tasks.rados.rados.0.smithi071.stdout:612: finishing write tid 2 to smithi07134777-7 2024-09-11T01:32:53.116 INFO:tasks.rados.rados.0.smithi071.stdout:612: finishing write tid 3 to smithi07134777-7 2024-09-11T01:32:53.116 INFO:tasks.rados.rados.0.smithi071.stdout:612: finishing write tid 4 to smithi07134777-7 2024-09-11T01:32:53.116 INFO:tasks.rados.rados.0.smithi071.stdout:612: oid 7 updating version 0 to 296 2024-09-11T01:32:53.116 INFO:tasks.rados.rados.0.smithi071.stdout:612: oid 7 updating version 296 to 297 2024-09-11T01:32:53.116 INFO:tasks.rados.rados.0.smithi071.stdout:612: oid 7 updating version 297 to 298 2024-09-11T01:32:53.116 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 298 (ObjNum 242 snap 59 seq_num 242) dirty exists 2024-09-11T01:32:53.116 INFO:tasks.rados.rados.0.smithi071.stdout:612: left oid 7 (ObjNum 242 snap 59 seq_num 242) 2024-09-11T01:32:53.118 INFO:tasks.rados.rados.0.smithi071.stdout:614: expect (ObjNum 216 snap 52 seq_num 216) 2024-09-11T01:32:53.149 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2454807 2024-09-11T01:32:53.449 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:32:53.450 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:32:53.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.560+0000 7f8519816640 10 monclient: get_monmap_and_config 2024-09-11T01:32:53.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.561+0000 7f8519816640 10 monclient: build_initial_monmap 2024-09-11T01:32:53.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.561+0000 7f8519816640 10 monclient: monmap: 2024-09-11T01:32:53.562 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:53.563 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:53.563 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:32:53.561756+0000 2024-09-11T01:32:53.563 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:32:53.561756+0000 2024-09-11T01:32:53.563 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:53.563 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:53.563 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:32:53.563 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:32:53.563 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:32:53.563 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:32:53.563 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:32:53.563 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:53.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.561+0000 7f8519816640 1 Processor -- start 2024-09-11T01:32:53.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.561+0000 7f8519816640 1 -- start start 2024-09-11T01:32:53.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.561+0000 7f8519816640 10 monclient: init 2024-09-11T01:32:53.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8519816640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:53.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8519816640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-11T01:32:53.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8519816640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8514138230 0x7f8514138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:53.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8519816640 10 monclient(hunting): picked mon.noname-b con 0x7f8514138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:53.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8519816640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f851409d030 0x7f85140a5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:53.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8519816640 10 monclient(hunting): picked mon.noname-a con 0x7f851409d030 addr v2:172.21.15.12:3300/0 2024-09-11T01:32:53.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8519816640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f851409d430 0x7f85140a5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:53.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8519816640 10 monclient(hunting): picked mon.noname-c con 0x7f851409d430 addr v2:172.21.15.71:3300/0 2024-09-11T01:32:53.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8519816640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:53.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8519816640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f851409c9c0 con 0x7f851409d030 2024-09-11T01:32:53.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8519816640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:53.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8519816640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f85140a6e30 con 0x7f851409d430 2024-09-11T01:32:53.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8519816640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:53.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8519816640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f851409ba20 con 0x7f8514138230 2024-09-11T01:32:53.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8519816640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:53.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8519816640 10 monclient(hunting): authenticate will time out at 980.004287s 2024-09-11T01:32:53.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8512ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8514138230 0x7f8514138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:53.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85127fc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f851409d030 0x7f85140a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:53.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8512ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8514138230 0x7f8514138610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:39390/0 (socket says 172.21.15.12:39390) 2024-09-11T01:32:53.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8512ffd640 1 -- 172.21.15.12:0/2710101277 learned_addr learned my addr 172.21.15.12:0/2710101277 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:53.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85127fc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f851409d030 0x7f85140a5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42842/0 (socket says 172.21.15.12:42842) 2024-09-11T01:32:53.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8512ffd640 10 monclient(hunting): get_auth_request con 0x7f8514138230 auth_method 0 2024-09-11T01:32:53.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8512ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:53.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8512ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:53.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8512ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:53.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85127fc640 10 monclient(hunting): get_auth_request con 0x7f851409d030 auth_method 0 2024-09-11T01:32:53.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:53.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85127fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:53.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:53.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85127fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:53.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85127fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:53.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85137fe640 1 --2- 172.21.15.12:0/2710101277 >> v2:172.21.15.71:3300/0 conn(0x7f851409d430 0x7f85140a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:53.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85127fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:53.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8512ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:53.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8512ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:53.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8512ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:53.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85137fe640 10 monclient(hunting): get_auth_request con 0x7f851409d430 auth_method 0 2024-09-11T01:32:53.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:53.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85137fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:53.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:53.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:53.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:53.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:53.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85127fc640 10 monclient(hunting): handle_auth_done global_id 24345 payload 274 2024-09-11T01:32:53.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85127fc640 1 -- 172.21.15.12:0/2710101277 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8514138230 msgr2=0x7f8514138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:53.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85127fc640 1 --2- 172.21.15.12:0/2710101277 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8514138230 0x7f8514138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:53.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85127fc640 1 -- 172.21.15.12:0/2710101277 >> v2:172.21.15.71:3300/0 conn(0x7f851409d430 msgr2=0x7f85140a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:53.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85127fc640 1 --2- 172.21.15.12:0/2710101277 >> v2:172.21.15.71:3300/0 conn(0x7f851409d430 0x7f85140a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:53.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85127fc640 10 monclient: _finish_hunting 0 2024-09-11T01:32:53.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85127fc640 1 monclient: found mon.noname-a 2024-09-11T01:32:53.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:53.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85127fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:32:53.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f85127fc640 1 -- 172.21.15.12:0/2710101277 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85140a66a0 con 0x7f851409d030 2024-09-11T01:32:53.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.562+0000 7f8512ffd640 1 --2- 172.21.15.12:0/2710101277 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8514138230 0x7f8514138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:32:53.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.565+0000 7f85127fc640 1 -- 172.21.15.12:0/2710101277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f851409d030 msgr2=0x7f85140a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:53.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.565+0000 7f85127fc640 1 --2- 172.21.15.12:0/2710101277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f851409d030 0x7f85140a5020 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f851409cf80 tx=0x7f84fc029fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e6086948cc74203 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:53.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.565+0000 7f8511ffb640 1 -- 172.21.15.12:0/2710101277 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84fc037070 con 0x7f851409d030 2024-09-11T01:32:53.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.565+0000 7f8511ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:53.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.565+0000 7f8511ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:32:53.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.565+0000 7f8511ffb640 10 monclient: dump: 2024-09-11T01:32:53.575 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:53.575 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:53.576 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:53.576 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:53.576 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:53.576 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:53.576 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:53.576 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:53.576 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:53.576 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:53.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.565+0000 7f8511ffb640 10 monclient: _finish_auth 0 2024-09-11T01:32:53.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.565+0000 7f8511ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:32:53.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.565+0000 7f8511ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:53.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.565+0000 7f8511ffb640 1 -- 172.21.15.12:0/2710101277 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84fc032440 con 0x7f851409d030 2024-09-11T01:32:53.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.565+0000 7f8511ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:53.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.565+0000 7f8511ffb640 1 -- 172.21.15.12:0/2710101277 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84fc032810 con 0x7f851409d030 2024-09-11T01:32:53.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.565+0000 7f8511ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:53.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.565+0000 7f8511ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:53.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.565+0000 7f8511ffb640 10 monclient: dump: 2024-09-11T01:32:53.577 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:53.577 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:53.577 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:53.577 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:53.577 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:53.577 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:53.577 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:53.577 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:53.577 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:53.577 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:53.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.565+0000 7f8519816640 5 monclient: authenticate success, global_id 24345 2024-09-11T01:32:53.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.565+0000 7f8519816640 10 monclient: get_monmap_and_config success 2024-09-11T01:32:53.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.565+0000 7f8519816640 4 set_mon_vals no callback set 2024-09-11T01:32:53.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.566+0000 7f8519816640 10 monclient: shutdown 2024-09-11T01:32:53.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.566+0000 7f8519816640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:53.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.566+0000 7f8519816640 1 -- 172.21.15.12:0/2710101277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f851409d030 msgr2=0x7f85140a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:53.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.566+0000 7f8519816640 1 --2- 172.21.15.12:0/2710101277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f851409d030 0x7f85140a5020 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f851409cf80 tx=0x7f84fc029fe0 comp rx=0 tx=0).stop 2024-09-11T01:32:53.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.566+0000 7f8519816640 1 -- 172.21.15.12:0/2710101277 shutdown_connections 2024-09-11T01:32:53.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.566+0000 7f8519816640 1 --2- 172.21.15.12:0/2710101277 >> v2:172.21.15.71:3300/0 conn(0x7f851409d430 0x7f85140a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:53.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.566+0000 7f8519816640 1 --2- 172.21.15.12:0/2710101277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f851409d030 0x7f85140a5020 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:53.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.566+0000 7f8519816640 1 --2- 172.21.15.12:0/2710101277 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8514138230 0x7f8514138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:53.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.566+0000 7f8519816640 1 -- 172.21.15.12:0/2710101277 >> 172.21.15.12:0/2710101277 conn(0x7f85140a7120 msgr2=0x7f85140a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:53.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.566+0000 7f8519816640 1 -- 172.21.15.12:0/2710101277 shutdown_connections 2024-09-11T01:32:53.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.566+0000 7f8519816640 1 -- 172.21.15.12:0/2710101277 wait complete. 2024-09-11T01:32:53.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.567+0000 7f8519816640 10 monclient: build_initial_monmap 2024-09-11T01:32:53.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.567+0000 7f8519816640 10 monclient: monmap: 2024-09-11T01:32:53.579 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:53.579 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:32:53.579 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:32:53.579 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:32:53.579 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:53.579 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:53.579 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:32:53.579 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:32:53.579 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:32:53.579 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:53.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.567+0000 7f8519816640 1 Processor -- start 2024-09-11T01:32:53.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.567+0000 7f8519816640 1 -- start start 2024-09-11T01:32:53.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.567+0000 7f8519816640 10 monclient: init 2024-09-11T01:32:53.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f8519816640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:53.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f8519816640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:32:53.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f8519816640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f851409d030 0x7f8514136fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:53.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f8519816640 10 monclient(hunting): picked mon.noname-b con 0x7f851409d030 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:32:53.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f8519816640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f851409d430 0x7f8514120c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:53.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f8519816640 10 monclient(hunting): picked mon.noname-a con 0x7f851409d430 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:32:53.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f8519816640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8514138230 0x7f85141211b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:53.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f8519816640 10 monclient(hunting): picked mon.noname-c con 0x7f8514138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:53.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f8519816640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:53.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f8519816640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f851406f970 con 0x7f851409d430 2024-09-11T01:32:53.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f8519816640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:53.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f8519816640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f851406f7f0 con 0x7f851409d030 2024-09-11T01:32:53.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f8519816640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:53.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f8519816640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f85140a5aa0 con 0x7f8514138230 2024-09-11T01:32:53.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f8519816640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:53.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f8519816640 10 monclient(hunting): authenticate will time out at 980.010169s 2024-09-11T01:32:53.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85127fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f851409d430 0x7f8514120c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:53.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85137fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8514138230 0x7f85141211b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:53.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85127fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f851409d430 0x7f8514120c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42850/0 (socket says 172.21.15.12:42850) 2024-09-11T01:32:53.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85127fc640 1 -- 172.21.15.12:0/118378025 learned_addr learned my addr 172.21.15.12:0/118378025 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:53.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85127fc640 10 monclient(hunting): get_auth_request con 0x7f851409d430 auth_method 0 2024-09-11T01:32:53.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:53.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85127fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:53.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:53.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85137fe640 10 monclient(hunting): get_auth_request con 0x7f8514138230 auth_method 0 2024-09-11T01:32:53.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:53.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85137fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:53.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:53.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85127fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:53.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85127fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:53.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85127fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:53.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:53.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:53.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:53.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f8512ffd640 1 --2- 172.21.15.12:0/118378025 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f851409d030 0x7f8514136fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:53.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f8512ffd640 10 monclient(hunting): get_auth_request con 0x7f851409d030 auth_method 0 2024-09-11T01:32:53.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f8512ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:53.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f8512ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:53.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f8512ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:53.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f8512ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:53.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f8512ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:53.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f8512ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:53.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85127fc640 10 monclient(hunting): handle_auth_done global_id 24348 payload 824 2024-09-11T01:32:53.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85127fc640 1 -- 172.21.15.12:0/118378025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8514138230 msgr2=0x7f85141211b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:53.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85127fc640 1 --2- 172.21.15.12:0/118378025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8514138230 0x7f85141211b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:53.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85127fc640 1 -- 172.21.15.12:0/118378025 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f851409d030 msgr2=0x7f8514136fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:53.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85127fc640 1 --2- 172.21.15.12:0/118378025 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f851409d030 0x7f8514136fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:53.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85127fc640 10 monclient: _finish_hunting 0 2024-09-11T01:32:53.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85127fc640 1 monclient: found mon.noname-a 2024-09-11T01:32:53.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:53.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85127fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:32:53.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85127fc640 1 -- 172.21.15.12:0/118378025 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85141223c0 con 0x7f851409d430 2024-09-11T01:32:53.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.568+0000 7f85137fe640 1 --2- 172.21.15.12:0/118378025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8514138230 0x7f85141211b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:32:53.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.570+0000 7f85127fc640 1 -- 172.21.15.12:0/118378025 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f851409d430 msgr2=0x7f8514120c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:53.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.570+0000 7f85127fc640 1 --2- 172.21.15.12:0/118378025 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f851409d430 0x7f8514120c70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f84fc02a5d0 tx=0x7f84fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:53.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.571+0000 7f850bfff640 1 -- 172.21.15.12:0/118378025 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84fc040020 con 0x7f851409d430 2024-09-11T01:32:53.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.571+0000 7f850bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:53.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.571+0000 7f850bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:32:53.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.571+0000 7f850bfff640 10 monclient: dump: 2024-09-11T01:32:53.585 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:53.585 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:53.585 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:53.585 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:53.585 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:53.585 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:53.585 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:53.585 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:53.586 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:53.586 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:53.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.571+0000 7f850bfff640 10 monclient: _finish_auth 0 2024-09-11T01:32:53.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.571+0000 7f850bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:32:53.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.571+0000 7f850bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:53.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.571+0000 7f8519816640 5 monclient: authenticate success, global_id 24348 2024-09-11T01:32:53.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.571+0000 7f850bfff640 1 -- 172.21.15.12:0/118378025 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84fc038040 con 0x7f851409d430 2024-09-11T01:32:53.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.571+0000 7f8519816640 10 monclient: _renew_subs 2024-09-11T01:32:53.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.571+0000 7f8519816640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:53.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.571+0000 7f8519816640 1 -- 172.21.15.12:0/118378025 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8514123e80 con 0x7f851409d430 2024-09-11T01:32:53.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.571+0000 7f850bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:53.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.571+0000 7f850bfff640 1 -- 172.21.15.12:0/118378025 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84fc037070 con 0x7f851409d430 2024-09-11T01:32:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.571+0000 7f850bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.571+0000 7f850bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.571+0000 7f850bfff640 10 monclient: dump: 2024-09-11T01:32:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:53.587 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.571+0000 7f8519816640 10 monclient: _renew_subs 2024-09-11T01:32:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.571+0000 7f8519816640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.571+0000 7f8519816640 1 -- 172.21.15.12:0/118378025 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8514124440 con 0x7f851409d430 2024-09-11T01:32:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.571+0000 7f85117fa640 4 set_mon_vals no callback set 2024-09-11T01:32:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.572+0000 7f850bfff640 1 -- 172.21.15.12:0/118378025 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f84fc03cb40 con 0x7f851409d430 2024-09-11T01:32:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.572+0000 7f8519816640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.572+0000 7f8519816640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.572+0000 7f8519816640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.572+0000 7f8519816640 1 -- 172.21.15.12:0/118378025 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8514138610 con 0x7f851409d430 2024-09-11T01:32:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.572+0000 7f850bfff640 1 --2- 172.21.15.12:0/118378025 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f84f403e260 0x7f84f4040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.572+0000 7f850bfff640 1 -- 172.21.15.12:0/118378025 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 4755+0+0 (secure 0 0 0) 0x7f84fc0730c0 con 0x7f851409d430 2024-09-11T01:32:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.572+0000 7f8512ffd640 1 --2- 172.21.15.12:0/118378025 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f84f403e260 0x7f84f4040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.572+0000 7f8512ffd640 10 monclient: get_auth_request con 0x7f84f403e260 auth_method 0 2024-09-11T01:32:53.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.575+0000 7f8512ffd640 1 --2- 172.21.15.12:0/118378025 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f84f403e260 0x7f84f4040720 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8504009630 tx=0x7f8504009240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:53.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.576+0000 7f850bfff640 1 -- 172.21.15.12:0/118378025 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f84fc04a030 con 0x7f851409d430 2024-09-11T01:32:53.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.576+0000 7f850bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:53.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.576+0000 7f850bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:32:53.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.833+0000 7f8519816640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:32:53.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.834+0000 7f8519816640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:32:53.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.834+0000 7f8519816640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:53.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.834+0000 7f8519816640 1 -- 172.21.15.12:0/118378025 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8514124640 con 0x7f851409d430 2024-09-11T01:32:53.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.834+0000 7f850bfff640 1 -- 172.21.15.12:0/118378025 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f84fc0313a0 con 0x7f851409d430 2024-09-11T01:32:53.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.834+0000 7f850bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:32:53.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.834+0000 7f850bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:32:53.836 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:32:53.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.836+0000 7f8519816640 1 -- 172.21.15.12:0/118378025 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f84f403e260 msgr2=0x7f84f4040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:53.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.836+0000 7f8519816640 1 --2- 172.21.15.12:0/118378025 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f84f403e260 0x7f84f4040720 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8504009630 tx=0x7f8504009240 comp rx=0 tx=0).stop 2024-09-11T01:32:53.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.836+0000 7f8519816640 10 monclient: shutdown 2024-09-11T01:32:53.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.836+0000 7f8519816640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:53.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.836+0000 7f8519816640 1 -- 172.21.15.12:0/118378025 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f851409d430 msgr2=0x7f8514120c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:53.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.836+0000 7f8519816640 1 --2- 172.21.15.12:0/118378025 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f851409d430 0x7f8514120c70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f84fc02a5d0 tx=0x7f84fc0046c0 comp rx=0 tx=0).stop 2024-09-11T01:32:53.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.836+0000 7f8519816640 1 -- 172.21.15.12:0/118378025 shutdown_connections 2024-09-11T01:32:53.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.836+0000 7f8519816640 1 --2- 172.21.15.12:0/118378025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8514138230 0x7f85141211b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:53.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.836+0000 7f8519816640 1 --2- 172.21.15.12:0/118378025 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f84f403e260 0x7f84f4040720 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:53.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.836+0000 7f8519816640 1 --2- 172.21.15.12:0/118378025 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f851409d430 0x7f8514120c70 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:53.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.836+0000 7f8519816640 1 --2- 172.21.15.12:0/118378025 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f851409d030 0x7f8514136fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:53.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.836+0000 7f8519816640 1 -- 172.21.15.12:0/118378025 >> 172.21.15.12:0/118378025 conn(0x7f85140a7120 msgr2=0x7f85140a32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:53.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.836+0000 7f8519816640 1 -- 172.21.15.12:0/118378025 shutdown_connections 2024-09-11T01:32:53.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.836+0000 7f8519816640 1 -- 172.21.15.12:0/118378025 wait complete. 2024-09-11T01:32:53.848 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:32:53.848 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:32:53.848 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:32:53.849 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:32:53.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.948+0000 7efedb17d640 10 monclient: get_monmap_and_config 2024-09-11T01:32:53.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.948+0000 7efedb17d640 10 monclient: build_initial_monmap 2024-09-11T01:32:53.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.948+0000 7efedb17d640 10 monclient: monmap: 2024-09-11T01:32:53.951 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:53.951 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:53.951 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:32:53.949703+0000 2024-09-11T01:32:53.951 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:32:53.949703+0000 2024-09-11T01:32:53.951 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:53.951 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:53.951 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:32:53.951 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:32:53.952 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:32:53.952 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:32:53.952 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:32:53.952 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:53.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.949+0000 7efedb17d640 1 Processor -- start 2024-09-11T01:32:53.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.949+0000 7efedb17d640 1 -- start start 2024-09-11T01:32:53.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.949+0000 7efedb17d640 10 monclient: init 2024-09-11T01:32:53.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efedb17d640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:53.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efedb17d640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-11T01:32:53.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efedb17d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efed4134ad0 addr v1:172.21.15.12:6789/0 2024-09-11T01:32:53.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efedb17d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efed40a2200 addr v1:172.21.15.71:6789/0 2024-09-11T01:32:53.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efedb17d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efed41355c0 0x7efed413d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:53.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efedb17d640 10 monclient(hunting): picked mon.noname-b con 0x7efed41355c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:53.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efedb17d640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efed409ed30 con 0x7efed4134ad0 2024-09-11T01:32:53.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efedb17d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efed409f010 con 0x7efed40a2200 2024-09-11T01:32:53.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efedb17d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:53.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efedb17d640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7efed409f5d0 con 0x7efed41355c0 2024-09-11T01:32:53.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efedb17d640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:53.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efedb17d640 10 monclient(hunting): authenticate will time out at 980.392239s 2024-09-11T01:32:53.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efed8f19640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7efed4134ad0 0x7efed4134eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:39816/0 (socket says 172.21.15.12:39816) 2024-09-11T01:32:53.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efed8f19640 1 -- 172.21.15.12:0/2711970858 learned_addr learned my addr 172.21.15.12:0/2711970858 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:32:53.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efed971a640 1 --2- 172.21.15.12:0/2711970858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efed41355c0 0x7efed413d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:53.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efed971a640 10 monclient(hunting): get_auth_request con 0x7efed41355c0 auth_method 0 2024-09-11T01:32:53.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efed971a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:53.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efed971a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:53.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efed971a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:53.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efed971a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:53.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efed971a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:53.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efed971a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:53.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efed971a640 10 monclient(hunting): handle_auth_done global_id 44123 payload 274 2024-09-11T01:32:53.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efed971a640 1 -- 172.21.15.12:0/2711970858 >> v1:172.21.15.71:6789/0 conn(0x7efed40a2200 legacy=0x7efed413a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:53.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efed971a640 1 -- 172.21.15.12:0/2711970858 >> v1:172.21.15.12:6789/0 conn(0x7efed4134ad0 legacy=0x7efed4134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:53.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efed971a640 10 monclient: _finish_hunting 0 2024-09-11T01:32:53.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efed971a640 1 monclient: found mon.noname-b 2024-09-11T01:32:53.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efed971a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:53.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efed971a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:32:53.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.950+0000 7efed971a640 1 -- 172.21.15.12:0/2711970858 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efed409f2f0 con 0x7efed41355c0 2024-09-11T01:32:53.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.953+0000 7efed971a640 1 -- 172.21.15.12:0/2711970858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efed41355c0 msgr2=0x7efed413d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:53.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.953+0000 7efed971a640 1 --2- 172.21.15.12:0/2711970858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efed41355c0 0x7efed413d730 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efec4009800 tx=0x7efec4029fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5a6c7383a2afbe3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:53.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.953+0000 7efed37fe640 1 -- 172.21.15.12:0/2711970858 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec4037070 con 0x7efed41355c0 2024-09-11T01:32:53.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.953+0000 7efed37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:53.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.953+0000 7efed37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:32:53.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.953+0000 7efed37fe640 10 monclient: dump: 2024-09-11T01:32:53.963 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:53.963 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:53.963 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:53.963 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:53.963 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:53.964 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:53.964 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:53.964 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:53.964 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:53.964 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:53.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.953+0000 7efed37fe640 10 monclient: _finish_auth 0 2024-09-11T01:32:53.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.953+0000 7efed37fe640 10 monclient: _check_auth_tickets 2024-09-11T01:32:53.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.953+0000 7efed37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:53.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.953+0000 7efedb17d640 5 monclient: authenticate success, global_id 44123 2024-09-11T01:32:53.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.953+0000 7efedb17d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:32:53.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.953+0000 7efed37fe640 1 -- 172.21.15.12:0/2711970858 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efec4032440 con 0x7efed41355c0 2024-09-11T01:32:53.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.953+0000 7efed37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:53.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.953+0000 7efedb17d640 10 monclient: get_monmap_and_config success 2024-09-11T01:32:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.953+0000 7efedb17d640 4 set_mon_vals no callback set 2024-09-11T01:32:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.953+0000 7efed37fe640 1 -- 172.21.15.12:0/2711970858 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec40327f0 con 0x7efed41355c0 2024-09-11T01:32:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.953+0000 7efed37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.953+0000 7efed37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:32:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.953+0000 7efed37fe640 10 monclient: dump: 2024-09-11T01:32:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:53.965 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:53.966 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:53.966 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:53.966 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:53.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.954+0000 7efedb17d640 10 monclient: shutdown 2024-09-11T01:32:53.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.954+0000 7efedb17d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:53.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.954+0000 7efedb17d640 1 -- 172.21.15.12:0/2711970858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efed41355c0 msgr2=0x7efed413d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:53.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.954+0000 7efedb17d640 1 --2- 172.21.15.12:0/2711970858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efed41355c0 0x7efed413d730 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efec4009800 tx=0x7efec4029fe0 comp rx=0 tx=0).stop 2024-09-11T01:32:53.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.954+0000 7efedb17d640 1 -- 172.21.15.12:0/2711970858 shutdown_connections 2024-09-11T01:32:53.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.954+0000 7efedb17d640 1 --2- 172.21.15.12:0/2711970858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efed41355c0 0x7efed413d730 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:53.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.954+0000 7efedb17d640 1 -- 172.21.15.12:0/2711970858 >> 172.21.15.12:0/2711970858 conn(0x7efed40a2f70 msgr2=0x7efed40a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:53.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.954+0000 7efedb17d640 1 -- 172.21.15.12:0/2711970858 shutdown_connections 2024-09-11T01:32:53.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.954+0000 7efedb17d640 1 -- 172.21.15.12:0/2711970858 wait complete. 2024-09-11T01:32:53.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.954+0000 7efedb17d640 10 monclient: build_initial_monmap 2024-09-11T01:32:53.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.954+0000 7efedb17d640 10 monclient: monmap: 2024-09-11T01:32:53.967 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:53.967 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:32:53.967 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:32:53.967 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:32:53.967 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:53.967 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:53.967 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:32:53.967 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:32:53.967 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:32:53.967 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:53.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.955+0000 7efedb17d640 1 Processor -- start 2024-09-11T01:32:53.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.955+0000 7efedb17d640 1 -- start start 2024-09-11T01:32:53.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.955+0000 7efedb17d640 10 monclient: init 2024-09-11T01:32:53.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.955+0000 7efedb17d640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:53.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.955+0000 7efedb17d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:32:53.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.955+0000 7efedb17d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efed40a2200 0x7efed411d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:53.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.955+0000 7efedb17d640 10 monclient(hunting): picked mon.noname-c con 0x7efed40a2200 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:53.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.955+0000 7efedb17d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efed4134ad0 0x7efed413b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:53.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.955+0000 7efedb17d640 10 monclient(hunting): picked mon.noname-a con 0x7efed4134ad0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:32:53.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.955+0000 7efedb17d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efed41355c0 0x7efed4133ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:53.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.955+0000 7efedb17d640 10 monclient(hunting): picked mon.noname-b con 0x7efed41355c0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:32:53.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.955+0000 7efedb17d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:53.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.955+0000 7efedb17d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7efed409cde0 con 0x7efed4134ad0 2024-09-11T01:32:53.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.955+0000 7efedb17d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:53.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.955+0000 7efedb17d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7efed40a01e0 con 0x7efed41355c0 2024-09-11T01:32:53.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.955+0000 7efedb17d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:53.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.955+0000 7efedb17d640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7efed413dc70 con 0x7efed40a2200 2024-09-11T01:32:53.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.955+0000 7efedb17d640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:53.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.955+0000 7efedb17d640 10 monclient(hunting): authenticate will time out at 980.397890s 2024-09-11T01:32:53.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.955+0000 7efed3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efed4134ad0 0x7efed413b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:53.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.955+0000 7efed8f19640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efed40a2200 0x7efed411d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:53.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efed4134ad0 0x7efed413b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42866/0 (socket says 172.21.15.12:42866) 2024-09-11T01:32:53.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed3fff640 1 -- 172.21.15.12:0/3598784752 learned_addr learned my addr 172.21.15.12:0/3598784752 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:53.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed3fff640 10 monclient(hunting): get_auth_request con 0x7efed4134ad0 auth_method 0 2024-09-11T01:32:53.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed8f19640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efed40a2200 0x7efed411d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:39420/0 (socket says 172.21.15.12:39420) 2024-09-11T01:32:53.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:53.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:53.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:53.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed8f19640 10 monclient(hunting): get_auth_request con 0x7efed40a2200 auth_method 0 2024-09-11T01:32:53.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed8f19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:53.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed8f19640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:53.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed8f19640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:53.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:53.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:53.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:53.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed8f19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:53.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed8f19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:53.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed971a640 1 --2- 172.21.15.12:0/3598784752 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efed41355c0 0x7efed4133ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:53.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed8f19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:53.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed971a640 10 monclient(hunting): get_auth_request con 0x7efed41355c0 auth_method 0 2024-09-11T01:32:53.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed971a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:53.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed971a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:53.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed971a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:53.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed971a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:53.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed971a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:53.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed971a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:53.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed3fff640 10 monclient(hunting): handle_auth_done global_id 24351 payload 824 2024-09-11T01:32:53.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed3fff640 1 -- 172.21.15.12:0/3598784752 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efed40a2200 msgr2=0x7efed411d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:53.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed3fff640 1 --2- 172.21.15.12:0/3598784752 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efed40a2200 0x7efed411d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:53.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed3fff640 1 -- 172.21.15.12:0/3598784752 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efed41355c0 msgr2=0x7efed4133ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:53.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed3fff640 1 --2- 172.21.15.12:0/3598784752 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efed41355c0 0x7efed4133ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:53.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed3fff640 10 monclient: _finish_hunting 0 2024-09-11T01:32:53.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed3fff640 1 monclient: found mon.noname-a 2024-09-11T01:32:53.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:53.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:32:53.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed3fff640 1 -- 172.21.15.12:0/3598784752 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efed40a4590 con 0x7efed4134ad0 2024-09-11T01:32:53.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.956+0000 7efed8f19640 1 --2- 172.21.15.12:0/3598784752 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efed40a2200 0x7efed411d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:32:53.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.958+0000 7efed3fff640 1 -- 172.21.15.12:0/3598784752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efed4134ad0 msgr2=0x7efed413b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:53.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.958+0000 7efed3fff640 1 --2- 172.21.15.12:0/3598784752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efed4134ad0 0x7efed413b770 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efec800c400 tx=0x7efec800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:53.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.958+0000 7efed17fa640 1 -- 172.21.15.12:0/3598784752 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec80119a0 con 0x7efed4134ad0 2024-09-11T01:32:53.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.958+0000 7efed17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:53.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.958+0000 7efed17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:32:53.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.958+0000 7efed17fa640 10 monclient: dump: 2024-09-11T01:32:53.973 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:53.973 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:53.973 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:53.973 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:53.973 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:53.973 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:53.973 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:53.973 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:53.973 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:53.973 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:53.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.958+0000 7efed17fa640 10 monclient: _finish_auth 0 2024-09-11T01:32:53.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.958+0000 7efed17fa640 10 monclient: _check_auth_tickets 2024-09-11T01:32:53.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.958+0000 7efed17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:53.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.958+0000 7efed17fa640 1 -- 172.21.15.12:0/3598784752 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efec8017070 con 0x7efed4134ad0 2024-09-11T01:32:53.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.958+0000 7efedb17d640 5 monclient: authenticate success, global_id 24351 2024-09-11T01:32:53.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.958+0000 7efedb17d640 10 monclient: _renew_subs 2024-09-11T01:32:53.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.958+0000 7efedb17d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:53.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.958+0000 7efedb17d640 1 -- 172.21.15.12:0/3598784752 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efed413cd20 con 0x7efed4134ad0 2024-09-11T01:32:53.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.958+0000 7efed17fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:53.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.958+0000 7efed17fa640 1 -- 172.21.15.12:0/3598784752 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec8011dc0 con 0x7efed4134ad0 2024-09-11T01:32:53.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.958+0000 7efed17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:53.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.958+0000 7efed17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:53.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.958+0000 7efed17fa640 10 monclient: dump: 2024-09-11T01:32:53.974 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:53.974 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:53.975 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:53.975 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:53.975 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:53.975 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:53.975 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:53.975 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:53.975 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:53.975 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:53.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.958+0000 7efed2ffd640 4 set_mon_vals no callback set 2024-09-11T01:32:53.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.958+0000 7efedb17d640 10 monclient: _renew_subs 2024-09-11T01:32:53.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.958+0000 7efedb17d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:53.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.958+0000 7efedb17d640 1 -- 172.21.15.12:0/3598784752 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efed413c6a0 con 0x7efed4134ad0 2024-09-11T01:32:53.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.959+0000 7efed17fa640 1 -- 172.21.15.12:0/3598784752 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7efec801c460 con 0x7efed4134ad0 2024-09-11T01:32:53.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.959+0000 7efedb17d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:53.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.959+0000 7efedb17d640 1 --2- 172.21.15.12:0/3598784752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efed4134430 0x7efed41cb240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:53.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.959+0000 7efedb17d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:53.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.959+0000 7efedb17d640 1 -- 172.21.15.12:0/3598784752 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7efed40a52f0 con 0x7efed4134430 2024-09-11T01:32:53.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.959+0000 7efed8f19640 1 --2- 172.21.15.12:0/3598784752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efed4134430 0x7efed41cb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:53.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.959+0000 7efed8f19640 10 monclient: get_auth_request con 0x7efed4134430 auth_method 0 2024-09-11T01:32:53.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.959+0000 7efed8f19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:53.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.959+0000 7efed8f19640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:53.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.959+0000 7efed8f19640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:53.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.959+0000 7efed17fa640 1 --2- 172.21.15.12:0/3598784752 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efec003e210 0x7efec00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:53.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.959+0000 7efed17fa640 1 -- 172.21.15.12:0/3598784752 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 4755+0+0 (secure 0 0 0) 0x7efec8052c10 con 0x7efed4134ad0 2024-09-11T01:32:53.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.959+0000 7efed8f19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:53.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.959+0000 7efed8f19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:53.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.960+0000 7efed8f19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:53.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.960+0000 7efed8f19640 10 monclient(hunting): handle_auth_done global_id 24354 payload 824 2024-09-11T01:32:53.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.960+0000 7efed8f19640 1 -- 172.21.15.12:0/3598784752 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efed4134eb0 con 0x7efed4134430 2024-09-11T01:32:53.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.960+0000 7efed971a640 1 --2- 172.21.15.12:0/3598784752 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efec003e210 0x7efec00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:53.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.960+0000 7efed971a640 10 monclient: get_auth_request con 0x7efec003e210 auth_method 0 2024-09-11T01:32:53.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.962+0000 7efed8f19640 1 -- 172.21.15.12:0/3598784752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efed4134430 msgr2=0x7efed41cb240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:53.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.962+0000 7efed8f19640 1 --2- 172.21.15.12:0/3598784752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efed4134430 0x7efed41cb240 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efecc00d380 tx=0x7efecc00d840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:53.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.962+0000 7efed17fa640 1 -- 172.21.15.12:0/3598784752 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efecc019070 con 0x7efed4134430 2024-09-11T01:32:53.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.962+0000 7efed17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:53.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.962+0000 7efed17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:53.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.962+0000 7efed17fa640 10 monclient: dump: 2024-09-11T01:32:53.980 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:53.980 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:53.980 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:53.980 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:53.980 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:53.980 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:53.980 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:53.980 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:53.980 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:53.980 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:53.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.962+0000 7efed17fa640 1 -- 172.21.15.12:0/3598784752 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7efecc007530 con 0x7efed4134430 2024-09-11T01:32:53.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.962+0000 7efed17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:53.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.962+0000 7efed17fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:32:53.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.962+0000 7efed17fa640 1 -- 172.21.15.12:0/3598784752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efed4134430 msgr2=0x7efed41cb240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:53.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.962+0000 7efed17fa640 1 --2- 172.21.15.12:0/3598784752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efed4134430 0x7efed41cb240 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efecc00d380 tx=0x7efecc00d840 comp rx=0 tx=0).stop 2024-09-11T01:32:53.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.962+0000 7efed17fa640 1 -- 172.21.15.12:0/3598784752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efed4134430 msgr2=0x7efed41cb240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:32:53.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.962+0000 7efed17fa640 1 --2- 172.21.15.12:0/3598784752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efed4134430 0x7efed41cb240 secure :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efecc00d380 tx=0x7efecc00d840 comp rx=0 tx=0).stop 2024-09-11T01:32:53.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.962+0000 7efed971a640 1 --2- 172.21.15.12:0/3598784752 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efec003e210 0x7efec00406d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efed40a72b0 tx=0x7efec402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:53.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.968+0000 7efedb17d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:32:53.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.968+0000 7efedb17d640 1 --2- 172.21.15.12:0/3598784752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efed41cb780 0x7efed41cbb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:53.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.968+0000 7efedb17d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:53.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.968+0000 7efedb17d640 1 -- 172.21.15.12:0/3598784752 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7efed41cc0d0 con 0x7efed41cb780 2024-09-11T01:32:53.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.969+0000 7efed8f19640 1 --2- 172.21.15.12:0/3598784752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efed41cb780 0x7efed41cbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:53.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.969+0000 7efed8f19640 10 monclient: get_auth_request con 0x7efed41cb780 auth_method 0 2024-09-11T01:32:53.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.969+0000 7efed8f19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:53.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.969+0000 7efed8f19640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:53.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.969+0000 7efed8f19640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:53.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.969+0000 7efed8f19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:53.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.969+0000 7efed8f19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:53.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.969+0000 7efed8f19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:53.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.969+0000 7efed8f19640 10 monclient(hunting): handle_auth_done global_id 24357 payload 824 2024-09-11T01:32:53.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.969+0000 7efed8f19640 1 -- 172.21.15.12:0/3598784752 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efed40a1a60 con 0x7efed41cb780 2024-09-11T01:32:53.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.971+0000 7efed8f19640 1 -- 172.21.15.12:0/3598784752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efed41cb780 msgr2=0x7efed41cbb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:53.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.971+0000 7efed8f19640 1 --2- 172.21.15.12:0/3598784752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efed41cb780 0x7efed41cbb90 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efecc00f240 tx=0x7efecc004cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:53.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.971+0000 7efed17fa640 1 -- 172.21.15.12:0/3598784752 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efecc019070 con 0x7efed41cb780 2024-09-11T01:32:53.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.971+0000 7efed17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:53.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.971+0000 7efed17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:53.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.971+0000 7efed17fa640 10 monclient: dump: 2024-09-11T01:32:53.983 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:53.983 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:53.983 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:53.983 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:53.983 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:53.983 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:53.983 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:53.983 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:53.983 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:53.983 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:53.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.971+0000 7efed17fa640 1 -- 172.21.15.12:0/3598784752 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4877 (secure 0 0 0) 0x7efecc01ba40 con 0x7efed41cb780 2024-09-11T01:32:53.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.971+0000 7efed17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:32:53.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.971+0000 7efed17fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:32:53.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.971+0000 7efed17fa640 1 -- 172.21.15.12:0/3598784752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efed41cb780 msgr2=0x7efed41cbb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:53.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.971+0000 7efed17fa640 1 --2- 172.21.15.12:0/3598784752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efed41cb780 0x7efed41cbb90 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efecc00f240 tx=0x7efecc004cc0 comp rx=0 tx=0).stop 2024-09-11T01:32:53.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.971+0000 7efed17fa640 1 -- 172.21.15.12:0/3598784752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efed41cb780 msgr2=0x7efed41cbb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:32:53.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.971+0000 7efed17fa640 1 --2- 172.21.15.12:0/3598784752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efed41cb780 0x7efed41cbb90 secure :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efecc00f240 tx=0x7efecc004cc0 comp rx=0 tx=0).stop 2024-09-11T01:32:53.984 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:32:53.984 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:32:53.984 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:32:53.984 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:32:53.985 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 38, 2024-09-11T01:32:53.985 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 111918, 2024-09-11T01:32:53.985 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:32:53.985 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:32:53.985 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:32:53.985 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:32:53.985 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:53.985 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 6, 2024-09-11T01:32:53.985 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:32:53.985 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:32:53.985 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:32:53.985 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:53.985 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:53.985 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:53.986 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:53.986 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:53.986 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:53.986 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:53.986 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:53.986 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:53.986 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:53.986 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:53.986 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:53.986 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:32:53.986 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:32:53.986 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:53.986 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:53.987 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:53.987 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:53.987 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:53.987 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:53.987 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:53.987 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:53.987 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:53.987 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:53.987 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:53.987 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:53.987 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:53.988 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:32:53.988 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:32:53.988 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:32:53.988 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:32:53.988 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:32:53.988 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:32:53.988 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:32:53.988 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:32:53.988 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:32:53.988 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:32:53.988 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:32:53.988 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:32:53.988 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:32:53.989 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:32:53.989 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:32:53.989 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:32:53.989 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:32:53.989 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:53.989 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:53.989 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:53.989 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:53.989 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:53.989 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:53.989 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:53.989 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:53.989 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:53.990 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:53.990 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:53.990 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:53.990 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:32:53.990 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:53.990 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:32:53.990 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:53.990 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:32:53.990 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:32:53.990 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:53.990 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:53.990 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:53.990 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:53.990 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:32:53.991 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:53.991 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:53.991 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:53.991 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:53.991 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:32:53.991 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:53.991 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:53.991 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:53.991 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:53.991 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:32:53.991 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:32:53.991 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:53.991 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:53.992 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:53.992 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:53.992 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:53.992 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:32:53.992 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:32:53.992 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:53.992 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:53.992 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:53.992 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:53.992 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:32:53.992 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:53.992 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:53.992 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:53.992 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:53.993 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:32:53.993 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:53.993 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:53.993 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:53.993 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:53.993 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:32:53.993 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:32:53.993 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:53.993 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:53.993 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:53.993 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:53.993 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:53.993 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:32:53.993 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:32:53.994 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:53.994 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:53.994 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:53.994 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:53.994 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:32:53.994 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:53.994 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:53.994 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:53.994 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:53.994 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:32:53.994 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:53.994 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:53.995 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:53.995 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:53.995 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:32:53.995 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:32:53.995 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:53.995 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:53.995 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:53.995 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:53.995 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:53.995 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:53.995 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:32:53.995 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:32:53.995 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:53.996 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:53.996 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:53.996 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:32:53.996 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:53.996 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:53.996 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:32:53.996 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:53.996 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:53.996 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:53.996 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:32:53.996 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:53.997 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:53.997 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:32:53.997 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:53.997 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:53.997 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:53.997 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:32:53.997 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:53.997 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:53.997 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:32:53.997 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:53.997 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:53.997 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:53.997 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:32:53.997 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:53.998 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:53.998 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:53.998 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:32:53.998 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:32:53.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.972+0000 7efedb17d640 1 -- 172.21.15.12:0/3598784752 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efec003e210 msgr2=0x7efec00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:54.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.972+0000 7efedb17d640 1 --2- 172.21.15.12:0/3598784752 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efec003e210 0x7efec00406d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efed40a72b0 tx=0x7efec402e000 comp rx=0 tx=0).stop 2024-09-11T01:32:54.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.972+0000 7efedb17d640 10 monclient: shutdown 2024-09-11T01:32:54.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.972+0000 7efedb17d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:54.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.972+0000 7efedb17d640 1 -- 172.21.15.12:0/3598784752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efed4134ad0 msgr2=0x7efed413b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:54.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.972+0000 7efedb17d640 1 --2- 172.21.15.12:0/3598784752 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efed4134ad0 0x7efed413b770 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efec800c400 tx=0x7efec800c8c0 comp rx=0 tx=0).stop 2024-09-11T01:32:54.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.972+0000 7efed971a640 1 -- 172.21.15.12:0/3598784752 reap_dead start 2024-09-11T01:32:54.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.972+0000 7efed971a640 1 -- 172.21.15.12:0/3598784752 reap_dead start 2024-09-11T01:32:54.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.972+0000 7efedb17d640 1 -- 172.21.15.12:0/3598784752 shutdown_connections 2024-09-11T01:32:54.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.972+0000 7efedb17d640 1 -- 172.21.15.12:0/3598784752 >> 172.21.15.12:0/3598784752 conn(0x7efed40a2f70 msgr2=0x7efed40a0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:54.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.972+0000 7efedb17d640 1 -- 172.21.15.12:0/3598784752 shutdown_connections 2024-09-11T01:32:54.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:53.972+0000 7efedb17d640 1 -- 172.21.15.12:0/3598784752 wait complete. 2024-09-11T01:32:54.001 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:32:54.001 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:32:54.001 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:32:54.089 INFO:tasks.rados.rados.0.smithi071.stdout:611: done (4 left) 2024-09-11T01:32:54.089 INFO:tasks.rados.rados.0.smithi071.stdout:612: done (3 left) 2024-09-11T01:32:54.089 INFO:tasks.rados.rados.0.smithi071.stdout:613: done (2 left) 2024-09-11T01:32:54.089 INFO:tasks.rados.rados.0.smithi071.stdout:614: done (1 left) 2024-09-11T01:32:54.089 INFO:tasks.rados.rados.0.smithi071.stdout:615: done (0 left) 2024-09-11T01:32:54.089 INFO:tasks.rados.rados.0.smithi071.stdout:616: copy_from oid 10 from oid 28 current snap is 60 2024-09-11T01:32:54.089 INFO:tasks.rados.rados.0.smithi071.stdout:617: snap_create 2024-09-11T01:32:54.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.105+0000 7f0061840640 10 monclient: get_monmap_and_config 2024-09-11T01:32:54.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.105+0000 7f0061840640 10 monclient: build_initial_monmap 2024-09-11T01:32:54.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.105+0000 7f0061840640 10 monclient: monmap: 2024-09-11T01:32:54.107 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:54.107 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:54.107 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:32:54.105783+0000 2024-09-11T01:32:54.107 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:32:54.105783+0000 2024-09-11T01:32:54.107 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:54.107 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:54.107 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:32:54.107 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:32:54.107 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:32:54.107 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:32:54.108 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:32:54.108 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:54.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.105+0000 7f0061840640 1 Processor -- start 2024-09-11T01:32:54.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.105+0000 7f0061840640 1 -- start start 2024-09-11T01:32:54.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.105+0000 7f0061840640 10 monclient: init 2024-09-11T01:32:54.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f0061840640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:54.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f0061840640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-11T01:32:54.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f0061840640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f005c137d50 0x7f005c09d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:54.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f0061840640 10 monclient(hunting): picked mon.noname-c con 0x7f005c137d50 addr v2:172.21.15.71:3300/0 2024-09-11T01:32:54.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f0061840640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f005c138130 addr v1:172.21.15.12:6789/0 2024-09-11T01:32:54.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f0061840640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c13d280 0x7f005c13f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:54.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f0061840640 10 monclient(hunting): picked mon.noname-b con 0x7f005c13d280 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:54.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f0061840640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f005c09ba20 con 0x7f005c138130 2024-09-11T01:32:54.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f0061840640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:54.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f0061840640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f005c09c650 con 0x7f005c137d50 2024-09-11T01:32:54.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f0061840640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:54.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f0061840640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f005c09c370 con 0x7f005c13d280 2024-09-11T01:32:54.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f0061840640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:54.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f0061840640 10 monclient(hunting): authenticate will time out at 980.548307s 2024-09-11T01:32:54.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f005a7fc640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f005c138130 0x7f005c138580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:39830/0 (socket says 172.21.15.12:39830) 2024-09-11T01:32:54.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f005a7fc640 1 -- 172.21.15.12:0/2223607662 learned_addr learned my addr 172.21.15.12:0/2223607662 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:32:54.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f005b7fe640 1 --2- 172.21.15.12:0/2223607662 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c13d280 0x7f005c13f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:54.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f005b7fe640 10 monclient(hunting): get_auth_request con 0x7f005c13d280 auth_method 0 2024-09-11T01:32:54.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f005b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:54.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f005b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:54.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f005b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:54.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f005affd640 1 --2- 172.21.15.12:0/2223607662 >> v2:172.21.15.71:3300/0 conn(0x7f005c137d50 0x7f005c09d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:54.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f005b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:54.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f005b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:54.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f005b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:54.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f005affd640 10 monclient(hunting): get_auth_request con 0x7f005c137d50 auth_method 0 2024-09-11T01:32:54.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f005affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:54.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f005affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:54.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f005affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:54.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f005b7fe640 10 monclient(hunting): handle_auth_done global_id 44129 payload 274 2024-09-11T01:32:54.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f005b7fe640 1 -- 172.21.15.12:0/2223607662 >> v2:172.21.15.71:3300/0 conn(0x7f005c137d50 msgr2=0x7f005c09d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:54.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f005b7fe640 1 --2- 172.21.15.12:0/2223607662 >> v2:172.21.15.71:3300/0 conn(0x7f005c137d50 0x7f005c09d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:54.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f005b7fe640 1 -- 172.21.15.12:0/2223607662 >> v1:172.21.15.12:6789/0 conn(0x7f005c138130 legacy=0x7f005c138580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:54.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f005b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:32:54.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f005b7fe640 1 monclient: found mon.noname-b 2024-09-11T01:32:54.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f005b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:54.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f005b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:32:54.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.106+0000 7f005b7fe640 1 -- 172.21.15.12:0/2223607662 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f005c0a6ce0 con 0x7f005c13d280 2024-09-11T01:32:54.114 INFO:tasks.rados.rados.0.smithi071.stdout:616: finishing copy_from racing read to smithi07134777-10 2024-09-11T01:32:54.120 INFO:tasks.rados.rados.0.smithi071.stdout:616: finishing copy_from to smithi07134777-10 2024-09-11T01:32:54.120 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 215 (ObjNum 200 snap 46 seq_num 200) dirty exists 2024-09-11T01:32:54.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.109+0000 7f005b7fe640 1 -- 172.21.15.12:0/2223607662 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c13d280 msgr2=0x7f005c13f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:54.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.109+0000 7f005b7fe640 1 --2- 172.21.15.12:0/2223607662 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c13d280 0x7f005c13f670 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f004c0097a0 tx=0x7f004c02a050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f94d0babfd423593 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:54.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.109+0000 7f0059ffb640 1 -- 172.21.15.12:0/2223607662 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004c037070 con 0x7f005c13d280 2024-09-11T01:32:54.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.109+0000 7f0059ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:54.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.109+0000 7f0059ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:32:54.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.109+0000 7f0059ffb640 10 monclient: dump: 2024-09-11T01:32:54.122 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:54.122 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:54.122 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.122 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.122 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:54.122 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:54.122 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:54.123 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:54.123 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:54.123 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:54.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.109+0000 7f0059ffb640 10 monclient: _finish_auth 0 2024-09-11T01:32:54.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.109+0000 7f0059ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:32:54.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.109+0000 7f0059ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:54.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.109+0000 7f0059ffb640 1 -- 172.21.15.12:0/2223607662 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f004c032440 con 0x7f005c13d280 2024-09-11T01:32:54.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.109+0000 7f0059ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:54.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.109+0000 7f0059ffb640 1 -- 172.21.15.12:0/2223607662 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004c032810 con 0x7f005c13d280 2024-09-11T01:32:54.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.109+0000 7f0059ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:54.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.109+0000 7f0059ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:32:54.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.109+0000 7f0059ffb640 10 monclient: dump: 2024-09-11T01:32:54.123 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:54.123 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:54.123 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.124 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.124 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:54.124 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:54.124 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:54.124 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:54.124 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:54.124 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:54.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.109+0000 7f0061840640 5 monclient: authenticate success, global_id 44129 2024-09-11T01:32:54.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.109+0000 7f0061840640 10 monclient: get_monmap_and_config success 2024-09-11T01:32:54.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.109+0000 7f0061840640 4 set_mon_vals no callback set 2024-09-11T01:32:54.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.110+0000 7f0061840640 10 monclient: shutdown 2024-09-11T01:32:54.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.110+0000 7f0061840640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:54.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.110+0000 7f0061840640 1 -- 172.21.15.12:0/2223607662 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c13d280 msgr2=0x7f005c13f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:54.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.110+0000 7f0061840640 1 --2- 172.21.15.12:0/2223607662 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c13d280 0x7f005c13f670 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f004c0097a0 tx=0x7f004c02a050 comp rx=0 tx=0).stop 2024-09-11T01:32:54.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.110+0000 7f0061840640 1 -- 172.21.15.12:0/2223607662 shutdown_connections 2024-09-11T01:32:54.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.110+0000 7f0061840640 1 --2- 172.21.15.12:0/2223607662 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c13d280 0x7f005c13f670 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:54.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.110+0000 7f0061840640 1 --2- 172.21.15.12:0/2223607662 >> v2:172.21.15.71:3300/0 conn(0x7f005c137d50 0x7f005c09d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:54.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.110+0000 7f0061840640 1 -- 172.21.15.12:0/2223607662 >> 172.21.15.12:0/2223607662 conn(0x7f005c0a7020 msgr2=0x7f005c0a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:54.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.110+0000 7f0061840640 1 -- 172.21.15.12:0/2223607662 shutdown_connections 2024-09-11T01:32:54.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.110+0000 7f0061840640 1 -- 172.21.15.12:0/2223607662 wait complete. 2024-09-11T01:32:54.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.111+0000 7f0061840640 10 monclient: build_initial_monmap 2024-09-11T01:32:54.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.111+0000 7f0061840640 10 monclient: monmap: 2024-09-11T01:32:54.125 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:54.125 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:32:54.125 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:32:54.125 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:32:54.125 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:54.125 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:54.126 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:32:54.126 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:32:54.126 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:32:54.126 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:54.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.112+0000 7f0061840640 1 Processor -- start 2024-09-11T01:32:54.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.112+0000 7f0061840640 1 -- start start 2024-09-11T01:32:54.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.112+0000 7f0061840640 10 monclient: init 2024-09-11T01:32:54.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.112+0000 7f0061840640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:54.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.112+0000 7f0061840640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:32:54.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.112+0000 7f0061840640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f005c137d50 0x7f005c1ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:54.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.112+0000 7f0061840640 10 monclient(hunting): picked mon.noname-a con 0x7f005c137d50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:32:54.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.112+0000 7f0061840640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c138130 0x7f005c136e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:54.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.112+0000 7f0061840640 10 monclient(hunting): picked mon.noname-c con 0x7f005c138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:54.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.112+0000 7f0061840640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f005c13d280 0x7f005c09e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:54.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.112+0000 7f0061840640 10 monclient(hunting): picked mon.noname-b con 0x7f005c13d280 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:32:54.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.112+0000 7f0061840640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:54.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.112+0000 7f0061840640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f005c09d7d0 con 0x7f005c137d50 2024-09-11T01:32:54.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.112+0000 7f0061840640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:54.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.112+0000 7f0061840640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f005c09d650 con 0x7f005c13d280 2024-09-11T01:32:54.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.112+0000 7f0061840640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:54.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.112+0000 7f0061840640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f005c13fbb0 con 0x7f005c138130 2024-09-11T01:32:54.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.112+0000 7f0061840640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:54.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.112+0000 7f0061840640 10 monclient(hunting): authenticate will time out at 980.554866s 2024-09-11T01:32:54.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.112+0000 7f005affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f005c137d50 0x7f005c1ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:54.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.112+0000 7f005a7fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c138130 0x7f005c136e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:54.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005a7fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c138130 0x7f005c136e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:39452/0 (socket says 172.21.15.12:39452) 2024-09-11T01:32:54.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f005c137d50 0x7f005c1ccd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42898/0 (socket says 172.21.15.12:42898) 2024-09-11T01:32:54.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005a7fc640 1 -- 172.21.15.12:0/3679780253 learned_addr learned my addr 172.21.15.12:0/3679780253 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:54.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005a7fc640 10 monclient(hunting): get_auth_request con 0x7f005c138130 auth_method 0 2024-09-11T01:32:54.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:54.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:54.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:54.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005affd640 10 monclient(hunting): get_auth_request con 0x7f005c137d50 auth_method 0 2024-09-11T01:32:54.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:54.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:54.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:54.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:54.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:54.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005b7fe640 1 --2- 172.21.15.12:0/3679780253 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f005c13d280 0x7f005c09e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:54.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:54.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:54.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:54.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:54.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005b7fe640 10 monclient(hunting): get_auth_request con 0x7f005c13d280 auth_method 0 2024-09-11T01:32:54.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:54.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:54.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:54.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:54.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:54.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:54.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005affd640 10 monclient(hunting): handle_auth_done global_id 24360 payload 824 2024-09-11T01:32:54.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005affd640 1 -- 172.21.15.12:0/3679780253 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c138130 msgr2=0x7f005c136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:54.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005affd640 1 --2- 172.21.15.12:0/3679780253 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c138130 0x7f005c136e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:54.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005affd640 1 -- 172.21.15.12:0/3679780253 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f005c13d280 msgr2=0x7f005c09e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:54.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005affd640 1 --2- 172.21.15.12:0/3679780253 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f005c13d280 0x7f005c09e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:54.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005affd640 10 monclient: _finish_hunting 0 2024-09-11T01:32:54.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005affd640 1 monclient: found mon.noname-a 2024-09-11T01:32:54.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:54.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:32:54.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005affd640 1 -- 172.21.15.12:0/3679780253 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f005c120b70 con 0x7f005c137d50 2024-09-11T01:32:54.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.113+0000 7f005a7fc640 1 --2- 172.21.15.12:0/3679780253 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c138130 0x7f005c136e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:32:54.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.115+0000 7f005affd640 1 -- 172.21.15.12:0/3679780253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f005c137d50 msgr2=0x7f005c1ccd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:54.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.115+0000 7f005affd640 1 --2- 172.21.15.12:0/3679780253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f005c137d50 0x7f005c1ccd50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0054007e90 tx=0x7f005400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:54.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.115+0000 7f0053fff640 1 -- 172.21.15.12:0/3679780253 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0054014620 con 0x7f005c137d50 2024-09-11T01:32:54.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.115+0000 7f0053fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:54.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.115+0000 7f0053fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:32:54.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.115+0000 7f0053fff640 10 monclient: dump: 2024-09-11T01:32:54.131 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:54.131 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:54.131 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.131 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.131 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:54.131 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:54.131 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:54.131 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:54.131 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:54.132 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:54.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.115+0000 7f0053fff640 10 monclient: _finish_auth 0 2024-09-11T01:32:54.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.115+0000 7f0053fff640 10 monclient: _check_auth_tickets 2024-09-11T01:32:54.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.115+0000 7f0053fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:54.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.115+0000 7f0053fff640 1 -- 172.21.15.12:0/3679780253 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0054019070 con 0x7f005c137d50 2024-09-11T01:32:54.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.115+0000 7f0061840640 5 monclient: authenticate success, global_id 24360 2024-09-11T01:32:54.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.115+0000 7f0053fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:54.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.115+0000 7f0053fff640 1 -- 172.21.15.12:0/3679780253 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00540149e0 con 0x7f005c137d50 2024-09-11T01:32:54.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.115+0000 7f0053fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:54.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.115+0000 7f0053fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:32:54.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.115+0000 7f0053fff640 10 monclient: dump: 2024-09-11T01:32:54.132 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:54.132 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:54.133 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.133 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.133 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:54.133 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:54.133 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:54.133 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:54.133 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:54.133 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:54.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.115+0000 7f0061840640 10 monclient: _renew_subs 2024-09-11T01:32:54.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.115+0000 7f0061840640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:54.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.115+0000 7f0058ff9640 4 set_mon_vals no callback set 2024-09-11T01:32:54.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.115+0000 7f0061840640 1 -- 172.21.15.12:0/3679780253 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f005c0a1710 con 0x7f005c137d50 2024-09-11T01:32:54.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.115+0000 7f0061840640 10 monclient: _renew_subs 2024-09-11T01:32:54.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.115+0000 7f0061840640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:32:54.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.115+0000 7f0061840640 1 -- 172.21.15.12:0/3679780253 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f005c1217f0 con 0x7f005c137d50 2024-09-11T01:32:54.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.116+0000 7f0061840640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:54.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.116+0000 7f0061840640 1 --2- 172.21.15.12:0/3679780253 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c1373a0 0x7f005c09e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:54.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.116+0000 7f0061840640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:54.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.116+0000 7f0061840640 1 -- 172.21.15.12:0/3679780253 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f005c09d110 con 0x7f005c1373a0 2024-09-11T01:32:54.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.116+0000 7f005a7fc640 1 --2- 172.21.15.12:0/3679780253 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c1373a0 0x7f005c09e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:54.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.116+0000 7f005a7fc640 10 monclient: get_auth_request con 0x7f005c1373a0 auth_method 0 2024-09-11T01:32:54.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.116+0000 7f005a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:54.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.116+0000 7f005a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:54.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.116+0000 7f005a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:54.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.116+0000 7f005a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:54.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.116+0000 7f005a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:54.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.116+0000 7f005a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:54.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.117+0000 7f0053fff640 1 -- 172.21.15.12:0/3679780253 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f0054014b80 con 0x7f005c137d50 2024-09-11T01:32:54.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.117+0000 7f005a7fc640 10 monclient(hunting): handle_auth_done global_id 44135 payload 824 2024-09-11T01:32:54.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.117+0000 7f005a7fc640 1 -- 172.21.15.12:0/3679780253 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f005c137b00 con 0x7f005c1373a0 2024-09-11T01:32:54.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.117+0000 7f0053fff640 1 --2- 172.21.15.12:0/3679780253 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f003c03e210 0x7f003c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:54.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.117+0000 7f0053fff640 1 -- 172.21.15.12:0/3679780253 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 4755+0+0 (secure 0 0 0) 0x7f0054053510 con 0x7f005c137d50 2024-09-11T01:32:54.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.117+0000 7f005b7fe640 1 --2- 172.21.15.12:0/3679780253 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f003c03e210 0x7f003c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:54.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.117+0000 7f005b7fe640 10 monclient: get_auth_request con 0x7f003c03e210 auth_method 0 2024-09-11T01:32:54.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.119+0000 7f005a7fc640 1 -- 172.21.15.12:0/3679780253 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c1373a0 msgr2=0x7f005c09e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:54.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.119+0000 7f005a7fc640 1 --2- 172.21.15.12:0/3679780253 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c1373a0 0x7f005c09e5c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f005c09de30 tx=0x7f0044007770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:54.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.119+0000 7f0053fff640 1 -- 172.21.15.12:0/3679780253 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004400fd30 con 0x7f005c1373a0 2024-09-11T01:32:54.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.119+0000 7f0053fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:54.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.119+0000 7f0053fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:32:54.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.119+0000 7f0053fff640 10 monclient: dump: 2024-09-11T01:32:54.136 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:54.136 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:54.136 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.136 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.136 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:54.136 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:54.136 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:54.136 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:54.136 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:54.136 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:54.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.119+0000 7f0053fff640 1 -- 172.21.15.12:0/3679780253 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f004400fd30 con 0x7f005c1373a0 2024-09-11T01:32:54.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.119+0000 7f0053fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:54.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.119+0000 7f0053fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:32:54.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.119+0000 7f0053fff640 1 -- 172.21.15.12:0/3679780253 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c1373a0 msgr2=0x7f005c09e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:54.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.119+0000 7f0053fff640 1 --2- 172.21.15.12:0/3679780253 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c1373a0 0x7f005c09e5c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f005c09de30 tx=0x7f0044007770 comp rx=0 tx=0).stop 2024-09-11T01:32:54.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.119+0000 7f0053fff640 1 -- 172.21.15.12:0/3679780253 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c1373a0 msgr2=0x7f005c09e5c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:32:54.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.119+0000 7f0053fff640 1 --2- 172.21.15.12:0/3679780253 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c1373a0 0x7f005c09e5c0 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f005c09de30 tx=0x7f0044007770 comp rx=0 tx=0).stop 2024-09-11T01:32:54.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.120+0000 7f005b7fe640 1 --2- 172.21.15.12:0/3679780253 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f003c03e210 0x7f003c0406d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f005c06fce0 tx=0x7f004c02e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:54.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.125+0000 7f0061840640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:32:54.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.125+0000 7f0061840640 1 --2- 172.21.15.12:0/3679780253 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c09eb00 0x7f005c139260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:54.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.125+0000 7f0061840640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:54.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.125+0000 7f0061840640 1 -- 172.21.15.12:0/3679780253 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f005c09ef10 con 0x7f005c09eb00 2024-09-11T01:32:54.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.126+0000 7f005a7fc640 1 --2- 172.21.15.12:0/3679780253 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c09eb00 0x7f005c139260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:54.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.126+0000 7f005a7fc640 10 monclient: get_auth_request con 0x7f005c09eb00 auth_method 0 2024-09-11T01:32:54.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.126+0000 7f005a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:54.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.126+0000 7f005a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:54.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.126+0000 7f005a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:54.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.126+0000 7f005a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:54.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.126+0000 7f005a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:54.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.126+0000 7f005a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:54.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.126+0000 7f005a7fc640 10 monclient(hunting): handle_auth_done global_id 44138 payload 824 2024-09-11T01:32:54.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.126+0000 7f005a7fc640 1 -- 172.21.15.12:0/3679780253 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f005c074a40 con 0x7f005c09eb00 2024-09-11T01:32:54.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.128+0000 7f005a7fc640 1 -- 172.21.15.12:0/3679780253 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c09eb00 msgr2=0x7f005c139260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:54.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.128+0000 7f005a7fc640 1 --2- 172.21.15.12:0/3679780253 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c09eb00 0x7f005c139260 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0044017000 tx=0x7f0044004940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:54.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.128+0000 7f0053fff640 1 -- 172.21.15.12:0/3679780253 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0044006030 con 0x7f005c09eb00 2024-09-11T01:32:54.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.128+0000 7f0053fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:54.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.128+0000 7f0053fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:32:54.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.128+0000 7f0053fff640 10 monclient: dump: 2024-09-11T01:32:54.141 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:54.141 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:54.141 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.141 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.141 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:54.141 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:54.141 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:54.141 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:54.142 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:54.142 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:54.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.129+0000 7f0053fff640 1 -- 172.21.15.12:0/3679780253 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4540 (secure 0 0 0) 0x7f0044006030 con 0x7f005c09eb00 2024-09-11T01:32:54.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.129+0000 7f0053fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:32:54.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.129+0000 7f0053fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:32:54.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.129+0000 7f0053fff640 1 -- 172.21.15.12:0/3679780253 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c09eb00 msgr2=0x7f005c139260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:54.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.129+0000 7f0053fff640 1 --2- 172.21.15.12:0/3679780253 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c09eb00 0x7f005c139260 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0044017000 tx=0x7f0044004940 comp rx=0 tx=0).stop 2024-09-11T01:32:54.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.129+0000 7f0053fff640 1 -- 172.21.15.12:0/3679780253 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c09eb00 msgr2=0x7f005c139260 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:32:54.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.129+0000 7f0053fff640 1 --2- 172.21.15.12:0/3679780253 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f005c09eb00 0x7f005c139260 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0044017000 tx=0x7f0044004940 comp rx=0 tx=0).stop 2024-09-11T01:32:54.143 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:32:54.143 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:32:54.143 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:32:54.143 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:32:54.143 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 38, 2024-09-11T01:32:54.143 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 18072, 2024-09-11T01:32:54.143 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:32:54.143 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:32:54.143 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:32:54.143 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:32:54.143 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:54.143 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 6, 2024-09-11T01:32:54.143 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:32:54.144 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:32:54.144 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:32:54.144 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:54.144 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:54.144 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:54.144 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:54.144 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:54.144 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:54.144 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:54.144 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:54.144 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:54.144 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:54.144 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:54.145 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:54.145 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:32:54.145 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:32:54.145 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:54.145 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:54.145 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:54.145 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:54.145 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:54.145 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:54.145 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:54.145 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:54.145 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:54.145 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:54.145 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:54.146 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:54.146 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:54.146 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:32:54.146 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:32:54.146 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:32:54.146 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:32:54.146 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:32:54.146 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:32:54.146 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:32:54.146 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:32:54.146 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:32:54.146 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:32:54.146 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:32:54.146 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:32:54.147 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:32:54.147 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:32:54.147 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:32:54.147 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:32:54.147 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:32:54.147 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:54.147 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:54.147 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:54.147 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:54.147 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:54.147 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:54.147 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:54.147 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:54.147 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:54.148 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:54.148 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:54.148 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:54.148 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:32:54.148 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:54.148 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:32:54.148 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:54.148 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:32:54.148 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:32:54.148 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:54.148 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:54.148 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:54.148 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:54.148 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:32:54.149 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:54.149 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:54.149 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:54.149 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:54.149 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:32:54.149 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:54.149 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:54.149 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:54.149 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:54.149 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:32:54.149 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:32:54.149 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:54.149 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:54.150 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:54.150 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:54.150 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:54.150 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:32:54.150 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:32:54.150 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:54.150 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:54.150 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:54.150 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:54.150 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:32:54.150 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:54.150 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:54.150 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:54.150 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:54.151 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:32:54.151 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:54.151 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:54.151 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:54.151 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:54.151 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:32:54.151 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:32:54.151 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:54.151 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:54.151 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:54.151 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:54.151 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:54.151 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:32:54.151 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:32:54.152 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:54.152 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:54.152 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:54.152 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:54.152 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:32:54.152 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:54.152 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:54.152 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:54.152 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:54.152 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:32:54.152 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:54.152 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:54.152 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:54.152 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:54.153 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:32:54.153 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:32:54.153 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:54.153 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:54.153 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:54.153 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:54.153 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:54.153 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:54.153 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:32:54.153 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:32:54.153 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:54.153 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:54.153 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:54.153 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:32:54.154 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:54.154 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:54.154 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:32:54.154 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:54.154 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:54.154 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:54.154 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:32:54.154 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:54.154 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:54.154 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:54.154 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:32:54.154 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:32:54.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.129+0000 7f0061840640 1 -- 172.21.15.12:0/3679780253 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f003c03e210 msgr2=0x7f003c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:54.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.129+0000 7f0061840640 1 --2- 172.21.15.12:0/3679780253 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f003c03e210 0x7f003c0406d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f005c06fce0 tx=0x7f004c02e000 comp rx=0 tx=0).stop 2024-09-11T01:32:54.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.129+0000 7f0061840640 10 monclient: shutdown 2024-09-11T01:32:54.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.129+0000 7f0061840640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:54.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.129+0000 7f0061840640 1 -- 172.21.15.12:0/3679780253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f005c137d50 msgr2=0x7f005c1ccd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:54.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.129+0000 7f0061840640 1 --2- 172.21.15.12:0/3679780253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f005c137d50 0x7f005c1ccd50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0054007e90 tx=0x7f005400e4b0 comp rx=0 tx=0).stop 2024-09-11T01:32:54.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.129+0000 7f005b7fe640 1 -- 172.21.15.12:0/3679780253 reap_dead start 2024-09-11T01:32:54.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.129+0000 7f005b7fe640 1 -- 172.21.15.12:0/3679780253 reap_dead start 2024-09-11T01:32:54.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.129+0000 7f0061840640 1 -- 172.21.15.12:0/3679780253 shutdown_connections 2024-09-11T01:32:54.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.129+0000 7f0061840640 1 -- 172.21.15.12:0/3679780253 >> 172.21.15.12:0/3679780253 conn(0x7f005c0a7020 msgr2=0x7f005c13e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:54.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.129+0000 7f0061840640 1 -- 172.21.15.12:0/3679780253 shutdown_connections 2024-09-11T01:32:54.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.129+0000 7f0061840640 1 -- 172.21.15.12:0/3679780253 wait complete. 2024-09-11T01:32:54.157 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:32:54.157 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:32:54.157 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:32:54.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.257+0000 7f6a6de47640 10 monclient: get_monmap_and_config 2024-09-11T01:32:54.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.257+0000 7f6a6de47640 10 monclient: build_initial_monmap 2024-09-11T01:32:54.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.257+0000 7f6a6de47640 10 monclient: monmap: 2024-09-11T01:32:54.259 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:54.259 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:54.259 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:32:54.258223+0000 2024-09-11T01:32:54.259 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:32:54.258223+0000 2024-09-11T01:32:54.259 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:54.259 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:54.259 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:32:54.259 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:32:54.260 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:32:54.260 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:32:54.260 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:32:54.260 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:54.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.258+0000 7f6a6de47640 1 Processor -- start 2024-09-11T01:32:54.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.258+0000 7f6a6de47640 1 -- start start 2024-09-11T01:32:54.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.258+0000 7f6a6de47640 10 monclient: init 2024-09-11T01:32:54.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.258+0000 7f6a6de47640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:54.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.258+0000 7f6a6de47640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-11T01:32:54.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.258+0000 7f6a6de47640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f6a680a2e20 0x7f6a6806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:54.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.258+0000 7f6a6de47640 10 monclient(hunting): picked mon.noname-c con 0x7f6a680a2e20 addr v2:172.21.15.71:3300/0 2024-09-11T01:32:54.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.258+0000 7f6a6de47640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6a680a4970 addr v1:172.21.15.12:6789/0 2024-09-11T01:32:54.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.258+0000 7f6a6de47640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6a6806f910 addr v1:172.21.15.71:6789/0 2024-09-11T01:32:54.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.258+0000 7f6a6de47640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a6806f4f0 con 0x7f6a680a4970 2024-09-11T01:32:54.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.258+0000 7f6a6de47640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a680a3910 con 0x7f6a6806f910 2024-09-11T01:32:54.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.258+0000 7f6a6de47640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:54.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.258+0000 7f6a6de47640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f6a6806ef30 con 0x7f6a680a2e20 2024-09-11T01:32:54.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.258+0000 7f6a6de47640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:54.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.258+0000 7f6a6de47640 10 monclient(hunting): authenticate will time out at 980.700934s 2024-09-11T01:32:54.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.259+0000 7f6a5ffff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6a680a4970 0x7f6a680a3200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:39834/0 (socket says 172.21.15.12:39834) 2024-09-11T01:32:54.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.259+0000 7f6a5ffff640 1 -- 172.21.15.12:0/1351793786 learned_addr learned my addr 172.21.15.12:0/1351793786 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:32:54.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.259+0000 7f6a677fe640 1 --2- 172.21.15.12:0/1351793786 >> v2:172.21.15.71:3300/0 conn(0x7f6a680a2e20 0x7f6a6806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:54.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.259+0000 7f6a677fe640 10 monclient(hunting): get_auth_request con 0x7f6a680a2e20 auth_method 0 2024-09-11T01:32:54.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.259+0000 7f6a677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:54.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.259+0000 7f6a677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:54.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.259+0000 7f6a677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:54.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.261+0000 7f6a5ffff640 1 -- 172.21.15.12:0/1351793786 >> v1:172.21.15.12:6789/0 conn(0x7f6a680a4970 legacy=0x7f6a680a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:54.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.261+0000 7f6a67fff640 1 -- 172.21.15.12:0/1351793786 >> v1:172.21.15.71:6789/0 conn(0x7f6a6806f910 legacy=0x7f6a6813f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:54.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.261+0000 7f6a677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:54.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.261+0000 7f6a677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:54.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.261+0000 7f6a677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:54.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.261+0000 7f6a66ffd640 1 -- 172.21.15.12:0/1351793786 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2347860080 0 0) 0x7f6a6806f4f0 con 0x7f6a680a4970 2024-09-11T01:32:54.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.261+0000 7f6a66ffd640 10 monclient(hunting): _negotiate 2024-09-11T01:32:54.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.261+0000 7f6a66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:54.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.261+0000 7f6a66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:54.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.261+0000 7f6a66ffd640 10 monclient(hunting): my global_id is 24363 2024-09-11T01:32:54.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.261+0000 7f6a66ffd640 1 -- 172.21.15.12:0/1351793786 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a54003520 con 0x7f6a680a4970 2024-09-11T01:32:54.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.261+0000 7f6a677fe640 10 monclient(hunting): handle_auth_done global_id 34219 payload 274 2024-09-11T01:32:54.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.261+0000 7f6a677fe640 1 -- 172.21.15.12:0/1351793786 >> v1:172.21.15.71:6789/0 conn(0x7f6a6806f910 legacy=0x7f6a6813f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:54.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.261+0000 7f6a677fe640 1 -- 172.21.15.12:0/1351793786 >> v1:172.21.15.12:6789/0 conn(0x7f6a680a4970 legacy=0x7f6a680a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:54.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.261+0000 7f6a66ffd640 1 -- 172.21.15.12:0/1351793786 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2063889299 0 0) 0x7f6a680a3910 con 0x7f6a6806f910 2024-09-11T01:32:54.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.261+0000 7f6a677fe640 10 monclient: _finish_hunting 0 2024-09-11T01:32:54.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.261+0000 7f6a677fe640 1 monclient: found mon.noname-c 2024-09-11T01:32:54.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.261+0000 7f6a677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:54.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.261+0000 7f6a677fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:32:54.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.261+0000 7f6a677fe640 1 -- 172.21.15.12:0/1351793786 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a680a3ae0 con 0x7f6a680a2e20 2024-09-11T01:32:54.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.261+0000 7f6a66ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T01:32:54.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.264+0000 7f6a677fe640 1 -- 172.21.15.12:0/1351793786 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a680a2e20 msgr2=0x7f6a6806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:54.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.264+0000 7f6a677fe640 1 --2- 172.21.15.12:0/1351793786 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a680a2e20 0x7f6a6806ded0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6a580098a0 tx=0x7f6a5802a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3130efc4430f37d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:54.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.264+0000 7f6a66ffd640 1 -- 172.21.15.12:0/1351793786 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a58037070 con 0x7f6a680a2e20 2024-09-11T01:32:54.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.264+0000 7f6a66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:54.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.264+0000 7f6a66ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:32:54.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.264+0000 7f6a66ffd640 10 monclient: dump: 2024-09-11T01:32:54.270 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:54.270 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:54.270 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.270 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:54.271 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.264+0000 7f6a66ffd640 10 monclient: _finish_auth 0 2024-09-11T01:32:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.264+0000 7f6a66ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:32:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.264+0000 7f6a66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.264+0000 7f6a6de47640 5 monclient: authenticate success, global_id 34219 2024-09-11T01:32:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.264+0000 7f6a6de47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:32:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.264+0000 7f6a66ffd640 1 -- 172.21.15.12:0/1351793786 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a58032640 con 0x7f6a680a2e20 2024-09-11T01:32:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.264+0000 7f6a66ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.264+0000 7f6a66ffd640 1 -- 172.21.15.12:0/1351793786 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a580329f0 con 0x7f6a680a2e20 2024-09-11T01:32:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.264+0000 7f6a6de47640 10 monclient: get_monmap_and_config success 2024-09-11T01:32:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.264+0000 7f6a6de47640 4 set_mon_vals no callback set 2024-09-11T01:32:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.264+0000 7f6a66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.264+0000 7f6a66ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:32:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.264+0000 7f6a66ffd640 10 monclient: dump: 2024-09-11T01:32:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:54.273 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:54.273 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:54.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.265+0000 7f6a6de47640 10 monclient: shutdown 2024-09-11T01:32:54.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.265+0000 7f6a6de47640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:54.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.265+0000 7f6a6de47640 1 -- 172.21.15.12:0/1351793786 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a680a2e20 msgr2=0x7f6a6806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:54.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.265+0000 7f6a6de47640 1 --2- 172.21.15.12:0/1351793786 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a680a2e20 0x7f6a6806ded0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6a580098a0 tx=0x7f6a5802a210 comp rx=0 tx=0).stop 2024-09-11T01:32:54.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.265+0000 7f6a6de47640 1 -- 172.21.15.12:0/1351793786 shutdown_connections 2024-09-11T01:32:54.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.265+0000 7f6a6de47640 1 --2- 172.21.15.12:0/1351793786 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a680a2e20 0x7f6a6806ded0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:54.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.265+0000 7f6a6de47640 1 -- 172.21.15.12:0/1351793786 >> 172.21.15.12:0/1351793786 conn(0x7f6a680a8600 msgr2=0x7f6a680a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:54.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.265+0000 7f6a6de47640 1 -- 172.21.15.12:0/1351793786 shutdown_connections 2024-09-11T01:32:54.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.265+0000 7f6a6de47640 1 -- 172.21.15.12:0/1351793786 wait complete. 2024-09-11T01:32:54.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.265+0000 7f6a6de47640 10 monclient: build_initial_monmap 2024-09-11T01:32:54.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.265+0000 7f6a6de47640 10 monclient: monmap: 2024-09-11T01:32:54.273 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:32:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:32:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:32:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:32:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:32:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:32:54.274 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a6de47640 1 Processor -- start 2024-09-11T01:32:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a6de47640 1 -- start start 2024-09-11T01:32:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a6de47640 10 monclient: init 2024-09-11T01:32:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a6de47640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a6de47640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:32:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a6de47640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a6806f910 0x7f6a6813f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a6de47640 10 monclient(hunting): picked mon.noname-b con 0x7f6a6806f910 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:32:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a6de47640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6a680a2e20 0x7f6a6813b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a6de47640 10 monclient(hunting): picked mon.noname-c con 0x7f6a680a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a6de47640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a680a4970 0x7f6a6813b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a6de47640 10 monclient(hunting): picked mon.noname-a con 0x7f6a680a4970 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:32:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a6de47640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a6de47640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6a68140d60 con 0x7f6a680a4970 2024-09-11T01:32:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a6de47640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a6de47640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6806ec50 con 0x7f6a6806f910 2024-09-11T01:32:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a6de47640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a6de47640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6a68140ee0 con 0x7f6a680a2e20 2024-09-11T01:32:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a6de47640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a6de47640 10 monclient(hunting): authenticate will time out at 980.708704s 2024-09-11T01:32:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a5ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6a680a2e20 0x7f6a6813b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a67fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a680a4970 0x7f6a6813b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a5ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6a680a2e20 0x7f6a6813b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:39506/0 (socket says 172.21.15.12:39506) 2024-09-11T01:32:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a5ffff640 1 -- 172.21.15.12:0/3651931571 learned_addr learned my addr 172.21.15.12:0/3651931571 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a5ffff640 10 monclient(hunting): get_auth_request con 0x7f6a680a2e20 auth_method 0 2024-09-11T01:32:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a67fff640 10 monclient(hunting): get_auth_request con 0x7f6a680a4970 auth_method 0 2024-09-11T01:32:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a67fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a677fe640 1 --2- 172.21.15.12:0/3651931571 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a6806f910 0x7f6a6813f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a677fe640 10 monclient(hunting): get_auth_request con 0x7f6a6806f910 auth_method 0 2024-09-11T01:32:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.266+0000 7f6a67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.267+0000 7f6a67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.267+0000 7f6a677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.267+0000 7f6a677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.267+0000 7f6a677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.267+0000 7f6a5ffff640 10 monclient(hunting): handle_auth_done global_id 44141 payload 824 2024-09-11T01:32:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.267+0000 7f6a5ffff640 1 -- 172.21.15.12:0/3651931571 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a6806f910 msgr2=0x7f6a6813f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:54.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.267+0000 7f6a5ffff640 1 --2- 172.21.15.12:0/3651931571 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a6806f910 0x7f6a6813f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:54.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.267+0000 7f6a5ffff640 1 -- 172.21.15.12:0/3651931571 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a680a4970 msgr2=0x7f6a6813b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:54.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.267+0000 7f6a5ffff640 1 --2- 172.21.15.12:0/3651931571 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a680a4970 0x7f6a6813b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:54.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.267+0000 7f6a5ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:32:54.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.267+0000 7f6a5ffff640 1 monclient: found mon.noname-c 2024-09-11T01:32:54.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.267+0000 7f6a5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:54.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.267+0000 7f6a5ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:32:54.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.267+0000 7f6a5ffff640 1 -- 172.21.15.12:0/3651931571 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a6813cc30 con 0x7f6a680a2e20 2024-09-11T01:32:54.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.267+0000 7f6a677fe640 1 --2- 172.21.15.12:0/3651931571 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a6806f910 0x7f6a6813f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:32:54.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.267+0000 7f6a67fff640 1 --2- 172.21.15.12:0/3651931571 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6a680a4970 0x7f6a6813b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:32:54.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.269+0000 7f6a5ffff640 1 -- 172.21.15.12:0/3651931571 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6a680a2e20 msgr2=0x7f6a6813b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:54.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.269+0000 7f6a5ffff640 1 --2- 172.21.15.12:0/3651931571 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6a680a2e20 0x7f6a6813b1b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6a5000c700 tx=0x7f6a5000cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:54.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.269+0000 7f6a64ff9640 1 -- 172.21.15.12:0/3651931571 DispatchQueue::entry inject delay of 0.002000 2024-09-11T01:32:54.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.271+0000 7f6a64ff9640 1 -- 172.21.15.12:0/3651931571 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a50017070 con 0x7f6a680a2e20 2024-09-11T01:32:54.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.271+0000 7f6a64ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:54.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.271+0000 7f6a64ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:32:54.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.271+0000 7f6a64ff9640 10 monclient: dump: 2024-09-11T01:32:54.281 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:54.281 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:54.281 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.282 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.282 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:54.282 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:54.282 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:54.282 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:54.282 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:54.282 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:54.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.271+0000 7f6a64ff9640 10 monclient: _finish_auth 0 2024-09-11T01:32:54.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.271+0000 7f6a64ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:32:54.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.271+0000 7f6a64ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:54.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.271+0000 7f6a64ff9640 1 -- 172.21.15.12:0/3651931571 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a50005d00 con 0x7f6a680a2e20 2024-09-11T01:32:54.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.271+0000 7f6a64ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:54.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.271+0000 7f6a64ff9640 1 -- 172.21.15.12:0/3651931571 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a500072b0 con 0x7f6a680a2e20 2024-09-11T01:32:54.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.271+0000 7f6a64ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:54.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.271+0000 7f6a64ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:32:54.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.271+0000 7f6a64ff9640 10 monclient: dump: 2024-09-11T01:32:54.283 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:54.283 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:54.283 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.283 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.283 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:54.283 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:54.283 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:54.283 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:54.283 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:54.283 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:54.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.271+0000 7f6a6de47640 5 monclient: authenticate success, global_id 44141 2024-09-11T01:32:54.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.271+0000 7f6a6de47640 10 monclient: _renew_subs 2024-09-11T01:32:54.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.271+0000 7f6a6de47640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:32:54.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.271+0000 7f6a6de47640 1 -- 172.21.15.12:0/3651931571 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a6809f610 con 0x7f6a680a2e20 2024-09-11T01:32:54.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.271+0000 7f6a667fc640 4 set_mon_vals no callback set 2024-09-11T01:32:54.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.271+0000 7f6a6de47640 10 monclient: _renew_subs 2024-09-11T01:32:54.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.271+0000 7f6a6de47640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:32:54.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.271+0000 7f6a6de47640 1 -- 172.21.15.12:0/3651931571 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a6813d790 con 0x7f6a680a2e20 2024-09-11T01:32:54.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.272+0000 7f6a6de47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:54.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.272+0000 7f6a6de47640 1 --2- 172.21.15.12:0/3651931571 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a6813bc30 0x7f6a681380f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:54.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.272+0000 7f6a6de47640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:54.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.272+0000 7f6a6de47640 1 -- 172.21.15.12:0/3651931571 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6806a060 con 0x7f6a6813bc30 2024-09-11T01:32:54.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.272+0000 7f6a64ff9640 1 -- 172.21.15.12:0/3651931571 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f6a500074d0 con 0x7f6a680a2e20 2024-09-11T01:32:54.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.272+0000 7f6a677fe640 1 --2- 172.21.15.12:0/3651931571 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a6813bc30 0x7f6a681380f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:54.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.272+0000 7f6a677fe640 10 monclient: get_auth_request con 0x7f6a6813bc30 auth_method 0 2024-09-11T01:32:54.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.272+0000 7f6a677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:54.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.272+0000 7f6a677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:54.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.272+0000 7f6a677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:54.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.273+0000 7f6a64ff9640 1 --2- 172.21.15.12:0/3651931571 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6a4803e260 0x7f6a48040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:54.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.273+0000 7f6a64ff9640 1 -- 172.21.15.12:0/3651931571 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(139..139 src has 1..139) ==== 4755+0+0 (secure 0 0 0) 0x7f6a50052390 con 0x7f6a680a2e20 2024-09-11T01:32:54.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.273+0000 7f6a677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:54.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.273+0000 7f6a677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:54.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.273+0000 7f6a677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:54.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.273+0000 7f6a67fff640 1 --2- 172.21.15.12:0/3651931571 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6a4803e260 0x7f6a48040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:54.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.273+0000 7f6a67fff640 10 monclient: get_auth_request con 0x7f6a4803e260 auth_method 0 2024-09-11T01:32:54.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.273+0000 7f6a677fe640 10 monclient(hunting): handle_auth_done global_id 34228 payload 824 2024-09-11T01:32:54.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.273+0000 7f6a677fe640 1 -- 172.21.15.12:0/3651931571 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6a68074a80 con 0x7f6a6813bc30 2024-09-11T01:32:54.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.275+0000 7f6a677fe640 1 -- 172.21.15.12:0/3651931571 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a6813bc30 msgr2=0x7f6a681380f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:54.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.275+0000 7f6a677fe640 1 --2- 172.21.15.12:0/3651931571 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a6813bc30 0x7f6a681380f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6a58034000 tx=0x7f6a580046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:54.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.275+0000 7f6a67fff640 1 --2- 172.21.15.12:0/3651931571 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6a4803e260 0x7f6a48040720 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6a600012e0 tx=0x7f6a60009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:54.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.275+0000 7f6a64ff9640 1 -- 172.21.15.12:0/3651931571 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a58042070 con 0x7f6a6813bc30 2024-09-11T01:32:54.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.275+0000 7f6a64ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:54.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.275+0000 7f6a64ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:32:54.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.275+0000 7f6a64ff9640 10 monclient: dump: 2024-09-11T01:32:54.288 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:54.288 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:54.288 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.288 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.288 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:54.288 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:54.288 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:54.288 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:54.288 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:54.288 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:54.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.276+0000 7f6a64ff9640 1 -- 172.21.15.12:0/3651931571 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6a580044d0 con 0x7f6a6813bc30 2024-09-11T01:32:54.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.276+0000 7f6a64ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:54.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.276+0000 7f6a64ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:32:54.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.276+0000 7f6a64ff9640 1 -- 172.21.15.12:0/3651931571 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a6813bc30 msgr2=0x7f6a681380f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:54.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.276+0000 7f6a64ff9640 1 --2- 172.21.15.12:0/3651931571 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a6813bc30 0x7f6a681380f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6a58034000 tx=0x7f6a580046c0 comp rx=0 tx=0).stop 2024-09-11T01:32:54.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.276+0000 7f6a64ff9640 1 -- 172.21.15.12:0/3651931571 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a6813bc30 msgr2=0x7f6a681380f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:32:54.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.276+0000 7f6a64ff9640 1 --2- 172.21.15.12:0/3651931571 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a6813bc30 0x7f6a681380f0 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6a58034000 tx=0x7f6a580046c0 comp rx=0 tx=0).stop 2024-09-11T01:32:54.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.282+0000 7f6a6de47640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:32:54.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.282+0000 7f6a6de47640 1 --2- 172.21.15.12:0/3651931571 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a68138630 0x7f6a68138a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:54.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.282+0000 7f6a6de47640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:54.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.282+0000 7f6a6de47640 1 -- 172.21.15.12:0/3651931571 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6a68138f80 con 0x7f6a68138630 2024-09-11T01:32:54.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.282+0000 7f6a677fe640 1 --2- 172.21.15.12:0/3651931571 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a68138630 0x7f6a68138a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:54.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.282+0000 7f6a677fe640 10 monclient: get_auth_request con 0x7f6a68138630 auth_method 0 2024-09-11T01:32:54.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.282+0000 7f6a677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:54.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.282+0000 7f6a677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:54.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.282+0000 7f6a677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:54.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.283+0000 7f6a677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:54.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.283+0000 7f6a677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:54.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.283+0000 7f6a677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:54.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.283+0000 7f6a677fe640 10 monclient(hunting): handle_auth_done global_id 34231 payload 824 2024-09-11T01:32:54.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.283+0000 7f6a677fe640 1 -- 172.21.15.12:0/3651931571 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6a6813c010 con 0x7f6a68138630 2024-09-11T01:32:54.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.285+0000 7f6a677fe640 1 -- 172.21.15.12:0/3651931571 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a68138630 msgr2=0x7f6a68138a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:54.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.285+0000 7f6a677fe640 1 --2- 172.21.15.12:0/3651931571 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a68138630 0x7f6a68138a40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6a58000c00 tx=0x7f6a58009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:54.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.285+0000 7f6a64ff9640 1 -- 172.21.15.12:0/3651931571 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a58037040 con 0x7f6a68138630 2024-09-11T01:32:54.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.285+0000 7f6a64ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:54.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.285+0000 7f6a64ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:32:54.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.285+0000 7f6a64ff9640 10 monclient: dump: 2024-09-11T01:32:54.292 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:54.292 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:54.292 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.292 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.292 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:54.292 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:54.292 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:54.292 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:54.293 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:54.293 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:54.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.286+0000 7f6a64ff9640 1 -- 172.21.15.12:0/3651931571 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4707 (secure 0 0 0) 0x7f6a58032440 con 0x7f6a68138630 2024-09-11T01:32:54.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.286+0000 7f6a64ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:32:54.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.286+0000 7f6a64ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:32:54.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.286+0000 7f6a64ff9640 1 -- 172.21.15.12:0/3651931571 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a68138630 msgr2=0x7f6a68138a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:54.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.286+0000 7f6a64ff9640 1 --2- 172.21.15.12:0/3651931571 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a68138630 0x7f6a68138a40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6a58000c00 tx=0x7f6a58009190 comp rx=0 tx=0).stop 2024-09-11T01:32:54.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.286+0000 7f6a64ff9640 1 -- 172.21.15.12:0/3651931571 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a68138630 msgr2=0x7f6a68138a40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:32:54.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.286+0000 7f6a64ff9640 1 --2- 172.21.15.12:0/3651931571 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a68138630 0x7f6a68138a40 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6a58000c00 tx=0x7f6a58009190 comp rx=0 tx=0).stop 2024-09-11T01:32:54.293 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:32:54.293 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:32:54.294 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:32:54.294 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:32:54.294 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 38, 2024-09-11T01:32:54.294 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 67499, 2024-09-11T01:32:54.294 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:32:54.294 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:32:54.294 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:32:54.294 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:32:54.294 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:54.294 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 6, 2024-09-11T01:32:54.294 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:32:54.294 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:32:54.294 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:32:54.295 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:54.295 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:54.295 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:54.295 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:54.295 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:54.295 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:54.295 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:54.295 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:54.295 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:54.295 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:54.295 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:54.295 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:54.295 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:32:54.295 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:32:54.295 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:54.296 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:54.296 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:54.296 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:54.296 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:54.296 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:54.296 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:54.296 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:54.296 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:54.296 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:54.296 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:54.296 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:54.296 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:54.296 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:32:54.296 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:32:54.297 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:32:54.297 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:32:54.297 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:32:54.297 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:32:54.297 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:32:54.297 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:32:54.297 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:32:54.297 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:32:54.297 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:32:54.297 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:32:54.297 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:32:54.297 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:32:54.297 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:32:54.297 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:32:54.298 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:32:54.298 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:32:54.298 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:32:54.298 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:32:54.298 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:32:54.298 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:32:54.298 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:32:54.298 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:32:54.298 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:32:54.298 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:32:54.298 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:32:54.298 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:32:54.298 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:54.298 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:32:54.299 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:54.299 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:32:54.299 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:54.299 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:32:54.299 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:32:54.299 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:54.299 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:54.299 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:54.299 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:54.299 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:32:54.299 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:54.299 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:54.299 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:54.299 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:54.300 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:32:54.300 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:54.300 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:54.300 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:54.300 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:54.300 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:32:54.300 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:32:54.300 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:54.300 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:54.300 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:54.300 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:54.300 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:54.300 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:32:54.300 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:32:54.301 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:54.301 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:54.301 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:54.301 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:54.301 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:32:54.301 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:54.301 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:54.301 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:54.301 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:54.301 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:32:54.301 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:54.301 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:54.301 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:54.301 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:54.302 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:32:54.302 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:32:54.302 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:54.302 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:54.302 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:54.302 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:54.302 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:54.302 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:32:54.302 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:32:54.302 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:32:54.302 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:32:54.302 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:54.302 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:32:54.302 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:32:54.302 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:54.303 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:54.303 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:54.303 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:32:54.303 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:32:54.303 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:32:54.303 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:54.303 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:54.303 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:54.303 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:32:54.303 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:32:54.303 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:32:54.303 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:32:54.303 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:32:54.304 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:54.304 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:54.304 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:54.304 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:32:54.304 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:32:54.304 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:54.304 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:54.304 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:54.304 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:32:54.304 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:54.304 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:54.304 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:32:54.304 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:54.304 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:54.305 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:54.305 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:32:54.305 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:54.305 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:32:54.305 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:32:54.305 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:32:54.305 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:32:54.305 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:32:54.305 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:32:54.305 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:32:54.305 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:32:54.305 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:32:54.305 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:32:54.305 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:32:54.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.286+0000 7f6a6de47640 1 -- 172.21.15.12:0/3651931571 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6a4803e260 msgr2=0x7f6a48040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:54.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.286+0000 7f6a6de47640 1 --2- 172.21.15.12:0/3651931571 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6a4803e260 0x7f6a48040720 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6a600012e0 tx=0x7f6a60009040 comp rx=0 tx=0).stop 2024-09-11T01:32:54.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.286+0000 7f6a6de47640 10 monclient: shutdown 2024-09-11T01:32:54.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.286+0000 7f6a6de47640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:54.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.286+0000 7f6a6de47640 1 -- 172.21.15.12:0/3651931571 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6a680a2e20 msgr2=0x7f6a6813b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:54.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.286+0000 7f6a6de47640 1 --2- 172.21.15.12:0/3651931571 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6a680a2e20 0x7f6a6813b1b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6a5000c700 tx=0x7f6a5000cbc0 comp rx=0 tx=0).stop 2024-09-11T01:32:54.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.286+0000 7f6a67fff640 1 -- 172.21.15.12:0/3651931571 reap_dead start 2024-09-11T01:32:54.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.286+0000 7f6a67fff640 1 -- 172.21.15.12:0/3651931571 reap_dead start 2024-09-11T01:32:54.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.286+0000 7f6a6de47640 1 -- 172.21.15.12:0/3651931571 shutdown_connections 2024-09-11T01:32:54.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.286+0000 7f6a6de47640 1 -- 172.21.15.12:0/3651931571 >> 172.21.15.12:0/3651931571 conn(0x7f6a680a8600 msgr2=0x7f6a6809e330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:54.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.286+0000 7f6a6de47640 1 -- 172.21.15.12:0/3651931571 shutdown_connections 2024-09-11T01:32:54.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.286+0000 7f6a6de47640 1 -- 172.21.15.12:0/3651931571 wait complete. 2024-09-11T01:32:54.308 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:32:54.308 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-11T01:32:54.308 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T01:32:54.308 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-11T01:32:54.308 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-11T01:32:54.309 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T01:32:54.309 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:32:54.320 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T01:32:54.320 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:32:54.320 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:32:54.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.459+0000 7fef89a23640 10 monclient: get_monmap_and_config 2024-09-11T01:32:54.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.459+0000 7fef89a23640 10 monclient: build_initial_monmap 2024-09-11T01:32:54.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.459+0000 7fef89a23640 10 monclient: monmap: 2024-09-11T01:32:54.461 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:54.462 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:54.462 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:32:54.460653+0000 2024-09-11T01:32:54.462 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:32:54.460653+0000 2024-09-11T01:32:54.462 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:54.462 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:54.462 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:32:54.462 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:32:54.462 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:32:54.462 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:32:54.462 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:32:54.462 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:54.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.460+0000 7fef89a23640 1 Processor -- start 2024-09-11T01:32:54.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.460+0000 7fef89a23640 1 -- start start 2024-09-11T01:32:54.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.460+0000 7fef89a23640 10 monclient: init 2024-09-11T01:32:54.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.460+0000 7fef89a23640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:54.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef89a23640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-11T01:32:54.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef89a23640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fef84138230 0x7fef84138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:54.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef89a23640 10 monclient(hunting): picked mon.noname-b con 0x7fef84138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:54.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef89a23640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fef8409d030 0x7fef840a5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:54.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef89a23640 10 monclient(hunting): picked mon.noname-a con 0x7fef8409d030 addr v2:172.21.15.12:3300/0 2024-09-11T01:32:54.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef89a23640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fef8409d430 addr v1:172.21.15.12:6789/0 2024-09-11T01:32:54.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef89a23640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef840a6e30 con 0x7fef8409d430 2024-09-11T01:32:54.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef89a23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:54.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef89a23640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fef8409c9c0 con 0x7fef8409d030 2024-09-11T01:32:54.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef89a23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:54.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef89a23640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fef8409ba20 con 0x7fef84138230 2024-09-11T01:32:54.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef89a23640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:54.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef89a23640 10 monclient(hunting): authenticate will time out at 980.903092s 2024-09-11T01:32:54.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef837fe640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7fef8409d430 legacy=0x7fef840a5560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:32:54.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef827fc640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7fef8409d030 msgr2=0x7fef840a5020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:32:54.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef827fc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fef8409d030 0x7fef840a5020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:54.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef82ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fef84138230 0x7fef84138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:54.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef82ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fef84138230 0x7fef84138610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:39540/0 (socket says 172.21.15.12:39540) 2024-09-11T01:32:54.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef82ffd640 1 -- 172.21.15.12:0/771021614 learned_addr learned my addr 172.21.15.12:0/771021614 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:54.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef82ffd640 10 monclient(hunting): get_auth_request con 0x7fef84138230 auth_method 0 2024-09-11T01:32:54.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:54.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:54.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:54.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef82ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:54.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef82ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:54.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef82ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:54.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef82ffd640 10 monclient(hunting): handle_auth_done global_id 44144 payload 274 2024-09-11T01:32:54.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef82ffd640 1 -- 172.21.15.12:0/771021614 >> v2:172.21.15.12:3300/0 conn(0x7fef8409d030 msgr2=0x7fef840a5020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:32:54.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef82ffd640 1 --2- 172.21.15.12:0/771021614 >> v2:172.21.15.12:3300/0 conn(0x7fef8409d030 0x7fef840a5020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:54.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef82ffd640 1 -- 172.21.15.12:0/771021614 >> v1:172.21.15.12:6789/0 conn(0x7fef8409d430 legacy=0x7fef840a5560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:32:54.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef82ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:32:54.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef82ffd640 1 monclient: found mon.noname-b 2024-09-11T01:32:54.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:54.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef82ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:32:54.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.461+0000 7fef82ffd640 1 -- 172.21.15.12:0/771021614 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef840a66a0 con 0x7fef84138230 2024-09-11T01:32:54.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.464+0000 7fef82ffd640 1 -- 172.21.15.12:0/771021614 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fef84138230 msgr2=0x7fef84138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:54.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.464+0000 7fef82ffd640 1 --2- 172.21.15.12:0/771021614 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fef84138230 0x7fef84138610 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fef8406d650 tx=0x7fef7402a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ad80938ef2e989 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:54.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.464+0000 7fef81ffb640 1 -- 172.21.15.12:0/771021614 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef74036070 con 0x7fef84138230 2024-09-11T01:32:54.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.464+0000 7fef81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:54.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.464+0000 7fef81ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:32:54.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.464+0000 7fef81ffb640 10 monclient: dump: 2024-09-11T01:32:54.472 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:54.472 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:54.473 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.473 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.473 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:54.473 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:54.473 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:54.473 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:54.473 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:54.473 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:54.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.464+0000 7fef81ffb640 10 monclient: _finish_auth 0 2024-09-11T01:32:54.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.464+0000 7fef81ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:32:54.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.464+0000 7fef81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:54.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.464+0000 7fef81ffb640 1 -- 172.21.15.12:0/771021614 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef74007800 con 0x7fef84138230 2024-09-11T01:32:54.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.464+0000 7fef89a23640 5 monclient: authenticate success, global_id 44144 2024-09-11T01:32:54.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.464+0000 7fef89a23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:32:54.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.464+0000 7fef81ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:54.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.464+0000 7fef89a23640 10 monclient: get_monmap_and_config success 2024-09-11T01:32:54.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.464+0000 7fef81ffb640 1 -- 172.21.15.12:0/771021614 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef74007bd0 con 0x7fef84138230 2024-09-11T01:32:54.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.464+0000 7fef89a23640 4 set_mon_vals no callback set 2024-09-11T01:32:54.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.464+0000 7fef81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:54.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.464+0000 7fef81ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:32:54.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.464+0000 7fef81ffb640 10 monclient: dump: 2024-09-11T01:32:54.474 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:54.474 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:54.474 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.474 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.474 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:54.475 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:54.475 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:54.475 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:54.475 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:54.475 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:54.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.464+0000 7fef89a23640 10 monclient: shutdown 2024-09-11T01:32:54.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.464+0000 7fef89a23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:54.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.464+0000 7fef89a23640 1 -- 172.21.15.12:0/771021614 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fef84138230 msgr2=0x7fef84138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:54.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.464+0000 7fef89a23640 1 --2- 172.21.15.12:0/771021614 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fef84138230 0x7fef84138610 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fef8406d650 tx=0x7fef7402a560 comp rx=0 tx=0).stop 2024-09-11T01:32:54.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.465+0000 7fef89a23640 1 -- 172.21.15.12:0/771021614 shutdown_connections 2024-09-11T01:32:54.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.465+0000 7fef89a23640 1 --2- 172.21.15.12:0/771021614 >> v2:172.21.15.12:3300/0 conn(0x7fef8409d030 0x7fef840a5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:54.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.465+0000 7fef89a23640 1 --2- 172.21.15.12:0/771021614 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fef84138230 0x7fef84138610 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:54.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.465+0000 7fef89a23640 1 -- 172.21.15.12:0/771021614 >> 172.21.15.12:0/771021614 conn(0x7fef840a7120 msgr2=0x7fef840a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:54.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.465+0000 7fef89a23640 1 -- 172.21.15.12:0/771021614 shutdown_connections 2024-09-11T01:32:54.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.465+0000 7fef89a23640 1 -- 172.21.15.12:0/771021614 wait complete. 2024-09-11T01:32:54.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.465+0000 7fef89a23640 10 monclient: build_initial_monmap 2024-09-11T01:32:54.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.465+0000 7fef89a23640 10 monclient: monmap: 2024-09-11T01:32:54.476 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:54.476 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:32:54.476 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:32:54.476 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:32:54.476 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:54.476 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:54.476 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:32:54.476 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:32:54.477 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:32:54.477 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:54.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.465+0000 7fef89a23640 1 Processor -- start 2024-09-11T01:32:54.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef89a23640 1 -- start start 2024-09-11T01:32:54.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef89a23640 10 monclient: init 2024-09-11T01:32:54.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef89a23640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:54.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef89a23640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:32:54.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef89a23640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef8409d030 0x7fef841cd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:54.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef89a23640 10 monclient(hunting): picked mon.noname-a con 0x7fef8409d030 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:32:54.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef89a23640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef8409d430 0x7fef84137220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:54.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef89a23640 10 monclient(hunting): picked mon.noname-b con 0x7fef8409d430 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:32:54.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef89a23640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fef84138230 0x7fef84137760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:54.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef89a23640 10 monclient(hunting): picked mon.noname-c con 0x7fef84138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:54.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef89a23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:54.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef89a23640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fef8406f970 con 0x7fef8409d030 2024-09-11T01:32:54.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef89a23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:54.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef89a23640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fef8406f7f0 con 0x7fef8409d430 2024-09-11T01:32:54.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef89a23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:54.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef89a23640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fef840a5c70 con 0x7fef84138230 2024-09-11T01:32:54.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef89a23640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:54.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef82ffd640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef8409d030 msgr2=0x7fef841cd260 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:32:54.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef89a23640 10 monclient(hunting): authenticate will time out at 980.908729s 2024-09-11T01:32:54.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef82ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef8409d030 0x7fef841cd260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:54.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef837fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fef84138230 0x7fef84137760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:54.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef837fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fef84138230 0x7fef84137760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:39546/0 (socket says 172.21.15.12:39546) 2024-09-11T01:32:54.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef837fe640 1 -- 172.21.15.12:0/1595762167 learned_addr learned my addr 172.21.15.12:0/1595762167 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:54.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef837fe640 10 monclient(hunting): get_auth_request con 0x7fef84138230 auth_method 0 2024-09-11T01:32:54.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:54.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef837fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:54.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:54.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:54.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:54.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.466+0000 7fef837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:54.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.467+0000 7fef837fe640 10 monclient(hunting): handle_auth_done global_id 44147 payload 824 2024-09-11T01:32:54.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.467+0000 7fef827fc640 1 --2- 172.21.15.12:0/1595762167 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef8409d430 0x7fef84137220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:54.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.467+0000 7fef837fe640 1 -- 172.21.15.12:0/1595762167 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef8409d430 msgr2=0x7fef84137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:54.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.467+0000 7fef837fe640 1 --2- 172.21.15.12:0/1595762167 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef8409d430 0x7fef84137220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:54.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.467+0000 7fef837fe640 1 -- 172.21.15.12:0/1595762167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef8409d030 msgr2=0x7fef841cd260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:32:54.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.467+0000 7fef837fe640 1 --2- 172.21.15.12:0/1595762167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef8409d030 0x7fef841cd260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:54.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.467+0000 7fef837fe640 10 monclient: _finish_hunting 0 2024-09-11T01:32:54.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.467+0000 7fef837fe640 1 monclient: found mon.noname-c 2024-09-11T01:32:54.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.467+0000 7fef837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:54.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.467+0000 7fef837fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:32:54.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.467+0000 7fef837fe640 1 -- 172.21.15.12:0/1595762167 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef840a1810 con 0x7fef84138230 2024-09-11T01:32:54.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.469+0000 7fef837fe640 1 -- 172.21.15.12:0/1595762167 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fef84138230 msgr2=0x7fef84137760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:54.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.469+0000 7fef837fe640 1 --2- 172.21.15.12:0/1595762167 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fef84138230 0x7fef84137760 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fef7c009fd0 tx=0x7fef7c00b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:54.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.469+0000 7fef7bfff640 1 -- 172.21.15.12:0/1595762167 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c017070 con 0x7fef84138230 2024-09-11T01:32:54.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.469+0000 7fef7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:54.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.469+0000 7fef7bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:32:54.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.469+0000 7fef7bfff640 10 monclient: dump: 2024-09-11T01:32:54.480 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:54.480 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:54.480 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.481 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.481 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:54.481 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:54.481 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:54.481 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:54.481 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:54.481 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:54.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.469+0000 7fef7bfff640 10 monclient: _finish_auth 0 2024-09-11T01:32:54.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.469+0000 7fef7bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:32:54.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.469+0000 7fef7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:54.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.469+0000 7fef7bfff640 1 -- 172.21.15.12:0/1595762167 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef7c0060d0 con 0x7fef84138230 2024-09-11T01:32:54.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.469+0000 7fef89a23640 5 monclient: authenticate success, global_id 44147 2024-09-11T01:32:54.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.469+0000 7fef89a23640 10 monclient: _renew_subs 2024-09-11T01:32:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.469+0000 7fef89a23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:32:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.469+0000 7fef89a23640 1 -- 172.21.15.12:0/1595762167 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef84124620 con 0x7fef84138230 2024-09-11T01:32:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.469+0000 7fef7bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.469+0000 7fef7bfff640 1 -- 172.21.15.12:0/1595762167 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c006480 con 0x7fef84138230 2024-09-11T01:32:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.469+0000 7fef7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.469+0000 7fef7bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:32:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.469+0000 7fef7bfff640 10 monclient: dump: 2024-09-11T01:32:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:54.483 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.469+0000 7fef817fa640 4 set_mon_vals no callback set 2024-09-11T01:32:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.469+0000 7fef89a23640 10 monclient: _renew_subs 2024-09-11T01:32:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.469+0000 7fef89a23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:32:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.469+0000 7fef89a23640 1 -- 172.21.15.12:0/1595762167 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fef840a1cd0 con 0x7fef84138230 2024-09-11T01:32:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.470+0000 7fef89a23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.470+0000 7fef89a23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.470+0000 7fef89a23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:32:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.470+0000 7fef7bfff640 1 -- 172.21.15.12:0/1595762167 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fef7c0066a0 con 0x7fef84138230 2024-09-11T01:32:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.470+0000 7fef89a23640 1 -- 172.21.15.12:0/1595762167 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef84138610 con 0x7fef84138230 2024-09-11T01:32:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.470+0000 7fef7bfff640 1 --2- 172.21.15.12:0/1595762167 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fef6403e1c0 0x7fef64040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.470+0000 7fef7bfff640 1 -- 172.21.15.12:0/1595762167 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(139..139 src has 1..139) ==== 4755+0+0 (secure 0 0 0) 0x7fef7c0523e0 con 0x7fef84138230 2024-09-11T01:32:54.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.470+0000 7fef82ffd640 1 --2- 172.21.15.12:0/1595762167 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fef6403e1c0 0x7fef64040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:54.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.470+0000 7fef82ffd640 10 monclient: get_auth_request con 0x7fef6403e1c0 auth_method 0 2024-09-11T01:32:54.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.473+0000 7fef82ffd640 1 --2- 172.21.15.12:0/1595762167 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fef6403e1c0 0x7fef64040680 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fef74033000 tx=0x7fef74009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:54.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.475+0000 7fef7bfff640 1 -- 172.21.15.12:0/1595762167 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fef840a1810 con 0x7fef84138230 2024-09-11T01:32:54.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.475+0000 7fef7bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:54.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.475+0000 7fef7bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:32:54.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.732+0000 7fef89a23640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:32:54.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.732+0000 7fef89a23640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:32:54.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.732+0000 7fef89a23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:32:54.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.732+0000 7fef89a23640 1 -- 172.21.15.12:0/1595762167 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fef840a15c0 con 0x7fef84138230 2024-09-11T01:32:54.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:54.732+0000 7fef837fe640 1 --2- 172.21.15.12:0/1595762167 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fef84138230 0x7fef84137760 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fef7c009fd0 tx=0x7fef7c00b870 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7fef7c050df0 mon_command_ack([{"prefix": "quorum_status"}]=0 v0) 2024-09-11T01:32:55.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:55.561+0000 7fef7bfff640 1 -- 172.21.15.12:0/1595762167 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fef7c050df0 con 0x7fef84138230 2024-09-11T01:32:55.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:55.561+0000 7fef7bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:32:55.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:55.561+0000 7fef7bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:32:55.562 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:32:55.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:55.563+0000 7fef89a23640 1 -- 172.21.15.12:0/1595762167 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fef6403e1c0 msgr2=0x7fef64040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:55.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:55.563+0000 7fef89a23640 1 --2- 172.21.15.12:0/1595762167 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fef6403e1c0 0x7fef64040680 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fef74033000 tx=0x7fef74009000 comp rx=0 tx=0).stop 2024-09-11T01:32:55.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:55.563+0000 7fef89a23640 10 monclient: shutdown 2024-09-11T01:32:55.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:55.563+0000 7fef89a23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:55.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:55.563+0000 7fef89a23640 1 -- 172.21.15.12:0/1595762167 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fef84138230 msgr2=0x7fef84137760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:55.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:55.563+0000 7fef89a23640 1 --2- 172.21.15.12:0/1595762167 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fef84138230 0x7fef84137760 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fef7c009fd0 tx=0x7fef7c00b870 comp rx=0 tx=0).stop 2024-09-11T01:32:55.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:55.564+0000 7fef89a23640 1 -- 172.21.15.12:0/1595762167 shutdown_connections 2024-09-11T01:32:55.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:55.564+0000 7fef89a23640 1 --2- 172.21.15.12:0/1595762167 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fef84138230 0x7fef84137760 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:55.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:55.564+0000 7fef89a23640 1 --2- 172.21.15.12:0/1595762167 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef8409d430 0x7fef84137220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:55.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:55.564+0000 7fef89a23640 1 --2- 172.21.15.12:0/1595762167 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fef6403e1c0 0x7fef64040680 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:55.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:55.564+0000 7fef89a23640 1 --2- 172.21.15.12:0/1595762167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef8409d030 0x7fef841cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:55.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:55.564+0000 7fef89a23640 1 -- 172.21.15.12:0/1595762167 >> 172.21.15.12:0/1595762167 conn(0x7fef840a7120 msgr2=0x7fef840a2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:55.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:55.564+0000 7fef89a23640 1 -- 172.21.15.12:0/1595762167 shutdown_connections 2024-09-11T01:32:55.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:55.564+0000 7fef89a23640 1 -- 172.21.15.12:0/1595762167 wait complete. 2024-09-11T01:32:58.581 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:32:58.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.693+0000 7ff68a657640 10 monclient: get_monmap_and_config 2024-09-11T01:32:58.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.693+0000 7ff68a657640 10 monclient: build_initial_monmap 2024-09-11T01:32:58.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.693+0000 7ff68a657640 10 monclient: monmap: 2024-09-11T01:32:58.695 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:58.695 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:58.696 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:32:58.694386+0000 2024-09-11T01:32:58.696 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:32:58.694386+0000 2024-09-11T01:32:58.696 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:58.696 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:58.696 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:32:58.696 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:32:58.696 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:32:58.696 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:32:58.696 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:32:58.696 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:58.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.694+0000 7ff68a657640 1 Processor -- start 2024-09-11T01:32:58.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.694+0000 7ff68a657640 1 -- start start 2024-09-11T01:32:58.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.694+0000 7ff68a657640 10 monclient: init 2024-09-11T01:32:58.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff68a657640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:58.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff68a657640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-11T01:32:58.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff68a657640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff68409d790 addr v1:172.21.15.12:6789/0 2024-09-11T01:32:58.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff68a657640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff68409e280 0x7ff684139f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:58.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff68a657640 10 monclient(hunting): picked mon.noname-b con 0x7ff68409e280 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:58.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff68a657640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff68409e6a0 0x7ff68413c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:58.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff68a657640 10 monclient(hunting): picked mon.noname-c con 0x7ff68409e6a0 addr v2:172.21.15.71:3300/0 2024-09-11T01:32:58.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff68a657640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6840a1da0 con 0x7ff68409d790 2024-09-11T01:32:58.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff68a657640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:58.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff68a657640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7ff6840a0760 con 0x7ff68409e6a0 2024-09-11T01:32:58.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff68a657640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:58.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff68a657640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff6840a1fa0 con 0x7ff68409e280 2024-09-11T01:32:58.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff68a657640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:58.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff68a657640 10 monclient(hunting): authenticate will time out at 985.137175s 2024-09-11T01:32:58.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff683fff640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7ff68409d790 legacy=0x7ff68409db70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:32:58.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff6837fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff68409e280 0x7ff684139f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:58.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff6837fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff68409e280 0x7ff684139f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:50214/0 (socket says 172.21.15.12:50214) 2024-09-11T01:32:58.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff6837fe640 1 -- 172.21.15.12:0/364350301 learned_addr learned my addr 172.21.15.12:0/364350301 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:58.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff6837fe640 10 monclient(hunting): get_auth_request con 0x7ff68409e280 auth_method 0 2024-09-11T01:32:58.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff6837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:58.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff6837fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:58.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff6837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:58.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff688bf4640 1 --2- 172.21.15.12:0/364350301 >> v2:172.21.15.71:3300/0 conn(0x7ff68409e6a0 0x7ff68413c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:58.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff6837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:58.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff6837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:58.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff6837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:58.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff688bf4640 10 monclient(hunting): get_auth_request con 0x7ff68409e6a0 auth_method 0 2024-09-11T01:32:58.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff688bf4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:58.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff688bf4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:58.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff688bf4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:58.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff688bf4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:58.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff688bf4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:58.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff688bf4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:58.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff6837fe640 10 monclient(hunting): handle_auth_done global_id 44150 payload 274 2024-09-11T01:32:58.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff6837fe640 1 -- 172.21.15.12:0/364350301 >> v2:172.21.15.71:3300/0 conn(0x7ff68409e6a0 msgr2=0x7ff68413c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:32:58.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff6837fe640 1 --2- 172.21.15.12:0/364350301 >> v2:172.21.15.71:3300/0 conn(0x7ff68409e6a0 0x7ff68413c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:58.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff6837fe640 1 -- 172.21.15.12:0/364350301 >> v1:172.21.15.12:6789/0 conn(0x7ff68409d790 legacy=0x7ff68409db70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:32:58.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff6837fe640 10 monclient: _finish_hunting 0 2024-09-11T01:32:58.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff6837fe640 1 monclient: found mon.noname-b 2024-09-11T01:32:58.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff6837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:58.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff6837fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:32:58.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.695+0000 7ff6837fe640 1 -- 172.21.15.12:0/364350301 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff68409eda0 con 0x7ff68409e280 2024-09-11T01:32:58.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.698+0000 7ff6837fe640 1 -- 172.21.15.12:0/364350301 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff68409e280 msgr2=0x7ff684139f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:58.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.698+0000 7ff6837fe640 1 --2- 172.21.15.12:0/364350301 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff68409e280 0x7ff684139f20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff678009870 tx=0x7ff67802a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3aa1a889abfa1555 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:58.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.698+0000 7ff682ffd640 1 -- 172.21.15.12:0/364350301 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff678037070 con 0x7ff68409e280 2024-09-11T01:32:58.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.698+0000 7ff682ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:58.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.698+0000 7ff682ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:32:58.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.698+0000 7ff682ffd640 10 monclient: dump: 2024-09-11T01:32:58.708 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:58.708 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:58.708 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:58.708 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:58.709 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:58.709 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:58.709 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:58.709 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:58.709 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:58.709 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:58.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.698+0000 7ff682ffd640 10 monclient: _finish_auth 0 2024-09-11T01:32:58.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.698+0000 7ff682ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:32:58.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.698+0000 7ff682ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:58.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.698+0000 7ff682ffd640 1 -- 172.21.15.12:0/364350301 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff678032640 con 0x7ff68409e280 2024-09-11T01:32:58.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.698+0000 7ff68a657640 5 monclient: authenticate success, global_id 44150 2024-09-11T01:32:58.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.698+0000 7ff68a657640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:32:58.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.698+0000 7ff682ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:58.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.698+0000 7ff68a657640 10 monclient: get_monmap_and_config success 2024-09-11T01:32:58.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.698+0000 7ff68a657640 4 set_mon_vals no callback set 2024-09-11T01:32:58.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.698+0000 7ff682ffd640 1 -- 172.21.15.12:0/364350301 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff678032a10 con 0x7ff68409e280 2024-09-11T01:32:58.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.698+0000 7ff682ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:58.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.698+0000 7ff682ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:32:58.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.698+0000 7ff682ffd640 10 monclient: dump: 2024-09-11T01:32:58.710 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:58.710 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:58.710 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:58.710 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:58.710 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:58.710 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:58.711 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:58.711 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:58.711 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:58.711 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:58.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.699+0000 7ff68a657640 10 monclient: shutdown 2024-09-11T01:32:58.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.699+0000 7ff68a657640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:58.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.699+0000 7ff68a657640 1 -- 172.21.15.12:0/364350301 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff68409e280 msgr2=0x7ff684139f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:58.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.699+0000 7ff68a657640 1 --2- 172.21.15.12:0/364350301 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff68409e280 0x7ff684139f20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff678009870 tx=0x7ff67802a130 comp rx=0 tx=0).stop 2024-09-11T01:32:58.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.699+0000 7ff68a657640 1 -- 172.21.15.12:0/364350301 shutdown_connections 2024-09-11T01:32:58.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.699+0000 7ff68a657640 1 --2- 172.21.15.12:0/364350301 >> v2:172.21.15.71:3300/0 conn(0x7ff68409e6a0 0x7ff68413c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:58.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.699+0000 7ff68a657640 1 --2- 172.21.15.12:0/364350301 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff68409e280 0x7ff684139f20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:58.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.699+0000 7ff68a657640 1 -- 172.21.15.12:0/364350301 >> 172.21.15.12:0/364350301 conn(0x7ff6840a2d90 msgr2=0x7ff6840a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:58.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.699+0000 7ff68a657640 1 -- 172.21.15.12:0/364350301 shutdown_connections 2024-09-11T01:32:58.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.699+0000 7ff68a657640 1 -- 172.21.15.12:0/364350301 wait complete. 2024-09-11T01:32:58.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.700+0000 7ff68a657640 10 monclient: build_initial_monmap 2024-09-11T01:32:58.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.700+0000 7ff68a657640 10 monclient: monmap: 2024-09-11T01:32:58.712 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:32:58.712 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:32:58.712 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:32:58.712 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:32:58.712 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:32:58.712 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:58.712 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:32:58.713 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:32:58.713 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:32:58.713 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:58.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.700+0000 7ff68a657640 1 Processor -- start 2024-09-11T01:32:58.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.700+0000 7ff68a657640 1 -- start start 2024-09-11T01:32:58.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.700+0000 7ff68a657640 10 monclient: init 2024-09-11T01:32:58.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff68a657640 10 monclient: _reopen_session rank -1 2024-09-11T01:32:58.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff68a657640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:32:58.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff68a657640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff68409d790 0x7ff6841363e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:58.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff68a657640 10 monclient(hunting): picked mon.noname-a con 0x7ff68409d790 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:32:58.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff68a657640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff68409e280 0x7ff684136920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:58.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff68a657640 10 monclient(hunting): picked mon.noname-c con 0x7ff68409e280 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:32:58.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff68a657640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff68409e6a0 0x7ff6841cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:58.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff68a657640 10 monclient(hunting): picked mon.noname-b con 0x7ff68409e6a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:32:58.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff68a657640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:58.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff68a657640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff68409ce20 con 0x7ff68409d790 2024-09-11T01:32:58.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff68a657640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:58.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff68a657640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff68409cca0 con 0x7ff68409e6a0 2024-09-11T01:32:58.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff68a657640 10 monclient(hunting): start opening mon connection 2024-09-11T01:32:58.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff68a657640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff6840a0c90 con 0x7ff68409e280 2024-09-11T01:32:58.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff68a657640 10 monclient(hunting): _renew_subs 2024-09-11T01:32:58.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff683fff640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff68409d790 msgr2=0x7ff6841363e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:32:58.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff68a657640 10 monclient(hunting): authenticate will time out at 985.143243s 2024-09-11T01:32:58.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff683fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff68409d790 0x7ff6841363e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:32:58.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff6837fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff68409e280 0x7ff684136920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:58.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff6837fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff68409e280 0x7ff684136920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:50230/0 (socket says 172.21.15.12:50230) 2024-09-11T01:32:58.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff6837fe640 1 -- 172.21.15.12:0/1396349990 learned_addr learned my addr 172.21.15.12:0/1396349990 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:32:58.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff6837fe640 10 monclient(hunting): get_auth_request con 0x7ff68409e280 auth_method 0 2024-09-11T01:32:58.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff6837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:58.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff6837fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:58.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff6837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:58.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff6837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:32:58.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff6837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:32:58.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff6837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:32:58.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff688bf4640 1 --2- 172.21.15.12:0/1396349990 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff68409e6a0 0x7ff6841cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:58.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff688bf4640 10 monclient(hunting): get_auth_request con 0x7ff68409e6a0 auth_method 0 2024-09-11T01:32:58.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff688bf4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:32:58.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff688bf4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:32:58.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff688bf4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:32:58.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff6837fe640 10 monclient(hunting): handle_auth_done global_id 44153 payload 824 2024-09-11T01:32:58.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff6837fe640 1 -- 172.21.15.12:0/1396349990 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff68409e6a0 msgr2=0x7ff6841cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:58.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff6837fe640 1 --2- 172.21.15.12:0/1396349990 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff68409e6a0 0x7ff6841cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:58.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff6837fe640 1 -- 172.21.15.12:0/1396349990 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff68409d790 msgr2=0x7ff6841363e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:32:58.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff6837fe640 1 --2- 172.21.15.12:0/1396349990 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff68409d790 0x7ff6841363e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:58.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff6837fe640 10 monclient: _finish_hunting 0 2024-09-11T01:32:58.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff6837fe640 1 monclient: found mon.noname-c 2024-09-11T01:32:58.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff6837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:32:58.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff6837fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:32:58.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff6837fe640 1 -- 172.21.15.12:0/1396349990 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff684074120 con 0x7ff68409e280 2024-09-11T01:32:58.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.701+0000 7ff688bf4640 1 --2- 172.21.15.12:0/1396349990 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff68409e6a0 0x7ff6841cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:32:58.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.704+0000 7ff6837fe640 1 -- 172.21.15.12:0/1396349990 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff68409e280 msgr2=0x7ff684136920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:32:58.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.704+0000 7ff6837fe640 1 --2- 172.21.15.12:0/1396349990 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff68409e280 0x7ff684136920 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff678009870 tx=0x7ff6780046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:58.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.704+0000 7ff680ff9640 1 -- 172.21.15.12:0/1396349990 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff678041070 con 0x7ff68409e280 2024-09-11T01:32:58.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.704+0000 7ff680ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:58.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.704+0000 7ff680ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:32:58.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.704+0000 7ff680ff9640 10 monclient: dump: 2024-09-11T01:32:58.717 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:58.717 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:58.717 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:58.717 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:58.717 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:58.717 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:58.717 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:58.717 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:58.717 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:58.717 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:58.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.704+0000 7ff680ff9640 10 monclient: _finish_auth 0 2024-09-11T01:32:58.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.704+0000 7ff680ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:32:58.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.704+0000 7ff680ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:32:58.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.704+0000 7ff68a657640 5 monclient: authenticate success, global_id 44153 2024-09-11T01:32:58.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.704+0000 7ff680ff9640 1 -- 172.21.15.12:0/1396349990 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff678032e50 con 0x7ff68409e280 2024-09-11T01:32:58.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.704+0000 7ff680ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:32:58.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.704+0000 7ff68a657640 10 monclient: _renew_subs 2024-09-11T01:32:58.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.704+0000 7ff680ff9640 1 -- 172.21.15.12:0/1396349990 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff678037040 con 0x7ff68409e280 2024-09-11T01:32:58.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.704+0000 7ff68a657640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:32:58.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.704+0000 7ff68a657640 1 -- 172.21.15.12:0/1396349990 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff68409d480 con 0x7ff68409e280 2024-09-11T01:32:58.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.704+0000 7ff680ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:32:58.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.704+0000 7ff680ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:32:58.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.704+0000 7ff680ff9640 10 monclient: dump: 2024-09-11T01:32:58.718 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:32:58.719 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:32:58.719 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:58.719 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:32:58.719 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:32:58.719 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:32:58.719 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:32:58.719 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:32:58.719 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:32:58.719 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:32:58.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.704+0000 7ff6827fc640 4 set_mon_vals no callback set 2024-09-11T01:32:58.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.704+0000 7ff68a657640 10 monclient: _renew_subs 2024-09-11T01:32:58.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.704+0000 7ff68a657640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:32:58.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.704+0000 7ff68a657640 1 -- 172.21.15.12:0/1396349990 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6840a5670 con 0x7ff68409e280 2024-09-11T01:32:58.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.705+0000 7ff68a657640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:58.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.705+0000 7ff68a657640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:58.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.705+0000 7ff68a657640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:32:58.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.705+0000 7ff68a657640 1 -- 172.21.15.12:0/1396349990 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff68409db70 con 0x7ff68409e280 2024-09-11T01:32:58.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.705+0000 7ff680ff9640 1 -- 172.21.15.12:0/1396349990 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff67803caa0 con 0x7ff68409e280 2024-09-11T01:32:58.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.706+0000 7ff680ff9640 1 --2- 172.21.15.12:0/1396349990 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff67003e240 0x7ff670040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:32:58.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.706+0000 7ff680ff9640 1 -- 172.21.15.12:0/1396349990 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(139..139 src has 1..139) ==== 4755+0+0 (secure 0 0 0) 0x7ff6780736b0 con 0x7ff68409e280 2024-09-11T01:32:58.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.706+0000 7ff683fff640 1 --2- 172.21.15.12:0/1396349990 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff67003e240 0x7ff670040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:32:58.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.706+0000 7ff683fff640 10 monclient: get_auth_request con 0x7ff67003e240 auth_method 0 2024-09-11T01:32:58.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.708+0000 7ff683fff640 1 --2- 172.21.15.12:0/1396349990 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff67003e240 0x7ff670040700 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff674003980 tx=0x7ff674006d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:32:58.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.710+0000 7ff680ff9640 1 -- 172.21.15.12:0/1396349990 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff68409d480 con 0x7ff68409e280 2024-09-11T01:32:58.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.710+0000 7ff680ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:32:58.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.710+0000 7ff680ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:32:58.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.967+0000 7ff68a657640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:32:58.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.967+0000 7ff68a657640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:32:58.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.967+0000 7ff68a657640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:32:58.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.967+0000 7ff68a657640 1 -- 172.21.15.12:0/1396349990 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff684138740 con 0x7ff68409e280 2024-09-11T01:32:58.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.967+0000 7ff680ff9640 1 -- 172.21.15.12:0/1396349990 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7ff67803cdc0 con 0x7ff68409e280 2024-09-11T01:32:58.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.967+0000 7ff680ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:32:58.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.967+0000 7ff680ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:32:58.970 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:32:58.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.969+0000 7ff68a657640 1 -- 172.21.15.12:0/1396349990 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff67003e240 msgr2=0x7ff670040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:58.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.969+0000 7ff68a657640 1 --2- 172.21.15.12:0/1396349990 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff67003e240 0x7ff670040700 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff674003980 tx=0x7ff674006d30 comp rx=0 tx=0).stop 2024-09-11T01:32:58.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.969+0000 7ff68a657640 10 monclient: shutdown 2024-09-11T01:32:58.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.969+0000 7ff68a657640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:32:58.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.969+0000 7ff68a657640 1 -- 172.21.15.12:0/1396349990 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff68409e280 msgr2=0x7ff684136920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:32:58.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.969+0000 7ff68a657640 1 --2- 172.21.15.12:0/1396349990 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff68409e280 0x7ff684136920 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff678009870 tx=0x7ff6780046c0 comp rx=0 tx=0).stop 2024-09-11T01:32:58.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.969+0000 7ff68a657640 1 -- 172.21.15.12:0/1396349990 shutdown_connections 2024-09-11T01:32:58.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.969+0000 7ff68a657640 1 --2- 172.21.15.12:0/1396349990 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff68409e6a0 0x7ff6841cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:58.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.969+0000 7ff68a657640 1 --2- 172.21.15.12:0/1396349990 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff68409e280 0x7ff684136920 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:58.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.969+0000 7ff68a657640 1 --2- 172.21.15.12:0/1396349990 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff67003e240 0x7ff670040700 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:58.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.969+0000 7ff68a657640 1 --2- 172.21.15.12:0/1396349990 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff68409d790 0x7ff6841363e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:32:58.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.969+0000 7ff68a657640 1 -- 172.21.15.12:0/1396349990 >> 172.21.15.12:0/1396349990 conn(0x7ff6840a2d90 msgr2=0x7ff684130a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:32:58.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.970+0000 7ff68a657640 1 -- 172.21.15.12:0/1396349990 shutdown_connections 2024-09-11T01:32:58.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:32:58.970+0000 7ff68a657640 1 -- 172.21.15.12:0/1396349990 wait complete. 2024-09-11T01:33:01.983 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:33:02.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.095+0000 7f8b9a7e8640 10 monclient: get_monmap_and_config 2024-09-11T01:33:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.095+0000 7f8b9a7e8640 10 monclient: build_initial_monmap 2024-09-11T01:33:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.095+0000 7f8b9a7e8640 10 monclient: monmap: 2024-09-11T01:33:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:33:02.096029+0000 2024-09-11T01:33:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:33:02.096029+0000 2024-09-11T01:33:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:33:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:33:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:33:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:33:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:33:02.098 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.096+0000 7f8b9a7e8640 1 Processor -- start 2024-09-11T01:33:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.096+0000 7f8b9a7e8640 1 -- start start 2024-09-11T01:33:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.096+0000 7f8b9a7e8640 10 monclient: init 2024-09-11T01:33:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.096+0000 7f8b9a7e8640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.096+0000 7f8b9a7e8640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-11T01:33:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.096+0000 7f8b9a7e8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8b94138230 addr v1:172.21.15.12:6789/0 2024-09-11T01:33:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.096+0000 7f8b9a7e8640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f8b9409d030 0x7f8b9413d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.096+0000 7f8b9a7e8640 10 monclient(hunting): picked mon.noname-c con 0x7f8b9409d030 addr v2:172.21.15.71:3300/0 2024-09-11T01:33:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.096+0000 7f8b9a7e8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b9409d460 0x7f8b9413f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:02.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.096+0000 7f8b9a7e8640 10 monclient(hunting): picked mon.noname-b con 0x7f8b9409d460 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:02.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.096+0000 7f8b9a7e8640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b940a6e30 con 0x7f8b94138230 2024-09-11T01:33:02.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.096+0000 7f8b9a7e8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:02.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.096+0000 7f8b9a7e8640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f8b9409c9c0 con 0x7f8b9409d030 2024-09-11T01:33:02.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.096+0000 7f8b9a7e8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:02.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.096+0000 7f8b9a7e8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8b9409ba20 con 0x7f8b9409d460 2024-09-11T01:33:02.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.096+0000 7f8b9a7e8640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:02.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.096+0000 7f8b9a7e8640 10 monclient(hunting): authenticate will time out at 988.538856s 2024-09-11T01:33:02.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.096+0000 7f8b93fff640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7f8b94138230 legacy=0x7f8b94138610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:33:02.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.096+0000 7f8b98d85640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b9409d460 0x7f8b9413f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:02.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.097+0000 7f8b98d85640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b9409d460 0x7f8b9413f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:50246/0 (socket says 172.21.15.12:50246) 2024-09-11T01:33:02.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.097+0000 7f8b98d85640 1 -- 172.21.15.12:0/2900666284 learned_addr learned my addr 172.21.15.12:0/2900666284 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.097+0000 7f8b98d85640 10 monclient(hunting): get_auth_request con 0x7f8b9409d460 auth_method 0 2024-09-11T01:33:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.097+0000 7f8b98d85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.097+0000 7f8b98d85640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.097+0000 7f8b98d85640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.097+0000 7f8b98d85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.097+0000 7f8b98d85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.097+0000 7f8b98d85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.097+0000 7f8b8bfff640 1 --2- 172.21.15.12:0/2900666284 >> v2:172.21.15.71:3300/0 conn(0x7f8b9409d030 0x7f8b9413d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.097+0000 7f8b8bfff640 10 monclient(hunting): get_auth_request con 0x7f8b9409d030 auth_method 0 2024-09-11T01:33:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.097+0000 7f8b8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.097+0000 7f8b8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.097+0000 7f8b8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.097+0000 7f8b98d85640 10 monclient(hunting): handle_auth_done global_id 44156 payload 274 2024-09-11T01:33:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.097+0000 7f8b98d85640 1 -- 172.21.15.12:0/2900666284 >> v2:172.21.15.71:3300/0 conn(0x7f8b9409d030 msgr2=0x7f8b9413d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:02.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.097+0000 7f8b98d85640 1 --2- 172.21.15.12:0/2900666284 >> v2:172.21.15.71:3300/0 conn(0x7f8b9409d030 0x7f8b9413d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:02.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.097+0000 7f8b98d85640 1 -- 172.21.15.12:0/2900666284 >> v1:172.21.15.12:6789/0 conn(0x7f8b94138230 legacy=0x7f8b94138610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:33:02.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.097+0000 7f8b98d85640 10 monclient: _finish_hunting 0 2024-09-11T01:33:02.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.097+0000 7f8b98d85640 1 monclient: found mon.noname-b 2024-09-11T01:33:02.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.097+0000 7f8b98d85640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:02.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.097+0000 7f8b98d85640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:33:02.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.097+0000 7f8b98d85640 1 -- 172.21.15.12:0/2900666284 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b940a66a0 con 0x7f8b9409d460 2024-09-11T01:33:02.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.100+0000 7f8b98d85640 1 -- 172.21.15.12:0/2900666284 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b9409d460 msgr2=0x7f8b9413f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:02.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.100+0000 7f8b98d85640 1 --2- 172.21.15.12:0/2900666284 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b9409d460 0x7f8b9413f5e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8b9409dae0 tx=0x7f8b8c02a540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4654794c0e7e42b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:02.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.100+0000 7f8b937fe640 1 -- 172.21.15.12:0/2900666284 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c036070 con 0x7f8b9409d460 2024-09-11T01:33:02.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.100+0000 7f8b937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:02.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.100+0000 7f8b937fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:33:02.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.100+0000 7f8b937fe640 10 monclient: dump: 2024-09-11T01:33:02.110 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:02.110 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:02.110 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:02.110 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:02.110 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:02.110 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:02.110 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:02.110 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:02.110 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:02.110 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:02.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.100+0000 7f8b937fe640 10 monclient: _finish_auth 0 2024-09-11T01:33:02.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.100+0000 7f8b937fe640 10 monclient: _check_auth_tickets 2024-09-11T01:33:02.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.100+0000 7f8b937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.100+0000 7f8b937fe640 1 -- 172.21.15.12:0/2900666284 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b8c007800 con 0x7f8b9409d460 2024-09-11T01:33:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.100+0000 7f8b9a7e8640 5 monclient: authenticate success, global_id 44156 2024-09-11T01:33:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.100+0000 7f8b937fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.100+0000 7f8b937fe640 1 -- 172.21.15.12:0/2900666284 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c007bd0 con 0x7f8b9409d460 2024-09-11T01:33:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.100+0000 7f8b9a7e8640 10 monclient: get_monmap_and_config success 2024-09-11T01:33:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.100+0000 7f8b937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.100+0000 7f8b9a7e8640 4 set_mon_vals no callback set 2024-09-11T01:33:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.100+0000 7f8b937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.100+0000 7f8b937fe640 10 monclient: dump: 2024-09-11T01:33:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:02.112 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:02.112 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:02.112 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:02.112 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:02.112 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:02.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.100+0000 7f8b9a7e8640 10 monclient: shutdown 2024-09-11T01:33:02.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.100+0000 7f8b9a7e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:02.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.100+0000 7f8b9a7e8640 1 -- 172.21.15.12:0/2900666284 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b9409d460 msgr2=0x7f8b9413f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:02.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.100+0000 7f8b9a7e8640 1 --2- 172.21.15.12:0/2900666284 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b9409d460 0x7f8b9413f5e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8b9409dae0 tx=0x7f8b8c02a540 comp rx=0 tx=0).stop 2024-09-11T01:33:02.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.101+0000 7f8b9a7e8640 1 -- 172.21.15.12:0/2900666284 shutdown_connections 2024-09-11T01:33:02.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.101+0000 7f8b9a7e8640 1 --2- 172.21.15.12:0/2900666284 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b9409d460 0x7f8b9413f5e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:02.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.101+0000 7f8b9a7e8640 1 --2- 172.21.15.12:0/2900666284 >> v2:172.21.15.71:3300/0 conn(0x7f8b9409d030 0x7f8b9413d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:02.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.101+0000 7f8b9a7e8640 1 -- 172.21.15.12:0/2900666284 >> 172.21.15.12:0/2900666284 conn(0x7f8b940a7120 msgr2=0x7f8b940a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:02.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.101+0000 7f8b9a7e8640 1 -- 172.21.15.12:0/2900666284 shutdown_connections 2024-09-11T01:33:02.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.101+0000 7f8b9a7e8640 1 -- 172.21.15.12:0/2900666284 wait complete. 2024-09-11T01:33:02.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.101+0000 7f8b9a7e8640 10 monclient: build_initial_monmap 2024-09-11T01:33:02.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.101+0000 7f8b9a7e8640 10 monclient: monmap: 2024-09-11T01:33:02.113 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:02.113 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:33:02.113 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:33:02.113 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:33:02.113 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:02.113 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:02.113 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:33:02.113 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:33:02.113 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:33:02.113 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b9a7e8640 1 Processor -- start 2024-09-11T01:33:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b9a7e8640 1 -- start start 2024-09-11T01:33:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b9a7e8640 10 monclient: init 2024-09-11T01:33:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b9a7e8640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b9a7e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:33:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b9a7e8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b9409d030 0x7f8b941366d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b9a7e8640 10 monclient(hunting): picked mon.noname-b con 0x7f8b9409d030 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:33:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b9a7e8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b9409d460 0x7f8b94136c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b9a7e8640 10 monclient(hunting): picked mon.noname-a con 0x7f8b9409d460 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:33:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b9a7e8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b94138230 0x7f8b9413c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b9a7e8640 10 monclient(hunting): picked mon.noname-c con 0x7f8b94138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b9a7e8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b9a7e8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9413fca0 con 0x7f8b9409d460 2024-09-11T01:33:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b9a7e8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:02.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b9a7e8640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9413fb20 con 0x7f8b9409d030 2024-09-11T01:33:02.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b9a7e8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:02.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b9a7e8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8b9413fe20 con 0x7f8b94138230 2024-09-11T01:33:02.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b9a7e8640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:02.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b9a7e8640 10 monclient(hunting): authenticate will time out at 988.544577s 2024-09-11T01:33:02.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b8bfff640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b9409d460 msgr2=0x7f8b94136c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:33:02.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b8bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b9409d460 0x7f8b94136c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:33:02.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b98d85640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b94138230 0x7f8b9413c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:02.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b98d85640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b94138230 0x7f8b9413c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:50256/0 (socket says 172.21.15.12:50256) 2024-09-11T01:33:02.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b98d85640 1 -- 172.21.15.12:0/1124455183 learned_addr learned my addr 172.21.15.12:0/1124455183 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:02.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b98d85640 10 monclient(hunting): get_auth_request con 0x7f8b94138230 auth_method 0 2024-09-11T01:33:02.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b98d85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:02.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b98d85640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:02.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b98d85640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:02.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b98d85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:02.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b98d85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:02.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b98d85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:02.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b93fff640 1 --2- 172.21.15.12:0/1124455183 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b9409d030 0x7f8b941366d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:02.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b93fff640 10 monclient(hunting): get_auth_request con 0x7f8b9409d030 auth_method 0 2024-09-11T01:33:02.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:02.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b93fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:02.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:02.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b98d85640 10 monclient(hunting): handle_auth_done global_id 44159 payload 824 2024-09-11T01:33:02.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b98d85640 1 -- 172.21.15.12:0/1124455183 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b9409d030 msgr2=0x7f8b941366d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:02.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b98d85640 1 --2- 172.21.15.12:0/1124455183 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b9409d030 0x7f8b941366d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:02.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b98d85640 1 -- 172.21.15.12:0/1124455183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b9409d460 msgr2=0x7f8b94136c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:33:02.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b98d85640 1 --2- 172.21.15.12:0/1124455183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b9409d460 0x7f8b94136c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:02.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b98d85640 10 monclient: _finish_hunting 0 2024-09-11T01:33:02.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b98d85640 1 monclient: found mon.noname-c 2024-09-11T01:33:02.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b98d85640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:02.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b98d85640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:33:02.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.102+0000 7f8b98d85640 1 -- 172.21.15.12:0/1124455183 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b940a1810 con 0x7f8b94138230 2024-09-11T01:33:02.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.105+0000 7f8b98d85640 1 -- 172.21.15.12:0/1124455183 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b94138230 msgr2=0x7f8b9413c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:02.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.105+0000 7f8b98d85640 1 --2- 172.21.15.12:0/1124455183 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b94138230 0x7f8b9413c990 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c034040 tx=0x7f8b8c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:02.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.105+0000 7f8b98d85640 1 --2- 172.21.15.12:0/1124455183 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b94138230 0x7f8b9413c990 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c034040 tx=0x7f8b8c004a40 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7f8b8c036050 mon_map magic: 0 2024-09-11T01:33:02.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.105+0000 7f8b917fa640 1 -- 172.21.15.12:0/1124455183 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c036050 con 0x7f8b94138230 2024-09-11T01:33:02.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.105+0000 7f8b917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:02.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.105+0000 7f8b917fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:33:02.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.105+0000 7f8b917fa640 10 monclient: dump: 2024-09-11T01:33:02.118 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:02.118 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:02.118 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:02.118 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:02.118 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:02.118 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:02.118 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:02.118 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:02.118 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:02.118 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:02.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.105+0000 7f8b917fa640 10 monclient: _finish_auth 0 2024-09-11T01:33:02.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.105+0000 7f8b917fa640 10 monclient: _check_auth_tickets 2024-09-11T01:33:02.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.105+0000 7f8b917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:02.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.105+0000 7f8b917fa640 1 -- 172.21.15.12:0/1124455183 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b8c042070 con 0x7f8b94138230 2024-09-11T01:33:02.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.105+0000 7f8b9a7e8640 5 monclient: authenticate success, global_id 44159 2024-09-11T01:33:02.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.105+0000 7f8b9a7e8640 10 monclient: _renew_subs 2024-09-11T01:33:02.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.105+0000 7f8b9a7e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:02.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.105+0000 7f8b9a7e8640 1 -- 172.21.15.12:0/1124455183 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b94121110 con 0x7f8b94138230 2024-09-11T01:33:02.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.105+0000 7f8b917fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:02.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.105+0000 7f8b92ffd640 4 set_mon_vals no callback set 2024-09-11T01:33:02.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.105+0000 7f8b9a7e8640 10 monclient: _renew_subs 2024-09-11T01:33:02.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.105+0000 7f8b9a7e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:02.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.105+0000 7f8b9a7e8640 1 -- 172.21.15.12:0/1124455183 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b9413eac0 con 0x7f8b94138230 2024-09-11T01:33:02.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.106+0000 7f8b9a7e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:02.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.106+0000 7f8b9a7e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:02.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.106+0000 7f8b9a7e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:02.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.106+0000 7f8b917fa640 1 -- 172.21.15.12:0/1124455183 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c031db0 con 0x7f8b94138230 2024-09-11T01:33:02.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.106+0000 7f8b9a7e8640 1 -- 172.21.15.12:0/1124455183 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b94138610 con 0x7f8b94138230 2024-09-11T01:33:02.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.106+0000 7f8b917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:02.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.106+0000 7f8b917fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:02.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.106+0000 7f8b917fa640 10 monclient: dump: 2024-09-11T01:33:02.120 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:02.120 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:02.120 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:02.121 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:02.121 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:02.121 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:02.121 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:02.121 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:02.121 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:02.121 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:02.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.106+0000 7f8b917fa640 1 -- 172.21.15.12:0/1124455183 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f8b8c03d450 con 0x7f8b94138230 2024-09-11T01:33:02.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.107+0000 7f8b917fa640 1 --2- 172.21.15.12:0/1124455183 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8b7403e1c0 0x7f8b74040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:02.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.107+0000 7f8b93fff640 1 --2- 172.21.15.12:0/1124455183 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8b7403e1c0 0x7f8b74040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:02.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.107+0000 7f8b93fff640 10 monclient: get_auth_request con 0x7f8b7403e1c0 auth_method 0 2024-09-11T01:33:02.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.109+0000 7f8b93fff640 1 --2- 172.21.15.12:0/1124455183 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8b7403e1c0 0x7f8b74040680 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8b84003980 tx=0x7f8b84008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:02.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.110+0000 7f8b917fa640 1 -- 172.21.15.12:0/1124455183 DispatchQueue::entry inject delay of 0.002000 2024-09-11T01:33:02.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.112+0000 7f8b917fa640 1 -- 172.21.15.12:0/1124455183 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(139..139 src has 1..139) ==== 4755+0+0 (secure 0 0 0) 0x7f8b8c073890 con 0x7f8b94138230 2024-09-11T01:33:02.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.728+0000 7f8b917fa640 1 -- 172.21.15.12:0/1124455183 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8b8c04b020 con 0x7f8b94138230 2024-09-11T01:33:02.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.728+0000 7f8b917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:02.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.728+0000 7f8b917fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:33:02.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.993+0000 7f8b9a7e8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:33:02.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.993+0000 7f8b9a7e8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:33:02.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.993+0000 7f8b9a7e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:02.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.993+0000 7f8b9a7e8640 1 -- 172.21.15.12:0/1124455183 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8b940698e0 con 0x7f8b94138230 2024-09-11T01:33:02.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.994+0000 7f8b917fa640 1 -- 172.21.15.12:0/1124455183 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f8b940a1810 con 0x7f8b94138230 2024-09-11T01:33:02.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.994+0000 7f8b917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:33:02.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.994+0000 7f8b917fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:33:02.996 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:33:02.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.996+0000 7f8b9a7e8640 1 -- 172.21.15.12:0/1124455183 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8b7403e1c0 msgr2=0x7f8b74040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:02.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.996+0000 7f8b9a7e8640 1 --2- 172.21.15.12:0/1124455183 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8b7403e1c0 0x7f8b74040680 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8b84003980 tx=0x7f8b84008040 comp rx=0 tx=0).stop 2024-09-11T01:33:02.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.996+0000 7f8b9a7e8640 10 monclient: shutdown 2024-09-11T01:33:02.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.996+0000 7f8b9a7e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:02.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.996+0000 7f8b9a7e8640 1 -- 172.21.15.12:0/1124455183 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b94138230 msgr2=0x7f8b9413c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:02.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.996+0000 7f8b9a7e8640 1 --2- 172.21.15.12:0/1124455183 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b94138230 0x7f8b9413c990 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c034040 tx=0x7f8b8c004a40 comp rx=0 tx=0).stop 2024-09-11T01:33:02.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.996+0000 7f8b9a7e8640 1 -- 172.21.15.12:0/1124455183 shutdown_connections 2024-09-11T01:33:02.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.996+0000 7f8b9a7e8640 1 --2- 172.21.15.12:0/1124455183 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b94138230 0x7f8b9413c990 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:02.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.996+0000 7f8b9a7e8640 1 --2- 172.21.15.12:0/1124455183 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8b7403e1c0 0x7f8b74040680 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:02.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.996+0000 7f8b9a7e8640 1 --2- 172.21.15.12:0/1124455183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b9409d460 0x7f8b94136c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:02.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.996+0000 7f8b9a7e8640 1 --2- 172.21.15.12:0/1124455183 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b9409d030 0x7f8b941366d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:02.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.996+0000 7f8b9a7e8640 1 -- 172.21.15.12:0/1124455183 >> 172.21.15.12:0/1124455183 conn(0x7f8b940a7120 msgr2=0x7f8b940a44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:02.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.996+0000 7f8b9a7e8640 1 -- 172.21.15.12:0/1124455183 shutdown_connections 2024-09-11T01:33:02.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:02.996+0000 7f8b9a7e8640 1 -- 172.21.15.12:0/1124455183 wait complete. 2024-09-11T01:33:06.010 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:33:06.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.122+0000 7f6f18ca9640 10 monclient: get_monmap_and_config 2024-09-11T01:33:06.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.122+0000 7f6f18ca9640 10 monclient: build_initial_monmap 2024-09-11T01:33:06.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.122+0000 7f6f18ca9640 10 monclient: monmap: 2024-09-11T01:33:06.124 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:06.124 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:06.124 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:33:06.123026+0000 2024-09-11T01:33:06.124 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:33:06.123026+0000 2024-09-11T01:33:06.124 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:06.124 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:06.124 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:33:06.124 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:33:06.124 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:33:06.125 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:33:06.125 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:33:06.125 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:06.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.122+0000 7f6f18ca9640 1 Processor -- start 2024-09-11T01:33:06.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.123+0000 7f6f18ca9640 1 -- start start 2024-09-11T01:33:06.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.123+0000 7f6f18ca9640 10 monclient: init 2024-09-11T01:33:06.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.123+0000 7f6f18ca9640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:06.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.123+0000 7f6f18ca9640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-11T01:33:06.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.123+0000 7f6f18ca9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6f14138230 0x7f6f14138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:06.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.123+0000 7f6f18ca9640 10 monclient(hunting): picked mon.noname-a con 0x7f6f14138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:33:06.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.123+0000 7f6f18ca9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6f1409d030 addr v1:172.21.15.71:6789/0 2024-09-11T01:33:06.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.123+0000 7f6f18ca9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f1409d460 0x7f6f140a5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:06.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.123+0000 7f6f18ca9640 10 monclient(hunting): picked mon.noname-b con 0x7f6f1409d460 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:06.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.123+0000 7f6f18ca9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f140a6e30 con 0x7f6f1409d030 2024-09-11T01:33:06.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.123+0000 7f6f18ca9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:06.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.123+0000 7f6f18ca9640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f6f1409c9c0 con 0x7f6f14138230 2024-09-11T01:33:06.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.123+0000 7f6f18ca9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:06.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.123+0000 7f6f1259c640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7f6f14138230 msgr2=0x7f6f14138610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:33:06.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.123+0000 7f6f18ca9640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6f1409ba20 con 0x7f6f1409d460 2024-09-11T01:33:06.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.123+0000 7f6f18ca9640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:06.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.123+0000 7f6f1259c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6f14138230 0x7f6f14138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:33:06.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.123+0000 7f6f18ca9640 10 monclient(hunting): authenticate will time out at 992.565884s 2024-09-11T01:33:06.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.123+0000 7f6f12d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f1409d460 0x7f6f140a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:06.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.123+0000 7f6f12d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f1409d460 0x7f6f140a5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:50270/0 (socket says 172.21.15.12:50270) 2024-09-11T01:33:06.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.124+0000 7f6f12d9d640 1 -- 172.21.15.12:0/3123350885 learned_addr learned my addr 172.21.15.12:0/3123350885 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:06.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.124+0000 7f6f12d9d640 10 monclient(hunting): get_auth_request con 0x7f6f1409d460 auth_method 0 2024-09-11T01:33:06.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.124+0000 7f6f12d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:06.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.124+0000 7f6f12d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:06.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.124+0000 7f6f12d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:06.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.124+0000 7f6f12d9d640 1 -- 172.21.15.12:0/3123350885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f1409d460 msgr2=0x7f6f140a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:33:06.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.124+0000 7f6f12d9d640 1 -- 172.21.15.12:0/3123350885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f1409d460 msgr2=0x7f6f140a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:06.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.124+0000 7f6f12d9d640 1 --2- 172.21.15.12:0/3123350885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f1409d460 0x7f6f140a5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:06.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.124+0000 7f6f12d9d640 1 --2- 172.21.15.12:0/3123350885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f1409d460 0x7f6f140a5730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:33:06.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.126+0000 7f6f11d9b640 1 -- 172.21.15.12:0/3123350885 >> v1:172.21.15.71:6789/0 conn(0x7f6f1409d030 legacy=0x7f6f140a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:06.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.324+0000 7f6f1259c640 1 -- 172.21.15.12:0/3123350885 >> v2:172.21.15.12:3300/0 conn(0x7f6f14138230 msgr2=0x7f6f14138610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:33:06.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.324+0000 7f6f1259c640 1 --2- 172.21.15.12:0/3123350885 >> v2:172.21.15.12:3300/0 conn(0x7f6f14138230 0x7f6f14138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:33:06.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.325+0000 7f6f12d9d640 1 --2- 172.21.15.12:0/3123350885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f1409d460 0x7f6f140a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:06.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.325+0000 7f6f12d9d640 10 monclient(hunting): get_auth_request con 0x7f6f1409d460 auth_method 0 2024-09-11T01:33:06.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.325+0000 7f6f12d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:06.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.325+0000 7f6f12d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:06.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.325+0000 7f6f12d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:06.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.325+0000 7f6f12d9d640 1 -- 172.21.15.12:0/3123350885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f1409d460 msgr2=0x7f6f140a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:33:06.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.325+0000 7f6f12d9d640 1 -- 172.21.15.12:0/3123350885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f1409d460 msgr2=0x7f6f140a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:06.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.325+0000 7f6f12d9d640 1 --2- 172.21.15.12:0/3123350885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f1409d460 0x7f6f140a5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:06.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.325+0000 7f6f12d9d640 1 --2- 172.21.15.12:0/3123350885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f1409d460 0x7f6f140a5730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:33:06.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.725+0000 7f6f1259c640 1 -- 172.21.15.12:0/3123350885 >> v2:172.21.15.12:3300/0 conn(0x7f6f14138230 msgr2=0x7f6f14138610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:33:06.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.725+0000 7f6f1259c640 1 --2- 172.21.15.12:0/3123350885 >> v2:172.21.15.12:3300/0 conn(0x7f6f14138230 0x7f6f14138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:33:06.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.726+0000 7f6f12d9d640 1 --2- 172.21.15.12:0/3123350885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f1409d460 0x7f6f140a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:06.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.726+0000 7f6f12d9d640 10 monclient(hunting): get_auth_request con 0x7f6f1409d460 auth_method 0 2024-09-11T01:33:06.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.726+0000 7f6f12d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:06.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.726+0000 7f6f12d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:06.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.726+0000 7f6f12d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:06.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.726+0000 7f6f12d9d640 1 -- 172.21.15.12:0/3123350885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f1409d460 msgr2=0x7f6f140a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:33:06.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.726+0000 7f6f12d9d640 1 -- 172.21.15.12:0/3123350885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f1409d460 msgr2=0x7f6f140a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:06.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.726+0000 7f6f12d9d640 1 --2- 172.21.15.12:0/3123350885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f1409d460 0x7f6f140a5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:06.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:06.726+0000 7f6f12d9d640 1 --2- 172.21.15.12:0/3123350885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f1409d460 0x7f6f140a5730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:33:07.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:07.525+0000 7f6f1259c640 1 -- 172.21.15.12:0/3123350885 >> v2:172.21.15.12:3300/0 conn(0x7f6f14138230 msgr2=0x7f6f14138610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:33:07.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:07.525+0000 7f6f1259c640 1 --2- 172.21.15.12:0/3123350885 >> v2:172.21.15.12:3300/0 conn(0x7f6f14138230 0x7f6f14138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:33:07.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:07.526+0000 7f6f12d9d640 1 --2- 172.21.15.12:0/3123350885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f1409d460 0x7f6f140a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:07.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:07.526+0000 7f6f12d9d640 10 monclient(hunting): get_auth_request con 0x7f6f1409d460 auth_method 0 2024-09-11T01:33:07.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:07.526+0000 7f6f12d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:07.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:07.526+0000 7f6f12d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:07.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:07.526+0000 7f6f12d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:07.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:07.526+0000 7f6f12d9d640 1 -- 172.21.15.12:0/3123350885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f1409d460 msgr2=0x7f6f140a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:33:07.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:07.526+0000 7f6f12d9d640 1 -- 172.21.15.12:0/3123350885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f1409d460 msgr2=0x7f6f140a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:07.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:07.526+0000 7f6f12d9d640 1 --2- 172.21.15.12:0/3123350885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f1409d460 0x7f6f140a5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:07.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:07.526+0000 7f6f12d9d640 1 --2- 172.21.15.12:0/3123350885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f1409d460 0x7f6f140a5730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:33:09.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f03fff640 10 monclient(hunting): tick 2024-09-11T01:33:09.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f03fff640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:33:09.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f03fff640 1 monclient(hunting): continuing hunt 2024-09-11T01:33:09.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f03fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:33:09.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f03fff640 1 -- 172.21.15.12:0/3123350885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f1409d460 msgr2=0x7f6f140a5730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:33:09.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f03fff640 1 --2- 172.21.15.12:0/3123350885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f1409d460 0x7f6f140a5730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:09.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f03fff640 1 -- 172.21.15.12:0/3123350885 >> v2:172.21.15.12:3300/0 conn(0x7f6f14138230 msgr2=0x7f6f14138610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:33:09.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f03fff640 1 --2- 172.21.15.12:0/3123350885 >> v2:172.21.15.12:3300/0 conn(0x7f6f14138230 0x7f6f14138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:09.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f03fff640 1 -- 172.21.15.12:0/3123350885 >> v1:172.21.15.71:6789/0 conn(0x7f6f1409d030 legacy=0x7f6f140a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f03fff640 10 monclient: _add_conns ranks=[0,4] 2024-09-11T01:33:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f03fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6ef4003760 addr v1:172.21.15.12:6789/0 2024-09-11T01:33:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f03fff640 1 --2- 172.21.15.12:0/3123350885 >> v2:172.21.15.71:3300/0 conn(0x7f6ef4007490 0x7f6ef4009930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f03fff640 10 monclient(hunting): picked mon.noname-c con 0x7f6ef4007490 addr v2:172.21.15.71:3300/0 2024-09-11T01:33:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f03fff640 1 -- 172.21.15.12:0/3123350885 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f140a66a0 con 0x7f6ef4003760 2024-09-11T01:33:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f03fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f03fff640 1 -- 172.21.15.12:0/3123350885 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f6f1409c9c0 con 0x7f6ef4007490 2024-09-11T01:33:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f03fff640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f1259c640 1 -- 172.21.15.12:0/3123350885 >> v1:172.21.15.12:6789/0 conn(0x7f6ef4003760 legacy=0x7f6ef4005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:33:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f11d9b640 1 --2- 172.21.15.12:0/3123350885 >> v2:172.21.15.71:3300/0 conn(0x7f6ef4007490 0x7f6ef4009930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f11d9b640 10 monclient(hunting): get_auth_request con 0x7f6ef4007490 auth_method 0 2024-09-11T01:33:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f11d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:09.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f11d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:09.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f11d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:09.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f11d9b640 1 -- 172.21.15.12:0/3123350885 >> v2:172.21.15.71:3300/0 conn(0x7f6ef4007490 msgr2=0x7f6ef4009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:33:09.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f11d9b640 1 -- 172.21.15.12:0/3123350885 >> v2:172.21.15.71:3300/0 conn(0x7f6ef4007490 msgr2=0x7f6ef4009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:09.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f11d9b640 1 --2- 172.21.15.12:0/3123350885 >> v2:172.21.15.71:3300/0 conn(0x7f6ef4007490 0x7f6ef4009930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:09.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.123+0000 7f6f11d9b640 1 --2- 172.21.15.12:0/3123350885 >> v2:172.21.15.71:3300/0 conn(0x7f6ef4007490 0x7f6ef4009930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:33:09.198 INFO:tasks.rados.rados.0.smithi071.stdout:616: done (1 left) 2024-09-11T01:33:09.198 INFO:tasks.rados.rados.0.smithi071.stdout:617: done (0 left) 2024-09-11T01:33:09.198 INFO:tasks.rados.rados.0.smithi071.stdout:618: rollback oid 41 current snap is 61 2024-09-11T01:33:09.198 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 41 to 58 2024-09-11T01:33:09.198 INFO:tasks.rados.rados.0.smithi071.stdout:619: copy_from oid 16 from oid 22 current snap is 61 2024-09-11T01:33:09.198 INFO:tasks.rados.rados.0.smithi071.stdout:620: copy_from oid 20 from oid 12 current snap is 61 2024-09-11T01:33:09.198 INFO:tasks.rados.rados.0.smithi071.stdout:621: rollback oid 13 current snap is 61 2024-09-11T01:33:09.198 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 13 to 57 2024-09-11T01:33:09.198 INFO:tasks.rados.rados.0.smithi071.stdout:622: copy_from oid 14 from oid 34 current snap is 61 2024-09-11T01:33:09.199 INFO:tasks.rados.rados.0.smithi071.stdout:623: rollback oid 29 current snap is 61 2024-09-11T01:33:09.199 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 29 to 57 2024-09-11T01:33:09.199 INFO:tasks.rados.rados.0.smithi071.stdout:624: write oid 47 current snap is 61 2024-09-11T01:33:09.199 INFO:tasks.rados.rados.0.smithi071.stdout:624: seq_num 243 ranges {484101=692450,1636889=447212} 2024-09-11T01:33:09.205 INFO:tasks.rados.rados.0.smithi071.stdout:624: writing smithi07134777-47 from 484101 to 1176551 tid 1 2024-09-11T01:33:09.211 INFO:tasks.rados.rados.0.smithi071.stdout:624: writing smithi07134777-47 from 1636889 to 2084101 tid 2 2024-09-11T01:33:09.211 INFO:tasks.rados.rados.0.smithi071.stdout:625: snap_remove snap 56 2024-09-11T01:33:09.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.324+0000 7f6f1259c640 1 -- 172.21.15.12:0/3123350885 >> v1:172.21.15.12:6789/0 conn(0x7f6ef4003760 legacy=0x7f6ef4005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:33:09.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.324+0000 7f6f11d9b640 1 --2- 172.21.15.12:0/3123350885 >> v2:172.21.15.71:3300/0 conn(0x7f6ef4007490 0x7f6ef4009930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:09.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.324+0000 7f6f11d9b640 10 monclient(hunting): get_auth_request con 0x7f6ef4007490 auth_method 0 2024-09-11T01:33:09.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.324+0000 7f6f11d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:09.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.324+0000 7f6f11d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:09.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.324+0000 7f6f11d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:09.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.324+0000 7f6f11d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:09.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.324+0000 7f6f11d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:09.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.324+0000 7f6f11d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:09.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.324+0000 7f6f11d9b640 10 monclient(hunting): handle_auth_done global_id 34246 payload 274 2024-09-11T01:33:09.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.324+0000 7f6f11d9b640 1 -- 172.21.15.12:0/3123350885 >> v1:172.21.15.12:6789/0 conn(0x7f6ef4003760 legacy=0x7f6ef4005bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:33:09.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.324+0000 7f6f11d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:33:09.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.324+0000 7f6f11d9b640 1 monclient: found mon.noname-c 2024-09-11T01:33:09.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.324+0000 7f6f11d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:09.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.324+0000 7f6f11d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:33:09.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.324+0000 7f6f11d9b640 1 -- 172.21.15.12:0/3123350885 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ef400b000 con 0x7f6ef4007490 2024-09-11T01:33:09.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.327+0000 7f6f11d9b640 1 -- 172.21.15.12:0/3123350885 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6ef4007490 msgr2=0x7f6ef4009930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:09.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.327+0000 7f6f11d9b640 1 --2- 172.21.15.12:0/3123350885 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6ef4007490 0x7f6ef4009930 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6f140a7000 tx=0x7f6f0802a310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0bed97840774f6a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:09.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.328+0000 7f6f1159a640 1 -- 172.21.15.12:0/3123350885 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f080072d0 con 0x7f6ef4007490 2024-09-11T01:33:09.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.328+0000 7f6f1159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:09.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.328+0000 7f6f1159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:33:09.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.328+0000 7f6f1159a640 10 monclient: dump: 2024-09-11T01:33:09.331 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:09.331 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:09.331 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.331 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.331 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:09.332 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:09.332 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:09.332 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:09.332 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:09.332 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:09.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.328+0000 7f6f1159a640 10 monclient: _finish_auth 0 2024-09-11T01:33:09.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.328+0000 7f6f1159a640 10 monclient: _check_auth_tickets 2024-09-11T01:33:09.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.328+0000 7f6f1159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:09.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.328+0000 7f6f1159a640 1 -- 172.21.15.12:0/3123350885 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f08007470 con 0x7f6ef4007490 2024-09-11T01:33:09.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.328+0000 7f6f18ca9640 5 monclient: authenticate success, global_id 34246 2024-09-11T01:33:09.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.328+0000 7f6f18ca9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:33:09.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.328+0000 7f6f1159a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:09.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.328+0000 7f6f18ca9640 10 monclient: get_monmap_and_config success 2024-09-11T01:33:09.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.328+0000 7f6f18ca9640 4 set_mon_vals no callback set 2024-09-11T01:33:09.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.328+0000 7f6f1159a640 1 -- 172.21.15.12:0/3123350885 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f080059e0 con 0x7f6ef4007490 2024-09-11T01:33:09.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.328+0000 7f6f1159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:09.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.328+0000 7f6f1159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:33:09.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.328+0000 7f6f1159a640 10 monclient: dump: 2024-09-11T01:33:09.333 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:09.333 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:09.333 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.333 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.333 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:09.333 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:09.333 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:09.333 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:09.334 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:09.334 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:09.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.329+0000 7f6f18ca9640 10 monclient: shutdown 2024-09-11T01:33:09.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.329+0000 7f6f18ca9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:09.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.329+0000 7f6f18ca9640 1 -- 172.21.15.12:0/3123350885 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6ef4007490 msgr2=0x7f6ef4009930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:09.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.329+0000 7f6f18ca9640 1 --2- 172.21.15.12:0/3123350885 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6ef4007490 0x7f6ef4009930 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6f140a7000 tx=0x7f6f0802a310 comp rx=0 tx=0).stop 2024-09-11T01:33:09.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.329+0000 7f6f12d9d640 1 -- 172.21.15.12:0/3123350885 reap_dead start 2024-09-11T01:33:09.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.329+0000 7f6f18ca9640 1 -- 172.21.15.12:0/3123350885 shutdown_connections 2024-09-11T01:33:09.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.329+0000 7f6f18ca9640 1 --2- 172.21.15.12:0/3123350885 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6ef4007490 0x7f6ef4009930 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:09.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.329+0000 7f6f18ca9640 1 -- 172.21.15.12:0/3123350885 >> 172.21.15.12:0/3123350885 conn(0x7f6f140a7120 msgr2=0x7f6f140a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:09.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.329+0000 7f6f18ca9640 1 -- 172.21.15.12:0/3123350885 shutdown_connections 2024-09-11T01:33:09.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.329+0000 7f6f18ca9640 1 -- 172.21.15.12:0/3123350885 wait complete. 2024-09-11T01:33:09.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.330+0000 7f6f18ca9640 10 monclient: build_initial_monmap 2024-09-11T01:33:09.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.330+0000 7f6f18ca9640 10 monclient: monmap: 2024-09-11T01:33:09.340 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:09.340 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:33:09.340 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:33:09.341 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:33:09.341 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:09.341 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:09.341 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:33:09.341 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:33:09.341 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:33:09.341 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:09.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.330+0000 7f6f18ca9640 1 Processor -- start 2024-09-11T01:33:09.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.330+0000 7f6f18ca9640 1 -- start start 2024-09-11T01:33:09.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.330+0000 7f6f18ca9640 10 monclient: init 2024-09-11T01:33:09.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f18ca9640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:09.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f18ca9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:33:09.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f18ca9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ef4007490 0x7f6f141368a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:09.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f18ca9640 10 monclient(hunting): picked mon.noname-a con 0x7f6ef4007490 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:33:09.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f18ca9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f14136de0 0x7f6f14120c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:09.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f18ca9640 10 monclient(hunting): picked mon.noname-c con 0x7f6f14136de0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:09.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f18ca9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f14137260 0x7f6f141211b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:09.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f18ca9640 10 monclient(hunting): picked mon.noname-b con 0x7f6f14137260 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:33:09.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f18ca9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:09.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f18ca9640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1406f970 con 0x7f6ef4007490 2024-09-11T01:33:09.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f18ca9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:09.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f18ca9640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1406f7f0 con 0x7f6f14137260 2024-09-11T01:33:09.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f18ca9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:09.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f18ca9640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6f140a5c70 con 0x7f6f14136de0 2024-09-11T01:33:09.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f18ca9640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:09.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f1259c640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ef4007490 msgr2=0x7f6f141368a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:33:09.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f18ca9640 10 monclient(hunting): authenticate will time out at 995.774421s 2024-09-11T01:33:09.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f1259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ef4007490 0x7f6f141368a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:33:09.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f11d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f14136de0 0x7f6f14120c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:09.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f11d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f14136de0 0x7f6f14120c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52930/0 (socket says 172.21.15.12:52930) 2024-09-11T01:33:09.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f11d9b640 1 -- 172.21.15.12:0/3123487784 learned_addr learned my addr 172.21.15.12:0/3123487784 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:09.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f11d9b640 10 monclient(hunting): get_auth_request con 0x7f6f14136de0 auth_method 0 2024-09-11T01:33:09.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f11d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:09.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f11d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:09.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f11d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:09.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f12d9d640 1 --2- 172.21.15.12:0/3123487784 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f14137260 0x7f6f141211b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:09.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f12d9d640 10 monclient(hunting): get_auth_request con 0x7f6f14137260 auth_method 0 2024-09-11T01:33:09.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f12d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:09.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f12d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:09.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f12d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:09.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f11d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:09.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f11d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:09.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f11d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:09.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f12d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:09.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f12d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:09.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f12d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:09.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f11d9b640 10 monclient(hunting): handle_auth_done global_id 44162 payload 824 2024-09-11T01:33:09.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f11d9b640 1 -- 172.21.15.12:0/3123487784 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f14137260 msgr2=0x7f6f141211b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:09.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.331+0000 7f6f11d9b640 1 --2- 172.21.15.12:0/3123487784 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f14137260 0x7f6f141211b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:09.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.332+0000 7f6f11d9b640 1 -- 172.21.15.12:0/3123487784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ef4007490 msgr2=0x7f6f141368a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:33:09.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.332+0000 7f6f11d9b640 1 --2- 172.21.15.12:0/3123487784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ef4007490 0x7f6f141368a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:09.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.332+0000 7f6f11d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:33:09.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.332+0000 7f6f11d9b640 1 monclient: found mon.noname-c 2024-09-11T01:33:09.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.332+0000 7f6f11d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:09.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.332+0000 7f6f11d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:33:09.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.332+0000 7f6f11d9b640 1 -- 172.21.15.12:0/3123487784 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f14124560 con 0x7f6f14136de0 2024-09-11T01:33:09.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.332+0000 7f6f12d9d640 1 --2- 172.21.15.12:0/3123487784 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f14137260 0x7f6f141211b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:33:09.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.334+0000 7f6f11d9b640 1 -- 172.21.15.12:0/3123487784 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f14136de0 msgr2=0x7f6f14120c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:09.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.334+0000 7f6f11d9b640 1 --2- 172.21.15.12:0/3123487784 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f14136de0 0x7f6f14120c70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6f0802a900 tx=0x7f6f08001560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:09.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.334+0000 7f6f02ffd640 1 -- 172.21.15.12:0/3123487784 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f08040020 con 0x7f6f14136de0 2024-09-11T01:33:09.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.334+0000 7f6f02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:09.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.334+0000 7f6f02ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:33:09.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.334+0000 7f6f02ffd640 10 monclient: dump: 2024-09-11T01:33:09.346 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:09.346 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:09.346 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.346 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.346 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:09.347 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:09.347 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:09.347 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:09.347 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:09.347 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:09.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.334+0000 7f6f02ffd640 10 monclient: _finish_auth 0 2024-09-11T01:33:09.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.334+0000 7f6f02ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:33:09.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.334+0000 7f6f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:09.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.334+0000 7f6f02ffd640 1 -- 172.21.15.12:0/3123487784 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f08007620 con 0x7f6f14136de0 2024-09-11T01:33:09.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.334+0000 7f6f18ca9640 5 monclient: authenticate success, global_id 44162 2024-09-11T01:33:09.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.334+0000 7f6f18ca9640 10 monclient: _renew_subs 2024-09-11T01:33:09.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.334+0000 7f6f18ca9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:09.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.334+0000 7f6f18ca9640 1 -- 172.21.15.12:0/3123487784 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f140a49e0 con 0x7f6f14136de0 2024-09-11T01:33:09.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.334+0000 7f6f02ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:09.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.334+0000 7f6f02ffd640 1 -- 172.21.15.12:0/3123487784 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0803cab0 con 0x7f6f14136de0 2024-09-11T01:33:09.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.334+0000 7f6f02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:09.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.334+0000 7f6f02ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:09.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.334+0000 7f6f02ffd640 10 monclient: dump: 2024-09-11T01:33:09.348 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:09.348 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:09.348 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.348 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.348 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:09.349 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:09.349 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:09.349 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:09.349 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:09.349 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:09.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.334+0000 7f6f10d99640 4 set_mon_vals no callback set 2024-09-11T01:33:09.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.334+0000 7f6f18ca9640 10 monclient: _renew_subs 2024-09-11T01:33:09.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.334+0000 7f6f18ca9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:09.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.334+0000 7f6f18ca9640 1 -- 172.21.15.12:0/3123487784 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f14124280 con 0x7f6f14136de0 2024-09-11T01:33:09.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.335+0000 7f6f02ffd640 1 -- 172.21.15.12:0/3123487784 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f6f0803ccd0 con 0x7f6f14136de0 2024-09-11T01:33:09.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.335+0000 7f6f18ca9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:09.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.335+0000 7f6f18ca9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:09.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.335+0000 7f6f02ffd640 1 --2- 172.21.15.12:0/3123487784 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6ef403f050 0x7f6ef4041510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:09.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.335+0000 7f6f18ca9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:09.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.335+0000 7f6f18ca9640 1 -- 172.21.15.12:0/3123487784 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f14138610 con 0x7f6f14136de0 2024-09-11T01:33:09.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.336+0000 7f6f02ffd640 1 -- 172.21.15.12:0/3123487784 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(140..140 src has 1..140) ==== 4755+0+0 (secure 0 0 0) 0x7f6f08073950 con 0x7f6f14136de0 2024-09-11T01:33:09.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.336+0000 7f6f1259c640 1 --2- 172.21.15.12:0/3123487784 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6ef403f050 0x7f6ef4041510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:09.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.336+0000 7f6f1259c640 10 monclient: get_auth_request con 0x7f6ef403f050 auth_method 0 2024-09-11T01:33:09.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.339+0000 7f6f1259c640 1 --2- 172.21.15.12:0/3123487784 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6ef403f050 0x7f6ef4041510 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6f04003d00 tx=0x7f6f04007310 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:09.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.341+0000 7f6f02ffd640 1 -- 172.21.15.12:0/3123487784 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6f0804a020 con 0x7f6f14136de0 2024-09-11T01:33:09.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.341+0000 7f6f02ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:09.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.341+0000 7f6f02ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:33:09.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.604+0000 7f6f18ca9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:33:09.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.604+0000 7f6f18ca9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:33:09.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.604+0000 7f6f18ca9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:09.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.604+0000 7f6f18ca9640 1 -- 172.21.15.12:0/3123487784 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6f140698e0 con 0x7f6f14136de0 2024-09-11T01:33:09.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.604+0000 7f6f02ffd640 1 -- 172.21.15.12:0/3123487784 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f6f08007400 con 0x7f6f14136de0 2024-09-11T01:33:09.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.604+0000 7f6f02ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:33:09.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.604+0000 7f6f02ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:33:09.608 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":40,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:33:09.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.607+0000 7f6f18ca9640 1 -- 172.21.15.12:0/3123487784 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6ef403f050 msgr2=0x7f6ef4041510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:09.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.607+0000 7f6f18ca9640 1 --2- 172.21.15.12:0/3123487784 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6ef403f050 0x7f6ef4041510 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6f04003d00 tx=0x7f6f04007310 comp rx=0 tx=0).stop 2024-09-11T01:33:09.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.607+0000 7f6f18ca9640 10 monclient: shutdown 2024-09-11T01:33:09.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.607+0000 7f6f18ca9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:09.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.607+0000 7f6f18ca9640 1 -- 172.21.15.12:0/3123487784 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f14136de0 msgr2=0x7f6f14120c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:09.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.607+0000 7f6f18ca9640 1 --2- 172.21.15.12:0/3123487784 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f14136de0 0x7f6f14120c70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6f0802a900 tx=0x7f6f08001560 comp rx=0 tx=0).stop 2024-09-11T01:33:09.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.607+0000 7f6f18ca9640 1 -- 172.21.15.12:0/3123487784 shutdown_connections 2024-09-11T01:33:09.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.607+0000 7f6f18ca9640 1 --2- 172.21.15.12:0/3123487784 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f14137260 0x7f6f141211b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:09.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.607+0000 7f6f18ca9640 1 --2- 172.21.15.12:0/3123487784 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f14136de0 0x7f6f14120c70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:09.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.607+0000 7f6f18ca9640 1 --2- 172.21.15.12:0/3123487784 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6ef403f050 0x7f6ef4041510 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:09.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.607+0000 7f6f18ca9640 1 --2- 172.21.15.12:0/3123487784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ef4007490 0x7f6f141368a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:09.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.607+0000 7f6f18ca9640 1 -- 172.21.15.12:0/3123487784 >> 172.21.15.12:0/3123487784 conn(0x7f6f140a7120 msgr2=0x7f6f140a3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:09.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.607+0000 7f6f18ca9640 1 -- 172.21.15.12:0/3123487784 shutdown_connections 2024-09-11T01:33:09.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.607+0000 7f6f18ca9640 1 -- 172.21.15.12:0/3123487784 wait complete. 2024-09-11T01:33:09.620 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:33:09.621 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T01:33:09.621 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:33:09.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.720+0000 7f90b5bbc640 10 monclient: get_monmap_and_config 2024-09-11T01:33:09.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.720+0000 7f90b5bbc640 10 monclient: build_initial_monmap 2024-09-11T01:33:09.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.720+0000 7f90b5bbc640 10 monclient: monmap: 2024-09-11T01:33:09.723 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:09.723 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:09.723 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:33:09.722392+0000 2024-09-11T01:33:09.723 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:33:09.722392+0000 2024-09-11T01:33:09.723 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:09.724 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:09.724 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:33:09.724 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:33:09.724 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:33:09.724 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:33:09.724 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:33:09.724 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:09.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.721+0000 7f90b5bbc640 1 Processor -- start 2024-09-11T01:33:09.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.721+0000 7f90b5bbc640 1 -- start start 2024-09-11T01:33:09.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.721+0000 7f90b5bbc640 10 monclient: init 2024-09-11T01:33:09.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.721+0000 7f90b5bbc640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:09.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.721+0000 7f90b5bbc640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-11T01:33:09.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.721+0000 7f90b5bbc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f90b00a2d40 addr v1:172.21.15.71:6789/0 2024-09-11T01:33:09.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90b5bbc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f90b006d260 addr v1:172.21.15.12:6789/0 2024-09-11T01:33:09.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90b5bbc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b006d670 0x7f90b0140c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:09.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90b5bbc640 10 monclient(hunting): picked mon.noname-b con 0x7f90b006d670 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:09.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90b5bbc640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90b009aed0 con 0x7f90b006d260 2024-09-11T01:33:09.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90aeffd640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7f90b006d260 legacy=0x7f90b013d4e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:33:09.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90b5bbc640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90b00a3920 con 0x7f90b00a2d40 2024-09-11T01:33:09.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90b5bbc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:09.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90b5bbc640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f90b006dd10 con 0x7f90b006d670 2024-09-11T01:33:09.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90b5bbc640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:09.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90b5bbc640 10 monclient(hunting): authenticate will time out at 996.165155s 2024-09-11T01:33:09.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90affff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b006d670 0x7f90b0140c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:09.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90affff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b006d670 0x7f90b0140c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52940/0 (socket says 172.21.15.12:52940) 2024-09-11T01:33:09.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90affff640 1 -- 172.21.15.12:0/1616343024 learned_addr learned my addr 172.21.15.12:0/1616343024 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:09.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90affff640 10 monclient(hunting): get_auth_request con 0x7f90b006d670 auth_method 0 2024-09-11T01:33:09.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:09.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90affff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:09.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:09.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:09.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:09.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:09.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90affff640 10 monclient(hunting): handle_auth_done global_id 44165 payload 274 2024-09-11T01:33:09.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90affff640 1 -- 172.21.15.12:0/1616343024 >> v1:172.21.15.71:6789/0 conn(0x7f90b00a2d40 legacy=0x7f90b006cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:09.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90affff640 1 -- 172.21.15.12:0/1616343024 >> v1:172.21.15.12:6789/0 conn(0x7f90b006d260 legacy=0x7f90b013d4e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:33:09.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90affff640 10 monclient: _finish_hunting 0 2024-09-11T01:33:09.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90affff640 1 monclient: found mon.noname-b 2024-09-11T01:33:09.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:09.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90affff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:33:09.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.722+0000 7f90affff640 1 -- 172.21.15.12:0/1616343024 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b00a3320 con 0x7f90b006d670 2024-09-11T01:33:09.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.725+0000 7f90affff640 1 -- 172.21.15.12:0/1616343024 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b006d670 msgr2=0x7f90b0140c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:09.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.725+0000 7f90affff640 1 --2- 172.21.15.12:0/1616343024 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b006d670 0x7f90b0140c10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f90b00700d0 tx=0x7f90a802a530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa086537123c3981 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:09.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.725+0000 7f90ae7fc640 1 -- 172.21.15.12:0/1616343024 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a8036070 con 0x7f90b006d670 2024-09-11T01:33:09.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.725+0000 7f90ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:09.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.725+0000 7f90ae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:33:09.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.725+0000 7f90ae7fc640 10 monclient: dump: 2024-09-11T01:33:09.732 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:09.732 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:09.732 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.732 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.732 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:09.732 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:09.732 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:09.732 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:09.732 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:09.732 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:09.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.725+0000 7f90ae7fc640 10 monclient: _finish_auth 0 2024-09-11T01:33:09.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.725+0000 7f90ae7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:33:09.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.725+0000 7f90ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:09.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.725+0000 7f90ae7fc640 1 -- 172.21.15.12:0/1616343024 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90a8007800 con 0x7f90b006d670 2024-09-11T01:33:09.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.725+0000 7f90ae7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:09.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.725+0000 7f90ae7fc640 1 -- 172.21.15.12:0/1616343024 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a8007bd0 con 0x7f90b006d670 2024-09-11T01:33:09.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.725+0000 7f90b5bbc640 5 monclient: authenticate success, global_id 44165 2024-09-11T01:33:09.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.725+0000 7f90b5bbc640 10 monclient: get_monmap_and_config success 2024-09-11T01:33:09.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.725+0000 7f90b5bbc640 4 set_mon_vals no callback set 2024-09-11T01:33:09.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.725+0000 7f90ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:09.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.725+0000 7f90ae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:09.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.725+0000 7f90ae7fc640 10 monclient: dump: 2024-09-11T01:33:09.733 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:09.733 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:09.733 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.733 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.733 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:09.734 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:09.734 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:09.734 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:09.734 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:09.734 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:09.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.725+0000 7f90b5bbc640 10 monclient: shutdown 2024-09-11T01:33:09.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.725+0000 7f90b5bbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:09.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.725+0000 7f90b5bbc640 1 -- 172.21.15.12:0/1616343024 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b006d670 msgr2=0x7f90b0140c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:09.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.725+0000 7f90b5bbc640 1 --2- 172.21.15.12:0/1616343024 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b006d670 0x7f90b0140c10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f90b00700d0 tx=0x7f90a802a530 comp rx=0 tx=0).stop 2024-09-11T01:33:09.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.726+0000 7f90b5bbc640 1 -- 172.21.15.12:0/1616343024 shutdown_connections 2024-09-11T01:33:09.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.726+0000 7f90b5bbc640 1 --2- 172.21.15.12:0/1616343024 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b006d670 0x7f90b0140c10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:09.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.726+0000 7f90b5bbc640 1 -- 172.21.15.12:0/1616343024 >> 172.21.15.12:0/1616343024 conn(0x7f90b00a3e30 msgr2=0x7f90b00a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:09.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.726+0000 7f90b5bbc640 1 -- 172.21.15.12:0/1616343024 shutdown_connections 2024-09-11T01:33:09.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.726+0000 7f90b5bbc640 1 -- 172.21.15.12:0/1616343024 wait complete. 2024-09-11T01:33:09.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.726+0000 7f90b5bbc640 10 monclient: build_initial_monmap 2024-09-11T01:33:09.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.726+0000 7f90b5bbc640 10 monclient: monmap: 2024-09-11T01:33:09.740 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:09.740 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:33:09.740 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:33:09.741 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:33:09.741 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:09.741 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:09.741 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:33:09.741 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:33:09.741 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:33:09.741 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:09.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.726+0000 7f90b5bbc640 1 Processor -- start 2024-09-11T01:33:09.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90b5bbc640 1 -- start start 2024-09-11T01:33:09.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90b5bbc640 10 monclient: init 2024-09-11T01:33:09.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90b5bbc640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:09.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90b5bbc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:33:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90b5bbc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f90b006d260 0x7f90b013cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90b5bbc640 10 monclient(hunting): picked mon.noname-b con 0x7f90b006d260 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:33:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90b5bbc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b006d670 0x7f90b013d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90b5bbc640 10 monclient(hunting): picked mon.noname-c con 0x7f90b006d670 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90b5bbc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b00a2d40 0x7f90b013d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90b5bbc640 10 monclient(hunting): picked mon.noname-a con 0x7f90b00a2d40 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:33:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90b5bbc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90b5bbc640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f90b00a3120 con 0x7f90b00a2d40 2024-09-11T01:33:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90b5bbc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90b5bbc640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f90b009bcf0 con 0x7f90b006d260 2024-09-11T01:33:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90b5bbc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90b5bbc640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f90b009ba10 con 0x7f90b006d670 2024-09-11T01:33:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90b5bbc640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:09.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90b5bbc640 10 monclient(hunting): authenticate will time out at 996.170730s 2024-09-11T01:33:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90affff640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b00a2d40 msgr2=0x7f90b013d640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:33:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90affff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b00a2d40 0x7f90b013d640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:33:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90aeffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b006d670 0x7f90b013d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90aeffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b006d670 0x7f90b013d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52956/0 (socket says 172.21.15.12:52956) 2024-09-11T01:33:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90aeffd640 1 -- 172.21.15.12:0/200004811 learned_addr learned my addr 172.21.15.12:0/200004811 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90aeffd640 10 monclient(hunting): get_auth_request con 0x7f90b006d670 auth_method 0 2024-09-11T01:33:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:09.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90af7fe640 1 --2- 172.21.15.12:0/200004811 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f90b006d260 0x7f90b013cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90af7fe640 10 monclient(hunting): get_auth_request con 0x7f90b006d260 auth_method 0 2024-09-11T01:33:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.727+0000 7f90af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.728+0000 7f90af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.728+0000 7f90af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.728+0000 7f90af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.728+0000 7f90aeffd640 10 monclient(hunting): handle_auth_done global_id 44168 payload 824 2024-09-11T01:33:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.728+0000 7f90aeffd640 1 -- 172.21.15.12:0/200004811 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f90b006d260 msgr2=0x7f90b013cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.728+0000 7f90aeffd640 1 --2- 172.21.15.12:0/200004811 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f90b006d260 0x7f90b013cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.728+0000 7f90aeffd640 1 -- 172.21.15.12:0/200004811 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b00a2d40 msgr2=0x7f90b013d640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:33:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.728+0000 7f90aeffd640 1 --2- 172.21.15.12:0/200004811 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b00a2d40 0x7f90b013d640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:09.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.728+0000 7f90aeffd640 10 monclient: _finish_hunting 0 2024-09-11T01:33:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.728+0000 7f90aeffd640 1 monclient: found mon.noname-c 2024-09-11T01:33:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.728+0000 7f90aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.728+0000 7f90aeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:33:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.728+0000 7f90aeffd640 1 -- 172.21.15.12:0/200004811 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b01409e0 con 0x7f90b006d670 2024-09-11T01:33:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.730+0000 7f90aeffd640 1 -- 172.21.15.12:0/200004811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b006d670 msgr2=0x7f90b013d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.730+0000 7f90aeffd640 1 --2- 172.21.15.12:0/200004811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b006d670 0x7f90b013d100 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f90a400b580 tx=0x7f90a400ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.730+0000 7f9097fff640 1 -- 172.21.15.12:0/200004811 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a40050f0 con 0x7f90b006d670 2024-09-11T01:33:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.730+0000 7f9097fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.730+0000 7f9097fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:33:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.730+0000 7f9097fff640 10 monclient: dump: 2024-09-11T01:33:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:09.745 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:09.746 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.730+0000 7f9097fff640 10 monclient: _finish_auth 0 2024-09-11T01:33:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.730+0000 7f9097fff640 10 monclient: _check_auth_tickets 2024-09-11T01:33:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.730+0000 7f9097fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.730+0000 7f9097fff640 1 -- 172.21.15.12:0/200004811 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90a4015070 con 0x7f90b006d670 2024-09-11T01:33:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.730+0000 7f9097fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.730+0000 7f9097fff640 1 -- 172.21.15.12:0/200004811 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a40054d0 con 0x7f90b006d670 2024-09-11T01:33:09.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.730+0000 7f90b5bbc640 5 monclient: authenticate success, global_id 44168 2024-09-11T01:33:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.730+0000 7f90b5bbc640 10 monclient: _renew_subs 2024-09-11T01:33:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.730+0000 7f90b5bbc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.730+0000 7f90b5bbc640 1 -- 172.21.15.12:0/200004811 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90b00a6980 con 0x7f90b006d670 2024-09-11T01:33:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.730+0000 7f9097fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.730+0000 7f9097fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.730+0000 7f9097fff640 10 monclient: dump: 2024-09-11T01:33:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.747 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:09.748 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.730+0000 7f90adffb640 4 set_mon_vals no callback set 2024-09-11T01:33:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.730+0000 7f90b5bbc640 10 monclient: _renew_subs 2024-09-11T01:33:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.730+0000 7f90b5bbc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.730+0000 7f90b5bbc640 1 -- 172.21.15.12:0/200004811 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f90b013e250 con 0x7f90b006d670 2024-09-11T01:33:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.731+0000 7f90b5bbc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.731+0000 7f90b5bbc640 1 --2- 172.21.15.12:0/200004811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b01384b0 0x7f90b0138890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.731+0000 7f90b5bbc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:09.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.731+0000 7f90b5bbc640 1 -- 172.21.15.12:0/200004811 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f90b006cb50 con 0x7f90b01384b0 2024-09-11T01:33:09.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.731+0000 7f9097fff640 1 -- 172.21.15.12:0/200004811 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f90a40056b0 con 0x7f90b006d670 2024-09-11T01:33:09.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.731+0000 7f90af7fe640 1 --2- 172.21.15.12:0/200004811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b01384b0 0x7f90b0138890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:09.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.731+0000 7f90af7fe640 10 monclient: get_auth_request con 0x7f90b01384b0 auth_method 0 2024-09-11T01:33:09.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.731+0000 7f90af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:09.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.731+0000 7f90af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:09.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.731+0000 7f90af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:09.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.731+0000 7f90af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:09.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.731+0000 7f90af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:09.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.731+0000 7f90af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:09.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.731+0000 7f90af7fe640 10 monclient(hunting): handle_auth_done global_id 44171 payload 824 2024-09-11T01:33:09.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.731+0000 7f9097fff640 1 --2- 172.21.15.12:0/200004811 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f909c03e240 0x7f909c040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:09.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.731+0000 7f90af7fe640 1 -- 172.21.15.12:0/200004811 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f90b009e520 con 0x7f90b01384b0 2024-09-11T01:33:09.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.731+0000 7f9097fff640 1 -- 172.21.15.12:0/200004811 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(140..140 src has 1..140) ==== 4755+0+0 (secure 0 0 0) 0x7f90a4052af0 con 0x7f90b006d670 2024-09-11T01:33:09.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.732+0000 7f90affff640 1 --2- 172.21.15.12:0/200004811 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f909c03e240 0x7f909c040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:09.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.732+0000 7f90affff640 10 monclient: get_auth_request con 0x7f909c03e240 auth_method 0 2024-09-11T01:33:09.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.733+0000 7f90af7fe640 1 -- 172.21.15.12:0/200004811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b01384b0 msgr2=0x7f90b0138890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:09.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.733+0000 7f90af7fe640 1 --2- 172.21.15.12:0/200004811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b01384b0 0x7f90b0138890 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f90a000f7a0 tx=0x7f90a000fc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:09.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.734+0000 7f9097fff640 1 -- 172.21.15.12:0/200004811 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a0018030 con 0x7f90b01384b0 2024-09-11T01:33:09.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.734+0000 7f9097fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:09.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.734+0000 7f9097fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:09.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.734+0000 7f9097fff640 10 monclient: dump: 2024-09-11T01:33:09.750 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:09.750 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:09.750 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.750 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.751 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:09.751 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:09.751 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:09.751 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:09.751 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:09.751 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:09.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.734+0000 7f9097fff640 1 -- 172.21.15.12:0/200004811 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f90a000d9b0 con 0x7f90b01384b0 2024-09-11T01:33:09.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.734+0000 7f9097fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:09.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.734+0000 7f9097fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:33:09.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.734+0000 7f9097fff640 1 -- 172.21.15.12:0/200004811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b01384b0 msgr2=0x7f90b0138890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:09.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.734+0000 7f9097fff640 1 --2- 172.21.15.12:0/200004811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b01384b0 0x7f90b0138890 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f90a000f7a0 tx=0x7f90a000fc60 comp rx=0 tx=0).stop 2024-09-11T01:33:09.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.734+0000 7f9097fff640 1 -- 172.21.15.12:0/200004811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b01384b0 msgr2=0x7f90b0138890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:33:09.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.734+0000 7f9097fff640 1 --2- 172.21.15.12:0/200004811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b01384b0 0x7f90b0138890 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f90a000f7a0 tx=0x7f90a000fc60 comp rx=0 tx=0).stop 2024-09-11T01:33:09.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.734+0000 7f90affff640 1 --2- 172.21.15.12:0/200004811 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f909c03e240 0x7f909c040700 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f90b006a070 tx=0x7f90a8009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:09.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.740+0000 7f90b5bbc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:33:09.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.740+0000 7f90b5bbc640 1 --2- 172.21.15.12:0/200004811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b0138dd0 0x7f90b0070f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:09.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.740+0000 7f90b5bbc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:09.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.740+0000 7f90b5bbc640 1 -- 172.21.15.12:0/200004811 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f90b01391e0 con 0x7f90b0138dd0 2024-09-11T01:33:09.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.740+0000 7f90af7fe640 1 --2- 172.21.15.12:0/200004811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b0138dd0 0x7f90b0070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:09.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.740+0000 7f90af7fe640 10 monclient: get_auth_request con 0x7f90b0138dd0 auth_method 0 2024-09-11T01:33:09.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.740+0000 7f90af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:09.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.740+0000 7f90af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:09.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.740+0000 7f90af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:09.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.740+0000 7f90af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:09.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.740+0000 7f90af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:09.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.740+0000 7f90af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:09.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.740+0000 7f90af7fe640 10 monclient(hunting): handle_auth_done global_id 44174 payload 824 2024-09-11T01:33:09.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.740+0000 7f90af7fe640 1 -- 172.21.15.12:0/200004811 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f90b0074430 con 0x7f90b0138dd0 2024-09-11T01:33:09.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.743+0000 7f90af7fe640 1 -- 172.21.15.12:0/200004811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b0138dd0 msgr2=0x7f90b0070f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:09.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.743+0000 7f90af7fe640 1 --2- 172.21.15.12:0/200004811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b0138dd0 0x7f90b0070f30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f90a000f7a0 tx=0x7f90a0003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:09.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.743+0000 7f9097fff640 1 -- 172.21.15.12:0/200004811 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a001dbe0 con 0x7f90b0138dd0 2024-09-11T01:33:09.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.743+0000 7f9097fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:09.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.743+0000 7f9097fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:09.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.743+0000 7f9097fff640 10 monclient: dump: 2024-09-11T01:33:09.755 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:09.755 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:09.755 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.755 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.755 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:09.755 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:09.755 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:09.755 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:09.755 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:09.755 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:09.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.743+0000 7f9097fff640 1 -- 172.21.15.12:0/200004811 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4863 (secure 0 0 0) 0x7f90a001dbe0 con 0x7f90b0138dd0 2024-09-11T01:33:09.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.743+0000 7f9097fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:33:09.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.743+0000 7f9097fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:33:09.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.743+0000 7f9097fff640 1 -- 172.21.15.12:0/200004811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b0138dd0 msgr2=0x7f90b0070f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:09.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.743+0000 7f9097fff640 1 --2- 172.21.15.12:0/200004811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b0138dd0 0x7f90b0070f30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f90a000f7a0 tx=0x7f90a0003760 comp rx=0 tx=0).stop 2024-09-11T01:33:09.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.743+0000 7f9097fff640 1 -- 172.21.15.12:0/200004811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b0138dd0 msgr2=0x7f90b0070f30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:33:09.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.743+0000 7f9097fff640 1 --2- 172.21.15.12:0/200004811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b0138dd0 0x7f90b0070f30 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f90a000f7a0 tx=0x7f90a0003760 comp rx=0 tx=0).stop 2024-09-11T01:33:09.756 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:33:09.756 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:33:09.757 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:33:09.757 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:33:09.757 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 40, 2024-09-11T01:33:09.757 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 33688, 2024-09-11T01:33:09.757 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:33:09.757 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:33:09.757 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:33:09.757 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:09.757 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:33:09.757 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:33:09.757 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:33:09.757 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:33:09.757 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:33:09.758 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:33:09.758 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:33:09.758 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:33:09.758 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:33:09.758 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:33:09.758 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:33:09.758 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:33:09.758 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:33:09.758 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:33:09.758 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:33:09.758 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:09.758 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:33:09.758 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:33:09.758 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:33:09.759 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:33:09.759 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:33:09.759 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:33:09.759 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:33:09.759 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:33:09.759 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:33:09.759 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:33:09.759 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:33:09.759 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:33:09.759 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:33:09.759 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:09.759 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:09.759 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:33:09.759 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:33:09.760 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:33:09.760 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:33:09.760 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:33:09.760 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:33:09.760 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:33:09.760 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:33:09.760 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:33:09.760 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:33:09.760 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:33:09.760 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:33:09.760 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:33:09.760 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:33:09.760 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:33:09.761 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:33:09.761 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:33:09.761 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:33:09.761 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:33:09.761 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:33:09.761 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:33:09.761 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:33:09.761 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:33:09.761 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:33:09.761 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:33:09.761 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:33:09.761 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:33:09.762 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:33:09.762 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:09.762 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:33:09.762 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:09.762 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:33:09.762 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:09.762 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:33:09.762 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:33:09.762 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:33:09.762 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:33:09.762 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:09.762 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:33:09.762 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:33:09.762 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:09.762 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:09.762 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:09.763 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:33:09.763 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:33:09.763 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:09.763 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:09.763 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:09.763 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:09.763 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:33:09.763 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:33:09.763 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:33:09.763 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:33:09.763 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:33:09.763 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:09.763 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:09.763 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:33:09.764 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:33:09.764 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:33:09.764 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:33:09.764 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:09.764 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:33:09.764 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:33:09.764 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:09.764 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:09.764 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:09.764 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:33:09.764 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:33:09.764 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:09.764 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:09.764 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:09.765 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:09.765 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:33:09.765 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:33:09.765 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:33:09.765 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:33:09.765 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:33:09.765 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:09.765 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:09.765 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:33:09.765 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:33:09.765 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:33:09.765 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:33:09.765 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:09.765 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:33:09.765 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:33:09.766 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:09.766 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:09.766 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:09.766 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:33:09.766 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:33:09.766 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:09.766 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:09.766 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:09.766 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:09.766 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:33:09.766 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:33:09.766 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:33:09.766 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:33:09.766 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:33:09.766 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:09.767 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:09.767 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:09.767 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:33:09.767 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:33:09.767 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:09.767 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:33:09.767 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:33:09.767 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:33:09.767 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:09.767 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:09.767 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:33:09.767 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:09.767 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:33:09.767 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:33:09.767 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:33:09.767 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:09.768 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:09.768 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:33:09.768 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:09.768 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:33:09.768 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:33:09.768 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:33:09.768 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:09.768 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:09.768 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:33:09.768 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:09.768 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:33:09.768 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:33:09.769 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:33:09.769 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:09.769 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:09.769 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:09.769 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:33:09.769 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:33:09.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.743+0000 7f90b5bbc640 1 -- 172.21.15.12:0/200004811 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f909c03e240 msgr2=0x7f909c040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:09.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.743+0000 7f90b5bbc640 1 --2- 172.21.15.12:0/200004811 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f909c03e240 0x7f909c040700 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f90b006a070 tx=0x7f90a8009000 comp rx=0 tx=0).stop 2024-09-11T01:33:09.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.743+0000 7f90b5bbc640 10 monclient: shutdown 2024-09-11T01:33:09.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.743+0000 7f90b5bbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:09.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.743+0000 7f90b5bbc640 1 -- 172.21.15.12:0/200004811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b006d670 msgr2=0x7f90b013d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:09.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.743+0000 7f90b5bbc640 1 --2- 172.21.15.12:0/200004811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b006d670 0x7f90b013d100 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f90a400b580 tx=0x7f90a400ba40 comp rx=0 tx=0).stop 2024-09-11T01:33:09.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.743+0000 7f90affff640 1 -- 172.21.15.12:0/200004811 reap_dead start 2024-09-11T01:33:09.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.743+0000 7f90b5bbc640 1 -- 172.21.15.12:0/200004811 shutdown_connections 2024-09-11T01:33:09.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.743+0000 7f90b5bbc640 1 --2- 172.21.15.12:0/200004811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b006d670 0x7f90b013d100 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:09.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.743+0000 7f90b5bbc640 1 -- 172.21.15.12:0/200004811 >> 172.21.15.12:0/200004811 conn(0x7f90b00a3e30 msgr2=0x7f90b00a5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:09.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.744+0000 7f90b5bbc640 1 -- 172.21.15.12:0/200004811 shutdown_connections 2024-09-11T01:33:09.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.744+0000 7f90b5bbc640 1 -- 172.21.15.12:0/200004811 wait complete. 2024-09-11T01:33:09.771 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:33:09.771 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T01:33:09.771 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:33:09.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.871+0000 7ff632fa1640 10 monclient: get_monmap_and_config 2024-09-11T01:33:09.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.871+0000 7ff632fa1640 10 monclient: build_initial_monmap 2024-09-11T01:33:09.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.871+0000 7ff632fa1640 10 monclient: monmap: 2024-09-11T01:33:09.874 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:09.874 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:09.874 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:33:09.873000+0000 2024-09-11T01:33:09.874 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:33:09.873000+0000 2024-09-11T01:33:09.874 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:09.874 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:09.874 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:33:09.874 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:33:09.874 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:33:09.875 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:33:09.875 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:33:09.875 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:09.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.871+0000 7ff632fa1640 1 Processor -- start 2024-09-11T01:33:09.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff632fa1640 1 -- start start 2024-09-11T01:33:09.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff632fa1640 10 monclient: init 2024-09-11T01:33:09.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff632fa1640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:09.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff632fa1640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-11T01:33:09.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff632fa1640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff62c137f50 0x7ff62c138330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:09.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff632fa1640 10 monclient(hunting): picked mon.noname-a con 0x7ff62c137f50 addr v2:172.21.15.12:3300/0 2024-09-11T01:33:09.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff632fa1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff62c138870 addr v1:172.21.15.12:6789/0 2024-09-11T01:33:09.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff632fa1640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c0a6550 0x7ff62c0a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:09.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff632fa1640 10 monclient(hunting): picked mon.noname-b con 0x7ff62c0a6550 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:09.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff631f9f640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7ff62c137f50 msgr2=0x7ff62c138330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:33:09.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff632fa1640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff62c09c4c0 con 0x7ff62c138870 2024-09-11T01:33:09.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff632fa1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:09.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff632fa1640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff62c09c1e0 con 0x7ff62c137f50 2024-09-11T01:33:09.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff63179e640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7ff62c138870 legacy=0x7ff62c0a4dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:33:09.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff631f9f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff62c137f50 0x7ff62c138330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:33:09.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff632fa1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:09.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff632fa1640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff62c09c7a0 con 0x7ff62c0a6550 2024-09-11T01:33:09.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff632fa1640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:09.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff632fa1640 10 monclient(hunting): authenticate will time out at 996.315684s 2024-09-11T01:33:09.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff6327a0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c0a6550 0x7ff62c0a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:09.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff6327a0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c0a6550 0x7ff62c0a54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52988/0 (socket says 172.21.15.12:52988) 2024-09-11T01:33:09.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff6327a0640 1 -- 172.21.15.12:0/1791783554 learned_addr learned my addr 172.21.15.12:0/1791783554 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:09.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff6327a0640 10 monclient(hunting): get_auth_request con 0x7ff62c0a6550 auth_method 0 2024-09-11T01:33:09.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff6327a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:09.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff6327a0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:09.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff6327a0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:09.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff6327a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:09.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff6327a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:09.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.872+0000 7ff6327a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:09.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.873+0000 7ff6327a0640 10 monclient(hunting): handle_auth_done global_id 44177 payload 274 2024-09-11T01:33:09.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.873+0000 7ff6327a0640 1 -- 172.21.15.12:0/1791783554 >> v2:172.21.15.12:3300/0 conn(0x7ff62c137f50 msgr2=0x7ff62c138330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:33:09.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.873+0000 7ff6327a0640 1 --2- 172.21.15.12:0/1791783554 >> v2:172.21.15.12:3300/0 conn(0x7ff62c137f50 0x7ff62c138330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:09.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.873+0000 7ff6327a0640 1 -- 172.21.15.12:0/1791783554 >> v1:172.21.15.12:6789/0 conn(0x7ff62c138870 legacy=0x7ff62c0a4dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:33:09.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.873+0000 7ff6327a0640 10 monclient: _finish_hunting 0 2024-09-11T01:33:09.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.873+0000 7ff6327a0640 1 monclient: found mon.noname-b 2024-09-11T01:33:09.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.873+0000 7ff6327a0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:09.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.873+0000 7ff6327a0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:33:09.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.873+0000 7ff6327a0640 1 -- 172.21.15.12:0/1791783554 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff62c09ba20 con 0x7ff62c0a6550 2024-09-11T01:33:09.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.875+0000 7ff6327a0640 1 -- 172.21.15.12:0/1791783554 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c0a6550 msgr2=0x7ff62c0a54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:09.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.875+0000 7ff6327a0640 1 --2- 172.21.15.12:0/1791783554 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c0a6550 0x7ff62c0a54e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff624009e50 tx=0x7ff62402a570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69cf46e1273154d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:09.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff630f9d640 1 -- 172.21.15.12:0/1791783554 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff624036070 con 0x7ff62c0a6550 2024-09-11T01:33:09.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff630f9d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:09.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff630f9d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:33:09.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff630f9d640 10 monclient: dump: 2024-09-11T01:33:09.889 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:09.889 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:09.889 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.889 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.889 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:09.890 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:09.890 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:09.890 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:09.890 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:09.890 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:09.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff630f9d640 10 monclient: _finish_auth 0 2024-09-11T01:33:09.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff630f9d640 10 monclient: _check_auth_tickets 2024-09-11T01:33:09.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff630f9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:09.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff632fa1640 5 monclient: authenticate success, global_id 44177 2024-09-11T01:33:09.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff632fa1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:33:09.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff630f9d640 1 -- 172.21.15.12:0/1791783554 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff624007800 con 0x7ff62c0a6550 2024-09-11T01:33:09.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff630f9d640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:09.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff632fa1640 10 monclient: get_monmap_and_config success 2024-09-11T01:33:09.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff632fa1640 4 set_mon_vals no callback set 2024-09-11T01:33:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff630f9d640 1 -- 172.21.15.12:0/1791783554 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff624007bd0 con 0x7ff62c0a6550 2024-09-11T01:33:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff630f9d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff630f9d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff630f9d640 10 monclient: dump: 2024-09-11T01:33:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:09.891 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:09.891 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:09.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff632fa1640 10 monclient: shutdown 2024-09-11T01:33:09.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff632fa1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:09.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff632fa1640 1 -- 172.21.15.12:0/1791783554 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c0a6550 msgr2=0x7ff62c0a54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:09.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff632fa1640 1 --2- 172.21.15.12:0/1791783554 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c0a6550 0x7ff62c0a54e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff624009e50 tx=0x7ff62402a570 comp rx=0 tx=0).stop 2024-09-11T01:33:09.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff632fa1640 1 -- 172.21.15.12:0/1791783554 shutdown_connections 2024-09-11T01:33:09.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff632fa1640 1 --2- 172.21.15.12:0/1791783554 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c0a6550 0x7ff62c0a54e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:09.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff632fa1640 1 --2- 172.21.15.12:0/1791783554 >> v2:172.21.15.12:3300/0 conn(0x7ff62c137f50 0x7ff62c138330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:09.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff632fa1640 1 -- 172.21.15.12:0/1791783554 >> 172.21.15.12:0/1791783554 conn(0x7ff62c0a6ed0 msgr2=0x7ff62c0a72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:09.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff632fa1640 1 -- 172.21.15.12:0/1791783554 shutdown_connections 2024-09-11T01:33:09.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.876+0000 7ff632fa1640 1 -- 172.21.15.12:0/1791783554 wait complete. 2024-09-11T01:33:09.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.877+0000 7ff632fa1640 10 monclient: build_initial_monmap 2024-09-11T01:33:09.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.877+0000 7ff632fa1640 10 monclient: monmap: 2024-09-11T01:33:09.892 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:33:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:33:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:33:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:33:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:33:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:33:09.893 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.877+0000 7ff632fa1640 1 Processor -- start 2024-09-11T01:33:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.877+0000 7ff632fa1640 1 -- start start 2024-09-11T01:33:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.877+0000 7ff632fa1640 10 monclient: init 2024-09-11T01:33:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff632fa1640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff632fa1640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:33:09.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff632fa1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff62c0a6550 0x7ff62c136620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:09.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff632fa1640 10 monclient(hunting): picked mon.noname-b con 0x7ff62c0a6550 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:33:09.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff632fa1640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c137f50 0x7ff62c136b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:09.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff632fa1640 10 monclient(hunting): picked mon.noname-c con 0x7ff62c137f50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:09.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff632fa1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c138870 0x7ff62c13cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:09.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff632fa1640 10 monclient(hunting): picked mon.noname-a con 0x7ff62c138870 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:33:09.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff632fa1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:09.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff632fa1640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff62c0a5a20 con 0x7ff62c138870 2024-09-11T01:33:09.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff632fa1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:09.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff632fa1640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff62c0a5ba0 con 0x7ff62c0a6550 2024-09-11T01:33:09.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff632fa1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:09.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff632fa1640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff62c13fed0 con 0x7ff62c137f50 2024-09-11T01:33:09.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff632fa1640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:09.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff632fa1640 10 monclient(hunting): authenticate will time out at 996.321272s 2024-09-11T01:33:09.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff6327a0640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c138870 msgr2=0x7ff62c13cb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:33:09.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff6327a0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c138870 0x7ff62c13cb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:33:09.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff63179e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c137f50 0x7ff62c136b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:09.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff63179e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c137f50 0x7ff62c136b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52994/0 (socket says 172.21.15.12:52994) 2024-09-11T01:33:09.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff63179e640 1 -- 172.21.15.12:0/1088423352 learned_addr learned my addr 172.21.15.12:0/1088423352 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:09.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff63179e640 10 monclient(hunting): get_auth_request con 0x7ff62c137f50 auth_method 0 2024-09-11T01:33:09.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff63179e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:09.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff63179e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:09.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff63179e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff63179e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff63179e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff63179e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff631f9f640 1 --2- 172.21.15.12:0/1088423352 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff62c0a6550 0x7ff62c136620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff631f9f640 10 monclient(hunting): get_auth_request con 0x7ff62c0a6550 auth_method 0 2024-09-11T01:33:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff631f9f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff631f9f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff631f9f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff63179e640 10 monclient(hunting): handle_auth_done global_id 44180 payload 824 2024-09-11T01:33:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff63179e640 1 -- 172.21.15.12:0/1088423352 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff62c0a6550 msgr2=0x7ff62c136620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff63179e640 1 --2- 172.21.15.12:0/1088423352 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff62c0a6550 0x7ff62c136620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff63179e640 1 -- 172.21.15.12:0/1088423352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c138870 msgr2=0x7ff62c13cb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:33:09.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff63179e640 1 --2- 172.21.15.12:0/1088423352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c138870 0x7ff62c13cb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:09.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff63179e640 10 monclient: _finish_hunting 0 2024-09-11T01:33:09.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff63179e640 1 monclient: found mon.noname-c 2024-09-11T01:33:09.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff63179e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:09.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff63179e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:33:09.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.878+0000 7ff63179e640 1 -- 172.21.15.12:0/1088423352 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff62c13f320 con 0x7ff62c137f50 2024-09-11T01:33:09.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.880+0000 7ff63179e640 1 -- 172.21.15.12:0/1088423352 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c137f50 msgr2=0x7ff62c136b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:09.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.880+0000 7ff63179e640 1 --2- 172.21.15.12:0/1088423352 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c137f50 0x7ff62c136b60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff62000b550 tx=0x7ff62000ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:09.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.881+0000 7ff61e7fc640 1 -- 172.21.15.12:0/1088423352 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff620005070 con 0x7ff62c137f50 2024-09-11T01:33:09.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.881+0000 7ff61e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:09.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.881+0000 7ff61e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:33:09.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.881+0000 7ff61e7fc640 10 monclient: dump: 2024-09-11T01:33:09.897 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:09.897 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:09.898 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.881+0000 7ff61e7fc640 10 monclient: _finish_auth 0 2024-09-11T01:33:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.881+0000 7ff61e7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:33:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.881+0000 7ff61e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.881+0000 7ff61e7fc640 1 -- 172.21.15.12:0/1088423352 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff620015070 con 0x7ff62c137f50 2024-09-11T01:33:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.881+0000 7ff632fa1640 5 monclient: authenticate success, global_id 44180 2024-09-11T01:33:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.881+0000 7ff632fa1640 10 monclient: _renew_subs 2024-09-11T01:33:09.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.881+0000 7ff632fa1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:09.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.881+0000 7ff632fa1640 1 -- 172.21.15.12:0/1088423352 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff62c1214f0 con 0x7ff62c137f50 2024-09-11T01:33:09.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.881+0000 7ff61e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:09.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.881+0000 7ff61e7fc640 1 -- 172.21.15.12:0/1088423352 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff620005450 con 0x7ff62c137f50 2024-09-11T01:33:09.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.881+0000 7ff61e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:09.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.881+0000 7ff61e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:09.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.881+0000 7ff61e7fc640 10 monclient: dump: 2024-09-11T01:33:09.899 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:09.899 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:09.899 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.899 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.899 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:09.899 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:09.899 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:09.900 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:09.900 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:09.900 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:09.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.881+0000 7ff632fa1640 10 monclient: _renew_subs 2024-09-11T01:33:09.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.881+0000 7ff61f7fe640 4 set_mon_vals no callback set 2024-09-11T01:33:09.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.881+0000 7ff632fa1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:09.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.881+0000 7ff632fa1640 1 -- 172.21.15.12:0/1088423352 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff62c13e4f0 con 0x7ff62c137f50 2024-09-11T01:33:09.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.881+0000 7ff61e7fc640 1 -- 172.21.15.12:0/1088423352 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff620005630 con 0x7ff62c137f50 2024-09-11T01:33:09.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.881+0000 7ff632fa1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:09.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.882+0000 7ff632fa1640 1 --2- 172.21.15.12:0/1088423352 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff62c13d060 0x7ff62c13d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:09.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.882+0000 7ff632fa1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:09.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.882+0000 7ff632fa1640 1 -- 172.21.15.12:0/1088423352 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff62c06f7c0 con 0x7ff62c13d060 2024-09-11T01:33:09.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.882+0000 7ff61e7fc640 1 --2- 172.21.15.12:0/1088423352 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff61003e1c0 0x7ff610040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:09.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.882+0000 7ff61e7fc640 1 -- 172.21.15.12:0/1088423352 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(140..140 src has 1..140) ==== 4755+0+0 (secure 0 0 0) 0x7ff620052b90 con 0x7ff62c137f50 2024-09-11T01:33:09.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.882+0000 7ff631f9f640 1 --2- 172.21.15.12:0/1088423352 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff62c13d060 0x7ff62c13d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:09.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.882+0000 7ff631f9f640 10 monclient: get_auth_request con 0x7ff62c13d060 auth_method 0 2024-09-11T01:33:09.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.882+0000 7ff631f9f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:09.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.882+0000 7ff631f9f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:09.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.882+0000 7ff631f9f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:09.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.882+0000 7ff6327a0640 1 --2- 172.21.15.12:0/1088423352 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff61003e1c0 0x7ff610040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:09.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.882+0000 7ff6327a0640 10 monclient: get_auth_request con 0x7ff61003e1c0 auth_method 0 2024-09-11T01:33:09.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.882+0000 7ff631f9f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:09.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.882+0000 7ff631f9f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:09.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.882+0000 7ff631f9f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:09.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.882+0000 7ff631f9f640 10 monclient(hunting): handle_auth_done global_id 34258 payload 824 2024-09-11T01:33:09.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.882+0000 7ff631f9f640 1 -- 172.21.15.12:0/1088423352 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff62c138330 con 0x7ff62c13d060 2024-09-11T01:33:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.885+0000 7ff6327a0640 1 --2- 172.21.15.12:0/1088423352 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff61003e1c0 0x7ff610040680 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff62c0a4a90 tx=0x7ff624009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.885+0000 7ff631f9f640 1 -- 172.21.15.12:0/1088423352 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff62c13d060 msgr2=0x7ff62c13d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.885+0000 7ff631f9f640 1 --2- 172.21.15.12:0/1088423352 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff62c13d060 0x7ff62c13d440 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff62800d700 tx=0x7ff62800dbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.885+0000 7ff61e7fc640 1 -- 172.21.15.12:0/1088423352 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff628003bc0 con 0x7ff62c13d060 2024-09-11T01:33:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.885+0000 7ff61e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.885+0000 7ff61e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:33:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.885+0000 7ff61e7fc640 10 monclient: dump: 2024-09-11T01:33:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:09.903 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.885+0000 7ff61e7fc640 1 -- 172.21.15.12:0/1088423352 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff628003bc0 con 0x7ff62c13d060 2024-09-11T01:33:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.885+0000 7ff61e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.885+0000 7ff61e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:33:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.885+0000 7ff61e7fc640 1 -- 172.21.15.12:0/1088423352 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff62c13d060 msgr2=0x7ff62c13d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.885+0000 7ff61e7fc640 1 --2- 172.21.15.12:0/1088423352 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff62c13d060 0x7ff62c13d440 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff62800d700 tx=0x7ff62800dbc0 comp rx=0 tx=0).stop 2024-09-11T01:33:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.885+0000 7ff61e7fc640 1 -- 172.21.15.12:0/1088423352 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff62c13d060 msgr2=0x7ff62c13d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:33:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.885+0000 7ff61e7fc640 1 --2- 172.21.15.12:0/1088423352 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff62c13d060 0x7ff62c13d440 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff62800d700 tx=0x7ff62800dbc0 comp rx=0 tx=0).stop 2024-09-11T01:33:09.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.891+0000 7ff632fa1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:33:09.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.891+0000 7ff632fa1640 1 --2- 172.21.15.12:0/1088423352 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff62c13d980 0x7ff62c138c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:09.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.891+0000 7ff632fa1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:09.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.891+0000 7ff632fa1640 1 -- 172.21.15.12:0/1088423352 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff62c13f050 con 0x7ff62c13d980 2024-09-11T01:33:09.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.892+0000 7ff631f9f640 1 --2- 172.21.15.12:0/1088423352 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff62c13d980 0x7ff62c138c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:09.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.892+0000 7ff631f9f640 10 monclient: get_auth_request con 0x7ff62c13d980 auth_method 0 2024-09-11T01:33:09.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.892+0000 7ff631f9f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:09.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.892+0000 7ff631f9f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:09.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.892+0000 7ff631f9f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:09.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.892+0000 7ff631f9f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:09.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.892+0000 7ff631f9f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:09.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.892+0000 7ff631f9f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:09.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.892+0000 7ff631f9f640 10 monclient(hunting): handle_auth_done global_id 34261 payload 824 2024-09-11T01:33:09.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.892+0000 7ff631f9f640 1 -- 172.21.15.12:0/1088423352 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff62c121150 con 0x7ff62c13d980 2024-09-11T01:33:09.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.895+0000 7ff631f9f640 1 -- 172.21.15.12:0/1088423352 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff62c13d980 msgr2=0x7ff62c138c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:09.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.895+0000 7ff631f9f640 1 --2- 172.21.15.12:0/1088423352 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff62c13d980 0x7ff62c138c80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff62800d700 tx=0x7ff6280059d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:09.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.895+0000 7ff61e7fc640 1 -- 172.21.15.12:0/1088423352 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62800bd50 con 0x7ff62c13d980 2024-09-11T01:33:09.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.895+0000 7ff61e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:09.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.895+0000 7ff61e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:33:09.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.895+0000 7ff61e7fc640 10 monclient: dump: 2024-09-11T01:33:09.907 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:09.907 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:09.907 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.907 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:09.907 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:09.907 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:09.907 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:09.907 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:09.907 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:09.907 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:09.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.895+0000 7ff61e7fc640 1 -- 172.21.15.12:0/1088423352 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4698 (secure 0 0 0) 0x7ff62800bd50 con 0x7ff62c13d980 2024-09-11T01:33:09.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.895+0000 7ff61e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:33:09.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.895+0000 7ff61e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:33:09.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.895+0000 7ff61e7fc640 1 -- 172.21.15.12:0/1088423352 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff62c13d980 msgr2=0x7ff62c138c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:09.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.895+0000 7ff61e7fc640 1 --2- 172.21.15.12:0/1088423352 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff62c13d980 0x7ff62c138c80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff62800d700 tx=0x7ff6280059d0 comp rx=0 tx=0).stop 2024-09-11T01:33:09.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.895+0000 7ff61e7fc640 1 -- 172.21.15.12:0/1088423352 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff62c13d980 msgr2=0x7ff62c138c80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:33:09.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.895+0000 7ff61e7fc640 1 --2- 172.21.15.12:0/1088423352 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff62c13d980 0x7ff62c138c80 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff62800d700 tx=0x7ff6280059d0 comp rx=0 tx=0).stop 2024-09-11T01:33:09.908 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:33:09.908 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:33:09.908 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:33:09.909 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:33:09.909 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 40, 2024-09-11T01:33:09.909 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 83110, 2024-09-11T01:33:09.909 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:33:09.909 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:33:09.909 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:33:09.909 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:09.909 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:33:09.909 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:33:09.909 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:33:09.909 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:33:09.909 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:33:09.909 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:33:09.910 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:33:09.910 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:33:09.910 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:33:09.910 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:33:09.910 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:33:09.910 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:33:09.910 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:33:09.910 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:33:09.910 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:33:09.910 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:09.910 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:33:09.910 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:33:09.910 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:33:09.910 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:33:09.911 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:33:09.911 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:33:09.911 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:33:09.911 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:33:09.911 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:33:09.911 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:33:09.911 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:33:09.911 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:33:09.911 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:33:09.911 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:09.911 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:09.911 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:33:09.911 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:33:09.911 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:33:09.912 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:33:09.912 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:33:09.912 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:33:09.912 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:33:09.912 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:33:09.912 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:33:09.912 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:33:09.912 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:33:09.912 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:33:09.912 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:33:09.912 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:33:09.912 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:33:09.912 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:33:09.913 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:33:09.913 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:33:09.913 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:33:09.913 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:33:09.913 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:33:09.913 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:33:09.913 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:33:09.913 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:33:09.913 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:33:09.913 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:33:09.913 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:33:09.913 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:33:09.913 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:09.913 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:33:09.914 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:09.914 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:33:09.914 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:09.914 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:33:09.914 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:33:09.914 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:33:09.914 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:33:09.914 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:09.914 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:33:09.914 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:33:09.914 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:09.915 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:09.915 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:09.915 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:33:09.915 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:33:09.915 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:09.915 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:09.915 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:09.915 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:09.915 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:33:09.915 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:33:09.915 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:33:09.915 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:33:09.915 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:33:09.916 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:09.916 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:09.916 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:33:09.916 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:33:09.916 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:33:09.916 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:33:09.916 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:09.916 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:33:09.916 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:33:09.916 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:09.916 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:09.916 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:09.916 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:33:09.916 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:33:09.917 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:09.917 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:09.917 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:09.917 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:09.917 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:33:09.917 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:33:09.917 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:33:09.917 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:33:09.917 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:33:09.917 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:09.917 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:09.917 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:33:09.917 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:33:09.917 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:33:09.917 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:33:09.918 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:09.918 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:33:09.918 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:33:09.918 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:09.918 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:09.918 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:09.918 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:33:09.918 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:33:09.918 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:09.918 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:09.918 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:09.918 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:09.918 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:33:09.918 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:33:09.919 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:33:09.919 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:33:09.919 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:33:09.919 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:09.919 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:09.919 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:09.919 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:33:09.919 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:33:09.919 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:09.919 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:33:09.919 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:33:09.919 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:33:09.919 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:09.920 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:09.920 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:33:09.920 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:09.920 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:33:09.920 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:33:09.920 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:33:09.920 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:09.920 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:09.920 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:33:09.920 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:09.920 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:33:09.920 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:33:09.920 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:33:09.920 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:09.921 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:09.921 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:09.921 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:33:09.921 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:33:09.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.895+0000 7ff632fa1640 1 -- 172.21.15.12:0/1088423352 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff61003e1c0 msgr2=0x7ff610040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:09.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.895+0000 7ff632fa1640 1 --2- 172.21.15.12:0/1088423352 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff61003e1c0 0x7ff610040680 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff62c0a4a90 tx=0x7ff624009000 comp rx=0 tx=0).stop 2024-09-11T01:33:09.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.895+0000 7ff632fa1640 10 monclient: shutdown 2024-09-11T01:33:09.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.895+0000 7ff632fa1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:09.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.895+0000 7ff632fa1640 1 -- 172.21.15.12:0/1088423352 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c137f50 msgr2=0x7ff62c136b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:09.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.895+0000 7ff632fa1640 1 --2- 172.21.15.12:0/1088423352 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c137f50 0x7ff62c136b60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff62000b550 tx=0x7ff62000ba10 comp rx=0 tx=0).stop 2024-09-11T01:33:09.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.895+0000 7ff6327a0640 1 -- 172.21.15.12:0/1088423352 reap_dead start 2024-09-11T01:33:09.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.895+0000 7ff6327a0640 1 -- 172.21.15.12:0/1088423352 reap_dead start 2024-09-11T01:33:09.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.895+0000 7ff632fa1640 1 -- 172.21.15.12:0/1088423352 shutdown_connections 2024-09-11T01:33:09.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.895+0000 7ff632fa1640 1 -- 172.21.15.12:0/1088423352 >> 172.21.15.12:0/1088423352 conn(0x7ff62c0a6ed0 msgr2=0x7ff62c0a3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:09.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.896+0000 7ff632fa1640 1 -- 172.21.15.12:0/1088423352 shutdown_connections 2024-09-11T01:33:09.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:09.896+0000 7ff632fa1640 1 -- 172.21.15.12:0/1088423352 wait complete. 2024-09-11T01:33:09.923 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:33:09.923 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:33:10.203 INFO:tasks.rados.rados.0.smithi071.stdout:625: done (7 left) 2024-09-11T01:33:10.203 INFO:tasks.rados.rados.0.smithi071.stdout:626: snap_remove snap 60 2024-09-11T01:33:11.214 INFO:tasks.rados.rados.0.smithi071.stdout:626: done (7 left) 2024-09-11T01:33:11.215 INFO:tasks.rados.rados.0.smithi071.stdout:627: copy_from oid 3 from oid 21 current snap is 61 2024-09-11T01:33:11.215 INFO:tasks.rados.rados.0.smithi071.stdout:618: finishing rollback tid 1 to smithi07134777-41 2024-09-11T01:33:11.215 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 0 (ObjNum 113 snap 0 seq_num 1) dirty dne 2024-09-11T01:33:11.215 INFO:tasks.rados.rados.0.smithi071.stdout:622: finishing copy_from to smithi07134777-14 2024-09-11T01:33:11.215 INFO:tasks.rados.rados.0.smithi071.stdout:622: got expected ENOENT (src dne) 2024-09-11T01:33:11.215 INFO:tasks.rados.rados.0.smithi071.stdout:622: finishing copy_from racing read to smithi07134777-14 2024-09-11T01:33:11.215 INFO:tasks.rados.rados.0.smithi071.stdout:623: finishing rollback tid 0 to smithi07134777-29 2024-09-11T01:33:11.215 INFO:tasks.rados.rados.0.smithi071.stdout:621: finishing rollback tid 0 to smithi07134777-13 2024-09-11T01:33:11.215 INFO:tasks.rados.rados.0.smithi071.stdout:623: finishing rollback tid 1 to smithi07134777-29 2024-09-11T01:33:11.215 INFO:tasks.rados.rados.0.smithi071.stdout:623: finishing rollback tid 2 to smithi07134777-29 2024-09-11T01:33:11.215 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 302 (ObjNum 191 snap 42 seq_num 191) dirty exists 2024-09-11T01:33:11.215 INFO:tasks.rados.rados.0.smithi071.stdout:621: finishing rollback tid 1 to smithi07134777-13 2024-09-11T01:33:11.216 INFO:tasks.rados.rados.0.smithi071.stdout:621: finishing rollback tid 2 to smithi07134777-13 2024-09-11T01:33:11.216 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 160 (ObjNum 205 snap 47 seq_num 205) dirty exists 2024-09-11T01:33:11.216 INFO:tasks.rados.rados.0.smithi071.stdout:619: finishing copy_from racing read to smithi07134777-16 2024-09-11T01:33:11.216 INFO:tasks.rados.rados.0.smithi071.stdout:624: finishing write tid 1 to smithi07134777-47 2024-09-11T01:33:11.216 INFO:tasks.rados.rados.0.smithi071.stdout:624: finishing write tid 2 to smithi07134777-47 2024-09-11T01:33:11.216 INFO:tasks.rados.rados.0.smithi071.stdout:624: finishing write tid 3 to smithi07134777-47 2024-09-11T01:33:11.216 INFO:tasks.rados.rados.0.smithi071.stdout:619: finishing copy_from to smithi07134777-16 2024-09-11T01:33:11.216 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 210 (ObjNum 240 snap 58 seq_num 240) dirty exists 2024-09-11T01:33:11.216 INFO:tasks.rados.rados.0.smithi071.stdout:624: finishing write tid 4 to smithi07134777-47 2024-09-11T01:33:11.216 INFO:tasks.rados.rados.0.smithi071.stdout:624: oid 47 updating version 0 to 164 2024-09-11T01:33:11.216 INFO:tasks.rados.rados.0.smithi071.stdout:624: oid 47 version 164 is already newer than 163 2024-09-11T01:33:11.216 INFO:tasks.rados.rados.0.smithi071.stdout:624: oid 47 version 164 is already newer than 162 2024-09-11T01:33:11.216 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 164 (ObjNum 243 snap 61 seq_num 243) dirty exists 2024-09-11T01:33:11.217 INFO:tasks.rados.rados.0.smithi071.stdout:624: left oid 47 (ObjNum 243 snap 61 seq_num 243) 2024-09-11T01:33:11.217 INFO:tasks.rados.rados.0.smithi071.stdout:620: finishing copy_from racing read to smithi07134777-20 2024-09-11T01:33:11.217 INFO:tasks.rados.rados.0.smithi071.stdout:620: finishing copy_from to smithi07134777-20 2024-09-11T01:33:11.217 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 135 (ObjNum 114 snap 27 seq_num 114) dirty exists 2024-09-11T01:33:11.217 INFO:tasks.rados.rados.0.smithi071.stdout:618: done (7 left) 2024-09-11T01:33:11.217 INFO:tasks.rados.rados.0.smithi071.stdout:619: done (6 left) 2024-09-11T01:33:11.217 INFO:tasks.rados.rados.0.smithi071.stdout:620: done (5 left) 2024-09-11T01:33:11.217 INFO:tasks.rados.rados.0.smithi071.stdout:621: done (4 left) 2024-09-11T01:33:11.217 INFO:tasks.rados.rados.0.smithi071.stdout:622: done (3 left) 2024-09-11T01:33:11.217 INFO:tasks.rados.rados.0.smithi071.stdout:623: done (2 left) 2024-09-11T01:33:11.217 INFO:tasks.rados.rados.0.smithi071.stdout:624: done (1 left) 2024-09-11T01:33:11.217 INFO:tasks.rados.rados.0.smithi071.stdout:628: read oid 14 snap -1 2024-09-11T01:33:11.217 INFO:tasks.rados.rados.0.smithi071.stdout:628: expect deleted 2024-09-11T01:33:11.217 INFO:tasks.rados.rados.0.smithi071.stdout:629: snap_remove snap 57 2024-09-11T01:33:11.807 INFO:tasks.rados.rados.0.smithi071.stdout:629: done (2 left) 2024-09-11T01:33:11.807 INFO:tasks.rados.rados.0.smithi071.stdout:630: write oid 26 current snap is 61 2024-09-11T01:33:11.807 INFO:tasks.rados.rados.0.smithi071.stdout:630: seq_num 244 ranges {500908=484052,1411724=689184} 2024-09-11T01:33:11.812 INFO:tasks.rados.rados.0.smithi071.stdout:630: writing smithi07134777-26 from 500908 to 984960 tid 1 2024-09-11T01:33:11.819 INFO:tasks.rados.rados.0.smithi071.stdout:630: writing smithi07134777-26 from 1411724 to 2100908 tid 2 2024-09-11T01:33:11.820 INFO:tasks.rados.rados.0.smithi071.stdout:627: finishing copy_from to smithi07134777-3 2024-09-11T01:33:11.820 INFO:tasks.rados.rados.0.smithi071.stdout:627: got expected ENOENT (src dne) 2024-09-11T01:33:11.820 INFO:tasks.rados.rados.0.smithi071.stdout:627: finishing copy_from racing read to smithi07134777-3 2024-09-11T01:33:11.820 INFO:tasks.rados.rados.0.smithi071.stdout:630: finishing write tid 1 to smithi07134777-26 2024-09-11T01:33:11.820 INFO:tasks.rados.rados.0.smithi071.stdout:627: done (2 left) 2024-09-11T01:33:11.820 INFO:tasks.rados.rados.0.smithi071.stdout:628: done (1 left) 2024-09-11T01:33:11.820 INFO:tasks.rados.rados.0.smithi071.stdout:631: delete oid 17 current snap is 61 2024-09-11T01:33:11.825 INFO:tasks.rados.rados.0.smithi071.stdout:630: finishing write tid 2 to smithi07134777-26 2024-09-11T01:33:11.826 INFO:tasks.rados.rados.0.smithi071.stdout:630: finishing write tid 3 to smithi07134777-26 2024-09-11T01:33:11.826 INFO:tasks.rados.rados.0.smithi071.stdout:630: finishing write tid 4 to smithi07134777-26 2024-09-11T01:33:11.826 INFO:tasks.rados.rados.0.smithi071.stdout:630: oid 26 updating version 0 to 214 2024-09-11T01:33:11.826 INFO:tasks.rados.rados.0.smithi071.stdout:630: oid 26 updating version 214 to 215 2024-09-11T01:33:11.826 INFO:tasks.rados.rados.0.smithi071.stdout:630: oid 26 version 215 is already newer than 213 2024-09-11T01:33:11.826 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 215 (ObjNum 244 snap 61 seq_num 244) dirty exists 2024-09-11T01:33:11.826 INFO:tasks.rados.rados.0.smithi071.stdout:630: left oid 26 (ObjNum 244 snap 61 seq_num 244) 2024-09-11T01:33:11.827 INFO:tasks.rados.rados.0.smithi071.stdout:630: done (1 left) 2024-09-11T01:33:11.827 INFO:tasks.rados.rados.0.smithi071.stdout:631: done (0 left) 2024-09-11T01:33:11.827 INFO:tasks.rados.rados.0.smithi071.stdout:632: snap_create 2024-09-11T01:33:12.809 INFO:tasks.rados.rados.0.smithi071.stdout:632: done (0 left) 2024-09-11T01:33:12.809 INFO:tasks.rados.rados.0.smithi071.stdout:633: write oid 19 current snap is 62 2024-09-11T01:33:12.809 INFO:tasks.rados.rados.0.smithi071.stdout:633: seq_num 246 ranges {534522=634550,1812335=322187} 2024-09-11T01:33:12.815 INFO:tasks.rados.rados.0.smithi071.stdout:633: writing smithi07134777-19 from 534522 to 1169072 tid 1 2024-09-11T01:33:12.820 INFO:tasks.rados.rados.0.smithi071.stdout:633: writing smithi07134777-19 from 1812335 to 2134522 tid 2 2024-09-11T01:33:12.820 INFO:tasks.rados.rados.0.smithi071.stdout:634: snap_remove snap 59 2024-09-11T01:33:13.815 INFO:tasks.rados.rados.0.smithi071.stdout:634: done (1 left) 2024-09-11T01:33:13.815 INFO:tasks.rados.rados.0.smithi071.stdout:635: read oid 25 snap -1 2024-09-11T01:33:13.815 INFO:tasks.rados.rados.0.smithi071.stdout:635: expect (ObjNum 235 snap 57 seq_num 235) 2024-09-11T01:33:13.815 INFO:tasks.rados.rados.0.smithi071.stdout:636: write (excl) oid 10 current snap is 62 2024-09-11T01:33:13.815 INFO:tasks.rados.rados.0.smithi071.stdout:636: seq_num 247 ranges {551329=709799,1870817=280512} 2024-09-11T01:33:13.822 INFO:tasks.rados.rados.0.smithi071.stdout:636: writing smithi07134777-10 from 551329 to 1261128 tid 1 2024-09-11T01:33:13.827 INFO:tasks.rados.rados.0.smithi071.stdout:636: writing smithi07134777-10 from 1870817 to 2151329 tid 2 2024-09-11T01:33:13.827 INFO:tasks.rados.rados.0.smithi071.stdout:637: snap_create 2024-09-11T01:33:13.827 INFO:tasks.rados.rados.0.smithi071.stdout:633: finishing write tid 1 to smithi07134777-19 2024-09-11T01:33:13.827 INFO:tasks.rados.rados.0.smithi071.stdout:633: finishing write tid 2 to smithi07134777-19 2024-09-11T01:33:13.828 INFO:tasks.rados.rados.0.smithi071.stdout:633: finishing write tid 3 to smithi07134777-19 2024-09-11T01:33:13.828 INFO:tasks.rados.rados.0.smithi071.stdout:633: finishing write tid 4 to smithi07134777-19 2024-09-11T01:33:13.828 INFO:tasks.rados.rados.0.smithi071.stdout:633: oid 19 updating version 0 to 118 2024-09-11T01:33:13.828 INFO:tasks.rados.rados.0.smithi071.stdout:633: oid 19 updating version 118 to 119 2024-09-11T01:33:13.828 INFO:tasks.rados.rados.0.smithi071.stdout:633: oid 19 updating version 119 to 120 2024-09-11T01:33:13.828 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 120 (ObjNum 246 snap 62 seq_num 246) dirty exists 2024-09-11T01:33:13.828 INFO:tasks.rados.rados.0.smithi071.stdout:633: left oid 19 (ObjNum 246 snap 62 seq_num 246) 2024-09-11T01:33:13.829 INFO:tasks.rados.rados.0.smithi071.stdout:636: finishing write tid 1 to smithi07134777-10 2024-09-11T01:33:13.832 INFO:tasks.rados.rados.0.smithi071.stdout:636: finishing write tid 2 to smithi07134777-10 2024-09-11T01:33:13.832 INFO:tasks.rados.rados.0.smithi071.stdout:636: finishing write tid 3 to smithi07134777-10 2024-09-11T01:33:13.833 INFO:tasks.rados.rados.0.smithi071.stdout:636: finishing write tid 4 to smithi07134777-10 2024-09-11T01:33:13.833 INFO:tasks.rados.rados.0.smithi071.stdout:636: oid 10 updating version 0 to 217 2024-09-11T01:33:13.833 INFO:tasks.rados.rados.0.smithi071.stdout:636: oid 10 updating version 217 to 218 2024-09-11T01:33:13.833 INFO:tasks.rados.rados.0.smithi071.stdout:636: oid 10 version 218 is already newer than 216 2024-09-11T01:33:13.833 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 218 (ObjNum 247 snap 62 seq_num 247) dirty exists 2024-09-11T01:33:13.833 INFO:tasks.rados.rados.0.smithi071.stdout:636: left oid 10 (ObjNum 247 snap 62 seq_num 247) 2024-09-11T01:33:13.837 INFO:tasks.rados.rados.0.smithi071.stdout:635: expect (ObjNum 235 snap 57 seq_num 235) 2024-09-11T01:33:13.837 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 129903 2024-09-11T01:33:13.907 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:33:13.977 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:33:14.815 INFO:tasks.rados.rados.0.smithi071.stdout:633: done (3 left) 2024-09-11T01:33:14.815 INFO:tasks.rados.rados.0.smithi071.stdout:635: done (2 left) 2024-09-11T01:33:14.816 INFO:tasks.rados.rados.0.smithi071.stdout:636: done (1 left) 2024-09-11T01:33:14.816 INFO:tasks.rados.rados.0.smithi071.stdout:637: done (0 left) 2024-09-11T01:33:14.816 INFO:tasks.rados.rados.0.smithi071.stdout:638: rollback oid 5 current snap is 63 2024-09-11T01:33:14.816 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 5 to 58 2024-09-11T01:33:14.816 INFO:tasks.rados.rados.0.smithi071.stdout:639: write oid 11 current snap is 63 2024-09-11T01:33:14.816 INFO:tasks.rados.rados.0.smithi071.stdout:639: seq_num 248 ranges {568136=785048,1929299=238837} 2024-09-11T01:33:14.823 INFO:tasks.rados.rados.0.smithi071.stdout:639: writing smithi07134777-11 from 568136 to 1353184 tid 1 2024-09-11T01:33:14.828 INFO:tasks.rados.rados.0.smithi071.stdout:639: writing smithi07134777-11 from 1929299 to 2168136 tid 2 2024-09-11T01:33:14.828 INFO:tasks.rados.rados.0.smithi071.stdout:640: read oid 22 snap 62 2024-09-11T01:33:14.828 INFO:tasks.rados.rados.0.smithi071.stdout:640: expect (ObjNum 240 snap 58 seq_num 240) 2024-09-11T01:33:14.828 INFO:tasks.rados.rados.0.smithi071.stdout:641: delete oid 49 current snap is 63 2024-09-11T01:33:14.828 INFO:tasks.rados.rados.0.smithi071.stdout:638: finishing rollback tid 0 to smithi07134777-5 2024-09-11T01:33:14.828 INFO:tasks.rados.rados.0.smithi071.stdout:638: finishing rollback tid 1 to smithi07134777-5 2024-09-11T01:33:14.828 INFO:tasks.rados.rados.0.smithi071.stdout:638: finishing rollback tid 2 to smithi07134777-5 2024-09-11T01:33:14.828 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 37 (ObjNum 201 snap 46 seq_num 201) dirty exists 2024-09-11T01:33:14.830 INFO:tasks.rados.rados.0.smithi071.stdout:638: done (3 left) 2024-09-11T01:33:14.830 INFO:tasks.rados.rados.0.smithi071.stdout:641: done (2 left) 2024-09-11T01:33:14.830 INFO:tasks.rados.rados.0.smithi071.stdout:642: read oid 16 snap -1 2024-09-11T01:33:14.830 INFO:tasks.rados.rados.0.smithi071.stdout:642: expect (ObjNum 240 snap 58 seq_num 240) 2024-09-11T01:33:14.830 INFO:tasks.rados.rados.0.smithi071.stdout:643: snap_remove snap 61 2024-09-11T01:33:15.820 INFO:tasks.rados.rados.0.smithi071.stdout:643: done (3 left) 2024-09-11T01:33:15.820 INFO:tasks.rados.rados.0.smithi071.stdout:644: write oid 25 current snap is 63 2024-09-11T01:33:15.820 INFO:tasks.rados.rados.0.smithi071.stdout:644: seq_num 250 ranges {601750=535546,1929910=271840} 2024-09-11T01:33:15.826 INFO:tasks.rados.rados.0.smithi071.stdout:644: writing smithi07134777-25 from 601750 to 1137296 tid 1 2024-09-11T01:33:15.832 INFO:tasks.rados.rados.0.smithi071.stdout:644: writing smithi07134777-25 from 1929910 to 2201750 tid 2 2024-09-11T01:33:15.832 INFO:tasks.rados.rados.0.smithi071.stdout:645: write (excl) oid 4 current snap is 63 2024-09-11T01:33:15.832 INFO:tasks.rados.rados.0.smithi071.stdout:645: seq_num 251 ranges {618557=727148,2104745=113812} 2024-09-11T01:33:15.839 INFO:tasks.rados.rados.0.smithi071.stdout:645: writing smithi07134777-4 from 618557 to 1345705 tid 1 2024-09-11T01:33:15.844 INFO:tasks.rados.rados.0.smithi071.stdout:645: writing smithi07134777-4 from 2104745 to 2218557 tid 2 2024-09-11T01:33:15.844 INFO:tasks.rados.rados.0.smithi071.stdout:646: read oid 17 snap -1 2024-09-11T01:33:15.844 INFO:tasks.rados.rados.0.smithi071.stdout:646: expect deleted 2024-09-11T01:33:15.844 INFO:tasks.rados.rados.0.smithi071.stdout:647: delete oid 38 current snap is 63 2024-09-11T01:33:15.844 INFO:tasks.rados.rados.0.smithi071.stdout:640: expect (ObjNum 240 snap 58 seq_num 240) 2024-09-11T01:33:15.844 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 90662 2024-09-11T01:33:15.909 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:33:15.941 INFO:tasks.rados.rados.0.smithi071.stdout:639: finishing write tid 1 to smithi07134777-11 2024-09-11T01:33:15.941 INFO:tasks.rados.rados.0.smithi071.stdout:639: finishing write tid 2 to smithi07134777-11 2024-09-11T01:33:15.941 INFO:tasks.rados.rados.0.smithi071.stdout:639: finishing write tid 3 to smithi07134777-11 2024-09-11T01:33:15.941 INFO:tasks.rados.rados.0.smithi071.stdout:639: finishing write tid 4 to smithi07134777-11 2024-09-11T01:33:15.942 INFO:tasks.rados.rados.0.smithi071.stdout:639: oid 11 updating version 0 to 216 2024-09-11T01:33:15.942 INFO:tasks.rados.rados.0.smithi071.stdout:639: oid 11 updating version 216 to 217 2024-09-11T01:33:15.942 INFO:tasks.rados.rados.0.smithi071.stdout:639: oid 11 version 217 is already newer than 215 2024-09-11T01:33:15.942 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 217 (ObjNum 248 snap 63 seq_num 248) dirty exists 2024-09-11T01:33:15.942 INFO:tasks.rados.rados.0.smithi071.stdout:639: left oid 11 (ObjNum 248 snap 63 seq_num 248) 2024-09-11T01:33:15.942 INFO:tasks.rados.rados.0.smithi071.stdout:642: expect (ObjNum 240 snap 58 seq_num 240) 2024-09-11T01:33:15.973 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:33:16.039 INFO:tasks.rados.rados.0.smithi071.stdout:644: finishing write tid 1 to smithi07134777-25 2024-09-11T01:33:16.040 INFO:tasks.rados.rados.0.smithi071.stdout:644: finishing write tid 2 to smithi07134777-25 2024-09-11T01:33:16.040 INFO:tasks.rados.rados.0.smithi071.stdout:639: done (6 left) 2024-09-11T01:33:16.040 INFO:tasks.rados.rados.0.smithi071.stdout:640: done (5 left) 2024-09-11T01:33:16.040 INFO:tasks.rados.rados.0.smithi071.stdout:642: done (4 left) 2024-09-11T01:33:16.040 INFO:tasks.rados.rados.0.smithi071.stdout:647: done (3 left) 2024-09-11T01:33:16.040 INFO:tasks.rados.rados.0.smithi071.stdout:648: snap_create 2024-09-11T01:33:16.040 INFO:tasks.rados.rados.0.smithi071.stdout:644: finishing write tid 3 to smithi07134777-25 2024-09-11T01:33:16.040 INFO:tasks.rados.rados.0.smithi071.stdout:644: finishing write tid 4 to smithi07134777-25 2024-09-11T01:33:16.040 INFO:tasks.rados.rados.0.smithi071.stdout:644: oid 25 updating version 0 to 78 2024-09-11T01:33:16.040 INFO:tasks.rados.rados.0.smithi071.stdout:644: oid 25 updating version 78 to 79 2024-09-11T01:33:16.040 INFO:tasks.rados.rados.0.smithi071.stdout:644: oid 25 version 79 is already newer than 77 2024-09-11T01:33:16.040 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 79 (ObjNum 250 snap 63 seq_num 250) dirty exists 2024-09-11T01:33:16.040 INFO:tasks.rados.rados.0.smithi071.stdout:644: left oid 25 (ObjNum 250 snap 63 seq_num 250) 2024-09-11T01:33:16.041 INFO:tasks.rados.rados.0.smithi071.stdout:645: finishing write tid 1 to smithi07134777-4 2024-09-11T01:33:16.041 INFO:tasks.rados.rados.0.smithi071.stdout:645: finishing write tid 2 to smithi07134777-4 2024-09-11T01:33:16.041 INFO:tasks.rados.rados.0.smithi071.stdout:645: finishing write tid 3 to smithi07134777-4 2024-09-11T01:33:16.041 INFO:tasks.rados.rados.0.smithi071.stdout:645: finishing write tid 4 to smithi07134777-4 2024-09-11T01:33:16.041 INFO:tasks.rados.rados.0.smithi071.stdout:645: oid 4 updating version 0 to 218 2024-09-11T01:33:16.041 INFO:tasks.rados.rados.0.smithi071.stdout:645: oid 4 updating version 218 to 220 2024-09-11T01:33:16.041 INFO:tasks.rados.rados.0.smithi071.stdout:645: oid 4 version 220 is already newer than 219 2024-09-11T01:33:16.041 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 220 (ObjNum 251 snap 63 seq_num 251) dirty exists 2024-09-11T01:33:16.041 INFO:tasks.rados.rados.0.smithi071.stdout:645: left oid 4 (ObjNum 251 snap 63 seq_num 251) 2024-09-11T01:33:16.807 INFO:tasks.rados.rados.0.smithi071.stdout:644: done (3 left) 2024-09-11T01:33:16.807 INFO:tasks.rados.rados.0.smithi071.stdout:645: done (2 left) 2024-09-11T01:33:16.807 INFO:tasks.rados.rados.0.smithi071.stdout:646: done (1 left) 2024-09-11T01:33:16.807 INFO:tasks.rados.rados.0.smithi071.stdout:648: done (0 left) 2024-09-11T01:33:16.807 INFO:tasks.rados.rados.0.smithi071.stdout:649: copy_from oid 13 from oid 48 current snap is 64 2024-09-11T01:33:16.807 INFO:tasks.rados.rados.0.smithi071.stdout:650: read oid 23 snap 63 2024-09-11T01:33:16.807 INFO:tasks.rados.rados.0.smithi071.stdout:650: expect (ObjNum 213 snap 50 seq_num 213) 2024-09-11T01:33:16.807 INFO:tasks.rados.rados.0.smithi071.stdout:651: snap_create 2024-09-11T01:33:16.820 INFO:tasks.rados.rados.0.smithi071.stdout:650: expect (ObjNum 213 snap 50 seq_num 213) 2024-09-11T01:33:16.963 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1356116 2024-09-11T01:33:17.008 INFO:tasks.rados.rados.0.smithi071.stdout:649: finishing copy_from racing read to smithi07134777-13 2024-09-11T01:33:17.008 INFO:tasks.rados.rados.0.smithi071.stdout:649: finishing copy_from to smithi07134777-13 2024-09-11T01:33:17.008 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 290 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-11T01:33:17.812 INFO:tasks.rados.rados.0.smithi071.stdout:649: done (2 left) 2024-09-11T01:33:17.812 INFO:tasks.rados.rados.0.smithi071.stdout:650: done (1 left) 2024-09-11T01:33:17.813 INFO:tasks.rados.rados.0.smithi071.stdout:651: done (0 left) 2024-09-11T01:33:17.813 INFO:tasks.rados.rados.0.smithi071.stdout:652: snap_create 2024-09-11T01:33:18.813 INFO:tasks.rados.rados.0.smithi071.stdout:652: done (0 left) 2024-09-11T01:33:18.814 INFO:tasks.rados.rados.0.smithi071.stdout:653: read oid 4 snap -1 2024-09-11T01:33:18.814 INFO:tasks.rados.rados.0.smithi071.stdout:653: expect (ObjNum 251 snap 63 seq_num 251) 2024-09-11T01:33:18.814 INFO:tasks.rados.rados.0.smithi071.stdout:654: write oid 15 current snap is 66 2024-09-11T01:33:18.814 INFO:tasks.rados.rados.0.smithi071.stdout:654: seq_num 253 ranges {652171=477646,1821709=430462} 2024-09-11T01:33:18.819 INFO:tasks.rados.rados.0.smithi071.stdout:654: writing smithi07134777-15 from 652171 to 1129817 tid 1 2024-09-11T01:33:18.827 INFO:tasks.rados.rados.0.smithi071.stdout:654: writing smithi07134777-15 from 1821709 to 2252171 tid 2 2024-09-11T01:33:18.827 INFO:tasks.rados.rados.0.smithi071.stdout:655: read oid 20 snap -1 2024-09-11T01:33:18.827 INFO:tasks.rados.rados.0.smithi071.stdout:655: expect (ObjNum 114 snap 27 seq_num 114) 2024-09-11T01:33:18.827 INFO:tasks.rados.rados.0.smithi071.stdout:656: copy_from oid 7 from oid 12 current snap is 66 2024-09-11T01:33:18.827 INFO:tasks.rados.rados.0.smithi071.stdout:654: finishing write tid 1 to smithi07134777-15 2024-09-11T01:33:18.827 INFO:tasks.rados.rados.0.smithi071.stdout:653: expect (ObjNum 251 snap 63 seq_num 251) 2024-09-11T01:33:18.827 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:33:19.006 INFO:tasks.rados.rados.0.smithi071.stdout:654: finishing write tid 2 to smithi07134777-15 2024-09-11T01:33:19.006 INFO:tasks.rados.rados.0.smithi071.stdout:654: finishing write tid 3 to smithi07134777-15 2024-09-11T01:33:19.006 INFO:tasks.rados.rados.0.smithi071.stdout:654: finishing write tid 4 to smithi07134777-15 2024-09-11T01:33:19.006 INFO:tasks.rados.rados.0.smithi071.stdout:654: oid 15 updating version 0 to 321 2024-09-11T01:33:19.006 INFO:tasks.rados.rados.0.smithi071.stdout:654: oid 15 version 321 is already newer than 320 2024-09-11T01:33:19.006 INFO:tasks.rados.rados.0.smithi071.stdout:654: oid 15 updating version 321 to 322 2024-09-11T01:33:19.007 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 322 (ObjNum 253 snap 66 seq_num 253) dirty exists 2024-09-11T01:33:19.007 INFO:tasks.rados.rados.0.smithi071.stdout:654: left oid 15 (ObjNum 253 snap 66 seq_num 253) 2024-09-11T01:33:19.007 INFO:tasks.rados.rados.0.smithi071.stdout:655: expect (ObjNum 114 snap 27 seq_num 114) 2024-09-11T01:33:19.009 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 624135 2024-09-11T01:33:19.120 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:33:19.179 INFO:tasks.rados.rados.0.smithi071.stdout:656: finishing copy_from racing read to smithi07134777-7 2024-09-11T01:33:19.179 INFO:tasks.rados.rados.0.smithi071.stdout:656: finishing copy_from to smithi07134777-7 2024-09-11T01:33:19.179 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 135 (ObjNum 114 snap 27 seq_num 114) dirty exists 2024-09-11T01:33:19.179 INFO:tasks.rados.rados.0.smithi071.stdout:653: done (3 left) 2024-09-11T01:33:19.179 INFO:tasks.rados.rados.0.smithi071.stdout:654: done (2 left) 2024-09-11T01:33:19.179 INFO:tasks.rados.rados.0.smithi071.stdout:655: done (1 left) 2024-09-11T01:33:19.179 INFO:tasks.rados.rados.0.smithi071.stdout:656: done (0 left) 2024-09-11T01:33:19.179 INFO:tasks.rados.rados.0.smithi071.stdout:657: copy_from oid 2 from oid 45 current snap is 66 2024-09-11T01:33:19.180 INFO:tasks.rados.rados.0.smithi071.stdout:658: copy_from oid 50 from oid 35 current snap is 66 2024-09-11T01:33:19.180 INFO:tasks.rados.rados.0.smithi071.stdout:659: write oid 5 current snap is 66 2024-09-11T01:33:19.180 INFO:tasks.rados.rados.0.smithi071.stdout:659: seq_num 254 ranges {668978=552895,1763838=505140} 2024-09-11T01:33:19.186 INFO:tasks.rados.rados.0.smithi071.stdout:659: writing smithi07134777-5 from 668978 to 1221873 tid 1 2024-09-11T01:33:19.193 INFO:tasks.rados.rados.0.smithi071.stdout:659: writing smithi07134777-5 from 1763838 to 2268978 tid 2 2024-09-11T01:33:19.193 INFO:tasks.rados.rados.0.smithi071.stdout:660: snap_create 2024-09-11T01:33:19.193 INFO:tasks.rados.rados.0.smithi071.stdout:657: finishing copy_from racing read to smithi07134777-2 2024-09-11T01:33:19.193 INFO:tasks.rados.rados.0.smithi071.stdout:658: finishing copy_from racing read to smithi07134777-50 2024-09-11T01:33:19.193 INFO:tasks.rados.rados.0.smithi071.stdout:657: finishing copy_from to smithi07134777-2 2024-09-11T01:33:19.193 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 98 (ObjNum 119 snap 28 seq_num 119) dirty exists 2024-09-11T01:33:19.194 INFO:tasks.rados.rados.0.smithi071.stdout:658: finishing copy_from to smithi07134777-50 2024-09-11T01:33:19.194 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 159 (ObjNum 204 snap 47 seq_num 204) dirty exists 2024-09-11T01:33:19.195 INFO:tasks.rados.rados.0.smithi071.stdout:659: finishing write tid 1 to smithi07134777-5 2024-09-11T01:33:19.195 INFO:tasks.rados.rados.0.smithi071.stdout:659: finishing write tid 2 to smithi07134777-5 2024-09-11T01:33:19.196 INFO:tasks.rados.rados.0.smithi071.stdout:659: finishing write tid 3 to smithi07134777-5 2024-09-11T01:33:19.196 INFO:tasks.rados.rados.0.smithi071.stdout:659: finishing write tid 4 to smithi07134777-5 2024-09-11T01:33:19.196 INFO:tasks.rados.rados.0.smithi071.stdout:659: oid 5 updating version 0 to 42 2024-09-11T01:33:19.196 INFO:tasks.rados.rados.0.smithi071.stdout:659: oid 5 version 42 is already newer than 41 2024-09-11T01:33:19.197 INFO:tasks.rados.rados.0.smithi071.stdout:659: oid 5 updating version 42 to 43 2024-09-11T01:33:19.197 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 43 (ObjNum 254 snap 66 seq_num 254) dirty exists 2024-09-11T01:33:19.197 INFO:tasks.rados.rados.0.smithi071.stdout:659: left oid 5 (ObjNum 254 snap 66 seq_num 254) 2024-09-11T01:33:19.820 INFO:tasks.rados.rados.0.smithi071.stdout:657: done (3 left) 2024-09-11T01:33:19.820 INFO:tasks.rados.rados.0.smithi071.stdout:658: done (2 left) 2024-09-11T01:33:19.821 INFO:tasks.rados.rados.0.smithi071.stdout:659: done (1 left) 2024-09-11T01:33:19.821 INFO:tasks.rados.rados.0.smithi071.stdout:660: done (0 left) 2024-09-11T01:33:19.821 INFO:tasks.rados.rados.0.smithi071.stdout:661: snap_remove snap 64 2024-09-11T01:33:20.824 INFO:tasks.rados.rados.0.smithi071.stdout:661: done (0 left) 2024-09-11T01:33:20.824 INFO:tasks.rados.rados.0.smithi071.stdout:662: write oid 19 current snap is 67 2024-09-11T01:33:20.824 INFO:tasks.rados.rados.0.smithi071.stdout:662: seq_num 255 ranges {685785=628144,1822320=463465} 2024-09-11T01:33:20.831 INFO:tasks.rados.rados.0.smithi071.stdout:662: writing smithi07134777-19 from 685785 to 1313929 tid 1 2024-09-11T01:33:20.837 INFO:tasks.rados.rados.0.smithi071.stdout:662: writing smithi07134777-19 from 1822320 to 2285785 tid 2 2024-09-11T01:33:20.837 INFO:tasks.rados.rados.0.smithi071.stdout:663: copy_from oid 37 from oid 34 current snap is 67 2024-09-11T01:33:20.837 INFO:tasks.rados.rados.0.smithi071.stdout:664: delete oid 30 current snap is 67 2024-09-11T01:33:20.841 INFO:tasks.rados.rados.0.smithi071.stdout:663: finishing copy_from to smithi07134777-37 2024-09-11T01:33:20.841 INFO:tasks.rados.rados.0.smithi071.stdout:663: got expected ENOENT (src dne) 2024-09-11T01:33:20.841 INFO:tasks.rados.rados.0.smithi071.stdout:663: finishing copy_from racing read to smithi07134777-37 2024-09-11T01:33:20.843 INFO:tasks.rados.rados.0.smithi071.stdout:662: finishing write tid 1 to smithi07134777-19 2024-09-11T01:33:20.843 INFO:tasks.rados.rados.0.smithi071.stdout:663: done (2 left) 2024-09-11T01:33:20.843 INFO:tasks.rados.rados.0.smithi071.stdout:664: done (1 left) 2024-09-11T01:33:20.843 INFO:tasks.rados.rados.0.smithi071.stdout:665: delete oid 18 current snap is 67 2024-09-11T01:33:20.843 INFO:tasks.rados.rados.0.smithi071.stdout:662: finishing write tid 2 to smithi07134777-19 2024-09-11T01:33:20.846 INFO:tasks.rados.rados.0.smithi071.stdout:662: finishing write tid 3 to smithi07134777-19 2024-09-11T01:33:20.846 INFO:tasks.rados.rados.0.smithi071.stdout:662: finishing write tid 4 to smithi07134777-19 2024-09-11T01:33:20.846 INFO:tasks.rados.rados.0.smithi071.stdout:662: oid 19 updating version 0 to 121 2024-09-11T01:33:20.846 INFO:tasks.rados.rados.0.smithi071.stdout:662: oid 19 updating version 121 to 123 2024-09-11T01:33:20.846 INFO:tasks.rados.rados.0.smithi071.stdout:662: oid 19 version 123 is already newer than 122 2024-09-11T01:33:20.846 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 123 (ObjNum 255 snap 67 seq_num 255) dirty exists 2024-09-11T01:33:20.846 INFO:tasks.rados.rados.0.smithi071.stdout:662: left oid 19 (ObjNum 255 snap 67 seq_num 255) 2024-09-11T01:33:20.846 INFO:tasks.rados.rados.0.smithi071.stdout:662: done (1 left) 2024-09-11T01:33:20.846 INFO:tasks.rados.rados.0.smithi071.stdout:665: done (0 left) 2024-09-11T01:33:20.846 INFO:tasks.rados.rados.0.smithi071.stdout:666: read oid 30 snap -1 2024-09-11T01:33:20.846 INFO:tasks.rados.rados.0.smithi071.stdout:666: expect deleted 2024-09-11T01:33:20.846 INFO:tasks.rados.rados.0.smithi071.stdout:667: write (excl) oid 9 current snap is 67 2024-09-11T01:33:20.847 INFO:tasks.rados.rados.0.smithi071.stdout:667: seq_num 258 ranges {736206=453891,1881413=454793} 2024-09-11T01:33:20.852 INFO:tasks.rados.rados.0.smithi071.stdout:667: writing smithi07134777-9 from 736206 to 1190097 tid 1 2024-09-11T01:33:20.858 INFO:tasks.rados.rados.0.smithi071.stdout:667: writing smithi07134777-9 from 1881413 to 2336206 tid 2 2024-09-11T01:33:20.858 INFO:tasks.rados.rados.0.smithi071.stdout:668: write oid 40 current snap is 67 2024-09-11T01:33:20.858 INFO:tasks.rados.rados.0.smithi071.stdout:668: seq_num 259 ranges {753013=645493,2056248=296765} 2024-09-11T01:33:20.866 INFO:tasks.rados.rados.0.smithi071.stdout:668: writing smithi07134777-40 from 753013 to 1398506 tid 1 2024-09-11T01:33:20.872 INFO:tasks.rados.rados.0.smithi071.stdout:668: writing smithi07134777-40 from 2056248 to 2353013 tid 2 2024-09-11T01:33:20.872 INFO:tasks.rados.rados.0.smithi071.stdout:669: copy_from oid 24 from oid 31 current snap is 67 2024-09-11T01:33:20.872 INFO:tasks.rados.rados.0.smithi071.stdout:667: finishing write tid 1 to smithi07134777-9 2024-09-11T01:33:20.872 INFO:tasks.rados.rados.0.smithi071.stdout:667: finishing write tid 2 to smithi07134777-9 2024-09-11T01:33:20.872 INFO:tasks.rados.rados.0.smithi071.stdout:667: finishing write tid 3 to smithi07134777-9 2024-09-11T01:33:20.872 INFO:tasks.rados.rados.0.smithi071.stdout:667: finishing write tid 4 to smithi07134777-9 2024-09-11T01:33:20.872 INFO:tasks.rados.rados.0.smithi071.stdout:667: oid 9 updating version 0 to 80 2024-09-11T01:33:20.872 INFO:tasks.rados.rados.0.smithi071.stdout:667: oid 9 updating version 80 to 82 2024-09-11T01:33:20.872 INFO:tasks.rados.rados.0.smithi071.stdout:667: oid 9 version 82 is already newer than 81 2024-09-11T01:33:20.872 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 82 (ObjNum 258 snap 67 seq_num 258) dirty exists 2024-09-11T01:33:20.872 INFO:tasks.rados.rados.0.smithi071.stdout:667: left oid 9 (ObjNum 258 snap 67 seq_num 258) 2024-09-11T01:33:20.872 INFO:tasks.rados.rados.0.smithi071.stdout:666: done (3 left) 2024-09-11T01:33:20.872 INFO:tasks.rados.rados.0.smithi071.stdout:667: done (2 left) 2024-09-11T01:33:20.872 INFO:tasks.rados.rados.0.smithi071.stdout:670: snap_create 2024-09-11T01:33:20.873 INFO:tasks.rados.rados.0.smithi071.stdout:668: finishing write tid 1 to smithi07134777-40 2024-09-11T01:33:20.876 INFO:tasks.rados.rados.0.smithi071.stdout:668: finishing write tid 2 to smithi07134777-40 2024-09-11T01:33:20.876 INFO:tasks.rados.rados.0.smithi071.stdout:668: finishing write tid 3 to smithi07134777-40 2024-09-11T01:33:20.876 INFO:tasks.rados.rados.0.smithi071.stdout:668: finishing write tid 4 to smithi07134777-40 2024-09-11T01:33:20.876 INFO:tasks.rados.rados.0.smithi071.stdout:668: oid 40 updating version 0 to 132 2024-09-11T01:33:20.876 INFO:tasks.rados.rados.0.smithi071.stdout:668: oid 40 updating version 132 to 133 2024-09-11T01:33:20.876 INFO:tasks.rados.rados.0.smithi071.stdout:668: oid 40 updating version 133 to 134 2024-09-11T01:33:20.876 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 134 (ObjNum 259 snap 67 seq_num 259) dirty exists 2024-09-11T01:33:20.876 INFO:tasks.rados.rados.0.smithi071.stdout:668: left oid 40 (ObjNum 259 snap 67 seq_num 259) 2024-09-11T01:33:20.887 INFO:tasks.rados.rados.0.smithi071.stdout:669: finishing copy_from racing read to smithi07134777-24 2024-09-11T01:33:20.889 INFO:tasks.rados.rados.0.smithi071.stdout:669: finishing copy_from to smithi07134777-24 2024-09-11T01:33:20.890 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 33 (ObjNum 201 snap 46 seq_num 201) dirty exists 2024-09-11T01:33:21.811 INFO:tasks.rados.rados.0.smithi071.stdout:668: done (2 left) 2024-09-11T01:33:21.811 INFO:tasks.rados.rados.0.smithi071.stdout:669: done (1 left) 2024-09-11T01:33:21.811 INFO:tasks.rados.rados.0.smithi071.stdout:670: done (0 left) 2024-09-11T01:33:21.811 INFO:tasks.rados.rados.0.smithi071.stdout:671: snap_create 2024-09-11T01:33:22.812 INFO:tasks.rados.rados.0.smithi071.stdout:671: done (0 left) 2024-09-11T01:33:22.813 INFO:tasks.rados.rados.0.smithi071.stdout:672: snap_remove snap 66 2024-09-11T01:33:23.822 INFO:tasks.rados.rados.0.smithi071.stdout:672: done (0 left) 2024-09-11T01:33:23.822 INFO:tasks.rados.rados.0.smithi071.stdout:673: delete oid 27 current snap is 69 2024-09-11T01:33:23.830 INFO:tasks.rados.rados.0.smithi071.stdout:673: done (0 left) 2024-09-11T01:33:23.830 INFO:tasks.rados.rados.0.smithi071.stdout:674: snap_create 2024-09-11T01:33:24.821 INFO:tasks.rados.rados.0.smithi071.stdout:674: done (0 left) 2024-09-11T01:33:24.822 INFO:tasks.rados.rados.0.smithi071.stdout:675: write (excl) oid 24 current snap is 70 2024-09-11T01:33:24.822 INFO:tasks.rados.rados.0.smithi071.stdout:675: seq_num 261 ranges {786627=795991,2173212=213415} 2024-09-11T01:33:24.830 INFO:tasks.rados.rados.0.smithi071.stdout:675: writing smithi07134777-24 from 786627 to 1582618 tid 1 2024-09-11T01:33:24.835 INFO:tasks.rados.rados.0.smithi071.stdout:675: writing smithi07134777-24 from 2173212 to 2386627 tid 2 2024-09-11T01:33:24.835 INFO:tasks.rados.rados.0.smithi071.stdout:676: rollback oid 23 current snap is 70 2024-09-11T01:33:24.835 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 23 to 62 2024-09-11T01:33:24.835 INFO:tasks.rados.rados.0.smithi071.stdout:677: rollback oid 32 current snap is 70 2024-09-11T01:33:24.835 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 32 to 68 2024-09-11T01:33:24.835 INFO:tasks.rados.rados.0.smithi071.stdout:678: delete oid 40 current snap is 70 2024-09-11T01:33:24.836 INFO:tasks.rados.rados.0.smithi071.stdout:677: finishing rollback tid 1 to smithi07134777-32 2024-09-11T01:33:24.836 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-11T01:33:24.841 INFO:tasks.rados.rados.0.smithi071.stdout:677: done (3 left) 2024-09-11T01:33:24.841 INFO:tasks.rados.rados.0.smithi071.stdout:678: done (2 left) 2024-09-11T01:33:24.841 INFO:tasks.rados.rados.0.smithi071.stdout:679: read oid 50 snap -1 2024-09-11T01:33:24.841 INFO:tasks.rados.rados.0.smithi071.stdout:679: expect (ObjNum 204 snap 47 seq_num 204) 2024-09-11T01:33:24.841 INFO:tasks.rados.rados.0.smithi071.stdout:680: copy_from oid 6 from oid 4 current snap is 70 2024-09-11T01:33:24.841 INFO:tasks.rados.rados.0.smithi071.stdout:681: write (excl) oid 11 current snap is 70 2024-09-11T01:33:24.841 INFO:tasks.rados.rados.0.smithi071.stdout:681: seq_num 263 ranges {420241=546489,1373823=544414} 2024-09-11T01:33:24.846 INFO:tasks.rados.rados.0.smithi071.stdout:681: writing smithi07134777-11 from 420241 to 966730 tid 1 2024-09-11T01:33:24.852 INFO:tasks.rados.rados.0.smithi071.stdout:681: writing smithi07134777-11 from 1373823 to 1918237 tid 2 2024-09-11T01:33:24.852 INFO:tasks.rados.rados.0.smithi071.stdout:682: delete oid 42 current snap is 70 2024-09-11T01:33:24.853 INFO:tasks.rados.rados.0.smithi071.stdout:676: finishing rollback tid 0 to smithi07134777-23 2024-09-11T01:33:24.853 INFO:tasks.rados.rados.0.smithi071.stdout:676: finishing rollback tid 1 to smithi07134777-23 2024-09-11T01:33:24.853 INFO:tasks.rados.rados.0.smithi071.stdout:675: finishing write tid 1 to smithi07134777-24 2024-09-11T01:33:24.853 INFO:tasks.rados.rados.0.smithi071.stdout:676: finishing rollback tid 2 to smithi07134777-23 2024-09-11T01:33:24.853 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 227 (ObjNum 213 snap 50 seq_num 213) dirty exists 2024-09-11T01:33:24.853 INFO:tasks.rados.rados.0.smithi071.stdout:675: finishing write tid 2 to smithi07134777-24 2024-09-11T01:33:24.853 INFO:tasks.rados.rados.0.smithi071.stdout:675: finishing write tid 3 to smithi07134777-24 2024-09-11T01:33:24.853 INFO:tasks.rados.rados.0.smithi071.stdout:675: finishing write tid 4 to smithi07134777-24 2024-09-11T01:33:24.853 INFO:tasks.rados.rados.0.smithi071.stdout:675: oid 24 updating version 0 to 218 2024-09-11T01:33:24.853 INFO:tasks.rados.rados.0.smithi071.stdout:675: oid 24 version 218 is already newer than 216 2024-09-11T01:33:24.853 INFO:tasks.rados.rados.0.smithi071.stdout:675: oid 24 version 218 is already newer than 217 2024-09-11T01:33:24.853 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 218 (ObjNum 261 snap 70 seq_num 261) dirty exists 2024-09-11T01:33:24.853 INFO:tasks.rados.rados.0.smithi071.stdout:675: left oid 24 (ObjNum 261 snap 70 seq_num 261) 2024-09-11T01:33:24.854 INFO:tasks.rados.rados.0.smithi071.stdout:680: finishing copy_from racing read to smithi07134777-6 2024-09-11T01:33:24.855 INFO:tasks.rados.rados.0.smithi071.stdout:680: finishing copy_from to smithi07134777-6 2024-09-11T01:33:24.855 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 220 (ObjNum 251 snap 63 seq_num 251) dirty exists 2024-09-11T01:33:24.855 INFO:tasks.rados.rados.0.smithi071.stdout:679: expect (ObjNum 204 snap 47 seq_num 204) 2024-09-11T01:33:24.900 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:33:24.953 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:33:25.006 INFO:tasks.rados.rados.0.smithi071.stdout:681: finishing write tid 1 to smithi07134777-11 2024-09-11T01:33:25.006 INFO:tasks.rados.rados.0.smithi071.stdout:681: finishing write tid 2 to smithi07134777-11 2024-09-11T01:33:25.006 INFO:tasks.rados.rados.0.smithi071.stdout:681: finishing write tid 3 to smithi07134777-11 2024-09-11T01:33:25.006 INFO:tasks.rados.rados.0.smithi071.stdout:681: finishing write tid 4 to smithi07134777-11 2024-09-11T01:33:25.006 INFO:tasks.rados.rados.0.smithi071.stdout:681: oid 11 updating version 0 to 225 2024-09-11T01:33:25.006 INFO:tasks.rados.rados.0.smithi071.stdout:681: oid 11 version 225 is already newer than 224 2024-09-11T01:33:25.006 INFO:tasks.rados.rados.0.smithi071.stdout:681: oid 11 version 225 is already newer than 223 2024-09-11T01:33:25.006 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 225 (ObjNum 263 snap 70 seq_num 263) dirty exists 2024-09-11T01:33:25.006 INFO:tasks.rados.rados.0.smithi071.stdout:681: left oid 11 (ObjNum 263 snap 70 seq_num 263) 2024-09-11T01:33:25.006 INFO:tasks.rados.rados.0.smithi071.stdout:675: done (5 left) 2024-09-11T01:33:25.007 INFO:tasks.rados.rados.0.smithi071.stdout:676: done (4 left) 2024-09-11T01:33:25.007 INFO:tasks.rados.rados.0.smithi071.stdout:679: done (3 left) 2024-09-11T01:33:25.007 INFO:tasks.rados.rados.0.smithi071.stdout:680: done (2 left) 2024-09-11T01:33:25.007 INFO:tasks.rados.rados.0.smithi071.stdout:681: done (1 left) 2024-09-11T01:33:25.007 INFO:tasks.rados.rados.0.smithi071.stdout:682: done (0 left) 2024-09-11T01:33:25.007 INFO:tasks.rados.rados.0.smithi071.stdout:683: read oid 37 snap -1 2024-09-11T01:33:25.007 INFO:tasks.rados.rados.0.smithi071.stdout:683: expect (ObjNum 170 snap 37 seq_num 170) 2024-09-11T01:33:25.007 INFO:tasks.rados.rados.0.smithi071.stdout:684: rollback oid 23 current snap is 70 2024-09-11T01:33:25.007 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 23 to 68 2024-09-11T01:33:25.007 INFO:tasks.rados.rados.0.smithi071.stdout:685: write (excl) oid 31 current snap is 70 2024-09-11T01:33:25.007 INFO:tasks.rados.rados.0.smithi071.stdout:685: seq_num 265 ranges {453855=696987,1890787=548083} 2024-09-11T01:33:25.011 INFO:tasks.rados.rados.0.smithi071.stdout:685: writing smithi07134777-31 from 453855 to 1150842 tid 1 2024-09-11T01:33:25.019 INFO:tasks.rados.rados.0.smithi071.stdout:685: writing smithi07134777-31 from 1890787 to 2438870 tid 2 2024-09-11T01:33:25.019 INFO:tasks.rados.rados.0.smithi071.stdout:686: rollback oid 9 current snap is 70 2024-09-11T01:33:25.019 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 9 to 63 2024-09-11T01:33:25.019 INFO:tasks.rados.rados.0.smithi071.stdout:684: finishing rollback tid 0 to smithi07134777-23 2024-09-11T01:33:25.019 INFO:tasks.rados.rados.0.smithi071.stdout:684: finishing rollback tid 1 to smithi07134777-23 2024-09-11T01:33:25.019 INFO:tasks.rados.rados.0.smithi071.stdout:684: finishing rollback tid 2 to smithi07134777-23 2024-09-11T01:33:25.019 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 230 (ObjNum 213 snap 50 seq_num 213) dirty exists 2024-09-11T01:33:25.019 INFO:tasks.rados.rados.0.smithi071.stdout:683: expect (ObjNum 170 snap 37 seq_num 170) 2024-09-11T01:33:25.104 INFO:tasks.rados.rados.0.smithi071.stdout:685: finishing write tid 1 to smithi07134777-31 2024-09-11T01:33:25.104 INFO:tasks.rados.rados.0.smithi071.stdout:686: finishing rollback tid 0 to smithi07134777-9 2024-09-11T01:33:25.105 INFO:tasks.rados.rados.0.smithi071.stdout:685: finishing write tid 2 to smithi07134777-31 2024-09-11T01:33:25.105 INFO:tasks.rados.rados.0.smithi071.stdout:685: finishing write tid 3 to smithi07134777-31 2024-09-11T01:33:25.105 INFO:tasks.rados.rados.0.smithi071.stdout:685: finishing write tid 4 to smithi07134777-31 2024-09-11T01:33:25.105 INFO:tasks.rados.rados.0.smithi071.stdout:685: oid 31 updating version 0 to 292 2024-09-11T01:33:25.105 INFO:tasks.rados.rados.0.smithi071.stdout:685: oid 31 updating version 292 to 293 2024-09-11T01:33:25.105 INFO:tasks.rados.rados.0.smithi071.stdout:685: oid 31 version 293 is already newer than 291 2024-09-11T01:33:25.105 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 293 (ObjNum 265 snap 70 seq_num 265) dirty exists 2024-09-11T01:33:25.105 INFO:tasks.rados.rados.0.smithi071.stdout:685: left oid 31 (ObjNum 265 snap 70 seq_num 265) 2024-09-11T01:33:25.105 INFO:tasks.rados.rados.0.smithi071.stdout:683: done (3 left) 2024-09-11T01:33:25.105 INFO:tasks.rados.rados.0.smithi071.stdout:684: done (2 left) 2024-09-11T01:33:25.105 INFO:tasks.rados.rados.0.smithi071.stdout:685: done (1 left) 2024-09-11T01:33:25.105 INFO:tasks.rados.rados.0.smithi071.stdout:687: read oid 43 snap -1 2024-09-11T01:33:25.105 INFO:tasks.rados.rados.0.smithi071.stdout:687: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:33:25.106 INFO:tasks.rados.rados.0.smithi071.stdout:688: snap_create 2024-09-11T01:33:25.229 INFO:tasks.rados.rados.0.smithi071.stdout:686: finishing rollback tid 1 to smithi07134777-9 2024-09-11T01:33:25.229 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 85 (ObjNum 33 snap 0 seq_num 3921778543) dirty dne 2024-09-11T01:33:25.236 INFO:tasks.rados.rados.0.smithi071.stdout:687: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:33:25.262 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1104813 2024-09-11T01:33:25.278 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:33:25.837 INFO:tasks.rados.rados.0.smithi071.stdout:686: done (2 left) 2024-09-11T01:33:25.838 INFO:tasks.rados.rados.0.smithi071.stdout:687: done (1 left) 2024-09-11T01:33:25.838 INFO:tasks.rados.rados.0.smithi071.stdout:688: done (0 left) 2024-09-11T01:33:25.838 INFO:tasks.rados.rados.0.smithi071.stdout:689: snap_remove snap 65 2024-09-11T01:33:26.809 INFO:tasks.rados.rados.0.smithi071.stdout:689: done (0 left) 2024-09-11T01:33:26.809 INFO:tasks.rados.rados.0.smithi071.stdout:690: copy_from oid 22 from oid 16 current snap is 71 2024-09-11T01:33:26.809 INFO:tasks.rados.rados.0.smithi071.stdout:691: delete oid 4 current snap is 71 2024-09-11T01:33:26.821 INFO:tasks.rados.rados.0.smithi071.stdout:691: done (1 left) 2024-09-11T01:33:26.821 INFO:tasks.rados.rados.0.smithi071.stdout:692: read oid 26 snap -1 2024-09-11T01:33:26.821 INFO:tasks.rados.rados.0.smithi071.stdout:692: expect (ObjNum 244 snap 61 seq_num 244) 2024-09-11T01:33:26.821 INFO:tasks.rados.rados.0.smithi071.stdout:693: copy_from oid 49 from oid 28 current snap is 71 2024-09-11T01:33:26.821 INFO:tasks.rados.rados.0.smithi071.stdout:694: read oid 46 snap -1 2024-09-11T01:33:26.821 INFO:tasks.rados.rados.0.smithi071.stdout:694: expect (ObjNum 211 snap 49 seq_num 211) 2024-09-11T01:33:26.821 INFO:tasks.rados.rados.0.smithi071.stdout:695: read oid 24 snap 63 2024-09-11T01:33:26.821 INFO:tasks.rados.rados.0.smithi071.stdout:695: expect (ObjNum 176 snap 37 seq_num 176) 2024-09-11T01:33:26.822 INFO:tasks.rados.rados.0.smithi071.stdout:696: read oid 6 snap 70 2024-09-11T01:33:26.822 INFO:tasks.rados.rados.0.smithi071.stdout:696: expect (ObjNum 251 snap 63 seq_num 251) 2024-09-11T01:33:26.822 INFO:tasks.rados.rados.0.smithi071.stdout:697: write oid 35 current snap is 71 2024-09-11T01:33:26.822 INFO:tasks.rados.rados.0.smithi071.stdout:697: seq_num 267 ranges {487469=563838,1607751=551752} 2024-09-11T01:33:26.827 INFO:tasks.rados.rados.0.smithi071.stdout:697: writing smithi07134777-35 from 487469 to 1051307 tid 1 2024-09-11T01:33:26.834 INFO:tasks.rados.rados.0.smithi071.stdout:697: writing smithi07134777-35 from 1607751 to 2159503 tid 2 2024-09-11T01:33:26.834 INFO:tasks.rados.rados.0.smithi071.stdout:698: delete oid 38 current snap is 71 2024-09-11T01:33:26.834 INFO:tasks.rados.rados.0.smithi071.stdout:692: expect (ObjNum 244 snap 61 seq_num 244) 2024-09-11T01:33:26.845 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1388415 2024-09-11T01:33:26.890 INFO:tasks.rados.rados.0.smithi071.stdout:694: expect (ObjNum 211 snap 49 seq_num 211) 2024-09-11T01:33:26.890 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:33:26.967 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 318235 2024-09-11T01:33:27.114 INFO:tasks.rados.rados.0.smithi071.stdout:690: finishing copy_from racing read to smithi07134777-22 2024-09-11T01:33:27.115 INFO:tasks.rados.rados.0.smithi071.stdout:695: expect (ObjNum 176 snap 37 seq_num 176) 2024-09-11T01:33:27.264 INFO:tasks.rados.rados.0.smithi071.stdout:690: finishing copy_from to smithi07134777-22 2024-09-11T01:33:27.265 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 210 (ObjNum 240 snap 58 seq_num 240) dirty exists 2024-09-11T01:33:27.265 INFO:tasks.rados.rados.0.smithi071.stdout:696: expect (ObjNum 251 snap 63 seq_num 251) 2024-09-11T01:33:27.265 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:33:27.429 INFO:tasks.rados.rados.0.smithi071.stdout:697: finishing write tid 1 to smithi07134777-35 2024-09-11T01:33:27.429 INFO:tasks.rados.rados.0.smithi071.stdout:693: finishing copy_from racing read to smithi07134777-49 2024-09-11T01:33:27.429 INFO:tasks.rados.rados.0.smithi071.stdout:693: finishing copy_from to smithi07134777-49 2024-09-11T01:33:27.429 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 215 (ObjNum 200 snap 46 seq_num 200) dirty exists 2024-09-11T01:33:27.429 INFO:tasks.rados.rados.0.smithi071.stdout:697: finishing write tid 2 to smithi07134777-35 2024-09-11T01:33:27.429 INFO:tasks.rados.rados.0.smithi071.stdout:697: finishing write tid 3 to smithi07134777-35 2024-09-11T01:33:27.429 INFO:tasks.rados.rados.0.smithi071.stdout:697: finishing write tid 4 to smithi07134777-35 2024-09-11T01:33:27.429 INFO:tasks.rados.rados.0.smithi071.stdout:697: oid 35 updating version 0 to 183 2024-09-11T01:33:27.429 INFO:tasks.rados.rados.0.smithi071.stdout:697: oid 35 version 183 is already newer than 182 2024-09-11T01:33:27.430 INFO:tasks.rados.rados.0.smithi071.stdout:697: oid 35 updating version 183 to 184 2024-09-11T01:33:27.430 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 184 (ObjNum 267 snap 71 seq_num 267) dirty exists 2024-09-11T01:33:27.430 INFO:tasks.rados.rados.0.smithi071.stdout:697: left oid 35 (ObjNum 267 snap 71 seq_num 267) 2024-09-11T01:33:27.430 INFO:tasks.rados.rados.0.smithi071.stdout:690: done (7 left) 2024-09-11T01:33:27.430 INFO:tasks.rados.rados.0.smithi071.stdout:692: done (6 left) 2024-09-11T01:33:27.430 INFO:tasks.rados.rados.0.smithi071.stdout:693: done (5 left) 2024-09-11T01:33:27.430 INFO:tasks.rados.rados.0.smithi071.stdout:694: done (4 left) 2024-09-11T01:33:27.430 INFO:tasks.rados.rados.0.smithi071.stdout:695: done (3 left) 2024-09-11T01:33:27.430 INFO:tasks.rados.rados.0.smithi071.stdout:696: done (2 left) 2024-09-11T01:33:27.430 INFO:tasks.rados.rados.0.smithi071.stdout:697: done (1 left) 2024-09-11T01:33:27.430 INFO:tasks.rados.rados.0.smithi071.stdout:698: done (0 left) 2024-09-11T01:33:27.430 INFO:tasks.rados.rados.0.smithi071.stdout:699: write (excl) oid 38 current snap is 71 2024-09-11T01:33:27.430 INFO:tasks.rados.rados.0.smithi071.stdout:699: seq_num 269 ranges {521083=714336,1724715=555421} 2024-09-11T01:33:27.435 INFO:tasks.rados.rados.0.smithi071.stdout:699: writing smithi07134777-38 from 521083 to 1235419 tid 1 2024-09-11T01:33:27.440 INFO:tasks.rados.rados.0.smithi071.stdout:699: writing smithi07134777-38 from 1724715 to 2280136 tid 2 2024-09-11T01:33:27.441 INFO:tasks.rados.rados.0.smithi071.stdout:700: snap_remove snap 69 2024-09-11T01:33:27.849 INFO:tasks.rados.rados.0.smithi071.stdout:700: done (1 left) 2024-09-11T01:33:27.849 INFO:tasks.rados.rados.0.smithi071.stdout:701: write oid 4 current snap is 71 2024-09-11T01:33:27.849 INFO:tasks.rados.rados.0.smithi071.stdout:701: seq_num 270 ranges {537890=789585,1783197=699079} 2024-09-11T01:33:27.856 INFO:tasks.rados.rados.0.smithi071.stdout:701: writing smithi07134777-4 from 537890 to 1327475 tid 1 2024-09-11T01:33:27.863 INFO:tasks.rados.rados.0.smithi071.stdout:701: writing smithi07134777-4 from 1783197 to 2482276 tid 2 2024-09-11T01:33:27.863 INFO:tasks.rados.rados.0.smithi071.stdout:702: delete oid 21 current snap is 71 2024-09-11T01:33:27.863 INFO:tasks.rados.rados.0.smithi071.stdout:699: finishing write tid 1 to smithi07134777-38 2024-09-11T01:33:27.863 INFO:tasks.rados.rados.0.smithi071.stdout:699: finishing write tid 2 to smithi07134777-38 2024-09-11T01:33:27.863 INFO:tasks.rados.rados.0.smithi071.stdout:699: finishing write tid 3 to smithi07134777-38 2024-09-11T01:33:27.863 INFO:tasks.rados.rados.0.smithi071.stdout:699: finishing write tid 4 to smithi07134777-38 2024-09-11T01:33:27.863 INFO:tasks.rados.rados.0.smithi071.stdout:699: oid 38 updating version 0 to 333 2024-09-11T01:33:27.863 INFO:tasks.rados.rados.0.smithi071.stdout:699: oid 38 version 333 is already newer than 332 2024-09-11T01:33:27.863 INFO:tasks.rados.rados.0.smithi071.stdout:699: oid 38 version 333 is already newer than 331 2024-09-11T01:33:27.863 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 333 (ObjNum 269 snap 71 seq_num 269) dirty exists 2024-09-11T01:33:27.863 INFO:tasks.rados.rados.0.smithi071.stdout:699: left oid 38 (ObjNum 269 snap 71 seq_num 269) 2024-09-11T01:33:27.864 INFO:tasks.rados.rados.0.smithi071.stdout:701: finishing write tid 1 to smithi07134777-4 2024-09-11T01:33:27.864 INFO:tasks.rados.rados.0.smithi071.stdout:699: done (2 left) 2024-09-11T01:33:27.865 INFO:tasks.rados.rados.0.smithi071.stdout:702: done (1 left) 2024-09-11T01:33:27.865 INFO:tasks.rados.rados.0.smithi071.stdout:703: copy_from oid 21 from oid 43 current snap is 71 2024-09-11T01:33:27.865 INFO:tasks.rados.rados.0.smithi071.stdout:704: rollback oid 28 current snap is 71 2024-09-11T01:33:27.865 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 28 to 67 2024-09-11T01:33:27.865 INFO:tasks.rados.rados.0.smithi071.stdout:705: snap_remove snap 70 2024-09-11T01:33:28.851 INFO:tasks.rados.rados.0.smithi071.stdout:705: done (3 left) 2024-09-11T01:33:28.851 INFO:tasks.rados.rados.0.smithi071.stdout:706: snap_create 2024-09-11T01:33:28.851 INFO:tasks.rados.rados.0.smithi071.stdout:701: finishing write tid 2 to smithi07134777-4 2024-09-11T01:33:28.851 INFO:tasks.rados.rados.0.smithi071.stdout:701: finishing write tid 3 to smithi07134777-4 2024-09-11T01:33:28.851 INFO:tasks.rados.rados.0.smithi071.stdout:701: finishing write tid 4 to smithi07134777-4 2024-09-11T01:33:28.851 INFO:tasks.rados.rados.0.smithi071.stdout:701: oid 4 updating version 0 to 227 2024-09-11T01:33:28.851 INFO:tasks.rados.rados.0.smithi071.stdout:701: oid 4 updating version 227 to 228 2024-09-11T01:33:28.851 INFO:tasks.rados.rados.0.smithi071.stdout:701: oid 4 updating version 228 to 229 2024-09-11T01:33:28.852 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 229 (ObjNum 270 snap 71 seq_num 270) dirty exists 2024-09-11T01:33:28.852 INFO:tasks.rados.rados.0.smithi071.stdout:701: left oid 4 (ObjNum 270 snap 71 seq_num 270) 2024-09-11T01:33:28.852 INFO:tasks.rados.rados.0.smithi071.stdout:704: finishing rollback tid 0 to smithi07134777-28 2024-09-11T01:33:28.852 INFO:tasks.rados.rados.0.smithi071.stdout:703: finishing copy_from racing read to smithi07134777-21 2024-09-11T01:33:28.852 INFO:tasks.rados.rados.0.smithi071.stdout:704: finishing rollback tid 1 to smithi07134777-28 2024-09-11T01:33:28.852 INFO:tasks.rados.rados.0.smithi071.stdout:704: finishing rollback tid 2 to smithi07134777-28 2024-09-11T01:33:28.852 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 221 (ObjNum 200 snap 46 seq_num 200) dirty exists 2024-09-11T01:33:28.852 INFO:tasks.rados.rados.0.smithi071.stdout:703: finishing copy_from to smithi07134777-21 2024-09-11T01:33:28.852 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 180 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-11T01:33:29.864 INFO:tasks.rados.rados.0.smithi071.stdout:701: done (3 left) 2024-09-11T01:33:29.864 INFO:tasks.rados.rados.0.smithi071.stdout:703: done (2 left) 2024-09-11T01:33:29.864 INFO:tasks.rados.rados.0.smithi071.stdout:704: done (1 left) 2024-09-11T01:33:29.864 INFO:tasks.rados.rados.0.smithi071.stdout:706: done (0 left) 2024-09-11T01:33:29.864 INFO:tasks.rados.rados.0.smithi071.stdout:707: write oid 7 current snap is 72 2024-09-11T01:33:29.864 INFO:tasks.rados.rados.0.smithi071.stdout:707: seq_num 272 ranges {571504=540083,1783808=702748} 2024-09-11T01:33:29.870 INFO:tasks.rados.rados.0.smithi071.stdout:707: writing smithi07134777-7 from 571504 to 1111587 tid 1 2024-09-11T01:33:29.878 INFO:tasks.rados.rados.0.smithi071.stdout:707: writing smithi07134777-7 from 1783808 to 2486556 tid 2 2024-09-11T01:33:29.878 INFO:tasks.rados.rados.0.smithi071.stdout:708: read oid 20 snap -1 2024-09-11T01:33:29.878 INFO:tasks.rados.rados.0.smithi071.stdout:708: expect (ObjNum 114 snap 27 seq_num 114) 2024-09-11T01:33:29.878 INFO:tasks.rados.rados.0.smithi071.stdout:709: write oid 21 current snap is 72 2024-09-11T01:33:29.878 INFO:tasks.rados.rados.0.smithi071.stdout:709: seq_num 273 ranges {588311=615332,1842290=562759} 2024-09-11T01:33:29.885 INFO:tasks.rados.rados.0.smithi071.stdout:709: writing smithi07134777-21 from 588311 to 1203643 tid 1 2024-09-11T01:33:29.892 INFO:tasks.rados.rados.0.smithi071.stdout:709: writing smithi07134777-21 from 1842290 to 2405049 tid 2 2024-09-11T01:33:29.892 INFO:tasks.rados.rados.0.smithi071.stdout:710: snap_remove snap 71 2024-09-11T01:33:29.925 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-11T01:33:29.925 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T01:33:29.925 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T01:33:29.928 INFO:tasks.ceph.mon.a:Started 2024-09-11T01:33:29.928 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:33:29.928 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:33:30.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.039+0000 7f75065e2640 10 monclient: get_monmap_and_config 2024-09-11T01:33:30.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.039+0000 7f75065e2640 10 monclient: build_initial_monmap 2024-09-11T01:33:30.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.039+0000 7f75065e2640 10 monclient: monmap: 2024-09-11T01:33:30.042 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:30.042 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:30.042 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:33:30.041058+0000 2024-09-11T01:33:30.042 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:33:30.041058+0000 2024-09-11T01:33:30.042 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:30.042 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:30.042 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:33:30.042 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:33:30.042 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:33:30.042 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:33:30.043 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:33:30.043 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:30.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.040+0000 7f75065e2640 1 Processor -- start 2024-09-11T01:33:30.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.040+0000 7f75065e2640 1 -- start start 2024-09-11T01:33:30.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.040+0000 7f75065e2640 10 monclient: init 2024-09-11T01:33:30.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.040+0000 7f75065e2640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:30.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.040+0000 7f75065e2640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-11T01:33:30.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.040+0000 7f75065e2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7500138230 addr v1:172.21.15.12:6789/0 2024-09-11T01:33:30.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.040+0000 7f75065e2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f750009d030 0x7f750013d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:30.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.040+0000 7f75065e2640 10 monclient(hunting): picked mon.noname-b con 0x7f750009d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:30.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.040+0000 7f75065e2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f750009d460 addr v1:172.21.15.71:6789/0 2024-09-11T01:33:30.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.040+0000 7f75065e2640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75000a6e30 con 0x7f7500138230 2024-09-11T01:33:30.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.040+0000 7f75065e2640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f750009ba20 con 0x7f750009d460 2024-09-11T01:33:30.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.040+0000 7f75065e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:30.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.040+0000 7f75065e2640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f750009c9c0 con 0x7f750009d030 2024-09-11T01:33:30.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.040+0000 7f75065e2640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:30.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.040+0000 7f75065e2640 10 monclient(hunting): authenticate will time out at 1016.483979s 2024-09-11T01:33:30.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.040+0000 7f74ff7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f750009d030 0x7f750013d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:30.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.041+0000 7f74ff7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f750009d030 0x7f750013d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:34834/0 (socket says 172.21.15.12:34834) 2024-09-11T01:33:30.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.041+0000 7f74fffff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7500138230 0x7f7500138610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46288/0 (socket says 172.21.15.12:46288) 2024-09-11T01:33:30.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.041+0000 7f74ff7fe640 1 -- 172.21.15.12:0/3468248443 learned_addr learned my addr 172.21.15.12:0/3468248443 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:30.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.041+0000 7f74ff7fe640 10 monclient(hunting): get_auth_request con 0x7f750009d030 auth_method 0 2024-09-11T01:33:30.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.041+0000 7f74ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:30.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.041+0000 7f74ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:30.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.041+0000 7f74ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:30.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.041+0000 7f74ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:30.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.041+0000 7f74ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:30.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.041+0000 7f74ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:30.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.041+0000 7f74ff7fe640 10 monclient(hunting): handle_auth_done global_id 44183 payload 274 2024-09-11T01:33:30.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.041+0000 7f74ff7fe640 1 -- 172.21.15.12:0/3468248443 >> v1:172.21.15.71:6789/0 conn(0x7f750009d460 legacy=0x7f750013f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:30.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.041+0000 7f74ff7fe640 1 -- 172.21.15.12:0/3468248443 >> v1:172.21.15.12:6789/0 conn(0x7f7500138230 legacy=0x7f7500138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:30.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.041+0000 7f74ff7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:33:30.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.041+0000 7f74ff7fe640 1 monclient: found mon.noname-b 2024-09-11T01:33:30.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.041+0000 7f74ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:30.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.041+0000 7f74ff7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:33:30.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.041+0000 7f74ff7fe640 1 -- 172.21.15.12:0/3468248443 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75000a66a0 con 0x7f750009d030 2024-09-11T01:33:30.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.044+0000 7f74ff7fe640 1 -- 172.21.15.12:0/3468248443 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f750009d030 msgr2=0x7f750013d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:30.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.044+0000 7f74ff7fe640 1 --2- 172.21.15.12:0/3468248443 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f750009d030 0x7f750013d090 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f750006e500 tx=0x7f74f402a050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c59491a2238725d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:30.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.044+0000 7f74feffd640 1 -- 172.21.15.12:0/3468248443 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f4037070 con 0x7f750009d030 2024-09-11T01:33:30.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.044+0000 7f74feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:30.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.044+0000 7f74feffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:33:30.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.044+0000 7f74feffd640 10 monclient: dump: 2024-09-11T01:33:30.058 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:30.058 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:30.058 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:30.058 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:30.058 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:30.058 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:30.058 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:30.058 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:30.058 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:30.058 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:30.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.044+0000 7f74feffd640 10 monclient: _finish_auth 0 2024-09-11T01:33:30.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.044+0000 7f74feffd640 10 monclient: _check_auth_tickets 2024-09-11T01:33:30.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.044+0000 7f74feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:30.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.044+0000 7f74feffd640 1 -- 172.21.15.12:0/3468248443 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74f4032640 con 0x7f750009d030 2024-09-11T01:33:30.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.044+0000 7f75065e2640 5 monclient: authenticate success, global_id 44183 2024-09-11T01:33:30.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.044+0000 7f75065e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:33:30.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.044+0000 7f74feffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:30.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.044+0000 7f75065e2640 10 monclient: get_monmap_and_config success 2024-09-11T01:33:30.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.044+0000 7f74feffd640 1 -- 172.21.15.12:0/3468248443 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f4032a10 con 0x7f750009d030 2024-09-11T01:33:30.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.044+0000 7f75065e2640 4 set_mon_vals no callback set 2024-09-11T01:33:30.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.044+0000 7f74feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:30.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.044+0000 7f74feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:30.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.044+0000 7f74feffd640 10 monclient: dump: 2024-09-11T01:33:30.060 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:30.060 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:30.060 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:30.060 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:30.060 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:30.060 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:30.060 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:30.060 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:30.060 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:30.060 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:30.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.044+0000 7f75065e2640 10 monclient: shutdown 2024-09-11T01:33:30.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.044+0000 7f75065e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:30.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.044+0000 7f75065e2640 1 -- 172.21.15.12:0/3468248443 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f750009d030 msgr2=0x7f750013d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.044+0000 7f75065e2640 1 --2- 172.21.15.12:0/3468248443 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f750009d030 0x7f750013d090 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f750006e500 tx=0x7f74f402a050 comp rx=0 tx=0).stop 2024-09-11T01:33:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.044+0000 7f75065e2640 1 -- 172.21.15.12:0/3468248443 shutdown_connections 2024-09-11T01:33:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.044+0000 7f75065e2640 1 --2- 172.21.15.12:0/3468248443 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f750009d030 0x7f750013d090 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.044+0000 7f75065e2640 1 -- 172.21.15.12:0/3468248443 >> 172.21.15.12:0/3468248443 conn(0x7f75000a7120 msgr2=0x7f75000a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.045+0000 7f75065e2640 1 -- 172.21.15.12:0/3468248443 shutdown_connections 2024-09-11T01:33:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.045+0000 7f75065e2640 1 -- 172.21.15.12:0/3468248443 wait complete. 2024-09-11T01:33:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.045+0000 7f75065e2640 10 monclient: build_initial_monmap 2024-09-11T01:33:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.045+0000 7f75065e2640 10 monclient: monmap: 2024-09-11T01:33:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:33:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:33:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:33:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:33:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:33:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:33:30.062 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.045+0000 7f75065e2640 1 Processor -- start 2024-09-11T01:33:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f75065e2640 1 -- start start 2024-09-11T01:33:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f75065e2640 10 monclient: init 2024-09-11T01:33:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f75065e2640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f75065e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:33:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f75065e2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f750009d030 0x7f75001cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f75065e2640 10 monclient(hunting): picked mon.noname-c con 0x7f750009d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f75065e2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f750009d460 0x7f7500137090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:30.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f75065e2640 10 monclient(hunting): picked mon.noname-b con 0x7f750009d460 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:33:30.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f75065e2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7500138230 0x7f750013cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:30.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f75065e2640 10 monclient(hunting): picked mon.noname-a con 0x7f7500138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:33:30.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f75065e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:30.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f75065e2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7500140e80 con 0x7f7500138230 2024-09-11T01:33:30.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f75065e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:30.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f75065e2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f7500140d00 con 0x7f750009d460 2024-09-11T01:33:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f75065e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f75065e2640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7500141000 con 0x7f750009d030 2024-09-11T01:33:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f75065e2640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f75065e2640 10 monclient(hunting): authenticate will time out at 1016.489636s 2024-09-11T01:33:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f74fffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f750009d030 0x7f75001cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f74fffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f750009d030 0x7f75001cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:34846/0 (socket says 172.21.15.12:34846) 2024-09-11T01:33:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f74fffff640 1 -- 172.21.15.12:0/1962325008 learned_addr learned my addr 172.21.15.12:0/1962325008 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f74fffff640 10 monclient(hunting): get_auth_request con 0x7f750009d030 auth_method 0 2024-09-11T01:33:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f74fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f74fffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f74fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f7504b7f640 1 --2- 172.21.15.12:0/1962325008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7500138230 0x7f750013cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f7504b7f640 10 monclient(hunting): get_auth_request con 0x7f7500138230 auth_method 0 2024-09-11T01:33:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f7504b7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f7504b7f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f7504b7f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f74fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f74fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f74fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f7504b7f640 1 -- 172.21.15.12:0/1962325008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7500138230 msgr2=0x7f750013cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:33:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f7504b7f640 1 -- 172.21.15.12:0/1962325008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7500138230 msgr2=0x7f750013cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:33:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f7504b7f640 1 --2- 172.21.15.12:0/1962325008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7500138230 0x7f750013cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f7504b7f640 1 --2- 172.21.15.12:0/1962325008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7500138230 0x7f750013cd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:33:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.046+0000 7f74ff7fe640 1 --2- 172.21.15.12:0/1962325008 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f750009d460 0x7f7500137090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.047+0000 7f74ff7fe640 10 monclient(hunting): get_auth_request con 0x7f750009d460 auth_method 0 2024-09-11T01:33:30.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.047+0000 7f74ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:30.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.047+0000 7f74ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:30.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.047+0000 7f74ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:30.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.047+0000 7f74fffff640 10 monclient(hunting): handle_auth_done global_id 44186 payload 824 2024-09-11T01:33:30.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.047+0000 7f74fffff640 1 -- 172.21.15.12:0/1962325008 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f750009d460 msgr2=0x7f7500137090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:30.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.047+0000 7f74fffff640 1 --2- 172.21.15.12:0/1962325008 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f750009d460 0x7f7500137090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:30.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.047+0000 7f74fffff640 1 -- 172.21.15.12:0/1962325008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7500138230 msgr2=0x7f750013cd10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:33:30.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.047+0000 7f74fffff640 1 --2- 172.21.15.12:0/1962325008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7500138230 0x7f750013cd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:30.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.047+0000 7f74fffff640 10 monclient: _finish_hunting 0 2024-09-11T01:33:30.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.047+0000 7f74fffff640 1 monclient: found mon.noname-c 2024-09-11T01:33:30.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.047+0000 7f74fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:30.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.047+0000 7f74fffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:33:30.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.047+0000 7f74fffff640 1 -- 172.21.15.12:0/1962325008 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7500140760 con 0x7f750009d030 2024-09-11T01:33:30.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.049+0000 7f74fffff640 1 -- 172.21.15.12:0/1962325008 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f750009d030 msgr2=0x7f75001cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:30.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.049+0000 7f74fffff640 1 --2- 172.21.15.12:0/1962325008 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f750009d030 0x7f75001cd1e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f74f000c450 tx=0x7f74f000c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:30.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.049+0000 7f74fcff9640 1 -- 172.21.15.12:0/1962325008 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f0017070 con 0x7f750009d030 2024-09-11T01:33:30.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.049+0000 7f74fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:30.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.049+0000 7f74fcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:33:30.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.049+0000 7f74fcff9640 10 monclient: dump: 2024-09-11T01:33:30.067 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:30.067 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:30.067 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:30.067 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:30.067 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:30.067 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:30.067 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:30.067 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:30.067 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:30.067 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:30.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.049+0000 7f74fcff9640 10 monclient: _finish_auth 0 2024-09-11T01:33:30.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.049+0000 7f74fcff9640 10 monclient: _check_auth_tickets 2024-09-11T01:33:30.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.049+0000 7f74fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:30.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.049+0000 7f74fcff9640 1 -- 172.21.15.12:0/1962325008 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74f0005810 con 0x7f750009d030 2024-09-11T01:33:30.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.049+0000 7f75065e2640 5 monclient: authenticate success, global_id 44186 2024-09-11T01:33:30.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.049+0000 7f75065e2640 10 monclient: _renew_subs 2024-09-11T01:33:30.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.049+0000 7f75065e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:30.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.049+0000 7f75065e2640 1 -- 172.21.15.12:0/1962325008 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75000a14a0 con 0x7f750009d030 2024-09-11T01:33:30.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.049+0000 7f74fcff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:30.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.049+0000 7f74fcff9640 1 -- 172.21.15.12:0/1962325008 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f0007690 con 0x7f750009d030 2024-09-11T01:33:30.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.049+0000 7f74fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:30.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.049+0000 7f74fcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:30.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.049+0000 7f74fcff9640 10 monclient: dump: 2024-09-11T01:33:30.069 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:30.069 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:30.069 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:30.069 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:30.069 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:30.069 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:30.069 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:30.069 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:30.069 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:30.069 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:30.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.049+0000 7f75065e2640 10 monclient: _renew_subs 2024-09-11T01:33:30.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.049+0000 7f75065e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:30.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.049+0000 7f75065e2640 1 -- 172.21.15.12:0/1962325008 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f75000a1810 con 0x7f750009d030 2024-09-11T01:33:30.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.049+0000 7f74fe7fc640 4 set_mon_vals no callback set 2024-09-11T01:33:30.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.050+0000 7f75065e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:30.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.050+0000 7f75065e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:30.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.050+0000 7f75065e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:30.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.050+0000 7f74fcff9640 1 -- 172.21.15.12:0/1962325008 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f74f0005c40 con 0x7f750009d030 2024-09-11T01:33:30.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.050+0000 7f75065e2640 1 -- 172.21.15.12:0/1962325008 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7500138610 con 0x7f750009d030 2024-09-11T01:33:30.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.050+0000 7f74fcff9640 1 --2- 172.21.15.12:0/1962325008 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f74ec03e210 0x7f74ec0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:30.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.050+0000 7f74fcff9640 1 -- 172.21.15.12:0/1962325008 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(161..161 src has 1..161) ==== 4755+0+0 (secure 0 0 0) 0x7f74f0052440 con 0x7f750009d030 2024-09-11T01:33:30.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.051+0000 7f74ff7fe640 1 --2- 172.21.15.12:0/1962325008 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f74ec03e210 0x7f74ec0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:30.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.051+0000 7f74ff7fe640 10 monclient: get_auth_request con 0x7f74ec03e210 auth_method 0 2024-09-11T01:33:30.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.053+0000 7f74ff7fe640 1 --2- 172.21.15.12:0/1962325008 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f74ec03e210 0x7f74ec0406d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f74f4034000 tx=0x7f74f402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.055+0000 7f74fcff9640 1 -- 172.21.15.12:0/1962325008 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f74f0050e50 con 0x7f750009d030 2024-09-11T01:33:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.055+0000 7f74fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.055+0000 7f74fcff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:33:30.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.309+0000 7f75065e2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:33:30.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.309+0000 7f75065e2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:33:30.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.309+0000 7f75065e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:30.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.309+0000 7f75065e2640 1 -- 172.21.15.12:0/1962325008 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7500140540 con 0x7f750009d030 2024-09-11T01:33:30.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.309+0000 7f74fcff9640 1 -- 172.21.15.12:0/1962325008 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f74f001bd90 con 0x7f750009d030 2024-09-11T01:33:30.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.309+0000 7f74fcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:33:30.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.309+0000 7f74fcff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:33:30.312 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":40,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:33:30.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.311+0000 7f75065e2640 1 -- 172.21.15.12:0/1962325008 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f74ec03e210 msgr2=0x7f74ec0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:30.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.311+0000 7f75065e2640 1 --2- 172.21.15.12:0/1962325008 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f74ec03e210 0x7f74ec0406d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f74f4034000 tx=0x7f74f402e000 comp rx=0 tx=0).stop 2024-09-11T01:33:30.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.311+0000 7f75065e2640 10 monclient: shutdown 2024-09-11T01:33:30.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.311+0000 7f75065e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:30.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.311+0000 7f75065e2640 1 -- 172.21.15.12:0/1962325008 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f750009d030 msgr2=0x7f75001cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:30.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.311+0000 7f75065e2640 1 --2- 172.21.15.12:0/1962325008 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f750009d030 0x7f75001cd1e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f74f000c450 tx=0x7f74f000c910 comp rx=0 tx=0).stop 2024-09-11T01:33:30.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.311+0000 7f75065e2640 1 -- 172.21.15.12:0/1962325008 shutdown_connections 2024-09-11T01:33:30.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.311+0000 7f75065e2640 1 --2- 172.21.15.12:0/1962325008 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f74ec03e210 0x7f74ec0406d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:30.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.311+0000 7f75065e2640 1 --2- 172.21.15.12:0/1962325008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7500138230 0x7f750013cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:30.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.311+0000 7f75065e2640 1 --2- 172.21.15.12:0/1962325008 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f750009d460 0x7f7500137090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:30.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.311+0000 7f75065e2640 1 --2- 172.21.15.12:0/1962325008 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f750009d030 0x7f75001cd1e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:30.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.311+0000 7f75065e2640 1 -- 172.21.15.12:0/1962325008 >> 172.21.15.12:0/1962325008 conn(0x7f75000a7120 msgr2=0x7f75000a0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:30.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.312+0000 7f75065e2640 1 -- 172.21.15.12:0/1962325008 shutdown_connections 2024-09-11T01:33:30.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:30.312+0000 7f75065e2640 1 -- 172.21.15.12:0/1962325008 wait complete. 2024-09-11T01:33:30.866 INFO:tasks.rados.rados.0.smithi071.stdout:710: done (3 left) 2024-09-11T01:33:30.866 INFO:tasks.rados.rados.0.smithi071.stdout:711: rollback oid 42 current snap is 72 2024-09-11T01:33:30.866 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 42 to 62 2024-09-11T01:33:30.866 INFO:tasks.rados.rados.0.smithi071.stdout:707: finishing write tid 1 to smithi07134777-7 2024-09-11T01:33:30.866 INFO:tasks.rados.rados.0.smithi071.stdout:707: finishing write tid 2 to smithi07134777-7 2024-09-11T01:33:30.866 INFO:tasks.rados.rados.0.smithi071.stdout:708: expect (ObjNum 114 snap 27 seq_num 114) 2024-09-11T01:33:30.866 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:33:30.940 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1702913 2024-09-11T01:33:30.977 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:33:31.037 INFO:tasks.rados.rados.0.smithi071.stdout:707: finishing write tid 3 to smithi07134777-7 2024-09-11T01:33:31.037 INFO:tasks.rados.rados.0.smithi071.stdout:707: finishing write tid 4 to smithi07134777-7 2024-09-11T01:33:31.038 INFO:tasks.rados.rados.0.smithi071.stdout:707: oid 7 updating version 0 to 337 2024-09-11T01:33:31.038 INFO:tasks.rados.rados.0.smithi071.stdout:707: oid 7 version 337 is already newer than 336 2024-09-11T01:33:31.038 INFO:tasks.rados.rados.0.smithi071.stdout:707: oid 7 version 337 is already newer than 335 2024-09-11T01:33:31.038 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 337 (ObjNum 272 snap 72 seq_num 272) dirty exists 2024-09-11T01:33:31.038 INFO:tasks.rados.rados.0.smithi071.stdout:707: left oid 7 (ObjNum 272 snap 72 seq_num 272) 2024-09-11T01:33:31.038 INFO:tasks.rados.rados.0.smithi071.stdout:709: finishing write tid 1 to smithi07134777-21 2024-09-11T01:33:31.038 INFO:tasks.rados.rados.0.smithi071.stdout:709: finishing write tid 2 to smithi07134777-21 2024-09-11T01:33:31.038 INFO:tasks.rados.rados.0.smithi071.stdout:709: finishing write tid 3 to smithi07134777-21 2024-09-11T01:33:31.038 INFO:tasks.rados.rados.0.smithi071.stdout:709: finishing write tid 4 to smithi07134777-21 2024-09-11T01:33:31.038 INFO:tasks.rados.rados.0.smithi071.stdout:709: oid 21 updating version 0 to 223 2024-09-11T01:33:31.038 INFO:tasks.rados.rados.0.smithi071.stdout:709: oid 21 updating version 223 to 224 2024-09-11T01:33:31.038 INFO:tasks.rados.rados.0.smithi071.stdout:709: oid 21 version 224 is already newer than 222 2024-09-11T01:33:31.038 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 224 (ObjNum 273 snap 72 seq_num 273) dirty exists 2024-09-11T01:33:31.039 INFO:tasks.rados.rados.0.smithi071.stdout:709: left oid 21 (ObjNum 273 snap 72 seq_num 273) 2024-09-11T01:33:31.039 INFO:tasks.rados.rados.0.smithi071.stdout:707: done (3 left) 2024-09-11T01:33:31.039 INFO:tasks.rados.rados.0.smithi071.stdout:708: done (2 left) 2024-09-11T01:33:31.039 INFO:tasks.rados.rados.0.smithi071.stdout:709: done (1 left) 2024-09-11T01:33:31.039 INFO:tasks.rados.rados.0.smithi071.stdout:712: read oid 19 snap 67 2024-09-11T01:33:31.039 INFO:tasks.rados.rados.0.smithi071.stdout:712: expect (ObjNum 255 snap 67 seq_num 255) 2024-09-11T01:33:31.039 INFO:tasks.rados.rados.0.smithi071.stdout:713: write (excl) oid 43 current snap is 72 2024-09-11T01:33:31.039 INFO:tasks.rados.rados.0.smithi071.stdout:711: finishing rollback tid 1 to smithi07134777-42 2024-09-11T01:33:31.039 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 0 (ObjNum 129 snap 0 seq_num 2370805456) dirty dne 2024-09-11T01:33:31.039 INFO:tasks.rados.rados.0.smithi071.stdout:713: seq_num 274 ranges {605118=406934,1617125=706417} 2024-09-11T01:33:31.042 INFO:tasks.rados.rados.0.smithi071.stdout:713: writing smithi07134777-43 from 605118 to 1012052 tid 1 2024-09-11T01:33:31.050 INFO:tasks.rados.rados.0.smithi071.stdout:713: writing smithi07134777-43 from 1617125 to 2323542 tid 2 2024-09-11T01:33:31.050 INFO:tasks.rados.rados.0.smithi071.stdout:711: done (2 left) 2024-09-11T01:33:31.050 INFO:tasks.rados.rados.0.smithi071.stdout:714: snap_create 2024-09-11T01:33:31.050 INFO:tasks.rados.rados.0.smithi071.stdout:712: expect (ObjNum 255 snap 67 seq_num 255) 2024-09-11T01:33:31.073 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:33:31.128 INFO:tasks.rados.rados.0.smithi071.stdout:713: finishing write tid 1 to smithi07134777-43 2024-09-11T01:33:31.128 INFO:tasks.rados.rados.0.smithi071.stdout:713: finishing write tid 2 to smithi07134777-43 2024-09-11T01:33:31.128 INFO:tasks.rados.rados.0.smithi071.stdout:713: finishing write tid 3 to smithi07134777-43 2024-09-11T01:33:31.128 INFO:tasks.rados.rados.0.smithi071.stdout:713: finishing write tid 4 to smithi07134777-43 2024-09-11T01:33:31.128 INFO:tasks.rados.rados.0.smithi071.stdout:713: oid 43 updating version 0 to 226 2024-09-11T01:33:31.128 INFO:tasks.rados.rados.0.smithi071.stdout:713: oid 43 version 226 is already newer than 225 2024-09-11T01:33:31.129 INFO:tasks.rados.rados.0.smithi071.stdout:713: oid 43 version 226 is already newer than 224 2024-09-11T01:33:31.129 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 226 (ObjNum 274 snap 72 seq_num 274) dirty exists 2024-09-11T01:33:31.129 INFO:tasks.rados.rados.0.smithi071.stdout:713: left oid 43 (ObjNum 274 snap 72 seq_num 274) 2024-09-11T01:33:31.810 INFO:tasks.rados.rados.0.smithi071.stdout:712: done (2 left) 2024-09-11T01:33:31.810 INFO:tasks.rados.rados.0.smithi071.stdout:713: done (1 left) 2024-09-11T01:33:31.810 INFO:tasks.rados.rados.0.smithi071.stdout:714: done (0 left) 2024-09-11T01:33:31.810 INFO:tasks.rados.rados.0.smithi071.stdout:715: read oid 15 snap -1 2024-09-11T01:33:31.810 INFO:tasks.rados.rados.0.smithi071.stdout:715: expect (ObjNum 253 snap 66 seq_num 253) 2024-09-11T01:33:31.810 INFO:tasks.rados.rados.0.smithi071.stdout:716: rollback oid 22 current snap is 73 2024-09-11T01:33:31.810 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 22 to 68 2024-09-11T01:33:31.811 INFO:tasks.rados.rados.0.smithi071.stdout:717: write (excl) oid 45 current snap is 73 2024-09-11T01:33:31.811 INFO:tasks.rados.rados.0.smithi071.stdout:717: seq_num 275 ranges {621925=482183,1559254=566428} 2024-09-11T01:33:31.816 INFO:tasks.rados.rados.0.smithi071.stdout:717: writing smithi07134777-45 from 621925 to 1104108 tid 1 2024-09-11T01:33:31.822 INFO:tasks.rados.rados.0.smithi071.stdout:717: writing smithi07134777-45 from 1559254 to 2125682 tid 2 2024-09-11T01:33:31.822 INFO:tasks.rados.rados.0.smithi071.stdout:718: rollback oid 30 current snap is 73 2024-09-11T01:33:31.822 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 30 to 63 2024-09-11T01:33:31.822 INFO:tasks.rados.rados.0.smithi071.stdout:715: expect (ObjNum 253 snap 66 seq_num 253) 2024-09-11T01:33:31.822 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:33:31.873 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:33:31.899 INFO:tasks.rados.rados.0.smithi071.stdout:716: finishing rollback tid 0 to smithi07134777-22 2024-09-11T01:33:31.899 INFO:tasks.rados.rados.0.smithi071.stdout:716: finishing rollback tid 1 to smithi07134777-22 2024-09-11T01:33:31.899 INFO:tasks.rados.rados.0.smithi071.stdout:716: finishing rollback tid 2 to smithi07134777-22 2024-09-11T01:33:31.899 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 213 (ObjNum 240 snap 58 seq_num 240) dirty exists 2024-09-11T01:33:31.899 INFO:tasks.rados.rados.0.smithi071.stdout:717: finishing write tid 1 to smithi07134777-45 2024-09-11T01:33:31.899 INFO:tasks.rados.rados.0.smithi071.stdout:717: finishing write tid 2 to smithi07134777-45 2024-09-11T01:33:31.899 INFO:tasks.rados.rados.0.smithi071.stdout:717: finishing write tid 3 to smithi07134777-45 2024-09-11T01:33:31.899 INFO:tasks.rados.rados.0.smithi071.stdout:718: finishing rollback tid 1 to smithi07134777-30 2024-09-11T01:33:31.899 INFO:tasks.rados.rados.0.smithi071.stdout:718: finishing rollback tid 2 to smithi07134777-30 2024-09-11T01:33:31.899 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 231 (ObjNum 217 snap 52 seq_num 217) dirty exists 2024-09-11T01:33:31.899 INFO:tasks.rados.rados.0.smithi071.stdout:717: finishing write tid 4 to smithi07134777-45 2024-09-11T01:33:31.899 INFO:tasks.rados.rados.0.smithi071.stdout:717: oid 45 updating version 0 to 225 2024-09-11T01:33:31.900 INFO:tasks.rados.rados.0.smithi071.stdout:717: oid 45 updating version 225 to 226 2024-09-11T01:33:31.900 INFO:tasks.rados.rados.0.smithi071.stdout:717: oid 45 updating version 226 to 227 2024-09-11T01:33:31.900 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 227 (ObjNum 275 snap 73 seq_num 275) dirty exists 2024-09-11T01:33:31.900 INFO:tasks.rados.rados.0.smithi071.stdout:717: left oid 45 (ObjNum 275 snap 73 seq_num 275) 2024-09-11T01:33:31.900 INFO:tasks.rados.rados.0.smithi071.stdout:715: done (3 left) 2024-09-11T01:33:31.900 INFO:tasks.rados.rados.0.smithi071.stdout:716: done (2 left) 2024-09-11T01:33:31.900 INFO:tasks.rados.rados.0.smithi071.stdout:717: done (1 left) 2024-09-11T01:33:31.900 INFO:tasks.rados.rados.0.smithi071.stdout:718: done (0 left) 2024-09-11T01:33:31.900 INFO:tasks.rados.rados.0.smithi071.stdout:719: read oid 35 snap -1 2024-09-11T01:33:31.900 INFO:tasks.rados.rados.0.smithi071.stdout:719: expect (ObjNum 267 snap 71 seq_num 267) 2024-09-11T01:33:31.900 INFO:tasks.rados.rados.0.smithi071.stdout:720: snap_remove snap 67 2024-09-11T01:33:32.887 INFO:tasks.rados.rados.0.smithi071.stdout:720: done (1 left) 2024-09-11T01:33:32.887 INFO:tasks.rados.rados.0.smithi071.stdout:721: read oid 41 snap -1 2024-09-11T01:33:32.887 INFO:tasks.rados.rados.0.smithi071.stdout:721: expect deleted 2024-09-11T01:33:32.887 INFO:tasks.rados.rados.0.smithi071.stdout:722: snap_remove snap 68 2024-09-11T01:33:33.326 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:33:33.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.437+0000 7f91ac233640 10 monclient: get_monmap_and_config 2024-09-11T01:33:33.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.437+0000 7f91ac233640 10 monclient: build_initial_monmap 2024-09-11T01:33:33.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.437+0000 7f91ac233640 10 monclient: monmap: 2024-09-11T01:33:33.440 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:33.440 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:33.440 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:33:33.439231+0000 2024-09-11T01:33:33.440 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:33:33.439231+0000 2024-09-11T01:33:33.440 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:33.440 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:33.440 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:33:33.441 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:33:33.441 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:33:33.441 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:33:33.441 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:33:33.441 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:33.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.438+0000 7f91ac233640 1 Processor -- start 2024-09-11T01:33:33.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.438+0000 7f91ac233640 1 -- start start 2024-09-11T01:33:33.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.438+0000 7f91ac233640 10 monclient: init 2024-09-11T01:33:33.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.438+0000 7f91ac233640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:33.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.438+0000 7f91ac233640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-11T01:33:33.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.438+0000 7f91ac233640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f91a40a2e90 0x7f91a4073d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:33.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.438+0000 7f91ac233640 10 monclient(hunting): picked mon.noname-a con 0x7f91a40a2e90 addr v2:172.21.15.12:3300/0 2024-09-11T01:33:33.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.438+0000 7f91ac233640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f91a40a3270 addr v1:172.21.15.12:6789/0 2024-09-11T01:33:33.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.438+0000 7f91ac233640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f91a413b150 0x7f91a413d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:33.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.438+0000 7f91ac233640 10 monclient(hunting): picked mon.noname-b con 0x7f91a413b150 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:33.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91ac233640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91a40a46c0 con 0x7f91a40a3270 2024-09-11T01:33:33.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91ac233640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:33.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91ac233640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f91a4072730 con 0x7f91a40a2e90 2024-09-11T01:33:33.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91ac233640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:33.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91ac233640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f91a4072fa0 con 0x7f91a413b150 2024-09-11T01:33:33.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91ac233640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:33.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91ac233640 10 monclient(hunting): authenticate will time out at 1019.882042s 2024-09-11T01:33:33.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91a9fcf640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f91a40a2e90 0x7f91a4073d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:33.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91aa7d0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f91a413b150 0x7f91a413d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:33.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91a9fcf640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f91a40a2e90 0x7f91a4073d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58762/0 (socket says 172.21.15.12:58762) 2024-09-11T01:33:33.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91a9fcf640 1 -- 172.21.15.12:0/3260351180 learned_addr learned my addr 172.21.15.12:0/3260351180 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:33.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91aa7d0640 10 monclient(hunting): get_auth_request con 0x7f91a413b150 auth_method 0 2024-09-11T01:33:33.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91aa7d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:33.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91aa7d0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:33.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91aa7d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:33.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91a9fcf640 10 monclient(hunting): get_auth_request con 0x7f91a40a2e90 auth_method 0 2024-09-11T01:33:33.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91a9fcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:33.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91a9fcf640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:33.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91a9fcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:33.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91aa7d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:33.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91aa7d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:33.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91a9fcf640 1 -- 172.21.15.12:0/3260351180 >> v2:172.21.15.12:3300/0 conn(0x7f91a40a2e90 msgr2=0x7f91a4073d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:33:33.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91a9fcf640 1 -- 172.21.15.12:0/3260351180 >> v2:172.21.15.12:3300/0 conn(0x7f91a40a2e90 msgr2=0x7f91a4073d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:33.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91aa7d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:33.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91a9fcf640 1 --2- 172.21.15.12:0/3260351180 >> v2:172.21.15.12:3300/0 conn(0x7f91a40a2e90 0x7f91a4073d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:33.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91a9fcf640 1 --2- 172.21.15.12:0/3260351180 >> v2:172.21.15.12:3300/0 conn(0x7f91a40a2e90 0x7f91a4073d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:33:33.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91aa7d0640 10 monclient(hunting): handle_auth_done global_id 44189 payload 274 2024-09-11T01:33:33.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91aa7d0640 1 -- 172.21.15.12:0/3260351180 >> v2:172.21.15.12:3300/0 conn(0x7f91a40a2e90 msgr2=0x7f91a4073d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:33:33.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91aa7d0640 1 --2- 172.21.15.12:0/3260351180 >> v2:172.21.15.12:3300/0 conn(0x7f91a40a2e90 0x7f91a4073d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:33.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91aa7d0640 1 -- 172.21.15.12:0/3260351180 >> v1:172.21.15.12:6789/0 conn(0x7f91a40a3270 legacy=0x7f91a40a36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:33.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91aa7d0640 10 monclient: _finish_hunting 0 2024-09-11T01:33:33.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91aa7d0640 1 monclient: found mon.noname-b 2024-09-11T01:33:33.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91aa7d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:33.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91aa7d0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:33:33.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.439+0000 7f91aa7d0640 1 -- 172.21.15.12:0/3260351180 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a4073280 con 0x7f91a413b150 2024-09-11T01:33:33.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.442+0000 7f91aa7d0640 1 -- 172.21.15.12:0/3260351180 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f91a413b150 msgr2=0x7f91a413d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:33.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.442+0000 7f91aa7d0640 1 --2- 172.21.15.12:0/3260351180 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f91a413b150 0x7f91a413d540 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f919c009d90 tx=0x7f919c02a490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31db15e5058939a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:33.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.442+0000 7f91a8fcd640 1 -- 172.21.15.12:0/3260351180 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919c007950 con 0x7f91a413b150 2024-09-11T01:33:33.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.442+0000 7f91a8fcd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:33.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.442+0000 7f91a8fcd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:33:33.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.442+0000 7f91a8fcd640 10 monclient: dump: 2024-09-11T01:33:33.452 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:33.452 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:33.452 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:33.452 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:33.452 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:33.452 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:33.453 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:33.453 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:33.453 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:33.453 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:33.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.442+0000 7f91a8fcd640 10 monclient: _finish_auth 0 2024-09-11T01:33:33.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.442+0000 7f91a8fcd640 10 monclient: _check_auth_tickets 2024-09-11T01:33:33.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.442+0000 7f91a8fcd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:33.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.442+0000 7f91ac233640 5 monclient: authenticate success, global_id 44189 2024-09-11T01:33:33.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.442+0000 7f91a8fcd640 1 -- 172.21.15.12:0/3260351180 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f919c007af0 con 0x7f91a413b150 2024-09-11T01:33:33.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.442+0000 7f91ac233640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:33:33.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.442+0000 7f91a8fcd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:33.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.442+0000 7f91ac233640 10 monclient: get_monmap_and_config success 2024-09-11T01:33:33.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.442+0000 7f91ac233640 4 set_mon_vals no callback set 2024-09-11T01:33:33.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.442+0000 7f91a8fcd640 1 -- 172.21.15.12:0/3260351180 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919c031400 con 0x7f91a413b150 2024-09-11T01:33:33.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.442+0000 7f91a8fcd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:33.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.442+0000 7f91a8fcd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:33.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.442+0000 7f91a8fcd640 10 monclient: dump: 2024-09-11T01:33:33.454 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:33.454 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:33.454 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:33.454 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:33.454 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:33.454 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:33.454 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:33.454 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:33.454 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:33.454 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:33.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.443+0000 7f91ac233640 10 monclient: shutdown 2024-09-11T01:33:33.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.443+0000 7f91ac233640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:33.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.443+0000 7f91ac233640 1 -- 172.21.15.12:0/3260351180 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f91a413b150 msgr2=0x7f91a413d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:33.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.443+0000 7f91ac233640 1 --2- 172.21.15.12:0/3260351180 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f91a413b150 0x7f91a413d540 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f919c009d90 tx=0x7f919c02a490 comp rx=0 tx=0).stop 2024-09-11T01:33:33.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.443+0000 7f91ac233640 1 -- 172.21.15.12:0/3260351180 shutdown_connections 2024-09-11T01:33:33.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.443+0000 7f91ac233640 1 --2- 172.21.15.12:0/3260351180 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f91a413b150 0x7f91a413d540 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:33.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.443+0000 7f91ac233640 1 --2- 172.21.15.12:0/3260351180 >> v2:172.21.15.12:3300/0 conn(0x7f91a40a2e90 0x7f91a4073d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:33.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.443+0000 7f91ac233640 1 -- 172.21.15.12:0/3260351180 >> 172.21.15.12:0/3260351180 conn(0x7f91a40a8490 msgr2=0x7f91a40a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:33.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.443+0000 7f91ac233640 1 -- 172.21.15.12:0/3260351180 shutdown_connections 2024-09-11T01:33:33.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.443+0000 7f91ac233640 1 -- 172.21.15.12:0/3260351180 wait complete. 2024-09-11T01:33:33.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.443+0000 7f91ac233640 10 monclient: build_initial_monmap 2024-09-11T01:33:33.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.443+0000 7f91ac233640 10 monclient: monmap: 2024-09-11T01:33:33.455 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:33.456 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:33:33.456 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:33:33.456 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:33:33.456 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:33.456 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:33.456 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:33:33.456 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:33:33.456 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:33:33.456 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:33.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.444+0000 7f91ac233640 1 Processor -- start 2024-09-11T01:33:33.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.444+0000 7f91ac233640 1 -- start start 2024-09-11T01:33:33.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.444+0000 7f91ac233640 10 monclient: init 2024-09-11T01:33:33.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.444+0000 7f91ac233640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:33.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.444+0000 7f91ac233640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:33:33.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.444+0000 7f91ac233640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91a40a2e90 0x7f91a41cce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:33.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.444+0000 7f91ac233640 10 monclient(hunting): picked mon.noname-b con 0x7f91a40a2e90 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:33:33.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.444+0000 7f91ac233640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f91a40a3270 0x7f91a4136020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:33.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.444+0000 7f91ac233640 10 monclient(hunting): picked mon.noname-c con 0x7f91a40a3270 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:33.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.444+0000 7f91ac233640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91a413b150 0x7f91a4137110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:33.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.444+0000 7f91ac233640 10 monclient(hunting): picked mon.noname-a con 0x7f91a413b150 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:33:33.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.444+0000 7f91ac233640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:33.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.444+0000 7f91ac233640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f91a413da80 con 0x7f91a413b150 2024-09-11T01:33:33.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.444+0000 7f91ac233640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:33.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.444+0000 7f91ac233640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f91a413dc00 con 0x7f91a40a2e90 2024-09-11T01:33:33.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.444+0000 7f91ac233640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:33.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.444+0000 7f91ac233640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f91a413dd80 con 0x7f91a40a3270 2024-09-11T01:33:33.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.444+0000 7f91ac233640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:33.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.444+0000 7f91ac233640 10 monclient(hunting): authenticate will time out at 1019.887881s 2024-09-11T01:33:33.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.444+0000 7f91a97ce640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f91a40a3270 0x7f91a4136020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:33.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.444+0000 7f91a97ce640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f91a40a3270 0x7f91a4136020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:34916/0 (socket says 172.21.15.12:34916) 2024-09-11T01:33:33.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91a97ce640 1 -- 172.21.15.12:0/647823788 learned_addr learned my addr 172.21.15.12:0/647823788 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:33.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91a97ce640 10 monclient(hunting): get_auth_request con 0x7f91a40a3270 auth_method 0 2024-09-11T01:33:33.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91a97ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:33.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91a97ce640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:33.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91a97ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:33.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91aa7d0640 1 --2- 172.21.15.12:0/647823788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91a413b150 0x7f91a4137110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:33.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91aa7d0640 10 monclient(hunting): get_auth_request con 0x7f91a413b150 auth_method 0 2024-09-11T01:33:33.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91aa7d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:33.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91aa7d0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:33.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91aa7d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:33.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91a97ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91a97ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91a97ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91a9fcf640 1 --2- 172.21.15.12:0/647823788 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91a40a2e90 0x7f91a41cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91a9fcf640 10 monclient(hunting): get_auth_request con 0x7f91a40a2e90 auth_method 0 2024-09-11T01:33:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91a9fcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91a9fcf640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91a9fcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91aa7d0640 1 -- 172.21.15.12:0/647823788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91a413b150 msgr2=0x7f91a4137110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:33:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91aa7d0640 1 -- 172.21.15.12:0/647823788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91a413b150 msgr2=0x7f91a4137110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:33:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91aa7d0640 1 --2- 172.21.15.12:0/647823788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91a413b150 0x7f91a4137110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91aa7d0640 1 --2- 172.21.15.12:0/647823788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91a413b150 0x7f91a4137110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:33:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91a9fcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:33.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91a9fcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:33.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91a9fcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:33.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91a97ce640 10 monclient(hunting): handle_auth_done global_id 44192 payload 824 2024-09-11T01:33:33.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91a97ce640 1 -- 172.21.15.12:0/647823788 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91a40a2e90 msgr2=0x7f91a41cce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:33.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91a97ce640 1 --2- 172.21.15.12:0/647823788 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91a40a2e90 0x7f91a41cce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:33.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91a97ce640 1 -- 172.21.15.12:0/647823788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91a413b150 msgr2=0x7f91a4137110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:33:33.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91a97ce640 1 --2- 172.21.15.12:0/647823788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91a413b150 0x7f91a4137110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:33.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91a97ce640 10 monclient: _finish_hunting 0 2024-09-11T01:33:33.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91a97ce640 1 monclient: found mon.noname-c 2024-09-11T01:33:33.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91a97ce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:33.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91a97ce640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:33:33.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.445+0000 7f91a97ce640 1 -- 172.21.15.12:0/647823788 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a406dc20 con 0x7f91a40a3270 2024-09-11T01:33:33.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.447+0000 7f91a97ce640 1 -- 172.21.15.12:0/647823788 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f91a40a3270 msgr2=0x7f91a4136020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:33.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.447+0000 7f91a97ce640 1 --2- 172.21.15.12:0/647823788 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f91a40a3270 0x7f91a4136020 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f919400c430 tx=0x7f919400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:33.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.447+0000 7f919a7fc640 1 -- 172.21.15.12:0/647823788 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91940119a0 con 0x7f91a40a3270 2024-09-11T01:33:33.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.447+0000 7f919a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:33.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.447+0000 7f919a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:33:33.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.447+0000 7f919a7fc640 10 monclient: dump: 2024-09-11T01:33:33.461 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:33.461 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:33.461 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:33.461 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:33.461 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:33.461 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:33.461 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:33.461 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:33.462 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:33.462 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:33.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.447+0000 7f919a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:33:33.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.447+0000 7f919a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:33:33.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.447+0000 7f919a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:33.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.447+0000 7f919a7fc640 1 -- 172.21.15.12:0/647823788 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9194017070 con 0x7f91a40a3270 2024-09-11T01:33:33.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.447+0000 7f919a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:33.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.447+0000 7f919a7fc640 1 -- 172.21.15.12:0/647823788 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9194011d80 con 0x7f91a40a3270 2024-09-11T01:33:33.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.447+0000 7f91ac233640 5 monclient: authenticate success, global_id 44192 2024-09-11T01:33:33.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.447+0000 7f91ac233640 10 monclient: _renew_subs 2024-09-11T01:33:33.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.447+0000 7f91ac233640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:33.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.447+0000 7f91ac233640 1 -- 172.21.15.12:0/647823788 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91a413aaf0 con 0x7f91a40a3270 2024-09-11T01:33:33.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.447+0000 7f919a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:33.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.447+0000 7f919a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:33.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.447+0000 7f919bfff640 4 set_mon_vals no callback set 2024-09-11T01:33:33.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.447+0000 7f919a7fc640 10 monclient: dump: 2024-09-11T01:33:33.463 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:33.463 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:33.463 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:33.463 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:33.463 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:33.463 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:33.463 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:33.463 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:33.463 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:33.463 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:33.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.447+0000 7f91ac233640 10 monclient: _renew_subs 2024-09-11T01:33:33.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.447+0000 7f91ac233640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:33.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.447+0000 7f91ac233640 1 -- 172.21.15.12:0/647823788 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f91a4138790 con 0x7f91a40a3270 2024-09-11T01:33:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.448+0000 7f919a7fc640 1 -- 172.21.15.12:0/647823788 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f919401c460 con 0x7f91a40a3270 2024-09-11T01:33:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.448+0000 7f91ac233640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.448+0000 7f91ac233640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.448+0000 7f91ac233640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.448+0000 7f91ac233640 1 -- 172.21.15.12:0/647823788 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91a4073d40 con 0x7f91a40a3270 2024-09-11T01:33:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.448+0000 7f919a7fc640 1 --2- 172.21.15.12:0/647823788 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f918803e290 0x7f9188040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.449+0000 7f919a7fc640 1 -- 172.21.15.12:0/647823788 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(164..164 src has 1..164) ==== 4799+0+0 (secure 0 0 0) 0x7f9194052be0 con 0x7f91a40a3270 2024-09-11T01:33:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.449+0000 7f91a9fcf640 1 --2- 172.21.15.12:0/647823788 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f918803e290 0x7f9188040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.449+0000 7f91a9fcf640 10 monclient: get_auth_request con 0x7f918803e290 auth_method 0 2024-09-11T01:33:33.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.451+0000 7f91a9fcf640 1 --2- 172.21.15.12:0/647823788 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f918803e290 0x7f9188040750 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f91a0009100 tx=0x7f91a0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:33.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.453+0000 7f919a7fc640 1 -- 172.21.15.12:0/647823788 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f91a406dc20 con 0x7f91a40a3270 2024-09-11T01:33:33.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.453+0000 7f919a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:33.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.453+0000 7f919a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:33:33.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.711+0000 7f91ac233640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:33:33.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.711+0000 7f91ac233640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:33:33.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.711+0000 7f91ac233640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:33.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.711+0000 7f91ac233640 1 -- 172.21.15.12:0/647823788 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f91a40698e0 con 0x7f91a40a3270 2024-09-11T01:33:33.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.712+0000 7f919a7fc640 1 -- 172.21.15.12:0/647823788 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f919401ac60 con 0x7f91a40a3270 2024-09-11T01:33:33.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.712+0000 7f919a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:33:33.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.712+0000 7f919a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:33:33.715 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":40,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:33:33.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.714+0000 7f91ac233640 1 -- 172.21.15.12:0/647823788 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f918803e290 msgr2=0x7f9188040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:33.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.714+0000 7f91ac233640 1 --2- 172.21.15.12:0/647823788 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f918803e290 0x7f9188040750 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f91a0009100 tx=0x7f91a0008040 comp rx=0 tx=0).stop 2024-09-11T01:33:33.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.714+0000 7f91ac233640 10 monclient: shutdown 2024-09-11T01:33:33.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.714+0000 7f91ac233640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:33.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.714+0000 7f91ac233640 1 -- 172.21.15.12:0/647823788 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f91a40a3270 msgr2=0x7f91a4136020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:33.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.714+0000 7f91ac233640 1 --2- 172.21.15.12:0/647823788 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f91a40a3270 0x7f91a4136020 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f919400c430 tx=0x7f919400c8f0 comp rx=0 tx=0).stop 2024-09-11T01:33:33.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.714+0000 7f91ac233640 1 -- 172.21.15.12:0/647823788 shutdown_connections 2024-09-11T01:33:33.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.714+0000 7f91ac233640 1 --2- 172.21.15.12:0/647823788 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f918803e290 0x7f9188040750 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:33.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.714+0000 7f91ac233640 1 --2- 172.21.15.12:0/647823788 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91a413b150 0x7f91a4137110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:33.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.714+0000 7f91ac233640 1 --2- 172.21.15.12:0/647823788 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f91a40a3270 0x7f91a4136020 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:33.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.714+0000 7f91ac233640 1 --2- 172.21.15.12:0/647823788 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91a40a2e90 0x7f91a41cce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:33.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.714+0000 7f91ac233640 1 -- 172.21.15.12:0/647823788 >> 172.21.15.12:0/647823788 conn(0x7f91a40a8490 msgr2=0x7f91a40a6290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:33.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.714+0000 7f91ac233640 1 -- 172.21.15.12:0/647823788 shutdown_connections 2024-09-11T01:33:33.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:33.714+0000 7f91ac233640 1 -- 172.21.15.12:0/647823788 wait complete. 2024-09-11T01:33:33.896 INFO:tasks.rados.rados.0.smithi071.stdout:722: done (2 left) 2024-09-11T01:33:33.897 INFO:tasks.rados.rados.0.smithi071.stdout:723: snap_remove snap 72 2024-09-11T01:33:34.896 INFO:tasks.rados.rados.0.smithi071.stdout:723: done (2 left) 2024-09-11T01:33:34.896 INFO:tasks.rados.rados.0.smithi071.stdout:724: write oid 24 current snap is 73 2024-09-11T01:33:34.896 INFO:tasks.rados.rados.0.smithi071.stdout:724: seq_num 276 ranges {638732=557432,1617736=710086} 2024-09-11T01:33:34.902 INFO:tasks.rados.rados.0.smithi071.stdout:724: writing smithi07134777-24 from 638732 to 1196164 tid 1 2024-09-11T01:33:34.909 INFO:tasks.rados.rados.0.smithi071.stdout:724: writing smithi07134777-24 from 1617736 to 2327822 tid 2 2024-09-11T01:33:34.909 INFO:tasks.rados.rados.0.smithi071.stdout:725: write oid 23 current snap is 73 2024-09-11T01:33:34.909 INFO:tasks.rados.rados.0.smithi071.stdout:725: seq_num 277 ranges {655539=632681,2076218=570097} 2024-09-11T01:33:34.916 INFO:tasks.rados.rados.0.smithi071.stdout:725: writing smithi07134777-23 from 655539 to 1288220 tid 1 2024-09-11T01:33:34.924 INFO:tasks.rados.rados.0.smithi071.stdout:725: writing smithi07134777-23 from 2076218 to 2646315 tid 2 2024-09-11T01:33:34.924 INFO:tasks.rados.rados.0.smithi071.stdout:726: snap_remove snap 63 2024-09-11T01:33:35.906 INFO:tasks.rados.rados.0.smithi071.stdout:726: done (4 left) 2024-09-11T01:33:35.907 INFO:tasks.rados.rados.0.smithi071.stdout:727: read oid 20 snap -1 2024-09-11T01:33:35.907 INFO:tasks.rados.rados.0.smithi071.stdout:727: expect (ObjNum 114 snap 27 seq_num 114) 2024-09-11T01:33:35.907 INFO:tasks.rados.rados.0.smithi071.stdout:728: delete oid 44 current snap is 73 2024-09-11T01:33:35.907 INFO:tasks.rados.rados.0.smithi071.stdout:719: expect (ObjNum 267 snap 71 seq_num 267) 2024-09-11T01:33:36.051 INFO:tasks.rados.rados.0.smithi071.stdout:724: finishing write tid 1 to smithi07134777-24 2024-09-11T01:33:36.051 INFO:tasks.rados.rados.0.smithi071.stdout:724: finishing write tid 2 to smithi07134777-24 2024-09-11T01:33:36.051 INFO:tasks.rados.rados.0.smithi071.stdout:724: finishing write tid 3 to smithi07134777-24 2024-09-11T01:33:36.051 INFO:tasks.rados.rados.0.smithi071.stdout:724: finishing write tid 4 to smithi07134777-24 2024-09-11T01:33:36.051 INFO:tasks.rados.rados.0.smithi071.stdout:724: oid 24 updating version 0 to 230 2024-09-11T01:33:36.051 INFO:tasks.rados.rados.0.smithi071.stdout:724: oid 24 version 230 is already newer than 229 2024-09-11T01:33:36.051 INFO:tasks.rados.rados.0.smithi071.stdout:724: oid 24 version 230 is already newer than 228 2024-09-11T01:33:36.051 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 230 (ObjNum 276 snap 73 seq_num 276) dirty exists 2024-09-11T01:33:36.051 INFO:tasks.rados.rados.0.smithi071.stdout:724: left oid 24 (ObjNum 276 snap 73 seq_num 276) 2024-09-11T01:33:36.051 INFO:tasks.rados.rados.0.smithi071.stdout:725: finishing write tid 1 to smithi07134777-23 2024-09-11T01:33:36.052 INFO:tasks.rados.rados.0.smithi071.stdout:725: finishing write tid 2 to smithi07134777-23 2024-09-11T01:33:36.052 INFO:tasks.rados.rados.0.smithi071.stdout:725: finishing write tid 3 to smithi07134777-23 2024-09-11T01:33:36.052 INFO:tasks.rados.rados.0.smithi071.stdout:719: done (5 left) 2024-09-11T01:33:36.052 INFO:tasks.rados.rados.0.smithi071.stdout:721: done (4 left) 2024-09-11T01:33:36.052 INFO:tasks.rados.rados.0.smithi071.stdout:724: done (3 left) 2024-09-11T01:33:36.052 INFO:tasks.rados.rados.0.smithi071.stdout:728: done (2 left) 2024-09-11T01:33:36.052 INFO:tasks.rados.rados.0.smithi071.stdout:729: delete oid 43 current snap is 73 2024-09-11T01:33:36.052 INFO:tasks.rados.rados.0.smithi071.stdout:725: finishing write tid 4 to smithi07134777-23 2024-09-11T01:33:36.052 INFO:tasks.rados.rados.0.smithi071.stdout:725: oid 23 updating version 0 to 231 2024-09-11T01:33:36.052 INFO:tasks.rados.rados.0.smithi071.stdout:725: oid 23 updating version 231 to 232 2024-09-11T01:33:36.052 INFO:tasks.rados.rados.0.smithi071.stdout:725: oid 23 updating version 232 to 233 2024-09-11T01:33:36.052 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 233 (ObjNum 277 snap 73 seq_num 277) dirty exists 2024-09-11T01:33:36.052 INFO:tasks.rados.rados.0.smithi071.stdout:725: left oid 23 (ObjNum 277 snap 73 seq_num 277) 2024-09-11T01:33:36.053 INFO:tasks.rados.rados.0.smithi071.stdout:727: expect (ObjNum 114 snap 27 seq_num 114) 2024-09-11T01:33:36.053 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:33:36.116 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 816990 2024-09-11T01:33:36.163 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:33:36.221 INFO:tasks.rados.rados.0.smithi071.stdout:725: done (2 left) 2024-09-11T01:33:36.221 INFO:tasks.rados.rados.0.smithi071.stdout:727: done (1 left) 2024-09-11T01:33:36.221 INFO:tasks.rados.rados.0.smithi071.stdout:729: done (0 left) 2024-09-11T01:33:36.221 INFO:tasks.rados.rados.0.smithi071.stdout:730: write (excl) oid 4 current snap is 73 2024-09-11T01:33:36.222 INFO:tasks.rados.rados.0.smithi071.stdout:730: seq_num 280 ranges {705960=458428,1735311=717424} 2024-09-11T01:33:36.227 INFO:tasks.rados.rados.0.smithi071.stdout:730: writing smithi07134777-4 from 705960 to 1164388 tid 1 2024-09-11T01:33:36.234 INFO:tasks.rados.rados.0.smithi071.stdout:730: writing smithi07134777-4 from 1735311 to 2452735 tid 2 2024-09-11T01:33:36.234 INFO:tasks.rados.rados.0.smithi071.stdout:731: snap_remove snap 58 2024-09-11T01:33:36.728 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:33:36.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.838+0000 7f9718272640 10 monclient: get_monmap_and_config 2024-09-11T01:33:36.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.838+0000 7f9718272640 10 monclient: build_initial_monmap 2024-09-11T01:33:36.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.838+0000 7f9718272640 10 monclient: monmap: 2024-09-11T01:33:36.841 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:36.842 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:36.842 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:33:36.840693+0000 2024-09-11T01:33:36.842 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:33:36.840693+0000 2024-09-11T01:33:36.842 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:36.842 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:36.842 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:33:36.842 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:33:36.842 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:33:36.842 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:33:36.842 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:33:36.842 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:36.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.839+0000 7f9718272640 1 Processor -- start 2024-09-11T01:33:36.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.839+0000 7f9718272640 1 -- start start 2024-09-11T01:33:36.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.839+0000 7f9718272640 10 monclient: init 2024-09-11T01:33:36.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f9718272640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:36.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f9718272640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-11T01:33:36.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f9718272640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f97100a0440 0x7f97100a2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:36.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f9718272640 10 monclient(hunting): picked mon.noname-a con 0x7f97100a0440 addr v2:172.21.15.12:3300/0 2024-09-11T01:33:36.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f9718272640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f97100a2740 0x7f97100a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:36.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f9718272640 10 monclient(hunting): picked mon.noname-c con 0x7f97100a2740 addr v2:172.21.15.71:3300/0 2024-09-11T01:33:36.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f9718272640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9710137ca0 addr v1:172.21.15.12:6789/0 2024-09-11T01:33:36.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f9718272640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9710069960 con 0x7f9710137ca0 2024-09-11T01:33:36.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f9718272640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:36.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f9718272640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f97100a0820 con 0x7f97100a0440 2024-09-11T01:33:36.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f9718272640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:36.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f9718272640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f971006ff60 con 0x7f97100a2740 2024-09-11T01:33:36.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f9718272640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:36.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f9718272640 10 monclient(hunting): authenticate will time out at 1023.283379s 2024-09-11T01:33:36.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f971600e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f97100a0440 0x7f97100a2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:36.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f971680f640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9710137ca0 0x7f9710138110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53870/0 (socket says 172.21.15.12:53870) 2024-09-11T01:33:36.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f971680f640 1 -- 172.21.15.12:0/378828871 learned_addr learned my addr 172.21.15.12:0/378828871 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:33:36.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f971600e640 10 monclient(hunting): get_auth_request con 0x7f97100a0440 auth_method 0 2024-09-11T01:33:36.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f971600e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:36.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f971600e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:36.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f971600e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:36.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f971580d640 1 --2- 172.21.15.12:0/378828871 >> v2:172.21.15.71:3300/0 conn(0x7f97100a2740 0x7f97100a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:36.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f971600e640 1 -- 172.21.15.12:0/378828871 >> v2:172.21.15.12:3300/0 conn(0x7f97100a0440 msgr2=0x7f97100a2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:33:36.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f971600e640 1 -- 172.21.15.12:0/378828871 >> v2:172.21.15.12:3300/0 conn(0x7f97100a0440 msgr2=0x7f97100a2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:36.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f971600e640 1 --2- 172.21.15.12:0/378828871 >> v2:172.21.15.12:3300/0 conn(0x7f97100a0440 0x7f97100a2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:36.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f971580d640 10 monclient(hunting): get_auth_request con 0x7f97100a2740 auth_method 0 2024-09-11T01:33:36.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f971580d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:36.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f971580d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:36.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f971580d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:36.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f971600e640 1 --2- 172.21.15.12:0/378828871 >> v2:172.21.15.12:3300/0 conn(0x7f97100a0440 0x7f97100a2200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:33:36.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f971580d640 1 -- 172.21.15.12:0/378828871 >> v2:172.21.15.71:3300/0 conn(0x7f97100a2740 msgr2=0x7f97100a2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:33:36.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f971580d640 1 -- 172.21.15.12:0/378828871 >> v2:172.21.15.71:3300/0 conn(0x7f97100a2740 msgr2=0x7f97100a2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:36.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f971580d640 1 --2- 172.21.15.12:0/378828871 >> v2:172.21.15.71:3300/0 conn(0x7f97100a2740 0x7f97100a2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:36.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.840+0000 7f971580d640 1 --2- 172.21.15.12:0/378828871 >> v2:172.21.15.71:3300/0 conn(0x7f97100a2740 0x7f97100a2b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:33:36.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:36.842+0000 7f971680f640 1 -- 172.21.15.12:0/378828871 >> v1:172.21.15.12:6789/0 conn(0x7f9710137ca0 legacy=0x7f9710138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:37.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.041+0000 7f971600e640 1 --2- 172.21.15.12:0/378828871 >> v2:172.21.15.12:3300/0 conn(0x7f97100a0440 0x7f97100a2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:37.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.041+0000 7f971600e640 10 monclient(hunting): get_auth_request con 0x7f97100a0440 auth_method 0 2024-09-11T01:33:37.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.041+0000 7f971600e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:37.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.041+0000 7f971600e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:37.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.041+0000 7f971600e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:37.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.041+0000 7f971600e640 1 -- 172.21.15.12:0/378828871 >> v2:172.21.15.12:3300/0 conn(0x7f97100a0440 msgr2=0x7f97100a2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:33:37.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.041+0000 7f971600e640 1 -- 172.21.15.12:0/378828871 >> v2:172.21.15.12:3300/0 conn(0x7f97100a0440 msgr2=0x7f97100a2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:37.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.041+0000 7f971600e640 1 --2- 172.21.15.12:0/378828871 >> v2:172.21.15.12:3300/0 conn(0x7f97100a0440 0x7f97100a2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:37.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.041+0000 7f971600e640 1 --2- 172.21.15.12:0/378828871 >> v2:172.21.15.12:3300/0 conn(0x7f97100a0440 0x7f97100a2200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:33:37.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.041+0000 7f971580d640 1 --2- 172.21.15.12:0/378828871 >> v2:172.21.15.71:3300/0 conn(0x7f97100a2740 0x7f97100a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:37.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.041+0000 7f971580d640 10 monclient(hunting): get_auth_request con 0x7f97100a2740 auth_method 0 2024-09-11T01:33:37.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.041+0000 7f971580d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:37.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.041+0000 7f971580d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:37.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.041+0000 7f971580d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:37.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.041+0000 7f971580d640 1 -- 172.21.15.12:0/378828871 >> v2:172.21.15.71:3300/0 conn(0x7f97100a2740 msgr2=0x7f97100a2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:33:37.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.041+0000 7f971580d640 1 -- 172.21.15.12:0/378828871 >> v2:172.21.15.71:3300/0 conn(0x7f97100a2740 msgr2=0x7f97100a2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:37.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.041+0000 7f971580d640 1 --2- 172.21.15.12:0/378828871 >> v2:172.21.15.71:3300/0 conn(0x7f97100a2740 0x7f97100a2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:37.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.041+0000 7f971580d640 1 --2- 172.21.15.12:0/378828871 >> v2:172.21.15.71:3300/0 conn(0x7f97100a2740 0x7f97100a2b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:33:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.442+0000 7f971600e640 1 --2- 172.21.15.12:0/378828871 >> v2:172.21.15.12:3300/0 conn(0x7f97100a0440 0x7f97100a2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.442+0000 7f971600e640 10 monclient(hunting): get_auth_request con 0x7f97100a0440 auth_method 0 2024-09-11T01:33:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.442+0000 7f971600e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.442+0000 7f971600e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.442+0000 7f971600e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.442+0000 7f971600e640 1 -- 172.21.15.12:0/378828871 >> v2:172.21.15.12:3300/0 conn(0x7f97100a0440 msgr2=0x7f97100a2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:33:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.442+0000 7f971600e640 1 -- 172.21.15.12:0/378828871 >> v2:172.21.15.12:3300/0 conn(0x7f97100a0440 msgr2=0x7f97100a2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.442+0000 7f971600e640 1 --2- 172.21.15.12:0/378828871 >> v2:172.21.15.12:3300/0 conn(0x7f97100a0440 0x7f97100a2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.442+0000 7f971600e640 1 --2- 172.21.15.12:0/378828871 >> v2:172.21.15.12:3300/0 conn(0x7f97100a0440 0x7f97100a2200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:33:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.442+0000 7f971580d640 1 --2- 172.21.15.12:0/378828871 >> v2:172.21.15.71:3300/0 conn(0x7f97100a2740 0x7f97100a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.442+0000 7f971580d640 10 monclient(hunting): get_auth_request con 0x7f97100a2740 auth_method 0 2024-09-11T01:33:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.442+0000 7f971580d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.442+0000 7f971580d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.442+0000 7f971580d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.442+0000 7f971580d640 1 -- 172.21.15.12:0/378828871 >> v2:172.21.15.71:3300/0 conn(0x7f97100a2740 msgr2=0x7f97100a2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:33:37.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.442+0000 7f971580d640 1 -- 172.21.15.12:0/378828871 >> v2:172.21.15.71:3300/0 conn(0x7f97100a2740 msgr2=0x7f97100a2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.442+0000 7f971580d640 1 --2- 172.21.15.12:0/378828871 >> v2:172.21.15.71:3300/0 conn(0x7f97100a2740 0x7f97100a2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:37.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:37.442+0000 7f971580d640 1 --2- 172.21.15.12:0/378828871 >> v2:172.21.15.71:3300/0 conn(0x7f97100a2740 0x7f97100a2b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:33:38.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:38.244+0000 7f971600e640 1 --2- 172.21.15.12:0/378828871 >> v2:172.21.15.12:3300/0 conn(0x7f97100a0440 0x7f97100a2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:38.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:38.244+0000 7f971600e640 10 monclient(hunting): get_auth_request con 0x7f97100a0440 auth_method 0 2024-09-11T01:33:38.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:38.244+0000 7f971600e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:38.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:38.244+0000 7f971600e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:38.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:38.244+0000 7f971600e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:38.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:38.244+0000 7f971600e640 1 -- 172.21.15.12:0/378828871 >> v2:172.21.15.12:3300/0 conn(0x7f97100a0440 msgr2=0x7f97100a2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:33:38.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:38.244+0000 7f971600e640 1 -- 172.21.15.12:0/378828871 >> v2:172.21.15.12:3300/0 conn(0x7f97100a0440 msgr2=0x7f97100a2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:38.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:38.244+0000 7f971600e640 1 --2- 172.21.15.12:0/378828871 >> v2:172.21.15.12:3300/0 conn(0x7f97100a0440 0x7f97100a2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:38.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:38.244+0000 7f971600e640 1 --2- 172.21.15.12:0/378828871 >> v2:172.21.15.12:3300/0 conn(0x7f97100a0440 0x7f97100a2200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:33:38.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:38.244+0000 7f971580d640 1 --2- 172.21.15.12:0/378828871 >> v2:172.21.15.71:3300/0 conn(0x7f97100a2740 0x7f97100a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:38.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:38.244+0000 7f971580d640 10 monclient(hunting): get_auth_request con 0x7f97100a2740 auth_method 0 2024-09-11T01:33:38.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:38.244+0000 7f971580d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:38.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:38.244+0000 7f971580d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:38.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:38.244+0000 7f971580d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:38.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:38.244+0000 7f971580d640 1 -- 172.21.15.12:0/378828871 >> v2:172.21.15.71:3300/0 conn(0x7f97100a2740 msgr2=0x7f97100a2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:33:38.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:38.244+0000 7f971580d640 1 -- 172.21.15.12:0/378828871 >> v2:172.21.15.71:3300/0 conn(0x7f97100a2740 msgr2=0x7f97100a2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:38.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:38.244+0000 7f971580d640 1 --2- 172.21.15.12:0/378828871 >> v2:172.21.15.71:3300/0 conn(0x7f97100a2740 0x7f97100a2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:38.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:38.244+0000 7f971580d640 1 --2- 172.21.15.12:0/378828871 >> v2:172.21.15.71:3300/0 conn(0x7f97100a2740 0x7f97100a2b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:33:39.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.841+0000 7f970ffff640 10 monclient(hunting): tick 2024-09-11T01:33:39.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.841+0000 7f970ffff640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:33:39.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.841+0000 7f970ffff640 1 monclient(hunting): continuing hunt 2024-09-11T01:33:39.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.841+0000 7f970ffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:33:39.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.841+0000 7f970ffff640 1 -- 172.21.15.12:0/378828871 >> v2:172.21.15.71:3300/0 conn(0x7f97100a2740 msgr2=0x7f97100a2b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:33:39.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.841+0000 7f970ffff640 1 --2- 172.21.15.12:0/378828871 >> v2:172.21.15.71:3300/0 conn(0x7f97100a2740 0x7f97100a2b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:39.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.841+0000 7f970ffff640 1 -- 172.21.15.12:0/378828871 >> v2:172.21.15.12:3300/0 conn(0x7f97100a0440 msgr2=0x7f97100a2200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:33:39.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.841+0000 7f970ffff640 1 --2- 172.21.15.12:0/378828871 >> v2:172.21.15.12:3300/0 conn(0x7f97100a0440 0x7f97100a2200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:39.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.841+0000 7f970ffff640 1 -- 172.21.15.12:0/378828871 >> v1:172.21.15.12:6789/0 conn(0x7f9710137ca0 legacy=0x7f9710138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:39.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.841+0000 7f970ffff640 10 monclient: _add_conns ranks=[1,2] 2024-09-11T01:33:39.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.841+0000 7f970ffff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f96fc003730 addr v1:172.21.15.71:6789/0 2024-09-11T01:33:39.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.841+0000 7f970ffff640 1 --2- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96fc007400 0x7f96fc0098a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:39.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.841+0000 7f970ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f96fc007400 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:39.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.841+0000 7f970ffff640 1 -- 172.21.15.12:0/378828871 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9710070240 con 0x7f96fc003730 2024-09-11T01:33:39.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.841+0000 7f970ffff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:39.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.841+0000 7f970ffff640 1 -- 172.21.15.12:0/378828871 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f97100a0820 con 0x7f96fc007400 2024-09-11T01:33:39.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.841+0000 7f970ffff640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:39.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.841+0000 7f971580d640 1 --2- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96fc007400 0x7f96fc0098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:39.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.841+0000 7f971580d640 10 monclient(hunting): get_auth_request con 0x7f96fc007400 auth_method 0 2024-09-11T01:33:39.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.841+0000 7f971580d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:39.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.841+0000 7f971580d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:39.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.841+0000 7f971580d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:39.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.841+0000 7f971580d640 1 -- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96fc007400 msgr2=0x7f96fc0098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:33:39.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.841+0000 7f971580d640 1 -- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96fc007400 msgr2=0x7f96fc0098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:39.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.841+0000 7f971580d640 1 --2- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96fc007400 0x7f96fc0098a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:39.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.841+0000 7f971580d640 1 --2- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96fc007400 0x7f96fc0098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:33:39.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:39.843+0000 7f971600e640 1 -- 172.21.15.12:0/378828871 >> v1:172.21.15.71:6789/0 conn(0x7f96fc003730 legacy=0x7f96fc005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:40.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:40.042+0000 7f971580d640 1 --2- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96fc007400 0x7f96fc0098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:40.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:40.042+0000 7f971580d640 10 monclient(hunting): get_auth_request con 0x7f96fc007400 auth_method 0 2024-09-11T01:33:40.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:40.042+0000 7f971580d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:40.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:40.042+0000 7f971580d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:40.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:40.042+0000 7f971580d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:40.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:40.042+0000 7f971580d640 1 -- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96fc007400 msgr2=0x7f96fc0098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:33:40.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:40.042+0000 7f971580d640 1 -- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96fc007400 msgr2=0x7f96fc0098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:40.042+0000 7f971580d640 1 --2- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96fc007400 0x7f96fc0098a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:40.042+0000 7f971580d640 1 --2- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96fc007400 0x7f96fc0098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:33:40.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:40.444+0000 7f971580d640 1 --2- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96fc007400 0x7f96fc0098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:40.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:40.444+0000 7f971580d640 10 monclient(hunting): get_auth_request con 0x7f96fc007400 auth_method 0 2024-09-11T01:33:40.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:40.444+0000 7f971580d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:40.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:40.444+0000 7f971580d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:40.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:40.444+0000 7f971580d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:40.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:40.444+0000 7f971580d640 1 -- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96fc007400 msgr2=0x7f96fc0098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:33:40.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:40.444+0000 7f971580d640 1 -- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96fc007400 msgr2=0x7f96fc0098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:40.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:40.444+0000 7f971580d640 1 --2- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96fc007400 0x7f96fc0098a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:40.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:40.444+0000 7f971580d640 1 --2- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96fc007400 0x7f96fc0098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:33:41.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.245+0000 7f971580d640 1 --2- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96fc007400 0x7f96fc0098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:41.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.245+0000 7f971580d640 10 monclient(hunting): get_auth_request con 0x7f96fc007400 auth_method 0 2024-09-11T01:33:41.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.245+0000 7f971580d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:41.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.245+0000 7f971580d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:41.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.245+0000 7f971580d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:41.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.245+0000 7f971580d640 1 -- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96fc007400 msgr2=0x7f96fc0098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:33:41.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.245+0000 7f971580d640 1 -- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96fc007400 msgr2=0x7f96fc0098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:41.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.245+0000 7f971580d640 1 --2- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96fc007400 0x7f96fc0098a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:41.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.245+0000 7f971580d640 1 --2- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96fc007400 0x7f96fc0098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:33:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.270+0000 7f971500c640 1 -- 172.21.15.12:0/378828871 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2606416808 0 0) 0x7f9710070240 con 0x7f96fc003730 2024-09-11T01:33:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.270+0000 7f971500c640 10 monclient(hunting): _negotiate 2024-09-11T01:33:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.270+0000 7f971500c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.270+0000 7f971500c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.270+0000 7f971500c640 10 monclient(hunting): my global_id is 34270 2024-09-11T01:33:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.270+0000 7f971500c640 1 -- 172.21.15.12:0/378828871 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96f8003520 con 0x7f96fc003730 2024-09-11T01:33:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.270+0000 7f971500c640 1 -- 172.21.15.12:0/378828871 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2999701952 0 0) 0x7f96f8003520 con 0x7f96fc003730 2024-09-11T01:33:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.270+0000 7f971500c640 1 -- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96fc007400 msgr2=0x7f96fc0098a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:33:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.270+0000 7f971500c640 1 --2- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96fc007400 0x7f96fc0098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.270+0000 7f971500c640 10 monclient: _finish_hunting 0 2024-09-11T01:33:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.270+0000 7f971500c640 1 monclient: found mon.noname-c-legacy 2024-09-11T01:33:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.270+0000 7f971500c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-09-11T01:33:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 1 -- 172.21.15.12:0/378828871 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96fc00b0e0 con 0x7f96fc003730 2024-09-11T01:33:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 10 monclient: _finish_auth 0 2024-09-11T01:33:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 10 monclient: _check_auth_tickets 2024-09-11T01:33:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 1 -- 172.21.15.12:0/378828871 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f9700009c20 con 0x7f96fc003730 2024-09-11T01:33:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-11T01:33:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 10 monclient: dump: 2024-09-11T01:33:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:41.277 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-09-11T01:33:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 1 -- 172.21.15.12:0/378828871 >> v1:172.21.15.71:6789/0 conn(0x7f96fc003730 legacy=0x7f96fc005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:33:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971680f640 1 -- 172.21.15.12:0/378828871 reap_dead start 2024-09-11T01:33:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 1 --2- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96f8003a50 0x7f96f8003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 10 monclient(hunting): picked mon.a con 0x7f96f8003a50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:33:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 1 --2- 172.21.15.12:0/378828871 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96f8004800 0x7f96f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 10 monclient(hunting): picked mon.b con 0x7f96f8004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:33:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 1 --2- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96f80053c0 0x7f96f800b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 10 monclient(hunting): picked mon.c con 0x7f96f80053c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 1 -- 172.21.15.12:0/378828871 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f97100a0820 con 0x7f96f8003a50 2024-09-11T01:33:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 1 -- 172.21.15.12:0/378828871 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f96f800c2c0 con 0x7f96f8004800 2024-09-11T01:33:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 1 -- 172.21.15.12:0/378828871 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f96f800c4c0 con 0x7f96f80053c0 2024-09-11T01:33:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971500c640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:33:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971680f640 1 --2- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96f80053c0 0x7f96f800b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971580d640 1 --2- 172.21.15.12:0/378828871 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96f8004800 0x7f96f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971680f640 10 monclient(hunting): get_auth_request con 0x7f96f80053c0 auth_method 0 2024-09-11T01:33:41.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971680f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:41.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971680f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:41.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971680f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:41.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971580d640 10 monclient(hunting): get_auth_request con 0x7f96f8004800 auth_method 0 2024-09-11T01:33:41.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971580d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:41.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971580d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:41.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971580d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:41.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971680f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:41.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971680f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:41.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971680f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:33:41.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971580d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:41.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971580d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:41.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971580d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:33:41.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971680f640 10 monclient(hunting): handle_auth_done global_id 34270 payload 293 2024-09-11T01:33:41.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971680f640 1 -- 172.21.15.12:0/378828871 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96f8004800 msgr2=0x7f96f8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:41.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971680f640 1 --2- 172.21.15.12:0/378828871 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96f8004800 0x7f96f8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:41.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971680f640 1 -- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96f8003a50 msgr2=0x7f96f8003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:41.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971680f640 1 --2- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96f8003a50 0x7f96f8003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:41.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971680f640 10 monclient: _finish_hunting 0 2024-09-11T01:33:41.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971680f640 1 monclient: found mon.c 2024-09-11T01:33:41.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971680f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:41.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971680f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:41.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971680f640 1 -- 172.21.15.12:0/378828871 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96f8003520 con 0x7f96f80053c0 2024-09-11T01:33:41.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971680f640 10 monclient: _finish_auth 0 2024-09-11T01:33:41.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971680f640 10 monclient: _check_auth_tickets 2024-09-11T01:33:41.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f971680f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:41.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f9718272640 5 monclient: authenticate success, global_id 34270 2024-09-11T01:33:41.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.271+0000 7f9718272640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:33:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.274+0000 7f971680f640 1 -- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96f80053c0 msgr2=0x7f96f800b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.274+0000 7f971680f640 1 --2- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96f80053c0 0x7f96f800b9e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f97100a0db0 tx=0x7f96f8005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8e6a09496d9c6fd server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.275+0000 7f971500c640 1 -- 172.21.15.12:0/378828871 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97080015a0 con 0x7f96f80053c0 2024-09-11T01:33:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.275+0000 7f971500c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.275+0000 7f971500c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.275+0000 7f971500c640 10 monclient: dump: 2024-09-11T01:33:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:41.288 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.275+0000 7f9718272640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:33:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.275+0000 7f971500c640 1 -- 172.21.15.12:0/378828871 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f970802e070 con 0x7f96f80053c0 2024-09-11T01:33:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.275+0000 7f971500c640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.275+0000 7f971500c640 1 -- 172.21.15.12:0/378828871 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9708004ce0 con 0x7f96f80053c0 2024-09-11T01:33:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.275+0000 7f9718272640 10 monclient: get_monmap_and_config success 2024-09-11T01:33:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.275+0000 7f9718272640 4 set_mon_vals no callback set 2024-09-11T01:33:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.275+0000 7f971500c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.275+0000 7f971500c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.275+0000 7f971500c640 10 monclient: dump: 2024-09-11T01:33:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:41.289 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.276+0000 7f9718272640 10 monclient: shutdown 2024-09-11T01:33:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.276+0000 7f9718272640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.276+0000 7f9718272640 1 -- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96f80053c0 msgr2=0x7f96f800b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.276+0000 7f9718272640 1 --2- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96f80053c0 0x7f96f800b9e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f97100a0db0 tx=0x7f96f8005830 comp rx=0 tx=0).stop 2024-09-11T01:33:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.276+0000 7f9718272640 1 -- 172.21.15.12:0/378828871 shutdown_connections 2024-09-11T01:33:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.276+0000 7f9718272640 1 --2- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96f80053c0 0x7f96f800b9e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.276+0000 7f9718272640 1 --2- 172.21.15.12:0/378828871 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96f8004800 0x7f96f8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.276+0000 7f9718272640 1 --2- 172.21.15.12:0/378828871 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96f8003a50 0x7f96f8003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.276+0000 7f9718272640 1 -- 172.21.15.12:0/378828871 >> 172.21.15.12:0/378828871 conn(0x7f97100a3de0 msgr2=0x7f9710068950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.276+0000 7f9718272640 1 -- 172.21.15.12:0/378828871 shutdown_connections 2024-09-11T01:33:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.276+0000 7f9718272640 1 -- 172.21.15.12:0/378828871 wait complete. 2024-09-11T01:33:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.277+0000 7f9718272640 10 monclient: build_initial_monmap 2024-09-11T01:33:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.277+0000 7f9718272640 10 monclient: monmap: 2024-09-11T01:33:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:33:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:33:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:33:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:33:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:33:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:33:41.291 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.277+0000 7f9718272640 1 Processor -- start 2024-09-11T01:33:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.277+0000 7f9718272640 1 -- start start 2024-09-11T01:33:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.277+0000 7f9718272640 10 monclient: init 2024-09-11T01:33:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f9718272640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f9718272640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:33:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f9718272640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96f8003a50 0x7f971011f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f9718272640 10 monclient(hunting): picked mon.noname-a con 0x7f96f8003a50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:33:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f9718272640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96f8004800 0x7f971011fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f9718272640 10 monclient(hunting): picked mon.noname-c con 0x7f96f8004800 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f9718272640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96f80053c0 0x7f971011ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f9718272640 10 monclient(hunting): picked mon.noname-b con 0x7f96f80053c0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:33:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f9718272640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f9718272640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f97100a2380 con 0x7f96f8003a50 2024-09-11T01:33:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f9718272640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f9718272640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f97100a2200 con 0x7f96f80053c0 2024-09-11T01:33:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f9718272640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f9718272640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f97100a2500 con 0x7f96f8004800 2024-09-11T01:33:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f9718272640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f9718272640 10 monclient(hunting): authenticate will time out at 1027.721255s 2024-09-11T01:33:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971600e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96f8003a50 0x7f971011f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971580d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96f8004800 0x7f971011fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971600e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96f8003a50 0x7f971011f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34428/0 (socket says 172.21.15.12:34428) 2024-09-11T01:33:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971600e640 1 -- 172.21.15.12:0/1229826215 learned_addr learned my addr 172.21.15.12:0/1229826215 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971600e640 10 monclient(hunting): get_auth_request con 0x7f96f8003a50 auth_method 0 2024-09-11T01:33:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971600e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971600e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971600e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971680f640 1 --2- 172.21.15.12:0/1229826215 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96f80053c0 0x7f971011ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971580d640 10 monclient(hunting): get_auth_request con 0x7f96f8004800 auth_method 0 2024-09-11T01:33:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971580d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971580d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971580d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971680f640 10 monclient(hunting): get_auth_request con 0x7f96f80053c0 auth_method 0 2024-09-11T01:33:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971680f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971680f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971680f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971600e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971600e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971600e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971580d640 1 -- 172.21.15.12:0/1229826215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96f8004800 msgr2=0x7f971011fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:33:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971580d640 1 -- 172.21.15.12:0/1229826215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96f8004800 msgr2=0x7f971011fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:33:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971680f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971580d640 1 --2- 172.21.15.12:0/1229826215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96f8004800 0x7f971011fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971680f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971680f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971580d640 1 --2- 172.21.15.12:0/1229826215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96f8004800 0x7f971011fa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:33:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971600e640 10 monclient(hunting): handle_auth_done global_id 54099 payload 824 2024-09-11T01:33:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971600e640 1 -- 172.21.15.12:0/1229826215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96f8004800 msgr2=0x7f971011fa90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:33:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971600e640 1 --2- 172.21.15.12:0/1229826215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96f8004800 0x7f971011fa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971600e640 1 -- 172.21.15.12:0/1229826215 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96f80053c0 msgr2=0x7f971011ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971600e640 1 --2- 172.21.15.12:0/1229826215 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96f80053c0 0x7f971011ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971600e640 10 monclient: _finish_hunting 0 2024-09-11T01:33:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971600e640 1 monclient: found mon.noname-a 2024-09-11T01:33:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971600e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971600e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:33:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971600e640 1 -- 172.21.15.12:0/1229826215 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9710123370 con 0x7f96f8003a50 2024-09-11T01:33:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.278+0000 7f971680f640 1 --2- 172.21.15.12:0/1229826215 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96f80053c0 0x7f971011ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:33:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.281+0000 7f971600e640 1 -- 172.21.15.12:0/1229826215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96f8003a50 msgr2=0x7f971011f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.281+0000 7f971600e640 1 --2- 172.21.15.12:0/1229826215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96f8003a50 0x7f971011f550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9700004640 tx=0x7f9700007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.281+0000 7f970effd640 1 -- 172.21.15.12:0/1229826215 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f970001a070 con 0x7f96f8003a50 2024-09-11T01:33:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.281+0000 7f970effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.281+0000 7f970effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:33:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.281+0000 7f970effd640 10 monclient: dump: 2024-09-11T01:33:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:41.296 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.281+0000 7f970effd640 10 monclient: _finish_auth 0 2024-09-11T01:33:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.281+0000 7f970effd640 10 monclient: _check_auth_tickets 2024-09-11T01:33:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.281+0000 7f970effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.281+0000 7f9718272640 5 monclient: authenticate success, global_id 54099 2024-09-11T01:33:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.281+0000 7f970effd640 1 -- 172.21.15.12:0/1229826215 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9700016620 con 0x7f96f8003a50 2024-09-11T01:33:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.281+0000 7f970effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.281+0000 7f970effd640 1 -- 172.21.15.12:0/1229826215 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9700016a10 con 0x7f96f8003a50 2024-09-11T01:33:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.281+0000 7f9718272640 10 monclient: _renew_subs 2024-09-11T01:33:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.281+0000 7f9718272640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:33:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.281+0000 7f9718272640 1 -- 172.21.15.12:0/1229826215 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9710135e80 con 0x7f96f8003a50 2024-09-11T01:33:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.281+0000 7f970effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.281+0000 7f970effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:33:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.281+0000 7f970effd640 10 monclient: dump: 2024-09-11T01:33:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:41.298 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.281+0000 7f971480b640 4 set_mon_vals no callback set 2024-09-11T01:33:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.281+0000 7f9718272640 10 monclient: _renew_subs 2024-09-11T01:33:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.281+0000 7f9718272640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:33:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.281+0000 7f9718272640 1 -- 172.21.15.12:0/1229826215 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9710121010 con 0x7f96f8003a50 2024-09-11T01:33:41.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.282+0000 7f9718272640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:41.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.282+0000 7f9718272640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:41.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.282+0000 7f9718272640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:33:41.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.282+0000 7f970effd640 1 -- 172.21.15.12:0/1229826215 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f9700016c30 con 0x7f96f8003a50 2024-09-11T01:33:41.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.282+0000 7f9718272640 1 -- 172.21.15.12:0/1229826215 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97100a2c20 con 0x7f96f8003a50 2024-09-11T01:33:41.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.283+0000 7f970effd640 1 --2- 172.21.15.12:0/1229826215 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f96fc03ed40 0x7f96fc041200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:41.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.283+0000 7f970effd640 1 -- 172.21.15.12:0/1229826215 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4799+0+0 (secure 0 0 0) 0x7f9700053cc0 con 0x7f96f8003a50 2024-09-11T01:33:41.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.283+0000 7f971580d640 1 --2- 172.21.15.12:0/1229826215 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f96fc03ed40 0x7f96fc041200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:41.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.283+0000 7f971580d640 10 monclient: get_auth_request con 0x7f96fc03ed40 auth_method 0 2024-09-11T01:33:41.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.285+0000 7f971580d640 1 --2- 172.21.15.12:0/1229826215 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f96fc03ed40 0x7f96fc041200 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9710121b00 tx=0x7f9704008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:41.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.288+0000 7f970effd640 1 -- 172.21.15.12:0/1229826215 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f97000526b0 con 0x7f96f8003a50 2024-09-11T01:33:41.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.288+0000 7f970effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:41.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.288+0000 7f970effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:33:41.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.546+0000 7f9718272640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:33:41.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.546+0000 7f9718272640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:33:41.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.546+0000 7f9718272640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:33:41.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.546+0000 7f9718272640 1 -- 172.21.15.12:0/1229826215 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f971006da60 con 0x7f96f8003a50 2024-09-11T01:33:41.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.547+0000 7f970effd640 1 -- 172.21.15.12:0/1229826215 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f9700015b70 con 0x7f96f8003a50 2024-09-11T01:33:41.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.547+0000 7f970effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:33:41.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.547+0000 7f970effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:33:41.551 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":44,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:33:41.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.549+0000 7f9718272640 1 -- 172.21.15.12:0/1229826215 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f96fc03ed40 msgr2=0x7f96fc041200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:41.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.549+0000 7f9718272640 1 --2- 172.21.15.12:0/1229826215 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f96fc03ed40 0x7f96fc041200 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9710121b00 tx=0x7f9704008040 comp rx=0 tx=0).stop 2024-09-11T01:33:41.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.549+0000 7f9718272640 10 monclient: shutdown 2024-09-11T01:33:41.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.549+0000 7f9718272640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:41.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.549+0000 7f9718272640 1 -- 172.21.15.12:0/1229826215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96f8003a50 msgr2=0x7f971011f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:41.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.549+0000 7f9718272640 1 --2- 172.21.15.12:0/1229826215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96f8003a50 0x7f971011f550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9700004640 tx=0x7f9700007620 comp rx=0 tx=0).stop 2024-09-11T01:33:41.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.549+0000 7f9718272640 1 -- 172.21.15.12:0/1229826215 shutdown_connections 2024-09-11T01:33:41.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.549+0000 7f9718272640 1 --2- 172.21.15.12:0/1229826215 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96f80053c0 0x7f971011ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:41.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.549+0000 7f9718272640 1 --2- 172.21.15.12:0/1229826215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96f8004800 0x7f971011fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:41.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.549+0000 7f9718272640 1 --2- 172.21.15.12:0/1229826215 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f96fc03ed40 0x7f96fc041200 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:41.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.549+0000 7f9718272640 1 --2- 172.21.15.12:0/1229826215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96f8003a50 0x7f971011f550 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:41.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.549+0000 7f9718272640 1 -- 172.21.15.12:0/1229826215 >> 172.21.15.12:0/1229826215 conn(0x7f97100a3de0 msgr2=0x7f9710072490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:41.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.549+0000 7f9718272640 1 -- 172.21.15.12:0/1229826215 shutdown_connections 2024-09-11T01:33:41.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:41.549+0000 7f9718272640 1 -- 172.21.15.12:0/1229826215 wait complete. 2024-09-11T01:33:44.564 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:33:44.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f42563e2640 10 monclient: get_monmap_and_config 2024-09-11T01:33:44.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f42563e2640 10 monclient: build_initial_monmap 2024-09-11T01:33:44.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f42563e2640 10 monclient: monmap: 2024-09-11T01:33:44.678 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:44.678 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:44.678 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:33:44.676946+0000 2024-09-11T01:33:44.678 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:33:44.676946+0000 2024-09-11T01:33:44.678 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:44.678 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:44.678 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:33:44.678 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:33:44.678 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:33:44.678 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:33:44.678 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:33:44.678 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f42563e2640 1 Processor -- start 2024-09-11T01:33:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f42563e2640 1 -- start start 2024-09-11T01:33:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f42563e2640 10 monclient: init 2024-09-11T01:33:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f42563e2640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f42563e2640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-11T01:33:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f42563e2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4250138230 addr v1:172.21.15.12:6789/0 2024-09-11T01:33:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f42563e2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f425009d030 addr v1:172.21.15.71:6789/0 2024-09-11T01:33:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f42563e2640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f425009d460 0x7f42501407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f42563e2640 10 monclient(hunting): picked mon.noname-a con 0x7f425009d460 addr v2:172.21.15.12:3300/0 2024-09-11T01:33:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f42563e2640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42500a6e30 con 0x7f4250138230 2024-09-11T01:33:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f42563e2640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f425009ba20 con 0x7f425009d030 2024-09-11T01:33:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f42563e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f42563e2640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f425009c9c0 con 0x7f425009d460 2024-09-11T01:33:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f42563e2640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f42563e2640 10 monclient(hunting): authenticate will time out at 1031.119638s 2024-09-11T01:33:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f425497f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f425009d460 0x7f42501407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f424ffff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4250138230 0x7f4250138610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53884/0 (socket says 172.21.15.12:53884) 2024-09-11T01:33:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f424ffff640 1 -- 172.21.15.12:0/129128262 learned_addr learned my addr 172.21.15.12:0/129128262 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:33:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f425497f640 10 monclient(hunting): get_auth_request con 0x7f425009d460 auth_method 0 2024-09-11T01:33:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f425497f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f425497f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f425497f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f425497f640 1 -- 172.21.15.12:0/129128262 >> v2:172.21.15.12:3300/0 conn(0x7f425009d460 msgr2=0x7f42501407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:33:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f425497f640 1 -- 172.21.15.12:0/129128262 >> v2:172.21.15.12:3300/0 conn(0x7f425009d460 msgr2=0x7f42501407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f425497f640 1 --2- 172.21.15.12:0/129128262 >> v2:172.21.15.12:3300/0 conn(0x7f425009d460 0x7f42501407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.676+0000 7f425497f640 1 --2- 172.21.15.12:0/129128262 >> v2:172.21.15.12:3300/0 conn(0x7f425009d460 0x7f42501407c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:33:44.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.678+0000 7f424ffff640 1 -- 172.21.15.12:0/129128262 >> v1:172.21.15.12:6789/0 conn(0x7f4250138230 legacy=0x7f4250138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:44.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.680+0000 7f424f7fe640 1 -- 172.21.15.12:0/129128262 >> v1:172.21.15.71:6789/0 conn(0x7f425009d030 legacy=0x7f425013d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:44.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.878+0000 7f425497f640 1 --2- 172.21.15.12:0/129128262 >> v2:172.21.15.12:3300/0 conn(0x7f425009d460 0x7f42501407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:44.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.878+0000 7f425497f640 10 monclient(hunting): get_auth_request con 0x7f425009d460 auth_method 0 2024-09-11T01:33:44.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.878+0000 7f425497f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:44.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.878+0000 7f425497f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:44.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.878+0000 7f425497f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:44.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.878+0000 7f425497f640 1 -- 172.21.15.12:0/129128262 >> v2:172.21.15.12:3300/0 conn(0x7f425009d460 msgr2=0x7f42501407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:33:44.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.878+0000 7f425497f640 1 -- 172.21.15.12:0/129128262 >> v2:172.21.15.12:3300/0 conn(0x7f425009d460 msgr2=0x7f42501407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.878+0000 7f425497f640 1 --2- 172.21.15.12:0/129128262 >> v2:172.21.15.12:3300/0 conn(0x7f425009d460 0x7f42501407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:44.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:44.878+0000 7f425497f640 1 --2- 172.21.15.12:0/129128262 >> v2:172.21.15.12:3300/0 conn(0x7f425009d460 0x7f42501407c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:33:45.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:45.279+0000 7f425497f640 1 --2- 172.21.15.12:0/129128262 >> v2:172.21.15.12:3300/0 conn(0x7f425009d460 0x7f42501407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:45.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:45.279+0000 7f425497f640 10 monclient(hunting): get_auth_request con 0x7f425009d460 auth_method 0 2024-09-11T01:33:45.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:45.279+0000 7f425497f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:45.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:45.279+0000 7f425497f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:45.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:45.279+0000 7f425497f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:45.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:45.279+0000 7f425497f640 1 -- 172.21.15.12:0/129128262 >> v2:172.21.15.12:3300/0 conn(0x7f425009d460 msgr2=0x7f42501407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:33:45.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:45.279+0000 7f425497f640 1 -- 172.21.15.12:0/129128262 >> v2:172.21.15.12:3300/0 conn(0x7f425009d460 msgr2=0x7f42501407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:45.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:45.279+0000 7f425497f640 1 --2- 172.21.15.12:0/129128262 >> v2:172.21.15.12:3300/0 conn(0x7f425009d460 0x7f42501407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:45.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:45.279+0000 7f425497f640 1 --2- 172.21.15.12:0/129128262 >> v2:172.21.15.12:3300/0 conn(0x7f425009d460 0x7f42501407c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:33:46.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:46.080+0000 7f425497f640 1 --2- 172.21.15.12:0/129128262 >> v2:172.21.15.12:3300/0 conn(0x7f425009d460 0x7f42501407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:46.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:46.080+0000 7f425497f640 10 monclient(hunting): get_auth_request con 0x7f425009d460 auth_method 0 2024-09-11T01:33:46.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:46.080+0000 7f425497f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:46.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:46.080+0000 7f425497f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:46.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:46.080+0000 7f425497f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:46.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:46.080+0000 7f425497f640 1 -- 172.21.15.12:0/129128262 >> v2:172.21.15.12:3300/0 conn(0x7f425009d460 msgr2=0x7f42501407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:33:46.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:46.080+0000 7f425497f640 1 -- 172.21.15.12:0/129128262 >> v2:172.21.15.12:3300/0 conn(0x7f425009d460 msgr2=0x7f42501407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:46.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:46.080+0000 7f425497f640 1 --2- 172.21.15.12:0/129128262 >> v2:172.21.15.12:3300/0 conn(0x7f425009d460 0x7f42501407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:46.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:46.080+0000 7f425497f640 1 --2- 172.21.15.12:0/129128262 >> v2:172.21.15.12:3300/0 conn(0x7f425009d460 0x7f42501407c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:33:47.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 1 -- 172.21.15.12:0/129128262 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1019646274 0 0) 0x7f425009ba20 con 0x7f425009d030 2024-09-11T01:33:47.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 10 monclient(hunting): _negotiate 2024-09-11T01:33:47.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:47.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:47.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 10 monclient(hunting): my global_id is 34276 2024-09-11T01:33:47.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 1 -- 172.21.15.12:0/129128262 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4238003520 con 0x7f425009d030 2024-09-11T01:33:47.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 1 -- 172.21.15.12:0/129128262 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1455041711 0 0) 0x7f4238003520 con 0x7f425009d030 2024-09-11T01:33:47.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 1 -- 172.21.15.12:0/129128262 >> v2:172.21.15.12:3300/0 conn(0x7f425009d460 msgr2=0x7f42501407c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:33:47.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 1 --2- 172.21.15.12:0/129128262 >> v2:172.21.15.12:3300/0 conn(0x7f425009d460 0x7f42501407c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:47.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 1 -- 172.21.15.12:0/129128262 >> v1:172.21.15.12:6789/0 conn(0x7f4250138230 legacy=0x7f4250138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:47.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 10 monclient: _finish_hunting 0 2024-09-11T01:33:47.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 1 monclient: found mon.noname-c-legacy 2024-09-11T01:33:47.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:47.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-09-11T01:33:47.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 1 -- 172.21.15.12:0/129128262 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42500a66a0 con 0x7f425009d030 2024-09-11T01:33:47.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 10 monclient: _finish_auth 0 2024-09-11T01:33:47.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 10 monclient: _check_auth_tickets 2024-09-11T01:33:47.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:47.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 1 -- 172.21.15.12:0/129128262 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f42440019f0 con 0x7f425009d030 2024-09-11T01:33:47.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f42563e2640 5 monclient: authenticate success, global_id 34276 2024-09-11T01:33:47.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f42563e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:33:47.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:47.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-11T01:33:47.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 10 monclient: dump: 2024-09-11T01:33:47.279 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:47.279 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:47.279 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:47.279 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:47.279 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:47.279 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:47.279 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:47.279 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:47.279 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:47.279 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:47.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-09-11T01:33:47.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:47.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 1 -- 172.21.15.12:0/129128262 >> v1:172.21.15.71:6789/0 conn(0x7f425009d030 legacy=0x7f425013d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:47.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:33:47.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 1 --2- 172.21.15.12:0/129128262 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4238003b20 0x7f4238003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:47.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 10 monclient(hunting): picked mon.c con 0x7f4238003b20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:47.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 1 --2- 172.21.15.12:0/129128262 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4238004800 0x7f4238004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:47.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 10 monclient(hunting): picked mon.b con 0x7f4238004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:33:47.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 1 --2- 172.21.15.12:0/129128262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42380099b0 0x7f423800bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:47.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 10 monclient(hunting): picked mon.a con 0x7f42380099b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:33:47.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:47.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 1 -- 172.21.15.12:0/129128262 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f425009c9c0 con 0x7f42380099b0 2024-09-11T01:33:47.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:47.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 1 -- 172.21.15.12:0/129128262 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f423800c3c0 con 0x7f4238004800 2024-09-11T01:33:47.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:47.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 1 -- 172.21.15.12:0/129128262 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f423800c5c0 con 0x7f4238003b20 2024-09-11T01:33:47.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:47.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424effd640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:33:47.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f42563e2640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:33:47.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424ffff640 1 --2- 172.21.15.12:0/129128262 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4238003b20 0x7f4238003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:47.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424ffff640 10 monclient(hunting): get_auth_request con 0x7f4238003b20 auth_method 0 2024-09-11T01:33:47.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:47.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:47.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:47.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424f7fe640 1 --2- 172.21.15.12:0/129128262 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4238004800 0x7f4238004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:47.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424f7fe640 10 monclient(hunting): get_auth_request con 0x7f4238004800 auth_method 0 2024-09-11T01:33:47.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:47.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:47.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.273+0000 7f424f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:47.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.274+0000 7f424ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:47.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.274+0000 7f424ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:47.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.274+0000 7f424ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:33:47.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.274+0000 7f424f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:47.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.274+0000 7f424f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:47.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.274+0000 7f424f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:33:47.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.274+0000 7f424ffff640 10 monclient(hunting): handle_auth_done global_id 34276 payload 293 2024-09-11T01:33:47.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.274+0000 7f424ffff640 1 -- 172.21.15.12:0/129128262 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4238004800 msgr2=0x7f4238004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:47.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.274+0000 7f424ffff640 1 --2- 172.21.15.12:0/129128262 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4238004800 0x7f4238004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:47.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.274+0000 7f424ffff640 1 -- 172.21.15.12:0/129128262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42380099b0 msgr2=0x7f423800bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:47.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.274+0000 7f424ffff640 1 --2- 172.21.15.12:0/129128262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42380099b0 0x7f423800bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:47.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.274+0000 7f424ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:33:47.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.274+0000 7f424ffff640 1 monclient: found mon.c 2024-09-11T01:33:47.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.274+0000 7f424ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:47.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.274+0000 7f424ffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:47.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.274+0000 7f424ffff640 1 -- 172.21.15.12:0/129128262 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4238003520 con 0x7f4238003b20 2024-09-11T01:33:47.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.274+0000 7f425497f640 1 -- 172.21.15.12:0/129128262 reap_dead start 2024-09-11T01:33:47.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.274+0000 7f424ffff640 10 monclient: _finish_auth 0 2024-09-11T01:33:47.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.274+0000 7f424ffff640 10 monclient: _check_auth_tickets 2024-09-11T01:33:47.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.274+0000 7f424ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:47.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.274+0000 7f424f7fe640 1 --2- 172.21.15.12:0/129128262 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4238004800 0x7f4238004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:33:47.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.277+0000 7f424ffff640 1 -- 172.21.15.12:0/129128262 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4238003b20 msgr2=0x7f4238003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:47.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.277+0000 7f424ffff640 1 --2- 172.21.15.12:0/129128262 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4238003b20 0x7f4238003f40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4240001a30 tx=0x7f4238005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96bc8ea50f8210db server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:47.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.277+0000 7f424ffff640 1 --2- 172.21.15.12:0/129128262 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4238003b20 0x7f4238003f40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4240001a30 tx=0x7f4238005830 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7f424002e070 config(0 keys) 2024-09-11T01:33:47.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.277+0000 7f424effd640 1 -- 172.21.15.12:0/129128262 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4240001760 con 0x7f4238003b20 2024-09-11T01:33:47.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.277+0000 7f424effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:47.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.277+0000 7f424effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:47.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.277+0000 7f424effd640 10 monclient: dump: 2024-09-11T01:33:47.291 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:47.291 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:47.291 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:47.291 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:47.291 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:47.292 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:47.292 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:47.292 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:47.292 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:47.292 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:47.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.277+0000 7f424effd640 1 -- 172.21.15.12:0/129128262 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f424002e070 con 0x7f4238003b20 2024-09-11T01:33:47.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.277+0000 7f424effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:47.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.277+0000 7f42563e2640 10 monclient: get_monmap_and_config success 2024-09-11T01:33:47.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.277+0000 7f42563e2640 4 set_mon_vals no callback set 2024-09-11T01:33:47.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.278+0000 7f42563e2640 10 monclient: shutdown 2024-09-11T01:33:47.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.278+0000 7f42563e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:47.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.278+0000 7f42563e2640 1 -- 172.21.15.12:0/129128262 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4238003b20 msgr2=0x7f4238003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:47.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.278+0000 7f42563e2640 1 --2- 172.21.15.12:0/129128262 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4238003b20 0x7f4238003f40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4240001a30 tx=0x7f4238005830 comp rx=0 tx=0).stop 2024-09-11T01:33:47.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.278+0000 7f424effd640 1 -- 172.21.15.12:0/129128262 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 0 0 0) 0x7f42400048a0 con 0x7f4238003b20 2024-09-11T01:33:47.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.278+0000 7f424effd640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-09-11T01:33:47.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.278+0000 7f42563e2640 1 -- 172.21.15.12:0/129128262 shutdown_connections 2024-09-11T01:33:47.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.278+0000 7f42563e2640 1 --2- 172.21.15.12:0/129128262 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4238003b20 0x7f4238003f40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:47.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.278+0000 7f42563e2640 1 -- 172.21.15.12:0/129128262 >> 172.21.15.12:0/129128262 conn(0x7f42500a7120 msgr2=0x7f42500a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:47.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.278+0000 7f42563e2640 1 -- 172.21.15.12:0/129128262 shutdown_connections 2024-09-11T01:33:47.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.278+0000 7f42563e2640 1 -- 172.21.15.12:0/129128262 wait complete. 2024-09-11T01:33:47.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.279+0000 7f42563e2640 10 monclient: build_initial_monmap 2024-09-11T01:33:47.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.279+0000 7f42563e2640 10 monclient: monmap: 2024-09-11T01:33:47.294 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:47.294 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:33:47.294 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:33:47.294 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:33:47.294 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:47.294 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:47.294 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:33:47.294 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:33:47.294 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:33:47.294 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:47.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.279+0000 7f42563e2640 1 Processor -- start 2024-09-11T01:33:47.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.280+0000 7f42563e2640 1 -- start start 2024-09-11T01:33:47.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.280+0000 7f42563e2640 10 monclient: init 2024-09-11T01:33:47.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.280+0000 7f42563e2640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:47.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.280+0000 7f42563e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:33:47.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.280+0000 7f42563e2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4238003b20 0x7f42501d12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:47.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.280+0000 7f42563e2640 10 monclient(hunting): picked mon.noname-a con 0x7f4238003b20 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:33:47.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.280+0000 7f42563e2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42501d1a10 0x7f425006fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:47.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.280+0000 7f42563e2640 10 monclient(hunting): picked mon.noname-c con 0x7f42501d1a10 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:47.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.280+0000 7f42563e2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f42501d1e70 0x7f42500703b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:47.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.280+0000 7f42563e2640 10 monclient(hunting): picked mon.noname-b con 0x7f42501d1e70 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:33:47.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.280+0000 7f42563e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:47.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.280+0000 7f42563e2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4250138790 con 0x7f4238003b20 2024-09-11T01:33:47.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.280+0000 7f42563e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:47.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.280+0000 7f42563e2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4250138610 con 0x7f42501d1e70 2024-09-11T01:33:47.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.280+0000 7f42563e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:47.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.280+0000 7f42563e2640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4250138910 con 0x7f42501d1a10 2024-09-11T01:33:47.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.280+0000 7f42563e2640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:47.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.280+0000 7f42563e2640 10 monclient(hunting): authenticate will time out at 1033.722812s 2024-09-11T01:33:47.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.280+0000 7f424f7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42501d1a10 0x7f425006fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:47.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.280+0000 7f424f7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42501d1a10 0x7f425006fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:47780/0 (socket says 172.21.15.12:47780) 2024-09-11T01:33:47.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.280+0000 7f424ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4238003b20 0x7f42501d12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:47.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.280+0000 7f424f7fe640 1 -- 172.21.15.12:0/504590060 learned_addr learned my addr 172.21.15.12:0/504590060 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:47.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.280+0000 7f424f7fe640 10 monclient(hunting): get_auth_request con 0x7f42501d1a10 auth_method 0 2024-09-11T01:33:47.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.280+0000 7f424f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:47.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.280+0000 7f424f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:47.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.280+0000 7f424f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:47.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f424ffff640 10 monclient(hunting): get_auth_request con 0x7f4238003b20 auth_method 0 2024-09-11T01:33:47.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f424ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:47.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f424ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:47.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f424ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:47.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f424f7fe640 1 -- 172.21.15.12:0/504590060 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42501d1a10 msgr2=0x7f425006fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:33:47.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f424f7fe640 1 -- 172.21.15.12:0/504590060 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42501d1a10 msgr2=0x7f425006fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:33:47.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f424f7fe640 1 --2- 172.21.15.12:0/504590060 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42501d1a10 0x7f425006fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:47.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f425497f640 1 --2- 172.21.15.12:0/504590060 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f42501d1e70 0x7f42500703b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:47.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f424f7fe640 1 --2- 172.21.15.12:0/504590060 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42501d1a10 0x7f425006fe70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:33:47.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f425497f640 10 monclient(hunting): get_auth_request con 0x7f42501d1e70 auth_method 0 2024-09-11T01:33:47.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f425497f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:47.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f425497f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:47.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f425497f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:47.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f424ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:47.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f424ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:47.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f424ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:47.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f425497f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:47.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f425497f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:47.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f425497f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:47.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f424ffff640 10 monclient(hunting): handle_auth_done global_id 54102 payload 824 2024-09-11T01:33:47.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f424ffff640 1 -- 172.21.15.12:0/504590060 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42501d1a10 msgr2=0x7f425006fe70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:33:47.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f424ffff640 1 --2- 172.21.15.12:0/504590060 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42501d1a10 0x7f425006fe70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:47.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f424ffff640 1 -- 172.21.15.12:0/504590060 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f42501d1e70 msgr2=0x7f42500703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:47.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f424ffff640 1 --2- 172.21.15.12:0/504590060 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f42501d1e70 0x7f42500703b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:47.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f424ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:33:47.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f424ffff640 1 monclient: found mon.noname-a 2024-09-11T01:33:47.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f424ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:47.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f424ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:33:47.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f424ffff640 1 -- 172.21.15.12:0/504590060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42500738c0 con 0x7f4238003b20 2024-09-11T01:33:47.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.281+0000 7f425497f640 1 --2- 172.21.15.12:0/504590060 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f42501d1e70 0x7f42500703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:33:47.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.283+0000 7f424ffff640 1 -- 172.21.15.12:0/504590060 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4238003b20 msgr2=0x7f42501d12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:47.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.283+0000 7f424ffff640 1 --2- 172.21.15.12:0/504590060 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4238003b20 0x7f42501d12e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4240032000 tx=0x7f4240001460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:47.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.284+0000 7f424cff9640 1 -- 172.21.15.12:0/504590060 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4240044070 con 0x7f4238003b20 2024-09-11T01:33:47.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.284+0000 7f424cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:47.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.284+0000 7f424cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:33:47.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.284+0000 7f424cff9640 10 monclient: dump: 2024-09-11T01:33:47.299 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:47.299 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:47.299 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:47.299 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:47.300 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:47.300 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:47.300 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:47.300 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:47.300 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:47.300 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:47.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.284+0000 7f424cff9640 10 monclient: _finish_auth 0 2024-09-11T01:33:47.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.284+0000 7f424cff9640 10 monclient: _check_auth_tickets 2024-09-11T01:33:47.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.284+0000 7f424cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:47.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.284+0000 7f424cff9640 1 -- 172.21.15.12:0/504590060 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4240004c90 con 0x7f4238003b20 2024-09-11T01:33:47.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.284+0000 7f42563e2640 5 monclient: authenticate success, global_id 54102 2024-09-11T01:33:47.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.284+0000 7f42563e2640 10 monclient: _renew_subs 2024-09-11T01:33:47.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.284+0000 7f42563e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:33:47.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.284+0000 7f42563e2640 1 -- 172.21.15.12:0/504590060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4250073ba0 con 0x7f4238003b20 2024-09-11T01:33:47.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.284+0000 7f424cff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:47.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.284+0000 7f424cff9640 1 -- 172.21.15.12:0/504590060 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42400072b0 con 0x7f4238003b20 2024-09-11T01:33:47.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.284+0000 7f424cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:47.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.284+0000 7f424cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:33:47.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.284+0000 7f424cff9640 10 monclient: dump: 2024-09-11T01:33:47.301 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:47.301 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:47.301 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:47.301 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:47.301 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:47.301 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:47.301 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:47.301 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:47.301 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:47.302 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.284+0000 7f42563e2640 10 monclient: _renew_subs 2024-09-11T01:33:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.284+0000 7f42563e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:33:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.284+0000 7f42563e2640 1 -- 172.21.15.12:0/504590060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42500a1430 con 0x7f4238003b20 2024-09-11T01:33:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.284+0000 7f424e7fc640 4 set_mon_vals no callback set 2024-09-11T01:33:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.285+0000 7f424cff9640 1 -- 172.21.15.12:0/504590060 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f4240007450 con 0x7f4238003b20 2024-09-11T01:33:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.285+0000 7f42563e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.285+0000 7f42563e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.285+0000 7f42563e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:33:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.285+0000 7f42563e2640 1 -- 172.21.15.12:0/504590060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42500a5020 con 0x7f4238003b20 2024-09-11T01:33:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.285+0000 7f424cff9640 1 --2- 172.21.15.12:0/504590060 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f423c03e210 0x7f423c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.285+0000 7f424cff9640 1 -- 172.21.15.12:0/504590060 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4799+0+0 (secure 0 0 0) 0x7f4240072500 con 0x7f4238003b20 2024-09-11T01:33:47.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.285+0000 7f424f7fe640 1 --2- 172.21.15.12:0/504590060 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f423c03e210 0x7f423c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:47.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.285+0000 7f424f7fe640 10 monclient: get_auth_request con 0x7f423c03e210 auth_method 0 2024-09-11T01:33:47.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.288+0000 7f424f7fe640 1 --2- 172.21.15.12:0/504590060 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f423c03e210 0x7f423c0406d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f42440014b0 tx=0x7f4244009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:47.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.290+0000 7f424cff9640 1 -- 172.21.15.12:0/504590060 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4240007770 con 0x7f4238003b20 2024-09-11T01:33:47.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.290+0000 7f424cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:47.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.290+0000 7f424cff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:33:47.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.547+0000 7f42563e2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:33:47.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.547+0000 7f42563e2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:33:47.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.547+0000 7f42563e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:33:47.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.547+0000 7f42563e2640 1 -- 172.21.15.12:0/504590060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f42500736a0 con 0x7f4238003b20 2024-09-11T01:33:47.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.548+0000 7f424cff9640 1 -- 172.21.15.12:0/504590060 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f424003d380 con 0x7f4238003b20 2024-09-11T01:33:47.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.548+0000 7f424cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:33:47.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.548+0000 7f424cff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:33:47.550 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":48,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:33:47.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.550+0000 7f42563e2640 1 -- 172.21.15.12:0/504590060 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f423c03e210 msgr2=0x7f423c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:47.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.550+0000 7f42563e2640 1 --2- 172.21.15.12:0/504590060 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f423c03e210 0x7f423c0406d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f42440014b0 tx=0x7f4244009040 comp rx=0 tx=0).stop 2024-09-11T01:33:47.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.550+0000 7f42563e2640 10 monclient: shutdown 2024-09-11T01:33:47.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.550+0000 7f42563e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:47.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.550+0000 7f42563e2640 1 -- 172.21.15.12:0/504590060 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4238003b20 msgr2=0x7f42501d12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:47.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.550+0000 7f42563e2640 1 --2- 172.21.15.12:0/504590060 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4238003b20 0x7f42501d12e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4240032000 tx=0x7f4240001460 comp rx=0 tx=0).stop 2024-09-11T01:33:47.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.550+0000 7f42563e2640 1 -- 172.21.15.12:0/504590060 shutdown_connections 2024-09-11T01:33:47.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.550+0000 7f42563e2640 1 --2- 172.21.15.12:0/504590060 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f42501d1e70 0x7f42500703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:47.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.550+0000 7f42563e2640 1 --2- 172.21.15.12:0/504590060 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42501d1a10 0x7f425006fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:47.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.550+0000 7f42563e2640 1 --2- 172.21.15.12:0/504590060 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f423c03e210 0x7f423c0406d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:47.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.550+0000 7f42563e2640 1 --2- 172.21.15.12:0/504590060 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4238003b20 0x7f42501d12e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:47.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.550+0000 7f42563e2640 1 -- 172.21.15.12:0/504590060 >> 172.21.15.12:0/504590060 conn(0x7f42500a7120 msgr2=0x7f42500a3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:47.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.550+0000 7f42563e2640 1 -- 172.21.15.12:0/504590060 shutdown_connections 2024-09-11T01:33:47.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:47.550+0000 7f42563e2640 1 -- 172.21.15.12:0/504590060 wait complete. 2024-09-11T01:33:50.563 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:33:50.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.674+0000 7fcd1ae08640 10 monclient: get_monmap_and_config 2024-09-11T01:33:50.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.674+0000 7fcd1ae08640 10 monclient: build_initial_monmap 2024-09-11T01:33:50.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.674+0000 7fcd1ae08640 10 monclient: monmap: 2024-09-11T01:33:50.676 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:50.677 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:50.677 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:33:50.675513+0000 2024-09-11T01:33:50.677 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:33:50.675513+0000 2024-09-11T01:33:50.677 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:50.677 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:50.677 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:33:50.677 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:33:50.677 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:33:50.677 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:33:50.677 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:33:50.677 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:50.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.675+0000 7fcd1ae08640 1 Processor -- start 2024-09-11T01:33:50.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.675+0000 7fcd1ae08640 1 -- start start 2024-09-11T01:33:50.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.675+0000 7fcd1ae08640 10 monclient: init 2024-09-11T01:33:50.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd1ae08640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:50.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd1ae08640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-11T01:33:50.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd1ae08640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 0x7fcd140a1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:50.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd1ae08640 10 monclient(hunting): picked mon.noname-b con 0x7fcd140a1380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:50.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd1ae08640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcd140a0c90 0x7fcd14135ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:50.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd1ae08640 10 monclient(hunting): picked mon.noname-a con 0x7fcd140a0c90 addr v2:172.21.15.12:3300/0 2024-09-11T01:33:50.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd1ae08640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcd14136020 addr v1:172.21.15.12:6789/0 2024-09-11T01:33:50.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd1ae08640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd140a2b40 con 0x7fcd14136020 2024-09-11T01:33:50.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd1ae08640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:50.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd1ae08640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fcd1409f9a0 con 0x7fcd140a0c90 2024-09-11T01:33:50.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd1ae08640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:50.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd1ae08640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcd140a2230 con 0x7fcd140a1380 2024-09-11T01:33:50.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd1ae08640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:50.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd1ae08640 10 monclient(hunting): authenticate will time out at 1037.118270s 2024-09-11T01:33:50.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd18ba4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 0x7fcd140a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:50.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd193a5640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fcd14136020 0x7fcd1413ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54010/0 (socket says 172.21.15.12:54010) 2024-09-11T01:33:50.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd193a5640 1 -- 172.21.15.12:0/2552392999 learned_addr learned my addr 172.21.15.12:0/2552392999 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:33:50.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd13fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcd140a0c90 0x7fcd14135ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:50.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd18ba4640 10 monclient(hunting): get_auth_request con 0x7fcd140a1380 auth_method 0 2024-09-11T01:33:50.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd18ba4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:50.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd18ba4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:50.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd18ba4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:50.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd13fff640 10 monclient(hunting): get_auth_request con 0x7fcd140a0c90 auth_method 0 2024-09-11T01:33:50.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:50.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd13fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:50.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd13fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:50.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd18ba4640 1 -- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 msgr2=0x7fcd140a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:33:50.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd13fff640 1 -- 172.21.15.12:0/2552392999 >> v2:172.21.15.12:3300/0 conn(0x7fcd140a0c90 msgr2=0x7fcd14135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:33:50.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd18ba4640 1 -- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 msgr2=0x7fcd140a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:50.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd13fff640 1 -- 172.21.15.12:0/2552392999 >> v2:172.21.15.12:3300/0 conn(0x7fcd140a0c90 msgr2=0x7fcd14135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:50.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd18ba4640 1 --2- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 0x7fcd140a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:50.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd13fff640 1 --2- 172.21.15.12:0/2552392999 >> v2:172.21.15.12:3300/0 conn(0x7fcd140a0c90 0x7fcd14135ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:50.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd13fff640 1 --2- 172.21.15.12:0/2552392999 >> v2:172.21.15.12:3300/0 conn(0x7fcd140a0c90 0x7fcd14135ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:33:50.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.676+0000 7fcd18ba4640 1 --2- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 0x7fcd140a1760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:33:50.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.678+0000 7fcd193a5640 1 -- 172.21.15.12:0/2552392999 >> v1:172.21.15.12:6789/0 conn(0x7fcd14136020 legacy=0x7fcd1413ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:50.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.878+0000 7fcd18ba4640 1 --2- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 0x7fcd140a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:50.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.878+0000 7fcd13fff640 1 --2- 172.21.15.12:0/2552392999 >> v2:172.21.15.12:3300/0 conn(0x7fcd140a0c90 0x7fcd14135ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:50.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.878+0000 7fcd18ba4640 10 monclient(hunting): get_auth_request con 0x7fcd140a1380 auth_method 0 2024-09-11T01:33:50.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.878+0000 7fcd18ba4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:50.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.878+0000 7fcd18ba4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:50.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.878+0000 7fcd18ba4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:50.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.878+0000 7fcd13fff640 10 monclient(hunting): get_auth_request con 0x7fcd140a0c90 auth_method 0 2024-09-11T01:33:50.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.878+0000 7fcd13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:50.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.878+0000 7fcd13fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:50.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.878+0000 7fcd13fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:50.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.878+0000 7fcd18ba4640 1 -- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 msgr2=0x7fcd140a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:33:50.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.878+0000 7fcd18ba4640 1 -- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 msgr2=0x7fcd140a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:50.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.878+0000 7fcd18ba4640 1 --2- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 0x7fcd140a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:50.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.878+0000 7fcd13fff640 1 -- 172.21.15.12:0/2552392999 >> v2:172.21.15.12:3300/0 conn(0x7fcd140a0c90 msgr2=0x7fcd14135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:33:50.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.878+0000 7fcd13fff640 1 -- 172.21.15.12:0/2552392999 >> v2:172.21.15.12:3300/0 conn(0x7fcd140a0c90 msgr2=0x7fcd14135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:50.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.878+0000 7fcd13fff640 1 --2- 172.21.15.12:0/2552392999 >> v2:172.21.15.12:3300/0 conn(0x7fcd140a0c90 0x7fcd14135ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:50.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.878+0000 7fcd18ba4640 1 --2- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 0x7fcd140a1760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:33:50.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:50.878+0000 7fcd13fff640 1 --2- 172.21.15.12:0/2552392999 >> v2:172.21.15.12:3300/0 conn(0x7fcd140a0c90 0x7fcd14135ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:33:51.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:51.280+0000 7fcd18ba4640 1 --2- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 0x7fcd140a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:51.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:51.280+0000 7fcd13fff640 1 --2- 172.21.15.12:0/2552392999 >> v2:172.21.15.12:3300/0 conn(0x7fcd140a0c90 0x7fcd14135ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:51.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:51.280+0000 7fcd18ba4640 10 monclient(hunting): get_auth_request con 0x7fcd140a1380 auth_method 0 2024-09-11T01:33:51.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:51.280+0000 7fcd18ba4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:51.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:51.280+0000 7fcd18ba4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:51.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:51.280+0000 7fcd18ba4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:51.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:51.280+0000 7fcd13fff640 10 monclient(hunting): get_auth_request con 0x7fcd140a0c90 auth_method 0 2024-09-11T01:33:51.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:51.280+0000 7fcd13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:51.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:51.280+0000 7fcd13fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:51.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:51.280+0000 7fcd13fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:51.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:51.280+0000 7fcd18ba4640 1 -- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 msgr2=0x7fcd140a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:33:51.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:51.280+0000 7fcd18ba4640 1 -- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 msgr2=0x7fcd140a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:51.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:51.280+0000 7fcd18ba4640 1 --2- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 0x7fcd140a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:51.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:51.280+0000 7fcd13fff640 1 -- 172.21.15.12:0/2552392999 >> v2:172.21.15.12:3300/0 conn(0x7fcd140a0c90 msgr2=0x7fcd14135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:33:51.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:51.280+0000 7fcd13fff640 1 -- 172.21.15.12:0/2552392999 >> v2:172.21.15.12:3300/0 conn(0x7fcd140a0c90 msgr2=0x7fcd14135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:51.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:51.280+0000 7fcd13fff640 1 --2- 172.21.15.12:0/2552392999 >> v2:172.21.15.12:3300/0 conn(0x7fcd140a0c90 0x7fcd14135ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:51.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:51.280+0000 7fcd18ba4640 1 --2- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 0x7fcd140a1760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:33:51.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:51.280+0000 7fcd13fff640 1 --2- 172.21.15.12:0/2552392999 >> v2:172.21.15.12:3300/0 conn(0x7fcd140a0c90 0x7fcd14135ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:33:52.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:52.081+0000 7fcd13fff640 1 --2- 172.21.15.12:0/2552392999 >> v2:172.21.15.12:3300/0 conn(0x7fcd140a0c90 0x7fcd14135ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:52.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:52.081+0000 7fcd18ba4640 1 --2- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 0x7fcd140a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:52.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:52.081+0000 7fcd13fff640 10 monclient(hunting): get_auth_request con 0x7fcd140a0c90 auth_method 0 2024-09-11T01:33:52.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:52.081+0000 7fcd13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:52.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:52.081+0000 7fcd13fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:52.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:52.081+0000 7fcd13fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:52.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:52.081+0000 7fcd18ba4640 10 monclient(hunting): get_auth_request con 0x7fcd140a1380 auth_method 0 2024-09-11T01:33:52.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:52.081+0000 7fcd18ba4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:52.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:52.081+0000 7fcd18ba4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:52.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:52.081+0000 7fcd18ba4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:52.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:52.081+0000 7fcd18ba4640 1 -- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 msgr2=0x7fcd140a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:33:52.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:52.081+0000 7fcd18ba4640 1 -- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 msgr2=0x7fcd140a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:52.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:52.081+0000 7fcd18ba4640 1 --2- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 0x7fcd140a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:52.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:52.081+0000 7fcd13fff640 1 -- 172.21.15.12:0/2552392999 >> v2:172.21.15.12:3300/0 conn(0x7fcd140a0c90 msgr2=0x7fcd14135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:33:52.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:52.081+0000 7fcd13fff640 1 -- 172.21.15.12:0/2552392999 >> v2:172.21.15.12:3300/0 conn(0x7fcd140a0c90 msgr2=0x7fcd14135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:33:52.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:52.081+0000 7fcd13fff640 1 --2- 172.21.15.12:0/2552392999 >> v2:172.21.15.12:3300/0 conn(0x7fcd140a0c90 0x7fcd14135ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:52.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:52.081+0000 7fcd18ba4640 1 --2- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 0x7fcd140a1760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:33:52.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:52.081+0000 7fcd13fff640 1 --2- 172.21.15.12:0/2552392999 >> v2:172.21.15.12:3300/0 conn(0x7fcd140a0c90 0x7fcd14135ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:33:53.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.277+0000 7fcd137fe640 1 -- 172.21.15.12:0/2552392999 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2819768647 0 0) 0x7fcd140a2b40 con 0x7fcd14136020 2024-09-11T01:33:53.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.277+0000 7fcd137fe640 10 monclient(hunting): _negotiate 2024-09-11T01:33:53.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.277+0000 7fcd137fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:53.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.277+0000 7fcd137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:53.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.277+0000 7fcd137fe640 10 monclient(hunting): my global_id is 54105 2024-09-11T01:33:53.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.277+0000 7fcd137fe640 1 -- 172.21.15.12:0/2552392999 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fccfc003520 con 0x7fcd14136020 2024-09-11T01:33:53.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.279+0000 7fcd137fe640 1 -- 172.21.15.12:0/2552392999 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3381784484 0 0) 0x7fccfc003520 con 0x7fcd14136020 2024-09-11T01:33:53.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.279+0000 7fcd137fe640 1 -- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 msgr2=0x7fcd140a1760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:33:53.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.279+0000 7fcd137fe640 1 --2- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 0x7fcd140a1760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:53.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.279+0000 7fcd137fe640 1 -- 172.21.15.12:0/2552392999 >> v2:172.21.15.12:3300/0 conn(0x7fcd140a0c90 msgr2=0x7fcd14135ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:33:53.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.279+0000 7fcd137fe640 1 --2- 172.21.15.12:0/2552392999 >> v2:172.21.15.12:3300/0 conn(0x7fcd140a0c90 0x7fcd14135ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:53.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.279+0000 7fcd137fe640 10 monclient: _finish_hunting 0 2024-09-11T01:33:53.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.279+0000 7fcd137fe640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:33:53.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.279+0000 7fcd137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:53.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.279+0000 7fcd137fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:33:53.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.279+0000 7fcd137fe640 1 -- 172.21.15.12:0/2552392999 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd140a2510 con 0x7fcd14136020 2024-09-11T01:33:53.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.279+0000 7fcd137fe640 10 monclient: _finish_auth 0 2024-09-11T01:33:53.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.279+0000 7fcd137fe640 10 monclient: _check_auth_tickets 2024-09-11T01:33:53.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.279+0000 7fcd137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:53.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.279+0000 7fcd137fe640 1 -- 172.21.15.12:0/2552392999 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7fcd0c001d20 con 0x7fcd14136020 2024-09-11T01:33:53.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.279+0000 7fcd1ae08640 5 monclient: authenticate success, global_id 54105 2024-09-11T01:33:53.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd1ae08640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:33:53.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:53.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd137fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:33:53.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd137fe640 10 monclient: dump: 2024-09-11T01:33:53.285 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:53.285 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:53.285 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:53.285 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:53.285 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:53.285 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:53.285 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:53.286 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd137fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:33:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd137fe640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd137fe640 1 -- 172.21.15.12:0/2552392999 >> v1:172.21.15.12:6789/0 conn(0x7fcd14136020 legacy=0x7fcd1413ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd137fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:33:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd137fe640 1 --2- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 0x7fccfc003ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd137fe640 10 monclient(hunting): picked mon.c con 0x7fcd140a1380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd137fe640 1 --2- 172.21.15.12:0/2552392999 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fccfc004150 0x7fccfc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd137fe640 10 monclient(hunting): picked mon.b con 0x7fccfc004150 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:33:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd137fe640 1 --2- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccfc004e60 0x7fccfc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:53.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd137fe640 10 monclient(hunting): picked mon.a con 0x7fccfc004e60 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:33:53.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd137fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:53.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd137fe640 1 -- 172.21.15.12:0/2552392999 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcd1409f9a0 con 0x7fccfc004e60 2024-09-11T01:33:53.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd137fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:53.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd137fe640 1 -- 172.21.15.12:0/2552392999 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fcd140a2230 con 0x7fccfc004150 2024-09-11T01:33:53.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd137fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:53.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd137fe640 1 -- 172.21.15.12:0/2552392999 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fccfc00bf00 con 0x7fcd140a1380 2024-09-11T01:33:53.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd137fe640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:53.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd137fe640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:33:53.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd1ae08640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:33:53.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd18ba4640 1 --2- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 0x7fccfc003ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:53.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd18ba4640 10 monclient(hunting): get_auth_request con 0x7fcd140a1380 auth_method 0 2024-09-11T01:33:53.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd18ba4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:53.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd18ba4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:53.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd18ba4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:53.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd13fff640 1 --2- 172.21.15.12:0/2552392999 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fccfc004150 0x7fccfc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:53.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd13fff640 10 monclient(hunting): get_auth_request con 0x7fccfc004150 auth_method 0 2024-09-11T01:33:53.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:53.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd13fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:53.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd13fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:53.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd193a5640 1 --2- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccfc004e60 0x7fccfc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:53.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd193a5640 10 monclient(hunting): get_auth_request con 0x7fccfc004e60 auth_method 0 2024-09-11T01:33:53.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd193a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:53.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd193a5640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:53.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd193a5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:53.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd18ba4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:53.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd18ba4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd18ba4640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:33:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd13fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:33:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd193a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd193a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd193a5640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:33:53.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd18ba4640 10 monclient(hunting): handle_auth_done global_id 54105 payload 293 2024-09-11T01:33:53.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd18ba4640 1 -- 172.21.15.12:0/2552392999 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fccfc004150 msgr2=0x7fccfc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:53.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd18ba4640 1 --2- 172.21.15.12:0/2552392999 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fccfc004150 0x7fccfc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:53.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd18ba4640 1 -- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccfc004e60 msgr2=0x7fccfc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:53.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd18ba4640 1 --2- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccfc004e60 0x7fccfc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:53.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd18ba4640 10 monclient: _finish_hunting 0 2024-09-11T01:33:53.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd18ba4640 1 monclient: found mon.c 2024-09-11T01:33:53.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd18ba4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:53.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd18ba4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:53.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd18ba4640 1 -- 172.21.15.12:0/2552392999 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccfc003520 con 0x7fcd140a1380 2024-09-11T01:33:53.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd18ba4640 10 monclient: _finish_auth 0 2024-09-11T01:33:53.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd18ba4640 10 monclient: _check_auth_tickets 2024-09-11T01:33:53.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd18ba4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:53.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.280+0000 7fcd13fff640 1 --2- 172.21.15.12:0/2552392999 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fccfc004150 0x7fccfc004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:33:53.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.283+0000 7fcd18ba4640 1 -- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 msgr2=0x7fccfc003ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:53.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.283+0000 7fcd18ba4640 1 --2- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 0x7fccfc003ad0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fcd04011b90 tx=0x7fccfc005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e583b5f9be35042e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:53.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.284+0000 7fcd137fe640 1 -- 172.21.15.12:0/2552392999 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd04031070 con 0x7fcd140a1380 2024-09-11T01:33:53.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.284+0000 7fcd137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:53.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.284+0000 7fcd137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:53.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.284+0000 7fcd137fe640 10 monclient: dump: 2024-09-11T01:33:53.301 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:53.301 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:53.301 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:53.301 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:53.301 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:53.301 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:53.301 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:53.301 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:53.301 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:53.302 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:53.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.284+0000 7fcd1ae08640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:33:53.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.284+0000 7fcd137fe640 1 -- 172.21.15.12:0/2552392999 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd0402b7e0 con 0x7fcd140a1380 2024-09-11T01:33:53.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.284+0000 7fcd137fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:53.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.284+0000 7fcd1ae08640 10 monclient: get_monmap_and_config success 2024-09-11T01:33:53.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.284+0000 7fcd1ae08640 4 set_mon_vals no callback set 2024-09-11T01:33:53.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.284+0000 7fcd137fe640 1 -- 172.21.15.12:0/2552392999 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd04011790 con 0x7fcd140a1380 2024-09-11T01:33:53.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.284+0000 7fcd137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:53.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.284+0000 7fcd137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:53.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.284+0000 7fcd137fe640 10 monclient: dump: 2024-09-11T01:33:53.302 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:53.302 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:53.302 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:53.303 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:53.303 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:53.303 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:53.303 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:53.303 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:53.303 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:53.303 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:53.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.284+0000 7fcd1ae08640 10 monclient: shutdown 2024-09-11T01:33:53.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.284+0000 7fcd1ae08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:53.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.284+0000 7fcd1ae08640 1 -- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 msgr2=0x7fccfc003ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:53.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.284+0000 7fcd1ae08640 1 --2- 172.21.15.12:0/2552392999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd140a1380 0x7fccfc003ad0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fcd04011b90 tx=0x7fccfc005830 comp rx=0 tx=0).stop 2024-09-11T01:33:53.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.284+0000 7fcd193a5640 1 -- 172.21.15.12:0/2552392999 reap_dead start 2024-09-11T01:33:53.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.285+0000 7fcd1ae08640 1 -- 172.21.15.12:0/2552392999 shutdown_connections 2024-09-11T01:33:53.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.285+0000 7fcd1ae08640 1 -- 172.21.15.12:0/2552392999 >> 172.21.15.12:0/2552392999 conn(0x7fcd140a2d90 msgr2=0x7fcd140a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:53.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.285+0000 7fcd1ae08640 1 -- 172.21.15.12:0/2552392999 shutdown_connections 2024-09-11T01:33:53.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.285+0000 7fcd1ae08640 1 -- 172.21.15.12:0/2552392999 wait complete. 2024-09-11T01:33:53.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.285+0000 7fcd1ae08640 10 monclient: build_initial_monmap 2024-09-11T01:33:53.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.285+0000 7fcd1ae08640 10 monclient: monmap: 2024-09-11T01:33:53.305 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:53.305 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:33:53.305 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:33:53.305 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:33:53.305 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:53.305 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:53.305 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:33:53.305 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:33:53.305 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:33:53.305 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:53.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.286+0000 7fcd1ae08640 1 Processor -- start 2024-09-11T01:33:53.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.286+0000 7fcd1ae08640 1 -- start start 2024-09-11T01:33:53.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.286+0000 7fcd1ae08640 10 monclient: init 2024-09-11T01:33:53.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd1ae08640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:53.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd1ae08640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:33:53.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd1ae08640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd140a1380 0x7fcd14134c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:53.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd1ae08640 10 monclient(hunting): picked mon.noname-a con 0x7fcd140a1380 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:33:53.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd1ae08640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd141cabe0 0x7fcd14135170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:53.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd1ae08640 10 monclient(hunting): picked mon.noname-c con 0x7fcd141cabe0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:53.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd1ae08640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd14122880 0x7fcd14122d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:53.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd1ae08640 10 monclient(hunting): picked mon.noname-b con 0x7fcd14122880 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:33:53.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd1ae08640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:53.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd1ae08640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcd140a0510 con 0x7fcd140a1380 2024-09-11T01:33:53.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd1ae08640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:53.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd1ae08640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fcd140a0390 con 0x7fcd14122880 2024-09-11T01:33:53.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd1ae08640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:53.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd1ae08640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcd14136470 con 0x7fcd141cabe0 2024-09-11T01:33:53.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd1ae08640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:53.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd1ae08640 10 monclient(hunting): authenticate will time out at 1039.729286s 2024-09-11T01:33:53.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd18ba4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd140a1380 0x7fcd14134c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:53.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd13fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd141cabe0 0x7fcd14135170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:53.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd18ba4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd140a1380 0x7fcd14134c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35972/0 (socket says 172.21.15.12:35972) 2024-09-11T01:33:53.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd18ba4640 1 -- 172.21.15.12:0/2385695765 learned_addr learned my addr 172.21.15.12:0/2385695765 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:53.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd13fff640 10 monclient(hunting): get_auth_request con 0x7fcd141cabe0 auth_method 0 2024-09-11T01:33:53.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:53.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd13fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:53.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd13fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:53.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd18ba4640 10 monclient(hunting): get_auth_request con 0x7fcd140a1380 auth_method 0 2024-09-11T01:33:53.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd18ba4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:53.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd18ba4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:53.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd18ba4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:53.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd18ba4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:53.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd18ba4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:53.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd18ba4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:53.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd13fff640 1 -- 172.21.15.12:0/2385695765 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd141cabe0 msgr2=0x7fcd14135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:33:53.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd13fff640 1 -- 172.21.15.12:0/2385695765 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd141cabe0 msgr2=0x7fcd14135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:33:53.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd13fff640 1 --2- 172.21.15.12:0/2385695765 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd141cabe0 0x7fcd14135170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:53.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd193a5640 1 --2- 172.21.15.12:0/2385695765 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd14122880 0x7fcd14122d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:53.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd193a5640 10 monclient(hunting): get_auth_request con 0x7fcd14122880 auth_method 0 2024-09-11T01:33:53.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd13fff640 1 --2- 172.21.15.12:0/2385695765 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd141cabe0 0x7fcd14135170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:33:53.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd193a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:53.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd193a5640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:53.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd193a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:53.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd193a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:53.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd193a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:53.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd193a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:53.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd18ba4640 10 monclient(hunting): handle_auth_done global_id 54108 payload 824 2024-09-11T01:33:53.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd18ba4640 1 -- 172.21.15.12:0/2385695765 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd141cabe0 msgr2=0x7fcd14135170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:33:53.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd18ba4640 1 --2- 172.21.15.12:0/2385695765 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd141cabe0 0x7fcd14135170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:53.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd18ba4640 1 -- 172.21.15.12:0/2385695765 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd14122880 msgr2=0x7fcd14122d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:53.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd18ba4640 1 --2- 172.21.15.12:0/2385695765 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd14122880 0x7fcd14122d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:53.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd18ba4640 10 monclient: _finish_hunting 0 2024-09-11T01:33:53.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd18ba4640 1 monclient: found mon.noname-a 2024-09-11T01:33:53.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd18ba4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:53.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd18ba4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:33:53.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.287+0000 7fcd18ba4640 1 -- 172.21.15.12:0/2385695765 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd141242f0 con 0x7fcd140a1380 2024-09-11T01:33:53.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.290+0000 7fcd18ba4640 1 -- 172.21.15.12:0/2385695765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd140a1380 msgr2=0x7fcd14134c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:53.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.290+0000 7fcd18ba4640 1 --2- 172.21.15.12:0/2385695765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd140a1380 0x7fcd14134c30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcd04037040 tx=0x7fcd040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:53.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.290+0000 7fcd117fa640 1 -- 172.21.15.12:0/2385695765 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd04043020 con 0x7fcd140a1380 2024-09-11T01:33:53.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.290+0000 7fcd117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:53.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.290+0000 7fcd117fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:33:53.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.290+0000 7fcd117fa640 10 monclient: dump: 2024-09-11T01:33:53.310 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:53.310 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:53.310 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:53.310 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:53.311 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:53.311 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:53.311 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:53.311 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:53.311 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:53.311 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:53.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.290+0000 7fcd117fa640 10 monclient: _finish_auth 0 2024-09-11T01:33:53.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.290+0000 7fcd117fa640 10 monclient: _check_auth_tickets 2024-09-11T01:33:53.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.290+0000 7fcd117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:53.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.290+0000 7fcd1ae08640 5 monclient: authenticate success, global_id 54108 2024-09-11T01:33:53.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.290+0000 7fcd117fa640 1 -- 172.21.15.12:0/2385695765 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd0403ea60 con 0x7fcd140a1380 2024-09-11T01:33:53.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.290+0000 7fcd1ae08640 10 monclient: _renew_subs 2024-09-11T01:33:53.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.290+0000 7fcd1ae08640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:33:53.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.290+0000 7fcd1ae08640 1 -- 172.21.15.12:0/2385695765 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd140a5610 con 0x7fcd140a1380 2024-09-11T01:33:53.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.290+0000 7fcd117fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:53.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.290+0000 7fcd117fa640 1 -- 172.21.15.12:0/2385695765 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd04011790 con 0x7fcd140a1380 2024-09-11T01:33:53.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.290+0000 7fcd117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:53.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.290+0000 7fcd117fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:33:53.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.290+0000 7fcd117fa640 10 monclient: dump: 2024-09-11T01:33:53.312 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:53.312 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:53.312 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:53.312 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:53.313 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:53.313 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:53.313 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:53.313 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:53.313 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:53.313 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:53.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.290+0000 7fcd1ae08640 10 monclient: _renew_subs 2024-09-11T01:33:53.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.290+0000 7fcd1ae08640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:33:53.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.290+0000 7fcd1ae08640 1 -- 172.21.15.12:0/2385695765 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd14126370 con 0x7fcd140a1380 2024-09-11T01:33:53.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.290+0000 7fcd12ffd640 4 set_mon_vals no callback set 2024-09-11T01:33:53.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.291+0000 7fcd117fa640 1 -- 172.21.15.12:0/2385695765 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fcd04035400 con 0x7fcd140a1380 2024-09-11T01:33:53.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.291+0000 7fcd1ae08640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:53.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.291+0000 7fcd1ae08640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:53.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.291+0000 7fcd1ae08640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:33:53.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.291+0000 7fcd1ae08640 1 -- 172.21.15.12:0/2385695765 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd141367f0 con 0x7fcd140a1380 2024-09-11T01:33:53.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.291+0000 7fcd117fa640 1 --2- 172.21.15.12:0/2385695765 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcd0003e260 0x7fcd00040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:53.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.291+0000 7fcd117fa640 1 -- 172.21.15.12:0/2385695765 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 4799+0+0 (secure 0 0 0) 0x7fcd04074090 con 0x7fcd140a1380 2024-09-11T01:33:53.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.292+0000 7fcd13fff640 1 --2- 172.21.15.12:0/2385695765 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcd0003e260 0x7fcd00040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:53.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.292+0000 7fcd13fff640 10 monclient: get_auth_request con 0x7fcd0003e260 auth_method 0 2024-09-11T01:33:53.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.294+0000 7fcd13fff640 1 --2- 172.21.15.12:0/2385695765 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcd0003e260 0x7fcd00040720 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcd08006fd0 tx=0x7fcd08008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:53.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.296+0000 7fcd117fa640 1 -- 172.21.15.12:0/2385695765 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fcd0404d030 con 0x7fcd140a1380 2024-09-11T01:33:53.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.296+0000 7fcd117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:53.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.296+0000 7fcd117fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:33:53.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.557+0000 7fcd1ae08640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:33:53.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.557+0000 7fcd1ae08640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:33:53.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.557+0000 7fcd1ae08640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:33:53.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.557+0000 7fcd1ae08640 1 -- 172.21.15.12:0/2385695765 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcd14124e50 con 0x7fcd140a1380 2024-09-11T01:33:53.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.558+0000 7fcd117fa640 1 -- 172.21.15.12:0/2385695765 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7fcd140a5610 con 0x7fcd140a1380 2024-09-11T01:33:53.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.558+0000 7fcd117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:33:53.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.558+0000 7fcd117fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:33:53.560 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":52,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:33:53.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.560+0000 7fcd1ae08640 1 -- 172.21.15.12:0/2385695765 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcd0003e260 msgr2=0x7fcd00040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:53.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.560+0000 7fcd1ae08640 1 --2- 172.21.15.12:0/2385695765 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcd0003e260 0x7fcd00040720 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcd08006fd0 tx=0x7fcd08008040 comp rx=0 tx=0).stop 2024-09-11T01:33:53.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.560+0000 7fcd1ae08640 10 monclient: shutdown 2024-09-11T01:33:53.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.560+0000 7fcd1ae08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:53.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.560+0000 7fcd1ae08640 1 -- 172.21.15.12:0/2385695765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd140a1380 msgr2=0x7fcd14134c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:53.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.560+0000 7fcd1ae08640 1 --2- 172.21.15.12:0/2385695765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd140a1380 0x7fcd14134c30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcd04037040 tx=0x7fcd040046c0 comp rx=0 tx=0).stop 2024-09-11T01:33:53.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.560+0000 7fcd1ae08640 1 -- 172.21.15.12:0/2385695765 shutdown_connections 2024-09-11T01:33:53.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.560+0000 7fcd1ae08640 1 --2- 172.21.15.12:0/2385695765 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd14122880 0x7fcd14122d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:53.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.560+0000 7fcd1ae08640 1 --2- 172.21.15.12:0/2385695765 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd141cabe0 0x7fcd14135170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:53.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.560+0000 7fcd1ae08640 1 --2- 172.21.15.12:0/2385695765 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcd0003e260 0x7fcd00040720 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:53.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.560+0000 7fcd1ae08640 1 --2- 172.21.15.12:0/2385695765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd140a1380 0x7fcd14134c30 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:53.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.560+0000 7fcd1ae08640 1 -- 172.21.15.12:0/2385695765 >> 172.21.15.12:0/2385695765 conn(0x7fcd140a2d90 msgr2=0x7fcd14070c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:53.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.560+0000 7fcd1ae08640 1 -- 172.21.15.12:0/2385695765 shutdown_connections 2024-09-11T01:33:53.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:53.560+0000 7fcd1ae08640 1 -- 172.21.15.12:0/2385695765 wait complete. 2024-09-11T01:33:54.000 INFO:tasks.rados.rados.0.smithi071.stdout:731: done (1 left) 2024-09-11T01:33:54.000 INFO:tasks.rados.rados.0.smithi071.stdout:732: delete oid 41 current snap is 73 2024-09-11T01:33:54.000 INFO:tasks.rados.rados.0.smithi071.stdout:730: finishing write tid 1 to smithi07134777-4 2024-09-11T01:33:54.000 INFO:tasks.rados.rados.0.smithi071.stdout:730: finishing write tid 2 to smithi07134777-4 2024-09-11T01:33:54.000 INFO:tasks.rados.rados.0.smithi071.stdout:730: finishing write tid 3 to smithi07134777-4 2024-09-11T01:33:54.000 INFO:tasks.rados.rados.0.smithi071.stdout:730: finishing write tid 4 to smithi07134777-4 2024-09-11T01:33:54.000 INFO:tasks.rados.rados.0.smithi071.stdout:730: oid 4 updating version 0 to 236 2024-09-11T01:33:54.000 INFO:tasks.rados.rados.0.smithi071.stdout:730: oid 4 updating version 236 to 237 2024-09-11T01:33:54.000 INFO:tasks.rados.rados.0.smithi071.stdout:730: oid 4 updating version 237 to 238 2024-09-11T01:33:54.000 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 238 (ObjNum 280 snap 73 seq_num 280) dirty exists 2024-09-11T01:33:54.000 INFO:tasks.rados.rados.0.smithi071.stdout:730: left oid 4 (ObjNum 280 snap 73 seq_num 280) 2024-09-11T01:33:54.011 INFO:tasks.rados.rados.0.smithi071.stdout:730: done (1 left) 2024-09-11T01:33:54.011 INFO:tasks.rados.rados.0.smithi071.stdout:732: done (0 left) 2024-09-11T01:33:54.011 INFO:tasks.rados.rados.0.smithi071.stdout:733: read oid 47 snap -1 2024-09-11T01:33:54.012 INFO:tasks.rados.rados.0.smithi071.stdout:733: expect (ObjNum 243 snap 61 seq_num 243) 2024-09-11T01:33:54.012 INFO:tasks.rados.rados.0.smithi071.stdout:734: copy_from oid 26 from oid 36 current snap is 73 2024-09-11T01:33:54.012 INFO:tasks.rados.rados.0.smithi071.stdout:735: rollback oid 19 current snap is 73 2024-09-11T01:33:54.012 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 19 to 62 2024-09-11T01:33:54.012 INFO:tasks.rados.rados.0.smithi071.stdout:736: write oid 50 current snap is 73 2024-09-11T01:33:54.012 INFO:tasks.rados.rados.0.smithi071.stdout:736: seq_num 282 ranges {739574=725279,1968628=721093} 2024-09-11T01:33:54.020 INFO:tasks.rados.rados.0.smithi071.stdout:736: writing smithi07134777-50 from 739574 to 1464853 tid 1 2024-09-11T01:33:54.027 INFO:tasks.rados.rados.0.smithi071.stdout:736: writing smithi07134777-50 from 1968628 to 2689721 tid 2 2024-09-11T01:33:54.027 INFO:tasks.rados.rados.0.smithi071.stdout:737: copy_from oid 5 from oid 37 current snap is 73 2024-09-11T01:33:54.027 INFO:tasks.rados.rados.0.smithi071.stdout:735: finishing rollback tid 0 to smithi07134777-19 2024-09-11T01:33:54.027 INFO:tasks.rados.rados.0.smithi071.stdout:735: finishing rollback tid 1 to smithi07134777-19 2024-09-11T01:33:54.028 INFO:tasks.rados.rados.0.smithi071.stdout:735: finishing rollback tid 2 to smithi07134777-19 2024-09-11T01:33:54.028 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 126 (ObjNum 246 snap 62 seq_num 246) dirty exists 2024-09-11T01:33:54.028 INFO:tasks.rados.rados.0.smithi071.stdout:735: done (4 left) 2024-09-11T01:33:54.028 INFO:tasks.rados.rados.0.smithi071.stdout:738: write (excl) oid 2 current snap is 73 2024-09-11T01:33:54.028 INFO:tasks.rados.rados.0.smithi071.stdout:738: seq_num 283 ranges {756381=400528,1910757=464751} 2024-09-11T01:33:54.034 INFO:tasks.rados.rados.0.smithi071.stdout:738: writing smithi07134777-2 from 756381 to 1156909 tid 1 2024-09-11T01:33:54.041 INFO:tasks.rados.rados.0.smithi071.stdout:738: writing smithi07134777-2 from 1910757 to 2375508 tid 2 2024-09-11T01:33:54.041 INFO:tasks.rados.rados.0.smithi071.stdout:739: read oid 24 snap -1 2024-09-11T01:33:54.041 INFO:tasks.rados.rados.0.smithi071.stdout:739: expect (ObjNum 276 snap 73 seq_num 276) 2024-09-11T01:33:54.041 INFO:tasks.rados.rados.0.smithi071.stdout:740: rollback oid 31 current snap is 73 2024-09-11T01:33:54.041 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 31 to 62 2024-09-11T01:33:54.042 INFO:tasks.rados.rados.0.smithi071.stdout:734: finishing copy_from racing read to smithi07134777-26 2024-09-11T01:33:54.042 INFO:tasks.rados.rados.0.smithi071.stdout:734: finishing copy_from to smithi07134777-26 2024-09-11T01:33:54.042 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 117 (ObjNum 187 snap 41 seq_num 187) dirty exists 2024-09-11T01:33:54.042 INFO:tasks.rados.rados.0.smithi071.stdout:736: finishing write tid 1 to smithi07134777-50 2024-09-11T01:33:54.042 INFO:tasks.rados.rados.0.smithi071.stdout:733: expect (ObjNum 243 snap 61 seq_num 243) 2024-09-11T01:33:54.042 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:33:54.139 INFO:tasks.rados.rados.0.smithi071.stdout:736: finishing write tid 2 to smithi07134777-50 2024-09-11T01:33:54.139 INFO:tasks.rados.rados.0.smithi071.stdout:736: finishing write tid 3 to smithi07134777-50 2024-09-11T01:33:54.139 INFO:tasks.rados.rados.0.smithi071.stdout:736: finishing write tid 4 to smithi07134777-50 2024-09-11T01:33:54.139 INFO:tasks.rados.rados.0.smithi071.stdout:736: oid 50 updating version 0 to 350 2024-09-11T01:33:54.139 INFO:tasks.rados.rados.0.smithi071.stdout:736: oid 50 updating version 350 to 351 2024-09-11T01:33:54.139 INFO:tasks.rados.rados.0.smithi071.stdout:736: oid 50 updating version 351 to 352 2024-09-11T01:33:54.139 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 352 (ObjNum 282 snap 73 seq_num 282) dirty exists 2024-09-11T01:33:54.139 INFO:tasks.rados.rados.0.smithi071.stdout:736: left oid 50 (ObjNum 282 snap 73 seq_num 282) 2024-09-11T01:33:54.139 INFO:tasks.rados.rados.0.smithi071.stdout:738: finishing write tid 1 to smithi07134777-2 2024-09-11T01:33:54.139 INFO:tasks.rados.rados.0.smithi071.stdout:733: done (6 left) 2024-09-11T01:33:54.139 INFO:tasks.rados.rados.0.smithi071.stdout:734: done (5 left) 2024-09-11T01:33:54.139 INFO:tasks.rados.rados.0.smithi071.stdout:736: done (4 left) 2024-09-11T01:33:54.140 INFO:tasks.rados.rados.0.smithi071.stdout:741: read oid 41 snap -1 2024-09-11T01:33:54.140 INFO:tasks.rados.rados.0.smithi071.stdout:741: expect deleted 2024-09-11T01:33:54.140 INFO:tasks.rados.rados.0.smithi071.stdout:742: write oid 45 current snap is 73 2024-09-11T01:33:54.140 INFO:tasks.rados.rados.0.smithi071.stdout:737: finishing copy_from racing read to smithi07134777-5 2024-09-11T01:33:54.140 INFO:tasks.rados.rados.0.smithi071.stdout:737: finishing copy_from to smithi07134777-5 2024-09-11T01:33:54.140 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 147 (ObjNum 170 snap 37 seq_num 170) dirty exists 2024-09-11T01:33:54.140 INFO:tasks.rados.rados.0.smithi071.stdout:738: finishing write tid 2 to smithi07134777-2 2024-09-11T01:33:54.140 INFO:tasks.rados.rados.0.smithi071.stdout:740: finishing rollback tid 0 to smithi07134777-31 2024-09-11T01:33:54.140 INFO:tasks.rados.rados.0.smithi071.stdout:740: finishing rollback tid 1 to smithi07134777-31 2024-09-11T01:33:54.140 INFO:tasks.rados.rados.0.smithi071.stdout:742: seq_num 284 ranges {773188=475777,1969239=724762} 2024-09-11T01:33:54.145 INFO:tasks.rados.rados.0.smithi071.stdout:742: writing smithi07134777-45 from 773188 to 1248965 tid 1 2024-09-11T01:33:54.154 INFO:tasks.rados.rados.0.smithi071.stdout:742: writing smithi07134777-45 from 1969239 to 2694001 tid 2 2024-09-11T01:33:54.154 INFO:tasks.rados.rados.0.smithi071.stdout:737: done (5 left) 2024-09-11T01:33:54.154 INFO:tasks.rados.rados.0.smithi071.stdout:743: rollback oid 33 current snap is 73 2024-09-11T01:33:54.154 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 33 to 62 2024-09-11T01:33:54.155 INFO:tasks.rados.rados.0.smithi071.stdout:744: rollback oid 8 current snap is 73 2024-09-11T01:33:54.155 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 8 to 62 2024-09-11T01:33:54.155 INFO:tasks.rados.rados.0.smithi071.stdout:738: finishing write tid 3 to smithi07134777-2 2024-09-11T01:33:54.155 INFO:tasks.rados.rados.0.smithi071.stdout:740: finishing rollback tid 2 to smithi07134777-31 2024-09-11T01:33:54.155 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 296 (ObjNum 201 snap 46 seq_num 201) dirty exists 2024-09-11T01:33:54.155 INFO:tasks.rados.rados.0.smithi071.stdout:738: finishing write tid 4 to smithi07134777-2 2024-09-11T01:33:54.155 INFO:tasks.rados.rados.0.smithi071.stdout:738: oid 2 updating version 0 to 132 2024-09-11T01:33:54.155 INFO:tasks.rados.rados.0.smithi071.stdout:738: oid 2 version 132 is already newer than 131 2024-09-11T01:33:54.155 INFO:tasks.rados.rados.0.smithi071.stdout:738: oid 2 version 132 is already newer than 130 2024-09-11T01:33:54.155 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 132 (ObjNum 283 snap 73 seq_num 283) dirty exists 2024-09-11T01:33:54.155 INFO:tasks.rados.rados.0.smithi071.stdout:738: left oid 2 (ObjNum 283 snap 73 seq_num 283) 2024-09-11T01:33:54.155 INFO:tasks.rados.rados.0.smithi071.stdout:738: done (6 left) 2024-09-11T01:33:54.155 INFO:tasks.rados.rados.0.smithi071.stdout:740: done (5 left) 2024-09-11T01:33:54.155 INFO:tasks.rados.rados.0.smithi071.stdout:745: read oid 34 snap -1 2024-09-11T01:33:54.155 INFO:tasks.rados.rados.0.smithi071.stdout:745: expect deleted 2024-09-11T01:33:54.155 INFO:tasks.rados.rados.0.smithi071.stdout:746: read oid 38 snap 62 2024-09-11T01:33:54.156 INFO:tasks.rados.rados.0.smithi071.stdout:746: expect deleted 2024-09-11T01:33:54.156 INFO:tasks.rados.rados.0.smithi071.stdout:747: copy_from oid 26 from oid 2 current snap is 73 2024-09-11T01:33:54.156 INFO:tasks.rados.rados.0.smithi071.stdout:739: expect (ObjNum 276 snap 73 seq_num 276) 2024-09-11T01:33:54.156 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:33:54.237 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:33:54.279 INFO:tasks.rados.rados.0.smithi071.stdout:742: finishing write tid 1 to smithi07134777-45 2024-09-11T01:33:54.279 INFO:tasks.rados.rados.0.smithi071.stdout:743: finishing rollback tid 1 to smithi07134777-33 2024-09-11T01:33:54.279 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 0 (ObjNum 129 snap 0 seq_num 2370823336) dirty dne 2024-09-11T01:33:54.280 INFO:tasks.rados.rados.0.smithi071.stdout:742: finishing write tid 2 to smithi07134777-45 2024-09-11T01:33:54.280 INFO:tasks.rados.rados.0.smithi071.stdout:742: finishing write tid 3 to smithi07134777-45 2024-09-11T01:33:54.280 INFO:tasks.rados.rados.0.smithi071.stdout:744: finishing rollback tid 0 to smithi07134777-8 2024-09-11T01:33:54.280 INFO:tasks.rados.rados.0.smithi071.stdout:742: finishing write tid 4 to smithi07134777-45 2024-09-11T01:33:54.280 INFO:tasks.rados.rados.0.smithi071.stdout:742: oid 45 updating version 0 to 242 2024-09-11T01:33:54.280 INFO:tasks.rados.rados.0.smithi071.stdout:742: oid 45 updating version 242 to 244 2024-09-11T01:33:54.280 INFO:tasks.rados.rados.0.smithi071.stdout:742: oid 45 version 244 is already newer than 243 2024-09-11T01:33:54.280 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 244 (ObjNum 284 snap 73 seq_num 284) dirty exists 2024-09-11T01:33:54.280 INFO:tasks.rados.rados.0.smithi071.stdout:742: left oid 45 (ObjNum 284 snap 73 seq_num 284) 2024-09-11T01:33:54.280 INFO:tasks.rados.rados.0.smithi071.stdout:744: finishing rollback tid 1 to smithi07134777-8 2024-09-11T01:33:54.280 INFO:tasks.rados.rados.0.smithi071.stdout:744: finishing rollback tid 2 to smithi07134777-8 2024-09-11T01:33:54.280 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 129 (ObjNum 226 snap 55 seq_num 226) dirty exists 2024-09-11T01:33:54.280 INFO:tasks.rados.rados.0.smithi071.stdout:747: finishing copy_from racing read to smithi07134777-26 2024-09-11T01:33:54.281 INFO:tasks.rados.rados.0.smithi071.stdout:747: finishing copy_from to smithi07134777-26 2024-09-11T01:33:54.281 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 132 (ObjNum 283 snap 73 seq_num 283) dirty exists 2024-09-11T01:33:54.281 INFO:tasks.rados.rados.0.smithi071.stdout:739: done (7 left) 2024-09-11T01:33:54.281 INFO:tasks.rados.rados.0.smithi071.stdout:741: done (6 left) 2024-09-11T01:33:54.281 INFO:tasks.rados.rados.0.smithi071.stdout:742: done (5 left) 2024-09-11T01:33:54.281 INFO:tasks.rados.rados.0.smithi071.stdout:743: done (4 left) 2024-09-11T01:33:54.281 INFO:tasks.rados.rados.0.smithi071.stdout:744: done (3 left) 2024-09-11T01:33:54.281 INFO:tasks.rados.rados.0.smithi071.stdout:745: done (2 left) 2024-09-11T01:33:54.281 INFO:tasks.rados.rados.0.smithi071.stdout:746: done (1 left) 2024-09-11T01:33:54.281 INFO:tasks.rados.rados.0.smithi071.stdout:747: done (0 left) 2024-09-11T01:33:54.281 INFO:tasks.rados.rados.0.smithi071.stdout:748: read oid 43 snap 62 2024-09-11T01:33:54.281 INFO:tasks.rados.rados.0.smithi071.stdout:748: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:33:54.281 INFO:tasks.rados.rados.0.smithi071.stdout:749: snap_create 2024-09-11T01:33:54.297 INFO:tasks.rados.rados.0.smithi071.stdout:748: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:33:54.297 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:33:54.325 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:33:55.531 INFO:tasks.rados.rados.0.smithi071.stdout:748: done (1 left) 2024-09-11T01:33:55.531 INFO:tasks.rados.rados.0.smithi071.stdout:749: done (0 left) 2024-09-11T01:33:55.531 INFO:tasks.rados.rados.0.smithi071.stdout:750: write (excl) oid 42 current snap is 74 2024-09-11T01:33:55.531 INFO:tasks.rados.rados.0.smithi071.stdout:750: seq_num 285 ranges {789995=551026,2027721=468420} 2024-09-11T01:33:55.538 INFO:tasks.rados.rados.0.smithi071.stdout:750: writing smithi07134777-42 from 789995 to 1341021 tid 1 2024-09-11T01:33:55.544 INFO:tasks.rados.rados.0.smithi071.stdout:750: writing smithi07134777-42 from 2027721 to 2496141 tid 2 2024-09-11T01:33:55.544 INFO:tasks.rados.rados.0.smithi071.stdout:751: snap_remove snap 73 2024-09-11T01:33:56.533 INFO:tasks.rados.rados.0.smithi071.stdout:751: done (1 left) 2024-09-11T01:33:56.534 INFO:tasks.rados.rados.0.smithi071.stdout:752: snap_remove snap 62 2024-09-11T01:33:56.573 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:33:56.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.682+0000 7f131cf05640 10 monclient: get_monmap_and_config 2024-09-11T01:33:56.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.682+0000 7f131cf05640 10 monclient: build_initial_monmap 2024-09-11T01:33:56.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.682+0000 7f131cf05640 10 monclient: monmap: 2024-09-11T01:33:56.684 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:56.684 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:56.684 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:33:56.683585+0000 2024-09-11T01:33:56.685 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:33:56.683585+0000 2024-09-11T01:33:56.685 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:56.685 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:56.685 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:33:56.685 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:33:56.685 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:33:56.685 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:33:56.685 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:33:56.685 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.683+0000 7f131cf05640 1 Processor -- start 2024-09-11T01:33:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.683+0000 7f131cf05640 1 -- start start 2024-09-11T01:33:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.683+0000 7f131cf05640 10 monclient: init 2024-09-11T01:33:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f131cf05640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f131cf05640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-11T01:33:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f131cf05640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f13180a2e90 addr v1:172.21.15.71:6789/0 2024-09-11T01:33:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f131cf05640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13180a3270 0x7f13180a3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f131cf05640 10 monclient(hunting): picked mon.noname-b con 0x7f13180a3270 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f131cf05640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f131813d490 0x7f131813f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f131cf05640 10 monclient(hunting): picked mon.noname-c con 0x7f131813d490 addr v2:172.21.15.71:3300/0 2024-09-11T01:33:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f131cf05640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13180a46c0 con 0x7f13180a2e90 2024-09-11T01:33:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f131cf05640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f131cf05640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f13180a4900 con 0x7f131813d490 2024-09-11T01:33:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f131cf05640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f131cf05640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f131806ee50 con 0x7f13180a3270 2024-09-11T01:33:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f131cf05640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f131cf05640 10 monclient(hunting): authenticate will time out at 1043.126256s 2024-09-11T01:33:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f130ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13180a3270 0x7f13180a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f130ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13180a3270 0x7f13180a3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:37198/0 (socket says 172.21.15.12:37198) 2024-09-11T01:33:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f130ffff640 1 -- 172.21.15.12:0/2081357931 learned_addr learned my addr 172.21.15.12:0/2081357931 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f130ffff640 10 monclient(hunting): get_auth_request con 0x7f13180a3270 auth_method 0 2024-09-11T01:33:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f130ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f130ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f130ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f130ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f130ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f1316d9d640 1 --2- 172.21.15.12:0/2081357931 >> v2:172.21.15.71:3300/0 conn(0x7f131813d490 0x7f131813f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f130ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f1316d9d640 10 monclient(hunting): get_auth_request con 0x7f131813d490 auth_method 0 2024-09-11T01:33:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f1316d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f1316d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f1316d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f1316d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f1316d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f1316d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f130ffff640 10 monclient(hunting): handle_auth_done global_id 44195 payload 274 2024-09-11T01:33:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f130ffff640 1 -- 172.21.15.12:0/2081357931 >> v2:172.21.15.71:3300/0 conn(0x7f131813d490 msgr2=0x7f131813f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:56.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f130ffff640 1 --2- 172.21.15.12:0/2081357931 >> v2:172.21.15.71:3300/0 conn(0x7f131813d490 0x7f131813f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:56.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f130ffff640 1 -- 172.21.15.12:0/2081357931 >> v1:172.21.15.71:6789/0 conn(0x7f13180a2e90 legacy=0x7f131806dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:56.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f130ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:33:56.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f130ffff640 1 monclient: found mon.noname-b 2024-09-11T01:33:56.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f130ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:56.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f130ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:33:56.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.684+0000 7f130ffff640 1 -- 172.21.15.12:0/2081357931 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f131806eb70 con 0x7f13180a3270 2024-09-11T01:33:56.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.687+0000 7f130ffff640 1 -- 172.21.15.12:0/2081357931 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13180a3270 msgr2=0x7f13180a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:56.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.687+0000 7f130ffff640 1 --2- 172.21.15.12:0/2081357931 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13180a3270 0x7f13180a3680 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f13180a4a80 tx=0x7f130002a050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bb0aa30a82a0c49 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:56.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.687+0000 7f1315d9b640 1 -- 172.21.15.12:0/2081357931 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1300037070 con 0x7f13180a3270 2024-09-11T01:33:56.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.687+0000 7f1315d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:56.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.687+0000 7f1315d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:33:56.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.687+0000 7f1315d9b640 10 monclient: dump: 2024-09-11T01:33:56.696 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:56.696 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:56.696 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:56.697 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:56.697 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:56.697 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:56.697 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:56.697 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:56.697 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:56.697 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:56.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.687+0000 7f1315d9b640 10 monclient: _finish_auth 0 2024-09-11T01:33:56.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.687+0000 7f1315d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:33:56.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.687+0000 7f1315d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:56.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.687+0000 7f1315d9b640 1 -- 172.21.15.12:0/2081357931 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1300032440 con 0x7f13180a3270 2024-09-11T01:33:56.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.687+0000 7f131cf05640 5 monclient: authenticate success, global_id 44195 2024-09-11T01:33:56.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.687+0000 7f131cf05640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:33:56.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.687+0000 7f1315d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:56.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.687+0000 7f1315d9b640 1 -- 172.21.15.12:0/2081357931 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1300032810 con 0x7f13180a3270 2024-09-11T01:33:56.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.687+0000 7f1315d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:56.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.687+0000 7f1315d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:56.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.687+0000 7f1315d9b640 10 monclient: dump: 2024-09-11T01:33:56.698 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:56.698 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:56.698 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:56.698 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:56.698 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:56.698 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:56.698 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:56.698 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:56.698 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:56.699 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:56.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.687+0000 7f131cf05640 10 monclient: get_monmap_and_config success 2024-09-11T01:33:56.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.687+0000 7f131cf05640 4 set_mon_vals no callback set 2024-09-11T01:33:56.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.688+0000 7f131cf05640 10 monclient: shutdown 2024-09-11T01:33:56.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.688+0000 7f131cf05640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:56.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.688+0000 7f131cf05640 1 -- 172.21.15.12:0/2081357931 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13180a3270 msgr2=0x7f13180a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:56.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.688+0000 7f131cf05640 1 --2- 172.21.15.12:0/2081357931 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13180a3270 0x7f13180a3680 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f13180a4a80 tx=0x7f130002a050 comp rx=0 tx=0).stop 2024-09-11T01:33:56.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.688+0000 7f131cf05640 1 -- 172.21.15.12:0/2081357931 shutdown_connections 2024-09-11T01:33:56.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.688+0000 7f131cf05640 1 --2- 172.21.15.12:0/2081357931 >> v2:172.21.15.71:3300/0 conn(0x7f131813d490 0x7f131813f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:56.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.688+0000 7f131cf05640 1 --2- 172.21.15.12:0/2081357931 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13180a3270 0x7f13180a3680 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:56.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.688+0000 7f131cf05640 1 -- 172.21.15.12:0/2081357931 >> 172.21.15.12:0/2081357931 conn(0x7f13180a8490 msgr2=0x7f13180a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:56.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.688+0000 7f131cf05640 1 -- 172.21.15.12:0/2081357931 shutdown_connections 2024-09-11T01:33:56.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.688+0000 7f131cf05640 1 -- 172.21.15.12:0/2081357931 wait complete. 2024-09-11T01:33:56.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.689+0000 7f131cf05640 10 monclient: build_initial_monmap 2024-09-11T01:33:56.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.689+0000 7f131cf05640 10 monclient: monmap: 2024-09-11T01:33:56.700 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:56.700 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:33:56.700 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:33:56.700 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:33:56.700 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:56.700 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:56.700 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:33:56.700 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:33:56.700 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:33:56.700 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:56.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.689+0000 7f131cf05640 1 Processor -- start 2024-09-11T01:33:56.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.689+0000 7f131cf05640 1 -- start start 2024-09-11T01:33:56.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.689+0000 7f131cf05640 10 monclient: init 2024-09-11T01:33:56.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131cf05640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:56.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131cf05640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:33:56.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131cf05640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13180a2e90 0x7f13180a1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:56.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131cf05640 10 monclient(hunting): picked mon.noname-a con 0x7f13180a2e90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:33:56.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131cf05640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13180a3270 0x7f131809c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:56.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131cf05640 10 monclient(hunting): picked mon.noname-b con 0x7f13180a3270 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:33:56.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131cf05640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f131813d490 0x7f131809cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:56.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131cf05640 10 monclient(hunting): picked mon.noname-c con 0x7f131813d490 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:56.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131cf05640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:56.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131cf05640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f131806fd20 con 0x7f13180a2e90 2024-09-11T01:33:56.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131cf05640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:56.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131cf05640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f131806fba0 con 0x7f13180a3270 2024-09-11T01:33:56.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131cf05640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:56.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131cf05640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f131813fdc0 con 0x7f131813d490 2024-09-11T01:33:56.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131cf05640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:56.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131cf05640 10 monclient(hunting): authenticate will time out at 1043.132201s 2024-09-11T01:33:56.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13180a2e90 0x7f13180a1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:56.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13180a2e90 0x7f13180a1cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42196/0 (socket says 172.21.15.12:42196) 2024-09-11T01:33:56.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131659c640 1 -- 172.21.15.12:0/1026428961 learned_addr learned my addr 172.21.15.12:0/1026428961 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:56.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131659c640 10 monclient(hunting): get_auth_request con 0x7f13180a2e90 auth_method 0 2024-09-11T01:33:56.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:56.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:56.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:56.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f1316d9d640 1 --2- 172.21.15.12:0/1026428961 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f131813d490 0x7f131809cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:56.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f1316d9d640 10 monclient(hunting): get_auth_request con 0x7f131813d490 auth_method 0 2024-09-11T01:33:56.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f1316d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:56.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f1316d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:56.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f1316d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:56.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:56.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:56.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:56.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f1316d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:56.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f1316d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:56.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f1316d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:56.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f130ffff640 1 --2- 172.21.15.12:0/1026428961 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13180a3270 0x7f131809c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:56.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131659c640 10 monclient(hunting): handle_auth_done global_id 54111 payload 824 2024-09-11T01:33:56.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131659c640 1 -- 172.21.15.12:0/1026428961 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f131813d490 msgr2=0x7f131809cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:56.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131659c640 1 --2- 172.21.15.12:0/1026428961 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f131813d490 0x7f131809cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:56.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131659c640 1 -- 172.21.15.12:0/1026428961 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13180a3270 msgr2=0x7f131809c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:56.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131659c640 1 --2- 172.21.15.12:0/1026428961 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13180a3270 0x7f131809c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:56.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131659c640 10 monclient: _finish_hunting 0 2024-09-11T01:33:56.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131659c640 1 monclient: found mon.noname-a 2024-09-11T01:33:56.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:56.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:33:56.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f131659c640 1 -- 172.21.15.12:0/1026428961 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f131809f180 con 0x7f13180a2e90 2024-09-11T01:33:56.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f130ffff640 10 monclient: get_auth_request con 0x7f13180a3270 auth_method 0 2024-09-11T01:33:56.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f130ffff640 1 --2- 172.21.15.12:0/1026428961 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13180a3270 0x7f131809c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:33:56.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.690+0000 7f1316d9d640 1 --2- 172.21.15.12:0/1026428961 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f131813d490 0x7f131809cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:33:56.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.692+0000 7f131659c640 1 -- 172.21.15.12:0/1026428961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13180a2e90 msgr2=0x7f13180a1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:56.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.692+0000 7f131659c640 1 --2- 172.21.15.12:0/1026428961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13180a2e90 0x7f13180a1cd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f130800c460 tx=0x7f130800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:56.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.693+0000 7f130effd640 1 -- 172.21.15.12:0/1026428961 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1308016020 con 0x7f13180a2e90 2024-09-11T01:33:56.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.693+0000 7f130effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:56.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.693+0000 7f130effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:33:56.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.693+0000 7f130effd640 10 monclient: dump: 2024-09-11T01:33:56.705 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:56.705 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:56.705 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:56.705 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:56.705 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:56.705 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:56.705 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:56.705 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:56.705 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:56.707 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:56.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.693+0000 7f130effd640 10 monclient: _finish_auth 0 2024-09-11T01:33:56.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.693+0000 7f130effd640 10 monclient: _check_auth_tickets 2024-09-11T01:33:56.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.693+0000 7f130effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:56.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.693+0000 7f131cf05640 5 monclient: authenticate success, global_id 54111 2024-09-11T01:33:56.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.693+0000 7f130effd640 1 -- 172.21.15.12:0/1026428961 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1308012a80 con 0x7f13180a2e90 2024-09-11T01:33:56.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.693+0000 7f131cf05640 10 monclient: _renew_subs 2024-09-11T01:33:56.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.693+0000 7f131cf05640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:33:56.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.693+0000 7f131cf05640 1 -- 172.21.15.12:0/1026428961 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13180a63b0 con 0x7f13180a2e90 2024-09-11T01:33:56.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.693+0000 7f130effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:56.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.693+0000 7f130effd640 1 -- 172.21.15.12:0/1026428961 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1308011380 con 0x7f13180a2e90 2024-09-11T01:33:56.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.693+0000 7f130effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:56.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.693+0000 7f130effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:33:56.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.693+0000 7f130effd640 10 monclient: dump: 2024-09-11T01:33:56.708 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:56.709 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:56.709 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:56.709 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:56.709 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:56.709 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:56.709 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:56.709 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:56.709 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:56.709 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:56.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.693+0000 7f131cf05640 10 monclient: _renew_subs 2024-09-11T01:33:56.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.693+0000 7f131559a640 4 set_mon_vals no callback set 2024-09-11T01:33:56.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.693+0000 7f131cf05640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:33:56.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.693+0000 7f131cf05640 1 -- 172.21.15.12:0/1026428961 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f131813c390 con 0x7f13180a2e90 2024-09-11T01:33:56.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.694+0000 7f131cf05640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:56.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.694+0000 7f131cf05640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:56.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.694+0000 7f131cf05640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:33:56.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.694+0000 7f130effd640 1 -- 172.21.15.12:0/1026428961 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f1308011560 con 0x7f13180a2e90 2024-09-11T01:33:56.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.694+0000 7f131cf05640 1 -- 172.21.15.12:0/1026428961 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13180a3680 con 0x7f13180a2e90 2024-09-11T01:33:56.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.694+0000 7f130effd640 1 --2- 172.21.15.12:0/1026428961 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f12f803e260 0x7f12f8040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:56.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.694+0000 7f130effd640 1 -- 172.21.15.12:0/1026428961 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(170..170 src has 1..170) ==== 4815+0+0 (secure 0 0 0) 0x7f1308052a20 con 0x7f13180a2e90 2024-09-11T01:33:56.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.694+0000 7f130ffff640 1 --2- 172.21.15.12:0/1026428961 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f12f803e260 0x7f12f8040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:56.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.694+0000 7f130ffff640 10 monclient: get_auth_request con 0x7f12f803e260 auth_method 0 2024-09-11T01:33:56.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.697+0000 7f130ffff640 1 --2- 172.21.15.12:0/1026428961 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f12f803e260 0x7f12f8040720 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1318071140 tx=0x7f130002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:56.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.699+0000 7f130effd640 1 -- 172.21.15.12:0/1026428961 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1308020070 con 0x7f13180a2e90 2024-09-11T01:33:56.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.699+0000 7f130effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:56.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.699+0000 7f130effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:33:56.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.954+0000 7f131cf05640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:33:56.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.954+0000 7f131cf05640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:33:56.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.954+0000 7f131cf05640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:33:56.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.954+0000 7f131cf05640 1 -- 172.21.15.12:0/1026428961 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f131809d2f0 con 0x7f13180a2e90 2024-09-11T01:33:56.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.954+0000 7f130effd640 1 -- 172.21.15.12:0/1026428961 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f1308051400 con 0x7f13180a2e90 2024-09-11T01:33:56.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.954+0000 7f130effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:33:56.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.954+0000 7f130effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:33:56.957 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:33:56.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.957+0000 7f131cf05640 1 -- 172.21.15.12:0/1026428961 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f12f803e260 msgr2=0x7f12f8040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:56.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.957+0000 7f131cf05640 1 --2- 172.21.15.12:0/1026428961 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f12f803e260 0x7f12f8040720 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1318071140 tx=0x7f130002e000 comp rx=0 tx=0).stop 2024-09-11T01:33:56.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.957+0000 7f131cf05640 10 monclient: shutdown 2024-09-11T01:33:56.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.957+0000 7f131cf05640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:56.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.957+0000 7f131cf05640 1 -- 172.21.15.12:0/1026428961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13180a2e90 msgr2=0x7f13180a1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:56.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.957+0000 7f131cf05640 1 --2- 172.21.15.12:0/1026428961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13180a2e90 0x7f13180a1cd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f130800c460 tx=0x7f130800c920 comp rx=0 tx=0).stop 2024-09-11T01:33:56.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.957+0000 7f131cf05640 1 -- 172.21.15.12:0/1026428961 shutdown_connections 2024-09-11T01:33:56.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.957+0000 7f131cf05640 1 --2- 172.21.15.12:0/1026428961 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f131813d490 0x7f131809cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:56.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.957+0000 7f131cf05640 1 --2- 172.21.15.12:0/1026428961 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13180a3270 0x7f131809c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:56.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.957+0000 7f131cf05640 1 --2- 172.21.15.12:0/1026428961 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f12f803e260 0x7f12f8040720 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:56.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.957+0000 7f131cf05640 1 --2- 172.21.15.12:0/1026428961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13180a2e90 0x7f13180a1cd0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:56.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.957+0000 7f131cf05640 1 -- 172.21.15.12:0/1026428961 >> 172.21.15.12:0/1026428961 conn(0x7f13180a8490 msgr2=0x7f13180a6d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:56.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.957+0000 7f131cf05640 1 -- 172.21.15.12:0/1026428961 shutdown_connections 2024-09-11T01:33:56.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:56.957+0000 7f131cf05640 1 -- 172.21.15.12:0/1026428961 wait complete. 2024-09-11T01:33:56.969 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:33:56.969 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:33:56.969 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:33:57.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.069+0000 7fc2babc6640 10 monclient: get_monmap_and_config 2024-09-11T01:33:57.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.069+0000 7fc2babc6640 10 monclient: build_initial_monmap 2024-09-11T01:33:57.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.069+0000 7fc2babc6640 10 monclient: monmap: 2024-09-11T01:33:57.071 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:57.071 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:57.071 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:33:57.069800+0000 2024-09-11T01:33:57.071 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:33:57.069800+0000 2024-09-11T01:33:57.071 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:57.071 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.071 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:33:57.071 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:33:57.071 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:33:57.071 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:33:57.071 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:33:57.071 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.069+0000 7fc2babc6640 1 Processor -- start 2024-09-11T01:33:57.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.069+0000 7fc2babc6640 1 -- start start 2024-09-11T01:33:57.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.069+0000 7fc2babc6640 10 monclient: init 2024-09-11T01:33:57.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2babc6640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:57.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2babc6640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-11T01:33:57.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2babc6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc2b4137d50 0x7fc2b409d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2babc6640 10 monclient(hunting): picked mon.noname-b con 0x7fc2b4137d50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:57.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2babc6640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc2b4138130 0x7fc2b4138550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2babc6640 10 monclient(hunting): picked mon.noname-a con 0x7fc2b4138130 addr v2:172.21.15.12:3300/0 2024-09-11T01:33:57.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2babc6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc2b4136cd0 0x7fc2b4137120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2babc6640 10 monclient(hunting): picked mon.noname-c con 0x7fc2b4136cd0 addr v2:172.21.15.71:3300/0 2024-09-11T01:33:57.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2babc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2babc6640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc2b409c650 con 0x7fc2b4138130 2024-09-11T01:33:57.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2babc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2babc6640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fc2b409ba20 con 0x7fc2b4136cd0 2024-09-11T01:33:57.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2babc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2babc6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc2b409c370 con 0x7fc2b4137d50 2024-09-11T01:33:57.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2babc6640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:57.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2babc6640 10 monclient(hunting): authenticate will time out at 1043.512439s 2024-09-11T01:33:57.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b8962640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc2b4137d50 0x7fc2b409d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b3fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc2b4138130 0x7fc2b4138550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b8962640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc2b4137d50 0x7fc2b409d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:37224/0 (socket says 172.21.15.12:37224) 2024-09-11T01:33:57.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b3fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc2b4138130 0x7fc2b4138550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42198/0 (socket says 172.21.15.12:42198) 2024-09-11T01:33:57.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b8962640 1 -- 172.21.15.12:0/2546803782 learned_addr learned my addr 172.21.15.12:0/2546803782 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:57.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b8962640 10 monclient(hunting): get_auth_request con 0x7fc2b4137d50 auth_method 0 2024-09-11T01:33:57.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b8962640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:57.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b8962640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:57.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b8962640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:57.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b3fff640 10 monclient(hunting): get_auth_request con 0x7fc2b4138130 auth_method 0 2024-09-11T01:33:57.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:57.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:57.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:57.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:57.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:57.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:57.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b8962640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:57.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b8962640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:57.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b8962640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:57.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b9163640 1 --2- 172.21.15.12:0/2546803782 >> v2:172.21.15.71:3300/0 conn(0x7fc2b4136cd0 0x7fc2b4137120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b8962640 10 monclient(hunting): handle_auth_done global_id 44201 payload 274 2024-09-11T01:33:57.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b8962640 1 -- 172.21.15.12:0/2546803782 >> v2:172.21.15.71:3300/0 conn(0x7fc2b4136cd0 msgr2=0x7fc2b4137120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:57.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b8962640 1 --2- 172.21.15.12:0/2546803782 >> v2:172.21.15.71:3300/0 conn(0x7fc2b4136cd0 0x7fc2b4137120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b8962640 1 -- 172.21.15.12:0/2546803782 >> v2:172.21.15.12:3300/0 conn(0x7fc2b4138130 msgr2=0x7fc2b4138550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:57.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b8962640 1 --2- 172.21.15.12:0/2546803782 >> v2:172.21.15.12:3300/0 conn(0x7fc2b4138130 0x7fc2b4138550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b8962640 10 monclient: _finish_hunting 0 2024-09-11T01:33:57.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b8962640 1 monclient: found mon.noname-b 2024-09-11T01:33:57.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b8962640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:57.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b8962640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:33:57.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b8962640 1 -- 172.21.15.12:0/2546803782 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b40a6ce0 con 0x7fc2b4137d50 2024-09-11T01:33:57.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b3fff640 1 --2- 172.21.15.12:0/2546803782 >> v2:172.21.15.12:3300/0 conn(0x7fc2b4138130 0x7fc2b4138550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:33:57.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b9163640 10 monclient: get_auth_request con 0x7fc2b4136cd0 auth_method 0 2024-09-11T01:33:57.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.070+0000 7fc2b9163640 1 --2- 172.21.15.12:0/2546803782 >> v2:172.21.15.71:3300/0 conn(0x7fc2b4136cd0 0x7fc2b4137120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:33:57.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.073+0000 7fc2b8962640 1 -- 172.21.15.12:0/2546803782 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc2b4137d50 msgr2=0x7fc2b409d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:57.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.073+0000 7fc2b8962640 1 --2- 172.21.15.12:0/2546803782 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc2b4137d50 0x7fc2b409d110 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc2b40a6f10 tx=0x7fc2ac02a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=337ec980c3c70e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:57.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.073+0000 7fc2b37fe640 1 -- 172.21.15.12:0/2546803782 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac036070 con 0x7fc2b4137d50 2024-09-11T01:33:57.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.073+0000 7fc2b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:57.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.073+0000 7fc2b37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:33:57.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.073+0000 7fc2b37fe640 10 monclient: dump: 2024-09-11T01:33:57.080 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:57.080 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:57.080 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.080 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.080 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:57.081 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.081 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:57.081 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:57.081 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:57.081 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.073+0000 7fc2b37fe640 10 monclient: _finish_auth 0 2024-09-11T01:33:57.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.073+0000 7fc2b37fe640 10 monclient: _check_auth_tickets 2024-09-11T01:33:57.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.073+0000 7fc2b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:57.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.073+0000 7fc2b37fe640 1 -- 172.21.15.12:0/2546803782 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2ac007800 con 0x7fc2b4137d50 2024-09-11T01:33:57.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.073+0000 7fc2babc6640 5 monclient: authenticate success, global_id 44201 2024-09-11T01:33:57.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.073+0000 7fc2babc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:33:57.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.073+0000 7fc2b37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:57.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.073+0000 7fc2babc6640 10 monclient: get_monmap_and_config success 2024-09-11T01:33:57.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.073+0000 7fc2b37fe640 1 -- 172.21.15.12:0/2546803782 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac007bd0 con 0x7fc2b4137d50 2024-09-11T01:33:57.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.073+0000 7fc2b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:57.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.073+0000 7fc2babc6640 4 set_mon_vals no callback set 2024-09-11T01:33:57.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.073+0000 7fc2b37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:57.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.073+0000 7fc2b37fe640 10 monclient: dump: 2024-09-11T01:33:57.082 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:57.082 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:57.082 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.082 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.082 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:57.082 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.082 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:57.082 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:57.083 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:57.083 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.074+0000 7fc2babc6640 10 monclient: shutdown 2024-09-11T01:33:57.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.074+0000 7fc2babc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:57.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.074+0000 7fc2babc6640 1 -- 172.21.15.12:0/2546803782 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc2b4137d50 msgr2=0x7fc2b409d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:57.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.074+0000 7fc2babc6640 1 --2- 172.21.15.12:0/2546803782 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc2b4137d50 0x7fc2b409d110 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc2b40a6f10 tx=0x7fc2ac02a5f0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.074+0000 7fc2babc6640 1 -- 172.21.15.12:0/2546803782 shutdown_connections 2024-09-11T01:33:57.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.074+0000 7fc2babc6640 1 --2- 172.21.15.12:0/2546803782 >> v2:172.21.15.71:3300/0 conn(0x7fc2b4136cd0 0x7fc2b4137120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.074+0000 7fc2babc6640 1 --2- 172.21.15.12:0/2546803782 >> v2:172.21.15.12:3300/0 conn(0x7fc2b4138130 0x7fc2b4138550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.074+0000 7fc2babc6640 1 --2- 172.21.15.12:0/2546803782 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc2b4137d50 0x7fc2b409d110 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.074+0000 7fc2babc6640 1 -- 172.21.15.12:0/2546803782 >> 172.21.15.12:0/2546803782 conn(0x7fc2b40a7020 msgr2=0x7fc2b40a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:57.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.074+0000 7fc2babc6640 1 -- 172.21.15.12:0/2546803782 shutdown_connections 2024-09-11T01:33:57.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.074+0000 7fc2babc6640 1 -- 172.21.15.12:0/2546803782 wait complete. 2024-09-11T01:33:57.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.075+0000 7fc2babc6640 10 monclient: build_initial_monmap 2024-09-11T01:33:57.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.075+0000 7fc2babc6640 10 monclient: monmap: 2024-09-11T01:33:57.089 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:57.089 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:33:57.089 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:33:57.089 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:33:57.089 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:57.089 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.089 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:33:57.089 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:33:57.089 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:33:57.089 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.075+0000 7fc2babc6640 1 Processor -- start 2024-09-11T01:33:57.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.075+0000 7fc2babc6640 1 -- start start 2024-09-11T01:33:57.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.075+0000 7fc2babc6640 10 monclient: init 2024-09-11T01:33:57.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.075+0000 7fc2babc6640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:57.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.075+0000 7fc2babc6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:33:57.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.075+0000 7fc2babc6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2b4136cd0 0x7fc2b4123300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.075+0000 7fc2babc6640 10 monclient(hunting): picked mon.noname-b con 0x7fc2b4136cd0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:33:57.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2babc6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc2b4137d50 0x7fc2b4123840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2babc6640 10 monclient(hunting): picked mon.noname-c con 0x7fc2b4137d50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:57.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2babc6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b4138130 0x7fc2b4123d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2babc6640 10 monclient(hunting): picked mon.noname-a con 0x7fc2b4138130 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:33:57.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2babc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2babc6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b406f850 con 0x7fc2b4138130 2024-09-11T01:33:57.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2babc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2babc6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b406f6d0 con 0x7fc2b4136cd0 2024-09-11T01:33:57.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2babc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2babc6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc2b409d650 con 0x7fc2b4137d50 2024-09-11T01:33:57.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2babc6640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:57.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2babc6640 10 monclient(hunting): authenticate will time out at 1043.518061s 2024-09-11T01:33:57.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b3fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc2b4137d50 0x7fc2b4123840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b9163640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b4138130 0x7fc2b4123d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b3fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc2b4137d50 0x7fc2b4123840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:37226/0 (socket says 172.21.15.12:37226) 2024-09-11T01:33:57.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b3fff640 1 -- 172.21.15.12:0/3671307858 learned_addr learned my addr 172.21.15.12:0/3671307858 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:57.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b3fff640 10 monclient(hunting): get_auth_request con 0x7fc2b4137d50 auth_method 0 2024-09-11T01:33:57.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:57.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:57.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:57.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b9163640 10 monclient(hunting): get_auth_request con 0x7fc2b4138130 auth_method 0 2024-09-11T01:33:57.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b9163640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:57.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b9163640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:57.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b9163640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:57.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:57.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:57.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:57.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b9163640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:57.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b9163640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:57.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b9163640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:57.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b8962640 1 --2- 172.21.15.12:0/3671307858 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2b4136cd0 0x7fc2b4123300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b8962640 10 monclient(hunting): get_auth_request con 0x7fc2b4136cd0 auth_method 0 2024-09-11T01:33:57.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b8962640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:57.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b8962640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:57.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b8962640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:57.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b3fff640 10 monclient(hunting): handle_auth_done global_id 44204 payload 824 2024-09-11T01:33:57.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b3fff640 1 -- 172.21.15.12:0/3671307858 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2b4136cd0 msgr2=0x7fc2b4123300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b3fff640 1 --2- 172.21.15.12:0/3671307858 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2b4136cd0 0x7fc2b4123300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b3fff640 1 -- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b4138130 msgr2=0x7fc2b4123d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b3fff640 1 --2- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b4138130 0x7fc2b4123d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b3fff640 10 monclient: _finish_hunting 0 2024-09-11T01:33:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b3fff640 1 monclient: found mon.noname-c 2024-09-11T01:33:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:33:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b3fff640 1 -- 172.21.15.12:0/3671307858 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b4127120 con 0x7fc2b4137d50 2024-09-11T01:33:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.076+0000 7fc2b9163640 1 --2- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b4138130 0x7fc2b4123d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:33:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.078+0000 7fc2b3fff640 1 -- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc2b4137d50 msgr2=0x7fc2b4123840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.078+0000 7fc2b3fff640 1 --2- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc2b4137d50 0x7fc2b4123840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc2a400ebe0 tx=0x7fc2a400c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.078+0000 7fc2b17fa640 1 -- 172.21.15.12:0/3671307858 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a4017070 con 0x7fc2b4137d50 2024-09-11T01:33:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.078+0000 7fc2b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.078+0000 7fc2b17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:33:57.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.078+0000 7fc2b17fa640 10 monclient: dump: 2024-09-11T01:33:57.094 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:57.094 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:57.094 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.094 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.094 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:57.094 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.094 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:57.094 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:57.094 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:57.094 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.078+0000 7fc2b17fa640 10 monclient: _finish_auth 0 2024-09-11T01:33:57.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.078+0000 7fc2b17fa640 10 monclient: _check_auth_tickets 2024-09-11T01:33:57.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.078+0000 7fc2b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:57.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.078+0000 7fc2b17fa640 1 -- 172.21.15.12:0/3671307858 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2a400ce40 con 0x7fc2b4137d50 2024-09-11T01:33:57.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.078+0000 7fc2babc6640 5 monclient: authenticate success, global_id 44204 2024-09-11T01:33:57.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.078+0000 7fc2babc6640 10 monclient: _renew_subs 2024-09-11T01:33:57.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.078+0000 7fc2babc6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:57.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.078+0000 7fc2babc6640 1 -- 172.21.15.12:0/3671307858 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2b40a1710 con 0x7fc2b4137d50 2024-09-11T01:33:57.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.078+0000 7fc2b17fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:57.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.078+0000 7fc2b17fa640 1 -- 172.21.15.12:0/3671307858 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a4013690 con 0x7fc2b4137d50 2024-09-11T01:33:57.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.078+0000 7fc2b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:57.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.078+0000 7fc2b17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:57.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.078+0000 7fc2b17fa640 10 monclient: dump: 2024-09-11T01:33:57.095 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:57.095 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:57.095 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.095 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.095 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:57.096 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.096 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:57.096 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:57.096 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:57.096 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.078+0000 7fc2b2ffd640 4 set_mon_vals no callback set 2024-09-11T01:33:57.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.078+0000 7fc2babc6640 10 monclient: _renew_subs 2024-09-11T01:33:57.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.078+0000 7fc2babc6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:57.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.078+0000 7fc2babc6640 1 -- 172.21.15.12:0/3671307858 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2b4126300 con 0x7fc2b4137d50 2024-09-11T01:33:57.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.079+0000 7fc2babc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:57.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.079+0000 7fc2babc6640 1 --2- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b409f050 0x7fc2b409f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.079+0000 7fc2babc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.079+0000 7fc2babc6640 1 -- 172.21.15.12:0/3671307858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b409d110 con 0x7fc2b409f050 2024-09-11T01:33:57.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.079+0000 7fc2b8962640 1 --2- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b409f050 0x7fc2b409f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.079+0000 7fc2b17fa640 1 -- 172.21.15.12:0/3671307858 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fc2a4024060 con 0x7fc2b4137d50 2024-09-11T01:33:57.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.079+0000 7fc2b8962640 10 monclient: get_auth_request con 0x7fc2b409f050 auth_method 0 2024-09-11T01:33:57.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.079+0000 7fc2b8962640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:57.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.079+0000 7fc2b8962640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:57.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.079+0000 7fc2b8962640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:57.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.080+0000 7fc2b8962640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:57.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.080+0000 7fc2b8962640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:57.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.080+0000 7fc2b8962640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:57.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.080+0000 7fc2b8962640 10 monclient(hunting): handle_auth_done global_id 54120 payload 824 2024-09-11T01:33:57.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.080+0000 7fc2b8962640 1 -- 172.21.15.12:0/3671307858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc2b4126f00 con 0x7fc2b409f050 2024-09-11T01:33:57.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.080+0000 7fc2b17fa640 1 --2- 172.21.15.12:0/3671307858 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc29403e210 0x7fc2940406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.080+0000 7fc2b17fa640 1 -- 172.21.15.12:0/3671307858 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(170..170 src has 1..170) ==== 4815+0+0 (secure 0 0 0) 0x7fc2a4052cb0 con 0x7fc2b4137d50 2024-09-11T01:33:57.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.080+0000 7fc2b9163640 1 --2- 172.21.15.12:0/3671307858 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc29403e210 0x7fc2940406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.080+0000 7fc2b9163640 10 monclient: get_auth_request con 0x7fc29403e210 auth_method 0 2024-09-11T01:33:57.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.082+0000 7fc2b8962640 1 -- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b409f050 msgr2=0x7fc2b409f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:57.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.082+0000 7fc2b8962640 1 --2- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b409f050 0x7fc2b409f430 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc2b40a6050 tx=0x7fc2ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:57.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.082+0000 7fc2b17fa640 1 -- 172.21.15.12:0/3671307858 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac036040 con 0x7fc2b409f050 2024-09-11T01:33:57.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.082+0000 7fc2b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:57.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.082+0000 7fc2b17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:33:57.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.082+0000 7fc2b17fa640 10 monclient: dump: 2024-09-11T01:33:57.098 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:57.098 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:57.098 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.098 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.098 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:57.098 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.098 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:57.099 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:57.099 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:57.099 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.082+0000 7fc2b17fa640 1 -- 172.21.15.12:0/3671307858 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc2ac03c850 con 0x7fc2b409f050 2024-09-11T01:33:57.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.082+0000 7fc2b17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:57.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.082+0000 7fc2b17fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:33:57.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.082+0000 7fc2b17fa640 1 -- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b409f050 msgr2=0x7fc2b409f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:57.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.082+0000 7fc2b17fa640 1 --2- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b409f050 0x7fc2b409f430 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc2b40a6050 tx=0x7fc2ac004a40 comp rx=0 tx=0).stop 2024-09-11T01:33:57.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.082+0000 7fc2b17fa640 1 -- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b409f050 msgr2=0x7fc2b409f430 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:33:57.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.082+0000 7fc2b17fa640 1 --2- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b409f050 0x7fc2b409f430 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc2b40a6050 tx=0x7fc2ac004a40 comp rx=0 tx=0).stop 2024-09-11T01:33:57.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.082+0000 7fc2b9163640 1 --2- 172.21.15.12:0/3671307858 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc29403e210 0x7fc2940406d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc2b41373a0 tx=0x7fc29c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:57.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.089+0000 7fc2babc6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:33:57.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.089+0000 7fc2babc6640 1 --2- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b4125b90 0x7fc2b409f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.089+0000 7fc2babc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.089+0000 7fc2babc6640 1 -- 172.21.15.12:0/3671307858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b4074a40 con 0x7fc2b4125b90 2024-09-11T01:33:57.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.089+0000 7fc2b8962640 1 --2- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b4125b90 0x7fc2b409f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.089+0000 7fc2b8962640 10 monclient: get_auth_request con 0x7fc2b4125b90 auth_method 0 2024-09-11T01:33:57.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.089+0000 7fc2b8962640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:57.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.089+0000 7fc2b8962640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:57.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.089+0000 7fc2b8962640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:57.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.089+0000 7fc2b8962640 1 -- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b4125b90 msgr2=0x7fc2b409f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:33:57.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.089+0000 7fc2b8962640 1 -- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b4125b90 msgr2=0x7fc2b409f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:33:57.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.089+0000 7fc2b8962640 1 --2- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b4125b90 0x7fc2b409f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:33:57.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.089+0000 7fc2b8962640 1 --2- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b4125b90 0x7fc2b409f970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:33:57.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.290+0000 7fc2b8962640 1 --2- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b4125b90 0x7fc2b409f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.290+0000 7fc2b8962640 10 monclient: get_auth_request con 0x7fc2b4125b90 auth_method 0 2024-09-11T01:33:57.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.290+0000 7fc2b8962640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:57.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.290+0000 7fc2b8962640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:57.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.290+0000 7fc2b8962640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:33:57.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.290+0000 7fc2b8962640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:57.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.290+0000 7fc2b8962640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:57.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.290+0000 7fc2b8962640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:57.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.290+0000 7fc2b8962640 10 monclient(hunting): handle_auth_done global_id 54126 payload 824 2024-09-11T01:33:57.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.291+0000 7fc2b8962640 1 -- 172.21.15.12:0/3671307858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc2b409feb0 con 0x7fc2b4125b90 2024-09-11T01:33:57.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.293+0000 7fc2b8962640 1 -- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b4125b90 msgr2=0x7fc2b409f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:57.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.293+0000 7fc2b8962640 1 --2- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b4125b90 0x7fc2b409f970 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac000f80 tx=0x7fc2ac02f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:57.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.293+0000 7fc2b17fa640 1 -- 172.21.15.12:0/3671307858 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac036040 con 0x7fc2b4125b90 2024-09-11T01:33:57.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.293+0000 7fc2b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:57.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.293+0000 7fc2b17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:33:57.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.293+0000 7fc2b17fa640 10 monclient: dump: 2024-09-11T01:33:57.295 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:57.295 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:57.295 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.295 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.296 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:57.296 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.296 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:57.296 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:57.296 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:57.296 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.293+0000 7fc2b17fa640 1 -- 172.21.15.12:0/3671307858 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7fc2ac045650 con 0x7fc2b4125b90 2024-09-11T01:33:57.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.293+0000 7fc2b17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:33:57.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.293+0000 7fc2b17fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:33:57.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.293+0000 7fc2b17fa640 1 -- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b4125b90 msgr2=0x7fc2b409f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:57.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.293+0000 7fc2b17fa640 1 --2- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b4125b90 0x7fc2b409f970 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac000f80 tx=0x7fc2ac02f000 comp rx=0 tx=0).stop 2024-09-11T01:33:57.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.293+0000 7fc2b17fa640 1 -- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b4125b90 msgr2=0x7fc2b409f970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:33:57.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.293+0000 7fc2b17fa640 1 --2- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc2b4125b90 0x7fc2b409f970 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac000f80 tx=0x7fc2ac02f000 comp rx=0 tx=0).stop 2024-09-11T01:33:57.297 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:33:57.297 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:33:57.297 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:33:57.297 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:33:57.297 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 56, 2024-09-11T01:33:57.297 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 27274, 2024-09-11T01:33:57.297 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:33:57.297 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:33:57.297 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:33:57.297 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:33:57.297 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:57.297 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:33:57.297 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:33:57.298 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:33:57.298 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:33:57.298 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:33:57.298 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:33:57.298 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:33:57.298 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:33:57.298 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:33:57.298 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:33:57.298 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:33:57.298 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:33:57.298 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:33:57.298 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:33:57.298 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:33:57.299 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:57.299 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:33:57.299 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:33:57.299 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:33:57.299 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:33:57.299 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:33:57.299 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:33:57.299 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:33:57.299 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:33:57.299 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:33:57.299 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:33:57.299 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:33:57.299 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:33:57.299 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:33:57.299 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:57.300 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.300 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:33:57.300 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:33:57.300 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:33:57.300 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:33:57.300 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:33:57.300 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:33:57.300 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:33:57.300 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:33:57.300 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:33:57.300 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:33:57.300 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:33:57.300 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:33:57.300 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:33:57.301 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:33:57.301 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:33:57.301 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:33:57.301 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:33:57.301 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:33:57.301 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:33:57.301 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:33:57.301 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:33:57.301 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:33:57.301 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:33:57.301 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:33:57.301 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:33:57.301 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:33:57.301 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:33:57.302 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:33:57.302 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:57.302 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:33:57.302 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.302 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:33:57.302 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.302 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:33:57.302 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:33:57.302 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:33:57.302 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:33:57.302 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.302 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:33:57.302 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:33:57.302 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:57.302 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.303 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.303 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:33:57.303 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:33:57.303 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:57.303 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:57.303 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:57.303 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.303 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:33:57.303 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:33:57.303 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:33:57.303 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:33:57.303 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:33:57.303 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.303 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.304 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:33:57.304 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:33:57.304 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:33:57.304 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:33:57.304 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.304 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:33:57.304 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:33:57.304 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:57.304 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.304 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.304 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:33:57.304 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:33:57.304 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:57.304 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:57.305 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:57.305 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.305 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:33:57.305 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:33:57.305 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:33:57.305 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:33:57.305 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:33:57.305 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.305 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.305 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:33:57.305 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:33:57.305 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:33:57.305 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:33:57.305 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.306 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:33:57.306 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:33:57.306 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:57.306 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.306 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.306 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:33:57.306 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:33:57.306 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:57.306 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:57.306 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:57.306 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.306 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:33:57.306 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:33:57.306 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:33:57.307 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:33:57.307 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:33:57.307 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:57.307 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:57.307 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.307 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:33:57.307 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:33:57.307 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.307 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:33:57.307 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:33:57.307 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:33:57.307 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:57.307 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:57.308 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:33:57.308 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.308 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:33:57.308 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:33:57.308 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:33:57.308 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:57.308 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:57.308 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:33:57.308 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.308 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:33:57.308 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:33:57.308 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:33:57.308 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:57.308 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:57.309 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.309 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:33:57.309 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:33:57.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.293+0000 7fc2babc6640 1 -- 172.21.15.12:0/3671307858 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc29403e210 msgr2=0x7fc2940406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:57.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.293+0000 7fc2babc6640 1 --2- 172.21.15.12:0/3671307858 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc29403e210 0x7fc2940406d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc2b41373a0 tx=0x7fc29c008040 comp rx=0 tx=0).stop 2024-09-11T01:33:57.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.293+0000 7fc2babc6640 10 monclient: shutdown 2024-09-11T01:33:57.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.293+0000 7fc2babc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:57.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.293+0000 7fc2babc6640 1 -- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc2b4137d50 msgr2=0x7fc2b4123840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:57.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.293+0000 7fc2babc6640 1 --2- 172.21.15.12:0/3671307858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc2b4137d50 0x7fc2b4123840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc2a400ebe0 tx=0x7fc2a400c520 comp rx=0 tx=0).stop 2024-09-11T01:33:57.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.293+0000 7fc2b9163640 1 -- 172.21.15.12:0/3671307858 reap_dead start 2024-09-11T01:33:57.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.294+0000 7fc2b9163640 1 -- 172.21.15.12:0/3671307858 reap_dead start 2024-09-11T01:33:57.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.294+0000 7fc2babc6640 1 -- 172.21.15.12:0/3671307858 shutdown_connections 2024-09-11T01:33:57.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.294+0000 7fc2babc6640 1 -- 172.21.15.12:0/3671307858 >> 172.21.15.12:0/3671307858 conn(0x7fc2b40a7020 msgr2=0x7fc2b40a8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:57.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.294+0000 7fc2babc6640 1 -- 172.21.15.12:0/3671307858 shutdown_connections 2024-09-11T01:33:57.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.294+0000 7fc2babc6640 1 -- 172.21.15.12:0/3671307858 wait complete. 2024-09-11T01:33:57.311 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:33:57.311 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:33:57.311 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:33:57.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.412+0000 7f2991394640 10 monclient: get_monmap_and_config 2024-09-11T01:33:57.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.412+0000 7f2991394640 10 monclient: build_initial_monmap 2024-09-11T01:33:57.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.412+0000 7f2991394640 10 monclient: monmap: 2024-09-11T01:33:57.414 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:57.414 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:57.415 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:33:57.413512+0000 2024-09-11T01:33:57.415 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:33:57.413512+0000 2024-09-11T01:33:57.415 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:57.415 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.415 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:33:57.415 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:33:57.415 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:33:57.415 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:33:57.415 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:33:57.415 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.413+0000 7f2991394640 1 Processor -- start 2024-09-11T01:33:57.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.413+0000 7f2991394640 1 -- start start 2024-09-11T01:33:57.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.413+0000 7f2991394640 10 monclient: init 2024-09-11T01:33:57.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.413+0000 7f2991394640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:57.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.413+0000 7f2991394640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-11T01:33:57.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.413+0000 7f2991394640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f298c1381c0 addr v1:172.21.15.12:6789/0 2024-09-11T01:33:57.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.413+0000 7f2991394640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f298c0a6aa0 0x7f298c09cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.413+0000 7f2991394640 10 monclient(hunting): picked mon.noname-c con 0x7f298c0a6aa0 addr v2:172.21.15.71:3300/0 2024-09-11T01:33:57.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.413+0000 7f2991394640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c13d000 0x7f298c13f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.413+0000 7f2991394640 10 monclient(hunting): picked mon.noname-b con 0x7f298c13d000 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:57.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.413+0000 7f2991394640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f298c09c310 con 0x7f298c1381c0 2024-09-11T01:33:57.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.413+0000 7f2991394640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.413+0000 7f2991394640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f298c06fc90 con 0x7f298c0a6aa0 2024-09-11T01:33:57.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.413+0000 7f2991394640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.413+0000 7f2991394640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f298c09c5f0 con 0x7f298c13d000 2024-09-11T01:33:57.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.413+0000 7f2991394640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:57.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.413+0000 7f2991394640 10 monclient(hunting): authenticate will time out at 1043.855987s 2024-09-11T01:33:57.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.414+0000 7f298b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c13d000 0x7f298c13f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.414+0000 7f298affd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f298c1381c0 0x7f298c1385a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:32990/0 (socket says 172.21.15.12:32990) 2024-09-11T01:33:57.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.414+0000 7f298affd640 1 -- 172.21.15.12:0/4157158256 learned_addr learned my addr 172.21.15.12:0/4157158256 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:33:57.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.414+0000 7f298b7fe640 10 monclient(hunting): get_auth_request con 0x7f298c13d000 auth_method 0 2024-09-11T01:33:57.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.414+0000 7f298b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:57.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.414+0000 7f298b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:57.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.414+0000 7f298b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:57.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.414+0000 7f298b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:57.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.414+0000 7f298b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:57.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.414+0000 7f298b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:57.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.414+0000 7f298a7fc640 1 --2- 172.21.15.12:0/4157158256 >> v2:172.21.15.71:3300/0 conn(0x7f298c0a6aa0 0x7f298c09cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.414+0000 7f298b7fe640 10 monclient(hunting): handle_auth_done global_id 44207 payload 274 2024-09-11T01:33:57.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.414+0000 7f298b7fe640 1 -- 172.21.15.12:0/4157158256 >> v2:172.21.15.71:3300/0 conn(0x7f298c0a6aa0 msgr2=0x7f298c09cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:57.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.414+0000 7f298b7fe640 1 --2- 172.21.15.12:0/4157158256 >> v2:172.21.15.71:3300/0 conn(0x7f298c0a6aa0 0x7f298c09cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.414+0000 7f298b7fe640 1 -- 172.21.15.12:0/4157158256 >> v1:172.21.15.12:6789/0 conn(0x7f298c1381c0 legacy=0x7f298c1385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:57.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.414+0000 7f298b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:33:57.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.414+0000 7f298b7fe640 1 monclient: found mon.noname-b 2024-09-11T01:33:57.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.414+0000 7f298b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:57.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.414+0000 7f298b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:33:57.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.414+0000 7f298b7fe640 1 -- 172.21.15.12:0/4157158256 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f298c09c8d0 con 0x7f298c13d000 2024-09-11T01:33:57.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f298b7fe640 1 -- 172.21.15.12:0/4157158256 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c13d000 msgr2=0x7f298c13f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:57.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f298b7fe640 1 --2- 172.21.15.12:0/4157158256 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c13d000 0x7f298c13f3f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f29840097a0 tx=0x7f2984029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75b919b1bfa94474 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:57.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2989ffb640 1 -- 172.21.15.12:0/4157158256 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2984037070 con 0x7f298c13d000 2024-09-11T01:33:57.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2989ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:57.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2989ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:33:57.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2989ffb640 10 monclient: dump: 2024-09-11T01:33:57.427 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:57.427 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:57.427 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.427 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.427 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:57.427 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.427 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:57.427 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:57.427 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:57.427 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2989ffb640 10 monclient: _finish_auth 0 2024-09-11T01:33:57.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2989ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:33:57.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2989ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:57.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2991394640 5 monclient: authenticate success, global_id 44207 2024-09-11T01:33:57.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2991394640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:33:57.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2989ffb640 1 -- 172.21.15.12:0/4157158256 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2984032440 con 0x7f298c13d000 2024-09-11T01:33:57.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2989ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:57.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2991394640 10 monclient: get_monmap_and_config success 2024-09-11T01:33:57.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2989ffb640 1 -- 172.21.15.12:0/4157158256 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2984032810 con 0x7f298c13d000 2024-09-11T01:33:57.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2991394640 4 set_mon_vals no callback set 2024-09-11T01:33:57.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2989ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:57.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2989ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:57.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2989ffb640 10 monclient: dump: 2024-09-11T01:33:57.428 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:57.428 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:57.428 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.428 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.428 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:57.429 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.429 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:57.429 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:57.429 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:57.429 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2991394640 10 monclient: shutdown 2024-09-11T01:33:57.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2991394640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:57.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2991394640 1 -- 172.21.15.12:0/4157158256 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c13d000 msgr2=0x7f298c13f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:57.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2991394640 1 --2- 172.21.15.12:0/4157158256 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c13d000 0x7f298c13f3f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f29840097a0 tx=0x7f2984029fb0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2991394640 1 -- 172.21.15.12:0/4157158256 shutdown_connections 2024-09-11T01:33:57.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2991394640 1 --2- 172.21.15.12:0/4157158256 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c13d000 0x7f298c13f3f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2991394640 1 --2- 172.21.15.12:0/4157158256 >> v2:172.21.15.71:3300/0 conn(0x7f298c0a6aa0 0x7f298c09cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2991394640 1 -- 172.21.15.12:0/4157158256 >> 172.21.15.12:0/4157158256 conn(0x7f298c0a7440 msgr2=0x7f298c0a7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:57.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2991394640 1 -- 172.21.15.12:0/4157158256 shutdown_connections 2024-09-11T01:33:57.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.417+0000 7f2991394640 1 -- 172.21.15.12:0/4157158256 wait complete. 2024-09-11T01:33:57.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.418+0000 7f2991394640 10 monclient: build_initial_monmap 2024-09-11T01:33:57.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.418+0000 7f2991394640 10 monclient: monmap: 2024-09-11T01:33:57.430 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:57.430 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:33:57.430 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:33:57.430 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:33:57.430 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:57.430 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.430 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:33:57.431 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:33:57.431 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:33:57.431 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.418+0000 7f2991394640 1 Processor -- start 2024-09-11T01:33:57.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.418+0000 7f2991394640 1 -- start start 2024-09-11T01:33:57.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.418+0000 7f2991394640 10 monclient: init 2024-09-11T01:33:57.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f2991394640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:57.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f2991394640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:33:57.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f2991394640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f298c0a6aa0 0x7f298c1ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f2991394640 10 monclient(hunting): picked mon.noname-a con 0x7f298c0a6aa0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:33:57.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f2991394640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f298c1381c0 0x7f298c136030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f2991394640 10 monclient(hunting): picked mon.noname-b con 0x7f298c1381c0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:33:57.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f2991394640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c13d000 0x7f298c139080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f2991394640 10 monclient(hunting): picked mon.noname-c con 0x7f298c13d000 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:57.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f2991394640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f2991394640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f298c13fab0 con 0x7f298c0a6aa0 2024-09-11T01:33:57.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f2991394640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f2991394640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f298c13f930 con 0x7f298c1381c0 2024-09-11T01:33:57.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f2991394640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f2991394640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f298c13fc30 con 0x7f298c13d000 2024-09-11T01:33:57.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f2991394640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:57.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f2991394640 10 monclient(hunting): authenticate will time out at 1043.861457s 2024-09-11T01:33:57.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f298c0a6aa0 0x7f298c1ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c13d000 0x7f298c139080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f298c0a6aa0 0x7f298c1ccf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42238/0 (socket says 172.21.15.12:42238) 2024-09-11T01:33:57.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298affd640 1 -- 172.21.15.12:0/898080103 learned_addr learned my addr 172.21.15.12:0/898080103 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:57.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298affd640 10 monclient(hunting): get_auth_request con 0x7f298c0a6aa0 auth_method 0 2024-09-11T01:33:57.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:57.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:57.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:57.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298b7fe640 10 monclient(hunting): get_auth_request con 0x7f298c13d000 auth_method 0 2024-09-11T01:33:57.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:57.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:57.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:57.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:57.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:57.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:57.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:57.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:57.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:57.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298a7fc640 1 --2- 172.21.15.12:0/898080103 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f298c1381c0 0x7f298c136030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298a7fc640 10 monclient(hunting): get_auth_request con 0x7f298c1381c0 auth_method 0 2024-09-11T01:33:57.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:57.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:57.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:57.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298b7fe640 10 monclient(hunting): handle_auth_done global_id 44210 payload 824 2024-09-11T01:33:57.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298b7fe640 1 -- 172.21.15.12:0/898080103 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f298c1381c0 msgr2=0x7f298c136030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:57.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298b7fe640 1 --2- 172.21.15.12:0/898080103 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f298c1381c0 0x7f298c136030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298b7fe640 1 -- 172.21.15.12:0/898080103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f298c0a6aa0 msgr2=0x7f298c1ccf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:57.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298b7fe640 1 --2- 172.21.15.12:0/898080103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f298c0a6aa0 0x7f298c1ccf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:33:57.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298b7fe640 1 monclient: found mon.noname-c 2024-09-11T01:33:57.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:57.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:33:57.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298b7fe640 1 -- 172.21.15.12:0/898080103 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f298c13b130 con 0x7f298c13d000 2024-09-11T01:33:57.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298a7fc640 1 --2- 172.21.15.12:0/898080103 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f298c1381c0 0x7f298c136030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:33:57.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.419+0000 7f298affd640 1 --2- 172.21.15.12:0/898080103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f298c0a6aa0 0x7f298c1ccf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:33:57.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.422+0000 7f298b7fe640 1 -- 172.21.15.12:0/898080103 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c13d000 msgr2=0x7f298c139080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:57.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.422+0000 7f298b7fe640 1 --2- 172.21.15.12:0/898080103 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c13d000 0x7f298c139080 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2984034000 tx=0x7f29840046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:57.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.422+0000 7f29737fe640 1 -- 172.21.15.12:0/898080103 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2984040020 con 0x7f298c13d000 2024-09-11T01:33:57.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.422+0000 7f29737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:57.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.422+0000 7f29737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:33:57.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.422+0000 7f29737fe640 10 monclient: dump: 2024-09-11T01:33:57.435 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:57.435 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:57.435 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.435 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.435 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:57.435 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:57.436 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.422+0000 7f29737fe640 10 monclient: _finish_auth 0 2024-09-11T01:33:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.422+0000 7f29737fe640 10 monclient: _check_auth_tickets 2024-09-11T01:33:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.422+0000 7f29737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.422+0000 7f29737fe640 1 -- 172.21.15.12:0/898080103 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2984038040 con 0x7f298c13d000 2024-09-11T01:33:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.422+0000 7f29737fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.422+0000 7f29737fe640 1 -- 172.21.15.12:0/898080103 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2984037070 con 0x7f298c13d000 2024-09-11T01:33:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.422+0000 7f2991394640 5 monclient: authenticate success, global_id 44210 2024-09-11T01:33:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.422+0000 7f2991394640 10 monclient: _renew_subs 2024-09-11T01:33:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.422+0000 7f2991394640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.422+0000 7f2991394640 1 -- 172.21.15.12:0/898080103 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f298c1cd480 con 0x7f298c13d000 2024-09-11T01:33:57.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.422+0000 7f29737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:57.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.422+0000 7f29737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:57.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.422+0000 7f29737fe640 10 monclient: dump: 2024-09-11T01:33:57.437 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:57.437 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:57.437 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.437 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.437 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:57.437 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.437 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:57.437 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:57.437 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:57.437 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.422+0000 7f29897fa640 4 set_mon_vals no callback set 2024-09-11T01:33:57.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.422+0000 7f2991394640 10 monclient: _renew_subs 2024-09-11T01:33:57.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.422+0000 7f2991394640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:57.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.422+0000 7f2991394640 1 -- 172.21.15.12:0/898080103 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f298c13b540 con 0x7f298c13d000 2024-09-11T01:33:57.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.423+0000 7f2991394640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:57.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.423+0000 7f2991394640 1 --2- 172.21.15.12:0/898080103 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c136570 0x7f298c1395c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.423+0000 7f2991394640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.423+0000 7f2991394640 1 -- 172.21.15.12:0/898080103 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f298c13c950 con 0x7f298c136570 2024-09-11T01:33:57.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.423+0000 7f29737fe640 1 -- 172.21.15.12:0/898080103 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f29840443b0 con 0x7f298c13d000 2024-09-11T01:33:57.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.423+0000 7f298affd640 1 --2- 172.21.15.12:0/898080103 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c136570 0x7f298c1395c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.423+0000 7f298affd640 10 monclient: get_auth_request con 0x7f298c136570 auth_method 0 2024-09-11T01:33:57.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.423+0000 7f298affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:57.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.423+0000 7f298affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:57.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.423+0000 7f298affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:57.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.423+0000 7f298affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:57.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.423+0000 7f298affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:57.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.423+0000 7f298affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:57.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.423+0000 7f29737fe640 1 --2- 172.21.15.12:0/898080103 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f297803e260 0x7f2978040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.423+0000 7f298affd640 10 monclient(hunting): handle_auth_done global_id 44213 payload 824 2024-09-11T01:33:57.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.423+0000 7f298affd640 1 -- 172.21.15.12:0/898080103 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f298c13f400 con 0x7f298c136570 2024-09-11T01:33:57.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.423+0000 7f29737fe640 1 -- 172.21.15.12:0/898080103 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(170..170 src has 1..170) ==== 4815+0+0 (secure 0 0 0) 0x7f2984073280 con 0x7f298c13d000 2024-09-11T01:33:57.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.423+0000 7f298a7fc640 1 --2- 172.21.15.12:0/898080103 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f297803e260 0x7f2978040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.423+0000 7f298a7fc640 10 monclient: get_auth_request con 0x7f297803e260 auth_method 0 2024-09-11T01:33:57.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.425+0000 7f298affd640 1 -- 172.21.15.12:0/898080103 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c136570 msgr2=0x7f298c1395c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:57.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.425+0000 7f298affd640 1 --2- 172.21.15.12:0/898080103 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c136570 0x7f298c1395c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f297c0072e0 tx=0x7f297c0077a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:57.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.425+0000 7f29737fe640 1 -- 172.21.15.12:0/898080103 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f297c00fd40 con 0x7f298c136570 2024-09-11T01:33:57.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.425+0000 7f29737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:57.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.425+0000 7f29737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:57.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.425+0000 7f29737fe640 10 monclient: dump: 2024-09-11T01:33:57.442 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:57.442 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:57.442 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.442 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.442 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:57.442 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.442 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:57.442 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:57.442 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:57.442 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.426+0000 7f29737fe640 1 -- 172.21.15.12:0/898080103 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f297c00fd40 con 0x7f298c136570 2024-09-11T01:33:57.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.426+0000 7f29737fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:57.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.426+0000 7f29737fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:33:57.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.426+0000 7f29737fe640 1 -- 172.21.15.12:0/898080103 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c136570 msgr2=0x7f298c1395c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:57.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.426+0000 7f29737fe640 1 --2- 172.21.15.12:0/898080103 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c136570 0x7f298c1395c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f297c0072e0 tx=0x7f297c0077a0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.426+0000 7f29737fe640 1 -- 172.21.15.12:0/898080103 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c136570 msgr2=0x7f298c1395c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:33:57.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.426+0000 7f29737fe640 1 --2- 172.21.15.12:0/898080103 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c136570 0x7f298c1395c0 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f297c0072e0 tx=0x7f297c0077a0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.426+0000 7f298a7fc640 1 --2- 172.21.15.12:0/898080103 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f297803e260 0x7f2978040720 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f298c13cf10 tx=0x7f2980007650 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:57.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.432+0000 7f2991394640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:33:57.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.432+0000 7f2991394640 1 --2- 172.21.15.12:0/898080103 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c139b00 0x7f298c13da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.432+0000 7f2991394640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.432+0000 7f2991394640 1 -- 172.21.15.12:0/898080103 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f298c139f10 con 0x7f298c139b00 2024-09-11T01:33:57.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.432+0000 7f298affd640 1 --2- 172.21.15.12:0/898080103 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c139b00 0x7f298c13da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.432+0000 7f298affd640 10 monclient: get_auth_request con 0x7f298c139b00 auth_method 0 2024-09-11T01:33:57.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.432+0000 7f298affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:57.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.432+0000 7f298affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:57.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.432+0000 7f298affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:57.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.432+0000 7f298affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:57.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.432+0000 7f298affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:57.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.432+0000 7f298affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:57.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.432+0000 7f298affd640 10 monclient(hunting): handle_auth_done global_id 44216 payload 824 2024-09-11T01:33:57.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.432+0000 7f298affd640 1 -- 172.21.15.12:0/898080103 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f298c0a1710 con 0x7f298c139b00 2024-09-11T01:33:57.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.434+0000 7f298affd640 1 -- 172.21.15.12:0/898080103 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c139b00 msgr2=0x7f298c13da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:57.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.434+0000 7f298affd640 1 --2- 172.21.15.12:0/898080103 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c139b00 0x7f298c13da80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f297c0072e0 tx=0x7f297c004940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:57.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.435+0000 7f29737fe640 1 -- 172.21.15.12:0/898080103 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f297c01ee40 con 0x7f298c139b00 2024-09-11T01:33:57.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.435+0000 7f29737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:57.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.435+0000 7f29737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:57.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.435+0000 7f29737fe640 10 monclient: dump: 2024-09-11T01:33:57.444 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:57.445 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:57.445 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.445 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.445 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:57.445 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.445 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:57.445 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:57.445 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:57.445 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.435+0000 7f29737fe640 1 -- 172.21.15.12:0/898080103 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4540 (secure 0 0 0) 0x7f297c006240 con 0x7f298c139b00 2024-09-11T01:33:57.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.435+0000 7f29737fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:33:57.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.435+0000 7f29737fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:33:57.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.435+0000 7f29737fe640 1 -- 172.21.15.12:0/898080103 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c139b00 msgr2=0x7f298c13da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:57.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.435+0000 7f29737fe640 1 --2- 172.21.15.12:0/898080103 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c139b00 0x7f298c13da80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f297c0072e0 tx=0x7f297c004940 comp rx=0 tx=0).stop 2024-09-11T01:33:57.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.435+0000 7f29737fe640 1 -- 172.21.15.12:0/898080103 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c139b00 msgr2=0x7f298c13da80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:33:57.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.435+0000 7f29737fe640 1 --2- 172.21.15.12:0/898080103 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c139b00 0x7f298c13da80 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f297c0072e0 tx=0x7f297c004940 comp rx=0 tx=0).stop 2024-09-11T01:33:57.446 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:33:57.446 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:33:57.446 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:33:57.446 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:33:57.446 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 56, 2024-09-11T01:33:57.446 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 81380, 2024-09-11T01:33:57.446 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:33:57.446 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:33:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:33:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:33:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:33:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:33:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:33:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:33:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:33:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:33:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:33:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:33:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:33:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:33:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:33:57.447 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:33:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:33:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:33:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:33:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:33:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:33:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:33:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:33:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:33:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:33:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:33:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:33:57.448 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:33:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:33:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:33:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:33:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:33:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:33:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:33:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:33:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:33:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:33:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:33:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:33:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:33:57.449 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:33:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:33:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:33:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:33:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:33:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:33:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:33:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:33:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:33:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:33:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:33:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:33:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:33:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:33:57.450 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:33:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:33:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:33:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:33:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:33:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:33:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:33:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:33:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:33:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:33:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:33:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:33:57.451 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:33:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:33:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:33:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:33:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:33:57.452 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:33:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:33:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:33:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:33:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:33:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:33:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:33:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:33:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:33:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:33:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:57.453 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:33:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:33:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:33:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:33:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:33:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:33:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:33:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.454 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:33:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:33:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:33:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:33:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:33:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:33:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:33:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:33:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:57.455 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:33:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:33:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:33:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:33:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:33:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:33:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:33:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.456 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:33:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:33:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:33:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:33:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:33:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:33:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:33:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.457 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:33:57.458 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:33:57.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.435+0000 7f2991394640 1 -- 172.21.15.12:0/898080103 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f297803e260 msgr2=0x7f2978040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:57.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.435+0000 7f2991394640 1 --2- 172.21.15.12:0/898080103 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f297803e260 0x7f2978040720 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f298c13cf10 tx=0x7f2980007650 comp rx=0 tx=0).stop 2024-09-11T01:33:57.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.435+0000 7f2991394640 10 monclient: shutdown 2024-09-11T01:33:57.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.435+0000 7f2991394640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:57.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.435+0000 7f2991394640 1 -- 172.21.15.12:0/898080103 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c13d000 msgr2=0x7f298c139080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:57.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.435+0000 7f2991394640 1 --2- 172.21.15.12:0/898080103 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c13d000 0x7f298c139080 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2984034000 tx=0x7f29840046c0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.435+0000 7f298b7fe640 1 -- 172.21.15.12:0/898080103 reap_dead start 2024-09-11T01:33:57.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.435+0000 7f2991394640 1 -- 172.21.15.12:0/898080103 shutdown_connections 2024-09-11T01:33:57.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.435+0000 7f2991394640 1 --2- 172.21.15.12:0/898080103 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f298c13d000 0x7f298c139080 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.435+0000 7f2991394640 1 -- 172.21.15.12:0/898080103 >> 172.21.15.12:0/898080103 conn(0x7f298c0a7440 msgr2=0x7f298c0a4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:57.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.435+0000 7f2991394640 1 -- 172.21.15.12:0/898080103 shutdown_connections 2024-09-11T01:33:57.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.435+0000 7f2991394640 1 -- 172.21.15.12:0/898080103 wait complete. 2024-09-11T01:33:57.460 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:33:57.460 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:33:57.460 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:33:57.535 INFO:tasks.rados.rados.0.smithi071.stdout:752: done (1 left) 2024-09-11T01:33:57.535 INFO:tasks.rados.rados.0.smithi071.stdout:753: write oid 47 current snap is 74 2024-09-11T01:33:57.535 INFO:tasks.rados.rados.0.smithi071.stdout:753: seq_num 286 ranges {406802=626275,1686203=728431} 2024-09-11T01:33:57.541 INFO:tasks.rados.rados.0.smithi071.stdout:753: writing smithi07134777-47 from 406802 to 1033077 tid 1 2024-09-11T01:33:57.550 INFO:tasks.rados.rados.0.smithi071.stdout:753: writing smithi07134777-47 from 1686203 to 2414634 tid 2 2024-09-11T01:33:57.550 INFO:tasks.rados.rados.0.smithi071.stdout:754: read oid 20 snap -1 2024-09-11T01:33:57.550 INFO:tasks.rados.rados.0.smithi071.stdout:754: expect (ObjNum 114 snap 27 seq_num 114) 2024-09-11T01:33:57.550 INFO:tasks.rados.rados.0.smithi071.stdout:755: write oid 50 current snap is 74 2024-09-11T01:33:57.550 INFO:tasks.rados.rados.0.smithi071.stdout:755: seq_num 287 ranges {423609=701524,1628332=472089,2648987=174622} 2024-09-11T01:33:57.557 INFO:tasks.rados.rados.0.smithi071.stdout:755: writing smithi07134777-50 from 423609 to 1125133 tid 1 2024-09-11T01:33:57.563 INFO:tasks.rados.rados.0.smithi071.stdout:755: writing smithi07134777-50 from 1628332 to 2100421 tid 2 2024-09-11T01:33:57.568 INFO:tasks.rados.rados.0.smithi071.stdout:755: writing smithi07134777-50 from 2648987 to 2823609 tid 3 2024-09-11T01:33:57.568 INFO:tasks.rados.rados.0.smithi071.stdout:756: copy_from oid 49 from oid 43 current snap is 74 2024-09-11T01:33:57.568 INFO:tasks.rados.rados.0.smithi071.stdout:750: finishing write tid 1 to smithi07134777-42 2024-09-11T01:33:57.569 INFO:tasks.rados.rados.0.smithi071.stdout:750: finishing write tid 2 to smithi07134777-42 2024-09-11T01:33:57.569 INFO:tasks.rados.rados.0.smithi071.stdout:750: finishing write tid 3 to smithi07134777-42 2024-09-11T01:33:57.569 INFO:tasks.rados.rados.0.smithi071.stdout:750: finishing write tid 4 to smithi07134777-42 2024-09-11T01:33:57.569 INFO:tasks.rados.rados.0.smithi071.stdout:750: oid 42 updating version 0 to 135 2024-09-11T01:33:57.569 INFO:tasks.rados.rados.0.smithi071.stdout:750: oid 42 version 135 is already newer than 134 2024-09-11T01:33:57.569 INFO:tasks.rados.rados.0.smithi071.stdout:750: oid 42 version 135 is already newer than 133 2024-09-11T01:33:57.569 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 135 (ObjNum 285 snap 74 seq_num 285) dirty exists 2024-09-11T01:33:57.569 INFO:tasks.rados.rados.0.smithi071.stdout:750: left oid 42 (ObjNum 285 snap 74 seq_num 285) 2024-09-11T01:33:57.569 INFO:tasks.rados.rados.0.smithi071.stdout:753: finishing write tid 1 to smithi07134777-47 2024-09-11T01:33:57.569 INFO:tasks.rados.rados.0.smithi071.stdout:753: finishing write tid 2 to smithi07134777-47 2024-09-11T01:33:57.569 INFO:tasks.rados.rados.0.smithi071.stdout:750: done (4 left) 2024-09-11T01:33:57.569 INFO:tasks.rados.rados.0.smithi071.stdout:757: copy_from oid 3 from oid 28 current snap is 74 2024-09-11T01:33:57.569 INFO:tasks.rados.rados.0.smithi071.stdout:753: finishing write tid 3 to smithi07134777-47 2024-09-11T01:33:57.569 INFO:tasks.rados.rados.0.smithi071.stdout:753: finishing write tid 4 to smithi07134777-47 2024-09-11T01:33:57.570 INFO:tasks.rados.rados.0.smithi071.stdout:753: oid 47 updating version 0 to 297 2024-09-11T01:33:57.570 INFO:tasks.rados.rados.0.smithi071.stdout:753: oid 47 updating version 297 to 299 2024-09-11T01:33:57.570 INFO:tasks.rados.rados.0.smithi071.stdout:753: oid 47 version 299 is already newer than 298 2024-09-11T01:33:57.570 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 299 (ObjNum 286 snap 74 seq_num 286) dirty exists 2024-09-11T01:33:57.570 INFO:tasks.rados.rados.0.smithi071.stdout:753: left oid 47 (ObjNum 286 snap 74 seq_num 286) 2024-09-11T01:33:57.570 INFO:tasks.rados.rados.0.smithi071.stdout:754: expect (ObjNum 114 snap 27 seq_num 114) 2024-09-11T01:33:57.570 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:33:57.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.569+0000 7f529f94b640 10 monclient: get_monmap_and_config 2024-09-11T01:33:57.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.569+0000 7f529f94b640 10 monclient: build_initial_monmap 2024-09-11T01:33:57.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.569+0000 7f529f94b640 10 monclient: monmap: 2024-09-11T01:33:57.571 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:57.571 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:57.571 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:33:57.570495+0000 2024-09-11T01:33:57.571 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:33:57.570495+0000 2024-09-11T01:33:57.572 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:57.572 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.572 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:33:57.572 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:33:57.572 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:33:57.572 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:33:57.572 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:33:57.572 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.570+0000 7f529f94b640 1 Processor -- start 2024-09-11T01:33:57.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.570+0000 7f529f94b640 1 -- start start 2024-09-11T01:33:57.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.570+0000 7f529f94b640 10 monclient: init 2024-09-11T01:33:57.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529f94b640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:57.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529f94b640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-11T01:33:57.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529f94b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5298137e50 addr v1:172.21.15.12:6789/0 2024-09-11T01:33:57.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529f94b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5298138230 0x7f5298138660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529f94b640 10 monclient(hunting): picked mon.noname-b con 0x7f5298138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:57.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529f94b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f529813d380 0x7f529813f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529f94b640 10 monclient(hunting): picked mon.noname-a con 0x7f529813d380 addr v2:172.21.15.12:3300/0 2024-09-11T01:33:57.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529f94b640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52980a6580 con 0x7f5298137e50 2024-09-11T01:33:57.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529f94b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529f94b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f52980a62d0 con 0x7f529813d380 2024-09-11T01:33:57.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529f94b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529f94b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f529809bb40 con 0x7f5298138230 2024-09-11T01:33:57.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529f94b640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:57.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529f94b640 10 monclient(hunting): authenticate will time out at 1044.013556s 2024-09-11T01:33:57.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529d6e7640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5298137e50 0x7f529809d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33004/0 (socket says 172.21.15.12:33004) 2024-09-11T01:33:57.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529d6e7640 1 -- 172.21.15.12:0/4086918049 learned_addr learned my addr 172.21.15.12:0/4086918049 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:33:57.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529cee6640 1 --2- 172.21.15.12:0/4086918049 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5298138230 0x7f5298138660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529dee8640 1 --2- 172.21.15.12:0/4086918049 >> v2:172.21.15.12:3300/0 conn(0x7f529813d380 0x7f529813f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529dee8640 10 monclient(hunting): get_auth_request con 0x7f529813d380 auth_method 0 2024-09-11T01:33:57.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529dee8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:57.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529dee8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:57.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529dee8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:57.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529cee6640 10 monclient(hunting): get_auth_request con 0x7f5298138230 auth_method 0 2024-09-11T01:33:57.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529cee6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:57.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529cee6640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:57.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529cee6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:57.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529cee6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:57.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529cee6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:57.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529cee6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:57.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529dee8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:57.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529dee8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:57.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529dee8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:57.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529cee6640 10 monclient(hunting): handle_auth_done global_id 44219 payload 274 2024-09-11T01:33:57.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.571+0000 7f529cee6640 1 -- 172.21.15.12:0/4086918049 >> v2:172.21.15.12:3300/0 conn(0x7f529813d380 msgr2=0x7f529813f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:57.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.572+0000 7f529cee6640 1 --2- 172.21.15.12:0/4086918049 >> v2:172.21.15.12:3300/0 conn(0x7f529813d380 0x7f529813f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.572+0000 7f529cee6640 1 -- 172.21.15.12:0/4086918049 >> v1:172.21.15.12:6789/0 conn(0x7f5298137e50 legacy=0x7f529809d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:57.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.572+0000 7f529cee6640 10 monclient: _finish_hunting 0 2024-09-11T01:33:57.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.572+0000 7f529cee6640 1 monclient: found mon.noname-b 2024-09-11T01:33:57.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.572+0000 7f529cee6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:57.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.572+0000 7f529cee6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:33:57.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.572+0000 7f529cee6640 1 -- 172.21.15.12:0/4086918049 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f529809c4a0 con 0x7f5298138230 2024-09-11T01:33:57.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.572+0000 7f529dee8640 1 --2- 172.21.15.12:0/4086918049 >> v2:172.21.15.12:3300/0 conn(0x7f529813d380 0x7f529813f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:33:57.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.574+0000 7f529cee6640 1 -- 172.21.15.12:0/4086918049 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5298138230 msgr2=0x7f5298138660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:57.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.574+0000 7f529cee6640 1 --2- 172.21.15.12:0/4086918049 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5298138230 0x7f5298138660 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f52800097a0 tx=0x7f5280029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aedbc4a0012776cd server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:57.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.575+0000 7f5297fff640 1 -- 172.21.15.12:0/4086918049 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5280037070 con 0x7f5298138230 2024-09-11T01:33:57.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.575+0000 7f5297fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:57.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.575+0000 7f5297fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:33:57.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.575+0000 7f5297fff640 10 monclient: dump: 2024-09-11T01:33:57.592 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:57.592 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:57.592 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.592 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.592 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:57.593 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.593 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:57.593 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:57.593 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:57.593 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.575+0000 7f5297fff640 10 monclient: _finish_auth 0 2024-09-11T01:33:57.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.575+0000 7f5297fff640 10 monclient: _check_auth_tickets 2024-09-11T01:33:57.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.575+0000 7f5297fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:57.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.575+0000 7f5297fff640 1 -- 172.21.15.12:0/4086918049 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5280032440 con 0x7f5298138230 2024-09-11T01:33:57.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.575+0000 7f529f94b640 5 monclient: authenticate success, global_id 44219 2024-09-11T01:33:57.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.575+0000 7f529f94b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:33:57.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.575+0000 7f5297fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:57.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.575+0000 7f529f94b640 10 monclient: get_monmap_and_config success 2024-09-11T01:33:57.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.575+0000 7f5297fff640 1 -- 172.21.15.12:0/4086918049 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5280032810 con 0x7f5298138230 2024-09-11T01:33:57.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.575+0000 7f5297fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:57.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.575+0000 7f529f94b640 4 set_mon_vals no callback set 2024-09-11T01:33:57.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.575+0000 7f5297fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:57.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.575+0000 7f5297fff640 10 monclient: dump: 2024-09-11T01:33:57.595 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:57.596 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:57.596 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.596 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.596 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:57.596 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.596 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:57.597 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:57.597 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:57.597 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.576+0000 7f529f94b640 10 monclient: shutdown 2024-09-11T01:33:57.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.576+0000 7f529f94b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:57.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.576+0000 7f529f94b640 1 -- 172.21.15.12:0/4086918049 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5298138230 msgr2=0x7f5298138660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:57.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.576+0000 7f529f94b640 1 --2- 172.21.15.12:0/4086918049 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5298138230 0x7f5298138660 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f52800097a0 tx=0x7f5280029fd0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.576+0000 7f529f94b640 1 -- 172.21.15.12:0/4086918049 shutdown_connections 2024-09-11T01:33:57.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.576+0000 7f529f94b640 1 --2- 172.21.15.12:0/4086918049 >> v2:172.21.15.12:3300/0 conn(0x7f529813d380 0x7f529813f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.576+0000 7f529f94b640 1 --2- 172.21.15.12:0/4086918049 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5298138230 0x7f5298138660 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.576+0000 7f529f94b640 1 -- 172.21.15.12:0/4086918049 >> 172.21.15.12:0/4086918049 conn(0x7f52980a7020 msgr2=0x7f52980a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:57.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.576+0000 7f529f94b640 1 -- 172.21.15.12:0/4086918049 shutdown_connections 2024-09-11T01:33:57.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.576+0000 7f529f94b640 1 -- 172.21.15.12:0/4086918049 wait complete. 2024-09-11T01:33:57.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.577+0000 7f529f94b640 10 monclient: build_initial_monmap 2024-09-11T01:33:57.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.577+0000 7f529f94b640 10 monclient: monmap: 2024-09-11T01:33:57.599 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:57.599 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:33:57.599 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:33:57.599 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:33:57.600 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:57.600 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.600 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:33:57.600 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:33:57.600 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:33:57.600 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.577+0000 7f529f94b640 1 Processor -- start 2024-09-11T01:33:57.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.577+0000 7f529f94b640 1 -- start start 2024-09-11T01:33:57.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.577+0000 7f529f94b640 10 monclient: init 2024-09-11T01:33:57.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529f94b640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:57.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529f94b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:33:57.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529f94b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5298137e50 0x7f52981ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529f94b640 10 monclient(hunting): picked mon.noname-a con 0x7f5298137e50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:33:57.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529f94b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5298138230 0x7f5298136f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529f94b640 10 monclient(hunting): picked mon.noname-b con 0x7f5298138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:33:57.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529f94b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f529813d380 0x7f5298139650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529f94b640 10 monclient(hunting): picked mon.noname-c con 0x7f529813d380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:57.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529f94b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529f94b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f529806f890 con 0x7f5298137e50 2024-09-11T01:33:57.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529f94b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529f94b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f529806f710 con 0x7f5298138230 2024-09-11T01:33:57.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529f94b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529f94b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f529813fcb0 con 0x7f529813d380 2024-09-11T01:33:57.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529f94b640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:57.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529f94b640 10 monclient(hunting): authenticate will time out at 1044.020124s 2024-09-11T01:33:57.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529dee8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f529813d380 0x7f5298139650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529d6e7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5298137e50 0x7f52981ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529dee8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f529813d380 0x7f5298139650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:37264/0 (socket says 172.21.15.12:37264) 2024-09-11T01:33:57.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529dee8640 1 -- 172.21.15.12:0/2086036941 learned_addr learned my addr 172.21.15.12:0/2086036941 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:57.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529dee8640 10 monclient(hunting): get_auth_request con 0x7f529813d380 auth_method 0 2024-09-11T01:33:57.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529dee8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:57.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529dee8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:57.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529dee8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:57.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529d6e7640 10 monclient(hunting): get_auth_request con 0x7f5298137e50 auth_method 0 2024-09-11T01:33:57.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529d6e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:57.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529d6e7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:57.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529d6e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:57.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529dee8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:57.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529dee8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:57.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529dee8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:57.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529d6e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:57.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529d6e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:57.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529d6e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:57.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529cee6640 1 --2- 172.21.15.12:0/2086036941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5298138230 0x7f5298136f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529cee6640 10 monclient(hunting): get_auth_request con 0x7f5298138230 auth_method 0 2024-09-11T01:33:57.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529cee6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:57.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529cee6640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:57.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529cee6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:57.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529cee6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:57.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529cee6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:57.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529cee6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:57.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529dee8640 10 monclient(hunting): handle_auth_done global_id 44222 payload 824 2024-09-11T01:33:57.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529dee8640 1 -- 172.21.15.12:0/2086036941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5298138230 msgr2=0x7f5298136f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:57.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529dee8640 1 --2- 172.21.15.12:0/2086036941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5298138230 0x7f5298136f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529dee8640 1 -- 172.21.15.12:0/2086036941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5298137e50 msgr2=0x7f52981ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:57.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529dee8640 1 --2- 172.21.15.12:0/2086036941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5298137e50 0x7f52981ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529dee8640 10 monclient: _finish_hunting 0 2024-09-11T01:33:57.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529dee8640 1 monclient: found mon.noname-c 2024-09-11T01:33:57.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529dee8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:57.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529dee8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:33:57.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529dee8640 1 -- 172.21.15.12:0/2086036941 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f529813afd0 con 0x7f529813d380 2024-09-11T01:33:57.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.578+0000 7f529d6e7640 1 --2- 172.21.15.12:0/2086036941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5298137e50 0x7f52981ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:33:57.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.580+0000 7f529dee8640 1 -- 172.21.15.12:0/2086036941 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f529813d380 msgr2=0x7f5298139650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:57.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.580+0000 7f529dee8640 1 --2- 172.21.15.12:0/2086036941 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f529813d380 0x7f5298139650 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f529000efc0 tx=0x7f5290009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:57.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.581+0000 7f5295ffb640 1 -- 172.21.15.12:0/2086036941 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5290018030 con 0x7f529813d380 2024-09-11T01:33:57.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.581+0000 7f5295ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:57.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.581+0000 7f5295ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:33:57.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.581+0000 7f5295ffb640 10 monclient: dump: 2024-09-11T01:33:57.610 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:57.610 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:57.610 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.610 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.611 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:57.611 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.611 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:57.611 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:57.611 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:57.611 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.581+0000 7f5295ffb640 10 monclient: _finish_auth 0 2024-09-11T01:33:57.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.581+0000 7f5295ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:33:57.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.581+0000 7f5295ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:57.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.581+0000 7f529f94b640 5 monclient: authenticate success, global_id 44222 2024-09-11T01:33:57.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.581+0000 7f5295ffb640 1 -- 172.21.15.12:0/2086036941 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5290003990 con 0x7f529813d380 2024-09-11T01:33:57.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.581+0000 7f529f94b640 10 monclient: _renew_subs 2024-09-11T01:33:57.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.581+0000 7f529f94b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:57.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.581+0000 7f529f94b640 1 -- 172.21.15.12:0/2086036941 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52980a1710 con 0x7f529813d380 2024-09-11T01:33:57.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.581+0000 7f5295ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:57.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.581+0000 7f5295ffb640 1 -- 172.21.15.12:0/2086036941 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5290003d60 con 0x7f529813d380 2024-09-11T01:33:57.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.581+0000 7f5295ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:57.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.581+0000 7f5295ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:57.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.581+0000 7f5295ffb640 10 monclient: dump: 2024-09-11T01:33:57.613 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:57.613 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:57.614 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.614 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.614 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:57.614 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.614 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:57.614 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:57.614 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:57.615 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.581+0000 7f52977fe640 4 set_mon_vals no callback set 2024-09-11T01:33:57.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.581+0000 7f529f94b640 10 monclient: _renew_subs 2024-09-11T01:33:57.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.581+0000 7f529f94b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:57.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.581+0000 7f529f94b640 1 -- 172.21.15.12:0/2086036941 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5298121060 con 0x7f529813d380 2024-09-11T01:33:57.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.582+0000 7f5295ffb640 1 -- 172.21.15.12:0/2086036941 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f5290004230 con 0x7f529813d380 2024-09-11T01:33:57.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.582+0000 7f529f94b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:57.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.582+0000 7f529f94b640 1 --2- 172.21.15.12:0/2086036941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52981374b0 0x7f5298139b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.582+0000 7f529f94b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.582+0000 7f529f94b640 1 -- 172.21.15.12:0/2086036941 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f529806a060 con 0x7f52981374b0 2024-09-11T01:33:57.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.582+0000 7f5295ffb640 1 --2- 172.21.15.12:0/2086036941 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f527803e210 0x7f52780406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.582+0000 7f5295ffb640 1 -- 172.21.15.12:0/2086036941 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(171..171 src has 1..171) ==== 4815+0+0 (secure 0 0 0) 0x7f5290053d90 con 0x7f529813d380 2024-09-11T01:33:57.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.582+0000 7f529d6e7640 1 --2- 172.21.15.12:0/2086036941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52981374b0 0x7f5298139b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.582+0000 7f529d6e7640 10 monclient: get_auth_request con 0x7f52981374b0 auth_method 0 2024-09-11T01:33:57.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.582+0000 7f529d6e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:57.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.582+0000 7f529d6e7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:57.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.582+0000 7f529d6e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:57.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.582+0000 7f529cee6640 1 --2- 172.21.15.12:0/2086036941 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f527803e210 0x7f52780406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.582+0000 7f529d6e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:57.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.582+0000 7f529d6e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:57.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.582+0000 7f529d6e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:57.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.582+0000 7f529cee6640 10 monclient: get_auth_request con 0x7f527803e210 auth_method 0 2024-09-11T01:33:57.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.583+0000 7f529d6e7640 10 monclient(hunting): handle_auth_done global_id 34297 payload 824 2024-09-11T01:33:57.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.583+0000 7f529d6e7640 1 -- 172.21.15.12:0/2086036941 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5298138660 con 0x7f52981374b0 2024-09-11T01:33:57.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.585+0000 7f529d6e7640 1 -- 172.21.15.12:0/2086036941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52981374b0 msgr2=0x7f5298139b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:57.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.585+0000 7f529d6e7640 1 --2- 172.21.15.12:0/2086036941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52981374b0 0x7f5298139b90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5288007fd0 tx=0x7f5288007700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:57.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.585+0000 7f529cee6640 1 --2- 172.21.15.12:0/2086036941 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f527803e210 0x7f52780406d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f52800097a0 tx=0x7f528002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:57.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.585+0000 7f5295ffb640 1 -- 172.21.15.12:0/2086036941 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5288017030 con 0x7f52981374b0 2024-09-11T01:33:57.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.585+0000 7f5295ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:57.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.585+0000 7f5295ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:33:57.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.585+0000 7f5295ffb640 10 monclient: dump: 2024-09-11T01:33:57.619 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:57.620 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:57.620 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.620 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.620 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:57.620 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.620 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:57.621 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:57.621 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:57.621 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.585+0000 7f5295ffb640 1 -- 172.21.15.12:0/2086036941 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f528800fc50 con 0x7f52981374b0 2024-09-11T01:33:57.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.585+0000 7f5295ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:57.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.585+0000 7f5295ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:33:57.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.585+0000 7f5295ffb640 1 -- 172.21.15.12:0/2086036941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52981374b0 msgr2=0x7f5298139b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:57.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.585+0000 7f5295ffb640 1 --2- 172.21.15.12:0/2086036941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52981374b0 0x7f5298139b90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5288007fd0 tx=0x7f5288007700 comp rx=0 tx=0).stop 2024-09-11T01:33:57.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.585+0000 7f5295ffb640 1 -- 172.21.15.12:0/2086036941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52981374b0 msgr2=0x7f5298139b90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:33:57.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.585+0000 7f5295ffb640 1 --2- 172.21.15.12:0/2086036941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52981374b0 0x7f5298139b90 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5288007fd0 tx=0x7f5288007700 comp rx=0 tx=0).stop 2024-09-11T01:33:57.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.592+0000 7f529f94b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:33:57.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.592+0000 7f529f94b640 1 --2- 172.21.15.12:0/2086036941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f529813a0d0 0x7f529813ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.592+0000 7f529f94b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.592+0000 7f529f94b640 1 -- 172.21.15.12:0/2086036941 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f529813a4e0 con 0x7f529813a0d0 2024-09-11T01:33:57.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.592+0000 7f529d6e7640 1 --2- 172.21.15.12:0/2086036941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f529813a0d0 0x7f529813ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.592+0000 7f529d6e7640 10 monclient: get_auth_request con 0x7f529813a0d0 auth_method 0 2024-09-11T01:33:57.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.592+0000 7f529d6e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:57.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.592+0000 7f529d6e7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:57.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.592+0000 7f529d6e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:57.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.592+0000 7f529d6e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:57.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.592+0000 7f529d6e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:57.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.592+0000 7f529d6e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:57.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.592+0000 7f529d6e7640 10 monclient(hunting): handle_auth_done global_id 34300 payload 824 2024-09-11T01:33:57.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.592+0000 7f529d6e7640 1 -- 172.21.15.12:0/2086036941 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f529813bb30 con 0x7f529813a0d0 2024-09-11T01:33:57.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.594+0000 7f529d6e7640 1 -- 172.21.15.12:0/2086036941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f529813a0d0 msgr2=0x7f529813ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:57.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.594+0000 7f529d6e7640 1 --2- 172.21.15.12:0/2086036941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f529813a0d0 0x7f529813ddb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5288007fd0 tx=0x7f5288004940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:57.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.595+0000 7f5295ffb640 1 -- 172.21.15.12:0/2086036941 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5288005fc0 con 0x7f529813a0d0 2024-09-11T01:33:57.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.595+0000 7f5295ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:57.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.595+0000 7f5295ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:33:57.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.595+0000 7f5295ffb640 10 monclient: dump: 2024-09-11T01:33:57.626 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:57.627 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:57.627 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.627 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.627 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:57.627 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.627 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:57.627 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:57.627 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:57.627 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.595+0000 7f5295ffb640 1 -- 172.21.15.12:0/2086036941 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7f5288005fc0 con 0x7f529813a0d0 2024-09-11T01:33:57.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.595+0000 7f5295ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:33:57.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.595+0000 7f5295ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:33:57.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.595+0000 7f5295ffb640 1 -- 172.21.15.12:0/2086036941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f529813a0d0 msgr2=0x7f529813ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:57.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.595+0000 7f5295ffb640 1 --2- 172.21.15.12:0/2086036941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f529813a0d0 0x7f529813ddb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5288007fd0 tx=0x7f5288004940 comp rx=0 tx=0).stop 2024-09-11T01:33:57.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.595+0000 7f5295ffb640 1 -- 172.21.15.12:0/2086036941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f529813a0d0 msgr2=0x7f529813ddb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:33:57.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.595+0000 7f5295ffb640 1 --2- 172.21.15.12:0/2086036941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f529813a0d0 0x7f529813ddb0 secure :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5288007fd0 tx=0x7f5288004940 comp rx=0 tx=0).stop 2024-09-11T01:33:57.628 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:33:57.628 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:33:57.628 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:33:57.628 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:33:57.628 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 56, 2024-09-11T01:33:57.628 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 130808, 2024-09-11T01:33:57.628 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:33:57.629 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:33:57.629 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:33:57.629 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:33:57.629 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:57.629 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:33:57.629 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:33:57.629 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:33:57.629 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:33:57.629 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:33:57.629 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:33:57.629 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:33:57.629 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:33:57.629 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:33:57.630 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:33:57.630 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:33:57.630 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:33:57.630 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:33:57.630 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:33:57.630 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:33:57.630 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:57.630 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:33:57.630 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:33:57.630 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:33:57.630 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:33:57.630 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:33:57.630 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:33:57.630 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:33:57.631 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:33:57.631 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:33:57.631 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:33:57.631 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:33:57.631 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:33:57.631 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:33:57.631 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:57.631 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.631 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:33:57.631 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:33:57.631 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:33:57.631 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:33:57.632 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:33:57.632 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:33:57.632 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:33:57.632 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:33:57.632 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:33:57.632 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:33:57.632 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:33:57.632 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:33:57.632 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:33:57.632 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:33:57.632 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:33:57.632 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:33:57.632 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:33:57.633 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:33:57.633 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:33:57.633 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:33:57.633 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:33:57.633 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:33:57.633 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:33:57.633 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:33:57.633 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:33:57.633 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:33:57.633 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:33:57.634 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:33:57.634 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:57.634 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:33:57.634 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.634 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:33:57.634 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.634 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:33:57.634 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:33:57.634 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:33:57.634 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:33:57.634 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.634 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:33:57.634 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:33:57.634 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:57.635 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.635 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.635 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:33:57.635 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:33:57.635 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:57.635 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:57.635 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:57.635 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.635 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:33:57.635 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:33:57.635 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:33:57.635 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:33:57.635 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:33:57.636 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.636 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.636 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:33:57.636 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:33:57.636 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:33:57.636 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:33:57.636 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.636 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:33:57.636 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:33:57.636 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:57.636 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.636 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.636 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:33:57.636 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:33:57.637 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:57.637 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:57.637 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:57.637 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.637 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:33:57.637 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:33:57.637 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:33:57.637 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:33:57.637 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:33:57.637 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.637 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.637 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:33:57.637 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:33:57.638 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:33:57.638 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:33:57.638 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.638 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:33:57.638 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:33:57.638 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:57.638 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.638 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.638 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:33:57.638 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:33:57.638 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:57.638 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:57.638 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:57.638 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.639 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:33:57.639 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:33:57.639 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:33:57.639 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:33:57.639 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:33:57.639 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:57.639 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:57.639 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.639 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:33:57.639 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:33:57.639 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.639 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:33:57.639 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:33:57.639 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:33:57.640 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:57.640 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:57.640 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:33:57.640 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.640 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:33:57.640 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:33:57.640 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:33:57.640 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:57.640 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:57.640 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:33:57.640 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.640 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:33:57.640 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:33:57.641 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:33:57.641 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:57.641 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:57.641 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:33:57.641 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:57.641 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:33:57.641 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:33:57.641 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:33:57.641 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:57.642 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:57.642 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:57.642 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:33:57.642 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:33:57.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.595+0000 7f529f94b640 1 -- 172.21.15.12:0/2086036941 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f527803e210 msgr2=0x7f52780406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:57.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.595+0000 7f529f94b640 1 --2- 172.21.15.12:0/2086036941 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f527803e210 0x7f52780406d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f52800097a0 tx=0x7f528002e000 comp rx=0 tx=0).stop 2024-09-11T01:33:57.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.595+0000 7f529f94b640 10 monclient: shutdown 2024-09-11T01:33:57.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.595+0000 7f529f94b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:57.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.595+0000 7f529f94b640 1 -- 172.21.15.12:0/2086036941 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f529813d380 msgr2=0x7f5298139650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:57.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.595+0000 7f529f94b640 1 --2- 172.21.15.12:0/2086036941 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f529813d380 0x7f5298139650 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f529000efc0 tx=0x7f5290009f90 comp rx=0 tx=0).stop 2024-09-11T01:33:57.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.595+0000 7f529dee8640 1 -- 172.21.15.12:0/2086036941 reap_dead start 2024-09-11T01:33:57.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.595+0000 7f529dee8640 1 -- 172.21.15.12:0/2086036941 reap_dead start 2024-09-11T01:33:57.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.595+0000 7f529f94b640 1 -- 172.21.15.12:0/2086036941 shutdown_connections 2024-09-11T01:33:57.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.595+0000 7f529f94b640 1 -- 172.21.15.12:0/2086036941 >> 172.21.15.12:0/2086036941 conn(0x7f52980a7020 msgr2=0x7f52980a8420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:57.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.595+0000 7f529f94b640 1 -- 172.21.15.12:0/2086036941 shutdown_connections 2024-09-11T01:33:57.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.595+0000 7f529f94b640 1 -- 172.21.15.12:0/2086036941 wait complete. 2024-09-11T01:33:57.644 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:33:57.644 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:33:57.645 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:33:57.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.743+0000 7f0587eb6640 10 monclient: get_monmap_and_config 2024-09-11T01:33:57.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.743+0000 7f0587eb6640 10 monclient: build_initial_monmap 2024-09-11T01:33:57.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.743+0000 7f0587eb6640 10 monclient: monmap: 2024-09-11T01:33:57.745 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:57.745 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:57.745 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:33:57.744075+0000 2024-09-11T01:33:57.745 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:33:57.744075+0000 2024-09-11T01:33:57.745 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:57.745 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.745 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:33:57.745 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:33:57.746 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:33:57.746 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:33:57.746 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:33:57.746 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.743+0000 7f0587eb6640 1 Processor -- start 2024-09-11T01:33:57.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0587eb6640 1 -- start start 2024-09-11T01:33:57.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0587eb6640 10 monclient: init 2024-09-11T01:33:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0587eb6640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0587eb6640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-11T01:33:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0587eb6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f05800a1380 0x7f05800a1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0587eb6640 10 monclient(hunting): picked mon.noname-c con 0x7f05800a1380 addr v2:172.21.15.71:3300/0 2024-09-11T01:33:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0587eb6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f05800a0370 addr v1:172.21.15.71:6789/0 2024-09-11T01:33:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0587eb6640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f05800a0c90 0x7f05801361f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0587eb6640 10 monclient(hunting): picked mon.noname-a con 0x7f05800a0c90 addr v2:172.21.15.12:3300/0 2024-09-11T01:33:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0587eb6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05800a1ca0 con 0x7f05800a0370 2024-09-11T01:33:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0587eb6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0587eb6640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f05800a21c0 con 0x7f05800a0c90 2024-09-11T01:33:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0587eb6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0587eb6640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f05800a24a0 con 0x7f05800a1380 2024-09-11T01:33:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0587eb6640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0587eb6640 10 monclient(hunting): authenticate will time out at 1044.186694s 2024-09-11T01:33:57.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0586453640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f05800a0c90 0x7f05801361f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0586453640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f05800a0c90 0x7f05801361f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42274/0 (socket says 172.21.15.12:42274) 2024-09-11T01:33:57.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0585c52640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f05800a1380 0x7f05800a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0586453640 1 -- 172.21.15.12:0/1138614131 learned_addr learned my addr 172.21.15.12:0/1138614131 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:57.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0586453640 10 monclient(hunting): get_auth_request con 0x7f05800a0c90 auth_method 0 2024-09-11T01:33:57.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0586453640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:57.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0586453640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:57.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0586453640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:57.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0585c52640 10 monclient(hunting): get_auth_request con 0x7f05800a1380 auth_method 0 2024-09-11T01:33:57.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0585c52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:57.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0585c52640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:57.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0585c52640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:57.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0586453640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:57.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0586453640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0586453640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0585c52640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.744+0000 7f0585c52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.745+0000 7f0585c52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.745+0000 7f0586453640 10 monclient(hunting): handle_auth_done global_id 54138 payload 274 2024-09-11T01:33:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.745+0000 7f0586453640 1 -- 172.21.15.12:0/1138614131 >> v2:172.21.15.71:3300/0 conn(0x7f05800a1380 msgr2=0x7f05800a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.745+0000 7f0586453640 1 --2- 172.21.15.12:0/1138614131 >> v2:172.21.15.71:3300/0 conn(0x7f05800a1380 0x7f05800a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.745+0000 7f0586453640 1 -- 172.21.15.12:0/1138614131 >> v1:172.21.15.71:6789/0 conn(0x7f05800a0370 legacy=0x7f0580135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.745+0000 7f0586453640 10 monclient: _finish_hunting 0 2024-09-11T01:33:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.745+0000 7f0586453640 1 monclient: found mon.noname-a 2024-09-11T01:33:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.745+0000 7f0586453640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.745+0000 7f0586453640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:33:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.745+0000 7f0586453640 1 -- 172.21.15.12:0/1138614131 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05800a2780 con 0x7f05800a0c90 2024-09-11T01:33:57.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.745+0000 7f0585c52640 1 --2- 172.21.15.12:0/1138614131 >> v2:172.21.15.71:3300/0 conn(0x7f05800a1380 0x7f05800a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:33:57.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.747+0000 7f0586453640 1 -- 172.21.15.12:0/1138614131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05800a0c90 msgr2=0x7f05801361f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:57.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.747+0000 7f0586453640 1 --2- 172.21.15.12:0/1138614131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05800a0c90 0x7f05801361f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f05800a72b0 tx=0x7f057802a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b440a88530efcd2d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:57.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0584c50640 1 -- 172.21.15.12:0/1138614131 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0578036070 con 0x7f05800a0c90 2024-09-11T01:33:57.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0584c50640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:57.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0584c50640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:33:57.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0584c50640 10 monclient: dump: 2024-09-11T01:33:57.758 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:57.758 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:57.758 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.759 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.759 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:57.759 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.759 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:57.759 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:57.759 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:57.759 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0584c50640 10 monclient: _finish_auth 0 2024-09-11T01:33:57.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0584c50640 10 monclient: _check_auth_tickets 2024-09-11T01:33:57.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0584c50640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:57.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0584c50640 1 -- 172.21.15.12:0/1138614131 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0578007800 con 0x7f05800a0c90 2024-09-11T01:33:57.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0587eb6640 5 monclient: authenticate success, global_id 54138 2024-09-11T01:33:57.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0587eb6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:33:57.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0584c50640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:57.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0584c50640 1 -- 172.21.15.12:0/1138614131 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0578007bd0 con 0x7f05800a0c90 2024-09-11T01:33:57.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0587eb6640 10 monclient: get_monmap_and_config success 2024-09-11T01:33:57.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0587eb6640 4 set_mon_vals no callback set 2024-09-11T01:33:57.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0584c50640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:57.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0584c50640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:33:57.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0584c50640 10 monclient: dump: 2024-09-11T01:33:57.760 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:57.760 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:57.760 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.760 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.760 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:57.760 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.760 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:57.760 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:57.761 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:57.761 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0587eb6640 10 monclient: shutdown 2024-09-11T01:33:57.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0587eb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:57.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0587eb6640 1 -- 172.21.15.12:0/1138614131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05800a0c90 msgr2=0x7f05801361f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:57.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0587eb6640 1 --2- 172.21.15.12:0/1138614131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05800a0c90 0x7f05801361f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f05800a72b0 tx=0x7f057802a570 comp rx=0 tx=0).stop 2024-09-11T01:33:57.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0587eb6640 1 -- 172.21.15.12:0/1138614131 shutdown_connections 2024-09-11T01:33:57.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0587eb6640 1 --2- 172.21.15.12:0/1138614131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05800a0c90 0x7f05801361f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0587eb6640 1 --2- 172.21.15.12:0/1138614131 >> v2:172.21.15.71:3300/0 conn(0x7f05800a1380 0x7f05800a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0587eb6640 1 -- 172.21.15.12:0/1138614131 >> 172.21.15.12:0/1138614131 conn(0x7f05800a2f70 msgr2=0x7f05800a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:57.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0587eb6640 1 -- 172.21.15.12:0/1138614131 shutdown_connections 2024-09-11T01:33:57.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.748+0000 7f0587eb6640 1 -- 172.21.15.12:0/1138614131 wait complete. 2024-09-11T01:33:57.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.749+0000 7f0587eb6640 10 monclient: build_initial_monmap 2024-09-11T01:33:57.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.749+0000 7f0587eb6640 10 monclient: monmap: 2024-09-11T01:33:57.762 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:57.762 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:33:57.762 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:33:57.762 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:33:57.762 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:57.762 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.762 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:33:57.762 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:33:57.762 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:33:57.762 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.749+0000 7f0587eb6640 1 Processor -- start 2024-09-11T01:33:57.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.749+0000 7f0587eb6640 1 -- start start 2024-09-11T01:33:57.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.749+0000 7f0587eb6640 10 monclient: init 2024-09-11T01:33:57.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0587eb6640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:57.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0587eb6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:33:57.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0587eb6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05800a0370 0x7f0580123c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0587eb6640 10 monclient(hunting): picked mon.noname-b con 0x7f05800a0370 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:33:57.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0587eb6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f05800a0c90 0x7f0580124150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0587eb6640 10 monclient(hunting): picked mon.noname-c con 0x7f05800a0c90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:57.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0587eb6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05800a1380 0x7f0580124690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0587eb6640 10 monclient(hunting): picked mon.noname-a con 0x7f05800a1380 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:33:57.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0587eb6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0587eb6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f058009ce20 con 0x7f05800a1380 2024-09-11T01:33:57.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0587eb6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0587eb6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f058009cca0 con 0x7f05800a0370 2024-09-11T01:33:57.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0587eb6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:57.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0587eb6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f0580136730 con 0x7f05800a0c90 2024-09-11T01:33:57.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0587eb6640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:57.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0587eb6640 10 monclient(hunting): authenticate will time out at 1044.192250s 2024-09-11T01:33:57.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0586453640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05800a1380 0x7f0580124690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0585451640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f05800a0c90 0x7f0580124150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0586453640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05800a1380 0x7f0580124690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42276/0 (socket says 172.21.15.12:42276) 2024-09-11T01:33:57.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0586453640 1 -- 172.21.15.12:0/1835506514 learned_addr learned my addr 172.21.15.12:0/1835506514 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:57.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0585451640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f05800a0c90 0x7f0580124150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:37276/0 (socket says 172.21.15.12:37276) 2024-09-11T01:33:57.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0586453640 10 monclient(hunting): get_auth_request con 0x7f05800a1380 auth_method 0 2024-09-11T01:33:57.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0586453640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:57.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0586453640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:57.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0586453640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:57.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0585451640 10 monclient(hunting): get_auth_request con 0x7f05800a0c90 auth_method 0 2024-09-11T01:33:57.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0585451640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:57.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0585451640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:57.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0585451640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:57.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0585c52640 1 --2- 172.21.15.12:0/1835506514 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05800a0370 0x7f0580123c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0585c52640 10 monclient(hunting): get_auth_request con 0x7f05800a0370 auth_method 0 2024-09-11T01:33:57.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0585c52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:57.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0585c52640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:57.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0585c52640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:57.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0586453640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:57.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0586453640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:57.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0586453640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:57.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0585451640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:57.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0585451640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:57.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0585451640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:57.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0585c52640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:57.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0585c52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:57.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0585c52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:57.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0586453640 10 monclient(hunting): handle_auth_done global_id 54141 payload 824 2024-09-11T01:33:57.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0586453640 1 -- 172.21.15.12:0/1835506514 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f05800a0c90 msgr2=0x7f0580124150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:57.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0586453640 1 --2- 172.21.15.12:0/1835506514 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f05800a0c90 0x7f0580124150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0586453640 1 -- 172.21.15.12:0/1835506514 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05800a0370 msgr2=0x7f0580123c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:57.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0586453640 1 --2- 172.21.15.12:0/1835506514 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05800a0370 0x7f0580123c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:57.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0586453640 10 monclient: _finish_hunting 0 2024-09-11T01:33:57.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0586453640 1 monclient: found mon.noname-a 2024-09-11T01:33:57.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0586453640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:57.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0586453640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:33:57.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0586453640 1 -- 172.21.15.12:0/1835506514 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0580127a30 con 0x7f05800a1380 2024-09-11T01:33:57.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0585c52640 1 --2- 172.21.15.12:0/1835506514 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05800a0370 0x7f0580123c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:33:57.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.750+0000 7f0585451640 1 --2- 172.21.15.12:0/1835506514 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f05800a0c90 0x7f0580124150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:33:57.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.752+0000 7f0586453640 1 -- 172.21.15.12:0/1835506514 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05800a1380 msgr2=0x7f0580124690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:57.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.752+0000 7f0586453640 1 --2- 172.21.15.12:0/1835506514 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05800a1380 0x7f0580124690 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f057802ab60 tx=0x7f0578004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:57.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.753+0000 7f05767fc640 1 -- 172.21.15.12:0/1835506514 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0578036050 con 0x7f05800a1380 2024-09-11T01:33:57.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.753+0000 7f05767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:57.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.753+0000 7f05767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:33:57.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.753+0000 7f05767fc640 10 monclient: dump: 2024-09-11T01:33:57.767 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:57.767 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:57.767 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.767 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.767 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:57.767 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.768 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:57.768 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:57.768 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:57.768 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.753+0000 7f05767fc640 10 monclient: _finish_auth 0 2024-09-11T01:33:57.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.753+0000 7f05767fc640 10 monclient: _check_auth_tickets 2024-09-11T01:33:57.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.753+0000 7f05767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:57.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.753+0000 7f05767fc640 1 -- 172.21.15.12:0/1835506514 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0578042070 con 0x7f05800a1380 2024-09-11T01:33:57.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.753+0000 7f05767fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:57.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.753+0000 7f05767fc640 1 -- 172.21.15.12:0/1835506514 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0578031d90 con 0x7f05800a1380 2024-09-11T01:33:57.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.753+0000 7f05767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:57.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.753+0000 7f05767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:33:57.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.753+0000 7f05767fc640 10 monclient: dump: 2024-09-11T01:33:57.768 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:57.769 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:57.769 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.769 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:57.769 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:57.769 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:57.769 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:57.769 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:57.769 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:57.769 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:57.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.753+0000 7f0587eb6640 5 monclient: authenticate success, global_id 54141 2024-09-11T01:33:57.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.753+0000 7f0587eb6640 10 monclient: _renew_subs 2024-09-11T01:33:57.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.753+0000 7f0587eb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:33:57.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.753+0000 7f0587eb6640 1 -- 172.21.15.12:0/1835506514 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0580070400 con 0x7f05800a1380 2024-09-11T01:33:57.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.753+0000 7f0577fff640 4 set_mon_vals no callback set 2024-09-11T01:33:57.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.753+0000 7f0587eb6640 10 monclient: _renew_subs 2024-09-11T01:33:57.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.753+0000 7f0587eb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:33:57.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.753+0000 7f0587eb6640 1 -- 172.21.15.12:0/1835506514 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05801256d0 con 0x7f05800a1380 2024-09-11T01:33:57.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.754+0000 7f05767fc640 1 -- 172.21.15.12:0/1835506514 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f0578047460 con 0x7f05800a1380 2024-09-11T01:33:57.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.754+0000 7f0587eb6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:57.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.754+0000 7f0587eb6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:57.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.754+0000 7f0587eb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:33:57.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.754+0000 7f0587eb6640 1 -- 172.21.15.12:0/1835506514 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0580069c00 con 0x7f05800a1380 2024-09-11T01:33:57.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.754+0000 7f05767fc640 1 --2- 172.21.15.12:0/1835506514 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f056403e210 0x7f05640406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:57.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.754+0000 7f05767fc640 1 -- 172.21.15.12:0/1835506514 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(171..171 src has 1..171) ==== 4815+0+0 (secure 0 0 0) 0x7f0578073ca0 con 0x7f05800a1380 2024-09-11T01:33:57.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.754+0000 7f0585c52640 1 --2- 172.21.15.12:0/1835506514 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f056403e210 0x7f05640406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:57.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.754+0000 7f0585c52640 10 monclient: get_auth_request con 0x7f056403e210 auth_method 0 2024-09-11T01:33:57.772 INFO:tasks.rados.rados.0.smithi071.stdout:755: finishing write tid 1 to smithi07134777-50 2024-09-11T01:33:57.773 INFO:tasks.rados.rados.0.smithi071.stdout:753: done (4 left) 2024-09-11T01:33:57.773 INFO:tasks.rados.rados.0.smithi071.stdout:754: done (3 left) 2024-09-11T01:33:57.773 INFO:tasks.rados.rados.0.smithi071.stdout:758: read oid 27 snap -1 2024-09-11T01:33:57.773 INFO:tasks.rados.rados.0.smithi071.stdout:758: expect deleted 2024-09-11T01:33:57.773 INFO:tasks.rados.rados.0.smithi071.stdout:759: write (excl) oid 2 current snap is 74 2024-09-11T01:33:57.773 INFO:tasks.rados.rados.0.smithi071.stdout:756: finishing copy_from to smithi07134777-49 2024-09-11T01:33:57.773 INFO:tasks.rados.rados.0.smithi071.stdout:756: got expected ENOENT (src dne) 2024-09-11T01:33:57.774 INFO:tasks.rados.rados.0.smithi071.stdout:756: finishing copy_from racing read to smithi07134777-49 2024-09-11T01:33:57.774 INFO:tasks.rados.rados.0.smithi071.stdout:755: finishing write tid 2 to smithi07134777-50 2024-09-11T01:33:57.774 INFO:tasks.rados.rados.0.smithi071.stdout:759: seq_num 288 ranges {440416=776773,1686814=732100} 2024-09-11T01:33:57.774 INFO:tasks.rados.rados.0.smithi071.stdout:759: writing smithi07134777-2 from 440416 to 1217189 tid 1 2024-09-11T01:33:57.774 INFO:tasks.rados.rados.0.smithi071.stdout:759: writing smithi07134777-2 from 1686814 to 2418914 tid 2 2024-09-11T01:33:57.774 INFO:tasks.rados.rados.0.smithi071.stdout:756: done (4 left) 2024-09-11T01:33:57.774 INFO:tasks.rados.rados.0.smithi071.stdout:760: read oid 7 snap -1 2024-09-11T01:33:57.774 INFO:tasks.rados.rados.0.smithi071.stdout:760: expect (ObjNum 272 snap 72 seq_num 272) 2024-09-11T01:33:57.774 INFO:tasks.rados.rados.0.smithi071.stdout:761: snap_create 2024-09-11T01:33:57.775 INFO:tasks.rados.rados.0.smithi071.stdout:755: finishing write tid 3 to smithi07134777-50 2024-09-11T01:33:57.775 INFO:tasks.rados.rados.0.smithi071.stdout:755: finishing write tid 4 to smithi07134777-50 2024-09-11T01:33:57.775 INFO:tasks.rados.rados.0.smithi071.stdout:755: finishing write tid 5 to smithi07134777-50 2024-09-11T01:33:57.775 INFO:tasks.rados.rados.0.smithi071.stdout:755: oid 50 updating version 0 to 358 2024-09-11T01:33:57.775 INFO:tasks.rados.rados.0.smithi071.stdout:755: oid 50 version 358 is already newer than 355 2024-09-11T01:33:57.775 INFO:tasks.rados.rados.0.smithi071.stdout:755: oid 50 version 358 is already newer than 356 2024-09-11T01:33:57.775 INFO:tasks.rados.rados.0.smithi071.stdout:755: oid 50 version 358 is already newer than 357 2024-09-11T01:33:57.776 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 358 (ObjNum 287 snap 74 seq_num 287) dirty exists 2024-09-11T01:33:57.776 INFO:tasks.rados.rados.0.smithi071.stdout:755: left oid 50 (ObjNum 287 snap 74 seq_num 287) 2024-09-11T01:33:57.776 INFO:tasks.rados.rados.0.smithi071.stdout:757: finishing copy_from racing read to smithi07134777-3 2024-09-11T01:33:57.776 INFO:tasks.rados.rados.0.smithi071.stdout:757: finishing copy_from to smithi07134777-3 2024-09-11T01:33:57.776 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 221 (ObjNum 200 snap 46 seq_num 200) dirty exists 2024-09-11T01:33:57.776 INFO:tasks.rados.rados.0.smithi071.stdout:759: finishing write tid 1 to smithi07134777-2 2024-09-11T01:33:57.776 INFO:tasks.rados.rados.0.smithi071.stdout:759: finishing write tid 2 to smithi07134777-2 2024-09-11T01:33:57.776 INFO:tasks.rados.rados.0.smithi071.stdout:760: expect (ObjNum 272 snap 72 seq_num 272) 2024-09-11T01:33:57.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.757+0000 7f0585c52640 1 --2- 172.21.15.12:0/1835506514 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f056403e210 0x7f05640406d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f057c009560 tx=0x7f057c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:57.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.759+0000 7f05767fc640 1 -- 172.21.15.12:0/1835506514 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0580069c00 con 0x7f05800a1380 2024-09-11T01:33:57.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.759+0000 7f05767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:57.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:57.759+0000 7f05767fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:33:57.892 INFO:tasks.rados.rados.0.smithi071.stdout:759: finishing write tid 3 to smithi07134777-2 2024-09-11T01:33:57.893 INFO:tasks.rados.rados.0.smithi071.stdout:759: finishing write tid 4 to smithi07134777-2 2024-09-11T01:33:57.893 INFO:tasks.rados.rados.0.smithi071.stdout:759: oid 2 updating version 0 to 138 2024-09-11T01:33:57.893 INFO:tasks.rados.rados.0.smithi071.stdout:759: oid 2 version 138 is already newer than 137 2024-09-11T01:33:57.893 INFO:tasks.rados.rados.0.smithi071.stdout:759: oid 2 version 138 is already newer than 136 2024-09-11T01:33:57.893 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 138 (ObjNum 288 snap 74 seq_num 288) dirty exists 2024-09-11T01:33:57.893 INFO:tasks.rados.rados.0.smithi071.stdout:759: left oid 2 (ObjNum 288 snap 74 seq_num 288) 2024-09-11T01:33:58.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:58.014+0000 7f0587eb6640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:33:58.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:58.014+0000 7f0587eb6640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:33:58.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:58.014+0000 7f0587eb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:33:58.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:58.014+0000 7f0587eb6640 1 -- 172.21.15.12:0/1835506514 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f05800a1760 con 0x7f05800a1380 2024-09-11T01:33:58.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:58.015+0000 7f05767fc640 1 -- 172.21.15.12:0/1835506514 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f0578072680 con 0x7f05800a1380 2024-09-11T01:33:58.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:58.015+0000 7f05767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:33:58.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:58.015+0000 7f05767fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:33:58.018 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.015+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.018 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.016+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=11,config=2,health=12,kv=11,logm=64} crc {auth=1202190055,config=2586702516,health=1535303502,kv=4240804263,logm=2974362868}) 2024-09-11T01:33:58.018 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.016+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=11,config=2,health=11,kv=11,logm=65} crc {auth=1946905999,config=2586702516,health=1096256309,kv=58696370,logm=1627312190}) 2024-09-11T01:33:58.018 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.016+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.018 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.016+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=11,config=2,health=12,kv=11,logm=64} crc {auth=1202190055,config=2586702516,health=1535303502,kv=4240804263,logm=2974362868}) 2024-09-11T01:33:58.018 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.016+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=11,config=2,health=11,kv=12,logm=64} crc {auth=2667280965,config=2586702516,health=1096256309,kv=3263697420,logm=3374575154}) 2024-09-11T01:33:58.019 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.017+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.019 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.017+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=572031810}) 2024-09-11T01:33:58.019 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.017+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2772094457}) 2024-09-11T01:33:58.019 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.017+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.019 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.017+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=572031810}) 2024-09-11T01:33:58.019 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.017+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=196664125}) 2024-09-11T01:33:58.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:58.017+0000 7f0587eb6640 1 -- 172.21.15.12:0/1835506514 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f056403e210 msgr2=0x7f05640406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:58.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:58.017+0000 7f0587eb6640 1 --2- 172.21.15.12:0/1835506514 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f056403e210 0x7f05640406d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f057c009560 tx=0x7f057c009190 comp rx=0 tx=0).stop 2024-09-11T01:33:58.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:58.017+0000 7f0587eb6640 10 monclient: shutdown 2024-09-11T01:33:58.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:58.017+0000 7f0587eb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:58.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:58.017+0000 7f0587eb6640 1 -- 172.21.15.12:0/1835506514 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05800a1380 msgr2=0x7f0580124690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:58.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:58.017+0000 7f0587eb6640 1 --2- 172.21.15.12:0/1835506514 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05800a1380 0x7f0580124690 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f057802ab60 tx=0x7f0578004a40 comp rx=0 tx=0).stop 2024-09-11T01:33:58.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:58.017+0000 7f0587eb6640 1 -- 172.21.15.12:0/1835506514 shutdown_connections 2024-09-11T01:33:58.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:58.017+0000 7f0587eb6640 1 --2- 172.21.15.12:0/1835506514 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f056403e210 0x7f05640406d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:58.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:58.017+0000 7f0587eb6640 1 --2- 172.21.15.12:0/1835506514 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05800a1380 0x7f0580124690 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:58.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:58.017+0000 7f0587eb6640 1 --2- 172.21.15.12:0/1835506514 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f05800a0c90 0x7f0580124150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:58.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:58.017+0000 7f0587eb6640 1 --2- 172.21.15.12:0/1835506514 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05800a0370 0x7f0580123c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:58.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:58.017+0000 7f0587eb6640 1 -- 172.21.15.12:0/1835506514 >> 172.21.15.12:0/1835506514 conn(0x7f05800a2f70 msgr2=0x7f0580071ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:58.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:58.017+0000 7f0587eb6640 1 -- 172.21.15.12:0/1835506514 shutdown_connections 2024-09-11T01:33:58.022 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.017+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:58.017+0000 7f0587eb6640 1 -- 172.21.15.12:0/1835506514 wait complete. 2024-09-11T01:33:58.027 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.017+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2934015505}) 2024-09-11T01:33:58.027 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.017+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1558541237}) 2024-09-11T01:33:58.027 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.017+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.027 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.017+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2934015505}) 2024-09-11T01:33:58.027 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.017+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2283420862}) 2024-09-11T01:33:58.027 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.018+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.027 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.018+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2904033283}) 2024-09-11T01:33:58.027 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.018+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2340487018}) 2024-09-11T01:33:58.027 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.018+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.027 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.018+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2904033283}) 2024-09-11T01:33:58.028 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.018+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=447349285}) 2024-09-11T01:33:58.028 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.019+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.028 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.019+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1697856193}) 2024-09-11T01:33:58.028 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.019+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=111486573}) 2024-09-11T01:33:58.028 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.019+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.028 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.019+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1697856193}) 2024-09-11T01:33:58.028 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.019+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1809153122}) 2024-09-11T01:33:58.028 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.020+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.028 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.020+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354146044}) 2024-09-11T01:33:58.028 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.020+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=680089797}) 2024-09-11T01:33:58.028 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.020+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.028 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.020+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354146044}) 2024-09-11T01:33:58.028 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.020+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=503191718}) 2024-09-11T01:33:58.029 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.021+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.029 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.021+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1373175589}) 2024-09-11T01:33:58.029 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.021+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1393223155}) 2024-09-11T01:33:58.029 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.021+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.029 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.021+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1373175589}) 2024-09-11T01:33:58.029 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.021+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=724220797}) 2024-09-11T01:33:58.029 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.021+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.029 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.021+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1038392676}) 2024-09-11T01:33:58.029 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.021+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2132685477}) 2024-09-11T01:33:58.029 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.021+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.029 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.021+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1038392676}) 2024-09-11T01:33:58.029 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.021+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1688586469}) 2024-09-11T01:33:58.029 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.022+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.029 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.022+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=867325199}) 2024-09-11T01:33:58.030 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.022+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3748689608}) 2024-09-11T01:33:58.030 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.022+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.030 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.022+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=867325199}) 2024-09-11T01:33:58.030 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.022+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1027165262}) 2024-09-11T01:33:58.030 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.023+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.030 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.023+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1174944162}) 2024-09-11T01:33:58.030 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.023+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3325851775}) 2024-09-11T01:33:58.030 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.023+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.030 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.023+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1174944162}) 2024-09-11T01:33:58.030 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.023+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=70725434}) 2024-09-11T01:33:58.030 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.023+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.030 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.023+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3673553082}) 2024-09-11T01:33:58.030 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.023+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1818391461}) 2024-09-11T01:33:58.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.023+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.023+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3673553082}) 2024-09-11T01:33:58.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.023+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=555984706}) 2024-09-11T01:33:58.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.024+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.024+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2423678211}) 2024-09-11T01:33:58.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.024+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2993676839}) 2024-09-11T01:33:58.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.024+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.024+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2423678211}) 2024-09-11T01:33:58.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.024+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2458485469}) 2024-09-11T01:33:58.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.025+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.025+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=48,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=18,mon_config_key=17,monmap=3,nvmeofgw=1,osd_metadata=1} crc {logm=4002964545,mdsmap=3760429986,mgr=2431795800,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=964353929,mon_config_key=3330371543,monmap=2933275274,nvmeofgw=1228700967,osd_metadata=3504647627}) 2024-09-11T01:33:58.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.025+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=46,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=18,mon_config_key=14,monmap=2,nvmeofgw=3,osd_metadata=5} crc {logm=1248375019,mdsmap=3760429986,mgr=2431795800,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=964353929,mon_config_key=3442574661,monmap=3140514009,nvmeofgw=598337686,osd_metadata=642414809}) 2024-09-11T01:33:58.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.025+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.032 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.025+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=48,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=18,mon_config_key=17,monmap=3,nvmeofgw=1,osd_metadata=1} crc {logm=4002964545,mdsmap=3760429986,mgr=2431795800,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=964353929,mon_config_key=3330371543,monmap=2933275274,nvmeofgw=1228700967,osd_metadata=3504647627}) 2024-09-11T01:33:58.032 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.025+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=47,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=18,mon_config_key=15,monmap=2,nvmeofgw=3,osd_metadata=2} crc {logm=1168594216,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=964353929,mon_config_key=1138777510,monmap=3140514009,nvmeofgw=598337686,osd_metadata=1622464708}) 2024-09-11T01:33:58.033 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.026+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.033 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.026+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osd_snap=29,osdmap=65} crc {osd_metadata=1404192910,osd_pg_creating=3538459402,osd_snap=656611222,osdmap=3958964929}) 2024-09-11T01:33:58.033 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.026+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osd_snap=31,osdmap=63} crc {osd_metadata=1404192910,osd_pg_creating=3538459402,osd_snap=63304325,osdmap=2473174740}) 2024-09-11T01:33:58.033 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.026+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.033 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.026+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osd_snap=29,osdmap=65} crc {osd_metadata=1404192910,osd_pg_creating=3538459402,osd_snap=656611222,osdmap=3958964929}) 2024-09-11T01:33:58.033 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.026+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=5,osd_snap=31,osdmap=64} crc {osd_metadata=1404192910,osd_snap=1455226998,osdmap=1040948587}) 2024-09-11T01:33:58.034 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:33:58.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.466+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.466+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1421631384}) 2024-09-11T01:33:58.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.466+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1613194303}) 2024-09-11T01:33:58.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.466+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.466+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1421631384}) 2024-09-11T01:33:58.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.466+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3404613276}) 2024-09-11T01:33:58.552 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.550+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.552 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.550+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=95} crc {osdmap=2236448474}) 2024-09-11T01:33:58.552 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.550+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=95} crc {osdmap=4225876989}) 2024-09-11T01:33:58.552 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.550+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:33:58.552 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.550+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=95} crc {osdmap=2236448474}) 2024-09-11T01:33:58.552 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:33:58.550+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=93} crc {osdmap=1070745614}) 2024-09-11T01:33:58.740 INFO:tasks.rados.rados.0.smithi071.stdout:755: done (5 left) 2024-09-11T01:33:58.740 INFO:tasks.rados.rados.0.smithi071.stdout:757: done (4 left) 2024-09-11T01:33:58.741 INFO:tasks.rados.rados.0.smithi071.stdout:758: done (3 left) 2024-09-11T01:33:58.741 INFO:tasks.rados.rados.0.smithi071.stdout:759: done (2 left) 2024-09-11T01:33:58.741 INFO:tasks.rados.rados.0.smithi071.stdout:760: done (1 left) 2024-09-11T01:33:58.741 INFO:tasks.rados.rados.0.smithi071.stdout:761: done (0 left) 2024-09-11T01:33:58.741 INFO:tasks.rados.rados.0.smithi071.stdout:762: snap_create 2024-09-11T01:33:59.035 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:33:59.036 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:33:59.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.147+0000 7f1c66cdd640 10 monclient: get_monmap_and_config 2024-09-11T01:33:59.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.147+0000 7f1c66cdd640 10 monclient: build_initial_monmap 2024-09-11T01:33:59.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.147+0000 7f1c66cdd640 10 monclient: monmap: 2024-09-11T01:33:59.149 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:59.149 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:59.149 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:33:59.148445+0000 2024-09-11T01:33:59.149 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:33:59.148445+0000 2024-09-11T01:33:59.149 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:59.150 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.150 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:33:59.150 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:33:59.150 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:33:59.150 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:33:59.150 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:33:59.150 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.148+0000 7f1c66cdd640 1 Processor -- start 2024-09-11T01:33:59.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.148+0000 7f1c66cdd640 1 -- start start 2024-09-11T01:33:59.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.148+0000 7f1c66cdd640 10 monclient: init 2024-09-11T01:33:59.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c66cdd640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:59.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c66cdd640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-11T01:33:59.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c66cdd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1c600a2e90 addr v1:172.21.15.71:6789/0 2024-09-11T01:33:59.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c66cdd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1c600a3270 addr v1:172.21.15.12:6789/0 2024-09-11T01:33:59.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c66cdd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c6013c160 0x7f1c6013e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c66cdd640 10 monclient(hunting): picked mon.noname-b con 0x7f1c6013c160 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:59.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c66cdd640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c600a46c0 con 0x7f1c600a3270 2024-09-11T01:33:59.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c66cdd640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c60072fa0 con 0x7f1c600a2e90 2024-09-11T01:33:59.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c66cdd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:59.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c66cdd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1c60072730 con 0x7f1c6013c160 2024-09-11T01:33:59.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c66cdd640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:59.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c66cdd640 10 monclient(hunting): authenticate will time out at 1045.591327s 2024-09-11T01:33:59.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c5ffff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1c600a3270 0x7f1c600a36a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33020/0 (socket says 172.21.15.12:33020) 2024-09-11T01:33:59.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c6527a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c6013c160 0x7f1c6013e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c5ffff640 1 -- 172.21.15.12:0/181970502 learned_addr learned my addr 172.21.15.12:0/181970502 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:33:59.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c6527a640 10 monclient(hunting): get_auth_request con 0x7f1c6013c160 auth_method 0 2024-09-11T01:33:59.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c6527a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:59.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c6527a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:59.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c6527a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:59.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c6527a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:59.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c6527a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:59.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c6527a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:59.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c6527a640 10 monclient(hunting): handle_auth_done global_id 44228 payload 274 2024-09-11T01:33:59.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c6527a640 1 -- 172.21.15.12:0/181970502 >> v1:172.21.15.71:6789/0 conn(0x7f1c600a2e90 legacy=0x7f1c60073d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:59.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c6527a640 1 -- 172.21.15.12:0/181970502 >> v1:172.21.15.12:6789/0 conn(0x7f1c600a3270 legacy=0x7f1c600a36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:59.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c6527a640 10 monclient: _finish_hunting 0 2024-09-11T01:33:59.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c6527a640 1 monclient: found mon.noname-b 2024-09-11T01:33:59.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c6527a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:59.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c6527a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:33:59.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.149+0000 7f1c6527a640 1 -- 172.21.15.12:0/181970502 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c60073280 con 0x7f1c6013c160 2024-09-11T01:33:59.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.152+0000 7f1c6527a640 1 -- 172.21.15.12:0/181970502 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c6013c160 msgr2=0x7f1c6013e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:59.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.152+0000 7f1c6527a640 1 --2- 172.21.15.12:0/181970502 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c6013c160 0x7f1c6013e550 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1c58009dc0 tx=0x7f1c5802a510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2399047397830bb3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:59.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.152+0000 7f1c5f7fe640 1 -- 172.21.15.12:0/181970502 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c58007a30 con 0x7f1c6013c160 2024-09-11T01:33:59.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.152+0000 7f1c5f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:59.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.152+0000 7f1c5f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:33:59.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.152+0000 7f1c5f7fe640 10 monclient: dump: 2024-09-11T01:33:59.165 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:59.165 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:59.165 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.165 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.165 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:59.166 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.166 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:59.166 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:59.166 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:59.166 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.152+0000 7f1c5f7fe640 10 monclient: _finish_auth 0 2024-09-11T01:33:59.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.152+0000 7f1c5f7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:33:59.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.152+0000 7f1c5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:59.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.152+0000 7f1c5f7fe640 1 -- 172.21.15.12:0/181970502 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c58007bd0 con 0x7f1c6013c160 2024-09-11T01:33:59.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.152+0000 7f1c66cdd640 5 monclient: authenticate success, global_id 44228 2024-09-11T01:33:59.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.152+0000 7f1c66cdd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:33:59.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.152+0000 7f1c5f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:59.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.152+0000 7f1c5f7fe640 1 -- 172.21.15.12:0/181970502 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c58031400 con 0x7f1c6013c160 2024-09-11T01:33:59.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.152+0000 7f1c5f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:59.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.152+0000 7f1c5f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:59.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.152+0000 7f1c5f7fe640 10 monclient: dump: 2024-09-11T01:33:59.167 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:59.167 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:59.167 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.167 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.167 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:59.167 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.167 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:59.167 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:59.167 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:59.167 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.152+0000 7f1c66cdd640 10 monclient: get_monmap_and_config success 2024-09-11T01:33:59.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.152+0000 7f1c66cdd640 4 set_mon_vals no callback set 2024-09-11T01:33:59.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.153+0000 7f1c66cdd640 10 monclient: shutdown 2024-09-11T01:33:59.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.153+0000 7f1c66cdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:59.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.153+0000 7f1c66cdd640 1 -- 172.21.15.12:0/181970502 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c6013c160 msgr2=0x7f1c6013e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:59.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.153+0000 7f1c66cdd640 1 --2- 172.21.15.12:0/181970502 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c6013c160 0x7f1c6013e550 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1c58009dc0 tx=0x7f1c5802a510 comp rx=0 tx=0).stop 2024-09-11T01:33:59.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.153+0000 7f1c66cdd640 1 -- 172.21.15.12:0/181970502 shutdown_connections 2024-09-11T01:33:59.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.153+0000 7f1c66cdd640 1 --2- 172.21.15.12:0/181970502 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c6013c160 0x7f1c6013e550 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.153+0000 7f1c66cdd640 1 -- 172.21.15.12:0/181970502 >> 172.21.15.12:0/181970502 conn(0x7f1c600a8490 msgr2=0x7f1c600a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:59.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.153+0000 7f1c66cdd640 1 -- 172.21.15.12:0/181970502 shutdown_connections 2024-09-11T01:33:59.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.153+0000 7f1c66cdd640 1 -- 172.21.15.12:0/181970502 wait complete. 2024-09-11T01:33:59.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.154+0000 7f1c66cdd640 10 monclient: build_initial_monmap 2024-09-11T01:33:59.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.154+0000 7f1c66cdd640 10 monclient: monmap: 2024-09-11T01:33:59.168 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:59.169 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:33:59.169 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:33:59.169 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:33:59.169 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:59.169 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.169 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:33:59.169 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:33:59.169 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:33:59.169 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.154+0000 7f1c66cdd640 1 Processor -- start 2024-09-11T01:33:59.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.154+0000 7f1c66cdd640 1 -- start start 2024-09-11T01:33:59.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.154+0000 7f1c66cdd640 10 monclient: init 2024-09-11T01:33:59.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c66cdd640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:59.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c66cdd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:33:59.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c66cdd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c600a2e90 0x7f1c60135880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c66cdd640 10 monclient(hunting): picked mon.noname-a con 0x7f1c600a2e90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:33:59.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c66cdd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c600a3270 0x7f1c60135dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c66cdd640 10 monclient(hunting): picked mon.noname-c con 0x7f1c600a3270 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:59.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c66cdd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c6013c160 0x7f1c60139040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c66cdd640 10 monclient(hunting): picked mon.noname-b con 0x7f1c6013c160 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:33:59.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c66cdd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:59.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c66cdd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6013ec10 con 0x7f1c600a2e90 2024-09-11T01:33:59.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c66cdd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:59.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c66cdd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6013ea90 con 0x7f1c6013c160 2024-09-11T01:33:59.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c66cdd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:59.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c66cdd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1c6013ed90 con 0x7f1c600a3270 2024-09-11T01:33:59.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c66cdd640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:59.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c66cdd640 10 monclient(hunting): authenticate will time out at 1045.597131s 2024-09-11T01:33:59.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c64a79640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c600a2e90 0x7f1c60135880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c5ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c600a3270 0x7f1c60135dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c64a79640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c600a2e90 0x7f1c60135880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42286/0 (socket says 172.21.15.12:42286) 2024-09-11T01:33:59.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c64a79640 1 -- 172.21.15.12:0/462197394 learned_addr learned my addr 172.21.15.12:0/462197394 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:59.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c64a79640 10 monclient(hunting): get_auth_request con 0x7f1c600a2e90 auth_method 0 2024-09-11T01:33:59.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c64a79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:59.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c64a79640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:59.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c64a79640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:59.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c5ffff640 10 monclient(hunting): get_auth_request con 0x7f1c600a3270 auth_method 0 2024-09-11T01:33:59.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:59.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:59.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:59.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c64a79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:59.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c64a79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:59.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c64a79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:59.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:59.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:59.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:59.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c6527a640 1 --2- 172.21.15.12:0/462197394 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c6013c160 0x7f1c60139040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c6527a640 10 monclient(hunting): get_auth_request con 0x7f1c6013c160 auth_method 0 2024-09-11T01:33:59.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c6527a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:59.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c6527a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:59.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c6527a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:59.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c64a79640 10 monclient(hunting): handle_auth_done global_id 54144 payload 824 2024-09-11T01:33:59.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c64a79640 1 -- 172.21.15.12:0/462197394 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c600a3270 msgr2=0x7f1c60135dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:59.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c64a79640 1 --2- 172.21.15.12:0/462197394 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c600a3270 0x7f1c60135dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c64a79640 1 -- 172.21.15.12:0/462197394 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c6013c160 msgr2=0x7f1c60139040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:59.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c64a79640 1 --2- 172.21.15.12:0/462197394 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c6013c160 0x7f1c60139040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c64a79640 10 monclient: _finish_hunting 0 2024-09-11T01:33:59.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c64a79640 1 monclient: found mon.noname-a 2024-09-11T01:33:59.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c64a79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:59.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c64a79640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:33:59.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.155+0000 7f1c64a79640 1 -- 172.21.15.12:0/462197394 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c6011f440 con 0x7f1c600a2e90 2024-09-11T01:33:59.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.157+0000 7f1c64a79640 1 -- 172.21.15.12:0/462197394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c600a2e90 msgr2=0x7f1c60135880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:59.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.157+0000 7f1c64a79640 1 --2- 172.21.15.12:0/462197394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c600a2e90 0x7f1c60135880 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1c5000c430 tx=0x7f1c5000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:59.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.157+0000 7f1c5d7fa640 1 -- 172.21.15.12:0/462197394 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c500119a0 con 0x7f1c600a2e90 2024-09-11T01:33:59.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.157+0000 7f1c5d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:59.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.157+0000 7f1c5d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:33:59.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.157+0000 7f1c5d7fa640 10 monclient: dump: 2024-09-11T01:33:59.174 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:59.174 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:59.174 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.174 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.174 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:59.174 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.174 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:59.174 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:59.174 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:59.174 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.157+0000 7f1c5d7fa640 10 monclient: _finish_auth 0 2024-09-11T01:33:59.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.157+0000 7f1c5d7fa640 10 monclient: _check_auth_tickets 2024-09-11T01:33:59.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.157+0000 7f1c5d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:59.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.157+0000 7f1c5d7fa640 1 -- 172.21.15.12:0/462197394 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c50017070 con 0x7f1c600a2e90 2024-09-11T01:33:59.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.157+0000 7f1c66cdd640 5 monclient: authenticate success, global_id 54144 2024-09-11T01:33:59.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.158+0000 7f1c5d7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:59.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.158+0000 7f1c66cdd640 10 monclient: _renew_subs 2024-09-11T01:33:59.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.158+0000 7f1c5d7fa640 1 -- 172.21.15.12:0/462197394 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c50011d80 con 0x7f1c600a2e90 2024-09-11T01:33:59.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.158+0000 7f1c66cdd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:33:59.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.158+0000 7f1c66cdd640 1 -- 172.21.15.12:0/462197394 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c6013c540 con 0x7f1c600a2e90 2024-09-11T01:33:59.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.158+0000 7f1c5d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:59.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.158+0000 7f1c5d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:33:59.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.158+0000 7f1c5d7fa640 10 monclient: dump: 2024-09-11T01:33:59.175 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:59.176 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:59.176 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.176 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.176 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:59.176 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.176 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:59.176 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:59.176 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:59.176 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.158+0000 7f1c5effd640 4 set_mon_vals no callback set 2024-09-11T01:33:59.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.158+0000 7f1c66cdd640 10 monclient: _renew_subs 2024-09-11T01:33:59.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.158+0000 7f1c66cdd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:33:59.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.158+0000 7f1c66cdd640 1 -- 172.21.15.12:0/462197394 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c6013b120 con 0x7f1c600a2e90 2024-09-11T01:33:59.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.158+0000 7f1c66cdd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:59.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.158+0000 7f1c66cdd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:59.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.158+0000 7f1c66cdd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:33:59.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.158+0000 7f1c66cdd640 1 -- 172.21.15.12:0/462197394 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c60073d40 con 0x7f1c600a2e90 2024-09-11T01:33:59.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.159+0000 7f1c5d7fa640 1 -- 172.21.15.12:0/462197394 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f1c5001c460 con 0x7f1c600a2e90 2024-09-11T01:33:59.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.159+0000 7f1c5d7fa640 1 --2- 172.21.15.12:0/462197394 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1c4c03e210 0x7f1c4c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.159+0000 7f1c5d7fa640 1 -- 172.21.15.12:0/462197394 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4787+0+0 (secure 0 0 0) 0x7f1c50052be0 con 0x7f1c600a2e90 2024-09-11T01:33:59.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.159+0000 7f1c5ffff640 1 --2- 172.21.15.12:0/462197394 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1c4c03e210 0x7f1c4c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.159+0000 7f1c5ffff640 10 monclient: get_auth_request con 0x7f1c4c03e210 auth_method 0 2024-09-11T01:33:59.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.162+0000 7f1c5ffff640 1 --2- 172.21.15.12:0/462197394 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1c4c03e210 0x7f1c4c0406d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1c54007c90 tx=0x7f1c54007be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:59.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.164+0000 7f1c5d7fa640 1 -- 172.21.15.12:0/462197394 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1c50012420 con 0x7f1c600a2e90 2024-09-11T01:33:59.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.164+0000 7f1c5d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:59.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.164+0000 7f1c5d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:33:59.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.419+0000 7f1c66cdd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:33:59.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.419+0000 7f1c66cdd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:33:59.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.419+0000 7f1c66cdd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:33:59.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.419+0000 7f1c66cdd640 1 -- 172.21.15.12:0/462197394 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1c6009f680 con 0x7f1c600a2e90 2024-09-11T01:33:59.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.420+0000 7f1c5d7fa640 1 -- 172.21.15.12:0/462197394 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f1c50010cc0 con 0x7f1c600a2e90 2024-09-11T01:33:59.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.420+0000 7f1c5d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:33:59.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.420+0000 7f1c5d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:33:59.422 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:33:59.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.422+0000 7f1c66cdd640 1 -- 172.21.15.12:0/462197394 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1c4c03e210 msgr2=0x7f1c4c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:59.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.422+0000 7f1c66cdd640 1 --2- 172.21.15.12:0/462197394 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1c4c03e210 0x7f1c4c0406d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1c54007c90 tx=0x7f1c54007be0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.422+0000 7f1c66cdd640 10 monclient: shutdown 2024-09-11T01:33:59.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.422+0000 7f1c66cdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:59.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.422+0000 7f1c66cdd640 1 -- 172.21.15.12:0/462197394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c600a2e90 msgr2=0x7f1c60135880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:59.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.422+0000 7f1c66cdd640 1 --2- 172.21.15.12:0/462197394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c600a2e90 0x7f1c60135880 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1c5000c430 tx=0x7f1c5000c8f0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.422+0000 7f1c66cdd640 1 -- 172.21.15.12:0/462197394 shutdown_connections 2024-09-11T01:33:59.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.422+0000 7f1c66cdd640 1 --2- 172.21.15.12:0/462197394 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c6013c160 0x7f1c60139040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.422+0000 7f1c66cdd640 1 --2- 172.21.15.12:0/462197394 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c600a3270 0x7f1c60135dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.422+0000 7f1c66cdd640 1 --2- 172.21.15.12:0/462197394 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1c4c03e210 0x7f1c4c0406d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.422+0000 7f1c66cdd640 1 --2- 172.21.15.12:0/462197394 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c600a2e90 0x7f1c60135880 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.422+0000 7f1c66cdd640 1 -- 172.21.15.12:0/462197394 >> 172.21.15.12:0/462197394 conn(0x7f1c600a8490 msgr2=0x7f1c600a19a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:59.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.422+0000 7f1c66cdd640 1 -- 172.21.15.12:0/462197394 shutdown_connections 2024-09-11T01:33:59.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.422+0000 7f1c66cdd640 1 -- 172.21.15.12:0/462197394 wait complete. 2024-09-11T01:33:59.434 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:33:59.435 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:33:59.435 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:33:59.435 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:33:59.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.536+0000 7fc73586e640 10 monclient: get_monmap_and_config 2024-09-11T01:33:59.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.536+0000 7fc73586e640 10 monclient: build_initial_monmap 2024-09-11T01:33:59.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.536+0000 7fc73586e640 10 monclient: monmap: 2024-09-11T01:33:59.538 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:59.538 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:59.538 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:33:59.537278+0000 2024-09-11T01:33:59.539 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:33:59.537278+0000 2024-09-11T01:33:59.539 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:59.539 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.539 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:33:59.539 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:33:59.539 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:33:59.539 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:33:59.539 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:33:59.539 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.537+0000 7fc73586e640 1 Processor -- start 2024-09-11T01:33:59.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.537+0000 7fc73586e640 1 -- start start 2024-09-11T01:33:59.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.537+0000 7fc73586e640 10 monclient: init 2024-09-11T01:33:59.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.537+0000 7fc73586e640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:59.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.537+0000 7fc73586e640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-11T01:33:59.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.537+0000 7fc73586e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc7301381c0 addr v1:172.21.15.12:6789/0 2024-09-11T01:33:59.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.537+0000 7fc73586e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc7300a6aa0 0x7fc73009cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.537+0000 7fc73586e640 10 monclient(hunting): picked mon.noname-a con 0x7fc7300a6aa0 addr v2:172.21.15.12:3300/0 2024-09-11T01:33:59.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.537+0000 7fc73586e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc73013d000 addr v1:172.21.15.71:6789/0 2024-09-11T01:33:59.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.537+0000 7fc73586e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc73009c310 con 0x7fc7301381c0 2024-09-11T01:33:59.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.537+0000 7fc73586e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc73009c8d0 con 0x7fc73013d000 2024-09-11T01:33:59.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.537+0000 7fc73586e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:59.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.537+0000 7fc73586e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc73006fc90 con 0x7fc7300a6aa0 2024-09-11T01:33:59.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.537+0000 7fc73586e640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:59.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.537+0000 7fc73586e640 10 monclient(hunting): authenticate will time out at 1045.979704s 2024-09-11T01:33:59.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.537+0000 7fc72e7fc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc7300a6aa0 0x7fc73009cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.537+0000 7fc72effd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc7301381c0 0x7fc7301385a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33030/0 (socket says 172.21.15.12:33030) 2024-09-11T01:33:59.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.537+0000 7fc72effd640 1 -- 172.21.15.12:0/256994413 learned_addr learned my addr 172.21.15.12:0/256994413 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:33:59.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.537+0000 7fc72e7fc640 10 monclient(hunting): get_auth_request con 0x7fc7300a6aa0 auth_method 0 2024-09-11T01:33:59.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.537+0000 7fc72e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:59.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.537+0000 7fc72e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:59.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.537+0000 7fc72e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:59.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.537+0000 7fc72e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:59.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.537+0000 7fc72e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:59.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.537+0000 7fc72e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:59.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.538+0000 7fc72e7fc640 10 monclient(hunting): handle_auth_done global_id 54147 payload 274 2024-09-11T01:33:59.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.538+0000 7fc72e7fc640 1 -- 172.21.15.12:0/256994413 >> v1:172.21.15.71:6789/0 conn(0x7fc73013d000 legacy=0x7fc73013f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:59.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.538+0000 7fc72e7fc640 1 -- 172.21.15.12:0/256994413 >> v1:172.21.15.12:6789/0 conn(0x7fc7301381c0 legacy=0x7fc7301385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:59.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.538+0000 7fc72e7fc640 10 monclient: _finish_hunting 0 2024-09-11T01:33:59.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.538+0000 7fc72e7fc640 1 monclient: found mon.noname-a 2024-09-11T01:33:59.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.538+0000 7fc72e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:59.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.538+0000 7fc72e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:33:59.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.538+0000 7fc72e7fc640 1 -- 172.21.15.12:0/256994413 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc73009d510 con 0x7fc7300a6aa0 2024-09-11T01:33:59.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.540+0000 7fc72e7fc640 1 -- 172.21.15.12:0/256994413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7300a6aa0 msgr2=0x7fc73009cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:59.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.540+0000 7fc72e7fc640 1 --2- 172.21.15.12:0/256994413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7300a6aa0 0x7fc73009cfd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc7240097a0 tx=0x7fc724029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7347f304a4294b4b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:59.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.541+0000 7fc72dffb640 1 -- 172.21.15.12:0/256994413 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc724037070 con 0x7fc7300a6aa0 2024-09-11T01:33:59.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.541+0000 7fc72dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:59.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.541+0000 7fc72dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:33:59.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.541+0000 7fc72dffb640 10 monclient: dump: 2024-09-11T01:33:59.550 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:59.551 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:59.551 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.551 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.551 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:59.551 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.551 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:59.551 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:59.551 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:59.551 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.541+0000 7fc72dffb640 10 monclient: _finish_auth 0 2024-09-11T01:33:59.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.541+0000 7fc72dffb640 10 monclient: _check_auth_tickets 2024-09-11T01:33:59.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.541+0000 7fc72dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:59.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.541+0000 7fc72dffb640 1 -- 172.21.15.12:0/256994413 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc724032640 con 0x7fc7300a6aa0 2024-09-11T01:33:59.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.541+0000 7fc73586e640 5 monclient: authenticate success, global_id 54147 2024-09-11T01:33:59.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.541+0000 7fc73586e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:33:59.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.541+0000 7fc72dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:59.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.541+0000 7fc72dffb640 1 -- 172.21.15.12:0/256994413 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc724032a10 con 0x7fc7300a6aa0 2024-09-11T01:33:59.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.541+0000 7fc73586e640 10 monclient: get_monmap_and_config success 2024-09-11T01:33:59.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.541+0000 7fc73586e640 4 set_mon_vals no callback set 2024-09-11T01:33:59.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.541+0000 7fc72dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:59.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.541+0000 7fc72dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:33:59.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.541+0000 7fc72dffb640 10 monclient: dump: 2024-09-11T01:33:59.552 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:59.552 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:59.552 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.552 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.553 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:59.553 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.553 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:59.553 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:59.553 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:59.553 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.541+0000 7fc73586e640 10 monclient: shutdown 2024-09-11T01:33:59.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.541+0000 7fc73586e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:59.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.541+0000 7fc73586e640 1 -- 172.21.15.12:0/256994413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7300a6aa0 msgr2=0x7fc73009cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:59.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.541+0000 7fc73586e640 1 --2- 172.21.15.12:0/256994413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7300a6aa0 0x7fc73009cfd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc7240097a0 tx=0x7fc724029fd0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.541+0000 7fc73586e640 1 -- 172.21.15.12:0/256994413 shutdown_connections 2024-09-11T01:33:59.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.541+0000 7fc73586e640 1 --2- 172.21.15.12:0/256994413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7300a6aa0 0x7fc73009cfd0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.541+0000 7fc73586e640 1 -- 172.21.15.12:0/256994413 >> 172.21.15.12:0/256994413 conn(0x7fc7300a7440 msgr2=0x7fc7300a7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:59.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.541+0000 7fc73586e640 1 -- 172.21.15.12:0/256994413 shutdown_connections 2024-09-11T01:33:59.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.541+0000 7fc73586e640 1 -- 172.21.15.12:0/256994413 wait complete. 2024-09-11T01:33:59.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.542+0000 7fc73586e640 10 monclient: build_initial_monmap 2024-09-11T01:33:59.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.542+0000 7fc73586e640 10 monclient: monmap: 2024-09-11T01:33:59.554 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:59.554 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:33:59.554 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:33:59.554 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:33:59.554 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:59.554 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.554 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:33:59.554 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:33:59.554 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:33:59.554 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.542+0000 7fc73586e640 1 Processor -- start 2024-09-11T01:33:59.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.542+0000 7fc73586e640 1 -- start start 2024-09-11T01:33:59.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.542+0000 7fc73586e640 10 monclient: init 2024-09-11T01:33:59.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc73586e640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:59.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc73586e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:33:59.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc73586e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7300a6aa0 0x7fc730136210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc73586e640 10 monclient(hunting): picked mon.noname-b con 0x7fc7300a6aa0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:33:59.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc73586e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7301381c0 0x7fc730136750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc73586e640 10 monclient(hunting): picked mon.noname-a con 0x7fc7301381c0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:33:59.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc73586e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc73013d000 0x7fc730138d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc73586e640 10 monclient(hunting): picked mon.noname-c con 0x7fc73013d000 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:59.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc73586e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:59.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc73586e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc730140d20 con 0x7fc7301381c0 2024-09-11T01:33:59.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc73586e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:59.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc73586e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc73009c5f0 con 0x7fc7300a6aa0 2024-09-11T01:33:59.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc73586e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:59.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc73586e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc730140ea0 con 0x7fc73013d000 2024-09-11T01:33:59.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc73586e640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:59.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc73586e640 10 monclient(hunting): authenticate will time out at 1045.985305s 2024-09-11T01:33:59.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72e7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7301381c0 0x7fc730136750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72e7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7301381c0 0x7fc730136750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42302/0 (socket says 172.21.15.12:42302) 2024-09-11T01:33:59.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72e7fc640 1 -- 172.21.15.12:0/1034191876 learned_addr learned my addr 172.21.15.12:0/1034191876 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:59.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72e7fc640 10 monclient(hunting): get_auth_request con 0x7fc7301381c0 auth_method 0 2024-09-11T01:33:59.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:59.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:59.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72f7fe640 1 --2- 172.21.15.12:0/1034191876 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc73013d000 0x7fc730138d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72f7fe640 10 monclient(hunting): get_auth_request con 0x7fc73013d000 auth_method 0 2024-09-11T01:33:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72effd640 1 --2- 172.21.15.12:0/1034191876 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7300a6aa0 0x7fc730136210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72effd640 10 monclient(hunting): get_auth_request con 0x7fc7300a6aa0 auth_method 0 2024-09-11T01:33:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:59.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:59.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:59.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72e7fc640 10 monclient(hunting): handle_auth_done global_id 54150 payload 824 2024-09-11T01:33:59.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72e7fc640 1 -- 172.21.15.12:0/1034191876 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc73013d000 msgr2=0x7fc730138d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:59.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72e7fc640 1 --2- 172.21.15.12:0/1034191876 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc73013d000 0x7fc730138d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72e7fc640 1 -- 172.21.15.12:0/1034191876 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7300a6aa0 msgr2=0x7fc730136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:59.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72e7fc640 1 --2- 172.21.15.12:0/1034191876 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7300a6aa0 0x7fc730136210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72e7fc640 10 monclient: _finish_hunting 0 2024-09-11T01:33:59.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72e7fc640 1 monclient: found mon.noname-a 2024-09-11T01:33:59.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:59.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:33:59.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72e7fc640 1 -- 172.21.15.12:0/1034191876 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc73013bd80 con 0x7fc7301381c0 2024-09-11T01:33:59.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.543+0000 7fc72effd640 1 --2- 172.21.15.12:0/1034191876 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7300a6aa0 0x7fc730136210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:33:59.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.546+0000 7fc72e7fc640 1 -- 172.21.15.12:0/1034191876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7301381c0 msgr2=0x7fc730136750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:59.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.546+0000 7fc72e7fc640 1 --2- 172.21.15.12:0/1034191876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7301381c0 0x7fc730136750 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc7240097a0 tx=0x7fc7240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:59.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.546+0000 7fc717fff640 1 -- 172.21.15.12:0/1034191876 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc724041070 con 0x7fc7301381c0 2024-09-11T01:33:59.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.546+0000 7fc717fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:59.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.546+0000 7fc717fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:33:59.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.546+0000 7fc717fff640 10 monclient: dump: 2024-09-11T01:33:59.559 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:59.559 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:59.559 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.559 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.559 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:59.559 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.559 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:59.559 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:59.560 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:59.560 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.546+0000 7fc717fff640 10 monclient: _finish_auth 0 2024-09-11T01:33:59.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.546+0000 7fc717fff640 10 monclient: _check_auth_tickets 2024-09-11T01:33:59.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.546+0000 7fc717fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:59.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.546+0000 7fc717fff640 1 -- 172.21.15.12:0/1034191876 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc72403c880 con 0x7fc7301381c0 2024-09-11T01:33:59.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.546+0000 7fc73586e640 5 monclient: authenticate success, global_id 54150 2024-09-11T01:33:59.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.546+0000 7fc73586e640 10 monclient: _renew_subs 2024-09-11T01:33:59.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.546+0000 7fc73586e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:33:59.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.546+0000 7fc73586e640 1 -- 172.21.15.12:0/1034191876 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc73013cca0 con 0x7fc7301381c0 2024-09-11T01:33:59.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.546+0000 7fc717fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:59.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.546+0000 7fc717fff640 1 -- 172.21.15.12:0/1034191876 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc724037040 con 0x7fc7301381c0 2024-09-11T01:33:59.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.546+0000 7fc717fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:59.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.546+0000 7fc717fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:33:59.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.546+0000 7fc717fff640 10 monclient: dump: 2024-09-11T01:33:59.561 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:59.561 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:59.561 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.561 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.561 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:59.561 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.561 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:59.561 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:59.561 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:59.561 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.546+0000 7fc73586e640 10 monclient: _renew_subs 2024-09-11T01:33:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.546+0000 7fc72d7fa640 4 set_mon_vals no callback set 2024-09-11T01:33:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.546+0000 7fc73586e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:33:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.546+0000 7fc73586e640 1 -- 172.21.15.12:0/1034191876 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc73013baa0 con 0x7fc7301381c0 2024-09-11T01:33:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.546+0000 7fc73586e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.547+0000 7fc73586e640 1 --2- 172.21.15.12:0/1034191876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7300a3f50 0x7fc7301392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.547+0000 7fc73586e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.547+0000 7fc73586e640 1 -- 172.21.15.12:0/1034191876 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc73009cfd0 con 0x7fc7300a3f50 2024-09-11T01:33:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.547+0000 7fc72effd640 1 --2- 172.21.15.12:0/1034191876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7300a3f50 0x7fc7301392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.547+0000 7fc72effd640 10 monclient: get_auth_request con 0x7fc7300a3f50 auth_method 0 2024-09-11T01:33:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.547+0000 7fc72effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.547+0000 7fc72effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.547+0000 7fc72effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.547+0000 7fc717fff640 1 -- 172.21.15.12:0/1034191876 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fc72403cc70 con 0x7fc7301381c0 2024-09-11T01:33:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.547+0000 7fc72effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:59.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.547+0000 7fc72effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:59.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.547+0000 7fc72effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:59.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.547+0000 7fc72effd640 10 monclient(hunting): handle_auth_done global_id 54153 payload 824 2024-09-11T01:33:59.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.547+0000 7fc72effd640 1 -- 172.21.15.12:0/1034191876 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc730136c90 con 0x7fc7300a3f50 2024-09-11T01:33:59.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.547+0000 7fc717fff640 1 --2- 172.21.15.12:0/1034191876 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc71c03e240 0x7fc71c040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.547+0000 7fc717fff640 1 -- 172.21.15.12:0/1034191876 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4787+0+0 (secure 0 0 0) 0x7fc724073670 con 0x7fc7301381c0 2024-09-11T01:33:59.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.547+0000 7fc72f7fe640 1 --2- 172.21.15.12:0/1034191876 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc71c03e240 0x7fc71c040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.547+0000 7fc72f7fe640 10 monclient: get_auth_request con 0x7fc71c03e240 auth_method 0 2024-09-11T01:33:59.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.549+0000 7fc72effd640 1 -- 172.21.15.12:0/1034191876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7300a3f50 msgr2=0x7fc7301392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:59.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.549+0000 7fc72effd640 1 --2- 172.21.15.12:0/1034191876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7300a3f50 0x7fc7301392d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc72000e520 tx=0x7fc72000e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:59.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.549+0000 7fc717fff640 1 -- 172.21.15.12:0/1034191876 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc720005cf0 con 0x7fc7300a3f50 2024-09-11T01:33:59.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.549+0000 7fc717fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:59.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.549+0000 7fc717fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:33:59.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.549+0000 7fc717fff640 10 monclient: dump: 2024-09-11T01:33:59.566 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:59.566 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:59.566 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.566 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.566 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:59.567 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.567 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:59.567 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:59.567 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:59.567 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.550+0000 7fc717fff640 1 -- 172.21.15.12:0/1034191876 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc720018070 con 0x7fc7300a3f50 2024-09-11T01:33:59.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.550+0000 7fc717fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:59.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.550+0000 7fc717fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:33:59.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.550+0000 7fc717fff640 1 -- 172.21.15.12:0/1034191876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7300a3f50 msgr2=0x7fc7301392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:59.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.550+0000 7fc717fff640 1 --2- 172.21.15.12:0/1034191876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7300a3f50 0x7fc7301392d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc72000e520 tx=0x7fc72000e9e0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.550+0000 7fc717fff640 1 -- 172.21.15.12:0/1034191876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7300a3f50 msgr2=0x7fc7301392d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:33:59.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.550+0000 7fc717fff640 1 --2- 172.21.15.12:0/1034191876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7300a3f50 0x7fc7301392d0 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc72000e520 tx=0x7fc72000e9e0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.550+0000 7fc72f7fe640 1 --2- 172.21.15.12:0/1034191876 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc71c03e240 0x7fc71c040700 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc73013d9e0 tx=0x7fc728003c60 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:59.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.556+0000 7fc73586e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:33:59.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.556+0000 7fc73586e640 1 --2- 172.21.15.12:0/1034191876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc730139810 0x7fc73009df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.556+0000 7fc73586e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:59.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.556+0000 7fc73586e640 1 -- 172.21.15.12:0/1034191876 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc730139c20 con 0x7fc730139810 2024-09-11T01:33:59.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.556+0000 7fc72effd640 1 --2- 172.21.15.12:0/1034191876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc730139810 0x7fc73009df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.556+0000 7fc72effd640 10 monclient: get_auth_request con 0x7fc730139810 auth_method 0 2024-09-11T01:33:59.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.556+0000 7fc72effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:59.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.556+0000 7fc72effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:59.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.556+0000 7fc72effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:59.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.556+0000 7fc72effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:59.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.556+0000 7fc72effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:59.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.556+0000 7fc72effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:59.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.556+0000 7fc72effd640 10 monclient(hunting): handle_auth_done global_id 54156 payload 824 2024-09-11T01:33:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.556+0000 7fc72effd640 1 -- 172.21.15.12:0/1034191876 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc730069840 con 0x7fc730139810 2024-09-11T01:33:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.558+0000 7fc72effd640 1 -- 172.21.15.12:0/1034191876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc730139810 msgr2=0x7fc73009df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.558+0000 7fc72effd640 1 --2- 172.21.15.12:0/1034191876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc730139810 0x7fc73009df90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc72000e520 tx=0x7fc720004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.559+0000 7fc717fff640 1 -- 172.21.15.12:0/1034191876 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7200072e0 con 0x7fc730139810 2024-09-11T01:33:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.559+0000 7fc717fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.559+0000 7fc717fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:33:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.559+0000 7fc717fff640 10 monclient: dump: 2024-09-11T01:33:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:59.570 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:59.570 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:59.570 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.559+0000 7fc717fff640 1 -- 172.21.15.12:0/1034191876 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7fc7200072e0 con 0x7fc730139810 2024-09-11T01:33:59.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.559+0000 7fc717fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:33:59.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.559+0000 7fc717fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:33:59.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.559+0000 7fc717fff640 1 -- 172.21.15.12:0/1034191876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc730139810 msgr2=0x7fc73009df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:59.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.559+0000 7fc717fff640 1 --2- 172.21.15.12:0/1034191876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc730139810 0x7fc73009df90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc72000e520 tx=0x7fc720004940 comp rx=0 tx=0).stop 2024-09-11T01:33:59.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.559+0000 7fc717fff640 1 -- 172.21.15.12:0/1034191876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc730139810 msgr2=0x7fc73009df90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:33:59.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.559+0000 7fc717fff640 1 --2- 172.21.15.12:0/1034191876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc730139810 0x7fc73009df90 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc72000e520 tx=0x7fc720004940 comp rx=0 tx=0).stop 2024-09-11T01:33:59.570 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:33:59.571 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:33:59.571 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:33:59.571 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:33:59.571 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 56, 2024-09-11T01:33:59.571 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 29540, 2024-09-11T01:33:59.571 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:33:59.571 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:33:59.571 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:33:59.571 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:33:59.571 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:59.571 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 4, 2024-09-11T01:33:59.571 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:33:59.571 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:33:59.572 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:33:59.572 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:33:59.572 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:33:59.572 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:33:59.572 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:33:59.572 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:33:59.572 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:33:59.572 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:33:59.572 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:33:59.572 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:33:59.572 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:33:59.572 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:33:59.573 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:59.573 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:33:59.573 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:33:59.573 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:33:59.573 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:33:59.573 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:33:59.573 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:33:59.573 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:33:59.573 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:33:59.573 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:33:59.573 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:33:59.573 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:33:59.573 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:33:59.573 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:33:59.574 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:59.574 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.574 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:33:59.574 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:33:59.574 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:33:59.574 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:33:59.574 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:33:59.574 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:33:59.574 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:33:59.574 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:33:59.574 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:33:59.574 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:33:59.574 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:33:59.575 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:33:59.575 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:33:59.575 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:33:59.575 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:33:59.575 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:33:59.575 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:33:59.575 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:33:59.575 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:33:59.575 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:33:59.575 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:33:59.575 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:33:59.575 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:33:59.575 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:33:59.575 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:33:59.576 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:33:59.576 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:33:59.576 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:33:59.576 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:59.576 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:33:59.576 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.576 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:33:59.576 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.576 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:33:59.576 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:33:59.576 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:33:59.576 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:33:59.577 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.577 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:33:59.577 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:33:59.577 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:59.577 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.577 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.577 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:33:59.577 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:33:59.577 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:59.577 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:59.577 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:59.577 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.577 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:33:59.577 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:33:59.578 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:33:59.578 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:33:59.578 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:33:59.578 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.578 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.578 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:33:59.578 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:33:59.578 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:33:59.578 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:33:59.578 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.578 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:33:59.578 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:33:59.578 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:59.579 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.579 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.579 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:33:59.579 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:33:59.579 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:59.579 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:59.579 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:59.579 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.579 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:33:59.579 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:33:59.579 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:33:59.579 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:33:59.579 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:33:59.580 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.580 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.580 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:33:59.580 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:33:59.580 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:33:59.580 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:33:59.580 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.580 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:33:59.580 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:33:59.580 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:59.580 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.580 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.580 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:33:59.580 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:33:59.581 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:59.581 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:59.581 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:59.581 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.581 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:33:59.581 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:33:59.581 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:33:59.581 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:33:59.581 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:33:59.581 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:59.581 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:59.581 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.581 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:33:59.581 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:33:59.582 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.582 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:33:59.582 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:33:59.582 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:33:59.582 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:59.582 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:59.582 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:33:59.582 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.582 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:33:59.582 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:33:59.583 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:33:59.583 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:59.583 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:59.583 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:33:59.583 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.583 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:33:59.583 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:33:59.583 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:33:59.583 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:59.583 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:59.583 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.583 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:33:59.583 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:33:59.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.559+0000 7fc73586e640 1 -- 172.21.15.12:0/1034191876 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc71c03e240 msgr2=0x7fc71c040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:59.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.559+0000 7fc73586e640 1 --2- 172.21.15.12:0/1034191876 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc71c03e240 0x7fc71c040700 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc73013d9e0 tx=0x7fc728003c60 comp rx=0 tx=0).stop 2024-09-11T01:33:59.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.559+0000 7fc73586e640 10 monclient: shutdown 2024-09-11T01:33:59.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.559+0000 7fc73586e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:59.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.559+0000 7fc73586e640 1 -- 172.21.15.12:0/1034191876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7301381c0 msgr2=0x7fc730136750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:59.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.559+0000 7fc73586e640 1 --2- 172.21.15.12:0/1034191876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7301381c0 0x7fc730136750 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc7240097a0 tx=0x7fc7240046c0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.559+0000 7fc72f7fe640 1 -- 172.21.15.12:0/1034191876 reap_dead start 2024-09-11T01:33:59.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.559+0000 7fc73586e640 1 -- 172.21.15.12:0/1034191876 shutdown_connections 2024-09-11T01:33:59.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.559+0000 7fc73586e640 1 --2- 172.21.15.12:0/1034191876 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7301381c0 0x7fc730136750 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.559+0000 7fc73586e640 1 -- 172.21.15.12:0/1034191876 >> 172.21.15.12:0/1034191876 conn(0x7fc7300a7440 msgr2=0x7fc73013fa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:59.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.559+0000 7fc73586e640 1 -- 172.21.15.12:0/1034191876 shutdown_connections 2024-09-11T01:33:59.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.559+0000 7fc73586e640 1 -- 172.21.15.12:0/1034191876 wait complete. 2024-09-11T01:33:59.586 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:33:59.586 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:33:59.586 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:33:59.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.687+0000 7fcacfcf6640 10 monclient: get_monmap_and_config 2024-09-11T01:33:59.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.687+0000 7fcacfcf6640 10 monclient: build_initial_monmap 2024-09-11T01:33:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.687+0000 7fcacfcf6640 10 monclient: monmap: 2024-09-11T01:33:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:33:59.687928+0000 2024-09-11T01:33:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:33:59.687928+0000 2024-09-11T01:33:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:33:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:33:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:33:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:33:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:33:59.690 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.687+0000 7fcacfcf6640 1 Processor -- start 2024-09-11T01:33:59.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.687+0000 7fcacfcf6640 1 -- start start 2024-09-11T01:33:59.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacfcf6640 10 monclient: init 2024-09-11T01:33:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacfcf6640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacfcf6640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-11T01:33:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacfcf6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcac8137e50 0x7fcac809d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacfcf6640 10 monclient(hunting): picked mon.noname-c con 0x7fcac8137e50 addr v2:172.21.15.71:3300/0 2024-09-11T01:33:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacfcf6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8138230 0x7fcac8138660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacfcf6640 10 monclient(hunting): picked mon.noname-b con 0x7fcac8138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacfcf6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcac8134cb0 addr v1:172.21.15.71:6789/0 2024-09-11T01:33:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacfcf6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcac80a6580 con 0x7fcac8134cb0 2024-09-11T01:33:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacfcf6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacfcf6640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fcac80a62d0 con 0x7fcac8137e50 2024-09-11T01:33:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacfcf6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacfcf6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcac809bb40 con 0x7fcac8138230 2024-09-11T01:33:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacfcf6640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacfcf6640 10 monclient(hunting): authenticate will time out at 1046.130530s 2024-09-11T01:33:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacd291640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8138230 0x7fcac8138660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacd291640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8138230 0x7fcac8138660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:37322/0 (socket says 172.21.15.12:37322) 2024-09-11T01:33:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacd291640 1 -- 172.21.15.12:0/1264251748 learned_addr learned my addr 172.21.15.12:0/1264251748 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacd291640 10 monclient(hunting): get_auth_request con 0x7fcac8138230 auth_method 0 2024-09-11T01:33:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacd291640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacd291640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacd291640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacd291640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacd291640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacd291640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacd291640 10 monclient(hunting): handle_auth_done global_id 44237 payload 274 2024-09-11T01:33:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacda92640 1 --2- 172.21.15.12:0/1264251748 >> v2:172.21.15.71:3300/0 conn(0x7fcac8137e50 0x7fcac809d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacd291640 1 -- 172.21.15.12:0/1264251748 >> v2:172.21.15.71:3300/0 conn(0x7fcac8137e50 msgr2=0x7fcac809d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:59.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacd291640 1 --2- 172.21.15.12:0/1264251748 >> v2:172.21.15.71:3300/0 conn(0x7fcac8137e50 0x7fcac809d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacd291640 1 -- 172.21.15.12:0/1264251748 >> v1:172.21.15.71:6789/0 conn(0x7fcac8134cb0 legacy=0x7fcac8135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:59.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacd291640 10 monclient: _finish_hunting 0 2024-09-11T01:33:59.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacd291640 1 monclient: found mon.noname-b 2024-09-11T01:33:59.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacd291640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:59.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacd291640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:33:59.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.688+0000 7fcacd291640 1 -- 172.21.15.12:0/1264251748 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcac809c4a0 con 0x7fcac8138230 2024-09-11T01:33:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.691+0000 7fcacd291640 1 -- 172.21.15.12:0/1264251748 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8138230 msgr2=0x7fcac8138660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.691+0000 7fcacd291640 1 --2- 172.21.15.12:0/1264251748 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8138230 0x7fcac8138660 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcab8009870 tx=0x7fcab802a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce4cd0da711792c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.692+0000 7fcacca90640 1 -- 172.21.15.12:0/1264251748 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab8037070 con 0x7fcac8138230 2024-09-11T01:33:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.692+0000 7fcacca90640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.692+0000 7fcacca90640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:33:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.692+0000 7fcacca90640 10 monclient: dump: 2024-09-11T01:33:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:59.700 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.692+0000 7fcacca90640 10 monclient: _finish_auth 0 2024-09-11T01:33:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.692+0000 7fcacca90640 10 monclient: _check_auth_tickets 2024-09-11T01:33:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.692+0000 7fcacca90640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.692+0000 7fcacfcf6640 5 monclient: authenticate success, global_id 44237 2024-09-11T01:33:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.692+0000 7fcacca90640 1 -- 172.21.15.12:0/1264251748 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcab8032640 con 0x7fcac8138230 2024-09-11T01:33:59.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.692+0000 7fcacca90640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:59.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.692+0000 7fcacfcf6640 10 monclient: get_monmap_and_config success 2024-09-11T01:33:59.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.692+0000 7fcacfcf6640 4 set_mon_vals no callback set 2024-09-11T01:33:59.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.692+0000 7fcacca90640 1 -- 172.21.15.12:0/1264251748 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab8032a10 con 0x7fcac8138230 2024-09-11T01:33:59.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.692+0000 7fcacca90640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:59.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.692+0000 7fcacca90640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:59.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.692+0000 7fcacca90640 10 monclient: dump: 2024-09-11T01:33:59.701 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:59.701 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:59.701 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.701 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.701 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:59.701 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.701 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:59.702 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:59.702 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:59.702 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.692+0000 7fcacfcf6640 10 monclient: shutdown 2024-09-11T01:33:59.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.692+0000 7fcacfcf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:59.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.692+0000 7fcacfcf6640 1 -- 172.21.15.12:0/1264251748 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8138230 msgr2=0x7fcac8138660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:59.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.692+0000 7fcacfcf6640 1 --2- 172.21.15.12:0/1264251748 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8138230 0x7fcac8138660 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcab8009870 tx=0x7fcab802a0a0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.692+0000 7fcacfcf6640 1 -- 172.21.15.12:0/1264251748 shutdown_connections 2024-09-11T01:33:59.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.692+0000 7fcacfcf6640 1 --2- 172.21.15.12:0/1264251748 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8138230 0x7fcac8138660 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.692+0000 7fcacfcf6640 1 --2- 172.21.15.12:0/1264251748 >> v2:172.21.15.71:3300/0 conn(0x7fcac8137e50 0x7fcac809d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.692+0000 7fcacfcf6640 1 -- 172.21.15.12:0/1264251748 >> 172.21.15.12:0/1264251748 conn(0x7fcac80a7020 msgr2=0x7fcac80a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:59.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.692+0000 7fcacfcf6640 1 -- 172.21.15.12:0/1264251748 shutdown_connections 2024-09-11T01:33:59.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.692+0000 7fcacfcf6640 1 -- 172.21.15.12:0/1264251748 wait complete. 2024-09-11T01:33:59.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.693+0000 7fcacfcf6640 10 monclient: build_initial_monmap 2024-09-11T01:33:59.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.693+0000 7fcacfcf6640 10 monclient: monmap: 2024-09-11T01:33:59.703 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:59.703 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:33:59.703 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:33:59.703 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:33:59.703 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:59.703 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.703 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:33:59.703 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:33:59.703 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:33:59.703 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.693+0000 7fcacfcf6640 1 Processor -- start 2024-09-11T01:33:59.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.693+0000 7fcacfcf6640 1 -- start start 2024-09-11T01:33:59.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.693+0000 7fcacfcf6640 10 monclient: init 2024-09-11T01:33:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacfcf6640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacfcf6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:33:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacfcf6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcac8134cb0 0x7fcac81cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacfcf6640 10 monclient(hunting): picked mon.noname-a con 0x7fcac8134cb0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:33:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacfcf6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8137e50 0x7fcac81cd910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacfcf6640 10 monclient(hunting): picked mon.noname-c con 0x7fcac8137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacfcf6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcac8138230 0x7fcac8121760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacfcf6640 10 monclient(hunting): picked mon.noname-b con 0x7fcac8138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:33:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacfcf6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacfcf6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcac806f710 con 0x7fcac8134cb0 2024-09-11T01:33:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacfcf6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacfcf6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fcac8138ba0 con 0x7fcac8138230 2024-09-11T01:33:59.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacfcf6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacfcf6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcac806f890 con 0x7fcac8137e50 2024-09-11T01:33:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacfcf6640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacfcf6640 10 monclient(hunting): authenticate will time out at 1046.136284s 2024-09-11T01:33:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacd291640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8137e50 0x7fcac81cd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacd291640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8137e50 0x7fcac81cd910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:37336/0 (socket says 172.21.15.12:37336) 2024-09-11T01:33:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacda92640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcac8134cb0 0x7fcac81cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacd291640 1 -- 172.21.15.12:0/3971861257 learned_addr learned my addr 172.21.15.12:0/3971861257 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacd291640 10 monclient(hunting): get_auth_request con 0x7fcac8137e50 auth_method 0 2024-09-11T01:33:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacd291640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacd291640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacd291640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacda92640 10 monclient(hunting): get_auth_request con 0x7fcac8134cb0 auth_method 0 2024-09-11T01:33:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacda92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacda92640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacda92640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacd291640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacd291640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacd291640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacda92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacda92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacda92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcace293640 1 --2- 172.21.15.12:0/3971861257 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcac8138230 0x7fcac8121760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcace293640 10 monclient(hunting): get_auth_request con 0x7fcac8138230 auth_method 0 2024-09-11T01:33:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcace293640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcace293640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcace293640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacd291640 10 monclient(hunting): handle_auth_done global_id 44240 payload 824 2024-09-11T01:33:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacd291640 1 -- 172.21.15.12:0/3971861257 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcac8138230 msgr2=0x7fcac8121760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacd291640 1 --2- 172.21.15.12:0/3971861257 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcac8138230 0x7fcac8121760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacd291640 1 -- 172.21.15.12:0/3971861257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcac8134cb0 msgr2=0x7fcac81cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacd291640 1 --2- 172.21.15.12:0/3971861257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcac8134cb0 0x7fcac81cd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacd291640 10 monclient: _finish_hunting 0 2024-09-11T01:33:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacd291640 1 monclient: found mon.noname-c 2024-09-11T01:33:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacd291640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacd291640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:33:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcacd291640 1 -- 172.21.15.12:0/3971861257 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcac80a5770 con 0x7fcac8137e50 2024-09-11T01:33:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.694+0000 7fcace293640 1 --2- 172.21.15.12:0/3971861257 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcac8138230 0x7fcac8121760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:33:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.696+0000 7fcacd291640 1 -- 172.21.15.12:0/3971861257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8137e50 msgr2=0x7fcac81cd910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.696+0000 7fcacd291640 1 --2- 172.21.15.12:0/3971861257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8137e50 0x7fcac81cd910 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcab8009870 tx=0x7fcab80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:59.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.697+0000 7fcabe7fc640 1 -- 172.21.15.12:0/3971861257 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab8041020 con 0x7fcac8137e50 2024-09-11T01:33:59.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.697+0000 7fcabe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:59.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.697+0000 7fcabe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:33:59.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.697+0000 7fcabe7fc640 10 monclient: dump: 2024-09-11T01:33:59.708 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:59.708 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:59.708 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.708 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.708 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:59.708 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.708 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:59.708 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:59.708 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:59.709 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.697+0000 7fcabe7fc640 10 monclient: _finish_auth 0 2024-09-11T01:33:59.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.697+0000 7fcabe7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:33:59.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.697+0000 7fcabe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:59.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.697+0000 7fcabe7fc640 1 -- 172.21.15.12:0/3971861257 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcab803d830 con 0x7fcac8137e50 2024-09-11T01:33:59.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.697+0000 7fcacfcf6640 5 monclient: authenticate success, global_id 44240 2024-09-11T01:33:59.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.697+0000 7fcacfcf6640 10 monclient: _renew_subs 2024-09-11T01:33:59.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.697+0000 7fcacfcf6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:59.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.697+0000 7fcacfcf6640 1 -- 172.21.15.12:0/3971861257 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcac80a1710 con 0x7fcac8137e50 2024-09-11T01:33:59.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.697+0000 7fcabe7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:59.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.697+0000 7fcabe7fc640 1 -- 172.21.15.12:0/3971861257 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab8037040 con 0x7fcac8137e50 2024-09-11T01:33:59.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.697+0000 7fcabe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:59.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.697+0000 7fcabe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:59.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.697+0000 7fcabe7fc640 10 monclient: dump: 2024-09-11T01:33:59.710 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:59.710 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:59.710 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.710 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.710 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:59.710 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.710 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:59.710 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:59.710 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:59.710 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.697+0000 7fcacfcf6640 10 monclient: _renew_subs 2024-09-11T01:33:59.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.697+0000 7fcacfcf6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:59.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.697+0000 7fcabffff640 4 set_mon_vals no callback set 2024-09-11T01:33:59.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.697+0000 7fcacfcf6640 1 -- 172.21.15.12:0/3971861257 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcac8125260 con 0x7fcac8137e50 2024-09-11T01:33:59.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.698+0000 7fcacfcf6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:59.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.698+0000 7fcacfcf6640 1 --2- 172.21.15.12:0/3971861257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8121ca0 0x7fcac8122080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.698+0000 7fcacfcf6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:59.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.698+0000 7fcacfcf6640 1 -- 172.21.15.12:0/3971861257 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcac806a060 con 0x7fcac8121ca0 2024-09-11T01:33:59.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.698+0000 7fcabe7fc640 1 -- 172.21.15.12:0/3971861257 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fcab803a030 con 0x7fcac8137e50 2024-09-11T01:33:59.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.698+0000 7fcacda92640 1 --2- 172.21.15.12:0/3971861257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8121ca0 0x7fcac8122080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.698+0000 7fcacda92640 10 monclient: get_auth_request con 0x7fcac8121ca0 auth_method 0 2024-09-11T01:33:59.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.698+0000 7fcacda92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:59.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.698+0000 7fcacda92640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:59.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.698+0000 7fcacda92640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:59.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.698+0000 7fcacda92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:59.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.698+0000 7fcacda92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:59.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.698+0000 7fcacda92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:59.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.698+0000 7fcacda92640 10 monclient(hunting): handle_auth_done global_id 44243 payload 824 2024-09-11T01:33:59.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.698+0000 7fcabe7fc640 1 --2- 172.21.15.12:0/3971861257 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcaac03e210 0x7fcaac0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.698+0000 7fcacda92640 1 -- 172.21.15.12:0/3971861257 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcac8135950 con 0x7fcac8121ca0 2024-09-11T01:33:59.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.698+0000 7fcabe7fc640 1 -- 172.21.15.12:0/3971861257 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(172..172 src has 1..172) ==== 4787+0+0 (secure 0 0 0) 0x7fcab80731f0 con 0x7fcac8137e50 2024-09-11T01:33:59.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.698+0000 7fcace293640 1 --2- 172.21.15.12:0/3971861257 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcaac03e210 0x7fcaac0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.698+0000 7fcace293640 10 monclient: get_auth_request con 0x7fcaac03e210 auth_method 0 2024-09-11T01:33:59.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.700+0000 7fcacda92640 1 -- 172.21.15.12:0/3971861257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8121ca0 msgr2=0x7fcac8122080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:59.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.700+0000 7fcacda92640 1 --2- 172.21.15.12:0/3971861257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8121ca0 0x7fcac8122080 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcac400f7e0 tx=0x7fcac400fca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:59.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.700+0000 7fcabe7fc640 1 -- 172.21.15.12:0/3971861257 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac400de30 con 0x7fcac8121ca0 2024-09-11T01:33:59.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.700+0000 7fcabe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:59.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.700+0000 7fcabe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:59.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.700+0000 7fcabe7fc640 10 monclient: dump: 2024-09-11T01:33:59.717 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:59.717 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:59.717 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.717 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.717 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:59.717 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.717 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:59.717 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:59.717 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:59.717 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.701+0000 7fcabe7fc640 1 -- 172.21.15.12:0/3971861257 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcac400de30 con 0x7fcac8121ca0 2024-09-11T01:33:59.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.701+0000 7fcabe7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:59.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.701+0000 7fcabe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:33:59.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.701+0000 7fcabe7fc640 1 -- 172.21.15.12:0/3971861257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8121ca0 msgr2=0x7fcac8122080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:59.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.701+0000 7fcabe7fc640 1 --2- 172.21.15.12:0/3971861257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8121ca0 0x7fcac8122080 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcac400f7e0 tx=0x7fcac400fca0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.701+0000 7fcabe7fc640 1 -- 172.21.15.12:0/3971861257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8121ca0 msgr2=0x7fcac8122080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:33:59.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.701+0000 7fcabe7fc640 1 --2- 172.21.15.12:0/3971861257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8121ca0 0x7fcac8122080 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcac400f7e0 tx=0x7fcac400fca0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.701+0000 7fcace293640 1 --2- 172.21.15.12:0/3971861257 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcaac03e210 0x7fcaac0406d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcac0001dd0 tx=0x7fcac0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:59.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.707+0000 7fcacfcf6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:33:59.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.707+0000 7fcacfcf6640 1 --2- 172.21.15.12:0/3971861257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8136b70 0x7fcac809e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.707+0000 7fcacfcf6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:59.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.707+0000 7fcacfcf6640 1 -- 172.21.15.12:0/3971861257 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcac81373e0 con 0x7fcac8136b70 2024-09-11T01:33:59.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.707+0000 7fcacda92640 1 --2- 172.21.15.12:0/3971861257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8136b70 0x7fcac809e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.707+0000 7fcacda92640 10 monclient: get_auth_request con 0x7fcac8136b70 auth_method 0 2024-09-11T01:33:59.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.707+0000 7fcacda92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:59.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.707+0000 7fcacda92640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:59.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.707+0000 7fcacda92640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:59.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.707+0000 7fcacda92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:59.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.707+0000 7fcacda92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:59.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.707+0000 7fcacda92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:59.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.707+0000 7fcacda92640 10 monclient(hunting): handle_auth_done global_id 44246 payload 824 2024-09-11T01:33:59.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.707+0000 7fcacda92640 1 -- 172.21.15.12:0/3971861257 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcac8069950 con 0x7fcac8136b70 2024-09-11T01:33:59.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.710+0000 7fcacda92640 1 -- 172.21.15.12:0/3971861257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8136b70 msgr2=0x7fcac809e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:59.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.710+0000 7fcacda92640 1 --2- 172.21.15.12:0/3971861257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8136b70 0x7fcac809e420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcac400f7e0 tx=0x7fcac40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:59.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.710+0000 7fcabe7fc640 1 -- 172.21.15.12:0/3971861257 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac4029020 con 0x7fcac8136b70 2024-09-11T01:33:59.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.710+0000 7fcabe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:59.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.710+0000 7fcabe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:59.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.710+0000 7fcabe7fc640 10 monclient: dump: 2024-09-11T01:33:59.720 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:59.720 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:59.720 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.720 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.720 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:59.720 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.720 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:59.720 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:59.720 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:59.720 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.710+0000 7fcabe7fc640 1 -- 172.21.15.12:0/3971861257 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4540 (secure 0 0 0) 0x7fcac4013440 con 0x7fcac8136b70 2024-09-11T01:33:59.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.710+0000 7fcabe7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:33:59.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.710+0000 7fcabe7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:33:59.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.710+0000 7fcabe7fc640 1 -- 172.21.15.12:0/3971861257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8136b70 msgr2=0x7fcac809e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:59.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.710+0000 7fcabe7fc640 1 --2- 172.21.15.12:0/3971861257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8136b70 0x7fcac809e420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcac400f7e0 tx=0x7fcac40046c0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.710+0000 7fcabe7fc640 1 -- 172.21.15.12:0/3971861257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8136b70 msgr2=0x7fcac809e420 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:33:59.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.710+0000 7fcabe7fc640 1 --2- 172.21.15.12:0/3971861257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8136b70 0x7fcac809e420 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcac400f7e0 tx=0x7fcac40046c0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.721 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:33:59.721 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:33:59.721 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:33:59.721 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:33:59.721 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 56, 2024-09-11T01:33:59.722 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 83655, 2024-09-11T01:33:59.722 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:33:59.722 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:33:59.722 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:33:59.722 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:33:59.722 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:59.722 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 4, 2024-09-11T01:33:59.722 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:33:59.722 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:33:59.722 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:33:59.722 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:33:59.722 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:33:59.722 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:33:59.722 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:33:59.723 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:33:59.723 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:33:59.723 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:33:59.723 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:33:59.723 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:33:59.723 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:33:59.723 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:33:59.723 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:59.723 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:33:59.723 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:33:59.723 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:33:59.723 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:33:59.723 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:33:59.724 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:33:59.724 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:33:59.724 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:33:59.724 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:33:59.724 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:33:59.724 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:33:59.724 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:33:59.724 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:33:59.724 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:59.724 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.724 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:33:59.724 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:33:59.724 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:33:59.725 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:33:59.725 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:33:59.725 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:33:59.725 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:33:59.725 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:33:59.725 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:33:59.725 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:33:59.725 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:33:59.725 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:33:59.725 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:33:59.725 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:33:59.725 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:33:59.725 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:33:59.725 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:33:59.726 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:33:59.726 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:33:59.726 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:33:59.726 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:33:59.726 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:33:59.726 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:33:59.726 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:33:59.726 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:33:59.726 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:33:59.726 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:33:59.726 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:33:59.726 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:59.726 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:33:59.726 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.727 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:33:59.727 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.727 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:33:59.727 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:33:59.727 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:33:59.727 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:33:59.727 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.727 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:33:59.727 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:33:59.727 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:59.727 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.727 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.727 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:33:59.728 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:33:59.728 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:59.728 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:59.728 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:59.728 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.728 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:33:59.728 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:33:59.728 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:33:59.728 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:33:59.728 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:33:59.728 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.728 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.728 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:33:59.728 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:33:59.729 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:33:59.729 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:33:59.729 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.729 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:33:59.729 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:33:59.729 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:59.729 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.729 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.729 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:33:59.729 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:33:59.729 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:59.729 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:59.729 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:59.730 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.730 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:33:59.730 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:33:59.730 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:33:59.730 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:33:59.730 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:33:59.730 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.730 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.730 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:33:59.730 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:33:59.730 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:33:59.730 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:33:59.730 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.731 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:33:59.731 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:33:59.731 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:59.731 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.731 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.731 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:33:59.731 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:33:59.731 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:59.731 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:59.731 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:59.731 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.731 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:33:59.731 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:33:59.732 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:33:59.732 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:33:59.732 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:33:59.732 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:59.732 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:59.732 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.732 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:33:59.732 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:33:59.732 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.732 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:33:59.732 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:33:59.732 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:33:59.732 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:59.732 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:59.733 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:33:59.733 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.733 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:33:59.733 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:33:59.733 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:33:59.733 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:59.733 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:59.733 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.733 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:33:59.733 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:33:59.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.710+0000 7fcacfcf6640 1 -- 172.21.15.12:0/3971861257 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcaac03e210 msgr2=0x7fcaac0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:59.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.710+0000 7fcacfcf6640 1 --2- 172.21.15.12:0/3971861257 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcaac03e210 0x7fcaac0406d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcac0001dd0 tx=0x7fcac0009040 comp rx=0 tx=0).stop 2024-09-11T01:33:59.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.710+0000 7fcacfcf6640 10 monclient: shutdown 2024-09-11T01:33:59.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.710+0000 7fcacfcf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:59.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.710+0000 7fcacfcf6640 1 -- 172.21.15.12:0/3971861257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8137e50 msgr2=0x7fcac81cd910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:59.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.710+0000 7fcacfcf6640 1 --2- 172.21.15.12:0/3971861257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcac8137e50 0x7fcac81cd910 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcab8009870 tx=0x7fcab80046c0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.710+0000 7fcace293640 1 -- 172.21.15.12:0/3971861257 reap_dead start 2024-09-11T01:33:59.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.711+0000 7fcace293640 1 -- 172.21.15.12:0/3971861257 reap_dead start 2024-09-11T01:33:59.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.711+0000 7fcacfcf6640 1 -- 172.21.15.12:0/3971861257 shutdown_connections 2024-09-11T01:33:59.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.711+0000 7fcacfcf6640 1 -- 172.21.15.12:0/3971861257 >> 172.21.15.12:0/3971861257 conn(0x7fcac80a7020 msgr2=0x7fcac80a3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:59.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.711+0000 7fcacfcf6640 1 -- 172.21.15.12:0/3971861257 shutdown_connections 2024-09-11T01:33:59.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.711+0000 7fcacfcf6640 1 -- 172.21.15.12:0/3971861257 wait complete. 2024-09-11T01:33:59.736 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:33:59.736 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:33:59.736 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:33:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.836+0000 7feeac0b8640 10 monclient: get_monmap_and_config 2024-09-11T01:33:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.836+0000 7feeac0b8640 10 monclient: build_initial_monmap 2024-09-11T01:33:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.836+0000 7feeac0b8640 10 monclient: monmap: 2024-09-11T01:33:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:33:59.837414+0000 2024-09-11T01:33:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:33:59.837414+0000 2024-09-11T01:33:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:33:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:33:59.843 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:33:59.843 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:33:59.843 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:33:59.843 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.837+0000 7feeac0b8640 1 Processor -- start 2024-09-11T01:33:59.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.837+0000 7feeac0b8640 1 -- start start 2024-09-11T01:33:59.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.837+0000 7feeac0b8640 10 monclient: init 2024-09-11T01:33:59.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.837+0000 7feeac0b8640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:59.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.837+0000 7feeac0b8640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-11T01:33:59.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.837+0000 7feeac0b8640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7feea40a2e20 0x7feea406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.837+0000 7feeac0b8640 10 monclient(hunting): picked mon.noname-c con 0x7feea40a2e20 addr v2:172.21.15.71:3300/0 2024-09-11T01:33:59.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.837+0000 7feeac0b8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feea40a4970 addr v1:172.21.15.12:6789/0 2024-09-11T01:33:59.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.837+0000 7feeac0b8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feea406f910 0x7feea413f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.837+0000 7feeac0b8640 10 monclient(hunting): picked mon.noname-b con 0x7feea406f910 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.837+0000 7feeac0b8640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feea406f4f0 con 0x7feea40a4970 2024-09-11T01:33:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.837+0000 7feeac0b8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.837+0000 7feeac0b8640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7feea406ef30 con 0x7feea40a2e20 2024-09-11T01:33:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.837+0000 7feeac0b8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.837+0000 7feeac0b8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7feea406ec50 con 0x7feea406f910 2024-09-11T01:33:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.837+0000 7feeac0b8640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.837+0000 7feeac0b8640 10 monclient(hunting): authenticate will time out at 1046.279979s 2024-09-11T01:33:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.838+0000 7feea9653640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7feea40a4970 0x7feea40a3200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33042/0 (socket says 172.21.15.12:33042) 2024-09-11T01:33:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.838+0000 7feea9653640 1 -- 172.21.15.12:0/4283342433 learned_addr learned my addr 172.21.15.12:0/4283342433 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:33:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.838+0000 7feeaa655640 1 --2- 172.21.15.12:0/4283342433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feea406f910 0x7feea413f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.838+0000 7feeaa655640 10 monclient(hunting): get_auth_request con 0x7feea406f910 auth_method 0 2024-09-11T01:33:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.838+0000 7feeaa655640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.838+0000 7feeaa655640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.838+0000 7feeaa655640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.838+0000 7feeaa655640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.838+0000 7feeaa655640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.838+0000 7feeaa655640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.838+0000 7feea9e54640 1 --2- 172.21.15.12:0/4283342433 >> v2:172.21.15.71:3300/0 conn(0x7feea40a2e20 0x7feea406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.838+0000 7feea9e54640 10 monclient(hunting): get_auth_request con 0x7feea40a2e20 auth_method 0 2024-09-11T01:33:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.838+0000 7feea9e54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.838+0000 7feea9e54640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.838+0000 7feea9e54640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.838+0000 7feeaa655640 10 monclient(hunting): handle_auth_done global_id 44249 payload 274 2024-09-11T01:33:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.838+0000 7feeaa655640 1 -- 172.21.15.12:0/4283342433 >> v2:172.21.15.71:3300/0 conn(0x7feea40a2e20 msgr2=0x7feea406ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.838+0000 7feeaa655640 1 --2- 172.21.15.12:0/4283342433 >> v2:172.21.15.71:3300/0 conn(0x7feea40a2e20 0x7feea406ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.838+0000 7feeaa655640 1 -- 172.21.15.12:0/4283342433 >> v1:172.21.15.12:6789/0 conn(0x7feea40a4970 legacy=0x7feea40a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:33:59.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.838+0000 7feeaa655640 10 monclient: _finish_hunting 0 2024-09-11T01:33:59.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.838+0000 7feeaa655640 1 monclient: found mon.noname-b 2024-09-11T01:33:59.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.838+0000 7feeaa655640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:59.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.838+0000 7feeaa655640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:33:59.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.838+0000 7feeaa655640 1 -- 172.21.15.12:0/4283342433 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feea40a3910 con 0x7feea406f910 2024-09-11T01:33:59.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feeaa655640 1 -- 172.21.15.12:0/4283342433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feea406f910 msgr2=0x7feea413f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:59.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feeaa655640 1 --2- 172.21.15.12:0/4283342433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feea406f910 0x7feea413f430 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fee9c009d90 tx=0x7fee9c02a490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaa1915700174af5 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:59.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feea8e52640 1 -- 172.21.15.12:0/4283342433 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9c007950 con 0x7feea406f910 2024-09-11T01:33:59.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feea8e52640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:59.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feea8e52640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:33:59.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feea8e52640 10 monclient: dump: 2024-09-11T01:33:59.852 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:59.852 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:59.852 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.852 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.853 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:59.853 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.853 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:59.853 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:59.853 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:59.853 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feea8e52640 10 monclient: _finish_auth 0 2024-09-11T01:33:59.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feea8e52640 10 monclient: _check_auth_tickets 2024-09-11T01:33:59.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feea8e52640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:59.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feea8e52640 1 -- 172.21.15.12:0/4283342433 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee9c007af0 con 0x7feea406f910 2024-09-11T01:33:59.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feeac0b8640 5 monclient: authenticate success, global_id 44249 2024-09-11T01:33:59.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feeac0b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:33:59.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feea8e52640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feeac0b8640 10 monclient: get_monmap_and_config success 2024-09-11T01:33:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feea8e52640 1 -- 172.21.15.12:0/4283342433 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9c031400 con 0x7feea406f910 2024-09-11T01:33:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feeac0b8640 4 set_mon_vals no callback set 2024-09-11T01:33:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feea8e52640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feea8e52640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feea8e52640 10 monclient: dump: 2024-09-11T01:33:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:59.855 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:59.855 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feeac0b8640 10 monclient: shutdown 2024-09-11T01:33:59.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feeac0b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:59.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feeac0b8640 1 -- 172.21.15.12:0/4283342433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feea406f910 msgr2=0x7feea413f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:59.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feeac0b8640 1 --2- 172.21.15.12:0/4283342433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feea406f910 0x7feea413f430 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fee9c009d90 tx=0x7fee9c02a490 comp rx=0 tx=0).stop 2024-09-11T01:33:59.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feeac0b8640 1 -- 172.21.15.12:0/4283342433 shutdown_connections 2024-09-11T01:33:59.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feeac0b8640 1 --2- 172.21.15.12:0/4283342433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feea406f910 0x7feea413f430 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feeac0b8640 1 --2- 172.21.15.12:0/4283342433 >> v2:172.21.15.71:3300/0 conn(0x7feea40a2e20 0x7feea406ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.841+0000 7feeac0b8640 1 -- 172.21.15.12:0/4283342433 >> 172.21.15.12:0/4283342433 conn(0x7feea40a8600 msgr2=0x7feea40a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:59.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.842+0000 7feeac0b8640 1 -- 172.21.15.12:0/4283342433 shutdown_connections 2024-09-11T01:33:59.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.842+0000 7feeac0b8640 1 -- 172.21.15.12:0/4283342433 wait complete. 2024-09-11T01:33:59.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.842+0000 7feeac0b8640 10 monclient: build_initial_monmap 2024-09-11T01:33:59.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.842+0000 7feeac0b8640 10 monclient: monmap: 2024-09-11T01:33:59.856 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:33:59.856 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:33:59.856 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:33:59.856 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:33:59.856 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:33:59.856 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.856 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:33:59.856 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:33:59.856 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:33:59.856 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.842+0000 7feeac0b8640 1 Processor -- start 2024-09-11T01:33:59.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feeac0b8640 1 -- start start 2024-09-11T01:33:59.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feeac0b8640 10 monclient: init 2024-09-11T01:33:59.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feeac0b8640 10 monclient: _reopen_session rank -1 2024-09-11T01:33:59.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feeac0b8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:33:59.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feeac0b8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feea406f910 0x7feea413e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feeac0b8640 10 monclient(hunting): picked mon.noname-a con 0x7feea406f910 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:33:59.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feeac0b8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feea40a2e20 0x7feea413ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feeac0b8640 10 monclient(hunting): picked mon.noname-c con 0x7feea40a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:33:59.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feeac0b8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feea40a4970 0x7feea41cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feeac0b8640 10 monclient(hunting): picked mon.noname-b con 0x7feea40a4970 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:33:59.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feeac0b8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:59.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feeac0b8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7feea413fd00 con 0x7feea406f910 2024-09-11T01:33:59.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feeac0b8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:59.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feeac0b8640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7feea413fb80 con 0x7feea40a4970 2024-09-11T01:33:59.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feeac0b8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:59.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feeac0b8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7feea413fe80 con 0x7feea40a2e20 2024-09-11T01:33:59.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feeac0b8640 10 monclient(hunting): _renew_subs 2024-09-11T01:33:59.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feeac0b8640 10 monclient(hunting): authenticate will time out at 1046.285493s 2024-09-11T01:33:59.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9e54640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feea406f910 0x7feea413e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9653640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feea40a2e20 0x7feea413ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9e54640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feea406f910 0x7feea413e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42334/0 (socket says 172.21.15.12:42334) 2024-09-11T01:33:59.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9e54640 1 -- 172.21.15.12:0/3966324951 learned_addr learned my addr 172.21.15.12:0/3966324951 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:33:59.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9e54640 10 monclient(hunting): get_auth_request con 0x7feea406f910 auth_method 0 2024-09-11T01:33:59.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9e54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:59.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9e54640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:59.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9e54640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:59.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9653640 10 monclient(hunting): get_auth_request con 0x7feea40a2e20 auth_method 0 2024-09-11T01:33:59.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9653640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:59.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9653640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:59.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9653640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:59.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9653640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:59.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9653640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:59.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9653640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:59.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9e54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:59.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9e54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:59.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9e54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:59.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feeaa655640 1 --2- 172.21.15.12:0/3966324951 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feea40a4970 0x7feea41cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9653640 10 monclient(hunting): handle_auth_done global_id 44252 payload 824 2024-09-11T01:33:59.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9653640 1 -- 172.21.15.12:0/3966324951 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feea40a4970 msgr2=0x7feea41cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:59.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9653640 1 --2- 172.21.15.12:0/3966324951 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feea40a4970 0x7feea41cd2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9653640 1 -- 172.21.15.12:0/3966324951 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feea406f910 msgr2=0x7feea413e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:59.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9653640 1 --2- 172.21.15.12:0/3966324951 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feea406f910 0x7feea413e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9653640 10 monclient: _finish_hunting 0 2024-09-11T01:33:59.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9653640 1 monclient: found mon.noname-c 2024-09-11T01:33:59.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9653640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:33:59.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9653640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:33:59.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.843+0000 7feea9653640 1 -- 172.21.15.12:0/3966324951 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feea409f0b0 con 0x7feea40a2e20 2024-09-11T01:33:59.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.846+0000 7feea9653640 1 -- 172.21.15.12:0/3966324951 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feea40a2e20 msgr2=0x7feea413ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:59.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.846+0000 7feea9653640 1 --2- 172.21.15.12:0/3966324951 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feea40a2e20 0x7feea413ed50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fee9400c3e0 tx=0x7fee9400c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:59.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.846+0000 7fee9a7fc640 1 -- 172.21.15.12:0/3966324951 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee94017070 con 0x7feea40a2e20 2024-09-11T01:33:59.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.846+0000 7fee9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:59.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.846+0000 7fee9a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:33:59.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.846+0000 7fee9a7fc640 10 monclient: dump: 2024-09-11T01:33:59.860 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:59.860 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:59.860 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.860 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.861 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:59.861 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.861 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:59.861 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:59.861 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:59.861 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.846+0000 7fee9a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:33:59.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.846+0000 7fee9a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:33:59.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.846+0000 7fee9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:33:59.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.846+0000 7fee9a7fc640 1 -- 172.21.15.12:0/3966324951 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee940070d0 con 0x7feea40a2e20 2024-09-11T01:33:59.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.846+0000 7fee9a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:33:59.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.846+0000 7fee9a7fc640 1 -- 172.21.15.12:0/3966324951 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee940074c0 con 0x7feea40a2e20 2024-09-11T01:33:59.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.846+0000 7feeac0b8640 5 monclient: authenticate success, global_id 44252 2024-09-11T01:33:59.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.846+0000 7feeac0b8640 10 monclient: _renew_subs 2024-09-11T01:33:59.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.846+0000 7feeac0b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:59.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.846+0000 7fee9bfff640 4 set_mon_vals no callback set 2024-09-11T01:33:59.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.846+0000 7feeac0b8640 1 -- 172.21.15.12:0/3966324951 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feea413ba30 con 0x7feea40a2e20 2024-09-11T01:33:59.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.846+0000 7fee9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:59.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.846+0000 7fee9a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:33:59.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.846+0000 7fee9a7fc640 10 monclient: dump: 2024-09-11T01:33:59.862 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:59.862 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:59.862 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.862 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.862 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:59.862 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.862 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:59.863 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:59.863 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:59.863 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.846+0000 7feeac0b8640 10 monclient: _renew_subs 2024-09-11T01:33:59.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.846+0000 7feeac0b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:33:59.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.846+0000 7feeac0b8640 1 -- 172.21.15.12:0/3966324951 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7feea409e870 con 0x7feea40a2e20 2024-09-11T01:33:59.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.847+0000 7feeac0b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:59.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.847+0000 7feeac0b8640 1 --2- 172.21.15.12:0/3966324951 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feea413bc00 0x7feea402d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.847+0000 7feeac0b8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:59.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.847+0000 7feeac0b8640 1 -- 172.21.15.12:0/3966324951 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7feea406a060 con 0x7feea413bc00 2024-09-11T01:33:59.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.847+0000 7fee9a7fc640 1 -- 172.21.15.12:0/3966324951 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fee9401b3b0 con 0x7feea40a2e20 2024-09-11T01:33:59.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.847+0000 7feea9e54640 1 --2- 172.21.15.12:0/3966324951 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feea413bc00 0x7feea402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.847+0000 7feea9e54640 10 monclient: get_auth_request con 0x7feea413bc00 auth_method 0 2024-09-11T01:33:59.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.847+0000 7feea9e54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:59.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.847+0000 7feea9e54640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:59.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.847+0000 7feea9e54640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:59.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.847+0000 7fee9a7fc640 1 --2- 172.21.15.12:0/3966324951 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fee8803e260 0x7fee88040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.847+0000 7feea9e54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:59.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.847+0000 7feea9e54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:59.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.847+0000 7feea9e54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:59.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.847+0000 7fee9a7fc640 1 -- 172.21.15.12:0/3966324951 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(172..172 src has 1..172) ==== 4787+0+0 (secure 0 0 0) 0x7fee940524d0 con 0x7feea40a2e20 2024-09-11T01:33:59.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.847+0000 7feeaa655640 1 --2- 172.21.15.12:0/3966324951 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fee8803e260 0x7fee88040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.847+0000 7feeaa655640 10 monclient: get_auth_request con 0x7fee8803e260 auth_method 0 2024-09-11T01:33:59.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.847+0000 7feea9e54640 10 monclient(hunting): handle_auth_done global_id 34321 payload 824 2024-09-11T01:33:59.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.847+0000 7feea9e54640 1 -- 172.21.15.12:0/3966324951 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feea406e4d0 con 0x7feea413bc00 2024-09-11T01:33:59.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.850+0000 7feea9e54640 1 -- 172.21.15.12:0/3966324951 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feea413bc00 msgr2=0x7feea402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:59.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.850+0000 7feea9e54640 1 --2- 172.21.15.12:0/3966324951 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feea413bc00 0x7feea402d440 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7feea413b330 tx=0x7feea0010dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:59.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.850+0000 7fee9a7fc640 1 -- 172.21.15.12:0/3966324951 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea000ec80 con 0x7feea413bc00 2024-09-11T01:33:59.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.850+0000 7fee9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:59.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.850+0000 7fee9a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:33:59.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.850+0000 7fee9a7fc640 10 monclient: dump: 2024-09-11T01:33:59.865 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:59.865 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:59.865 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.865 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.865 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:59.865 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.865 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:59.865 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:59.865 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:59.865 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.850+0000 7feeaa655640 1 --2- 172.21.15.12:0/3966324951 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fee8803e260 0x7fee88040720 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7feea4074f40 tx=0x7fee9c02f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:59.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.850+0000 7fee9a7fc640 1 -- 172.21.15.12:0/3966324951 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7feea000ec80 con 0x7feea413bc00 2024-09-11T01:33:59.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.850+0000 7fee9a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:33:59.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.850+0000 7fee9a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:33:59.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.850+0000 7fee9a7fc640 1 -- 172.21.15.12:0/3966324951 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feea413bc00 msgr2=0x7feea402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:59.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.850+0000 7fee9a7fc640 1 --2- 172.21.15.12:0/3966324951 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feea413bc00 0x7feea402d440 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7feea413b330 tx=0x7feea0010dc0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.850+0000 7fee9a7fc640 1 -- 172.21.15.12:0/3966324951 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feea413bc00 msgr2=0x7feea402d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:33:59.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.850+0000 7fee9a7fc640 1 --2- 172.21.15.12:0/3966324951 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feea413bc00 0x7feea402d440 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7feea413b330 tx=0x7feea0010dc0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.856+0000 7feeac0b8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:33:59.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.856+0000 7feeac0b8640 1 --2- 172.21.15.12:0/3966324951 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feea41398c0 0x7feea402d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:33:59.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.856+0000 7feeac0b8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:33:59.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.856+0000 7feeac0b8640 1 -- 172.21.15.12:0/3966324951 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7feea406ded0 con 0x7feea41398c0 2024-09-11T01:33:59.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.857+0000 7feea9e54640 1 --2- 172.21.15.12:0/3966324951 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feea41398c0 0x7feea402d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:33:59.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.857+0000 7feea9e54640 10 monclient: get_auth_request con 0x7feea41398c0 auth_method 0 2024-09-11T01:33:59.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.857+0000 7feea9e54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:33:59.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.857+0000 7feea9e54640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:33:59.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.857+0000 7feea9e54640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:33:59.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.857+0000 7feea9e54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:33:59.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.857+0000 7feea9e54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:33:59.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.857+0000 7feea9e54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:33:59.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.857+0000 7feea9e54640 10 monclient(hunting): handle_auth_done global_id 34324 payload 824 2024-09-11T01:33:59.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.857+0000 7feea9e54640 1 -- 172.21.15.12:0/3966324951 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7feea40a0f30 con 0x7feea41398c0 2024-09-11T01:33:59.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.859+0000 7feea9e54640 1 -- 172.21.15.12:0/3966324951 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feea41398c0 msgr2=0x7feea402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:33:59.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.859+0000 7feea9e54640 1 --2- 172.21.15.12:0/3966324951 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feea41398c0 0x7feea402d980 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7feea0011010 tx=0x7feea00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:33:59.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.860+0000 7fee9a7fc640 1 -- 172.21.15.12:0/3966324951 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea0003e00 con 0x7feea41398c0 2024-09-11T01:33:59.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.860+0000 7fee9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:33:59.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.860+0000 7fee9a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:33:59.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.860+0000 7fee9a7fc640 10 monclient: dump: 2024-09-11T01:33:59.869 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:33:59.869 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:33:59.870 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.870 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:33:59.870 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:33:59.870 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:33:59.870 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:33:59.870 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:33:59.870 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:33:59.870 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:33:59.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.860+0000 7fee9a7fc640 1 -- 172.21.15.12:0/3966324951 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7feea0003e00 con 0x7feea41398c0 2024-09-11T01:33:59.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.860+0000 7fee9a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:33:59.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.860+0000 7fee9a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:33:59.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.860+0000 7fee9a7fc640 1 -- 172.21.15.12:0/3966324951 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feea41398c0 msgr2=0x7feea402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:59.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.860+0000 7fee9a7fc640 1 --2- 172.21.15.12:0/3966324951 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feea41398c0 0x7feea402d980 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7feea0011010 tx=0x7feea00046c0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.860+0000 7fee9a7fc640 1 -- 172.21.15.12:0/3966324951 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feea41398c0 msgr2=0x7feea402d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:33:59.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.860+0000 7fee9a7fc640 1 --2- 172.21.15.12:0/3966324951 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feea41398c0 0x7feea402d980 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7feea0011010 tx=0x7feea00046c0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.871 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:33:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:33:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:33:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:33:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 56, 2024-09-11T01:33:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 133073, 2024-09-11T01:33:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:33:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:33:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:33:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:33:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 4, 2024-09-11T01:33:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:33:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:33:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:33:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:33:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:33:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:33:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:33:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:33:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:33:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:33:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:33:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:33:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:33:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:33:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:33:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:33:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:33:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:33:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:33:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:33:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:33:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:33:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:33:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:33:59.873 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:33:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:33:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:33:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:33:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:33:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:33:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:33:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:33:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:33:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:33:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:33:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:33:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:33:59.874 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:33:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:33:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:33:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:33:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:33:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:33:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:33:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:33:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:33:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:33:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:33:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:33:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:33:59.875 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:33:59.876 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:33:59.876 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:33:59.876 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:33:59.876 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:33:59.876 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:59.876 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:33:59.876 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.876 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:33:59.876 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.876 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:33:59.876 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:33:59.876 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:33:59.876 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:33:59.876 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.877 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:33:59.877 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:33:59.877 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:59.877 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.877 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.877 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:33:59.877 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:33:59.877 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:59.877 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:59.877 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:59.877 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.877 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:33:59.877 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:33:59.877 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:33:59.878 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:33:59.878 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:33:59.878 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.878 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.878 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:33:59.878 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:33:59.878 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:33:59.878 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:33:59.878 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.878 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:33:59.878 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:33:59.878 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:59.878 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.878 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.879 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:33:59.879 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:33:59.879 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:59.879 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:59.879 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:59.879 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.879 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:33:59.879 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:33:59.879 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:33:59.879 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:33:59.879 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:33:59.879 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.879 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.879 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:33:59.880 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:33:59.880 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:33:59.880 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:33:59.880 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.880 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:33:59.880 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:33:59.880 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:59.880 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.880 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.880 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:33:59.880 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:33:59.880 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:33:59.880 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:59.880 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:59.880 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.881 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:33:59.881 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:33:59.881 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:33:59.881 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:33:59.881 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:33:59.881 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:59.881 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:59.881 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.881 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:33:59.881 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:33:59.881 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.881 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:33:59.881 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:33:59.881 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:33:59.881 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:59.882 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:59.882 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:33:59.882 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.882 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:33:59.882 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:33:59.882 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:33:59.882 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:59.882 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:59.882 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:33:59.882 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.882 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:33:59.882 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:33:59.882 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:33:59.882 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:59.882 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:33:59.883 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:33:59.883 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:33:59.883 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:33:59.883 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:33:59.883 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:33:59.883 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:33:59.883 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:33:59.883 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:33:59.883 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:33:59.883 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:33:59.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.860+0000 7feeac0b8640 1 -- 172.21.15.12:0/3966324951 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fee8803e260 msgr2=0x7fee88040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:59.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.860+0000 7feeac0b8640 1 --2- 172.21.15.12:0/3966324951 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fee8803e260 0x7fee88040720 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7feea4074f40 tx=0x7fee9c02f000 comp rx=0 tx=0).stop 2024-09-11T01:33:59.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.861+0000 7feeac0b8640 10 monclient: shutdown 2024-09-11T01:33:59.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.861+0000 7feeac0b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:33:59.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.861+0000 7feeac0b8640 1 -- 172.21.15.12:0/3966324951 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feea40a2e20 msgr2=0x7feea413ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:33:59.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.861+0000 7feeac0b8640 1 --2- 172.21.15.12:0/3966324951 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feea40a2e20 0x7feea413ed50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fee9400c3e0 tx=0x7fee9400c8a0 comp rx=0 tx=0).stop 2024-09-11T01:33:59.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.861+0000 7feeaa655640 1 -- 172.21.15.12:0/3966324951 reap_dead start 2024-09-11T01:33:59.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.861+0000 7feeaa655640 1 -- 172.21.15.12:0/3966324951 reap_dead start 2024-09-11T01:33:59.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.861+0000 7feeac0b8640 1 -- 172.21.15.12:0/3966324951 shutdown_connections 2024-09-11T01:33:59.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.861+0000 7feeac0b8640 1 -- 172.21.15.12:0/3966324951 >> 172.21.15.12:0/3966324951 conn(0x7feea40a8600 msgr2=0x7feea40736b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:33:59.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.861+0000 7feeac0b8640 1 -- 172.21.15.12:0/3966324951 shutdown_connections 2024-09-11T01:33:59.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:33:59.861+0000 7feeac0b8640 1 -- 172.21.15.12:0/3966324951 wait complete. 2024-09-11T01:33:59.886 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:33:59.886 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-11T01:33:59.886 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-11T01:33:59.886 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-11T01:33:59.886 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-11T01:33:59.886 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-11T01:33:59.886 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:33:59.897 INFO:tasks.ceph.mon.c:Stopped 2024-09-11T01:33:59.897 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-11T01:33:59.898 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-11T01:33:59.898 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T01:34:14.899 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-11T01:34:14.900 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-11T01:34:14.900 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:34:14.900 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:34:15.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.011+0000 7f188f0b1640 10 monclient: get_monmap_and_config 2024-09-11T01:34:15.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.011+0000 7f188f0b1640 10 monclient: build_initial_monmap 2024-09-11T01:34:15.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.011+0000 7f188f0b1640 10 monclient: monmap: 2024-09-11T01:34:15.013 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:34:15.013 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:34:15.013 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:34:15.011852+0000 2024-09-11T01:34:15.013 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:34:15.011852+0000 2024-09-11T01:34:15.013 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:34:15.013 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:34:15.013 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:34:15.014 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:34:15.014 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:34:15.014 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:34:15.014 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:34:15.014 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:34:15.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.011+0000 7f188f0b1640 1 Processor -- start 2024-09-11T01:34:15.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.011+0000 7f188f0b1640 1 -- start start 2024-09-11T01:34:15.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.011+0000 7f188f0b1640 10 monclient: init 2024-09-11T01:34:15.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.012+0000 7f188f0b1640 10 monclient: _reopen_session rank -1 2024-09-11T01:34:15.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.012+0000 7f188f0b1640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-11T01:34:15.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.012+0000 7f188f0b1640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1888138230 0x7f1888138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:15.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.012+0000 7f188f0b1640 10 monclient(hunting): picked mon.noname-c con 0x7f1888138230 addr v2:172.21.15.71:3300/0 2024-09-11T01:34:15.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.012+0000 7f188f0b1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f188809d030 addr v1:172.21.15.12:6789/0 2024-09-11T01:34:15.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.012+0000 7f188f0b1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f188809d460 addr v1:172.21.15.71:6789/0 2024-09-11T01:34:15.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.012+0000 7f188f0b1640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18880a5e40 con 0x7f188809d030 2024-09-11T01:34:15.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.012+0000 7f188f0b1640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18880a6e30 con 0x7f188809d460 2024-09-11T01:34:15.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.012+0000 7f188f0b1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:15.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.012+0000 7f188f0b1640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f188809c9c0 con 0x7f1888138230 2024-09-11T01:34:15.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.012+0000 7f188f0b1640 10 monclient(hunting): _renew_subs 2024-09-11T01:34:15.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.012+0000 7f188f0b1640 10 monclient(hunting): authenticate will time out at 1061.454521s 2024-09-11T01:34:15.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.012+0000 7f1887fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f188809d030 0x7f18880a5020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46672/0 (socket says 172.21.15.12:46672) 2024-09-11T01:34:15.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.012+0000 7f1887fff640 1 -- 172.21.15.12:0/3526109494 learned_addr learned my addr 172.21.15.12:0/3526109494 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:34:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.012+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.71:3300/0 conn(0x7f1888138230 0x7f1888138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.012+0000 7f188ce4d640 10 monclient(hunting): get_auth_request con 0x7f1888138230 auth_method 0 2024-09-11T01:34:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.012+0000 7f188ce4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.012+0000 7f188ce4d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.012+0000 7f188ce4d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:34:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.013+0000 7f188ce4d640 1 -- 172.21.15.12:0/3526109494 >> v2:172.21.15.71:3300/0 conn(0x7f1888138230 msgr2=0x7f1888138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:34:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.013+0000 7f188ce4d640 1 -- 172.21.15.12:0/3526109494 >> v2:172.21.15.71:3300/0 conn(0x7f1888138230 msgr2=0x7f1888138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:34:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.013+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.71:3300/0 conn(0x7f1888138230 0x7f1888138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:34:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.013+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.71:3300/0 conn(0x7f1888138230 0x7f1888138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:34:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.014+0000 7f1887fff640 1 -- 172.21.15.12:0/3526109494 >> v1:172.21.15.12:6789/0 conn(0x7f188809d030 legacy=0x7f18880a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:34:15.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.015+0000 7f188d64e640 1 -- 172.21.15.12:0/3526109494 >> v1:172.21.15.71:6789/0 conn(0x7f188809d460 legacy=0x7f18880a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:34:15.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.214+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.71:3300/0 conn(0x7f1888138230 0x7f1888138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:15.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.214+0000 7f188ce4d640 10 monclient(hunting): get_auth_request con 0x7f1888138230 auth_method 0 2024-09-11T01:34:15.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.214+0000 7f188ce4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:15.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.214+0000 7f188ce4d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:15.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.214+0000 7f188ce4d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:34:15.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.214+0000 7f188ce4d640 1 -- 172.21.15.12:0/3526109494 >> v2:172.21.15.71:3300/0 conn(0x7f1888138230 msgr2=0x7f1888138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:34:15.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.215+0000 7f188ce4d640 1 -- 172.21.15.12:0/3526109494 >> v2:172.21.15.71:3300/0 conn(0x7f1888138230 msgr2=0x7f1888138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:34:15.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.215+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.71:3300/0 conn(0x7f1888138230 0x7f1888138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:34:15.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.215+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.71:3300/0 conn(0x7f1888138230 0x7f1888138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:34:15.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.617+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.71:3300/0 conn(0x7f1888138230 0x7f1888138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:15.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.617+0000 7f188ce4d640 10 monclient(hunting): get_auth_request con 0x7f1888138230 auth_method 0 2024-09-11T01:34:15.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.617+0000 7f188ce4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:15.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.617+0000 7f188ce4d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:15.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.617+0000 7f188ce4d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:34:15.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.617+0000 7f188ce4d640 1 -- 172.21.15.12:0/3526109494 >> v2:172.21.15.71:3300/0 conn(0x7f1888138230 msgr2=0x7f1888138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:34:15.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.617+0000 7f188ce4d640 1 -- 172.21.15.12:0/3526109494 >> v2:172.21.15.71:3300/0 conn(0x7f1888138230 msgr2=0x7f1888138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:34:15.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.617+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.71:3300/0 conn(0x7f1888138230 0x7f1888138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:34:15.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:15.617+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.71:3300/0 conn(0x7f1888138230 0x7f1888138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:34:16.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:16.418+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.71:3300/0 conn(0x7f1888138230 0x7f1888138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:16.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:16.418+0000 7f188ce4d640 10 monclient(hunting): get_auth_request con 0x7f1888138230 auth_method 0 2024-09-11T01:34:16.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:16.418+0000 7f188ce4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:16.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:16.418+0000 7f188ce4d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:16.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:16.418+0000 7f188ce4d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:34:16.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:16.418+0000 7f188ce4d640 1 -- 172.21.15.12:0/3526109494 >> v2:172.21.15.71:3300/0 conn(0x7f1888138230 msgr2=0x7f1888138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:34:16.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:16.418+0000 7f188ce4d640 1 -- 172.21.15.12:0/3526109494 >> v2:172.21.15.71:3300/0 conn(0x7f1888138230 msgr2=0x7f1888138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:34:16.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:16.418+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.71:3300/0 conn(0x7f1888138230 0x7f1888138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:34:16.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:16.418+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.71:3300/0 conn(0x7f1888138230 0x7f1888138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:34:18.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.012+0000 7f18867fc640 10 monclient(hunting): tick 2024-09-11T01:34:18.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.012+0000 7f18867fc640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:34:18.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.012+0000 7f18867fc640 1 monclient(hunting): continuing hunt 2024-09-11T01:34:18.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.012+0000 7f18867fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:34:18.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.012+0000 7f18867fc640 1 -- 172.21.15.12:0/3526109494 >> v2:172.21.15.71:3300/0 conn(0x7f1888138230 msgr2=0x7f1888138610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:34:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.012+0000 7f18867fc640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.71:3300/0 conn(0x7f1888138230 0x7f1888138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.012+0000 7f18867fc640 1 -- 172.21.15.12:0/3526109494 >> v1:172.21.15.71:6789/0 conn(0x7f188809d460 legacy=0x7f18880a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:34:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.012+0000 7f18867fc640 1 -- 172.21.15.12:0/3526109494 >> v1:172.21.15.12:6789/0 conn(0x7f188809d030 legacy=0x7f18880a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:34:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.012+0000 7f18867fc640 10 monclient: _add_conns ranks=[3,2] 2024-09-11T01:34:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.012+0000 7f18867fc640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f18740037e0 0x7f1874005c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.012+0000 7f18867fc640 10 monclient(hunting): picked mon.noname-a con 0x7f18740037e0 addr v2:172.21.15.12:3300/0 2024-09-11T01:34:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.012+0000 7f18867fc640 1 --2- 172.21.15.12:0/3526109494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1874006320 0x7f18740087a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.012+0000 7f18867fc640 10 monclient(hunting): picked mon.noname-b con 0x7f1874006320 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:34:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.012+0000 7f18867fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.012+0000 7f18867fc640 1 -- 172.21.15.12:0/3526109494 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f188809c9c0 con 0x7f18740037e0 2024-09-11T01:34:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.012+0000 7f18867fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.012+0000 7f18867fc640 1 -- 172.21.15.12:0/3526109494 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1874008ed0 con 0x7f1874006320 2024-09-11T01:34:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.012+0000 7f18867fc640 10 monclient(hunting): _renew_subs 2024-09-11T01:34:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.013+0000 7f1887fff640 1 -- 172.21.15.12:0/3526109494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1874006320 msgr2=0x7f18740087a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:34:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.013+0000 7f1887fff640 1 --2- 172.21.15.12:0/3526109494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1874006320 0x7f18740087a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:34:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.013+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f18740037e0 0x7f1874005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.013+0000 7f188ce4d640 10 monclient(hunting): get_auth_request con 0x7f18740037e0 auth_method 0 2024-09-11T01:34:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.013+0000 7f188ce4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.013+0000 7f188ce4d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.013+0000 7f188ce4d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:34:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.013+0000 7f188ce4d640 1 -- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f18740037e0 msgr2=0x7f1874005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:34:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.013+0000 7f188ce4d640 1 -- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f18740037e0 msgr2=0x7f1874005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:34:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.013+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f18740037e0 0x7f1874005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:34:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.013+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f18740037e0 0x7f1874005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:34:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.214+0000 7f1887fff640 1 -- 172.21.15.12:0/3526109494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1874006320 msgr2=0x7f18740087a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:34:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.214+0000 7f1887fff640 1 --2- 172.21.15.12:0/3526109494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1874006320 0x7f18740087a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:34:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.214+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f18740037e0 0x7f1874005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.214+0000 7f188ce4d640 10 monclient(hunting): get_auth_request con 0x7f18740037e0 auth_method 0 2024-09-11T01:34:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.214+0000 7f188ce4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.214+0000 7f188ce4d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.214+0000 7f188ce4d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:34:18.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.215+0000 7f188ce4d640 1 -- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f18740037e0 msgr2=0x7f1874005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:34:18.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.215+0000 7f188ce4d640 1 -- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f18740037e0 msgr2=0x7f1874005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:34:18.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.215+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f18740037e0 0x7f1874005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:34:18.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.215+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f18740037e0 0x7f1874005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:34:18.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.615+0000 7f1887fff640 1 -- 172.21.15.12:0/3526109494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1874006320 msgr2=0x7f18740087a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:34:18.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.615+0000 7f1887fff640 1 --2- 172.21.15.12:0/3526109494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1874006320 0x7f18740087a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:34:18.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.616+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f18740037e0 0x7f1874005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:18.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.616+0000 7f188ce4d640 10 monclient(hunting): get_auth_request con 0x7f18740037e0 auth_method 0 2024-09-11T01:34:18.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.616+0000 7f188ce4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:18.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.616+0000 7f188ce4d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:18.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.616+0000 7f188ce4d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:34:18.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.616+0000 7f188ce4d640 1 -- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f18740037e0 msgr2=0x7f1874005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:34:18.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.616+0000 7f188ce4d640 1 -- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f18740037e0 msgr2=0x7f1874005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:34:18.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.616+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f18740037e0 0x7f1874005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:34:18.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:18.616+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f18740037e0 0x7f1874005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:34:19.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:19.416+0000 7f1887fff640 1 -- 172.21.15.12:0/3526109494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1874006320 msgr2=0x7f18740087a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:34:19.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:19.416+0000 7f1887fff640 1 --2- 172.21.15.12:0/3526109494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1874006320 0x7f18740087a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:34:19.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:19.418+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f18740037e0 0x7f1874005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:19.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:19.418+0000 7f188ce4d640 10 monclient(hunting): get_auth_request con 0x7f18740037e0 auth_method 0 2024-09-11T01:34:19.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:19.418+0000 7f188ce4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:19.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:19.418+0000 7f188ce4d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:19.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:19.418+0000 7f188ce4d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:34:19.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:19.418+0000 7f188ce4d640 1 -- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f18740037e0 msgr2=0x7f1874005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:34:19.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:19.418+0000 7f188ce4d640 1 -- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f18740037e0 msgr2=0x7f1874005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:34:19.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:19.418+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f18740037e0 0x7f1874005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:34:19.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:19.418+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f18740037e0 0x7f1874005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:34:20.034 INFO:tasks.rados.rados.0.smithi071.stdout:762: done (0 left) 2024-09-11T01:34:20.034 INFO:tasks.rados.rados.0.smithi071.stdout:763: snap_remove snap 75 2024-09-11T01:34:21.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f18867fc640 10 monclient(hunting): tick 2024-09-11T01:34:21.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f18867fc640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:34:21.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f18867fc640 1 monclient(hunting): continuing hunt 2024-09-11T01:34:21.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f18867fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:34:21.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f18867fc640 1 -- 172.21.15.12:0/3526109494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1874006320 msgr2=0x7f18740087a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:34:21.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f18867fc640 1 --2- 172.21.15.12:0/3526109494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1874006320 0x7f18740087a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f18867fc640 1 -- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f18740037e0 msgr2=0x7f1874005c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:34:21.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f18867fc640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f18740037e0 0x7f1874005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f188d64e640 1 -- 172.21.15.12:0/3526109494 reap_dead start 2024-09-11T01:34:21.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f18867fc640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-11T01:34:21.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f18867fc640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f1874003c30 0x7f1874006790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f18867fc640 10 monclient(hunting): picked mon.noname-a con 0x7f1874003c30 addr v2:172.21.15.12:3300/0 2024-09-11T01:34:21.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f18867fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1874006cd0 addr v1:172.21.15.71:6789/0 2024-09-11T01:34:21.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f18867fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1874008850 addr v1:172.21.15.12:6789/0 2024-09-11T01:34:21.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f18867fc640 1 -- 172.21.15.12:0/3526109494 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f188809ba20 con 0x7f1874008850 2024-09-11T01:34:21.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f18867fc640 1 -- 172.21.15.12:0/3526109494 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f187400fc30 con 0x7f1874006cd0 2024-09-11T01:34:21.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f18867fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:21.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f18867fc640 1 -- 172.21.15.12:0/3526109494 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f188809c9c0 con 0x7f1874003c30 2024-09-11T01:34:21.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f18867fc640 10 monclient(hunting): _renew_subs 2024-09-11T01:34:21.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f1874003c30 0x7f1874006790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:21.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f188ce4d640 10 monclient(hunting): get_auth_request con 0x7f1874003c30 auth_method 0 2024-09-11T01:34:21.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f188ce4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:21.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f188ce4d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:21.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f188ce4d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:34:21.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f188ce4d640 1 -- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f1874003c30 msgr2=0x7f1874006790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:34:21.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f188ce4d640 1 -- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f1874003c30 msgr2=0x7f1874006790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:34:21.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f1874003c30 0x7f1874006790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:34:21.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.013+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f1874003c30 0x7f1874006790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:34:21.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.015+0000 7f188d64e640 1 -- 172.21.15.12:0/3526109494 >> v1:172.21.15.12:6789/0 conn(0x7f1874008850 legacy=0x7f187400d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:34:21.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.015+0000 7f1887fff640 1 -- 172.21.15.12:0/3526109494 >> v1:172.21.15.71:6789/0 conn(0x7f1874006cd0 legacy=0x7f18740070d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:34:21.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 1 -- 172.21.15.12:0/3526109494 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3817873657 0 0) 0x7f188809ba20 con 0x7f1874008850 2024-09-11T01:34:21.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 10 monclient(hunting): _negotiate 2024-09-11T01:34:21.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:21.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:34:21.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 10 monclient(hunting): my global_id is 54165 2024-09-11T01:34:21.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 1 -- 172.21.15.12:0/3526109494 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1870003520 con 0x7f1874008850 2024-09-11T01:34:21.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 1 -- 172.21.15.12:0/3526109494 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1277029598 0 0) 0x7f187400fc30 con 0x7f1874006cd0 2024-09-11T01:34:21.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 10 monclient(hunting): _negotiate 2024-09-11T01:34:21.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:21.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:34:21.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 10 monclient(hunting): my global_id is 34327 2024-09-11T01:34:21.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 1 -- 172.21.15.12:0/3526109494 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f188809ba20 con 0x7f1874006cd0 2024-09-11T01:34:21.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 1 -- 172.21.15.12:0/3526109494 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 813160903 0 0) 0x7f1870003520 con 0x7f1874008850 2024-09-11T01:34:21.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 1 -- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f1874003c30 msgr2=0x7f1874006790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:34:21.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 1 --2- 172.21.15.12:0/3526109494 >> v2:172.21.15.12:3300/0 conn(0x7f1874003c30 0x7f1874006790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 1 -- 172.21.15.12:0/3526109494 >> v1:172.21.15.71:6789/0 conn(0x7f1874006cd0 legacy=0x7f18740070d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:34:21.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 10 monclient: _finish_hunting 0 2024-09-11T01:34:21.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:34:21.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:34:21.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:34:21.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 1 -- 172.21.15.12:0/3526109494 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1874010ed0 con 0x7f1874008850 2024-09-11T01:34:21.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 10 monclient: _finish_auth 0 2024-09-11T01:34:21.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 10 monclient: _check_auth_tickets 2024-09-11T01:34:21.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:34:21.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 1 -- 172.21.15.12:0/3526109494 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f1880001c40 con 0x7f1874008850 2024-09-11T01:34:21.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:34:21.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:34:21.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 10 monclient: dump: 2024-09-11T01:34:21.025 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:34:21.025 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:34:21.025 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.025 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.025 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:34:21.025 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:34:21.026 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:34:21.026 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:34:21.026 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:34:21.026 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:34:21.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:34:21.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 10 monclient: _reopen_session rank -1 2024-09-11T01:34:21.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 1 -- 172.21.15.12:0/3526109494 >> v1:172.21.15.12:6789/0 conn(0x7f1874008850 legacy=0x7f187400d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:34:21.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:34:21.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 1 --2- 172.21.15.12:0/3526109494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1870004bd0 0x7f1870004fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 10 monclient(hunting): picked mon.c con 0x7f1870004bd0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:34:21.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 1 --2- 172.21.15.12:0/3526109494 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18700037a0 0x7f1870003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 10 monclient(hunting): picked mon.a con 0x7f18700037a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:34:21.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 1 --2- 172.21.15.12:0/3526109494 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1870004280 0x7f187000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 10 monclient(hunting): picked mon.b con 0x7f1870004280 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:34:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 1 -- 172.21.15.12:0/3526109494 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f188809c9c0 con 0x7f18700037a0 2024-09-11T01:34:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 1 -- 172.21.15.12:0/3526109494 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f187000c1e0 con 0x7f1870004280 2024-09-11T01:34:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 1 -- 172.21.15.12:0/3526109494 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f187000c3e0 con 0x7f1870004bd0 2024-09-11T01:34:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 10 monclient(hunting): _renew_subs 2024-09-11T01:34:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f188ce4d640 1 -- 172.21.15.12:0/3526109494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1870004bd0 msgr2=0x7f1870004fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:34:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f18877fe640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:34:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f188ce4d640 1 --2- 172.21.15.12:0/3526109494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1870004bd0 0x7f1870004fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:34:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f1887fff640 1 --2- 172.21.15.12:0/3526109494 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18700037a0 0x7f1870003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f1887fff640 10 monclient(hunting): get_auth_request con 0x7f18700037a0 auth_method 0 2024-09-11T01:34:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f1887fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:21.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f1887fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:21.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f1887fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:34:21.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f188d64e640 1 --2- 172.21.15.12:0/3526109494 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1870004280 0x7f187000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:21.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f188d64e640 10 monclient(hunting): get_auth_request con 0x7f1870004280 auth_method 0 2024-09-11T01:34:21.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f188d64e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:21.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f188d64e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:21.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f188d64e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:34:21.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f1887fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:34:21.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f1887fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:34:21.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.016+0000 7f1887fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:34:21.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.017+0000 7f188d64e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:34:21.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.017+0000 7f188d64e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:34:21.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.017+0000 7f188d64e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:34:21.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.017+0000 7f1887fff640 10 monclient(hunting): handle_auth_done global_id 54165 payload 293 2024-09-11T01:34:21.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.017+0000 7f1887fff640 1 -- 172.21.15.12:0/3526109494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1870004bd0 msgr2=0x7f1870004fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:34:21.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.017+0000 7f1887fff640 1 --2- 172.21.15.12:0/3526109494 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1870004bd0 0x7f1870004fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.017+0000 7f1887fff640 1 -- 172.21.15.12:0/3526109494 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1870004280 msgr2=0x7f187000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:34:21.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.017+0000 7f1887fff640 1 --2- 172.21.15.12:0/3526109494 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1870004280 0x7f187000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.017+0000 7f1887fff640 10 monclient: _finish_hunting 0 2024-09-11T01:34:21.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.017+0000 7f1887fff640 1 monclient: found mon.a 2024-09-11T01:34:21.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.017+0000 7f1887fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:34:21.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.017+0000 7f1887fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:34:21.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.017+0000 7f1887fff640 1 -- 172.21.15.12:0/3526109494 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1870003520 con 0x7f18700037a0 2024-09-11T01:34:21.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.017+0000 7f1887fff640 10 monclient: _finish_auth 0 2024-09-11T01:34:21.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.017+0000 7f1887fff640 10 monclient: _check_auth_tickets 2024-09-11T01:34:21.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.017+0000 7f1887fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:34:21.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.017+0000 7f188f0b1640 5 monclient: authenticate success, global_id 54165 2024-09-11T01:34:21.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.017+0000 7f188f0b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:34:21.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.017+0000 7f188d64e640 1 --2- 172.21.15.12:0/3526109494 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1870004280 0x7f187000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:34:21.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.017+0000 7f188d64e640 1 -- 172.21.15.12:0/3526109494 reap_dead start 2024-09-11T01:34:21.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.020+0000 7f1887fff640 1 -- 172.21.15.12:0/3526109494 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18700037a0 msgr2=0x7f1870003c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:34:21.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.020+0000 7f1887fff640 1 --2- 172.21.15.12:0/3526109494 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18700037a0 0x7f1870003c20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f187c001df0 tx=0x7f1870005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adc7b8ddef57c99e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:34:21.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.020+0000 7f18877fe640 1 -- 172.21.15.12:0/3526109494 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f187c00c200 con 0x7f18700037a0 2024-09-11T01:34:21.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.020+0000 7f18877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:34:21.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.020+0000 7f18877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:34:21.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.020+0000 7f18877fe640 10 monclient: dump: 2024-09-11T01:34:21.030 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:34:21.031 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:34:21.031 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.031 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.031 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:34:21.031 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:34:21.031 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:34:21.031 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:34:21.031 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:34:21.031 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:34:21.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.020+0000 7f18877fe640 1 -- 172.21.15.12:0/3526109494 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f187c005990 con 0x7f18700037a0 2024-09-11T01:34:21.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.020+0000 7f188f0b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:34:21.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.020+0000 7f18877fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:34:21.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.020+0000 7f188f0b1640 10 monclient: get_monmap_and_config success 2024-09-11T01:34:21.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.020+0000 7f18877fe640 1 -- 172.21.15.12:0/3526109494 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f187c005e00 con 0x7f18700037a0 2024-09-11T01:34:21.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.020+0000 7f18877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:34:21.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.020+0000 7f188f0b1640 4 set_mon_vals no callback set 2024-09-11T01:34:21.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.020+0000 7f18877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:34:21.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.020+0000 7f18877fe640 10 monclient: dump: 2024-09-11T01:34:21.032 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:34:21.032 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:34:21.032 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.032 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.032 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:34:21.032 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:34:21.032 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:34:21.032 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:34:21.032 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:34:21.033 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:34:21.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.021+0000 7f188f0b1640 10 monclient: shutdown 2024-09-11T01:34:21.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.021+0000 7f188f0b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:34:21.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.021+0000 7f188f0b1640 1 -- 172.21.15.12:0/3526109494 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18700037a0 msgr2=0x7f1870003c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:34:21.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.021+0000 7f188f0b1640 1 --2- 172.21.15.12:0/3526109494 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18700037a0 0x7f1870003c20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f187c001df0 tx=0x7f1870005830 comp rx=0 tx=0).stop 2024-09-11T01:34:21.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.021+0000 7f188f0b1640 1 -- 172.21.15.12:0/3526109494 shutdown_connections 2024-09-11T01:34:21.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.021+0000 7f188f0b1640 1 --2- 172.21.15.12:0/3526109494 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18700037a0 0x7f1870003c20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.021+0000 7f188f0b1640 1 -- 172.21.15.12:0/3526109494 >> 172.21.15.12:0/3526109494 conn(0x7f18880a7120 msgr2=0x7f18880a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:34:21.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.021+0000 7f188f0b1640 1 -- 172.21.15.12:0/3526109494 shutdown_connections 2024-09-11T01:34:21.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.021+0000 7f188f0b1640 1 -- 172.21.15.12:0/3526109494 wait complete. 2024-09-11T01:34:21.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.022+0000 7f188f0b1640 10 monclient: build_initial_monmap 2024-09-11T01:34:21.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.022+0000 7f188f0b1640 10 monclient: monmap: 2024-09-11T01:34:21.036 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:34:21.036 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:34:21.036 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:34:21.036 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:34:21.036 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:34:21.036 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:34:21.036 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:34:21.037 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:34:21.037 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:34:21.037 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:34:21.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.022+0000 7f188f0b1640 1 Processor -- start 2024-09-11T01:34:21.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.022+0000 7f188f0b1640 1 -- start start 2024-09-11T01:34:21.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.022+0000 7f188f0b1640 10 monclient: init 2024-09-11T01:34:21.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188f0b1640 10 monclient: _reopen_session rank -1 2024-09-11T01:34:21.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188f0b1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:34:21.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188f0b1640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18700037a0 0x7f18881cce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188f0b1640 10 monclient(hunting): picked mon.noname-c con 0x7f18700037a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:34:21.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188f0b1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1888136ff0 0x7f1888137470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188f0b1640 10 monclient(hunting): picked mon.noname-b con 0x7f1888136ff0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:34:21.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188f0b1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1888120c70 0x7f18881210f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188f0b1640 10 monclient(hunting): picked mon.noname-a con 0x7f1888120c70 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:34:21.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188f0b1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:21.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188f0b1640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f188806f970 con 0x7f1888120c70 2024-09-11T01:34:21.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188f0b1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:21.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188f0b1640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f188806f7f0 con 0x7f1888136ff0 2024-09-11T01:34:21.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188f0b1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:21.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188f0b1640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1888138610 con 0x7f18700037a0 2024-09-11T01:34:21.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188f0b1640 10 monclient(hunting): _renew_subs 2024-09-11T01:34:21.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188f0b1640 10 monclient(hunting): authenticate will time out at 1067.465580s 2024-09-11T01:34:21.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188ce4d640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18700037a0 msgr2=0x7f18881cce40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:34:21.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188ce4d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18700037a0 0x7f18881cce40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:34:21.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188d64e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1888120c70 0x7f18881210f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:21.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188d64e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1888120c70 0x7f18881210f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51334/0 (socket says 172.21.15.12:51334) 2024-09-11T01:34:21.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188d64e640 1 -- 172.21.15.12:0/4240755685 learned_addr learned my addr 172.21.15.12:0/4240755685 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:34:21.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188d64e640 10 monclient(hunting): get_auth_request con 0x7f1888120c70 auth_method 0 2024-09-11T01:34:21.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188d64e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:21.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188d64e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:21.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188d64e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:34:21.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f1887fff640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1888136ff0 0x7f1888137470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:21.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188d64e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:34:21.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188d64e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:34:21.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f188d64e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:34:21.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f1887fff640 10 monclient(hunting): get_auth_request con 0x7f1888136ff0 auth_method 0 2024-09-11T01:34:21.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f1887fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:21.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f1887fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:21.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f1887fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:34:21.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.023+0000 7f1887fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:34:21.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.024+0000 7f1887fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:34:21.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.024+0000 7f1887fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:34:21.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.024+0000 7f188d64e640 10 monclient(hunting): handle_auth_done global_id 54168 payload 824 2024-09-11T01:34:21.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.024+0000 7f188d64e640 1 -- 172.21.15.12:0/4240755685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18700037a0 msgr2=0x7f18881cce40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:34:21.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.024+0000 7f188d64e640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18700037a0 0x7f18881cce40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.024+0000 7f188d64e640 1 -- 172.21.15.12:0/4240755685 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1888136ff0 msgr2=0x7f1888137470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:34:21.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.024+0000 7f188d64e640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1888136ff0 0x7f1888137470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.024+0000 7f188d64e640 10 monclient: _finish_hunting 0 2024-09-11T01:34:21.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.024+0000 7f188d64e640 1 monclient: found mon.noname-a 2024-09-11T01:34:21.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.024+0000 7f188d64e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:34:21.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.024+0000 7f188d64e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:34:21.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.024+0000 7f188d64e640 1 -- 172.21.15.12:0/4240755685 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18881246f0 con 0x7f1888120c70 2024-09-11T01:34:21.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.024+0000 7f188d64e640 1 -- 172.21.15.12:0/4240755685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1888120c70 msgr2=0x7f18881210f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:34:21.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.024+0000 7f188d64e640 1 -- 172.21.15.12:0/4240755685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1888120c70 msgr2=0x7f18881210f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:34:21.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.024+0000 7f188d64e640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1888120c70 0x7f18881210f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f1880001460 tx=0x7f188000f5a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:34:21.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.024+0000 7f188d64e640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1888120c70 0x7f18881210f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:34:21.041 INFO:tasks.rados.rados.0.smithi071.stdout:763: done (0 left) 2024-09-11T01:34:21.041 INFO:tasks.rados.rados.0.smithi071.stdout:764: write (excl) oid 12 current snap is 76 2024-09-11T01:34:21.041 INFO:tasks.rados.rados.0.smithi071.stdout:764: seq_num 289 ranges {457223=568375,1461649=475758,2420186=437037} 2024-09-11T01:34:21.041 INFO:tasks.rados.rados.0.smithi071.stdout:764: writing smithi07134777-12 from 457223 to 1025598 tid 1 2024-09-11T01:34:21.045 INFO:tasks.rados.rados.0.smithi071.stdout:764: writing smithi07134777-12 from 1461649 to 1937407 tid 2 2024-09-11T01:34:21.050 INFO:tasks.rados.rados.0.smithi071.stdout:764: writing smithi07134777-12 from 2420186 to 2857223 tid 3 2024-09-11T01:34:21.051 INFO:tasks.rados.rados.0.smithi071.stdout:765: snap_create 2024-09-11T01:34:21.051 INFO:tasks.rados.rados.0.smithi071.stdout:764: finishing write tid 1 to smithi07134777-12 2024-09-11T01:34:21.053 INFO:tasks.rados.rados.0.smithi071.stdout:764: finishing write tid 2 to smithi07134777-12 2024-09-11T01:34:21.056 INFO:tasks.rados.rados.0.smithi071.stdout:764: finishing write tid 3 to smithi07134777-12 2024-09-11T01:34:21.059 INFO:tasks.rados.rados.0.smithi071.stdout:764: finishing write tid 4 to smithi07134777-12 2024-09-11T01:34:21.059 INFO:tasks.rados.rados.0.smithi071.stdout:764: finishing write tid 5 to smithi07134777-12 2024-09-11T01:34:21.059 INFO:tasks.rados.rados.0.smithi071.stdout:764: oid 12 updating version 0 to 256 2024-09-11T01:34:21.059 INFO:tasks.rados.rados.0.smithi071.stdout:764: oid 12 updating version 256 to 257 2024-09-11T01:34:21.059 INFO:tasks.rados.rados.0.smithi071.stdout:764: oid 12 updating version 257 to 259 2024-09-11T01:34:21.059 INFO:tasks.rados.rados.0.smithi071.stdout:764: oid 12 version 259 is already newer than 258 2024-09-11T01:34:21.059 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 259 (ObjNum 289 snap 76 seq_num 289) dirty exists 2024-09-11T01:34:21.059 INFO:tasks.rados.rados.0.smithi071.stdout:764: left oid 12 (ObjNum 289 snap 76 seq_num 289) 2024-09-11T01:34:21.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.225+0000 7f188d64e640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1888120c70 0x7f18881210f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:21.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f188d64e640 10 monclient: get_auth_request con 0x7f1888120c70 auth_method 0 2024-09-11T01:34:21.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f188d64e640 0 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1888120c70 0x7f18881210f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-11T01:34:21.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f188d64e640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1888120c70 0x7f18881210f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f18857fa640 10 monclient: ms_handle_reset current mon [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:34:21.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f18857fa640 10 monclient: _reopen_session rank -1 2024-09-11T01:34:21.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f18857fa640 1 -- 172.21.15.12:0/4240755685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1888120c70 msgr2=0x7f18881210f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:34:21.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f18857fa640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1888120c70 0x7f18881210f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f18857fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:34:21.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f18857fa640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1874006700 0x7f1874008c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f18857fa640 10 monclient(hunting): picked mon.noname-a con 0x7f1874006700 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:34:21.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f18857fa640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18700037a0 0x7f1874002d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f18857fa640 10 monclient(hunting): picked mon.noname-c con 0x7f18700037a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:34:21.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f18857fa640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1888136ff0 0x7f18740050d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f18857fa640 10 monclient(hunting): picked mon.noname-b con 0x7f1888136ff0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:34:21.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f18857fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f18857fa640 1 -- 172.21.15.12:0/4240755685 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1874005ac0 con 0x7f1874006700 2024-09-11T01:34:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f18857fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f18857fa640 1 -- 172.21.15.12:0/4240755685 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1874005c40 con 0x7f1888136ff0 2024-09-11T01:34:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f18857fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f18857fa640 1 -- 172.21.15.12:0/4240755685 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1874005e40 con 0x7f18700037a0 2024-09-11T01:34:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f18857fa640 10 monclient(hunting): _renew_subs 2024-09-11T01:34:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f1887fff640 1 -- 172.21.15.12:0/4240755685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18700037a0 msgr2=0x7f1874002d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:34:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f1887fff640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18700037a0 0x7f1874002d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:34:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f188ce4d640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1874006700 0x7f1874008c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f188ce4d640 10 monclient(hunting): get_auth_request con 0x7f1874006700 auth_method 0 2024-09-11T01:34:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f188ce4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f188ce4d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f188ce4d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:34:21.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f188ce4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:34:21.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f188ce4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:34:21.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f188ce4d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:34:21.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f188d64e640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1888136ff0 0x7f18740050d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:21.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f188ce4d640 10 monclient(hunting): handle_auth_done global_id 54168 payload 293 2024-09-11T01:34:21.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f188ce4d640 1 -- 172.21.15.12:0/4240755685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18700037a0 msgr2=0x7f1874002d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:34:21.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f188ce4d640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18700037a0 0x7f1874002d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f188ce4d640 1 -- 172.21.15.12:0/4240755685 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1888136ff0 msgr2=0x7f18740050d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:34:21.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f188ce4d640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1888136ff0 0x7f18740050d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f188ce4d640 10 monclient: _finish_hunting 0 2024-09-11T01:34:21.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f188ce4d640 1 monclient: found mon.noname-a 2024-09-11T01:34:21.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f188ce4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:34:21.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f188ce4d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:34:21.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f188ce4d640 1 -- 172.21.15.12:0/4240755685 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f187400d330 con 0x7f1874006700 2024-09-11T01:34:21.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f188d64e640 10 monclient: get_auth_request con 0x7f1888136ff0 auth_method 0 2024-09-11T01:34:21.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.226+0000 7f188d64e640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1888136ff0 0x7f18740050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:34:21.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.229+0000 7f188ce4d640 1 -- 172.21.15.12:0/4240755685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1874006700 msgr2=0x7f1874008c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:34:21.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.229+0000 7f188ce4d640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1874006700 0x7f1874008c30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1878006fd0 tx=0x7f1880009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:34:21.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.229+0000 7f18857fa640 1 -- 172.21.15.12:0/4240755685 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18780178e0 con 0x7f1874006700 2024-09-11T01:34:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.229+0000 7f18857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:34:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.229+0000 7f18857fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:34:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.229+0000 7f18857fa640 10 monclient: dump: 2024-09-11T01:34:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:34:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:34:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:34:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:34:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:34:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:34:21.235 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:34:21.235 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:34:21.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.229+0000 7f18857fa640 10 monclient: _finish_auth 0 2024-09-11T01:34:21.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.229+0000 7f18857fa640 10 monclient: _check_auth_tickets 2024-09-11T01:34:21.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.229+0000 7f18857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:34:21.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.229+0000 7f188f0b1640 5 monclient: authenticate success, global_id 54168 2024-09-11T01:34:21.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.229+0000 7f188f0b1640 10 monclient: _renew_subs 2024-09-11T01:34:21.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.229+0000 7f188f0b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:34:21.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.229+0000 7f188f0b1640 1 -- 172.21.15.12:0/4240755685 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18881249d0 con 0x7f1874006700 2024-09-11T01:34:21.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.229+0000 7f18857fa640 1 -- 172.21.15.12:0/4240755685 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f187801a0d0 con 0x7f1874006700 2024-09-11T01:34:21.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.229+0000 7f18857fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:34:21.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.229+0000 7f18857fa640 1 -- 172.21.15.12:0/4240755685 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f187801a2a0 con 0x7f1874006700 2024-09-11T01:34:21.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.229+0000 7f18857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:34:21.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.229+0000 7f18857fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:34:21.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.229+0000 7f18857fa640 10 monclient: dump: 2024-09-11T01:34:21.237 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:34:21.237 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:34:21.237 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.237 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.237 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:34:21.237 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:34:21.237 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:34:21.237 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:34:21.237 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:34:21.237 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:34:21.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.229+0000 7f1886ffd640 4 set_mon_vals no callback set 2024-09-11T01:34:21.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.229+0000 7f188f0b1640 10 monclient: _renew_subs 2024-09-11T01:34:21.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.229+0000 7f188f0b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:34:21.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.229+0000 7f188f0b1640 1 -- 172.21.15.12:0/4240755685 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1888122670 con 0x7f1874006700 2024-09-11T01:34:21.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.230+0000 7f18857fa640 1 -- 172.21.15.12:0/4240755685 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f187801aab0 con 0x7f1874006700 2024-09-11T01:34:21.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.230+0000 7f188f0b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:34:21.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.230+0000 7f188f0b1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:34:21.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.230+0000 7f188f0b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:34:21.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.230+0000 7f188f0b1640 1 -- 172.21.15.12:0/4240755685 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18881389a0 con 0x7f1874006700 2024-09-11T01:34:21.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.230+0000 7f18857fa640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1888120c70 0x7f187404a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.230+0000 7f18857fa640 1 -- 172.21.15.12:0/4240755685 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(175..175 src has 1..175) ==== 4799+0+0 (secure 0 0 0) 0x7f1878052c50 con 0x7f1874006700 2024-09-11T01:34:21.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.231+0000 7f1887fff640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1888120c70 0x7f187404a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:21.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.231+0000 7f1887fff640 10 monclient: get_auth_request con 0x7f1888120c70 auth_method 0 2024-09-11T01:34:21.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.233+0000 7f1887fff640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1888120c70 0x7f187404a140 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f187c007fd0 tx=0x7f187c02e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:34:21.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.235+0000 7f18857fa640 1 -- 172.21.15.12:0/4240755685 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1878051640 con 0x7f1874006700 2024-09-11T01:34:21.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.235+0000 7f18857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:34:21.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.235+0000 7f18857fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:34:21.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.496+0000 7f188f0b1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:34:21.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.496+0000 7f188f0b1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:34:21.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.496+0000 7f188f0b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:34:21.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.496+0000 7f188f0b1640 1 -- 172.21.15.12:0/4240755685 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f18880698e0 con 0x7f1874006700 2024-09-11T01:34:21.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.497+0000 7f18857fa640 1 -- 172.21.15.12:0/4240755685 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f1878004030 con 0x7f1874006700 2024-09-11T01:34:21.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.497+0000 7f18857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:34:21.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.497+0000 7f18857fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:34:21.499 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":58,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:34:21.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.499+0000 7f188f0b1640 1 -- 172.21.15.12:0/4240755685 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1888120c70 msgr2=0x7f187404a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:34:21.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.499+0000 7f188f0b1640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1888120c70 0x7f187404a140 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f187c007fd0 tx=0x7f187c02e000 comp rx=0 tx=0).stop 2024-09-11T01:34:21.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.499+0000 7f188f0b1640 10 monclient: shutdown 2024-09-11T01:34:21.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.499+0000 7f188f0b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:34:21.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.499+0000 7f188f0b1640 1 -- 172.21.15.12:0/4240755685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1874006700 msgr2=0x7f1874008c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:34:21.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.499+0000 7f188f0b1640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1874006700 0x7f1874008c30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1878006fd0 tx=0x7f1880009040 comp rx=0 tx=0).stop 2024-09-11T01:34:21.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.499+0000 7f188f0b1640 1 -- 172.21.15.12:0/4240755685 shutdown_connections 2024-09-11T01:34:21.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.499+0000 7f188f0b1640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1888136ff0 0x7f18740050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.499+0000 7f188f0b1640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18700037a0 0x7f1874002d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.499+0000 7f188f0b1640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1888120c70 0x7f187404a140 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.499+0000 7f188f0b1640 1 --2- 172.21.15.12:0/4240755685 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1874006700 0x7f1874008c30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.499+0000 7f188f0b1640 1 -- 172.21.15.12:0/4240755685 >> 172.21.15.12:0/4240755685 conn(0x7f18880a7120 msgr2=0x7f18880a3540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:34:21.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.499+0000 7f188f0b1640 1 -- 172.21.15.12:0/4240755685 shutdown_connections 2024-09-11T01:34:21.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.499+0000 7f188f0b1640 1 -- 172.21.15.12:0/4240755685 wait complete. 2024-09-11T01:34:21.512 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:34:21.512 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-11T01:34:21.512 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:34:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.612+0000 7fbf05f9f640 10 monclient: get_monmap_and_config 2024-09-11T01:34:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.612+0000 7fbf05f9f640 10 monclient: build_initial_monmap 2024-09-11T01:34:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.612+0000 7fbf05f9f640 10 monclient: monmap: 2024-09-11T01:34:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:34:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:34:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:34:21.613360+0000 2024-09-11T01:34:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:34:21.613360+0000 2024-09-11T01:34:21.614 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:34:21.615 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:34:21.615 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:34:21.615 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:34:21.615 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:34:21.615 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:34:21.615 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:34:21.615 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:34:21.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.613+0000 7fbf05f9f640 1 Processor -- start 2024-09-11T01:34:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.613+0000 7fbf05f9f640 1 -- start start 2024-09-11T01:34:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.613+0000 7fbf05f9f640 10 monclient: init 2024-09-11T01:34:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.613+0000 7fbf05f9f640 10 monclient: _reopen_session rank -1 2024-09-11T01:34:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.613+0000 7fbf05f9f640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-11T01:34:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.613+0000 7fbf05f9f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbf00136fb0 0x7fbf00137390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.613+0000 7fbf05f9f640 10 monclient(hunting): picked mon.noname-b con 0x7fbf00136fb0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:34:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.613+0000 7fbf05f9f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbf0009cff0 0x7fbf0009d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.613+0000 7fbf05f9f640 10 monclient(hunting): picked mon.noname-c con 0x7fbf0009cff0 addr v2:172.21.15.71:3300/0 2024-09-11T01:34:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.613+0000 7fbf05f9f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbf00137ce0 addr v1:172.21.15.12:6789/0 2024-09-11T01:34:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbf05f9f640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf0009c330 con 0x7fbf00137ce0 2024-09-11T01:34:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbf05f9f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbf05f9f640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fbf000a65a0 con 0x7fbf0009cff0 2024-09-11T01:34:21.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbf05f9f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:21.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbf05f9f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fbf0009c610 con 0x7fbf00136fb0 2024-09-11T01:34:21.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbf05f9f640 10 monclient(hunting): _renew_subs 2024-09-11T01:34:21.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbf05f9f640 10 monclient(hunting): authenticate will time out at 1068.056033s 2024-09-11T01:34:21.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbf04f9d640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbf00136fb0 msgr2=0x7fbf00137390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:34:21.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbf04f9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbf00136fb0 0x7fbf00137390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:34:21.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbf0579e640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbf00137ce0 0x7fbf00138170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58852/0 (socket says 172.21.15.12:58852) 2024-09-11T01:34:21.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbf0579e640 1 -- 172.21.15.12:0/2583145800 learned_addr learned my addr 172.21.15.12:0/2583145800 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:34:21.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbefffff640 1 --2- 172.21.15.12:0/2583145800 >> v2:172.21.15.71:3300/0 conn(0x7fbf0009cff0 0x7fbf0009d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:21.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbefffff640 10 monclient(hunting): get_auth_request con 0x7fbf0009cff0 auth_method 0 2024-09-11T01:34:21.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbefffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:21.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbefffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:21.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbefffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:34:21.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbefffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:34:21.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbefffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:34:21.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbefffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:34:21.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbefffff640 10 monclient(hunting): handle_auth_done global_id 34333 payload 274 2024-09-11T01:34:21.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbefffff640 1 -- 172.21.15.12:0/2583145800 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbf00136fb0 msgr2=0x7fbf00137390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:34:21.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbefffff640 1 --2- 172.21.15.12:0/2583145800 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbf00136fb0 0x7fbf00137390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbefffff640 1 -- 172.21.15.12:0/2583145800 >> v1:172.21.15.12:6789/0 conn(0x7fbf00137ce0 legacy=0x7fbf00138170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:34:21.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbefffff640 10 monclient: _finish_hunting 0 2024-09-11T01:34:21.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbefffff640 1 monclient: found mon.noname-c 2024-09-11T01:34:21.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbefffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:34:21.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbefffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:34:21.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.614+0000 7fbefffff640 1 -- 172.21.15.12:0/2583145800 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf0009c8f0 con 0x7fbf0009cff0 2024-09-11T01:34:21.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.617+0000 7fbefffff640 1 -- 172.21.15.12:0/2583145800 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf0009cff0 msgr2=0x7fbf0009d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:34:21.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.617+0000 7fbefffff640 1 --2- 172.21.15.12:0/2583145800 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf0009cff0 0x7fbf0009d430 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbeec0097a0 tx=0x7fbeec029fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92a45b157e20b6ad server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:34:21.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbeff7fe640 1 -- 172.21.15.12:0/2583145800 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeec037070 con 0x7fbf0009cff0 2024-09-11T01:34:21.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbeff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:34:21.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbeff7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:34:21.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbeff7fe640 10 monclient: dump: 2024-09-11T01:34:21.627 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:34:21.627 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:34:21.627 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.628 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.628 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:34:21.628 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:34:21.628 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:34:21.628 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:34:21.628 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:34:21.628 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:34:21.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbeff7fe640 10 monclient: _finish_auth 0 2024-09-11T01:34:21.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbeff7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:34:21.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbeff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:34:21.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbeff7fe640 1 -- 172.21.15.12:0/2583145800 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbeec032640 con 0x7fbf0009cff0 2024-09-11T01:34:21.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbf05f9f640 5 monclient: authenticate success, global_id 34333 2024-09-11T01:34:21.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbf05f9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:34:21.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbeff7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:34:21.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbf05f9f640 10 monclient: get_monmap_and_config success 2024-09-11T01:34:21.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbeff7fe640 1 -- 172.21.15.12:0/2583145800 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeec032a10 con 0x7fbf0009cff0 2024-09-11T01:34:21.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbf05f9f640 4 set_mon_vals no callback set 2024-09-11T01:34:21.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbeff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:34:21.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbeff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:34:21.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbeff7fe640 10 monclient: dump: 2024-09-11T01:34:21.629 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:34:21.629 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:34:21.629 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.629 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.629 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:34:21.629 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:34:21.630 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:34:21.630 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:34:21.630 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:34:21.630 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:34:21.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbf05f9f640 10 monclient: shutdown 2024-09-11T01:34:21.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbf05f9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:34:21.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbf05f9f640 1 -- 172.21.15.12:0/2583145800 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf0009cff0 msgr2=0x7fbf0009d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:34:21.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbf05f9f640 1 --2- 172.21.15.12:0/2583145800 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf0009cff0 0x7fbf0009d430 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbeec0097a0 tx=0x7fbeec029fe0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbf05f9f640 1 -- 172.21.15.12:0/2583145800 shutdown_connections 2024-09-11T01:34:21.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbf05f9f640 1 --2- 172.21.15.12:0/2583145800 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf0009cff0 0x7fbf0009d430 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbf05f9f640 1 --2- 172.21.15.12:0/2583145800 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbf00136fb0 0x7fbf00137390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbf05f9f640 1 -- 172.21.15.12:0/2583145800 >> 172.21.15.12:0/2583145800 conn(0x7fbf000ab190 msgr2=0x7fbf000a6fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:34:21.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbf05f9f640 1 -- 172.21.15.12:0/2583145800 shutdown_connections 2024-09-11T01:34:21.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.618+0000 7fbf05f9f640 1 -- 172.21.15.12:0/2583145800 wait complete. 2024-09-11T01:34:21.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.619+0000 7fbf05f9f640 10 monclient: build_initial_monmap 2024-09-11T01:34:21.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.619+0000 7fbf05f9f640 10 monclient: monmap: 2024-09-11T01:34:21.631 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:34:21.631 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:34:21.631 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:34:21.631 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:34:21.631 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:34:21.631 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:34:21.631 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:34:21.631 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:34:21.631 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:34:21.631 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:34:21.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.619+0000 7fbf05f9f640 1 Processor -- start 2024-09-11T01:34:21.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.619+0000 7fbf05f9f640 1 -- start start 2024-09-11T01:34:21.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.619+0000 7fbf05f9f640 10 monclient: init 2024-09-11T01:34:21.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbf05f9f640 10 monclient: _reopen_session rank -1 2024-09-11T01:34:21.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbf05f9f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:34:21.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbf05f9f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf0009cff0 0x7fbf00135af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbf05f9f640 10 monclient(hunting): picked mon.noname-b con 0x7fbf0009cff0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:34:21.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbf05f9f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf00136fb0 0x7fbf00136030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbf05f9f640 10 monclient(hunting): picked mon.noname-a con 0x7fbf00136fb0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:34:21.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbf05f9f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbf00137ce0 0x7fbf0013bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbf05f9f640 10 monclient(hunting): picked mon.noname-c con 0x7fbf00137ce0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:34:21.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbf05f9f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:21.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbf05f9f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbf00138880 con 0x7fbf00136fb0 2024-09-11T01:34:21.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbf05f9f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:21.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbf05f9f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbf00138a00 con 0x7fbf0009cff0 2024-09-11T01:34:21.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbf05f9f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbf05f9f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fbf0013e140 con 0x7fbf00137ce0 2024-09-11T01:34:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbf05f9f640 10 monclient(hunting): _renew_subs 2024-09-11T01:34:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbf05f9f640 10 monclient(hunting): authenticate will time out at 1068.062459s 2024-09-11T01:34:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbf0579e640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbf00137ce0 msgr2=0x7fbf0013bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:34:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbf0579e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbf00137ce0 0x7fbf0013bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:34:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbefffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf00136fb0 0x7fbf00136030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbefffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf00136fb0 0x7fbf00136030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51372/0 (socket says 172.21.15.12:51372) 2024-09-11T01:34:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbefffff640 1 -- 172.21.15.12:0/4192134897 learned_addr learned my addr 172.21.15.12:0/4192134897 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:34:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbefffff640 10 monclient(hunting): get_auth_request con 0x7fbf00136fb0 auth_method 0 2024-09-11T01:34:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbefffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbefffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbefffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:34:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbf04f9d640 1 --2- 172.21.15.12:0/4192134897 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf0009cff0 0x7fbf00135af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:21.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbefffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:34:21.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbefffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:34:21.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbefffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:34:21.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbf04f9d640 10 monclient(hunting): get_auth_request con 0x7fbf0009cff0 auth_method 0 2024-09-11T01:34:21.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbf04f9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:21.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbf04f9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:21.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbf04f9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:34:21.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbf04f9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:34:21.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbf04f9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:34:21.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbf04f9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:34:21.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbefffff640 10 monclient(hunting): handle_auth_done global_id 54171 payload 824 2024-09-11T01:34:21.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbefffff640 1 -- 172.21.15.12:0/4192134897 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbf00137ce0 msgr2=0x7fbf0013bb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:34:21.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbefffff640 1 --2- 172.21.15.12:0/4192134897 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbf00137ce0 0x7fbf0013bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbefffff640 1 -- 172.21.15.12:0/4192134897 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf0009cff0 msgr2=0x7fbf00135af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:34:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbefffff640 1 --2- 172.21.15.12:0/4192134897 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf0009cff0 0x7fbf00135af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbefffff640 10 monclient: _finish_hunting 0 2024-09-11T01:34:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbefffff640 1 monclient: found mon.noname-a 2024-09-11T01:34:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbefffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:34:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbefffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:34:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.620+0000 7fbefffff640 1 -- 172.21.15.12:0/4192134897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf0013f660 con 0x7fbf00136fb0 2024-09-11T01:34:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.623+0000 7fbefffff640 1 -- 172.21.15.12:0/4192134897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf00136fb0 msgr2=0x7fbf00136030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:34:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.623+0000 7fbefffff640 1 --2- 172.21.15.12:0/4192134897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf00136fb0 0x7fbf00136030 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbeec0098c0 tx=0x7fbeec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:34:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.623+0000 7fbefd7fa640 1 -- 172.21.15.12:0/4192134897 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeec042020 con 0x7fbf00136fb0 2024-09-11T01:34:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.623+0000 7fbefd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:34:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.623+0000 7fbefd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:34:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.623+0000 7fbefd7fa640 10 monclient: dump: 2024-09-11T01:34:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:34:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:34:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:34:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:34:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:34:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:34:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:34:21.636 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:34:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.623+0000 7fbefd7fa640 10 monclient: _finish_auth 0 2024-09-11T01:34:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.623+0000 7fbefd7fa640 10 monclient: _check_auth_tickets 2024-09-11T01:34:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.623+0000 7fbefd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:34:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.623+0000 7fbefd7fa640 1 -- 172.21.15.12:0/4192134897 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbeec032e10 con 0x7fbf00136fb0 2024-09-11T01:34:21.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.623+0000 7fbf05f9f640 5 monclient: authenticate success, global_id 54171 2024-09-11T01:34:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.623+0000 7fbf05f9f640 10 monclient: _renew_subs 2024-09-11T01:34:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.623+0000 7fbf05f9f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:34:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.623+0000 7fbf05f9f640 1 -- 172.21.15.12:0/4192134897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf0013f940 con 0x7fbf00136fb0 2024-09-11T01:34:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.623+0000 7fbefd7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:34:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.623+0000 7fbefd7fa640 1 -- 172.21.15.12:0/4192134897 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeec037040 con 0x7fbf00136fb0 2024-09-11T01:34:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.623+0000 7fbefd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:34:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.623+0000 7fbefd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:34:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.623+0000 7fbefd7fa640 10 monclient: dump: 2024-09-11T01:34:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:34:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:34:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.637 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:34:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:34:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:34:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:34:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:34:21.638 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:34:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.623+0000 7fbf05f9f640 10 monclient: _renew_subs 2024-09-11T01:34:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.623+0000 7fbf05f9f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:34:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.623+0000 7fbefeffd640 4 set_mon_vals no callback set 2024-09-11T01:34:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.623+0000 7fbf05f9f640 1 -- 172.21.15.12:0/4192134897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf000a16a0 con 0x7fbf00136fb0 2024-09-11T01:34:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.624+0000 7fbf05f9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:34:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.624+0000 7fbf05f9f640 1 --2- 172.21.15.12:0/4192134897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf000a25f0 0x7fbf000a29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.624+0000 7fbf05f9f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.624+0000 7fbf05f9f640 1 -- 172.21.15.12:0/4192134897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbf00137ae0 con 0x7fbf000a25f0 2024-09-11T01:34:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.624+0000 7fbf04f9d640 1 --2- 172.21.15.12:0/4192134897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf000a25f0 0x7fbf000a29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.624+0000 7fbefd7fa640 1 -- 172.21.15.12:0/4192134897 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fbeec030aa0 con 0x7fbf00136fb0 2024-09-11T01:34:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.624+0000 7fbf04f9d640 10 monclient: get_auth_request con 0x7fbf000a25f0 auth_method 0 2024-09-11T01:34:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.624+0000 7fbf04f9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.624+0000 7fbf04f9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.624+0000 7fbf04f9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:34:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.624+0000 7fbf04f9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:34:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.624+0000 7fbf04f9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:34:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.624+0000 7fbf04f9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:34:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.624+0000 7fbf04f9d640 10 monclient(hunting): handle_auth_done global_id 54174 payload 824 2024-09-11T01:34:21.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.624+0000 7fbefd7fa640 1 --2- 172.21.15.12:0/4192134897 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbee403e1c0 0x7fbee4040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.624+0000 7fbf04f9d640 1 -- 172.21.15.12:0/4192134897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbf0009d430 con 0x7fbf000a25f0 2024-09-11T01:34:21.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.624+0000 7fbefd7fa640 1 -- 172.21.15.12:0/4192134897 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(175..175 src has 1..175) ==== 4799+0+0 (secure 0 0 0) 0x7fbeec073110 con 0x7fbf00136fb0 2024-09-11T01:34:21.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.624+0000 7fbf0579e640 1 --2- 172.21.15.12:0/4192134897 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbee403e1c0 0x7fbee4040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:21.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.624+0000 7fbf0579e640 10 monclient: get_auth_request con 0x7fbee403e1c0 auth_method 0 2024-09-11T01:34:21.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.626+0000 7fbf04f9d640 1 -- 172.21.15.12:0/4192134897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf000a25f0 msgr2=0x7fbf000a29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:34:21.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.626+0000 7fbf04f9d640 1 --2- 172.21.15.12:0/4192134897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf000a25f0 0x7fbf000a29d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009610 tx=0x7fbef4009ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:34:21.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.626+0000 7fbefd7fa640 1 -- 172.21.15.12:0/4192134897 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef4016070 con 0x7fbf000a25f0 2024-09-11T01:34:21.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.626+0000 7fbefd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:34:21.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.626+0000 7fbefd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:34:21.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.626+0000 7fbefd7fa640 10 monclient: dump: 2024-09-11T01:34:21.642 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:34:21.643 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:34:21.643 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.643 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.643 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:34:21.643 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:34:21.643 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:34:21.643 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:34:21.643 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:34:21.643 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:34:21.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.627+0000 7fbefd7fa640 1 -- 172.21.15.12:0/4192134897 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbef4005a40 con 0x7fbf000a25f0 2024-09-11T01:34:21.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.627+0000 7fbefd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:34:21.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.627+0000 7fbefd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:34:21.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.627+0000 7fbf0579e640 1 --2- 172.21.15.12:0/4192134897 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbee403e1c0 0x7fbee4040680 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbef8004b90 tx=0x7fbef8005080 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:34:21.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.627+0000 7fbefd7fa640 1 -- 172.21.15.12:0/4192134897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf000a25f0 msgr2=0x7fbf000a29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:34:21.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.627+0000 7fbefd7fa640 1 --2- 172.21.15.12:0/4192134897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf000a25f0 0x7fbf000a29d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009610 tx=0x7fbef4009ad0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.627+0000 7fbefd7fa640 1 -- 172.21.15.12:0/4192134897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf000a25f0 msgr2=0x7fbf000a29d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:34:21.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.627+0000 7fbefd7fa640 1 --2- 172.21.15.12:0/4192134897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf000a25f0 0x7fbf000a29d0 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009610 tx=0x7fbef4009ad0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.633+0000 7fbf05f9f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:34:21.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.633+0000 7fbf05f9f640 1 --2- 172.21.15.12:0/4192134897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf000a2f10 0x7fbf0013c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.633+0000 7fbf05f9f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:21.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.633+0000 7fbf05f9f640 1 -- 172.21.15.12:0/4192134897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbf000a7b70 con 0x7fbf000a2f10 2024-09-11T01:34:21.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.633+0000 7fbf04f9d640 1 --2- 172.21.15.12:0/4192134897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf000a2f10 0x7fbf0013c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:21.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.633+0000 7fbf04f9d640 10 monclient: get_auth_request con 0x7fbf000a2f10 auth_method 0 2024-09-11T01:34:21.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.633+0000 7fbf04f9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:21.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.633+0000 7fbf04f9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:21.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.633+0000 7fbf04f9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:34:21.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.633+0000 7fbf04f9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:34:21.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.633+0000 7fbf04f9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:34:21.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.633+0000 7fbf04f9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:34:21.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.633+0000 7fbf04f9d640 10 monclient(hunting): handle_auth_done global_id 54177 payload 824 2024-09-11T01:34:21.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.633+0000 7fbf04f9d640 1 -- 172.21.15.12:0/4192134897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbf0013f3e0 con 0x7fbf000a2f10 2024-09-11T01:34:21.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.635+0000 7fbf04f9d640 1 -- 172.21.15.12:0/4192134897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf000a2f10 msgr2=0x7fbf0013c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:34:21.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.635+0000 7fbf04f9d640 1 --2- 172.21.15.12:0/4192134897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf000a2f10 0x7fbf0013c060 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009610 tx=0x7fbef40059d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:34:21.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.635+0000 7fbefd7fa640 1 -- 172.21.15.12:0/4192134897 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef4016050 con 0x7fbf000a2f10 2024-09-11T01:34:21.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.635+0000 7fbefd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:34:21.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.635+0000 7fbefd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:34:21.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.635+0000 7fbefd7fa640 10 monclient: dump: 2024-09-11T01:34:21.646 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:34:21.646 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:34:21.646 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.646 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.646 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:34:21.646 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:34:21.646 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:34:21.646 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:34:21.646 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:34:21.646 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:34:21.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.636+0000 7fbefd7fa640 1 -- 172.21.15.12:0/4192134897 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4698 (secure 0 0 0) 0x7fbef400e3f0 con 0x7fbf000a2f10 2024-09-11T01:34:21.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.636+0000 7fbefd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:34:21.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.636+0000 7fbefd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:34:21.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.636+0000 7fbefd7fa640 1 -- 172.21.15.12:0/4192134897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf000a2f10 msgr2=0x7fbf0013c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:34:21.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.636+0000 7fbefd7fa640 1 --2- 172.21.15.12:0/4192134897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf000a2f10 0x7fbf0013c060 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009610 tx=0x7fbef40059d0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.636+0000 7fbefd7fa640 1 -- 172.21.15.12:0/4192134897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf000a2f10 msgr2=0x7fbf0013c060 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:34:21.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.636+0000 7fbefd7fa640 1 --2- 172.21.15.12:0/4192134897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf000a2f10 0x7fbf0013c060 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009610 tx=0x7fbef40059d0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.647 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:34:21.647 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:34:21.647 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:34:21.647 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:34:21.647 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 58, 2024-09-11T01:34:21.648 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 51616, 2024-09-11T01:34:21.648 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:34:21.648 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:34:21.648 INFO:teuthology.orchestra.run.smithi012.stdout: 1 2024-09-11T01:34:21.648 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:34:21.648 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:34:21.648 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:34:21.648 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:34:21.648 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:34:21.648 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:34:21.648 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:34:21.648 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:34:21.648 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:34:21.649 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:34:21.649 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:34:21.649 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:34:21.649 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:34:21.649 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:34:21.649 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:34:21.649 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:34:21.649 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:34:21.649 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:34:21.649 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:34:21.649 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:34:21.649 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:34:21.649 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:34:21.649 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:34:21.650 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:34:21.650 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:34:21.650 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:34:21.650 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:34:21.650 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:34:21.650 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:34:21.650 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:34:21.650 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:34:21.650 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:34:21.650 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:34:21.650 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:34:21.650 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:34:21.650 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:34:21.651 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:34:21.651 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:34:21.651 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:34:21.651 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:34:21.651 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:34:21.651 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:34:21.651 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:34:21.651 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:34:21.651 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:34:21.651 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:34:21.651 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:34:21.651 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:34:21.651 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:34:21.651 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:34:21.651 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:34:21.652 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:34:21.652 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:34:21.652 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:34:21.652 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:34:21.652 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:34:21.652 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:34:21.652 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:34:21.652 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:34:21.652 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:34:21.652 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:34:21.652 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:34:21.652 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:34:21.652 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:34:21.653 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:34:21.653 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:34:21.653 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:34:21.653 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:34:21.653 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:34:21.653 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:34:21.653 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:34:21.653 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:34:21.653 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:34:21.653 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:34:21.653 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:34:21.653 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:34:21.654 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:34:21.654 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:34:21.654 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:34:21.654 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:34:21.654 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:34:21.654 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:34:21.654 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:34:21.654 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:34:21.654 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:34:21.654 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:34:21.654 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:34:21.654 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:34:21.654 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:34:21.654 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:34:21.654 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:34:21.655 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:34:21.655 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:34:21.655 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:34:21.655 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:34:21.655 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:34:21.655 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:34:21.655 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:34:21.655 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:34:21.655 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:34:21.655 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:34:21.655 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:34:21.655 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:34:21.655 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:34:21.656 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:34:21.656 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:34:21.656 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:34:21.656 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:34:21.656 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:34:21.656 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:34:21.656 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:34:21.656 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:34:21.656 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:34:21.656 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:34:21.656 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:34:21.656 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:34:21.656 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:34:21.656 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:34:21.657 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:34:21.657 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:34:21.657 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:34:21.657 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:34:21.657 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:34:21.657 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:34:21.657 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:34:21.657 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:34:21.657 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:34:21.657 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:34:21.657 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:34:21.657 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:34:21.657 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:34:21.658 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:34:21.658 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:34:21.658 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:34:21.658 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:34:21.658 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:34:21.658 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:34:21.658 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:34:21.658 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:34:21.658 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:34:21.658 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:34:21.658 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:34:21.658 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:34:21.658 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:34:21.659 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:34:21.659 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:34:21.659 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:34:21.659 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:34:21.659 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:34:21.659 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:34:21.659 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:34:21.659 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:34:21.659 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:34:21.659 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:34:21.659 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:34:21.659 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:34:21.659 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:34:21.659 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:34:21.660 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:34:21.660 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:34:21.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.636+0000 7fbf05f9f640 1 -- 172.21.15.12:0/4192134897 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbee403e1c0 msgr2=0x7fbee4040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:34:21.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.636+0000 7fbf05f9f640 1 --2- 172.21.15.12:0/4192134897 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbee403e1c0 0x7fbee4040680 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbef8004b90 tx=0x7fbef8005080 comp rx=0 tx=0).stop 2024-09-11T01:34:21.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.636+0000 7fbf05f9f640 10 monclient: shutdown 2024-09-11T01:34:21.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.636+0000 7fbf05f9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:34:21.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.636+0000 7fbf05f9f640 1 -- 172.21.15.12:0/4192134897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf00136fb0 msgr2=0x7fbf00136030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:34:21.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.636+0000 7fbf05f9f640 1 --2- 172.21.15.12:0/4192134897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf00136fb0 0x7fbf00136030 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbeec0098c0 tx=0x7fbeec0046c0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.636+0000 7fbf0579e640 1 -- 172.21.15.12:0/4192134897 reap_dead start 2024-09-11T01:34:21.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.636+0000 7fbf05f9f640 1 -- 172.21.15.12:0/4192134897 shutdown_connections 2024-09-11T01:34:21.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.636+0000 7fbf05f9f640 1 --2- 172.21.15.12:0/4192134897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf00136fb0 0x7fbf00136030 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.636+0000 7fbf05f9f640 1 -- 172.21.15.12:0/4192134897 >> 172.21.15.12:0/4192134897 conn(0x7fbf000ab190 msgr2=0x7fbf000a1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:34:21.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.636+0000 7fbf05f9f640 1 -- 172.21.15.12:0/4192134897 shutdown_connections 2024-09-11T01:34:21.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.636+0000 7fbf05f9f640 1 -- 172.21.15.12:0/4192134897 wait complete. 2024-09-11T01:34:21.662 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:34:21.662 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T01:34:21.663 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:34:21.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.763+0000 7f31731f8640 10 monclient: get_monmap_and_config 2024-09-11T01:34:21.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.763+0000 7f31731f8640 10 monclient: build_initial_monmap 2024-09-11T01:34:21.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.763+0000 7f31731f8640 10 monclient: monmap: 2024-09-11T01:34:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:34:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:34:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:34:21.764236+0000 2024-09-11T01:34:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:34:21.764236+0000 2024-09-11T01:34:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:34:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:34:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:34:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:34:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:34:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:34:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:34:21.766 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:34:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.764+0000 7f31731f8640 1 Processor -- start 2024-09-11T01:34:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.764+0000 7f31731f8640 1 -- start start 2024-09-11T01:34:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.764+0000 7f31731f8640 10 monclient: init 2024-09-11T01:34:21.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.764+0000 7f31731f8640 10 monclient: _reopen_session rank -1 2024-09-11T01:34:21.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.764+0000 7f31731f8640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-11T01:34:21.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.764+0000 7f31731f8640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f316c134ad0 0x7f316c134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.764+0000 7f31731f8640 10 monclient(hunting): picked mon.noname-c con 0x7f316c134ad0 addr v2:172.21.15.71:3300/0 2024-09-11T01:34:21.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.764+0000 7f31731f8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f316c0a2200 addr v1:172.21.15.12:6789/0 2024-09-11T01:34:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.764+0000 7f31731f8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f316c1353f0 0x7f316c134160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.764+0000 7f31731f8640 10 monclient(hunting): picked mon.noname-b con 0x7f316c1353f0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:34:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.764+0000 7f31731f8640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f316c09ed30 con 0x7f316c0a2200 2024-09-11T01:34:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.764+0000 7f31731f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.764+0000 7f31731f8640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f316c09f5d0 con 0x7f316c134ad0 2024-09-11T01:34:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.764+0000 7f31731f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.764+0000 7f31731f8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f316c09f010 con 0x7f316c1353f0 2024-09-11T01:34:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.764+0000 7f31731f8640 10 monclient(hunting): _renew_subs 2024-09-11T01:34:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.764+0000 7f31731f8640 10 monclient(hunting): authenticate will time out at 1068.206799s 2024-09-11T01:34:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.764+0000 7f3171795640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f316c1353f0 msgr2=0x7f316c134160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:34:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.764+0000 7f3171795640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f316c1353f0 0x7f316c134160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:34:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.764+0000 7f316bfff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f316c0a2200 0x7f316c133a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58868/0 (socket says 172.21.15.12:58868) 2024-09-11T01:34:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.764+0000 7f316bfff640 1 -- 172.21.15.12:0/1591263467 learned_addr learned my addr 172.21.15.12:0/1591263467 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:34:21.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.765+0000 7f3170f94640 1 --2- 172.21.15.12:0/1591263467 >> v2:172.21.15.71:3300/0 conn(0x7f316c134ad0 0x7f316c134eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.765+0000 7f3170f94640 10 monclient(hunting): get_auth_request con 0x7f316c134ad0 auth_method 0 2024-09-11T01:34:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.765+0000 7f3170f94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.765+0000 7f3170f94640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.765+0000 7f3170f94640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:34:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.765+0000 7f3170f94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:34:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.765+0000 7f3170f94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:34:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.765+0000 7f3170f94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:34:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.765+0000 7f3170f94640 10 monclient(hunting): handle_auth_done global_id 34339 payload 274 2024-09-11T01:34:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.765+0000 7f3170f94640 1 -- 172.21.15.12:0/1591263467 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f316c1353f0 msgr2=0x7f316c134160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:34:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.765+0000 7f3170f94640 1 --2- 172.21.15.12:0/1591263467 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f316c1353f0 0x7f316c134160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.765+0000 7f3170f94640 1 -- 172.21.15.12:0/1591263467 >> v1:172.21.15.12:6789/0 conn(0x7f316c0a2200 legacy=0x7f316c133a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:34:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.765+0000 7f3170f94640 10 monclient: _finish_hunting 0 2024-09-11T01:34:21.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.765+0000 7f3170f94640 1 monclient: found mon.noname-c 2024-09-11T01:34:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.765+0000 7f3170f94640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:34:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.765+0000 7f3170f94640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:34:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.765+0000 7f3170f94640 1 -- 172.21.15.12:0/1591263467 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f316c09f2f0 con 0x7f316c134ad0 2024-09-11T01:34:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.768+0000 7f3170f94640 1 -- 172.21.15.12:0/1591263467 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f316c134ad0 msgr2=0x7f316c134eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:34:21.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.768+0000 7f3170f94640 1 --2- 172.21.15.12:0/1591263467 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f316c134ad0 0x7f316c134eb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f31600097d0 tx=0x7f316002a010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0a28cf1dd1fa864 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:34:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.768+0000 7f316b7fe640 1 -- 172.21.15.12:0/1591263467 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3160037070 con 0x7f316c134ad0 2024-09-11T01:34:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.768+0000 7f316b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:34:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.768+0000 7f316b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:34:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.768+0000 7f316b7fe640 10 monclient: dump: 2024-09-11T01:34:21.777 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:34:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:34:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:34:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:34:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:34:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:34:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:34:21.778 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:34:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.768+0000 7f316b7fe640 10 monclient: _finish_auth 0 2024-09-11T01:34:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.768+0000 7f316b7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:34:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.768+0000 7f316b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:34:21.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.768+0000 7f31731f8640 5 monclient: authenticate success, global_id 34339 2024-09-11T01:34:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.768+0000 7f316b7fe640 1 -- 172.21.15.12:0/1591263467 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3160032640 con 0x7f316c134ad0 2024-09-11T01:34:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.768+0000 7f31731f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:34:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.768+0000 7f316b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:34:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.768+0000 7f31731f8640 10 monclient: get_monmap_and_config success 2024-09-11T01:34:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.768+0000 7f31731f8640 4 set_mon_vals no callback set 2024-09-11T01:34:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.768+0000 7f316b7fe640 1 -- 172.21.15.12:0/1591263467 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3160032a10 con 0x7f316c134ad0 2024-09-11T01:34:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.768+0000 7f316b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:34:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.768+0000 7f316b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:34:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.768+0000 7f316b7fe640 10 monclient: dump: 2024-09-11T01:34:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:34:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:34:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.779 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.780 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:34:21.780 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:34:21.780 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:34:21.780 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:34:21.780 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:34:21.780 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:34:21.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.769+0000 7f31731f8640 10 monclient: shutdown 2024-09-11T01:34:21.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.769+0000 7f31731f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:34:21.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.769+0000 7f31731f8640 1 -- 172.21.15.12:0/1591263467 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f316c134ad0 msgr2=0x7f316c134eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:34:21.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.769+0000 7f31731f8640 1 --2- 172.21.15.12:0/1591263467 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f316c134ad0 0x7f316c134eb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f31600097d0 tx=0x7f316002a010 comp rx=0 tx=0).stop 2024-09-11T01:34:21.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.769+0000 7f31731f8640 1 -- 172.21.15.12:0/1591263467 shutdown_connections 2024-09-11T01:34:21.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.769+0000 7f31731f8640 1 --2- 172.21.15.12:0/1591263467 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f316c1353f0 0x7f316c134160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.769+0000 7f31731f8640 1 --2- 172.21.15.12:0/1591263467 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f316c134ad0 0x7f316c134eb0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.769+0000 7f31731f8640 1 -- 172.21.15.12:0/1591263467 >> 172.21.15.12:0/1591263467 conn(0x7f316c0a2f70 msgr2=0x7f316c0a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:34:21.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.769+0000 7f31731f8640 1 -- 172.21.15.12:0/1591263467 shutdown_connections 2024-09-11T01:34:21.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.769+0000 7f31731f8640 1 -- 172.21.15.12:0/1591263467 wait complete. 2024-09-11T01:34:21.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.769+0000 7f31731f8640 10 monclient: build_initial_monmap 2024-09-11T01:34:21.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.769+0000 7f31731f8640 10 monclient: monmap: 2024-09-11T01:34:21.781 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:34:21.781 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:34:21.781 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:34:21.781 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:34:21.781 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:34:21.781 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:34:21.781 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:34:21.781 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:34:21.782 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:34:21.782 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:34:21.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f31731f8640 1 Processor -- start 2024-09-11T01:34:21.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f31731f8640 1 -- start start 2024-09-11T01:34:21.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f31731f8640 10 monclient: init 2024-09-11T01:34:21.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f31731f8640 10 monclient: _reopen_session rank -1 2024-09-11T01:34:21.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f31731f8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:34:21.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f31731f8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f316c0a2200 0x7f316c1cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f31731f8640 10 monclient(hunting): picked mon.noname-b con 0x7f316c0a2200 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:34:21.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f31731f8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f316c134ad0 0x7f316c1339b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f31731f8640 10 monclient(hunting): picked mon.noname-a con 0x7f316c134ad0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:34:21.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f31731f8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f316c1353f0 0x7f316c13e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f31731f8640 10 monclient(hunting): picked mon.noname-c con 0x7f316c1353f0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:34:21.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f31731f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:21.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f31731f8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f316c134820 con 0x7f316c134ad0 2024-09-11T01:34:21.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f31731f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:21.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f31731f8640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f316c1346a0 con 0x7f316c0a2200 2024-09-11T01:34:21.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f31731f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:21.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f31731f8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f316c09ffd0 con 0x7f316c1353f0 2024-09-11T01:34:21.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f31731f8640 10 monclient(hunting): _renew_subs 2024-09-11T01:34:21.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f31731f8640 10 monclient(hunting): authenticate will time out at 1068.212879s 2024-09-11T01:34:21.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f3171795640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f316c1353f0 msgr2=0x7f316c13e030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:34:21.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f3171795640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f316c1353f0 0x7f316c13e030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:34:21.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f316bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f316c134ad0 0x7f316c1339b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:21.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f316bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f316c134ad0 0x7f316c1339b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51394/0 (socket says 172.21.15.12:51394) 2024-09-11T01:34:21.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f316bfff640 1 -- 172.21.15.12:0/1984522508 learned_addr learned my addr 172.21.15.12:0/1984522508 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:34:21.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f316bfff640 10 monclient(hunting): get_auth_request con 0x7f316c134ad0 auth_method 0 2024-09-11T01:34:21.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f316bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:21.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f316bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:21.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.770+0000 7f316bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:34:21.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.771+0000 7f316bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:34:21.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.771+0000 7f316bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:34:21.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.771+0000 7f316bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:34:21.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.771+0000 7f316bfff640 10 monclient(hunting): handle_auth_done global_id 54180 payload 824 2024-09-11T01:34:21.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.771+0000 7f3170f94640 1 --2- 172.21.15.12:0/1984522508 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f316c0a2200 0x7f316c1cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:21.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.771+0000 7f316bfff640 1 -- 172.21.15.12:0/1984522508 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f316c1353f0 msgr2=0x7f316c13e030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:34:21.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.771+0000 7f316bfff640 1 --2- 172.21.15.12:0/1984522508 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f316c1353f0 0x7f316c13e030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.771+0000 7f316bfff640 1 -- 172.21.15.12:0/1984522508 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f316c0a2200 msgr2=0x7f316c1cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:34:21.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.771+0000 7f316bfff640 1 --2- 172.21.15.12:0/1984522508 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f316c0a2200 0x7f316c1cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.771+0000 7f316bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:34:21.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.771+0000 7f316bfff640 1 monclient: found mon.noname-a 2024-09-11T01:34:21.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.771+0000 7f316bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:34:21.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.771+0000 7f316bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:34:21.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.771+0000 7f316bfff640 1 -- 172.21.15.12:0/1984522508 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f316c0a4390 con 0x7f316c134ad0 2024-09-11T01:34:21.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.773+0000 7f316bfff640 1 -- 172.21.15.12:0/1984522508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f316c134ad0 msgr2=0x7f316c1339b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:34:21.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.773+0000 7f316bfff640 1 --2- 172.21.15.12:0/1984522508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f316c134ad0 0x7f316c1339b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f315800c430 tx=0x7f315800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:34:21.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.773+0000 7f31697fa640 1 -- 172.21.15.12:0/1984522508 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3158017070 con 0x7f316c134ad0 2024-09-11T01:34:21.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.773+0000 7f31697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:34:21.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.773+0000 7f31697fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:34:21.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.773+0000 7f31697fa640 10 monclient: dump: 2024-09-11T01:34:21.785 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:34:21.786 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:34:21.786 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.786 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.786 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:34:21.786 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:34:21.786 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:34:21.786 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:34:21.786 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:34:21.786 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:34:21.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.773+0000 7f31697fa640 10 monclient: _finish_auth 0 2024-09-11T01:34:21.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.773+0000 7f31697fa640 10 monclient: _check_auth_tickets 2024-09-11T01:34:21.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.773+0000 7f31697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:34:21.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.773+0000 7f31697fa640 1 -- 172.21.15.12:0/1984522508 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31580070d0 con 0x7f316c134ad0 2024-09-11T01:34:21.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.773+0000 7f31697fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:34:21.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.773+0000 7f31697fa640 1 -- 172.21.15.12:0/1984522508 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31580074c0 con 0x7f316c134ad0 2024-09-11T01:34:21.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.773+0000 7f31697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:34:21.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.773+0000 7f31697fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:34:21.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.773+0000 7f31697fa640 10 monclient: dump: 2024-09-11T01:34:21.787 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:34:21.787 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:34:21.787 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.787 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.787 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:34:21.787 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:34:21.787 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:34:21.788 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:34:21.788 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:34:21.788 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:34:21.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.773+0000 7f31731f8640 5 monclient: authenticate success, global_id 54180 2024-09-11T01:34:21.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.773+0000 7f316affd640 4 set_mon_vals no callback set 2024-09-11T01:34:21.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.773+0000 7f31731f8640 10 monclient: _renew_subs 2024-09-11T01:34:21.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.773+0000 7f31731f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:34:21.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.773+0000 7f31731f8640 1 -- 172.21.15.12:0/1984522508 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f316c09d480 con 0x7f316c134ad0 2024-09-11T01:34:21.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.773+0000 7f31731f8640 10 monclient: _renew_subs 2024-09-11T01:34:21.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.773+0000 7f31731f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:34:21.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.773+0000 7f31731f8640 1 -- 172.21.15.12:0/1984522508 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f316c11cc40 con 0x7f316c134ad0 2024-09-11T01:34:21.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.774+0000 7f31731f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:34:21.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.774+0000 7f31731f8640 1 --2- 172.21.15.12:0/1984522508 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f316c133ef0 0x7f316c13e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.774+0000 7f31731f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:21.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.774+0000 7f31731f8640 1 -- 172.21.15.12:0/1984522508 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f316c134eb0 con 0x7f316c133ef0 2024-09-11T01:34:21.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.774+0000 7f31697fa640 1 -- 172.21.15.12:0/1984522508 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f315801b3b0 con 0x7f316c134ad0 2024-09-11T01:34:21.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.774+0000 7f3170f94640 1 --2- 172.21.15.12:0/1984522508 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f316c133ef0 0x7f316c13e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:21.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.774+0000 7f3170f94640 10 monclient: get_auth_request con 0x7f316c133ef0 auth_method 0 2024-09-11T01:34:21.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.774+0000 7f3170f94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:21.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.774+0000 7f3170f94640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:21.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.774+0000 7f3170f94640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:34:21.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.774+0000 7f31697fa640 1 --2- 172.21.15.12:0/1984522508 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f315003e1f0 0x7f31500406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.774+0000 7f31697fa640 1 -- 172.21.15.12:0/1984522508 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(175..175 src has 1..175) ==== 4799+0+0 (secure 0 0 0) 0x7f31580525e0 con 0x7f316c134ad0 2024-09-11T01:34:21.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.774+0000 7f3170f94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:34:21.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.774+0000 7f3170f94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:34:21.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.774+0000 7f3170f94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:34:21.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.775+0000 7f3171795640 1 --2- 172.21.15.12:0/1984522508 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f315003e1f0 0x7f31500406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:21.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.775+0000 7f3171795640 10 monclient: get_auth_request con 0x7f315003e1f0 auth_method 0 2024-09-11T01:34:21.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.775+0000 7f3170f94640 10 monclient(hunting): handle_auth_done global_id 34342 payload 824 2024-09-11T01:34:21.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.775+0000 7f3170f94640 1 -- 172.21.15.12:0/1984522508 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f316c0a1a60 con 0x7f316c133ef0 2024-09-11T01:34:21.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.777+0000 7f3170f94640 1 -- 172.21.15.12:0/1984522508 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f316c133ef0 msgr2=0x7f316c13e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:34:21.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.777+0000 7f3170f94640 1 --2- 172.21.15.12:0/1984522508 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f316c133ef0 0x7f316c13e570 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f31600097a0 tx=0x7f31600046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:34:21.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.777+0000 7f31697fa640 1 -- 172.21.15.12:0/1984522508 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3160042070 con 0x7f316c133ef0 2024-09-11T01:34:21.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.777+0000 7f31697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:34:21.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.777+0000 7f31697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:34:21.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.777+0000 7f31697fa640 10 monclient: dump: 2024-09-11T01:34:21.792 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:34:21.793 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:34:21.793 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.793 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.793 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:34:21.793 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:34:21.793 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:34:21.793 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:34:21.793 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:34:21.793 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:34:21.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.777+0000 7f3171795640 1 --2- 172.21.15.12:0/1984522508 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f315003e1f0 0x7f31500406b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3164003cb0 tx=0x7f31640091f0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:34:21.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.777+0000 7f31697fa640 1 -- 172.21.15.12:0/1984522508 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f316003e870 con 0x7f316c133ef0 2024-09-11T01:34:21.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.777+0000 7f31697fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:34:21.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.777+0000 7f31697fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:34:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.777+0000 7f31697fa640 1 -- 172.21.15.12:0/1984522508 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f316c133ef0 msgr2=0x7f316c13e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:34:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.777+0000 7f31697fa640 1 --2- 172.21.15.12:0/1984522508 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f316c133ef0 0x7f316c13e570 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f31600097a0 tx=0x7f31600046c0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.777+0000 7f31697fa640 1 -- 172.21.15.12:0/1984522508 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f316c133ef0 msgr2=0x7f316c13e570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:34:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.777+0000 7f31697fa640 1 --2- 172.21.15.12:0/1984522508 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f316c133ef0 0x7f316c13e570 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f31600097a0 tx=0x7f31600046c0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.784+0000 7f31731f8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:34:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.784+0000 7f31731f8640 1 --2- 172.21.15.12:0/1984522508 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f316c13eab0 0x7f316c0726b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.784+0000 7f31731f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.784+0000 7f31731f8640 1 -- 172.21.15.12:0/1984522508 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f316c13eec0 con 0x7f316c13eab0 2024-09-11T01:34:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.784+0000 7f3170f94640 1 --2- 172.21.15.12:0/1984522508 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f316c13eab0 0x7f316c0726b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.784+0000 7f3170f94640 10 monclient: get_auth_request con 0x7f316c13eab0 auth_method 0 2024-09-11T01:34:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.784+0000 7f3170f94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.784+0000 7f3170f94640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.784+0000 7f3170f94640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:34:21.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.784+0000 7f3170f94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:34:21.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.784+0000 7f3170f94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:34:21.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.784+0000 7f3170f94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:34:21.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.784+0000 7f3170f94640 10 monclient(hunting): handle_auth_done global_id 34345 payload 824 2024-09-11T01:34:21.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.784+0000 7f3170f94640 1 -- 172.21.15.12:0/1984522508 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f316c11df60 con 0x7f316c13eab0 2024-09-11T01:34:21.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.787+0000 7f3170f94640 1 -- 172.21.15.12:0/1984522508 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f316c13eab0 msgr2=0x7f316c0726b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:34:21.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.787+0000 7f3170f94640 1 --2- 172.21.15.12:0/1984522508 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f316c13eab0 0x7f316c0726b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f31600097d0 tx=0x7f3160008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:34:21.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.787+0000 7f31697fa640 1 -- 172.21.15.12:0/1984522508 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3160037040 con 0x7f316c13eab0 2024-09-11T01:34:21.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.787+0000 7f31697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:34:21.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.787+0000 7f31697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:34:21.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.787+0000 7f31697fa640 10 monclient: dump: 2024-09-11T01:34:21.795 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:34:21.795 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:34:21.795 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.796 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:34:21.796 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:34:21.796 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:34:21.796 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:34:21.796 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:34:21.796 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:34:21.796 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:34:21.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.787+0000 7f31697fa640 1 -- 172.21.15.12:0/1984522508 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4864 (secure 0 0 0) 0x7f31600044d0 con 0x7f316c13eab0 2024-09-11T01:34:21.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.787+0000 7f31697fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:34:21.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.787+0000 7f31697fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:34:21.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.787+0000 7f31697fa640 1 -- 172.21.15.12:0/1984522508 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f316c13eab0 msgr2=0x7f316c0726b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:34:21.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.787+0000 7f31697fa640 1 --2- 172.21.15.12:0/1984522508 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f316c13eab0 0x7f316c0726b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f31600097d0 tx=0x7f3160008030 comp rx=0 tx=0).stop 2024-09-11T01:34:21.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.787+0000 7f31697fa640 1 -- 172.21.15.12:0/1984522508 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f316c13eab0 msgr2=0x7f316c0726b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:34:21.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.787+0000 7f31697fa640 1 --2- 172.21.15.12:0/1984522508 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f316c13eab0 0x7f316c0726b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.797 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:34:21.797 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:34:21.797 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:34:21.797 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:34:21.797 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 58, 2024-09-11T01:34:21.798 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 155001, 2024-09-11T01:34:21.798 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:34:21.798 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:34:21.798 INFO:teuthology.orchestra.run.smithi012.stdout: 1 2024-09-11T01:34:21.798 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:34:21.798 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:34:21.798 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:34:21.798 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:34:21.798 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:34:21.798 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:34:21.798 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:34:21.798 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:34:21.798 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:34:21.798 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:34:21.799 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:34:21.799 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:34:21.799 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:34:21.799 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:34:21.799 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:34:21.799 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:34:21.799 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:34:21.799 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:34:21.799 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:34:21.799 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:34:21.799 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:34:21.799 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:34:21.799 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:34:21.800 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:34:21.800 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:34:21.800 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:34:21.800 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:34:21.800 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:34:21.800 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:34:21.800 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:34:21.800 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:34:21.800 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:34:21.800 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:34:21.800 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:34:21.800 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:34:21.800 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:34:21.801 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:34:21.801 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:34:21.801 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:34:21.801 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:34:21.801 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:34:21.801 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:34:21.801 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:34:21.801 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:34:21.801 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:34:21.801 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:34:21.801 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:34:21.801 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:34:21.801 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:34:21.802 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:34:21.802 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:34:21.802 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:34:21.802 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:34:21.802 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:34:21.802 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:34:21.802 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:34:21.802 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:34:21.802 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:34:21.802 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:34:21.802 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:34:21.802 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:34:21.802 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:34:21.802 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:34:21.803 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:34:21.803 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:34:21.803 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:34:21.803 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:34:21.803 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:34:21.803 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:34:21.803 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:34:21.803 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:34:21.803 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:34:21.803 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:34:21.803 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:34:21.803 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:34:21.803 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:34:21.803 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:34:21.804 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:34:21.804 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:34:21.804 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:34:21.804 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:34:21.804 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:34:21.804 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:34:21.804 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:34:21.804 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:34:21.804 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:34:21.804 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:34:21.804 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:34:21.804 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:34:21.804 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:34:21.804 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:34:21.804 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:34:21.805 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:34:21.805 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:34:21.805 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:34:21.805 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:34:21.805 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:34:21.805 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:34:21.805 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:34:21.805 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:34:21.805 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:34:21.805 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:34:21.805 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:34:21.805 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:34:21.806 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:34:21.806 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:34:21.806 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:34:21.806 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:34:21.806 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:34:21.806 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:34:21.806 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:34:21.806 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:34:21.806 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:34:21.806 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:34:21.806 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:34:21.806 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:34:21.806 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:34:21.806 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:34:21.807 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:34:21.807 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:34:21.807 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:34:21.807 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:34:21.807 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:34:21.807 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:34:21.807 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:34:21.807 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:34:21.807 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:34:21.807 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:34:21.807 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:34:21.807 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:34:21.807 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:34:21.807 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:34:21.808 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:34:21.808 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:34:21.808 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:34:21.808 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:34:21.808 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:34:21.808 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:34:21.808 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:34:21.808 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:34:21.808 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:34:21.808 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:34:21.808 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:34:21.808 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:34:21.808 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:34:21.809 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:34:21.809 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:34:21.809 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:34:21.809 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:34:21.809 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:34:21.809 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:34:21.809 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:34:21.809 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:34:21.809 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:34:21.809 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:34:21.809 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:34:21.809 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:34:21.809 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:34:21.810 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:34:21.810 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:34:21.810 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:34:21.810 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:34:21.810 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:34:21.810 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:34:21.810 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:34:21.810 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:34:21.810 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:34:21.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.787+0000 7f31731f8640 1 -- 172.21.15.12:0/1984522508 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f315003e1f0 msgr2=0x7f31500406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:34:21.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.787+0000 7f31731f8640 1 --2- 172.21.15.12:0/1984522508 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f315003e1f0 0x7f31500406b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3164003cb0 tx=0x7f31640091f0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.787+0000 7f31731f8640 10 monclient: shutdown 2024-09-11T01:34:21.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.787+0000 7f31731f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:34:21.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.787+0000 7f31731f8640 1 -- 172.21.15.12:0/1984522508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f316c134ad0 msgr2=0x7f316c1339b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:34:21.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.787+0000 7f3171795640 1 -- 172.21.15.12:0/1984522508 reap_dead start 2024-09-11T01:34:21.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.787+0000 7f31731f8640 1 --2- 172.21.15.12:0/1984522508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f316c134ad0 0x7f316c1339b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f315800c430 tx=0x7f315800c8f0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.788+0000 7f31731f8640 1 -- 172.21.15.12:0/1984522508 shutdown_connections 2024-09-11T01:34:21.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.788+0000 7f31731f8640 1 --2- 172.21.15.12:0/1984522508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f316c134ad0 0x7f316c1339b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:21.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.788+0000 7f31731f8640 1 -- 172.21.15.12:0/1984522508 >> 172.21.15.12:0/1984522508 conn(0x7f316c0a2f70 msgr2=0x7f316c0a0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:34:21.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.788+0000 7f31731f8640 1 -- 172.21.15.12:0/1984522508 shutdown_connections 2024-09-11T01:34:21.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:21.788+0000 7f31731f8640 1 -- 172.21.15.12:0/1984522508 wait complete. 2024-09-11T01:34:21.813 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:34:21.813 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:34:22.042 INFO:tasks.rados.rados.0.smithi071.stdout:764: done (1 left) 2024-09-11T01:34:22.042 INFO:tasks.rados.rados.0.smithi071.stdout:765: done (0 left) 2024-09-11T01:34:22.042 INFO:tasks.rados.rados.0.smithi071.stdout:766: snap_remove snap 76 2024-09-11T01:34:23.049 INFO:tasks.rados.rados.0.smithi071.stdout:766: done (0 left) 2024-09-11T01:34:23.049 INFO:tasks.rados.rados.0.smithi071.stdout:767: rollback oid 15 current snap is 77 2024-09-11T01:34:23.049 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 15 to 74 2024-09-11T01:34:23.049 INFO:tasks.rados.rados.0.smithi071.stdout:768: copy_from oid 46 from oid 16 current snap is 77 2024-09-11T01:34:23.049 INFO:tasks.rados.rados.0.smithi071.stdout:769: snap_create 2024-09-11T01:34:23.056 INFO:tasks.rados.rados.0.smithi071.stdout:767: finishing rollback tid 0 to smithi07134777-15 2024-09-11T01:34:23.056 INFO:tasks.rados.rados.0.smithi071.stdout:767: finishing rollback tid 1 to smithi07134777-15 2024-09-11T01:34:23.057 INFO:tasks.rados.rados.0.smithi071.stdout:767: finishing rollback tid 2 to smithi07134777-15 2024-09-11T01:34:23.057 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 362 (ObjNum 253 snap 66 seq_num 253) dirty exists 2024-09-11T01:34:23.061 INFO:tasks.rados.rados.0.smithi071.stdout:768: finishing copy_from racing read to smithi07134777-46 2024-09-11T01:34:23.065 INFO:tasks.rados.rados.0.smithi071.stdout:768: finishing copy_from to smithi07134777-46 2024-09-11T01:34:23.065 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 210 (ObjNum 240 snap 58 seq_num 240) dirty exists 2024-09-11T01:34:24.053 INFO:tasks.rados.rados.0.smithi071.stdout:767: done (2 left) 2024-09-11T01:34:24.053 INFO:tasks.rados.rados.0.smithi071.stdout:768: done (1 left) 2024-09-11T01:34:24.053 INFO:tasks.rados.rados.0.smithi071.stdout:769: done (0 left) 2024-09-11T01:34:24.053 INFO:tasks.rados.rados.0.smithi071.stdout:770: rollback oid 11 current snap is 78 2024-09-11T01:34:24.053 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 11 to 77 2024-09-11T01:34:24.053 INFO:tasks.rados.rados.0.smithi071.stdout:771: rollback oid 4 current snap is 78 2024-09-11T01:34:24.053 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 4 to 77 2024-09-11T01:34:24.054 INFO:tasks.rados.rados.0.smithi071.stdout:772: write oid 15 current snap is 78 2024-09-11T01:34:24.054 INFO:tasks.rados.rados.0.smithi071.stdout:772: seq_num 290 ranges {474030=643624,1520131=735769} 2024-09-11T01:34:24.059 INFO:tasks.rados.rados.0.smithi071.stdout:772: writing smithi07134777-15 from 474030 to 1117654 tid 1 2024-09-11T01:34:24.066 INFO:tasks.rados.rados.0.smithi071.stdout:772: writing smithi07134777-15 from 1520131 to 2255900 tid 2 2024-09-11T01:34:24.066 INFO:tasks.rados.rados.0.smithi071.stdout:773: rollback oid 22 current snap is 78 2024-09-11T01:34:24.066 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 22 to 74 2024-09-11T01:34:24.066 INFO:tasks.rados.rados.0.smithi071.stdout:770: finishing rollback tid 0 to smithi07134777-11 2024-09-11T01:34:24.066 INFO:tasks.rados.rados.0.smithi071.stdout:770: finishing rollback tid 1 to smithi07134777-11 2024-09-11T01:34:24.066 INFO:tasks.rados.rados.0.smithi071.stdout:770: finishing rollback tid 2 to smithi07134777-11 2024-09-11T01:34:24.066 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 258 (ObjNum 263 snap 70 seq_num 263) dirty exists 2024-09-11T01:34:24.067 INFO:tasks.rados.rados.0.smithi071.stdout:771: finishing rollback tid 0 to smithi07134777-4 2024-09-11T01:34:24.067 INFO:tasks.rados.rados.0.smithi071.stdout:771: finishing rollback tid 1 to smithi07134777-4 2024-09-11T01:34:24.067 INFO:tasks.rados.rados.0.smithi071.stdout:771: finishing rollback tid 2 to smithi07134777-4 2024-09-11T01:34:24.067 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 262 (ObjNum 280 snap 73 seq_num 280) dirty exists 2024-09-11T01:34:24.067 INFO:tasks.rados.rados.0.smithi071.stdout:770: done (3 left) 2024-09-11T01:34:24.067 INFO:tasks.rados.rados.0.smithi071.stdout:771: done (2 left) 2024-09-11T01:34:24.067 INFO:tasks.rados.rados.0.smithi071.stdout:774: snap_remove snap 77 2024-09-11T01:34:25.035 INFO:tasks.rados.rados.0.smithi071.stdout:774: done (2 left) 2024-09-11T01:34:25.035 INFO:tasks.rados.rados.0.smithi071.stdout:775: read oid 46 snap 74 2024-09-11T01:34:25.035 INFO:tasks.rados.rados.0.smithi071.stdout:775: expect (ObjNum 211 snap 49 seq_num 211) 2024-09-11T01:34:25.035 INFO:tasks.rados.rados.0.smithi071.stdout:776: write oid 45 current snap is 78 2024-09-11T01:34:25.035 INFO:tasks.rados.rados.0.smithi071.stdout:776: seq_num 291 ranges {490837=718873,1862260=479427,2875032=15805} 2024-09-11T01:34:25.041 INFO:tasks.rados.rados.0.smithi071.stdout:776: writing smithi07134777-45 from 490837 to 1209710 tid 1 2024-09-11T01:34:25.048 INFO:tasks.rados.rados.0.smithi071.stdout:776: writing smithi07134777-45 from 1862260 to 2341687 tid 2 2024-09-11T01:34:25.051 INFO:tasks.rados.rados.0.smithi071.stdout:776: writing smithi07134777-45 from 2875032 to 2890837 tid 3 2024-09-11T01:34:25.052 INFO:tasks.rados.rados.0.smithi071.stdout:777: snap_create 2024-09-11T01:34:25.052 INFO:tasks.rados.rados.0.smithi071.stdout:772: finishing write tid 1 to smithi07134777-15 2024-09-11T01:34:25.052 INFO:tasks.rados.rados.0.smithi071.stdout:773: finishing rollback tid 0 to smithi07134777-22 2024-09-11T01:34:25.052 INFO:tasks.rados.rados.0.smithi071.stdout:773: finishing rollback tid 1 to smithi07134777-22 2024-09-11T01:34:25.052 INFO:tasks.rados.rados.0.smithi071.stdout:773: finishing rollback tid 2 to smithi07134777-22 2024-09-11T01:34:25.052 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 224 (ObjNum 240 snap 58 seq_num 240) dirty exists 2024-09-11T01:34:25.052 INFO:tasks.rados.rados.0.smithi071.stdout:772: finishing write tid 2 to smithi07134777-15 2024-09-11T01:34:25.053 INFO:tasks.rados.rados.0.smithi071.stdout:772: finishing write tid 3 to smithi07134777-15 2024-09-11T01:34:25.053 INFO:tasks.rados.rados.0.smithi071.stdout:772: finishing write tid 4 to smithi07134777-15 2024-09-11T01:34:25.053 INFO:tasks.rados.rados.0.smithi071.stdout:772: oid 15 updating version 0 to 366 2024-09-11T01:34:25.053 INFO:tasks.rados.rados.0.smithi071.stdout:772: oid 15 version 366 is already newer than 365 2024-09-11T01:34:25.053 INFO:tasks.rados.rados.0.smithi071.stdout:772: oid 15 version 366 is already newer than 364 2024-09-11T01:34:25.053 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 366 (ObjNum 290 snap 78 seq_num 290) dirty exists 2024-09-11T01:34:25.053 INFO:tasks.rados.rados.0.smithi071.stdout:772: left oid 15 (ObjNum 290 snap 78 seq_num 290) 2024-09-11T01:34:25.053 INFO:tasks.rados.rados.0.smithi071.stdout:776: finishing write tid 1 to smithi07134777-45 2024-09-11T01:34:25.054 INFO:tasks.rados.rados.0.smithi071.stdout:775: expect (ObjNum 211 snap 49 seq_num 211) 2024-09-11T01:34:25.078 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1713612 2024-09-11T01:34:25.249 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2909382 2024-09-11T01:34:25.259 INFO:tasks.rados.rados.0.smithi071.stdout:776: finishing write tid 2 to smithi07134777-45 2024-09-11T01:34:25.259 INFO:tasks.rados.rados.0.smithi071.stdout:776: finishing write tid 3 to smithi07134777-45 2024-09-11T01:34:25.259 INFO:tasks.rados.rados.0.smithi071.stdout:776: finishing write tid 4 to smithi07134777-45 2024-09-11T01:34:25.260 INFO:tasks.rados.rados.0.smithi071.stdout:776: finishing write tid 5 to smithi07134777-45 2024-09-11T01:34:25.260 INFO:tasks.rados.rados.0.smithi071.stdout:776: oid 45 updating version 0 to 255 2024-09-11T01:34:25.260 INFO:tasks.rados.rados.0.smithi071.stdout:776: oid 45 version 255 is already newer than 253 2024-09-11T01:34:25.260 INFO:tasks.rados.rados.0.smithi071.stdout:776: oid 45 version 255 is already newer than 254 2024-09-11T01:34:25.260 INFO:tasks.rados.rados.0.smithi071.stdout:776: oid 45 version 255 is already newer than 252 2024-09-11T01:34:25.260 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 255 (ObjNum 291 snap 78 seq_num 291) dirty exists 2024-09-11T01:34:25.260 INFO:tasks.rados.rados.0.smithi071.stdout:776: left oid 45 (ObjNum 291 snap 78 seq_num 291) 2024-09-11T01:34:26.306 INFO:tasks.rados.rados.0.smithi071.stdout:772: done (4 left) 2024-09-11T01:34:26.306 INFO:tasks.rados.rados.0.smithi071.stdout:773: done (3 left) 2024-09-11T01:34:26.306 INFO:tasks.rados.rados.0.smithi071.stdout:775: done (2 left) 2024-09-11T01:34:26.306 INFO:tasks.rados.rados.0.smithi071.stdout:776: done (1 left) 2024-09-11T01:34:26.307 INFO:tasks.rados.rados.0.smithi071.stdout:777: done (0 left) 2024-09-11T01:34:26.307 INFO:tasks.rados.rados.0.smithi071.stdout:778: write (excl) oid 34 current snap is 79 2024-09-11T01:34:26.307 INFO:tasks.rados.rados.0.smithi071.stdout:778: seq_num 292 ranges {507644=794122,1920742=623085} 2024-09-11T01:34:26.313 INFO:tasks.rados.rados.0.smithi071.stdout:778: writing smithi07134777-34 from 507644 to 1301766 tid 1 2024-09-11T01:34:26.320 INFO:tasks.rados.rados.0.smithi071.stdout:778: writing smithi07134777-34 from 1920742 to 2543827 tid 2 2024-09-11T01:34:26.320 INFO:tasks.rados.rados.0.smithi071.stdout:779: snap_remove snap 74 2024-09-11T01:34:27.323 INFO:tasks.rados.rados.0.smithi071.stdout:779: done (1 left) 2024-09-11T01:34:27.323 INFO:tasks.rados.rados.0.smithi071.stdout:780: read oid 12 snap -1 2024-09-11T01:34:27.323 INFO:tasks.rados.rados.0.smithi071.stdout:780: expect (ObjNum 289 snap 76 seq_num 289) 2024-09-11T01:34:27.323 INFO:tasks.rados.rados.0.smithi071.stdout:781: snap_create 2024-09-11T01:34:27.323 INFO:tasks.rados.rados.0.smithi071.stdout:778: finishing write tid 1 to smithi07134777-34 2024-09-11T01:34:27.323 INFO:tasks.rados.rados.0.smithi071.stdout:778: finishing write tid 2 to smithi07134777-34 2024-09-11T01:34:27.323 INFO:tasks.rados.rados.0.smithi071.stdout:778: finishing write tid 3 to smithi07134777-34 2024-09-11T01:34:27.323 INFO:tasks.rados.rados.0.smithi071.stdout:778: finishing write tid 4 to smithi07134777-34 2024-09-11T01:34:27.323 INFO:tasks.rados.rados.0.smithi071.stdout:778: oid 34 updating version 0 to 262 2024-09-11T01:34:27.324 INFO:tasks.rados.rados.0.smithi071.stdout:778: oid 34 version 262 is already newer than 260 2024-09-11T01:34:27.324 INFO:tasks.rados.rados.0.smithi071.stdout:778: oid 34 version 262 is already newer than 261 2024-09-11T01:34:27.324 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 262 (ObjNum 292 snap 79 seq_num 292) dirty exists 2024-09-11T01:34:27.324 INFO:tasks.rados.rados.0.smithi071.stdout:778: left oid 34 (ObjNum 292 snap 79 seq_num 292) 2024-09-11T01:34:27.334 INFO:tasks.rados.rados.0.smithi071.stdout:780: expect (ObjNum 289 snap 76 seq_num 289) 2024-09-11T01:34:27.342 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1406648 2024-09-11T01:34:27.446 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2067294 2024-09-11T01:34:28.322 INFO:tasks.rados.rados.0.smithi071.stdout:778: done (2 left) 2024-09-11T01:34:28.322 INFO:tasks.rados.rados.0.smithi071.stdout:780: done (1 left) 2024-09-11T01:34:28.323 INFO:tasks.rados.rados.0.smithi071.stdout:781: done (0 left) 2024-09-11T01:34:28.323 INFO:tasks.rados.rados.0.smithi071.stdout:782: delete oid 35 current snap is 80 2024-09-11T01:34:28.326 INFO:tasks.rados.rados.0.smithi071.stdout:782: done (0 left) 2024-09-11T01:34:28.326 INFO:tasks.rados.rados.0.smithi071.stdout:783: copy_from oid 29 from oid 31 current snap is 80 2024-09-11T01:34:28.326 INFO:tasks.rados.rados.0.smithi071.stdout:784: write (excl) oid 42 current snap is 80 2024-09-11T01:34:28.326 INFO:tasks.rados.rados.0.smithi071.stdout:784: seq_num 294 ranges {541258=544620,1637706=626754,2840948=100310} 2024-09-11T01:34:28.332 INFO:tasks.rados.rados.0.smithi071.stdout:784: writing smithi07134777-42 from 541258 to 1085878 tid 1 2024-09-11T01:34:28.340 INFO:tasks.rados.rados.0.smithi071.stdout:784: writing smithi07134777-42 from 1637706 to 2264460 tid 2 2024-09-11T01:34:28.343 INFO:tasks.rados.rados.0.smithi071.stdout:784: writing smithi07134777-42 from 2840948 to 2941258 tid 3 2024-09-11T01:34:28.344 INFO:tasks.rados.rados.0.smithi071.stdout:785: read oid 1 snap -1 2024-09-11T01:34:28.344 INFO:tasks.rados.rados.0.smithi071.stdout:785: expect (ObjNum 232 snap 57 seq_num 232) 2024-09-11T01:34:28.344 INFO:tasks.rados.rados.0.smithi071.stdout:786: write (excl) oid 37 current snap is 80 2024-09-11T01:34:28.344 INFO:tasks.rados.rados.0.smithi071.stdout:784: finishing write tid 1 to smithi07134777-42 2024-09-11T01:34:28.344 INFO:tasks.rados.rados.0.smithi071.stdout:786: seq_num 295 ranges {558065=619869,1579835=486765,2468371=489694} 2024-09-11T01:34:28.350 INFO:tasks.rados.rados.0.smithi071.stdout:786: writing smithi07134777-37 from 558065 to 1177934 tid 1 2024-09-11T01:34:28.355 INFO:tasks.rados.rados.0.smithi071.stdout:786: writing smithi07134777-37 from 1579835 to 2066600 tid 2 2024-09-11T01:34:28.360 INFO:tasks.rados.rados.0.smithi071.stdout:786: writing smithi07134777-37 from 2468371 to 2958065 tid 3 2024-09-11T01:34:28.360 INFO:tasks.rados.rados.0.smithi071.stdout:787: read oid 50 snap -1 2024-09-11T01:34:28.360 INFO:tasks.rados.rados.0.smithi071.stdout:787: expect (ObjNum 287 snap 74 seq_num 287) 2024-09-11T01:34:28.360 INFO:tasks.rados.rados.0.smithi071.stdout:788: write oid 17 current snap is 80 2024-09-11T01:34:28.361 INFO:tasks.rados.rados.0.smithi071.stdout:788: seq_num 296 ranges {574872=695118,2038317=630423} 2024-09-11T01:34:28.367 INFO:tasks.rados.rados.0.smithi071.stdout:788: writing smithi07134777-17 from 574872 to 1269990 tid 1 2024-09-11T01:34:28.375 INFO:tasks.rados.rados.0.smithi071.stdout:788: writing smithi07134777-17 from 2038317 to 2668740 tid 2 2024-09-11T01:34:28.375 INFO:tasks.rados.rados.0.smithi071.stdout:789: write (excl) oid 45 current snap is 80 2024-09-11T01:34:28.375 INFO:tasks.rados.rados.0.smithi071.stdout:783: finishing copy_from racing read to smithi07134777-29 2024-09-11T01:34:28.375 INFO:tasks.rados.rados.0.smithi071.stdout:789: seq_num 297 ranges {591679=486720,1813152=490434} 2024-09-11T01:34:28.381 INFO:tasks.rados.rados.0.smithi071.stdout:789: writing smithi07134777-45 from 591679 to 1078399 tid 1 2024-09-11T01:34:28.389 INFO:tasks.rados.rados.0.smithi071.stdout:789: writing smithi07134777-45 from 1813152 to 2303586 tid 2 2024-09-11T01:34:28.389 INFO:tasks.rados.rados.0.smithi071.stdout:790: read oid 6 snap -1 2024-09-11T01:34:28.389 INFO:tasks.rados.rados.0.smithi071.stdout:790: expect (ObjNum 251 snap 63 seq_num 251) 2024-09-11T01:34:28.389 INFO:tasks.rados.rados.0.smithi071.stdout:791: write oid 10 current snap is 80 2024-09-11T01:34:28.389 INFO:tasks.rados.rados.0.smithi071.stdout:783: finishing copy_from to smithi07134777-29 2024-09-11T01:34:28.389 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 296 (ObjNum 201 snap 46 seq_num 201) dirty exists 2024-09-11T01:34:28.389 INFO:tasks.rados.rados.0.smithi071.stdout:784: finishing write tid 2 to smithi07134777-42 2024-09-11T01:34:28.389 INFO:tasks.rados.rados.0.smithi071.stdout:784: finishing write tid 3 to smithi07134777-42 2024-09-11T01:34:28.389 INFO:tasks.rados.rados.0.smithi071.stdout:784: finishing write tid 4 to smithi07134777-42 2024-09-11T01:34:28.389 INFO:tasks.rados.rados.0.smithi071.stdout:784: finishing write tid 5 to smithi07134777-42 2024-09-11T01:34:28.389 INFO:tasks.rados.rados.0.smithi071.stdout:784: oid 42 updating version 0 to 140 2024-09-11T01:34:28.389 INFO:tasks.rados.rados.0.smithi071.stdout:784: oid 42 updating version 140 to 142 2024-09-11T01:34:28.390 INFO:tasks.rados.rados.0.smithi071.stdout:784: oid 42 version 142 is already newer than 139 2024-09-11T01:34:28.390 INFO:tasks.rados.rados.0.smithi071.stdout:784: oid 42 version 142 is already newer than 141 2024-09-11T01:34:28.390 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 142 (ObjNum 294 snap 80 seq_num 294) dirty exists 2024-09-11T01:34:28.390 INFO:tasks.rados.rados.0.smithi071.stdout:784: left oid 42 (ObjNum 294 snap 80 seq_num 294) 2024-09-11T01:34:28.390 INFO:tasks.rados.rados.0.smithi071.stdout:785: expect (ObjNum 232 snap 57 seq_num 232) 2024-09-11T01:34:28.390 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:34:28.469 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 888506 2024-09-11T01:34:28.550 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1856151 2024-09-11T01:34:28.595 INFO:tasks.rados.rados.0.smithi071.stdout:786: finishing write tid 1 to smithi07134777-37 2024-09-11T01:34:28.595 INFO:tasks.rados.rados.0.smithi071.stdout:786: finishing write tid 2 to smithi07134777-37 2024-09-11T01:34:28.595 INFO:tasks.rados.rados.0.smithi071.stdout:787: expect (ObjNum 287 snap 74 seq_num 287) 2024-09-11T01:34:28.600 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 878377 2024-09-11T01:34:28.665 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2192386 2024-09-11T01:34:28.714 INFO:tasks.rados.rados.0.smithi071.stdout:786: finishing write tid 3 to smithi07134777-37 2024-09-11T01:34:28.714 INFO:tasks.rados.rados.0.smithi071.stdout:786: finishing write tid 4 to smithi07134777-37 2024-09-11T01:34:28.714 INFO:tasks.rados.rados.0.smithi071.stdout:786: finishing write tid 5 to smithi07134777-37 2024-09-11T01:34:28.714 INFO:tasks.rados.rados.0.smithi071.stdout:786: oid 37 updating version 0 to 204 2024-09-11T01:34:28.714 INFO:tasks.rados.rados.0.smithi071.stdout:786: oid 37 version 204 is already newer than 203 2024-09-11T01:34:28.714 INFO:tasks.rados.rados.0.smithi071.stdout:786: oid 37 version 204 is already newer than 202 2024-09-11T01:34:28.715 INFO:tasks.rados.rados.0.smithi071.stdout:786: oid 37 updating version 204 to 205 2024-09-11T01:34:28.715 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 205 (ObjNum 295 snap 80 seq_num 295) dirty exists 2024-09-11T01:34:28.715 INFO:tasks.rados.rados.0.smithi071.stdout:786: left oid 37 (ObjNum 295 snap 80 seq_num 295) 2024-09-11T01:34:28.715 INFO:tasks.rados.rados.0.smithi071.stdout:788: finishing write tid 1 to smithi07134777-17 2024-09-11T01:34:28.715 INFO:tasks.rados.rados.0.smithi071.stdout:788: finishing write tid 2 to smithi07134777-17 2024-09-11T01:34:28.715 INFO:tasks.rados.rados.0.smithi071.stdout:791: seq_num 298 ranges {608486=561969,1871634=634092,2950640=57846} 2024-09-11T01:34:28.720 INFO:tasks.rados.rados.0.smithi071.stdout:791: writing smithi07134777-10 from 608486 to 1170455 tid 1 2024-09-11T01:34:28.727 INFO:tasks.rados.rados.0.smithi071.stdout:791: writing smithi07134777-10 from 1871634 to 2505726 tid 2 2024-09-11T01:34:28.731 INFO:tasks.rados.rados.0.smithi071.stdout:791: writing smithi07134777-10 from 2950640 to 3008486 tid 3 2024-09-11T01:34:28.731 INFO:tasks.rados.rados.0.smithi071.stdout:783: done (8 left) 2024-09-11T01:34:28.731 INFO:tasks.rados.rados.0.smithi071.stdout:784: done (7 left) 2024-09-11T01:34:28.731 INFO:tasks.rados.rados.0.smithi071.stdout:785: done (6 left) 2024-09-11T01:34:28.731 INFO:tasks.rados.rados.0.smithi071.stdout:786: done (5 left) 2024-09-11T01:34:28.731 INFO:tasks.rados.rados.0.smithi071.stdout:787: done (4 left) 2024-09-11T01:34:28.731 INFO:tasks.rados.rados.0.smithi071.stdout:792: snap_create 2024-09-11T01:34:28.731 INFO:tasks.rados.rados.0.smithi071.stdout:788: finishing write tid 3 to smithi07134777-17 2024-09-11T01:34:28.731 INFO:tasks.rados.rados.0.smithi071.stdout:788: finishing write tid 4 to smithi07134777-17 2024-09-11T01:34:28.732 INFO:tasks.rados.rados.0.smithi071.stdout:788: oid 17 updating version 0 to 300 2024-09-11T01:34:28.732 INFO:tasks.rados.rados.0.smithi071.stdout:788: oid 17 updating version 300 to 302 2024-09-11T01:34:28.732 INFO:tasks.rados.rados.0.smithi071.stdout:788: oid 17 version 302 is already newer than 301 2024-09-11T01:34:28.732 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 302 (ObjNum 296 snap 80 seq_num 296) dirty exists 2024-09-11T01:34:28.732 INFO:tasks.rados.rados.0.smithi071.stdout:788: left oid 17 (ObjNum 296 snap 80 seq_num 296) 2024-09-11T01:34:28.732 INFO:tasks.rados.rados.0.smithi071.stdout:789: finishing write tid 1 to smithi07134777-45 2024-09-11T01:34:28.732 INFO:tasks.rados.rados.0.smithi071.stdout:789: finishing write tid 2 to smithi07134777-45 2024-09-11T01:34:28.732 INFO:tasks.rados.rados.0.smithi071.stdout:789: finishing write tid 3 to smithi07134777-45 2024-09-11T01:34:28.732 INFO:tasks.rados.rados.0.smithi071.stdout:790: expect (ObjNum 251 snap 63 seq_num 251) 2024-09-11T01:34:28.908 INFO:tasks.rados.rados.0.smithi071.stdout:789: finishing write tid 4 to smithi07134777-45 2024-09-11T01:34:28.908 INFO:tasks.rados.rados.0.smithi071.stdout:789: oid 45 updating version 0 to 261 2024-09-11T01:34:28.908 INFO:tasks.rados.rados.0.smithi071.stdout:789: oid 45 version 261 is already newer than 259 2024-09-11T01:34:28.908 INFO:tasks.rados.rados.0.smithi071.stdout:789: oid 45 version 261 is already newer than 260 2024-09-11T01:34:28.908 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 261 (ObjNum 297 snap 80 seq_num 297) dirty exists 2024-09-11T01:34:28.908 INFO:tasks.rados.rados.0.smithi071.stdout:789: left oid 45 (ObjNum 297 snap 80 seq_num 297) 2024-09-11T01:34:28.909 INFO:tasks.rados.rados.0.smithi071.stdout:791: finishing write tid 1 to smithi07134777-10 2024-09-11T01:34:28.909 INFO:tasks.rados.rados.0.smithi071.stdout:791: finishing write tid 2 to smithi07134777-10 2024-09-11T01:34:28.909 INFO:tasks.rados.rados.0.smithi071.stdout:791: finishing write tid 3 to smithi07134777-10 2024-09-11T01:34:28.909 INFO:tasks.rados.rados.0.smithi071.stdout:791: finishing write tid 4 to smithi07134777-10 2024-09-11T01:34:28.909 INFO:tasks.rados.rados.0.smithi071.stdout:791: finishing write tid 5 to smithi07134777-10 2024-09-11T01:34:28.909 INFO:tasks.rados.rados.0.smithi071.stdout:791: oid 10 updating version 0 to 219 2024-09-11T01:34:28.909 INFO:tasks.rados.rados.0.smithi071.stdout:791: oid 10 updating version 219 to 221 2024-09-11T01:34:28.909 INFO:tasks.rados.rados.0.smithi071.stdout:791: oid 10 updating version 221 to 222 2024-09-11T01:34:28.909 INFO:tasks.rados.rados.0.smithi071.stdout:791: oid 10 version 222 is already newer than 220 2024-09-11T01:34:28.909 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 222 (ObjNum 298 snap 80 seq_num 298) dirty exists 2024-09-11T01:34:28.909 INFO:tasks.rados.rados.0.smithi071.stdout:791: left oid 10 (ObjNum 298 snap 80 seq_num 298) 2024-09-11T01:34:29.331 INFO:tasks.rados.rados.0.smithi071.stdout:788: done (4 left) 2024-09-11T01:34:29.331 INFO:tasks.rados.rados.0.smithi071.stdout:789: done (3 left) 2024-09-11T01:34:29.331 INFO:tasks.rados.rados.0.smithi071.stdout:790: done (2 left) 2024-09-11T01:34:29.331 INFO:tasks.rados.rados.0.smithi071.stdout:791: done (1 left) 2024-09-11T01:34:29.331 INFO:tasks.rados.rados.0.smithi071.stdout:792: done (0 left) 2024-09-11T01:34:29.331 INFO:tasks.rados.rados.0.smithi071.stdout:793: rollback oid 20 current snap is 81 2024-09-11T01:34:29.332 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 20 to 78 2024-09-11T01:34:29.332 INFO:tasks.rados.rados.0.smithi071.stdout:794: rollback oid 23 current snap is 81 2024-09-11T01:34:29.332 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 23 to 80 2024-09-11T01:34:29.332 INFO:tasks.rados.rados.0.smithi071.stdout:795: copy_from oid 17 from oid 25 current snap is 81 2024-09-11T01:34:29.332 INFO:tasks.rados.rados.0.smithi071.stdout:796: snap_create 2024-09-11T01:34:29.339 INFO:tasks.rados.rados.0.smithi071.stdout:793: finishing rollback tid 0 to smithi07134777-20 2024-09-11T01:34:29.339 INFO:tasks.rados.rados.0.smithi071.stdout:793: finishing rollback tid 1 to smithi07134777-20 2024-09-11T01:34:29.340 INFO:tasks.rados.rados.0.smithi071.stdout:793: finishing rollback tid 2 to smithi07134777-20 2024-09-11T01:34:29.340 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 237 (ObjNum 114 snap 27 seq_num 114) dirty exists 2024-09-11T01:34:29.340 INFO:tasks.rados.rados.0.smithi071.stdout:794: finishing rollback tid 0 to smithi07134777-23 2024-09-11T01:34:29.343 INFO:tasks.rados.rados.0.smithi071.stdout:794: finishing rollback tid 1 to smithi07134777-23 2024-09-11T01:34:29.343 INFO:tasks.rados.rados.0.smithi071.stdout:794: finishing rollback tid 2 to smithi07134777-23 2024-09-11T01:34:29.343 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 240 (ObjNum 277 snap 73 seq_num 277) dirty exists 2024-09-11T01:34:29.344 INFO:tasks.rados.rados.0.smithi071.stdout:795: finishing copy_from racing read to smithi07134777-17 2024-09-11T01:34:29.348 INFO:tasks.rados.rados.0.smithi071.stdout:795: finishing copy_from to smithi07134777-17 2024-09-11T01:34:29.348 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 79 (ObjNum 250 snap 63 seq_num 250) dirty exists 2024-09-11T01:34:30.036 INFO:tasks.rados.rados.0.smithi071.stdout:793: done (3 left) 2024-09-11T01:34:30.036 INFO:tasks.rados.rados.0.smithi071.stdout:794: done (2 left) 2024-09-11T01:34:30.036 INFO:tasks.rados.rados.0.smithi071.stdout:795: done (1 left) 2024-09-11T01:34:30.036 INFO:tasks.rados.rados.0.smithi071.stdout:796: done (0 left) 2024-09-11T01:34:30.036 INFO:tasks.rados.rados.0.smithi071.stdout:797: read oid 49 snap -1 2024-09-11T01:34:30.036 INFO:tasks.rados.rados.0.smithi071.stdout:797: expect (ObjNum 200 snap 46 seq_num 200) 2024-09-11T01:34:30.037 INFO:tasks.rados.rados.0.smithi071.stdout:798: write (excl) oid 18 current snap is 82 2024-09-11T01:34:30.037 INFO:tasks.rados.rados.0.smithi071.stdout:798: seq_num 299 ranges {625293=637218,1813763=494103,2978063=47230} 2024-09-11T01:34:30.043 INFO:tasks.rados.rados.0.smithi071.stdout:798: writing smithi07134777-18 from 625293 to 1262511 tid 1 2024-09-11T01:34:30.048 INFO:tasks.rados.rados.0.smithi071.stdout:798: writing smithi07134777-18 from 1813763 to 2307866 tid 2 2024-09-11T01:34:30.053 INFO:tasks.rados.rados.0.smithi071.stdout:798: writing smithi07134777-18 from 2978063 to 3025293 tid 3 2024-09-11T01:34:30.053 INFO:tasks.rados.rados.0.smithi071.stdout:799: read oid 39 snap 80 2024-09-11T01:34:30.053 INFO:tasks.rados.rados.0.smithi071.stdout:799: expect (ObjNum 229 snap 55 seq_num 229) 2024-09-11T01:34:30.053 INFO:tasks.rados.rados.0.smithi071.stdout:800: delete oid 43 current snap is 82 2024-09-11T01:34:30.053 INFO:tasks.rados.rados.0.smithi071.stdout:797: expect (ObjNum 200 snap 46 seq_num 200) 2024-09-11T01:34:30.111 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 331588 2024-09-11T01:34:30.242 INFO:tasks.rados.rados.0.smithi071.stdout:798: finishing write tid 1 to smithi07134777-18 2024-09-11T01:34:30.243 INFO:tasks.rados.rados.0.smithi071.stdout:798: finishing write tid 2 to smithi07134777-18 2024-09-11T01:34:30.243 INFO:tasks.rados.rados.0.smithi071.stdout:798: finishing write tid 3 to smithi07134777-18 2024-09-11T01:34:30.243 INFO:tasks.rados.rados.0.smithi071.stdout:798: finishing write tid 4 to smithi07134777-18 2024-09-11T01:34:30.243 INFO:tasks.rados.rados.0.smithi071.stdout:798: finishing write tid 5 to smithi07134777-18 2024-09-11T01:34:30.243 INFO:tasks.rados.rados.0.smithi071.stdout:798: oid 18 updating version 0 to 273 2024-09-11T01:34:30.243 INFO:tasks.rados.rados.0.smithi071.stdout:798: oid 18 updating version 273 to 274 2024-09-11T01:34:30.243 INFO:tasks.rados.rados.0.smithi071.stdout:798: oid 18 version 274 is already newer than 271 2024-09-11T01:34:30.243 INFO:tasks.rados.rados.0.smithi071.stdout:798: oid 18 version 274 is already newer than 272 2024-09-11T01:34:30.243 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 274 (ObjNum 299 snap 82 seq_num 299) dirty exists 2024-09-11T01:34:30.243 INFO:tasks.rados.rados.0.smithi071.stdout:798: left oid 18 (ObjNum 299 snap 82 seq_num 299) 2024-09-11T01:34:30.243 INFO:tasks.rados.rados.0.smithi071.stdout:799: expect (ObjNum 229 snap 55 seq_num 229) 2024-09-11T01:34:30.255 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2417045 2024-09-11T01:34:30.324 INFO:tasks.rados.rados.0.smithi071.stdout:797: done (3 left) 2024-09-11T01:34:30.324 INFO:tasks.rados.rados.0.smithi071.stdout:798: done (2 left) 2024-09-11T01:34:30.325 INFO:tasks.rados.rados.0.smithi071.stdout:799: done (1 left) 2024-09-11T01:34:30.325 INFO:tasks.rados.rados.0.smithi071.stdout:800: done (0 left) 2024-09-11T01:34:30.325 INFO:tasks.rados.rados.0.smithi071.stdout:801: copy_from oid 10 from oid 30 current snap is 82 2024-09-11T01:34:30.325 INFO:tasks.rados.rados.0.smithi071.stdout:802: snap_remove snap 81 2024-09-11T01:34:31.041 INFO:tasks.rados.rados.0.smithi071.stdout:802: done (1 left) 2024-09-11T01:34:31.041 INFO:tasks.rados.rados.0.smithi071.stdout:803: delete oid 43 current snap is 82 2024-09-11T01:34:31.041 INFO:tasks.rados.rados.0.smithi071.stdout:801: finishing copy_from racing read to smithi07134777-10 2024-09-11T01:34:31.042 INFO:tasks.rados.rados.0.smithi071.stdout:801: finishing copy_from to smithi07134777-10 2024-09-11T01:34:31.042 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 231 (ObjNum 217 snap 52 seq_num 217) dirty exists 2024-09-11T01:34:31.044 INFO:tasks.rados.rados.0.smithi071.stdout:801: done (1 left) 2024-09-11T01:34:31.044 INFO:tasks.rados.rados.0.smithi071.stdout:803: done (0 left) 2024-09-11T01:34:31.044 INFO:tasks.rados.rados.0.smithi071.stdout:804: write (excl) oid 32 current snap is 82 2024-09-11T01:34:31.044 INFO:tasks.rados.rados.0.smithi071.stdout:804: seq_num 302 ranges {675714=462965,1589209=641430,2943979=131735} 2024-09-11T01:34:31.050 INFO:tasks.rados.rados.0.smithi071.stdout:804: writing smithi07134777-32 from 675714 to 1138679 tid 1 2024-09-11T01:34:31.056 INFO:tasks.rados.rados.0.smithi071.stdout:804: writing smithi07134777-32 from 1589209 to 2230639 tid 2 2024-09-11T01:34:31.060 INFO:tasks.rados.rados.0.smithi071.stdout:804: writing smithi07134777-32 from 2943979 to 3075714 tid 3 2024-09-11T01:34:31.061 INFO:tasks.rados.rados.0.smithi071.stdout:805: write oid 19 current snap is 82 2024-09-11T01:34:31.061 INFO:tasks.rados.rados.0.smithi071.stdout:805: seq_num 303 ranges {692521=538214,1931338=785088} 2024-09-11T01:34:31.067 INFO:tasks.rados.rados.0.smithi071.stdout:805: writing smithi07134777-19 from 692521 to 1230735 tid 1 2024-09-11T01:34:31.076 INFO:tasks.rados.rados.0.smithi071.stdout:805: writing smithi07134777-19 from 1931338 to 2716426 tid 2 2024-09-11T01:34:31.076 INFO:tasks.rados.rados.0.smithi071.stdout:806: delete oid 42 current snap is 82 2024-09-11T01:34:31.076 INFO:tasks.rados.rados.0.smithi071.stdout:804: finishing write tid 1 to smithi07134777-32 2024-09-11T01:34:31.076 INFO:tasks.rados.rados.0.smithi071.stdout:804: finishing write tid 2 to smithi07134777-32 2024-09-11T01:34:31.076 INFO:tasks.rados.rados.0.smithi071.stdout:804: finishing write tid 3 to smithi07134777-32 2024-09-11T01:34:31.077 INFO:tasks.rados.rados.0.smithi071.stdout:804: finishing write tid 4 to smithi07134777-32 2024-09-11T01:34:31.077 INFO:tasks.rados.rados.0.smithi071.stdout:804: finishing write tid 5 to smithi07134777-32 2024-09-11T01:34:31.077 INFO:tasks.rados.rados.0.smithi071.stdout:804: oid 32 updating version 0 to 208 2024-09-11T01:34:31.077 INFO:tasks.rados.rados.0.smithi071.stdout:804: oid 32 version 208 is already newer than 206 2024-09-11T01:34:31.077 INFO:tasks.rados.rados.0.smithi071.stdout:804: oid 32 updating version 208 to 209 2024-09-11T01:34:31.077 INFO:tasks.rados.rados.0.smithi071.stdout:804: oid 32 version 209 is already newer than 207 2024-09-11T01:34:31.078 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 209 (ObjNum 302 snap 82 seq_num 302) dirty exists 2024-09-11T01:34:31.078 INFO:tasks.rados.rados.0.smithi071.stdout:804: left oid 32 (ObjNum 302 snap 82 seq_num 302) 2024-09-11T01:34:31.078 INFO:tasks.rados.rados.0.smithi071.stdout:805: finishing write tid 1 to smithi07134777-19 2024-09-11T01:34:31.080 INFO:tasks.rados.rados.0.smithi071.stdout:805: finishing write tid 2 to smithi07134777-19 2024-09-11T01:34:31.080 INFO:tasks.rados.rados.0.smithi071.stdout:805: finishing write tid 3 to smithi07134777-19 2024-09-11T01:34:31.081 INFO:tasks.rados.rados.0.smithi071.stdout:805: finishing write tid 4 to smithi07134777-19 2024-09-11T01:34:31.081 INFO:tasks.rados.rados.0.smithi071.stdout:805: oid 19 updating version 0 to 132 2024-09-11T01:34:31.081 INFO:tasks.rados.rados.0.smithi071.stdout:805: oid 19 version 132 is already newer than 130 2024-09-11T01:34:31.082 INFO:tasks.rados.rados.0.smithi071.stdout:805: oid 19 version 132 is already newer than 131 2024-09-11T01:34:31.082 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 132 (ObjNum 303 snap 82 seq_num 303) dirty exists 2024-09-11T01:34:31.082 INFO:tasks.rados.rados.0.smithi071.stdout:805: left oid 19 (ObjNum 303 snap 82 seq_num 303) 2024-09-11T01:34:31.082 INFO:tasks.rados.rados.0.smithi071.stdout:804: done (2 left) 2024-09-11T01:34:31.082 INFO:tasks.rados.rados.0.smithi071.stdout:805: done (1 left) 2024-09-11T01:34:31.082 INFO:tasks.rados.rados.0.smithi071.stdout:806: done (0 left) 2024-09-11T01:34:31.082 INFO:tasks.rados.rados.0.smithi071.stdout:807: delete oid 12 current snap is 82 2024-09-11T01:34:31.084 INFO:tasks.rados.rados.0.smithi071.stdout:807: done (0 left) 2024-09-11T01:34:31.084 INFO:tasks.rados.rados.0.smithi071.stdout:808: delete oid 32 current snap is 82 2024-09-11T01:34:31.086 INFO:tasks.rados.rados.0.smithi071.stdout:808: done (0 left) 2024-09-11T01:34:31.086 INFO:tasks.rados.rados.0.smithi071.stdout:809: read oid 46 snap 78 2024-09-11T01:34:31.086 INFO:tasks.rados.rados.0.smithi071.stdout:809: expect (ObjNum 240 snap 58 seq_num 240) 2024-09-11T01:34:31.086 INFO:tasks.rados.rados.0.smithi071.stdout:810: snap_create 2024-09-11T01:34:31.093 INFO:tasks.rados.rados.0.smithi071.stdout:809: expect (ObjNum 240 snap 58 seq_num 240) 2024-09-11T01:34:31.093 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:34:32.043 INFO:tasks.rados.rados.0.smithi071.stdout:809: done (1 left) 2024-09-11T01:34:32.043 INFO:tasks.rados.rados.0.smithi071.stdout:810: done (0 left) 2024-09-11T01:34:32.043 INFO:tasks.rados.rados.0.smithi071.stdout:811: write oid 18 current snap is 83 2024-09-11T01:34:32.043 INFO:tasks.rados.rados.0.smithi071.stdout:811: seq_num 307 ranges {759749=555563,1765266=792426,2964741=195008} 2024-09-11T01:34:32.049 INFO:tasks.rados.rados.0.smithi071.stdout:811: writing smithi07134777-18 from 759749 to 1315312 tid 1 2024-09-11T01:34:32.056 INFO:tasks.rados.rados.0.smithi071.stdout:811: writing smithi07134777-18 from 1765266 to 2557692 tid 2 2024-09-11T01:34:32.059 INFO:tasks.rados.rados.0.smithi071.stdout:811: writing smithi07134777-18 from 2964741 to 3159749 tid 3 2024-09-11T01:34:32.059 INFO:tasks.rados.rados.0.smithi071.stdout:812: delete oid 30 current snap is 83 2024-09-11T01:34:32.060 INFO:tasks.rados.rados.0.smithi071.stdout:811: finishing write tid 1 to smithi07134777-18 2024-09-11T01:34:32.065 INFO:tasks.rados.rados.0.smithi071.stdout:811: finishing write tid 2 to smithi07134777-18 2024-09-11T01:34:32.067 INFO:tasks.rados.rados.0.smithi071.stdout:811: finishing write tid 3 to smithi07134777-18 2024-09-11T01:34:32.067 INFO:tasks.rados.rados.0.smithi071.stdout:811: finishing write tid 4 to smithi07134777-18 2024-09-11T01:34:32.067 INFO:tasks.rados.rados.0.smithi071.stdout:811: finishing write tid 5 to smithi07134777-18 2024-09-11T01:34:32.068 INFO:tasks.rados.rados.0.smithi071.stdout:811: oid 18 updating version 0 to 277 2024-09-11T01:34:32.068 INFO:tasks.rados.rados.0.smithi071.stdout:811: oid 18 updating version 277 to 279 2024-09-11T01:34:32.068 INFO:tasks.rados.rados.0.smithi071.stdout:811: oid 18 version 279 is already newer than 276 2024-09-11T01:34:32.068 INFO:tasks.rados.rados.0.smithi071.stdout:811: oid 18 version 279 is already newer than 278 2024-09-11T01:34:32.068 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 279 (ObjNum 307 snap 83 seq_num 307) dirty exists 2024-09-11T01:34:32.068 INFO:tasks.rados.rados.0.smithi071.stdout:811: left oid 18 (ObjNum 307 snap 83 seq_num 307) 2024-09-11T01:34:32.068 INFO:tasks.rados.rados.0.smithi071.stdout:811: done (1 left) 2024-09-11T01:34:32.068 INFO:tasks.rados.rados.0.smithi071.stdout:812: done (0 left) 2024-09-11T01:34:32.068 INFO:tasks.rados.rados.0.smithi071.stdout:813: snap_create 2024-09-11T01:34:33.055 INFO:tasks.rados.rados.0.smithi071.stdout:813: done (0 left) 2024-09-11T01:34:33.055 INFO:tasks.rados.rados.0.smithi071.stdout:814: snap_create 2024-09-11T01:34:34.057 INFO:tasks.rados.rados.0.smithi071.stdout:814: done (0 left) 2024-09-11T01:34:34.057 INFO:tasks.rados.rados.0.smithi071.stdout:815: snap_remove snap 80 2024-09-11T01:34:35.039 INFO:tasks.rados.rados.0.smithi071.stdout:815: done (0 left) 2024-09-11T01:34:35.040 INFO:tasks.rados.rados.0.smithi071.stdout:816: snap_create 2024-09-11T01:34:36.038 INFO:tasks.rados.rados.0.smithi071.stdout:816: done (0 left) 2024-09-11T01:34:36.039 INFO:tasks.rados.rados.0.smithi071.stdout:817: delete oid 19 current snap is 86 2024-09-11T01:34:36.045 INFO:tasks.rados.rados.0.smithi071.stdout:817: done (0 left) 2024-09-11T01:34:36.045 INFO:tasks.rados.rados.0.smithi071.stdout:818: rollback oid 34 current snap is 86 2024-09-11T01:34:36.045 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 34 to 83 2024-09-11T01:34:36.045 INFO:tasks.rados.rados.0.smithi071.stdout:819: write oid 26 current snap is 86 2024-09-11T01:34:36.045 INFO:tasks.rados.rados.0.smithi071.stdout:819: seq_num 310 ranges {410170=781310,1940712=656106,3163363=46807} 2024-09-11T01:34:36.051 INFO:tasks.rados.rados.0.smithi071.stdout:819: writing smithi07134777-26 from 410170 to 1191480 tid 1 2024-09-11T01:34:36.058 INFO:tasks.rados.rados.0.smithi071.stdout:819: writing smithi07134777-26 from 1940712 to 2596818 tid 2 2024-09-11T01:34:36.062 INFO:tasks.rados.rados.0.smithi071.stdout:819: writing smithi07134777-26 from 3163363 to 3210170 tid 3 2024-09-11T01:34:36.062 INFO:tasks.rados.rados.0.smithi071.stdout:820: snap_create 2024-09-11T01:34:36.062 INFO:tasks.rados.rados.0.smithi071.stdout:818: finishing rollback tid 0 to smithi07134777-34 2024-09-11T01:34:36.062 INFO:tasks.rados.rados.0.smithi071.stdout:818: finishing rollback tid 1 to smithi07134777-34 2024-09-11T01:34:36.063 INFO:tasks.rados.rados.0.smithi071.stdout:818: finishing rollback tid 2 to smithi07134777-34 2024-09-11T01:34:36.063 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 274 (ObjNum 292 snap 79 seq_num 292) dirty exists 2024-09-11T01:34:36.064 INFO:tasks.rados.rados.0.smithi071.stdout:819: finishing write tid 1 to smithi07134777-26 2024-09-11T01:34:36.064 INFO:tasks.rados.rados.0.smithi071.stdout:819: finishing write tid 2 to smithi07134777-26 2024-09-11T01:34:36.065 INFO:tasks.rados.rados.0.smithi071.stdout:819: finishing write tid 3 to smithi07134777-26 2024-09-11T01:34:36.065 INFO:tasks.rados.rados.0.smithi071.stdout:819: finishing write tid 4 to smithi07134777-26 2024-09-11T01:34:36.065 INFO:tasks.rados.rados.0.smithi071.stdout:819: finishing write tid 5 to smithi07134777-26 2024-09-11T01:34:36.065 INFO:tasks.rados.rados.0.smithi071.stdout:819: oid 26 updating version 0 to 277 2024-09-11T01:34:36.065 INFO:tasks.rados.rados.0.smithi071.stdout:819: oid 26 updating version 277 to 278 2024-09-11T01:34:36.065 INFO:tasks.rados.rados.0.smithi071.stdout:819: oid 26 updating version 278 to 279 2024-09-11T01:34:36.065 INFO:tasks.rados.rados.0.smithi071.stdout:819: oid 26 version 279 is already newer than 276 2024-09-11T01:34:36.065 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 279 (ObjNum 310 snap 86 seq_num 310) dirty exists 2024-09-11T01:34:36.066 INFO:tasks.rados.rados.0.smithi071.stdout:819: left oid 26 (ObjNum 310 snap 86 seq_num 310) 2024-09-11T01:34:37.040 INFO:tasks.rados.rados.0.smithi071.stdout:818: done (2 left) 2024-09-11T01:34:37.040 INFO:tasks.rados.rados.0.smithi071.stdout:819: done (1 left) 2024-09-11T01:34:37.040 INFO:tasks.rados.rados.0.smithi071.stdout:820: done (0 left) 2024-09-11T01:34:37.040 INFO:tasks.rados.rados.0.smithi071.stdout:821: snap_remove snap 79 2024-09-11T01:34:38.042 INFO:tasks.rados.rados.0.smithi071.stdout:821: done (0 left) 2024-09-11T01:34:38.042 INFO:tasks.rados.rados.0.smithi071.stdout:822: read oid 17 snap 83 2024-09-11T01:34:38.042 INFO:tasks.rados.rados.0.smithi071.stdout:822: expect (ObjNum 250 snap 63 seq_num 250) 2024-09-11T01:34:38.042 INFO:tasks.rados.rados.0.smithi071.stdout:823: copy_from oid 23 from oid 44 current snap is 87 2024-09-11T01:34:38.042 INFO:tasks.rados.rados.0.smithi071.stdout:824: read oid 7 snap -1 2024-09-11T01:34:38.042 INFO:tasks.rados.rados.0.smithi071.stdout:824: expect (ObjNum 272 snap 72 seq_num 272) 2024-09-11T01:34:38.042 INFO:tasks.rados.rados.0.smithi071.stdout:825: read oid 46 snap -1 2024-09-11T01:34:38.042 INFO:tasks.rados.rados.0.smithi071.stdout:825: expect (ObjNum 240 snap 58 seq_num 240) 2024-09-11T01:34:38.042 INFO:tasks.rados.rados.0.smithi071.stdout:826: write (excl) oid 16 current snap is 87 2024-09-11T01:34:38.042 INFO:tasks.rados.rados.0.smithi071.stdout:826: seq_num 311 ranges {426977=456559,1482841=799764,2958080=268897} 2024-09-11T01:34:38.047 INFO:tasks.rados.rados.0.smithi071.stdout:826: writing smithi07134777-16 from 426977 to 883536 tid 1 2024-09-11T01:34:38.056 INFO:tasks.rados.rados.0.smithi071.stdout:826: writing smithi07134777-16 from 1482841 to 2282605 tid 2 2024-09-11T01:34:38.061 INFO:tasks.rados.rados.0.smithi071.stdout:826: writing smithi07134777-16 from 2958080 to 3226977 tid 3 2024-09-11T01:34:38.061 INFO:tasks.rados.rados.0.smithi071.stdout:827: write (excl) oid 35 current snap is 87 2024-09-11T01:34:38.061 INFO:tasks.rados.rados.0.smithi071.stdout:827: seq_num 312 ranges {443784=648161,1657676=659775,2818209=425575} 2024-09-11T01:34:38.067 INFO:tasks.rados.rados.0.smithi071.stdout:827: writing smithi07134777-35 from 443784 to 1091945 tid 1 2024-09-11T01:34:38.074 INFO:tasks.rados.rados.0.smithi071.stdout:827: writing smithi07134777-35 from 1657676 to 2317451 tid 2 2024-09-11T01:34:38.079 INFO:tasks.rados.rados.0.smithi071.stdout:827: writing smithi07134777-35 from 2818209 to 3243784 tid 3 2024-09-11T01:34:38.079 INFO:tasks.rados.rados.0.smithi071.stdout:828: snap_create 2024-09-11T01:34:38.079 INFO:tasks.rados.rados.0.smithi071.stdout:823: finishing copy_from to smithi07134777-23 2024-09-11T01:34:38.079 INFO:tasks.rados.rados.0.smithi071.stdout:823: got expected ENOENT (src dne) 2024-09-11T01:34:38.080 INFO:tasks.rados.rados.0.smithi071.stdout:823: finishing copy_from racing read to smithi07134777-23 2024-09-11T01:34:38.080 INFO:tasks.rados.rados.0.smithi071.stdout:825: expect (ObjNum 240 snap 58 seq_num 240) 2024-09-11T01:34:38.101 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2004633 2024-09-11T01:34:38.105 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 738073 2024-09-11T01:34:38.162 INFO:tasks.rados.rados.0.smithi071.stdout:824: expect (ObjNum 272 snap 72 seq_num 272) 2024-09-11T01:34:38.162 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:34:38.211 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 834748 2024-09-11T01:34:38.293 INFO:tasks.rados.rados.0.smithi071.stdout:822: expect (ObjNum 250 snap 63 seq_num 250) 2024-09-11T01:34:38.372 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:34:38.414 INFO:tasks.rados.rados.0.smithi071.stdout:826: finishing write tid 1 to smithi07134777-16 2024-09-11T01:34:38.415 INFO:tasks.rados.rados.0.smithi071.stdout:826: finishing write tid 2 to smithi07134777-16 2024-09-11T01:34:38.415 INFO:tasks.rados.rados.0.smithi071.stdout:826: finishing write tid 3 to smithi07134777-16 2024-09-11T01:34:38.415 INFO:tasks.rados.rados.0.smithi071.stdout:826: finishing write tid 4 to smithi07134777-16 2024-09-11T01:34:38.415 INFO:tasks.rados.rados.0.smithi071.stdout:826: finishing write tid 5 to smithi07134777-16 2024-09-11T01:34:38.415 INFO:tasks.rados.rados.0.smithi071.stdout:826: oid 16 updating version 0 to 268 2024-09-11T01:34:38.415 INFO:tasks.rados.rados.0.smithi071.stdout:826: oid 16 version 268 is already newer than 267 2024-09-11T01:34:38.415 INFO:tasks.rados.rados.0.smithi071.stdout:826: oid 16 version 268 is already newer than 266 2024-09-11T01:34:38.415 INFO:tasks.rados.rados.0.smithi071.stdout:826: oid 16 version 268 is already newer than 265 2024-09-11T01:34:38.415 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 268 (ObjNum 311 snap 87 seq_num 311) dirty exists 2024-09-11T01:34:38.415 INFO:tasks.rados.rados.0.smithi071.stdout:826: left oid 16 (ObjNum 311 snap 87 seq_num 311) 2024-09-11T01:34:38.415 INFO:tasks.rados.rados.0.smithi071.stdout:827: finishing write tid 1 to smithi07134777-35 2024-09-11T01:34:38.415 INFO:tasks.rados.rados.0.smithi071.stdout:827: finishing write tid 2 to smithi07134777-35 2024-09-11T01:34:38.415 INFO:tasks.rados.rados.0.smithi071.stdout:827: finishing write tid 3 to smithi07134777-35 2024-09-11T01:34:38.416 INFO:tasks.rados.rados.0.smithi071.stdout:827: finishing write tid 4 to smithi07134777-35 2024-09-11T01:34:38.416 INFO:tasks.rados.rados.0.smithi071.stdout:827: finishing write tid 5 to smithi07134777-35 2024-09-11T01:34:38.416 INFO:tasks.rados.rados.0.smithi071.stdout:827: oid 35 updating version 0 to 218 2024-09-11T01:34:38.416 INFO:tasks.rados.rados.0.smithi071.stdout:827: oid 35 version 218 is already newer than 215 2024-09-11T01:34:38.416 INFO:tasks.rados.rados.0.smithi071.stdout:827: oid 35 version 218 is already newer than 216 2024-09-11T01:34:38.416 INFO:tasks.rados.rados.0.smithi071.stdout:827: oid 35 version 218 is already newer than 217 2024-09-11T01:34:38.416 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 218 (ObjNum 312 snap 87 seq_num 312) dirty exists 2024-09-11T01:34:38.416 INFO:tasks.rados.rados.0.smithi071.stdout:827: left oid 35 (ObjNum 312 snap 87 seq_num 312) 2024-09-11T01:34:39.050 INFO:tasks.rados.rados.0.smithi071.stdout:822: done (6 left) 2024-09-11T01:34:39.050 INFO:tasks.rados.rados.0.smithi071.stdout:823: done (5 left) 2024-09-11T01:34:39.050 INFO:tasks.rados.rados.0.smithi071.stdout:824: done (4 left) 2024-09-11T01:34:39.050 INFO:tasks.rados.rados.0.smithi071.stdout:825: done (3 left) 2024-09-11T01:34:39.050 INFO:tasks.rados.rados.0.smithi071.stdout:826: done (2 left) 2024-09-11T01:34:39.050 INFO:tasks.rados.rados.0.smithi071.stdout:827: done (1 left) 2024-09-11T01:34:39.051 INFO:tasks.rados.rados.0.smithi071.stdout:828: done (0 left) 2024-09-11T01:34:39.051 INFO:tasks.rados.rados.0.smithi071.stdout:829: read oid 13 snap -1 2024-09-11T01:34:39.051 INFO:tasks.rados.rados.0.smithi071.stdout:829: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:34:39.051 INFO:tasks.rados.rados.0.smithi071.stdout:830: snap_create 2024-09-11T01:34:39.058 INFO:tasks.rados.rados.0.smithi071.stdout:829: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:34:39.062 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 817651 2024-09-11T01:34:39.081 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:34:39.109 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:34:40.038 INFO:tasks.rados.rados.0.smithi071.stdout:829: done (1 left) 2024-09-11T01:34:40.038 INFO:tasks.rados.rados.0.smithi071.stdout:830: done (0 left) 2024-09-11T01:34:40.038 INFO:tasks.rados.rados.0.smithi071.stdout:831: write (excl) oid 48 current snap is 89 2024-09-11T01:34:40.038 INFO:tasks.rados.rados.0.smithi071.stdout:831: seq_num 313 ranges {460591=723410,1716158=403433,2729279=440140} 2024-09-11T01:34:40.045 INFO:tasks.rados.rados.0.smithi071.stdout:831: writing smithi07134777-48 from 460591 to 1184001 tid 1 2024-09-11T01:34:40.050 INFO:tasks.rados.rados.0.smithi071.stdout:831: writing smithi07134777-48 from 1716158 to 2119591 tid 2 2024-09-11T01:34:40.057 INFO:tasks.rados.rados.0.smithi071.stdout:831: writing smithi07134777-48 from 2729279 to 3169419 tid 3 2024-09-11T01:34:40.058 INFO:tasks.rados.rados.0.smithi071.stdout:832: read oid 14 snap -1 2024-09-11T01:34:40.058 INFO:tasks.rados.rados.0.smithi071.stdout:832: expect deleted 2024-09-11T01:34:40.058 INFO:tasks.rados.rados.0.smithi071.stdout:833: read oid 49 snap -1 2024-09-11T01:34:40.058 INFO:tasks.rados.rados.0.smithi071.stdout:833: expect (ObjNum 200 snap 46 seq_num 200) 2024-09-11T01:34:40.058 INFO:tasks.rados.rados.0.smithi071.stdout:834: read oid 29 snap -1 2024-09-11T01:34:40.058 INFO:tasks.rados.rados.0.smithi071.stdout:834: expect (ObjNum 201 snap 46 seq_num 201) 2024-09-11T01:34:40.058 INFO:tasks.rados.rados.0.smithi071.stdout:835: snap_remove snap 78 2024-09-11T01:34:41.058 INFO:tasks.rados.rados.0.smithi071.stdout:835: done (4 left) 2024-09-11T01:34:41.058 INFO:tasks.rados.rados.0.smithi071.stdout:836: snap_create 2024-09-11T01:34:41.058 INFO:tasks.rados.rados.0.smithi071.stdout:831: finishing write tid 1 to smithi07134777-48 2024-09-11T01:34:41.058 INFO:tasks.rados.rados.0.smithi071.stdout:831: finishing write tid 2 to smithi07134777-48 2024-09-11T01:34:41.058 INFO:tasks.rados.rados.0.smithi071.stdout:831: finishing write tid 3 to smithi07134777-48 2024-09-11T01:34:41.058 INFO:tasks.rados.rados.0.smithi071.stdout:831: finishing write tid 4 to smithi07134777-48 2024-09-11T01:34:41.058 INFO:tasks.rados.rados.0.smithi071.stdout:831: finishing write tid 5 to smithi07134777-48 2024-09-11T01:34:41.059 INFO:tasks.rados.rados.0.smithi071.stdout:831: oid 48 updating version 0 to 377 2024-09-11T01:34:41.059 INFO:tasks.rados.rados.0.smithi071.stdout:831: oid 48 version 377 is already newer than 376 2024-09-11T01:34:41.059 INFO:tasks.rados.rados.0.smithi071.stdout:831: oid 48 updating version 377 to 378 2024-09-11T01:34:41.059 INFO:tasks.rados.rados.0.smithi071.stdout:831: oid 48 updating version 378 to 379 2024-09-11T01:34:41.059 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 379 (ObjNum 313 snap 89 seq_num 313) dirty exists 2024-09-11T01:34:41.059 INFO:tasks.rados.rados.0.smithi071.stdout:831: left oid 48 (ObjNum 313 snap 89 seq_num 313) 2024-09-11T01:34:41.059 INFO:tasks.rados.rados.0.smithi071.stdout:833: expect (ObjNum 200 snap 46 seq_num 200) 2024-09-11T01:34:41.135 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:34:41.207 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:34:41.278 INFO:tasks.rados.rados.0.smithi071.stdout:834: expect (ObjNum 201 snap 46 seq_num 201) 2024-09-11T01:34:41.278 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:34:41.317 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1343004 2024-09-11T01:34:41.343 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:34:41.814 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-11T01:34:41.814 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-11T01:34:41.815 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-11T01:34:41.819 INFO:tasks.ceph.mon.c:Started 2024-09-11T01:34:41.819 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-11T01:34:41.819 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-11T01:34:41.819 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T01:34:56.820 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-11T01:34:56.820 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-11T01:34:56.821 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:34:56.821 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:34:56.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.932+0000 7f96efeb1640 10 monclient: get_monmap_and_config 2024-09-11T01:34:56.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.932+0000 7f96efeb1640 10 monclient: build_initial_monmap 2024-09-11T01:34:56.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.932+0000 7f96efeb1640 10 monclient: monmap: 2024-09-11T01:34:56.934 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:34:56.934 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:34:56.934 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:34:56.932924+0000 2024-09-11T01:34:56.934 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:34:56.932924+0000 2024-09-11T01:34:56.934 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:34:56.934 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:34:56.934 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:34:56.935 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:34:56.935 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:34:56.935 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:34:56.935 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:34:56.935 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:34:56.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.932+0000 7f96efeb1640 1 Processor -- start 2024-09-11T01:34:56.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96efeb1640 1 -- start start 2024-09-11T01:34:56.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96efeb1640 10 monclient: init 2024-09-11T01:34:56.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96efeb1640 10 monclient: _reopen_session rank -1 2024-09-11T01:34:56.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96efeb1640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-11T01:34:56.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96efeb1640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f96e8138230 0x7f96e8138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:56.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96efeb1640 10 monclient(hunting): picked mon.noname-a con 0x7f96e8138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:34:56.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96efeb1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f96e809d030 addr v1:172.21.15.71:6789/0 2024-09-11T01:34:56.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96efeb1640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 0x7f96e80a5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:56.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96efeb1640 10 monclient(hunting): picked mon.noname-b con 0x7f96e809d460 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:34:56.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96efeb1640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96e80a6e30 con 0x7f96e809d030 2024-09-11T01:34:56.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96efeb1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:56.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96efeb1640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f96e809c9c0 con 0x7f96e8138230 2024-09-11T01:34:56.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96efeb1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:56.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96efeb1640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f96e809ba20 con 0x7f96e809d460 2024-09-11T01:34:56.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96efeb1640 10 monclient(hunting): _renew_subs 2024-09-11T01:34:56.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96efeb1640 10 monclient(hunting): authenticate will time out at 1103.375610s 2024-09-11T01:34:56.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96edc4d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f96e8138230 0x7f96e8138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:56.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96ee44e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 0x7f96e80a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:56.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96edc4d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f96e8138230 0x7f96e8138610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48800/0 (socket says 172.21.15.12:48800) 2024-09-11T01:34:56.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96edc4d640 1 -- 172.21.15.12:0/2758914264 learned_addr learned my addr 172.21.15.12:0/2758914264 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:34:56.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96edc4d640 10 monclient(hunting): get_auth_request con 0x7f96e8138230 auth_method 0 2024-09-11T01:34:56.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96edc4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:56.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96edc4d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:56.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96edc4d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:34:56.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96ee44e640 10 monclient(hunting): get_auth_request con 0x7f96e809d460 auth_method 0 2024-09-11T01:34:56.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96ee44e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:56.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96ee44e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:56.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96ee44e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:34:56.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96ee44e640 1 -- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 msgr2=0x7f96e80a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:34:56.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96ee44e640 1 -- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 msgr2=0x7f96e80a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:34:56.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96ee44e640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 0x7f96e80a5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:34:56.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96edc4d640 1 -- 172.21.15.12:0/2758914264 >> v2:172.21.15.12:3300/0 conn(0x7f96e8138230 msgr2=0x7f96e8138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:34:56.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96edc4d640 1 -- 172.21.15.12:0/2758914264 >> v2:172.21.15.12:3300/0 conn(0x7f96e8138230 msgr2=0x7f96e8138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:34:56.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96edc4d640 1 --2- 172.21.15.12:0/2758914264 >> v2:172.21.15.12:3300/0 conn(0x7f96e8138230 0x7f96e8138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:34:56.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96ee44e640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 0x7f96e80a5730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:34:56.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.933+0000 7f96edc4d640 1 --2- 172.21.15.12:0/2758914264 >> v2:172.21.15.12:3300/0 conn(0x7f96e8138230 0x7f96e8138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:34:56.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:56.936+0000 7f96ed44c640 1 -- 172.21.15.12:0/2758914264 >> v1:172.21.15.71:6789/0 conn(0x7f96e809d030 legacy=0x7f96e80a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:34:57.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.134+0000 7f96edc4d640 1 --2- 172.21.15.12:0/2758914264 >> v2:172.21.15.12:3300/0 conn(0x7f96e8138230 0x7f96e8138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:57.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.134+0000 7f96ee44e640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 0x7f96e80a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:57.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.134+0000 7f96edc4d640 10 monclient(hunting): get_auth_request con 0x7f96e8138230 auth_method 0 2024-09-11T01:34:57.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.134+0000 7f96edc4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:57.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.134+0000 7f96edc4d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:57.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.134+0000 7f96edc4d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:34:57.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.134+0000 7f96ee44e640 10 monclient(hunting): get_auth_request con 0x7f96e809d460 auth_method 0 2024-09-11T01:34:57.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.134+0000 7f96ee44e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:57.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.134+0000 7f96ee44e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:57.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.134+0000 7f96ee44e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:34:57.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.134+0000 7f96edc4d640 1 -- 172.21.15.12:0/2758914264 >> v2:172.21.15.12:3300/0 conn(0x7f96e8138230 msgr2=0x7f96e8138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:34:57.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.134+0000 7f96edc4d640 1 -- 172.21.15.12:0/2758914264 >> v2:172.21.15.12:3300/0 conn(0x7f96e8138230 msgr2=0x7f96e8138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:34:57.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.134+0000 7f96edc4d640 1 --2- 172.21.15.12:0/2758914264 >> v2:172.21.15.12:3300/0 conn(0x7f96e8138230 0x7f96e8138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:34:57.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.134+0000 7f96ee44e640 1 -- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 msgr2=0x7f96e80a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:34:57.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.134+0000 7f96ee44e640 1 -- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 msgr2=0x7f96e80a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:34:57.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.134+0000 7f96ee44e640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 0x7f96e80a5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:34:57.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.134+0000 7f96edc4d640 1 --2- 172.21.15.12:0/2758914264 >> v2:172.21.15.12:3300/0 conn(0x7f96e8138230 0x7f96e8138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:34:57.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.134+0000 7f96ee44e640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 0x7f96e80a5730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:34:57.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.535+0000 7f96edc4d640 1 --2- 172.21.15.12:0/2758914264 >> v2:172.21.15.12:3300/0 conn(0x7f96e8138230 0x7f96e8138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.535+0000 7f96edc4d640 10 monclient(hunting): get_auth_request con 0x7f96e8138230 auth_method 0 2024-09-11T01:34:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.535+0000 7f96edc4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.535+0000 7f96ee44e640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 0x7f96e80a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.535+0000 7f96edc4d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.535+0000 7f96edc4d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:34:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.535+0000 7f96ee44e640 10 monclient(hunting): get_auth_request con 0x7f96e809d460 auth_method 0 2024-09-11T01:34:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.535+0000 7f96ee44e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.535+0000 7f96ee44e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.535+0000 7f96ee44e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:34:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.535+0000 7f96edc4d640 1 -- 172.21.15.12:0/2758914264 >> v2:172.21.15.12:3300/0 conn(0x7f96e8138230 msgr2=0x7f96e8138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:34:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.535+0000 7f96edc4d640 1 -- 172.21.15.12:0/2758914264 >> v2:172.21.15.12:3300/0 conn(0x7f96e8138230 msgr2=0x7f96e8138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:34:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.535+0000 7f96edc4d640 1 --2- 172.21.15.12:0/2758914264 >> v2:172.21.15.12:3300/0 conn(0x7f96e8138230 0x7f96e8138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:34:57.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.535+0000 7f96ee44e640 1 -- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 msgr2=0x7f96e80a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:34:57.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.535+0000 7f96ee44e640 1 -- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 msgr2=0x7f96e80a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:34:57.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.535+0000 7f96ee44e640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 0x7f96e80a5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:34:57.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.535+0000 7f96edc4d640 1 --2- 172.21.15.12:0/2758914264 >> v2:172.21.15.12:3300/0 conn(0x7f96e8138230 0x7f96e8138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:34:57.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:57.535+0000 7f96ee44e640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 0x7f96e80a5730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:34:58.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:58.337+0000 7f96edc4d640 1 --2- 172.21.15.12:0/2758914264 >> v2:172.21.15.12:3300/0 conn(0x7f96e8138230 0x7f96e8138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:58.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:58.337+0000 7f96edc4d640 10 monclient(hunting): get_auth_request con 0x7f96e8138230 auth_method 0 2024-09-11T01:34:58.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:58.337+0000 7f96edc4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:58.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:58.337+0000 7f96edc4d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:58.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:58.337+0000 7f96edc4d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:34:58.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:58.337+0000 7f96ee44e640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 0x7f96e80a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:58.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:58.337+0000 7f96ee44e640 10 monclient(hunting): get_auth_request con 0x7f96e809d460 auth_method 0 2024-09-11T01:34:58.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:58.337+0000 7f96ee44e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:58.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:58.337+0000 7f96ee44e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:58.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:58.337+0000 7f96ee44e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:34:58.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:58.337+0000 7f96edc4d640 1 -- 172.21.15.12:0/2758914264 >> v2:172.21.15.12:3300/0 conn(0x7f96e8138230 msgr2=0x7f96e8138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:34:58.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:58.337+0000 7f96edc4d640 1 -- 172.21.15.12:0/2758914264 >> v2:172.21.15.12:3300/0 conn(0x7f96e8138230 msgr2=0x7f96e8138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:34:58.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:58.337+0000 7f96edc4d640 1 --2- 172.21.15.12:0/2758914264 >> v2:172.21.15.12:3300/0 conn(0x7f96e8138230 0x7f96e8138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:34:58.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:58.337+0000 7f96ee44e640 1 -- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 msgr2=0x7f96e80a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:34:58.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:58.337+0000 7f96ee44e640 1 -- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 msgr2=0x7f96e80a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:34:58.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:58.337+0000 7f96ee44e640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 0x7f96e80a5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:34:58.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:58.337+0000 7f96edc4d640 1 --2- 172.21.15.12:0/2758914264 >> v2:172.21.15.12:3300/0 conn(0x7f96e8138230 0x7f96e8138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:34:58.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:58.337+0000 7f96ee44e640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 0x7f96e80a5730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:34:59.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.934+0000 7f96df7fe640 10 monclient(hunting): tick 2024-09-11T01:34:59.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.934+0000 7f96df7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:34:59.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.934+0000 7f96df7fe640 1 monclient(hunting): continuing hunt 2024-09-11T01:34:59.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.934+0000 7f96df7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:34:59.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.934+0000 7f96df7fe640 1 -- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 msgr2=0x7f96e80a5730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:34:59.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.934+0000 7f96df7fe640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 0x7f96e80a5730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:59.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.934+0000 7f96df7fe640 1 -- 172.21.15.12:0/2758914264 >> v2:172.21.15.12:3300/0 conn(0x7f96e8138230 msgr2=0x7f96e8138610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:34:59.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.934+0000 7f96df7fe640 1 --2- 172.21.15.12:0/2758914264 >> v2:172.21.15.12:3300/0 conn(0x7f96e8138230 0x7f96e8138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:34:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.934+0000 7f96df7fe640 1 -- 172.21.15.12:0/2758914264 >> v1:172.21.15.71:6789/0 conn(0x7f96e809d030 legacy=0x7f96e80a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:34:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.934+0000 7f96df7fe640 10 monclient: _add_conns ranks=[4,0] 2024-09-11T01:34:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.934+0000 7f96df7fe640 1 --2- 172.21.15.12:0/2758914264 >> v2:172.21.15.71:3300/0 conn(0x7f96cc003740 0x7f96cc005ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:34:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.934+0000 7f96df7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f96cc003740 addr v2:172.21.15.71:3300/0 2024-09-11T01:34:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.934+0000 7f96df7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f96cc006290 addr v1:172.21.15.12:6789/0 2024-09-11T01:34:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.934+0000 7f96df7fe640 1 -- 172.21.15.12:0/2758914264 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96e80a66a0 con 0x7f96cc006290 2024-09-11T01:34:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.934+0000 7f96df7fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:34:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.934+0000 7f96df7fe640 1 -- 172.21.15.12:0/2758914264 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f96e809c9c0 con 0x7f96cc003740 2024-09-11T01:34:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.934+0000 7f96df7fe640 10 monclient(hunting): _renew_subs 2024-09-11T01:34:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.934+0000 7f96edc4d640 1 --2- 172.21.15.12:0/2758914264 >> v2:172.21.15.71:3300/0 conn(0x7f96cc003740 0x7f96cc005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:34:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.934+0000 7f96edc4d640 10 monclient(hunting): get_auth_request con 0x7f96cc003740 auth_method 0 2024-09-11T01:34:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.934+0000 7f96edc4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:34:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.934+0000 7f96edc4d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:34:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.934+0000 7f96edc4d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:34:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.934+0000 7f96edc4d640 1 -- 172.21.15.12:0/2758914264 >> v2:172.21.15.71:3300/0 conn(0x7f96cc003740 msgr2=0x7f96cc005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:34:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.934+0000 7f96edc4d640 1 -- 172.21.15.12:0/2758914264 >> v2:172.21.15.71:3300/0 conn(0x7f96cc003740 msgr2=0x7f96cc005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:34:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.934+0000 7f96edc4d640 1 --2- 172.21.15.12:0/2758914264 >> v2:172.21.15.71:3300/0 conn(0x7f96cc003740 0x7f96cc005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:34:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.934+0000 7f96edc4d640 1 --2- 172.21.15.12:0/2758914264 >> v2:172.21.15.71:3300/0 conn(0x7f96cc003740 0x7f96cc005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:34:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:34:59.936+0000 7f96ed44c640 1 -- 172.21.15.12:0/2758914264 >> v1:172.21.15.12:6789/0 conn(0x7f96cc006290 legacy=0x7f96cc008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:00.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:00.135+0000 7f96edc4d640 1 --2- 172.21.15.12:0/2758914264 >> v2:172.21.15.71:3300/0 conn(0x7f96cc003740 0x7f96cc005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:00.135+0000 7f96edc4d640 10 monclient(hunting): get_auth_request con 0x7f96cc003740 auth_method 0 2024-09-11T01:35:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:00.135+0000 7f96edc4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:00.135+0000 7f96edc4d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:00.135+0000 7f96edc4d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:35:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:00.135+0000 7f96edc4d640 1 -- 172.21.15.12:0/2758914264 >> v2:172.21.15.71:3300/0 conn(0x7f96cc003740 msgr2=0x7f96cc005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:35:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:00.135+0000 7f96edc4d640 1 -- 172.21.15.12:0/2758914264 >> v2:172.21.15.71:3300/0 conn(0x7f96cc003740 msgr2=0x7f96cc005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:35:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:00.135+0000 7f96edc4d640 1 --2- 172.21.15.12:0/2758914264 >> v2:172.21.15.71:3300/0 conn(0x7f96cc003740 0x7f96cc005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:35:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:00.135+0000 7f96edc4d640 1 --2- 172.21.15.12:0/2758914264 >> v2:172.21.15.71:3300/0 conn(0x7f96cc003740 0x7f96cc005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:35:00.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:00.536+0000 7f96edc4d640 1 --2- 172.21.15.12:0/2758914264 >> v2:172.21.15.71:3300/0 conn(0x7f96cc003740 0x7f96cc005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:00.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:00.536+0000 7f96edc4d640 10 monclient(hunting): get_auth_request con 0x7f96cc003740 auth_method 0 2024-09-11T01:35:00.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:00.536+0000 7f96edc4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:00.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:00.536+0000 7f96edc4d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:00.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:00.536+0000 7f96edc4d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:35:00.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:00.536+0000 7f96edc4d640 1 -- 172.21.15.12:0/2758914264 >> v2:172.21.15.71:3300/0 conn(0x7f96cc003740 msgr2=0x7f96cc005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:35:00.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:00.536+0000 7f96edc4d640 1 -- 172.21.15.12:0/2758914264 >> v2:172.21.15.71:3300/0 conn(0x7f96cc003740 msgr2=0x7f96cc005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:35:00.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:00.536+0000 7f96edc4d640 1 --2- 172.21.15.12:0/2758914264 >> v2:172.21.15.71:3300/0 conn(0x7f96cc003740 0x7f96cc005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:35:00.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:00.536+0000 7f96edc4d640 1 --2- 172.21.15.12:0/2758914264 >> v2:172.21.15.71:3300/0 conn(0x7f96cc003740 0x7f96cc005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:35:01.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.065+0000 7f96ecc4b640 1 -- 172.21.15.12:0/2758914264 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 297494210 0 0) 0x7f96e80a66a0 con 0x7f96cc006290 2024-09-11T01:35:01.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.065+0000 7f96ecc4b640 10 monclient(hunting): _negotiate 2024-09-11T01:35:01.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.065+0000 7f96ecc4b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:01.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.065+0000 7f96ecc4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:01.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.065+0000 7f96ecc4b640 10 monclient(hunting): my global_id is 54183 2024-09-11T01:35:01.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.065+0000 7f96ecc4b640 1 -- 172.21.15.12:0/2758914264 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96d4003520 con 0x7f96cc006290 2024-09-11T01:35:01.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 1 -- 172.21.15.12:0/2758914264 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2625669158 0 0) 0x7f96d4003520 con 0x7f96cc006290 2024-09-11T01:35:01.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 1 -- 172.21.15.12:0/2758914264 >> v2:172.21.15.71:3300/0 conn(0x7f96cc003740 msgr2=0x7f96cc005ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:35:01.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 1 --2- 172.21.15.12:0/2758914264 >> v2:172.21.15.71:3300/0 conn(0x7f96cc003740 0x7f96cc005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:01.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 10 monclient: _finish_hunting 0 2024-09-11T01:35:01.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:35:01.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:01.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:35:01.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 1 -- 172.21.15.12:0/2758914264 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96cc00b000 con 0x7f96cc006290 2024-09-11T01:35:01.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 10 monclient: _finish_auth 0 2024-09-11T01:35:01.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 10 monclient: _check_auth_tickets 2024-09-11T01:35:01.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:01.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 1 -- 172.21.15.12:0/2758914264 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f96d8001a20 con 0x7f96cc006290 2024-09-11T01:35:01.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:01.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:35:01.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 10 monclient: dump: 2024-09-11T01:35:01.072 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:01.072 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:01.073 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:01.073 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:01.073 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:01.073 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:01.073 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:01.073 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:01.073 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:01.073 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:01.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:35:01.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:01.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 1 -- 172.21.15.12:0/2758914264 >> v1:172.21.15.12:6789/0 conn(0x7f96cc006290 legacy=0x7f96cc008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:01.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:35:01.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ee44e640 1 -- 172.21.15.12:0/2758914264 reap_dead start 2024-09-11T01:35:01.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 0x7f96d4003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:01.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 10 monclient(hunting): picked mon.c con 0x7f96e809d460 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:35:01.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96d4004100 0x7f96d4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:01.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 10 monclient(hunting): picked mon.a con 0x7f96d4004100 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:35:01.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96d4004f40 0x7f96d400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:01.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 10 monclient(hunting): picked mon.b con 0x7f96d4004f40 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:35:01.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:01.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 1 -- 172.21.15.12:0/2758914264 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f96e809c9c0 con 0x7f96d4004100 2024-09-11T01:35:01.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:01.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 1 -- 172.21.15.12:0/2758914264 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f96d400bf00 con 0x7f96d4004f40 2024-09-11T01:35:01.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:01.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 1 -- 172.21.15.12:0/2758914264 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f96d400c080 con 0x7f96e809d460 2024-09-11T01:35:01.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:01.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ecc4b640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:35:01.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96edc4d640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 0x7f96d4003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:01.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ed44c640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96d4004100 0x7f96d4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:01.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96edc4d640 10 monclient(hunting): get_auth_request con 0x7f96e809d460 auth_method 0 2024-09-11T01:35:01.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96edc4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:01.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96edc4d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:01.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96edc4d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:35:01.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ed44c640 10 monclient(hunting): get_auth_request con 0x7f96d4004100 auth_method 0 2024-09-11T01:35:01.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ed44c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:01.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ed44c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:01.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.066+0000 7f96ed44c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:35:01.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96edc4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:01.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96edc4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:01.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96edc4d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:35:01.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96ed44c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:01.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96ed44c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:01.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96ed44c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:35:01.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96ee44e640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96d4004f40 0x7f96d400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:01.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96ee44e640 10 monclient(hunting): get_auth_request con 0x7f96d4004f40 auth_method 0 2024-09-11T01:35:01.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96ee44e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:01.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96ee44e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:01.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96ee44e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:35:01.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96edc4d640 10 monclient(hunting): handle_auth_done global_id 54183 payload 293 2024-09-11T01:35:01.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96edc4d640 1 -- 172.21.15.12:0/2758914264 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96d4004f40 msgr2=0x7f96d400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:35:01.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96edc4d640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96d4004f40 0x7f96d400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:01.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96edc4d640 1 -- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96d4004100 msgr2=0x7f96d4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:35:01.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96edc4d640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96d4004100 0x7f96d4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:01.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96edc4d640 10 monclient: _finish_hunting 0 2024-09-11T01:35:01.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96edc4d640 1 monclient: found mon.c 2024-09-11T01:35:01.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96edc4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:01.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96edc4d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:35:01.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96edc4d640 1 -- 172.21.15.12:0/2758914264 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96d4003520 con 0x7f96e809d460 2024-09-11T01:35:01.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96edc4d640 10 monclient: _finish_auth 0 2024-09-11T01:35:01.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96edc4d640 10 monclient: _check_auth_tickets 2024-09-11T01:35:01.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96edc4d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:01.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96efeb1640 5 monclient: authenticate success, global_id 54183 2024-09-11T01:35:01.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96efeb1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:35:01.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96ee44e640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96d4004f40 0x7f96d400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:35:01.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.067+0000 7f96ed44c640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96d4004100 0x7f96d4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:35:01.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.070+0000 7f96edc4d640 1 -- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 msgr2=0x7f96d4003aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:01.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.070+0000 7f96edc4d640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 0x7f96d4003aa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f96e4018ae0 tx=0x7f96d4005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=318b57011fcd5f68 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:01.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.070+0000 7f96ecc4b640 1 -- 172.21.15.12:0/2758914264 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e400f040 con 0x7f96e809d460 2024-09-11T01:35:01.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.070+0000 7f96ecc4b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:01.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.070+0000 7f96ecc4b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:35:01.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.070+0000 7f96ecc4b640 10 monclient: dump: 2024-09-11T01:35:01.085 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:01.085 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:01.085 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:01.085 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:01.085 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:01.085 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:01.085 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:01.085 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:01.086 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:01.086 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:01.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.070+0000 7f96ecc4b640 1 -- 172.21.15.12:0/2758914264 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96e402d6c0 con 0x7f96e809d460 2024-09-11T01:35:01.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.070+0000 7f96efeb1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:35:01.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.070+0000 7f96ecc4b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:35:01.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.070+0000 7f96efeb1640 10 monclient: get_monmap_and_config success 2024-09-11T01:35:01.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.070+0000 7f96ecc4b640 1 -- 172.21.15.12:0/2758914264 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e402dab0 con 0x7f96e809d460 2024-09-11T01:35:01.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.070+0000 7f96ecc4b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:01.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.070+0000 7f96efeb1640 4 set_mon_vals no callback set 2024-09-11T01:35:01.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.070+0000 7f96ecc4b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:35:01.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.070+0000 7f96ecc4b640 10 monclient: dump: 2024-09-11T01:35:01.086 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:01.087 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:01.087 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:01.087 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:01.087 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:01.087 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:01.087 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:01.087 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:01.087 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:01.087 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:01.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.071+0000 7f96efeb1640 10 monclient: shutdown 2024-09-11T01:35:01.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.071+0000 7f96efeb1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:35:01.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.071+0000 7f96efeb1640 1 -- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 msgr2=0x7f96d4003aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:01.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.071+0000 7f96efeb1640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 0x7f96d4003aa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f96e4018ae0 tx=0x7f96d4005830 comp rx=0 tx=0).stop 2024-09-11T01:35:01.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.071+0000 7f96efeb1640 1 -- 172.21.15.12:0/2758914264 shutdown_connections 2024-09-11T01:35:01.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.071+0000 7f96efeb1640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96d4004f40 0x7f96d400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:01.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.071+0000 7f96efeb1640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96d4004100 0x7f96d4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:01.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.071+0000 7f96efeb1640 1 --2- 172.21.15.12:0/2758914264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 0x7f96d4003aa0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:01.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.071+0000 7f96efeb1640 1 -- 172.21.15.12:0/2758914264 >> 172.21.15.12:0/2758914264 conn(0x7f96e80a7120 msgr2=0x7f96e80a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:35:01.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.071+0000 7f96efeb1640 1 -- 172.21.15.12:0/2758914264 shutdown_connections 2024-09-11T01:35:01.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.071+0000 7f96efeb1640 1 -- 172.21.15.12:0/2758914264 wait complete. 2024-09-11T01:35:01.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.072+0000 7f96efeb1640 10 monclient: build_initial_monmap 2024-09-11T01:35:01.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.072+0000 7f96efeb1640 10 monclient: monmap: 2024-09-11T01:35:01.088 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:35:01.088 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:35:01.088 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:35:01.088 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:35:01.088 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:35:01.088 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:01.089 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:35:01.089 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:35:01.089 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:35:01.089 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:01.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.072+0000 7f96efeb1640 1 Processor -- start 2024-09-11T01:35:01.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.072+0000 7f96efeb1640 1 -- start start 2024-09-11T01:35:01.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.072+0000 7f96efeb1640 10 monclient: init 2024-09-11T01:35:01.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96efeb1640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:01.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96efeb1640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:35:01.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96efeb1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96d4004100 0x7f96e8136dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:01.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96efeb1640 10 monclient(hunting): picked mon.noname-a con 0x7f96d4004100 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:35:01.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96efeb1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96d4004f40 0x7f96e8137300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:01.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96efeb1640 10 monclient(hunting): picked mon.noname-b con 0x7f96d4004f40 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:35:01.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96efeb1640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 0x7f96e8124a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:01.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96efeb1640 10 monclient(hunting): picked mon.noname-c con 0x7f96e809d460 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:35:01.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96efeb1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:01.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96efeb1640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f96e8138790 con 0x7f96d4004100 2024-09-11T01:35:01.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96efeb1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:01.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96efeb1640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f96e8138610 con 0x7f96d4004f40 2024-09-11T01:35:01.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96efeb1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:01.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96efeb1640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f96e8138910 con 0x7f96e809d460 2024-09-11T01:35:01.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96efeb1640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:01.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96efeb1640 10 monclient(hunting): authenticate will time out at 1107.515479s 2024-09-11T01:35:01.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96edc4d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96d4004100 0x7f96e8136dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:01.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96ee44e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 0x7f96e8124a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:01.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96edc4d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96d4004100 0x7f96e8136dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48840/0 (socket says 172.21.15.12:48840) 2024-09-11T01:35:01.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96edc4d640 1 -- 172.21.15.12:0/3870065622 learned_addr learned my addr 172.21.15.12:0/3870065622 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:35:01.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96edc4d640 10 monclient(hunting): get_auth_request con 0x7f96d4004100 auth_method 0 2024-09-11T01:35:01.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96edc4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:01.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96edc4d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:01.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96edc4d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:01.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96ee44e640 10 monclient(hunting): get_auth_request con 0x7f96e809d460 auth_method 0 2024-09-11T01:35:01.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96ee44e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:01.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96ee44e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:01.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96ee44e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:01.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96ed44c640 1 --2- 172.21.15.12:0/3870065622 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96d4004f40 0x7f96e8137300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:01.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96edc4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:01.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96edc4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:01.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96edc4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:01.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96ee44e640 1 -- 172.21.15.12:0/3870065622 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 msgr2=0x7f96e8124a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:35:01.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96ed44c640 10 monclient(hunting): get_auth_request con 0x7f96d4004f40 auth_method 0 2024-09-11T01:35:01.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96ee44e640 1 -- 172.21.15.12:0/3870065622 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 msgr2=0x7f96e8124a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:35:01.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96ed44c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:01.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96ed44c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:01.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96ed44c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:01.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96ee44e640 1 --2- 172.21.15.12:0/3870065622 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 0x7f96e8124a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:35:01.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96ee44e640 1 --2- 172.21.15.12:0/3870065622 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 0x7f96e8124a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:35:01.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96ed44c640 1 -- 172.21.15.12:0/3870065622 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96d4004f40 msgr2=0x7f96e8137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:35:01.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96ed44c640 1 -- 172.21.15.12:0/3870065622 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96d4004f40 msgr2=0x7f96e8137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:35:01.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96ed44c640 1 --2- 172.21.15.12:0/3870065622 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96d4004f40 0x7f96e8137300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:35:01.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.073+0000 7f96ed44c640 1 --2- 172.21.15.12:0/3870065622 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96d4004f40 0x7f96e8137300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:35:01.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.074+0000 7f96edc4d640 10 monclient(hunting): handle_auth_done global_id 54186 payload 824 2024-09-11T01:35:01.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.074+0000 7f96edc4d640 1 -- 172.21.15.12:0/3870065622 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 msgr2=0x7f96e8124a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:35:01.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.074+0000 7f96edc4d640 1 --2- 172.21.15.12:0/3870065622 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 0x7f96e8124a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:01.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.074+0000 7f96edc4d640 1 -- 172.21.15.12:0/3870065622 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96d4004f40 msgr2=0x7f96e8137300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:35:01.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.074+0000 7f96edc4d640 1 --2- 172.21.15.12:0/3870065622 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96d4004f40 0x7f96e8137300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:01.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.074+0000 7f96edc4d640 10 monclient: _finish_hunting 0 2024-09-11T01:35:01.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.074+0000 7f96edc4d640 1 monclient: found mon.noname-a 2024-09-11T01:35:01.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.074+0000 7f96edc4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:01.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.074+0000 7f96edc4d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:35:01.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.074+0000 7f96edc4d640 1 -- 172.21.15.12:0/3870065622 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96e81cdbf0 con 0x7f96d4004100 2024-09-11T01:35:01.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.076+0000 7f96edc4d640 1 -- 172.21.15.12:0/3870065622 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96d4004100 msgr2=0x7f96e8136dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:01.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.076+0000 7f96edc4d640 1 --2- 172.21.15.12:0/3870065622 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96d4004100 0x7f96e8136dc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f96e4000c00 tx=0x7f96e4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:01.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.076+0000 7f96de7fc640 1 -- 172.21.15.12:0/3870065622 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e4044020 con 0x7f96d4004100 2024-09-11T01:35:01.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.076+0000 7f96de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:01.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.076+0000 7f96de7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:35:01.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.076+0000 7f96de7fc640 10 monclient: dump: 2024-09-11T01:35:01.094 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:01.094 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:01.094 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:01.094 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:01.094 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:01.094 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:01.094 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:01.094 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:01.094 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:01.094 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:01.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.076+0000 7f96de7fc640 10 monclient: _finish_auth 0 2024-09-11T01:35:01.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.076+0000 7f96de7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:35:01.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.076+0000 7f96de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:01.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.076+0000 7f96de7fc640 1 -- 172.21.15.12:0/3870065622 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96e4040600 con 0x7f96d4004100 2024-09-11T01:35:01.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.076+0000 7f96efeb1640 5 monclient: authenticate success, global_id 54186 2024-09-11T01:35:01.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.076+0000 7f96efeb1640 10 monclient: _renew_subs 2024-09-11T01:35:01.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.076+0000 7f96efeb1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:35:01.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.076+0000 7f96efeb1640 1 -- 172.21.15.12:0/3870065622 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96e809db10 con 0x7f96d4004100 2024-09-11T01:35:01.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.076+0000 7f96de7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:35:01.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.076+0000 7f96de7fc640 1 -- 172.21.15.12:0/3870065622 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e403c040 con 0x7f96d4004100 2024-09-11T01:35:01.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.076+0000 7f96de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:01.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.076+0000 7f96de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:35:01.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.076+0000 7f96de7fc640 10 monclient: dump: 2024-09-11T01:35:01.095 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:01.095 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:01.095 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:01.095 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:01.096 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:01.096 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:01.096 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:01.096 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:01.096 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:01.096 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:01.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.076+0000 7f96dffff640 4 set_mon_vals no callback set 2024-09-11T01:35:01.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.076+0000 7f96efeb1640 10 monclient: _renew_subs 2024-09-11T01:35:01.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.076+0000 7f96efeb1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:35:01.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.076+0000 7f96efeb1640 1 -- 172.21.15.12:0/3870065622 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96e8127f20 con 0x7f96d4004100 2024-09-11T01:35:01.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.078+0000 7f96efeb1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:01.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.078+0000 7f96efeb1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:01.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.078+0000 7f96efeb1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:35:01.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.078+0000 7f96de7fc640 1 -- 172.21.15.12:0/3870065622 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f96e403f030 con 0x7f96d4004100 2024-09-11T01:35:01.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.078+0000 7f96efeb1640 1 -- 172.21.15.12:0/3870065622 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96e8138000 con 0x7f96d4004100 2024-09-11T01:35:01.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.078+0000 7f96de7fc640 1 --2- 172.21.15.12:0/3870065622 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f96cc03ec30 0x7f96cc0410f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:01.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.078+0000 7f96de7fc640 1 -- 172.21.15.12:0/3870065622 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 4767+0+0 (secure 0 0 0) 0x7f96e4074930 con 0x7f96d4004100 2024-09-11T01:35:01.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.078+0000 7f96ed44c640 1 --2- 172.21.15.12:0/3870065622 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f96cc03ec30 0x7f96cc0410f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:01.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.078+0000 7f96ed44c640 10 monclient: get_auth_request con 0x7f96cc03ec30 auth_method 0 2024-09-11T01:35:01.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.081+0000 7f96ed44c640 1 --2- 172.21.15.12:0/3870065622 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f96cc03ec30 0x7f96cc0410f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f96e80aa220 tx=0x7f96d8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:01.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.083+0000 7f96de7fc640 1 -- 172.21.15.12:0/3870065622 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f96e8138000 con 0x7f96d4004100 2024-09-11T01:35:01.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.083+0000 7f96de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:01.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.083+0000 7f96de7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:35:01.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.341+0000 7f96efeb1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:35:01.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.341+0000 7f96efeb1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:35:01.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.341+0000 7f96efeb1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:35:01.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.341+0000 7f96efeb1640 1 -- 172.21.15.12:0/3870065622 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f96e80a1810 con 0x7f96d4004100 2024-09-11T01:35:01.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.341+0000 7f96de7fc640 1 -- 172.21.15.12:0/3870065622 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f96e404e030 con 0x7f96d4004100 2024-09-11T01:35:01.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.341+0000 7f96de7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:35:01.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.341+0000 7f96de7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:35:01.344 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":60,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:35:01.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.343+0000 7f96efeb1640 1 -- 172.21.15.12:0/3870065622 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f96cc03ec30 msgr2=0x7f96cc0410f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:01.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.343+0000 7f96efeb1640 1 --2- 172.21.15.12:0/3870065622 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f96cc03ec30 0x7f96cc0410f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f96e80aa220 tx=0x7f96d8009040 comp rx=0 tx=0).stop 2024-09-11T01:35:01.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.343+0000 7f96efeb1640 10 monclient: shutdown 2024-09-11T01:35:01.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.343+0000 7f96efeb1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:35:01.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.343+0000 7f96efeb1640 1 -- 172.21.15.12:0/3870065622 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96d4004100 msgr2=0x7f96e8136dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:01.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.343+0000 7f96efeb1640 1 --2- 172.21.15.12:0/3870065622 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96d4004100 0x7f96e8136dc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f96e4000c00 tx=0x7f96e4004730 comp rx=0 tx=0).stop 2024-09-11T01:35:01.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.343+0000 7f96efeb1640 1 -- 172.21.15.12:0/3870065622 shutdown_connections 2024-09-11T01:35:01.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.343+0000 7f96efeb1640 1 --2- 172.21.15.12:0/3870065622 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96e809d460 0x7f96e8124a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:01.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.343+0000 7f96efeb1640 1 --2- 172.21.15.12:0/3870065622 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96d4004f40 0x7f96e8137300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:01.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.343+0000 7f96efeb1640 1 --2- 172.21.15.12:0/3870065622 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f96cc03ec30 0x7f96cc0410f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:01.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.343+0000 7f96efeb1640 1 --2- 172.21.15.12:0/3870065622 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96d4004100 0x7f96e8136dc0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:01.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.343+0000 7f96efeb1640 1 -- 172.21.15.12:0/3870065622 >> 172.21.15.12:0/3870065622 conn(0x7f96e80a7120 msgr2=0x7f96e80a4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:35:01.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.343+0000 7f96efeb1640 1 -- 172.21.15.12:0/3870065622 shutdown_connections 2024-09-11T01:35:01.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:01.343+0000 7f96efeb1640 1 -- 172.21.15.12:0/3870065622 wait complete. 2024-09-11T01:35:02.830 INFO:tasks.rados.rados.0.smithi071.stdout:831: done (4 left) 2024-09-11T01:35:02.830 INFO:tasks.rados.rados.0.smithi071.stdout:832: done (3 left) 2024-09-11T01:35:02.830 INFO:tasks.rados.rados.0.smithi071.stdout:833: done (2 left) 2024-09-11T01:35:02.830 INFO:tasks.rados.rados.0.smithi071.stdout:834: done (1 left) 2024-09-11T01:35:02.830 INFO:tasks.rados.rados.0.smithi071.stdout:836: done (0 left) 2024-09-11T01:35:02.830 INFO:tasks.rados.rados.0.smithi071.stdout:837: read oid 22 snap -1 2024-09-11T01:35:02.830 INFO:tasks.rados.rados.0.smithi071.stdout:837: expect (ObjNum 240 snap 58 seq_num 240) 2024-09-11T01:35:02.830 INFO:tasks.rados.rados.0.smithi071.stdout:838: read oid 27 snap -1 2024-09-11T01:35:02.830 INFO:tasks.rados.rados.0.smithi071.stdout:838: expect deleted 2024-09-11T01:35:02.830 INFO:tasks.rados.rados.0.smithi071.stdout:839: read oid 45 snap -1 2024-09-11T01:35:02.830 INFO:tasks.rados.rados.0.smithi071.stdout:839: expect (ObjNum 297 snap 80 seq_num 297) 2024-09-11T01:35:02.831 INFO:tasks.rados.rados.0.smithi071.stdout:840: write oid 47 current snap is 90 2024-09-11T01:35:02.831 INFO:tasks.rados.rados.0.smithi071.stdout:840: seq_num 314 ranges {477398=798659,1774640=663444,2873055=404343} 2024-09-11T01:35:02.837 INFO:tasks.rados.rados.0.smithi071.stdout:840: writing smithi07134777-47 from 477398 to 1276057 tid 1 2024-09-11T01:35:02.843 INFO:tasks.rados.rados.0.smithi071.stdout:840: writing smithi07134777-47 from 1774640 to 2438084 tid 2 2024-09-11T01:35:02.848 INFO:tasks.rados.rados.0.smithi071.stdout:840: writing smithi07134777-47 from 2873055 to 3277398 tid 3 2024-09-11T01:35:02.848 INFO:tasks.rados.rados.0.smithi071.stdout:841: write oid 3 current snap is 90 2024-09-11T01:35:02.848 INFO:tasks.rados.rados.0.smithi071.stdout:841: seq_num 315 ranges {494205=473908,1433122=407102,2384125=462684} 2024-09-11T01:35:02.854 INFO:tasks.rados.rados.0.smithi071.stdout:841: writing smithi07134777-3 from 494205 to 968113 tid 1 2024-09-11T01:35:02.858 INFO:tasks.rados.rados.0.smithi071.stdout:841: writing smithi07134777-3 from 1433122 to 1840224 tid 2 2024-09-11T01:35:02.864 INFO:tasks.rados.rados.0.smithi071.stdout:841: writing smithi07134777-3 from 2384125 to 2846809 tid 3 2024-09-11T01:35:02.864 INFO:tasks.rados.rados.0.smithi071.stdout:842: delete oid 6 current snap is 90 2024-09-11T01:35:02.864 INFO:tasks.rados.rados.0.smithi071.stdout:837: expect (ObjNum 240 snap 58 seq_num 240) 2024-09-11T01:35:02.864 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:35:02.897 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:35:02.929 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:35:02.962 INFO:tasks.rados.rados.0.smithi071.stdout:840: finishing write tid 1 to smithi07134777-47 2024-09-11T01:35:02.962 INFO:tasks.rados.rados.0.smithi071.stdout:840: finishing write tid 2 to smithi07134777-47 2024-09-11T01:35:02.962 INFO:tasks.rados.rados.0.smithi071.stdout:840: finishing write tid 3 to smithi07134777-47 2024-09-11T01:35:02.962 INFO:tasks.rados.rados.0.smithi071.stdout:840: finishing write tid 4 to smithi07134777-47 2024-09-11T01:35:02.963 INFO:tasks.rados.rados.0.smithi071.stdout:840: finishing write tid 5 to smithi07134777-47 2024-09-11T01:35:02.963 INFO:tasks.rados.rados.0.smithi071.stdout:840: oid 47 updating version 0 to 306 2024-09-11T01:35:02.963 INFO:tasks.rados.rados.0.smithi071.stdout:840: oid 47 version 306 is already newer than 304 2024-09-11T01:35:02.963 INFO:tasks.rados.rados.0.smithi071.stdout:840: oid 47 version 306 is already newer than 305 2024-09-11T01:35:02.963 INFO:tasks.rados.rados.0.smithi071.stdout:840: oid 47 version 306 is already newer than 303 2024-09-11T01:35:02.963 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 306 (ObjNum 314 snap 90 seq_num 314) dirty exists 2024-09-11T01:35:02.963 INFO:tasks.rados.rados.0.smithi071.stdout:840: left oid 47 (ObjNum 314 snap 90 seq_num 314) 2024-09-11T01:35:02.963 INFO:tasks.rados.rados.0.smithi071.stdout:841: finishing write tid 1 to smithi07134777-3 2024-09-11T01:35:02.963 INFO:tasks.rados.rados.0.smithi071.stdout:841: finishing write tid 2 to smithi07134777-3 2024-09-11T01:35:02.963 INFO:tasks.rados.rados.0.smithi071.stdout:841: finishing write tid 3 to smithi07134777-3 2024-09-11T01:35:02.963 INFO:tasks.rados.rados.0.smithi071.stdout:841: finishing write tid 4 to smithi07134777-3 2024-09-11T01:35:02.963 INFO:tasks.rados.rados.0.smithi071.stdout:841: finishing write tid 5 to smithi07134777-3 2024-09-11T01:35:02.963 INFO:tasks.rados.rados.0.smithi071.stdout:841: oid 3 updating version 0 to 228 2024-09-11T01:35:02.963 INFO:tasks.rados.rados.0.smithi071.stdout:841: oid 3 version 228 is already newer than 225 2024-09-11T01:35:02.964 INFO:tasks.rados.rados.0.smithi071.stdout:841: oid 3 version 228 is already newer than 226 2024-09-11T01:35:02.964 INFO:tasks.rados.rados.0.smithi071.stdout:841: oid 3 version 228 is already newer than 227 2024-09-11T01:35:02.964 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 228 (ObjNum 315 snap 90 seq_num 315) dirty exists 2024-09-11T01:35:02.964 INFO:tasks.rados.rados.0.smithi071.stdout:841: left oid 3 (ObjNum 315 snap 90 seq_num 315) 2024-09-11T01:35:02.984 INFO:tasks.rados.rados.0.smithi071.stdout:837: done (5 left) 2024-09-11T01:35:02.985 INFO:tasks.rados.rados.0.smithi071.stdout:838: done (4 left) 2024-09-11T01:35:02.985 INFO:tasks.rados.rados.0.smithi071.stdout:840: done (3 left) 2024-09-11T01:35:02.985 INFO:tasks.rados.rados.0.smithi071.stdout:841: done (2 left) 2024-09-11T01:35:02.985 INFO:tasks.rados.rados.0.smithi071.stdout:842: done (1 left) 2024-09-11T01:35:02.985 INFO:tasks.rados.rados.0.smithi071.stdout:843: snap_create 2024-09-11T01:35:02.989 INFO:tasks.rados.rados.0.smithi071.stdout:839: expect (ObjNum 297 snap 80 seq_num 297) 2024-09-11T01:35:03.047 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2363037 2024-09-11T01:35:03.142 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:35:03.834 INFO:tasks.rados.rados.0.smithi071.stdout:839: done (1 left) 2024-09-11T01:35:03.835 INFO:tasks.rados.rados.0.smithi071.stdout:843: done (0 left) 2024-09-11T01:35:03.835 INFO:tasks.rados.rados.0.smithi071.stdout:844: copy_from oid 16 from oid 8 current snap is 91 2024-09-11T01:35:03.835 INFO:tasks.rados.rados.0.smithi071.stdout:845: read oid 49 snap -1 2024-09-11T01:35:03.835 INFO:tasks.rados.rados.0.smithi071.stdout:845: expect (ObjNum 200 snap 46 seq_num 200) 2024-09-11T01:35:03.835 INFO:tasks.rados.rados.0.smithi071.stdout:846: delete oid 40 current snap is 91 2024-09-11T01:35:03.839 INFO:tasks.rados.rados.0.smithi071.stdout:846: done (2 left) 2024-09-11T01:35:03.839 INFO:tasks.rados.rados.0.smithi071.stdout:847: write (excl) oid 36 current snap is 91 2024-09-11T01:35:03.839 INFO:tasks.rados.rados.0.smithi071.stdout:847: seq_num 318 ranges {544626=699655,1892215=554429,3150041=194585} 2024-09-11T01:35:03.846 INFO:tasks.rados.rados.0.smithi071.stdout:847: writing smithi07134777-36 from 544626 to 1244281 tid 1 2024-09-11T01:35:03.853 INFO:tasks.rados.rados.0.smithi071.stdout:847: writing smithi07134777-36 from 1892215 to 2446644 tid 2 2024-09-11T01:35:03.858 INFO:tasks.rados.rados.0.smithi071.stdout:847: writing smithi07134777-36 from 3150041 to 3344626 tid 3 2024-09-11T01:35:03.858 INFO:tasks.rados.rados.0.smithi071.stdout:848: write (excl) oid 18 current snap is 91 2024-09-11T01:35:03.858 INFO:tasks.rados.rados.0.smithi071.stdout:848: seq_num 319 ranges {561433=774904,1950697=414440,2777464=583969} 2024-09-11T01:35:03.866 INFO:tasks.rados.rados.0.smithi071.stdout:848: writing smithi07134777-18 from 561433 to 1336337 tid 1 2024-09-11T01:35:03.871 INFO:tasks.rados.rados.0.smithi071.stdout:848: writing smithi07134777-18 from 1950697 to 2365137 tid 2 2024-09-11T01:35:03.877 INFO:tasks.rados.rados.0.smithi071.stdout:848: writing smithi07134777-18 from 2777464 to 3361433 tid 3 2024-09-11T01:35:03.877 INFO:tasks.rados.rados.0.smithi071.stdout:849: snap_remove snap 84 2024-09-11T01:35:04.357 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:35:04.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.468+0000 7f55ef90e640 10 monclient: get_monmap_and_config 2024-09-11T01:35:04.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.468+0000 7f55ef90e640 10 monclient: build_initial_monmap 2024-09-11T01:35:04.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.468+0000 7f55ef90e640 10 monclient: monmap: 2024-09-11T01:35:04.470 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:35:04.470 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:04.470 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:35:04.469438+0000 2024-09-11T01:35:04.470 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:35:04.469438+0000 2024-09-11T01:35:04.470 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:35:04.471 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:04.471 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:35:04.471 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:35:04.471 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:35:04.471 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:35:04.471 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:35:04.471 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:04.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.469+0000 7f55ef90e640 1 Processor -- start 2024-09-11T01:35:04.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.469+0000 7f55ef90e640 1 -- start start 2024-09-11T01:35:04.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.469+0000 7f55ef90e640 10 monclient: init 2024-09-11T01:35:04.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ef90e640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:04.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ef90e640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-11T01:35:04.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ef90e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f55e80a0c90 0x7f55e80a1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:04.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ef90e640 10 monclient(hunting): picked mon.noname-c con 0x7f55e80a0c90 addr v2:172.21.15.71:3300/0 2024-09-11T01:35:04.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ef90e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f55e80a2120 0x7f55e80a15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:04.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ef90e640 10 monclient(hunting): picked mon.noname-a con 0x7f55e80a2120 addr v2:172.21.15.12:3300/0 2024-09-11T01:35:04.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ef90e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f55e813c260 addr v1:172.21.15.71:6789/0 2024-09-11T01:35:04.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ef90e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55e809d790 con 0x7f55e813c260 2024-09-11T01:35:04.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ef90e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:04.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ef90e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f55e809e290 con 0x7f55e80a2120 2024-09-11T01:35:04.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ef90e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:04.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ef90e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f55e80a1ea0 con 0x7f55e80a0c90 2024-09-11T01:35:04.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ef90e640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:04.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ef90e640 10 monclient(hunting): authenticate will time out at 1110.912258s 2024-09-11T01:35:04.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ecea9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f55e80a2120 0x7f55e80a15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:04.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ecea9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f55e80a2120 0x7f55e80a15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48872/0 (socket says 172.21.15.12:48872) 2024-09-11T01:35:04.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ecea9640 1 -- 172.21.15.12:0/199522968 learned_addr learned my addr 172.21.15.12:0/199522968 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:35:04.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ecea9640 10 monclient(hunting): get_auth_request con 0x7f55e80a2120 auth_method 0 2024-09-11T01:35:04.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ecea9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:04.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ecea9640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:04.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ecea9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:04.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ecea9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:04.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ecea9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:04.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ecea9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:04.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ed6aa640 1 --2- 172.21.15.12:0/199522968 >> v2:172.21.15.71:3300/0 conn(0x7f55e80a0c90 0x7f55e80a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:04.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ed6aa640 10 monclient(hunting): get_auth_request con 0x7f55e80a0c90 auth_method 0 2024-09-11T01:35:04.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ed6aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:04.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ed6aa640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:04.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ed6aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:04.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ecea9640 10 monclient(hunting): handle_auth_done global_id 54189 payload 274 2024-09-11T01:35:04.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ecea9640 1 -- 172.21.15.12:0/199522968 >> v2:172.21.15.71:3300/0 conn(0x7f55e80a0c90 msgr2=0x7f55e80a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:35:04.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ecea9640 1 --2- 172.21.15.12:0/199522968 >> v2:172.21.15.71:3300/0 conn(0x7f55e80a0c90 0x7f55e80a1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:04.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ecea9640 1 -- 172.21.15.12:0/199522968 >> v1:172.21.15.71:6789/0 conn(0x7f55e813c260 legacy=0x7f55e813c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:35:04.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ecea9640 10 monclient: _finish_hunting 0 2024-09-11T01:35:04.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ecea9640 1 monclient: found mon.noname-a 2024-09-11T01:35:04.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ecea9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:04.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ecea9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:35:04.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.470+0000 7f55ecea9640 1 -- 172.21.15.12:0/199522968 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55e80a06b0 con 0x7f55e80a2120 2024-09-11T01:35:04.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.473+0000 7f55ecea9640 1 -- 172.21.15.12:0/199522968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55e80a2120 msgr2=0x7f55e80a15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:04.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.473+0000 7f55ecea9640 1 --2- 172.21.15.12:0/199522968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55e80a2120 0x7f55e80a15b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f55e809d450 tx=0x7f55dc02a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2574b69d8d300bbf server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:04.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.473+0000 7f55dbfff640 1 -- 172.21.15.12:0/199522968 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55dc037070 con 0x7f55e80a2120 2024-09-11T01:35:04.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.473+0000 7f55dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:04.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.473+0000 7f55dbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:35:04.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.473+0000 7f55dbfff640 10 monclient: dump: 2024-09-11T01:35:04.485 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:04.485 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:04.485 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:04.485 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:04.485 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:04.485 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:04.485 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:04.485 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:04.485 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:04.485 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:04.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.473+0000 7f55dbfff640 10 monclient: _finish_auth 0 2024-09-11T01:35:04.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.473+0000 7f55dbfff640 10 monclient: _check_auth_tickets 2024-09-11T01:35:04.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.473+0000 7f55dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:04.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.473+0000 7f55ef90e640 5 monclient: authenticate success, global_id 54189 2024-09-11T01:35:04.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.473+0000 7f55dbfff640 1 -- 172.21.15.12:0/199522968 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55dc032640 con 0x7f55e80a2120 2024-09-11T01:35:04.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.473+0000 7f55ef90e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:35:04.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.473+0000 7f55dbfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:35:04.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.473+0000 7f55ef90e640 10 monclient: get_monmap_and_config success 2024-09-11T01:35:04.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.473+0000 7f55dbfff640 1 -- 172.21.15.12:0/199522968 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55dc032a10 con 0x7f55e80a2120 2024-09-11T01:35:04.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.473+0000 7f55ef90e640 4 set_mon_vals no callback set 2024-09-11T01:35:04.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.473+0000 7f55dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:04.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.473+0000 7f55dbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:35:04.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.473+0000 7f55dbfff640 10 monclient: dump: 2024-09-11T01:35:04.487 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:04.487 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:04.487 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:04.487 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:04.487 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:04.487 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:04.487 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:04.487 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:04.487 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:04.487 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:04.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.474+0000 7f55ef90e640 10 monclient: shutdown 2024-09-11T01:35:04.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.474+0000 7f55ef90e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:35:04.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.474+0000 7f55ef90e640 1 -- 172.21.15.12:0/199522968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55e80a2120 msgr2=0x7f55e80a15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:04.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.474+0000 7f55ef90e640 1 --2- 172.21.15.12:0/199522968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55e80a2120 0x7f55e80a15b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f55e809d450 tx=0x7f55dc02a0b0 comp rx=0 tx=0).stop 2024-09-11T01:35:04.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.474+0000 7f55ef90e640 1 -- 172.21.15.12:0/199522968 shutdown_connections 2024-09-11T01:35:04.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.474+0000 7f55ef90e640 1 --2- 172.21.15.12:0/199522968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55e80a2120 0x7f55e80a15b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:04.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.474+0000 7f55ef90e640 1 --2- 172.21.15.12:0/199522968 >> v2:172.21.15.71:3300/0 conn(0x7f55e80a0c90 0x7f55e80a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:04.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.474+0000 7f55ef90e640 1 -- 172.21.15.12:0/199522968 >> 172.21.15.12:0/199522968 conn(0x7f55e80a2d90 msgr2=0x7f55e80a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:35:04.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.474+0000 7f55ef90e640 1 -- 172.21.15.12:0/199522968 shutdown_connections 2024-09-11T01:35:04.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.474+0000 7f55ef90e640 1 -- 172.21.15.12:0/199522968 wait complete. 2024-09-11T01:35:04.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.475+0000 7f55ef90e640 10 monclient: build_initial_monmap 2024-09-11T01:35:04.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.475+0000 7f55ef90e640 10 monclient: monmap: 2024-09-11T01:35:04.489 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:35:04.489 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:35:04.489 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:35:04.489 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:35:04.489 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:35:04.489 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:04.489 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:35:04.489 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:35:04.489 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:35:04.489 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:04.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.475+0000 7f55ef90e640 1 Processor -- start 2024-09-11T01:35:04.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.475+0000 7f55ef90e640 1 -- start start 2024-09-11T01:35:04.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.475+0000 7f55ef90e640 10 monclient: init 2024-09-11T01:35:04.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ef90e640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:04.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ef90e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:35:04.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ef90e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f55e80a0c90 0x7f55e81d17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:04.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ef90e640 10 monclient(hunting): picked mon.noname-c con 0x7f55e80a0c90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:35:04.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ef90e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55e80a2120 0x7f55e81d1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:04.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ef90e640 10 monclient(hunting): picked mon.noname-b con 0x7f55e80a2120 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:35:04.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ef90e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55e813c260 0x7f55e8128b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:04.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ef90e640 10 monclient(hunting): picked mon.noname-a con 0x7f55e813c260 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:35:04.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ef90e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:04.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ef90e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f55e80a1c70 con 0x7f55e813c260 2024-09-11T01:35:04.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ef90e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:04.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ef90e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f55e80a1af0 con 0x7f55e80a2120 2024-09-11T01:35:04.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ef90e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:04.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ef90e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f55e813cde0 con 0x7f55e80a0c90 2024-09-11T01:35:04.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ef90e640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:04.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ef90e640 10 monclient(hunting): authenticate will time out at 1110.918121s 2024-09-11T01:35:04.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ed6aa640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f55e80a0c90 0x7f55e81d17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:04.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55edeab640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55e813c260 0x7f55e8128b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:04.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ed6aa640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f55e80a0c90 0x7f55e81d17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:35438/0 (socket says 172.21.15.12:35438) 2024-09-11T01:35:04.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55edeab640 10 monclient(hunting): get_auth_request con 0x7f55e813c260 auth_method 0 2024-09-11T01:35:04.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ed6aa640 1 -- 172.21.15.12:0/2168397263 learned_addr learned my addr 172.21.15.12:0/2168397263 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:35:04.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55edeab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:04.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55edeab640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:04.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55edeab640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:04.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ed6aa640 10 monclient(hunting): get_auth_request con 0x7f55e80a0c90 auth_method 0 2024-09-11T01:35:04.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ed6aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:04.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ed6aa640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:04.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ed6aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:04.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ecea9640 1 --2- 172.21.15.12:0/2168397263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55e80a2120 0x7f55e81d1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:04.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55edeab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:04.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55edeab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:04.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55edeab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:04.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ecea9640 10 monclient(hunting): get_auth_request con 0x7f55e80a2120 auth_method 0 2024-09-11T01:35:04.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ecea9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:04.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ecea9640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:04.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ecea9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:04.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ed6aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:04.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ed6aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:04.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ed6aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:04.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ecea9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:04.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ecea9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:04.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55ecea9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:04.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55edeab640 10 monclient(hunting): handle_auth_done global_id 54192 payload 824 2024-09-11T01:35:04.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55edeab640 1 -- 172.21.15.12:0/2168397263 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f55e80a0c90 msgr2=0x7f55e81d17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:04.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55edeab640 1 --2- 172.21.15.12:0/2168397263 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f55e80a0c90 0x7f55e81d17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:04.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55edeab640 1 -- 172.21.15.12:0/2168397263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55e80a2120 msgr2=0x7f55e81d1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:04.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55edeab640 1 --2- 172.21.15.12:0/2168397263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55e80a2120 0x7f55e81d1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:04.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55edeab640 10 monclient: _finish_hunting 0 2024-09-11T01:35:04.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55edeab640 1 monclient: found mon.noname-a 2024-09-11T01:35:04.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55edeab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:04.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55edeab640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:35:04.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.476+0000 7f55edeab640 1 -- 172.21.15.12:0/2168397263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55e813bea0 con 0x7f55e813c260 2024-09-11T01:35:04.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.478+0000 7f55edeab640 1 -- 172.21.15.12:0/2168397263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55e813c260 msgr2=0x7f55e8128b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:04.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.478+0000 7f55edeab640 1 --2- 172.21.15.12:0/2168397263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55e813c260 0x7f55e8128b00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f55e000c7b0 tx=0x7f55e000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:04.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.479+0000 7f55d9ffb640 1 -- 172.21.15.12:0/2168397263 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e0016020 con 0x7f55e813c260 2024-09-11T01:35:04.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.479+0000 7f55d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:04.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.479+0000 7f55d9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:35:04.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.479+0000 7f55d9ffb640 10 monclient: dump: 2024-09-11T01:35:04.494 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:04.494 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:04.494 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:04.495 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:04.495 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:04.495 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:04.495 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:04.495 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:04.495 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:04.495 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:04.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.479+0000 7f55d9ffb640 10 monclient: _finish_auth 0 2024-09-11T01:35:04.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.479+0000 7f55d9ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:35:04.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.479+0000 7f55d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:04.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.479+0000 7f55d9ffb640 1 -- 172.21.15.12:0/2168397263 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55e000ac60 con 0x7f55e813c260 2024-09-11T01:35:04.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.479+0000 7f55d9ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:35:04.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.479+0000 7f55d9ffb640 1 -- 172.21.15.12:0/2168397263 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e000ce60 con 0x7f55e813c260 2024-09-11T01:35:04.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.479+0000 7f55d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:04.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.479+0000 7f55d9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:35:04.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.479+0000 7f55d9ffb640 10 monclient: dump: 2024-09-11T01:35:04.496 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:04.496 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:04.496 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:04.496 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:04.496 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:04.496 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:04.497 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:04.497 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:04.497 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:04.497 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:04.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.479+0000 7f55ef90e640 5 monclient: authenticate success, global_id 54192 2024-09-11T01:35:04.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.479+0000 7f55ef90e640 10 monclient: _renew_subs 2024-09-11T01:35:04.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.479+0000 7f55db7fe640 4 set_mon_vals no callback set 2024-09-11T01:35:04.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.479+0000 7f55ef90e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:35:04.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.479+0000 7f55ef90e640 1 -- 172.21.15.12:0/2168397263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55e812c8e0 con 0x7f55e813c260 2024-09-11T01:35:04.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.479+0000 7f55ef90e640 10 monclient: _renew_subs 2024-09-11T01:35:04.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.479+0000 7f55ef90e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:35:04.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.479+0000 7f55ef90e640 1 -- 172.21.15.12:0/2168397263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55e812c600 con 0x7f55e813c260 2024-09-11T01:35:04.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.480+0000 7f55d9ffb640 1 -- 172.21.15.12:0/2168397263 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f55e00126a0 con 0x7f55e813c260 2024-09-11T01:35:04.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.480+0000 7f55ef90e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:04.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.480+0000 7f55ef90e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:04.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.480+0000 7f55ef90e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:35:04.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.480+0000 7f55ef90e640 1 -- 172.21.15.12:0/2168397263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55e813c6d0 con 0x7f55e813c260 2024-09-11T01:35:04.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.480+0000 7f55d9ffb640 1 --2- 172.21.15.12:0/2168397263 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f55cc03e260 0x7f55cc040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:04.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.480+0000 7f55d9ffb640 1 -- 172.21.15.12:0/2168397263 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(197..197 src has 1..197) ==== 4739+0+0 (secure 0 0 0) 0x7f55e0052eb0 con 0x7f55e813c260 2024-09-11T01:35:04.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.480+0000 7f55ed6aa640 1 --2- 172.21.15.12:0/2168397263 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f55cc03e260 0x7f55cc040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:04.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.480+0000 7f55ed6aa640 10 monclient: get_auth_request con 0x7f55cc03e260 auth_method 0 2024-09-11T01:35:04.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.483+0000 7f55ed6aa640 1 --2- 172.21.15.12:0/2168397263 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f55cc03e260 0x7f55cc040720 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f55e4006fd0 tx=0x7f55e4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:04.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.485+0000 7f55d9ffb640 1 -- 172.21.15.12:0/2168397263 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f55e0020030 con 0x7f55e813c260 2024-09-11T01:35:04.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.485+0000 7f55d9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:04.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.485+0000 7f55d9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:35:04.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.742+0000 7f55ef90e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:35:04.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.742+0000 7f55ef90e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:35:04.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.742+0000 7f55ef90e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:35:04.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.742+0000 7f55ef90e640 1 -- 172.21.15.12:0/2168397263 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f55e812b0e0 con 0x7f55e813c260 2024-09-11T01:35:04.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.743+0000 7f55d9ffb640 1 -- 172.21.15.12:0/2168397263 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f55e000ae10 con 0x7f55e813c260 2024-09-11T01:35:04.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.743+0000 7f55d9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:35:04.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.743+0000 7f55d9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:35:04.745 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":64,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:35:04.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.745+0000 7f55ef90e640 1 -- 172.21.15.12:0/2168397263 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f55cc03e260 msgr2=0x7f55cc040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:04.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.745+0000 7f55ef90e640 1 --2- 172.21.15.12:0/2168397263 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f55cc03e260 0x7f55cc040720 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f55e4006fd0 tx=0x7f55e4008040 comp rx=0 tx=0).stop 2024-09-11T01:35:04.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.745+0000 7f55ef90e640 10 monclient: shutdown 2024-09-11T01:35:04.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.745+0000 7f55ef90e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:35:04.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.745+0000 7f55ef90e640 1 -- 172.21.15.12:0/2168397263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55e813c260 msgr2=0x7f55e8128b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:04.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.745+0000 7f55ef90e640 1 --2- 172.21.15.12:0/2168397263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55e813c260 0x7f55e8128b00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f55e000c7b0 tx=0x7f55e000cc70 comp rx=0 tx=0).stop 2024-09-11T01:35:04.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.745+0000 7f55ef90e640 1 -- 172.21.15.12:0/2168397263 shutdown_connections 2024-09-11T01:35:04.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.745+0000 7f55ef90e640 1 --2- 172.21.15.12:0/2168397263 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f55cc03e260 0x7f55cc040720 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:04.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.745+0000 7f55ef90e640 1 --2- 172.21.15.12:0/2168397263 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55e813c260 0x7f55e8128b00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:04.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.745+0000 7f55ef90e640 1 --2- 172.21.15.12:0/2168397263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55e80a2120 0x7f55e81d1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:04.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.745+0000 7f55ef90e640 1 --2- 172.21.15.12:0/2168397263 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f55e80a0c90 0x7f55e81d17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:04.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.745+0000 7f55ef90e640 1 -- 172.21.15.12:0/2168397263 >> 172.21.15.12:0/2168397263 conn(0x7f55e80a2d90 msgr2=0x7f55e809ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:35:04.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.745+0000 7f55ef90e640 1 -- 172.21.15.12:0/2168397263 shutdown_connections 2024-09-11T01:35:04.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.745+0000 7f55ef90e640 1 -- 172.21.15.12:0/2168397263 wait complete. 2024-09-11T01:35:04.757 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:35:04.757 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:35:04.757 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:35:04.838 INFO:tasks.rados.rados.0.smithi071.stdout:849: done (4 left) 2024-09-11T01:35:04.838 INFO:tasks.rados.rados.0.smithi071.stdout:850: write oid 12 current snap is 91 2024-09-11T01:35:04.838 INFO:tasks.rados.rados.0.smithi071.stdout:850: seq_num 320 ranges {578240=566506,1609179=558098,2804887=551750} 2024-09-11T01:35:04.844 INFO:tasks.rados.rados.0.smithi071.stdout:850: writing smithi07134777-12 from 578240 to 1144746 tid 1 2024-09-11T01:35:04.850 INFO:tasks.rados.rados.0.smithi071.stdout:850: writing smithi07134777-12 from 1609179 to 2167277 tid 2 2024-09-11T01:35:04.857 INFO:tasks.rados.rados.0.smithi071.stdout:850: writing smithi07134777-12 from 2804887 to 3356637 tid 3 2024-09-11T01:35:04.857 INFO:tasks.rados.rados.0.smithi071.stdout:851: rollback oid 3 current snap is 91 2024-09-11T01:35:04.858 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 3 to 85 2024-09-11T01:35:04.858 INFO:tasks.rados.rados.0.smithi071.stdout:845: expect (ObjNum 200 snap 46 seq_num 200) 2024-09-11T01:35:04.858 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:35:04.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.864+0000 7f90c012b640 10 monclient: get_monmap_and_config 2024-09-11T01:35:04.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.864+0000 7f90c012b640 10 monclient: build_initial_monmap 2024-09-11T01:35:04.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.864+0000 7f90c012b640 10 monclient: monmap: 2024-09-11T01:35:04.866 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:35:04.866 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:04.866 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:35:04.865635+0000 2024-09-11T01:35:04.866 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:35:04.865635+0000 2024-09-11T01:35:04.866 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:35:04.867 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:04.867 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:35:04.867 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:35:04.867 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:35:04.867 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:35:04.867 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:35:04.867 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:04.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.865+0000 7f90c012b640 1 Processor -- start 2024-09-11T01:35:04.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.865+0000 7f90c012b640 1 -- start start 2024-09-11T01:35:04.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.865+0000 7f90c012b640 10 monclient: init 2024-09-11T01:35:04.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90c012b640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:04.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90c012b640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-11T01:35:04.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90c012b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f90b806f540 addr v1:172.21.15.71:6789/0 2024-09-11T01:35:04.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90c012b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f90b809cff0 0x7f90b80a5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:04.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90c012b640 10 monclient(hunting): picked mon.noname-c con 0x7f90b809cff0 addr v2:172.21.15.71:3300/0 2024-09-11T01:35:04.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90c012b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f90b813d230 0x7f90b813f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:04.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90c012b640 10 monclient(hunting): picked mon.noname-a con 0x7f90b813d230 addr v2:172.21.15.12:3300/0 2024-09-11T01:35:04.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90c012b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90b80a6ea0 con 0x7f90b806f540 2024-09-11T01:35:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90c012b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90c012b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f90b806f920 con 0x7f90b813d230 2024-09-11T01:35:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90c012b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90c012b640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f90b809d4d0 con 0x7f90b809cff0 2024-09-11T01:35:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90c012b640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90c012b640 10 monclient(hunting): authenticate will time out at 1111.308310s 2024-09-11T01:35:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90be6c8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f90b813d230 0x7f90b813f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90be6c8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f90b813d230 0x7f90b813f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48898/0 (socket says 172.21.15.12:48898) 2024-09-11T01:35:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90be6c8640 1 -- 172.21.15.12:0/2297520112 learned_addr learned my addr 172.21.15.12:0/2297520112 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:35:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90be6c8640 10 monclient(hunting): get_auth_request con 0x7f90b813d230 auth_method 0 2024-09-11T01:35:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90be6c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90be6c8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90be6c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90bd6c6640 1 --2- 172.21.15.12:0/2297520112 >> v2:172.21.15.71:3300/0 conn(0x7f90b809cff0 0x7f90b80a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90bd6c6640 10 monclient(hunting): get_auth_request con 0x7f90b809cff0 auth_method 0 2024-09-11T01:35:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90bd6c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90bd6c6640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90bd6c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90be6c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90be6c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90be6c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90bd6c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90bd6c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90bd6c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90be6c8640 10 monclient(hunting): handle_auth_done global_id 54195 payload 274 2024-09-11T01:35:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90be6c8640 1 -- 172.21.15.12:0/2297520112 >> v2:172.21.15.71:3300/0 conn(0x7f90b809cff0 msgr2=0x7f90b80a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:35:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90be6c8640 1 --2- 172.21.15.12:0/2297520112 >> v2:172.21.15.71:3300/0 conn(0x7f90b809cff0 0x7f90b80a5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:04.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90be6c8640 1 -- 172.21.15.12:0/2297520112 >> v1:172.21.15.71:6789/0 conn(0x7f90b806f540 legacy=0x7f90b80a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:35:04.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90be6c8640 10 monclient: _finish_hunting 0 2024-09-11T01:35:04.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90be6c8640 1 monclient: found mon.noname-a 2024-09-11T01:35:04.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90be6c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:04.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90be6c8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:35:04.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.866+0000 7f90be6c8640 1 -- 172.21.15.12:0/2297520112 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b809c330 con 0x7f90b813d230 2024-09-11T01:35:04.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.869+0000 7f90be6c8640 1 -- 172.21.15.12:0/2297520112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b813d230 msgr2=0x7f90b813f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:04.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.869+0000 7f90be6c8640 1 --2- 172.21.15.12:0/2297520112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b813d230 0x7f90b813f620 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f90b809ca80 tx=0x7f90b002a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9deab0bd3e4a7cce server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:04.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.869+0000 7f90bcec5640 1 -- 172.21.15.12:0/2297520112 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b0036070 con 0x7f90b813d230 2024-09-11T01:35:04.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.869+0000 7f90bcec5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:04.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.869+0000 7f90bcec5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:35:04.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.869+0000 7f90bcec5640 10 monclient: dump: 2024-09-11T01:35:04.879 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:04.879 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:04.879 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:04.879 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:04.879 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:04.880 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:04.880 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:04.880 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:04.880 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:04.880 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:04.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.869+0000 7f90bcec5640 10 monclient: _finish_auth 0 2024-09-11T01:35:04.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.869+0000 7f90bcec5640 10 monclient: _check_auth_tickets 2024-09-11T01:35:04.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.869+0000 7f90bcec5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:04.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.869+0000 7f90bcec5640 1 -- 172.21.15.12:0/2297520112 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90b0007800 con 0x7f90b813d230 2024-09-11T01:35:04.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.869+0000 7f90c012b640 5 monclient: authenticate success, global_id 54195 2024-09-11T01:35:04.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.869+0000 7f90c012b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:35:04.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.869+0000 7f90bcec5640 10 monclient: handle_config config(0 keys) 2024-09-11T01:35:04.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.869+0000 7f90c012b640 10 monclient: get_monmap_and_config success 2024-09-11T01:35:04.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.869+0000 7f90bcec5640 1 -- 172.21.15.12:0/2297520112 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b0007bd0 con 0x7f90b813d230 2024-09-11T01:35:04.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.869+0000 7f90c012b640 4 set_mon_vals no callback set 2024-09-11T01:35:04.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.869+0000 7f90bcec5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:04.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.869+0000 7f90bcec5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:35:04.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.869+0000 7f90bcec5640 10 monclient: dump: 2024-09-11T01:35:04.881 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:04.881 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:04.881 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:04.881 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:04.881 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:04.881 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:04.881 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:04.881 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:04.882 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:04.882 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:04.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.870+0000 7f90c012b640 10 monclient: shutdown 2024-09-11T01:35:04.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.870+0000 7f90c012b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:35:04.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.870+0000 7f90c012b640 1 -- 172.21.15.12:0/2297520112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b813d230 msgr2=0x7f90b813f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:04.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.870+0000 7f90c012b640 1 --2- 172.21.15.12:0/2297520112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b813d230 0x7f90b813f620 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f90b809ca80 tx=0x7f90b002a570 comp rx=0 tx=0).stop 2024-09-11T01:35:04.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.870+0000 7f90c012b640 1 -- 172.21.15.12:0/2297520112 shutdown_connections 2024-09-11T01:35:04.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.870+0000 7f90c012b640 1 --2- 172.21.15.12:0/2297520112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b813d230 0x7f90b813f620 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:04.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.870+0000 7f90c012b640 1 --2- 172.21.15.12:0/2297520112 >> v2:172.21.15.71:3300/0 conn(0x7f90b809cff0 0x7f90b80a5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:04.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.870+0000 7f90c012b640 1 -- 172.21.15.12:0/2297520112 >> 172.21.15.12:0/2297520112 conn(0x7f90b80a7300 msgr2=0x7f90b80a7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:35:04.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.870+0000 7f90c012b640 1 -- 172.21.15.12:0/2297520112 shutdown_connections 2024-09-11T01:35:04.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.870+0000 7f90c012b640 1 -- 172.21.15.12:0/2297520112 wait complete. 2024-09-11T01:35:04.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.871+0000 7f90c012b640 10 monclient: build_initial_monmap 2024-09-11T01:35:04.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.871+0000 7f90c012b640 10 monclient: monmap: 2024-09-11T01:35:04.884 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:35:04.884 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:35:04.884 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:35:04.885 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:35:04.885 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:35:04.885 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:04.885 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:35:04.885 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:35:04.885 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:35:04.885 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:04.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.871+0000 7f90c012b640 1 Processor -- start 2024-09-11T01:35:04.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.871+0000 7f90c012b640 1 -- start start 2024-09-11T01:35:04.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.871+0000 7f90c012b640 10 monclient: init 2024-09-11T01:35:04.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.871+0000 7f90c012b640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:04.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.871+0000 7f90c012b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:35:04.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.871+0000 7f90c012b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b806f540 0x7f90b8121330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:04.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.871+0000 7f90c012b640 10 monclient(hunting): picked mon.noname-c con 0x7f90b806f540 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:35:04.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.871+0000 7f90c012b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f90b809cff0 0x7f90b81381a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:04.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.871+0000 7f90c012b640 10 monclient(hunting): picked mon.noname-b con 0x7f90b809cff0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:35:04.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.871+0000 7f90c012b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b813d230 0x7f90b81cce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:04.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.871+0000 7f90c012b640 10 monclient(hunting): picked mon.noname-a con 0x7f90b813d230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:35:04.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.871+0000 7f90c012b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:04.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.871+0000 7f90c012b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f90b80a5df0 con 0x7f90b813d230 2024-09-11T01:35:04.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.871+0000 7f90c012b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:04.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.871+0000 7f90c012b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f90b80a5c70 con 0x7f90b809cff0 2024-09-11T01:35:04.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.871+0000 7f90c012b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:04.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.871+0000 7f90c012b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f90b813fb60 con 0x7f90b806f540 2024-09-11T01:35:04.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.871+0000 7f90c012b640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:04.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.871+0000 7f90c012b640 10 monclient(hunting): authenticate will time out at 1111.313995s 2024-09-11T01:35:04.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90bdec7640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b806f540 0x7f90b8121330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:04.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90be6c8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b813d230 0x7f90b81cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:04.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90bdec7640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b806f540 0x7f90b8121330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:35442/0 (socket says 172.21.15.12:35442) 2024-09-11T01:35:04.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90be6c8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b813d230 0x7f90b81cce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48914/0 (socket says 172.21.15.12:48914) 2024-09-11T01:35:04.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90bdec7640 1 -- 172.21.15.12:0/2074723002 learned_addr learned my addr 172.21.15.12:0/2074723002 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:35:04.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90bdec7640 10 monclient(hunting): get_auth_request con 0x7f90b806f540 auth_method 0 2024-09-11T01:35:04.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90bdec7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:04.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90bdec7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:04.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90bdec7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:04.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90be6c8640 10 monclient(hunting): get_auth_request con 0x7f90b813d230 auth_method 0 2024-09-11T01:35:04.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90be6c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:04.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90be6c8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:04.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90be6c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:04.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90bd6c6640 1 --2- 172.21.15.12:0/2074723002 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f90b809cff0 0x7f90b81381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:04.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90bd6c6640 10 monclient(hunting): get_auth_request con 0x7f90b809cff0 auth_method 0 2024-09-11T01:35:04.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90bd6c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:04.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90bd6c6640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:04.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90bd6c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:04.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90be6c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:04.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90be6c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:04.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90be6c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:04.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90bdec7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:04.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90bdec7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:04.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90bdec7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:04.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90bd6c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:04.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90bd6c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:04.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90bd6c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:04.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90be6c8640 10 monclient(hunting): handle_auth_done global_id 54198 payload 824 2024-09-11T01:35:04.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90be6c8640 1 -- 172.21.15.12:0/2074723002 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b806f540 msgr2=0x7f90b8121330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:04.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90be6c8640 1 --2- 172.21.15.12:0/2074723002 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90b806f540 0x7f90b8121330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:04.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90be6c8640 1 -- 172.21.15.12:0/2074723002 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f90b809cff0 msgr2=0x7f90b81381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:04.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90be6c8640 1 --2- 172.21.15.12:0/2074723002 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f90b809cff0 0x7f90b81381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:04.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90be6c8640 10 monclient: _finish_hunting 0 2024-09-11T01:35:04.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90be6c8640 1 monclient: found mon.noname-a 2024-09-11T01:35:04.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90be6c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:04.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90be6c8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:35:04.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.872+0000 7f90be6c8640 1 -- 172.21.15.12:0/2074723002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b8122b30 con 0x7f90b813d230 2024-09-11T01:35:04.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.874+0000 7f90be6c8640 1 -- 172.21.15.12:0/2074723002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b813d230 msgr2=0x7f90b81cce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:04.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.874+0000 7f90be6c8640 1 --2- 172.21.15.12:0/2074723002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b813d230 0x7f90b81cce50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f90b0030000 tx=0x7f90b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:04.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.875+0000 7f90ae7fc640 1 -- 172.21.15.12:0/2074723002 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b0041020 con 0x7f90b813d230 2024-09-11T01:35:04.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.875+0000 7f90ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:04.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.875+0000 7f90ae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:35:04.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.875+0000 7f90ae7fc640 10 monclient: dump: 2024-09-11T01:35:04.890 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:04.890 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:04.890 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:04.890 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:04.890 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:04.890 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:04.890 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:04.890 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:04.890 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:04.890 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:04.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.875+0000 7f90ae7fc640 10 monclient: _finish_auth 0 2024-09-11T01:35:04.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.875+0000 7f90ae7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:35:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.875+0000 7f90ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.875+0000 7f90ae7fc640 1 -- 172.21.15.12:0/2074723002 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90b0032950 con 0x7f90b813d230 2024-09-11T01:35:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.875+0000 7f90c012b640 5 monclient: authenticate success, global_id 54198 2024-09-11T01:35:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.875+0000 7f90c012b640 10 monclient: _renew_subs 2024-09-11T01:35:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.875+0000 7f90c012b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:35:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.875+0000 7f90c012b640 1 -- 172.21.15.12:0/2074723002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90b813f8c0 con 0x7f90b813d230 2024-09-11T01:35:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.875+0000 7f90ae7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:35:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.875+0000 7f90ae7fc640 1 -- 172.21.15.12:0/2074723002 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b0036040 con 0x7f90b813d230 2024-09-11T01:35:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.875+0000 7f90ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.875+0000 7f90ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:35:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.875+0000 7f90ae7fc640 10 monclient: dump: 2024-09-11T01:35:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:04.891 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:04.892 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.875+0000 7f90affff640 4 set_mon_vals no callback set 2024-09-11T01:35:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.875+0000 7f90c012b640 10 monclient: _renew_subs 2024-09-11T01:35:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.875+0000 7f90c012b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:35:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.875+0000 7f90c012b640 1 -- 172.21.15.12:0/2074723002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90b80a11f0 con 0x7f90b813d230 2024-09-11T01:35:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.875+0000 7f90c012b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:04.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.876+0000 7f90c012b640 1 --2- 172.21.15.12:0/2074723002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b81386e0 0x7f90b81cd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:04.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.876+0000 7f90c012b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:04.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.876+0000 7f90c012b640 1 -- 172.21.15.12:0/2074723002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f90b806a060 con 0x7f90b81386e0 2024-09-11T01:35:04.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.876+0000 7f90bdec7640 1 --2- 172.21.15.12:0/2074723002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b81386e0 0x7f90b81cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:04.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.876+0000 7f90ae7fc640 1 -- 172.21.15.12:0/2074723002 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f90b003c470 con 0x7f90b813d230 2024-09-11T01:35:04.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.876+0000 7f90bdec7640 10 monclient: get_auth_request con 0x7f90b81386e0 auth_method 0 2024-09-11T01:35:04.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.876+0000 7f90bdec7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:04.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.876+0000 7f90bdec7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:04.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.876+0000 7f90bdec7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:04.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.876+0000 7f90bdec7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:04.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.876+0000 7f90bdec7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:04.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.876+0000 7f90bdec7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:04.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.876+0000 7f90bdec7640 10 monclient(hunting): handle_auth_done global_id 54201 payload 824 2024-09-11T01:35:04.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.876+0000 7f90ae7fc640 1 --2- 172.21.15.12:0/2074723002 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f90a403e260 0x7f90a4040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:04.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.876+0000 7f90bdec7640 1 -- 172.21.15.12:0/2074723002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f90b80a5020 con 0x7f90b81386e0 2024-09-11T01:35:04.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.876+0000 7f90ae7fc640 1 -- 172.21.15.12:0/2074723002 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(198..198 src has 1..198) ==== 4783+0+0 (secure 0 0 0) 0x7f90b0073b00 con 0x7f90b813d230 2024-09-11T01:35:04.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.876+0000 7f90bd6c6640 1 --2- 172.21.15.12:0/2074723002 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f90a403e260 0x7f90a4040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:04.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.876+0000 7f90bd6c6640 10 monclient: get_auth_request con 0x7f90a403e260 auth_method 0 2024-09-11T01:35:04.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.878+0000 7f90bdec7640 1 -- 172.21.15.12:0/2074723002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b81386e0 msgr2=0x7f90b81cd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:04.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.878+0000 7f90bdec7640 1 --2- 172.21.15.12:0/2074723002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b81386e0 0x7f90b81cd390 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f90a800e7a0 tx=0x7f90a800ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:04.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.878+0000 7f90ae7fc640 1 -- 172.21.15.12:0/2074723002 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a8017070 con 0x7f90b81386e0 2024-09-11T01:35:04.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.878+0000 7f90ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:04.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.878+0000 7f90ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:35:04.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.878+0000 7f90ae7fc640 10 monclient: dump: 2024-09-11T01:35:04.901 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:04.901 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:04.901 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:04.901 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:04.902 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:04.902 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:04.902 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:04.902 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:04.902 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:04.902 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:04.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.878+0000 7f90ae7fc640 1 -- 172.21.15.12:0/2074723002 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f90a800ee70 con 0x7f90b81386e0 2024-09-11T01:35:04.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.878+0000 7f90ae7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:04.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.878+0000 7f90ae7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:35:04.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.879+0000 7f90ae7fc640 1 -- 172.21.15.12:0/2074723002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b81386e0 msgr2=0x7f90b81cd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:04.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.879+0000 7f90ae7fc640 1 --2- 172.21.15.12:0/2074723002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b81386e0 0x7f90b81cd390 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f90a800e7a0 tx=0x7f90a800ec60 comp rx=0 tx=0).stop 2024-09-11T01:35:04.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.879+0000 7f90ae7fc640 1 -- 172.21.15.12:0/2074723002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b81386e0 msgr2=0x7f90b81cd390 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:35:04.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.879+0000 7f90ae7fc640 1 --2- 172.21.15.12:0/2074723002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b81386e0 0x7f90b81cd390 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f90a800e7a0 tx=0x7f90a800ec60 comp rx=0 tx=0).stop 2024-09-11T01:35:04.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.879+0000 7f90bd6c6640 1 --2- 172.21.15.12:0/2074723002 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f90a403e260 0x7f90a4040720 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f90b81cdf10 tx=0x7f90b4009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:04.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.885+0000 7f90c012b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:35:04.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.885+0000 7f90c012b640 1 --2- 172.21.15.12:0/2074723002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b809fbf0 0x7f90b81cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:04.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.885+0000 7f90c012b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:04.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.885+0000 7f90c012b640 1 -- 172.21.15.12:0/2074723002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f90b8069990 con 0x7f90b809fbf0 2024-09-11T01:35:04.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.885+0000 7f90bdec7640 1 --2- 172.21.15.12:0/2074723002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b809fbf0 0x7f90b81cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:04.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.885+0000 7f90bdec7640 10 monclient: get_auth_request con 0x7f90b809fbf0 auth_method 0 2024-09-11T01:35:04.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.885+0000 7f90bdec7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:04.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.885+0000 7f90bdec7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:04.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.885+0000 7f90bdec7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:04.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.885+0000 7f90bdec7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:04.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.885+0000 7f90bdec7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:04.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.885+0000 7f90bdec7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:04.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.885+0000 7f90bdec7640 10 monclient(hunting): handle_auth_done global_id 54204 payload 824 2024-09-11T01:35:04.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.885+0000 7f90bdec7640 1 -- 172.21.15.12:0/2074723002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f90b8122350 con 0x7f90b809fbf0 2024-09-11T01:35:04.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.887+0000 7f90bdec7640 1 -- 172.21.15.12:0/2074723002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b809fbf0 msgr2=0x7f90b81cd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:04.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.887+0000 7f90bdec7640 1 --2- 172.21.15.12:0/2074723002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b809fbf0 0x7f90b81cd8d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f90a8018000 tx=0x7f90a8004680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:04.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.887+0000 7f90ae7fc640 1 -- 172.21.15.12:0/2074723002 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a8017030 con 0x7f90b809fbf0 2024-09-11T01:35:04.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.887+0000 7f90ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:04.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.887+0000 7f90ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:35:04.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.887+0000 7f90ae7fc640 10 monclient: dump: 2024-09-11T01:35:04.907 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:04.908 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:04.908 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:04.908 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:04.908 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:04.908 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:04.908 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:04.909 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:04.909 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:04.909 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:04.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.888+0000 7f90ae7fc640 1 -- 172.21.15.12:0/2074723002 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7f90a80035f0 con 0x7f90b809fbf0 2024-09-11T01:35:04.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.888+0000 7f90ae7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:35:04.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.888+0000 7f90ae7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:35:04.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.888+0000 7f90ae7fc640 1 -- 172.21.15.12:0/2074723002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b809fbf0 msgr2=0x7f90b81cd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:04.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.888+0000 7f90ae7fc640 1 --2- 172.21.15.12:0/2074723002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b809fbf0 0x7f90b81cd8d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f90a8018000 tx=0x7f90a8004680 comp rx=0 tx=0).stop 2024-09-11T01:35:04.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.888+0000 7f90ae7fc640 1 -- 172.21.15.12:0/2074723002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b809fbf0 msgr2=0x7f90b81cd8d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:35:04.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.888+0000 7f90ae7fc640 1 --2- 172.21.15.12:0/2074723002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b809fbf0 0x7f90b81cd8d0 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f90a8018000 tx=0x7f90a8004680 comp rx=0 tx=0).stop 2024-09-11T01:35:04.911 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:35:04.911 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:35:04.911 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:35:04.911 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:35:04.911 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 64, 2024-09-11T01:35:04.911 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 94868, 2024-09-11T01:35:04.911 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:35:04.911 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:35:04.911 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:35:04.912 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:35:04.912 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:04.912 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:35:04.912 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:35:04.912 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:35:04.912 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:35:04.912 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:35:04.912 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:35:04.912 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:35:04.912 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:35:04.912 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:35:04.912 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:35:04.913 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:35:04.913 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:35:04.913 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:35:04.913 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:35:04.913 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:35:04.913 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:04.913 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:35:04.913 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:35:04.913 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:35:04.914 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:35:04.914 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:35:04.914 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:35:04.914 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:35:04.914 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:35:04.914 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:35:04.914 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:35:04.914 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:35:04.914 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:35:04.914 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:35:04.914 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:04.914 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:04.915 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:35:04.915 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:35:04.915 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:35:04.915 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:35:04.915 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:35:04.915 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:35:04.915 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:35:04.915 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:35:04.915 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:35:04.915 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:35:04.915 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:35:04.915 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:35:04.915 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:35:04.915 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:35:04.916 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:35:04.916 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:35:04.916 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:35:04.916 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:35:04.916 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:35:04.916 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:35:04.916 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:35:04.916 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:35:04.916 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:35:04.916 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:35:04.916 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:35:04.917 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:35:04.917 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:35:04.917 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:35:04.917 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:04.917 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:35:04.917 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:04.917 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:35:04.917 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:04.917 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:35:04.917 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:35:04.917 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:35:04.918 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:35:04.918 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:04.918 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:35:04.918 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:35:04.918 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:04.918 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:04.918 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:04.918 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:35:04.918 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:35:04.918 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:04.918 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:04.918 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:04.918 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:04.918 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:35:04.919 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:35:04.919 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:35:04.919 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:35:04.919 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:35:04.919 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:04.919 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:04.919 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:35:04.919 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:35:04.919 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:35:04.919 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:35:04.919 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:04.919 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:35:04.919 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:35:04.919 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:04.920 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:04.920 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:04.920 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:35:04.920 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:35:04.920 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:04.920 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:04.920 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:04.920 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:04.920 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:35:04.920 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:35:04.920 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:35:04.921 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:35:04.921 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:35:04.921 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:04.921 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:04.921 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:35:04.921 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:35:04.921 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:35:04.921 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:35:04.921 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:04.921 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:35:04.921 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:35:04.921 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:04.921 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:04.922 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:04.922 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:35:04.922 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:35:04.922 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:04.922 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:04.922 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:04.922 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:04.922 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:35:04.922 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:35:04.922 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:35:04.922 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:35:04.922 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:35:04.922 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:04.922 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:04.923 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:04.923 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:35:04.923 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:35:04.923 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:04.923 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:35:04.923 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:35:04.923 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:35:04.923 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:04.923 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:04.923 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:35:04.923 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:04.923 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:35:04.924 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:35:04.924 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:35:04.924 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:04.924 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:04.924 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:35:04.924 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:04.924 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:35:04.924 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:35:04.924 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:35:04.924 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:04.925 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:04.925 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:04.925 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:35:04.925 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:35:04.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.888+0000 7f90c012b640 1 -- 172.21.15.12:0/2074723002 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f90a403e260 msgr2=0x7f90a4040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:04.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.888+0000 7f90c012b640 1 --2- 172.21.15.12:0/2074723002 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f90a403e260 0x7f90a4040720 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f90b81cdf10 tx=0x7f90b4009190 comp rx=0 tx=0).stop 2024-09-11T01:35:04.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.888+0000 7f90c012b640 10 monclient: shutdown 2024-09-11T01:35:04.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.888+0000 7f90c012b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:35:04.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.888+0000 7f90c012b640 1 -- 172.21.15.12:0/2074723002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b813d230 msgr2=0x7f90b81cce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:04.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.888+0000 7f90be6c8640 1 -- 172.21.15.12:0/2074723002 reap_dead start 2024-09-11T01:35:04.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.888+0000 7f90c012b640 1 --2- 172.21.15.12:0/2074723002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b813d230 0x7f90b81cce50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f90b0030000 tx=0x7f90b0004a40 comp rx=0 tx=0).stop 2024-09-11T01:35:04.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.888+0000 7f90c012b640 1 -- 172.21.15.12:0/2074723002 shutdown_connections 2024-09-11T01:35:04.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.888+0000 7f90c012b640 1 --2- 172.21.15.12:0/2074723002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90b813d230 0x7f90b81cce50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:04.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.888+0000 7f90c012b640 1 -- 172.21.15.12:0/2074723002 >> 172.21.15.12:0/2074723002 conn(0x7f90b80a7300 msgr2=0x7f90b80a3c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:35:04.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.888+0000 7f90c012b640 1 -- 172.21.15.12:0/2074723002 shutdown_connections 2024-09-11T01:35:04.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:04.888+0000 7f90c012b640 1 -- 172.21.15.12:0/2074723002 wait complete. 2024-09-11T01:35:04.928 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:35:04.928 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:35:04.928 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:35:05.000 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:35:05.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.028+0000 7f059e0c6640 10 monclient: get_monmap_and_config 2024-09-11T01:35:05.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.028+0000 7f059e0c6640 10 monclient: build_initial_monmap 2024-09-11T01:35:05.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.028+0000 7f059e0c6640 10 monclient: monmap: 2024-09-11T01:35:05.030 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:35:05.030 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:05.030 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:35:05.029345+0000 2024-09-11T01:35:05.030 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:35:05.029345+0000 2024-09-11T01:35:05.031 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:35:05.031 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:05.031 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:35:05.031 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:35:05.031 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:35:05.031 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:35:05.031 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:35:05.031 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:05.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.029+0000 7f059e0c6640 1 Processor -- start 2024-09-11T01:35:05.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.029+0000 7f059e0c6640 1 -- start start 2024-09-11T01:35:05.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.029+0000 7f059e0c6640 10 monclient: init 2024-09-11T01:35:05.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.029+0000 7f059e0c6640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:05.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.029+0000 7f059e0c6640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-11T01:35:05.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.029+0000 7f059e0c6640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f05980a03a0 0x7f059809f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:05.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.029+0000 7f059e0c6640 10 monclient(hunting): picked mon.noname-a con 0x7f05980a03a0 addr v2:172.21.15.12:3300/0 2024-09-11T01:35:05.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.029+0000 7f059e0c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f05980a2530 0x7f05980a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:05.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.029+0000 7f059e0c6640 10 monclient(hunting): picked mon.noname-b con 0x7f05980a2530 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:35:05.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f059e0c6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f05980a11d0 0x7f05980a1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:05.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f059e0c6640 10 monclient(hunting): picked mon.noname-c con 0x7f05980a11d0 addr v2:172.21.15.71:3300/0 2024-09-11T01:35:05.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f059e0c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:05.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f059e0c6640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f05980a29b0 con 0x7f05980a03a0 2024-09-11T01:35:05.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f059e0c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:05.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f059e0c6640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f05980a0780 con 0x7f05980a11d0 2024-09-11T01:35:05.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f059e0c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:05.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f059e0c6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f059809e510 con 0x7f05980a2530 2024-09-11T01:35:05.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f059e0c6640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:05.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f059e0c6640 10 monclient(hunting): authenticate will time out at 1111.472047s 2024-09-11T01:35:05.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f05977fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f05980a03a0 0x7f059809f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:05.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f058ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f05980a2530 0x7f05980a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:05.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f05977fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f05980a03a0 0x7f059809f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48958/0 (socket says 172.21.15.12:48958) 2024-09-11T01:35:05.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f05977fe640 1 -- 172.21.15.12:0/1438402093 learned_addr learned my addr 172.21.15.12:0/1438402093 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:35:05.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f05977fe640 10 monclient(hunting): get_auth_request con 0x7f05980a03a0 auth_method 0 2024-09-11T01:35:05.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f05977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:05.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f05977fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:05.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f05977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:05.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f058ffff640 10 monclient(hunting): get_auth_request con 0x7f05980a2530 auth_method 0 2024-09-11T01:35:05.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f058ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:05.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f058ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:05.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f058ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:05.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f0597fff640 1 --2- 172.21.15.12:0/1438402093 >> v2:172.21.15.71:3300/0 conn(0x7f05980a11d0 0x7f05980a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:05.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f0597fff640 10 monclient(hunting): get_auth_request con 0x7f05980a11d0 auth_method 0 2024-09-11T01:35:05.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f0597fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:05.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f0597fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:05.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f0597fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:05.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f05977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:05.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f05977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:05.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f05977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:05.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f058ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:05.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f058ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:05.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f058ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:05.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f0597fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:05.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f0597fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:05.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f0597fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:05.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f05977fe640 10 monclient(hunting): handle_auth_done global_id 54207 payload 274 2024-09-11T01:35:05.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f05977fe640 1 -- 172.21.15.12:0/1438402093 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f05980a2530 msgr2=0x7f05980a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:35:05.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f05977fe640 1 --2- 172.21.15.12:0/1438402093 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f05980a2530 0x7f05980a0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f05977fe640 1 -- 172.21.15.12:0/1438402093 >> v2:172.21.15.71:3300/0 conn(0x7f05980a11d0 msgr2=0x7f05980a1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:35:05.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f05977fe640 1 --2- 172.21.15.12:0/1438402093 >> v2:172.21.15.71:3300/0 conn(0x7f05980a11d0 0x7f05980a1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f05977fe640 10 monclient: _finish_hunting 0 2024-09-11T01:35:05.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f05977fe640 1 monclient: found mon.noname-a 2024-09-11T01:35:05.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f05977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:05.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f05977fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:35:05.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f05977fe640 1 -- 172.21.15.12:0/1438402093 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f059809edb0 con 0x7f05980a03a0 2024-09-11T01:35:05.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.030+0000 7f058ffff640 1 --2- 172.21.15.12:0/1438402093 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f05980a2530 0x7f05980a0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:35:05.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.033+0000 7f05977fe640 1 -- 172.21.15.12:0/1438402093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05980a03a0 msgr2=0x7f059809f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:05.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.033+0000 7f05977fe640 1 --2- 172.21.15.12:0/1438402093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05980a03a0 0x7f059809f1d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f059809e880 tx=0x7f058802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9172d652cee0cdbb server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:05.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.033+0000 7f0596ffd640 1 -- 172.21.15.12:0/1438402093 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0588037070 con 0x7f05980a03a0 2024-09-11T01:35:05.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.033+0000 7f0596ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:05.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.033+0000 7f0596ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:35:05.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.033+0000 7f0596ffd640 10 monclient: dump: 2024-09-11T01:35:05.044 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:05.044 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:05.044 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.044 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.044 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:05.045 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:05.045 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:05.045 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:05.045 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:05.045 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:05.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.033+0000 7f0596ffd640 10 monclient: _finish_auth 0 2024-09-11T01:35:05.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.033+0000 7f0596ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:35:05.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.033+0000 7f0596ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:05.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.033+0000 7f0596ffd640 1 -- 172.21.15.12:0/1438402093 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0588032640 con 0x7f05980a03a0 2024-09-11T01:35:05.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.033+0000 7f059e0c6640 5 monclient: authenticate success, global_id 54207 2024-09-11T01:35:05.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.033+0000 7f059e0c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:35:05.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.033+0000 7f0596ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:35:05.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.033+0000 7f059e0c6640 10 monclient: get_monmap_and_config success 2024-09-11T01:35:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.033+0000 7f0596ffd640 1 -- 172.21.15.12:0/1438402093 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0588032a10 con 0x7f05980a03a0 2024-09-11T01:35:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.033+0000 7f0596ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.033+0000 7f059e0c6640 4 set_mon_vals no callback set 2024-09-11T01:35:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.033+0000 7f0596ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:35:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.033+0000 7f0596ffd640 10 monclient: dump: 2024-09-11T01:35:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:05.046 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:05.047 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.034+0000 7f059e0c6640 10 monclient: shutdown 2024-09-11T01:35:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.034+0000 7f059e0c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:35:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.034+0000 7f059e0c6640 1 -- 172.21.15.12:0/1438402093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05980a03a0 msgr2=0x7f059809f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.034+0000 7f059e0c6640 1 --2- 172.21.15.12:0/1438402093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05980a03a0 0x7f059809f1d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f059809e880 tx=0x7f058802a100 comp rx=0 tx=0).stop 2024-09-11T01:35:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.034+0000 7f059e0c6640 1 -- 172.21.15.12:0/1438402093 shutdown_connections 2024-09-11T01:35:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.034+0000 7f059e0c6640 1 --2- 172.21.15.12:0/1438402093 >> v2:172.21.15.71:3300/0 conn(0x7f05980a11d0 0x7f05980a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.034+0000 7f059e0c6640 1 --2- 172.21.15.12:0/1438402093 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f05980a2530 0x7f05980a0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.034+0000 7f059e0c6640 1 --2- 172.21.15.12:0/1438402093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05980a03a0 0x7f059809f1d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.034+0000 7f059e0c6640 1 -- 172.21.15.12:0/1438402093 >> 172.21.15.12:0/1438402093 conn(0x7f05980a2f70 msgr2=0x7f05980a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:35:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.034+0000 7f059e0c6640 1 -- 172.21.15.12:0/1438402093 shutdown_connections 2024-09-11T01:35:05.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.034+0000 7f059e0c6640 1 -- 172.21.15.12:0/1438402093 wait complete. 2024-09-11T01:35:05.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.034+0000 7f059e0c6640 10 monclient: build_initial_monmap 2024-09-11T01:35:05.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.034+0000 7f059e0c6640 10 monclient: monmap: 2024-09-11T01:35:05.048 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:35:05.048 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:35:05.048 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:35:05.048 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:35:05.048 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:35:05.048 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:05.048 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:35:05.048 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:35:05.048 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:35:05.048 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:05.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f059e0c6640 1 Processor -- start 2024-09-11T01:35:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f059e0c6640 1 -- start start 2024-09-11T01:35:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f059e0c6640 10 monclient: init 2024-09-11T01:35:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f059e0c6640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f059e0c6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:35:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f059e0c6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05980a03a0 0x7f0598137d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f059e0c6640 10 monclient(hunting): picked mon.noname-b con 0x7f05980a03a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:35:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f059e0c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f05980a11d0 0x7f05981382b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f059e0c6640 10 monclient(hunting): picked mon.noname-c con 0x7f05980a11d0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:35:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f059e0c6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05980a2530 0x7f059813bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f059e0c6640 10 monclient(hunting): picked mon.noname-a con 0x7f05980a2530 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:35:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f059e0c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f059e0c6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f059809cdc0 con 0x7f05980a2530 2024-09-11T01:35:05.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f059e0c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f059e0c6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f059809cc40 con 0x7f05980a03a0 2024-09-11T01:35:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f059e0c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f059e0c6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f059809cf40 con 0x7f05980a11d0 2024-09-11T01:35:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f059e0c6640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f059e0c6640 10 monclient(hunting): authenticate will time out at 1111.477594s 2024-09-11T01:35:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f0597fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05980a2530 0x7f059813bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f05977fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05980a03a0 0x7f0598137d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f058ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f05980a11d0 0x7f05981382b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f05977fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05980a03a0 0x7f0598137d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:33674/0 (socket says 172.21.15.12:33674) 2024-09-11T01:35:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f05977fe640 1 -- 172.21.15.12:0/1322026592 learned_addr learned my addr 172.21.15.12:0/1322026592 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:35:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f05977fe640 10 monclient(hunting): get_auth_request con 0x7f05980a03a0 auth_method 0 2024-09-11T01:35:05.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f05977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f058ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f05980a11d0 0x7f05981382b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:35456/0 (socket says 172.21.15.12:35456) 2024-09-11T01:35:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f0597fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05980a2530 0x7f059813bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48964/0 (socket says 172.21.15.12:48964) 2024-09-11T01:35:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f05977fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f05977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f058ffff640 10 monclient(hunting): get_auth_request con 0x7f05980a11d0 auth_method 0 2024-09-11T01:35:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f058ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f058ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f058ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f0597fff640 10 monclient(hunting): get_auth_request con 0x7f05980a2530 auth_method 0 2024-09-11T01:35:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f0597fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f0597fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f0597fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f05977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:05.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f05977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:05.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f05977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:05.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f058ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:05.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f058ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:05.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f058ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:05.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f0597fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:05.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f0597fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:05.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f0597fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:05.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.035+0000 7f05977fe640 10 monclient(hunting): handle_auth_done global_id 34363 payload 824 2024-09-11T01:35:05.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.036+0000 7f05977fe640 1 -- 172.21.15.12:0/1322026592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f05980a11d0 msgr2=0x7f05981382b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:05.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.036+0000 7f05977fe640 1 --2- 172.21.15.12:0/1322026592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f05980a11d0 0x7f05981382b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.036+0000 7f05977fe640 1 -- 172.21.15.12:0/1322026592 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05980a2530 msgr2=0x7f059813bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:05.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.036+0000 7f05977fe640 1 --2- 172.21.15.12:0/1322026592 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05980a2530 0x7f059813bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.036+0000 7f05977fe640 10 monclient: _finish_hunting 0 2024-09-11T01:35:05.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.036+0000 7f05977fe640 1 monclient: found mon.noname-b 2024-09-11T01:35:05.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.036+0000 7f05977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:05.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.036+0000 7f05977fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-09-11T01:35:05.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.036+0000 7f05977fe640 1 -- 172.21.15.12:0/1322026592 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05981cdd00 con 0x7f05980a03a0 2024-09-11T01:35:05.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.036+0000 7f058ffff640 1 --2- 172.21.15.12:0/1322026592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f05980a11d0 0x7f05981382b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:35:05.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.036+0000 7f0597fff640 1 --2- 172.21.15.12:0/1322026592 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05980a2530 0x7f059813bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:35:05.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.038+0000 7f05977fe640 1 -- 172.21.15.12:0/1322026592 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05980a03a0 msgr2=0x7f0598137d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:05.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.038+0000 7f05977fe640 1 --2- 172.21.15.12:0/1322026592 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05980a03a0 0x7f0598137d70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f05880090b0 tx=0x7f05880046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:05.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.038+0000 7f0594ff9640 1 -- 172.21.15.12:0/1322026592 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0588041020 con 0x7f05980a03a0 2024-09-11T01:35:05.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.038+0000 7f0594ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:05.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.038+0000 7f0594ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:35:05.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.038+0000 7f0594ff9640 10 monclient: dump: 2024-09-11T01:35:05.053 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:05.053 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:05.054 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.054 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.054 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:05.054 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:05.054 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:05.054 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:05.054 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:05.054 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:05.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.038+0000 7f0594ff9640 10 monclient: _finish_auth 0 2024-09-11T01:35:05.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.038+0000 7f0594ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:35:05.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.038+0000 7f0594ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:05.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.038+0000 7f0594ff9640 1 -- 172.21.15.12:0/1322026592 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f058803d850 con 0x7f05980a03a0 2024-09-11T01:35:05.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.038+0000 7f0594ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:35:05.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.038+0000 7f0594ff9640 1 -- 172.21.15.12:0/1322026592 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0588037040 con 0x7f05980a03a0 2024-09-11T01:35:05.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.038+0000 7f0594ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:05.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.038+0000 7f0594ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:35:05.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.038+0000 7f0594ff9640 10 monclient: dump: 2024-09-11T01:35:05.055 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:05.055 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:05.055 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.055 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.055 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:05.055 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:05.055 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:05.055 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:05.055 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:05.056 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:05.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.038+0000 7f05967fc640 4 set_mon_vals no callback set 2024-09-11T01:35:05.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.039+0000 7f059e0c6640 5 monclient: authenticate success, global_id 34363 2024-09-11T01:35:05.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.039+0000 7f059e0c6640 10 monclient: _renew_subs 2024-09-11T01:35:05.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.039+0000 7f059e0c6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:35:05.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.039+0000 7f059e0c6640 1 -- 172.21.15.12:0/1322026592 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f059813d690 con 0x7f05980a03a0 2024-09-11T01:35:05.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.039+0000 7f059e0c6640 10 monclient: _renew_subs 2024-09-11T01:35:05.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.039+0000 7f059e0c6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:35:05.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.039+0000 7f059e0c6640 1 -- 172.21.15.12:0/1322026592 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f059809c540 con 0x7f05980a03a0 2024-09-11T01:35:05.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.039+0000 7f059e0c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:05.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.039+0000 7f0594ff9640 1 -- 172.21.15.12:0/1322026592 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f058803a030 con 0x7f05980a03a0 2024-09-11T01:35:05.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.039+0000 7f059e0c6640 1 --2- 172.21.15.12:0/1322026592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0598122f40 0x7f059813c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:05.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.039+0000 7f059e0c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:05.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.039+0000 7f059e0c6640 1 -- 172.21.15.12:0/1322026592 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f059809d480 con 0x7f0598122f40 2024-09-11T01:35:05.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.040+0000 7f058ffff640 1 --2- 172.21.15.12:0/1322026592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0598122f40 0x7f059813c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:05.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.040+0000 7f058ffff640 10 monclient: get_auth_request con 0x7f0598122f40 auth_method 0 2024-09-11T01:35:05.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.040+0000 7f058ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:05.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.040+0000 7f058ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:05.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.040+0000 7f058ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:05.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.040+0000 7f058ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:05.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.040+0000 7f058ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:05.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.040+0000 7f058ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:05.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.040+0000 7f0594ff9640 1 --2- 172.21.15.12:0/1322026592 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f057803e260 0x7f0578040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:05.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.040+0000 7f0594ff9640 1 -- 172.21.15.12:0/1322026592 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(199..199 src has 1..199) ==== 4767+0+0 (secure 0 0 0) 0x7f0588073160 con 0x7f05980a03a0 2024-09-11T01:35:05.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.040+0000 7f058ffff640 10 monclient(hunting): handle_auth_done global_id 64112 payload 824 2024-09-11T01:35:05.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.040+0000 7f058ffff640 1 -- 172.21.15.12:0/1322026592 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f05980a1600 con 0x7f0598122f40 2024-09-11T01:35:05.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.040+0000 7f0597fff640 1 --2- 172.21.15.12:0/1322026592 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f057803e260 0x7f0578040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:05.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.040+0000 7f0597fff640 10 monclient: get_auth_request con 0x7f057803e260 auth_method 0 2024-09-11T01:35:05.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.042+0000 7f058ffff640 1 -- 172.21.15.12:0/1322026592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0598122f40 msgr2=0x7f059813c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:05.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.042+0000 7f058ffff640 1 --2- 172.21.15.12:0/1322026592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0598122f40 0x7f059813c110 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f05980a0140 tx=0x7f058000f610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:05.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.042+0000 7f0594ff9640 1 -- 172.21.15.12:0/1322026592 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0580014650 con 0x7f0598122f40 2024-09-11T01:35:05.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.042+0000 7f0594ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:05.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.042+0000 7f0594ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:35:05.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.042+0000 7f0594ff9640 10 monclient: dump: 2024-09-11T01:35:05.058 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:05.058 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:05.058 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.059 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.059 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:05.059 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:05.059 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:05.059 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:05.059 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:05.059 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:05.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.042+0000 7f0597fff640 1 --2- 172.21.15.12:0/1322026592 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f057803e260 0x7f0578040720 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f059000e740 tx=0x7f059000b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:05.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.043+0000 7f0594ff9640 1 -- 172.21.15.12:0/1322026592 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0580014650 con 0x7f0598122f40 2024-09-11T01:35:05.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.043+0000 7f0594ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:05.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.043+0000 7f0594ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:35:05.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.043+0000 7f0594ff9640 1 -- 172.21.15.12:0/1322026592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0598122f40 msgr2=0x7f059813c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:05.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.043+0000 7f0594ff9640 1 --2- 172.21.15.12:0/1322026592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0598122f40 0x7f059813c110 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f05980a0140 tx=0x7f058000f610 comp rx=0 tx=0).stop 2024-09-11T01:35:05.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.043+0000 7f0594ff9640 1 -- 172.21.15.12:0/1322026592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0598122f40 msgr2=0x7f059813c110 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:35:05.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.043+0000 7f0594ff9640 1 --2- 172.21.15.12:0/1322026592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0598122f40 0x7f059813c110 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f05980a0140 tx=0x7f058000f610 comp rx=0 tx=0).stop 2024-09-11T01:35:05.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.049+0000 7f059e0c6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:35:05.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.049+0000 7f059e0c6640 1 --2- 172.21.15.12:0/1322026592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f059813c650 0x7f0598071440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:05.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.049+0000 7f059e0c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:05.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.049+0000 7f059e0c6640 1 -- 172.21.15.12:0/1322026592 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f05981246a0 con 0x7f059813c650 2024-09-11T01:35:05.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.049+0000 7f058ffff640 1 --2- 172.21.15.12:0/1322026592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f059813c650 0x7f0598071440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:05.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.049+0000 7f058ffff640 10 monclient: get_auth_request con 0x7f059813c650 auth_method 0 2024-09-11T01:35:05.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.049+0000 7f058ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:05.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.049+0000 7f058ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:05.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.049+0000 7f058ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:05.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.049+0000 7f058ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:05.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.049+0000 7f058ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:05.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.049+0000 7f058ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:05.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.049+0000 7f058ffff640 10 monclient(hunting): handle_auth_done global_id 64115 payload 824 2024-09-11T01:35:05.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.049+0000 7f058ffff640 1 -- 172.21.15.12:0/1322026592 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f059806d480 con 0x7f059813c650 2024-09-11T01:35:05.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.052+0000 7f058ffff640 1 -- 172.21.15.12:0/1322026592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f059813c650 msgr2=0x7f0598071440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:05.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.052+0000 7f058ffff640 1 --2- 172.21.15.12:0/1322026592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f059813c650 0x7f0598071440 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f059806cfd0 tx=0x7f05800046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:05.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.052+0000 7f0594ff9640 1 -- 172.21.15.12:0/1322026592 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0580014e60 con 0x7f059813c650 2024-09-11T01:35:05.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.052+0000 7f0594ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:05.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.052+0000 7f0594ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:35:05.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.052+0000 7f0594ff9640 10 monclient: dump: 2024-09-11T01:35:05.063 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:05.063 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:05.063 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.063 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.063 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:05.064 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:05.064 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:05.064 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:05.064 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:05.064 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:05.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.052+0000 7f0594ff9640 1 -- 172.21.15.12:0/1322026592 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4540 (secure 0 0 0) 0x7f058001d400 con 0x7f059813c650 2024-09-11T01:35:05.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.052+0000 7f0594ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:35:05.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.052+0000 7f0594ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:35:05.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.052+0000 7f0594ff9640 1 -- 172.21.15.12:0/1322026592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f059813c650 msgr2=0x7f0598071440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:05.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.052+0000 7f0594ff9640 1 --2- 172.21.15.12:0/1322026592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f059813c650 0x7f0598071440 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f059806cfd0 tx=0x7f05800046c0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.052+0000 7f0594ff9640 1 -- 172.21.15.12:0/1322026592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f059813c650 msgr2=0x7f0598071440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:35:05.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.052+0000 7f0594ff9640 1 --2- 172.21.15.12:0/1322026592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f059813c650 0x7f0598071440 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f059806cfd0 tx=0x7f05800046c0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.065 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:35:05.065 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:35:05.065 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:35:05.065 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:35:05.065 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 64, 2024-09-11T01:35:05.065 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 23144, 2024-09-11T01:35:05.065 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:35:05.065 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:35:05.065 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:35:05.065 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:35:05.065 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:05.065 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:35:05.066 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:35:05.066 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:35:05.066 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:35:05.066 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:35:05.066 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:35:05.066 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:35:05.066 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:35:05.066 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:35:05.066 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:35:05.066 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:35:05.066 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:35:05.066 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:35:05.066 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:35:05.066 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:35:05.067 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:05.067 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:35:05.067 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:35:05.067 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:35:05.067 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:35:05.067 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:35:05.067 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:35:05.067 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:35:05.067 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:35:05.067 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:35:05.067 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:35:05.067 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:35:05.067 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:35:05.067 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:35:05.068 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:05.068 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:05.068 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:35:05.068 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:35:05.068 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:35:05.068 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:35:05.068 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:35:05.068 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:35:05.068 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:35:05.068 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:35:05.068 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:35:05.068 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:35:05.068 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:35:05.069 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:35:05.069 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:35:05.069 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:35:05.069 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:35:05.069 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:35:05.069 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:35:05.069 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:35:05.069 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:35:05.069 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:35:05.069 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:35:05.069 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:35:05.069 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:35:05.069 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:35:05.069 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:35:05.070 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:35:05.070 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:35:05.070 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:35:05.070 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:05.070 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:35:05.070 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:05.070 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:35:05.070 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:05.070 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:35:05.070 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:35:05.070 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:35:05.070 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:35:05.070 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:05.070 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:35:05.070 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:35:05.070 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:05.071 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:05.071 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:05.071 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:35:05.071 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:35:05.071 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:05.071 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:05.071 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:05.071 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:05.071 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:35:05.071 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:35:05.071 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:35:05.071 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:35:05.071 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:35:05.071 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:05.071 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:05.072 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:35:05.072 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:35:05.072 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:35:05.072 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:35:05.072 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:05.072 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:35:05.072 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:35:05.072 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:05.072 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:05.072 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:05.072 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:35:05.072 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:35:05.072 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:05.072 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:05.073 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:05.073 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:05.073 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:35:05.073 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:35:05.073 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:35:05.073 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:35:05.073 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:35:05.073 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:05.073 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:05.073 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:35:05.073 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:35:05.073 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:35:05.073 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:35:05.073 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:05.074 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:35:05.074 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:35:05.074 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:05.074 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:05.074 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:05.074 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:35:05.074 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:35:05.074 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:05.074 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:05.074 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:05.074 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:05.074 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:35:05.074 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:35:05.074 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:35:05.074 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:35:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:35:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:35:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:35:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:35:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:35:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:35:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:35:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:05.076 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:35:05.076 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:35:05.076 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:35:05.076 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:05.076 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:05.076 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:05.076 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:35:05.076 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:35:05.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.052+0000 7f059e0c6640 1 -- 172.21.15.12:0/1322026592 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f057803e260 msgr2=0x7f0578040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:05.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.052+0000 7f059e0c6640 1 --2- 172.21.15.12:0/1322026592 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f057803e260 0x7f0578040720 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f059000e740 tx=0x7f059000b040 comp rx=0 tx=0).stop 2024-09-11T01:35:05.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.052+0000 7f059e0c6640 10 monclient: shutdown 2024-09-11T01:35:05.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.052+0000 7f059e0c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:35:05.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.052+0000 7f059e0c6640 1 -- 172.21.15.12:0/1322026592 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05980a03a0 msgr2=0x7f0598137d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.052+0000 7f059e0c6640 1 --2- 172.21.15.12:0/1322026592 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05980a03a0 0x7f0598137d70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f05880090b0 tx=0x7f05880046c0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.052+0000 7f0597fff640 1 -- 172.21.15.12:0/1322026592 reap_dead start 2024-09-11T01:35:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.052+0000 7f0597fff640 1 -- 172.21.15.12:0/1322026592 reap_dead start 2024-09-11T01:35:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.052+0000 7f059e0c6640 1 -- 172.21.15.12:0/1322026592 shutdown_connections 2024-09-11T01:35:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.052+0000 7f059e0c6640 1 -- 172.21.15.12:0/1322026592 >> 172.21.15.12:0/1322026592 conn(0x7f05980a2f70 msgr2=0x7f05980a57f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:35:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.052+0000 7f059e0c6640 1 -- 172.21.15.12:0/1322026592 shutdown_connections 2024-09-11T01:35:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.052+0000 7f059e0c6640 1 -- 172.21.15.12:0/1322026592 wait complete. 2024-09-11T01:35:05.079 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:35:05.079 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:35:05.080 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:35:05.080 INFO:tasks.rados.rados.0.smithi071.stdout:847: finishing write tid 1 to smithi07134777-36 2024-09-11T01:35:05.080 INFO:tasks.rados.rados.0.smithi071.stdout:847: finishing write tid 2 to smithi07134777-36 2024-09-11T01:35:05.080 INFO:tasks.rados.rados.0.smithi071.stdout:847: finishing write tid 3 to smithi07134777-36 2024-09-11T01:35:05.081 INFO:tasks.rados.rados.0.smithi071.stdout:844: finishing copy_from racing read to smithi07134777-16 2024-09-11T01:35:05.081 INFO:tasks.rados.rados.0.smithi071.stdout:847: finishing write tid 4 to smithi07134777-36 2024-09-11T01:35:05.081 INFO:tasks.rados.rados.0.smithi071.stdout:847: finishing write tid 5 to smithi07134777-36 2024-09-11T01:35:05.081 INFO:tasks.rados.rados.0.smithi071.stdout:847: oid 36 updating version 0 to 137 2024-09-11T01:35:05.081 INFO:tasks.rados.rados.0.smithi071.stdout:847: oid 36 version 137 is already newer than 134 2024-09-11T01:35:05.081 INFO:tasks.rados.rados.0.smithi071.stdout:847: oid 36 version 137 is already newer than 136 2024-09-11T01:35:05.081 INFO:tasks.rados.rados.0.smithi071.stdout:847: oid 36 version 137 is already newer than 135 2024-09-11T01:35:05.081 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 137 (ObjNum 318 snap 91 seq_num 318) dirty exists 2024-09-11T01:35:05.081 INFO:tasks.rados.rados.0.smithi071.stdout:847: left oid 36 (ObjNum 318 snap 91 seq_num 318) 2024-09-11T01:35:05.081 INFO:tasks.rados.rados.0.smithi071.stdout:844: finishing copy_from to smithi07134777-16 2024-09-11T01:35:05.081 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 129 (ObjNum 226 snap 55 seq_num 226) dirty exists 2024-09-11T01:35:05.081 INFO:tasks.rados.rados.0.smithi071.stdout:848: finishing write tid 1 to smithi07134777-18 2024-09-11T01:35:05.081 INFO:tasks.rados.rados.0.smithi071.stdout:848: finishing write tid 2 to smithi07134777-18 2024-09-11T01:35:05.081 INFO:tasks.rados.rados.0.smithi071.stdout:848: finishing write tid 3 to smithi07134777-18 2024-09-11T01:35:05.081 INFO:tasks.rados.rados.0.smithi071.stdout:844: done (5 left) 2024-09-11T01:35:05.082 INFO:tasks.rados.rados.0.smithi071.stdout:845: done (4 left) 2024-09-11T01:35:05.082 INFO:tasks.rados.rados.0.smithi071.stdout:847: done (3 left) 2024-09-11T01:35:05.082 INFO:tasks.rados.rados.0.smithi071.stdout:852: snap_create 2024-09-11T01:35:05.082 INFO:tasks.rados.rados.0.smithi071.stdout:848: finishing write tid 4 to smithi07134777-18 2024-09-11T01:35:05.082 INFO:tasks.rados.rados.0.smithi071.stdout:848: finishing write tid 5 to smithi07134777-18 2024-09-11T01:35:05.082 INFO:tasks.rados.rados.0.smithi071.stdout:848: oid 18 updating version 0 to 292 2024-09-11T01:35:05.082 INFO:tasks.rados.rados.0.smithi071.stdout:848: oid 18 version 292 is already newer than 290 2024-09-11T01:35:05.082 INFO:tasks.rados.rados.0.smithi071.stdout:848: oid 18 version 292 is already newer than 289 2024-09-11T01:35:05.082 INFO:tasks.rados.rados.0.smithi071.stdout:848: oid 18 version 292 is already newer than 291 2024-09-11T01:35:05.082 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 292 (ObjNum 319 snap 91 seq_num 319) dirty exists 2024-09-11T01:35:05.082 INFO:tasks.rados.rados.0.smithi071.stdout:848: left oid 18 (ObjNum 319 snap 91 seq_num 319) 2024-09-11T01:35:05.082 INFO:tasks.rados.rados.0.smithi071.stdout:850: finishing write tid 1 to smithi07134777-12 2024-09-11T01:35:05.082 INFO:tasks.rados.rados.0.smithi071.stdout:850: finishing write tid 2 to smithi07134777-12 2024-09-11T01:35:05.083 INFO:tasks.rados.rados.0.smithi071.stdout:851: finishing rollback tid 0 to smithi07134777-3 2024-09-11T01:35:05.083 INFO:tasks.rados.rados.0.smithi071.stdout:851: finishing rollback tid 1 to smithi07134777-3 2024-09-11T01:35:05.083 INFO:tasks.rados.rados.0.smithi071.stdout:850: finishing write tid 3 to smithi07134777-12 2024-09-11T01:35:05.083 INFO:tasks.rados.rados.0.smithi071.stdout:850: finishing write tid 4 to smithi07134777-12 2024-09-11T01:35:05.083 INFO:tasks.rados.rados.0.smithi071.stdout:850: finishing write tid 5 to smithi07134777-12 2024-09-11T01:35:05.083 INFO:tasks.rados.rados.0.smithi071.stdout:850: oid 12 updating version 0 to 301 2024-09-11T01:35:05.083 INFO:tasks.rados.rados.0.smithi071.stdout:850: oid 12 version 301 is already newer than 300 2024-09-11T01:35:05.083 INFO:tasks.rados.rados.0.smithi071.stdout:850: oid 12 version 301 is already newer than 296 2024-09-11T01:35:05.083 INFO:tasks.rados.rados.0.smithi071.stdout:850: oid 12 version 301 is already newer than 299 2024-09-11T01:35:05.083 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 301 (ObjNum 320 snap 91 seq_num 320) dirty exists 2024-09-11T01:35:05.083 INFO:tasks.rados.rados.0.smithi071.stdout:850: left oid 12 (ObjNum 320 snap 91 seq_num 320) 2024-09-11T01:35:05.083 INFO:tasks.rados.rados.0.smithi071.stdout:851: finishing rollback tid 2 to smithi07134777-3 2024-09-11T01:35:05.083 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 231 (ObjNum 200 snap 46 seq_num 200) dirty exists 2024-09-11T01:35:05.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.184+0000 7fd399c6e640 10 monclient: get_monmap_and_config 2024-09-11T01:35:05.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.184+0000 7fd399c6e640 10 monclient: build_initial_monmap 2024-09-11T01:35:05.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.184+0000 7fd399c6e640 10 monclient: monmap: 2024-09-11T01:35:05.185 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:35:05.186 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:05.186 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:35:05.184758+0000 2024-09-11T01:35:05.186 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:35:05.184758+0000 2024-09-11T01:35:05.186 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:35:05.186 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:05.186 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:35:05.186 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:35:05.186 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:35:05.186 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:35:05.186 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:35:05.186 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:05.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.184+0000 7fd399c6e640 1 Processor -- start 2024-09-11T01:35:05.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.184+0000 7fd399c6e640 1 -- start start 2024-09-11T01:35:05.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.184+0000 7fd399c6e640 10 monclient: init 2024-09-11T01:35:05.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd399c6e640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:05.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd399c6e640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-11T01:35:05.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd399c6e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd3941381c0 0x7fd3941385a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:05.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd399c6e640 10 monclient(hunting): picked mon.noname-b con 0x7fd3941381c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:35:05.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd399c6e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd3940a6aa0 0x7fd39409cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:05.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd399c6e640 10 monclient(hunting): picked mon.noname-c con 0x7fd3940a6aa0 addr v2:172.21.15.71:3300/0 2024-09-11T01:35:05.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd399c6e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd3940a4f20 addr v1:172.21.15.71:6789/0 2024-09-11T01:35:05.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd399c6e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd39409c310 con 0x7fd3940a4f20 2024-09-11T01:35:05.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd399c6e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:05.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd399c6e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fd39406fc90 con 0x7fd3940a6aa0 2024-09-11T01:35:05.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd399c6e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:05.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd399c6e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd39409c5f0 con 0x7fd3941381c0 2024-09-11T01:35:05.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd399c6e640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:05.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd399c6e640 10 monclient(hunting): authenticate will time out at 1111.627358s 2024-09-11T01:35:05.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd3937fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd3941381c0 0x7fd3941385a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:05.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd3937fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd3941381c0 0x7fd3941385a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:35484/0 (socket says 172.21.15.12:35484) 2024-09-11T01:35:05.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd3937fe640 1 -- 172.21.15.12:0/1782281825 learned_addr learned my addr 172.21.15.12:0/1782281825 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:35:05.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd3937fe640 10 monclient(hunting): get_auth_request con 0x7fd3941381c0 auth_method 0 2024-09-11T01:35:05.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd3937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:05.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd3937fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:05.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd3937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:05.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd392ffd640 1 --2- 172.21.15.12:0/1782281825 >> v2:172.21.15.71:3300/0 conn(0x7fd3940a6aa0 0x7fd39409cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:05.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd3937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:05.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd3937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:05.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd3937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:05.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd392ffd640 10 monclient(hunting): get_auth_request con 0x7fd3940a6aa0 auth_method 0 2024-09-11T01:35:05.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd392ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:05.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd392ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:05.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:05.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd392ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:05.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd392ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:05.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd392ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:05.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd3937fe640 10 monclient(hunting): handle_auth_done global_id 64118 payload 274 2024-09-11T01:35:05.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd3937fe640 1 -- 172.21.15.12:0/1782281825 >> v2:172.21.15.71:3300/0 conn(0x7fd3940a6aa0 msgr2=0x7fd39409cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:35:05.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd3937fe640 1 --2- 172.21.15.12:0/1782281825 >> v2:172.21.15.71:3300/0 conn(0x7fd3940a6aa0 0x7fd39409cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd3937fe640 1 -- 172.21.15.12:0/1782281825 >> v1:172.21.15.71:6789/0 conn(0x7fd3940a4f20 legacy=0x7fd3940a5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:35:05.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd3937fe640 10 monclient: _finish_hunting 0 2024-09-11T01:35:05.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd3937fe640 1 monclient: found mon.noname-b 2024-09-11T01:35:05.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd3937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:05.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd3937fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:35:05.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.185+0000 7fd3937fe640 1 -- 172.21.15.12:0/1782281825 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd39409c8d0 con 0x7fd3941381c0 2024-09-11T01:35:05.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.188+0000 7fd3937fe640 1 -- 172.21.15.12:0/1782281825 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd3941381c0 msgr2=0x7fd3941385a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:05.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.188+0000 7fd3937fe640 1 --2- 172.21.15.12:0/1782281825 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd3941381c0 0x7fd3941385a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd384009870 tx=0x7fd38402a120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b87225c53a55c5fa server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:05.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.188+0000 7fd3927fc640 1 -- 172.21.15.12:0/1782281825 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd384037070 con 0x7fd3941381c0 2024-09-11T01:35:05.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.188+0000 7fd3927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:05.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.188+0000 7fd3927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:35:05.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.188+0000 7fd3927fc640 10 monclient: dump: 2024-09-11T01:35:05.200 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:05.200 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:05.200 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.200 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.200 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:05.200 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:05.200 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:05.200 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:05.200 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:05.200 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:05.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.188+0000 7fd3927fc640 10 monclient: _finish_auth 0 2024-09-11T01:35:05.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.188+0000 7fd3927fc640 10 monclient: _check_auth_tickets 2024-09-11T01:35:05.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.188+0000 7fd3927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:05.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.188+0000 7fd3927fc640 1 -- 172.21.15.12:0/1782281825 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd384032640 con 0x7fd3941381c0 2024-09-11T01:35:05.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.188+0000 7fd399c6e640 5 monclient: authenticate success, global_id 64118 2024-09-11T01:35:05.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.188+0000 7fd399c6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:35:05.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.188+0000 7fd3927fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:35:05.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.188+0000 7fd3927fc640 1 -- 172.21.15.12:0/1782281825 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd384032a10 con 0x7fd3941381c0 2024-09-11T01:35:05.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.188+0000 7fd399c6e640 10 monclient: get_monmap_and_config success 2024-09-11T01:35:05.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.188+0000 7fd399c6e640 4 set_mon_vals no callback set 2024-09-11T01:35:05.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.188+0000 7fd3927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:05.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.188+0000 7fd3927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:35:05.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.188+0000 7fd3927fc640 10 monclient: dump: 2024-09-11T01:35:05.202 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:05.202 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:05.202 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.202 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.202 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:05.202 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:05.202 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:05.202 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:05.202 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:05.202 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:05.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.189+0000 7fd399c6e640 10 monclient: shutdown 2024-09-11T01:35:05.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.189+0000 7fd399c6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:35:05.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.189+0000 7fd399c6e640 1 -- 172.21.15.12:0/1782281825 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd3941381c0 msgr2=0x7fd3941385a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:05.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.189+0000 7fd399c6e640 1 --2- 172.21.15.12:0/1782281825 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd3941381c0 0x7fd3941385a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd384009870 tx=0x7fd38402a120 comp rx=0 tx=0).stop 2024-09-11T01:35:05.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.189+0000 7fd399c6e640 1 -- 172.21.15.12:0/1782281825 shutdown_connections 2024-09-11T01:35:05.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.189+0000 7fd399c6e640 1 --2- 172.21.15.12:0/1782281825 >> v2:172.21.15.71:3300/0 conn(0x7fd3940a6aa0 0x7fd39409cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.189+0000 7fd399c6e640 1 --2- 172.21.15.12:0/1782281825 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd3941381c0 0x7fd3941385a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.189+0000 7fd399c6e640 1 -- 172.21.15.12:0/1782281825 >> 172.21.15.12:0/1782281825 conn(0x7fd3940a7440 msgr2=0x7fd3940a7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:35:05.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.189+0000 7fd399c6e640 1 -- 172.21.15.12:0/1782281825 shutdown_connections 2024-09-11T01:35:05.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.189+0000 7fd399c6e640 1 -- 172.21.15.12:0/1782281825 wait complete. 2024-09-11T01:35:05.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.190+0000 7fd399c6e640 10 monclient: build_initial_monmap 2024-09-11T01:35:05.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.190+0000 7fd399c6e640 10 monclient: monmap: 2024-09-11T01:35:05.203 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:35:05.203 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:35:05.203 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:35:05.203 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:35:05.204 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:35:05.204 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:05.204 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:35:05.204 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:35:05.204 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:35:05.204 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:05.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.190+0000 7fd399c6e640 1 Processor -- start 2024-09-11T01:35:05.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.190+0000 7fd399c6e640 1 -- start start 2024-09-11T01:35:05.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.190+0000 7fd399c6e640 10 monclient: init 2024-09-11T01:35:05.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd399c6e640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:05.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd399c6e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:35:05.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd399c6e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd3940a4f20 0x7fd394135db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:05.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd399c6e640 10 monclient(hunting): picked mon.noname-c con 0x7fd3940a4f20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:35:05.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd399c6e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3940a6aa0 0x7fd3941362f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:05.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd399c6e640 10 monclient(hunting): picked mon.noname-b con 0x7fd3940a6aa0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:35:05.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd399c6e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3941381c0 0x7fd394123990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:05.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd399c6e640 10 monclient(hunting): picked mon.noname-a con 0x7fd3941381c0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:35:05.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd399c6e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:05.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd399c6e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd39409d690 con 0x7fd3941381c0 2024-09-11T01:35:05.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd399c6e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:05.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd399c6e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd39409d510 con 0x7fd3940a6aa0 2024-09-11T01:35:05.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd399c6e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:05.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd399c6e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd3940a5aa0 con 0x7fd3940a4f20 2024-09-11T01:35:05.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd399c6e640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:05.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd399c6e640 10 monclient(hunting): authenticate will time out at 1111.633122s 2024-09-11T01:35:05.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd3937fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd3940a4f20 0x7fd394135db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:05.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd3937fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd3940a4f20 0x7fd394135db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:35494/0 (socket says 172.21.15.12:35494) 2024-09-11T01:35:05.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd3937fe640 1 -- 172.21.15.12:0/3429246907 learned_addr learned my addr 172.21.15.12:0/3429246907 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:35:05.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd3937fe640 10 monclient(hunting): get_auth_request con 0x7fd3940a4f20 auth_method 0 2024-09-11T01:35:05.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd3937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:05.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd3937fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:05.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd3937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:05.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd393fff640 1 --2- 172.21.15.12:0/3429246907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3941381c0 0x7fd394123990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:05.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd393fff640 10 monclient(hunting): get_auth_request con 0x7fd3941381c0 auth_method 0 2024-09-11T01:35:05.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd393fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:05.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd393fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:05.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd393fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:05.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd3937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:05.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd3937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:05.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd3937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:05.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd393fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:05.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd393fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:05.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd393fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:05.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd392ffd640 1 --2- 172.21.15.12:0/3429246907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3940a6aa0 0x7fd3941362f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:05.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd392ffd640 10 monclient(hunting): get_auth_request con 0x7fd3940a6aa0 auth_method 0 2024-09-11T01:35:05.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd392ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:05.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd392ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:05.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:05.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd392ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:05.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd392ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:05.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd392ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:05.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd3937fe640 10 monclient(hunting): handle_auth_done global_id 64121 payload 824 2024-09-11T01:35:05.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd3937fe640 1 -- 172.21.15.12:0/3429246907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3940a6aa0 msgr2=0x7fd3941362f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:05.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd3937fe640 1 --2- 172.21.15.12:0/3429246907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3940a6aa0 0x7fd3941362f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd3937fe640 1 -- 172.21.15.12:0/3429246907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3941381c0 msgr2=0x7fd394123990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:05.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd3937fe640 1 --2- 172.21.15.12:0/3429246907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3941381c0 0x7fd394123990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd3937fe640 10 monclient: _finish_hunting 0 2024-09-11T01:35:05.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd3937fe640 1 monclient: found mon.noname-c 2024-09-11T01:35:05.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd3937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:05.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd3937fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:35:05.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.191+0000 7fd3937fe640 1 -- 172.21.15.12:0/3429246907 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3941cdbb0 con 0x7fd3940a4f20 2024-09-11T01:35:05.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.193+0000 7fd3937fe640 1 -- 172.21.15.12:0/3429246907 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd3940a4f20 msgr2=0x7fd394135db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:05.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.193+0000 7fd3937fe640 1 --2- 172.21.15.12:0/3429246907 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd3940a4f20 0x7fd394135db0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd384000c00 tx=0x7fd3840046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:05.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.193+0000 7fd37bfff640 1 -- 172.21.15.12:0/3429246907 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd384041020 con 0x7fd3940a4f20 2024-09-11T01:35:05.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.193+0000 7fd37bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:05.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.193+0000 7fd37bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:35:05.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.193+0000 7fd37bfff640 10 monclient: dump: 2024-09-11T01:35:05.209 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:05.209 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:05.209 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.209 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.209 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:05.209 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:05.209 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:05.209 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:05.210 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:05.210 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:05.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.194+0000 7fd37bfff640 10 monclient: _finish_auth 0 2024-09-11T01:35:05.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.194+0000 7fd37bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:35:05.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.194+0000 7fd37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:05.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.194+0000 7fd37bfff640 1 -- 172.21.15.12:0/3429246907 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd38403d830 con 0x7fd3940a4f20 2024-09-11T01:35:05.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.194+0000 7fd399c6e640 5 monclient: authenticate success, global_id 64121 2024-09-11T01:35:05.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.194+0000 7fd399c6e640 10 monclient: _renew_subs 2024-09-11T01:35:05.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.194+0000 7fd399c6e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:35:05.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.194+0000 7fd399c6e640 1 -- 172.21.15.12:0/3429246907 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3941277b0 con 0x7fd3940a4f20 2024-09-11T01:35:05.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.194+0000 7fd37bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:35:05.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.194+0000 7fd37bfff640 1 -- 172.21.15.12:0/3429246907 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd384037040 con 0x7fd3940a4f20 2024-09-11T01:35:05.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.194+0000 7fd37bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:05.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.194+0000 7fd37bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:35:05.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.194+0000 7fd37bfff640 10 monclient: dump: 2024-09-11T01:35:05.211 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:05.211 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:05.211 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.211 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.211 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:05.211 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:05.211 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:05.211 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:05.211 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:05.211 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:05.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.194+0000 7fd399c6e640 10 monclient: _renew_subs 2024-09-11T01:35:05.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.194+0000 7fd399c6e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:35:05.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.194+0000 7fd399c6e640 1 -- 172.21.15.12:0/3429246907 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd394136830 con 0x7fd3940a4f20 2024-09-11T01:35:05.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.194+0000 7fd3917fa640 4 set_mon_vals no callback set 2024-09-11T01:35:05.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.194+0000 7fd399c6e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:05.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.194+0000 7fd399c6e640 1 --2- 172.21.15.12:0/3429246907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3941ccf00 0x7fd394123ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:05.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.194+0000 7fd399c6e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:05.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.194+0000 7fd399c6e640 1 -- 172.21.15.12:0/3429246907 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd3941263c0 con 0x7fd3941ccf00 2024-09-11T01:35:05.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.195+0000 7fd392ffd640 1 --2- 172.21.15.12:0/3429246907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3941ccf00 0x7fd394123ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:05.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.195+0000 7fd37bfff640 1 -- 172.21.15.12:0/3429246907 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fd38403a030 con 0x7fd3940a4f20 2024-09-11T01:35:05.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.195+0000 7fd392ffd640 10 monclient: get_auth_request con 0x7fd3941ccf00 auth_method 0 2024-09-11T01:35:05.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.195+0000 7fd392ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:05.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.195+0000 7fd392ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:05.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.195+0000 7fd392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:05.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.195+0000 7fd392ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:05.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.195+0000 7fd392ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:05.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.195+0000 7fd392ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:05.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.195+0000 7fd37bfff640 1 --2- 172.21.15.12:0/3429246907 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd38003e210 0x7fd3800406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:05.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.195+0000 7fd392ffd640 10 monclient(hunting): handle_auth_done global_id 34372 payload 824 2024-09-11T01:35:05.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.195+0000 7fd37bfff640 1 -- 172.21.15.12:0/3429246907 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(199..199 src has 1..199) ==== 4767+0+0 (secure 0 0 0) 0x7fd384073230 con 0x7fd3940a4f20 2024-09-11T01:35:05.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.195+0000 7fd392ffd640 1 -- 172.21.15.12:0/3429246907 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd3941270a0 con 0x7fd3941ccf00 2024-09-11T01:35:05.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.195+0000 7fd393fff640 1 --2- 172.21.15.12:0/3429246907 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd38003e210 0x7fd3800406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:05.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.195+0000 7fd393fff640 10 monclient: get_auth_request con 0x7fd38003e210 auth_method 0 2024-09-11T01:35:05.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.197+0000 7fd392ffd640 1 -- 172.21.15.12:0/3429246907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3941ccf00 msgr2=0x7fd394123ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:05.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.197+0000 7fd392ffd640 1 --2- 172.21.15.12:0/3429246907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3941ccf00 0x7fd394123ed0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd394126a60 tx=0x7fd388011d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:05.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.197+0000 7fd37bfff640 1 -- 172.21.15.12:0/3429246907 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd38800fe20 con 0x7fd3941ccf00 2024-09-11T01:35:05.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.197+0000 7fd37bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:05.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.197+0000 7fd37bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:35:05.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.197+0000 7fd37bfff640 10 monclient: dump: 2024-09-11T01:35:05.214 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:05.214 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:05.214 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.214 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.214 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:05.214 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:05.214 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:05.214 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:05.214 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:05.214 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:05.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.198+0000 7fd393fff640 1 --2- 172.21.15.12:0/3429246907 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd38003e210 0x7fd3800406d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd394125880 tx=0x7fd38c003c60 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:05.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.198+0000 7fd37bfff640 1 -- 172.21.15.12:0/3429246907 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd38800fe20 con 0x7fd3941ccf00 2024-09-11T01:35:05.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.198+0000 7fd37bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:05.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.198+0000 7fd37bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:35:05.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.198+0000 7fd37bfff640 1 -- 172.21.15.12:0/3429246907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3941ccf00 msgr2=0x7fd394123ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:05.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.198+0000 7fd37bfff640 1 --2- 172.21.15.12:0/3429246907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3941ccf00 0x7fd394123ed0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd394126a60 tx=0x7fd388011d40 comp rx=0 tx=0).stop 2024-09-11T01:35:05.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.198+0000 7fd37bfff640 1 -- 172.21.15.12:0/3429246907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3941ccf00 msgr2=0x7fd394123ed0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:35:05.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.198+0000 7fd37bfff640 1 --2- 172.21.15.12:0/3429246907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3941ccf00 0x7fd394123ed0 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd394126a60 tx=0x7fd388011d40 comp rx=0 tx=0).stop 2024-09-11T01:35:05.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.204+0000 7fd399c6e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:35:05.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.204+0000 7fd399c6e640 1 --2- 172.21.15.12:0/3429246907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd394124410 0x7fd3940a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:05.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.204+0000 7fd399c6e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:05.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.204+0000 7fd399c6e640 1 -- 172.21.15.12:0/3429246907 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd3941247f0 con 0x7fd394124410 2024-09-11T01:35:05.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.204+0000 7fd392ffd640 1 --2- 172.21.15.12:0/3429246907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd394124410 0x7fd3940a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:05.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.204+0000 7fd392ffd640 10 monclient: get_auth_request con 0x7fd394124410 auth_method 0 2024-09-11T01:35:05.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.204+0000 7fd392ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:05.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.204+0000 7fd392ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:05.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.204+0000 7fd392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:05.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.204+0000 7fd392ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:05.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.204+0000 7fd392ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:05.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.204+0000 7fd392ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:05.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.204+0000 7fd392ffd640 10 monclient(hunting): handle_auth_done global_id 34375 payload 824 2024-09-11T01:35:05.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.204+0000 7fd392ffd640 1 -- 172.21.15.12:0/3429246907 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd39409cfd0 con 0x7fd394124410 2024-09-11T01:35:05.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.207+0000 7fd392ffd640 1 -- 172.21.15.12:0/3429246907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd394124410 msgr2=0x7fd3940a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:05.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.207+0000 7fd392ffd640 1 --2- 172.21.15.12:0/3429246907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd394124410 0x7fd3940a18f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd38801b000 tx=0x7fd3880046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:05.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.207+0000 7fd37bfff640 1 -- 172.21.15.12:0/3429246907 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd38802b020 con 0x7fd394124410 2024-09-11T01:35:05.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.207+0000 7fd37bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:05.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.207+0000 7fd37bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:35:05.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.207+0000 7fd37bfff640 10 monclient: dump: 2024-09-11T01:35:05.218 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:05.218 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:05.218 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.218 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.218 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:05.218 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:05.218 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:05.219 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:05.219 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:05.219 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:05.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.207+0000 7fd37bfff640 1 -- 172.21.15.12:0/3429246907 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7fd38801c3a0 con 0x7fd394124410 2024-09-11T01:35:05.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.207+0000 7fd37bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:35:05.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.207+0000 7fd37bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:35:05.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.207+0000 7fd37bfff640 1 -- 172.21.15.12:0/3429246907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd394124410 msgr2=0x7fd3940a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:05.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.207+0000 7fd37bfff640 1 --2- 172.21.15.12:0/3429246907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd394124410 0x7fd3940a18f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd38801b000 tx=0x7fd3880046c0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.207+0000 7fd37bfff640 1 -- 172.21.15.12:0/3429246907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd394124410 msgr2=0x7fd3940a18f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:35:05.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.207+0000 7fd37bfff640 1 --2- 172.21.15.12:0/3429246907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd394124410 0x7fd3940a18f0 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd38801b000 tx=0x7fd3880046c0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.219 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:35:05.220 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:35:05.220 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:35:05.220 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:35:05.220 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 64, 2024-09-11T01:35:05.220 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 198421, 2024-09-11T01:35:05.220 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:35:05.220 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:35:05.220 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:35:05.220 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:35:05.220 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:05.220 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:35:05.221 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:35:05.221 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:35:05.221 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:35:05.221 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:35:05.221 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:35:05.221 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:35:05.221 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:35:05.221 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:35:05.221 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:35:05.221 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:35:05.221 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:35:05.221 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:35:05.222 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:35:05.222 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:35:05.222 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:05.222 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:35:05.222 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:35:05.222 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:35:05.222 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:35:05.222 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:35:05.222 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:35:05.222 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:35:05.222 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:35:05.222 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:35:05.222 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:35:05.223 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:35:05.223 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:35:05.223 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:35:05.223 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:05.223 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:05.223 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:35:05.223 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:35:05.223 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:35:05.223 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:35:05.223 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:35:05.223 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:35:05.223 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:35:05.223 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:35:05.223 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:35:05.224 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:35:05.224 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:35:05.224 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:35:05.224 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:35:05.224 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:35:05.224 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:35:05.224 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:35:05.224 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:35:05.224 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:35:05.224 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:35:05.224 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:35:05.224 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:35:05.224 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:35:05.225 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:35:05.225 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:35:05.225 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:35:05.225 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:35:05.225 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:35:05.225 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:35:05.225 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:05.225 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:35:05.225 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:05.225 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:35:05.225 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:05.225 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:35:05.225 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:35:05.225 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:35:05.225 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:35:05.226 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:05.226 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:35:05.226 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:35:05.226 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:05.226 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:05.226 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:05.226 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:35:05.226 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:35:05.226 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:05.226 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:05.226 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:05.226 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:05.226 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:35:05.227 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:35:05.227 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:35:05.227 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:35:05.227 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:35:05.227 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:05.227 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:05.227 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:35:05.227 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:35:05.227 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:35:05.227 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:35:05.227 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:05.227 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:35:05.228 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:35:05.228 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:05.228 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:05.228 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:05.228 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:35:05.228 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:35:05.228 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:05.228 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:05.228 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:05.228 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:05.228 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:35:05.228 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:35:05.228 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:35:05.229 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:35:05.229 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:35:05.229 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:05.229 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:05.229 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:35:05.229 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:35:05.229 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:35:05.229 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:35:05.229 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:05.229 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:35:05.229 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:35:05.229 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:05.229 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:05.230 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:05.230 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:35:05.230 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:35:05.230 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:05.230 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:05.230 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:05.230 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:05.230 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:35:05.230 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:35:05.230 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:35:05.230 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:35:05.230 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:35:05.230 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:05.230 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:05.231 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:05.231 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:35:05.231 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:35:05.231 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:05.231 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:35:05.231 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:35:05.231 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:35:05.231 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:05.231 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:05.231 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:35:05.231 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:05.231 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:35:05.231 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:35:05.232 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:35:05.232 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:05.232 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:05.232 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:35:05.232 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:05.232 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:35:05.232 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:35:05.232 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:35:05.232 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:05.232 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:05.232 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:35:05.232 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:05.232 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:35:05.232 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:35:05.233 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:35:05.233 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:05.233 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:05.233 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:05.233 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:35:05.233 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:35:05.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.208+0000 7fd399c6e640 1 -- 172.21.15.12:0/3429246907 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd38003e210 msgr2=0x7fd3800406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:05.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.208+0000 7fd399c6e640 1 --2- 172.21.15.12:0/3429246907 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd38003e210 0x7fd3800406d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd394125880 tx=0x7fd38c003c60 comp rx=0 tx=0).stop 2024-09-11T01:35:05.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.208+0000 7fd399c6e640 10 monclient: shutdown 2024-09-11T01:35:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.208+0000 7fd399c6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:35:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.208+0000 7fd399c6e640 1 -- 172.21.15.12:0/3429246907 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd3940a4f20 msgr2=0x7fd394135db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.208+0000 7fd399c6e640 1 --2- 172.21.15.12:0/3429246907 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd3940a4f20 0x7fd394135db0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd384000c00 tx=0x7fd3840046c0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.208+0000 7fd393fff640 1 -- 172.21.15.12:0/3429246907 reap_dead start 2024-09-11T01:35:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.208+0000 7fd393fff640 1 -- 172.21.15.12:0/3429246907 reap_dead start 2024-09-11T01:35:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.208+0000 7fd399c6e640 1 -- 172.21.15.12:0/3429246907 shutdown_connections 2024-09-11T01:35:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.208+0000 7fd399c6e640 1 -- 172.21.15.12:0/3429246907 >> 172.21.15.12:0/3429246907 conn(0x7fd3940a7440 msgr2=0x7fd39409ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:35:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.208+0000 7fd399c6e640 1 -- 172.21.15.12:0/3429246907 shutdown_connections 2024-09-11T01:35:05.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.208+0000 7fd399c6e640 1 -- 172.21.15.12:0/3429246907 wait complete. 2024-09-11T01:35:05.235 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:35:05.236 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:35:05.236 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:35:05.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.334+0000 7fc951a57640 10 monclient: get_monmap_and_config 2024-09-11T01:35:05.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.334+0000 7fc951a57640 10 monclient: build_initial_monmap 2024-09-11T01:35:05.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.334+0000 7fc951a57640 10 monclient: monmap: 2024-09-11T01:35:05.337 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:35:05.337 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:05.337 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:35:05.335590+0000 2024-09-11T01:35:05.337 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:35:05.335590+0000 2024-09-11T01:35:05.337 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:35:05.337 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:05.337 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:35:05.337 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:35:05.337 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:35:05.337 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:35:05.337 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:35:05.338 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:05.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.335+0000 7fc951a57640 1 Processor -- start 2024-09-11T01:35:05.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.335+0000 7fc951a57640 1 -- start start 2024-09-11T01:35:05.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.335+0000 7fc951a57640 10 monclient: init 2024-09-11T01:35:05.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc951a57640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:05.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc951a57640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-11T01:35:05.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc951a57640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc94c09dcd0 addr v1:172.21.15.71:6789/0 2024-09-11T01:35:05.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc951a57640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc94c09e0b0 0x7fc94c0a2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:05.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc951a57640 10 monclient(hunting): picked mon.noname-b con 0x7fc94c09e0b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:35:05.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc951a57640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc94c13b6b0 0x7fc94c13daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:05.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc951a57640 10 monclient(hunting): picked mon.noname-a con 0x7fc94c13b6b0 addr v2:172.21.15.12:3300/0 2024-09-11T01:35:05.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc951a57640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc94c09caa0 con 0x7fc94c09dcd0 2024-09-11T01:35:05.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc951a57640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:05.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc951a57640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc94c0a3b30 con 0x7fc94c13b6b0 2024-09-11T01:35:05.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc951a57640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:05.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc951a57640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc94c09c4e0 con 0x7fc94c09e0b0 2024-09-11T01:35:05.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc951a57640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:05.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc951a57640 10 monclient(hunting): authenticate will time out at 1111.778236s 2024-09-11T01:35:05.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc94b7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc94c13b6b0 0x7fc94c13daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:05.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc943fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc94c09e0b0 0x7fc94c0a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:05.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc94b7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc94c13b6b0 0x7fc94c13daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48994/0 (socket says 172.21.15.12:48994) 2024-09-11T01:35:05.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc94b7fe640 1 -- 172.21.15.12:0/3337052010 learned_addr learned my addr 172.21.15.12:0/3337052010 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:35:05.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc94b7fe640 10 monclient(hunting): get_auth_request con 0x7fc94c13b6b0 auth_method 0 2024-09-11T01:35:05.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc94b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:05.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc94b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:05.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc94b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:05.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc943fff640 10 monclient(hunting): get_auth_request con 0x7fc94c09e0b0 auth_method 0 2024-09-11T01:35:05.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc943fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:05.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc943fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:05.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc943fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:05.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc94b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:05.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc94b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:05.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc94b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:05.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc943fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:05.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc943fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:05.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc943fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:05.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc94b7fe640 10 monclient(hunting): handle_auth_done global_id 54216 payload 274 2024-09-11T01:35:05.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc94b7fe640 1 -- 172.21.15.12:0/3337052010 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc94c09e0b0 msgr2=0x7fc94c0a2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:35:05.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc94b7fe640 1 --2- 172.21.15.12:0/3337052010 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc94c09e0b0 0x7fc94c0a2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc94b7fe640 1 -- 172.21.15.12:0/3337052010 >> v1:172.21.15.71:6789/0 conn(0x7fc94c09dcd0 legacy=0x7fc94c0a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:35:05.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc94b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:35:05.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc94b7fe640 1 monclient: found mon.noname-a 2024-09-11T01:35:05.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc94b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:05.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc94b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:35:05.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.336+0000 7fc94b7fe640 1 -- 172.21.15.12:0/3337052010 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc94c0a3470 con 0x7fc94c13b6b0 2024-09-11T01:35:05.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.339+0000 7fc94b7fe640 1 -- 172.21.15.12:0/3337052010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc94c13b6b0 msgr2=0x7fc94c13daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:05.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.339+0000 7fc94b7fe640 1 --2- 172.21.15.12:0/3337052010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc94c13b6b0 0x7fc94c13daa0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc944009e50 tx=0x7fc94402a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e525b0a4064cb897 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:05.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.341+0000 7fc94a7fc640 1 -- 172.21.15.12:0/3337052010 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc944036070 con 0x7fc94c13b6b0 2024-09-11T01:35:05.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.341+0000 7fc94a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:05.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.341+0000 7fc94a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:35:05.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.341+0000 7fc94a7fc640 10 monclient: dump: 2024-09-11T01:35:05.348 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:05.348 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:05.348 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.348 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.348 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:05.348 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:05.348 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:05.348 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:05.348 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:05.348 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:05.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.341+0000 7fc94a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:35:05.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.341+0000 7fc94a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:35:05.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.341+0000 7fc94a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:05.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.341+0000 7fc94a7fc640 1 -- 172.21.15.12:0/3337052010 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc944007800 con 0x7fc94c13b6b0 2024-09-11T01:35:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.341+0000 7fc951a57640 5 monclient: authenticate success, global_id 54216 2024-09-11T01:35:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.341+0000 7fc951a57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:35:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.341+0000 7fc94a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:35:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.341+0000 7fc951a57640 10 monclient: get_monmap_and_config success 2024-09-11T01:35:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.341+0000 7fc94a7fc640 1 -- 172.21.15.12:0/3337052010 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc944007bd0 con 0x7fc94c13b6b0 2024-09-11T01:35:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.341+0000 7fc94a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.341+0000 7fc951a57640 4 set_mon_vals no callback set 2024-09-11T01:35:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.341+0000 7fc94a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:35:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.341+0000 7fc94a7fc640 10 monclient: dump: 2024-09-11T01:35:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.349 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:05.350 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.342+0000 7fc951a57640 10 monclient: shutdown 2024-09-11T01:35:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.342+0000 7fc951a57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:35:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.342+0000 7fc951a57640 1 -- 172.21.15.12:0/3337052010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc94c13b6b0 msgr2=0x7fc94c13daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.342+0000 7fc951a57640 1 --2- 172.21.15.12:0/3337052010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc94c13b6b0 0x7fc94c13daa0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc944009e50 tx=0x7fc94402a560 comp rx=0 tx=0).stop 2024-09-11T01:35:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.342+0000 7fc951a57640 1 -- 172.21.15.12:0/3337052010 shutdown_connections 2024-09-11T01:35:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.342+0000 7fc951a57640 1 --2- 172.21.15.12:0/3337052010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc94c13b6b0 0x7fc94c13daa0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.342+0000 7fc951a57640 1 --2- 172.21.15.12:0/3337052010 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc94c09e0b0 0x7fc94c0a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.342+0000 7fc951a57640 1 -- 172.21.15.12:0/3337052010 >> 172.21.15.12:0/3337052010 conn(0x7fc94c0a7da0 msgr2=0x7fc94c0a3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:35:05.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.342+0000 7fc951a57640 1 -- 172.21.15.12:0/3337052010 shutdown_connections 2024-09-11T01:35:05.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.342+0000 7fc951a57640 1 -- 172.21.15.12:0/3337052010 wait complete. 2024-09-11T01:35:05.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.343+0000 7fc951a57640 10 monclient: build_initial_monmap 2024-09-11T01:35:05.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.343+0000 7fc951a57640 10 monclient: monmap: 2024-09-11T01:35:05.351 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:35:05.351 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:35:05.351 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:35:05.351 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:35:05.351 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:35:05.351 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:05.351 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:35:05.351 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:35:05.351 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:35:05.351 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:05.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.343+0000 7fc951a57640 1 Processor -- start 2024-09-11T01:35:05.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.343+0000 7fc951a57640 1 -- start start 2024-09-11T01:35:05.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.343+0000 7fc951a57640 10 monclient: init 2024-09-11T01:35:05.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.343+0000 7fc951a57640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:05.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.343+0000 7fc951a57640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:35:05.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.343+0000 7fc951a57640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc94c09dcd0 0x7fc94c08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:05.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.343+0000 7fc951a57640 10 monclient(hunting): picked mon.noname-a con 0x7fc94c09dcd0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:35:05.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.343+0000 7fc951a57640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc94c09e0b0 0x7fc94c08da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:05.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.343+0000 7fc951a57640 10 monclient(hunting): picked mon.noname-c con 0x7fc94c09e0b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:35:05.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.343+0000 7fc951a57640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc94c13b6b0 0x7fc94c08dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:05.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.343+0000 7fc951a57640 10 monclient(hunting): picked mon.noname-b con 0x7fc94c13b6b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:35:05.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.343+0000 7fc951a57640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:05.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.343+0000 7fc951a57640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc94c074cd0 con 0x7fc94c09dcd0 2024-09-11T01:35:05.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.343+0000 7fc951a57640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:05.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.343+0000 7fc951a57640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc94c074b50 con 0x7fc94c13b6b0 2024-09-11T01:35:05.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.343+0000 7fc951a57640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:05.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.343+0000 7fc951a57640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc94c0a2980 con 0x7fc94c09e0b0 2024-09-11T01:35:05.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.343+0000 7fc951a57640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:05.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.343+0000 7fc951a57640 10 monclient(hunting): authenticate will time out at 1111.785911s 2024-09-11T01:35:05.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.343+0000 7fc94affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc94c09dcd0 0x7fc94c08d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:05.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc943fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc94c09e0b0 0x7fc94c08da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:05.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc94affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc94c09dcd0 0x7fc94c08d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49002/0 (socket says 172.21.15.12:49002) 2024-09-11T01:35:05.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc943fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc94c09e0b0 0x7fc94c08da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:35500/0 (socket says 172.21.15.12:35500) 2024-09-11T01:35:05.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc94affd640 1 -- 172.21.15.12:0/231816759 learned_addr learned my addr 172.21.15.12:0/231816759 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:35:05.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc94affd640 10 monclient(hunting): get_auth_request con 0x7fc94c09dcd0 auth_method 0 2024-09-11T01:35:05.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc94affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:05.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc94affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:05.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc94affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:05.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc943fff640 10 monclient(hunting): get_auth_request con 0x7fc94c09e0b0 auth_method 0 2024-09-11T01:35:05.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc943fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:05.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc943fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:05.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc943fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:05.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc94affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:05.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc94affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:05.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc94affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:05.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc943fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:05.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc943fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:05.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc943fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:05.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc94b7fe640 1 --2- 172.21.15.12:0/231816759 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc94c13b6b0 0x7fc94c08dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:05.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc94b7fe640 10 monclient(hunting): get_auth_request con 0x7fc94c13b6b0 auth_method 0 2024-09-11T01:35:05.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc94b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:05.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc94b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:05.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc94b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:05.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc943fff640 10 monclient(hunting): handle_auth_done global_id 64127 payload 824 2024-09-11T01:35:05.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc943fff640 1 -- 172.21.15.12:0/231816759 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc94c13b6b0 msgr2=0x7fc94c08dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:05.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc943fff640 1 --2- 172.21.15.12:0/231816759 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc94c13b6b0 0x7fc94c08dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc943fff640 1 -- 172.21.15.12:0/231816759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc94c09dcd0 msgr2=0x7fc94c08d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:05.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc943fff640 1 --2- 172.21.15.12:0/231816759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc94c09dcd0 0x7fc94c08d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc943fff640 10 monclient: _finish_hunting 0 2024-09-11T01:35:05.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc943fff640 1 monclient: found mon.noname-c 2024-09-11T01:35:05.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc943fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:05.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc943fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:35:05.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc943fff640 1 -- 172.21.15.12:0/231816759 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc94c0a15a0 con 0x7fc94c09e0b0 2024-09-11T01:35:05.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc94affd640 1 --2- 172.21.15.12:0/231816759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc94c09dcd0 0x7fc94c08d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:35:05.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.344+0000 7fc94b7fe640 1 --2- 172.21.15.12:0/231816759 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc94c13b6b0 0x7fc94c08dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:35:05.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.346+0000 7fc943fff640 1 -- 172.21.15.12:0/231816759 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc94c09e0b0 msgr2=0x7fc94c08da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:05.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.346+0000 7fc943fff640 1 --2- 172.21.15.12:0/231816759 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc94c09e0b0 0x7fc94c08da90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc93400e8a0 tx=0x7fc93400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:05.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.346+0000 7fc948ff9640 1 -- 172.21.15.12:0/231816759 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc934017070 con 0x7fc94c09e0b0 2024-09-11T01:35:05.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.346+0000 7fc948ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:05.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.346+0000 7fc948ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:35:05.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.346+0000 7fc948ff9640 10 monclient: dump: 2024-09-11T01:35:05.358 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:05.358 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:05.358 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.358 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.358 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:05.358 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:05.358 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:05.358 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:05.358 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:05.358 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:05.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.346+0000 7fc948ff9640 10 monclient: _finish_auth 0 2024-09-11T01:35:05.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.346+0000 7fc948ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:35:05.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.346+0000 7fc948ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:05.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.346+0000 7fc948ff9640 1 -- 172.21.15.12:0/231816759 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc93400cb50 con 0x7fc94c09e0b0 2024-09-11T01:35:05.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.346+0000 7fc948ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:35:05.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.346+0000 7fc948ff9640 1 -- 172.21.15.12:0/231816759 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc934013470 con 0x7fc94c09e0b0 2024-09-11T01:35:05.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.346+0000 7fc948ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:05.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.346+0000 7fc948ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:35:05.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.346+0000 7fc948ff9640 10 monclient: dump: 2024-09-11T01:35:05.359 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:05.359 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:05.359 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.359 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:05.359 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:05.359 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:05.360 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.346+0000 7fc951a57640 5 monclient: authenticate success, global_id 64127 2024-09-11T01:35:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.346+0000 7fc951a57640 10 monclient: _renew_subs 2024-09-11T01:35:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.346+0000 7fc951a57640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:35:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.346+0000 7fc951a57640 1 -- 172.21.15.12:0/231816759 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc94c09e520 con 0x7fc94c09e0b0 2024-09-11T01:35:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.346+0000 7fc949ffb640 4 set_mon_vals no callback set 2024-09-11T01:35:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.346+0000 7fc951a57640 10 monclient: _renew_subs 2024-09-11T01:35:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.346+0000 7fc951a57640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:35:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.346+0000 7fc951a57640 1 -- 172.21.15.12:0/231816759 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc94c090f20 con 0x7fc94c09e0b0 2024-09-11T01:35:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.347+0000 7fc948ff9640 1 -- 172.21.15.12:0/231816759 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fc934024060 con 0x7fc94c09e0b0 2024-09-11T01:35:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.347+0000 7fc951a57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:05.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.347+0000 7fc951a57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:05.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.347+0000 7fc951a57640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:35:05.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.347+0000 7fc951a57640 1 -- 172.21.15.12:0/231816759 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc94c0a1d30 con 0x7fc94c09e0b0 2024-09-11T01:35:05.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.348+0000 7fc948ff9640 1 --2- 172.21.15.12:0/231816759 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc92c03e260 0x7fc92c040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:05.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.348+0000 7fc948ff9640 1 -- 172.21.15.12:0/231816759 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(199..199 src has 1..199) ==== 4767+0+0 (secure 0 0 0) 0x7fc934052c30 con 0x7fc94c09e0b0 2024-09-11T01:35:05.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.348+0000 7fc94affd640 1 --2- 172.21.15.12:0/231816759 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc92c03e260 0x7fc92c040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:05.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.348+0000 7fc94affd640 10 monclient: get_auth_request con 0x7fc92c03e260 auth_method 0 2024-09-11T01:35:05.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.350+0000 7fc94affd640 1 --2- 172.21.15.12:0/231816759 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc92c03e260 0x7fc92c040720 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc93c0043e0 tx=0x7fc93c007ef0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:05.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.353+0000 7fc948ff9640 1 -- 172.21.15.12:0/231816759 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc934051640 con 0x7fc94c09e0b0 2024-09-11T01:35:05.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.353+0000 7fc948ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:05.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.353+0000 7fc948ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:35:05.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.612+0000 7fc951a57640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:35:05.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.612+0000 7fc951a57640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:35:05.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.612+0000 7fc951a57640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:35:05.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.612+0000 7fc951a57640 1 -- 172.21.15.12:0/231816759 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fc94c08fa00 con 0x7fc94c09e0b0 2024-09-11T01:35:05.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.614+0000 7fc948ff9640 1 -- 172.21.15.12:0/231816759 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fc934023d00 con 0x7fc94c09e0b0 2024-09-11T01:35:05.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.614+0000 7fc948ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:35:05.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.614+0000 7fc948ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:35:05.616 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.614+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.616 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.614+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=11,kv=12,logm=62} crc {auth=416146424,config=2586702516,health=3226445885,kv=3263697420,logm=1161933660}) 2024-09-11T01:35:05.616 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.614+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=13,config=2,health=12,kv=10,logm=63} crc {auth=416146424,config=2586702516,health=705117743,kv=4051853960,logm=537777366}) 2024-09-11T01:35:05.616 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.614+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.616 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.614+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=11,kv=12,logm=62} crc {auth=416146424,config=2586702516,health=3226445885,kv=3263697420,logm=1161933660}) 2024-09-11T01:35:05.616 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.614+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=1,health=10,kv=12,logm=64} crc {auth=416146424,config=1228700967,health=3550853401,kv=3263697420,logm=2554807222}) 2024-09-11T01:35:05.618 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.615+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.618 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.615+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2220947729}) 2024-09-11T01:35:05.618 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.615+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3461503566}) 2024-09-11T01:35:05.618 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.615+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.618 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.615+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2220947729}) 2024-09-11T01:35:05.618 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.615+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3930297494}) 2024-09-11T01:35:05.618 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.616+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.618 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.616+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2495919744}) 2024-09-11T01:35:05.618 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.616+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2513743458}) 2024-09-11T01:35:05.618 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.616+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.618 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.616+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2495919744}) 2024-09-11T01:35:05.618 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.616+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1947490990}) 2024-09-11T01:35:05.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.616+0000 7fc951a57640 1 -- 172.21.15.12:0/231816759 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc92c03e260 msgr2=0x7fc92c040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:05.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.616+0000 7fc951a57640 1 --2- 172.21.15.12:0/231816759 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc92c03e260 0x7fc92c040720 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc93c0043e0 tx=0x7fc93c007ef0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.616+0000 7fc951a57640 10 monclient: shutdown 2024-09-11T01:35:05.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.616+0000 7fc951a57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:35:05.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.616+0000 7fc951a57640 1 -- 172.21.15.12:0/231816759 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc94c09e0b0 msgr2=0x7fc94c08da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:05.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.616+0000 7fc951a57640 1 --2- 172.21.15.12:0/231816759 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc94c09e0b0 0x7fc94c08da90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc93400e8a0 tx=0x7fc93400ed60 comp rx=0 tx=0).stop 2024-09-11T01:35:05.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.616+0000 7fc951a57640 1 -- 172.21.15.12:0/231816759 shutdown_connections 2024-09-11T01:35:05.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.616+0000 7fc951a57640 1 --2- 172.21.15.12:0/231816759 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc94c13b6b0 0x7fc94c08dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.616+0000 7fc951a57640 1 --2- 172.21.15.12:0/231816759 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc94c09e0b0 0x7fc94c08da90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.616+0000 7fc951a57640 1 --2- 172.21.15.12:0/231816759 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc92c03e260 0x7fc92c040720 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.616+0000 7fc951a57640 1 --2- 172.21.15.12:0/231816759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc94c09dcd0 0x7fc94c08d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:05.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.616+0000 7fc951a57640 1 -- 172.21.15.12:0/231816759 >> 172.21.15.12:0/231816759 conn(0x7fc94c0a7da0 msgr2=0x7fc94c09a420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:35:05.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.617+0000 7fc951a57640 1 -- 172.21.15.12:0/231816759 shutdown_connections 2024-09-11T01:35:05.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:05.617+0000 7fc951a57640 1 -- 172.21.15.12:0/231816759 wait complete. 2024-09-11T01:35:05.622 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.617+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.622 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.617+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1971998355}) 2024-09-11T01:35:05.622 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.617+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2518196411}) 2024-09-11T01:35:05.622 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.617+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.622 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.617+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1971998355}) 2024-09-11T01:35:05.622 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.617+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2146563159}) 2024-09-11T01:35:05.622 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.618+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.622 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.618+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1110384098}) 2024-09-11T01:35:05.622 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.618+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=101496093}) 2024-09-11T01:35:05.623 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.618+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.623 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.618+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1110384098}) 2024-09-11T01:35:05.623 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.618+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2472075983}) 2024-09-11T01:35:05.623 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.619+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.623 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.619+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=801948948}) 2024-09-11T01:35:05.623 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.619+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2140671225}) 2024-09-11T01:35:05.623 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.619+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.623 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.619+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=801948948}) 2024-09-11T01:35:05.623 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.619+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1535399066}) 2024-09-11T01:35:05.623 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.620+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.623 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.620+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2200907224}) 2024-09-11T01:35:05.623 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.620+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=536859568}) 2024-09-11T01:35:05.623 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.620+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.623 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.620+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2200907224}) 2024-09-11T01:35:05.624 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.620+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=823058513}) 2024-09-11T01:35:05.625 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.621+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.625 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.621+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=360388627}) 2024-09-11T01:35:05.625 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.621+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1567545512}) 2024-09-11T01:35:05.625 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.621+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.625 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.621+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=360388627}) 2024-09-11T01:35:05.625 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.621+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=484158254}) 2024-09-11T01:35:05.625 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.622+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.625 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.622+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3691049931}) 2024-09-11T01:35:05.625 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.622+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3367887705}) 2024-09-11T01:35:05.625 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.622+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.625 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.622+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3691049931}) 2024-09-11T01:35:05.625 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.622+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3250594757}) 2024-09-11T01:35:05.626 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.623+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.626 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.623+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715960209}) 2024-09-11T01:35:05.626 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.623+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3713798269}) 2024-09-11T01:35:05.626 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.623+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.626 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.623+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715960209}) 2024-09-11T01:35:05.626 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.623+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1693486080}) 2024-09-11T01:35:05.627 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.624+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.627 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.624+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1900949216}) 2024-09-11T01:35:05.627 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.624+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4137806010}) 2024-09-11T01:35:05.627 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.624+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.627 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.624+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1900949216}) 2024-09-11T01:35:05.627 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.624+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1552966034}) 2024-09-11T01:35:05.627 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.625+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.628 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.625+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3828409614}) 2024-09-11T01:35:05.628 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.625+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1640306388}) 2024-09-11T01:35:05.628 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.625+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.628 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.625+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3828409614}) 2024-09-11T01:35:05.628 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.625+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3726856193}) 2024-09-11T01:35:05.628 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.625+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.628 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.625+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=898663547}) 2024-09-11T01:35:05.628 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.625+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=615802869}) 2024-09-11T01:35:05.628 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.625+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.628 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.625+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=898663547}) 2024-09-11T01:35:05.628 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.625+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4017872060}) 2024-09-11T01:35:05.629 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.626+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.630 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.626+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=28599811}) 2024-09-11T01:35:05.630 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.626+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1498468428}) 2024-09-11T01:35:05.630 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.626+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.630 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.626+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=28599811}) 2024-09-11T01:35:05.630 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.626+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2369398957}) 2024-09-11T01:35:05.630 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.627+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.630 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.627+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=77,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=2} crc {logm=650838274,mdsmap=3140514009,mgr=4204056853,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2746223661,mon_config_key=243754935}) 2024-09-11T01:35:05.630 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.627+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=78,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9} crc {logm=1110844826,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1763914508}) 2024-09-11T01:35:05.630 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.627+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.630 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.627+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=77,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=2} crc {logm=650838274,mdsmap=3140514009,mgr=4204056853,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2746223661,mon_config_key=243754935}) 2024-09-11T01:35:05.630 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.627+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=81,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=7} crc {logm=3626979020,mdsmap=3760429986,mgr=2672916156,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=80213274}) 2024-09-11T01:35:05.630 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.628+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.630 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.628+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=36} crc {mon_config_key=2796119330,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=351180295,osd_snap=3806084405,osdmap=3872555243}) 2024-09-11T01:35:05.631 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.628+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=14,monmap=3,nvmeofgw=2,osd_metadata=5,osd_pg_creating=1,osd_snap=35,osdmap=40} crc {mon_config_key=3987932583,monmap=1972551139,nvmeofgw=3140514009,osd_metadata=2239697408,osd_pg_creating=351180295,osd_snap=596897224,osdmap=4180877242}) 2024-09-11T01:35:05.631 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.628+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.631 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.628+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=36} crc {mon_config_key=2796119330,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=351180295,osd_snap=3806084405,osdmap=3872555243}) 2024-09-11T01:35:05.631 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.628+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=36} crc {mon_config_key=3139524022,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=351180295,osd_snap=3806084405,osdmap=2185535766}) 2024-09-11T01:35:05.632 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.628+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.632 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.628+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=95662100}) 2024-09-11T01:35:05.632 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.629+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2031739145}) 2024-09-11T01:35:05.632 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.629+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.632 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.629+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=95662100}) 2024-09-11T01:35:05.632 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.629+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1583842087}) 2024-09-11T01:35:05.633 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:35:05.633 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.630+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.633 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.630+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2212324673}) 2024-09-11T01:35:05.633 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.630+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1001774390}) 2024-09-11T01:35:05.634 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.630+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.634 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.630+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2212324673}) 2024-09-11T01:35:05.634 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.630+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2759872312}) 2024-09-11T01:35:05.635 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.633+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.635 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.633+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=63} crc {osdmap=3972144522}) 2024-09-11T01:35:05.635 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.633+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=67} crc {osdmap=452209575}) 2024-09-11T01:35:05.635 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.633+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:35:05.635 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.633+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=63} crc {osdmap=3972144522}) 2024-09-11T01:35:05.635 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:35:05.633+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=67} crc {osdmap=515447372}) 2024-09-11T01:35:06.040 INFO:tasks.rados.rados.0.smithi071.stdout:848: done (3 left) 2024-09-11T01:35:06.040 INFO:tasks.rados.rados.0.smithi071.stdout:850: done (2 left) 2024-09-11T01:35:06.040 INFO:tasks.rados.rados.0.smithi071.stdout:851: done (1 left) 2024-09-11T01:35:06.040 INFO:tasks.rados.rados.0.smithi071.stdout:852: done (0 left) 2024-09-11T01:35:06.041 INFO:tasks.rados.rados.0.smithi071.stdout:853: write (excl) oid 14 current snap is 92 2024-09-11T01:35:06.041 INFO:tasks.rados.rados.0.smithi071.stdout:853: seq_num 321 ranges {595047=641755,1667661=418109,2548663=763022} 2024-09-11T01:35:06.048 INFO:tasks.rados.rados.0.smithi071.stdout:853: writing smithi07134777-14 from 595047 to 1236802 tid 1 2024-09-11T01:35:06.052 INFO:tasks.rados.rados.0.smithi071.stdout:853: writing smithi07134777-14 from 1667661 to 2085770 tid 2 2024-09-11T01:35:06.059 INFO:tasks.rados.rados.0.smithi071.stdout:853: writing smithi07134777-14 from 2548663 to 3311685 tid 3 2024-09-11T01:35:06.059 INFO:tasks.rados.rados.0.smithi071.stdout:854: write (excl) oid 5 current snap is 92 2024-09-11T01:35:06.059 INFO:tasks.rados.rados.0.smithi071.stdout:853: finishing write tid 1 to smithi07134777-14 2024-09-11T01:35:06.059 INFO:tasks.rados.rados.0.smithi071.stdout:853: finishing write tid 2 to smithi07134777-14 2024-09-11T01:35:06.059 INFO:tasks.rados.rados.0.smithi071.stdout:854: seq_num 322 ranges {611854=717004,2126143=561767,3259733=152121} 2024-09-11T01:35:06.066 INFO:tasks.rados.rados.0.smithi071.stdout:854: writing smithi07134777-5 from 611854 to 1328858 tid 1 2024-09-11T01:35:06.074 INFO:tasks.rados.rados.0.smithi071.stdout:854: writing smithi07134777-5 from 2126143 to 2687910 tid 2 2024-09-11T01:35:06.079 INFO:tasks.rados.rados.0.smithi071.stdout:854: writing smithi07134777-5 from 3259733 to 3411854 tid 3 2024-09-11T01:35:06.079 INFO:tasks.rados.rados.0.smithi071.stdout:855: read oid 15 snap -1 2024-09-11T01:35:06.079 INFO:tasks.rados.rados.0.smithi071.stdout:855: expect (ObjNum 290 snap 78 seq_num 290) 2024-09-11T01:35:06.079 INFO:tasks.rados.rados.0.smithi071.stdout:856: rollback oid 27 current snap is 92 2024-09-11T01:35:06.079 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 27 to 82 2024-09-11T01:35:06.079 INFO:tasks.rados.rados.0.smithi071.stdout:853: finishing write tid 3 to smithi07134777-14 2024-09-11T01:35:06.079 INFO:tasks.rados.rados.0.smithi071.stdout:853: finishing write tid 4 to smithi07134777-14 2024-09-11T01:35:06.079 INFO:tasks.rados.rados.0.smithi071.stdout:853: finishing write tid 5 to smithi07134777-14 2024-09-11T01:35:06.079 INFO:tasks.rados.rados.0.smithi071.stdout:853: oid 14 updating version 0 to 386 2024-09-11T01:35:06.079 INFO:tasks.rados.rados.0.smithi071.stdout:853: oid 14 updating version 386 to 388 2024-09-11T01:35:06.079 INFO:tasks.rados.rados.0.smithi071.stdout:853: oid 14 updating version 388 to 389 2024-09-11T01:35:06.079 INFO:tasks.rados.rados.0.smithi071.stdout:853: oid 14 version 389 is already newer than 387 2024-09-11T01:35:06.080 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 389 (ObjNum 321 snap 92 seq_num 321) dirty exists 2024-09-11T01:35:06.080 INFO:tasks.rados.rados.0.smithi071.stdout:853: left oid 14 (ObjNum 321 snap 92 seq_num 321) 2024-09-11T01:35:06.080 INFO:tasks.rados.rados.0.smithi071.stdout:854: finishing write tid 1 to smithi07134777-5 2024-09-11T01:35:06.080 INFO:tasks.rados.rados.0.smithi071.stdout:853: done (3 left) 2024-09-11T01:35:06.080 INFO:tasks.rados.rados.0.smithi071.stdout:857: write (excl) oid 20 current snap is 92 2024-09-11T01:35:06.080 INFO:tasks.rados.rados.0.smithi071.stdout:857: seq_num 323 ranges {628661=792253,2184625=421778,3403509=25152} 2024-09-11T01:35:06.086 INFO:tasks.rados.rados.0.smithi071.stdout:857: writing smithi07134777-20 from 628661 to 1420914 tid 1 2024-09-11T01:35:06.093 INFO:tasks.rados.rados.0.smithi071.stdout:857: writing smithi07134777-20 from 2184625 to 2606403 tid 2 2024-09-11T01:35:06.098 INFO:tasks.rados.rados.0.smithi071.stdout:857: writing smithi07134777-20 from 3403509 to 3428661 tid 3 2024-09-11T01:35:06.098 INFO:tasks.rados.rados.0.smithi071.stdout:858: write (excl) oid 2 current snap is 92 2024-09-11T01:35:06.098 INFO:tasks.rados.rados.0.smithi071.stdout:854: finishing write tid 2 to smithi07134777-5 2024-09-11T01:35:06.098 INFO:tasks.rados.rados.0.smithi071.stdout:858: seq_num 324 ranges {645468=467502,1726754=565436,2798226=596838} 2024-09-11T01:35:06.104 INFO:tasks.rados.rados.0.smithi071.stdout:858: writing smithi07134777-2 from 645468 to 1112970 tid 1 2024-09-11T01:35:06.110 INFO:tasks.rados.rados.0.smithi071.stdout:858: writing smithi07134777-2 from 1726754 to 2292190 tid 2 2024-09-11T01:35:06.116 INFO:tasks.rados.rados.0.smithi071.stdout:858: writing smithi07134777-2 from 2798226 to 3395064 tid 3 2024-09-11T01:35:06.117 INFO:tasks.rados.rados.0.smithi071.stdout:859: snap_remove snap 89 2024-09-11T01:35:06.634 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:35:06.634 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:35:06.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.746+0000 7fb809a7f640 10 monclient: get_monmap_and_config 2024-09-11T01:35:06.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.746+0000 7fb809a7f640 10 monclient: build_initial_monmap 2024-09-11T01:35:06.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.746+0000 7fb809a7f640 10 monclient: monmap: 2024-09-11T01:35:06.748 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:35:06.748 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:06.748 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:35:06.747000+0000 2024-09-11T01:35:06.748 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:35:06.747000+0000 2024-09-11T01:35:06.748 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:35:06.748 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:06.748 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:35:06.748 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:35:06.748 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:35:06.749 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:35:06.749 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:35:06.749 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:06.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.747+0000 7fb809a7f640 1 Processor -- start 2024-09-11T01:35:06.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.747+0000 7fb809a7f640 1 -- start start 2024-09-11T01:35:06.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.747+0000 7fb809a7f640 10 monclient: init 2024-09-11T01:35:06.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.747+0000 7fb809a7f640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:06.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.747+0000 7fb809a7f640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-11T01:35:06.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.747+0000 7fb809a7f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb804138230 addr v1:172.21.15.71:6789/0 2024-09-11T01:35:06.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.747+0000 7fb809a7f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb80409d030 addr v1:172.21.15.12:6789/0 2024-09-11T01:35:06.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.747+0000 7fb809a7f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb80409d460 0x7fb8041407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:06.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.747+0000 7fb809a7f640 10 monclient(hunting): picked mon.noname-b con 0x7fb80409d460 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:35:06.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.747+0000 7fb809a7f640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8040a6e30 con 0x7fb80409d030 2024-09-11T01:35:06.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.747+0000 7fb809a7f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb80409ba20 con 0x7fb804138230 2024-09-11T01:35:06.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.747+0000 7fb809a7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:06.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.747+0000 7fb809a7f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb80409c9c0 con 0x7fb80409d460 2024-09-11T01:35:06.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.747+0000 7fb809a7f640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:06.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.747+0000 7fb809a7f640 10 monclient(hunting): authenticate will time out at 1113.189679s 2024-09-11T01:35:06.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.747+0000 7fb803fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb80409d460 0x7fb8041407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:06.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.747+0000 7fb802ffd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb80409d030 0x7fb80413d090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46216/0 (socket says 172.21.15.12:46216) 2024-09-11T01:35:06.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.747+0000 7fb802ffd640 1 -- 172.21.15.12:0/4031064474 learned_addr learned my addr 172.21.15.12:0/4031064474 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:35:06.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.747+0000 7fb803fff640 10 monclient(hunting): get_auth_request con 0x7fb80409d460 auth_method 0 2024-09-11T01:35:06.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.747+0000 7fb803fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:06.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.747+0000 7fb803fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:06.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.747+0000 7fb803fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:06.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.747+0000 7fb803fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:06.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.747+0000 7fb803fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:06.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.747+0000 7fb803fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:06.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.748+0000 7fb803fff640 10 monclient(hunting): handle_auth_done global_id 64130 payload 274 2024-09-11T01:35:06.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.748+0000 7fb803fff640 1 -- 172.21.15.12:0/4031064474 >> v1:172.21.15.71:6789/0 conn(0x7fb804138230 legacy=0x7fb804138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:35:06.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.748+0000 7fb803fff640 1 -- 172.21.15.12:0/4031064474 >> v1:172.21.15.12:6789/0 conn(0x7fb80409d030 legacy=0x7fb80413d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:35:06.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.748+0000 7fb803fff640 10 monclient: _finish_hunting 0 2024-09-11T01:35:06.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.748+0000 7fb803fff640 1 monclient: found mon.noname-b 2024-09-11T01:35:06.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.748+0000 7fb803fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:06.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.748+0000 7fb803fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:35:06.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.748+0000 7fb803fff640 1 -- 172.21.15.12:0/4031064474 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8040a66a0 con 0x7fb80409d460 2024-09-11T01:35:06.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.750+0000 7fb803fff640 1 -- 172.21.15.12:0/4031064474 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb80409d460 msgr2=0x7fb8041407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:06.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.750+0000 7fb803fff640 1 --2- 172.21.15.12:0/4031064474 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb80409d460 0x7fb8041407c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb8040a17e0 tx=0x7fb7fc02a440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7589004214bd99e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:06.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.751+0000 7fb8027fc640 1 -- 172.21.15.12:0/4031064474 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc007a30 con 0x7fb80409d460 2024-09-11T01:35:06.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.751+0000 7fb8027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:06.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.751+0000 7fb8027fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:35:06.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.751+0000 7fb8027fc640 10 monclient: dump: 2024-09-11T01:35:06.758 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:06.758 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:06.758 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:06.758 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:06.758 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:06.758 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:06.758 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:06.758 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:06.758 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:06.758 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:06.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.751+0000 7fb8027fc640 10 monclient: _finish_auth 0 2024-09-11T01:35:06.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.751+0000 7fb8027fc640 10 monclient: _check_auth_tickets 2024-09-11T01:35:06.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.751+0000 7fb8027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:06.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.751+0000 7fb809a7f640 5 monclient: authenticate success, global_id 64130 2024-09-11T01:35:06.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.751+0000 7fb809a7f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:35:06.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.751+0000 7fb8027fc640 1 -- 172.21.15.12:0/4031064474 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7fc007bd0 con 0x7fb80409d460 2024-09-11T01:35:06.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.751+0000 7fb8027fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:35:06.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.751+0000 7fb809a7f640 10 monclient: get_monmap_and_config success 2024-09-11T01:35:06.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.751+0000 7fb809a7f640 4 set_mon_vals no callback set 2024-09-11T01:35:06.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.751+0000 7fb8027fc640 1 -- 172.21.15.12:0/4031064474 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc031400 con 0x7fb80409d460 2024-09-11T01:35:06.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.751+0000 7fb8027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:06.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.751+0000 7fb8027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:35:06.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.751+0000 7fb8027fc640 10 monclient: dump: 2024-09-11T01:35:06.759 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:06.759 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:06.759 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:06.760 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:06.760 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:06.760 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:06.760 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:06.760 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:06.760 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:06.760 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:06.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.751+0000 7fb809a7f640 10 monclient: shutdown 2024-09-11T01:35:06.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.751+0000 7fb809a7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:35:06.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.751+0000 7fb809a7f640 1 -- 172.21.15.12:0/4031064474 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb80409d460 msgr2=0x7fb8041407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:06.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.751+0000 7fb809a7f640 1 --2- 172.21.15.12:0/4031064474 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb80409d460 0x7fb8041407c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb8040a17e0 tx=0x7fb7fc02a440 comp rx=0 tx=0).stop 2024-09-11T01:35:06.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.751+0000 7fb809a7f640 1 -- 172.21.15.12:0/4031064474 shutdown_connections 2024-09-11T01:35:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.751+0000 7fb809a7f640 1 --2- 172.21.15.12:0/4031064474 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb80409d460 0x7fb8041407c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.751+0000 7fb809a7f640 1 -- 172.21.15.12:0/4031064474 >> 172.21.15.12:0/4031064474 conn(0x7fb8040a7120 msgr2=0x7fb8040a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:35:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.751+0000 7fb809a7f640 1 -- 172.21.15.12:0/4031064474 shutdown_connections 2024-09-11T01:35:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.751+0000 7fb809a7f640 1 -- 172.21.15.12:0/4031064474 wait complete. 2024-09-11T01:35:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.752+0000 7fb809a7f640 10 monclient: build_initial_monmap 2024-09-11T01:35:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.752+0000 7fb809a7f640 10 monclient: monmap: 2024-09-11T01:35:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:35:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:35:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:35:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:35:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:35:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:06.761 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:35:06.762 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:35:06.762 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:35:06.762 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:06.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.752+0000 7fb809a7f640 1 Processor -- start 2024-09-11T01:35:06.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.752+0000 7fb809a7f640 1 -- start start 2024-09-11T01:35:06.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb809a7f640 10 monclient: init 2024-09-11T01:35:06.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb809a7f640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:06.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb809a7f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:35:06.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb809a7f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb80409d030 0x7fb80413f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:06.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb809a7f640 10 monclient(hunting): picked mon.noname-b con 0x7fb80409d030 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:35:06.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb809a7f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb80409d460 0x7fb804136e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:06.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb809a7f640 10 monclient(hunting): picked mon.noname-a con 0x7fb80409d460 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:35:06.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb809a7f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb804138230 0x7fb80413ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:06.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb809a7f640 10 monclient(hunting): picked mon.noname-c con 0x7fb804138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:35:06.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb809a7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:06.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb809a7f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb804140e80 con 0x7fb80409d460 2024-09-11T01:35:06.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb809a7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:06.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb809a7f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb804140d00 con 0x7fb80409d030 2024-09-11T01:35:06.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb809a7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:06.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb809a7f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb804141000 con 0x7fb804138230 2024-09-11T01:35:06.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb809a7f640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:06.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb809a7f640 10 monclient(hunting): authenticate will time out at 1113.195464s 2024-09-11T01:35:06.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb802ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb80409d460 0x7fb804136e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:06.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb803fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb804138230 0x7fb80413ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:06.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb802ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb80409d460 0x7fb804136e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45364/0 (socket says 172.21.15.12:45364) 2024-09-11T01:35:06.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb803fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb804138230 0x7fb80413ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:58020/0 (socket says 172.21.15.12:58020) 2024-09-11T01:35:06.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb802ffd640 1 -- 172.21.15.12:0/2244157629 learned_addr learned my addr 172.21.15.12:0/2244157629 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:35:06.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb802ffd640 10 monclient(hunting): get_auth_request con 0x7fb80409d460 auth_method 0 2024-09-11T01:35:06.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb802ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:06.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb802ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:06.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb802ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:06.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb803fff640 10 monclient(hunting): get_auth_request con 0x7fb804138230 auth_method 0 2024-09-11T01:35:06.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb803fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:06.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb803fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:06.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb803fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:06.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb802ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:06.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb802ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:06.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb8037fe640 1 --2- 172.21.15.12:0/2244157629 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb80409d030 0x7fb80413f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:06.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb802ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:06.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb803fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:06.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb803fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:06.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb803fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:06.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb8037fe640 10 monclient(hunting): get_auth_request con 0x7fb80409d030 auth_method 0 2024-09-11T01:35:06.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb8037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:06.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb8037fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:06.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb8037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:06.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb8037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:06.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb8037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:06.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb8037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:06.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb803fff640 10 monclient(hunting): handle_auth_done global_id 64133 payload 824 2024-09-11T01:35:06.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb803fff640 1 -- 172.21.15.12:0/2244157629 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb80409d030 msgr2=0x7fb80413f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:06.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb803fff640 1 --2- 172.21.15.12:0/2244157629 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb80409d030 0x7fb80413f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:06.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb803fff640 1 -- 172.21.15.12:0/2244157629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb80409d460 msgr2=0x7fb804136e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:06.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb803fff640 1 --2- 172.21.15.12:0/2244157629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb80409d460 0x7fb804136e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:06.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb803fff640 10 monclient: _finish_hunting 0 2024-09-11T01:35:06.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.753+0000 7fb803fff640 1 monclient: found mon.noname-c 2024-09-11T01:35:06.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.754+0000 7fb803fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:06.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.754+0000 7fb803fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:35:06.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.754+0000 7fb803fff640 1 -- 172.21.15.12:0/2244157629 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb80413f9f0 con 0x7fb804138230 2024-09-11T01:35:06.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.754+0000 7fb802ffd640 1 --2- 172.21.15.12:0/2244157629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb80409d460 0x7fb804136e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:35:06.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.756+0000 7fb803fff640 1 -- 172.21.15.12:0/2244157629 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb804138230 msgr2=0x7fb80413ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:06.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.756+0000 7fb803fff640 1 --2- 172.21.15.12:0/2244157629 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb804138230 0x7fb80413ca40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc031000 tx=0x7fb7fc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:06.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.756+0000 7fb7ebfff640 1 -- 172.21.15.12:0/2244157629 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc040070 con 0x7fb804138230 2024-09-11T01:35:06.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.756+0000 7fb7ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:06.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.756+0000 7fb7ebfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:35:06.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.756+0000 7fb7ebfff640 10 monclient: dump: 2024-09-11T01:35:06.768 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:06.768 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:06.768 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:06.768 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:06.769 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:06.769 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:06.769 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:06.769 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:06.769 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:06.769 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:06.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.756+0000 7fb7ebfff640 10 monclient: _finish_auth 0 2024-09-11T01:35:06.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.756+0000 7fb7ebfff640 10 monclient: _check_auth_tickets 2024-09-11T01:35:06.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.756+0000 7fb7ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:06.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.756+0000 7fb7ebfff640 1 -- 172.21.15.12:0/2244157629 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7fc045680 con 0x7fb804138230 2024-09-11T01:35:06.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.756+0000 7fb809a7f640 5 monclient: authenticate success, global_id 64133 2024-09-11T01:35:06.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.756+0000 7fb809a7f640 10 monclient: _renew_subs 2024-09-11T01:35:06.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.756+0000 7fb809a7f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:35:06.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.756+0000 7fb809a7f640 1 -- 172.21.15.12:0/2244157629 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb804140580 con 0x7fb804138230 2024-09-11T01:35:06.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.756+0000 7fb7ebfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:35:06.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.756+0000 7fb7ebfff640 1 -- 172.21.15.12:0/2244157629 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc0314c0 con 0x7fb804138230 2024-09-11T01:35:06.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.756+0000 7fb7ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:06.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.756+0000 7fb7ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:35:06.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.756+0000 7fb7ebfff640 10 monclient: dump: 2024-09-11T01:35:06.770 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:06.770 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:06.770 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:06.770 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:06.770 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:06.770 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:06.771 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:06.771 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:06.771 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:06.771 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:06.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.756+0000 7fb809a7f640 10 monclient: _renew_subs 2024-09-11T01:35:06.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.756+0000 7fb809a7f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:35:06.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.756+0000 7fb809a7f640 1 -- 172.21.15.12:0/2244157629 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8041402a0 con 0x7fb804138230 2024-09-11T01:35:06.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.756+0000 7fb801ffb640 4 set_mon_vals no callback set 2024-09-11T01:35:06.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.757+0000 7fb809a7f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:06.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.757+0000 7fb809a7f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:06.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.757+0000 7fb809a7f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:35:06.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.757+0000 7fb7ebfff640 1 -- 172.21.15.12:0/2244157629 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fb7fc007a40 con 0x7fb804138230 2024-09-11T01:35:06.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.757+0000 7fb809a7f640 1 -- 172.21.15.12:0/2244157629 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb80409db10 con 0x7fb804138230 2024-09-11T01:35:06.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.757+0000 7fb7ebfff640 1 --2- 172.21.15.12:0/2244157629 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb7f003e260 0x7fb7f0040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:06.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.757+0000 7fb7ebfff640 1 -- 172.21.15.12:0/2244157629 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(200..200 src has 1..200) ==== 4739+0+0 (secure 0 0 0) 0x7fb7fc074a80 con 0x7fb804138230 2024-09-11T01:35:06.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.758+0000 7fb8037fe640 1 --2- 172.21.15.12:0/2244157629 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb7f003e260 0x7fb7f0040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:06.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.758+0000 7fb8037fe640 10 monclient: get_auth_request con 0x7fb7f003e260 auth_method 0 2024-09-11T01:35:06.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.760+0000 7fb8037fe640 1 --2- 172.21.15.12:0/2244157629 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb7f003e260 0x7fb7f0040720 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb7f4004f60 tx=0x7fb7f4007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:06.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.762+0000 7fb7ebfff640 1 -- 172.21.15.12:0/2244157629 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb80409db10 con 0x7fb804138230 2024-09-11T01:35:06.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.762+0000 7fb7ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:06.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:06.762+0000 7fb7ebfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:35:07.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.018+0000 7fb809a7f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:35:07.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.018+0000 7fb809a7f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:35:07.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.018+0000 7fb809a7f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:35:07.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.018+0000 7fb809a7f640 1 -- 172.21.15.12:0/2244157629 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb804138610 con 0x7fb804138230 2024-09-11T01:35:07.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.018+0000 7fb7ebfff640 1 -- 172.21.15.12:0/2244157629 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fb7fc074e20 con 0x7fb804138230 2024-09-11T01:35:07.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.018+0000 7fb7ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:35:07.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.018+0000 7fb7ebfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:35:07.021 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":64,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:35:07.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.020+0000 7fb809a7f640 1 -- 172.21.15.12:0/2244157629 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb7f003e260 msgr2=0x7fb7f0040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:07.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.020+0000 7fb809a7f640 1 --2- 172.21.15.12:0/2244157629 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb7f003e260 0x7fb7f0040720 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb7f4004f60 tx=0x7fb7f4007e30 comp rx=0 tx=0).stop 2024-09-11T01:35:07.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.020+0000 7fb809a7f640 10 monclient: shutdown 2024-09-11T01:35:07.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.020+0000 7fb809a7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:35:07.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.020+0000 7fb809a7f640 1 -- 172.21.15.12:0/2244157629 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb804138230 msgr2=0x7fb80413ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:07.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.020+0000 7fb809a7f640 1 --2- 172.21.15.12:0/2244157629 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb804138230 0x7fb80413ca40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc031000 tx=0x7fb7fc004a40 comp rx=0 tx=0).stop 2024-09-11T01:35:07.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.020+0000 7fb809a7f640 1 -- 172.21.15.12:0/2244157629 shutdown_connections 2024-09-11T01:35:07.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.020+0000 7fb809a7f640 1 --2- 172.21.15.12:0/2244157629 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb804138230 0x7fb80413ca40 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.020+0000 7fb809a7f640 1 --2- 172.21.15.12:0/2244157629 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb7f003e260 0x7fb7f0040720 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.020+0000 7fb809a7f640 1 --2- 172.21.15.12:0/2244157629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb80409d460 0x7fb804136e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.020+0000 7fb809a7f640 1 --2- 172.21.15.12:0/2244157629 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb80409d030 0x7fb80413f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.020+0000 7fb809a7f640 1 -- 172.21.15.12:0/2244157629 >> 172.21.15.12:0/2244157629 conn(0x7fb8040a7120 msgr2=0x7fb8040a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:35:07.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.021+0000 7fb809a7f640 1 -- 172.21.15.12:0/2244157629 shutdown_connections 2024-09-11T01:35:07.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.021+0000 7fb809a7f640 1 -- 172.21.15.12:0/2244157629 wait complete. 2024-09-11T01:35:07.033 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:35:07.033 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:35:07.033 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:35:07.033 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:35:07.051 INFO:tasks.rados.rados.0.smithi071.stdout:859: done (5 left) 2024-09-11T01:35:07.051 INFO:tasks.rados.rados.0.smithi071.stdout:860: copy_from oid 31 from oid 11 current snap is 92 2024-09-11T01:35:07.051 INFO:tasks.rados.rados.0.smithi071.stdout:854: finishing write tid 3 to smithi07134777-5 2024-09-11T01:35:07.051 INFO:tasks.rados.rados.0.smithi071.stdout:856: finishing rollback tid 1 to smithi07134777-27 2024-09-11T01:35:07.051 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 0 (ObjNum 945 snap 0 seq_num 3922266119) dirty dne 2024-09-11T01:35:07.051 INFO:tasks.rados.rados.0.smithi071.stdout:854: finishing write tid 4 to smithi07134777-5 2024-09-11T01:35:07.051 INFO:tasks.rados.rados.0.smithi071.stdout:854: finishing write tid 5 to smithi07134777-5 2024-09-11T01:35:07.051 INFO:tasks.rados.rados.0.smithi071.stdout:854: oid 5 updating version 0 to 149 2024-09-11T01:35:07.051 INFO:tasks.rados.rados.0.smithi071.stdout:854: oid 5 updating version 149 to 151 2024-09-11T01:35:07.051 INFO:tasks.rados.rados.0.smithi071.stdout:854: oid 5 version 151 is already newer than 148 2024-09-11T01:35:07.051 INFO:tasks.rados.rados.0.smithi071.stdout:854: oid 5 version 151 is already newer than 150 2024-09-11T01:35:07.052 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 151 (ObjNum 322 snap 92 seq_num 322) dirty exists 2024-09-11T01:35:07.052 INFO:tasks.rados.rados.0.smithi071.stdout:854: left oid 5 (ObjNum 322 snap 92 seq_num 322) 2024-09-11T01:35:07.052 INFO:tasks.rados.rados.0.smithi071.stdout:855: expect (ObjNum 290 snap 78 seq_num 290) 2024-09-11T01:35:07.101 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 616242 2024-09-11T01:35:07.136 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 60107 2024-09-11T01:35:07.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.138+0000 7f33c43cd640 10 monclient: get_monmap_and_config 2024-09-11T01:35:07.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.138+0000 7f33c43cd640 10 monclient: build_initial_monmap 2024-09-11T01:35:07.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.138+0000 7f33c43cd640 10 monclient: monmap: 2024-09-11T01:35:07.140 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:35:07.140 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:07.140 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:35:07.138999+0000 2024-09-11T01:35:07.140 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:35:07.138999+0000 2024-09-11T01:35:07.140 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:35:07.140 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:07.140 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:35:07.140 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:35:07.140 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:35:07.140 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:35:07.140 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:35:07.140 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:07.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.138+0000 7f33c43cd640 1 Processor -- start 2024-09-11T01:35:07.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.138+0000 7f33c43cd640 1 -- start start 2024-09-11T01:35:07.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.138+0000 7f33c43cd640 10 monclient: init 2024-09-11T01:35:07.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c43cd640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:07.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c43cd640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-11T01:35:07.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c43cd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f33bc137e50 0x7f33bc09d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:07.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c43cd640 10 monclient(hunting): picked mon.noname-b con 0x7f33bc137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:35:07.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c43cd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f33bc138230 0x7f33bc138630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:07.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c43cd640 10 monclient(hunting): picked mon.noname-a con 0x7f33bc138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:35:07.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c43cd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f33bc134cb0 addr v1:172.21.15.71:6789/0 2024-09-11T01:35:07.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c43cd640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33bc0a62d0 con 0x7f33bc134cb0 2024-09-11T01:35:07.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c43cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:07.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c43cd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f33bc0a65b0 con 0x7f33bc138230 2024-09-11T01:35:07.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c43cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:07.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c43cd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f33bc09bb90 con 0x7f33bc137e50 2024-09-11T01:35:07.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c43cd640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:07.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c43cd640 10 monclient(hunting): authenticate will time out at 1113.581689s 2024-09-11T01:35:07.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c1968640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f33bc138230 0x7f33bc138630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:07.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c2169640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f33bc137e50 0x7f33bc09d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:07.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c1968640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f33bc138230 0x7f33bc138630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45368/0 (socket says 172.21.15.12:45368) 2024-09-11T01:35:07.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c1968640 1 -- 172.21.15.12:0/1558681430 learned_addr learned my addr 172.21.15.12:0/1558681430 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:35:07.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c1968640 10 monclient(hunting): get_auth_request con 0x7f33bc138230 auth_method 0 2024-09-11T01:35:07.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c1968640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:07.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c1968640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:07.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c1968640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:07.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c2169640 10 monclient(hunting): get_auth_request con 0x7f33bc137e50 auth_method 0 2024-09-11T01:35:07.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c2169640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:07.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c2169640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:07.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c2169640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:07.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c1968640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:07.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c1968640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:07.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c1968640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:07.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c2169640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:07.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c2169640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:07.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.139+0000 7f33c2169640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:07.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.140+0000 7f33c1968640 10 monclient(hunting): handle_auth_done global_id 54225 payload 274 2024-09-11T01:35:07.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.140+0000 7f33c1968640 1 -- 172.21.15.12:0/1558681430 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f33bc137e50 msgr2=0x7f33bc09d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:35:07.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.140+0000 7f33c1968640 1 --2- 172.21.15.12:0/1558681430 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f33bc137e50 0x7f33bc09d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.140+0000 7f33c1968640 1 -- 172.21.15.12:0/1558681430 >> v1:172.21.15.71:6789/0 conn(0x7f33bc134cb0 legacy=0x7f33bc135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:35:07.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.140+0000 7f33c1968640 10 monclient: _finish_hunting 0 2024-09-11T01:35:07.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.140+0000 7f33c1968640 1 monclient: found mon.noname-a 2024-09-11T01:35:07.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.140+0000 7f33c1968640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:07.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.140+0000 7f33c1968640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:35:07.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.140+0000 7f33c1968640 1 -- 172.21.15.12:0/1558681430 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33bc09c4f0 con 0x7f33bc138230 2024-09-11T01:35:07.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.140+0000 7f33c2169640 1 --2- 172.21.15.12:0/1558681430 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f33bc137e50 0x7f33bc09d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:35:07.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.142+0000 7f33c1968640 1 -- 172.21.15.12:0/1558681430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc138230 msgr2=0x7f33bc138630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:07.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.142+0000 7f33c1968640 1 --2- 172.21.15.12:0/1558681430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc138230 0x7f33bc138630 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f33ac009870 tx=0x7f33ac02a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4dd63c2b20eb825 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:07.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c1167640 1 -- 172.21.15.12:0/1558681430 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33ac037070 con 0x7f33bc138230 2024-09-11T01:35:07.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c1167640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:07.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c1167640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:35:07.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c1167640 10 monclient: dump: 2024-09-11T01:35:07.152 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:07.152 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:07.152 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.152 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.152 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:07.152 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:07.152 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:07.152 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:07.153 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:07.153 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:07.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c1167640 10 monclient: _finish_auth 0 2024-09-11T01:35:07.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c1167640 10 monclient: _check_auth_tickets 2024-09-11T01:35:07.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c1167640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:07.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c43cd640 5 monclient: authenticate success, global_id 54225 2024-09-11T01:35:07.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c43cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:35:07.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c1167640 1 -- 172.21.15.12:0/1558681430 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33ac032440 con 0x7f33bc138230 2024-09-11T01:35:07.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c1167640 10 monclient: handle_config config(0 keys) 2024-09-11T01:35:07.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c43cd640 10 monclient: get_monmap_and_config success 2024-09-11T01:35:07.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c43cd640 4 set_mon_vals no callback set 2024-09-11T01:35:07.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c1167640 1 -- 172.21.15.12:0/1558681430 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33ac032810 con 0x7f33bc138230 2024-09-11T01:35:07.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c1167640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:07.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c1167640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:35:07.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c1167640 10 monclient: dump: 2024-09-11T01:35:07.154 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:07.154 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:07.154 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.154 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.154 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:07.154 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:07.154 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:07.154 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:07.154 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:07.154 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:07.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c43cd640 10 monclient: shutdown 2024-09-11T01:35:07.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c43cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:35:07.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c43cd640 1 -- 172.21.15.12:0/1558681430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc138230 msgr2=0x7f33bc138630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:07.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c43cd640 1 --2- 172.21.15.12:0/1558681430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc138230 0x7f33bc138630 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f33ac009870 tx=0x7f33ac02a0a0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c43cd640 1 -- 172.21.15.12:0/1558681430 shutdown_connections 2024-09-11T01:35:07.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c43cd640 1 --2- 172.21.15.12:0/1558681430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc138230 0x7f33bc138630 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c43cd640 1 --2- 172.21.15.12:0/1558681430 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f33bc137e50 0x7f33bc09d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c43cd640 1 -- 172.21.15.12:0/1558681430 >> 172.21.15.12:0/1558681430 conn(0x7f33bc0a7020 msgr2=0x7f33bc0a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:35:07.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c43cd640 1 -- 172.21.15.12:0/1558681430 shutdown_connections 2024-09-11T01:35:07.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.143+0000 7f33c43cd640 1 -- 172.21.15.12:0/1558681430 wait complete. 2024-09-11T01:35:07.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.144+0000 7f33c43cd640 10 monclient: build_initial_monmap 2024-09-11T01:35:07.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.144+0000 7f33c43cd640 10 monclient: monmap: 2024-09-11T01:35:07.155 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:35:07.155 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:35:07.155 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:35:07.156 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:35:07.156 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:35:07.156 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:07.156 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:35:07.156 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:35:07.156 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:35:07.156 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:07.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.144+0000 7f33c43cd640 1 Processor -- start 2024-09-11T01:35:07.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.144+0000 7f33c43cd640 1 -- start start 2024-09-11T01:35:07.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.144+0000 7f33c43cd640 10 monclient: init 2024-09-11T01:35:07.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c43cd640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:07.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c43cd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:35:07.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c43cd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f33bc134cb0 0x7f33bc1cd750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:07.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c43cd640 10 monclient(hunting): picked mon.noname-c con 0x7f33bc134cb0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:35:07.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c43cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc137e50 0x7f33bc134020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:07.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c43cd640 10 monclient(hunting): picked mon.noname-a con 0x7f33bc137e50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:35:07.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c43cd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33bc138230 0x7f33bc134560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:07.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c43cd640 10 monclient(hunting): picked mon.noname-b con 0x7f33bc138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:35:07.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c43cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:07.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c43cd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f33bc06f710 con 0x7f33bc137e50 2024-09-11T01:35:07.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c43cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:07.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c43cd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f33bc138b70 con 0x7f33bc138230 2024-09-11T01:35:07.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c43cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:07.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c43cd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f33bc06f890 con 0x7f33bc134cb0 2024-09-11T01:35:07.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c43cd640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:07.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c43cd640 10 monclient(hunting): authenticate will time out at 1113.587229s 2024-09-11T01:35:07.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c1968640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc137e50 0x7f33bc134020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:07.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c2169640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f33bc134cb0 0x7f33bc1cd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:07.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c1968640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc137e50 0x7f33bc134020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45382/0 (socket says 172.21.15.12:45382) 2024-09-11T01:35:07.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c1968640 1 -- 172.21.15.12:0/2870194624 learned_addr learned my addr 172.21.15.12:0/2870194624 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:35:07.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c1968640 10 monclient(hunting): get_auth_request con 0x7f33bc137e50 auth_method 0 2024-09-11T01:35:07.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c1968640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:07.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c1968640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:07.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c1968640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:07.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c2169640 10 monclient(hunting): get_auth_request con 0x7f33bc134cb0 auth_method 0 2024-09-11T01:35:07.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c2169640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:07.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c2169640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:07.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c2169640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:07.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c1968640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:07.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c1968640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:07.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c296a640 1 --2- 172.21.15.12:0/2870194624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33bc138230 0x7f33bc134560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:07.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c1968640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:07.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c2169640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:07.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c2169640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:07.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c2169640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:07.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c296a640 10 monclient(hunting): get_auth_request con 0x7f33bc138230 auth_method 0 2024-09-11T01:35:07.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c296a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:07.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c296a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:07.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c296a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:07.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c296a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:07.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c296a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:07.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c296a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:07.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c1968640 10 monclient(hunting): handle_auth_done global_id 54228 payload 824 2024-09-11T01:35:07.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c1968640 1 -- 172.21.15.12:0/2870194624 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f33bc134cb0 msgr2=0x7f33bc1cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:07.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c1968640 1 --2- 172.21.15.12:0/2870194624 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f33bc134cb0 0x7f33bc1cd750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c1968640 1 -- 172.21.15.12:0/2870194624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33bc138230 msgr2=0x7f33bc134560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:07.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c1968640 1 --2- 172.21.15.12:0/2870194624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33bc138230 0x7f33bc134560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c1968640 10 monclient: _finish_hunting 0 2024-09-11T01:35:07.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c1968640 1 monclient: found mon.noname-a 2024-09-11T01:35:07.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c1968640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:07.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c1968640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:35:07.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.145+0000 7f33c1968640 1 -- 172.21.15.12:0/2870194624 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33bc125740 con 0x7f33bc137e50 2024-09-11T01:35:07.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.147+0000 7f33c1968640 1 -- 172.21.15.12:0/2870194624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc137e50 msgr2=0x7f33bc134020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:07.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.147+0000 7f33c1968640 1 --2- 172.21.15.12:0/2870194624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc137e50 0x7f33bc134020 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f33ac0090b0 tx=0x7f33ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:07.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.147+0000 7f33b2ffd640 1 -- 172.21.15.12:0/2870194624 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33ac037040 con 0x7f33bc137e50 2024-09-11T01:35:07.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.148+0000 7f33b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:07.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.148+0000 7f33b2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:35:07.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.148+0000 7f33b2ffd640 10 monclient: dump: 2024-09-11T01:35:07.161 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:07.161 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:07.161 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.161 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.161 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:07.161 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:07.161 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:07.161 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:07.162 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:07.162 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:07.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.148+0000 7f33b2ffd640 10 monclient: _finish_auth 0 2024-09-11T01:35:07.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.148+0000 7f33b2ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:35:07.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.148+0000 7f33b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:07.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.148+0000 7f33b2ffd640 1 -- 172.21.15.12:0/2870194624 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33ac03b600 con 0x7f33bc137e50 2024-09-11T01:35:07.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.148+0000 7f33c43cd640 5 monclient: authenticate success, global_id 54228 2024-09-11T01:35:07.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.148+0000 7f33c43cd640 10 monclient: _renew_subs 2024-09-11T01:35:07.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.148+0000 7f33c43cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:35:07.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.148+0000 7f33c43cd640 1 -- 172.21.15.12:0/2870194624 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33bc124220 con 0x7f33bc137e50 2024-09-11T01:35:07.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.148+0000 7f33b2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:35:07.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.148+0000 7f33b2ffd640 1 -- 172.21.15.12:0/2870194624 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33ac03b7d0 con 0x7f33bc137e50 2024-09-11T01:35:07.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.148+0000 7f33b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:07.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.148+0000 7f33b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:35:07.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.148+0000 7f33b2ffd640 10 monclient: dump: 2024-09-11T01:35:07.163 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:07.163 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:07.163 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.163 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.163 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:07.163 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:07.163 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:07.163 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:07.163 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:07.163 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:07.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.148+0000 7f33c43cd640 10 monclient: _renew_subs 2024-09-11T01:35:07.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.148+0000 7f33c0966640 4 set_mon_vals no callback set 2024-09-11T01:35:07.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.148+0000 7f33c43cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:35:07.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.148+0000 7f33c43cd640 1 -- 172.21.15.12:0/2870194624 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33bc124800 con 0x7f33bc137e50 2024-09-11T01:35:07.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.148+0000 7f33c43cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:07.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.148+0000 7f33c43cd640 1 --2- 172.21.15.12:0/2870194624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc125180 0x7f33bc121c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:07.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.148+0000 7f33c43cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:07.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.148+0000 7f33c43cd640 1 -- 172.21.15.12:0/2870194624 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f33bc06a060 con 0x7f33bc125180 2024-09-11T01:35:07.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.149+0000 7f33c2169640 1 --2- 172.21.15.12:0/2870194624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc125180 0x7f33bc121c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:07.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.149+0000 7f33b2ffd640 1 -- 172.21.15.12:0/2870194624 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f33ac03a030 con 0x7f33bc137e50 2024-09-11T01:35:07.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.149+0000 7f33c2169640 10 monclient: get_auth_request con 0x7f33bc125180 auth_method 0 2024-09-11T01:35:07.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.149+0000 7f33c2169640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:07.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.149+0000 7f33c2169640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:07.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.149+0000 7f33c2169640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:07.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.149+0000 7f33c2169640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:07.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.149+0000 7f33c2169640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:07.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.149+0000 7f33c2169640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:07.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.149+0000 7f33c2169640 10 monclient(hunting): handle_auth_done global_id 54231 payload 824 2024-09-11T01:35:07.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.149+0000 7f33b2ffd640 1 --2- 172.21.15.12:0/2870194624 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f33a003e210 0x7f33a00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:07.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.149+0000 7f33c2169640 1 -- 172.21.15.12:0/2870194624 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f33bc135950 con 0x7f33bc125180 2024-09-11T01:35:07.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.149+0000 7f33b2ffd640 1 -- 172.21.15.12:0/2870194624 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(201..201 src has 1..201) ==== 4783+0+0 (secure 0 0 0) 0x7f33ac072970 con 0x7f33bc137e50 2024-09-11T01:35:07.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.149+0000 7f33c296a640 1 --2- 172.21.15.12:0/2870194624 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f33a003e210 0x7f33a00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:07.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.149+0000 7f33c296a640 10 monclient: get_auth_request con 0x7f33a003e210 auth_method 0 2024-09-11T01:35:07.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.151+0000 7f33c2169640 1 -- 172.21.15.12:0/2870194624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc125180 msgr2=0x7f33bc121c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:07.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.151+0000 7f33c2169640 1 --2- 172.21.15.12:0/2870194624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc125180 0x7f33bc121c00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f33b80118e0 tx=0x7f33b8011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:07.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.151+0000 7f33b2ffd640 1 -- 172.21.15.12:0/2870194624 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b8014440 con 0x7f33bc125180 2024-09-11T01:35:07.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.151+0000 7f33b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:07.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.151+0000 7f33b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:35:07.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.151+0000 7f33b2ffd640 10 monclient: dump: 2024-09-11T01:35:07.168 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:07.168 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:07.168 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.168 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.168 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:07.168 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:07.168 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:07.168 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:07.168 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:07.168 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:07.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.151+0000 7f33b2ffd640 1 -- 172.21.15.12:0/2870194624 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f33b8014440 con 0x7f33bc125180 2024-09-11T01:35:07.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.151+0000 7f33b2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:07.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.151+0000 7f33b2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:35:07.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.151+0000 7f33b2ffd640 1 -- 172.21.15.12:0/2870194624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc125180 msgr2=0x7f33bc121c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:07.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.151+0000 7f33b2ffd640 1 --2- 172.21.15.12:0/2870194624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc125180 0x7f33bc121c00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f33b80118e0 tx=0x7f33b8011da0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.151+0000 7f33b2ffd640 1 -- 172.21.15.12:0/2870194624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc125180 msgr2=0x7f33bc121c00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:35:07.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.152+0000 7f33b2ffd640 1 --2- 172.21.15.12:0/2870194624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc125180 0x7f33bc121c00 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f33b80118e0 tx=0x7f33b8011da0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.152+0000 7f33c296a640 1 --2- 172.21.15.12:0/2870194624 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f33a003e210 0x7f33a00406d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f33bc137d00 tx=0x7f33b4003d10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:07.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.158+0000 7f33c43cd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:35:07.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.158+0000 7f33c43cd640 1 --2- 172.21.15.12:0/2870194624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc122140 0x7f33bc122550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:07.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.158+0000 7f33c43cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:07.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.158+0000 7f33c43cd640 1 -- 172.21.15.12:0/2870194624 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f33bc122a90 con 0x7f33bc122140 2024-09-11T01:35:07.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.158+0000 7f33c2169640 1 --2- 172.21.15.12:0/2870194624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc122140 0x7f33bc122550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:07.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.158+0000 7f33c2169640 10 monclient: get_auth_request con 0x7f33bc122140 auth_method 0 2024-09-11T01:35:07.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.158+0000 7f33c2169640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:07.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.158+0000 7f33c2169640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:07.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.158+0000 7f33c2169640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:07.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.158+0000 7f33c2169640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:07.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.158+0000 7f33c2169640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:07.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.158+0000 7f33c2169640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:07.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.158+0000 7f33c2169640 10 monclient(hunting): handle_auth_done global_id 54234 payload 824 2024-09-11T01:35:07.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.158+0000 7f33c2169640 1 -- 172.21.15.12:0/2870194624 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f33bc136710 con 0x7f33bc122140 2024-09-11T01:35:07.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.160+0000 7f33c2169640 1 -- 172.21.15.12:0/2870194624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc122140 msgr2=0x7f33bc122550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:07.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.160+0000 7f33c2169640 1 --2- 172.21.15.12:0/2870194624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc122140 0x7f33bc122550 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f33b80118e0 tx=0x7f33b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:07.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.161+0000 7f33b2ffd640 1 -- 172.21.15.12:0/2870194624 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b8022980 con 0x7f33bc122140 2024-09-11T01:35:07.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.161+0000 7f33b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:07.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.161+0000 7f33b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:35:07.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.161+0000 7f33b2ffd640 10 monclient: dump: 2024-09-11T01:35:07.171 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:07.171 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:07.171 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.171 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.171 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:07.171 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:07.171 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:07.171 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:07.171 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:07.171 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:07.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.161+0000 7f33b2ffd640 1 -- 172.21.15.12:0/2870194624 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7f33b802b470 con 0x7f33bc122140 2024-09-11T01:35:07.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.161+0000 7f33b2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:35:07.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.161+0000 7f33b2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:35:07.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.161+0000 7f33b2ffd640 1 -- 172.21.15.12:0/2870194624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc122140 msgr2=0x7f33bc122550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:07.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.161+0000 7f33b2ffd640 1 --2- 172.21.15.12:0/2870194624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc122140 0x7f33bc122550 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f33b80118e0 tx=0x7f33b80046c0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.161+0000 7f33b2ffd640 1 -- 172.21.15.12:0/2870194624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc122140 msgr2=0x7f33bc122550 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:35:07.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.161+0000 7f33b2ffd640 1 --2- 172.21.15.12:0/2870194624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc122140 0x7f33bc122550 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f33b80118e0 tx=0x7f33b80046c0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.172 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:35:07.172 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:35:07.172 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:35:07.172 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:35:07.172 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 64, 2024-09-11T01:35:07.172 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 97142, 2024-09-11T01:35:07.173 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:35:07.173 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:35:07.173 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:35:07.173 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:35:07.173 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:07.173 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 5, 2024-09-11T01:35:07.173 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:35:07.173 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:35:07.173 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:35:07.173 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:35:07.173 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:35:07.173 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:35:07.173 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:35:07.174 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:35:07.174 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:35:07.174 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:35:07.174 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:35:07.174 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:35:07.174 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:35:07.174 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:35:07.174 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:07.174 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:35:07.174 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:35:07.174 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:35:07.174 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:35:07.174 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:35:07.174 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:35:07.175 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:35:07.175 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:35:07.175 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:35:07.175 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:35:07.175 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:35:07.175 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:35:07.175 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:35:07.175 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:07.175 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.175 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:35:07.175 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:35:07.175 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:35:07.175 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:35:07.175 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:35:07.176 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:35:07.176 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:35:07.176 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:35:07.176 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:35:07.176 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:35:07.176 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:35:07.176 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:35:07.176 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:35:07.176 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:35:07.176 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:35:07.176 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:35:07.176 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:35:07.176 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:35:07.177 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:35:07.177 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:35:07.177 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:35:07.177 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:35:07.177 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:35:07.177 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:35:07.177 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:35:07.177 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:35:07.177 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:35:07.177 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:35:07.177 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:07.177 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:35:07.177 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.177 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:35:07.178 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.178 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:35:07.178 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:35:07.178 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:35:07.178 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:35:07.178 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.178 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:35:07.178 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:35:07.178 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:07.178 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.178 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.178 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:35:07.178 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:35:07.178 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:07.179 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:07.179 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:07.179 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.179 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:35:07.179 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:35:07.179 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:35:07.179 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:35:07.179 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:35:07.179 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.179 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.179 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:35:07.179 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:35:07.179 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:35:07.180 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:35:07.180 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.180 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:35:07.180 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:35:07.180 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:07.180 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.180 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.180 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:35:07.180 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:35:07.180 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:07.180 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:07.181 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:07.181 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.181 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:35:07.181 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:35:07.181 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:35:07.181 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:35:07.181 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:35:07.181 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.181 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.181 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:35:07.181 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:35:07.181 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:35:07.182 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:35:07.182 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.182 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:35:07.182 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:35:07.182 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:07.182 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.182 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.182 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:35:07.182 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:35:07.182 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:07.182 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:07.182 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:07.182 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.182 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:35:07.183 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:35:07.183 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:35:07.183 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:35:07.183 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:35:07.183 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:07.183 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:07.183 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.183 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:35:07.183 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:35:07.183 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.183 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:35:07.183 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:35:07.183 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:35:07.183 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:07.184 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:07.184 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:35:07.184 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.184 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:35:07.184 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:35:07.184 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:35:07.184 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:07.184 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:07.184 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:35:07.184 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.184 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:35:07.184 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:35:07.184 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:35:07.184 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:07.185 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:07.185 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.185 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:35:07.185 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:35:07.186 INFO:tasks.rados.rados.0.smithi071.stdout:857: finishing write tid 1 to smithi07134777-20 2024-09-11T01:35:07.186 INFO:tasks.rados.rados.0.smithi071.stdout:857: finishing write tid 2 to smithi07134777-20 2024-09-11T01:35:07.186 INFO:tasks.rados.rados.0.smithi071.stdout:854: done (5 left) 2024-09-11T01:35:07.186 INFO:tasks.rados.rados.0.smithi071.stdout:855: done (4 left) 2024-09-11T01:35:07.187 INFO:tasks.rados.rados.0.smithi071.stdout:856: done (3 left) 2024-09-11T01:35:07.187 INFO:tasks.rados.rados.0.smithi071.stdout:861: write (excl) oid 45 current snap is 92 2024-09-11T01:35:07.187 INFO:tasks.rados.rados.0.smithi071.stdout:861: seq_num 325 ranges {662275=542751,1785236=425447,2942002=520273} 2024-09-11T01:35:07.187 INFO:tasks.rados.rados.0.smithi071.stdout:861: writing smithi07134777-45 from 662275 to 1205026 tid 1 2024-09-11T01:35:07.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.161+0000 7f33c43cd640 1 -- 172.21.15.12:0/2870194624 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f33a003e210 msgr2=0x7f33a00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:07.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.161+0000 7f33c43cd640 1 --2- 172.21.15.12:0/2870194624 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f33a003e210 0x7f33a00406d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f33bc137d00 tx=0x7f33b4003d10 comp rx=0 tx=0).stop 2024-09-11T01:35:07.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.161+0000 7f33c43cd640 10 monclient: shutdown 2024-09-11T01:35:07.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.161+0000 7f33c43cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:35:07.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.161+0000 7f33c43cd640 1 -- 172.21.15.12:0/2870194624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc137e50 msgr2=0x7f33bc134020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:07.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.161+0000 7f33c43cd640 1 --2- 172.21.15.12:0/2870194624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33bc137e50 0x7f33bc134020 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f33ac0090b0 tx=0x7f33ac0046c0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.161+0000 7f33c296a640 1 -- 172.21.15.12:0/2870194624 reap_dead start 2024-09-11T01:35:07.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.161+0000 7f33c296a640 1 -- 172.21.15.12:0/2870194624 reap_dead start 2024-09-11T01:35:07.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.161+0000 7f33c43cd640 1 -- 172.21.15.12:0/2870194624 shutdown_connections 2024-09-11T01:35:07.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.161+0000 7f33c43cd640 1 -- 172.21.15.12:0/2870194624 >> 172.21.15.12:0/2870194624 conn(0x7f33bc0a7020 msgr2=0x7f33bc136000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:35:07.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.161+0000 7f33c43cd640 1 -- 172.21.15.12:0/2870194624 shutdown_connections 2024-09-11T01:35:07.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.161+0000 7f33c43cd640 1 -- 172.21.15.12:0/2870194624 wait complete. 2024-09-11T01:35:07.189 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:35:07.189 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:35:07.189 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:35:07.190 INFO:tasks.rados.rados.0.smithi071.stdout:861: writing smithi07134777-45 from 1785236 to 2210683 tid 2 2024-09-11T01:35:07.197 INFO:tasks.rados.rados.0.smithi071.stdout:861: writing smithi07134777-45 from 2942002 to 3462275 tid 3 2024-09-11T01:35:07.198 INFO:tasks.rados.rados.0.smithi071.stdout:862: write oid 8 current snap is 92 2024-09-11T01:35:07.198 INFO:tasks.rados.rados.0.smithi071.stdout:857: finishing write tid 3 to smithi07134777-20 2024-09-11T01:35:07.198 INFO:tasks.rados.rados.0.smithi071.stdout:857: finishing write tid 4 to smithi07134777-20 2024-09-11T01:35:07.198 INFO:tasks.rados.rados.0.smithi071.stdout:857: finishing write tid 5 to smithi07134777-20 2024-09-11T01:35:07.198 INFO:tasks.rados.rados.0.smithi071.stdout:857: oid 20 updating version 0 to 243 2024-09-11T01:35:07.198 INFO:tasks.rados.rados.0.smithi071.stdout:857: oid 20 version 243 is already newer than 241 2024-09-11T01:35:07.198 INFO:tasks.rados.rados.0.smithi071.stdout:857: oid 20 version 243 is already newer than 242 2024-09-11T01:35:07.198 INFO:tasks.rados.rados.0.smithi071.stdout:857: oid 20 updating version 243 to 244 2024-09-11T01:35:07.198 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 244 (ObjNum 323 snap 92 seq_num 323) dirty exists 2024-09-11T01:35:07.198 INFO:tasks.rados.rados.0.smithi071.stdout:857: left oid 20 (ObjNum 323 snap 92 seq_num 323) 2024-09-11T01:35:07.198 INFO:tasks.rados.rados.0.smithi071.stdout:858: finishing write tid 1 to smithi07134777-2 2024-09-11T01:35:07.198 INFO:tasks.rados.rados.0.smithi071.stdout:862: seq_num 326 ranges {679082=618000,1843718=569105,2853072=626010} 2024-09-11T01:35:07.204 INFO:tasks.rados.rados.0.smithi071.stdout:862: writing smithi07134777-8 from 679082 to 1297082 tid 1 2024-09-11T01:35:07.210 INFO:tasks.rados.rados.0.smithi071.stdout:862: writing smithi07134777-8 from 1843718 to 2412823 tid 2 2024-09-11T01:35:07.216 INFO:tasks.rados.rados.0.smithi071.stdout:862: writing smithi07134777-8 from 2853072 to 3479082 tid 3 2024-09-11T01:35:07.216 INFO:tasks.rados.rados.0.smithi071.stdout:857: done (4 left) 2024-09-11T01:35:07.216 INFO:tasks.rados.rados.0.smithi071.stdout:863: write oid 25 current snap is 92 2024-09-11T01:35:07.217 INFO:tasks.rados.rados.0.smithi071.stdout:858: finishing write tid 2 to smithi07134777-2 2024-09-11T01:35:07.217 INFO:tasks.rados.rados.0.smithi071.stdout:858: finishing write tid 3 to smithi07134777-2 2024-09-11T01:35:07.217 INFO:tasks.rados.rados.0.smithi071.stdout:858: finishing write tid 4 to smithi07134777-2 2024-09-11T01:35:07.217 INFO:tasks.rados.rados.0.smithi071.stdout:858: finishing write tid 5 to smithi07134777-2 2024-09-11T01:35:07.217 INFO:tasks.rados.rados.0.smithi071.stdout:858: oid 2 updating version 0 to 147 2024-09-11T01:35:07.217 INFO:tasks.rados.rados.0.smithi071.stdout:858: oid 2 version 147 is already newer than 145 2024-09-11T01:35:07.217 INFO:tasks.rados.rados.0.smithi071.stdout:858: oid 2 version 147 is already newer than 146 2024-09-11T01:35:07.217 INFO:tasks.rados.rados.0.smithi071.stdout:858: oid 2 version 147 is already newer than 144 2024-09-11T01:35:07.217 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 147 (ObjNum 324 snap 92 seq_num 324) dirty exists 2024-09-11T01:35:07.217 INFO:tasks.rados.rados.0.smithi071.stdout:858: left oid 2 (ObjNum 324 snap 92 seq_num 324) 2024-09-11T01:35:07.217 INFO:tasks.rados.rados.0.smithi071.stdout:860: finishing copy_from racing read to smithi07134777-31 2024-09-11T01:35:07.217 INFO:tasks.rados.rados.0.smithi071.stdout:860: finishing copy_from to smithi07134777-31 2024-09-11T01:35:07.217 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 258 (ObjNum 263 snap 70 seq_num 263) dirty exists 2024-09-11T01:35:07.217 INFO:tasks.rados.rados.0.smithi071.stdout:863: seq_num 327 ranges {695889=409602,1618553=712763,2996848=499041} 2024-09-11T01:35:07.222 INFO:tasks.rados.rados.0.smithi071.stdout:863: writing smithi07134777-25 from 695889 to 1105491 tid 1 2024-09-11T01:35:07.228 INFO:tasks.rados.rados.0.smithi071.stdout:863: writing smithi07134777-25 from 1618553 to 2331316 tid 2 2024-09-11T01:35:07.235 INFO:tasks.rados.rados.0.smithi071.stdout:863: writing smithi07134777-25 from 2996848 to 3495889 tid 3 2024-09-11T01:35:07.235 INFO:tasks.rados.rados.0.smithi071.stdout:858: done (4 left) 2024-09-11T01:35:07.235 INFO:tasks.rados.rados.0.smithi071.stdout:860: done (3 left) 2024-09-11T01:35:07.235 INFO:tasks.rados.rados.0.smithi071.stdout:864: snap_create 2024-09-11T01:35:07.235 INFO:tasks.rados.rados.0.smithi071.stdout:861: finishing write tid 1 to smithi07134777-45 2024-09-11T01:35:07.235 INFO:tasks.rados.rados.0.smithi071.stdout:861: finishing write tid 2 to smithi07134777-45 2024-09-11T01:35:07.235 INFO:tasks.rados.rados.0.smithi071.stdout:861: finishing write tid 3 to smithi07134777-45 2024-09-11T01:35:07.236 INFO:tasks.rados.rados.0.smithi071.stdout:861: finishing write tid 4 to smithi07134777-45 2024-09-11T01:35:07.236 INFO:tasks.rados.rados.0.smithi071.stdout:861: finishing write tid 5 to smithi07134777-45 2024-09-11T01:35:07.236 INFO:tasks.rados.rados.0.smithi071.stdout:861: oid 45 updating version 0 to 280 2024-09-11T01:35:07.236 INFO:tasks.rados.rados.0.smithi071.stdout:861: oid 45 updating version 280 to 281 2024-09-11T01:35:07.236 INFO:tasks.rados.rados.0.smithi071.stdout:861: oid 45 updating version 281 to 282 2024-09-11T01:35:07.236 INFO:tasks.rados.rados.0.smithi071.stdout:861: oid 45 updating version 282 to 283 2024-09-11T01:35:07.236 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 283 (ObjNum 325 snap 92 seq_num 325) dirty exists 2024-09-11T01:35:07.236 INFO:tasks.rados.rados.0.smithi071.stdout:861: left oid 45 (ObjNum 325 snap 92 seq_num 325) 2024-09-11T01:35:07.236 INFO:tasks.rados.rados.0.smithi071.stdout:862: finishing write tid 1 to smithi07134777-8 2024-09-11T01:35:07.236 INFO:tasks.rados.rados.0.smithi071.stdout:862: finishing write tid 2 to smithi07134777-8 2024-09-11T01:35:07.236 INFO:tasks.rados.rados.0.smithi071.stdout:862: finishing write tid 3 to smithi07134777-8 2024-09-11T01:35:07.236 INFO:tasks.rados.rados.0.smithi071.stdout:862: finishing write tid 4 to smithi07134777-8 2024-09-11T01:35:07.236 INFO:tasks.rados.rados.0.smithi071.stdout:862: finishing write tid 5 to smithi07134777-8 2024-09-11T01:35:07.236 INFO:tasks.rados.rados.0.smithi071.stdout:862: oid 8 updating version 0 to 140 2024-09-11T01:35:07.237 INFO:tasks.rados.rados.0.smithi071.stdout:862: oid 8 version 140 is already newer than 138 2024-09-11T01:35:07.237 INFO:tasks.rados.rados.0.smithi071.stdout:862: oid 8 version 140 is already newer than 139 2024-09-11T01:35:07.237 INFO:tasks.rados.rados.0.smithi071.stdout:862: oid 8 updating version 140 to 141 2024-09-11T01:35:07.237 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 141 (ObjNum 326 snap 92 seq_num 326) dirty exists 2024-09-11T01:35:07.237 INFO:tasks.rados.rados.0.smithi071.stdout:862: left oid 8 (ObjNum 326 snap 92 seq_num 326) 2024-09-11T01:35:07.237 INFO:tasks.rados.rados.0.smithi071.stdout:863: finishing write tid 1 to smithi07134777-25 2024-09-11T01:35:07.237 INFO:tasks.rados.rados.0.smithi071.stdout:863: finishing write tid 2 to smithi07134777-25 2024-09-11T01:35:07.238 INFO:tasks.rados.rados.0.smithi071.stdout:863: finishing write tid 3 to smithi07134777-25 2024-09-11T01:35:07.238 INFO:tasks.rados.rados.0.smithi071.stdout:863: finishing write tid 4 to smithi07134777-25 2024-09-11T01:35:07.238 INFO:tasks.rados.rados.0.smithi071.stdout:863: finishing write tid 5 to smithi07134777-25 2024-09-11T01:35:07.238 INFO:tasks.rados.rados.0.smithi071.stdout:863: oid 25 updating version 0 to 99 2024-09-11T01:35:07.238 INFO:tasks.rados.rados.0.smithi071.stdout:863: oid 25 version 99 is already newer than 98 2024-09-11T01:35:07.238 INFO:tasks.rados.rados.0.smithi071.stdout:863: oid 25 version 99 is already newer than 97 2024-09-11T01:35:07.238 INFO:tasks.rados.rados.0.smithi071.stdout:863: oid 25 updating version 99 to 100 2024-09-11T01:35:07.238 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 100 (ObjNum 327 snap 92 seq_num 327) dirty exists 2024-09-11T01:35:07.238 INFO:tasks.rados.rados.0.smithi071.stdout:863: left oid 25 (ObjNum 327 snap 92 seq_num 327) 2024-09-11T01:35:07.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.294+0000 7f181000f640 10 monclient: get_monmap_and_config 2024-09-11T01:35:07.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.294+0000 7f181000f640 10 monclient: build_initial_monmap 2024-09-11T01:35:07.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.294+0000 7f181000f640 10 monclient: monmap: 2024-09-11T01:35:07.296 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:35:07.296 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:07.296 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:35:07.295025+0000 2024-09-11T01:35:07.296 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:35:07.295025+0000 2024-09-11T01:35:07.296 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:35:07.296 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:07.296 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:35:07.296 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:35:07.296 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:35:07.296 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:35:07.296 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:35:07.297 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:07.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.294+0000 7f181000f640 1 Processor -- start 2024-09-11T01:35:07.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.295+0000 7f181000f640 1 -- start start 2024-09-11T01:35:07.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.295+0000 7f181000f640 10 monclient: init 2024-09-11T01:35:07.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.295+0000 7f181000f640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:07.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.295+0000 7f181000f640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-11T01:35:07.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.295+0000 7f181000f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1808137e50 0x7f180809d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:07.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.295+0000 7f181000f640 10 monclient(hunting): picked mon.noname-a con 0x7f1808137e50 addr v2:172.21.15.12:3300/0 2024-09-11T01:35:07.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.295+0000 7f181000f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1808138230 addr v1:172.21.15.71:6789/0 2024-09-11T01:35:07.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.295+0000 7f181000f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f180813d380 addr v1:172.21.15.12:6789/0 2024-09-11T01:35:07.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.295+0000 7f181000f640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18080a62d0 con 0x7f180813d380 2024-09-11T01:35:07.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.295+0000 7f181000f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f180809bb40 con 0x7f1808138230 2024-09-11T01:35:07.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.295+0000 7f181000f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:07.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.295+0000 7f181000f640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f180809c4a0 con 0x7f1808137e50 2024-09-11T01:35:07.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.295+0000 7f181000f640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:07.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.295+0000 7f181000f640 10 monclient(hunting): authenticate will time out at 1113.737655s 2024-09-11T01:35:07.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.295+0000 7f180ddab640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1808137e50 0x7f180809d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:07.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.295+0000 7f180ddab640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1808137e50 0x7f180809d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45404/0 (socket says 172.21.15.12:45404) 2024-09-11T01:35:07.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.295+0000 7f180e5ac640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f180813d380 0x7f180813f770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46232/0 (socket says 172.21.15.12:46232) 2024-09-11T01:35:07.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.295+0000 7f180ddab640 1 -- 172.21.15.12:0/3553922529 learned_addr learned my addr 172.21.15.12:0/3553922529 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:35:07.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.295+0000 7f180ddab640 10 monclient(hunting): get_auth_request con 0x7f1808137e50 auth_method 0 2024-09-11T01:35:07.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.295+0000 7f180ddab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:07.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.295+0000 7f180ddab640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:07.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.295+0000 7f180ddab640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:07.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.295+0000 7f180ddab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:07.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.295+0000 7f180ddab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:07.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.295+0000 7f180ddab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:07.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.296+0000 7f180ddab640 10 monclient(hunting): handle_auth_done global_id 54237 payload 274 2024-09-11T01:35:07.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.296+0000 7f180ddab640 1 -- 172.21.15.12:0/3553922529 >> v1:172.21.15.71:6789/0 conn(0x7f1808138230 legacy=0x7f1808138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:35:07.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.296+0000 7f180ddab640 1 -- 172.21.15.12:0/3553922529 >> v1:172.21.15.12:6789/0 conn(0x7f180813d380 legacy=0x7f180813f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:35:07.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.296+0000 7f180ddab640 10 monclient: _finish_hunting 0 2024-09-11T01:35:07.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.296+0000 7f180ddab640 1 monclient: found mon.noname-a 2024-09-11T01:35:07.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.296+0000 7f180ddab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:07.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.296+0000 7f180ddab640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:35:07.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.296+0000 7f180ddab640 1 -- 172.21.15.12:0/3553922529 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f180809c780 con 0x7f1808137e50 2024-09-11T01:35:07.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.298+0000 7f180ddab640 1 -- 172.21.15.12:0/3553922529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1808137e50 msgr2=0x7f180809d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:07.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.298+0000 7f180ddab640 1 --2- 172.21.15.12:0/3553922529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1808137e50 0x7f180809d160 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f180809cf30 tx=0x7f180402a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46fa6f3e2b523153 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:07.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.299+0000 7f180cda9640 1 -- 172.21.15.12:0/3553922529 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1804037070 con 0x7f1808137e50 2024-09-11T01:35:07.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.299+0000 7f180cda9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:07.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.299+0000 7f180cda9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:35:07.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.299+0000 7f180cda9640 10 monclient: dump: 2024-09-11T01:35:07.308 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:07.308 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:07.308 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.308 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.308 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:07.308 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:07.308 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:07.308 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:07.308 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:07.309 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:07.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.299+0000 7f180cda9640 10 monclient: _finish_auth 0 2024-09-11T01:35:07.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.299+0000 7f180cda9640 10 monclient: _check_auth_tickets 2024-09-11T01:35:07.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.299+0000 7f180cda9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:07.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.299+0000 7f180cda9640 1 -- 172.21.15.12:0/3553922529 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1804032640 con 0x7f1808137e50 2024-09-11T01:35:07.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.299+0000 7f181000f640 5 monclient: authenticate success, global_id 54237 2024-09-11T01:35:07.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.299+0000 7f181000f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:35:07.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.299+0000 7f180cda9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:35:07.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.299+0000 7f181000f640 10 monclient: get_monmap_and_config success 2024-09-11T01:35:07.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.299+0000 7f180cda9640 1 -- 172.21.15.12:0/3553922529 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1804032a10 con 0x7f1808137e50 2024-09-11T01:35:07.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.299+0000 7f180cda9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:07.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.299+0000 7f181000f640 4 set_mon_vals no callback set 2024-09-11T01:35:07.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.299+0000 7f180cda9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:35:07.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.299+0000 7f180cda9640 10 monclient: dump: 2024-09-11T01:35:07.310 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:07.310 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:07.310 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.310 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.310 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:07.310 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:07.310 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:07.310 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:07.310 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:07.310 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:07.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.299+0000 7f181000f640 10 monclient: shutdown 2024-09-11T01:35:07.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.299+0000 7f181000f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:35:07.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.299+0000 7f181000f640 1 -- 172.21.15.12:0/3553922529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1808137e50 msgr2=0x7f180809d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:07.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.299+0000 7f181000f640 1 --2- 172.21.15.12:0/3553922529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1808137e50 0x7f180809d160 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f180809cf30 tx=0x7f180402a060 comp rx=0 tx=0).stop 2024-09-11T01:35:07.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.299+0000 7f181000f640 1 -- 172.21.15.12:0/3553922529 shutdown_connections 2024-09-11T01:35:07.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.299+0000 7f181000f640 1 --2- 172.21.15.12:0/3553922529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1808137e50 0x7f180809d160 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.299+0000 7f181000f640 1 -- 172.21.15.12:0/3553922529 >> 172.21.15.12:0/3553922529 conn(0x7f18080a7020 msgr2=0x7f18080a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:35:07.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.299+0000 7f181000f640 1 -- 172.21.15.12:0/3553922529 shutdown_connections 2024-09-11T01:35:07.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.299+0000 7f181000f640 1 -- 172.21.15.12:0/3553922529 wait complete. 2024-09-11T01:35:07.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.300+0000 7f181000f640 10 monclient: build_initial_monmap 2024-09-11T01:35:07.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.300+0000 7f181000f640 10 monclient: monmap: 2024-09-11T01:35:07.311 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:35:07.311 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:35:07.311 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:35:07.311 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:35:07.311 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:35:07.312 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:07.312 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:35:07.312 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:35:07.312 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:35:07.312 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:07.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.300+0000 7f181000f640 1 Processor -- start 2024-09-11T01:35:07.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.300+0000 7f181000f640 1 -- start start 2024-09-11T01:35:07.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.300+0000 7f181000f640 10 monclient: init 2024-09-11T01:35:07.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f181000f640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:07.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f181000f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:35:07.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f181000f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1808137e50 0x7f18080a3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:07.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f181000f640 10 monclient(hunting): picked mon.noname-c con 0x7f1808137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:35:07.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f181000f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1808138230 0x7f1808136e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:07.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f181000f640 10 monclient(hunting): picked mon.noname-a con 0x7f1808138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:35:07.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f181000f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f180813d380 0x7f18080a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:07.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f181000f640 10 monclient(hunting): picked mon.noname-b con 0x7f180813d380 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:35:07.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f181000f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:07.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f181000f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f180806f890 con 0x7f1808138230 2024-09-11T01:35:07.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f181000f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:07.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f181000f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f180806f710 con 0x7f180813d380 2024-09-11T01:35:07.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f181000f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:07.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f181000f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1808140e90 con 0x7f1808137e50 2024-09-11T01:35:07.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f181000f640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:07.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f181000f640 10 monclient(hunting): authenticate will time out at 1113.743093s 2024-09-11T01:35:07.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180ddab640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1808137e50 0x7f18080a3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:07.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180ddab640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1808137e50 0x7f18080a3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:58050/0 (socket says 172.21.15.12:58050) 2024-09-11T01:35:07.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180d5aa640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1808138230 0x7f1808136e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:07.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180ddab640 1 -- 172.21.15.12:0/1052400043 learned_addr learned my addr 172.21.15.12:0/1052400043 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:35:07.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180ddab640 10 monclient(hunting): get_auth_request con 0x7f1808137e50 auth_method 0 2024-09-11T01:35:07.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180ddab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:07.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180ddab640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:07.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180ddab640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:07.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180d5aa640 10 monclient(hunting): get_auth_request con 0x7f1808138230 auth_method 0 2024-09-11T01:35:07.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180d5aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:07.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180d5aa640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:07.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180d5aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:07.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180ddab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:07.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180ddab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:07.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180ddab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:07.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180d5aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:07.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180d5aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:07.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180d5aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:07.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180e5ac640 1 --2- 172.21.15.12:0/1052400043 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f180813d380 0x7f18080a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:07.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180e5ac640 10 monclient(hunting): get_auth_request con 0x7f180813d380 auth_method 0 2024-09-11T01:35:07.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180e5ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:07.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180e5ac640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:07.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180e5ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:07.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180ddab640 10 monclient(hunting): handle_auth_done global_id 64142 payload 824 2024-09-11T01:35:07.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180ddab640 1 -- 172.21.15.12:0/1052400043 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f180813d380 msgr2=0x7f18080a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:07.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180ddab640 1 --2- 172.21.15.12:0/1052400043 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f180813d380 0x7f18080a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180ddab640 1 -- 172.21.15.12:0/1052400043 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1808138230 msgr2=0x7f1808136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:07.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180ddab640 1 --2- 172.21.15.12:0/1052400043 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1808138230 0x7f1808136e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180ddab640 10 monclient: _finish_hunting 0 2024-09-11T01:35:07.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180ddab640 1 monclient: found mon.noname-c 2024-09-11T01:35:07.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180ddab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:07.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180ddab640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:35:07.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180ddab640 1 -- 172.21.15.12:0/1052400043 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18081cde70 con 0x7f1808137e50 2024-09-11T01:35:07.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.301+0000 7f180e5ac640 1 --2- 172.21.15.12:0/1052400043 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f180813d380 0x7f18080a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:35:07.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.303+0000 7f180ddab640 1 -- 172.21.15.12:0/1052400043 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1808137e50 msgr2=0x7f18080a3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:07.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.303+0000 7f180ddab640 1 --2- 172.21.15.12:0/1052400043 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1808137e50 0x7f18080a3030 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f18040090b0 tx=0x7f18040046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:07.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.303+0000 7f17fe7fc640 1 -- 172.21.15.12:0/1052400043 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1804041070 con 0x7f1808137e50 2024-09-11T01:35:07.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.303+0000 7f17fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:07.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.303+0000 7f17fe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:35:07.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.303+0000 7f17fe7fc640 10 monclient: dump: 2024-09-11T01:35:07.317 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:07.317 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:07.317 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.317 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.317 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:07.317 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:07.317 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:07.317 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:07.317 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:07.317 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:07.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.303+0000 7f17fe7fc640 10 monclient: _finish_auth 0 2024-09-11T01:35:07.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.303+0000 7f17fe7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:35:07.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.303+0000 7f17fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:07.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.303+0000 7f17fe7fc640 1 -- 172.21.15.12:0/1052400043 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f180403c880 con 0x7f1808137e50 2024-09-11T01:35:07.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.303+0000 7f17fe7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:35:07.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.303+0000 7f17fe7fc640 1 -- 172.21.15.12:0/1052400043 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1804037040 con 0x7f1808137e50 2024-09-11T01:35:07.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.303+0000 7f181000f640 5 monclient: authenticate success, global_id 64142 2024-09-11T01:35:07.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.303+0000 7f181000f640 10 monclient: _renew_subs 2024-09-11T01:35:07.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.303+0000 7f181000f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:35:07.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.303+0000 7f181000f640 1 -- 172.21.15.12:0/1052400043 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18080a39c0 con 0x7f1808137e50 2024-09-11T01:35:07.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.303+0000 7f17fffff640 4 set_mon_vals no callback set 2024-09-11T01:35:07.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.304+0000 7f17fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:07.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.304+0000 7f17fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:35:07.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.304+0000 7f17fe7fc640 10 monclient: dump: 2024-09-11T01:35:07.318 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:07.319 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:07.319 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.319 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.319 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:07.319 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:07.319 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:07.319 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:07.319 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:07.319 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:07.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.304+0000 7f181000f640 10 monclient: _renew_subs 2024-09-11T01:35:07.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.304+0000 7f181000f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:35:07.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.304+0000 7f181000f640 1 -- 172.21.15.12:0/1052400043 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f18080a3f90 con 0x7f1808137e50 2024-09-11T01:35:07.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.304+0000 7f181000f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:07.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.304+0000 7f181000f640 1 --2- 172.21.15.12:0/1052400043 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18080a1c50 0x7f18080a2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:07.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.304+0000 7f181000f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:07.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.304+0000 7f181000f640 1 -- 172.21.15.12:0/1052400043 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f180809d160 con 0x7f18080a1c50 2024-09-11T01:35:07.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.304+0000 7f17fe7fc640 1 -- 172.21.15.12:0/1052400043 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f180403ccd0 con 0x7f1808137e50 2024-09-11T01:35:07.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.304+0000 7f180d5aa640 1 --2- 172.21.15.12:0/1052400043 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18080a1c50 0x7f18080a2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:07.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.304+0000 7f180d5aa640 10 monclient: get_auth_request con 0x7f18080a1c50 auth_method 0 2024-09-11T01:35:07.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.304+0000 7f180d5aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:07.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.304+0000 7f180d5aa640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:07.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.304+0000 7f180d5aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:07.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.304+0000 7f180d5aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:07.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.304+0000 7f180d5aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:07.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.305+0000 7f180d5aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:07.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.305+0000 7f17fe7fc640 1 --2- 172.21.15.12:0/1052400043 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f17ec03e240 0x7f17ec040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:07.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.305+0000 7f17fe7fc640 1 -- 172.21.15.12:0/1052400043 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(201..201 src has 1..201) ==== 4783+0+0 (secure 0 0 0) 0x7f1804073550 con 0x7f1808137e50 2024-09-11T01:35:07.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.305+0000 7f180d5aa640 10 monclient(hunting): handle_auth_done global_id 64145 payload 824 2024-09-11T01:35:07.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.305+0000 7f180d5aa640 1 -- 172.21.15.12:0/1052400043 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f18080a43b0 con 0x7f18080a1c50 2024-09-11T01:35:07.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.305+0000 7f180e5ac640 1 --2- 172.21.15.12:0/1052400043 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f17ec03e240 0x7f17ec040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:07.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.305+0000 7f180e5ac640 10 monclient: get_auth_request con 0x7f17ec03e240 auth_method 0 2024-09-11T01:35:07.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.307+0000 7f180d5aa640 1 -- 172.21.15.12:0/1052400043 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18080a1c50 msgr2=0x7f18080a2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:07.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.307+0000 7f180d5aa640 1 --2- 172.21.15.12:0/1052400043 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18080a1c50 0x7f18080a2030 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f18080a2fc0 tx=0x7f17f800fc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:07.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.307+0000 7f17fe7fc640 1 -- 172.21.15.12:0/1052400043 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f8018030 con 0x7f18080a1c50 2024-09-11T01:35:07.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.307+0000 7f17fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:07.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.307+0000 7f17fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:35:07.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.307+0000 7f17fe7fc640 10 monclient: dump: 2024-09-11T01:35:07.324 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:07.324 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:07.324 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.324 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.324 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:07.324 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:07.324 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:07.325 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:07.325 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:07.325 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:07.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.307+0000 7f17fe7fc640 1 -- 172.21.15.12:0/1052400043 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f17f8003940 con 0x7f18080a1c50 2024-09-11T01:35:07.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.307+0000 7f17fe7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:07.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.307+0000 7f17fe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:35:07.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.307+0000 7f17fe7fc640 1 -- 172.21.15.12:0/1052400043 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18080a1c50 msgr2=0x7f18080a2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:07.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.307+0000 7f17fe7fc640 1 --2- 172.21.15.12:0/1052400043 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18080a1c50 0x7f18080a2030 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f18080a2fc0 tx=0x7f17f800fc00 comp rx=0 tx=0).stop 2024-09-11T01:35:07.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.307+0000 7f17fe7fc640 1 -- 172.21.15.12:0/1052400043 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18080a1c50 msgr2=0x7f18080a2030 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:35:07.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.307+0000 7f17fe7fc640 1 --2- 172.21.15.12:0/1052400043 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18080a1c50 0x7f18080a2030 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.307+0000 7f180e5ac640 1 --2- 172.21.15.12:0/1052400043 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f17ec03e240 0x7f17ec040700 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1800005710 tx=0x7f1800009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:07.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.313+0000 7f181000f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:35:07.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.313+0000 7f181000f640 1 --2- 172.21.15.12:0/1052400043 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18080a2570 0x7f180809f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:07.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.313+0000 7f181000f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:07.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.313+0000 7f181000f640 1 -- 172.21.15.12:0/1052400043 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1808069950 con 0x7f18080a2570 2024-09-11T01:35:07.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.314+0000 7f180d5aa640 1 --2- 172.21.15.12:0/1052400043 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18080a2570 0x7f180809f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:07.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.314+0000 7f180d5aa640 10 monclient: get_auth_request con 0x7f18080a2570 auth_method 0 2024-09-11T01:35:07.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.314+0000 7f180d5aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:07.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.314+0000 7f180d5aa640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:07.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.314+0000 7f180d5aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:07.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.314+0000 7f180d5aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:07.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.314+0000 7f180d5aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:07.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.314+0000 7f180d5aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:07.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.314+0000 7f180d5aa640 10 monclient(hunting): handle_auth_done global_id 64148 payload 824 2024-09-11T01:35:07.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.314+0000 7f180d5aa640 1 -- 172.21.15.12:0/1052400043 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f18081c5f70 con 0x7f18080a2570 2024-09-11T01:35:07.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.316+0000 7f180d5aa640 1 -- 172.21.15.12:0/1052400043 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18080a2570 msgr2=0x7f180809f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:07.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.316+0000 7f180d5aa640 1 --2- 172.21.15.12:0/1052400043 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18080a2570 0x7f180809f500 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f18080a2fc0 tx=0x7f17f8003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:07.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.316+0000 7f17fe7fc640 1 -- 172.21.15.12:0/1052400043 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f801dcc0 con 0x7f18080a2570 2024-09-11T01:35:07.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.316+0000 7f17fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:07.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.316+0000 7f17fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:35:07.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.316+0000 7f17fe7fc640 10 monclient: dump: 2024-09-11T01:35:07.327 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:07.327 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:07.327 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.327 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.327 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:07.327 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:07.327 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:07.327 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:07.328 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:07.328 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:07.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.316+0000 7f17fe7fc640 1 -- 172.21.15.12:0/1052400043 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4540 (secure 0 0 0) 0x7f17f801dcc0 con 0x7f18080a2570 2024-09-11T01:35:07.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.316+0000 7f17fe7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:35:07.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.316+0000 7f17fe7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:35:07.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.316+0000 7f17fe7fc640 1 -- 172.21.15.12:0/1052400043 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18080a2570 msgr2=0x7f180809f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:07.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.316+0000 7f17fe7fc640 1 --2- 172.21.15.12:0/1052400043 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18080a2570 0x7f180809f500 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f18080a2fc0 tx=0x7f17f8003760 comp rx=0 tx=0).stop 2024-09-11T01:35:07.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.316+0000 7f17fe7fc640 1 -- 172.21.15.12:0/1052400043 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18080a2570 msgr2=0x7f180809f500 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:35:07.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.316+0000 7f17fe7fc640 1 --2- 172.21.15.12:0/1052400043 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18080a2570 0x7f180809f500 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f18080a2fc0 tx=0x7f17f8003760 comp rx=0 tx=0).stop 2024-09-11T01:35:07.328 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:35:07.329 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:35:07.329 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:35:07.329 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:35:07.329 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 64, 2024-09-11T01:35:07.329 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 25408, 2024-09-11T01:35:07.329 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:35:07.329 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:35:07.329 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:35:07.329 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:35:07.329 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:07.329 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 5, 2024-09-11T01:35:07.329 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:35:07.329 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:35:07.330 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:35:07.330 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:35:07.330 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:35:07.330 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:35:07.330 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:35:07.330 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:35:07.330 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:35:07.330 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:35:07.330 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:35:07.330 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:35:07.330 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:35:07.330 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:35:07.330 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:07.331 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:35:07.331 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:35:07.331 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:35:07.331 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:35:07.331 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:35:07.331 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:35:07.331 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:35:07.331 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:35:07.331 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:35:07.331 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:35:07.331 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:35:07.331 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:35:07.331 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:35:07.331 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:07.332 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.332 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:35:07.332 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:35:07.332 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:35:07.332 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:35:07.332 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:35:07.332 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:35:07.332 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:35:07.332 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:35:07.332 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:35:07.332 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:35:07.332 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:35:07.332 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:35:07.333 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:35:07.333 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:35:07.333 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:35:07.333 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:35:07.333 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:35:07.333 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:35:07.333 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:35:07.333 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:35:07.333 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:35:07.333 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:35:07.333 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:35:07.333 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:35:07.333 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:35:07.334 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:35:07.334 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:35:07.334 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:35:07.334 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:07.334 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:35:07.334 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.334 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:35:07.334 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.334 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:35:07.334 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:35:07.334 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:35:07.334 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:35:07.334 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.335 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:35:07.335 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:35:07.335 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:07.335 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.335 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.335 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:35:07.335 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:35:07.335 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:07.335 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:07.335 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:07.335 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.335 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:35:07.335 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:35:07.335 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:35:07.336 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:35:07.336 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:35:07.336 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.336 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.336 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:35:07.336 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:35:07.336 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:35:07.336 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:35:07.336 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.336 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:35:07.336 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:35:07.336 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:07.336 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.337 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.337 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:35:07.337 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:35:07.337 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:07.337 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:07.337 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:07.337 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.337 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:35:07.337 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:35:07.337 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:35:07.337 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:35:07.337 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:35:07.337 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.338 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.338 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:35:07.338 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:35:07.338 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:35:07.338 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:35:07.338 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.338 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:35:07.338 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:35:07.338 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:07.338 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.338 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.338 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:35:07.338 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:35:07.338 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:07.340 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:07.340 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:07.340 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.340 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:35:07.340 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:35:07.340 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:35:07.340 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:35:07.340 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:35:07.341 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:07.341 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:07.341 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.341 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:35:07.341 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:35:07.341 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.341 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:35:07.341 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:35:07.341 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:35:07.341 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:07.341 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:07.341 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:35:07.341 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.341 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:35:07.342 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:35:07.342 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:35:07.342 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:07.342 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:07.342 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.342 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:35:07.342 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:35:07.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.317+0000 7f181000f640 1 -- 172.21.15.12:0/1052400043 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f17ec03e240 msgr2=0x7f17ec040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.317+0000 7f181000f640 1 --2- 172.21.15.12:0/1052400043 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f17ec03e240 0x7f17ec040700 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1800005710 tx=0x7f1800009040 comp rx=0 tx=0).stop 2024-09-11T01:35:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.317+0000 7f181000f640 10 monclient: shutdown 2024-09-11T01:35:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.317+0000 7f181000f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:35:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.317+0000 7f181000f640 1 -- 172.21.15.12:0/1052400043 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1808137e50 msgr2=0x7f18080a3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.317+0000 7f180e5ac640 1 -- 172.21.15.12:0/1052400043 reap_dead start 2024-09-11T01:35:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.317+0000 7f181000f640 1 --2- 172.21.15.12:0/1052400043 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1808137e50 0x7f18080a3030 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f18040090b0 tx=0x7f18040046c0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.317+0000 7f181000f640 1 -- 172.21.15.12:0/1052400043 shutdown_connections 2024-09-11T01:35:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.317+0000 7f181000f640 1 --2- 172.21.15.12:0/1052400043 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1808137e50 0x7f18080a3030 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.317+0000 7f181000f640 1 -- 172.21.15.12:0/1052400043 >> 172.21.15.12:0/1052400043 conn(0x7f18080a7020 msgr2=0x7f180813c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:35:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.317+0000 7f181000f640 1 -- 172.21.15.12:0/1052400043 shutdown_connections 2024-09-11T01:35:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.317+0000 7f181000f640 1 -- 172.21.15.12:0/1052400043 wait complete. 2024-09-11T01:35:07.345 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:35:07.345 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:35:07.345 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:35:07.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.446+0000 7f3e487ec640 10 monclient: get_monmap_and_config 2024-09-11T01:35:07.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.446+0000 7f3e487ec640 10 monclient: build_initial_monmap 2024-09-11T01:35:07.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.446+0000 7f3e487ec640 10 monclient: monmap: 2024-09-11T01:35:07.448 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:35:07.448 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:07.448 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:35:07.447083+0000 2024-09-11T01:35:07.448 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:35:07.447083+0000 2024-09-11T01:35:07.448 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:35:07.448 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:07.448 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:35:07.448 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:35:07.448 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:35:07.449 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:35:07.449 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:35:07.449 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:07.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.446+0000 7f3e487ec640 1 Processor -- start 2024-09-11T01:35:07.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.446+0000 7f3e487ec640 1 -- start start 2024-09-11T01:35:07.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.446+0000 7f3e487ec640 10 monclient: init 2024-09-11T01:35:07.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e487ec640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:07.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e487ec640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-11T01:35:07.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e487ec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3e400a2e20 addr v1:172.21.15.71:6789/0 2024-09-11T01:35:07.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e487ec640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3e400a4970 0x7f3e4006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:07.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e487ec640 10 monclient(hunting): picked mon.noname-b con 0x7f3e400a4970 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:35:07.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e487ec640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3e4006f910 0x7f3e4013f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:07.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e487ec640 10 monclient(hunting): picked mon.noname-c con 0x7f3e4006f910 addr v2:172.21.15.71:3300/0 2024-09-11T01:35:07.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e487ec640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e4006f4f0 con 0x7f3e400a2e20 2024-09-11T01:35:07.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e487ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:07.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e487ec640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f3e4006ef30 con 0x7f3e4006f910 2024-09-11T01:35:07.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e487ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:07.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e487ec640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f3e4006ec50 con 0x7f3e400a4970 2024-09-11T01:35:07.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e487ec640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:07.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e487ec640 10 monclient(hunting): authenticate will time out at 1113.889660s 2024-09-11T01:35:07.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e45d87640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3e400a4970 0x7f3e4006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:07.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e45d87640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3e400a4970 0x7f3e4006ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:58072/0 (socket says 172.21.15.12:58072) 2024-09-11T01:35:07.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e45d87640 1 -- 172.21.15.12:0/2461003863 learned_addr learned my addr 172.21.15.12:0/2461003863 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:35:07.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e45d87640 10 monclient(hunting): get_auth_request con 0x7f3e400a4970 auth_method 0 2024-09-11T01:35:07.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e45d87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:07.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e45d87640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:07.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e45d87640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:07.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e46d89640 1 --2- 172.21.15.12:0/2461003863 >> v2:172.21.15.71:3300/0 conn(0x7f3e4006f910 0x7f3e4013f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:07.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e46d89640 10 monclient(hunting): get_auth_request con 0x7f3e4006f910 auth_method 0 2024-09-11T01:35:07.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e46d89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:07.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e46d89640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:07.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e46d89640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:07.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e45d87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:07.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e45d87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:07.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e45d87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:07.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e46d89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:07.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.447+0000 7f3e46d89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:07.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.448+0000 7f3e46d89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:07.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.448+0000 7f3e45d87640 10 monclient(hunting): handle_auth_done global_id 64151 payload 274 2024-09-11T01:35:07.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.448+0000 7f3e45d87640 1 -- 172.21.15.12:0/2461003863 >> v2:172.21.15.71:3300/0 conn(0x7f3e4006f910 msgr2=0x7f3e4013f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:35:07.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.448+0000 7f3e45d87640 1 --2- 172.21.15.12:0/2461003863 >> v2:172.21.15.71:3300/0 conn(0x7f3e4006f910 0x7f3e4013f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.448+0000 7f3e45d87640 1 -- 172.21.15.12:0/2461003863 >> v1:172.21.15.71:6789/0 conn(0x7f3e400a2e20 legacy=0x7f3e400a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:35:07.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.448+0000 7f3e45d87640 10 monclient: _finish_hunting 0 2024-09-11T01:35:07.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.448+0000 7f3e45d87640 1 monclient: found mon.noname-b 2024-09-11T01:35:07.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.448+0000 7f3e45d87640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:07.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.448+0000 7f3e45d87640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:35:07.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.448+0000 7f3e45d87640 1 -- 172.21.15.12:0/2461003863 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e400a3910 con 0x7f3e400a4970 2024-09-11T01:35:07.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.450+0000 7f3e45d87640 1 -- 172.21.15.12:0/2461003863 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3e400a4970 msgr2=0x7f3e4006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:07.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.450+0000 7f3e45d87640 1 --2- 172.21.15.12:0/2461003863 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3e400a4970 0x7f3e4006ded0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3e30009870 tx=0x7f3e3002a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e607854357618c74 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:07.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e45586640 1 -- 172.21.15.12:0/2461003863 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e30037070 con 0x7f3e400a4970 2024-09-11T01:35:07.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e45586640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:07.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e45586640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:35:07.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e45586640 10 monclient: dump: 2024-09-11T01:35:07.461 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:07.461 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:07.461 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.461 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.461 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:07.461 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:07.461 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:07.461 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:07.461 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:07.462 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:07.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e45586640 10 monclient: _finish_auth 0 2024-09-11T01:35:07.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e45586640 10 monclient: _check_auth_tickets 2024-09-11T01:35:07.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e45586640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:07.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e487ec640 5 monclient: authenticate success, global_id 64151 2024-09-11T01:35:07.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e487ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:35:07.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e45586640 1 -- 172.21.15.12:0/2461003863 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e30032440 con 0x7f3e400a4970 2024-09-11T01:35:07.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e45586640 10 monclient: handle_config config(0 keys) 2024-09-11T01:35:07.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e45586640 1 -- 172.21.15.12:0/2461003863 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e30032810 con 0x7f3e400a4970 2024-09-11T01:35:07.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e45586640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:07.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e45586640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:35:07.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e45586640 10 monclient: dump: 2024-09-11T01:35:07.462 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:07.462 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:07.463 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.463 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.463 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:07.463 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:07.463 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:07.463 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:07.463 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:07.463 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:07.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e487ec640 10 monclient: get_monmap_and_config success 2024-09-11T01:35:07.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e487ec640 4 set_mon_vals no callback set 2024-09-11T01:35:07.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e487ec640 10 monclient: shutdown 2024-09-11T01:35:07.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e487ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:35:07.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e487ec640 1 -- 172.21.15.12:0/2461003863 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3e400a4970 msgr2=0x7f3e4006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:07.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e487ec640 1 --2- 172.21.15.12:0/2461003863 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3e400a4970 0x7f3e4006ded0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3e30009870 tx=0x7f3e3002a0a0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e487ec640 1 -- 172.21.15.12:0/2461003863 shutdown_connections 2024-09-11T01:35:07.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e487ec640 1 --2- 172.21.15.12:0/2461003863 >> v2:172.21.15.71:3300/0 conn(0x7f3e4006f910 0x7f3e4013f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e487ec640 1 --2- 172.21.15.12:0/2461003863 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3e400a4970 0x7f3e4006ded0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e487ec640 1 -- 172.21.15.12:0/2461003863 >> 172.21.15.12:0/2461003863 conn(0x7f3e400a8600 msgr2=0x7f3e400a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:35:07.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e487ec640 1 -- 172.21.15.12:0/2461003863 shutdown_connections 2024-09-11T01:35:07.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.451+0000 7f3e487ec640 1 -- 172.21.15.12:0/2461003863 wait complete. 2024-09-11T01:35:07.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.452+0000 7f3e487ec640 10 monclient: build_initial_monmap 2024-09-11T01:35:07.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.452+0000 7f3e487ec640 10 monclient: monmap: 2024-09-11T01:35:07.464 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:35:07.464 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:35:07.464 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:35:07.464 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:35:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:35:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:35:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:35:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:35:07.465 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.452+0000 7f3e487ec640 1 Processor -- start 2024-09-11T01:35:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.452+0000 7f3e487ec640 1 -- start start 2024-09-11T01:35:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.452+0000 7f3e487ec640 10 monclient: init 2024-09-11T01:35:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e487ec640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e487ec640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:35:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e487ec640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3e4006f910 0x7f3e401398c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e487ec640 10 monclient(hunting): picked mon.noname-b con 0x7f3e4006f910 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:35:07.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e487ec640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3e400a2e20 0x7f3e40139e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:07.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e487ec640 10 monclient(hunting): picked mon.noname-c con 0x7f3e400a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:35:07.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e487ec640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e400a4970 0x7f3e401ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:07.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e487ec640 10 monclient(hunting): picked mon.noname-a con 0x7f3e400a4970 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:35:07.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e487ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:07.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e487ec640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4013fd00 con 0x7f3e400a4970 2024-09-11T01:35:07.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e487ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:07.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e487ec640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4013fb80 con 0x7f3e4006f910 2024-09-11T01:35:07.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e487ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:07.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e487ec640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f3e4013fe80 con 0x7f3e400a2e20 2024-09-11T01:35:07.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e487ec640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:07.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e487ec640 10 monclient(hunting): authenticate will time out at 1113.895320s 2024-09-11T01:35:07.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e45d87640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3e400a2e20 0x7f3e40139e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:07.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e46d89640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e400a4970 0x7f3e401ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e45d87640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3e400a2e20 0x7f3e40139e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:58084/0 (socket says 172.21.15.12:58084) 2024-09-11T01:35:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e46d89640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e400a4970 0x7f3e401ccfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45424/0 (socket says 172.21.15.12:45424) 2024-09-11T01:35:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e45d87640 1 -- 172.21.15.12:0/1103114834 learned_addr learned my addr 172.21.15.12:0/1103114834 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:35:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e45d87640 10 monclient(hunting): get_auth_request con 0x7f3e400a2e20 auth_method 0 2024-09-11T01:35:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e45d87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e45d87640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e45d87640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e46d89640 10 monclient(hunting): get_auth_request con 0x7f3e400a4970 auth_method 0 2024-09-11T01:35:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e46d89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e46d89640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e46d89640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e45d87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:07.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e45d87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e45d87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e46d89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e46d89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e46588640 1 --2- 172.21.15.12:0/1103114834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3e4006f910 0x7f3e401398c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e46d89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e46588640 10 monclient(hunting): get_auth_request con 0x7f3e4006f910 auth_method 0 2024-09-11T01:35:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e46588640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e46588640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e46588640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e46588640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e46588640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e46588640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:07.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e45d87640 10 monclient(hunting): handle_auth_done global_id 64154 payload 824 2024-09-11T01:35:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e45d87640 1 -- 172.21.15.12:0/1103114834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3e4006f910 msgr2=0x7f3e401398c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e45d87640 1 --2- 172.21.15.12:0/1103114834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3e4006f910 0x7f3e401398c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e45d87640 1 -- 172.21.15.12:0/1103114834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e400a4970 msgr2=0x7f3e401ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e45d87640 1 --2- 172.21.15.12:0/1103114834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e400a4970 0x7f3e401ccfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e45d87640 10 monclient: _finish_hunting 0 2024-09-11T01:35:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e45d87640 1 monclient: found mon.noname-c 2024-09-11T01:35:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e45d87640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e45d87640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:35:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e45d87640 1 -- 172.21.15.12:0/1103114834 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e400a2310 con 0x7f3e400a2e20 2024-09-11T01:35:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.453+0000 7f3e46d89640 1 --2- 172.21.15.12:0/1103114834 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e400a4970 0x7f3e401ccfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:35:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.456+0000 7f3e45d87640 1 -- 172.21.15.12:0/1103114834 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3e400a2e20 msgr2=0x7f3e40139e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.456+0000 7f3e45d87640 1 --2- 172.21.15.12:0/1103114834 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3e400a2e20 0x7f3e40139e00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3e300090b0 tx=0x7f3e300046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:07.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.456+0000 7f3e36ffd640 1 -- 172.21.15.12:0/1103114834 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e30037040 con 0x7f3e400a2e20 2024-09-11T01:35:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.456+0000 7f3e36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.456+0000 7f3e36ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:35:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.456+0000 7f3e36ffd640 10 monclient: dump: 2024-09-11T01:35:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:07.470 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:07.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.456+0000 7f3e36ffd640 10 monclient: _finish_auth 0 2024-09-11T01:35:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.456+0000 7f3e36ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:35:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.456+0000 7f3e36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.456+0000 7f3e36ffd640 1 -- 172.21.15.12:0/1103114834 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e30042070 con 0x7f3e400a2e20 2024-09-11T01:35:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.456+0000 7f3e36ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:35:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.456+0000 7f3e36ffd640 1 -- 172.21.15.12:0/1103114834 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3003b860 con 0x7f3e400a2e20 2024-09-11T01:35:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.456+0000 7f3e36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.456+0000 7f3e36ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:35:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.456+0000 7f3e36ffd640 10 monclient: dump: 2024-09-11T01:35:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:07.471 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:07.472 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:07.472 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:07.472 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:07.472 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:07.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.456+0000 7f3e487ec640 5 monclient: authenticate success, global_id 64154 2024-09-11T01:35:07.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.456+0000 7f3e487ec640 10 monclient: _renew_subs 2024-09-11T01:35:07.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.456+0000 7f3e487ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:35:07.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.456+0000 7f3e487ec640 1 -- 172.21.15.12:0/1103114834 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e400a5d90 con 0x7f3e400a2e20 2024-09-11T01:35:07.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.456+0000 7f3e44d85640 4 set_mon_vals no callback set 2024-09-11T01:35:07.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.456+0000 7f3e487ec640 10 monclient: _renew_subs 2024-09-11T01:35:07.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.456+0000 7f3e487ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:35:07.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.456+0000 7f3e487ec640 1 -- 172.21.15.12:0/1103114834 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e4013b790 con 0x7f3e400a2e20 2024-09-11T01:35:07.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.457+0000 7f3e36ffd640 1 -- 172.21.15.12:0/1103114834 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f3e3003a030 con 0x7f3e400a2e20 2024-09-11T01:35:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.457+0000 7f3e487ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.457+0000 7f3e487ec640 1 --2- 172.21.15.12:0/1103114834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3e4002d440 0x7f3e4002d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.457+0000 7f3e487ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.457+0000 7f3e487ec640 1 -- 172.21.15.12:0/1103114834 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4006a060 con 0x7f3e4002d440 2024-09-11T01:35:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.457+0000 7f3e36ffd640 1 --2- 172.21.15.12:0/1103114834 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3e2403e210 0x7f3e240406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.457+0000 7f3e36ffd640 1 -- 172.21.15.12:0/1103114834 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(201..201 src has 1..201) ==== 4783+0+0 (secure 0 0 0) 0x7f3e30072f30 con 0x7f3e400a2e20 2024-09-11T01:35:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.457+0000 7f3e46588640 1 --2- 172.21.15.12:0/1103114834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3e4002d440 0x7f3e4002d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.457+0000 7f3e46588640 10 monclient: get_auth_request con 0x7f3e4002d440 auth_method 0 2024-09-11T01:35:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.457+0000 7f3e46588640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.457+0000 7f3e46588640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.457+0000 7f3e46588640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.457+0000 7f3e46d89640 1 --2- 172.21.15.12:0/1103114834 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3e2403e210 0x7f3e240406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.457+0000 7f3e46d89640 10 monclient: get_auth_request con 0x7f3e2403e210 auth_method 0 2024-09-11T01:35:07.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.457+0000 7f3e46588640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:07.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.457+0000 7f3e46588640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:07.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.457+0000 7f3e46588640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:07.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.457+0000 7f3e46588640 10 monclient(hunting): handle_auth_done global_id 34396 payload 824 2024-09-11T01:35:07.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.457+0000 7f3e46588640 1 -- 172.21.15.12:0/1103114834 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3e4006ded0 con 0x7f3e4002d440 2024-09-11T01:35:07.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.460+0000 7f3e46d89640 1 --2- 172.21.15.12:0/1103114834 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3e2403e210 0x7f3e240406d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3e4009cad0 tx=0x7f3e3800b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:07.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.460+0000 7f3e46588640 1 -- 172.21.15.12:0/1103114834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3e4002d440 msgr2=0x7f3e4002d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:07.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.460+0000 7f3e46588640 1 --2- 172.21.15.12:0/1103114834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3e4002d440 0x7f3e4002d820 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00f7a0 tx=0x7f3e3c00fc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.460+0000 7f3e36ffd640 1 -- 172.21.15.12:0/1103114834 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c018030 con 0x7f3e4002d440 2024-09-11T01:35:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.460+0000 7f3e36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.460+0000 7f3e36ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:35:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.460+0000 7f3e36ffd640 10 monclient: dump: 2024-09-11T01:35:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:07.477 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:07.478 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.460+0000 7f3e36ffd640 1 -- 172.21.15.12:0/1103114834 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3e3c00d9b0 con 0x7f3e4002d440 2024-09-11T01:35:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.460+0000 7f3e36ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.460+0000 7f3e36ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:35:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.460+0000 7f3e36ffd640 1 -- 172.21.15.12:0/1103114834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3e4002d440 msgr2=0x7f3e4002d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.460+0000 7f3e36ffd640 1 --2- 172.21.15.12:0/1103114834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3e4002d440 0x7f3e4002d820 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00f7a0 tx=0x7f3e3c00fc60 comp rx=0 tx=0).stop 2024-09-11T01:35:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.460+0000 7f3e36ffd640 1 -- 172.21.15.12:0/1103114834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3e4002d440 msgr2=0x7f3e4002d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:35:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.460+0000 7f3e36ffd640 1 --2- 172.21.15.12:0/1103114834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3e4002d440 0x7f3e4002d820 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.467+0000 7f3e487ec640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:35:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.467+0000 7f3e487ec640 1 --2- 172.21.15.12:0/1103114834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3e4002dd60 0x7f3e4002e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.467+0000 7f3e487ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.467+0000 7f3e487ec640 1 -- 172.21.15.12:0/1103114834 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3e40137960 con 0x7f3e4002dd60 2024-09-11T01:35:07.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.467+0000 7f3e46588640 1 --2- 172.21.15.12:0/1103114834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3e4002dd60 0x7f3e4002e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:07.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.467+0000 7f3e46588640 10 monclient: get_auth_request con 0x7f3e4002dd60 auth_method 0 2024-09-11T01:35:07.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.467+0000 7f3e46588640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:07.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.467+0000 7f3e46588640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:07.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.467+0000 7f3e46588640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:07.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.467+0000 7f3e46588640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:07.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.467+0000 7f3e46588640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:07.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.467+0000 7f3e46588640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:07.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.467+0000 7f3e46588640 10 monclient(hunting): handle_auth_done global_id 34399 payload 824 2024-09-11T01:35:07.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.467+0000 7f3e46588640 1 -- 172.21.15.12:0/1103114834 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3e40069950 con 0x7f3e4002dd60 2024-09-11T01:35:07.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.470+0000 7f3e46588640 1 -- 172.21.15.12:0/1103114834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3e4002dd60 msgr2=0x7f3e4002e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:07.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.470+0000 7f3e46588640 1 --2- 172.21.15.12:0/1103114834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3e4002dd60 0x7f3e4002e170 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00f7a0 tx=0x7f3e3c003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:07.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.470+0000 7f3e36ffd640 1 -- 172.21.15.12:0/1103114834 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c01dd70 con 0x7f3e4002dd60 2024-09-11T01:35:07.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.470+0000 7f3e36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:07.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.470+0000 7f3e36ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:35:07.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.470+0000 7f3e36ffd640 10 monclient: dump: 2024-09-11T01:35:07.480 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:07.480 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:07.480 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.480 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:07.480 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:07.480 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:07.480 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:07.480 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:07.480 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:07.480 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:07.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.470+0000 7f3e36ffd640 1 -- 172.21.15.12:0/1103114834 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7f3e3c01dd70 con 0x7f3e4002dd60 2024-09-11T01:35:07.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.470+0000 7f3e36ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:35:07.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.470+0000 7f3e36ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:35:07.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.470+0000 7f3e36ffd640 1 -- 172.21.15.12:0/1103114834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3e4002dd60 msgr2=0x7f3e4002e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:07.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.470+0000 7f3e36ffd640 1 --2- 172.21.15.12:0/1103114834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3e4002dd60 0x7f3e4002e170 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00f7a0 tx=0x7f3e3c003760 comp rx=0 tx=0).stop 2024-09-11T01:35:07.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.470+0000 7f3e36ffd640 1 -- 172.21.15.12:0/1103114834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3e4002dd60 msgr2=0x7f3e4002e170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:35:07.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.470+0000 7f3e36ffd640 1 --2- 172.21.15.12:0/1103114834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3e4002dd60 0x7f3e4002e170 secure :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00f7a0 tx=0x7f3e3c003760 comp rx=0 tx=0).stop 2024-09-11T01:35:07.481 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:35:07.481 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:35:07.481 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:35:07.481 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:35:07.482 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 64, 2024-09-11T01:35:07.482 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 200684, 2024-09-11T01:35:07.482 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:35:07.482 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:35:07.482 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:35:07.482 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:35:07.482 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:07.482 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 5, 2024-09-11T01:35:07.482 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:35:07.482 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:35:07.482 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:35:07.482 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:35:07.482 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:35:07.483 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:35:07.483 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:35:07.483 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:35:07.483 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:35:07.483 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:35:07.483 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:35:07.483 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:35:07.483 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:35:07.483 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:35:07.483 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:07.483 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:35:07.483 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:35:07.483 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:35:07.483 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:35:07.484 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:35:07.484 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:35:07.484 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:35:07.484 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:35:07.484 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:35:07.484 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:35:07.484 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:35:07.484 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:35:07.484 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:35:07.484 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:07.484 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.484 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:35:07.484 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:35:07.485 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:35:07.485 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:35:07.485 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:35:07.485 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:35:07.485 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:35:07.485 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:35:07.485 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:35:07.485 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:35:07.485 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:35:07.485 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:35:07.485 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:35:07.485 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:35:07.485 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:35:07.486 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:35:07.486 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:35:07.486 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:35:07.486 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:35:07.486 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:35:07.486 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:35:07.486 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:35:07.486 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:35:07.486 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:35:07.486 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:35:07.486 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:35:07.486 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:35:07.486 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:35:07.486 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:07.487 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:35:07.487 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.487 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:35:07.487 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.487 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:35:07.487 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:35:07.487 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:35:07.487 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:35:07.487 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.487 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:35:07.487 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:35:07.487 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:07.487 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.488 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.488 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:35:07.488 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:35:07.488 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:07.488 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:07.488 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:07.488 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.488 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:35:07.488 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:35:07.488 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:35:07.488 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:35:07.488 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:35:07.488 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.488 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.489 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:35:07.489 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:35:07.489 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:35:07.489 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:35:07.489 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.489 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:35:07.489 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:35:07.489 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:07.489 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.489 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.489 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:35:07.489 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:35:07.489 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:07.490 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:07.490 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:07.490 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.490 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:35:07.490 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:35:07.490 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:35:07.490 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:35:07.490 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:35:07.490 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.490 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.490 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:35:07.490 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:35:07.490 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:35:07.490 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:35:07.491 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.491 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:35:07.491 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:35:07.491 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:07.491 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.491 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.491 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:35:07.491 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:35:07.491 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:07.491 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:07.491 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:07.491 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.491 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:35:07.491 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:35:07.492 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:35:07.492 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:35:07.492 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:35:07.492 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:07.492 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:07.492 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.492 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:35:07.492 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:35:07.492 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.492 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:35:07.492 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:35:07.492 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:35:07.492 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:07.493 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:07.493 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:35:07.493 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.493 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:35:07.493 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:35:07.493 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:35:07.493 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:07.493 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:07.493 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:35:07.493 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.493 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:35:07.493 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:35:07.493 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:35:07.493 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:07.494 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:07.494 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:35:07.494 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:07.494 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:35:07.494 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:35:07.494 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:35:07.494 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:07.494 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:07.494 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:07.494 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:35:07.494 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:35:07.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.470+0000 7f3e487ec640 1 -- 172.21.15.12:0/1103114834 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3e2403e210 msgr2=0x7f3e240406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:07.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.470+0000 7f3e487ec640 1 --2- 172.21.15.12:0/1103114834 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3e2403e210 0x7f3e240406d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3e4009cad0 tx=0x7f3e3800b040 comp rx=0 tx=0).stop 2024-09-11T01:35:07.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.470+0000 7f3e487ec640 10 monclient: shutdown 2024-09-11T01:35:07.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.470+0000 7f3e487ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:35:07.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.470+0000 7f3e487ec640 1 -- 172.21.15.12:0/1103114834 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3e400a2e20 msgr2=0x7f3e40139e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:07.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.470+0000 7f3e487ec640 1 --2- 172.21.15.12:0/1103114834 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3e400a2e20 0x7f3e40139e00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3e300090b0 tx=0x7f3e300046c0 comp rx=0 tx=0).stop 2024-09-11T01:35:07.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.470+0000 7f3e46d89640 1 -- 172.21.15.12:0/1103114834 reap_dead start 2024-09-11T01:35:07.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.470+0000 7f3e46d89640 1 -- 172.21.15.12:0/1103114834 reap_dead start 2024-09-11T01:35:07.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.470+0000 7f3e487ec640 1 -- 172.21.15.12:0/1103114834 shutdown_connections 2024-09-11T01:35:07.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.470+0000 7f3e487ec640 1 -- 172.21.15.12:0/1103114834 >> 172.21.15.12:0/1103114834 conn(0x7f3e400a8600 msgr2=0x7f3e4009eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:35:07.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.471+0000 7f3e487ec640 1 -- 172.21.15.12:0/1103114834 shutdown_connections 2024-09-11T01:35:07.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:07.471+0000 7f3e487ec640 1 -- 172.21.15.12:0/1103114834 wait complete. 2024-09-11T01:35:07.497 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:35:07.497 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-11T01:35:07.497 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-11T01:35:07.497 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-11T01:35:07.497 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-11T01:35:07.497 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-11T01:35:07.498 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:35:07.509 INFO:tasks.ceph.mon.b:Stopped 2024-09-11T01:35:07.509 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-11T01:35:07.509 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-11T01:35:07.509 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T01:35:22.511 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-11T01:35:22.511 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-11T01:35:22.512 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:35:22.512 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:35:22.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.623+0000 7f801cf4b640 10 monclient: get_monmap_and_config 2024-09-11T01:35:22.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.623+0000 7f801cf4b640 10 monclient: build_initial_monmap 2024-09-11T01:35:22.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.623+0000 7f801cf4b640 10 monclient: monmap: 2024-09-11T01:35:22.626 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:35:22.626 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:22.626 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:35:22.625304+0000 2024-09-11T01:35:22.626 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:35:22.625304+0000 2024-09-11T01:35:22.626 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:35:22.626 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:22.627 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:35:22.627 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:35:22.627 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:35:22.627 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:35:22.627 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:35:22.627 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:22.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.624+0000 7f801cf4b640 1 Processor -- start 2024-09-11T01:35:22.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.624+0000 7f801cf4b640 1 -- start start 2024-09-11T01:35:22.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.624+0000 7f801cf4b640 10 monclient: init 2024-09-11T01:35:22.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.624+0000 7f801cf4b640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:22.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.624+0000 7f801cf4b640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-11T01:35:22.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.624+0000 7f801cf4b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8018138230 0x7f8018138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.624+0000 7f801cf4b640 10 monclient(hunting): picked mon.noname-b con 0x7f8018138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:35:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.624+0000 7f801cf4b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f801809d030 addr v1:172.21.15.71:6789/0 2024-09-11T01:35:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.624+0000 7f801cf4b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f801809d430 0x7f80180a5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.624+0000 7f801cf4b640 10 monclient(hunting): picked mon.noname-a con 0x7f801809d430 addr v2:172.21.15.12:3300/0 2024-09-11T01:35:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.624+0000 7f801cf4b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80180a6e30 con 0x7f801809d030 2024-09-11T01:35:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.624+0000 7f801cf4b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.624+0000 7f801cf4b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f801809c9c0 con 0x7f801809d430 2024-09-11T01:35:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.624+0000 7f801cf4b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.624+0000 7f801cf4b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f801809ba20 con 0x7f8018138230 2024-09-11T01:35:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.624+0000 7f801cf4b640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.624+0000 7f801cf4b640 10 monclient(hunting): authenticate will time out at 1129.067948s 2024-09-11T01:35:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.625+0000 7f801659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8018138230 0x7f8018138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:22.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.625+0000 7f800ffff640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f801809d030 legacy=0x7f80180a5020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:35:22.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.625+0000 7f8016d9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f801809d430 0x7f80180a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:22.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.625+0000 7f801659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8018138230 0x7f8018138610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41108/0 (socket says 172.21.15.12:41108) 2024-09-11T01:35:22.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.625+0000 7f801659c640 1 -- 172.21.15.12:0/4092600457 learned_addr learned my addr 172.21.15.12:0/4092600457 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:35:22.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.625+0000 7f801659c640 10 monclient(hunting): get_auth_request con 0x7f8018138230 auth_method 0 2024-09-11T01:35:22.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.625+0000 7f801659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:22.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.625+0000 7f801659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:22.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.625+0000 7f801659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:22.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.625+0000 7f8016d9d640 10 monclient(hunting): get_auth_request con 0x7f801809d430 auth_method 0 2024-09-11T01:35:22.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.625+0000 7f8016d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:22.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.625+0000 7f8016d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:22.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.625+0000 7f8016d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:22.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.625+0000 7f8016d9d640 1 -- 172.21.15.12:0/4092600457 >> v2:172.21.15.12:3300/0 conn(0x7f801809d430 msgr2=0x7f80180a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:35:22.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.625+0000 7f8016d9d640 1 -- 172.21.15.12:0/4092600457 >> v2:172.21.15.12:3300/0 conn(0x7f801809d430 msgr2=0x7f80180a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:35:22.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.625+0000 7f801659c640 1 -- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8018138230 msgr2=0x7f8018138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:35:22.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.625+0000 7f8016d9d640 1 --2- 172.21.15.12:0/4092600457 >> v2:172.21.15.12:3300/0 conn(0x7f801809d430 0x7f80180a5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:35:22.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.625+0000 7f801659c640 1 -- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8018138230 msgr2=0x7f8018138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:35:22.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.625+0000 7f801659c640 1 --2- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8018138230 0x7f8018138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:35:22.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.625+0000 7f8016d9d640 1 --2- 172.21.15.12:0/4092600457 >> v2:172.21.15.12:3300/0 conn(0x7f801809d430 0x7f80180a5730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:35:22.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.625+0000 7f801659c640 1 --2- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8018138230 0x7f8018138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:35:22.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.826+0000 7f800ffff640 1 -- 172.21.15.12:0/4092600457 >> v1:172.21.15.71:6789/0 conn(0x7f801809d030 legacy=0x7f80180a5020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:35:22.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.826+0000 7f801659c640 1 --2- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8018138230 0x7f8018138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:22.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.826+0000 7f8016d9d640 1 --2- 172.21.15.12:0/4092600457 >> v2:172.21.15.12:3300/0 conn(0x7f801809d430 0x7f80180a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:22.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.826+0000 7f8016d9d640 10 monclient(hunting): get_auth_request con 0x7f801809d430 auth_method 0 2024-09-11T01:35:22.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.826+0000 7f8016d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:22.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.826+0000 7f8016d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:22.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.826+0000 7f8016d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:35:22.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.826+0000 7f801659c640 10 monclient(hunting): get_auth_request con 0x7f8018138230 auth_method 0 2024-09-11T01:35:22.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.826+0000 7f801659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:22.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.826+0000 7f801659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:22.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.826+0000 7f801659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:35:22.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.826+0000 7f8016d9d640 1 -- 172.21.15.12:0/4092600457 >> v2:172.21.15.12:3300/0 conn(0x7f801809d430 msgr2=0x7f80180a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:35:22.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.826+0000 7f8016d9d640 1 -- 172.21.15.12:0/4092600457 >> v2:172.21.15.12:3300/0 conn(0x7f801809d430 msgr2=0x7f80180a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:35:22.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.826+0000 7f801659c640 1 -- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8018138230 msgr2=0x7f8018138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:35:22.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.826+0000 7f8016d9d640 1 --2- 172.21.15.12:0/4092600457 >> v2:172.21.15.12:3300/0 conn(0x7f801809d430 0x7f80180a5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:35:22.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.826+0000 7f801659c640 1 -- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8018138230 msgr2=0x7f8018138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:35:22.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.826+0000 7f801659c640 1 --2- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8018138230 0x7f8018138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:35:22.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.826+0000 7f8016d9d640 1 --2- 172.21.15.12:0/4092600457 >> v2:172.21.15.12:3300/0 conn(0x7f801809d430 0x7f80180a5730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:35:22.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:22.826+0000 7f801659c640 1 --2- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8018138230 0x7f8018138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:35:23.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:23.227+0000 7f8016d9d640 1 --2- 172.21.15.12:0/4092600457 >> v2:172.21.15.12:3300/0 conn(0x7f801809d430 0x7f80180a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:23.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:23.227+0000 7f800ffff640 1 -- 172.21.15.12:0/4092600457 >> v1:172.21.15.71:6789/0 conn(0x7f801809d030 legacy=0x7f80180a5020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:35:23.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:23.227+0000 7f8016d9d640 10 monclient(hunting): get_auth_request con 0x7f801809d430 auth_method 0 2024-09-11T01:35:23.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:23.227+0000 7f8016d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:23.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:23.227+0000 7f8016d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:23.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:23.227+0000 7f8016d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:35:23.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:23.227+0000 7f8016d9d640 1 -- 172.21.15.12:0/4092600457 >> v2:172.21.15.12:3300/0 conn(0x7f801809d430 msgr2=0x7f80180a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:35:23.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:23.227+0000 7f8016d9d640 1 -- 172.21.15.12:0/4092600457 >> v2:172.21.15.12:3300/0 conn(0x7f801809d430 msgr2=0x7f80180a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:35:23.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:23.227+0000 7f8016d9d640 1 --2- 172.21.15.12:0/4092600457 >> v2:172.21.15.12:3300/0 conn(0x7f801809d430 0x7f80180a5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:35:23.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:23.227+0000 7f801659c640 1 --2- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8018138230 0x7f8018138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:23.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:23.227+0000 7f8016d9d640 1 --2- 172.21.15.12:0/4092600457 >> v2:172.21.15.12:3300/0 conn(0x7f801809d430 0x7f80180a5730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:35:23.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:23.227+0000 7f801659c640 10 monclient(hunting): get_auth_request con 0x7f8018138230 auth_method 0 2024-09-11T01:35:23.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:23.227+0000 7f801659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:23.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:23.227+0000 7f801659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:23.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:23.227+0000 7f801659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:35:23.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:23.227+0000 7f801659c640 1 -- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8018138230 msgr2=0x7f8018138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:35:23.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:23.227+0000 7f801659c640 1 -- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8018138230 msgr2=0x7f8018138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:35:23.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:23.227+0000 7f801659c640 1 --2- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8018138230 0x7f8018138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:35:23.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:23.227+0000 7f801659c640 1 --2- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8018138230 0x7f8018138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:35:24.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:24.028+0000 7f800ffff640 1 -- 172.21.15.12:0/4092600457 >> v1:172.21.15.71:6789/0 conn(0x7f801809d030 legacy=0x7f80180a5020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:35:24.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:24.028+0000 7f8016d9d640 1 --2- 172.21.15.12:0/4092600457 >> v2:172.21.15.12:3300/0 conn(0x7f801809d430 0x7f80180a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:24.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:24.028+0000 7f801659c640 1 --2- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8018138230 0x7f8018138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:24.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:24.028+0000 7f8016d9d640 10 monclient(hunting): get_auth_request con 0x7f801809d430 auth_method 0 2024-09-11T01:35:24.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:24.028+0000 7f8016d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:24.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:24.028+0000 7f8016d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:24.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:24.028+0000 7f8016d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:35:24.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:24.028+0000 7f801659c640 10 monclient(hunting): get_auth_request con 0x7f8018138230 auth_method 0 2024-09-11T01:35:24.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:24.028+0000 7f801659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:24.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:24.028+0000 7f801659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:24.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:24.028+0000 7f801659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:35:24.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:24.028+0000 7f8016d9d640 1 -- 172.21.15.12:0/4092600457 >> v2:172.21.15.12:3300/0 conn(0x7f801809d430 msgr2=0x7f80180a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:35:24.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:24.028+0000 7f8016d9d640 1 -- 172.21.15.12:0/4092600457 >> v2:172.21.15.12:3300/0 conn(0x7f801809d430 msgr2=0x7f80180a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:35:24.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:24.028+0000 7f8016d9d640 1 --2- 172.21.15.12:0/4092600457 >> v2:172.21.15.12:3300/0 conn(0x7f801809d430 0x7f80180a5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:35:24.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:24.028+0000 7f8016d9d640 1 --2- 172.21.15.12:0/4092600457 >> v2:172.21.15.12:3300/0 conn(0x7f801809d430 0x7f80180a5730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:35:24.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:24.028+0000 7f801659c640 1 -- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8018138230 msgr2=0x7f8018138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:35:24.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:24.028+0000 7f801659c640 1 -- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8018138230 msgr2=0x7f8018138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:35:24.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:24.028+0000 7f801659c640 1 --2- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8018138230 0x7f8018138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:35:24.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:24.028+0000 7f801659c640 1 --2- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8018138230 0x7f8018138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:35:25.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:25.625+0000 7f8014d99640 10 monclient(hunting): tick 2024-09-11T01:35:25.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:25.625+0000 7f8014d99640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:35:25.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:25.625+0000 7f8014d99640 1 monclient(hunting): continuing hunt 2024-09-11T01:35:25.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:25.625+0000 7f8014d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:35:25.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:25.625+0000 7f8014d99640 1 -- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8018138230 msgr2=0x7f8018138610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:35:25.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:25.625+0000 7f8014d99640 1 --2- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8018138230 0x7f8018138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:25.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:25.625+0000 7f8014d99640 1 -- 172.21.15.12:0/4092600457 >> v2:172.21.15.12:3300/0 conn(0x7f801809d430 msgr2=0x7f80180a5730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:35:25.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:25.625+0000 7f8014d99640 1 --2- 172.21.15.12:0/4092600457 >> v2:172.21.15.12:3300/0 conn(0x7f801809d430 0x7f80180a5730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:25.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:25.625+0000 7f8014d99640 1 -- 172.21.15.12:0/4092600457 >> v1:172.21.15.71:6789/0 conn(0x7f801809d030 legacy=0x7f80180a5020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:35:25.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:25.625+0000 7f8014d99640 10 monclient: _add_conns ranks=[4,0] 2024-09-11T01:35:25.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:25.625+0000 7f8014d99640 1 --2- 172.21.15.12:0/4092600457 >> v2:172.21.15.71:3300/0 conn(0x7f7ff8003670 0x7f7ff8005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:25.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:25.625+0000 7f8014d99640 10 monclient(hunting): picked mon.noname-c con 0x7f7ff8003670 addr v2:172.21.15.71:3300/0 2024-09-11T01:35:25.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:25.625+0000 7f8014d99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7ff80061e0 addr v1:172.21.15.12:6789/0 2024-09-11T01:35:25.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:25.625+0000 7f8014d99640 1 -- 172.21.15.12:0/4092600457 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80180a66a0 con 0x7f7ff80061e0 2024-09-11T01:35:25.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:25.625+0000 7f8014d99640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:25.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:25.625+0000 7f8014d99640 1 -- 172.21.15.12:0/4092600457 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f801809c9c0 con 0x7f7ff8003670 2024-09-11T01:35:25.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:25.625+0000 7f8014d99640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:25.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:25.625+0000 7f801659c640 1 -- 172.21.15.12:0/4092600457 >> v2:172.21.15.71:3300/0 conn(0x7f7ff8003670 msgr2=0x7f7ff8005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:35:25.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:25.625+0000 7f801659c640 1 --2- 172.21.15.12:0/4092600457 >> v2:172.21.15.71:3300/0 conn(0x7f7ff8003670 0x7f7ff8005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:35:25.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:25.627+0000 7f800ffff640 1 -- 172.21.15.12:0/4092600457 >> v1:172.21.15.12:6789/0 conn(0x7f7ff80061e0 legacy=0x7f7ff80086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:25.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:25.826+0000 7f801659c640 1 -- 172.21.15.12:0/4092600457 >> v2:172.21.15.71:3300/0 conn(0x7f7ff8003670 msgr2=0x7f7ff8005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:35:25.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:25.826+0000 7f801659c640 1 --2- 172.21.15.12:0/4092600457 >> v2:172.21.15.71:3300/0 conn(0x7f7ff8003670 0x7f7ff8005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:35:26.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:26.227+0000 7f801659c640 1 -- 172.21.15.12:0/4092600457 >> v2:172.21.15.71:3300/0 conn(0x7f7ff8003670 msgr2=0x7f7ff8005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:35:26.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:26.227+0000 7f801659c640 1 --2- 172.21.15.12:0/4092600457 >> v2:172.21.15.71:3300/0 conn(0x7f7ff8003670 0x7f7ff8005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:35:27.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.028+0000 7f801659c640 1 -- 172.21.15.12:0/4092600457 >> v2:172.21.15.71:3300/0 conn(0x7f7ff8003670 msgr2=0x7f7ff8005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:35:27.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.028+0000 7f801659c640 1 --2- 172.21.15.12:0/4092600457 >> v2:172.21.15.71:3300/0 conn(0x7f7ff8003670 0x7f7ff8005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:35:27.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.530+0000 7f8015d9b640 1 -- 172.21.15.12:0/4092600457 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3438398641 0 0) 0x7f80180a66a0 con 0x7f7ff80061e0 2024-09-11T01:35:27.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.530+0000 7f8015d9b640 10 monclient(hunting): _negotiate 2024-09-11T01:35:27.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.530+0000 7f8015d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:27.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.530+0000 7f8015d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:27.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.530+0000 7f8015d9b640 10 monclient(hunting): my global_id is 54246 2024-09-11T01:35:27.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.530+0000 7f8015d9b640 1 -- 172.21.15.12:0/4092600457 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8004003520 con 0x7f7ff80061e0 2024-09-11T01:35:27.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 1 -- 172.21.15.12:0/4092600457 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1866926443 0 0) 0x7f8004003520 con 0x7f7ff80061e0 2024-09-11T01:35:27.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 1 -- 172.21.15.12:0/4092600457 >> v2:172.21.15.71:3300/0 conn(0x7f7ff8003670 msgr2=0x7f7ff8005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:35:27.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 1 --2- 172.21.15.12:0/4092600457 >> v2:172.21.15.71:3300/0 conn(0x7f7ff8003670 0x7f7ff8005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:27.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:35:27.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:35:27.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:27.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:35:27.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 1 -- 172.21.15.12:0/4092600457 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80180a6e30 con 0x7f7ff80061e0 2024-09-11T01:35:27.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 10 monclient: _finish_auth 0 2024-09-11T01:35:27.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:35:27.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:27.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 1 -- 172.21.15.12:0/4092600457 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f8000004500 con 0x7f7ff80061e0 2024-09-11T01:35:27.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f801cf4b640 5 monclient: authenticate success, global_id 54246 2024-09-11T01:35:27.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f801cf4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:35:27.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:27.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:35:27.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 10 monclient: dump: 2024-09-11T01:35:27.539 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:27.539 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:27.539 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:27.539 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:27.539 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:27.539 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:27.539 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:27.539 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:27.539 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:27.539 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:27.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:35:27.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:27.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 1 -- 172.21.15.12:0/4092600457 >> v1:172.21.15.12:6789/0 conn(0x7f7ff80061e0 legacy=0x7f7ff80086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:27.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:35:27.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8016d9d640 1 -- 172.21.15.12:0/4092600457 reap_dead start 2024-09-11T01:35:27.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 1 --2- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80040039b0 0x7f8004003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:27.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 10 monclient(hunting): picked mon.a con 0x7f80040039b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:35:27.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 1 --2- 172.21.15.12:0/4092600457 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8004004800 0x7f8004004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:27.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 10 monclient(hunting): picked mon.b con 0x7f8004004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:35:27.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 1 --2- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8004005370 0x7f800400b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:27.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 10 monclient(hunting): picked mon.c con 0x7f8004005370 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:35:27.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:27.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 1 -- 172.21.15.12:0/4092600457 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f801809c9c0 con 0x7f80040039b0 2024-09-11T01:35:27.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:27.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 1 -- 172.21.15.12:0/4092600457 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f800400c2c0 con 0x7f8004004800 2024-09-11T01:35:27.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:27.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 1 -- 172.21.15.12:0/4092600457 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f800400c4c0 con 0x7f8004005370 2024-09-11T01:35:27.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:27.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8015d9b640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:35:27.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f801cf4b640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:35:27.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f800ffff640 1 -- 172.21.15.12:0/4092600457 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8004004800 msgr2=0x7f8004004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:35:27.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f801659c640 1 --2- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80040039b0 0x7f8004003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:27.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8016d9d640 1 --2- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8004005370 0x7f800400b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:27.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f800ffff640 1 --2- 172.21.15.12:0/4092600457 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8004004800 0x7f8004004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:35:27.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8016d9d640 10 monclient(hunting): get_auth_request con 0x7f8004005370 auth_method 0 2024-09-11T01:35:27.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8016d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:27.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8016d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:27.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8016d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:35:27.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f801659c640 10 monclient(hunting): get_auth_request con 0x7f80040039b0 auth_method 0 2024-09-11T01:35:27.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f801659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:27.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f801659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:27.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f801659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:35:27.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8016d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:27.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8016d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:27.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8016d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:35:27.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f801659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:27.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f801659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:27.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f801659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:35:27.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8016d9d640 10 monclient(hunting): handle_auth_done global_id 54246 payload 293 2024-09-11T01:35:27.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8016d9d640 1 -- 172.21.15.12:0/4092600457 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8004004800 msgr2=0x7f8004004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:35:27.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8016d9d640 1 --2- 172.21.15.12:0/4092600457 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8004004800 0x7f8004004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:27.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8016d9d640 1 -- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80040039b0 msgr2=0x7f8004003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:35:27.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8016d9d640 1 --2- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80040039b0 0x7f8004003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:27.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8016d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:35:27.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8016d9d640 1 monclient: found mon.c 2024-09-11T01:35:27.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8016d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:27.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8016d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:35:27.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8016d9d640 1 -- 172.21.15.12:0/4092600457 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8004003520 con 0x7f8004005370 2024-09-11T01:35:27.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8016d9d640 10 monclient: _finish_auth 0 2024-09-11T01:35:27.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8016d9d640 10 monclient: _check_auth_tickets 2024-09-11T01:35:27.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.533+0000 7f8016d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:27.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.536+0000 7f8016d9d640 1 -- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8004005370 msgr2=0x7f800400b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:27.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.536+0000 7f8016d9d640 1 --2- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8004005370 0x7f800400b9e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8018069730 tx=0x7f8004005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a178227f2676a29 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:27.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.537+0000 7f8015d9b640 1 -- 172.21.15.12:0/4092600457 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8010031070 con 0x7f8004005370 2024-09-11T01:35:27.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.537+0000 7f8015d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:27.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.537+0000 7f8015d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:35:27.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.537+0000 7f8015d9b640 10 monclient: dump: 2024-09-11T01:35:27.549 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:27.549 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:27.549 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:27.549 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:27.549 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:27.549 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:27.550 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:27.550 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:27.550 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:27.550 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:27.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.537+0000 7f8015d9b640 1 -- 172.21.15.12:0/4092600457 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f801002b9b0 con 0x7f8004005370 2024-09-11T01:35:27.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.537+0000 7f801cf4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:35:27.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.537+0000 7f8015d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:35:27.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.537+0000 7f801cf4b640 10 monclient: get_monmap_and_config success 2024-09-11T01:35:27.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.537+0000 7f8015d9b640 1 -- 172.21.15.12:0/4092600457 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8010003e30 con 0x7f8004005370 2024-09-11T01:35:27.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.537+0000 7f8015d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:27.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.537+0000 7f801cf4b640 4 set_mon_vals no callback set 2024-09-11T01:35:27.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.537+0000 7f8015d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:35:27.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.537+0000 7f8015d9b640 10 monclient: dump: 2024-09-11T01:35:27.550 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:27.551 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:27.551 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:27.551 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:27.551 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:27.551 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:27.551 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:27.551 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:27.551 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:27.551 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:27.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.537+0000 7f801cf4b640 10 monclient: shutdown 2024-09-11T01:35:27.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.537+0000 7f801cf4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:35:27.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.537+0000 7f801cf4b640 1 -- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8004005370 msgr2=0x7f800400b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:27.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.537+0000 7f801cf4b640 1 --2- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8004005370 0x7f800400b9e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8018069730 tx=0x7f8004005830 comp rx=0 tx=0).stop 2024-09-11T01:35:27.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.537+0000 7f801cf4b640 1 -- 172.21.15.12:0/4092600457 shutdown_connections 2024-09-11T01:35:27.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.537+0000 7f801cf4b640 1 --2- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8004005370 0x7f800400b9e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:27.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.537+0000 7f801cf4b640 1 --2- 172.21.15.12:0/4092600457 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8004004800 0x7f8004004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:27.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.537+0000 7f801cf4b640 1 --2- 172.21.15.12:0/4092600457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80040039b0 0x7f8004003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:27.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.537+0000 7f801cf4b640 1 -- 172.21.15.12:0/4092600457 >> 172.21.15.12:0/4092600457 conn(0x7f80180a7120 msgr2=0x7f80180a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:35:27.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.538+0000 7f801cf4b640 1 -- 172.21.15.12:0/4092600457 shutdown_connections 2024-09-11T01:35:27.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.538+0000 7f801cf4b640 1 -- 172.21.15.12:0/4092600457 wait complete. 2024-09-11T01:35:27.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.539+0000 7f801cf4b640 10 monclient: build_initial_monmap 2024-09-11T01:35:27.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.539+0000 7f801cf4b640 10 monclient: monmap: 2024-09-11T01:35:27.552 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:35:27.552 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:35:27.552 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:35:27.552 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:35:27.553 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:35:27.553 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:27.553 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:35:27.553 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:35:27.553 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:35:27.553 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:27.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.539+0000 7f801cf4b640 1 Processor -- start 2024-09-11T01:35:27.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.539+0000 7f801cf4b640 1 -- start start 2024-09-11T01:35:27.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.539+0000 7f801cf4b640 10 monclient: init 2024-09-11T01:35:27.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801cf4b640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:27.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801cf4b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:35:27.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801cf4b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f80040039b0 0x7f80181cce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:27.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801cf4b640 10 monclient(hunting): picked mon.noname-c con 0x7f80040039b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:35:27.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801cf4b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8004004800 0x7f80181cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:27.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801cf4b640 10 monclient(hunting): picked mon.noname-b con 0x7f8004004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:35:27.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801cf4b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8004005370 0x7f80181245e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:27.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801cf4b640 10 monclient(hunting): picked mon.noname-a con 0x7f8004005370 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:35:27.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801cf4b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:27.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801cf4b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8018138790 con 0x7f8004005370 2024-09-11T01:35:27.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801cf4b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:27.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801cf4b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8018138610 con 0x7f8004004800 2024-09-11T01:35:27.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801cf4b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:27.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801cf4b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8018138910 con 0x7f80040039b0 2024-09-11T01:35:27.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801cf4b640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:27.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801cf4b640 10 monclient(hunting): authenticate will time out at 1133.983229s 2024-09-11T01:35:27.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f80040039b0 0x7f80181cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:27.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f800ffff640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8004004800 msgr2=0x7f80181cd3d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:35:27.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f80040039b0 0x7f80181cce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45412/0 (socket says 172.21.15.12:45412) 2024-09-11T01:35:27.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801659c640 1 -- 172.21.15.12:0/678954205 learned_addr learned my addr 172.21.15.12:0/678954205 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:35:27.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801659c640 10 monclient(hunting): get_auth_request con 0x7f80040039b0 auth_method 0 2024-09-11T01:35:27.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:27.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:27.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:27.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f8016d9d640 1 --2- 172.21.15.12:0/678954205 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8004005370 0x7f80181245e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:27.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f800ffff640 1 --2- 172.21.15.12:0/678954205 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8004004800 0x7f80181cd3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:35:27.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f8016d9d640 10 monclient(hunting): get_auth_request con 0x7f8004005370 auth_method 0 2024-09-11T01:35:27.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f8016d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:27.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f8016d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:27.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f8016d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:27.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:27.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:27.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:27.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f8016d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:27.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f8016d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:27.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f8016d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:27.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801659c640 10 monclient(hunting): handle_auth_done global_id 64157 payload 824 2024-09-11T01:35:27.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801659c640 1 -- 172.21.15.12:0/678954205 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8004004800 msgr2=0x7f80181cd3d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:35:27.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801659c640 1 --2- 172.21.15.12:0/678954205 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8004004800 0x7f80181cd3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:27.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801659c640 1 -- 172.21.15.12:0/678954205 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8004005370 msgr2=0x7f80181245e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:27.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801659c640 1 --2- 172.21.15.12:0/678954205 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8004005370 0x7f80181245e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:27.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801659c640 10 monclient: _finish_hunting 0 2024-09-11T01:35:27.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801659c640 1 monclient: found mon.noname-c 2024-09-11T01:35:27.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:27.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:35:27.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f801659c640 1 -- 172.21.15.12:0/678954205 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80181379c0 con 0x7f80040039b0 2024-09-11T01:35:27.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.540+0000 7f8016d9d640 1 --2- 172.21.15.12:0/678954205 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8004005370 0x7f80181245e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:35:27.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.543+0000 7f801659c640 1 -- 172.21.15.12:0/678954205 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f80040039b0 msgr2=0x7f80181cce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:27.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.543+0000 7f801659c640 1 --2- 172.21.15.12:0/678954205 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f80040039b0 0x7f80181cce90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8008017a50 tx=0x7f8008017f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:27.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.543+0000 7f800effd640 1 -- 172.21.15.12:0/678954205 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f800800f040 con 0x7f80040039b0 2024-09-11T01:35:27.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.543+0000 7f800effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:27.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.543+0000 7f800effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:35:27.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.543+0000 7f800effd640 10 monclient: dump: 2024-09-11T01:35:27.557 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:27.557 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:27.557 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:27.557 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:27.558 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:27.558 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:27.558 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:27.558 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:27.558 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:27.558 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:27.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.543+0000 7f800effd640 10 monclient: _finish_auth 0 2024-09-11T01:35:27.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.543+0000 7f800effd640 10 monclient: _check_auth_tickets 2024-09-11T01:35:27.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.543+0000 7f800effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:27.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.543+0000 7f800effd640 1 -- 172.21.15.12:0/678954205 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8008015cd0 con 0x7f80040039b0 2024-09-11T01:35:27.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.543+0000 7f800effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:35:27.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.543+0000 7f800effd640 1 -- 172.21.15.12:0/678954205 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8008018670 con 0x7f80040039b0 2024-09-11T01:35:27.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.543+0000 7f800effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:27.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.543+0000 7f800effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:35:27.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.543+0000 7f800effd640 10 monclient: dump: 2024-09-11T01:35:27.559 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:27.559 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:27.559 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:27.559 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:27.559 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:27.559 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:27.559 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:27.559 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:27.559 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:27.559 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:27.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.543+0000 7f801cf4b640 5 monclient: authenticate success, global_id 64157 2024-09-11T01:35:27.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.543+0000 7f801cf4b640 10 monclient: _renew_subs 2024-09-11T01:35:27.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.543+0000 7f801cf4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:35:27.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.543+0000 7f801cf4b640 1 -- 172.21.15.12:0/678954205 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8018128400 con 0x7f80040039b0 2024-09-11T01:35:27.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.543+0000 7f801559a640 4 set_mon_vals no callback set 2024-09-11T01:35:27.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.543+0000 7f801cf4b640 10 monclient: _renew_subs 2024-09-11T01:35:27.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.543+0000 7f801cf4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:35:27.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.543+0000 7f801cf4b640 1 -- 172.21.15.12:0/678954205 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f80180a9cf0 con 0x7f80040039b0 2024-09-11T01:35:27.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.544+0000 7f800effd640 1 -- 172.21.15.12:0/678954205 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f8008015040 con 0x7f80040039b0 2024-09-11T01:35:27.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.544+0000 7f801cf4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:27.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.544+0000 7f801cf4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:27.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.544+0000 7f801cf4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:35:27.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.544+0000 7f801cf4b640 1 -- 172.21.15.12:0/678954205 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80180a5020 con 0x7f80040039b0 2024-09-11T01:35:27.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.544+0000 7f800effd640 1 --2- 172.21.15.12:0/678954205 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7ff803e910 0x7f7ff8040dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:27.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.545+0000 7f800effd640 1 -- 172.21.15.12:0/678954205 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(202..202 src has 1..202) ==== 4755+0+0 (secure 0 0 0) 0x7f8008053e10 con 0x7f80040039b0 2024-09-11T01:35:27.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.545+0000 7f800ffff640 1 --2- 172.21.15.12:0/678954205 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7ff803e910 0x7f7ff8040dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:27.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.545+0000 7f800ffff640 10 monclient: get_auth_request con 0x7f7ff803e910 auth_method 0 2024-09-11T01:35:27.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.548+0000 7f800ffff640 1 --2- 172.21.15.12:0/678954205 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7ff803e910 0x7f7ff8040dd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8018127190 tx=0x7f8000004280 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:27.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.550+0000 7f800effd640 1 -- 172.21.15.12:0/678954205 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f800801f380 con 0x7f80040039b0 2024-09-11T01:35:27.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.550+0000 7f800effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:27.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.550+0000 7f800effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:35:27.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.810+0000 7f801cf4b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:35:27.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.810+0000 7f801cf4b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:35:27.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.810+0000 7f801cf4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:35:27.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.810+0000 7f801cf4b640 1 -- 172.21.15.12:0/678954205 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8018126c00 con 0x7f80040039b0 2024-09-11T01:35:27.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.810+0000 7f800effd640 1 -- 172.21.15.12:0/678954205 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f8008055020 con 0x7f80040039b0 2024-09-11T01:35:27.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.810+0000 7f800effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:35:27.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.810+0000 7f800effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:35:27.813 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":68,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:35:27.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.812+0000 7f801cf4b640 1 -- 172.21.15.12:0/678954205 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7ff803e910 msgr2=0x7f7ff8040dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:27.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.812+0000 7f801cf4b640 1 --2- 172.21.15.12:0/678954205 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7ff803e910 0x7f7ff8040dd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8018127190 tx=0x7f8000004280 comp rx=0 tx=0).stop 2024-09-11T01:35:27.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.812+0000 7f801cf4b640 10 monclient: shutdown 2024-09-11T01:35:27.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.812+0000 7f801cf4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:35:27.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.812+0000 7f801cf4b640 1 -- 172.21.15.12:0/678954205 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f80040039b0 msgr2=0x7f80181cce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:27.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.812+0000 7f801cf4b640 1 --2- 172.21.15.12:0/678954205 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f80040039b0 0x7f80181cce90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8008017a50 tx=0x7f8008017f10 comp rx=0 tx=0).stop 2024-09-11T01:35:27.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.812+0000 7f801cf4b640 1 -- 172.21.15.12:0/678954205 shutdown_connections 2024-09-11T01:35:27.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.812+0000 7f801cf4b640 1 --2- 172.21.15.12:0/678954205 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7ff803e910 0x7f7ff8040dd0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:27.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.812+0000 7f801cf4b640 1 --2- 172.21.15.12:0/678954205 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8004005370 0x7f80181245e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:27.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.812+0000 7f801cf4b640 1 --2- 172.21.15.12:0/678954205 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8004004800 0x7f80181cd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:27.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.812+0000 7f801cf4b640 1 --2- 172.21.15.12:0/678954205 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f80040039b0 0x7f80181cce90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:27.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.812+0000 7f801cf4b640 1 -- 172.21.15.12:0/678954205 >> 172.21.15.12:0/678954205 conn(0x7f80180a7120 msgr2=0x7f80180a2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:35:27.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.812+0000 7f801cf4b640 1 -- 172.21.15.12:0/678954205 shutdown_connections 2024-09-11T01:35:27.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.812+0000 7f801cf4b640 1 -- 172.21.15.12:0/678954205 wait complete. 2024-09-11T01:35:27.826 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:35:27.826 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-11T01:35:27.826 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:35:27.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.925+0000 7ff02b3b5640 10 monclient: get_monmap_and_config 2024-09-11T01:35:27.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.925+0000 7ff02b3b5640 10 monclient: build_initial_monmap 2024-09-11T01:35:27.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.925+0000 7ff02b3b5640 10 monclient: monmap: 2024-09-11T01:35:27.928 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:35:27.928 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:27.928 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:35:27.927409+0000 2024-09-11T01:35:27.928 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:35:27.927409+0000 2024-09-11T01:35:27.928 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:35:27.929 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:27.929 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:35:27.929 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:35:27.929 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:35:27.929 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:35:27.929 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:35:27.929 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:27.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.926+0000 7ff02b3b5640 1 Processor -- start 2024-09-11T01:35:27.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.926+0000 7ff02b3b5640 1 -- start start 2024-09-11T01:35:27.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.926+0000 7ff02b3b5640 10 monclient: init 2024-09-11T01:35:27.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.926+0000 7ff02b3b5640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:27.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.926+0000 7ff02b3b5640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-11T01:35:27.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.926+0000 7ff02b3b5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff0240a2d40 addr v1:172.21.15.71:6789/0 2024-09-11T01:35:27.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.926+0000 7ff02b3b5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff02406d260 addr v1:172.21.15.12:6789/0 2024-09-11T01:35:27.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.926+0000 7ff02b3b5640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff02406d670 0x7ff024140c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:27.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.926+0000 7ff02b3b5640 10 monclient(hunting): picked mon.noname-c con 0x7ff02406d670 addr v2:172.21.15.71:3300/0 2024-09-11T01:35:27.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.926+0000 7ff02b3b5640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff02409aed0 con 0x7ff02406d260 2024-09-11T01:35:27.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.926+0000 7ff02b3b5640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0240a3920 con 0x7ff0240a2d40 2024-09-11T01:35:27.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.926+0000 7ff02b3b5640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:27.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.926+0000 7ff02b3b5640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7ff02406dd10 con 0x7ff02406d670 2024-09-11T01:35:27.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.926+0000 7ff02b3b5640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:27.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.926+0000 7ff02b3b5640 10 monclient(hunting): authenticate will time out at 1134.369917s 2024-09-11T01:35:27.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.926+0000 7ff029151640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7ff0240a2d40 legacy=0x7ff02406cb50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:35:27.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.927+0000 7ff029952640 1 -- >> v2:172.21.15.71:3300/0 conn(0x7ff02406d670 msgr2=0x7ff024140c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:35:27.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.927+0000 7ff029952640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff02406d670 0x7ff024140c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:35:27.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.927+0000 7ff028950640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff02406d260 0x7ff02413d4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:49218/0 (socket says 172.21.15.12:49218) 2024-09-11T01:35:27.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.927+0000 7ff028950640 1 -- 172.21.15.12:0/1227464531 learned_addr learned my addr 172.21.15.12:0/1227464531 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:35:27.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff028950640 1 -- 172.21.15.12:0/1227464531 >> v1:172.21.15.12:6789/0 conn(0x7ff02406d260 legacy=0x7ff02413d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:27.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 1 -- 172.21.15.12:0/1227464531 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1579585695 0 0) 0x7ff02409aed0 con 0x7ff02406d260 2024-09-11T01:35:27.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 10 monclient(hunting): _negotiate 2024-09-11T01:35:27.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:27.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:27.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 10 monclient(hunting): my global_id is 54252 2024-09-11T01:35:27.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 1 -- 172.21.15.12:0/1227464531 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff010003520 con 0x7ff02406d260 2024-09-11T01:35:27.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 1 -- 172.21.15.12:0/1227464531 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2254306243 0 0) 0x7ff010003520 con 0x7ff02406d260 2024-09-11T01:35:27.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 1 -- 172.21.15.12:0/1227464531 >> v2:172.21.15.71:3300/0 conn(0x7ff02406d670 msgr2=0x7ff024140c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:35:27.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 1 --2- 172.21.15.12:0/1227464531 >> v2:172.21.15.71:3300/0 conn(0x7ff02406d670 0x7ff024140c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:27.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 1 -- 172.21.15.12:0/1227464531 >> v1:172.21.15.71:6789/0 conn(0x7ff0240a2d40 legacy=0x7ff02406cb50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:35:27.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:35:27.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:35:27.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:27.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:35:27.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 1 -- 172.21.15.12:0/1227464531 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0240a3320 con 0x7ff02406d260 2024-09-11T01:35:27.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 10 monclient: _finish_auth 0 2024-09-11T01:35:27.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:35:27.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:27.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 1 -- 172.21.15.12:0/1227464531 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7ff0140019a0 con 0x7ff02406d260 2024-09-11T01:35:27.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff02b3b5640 5 monclient: authenticate success, global_id 54252 2024-09-11T01:35:27.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff02b3b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:35:27.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:27.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:35:27.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 10 monclient: dump: 2024-09-11T01:35:27.936 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:27.936 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:27.937 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:27.937 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:27.937 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:27.937 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:27.937 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:27.937 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:27.937 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:27.937 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:27.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:35:27.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:27.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 1 -- 172.21.15.12:0/1227464531 >> v1:172.21.15.12:6789/0 conn(0x7ff02406d260 legacy=0x7ff02413d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:27.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:35:27.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 1 --2- 172.21.15.12:0/1227464531 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff010003a00 0x7ff010003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:27.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 10 monclient(hunting): picked mon.c con 0x7ff010003a00 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:35:27.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 1 --2- 172.21.15.12:0/1227464531 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff010004800 0x7ff010004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:27.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 10 monclient(hunting): picked mon.b con 0x7ff010004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:35:27.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 1 --2- 172.21.15.12:0/1227464531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff010005410 0x7ff01000ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:27.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 10 monclient(hunting): picked mon.a con 0x7ff010005410 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:35:27.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:27.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 1 -- 172.21.15.12:0/1227464531 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff01000c320 con 0x7ff010005410 2024-09-11T01:35:27.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:27.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 1 -- 172.21.15.12:0/1227464531 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff01000c4d0 con 0x7ff010004800 2024-09-11T01:35:27.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:27.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 1 -- 172.21.15.12:0/1227464531 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff01000c6d0 con 0x7ff010003a00 2024-09-11T01:35:27.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:27.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff01bfff640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:35:27.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff02b3b5640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:35:27.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff029151640 1 --2- 172.21.15.12:0/1227464531 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff010003a00 0x7ff010003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:27.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff029952640 1 --2- 172.21.15.12:0/1227464531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff010005410 0x7ff01000ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:27.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff028950640 1 -- 172.21.15.12:0/1227464531 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff010004800 msgr2=0x7ff010004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:35:27.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff029151640 10 monclient(hunting): get_auth_request con 0x7ff010003a00 auth_method 0 2024-09-11T01:35:27.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff029151640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:27.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff029151640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:27.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff029151640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:35:27.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff029952640 10 monclient(hunting): get_auth_request con 0x7ff010005410 auth_method 0 2024-09-11T01:35:27.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff029952640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:27.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff029952640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:27.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff029952640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:35:27.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.929+0000 7ff028950640 1 --2- 172.21.15.12:0/1227464531 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff010004800 0x7ff010004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:35:27.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.930+0000 7ff029151640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:27.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.930+0000 7ff029151640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:27.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.930+0000 7ff029151640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:35:27.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.930+0000 7ff029952640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:27.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.930+0000 7ff029952640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:27.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.930+0000 7ff029952640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:35:27.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.930+0000 7ff029151640 10 monclient(hunting): handle_auth_done global_id 54252 payload 293 2024-09-11T01:35:27.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.930+0000 7ff029151640 1 -- 172.21.15.12:0/1227464531 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff010004800 msgr2=0x7ff010004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:35:27.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.930+0000 7ff029151640 1 --2- 172.21.15.12:0/1227464531 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff010004800 0x7ff010004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:27.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.930+0000 7ff029151640 1 -- 172.21.15.12:0/1227464531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff010005410 msgr2=0x7ff01000ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:35:27.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.930+0000 7ff029151640 1 --2- 172.21.15.12:0/1227464531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff010005410 0x7ff01000ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:27.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.930+0000 7ff029151640 10 monclient: _finish_hunting 0 2024-09-11T01:35:27.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.930+0000 7ff029151640 1 monclient: found mon.c 2024-09-11T01:35:27.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.930+0000 7ff029151640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:27.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.930+0000 7ff029151640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:35:27.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.930+0000 7ff029151640 1 -- 172.21.15.12:0/1227464531 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff010003520 con 0x7ff010003a00 2024-09-11T01:35:27.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.930+0000 7ff029151640 10 monclient: _finish_auth 0 2024-09-11T01:35:27.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.930+0000 7ff029151640 10 monclient: _check_auth_tickets 2024-09-11T01:35:27.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.930+0000 7ff029151640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:27.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.930+0000 7ff029952640 1 --2- 172.21.15.12:0/1227464531 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff010005410 0x7ff01000ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:35:27.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.930+0000 7ff029952640 1 -- 172.21.15.12:0/1227464531 reap_dead start 2024-09-11T01:35:27.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.932+0000 7ff029151640 1 -- 172.21.15.12:0/1227464531 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff010003a00 msgr2=0x7ff010003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:27.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.932+0000 7ff029151640 1 --2- 172.21.15.12:0/1227464531 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff010003a00 0x7ff010003e00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff020009900 tx=0x7ff010005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c6e8c6569cd36b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:27.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.933+0000 7ff01bfff640 1 -- 172.21.15.12:0/1227464531 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff020009500 con 0x7ff010003a00 2024-09-11T01:35:27.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.933+0000 7ff01bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:27.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.933+0000 7ff01bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:35:27.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.933+0000 7ff01bfff640 10 monclient: dump: 2024-09-11T01:35:27.948 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:27.948 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:27.948 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:27.948 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:27.949 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:27.949 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:27.949 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:27.949 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:27.949 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:27.949 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:27.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.933+0000 7ff02b3b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:35:27.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.933+0000 7ff01bfff640 1 -- 172.21.15.12:0/1227464531 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff02002e070 con 0x7ff010003a00 2024-09-11T01:35:27.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.933+0000 7ff01bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:35:27.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.933+0000 7ff02b3b5640 10 monclient: get_monmap_and_config success 2024-09-11T01:35:27.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.933+0000 7ff01bfff640 1 -- 172.21.15.12:0/1227464531 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02002a910 con 0x7ff010003a00 2024-09-11T01:35:27.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.933+0000 7ff01bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:27.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.933+0000 7ff02b3b5640 4 set_mon_vals no callback set 2024-09-11T01:35:27.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.933+0000 7ff01bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:35:27.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.933+0000 7ff01bfff640 10 monclient: dump: 2024-09-11T01:35:27.950 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:27.950 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:27.950 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:27.950 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:27.950 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:27.950 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:27.950 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:27.950 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:27.950 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:27.950 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:27.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.933+0000 7ff02b3b5640 10 monclient: shutdown 2024-09-11T01:35:27.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.933+0000 7ff02b3b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:35:27.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.933+0000 7ff02b3b5640 1 -- 172.21.15.12:0/1227464531 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff010003a00 msgr2=0x7ff010003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:27.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.933+0000 7ff02b3b5640 1 --2- 172.21.15.12:0/1227464531 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff010003a00 0x7ff010003e00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff020009900 tx=0x7ff010005830 comp rx=0 tx=0).stop 2024-09-11T01:35:27.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.933+0000 7ff02b3b5640 1 -- 172.21.15.12:0/1227464531 shutdown_connections 2024-09-11T01:35:27.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.933+0000 7ff02b3b5640 1 --2- 172.21.15.12:0/1227464531 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff010003a00 0x7ff010003e00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:27.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.933+0000 7ff02b3b5640 1 -- 172.21.15.12:0/1227464531 >> 172.21.15.12:0/1227464531 conn(0x7ff0240a3e30 msgr2=0x7ff0240a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:35:27.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.933+0000 7ff02b3b5640 1 -- 172.21.15.12:0/1227464531 shutdown_connections 2024-09-11T01:35:27.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.933+0000 7ff02b3b5640 1 -- 172.21.15.12:0/1227464531 wait complete. 2024-09-11T01:35:27.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.934+0000 7ff02b3b5640 10 monclient: build_initial_monmap 2024-09-11T01:35:27.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.934+0000 7ff02b3b5640 10 monclient: monmap: 2024-09-11T01:35:27.951 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:35:27.951 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:35:27.951 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:35:27.951 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:35:27.951 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:35:27.951 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:27.952 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:35:27.952 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:35:27.952 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:35:27.952 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:27.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.934+0000 7ff02b3b5640 1 Processor -- start 2024-09-11T01:35:27.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.934+0000 7ff02b3b5640 1 -- start start 2024-09-11T01:35:27.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.934+0000 7ff02b3b5640 10 monclient: init 2024-09-11T01:35:27.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff02b3b5640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:27.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff02b3b5640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:35:27.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff02b3b5640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff010003a00 0x7ff024099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:27.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff02b3b5640 10 monclient(hunting): picked mon.noname-b con 0x7ff010003a00 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:35:27.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff02b3b5640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff024093aa0 0x7ff024093f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:27.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff02b3b5640 10 monclient(hunting): picked mon.noname-c con 0x7ff024093aa0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:35:27.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff02b3b5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024094440 0x7ff0241d3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:27.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff02b3b5640 10 monclient(hunting): picked mon.noname-a con 0x7ff024094440 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:35:27.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff02b3b5640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:27.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff02b3b5640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff0240a3120 con 0x7ff024094440 2024-09-11T01:35:27.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff02b3b5640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:27.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff02b3b5640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff02409bcf0 con 0x7ff010003a00 2024-09-11T01:35:27.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff02b3b5640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:27.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff02b3b5640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff02409ba10 con 0x7ff024093aa0 2024-09-11T01:35:27.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff02b3b5640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:27.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff02b3b5640 10 monclient(hunting): authenticate will time out at 1134.378186s 2024-09-11T01:35:27.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff029151640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff010003a00 msgr2=0x7ff024099ea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:35:27.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff029151640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff010003a00 0x7ff024099ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:35:27.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff028950640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff024093aa0 0x7ff024093f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:27.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff029952640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024094440 0x7ff0241d3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:27.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff028950640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff024093aa0 0x7ff024093f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45428/0 (socket says 172.21.15.12:45428) 2024-09-11T01:35:27.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff028950640 1 -- 172.21.15.12:0/1015595028 learned_addr learned my addr 172.21.15.12:0/1015595028 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:35:27.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff029952640 10 monclient(hunting): get_auth_request con 0x7ff024094440 auth_method 0 2024-09-11T01:35:27.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff029952640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:27.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff029952640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:27.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff029952640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:27.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff028950640 10 monclient(hunting): get_auth_request con 0x7ff024093aa0 auth_method 0 2024-09-11T01:35:27.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff028950640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:27.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff028950640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:27.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff028950640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:27.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff029952640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:27.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff029952640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:27.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff029952640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:27.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff028950640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:27.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff028950640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:27.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff028950640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:27.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff029952640 10 monclient(hunting): handle_auth_done global_id 54255 payload 824 2024-09-11T01:35:27.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff029952640 1 -- 172.21.15.12:0/1015595028 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff024093aa0 msgr2=0x7ff024093f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:27.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff029952640 1 --2- 172.21.15.12:0/1015595028 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff024093aa0 0x7ff024093f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:27.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff029952640 1 -- 172.21.15.12:0/1015595028 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff010003a00 msgr2=0x7ff024099ea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:35:27.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff029952640 1 --2- 172.21.15.12:0/1015595028 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff010003a00 0x7ff024099ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:27.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff029952640 10 monclient: _finish_hunting 0 2024-09-11T01:35:27.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff029952640 1 monclient: found mon.noname-a 2024-09-11T01:35:27.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff029952640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:27.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff029952640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:35:27.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff029952640 1 -- 172.21.15.12:0/1015595028 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff024095f80 con 0x7ff024094440 2024-09-11T01:35:27.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.935+0000 7ff028950640 1 --2- 172.21.15.12:0/1015595028 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff024093aa0 0x7ff024093f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:35:27.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.937+0000 7ff029952640 1 -- 172.21.15.12:0/1015595028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024094440 msgr2=0x7ff0241d3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:27.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.937+0000 7ff029952640 1 --2- 172.21.15.12:0/1015595028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024094440 0x7ff0241d3540 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff02406dc30 tx=0x7ff01c00c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:27.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.937+0000 7ff019ffb640 1 -- 172.21.15.12:0/1015595028 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff01c009a30 con 0x7ff024094440 2024-09-11T01:35:27.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.937+0000 7ff019ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:27.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.937+0000 7ff019ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:35:27.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.937+0000 7ff019ffb640 10 monclient: dump: 2024-09-11T01:35:27.956 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:27.956 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:27.956 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:27.957 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:27.957 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:27.957 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:27.957 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:27.957 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:27.957 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:27.957 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:27.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.938+0000 7ff019ffb640 10 monclient: _finish_auth 0 2024-09-11T01:35:27.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.938+0000 7ff019ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:35:27.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.938+0000 7ff019ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:27.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.938+0000 7ff019ffb640 1 -- 172.21.15.12:0/1015595028 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff01c005320 con 0x7ff024094440 2024-09-11T01:35:27.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.938+0000 7ff02b3b5640 5 monclient: authenticate success, global_id 54255 2024-09-11T01:35:27.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.938+0000 7ff02b3b5640 10 monclient: _renew_subs 2024-09-11T01:35:27.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.938+0000 7ff02b3b5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:35:27.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.938+0000 7ff02b3b5640 1 -- 172.21.15.12:0/1015595028 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0240948d0 con 0x7ff024094440 2024-09-11T01:35:27.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.938+0000 7ff019ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:35:27.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.938+0000 7ff019ffb640 1 -- 172.21.15.12:0/1015595028 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff01c0056f0 con 0x7ff024094440 2024-09-11T01:35:27.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.938+0000 7ff019ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:27.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.938+0000 7ff019ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:35:27.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.938+0000 7ff019ffb640 10 monclient: dump: 2024-09-11T01:35:27.958 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:27.958 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:27.958 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:27.958 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:27.958 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:27.958 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:27.958 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:27.958 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:27.959 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:27.959 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:27.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.938+0000 7ff02b3b5640 10 monclient: _renew_subs 2024-09-11T01:35:27.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.938+0000 7ff01b7fe640 4 set_mon_vals no callback set 2024-09-11T01:35:27.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.938+0000 7ff02b3b5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:35:27.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.938+0000 7ff02b3b5640 1 -- 172.21.15.12:0/1015595028 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff024099370 con 0x7ff024094440 2024-09-11T01:35:27.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.938+0000 7ff019ffb640 1 -- 172.21.15.12:0/1015595028 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff01c005890 con 0x7ff024094440 2024-09-11T01:35:27.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.938+0000 7ff02b3b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:27.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.938+0000 7ff02b3b5640 1 --2- 172.21.15.12:0/1015595028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff02402e5c0 0x7ff024063d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:27.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.938+0000 7ff02b3b5640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:27.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.938+0000 7ff02b3b5640 1 -- 172.21.15.12:0/1015595028 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff02402e9a0 con 0x7ff02402e5c0 2024-09-11T01:35:27.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.939+0000 7ff029151640 1 --2- 172.21.15.12:0/1015595028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff02402e5c0 0x7ff024063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:27.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.939+0000 7ff029151640 10 monclient: get_auth_request con 0x7ff02402e5c0 auth_method 0 2024-09-11T01:35:27.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.939+0000 7ff029151640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:27.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.939+0000 7ff029151640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:27.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.939+0000 7ff029151640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:27.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.939+0000 7ff019ffb640 1 --2- 172.21.15.12:0/1015595028 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff00803e240 0x7ff008040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:27.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.939+0000 7ff029151640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:27.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.939+0000 7ff029151640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:27.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.939+0000 7ff029151640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:27.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.939+0000 7ff019ffb640 1 -- 172.21.15.12:0/1015595028 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(202..202 src has 1..202) ==== 4755+0+0 (secure 0 0 0) 0x7ff01c053480 con 0x7ff024094440 2024-09-11T01:35:27.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.939+0000 7ff029151640 10 monclient(hunting): handle_auth_done global_id 54258 payload 824 2024-09-11T01:35:27.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.939+0000 7ff029151640 1 -- 172.21.15.12:0/1015595028 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff02406cc60 con 0x7ff02402e5c0 2024-09-11T01:35:27.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.939+0000 7ff028950640 1 --2- 172.21.15.12:0/1015595028 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff00803e240 0x7ff008040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:27.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.939+0000 7ff028950640 10 monclient: get_auth_request con 0x7ff00803e240 auth_method 0 2024-09-11T01:35:27.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.941+0000 7ff029151640 1 -- 172.21.15.12:0/1015595028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff02402e5c0 msgr2=0x7ff024063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:27.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.941+0000 7ff029151640 1 --2- 172.21.15.12:0/1015595028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff02402e5c0 0x7ff024063d50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff020003930 tx=0x7ff020005a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:27.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.941+0000 7ff019ffb640 1 -- 172.21.15.12:0/1015595028 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff020005280 con 0x7ff02402e5c0 2024-09-11T01:35:27.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.941+0000 7ff019ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:27.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.941+0000 7ff019ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:35:27.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.941+0000 7ff019ffb640 10 monclient: dump: 2024-09-11T01:35:27.961 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:27.961 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:27.961 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:27.961 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:27.961 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:27.961 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:27.961 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:27.961 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:27.961 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:27.962 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:27.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.942+0000 7ff019ffb640 1 -- 172.21.15.12:0/1015595028 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff02002e040 con 0x7ff02402e5c0 2024-09-11T01:35:27.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.942+0000 7ff019ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:27.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.942+0000 7ff019ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:35:27.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.942+0000 7ff019ffb640 1 -- 172.21.15.12:0/1015595028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff02402e5c0 msgr2=0x7ff024063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:27.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.942+0000 7ff019ffb640 1 --2- 172.21.15.12:0/1015595028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff02402e5c0 0x7ff024063d50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff020003930 tx=0x7ff020005a80 comp rx=0 tx=0).stop 2024-09-11T01:35:27.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.942+0000 7ff019ffb640 1 -- 172.21.15.12:0/1015595028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff02402e5c0 msgr2=0x7ff024063d50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:35:27.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.942+0000 7ff019ffb640 1 --2- 172.21.15.12:0/1015595028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff02402e5c0 0x7ff024063d50 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff020003930 tx=0x7ff020005a80 comp rx=0 tx=0).stop 2024-09-11T01:35:27.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.942+0000 7ff028950640 1 --2- 172.21.15.12:0/1015595028 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff00803e240 0x7ff008040700 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff014001b90 tx=0x7ff014001a20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:27.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.948+0000 7ff02b3b5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:35:27.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.948+0000 7ff02b3b5640 1 --2- 172.21.15.12:0/1015595028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024064290 0x7ff0241d7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:27.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.948+0000 7ff02b3b5640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:27.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.948+0000 7ff02b3b5640 1 -- 172.21.15.12:0/1015595028 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff024097840 con 0x7ff024064290 2024-09-11T01:35:27.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.948+0000 7ff029151640 1 --2- 172.21.15.12:0/1015595028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024064290 0x7ff0241d7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:27.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.948+0000 7ff029151640 10 monclient: get_auth_request con 0x7ff024064290 auth_method 0 2024-09-11T01:35:27.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.948+0000 7ff029151640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:27.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.948+0000 7ff029151640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:27.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.948+0000 7ff029151640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:27.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.948+0000 7ff029151640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:27.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.948+0000 7ff029151640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:27.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.948+0000 7ff029151640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:27.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.948+0000 7ff029151640 10 monclient(hunting): handle_auth_done global_id 54261 payload 824 2024-09-11T01:35:27.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.948+0000 7ff029151640 1 -- 172.21.15.12:0/1015595028 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff024073d60 con 0x7ff024064290 2024-09-11T01:35:27.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.950+0000 7ff029151640 1 -- 172.21.15.12:0/1015595028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024064290 msgr2=0x7ff0241d7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:27.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.951+0000 7ff029151640 1 --2- 172.21.15.12:0/1015595028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024064290 0x7ff0241d7aa0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff020009900 tx=0x7ff020005a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:27.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.951+0000 7ff019ffb640 1 -- 172.21.15.12:0/1015595028 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff020005690 con 0x7ff024064290 2024-09-11T01:35:27.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.951+0000 7ff019ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:27.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.951+0000 7ff019ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:35:27.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.951+0000 7ff019ffb640 10 monclient: dump: 2024-09-11T01:35:27.965 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:27.965 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:27.965 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:27.966 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:27.966 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:27.966 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:27.966 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:27.966 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:27.966 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:27.966 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:27.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.951+0000 7ff019ffb640 1 -- 172.21.15.12:0/1015595028 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4866 (secure 0 0 0) 0x7ff020005690 con 0x7ff024064290 2024-09-11T01:35:27.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.951+0000 7ff019ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:35:27.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.951+0000 7ff019ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:35:27.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.951+0000 7ff019ffb640 1 -- 172.21.15.12:0/1015595028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024064290 msgr2=0x7ff0241d7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:27.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.951+0000 7ff019ffb640 1 --2- 172.21.15.12:0/1015595028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024064290 0x7ff0241d7aa0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff020009900 tx=0x7ff020005a10 comp rx=0 tx=0).stop 2024-09-11T01:35:27.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.951+0000 7ff019ffb640 1 -- 172.21.15.12:0/1015595028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024064290 msgr2=0x7ff0241d7aa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:35:27.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.951+0000 7ff019ffb640 1 --2- 172.21.15.12:0/1015595028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024064290 0x7ff0241d7aa0 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff020009900 tx=0x7ff020005a10 comp rx=0 tx=0).stop 2024-09-11T01:35:27.967 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:35:27.967 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:35:27.967 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:35:27.967 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:35:27.967 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 68, 2024-09-11T01:35:27.967 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 117933, 2024-09-11T01:35:27.967 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:35:27.967 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:35:27.967 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:35:27.967 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:27.968 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:35:27.968 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:35:27.968 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:35:27.968 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:35:27.968 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:35:27.968 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:35:27.968 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:35:27.968 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:35:27.968 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:35:27.968 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:35:27.968 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:35:27.968 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:35:27.968 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:35:27.969 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:35:27.969 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:35:27.969 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:27.969 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:35:27.969 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:35:27.969 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:35:27.969 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:35:27.969 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:35:27.969 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:35:27.969 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:35:27.969 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:35:27.969 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:35:27.969 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:35:27.970 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:35:27.970 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:35:27.970 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:35:27.970 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:27.970 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:27.970 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:35:27.970 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:35:27.970 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:35:27.970 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:35:27.970 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:35:27.970 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:35:27.970 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:35:27.970 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:35:27.970 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:35:27.971 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:35:27.971 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:35:27.971 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:35:27.971 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:35:27.971 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:35:27.971 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:35:27.971 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:35:27.971 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:35:27.971 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:35:27.971 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:35:27.971 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:35:27.971 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:35:27.971 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:35:27.971 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:35:27.972 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:35:27.972 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:35:27.972 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:35:27.972 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:35:27.972 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:35:27.972 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:27.972 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:35:27.972 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:27.972 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:35:27.972 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:27.972 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:35:27.972 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:35:27.972 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:35:27.972 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:35:27.973 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:27.973 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:35:27.973 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:35:27.973 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:27.973 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:27.973 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:27.973 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:35:27.973 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:35:27.973 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:27.973 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:27.973 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:27.973 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:27.973 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:35:27.974 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:35:27.974 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:35:27.974 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:35:27.974 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:35:27.974 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:27.974 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:27.974 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:35:27.974 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:35:27.974 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:35:27.974 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:35:27.974 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:27.974 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:35:27.974 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:35:27.974 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:27.975 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:27.975 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:27.975 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:35:27.975 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:35:27.975 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:27.975 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:27.975 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:27.975 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:27.975 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:35:27.975 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:35:27.975 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:35:27.975 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:35:27.975 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:35:27.975 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:27.976 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:27.976 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:35:27.976 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:35:27.976 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:35:27.976 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:35:27.976 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:27.976 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:35:27.976 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:35:27.976 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:27.976 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:27.976 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:27.976 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:35:27.976 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:35:27.976 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:27.977 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:27.977 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:27.977 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:27.977 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:35:27.977 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:35:27.977 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:35:27.977 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:35:27.977 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:35:27.977 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:27.977 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:27.977 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:27.977 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:35:27.977 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:35:27.977 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:27.978 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:35:27.978 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:35:27.978 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:35:27.978 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:27.978 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:27.978 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:35:27.978 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:27.978 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:35:27.978 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:35:27.978 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:35:27.978 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:27.978 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:27.978 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:35:27.978 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:27.979 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:35:27.979 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:35:27.979 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:35:27.979 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:27.979 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:27.979 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:35:27.979 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:27.979 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:35:27.979 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:35:27.979 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:35:27.979 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:27.979 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:27.979 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:27.979 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:35:27.979 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:35:27.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.951+0000 7ff02b3b5640 1 -- 172.21.15.12:0/1015595028 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff00803e240 msgr2=0x7ff008040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:27.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.951+0000 7ff02b3b5640 1 --2- 172.21.15.12:0/1015595028 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff00803e240 0x7ff008040700 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff014001b90 tx=0x7ff014001a20 comp rx=0 tx=0).stop 2024-09-11T01:35:27.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.951+0000 7ff02b3b5640 10 monclient: shutdown 2024-09-11T01:35:27.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.951+0000 7ff02b3b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:35:27.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.951+0000 7ff02b3b5640 1 -- 172.21.15.12:0/1015595028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024094440 msgr2=0x7ff0241d3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:27.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.951+0000 7ff02b3b5640 1 --2- 172.21.15.12:0/1015595028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024094440 0x7ff0241d3540 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff02406dc30 tx=0x7ff01c00c750 comp rx=0 tx=0).stop 2024-09-11T01:35:27.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.951+0000 7ff029952640 1 -- 172.21.15.12:0/1015595028 reap_dead start 2024-09-11T01:35:27.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.951+0000 7ff029952640 1 -- 172.21.15.12:0/1015595028 reap_dead start 2024-09-11T01:35:27.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.951+0000 7ff02b3b5640 1 -- 172.21.15.12:0/1015595028 shutdown_connections 2024-09-11T01:35:27.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.951+0000 7ff02b3b5640 1 -- 172.21.15.12:0/1015595028 >> 172.21.15.12:0/1015595028 conn(0x7ff0240a3e30 msgr2=0x7ff024070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:35:27.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.951+0000 7ff02b3b5640 1 -- 172.21.15.12:0/1015595028 shutdown_connections 2024-09-11T01:35:27.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:27.951+0000 7ff02b3b5640 1 -- 172.21.15.12:0/1015595028 wait complete. 2024-09-11T01:35:27.982 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:35:27.982 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T01:35:27.982 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:35:28.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.081+0000 7f809ca87640 10 monclient: get_monmap_and_config 2024-09-11T01:35:28.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.081+0000 7f809ca87640 10 monclient: build_initial_monmap 2024-09-11T01:35:28.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.081+0000 7f809ca87640 10 monclient: monmap: 2024-09-11T01:35:28.084 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:35:28.084 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:28.085 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:35:28.083595+0000 2024-09-11T01:35:28.085 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:35:28.083595+0000 2024-09-11T01:35:28.085 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:35:28.085 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:28.085 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:35:28.085 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:35:28.085 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:35:28.085 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:35:28.085 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:35:28.085 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:28.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.082+0000 7f809ca87640 1 Processor -- start 2024-09-11T01:35:28.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.082+0000 7f809ca87640 1 -- start start 2024-09-11T01:35:28.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.082+0000 7f809ca87640 10 monclient: init 2024-09-11T01:35:28.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f809ca87640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:28.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f809ca87640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-11T01:35:28.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f809ca87640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f80980a1380 addr v1:172.21.15.71:6789/0 2024-09-11T01:35:28.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f809ca87640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f80980a0370 0x7f809813ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:28.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f809ca87640 10 monclient(hunting): picked mon.noname-c con 0x7f80980a0370 addr v2:172.21.15.71:3300/0 2024-09-11T01:35:28.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f809ca87640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f80980a0c90 0x7f809813d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:28.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f809ca87640 10 monclient(hunting): picked mon.noname-a con 0x7f80980a0c90 addr v2:172.21.15.12:3300/0 2024-09-11T01:35:28.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f809ca87640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80980a21c0 con 0x7f80980a1380 2024-09-11T01:35:28.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f809ca87640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:28.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f809ca87640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f80980a24a0 con 0x7f80980a0c90 2024-09-11T01:35:28.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f809ca87640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:28.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f809ca87640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f80980a2780 con 0x7f80980a0370 2024-09-11T01:35:28.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f809ca87640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:28.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f809ca87640 10 monclient(hunting): authenticate will time out at 1134.526228s 2024-09-11T01:35:28.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f809659c640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f80980a1380 legacy=0x7f80980a1760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:35:28.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f8095d9b640 1 -- >> v2:172.21.15.71:3300/0 conn(0x7f80980a0370 msgr2=0x7f809813ae00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:35:28.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f8095d9b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f80980a0370 0x7f809813ae00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:35:28.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f8096d9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f80980a0c90 0x7f809813d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:28.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f8096d9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f80980a0c90 0x7f809813d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35792/0 (socket says 172.21.15.12:35792) 2024-09-11T01:35:28.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f8096d9d640 1 -- 172.21.15.12:0/3790110944 learned_addr learned my addr 172.21.15.12:0/3790110944 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:35:28.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f8096d9d640 10 monclient(hunting): get_auth_request con 0x7f80980a0c90 auth_method 0 2024-09-11T01:35:28.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f8096d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:28.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f8096d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:28.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f8096d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:28.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f8096d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:28.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f8096d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f8096d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f8096d9d640 10 monclient(hunting): handle_auth_done global_id 54264 payload 274 2024-09-11T01:35:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f8096d9d640 1 -- 172.21.15.12:0/3790110944 >> v2:172.21.15.71:3300/0 conn(0x7f80980a0370 msgr2=0x7f809813ae00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:35:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f8096d9d640 1 --2- 172.21.15.12:0/3790110944 >> v2:172.21.15.71:3300/0 conn(0x7f80980a0370 0x7f809813ae00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f8096d9d640 1 -- 172.21.15.12:0/3790110944 >> v1:172.21.15.71:6789/0 conn(0x7f80980a1380 legacy=0x7f80980a1760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:35:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f8096d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:35:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f8096d9d640 1 monclient: found mon.noname-a 2024-09-11T01:35:28.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f8096d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:28.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f8096d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:35:28.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.083+0000 7f8096d9d640 1 -- 172.21.15.12:0/3790110944 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f809809fb60 con 0x7f80980a0c90 2024-09-11T01:35:28.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.086+0000 7f8096d9d640 1 -- 172.21.15.12:0/3790110944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80980a0c90 msgr2=0x7f809813d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:28.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.086+0000 7f8096d9d640 1 --2- 172.21.15.12:0/3790110944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80980a0c90 0x7f809813d350 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f80980a4140 tx=0x7f809002a590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81467fd3f1cc7dc8 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:28.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.086+0000 7f809559a640 1 -- 172.21.15.12:0/3790110944 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8090036070 con 0x7f80980a0c90 2024-09-11T01:35:28.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.086+0000 7f809559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:28.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.086+0000 7f809559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:35:28.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.086+0000 7f809559a640 10 monclient: dump: 2024-09-11T01:35:28.102 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:28.102 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:28.102 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:28.102 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:28.102 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:28.102 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:28.102 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:28.103 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:28.103 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:28.103 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:28.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.086+0000 7f809559a640 10 monclient: _finish_auth 0 2024-09-11T01:35:28.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.086+0000 7f809559a640 10 monclient: _check_auth_tickets 2024-09-11T01:35:28.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.086+0000 7f809559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:28.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.086+0000 7f809559a640 1 -- 172.21.15.12:0/3790110944 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8090007800 con 0x7f80980a0c90 2024-09-11T01:35:28.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.086+0000 7f809ca87640 5 monclient: authenticate success, global_id 54264 2024-09-11T01:35:28.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.086+0000 7f809ca87640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:35:28.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.086+0000 7f809559a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:35:28.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.086+0000 7f809ca87640 10 monclient: get_monmap_and_config success 2024-09-11T01:35:28.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.086+0000 7f809559a640 1 -- 172.21.15.12:0/3790110944 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8090007bd0 con 0x7f80980a0c90 2024-09-11T01:35:28.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.086+0000 7f809559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:28.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.086+0000 7f809ca87640 4 set_mon_vals no callback set 2024-09-11T01:35:28.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.086+0000 7f809559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:35:28.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.086+0000 7f809559a640 10 monclient: dump: 2024-09-11T01:35:28.104 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:28.104 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:28.104 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:28.104 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:28.104 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:28.104 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:28.104 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:28.104 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:28.104 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:28.104 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:28.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.087+0000 7f809ca87640 10 monclient: shutdown 2024-09-11T01:35:28.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.087+0000 7f809ca87640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:35:28.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.087+0000 7f809ca87640 1 -- 172.21.15.12:0/3790110944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80980a0c90 msgr2=0x7f809813d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:28.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.087+0000 7f809ca87640 1 --2- 172.21.15.12:0/3790110944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80980a0c90 0x7f809813d350 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f80980a4140 tx=0x7f809002a590 comp rx=0 tx=0).stop 2024-09-11T01:35:28.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.087+0000 7f809ca87640 1 -- 172.21.15.12:0/3790110944 shutdown_connections 2024-09-11T01:35:28.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.087+0000 7f809ca87640 1 --2- 172.21.15.12:0/3790110944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80980a0c90 0x7f809813d350 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:28.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.087+0000 7f809ca87640 1 --2- 172.21.15.12:0/3790110944 >> v2:172.21.15.71:3300/0 conn(0x7f80980a0370 0x7f809813ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:28.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.087+0000 7f809ca87640 1 -- 172.21.15.12:0/3790110944 >> 172.21.15.12:0/3790110944 conn(0x7f80980a2f70 msgr2=0x7f80980a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:35:28.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.087+0000 7f809ca87640 1 -- 172.21.15.12:0/3790110944 shutdown_connections 2024-09-11T01:35:28.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.087+0000 7f809ca87640 1 -- 172.21.15.12:0/3790110944 wait complete. 2024-09-11T01:35:28.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.087+0000 7f809ca87640 10 monclient: build_initial_monmap 2024-09-11T01:35:28.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.087+0000 7f809ca87640 10 monclient: monmap: 2024-09-11T01:35:28.105 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:35:28.105 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:35:28.105 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:35:28.106 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:35:28.106 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:35:28.106 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:28.106 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:35:28.106 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:35:28.106 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:35:28.106 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:28.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.088+0000 7f809ca87640 1 Processor -- start 2024-09-11T01:35:28.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.088+0000 7f809ca87640 1 -- start start 2024-09-11T01:35:28.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.088+0000 7f809ca87640 10 monclient: init 2024-09-11T01:35:28.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.088+0000 7f809ca87640 10 monclient: _reopen_session rank -1 2024-09-11T01:35:28.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.088+0000 7f809ca87640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:35:28.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.088+0000 7f809ca87640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80980a0370 0x7f8098125a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:28.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.088+0000 7f809ca87640 10 monclient(hunting): picked mon.noname-a con 0x7f80980a0370 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:35:28.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.088+0000 7f809ca87640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f80980a0c90 0x7f809813c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:28.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.088+0000 7f809ca87640 10 monclient(hunting): picked mon.noname-c con 0x7f80980a0c90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:35:28.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.088+0000 7f809ca87640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80980a1380 0x7f80981caba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:28.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.088+0000 7f809ca87640 10 monclient(hunting): picked mon.noname-b con 0x7f80980a1380 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:35:28.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.088+0000 7f809ca87640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:28.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.088+0000 7f809ca87640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f809813da10 con 0x7f80980a0370 2024-09-11T01:35:28.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.088+0000 7f809ca87640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:28.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.088+0000 7f809ca87640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f809813d890 con 0x7f80980a1380 2024-09-11T01:35:28.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.088+0000 7f809ca87640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:28.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.088+0000 7f809ca87640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f809813db90 con 0x7f80980a0c90 2024-09-11T01:35:28.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.088+0000 7f809ca87640 10 monclient(hunting): _renew_subs 2024-09-11T01:35:28.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.088+0000 7f809ca87640 10 monclient(hunting): authenticate will time out at 1134.531814s 2024-09-11T01:35:28.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.088+0000 7f809659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80980a0370 0x7f8098125a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:28.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.088+0000 7f8095d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f80980a0c90 0x7f809813c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:28.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.088+0000 7f8096d9d640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80980a1380 msgr2=0x7f80981caba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:35:28.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.088+0000 7f8096d9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80980a1380 0x7f80981caba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:35:28.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.088+0000 7f809659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80980a0370 0x7f8098125a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35806/0 (socket says 172.21.15.12:35806) 2024-09-11T01:35:28.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.088+0000 7f8095d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f80980a0c90 0x7f809813c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45430/0 (socket says 172.21.15.12:45430) 2024-09-11T01:35:28.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.089+0000 7f809659c640 1 -- 172.21.15.12:0/1121431196 learned_addr learned my addr 172.21.15.12:0/1121431196 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:35:28.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.089+0000 7f809659c640 10 monclient(hunting): get_auth_request con 0x7f80980a0370 auth_method 0 2024-09-11T01:35:28.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.089+0000 7f809659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:28.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.089+0000 7f809659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:28.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.089+0000 7f809659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:28.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.089+0000 7f8095d9b640 10 monclient(hunting): get_auth_request con 0x7f80980a0c90 auth_method 0 2024-09-11T01:35:28.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.089+0000 7f8095d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:28.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.089+0000 7f8095d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:28.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.089+0000 7f8095d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:28.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.089+0000 7f809659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:28.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.089+0000 7f809659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:28.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.089+0000 7f809659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:28.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.089+0000 7f8095d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:28.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.089+0000 7f8095d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:28.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.089+0000 7f8095d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:28.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.089+0000 7f809659c640 10 monclient(hunting): handle_auth_done global_id 54267 payload 824 2024-09-11T01:35:28.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.089+0000 7f809659c640 1 -- 172.21.15.12:0/1121431196 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f80980a0c90 msgr2=0x7f809813c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:28.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.089+0000 7f809659c640 1 --2- 172.21.15.12:0/1121431196 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f80980a0c90 0x7f809813c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:28.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.089+0000 7f809659c640 1 -- 172.21.15.12:0/1121431196 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80980a1380 msgr2=0x7f80981caba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:35:28.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.089+0000 7f809659c640 1 --2- 172.21.15.12:0/1121431196 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f80980a1380 0x7f80981caba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:28.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.089+0000 7f809659c640 10 monclient: _finish_hunting 0 2024-09-11T01:35:28.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.089+0000 7f809659c640 1 monclient: found mon.noname-a 2024-09-11T01:35:28.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.089+0000 7f809659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:35:28.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.089+0000 7f809659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:35:28.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.089+0000 7f809659c640 1 -- 172.21.15.12:0/1121431196 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8098124f20 con 0x7f80980a0370 2024-09-11T01:35:28.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.091+0000 7f809659c640 1 -- 172.21.15.12:0/1121431196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80980a0370 msgr2=0x7f8098125a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:28.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.091+0000 7f809659c640 1 --2- 172.21.15.12:0/1121431196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80980a0370 0x7f8098125a80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f808800b550 tx=0x7f808800ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:28.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.091+0000 7f8086ffd640 1 -- 172.21.15.12:0/1121431196 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8088013020 con 0x7f80980a0370 2024-09-11T01:35:28.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.091+0000 7f8086ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:28.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.091+0000 7f8086ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:35:28.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.091+0000 7f8086ffd640 10 monclient: dump: 2024-09-11T01:35:28.111 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:28.111 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:28.112 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:28.112 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:28.112 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:28.112 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:28.112 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:28.112 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:28.112 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:28.112 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:28.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.091+0000 7f8086ffd640 10 monclient: _finish_auth 0 2024-09-11T01:35:28.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.091+0000 7f8086ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:35:28.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.091+0000 7f8086ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:35:28.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.091+0000 7f8086ffd640 1 -- 172.21.15.12:0/1121431196 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8088005070 con 0x7f80980a0370 2024-09-11T01:35:28.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.091+0000 7f809ca87640 5 monclient: authenticate success, global_id 54267 2024-09-11T01:35:28.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.091+0000 7f809ca87640 10 monclient: _renew_subs 2024-09-11T01:35:28.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.091+0000 7f809ca87640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:35:28.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.091+0000 7f809ca87640 1 -- 172.21.15.12:0/1121431196 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f809813d590 con 0x7f80980a0370 2024-09-11T01:35:28.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.091+0000 7f8086ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:35:28.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.091+0000 7f8086ffd640 1 -- 172.21.15.12:0/1121431196 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8088005480 con 0x7f80980a0370 2024-09-11T01:35:28.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.091+0000 7f8086ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:28.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.091+0000 7f8086ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:35:28.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.091+0000 7f8086ffd640 10 monclient: dump: 2024-09-11T01:35:28.113 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:28.113 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:28.113 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:28.113 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:28.113 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:28.113 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:28.113 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:28.114 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.091+0000 7f809ca87640 10 monclient: _renew_subs 2024-09-11T01:35:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.091+0000 7f8094d99640 4 set_mon_vals no callback set 2024-09-11T01:35:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.091+0000 7f809ca87640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:35:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.091+0000 7f809ca87640 1 -- 172.21.15.12:0/1121431196 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8098123dc0 con 0x7f80980a0370 2024-09-11T01:35:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.092+0000 7f809ca87640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.092+0000 7f809ca87640 1 --2- 172.21.15.12:0/1121431196 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f809813ce30 0x7f809802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.092+0000 7f809ca87640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.092+0000 7f809ca87640 1 -- 172.21.15.12:0/1121431196 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f80980a1760 con 0x7f809813ce30 2024-09-11T01:35:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.092+0000 7f8086ffd640 1 -- 172.21.15.12:0/1121431196 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f8088005620 con 0x7f80980a0370 2024-09-11T01:35:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.092+0000 7f8095d9b640 1 --2- 172.21.15.12:0/1121431196 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f809813ce30 0x7f809802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:28.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.092+0000 7f8095d9b640 10 monclient: get_auth_request con 0x7f809813ce30 auth_method 0 2024-09-11T01:35:28.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.092+0000 7f8095d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:28.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.092+0000 7f8095d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:28.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.092+0000 7f8095d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:28.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.092+0000 7f8095d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:28.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.092+0000 7f8095d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:28.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.092+0000 7f8095d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:28.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.093+0000 7f8095d9b640 10 monclient(hunting): handle_auth_done global_id 64166 payload 824 2024-09-11T01:35:28.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.093+0000 7f8095d9b640 1 -- 172.21.15.12:0/1121431196 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8098120000 con 0x7f809813ce30 2024-09-11T01:35:28.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.093+0000 7f8086ffd640 1 --2- 172.21.15.12:0/1121431196 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f807803e1c0 0x7f8078040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:28.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.093+0000 7f8086ffd640 1 -- 172.21.15.12:0/1121431196 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(202..202 src has 1..202) ==== 4755+0+0 (secure 0 0 0) 0x7f8088052da0 con 0x7f80980a0370 2024-09-11T01:35:28.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.093+0000 7f8096d9d640 1 --2- 172.21.15.12:0/1121431196 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f807803e1c0 0x7f8078040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:28.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.093+0000 7f8096d9d640 10 monclient: get_auth_request con 0x7f807803e1c0 auth_method 0 2024-09-11T01:35:28.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.095+0000 7f8095d9b640 1 -- 172.21.15.12:0/1121431196 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f809813ce30 msgr2=0x7f809802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:28.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.095+0000 7f8095d9b640 1 --2- 172.21.15.12:0/1121431196 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f809813ce30 0x7f809802d440 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f809806ff10 tx=0x7f808c009b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:28.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.095+0000 7f8086ffd640 1 -- 172.21.15.12:0/1121431196 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808c016070 con 0x7f809813ce30 2024-09-11T01:35:28.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.095+0000 7f8086ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:28.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.095+0000 7f8086ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:35:28.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.095+0000 7f8086ffd640 10 monclient: dump: 2024-09-11T01:35:28.116 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:28.116 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:28.116 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:28.116 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:28.116 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:28.116 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:28.116 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:28.116 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:28.117 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:28.117 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:28.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.095+0000 7f8086ffd640 1 -- 172.21.15.12:0/1121431196 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f808c0128f0 con 0x7f809813ce30 2024-09-11T01:35:28.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.095+0000 7f8086ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:35:28.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.095+0000 7f8086ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:35:28.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.095+0000 7f8086ffd640 1 -- 172.21.15.12:0/1121431196 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f809813ce30 msgr2=0x7f809802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:28.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.095+0000 7f8086ffd640 1 --2- 172.21.15.12:0/1121431196 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f809813ce30 0x7f809802d440 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f809806ff10 tx=0x7f808c009b00 comp rx=0 tx=0).stop 2024-09-11T01:35:28.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.095+0000 7f8086ffd640 1 -- 172.21.15.12:0/1121431196 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f809813ce30 msgr2=0x7f809802d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:35:28.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.095+0000 7f8086ffd640 1 --2- 172.21.15.12:0/1121431196 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f809813ce30 0x7f809802d440 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f809806ff10 tx=0x7f808c009b00 comp rx=0 tx=0).stop 2024-09-11T01:35:28.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.096+0000 7f8096d9d640 1 --2- 172.21.15.12:0/1121431196 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f807803e1c0 0x7f8078040680 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f80980a72b0 tx=0x7f8090009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:28.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.101+0000 7f809ca87640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:35:28.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.101+0000 7f809ca87640 1 --2- 172.21.15.12:0/1121431196 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f809802d980 0x7f809802dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:35:28.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.101+0000 7f809ca87640 10 monclient(hunting): start opening mon connection 2024-09-11T01:35:28.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.101+0000 7f809ca87640 1 -- 172.21.15.12:0/1121431196 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8098126d20 con 0x7f809802d980 2024-09-11T01:35:28.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.101+0000 7f8095d9b640 1 --2- 172.21.15.12:0/1121431196 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f809802d980 0x7f809802dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:35:28.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.101+0000 7f8095d9b640 10 monclient: get_auth_request con 0x7f809802d980 auth_method 0 2024-09-11T01:35:28.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.101+0000 7f8095d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:35:28.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.101+0000 7f8095d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:35:28.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.101+0000 7f8095d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:35:28.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.102+0000 7f8095d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:35:28.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.102+0000 7f8095d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:35:28.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.102+0000 7f8095d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:35:28.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.102+0000 7f8095d9b640 10 monclient(hunting): handle_auth_done global_id 64169 payload 824 2024-09-11T01:35:28.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.102+0000 7f8095d9b640 1 -- 172.21.15.12:0/1121431196 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f809806d480 con 0x7f809802d980 2024-09-11T01:35:28.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.104+0000 7f8095d9b640 1 -- 172.21.15.12:0/1121431196 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f809802d980 msgr2=0x7f809802dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:35:28.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.104+0000 7f8095d9b640 1 --2- 172.21.15.12:0/1121431196 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f809802d980 0x7f809802dd60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f808c003fd0 tx=0x7f808c0059d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:35:28.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.104+0000 7f8086ffd640 1 -- 172.21.15.12:0/1121431196 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808c016040 con 0x7f809802d980 2024-09-11T01:35:28.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.104+0000 7f8086ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:35:28.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.104+0000 7f8086ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:35:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.104+0000 7f8086ffd640 10 monclient: dump: 2024-09-11T01:35:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:35:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:35:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:35:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:35:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:35:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:35:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:35:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:35:28.121 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:35:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.104+0000 7f8086ffd640 1 -- 172.21.15.12:0/1121431196 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4696 (secure 0 0 0) 0x7f808c01b450 con 0x7f809802d980 2024-09-11T01:35:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.104+0000 7f8086ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:35:28.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.104+0000 7f8086ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:35:28.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.104+0000 7f8086ffd640 1 -- 172.21.15.12:0/1121431196 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f809802d980 msgr2=0x7f809802dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:28.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.104+0000 7f8086ffd640 1 --2- 172.21.15.12:0/1121431196 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f809802d980 0x7f809802dd60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f808c003fd0 tx=0x7f808c0059d0 comp rx=0 tx=0).stop 2024-09-11T01:35:28.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.104+0000 7f8086ffd640 1 -- 172.21.15.12:0/1121431196 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f809802d980 msgr2=0x7f809802dd60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:35:28.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.104+0000 7f8086ffd640 1 --2- 172.21.15.12:0/1121431196 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f809802d980 0x7f809802dd60 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f808c003fd0 tx=0x7f808c0059d0 comp rx=0 tx=0).stop 2024-09-11T01:35:28.122 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:35:28.122 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:35:28.122 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:35:28.122 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:35:28.122 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 68, 2024-09-11T01:35:28.122 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 46197, 2024-09-11T01:35:28.123 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:35:28.123 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:35:28.123 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:35:28.123 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:28.123 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:35:28.123 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:35:28.123 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:35:28.123 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:35:28.123 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:35:28.123 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:35:28.123 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:35:28.123 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:35:28.123 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:35:28.123 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:35:28.124 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:35:28.124 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:35:28.124 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:35:28.124 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:35:28.124 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:35:28.124 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:28.124 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:35:28.124 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:35:28.124 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:35:28.124 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:35:28.124 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:35:28.124 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:35:28.124 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:35:28.125 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:35:28.125 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:35:28.125 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:35:28.125 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:35:28.125 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:35:28.125 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:35:28.125 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:28.125 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:28.125 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:35:28.125 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:35:28.125 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:35:28.125 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:35:28.125 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:35:28.126 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:35:28.126 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:35:28.126 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:35:28.126 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:35:28.126 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:35:28.126 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:35:28.126 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:35:28.126 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:35:28.126 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:35:28.126 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:35:28.126 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:35:28.126 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:35:28.126 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:35:28.127 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:35:28.127 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:35:28.127 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:35:28.127 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:35:28.127 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:35:28.127 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:35:28.127 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:35:28.127 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:35:28.127 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:35:28.127 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:35:28.127 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:28.127 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:35:28.128 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:28.128 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:35:28.128 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:28.128 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:35:28.128 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:35:28.128 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:35:28.128 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:35:28.128 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:28.128 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:35:28.128 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:35:28.128 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:28.128 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:28.128 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:28.128 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:35:28.129 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:35:28.129 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:28.129 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:28.129 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:28.129 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:28.129 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:35:28.129 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:35:28.129 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:35:28.129 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:35:28.129 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:35:28.129 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:28.129 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:28.129 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:35:28.130 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:35:28.130 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:35:28.130 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:35:28.130 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:28.130 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:35:28.130 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:35:28.130 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:28.130 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:28.130 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:28.130 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:35:28.130 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:35:28.131 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:28.131 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:28.131 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:28.131 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:28.131 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:35:28.131 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:35:28.131 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:35:28.131 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:35:28.131 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:35:28.132 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:28.132 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:28.132 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:35:28.132 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:35:28.132 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:35:28.132 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:35:28.132 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:28.132 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:35:28.132 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:35:28.132 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:28.132 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:28.132 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:28.133 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:35:28.133 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:35:28.133 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:35:28.133 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:28.133 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:28.133 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:28.133 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:35:28.133 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:35:28.133 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:35:28.133 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:35:28.133 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:35:28.133 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:28.133 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:28.134 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:28.134 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:35:28.134 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:35:28.134 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:28.134 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:35:28.134 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:35:28.134 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:35:28.134 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:28.134 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:28.134 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:35:28.134 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:28.134 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:35:28.134 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:35:28.134 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:35:28.135 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:28.135 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:35:28.135 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:35:28.135 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:35:28.135 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:35:28.135 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:35:28.135 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:35:28.135 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:35:28.135 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:35:28.135 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:35:28.135 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:35:28.135 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:35:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.105+0000 7f809ca87640 1 -- 172.21.15.12:0/1121431196 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f807803e1c0 msgr2=0x7f8078040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.105+0000 7f809ca87640 1 --2- 172.21.15.12:0/1121431196 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f807803e1c0 0x7f8078040680 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f80980a72b0 tx=0x7f8090009000 comp rx=0 tx=0).stop 2024-09-11T01:35:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.105+0000 7f809ca87640 10 monclient: shutdown 2024-09-11T01:35:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.105+0000 7f809ca87640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:35:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.105+0000 7f809ca87640 1 -- 172.21.15.12:0/1121431196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80980a0370 msgr2=0x7f8098125a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:35:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.105+0000 7f809ca87640 1 --2- 172.21.15.12:0/1121431196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80980a0370 0x7f8098125a80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f808800b550 tx=0x7f808800ba10 comp rx=0 tx=0).stop 2024-09-11T01:35:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.105+0000 7f8096d9d640 1 -- 172.21.15.12:0/1121431196 reap_dead start 2024-09-11T01:35:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.105+0000 7f809ca87640 1 -- 172.21.15.12:0/1121431196 shutdown_connections 2024-09-11T01:35:28.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.105+0000 7f809ca87640 1 --2- 172.21.15.12:0/1121431196 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80980a0370 0x7f8098125a80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:35:28.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.105+0000 7f809ca87640 1 -- 172.21.15.12:0/1121431196 >> 172.21.15.12:0/1121431196 conn(0x7f80980a2f70 msgr2=0x7f80980713c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:35:28.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.105+0000 7f809ca87640 1 -- 172.21.15.12:0/1121431196 shutdown_connections 2024-09-11T01:35:28.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:35:28.105+0000 7f809ca87640 1 -- 172.21.15.12:0/1121431196 wait complete. 2024-09-11T01:35:28.138 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:35:28.138 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:35:28.530 INFO:tasks.rados.rados.0.smithi071.stdout:861: done (3 left) 2024-09-11T01:35:28.530 INFO:tasks.rados.rados.0.smithi071.stdout:862: done (2 left) 2024-09-11T01:35:28.530 INFO:tasks.rados.rados.0.smithi071.stdout:863: done (1 left) 2024-09-11T01:35:28.530 INFO:tasks.rados.rados.0.smithi071.stdout:864: done (0 left) 2024-09-11T01:35:28.530 INFO:tasks.rados.rados.0.smithi071.stdout:865: delete oid 40 current snap is 93 2024-09-11T01:35:28.534 INFO:tasks.rados.rados.0.smithi071.stdout:865: done (0 left) 2024-09-11T01:35:28.534 INFO:tasks.rados.rados.0.smithi071.stdout:866: read oid 21 snap -1 2024-09-11T01:35:28.534 INFO:tasks.rados.rados.0.smithi071.stdout:866: expect (ObjNum 273 snap 72 seq_num 273) 2024-09-11T01:35:28.534 INFO:tasks.rados.rados.0.smithi071.stdout:867: rollback oid 31 current snap is 93 2024-09-11T01:35:28.534 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 31 to 86 2024-09-11T01:35:28.535 INFO:tasks.rados.rados.0.smithi071.stdout:868: snap_create 2024-09-11T01:35:28.540 INFO:tasks.rados.rados.0.smithi071.stdout:867: finishing rollback tid 0 to smithi07134777-31 2024-09-11T01:35:28.541 INFO:tasks.rados.rados.0.smithi071.stdout:867: finishing rollback tid 1 to smithi07134777-31 2024-09-11T01:35:28.541 INFO:tasks.rados.rados.0.smithi071.stdout:867: finishing rollback tid 2 to smithi07134777-31 2024-09-11T01:35:28.541 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 309 (ObjNum 201 snap 46 seq_num 201) dirty exists 2024-09-11T01:35:28.543 INFO:tasks.rados.rados.0.smithi071.stdout:866: expect (ObjNum 273 snap 72 seq_num 273) 2024-09-11T01:35:28.617 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:35:29.545 INFO:tasks.rados.rados.0.smithi071.stdout:866: done (2 left) 2024-09-11T01:35:29.545 INFO:tasks.rados.rados.0.smithi071.stdout:867: done (1 left) 2024-09-11T01:35:29.545 INFO:tasks.rados.rados.0.smithi071.stdout:868: done (0 left) 2024-09-11T01:35:29.545 INFO:tasks.rados.rados.0.smithi071.stdout:869: copy_from oid 21 from oid 44 current snap is 94 2024-09-11T01:35:29.545 INFO:tasks.rados.rados.0.smithi071.stdout:870: write oid 48 current snap is 94 2024-09-11T01:35:29.545 INFO:tasks.rados.rados.0.smithi071.stdout:870: seq_num 329 ranges {729503=560100,2019164=716432,3335341=194162} 2024-09-11T01:35:29.551 INFO:tasks.rados.rados.0.smithi071.stdout:870: writing smithi07134777-48 from 729503 to 1289603 tid 1 2024-09-11T01:35:29.559 INFO:tasks.rados.rados.0.smithi071.stdout:870: writing smithi07134777-48 from 2019164 to 2735596 tid 2 2024-09-11T01:35:29.564 INFO:tasks.rados.rados.0.smithi071.stdout:870: writing smithi07134777-48 from 3335341 to 3529503 tid 3 2024-09-11T01:35:29.564 INFO:tasks.rados.rados.0.smithi071.stdout:871: write oid 49 current snap is 94 2024-09-11T01:35:29.564 INFO:tasks.rados.rados.0.smithi071.stdout:871: seq_num 330 ranges {746310=635349,2077646=576443,3362764=183546} 2024-09-11T01:35:29.571 INFO:tasks.rados.rados.0.smithi071.stdout:871: writing smithi07134777-49 from 746310 to 1381659 tid 1 2024-09-11T01:35:29.578 INFO:tasks.rados.rados.0.smithi071.stdout:871: writing smithi07134777-49 from 2077646 to 2654089 tid 2 2024-09-11T01:35:29.584 INFO:tasks.rados.rados.0.smithi071.stdout:871: writing smithi07134777-49 from 3362764 to 3546310 tid 3 2024-09-11T01:35:29.584 INFO:tasks.rados.rados.0.smithi071.stdout:872: snap_create 2024-09-11T01:35:29.584 INFO:tasks.rados.rados.0.smithi071.stdout:869: finishing copy_from to smithi07134777-21 2024-09-11T01:35:29.584 INFO:tasks.rados.rados.0.smithi071.stdout:869: got expected ENOENT (src dne) 2024-09-11T01:35:29.584 INFO:tasks.rados.rados.0.smithi071.stdout:869: finishing copy_from racing read to smithi07134777-21 2024-09-11T01:35:29.584 INFO:tasks.rados.rados.0.smithi071.stdout:870: finishing write tid 1 to smithi07134777-48 2024-09-11T01:35:29.584 INFO:tasks.rados.rados.0.smithi071.stdout:870: finishing write tid 2 to smithi07134777-48 2024-09-11T01:35:29.584 INFO:tasks.rados.rados.0.smithi071.stdout:870: finishing write tid 3 to smithi07134777-48 2024-09-11T01:35:29.584 INFO:tasks.rados.rados.0.smithi071.stdout:870: finishing write tid 4 to smithi07134777-48 2024-09-11T01:35:29.585 INFO:tasks.rados.rados.0.smithi071.stdout:870: finishing write tid 5 to smithi07134777-48 2024-09-11T01:35:29.585 INFO:tasks.rados.rados.0.smithi071.stdout:870: oid 48 updating version 0 to 393 2024-09-11T01:35:29.585 INFO:tasks.rados.rados.0.smithi071.stdout:870: oid 48 updating version 393 to 394 2024-09-11T01:35:29.585 INFO:tasks.rados.rados.0.smithi071.stdout:870: oid 48 version 394 is already newer than 392 2024-09-11T01:35:29.585 INFO:tasks.rados.rados.0.smithi071.stdout:870: oid 48 version 394 is already newer than 391 2024-09-11T01:35:29.585 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 394 (ObjNum 329 snap 94 seq_num 329) dirty exists 2024-09-11T01:35:29.585 INFO:tasks.rados.rados.0.smithi071.stdout:870: left oid 48 (ObjNum 329 snap 94 seq_num 329) 2024-09-11T01:35:29.585 INFO:tasks.rados.rados.0.smithi071.stdout:871: finishing write tid 1 to smithi07134777-49 2024-09-11T01:35:29.585 INFO:tasks.rados.rados.0.smithi071.stdout:871: finishing write tid 2 to smithi07134777-49 2024-09-11T01:35:29.586 INFO:tasks.rados.rados.0.smithi071.stdout:871: finishing write tid 3 to smithi07134777-49 2024-09-11T01:35:29.586 INFO:tasks.rados.rados.0.smithi071.stdout:871: finishing write tid 4 to smithi07134777-49 2024-09-11T01:35:29.587 INFO:tasks.rados.rados.0.smithi071.stdout:871: finishing write tid 5 to smithi07134777-49 2024-09-11T01:35:29.587 INFO:tasks.rados.rados.0.smithi071.stdout:871: oid 49 updating version 0 to 219 2024-09-11T01:35:29.587 INFO:tasks.rados.rados.0.smithi071.stdout:871: oid 49 version 219 is already newer than 218 2024-09-11T01:35:29.587 INFO:tasks.rados.rados.0.smithi071.stdout:871: oid 49 version 219 is already newer than 216 2024-09-11T01:35:29.587 INFO:tasks.rados.rados.0.smithi071.stdout:871: oid 49 version 219 is already newer than 217 2024-09-11T01:35:29.587 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 219 (ObjNum 330 snap 94 seq_num 330) dirty exists 2024-09-11T01:35:29.587 INFO:tasks.rados.rados.0.smithi071.stdout:871: left oid 49 (ObjNum 330 snap 94 seq_num 330) 2024-09-11T01:35:30.543 INFO:tasks.rados.rados.0.smithi071.stdout:869: done (3 left) 2024-09-11T01:35:30.543 INFO:tasks.rados.rados.0.smithi071.stdout:870: done (2 left) 2024-09-11T01:35:30.543 INFO:tasks.rados.rados.0.smithi071.stdout:871: done (1 left) 2024-09-11T01:35:30.543 INFO:tasks.rados.rados.0.smithi071.stdout:872: done (0 left) 2024-09-11T01:35:30.543 INFO:tasks.rados.rados.0.smithi071.stdout:873: write (excl) oid 28 current snap is 95 2024-09-11T01:35:30.543 INFO:tasks.rados.rados.0.smithi071.stdout:873: seq_num 331 ranges {763117=710598,2136128=720101,3390187=172930} 2024-09-11T01:35:30.550 INFO:tasks.rados.rados.0.smithi071.stdout:873: writing smithi07134777-28 from 763117 to 1473715 tid 1 2024-09-11T01:35:30.558 INFO:tasks.rados.rados.0.smithi071.stdout:873: writing smithi07134777-28 from 2136128 to 2856229 tid 2 2024-09-11T01:35:30.562 INFO:tasks.rados.rados.0.smithi071.stdout:873: writing smithi07134777-28 from 3390187 to 3563117 tid 3 2024-09-11T01:35:30.562 INFO:tasks.rados.rados.0.smithi071.stdout:874: snap_create 2024-09-11T01:35:30.562 INFO:tasks.rados.rados.0.smithi071.stdout:873: finishing write tid 1 to smithi07134777-28 2024-09-11T01:35:30.563 INFO:tasks.rados.rados.0.smithi071.stdout:873: finishing write tid 2 to smithi07134777-28 2024-09-11T01:35:30.565 INFO:tasks.rados.rados.0.smithi071.stdout:873: finishing write tid 3 to smithi07134777-28 2024-09-11T01:35:30.565 INFO:tasks.rados.rados.0.smithi071.stdout:873: finishing write tid 4 to smithi07134777-28 2024-09-11T01:35:30.565 INFO:tasks.rados.rados.0.smithi071.stdout:873: finishing write tid 5 to smithi07134777-28 2024-09-11T01:35:30.566 INFO:tasks.rados.rados.0.smithi071.stdout:873: oid 28 updating version 0 to 288 2024-09-11T01:35:30.566 INFO:tasks.rados.rados.0.smithi071.stdout:873: oid 28 version 288 is already newer than 287 2024-09-11T01:35:30.566 INFO:tasks.rados.rados.0.smithi071.stdout:873: oid 28 version 288 is already newer than 286 2024-09-11T01:35:30.566 INFO:tasks.rados.rados.0.smithi071.stdout:873: oid 28 version 288 is already newer than 285 2024-09-11T01:35:30.566 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 288 (ObjNum 331 snap 95 seq_num 331) dirty exists 2024-09-11T01:35:30.566 INFO:tasks.rados.rados.0.smithi071.stdout:873: left oid 28 (ObjNum 331 snap 95 seq_num 331) 2024-09-11T01:35:31.552 INFO:tasks.rados.rados.0.smithi071.stdout:873: done (1 left) 2024-09-11T01:35:31.552 INFO:tasks.rados.rados.0.smithi071.stdout:874: done (0 left) 2024-09-11T01:35:31.552 INFO:tasks.rados.rados.0.smithi071.stdout:875: write oid 33 current snap is 96 2024-09-11T01:35:31.552 INFO:tasks.rados.rados.0.smithi071.stdout:875: seq_num 332 ranges {779924=785847,2078257=580112,3301257=278667} 2024-09-11T01:35:31.560 INFO:tasks.rados.rados.0.smithi071.stdout:875: writing smithi07134777-33 from 779924 to 1565771 tid 1 2024-09-11T01:35:31.566 INFO:tasks.rados.rados.0.smithi071.stdout:875: writing smithi07134777-33 from 2078257 to 2658369 tid 2 2024-09-11T01:35:31.571 INFO:tasks.rados.rados.0.smithi071.stdout:875: writing smithi07134777-33 from 3301257 to 3579924 tid 3 2024-09-11T01:35:31.571 INFO:tasks.rados.rados.0.smithi071.stdout:876: delete oid 44 current snap is 96 2024-09-11T01:35:31.571 INFO:tasks.rados.rados.0.smithi071.stdout:875: finishing write tid 1 to smithi07134777-33 2024-09-11T01:35:31.573 INFO:tasks.rados.rados.0.smithi071.stdout:875: finishing write tid 2 to smithi07134777-33 2024-09-11T01:35:31.574 INFO:tasks.rados.rados.0.smithi071.stdout:876: done (1 left) 2024-09-11T01:35:31.574 INFO:tasks.rados.rados.0.smithi071.stdout:877: snap_remove snap 94 2024-09-11T01:35:32.524 INFO:tasks.rados.rados.0.smithi071.stdout:877: done (1 left) 2024-09-11T01:35:32.525 INFO:tasks.rados.rados.0.smithi071.stdout:878: snap_remove snap 93 2024-09-11T01:35:33.521 INFO:tasks.rados.rados.0.smithi071.stdout:878: done (1 left) 2024-09-11T01:35:33.522 INFO:tasks.rados.rados.0.smithi071.stdout:879: copy_from oid 50 from oid 16 current snap is 96 2024-09-11T01:35:33.522 INFO:tasks.rados.rados.0.smithi071.stdout:875: finishing write tid 3 to smithi07134777-33 2024-09-11T01:35:33.522 INFO:tasks.rados.rados.0.smithi071.stdout:875: finishing write tid 4 to smithi07134777-33 2024-09-11T01:35:33.522 INFO:tasks.rados.rados.0.smithi071.stdout:875: finishing write tid 5 to smithi07134777-33 2024-09-11T01:35:33.522 INFO:tasks.rados.rados.0.smithi071.stdout:875: oid 33 updating version 0 to 306 2024-09-11T01:35:33.522 INFO:tasks.rados.rados.0.smithi071.stdout:875: oid 33 version 306 is already newer than 304 2024-09-11T01:35:33.522 INFO:tasks.rados.rados.0.smithi071.stdout:875: oid 33 version 306 is already newer than 305 2024-09-11T01:35:33.522 INFO:tasks.rados.rados.0.smithi071.stdout:875: oid 33 updating version 306 to 307 2024-09-11T01:35:33.522 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 307 (ObjNum 332 snap 96 seq_num 332) dirty exists 2024-09-11T01:35:33.522 INFO:tasks.rados.rados.0.smithi071.stdout:875: left oid 33 (ObjNum 332 snap 96 seq_num 332) 2024-09-11T01:35:33.522 INFO:tasks.rados.rados.0.smithi071.stdout:875: done (1 left) 2024-09-11T01:35:33.522 INFO:tasks.rados.rados.0.smithi071.stdout:880: read oid 3 snap -1 2024-09-11T01:35:33.522 INFO:tasks.rados.rados.0.smithi071.stdout:880: expect (ObjNum 200 snap 46 seq_num 200) 2024-09-11T01:35:33.523 INFO:tasks.rados.rados.0.smithi071.stdout:881: write oid 14 current snap is 96 2024-09-11T01:35:33.523 INFO:tasks.rados.rados.0.smithi071.stdout:881: seq_num 334 ranges {413538=536345,1395221=583781,2556103=658617} 2024-09-11T01:35:33.526 INFO:tasks.rados.rados.0.smithi071.stdout:881: writing smithi07134777-14 from 413538 to 949883 tid 1 2024-09-11T01:35:33.532 INFO:tasks.rados.rados.0.smithi071.stdout:881: writing smithi07134777-14 from 1395221 to 1979002 tid 2 2024-09-11T01:35:33.540 INFO:tasks.rados.rados.0.smithi071.stdout:881: writing smithi07134777-14 from 2556103 to 3214720 tid 3 2024-09-11T01:35:33.540 INFO:tasks.rados.rados.0.smithi071.stdout:882: copy_from oid 20 from oid 32 current snap is 96 2024-09-11T01:35:33.540 INFO:tasks.rados.rados.0.smithi071.stdout:880: expect (ObjNum 200 snap 46 seq_num 200) 2024-09-11T01:35:33.615 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1080339 2024-09-11T01:35:33.692 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1534501 2024-09-11T01:35:33.736 INFO:tasks.rados.rados.0.smithi071.stdout:879: finishing copy_from racing read to smithi07134777-50 2024-09-11T01:35:33.736 INFO:tasks.rados.rados.0.smithi071.stdout:881: finishing write tid 1 to smithi07134777-14 2024-09-11T01:35:33.736 INFO:tasks.rados.rados.0.smithi071.stdout:882: finishing copy_from to smithi07134777-20 2024-09-11T01:35:33.736 INFO:tasks.rados.rados.0.smithi071.stdout:882: got expected ENOENT (src dne) 2024-09-11T01:35:33.736 INFO:tasks.rados.rados.0.smithi071.stdout:882: finishing copy_from racing read to smithi07134777-20 2024-09-11T01:35:33.736 INFO:tasks.rados.rados.0.smithi071.stdout:879: finishing copy_from to smithi07134777-50 2024-09-11T01:35:33.736 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 129 (ObjNum 226 snap 55 seq_num 226) dirty exists 2024-09-11T01:35:33.736 INFO:tasks.rados.rados.0.smithi071.stdout:881: finishing write tid 2 to smithi07134777-14 2024-09-11T01:35:33.736 INFO:tasks.rados.rados.0.smithi071.stdout:881: finishing write tid 3 to smithi07134777-14 2024-09-11T01:35:33.737 INFO:tasks.rados.rados.0.smithi071.stdout:881: finishing write tid 4 to smithi07134777-14 2024-09-11T01:35:33.737 INFO:tasks.rados.rados.0.smithi071.stdout:881: finishing write tid 5 to smithi07134777-14 2024-09-11T01:35:33.737 INFO:tasks.rados.rados.0.smithi071.stdout:881: oid 14 updating version 0 to 403 2024-09-11T01:35:33.737 INFO:tasks.rados.rados.0.smithi071.stdout:881: oid 14 version 403 is already newer than 401 2024-09-11T01:35:33.737 INFO:tasks.rados.rados.0.smithi071.stdout:881: oid 14 version 403 is already newer than 402 2024-09-11T01:35:33.737 INFO:tasks.rados.rados.0.smithi071.stdout:881: oid 14 version 403 is already newer than 398 2024-09-11T01:35:33.737 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 403 (ObjNum 334 snap 96 seq_num 334) dirty exists 2024-09-11T01:35:33.737 INFO:tasks.rados.rados.0.smithi071.stdout:881: left oid 14 (ObjNum 334 snap 96 seq_num 334) 2024-09-11T01:35:33.737 INFO:tasks.rados.rados.0.smithi071.stdout:879: done (3 left) 2024-09-11T01:35:33.737 INFO:tasks.rados.rados.0.smithi071.stdout:880: done (2 left) 2024-09-11T01:35:33.737 INFO:tasks.rados.rados.0.smithi071.stdout:881: done (1 left) 2024-09-11T01:35:33.737 INFO:tasks.rados.rados.0.smithi071.stdout:882: done (0 left) 2024-09-11T01:35:33.737 INFO:tasks.rados.rados.0.smithi071.stdout:883: read oid 21 snap 91 2024-09-11T01:35:33.738 INFO:tasks.rados.rados.0.smithi071.stdout:883: expect (ObjNum 273 snap 72 seq_num 273) 2024-09-11T01:35:33.738 INFO:tasks.rados.rados.0.smithi071.stdout:884: rollback oid 27 current snap is 96 2024-09-11T01:35:33.738 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 27 to 82 2024-09-11T01:35:33.738 INFO:tasks.rados.rados.0.smithi071.stdout:885: copy_from oid 29 from oid 50 current snap is 96 2024-09-11T01:35:33.738 INFO:tasks.rados.rados.0.smithi071.stdout:886: rollback oid 1 current snap is 96 2024-09-11T01:35:33.738 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 1 to 91 2024-09-11T01:35:33.738 INFO:tasks.rados.rados.0.smithi071.stdout:887: snap_remove snap 88 2024-09-11T01:35:34.562 INFO:tasks.rados.rados.0.smithi071.stdout:887: done (4 left) 2024-09-11T01:35:34.562 INFO:tasks.rados.rados.0.smithi071.stdout:888: snap_remove snap 95 2024-09-11T01:35:35.572 INFO:tasks.rados.rados.0.smithi071.stdout:888: done (4 left) 2024-09-11T01:35:35.572 INFO:tasks.rados.rados.0.smithi071.stdout:889: copy_from oid 18 from oid 39 current snap is 96 2024-09-11T01:35:35.572 INFO:tasks.rados.rados.0.smithi071.stdout:886: finishing rollback tid 0 to smithi07134777-1 2024-09-11T01:35:35.572 INFO:tasks.rados.rados.0.smithi071.stdout:884: finishing rollback tid 1 to smithi07134777-27 2024-09-11T01:35:35.572 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-11T01:35:35.573 INFO:tasks.rados.rados.0.smithi071.stdout:886: finishing rollback tid 1 to smithi07134777-1 2024-09-11T01:35:35.573 INFO:tasks.rados.rados.0.smithi071.stdout:886: finishing rollback tid 2 to smithi07134777-1 2024-09-11T01:35:35.573 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 312 (ObjNum 232 snap 57 seq_num 232) dirty exists 2024-09-11T01:35:35.573 INFO:tasks.rados.rados.0.smithi071.stdout:883: expect (ObjNum 273 snap 72 seq_num 273) 2024-09-11T01:35:35.573 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:35:35.681 INFO:tasks.rados.rados.0.smithi071.stdout:885: finishing copy_from racing read to smithi07134777-29 2024-09-11T01:35:35.681 INFO:tasks.rados.rados.0.smithi071.stdout:885: finishing copy_from to smithi07134777-29 2024-09-11T01:35:35.681 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 129 (ObjNum 226 snap 55 seq_num 226) dirty exists 2024-09-11T01:35:35.681 INFO:tasks.rados.rados.0.smithi071.stdout:889: finishing copy_from racing read to smithi07134777-18 2024-09-11T01:35:35.681 INFO:tasks.rados.rados.0.smithi071.stdout:889: finishing copy_from to smithi07134777-18 2024-09-11T01:35:35.681 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 173 (ObjNum 229 snap 55 seq_num 229) dirty exists 2024-09-11T01:35:35.681 INFO:tasks.rados.rados.0.smithi071.stdout:883: done (4 left) 2024-09-11T01:35:35.681 INFO:tasks.rados.rados.0.smithi071.stdout:884: done (3 left) 2024-09-11T01:35:35.681 INFO:tasks.rados.rados.0.smithi071.stdout:885: done (2 left) 2024-09-11T01:35:35.682 INFO:tasks.rados.rados.0.smithi071.stdout:886: done (1 left) 2024-09-11T01:35:35.682 INFO:tasks.rados.rados.0.smithi071.stdout:889: done (0 left) 2024-09-11T01:35:35.682 INFO:tasks.rados.rados.0.smithi071.stdout:890: write oid 49 current snap is 96 2024-09-11T01:35:35.682 INFO:tasks.rados.rados.0.smithi071.stdout:890: seq_num 335 ranges {430345=727947,1570056=727439,2699879=469889} 2024-09-11T01:35:35.686 INFO:tasks.rados.rados.0.smithi071.stdout:890: writing smithi07134777-49 from 430345 to 1158292 tid 1 2024-09-11T01:35:35.692 INFO:tasks.rados.rados.0.smithi071.stdout:890: writing smithi07134777-49 from 1570056 to 2297495 tid 2 2024-09-11T01:35:35.697 INFO:tasks.rados.rados.0.smithi071.stdout:890: writing smithi07134777-49 from 2699879 to 3169768 tid 3 2024-09-11T01:35:35.697 INFO:tasks.rados.rados.0.smithi071.stdout:891: snap_remove snap 87 2024-09-11T01:35:36.575 INFO:tasks.rados.rados.0.smithi071.stdout:891: done (1 left) 2024-09-11T01:35:36.575 INFO:tasks.rados.rados.0.smithi071.stdout:892: copy_from oid 48 from oid 40 current snap is 96 2024-09-11T01:35:36.575 INFO:tasks.rados.rados.0.smithi071.stdout:890: finishing write tid 1 to smithi07134777-49 2024-09-11T01:35:36.575 INFO:tasks.rados.rados.0.smithi071.stdout:890: finishing write tid 2 to smithi07134777-49 2024-09-11T01:35:36.575 INFO:tasks.rados.rados.0.smithi071.stdout:890: finishing write tid 3 to smithi07134777-49 2024-09-11T01:35:36.575 INFO:tasks.rados.rados.0.smithi071.stdout:890: finishing write tid 4 to smithi07134777-49 2024-09-11T01:35:36.575 INFO:tasks.rados.rados.0.smithi071.stdout:890: finishing write tid 5 to smithi07134777-49 2024-09-11T01:35:36.575 INFO:tasks.rados.rados.0.smithi071.stdout:890: oid 49 updating version 0 to 223 2024-09-11T01:35:36.575 INFO:tasks.rados.rados.0.smithi071.stdout:890: oid 49 version 223 is already newer than 220 2024-09-11T01:35:36.576 INFO:tasks.rados.rados.0.smithi071.stdout:890: oid 49 version 223 is already newer than 222 2024-09-11T01:35:36.576 INFO:tasks.rados.rados.0.smithi071.stdout:890: oid 49 version 223 is already newer than 221 2024-09-11T01:35:36.576 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 223 (ObjNum 335 snap 96 seq_num 335) dirty exists 2024-09-11T01:35:36.576 INFO:tasks.rados.rados.0.smithi071.stdout:890: left oid 49 (ObjNum 335 snap 96 seq_num 335) 2024-09-11T01:35:36.576 INFO:tasks.rados.rados.0.smithi071.stdout:890: done (1 left) 2024-09-11T01:35:36.576 INFO:tasks.rados.rados.0.smithi071.stdout:893: write (excl) oid 9 current snap is 96 2024-09-11T01:35:36.576 INFO:tasks.rados.rados.0.smithi071.stdout:893: seq_num 336 ranges {447152=403196,1512185=587450,2727302=681161} 2024-09-11T01:35:36.579 INFO:tasks.rados.rados.0.smithi071.stdout:893: writing smithi07134777-9 from 447152 to 850348 tid 1 2024-09-11T01:35:36.587 INFO:tasks.rados.rados.0.smithi071.stdout:893: writing smithi07134777-9 from 1512185 to 2099635 tid 2 2024-09-11T01:35:36.594 INFO:tasks.rados.rados.0.smithi071.stdout:893: writing smithi07134777-9 from 2727302 to 3408463 tid 3 2024-09-11T01:35:36.595 INFO:tasks.rados.rados.0.smithi071.stdout:894: write (excl) oid 4 current snap is 96 2024-09-11T01:35:36.595 INFO:tasks.rados.rados.0.smithi071.stdout:892: finishing copy_from to smithi07134777-48 2024-09-11T01:35:36.595 INFO:tasks.rados.rados.0.smithi071.stdout:892: got expected ENOENT (src dne) 2024-09-11T01:35:36.595 INFO:tasks.rados.rados.0.smithi071.stdout:894: seq_num 337 ranges {463959=478445,1570667=731108,3038372=492433} 2024-09-11T01:35:36.599 INFO:tasks.rados.rados.0.smithi071.stdout:894: writing smithi07134777-4 from 463959 to 942404 tid 1 2024-09-11T01:35:36.607 INFO:tasks.rados.rados.0.smithi071.stdout:894: writing smithi07134777-4 from 1570667 to 2301775 tid 2 2024-09-11T01:35:36.614 INFO:tasks.rados.rados.0.smithi071.stdout:894: writing smithi07134777-4 from 3038372 to 3530805 tid 3 2024-09-11T01:35:36.614 INFO:tasks.rados.rados.0.smithi071.stdout:895: copy_from oid 26 from oid 16 current snap is 96 2024-09-11T01:35:36.614 INFO:tasks.rados.rados.0.smithi071.stdout:892: finishing copy_from racing read to smithi07134777-48 2024-09-11T01:35:36.614 INFO:tasks.rados.rados.0.smithi071.stdout:893: finishing write tid 1 to smithi07134777-9 2024-09-11T01:35:36.614 INFO:tasks.rados.rados.0.smithi071.stdout:893: finishing write tid 2 to smithi07134777-9 2024-09-11T01:35:36.614 INFO:tasks.rados.rados.0.smithi071.stdout:893: finishing write tid 3 to smithi07134777-9 2024-09-11T01:35:36.616 INFO:tasks.rados.rados.0.smithi071.stdout:893: finishing write tid 4 to smithi07134777-9 2024-09-11T01:35:36.616 INFO:tasks.rados.rados.0.smithi071.stdout:893: finishing write tid 5 to smithi07134777-9 2024-09-11T01:35:36.617 INFO:tasks.rados.rados.0.smithi071.stdout:893: oid 9 updating version 0 to 107 2024-09-11T01:35:36.617 INFO:tasks.rados.rados.0.smithi071.stdout:893: oid 9 version 107 is already newer than 106 2024-09-11T01:35:36.617 INFO:tasks.rados.rados.0.smithi071.stdout:893: oid 9 version 107 is already newer than 105 2024-09-11T01:35:36.617 INFO:tasks.rados.rados.0.smithi071.stdout:893: oid 9 updating version 107 to 108 2024-09-11T01:35:36.617 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 108 (ObjNum 336 snap 96 seq_num 336) dirty exists 2024-09-11T01:35:36.617 INFO:tasks.rados.rados.0.smithi071.stdout:893: left oid 9 (ObjNum 336 snap 96 seq_num 336) 2024-09-11T01:35:36.617 INFO:tasks.rados.rados.0.smithi071.stdout:894: finishing write tid 1 to smithi07134777-4 2024-09-11T01:35:36.617 INFO:tasks.rados.rados.0.smithi071.stdout:894: finishing write tid 2 to smithi07134777-4 2024-09-11T01:35:36.617 INFO:tasks.rados.rados.0.smithi071.stdout:892: done (3 left) 2024-09-11T01:35:36.617 INFO:tasks.rados.rados.0.smithi071.stdout:893: done (2 left) 2024-09-11T01:35:36.617 INFO:tasks.rados.rados.0.smithi071.stdout:896: delete oid 46 current snap is 96 2024-09-11T01:35:36.618 INFO:tasks.rados.rados.0.smithi071.stdout:896: done (2 left) 2024-09-11T01:35:36.618 INFO:tasks.rados.rados.0.smithi071.stdout:897: delete oid 40 current snap is 96 2024-09-11T01:35:36.619 INFO:tasks.rados.rados.0.smithi071.stdout:897: done (2 left) 2024-09-11T01:35:36.619 INFO:tasks.rados.rados.0.smithi071.stdout:898: write (excl) oid 33 current snap is 96 2024-09-11T01:35:36.619 INFO:tasks.rados.rados.0.smithi071.stdout:898: seq_num 340 ranges {514380=704192,1629760=478435,2604288=442602} 2024-09-11T01:35:36.626 INFO:tasks.rados.rados.0.smithi071.stdout:898: writing smithi07134777-33 from 514380 to 1218572 tid 1 2024-09-11T01:35:36.631 INFO:tasks.rados.rados.0.smithi071.stdout:898: writing smithi07134777-33 from 1629760 to 2108195 tid 2 2024-09-11T01:35:36.636 INFO:tasks.rados.rados.0.smithi071.stdout:898: writing smithi07134777-33 from 2604288 to 3046890 tid 3 2024-09-11T01:35:36.636 INFO:tasks.rados.rados.0.smithi071.stdout:899: write (excl) oid 40 current snap is 96 2024-09-11T01:35:36.636 INFO:tasks.rados.rados.0.smithi071.stdout:899: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2024-09-11T01:35:36.643 INFO:tasks.rados.rados.0.smithi071.stdout:899: writing smithi07134777-40 from 531187 to 1310628 tid 1 2024-09-11T01:35:36.651 INFO:tasks.rados.rados.0.smithi071.stdout:899: writing smithi07134777-40 from 2088242 to 2826688 tid 2 2024-09-11T01:35:36.657 INFO:tasks.rados.rados.0.smithi071.stdout:899: writing smithi07134777-40 from 3431711 to 3731187 tid 3 2024-09-11T01:35:36.657 INFO:tasks.rados.rados.0.smithi071.stdout:900: snap_create 2024-09-11T01:35:36.657 INFO:tasks.rados.rados.0.smithi071.stdout:894: finishing write tid 3 to smithi07134777-4 2024-09-11T01:35:36.658 INFO:tasks.rados.rados.0.smithi071.stdout:894: finishing write tid 4 to smithi07134777-4 2024-09-11T01:35:36.658 INFO:tasks.rados.rados.0.smithi071.stdout:894: finishing write tid 5 to smithi07134777-4 2024-09-11T01:35:36.658 INFO:tasks.rados.rados.0.smithi071.stdout:894: oid 4 updating version 0 to 307 2024-09-11T01:35:36.658 INFO:tasks.rados.rados.0.smithi071.stdout:894: oid 4 version 307 is already newer than 304 2024-09-11T01:35:36.658 INFO:tasks.rados.rados.0.smithi071.stdout:894: oid 4 version 307 is already newer than 305 2024-09-11T01:35:36.658 INFO:tasks.rados.rados.0.smithi071.stdout:894: oid 4 version 307 is already newer than 306 2024-09-11T01:35:36.658 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 307 (ObjNum 337 snap 96 seq_num 337) dirty exists 2024-09-11T01:35:36.658 INFO:tasks.rados.rados.0.smithi071.stdout:894: left oid 4 (ObjNum 337 snap 96 seq_num 337) 2024-09-11T01:35:36.658 INFO:tasks.rados.rados.0.smithi071.stdout:895: finishing copy_from racing read to smithi07134777-26 2024-09-11T01:35:36.658 INFO:tasks.rados.rados.0.smithi071.stdout:895: finishing copy_from to smithi07134777-26 2024-09-11T01:35:36.658 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 129 (ObjNum 226 snap 55 seq_num 226) dirty exists 2024-09-11T01:35:36.658 INFO:tasks.rados.rados.0.smithi071.stdout:898: finishing write tid 1 to smithi07134777-33 2024-09-11T01:35:36.659 INFO:tasks.rados.rados.0.smithi071.stdout:898: finishing write tid 2 to smithi07134777-33 2024-09-11T01:35:36.659 INFO:tasks.rados.rados.0.smithi071.stdout:898: finishing write tid 3 to smithi07134777-33 2024-09-11T01:35:36.659 INFO:tasks.rados.rados.0.smithi071.stdout:898: finishing write tid 4 to smithi07134777-33 2024-09-11T01:35:36.659 INFO:tasks.rados.rados.0.smithi071.stdout:898: finishing write tid 5 to smithi07134777-33 2024-09-11T01:35:36.659 INFO:tasks.rados.rados.0.smithi071.stdout:898: oid 33 updating version 0 to 318 2024-09-11T01:35:36.659 INFO:tasks.rados.rados.0.smithi071.stdout:898: oid 33 version 318 is already newer than 317 2024-09-11T01:35:36.659 INFO:tasks.rados.rados.0.smithi071.stdout:898: oid 33 version 318 is already newer than 315 2024-09-11T01:35:36.659 INFO:tasks.rados.rados.0.smithi071.stdout:898: oid 33 version 318 is already newer than 316 2024-09-11T01:35:36.659 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 318 (ObjNum 340 snap 96 seq_num 340) dirty exists 2024-09-11T01:35:36.659 INFO:tasks.rados.rados.0.smithi071.stdout:898: left oid 33 (ObjNum 340 snap 96 seq_num 340) 2024-09-11T01:35:36.659 INFO:tasks.rados.rados.0.smithi071.stdout:899: finishing write tid 1 to smithi07134777-40 2024-09-11T01:35:36.659 INFO:tasks.rados.rados.0.smithi071.stdout:899: finishing write tid 2 to smithi07134777-40 2024-09-11T01:35:36.660 INFO:tasks.rados.rados.0.smithi071.stdout:899: finishing write tid 3 to smithi07134777-40 2024-09-11T01:35:36.660 INFO:tasks.rados.rados.0.smithi071.stdout:899: finishing write tid 4 to smithi07134777-40 2024-09-11T01:35:36.660 INFO:tasks.rados.rados.0.smithi071.stdout:899: finishing write tid 5 to smithi07134777-40 2024-09-11T01:35:36.660 INFO:tasks.rados.rados.0.smithi071.stdout:899: oid 40 updating version 0 to 213 2024-09-11T01:35:36.660 INFO:tasks.rados.rados.0.smithi071.stdout:899: oid 40 version 213 is already newer than 212 2024-09-11T01:35:36.660 INFO:tasks.rados.rados.0.smithi071.stdout:899: oid 40 updating version 213 to 215 2024-09-11T01:35:36.660 INFO:tasks.rados.rados.0.smithi071.stdout:899: oid 40 version 215 is already newer than 214 2024-09-11T01:35:36.661 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 215 (ObjNum 341 snap 96 seq_num 341) dirty exists 2024-09-11T01:35:36.661 INFO:tasks.rados.rados.0.smithi071.stdout:899: left oid 40 (ObjNum 341 snap 96 seq_num 341) 2024-09-11T01:35:37.584 INFO:tasks.rados.rados.0.smithi071.stdout:894: done (4 left) 2024-09-11T01:35:37.584 INFO:tasks.rados.rados.0.smithi071.stdout:895: done (3 left) 2024-09-11T01:35:37.585 INFO:tasks.rados.rados.0.smithi071.stdout:898: done (2 left) 2024-09-11T01:35:37.585 INFO:tasks.rados.rados.0.smithi071.stdout:899: done (1 left) 2024-09-11T01:35:37.585 INFO:tasks.rados.rados.0.smithi071.stdout:900: done (0 left) 2024-09-11T01:35:37.585 INFO:tasks.rados.rados.0.smithi071.stdout:901: rollback oid 37 current snap is 97 2024-09-11T01:35:37.585 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 37 to 90 2024-09-11T01:35:37.585 INFO:tasks.rados.rados.0.smithi071.stdout:902: snap_remove snap 82 2024-09-11T01:35:38.588 INFO:tasks.rados.rados.0.smithi071.stdout:902: done (1 left) 2024-09-11T01:35:38.588 INFO:tasks.rados.rados.0.smithi071.stdout:903: rollback oid 49 current snap is 97 2024-09-11T01:35:38.588 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 49 to 85 2024-09-11T01:35:38.588 INFO:tasks.rados.rados.0.smithi071.stdout:901: finishing rollback tid 0 to smithi07134777-37 2024-09-11T01:35:38.588 INFO:tasks.rados.rados.0.smithi071.stdout:901: finishing rollback tid 1 to smithi07134777-37 2024-09-11T01:35:38.588 INFO:tasks.rados.rados.0.smithi071.stdout:901: finishing rollback tid 2 to smithi07134777-37 2024-09-11T01:35:38.589 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 226 (ObjNum 295 snap 80 seq_num 295) dirty exists 2024-09-11T01:35:38.589 INFO:tasks.rados.rados.0.smithi071.stdout:901: done (1 left) 2024-09-11T01:35:38.589 INFO:tasks.rados.rados.0.smithi071.stdout:904: snap_create 2024-09-11T01:35:38.600 INFO:tasks.rados.rados.0.smithi071.stdout:903: finishing rollback tid 0 to smithi07134777-49 2024-09-11T01:35:38.603 INFO:tasks.rados.rados.0.smithi071.stdout:903: finishing rollback tid 1 to smithi07134777-49 2024-09-11T01:35:38.603 INFO:tasks.rados.rados.0.smithi071.stdout:903: finishing rollback tid 2 to smithi07134777-49 2024-09-11T01:35:38.603 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 226 (ObjNum 200 snap 46 seq_num 200) dirty exists 2024-09-11T01:35:39.593 INFO:tasks.rados.rados.0.smithi071.stdout:903: done (1 left) 2024-09-11T01:35:39.593 INFO:tasks.rados.rados.0.smithi071.stdout:904: done (0 left) 2024-09-11T01:35:39.593 INFO:tasks.rados.rados.0.smithi071.stdout:905: write oid 31 current snap is 98 2024-09-11T01:35:39.593 INFO:tasks.rados.rados.0.smithi071.stdout:905: seq_num 342 ranges {547994=454690,1746724=482104,2659134=465146,3605976=142018} 2024-09-11T01:35:39.598 INFO:tasks.rados.rados.0.smithi071.stdout:905: writing smithi07134777-31 from 547994 to 1002684 tid 1 2024-09-11T01:35:39.606 INFO:tasks.rados.rados.0.smithi071.stdout:905: writing smithi07134777-31 from 1746724 to 2228828 tid 2 2024-09-11T01:35:39.611 INFO:tasks.rados.rados.0.smithi071.stdout:905: writing smithi07134777-31 from 2659134 to 3124280 tid 3 2024-09-11T01:35:39.615 INFO:tasks.rados.rados.0.smithi071.stdout:905: writing smithi07134777-31 from 3605976 to 3747994 tid 4 2024-09-11T01:35:39.615 INFO:tasks.rados.rados.0.smithi071.stdout:906: rollback oid 26 current snap is 98 2024-09-11T01:35:39.615 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 26 to 83 2024-09-11T01:35:39.615 INFO:tasks.rados.rados.0.smithi071.stdout:905: finishing write tid 1 to smithi07134777-31 2024-09-11T01:35:39.615 INFO:tasks.rados.rados.0.smithi071.stdout:905: finishing write tid 2 to smithi07134777-31 2024-09-11T01:35:39.615 INFO:tasks.rados.rados.0.smithi071.stdout:907: snap_remove snap 90 2024-09-11T01:35:40.597 INFO:tasks.rados.rados.0.smithi071.stdout:907: done (2 left) 2024-09-11T01:35:40.597 INFO:tasks.rados.rados.0.smithi071.stdout:908: delete oid 3 current snap is 98 2024-09-11T01:35:40.597 INFO:tasks.rados.rados.0.smithi071.stdout:905: finishing write tid 3 to smithi07134777-31 2024-09-11T01:35:40.597 INFO:tasks.rados.rados.0.smithi071.stdout:905: finishing write tid 4 to smithi07134777-31 2024-09-11T01:35:40.597 INFO:tasks.rados.rados.0.smithi071.stdout:905: finishing write tid 5 to smithi07134777-31 2024-09-11T01:35:40.597 INFO:tasks.rados.rados.0.smithi071.stdout:905: finishing write tid 6 to smithi07134777-31 2024-09-11T01:35:40.597 INFO:tasks.rados.rados.0.smithi071.stdout:905: oid 31 updating version 0 to 315 2024-09-11T01:35:40.597 INFO:tasks.rados.rados.0.smithi071.stdout:905: oid 31 updating version 315 to 317 2024-09-11T01:35:40.598 INFO:tasks.rados.rados.0.smithi071.stdout:905: oid 31 version 317 is already newer than 314 2024-09-11T01:35:40.598 INFO:tasks.rados.rados.0.smithi071.stdout:905: oid 31 version 317 is already newer than 313 2024-09-11T01:35:40.598 INFO:tasks.rados.rados.0.smithi071.stdout:905: oid 31 version 317 is already newer than 316 2024-09-11T01:35:40.598 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 317 (ObjNum 342 snap 98 seq_num 342) dirty exists 2024-09-11T01:35:40.598 INFO:tasks.rados.rados.0.smithi071.stdout:905: left oid 31 (ObjNum 342 snap 98 seq_num 342) 2024-09-11T01:35:40.598 INFO:tasks.rados.rados.0.smithi071.stdout:906: finishing rollback tid 0 to smithi07134777-26 2024-09-11T01:35:40.598 INFO:tasks.rados.rados.0.smithi071.stdout:906: finishing rollback tid 1 to smithi07134777-26 2024-09-11T01:35:40.598 INFO:tasks.rados.rados.0.smithi071.stdout:906: finishing rollback tid 2 to smithi07134777-26 2024-09-11T01:35:40.598 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 328 (ObjNum 283 snap 73 seq_num 283) dirty exists 2024-09-11T01:35:40.605 INFO:tasks.rados.rados.0.smithi071.stdout:905: done (2 left) 2024-09-11T01:35:40.605 INFO:tasks.rados.rados.0.smithi071.stdout:906: done (1 left) 2024-09-11T01:35:40.605 INFO:tasks.rados.rados.0.smithi071.stdout:908: done (0 left) 2024-09-11T01:35:40.605 INFO:tasks.rados.rados.0.smithi071.stdout:909: copy_from oid 18 from oid 48 current snap is 98 2024-09-11T01:35:40.605 INFO:tasks.rados.rados.0.smithi071.stdout:910: read oid 39 snap -1 2024-09-11T01:35:40.606 INFO:tasks.rados.rados.0.smithi071.stdout:910: expect (ObjNum 229 snap 55 seq_num 229) 2024-09-11T01:35:40.606 INFO:tasks.rados.rados.0.smithi071.stdout:911: read oid 44 snap -1 2024-09-11T01:35:40.606 INFO:tasks.rados.rados.0.smithi071.stdout:911: expect deleted 2024-09-11T01:35:40.606 INFO:tasks.rados.rados.0.smithi071.stdout:912: rollback oid 46 current snap is 98 2024-09-11T01:35:40.606 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 46 to 83 2024-09-11T01:35:40.606 INFO:tasks.rados.rados.0.smithi071.stdout:913: snap_remove snap 85 2024-09-11T01:35:41.609 INFO:tasks.rados.rados.0.smithi071.stdout:912: finishing rollback tid 1 to smithi07134777-46 2024-09-11T01:35:41.609 INFO:tasks.rados.rados.0.smithi071.stdout:912: finishing rollback tid 2 to smithi07134777-46 2024-09-11T01:35:41.609 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 217 (ObjNum 240 snap 58 seq_num 240) dirty exists 2024-09-11T01:35:41.609 INFO:tasks.rados.rados.0.smithi071.stdout:910: expect (ObjNum 229 snap 55 seq_num 229) 2024-09-11T01:35:41.650 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:35:41.711 INFO:tasks.rados.rados.0.smithi071.stdout:909: finishing copy_from racing read to smithi07134777-18 2024-09-11T01:35:41.712 INFO:tasks.rados.rados.0.smithi071.stdout:909: finishing copy_from to smithi07134777-18 2024-09-11T01:35:41.712 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 394 (ObjNum 329 snap 94 seq_num 329) dirty exists 2024-09-11T01:35:41.712 INFO:tasks.rados.rados.0.smithi071.stdout:909: done (4 left) 2024-09-11T01:35:41.712 INFO:tasks.rados.rados.0.smithi071.stdout:910: done (3 left) 2024-09-11T01:35:41.712 INFO:tasks.rados.rados.0.smithi071.stdout:911: done (2 left) 2024-09-11T01:35:41.712 INFO:tasks.rados.rados.0.smithi071.stdout:912: done (1 left) 2024-09-11T01:35:41.712 INFO:tasks.rados.rados.0.smithi071.stdout:913: done (0 left) 2024-09-11T01:35:41.712 INFO:tasks.rados.rados.0.smithi071.stdout:914: delete oid 34 current snap is 98 2024-09-11T01:35:41.714 INFO:tasks.rados.rados.0.smithi071.stdout:914: done (0 left) 2024-09-11T01:35:41.714 INFO:tasks.rados.rados.0.smithi071.stdout:915: delete oid 2 current snap is 98 2024-09-11T01:35:41.717 INFO:tasks.rados.rados.0.smithi071.stdout:915: done (0 left) 2024-09-11T01:35:41.717 INFO:tasks.rados.rados.0.smithi071.stdout:916: write (excl) oid 7 current snap is 98 2024-09-11T01:35:41.717 INFO:tasks.rados.rados.0.smithi071.stdout:916: seq_num 346 ranges {615222=472039,1580652=489442,2768826=626587} 2024-09-11T01:35:41.723 INFO:tasks.rados.rados.0.smithi071.stdout:916: writing smithi07134777-7 from 615222 to 1087261 tid 1 2024-09-11T01:35:41.728 INFO:tasks.rados.rados.0.smithi071.stdout:916: writing smithi07134777-7 from 1580652 to 2070094 tid 2 2024-09-11T01:35:41.736 INFO:tasks.rados.rados.0.smithi071.stdout:916: writing smithi07134777-7 from 2768826 to 3395413 tid 3 2024-09-11T01:35:41.736 INFO:tasks.rados.rados.0.smithi071.stdout:917: read oid 31 snap -1 2024-09-11T01:35:41.736 INFO:tasks.rados.rados.0.smithi071.stdout:917: expect (ObjNum 342 snap 98 seq_num 342) 2024-09-11T01:35:41.737 INFO:tasks.rados.rados.0.smithi071.stdout:918: rollback oid 5 current snap is 98 2024-09-11T01:35:41.737 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 5 to 96 2024-09-11T01:35:41.737 INFO:tasks.rados.rados.0.smithi071.stdout:916: finishing write tid 1 to smithi07134777-7 2024-09-11T01:35:41.737 INFO:tasks.rados.rados.0.smithi071.stdout:916: finishing write tid 2 to smithi07134777-7 2024-09-11T01:35:41.737 INFO:tasks.rados.rados.0.smithi071.stdout:919: copy_from oid 10 from oid 26 current snap is 98 2024-09-11T01:35:41.737 INFO:tasks.rados.rados.0.smithi071.stdout:920: write (excl) oid 22 current snap is 98 2024-09-11T01:35:41.737 INFO:tasks.rados.rados.0.smithi071.stdout:920: seq_num 347 ranges {632029=547288,1639134=749453,2796249=437859,3653524=178505} 2024-09-11T01:35:41.742 INFO:tasks.rados.rados.0.smithi071.stdout:920: writing smithi07134777-22 from 632029 to 1179317 tid 1 2024-09-11T01:35:41.750 INFO:tasks.rados.rados.0.smithi071.stdout:920: writing smithi07134777-22 from 1639134 to 2388587 tid 2 2024-09-11T01:35:41.754 INFO:tasks.rados.rados.0.smithi071.stdout:920: writing smithi07134777-22 from 2796249 to 3234108 tid 3 2024-09-11T01:35:41.758 INFO:tasks.rados.rados.0.smithi071.stdout:920: writing smithi07134777-22 from 3653524 to 3832029 tid 4 2024-09-11T01:35:41.758 INFO:tasks.rados.rados.0.smithi071.stdout:921: delete oid 2 current snap is 98 2024-09-11T01:35:41.758 INFO:tasks.rados.rados.0.smithi071.stdout:918: finishing rollback tid 0 to smithi07134777-5 2024-09-11T01:35:41.759 INFO:tasks.rados.rados.0.smithi071.stdout:916: finishing write tid 3 to smithi07134777-7 2024-09-11T01:35:41.759 INFO:tasks.rados.rados.0.smithi071.stdout:918: finishing rollback tid 1 to smithi07134777-5 2024-09-11T01:35:41.759 INFO:tasks.rados.rados.0.smithi071.stdout:916: finishing write tid 4 to smithi07134777-7 2024-09-11T01:35:41.759 INFO:tasks.rados.rados.0.smithi071.stdout:916: finishing write tid 5 to smithi07134777-7 2024-09-11T01:35:41.759 INFO:tasks.rados.rados.0.smithi071.stdout:916: oid 7 updating version 0 to 443 2024-09-11T01:35:41.759 INFO:tasks.rados.rados.0.smithi071.stdout:916: oid 7 updating version 443 to 446 2024-09-11T01:35:41.759 INFO:tasks.rados.rados.0.smithi071.stdout:916: oid 7 version 446 is already newer than 445 2024-09-11T01:35:41.759 INFO:tasks.rados.rados.0.smithi071.stdout:916: oid 7 version 446 is already newer than 444 2024-09-11T01:35:41.759 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 446 (ObjNum 346 snap 98 seq_num 346) dirty exists 2024-09-11T01:35:41.759 INFO:tasks.rados.rados.0.smithi071.stdout:916: left oid 7 (ObjNum 346 snap 98 seq_num 346) 2024-09-11T01:35:41.760 INFO:tasks.rados.rados.0.smithi071.stdout:917: expect (ObjNum 342 snap 98 seq_num 342) 2024-09-11T01:35:41.823 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1312060 2024-09-11T01:35:41.864 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:35:41.920 INFO:tasks.rados.rados.0.smithi071.stdout:918: finishing rollback tid 2 to smithi07134777-5 2024-09-11T01:35:41.921 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 154 (ObjNum 322 snap 92 seq_num 322) dirty exists 2024-09-11T01:35:41.921 INFO:tasks.rados.rados.0.smithi071.stdout:919: finishing copy_from racing read to smithi07134777-10 2024-09-11T01:35:41.921 INFO:tasks.rados.rados.0.smithi071.stdout:920: finishing write tid 1 to smithi07134777-22 2024-09-11T01:35:41.921 INFO:tasks.rados.rados.0.smithi071.stdout:919: finishing copy_from to smithi07134777-10 2024-09-11T01:35:41.921 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 328 (ObjNum 283 snap 73 seq_num 283) dirty exists 2024-09-11T01:35:41.921 INFO:tasks.rados.rados.0.smithi071.stdout:916: done (5 left) 2024-09-11T01:35:41.921 INFO:tasks.rados.rados.0.smithi071.stdout:917: done (4 left) 2024-09-11T01:35:41.921 INFO:tasks.rados.rados.0.smithi071.stdout:918: done (3 left) 2024-09-11T01:35:41.921 INFO:tasks.rados.rados.0.smithi071.stdout:919: done (2 left) 2024-09-11T01:35:41.921 INFO:tasks.rados.rados.0.smithi071.stdout:921: done (1 left) 2024-09-11T01:35:41.921 INFO:tasks.rados.rados.0.smithi071.stdout:922: delete oid 1 current snap is 98 2024-09-11T01:35:41.921 INFO:tasks.rados.rados.0.smithi071.stdout:920: finishing write tid 2 to smithi07134777-22 2024-09-11T01:35:41.922 INFO:tasks.rados.rados.0.smithi071.stdout:920: finishing write tid 3 to smithi07134777-22 2024-09-11T01:35:41.922 INFO:tasks.rados.rados.0.smithi071.stdout:920: finishing write tid 4 to smithi07134777-22 2024-09-11T01:35:41.922 INFO:tasks.rados.rados.0.smithi071.stdout:920: finishing write tid 5 to smithi07134777-22 2024-09-11T01:35:41.922 INFO:tasks.rados.rados.0.smithi071.stdout:920: finishing write tid 6 to smithi07134777-22 2024-09-11T01:35:41.922 INFO:tasks.rados.rados.0.smithi071.stdout:920: oid 22 updating version 0 to 233 2024-09-11T01:35:41.922 INFO:tasks.rados.rados.0.smithi071.stdout:920: oid 22 updating version 233 to 237 2024-09-11T01:35:41.922 INFO:tasks.rados.rados.0.smithi071.stdout:920: oid 22 version 237 is already newer than 235 2024-09-11T01:35:41.922 INFO:tasks.rados.rados.0.smithi071.stdout:920: oid 22 version 237 is already newer than 236 2024-09-11T01:35:41.922 INFO:tasks.rados.rados.0.smithi071.stdout:920: oid 22 version 237 is already newer than 234 2024-09-11T01:35:41.922 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 237 (ObjNum 347 snap 98 seq_num 347) dirty exists 2024-09-11T01:35:41.922 INFO:tasks.rados.rados.0.smithi071.stdout:920: left oid 22 (ObjNum 347 snap 98 seq_num 347) 2024-09-11T01:35:41.923 INFO:tasks.rados.rados.0.smithi071.stdout:920: done (1 left) 2024-09-11T01:35:41.923 INFO:tasks.rados.rados.0.smithi071.stdout:922: done (0 left) 2024-09-11T01:35:41.923 INFO:tasks.rados.rados.0.smithi071.stdout:923: read oid 25 snap 97 2024-09-11T01:35:41.923 INFO:tasks.rados.rados.0.smithi071.stdout:923: expect (ObjNum 327 snap 92 seq_num 327) 2024-09-11T01:35:41.923 INFO:tasks.rados.rados.0.smithi071.stdout:924: snap_remove snap 83 2024-09-11T01:35:42.528 INFO:tasks.rados.rados.0.smithi071.stdout:924: done (1 left) 2024-09-11T01:35:42.528 INFO:tasks.rados.rados.0.smithi071.stdout:925: read oid 3 snap 92 2024-09-11T01:35:42.529 INFO:tasks.rados.rados.0.smithi071.stdout:925: expect (ObjNum 200 snap 46 seq_num 200) 2024-09-11T01:35:42.529 INFO:tasks.rados.rados.0.smithi071.stdout:926: delete oid 47 current snap is 98 2024-09-11T01:35:42.529 INFO:tasks.rados.rados.0.smithi071.stdout:923: expect (ObjNum 327 snap 92 seq_num 327) 2024-09-11T01:35:42.529 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:35:42.655 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:35:42.715 INFO:tasks.rados.rados.0.smithi071.stdout:925: expect (ObjNum 200 snap 46 seq_num 200) 2024-09-11T01:35:42.716 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:35:42.831 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2551300 2024-09-11T01:35:42.848 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:35:42.919 INFO:tasks.rados.rados.0.smithi071.stdout:923: done (2 left) 2024-09-11T01:35:42.920 INFO:tasks.rados.rados.0.smithi071.stdout:925: done (1 left) 2024-09-11T01:35:42.920 INFO:tasks.rados.rados.0.smithi071.stdout:926: done (0 left) 2024-09-11T01:35:42.920 INFO:tasks.rados.rados.0.smithi071.stdout:927: write oid 27 current snap is 98 2024-09-11T01:35:42.920 INFO:tasks.rados.rados.0.smithi071.stdout:927: seq_num 351 ranges {699257=564637,1873062=640438,3305941=593316} 2024-09-11T01:35:42.926 INFO:tasks.rados.rados.0.smithi071.stdout:927: writing smithi07134777-27 from 699257 to 1263894 tid 1 2024-09-11T01:35:42.932 INFO:tasks.rados.rados.0.smithi071.stdout:927: writing smithi07134777-27 from 1873062 to 2513500 tid 2 2024-09-11T01:35:42.940 INFO:tasks.rados.rados.0.smithi071.stdout:927: writing smithi07134777-27 from 3305941 to 3899257 tid 3 2024-09-11T01:35:42.940 INFO:tasks.rados.rados.0.smithi071.stdout:928: read oid 7 snap -1 2024-09-11T01:35:42.941 INFO:tasks.rados.rados.0.smithi071.stdout:928: expect (ObjNum 346 snap 98 seq_num 346) 2024-09-11T01:35:42.941 INFO:tasks.rados.rados.0.smithi071.stdout:929: snap_create 2024-09-11T01:35:42.941 INFO:tasks.rados.rados.0.smithi071.stdout:927: finishing write tid 1 to smithi07134777-27 2024-09-11T01:35:42.941 INFO:tasks.rados.rados.0.smithi071.stdout:927: finishing write tid 2 to smithi07134777-27 2024-09-11T01:35:42.945 INFO:tasks.rados.rados.0.smithi071.stdout:927: finishing write tid 3 to smithi07134777-27 2024-09-11T01:35:42.946 INFO:tasks.rados.rados.0.smithi071.stdout:928: expect (ObjNum 346 snap 98 seq_num 346) 2024-09-11T01:35:42.946 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:35:43.072 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3358183 2024-09-11T01:35:43.082 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:35:43.158 INFO:tasks.rados.rados.0.smithi071.stdout:927: finishing write tid 4 to smithi07134777-27 2024-09-11T01:35:43.158 INFO:tasks.rados.rados.0.smithi071.stdout:927: finishing write tid 5 to smithi07134777-27 2024-09-11T01:35:43.158 INFO:tasks.rados.rados.0.smithi071.stdout:927: oid 27 updating version 0 to 347 2024-09-11T01:35:43.159 INFO:tasks.rados.rados.0.smithi071.stdout:927: oid 27 version 347 is already newer than 345 2024-09-11T01:35:43.159 INFO:tasks.rados.rados.0.smithi071.stdout:927: oid 27 updating version 347 to 348 2024-09-11T01:35:43.159 INFO:tasks.rados.rados.0.smithi071.stdout:927: oid 27 version 348 is already newer than 346 2024-09-11T01:35:43.159 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 348 (ObjNum 351 snap 98 seq_num 351) dirty exists 2024-09-11T01:35:43.159 INFO:tasks.rados.rados.0.smithi071.stdout:927: left oid 27 (ObjNum 351 snap 98 seq_num 351) 2024-09-11T01:35:43.621 INFO:tasks.rados.rados.0.smithi071.stdout:927: done (2 left) 2024-09-11T01:35:43.621 INFO:tasks.rados.rados.0.smithi071.stdout:928: done (1 left) 2024-09-11T01:35:43.621 INFO:tasks.rados.rados.0.smithi071.stdout:929: done (0 left) 2024-09-11T01:35:43.622 INFO:tasks.rados.rados.0.smithi071.stdout:930: copy_from oid 15 from oid 20 current snap is 99 2024-09-11T01:35:43.622 INFO:tasks.rados.rados.0.smithi071.stdout:931: copy_from oid 18 from oid 33 current snap is 99 2024-09-11T01:35:43.622 INFO:tasks.rados.rados.0.smithi071.stdout:932: snap_create 2024-09-11T01:35:43.638 INFO:tasks.rados.rados.0.smithi071.stdout:930: finishing copy_from racing read to smithi07134777-15 2024-09-11T01:35:43.640 INFO:tasks.rados.rados.0.smithi071.stdout:931: finishing copy_from racing read to smithi07134777-18 2024-09-11T01:35:43.643 INFO:tasks.rados.rados.0.smithi071.stdout:930: finishing copy_from to smithi07134777-15 2024-09-11T01:35:43.643 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 244 (ObjNum 323 snap 92 seq_num 323) dirty exists 2024-09-11T01:35:43.645 INFO:tasks.rados.rados.0.smithi071.stdout:931: finishing copy_from to smithi07134777-18 2024-09-11T01:35:43.646 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 318 (ObjNum 340 snap 96 seq_num 340) dirty exists 2024-09-11T01:35:44.619 INFO:tasks.rados.rados.0.smithi071.stdout:930: done (2 left) 2024-09-11T01:35:44.619 INFO:tasks.rados.rados.0.smithi071.stdout:931: done (1 left) 2024-09-11T01:35:44.619 INFO:tasks.rados.rados.0.smithi071.stdout:932: done (0 left) 2024-09-11T01:35:44.619 INFO:tasks.rados.rados.0.smithi071.stdout:933: read oid 37 snap -1 2024-09-11T01:35:44.619 INFO:tasks.rados.rados.0.smithi071.stdout:933: expect (ObjNum 295 snap 80 seq_num 295) 2024-09-11T01:35:44.619 INFO:tasks.rados.rados.0.smithi071.stdout:934: snap_create 2024-09-11T01:35:44.633 INFO:tasks.rados.rados.0.smithi071.stdout:933: expect (ObjNum 295 snap 80 seq_num 295) 2024-09-11T01:35:44.633 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:35:44.714 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1854574 2024-09-11T01:35:45.634 INFO:tasks.rados.rados.0.smithi071.stdout:933: done (1 left) 2024-09-11T01:35:45.635 INFO:tasks.rados.rados.0.smithi071.stdout:934: done (0 left) 2024-09-11T01:35:45.635 INFO:tasks.rados.rados.0.smithi071.stdout:935: read oid 13 snap -1 2024-09-11T01:35:45.635 INFO:tasks.rados.rados.0.smithi071.stdout:935: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:35:45.635 INFO:tasks.rados.rados.0.smithi071.stdout:936: write (excl) oid 33 current snap is 101 2024-09-11T01:35:45.635 INFO:tasks.rados.rados.0.smithi071.stdout:936: seq_num 352 ranges {716064=639886,1815191=500449,2817011=526925} 2024-09-11T01:35:45.642 INFO:tasks.rados.rados.0.smithi071.stdout:936: writing smithi07134777-33 from 716064 to 1355950 tid 1 2024-09-11T01:35:45.647 INFO:tasks.rados.rados.0.smithi071.stdout:936: writing smithi07134777-33 from 1815191 to 2315640 tid 2 2024-09-11T01:35:45.653 INFO:tasks.rados.rados.0.smithi071.stdout:936: writing smithi07134777-33 from 2817011 to 3343936 tid 3 2024-09-11T01:35:45.653 INFO:tasks.rados.rados.0.smithi071.stdout:937: write (excl) oid 12 current snap is 101 2024-09-11T01:35:45.653 INFO:tasks.rados.rados.0.smithi071.stdout:937: seq_num 353 ranges {732871=715135,1873673=644107,3244434=688437} 2024-09-11T01:35:45.660 INFO:tasks.rados.rados.0.smithi071.stdout:937: writing smithi07134777-12 from 732871 to 1448006 tid 1 2024-09-11T01:35:45.666 INFO:tasks.rados.rados.0.smithi071.stdout:937: writing smithi07134777-12 from 1873673 to 2517780 tid 2 2024-09-11T01:35:45.674 INFO:tasks.rados.rados.0.smithi071.stdout:937: writing smithi07134777-12 from 3244434 to 3932871 tid 3 2024-09-11T01:35:45.674 INFO:tasks.rados.rados.0.smithi071.stdout:938: write oid 32 current snap is 101 2024-09-11T01:35:45.674 INFO:tasks.rados.rados.0.smithi071.stdout:938: seq_num 354 ranges {749678=790384,2332155=504118,3271857=549469} 2024-09-11T01:35:45.682 INFO:tasks.rados.rados.0.smithi071.stdout:938: writing smithi07134777-32 from 749678 to 1540062 tid 1 2024-09-11T01:35:45.688 INFO:tasks.rados.rados.0.smithi071.stdout:938: writing smithi07134777-32 from 2332155 to 2836273 tid 2 2024-09-11T01:35:45.693 INFO:tasks.rados.rados.0.smithi071.stdout:938: writing smithi07134777-32 from 3271857 to 3821326 tid 3 2024-09-11T01:35:45.694 INFO:tasks.rados.rados.0.smithi071.stdout:939: snap_remove snap 99 2024-09-11T01:35:46.636 INFO:tasks.rados.rados.0.smithi071.stdout:939: done (4 left) 2024-09-11T01:35:46.636 INFO:tasks.rados.rados.0.smithi071.stdout:940: delete oid 5 current snap is 101 2024-09-11T01:35:46.636 INFO:tasks.rados.rados.0.smithi071.stdout:935: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:35:46.665 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:35:46.693 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:35:46.723 INFO:tasks.rados.rados.0.smithi071.stdout:936: finishing write tid 1 to smithi07134777-33 2024-09-11T01:35:46.723 INFO:tasks.rados.rados.0.smithi071.stdout:936: finishing write tid 2 to smithi07134777-33 2024-09-11T01:35:46.723 INFO:tasks.rados.rados.0.smithi071.stdout:935: done (4 left) 2024-09-11T01:35:46.723 INFO:tasks.rados.rados.0.smithi071.stdout:940: done (3 left) 2024-09-11T01:35:46.723 INFO:tasks.rados.rados.0.smithi071.stdout:941: rollback oid 18 current snap is 101 2024-09-11T01:35:46.723 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 18 to 91 2024-09-11T01:35:46.723 INFO:tasks.rados.rados.0.smithi071.stdout:936: finishing write tid 3 to smithi07134777-33 2024-09-11T01:35:46.723 INFO:tasks.rados.rados.0.smithi071.stdout:936: finishing write tid 4 to smithi07134777-33 2024-09-11T01:35:46.723 INFO:tasks.rados.rados.0.smithi071.stdout:936: finishing write tid 5 to smithi07134777-33 2024-09-11T01:35:46.723 INFO:tasks.rados.rados.0.smithi071.stdout:936: oid 33 updating version 0 to 350 2024-09-11T01:35:46.724 INFO:tasks.rados.rados.0.smithi071.stdout:936: oid 33 updating version 350 to 352 2024-09-11T01:35:46.724 INFO:tasks.rados.rados.0.smithi071.stdout:936: oid 33 version 352 is already newer than 351 2024-09-11T01:35:46.724 INFO:tasks.rados.rados.0.smithi071.stdout:936: oid 33 updating version 352 to 353 2024-09-11T01:35:46.724 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 353 (ObjNum 352 snap 101 seq_num 352) dirty exists 2024-09-11T01:35:46.724 INFO:tasks.rados.rados.0.smithi071.stdout:936: left oid 33 (ObjNum 352 snap 101 seq_num 352) 2024-09-11T01:35:46.724 INFO:tasks.rados.rados.0.smithi071.stdout:937: finishing write tid 1 to smithi07134777-12 2024-09-11T01:35:46.724 INFO:tasks.rados.rados.0.smithi071.stdout:937: finishing write tid 2 to smithi07134777-12 2024-09-11T01:35:46.724 INFO:tasks.rados.rados.0.smithi071.stdout:937: finishing write tid 3 to smithi07134777-12 2024-09-11T01:35:46.724 INFO:tasks.rados.rados.0.smithi071.stdout:937: finishing write tid 4 to smithi07134777-12 2024-09-11T01:35:46.724 INFO:tasks.rados.rados.0.smithi071.stdout:937: finishing write tid 5 to smithi07134777-12 2024-09-11T01:35:46.724 INFO:tasks.rados.rados.0.smithi071.stdout:937: oid 12 updating version 0 to 357 2024-09-11T01:35:46.724 INFO:tasks.rados.rados.0.smithi071.stdout:937: oid 12 updating version 357 to 358 2024-09-11T01:35:46.724 INFO:tasks.rados.rados.0.smithi071.stdout:937: oid 12 version 358 is already newer than 355 2024-09-11T01:35:46.724 INFO:tasks.rados.rados.0.smithi071.stdout:937: oid 12 version 358 is already newer than 356 2024-09-11T01:35:46.725 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 358 (ObjNum 353 snap 101 seq_num 353) dirty exists 2024-09-11T01:35:46.725 INFO:tasks.rados.rados.0.smithi071.stdout:937: left oid 12 (ObjNum 353 snap 101 seq_num 353) 2024-09-11T01:35:46.725 INFO:tasks.rados.rados.0.smithi071.stdout:938: finishing write tid 1 to smithi07134777-32 2024-09-11T01:35:46.725 INFO:tasks.rados.rados.0.smithi071.stdout:938: finishing write tid 2 to smithi07134777-32 2024-09-11T01:35:46.725 INFO:tasks.rados.rados.0.smithi071.stdout:938: finishing write tid 3 to smithi07134777-32 2024-09-11T01:35:46.725 INFO:tasks.rados.rados.0.smithi071.stdout:938: finishing write tid 4 to smithi07134777-32 2024-09-11T01:35:46.725 INFO:tasks.rados.rados.0.smithi071.stdout:938: finishing write tid 5 to smithi07134777-32 2024-09-11T01:35:46.725 INFO:tasks.rados.rados.0.smithi071.stdout:938: oid 32 updating version 0 to 238 2024-09-11T01:35:46.725 INFO:tasks.rados.rados.0.smithi071.stdout:938: oid 32 version 238 is already newer than 236 2024-09-11T01:35:46.725 INFO:tasks.rados.rados.0.smithi071.stdout:938: oid 32 version 238 is already newer than 237 2024-09-11T01:35:46.725 INFO:tasks.rados.rados.0.smithi071.stdout:938: oid 32 version 238 is already newer than 235 2024-09-11T01:35:46.725 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 238 (ObjNum 354 snap 101 seq_num 354) dirty exists 2024-09-11T01:35:46.726 INFO:tasks.rados.rados.0.smithi071.stdout:938: left oid 32 (ObjNum 354 snap 101 seq_num 354) 2024-09-11T01:35:46.726 INFO:tasks.rados.rados.0.smithi071.stdout:936: done (3 left) 2024-09-11T01:35:46.726 INFO:tasks.rados.rados.0.smithi071.stdout:937: done (2 left) 2024-09-11T01:35:46.726 INFO:tasks.rados.rados.0.smithi071.stdout:938: done (1 left) 2024-09-11T01:35:46.726 INFO:tasks.rados.rados.0.smithi071.stdout:942: write oid 8 current snap is 101 2024-09-11T01:35:46.726 INFO:tasks.rados.rados.0.smithi071.stdout:942: seq_num 356 ranges {783292=540882,1932766=507787,3210350=572013} 2024-09-11T01:35:46.729 INFO:tasks.rados.rados.0.smithi071.stdout:942: writing smithi07134777-8 from 783292 to 1324174 tid 1 2024-09-11T01:35:46.735 INFO:tasks.rados.rados.0.smithi071.stdout:942: writing smithi07134777-8 from 1932766 to 2440553 tid 2 2024-09-11T01:35:46.742 INFO:tasks.rados.rados.0.smithi071.stdout:942: writing smithi07134777-8 from 3210350 to 3782363 tid 3 2024-09-11T01:35:46.743 INFO:tasks.rados.rados.0.smithi071.stdout:943: delete oid 3 current snap is 101 2024-09-11T01:35:46.743 INFO:tasks.rados.rados.0.smithi071.stdout:941: finishing rollback tid 0 to smithi07134777-18 2024-09-11T01:35:46.743 INFO:tasks.rados.rados.0.smithi071.stdout:941: finishing rollback tid 1 to smithi07134777-18 2024-09-11T01:35:46.743 INFO:tasks.rados.rados.0.smithi071.stdout:941: finishing rollback tid 2 to smithi07134777-18 2024-09-11T01:35:46.743 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 397 (ObjNum 319 snap 91 seq_num 319) dirty exists 2024-09-11T01:35:46.743 INFO:tasks.rados.rados.0.smithi071.stdout:942: finishing write tid 1 to smithi07134777-8 2024-09-11T01:35:46.743 INFO:tasks.rados.rados.0.smithi071.stdout:942: finishing write tid 2 to smithi07134777-8 2024-09-11T01:35:46.747 INFO:tasks.rados.rados.0.smithi071.stdout:941: done (2 left) 2024-09-11T01:35:46.747 INFO:tasks.rados.rados.0.smithi071.stdout:943: done (1 left) 2024-09-11T01:35:46.747 INFO:tasks.rados.rados.0.smithi071.stdout:944: rollback oid 50 current snap is 101 2024-09-11T01:35:46.747 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 50 to 92 2024-09-11T01:35:46.747 INFO:tasks.rados.rados.0.smithi071.stdout:942: finishing write tid 3 to smithi07134777-8 2024-09-11T01:35:46.747 INFO:tasks.rados.rados.0.smithi071.stdout:945: write (excl) oid 28 current snap is 101 2024-09-11T01:35:46.747 INFO:tasks.rados.rados.0.smithi071.stdout:945: seq_num 358 ranges {416906=407733,1366083=511456} 2024-09-11T01:35:46.751 INFO:tasks.rados.rados.0.smithi071.stdout:945: writing smithi07134777-28 from 416906 to 824639 tid 1 2024-09-11T01:35:46.758 INFO:tasks.rados.rados.0.smithi071.stdout:945: writing smithi07134777-28 from 1366083 to 1877539 tid 2 2024-09-11T01:35:46.758 INFO:tasks.rados.rados.0.smithi071.stdout:946: copy_from oid 31 from oid 14 current snap is 101 2024-09-11T01:35:46.758 INFO:tasks.rados.rados.0.smithi071.stdout:942: finishing write tid 4 to smithi07134777-8 2024-09-11T01:35:46.758 INFO:tasks.rados.rados.0.smithi071.stdout:942: finishing write tid 5 to smithi07134777-8 2024-09-11T01:35:46.758 INFO:tasks.rados.rados.0.smithi071.stdout:942: oid 8 updating version 0 to 170 2024-09-11T01:35:46.758 INFO:tasks.rados.rados.0.smithi071.stdout:942: oid 8 updating version 170 to 173 2024-09-11T01:35:46.758 INFO:tasks.rados.rados.0.smithi071.stdout:942: oid 8 version 173 is already newer than 171 2024-09-11T01:35:46.758 INFO:tasks.rados.rados.0.smithi071.stdout:942: oid 8 version 173 is already newer than 172 2024-09-11T01:35:46.758 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 173 (ObjNum 356 snap 101 seq_num 356) dirty exists 2024-09-11T01:35:46.758 INFO:tasks.rados.rados.0.smithi071.stdout:942: left oid 8 (ObjNum 356 snap 101 seq_num 356) 2024-09-11T01:35:46.758 INFO:tasks.rados.rados.0.smithi071.stdout:944: finishing rollback tid 0 to smithi07134777-50 2024-09-11T01:35:46.758 INFO:tasks.rados.rados.0.smithi071.stdout:944: finishing rollback tid 1 to smithi07134777-50 2024-09-11T01:35:46.759 INFO:tasks.rados.rados.0.smithi071.stdout:942: done (3 left) 2024-09-11T01:35:46.759 INFO:tasks.rados.rados.0.smithi071.stdout:947: write (excl) oid 32 current snap is 101 2024-09-11T01:35:46.759 INFO:tasks.rados.rados.0.smithi071.stdout:947: seq_num 359 ranges {433713=482982,1424565=609148} 2024-09-11T01:35:46.762 INFO:tasks.rados.rados.0.smithi071.stdout:947: writing smithi07134777-32 from 433713 to 916695 tid 1 2024-09-11T01:35:46.769 INFO:tasks.rados.rados.0.smithi071.stdout:947: writing smithi07134777-32 from 1424565 to 2033713 tid 2 2024-09-11T01:35:46.769 INFO:tasks.rados.rados.0.smithi071.stdout:948: delete oid 1 current snap is 101 2024-09-11T01:35:46.769 INFO:tasks.rados.rados.0.smithi071.stdout:944: finishing rollback tid 2 to smithi07134777-50 2024-09-11T01:35:46.769 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 460 (ObjNum 287 snap 74 seq_num 287) dirty exists 2024-09-11T01:35:46.769 INFO:tasks.rados.rados.0.smithi071.stdout:947: finishing write tid 1 to smithi07134777-32 2024-09-11T01:35:46.772 INFO:tasks.rados.rados.0.smithi071.stdout:946: finishing copy_from racing read to smithi07134777-31 2024-09-11T01:35:46.773 INFO:tasks.rados.rados.0.smithi071.stdout:945: finishing write tid 1 to smithi07134777-28 2024-09-11T01:35:46.773 INFO:tasks.rados.rados.0.smithi071.stdout:945: finishing write tid 2 to smithi07134777-28 2024-09-11T01:35:46.775 INFO:tasks.rados.rados.0.smithi071.stdout:947: finishing write tid 2 to smithi07134777-32 2024-09-11T01:35:46.776 INFO:tasks.rados.rados.0.smithi071.stdout:945: finishing write tid 3 to smithi07134777-28 2024-09-11T01:35:46.776 INFO:tasks.rados.rados.0.smithi071.stdout:945: finishing write tid 4 to smithi07134777-28 2024-09-11T01:35:46.776 INFO:tasks.rados.rados.0.smithi071.stdout:945: oid 28 updating version 0 to 331 2024-09-11T01:35:46.776 INFO:tasks.rados.rados.0.smithi071.stdout:945: oid 28 updating version 331 to 332 2024-09-11T01:35:46.776 INFO:tasks.rados.rados.0.smithi071.stdout:945: oid 28 version 332 is already newer than 330 2024-09-11T01:35:46.776 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 332 (ObjNum 358 snap 101 seq_num 358) dirty exists 2024-09-11T01:35:46.776 INFO:tasks.rados.rados.0.smithi071.stdout:945: left oid 28 (ObjNum 358 snap 101 seq_num 358) 2024-09-11T01:35:46.777 INFO:tasks.rados.rados.0.smithi071.stdout:947: finishing write tid 3 to smithi07134777-32 2024-09-11T01:35:46.778 INFO:tasks.rados.rados.0.smithi071.stdout:947: finishing write tid 4 to smithi07134777-32 2024-09-11T01:35:46.778 INFO:tasks.rados.rados.0.smithi071.stdout:947: oid 32 updating version 0 to 240 2024-09-11T01:35:46.778 INFO:tasks.rados.rados.0.smithi071.stdout:947: oid 32 version 240 is already newer than 239 2024-09-11T01:35:46.778 INFO:tasks.rados.rados.0.smithi071.stdout:947: oid 32 updating version 240 to 241 2024-09-11T01:35:46.778 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 241 (ObjNum 359 snap 101 seq_num 359) dirty exists 2024-09-11T01:35:46.778 INFO:tasks.rados.rados.0.smithi071.stdout:947: left oid 32 (ObjNum 359 snap 101 seq_num 359) 2024-09-11T01:35:46.779 INFO:tasks.rados.rados.0.smithi071.stdout:946: finishing copy_from to smithi07134777-31 2024-09-11T01:35:46.779 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 403 (ObjNum 334 snap 96 seq_num 334) dirty exists 2024-09-11T01:35:46.779 INFO:tasks.rados.rados.0.smithi071.stdout:944: done (4 left) 2024-09-11T01:35:46.779 INFO:tasks.rados.rados.0.smithi071.stdout:945: done (3 left) 2024-09-11T01:35:46.779 INFO:tasks.rados.rados.0.smithi071.stdout:946: done (2 left) 2024-09-11T01:35:46.779 INFO:tasks.rados.rados.0.smithi071.stdout:947: done (1 left) 2024-09-11T01:35:46.779 INFO:tasks.rados.rados.0.smithi071.stdout:948: done (0 left) 2024-09-11T01:35:46.779 INFO:tasks.rados.rados.0.smithi071.stdout:949: rollback oid 14 current snap is 101 2024-09-11T01:35:46.779 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 14 to 96 2024-09-11T01:35:46.779 INFO:tasks.rados.rados.0.smithi071.stdout:950: copy_from oid 31 from oid 28 current snap is 101 2024-09-11T01:35:46.780 INFO:tasks.rados.rados.0.smithi071.stdout:951: snap_remove snap 100 2024-09-11T01:35:47.528 INFO:tasks.rados.rados.0.smithi071.stdout:951: done (2 left) 2024-09-11T01:35:47.528 INFO:tasks.rados.rados.0.smithi071.stdout:952: read oid 2 snap -1 2024-09-11T01:35:47.528 INFO:tasks.rados.rados.0.smithi071.stdout:952: expect deleted 2024-09-11T01:35:47.528 INFO:tasks.rados.rados.0.smithi071.stdout:953: read oid 5 snap 98 2024-09-11T01:35:47.528 INFO:tasks.rados.rados.0.smithi071.stdout:953: expect (ObjNum 322 snap 92 seq_num 322) 2024-09-11T01:35:47.528 INFO:tasks.rados.rados.0.smithi071.stdout:954: write oid 34 current snap is 101 2024-09-11T01:35:47.528 INFO:tasks.rados.rados.0.smithi071.stdout:954: seq_num 361 ranges {467327=633480,1825176=242151} 2024-09-11T01:35:47.534 INFO:tasks.rados.rados.0.smithi071.stdout:954: writing smithi07134777-34 from 467327 to 1100807 tid 1 2024-09-11T01:35:47.540 INFO:tasks.rados.rados.0.smithi071.stdout:954: writing smithi07134777-34 from 1825176 to 2067327 tid 2 2024-09-11T01:35:47.541 INFO:tasks.rados.rados.0.smithi071.stdout:955: rollback oid 18 current snap is 101 2024-09-11T01:35:47.541 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 18 to 97 2024-09-11T01:35:47.541 INFO:tasks.rados.rados.0.smithi071.stdout:949: finishing rollback tid 0 to smithi07134777-14 2024-09-11T01:35:47.541 INFO:tasks.rados.rados.0.smithi071.stdout:949: finishing rollback tid 1 to smithi07134777-14 2024-09-11T01:35:47.541 INFO:tasks.rados.rados.0.smithi071.stdout:949: finishing rollback tid 2 to smithi07134777-14 2024-09-11T01:35:47.541 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 464 (ObjNum 334 snap 96 seq_num 334) dirty exists 2024-09-11T01:35:47.541 INFO:tasks.rados.rados.0.smithi071.stdout:950: finishing copy_from racing read to smithi07134777-31 2024-09-11T01:35:47.541 INFO:tasks.rados.rados.0.smithi071.stdout:950: finishing copy_from to smithi07134777-31 2024-09-11T01:35:47.541 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 332 (ObjNum 358 snap 101 seq_num 358) dirty exists 2024-09-11T01:35:47.541 INFO:tasks.rados.rados.0.smithi071.stdout:949: done (5 left) 2024-09-11T01:35:47.541 INFO:tasks.rados.rados.0.smithi071.stdout:950: done (4 left) 2024-09-11T01:35:47.541 INFO:tasks.rados.rados.0.smithi071.stdout:952: done (3 left) 2024-09-11T01:35:47.541 INFO:tasks.rados.rados.0.smithi071.stdout:956: rollback oid 16 current snap is 101 2024-09-11T01:35:47.541 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 16 to 86 2024-09-11T01:35:47.541 INFO:tasks.rados.rados.0.smithi071.stdout:957: read oid 33 snap -1 2024-09-11T01:35:47.542 INFO:tasks.rados.rados.0.smithi071.stdout:957: expect (ObjNum 352 snap 101 seq_num 352) 2024-09-11T01:35:47.542 INFO:tasks.rados.rados.0.smithi071.stdout:958: write oid 21 current snap is 101 2024-09-11T01:35:47.542 INFO:tasks.rados.rados.0.smithi071.stdout:958: seq_num 362 ranges {484134=708729,1883658=200476} 2024-09-11T01:35:47.547 INFO:tasks.rados.rados.0.smithi071.stdout:958: writing smithi07134777-21 from 484134 to 1192863 tid 1 2024-09-11T01:35:47.552 INFO:tasks.rados.rados.0.smithi071.stdout:958: writing smithi07134777-21 from 1883658 to 2084134 tid 2 2024-09-11T01:35:47.553 INFO:tasks.rados.rados.0.smithi071.stdout:959: write (excl) oid 25 current snap is 101 2024-09-11T01:35:47.553 INFO:tasks.rados.rados.0.smithi071.stdout:959: seq_num 363 ranges {500941=783978,1942140=158801} 2024-09-11T01:35:47.560 INFO:tasks.rados.rados.0.smithi071.stdout:959: writing smithi07134777-25 from 500941 to 1284919 tid 1 2024-09-11T01:35:47.565 INFO:tasks.rados.rados.0.smithi071.stdout:959: writing smithi07134777-25 from 1942140 to 2100941 tid 2 2024-09-11T01:35:47.565 INFO:tasks.rados.rados.0.smithi071.stdout:960: rollback oid 13 current snap is 101 2024-09-11T01:35:47.565 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 13 to 91 2024-09-11T01:35:47.565 INFO:tasks.rados.rados.0.smithi071.stdout:953: expect (ObjNum 322 snap 92 seq_num 322) 2024-09-11T01:35:47.565 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:35:47.615 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:35:47.705 INFO:tasks.rados.rados.0.smithi071.stdout:955: finishing rollback tid 0 to smithi07134777-18 2024-09-11T01:35:47.706 INFO:tasks.rados.rados.0.smithi071.stdout:955: finishing rollback tid 1 to smithi07134777-18 2024-09-11T01:35:47.706 INFO:tasks.rados.rados.0.smithi071.stdout:955: finishing rollback tid 2 to smithi07134777-18 2024-09-11T01:35:47.706 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 400 (ObjNum 229 snap 55 seq_num 229) dirty exists 2024-09-11T01:35:47.706 INFO:tasks.rados.rados.0.smithi071.stdout:954: finishing write tid 1 to smithi07134777-34 2024-09-11T01:35:47.706 INFO:tasks.rados.rados.0.smithi071.stdout:954: finishing write tid 2 to smithi07134777-34 2024-09-11T01:35:47.706 INFO:tasks.rados.rados.0.smithi071.stdout:956: finishing rollback tid 0 to smithi07134777-16 2024-09-11T01:35:47.706 INFO:tasks.rados.rados.0.smithi071.stdout:957: expect (ObjNum 352 snap 101 seq_num 352) 2024-09-11T01:35:47.847 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:35:47.921 INFO:tasks.rados.rados.0.smithi071.stdout:954: finishing write tid 3 to smithi07134777-34 2024-09-11T01:35:47.921 INFO:tasks.rados.rados.0.smithi071.stdout:954: finishing write tid 4 to smithi07134777-34 2024-09-11T01:35:47.922 INFO:tasks.rados.rados.0.smithi071.stdout:954: oid 34 updating version 0 to 367 2024-09-11T01:35:47.922 INFO:tasks.rados.rados.0.smithi071.stdout:954: oid 34 updating version 367 to 369 2024-09-11T01:35:47.922 INFO:tasks.rados.rados.0.smithi071.stdout:954: oid 34 version 369 is already newer than 368 2024-09-11T01:35:47.922 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 369 (ObjNum 361 snap 101 seq_num 361) dirty exists 2024-09-11T01:35:47.922 INFO:tasks.rados.rados.0.smithi071.stdout:954: left oid 34 (ObjNum 361 snap 101 seq_num 361) 2024-09-11T01:35:47.922 INFO:tasks.rados.rados.0.smithi071.stdout:956: finishing rollback tid 1 to smithi07134777-16 2024-09-11T01:35:47.922 INFO:tasks.rados.rados.0.smithi071.stdout:956: finishing rollback tid 2 to smithi07134777-16 2024-09-11T01:35:47.922 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 338 (ObjNum 240 snap 58 seq_num 240) dirty exists 2024-09-11T01:35:47.922 INFO:tasks.rados.rados.0.smithi071.stdout:958: finishing write tid 1 to smithi07134777-21 2024-09-11T01:35:47.922 INFO:tasks.rados.rados.0.smithi071.stdout:953: done (7 left) 2024-09-11T01:35:47.923 INFO:tasks.rados.rados.0.smithi071.stdout:954: done (6 left) 2024-09-11T01:35:47.923 INFO:tasks.rados.rados.0.smithi071.stdout:955: done (5 left) 2024-09-11T01:35:47.923 INFO:tasks.rados.rados.0.smithi071.stdout:956: done (4 left) 2024-09-11T01:35:47.923 INFO:tasks.rados.rados.0.smithi071.stdout:957: done (3 left) 2024-09-11T01:35:47.923 INFO:tasks.rados.rados.0.smithi071.stdout:961: snap_remove snap 92 2024-09-11T01:35:48.139 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-11T01:35:48.139 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-11T01:35:48.140 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-11T01:35:48.142 INFO:tasks.ceph.mon.b:Started 2024-09-11T01:35:48.142 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-11T01:35:48.143 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-11T01:35:48.143 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T01:36:03.143 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-11T01:36:03.144 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-11T01:36:03.144 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:36:03.145 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:36:03.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.256+0000 7f125fd94640 10 monclient: get_monmap_and_config 2024-09-11T01:36:03.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.256+0000 7f125fd94640 10 monclient: build_initial_monmap 2024-09-11T01:36:03.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.256+0000 7f125fd94640 10 monclient: monmap: 2024-09-11T01:36:03.258 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:03.258 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:03.258 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:03.257623+0000 2024-09-11T01:36:03.259 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:03.257623+0000 2024-09-11T01:36:03.259 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:03.259 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:03.259 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:03.259 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:03.259 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:03.259 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:03.259 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:03.259 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:03.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.257+0000 7f125fd94640 1 Processor -- start 2024-09-11T01:36:03.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.257+0000 7f125fd94640 1 -- start start 2024-09-11T01:36:03.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.257+0000 7f125fd94640 10 monclient: init 2024-09-11T01:36:03.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125fd94640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:03.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125fd94640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-11T01:36:03.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125fd94640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f12580a2e90 0x7f125806dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:03.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125fd94640 10 monclient(hunting): picked mon.noname-c con 0x7f12580a2e90 addr v2:172.21.15.71:3300/0 2024-09-11T01:36:03.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125fd94640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f12580a3270 addr v1:172.21.15.71:6789/0 2024-09-11T01:36:03.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125fd94640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f125813d490 0x7f125813f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:03.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125fd94640 10 monclient(hunting): picked mon.noname-a con 0x7f125813d490 addr v2:172.21.15.12:3300/0 2024-09-11T01:36:03.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125fd94640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12580a46c0 con 0x7f12580a3270 2024-09-11T01:36:03.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125fd94640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:03.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125fd94640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f12580a4900 con 0x7f125813d490 2024-09-11T01:36:03.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125fd94640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:03.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125fd94640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f125806ee50 con 0x7f12580a2e90 2024-09-11T01:36:03.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125fd94640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:03.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125fd94640 10 monclient(hunting): authenticate will time out at 1169.700348s 2024-09-11T01:36:03.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125e331640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f125813d490 0x7f125813f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:03.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125e331640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f125813d490 0x7f125813f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59766/0 (socket says 172.21.15.12:59766) 2024-09-11T01:36:03.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125e331640 1 -- 172.21.15.12:0/3912751821 learned_addr learned my addr 172.21.15.12:0/3912751821 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:03.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125e331640 10 monclient(hunting): get_auth_request con 0x7f125813d490 auth_method 0 2024-09-11T01:36:03.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125e331640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:03.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125e331640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:03.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125e331640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:03.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125e331640 1 -- 172.21.15.12:0/3912751821 >> v2:172.21.15.12:3300/0 conn(0x7f125813d490 msgr2=0x7f125813f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:36:03.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125e331640 1 -- 172.21.15.12:0/3912751821 >> v2:172.21.15.12:3300/0 conn(0x7f125813d490 msgr2=0x7f125813f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:36:03.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125e331640 1 --2- 172.21.15.12:0/3912751821 >> v2:172.21.15.12:3300/0 conn(0x7f125813d490 0x7f125813f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:03.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125e331640 1 --2- 172.21.15.12:0/3912751821 >> v2:172.21.15.12:3300/0 conn(0x7f125813d490 0x7f125813f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:03.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125db30640 1 --2- 172.21.15.12:0/3912751821 >> v2:172.21.15.71:3300/0 conn(0x7f12580a2e90 0x7f125806dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:03.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125db30640 10 monclient(hunting): get_auth_request con 0x7f12580a2e90 auth_method 0 2024-09-11T01:36:03.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125db30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:03.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125db30640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:03.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125db30640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:03.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125db30640 1 -- 172.21.15.12:0/3912751821 >> v2:172.21.15.71:3300/0 conn(0x7f12580a2e90 msgr2=0x7f125806dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:36:03.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125db30640 1 -- 172.21.15.12:0/3912751821 >> v2:172.21.15.71:3300/0 conn(0x7f12580a2e90 msgr2=0x7f125806dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:36:03.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125db30640 1 --2- 172.21.15.12:0/3912751821 >> v2:172.21.15.71:3300/0 conn(0x7f12580a2e90 0x7f125806dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:03.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.258+0000 7f125db30640 1 --2- 172.21.15.12:0/3912751821 >> v2:172.21.15.71:3300/0 conn(0x7f12580a2e90 0x7f125806dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:03.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.260+0000 7f125d32f640 1 -- 172.21.15.12:0/3912751821 >> v1:172.21.15.71:6789/0 conn(0x7f12580a3270 legacy=0x7f12580a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:03.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.459+0000 7f125db30640 1 --2- 172.21.15.12:0/3912751821 >> v2:172.21.15.71:3300/0 conn(0x7f12580a2e90 0x7f125806dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:03.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.459+0000 7f125db30640 10 monclient(hunting): get_auth_request con 0x7f12580a2e90 auth_method 0 2024-09-11T01:36:03.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.459+0000 7f125db30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:03.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.459+0000 7f125db30640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:03.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.459+0000 7f125db30640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:36:03.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.459+0000 7f125db30640 1 -- 172.21.15.12:0/3912751821 >> v2:172.21.15.71:3300/0 conn(0x7f12580a2e90 msgr2=0x7f125806dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:36:03.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.459+0000 7f125db30640 1 -- 172.21.15.12:0/3912751821 >> v2:172.21.15.71:3300/0 conn(0x7f12580a2e90 msgr2=0x7f125806dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:36:03.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.459+0000 7f125db30640 1 --2- 172.21.15.12:0/3912751821 >> v2:172.21.15.71:3300/0 conn(0x7f12580a2e90 0x7f125806dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:03.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.459+0000 7f125db30640 1 --2- 172.21.15.12:0/3912751821 >> v2:172.21.15.71:3300/0 conn(0x7f12580a2e90 0x7f125806dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:36:03.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.460+0000 7f125e331640 1 --2- 172.21.15.12:0/3912751821 >> v2:172.21.15.12:3300/0 conn(0x7f125813d490 0x7f125813f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:03.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.460+0000 7f125e331640 10 monclient(hunting): get_auth_request con 0x7f125813d490 auth_method 0 2024-09-11T01:36:03.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.460+0000 7f125e331640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.460+0000 7f125e331640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.460+0000 7f125e331640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:36:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.460+0000 7f125e331640 1 -- 172.21.15.12:0/3912751821 >> v2:172.21.15.12:3300/0 conn(0x7f125813d490 msgr2=0x7f125813f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:36:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.460+0000 7f125e331640 1 -- 172.21.15.12:0/3912751821 >> v2:172.21.15.12:3300/0 conn(0x7f125813d490 msgr2=0x7f125813f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:36:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.460+0000 7f125e331640 1 --2- 172.21.15.12:0/3912751821 >> v2:172.21.15.12:3300/0 conn(0x7f125813d490 0x7f125813f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.460+0000 7f125e331640 1 --2- 172.21.15.12:0/3912751821 >> v2:172.21.15.12:3300/0 conn(0x7f125813d490 0x7f125813f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:36:03.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.860+0000 7f125db30640 1 --2- 172.21.15.12:0/3912751821 >> v2:172.21.15.71:3300/0 conn(0x7f12580a2e90 0x7f125806dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:03.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.860+0000 7f125db30640 10 monclient(hunting): get_auth_request con 0x7f12580a2e90 auth_method 0 2024-09-11T01:36:03.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.860+0000 7f125db30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:03.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.860+0000 7f125db30640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:03.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.860+0000 7f125db30640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:36:03.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.860+0000 7f125db30640 1 -- 172.21.15.12:0/3912751821 >> v2:172.21.15.71:3300/0 conn(0x7f12580a2e90 msgr2=0x7f125806dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:36:03.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.860+0000 7f125db30640 1 -- 172.21.15.12:0/3912751821 >> v2:172.21.15.71:3300/0 conn(0x7f12580a2e90 msgr2=0x7f125806dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:36:03.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.860+0000 7f125db30640 1 --2- 172.21.15.12:0/3912751821 >> v2:172.21.15.71:3300/0 conn(0x7f12580a2e90 0x7f125806dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:03.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.860+0000 7f125db30640 1 --2- 172.21.15.12:0/3912751821 >> v2:172.21.15.71:3300/0 conn(0x7f12580a2e90 0x7f125806dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:36:03.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.861+0000 7f125e331640 1 --2- 172.21.15.12:0/3912751821 >> v2:172.21.15.12:3300/0 conn(0x7f125813d490 0x7f125813f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:03.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.862+0000 7f125e331640 10 monclient(hunting): get_auth_request con 0x7f125813d490 auth_method 0 2024-09-11T01:36:03.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.862+0000 7f125e331640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:03.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.862+0000 7f125e331640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:03.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.862+0000 7f125e331640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:36:03.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.862+0000 7f125e331640 1 -- 172.21.15.12:0/3912751821 >> v2:172.21.15.12:3300/0 conn(0x7f125813d490 msgr2=0x7f125813f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:36:03.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.862+0000 7f125e331640 1 -- 172.21.15.12:0/3912751821 >> v2:172.21.15.12:3300/0 conn(0x7f125813d490 msgr2=0x7f125813f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:36:03.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.862+0000 7f125e331640 1 --2- 172.21.15.12:0/3912751821 >> v2:172.21.15.12:3300/0 conn(0x7f125813d490 0x7f125813f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:03.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:03.862+0000 7f125e331640 1 --2- 172.21.15.12:0/3912751821 >> v2:172.21.15.12:3300/0 conn(0x7f125813d490 0x7f125813f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:36:04.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:04.661+0000 7f125db30640 1 --2- 172.21.15.12:0/3912751821 >> v2:172.21.15.71:3300/0 conn(0x7f12580a2e90 0x7f125806dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:04.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:04.661+0000 7f125db30640 10 monclient(hunting): get_auth_request con 0x7f12580a2e90 auth_method 0 2024-09-11T01:36:04.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:04.661+0000 7f125db30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:04.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:04.661+0000 7f125db30640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:04.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:04.661+0000 7f125db30640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:36:04.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:04.661+0000 7f125db30640 1 -- 172.21.15.12:0/3912751821 >> v2:172.21.15.71:3300/0 conn(0x7f12580a2e90 msgr2=0x7f125806dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:36:04.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:04.661+0000 7f125db30640 1 -- 172.21.15.12:0/3912751821 >> v2:172.21.15.71:3300/0 conn(0x7f12580a2e90 msgr2=0x7f125806dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:36:04.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:04.661+0000 7f125db30640 1 --2- 172.21.15.12:0/3912751821 >> v2:172.21.15.71:3300/0 conn(0x7f12580a2e90 0x7f125806dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:04.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:04.662+0000 7f125db30640 1 --2- 172.21.15.12:0/3912751821 >> v2:172.21.15.71:3300/0 conn(0x7f12580a2e90 0x7f125806dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:36:04.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:04.664+0000 7f125e331640 1 --2- 172.21.15.12:0/3912751821 >> v2:172.21.15.12:3300/0 conn(0x7f125813d490 0x7f125813f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:04.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:04.664+0000 7f125e331640 10 monclient(hunting): get_auth_request con 0x7f125813d490 auth_method 0 2024-09-11T01:36:04.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:04.664+0000 7f125e331640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:04.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:04.664+0000 7f125e331640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:04.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:04.664+0000 7f125e331640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:36:04.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:04.664+0000 7f125e331640 1 -- 172.21.15.12:0/3912751821 >> v2:172.21.15.12:3300/0 conn(0x7f125813d490 msgr2=0x7f125813f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:36:04.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:04.664+0000 7f125e331640 1 -- 172.21.15.12:0/3912751821 >> v2:172.21.15.12:3300/0 conn(0x7f125813d490 msgr2=0x7f125813f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:36:04.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:04.664+0000 7f125e331640 1 --2- 172.21.15.12:0/3912751821 >> v2:172.21.15.12:3300/0 conn(0x7f125813d490 0x7f125813f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:04.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:04.664+0000 7f125e331640 1 --2- 172.21.15.12:0/3912751821 >> v2:172.21.15.12:3300/0 conn(0x7f125813d490 0x7f125813f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:36:06.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.258+0000 7f124f7fe640 10 monclient(hunting): tick 2024-09-11T01:36:06.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.258+0000 7f124f7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:36:06.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.258+0000 7f124f7fe640 1 monclient(hunting): continuing hunt 2024-09-11T01:36:06.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.258+0000 7f124f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:36:06.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.258+0000 7f124f7fe640 1 -- 172.21.15.12:0/3912751821 >> v2:172.21.15.71:3300/0 conn(0x7f12580a2e90 msgr2=0x7f125806dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:36:06.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.258+0000 7f124f7fe640 1 --2- 172.21.15.12:0/3912751821 >> v2:172.21.15.71:3300/0 conn(0x7f12580a2e90 0x7f125806dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:06.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.258+0000 7f124f7fe640 1 -- 172.21.15.12:0/3912751821 >> v2:172.21.15.12:3300/0 conn(0x7f125813d490 msgr2=0x7f125813f880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:36:06.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.258+0000 7f124f7fe640 1 --2- 172.21.15.12:0/3912751821 >> v2:172.21.15.12:3300/0 conn(0x7f125813d490 0x7f125813f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:06.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.258+0000 7f124f7fe640 1 -- 172.21.15.12:0/3912751821 >> v1:172.21.15.71:6789/0 conn(0x7f12580a3270 legacy=0x7f12580a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:06.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.258+0000 7f124f7fe640 10 monclient: _add_conns ranks=[0,2] 2024-09-11T01:36:06.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.258+0000 7f124f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f123c003730 addr v1:172.21.15.12:6789/0 2024-09-11T01:36:06.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.258+0000 7f124f7fe640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 0x7f123c0098a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:06.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.258+0000 7f124f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f123c0073e0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.258+0000 7f124f7fe640 1 -- 172.21.15.12:0/3912751821 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f125806eb70 con 0x7f123c003730 2024-09-11T01:36:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.258+0000 7f124f7fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.258+0000 7f124f7fe640 1 -- 172.21.15.12:0/3912751821 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f12580a4900 con 0x7f123c0073e0 2024-09-11T01:36:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.258+0000 7f124f7fe640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.258+0000 7f125d32f640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 0x7f123c0098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.258+0000 7f125d32f640 10 monclient(hunting): get_auth_request con 0x7f123c0073e0 auth_method 0 2024-09-11T01:36:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.258+0000 7f125d32f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.258+0000 7f125d32f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.258+0000 7f125d32f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.258+0000 7f125d32f640 1 -- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 msgr2=0x7f123c0098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:36:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.258+0000 7f125d32f640 1 -- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 msgr2=0x7f123c0098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:36:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.258+0000 7f125d32f640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 0x7f123c0098a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.259+0000 7f125d32f640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 0x7f123c0098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.261+0000 7f125db30640 1 -- 172.21.15.12:0/3912751821 >> v1:172.21.15.12:6789/0 conn(0x7f123c003730 legacy=0x7f123c005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.460+0000 7f125d32f640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 0x7f123c0098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.460+0000 7f125d32f640 10 monclient(hunting): get_auth_request con 0x7f123c0073e0 auth_method 0 2024-09-11T01:36:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.460+0000 7f125d32f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.460+0000 7f125d32f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.460+0000 7f125d32f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:36:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.460+0000 7f125d32f640 1 -- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 msgr2=0x7f123c0098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:36:06.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.460+0000 7f125d32f640 1 -- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 msgr2=0x7f123c0098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:36:06.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.460+0000 7f125d32f640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 0x7f123c0098a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:06.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.460+0000 7f125d32f640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 0x7f123c0098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:36:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.861+0000 7f125d32f640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 0x7f123c0098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.861+0000 7f125d32f640 10 monclient(hunting): get_auth_request con 0x7f123c0073e0 auth_method 0 2024-09-11T01:36:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.861+0000 7f125d32f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.861+0000 7f125d32f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.861+0000 7f125d32f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:36:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.861+0000 7f125d32f640 1 -- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 msgr2=0x7f123c0098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:36:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.861+0000 7f125d32f640 1 -- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 msgr2=0x7f123c0098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:36:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.861+0000 7f125d32f640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 0x7f123c0098a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:06.861+0000 7f125d32f640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 0x7f123c0098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:36:07.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:07.662+0000 7f125d32f640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 0x7f123c0098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:07.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:07.662+0000 7f125d32f640 10 monclient(hunting): get_auth_request con 0x7f123c0073e0 auth_method 0 2024-09-11T01:36:07.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:07.662+0000 7f125d32f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:07.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:07.662+0000 7f125d32f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:07.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:07.662+0000 7f125d32f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:36:07.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:07.662+0000 7f125d32f640 1 -- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 msgr2=0x7f123c0098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:36:07.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:07.662+0000 7f125d32f640 1 -- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 msgr2=0x7f123c0098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:36:07.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:07.662+0000 7f125d32f640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 0x7f123c0098a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:07.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:07.662+0000 7f125d32f640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 0x7f123c0098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:36:08.157 INFO:tasks.rados.rados.0.smithi071.stdout:961: done (3 left) 2024-09-11T01:36:08.157 INFO:tasks.rados.rados.0.smithi071.stdout:962: copy_from oid 27 from oid 16 current snap is 101 2024-09-11T01:36:08.157 INFO:tasks.rados.rados.0.smithi071.stdout:958: finishing write tid 2 to smithi07134777-21 2024-09-11T01:36:08.157 INFO:tasks.rados.rados.0.smithi071.stdout:958: finishing write tid 3 to smithi07134777-21 2024-09-11T01:36:08.158 INFO:tasks.rados.rados.0.smithi071.stdout:958: finishing write tid 4 to smithi07134777-21 2024-09-11T01:36:08.158 INFO:tasks.rados.rados.0.smithi071.stdout:958: oid 21 updating version 0 to 341 2024-09-11T01:36:08.158 INFO:tasks.rados.rados.0.smithi071.stdout:958: oid 21 version 341 is already newer than 340 2024-09-11T01:36:08.158 INFO:tasks.rados.rados.0.smithi071.stdout:958: oid 21 updating version 341 to 342 2024-09-11T01:36:08.158 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 342 (ObjNum 362 snap 101 seq_num 362) dirty exists 2024-09-11T01:36:08.158 INFO:tasks.rados.rados.0.smithi071.stdout:958: left oid 21 (ObjNum 362 snap 101 seq_num 362) 2024-09-11T01:36:08.158 INFO:tasks.rados.rados.0.smithi071.stdout:959: finishing write tid 1 to smithi07134777-25 2024-09-11T01:36:08.158 INFO:tasks.rados.rados.0.smithi071.stdout:959: finishing write tid 2 to smithi07134777-25 2024-09-11T01:36:08.158 INFO:tasks.rados.rados.0.smithi071.stdout:960: finishing rollback tid 0 to smithi07134777-13 2024-09-11T01:36:08.158 INFO:tasks.rados.rados.0.smithi071.stdout:959: finishing write tid 3 to smithi07134777-25 2024-09-11T01:36:08.158 INFO:tasks.rados.rados.0.smithi071.stdout:959: finishing write tid 4 to smithi07134777-25 2024-09-11T01:36:08.158 INFO:tasks.rados.rados.0.smithi071.stdout:959: oid 25 updating version 0 to 119 2024-09-11T01:36:08.158 INFO:tasks.rados.rados.0.smithi071.stdout:959: oid 25 version 119 is already newer than 118 2024-09-11T01:36:08.158 INFO:tasks.rados.rados.0.smithi071.stdout:959: oid 25 updating version 119 to 120 2024-09-11T01:36:08.159 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 120 (ObjNum 363 snap 101 seq_num 363) dirty exists 2024-09-11T01:36:08.159 INFO:tasks.rados.rados.0.smithi071.stdout:959: left oid 25 (ObjNum 363 snap 101 seq_num 363) 2024-09-11T01:36:08.159 INFO:tasks.rados.rados.0.smithi071.stdout:960: finishing rollback tid 1 to smithi07134777-13 2024-09-11T01:36:08.159 INFO:tasks.rados.rados.0.smithi071.stdout:960: finishing rollback tid 2 to smithi07134777-13 2024-09-11T01:36:08.159 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 406 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-11T01:36:08.159 INFO:tasks.rados.rados.0.smithi071.stdout:958: done (3 left) 2024-09-11T01:36:08.159 INFO:tasks.rados.rados.0.smithi071.stdout:959: done (2 left) 2024-09-11T01:36:08.159 INFO:tasks.rados.rados.0.smithi071.stdout:960: done (1 left) 2024-09-11T01:36:08.159 INFO:tasks.rados.rados.0.smithi071.stdout:963: read oid 44 snap -1 2024-09-11T01:36:08.159 INFO:tasks.rados.rados.0.smithi071.stdout:963: expect deleted 2024-09-11T01:36:08.159 INFO:tasks.rados.rados.0.smithi071.stdout:964: copy_from oid 23 from oid 15 current snap is 101 2024-09-11T01:36:08.159 INFO:tasks.rados.rados.0.smithi071.stdout:965: read oid 14 snap -1 2024-09-11T01:36:08.159 INFO:tasks.rados.rados.0.smithi071.stdout:965: expect (ObjNum 334 snap 96 seq_num 334) 2024-09-11T01:36:08.160 INFO:tasks.rados.rados.0.smithi071.stdout:966: delete oid 7 current snap is 101 2024-09-11T01:36:08.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.155+0000 7f125cb2e640 1 -- 172.21.15.12:0/3912751821 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2322296758 0 0) 0x7f125806eb70 con 0x7f123c003730 2024-09-11T01:36:08.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.155+0000 7f125cb2e640 10 monclient(hunting): _negotiate 2024-09-11T01:36:08.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.155+0000 7f125cb2e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:08.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.155+0000 7f125cb2e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:08.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.155+0000 7f125cb2e640 10 monclient(hunting): my global_id is 54270 2024-09-11T01:36:08.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.155+0000 7f125cb2e640 1 -- 172.21.15.12:0/3912751821 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1244003520 con 0x7f123c003730 2024-09-11T01:36:08.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 1 -- 172.21.15.12:0/3912751821 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1262720823 0 0) 0x7f1244003520 con 0x7f123c003730 2024-09-11T01:36:08.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 1 -- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 msgr2=0x7f123c0098a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:36:08.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 0x7f123c0098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 10 monclient: _finish_hunting 0 2024-09-11T01:36:08.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:36:08.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:08.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:36:08.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 1 -- 172.21.15.12:0/3912751821 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f123c00b140 con 0x7f123c003730 2024-09-11T01:36:08.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 10 monclient: _finish_auth 0 2024-09-11T01:36:08.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 10 monclient: _check_auth_tickets 2024-09-11T01:36:08.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:08.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 1 -- 172.21.15.12:0/3912751821 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f125400d680 con 0x7f123c003730 2024-09-11T01:36:08.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:08.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:36:08.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 10 monclient: dump: 2024-09-11T01:36:08.163 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:08.163 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:08.163 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.163 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.164 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:08.164 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.164 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:08.164 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:08.164 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:08.164 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:36:08.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:08.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 1 -- 172.21.15.12:0/3912751821 >> v1:172.21.15.12:6789/0 conn(0x7f123c003730 legacy=0x7f123c005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:08.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:36:08.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1244003a00 0x7f1244003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 10 monclient(hunting): picked mon.a con 0x7f1244003a00 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:08.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1244004800 0x7f1244004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 10 monclient(hunting): picked mon.b con 0x7f1244004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:08.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 0x7f124400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 10 monclient(hunting): picked mon.c con 0x7f123c0073e0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:08.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 1 -- 172.21.15.12:0/3912751821 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f12580a4900 con 0x7f1244003a00 2024-09-11T01:36:08.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 1 -- 172.21.15.12:0/3912751821 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1244005540 con 0x7f1244004800 2024-09-11T01:36:08.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 1 -- 172.21.15.12:0/3912751821 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f124400bf00 con 0x7f123c0073e0 2024-09-11T01:36:08.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:08.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125cb2e640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:36:08.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.157+0000 7f125e331640 1 -- 172.21.15.12:0/3912751821 reap_dead start 2024-09-11T01:36:08.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.158+0000 7f125e331640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 0x7f124400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.158+0000 7f125e331640 10 monclient(hunting): get_auth_request con 0x7f123c0073e0 auth_method 0 2024-09-11T01:36:08.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.158+0000 7f125e331640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:08.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.158+0000 7f125e331640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:08.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.158+0000 7f125e331640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:36:08.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.158+0000 7f125e331640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:08.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.158+0000 7f125e331640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:08.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.158+0000 7f125db30640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1244003a00 0x7f1244003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.158+0000 7f125d32f640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1244004800 0x7f1244004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.158+0000 7f125e331640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:36:08.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.158+0000 7f125e331640 10 monclient(hunting): handle_auth_done global_id 54270 payload 293 2024-09-11T01:36:08.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.159+0000 7f125e331640 1 -- 172.21.15.12:0/3912751821 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1244004800 msgr2=0x7f1244004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:08.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.159+0000 7f125e331640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1244004800 0x7f1244004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.159+0000 7f125e331640 1 -- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1244003a00 msgr2=0x7f1244003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:08.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.159+0000 7f125e331640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1244003a00 0x7f1244003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.159+0000 7f125e331640 10 monclient: _finish_hunting 0 2024-09-11T01:36:08.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.159+0000 7f125e331640 1 monclient: found mon.c 2024-09-11T01:36:08.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.159+0000 7f125e331640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:08.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.159+0000 7f125e331640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:36:08.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.159+0000 7f125e331640 1 -- 172.21.15.12:0/3912751821 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f123c00b140 con 0x7f123c0073e0 2024-09-11T01:36:08.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.159+0000 7f125e331640 10 monclient: _finish_auth 0 2024-09-11T01:36:08.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.159+0000 7f125e331640 10 monclient: _check_auth_tickets 2024-09-11T01:36:08.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.159+0000 7f125e331640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:08.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.160+0000 7f125fd94640 5 monclient: authenticate success, global_id 54270 2024-09-11T01:36:08.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.160+0000 7f125fd94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:36:08.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.160+0000 7f125d32f640 10 monclient: get_auth_request con 0x7f1244004800 auth_method 0 2024-09-11T01:36:08.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.160+0000 7f125d32f640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1244004800 0x7f1244004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:36:08.168 INFO:tasks.rados.rados.0.smithi071.stdout:965: expect (ObjNum 334 snap 96 seq_num 334) 2024-09-11T01:36:08.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.161+0000 7f125db30640 10 monclient: get_auth_request con 0x7f1244003a00 auth_method 0 2024-09-11T01:36:08.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.161+0000 7f125db30640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1244003a00 0x7f1244003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:36:08.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.163+0000 7f125e331640 1 -- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 msgr2=0x7f124400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:08.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.163+0000 7f125e331640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 0x7f124400b9c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f125806f190 tx=0x7f1244005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da34558d5aeb93b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:08.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.163+0000 7f125cb2e640 1 -- 172.21.15.12:0/3912751821 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1250039070 con 0x7f123c0073e0 2024-09-11T01:36:08.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.163+0000 7f125cb2e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:08.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.163+0000 7f125cb2e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:08.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.163+0000 7f125cb2e640 10 monclient: dump: 2024-09-11T01:36:08.172 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:08.172 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:08.172 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.172 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.172 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:08.172 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.172 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:08.172 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:08.173 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:08.173 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.163+0000 7f125cb2e640 1 -- 172.21.15.12:0/3912751821 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1250033d20 con 0x7f123c0073e0 2024-09-11T01:36:08.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.163+0000 7f125cb2e640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:08.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.163+0000 7f125cb2e640 1 -- 172.21.15.12:0/3912751821 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12500199e0 con 0x7f123c0073e0 2024-09-11T01:36:08.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.163+0000 7f125cb2e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:08.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.163+0000 7f125cb2e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:08.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.163+0000 7f125cb2e640 10 monclient: dump: 2024-09-11T01:36:08.173 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:08.173 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:08.173 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.173 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.173 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:08.174 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.174 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:08.174 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:08.174 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:08.174 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.163+0000 7f125fd94640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:08.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.163+0000 7f125fd94640 4 set_mon_vals no callback set 2024-09-11T01:36:08.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.164+0000 7f125fd94640 10 monclient: shutdown 2024-09-11T01:36:08.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.164+0000 7f125fd94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:08.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.164+0000 7f125fd94640 1 -- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 msgr2=0x7f124400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:08.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.164+0000 7f125fd94640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 0x7f124400b9c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f125806f190 tx=0x7f1244005830 comp rx=0 tx=0).stop 2024-09-11T01:36:08.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.165+0000 7f125fd94640 1 -- 172.21.15.12:0/3912751821 shutdown_connections 2024-09-11T01:36:08.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.165+0000 7f125fd94640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1244004800 0x7f1244004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.165+0000 7f125fd94640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f123c0073e0 0x7f124400b9c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.165+0000 7f125fd94640 1 --2- 172.21.15.12:0/3912751821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1244003a00 0x7f1244003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.165+0000 7f125fd94640 1 -- 172.21.15.12:0/3912751821 >> 172.21.15.12:0/3912751821 conn(0x7f12580a8490 msgr2=0x7f12580a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:08.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.165+0000 7f125fd94640 1 -- 172.21.15.12:0/3912751821 shutdown_connections 2024-09-11T01:36:08.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.165+0000 7f125fd94640 1 -- 172.21.15.12:0/3912751821 wait complete. 2024-09-11T01:36:08.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.167+0000 7f125fd94640 10 monclient: build_initial_monmap 2024-09-11T01:36:08.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.167+0000 7f125fd94640 10 monclient: monmap: 2024-09-11T01:36:08.175 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:08.175 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:08.175 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:08.175 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:08.175 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:08.175 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.175 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:08.176 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:08.176 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:08.176 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.167+0000 7f125fd94640 1 Processor -- start 2024-09-11T01:36:08.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.167+0000 7f125fd94640 1 -- start start 2024-09-11T01:36:08.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.167+0000 7f125fd94640 10 monclient: init 2024-09-11T01:36:08.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125fd94640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:08.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125fd94640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:36:08.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125fd94640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f123c0073e0 0x7f125813b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125fd94640 10 monclient(hunting): picked mon.noname-b con 0x7f123c0073e0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:08.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125fd94640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1244003a00 0x7f125813b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125fd94640 10 monclient(hunting): picked mon.noname-a con 0x7f1244003a00 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:08.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125fd94640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1244004800 0x7f125813bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125fd94640 10 monclient(hunting): picked mon.noname-c con 0x7f1244004800 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:08.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125fd94640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125fd94640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f125806dde0 con 0x7f1244003a00 2024-09-11T01:36:08.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125fd94640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125fd94640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f125806dc60 con 0x7f123c0073e0 2024-09-11T01:36:08.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125fd94640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125fd94640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f125806df60 con 0x7f1244004800 2024-09-11T01:36:08.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125fd94640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:08.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125fd94640 10 monclient(hunting): authenticate will time out at 1174.610344s 2024-09-11T01:36:08.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125d32f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1244003a00 0x7f125813b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125db30640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f123c0073e0 0x7f125813b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125db30640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f123c0073e0 0x7f125813b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:34062/0 (socket says 172.21.15.12:34062) 2024-09-11T01:36:08.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125db30640 1 -- 172.21.15.12:0/2237308904 learned_addr learned my addr 172.21.15.12:0/2237308904 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125db30640 10 monclient(hunting): get_auth_request con 0x7f123c0073e0 auth_method 0 2024-09-11T01:36:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125db30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125db30640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125db30640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125e331640 1 --2- 172.21.15.12:0/2237308904 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1244004800 0x7f125813bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125d32f640 10 monclient(hunting): get_auth_request con 0x7f1244003a00 auth_method 0 2024-09-11T01:36:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125d32f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125d32f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125d32f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125e331640 10 monclient(hunting): get_auth_request con 0x7f1244004800 auth_method 0 2024-09-11T01:36:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125e331640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125e331640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:08.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125e331640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:08.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125db30640 1 -- 172.21.15.12:0/2237308904 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f123c0073e0 msgr2=0x7f125813b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:36:08.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125db30640 1 -- 172.21.15.12:0/2237308904 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f123c0073e0 msgr2=0x7f125813b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:36:08.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125db30640 1 --2- 172.21.15.12:0/2237308904 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f123c0073e0 0x7f125813b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:08.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125d32f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:08.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125d32f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:08.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125db30640 1 --2- 172.21.15.12:0/2237308904 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f123c0073e0 0x7f125813b2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:08.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125d32f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:08.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125e331640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:08.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125e331640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:08.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.168+0000 7f125e331640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:08.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.169+0000 7f125d32f640 10 monclient(hunting): handle_auth_done global_id 54273 payload 824 2024-09-11T01:36:08.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.169+0000 7f125d32f640 1 -- 172.21.15.12:0/2237308904 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1244004800 msgr2=0x7f125813bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:08.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.169+0000 7f125d32f640 1 --2- 172.21.15.12:0/2237308904 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1244004800 0x7f125813bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.169+0000 7f125d32f640 1 -- 172.21.15.12:0/2237308904 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f123c0073e0 msgr2=0x7f125813b2e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:36:08.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.169+0000 7f125d32f640 1 --2- 172.21.15.12:0/2237308904 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f123c0073e0 0x7f125813b2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.169+0000 7f125d32f640 10 monclient: _finish_hunting 0 2024-09-11T01:36:08.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.169+0000 7f125d32f640 1 monclient: found mon.noname-a 2024-09-11T01:36:08.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.169+0000 7f125d32f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:08.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.169+0000 7f125d32f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:08.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.169+0000 7f125d32f640 1 -- 172.21.15.12:0/2237308904 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12581204c0 con 0x7f1244003a00 2024-09-11T01:36:08.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.169+0000 7f125e331640 1 --2- 172.21.15.12:0/2237308904 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1244004800 0x7f125813bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:36:08.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.171+0000 7f125d32f640 1 -- 172.21.15.12:0/2237308904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1244003a00 msgr2=0x7f125813b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:08.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.171+0000 7f125d32f640 1 --2- 172.21.15.12:0/2237308904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1244003a00 0x7f125813b820 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1248005de0 tx=0x7f1248001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:08.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.171+0000 7f124e7fc640 1 -- 172.21.15.12:0/2237308904 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1248001d40 con 0x7f1244003a00 2024-09-11T01:36:08.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.171+0000 7f124e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:08.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.171+0000 7f124e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:08.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.171+0000 7f124e7fc640 10 monclient: dump: 2024-09-11T01:36:08.183 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:08.183 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:08.183 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.183 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.183 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:08.183 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.183 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:08.183 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:08.183 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:08.183 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.171+0000 7f124e7fc640 10 monclient: _finish_auth 0 2024-09-11T01:36:08.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.171+0000 7f124e7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:36:08.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.171+0000 7f124e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:08.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.171+0000 7f124e7fc640 1 -- 172.21.15.12:0/2237308904 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1248014ad0 con 0x7f1244003a00 2024-09-11T01:36:08.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.171+0000 7f125fd94640 5 monclient: authenticate success, global_id 54273 2024-09-11T01:36:08.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.171+0000 7f125fd94640 10 monclient: _renew_subs 2024-09-11T01:36:08.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.171+0000 7f125fd94640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:08.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.171+0000 7f125fd94640 1 -- 172.21.15.12:0/2237308904 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12581207a0 con 0x7f1244003a00 2024-09-11T01:36:08.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.171+0000 7f124e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:08.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.172+0000 7f124e7fc640 1 -- 172.21.15.12:0/2237308904 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1248014ea0 con 0x7f1244003a00 2024-09-11T01:36:08.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.172+0000 7f124e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:08.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.172+0000 7f124e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:08.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.172+0000 7f124e7fc640 10 monclient: dump: 2024-09-11T01:36:08.184 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:08.184 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:08.184 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.185 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.185 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:08.185 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.185 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:08.185 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:08.185 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:08.185 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.172+0000 7f124ffff640 4 set_mon_vals no callback set 2024-09-11T01:36:08.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.172+0000 7f125fd94640 10 monclient: _renew_subs 2024-09-11T01:36:08.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.172+0000 7f125fd94640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:08.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.172+0000 7f125fd94640 1 -- 172.21.15.12:0/2237308904 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f125811f930 con 0x7f1244003a00 2024-09-11T01:36:08.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.173+0000 7f124e7fc640 1 -- 172.21.15.12:0/2237308904 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f1248017460 con 0x7f1244003a00 2024-09-11T01:36:08.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.173+0000 7f125fd94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:08.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.173+0000 7f125fd94640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:08.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.173+0000 7f125fd94640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:08.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.173+0000 7f125fd94640 1 -- 172.21.15.12:0/2237308904 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12580698e0 con 0x7f1244003a00 2024-09-11T01:36:08.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.173+0000 7f124e7fc640 1 --2- 172.21.15.12:0/2237308904 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f123c03eeb0 0x7f123c041370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.173+0000 7f124e7fc640 1 -- 172.21.15.12:0/2237308904 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(223..223 src has 186..223) ==== 4799+0+0 (secure 0 0 0) 0x7f12480537a0 con 0x7f1244003a00 2024-09-11T01:36:08.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.174+0000 7f125db30640 1 --2- 172.21.15.12:0/2237308904 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f123c03eeb0 0x7f123c041370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.174+0000 7f125db30640 10 monclient: get_auth_request con 0x7f123c03eeb0 auth_method 0 2024-09-11T01:36:08.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.176+0000 7f125db30640 1 --2- 172.21.15.12:0/2237308904 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f123c03eeb0 0x7f123c041370 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1254006e80 tx=0x7f1254008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:08.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.181+0000 7f124e7fc640 1 -- 172.21.15.12:0/2237308904 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f124801b020 con 0x7f1244003a00 2024-09-11T01:36:08.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.181+0000 7f124e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:08.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.181+0000 7f124e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:08.197 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2993977 2024-09-11T01:36:08.292 INFO:tasks.rados.rados.0.smithi071.stdout:962: finishing copy_from racing read to smithi07134777-27 2024-09-11T01:36:08.293 INFO:tasks.rados.rados.0.smithi071.stdout:962: finishing copy_from to smithi07134777-27 2024-09-11T01:36:08.293 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 338 (ObjNum 240 snap 58 seq_num 240) dirty exists 2024-09-11T01:36:08.293 INFO:tasks.rados.rados.0.smithi071.stdout:964: finishing copy_from racing read to smithi07134777-23 2024-09-11T01:36:08.293 INFO:tasks.rados.rados.0.smithi071.stdout:964: finishing copy_from to smithi07134777-23 2024-09-11T01:36:08.293 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 244 (ObjNum 323 snap 92 seq_num 323) dirty exists 2024-09-11T01:36:08.293 INFO:tasks.rados.rados.0.smithi071.stdout:962: done (4 left) 2024-09-11T01:36:08.293 INFO:tasks.rados.rados.0.smithi071.stdout:963: done (3 left) 2024-09-11T01:36:08.293 INFO:tasks.rados.rados.0.smithi071.stdout:964: done (2 left) 2024-09-11T01:36:08.293 INFO:tasks.rados.rados.0.smithi071.stdout:965: done (1 left) 2024-09-11T01:36:08.293 INFO:tasks.rados.rados.0.smithi071.stdout:966: done (0 left) 2024-09-11T01:36:08.293 INFO:tasks.rados.rados.0.smithi071.stdout:967: write (excl) oid 31 current snap is 101 2024-09-11T01:36:08.293 INFO:tasks.rados.rados.0.smithi071.stdout:967: seq_num 365 ranges {534555=650829,1659104=475451} 2024-09-11T01:36:08.298 INFO:tasks.rados.rados.0.smithi071.stdout:967: writing smithi07134777-31 from 534555 to 1185384 tid 1 2024-09-11T01:36:08.305 INFO:tasks.rados.rados.0.smithi071.stdout:967: writing smithi07134777-31 from 1659104 to 2134555 tid 2 2024-09-11T01:36:08.305 INFO:tasks.rados.rados.0.smithi071.stdout:968: read oid 1 snap -1 2024-09-11T01:36:08.305 INFO:tasks.rados.rados.0.smithi071.stdout:968: expect deleted 2024-09-11T01:36:08.305 INFO:tasks.rados.rados.0.smithi071.stdout:969: rollback oid 13 current snap is 101 2024-09-11T01:36:08.305 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 13 to 98 2024-09-11T01:36:08.305 INFO:tasks.rados.rados.0.smithi071.stdout:967: finishing write tid 1 to smithi07134777-31 2024-09-11T01:36:08.305 INFO:tasks.rados.rados.0.smithi071.stdout:970: copy_from oid 26 from oid 22 current snap is 101 2024-09-11T01:36:08.305 INFO:tasks.rados.rados.0.smithi071.stdout:971: read oid 6 snap -1 2024-09-11T01:36:08.305 INFO:tasks.rados.rados.0.smithi071.stdout:971: expect deleted 2024-09-11T01:36:08.305 INFO:tasks.rados.rados.0.smithi071.stdout:972: copy_from oid 33 from oid 30 current snap is 101 2024-09-11T01:36:08.306 INFO:tasks.rados.rados.0.smithi071.stdout:973: delete oid 18 current snap is 101 2024-09-11T01:36:08.306 INFO:tasks.rados.rados.0.smithi071.stdout:972: finishing copy_from to smithi07134777-33 2024-09-11T01:36:08.306 INFO:tasks.rados.rados.0.smithi071.stdout:972: got expected ENOENT (src dne) 2024-09-11T01:36:08.306 INFO:tasks.rados.rados.0.smithi071.stdout:972: finishing copy_from racing read to smithi07134777-33 2024-09-11T01:36:08.306 INFO:tasks.rados.rados.0.smithi071.stdout:971: done (6 left) 2024-09-11T01:36:08.307 INFO:tasks.rados.rados.0.smithi071.stdout:972: done (5 left) 2024-09-11T01:36:08.307 INFO:tasks.rados.rados.0.smithi071.stdout:973: done (4 left) 2024-09-11T01:36:08.307 INFO:tasks.rados.rados.0.smithi071.stdout:974: snap_create 2024-09-11T01:36:08.309 INFO:tasks.rados.rados.0.smithi071.stdout:967: finishing write tid 2 to smithi07134777-31 2024-09-11T01:36:08.310 INFO:tasks.rados.rados.0.smithi071.stdout:967: finishing write tid 3 to smithi07134777-31 2024-09-11T01:36:08.310 INFO:tasks.rados.rados.0.smithi071.stdout:967: finishing write tid 4 to smithi07134777-31 2024-09-11T01:36:08.310 INFO:tasks.rados.rados.0.smithi071.stdout:967: oid 31 updating version 0 to 407 2024-09-11T01:36:08.310 INFO:tasks.rados.rados.0.smithi071.stdout:967: oid 31 updating version 407 to 409 2024-09-11T01:36:08.310 INFO:tasks.rados.rados.0.smithi071.stdout:967: oid 31 version 409 is already newer than 408 2024-09-11T01:36:08.310 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 409 (ObjNum 365 snap 101 seq_num 365) dirty exists 2024-09-11T01:36:08.310 INFO:tasks.rados.rados.0.smithi071.stdout:967: left oid 31 (ObjNum 365 snap 101 seq_num 365) 2024-09-11T01:36:08.312 INFO:tasks.rados.rados.0.smithi071.stdout:969: finishing rollback tid 0 to smithi07134777-13 2024-09-11T01:36:08.312 INFO:tasks.rados.rados.0.smithi071.stdout:969: finishing rollback tid 1 to smithi07134777-13 2024-09-11T01:36:08.312 INFO:tasks.rados.rados.0.smithi071.stdout:969: finishing rollback tid 2 to smithi07134777-13 2024-09-11T01:36:08.312 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 412 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-11T01:36:08.317 INFO:tasks.rados.rados.0.smithi071.stdout:970: finishing copy_from racing read to smithi07134777-26 2024-09-11T01:36:08.322 INFO:tasks.rados.rados.0.smithi071.stdout:970: finishing copy_from to smithi07134777-26 2024-09-11T01:36:08.322 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 237 (ObjNum 347 snap 98 seq_num 347) dirty exists 2024-09-11T01:36:08.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.445+0000 7f125fd94640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:36:08.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.445+0000 7f125fd94640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:36:08.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.445+0000 7f125fd94640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:08.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.445+0000 7f125fd94640 1 -- 172.21.15.12:0/2237308904 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1258120bc0 con 0x7f1244003a00 2024-09-11T01:36:08.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.446+0000 7f124e7fc640 1 -- 172.21.15.12:0/2237308904 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f1248052190 con 0x7f1244003a00 2024-09-11T01:36:08.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.446+0000 7f124e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:36:08.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.446+0000 7f124e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:08.448 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":74,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:36:08.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.448+0000 7f125fd94640 1 -- 172.21.15.12:0/2237308904 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f123c03eeb0 msgr2=0x7f123c041370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:08.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.448+0000 7f125fd94640 1 --2- 172.21.15.12:0/2237308904 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f123c03eeb0 0x7f123c041370 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1254006e80 tx=0x7f1254008040 comp rx=0 tx=0).stop 2024-09-11T01:36:08.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.448+0000 7f125fd94640 10 monclient: shutdown 2024-09-11T01:36:08.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.448+0000 7f125fd94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:08.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.448+0000 7f125fd94640 1 -- 172.21.15.12:0/2237308904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1244003a00 msgr2=0x7f125813b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:08.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.448+0000 7f125fd94640 1 --2- 172.21.15.12:0/2237308904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1244003a00 0x7f125813b820 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1248005de0 tx=0x7f1248001a00 comp rx=0 tx=0).stop 2024-09-11T01:36:08.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.448+0000 7f125fd94640 1 -- 172.21.15.12:0/2237308904 shutdown_connections 2024-09-11T01:36:08.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.448+0000 7f125fd94640 1 --2- 172.21.15.12:0/2237308904 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1244004800 0x7f125813bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.448+0000 7f125fd94640 1 --2- 172.21.15.12:0/2237308904 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f123c03eeb0 0x7f123c041370 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.448+0000 7f125fd94640 1 --2- 172.21.15.12:0/2237308904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1244003a00 0x7f125813b820 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.448+0000 7f125fd94640 1 --2- 172.21.15.12:0/2237308904 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f123c0073e0 0x7f125813b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.448+0000 7f125fd94640 1 -- 172.21.15.12:0/2237308904 >> 172.21.15.12:0/2237308904 conn(0x7f12580a8490 msgr2=0x7f125809e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:08.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.448+0000 7f125fd94640 1 -- 172.21.15.12:0/2237308904 shutdown_connections 2024-09-11T01:36:08.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.448+0000 7f125fd94640 1 -- 172.21.15.12:0/2237308904 wait complete. 2024-09-11T01:36:08.460 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:36:08.460 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:36:08.460 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:36:08.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.560+0000 7ffb6c5f3640 10 monclient: get_monmap_and_config 2024-09-11T01:36:08.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.560+0000 7ffb6c5f3640 10 monclient: build_initial_monmap 2024-09-11T01:36:08.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.560+0000 7ffb6c5f3640 10 monclient: monmap: 2024-09-11T01:36:08.562 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:08.562 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:08.563 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:08.561532+0000 2024-09-11T01:36:08.563 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:08.561532+0000 2024-09-11T01:36:08.563 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:08.563 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.563 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:08.563 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:08.563 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:08.563 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:08.563 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:08.563 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.561+0000 7ffb6c5f3640 1 Processor -- start 2024-09-11T01:36:08.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.561+0000 7ffb6c5f3640 1 -- start start 2024-09-11T01:36:08.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.561+0000 7ffb6c5f3640 10 monclient: init 2024-09-11T01:36:08.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6c5f3640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6c5f3640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-11T01:36:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6c5f3640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb640a2d40 0x7ffb6406cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6c5f3640 10 monclient(hunting): picked mon.noname-b con 0x7ffb640a2d40 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6c5f3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffb6406d090 addr v1:172.21.15.71:6789/0 2024-09-11T01:36:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6c5f3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffb6406d470 addr v1:172.21.15.12:6789/0 2024-09-11T01:36:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6c5f3640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb6409aed0 con 0x7ffb6406d470 2024-09-11T01:36:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6c5f3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb640a3920 con 0x7ffb6406d090 2024-09-11T01:36:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6c5f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6c5f3640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ffb6406dd10 con 0x7ffb640a2d40 2024-09-11T01:36:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6c5f3640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6c5f3640 10 monclient(hunting): authenticate will time out at 1175.004131s 2024-09-11T01:36:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6a38f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb640a2d40 0x7ffb6406cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6ab90640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ffb6406d470 0x7ffb6413fc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42100/0 (socket says 172.21.15.12:42100) 2024-09-11T01:36:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6ab90640 1 -- 172.21.15.12:0/432693169 learned_addr learned my addr 172.21.15.12:0/432693169 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:36:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6a38f640 10 monclient(hunting): get_auth_request con 0x7ffb640a2d40 auth_method 0 2024-09-11T01:36:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6a38f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6a38f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6a38f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6a38f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6a38f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6a38f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6a38f640 10 monclient(hunting): handle_auth_done global_id 64175 payload 274 2024-09-11T01:36:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6a38f640 1 -- 172.21.15.12:0/432693169 >> v1:172.21.15.71:6789/0 conn(0x7ffb6406d090 legacy=0x7ffb6413c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6a38f640 1 -- 172.21.15.12:0/432693169 >> v1:172.21.15.12:6789/0 conn(0x7ffb6406d470 legacy=0x7ffb6413fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6a38f640 10 monclient: _finish_hunting 0 2024-09-11T01:36:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6a38f640 1 monclient: found mon.noname-b 2024-09-11T01:36:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6a38f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:08.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6a38f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:36:08.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.562+0000 7ffb6a38f640 1 -- 172.21.15.12:0/432693169 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb640a3320 con 0x7ffb640a2d40 2024-09-11T01:36:08.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.565+0000 7ffb6a38f640 1 -- 172.21.15.12:0/432693169 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb640a2d40 msgr2=0x7ffb6406cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:08.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.565+0000 7ffb6a38f640 1 --2- 172.21.15.12:0/432693169 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb640a2d40 0x7ffb6406cb50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ffb6406dc30 tx=0x7ffb60029f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f84c7d326a30d8ed server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:08.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.565+0000 7ffb6938d640 1 -- 172.21.15.12:0/432693169 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb60037070 con 0x7ffb640a2d40 2024-09-11T01:36:08.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.565+0000 7ffb6938d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:08.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.565+0000 7ffb6938d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:36:08.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.565+0000 7ffb6938d640 10 monclient: dump: 2024-09-11T01:36:08.574 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:08.574 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:08.574 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.574 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.574 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:08.575 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.575 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:08.575 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:08.575 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:08.575 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.565+0000 7ffb6938d640 10 monclient: _finish_auth 0 2024-09-11T01:36:08.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.565+0000 7ffb6938d640 10 monclient: _check_auth_tickets 2024-09-11T01:36:08.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.565+0000 7ffb6938d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:08.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.565+0000 7ffb6938d640 1 -- 172.21.15.12:0/432693169 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb60032440 con 0x7ffb640a2d40 2024-09-11T01:36:08.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.565+0000 7ffb6c5f3640 5 monclient: authenticate success, global_id 64175 2024-09-11T01:36:08.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.565+0000 7ffb6c5f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:36:08.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.565+0000 7ffb6938d640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:08.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.565+0000 7ffb6938d640 1 -- 172.21.15.12:0/432693169 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb60032810 con 0x7ffb640a2d40 2024-09-11T01:36:08.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.565+0000 7ffb6c5f3640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:08.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.565+0000 7ffb6c5f3640 4 set_mon_vals no callback set 2024-09-11T01:36:08.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.565+0000 7ffb6938d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:08.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.565+0000 7ffb6938d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:08.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.565+0000 7ffb6938d640 10 monclient: dump: 2024-09-11T01:36:08.576 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:08.576 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:08.576 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.576 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.576 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:08.576 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.576 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:08.576 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:08.576 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:08.577 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.566+0000 7ffb6c5f3640 10 monclient: shutdown 2024-09-11T01:36:08.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.566+0000 7ffb6c5f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:08.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.566+0000 7ffb6c5f3640 1 -- 172.21.15.12:0/432693169 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb640a2d40 msgr2=0x7ffb6406cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:08.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.566+0000 7ffb6c5f3640 1 --2- 172.21.15.12:0/432693169 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb640a2d40 0x7ffb6406cb50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ffb6406dc30 tx=0x7ffb60029f80 comp rx=0 tx=0).stop 2024-09-11T01:36:08.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.566+0000 7ffb6c5f3640 1 -- 172.21.15.12:0/432693169 shutdown_connections 2024-09-11T01:36:08.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.566+0000 7ffb6c5f3640 1 --2- 172.21.15.12:0/432693169 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb640a2d40 0x7ffb6406cb50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.566+0000 7ffb6c5f3640 1 -- 172.21.15.12:0/432693169 >> 172.21.15.12:0/432693169 conn(0x7ffb640a3e30 msgr2=0x7ffb640a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:08.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.566+0000 7ffb6c5f3640 1 -- 172.21.15.12:0/432693169 shutdown_connections 2024-09-11T01:36:08.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.566+0000 7ffb6c5f3640 1 -- 172.21.15.12:0/432693169 wait complete. 2024-09-11T01:36:08.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.566+0000 7ffb6c5f3640 10 monclient: build_initial_monmap 2024-09-11T01:36:08.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.566+0000 7ffb6c5f3640 10 monclient: monmap: 2024-09-11T01:36:08.577 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:08.577 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:08.578 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:08.578 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:08.578 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:08.578 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.578 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:08.578 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:08.578 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:08.578 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6c5f3640 1 Processor -- start 2024-09-11T01:36:08.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6c5f3640 1 -- start start 2024-09-11T01:36:08.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6c5f3640 10 monclient: init 2024-09-11T01:36:08.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6c5f3640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:08.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6c5f3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:36:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6c5f3640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb6406d090 0x7ffb64140af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6c5f3640 10 monclient(hunting): picked mon.noname-c con 0x7ffb6406d090 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6c5f3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb6406d470 0x7ffb64093da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6c5f3640 10 monclient(hunting): picked mon.noname-a con 0x7ffb6406d470 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6c5f3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffb640a2d40 0x7ffb640942e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6c5f3640 10 monclient(hunting): picked mon.noname-b con 0x7ffb640a2d40 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6c5f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6c5f3640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6409ba10 con 0x7ffb6406d470 2024-09-11T01:36:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6c5f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6c5f3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ffb640a3120 con 0x7ffb640a2d40 2024-09-11T01:36:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6c5f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6c5f3640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ffb6409b3a0 con 0x7ffb6406d090 2024-09-11T01:36:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6c5f3640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6c5f3640 10 monclient(hunting): authenticate will time out at 1175.009547s 2024-09-11T01:36:08.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6a38f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb6406d090 0x7ffb64140af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6a38f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb6406d090 0x7ffb64140af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:49446/0 (socket says 172.21.15.12:49446) 2024-09-11T01:36:08.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6a38f640 1 -- 172.21.15.12:0/85373501 learned_addr learned my addr 172.21.15.12:0/85373501 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:08.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6a38f640 10 monclient(hunting): get_auth_request con 0x7ffb6406d090 auth_method 0 2024-09-11T01:36:08.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6a38f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:08.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6a38f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:08.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6a38f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:08.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb69b8e640 1 --2- 172.21.15.12:0/85373501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb6406d470 0x7ffb64093da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb69b8e640 10 monclient(hunting): get_auth_request con 0x7ffb6406d470 auth_method 0 2024-09-11T01:36:08.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb69b8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:08.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb69b8e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:08.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb69b8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:08.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6a38f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:08.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6a38f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:08.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6a38f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:08.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb69b8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:08.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb69b8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:08.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb69b8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:08.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6ab90640 1 --2- 172.21.15.12:0/85373501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffb640a2d40 0x7ffb640942e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6ab90640 10 monclient(hunting): get_auth_request con 0x7ffb640a2d40 auth_method 0 2024-09-11T01:36:08.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6ab90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:08.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6ab90640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:08.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6ab90640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:08.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6a38f640 10 monclient(hunting): handle_auth_done global_id 64178 payload 824 2024-09-11T01:36:08.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6a38f640 1 -- 172.21.15.12:0/85373501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffb640a2d40 msgr2=0x7ffb640942e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:08.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6ab90640 1 -- 172.21.15.12:0/85373501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffb640a2d40 msgr2=0x7ffb640942e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:36:08.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6ab90640 1 -- 172.21.15.12:0/85373501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffb640a2d40 msgr2=0x7ffb640942e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:36:08.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6ab90640 1 --2- 172.21.15.12:0/85373501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffb640a2d40 0x7ffb640942e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:08.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.567+0000 7ffb6ab90640 1 --2- 172.21.15.12:0/85373501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffb640a2d40 0x7ffb640942e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:08.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.568+0000 7ffb6a38f640 1 --2- 172.21.15.12:0/85373501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffb640a2d40 0x7ffb640942e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.568+0000 7ffb6a38f640 1 -- 172.21.15.12:0/85373501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb6406d470 msgr2=0x7ffb64093da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:08.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.568+0000 7ffb6a38f640 1 --2- 172.21.15.12:0/85373501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb6406d470 0x7ffb64093da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.568+0000 7ffb6a38f640 10 monclient: _finish_hunting 0 2024-09-11T01:36:08.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.568+0000 7ffb6a38f640 1 monclient: found mon.noname-c 2024-09-11T01:36:08.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.568+0000 7ffb6a38f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:08.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.568+0000 7ffb6a38f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:36:08.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.568+0000 7ffb6a38f640 1 -- 172.21.15.12:0/85373501 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb64097b80 con 0x7ffb6406d090 2024-09-11T01:36:08.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.568+0000 7ffb69b8e640 1 --2- 172.21.15.12:0/85373501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb6406d470 0x7ffb64093da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:36:08.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.570+0000 7ffb6a38f640 1 -- 172.21.15.12:0/85373501 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb6406d090 msgr2=0x7ffb64140af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:08.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.570+0000 7ffb6a38f640 1 --2- 172.21.15.12:0/85373501 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb6406d090 0x7ffb64140af0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ffb60035040 tx=0x7ffb600046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:08.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.570+0000 7ffb5affd640 1 -- 172.21.15.12:0/85373501 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb60037050 con 0x7ffb6406d090 2024-09-11T01:36:08.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.570+0000 7ffb5affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:08.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.570+0000 7ffb5affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:36:08.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.570+0000 7ffb5affd640 10 monclient: dump: 2024-09-11T01:36:08.584 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:08.584 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:08.584 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.584 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.584 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:08.584 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.584 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:08.584 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:08.584 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:08.584 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.570+0000 7ffb5affd640 10 monclient: _finish_auth 0 2024-09-11T01:36:08.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.570+0000 7ffb5affd640 10 monclient: _check_auth_tickets 2024-09-11T01:36:08.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.570+0000 7ffb5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:08.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.570+0000 7ffb5affd640 1 -- 172.21.15.12:0/85373501 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb60043070 con 0x7ffb6406d090 2024-09-11T01:36:08.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.570+0000 7ffb5affd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:08.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.570+0000 7ffb5affd640 1 -- 172.21.15.12:0/85373501 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb6003cd60 con 0x7ffb6406d090 2024-09-11T01:36:08.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.570+0000 7ffb6c5f3640 5 monclient: authenticate success, global_id 64178 2024-09-11T01:36:08.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.570+0000 7ffb6c5f3640 10 monclient: _renew_subs 2024-09-11T01:36:08.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.570+0000 7ffb6c5f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:36:08.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.570+0000 7ffb6c5f3640 1 -- 172.21.15.12:0/85373501 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb6409e520 con 0x7ffb6406d090 2024-09-11T01:36:08.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.570+0000 7ffb68b8c640 4 set_mon_vals no callback set 2024-09-11T01:36:08.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.570+0000 7ffb5affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:08.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.570+0000 7ffb5affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:08.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.570+0000 7ffb5affd640 10 monclient: dump: 2024-09-11T01:36:08.585 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:08.586 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:08.586 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.586 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.586 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:08.586 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.586 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:08.586 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:08.586 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:08.586 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.570+0000 7ffb6c5f3640 10 monclient: _renew_subs 2024-09-11T01:36:08.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.570+0000 7ffb6c5f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:36:08.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.570+0000 7ffb6c5f3640 1 -- 172.21.15.12:0/85373501 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb64095820 con 0x7ffb6406d090 2024-09-11T01:36:08.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.571+0000 7ffb6c5f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:08.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.571+0000 7ffb6c5f3640 1 --2- 172.21.15.12:0/85373501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb64094820 0x7ffb64094c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.571+0000 7ffb6c5f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.571+0000 7ffb6c5f3640 1 -- 172.21.15.12:0/85373501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6406cbd0 con 0x7ffb64094820 2024-09-11T01:36:08.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.571+0000 7ffb69b8e640 1 --2- 172.21.15.12:0/85373501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb64094820 0x7ffb64094c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.571+0000 7ffb69b8e640 10 monclient: get_auth_request con 0x7ffb64094820 auth_method 0 2024-09-11T01:36:08.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.571+0000 7ffb69b8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:08.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.571+0000 7ffb69b8e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:08.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.571+0000 7ffb69b8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:08.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.571+0000 7ffb69b8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:08.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.571+0000 7ffb69b8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:08.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.571+0000 7ffb69b8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:08.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.571+0000 7ffb5affd640 1 -- 172.21.15.12:0/85373501 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ffb60048460 con 0x7ffb6406d090 2024-09-11T01:36:08.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.571+0000 7ffb69b8e640 10 monclient(hunting): handle_auth_done global_id 54279 payload 824 2024-09-11T01:36:08.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.571+0000 7ffb69b8e640 1 -- 172.21.15.12:0/85373501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffb64095140 con 0x7ffb64094820 2024-09-11T01:36:08.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.571+0000 7ffb5affd640 1 --2- 172.21.15.12:0/85373501 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffb4803e260 0x7ffb48040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.571+0000 7ffb5affd640 1 -- 172.21.15.12:0/85373501 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(223..223 src has 186..223) ==== 4799+0+0 (secure 0 0 0) 0x7ffb60073910 con 0x7ffb6406d090 2024-09-11T01:36:08.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.572+0000 7ffb6ab90640 1 --2- 172.21.15.12:0/85373501 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffb4803e260 0x7ffb48040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.572+0000 7ffb6ab90640 10 monclient: get_auth_request con 0x7ffb4803e260 auth_method 0 2024-09-11T01:36:08.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.573+0000 7ffb69b8e640 1 -- 172.21.15.12:0/85373501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb64094820 msgr2=0x7ffb64094c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:08.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.573+0000 7ffb69b8e640 1 --2- 172.21.15.12:0/85373501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb64094820 0x7ffb64094c00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ffb5400fae0 tx=0x7ffb540044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:08.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.574+0000 7ffb5affd640 1 -- 172.21.15.12:0/85373501 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb54018020 con 0x7ffb64094820 2024-09-11T01:36:08.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.574+0000 7ffb5affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:08.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.574+0000 7ffb5affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:08.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.574+0000 7ffb5affd640 10 monclient: dump: 2024-09-11T01:36:08.591 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:08.591 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:08.591 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.591 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.591 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:08.591 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.591 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:08.591 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:08.591 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:08.591 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.574+0000 7ffb5affd640 1 -- 172.21.15.12:0/85373501 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ffb5400dd30 con 0x7ffb64094820 2024-09-11T01:36:08.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.574+0000 7ffb5affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:08.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.574+0000 7ffb5affd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:08.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.574+0000 7ffb5affd640 1 -- 172.21.15.12:0/85373501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb64094820 msgr2=0x7ffb64094c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:08.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.574+0000 7ffb5affd640 1 --2- 172.21.15.12:0/85373501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb64094820 0x7ffb64094c00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ffb5400fae0 tx=0x7ffb540044c0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.574+0000 7ffb5affd640 1 -- 172.21.15.12:0/85373501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb64094820 msgr2=0x7ffb64094c00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:08.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.574+0000 7ffb5affd640 1 --2- 172.21.15.12:0/85373501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb64094820 0x7ffb64094c00 secure :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ffb5400fae0 tx=0x7ffb540044c0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.574+0000 7ffb6ab90640 1 --2- 172.21.15.12:0/85373501 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffb4803e260 0x7ffb48040720 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ffb64096110 tx=0x7ffb5c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:08.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.580+0000 7ffb6c5f3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:36:08.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.580+0000 7ffb6c5f3640 1 --2- 172.21.15.12:0/85373501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb640913c0 0x7ffb640917d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.580+0000 7ffb6c5f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.580+0000 7ffb6c5f3640 1 -- 172.21.15.12:0/85373501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ffb640973e0 con 0x7ffb640913c0 2024-09-11T01:36:08.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.580+0000 7ffb69b8e640 1 --2- 172.21.15.12:0/85373501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb640913c0 0x7ffb640917d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.580+0000 7ffb69b8e640 10 monclient: get_auth_request con 0x7ffb640913c0 auth_method 0 2024-09-11T01:36:08.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.580+0000 7ffb69b8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.580+0000 7ffb69b8e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.580+0000 7ffb69b8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.580+0000 7ffb69b8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.580+0000 7ffb69b8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.580+0000 7ffb69b8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.580+0000 7ffb69b8e640 10 monclient(hunting): handle_auth_done global_id 54282 payload 824 2024-09-11T01:36:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.580+0000 7ffb69b8e640 1 -- 172.21.15.12:0/85373501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ffb64096380 con 0x7ffb640913c0 2024-09-11T01:36:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.582+0000 7ffb69b8e640 1 -- 172.21.15.12:0/85373501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb640913c0 msgr2=0x7ffb640917d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.582+0000 7ffb69b8e640 1 --2- 172.21.15.12:0/85373501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb640913c0 0x7ffb640917d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ffb54019000 tx=0x7ffb540036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.583+0000 7ffb5affd640 1 -- 172.21.15.12:0/85373501 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb54018030 con 0x7ffb640913c0 2024-09-11T01:36:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.583+0000 7ffb5affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.583+0000 7ffb5affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:08.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.583+0000 7ffb5affd640 10 monclient: dump: 2024-09-11T01:36:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:08.594 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.583+0000 7ffb5affd640 1 -- 172.21.15.12:0/85373501 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4710 (secure 0 0 0) 0x7ffb54014b80 con 0x7ffb640913c0 2024-09-11T01:36:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.583+0000 7ffb5affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:36:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.583+0000 7ffb5affd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:08.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.583+0000 7ffb5affd640 1 -- 172.21.15.12:0/85373501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb640913c0 msgr2=0x7ffb640917d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:08.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.583+0000 7ffb5affd640 1 --2- 172.21.15.12:0/85373501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb640913c0 0x7ffb640917d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ffb54019000 tx=0x7ffb540036f0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.583+0000 7ffb5affd640 1 -- 172.21.15.12:0/85373501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb640913c0 msgr2=0x7ffb640917d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:08.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.583+0000 7ffb5affd640 1 --2- 172.21.15.12:0/85373501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb640913c0 0x7ffb640917d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.595 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:36:08.595 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:36:08.595 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:36:08.595 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:36:08.595 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 74, 2024-09-11T01:36:08.595 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 158564, 2024-09-11T01:36:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:36:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:36:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:36:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:36:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:36:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:36:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:36:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:36:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:36:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:08.598 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:08.598 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:08.598 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:08.598 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:08.598 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:08.598 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:08.598 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:08.598 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:08.598 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:08.598 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.599 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:36:08.599 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:36:08.599 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:36:08.599 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:36:08.599 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:36:08.599 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:36:08.599 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:08.599 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:08.599 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:36:08.599 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:36:08.599 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:36:08.600 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:36:08.600 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:36:08.600 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:36:08.600 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:36:08.600 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:08.600 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:36:08.600 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:08.600 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:08.600 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:08.600 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:08.600 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:08.600 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:08.601 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:08.601 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:08.601 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:08.601 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:08.601 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:08.601 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:08.601 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:36:08.601 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.601 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:36:08.601 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.601 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:36:08.601 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:36:08.602 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:08.602 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:08.602 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.602 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:08.602 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:36:08.602 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:08.602 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.602 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.602 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:08.602 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:36:08.602 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:08.602 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:08.602 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:08.602 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.603 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:36:08.603 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:36:08.603 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:08.603 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:08.603 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:08.603 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.603 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.603 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:36:08.603 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:36:08.603 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:08.603 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:08.603 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.603 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:08.604 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:36:08.604 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:08.604 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.604 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.604 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:08.604 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:36:08.604 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:08.604 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:08.604 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:08.604 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.604 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:36:08.604 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:36:08.605 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:08.605 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:08.605 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:08.605 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.605 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.605 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:36:08.605 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:36:08.605 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:08.605 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:08.605 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.605 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:08.605 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:36:08.605 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:08.605 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.606 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.606 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:08.606 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:36:08.606 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:08.606 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:08.606 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:08.606 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.606 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:36:08.606 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:36:08.606 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:08.606 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:08.606 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:08.606 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:08.607 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:08.607 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.607 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:36:08.607 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:36:08.607 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.607 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:08.607 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:08.607 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:36:08.607 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:08.607 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:08.607 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:36:08.607 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.607 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:08.608 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:08.608 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:36:08.608 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:08.608 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:08.608 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:36:08.608 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.608 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:08.608 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:08.608 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:36:08.608 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:08.608 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:08.608 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.608 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:36:08.609 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:36:08.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.583+0000 7ffb6c5f3640 1 -- 172.21.15.12:0/85373501 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffb4803e260 msgr2=0x7ffb48040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:08.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.583+0000 7ffb6c5f3640 1 --2- 172.21.15.12:0/85373501 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffb4803e260 0x7ffb48040720 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ffb64096110 tx=0x7ffb5c009040 comp rx=0 tx=0).stop 2024-09-11T01:36:08.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.583+0000 7ffb6c5f3640 10 monclient: shutdown 2024-09-11T01:36:08.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.583+0000 7ffb6c5f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:08.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.583+0000 7ffb6c5f3640 1 -- 172.21.15.12:0/85373501 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb6406d090 msgr2=0x7ffb64140af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:08.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.583+0000 7ffb6c5f3640 1 --2- 172.21.15.12:0/85373501 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb6406d090 0x7ffb64140af0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ffb60035040 tx=0x7ffb600046c0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.583+0000 7ffb6ab90640 1 -- 172.21.15.12:0/85373501 reap_dead start 2024-09-11T01:36:08.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.583+0000 7ffb6ab90640 1 -- 172.21.15.12:0/85373501 reap_dead start 2024-09-11T01:36:08.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.583+0000 7ffb6c5f3640 1 -- 172.21.15.12:0/85373501 shutdown_connections 2024-09-11T01:36:08.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.583+0000 7ffb6c5f3640 1 -- 172.21.15.12:0/85373501 >> 172.21.15.12:0/85373501 conn(0x7ffb640a3e30 msgr2=0x7ffb6409f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:08.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.583+0000 7ffb6c5f3640 1 -- 172.21.15.12:0/85373501 shutdown_connections 2024-09-11T01:36:08.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.583+0000 7ffb6c5f3640 1 -- 172.21.15.12:0/85373501 wait complete. 2024-09-11T01:36:08.611 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:36:08.611 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:36:08.611 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:36:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.711+0000 7f5d1e67f640 10 monclient: get_monmap_and_config 2024-09-11T01:36:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.711+0000 7f5d1e67f640 10 monclient: build_initial_monmap 2024-09-11T01:36:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.711+0000 7f5d1e67f640 10 monclient: monmap: 2024-09-11T01:36:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:08.712121+0000 2024-09-11T01:36:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:08.712121+0000 2024-09-11T01:36:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.713 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:08.714 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:08.714 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:08.714 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:08.714 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:08.714 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1e67f640 1 Processor -- start 2024-09-11T01:36:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1e67f640 1 -- start start 2024-09-11T01:36:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1e67f640 10 monclient: init 2024-09-11T01:36:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1e67f640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1e67f640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-11T01:36:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1e67f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5d180a2d40 addr v1:172.21.15.12:6789/0 2024-09-11T01:36:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1e67f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5d1806d260 0x7f5d1813d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1e67f640 10 monclient(hunting): picked mon.noname-a con 0x7f5d1806d260 addr v2:172.21.15.12:3300/0 2024-09-11T01:36:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1e67f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d1806d670 0x7f5d1813fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1e67f640 10 monclient(hunting): picked mon.noname-b con 0x7f5d1806d670 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1e67f640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d1809aed0 con 0x7f5d180a2d40 2024-09-11T01:36:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1e67f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1e67f640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f5d1806dd10 con 0x7f5d1806d260 2024-09-11T01:36:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1e67f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1e67f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5d1809bcf0 con 0x7f5d1806d670 2024-09-11T01:36:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1e67f640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1e67f640 10 monclient(hunting): authenticate will time out at 1175.154703s 2024-09-11T01:36:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d177fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5d1806d260 0x7f5d1813d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1cc1c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d1806d670 0x7f5d1813fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1cc1c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d1806d670 0x7f5d1813fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:49456/0 (socket says 172.21.15.12:49456) 2024-09-11T01:36:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d177fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5d1806d260 0x7f5d1813d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48898/0 (socket says 172.21.15.12:48898) 2024-09-11T01:36:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1cc1c640 1 -- 172.21.15.12:0/3433070903 learned_addr learned my addr 172.21.15.12:0/3433070903 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1cc1c640 10 monclient(hunting): get_auth_request con 0x7f5d1806d670 auth_method 0 2024-09-11T01:36:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1cc1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1cc1c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1cc1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d177fe640 10 monclient(hunting): get_auth_request con 0x7f5d1806d260 auth_method 0 2024-09-11T01:36:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d177fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1cc1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1cc1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d1cc1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.712+0000 7f5d177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.713+0000 7f5d177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.713+0000 7f5d1cc1c640 10 monclient(hunting): handle_auth_done global_id 64181 payload 274 2024-09-11T01:36:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.713+0000 7f5d1cc1c640 1 -- 172.21.15.12:0/3433070903 >> v2:172.21.15.12:3300/0 conn(0x7f5d1806d260 msgr2=0x7f5d1813d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.713+0000 7f5d1cc1c640 1 --2- 172.21.15.12:0/3433070903 >> v2:172.21.15.12:3300/0 conn(0x7f5d1806d260 0x7f5d1813d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.713+0000 7f5d1cc1c640 1 -- 172.21.15.12:0/3433070903 >> v1:172.21.15.12:6789/0 conn(0x7f5d180a2d40 legacy=0x7f5d1806cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.713+0000 7f5d1cc1c640 10 monclient: _finish_hunting 0 2024-09-11T01:36:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.713+0000 7f5d1cc1c640 1 monclient: found mon.noname-b 2024-09-11T01:36:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.713+0000 7f5d1cc1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.713+0000 7f5d1cc1c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:36:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.713+0000 7f5d1cc1c640 1 -- 172.21.15.12:0/3433070903 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d180a3920 con 0x7f5d1806d670 2024-09-11T01:36:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d1cc1c640 1 -- 172.21.15.12:0/3433070903 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d1806d670 msgr2=0x7f5d1813fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d1cc1c640 1 --2- 172.21.15.12:0/3433070903 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d1806d670 0x7f5d1813fa30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5d1806a070 tx=0x7f5d1002a530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7500f3d8cf1047fd server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d16ffd640 1 -- 172.21.15.12:0/3433070903 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d10036070 con 0x7f5d1806d670 2024-09-11T01:36:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d16ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:36:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d16ffd640 10 monclient: dump: 2024-09-11T01:36:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:08.727 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d16ffd640 10 monclient: _finish_auth 0 2024-09-11T01:36:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d16ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:36:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d1e67f640 5 monclient: authenticate success, global_id 64181 2024-09-11T01:36:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d16ffd640 1 -- 172.21.15.12:0/3433070903 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d10007800 con 0x7f5d1806d670 2024-09-11T01:36:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d16ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d1e67f640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d16ffd640 1 -- 172.21.15.12:0/3433070903 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d10007bd0 con 0x7f5d1806d670 2024-09-11T01:36:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d1e67f640 4 set_mon_vals no callback set 2024-09-11T01:36:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d16ffd640 10 monclient: dump: 2024-09-11T01:36:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:08.729 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d1e67f640 10 monclient: shutdown 2024-09-11T01:36:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d1e67f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d1e67f640 1 -- 172.21.15.12:0/3433070903 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d1806d670 msgr2=0x7f5d1813fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d1e67f640 1 --2- 172.21.15.12:0/3433070903 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d1806d670 0x7f5d1813fa30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5d1806a070 tx=0x7f5d1002a530 comp rx=0 tx=0).stop 2024-09-11T01:36:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d1e67f640 1 -- 172.21.15.12:0/3433070903 shutdown_connections 2024-09-11T01:36:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d1e67f640 1 --2- 172.21.15.12:0/3433070903 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d1806d670 0x7f5d1813fa30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d1e67f640 1 --2- 172.21.15.12:0/3433070903 >> v2:172.21.15.12:3300/0 conn(0x7f5d1806d260 0x7f5d1813d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d1e67f640 1 -- 172.21.15.12:0/3433070903 >> 172.21.15.12:0/3433070903 conn(0x7f5d180a3e30 msgr2=0x7f5d180a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d1e67f640 1 -- 172.21.15.12:0/3433070903 shutdown_connections 2024-09-11T01:36:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.716+0000 7f5d1e67f640 1 -- 172.21.15.12:0/3433070903 wait complete. 2024-09-11T01:36:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.717+0000 7f5d1e67f640 10 monclient: build_initial_monmap 2024-09-11T01:36:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.717+0000 7f5d1e67f640 10 monclient: monmap: 2024-09-11T01:36:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:08.730 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.717+0000 7f5d1e67f640 1 Processor -- start 2024-09-11T01:36:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.717+0000 7f5d1e67f640 1 -- start start 2024-09-11T01:36:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.717+0000 7f5d1e67f640 10 monclient: init 2024-09-11T01:36:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d1e67f640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d1e67f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:36:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d1e67f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d1806d260 0x7f5d18072f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d1e67f640 10 monclient(hunting): picked mon.noname-b con 0x7f5d1806d260 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d1e67f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d1806d670 0x7f5d18099990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d1e67f640 10 monclient(hunting): picked mon.noname-a con 0x7f5d1806d670 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d1e67f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d180a2d40 0x7f5d18099ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d1e67f640 10 monclient(hunting): picked mon.noname-c con 0x7f5d180a2d40 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d1e67f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d1e67f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1809ba10 con 0x7f5d1806d670 2024-09-11T01:36:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d1e67f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d1e67f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5d180a3120 con 0x7f5d1806d260 2024-09-11T01:36:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d1e67f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d1e67f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5d1809b3a0 con 0x7f5d180a2d40 2024-09-11T01:36:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d1e67f640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d1e67f640 10 monclient(hunting): authenticate will time out at 1175.160478s 2024-09-11T01:36:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d177fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d1806d670 0x7f5d18099990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d1cc1c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d180a2d40 0x7f5d18099ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d177fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d1806d670 0x7f5d18099990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48910/0 (socket says 172.21.15.12:48910) 2024-09-11T01:36:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d177fe640 1 -- 172.21.15.12:0/1481814209 learned_addr learned my addr 172.21.15.12:0/1481814209 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d177fe640 10 monclient(hunting): get_auth_request con 0x7f5d1806d670 auth_method 0 2024-09-11T01:36:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d177fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d1cc1c640 10 monclient(hunting): get_auth_request con 0x7f5d180a2d40 auth_method 0 2024-09-11T01:36:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d1cc1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d1cc1c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d1cc1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d1cc1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d1cc1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d1cc1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d17fff640 1 --2- 172.21.15.12:0/1481814209 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d1806d260 0x7f5d18072f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d17fff640 10 monclient(hunting): get_auth_request con 0x7f5d1806d260 auth_method 0 2024-09-11T01:36:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d17fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d177fe640 10 monclient(hunting): handle_auth_done global_id 54288 payload 824 2024-09-11T01:36:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d177fe640 1 -- 172.21.15.12:0/1481814209 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d180a2d40 msgr2=0x7f5d18099ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d177fe640 1 --2- 172.21.15.12:0/1481814209 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d180a2d40 0x7f5d18099ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d177fe640 1 -- 172.21.15.12:0/1481814209 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d1806d260 msgr2=0x7f5d18072f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d177fe640 1 --2- 172.21.15.12:0/1481814209 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d1806d260 0x7f5d18072f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d177fe640 10 monclient: _finish_hunting 0 2024-09-11T01:36:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d177fe640 1 monclient: found mon.noname-a 2024-09-11T01:36:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d177fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d177fe640 1 -- 172.21.15.12:0/1481814209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d18073460 con 0x7f5d1806d670 2024-09-11T01:36:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.718+0000 7f5d1cc1c640 1 --2- 172.21.15.12:0/1481814209 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d180a2d40 0x7f5d18099ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:36:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.721+0000 7f5d177fe640 1 -- 172.21.15.12:0/1481814209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d1806d670 msgr2=0x7f5d18099990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.721+0000 7f5d177fe640 1 --2- 172.21.15.12:0/1481814209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d1806d670 0x7f5d18099990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c00e9b0 tx=0x7f5d0c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.721+0000 7f5d14ff9640 1 -- 172.21.15.12:0/1481814209 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c017070 con 0x7f5d1806d670 2024-09-11T01:36:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.721+0000 7f5d14ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.721+0000 7f5d14ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.721+0000 7f5d14ff9640 10 monclient: dump: 2024-09-11T01:36:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:08.736 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.721+0000 7f5d14ff9640 10 monclient: _finish_auth 0 2024-09-11T01:36:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.721+0000 7f5d14ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:36:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.721+0000 7f5d14ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.721+0000 7f5d14ff9640 1 -- 172.21.15.12:0/1481814209 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d0c00cbb0 con 0x7f5d1806d670 2024-09-11T01:36:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.721+0000 7f5d1e67f640 5 monclient: authenticate success, global_id 54288 2024-09-11T01:36:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.721+0000 7f5d1e67f640 10 monclient: _renew_subs 2024-09-11T01:36:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.721+0000 7f5d1e67f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.721+0000 7f5d1e67f640 1 -- 172.21.15.12:0/1481814209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d18074470 con 0x7f5d1806d670 2024-09-11T01:36:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.721+0000 7f5d14ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.721+0000 7f5d14ff9640 1 -- 172.21.15.12:0/1481814209 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c013470 con 0x7f5d1806d670 2024-09-11T01:36:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.721+0000 7f5d14ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.721+0000 7f5d14ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.721+0000 7f5d14ff9640 10 monclient: dump: 2024-09-11T01:36:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:08.738 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:08.738 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.721+0000 7f5d1e67f640 10 monclient: _renew_subs 2024-09-11T01:36:08.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.721+0000 7f5d1e67f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:08.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.721+0000 7f5d1e67f640 1 -- 172.21.15.12:0/1481814209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d18074a30 con 0x7f5d1806d670 2024-09-11T01:36:08.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.721+0000 7f5d167fc640 4 set_mon_vals no callback set 2024-09-11T01:36:08.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.722+0000 7f5d1e67f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:08.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.722+0000 7f5d1e67f640 1 --2- 172.21.15.12:0/1481814209 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d180713b0 0x7f5d18071790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.722+0000 7f5d1e67f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.722+0000 7f5d1e67f640 1 -- 172.21.15.12:0/1481814209 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5d1806cb50 con 0x7f5d180713b0 2024-09-11T01:36:08.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.722+0000 7f5d17fff640 1 --2- 172.21.15.12:0/1481814209 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d180713b0 0x7f5d18071790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.722+0000 7f5d17fff640 10 monclient: get_auth_request con 0x7f5d180713b0 auth_method 0 2024-09-11T01:36:08.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.722+0000 7f5d17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:08.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.722+0000 7f5d17fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:08.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.722+0000 7f5d17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:08.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.722+0000 7f5d17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:08.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.722+0000 7f5d17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:08.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.722+0000 7f5d177fe640 1 --2- 172.21.15.12:0/1481814209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d1806d670 0x7f5d18099990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c00e9b0 tx=0x7f5d0c00ee70 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7f5d0c024060 mgrmap(e 5) 2024-09-11T01:36:08.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.722+0000 7f5d17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:08.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.722+0000 7f5d14ff9640 1 -- 172.21.15.12:0/1481814209 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f5d0c024060 con 0x7f5d1806d670 2024-09-11T01:36:08.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.722+0000 7f5d17fff640 10 monclient(hunting): handle_auth_done global_id 64187 payload 824 2024-09-11T01:36:08.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.722+0000 7f5d17fff640 1 -- 172.21.15.12:0/1481814209 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5d180008d0 con 0x7f5d180713b0 2024-09-11T01:36:08.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.722+0000 7f5d14ff9640 1 --2- 172.21.15.12:0/1481814209 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5d0403e260 0x7f5d04040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.723+0000 7f5d1cc1c640 1 --2- 172.21.15.12:0/1481814209 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5d0403e260 0x7f5d04040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.723+0000 7f5d1cc1c640 10 monclient: get_auth_request con 0x7f5d0403e260 auth_method 0 2024-09-11T01:36:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.724+0000 7f5d17fff640 1 -- 172.21.15.12:0/1481814209 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d180713b0 msgr2=0x7f5d18071790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.724+0000 7f5d17fff640 1 --2- 172.21.15.12:0/1481814209 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d180713b0 0x7f5d18071790 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5d0800e4d0 tx=0x7f5d0800e990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.725+0000 7f5d14ff9640 1 -- 172.21.15.12:0/1481814209 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d08007450 con 0x7f5d180713b0 2024-09-11T01:36:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.725+0000 7f5d14ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.725+0000 7f5d14ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.725+0000 7f5d14ff9640 10 monclient: dump: 2024-09-11T01:36:08.742 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:08.743 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.725+0000 7f5d14ff9640 1 -- 172.21.15.12:0/1481814209 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5d08007450 con 0x7f5d180713b0 2024-09-11T01:36:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.725+0000 7f5d14ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.725+0000 7f5d14ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:08.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.725+0000 7f5d14ff9640 1 -- 172.21.15.12:0/1481814209 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d180713b0 msgr2=0x7f5d18071790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:08.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.725+0000 7f5d14ff9640 1 --2- 172.21.15.12:0/1481814209 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d180713b0 0x7f5d18071790 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5d0800e4d0 tx=0x7f5d0800e990 comp rx=0 tx=0).stop 2024-09-11T01:36:08.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.725+0000 7f5d14ff9640 1 -- 172.21.15.12:0/1481814209 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d180713b0 msgr2=0x7f5d18071790 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:08.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.725+0000 7f5d14ff9640 1 --2- 172.21.15.12:0/1481814209 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d180713b0 0x7f5d18071790 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5d0800e4d0 tx=0x7f5d0800e990 comp rx=0 tx=0).stop 2024-09-11T01:36:08.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.725+0000 7f5d1cc1c640 1 --2- 172.21.15.12:0/1481814209 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5d0403e260 0x7f5d04040720 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5d180700d0 tx=0x7f5d10009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:08.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.731+0000 7f5d1e67f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:36:08.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.731+0000 7f5d1e67f640 1 --2- 172.21.15.12:0/1481814209 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d18071cd0 0x7f5d18135690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.731+0000 7f5d1e67f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.731+0000 7f5d1e67f640 1 -- 172.21.15.12:0/1481814209 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5d180720e0 con 0x7f5d18071cd0 2024-09-11T01:36:08.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.731+0000 7f5d17fff640 1 --2- 172.21.15.12:0/1481814209 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d18071cd0 0x7f5d18135690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.731+0000 7f5d17fff640 10 monclient: get_auth_request con 0x7f5d18071cd0 auth_method 0 2024-09-11T01:36:08.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.731+0000 7f5d17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:08.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.731+0000 7f5d17fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.731+0000 7f5d17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.731+0000 7f5d17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.731+0000 7f5d17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.731+0000 7f5d17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.731+0000 7f5d17fff640 10 monclient(hunting): handle_auth_done global_id 64190 payload 824 2024-09-11T01:36:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.731+0000 7f5d17fff640 1 -- 172.21.15.12:0/1481814209 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5d1813bdc0 con 0x7f5d18071cd0 2024-09-11T01:36:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.733+0000 7f5d17fff640 1 -- 172.21.15.12:0/1481814209 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d18071cd0 msgr2=0x7f5d18135690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.734+0000 7f5d17fff640 1 --2- 172.21.15.12:0/1481814209 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d18071cd0 0x7f5d18135690 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5d0800e4d0 tx=0x7f5d08003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.734+0000 7f5d14ff9640 1 -- 172.21.15.12:0/1481814209 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d0801eb70 con 0x7f5d18071cd0 2024-09-11T01:36:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.734+0000 7f5d14ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.734+0000 7f5d14ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:08.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.734+0000 7f5d14ff9640 10 monclient: dump: 2024-09-11T01:36:08.746 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:08.746 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:08.746 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.746 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.746 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:08.746 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.746 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:08.746 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:08.746 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:08.746 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.734+0000 7f5d14ff9640 1 -- 172.21.15.12:0/1481814209 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4707 (secure 0 0 0) 0x7f5d08028430 con 0x7f5d18071cd0 2024-09-11T01:36:08.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.734+0000 7f5d14ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:36:08.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.734+0000 7f5d14ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:08.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.734+0000 7f5d14ff9640 1 -- 172.21.15.12:0/1481814209 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d18071cd0 msgr2=0x7f5d18135690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:08.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.734+0000 7f5d14ff9640 1 --2- 172.21.15.12:0/1481814209 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d18071cd0 0x7f5d18135690 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5d0800e4d0 tx=0x7f5d08003760 comp rx=0 tx=0).stop 2024-09-11T01:36:08.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.734+0000 7f5d14ff9640 1 -- 172.21.15.12:0/1481814209 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d18071cd0 msgr2=0x7f5d18135690 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:08.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.734+0000 7f5d14ff9640 1 --2- 172.21.15.12:0/1481814209 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d18071cd0 0x7f5d18135690 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5d0800e4d0 tx=0x7f5d08003760 comp rx=0 tx=0).stop 2024-09-11T01:36:08.747 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:36:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:36:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:36:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:36:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 74, 2024-09-11T01:36:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 86827, 2024-09-11T01:36:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:36:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:36:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:36:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:36:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:36:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:36:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:36:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:36:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:36:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:36:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:36:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:36:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:36:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:36:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:36:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:36:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:36:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:36:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:36:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:36:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:36:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:36:08.751 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:36:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:08.752 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:36:08.753 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.753 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:36:08.753 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.753 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:36:08.753 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:36:08.753 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:08.753 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:08.753 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.753 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:08.753 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:36:08.754 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:08.754 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.754 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.754 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:08.754 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:36:08.754 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:08.754 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:08.754 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:08.754 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.754 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:36:08.754 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:36:08.754 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:08.755 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:08.755 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:08.755 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.755 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.755 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:36:08.755 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:36:08.755 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:08.755 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:08.755 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.755 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:08.755 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:36:08.755 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:08.755 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.756 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.756 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:08.756 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:36:08.756 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:08.756 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:08.756 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:08.756 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.756 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:36:08.756 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:36:08.756 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:08.756 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:08.756 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:08.756 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.757 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.757 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:36:08.757 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:36:08.757 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:08.757 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:08.757 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.757 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:08.757 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:36:08.757 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:08.757 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.757 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.757 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:08.757 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:36:08.758 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:08.758 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:08.758 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:08.758 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.758 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:36:08.758 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:36:08.758 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:08.758 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:08.758 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:08.758 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:08.758 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:08.758 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.758 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:36:08.759 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:36:08.759 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.759 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:08.759 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:08.759 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:36:08.759 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:08.759 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:08.759 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:36:08.759 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.759 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:08.759 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:08.759 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:36:08.759 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:08.760 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:08.760 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:36:08.760 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.760 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:08.760 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:08.760 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:36:08.760 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:08.760 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:08.760 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.760 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:36:08.760 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:36:08.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.734+0000 7f5d1e67f640 1 -- 172.21.15.12:0/1481814209 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5d0403e260 msgr2=0x7f5d04040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:08.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.734+0000 7f5d1e67f640 1 --2- 172.21.15.12:0/1481814209 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5d0403e260 0x7f5d04040720 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5d180700d0 tx=0x7f5d10009000 comp rx=0 tx=0).stop 2024-09-11T01:36:08.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.734+0000 7f5d1e67f640 10 monclient: shutdown 2024-09-11T01:36:08.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.734+0000 7f5d1e67f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:08.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.734+0000 7f5d1e67f640 1 -- 172.21.15.12:0/1481814209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d1806d670 msgr2=0x7f5d18099990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:08.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.734+0000 7f5d1e67f640 1 --2- 172.21.15.12:0/1481814209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d1806d670 0x7f5d18099990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c00e9b0 tx=0x7f5d0c00ee70 comp rx=0 tx=0).stop 2024-09-11T01:36:08.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.734+0000 7f5d1cc1c640 1 -- 172.21.15.12:0/1481814209 reap_dead start 2024-09-11T01:36:08.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.734+0000 7f5d14ff9640 1 -- 172.21.15.12:0/1481814209 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(223..223 src has 186..223) ==== 4799+0+0 (unknown 0 0 0) 0x7f5d0c052d30 con 0x7f5d1806d670 2024-09-11T01:36:08.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.734+0000 7f5d1cc1c640 1 -- 172.21.15.12:0/1481814209 reap_dead start 2024-09-11T01:36:08.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.734+0000 7f5d1e67f640 1 -- 172.21.15.12:0/1481814209 shutdown_connections 2024-09-11T01:36:08.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.734+0000 7f5d1e67f640 1 -- 172.21.15.12:0/1481814209 >> 172.21.15.12:0/1481814209 conn(0x7f5d180a3e30 msgr2=0x7f5d180a5210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:08.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.734+0000 7f5d1e67f640 1 -- 172.21.15.12:0/1481814209 shutdown_connections 2024-09-11T01:36:08.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.734+0000 7f5d1e67f640 1 -- 172.21.15.12:0/1481814209 wait complete. 2024-09-11T01:36:08.763 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:36:08.763 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:36:08.763 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:36:08.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.864+0000 7fca3f8e6640 10 monclient: get_monmap_and_config 2024-09-11T01:36:08.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.864+0000 7fca3f8e6640 10 monclient: build_initial_monmap 2024-09-11T01:36:08.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.864+0000 7fca3f8e6640 10 monclient: monmap: 2024-09-11T01:36:08.866 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:08.866 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:08.866 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:08.865059+0000 2024-09-11T01:36:08.866 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:08.865059+0000 2024-09-11T01:36:08.866 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:08.866 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.866 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:08.866 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:08.866 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:08.866 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:08.866 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:08.867 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.864+0000 7fca3f8e6640 1 Processor -- start 2024-09-11T01:36:08.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3f8e6640 1 -- start start 2024-09-11T01:36:08.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3f8e6640 10 monclient: init 2024-09-11T01:36:08.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3f8e6640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:08.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3f8e6640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-11T01:36:08.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3f8e6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fca38137e50 addr v1:172.21.15.12:6789/0 2024-09-11T01:36:08.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3f8e6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fca38138230 0x7fca38138640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3f8e6640 10 monclient(hunting): picked mon.noname-c con 0x7fca38138230 addr v2:172.21.15.71:3300/0 2024-09-11T01:36:08.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3f8e6640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fca3813d380 0x7fca3813f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3f8e6640 10 monclient(hunting): picked mon.noname-a con 0x7fca3813d380 addr v2:172.21.15.12:3300/0 2024-09-11T01:36:08.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3f8e6640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca380a62d0 con 0x7fca38137e50 2024-09-11T01:36:08.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3f8e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3f8e6640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fca3809bb40 con 0x7fca3813d380 2024-09-11T01:36:08.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3f8e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3f8e6640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fca3809c4a0 con 0x7fca38138230 2024-09-11T01:36:08.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3f8e6640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:08.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3f8e6640 10 monclient(hunting): authenticate will time out at 1175.307605s 2024-09-11T01:36:08.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3de83640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fca3813d380 0x7fca3813f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3de83640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fca3813d380 0x7fca3813f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48924/0 (socket says 172.21.15.12:48924) 2024-09-11T01:36:08.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3de83640 1 -- 172.21.15.12:0/3792459662 learned_addr learned my addr 172.21.15.12:0/3792459662 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:08.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3de83640 10 monclient(hunting): get_auth_request con 0x7fca3813d380 auth_method 0 2024-09-11T01:36:08.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3de83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:08.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3de83640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:08.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3de83640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:08.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3de83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:08.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3de83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:08.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3ce81640 1 --2- 172.21.15.12:0/3792459662 >> v2:172.21.15.71:3300/0 conn(0x7fca38138230 0x7fca38138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3de83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:08.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3ce81640 10 monclient(hunting): get_auth_request con 0x7fca38138230 auth_method 0 2024-09-11T01:36:08.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3ce81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:08.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3ce81640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:08.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3ce81640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:08.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3ce81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:08.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3ce81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:08.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.865+0000 7fca3ce81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:08.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.866+0000 7fca3ce81640 10 monclient(hunting): handle_auth_done global_id 74098 payload 274 2024-09-11T01:36:08.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.866+0000 7fca3ce81640 1 -- 172.21.15.12:0/3792459662 >> v2:172.21.15.12:3300/0 conn(0x7fca3813d380 msgr2=0x7fca3813f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:08.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.866+0000 7fca3ce81640 1 --2- 172.21.15.12:0/3792459662 >> v2:172.21.15.12:3300/0 conn(0x7fca3813d380 0x7fca3813f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.866+0000 7fca3ce81640 1 -- 172.21.15.12:0/3792459662 >> v1:172.21.15.12:6789/0 conn(0x7fca38137e50 legacy=0x7fca3809d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:08.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.866+0000 7fca3ce81640 10 monclient: _finish_hunting 0 2024-09-11T01:36:08.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.866+0000 7fca3ce81640 1 monclient: found mon.noname-c 2024-09-11T01:36:08.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.866+0000 7fca3ce81640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:08.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.866+0000 7fca3ce81640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:36:08.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.866+0000 7fca3ce81640 1 -- 172.21.15.12:0/3792459662 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca3809c780 con 0x7fca38138230 2024-09-11T01:36:08.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.869+0000 7fca3ce81640 1 -- 172.21.15.12:0/3792459662 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca38138230 msgr2=0x7fca38138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:08.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.869+0000 7fca3ce81640 1 --2- 172.21.15.12:0/3792459662 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca38138230 0x7fca38138640 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fca380a6f80 tx=0x7fca2c029fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86d08703ad60aab server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:08.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.869+0000 7fca37fff640 1 -- 172.21.15.12:0/3792459662 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca2c037070 con 0x7fca38138230 2024-09-11T01:36:08.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.869+0000 7fca37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:08.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.869+0000 7fca37fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:36:08.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.869+0000 7fca37fff640 10 monclient: dump: 2024-09-11T01:36:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:08.884 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:08.885 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.869+0000 7fca37fff640 10 monclient: _finish_auth 0 2024-09-11T01:36:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.869+0000 7fca37fff640 10 monclient: _check_auth_tickets 2024-09-11T01:36:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.869+0000 7fca37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.869+0000 7fca37fff640 1 -- 172.21.15.12:0/3792459662 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca2c032440 con 0x7fca38138230 2024-09-11T01:36:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.869+0000 7fca3f8e6640 5 monclient: authenticate success, global_id 74098 2024-09-11T01:36:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.869+0000 7fca3f8e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:36:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.869+0000 7fca37fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.869+0000 7fca3f8e6640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:08.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.869+0000 7fca37fff640 1 -- 172.21.15.12:0/3792459662 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca2c032810 con 0x7fca38138230 2024-09-11T01:36:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.869+0000 7fca3f8e6640 4 set_mon_vals no callback set 2024-09-11T01:36:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.869+0000 7fca37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.869+0000 7fca37fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:36:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.869+0000 7fca37fff640 10 monclient: dump: 2024-09-11T01:36:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:08.886 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:08.887 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.870+0000 7fca3f8e6640 10 monclient: shutdown 2024-09-11T01:36:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.870+0000 7fca3f8e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.870+0000 7fca3f8e6640 1 -- 172.21.15.12:0/3792459662 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca38138230 msgr2=0x7fca38138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.870+0000 7fca3f8e6640 1 --2- 172.21.15.12:0/3792459662 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca38138230 0x7fca38138640 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fca380a6f80 tx=0x7fca2c029fb0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.870+0000 7fca3f8e6640 1 -- 172.21.15.12:0/3792459662 shutdown_connections 2024-09-11T01:36:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.870+0000 7fca3f8e6640 1 --2- 172.21.15.12:0/3792459662 >> v2:172.21.15.12:3300/0 conn(0x7fca3813d380 0x7fca3813f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.870+0000 7fca3f8e6640 1 --2- 172.21.15.12:0/3792459662 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca38138230 0x7fca38138640 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.870+0000 7fca3f8e6640 1 -- 172.21.15.12:0/3792459662 >> 172.21.15.12:0/3792459662 conn(0x7fca380a7020 msgr2=0x7fca380a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.870+0000 7fca3f8e6640 1 -- 172.21.15.12:0/3792459662 shutdown_connections 2024-09-11T01:36:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.870+0000 7fca3f8e6640 1 -- 172.21.15.12:0/3792459662 wait complete. 2024-09-11T01:36:08.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.870+0000 7fca3f8e6640 10 monclient: build_initial_monmap 2024-09-11T01:36:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.870+0000 7fca3f8e6640 10 monclient: monmap: 2024-09-11T01:36:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:08.888 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3f8e6640 1 Processor -- start 2024-09-11T01:36:08.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3f8e6640 1 -- start start 2024-09-11T01:36:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3f8e6640 10 monclient: init 2024-09-11T01:36:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3f8e6640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3f8e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:36:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3f8e6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fca38137e50 0x7fca381cd000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3f8e6640 10 monclient(hunting): picked mon.noname-c con 0x7fca38137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3f8e6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca38138230 0x7fca38137050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3f8e6640 10 monclient(hunting): picked mon.noname-a con 0x7fca38138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3f8e6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca3813d380 0x7fca380a1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3f8e6640 10 monclient(hunting): picked mon.noname-b con 0x7fca3813d380 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3f8e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3f8e6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fca3806f890 con 0x7fca38138230 2024-09-11T01:36:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3f8e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3f8e6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fca3806f710 con 0x7fca3813d380 2024-09-11T01:36:08.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3f8e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3f8e6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fca3813fcb0 con 0x7fca38137e50 2024-09-11T01:36:08.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3f8e6640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:08.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3f8e6640 10 monclient(hunting): authenticate will time out at 1175.313597s 2024-09-11T01:36:08.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3d682640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fca38137e50 0x7fca381cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3ce81640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca38138230 0x7fca38137050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3d682640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fca38137e50 0x7fca381cd000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:49502/0 (socket says 172.21.15.12:49502) 2024-09-11T01:36:08.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3d682640 1 -- 172.21.15.12:0/3806058787 learned_addr learned my addr 172.21.15.12:0/3806058787 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:08.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3ce81640 10 monclient(hunting): get_auth_request con 0x7fca38138230 auth_method 0 2024-09-11T01:36:08.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3ce81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:08.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3ce81640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:08.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3ce81640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:08.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3d682640 10 monclient(hunting): get_auth_request con 0x7fca38137e50 auth_method 0 2024-09-11T01:36:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3d682640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3d682640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3d682640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3ce81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3ce81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3ce81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3d682640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3d682640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3d682640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3de83640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca3813d380 0x7fca380a1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3de83640 10 monclient(hunting): get_auth_request con 0x7fca3813d380 auth_method 0 2024-09-11T01:36:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3de83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3de83640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:08.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3de83640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:08.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3de83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:08.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3de83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:08.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3de83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:08.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3ce81640 10 monclient(hunting): handle_auth_done global_id 54294 payload 824 2024-09-11T01:36:08.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.871+0000 7fca3ce81640 1 -- 172.21.15.12:0/3806058787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fca38137e50 msgr2=0x7fca381cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:08.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.872+0000 7fca3ce81640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fca38137e50 0x7fca381cd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.872+0000 7fca3ce81640 1 -- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca3813d380 msgr2=0x7fca380a1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:08.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.872+0000 7fca3ce81640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca3813d380 0x7fca380a1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.872+0000 7fca3ce81640 10 monclient: _finish_hunting 0 2024-09-11T01:36:08.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.872+0000 7fca3ce81640 1 monclient: found mon.noname-a 2024-09-11T01:36:08.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.872+0000 7fca3ce81640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:08.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.872+0000 7fca3ce81640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:08.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.872+0000 7fca3ce81640 1 -- 172.21.15.12:0/3806058787 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca380a1710 con 0x7fca38138230 2024-09-11T01:36:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.872+0000 7fca3d682640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fca38137e50 0x7fca381cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:36:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.874+0000 7fca3ce81640 1 -- 172.21.15.12:0/3806058787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca38138230 msgr2=0x7fca38137050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.874+0000 7fca3ce81640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca38138230 0x7fca38137050 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fca2c034000 tx=0x7fca2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.874+0000 7fca35ffb640 1 -- 172.21.15.12:0/3806058787 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca2c040020 con 0x7fca38138230 2024-09-11T01:36:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.874+0000 7fca35ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.874+0000 7fca35ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.874+0000 7fca35ffb640 10 monclient: dump: 2024-09-11T01:36:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:08.894 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:08.894 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:08.894 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.874+0000 7fca35ffb640 10 monclient: _finish_auth 0 2024-09-11T01:36:08.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.874+0000 7fca35ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:36:08.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.874+0000 7fca35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:08.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.874+0000 7fca3f8e6640 5 monclient: authenticate success, global_id 54294 2024-09-11T01:36:08.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.874+0000 7fca35ffb640 1 -- 172.21.15.12:0/3806058787 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca2c038040 con 0x7fca38138230 2024-09-11T01:36:08.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.874+0000 7fca35ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:08.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.874+0000 7fca35ffb640 1 -- 172.21.15.12:0/3806058787 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca2c037070 con 0x7fca38138230 2024-09-11T01:36:08.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.874+0000 7fca35ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:08.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.874+0000 7fca35ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:08.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.874+0000 7fca35ffb640 10 monclient: dump: 2024-09-11T01:36:08.894 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:08.895 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.874+0000 7fca377fe640 4 set_mon_vals no callback set 2024-09-11T01:36:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.874+0000 7fca3f8e6640 10 monclient: _renew_subs 2024-09-11T01:36:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.874+0000 7fca3f8e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.874+0000 7fca3f8e6640 1 -- 172.21.15.12:0/3806058787 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca3813fa60 con 0x7fca38138230 2024-09-11T01:36:08.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.874+0000 7fca3f8e6640 10 monclient: _renew_subs 2024-09-11T01:36:08.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.874+0000 7fca3f8e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:08.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.874+0000 7fca3f8e6640 1 -- 172.21.15.12:0/3806058787 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca38121140 con 0x7fca38138230 2024-09-11T01:36:08.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.875+0000 7fca3f8e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:08.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.875+0000 7fca3f8e6640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca380a22a0 0x7fca380a2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.875+0000 7fca3f8e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.875+0000 7fca3f8e6640 1 -- 172.21.15.12:0/3806058787 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fca3806a060 con 0x7fca380a22a0 2024-09-11T01:36:08.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.875+0000 7fca35ffb640 1 -- 172.21.15.12:0/3806058787 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fca2c0443b0 con 0x7fca38138230 2024-09-11T01:36:08.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.875+0000 7fca3d682640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca380a22a0 0x7fca380a2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.875+0000 7fca3d682640 10 monclient: get_auth_request con 0x7fca380a22a0 auth_method 0 2024-09-11T01:36:08.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.875+0000 7fca3d682640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:08.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.875+0000 7fca3d682640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:08.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.875+0000 7fca3d682640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:08.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.875+0000 7fca3d682640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:08.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.875+0000 7fca3d682640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:08.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.875+0000 7fca35ffb640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fca2403e240 0x7fca24040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.875+0000 7fca3d682640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:08.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.876+0000 7fca35ffb640 1 -- 172.21.15.12:0/3806058787 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(223..223 src has 186..223) ==== 4799+0+0 (secure 0 0 0) 0x7fca2c073240 con 0x7fca38138230 2024-09-11T01:36:08.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.876+0000 7fca3d682640 10 monclient(hunting): handle_auth_done global_id 74104 payload 824 2024-09-11T01:36:08.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.876+0000 7fca3d682640 1 -- 172.21.15.12:0/3806058787 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fca380a2bc0 con 0x7fca380a22a0 2024-09-11T01:36:08.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.876+0000 7fca3de83640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fca2403e240 0x7fca24040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.876+0000 7fca3de83640 10 monclient: get_auth_request con 0x7fca2403e240 auth_method 0 2024-09-11T01:36:08.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.878+0000 7fca3d682640 1 -- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca380a22a0 msgr2=0x7fca380a2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:08.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.878+0000 7fca3d682640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca380a22a0 0x7fca380a2680 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fca2800fe10 tx=0x7fca2800d520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:08.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.878+0000 7fca35ffb640 1 -- 172.21.15.12:0/3806058787 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca28018030 con 0x7fca380a22a0 2024-09-11T01:36:08.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.878+0000 7fca35ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:08.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.878+0000 7fca35ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:36:08.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.878+0000 7fca35ffb640 10 monclient: dump: 2024-09-11T01:36:08.898 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:08.898 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:08.898 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.898 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.898 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:08.899 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.878+0000 7fca3de83640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fca2403e240 0x7fca24040700 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fca30009b50 tx=0x7fca30009240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.878+0000 7fca35ffb640 1 -- 172.21.15.12:0/3806058787 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fca28004980 con 0x7fca380a22a0 2024-09-11T01:36:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.878+0000 7fca35ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.878+0000 7fca35ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.878+0000 7fca35ffb640 1 -- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca380a22a0 msgr2=0x7fca380a2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.878+0000 7fca35ffb640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca380a22a0 0x7fca380a2680 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fca2800fe10 tx=0x7fca2800d520 comp rx=0 tx=0).stop 2024-09-11T01:36:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.878+0000 7fca35ffb640 1 -- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca380a22a0 msgr2=0x7fca380a2680 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.878+0000 7fca35ffb640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca380a22a0 0x7fca380a2680 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fca2800fe10 tx=0x7fca2800d520 comp rx=0 tx=0).stop 2024-09-11T01:36:08.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.884+0000 7fca3f8e6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:36:08.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.884+0000 7fca3f8e6640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca3809e420 0x7fca3809e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.884+0000 7fca3f8e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.884+0000 7fca3f8e6640 1 -- 172.21.15.12:0/3806058787 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fca38138640 con 0x7fca3809e420 2024-09-11T01:36:08.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.885+0000 7fca3d682640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca3809e420 0x7fca3809e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.885+0000 7fca3d682640 10 monclient: get_auth_request con 0x7fca3809e420 auth_method 0 2024-09-11T01:36:08.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.885+0000 7fca3d682640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:08.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.885+0000 7fca3d682640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:08.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.885+0000 7fca3d682640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:08.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.885+0000 7fca3d682640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:08.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.885+0000 7fca3d682640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:08.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.885+0000 7fca3d682640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:08.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.885+0000 7fca3d682640 10 monclient(hunting): handle_auth_done global_id 74107 payload 824 2024-09-11T01:36:08.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.885+0000 7fca3d682640 1 -- 172.21.15.12:0/3806058787 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fca38137b40 con 0x7fca3809e420 2024-09-11T01:36:08.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.887+0000 7fca3d682640 1 -- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca3809e420 msgr2=0x7fca3809e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:08.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.887+0000 7fca3d682640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca3809e420 0x7fca3809e830 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fca2800fe10 tx=0x7fca28003ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:08.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.887+0000 7fca3d682640 1 -- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca3809e420 msgr2=0x7fca3809e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:36:08.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.887+0000 7fca3d682640 1 -- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca3809e420 msgr2=0x7fca3809e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:36:08.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.887+0000 7fca3d682640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca3809e420 0x7fca3809e830 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fca2800fe10 tx=0x7fca28003ae0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:08.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.887+0000 7fca3d682640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca3809e420 0x7fca3809e830 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fca2800fe10 tx=0x7fca28003ae0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.887+0000 7fca35ffb640 1 -- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca3809e420 msgr2=0x7fca3809e830 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:08.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.887+0000 7fca35ffb640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca3809e420 0x7fca3809e830 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.887+0000 7fca35ffb640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca24044470 0x7fca24046860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:08.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.887+0000 7fca35ffb640 1 -- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca3809e420 msgr2=0x7fca3809e830 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:08.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.887+0000 7fca35ffb640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca3809e420 0x7fca3809e830 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.887+0000 7fca35ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:08.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.887+0000 7fca35ffb640 1 -- 172.21.15.12:0/3806058787 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fca24046e70 con 0x7fca24044470 2024-09-11T01:36:08.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.887+0000 7fca3d682640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca24044470 0x7fca24046860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:08.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.888+0000 7fca3d682640 10 monclient: get_auth_request con 0x7fca24044470 auth_method 0 2024-09-11T01:36:08.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.888+0000 7fca3d682640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:08.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.888+0000 7fca3d682640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:08.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.888+0000 7fca3d682640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:08.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.888+0000 7fca3d682640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:08.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.888+0000 7fca3d682640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:08.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.888+0000 7fca3d682640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:08.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.888+0000 7fca3d682640 10 monclient(hunting): handle_auth_done global_id 74110 payload 824 2024-09-11T01:36:08.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.888+0000 7fca3d682640 1 -- 172.21.15.12:0/3806058787 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fca28004980 con 0x7fca24044470 2024-09-11T01:36:08.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.890+0000 7fca3d682640 1 -- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca24044470 msgr2=0x7fca24046860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:08.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.890+0000 7fca3d682640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca24044470 0x7fca24046860 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fca2800d560 tx=0x7fca28003e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:08.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.890+0000 7fca35ffb640 1 -- 172.21.15.12:0/3806058787 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca28027630 con 0x7fca24044470 2024-09-11T01:36:08.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.890+0000 7fca35ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:08.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.890+0000 7fca35ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:36:08.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.890+0000 7fca35ffb640 10 monclient: dump: 2024-09-11T01:36:08.907 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:08.907 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:08.907 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.907 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:08.907 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:08.907 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:08.907 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:08.907 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:08.908 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:08.908 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:08.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.891+0000 7fca35ffb640 1 -- 172.21.15.12:0/3806058787 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4707 (secure 0 0 0) 0x7fca28027630 con 0x7fca24044470 2024-09-11T01:36:08.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.891+0000 7fca35ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:36:08.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.891+0000 7fca35ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:08.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.891+0000 7fca35ffb640 1 -- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca24044470 msgr2=0x7fca24046860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:08.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.891+0000 7fca35ffb640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca24044470 0x7fca24046860 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fca2800d560 tx=0x7fca28003e60 comp rx=0 tx=0).stop 2024-09-11T01:36:08.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.891+0000 7fca35ffb640 1 -- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca24044470 msgr2=0x7fca24046860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:08.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.891+0000 7fca35ffb640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca24044470 0x7fca24046860 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fca2800d560 tx=0x7fca28003e60 comp rx=0 tx=0).stop 2024-09-11T01:36:08.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.891+0000 7fca3de83640 1 -- 172.21.15.12:0/3806058787 reap_dead start 2024-09-11T01:36:08.909 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:36:08.910 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:36:08.910 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:36:08.910 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:36:08.910 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 74, 2024-09-11T01:36:08.910 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 20669, 2024-09-11T01:36:08.910 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:36:08.910 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:36:08.910 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:36:08.911 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:36:08.911 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:08.911 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:36:08.911 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:08.911 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:36:08.911 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:36:08.911 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:08.912 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:08.912 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:08.912 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:08.912 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:08.912 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:08.912 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:08.912 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:08.913 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:08.913 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:08.913 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:08.913 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:08.913 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:36:08.914 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:36:08.914 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:08.914 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:08.914 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:08.914 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:08.914 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:08.915 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:08.915 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:08.915 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:08.915 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:08.915 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:08.916 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:08.916 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:08.916 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.916 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:36:08.916 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:36:08.917 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:36:08.917 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:36:08.917 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:36:08.917 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:36:08.917 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:08.918 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:08.918 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:36:08.918 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:36:08.918 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:36:08.918 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:36:08.919 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:36:08.919 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:36:08.919 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:36:08.919 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:08.919 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:36:08.920 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:08.920 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:08.920 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:08.920 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:08.920 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:08.920 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:08.921 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:08.921 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:08.921 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:08.921 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:08.921 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:08.922 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:08.922 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:36:08.922 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.922 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:36:08.922 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.923 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:36:08.923 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:36:08.923 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:08.923 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:08.923 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.923 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:08.924 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:36:08.924 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:08.924 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.924 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.924 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:08.925 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:36:08.925 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:08.925 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:08.925 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:08.925 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.926 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:36:08.926 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:36:08.926 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:08.926 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:08.926 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:08.927 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.927 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.927 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:36:08.927 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:36:08.927 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:08.927 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:08.928 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.928 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:08.928 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:36:08.928 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:08.928 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.929 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.929 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:08.929 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:36:08.929 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:08.929 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:08.930 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:08.930 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.930 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:36:08.930 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:36:08.930 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:08.930 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:08.931 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:08.931 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.931 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.931 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:36:08.931 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:36:08.932 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:08.932 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:08.932 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.932 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:08.932 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:36:08.933 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:08.933 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.933 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.933 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:08.933 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:36:08.933 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:08.934 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:08.934 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:08.934 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.934 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:36:08.934 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:36:08.935 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:08.935 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:08.935 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:08.935 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:08.935 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:08.935 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.936 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:36:08.936 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:36:08.936 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.936 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:08.936 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:08.937 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:36:08.937 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:08.937 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:08.937 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:36:08.937 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.938 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:08.938 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:08.938 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:36:08.938 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:08.938 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:08.938 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:36:08.939 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:08.939 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:08.939 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:08.939 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:36:08.939 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:08.940 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:08.940 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:08.940 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:36:08.940 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:36:08.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.891+0000 7fca3f8e6640 1 -- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fca2403e240 msgr2=0x7fca24040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:08.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.891+0000 7fca3f8e6640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fca2403e240 0x7fca24040700 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fca30009b50 tx=0x7fca30009240 comp rx=0 tx=0).stop 2024-09-11T01:36:08.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.891+0000 7fca3f8e6640 10 monclient: shutdown 2024-09-11T01:36:08.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.891+0000 7fca3f8e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:08.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.891+0000 7fca3f8e6640 1 -- 172.21.15.12:0/3806058787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca38138230 msgr2=0x7fca38137050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:08.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.891+0000 7fca3f8e6640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca38138230 0x7fca38137050 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fca2c034000 tx=0x7fca2c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.891+0000 7fca3f8e6640 1 -- 172.21.15.12:0/3806058787 shutdown_connections 2024-09-11T01:36:08.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.891+0000 7fca3f8e6640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fca2403e240 0x7fca24040700 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.891+0000 7fca3f8e6640 1 --2- 172.21.15.12:0/3806058787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca38138230 0x7fca38137050 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:08.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.891+0000 7fca3f8e6640 1 -- 172.21.15.12:0/3806058787 >> 172.21.15.12:0/3806058787 conn(0x7fca380a7020 msgr2=0x7fca3813e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:08.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.891+0000 7fca3f8e6640 1 -- 172.21.15.12:0/3806058787 shutdown_connections 2024-09-11T01:36:08.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:08.891+0000 7fca3f8e6640 1 -- 172.21.15.12:0/3806058787 wait complete. 2024-09-11T01:36:08.945 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:36:08.945 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:36:08.946 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:36:09.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.045+0000 7fa9d47c6640 10 monclient: get_monmap_and_config 2024-09-11T01:36:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.045+0000 7fa9d47c6640 10 monclient: build_initial_monmap 2024-09-11T01:36:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.045+0000 7fa9d47c6640 10 monclient: monmap: 2024-09-11T01:36:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:09.048 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:09.046292+0000 2024-09-11T01:36:09.049 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:09.046292+0000 2024-09-11T01:36:09.049 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:09.049 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:09.049 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:09.049 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:09.050 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:09.050 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:09.050 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:09.050 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:09.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.046+0000 7fa9d47c6640 1 Processor -- start 2024-09-11T01:36:09.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.046+0000 7fa9d47c6640 1 -- start start 2024-09-11T01:36:09.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.046+0000 7fa9d47c6640 10 monclient: init 2024-09-11T01:36:09.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.046+0000 7fa9d47c6640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:09.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.046+0000 7fa9d47c6640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-11T01:36:09.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.046+0000 7fa9d47c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa9cc134ad0 addr v1:172.21.15.12:6789/0 2024-09-11T01:36:09.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.046+0000 7fa9d47c6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa9cc0a2200 addr v1:172.21.15.71:6789/0 2024-09-11T01:36:09.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.046+0000 7fa9d47c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9cc1355c0 0x7fa9cc13d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:09.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.046+0000 7fa9d47c6640 10 monclient(hunting): picked mon.noname-b con 0x7fa9cc1355c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:09.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.046+0000 7fa9d47c6640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9cc09ed30 con 0x7fa9cc134ad0 2024-09-11T01:36:09.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.046+0000 7fa9d47c6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9cc09f010 con 0x7fa9cc0a2200 2024-09-11T01:36:09.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.046+0000 7fa9d47c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:09.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.046+0000 7fa9d47c6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa9cc09f5d0 con 0x7fa9cc1355c0 2024-09-11T01:36:09.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.046+0000 7fa9d47c6640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:09.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.046+0000 7fa9d47c6640 10 monclient(hunting): authenticate will time out at 1175.488864s 2024-09-11T01:36:09.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.046+0000 7fa9d2d63640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9cc1355c0 0x7fa9cc13d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:09.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.046+0000 7fa9d2562640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa9cc134ad0 0x7fa9cc134eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42124/0 (socket says 172.21.15.12:42124) 2024-09-11T01:36:09.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.047+0000 7fa9d2562640 1 -- 172.21.15.12:0/172868934 learned_addr learned my addr 172.21.15.12:0/172868934 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:36:09.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.047+0000 7fa9d2d63640 10 monclient(hunting): get_auth_request con 0x7fa9cc1355c0 auth_method 0 2024-09-11T01:36:09.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.047+0000 7fa9d2d63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:09.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.047+0000 7fa9d2d63640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:09.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.047+0000 7fa9d2d63640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:09.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.047+0000 7fa9d2d63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:09.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.047+0000 7fa9d2d63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:09.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.047+0000 7fa9d2d63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:09.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.047+0000 7fa9d2d63640 10 monclient(hunting): handle_auth_done global_id 64196 payload 274 2024-09-11T01:36:09.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.047+0000 7fa9d2d63640 1 -- 172.21.15.12:0/172868934 >> v1:172.21.15.71:6789/0 conn(0x7fa9cc0a2200 legacy=0x7fa9cc13a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:09.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.047+0000 7fa9d2d63640 1 -- 172.21.15.12:0/172868934 >> v1:172.21.15.12:6789/0 conn(0x7fa9cc134ad0 legacy=0x7fa9cc134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:09.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.047+0000 7fa9d2d63640 10 monclient: _finish_hunting 0 2024-09-11T01:36:09.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.047+0000 7fa9d2d63640 1 monclient: found mon.noname-b 2024-09-11T01:36:09.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.047+0000 7fa9d2d63640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:09.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.047+0000 7fa9d2d63640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:36:09.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.047+0000 7fa9d2d63640 1 -- 172.21.15.12:0/172868934 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9cc09f2f0 con 0x7fa9cc1355c0 2024-09-11T01:36:09.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.050+0000 7fa9d2d63640 1 -- 172.21.15.12:0/172868934 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9cc1355c0 msgr2=0x7fa9cc13d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:09.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.050+0000 7fa9d2d63640 1 --2- 172.21.15.12:0/172868934 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9cc1355c0 0x7fa9cc13d730 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa9c400d050 tx=0x7fa9c402c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bd9d4dd24e54f29 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:09.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.050+0000 7fa9d1560640 1 -- 172.21.15.12:0/172868934 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c4038070 con 0x7fa9cc1355c0 2024-09-11T01:36:09.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.050+0000 7fa9d1560640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:09.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.050+0000 7fa9d1560640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:36:09.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.050+0000 7fa9d1560640 10 monclient: dump: 2024-09-11T01:36:09.075 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:09.075 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:09.075 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:09.075 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:09.075 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:09.076 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:09.076 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:09.076 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:09.076 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:09.076 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:09.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.050+0000 7fa9d1560640 10 monclient: _finish_auth 0 2024-09-11T01:36:09.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.050+0000 7fa9d1560640 10 monclient: _check_auth_tickets 2024-09-11T01:36:09.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.050+0000 7fa9d1560640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:09.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.050+0000 7fa9d1560640 1 -- 172.21.15.12:0/172868934 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9c4009950 con 0x7fa9cc1355c0 2024-09-11T01:36:09.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.050+0000 7fa9d47c6640 5 monclient: authenticate success, global_id 64196 2024-09-11T01:36:09.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.050+0000 7fa9d47c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:36:09.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.050+0000 7fa9d1560640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:09.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.050+0000 7fa9d47c6640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:09.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.050+0000 7fa9d1560640 1 -- 172.21.15.12:0/172868934 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c400b3c0 con 0x7fa9cc1355c0 2024-09-11T01:36:09.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.050+0000 7fa9d1560640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:09.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.050+0000 7fa9d47c6640 4 set_mon_vals no callback set 2024-09-11T01:36:09.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.050+0000 7fa9d1560640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:09.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.050+0000 7fa9d1560640 10 monclient: dump: 2024-09-11T01:36:09.079 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:09.079 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:09.080 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:09.080 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:09.080 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:09.080 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:09.080 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:09.080 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:09.081 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:09.081 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:09.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.050+0000 7fa9d47c6640 10 monclient: shutdown 2024-09-11T01:36:09.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.050+0000 7fa9d47c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:09.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.050+0000 7fa9d47c6640 1 -- 172.21.15.12:0/172868934 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9cc1355c0 msgr2=0x7fa9cc13d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:09.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.050+0000 7fa9d47c6640 1 --2- 172.21.15.12:0/172868934 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9cc1355c0 0x7fa9cc13d730 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa9c400d050 tx=0x7fa9c402c6c0 comp rx=0 tx=0).stop 2024-09-11T01:36:09.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.051+0000 7fa9d47c6640 1 -- 172.21.15.12:0/172868934 shutdown_connections 2024-09-11T01:36:09.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.051+0000 7fa9d47c6640 1 --2- 172.21.15.12:0/172868934 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9cc1355c0 0x7fa9cc13d730 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:09.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.051+0000 7fa9d47c6640 1 -- 172.21.15.12:0/172868934 >> 172.21.15.12:0/172868934 conn(0x7fa9cc0a2f70 msgr2=0x7fa9cc0a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:09.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.051+0000 7fa9d47c6640 1 -- 172.21.15.12:0/172868934 shutdown_connections 2024-09-11T01:36:09.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.051+0000 7fa9d47c6640 1 -- 172.21.15.12:0/172868934 wait complete. 2024-09-11T01:36:09.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.051+0000 7fa9d47c6640 10 monclient: build_initial_monmap 2024-09-11T01:36:09.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.051+0000 7fa9d47c6640 10 monclient: monmap: 2024-09-11T01:36:09.083 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:09.083 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:09.084 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:09.084 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:09.084 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:09.084 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:09.084 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:09.084 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:09.085 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:09.085 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:09.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d47c6640 1 Processor -- start 2024-09-11T01:36:09.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d47c6640 1 -- start start 2024-09-11T01:36:09.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d47c6640 10 monclient: init 2024-09-11T01:36:09.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d47c6640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:09.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d47c6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:36:09.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d47c6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9cc0a2200 0x7fa9cc1c9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:09.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d47c6640 10 monclient(hunting): picked mon.noname-b con 0x7fa9cc0a2200 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:09.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d47c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9cc134ad0 0x7fa9cc1c9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:09.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d47c6640 10 monclient(hunting): picked mon.noname-c con 0x7fa9cc134ad0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:09.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d47c6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9cc1355c0 0x7fa9cc120310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:09.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d47c6640 10 monclient(hunting): picked mon.noname-a con 0x7fa9cc1355c0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:09.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d47c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:09.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d47c6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa9cc09cde0 con 0x7fa9cc1355c0 2024-09-11T01:36:09.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d47c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:09.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d47c6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa9cc0a01e0 con 0x7fa9cc0a2200 2024-09-11T01:36:09.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d47c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:09.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d47c6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa9cc0a0c90 con 0x7fa9cc134ad0 2024-09-11T01:36:09.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d47c6640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:09.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d47c6640 10 monclient(hunting): authenticate will time out at 1175.494615s 2024-09-11T01:36:09.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d2d63640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9cc1355c0 0x7fa9cc120310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:09.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d1d61640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9cc134ad0 0x7fa9cc1c9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:09.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d2d63640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9cc1355c0 0x7fa9cc120310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48942/0 (socket says 172.21.15.12:48942) 2024-09-11T01:36:09.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d2d63640 1 -- 172.21.15.12:0/4282313098 learned_addr learned my addr 172.21.15.12:0/4282313098 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:09.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d2d63640 10 monclient(hunting): get_auth_request con 0x7fa9cc1355c0 auth_method 0 2024-09-11T01:36:09.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d2d63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:09.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d2d63640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:09.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d2d63640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:09.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d1d61640 10 monclient(hunting): get_auth_request con 0x7fa9cc134ad0 auth_method 0 2024-09-11T01:36:09.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d1d61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:09.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d1d61640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:09.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d1d61640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:09.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d2d63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:09.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d2d63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:09.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d2d63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:09.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d1d61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:09.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d1d61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:09.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d1d61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:09.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d2562640 1 --2- 172.21.15.12:0/4282313098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9cc0a2200 0x7fa9cc1c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:09.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d2562640 10 monclient(hunting): get_auth_request con 0x7fa9cc0a2200 auth_method 0 2024-09-11T01:36:09.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d2562640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:09.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d2562640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:09.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.052+0000 7fa9d2562640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:09.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.053+0000 7fa9d2d63640 10 monclient(hunting): handle_auth_done global_id 54297 payload 824 2024-09-11T01:36:09.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.053+0000 7fa9d2d63640 1 -- 172.21.15.12:0/4282313098 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9cc134ad0 msgr2=0x7fa9cc1c9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:09.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.053+0000 7fa9d2d63640 1 --2- 172.21.15.12:0/4282313098 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9cc134ad0 0x7fa9cc1c9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:09.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.053+0000 7fa9d2d63640 1 -- 172.21.15.12:0/4282313098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9cc0a2200 msgr2=0x7fa9cc1c9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:09.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.053+0000 7fa9d2d63640 1 --2- 172.21.15.12:0/4282313098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9cc0a2200 0x7fa9cc1c9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:09.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.053+0000 7fa9d2d63640 10 monclient: _finish_hunting 0 2024-09-11T01:36:09.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.053+0000 7fa9d2d63640 1 monclient: found mon.noname-a 2024-09-11T01:36:09.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.053+0000 7fa9d2d63640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:09.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.053+0000 7fa9d2d63640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:09.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.053+0000 7fa9d2d63640 1 -- 172.21.15.12:0/4282313098 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9cc1346c0 con 0x7fa9cc1355c0 2024-09-11T01:36:09.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.053+0000 7fa9d2562640 1 --2- 172.21.15.12:0/4282313098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9cc0a2200 0x7fa9cc1c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:36:09.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.053+0000 7fa9d1d61640 1 --2- 172.21.15.12:0/4282313098 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9cc134ad0 0x7fa9cc1c9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:36:09.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.055+0000 7fa9d2d63640 1 -- 172.21.15.12:0/4282313098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9cc1355c0 msgr2=0x7fa9cc120310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:09.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.055+0000 7fa9d2d63640 1 --2- 172.21.15.12:0/4282313098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9cc1355c0 0x7fa9cc120310 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa9c400d170 tx=0x7fa9c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:09.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.055+0000 7fa9beffd640 1 -- 172.21.15.12:0/4282313098 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c4043020 con 0x7fa9cc1355c0 2024-09-11T01:36:09.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.055+0000 7fa9beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:09.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.055+0000 7fa9beffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:09.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.055+0000 7fa9beffd640 10 monclient: dump: 2024-09-11T01:36:09.097 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:09.097 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:09.097 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:09.097 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:09.097 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:09.097 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:09.098 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:09.098 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:09.098 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:09.099 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:09.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.055+0000 7fa9beffd640 10 monclient: _finish_auth 0 2024-09-11T01:36:09.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.055+0000 7fa9beffd640 10 monclient: _check_auth_tickets 2024-09-11T01:36:09.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.055+0000 7fa9beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:09.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.055+0000 7fa9beffd640 1 -- 172.21.15.12:0/4282313098 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9c4033ac0 con 0x7fa9cc1355c0 2024-09-11T01:36:09.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.055+0000 7fa9beffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:09.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.055+0000 7fa9beffd640 1 -- 172.21.15.12:0/4282313098 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c4038040 con 0x7fa9cc1355c0 2024-09-11T01:36:09.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.055+0000 7fa9d47c6640 5 monclient: authenticate success, global_id 54297 2024-09-11T01:36:09.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.055+0000 7fa9d47c6640 10 monclient: _renew_subs 2024-09-11T01:36:09.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.055+0000 7fa9d47c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:09.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.055+0000 7fa9d47c6640 1 -- 172.21.15.12:0/4282313098 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9cc1240f0 con 0x7fa9cc1355c0 2024-09-11T01:36:09.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.055+0000 7fa9d0d5f640 4 set_mon_vals no callback set 2024-09-11T01:36:09.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.055+0000 7fa9beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:09.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.055+0000 7fa9beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:09.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.055+0000 7fa9beffd640 10 monclient: dump: 2024-09-11T01:36:09.101 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:09.102 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:09.102 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:09.102 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:09.102 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:09.102 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:09.103 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:09.103 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:09.103 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:09.103 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:09.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.055+0000 7fa9d47c6640 10 monclient: _renew_subs 2024-09-11T01:36:09.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.055+0000 7fa9d47c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:09.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.055+0000 7fa9d47c6640 1 -- 172.21.15.12:0/4282313098 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9cc123e10 con 0x7fa9cc1355c0 2024-09-11T01:36:09.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.056+0000 7fa9beffd640 1 -- 172.21.15.12:0/4282313098 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fa9c403e470 con 0x7fa9cc1355c0 2024-09-11T01:36:09.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.056+0000 7fa9d47c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:09.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.056+0000 7fa9d47c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:09.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.056+0000 7fa9d47c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:09.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.056+0000 7fa9d47c6640 1 -- 172.21.15.12:0/4282313098 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9cc134eb0 con 0x7fa9cc1355c0 2024-09-11T01:36:09.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.056+0000 7fa9beffd640 1 --2- 172.21.15.12:0/4282313098 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa9b003e240 0x7fa9b0040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:09.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.056+0000 7fa9beffd640 1 -- 172.21.15.12:0/4282313098 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(223..223 src has 186..223) ==== 4799+0+0 (secure 0 0 0) 0x7fa9c4075da0 con 0x7fa9cc1355c0 2024-09-11T01:36:09.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.057+0000 7fa9d2562640 1 --2- 172.21.15.12:0/4282313098 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa9b003e240 0x7fa9b0040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:09.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.057+0000 7fa9d2562640 10 monclient: get_auth_request con 0x7fa9b003e240 auth_method 0 2024-09-11T01:36:09.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.059+0000 7fa9d2562640 1 --2- 172.21.15.12:0/4282313098 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa9b003e240 0x7fa9b0040700 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8005160 tx=0x7fa9c8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:09.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.061+0000 7fa9beffd640 1 -- 172.21.15.12:0/4282313098 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa9c404d070 con 0x7fa9cc1355c0 2024-09-11T01:36:09.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.061+0000 7fa9beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:09.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.061+0000 7fa9beffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:09.235 INFO:tasks.rados.rados.0.smithi071.stdout:967: done (4 left) 2024-09-11T01:36:09.235 INFO:tasks.rados.rados.0.smithi071.stdout:968: done (3 left) 2024-09-11T01:36:09.235 INFO:tasks.rados.rados.0.smithi071.stdout:969: done (2 left) 2024-09-11T01:36:09.235 INFO:tasks.rados.rados.0.smithi071.stdout:970: done (1 left) 2024-09-11T01:36:09.235 INFO:tasks.rados.rados.0.smithi071.stdout:974: done (0 left) 2024-09-11T01:36:09.236 INFO:tasks.rados.rados.0.smithi071.stdout:975: read oid 20 snap 98 2024-09-11T01:36:09.236 INFO:tasks.rados.rados.0.smithi071.stdout:975: expect (ObjNum 323 snap 92 seq_num 323) 2024-09-11T01:36:09.236 INFO:tasks.rados.rados.0.smithi071.stdout:976: rollback oid 42 current snap is 102 2024-09-11T01:36:09.236 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 42 to 91 2024-09-11T01:36:09.236 INFO:tasks.rados.rados.0.smithi071.stdout:977: read oid 9 snap -1 2024-09-11T01:36:09.236 INFO:tasks.rados.rados.0.smithi071.stdout:977: expect (ObjNum 336 snap 96 seq_num 336) 2024-09-11T01:36:09.236 INFO:tasks.rados.rados.0.smithi071.stdout:978: delete oid 50 current snap is 102 2024-09-11T01:36:09.241 INFO:tasks.rados.rados.0.smithi071.stdout:976: finishing rollback tid 1 to smithi07134777-42 2024-09-11T01:36:09.241 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 0 (ObjNum 97 snap 0 seq_num 1) dirty dne 2024-09-11T01:36:09.242 INFO:tasks.rados.rados.0.smithi071.stdout:975: expect (ObjNum 323 snap 92 seq_num 323) 2024-09-11T01:36:09.242 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:36:09.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.329+0000 7fa9d47c6640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:36:09.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.329+0000 7fa9d47c6640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:36:09.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.329+0000 7fa9d47c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:09.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.329+0000 7fa9d47c6640 1 -- 172.21.15.12:0/4282313098 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fa9cc1228f0 con 0x7fa9cc1355c0 2024-09-11T01:36:09.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.330+0000 7fa9beffd640 1 -- 172.21.15.12:0/4282313098 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fa9c4074790 con 0x7fa9cc1355c0 2024-09-11T01:36:09.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.330+0000 7fa9beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:36:09.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.330+0000 7fa9beffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:09.333 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.331+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.333 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.331+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=10,kv=12,logm=62} crc {auth=1402955452,config=2586702516,health=1228072951,kv=451678197,logm=1963743071}) 2024-09-11T01:36:09.333 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.331+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=13,config=2,health=11,kv=12,logm=62} crc {auth=1827111481,config=2586702516,health=649431983,kv=3263697420,logm=4273711429}) 2024-09-11T01:36:09.333 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.331+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.333 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.331+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=10,kv=12,logm=62} crc {auth=1402955452,config=2586702516,health=1228072951,kv=451678197,logm=1963743071}) 2024-09-11T01:36:09.333 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.331+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=12,kv=12,logm=61} crc {auth=2704632225,config=2586702516,health=3837507830,kv=3736942389,logm=1122501222}) 2024-09-11T01:36:09.334 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.332+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.334 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.332+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3571634582}) 2024-09-11T01:36:09.334 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.332+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2811385259}) 2024-09-11T01:36:09.334 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.332+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.334 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.332+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3571634582}) 2024-09-11T01:36:09.334 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.332+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2184481269}) 2024-09-11T01:36:09.334 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.333+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.335 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.333+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4102490900}) 2024-09-11T01:36:09.335 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.333+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=965886962}) 2024-09-11T01:36:09.335 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.333+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.335 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.333+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4102490900}) 2024-09-11T01:36:09.335 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.333+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=995414796}) 2024-09-11T01:36:09.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.333+0000 7fa9d47c6640 1 -- 172.21.15.12:0/4282313098 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa9b003e240 msgr2=0x7fa9b0040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:09.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.333+0000 7fa9d47c6640 1 --2- 172.21.15.12:0/4282313098 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa9b003e240 0x7fa9b0040700 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8005160 tx=0x7fa9c8009040 comp rx=0 tx=0).stop 2024-09-11T01:36:09.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.333+0000 7fa9d47c6640 10 monclient: shutdown 2024-09-11T01:36:09.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.333+0000 7fa9d47c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:09.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.333+0000 7fa9d47c6640 1 -- 172.21.15.12:0/4282313098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9cc1355c0 msgr2=0x7fa9cc120310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:09.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.333+0000 7fa9d47c6640 1 --2- 172.21.15.12:0/4282313098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9cc1355c0 0x7fa9cc120310 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa9c400d170 tx=0x7fa9c4004a40 comp rx=0 tx=0).stop 2024-09-11T01:36:09.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.333+0000 7fa9d47c6640 1 -- 172.21.15.12:0/4282313098 shutdown_connections 2024-09-11T01:36:09.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.333+0000 7fa9d47c6640 1 --2- 172.21.15.12:0/4282313098 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa9b003e240 0x7fa9b0040700 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:09.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.333+0000 7fa9d47c6640 1 --2- 172.21.15.12:0/4282313098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9cc1355c0 0x7fa9cc120310 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:09.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.333+0000 7fa9d47c6640 1 --2- 172.21.15.12:0/4282313098 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9cc134ad0 0x7fa9cc1c9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:09.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.333+0000 7fa9d47c6640 1 --2- 172.21.15.12:0/4282313098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9cc0a2200 0x7fa9cc1c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:09.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.333+0000 7fa9d47c6640 1 -- 172.21.15.12:0/4282313098 >> 172.21.15.12:0/4282313098 conn(0x7fa9cc0a2f70 msgr2=0x7fa9cc09d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:09.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.333+0000 7fa9d47c6640 1 -- 172.21.15.12:0/4282313098 shutdown_connections 2024-09-11T01:36:09.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:09.333+0000 7fa9d47c6640 1 -- 172.21.15.12:0/4282313098 wait complete. 2024-09-11T01:36:09.338 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.334+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.338 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.334+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1535747854}) 2024-09-11T01:36:09.338 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.334+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1058892707}) 2024-09-11T01:36:09.338 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.334+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.338 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.334+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1535747854}) 2024-09-11T01:36:09.338 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.334+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2325630416}) 2024-09-11T01:36:09.339 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.335+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.339 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.335+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=890738580}) 2024-09-11T01:36:09.339 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.335+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2254860452}) 2024-09-11T01:36:09.339 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.335+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.339 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.335+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=890738580}) 2024-09-11T01:36:09.339 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.335+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3623153676}) 2024-09-11T01:36:09.339 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.335+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.339 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.335+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=277822396}) 2024-09-11T01:36:09.339 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.335+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2630761526}) 2024-09-11T01:36:09.339 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.335+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.339 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.335+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=277822396}) 2024-09-11T01:36:09.339 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.335+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3003414463}) 2024-09-11T01:36:09.345 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:36:09.388 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 743889 2024-09-11T01:36:09.444 INFO:tasks.rados.rados.0.smithi071.stdout:975: done (3 left) 2024-09-11T01:36:09.444 INFO:tasks.rados.rados.0.smithi071.stdout:976: done (2 left) 2024-09-11T01:36:09.444 INFO:tasks.rados.rados.0.smithi071.stdout:978: done (1 left) 2024-09-11T01:36:09.444 INFO:tasks.rados.rados.0.smithi071.stdout:979: read oid 35 snap 101 2024-09-11T01:36:09.444 INFO:tasks.rados.rados.0.smithi071.stdout:979: expect (ObjNum 312 snap 87 seq_num 312) 2024-09-11T01:36:09.444 INFO:tasks.rados.rados.0.smithi071.stdout:980: delete oid 23 current snap is 102 2024-09-11T01:36:09.444 INFO:tasks.rados.rados.0.smithi071.stdout:977: expect (ObjNum 336 snap 96 seq_num 336) 2024-09-11T01:36:09.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.473+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.473+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=749179893}) 2024-09-11T01:36:09.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.473+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3459986764}) 2024-09-11T01:36:09.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.473+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.473+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=749179893}) 2024-09-11T01:36:09.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.473+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1879224676}) 2024-09-11T01:36:09.476 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.473+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.476 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.473+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4220330510}) 2024-09-11T01:36:09.476 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.473+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=627485606}) 2024-09-11T01:36:09.476 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.473+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.476 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.474+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4220330510}) 2024-09-11T01:36:09.476 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.474+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1900138731}) 2024-09-11T01:36:09.476 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.474+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.476 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.474+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1334877699}) 2024-09-11T01:36:09.476 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.474+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3880827673}) 2024-09-11T01:36:09.476 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.474+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.476 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.474+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1334877699}) 2024-09-11T01:36:09.477 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.474+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3253718997}) 2024-09-11T01:36:09.478 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.475+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.478 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.475+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1236189871}) 2024-09-11T01:36:09.478 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.475+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=678569603}) 2024-09-11T01:36:09.478 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.475+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.478 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.475+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1236189871}) 2024-09-11T01:36:09.478 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.475+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=94649608}) 2024-09-11T01:36:09.478 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.476+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.478 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.476+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4125622027}) 2024-09-11T01:36:09.478 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.476+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2498074522}) 2024-09-11T01:36:09.478 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.476+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.479 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.476+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4125622027}) 2024-09-11T01:36:09.479 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.476+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=104843440}) 2024-09-11T01:36:09.479 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.476+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.479 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.476+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2783613887}) 2024-09-11T01:36:09.479 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.476+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1830686406}) 2024-09-11T01:36:09.479 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.476+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.479 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.476+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2783613887}) 2024-09-11T01:36:09.479 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.476+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2434559364}) 2024-09-11T01:36:09.481 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.477+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.481 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.477+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=793559186}) 2024-09-11T01:36:09.481 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.477+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2349360163}) 2024-09-11T01:36:09.481 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.477+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.481 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.477+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=793559186}) 2024-09-11T01:36:09.481 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.477+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4257779512}) 2024-09-11T01:36:09.481 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.478+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.481 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.478+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145056424}) 2024-09-11T01:36:09.481 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.478+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3126262090}) 2024-09-11T01:36:09.482 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.478+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.482 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.478+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145056424}) 2024-09-11T01:36:09.482 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.478+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1433150970}) 2024-09-11T01:36:09.482 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.478+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.482 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.479+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=413883432}) 2024-09-11T01:36:09.482 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.479+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=178129584}) 2024-09-11T01:36:09.482 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.479+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.482 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.479+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=413883432}) 2024-09-11T01:36:09.482 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.479+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3485068131}) 2024-09-11T01:36:09.482 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.479+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.482 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.479+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3173427441}) 2024-09-11T01:36:09.482 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.479+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2159887891}) 2024-09-11T01:36:09.482 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.479+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.482 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.479+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3173427441}) 2024-09-11T01:36:09.483 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.479+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4187715403}) 2024-09-11T01:36:09.483 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.480+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.484 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.480+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=993694974,mdsmap=2682192099}) 2024-09-11T01:36:09.484 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.480+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=2225786132,mdsmap=2682192099}) 2024-09-11T01:36:09.484 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.480+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.484 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.480+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=993694974,mdsmap=2682192099}) 2024-09-11T01:36:09.484 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.480+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1814383802}) 2024-09-11T01:36:09.484 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.481+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.484 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.481+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=7} crc {mdsmap=3140514009,mgr=4080174641,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=15673428,mon_config_key=3330371543,monmap=3140514009,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=3665319007,osd_snap=810101447,osdmap=1620903012}) 2024-09-11T01:36:09.484 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.481+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mdsmap=1,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=12} crc {mdsmap=1228700967,mgr=2672916156,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2105484724,mon_config_key=3330371543,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=3665319007,osd_snap=2058421305,osdmap=2096932793}) 2024-09-11T01:36:09.484 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.481+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.484 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.481+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=7} crc {mdsmap=3140514009,mgr=4080174641,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=15673428,mon_config_key=3330371543,monmap=3140514009,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=3665319007,osd_snap=810101447,osdmap=1620903012}) 2024-09-11T01:36:09.484 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.481+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=43,osdmap=6} crc {mdsmap=3140514009,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=55585028,mon_config_key=508202868,monmap=1557804916,nvmeofgw=598337686,osd_metadata=118787097,osd_pg_creating=3665319007,osd_snap=187616840,osdmap=538511488}) 2024-09-11T01:36:09.486 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.484+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.486 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.484+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=64} crc {osdmap=3335429493}) 2024-09-11T01:36:09.486 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.484+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=69} crc {osdmap=264257101}) 2024-09-11T01:36:09.486 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.484+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:09.486 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.484+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=64} crc {osdmap=3335429493}) 2024-09-11T01:36:09.486 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:09.484+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=69} crc {osdmap=4243357816}) 2024-09-11T01:36:09.552 INFO:tasks.rados.rados.0.smithi071.stdout:977: done (2 left) 2024-09-11T01:36:09.552 INFO:tasks.rados.rados.0.smithi071.stdout:980: done (1 left) 2024-09-11T01:36:09.552 INFO:tasks.rados.rados.0.smithi071.stdout:981: delete oid 6 current snap is 102 2024-09-11T01:36:09.552 INFO:tasks.rados.rados.0.smithi071.stdout:979: expect (ObjNum 312 snap 87 seq_num 312) 2024-09-11T01:36:09.585 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:36:09.619 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:36:09.653 INFO:tasks.rados.rados.0.smithi071.stdout:979: done (1 left) 2024-09-11T01:36:09.653 INFO:tasks.rados.rados.0.smithi071.stdout:981: done (0 left) 2024-09-11T01:36:09.653 INFO:tasks.rados.rados.0.smithi071.stdout:982: rollback oid 29 current snap is 102 2024-09-11T01:36:09.653 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 29 to 98 2024-09-11T01:36:09.653 INFO:tasks.rados.rados.0.smithi071.stdout:983: snap_create 2024-09-11T01:36:09.656 INFO:tasks.rados.rados.0.smithi071.stdout:982: finishing rollback tid 0 to smithi07134777-29 2024-09-11T01:36:09.657 INFO:tasks.rados.rados.0.smithi071.stdout:982: finishing rollback tid 1 to smithi07134777-29 2024-09-11T01:36:09.659 INFO:tasks.rados.rados.0.smithi071.stdout:982: finishing rollback tid 2 to smithi07134777-29 2024-09-11T01:36:09.659 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 488 (ObjNum 226 snap 55 seq_num 226) dirty exists 2024-09-11T01:36:10.243 INFO:tasks.rados.rados.0.smithi071.stdout:982: done (1 left) 2024-09-11T01:36:10.243 INFO:tasks.rados.rados.0.smithi071.stdout:983: done (0 left) 2024-09-11T01:36:10.243 INFO:tasks.rados.rados.0.smithi071.stdout:984: snap_create 2024-09-11T01:36:10.346 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:36:10.347 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:36:10.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.458+0000 7f195f12b640 10 monclient: get_monmap_and_config 2024-09-11T01:36:10.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.458+0000 7f195f12b640 10 monclient: build_initial_monmap 2024-09-11T01:36:10.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.458+0000 7f195f12b640 10 monclient: monmap: 2024-09-11T01:36:10.460 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:10.460 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:10.460 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:10.459477+0000 2024-09-11T01:36:10.461 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:10.459477+0000 2024-09-11T01:36:10.461 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:10.461 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:10.461 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:10.461 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:10.461 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:10.461 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:10.461 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:10.461 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.459+0000 7f195f12b640 1 Processor -- start 2024-09-11T01:36:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.459+0000 7f195f12b640 1 -- start start 2024-09-11T01:36:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.459+0000 7f195f12b640 10 monclient: init 2024-09-11T01:36:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195f12b640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195f12b640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-11T01:36:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195f12b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1958138230 0x7f1958138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195f12b640 10 monclient(hunting): picked mon.noname-b con 0x7f1958138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:10.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195f12b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f195809d030 0x7f19580a5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:10.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195f12b640 10 monclient(hunting): picked mon.noname-a con 0x7f195809d030 addr v2:172.21.15.12:3300/0 2024-09-11T01:36:10.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195f12b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f195809d430 0x7f19580a5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:10.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195f12b640 10 monclient(hunting): picked mon.noname-c con 0x7f195809d430 addr v2:172.21.15.71:3300/0 2024-09-11T01:36:10.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195f12b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:10.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195f12b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f195809c9c0 con 0x7f195809d030 2024-09-11T01:36:10.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195f12b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:10.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195f12b640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f19580a6e30 con 0x7f195809d430 2024-09-11T01:36:10.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195f12b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:10.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195f12b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f195809ba20 con 0x7f1958138230 2024-09-11T01:36:10.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195f12b640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:10.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195f12b640 10 monclient(hunting): authenticate will time out at 1176.902216s 2024-09-11T01:36:10.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195cec7640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1958138230 0x7f1958138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:10.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195cec7640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1958138230 0x7f1958138610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:49534/0 (socket says 172.21.15.12:49534) 2024-09-11T01:36:10.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f1957fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f195809d030 0x7f19580a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:10.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195cec7640 1 -- 172.21.15.12:0/4184727363 learned_addr learned my addr 172.21.15.12:0/4184727363 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:10.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195cec7640 10 monclient(hunting): get_auth_request con 0x7f1958138230 auth_method 0 2024-09-11T01:36:10.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195cec7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:10.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195cec7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:10.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195cec7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:10.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f1957fff640 10 monclient(hunting): get_auth_request con 0x7f195809d030 auth_method 0 2024-09-11T01:36:10.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f1957fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:10.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f1957fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f1957fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195cec7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195cec7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195cec7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f1957fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f1957fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f1957fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195d6c8640 1 --2- 172.21.15.12:0/4184727363 >> v2:172.21.15.71:3300/0 conn(0x7f195809d430 0x7f19580a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195d6c8640 10 monclient(hunting): get_auth_request con 0x7f195809d430 auth_method 0 2024-09-11T01:36:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195d6c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195d6c8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195d6c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f1957fff640 10 monclient(hunting): handle_auth_done global_id 54300 payload 274 2024-09-11T01:36:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f1957fff640 1 -- 172.21.15.12:0/4184727363 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1958138230 msgr2=0x7f1958138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f1957fff640 1 --2- 172.21.15.12:0/4184727363 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1958138230 0x7f1958138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f1957fff640 1 -- 172.21.15.12:0/4184727363 >> v2:172.21.15.71:3300/0 conn(0x7f195809d430 msgr2=0x7f19580a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f1957fff640 1 --2- 172.21.15.12:0/4184727363 >> v2:172.21.15.71:3300/0 conn(0x7f195809d430 0x7f19580a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f1957fff640 10 monclient: _finish_hunting 0 2024-09-11T01:36:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f1957fff640 1 monclient: found mon.noname-a 2024-09-11T01:36:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f1957fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f1957fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f1957fff640 1 -- 172.21.15.12:0/4184727363 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19580a66a0 con 0x7f195809d030 2024-09-11T01:36:10.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.460+0000 7f195cec7640 1 --2- 172.21.15.12:0/4184727363 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1958138230 0x7f1958138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:36:10.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.463+0000 7f1957fff640 1 -- 172.21.15.12:0/4184727363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195809d030 msgr2=0x7f19580a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:10.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.463+0000 7f1957fff640 1 --2- 172.21.15.12:0/4184727363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195809d030 0x7f19580a5020 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f195809cf80 tx=0x7f194802a4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b06d63a2f9cfb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:10.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.463+0000 7f19577fe640 1 -- 172.21.15.12:0/4184727363 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1948007950 con 0x7f195809d030 2024-09-11T01:36:10.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.463+0000 7f19577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:10.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.463+0000 7f19577fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:10.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.463+0000 7f19577fe640 10 monclient: dump: 2024-09-11T01:36:10.475 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:10.475 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:10.475 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:10.475 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:10.475 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:10.475 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:10.475 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:10.475 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:10.475 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:10.475 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:10.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.463+0000 7f19577fe640 10 monclient: _finish_auth 0 2024-09-11T01:36:10.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.463+0000 7f19577fe640 10 monclient: _check_auth_tickets 2024-09-11T01:36:10.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.463+0000 7f19577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:10.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.463+0000 7f19577fe640 1 -- 172.21.15.12:0/4184727363 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1948007af0 con 0x7f195809d030 2024-09-11T01:36:10.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.463+0000 7f195f12b640 5 monclient: authenticate success, global_id 54300 2024-09-11T01:36:10.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.463+0000 7f195f12b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:36:10.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.463+0000 7f19577fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:10.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.463+0000 7f195f12b640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:10.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.463+0000 7f19577fe640 1 -- 172.21.15.12:0/4184727363 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1948031400 con 0x7f195809d030 2024-09-11T01:36:10.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.463+0000 7f19577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:10.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.463+0000 7f195f12b640 4 set_mon_vals no callback set 2024-09-11T01:36:10.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.463+0000 7f19577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:10.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.463+0000 7f19577fe640 10 monclient: dump: 2024-09-11T01:36:10.476 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:10.476 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:10.476 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:10.477 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:10.477 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:10.477 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:10.477 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:10.477 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:10.477 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:10.477 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:10.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.464+0000 7f195f12b640 10 monclient: shutdown 2024-09-11T01:36:10.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.464+0000 7f195f12b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:10.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.464+0000 7f195f12b640 1 -- 172.21.15.12:0/4184727363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195809d030 msgr2=0x7f19580a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:10.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.464+0000 7f195f12b640 1 --2- 172.21.15.12:0/4184727363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195809d030 0x7f19580a5020 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f195809cf80 tx=0x7f194802a4b0 comp rx=0 tx=0).stop 2024-09-11T01:36:10.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.464+0000 7f195f12b640 1 -- 172.21.15.12:0/4184727363 shutdown_connections 2024-09-11T01:36:10.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.464+0000 7f195f12b640 1 --2- 172.21.15.12:0/4184727363 >> v2:172.21.15.71:3300/0 conn(0x7f195809d430 0x7f19580a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:10.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.464+0000 7f195f12b640 1 --2- 172.21.15.12:0/4184727363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195809d030 0x7f19580a5020 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:10.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.464+0000 7f195f12b640 1 --2- 172.21.15.12:0/4184727363 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1958138230 0x7f1958138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:10.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.464+0000 7f195f12b640 1 -- 172.21.15.12:0/4184727363 >> 172.21.15.12:0/4184727363 conn(0x7f19580a7120 msgr2=0x7f19580a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:10.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.464+0000 7f195f12b640 1 -- 172.21.15.12:0/4184727363 shutdown_connections 2024-09-11T01:36:10.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.464+0000 7f195f12b640 1 -- 172.21.15.12:0/4184727363 wait complete. 2024-09-11T01:36:10.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.464+0000 7f195f12b640 10 monclient: build_initial_monmap 2024-09-11T01:36:10.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.465+0000 7f195f12b640 10 monclient: monmap: 2024-09-11T01:36:10.478 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:10.478 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:10.478 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:10.478 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:10.478 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:10.478 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:10.478 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:10.478 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:10.478 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:10.479 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:10.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.465+0000 7f195f12b640 1 Processor -- start 2024-09-11T01:36:10.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.465+0000 7f195f12b640 1 -- start start 2024-09-11T01:36:10.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.465+0000 7f195f12b640 10 monclient: init 2024-09-11T01:36:10.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.465+0000 7f195f12b640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:10.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.465+0000 7f195f12b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:36:10.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195f12b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195809d030 0x7f19581cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:10.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195f12b640 10 monclient(hunting): picked mon.noname-a con 0x7f195809d030 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:10.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195f12b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f195809d430 0x7f1958136ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:10.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195f12b640 10 monclient(hunting): picked mon.noname-b con 0x7f195809d430 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:10.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195f12b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1958138230 0x7f1958137410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:10.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195f12b640 10 monclient(hunting): picked mon.noname-c con 0x7f1958138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:10.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195f12b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:10.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195f12b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f195806f970 con 0x7f195809d030 2024-09-11T01:36:10.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195f12b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:10.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195f12b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f195806f7f0 con 0x7f195809d430 2024-09-11T01:36:10.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195f12b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:10.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195f12b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f19580a5aa0 con 0x7f1958138230 2024-09-11T01:36:10.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195f12b640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:10.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195f12b640 10 monclient(hunting): authenticate will time out at 1176.908092s 2024-09-11T01:36:10.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195cec7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195809d030 0x7f19581cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:10.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195cec7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195809d030 0x7f19581cccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48956/0 (socket says 172.21.15.12:48956) 2024-09-11T01:36:10.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195d6c8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1958138230 0x7f1958137410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:10.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195cec7640 1 -- 172.21.15.12:0/2948804698 learned_addr learned my addr 172.21.15.12:0/2948804698 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:10.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195cec7640 10 monclient(hunting): get_auth_request con 0x7f195809d030 auth_method 0 2024-09-11T01:36:10.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195cec7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:10.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195cec7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:10.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195cec7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:10.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195d6c8640 10 monclient(hunting): get_auth_request con 0x7f1958138230 auth_method 0 2024-09-11T01:36:10.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195d6c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:10.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195d6c8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:10.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195d6c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:10.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195cec7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:10.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195cec7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:10.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195cec7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:10.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195d6c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:10.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195d6c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:10.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195d6c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:10.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f1957fff640 1 --2- 172.21.15.12:0/2948804698 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f195809d430 0x7f1958136ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:10.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f1957fff640 10 monclient(hunting): get_auth_request con 0x7f195809d430 auth_method 0 2024-09-11T01:36:10.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f1957fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:10.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f1957fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:10.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f1957fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:10.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195d6c8640 10 monclient(hunting): handle_auth_done global_id 64205 payload 824 2024-09-11T01:36:10.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195d6c8640 1 -- 172.21.15.12:0/2948804698 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f195809d430 msgr2=0x7f1958136ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:10.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195d6c8640 1 --2- 172.21.15.12:0/2948804698 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f195809d430 0x7f1958136ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:10.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195d6c8640 1 -- 172.21.15.12:0/2948804698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195809d030 msgr2=0x7f19581cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:10.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195d6c8640 1 --2- 172.21.15.12:0/2948804698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195809d030 0x7f19581cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:10.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195d6c8640 10 monclient: _finish_hunting 0 2024-09-11T01:36:10.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195d6c8640 1 monclient: found mon.noname-c 2024-09-11T01:36:10.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195d6c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:10.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195d6c8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:36:10.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195d6c8640 1 -- 172.21.15.12:0/2948804698 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19581cd210 con 0x7f1958138230 2024-09-11T01:36:10.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.466+0000 7f195cec7640 1 --2- 172.21.15.12:0/2948804698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195809d030 0x7f19581cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:36:10.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.468+0000 7f195d6c8640 1 -- 172.21.15.12:0/2948804698 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1958138230 msgr2=0x7f1958137410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:10.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.468+0000 7f195d6c8640 1 --2- 172.21.15.12:0/2948804698 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1958138230 0x7f1958137410 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f194000d530 tx=0x7f194000d9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:10.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.469+0000 7f19557fa640 1 -- 172.21.15.12:0/2948804698 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1940012800 con 0x7f1958138230 2024-09-11T01:36:10.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.469+0000 7f19557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:10.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.469+0000 7f19557fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:36:10.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.469+0000 7f19557fa640 10 monclient: dump: 2024-09-11T01:36:10.483 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:10.483 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:10.483 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:10.483 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:10.483 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:10.483 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:10.483 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:10.483 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:10.484 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:10.484 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:10.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.469+0000 7f19557fa640 10 monclient: _finish_auth 0 2024-09-11T01:36:10.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.469+0000 7f19557fa640 10 monclient: _check_auth_tickets 2024-09-11T01:36:10.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.469+0000 7f19557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:10.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.469+0000 7f19557fa640 1 -- 172.21.15.12:0/2948804698 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1940019070 con 0x7f1958138230 2024-09-11T01:36:10.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.469+0000 7f195f12b640 5 monclient: authenticate success, global_id 64205 2024-09-11T01:36:10.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.469+0000 7f195f12b640 10 monclient: _renew_subs 2024-09-11T01:36:10.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.469+0000 7f195f12b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:36:10.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.469+0000 7f195f12b640 1 -- 172.21.15.12:0/2948804698 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1958126570 con 0x7f1958138230 2024-09-11T01:36:10.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.469+0000 7f19557fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:10.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.469+0000 7f19557fa640 1 -- 172.21.15.12:0/2948804698 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1940012bc0 con 0x7f1958138230 2024-09-11T01:36:10.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.469+0000 7f19557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:10.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.469+0000 7f19557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:10.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.469+0000 7f19557fa640 10 monclient: dump: 2024-09-11T01:36:10.485 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:10.485 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:10.485 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:10.485 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:10.485 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:10.485 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:10.485 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:10.485 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:10.485 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:10.485 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:10.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.469+0000 7f195f12b640 10 monclient: _renew_subs 2024-09-11T01:36:10.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.469+0000 7f195f12b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:36:10.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.469+0000 7f195f12b640 1 -- 172.21.15.12:0/2948804698 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1958128030 con 0x7f1958138230 2024-09-11T01:36:10.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.469+0000 7f1956ffd640 4 set_mon_vals no callback set 2024-09-11T01:36:10.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.470+0000 7f19557fa640 1 -- 172.21.15.12:0/2948804698 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f1940013070 con 0x7f1958138230 2024-09-11T01:36:10.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.470+0000 7f195f12b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:10.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.470+0000 7f195f12b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:10.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.470+0000 7f195f12b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:36:10.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.470+0000 7f195f12b640 1 -- 172.21.15.12:0/2948804698 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1958138610 con 0x7f1958138230 2024-09-11T01:36:10.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.470+0000 7f19557fa640 1 --2- 172.21.15.12:0/2948804698 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f193803e260 0x7f1938040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:10.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.470+0000 7f19557fa640 1 -- 172.21.15.12:0/2948804698 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(225..225 src has 186..225) ==== 4771+0+0 (secure 0 0 0) 0x7f1940052d20 con 0x7f1958138230 2024-09-11T01:36:10.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.470+0000 7f195cec7640 1 --2- 172.21.15.12:0/2948804698 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f193803e260 0x7f1938040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:10.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.470+0000 7f195cec7640 10 monclient: get_auth_request con 0x7f193803e260 auth_method 0 2024-09-11T01:36:10.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.473+0000 7f195cec7640 1 --2- 172.21.15.12:0/2948804698 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f193803e260 0x7f1938040720 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1950009630 tx=0x7f1950009240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:10.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.475+0000 7f19557fa640 1 -- 172.21.15.12:0/2948804698 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1940051720 con 0x7f1958138230 2024-09-11T01:36:10.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.475+0000 7f19557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:10.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.475+0000 7f19557fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:10.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.733+0000 7f195f12b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:36:10.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.733+0000 7f195f12b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:36:10.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.733+0000 7f195f12b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:36:10.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.733+0000 7f195f12b640 1 -- 172.21.15.12:0/2948804698 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1958137950 con 0x7f1958138230 2024-09-11T01:36:10.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.733+0000 7f19557fa640 1 -- 172.21.15.12:0/2948804698 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f1940012d60 con 0x7f1958138230 2024-09-11T01:36:10.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.733+0000 7f19557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:36:10.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.733+0000 7f19557fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:10.736 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":74,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:36:10.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.735+0000 7f195f12b640 1 -- 172.21.15.12:0/2948804698 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f193803e260 msgr2=0x7f1938040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:10.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.735+0000 7f195f12b640 1 --2- 172.21.15.12:0/2948804698 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f193803e260 0x7f1938040720 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1950009630 tx=0x7f1950009240 comp rx=0 tx=0).stop 2024-09-11T01:36:10.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.735+0000 7f195f12b640 10 monclient: shutdown 2024-09-11T01:36:10.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.735+0000 7f195f12b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:10.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.735+0000 7f195f12b640 1 -- 172.21.15.12:0/2948804698 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1958138230 msgr2=0x7f1958137410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:10.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.735+0000 7f195f12b640 1 --2- 172.21.15.12:0/2948804698 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1958138230 0x7f1958137410 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f194000d530 tx=0x7f194000d9f0 comp rx=0 tx=0).stop 2024-09-11T01:36:10.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.736+0000 7f195f12b640 1 -- 172.21.15.12:0/2948804698 shutdown_connections 2024-09-11T01:36:10.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.736+0000 7f195f12b640 1 --2- 172.21.15.12:0/2948804698 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1958138230 0x7f1958137410 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:10.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.736+0000 7f195f12b640 1 --2- 172.21.15.12:0/2948804698 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f195809d430 0x7f1958136ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:10.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.736+0000 7f195f12b640 1 --2- 172.21.15.12:0/2948804698 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f193803e260 0x7f1938040720 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:10.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.736+0000 7f195f12b640 1 --2- 172.21.15.12:0/2948804698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195809d030 0x7f19581cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:10.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.736+0000 7f195f12b640 1 -- 172.21.15.12:0/2948804698 >> 172.21.15.12:0/2948804698 conn(0x7f19580a7120 msgr2=0x7f19580a33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:10.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.736+0000 7f195f12b640 1 -- 172.21.15.12:0/2948804698 shutdown_connections 2024-09-11T01:36:10.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.736+0000 7f195f12b640 1 -- 172.21.15.12:0/2948804698 wait complete. 2024-09-11T01:36:10.747 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:36:10.748 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:36:10.748 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:36:10.748 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:36:10.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.848+0000 7f93373e2640 10 monclient: get_monmap_and_config 2024-09-11T01:36:10.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.848+0000 7f93373e2640 10 monclient: build_initial_monmap 2024-09-11T01:36:10.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.848+0000 7f93373e2640 10 monclient: monmap: 2024-09-11T01:36:10.850 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:10.851 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:10.851 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:10.849090+0000 2024-09-11T01:36:10.851 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:10.849090+0000 2024-09-11T01:36:10.851 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:10.851 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:10.851 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:10.851 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:10.851 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:10.851 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:10.851 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:10.851 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:10.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.848+0000 7f93373e2640 1 Processor -- start 2024-09-11T01:36:10.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f93373e2640 1 -- start start 2024-09-11T01:36:10.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f93373e2640 10 monclient: init 2024-09-11T01:36:10.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f93373e2640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:10.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f93373e2640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-11T01:36:10.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f93373e2640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9330134ad0 0x7f9330134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:10.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f93373e2640 10 monclient(hunting): picked mon.noname-c con 0x7f9330134ad0 addr v2:172.21.15.71:3300/0 2024-09-11T01:36:10.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f93373e2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93300a2200 0x7f93301353f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:10.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f93373e2640 10 monclient(hunting): picked mon.noname-b con 0x7f93300a2200 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:10.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f93373e2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9330133a50 addr v1:172.21.15.71:6789/0 2024-09-11T01:36:10.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f93373e2640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f933009ed30 con 0x7f9330133a50 2024-09-11T01:36:10.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f93373e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:10.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f93373e2640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f933009f5d0 con 0x7f9330134ad0 2024-09-11T01:36:10.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f93373e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:10.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f93373e2640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f9330135930 con 0x7f93300a2200 2024-09-11T01:36:10.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f93373e2640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:10.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f93373e2640 10 monclient(hunting): authenticate will time out at 1177.291637s 2024-09-11T01:36:10.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f933497d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93300a2200 0x7f93301353f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:10.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f933497d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93300a2200 0x7f93301353f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:49554/0 (socket says 172.21.15.12:49554) 2024-09-11T01:36:10.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f933497d640 1 -- 172.21.15.12:0/3139981826 learned_addr learned my addr 172.21.15.12:0/3139981826 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:10.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f933497d640 10 monclient(hunting): get_auth_request con 0x7f93300a2200 auth_method 0 2024-09-11T01:36:10.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f933497d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:10.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f933497d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:10.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f933497d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:10.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f933497d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:10.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f933497d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:10.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f933497d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:10.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.849+0000 7f933497d640 10 monclient(hunting): handle_auth_done global_id 64208 payload 274 2024-09-11T01:36:10.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.850+0000 7f933497d640 1 -- 172.21.15.12:0/3139981826 >> v2:172.21.15.71:3300/0 conn(0x7f9330134ad0 msgr2=0x7f9330134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:10.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.850+0000 7f933497d640 1 --2- 172.21.15.12:0/3139981826 >> v2:172.21.15.71:3300/0 conn(0x7f9330134ad0 0x7f9330134eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:10.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.850+0000 7f933497d640 1 -- 172.21.15.12:0/3139981826 >> v1:172.21.15.71:6789/0 conn(0x7f9330133a50 legacy=0x7f9330133ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:10.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.850+0000 7f933497d640 10 monclient: _finish_hunting 0 2024-09-11T01:36:10.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.850+0000 7f933497d640 1 monclient: found mon.noname-b 2024-09-11T01:36:10.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.850+0000 7f933497d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:10.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.850+0000 7f933497d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:36:10.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.850+0000 7f933497d640 1 -- 172.21.15.12:0/3139981826 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f933009f010 con 0x7f93300a2200 2024-09-11T01:36:10.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.852+0000 7f933497d640 1 -- 172.21.15.12:0/3139981826 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93300a2200 msgr2=0x7f93301353f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:10.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.852+0000 7f933497d640 1 --2- 172.21.15.12:0/3139981826 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93300a2200 0x7f93301353f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9324009870 tx=0x7f932402c2b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1d2abcd14d34a08 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:10.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.852+0000 7f9323fff640 1 -- 172.21.15.12:0/3139981826 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9324039070 con 0x7f93300a2200 2024-09-11T01:36:10.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.853+0000 7f9323fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:10.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.853+0000 7f9323fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:36:10.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.853+0000 7f9323fff640 10 monclient: dump: 2024-09-11T01:36:10.862 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:10.862 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:10.862 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:10.863 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:10.863 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:10.863 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:10.863 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:10.863 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:10.863 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:10.863 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:10.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.853+0000 7f9323fff640 10 monclient: _finish_auth 0 2024-09-11T01:36:10.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.853+0000 7f9323fff640 10 monclient: _check_auth_tickets 2024-09-11T01:36:10.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.853+0000 7f9323fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:10.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.853+0000 7f9323fff640 1 -- 172.21.15.12:0/3139981826 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9324034640 con 0x7f93300a2200 2024-09-11T01:36:10.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.853+0000 7f93373e2640 5 monclient: authenticate success, global_id 64208 2024-09-11T01:36:10.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.853+0000 7f93373e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:36:10.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.853+0000 7f9323fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.853+0000 7f93373e2640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.853+0000 7f9323fff640 1 -- 172.21.15.12:0/3139981826 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9324034a10 con 0x7f93300a2200 2024-09-11T01:36:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.853+0000 7f93373e2640 4 set_mon_vals no callback set 2024-09-11T01:36:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.853+0000 7f9323fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.853+0000 7f9323fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.853+0000 7f9323fff640 10 monclient: dump: 2024-09-11T01:36:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:10.864 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:10.865 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:10.865 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:10.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.853+0000 7f93373e2640 10 monclient: shutdown 2024-09-11T01:36:10.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.853+0000 7f93373e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:10.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.853+0000 7f93373e2640 1 -- 172.21.15.12:0/3139981826 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93300a2200 msgr2=0x7f93301353f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:10.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.853+0000 7f93373e2640 1 --2- 172.21.15.12:0/3139981826 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93300a2200 0x7f93301353f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9324009870 tx=0x7f932402c2b0 comp rx=0 tx=0).stop 2024-09-11T01:36:10.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.853+0000 7f93373e2640 1 -- 172.21.15.12:0/3139981826 shutdown_connections 2024-09-11T01:36:10.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.853+0000 7f93373e2640 1 --2- 172.21.15.12:0/3139981826 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93300a2200 0x7f93301353f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:10.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.853+0000 7f93373e2640 1 --2- 172.21.15.12:0/3139981826 >> v2:172.21.15.71:3300/0 conn(0x7f9330134ad0 0x7f9330134eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:10.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.853+0000 7f93373e2640 1 -- 172.21.15.12:0/3139981826 >> 172.21.15.12:0/3139981826 conn(0x7f93300a2f70 msgr2=0x7f93300a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:10.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.853+0000 7f93373e2640 1 -- 172.21.15.12:0/3139981826 shutdown_connections 2024-09-11T01:36:10.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.853+0000 7f93373e2640 1 -- 172.21.15.12:0/3139981826 wait complete. 2024-09-11T01:36:10.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.854+0000 7f93373e2640 10 monclient: build_initial_monmap 2024-09-11T01:36:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.854+0000 7f93373e2640 10 monclient: monmap: 2024-09-11T01:36:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:10.866 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.854+0000 7f93373e2640 1 Processor -- start 2024-09-11T01:36:10.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.854+0000 7f93373e2640 1 -- start start 2024-09-11T01:36:10.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.854+0000 7f93373e2640 10 monclient: init 2024-09-11T01:36:10.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f93373e2640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:10.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f93373e2640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:36:10.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f93373e2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93300a2200 0x7f93301c9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:10.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f93373e2640 10 monclient(hunting): picked mon.noname-a con 0x7f93300a2200 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:10.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f93373e2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9330133a50 0x7f93301c9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:10.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f93373e2640 10 monclient(hunting): picked mon.noname-b con 0x7f9330133a50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:10.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f93373e2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9330134ad0 0x7f933011f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:10.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f93373e2640 10 monclient(hunting): picked mon.noname-c con 0x7f9330134ad0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:10.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f93373e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:10.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f93373e2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f93300a0150 con 0x7f93300a2200 2024-09-11T01:36:10.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f93373e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:10.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f93373e2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f933009ffd0 con 0x7f9330133a50 2024-09-11T01:36:10.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f93373e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:10.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f93373e2640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f93301345f0 con 0x7f9330134ad0 2024-09-11T01:36:10.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f93373e2640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:10.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f93373e2640 10 monclient(hunting): authenticate will time out at 1177.297196s 2024-09-11T01:36:10.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933517e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93300a2200 0x7f93301c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:10.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933517e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93300a2200 0x7f93301c9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48958/0 (socket says 172.21.15.12:48958) 2024-09-11T01:36:10.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933517e640 1 -- 172.21.15.12:0/1177410372 learned_addr learned my addr 172.21.15.12:0/1177410372 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:10.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933517e640 10 monclient(hunting): get_auth_request con 0x7f93300a2200 auth_method 0 2024-09-11T01:36:10.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933517e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:10.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933517e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:10.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933517e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:10.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933597f640 1 --2- 172.21.15.12:0/1177410372 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9330134ad0 0x7f933011f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:10.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933597f640 10 monclient(hunting): get_auth_request con 0x7f9330134ad0 auth_method 0 2024-09-11T01:36:10.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933597f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933597f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933597f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933517e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933517e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933517e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933597f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933597f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933597f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933497d640 1 --2- 172.21.15.12:0/1177410372 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9330133a50 0x7f93301c9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933497d640 10 monclient(hunting): get_auth_request con 0x7f9330133a50 auth_method 0 2024-09-11T01:36:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933497d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933497d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933497d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933517e640 10 monclient(hunting): handle_auth_done global_id 54306 payload 824 2024-09-11T01:36:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933517e640 1 -- 172.21.15.12:0/1177410372 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9330134ad0 msgr2=0x7f933011f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933517e640 1 --2- 172.21.15.12:0/1177410372 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9330134ad0 0x7f933011f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933517e640 1 -- 172.21.15.12:0/1177410372 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9330133a50 msgr2=0x7f93301c9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933517e640 1 --2- 172.21.15.12:0/1177410372 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9330133a50 0x7f93301c9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933517e640 10 monclient: _finish_hunting 0 2024-09-11T01:36:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933517e640 1 monclient: found mon.noname-a 2024-09-11T01:36:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933517e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933517e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.855+0000 7f933517e640 1 -- 172.21.15.12:0/1177410372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93301336c0 con 0x7f93300a2200 2024-09-11T01:36:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.857+0000 7f933517e640 1 -- 172.21.15.12:0/1177410372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93300a2200 msgr2=0x7f93301c9120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.857+0000 7f933517e640 1 --2- 172.21.15.12:0/1177410372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93300a2200 0x7f93301c9120 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f932c00b4d0 tx=0x7f932c00b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.857+0000 7f9321ffb640 1 -- 172.21.15.12:0/1177410372 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f932c014070 con 0x7f93300a2200 2024-09-11T01:36:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.857+0000 7f9321ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.857+0000 7f9321ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.857+0000 7f9321ffb640 10 monclient: dump: 2024-09-11T01:36:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:10.871 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.857+0000 7f9321ffb640 10 monclient: _finish_auth 0 2024-09-11T01:36:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.857+0000 7f9321ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:36:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.857+0000 7f9321ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.858+0000 7f93373e2640 5 monclient: authenticate success, global_id 54306 2024-09-11T01:36:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.858+0000 7f9321ffb640 1 -- 172.21.15.12:0/1177410372 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f932c010a80 con 0x7f93300a2200 2024-09-11T01:36:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.858+0000 7f93373e2640 10 monclient: _renew_subs 2024-09-11T01:36:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.858+0000 7f93373e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.858+0000 7f93373e2640 1 -- 172.21.15.12:0/1177410372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93301230f0 con 0x7f93300a2200 2024-09-11T01:36:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.858+0000 7f9321ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.858+0000 7f9321ffb640 1 -- 172.21.15.12:0/1177410372 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f932c00f3c0 con 0x7f93300a2200 2024-09-11T01:36:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.858+0000 7f9321ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.858+0000 7f9321ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.858+0000 7f9321ffb640 10 monclient: dump: 2024-09-11T01:36:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:10.873 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:10.873 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:10.873 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:10.873 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:10.873 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:10.873 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:10.873 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:10.873 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:10.873 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:10.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.858+0000 7f93373e2640 10 monclient: _renew_subs 2024-09-11T01:36:10.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.858+0000 7f93237fe640 4 set_mon_vals no callback set 2024-09-11T01:36:10.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.858+0000 7f93373e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:10.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.858+0000 7f93373e2640 1 -- 172.21.15.12:0/1177410372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9330122e10 con 0x7f93300a2200 2024-09-11T01:36:10.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.858+0000 7f93373e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:10.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.858+0000 7f93373e2640 1 --2- 172.21.15.12:0/1177410372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f933011f810 0x7f933011fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:10.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.858+0000 7f93373e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:10.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.858+0000 7f93373e2640 1 -- 172.21.15.12:0/1177410372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f93300a17c0 con 0x7f933011f810 2024-09-11T01:36:10.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.859+0000 7f933497d640 1 --2- 172.21.15.12:0/1177410372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f933011f810 0x7f933011fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:10.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.859+0000 7f933497d640 10 monclient: get_auth_request con 0x7f933011f810 auth_method 0 2024-09-11T01:36:10.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.859+0000 7f933497d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:10.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.859+0000 7f933497d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:10.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.859+0000 7f933497d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:10.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.859+0000 7f9321ffb640 1 -- 172.21.15.12:0/1177410372 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f932c00f560 con 0x7f93300a2200 2024-09-11T01:36:10.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.859+0000 7f933497d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:10.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.859+0000 7f933497d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:10.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.859+0000 7f933497d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:10.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.859+0000 7f933497d640 10 monclient(hunting): handle_auth_done global_id 54309 payload 824 2024-09-11T01:36:10.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.859+0000 7f933497d640 1 -- 172.21.15.12:0/1177410372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9330133ee0 con 0x7f933011f810 2024-09-11T01:36:10.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.859+0000 7f9321ffb640 1 --2- 172.21.15.12:0/1177410372 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f931403e260 0x7f9314040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:10.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.859+0000 7f9321ffb640 1 -- 172.21.15.12:0/1177410372 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(225..225 src has 186..225) ==== 4771+0+0 (secure 0 0 0) 0x7f932c052de0 con 0x7f93300a2200 2024-09-11T01:36:10.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.859+0000 7f933597f640 1 --2- 172.21.15.12:0/1177410372 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f931403e260 0x7f9314040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:10.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.859+0000 7f933597f640 10 monclient: get_auth_request con 0x7f931403e260 auth_method 0 2024-09-11T01:36:10.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.861+0000 7f933497d640 1 -- 172.21.15.12:0/1177410372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f933011f810 msgr2=0x7f933011fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:10.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.861+0000 7f933497d640 1 --2- 172.21.15.12:0/1177410372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f933011f810 0x7f933011fbf0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9324036000 tx=0x7f93240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:10.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.862+0000 7f9321ffb640 1 -- 172.21.15.12:0/1177410372 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9324039040 con 0x7f933011f810 2024-09-11T01:36:10.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.862+0000 7f9321ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:10.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.862+0000 7f9321ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:10.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.862+0000 7f9321ffb640 10 monclient: dump: 2024-09-11T01:36:10.878 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:10.878 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:10.878 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:10.878 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:10.878 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:10.878 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:10.878 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:10.878 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:10.878 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:10.878 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:10.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.862+0000 7f933597f640 1 --2- 172.21.15.12:0/1177410372 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f931403e260 0x7f9314040720 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f93301235e0 tx=0x7f9328009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:10.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.862+0000 7f9321ffb640 1 -- 172.21.15.12:0/1177410372 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f932403e640 con 0x7f933011f810 2024-09-11T01:36:10.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.862+0000 7f9321ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:10.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.862+0000 7f9321ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:10.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.862+0000 7f9321ffb640 1 -- 172.21.15.12:0/1177410372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f933011f810 msgr2=0x7f933011fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:10.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.862+0000 7f9321ffb640 1 --2- 172.21.15.12:0/1177410372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f933011f810 0x7f933011fbf0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9324036000 tx=0x7f93240046c0 comp rx=0 tx=0).stop 2024-09-11T01:36:10.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.862+0000 7f9321ffb640 1 -- 172.21.15.12:0/1177410372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f933011f810 msgr2=0x7f933011fbf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:10.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.862+0000 7f9321ffb640 1 --2- 172.21.15.12:0/1177410372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f933011f810 0x7f933011fbf0 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9324036000 tx=0x7f93240046c0 comp rx=0 tx=0).stop 2024-09-11T01:36:10.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.868+0000 7f93373e2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:36:10.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.868+0000 7f93373e2640 1 --2- 172.21.15.12:0/1177410372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9330120130 0x7f93300726b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:10.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.868+0000 7f93373e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:10.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.868+0000 7f93373e2640 1 -- 172.21.15.12:0/1177410372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f93301218f0 con 0x7f9330120130 2024-09-11T01:36:10.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.868+0000 7f933497d640 1 --2- 172.21.15.12:0/1177410372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9330120130 0x7f93300726b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:10.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.868+0000 7f933497d640 10 monclient: get_auth_request con 0x7f9330120130 auth_method 0 2024-09-11T01:36:10.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.868+0000 7f933497d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:10.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.868+0000 7f933497d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:10.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.868+0000 7f933497d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:10.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.868+0000 7f933497d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:10.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.868+0000 7f933497d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:10.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.868+0000 7f933497d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:10.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.868+0000 7f933497d640 10 monclient(hunting): handle_auth_done global_id 54312 payload 824 2024-09-11T01:36:10.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.868+0000 7f933497d640 1 -- 172.21.15.12:0/1177410372 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f93301c9ba0 con 0x7f9330120130 2024-09-11T01:36:10.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.871+0000 7f933497d640 1 -- 172.21.15.12:0/1177410372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9330120130 msgr2=0x7f93300726b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:10.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.871+0000 7f933497d640 1 --2- 172.21.15.12:0/1177410372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9330120130 0x7f93300726b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9324000c00 tx=0x7f9324008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:10.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.871+0000 7f9321ffb640 1 -- 172.21.15.12:0/1177410372 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f932404f070 con 0x7f9330120130 2024-09-11T01:36:10.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.871+0000 7f9321ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:10.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.871+0000 7f9321ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:10.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.871+0000 7f9321ffb640 10 monclient: dump: 2024-09-11T01:36:10.880 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:10.880 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:10.880 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:10.881 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:10.881 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:10.881 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:10.881 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:10.881 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:10.881 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:10.881 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:10.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.871+0000 7f9321ffb640 1 -- 172.21.15.12:0/1177410372 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4710 (secure 0 0 0) 0x7f932404b290 con 0x7f9330120130 2024-09-11T01:36:10.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.871+0000 7f9321ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:36:10.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.871+0000 7f9321ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:10.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.871+0000 7f9321ffb640 1 -- 172.21.15.12:0/1177410372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9330120130 msgr2=0x7f93300726b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:10.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.871+0000 7f9321ffb640 1 --2- 172.21.15.12:0/1177410372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9330120130 0x7f93300726b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9324000c00 tx=0x7f9324008030 comp rx=0 tx=0).stop 2024-09-11T01:36:10.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.871+0000 7f9321ffb640 1 -- 172.21.15.12:0/1177410372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9330120130 msgr2=0x7f93300726b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:10.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.871+0000 7f9321ffb640 1 --2- 172.21.15.12:0/1177410372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9330120130 0x7f93300726b0 secure :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9324000c00 tx=0x7f9324008030 comp rx=0 tx=0).stop 2024-09-11T01:36:10.882 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:36:10.882 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:36:10.882 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:36:10.882 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:36:10.882 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 74, 2024-09-11T01:36:10.882 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 160852, 2024-09-11T01:36:10.882 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:36:10.882 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:36:10.882 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:36:10.882 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:36:10.883 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:10.883 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:36:10.883 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:10.883 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:36:10.883 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:36:10.883 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:10.883 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:10.883 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:10.883 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:10.883 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:10.883 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:10.883 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:10.883 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:10.883 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:10.884 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:10.884 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:10.884 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:10.884 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:36:10.884 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:36:10.884 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:10.884 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:10.884 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:10.884 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:10.884 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:10.884 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:10.884 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:10.884 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:10.885 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:10.885 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:10.885 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:10.885 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:10.885 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:10.885 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:36:10.885 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:36:10.885 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:36:10.885 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:36:10.885 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:36:10.885 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:36:10.885 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:10.885 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:10.886 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:36:10.886 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:36:10.886 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:36:10.886 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:36:10.886 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:36:10.886 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:36:10.886 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:36:10.886 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:10.886 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:36:10.886 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:10.886 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:10.886 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:10.886 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:10.887 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:10.887 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:10.887 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:10.887 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:10.887 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:10.887 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:10.887 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:10.887 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:10.887 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:36:10.887 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:10.887 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:36:10.887 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:10.887 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:36:10.887 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:36:10.888 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:10.888 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:10.888 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:10.888 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:10.888 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:36:10.888 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:10.888 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:10.888 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:10.888 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:10.888 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:36:10.888 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:10.888 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:10.888 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:10.889 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:10.889 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:36:10.889 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:36:10.889 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:10.889 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:10.889 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:10.889 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:10.889 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:10.889 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:36:10.889 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:36:10.889 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:10.889 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:10.889 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:10.889 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:10.890 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:36:10.890 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:10.890 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:10.890 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:10.890 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:10.890 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:36:10.890 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:10.890 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:10.890 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:10.890 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:10.890 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:36:10.890 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:36:10.890 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:10.890 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:10.890 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:10.891 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:10.891 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:10.891 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:36:10.891 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:36:10.891 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:10.891 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:10.891 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:10.891 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:10.891 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:36:10.891 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:10.891 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:10.891 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:10.891 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:10.891 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:36:10.891 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:10.892 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:10.892 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:10.892 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:10.892 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:36:10.892 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:36:10.892 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:10.892 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:10.892 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:10.892 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:10.892 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:10.892 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:10.892 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:36:10.892 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:36:10.893 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:10.893 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:10.893 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:10.893 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:36:10.893 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:10.893 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:10.893 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:36:10.893 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:10.893 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:10.893 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:10.893 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:36:10.893 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:10.893 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:10.894 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:36:10.894 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:10.894 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:10.894 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:10.894 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:36:10.894 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:10.894 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:10.894 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:10.894 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:36:10.894 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:36:10.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.871+0000 7f93373e2640 1 -- 172.21.15.12:0/1177410372 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f931403e260 msgr2=0x7f9314040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:10.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.871+0000 7f93373e2640 1 --2- 172.21.15.12:0/1177410372 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f931403e260 0x7f9314040720 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f93301235e0 tx=0x7f9328009040 comp rx=0 tx=0).stop 2024-09-11T01:36:10.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.871+0000 7f93373e2640 10 monclient: shutdown 2024-09-11T01:36:10.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.871+0000 7f93373e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:10.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.871+0000 7f93373e2640 1 -- 172.21.15.12:0/1177410372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93300a2200 msgr2=0x7f93301c9120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:10.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.871+0000 7f93373e2640 1 --2- 172.21.15.12:0/1177410372 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93300a2200 0x7f93301c9120 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f932c00b4d0 tx=0x7f932c00b990 comp rx=0 tx=0).stop 2024-09-11T01:36:10.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.871+0000 7f933597f640 1 -- 172.21.15.12:0/1177410372 reap_dead start 2024-09-11T01:36:10.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.871+0000 7f933597f640 1 -- 172.21.15.12:0/1177410372 reap_dead start 2024-09-11T01:36:10.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.871+0000 7f93373e2640 1 -- 172.21.15.12:0/1177410372 shutdown_connections 2024-09-11T01:36:10.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.871+0000 7f93373e2640 1 -- 172.21.15.12:0/1177410372 >> 172.21.15.12:0/1177410372 conn(0x7f93300a2f70 msgr2=0x7f933009d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:10.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.871+0000 7f93373e2640 1 -- 172.21.15.12:0/1177410372 shutdown_connections 2024-09-11T01:36:10.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.871+0000 7f93373e2640 1 -- 172.21.15.12:0/1177410372 wait complete. 2024-09-11T01:36:10.897 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:36:10.897 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:36:10.897 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:36:10.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.997+0000 7f410afbc640 10 monclient: get_monmap_and_config 2024-09-11T01:36:10.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.997+0000 7f410afbc640 10 monclient: build_initial_monmap 2024-09-11T01:36:10.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.997+0000 7f410afbc640 10 monclient: monmap: 2024-09-11T01:36:10.999 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:10.999 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:10.999 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:10.998378+0000 2024-09-11T01:36:10.999 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:10.998378+0000 2024-09-11T01:36:10.999 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:10.999 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:10.999 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:10.999 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:11.000 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:11.000 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:11.000 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:11.000 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.998+0000 7f410afbc640 1 Processor -- start 2024-09-11T01:36:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.998+0000 7f410afbc640 1 -- start start 2024-09-11T01:36:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.998+0000 7f410afbc640 10 monclient: init 2024-09-11T01:36:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.998+0000 7f410afbc640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.998+0000 7f410afbc640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-11T01:36:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.998+0000 7f410afbc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f41040a2e20 addr v1:172.21.15.12:6789/0 2024-09-11T01:36:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.998+0000 7f410afbc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f41040a4970 0x7f410406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.998+0000 7f410afbc640 10 monclient(hunting): picked mon.noname-a con 0x7f41040a4970 addr v2:172.21.15.12:3300/0 2024-09-11T01:36:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.998+0000 7f410afbc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f410406f910 addr v1:172.21.15.71:6789/0 2024-09-11T01:36:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.998+0000 7f410afbc640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f410406f4f0 con 0x7f41040a2e20 2024-09-11T01:36:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.998+0000 7f410afbc640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41040a3910 con 0x7f410406f910 2024-09-11T01:36:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.998+0000 7f410afbc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.998+0000 7f410afbc640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f410406ef30 con 0x7f41040a4970 2024-09-11T01:36:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.998+0000 7f410afbc640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.998+0000 7f410afbc640 10 monclient(hunting): authenticate will time out at 1177.440932s 2024-09-11T01:36:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.999+0000 7f4103fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f41040a4970 0x7f410406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.999+0000 7f4103fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f41040a4970 0x7f410406ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48990/0 (socket says 172.21.15.12:48990) 2024-09-11T01:36:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.999+0000 7f4103fff640 1 -- 172.21.15.12:0/2364131521 learned_addr learned my addr 172.21.15.12:0/2364131521 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:11.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.999+0000 7f4103fff640 10 monclient(hunting): get_auth_request con 0x7f41040a4970 auth_method 0 2024-09-11T01:36:11.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.999+0000 7f4103fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:11.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.999+0000 7f4103fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:11.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.999+0000 7f4103fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:11.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.999+0000 7f4103fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:11.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.999+0000 7f4103fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:11.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:10.999+0000 7f4103fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:11.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.001+0000 7f4108d58640 1 -- 172.21.15.12:0/2364131521 >> v1:172.21.15.12:6789/0 conn(0x7f41040a2e20 legacy=0x7f41040a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:11.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.001+0000 7f4103fff640 10 monclient(hunting): handle_auth_done global_id 54315 payload 274 2024-09-11T01:36:11.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.001+0000 7f4103fff640 1 -- 172.21.15.12:0/2364131521 >> v1:172.21.15.71:6789/0 conn(0x7f410406f910 legacy=0x7f410413f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:11.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.001+0000 7f4103fff640 1 -- 172.21.15.12:0/2364131521 >> v1:172.21.15.12:6789/0 conn(0x7f41040a2e20 legacy=0x7f41040a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:11.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.001+0000 7f4103fff640 10 monclient: _finish_hunting 0 2024-09-11T01:36:11.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.001+0000 7f4103fff640 1 monclient: found mon.noname-a 2024-09-11T01:36:11.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.001+0000 7f4103fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:11.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.001+0000 7f4103fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:11.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.001+0000 7f4103fff640 1 -- 172.21.15.12:0/2364131521 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41040a3ae0 con 0x7f41040a4970 2024-09-11T01:36:11.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f4103fff640 1 -- 172.21.15.12:0/2364131521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41040a4970 msgr2=0x7f410406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:11.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f4103fff640 1 --2- 172.21.15.12:0/2364131521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41040a4970 0x7f410406ded0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f40f00098f0 tx=0x7f40f002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ef9b9b53e39fd94 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:11.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f41037fe640 1 -- 172.21.15.12:0/2364131521 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f0037070 con 0x7f41040a4970 2024-09-11T01:36:11.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f41037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:11.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f41037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:11.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f41037fe640 10 monclient: dump: 2024-09-11T01:36:11.007 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:11.007 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:11.007 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.007 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.007 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:11.007 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:11.007 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:11.007 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:11.007 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:11.007 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:11.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f41037fe640 10 monclient: _finish_auth 0 2024-09-11T01:36:11.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f41037fe640 10 monclient: _check_auth_tickets 2024-09-11T01:36:11.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f41037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:11.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f410afbc640 5 monclient: authenticate success, global_id 54315 2024-09-11T01:36:11.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f41037fe640 1 -- 172.21.15.12:0/2364131521 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40f0032640 con 0x7f41040a4970 2024-09-11T01:36:11.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f410afbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:36:11.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f41037fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:11.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f410afbc640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:11.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f410afbc640 4 set_mon_vals no callback set 2024-09-11T01:36:11.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f41037fe640 1 -- 172.21.15.12:0/2364131521 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f0032a10 con 0x7f41040a4970 2024-09-11T01:36:11.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f41037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:11.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f41037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:11.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f41037fe640 10 monclient: dump: 2024-09-11T01:36:11.009 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:11.009 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:11.009 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.009 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.009 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:11.009 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:11.009 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:11.009 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:11.009 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:11.009 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:11.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f410afbc640 10 monclient: shutdown 2024-09-11T01:36:11.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f410afbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:11.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f410afbc640 1 -- 172.21.15.12:0/2364131521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41040a4970 msgr2=0x7f410406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:11.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f410afbc640 1 --2- 172.21.15.12:0/2364131521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41040a4970 0x7f410406ded0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f40f00098f0 tx=0x7f40f002a1c0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f410afbc640 1 -- 172.21.15.12:0/2364131521 shutdown_connections 2024-09-11T01:36:11.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f410afbc640 1 --2- 172.21.15.12:0/2364131521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41040a4970 0x7f410406ded0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f410afbc640 1 -- 172.21.15.12:0/2364131521 >> 172.21.15.12:0/2364131521 conn(0x7f41040a8600 msgr2=0x7f41040a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:11.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f410afbc640 1 -- 172.21.15.12:0/2364131521 shutdown_connections 2024-09-11T01:36:11.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.004+0000 7f410afbc640 1 -- 172.21.15.12:0/2364131521 wait complete. 2024-09-11T01:36:11.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.005+0000 7f410afbc640 10 monclient: build_initial_monmap 2024-09-11T01:36:11.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.005+0000 7f410afbc640 10 monclient: monmap: 2024-09-11T01:36:11.010 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:11.010 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:11.010 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:11.010 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:11.010 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:11.011 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:11.011 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:11.011 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:11.011 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:11.011 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:11.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.005+0000 7f410afbc640 1 Processor -- start 2024-09-11T01:36:11.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.005+0000 7f410afbc640 1 -- start start 2024-09-11T01:36:11.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.005+0000 7f410afbc640 10 monclient: init 2024-09-11T01:36:11.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f410afbc640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:11.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f410afbc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:36:11.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f410afbc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f410406f910 0x7f4104134200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:11.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f410afbc640 10 monclient(hunting): picked mon.noname-c con 0x7f410406f910 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:11.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f410afbc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41040a2e20 0x7f4104134740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:11.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f410afbc640 10 monclient(hunting): picked mon.noname-b con 0x7f41040a2e20 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:11.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f410afbc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41040a4970 0x7f4104137020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:11.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f410afbc640 10 monclient(hunting): picked mon.noname-a con 0x7f41040a4970 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:11.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f410afbc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:11.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f410afbc640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4104140d60 con 0x7f41040a4970 2024-09-11T01:36:11.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f410afbc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:11.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f410afbc640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f410406ec50 con 0x7f41040a2e20 2024-09-11T01:36:11.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f410afbc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:11.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f410afbc640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4104140ee0 con 0x7f410406f910 2024-09-11T01:36:11.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f410afbc640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:11.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f410afbc640 10 monclient(hunting): authenticate will time out at 1177.448396s 2024-09-11T01:36:11.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4108d58640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f410406f910 0x7f4104134200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4108d58640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f410406f910 0x7f4104134200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:49570/0 (socket says 172.21.15.12:49570) 2024-09-11T01:36:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4108d58640 1 -- 172.21.15.12:0/2368689564 learned_addr learned my addr 172.21.15.12:0/2368689564 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4108d58640 10 monclient(hunting): get_auth_request con 0x7f410406f910 auth_method 0 2024-09-11T01:36:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4108d58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4108d58640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4108d58640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4109559640 1 --2- 172.21.15.12:0/2368689564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41040a4970 0x7f4104137020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4109559640 10 monclient(hunting): get_auth_request con 0x7f41040a4970 auth_method 0 2024-09-11T01:36:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4109559640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4109559640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4109559640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4103fff640 1 --2- 172.21.15.12:0/2368689564 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41040a2e20 0x7f4104134740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:11.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4103fff640 10 monclient(hunting): get_auth_request con 0x7f41040a2e20 auth_method 0 2024-09-11T01:36:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4103fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4103fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4103fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4108d58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4108d58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4108d58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4109559640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4109559640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4109559640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4103fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4103fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4103fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4108d58640 10 monclient(hunting): handle_auth_done global_id 64214 payload 824 2024-09-11T01:36:11.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4108d58640 1 -- 172.21.15.12:0/2368689564 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41040a2e20 msgr2=0x7f4104134740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:11.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4108d58640 1 --2- 172.21.15.12:0/2368689564 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41040a2e20 0x7f4104134740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4108d58640 1 -- 172.21.15.12:0/2368689564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41040a4970 msgr2=0x7f4104137020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:11.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4108d58640 1 --2- 172.21.15.12:0/2368689564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41040a4970 0x7f4104137020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4108d58640 10 monclient: _finish_hunting 0 2024-09-11T01:36:11.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4108d58640 1 monclient: found mon.noname-c 2024-09-11T01:36:11.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4108d58640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:11.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4108d58640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:36:11.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4108d58640 1 -- 172.21.15.12:0/2368689564 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f410413b080 con 0x7f410406f910 2024-09-11T01:36:11.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.006+0000 7f4109559640 1 --2- 172.21.15.12:0/2368689564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41040a4970 0x7f4104137020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:36:11.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.009+0000 7f4108d58640 1 -- 172.21.15.12:0/2368689564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f410406f910 msgr2=0x7f4104134200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:11.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.009+0000 7f4108d58640 1 --2- 172.21.15.12:0/2368689564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f410406f910 0x7f4104134200 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f40f800c450 tx=0x7f40f800c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:11.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.009+0000 7f41017fa640 1 -- 172.21.15.12:0/2368689564 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f8017070 con 0x7f410406f910 2024-09-11T01:36:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.009+0000 7f41017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.009+0000 7f41017fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:36:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.009+0000 7f41017fa640 10 monclient: dump: 2024-09-11T01:36:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:11.021 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:11.022 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:11.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.009+0000 7f41017fa640 10 monclient: _finish_auth 0 2024-09-11T01:36:11.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.009+0000 7f41017fa640 10 monclient: _check_auth_tickets 2024-09-11T01:36:11.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.009+0000 7f41017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:11.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.009+0000 7f41017fa640 1 -- 172.21.15.12:0/2368689564 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40f80057c0 con 0x7f410406f910 2024-09-11T01:36:11.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.009+0000 7f41017fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:11.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.009+0000 7f41017fa640 1 -- 172.21.15.12:0/2368689564 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f8005bb0 con 0x7f410406f910 2024-09-11T01:36:11.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.009+0000 7f41017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:11.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.009+0000 7f41017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:11.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.009+0000 7f41017fa640 10 monclient: dump: 2024-09-11T01:36:11.022 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:11.022 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:11.022 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.022 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.023 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:11.023 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:11.023 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:11.023 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:11.023 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:11.023 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:11.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.009+0000 7f410afbc640 5 monclient: authenticate success, global_id 64214 2024-09-11T01:36:11.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.009+0000 7f410afbc640 10 monclient: _renew_subs 2024-09-11T01:36:11.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.009+0000 7f410afbc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:36:11.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.009+0000 7f410afbc640 1 -- 172.21.15.12:0/2368689564 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f410409f610 con 0x7f410406f910 2024-09-11T01:36:11.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.009+0000 7f4102ffd640 4 set_mon_vals no callback set 2024-09-11T01:36:11.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.009+0000 7f410afbc640 10 monclient: _renew_subs 2024-09-11T01:36:11.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.009+0000 7f410afbc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:36:11.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.009+0000 7f410afbc640 1 -- 172.21.15.12:0/2368689564 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f410413ffe0 con 0x7f410406f910 2024-09-11T01:36:11.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.009+0000 7f410afbc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:11.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.010+0000 7f410afbc640 1 --2- 172.21.15.12:0/2368689564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4104137d20 0x7f410402d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:11.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.010+0000 7f410afbc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:11.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.010+0000 7f410afbc640 1 -- 172.21.15.12:0/2368689564 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f410406a060 con 0x7f4104137d20 2024-09-11T01:36:11.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.010+0000 7f4103fff640 1 --2- 172.21.15.12:0/2368689564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4104137d20 0x7f410402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:11.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.010+0000 7f4103fff640 10 monclient: get_auth_request con 0x7f4104137d20 auth_method 0 2024-09-11T01:36:11.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.010+0000 7f4103fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:11.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.010+0000 7f4103fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:11.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.010+0000 7f4103fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:11.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.010+0000 7f4103fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:11.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.010+0000 7f4103fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:11.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.010+0000 7f4103fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:11.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.010+0000 7f41017fa640 1 -- 172.21.15.12:0/2368689564 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f40f801b3b0 con 0x7f410406f910 2024-09-11T01:36:11.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.010+0000 7f4103fff640 10 monclient(hunting): handle_auth_done global_id 64217 payload 824 2024-09-11T01:36:11.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.010+0000 7f4103fff640 1 -- 172.21.15.12:0/2368689564 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f410406ded0 con 0x7f4104137d20 2024-09-11T01:36:11.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.010+0000 7f41017fa640 1 --2- 172.21.15.12:0/2368689564 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f40e803e210 0x7f40e80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:11.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.010+0000 7f41017fa640 1 -- 172.21.15.12:0/2368689564 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(225..225 src has 186..225) ==== 4771+0+0 (secure 0 0 0) 0x7f40f80522f0 con 0x7f410406f910 2024-09-11T01:36:11.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.010+0000 7f4109559640 1 --2- 172.21.15.12:0/2368689564 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f40e803e210 0x7f40e80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:11.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.010+0000 7f4109559640 10 monclient: get_auth_request con 0x7f40e803e210 auth_method 0 2024-09-11T01:36:11.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.012+0000 7f4103fff640 1 -- 172.21.15.12:0/2368689564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4104137d20 msgr2=0x7f410402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:11.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.012+0000 7f4103fff640 1 --2- 172.21.15.12:0/2368689564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4104137d20 0x7f410402d440 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f40f0034000 tx=0x7f40f00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:11.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.012+0000 7f41017fa640 1 -- 172.21.15.12:0/2368689564 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f0037040 con 0x7f4104137d20 2024-09-11T01:36:11.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.012+0000 7f41017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:11.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.012+0000 7f41017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:11.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.012+0000 7f41017fa640 10 monclient: dump: 2024-09-11T01:36:11.026 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:11.026 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:11.026 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.026 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.026 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:11.026 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:11.026 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:11.026 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:11.026 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:11.026 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:11.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.013+0000 7f41017fa640 1 -- 172.21.15.12:0/2368689564 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f40f003c820 con 0x7f4104137d20 2024-09-11T01:36:11.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.013+0000 7f41017fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:11.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.013+0000 7f41017fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:11.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.013+0000 7f41017fa640 1 -- 172.21.15.12:0/2368689564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4104137d20 msgr2=0x7f410402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:11.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.013+0000 7f41017fa640 1 --2- 172.21.15.12:0/2368689564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4104137d20 0x7f410402d440 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f40f0034000 tx=0x7f40f00046c0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.013+0000 7f41017fa640 1 -- 172.21.15.12:0/2368689564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4104137d20 msgr2=0x7f410402d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:11.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.013+0000 7f41017fa640 1 --2- 172.21.15.12:0/2368689564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4104137d20 0x7f410402d440 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f40f0034000 tx=0x7f40f00046c0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.013+0000 7f4109559640 1 --2- 172.21.15.12:0/2368689564 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f40e803e210 0x7f40e80406d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f40fc00f1d0 tx=0x7f40fc003d10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:11.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.019+0000 7f410afbc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:36:11.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.019+0000 7f410afbc640 1 --2- 172.21.15.12:0/2368689564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4104138100 0x7f410402d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:11.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.019+0000 7f410afbc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:11.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.019+0000 7f410afbc640 1 -- 172.21.15.12:0/2368689564 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f41040a0ef0 con 0x7f4104138100 2024-09-11T01:36:11.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.019+0000 7f4103fff640 1 --2- 172.21.15.12:0/2368689564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4104138100 0x7f410402d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:11.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.019+0000 7f4103fff640 10 monclient: get_auth_request con 0x7f4104138100 auth_method 0 2024-09-11T01:36:11.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.019+0000 7f4103fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:11.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.019+0000 7f4103fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:11.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.019+0000 7f4103fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:11.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.019+0000 7f4103fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:11.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.019+0000 7f4103fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:11.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.019+0000 7f4103fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:11.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.019+0000 7f4103fff640 10 monclient(hunting): handle_auth_done global_id 64220 payload 824 2024-09-11T01:36:11.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.019+0000 7f4103fff640 1 -- 172.21.15.12:0/2368689564 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4104069950 con 0x7f4104138100 2024-09-11T01:36:11.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.021+0000 7f4103fff640 1 -- 172.21.15.12:0/2368689564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4104138100 msgr2=0x7f410402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:11.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.021+0000 7f4103fff640 1 --2- 172.21.15.12:0/2368689564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4104138100 0x7f410402d980 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f40f002a200 tx=0x7f40f0009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:11.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.021+0000 7f41017fa640 1 -- 172.21.15.12:0/2368689564 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f0032e60 con 0x7f4104138100 2024-09-11T01:36:11.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.021+0000 7f41017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:11.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.021+0000 7f41017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:11.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.021+0000 7f41017fa640 10 monclient: dump: 2024-09-11T01:36:11.030 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:11.030 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:11.030 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.030 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.030 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:11.030 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:11.030 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:11.031 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:11.031 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:11.031 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:11.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.022+0000 7f41017fa640 1 -- 172.21.15.12:0/2368689564 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4707 (secure 0 0 0) 0x7f40f0046b70 con 0x7f4104138100 2024-09-11T01:36:11.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.022+0000 7f41017fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:36:11.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.022+0000 7f41017fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:11.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.022+0000 7f41017fa640 1 -- 172.21.15.12:0/2368689564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4104138100 msgr2=0x7f410402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:11.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.022+0000 7f41017fa640 1 --2- 172.21.15.12:0/2368689564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4104138100 0x7f410402d980 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f40f002a200 tx=0x7f40f0009240 comp rx=0 tx=0).stop 2024-09-11T01:36:11.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.022+0000 7f41017fa640 1 -- 172.21.15.12:0/2368689564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4104138100 msgr2=0x7f410402d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:11.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.022+0000 7f41017fa640 1 --2- 172.21.15.12:0/2368689564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4104138100 0x7f410402d980 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f40f002a200 tx=0x7f40f0009240 comp rx=0 tx=0).stop 2024-09-11T01:36:11.032 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:36:11.032 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:36:11.032 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:36:11.032 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:36:11.032 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 74, 2024-09-11T01:36:11.032 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 89115, 2024-09-11T01:36:11.032 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:36:11.032 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:36:11.032 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:36:11.032 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:36:11.032 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:11.032 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:36:11.033 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:11.033 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:36:11.033 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:36:11.033 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:11.033 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:11.033 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:11.033 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:11.033 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:11.033 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:11.033 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:11.033 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:11.033 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:11.033 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:11.033 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:11.034 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:11.034 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:36:11.034 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:36:11.034 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:11.034 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:11.034 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:11.034 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:11.034 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:11.034 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:11.034 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:11.034 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:11.034 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:11.034 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:11.034 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:11.035 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:11.035 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:11.035 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:36:11.035 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:36:11.035 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:36:11.035 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:36:11.035 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:36:11.035 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:36:11.035 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:11.035 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:11.035 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:36:11.035 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:36:11.035 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:36:11.035 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:36:11.035 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:36:11.036 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:36:11.036 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:36:11.036 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:11.036 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:36:11.036 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:11.036 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:11.036 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:11.036 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:11.036 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:11.036 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:11.036 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:11.036 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:11.037 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:11.037 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:11.037 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:11.037 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:11.037 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:36:11.037 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:11.037 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:36:11.037 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:11.037 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:36:11.037 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:36:11.037 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:11.037 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:11.037 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:11.037 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:11.038 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:36:11.038 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:11.038 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:11.038 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:11.038 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:11.038 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:36:11.038 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:11.038 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:11.038 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:11.038 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:11.038 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:36:11.038 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:36:11.038 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:11.038 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:11.038 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:11.039 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:11.039 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:11.039 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:36:11.039 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:36:11.039 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:11.039 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:11.039 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:11.039 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:11.039 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:36:11.039 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:11.039 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:11.039 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:11.039 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:11.039 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:36:11.039 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:11.040 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:11.040 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:11.040 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:11.040 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:36:11.040 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:36:11.040 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:11.040 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:11.040 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:11.040 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:11.040 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:11.040 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:36:11.040 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:36:11.040 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:11.041 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:11.041 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:11.041 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:11.041 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:36:11.041 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:11.041 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:11.041 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:11.041 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:11.041 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:36:11.041 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:11.041 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:11.041 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:11.041 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:11.042 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:36:11.042 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:36:11.042 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:11.042 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:11.042 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:11.042 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:11.042 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:11.042 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:11.042 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:36:11.042 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:36:11.042 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:11.042 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:11.042 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:11.042 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:36:11.042 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:11.043 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:11.043 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:36:11.043 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:11.043 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:11.043 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:11.043 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:36:11.043 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:11.043 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:11.043 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:36:11.043 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:11.043 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:11.043 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:11.043 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:36:11.043 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:11.043 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:11.043 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:11.044 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:36:11.044 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:36:11.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.022+0000 7f410afbc640 1 -- 172.21.15.12:0/2368689564 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f40e803e210 msgr2=0x7f40e80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:11.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.022+0000 7f410afbc640 1 --2- 172.21.15.12:0/2368689564 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f40e803e210 0x7f40e80406d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f40fc00f1d0 tx=0x7f40fc003d10 comp rx=0 tx=0).stop 2024-09-11T01:36:11.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.022+0000 7f410afbc640 10 monclient: shutdown 2024-09-11T01:36:11.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.022+0000 7f410afbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:11.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.022+0000 7f410afbc640 1 -- 172.21.15.12:0/2368689564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f410406f910 msgr2=0x7f4104134200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:11.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.022+0000 7f410afbc640 1 --2- 172.21.15.12:0/2368689564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f410406f910 0x7f4104134200 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f40f800c450 tx=0x7f40f800c910 comp rx=0 tx=0).stop 2024-09-11T01:36:11.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.022+0000 7f4109559640 1 -- 172.21.15.12:0/2368689564 reap_dead start 2024-09-11T01:36:11.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.022+0000 7f410afbc640 1 -- 172.21.15.12:0/2368689564 shutdown_connections 2024-09-11T01:36:11.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.022+0000 7f410afbc640 1 --2- 172.21.15.12:0/2368689564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f410406f910 0x7f4104134200 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.022+0000 7f410afbc640 1 -- 172.21.15.12:0/2368689564 >> 172.21.15.12:0/2368689564 conn(0x7f41040a8600 msgr2=0x7f410409e820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:11.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.022+0000 7f410afbc640 1 -- 172.21.15.12:0/2368689564 shutdown_connections 2024-09-11T01:36:11.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.022+0000 7f410afbc640 1 -- 172.21.15.12:0/2368689564 wait complete. 2024-09-11T01:36:11.046 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:36:11.046 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:36:11.046 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:36:11.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.146+0000 7f3954975640 10 monclient: get_monmap_and_config 2024-09-11T01:36:11.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.146+0000 7f3954975640 10 monclient: build_initial_monmap 2024-09-11T01:36:11.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.146+0000 7f3954975640 10 monclient: monmap: 2024-09-11T01:36:11.148 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:11.148 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:11.148 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:11.147072+0000 2024-09-11T01:36:11.148 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:11.147072+0000 2024-09-11T01:36:11.148 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:11.148 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:11.148 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:11.148 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:11.148 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:11.148 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:11.149 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:11.149 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:11.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f3954975640 1 Processor -- start 2024-09-11T01:36:11.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f3954975640 1 -- start start 2024-09-11T01:36:11.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f3954975640 10 monclient: init 2024-09-11T01:36:11.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f3954975640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:11.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f3954975640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-11T01:36:11.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f3954975640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3950134ad0 0x7f3950134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:11.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f3954975640 10 monclient(hunting): picked mon.noname-a con 0x7f3950134ad0 addr v2:172.21.15.12:3300/0 2024-09-11T01:36:11.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f3954975640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f39500a2200 0x7f39501353f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:11.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f3954975640 10 monclient(hunting): picked mon.noname-c con 0x7f39500a2200 addr v2:172.21.15.71:3300/0 2024-09-11T01:36:11.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f3954975640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3950133a50 0x7f3950133f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:11.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f3954975640 10 monclient(hunting): picked mon.noname-b con 0x7f3950133a50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:11.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f3954975640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:11.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f3954975640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f395009f5d0 con 0x7f3950134ad0 2024-09-11T01:36:11.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f3954975640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:11.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f3954975640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f3950135930 con 0x7f39500a2200 2024-09-11T01:36:11.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f3954975640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:11.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f3954975640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f395009ed30 con 0x7f3950133a50 2024-09-11T01:36:11.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f3954975640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:11.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f3954975640 10 monclient(hunting): authenticate will time out at 1177.589681s 2024-09-11T01:36:11.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f394e59c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3950134ad0 0x7f3950134eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:11.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f394e59c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3950134ad0 0x7f3950134eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49002/0 (socket says 172.21.15.12:49002) 2024-09-11T01:36:11.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f394e59c640 1 -- 172.21.15.12:0/2533861824 learned_addr learned my addr 172.21.15.12:0/2533861824 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:11.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f394e59c640 10 monclient(hunting): get_auth_request con 0x7f3950134ad0 auth_method 0 2024-09-11T01:36:11.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f394ed9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3950133a50 0x7f3950133f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:11.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f394e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:11.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f394e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:11.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f394e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:11.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f394ed9d640 10 monclient(hunting): get_auth_request con 0x7f3950133a50 auth_method 0 2024-09-11T01:36:11.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f394ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:11.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f394ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:11.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f394ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:11.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f394e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:11.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f394e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:11.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f394e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:11.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f3947fff640 1 --2- 172.21.15.12:0/2533861824 >> v2:172.21.15.71:3300/0 conn(0x7f39500a2200 0x7f39501353f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:11.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f394ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:11.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f394ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:11.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f394ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:11.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f3947fff640 10 monclient(hunting): get_auth_request con 0x7f39500a2200 auth_method 0 2024-09-11T01:36:11.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f3947fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:11.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f3947fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:11.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.147+0000 7f3947fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:11.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.148+0000 7f3947fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:11.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.148+0000 7f3947fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:11.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.148+0000 7f3947fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:11.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.148+0000 7f394ed9d640 10 monclient(hunting): handle_auth_done global_id 64223 payload 274 2024-09-11T01:36:11.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.148+0000 7f394ed9d640 1 -- 172.21.15.12:0/2533861824 >> v2:172.21.15.71:3300/0 conn(0x7f39500a2200 msgr2=0x7f39501353f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:11.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.148+0000 7f394ed9d640 1 --2- 172.21.15.12:0/2533861824 >> v2:172.21.15.71:3300/0 conn(0x7f39500a2200 0x7f39501353f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.148+0000 7f394ed9d640 1 -- 172.21.15.12:0/2533861824 >> v2:172.21.15.12:3300/0 conn(0x7f3950134ad0 msgr2=0x7f3950134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:11.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.148+0000 7f394ed9d640 1 --2- 172.21.15.12:0/2533861824 >> v2:172.21.15.12:3300/0 conn(0x7f3950134ad0 0x7f3950134eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.148+0000 7f394ed9d640 10 monclient: _finish_hunting 0 2024-09-11T01:36:11.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.148+0000 7f394ed9d640 1 monclient: found mon.noname-b 2024-09-11T01:36:11.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.148+0000 7f394ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:11.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.148+0000 7f394ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:36:11.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.148+0000 7f394ed9d640 1 -- 172.21.15.12:0/2533861824 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f395009f010 con 0x7f3950133a50 2024-09-11T01:36:11.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.148+0000 7f394e59c640 1 --2- 172.21.15.12:0/2533861824 >> v2:172.21.15.12:3300/0 conn(0x7f3950134ad0 0x7f3950134eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:36:11.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.150+0000 7f394ed9d640 1 -- 172.21.15.12:0/2533861824 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3950133a50 msgr2=0x7f3950133f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:11.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.150+0000 7f394ed9d640 1 --2- 172.21.15.12:0/2533861824 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3950133a50 0x7f3950133f00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3948009d90 tx=0x7f394802a4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=beba2affe5d43ed4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:11.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f394dd9b640 1 -- 172.21.15.12:0/2533861824 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3948007950 con 0x7f3950133a50 2024-09-11T01:36:11.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f394dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:11.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f394dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:36:11.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f394dd9b640 10 monclient: dump: 2024-09-11T01:36:11.162 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:11.162 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:11.162 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.163 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.163 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:11.163 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:11.163 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:11.163 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:11.163 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:11.163 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:11.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f394dd9b640 10 monclient: _finish_auth 0 2024-09-11T01:36:11.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f394dd9b640 10 monclient: _check_auth_tickets 2024-09-11T01:36:11.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f394dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:11.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f394dd9b640 1 -- 172.21.15.12:0/2533861824 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3948007af0 con 0x7f3950133a50 2024-09-11T01:36:11.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f3954975640 5 monclient: authenticate success, global_id 64223 2024-09-11T01:36:11.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f394dd9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:11.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f394dd9b640 1 -- 172.21.15.12:0/2533861824 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3948031400 con 0x7f3950133a50 2024-09-11T01:36:11.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f394dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:11.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f394dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:11.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f394dd9b640 10 monclient: dump: 2024-09-11T01:36:11.164 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:11.164 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:11.164 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.164 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.164 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:11.164 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:11.164 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:11.164 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:11.164 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:11.164 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:11.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f3954975640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:11.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f3954975640 4 set_mon_vals no callback set 2024-09-11T01:36:11.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f3954975640 10 monclient: shutdown 2024-09-11T01:36:11.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f3954975640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:11.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f3954975640 1 -- 172.21.15.12:0/2533861824 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3950133a50 msgr2=0x7f3950133f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:11.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f3954975640 1 --2- 172.21.15.12:0/2533861824 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3950133a50 0x7f3950133f00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3948009d90 tx=0x7f394802a4e0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f3954975640 1 -- 172.21.15.12:0/2533861824 shutdown_connections 2024-09-11T01:36:11.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f3954975640 1 --2- 172.21.15.12:0/2533861824 >> v2:172.21.15.71:3300/0 conn(0x7f39500a2200 0x7f39501353f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f3954975640 1 --2- 172.21.15.12:0/2533861824 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3950133a50 0x7f3950133f00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f3954975640 1 --2- 172.21.15.12:0/2533861824 >> v2:172.21.15.12:3300/0 conn(0x7f3950134ad0 0x7f3950134eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f3954975640 1 -- 172.21.15.12:0/2533861824 >> 172.21.15.12:0/2533861824 conn(0x7f39500a2f70 msgr2=0x7f39500a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:11.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f3954975640 1 -- 172.21.15.12:0/2533861824 shutdown_connections 2024-09-11T01:36:11.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.151+0000 7f3954975640 1 -- 172.21.15.12:0/2533861824 wait complete. 2024-09-11T01:36:11.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.152+0000 7f3954975640 10 monclient: build_initial_monmap 2024-09-11T01:36:11.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.152+0000 7f3954975640 10 monclient: monmap: 2024-09-11T01:36:11.166 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:11.166 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:11.166 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:11.166 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:11.166 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:11.166 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:11.166 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:11.166 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:11.166 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:11.166 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.152+0000 7f3954975640 1 Processor -- start 2024-09-11T01:36:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.152+0000 7f3954975640 1 -- start start 2024-09-11T01:36:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.152+0000 7f3954975640 10 monclient: init 2024-09-11T01:36:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f3954975640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f3954975640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:36:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f3954975640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39500a2200 0x7f39501cb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f3954975640 10 monclient(hunting): picked mon.noname-a con 0x7f39500a2200 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f3954975640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3950133a50 0x7f39501cb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f3954975640 10 monclient(hunting): picked mon.noname-c con 0x7f3950133a50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f3954975640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3950134ad0 0x7f39501388b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f3954975640 10 monclient(hunting): picked mon.noname-b con 0x7f3950134ad0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f3954975640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f3954975640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f395009ffd0 con 0x7f39500a2200 2024-09-11T01:36:11.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f3954975640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:11.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f3954975640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f395009f380 con 0x7f3950134ad0 2024-09-11T01:36:11.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f3954975640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:11.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f3954975640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f39500a0150 con 0x7f3950133a50 2024-09-11T01:36:11.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f3954975640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:11.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f3954975640 10 monclient(hunting): authenticate will time out at 1177.595446s 2024-09-11T01:36:11.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f394e59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39500a2200 0x7f39501cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:11.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f3947fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3950133a50 0x7f39501cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:11.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f394e59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39500a2200 0x7f39501cb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49014/0 (socket says 172.21.15.12:49014) 2024-09-11T01:36:11.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f394e59c640 1 -- 172.21.15.12:0/3027960403 learned_addr learned my addr 172.21.15.12:0/3027960403 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:11.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f394e59c640 10 monclient(hunting): get_auth_request con 0x7f39500a2200 auth_method 0 2024-09-11T01:36:11.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f394e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:11.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f394e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:11.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f394e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:11.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f3947fff640 10 monclient(hunting): get_auth_request con 0x7f3950133a50 auth_method 0 2024-09-11T01:36:11.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f3947fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:11.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f3947fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:11.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f3947fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:11.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f394e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:11.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f394e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:11.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f394e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:11.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f3947fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:11.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f3947fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:11.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f3947fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:11.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f394ed9d640 1 --2- 172.21.15.12:0/3027960403 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3950134ad0 0x7f39501388b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:11.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f394ed9d640 10 monclient(hunting): get_auth_request con 0x7f3950134ad0 auth_method 0 2024-09-11T01:36:11.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f394ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:11.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f394ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:11.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f394ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:11.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f394e59c640 10 monclient(hunting): handle_auth_done global_id 54324 payload 824 2024-09-11T01:36:11.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f394e59c640 1 -- 172.21.15.12:0/3027960403 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3950133a50 msgr2=0x7f39501cb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:11.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f394e59c640 1 --2- 172.21.15.12:0/3027960403 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3950133a50 0x7f39501cb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f394e59c640 1 -- 172.21.15.12:0/3027960403 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3950134ad0 msgr2=0x7f39501388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:11.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f394e59c640 1 --2- 172.21.15.12:0/3027960403 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3950134ad0 0x7f39501388b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f394e59c640 10 monclient: _finish_hunting 0 2024-09-11T01:36:11.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f394e59c640 1 monclient: found mon.noname-a 2024-09-11T01:36:11.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f394e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:11.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f394e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:11.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f394e59c640 1 -- 172.21.15.12:0/3027960403 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f395013a810 con 0x7f39500a2200 2024-09-11T01:36:11.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.153+0000 7f3947fff640 1 --2- 172.21.15.12:0/3027960403 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3950133a50 0x7f39501cb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:36:11.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.156+0000 7f394e59c640 1 -- 172.21.15.12:0/3027960403 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39500a2200 msgr2=0x7f39501cb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:11.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.156+0000 7f394e59c640 1 --2- 172.21.15.12:0/3027960403 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39500a2200 0x7f39501cb220 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f394000e9d0 tx=0x7f394000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:11.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.156+0000 7f3946ffd640 1 -- 172.21.15.12:0/3027960403 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3940017070 con 0x7f39500a2200 2024-09-11T01:36:11.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.156+0000 7f3946ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:11.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.156+0000 7f3946ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:11.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.156+0000 7f3946ffd640 10 monclient: dump: 2024-09-11T01:36:11.171 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:11.171 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:11.171 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.171 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.171 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:11.171 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:11.171 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:11.171 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:11.172 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:11.172 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:11.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.156+0000 7f3946ffd640 10 monclient: _finish_auth 0 2024-09-11T01:36:11.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.156+0000 7f3946ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:36:11.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.156+0000 7f3946ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:11.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.156+0000 7f3954975640 5 monclient: authenticate success, global_id 54324 2024-09-11T01:36:11.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.156+0000 7f3954975640 10 monclient: _renew_subs 2024-09-11T01:36:11.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.156+0000 7f3946ffd640 1 -- 172.21.15.12:0/3027960403 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f394000cc70 con 0x7f39500a2200 2024-09-11T01:36:11.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.156+0000 7f3954975640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:11.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.156+0000 7f3954975640 1 -- 172.21.15.12:0/3027960403 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f395009d480 con 0x7f39500a2200 2024-09-11T01:36:11.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.156+0000 7f3946ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:11.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.156+0000 7f3946ffd640 1 -- 172.21.15.12:0/3027960403 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3940013670 con 0x7f39500a2200 2024-09-11T01:36:11.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.156+0000 7f3946ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:11.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.156+0000 7f3946ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:11.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.156+0000 7f3946ffd640 10 monclient: dump: 2024-09-11T01:36:11.173 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:11.173 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:11.173 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.173 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.173 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:11.173 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:11.173 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:11.173 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:11.173 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:11.173 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:11.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.156+0000 7f394d59a640 4 set_mon_vals no callback set 2024-09-11T01:36:11.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.156+0000 7f3954975640 10 monclient: _renew_subs 2024-09-11T01:36:11.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.156+0000 7f3954975640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:11.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.156+0000 7f3954975640 1 -- 172.21.15.12:0/3027960403 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f395011ca10 con 0x7f39500a2200 2024-09-11T01:36:11.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.157+0000 7f3954975640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:11.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.157+0000 7f3954975640 1 --2- 172.21.15.12:0/3027960403 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3950138df0 0x7f39501391d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:11.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.157+0000 7f3954975640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:11.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.157+0000 7f3954975640 1 -- 172.21.15.12:0/3027960403 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f395013acb0 con 0x7f3950138df0 2024-09-11T01:36:11.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.157+0000 7f3946ffd640 1 -- 172.21.15.12:0/3027960403 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f3940024060 con 0x7f39500a2200 2024-09-11T01:36:11.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.157+0000 7f3947fff640 1 --2- 172.21.15.12:0/3027960403 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3950138df0 0x7f39501391d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:11.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.157+0000 7f3947fff640 10 monclient: get_auth_request con 0x7f3950138df0 auth_method 0 2024-09-11T01:36:11.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.157+0000 7f3947fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:11.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.157+0000 7f3947fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:11.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.157+0000 7f3947fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.157+0000 7f3947fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.157+0000 7f3947fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.157+0000 7f3946ffd640 1 --2- 172.21.15.12:0/3027960403 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f393003e240 0x7f3930040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.157+0000 7f3947fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.157+0000 7f3946ffd640 1 -- 172.21.15.12:0/3027960403 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(225..225 src has 186..225) ==== 4771+0+0 (secure 0 0 0) 0x7f3940052c80 con 0x7f39500a2200 2024-09-11T01:36:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.157+0000 7f394ed9d640 1 --2- 172.21.15.12:0/3027960403 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f393003e240 0x7f3930040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.157+0000 7f3947fff640 10 monclient(hunting): handle_auth_done global_id 74134 payload 824 2024-09-11T01:36:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.157+0000 7f3947fff640 1 -- 172.21.15.12:0/3027960403 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3950134560 con 0x7f3950138df0 2024-09-11T01:36:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.157+0000 7f394ed9d640 10 monclient: get_auth_request con 0x7f393003e240 auth_method 0 2024-09-11T01:36:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.160+0000 7f3947fff640 1 -- 172.21.15.12:0/3027960403 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3950138df0 msgr2=0x7f39501391d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.160+0000 7f3947fff640 1 --2- 172.21.15.12:0/3027960403 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3950138df0 0x7f39501391d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3938011cf0 tx=0x7f393800f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.160+0000 7f3946ffd640 1 -- 172.21.15.12:0/3027960403 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3938004030 con 0x7f3950138df0 2024-09-11T01:36:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.160+0000 7f3946ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:11.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.160+0000 7f3946ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:36:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.160+0000 7f3946ffd640 10 monclient: dump: 2024-09-11T01:36:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:11.176 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.160+0000 7f394ed9d640 1 --2- 172.21.15.12:0/3027960403 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f393003e240 0x7f3930040700 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f39500a72b0 tx=0x7f394802f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:11.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.160+0000 7f3946ffd640 1 -- 172.21.15.12:0/3027960403 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3938004030 con 0x7f3950138df0 2024-09-11T01:36:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.160+0000 7f3946ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.160+0000 7f3946ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.160+0000 7f3946ffd640 1 -- 172.21.15.12:0/3027960403 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3950138df0 msgr2=0x7f39501391d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.160+0000 7f3946ffd640 1 --2- 172.21.15.12:0/3027960403 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3950138df0 0x7f39501391d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3938011cf0 tx=0x7f393800f610 comp rx=0 tx=0).stop 2024-09-11T01:36:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.160+0000 7f3946ffd640 1 -- 172.21.15.12:0/3027960403 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3950138df0 msgr2=0x7f39501391d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:11.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.160+0000 7f3946ffd640 1 --2- 172.21.15.12:0/3027960403 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3950138df0 0x7f39501391d0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3938011cf0 tx=0x7f393800f610 comp rx=0 tx=0).stop 2024-09-11T01:36:11.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.167+0000 7f3954975640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:36:11.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.167+0000 7f3954975640 1 --2- 172.21.15.12:0/3027960403 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3950139710 0x7f39500726b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:11.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.167+0000 7f3954975640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:11.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.167+0000 7f3954975640 1 -- 172.21.15.12:0/3027960403 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f395006d480 con 0x7f3950139710 2024-09-11T01:36:11.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.167+0000 7f3947fff640 1 --2- 172.21.15.12:0/3027960403 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3950139710 0x7f39500726b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:11.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.167+0000 7f3947fff640 10 monclient: get_auth_request con 0x7f3950139710 auth_method 0 2024-09-11T01:36:11.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.167+0000 7f3947fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:11.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.167+0000 7f3947fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:11.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.167+0000 7f3947fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:11.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.167+0000 7f3947fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:11.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.167+0000 7f3947fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:11.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.167+0000 7f3947fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:11.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.167+0000 7f3947fff640 10 monclient(hunting): handle_auth_done global_id 74137 payload 824 2024-09-11T01:36:11.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.167+0000 7f3947fff640 1 -- 172.21.15.12:0/3027960403 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f39501cbca0 con 0x7f3950139710 2024-09-11T01:36:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.170+0000 7f3947fff640 1 -- 172.21.15.12:0/3027960403 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3950139710 msgr2=0x7f39500726b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.170+0000 7f3947fff640 1 --2- 172.21.15.12:0/3027960403 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3950139710 0x7f39500726b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f393801b000 tx=0x7f39380046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.172+0000 7f3946ffd640 1 -- 172.21.15.12:0/3027960403 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3938003c30 con 0x7f3950139710 2024-09-11T01:36:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.172+0000 7f3946ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.172+0000 7f3946ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:36:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.172+0000 7f3946ffd640 10 monclient: dump: 2024-09-11T01:36:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:11.180 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:11.181 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.172+0000 7f3946ffd640 1 -- 172.21.15.12:0/3027960403 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4707 (secure 0 0 0) 0x7f3938003c30 con 0x7f3950139710 2024-09-11T01:36:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.172+0000 7f3946ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:36:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.172+0000 7f3946ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.172+0000 7f3946ffd640 1 -- 172.21.15.12:0/3027960403 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3950139710 msgr2=0x7f39500726b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.172+0000 7f3946ffd640 1 --2- 172.21.15.12:0/3027960403 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3950139710 0x7f39500726b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f393801b000 tx=0x7f39380046c0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.172+0000 7f3946ffd640 1 -- 172.21.15.12:0/3027960403 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3950139710 msgr2=0x7f39500726b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:11.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.172+0000 7f3946ffd640 1 --2- 172.21.15.12:0/3027960403 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3950139710 0x7f39500726b0 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f393801b000 tx=0x7f39380046c0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.181 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:36:11.182 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:36:11.182 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:36:11.182 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:36:11.182 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 74, 2024-09-11T01:36:11.182 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 22951, 2024-09-11T01:36:11.182 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:36:11.182 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:36:11.182 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:36:11.182 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:36:11.182 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:11.182 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:36:11.182 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:11.182 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:36:11.182 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:36:11.182 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:11.183 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:11.183 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:11.183 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:11.183 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:11.183 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:11.183 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:11.183 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:11.183 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:11.183 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:11.183 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:11.183 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:11.183 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:36:11.183 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:36:11.184 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:11.184 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:11.184 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:11.184 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:11.184 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:11.184 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:11.184 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:11.184 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:11.184 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:11.184 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:11.184 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:11.184 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:11.184 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:11.184 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:36:11.185 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:36:11.185 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:36:11.185 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:36:11.185 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:36:11.185 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:36:11.185 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:11.185 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:11.185 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:36:11.185 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:36:11.185 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:36:11.185 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:36:11.185 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:36:11.185 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:36:11.185 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:36:11.186 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:11.186 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:36:11.186 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:11.186 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:11.186 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:11.186 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:11.186 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:11.186 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:11.186 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:11.186 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:11.186 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:11.186 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:11.186 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:11.186 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:11.187 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:36:11.187 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:11.187 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:36:11.187 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:11.187 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:36:11.187 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:36:11.187 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:11.187 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:11.187 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:11.187 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:11.187 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:36:11.187 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:11.187 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:11.187 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:11.187 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:11.188 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:36:11.188 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:11.188 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:11.188 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:11.188 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:11.188 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:36:11.188 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:36:11.188 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:11.188 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:11.188 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:11.188 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:11.188 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:11.188 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:36:11.188 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:36:11.189 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:11.189 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:11.189 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:11.189 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:11.189 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:36:11.189 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:11.189 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:11.189 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:11.189 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:11.189 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:36:11.189 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:11.189 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:11.189 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:11.190 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:11.190 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:36:11.190 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:36:11.190 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:11.190 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:11.190 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:11.190 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:11.190 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:11.190 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:36:11.190 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:36:11.190 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:11.190 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:11.190 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:11.190 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:11.190 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:36:11.191 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:11.191 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:11.191 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:11.191 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:11.191 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:36:11.191 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:11.191 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:11.191 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:11.191 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:11.191 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:36:11.191 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:36:11.191 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:11.191 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:11.191 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:11.192 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:11.192 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:11.192 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:11.192 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:36:11.192 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:36:11.192 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:11.192 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:11.192 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:11.192 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:36:11.192 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:11.192 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:11.192 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:36:11.192 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:11.192 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:11.193 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:11.193 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:36:11.193 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:11.193 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:11.193 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:36:11.193 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:11.193 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:11.193 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:11.193 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:36:11.193 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:11.193 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:11.193 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:11.193 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:36:11.193 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:36:11.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.172+0000 7f3954975640 1 -- 172.21.15.12:0/3027960403 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f393003e240 msgr2=0x7f3930040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:11.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.172+0000 7f3954975640 1 --2- 172.21.15.12:0/3027960403 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f393003e240 0x7f3930040700 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f39500a72b0 tx=0x7f394802f000 comp rx=0 tx=0).stop 2024-09-11T01:36:11.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.172+0000 7f3954975640 10 monclient: shutdown 2024-09-11T01:36:11.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.172+0000 7f3954975640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:11.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.172+0000 7f3954975640 1 -- 172.21.15.12:0/3027960403 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39500a2200 msgr2=0x7f39501cb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:11.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.172+0000 7f3954975640 1 --2- 172.21.15.12:0/3027960403 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39500a2200 0x7f39501cb220 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f394000e9d0 tx=0x7f394000ee90 comp rx=0 tx=0).stop 2024-09-11T01:36:11.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.173+0000 7f394ed9d640 1 -- 172.21.15.12:0/3027960403 reap_dead start 2024-09-11T01:36:11.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.173+0000 7f394ed9d640 1 -- 172.21.15.12:0/3027960403 reap_dead start 2024-09-11T01:36:11.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.173+0000 7f3954975640 1 -- 172.21.15.12:0/3027960403 shutdown_connections 2024-09-11T01:36:11.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.173+0000 7f3954975640 1 -- 172.21.15.12:0/3027960403 >> 172.21.15.12:0/3027960403 conn(0x7f39500a2f70 msgr2=0x7f395009d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:11.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.173+0000 7f3954975640 1 -- 172.21.15.12:0/3027960403 shutdown_connections 2024-09-11T01:36:11.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.173+0000 7f3954975640 1 -- 172.21.15.12:0/3027960403 wait complete. 2024-09-11T01:36:11.196 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:36:11.196 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-11T01:36:11.196 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T01:36:11.197 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-11T01:36:11.197 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-11T01:36:11.197 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-11T01:36:11.197 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:36:11.209 INFO:tasks.ceph.mon.c:Stopped 2024-09-11T01:36:11.209 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:36:11.209 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:36:11.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.348+0000 7f0bcc3ec640 10 monclient: get_monmap_and_config 2024-09-11T01:36:11.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.348+0000 7f0bcc3ec640 10 monclient: build_initial_monmap 2024-09-11T01:36:11.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.348+0000 7f0bcc3ec640 10 monclient: monmap: 2024-09-11T01:36:11.350 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:11.350 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:11.350 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:11.348927+0000 2024-09-11T01:36:11.350 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:11.348927+0000 2024-09-11T01:36:11.350 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:11.350 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:11.350 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:11.350 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:11.350 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:11.350 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:11.350 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:11.350 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:11.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.348+0000 7f0bcc3ec640 1 Processor -- start 2024-09-11T01:36:11.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.348+0000 7f0bcc3ec640 1 -- start start 2024-09-11T01:36:11.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.348+0000 7f0bcc3ec640 10 monclient: init 2024-09-11T01:36:11.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.349+0000 7f0bcc3ec640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:11.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.349+0000 7f0bcc3ec640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-11T01:36:11.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.349+0000 7f0bcc3ec640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0bc40a2e90 0x7f0bc406dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:11.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.349+0000 7f0bcc3ec640 10 monclient(hunting): picked mon.noname-a con 0x7f0bc40a2e90 addr v2:172.21.15.12:3300/0 2024-09-11T01:36:11.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.349+0000 7f0bcc3ec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0bc40a3270 addr v1:172.21.15.12:6789/0 2024-09-11T01:36:11.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.349+0000 7f0bcc3ec640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0bc413d490 0x7f0bc413f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:11.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.349+0000 7f0bcc3ec640 10 monclient(hunting): picked mon.noname-c con 0x7f0bc413d490 addr v2:172.21.15.71:3300/0 2024-09-11T01:36:11.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.349+0000 7f0bcc3ec640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bc40a46c0 con 0x7f0bc40a3270 2024-09-11T01:36:11.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.349+0000 7f0bcc3ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:11.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.349+0000 7f0bcc3ec640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f0bc40a4900 con 0x7f0bc40a2e90 2024-09-11T01:36:11.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.349+0000 7f0bcc3ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:11.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.349+0000 7f0bcc3ec640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f0bc406ee50 con 0x7f0bc413d490 2024-09-11T01:36:11.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.349+0000 7f0bcc3ec640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:11.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.349+0000 7f0bcc3ec640 10 monclient(hunting): authenticate will time out at 1177.791532s 2024-09-11T01:36:11.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.349+0000 7f0bca188640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0bc40a2e90 0x7f0bc406dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:11.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.349+0000 7f0bca188640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0bc40a2e90 0x7f0bc406dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49038/0 (socket says 172.21.15.12:49038) 2024-09-11T01:36:11.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.349+0000 7f0bca188640 1 -- 172.21.15.12:0/30901588 learned_addr learned my addr 172.21.15.12:0/30901588 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:11.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.349+0000 7f0bca188640 10 monclient(hunting): get_auth_request con 0x7f0bc40a2e90 auth_method 0 2024-09-11T01:36:11.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.349+0000 7f0bca188640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:11.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.349+0000 7f0bca188640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:11.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.349+0000 7f0bca188640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:11.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.349+0000 7f0bca188640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:11.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.349+0000 7f0bca188640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:11.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.349+0000 7f0bca188640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:11.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.349+0000 7f0bca989640 1 --2- 172.21.15.12:0/30901588 >> v2:172.21.15.71:3300/0 conn(0x7f0bc413d490 0x7f0bc413f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:11.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.349+0000 7f0bca989640 10 monclient(hunting): get_auth_request con 0x7f0bc413d490 auth_method 0 2024-09-11T01:36:11.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.350+0000 7f0bca989640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:11.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.350+0000 7f0bca989640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:11.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.350+0000 7f0bca989640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:11.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.350+0000 7f0bca989640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:11.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.350+0000 7f0bca989640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:11.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.350+0000 7f0bca989640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:11.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.350+0000 7f0bca989640 10 monclient(hunting): handle_auth_done global_id 74140 payload 274 2024-09-11T01:36:11.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.350+0000 7f0bca989640 1 -- 172.21.15.12:0/30901588 >> v2:172.21.15.12:3300/0 conn(0x7f0bc40a2e90 msgr2=0x7f0bc406dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:11.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.350+0000 7f0bca989640 1 --2- 172.21.15.12:0/30901588 >> v2:172.21.15.12:3300/0 conn(0x7f0bc40a2e90 0x7f0bc406dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.350+0000 7f0bca989640 1 -- 172.21.15.12:0/30901588 >> v1:172.21.15.12:6789/0 conn(0x7f0bc40a3270 legacy=0x7f0bc40a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:11.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.350+0000 7f0bca989640 10 monclient: _finish_hunting 0 2024-09-11T01:36:11.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.350+0000 7f0bca989640 1 monclient: found mon.noname-c 2024-09-11T01:36:11.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.350+0000 7f0bca989640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:11.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.350+0000 7f0bca989640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:36:11.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.350+0000 7f0bca989640 1 -- 172.21.15.12:0/30901588 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bc406eb70 con 0x7f0bc413d490 2024-09-11T01:36:11.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.353+0000 7f0bca989640 1 -- 172.21.15.12:0/30901588 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0bc413d490 msgr2=0x7f0bc413f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:11.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.353+0000 7f0bca989640 1 --2- 172.21.15.12:0/30901588 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0bc413d490 0x7f0bc413f880 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4071140 tx=0x7f0bbc02a480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26949cf5334768b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:11.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.353+0000 7f0bc9186640 1 -- 172.21.15.12:0/30901588 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc007950 con 0x7f0bc413d490 2024-09-11T01:36:11.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.353+0000 7f0bc9186640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:11.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.353+0000 7f0bc9186640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:36:11.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.353+0000 7f0bc9186640 10 monclient: dump: 2024-09-11T01:36:11.363 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:11.363 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:11.363 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.363 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.363 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:11.363 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:11.364 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:11.364 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:11.364 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:11.364 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:11.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.353+0000 7f0bc9186640 10 monclient: _finish_auth 0 2024-09-11T01:36:11.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.353+0000 7f0bc9186640 10 monclient: _check_auth_tickets 2024-09-11T01:36:11.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.353+0000 7f0bc9186640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:11.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.353+0000 7f0bc9186640 1 -- 172.21.15.12:0/30901588 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bbc007af0 con 0x7f0bc413d490 2024-09-11T01:36:11.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.353+0000 7f0bcc3ec640 5 monclient: authenticate success, global_id 74140 2024-09-11T01:36:11.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.353+0000 7f0bcc3ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:36:11.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.353+0000 7f0bc9186640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:11.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.353+0000 7f0bc9186640 1 -- 172.21.15.12:0/30901588 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc031400 con 0x7f0bc413d490 2024-09-11T01:36:11.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.353+0000 7f0bcc3ec640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.353+0000 7f0bcc3ec640 4 set_mon_vals no callback set 2024-09-11T01:36:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.353+0000 7f0bc9186640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.353+0000 7f0bc9186640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:36:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.353+0000 7f0bc9186640 10 monclient: dump: 2024-09-11T01:36:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:11.365 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:11.366 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:11.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.354+0000 7f0bcc3ec640 10 monclient: shutdown 2024-09-11T01:36:11.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.354+0000 7f0bcc3ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:11.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.354+0000 7f0bcc3ec640 1 -- 172.21.15.12:0/30901588 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0bc413d490 msgr2=0x7f0bc413f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:11.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.354+0000 7f0bcc3ec640 1 --2- 172.21.15.12:0/30901588 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0bc413d490 0x7f0bc413f880 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4071140 tx=0x7f0bbc02a480 comp rx=0 tx=0).stop 2024-09-11T01:36:11.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.354+0000 7f0bcc3ec640 1 -- 172.21.15.12:0/30901588 shutdown_connections 2024-09-11T01:36:11.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.354+0000 7f0bcc3ec640 1 --2- 172.21.15.12:0/30901588 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0bc413d490 0x7f0bc413f880 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.354+0000 7f0bcc3ec640 1 --2- 172.21.15.12:0/30901588 >> v2:172.21.15.12:3300/0 conn(0x7f0bc40a2e90 0x7f0bc406dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.354+0000 7f0bcc3ec640 1 -- 172.21.15.12:0/30901588 >> 172.21.15.12:0/30901588 conn(0x7f0bc40a8490 msgr2=0x7f0bc40a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:11.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.354+0000 7f0bcc3ec640 1 -- 172.21.15.12:0/30901588 shutdown_connections 2024-09-11T01:36:11.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.354+0000 7f0bcc3ec640 1 -- 172.21.15.12:0/30901588 wait complete. 2024-09-11T01:36:11.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.354+0000 7f0bcc3ec640 10 monclient: build_initial_monmap 2024-09-11T01:36:11.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.354+0000 7f0bcc3ec640 10 monclient: monmap: 2024-09-11T01:36:11.366 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:11.367 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:11.367 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:11.367 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:11.367 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:11.367 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:11.367 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:11.367 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:11.367 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:11.367 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:11.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bcc3ec640 1 Processor -- start 2024-09-11T01:36:11.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bcc3ec640 1 -- start start 2024-09-11T01:36:11.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bcc3ec640 10 monclient: init 2024-09-11T01:36:11.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bcc3ec640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:11.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bcc3ec640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:36:11.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bcc3ec640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bc40a2e90 0x7f0bc409c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:11.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bcc3ec640 10 monclient(hunting): picked mon.noname-a con 0x7f0bc40a2e90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:11.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bcc3ec640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0bc40a3270 0x7f0bc409c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:11.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bcc3ec640 10 monclient(hunting): picked mon.noname-b con 0x7f0bc40a3270 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:11.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bcc3ec640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0bc413d490 0x7f0bc409ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:11.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bcc3ec640 10 monclient(hunting): picked mon.noname-c con 0x7f0bc413d490 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:11.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bcc3ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:11.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bcc3ec640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0bc406fd20 con 0x7f0bc40a2e90 2024-09-11T01:36:11.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bcc3ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:11.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bcc3ec640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0bc406fba0 con 0x7f0bc40a3270 2024-09-11T01:36:11.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bcc3ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:11.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bcc3ec640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f0bc413fdc0 con 0x7f0bc413d490 2024-09-11T01:36:11.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bcc3ec640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:11.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bcc3ec640 10 monclient(hunting): authenticate will time out at 1177.797752s 2024-09-11T01:36:11.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bca989640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0bc413d490 msgr2=0x7f0bc409ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:36:11.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bca989640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0bc413d490 0x7f0bc409ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:11.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bca188640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bc40a2e90 0x7f0bc409c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:11.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bca188640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bc40a2e90 0x7f0bc409c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49054/0 (socket says 172.21.15.12:49054) 2024-09-11T01:36:11.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bca188640 1 -- 172.21.15.12:0/1195354410 learned_addr learned my addr 172.21.15.12:0/1195354410 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:11.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bca188640 10 monclient(hunting): get_auth_request con 0x7f0bc40a2e90 auth_method 0 2024-09-11T01:36:11.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bca188640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:11.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bca188640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:11.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bca188640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:11.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bca188640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:11.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bca188640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:11.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.355+0000 7f0bca188640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:11.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.356+0000 7f0bca188640 10 monclient(hunting): handle_auth_done global_id 54330 payload 824 2024-09-11T01:36:11.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.356+0000 7f0bca188640 1 -- 172.21.15.12:0/1195354410 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0bc413d490 msgr2=0x7f0bc409ce10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:36:11.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.356+0000 7f0bca188640 1 --2- 172.21.15.12:0/1195354410 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0bc413d490 0x7f0bc409ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.356+0000 7f0bca188640 1 -- 172.21.15.12:0/1195354410 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0bc40a3270 msgr2=0x7f0bc409c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:11.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.356+0000 7f0bca188640 1 --2- 172.21.15.12:0/1195354410 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0bc40a3270 0x7f0bc409c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.356+0000 7f0bca188640 10 monclient: _finish_hunting 0 2024-09-11T01:36:11.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.356+0000 7f0bca188640 1 monclient: found mon.noname-a 2024-09-11T01:36:11.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.356+0000 7f0bca188640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:11.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.356+0000 7f0bca188640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:11.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.356+0000 7f0bca188640 1 -- 172.21.15.12:0/1195354410 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bc413f920 con 0x7f0bc40a2e90 2024-09-11T01:36:11.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.358+0000 7f0bca188640 1 -- 172.21.15.12:0/1195354410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bc40a2e90 msgr2=0x7f0bc409c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:11.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.358+0000 7f0bca188640 1 --2- 172.21.15.12:0/1195354410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bc40a2e90 0x7f0bc409c390 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0009fd0 tx=0x7f0bc000ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:11.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.358+0000 7f0bbaffd640 1 -- 172.21.15.12:0/1195354410 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0017070 con 0x7f0bc40a2e90 2024-09-11T01:36:11.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.358+0000 7f0bbaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:11.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.358+0000 7f0bbaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:11.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.358+0000 7f0bbaffd640 10 monclient: dump: 2024-09-11T01:36:11.371 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:11.371 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:11.371 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.371 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.372 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:11.372 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:11.372 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:11.372 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:11.372 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:11.372 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:11.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.358+0000 7f0bbaffd640 10 monclient: _finish_auth 0 2024-09-11T01:36:11.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.358+0000 7f0bbaffd640 10 monclient: _check_auth_tickets 2024-09-11T01:36:11.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.358+0000 7f0bbaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:11.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.358+0000 7f0bcc3ec640 5 monclient: authenticate success, global_id 54330 2024-09-11T01:36:11.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.358+0000 7f0bbaffd640 1 -- 172.21.15.12:0/1195354410 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bc000cb50 con 0x7f0bc40a2e90 2024-09-11T01:36:11.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.358+0000 7f0bcc3ec640 10 monclient: _renew_subs 2024-09-11T01:36:11.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.358+0000 7f0bcc3ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:11.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.358+0000 7f0bcc3ec640 1 -- 172.21.15.12:0/1195354410 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bc40a6400 con 0x7f0bc40a2e90 2024-09-11T01:36:11.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.358+0000 7f0bbaffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:11.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.358+0000 7f0bbaffd640 1 -- 172.21.15.12:0/1195354410 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0013470 con 0x7f0bc40a2e90 2024-09-11T01:36:11.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.358+0000 7f0bbaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:11.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.358+0000 7f0bbaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:11.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.358+0000 7f0bbaffd640 10 monclient: dump: 2024-09-11T01:36:11.373 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:11.373 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:11.373 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.373 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:11.373 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:11.373 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:11.373 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:11.373 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:11.374 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:11.374 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:11.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.358+0000 7f0bcc3ec640 10 monclient: _renew_subs 2024-09-11T01:36:11.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.358+0000 7f0bc8985640 4 set_mon_vals no callback set 2024-09-11T01:36:11.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.358+0000 7f0bcc3ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:11.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.358+0000 7f0bcc3ec640 1 -- 172.21.15.12:0/1195354410 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bc409f310 con 0x7f0bc40a2e90 2024-09-11T01:36:11.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.359+0000 7f0bcc3ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:11.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.359+0000 7f0bcc3ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:11.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.359+0000 7f0bcc3ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:11.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.359+0000 7f0bcc3ec640 1 -- 172.21.15.12:0/1195354410 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bc406dc60 con 0x7f0bc40a2e90 2024-09-11T01:36:11.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.359+0000 7f0bbaffd640 1 -- 172.21.15.12:0/1195354410 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f0bc0024060 con 0x7f0bc40a2e90 2024-09-11T01:36:11.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.359+0000 7f0bbaffd640 1 --2- 172.21.15.12:0/1195354410 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0ba803e1c0 0x7f0ba8040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:11.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.359+0000 7f0bbaffd640 1 -- 172.21.15.12:0/1195354410 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(225..225 src has 186..225) ==== 4771+0+0 (secure 0 0 0) 0x7f0bc0052da0 con 0x7f0bc40a2e90 2024-09-11T01:36:11.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.360+0000 7f0bc9987640 1 --2- 172.21.15.12:0/1195354410 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0ba803e1c0 0x7f0ba8040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:11.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.360+0000 7f0bc9987640 10 monclient: get_auth_request con 0x7f0ba803e1c0 auth_method 0 2024-09-11T01:36:11.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.362+0000 7f0bc9987640 1 --2- 172.21.15.12:0/1195354410 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0ba803e1c0 0x7f0ba8040680 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0bc409e930 tx=0x7f0bb40079b0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:11.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.364+0000 7f0bbaffd640 1 -- 172.21.15.12:0/1195354410 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0bc0023d00 con 0x7f0bc40a2e90 2024-09-11T01:36:11.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.364+0000 7f0bbaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:11.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.364+0000 7f0bbaffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:11.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.620+0000 7f0bcc3ec640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:36:11.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.620+0000 7f0bcc3ec640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:36:11.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.620+0000 7f0bcc3ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:11.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.620+0000 7f0bcc3ec640 1 -- 172.21.15.12:0/1195354410 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0bc409f0f0 con 0x7f0bc40a2e90 2024-09-11T01:36:11.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.620+0000 7f0bbaffd640 1 -- 172.21.15.12:0/1195354410 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f0bc001b3b0 con 0x7f0bc40a2e90 2024-09-11T01:36:11.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.620+0000 7f0bbaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:36:11.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.620+0000 7f0bbaffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:11.623 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":74,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:36:11.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.622+0000 7f0bcc3ec640 1 -- 172.21.15.12:0/1195354410 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0ba803e1c0 msgr2=0x7f0ba8040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:11.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.622+0000 7f0bcc3ec640 1 --2- 172.21.15.12:0/1195354410 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0ba803e1c0 0x7f0ba8040680 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0bc409e930 tx=0x7f0bb40079b0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.622+0000 7f0bcc3ec640 10 monclient: shutdown 2024-09-11T01:36:11.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.622+0000 7f0bcc3ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:11.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.622+0000 7f0bcc3ec640 1 -- 172.21.15.12:0/1195354410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bc40a2e90 msgr2=0x7f0bc409c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:11.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.622+0000 7f0bcc3ec640 1 --2- 172.21.15.12:0/1195354410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bc40a2e90 0x7f0bc409c390 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0009fd0 tx=0x7f0bc000ed40 comp rx=0 tx=0).stop 2024-09-11T01:36:11.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.622+0000 7f0bcc3ec640 1 -- 172.21.15.12:0/1195354410 shutdown_connections 2024-09-11T01:36:11.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.622+0000 7f0bcc3ec640 1 --2- 172.21.15.12:0/1195354410 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0bc413d490 0x7f0bc409ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.622+0000 7f0bcc3ec640 1 --2- 172.21.15.12:0/1195354410 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0bc40a3270 0x7f0bc409c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.622+0000 7f0bcc3ec640 1 --2- 172.21.15.12:0/1195354410 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0ba803e1c0 0x7f0ba8040680 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.622+0000 7f0bcc3ec640 1 --2- 172.21.15.12:0/1195354410 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bc40a2e90 0x7f0bc409c390 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:11.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.623+0000 7f0bcc3ec640 1 -- 172.21.15.12:0/1195354410 >> 172.21.15.12:0/1195354410 conn(0x7f0bc40a8490 msgr2=0x7f0bc413c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:11.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.623+0000 7f0bcc3ec640 1 -- 172.21.15.12:0/1195354410 shutdown_connections 2024-09-11T01:36:11.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:11.623+0000 7f0bcc3ec640 1 -- 172.21.15.12:0/1195354410 wait complete. 2024-09-11T01:36:14.636 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:36:14.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.748+0000 7f08918e2640 10 monclient: get_monmap_and_config 2024-09-11T01:36:14.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.748+0000 7f08918e2640 10 monclient: build_initial_monmap 2024-09-11T01:36:14.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.748+0000 7f08918e2640 10 monclient: monmap: 2024-09-11T01:36:14.750 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:14.750 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:14.750 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:14.749154+0000 2024-09-11T01:36:14.750 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:14.749154+0000 2024-09-11T01:36:14.750 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:14.750 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:14.750 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:14.751 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:14.751 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:14.751 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:14.751 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:14.751 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:14.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f08918e2640 1 Processor -- start 2024-09-11T01:36:14.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f08918e2640 1 -- start start 2024-09-11T01:36:14.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f08918e2640 10 monclient: init 2024-09-11T01:36:14.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f08918e2640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:14.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f08918e2640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-11T01:36:14.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f08918e2640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f088c0a2e90 0x7f088c06dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:14.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f08918e2640 10 monclient(hunting): picked mon.noname-c con 0x7f088c0a2e90 addr v2:172.21.15.71:3300/0 2024-09-11T01:36:14.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f08918e2640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f088c0a3270 0x7f088c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:14.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f08918e2640 10 monclient(hunting): picked mon.noname-a con 0x7f088c0a3270 addr v2:172.21.15.12:3300/0 2024-09-11T01:36:14.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f08918e2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f088c138040 0x7f088c1384a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:14.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f08918e2640 10 monclient(hunting): picked mon.noname-b con 0x7f088c138040 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:14.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f08918e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:14.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f08918e2640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f088c0a4900 con 0x7f088c0a3270 2024-09-11T01:36:14.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f08918e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:14.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f08918e2640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f088c0a46c0 con 0x7f088c0a2e90 2024-09-11T01:36:14.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f08918e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:14.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f08918e2640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f088c06ee50 con 0x7f088c138040 2024-09-11T01:36:14.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f08918e2640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:14.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f08918e2640 10 monclient(hunting): authenticate will time out at 1181.191627s 2024-09-11T01:36:14.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f088b7fe640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f088c138040 msgr2=0x7f088c1384a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:36:14.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f088b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f088c138040 0x7f088c1384a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:14.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f088a7fc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f088c0a3270 0x7f088c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:14.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f088a7fc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f088c0a3270 0x7f088c0a3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49056/0 (socket says 172.21.15.12:49056) 2024-09-11T01:36:14.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f088a7fc640 1 -- 172.21.15.12:0/164261525 learned_addr learned my addr 172.21.15.12:0/164261525 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:14.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f088a7fc640 10 monclient(hunting): get_auth_request con 0x7f088c0a3270 auth_method 0 2024-09-11T01:36:14.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f088a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:14.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f088a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:14.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f088a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:14.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f088a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:14.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f088a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:14.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.749+0000 7f088a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:14.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.750+0000 7f088affd640 1 --2- 172.21.15.12:0/164261525 >> v2:172.21.15.71:3300/0 conn(0x7f088c0a2e90 0x7f088c06dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:14.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.750+0000 7f088affd640 10 monclient(hunting): get_auth_request con 0x7f088c0a2e90 auth_method 0 2024-09-11T01:36:14.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.750+0000 7f088affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:14.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.750+0000 7f088affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:14.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.750+0000 7f088affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:14.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.750+0000 7f088a7fc640 10 monclient(hunting): handle_auth_done global_id 54333 payload 274 2024-09-11T01:36:14.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.750+0000 7f088a7fc640 1 -- 172.21.15.12:0/164261525 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f088c138040 msgr2=0x7f088c1384a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:36:14.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.750+0000 7f088a7fc640 1 --2- 172.21.15.12:0/164261525 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f088c138040 0x7f088c1384a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:14.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.750+0000 7f088a7fc640 1 -- 172.21.15.12:0/164261525 >> v2:172.21.15.71:3300/0 conn(0x7f088c0a2e90 msgr2=0x7f088c06dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:14.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.750+0000 7f088a7fc640 1 --2- 172.21.15.12:0/164261525 >> v2:172.21.15.71:3300/0 conn(0x7f088c0a2e90 0x7f088c06dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:14.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.750+0000 7f088a7fc640 10 monclient: _finish_hunting 0 2024-09-11T01:36:14.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.750+0000 7f088a7fc640 1 monclient: found mon.noname-a 2024-09-11T01:36:14.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.750+0000 7f088a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:14.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.750+0000 7f088a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:14.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.750+0000 7f088a7fc640 1 -- 172.21.15.12:0/164261525 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f088c06eb70 con 0x7f088c0a3270 2024-09-11T01:36:14.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f088a7fc640 1 -- 172.21.15.12:0/164261525 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f088c0a3270 msgr2=0x7f088c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:14.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f088a7fc640 1 --2- 172.21.15.12:0/164261525 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f088c0a3270 0x7f088c0a3680 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f088c071140 tx=0x7f087c02a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1322aa96457d472 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:14.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f0889ffb640 1 -- 172.21.15.12:0/164261525 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c037070 con 0x7f088c0a3270 2024-09-11T01:36:14.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f0889ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:14.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f0889ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:14.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f0889ffb640 10 monclient: dump: 2024-09-11T01:36:14.761 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:14.761 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:14.761 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:14.761 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:14.762 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:14.762 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:14.762 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:14.762 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:14.762 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:14.762 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:14.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f0889ffb640 10 monclient: _finish_auth 0 2024-09-11T01:36:14.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f0889ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:36:14.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f0889ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:14.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f0889ffb640 1 -- 172.21.15.12:0/164261525 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f087c032640 con 0x7f088c0a3270 2024-09-11T01:36:14.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f08918e2640 5 monclient: authenticate success, global_id 54333 2024-09-11T01:36:14.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f08918e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:36:14.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f0889ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:14.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f0889ffb640 1 -- 172.21.15.12:0/164261525 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c032a10 con 0x7f088c0a3270 2024-09-11T01:36:14.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f0889ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:14.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f0889ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:14.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f0889ffb640 10 monclient: dump: 2024-09-11T01:36:14.763 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:14.763 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:14.763 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:14.763 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:14.763 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:14.763 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:14.763 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:14.764 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:14.764 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:14.764 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:14.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f08918e2640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:14.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f08918e2640 4 set_mon_vals no callback set 2024-09-11T01:36:14.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f08918e2640 10 monclient: shutdown 2024-09-11T01:36:14.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f08918e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:14.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f08918e2640 1 -- 172.21.15.12:0/164261525 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f088c0a3270 msgr2=0x7f088c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:14.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f08918e2640 1 --2- 172.21.15.12:0/164261525 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f088c0a3270 0x7f088c0a3680 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f088c071140 tx=0x7f087c02a0d0 comp rx=0 tx=0).stop 2024-09-11T01:36:14.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f08918e2640 1 -- 172.21.15.12:0/164261525 shutdown_connections 2024-09-11T01:36:14.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f08918e2640 1 --2- 172.21.15.12:0/164261525 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f088c138040 0x7f088c1384a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:14.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f08918e2640 1 --2- 172.21.15.12:0/164261525 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f088c0a3270 0x7f088c0a3680 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:14.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f08918e2640 1 --2- 172.21.15.12:0/164261525 >> v2:172.21.15.71:3300/0 conn(0x7f088c0a2e90 0x7f088c06dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f08918e2640 1 -- 172.21.15.12:0/164261525 >> 172.21.15.12:0/164261525 conn(0x7f088c0a8490 msgr2=0x7f088c0a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f08918e2640 1 -- 172.21.15.12:0/164261525 shutdown_connections 2024-09-11T01:36:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.753+0000 7f08918e2640 1 -- 172.21.15.12:0/164261525 wait complete. 2024-09-11T01:36:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.754+0000 7f08918e2640 10 monclient: build_initial_monmap 2024-09-11T01:36:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.754+0000 7f08918e2640 10 monclient: monmap: 2024-09-11T01:36:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:14.766 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:14.766 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:14.766 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:14.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.754+0000 7f08918e2640 1 Processor -- start 2024-09-11T01:36:14.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f08918e2640 1 -- start start 2024-09-11T01:36:14.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f08918e2640 10 monclient: init 2024-09-11T01:36:14.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f08918e2640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:14.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f08918e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:36:14.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f08918e2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f088c0a2e90 0x7f088c1cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:14.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f08918e2640 10 monclient(hunting): picked mon.noname-a con 0x7f088c0a2e90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:14.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f08918e2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f088c0a3270 0x7f088c1cda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:14.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f08918e2640 10 monclient(hunting): picked mon.noname-c con 0x7f088c0a3270 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:14.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f08918e2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f088c138040 0x7f088c1248f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:14.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f08918e2640 10 monclient(hunting): picked mon.noname-b con 0x7f088c138040 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:14.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f08918e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:14.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f08918e2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f088c06fba0 con 0x7f088c0a2e90 2024-09-11T01:36:14.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f08918e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:14.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f08918e2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f088c0a3bc0 con 0x7f088c138040 2024-09-11T01:36:14.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f08918e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:14.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f08918e2640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f088c06fd20 con 0x7f088c0a3270 2024-09-11T01:36:14.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f08918e2640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:14.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f08918e2640 10 monclient(hunting): authenticate will time out at 1181.197698s 2024-09-11T01:36:14.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f088a7fc640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f088c0a3270 msgr2=0x7f088c1cda20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:36:14.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f088a7fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f088c0a3270 0x7f088c1cda20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:14.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f088affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f088c0a2e90 0x7f088c1cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:14.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f088affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f088c0a2e90 0x7f088c1cd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49058/0 (socket says 172.21.15.12:49058) 2024-09-11T01:36:14.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f088affd640 1 -- 172.21.15.12:0/1483388395 learned_addr learned my addr 172.21.15.12:0/1483388395 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:14.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f088affd640 10 monclient(hunting): get_auth_request con 0x7f088c0a2e90 auth_method 0 2024-09-11T01:36:14.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f088affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:14.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f088affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:14.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f088affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:14.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f088affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:14.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f088affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:14.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f088affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:14.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f088b7fe640 1 --2- 172.21.15.12:0/1483388395 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f088c138040 0x7f088c1248f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:14.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f088b7fe640 10 monclient(hunting): get_auth_request con 0x7f088c138040 auth_method 0 2024-09-11T01:36:14.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f088b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:14.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f088b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:14.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.755+0000 7f088b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:14.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.756+0000 7f088affd640 10 monclient(hunting): handle_auth_done global_id 54336 payload 824 2024-09-11T01:36:14.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.756+0000 7f088affd640 1 -- 172.21.15.12:0/1483388395 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f088c0a3270 msgr2=0x7f088c1cda20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:36:14.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.756+0000 7f088affd640 1 --2- 172.21.15.12:0/1483388395 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f088c0a3270 0x7f088c1cda20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:14.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.756+0000 7f088affd640 1 -- 172.21.15.12:0/1483388395 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f088c138040 msgr2=0x7f088c1248f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:14.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.756+0000 7f088affd640 1 --2- 172.21.15.12:0/1483388395 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f088c138040 0x7f088c1248f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:14.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.756+0000 7f088affd640 10 monclient: _finish_hunting 0 2024-09-11T01:36:14.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.756+0000 7f088affd640 1 monclient: found mon.noname-a 2024-09-11T01:36:14.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.756+0000 7f088affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:14.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.756+0000 7f088affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:14.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.756+0000 7f088affd640 1 -- 172.21.15.12:0/1483388395 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f088c128650 con 0x7f088c0a2e90 2024-09-11T01:36:14.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.756+0000 7f088b7fe640 1 --2- 172.21.15.12:0/1483388395 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f088c138040 0x7f088c1248f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:36:14.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.758+0000 7f088affd640 1 -- 172.21.15.12:0/1483388395 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f088c0a2e90 msgr2=0x7f088c1cd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:14.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.758+0000 7f088affd640 1 --2- 172.21.15.12:0/1483388395 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f088c0a2e90 0x7f088c1cd4e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f087400d800 tx=0x7f087400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:14.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.758+0000 7f0883fff640 1 -- 172.21.15.12:0/1483388395 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087400b870 con 0x7f088c0a2e90 2024-09-11T01:36:14.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.758+0000 7f0883fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:14.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.758+0000 7f0883fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:14.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.758+0000 7f0883fff640 10 monclient: dump: 2024-09-11T01:36:14.770 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:14.770 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:14.770 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:14.770 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:14.770 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:14.770 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:14.770 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:14.770 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:14.770 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:14.771 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:14.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.758+0000 7f0883fff640 10 monclient: _finish_auth 0 2024-09-11T01:36:14.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.758+0000 7f0883fff640 10 monclient: _check_auth_tickets 2024-09-11T01:36:14.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.758+0000 7f0883fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:14.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.758+0000 7f0883fff640 1 -- 172.21.15.12:0/1483388395 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0874019070 con 0x7f088c0a2e90 2024-09-11T01:36:14.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.758+0000 7f08918e2640 5 monclient: authenticate success, global_id 54336 2024-09-11T01:36:14.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.758+0000 7f08918e2640 10 monclient: _renew_subs 2024-09-11T01:36:14.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.758+0000 7f08918e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:14.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.758+0000 7f08918e2640 1 -- 172.21.15.12:0/1483388395 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f088c09f680 con 0x7f088c0a2e90 2024-09-11T01:36:14.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.758+0000 7f0883fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:14.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.758+0000 7f0883fff640 1 -- 172.21.15.12:0/1483388395 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087400bc30 con 0x7f088c0a2e90 2024-09-11T01:36:14.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.758+0000 7f0883fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:14.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.758+0000 7f0883fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:14.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.758+0000 7f0883fff640 10 monclient: dump: 2024-09-11T01:36:14.772 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:14.772 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:14.772 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:14.772 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:14.772 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:14.772 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:14.772 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:14.772 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:14.772 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:14.772 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:14.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.758+0000 7f08897fa640 4 set_mon_vals no callback set 2024-09-11T01:36:14.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.758+0000 7f08918e2640 10 monclient: _renew_subs 2024-09-11T01:36:14.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.758+0000 7f08918e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:14.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.758+0000 7f08918e2640 1 -- 172.21.15.12:0/1483388395 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f088c1262f0 con 0x7f088c0a2e90 2024-09-11T01:36:14.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.759+0000 7f08918e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:14.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.759+0000 7f08918e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:14.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.759+0000 7f08918e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:14.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.759+0000 7f0883fff640 1 -- 172.21.15.12:0/1483388395 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f0874013070 con 0x7f088c0a2e90 2024-09-11T01:36:14.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.759+0000 7f08918e2640 1 -- 172.21.15.12:0/1483388395 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f088c06dc60 con 0x7f088c0a2e90 2024-09-11T01:36:14.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.759+0000 7f0883fff640 1 --2- 172.21.15.12:0/1483388395 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f086c03e210 0x7f086c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:14.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.759+0000 7f0883fff640 1 -- 172.21.15.12:0/1483388395 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(225..225 src has 186..225) ==== 4771+0+0 (secure 0 0 0) 0x7f0874052e80 con 0x7f088c0a2e90 2024-09-11T01:36:14.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.760+0000 7f088a7fc640 1 --2- 172.21.15.12:0/1483388395 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f086c03e210 0x7f086c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:14.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.760+0000 7f088a7fc640 10 monclient: get_auth_request con 0x7f086c03e210 auth_method 0 2024-09-11T01:36:14.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.762+0000 7f088a7fc640 1 --2- 172.21.15.12:0/1483388395 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f086c03e210 0x7f086c0406d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f088c071820 tx=0x7f087c02e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:14.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.764+0000 7f0883fff640 1 -- 172.21.15.12:0/1483388395 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0874051880 con 0x7f088c0a2e90 2024-09-11T01:36:14.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.764+0000 7f0883fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:14.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:14.764+0000 7f0883fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:15.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:15.020+0000 7f08918e2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:36:15.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:15.020+0000 7f08918e2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:36:15.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:15.020+0000 7f08918e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:15.020+0000 7f08918e2640 1 -- 172.21.15.12:0/1483388395 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f088c126e50 con 0x7f088c0a2e90 2024-09-11T01:36:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:15.021+0000 7f0883fff640 1 -- 172.21.15.12:0/1483388395 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f087400bdd0 con 0x7f088c0a2e90 2024-09-11T01:36:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:15.021+0000 7f0883fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:36:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:15.021+0000 7f0883fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:15.023 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":74,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:36:15.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:15.023+0000 7f08918e2640 1 -- 172.21.15.12:0/1483388395 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f086c03e210 msgr2=0x7f086c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:15.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:15.023+0000 7f08918e2640 1 --2- 172.21.15.12:0/1483388395 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f086c03e210 0x7f086c0406d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f088c071820 tx=0x7f087c02e000 comp rx=0 tx=0).stop 2024-09-11T01:36:15.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:15.023+0000 7f08918e2640 10 monclient: shutdown 2024-09-11T01:36:15.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:15.023+0000 7f08918e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:15.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:15.023+0000 7f08918e2640 1 -- 172.21.15.12:0/1483388395 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f088c0a2e90 msgr2=0x7f088c1cd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:15.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:15.023+0000 7f08918e2640 1 --2- 172.21.15.12:0/1483388395 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f088c0a2e90 0x7f088c1cd4e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f087400d800 tx=0x7f087400dcc0 comp rx=0 tx=0).stop 2024-09-11T01:36:15.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:15.023+0000 7f08918e2640 1 -- 172.21.15.12:0/1483388395 shutdown_connections 2024-09-11T01:36:15.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:15.023+0000 7f08918e2640 1 --2- 172.21.15.12:0/1483388395 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f088c138040 0x7f088c1248f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:15.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:15.023+0000 7f08918e2640 1 --2- 172.21.15.12:0/1483388395 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f088c0a3270 0x7f088c1cda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:15.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:15.023+0000 7f08918e2640 1 --2- 172.21.15.12:0/1483388395 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f086c03e210 0x7f086c0406d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:15.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:15.023+0000 7f08918e2640 1 --2- 172.21.15.12:0/1483388395 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f088c0a2e90 0x7f088c1cd4e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:15.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:15.023+0000 7f08918e2640 1 -- 172.21.15.12:0/1483388395 >> 172.21.15.12:0/1483388395 conn(0x7f088c0a8490 msgr2=0x7f088c072090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:15.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:15.023+0000 7f08918e2640 1 -- 172.21.15.12:0/1483388395 shutdown_connections 2024-09-11T01:36:15.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:15.023+0000 7f08918e2640 1 -- 172.21.15.12:0/1483388395 wait complete. 2024-09-11T01:36:18.036 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:36:18.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.148+0000 7f7e36794640 10 monclient: get_monmap_and_config 2024-09-11T01:36:18.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.148+0000 7f7e36794640 10 monclient: build_initial_monmap 2024-09-11T01:36:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.148+0000 7f7e36794640 10 monclient: monmap: 2024-09-11T01:36:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:18.148890+0000 2024-09-11T01:36:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:18.148890+0000 2024-09-11T01:36:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:18.150 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:18.150 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:18.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.148+0000 7f7e36794640 1 Processor -- start 2024-09-11T01:36:18.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.149+0000 7f7e36794640 1 -- start start 2024-09-11T01:36:18.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.149+0000 7f7e36794640 10 monclient: init 2024-09-11T01:36:18.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.149+0000 7f7e36794640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:18.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.149+0000 7f7e36794640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-11T01:36:18.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.149+0000 7f7e36794640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7e30138230 addr v1:172.21.15.12:6789/0 2024-09-11T01:36:18.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.149+0000 7f7e36794640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7e3009d030 0x7f7e3013d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:18.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.149+0000 7f7e36794640 10 monclient(hunting): picked mon.noname-a con 0x7f7e3009d030 addr v2:172.21.15.12:3300/0 2024-09-11T01:36:18.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.149+0000 7f7e36794640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7e3009d460 addr v1:172.21.15.71:6789/0 2024-09-11T01:36:18.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.149+0000 7f7e36794640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e300a6e30 con 0x7f7e30138230 2024-09-11T01:36:18.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.149+0000 7f7e36794640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e3009ba20 con 0x7f7e3009d460 2024-09-11T01:36:18.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.149+0000 7f7e36794640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:18.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.149+0000 7f7e36794640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f7e3009c9c0 con 0x7f7e3009d030 2024-09-11T01:36:18.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.149+0000 7f7e36794640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:18.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.149+0000 7f7e36794640 10 monclient(hunting): authenticate will time out at 1184.591582s 2024-09-11T01:36:18.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.149+0000 7f7e2f7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7e3009d030 0x7f7e3013d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:18.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.149+0000 7f7e2f7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7e3009d030 0x7f7e3013d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56462/0 (socket says 172.21.15.12:56462) 2024-09-11T01:36:18.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.149+0000 7f7e2f7fe640 1 -- 172.21.15.12:0/3768625909 learned_addr learned my addr 172.21.15.12:0/3768625909 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:18.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.149+0000 7f7e2f7fe640 10 monclient(hunting): get_auth_request con 0x7f7e3009d030 auth_method 0 2024-09-11T01:36:18.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.149+0000 7f7e2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:18.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.149+0000 7f7e2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:18.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.149+0000 7f7e2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:18.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.151+0000 7f7e2ffff640 1 -- 172.21.15.12:0/3768625909 >> v1:172.21.15.12:6789/0 conn(0x7f7e30138230 legacy=0x7f7e30138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.151+0000 7f7e2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.151+0000 7f7e2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.151+0000 7f7e2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.152+0000 7f7e2f7fe640 10 monclient(hunting): handle_auth_done global_id 54339 payload 274 2024-09-11T01:36:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.152+0000 7f7e2f7fe640 1 -- 172.21.15.12:0/3768625909 >> v1:172.21.15.71:6789/0 conn(0x7f7e3009d460 legacy=0x7f7e3013f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.152+0000 7f7e2f7fe640 1 -- 172.21.15.12:0/3768625909 >> v1:172.21.15.12:6789/0 conn(0x7f7e30138230 legacy=0x7f7e30138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.152+0000 7f7e2f7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:36:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.152+0000 7f7e2f7fe640 1 monclient: found mon.noname-a 2024-09-11T01:36:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.152+0000 7f7e2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.152+0000 7f7e2f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.152+0000 7f7e2f7fe640 1 -- 172.21.15.12:0/3768625909 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e300a66a0 con 0x7f7e3009d030 2024-09-11T01:36:18.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e2f7fe640 1 -- 172.21.15.12:0/3768625909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e3009d030 msgr2=0x7f7e3013d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e2f7fe640 1 --2- 172.21.15.12:0/3768625909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e3009d030 0x7f7e3013d090 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7e240098f0 tx=0x7f7e2402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23bf56ffbed48c52 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e2effd640 1 -- 172.21.15.12:0/3768625909 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e24037070 con 0x7f7e3009d030 2024-09-11T01:36:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e2effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e2effd640 10 monclient: dump: 2024-09-11T01:36:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:18.159 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:18.159 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:18.159 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:18.159 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:18.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e2effd640 10 monclient: _finish_auth 0 2024-09-11T01:36:18.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e2effd640 10 monclient: _check_auth_tickets 2024-09-11T01:36:18.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:18.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e2effd640 1 -- 172.21.15.12:0/3768625909 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e24032640 con 0x7f7e3009d030 2024-09-11T01:36:18.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e36794640 5 monclient: authenticate success, global_id 54339 2024-09-11T01:36:18.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e36794640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:36:18.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e2effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:18.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e36794640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:18.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e2effd640 1 -- 172.21.15.12:0/3768625909 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e24032a10 con 0x7f7e3009d030 2024-09-11T01:36:18.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:18.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e36794640 4 set_mon_vals no callback set 2024-09-11T01:36:18.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e2effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:18.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e2effd640 10 monclient: dump: 2024-09-11T01:36:18.160 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:18.160 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:18.160 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:18.160 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:18.161 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:18.161 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:18.161 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:18.161 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:18.161 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:18.161 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:18.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e36794640 10 monclient: shutdown 2024-09-11T01:36:18.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e36794640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:18.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e36794640 1 -- 172.21.15.12:0/3768625909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e3009d030 msgr2=0x7f7e3013d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:18.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e36794640 1 --2- 172.21.15.12:0/3768625909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e3009d030 0x7f7e3013d090 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7e240098f0 tx=0x7f7e2402a1c0 comp rx=0 tx=0).stop 2024-09-11T01:36:18.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e36794640 1 -- 172.21.15.12:0/3768625909 shutdown_connections 2024-09-11T01:36:18.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e36794640 1 --2- 172.21.15.12:0/3768625909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e3009d030 0x7f7e3013d090 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:18.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e36794640 1 -- 172.21.15.12:0/3768625909 >> 172.21.15.12:0/3768625909 conn(0x7f7e300a7120 msgr2=0x7f7e300a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:18.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e36794640 1 -- 172.21.15.12:0/3768625909 shutdown_connections 2024-09-11T01:36:18.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.155+0000 7f7e36794640 1 -- 172.21.15.12:0/3768625909 wait complete. 2024-09-11T01:36:18.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.156+0000 7f7e36794640 10 monclient: build_initial_monmap 2024-09-11T01:36:18.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.156+0000 7f7e36794640 10 monclient: monmap: 2024-09-11T01:36:18.162 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:18.162 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:18.162 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:18.162 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:18.162 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:18.162 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:18.162 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:18.163 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:18.163 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:18.163 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:18.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.156+0000 7f7e36794640 1 Processor -- start 2024-09-11T01:36:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e36794640 1 -- start start 2024-09-11T01:36:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e36794640 10 monclient: init 2024-09-11T01:36:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e36794640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e36794640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:36:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e36794640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e3009d030 0x7f7e301cd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e36794640 10 monclient(hunting): picked mon.noname-b con 0x7f7e3009d030 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e36794640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e3009d460 0x7f7e30136e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e36794640 10 monclient(hunting): picked mon.noname-a con 0x7f7e3009d460 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e36794640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e30138230 0x7f7e3013cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e36794640 10 monclient(hunting): picked mon.noname-c con 0x7f7e30138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e36794640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e36794640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7e30140e80 con 0x7f7e3009d460 2024-09-11T01:36:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e36794640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e36794640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f7e30140d00 con 0x7f7e3009d030 2024-09-11T01:36:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e36794640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e36794640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7e30141000 con 0x7f7e30138230 2024-09-11T01:36:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e36794640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e36794640 10 monclient(hunting): authenticate will time out at 1184.599491s 2024-09-11T01:36:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e34d31640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e30138230 msgr2=0x7f7e3013cc00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:36:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e34d31640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e30138230 0x7f7e3013cc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e3009d460 0x7f7e30136e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e3009d460 0x7f7e30136e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56476/0 (socket says 172.21.15.12:56476) 2024-09-11T01:36:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2f7fe640 1 -- 172.21.15.12:0/1340073744 learned_addr learned my addr 172.21.15.12:0/1340073744 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2f7fe640 10 monclient(hunting): get_auth_request con 0x7f7e3009d460 auth_method 0 2024-09-11T01:36:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2ffff640 1 --2- 172.21.15.12:0/1340073744 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e3009d030 0x7f7e301cd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2ffff640 10 monclient(hunting): get_auth_request con 0x7f7e3009d030 auth_method 0 2024-09-11T01:36:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2f7fe640 10 monclient(hunting): handle_auth_done global_id 54342 payload 824 2024-09-11T01:36:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2f7fe640 1 -- 172.21.15.12:0/1340073744 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e30138230 msgr2=0x7f7e3013cc00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:36:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2f7fe640 1 --2- 172.21.15.12:0/1340073744 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e30138230 0x7f7e3013cc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2f7fe640 1 -- 172.21.15.12:0/1340073744 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e3009d030 msgr2=0x7f7e301cd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2f7fe640 1 --2- 172.21.15.12:0/1340073744 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e3009d030 0x7f7e301cd0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2f7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:36:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2f7fe640 1 monclient: found mon.noname-a 2024-09-11T01:36:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.157+0000 7f7e2f7fe640 1 -- 172.21.15.12:0/1340073744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e30140650 con 0x7f7e3009d460 2024-09-11T01:36:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.160+0000 7f7e2f7fe640 1 -- 172.21.15.12:0/1340073744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e3009d460 msgr2=0x7f7e30136e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.160+0000 7f7e2f7fe640 1 --2- 172.21.15.12:0/1340073744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e3009d460 0x7f7e30136e80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7e240098f0 tx=0x7f7e240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.160+0000 7f7e2cff9640 1 -- 172.21.15.12:0/1340073744 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e24042070 con 0x7f7e3009d460 2024-09-11T01:36:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.160+0000 7f7e2cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.160+0000 7f7e2cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.160+0000 7f7e2cff9640 10 monclient: dump: 2024-09-11T01:36:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:18.173 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.160+0000 7f7e2cff9640 10 monclient: _finish_auth 0 2024-09-11T01:36:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.160+0000 7f7e2cff9640 10 monclient: _check_auth_tickets 2024-09-11T01:36:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.160+0000 7f7e2cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.160+0000 7f7e2cff9640 1 -- 172.21.15.12:0/1340073744 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e2403e880 con 0x7f7e3009d460 2024-09-11T01:36:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.160+0000 7f7e2cff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.160+0000 7f7e2cff9640 1 -- 172.21.15.12:0/1340073744 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e24037040 con 0x7f7e3009d460 2024-09-11T01:36:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.160+0000 7f7e36794640 5 monclient: authenticate success, global_id 54342 2024-09-11T01:36:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.160+0000 7f7e36794640 10 monclient: _renew_subs 2024-09-11T01:36:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.160+0000 7f7e36794640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.160+0000 7f7e36794640 1 -- 172.21.15.12:0/1340073744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e30140930 con 0x7f7e3009d460 2024-09-11T01:36:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.160+0000 7f7e2cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.160+0000 7f7e2cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.160+0000 7f7e2cff9640 10 monclient: dump: 2024-09-11T01:36:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:18.175 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.160+0000 7f7e36794640 10 monclient: _renew_subs 2024-09-11T01:36:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.160+0000 7f7e36794640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.160+0000 7f7e36794640 1 -- 172.21.15.12:0/1340073744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e300a1810 con 0x7f7e3009d460 2024-09-11T01:36:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.160+0000 7f7e2e7fc640 4 set_mon_vals no callback set 2024-09-11T01:36:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.161+0000 7f7e36794640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.161+0000 7f7e36794640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.161+0000 7f7e36794640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.161+0000 7f7e36794640 1 -- 172.21.15.12:0/1340073744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e3009bc50 con 0x7f7e3009d460 2024-09-11T01:36:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.161+0000 7f7e2cff9640 1 -- 172.21.15.12:0/1340073744 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f7e2403ec50 con 0x7f7e3009d460 2024-09-11T01:36:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.162+0000 7f7e2cff9640 1 --2- 172.21.15.12:0/1340073744 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7e1c03e1c0 0x7f7e1c040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.162+0000 7f7e2cff9640 1 -- 172.21.15.12:0/1340073744 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(225..225 src has 186..225) ==== 4771+0+0 (secure 0 0 0) 0x7f7e24073520 con 0x7f7e3009d460 2024-09-11T01:36:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.162+0000 7f7e2ffff640 1 --2- 172.21.15.12:0/1340073744 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7e1c03e1c0 0x7f7e1c040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.162+0000 7f7e2ffff640 10 monclient: get_auth_request con 0x7f7e1c03e1c0 auth_method 0 2024-09-11T01:36:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.165+0000 7f7e2ffff640 1 --2- 172.21.15.12:0/1340073744 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7e1c03e1c0 0x7f7e1c040680 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7e20000f60 tx=0x7f7e20009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.166+0000 7f7e2cff9640 1 -- 172.21.15.12:0/1340073744 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7e3009bc50 con 0x7f7e3009d460 2024-09-11T01:36:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.166+0000 7f7e2cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.166+0000 7f7e2cff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:18.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.420+0000 7f7e36794640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:36:18.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.421+0000 7f7e36794640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:36:18.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.421+0000 7f7e36794640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:18.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.421+0000 7f7e36794640 1 -- 172.21.15.12:0/1340073744 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7e30138610 con 0x7f7e3009d460 2024-09-11T01:36:18.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.421+0000 7f7e2cff9640 1 -- 172.21.15.12:0/1340073744 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f7e2404c070 con 0x7f7e3009d460 2024-09-11T01:36:18.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.421+0000 7f7e2cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:36:18.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.421+0000 7f7e2cff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:18.426 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":74,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:36:18.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.423+0000 7f7e36794640 1 -- 172.21.15.12:0/1340073744 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7e1c03e1c0 msgr2=0x7f7e1c040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:18.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.423+0000 7f7e36794640 1 --2- 172.21.15.12:0/1340073744 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7e1c03e1c0 0x7f7e1c040680 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7e20000f60 tx=0x7f7e20009040 comp rx=0 tx=0).stop 2024-09-11T01:36:18.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.423+0000 7f7e36794640 10 monclient: shutdown 2024-09-11T01:36:18.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.423+0000 7f7e36794640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:18.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.423+0000 7f7e36794640 1 -- 172.21.15.12:0/1340073744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e3009d460 msgr2=0x7f7e30136e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:18.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.423+0000 7f7e36794640 1 --2- 172.21.15.12:0/1340073744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e3009d460 0x7f7e30136e80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7e240098f0 tx=0x7f7e240046c0 comp rx=0 tx=0).stop 2024-09-11T01:36:18.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.423+0000 7f7e36794640 1 -- 172.21.15.12:0/1340073744 shutdown_connections 2024-09-11T01:36:18.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.423+0000 7f7e36794640 1 --2- 172.21.15.12:0/1340073744 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e30138230 0x7f7e3013cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:18.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.423+0000 7f7e36794640 1 --2- 172.21.15.12:0/1340073744 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7e1c03e1c0 0x7f7e1c040680 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:18.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.423+0000 7f7e36794640 1 --2- 172.21.15.12:0/1340073744 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e3009d460 0x7f7e30136e80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:18.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.423+0000 7f7e36794640 1 --2- 172.21.15.12:0/1340073744 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e3009d030 0x7f7e301cd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:18.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.423+0000 7f7e36794640 1 -- 172.21.15.12:0/1340073744 >> 172.21.15.12:0/1340073744 conn(0x7f7e300a7120 msgr2=0x7f7e300a0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:18.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.423+0000 7f7e36794640 1 -- 172.21.15.12:0/1340073744 shutdown_connections 2024-09-11T01:36:18.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:18.423+0000 7f7e36794640 1 -- 172.21.15.12:0/1340073744 wait complete. 2024-09-11T01:36:21.437 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:36:21.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.548+0000 7f08337cd640 10 monclient: get_monmap_and_config 2024-09-11T01:36:21.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.548+0000 7f08337cd640 10 monclient: build_initial_monmap 2024-09-11T01:36:21.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.548+0000 7f08337cd640 10 monclient: monmap: 2024-09-11T01:36:21.550 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:21.550 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:21.550 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:21.548893+0000 2024-09-11T01:36:21.550 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:21.548893+0000 2024-09-11T01:36:21.550 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:21.550 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:21.550 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:21.550 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:21.550 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:21.550 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:21.550 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:21.550 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:21.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.548+0000 7f08337cd640 1 Processor -- start 2024-09-11T01:36:21.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.549+0000 7f08337cd640 1 -- start start 2024-09-11T01:36:21.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.549+0000 7f08337cd640 10 monclient: init 2024-09-11T01:36:21.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.549+0000 7f08337cd640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:21.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.549+0000 7f08337cd640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-11T01:36:21.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.549+0000 7f08337cd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f082c138230 0x7f082c138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:21.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.549+0000 7f08337cd640 10 monclient(hunting): picked mon.noname-c con 0x7f082c138230 addr v2:172.21.15.71:3300/0 2024-09-11T01:36:21.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.549+0000 7f08337cd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f082c09d030 0x7f082c0a5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:21.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.549+0000 7f08337cd640 10 monclient(hunting): picked mon.noname-b con 0x7f082c09d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:21.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.549+0000 7f08337cd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f082c09d460 addr v1:172.21.15.12:6789/0 2024-09-11T01:36:21.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.549+0000 7f08337cd640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f082c0a6e30 con 0x7f082c09d460 2024-09-11T01:36:21.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.549+0000 7f08337cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:21.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.549+0000 7f08337cd640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f082c09c9c0 con 0x7f082c138230 2024-09-11T01:36:21.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.549+0000 7f08337cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:21.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.549+0000 7f08337cd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f082c09ba20 con 0x7f082c09d030 2024-09-11T01:36:21.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.549+0000 7f08337cd640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:21.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.549+0000 7f08337cd640 10 monclient(hunting): authenticate will time out at 1187.991605s 2024-09-11T01:36:21.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.549+0000 7f0830d68640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f082c09d030 msgr2=0x7f082c0a5020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:36:21.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.549+0000 7f0830d68640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f082c09d030 0x7f082c0a5020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:21.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.549+0000 7f0831d6a640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f082c09d460 0x7f082c0a5560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53924/0 (socket says 172.21.15.12:53924) 2024-09-11T01:36:21.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.549+0000 7f0831d6a640 1 -- 172.21.15.12:0/1366389665 learned_addr learned my addr 172.21.15.12:0/1366389665 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:36:21.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.549+0000 7f0831569640 1 --2- 172.21.15.12:0/1366389665 >> v2:172.21.15.71:3300/0 conn(0x7f082c138230 0x7f082c138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:21.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.549+0000 7f0831569640 10 monclient(hunting): get_auth_request con 0x7f082c138230 auth_method 0 2024-09-11T01:36:21.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.549+0000 7f0831569640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:21.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.549+0000 7f0831569640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:21.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.549+0000 7f0831569640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:21.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.550+0000 7f0831569640 1 -- 172.21.15.12:0/1366389665 >> v2:172.21.15.71:3300/0 conn(0x7f082c138230 msgr2=0x7f082c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:36:21.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.550+0000 7f0831569640 1 -- 172.21.15.12:0/1366389665 >> v2:172.21.15.71:3300/0 conn(0x7f082c138230 msgr2=0x7f082c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:36:21.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.550+0000 7f0831569640 1 --2- 172.21.15.12:0/1366389665 >> v2:172.21.15.71:3300/0 conn(0x7f082c138230 0x7f082c138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:21.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.550+0000 7f0831569640 1 --2- 172.21.15.12:0/1366389665 >> v2:172.21.15.71:3300/0 conn(0x7f082c138230 0x7f082c138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:21.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.551+0000 7f0831d6a640 1 -- 172.21.15.12:0/1366389665 >> v1:172.21.15.12:6789/0 conn(0x7f082c09d460 legacy=0x7f082c0a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:21.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.751+0000 7f0830d68640 1 -- 172.21.15.12:0/1366389665 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f082c09d030 msgr2=0x7f082c0a5020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:36:21.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.751+0000 7f0830d68640 1 --2- 172.21.15.12:0/1366389665 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f082c09d030 0x7f082c0a5020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:36:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.751+0000 7f0831569640 1 --2- 172.21.15.12:0/1366389665 >> v2:172.21.15.71:3300/0 conn(0x7f082c138230 0x7f082c138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.751+0000 7f0831569640 10 monclient(hunting): get_auth_request con 0x7f082c138230 auth_method 0 2024-09-11T01:36:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.751+0000 7f0831569640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.751+0000 7f0831569640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:21.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.751+0000 7f0831569640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:36:21.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.751+0000 7f0831569640 1 -- 172.21.15.12:0/1366389665 >> v2:172.21.15.71:3300/0 conn(0x7f082c138230 msgr2=0x7f082c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:36:21.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.751+0000 7f0831569640 1 -- 172.21.15.12:0/1366389665 >> v2:172.21.15.71:3300/0 conn(0x7f082c138230 msgr2=0x7f082c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:36:21.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.751+0000 7f0831569640 1 --2- 172.21.15.12:0/1366389665 >> v2:172.21.15.71:3300/0 conn(0x7f082c138230 0x7f082c138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:21.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:21.752+0000 7f0831569640 1 --2- 172.21.15.12:0/1366389665 >> v2:172.21.15.71:3300/0 conn(0x7f082c138230 0x7f082c138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:36:22.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:22.152+0000 7f0830d68640 1 -- 172.21.15.12:0/1366389665 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f082c09d030 msgr2=0x7f082c0a5020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:36:22.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:22.152+0000 7f0830d68640 1 --2- 172.21.15.12:0/1366389665 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f082c09d030 0x7f082c0a5020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:36:22.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:22.154+0000 7f0831569640 1 --2- 172.21.15.12:0/1366389665 >> v2:172.21.15.71:3300/0 conn(0x7f082c138230 0x7f082c138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:22.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:22.154+0000 7f0831569640 10 monclient(hunting): get_auth_request con 0x7f082c138230 auth_method 0 2024-09-11T01:36:22.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:22.154+0000 7f0831569640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:22.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:22.154+0000 7f0831569640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:22.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:22.154+0000 7f0831569640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:36:22.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:22.154+0000 7f0831569640 1 -- 172.21.15.12:0/1366389665 >> v2:172.21.15.71:3300/0 conn(0x7f082c138230 msgr2=0x7f082c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:36:22.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:22.154+0000 7f0831569640 1 -- 172.21.15.12:0/1366389665 >> v2:172.21.15.71:3300/0 conn(0x7f082c138230 msgr2=0x7f082c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:36:22.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:22.154+0000 7f0831569640 1 --2- 172.21.15.12:0/1366389665 >> v2:172.21.15.71:3300/0 conn(0x7f082c138230 0x7f082c138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:22.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:22.154+0000 7f0831569640 1 --2- 172.21.15.12:0/1366389665 >> v2:172.21.15.71:3300/0 conn(0x7f082c138230 0x7f082c138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:36:22.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:22.953+0000 7f0830d68640 1 -- 172.21.15.12:0/1366389665 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f082c09d030 msgr2=0x7f082c0a5020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:36:22.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:22.953+0000 7f0830d68640 1 --2- 172.21.15.12:0/1366389665 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f082c09d030 0x7f082c0a5020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:36:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:22.955+0000 7f0831569640 1 --2- 172.21.15.12:0/1366389665 >> v2:172.21.15.71:3300/0 conn(0x7f082c138230 0x7f082c138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:22.955+0000 7f0831569640 10 monclient(hunting): get_auth_request con 0x7f082c138230 auth_method 0 2024-09-11T01:36:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:22.955+0000 7f0831569640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:22.955+0000 7f0831569640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:22.955+0000 7f0831569640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:36:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:22.955+0000 7f0831569640 1 -- 172.21.15.12:0/1366389665 >> v2:172.21.15.71:3300/0 conn(0x7f082c138230 msgr2=0x7f082c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:36:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:22.955+0000 7f0831569640 1 -- 172.21.15.12:0/1366389665 >> v2:172.21.15.71:3300/0 conn(0x7f082c138230 msgr2=0x7f082c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:36:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:22.955+0000 7f0831569640 1 --2- 172.21.15.12:0/1366389665 >> v2:172.21.15.71:3300/0 conn(0x7f082c138230 0x7f082c138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:22.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:22.955+0000 7f0831569640 1 --2- 172.21.15.12:0/1366389665 >> v2:172.21.15.71:3300/0 conn(0x7f082c138230 0x7f082c138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:36:24.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.549+0000 7f0822ffd640 10 monclient(hunting): tick 2024-09-11T01:36:24.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.549+0000 7f0822ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:36:24.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.549+0000 7f0822ffd640 1 monclient(hunting): continuing hunt 2024-09-11T01:36:24.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.549+0000 7f0822ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:36:24.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.549+0000 7f0822ffd640 1 -- 172.21.15.12:0/1366389665 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f082c09d030 msgr2=0x7f082c0a5020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:36:24.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.549+0000 7f0822ffd640 1 --2- 172.21.15.12:0/1366389665 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f082c09d030 0x7f082c0a5020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:24.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.549+0000 7f0822ffd640 1 -- 172.21.15.12:0/1366389665 >> v2:172.21.15.71:3300/0 conn(0x7f082c138230 msgr2=0x7f082c138610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:36:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.549+0000 7f0822ffd640 1 --2- 172.21.15.12:0/1366389665 >> v2:172.21.15.71:3300/0 conn(0x7f082c138230 0x7f082c138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.549+0000 7f0822ffd640 1 -- 172.21.15.12:0/1366389665 >> v1:172.21.15.12:6789/0 conn(0x7f082c09d460 legacy=0x7f082c0a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.549+0000 7f0822ffd640 10 monclient: _add_conns ranks=[3,1] 2024-09-11T01:36:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.549+0000 7f0822ffd640 1 --2- 172.21.15.12:0/1366389665 >> v2:172.21.15.12:3300/0 conn(0x7f0810003760 0x7f0810005be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.549+0000 7f0822ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f0810003760 addr v2:172.21.15.12:3300/0 2024-09-11T01:36:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.549+0000 7f0822ffd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f08100062b0 addr v1:172.21.15.71:6789/0 2024-09-11T01:36:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.549+0000 7f0822ffd640 1 -- 172.21.15.12:0/1366389665 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f082c0a66a0 con 0x7f08100062b0 2024-09-11T01:36:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.549+0000 7f0822ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.549+0000 7f0822ffd640 1 -- 172.21.15.12:0/1366389665 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f082c09c9c0 con 0x7f0810003760 2024-09-11T01:36:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.549+0000 7f0822ffd640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.550+0000 7f0831569640 1 --2- 172.21.15.12:0/1366389665 >> v2:172.21.15.12:3300/0 conn(0x7f0810003760 0x7f0810005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.550+0000 7f0831569640 10 monclient(hunting): get_auth_request con 0x7f0810003760 auth_method 0 2024-09-11T01:36:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.550+0000 7f0831569640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:24.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.550+0000 7f0831569640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:24.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.550+0000 7f0831569640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:24.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.550+0000 7f0831569640 1 -- 172.21.15.12:0/1366389665 >> v2:172.21.15.12:3300/0 conn(0x7f0810003760 msgr2=0x7f0810005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:36:24.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.550+0000 7f0831569640 1 -- 172.21.15.12:0/1366389665 >> v2:172.21.15.12:3300/0 conn(0x7f0810003760 msgr2=0x7f0810005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:36:24.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.550+0000 7f0831569640 1 --2- 172.21.15.12:0/1366389665 >> v2:172.21.15.12:3300/0 conn(0x7f0810003760 0x7f0810005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:24.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.550+0000 7f0831569640 1 --2- 172.21.15.12:0/1366389665 >> v2:172.21.15.12:3300/0 conn(0x7f0810003760 0x7f0810005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:24.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.552+0000 7f0830d68640 1 -- 172.21.15.12:0/1366389665 >> v1:172.21.15.71:6789/0 conn(0x7f08100062b0 legacy=0x7f0810008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.751+0000 7f0831569640 1 --2- 172.21.15.12:0/1366389665 >> v2:172.21.15.12:3300/0 conn(0x7f0810003760 0x7f0810005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.752+0000 7f0831569640 10 monclient(hunting): get_auth_request con 0x7f0810003760 auth_method 0 2024-09-11T01:36:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.752+0000 7f0831569640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.752+0000 7f0831569640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.752+0000 7f0831569640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:36:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.752+0000 7f0831569640 1 -- 172.21.15.12:0/1366389665 >> v2:172.21.15.12:3300/0 conn(0x7f0810003760 msgr2=0x7f0810005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:36:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.752+0000 7f0831569640 1 -- 172.21.15.12:0/1366389665 >> v2:172.21.15.12:3300/0 conn(0x7f0810003760 msgr2=0x7f0810005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:36:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.752+0000 7f0831569640 1 --2- 172.21.15.12:0/1366389665 >> v2:172.21.15.12:3300/0 conn(0x7f0810003760 0x7f0810005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:24.752+0000 7f0831569640 1 --2- 172.21.15.12:0/1366389665 >> v2:172.21.15.12:3300/0 conn(0x7f0810003760 0x7f0810005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:36:25.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.154+0000 7f0831569640 1 --2- 172.21.15.12:0/1366389665 >> v2:172.21.15.12:3300/0 conn(0x7f0810003760 0x7f0810005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:25.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.154+0000 7f0831569640 10 monclient(hunting): get_auth_request con 0x7f0810003760 auth_method 0 2024-09-11T01:36:25.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.154+0000 7f0831569640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:25.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.154+0000 7f0831569640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:25.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.154+0000 7f0831569640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:36:25.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.154+0000 7f0831569640 1 -- 172.21.15.12:0/1366389665 >> v2:172.21.15.12:3300/0 conn(0x7f0810003760 msgr2=0x7f0810005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:36:25.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.154+0000 7f0831569640 1 -- 172.21.15.12:0/1366389665 >> v2:172.21.15.12:3300/0 conn(0x7f0810003760 msgr2=0x7f0810005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:36:25.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.154+0000 7f0831569640 1 --2- 172.21.15.12:0/1366389665 >> v2:172.21.15.12:3300/0 conn(0x7f0810003760 0x7f0810005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:25.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.154+0000 7f0831569640 1 --2- 172.21.15.12:0/1366389665 >> v2:172.21.15.12:3300/0 conn(0x7f0810003760 0x7f0810005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:36:25.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.250+0000 7f0823fff640 1 -- 172.21.15.12:0/1366389665 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4020482375 0 0) 0x7f082c0a66a0 con 0x7f08100062b0 2024-09-11T01:36:25.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.250+0000 7f0823fff640 10 monclient(hunting): _negotiate 2024-09-11T01:36:25.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.250+0000 7f0823fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:25.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.250+0000 7f0823fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:25.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.250+0000 7f0823fff640 10 monclient(hunting): my global_id is 74152 2024-09-11T01:36:25.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.250+0000 7f0823fff640 1 -- 172.21.15.12:0/1366389665 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0818003520 con 0x7f08100062b0 2024-09-11T01:36:25.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 1 -- 172.21.15.12:0/1366389665 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3171580168 0 0) 0x7f0818003520 con 0x7f08100062b0 2024-09-11T01:36:25.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 1 -- 172.21.15.12:0/1366389665 >> v2:172.21.15.12:3300/0 conn(0x7f0810003760 msgr2=0x7f0810005be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:36:25.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 1 --2- 172.21.15.12:0/1366389665 >> v2:172.21.15.12:3300/0 conn(0x7f0810003760 0x7f0810005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 10 monclient: _finish_hunting 0 2024-09-11T01:36:25.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 1 monclient: found mon.noname-c-legacy 2024-09-11T01:36:25.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:25.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-09-11T01:36:25.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 1 -- 172.21.15.12:0/1366389665 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f081000b080 con 0x7f08100062b0 2024-09-11T01:36:25.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 10 monclient: _finish_auth 0 2024-09-11T01:36:25.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 10 monclient: _check_auth_tickets 2024-09-11T01:36:25.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:25.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 1 -- 172.21.15.12:0/1366389665 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f081c004600 con 0x7f08100062b0 2024-09-11T01:36:25.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:25.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-11T01:36:25.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 10 monclient: dump: 2024-09-11T01:36:25.268 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:25.268 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:25.268 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.268 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.268 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:25.268 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:25.268 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:25.268 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:25.268 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:25.269 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:25.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-09-11T01:36:25.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:25.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 1 -- 172.21.15.12:0/1366389665 >> v1:172.21.15.71:6789/0 conn(0x7f08100062b0 legacy=0x7f0810008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:25.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:36:25.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0831d6a640 1 -- 172.21.15.12:0/1366389665 reap_dead start 2024-09-11T01:36:25.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 1 --2- 172.21.15.12:0/1366389665 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f08180039b0 0x7f0818003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:25.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 10 monclient(hunting): picked mon.b con 0x7f08180039b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:25.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 1 --2- 172.21.15.12:0/1366389665 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0818004800 0x7f0818004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:25.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 10 monclient(hunting): picked mon.c con 0x7f0818004800 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:25.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 1 --2- 172.21.15.12:0/1366389665 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0818005290 0x7f081800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:25.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 10 monclient(hunting): picked mon.a con 0x7f0818005290 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:25.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:25.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 1 -- 172.21.15.12:0/1366389665 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f082c09c9c0 con 0x7f0818005290 2024-09-11T01:36:25.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:25.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 1 -- 172.21.15.12:0/1366389665 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f081800c1e0 con 0x7f08180039b0 2024-09-11T01:36:25.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:25.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 1 -- 172.21.15.12:0/1366389665 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f081800c3e0 con 0x7f0818004800 2024-09-11T01:36:25.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:25.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0823fff640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:36:25.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.263+0000 7f0830d68640 1 -- 172.21.15.12:0/1366389665 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0818004800 msgr2=0x7f0818004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:36:25.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0830d68640 1 --2- 172.21.15.12:0/1366389665 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0818004800 0x7f0818004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:25.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831d6a640 1 --2- 172.21.15.12:0/1366389665 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0818005290 0x7f081800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:25.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831569640 1 --2- 172.21.15.12:0/1366389665 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f08180039b0 0x7f0818003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:25.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831569640 10 monclient(hunting): get_auth_request con 0x7f08180039b0 auth_method 0 2024-09-11T01:36:25.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831569640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:25.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831569640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:25.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831569640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:36:25.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831d6a640 10 monclient(hunting): get_auth_request con 0x7f0818005290 auth_method 0 2024-09-11T01:36:25.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831d6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:25.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831d6a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:25.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831d6a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:36:25.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831569640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:25.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831569640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:25.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831569640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:36:25.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831d6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:25.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831d6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:25.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831d6a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:36:25.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831569640 10 monclient(hunting): handle_auth_done global_id 74152 payload 293 2024-09-11T01:36:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831569640 1 -- 172.21.15.12:0/1366389665 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0818004800 msgr2=0x7f0818004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:36:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831569640 1 --2- 172.21.15.12:0/1366389665 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0818004800 0x7f0818004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831569640 1 -- 172.21.15.12:0/1366389665 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0818005290 msgr2=0x7f081800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831569640 1 --2- 172.21.15.12:0/1366389665 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0818005290 0x7f081800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831569640 10 monclient: _finish_hunting 0 2024-09-11T01:36:25.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831569640 1 monclient: found mon.b 2024-09-11T01:36:25.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831569640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:25.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831569640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:36:25.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831569640 1 -- 172.21.15.12:0/1366389665 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0818003520 con 0x7f08180039b0 2024-09-11T01:36:25.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831569640 10 monclient: _finish_auth 0 2024-09-11T01:36:25.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831569640 10 monclient: _check_auth_tickets 2024-09-11T01:36:25.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831569640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:25.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f0831d6a640 1 --2- 172.21.15.12:0/1366389665 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0818005290 0x7f081800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:36:25.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f08337cd640 5 monclient: authenticate success, global_id 74152 2024-09-11T01:36:25.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.264+0000 7f08337cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:36:25.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.267+0000 7f0831569640 1 -- 172.21.15.12:0/1366389665 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f08180039b0 msgr2=0x7f0818003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:25.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.267+0000 7f0831569640 1 --2- 172.21.15.12:0/1366389665 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f08180039b0 0x7f0818003db0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0828017cf0 tx=0x7f0818005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0cefca2671e9ef7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:25.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.268+0000 7f0823fff640 1 -- 172.21.15.12:0/1366389665 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0828032070 con 0x7f08180039b0 2024-09-11T01:36:25.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.268+0000 7f0823fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:25.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.268+0000 7f0823fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:36:25.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.268+0000 7f0823fff640 10 monclient: dump: 2024-09-11T01:36:25.283 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:25.284 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:25.284 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.284 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.284 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:25.284 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:25.284 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:25.284 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:25.284 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:25.284 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:25.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.268+0000 7f0823fff640 1 -- 172.21.15.12:0/1366389665 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f082802c930 con 0x7f08180039b0 2024-09-11T01:36:25.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.268+0000 7f0823fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:25.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.268+0000 7f0823fff640 1 -- 172.21.15.12:0/1366389665 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08280178f0 con 0x7f08180039b0 2024-09-11T01:36:25.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.268+0000 7f08337cd640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:25.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.268+0000 7f08337cd640 4 set_mon_vals no callback set 2024-09-11T01:36:25.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.268+0000 7f0823fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:25.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.268+0000 7f0823fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:36:25.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.268+0000 7f0823fff640 10 monclient: dump: 2024-09-11T01:36:25.285 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:25.285 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:25.285 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.285 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.285 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:25.285 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:25.285 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:25.285 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:25.286 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:25.286 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:25.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.268+0000 7f08337cd640 10 monclient: shutdown 2024-09-11T01:36:25.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.268+0000 7f08337cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:25.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.268+0000 7f08337cd640 1 -- 172.21.15.12:0/1366389665 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f08180039b0 msgr2=0x7f0818003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:25.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.268+0000 7f08337cd640 1 --2- 172.21.15.12:0/1366389665 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f08180039b0 0x7f0818003db0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0828017cf0 tx=0x7f0818005830 comp rx=0 tx=0).stop 2024-09-11T01:36:25.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.269+0000 7f08337cd640 1 -- 172.21.15.12:0/1366389665 shutdown_connections 2024-09-11T01:36:25.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.269+0000 7f08337cd640 1 --2- 172.21.15.12:0/1366389665 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0818005290 0x7f081800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.269+0000 7f08337cd640 1 --2- 172.21.15.12:0/1366389665 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0818004800 0x7f0818004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.269+0000 7f08337cd640 1 --2- 172.21.15.12:0/1366389665 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f08180039b0 0x7f0818003db0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.269+0000 7f08337cd640 1 -- 172.21.15.12:0/1366389665 >> 172.21.15.12:0/1366389665 conn(0x7f082c0a7120 msgr2=0x7f082c0a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:25.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.269+0000 7f08337cd640 1 -- 172.21.15.12:0/1366389665 shutdown_connections 2024-09-11T01:36:25.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.269+0000 7f08337cd640 1 -- 172.21.15.12:0/1366389665 wait complete. 2024-09-11T01:36:25.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.269+0000 7f08337cd640 10 monclient: build_initial_monmap 2024-09-11T01:36:25.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.269+0000 7f08337cd640 10 monclient: monmap: 2024-09-11T01:36:25.287 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:25.287 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:25.287 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:25.287 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:25.287 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:25.287 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:25.287 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:25.287 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:25.287 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:25.287 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.270+0000 7f08337cd640 1 Processor -- start 2024-09-11T01:36:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.270+0000 7f08337cd640 1 -- start start 2024-09-11T01:36:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.270+0000 7f08337cd640 10 monclient: init 2024-09-11T01:36:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.270+0000 7f08337cd640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.270+0000 7f08337cd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:36:25.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.270+0000 7f08337cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08180039b0 0x7f082c1ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:25.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.270+0000 7f08337cd640 10 monclient(hunting): picked mon.noname-a con 0x7f08180039b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:25.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.270+0000 7f08337cd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0818004800 0x7f082c1cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:25.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.270+0000 7f08337cd640 10 monclient(hunting): picked mon.noname-c con 0x7f0818004800 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:25.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.270+0000 7f08337cd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0818005290 0x7f082c124870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:25.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.270+0000 7f08337cd640 10 monclient(hunting): picked mon.noname-b con 0x7f0818005290 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:25.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.270+0000 7f08337cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:25.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.270+0000 7f08337cd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f082c138790 con 0x7f08180039b0 2024-09-11T01:36:25.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.270+0000 7f08337cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:25.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.270+0000 7f08337cd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f082c138610 con 0x7f0818005290 2024-09-11T01:36:25.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.270+0000 7f08337cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:25.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.270+0000 7f08337cd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f082c138910 con 0x7f0818004800 2024-09-11T01:36:25.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.270+0000 7f08337cd640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:25.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.270+0000 7f08337cd640 10 monclient(hunting): authenticate will time out at 1191.712990s 2024-09-11T01:36:25.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0830d68640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0818004800 msgr2=0x7f082c1cd2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:36:25.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0830d68640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0818004800 0x7f082c1cd2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:25.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0831569640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08180039b0 0x7f082c1ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:25.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0831569640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08180039b0 0x7f082c1ccdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56526/0 (socket says 172.21.15.12:56526) 2024-09-11T01:36:25.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0831569640 1 -- 172.21.15.12:0/2670222687 learned_addr learned my addr 172.21.15.12:0/2670222687 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:25.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0831569640 10 monclient(hunting): get_auth_request con 0x7f08180039b0 auth_method 0 2024-09-11T01:36:25.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0831569640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:25.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0831569640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:25.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0831569640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:25.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0831569640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:25.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0831569640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:25.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0831569640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:25.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0831d6a640 1 --2- 172.21.15.12:0/2670222687 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0818005290 0x7f082c124870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:25.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0831d6a640 10 monclient(hunting): get_auth_request con 0x7f0818005290 auth_method 0 2024-09-11T01:36:25.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0831d6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:25.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0831d6a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:25.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0831d6a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:25.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0831569640 10 monclient(hunting): handle_auth_done global_id 54345 payload 824 2024-09-11T01:36:25.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0831569640 1 -- 172.21.15.12:0/2670222687 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0818004800 msgr2=0x7f082c1cd2f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:36:25.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0831569640 1 --2- 172.21.15.12:0/2670222687 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0818004800 0x7f082c1cd2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0831569640 1 -- 172.21.15.12:0/2670222687 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0818005290 msgr2=0x7f082c124870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:25.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0831569640 1 --2- 172.21.15.12:0/2670222687 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0818005290 0x7f082c124870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0831569640 10 monclient: _finish_hunting 0 2024-09-11T01:36:25.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0831569640 1 monclient: found mon.noname-a 2024-09-11T01:36:25.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0831569640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:25.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0831569640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:25.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.271+0000 7f0831569640 1 -- 172.21.15.12:0/2670222687 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f082c126270 con 0x7f08180039b0 2024-09-11T01:36:25.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.273+0000 7f0831569640 1 -- 172.21.15.12:0/2670222687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08180039b0 msgr2=0x7f082c1ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:25.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.273+0000 7f0831569640 1 --2- 172.21.15.12:0/2670222687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08180039b0 0x7f082c1ccdb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0828000c00 tx=0x7f0828004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:25.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.274+0000 7f0821ffb640 1 -- 172.21.15.12:0/2670222687 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08280178f0 con 0x7f08180039b0 2024-09-11T01:36:25.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.274+0000 7f0821ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:25.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.274+0000 7f0821ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:25.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.274+0000 7f0821ffb640 10 monclient: dump: 2024-09-11T01:36:25.292 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:25.292 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:25.292 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.293 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.293 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:25.293 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:25.293 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:25.293 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:25.293 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:25.293 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:25.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.274+0000 7f0821ffb640 10 monclient: _finish_auth 0 2024-09-11T01:36:25.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.274+0000 7f0821ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:36:25.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.274+0000 7f0821ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:25.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.274+0000 7f0821ffb640 1 -- 172.21.15.12:0/2670222687 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0828046070 con 0x7f08180039b0 2024-09-11T01:36:25.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.274+0000 7f08337cd640 5 monclient: authenticate success, global_id 54345 2024-09-11T01:36:25.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.274+0000 7f08337cd640 10 monclient: _renew_subs 2024-09-11T01:36:25.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.274+0000 7f08337cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:25.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.274+0000 7f08337cd640 1 -- 172.21.15.12:0/2670222687 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f082c1285d0 con 0x7f08180039b0 2024-09-11T01:36:25.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.274+0000 7f0821ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:25.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.274+0000 7f0821ffb640 1 -- 172.21.15.12:0/2670222687 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0828036cf0 con 0x7f08180039b0 2024-09-11T01:36:25.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.274+0000 7f0821ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:25.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.274+0000 7f0821ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:25.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.274+0000 7f0821ffb640 10 monclient: dump: 2024-09-11T01:36:25.294 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:25.294 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:25.294 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.294 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.294 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:25.294 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:25.295 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:25.295 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:25.295 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:25.295 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:25.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.274+0000 7f08337cd640 10 monclient: _renew_subs 2024-09-11T01:36:25.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.274+0000 7f08337cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:25.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.274+0000 7f08337cd640 1 -- 172.21.15.12:0/2670222687 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f082c137b90 con 0x7f08180039b0 2024-09-11T01:36:25.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.274+0000 7f08237fe640 4 set_mon_vals no callback set 2024-09-11T01:36:25.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.275+0000 7f0821ffb640 1 -- 172.21.15.12:0/2670222687 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f082803f480 con 0x7f08180039b0 2024-09-11T01:36:25.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.275+0000 7f08337cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:25.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.275+0000 7f08337cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:25.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.275+0000 7f08337cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:25.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.275+0000 7f08337cd640 1 -- 172.21.15.12:0/2670222687 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f082c0a5020 con 0x7f08180039b0 2024-09-11T01:36:25.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.275+0000 7f0821ffb640 1 --2- 172.21.15.12:0/2670222687 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f081003ec30 0x7f08100410f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:25.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.275+0000 7f0821ffb640 1 -- 172.21.15.12:0/2670222687 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(225..225 src has 186..225) ==== 4771+0+0 (secure 0 0 0) 0x7f08280754c0 con 0x7f08180039b0 2024-09-11T01:36:25.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.276+0000 7f0830d68640 1 --2- 172.21.15.12:0/2670222687 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f081003ec30 0x7f08100410f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:25.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.276+0000 7f0830d68640 10 monclient: get_auth_request con 0x7f081003ec30 auth_method 0 2024-09-11T01:36:25.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.278+0000 7f0830d68640 1 --2- 172.21.15.12:0/2670222687 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f081003ec30 0x7f08100410f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f081c003e70 tx=0x7f081c00c4b0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:25.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.280+0000 7f0821ffb640 1 -- 172.21.15.12:0/2670222687 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0828073ec0 con 0x7f08180039b0 2024-09-11T01:36:25.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.280+0000 7f0821ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:25.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.280+0000 7f0821ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.541+0000 7f08337cd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:36:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.541+0000 7f08337cd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:36:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.541+0000 7f08337cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:25.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.541+0000 7f08337cd640 1 -- 172.21.15.12:0/2670222687 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f082c0a14a0 con 0x7f08180039b0 2024-09-11T01:36:25.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.542+0000 7f0821ffb640 1 -- 172.21.15.12:0/2670222687 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f082803ec60 con 0x7f08180039b0 2024-09-11T01:36:25.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.542+0000 7f0821ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:36:25.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.542+0000 7f0821ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:25.544 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":78,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:36:25.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.544+0000 7f08337cd640 1 -- 172.21.15.12:0/2670222687 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f081003ec30 msgr2=0x7f08100410f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:25.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.544+0000 7f08337cd640 1 --2- 172.21.15.12:0/2670222687 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f081003ec30 0x7f08100410f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f081c003e70 tx=0x7f081c00c4b0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.544+0000 7f08337cd640 10 monclient: shutdown 2024-09-11T01:36:25.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.544+0000 7f08337cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:25.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.544+0000 7f08337cd640 1 -- 172.21.15.12:0/2670222687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08180039b0 msgr2=0x7f082c1ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:25.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.544+0000 7f08337cd640 1 --2- 172.21.15.12:0/2670222687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08180039b0 0x7f082c1ccdb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0828000c00 tx=0x7f0828004730 comp rx=0 tx=0).stop 2024-09-11T01:36:25.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.544+0000 7f08337cd640 1 -- 172.21.15.12:0/2670222687 shutdown_connections 2024-09-11T01:36:25.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.544+0000 7f08337cd640 1 --2- 172.21.15.12:0/2670222687 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0818005290 0x7f082c124870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.544+0000 7f08337cd640 1 --2- 172.21.15.12:0/2670222687 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0818004800 0x7f082c1cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.544+0000 7f08337cd640 1 --2- 172.21.15.12:0/2670222687 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f081003ec30 0x7f08100410f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.544+0000 7f08337cd640 1 --2- 172.21.15.12:0/2670222687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08180039b0 0x7f082c1ccdb0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.544+0000 7f08337cd640 1 -- 172.21.15.12:0/2670222687 >> 172.21.15.12:0/2670222687 conn(0x7f082c0a7120 msgr2=0x7f082c0a0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:25.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.544+0000 7f08337cd640 1 -- 172.21.15.12:0/2670222687 shutdown_connections 2024-09-11T01:36:25.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.544+0000 7f08337cd640 1 -- 172.21.15.12:0/2670222687 wait complete. 2024-09-11T01:36:25.556 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:36:25.556 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-11T01:36:25.557 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:36:25.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.656+0000 7f5db4d18640 10 monclient: get_monmap_and_config 2024-09-11T01:36:25.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.656+0000 7f5db4d18640 10 monclient: build_initial_monmap 2024-09-11T01:36:25.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.656+0000 7f5db4d18640 10 monclient: monmap: 2024-09-11T01:36:25.658 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:25.658 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:25.658 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:25.657284+0000 2024-09-11T01:36:25.658 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:25.657284+0000 2024-09-11T01:36:25.658 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:25.658 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:25.658 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:25.659 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:25.659 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:25.659 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:25.659 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:25.659 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:25.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.657+0000 7f5db4d18640 1 Processor -- start 2024-09-11T01:36:25.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.657+0000 7f5db4d18640 1 -- start start 2024-09-11T01:36:25.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.657+0000 7f5db4d18640 10 monclient: init 2024-09-11T01:36:25.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.657+0000 7f5db4d18640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:25.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.657+0000 7f5db4d18640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-11T01:36:25.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.657+0000 7f5db4d18640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5db0135f70 addr v1:172.21.15.71:6789/0 2024-09-11T01:36:25.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.657+0000 7f5db4d18640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5db009e020 0x7f5db013ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:25.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.657+0000 7f5db4d18640 10 monclient(hunting): picked mon.noname-b con 0x7f5db009e020 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:25.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.657+0000 7f5db4d18640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5db013b320 0x7f5db013d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:25.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.657+0000 7f5db4d18640 10 monclient(hunting): picked mon.noname-c con 0x7f5db013b320 addr v2:172.21.15.71:3300/0 2024-09-11T01:36:25.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.657+0000 7f5db4d18640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5db009da70 con 0x7f5db0135f70 2024-09-11T01:36:25.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.657+0000 7f5db4d18640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:25.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.657+0000 7f5db4d18640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f5db009dde0 con 0x7f5db013b320 2024-09-11T01:36:25.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.657+0000 7f5db4d18640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:25.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.657+0000 7f5db4d18640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5db006cc50 con 0x7f5db009e020 2024-09-11T01:36:25.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.657+0000 7f5db4d18640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:25.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.657+0000 7f5db4d18640 10 monclient(hunting): authenticate will time out at 1192.099877s 2024-09-11T01:36:25.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.657+0000 7f5dadd9b640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5db009e020 msgr2=0x7f5db013ade0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:36:25.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.657+0000 7f5dadd9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5db009e020 0x7f5db013ade0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:25.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.658+0000 7f5dae59c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f5db0135f70 0x7f5db0136350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.12:45370/0 (socket says 172.21.15.12:45370) 2024-09-11T01:36:25.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.658+0000 7f5dae59c640 1 -- 172.21.15.12:0/808974396 learned_addr learned my addr 172.21.15.12:0/808974396 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:36:25.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.658+0000 7f5daed9d640 1 --2- 172.21.15.12:0/808974396 >> v2:172.21.15.71:3300/0 conn(0x7f5db013b320 0x7f5db013d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:25.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.658+0000 7f5daed9d640 10 monclient(hunting): get_auth_request con 0x7f5db013b320 auth_method 0 2024-09-11T01:36:25.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.658+0000 7f5daed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:25.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.658+0000 7f5daed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:25.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.658+0000 7f5daed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:25.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.658+0000 7f5daed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:25.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.658+0000 7f5daed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:25.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.658+0000 7f5daed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:25.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.658+0000 7f5daed9d640 10 monclient(hunting): handle_auth_done global_id 74158 payload 274 2024-09-11T01:36:25.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.658+0000 7f5daed9d640 1 -- 172.21.15.12:0/808974396 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5db009e020 msgr2=0x7f5db013ade0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:36:25.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.658+0000 7f5daed9d640 1 --2- 172.21.15.12:0/808974396 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5db009e020 0x7f5db013ade0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.658+0000 7f5daed9d640 1 -- 172.21.15.12:0/808974396 >> v1:172.21.15.71:6789/0 conn(0x7f5db0135f70 legacy=0x7f5db0136350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:25.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.658+0000 7f5daed9d640 10 monclient: _finish_hunting 0 2024-09-11T01:36:25.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.658+0000 7f5daed9d640 1 monclient: found mon.noname-c 2024-09-11T01:36:25.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.658+0000 7f5daed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:25.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.658+0000 7f5daed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:36:25.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.658+0000 7f5daed9d640 1 -- 172.21.15.12:0/808974396 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db006d3b0 con 0x7f5db013b320 2024-09-11T01:36:25.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.661+0000 7f5daed9d640 1 -- 172.21.15.12:0/808974396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5db013b320 msgr2=0x7f5db013d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:25.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.661+0000 7f5daed9d640 1 --2- 172.21.15.12:0/808974396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5db013b320 0x7f5db013d710 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5da8009d90 tx=0x7f5da802a4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f8d33f580696022 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:25.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.662+0000 7f5dad59a640 1 -- 172.21.15.12:0/808974396 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da8007a30 con 0x7f5db013b320 2024-09-11T01:36:25.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.662+0000 7f5dad59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:25.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.662+0000 7f5dad59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:36:25.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.662+0000 7f5dad59a640 10 monclient: dump: 2024-09-11T01:36:25.668 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:25.668 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:25.668 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.668 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.668 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:25.668 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:25.668 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:25.668 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:25.669 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:25.669 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:25.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.662+0000 7f5dad59a640 10 monclient: _finish_auth 0 2024-09-11T01:36:25.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.662+0000 7f5dad59a640 10 monclient: _check_auth_tickets 2024-09-11T01:36:25.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.662+0000 7f5dad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:25.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.662+0000 7f5dad59a640 1 -- 172.21.15.12:0/808974396 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5da8007bd0 con 0x7f5db013b320 2024-09-11T01:36:25.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.662+0000 7f5dad59a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:25.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.662+0000 7f5dad59a640 1 -- 172.21.15.12:0/808974396 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da8031400 con 0x7f5db013b320 2024-09-11T01:36:25.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.662+0000 7f5db4d18640 5 monclient: authenticate success, global_id 74158 2024-09-11T01:36:25.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.662+0000 7f5db4d18640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:25.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.662+0000 7f5db4d18640 4 set_mon_vals no callback set 2024-09-11T01:36:25.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.662+0000 7f5dad59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:25.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.662+0000 7f5dad59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:36:25.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.662+0000 7f5dad59a640 10 monclient: dump: 2024-09-11T01:36:25.670 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:25.670 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:25.670 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.670 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.670 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:25.670 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:25.670 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:25.670 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:25.670 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:25.670 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:25.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.662+0000 7f5db4d18640 10 monclient: shutdown 2024-09-11T01:36:25.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.662+0000 7f5db4d18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:25.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.662+0000 7f5db4d18640 1 -- 172.21.15.12:0/808974396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5db013b320 msgr2=0x7f5db013d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:25.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.662+0000 7f5db4d18640 1 --2- 172.21.15.12:0/808974396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5db013b320 0x7f5db013d710 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5da8009d90 tx=0x7f5da802a4b0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.662+0000 7f5db4d18640 1 -- 172.21.15.12:0/808974396 shutdown_connections 2024-09-11T01:36:25.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.662+0000 7f5db4d18640 1 --2- 172.21.15.12:0/808974396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5db013b320 0x7f5db013d710 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.662+0000 7f5db4d18640 1 --2- 172.21.15.12:0/808974396 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5db009e020 0x7f5db013ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.662+0000 7f5db4d18640 1 -- 172.21.15.12:0/808974396 >> 172.21.15.12:0/808974396 conn(0x7f5db00a3e30 msgr2=0x7f5db00a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:25.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.663+0000 7f5db4d18640 1 -- 172.21.15.12:0/808974396 shutdown_connections 2024-09-11T01:36:25.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.663+0000 7f5db4d18640 1 -- 172.21.15.12:0/808974396 wait complete. 2024-09-11T01:36:25.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.663+0000 7f5db4d18640 10 monclient: build_initial_monmap 2024-09-11T01:36:25.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.663+0000 7f5db4d18640 10 monclient: monmap: 2024-09-11T01:36:25.671 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:25.672 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:25.672 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:25.672 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:25.672 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:25.672 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:25.672 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:25.672 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:25.672 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:25.672 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:25.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.663+0000 7f5db4d18640 1 Processor -- start 2024-09-11T01:36:25.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5db4d18640 1 -- start start 2024-09-11T01:36:25.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5db4d18640 10 monclient: init 2024-09-11T01:36:25.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5db4d18640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:25.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5db4d18640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:36:25.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5db4d18640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5db009e020 0x7f5db012f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:25.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5db4d18640 10 monclient(hunting): picked mon.noname-b con 0x7f5db009e020 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:25.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5db4d18640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5db0135f70 0x7f5db009b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:25.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5db4d18640 10 monclient(hunting): picked mon.noname-c con 0x7f5db0135f70 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:25.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5db4d18640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db013b320 0x7f5db009b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:25.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5db4d18640 10 monclient(hunting): picked mon.noname-a con 0x7f5db013b320 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:25.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5db4d18640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:25.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5db4d18640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5db013dc50 con 0x7f5db013b320 2024-09-11T01:36:25.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5db4d18640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:25.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5db4d18640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5db009f020 con 0x7f5db009e020 2024-09-11T01:36:25.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5db4d18640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:25.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5db4d18640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5db013ddd0 con 0x7f5db0135f70 2024-09-11T01:36:25.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5db4d18640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:25.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5db4d18640 10 monclient(hunting): authenticate will time out at 1192.106517s 2024-09-11T01:36:25.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5dadd9b640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5db0135f70 msgr2=0x7f5db009b480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:36:25.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5dadd9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5db0135f70 0x7f5db009b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:25.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5daed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db013b320 0x7f5db009b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:25.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5daed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db013b320 0x7f5db009b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56530/0 (socket says 172.21.15.12:56530) 2024-09-11T01:36:25.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5daed9d640 1 -- 172.21.15.12:0/3156102911 learned_addr learned my addr 172.21.15.12:0/3156102911 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:25.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5daed9d640 10 monclient(hunting): get_auth_request con 0x7f5db013b320 auth_method 0 2024-09-11T01:36:25.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5daed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:25.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5daed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:25.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5daed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:25.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5daed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:25.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5daed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:25.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5dae59c640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5db009e020 0x7f5db012f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:25.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5daed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:25.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5dae59c640 10 monclient(hunting): get_auth_request con 0x7f5db009e020 auth_method 0 2024-09-11T01:36:25.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5dae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:25.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5dae59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:25.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5dae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:25.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5dae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:25.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5dae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:25.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5dae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:25.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5daed9d640 10 monclient(hunting): handle_auth_done global_id 54348 payload 824 2024-09-11T01:36:25.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5daed9d640 1 -- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5db0135f70 msgr2=0x7f5db009b480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:36:25.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5daed9d640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5db0135f70 0x7f5db009b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5daed9d640 1 -- 172.21.15.12:0/3156102911 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5db009e020 msgr2=0x7f5db012f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:25.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5daed9d640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5db009e020 0x7f5db012f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5daed9d640 10 monclient: _finish_hunting 0 2024-09-11T01:36:25.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5daed9d640 1 monclient: found mon.noname-a 2024-09-11T01:36:25.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5daed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:25.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5daed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:25.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.664+0000 7f5daed9d640 1 -- 172.21.15.12:0/3156102911 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db012fe40 con 0x7f5db013b320 2024-09-11T01:36:25.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.667+0000 7f5daed9d640 1 -- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db013b320 msgr2=0x7f5db009b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:25.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.667+0000 7f5daed9d640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db013b320 0x7f5db009b9c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5da8009eb0 tx=0x7f5da8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:25.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.667+0000 7f5daed9d640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db013b320 0x7f5db009b9c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5da8009eb0 tx=0x7f5da8004a40 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7f5da8040070 mon_map magic: 0 2024-09-11T01:36:25.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.667+0000 7f5d9effd640 1 -- 172.21.15.12:0/3156102911 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da8040070 con 0x7f5db013b320 2024-09-11T01:36:25.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.667+0000 7f5d9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:25.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.667+0000 7f5d9effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:25.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.667+0000 7f5d9effd640 10 monclient: dump: 2024-09-11T01:36:25.680 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:25.680 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:25.680 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.681 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.681 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:25.681 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:25.681 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:25.681 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:25.681 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:25.681 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:25.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.667+0000 7f5d9effd640 10 monclient: _finish_auth 0 2024-09-11T01:36:25.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.667+0000 7f5d9effd640 10 monclient: _check_auth_tickets 2024-09-11T01:36:25.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.667+0000 7f5d9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:25.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.667+0000 7f5d9effd640 1 -- 172.21.15.12:0/3156102911 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5da8045660 con 0x7f5db013b320 2024-09-11T01:36:25.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.667+0000 7f5db4d18640 5 monclient: authenticate success, global_id 54348 2024-09-11T01:36:25.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.667+0000 7f5db4d18640 10 monclient: _renew_subs 2024-09-11T01:36:25.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.667+0000 7f5db4d18640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:25.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.667+0000 7f5db4d18640 1 -- 172.21.15.12:0/3156102911 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5db009e520 con 0x7f5db013b320 2024-09-11T01:36:25.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.667+0000 7f5d9effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:25.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.667+0000 7f5dacd99640 4 set_mon_vals no callback set 2024-09-11T01:36:25.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.667+0000 7f5db4d18640 10 monclient: _renew_subs 2024-09-11T01:36:25.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.667+0000 7f5db4d18640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:25.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.667+0000 7f5db4d18640 1 -- 172.21.15.12:0/3156102911 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5db012c810 con 0x7f5db013b320 2024-09-11T01:36:25.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.668+0000 7f5db4d18640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:25.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.668+0000 7f5db4d18640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db009bf00 0x7f5db0130530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:25.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.668+0000 7f5db4d18640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:25.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.668+0000 7f5db4d18640 1 -- 172.21.15.12:0/3156102911 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5db009c2e0 con 0x7f5db009bf00 2024-09-11T01:36:25.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.668+0000 7f5dae59c640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db009bf00 0x7f5db0130530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:25.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.668+0000 7f5dae59c640 10 monclient: get_auth_request con 0x7f5db009bf00 auth_method 0 2024-09-11T01:36:25.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.668+0000 7f5dae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:25.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.668+0000 7f5dae59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:25.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.668+0000 7f5dae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:25.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.668+0000 7f5d9effd640 1 -- 172.21.15.12:0/3156102911 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da8045a30 con 0x7f5db013b320 2024-09-11T01:36:25.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.668+0000 7f5d9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:25.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.668+0000 7f5d9effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:25.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.668+0000 7f5d9effd640 10 monclient: dump: 2024-09-11T01:36:25.683 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:25.683 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:25.683 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.683 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.683 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:25.683 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:25.683 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:25.683 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:25.684 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:25.684 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:25.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.668+0000 7f5dae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:25.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.668+0000 7f5dae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:25.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.668+0000 7f5d9effd640 1 -- 172.21.15.12:0/3156102911 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f5da8007a40 con 0x7f5db013b320 2024-09-11T01:36:25.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.668+0000 7f5dae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:25.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.668+0000 7f5dae59c640 10 monclient(hunting): handle_auth_done global_id 54351 payload 824 2024-09-11T01:36:25.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.668+0000 7f5dae59c640 1 -- 172.21.15.12:0/3156102911 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5db0136350 con 0x7f5db009bf00 2024-09-11T01:36:25.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.668+0000 7f5d9effd640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5d9003e1c0 0x7f5d90040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:25.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.669+0000 7f5dadd9b640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5d9003e1c0 0x7f5d90040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:25.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.669+0000 7f5dadd9b640 10 monclient: get_auth_request con 0x7f5d9003e1c0 auth_method 0 2024-09-11T01:36:25.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.670+0000 7f5dae59c640 1 -- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db009bf00 msgr2=0x7f5db0130530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:25.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.670+0000 7f5dae59c640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db009bf00 0x7f5db0130530 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5da000ea70 tx=0x7f5da000ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:25.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.670+0000 7f5dae59c640 1 -- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db009bf00 msgr2=0x7f5db0130530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:36:25.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.670+0000 7f5dae59c640 1 -- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db009bf00 msgr2=0x7f5db0130530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:36:25.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.670+0000 7f5dae59c640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db009bf00 0x7f5db0130530 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5da000ea70 tx=0x7f5da000ef30 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:25.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.671+0000 7f5dae59c640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db009bf00 0x7f5db0130530 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5da000ea70 tx=0x7f5da000ef30 comp rx=0 tx=0).stop 2024-09-11T01:36:25.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.671+0000 7f5d9effd640 1 -- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db009bf00 msgr2=0x7f5db0130530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:25.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.671+0000 7f5d9effd640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db009bf00 0x7f5db0130530 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.671+0000 7f5d9effd640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d90040d90 0x7f5d90043180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:25.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.671+0000 7f5d9effd640 1 -- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db009bf00 msgr2=0x7f5db0130530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:25.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.671+0000 7f5d9effd640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db009bf00 0x7f5db0130530 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.671+0000 7f5d9effd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:25.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.671+0000 7f5d9effd640 1 -- 172.21.15.12:0/3156102911 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5d90043760 con 0x7f5d90040d90 2024-09-11T01:36:25.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.671+0000 7f5dae59c640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d90040d90 0x7f5d90043180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:25.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.671+0000 7f5dae59c640 10 monclient: get_auth_request con 0x7f5d90040d90 auth_method 0 2024-09-11T01:36:25.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.671+0000 7f5dae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:25.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.671+0000 7f5dae59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:25.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.671+0000 7f5dae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:25.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.671+0000 7f5dae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:25.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.671+0000 7f5dae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:25.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.671+0000 7f5dae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:25.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.671+0000 7f5dae59c640 10 monclient(hunting): handle_auth_done global_id 54354 payload 824 2024-09-11T01:36:25.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.671+0000 7f5dae59c640 1 -- 172.21.15.12:0/3156102911 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5da8047e50 con 0x7f5d90040d90 2024-09-11T01:36:25.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.671+0000 7f5dadd9b640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5d9003e1c0 0x7f5d90040680 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5db009d4c0 tx=0x7f5da4006d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:25.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.673+0000 7f5dae59c640 1 -- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d90040d90 msgr2=0x7f5d90043180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:25.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.673+0000 7f5dae59c640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d90040d90 0x7f5d90043180 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5da000c610 tx=0x7f5da0004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:25.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.673+0000 7f5d9effd640 1 -- 172.21.15.12:0/3156102911 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da0024070 con 0x7f5d90040d90 2024-09-11T01:36:25.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.673+0000 7f5d9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:25.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.673+0000 7f5d9effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:25.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.673+0000 7f5d9effd640 10 monclient: dump: 2024-09-11T01:36:25.687 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:25.687 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:25.687 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.687 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.687 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:25.687 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:25.687 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:25.687 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:25.687 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:25.687 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:25.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.673+0000 7f5d9effd640 1 -- 172.21.15.12:0/3156102911 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5da0020630 con 0x7f5d90040d90 2024-09-11T01:36:25.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.673+0000 7f5d9effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:25.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.673+0000 7f5d9effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:25.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.673+0000 7f5d9effd640 1 -- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d90040d90 msgr2=0x7f5d90043180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:25.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.673+0000 7f5d9effd640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d90040d90 0x7f5d90043180 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5da000c610 tx=0x7f5da0004940 comp rx=0 tx=0).stop 2024-09-11T01:36:25.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.674+0000 7f5d9effd640 1 -- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d90040d90 msgr2=0x7f5d90043180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:25.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.674+0000 7f5d9effd640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d90040d90 0x7f5d90043180 secure :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5da000c610 tx=0x7f5da0004940 comp rx=0 tx=0).stop 2024-09-11T01:36:25.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.680+0000 7f5db4d18640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:36:25.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.680+0000 7f5db4d18640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db0130a70 0x7f5db0130e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:25.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.680+0000 7f5db4d18640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:25.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.680+0000 7f5db4d18640 1 -- 172.21.15.12:0/3156102911 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5db006e520 con 0x7f5db0130a70 2024-09-11T01:36:25.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.680+0000 7f5dae59c640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db0130a70 0x7f5db0130e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:25.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.680+0000 7f5dae59c640 10 monclient: get_auth_request con 0x7f5db0130a70 auth_method 0 2024-09-11T01:36:25.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.680+0000 7f5dae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:25.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.680+0000 7f5dae59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:25.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.680+0000 7f5dae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:25.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.680+0000 7f5dae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:25.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.680+0000 7f5dae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:25.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.680+0000 7f5dae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:25.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.680+0000 7f5dae59c640 10 monclient(hunting): handle_auth_done global_id 54357 payload 824 2024-09-11T01:36:25.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.680+0000 7f5dae59c640 1 -- 172.21.15.12:0/3156102911 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5db01313c0 con 0x7f5db0130a70 2024-09-11T01:36:25.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.682+0000 7f5dae59c640 1 -- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db0130a70 msgr2=0x7f5db0130e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:25.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.682+0000 7f5dae59c640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db0130a70 0x7f5db0130e80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5da000e4a0 tx=0x7f5da0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:25.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.682+0000 7f5d9effd640 1 -- 172.21.15.12:0/3156102911 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da001a070 con 0x7f5db0130a70 2024-09-11T01:36:25.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.682+0000 7f5d9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:25.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.682+0000 7f5d9effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:25.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.682+0000 7f5d9effd640 10 monclient: dump: 2024-09-11T01:36:25.691 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:25.691 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:25.691 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.691 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.691 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:25.692 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:25.692 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:25.692 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:25.692 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:25.692 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:25.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.683+0000 7f5d9effd640 1 -- 172.21.15.12:0/3156102911 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4699 (secure 0 0 0) 0x7f5da0020e30 con 0x7f5db0130a70 2024-09-11T01:36:25.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.683+0000 7f5d9effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:36:25.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.683+0000 7f5d9effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:25.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.683+0000 7f5d9effd640 1 -- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db0130a70 msgr2=0x7f5db0130e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:25.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.683+0000 7f5d9effd640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db0130a70 0x7f5db0130e80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5da000e4a0 tx=0x7f5da0018000 comp rx=0 tx=0).stop 2024-09-11T01:36:25.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.683+0000 7f5d9effd640 1 -- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db0130a70 msgr2=0x7f5db0130e80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:25.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.683+0000 7f5d9effd640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db0130a70 0x7f5db0130e80 secure :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5da000e4a0 tx=0x7f5da0018000 comp rx=0 tx=0).stop 2024-09-11T01:36:25.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.683+0000 7f5daed9d640 1 -- 172.21.15.12:0/3156102911 reap_dead start 2024-09-11T01:36:25.693 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:36:25.693 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:36:25.693 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:36:25.693 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:36:25.693 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 78, 2024-09-11T01:36:25.693 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 175663, 2024-09-11T01:36:25.693 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:36:25.693 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:36:25.693 INFO:teuthology.orchestra.run.smithi012.stdout: 1 2024-09-11T01:36:25.694 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:25.694 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:36:25.694 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:25.694 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:36:25.694 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:36:25.694 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:25.694 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:25.694 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:25.694 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:25.694 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:25.694 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:25.694 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:25.694 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:25.695 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:25.695 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:25.695 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:25.695 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:25.695 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:36:25.695 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:36:25.695 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:25.695 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:25.695 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:25.695 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:25.695 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:25.695 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:25.695 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:25.695 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:25.696 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:25.696 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:25.696 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:25.696 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:25.696 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:25.696 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:36:25.696 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:36:25.696 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:36:25.696 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:36:25.696 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:36:25.696 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:36:25.696 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:25.697 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:25.697 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:36:25.697 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:36:25.697 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:36:25.697 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:36:25.697 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:36:25.697 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:36:25.697 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:36:25.697 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:25.697 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:36:25.698 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:25.698 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:25.698 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:25.698 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:25.698 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:25.698 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:25.698 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:25.698 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:25.698 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:25.698 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:25.698 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:25.698 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:25.698 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:36:25.698 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:25.698 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:36:25.698 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:25.699 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:36:25.699 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:36:25.699 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:25.699 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:25.699 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:25.699 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:25.699 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:36:25.699 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:25.699 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:25.699 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:25.699 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:25.699 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:36:25.699 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:25.700 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:25.700 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:25.700 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:25.700 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:36:25.700 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:36:25.700 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:25.700 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:25.700 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:25.700 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:25.700 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:25.700 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:36:25.700 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:36:25.700 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:25.700 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:25.701 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:25.701 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:25.701 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:36:25.701 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:25.701 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:25.701 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:25.701 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:25.701 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:36:25.701 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:25.701 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:25.701 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:25.701 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:25.701 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:36:25.701 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:36:25.702 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:25.702 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:25.702 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:25.702 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:25.702 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:25.702 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:36:25.702 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:36:25.702 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:25.702 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:25.702 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:25.702 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:25.702 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:36:25.702 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:25.702 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:25.703 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:25.703 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:25.703 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:36:25.703 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:25.703 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:25.703 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:25.703 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:25.703 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:36:25.703 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:36:25.703 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:25.703 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:25.703 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:25.703 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:25.703 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:25.704 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:25.704 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:36:25.704 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:36:25.704 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:25.704 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:25.704 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:25.704 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:36:25.704 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:25.704 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:25.704 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:36:25.704 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:25.704 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:25.704 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:25.704 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:36:25.705 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:25.705 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:25.705 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:36:25.705 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:25.705 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:25.705 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:25.705 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:36:25.705 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:25.705 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:25.705 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:25.705 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:36:25.705 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:36:25.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.683+0000 7f5db4d18640 1 -- 172.21.15.12:0/3156102911 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5d9003e1c0 msgr2=0x7f5d90040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:25.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.683+0000 7f5db4d18640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5d9003e1c0 0x7f5d90040680 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5db009d4c0 tx=0x7f5da4006d30 comp rx=0 tx=0).stop 2024-09-11T01:36:25.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.683+0000 7f5db4d18640 10 monclient: shutdown 2024-09-11T01:36:25.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.683+0000 7f5db4d18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:25.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.683+0000 7f5db4d18640 1 -- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db013b320 msgr2=0x7f5db009b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:25.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.683+0000 7f5db4d18640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db013b320 0x7f5db009b9c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5da8009eb0 tx=0x7f5da8004a40 comp rx=0 tx=0).stop 2024-09-11T01:36:25.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.683+0000 7f5d9effd640 1 -- 172.21.15.12:0/3156102911 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(225..225 src has 186..225) ==== 4771+0+0 (unknown 0 0 0) 0x7f5da80748b0 con 0x7f5db013b320 2024-09-11T01:36:25.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.683+0000 7f5db4d18640 1 -- 172.21.15.12:0/3156102911 shutdown_connections 2024-09-11T01:36:25.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.683+0000 7f5db4d18640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5d9003e1c0 0x7f5d90040680 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.683+0000 7f5db4d18640 1 --2- 172.21.15.12:0/3156102911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5db013b320 0x7f5db009b9c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.683+0000 7f5db4d18640 1 -- 172.21.15.12:0/3156102911 >> 172.21.15.12:0/3156102911 conn(0x7f5db00a3e30 msgr2=0x7f5db00a5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:25.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.683+0000 7f5db4d18640 1 -- 172.21.15.12:0/3156102911 shutdown_connections 2024-09-11T01:36:25.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.683+0000 7f5db4d18640 1 -- 172.21.15.12:0/3156102911 wait complete. 2024-09-11T01:36:25.708 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:36:25.708 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T01:36:25.708 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:36:25.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.808+0000 7f2ec5847640 10 monclient: get_monmap_and_config 2024-09-11T01:36:25.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.808+0000 7f2ec5847640 10 monclient: build_initial_monmap 2024-09-11T01:36:25.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.808+0000 7f2ec5847640 10 monclient: monmap: 2024-09-11T01:36:25.810 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:25.810 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:25.810 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:25.809237+0000 2024-09-11T01:36:25.810 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:25.809237+0000 2024-09-11T01:36:25.810 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:25.811 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.809+0000 7f2ec5847640 1 Processor -- start 2024-09-11T01:36:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.809+0000 7f2ec5847640 1 -- start start 2024-09-11T01:36:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.809+0000 7f2ec5847640 10 monclient: init 2024-09-11T01:36:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.809+0000 7f2ec5847640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.809+0000 7f2ec5847640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-11T01:36:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.809+0000 7f2ec5847640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2ec0134ad0 0x7f2ec0134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.809+0000 7f2ec5847640 10 monclient(hunting): picked mon.noname-a con 0x7f2ec0134ad0 addr v2:172.21.15.12:3300/0 2024-09-11T01:36:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.809+0000 7f2ec5847640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2ec00a2200 0x7f2ec01353f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.809+0000 7f2ec5847640 10 monclient(hunting): picked mon.noname-c con 0x7f2ec00a2200 addr v2:172.21.15.71:3300/0 2024-09-11T01:36:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.809+0000 7f2ec5847640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2ec0133a50 0x7f2ec0133f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.809+0000 7f2ec5847640 10 monclient(hunting): picked mon.noname-b con 0x7f2ec0133a50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.809+0000 7f2ec5847640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.809+0000 7f2ec5847640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f2ec009f5d0 con 0x7f2ec0134ad0 2024-09-11T01:36:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.809+0000 7f2ec5847640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.809+0000 7f2ec5847640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f2ec0135930 con 0x7f2ec00a2200 2024-09-11T01:36:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.809+0000 7f2ec5847640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.809+0000 7f2ec5847640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2ec009ed30 con 0x7f2ec0133a50 2024-09-11T01:36:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.809+0000 7f2ec5847640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.809+0000 7f2ec5847640 10 monclient(hunting): authenticate will time out at 1192.251933s 2024-09-11T01:36:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.809+0000 7f2ebf7fe640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2ec0133a50 msgr2=0x7f2ec0133f00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:36:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.809+0000 7f2ebf7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2ec0133a50 0x7f2ec0133f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.810+0000 7f2ebeffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2ec0134ad0 0x7f2ec0134eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.810+0000 7f2ebeffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2ec0134ad0 0x7f2ec0134eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56562/0 (socket says 172.21.15.12:56562) 2024-09-11T01:36:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.810+0000 7f2ebeffd640 1 -- 172.21.15.12:0/2792806356 learned_addr learned my addr 172.21.15.12:0/2792806356 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.810+0000 7f2ebeffd640 10 monclient(hunting): get_auth_request con 0x7f2ec0134ad0 auth_method 0 2024-09-11T01:36:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.810+0000 7f2ebeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.810+0000 7f2ebeffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.810+0000 7f2ebeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.810+0000 7f2ebeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.810+0000 7f2ebeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.810+0000 7f2ebeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.810+0000 7f2eb7fff640 1 --2- 172.21.15.12:0/2792806356 >> v2:172.21.15.71:3300/0 conn(0x7f2ec00a2200 0x7f2ec01353f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.810+0000 7f2eb7fff640 10 monclient(hunting): get_auth_request con 0x7f2ec00a2200 auth_method 0 2024-09-11T01:36:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.810+0000 7f2eb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.810+0000 7f2eb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.810+0000 7f2eb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.810+0000 7f2ebeffd640 10 monclient(hunting): handle_auth_done global_id 54360 payload 274 2024-09-11T01:36:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.810+0000 7f2ebeffd640 1 -- 172.21.15.12:0/2792806356 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2ec0133a50 msgr2=0x7f2ec0133f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:36:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.810+0000 7f2ebeffd640 1 --2- 172.21.15.12:0/2792806356 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2ec0133a50 0x7f2ec0133f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.810+0000 7f2ebeffd640 1 -- 172.21.15.12:0/2792806356 >> v2:172.21.15.71:3300/0 conn(0x7f2ec00a2200 msgr2=0x7f2ec01353f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.810+0000 7f2ebeffd640 1 --2- 172.21.15.12:0/2792806356 >> v2:172.21.15.71:3300/0 conn(0x7f2ec00a2200 0x7f2ec01353f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.810+0000 7f2ebeffd640 10 monclient: _finish_hunting 0 2024-09-11T01:36:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.810+0000 7f2ebeffd640 1 monclient: found mon.noname-a 2024-09-11T01:36:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.810+0000 7f2ebeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.810+0000 7f2ebeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.810+0000 7f2ebeffd640 1 -- 172.21.15.12:0/2792806356 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ec009f010 con 0x7f2ec0134ad0 2024-09-11T01:36:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.810+0000 7f2eb7fff640 1 --2- 172.21.15.12:0/2792806356 >> v2:172.21.15.71:3300/0 conn(0x7f2ec00a2200 0x7f2ec01353f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:36:25.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.813+0000 7f2ebeffd640 1 -- 172.21.15.12:0/2792806356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ec0134ad0 msgr2=0x7f2ec0134eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:25.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.813+0000 7f2ebeffd640 1 --2- 172.21.15.12:0/2792806356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ec0134ad0 0x7f2ec0134eb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2eb000bb10 tx=0x7f2eb002c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eff44a42dea188a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:25.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.813+0000 7f2ebe7fc640 1 -- 172.21.15.12:0/2792806356 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0031920 con 0x7f2ec0134ad0 2024-09-11T01:36:25.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.813+0000 7f2ebe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:25.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.813+0000 7f2ebe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:25.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.813+0000 7f2ebe7fc640 10 monclient: dump: 2024-09-11T01:36:25.823 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:25.824 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:25.824 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.824 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.824 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:25.824 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:25.824 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:25.824 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:25.824 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:25.824 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:25.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.813+0000 7f2ebe7fc640 10 monclient: _finish_auth 0 2024-09-11T01:36:25.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.813+0000 7f2ebe7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:36:25.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.813+0000 7f2ebe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:25.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.813+0000 7f2ebe7fc640 1 -- 172.21.15.12:0/2792806356 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2eb0031ac0 con 0x7f2ec0134ad0 2024-09-11T01:36:25.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.813+0000 7f2ec5847640 5 monclient: authenticate success, global_id 54360 2024-09-11T01:36:25.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.813+0000 7f2ec5847640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:36:25.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.813+0000 7f2ebe7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:25.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.813+0000 7f2ebe7fc640 1 -- 172.21.15.12:0/2792806356 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0033400 con 0x7f2ec0134ad0 2024-09-11T01:36:25.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.813+0000 7f2ec5847640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:25.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.813+0000 7f2ec5847640 4 set_mon_vals no callback set 2024-09-11T01:36:25.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.813+0000 7f2ebe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:25.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.813+0000 7f2ebe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:25.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.813+0000 7f2ebe7fc640 10 monclient: dump: 2024-09-11T01:36:25.825 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:25.825 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:25.825 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.826 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.826 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:25.826 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:25.826 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:25.826 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:25.826 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:25.826 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:25.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.814+0000 7f2ec5847640 10 monclient: shutdown 2024-09-11T01:36:25.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.814+0000 7f2ec5847640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:25.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.814+0000 7f2ec5847640 1 -- 172.21.15.12:0/2792806356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ec0134ad0 msgr2=0x7f2ec0134eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:25.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.814+0000 7f2ec5847640 1 --2- 172.21.15.12:0/2792806356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ec0134ad0 0x7f2ec0134eb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2eb000bb10 tx=0x7f2eb002c7d0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.814+0000 7f2ec5847640 1 -- 172.21.15.12:0/2792806356 shutdown_connections 2024-09-11T01:36:25.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.814+0000 7f2ec5847640 1 --2- 172.21.15.12:0/2792806356 >> v2:172.21.15.71:3300/0 conn(0x7f2ec00a2200 0x7f2ec01353f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.814+0000 7f2ec5847640 1 --2- 172.21.15.12:0/2792806356 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2ec0133a50 0x7f2ec0133f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.814+0000 7f2ec5847640 1 --2- 172.21.15.12:0/2792806356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ec0134ad0 0x7f2ec0134eb0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.814+0000 7f2ec5847640 1 -- 172.21.15.12:0/2792806356 >> 172.21.15.12:0/2792806356 conn(0x7f2ec00a2f70 msgr2=0x7f2ec00a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:25.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.814+0000 7f2ec5847640 1 -- 172.21.15.12:0/2792806356 shutdown_connections 2024-09-11T01:36:25.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.814+0000 7f2ec5847640 1 -- 172.21.15.12:0/2792806356 wait complete. 2024-09-11T01:36:25.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.814+0000 7f2ec5847640 10 monclient: build_initial_monmap 2024-09-11T01:36:25.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.814+0000 7f2ec5847640 10 monclient: monmap: 2024-09-11T01:36:25.827 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:25.827 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:25.827 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:25.827 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:25.827 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:25.827 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:25.828 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:25.828 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:25.828 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:25.828 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:25.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ec5847640 1 Processor -- start 2024-09-11T01:36:25.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ec5847640 1 -- start start 2024-09-11T01:36:25.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ec5847640 10 monclient: init 2024-09-11T01:36:25.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ec5847640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:25.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ec5847640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:36:25.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ec5847640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ec00a2200 0x7f2ec011f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:25.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ec5847640 10 monclient(hunting): picked mon.noname-a con 0x7f2ec00a2200 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:25.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ec5847640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2ec0133a50 0x7f2ec011f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:25.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ec5847640 10 monclient(hunting): picked mon.noname-c con 0x7f2ec0133a50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:25.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ec5847640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2ec0134ad0 0x7f2ec011fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:25.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ec5847640 10 monclient(hunting): picked mon.noname-b con 0x7f2ec0134ad0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:25.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ec5847640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:25.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ec5847640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec009ffd0 con 0x7f2ec00a2200 2024-09-11T01:36:25.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ec5847640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:25.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ec5847640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec009f380 con 0x7f2ec0134ad0 2024-09-11T01:36:25.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ec5847640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:25.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ec5847640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2ec00a0150 con 0x7f2ec0133a50 2024-09-11T01:36:25.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ec5847640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:25.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2eb7fff640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2ec0133a50 msgr2=0x7f2ec011f640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:36:25.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ec5847640 10 monclient(hunting): authenticate will time out at 1192.257695s 2024-09-11T01:36:25.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2eb7fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2ec0133a50 0x7f2ec011f640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:25.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ebeffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ec00a2200 0x7f2ec011f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:25.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ebeffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ec00a2200 0x7f2ec011f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56578/0 (socket says 172.21.15.12:56578) 2024-09-11T01:36:25.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ebeffd640 1 -- 172.21.15.12:0/394914798 learned_addr learned my addr 172.21.15.12:0/394914798 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:25.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ebeffd640 10 monclient(hunting): get_auth_request con 0x7f2ec00a2200 auth_method 0 2024-09-11T01:36:25.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ebeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:25.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ebeffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:25.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ebeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:25.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ebeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:25.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ebeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:25.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.815+0000 7f2ebeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:25.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.816+0000 7f2ebeffd640 10 monclient(hunting): handle_auth_done global_id 54363 payload 824 2024-09-11T01:36:25.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.816+0000 7f2ebf7fe640 1 --2- 172.21.15.12:0/394914798 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2ec0134ad0 0x7f2ec011fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:25.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.816+0000 7f2ebeffd640 1 -- 172.21.15.12:0/394914798 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2ec0133a50 msgr2=0x7f2ec011f640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:36:25.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.816+0000 7f2ebeffd640 1 --2- 172.21.15.12:0/394914798 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2ec0133a50 0x7f2ec011f640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.816+0000 7f2ebeffd640 1 -- 172.21.15.12:0/394914798 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2ec0134ad0 msgr2=0x7f2ec011fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:25.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.816+0000 7f2ebeffd640 1 --2- 172.21.15.12:0/394914798 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2ec0134ad0 0x7f2ec011fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.816+0000 7f2ebeffd640 10 monclient: _finish_hunting 0 2024-09-11T01:36:25.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.816+0000 7f2ebeffd640 1 monclient: found mon.noname-a 2024-09-11T01:36:25.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.816+0000 7f2ebeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:25.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.816+0000 7f2ebeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:25.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.816+0000 7f2ebeffd640 1 -- 172.21.15.12:0/394914798 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ec0122c40 con 0x7f2ec00a2200 2024-09-11T01:36:25.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.818+0000 7f2ebeffd640 1 -- 172.21.15.12:0/394914798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ec00a2200 msgr2=0x7f2ec011f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:25.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.818+0000 7f2ebeffd640 1 --2- 172.21.15.12:0/394914798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ec00a2200 0x7f2ec011f100 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2eb000bb10 tx=0x7f2eb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:25.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.818+0000 7f2ebcff9640 1 -- 172.21.15.12:0/394914798 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0041070 con 0x7f2ec00a2200 2024-09-11T01:36:25.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.818+0000 7f2ebcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:25.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.818+0000 7f2ebcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:25.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.818+0000 7f2ebcff9640 10 monclient: dump: 2024-09-11T01:36:25.832 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:25.832 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:25.832 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.832 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.832 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:25.832 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:25.832 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:25.832 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:25.832 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:25.832 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:25.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.818+0000 7f2ebcff9640 10 monclient: _finish_auth 0 2024-09-11T01:36:25.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.818+0000 7f2ebcff9640 10 monclient: _check_auth_tickets 2024-09-11T01:36:25.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.818+0000 7f2ebcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:25.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.818+0000 7f2ebcff9640 1 -- 172.21.15.12:0/394914798 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2eb0046660 con 0x7f2ec00a2200 2024-09-11T01:36:25.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.818+0000 7f2ebcff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:25.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.818+0000 7f2ebcff9640 1 -- 172.21.15.12:0/394914798 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0046a50 con 0x7f2ec00a2200 2024-09-11T01:36:25.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.818+0000 7f2ec5847640 5 monclient: authenticate success, global_id 54363 2024-09-11T01:36:25.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.818+0000 7f2ec5847640 10 monclient: _renew_subs 2024-09-11T01:36:25.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.818+0000 7f2ec5847640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:25.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.818+0000 7f2ec5847640 1 -- 172.21.15.12:0/394914798 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ec0122f20 con 0x7f2ec00a2200 2024-09-11T01:36:25.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.818+0000 7f2ebcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:25.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.818+0000 7f2ebcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:25.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.818+0000 7f2ebdffb640 4 set_mon_vals no callback set 2024-09-11T01:36:25.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.818+0000 7f2ebcff9640 10 monclient: dump: 2024-09-11T01:36:25.833 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:25.834 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:25.834 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.834 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.834 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:25.834 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:25.834 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:25.834 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:25.834 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:25.834 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:25.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.818+0000 7f2ec5847640 10 monclient: _renew_subs 2024-09-11T01:36:25.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.818+0000 7f2ec5847640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:25.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.818+0000 7f2ec5847640 1 -- 172.21.15.12:0/394914798 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ec01220b0 con 0x7f2ec00a2200 2024-09-11T01:36:25.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.819+0000 7f2ec5847640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:25.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.819+0000 7f2ec5847640 1 --2- 172.21.15.12:0/394914798 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2ec00726b0 0x7f2ec0072a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:25.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.819+0000 7f2ec5847640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:25.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.819+0000 7f2ebcff9640 1 -- 172.21.15.12:0/394914798 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f2eb000e040 con 0x7f2ec00a2200 2024-09-11T01:36:25.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.819+0000 7f2ec5847640 1 -- 172.21.15.12:0/394914798 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec0134560 con 0x7f2ec00726b0 2024-09-11T01:36:25.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.819+0000 7f2ebcff9640 1 --2- 172.21.15.12:0/394914798 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2ea0042600 0x7f2ea0044ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:25.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.819+0000 7f2ebcff9640 1 -- 172.21.15.12:0/394914798 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(225..225 src has 186..225) ==== 4771+0+0 (secure 0 0 0) 0x7f2eb0073fc0 con 0x7f2ec00a2200 2024-09-11T01:36:25.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.819+0000 7f2eb7fff640 1 --2- 172.21.15.12:0/394914798 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2ec00726b0 0x7f2ec0072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:25.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.819+0000 7f2eb7fff640 10 monclient: get_auth_request con 0x7f2ec00726b0 auth_method 0 2024-09-11T01:36:25.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.819+0000 7f2eb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:25.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.819+0000 7f2eb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:25.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.819+0000 7f2eb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:25.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.819+0000 7f2eb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:25.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.819+0000 7f2eb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:25.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.819+0000 7f2eb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:25.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.819+0000 7f2ebf7fe640 1 --2- 172.21.15.12:0/394914798 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2ea0042600 0x7f2ea0044ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:25.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.819+0000 7f2ebf7fe640 10 monclient: get_auth_request con 0x7f2ea0042600 auth_method 0 2024-09-11T01:36:25.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.820+0000 7f2eb7fff640 10 monclient(hunting): handle_auth_done global_id 74167 payload 824 2024-09-11T01:36:25.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.820+0000 7f2eb7fff640 1 -- 172.21.15.12:0/394914798 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2ec00008d0 con 0x7f2ec00726b0 2024-09-11T01:36:25.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.822+0000 7f2eb7fff640 1 -- 172.21.15.12:0/394914798 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2ec00726b0 msgr2=0x7f2ec0072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:25.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.822+0000 7f2eb7fff640 1 --2- 172.21.15.12:0/394914798 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2ec00726b0 0x7f2ec0072a90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2ec00a2630 tx=0x7f2ea800eba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:25.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.822+0000 7f2ebf7fe640 1 --2- 172.21.15.12:0/394914798 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2ea0042600 0x7f2ea0044ac0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2ec01214b0 tx=0x7f2eb80071c0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:25.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.822+0000 7f2ebcff9640 1 -- 172.21.15.12:0/394914798 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8017070 con 0x7f2ec00726b0 2024-09-11T01:36:25.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.822+0000 7f2ebcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:25.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.822+0000 7f2ebcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:36:25.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.822+0000 7f2ebcff9640 10 monclient: dump: 2024-09-11T01:36:25.838 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:25.839 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:25.839 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.839 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.839 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:25.839 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:25.839 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:25.839 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:25.839 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:25.839 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:25.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.823+0000 7f2ebcff9640 1 -- 172.21.15.12:0/394914798 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2ea800c940 con 0x7f2ec00726b0 2024-09-11T01:36:25.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.823+0000 7f2ebcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:25.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.823+0000 7f2ebcff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:25.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.823+0000 7f2ebcff9640 1 -- 172.21.15.12:0/394914798 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2ec00726b0 msgr2=0x7f2ec0072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:25.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.823+0000 7f2ebcff9640 1 --2- 172.21.15.12:0/394914798 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2ec00726b0 0x7f2ec0072a90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2ec00a2630 tx=0x7f2ea800eba0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.823+0000 7f2ebcff9640 1 -- 172.21.15.12:0/394914798 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2ec00726b0 msgr2=0x7f2ec0072a90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:25.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.823+0000 7f2ebcff9640 1 --2- 172.21.15.12:0/394914798 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2ec00726b0 0x7f2ec0072a90 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2ec00a2630 tx=0x7f2ea800eba0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.829+0000 7f2ec5847640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:36:25.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.829+0000 7f2ec5847640 1 --2- 172.21.15.12:0/394914798 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2ec0072fd0 0x7f2ec01d2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:25.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.829+0000 7f2ec5847640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:25.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.829+0000 7f2ec5847640 1 -- 172.21.15.12:0/394914798 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec0121ab0 con 0x7f2ec0072fd0 2024-09-11T01:36:25.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.829+0000 7f2eb7fff640 1 --2- 172.21.15.12:0/394914798 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2ec0072fd0 0x7f2ec01d2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:25.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.829+0000 7f2eb7fff640 10 monclient: get_auth_request con 0x7f2ec0072fd0 auth_method 0 2024-09-11T01:36:25.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.829+0000 7f2eb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:25.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.829+0000 7f2eb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:25.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.829+0000 7f2eb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.829+0000 7f2eb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.829+0000 7f2eb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.829+0000 7f2eb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.829+0000 7f2eb7fff640 10 monclient(hunting): handle_auth_done global_id 74170 payload 824 2024-09-11T01:36:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.829+0000 7f2eb7fff640 1 -- 172.21.15.12:0/394914798 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2ec01228c0 con 0x7f2ec0072fd0 2024-09-11T01:36:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.832+0000 7f2eb7fff640 1 -- 172.21.15.12:0/394914798 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2ec0072fd0 msgr2=0x7f2ec01d2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.832+0000 7f2eb7fff640 1 --2- 172.21.15.12:0/394914798 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2ec0072fd0 0x7f2ec01d2e20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2ec00a2630 tx=0x7f2ea80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.832+0000 7f2ebcff9640 1 -- 172.21.15.12:0/394914798 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea800f040 con 0x7f2ec0072fd0 2024-09-11T01:36:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.832+0000 7f2ebcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.832+0000 7f2ebcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:36:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.832+0000 7f2ebcff9640 10 monclient: dump: 2024-09-11T01:36:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:25.841 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.842 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:25.842 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:25.842 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:25.842 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:25.842 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:25.842 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:25.842 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:25.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.832+0000 7f2ebcff9640 1 -- 172.21.15.12:0/394914798 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4863 (secure 0 0 0) 0x7f2ea8013610 con 0x7f2ec0072fd0 2024-09-11T01:36:25.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.832+0000 7f2ebcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:36:25.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.832+0000 7f2ebcff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:25.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.832+0000 7f2ebcff9640 1 -- 172.21.15.12:0/394914798 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2ec0072fd0 msgr2=0x7f2ec01d2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:25.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.832+0000 7f2ebcff9640 1 --2- 172.21.15.12:0/394914798 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2ec0072fd0 0x7f2ec01d2e20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2ec00a2630 tx=0x7f2ea80046c0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.832+0000 7f2ebcff9640 1 -- 172.21.15.12:0/394914798 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2ec0072fd0 msgr2=0x7f2ec01d2e20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:25.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.832+0000 7f2ebcff9640 1 --2- 172.21.15.12:0/394914798 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2ec0072fd0 0x7f2ec01d2e20 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2ec00a2630 tx=0x7f2ea80046c0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.843 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:36:25.843 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:36:25.843 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:36:25.843 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:36:25.843 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 78, 2024-09-11T01:36:25.843 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 37612, 2024-09-11T01:36:25.843 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:36:25.843 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:36:25.843 INFO:teuthology.orchestra.run.smithi012.stdout: 1 2024-09-11T01:36:25.843 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:25.844 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:36:25.844 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:25.844 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:36:25.844 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:36:25.844 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:25.844 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:25.844 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:25.844 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:25.844 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:25.844 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:25.844 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:25.844 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:25.845 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:25.845 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:25.845 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:25.845 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:25.845 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:36:25.845 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:36:25.845 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:25.845 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:25.845 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:25.845 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:25.845 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:25.845 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:25.845 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:25.846 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:25.846 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:25.846 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:25.846 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:25.846 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:25.846 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:25.846 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:36:25.846 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:36:25.846 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:36:25.846 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:36:25.846 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:36:25.846 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:36:25.846 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:25.847 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:25.847 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:36:25.847 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:36:25.847 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:36:25.847 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:36:25.847 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:36:25.847 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:36:25.847 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:36:25.847 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:25.847 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:36:25.847 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:25.847 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:25.847 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:25.847 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:25.848 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:25.848 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:25.848 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:25.848 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:25.848 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:25.848 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:25.848 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:25.848 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:25.848 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:36:25.848 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:25.848 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:36:25.848 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:25.848 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:36:25.849 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:36:25.849 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:25.849 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:25.849 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:25.849 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:25.849 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:36:25.849 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:25.849 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:25.849 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:25.849 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:25.849 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:36:25.849 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:25.849 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:25.849 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:25.850 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:25.850 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:36:25.850 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:36:25.850 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:25.850 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:25.850 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:25.850 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:25.850 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:25.850 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:36:25.850 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:36:25.850 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:25.850 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:25.850 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:25.851 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:25.851 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:36:25.851 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:25.851 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:25.851 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:25.851 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:25.851 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:36:25.851 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:25.851 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:25.851 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:25.851 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:25.851 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:36:25.851 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:36:25.851 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:25.852 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:25.852 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:25.852 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:25.852 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:25.852 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:36:25.852 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:36:25.852 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:25.852 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:25.852 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:25.852 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:25.852 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:36:25.852 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:25.852 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:25.853 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:25.853 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:25.853 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:36:25.853 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:25.853 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:25.853 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:25.853 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:25.853 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:36:25.853 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:36:25.853 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:25.854 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:25.854 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:25.854 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:25.854 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:25.854 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:25.854 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:36:25.854 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:36:25.854 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:25.854 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:25.854 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:25.854 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:36:25.854 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:25.854 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:25.855 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:36:25.855 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:25.855 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:25.855 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:25.855 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:36:25.855 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:25.855 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:25.855 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:36:25.855 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:25.855 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:25.855 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:25.855 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:36:25.855 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:25.855 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:25.856 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:36:25.856 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:25.856 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:25.856 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:25.856 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:36:25.856 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:25.856 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:25.856 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:25.856 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:36:25.856 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:36:25.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.832+0000 7f2ec5847640 1 -- 172.21.15.12:0/394914798 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2ea0042600 msgr2=0x7f2ea0044ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:25.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.832+0000 7f2ec5847640 1 --2- 172.21.15.12:0/394914798 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2ea0042600 0x7f2ea0044ac0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2ec01214b0 tx=0x7f2eb80071c0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.832+0000 7f2ec5847640 10 monclient: shutdown 2024-09-11T01:36:25.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.832+0000 7f2ec5847640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:25.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.832+0000 7f2ec5847640 1 -- 172.21.15.12:0/394914798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ec00a2200 msgr2=0x7f2ec011f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:25.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.832+0000 7f2ec5847640 1 --2- 172.21.15.12:0/394914798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ec00a2200 0x7f2ec011f100 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2eb000bb10 tx=0x7f2eb00046c0 comp rx=0 tx=0).stop 2024-09-11T01:36:25.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.832+0000 7f2ebf7fe640 1 -- 172.21.15.12:0/394914798 reap_dead start 2024-09-11T01:36:25.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.833+0000 7f2ebf7fe640 1 -- 172.21.15.12:0/394914798 reap_dead start 2024-09-11T01:36:25.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.833+0000 7f2ec5847640 1 -- 172.21.15.12:0/394914798 shutdown_connections 2024-09-11T01:36:25.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.833+0000 7f2ec5847640 1 -- 172.21.15.12:0/394914798 >> 172.21.15.12:0/394914798 conn(0x7f2ec00a2f70 msgr2=0x7f2ec009d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:25.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.833+0000 7f2ec5847640 1 -- 172.21.15.12:0/394914798 shutdown_connections 2024-09-11T01:36:25.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:25.833+0000 7f2ec5847640 1 -- 172.21.15.12:0/394914798 wait complete. 2024-09-11T01:36:25.859 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:36:25.859 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:36:26.256 INFO:tasks.rados.rados.0.smithi071.stdout:984: done (0 left) 2024-09-11T01:36:26.256 INFO:tasks.rados.rados.0.smithi071.stdout:985: snap_remove snap 86 2024-09-11T01:36:27.261 INFO:tasks.rados.rados.0.smithi071.stdout:985: done (0 left) 2024-09-11T01:36:27.262 INFO:tasks.rados.rados.0.smithi071.stdout:986: write (excl) oid 10 current snap is 104 2024-09-11T01:36:27.262 INFO:tasks.rados.rados.0.smithi071.stdout:986: seq_num 370 ranges {618590=627074,1835161=383429} 2024-09-11T01:36:27.269 INFO:tasks.rados.rados.0.smithi071.stdout:986: writing smithi07134777-10 from 618590 to 1245664 tid 1 2024-09-11T01:36:27.276 INFO:tasks.rados.rados.0.smithi071.stdout:986: writing smithi07134777-10 from 1835161 to 2218590 tid 2 2024-09-11T01:36:27.276 INFO:tasks.rados.rados.0.smithi071.stdout:987: snap_create 2024-09-11T01:36:27.280 INFO:tasks.rados.rados.0.smithi071.stdout:986: finishing write tid 1 to smithi07134777-10 2024-09-11T01:36:27.283 INFO:tasks.rados.rados.0.smithi071.stdout:986: finishing write tid 2 to smithi07134777-10 2024-09-11T01:36:27.283 INFO:tasks.rados.rados.0.smithi071.stdout:986: finishing write tid 3 to smithi07134777-10 2024-09-11T01:36:27.283 INFO:tasks.rados.rados.0.smithi071.stdout:986: finishing write tid 4 to smithi07134777-10 2024-09-11T01:36:27.284 INFO:tasks.rados.rados.0.smithi071.stdout:986: oid 10 updating version 0 to 329 2024-09-11T01:36:27.284 INFO:tasks.rados.rados.0.smithi071.stdout:986: oid 10 updating version 329 to 331 2024-09-11T01:36:27.284 INFO:tasks.rados.rados.0.smithi071.stdout:986: oid 10 version 331 is already newer than 330 2024-09-11T01:36:27.284 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 331 (ObjNum 370 snap 104 seq_num 370) dirty exists 2024-09-11T01:36:27.284 INFO:tasks.rados.rados.0.smithi071.stdout:986: left oid 10 (ObjNum 370 snap 104 seq_num 370) 2024-09-11T01:36:28.154 INFO:tasks.rados.rados.0.smithi071.stdout:986: done (1 left) 2024-09-11T01:36:28.154 INFO:tasks.rados.rados.0.smithi071.stdout:987: done (0 left) 2024-09-11T01:36:28.154 INFO:tasks.rados.rados.0.smithi071.stdout:988: snap_create 2024-09-11T01:36:29.153 INFO:tasks.rados.rados.0.smithi071.stdout:988: done (0 left) 2024-09-11T01:36:29.153 INFO:tasks.rados.rados.0.smithi071.stdout:989: snap_create 2024-09-11T01:36:30.156 INFO:tasks.rados.rados.0.smithi071.stdout:989: done (0 left) 2024-09-11T01:36:30.156 INFO:tasks.rados.rados.0.smithi071.stdout:990: rollback oid 4 current snap is 107 2024-09-11T01:36:30.156 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 4 to 97 2024-09-11T01:36:30.156 INFO:tasks.rados.rados.0.smithi071.stdout:991: read oid 20 snap -1 2024-09-11T01:36:30.156 INFO:tasks.rados.rados.0.smithi071.stdout:991: expect (ObjNum 323 snap 92 seq_num 323) 2024-09-11T01:36:30.156 INFO:tasks.rados.rados.0.smithi071.stdout:992: snap_remove snap 102 2024-09-11T01:36:31.160 INFO:tasks.rados.rados.0.smithi071.stdout:992: done (2 left) 2024-09-11T01:36:31.160 INFO:tasks.rados.rados.0.smithi071.stdout:993: write (excl) oid 1 current snap is 107 2024-09-11T01:36:31.160 INFO:tasks.rados.rados.0.smithi071.stdout:993: seq_num 371 ranges {635397=702323,1893643=341754} 2024-09-11T01:36:31.167 INFO:tasks.rados.rados.0.smithi071.stdout:993: writing smithi07134777-1 from 635397 to 1337720 tid 1 2024-09-11T01:36:31.172 INFO:tasks.rados.rados.0.smithi071.stdout:993: writing smithi07134777-1 from 1893643 to 2235397 tid 2 2024-09-11T01:36:31.172 INFO:tasks.rados.rados.0.smithi071.stdout:994: rollback oid 48 current snap is 107 2024-09-11T01:36:31.172 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 48 to 106 2024-09-11T01:36:31.173 INFO:tasks.rados.rados.0.smithi071.stdout:991: expect (ObjNum 323 snap 92 seq_num 323) 2024-09-11T01:36:31.173 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:36:31.255 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1747597 2024-09-11T01:36:31.369 INFO:tasks.rados.rados.0.smithi071.stdout:990: finishing rollback tid 0 to smithi07134777-4 2024-09-11T01:36:31.369 INFO:tasks.rados.rados.0.smithi071.stdout:990: finishing rollback tid 1 to smithi07134777-4 2024-09-11T01:36:31.369 INFO:tasks.rados.rados.0.smithi071.stdout:990: finishing rollback tid 2 to smithi07134777-4 2024-09-11T01:36:31.370 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 404 (ObjNum 337 snap 96 seq_num 337) dirty exists 2024-09-11T01:36:31.370 INFO:tasks.rados.rados.0.smithi071.stdout:993: finishing write tid 1 to smithi07134777-1 2024-09-11T01:36:31.370 INFO:tasks.rados.rados.0.smithi071.stdout:993: finishing write tid 2 to smithi07134777-1 2024-09-11T01:36:31.370 INFO:tasks.rados.rados.0.smithi071.stdout:993: finishing write tid 3 to smithi07134777-1 2024-09-11T01:36:31.370 INFO:tasks.rados.rados.0.smithi071.stdout:993: finishing write tid 4 to smithi07134777-1 2024-09-11T01:36:31.370 INFO:tasks.rados.rados.0.smithi071.stdout:993: oid 1 updating version 0 to 415 2024-09-11T01:36:31.370 INFO:tasks.rados.rados.0.smithi071.stdout:993: oid 1 version 415 is already newer than 414 2024-09-11T01:36:31.370 INFO:tasks.rados.rados.0.smithi071.stdout:993: oid 1 version 415 is already newer than 413 2024-09-11T01:36:31.370 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 415 (ObjNum 371 snap 107 seq_num 371) dirty exists 2024-09-11T01:36:31.370 INFO:tasks.rados.rados.0.smithi071.stdout:993: left oid 1 (ObjNum 371 snap 107 seq_num 371) 2024-09-11T01:36:31.370 INFO:tasks.rados.rados.0.smithi071.stdout:994: finishing rollback tid 0 to smithi07134777-48 2024-09-11T01:36:31.370 INFO:tasks.rados.rados.0.smithi071.stdout:994: finishing rollback tid 1 to smithi07134777-48 2024-09-11T01:36:31.370 INFO:tasks.rados.rados.0.smithi071.stdout:994: finishing rollback tid 2 to smithi07134777-48 2024-09-11T01:36:31.371 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 502 (ObjNum 329 snap 94 seq_num 329) dirty exists 2024-09-11T01:36:31.371 INFO:tasks.rados.rados.0.smithi071.stdout:990: done (3 left) 2024-09-11T01:36:31.371 INFO:tasks.rados.rados.0.smithi071.stdout:991: done (2 left) 2024-09-11T01:36:31.371 INFO:tasks.rados.rados.0.smithi071.stdout:993: done (1 left) 2024-09-11T01:36:31.371 INFO:tasks.rados.rados.0.smithi071.stdout:994: done (0 left) 2024-09-11T01:36:31.371 INFO:tasks.rados.rados.0.smithi071.stdout:995: write oid 32 current snap is 107 2024-09-11T01:36:31.371 INFO:tasks.rados.rados.0.smithi071.stdout:995: seq_num 372 ranges {652204=777572,1952125=300079} 2024-09-11T01:36:31.377 INFO:tasks.rados.rados.0.smithi071.stdout:995: writing smithi07134777-32 from 652204 to 1429776 tid 1 2024-09-11T01:36:31.381 INFO:tasks.rados.rados.0.smithi071.stdout:995: writing smithi07134777-32 from 1952125 to 2252204 tid 2 2024-09-11T01:36:31.381 INFO:tasks.rados.rados.0.smithi071.stdout:996: snap_create 2024-09-11T01:36:31.388 INFO:tasks.rados.rados.0.smithi071.stdout:995: finishing write tid 1 to smithi07134777-32 2024-09-11T01:36:31.390 INFO:tasks.rados.rados.0.smithi071.stdout:995: finishing write tid 2 to smithi07134777-32 2024-09-11T01:36:31.393 INFO:tasks.rados.rados.0.smithi071.stdout:995: finishing write tid 3 to smithi07134777-32 2024-09-11T01:36:31.394 INFO:tasks.rados.rados.0.smithi071.stdout:995: finishing write tid 4 to smithi07134777-32 2024-09-11T01:36:31.394 INFO:tasks.rados.rados.0.smithi071.stdout:995: oid 32 updating version 0 to 247 2024-09-11T01:36:31.394 INFO:tasks.rados.rados.0.smithi071.stdout:995: oid 32 updating version 247 to 249 2024-09-11T01:36:31.394 INFO:tasks.rados.rados.0.smithi071.stdout:995: oid 32 version 249 is already newer than 248 2024-09-11T01:36:31.394 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 249 (ObjNum 372 snap 107 seq_num 372) dirty exists 2024-09-11T01:36:31.394 INFO:tasks.rados.rados.0.smithi071.stdout:995: left oid 32 (ObjNum 372 snap 107 seq_num 372) 2024-09-11T01:36:32.175 INFO:tasks.rados.rados.0.smithi071.stdout:995: done (1 left) 2024-09-11T01:36:32.175 INFO:tasks.rados.rados.0.smithi071.stdout:996: done (0 left) 2024-09-11T01:36:32.175 INFO:tasks.rados.rados.0.smithi071.stdout:997: rollback oid 17 current snap is 108 2024-09-11T01:36:32.175 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 17 to 106 2024-09-11T01:36:32.175 INFO:tasks.rados.rados.0.smithi071.stdout:998: write (excl) oid 32 current snap is 108 2024-09-11T01:36:32.175 INFO:tasks.rados.rados.0.smithi071.stdout:998: seq_num 373 ranges {669011=569174,2010607=258404} 2024-09-11T01:36:32.181 INFO:tasks.rados.rados.0.smithi071.stdout:998: writing smithi07134777-32 from 669011 to 1238185 tid 1 2024-09-11T01:36:32.187 INFO:tasks.rados.rados.0.smithi071.stdout:998: writing smithi07134777-32 from 2010607 to 2269011 tid 2 2024-09-11T01:36:32.187 INFO:tasks.rados.rados.0.smithi071.stdout:999: delete oid 34 current snap is 108 2024-09-11T01:36:32.187 INFO:tasks.rados.rados.0.smithi071.stdout:997: finishing rollback tid 0 to smithi07134777-17 2024-09-11T01:36:32.191 INFO:tasks.rados.rados.0.smithi071.stdout:998: finishing write tid 1 to smithi07134777-32 2024-09-11T01:36:32.191 INFO:tasks.rados.rados.0.smithi071.stdout:998: finishing write tid 2 to smithi07134777-32 2024-09-11T01:36:32.191 INFO:tasks.rados.rados.0.smithi071.stdout:997: finishing rollback tid 1 to smithi07134777-17 2024-09-11T01:36:32.191 INFO:tasks.rados.rados.0.smithi071.stdout:997: finishing rollback tid 2 to smithi07134777-17 2024-09-11T01:36:32.191 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 418 (ObjNum 250 snap 63 seq_num 250) dirty exists 2024-09-11T01:36:32.193 INFO:tasks.rados.rados.0.smithi071.stdout:998: finishing write tid 3 to smithi07134777-32 2024-09-11T01:36:32.193 INFO:tasks.rados.rados.0.smithi071.stdout:997: done (2 left) 2024-09-11T01:36:32.193 INFO:tasks.rados.rados.0.smithi071.stdout:999: done (1 left) 2024-09-11T01:36:32.193 INFO:tasks.rados.rados.0.smithi071.stdout:1000: rollback oid 28 current snap is 108 2024-09-11T01:36:32.194 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 28 to 104 2024-09-11T01:36:32.194 INFO:tasks.rados.rados.0.smithi071.stdout:1001: delete oid 37 current snap is 108 2024-09-11T01:36:32.194 INFO:tasks.rados.rados.0.smithi071.stdout:998: finishing write tid 4 to smithi07134777-32 2024-09-11T01:36:32.194 INFO:tasks.rados.rados.0.smithi071.stdout:998: oid 32 updating version 0 to 251 2024-09-11T01:36:32.194 INFO:tasks.rados.rados.0.smithi071.stdout:998: oid 32 updating version 251 to 252 2024-09-11T01:36:32.194 INFO:tasks.rados.rados.0.smithi071.stdout:998: oid 32 updating version 252 to 253 2024-09-11T01:36:32.194 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 253 (ObjNum 373 snap 108 seq_num 373) dirty exists 2024-09-11T01:36:32.194 INFO:tasks.rados.rados.0.smithi071.stdout:998: left oid 32 (ObjNum 373 snap 108 seq_num 373) 2024-09-11T01:36:32.196 INFO:tasks.rados.rados.0.smithi071.stdout:998: done (2 left) 2024-09-11T01:36:32.197 INFO:tasks.rados.rados.0.smithi071.stdout:1001: done (1 left) 2024-09-11T01:36:32.197 INFO:tasks.rados.rados.0.smithi071.stdout:1002: copy_from oid 10 from oid 26 current snap is 108 2024-09-11T01:36:32.197 INFO:tasks.rados.rados.0.smithi071.stdout:1003: snap_remove snap 105 2024-09-11T01:36:33.157 INFO:tasks.rados.rados.0.smithi071.stdout:1003: done (2 left) 2024-09-11T01:36:33.157 INFO:tasks.rados.rados.0.smithi071.stdout:1004: read oid 20 snap 103 2024-09-11T01:36:33.158 INFO:tasks.rados.rados.0.smithi071.stdout:1004: expect (ObjNum 323 snap 92 seq_num 323) 2024-09-11T01:36:33.158 INFO:tasks.rados.rados.0.smithi071.stdout:1005: read oid 43 snap 107 2024-09-11T01:36:33.158 INFO:tasks.rados.rados.0.smithi071.stdout:1005: expect deleted 2024-09-11T01:36:33.158 INFO:tasks.rados.rados.0.smithi071.stdout:1000: finishing rollback tid 0 to smithi07134777-28 2024-09-11T01:36:33.158 INFO:tasks.rados.rados.0.smithi071.stdout:1000: finishing rollback tid 1 to smithi07134777-28 2024-09-11T01:36:33.158 INFO:tasks.rados.rados.0.smithi071.stdout:1000: finishing rollback tid 2 to smithi07134777-28 2024-09-11T01:36:33.159 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 360 (ObjNum 358 snap 101 seq_num 358) dirty exists 2024-09-11T01:36:33.159 INFO:tasks.rados.rados.0.smithi071.stdout:1002: finishing copy_from racing read to smithi07134777-10 2024-09-11T01:36:33.159 INFO:tasks.rados.rados.0.smithi071.stdout:1002: finishing copy_from to smithi07134777-10 2024-09-11T01:36:33.159 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 237 (ObjNum 347 snap 98 seq_num 347) dirty exists 2024-09-11T01:36:33.159 INFO:tasks.rados.rados.0.smithi071.stdout:1000: done (3 left) 2024-09-11T01:36:33.160 INFO:tasks.rados.rados.0.smithi071.stdout:1002: done (2 left) 2024-09-11T01:36:33.160 INFO:tasks.rados.rados.0.smithi071.stdout:1006: snap_remove snap 104 2024-09-11T01:36:34.156 INFO:tasks.rados.rados.0.smithi071.stdout:1006: done (2 left) 2024-09-11T01:36:34.156 INFO:tasks.rados.rados.0.smithi071.stdout:1007: copy_from oid 6 from oid 2 current snap is 108 2024-09-11T01:36:34.156 INFO:tasks.rados.rados.0.smithi071.stdout:1004: expect (ObjNum 323 snap 92 seq_num 323) 2024-09-11T01:36:34.156 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:36:34.249 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2183685 2024-09-11T01:36:34.293 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1345675 2024-09-11T01:36:34.344 INFO:tasks.rados.rados.0.smithi071.stdout:1007: finishing copy_from to smithi07134777-6 2024-09-11T01:36:34.345 INFO:tasks.rados.rados.0.smithi071.stdout:1007: got expected ENOENT (src dne) 2024-09-11T01:36:34.345 INFO:tasks.rados.rados.0.smithi071.stdout:1007: finishing copy_from racing read to smithi07134777-6 2024-09-11T01:36:34.345 INFO:tasks.rados.rados.0.smithi071.stdout:1004: done (2 left) 2024-09-11T01:36:34.345 INFO:tasks.rados.rados.0.smithi071.stdout:1005: done (1 left) 2024-09-11T01:36:34.345 INFO:tasks.rados.rados.0.smithi071.stdout:1007: done (0 left) 2024-09-11T01:36:34.345 INFO:tasks.rados.rados.0.smithi071.stdout:1008: snap_remove snap 107 2024-09-11T01:36:35.192 INFO:tasks.rados.rados.0.smithi071.stdout:1008: done (0 left) 2024-09-11T01:36:35.193 INFO:tasks.rados.rados.0.smithi071.stdout:1009: write oid 8 current snap is 108 2024-09-11T01:36:35.193 INFO:tasks.rados.rados.0.smithi071.stdout:1009: seq_num 376 ranges {719432=794921,2186053=133379} 2024-09-11T01:36:35.201 INFO:tasks.rados.rados.0.smithi071.stdout:1009: writing smithi07134777-8 from 719432 to 1514353 tid 1 2024-09-11T01:36:35.206 INFO:tasks.rados.rados.0.smithi071.stdout:1009: writing smithi07134777-8 from 2186053 to 2319432 tid 2 2024-09-11T01:36:35.206 INFO:tasks.rados.rados.0.smithi071.stdout:1010: copy_from oid 39 from oid 34 current snap is 108 2024-09-11T01:36:35.206 INFO:tasks.rados.rados.0.smithi071.stdout:1011: write oid 32 current snap is 108 2024-09-11T01:36:35.206 INFO:tasks.rados.rados.0.smithi071.stdout:1011: seq_num 377 ranges {736239=470170,1728182=571782} 2024-09-11T01:36:35.212 INFO:tasks.rados.rados.0.smithi071.stdout:1011: writing smithi07134777-32 from 736239 to 1206409 tid 1 2024-09-11T01:36:35.218 INFO:tasks.rados.rados.0.smithi071.stdout:1011: writing smithi07134777-32 from 1728182 to 2299964 tid 2 2024-09-11T01:36:35.218 INFO:tasks.rados.rados.0.smithi071.stdout:1012: write oid 3 current snap is 108 2024-09-11T01:36:35.218 INFO:tasks.rados.rados.0.smithi071.stdout:1012: seq_num 378 ranges {753046=545419,1786664=431793} 2024-09-11T01:36:35.225 INFO:tasks.rados.rados.0.smithi071.stdout:1012: writing smithi07134777-3 from 753046 to 1298465 tid 1 2024-09-11T01:36:35.231 INFO:tasks.rados.rados.0.smithi071.stdout:1012: writing smithi07134777-3 from 1786664 to 2218457 tid 2 2024-09-11T01:36:35.231 INFO:tasks.rados.rados.0.smithi071.stdout:1013: delete oid 12 current snap is 108 2024-09-11T01:36:35.231 INFO:tasks.rados.rados.0.smithi071.stdout:1010: finishing copy_from to smithi07134777-39 2024-09-11T01:36:35.231 INFO:tasks.rados.rados.0.smithi071.stdout:1010: got expected ENOENT (src dne) 2024-09-11T01:36:35.231 INFO:tasks.rados.rados.0.smithi071.stdout:1010: finishing copy_from racing read to smithi07134777-39 2024-09-11T01:36:35.231 INFO:tasks.rados.rados.0.smithi071.stdout:1009: finishing write tid 1 to smithi07134777-8 2024-09-11T01:36:35.231 INFO:tasks.rados.rados.0.smithi071.stdout:1009: finishing write tid 2 to smithi07134777-8 2024-09-11T01:36:35.232 INFO:tasks.rados.rados.0.smithi071.stdout:1009: finishing write tid 3 to smithi07134777-8 2024-09-11T01:36:35.232 INFO:tasks.rados.rados.0.smithi071.stdout:1009: finishing write tid 4 to smithi07134777-8 2024-09-11T01:36:35.232 INFO:tasks.rados.rados.0.smithi071.stdout:1009: oid 8 updating version 0 to 185 2024-09-11T01:36:35.232 INFO:tasks.rados.rados.0.smithi071.stdout:1009: oid 8 version 185 is already newer than 184 2024-09-11T01:36:35.232 INFO:tasks.rados.rados.0.smithi071.stdout:1009: oid 8 version 185 is already newer than 183 2024-09-11T01:36:35.232 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 185 (ObjNum 376 snap 108 seq_num 376) dirty exists 2024-09-11T01:36:35.232 INFO:tasks.rados.rados.0.smithi071.stdout:1009: left oid 8 (ObjNum 376 snap 108 seq_num 376) 2024-09-11T01:36:35.232 INFO:tasks.rados.rados.0.smithi071.stdout:1011: finishing write tid 1 to smithi07134777-32 2024-09-11T01:36:35.232 INFO:tasks.rados.rados.0.smithi071.stdout:1011: finishing write tid 2 to smithi07134777-32 2024-09-11T01:36:35.232 INFO:tasks.rados.rados.0.smithi071.stdout:1011: finishing write tid 3 to smithi07134777-32 2024-09-11T01:36:35.232 INFO:tasks.rados.rados.0.smithi071.stdout:1011: finishing write tid 4 to smithi07134777-32 2024-09-11T01:36:35.232 INFO:tasks.rados.rados.0.smithi071.stdout:1011: oid 32 updating version 0 to 270 2024-09-11T01:36:35.232 INFO:tasks.rados.rados.0.smithi071.stdout:1011: oid 32 version 270 is already newer than 269 2024-09-11T01:36:35.232 INFO:tasks.rados.rados.0.smithi071.stdout:1011: oid 32 version 270 is already newer than 268 2024-09-11T01:36:35.233 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 270 (ObjNum 377 snap 108 seq_num 377) dirty exists 2024-09-11T01:36:35.233 INFO:tasks.rados.rados.0.smithi071.stdout:1011: left oid 32 (ObjNum 377 snap 108 seq_num 377) 2024-09-11T01:36:35.233 INFO:tasks.rados.rados.0.smithi071.stdout:1012: finishing write tid 1 to smithi07134777-3 2024-09-11T01:36:35.234 INFO:tasks.rados.rados.0.smithi071.stdout:1009: done (4 left) 2024-09-11T01:36:35.234 INFO:tasks.rados.rados.0.smithi071.stdout:1010: done (3 left) 2024-09-11T01:36:35.234 INFO:tasks.rados.rados.0.smithi071.stdout:1011: done (2 left) 2024-09-11T01:36:35.234 INFO:tasks.rados.rados.0.smithi071.stdout:1013: done (1 left) 2024-09-11T01:36:35.235 INFO:tasks.rados.rados.0.smithi071.stdout:1014: snap_remove snap 91 2024-09-11T01:36:36.197 INFO:tasks.rados.rados.0.smithi071.stdout:1014: done (1 left) 2024-09-11T01:36:36.197 INFO:tasks.rados.rados.0.smithi071.stdout:1015: rollback oid 2 current snap is 108 2024-09-11T01:36:36.197 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 2 to 106 2024-09-11T01:36:36.197 INFO:tasks.rados.rados.0.smithi071.stdout:1012: finishing write tid 2 to smithi07134777-3 2024-09-11T01:36:36.198 INFO:tasks.rados.rados.0.smithi071.stdout:1012: finishing write tid 3 to smithi07134777-3 2024-09-11T01:36:36.198 INFO:tasks.rados.rados.0.smithi071.stdout:1012: finishing write tid 4 to smithi07134777-3 2024-09-11T01:36:36.198 INFO:tasks.rados.rados.0.smithi071.stdout:1012: oid 3 updating version 0 to 240 2024-09-11T01:36:36.198 INFO:tasks.rados.rados.0.smithi071.stdout:1012: oid 3 version 240 is already newer than 239 2024-09-11T01:36:36.198 INFO:tasks.rados.rados.0.smithi071.stdout:1012: oid 3 version 240 is already newer than 238 2024-09-11T01:36:36.198 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 240 (ObjNum 378 snap 108 seq_num 378) dirty exists 2024-09-11T01:36:36.198 INFO:tasks.rados.rados.0.smithi071.stdout:1012: left oid 3 (ObjNum 378 snap 108 seq_num 378) 2024-09-11T01:36:36.198 INFO:tasks.rados.rados.0.smithi071.stdout:1012: done (1 left) 2024-09-11T01:36:36.198 INFO:tasks.rados.rados.0.smithi071.stdout:1016: snap_create 2024-09-11T01:36:36.208 INFO:tasks.rados.rados.0.smithi071.stdout:1015: finishing rollback tid 1 to smithi07134777-2 2024-09-11T01:36:36.209 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 148 (ObjNum 177 snap 0 seq_num 2370497776) dirty dne 2024-09-11T01:36:37.202 INFO:tasks.rados.rados.0.smithi071.stdout:1015: done (1 left) 2024-09-11T01:36:37.203 INFO:tasks.rados.rados.0.smithi071.stdout:1016: done (0 left) 2024-09-11T01:36:37.203 INFO:tasks.rados.rados.0.smithi071.stdout:1017: snap_create 2024-09-11T01:36:38.154 INFO:tasks.rados.rados.0.smithi071.stdout:1017: done (0 left) 2024-09-11T01:36:38.154 INFO:tasks.rados.rados.0.smithi071.stdout:1018: write oid 19 current snap is 110 2024-09-11T01:36:38.154 INFO:tasks.rados.rados.0.smithi071.stdout:1018: seq_num 380 ranges {786660=695917,1903628=435462} 2024-09-11T01:36:38.161 INFO:tasks.rados.rados.0.smithi071.stdout:1018: writing smithi07134777-19 from 786660 to 1482577 tid 1 2024-09-11T01:36:38.166 INFO:tasks.rados.rados.0.smithi071.stdout:1018: writing smithi07134777-19 from 1903628 to 2339090 tid 2 2024-09-11T01:36:38.166 INFO:tasks.rados.rados.0.smithi071.stdout:1019: read oid 37 snap 109 2024-09-11T01:36:38.167 INFO:tasks.rados.rados.0.smithi071.stdout:1019: expect deleted 2024-09-11T01:36:38.167 INFO:tasks.rados.rados.0.smithi071.stdout:1020: write (excl) oid 11 current snap is 110 2024-09-11T01:36:38.167 INFO:tasks.rados.rados.0.smithi071.stdout:1020: seq_num 381 ranges {403467=487519,1562110=579120} 2024-09-11T01:36:38.171 INFO:tasks.rados.rados.0.smithi071.stdout:1020: writing smithi07134777-11 from 403467 to 890986 tid 1 2024-09-11T01:36:38.179 INFO:tasks.rados.rados.0.smithi071.stdout:1020: writing smithi07134777-11 from 1562110 to 2141230 tid 2 2024-09-11T01:36:38.179 INFO:tasks.rados.rados.0.smithi071.stdout:1021: read oid 5 snap -1 2024-09-11T01:36:38.179 INFO:tasks.rados.rados.0.smithi071.stdout:1021: expect deleted 2024-09-11T01:36:38.179 INFO:tasks.rados.rados.0.smithi071.stdout:1022: read oid 40 snap -1 2024-09-11T01:36:38.179 INFO:tasks.rados.rados.0.smithi071.stdout:1022: expect (ObjNum 341 snap 96 seq_num 341) 2024-09-11T01:36:38.180 INFO:tasks.rados.rados.0.smithi071.stdout:1023: read oid 13 snap 103 2024-09-11T01:36:38.180 INFO:tasks.rados.rados.0.smithi071.stdout:1023: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:36:38.180 INFO:tasks.rados.rados.0.smithi071.stdout:1024: read oid 27 snap -1 2024-09-11T01:36:38.180 INFO:tasks.rados.rados.0.smithi071.stdout:1024: expect (ObjNum 240 snap 58 seq_num 240) 2024-09-11T01:36:38.180 INFO:tasks.rados.rados.0.smithi071.stdout:1025: copy_from oid 34 from oid 30 current snap is 110 2024-09-11T01:36:38.180 INFO:tasks.rados.rados.0.smithi071.stdout:1018: finishing write tid 1 to smithi07134777-19 2024-09-11T01:36:38.180 INFO:tasks.rados.rados.0.smithi071.stdout:1018: finishing write tid 2 to smithi07134777-19 2024-09-11T01:36:38.180 INFO:tasks.rados.rados.0.smithi071.stdout:1018: finishing write tid 3 to smithi07134777-19 2024-09-11T01:36:38.180 INFO:tasks.rados.rados.0.smithi071.stdout:1018: finishing write tid 4 to smithi07134777-19 2024-09-11T01:36:38.180 INFO:tasks.rados.rados.0.smithi071.stdout:1018: oid 19 updating version 0 to 189 2024-09-11T01:36:38.180 INFO:tasks.rados.rados.0.smithi071.stdout:1018: oid 19 version 189 is already newer than 188 2024-09-11T01:36:38.180 INFO:tasks.rados.rados.0.smithi071.stdout:1018: oid 19 updating version 189 to 190 2024-09-11T01:36:38.180 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 190 (ObjNum 380 snap 110 seq_num 380) dirty exists 2024-09-11T01:36:38.181 INFO:tasks.rados.rados.0.smithi071.stdout:1018: left oid 19 (ObjNum 380 snap 110 seq_num 380) 2024-09-11T01:36:38.181 INFO:tasks.rados.rados.0.smithi071.stdout:1020: finishing write tid 1 to smithi07134777-11 2024-09-11T01:36:38.181 INFO:tasks.rados.rados.0.smithi071.stdout:1018: done (7 left) 2024-09-11T01:36:38.181 INFO:tasks.rados.rados.0.smithi071.stdout:1019: done (6 left) 2024-09-11T01:36:38.181 INFO:tasks.rados.rados.0.smithi071.stdout:1026: snap_create 2024-09-11T01:36:38.182 INFO:tasks.rados.rados.0.smithi071.stdout:1020: finishing write tid 2 to smithi07134777-11 2024-09-11T01:36:38.183 INFO:tasks.rados.rados.0.smithi071.stdout:1022: expect (ObjNum 341 snap 96 seq_num 341) 2024-09-11T01:36:38.289 INFO:tasks.rados.rados.0.smithi071.stdout:1020: finishing write tid 3 to smithi07134777-11 2024-09-11T01:36:38.289 INFO:tasks.rados.rados.0.smithi071.stdout:1020: finishing write tid 4 to smithi07134777-11 2024-09-11T01:36:38.290 INFO:tasks.rados.rados.0.smithi071.stdout:1020: oid 11 updating version 0 to 407 2024-09-11T01:36:38.290 INFO:tasks.rados.rados.0.smithi071.stdout:1020: oid 11 version 407 is already newer than 405 2024-09-11T01:36:38.290 INFO:tasks.rados.rados.0.smithi071.stdout:1020: oid 11 version 407 is already newer than 406 2024-09-11T01:36:38.290 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 407 (ObjNum 381 snap 110 seq_num 381) dirty exists 2024-09-11T01:36:38.290 INFO:tasks.rados.rados.0.smithi071.stdout:1020: left oid 11 (ObjNum 381 snap 110 seq_num 381) 2024-09-11T01:36:38.290 INFO:tasks.rados.rados.0.smithi071.stdout:1024: expect (ObjNum 240 snap 58 seq_num 240) 2024-09-11T01:36:38.292 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 582306 2024-09-11T01:36:38.318 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:36:38.350 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3415 2024-09-11T01:36:38.382 INFO:tasks.rados.rados.0.smithi071.stdout:1025: finishing copy_from to smithi07134777-34 2024-09-11T01:36:38.383 INFO:tasks.rados.rados.0.smithi071.stdout:1025: got expected ENOENT (src dne) 2024-09-11T01:36:38.383 INFO:tasks.rados.rados.0.smithi071.stdout:1025: finishing copy_from racing read to smithi07134777-34 2024-09-11T01:36:38.383 INFO:tasks.rados.rados.0.smithi071.stdout:1023: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-11T01:36:38.427 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:36:39.163 INFO:tasks.rados.rados.0.smithi071.stdout:1020: done (6 left) 2024-09-11T01:36:39.163 INFO:tasks.rados.rados.0.smithi071.stdout:1021: done (5 left) 2024-09-11T01:36:39.163 INFO:tasks.rados.rados.0.smithi071.stdout:1022: done (4 left) 2024-09-11T01:36:39.163 INFO:tasks.rados.rados.0.smithi071.stdout:1023: done (3 left) 2024-09-11T01:36:39.163 INFO:tasks.rados.rados.0.smithi071.stdout:1024: done (2 left) 2024-09-11T01:36:39.164 INFO:tasks.rados.rados.0.smithi071.stdout:1025: done (1 left) 2024-09-11T01:36:39.164 INFO:tasks.rados.rados.0.smithi071.stdout:1026: done (0 left) 2024-09-11T01:36:39.164 INFO:tasks.rados.rados.0.smithi071.stdout:1027: write oid 12 current snap is 111 2024-09-11T01:36:39.164 INFO:tasks.rados.rados.0.smithi071.stdout:1027: seq_num 382 ranges {420274=562768,1620592=439131} 2024-09-11T01:36:39.168 INFO:tasks.rados.rados.0.smithi071.stdout:1027: writing smithi07134777-12 from 420274 to 983042 tid 1 2024-09-11T01:36:39.174 INFO:tasks.rados.rados.0.smithi071.stdout:1027: writing smithi07134777-12 from 1620592 to 2059723 tid 2 2024-09-11T01:36:39.174 INFO:tasks.rados.rados.0.smithi071.stdout:1028: read oid 18 snap -1 2024-09-11T01:36:39.174 INFO:tasks.rados.rados.0.smithi071.stdout:1028: expect deleted 2024-09-11T01:36:39.174 INFO:tasks.rados.rados.0.smithi071.stdout:1029: write oid 3 current snap is 111 2024-09-11T01:36:39.174 INFO:tasks.rados.rados.0.smithi071.stdout:1027: finishing write tid 1 to smithi07134777-12 2024-09-11T01:36:39.174 INFO:tasks.rados.rados.0.smithi071.stdout:1029: seq_num 383 ranges {437081=638017,1679074=582789} 2024-09-11T01:36:39.180 INFO:tasks.rados.rados.0.smithi071.stdout:1029: writing smithi07134777-3 from 437081 to 1075098 tid 1 2024-09-11T01:36:39.187 INFO:tasks.rados.rados.0.smithi071.stdout:1029: writing smithi07134777-3 from 1679074 to 2261863 tid 2 2024-09-11T01:36:39.187 INFO:tasks.rados.rados.0.smithi071.stdout:1030: snap_remove snap 109 2024-09-11T01:36:40.174 INFO:tasks.rados.rados.0.smithi071.stdout:1030: done (3 left) 2024-09-11T01:36:40.174 INFO:tasks.rados.rados.0.smithi071.stdout:1031: read oid 31 snap -1 2024-09-11T01:36:40.174 INFO:tasks.rados.rados.0.smithi071.stdout:1031: expect (ObjNum 365 snap 101 seq_num 365) 2024-09-11T01:36:40.174 INFO:tasks.rados.rados.0.smithi071.stdout:1032: write oid 37 current snap is 111 2024-09-11T01:36:40.174 INFO:tasks.rados.rados.0.smithi071.stdout:1032: seq_num 384 ranges {453888=713266,1737556=442800} 2024-09-11T01:36:40.180 INFO:tasks.rados.rados.0.smithi071.stdout:1032: writing smithi07134777-37 from 453888 to 1167154 tid 1 2024-09-11T01:36:40.187 INFO:tasks.rados.rados.0.smithi071.stdout:1032: writing smithi07134777-37 from 1737556 to 2180356 tid 2 2024-09-11T01:36:40.187 INFO:tasks.rados.rados.0.smithi071.stdout:1033: rollback oid 44 current snap is 111 2024-09-11T01:36:40.187 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 44 to 98 2024-09-11T01:36:40.187 INFO:tasks.rados.rados.0.smithi071.stdout:1027: finishing write tid 2 to smithi07134777-12 2024-09-11T01:36:40.187 INFO:tasks.rados.rados.0.smithi071.stdout:1027: finishing write tid 3 to smithi07134777-12 2024-09-11T01:36:40.187 INFO:tasks.rados.rados.0.smithi071.stdout:1027: finishing write tid 4 to smithi07134777-12 2024-09-11T01:36:40.187 INFO:tasks.rados.rados.0.smithi071.stdout:1027: oid 12 updating version 0 to 403 2024-09-11T01:36:40.187 INFO:tasks.rados.rados.0.smithi071.stdout:1027: oid 12 updating version 403 to 404 2024-09-11T01:36:40.187 INFO:tasks.rados.rados.0.smithi071.stdout:1027: oid 12 updating version 404 to 405 2024-09-11T01:36:40.187 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 405 (ObjNum 382 snap 111 seq_num 382) dirty exists 2024-09-11T01:36:40.187 INFO:tasks.rados.rados.0.smithi071.stdout:1027: left oid 12 (ObjNum 382 snap 111 seq_num 382) 2024-09-11T01:36:40.187 INFO:tasks.rados.rados.0.smithi071.stdout:1029: finishing write tid 1 to smithi07134777-3 2024-09-11T01:36:40.188 INFO:tasks.rados.rados.0.smithi071.stdout:1029: finishing write tid 2 to smithi07134777-3 2024-09-11T01:36:40.188 INFO:tasks.rados.rados.0.smithi071.stdout:1029: finishing write tid 3 to smithi07134777-3 2024-09-11T01:36:40.188 INFO:tasks.rados.rados.0.smithi071.stdout:1029: finishing write tid 4 to smithi07134777-3 2024-09-11T01:36:40.188 INFO:tasks.rados.rados.0.smithi071.stdout:1029: oid 3 updating version 0 to 243 2024-09-11T01:36:40.188 INFO:tasks.rados.rados.0.smithi071.stdout:1029: oid 3 version 243 is already newer than 241 2024-09-11T01:36:40.188 INFO:tasks.rados.rados.0.smithi071.stdout:1029: oid 3 version 243 is already newer than 242 2024-09-11T01:36:40.188 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 243 (ObjNum 383 snap 111 seq_num 383) dirty exists 2024-09-11T01:36:40.188 INFO:tasks.rados.rados.0.smithi071.stdout:1029: left oid 3 (ObjNum 383 snap 111 seq_num 383) 2024-09-11T01:36:40.188 INFO:tasks.rados.rados.0.smithi071.stdout:1027: done (5 left) 2024-09-11T01:36:40.188 INFO:tasks.rados.rados.0.smithi071.stdout:1028: done (4 left) 2024-09-11T01:36:40.188 INFO:tasks.rados.rados.0.smithi071.stdout:1029: done (3 left) 2024-09-11T01:36:40.188 INFO:tasks.rados.rados.0.smithi071.stdout:1034: read oid 30 snap -1 2024-09-11T01:36:40.189 INFO:tasks.rados.rados.0.smithi071.stdout:1034: expect deleted 2024-09-11T01:36:40.189 INFO:tasks.rados.rados.0.smithi071.stdout:1035: write (excl) oid 32 current snap is 111 2024-09-11T01:36:40.189 INFO:tasks.rados.rados.0.smithi071.stdout:1035: seq_num 385 ranges {470695=788515,1679685=586458} 2024-09-11T01:36:40.193 INFO:tasks.rados.rados.0.smithi071.stdout:1035: writing smithi07134777-32 from 470695 to 1259210 tid 1 2024-09-11T01:36:40.199 INFO:tasks.rados.rados.0.smithi071.stdout:1035: writing smithi07134777-32 from 1679685 to 2266143 tid 2 2024-09-11T01:36:40.199 INFO:tasks.rados.rados.0.smithi071.stdout:1036: write oid 13 current snap is 111 2024-09-11T01:36:40.199 INFO:tasks.rados.rados.0.smithi071.stdout:1036: seq_num 386 ranges {487502=463764,1738167=446469} 2024-09-11T01:36:40.204 INFO:tasks.rados.rados.0.smithi071.stdout:1036: writing smithi07134777-13 from 487502 to 951266 tid 1 2024-09-11T01:36:40.211 INFO:tasks.rados.rados.0.smithi071.stdout:1036: writing smithi07134777-13 from 1738167 to 2184636 tid 2 2024-09-11T01:36:40.211 INFO:tasks.rados.rados.0.smithi071.stdout:1037: write oid 34 current snap is 111 2024-09-11T01:36:40.211 INFO:tasks.rados.rados.0.smithi071.stdout:1037: seq_num 387 ranges {504309=539013,1796649=590127} 2024-09-11T01:36:40.216 INFO:tasks.rados.rados.0.smithi071.stdout:1037: writing smithi07134777-34 from 504309 to 1043322 tid 1 2024-09-11T01:36:40.224 INFO:tasks.rados.rados.0.smithi071.stdout:1037: writing smithi07134777-34 from 1796649 to 2386776 tid 2 2024-09-11T01:36:40.224 INFO:tasks.rados.rados.0.smithi071.stdout:1038: rollback oid 24 current snap is 111 2024-09-11T01:36:40.224 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 24 to 103 2024-09-11T01:36:40.224 INFO:tasks.rados.rados.0.smithi071.stdout:1031: expect (ObjNum 365 snap 101 seq_num 365) 2024-09-11T01:36:40.250 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:36:40.286 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:36:40.323 INFO:tasks.rados.rados.0.smithi071.stdout:1033: finishing rollback tid 1 to smithi07134777-44 2024-09-11T01:36:40.323 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 0 (ObjNum 129 snap 0 seq_num 2370941568) dirty dne 2024-09-11T01:36:40.323 INFO:tasks.rados.rados.0.smithi071.stdout:1032: finishing write tid 1 to smithi07134777-37 2024-09-11T01:36:40.323 INFO:tasks.rados.rados.0.smithi071.stdout:1032: finishing write tid 2 to smithi07134777-37 2024-09-11T01:36:40.323 INFO:tasks.rados.rados.0.smithi071.stdout:1032: finishing write tid 3 to smithi07134777-37 2024-09-11T01:36:40.323 INFO:tasks.rados.rados.0.smithi071.stdout:1032: finishing write tid 4 to smithi07134777-37 2024-09-11T01:36:40.323 INFO:tasks.rados.rados.0.smithi071.stdout:1032: oid 37 updating version 0 to 275 2024-09-11T01:36:40.323 INFO:tasks.rados.rados.0.smithi071.stdout:1032: oid 37 version 275 is already newer than 273 2024-09-11T01:36:40.324 INFO:tasks.rados.rados.0.smithi071.stdout:1032: oid 37 version 275 is already newer than 274 2024-09-11T01:36:40.324 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 275 (ObjNum 384 snap 111 seq_num 384) dirty exists 2024-09-11T01:36:40.324 INFO:tasks.rados.rados.0.smithi071.stdout:1032: left oid 37 (ObjNum 384 snap 111 seq_num 384) 2024-09-11T01:36:40.324 INFO:tasks.rados.rados.0.smithi071.stdout:1035: finishing write tid 1 to smithi07134777-32 2024-09-11T01:36:40.324 INFO:tasks.rados.rados.0.smithi071.stdout:1031: done (7 left) 2024-09-11T01:36:40.324 INFO:tasks.rados.rados.0.smithi071.stdout:1032: done (6 left) 2024-09-11T01:36:40.324 INFO:tasks.rados.rados.0.smithi071.stdout:1033: done (5 left) 2024-09-11T01:36:40.324 INFO:tasks.rados.rados.0.smithi071.stdout:1034: done (4 left) 2024-09-11T01:36:40.324 INFO:tasks.rados.rados.0.smithi071.stdout:1039: read oid 25 snap -1 2024-09-11T01:36:40.324 INFO:tasks.rados.rados.0.smithi071.stdout:1039: expect (ObjNum 363 snap 101 seq_num 363) 2024-09-11T01:36:40.324 INFO:tasks.rados.rados.0.smithi071.stdout:1035: finishing write tid 2 to smithi07134777-32 2024-09-11T01:36:40.324 INFO:tasks.rados.rados.0.smithi071.stdout:1035: finishing write tid 3 to smithi07134777-32 2024-09-11T01:36:40.325 INFO:tasks.rados.rados.0.smithi071.stdout:1035: finishing write tid 4 to smithi07134777-32 2024-09-11T01:36:40.325 INFO:tasks.rados.rados.0.smithi071.stdout:1035: oid 32 updating version 0 to 279 2024-09-11T01:36:40.325 INFO:tasks.rados.rados.0.smithi071.stdout:1035: oid 32 version 279 is already newer than 277 2024-09-11T01:36:40.325 INFO:tasks.rados.rados.0.smithi071.stdout:1035: oid 32 version 279 is already newer than 278 2024-09-11T01:36:40.325 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 279 (ObjNum 385 snap 111 seq_num 385) dirty exists 2024-09-11T01:36:40.325 INFO:tasks.rados.rados.0.smithi071.stdout:1035: left oid 32 (ObjNum 385 snap 111 seq_num 385) 2024-09-11T01:36:40.325 INFO:tasks.rados.rados.0.smithi071.stdout:1036: finishing write tid 1 to smithi07134777-13 2024-09-11T01:36:40.325 INFO:tasks.rados.rados.0.smithi071.stdout:1036: finishing write tid 2 to smithi07134777-13 2024-09-11T01:36:40.325 INFO:tasks.rados.rados.0.smithi071.stdout:1036: finishing write tid 3 to smithi07134777-13 2024-09-11T01:36:40.325 INFO:tasks.rados.rados.0.smithi071.stdout:1040: read oid 9 snap -1 2024-09-11T01:36:40.325 INFO:tasks.rados.rados.0.smithi071.stdout:1040: expect (ObjNum 336 snap 96 seq_num 336) 2024-09-11T01:36:40.325 INFO:tasks.rados.rados.0.smithi071.stdout:1035: done (5 left) 2024-09-11T01:36:40.326 INFO:tasks.rados.rados.0.smithi071.stdout:1041: snap_create 2024-09-11T01:36:40.326 INFO:tasks.rados.rados.0.smithi071.stdout:1036: finishing write tid 4 to smithi07134777-13 2024-09-11T01:36:40.326 INFO:tasks.rados.rados.0.smithi071.stdout:1036: oid 13 updating version 0 to 420 2024-09-11T01:36:40.326 INFO:tasks.rados.rados.0.smithi071.stdout:1036: oid 13 version 420 is already newer than 419 2024-09-11T01:36:40.326 INFO:tasks.rados.rados.0.smithi071.stdout:1036: oid 13 updating version 420 to 421 2024-09-11T01:36:40.326 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 421 (ObjNum 386 snap 111 seq_num 386) dirty exists 2024-09-11T01:36:40.326 INFO:tasks.rados.rados.0.smithi071.stdout:1036: left oid 13 (ObjNum 386 snap 111 seq_num 386) 2024-09-11T01:36:40.326 INFO:tasks.rados.rados.0.smithi071.stdout:1037: finishing write tid 1 to smithi07134777-34 2024-09-11T01:36:40.326 INFO:tasks.rados.rados.0.smithi071.stdout:1037: finishing write tid 2 to smithi07134777-34 2024-09-11T01:36:40.326 INFO:tasks.rados.rados.0.smithi071.stdout:1037: finishing write tid 3 to smithi07134777-34 2024-09-11T01:36:40.326 INFO:tasks.rados.rados.0.smithi071.stdout:1037: finishing write tid 4 to smithi07134777-34 2024-09-11T01:36:40.326 INFO:tasks.rados.rados.0.smithi071.stdout:1037: oid 34 updating version 0 to 407 2024-09-11T01:36:40.326 INFO:tasks.rados.rados.0.smithi071.stdout:1037: oid 34 version 407 is already newer than 406 2024-09-11T01:36:40.326 INFO:tasks.rados.rados.0.smithi071.stdout:1037: oid 34 updating version 407 to 408 2024-09-11T01:36:40.326 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 408 (ObjNum 387 snap 111 seq_num 387) dirty exists 2024-09-11T01:36:40.327 INFO:tasks.rados.rados.0.smithi071.stdout:1037: left oid 34 (ObjNum 387 snap 111 seq_num 387) 2024-09-11T01:36:40.327 INFO:tasks.rados.rados.0.smithi071.stdout:1038: finishing rollback tid 0 to smithi07134777-24 2024-09-11T01:36:40.327 INFO:tasks.rados.rados.0.smithi071.stdout:1038: finishing rollback tid 1 to smithi07134777-24 2024-09-11T01:36:40.327 INFO:tasks.rados.rados.0.smithi071.stdout:1038: finishing rollback tid 2 to smithi07134777-24 2024-09-11T01:36:40.327 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 378 (ObjNum 276 snap 73 seq_num 276) dirty exists 2024-09-11T01:36:40.328 INFO:tasks.rados.rados.0.smithi071.stdout:1039: expect (ObjNum 363 snap 101 seq_num 363) 2024-09-11T01:36:40.442 INFO:tasks.rados.rados.0.smithi071.stdout:1040: expect (ObjNum 336 snap 96 seq_num 336) 2024-09-11T01:36:40.481 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:36:41.177 INFO:tasks.rados.rados.0.smithi071.stdout:1036: done (5 left) 2024-09-11T01:36:41.177 INFO:tasks.rados.rados.0.smithi071.stdout:1037: done (4 left) 2024-09-11T01:36:41.177 INFO:tasks.rados.rados.0.smithi071.stdout:1038: done (3 left) 2024-09-11T01:36:41.177 INFO:tasks.rados.rados.0.smithi071.stdout:1039: done (2 left) 2024-09-11T01:36:41.177 INFO:tasks.rados.rados.0.smithi071.stdout:1040: done (1 left) 2024-09-11T01:36:41.177 INFO:tasks.rados.rados.0.smithi071.stdout:1041: done (0 left) 2024-09-11T01:36:41.177 INFO:tasks.rados.rados.0.smithi071.stdout:1042: copy_from oid 37 from oid 10 current snap is 112 2024-09-11T01:36:41.177 INFO:tasks.rados.rados.0.smithi071.stdout:1043: delete oid 43 current snap is 112 2024-09-11T01:36:41.179 INFO:tasks.rados.rados.0.smithi071.stdout:1043: done (1 left) 2024-09-11T01:36:41.179 INFO:tasks.rados.rados.0.smithi071.stdout:1044: read oid 16 snap 96 2024-09-11T01:36:41.179 INFO:tasks.rados.rados.0.smithi071.stdout:1044: expect (ObjNum 226 snap 55 seq_num 226) 2024-09-11T01:36:41.179 INFO:tasks.rados.rados.0.smithi071.stdout:1045: write (excl) oid 27 current snap is 112 2024-09-11T01:36:41.179 INFO:tasks.rados.rados.0.smithi071.stdout:1045: seq_num 389 ranges {537923=405864,1513613=593796} 2024-09-11T01:36:41.184 INFO:tasks.rados.rados.0.smithi071.stdout:1045: writing smithi07134777-27 from 537923 to 943787 tid 1 2024-09-11T01:36:41.191 INFO:tasks.rados.rados.0.smithi071.stdout:1045: writing smithi07134777-27 from 1513613 to 2107409 tid 2 2024-09-11T01:36:41.191 INFO:tasks.rados.rados.0.smithi071.stdout:1046: snap_create 2024-09-11T01:36:41.191 INFO:tasks.rados.rados.0.smithi071.stdout:1044: expect (ObjNum 226 snap 55 seq_num 226) 2024-09-11T01:36:41.231 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 469181 2024-09-11T01:36:41.307 INFO:tasks.rados.rados.0.smithi071.stdout:1045: finishing write tid 1 to smithi07134777-27 2024-09-11T01:36:41.307 INFO:tasks.rados.rados.0.smithi071.stdout:1045: finishing write tid 2 to smithi07134777-27 2024-09-11T01:36:41.307 INFO:tasks.rados.rados.0.smithi071.stdout:1045: finishing write tid 3 to smithi07134777-27 2024-09-11T01:36:41.307 INFO:tasks.rados.rados.0.smithi071.stdout:1045: finishing write tid 4 to smithi07134777-27 2024-09-11T01:36:41.307 INFO:tasks.rados.rados.0.smithi071.stdout:1045: oid 27 updating version 0 to 411 2024-09-11T01:36:41.308 INFO:tasks.rados.rados.0.smithi071.stdout:1045: oid 27 updating version 411 to 413 2024-09-11T01:36:41.308 INFO:tasks.rados.rados.0.smithi071.stdout:1045: oid 27 version 413 is already newer than 412 2024-09-11T01:36:41.308 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 413 (ObjNum 389 snap 112 seq_num 389) dirty exists 2024-09-11T01:36:41.308 INFO:tasks.rados.rados.0.smithi071.stdout:1045: left oid 27 (ObjNum 389 snap 112 seq_num 389) 2024-09-11T01:36:41.308 INFO:tasks.rados.rados.0.smithi071.stdout:1042: finishing copy_from racing read to smithi07134777-37 2024-09-11T01:36:41.308 INFO:tasks.rados.rados.0.smithi071.stdout:1042: finishing copy_from to smithi07134777-37 2024-09-11T01:36:41.308 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 237 (ObjNum 347 snap 98 seq_num 347) dirty exists 2024-09-11T01:36:42.178 INFO:tasks.rados.rados.0.smithi071.stdout:1042: done (3 left) 2024-09-11T01:36:42.179 INFO:tasks.rados.rados.0.smithi071.stdout:1044: done (2 left) 2024-09-11T01:36:42.179 INFO:tasks.rados.rados.0.smithi071.stdout:1045: done (1 left) 2024-09-11T01:36:42.179 INFO:tasks.rados.rados.0.smithi071.stdout:1046: done (0 left) 2024-09-11T01:36:42.179 INFO:tasks.rados.rados.0.smithi071.stdout:1047: delete oid 48 current snap is 113 2024-09-11T01:36:42.185 INFO:tasks.rados.rados.0.smithi071.stdout:1047: done (0 left) 2024-09-11T01:36:42.185 INFO:tasks.rados.rados.0.smithi071.stdout:1048: copy_from oid 36 from oid 8 current snap is 113 2024-09-11T01:36:42.185 INFO:tasks.rados.rados.0.smithi071.stdout:1049: rollback oid 46 current snap is 113 2024-09-11T01:36:42.185 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 46 to 108 2024-09-11T01:36:42.185 INFO:tasks.rados.rados.0.smithi071.stdout:1050: snap_create 2024-09-11T01:36:42.188 INFO:tasks.rados.rados.0.smithi071.stdout:1049: finishing rollback tid 0 to smithi07134777-46 2024-09-11T01:36:42.189 INFO:tasks.rados.rados.0.smithi071.stdout:1049: finishing rollback tid 1 to smithi07134777-46 2024-09-11T01:36:42.189 INFO:tasks.rados.rados.0.smithi071.stdout:1049: finishing rollback tid 2 to smithi07134777-46 2024-09-11T01:36:42.189 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 220 (ObjNum 240 snap 58 seq_num 240) dirty exists 2024-09-11T01:36:42.203 INFO:tasks.rados.rados.0.smithi071.stdout:1048: finishing copy_from racing read to smithi07134777-36 2024-09-11T01:36:42.204 INFO:tasks.rados.rados.0.smithi071.stdout:1048: finishing copy_from to smithi07134777-36 2024-09-11T01:36:42.204 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 185 (ObjNum 376 snap 108 seq_num 376) dirty exists 2024-09-11T01:36:43.155 INFO:tasks.rados.rados.0.smithi071.stdout:1048: done (2 left) 2024-09-11T01:36:43.156 INFO:tasks.rados.rados.0.smithi071.stdout:1049: done (1 left) 2024-09-11T01:36:43.156 INFO:tasks.rados.rados.0.smithi071.stdout:1050: done (0 left) 2024-09-11T01:36:43.156 INFO:tasks.rados.rados.0.smithi071.stdout:1051: snap_remove snap 111 2024-09-11T01:36:44.159 INFO:tasks.rados.rados.0.smithi071.stdout:1051: done (0 left) 2024-09-11T01:36:44.160 INFO:tasks.rados.rados.0.smithi071.stdout:1052: delete oid 27 current snap is 114 2024-09-11T01:36:44.163 INFO:tasks.rados.rados.0.smithi071.stdout:1052: done (0 left) 2024-09-11T01:36:44.163 INFO:tasks.rados.rados.0.smithi071.stdout:1053: snap_create 2024-09-11T01:36:45.160 INFO:tasks.rados.rados.0.smithi071.stdout:1053: done (0 left) 2024-09-11T01:36:45.160 INFO:tasks.rados.rados.0.smithi071.stdout:1054: snap_remove snap 113 2024-09-11T01:36:45.860 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-11T01:36:45.860 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-11T01:36:45.860 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-11T01:36:45.863 INFO:tasks.ceph.mon.c:Started 2024-09-11T01:36:45.863 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:36:45.863 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:36:45.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.990+0000 7f88ffb10640 10 monclient: get_monmap_and_config 2024-09-11T01:36:45.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.990+0000 7f88ffb10640 10 monclient: build_initial_monmap 2024-09-11T01:36:45.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.990+0000 7f88ffb10640 10 monclient: monmap: 2024-09-11T01:36:45.993 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:45.993 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:45.993 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:45.991479+0000 2024-09-11T01:36:45.993 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:45.991479+0000 2024-09-11T01:36:45.993 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:45.993 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:45.993 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:45.993 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:45.993 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:45.993 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:45.993 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:45.993 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:45.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.991+0000 7f88ffb10640 1 Processor -- start 2024-09-11T01:36:46.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.991+0000 7f88ffb10640 1 -- start start 2024-09-11T01:36:46.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.991+0000 7f88ffb10640 10 monclient: init 2024-09-11T01:36:46.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.992+0000 7f88ffb10640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:46.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.992+0000 7f88ffb10640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-11T01:36:46.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.992+0000 7f88ffb10640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f88f80a2e90 addr v1:172.21.15.12:6789/0 2024-09-11T01:36:46.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.992+0000 7f88ffb10640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f88f80a3270 addr v1:172.21.15.71:6789/0 2024-09-11T01:36:46.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.992+0000 7f88ffb10640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88f813e4a0 0x7f88f8140890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:46.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.992+0000 7f88ffb10640 10 monclient(hunting): picked mon.noname-b con 0x7f88f813e4a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:46.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.992+0000 7f88ffb10640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88f80a46c0 con 0x7f88f80a2e90 2024-09-11T01:36:46.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.992+0000 7f88ffb10640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88f806ee50 con 0x7f88f80a3270 2024-09-11T01:36:46.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.992+0000 7f88ffb10640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:46.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.992+0000 7f88ffb10640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f88f80a4900 con 0x7f88f813e4a0 2024-09-11T01:36:46.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.992+0000 7f88ffb10640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:46.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.992+0000 7f88ffb10640 10 monclient(hunting): authenticate will time out at 1212.434300s 2024-09-11T01:36:46.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.992+0000 7f88fd8ac640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f88f80a2e90 0x7f88f806dc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35978/0 (socket says 172.21.15.12:35978) 2024-09-11T01:36:46.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.992+0000 7f88fd8ac640 1 -- 172.21.15.12:0/1613602353 learned_addr learned my addr 172.21.15.12:0/1613602353 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:36:46.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.992+0000 7f88fe0ad640 1 --2- 172.21.15.12:0/1613602353 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88f813e4a0 0x7f88f8140890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:46.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.992+0000 7f88fe0ad640 10 monclient(hunting): get_auth_request con 0x7f88f813e4a0 auth_method 0 2024-09-11T01:36:46.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.992+0000 7f88fe0ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:46.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.992+0000 7f88fe0ad640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:46.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.992+0000 7f88fe0ad640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:46.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.992+0000 7f88fe0ad640 1 -- 172.21.15.12:0/1613602353 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88f813e4a0 msgr2=0x7f88f8140890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:36:46.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.992+0000 7f88fe0ad640 1 -- 172.21.15.12:0/1613602353 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88f813e4a0 msgr2=0x7f88f8140890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:36:46.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.992+0000 7f88fe0ad640 1 --2- 172.21.15.12:0/1613602353 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88f813e4a0 0x7f88f8140890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:46.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.992+0000 7f88fe0ad640 1 --2- 172.21.15.12:0/1613602353 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88f813e4a0 0x7f88f8140890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:46.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.994+0000 7f88fd8ac640 1 -- 172.21.15.12:0/1613602353 >> v1:172.21.15.12:6789/0 conn(0x7f88f80a2e90 legacy=0x7f88f806dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:46.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.994+0000 7f88fd0ab640 1 -- 172.21.15.12:0/1613602353 >> v1:172.21.15.71:6789/0 conn(0x7f88f80a3270 legacy=0x7f88f80a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:46.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.994+0000 7f88fc8aa640 1 -- 172.21.15.12:0/1613602353 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2690682615 0 0) 0x7f88f80a46c0 con 0x7f88f80a2e90 2024-09-11T01:36:46.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.994+0000 7f88fc8aa640 10 monclient(hunting): _negotiate 2024-09-11T01:36:46.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.994+0000 7f88fc8aa640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:46.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.994+0000 7f88fc8aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:46.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.994+0000 7f88fc8aa640 10 monclient(hunting): my global_id is 54366 2024-09-11T01:36:46.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.994+0000 7f88fc8aa640 1 -- 172.21.15.12:0/1613602353 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88e4003520 con 0x7f88f80a2e90 2024-09-11T01:36:46.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.994+0000 7f88fc8aa640 1 -- 172.21.15.12:0/1613602353 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 695064067 0 0) 0x7f88f806ee50 con 0x7f88f80a3270 2024-09-11T01:36:46.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.994+0000 7f88fc8aa640 10 monclient(hunting): _negotiate 2024-09-11T01:36:46.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.994+0000 7f88fc8aa640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:46.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.994+0000 7f88fc8aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:46.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.994+0000 7f88fc8aa640 10 monclient(hunting): my global_id is 74173 2024-09-11T01:36:46.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.994+0000 7f88fc8aa640 1 -- 172.21.15.12:0/1613602353 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88f80a46c0 con 0x7f88f80a3270 2024-09-11T01:36:46.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 1 -- 172.21.15.12:0/1613602353 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 20969838 0 0) 0x7f88e4003520 con 0x7f88f80a2e90 2024-09-11T01:36:46.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 1 -- 172.21.15.12:0/1613602353 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88f813e4a0 msgr2=0x7f88f8140890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:36:46.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 1 --2- 172.21.15.12:0/1613602353 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88f813e4a0 0x7f88f8140890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:46.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 1 -- 172.21.15.12:0/1613602353 >> v1:172.21.15.71:6789/0 conn(0x7f88f80a3270 legacy=0x7f88f80a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:46.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 10 monclient: _finish_hunting 0 2024-09-11T01:36:46.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:36:46.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:46.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:36:46.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 1 -- 172.21.15.12:0/1613602353 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f806eb70 con 0x7f88f80a2e90 2024-09-11T01:36:46.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 10 monclient: _finish_auth 0 2024-09-11T01:36:46.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 10 monclient: _check_auth_tickets 2024-09-11T01:36:46.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:46.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 1 -- 172.21.15.12:0/1613602353 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f88f4001990 con 0x7f88f80a2e90 2024-09-11T01:36:46.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88ffb10640 5 monclient: authenticate success, global_id 54366 2024-09-11T01:36:46.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88ffb10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:36:46.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:46.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:36:46.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 10 monclient: dump: 2024-09-11T01:36:46.005 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:46.006 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:46.006 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:46.006 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:46.006 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:46.006 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:46.006 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:46.006 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:46.006 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:46.006 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:46.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:36:46.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:46.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 1 -- 172.21.15.12:0/1613602353 >> v1:172.21.15.12:6789/0 conn(0x7f88f80a2e90 legacy=0x7f88f806dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:46.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:36:46.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 1 --2- 172.21.15.12:0/1613602353 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88e4003c60 0x7f88e4004060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:46.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 10 monclient(hunting): picked mon.b con 0x7f88e4003c60 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:46.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 1 --2- 172.21.15.12:0/1613602353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88e4004800 0x7f88e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:46.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 10 monclient(hunting): picked mon.a con 0x7f88e4004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:46.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 1 --2- 172.21.15.12:0/1613602353 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88f813e4a0 0x7f88e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:46.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 10 monclient(hunting): picked mon.c con 0x7f88f813e4a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:46.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:46.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fd0ab640 1 --2- 172.21.15.12:0/1613602353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88e4004800 0x7f88e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:46.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 1 -- 172.21.15.12:0/1613602353 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f88f80a4900 con 0x7f88e4004800 2024-09-11T01:36:46.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:46.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 1 -- 172.21.15.12:0/1613602353 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f88e4005660 con 0x7f88e4003c60 2024-09-11T01:36:46.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:46.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 1 -- 172.21.15.12:0/1613602353 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f88e400bf50 con 0x7f88f813e4a0 2024-09-11T01:36:46.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:46.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fe0ad640 1 --2- 172.21.15.12:0/1613602353 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88f813e4a0 0x7f88e400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:46.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fd8ac640 1 --2- 172.21.15.12:0/1613602353 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88e4003c60 0x7f88e4004060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:46.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fc8aa640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:36:46.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fd0ab640 10 monclient(hunting): get_auth_request con 0x7f88e4004800 auth_method 0 2024-09-11T01:36:46.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fd0ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:46.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fd0ab640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:46.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fd0ab640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:36:46.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fe0ad640 10 monclient(hunting): get_auth_request con 0x7f88f813e4a0 auth_method 0 2024-09-11T01:36:46.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fe0ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:46.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fe0ad640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:46.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fe0ad640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:36:46.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fd8ac640 10 monclient(hunting): get_auth_request con 0x7f88e4003c60 auth_method 0 2024-09-11T01:36:46.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fd8ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:46.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fd8ac640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:46.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fd8ac640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:36:46.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88ffb10640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:36:46.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fe0ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:46.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fe0ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:46.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fe0ad640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:36:46.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fd0ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:46.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fd0ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:46.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fd0ab640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:36:46.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fd8ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:46.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fd8ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:46.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fd8ac640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:36:46.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fe0ad640 10 monclient(hunting): handle_auth_done global_id 54366 payload 293 2024-09-11T01:36:46.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fe0ad640 1 -- 172.21.15.12:0/1613602353 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88e4003c60 msgr2=0x7f88e4004060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:46.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fe0ad640 1 --2- 172.21.15.12:0/1613602353 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88e4003c60 0x7f88e4004060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:46.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fe0ad640 1 -- 172.21.15.12:0/1613602353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88e4004800 msgr2=0x7f88e4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:46.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fe0ad640 1 --2- 172.21.15.12:0/1613602353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88e4004800 0x7f88e4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:46.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fe0ad640 10 monclient: _finish_hunting 0 2024-09-11T01:36:46.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fe0ad640 1 monclient: found mon.c 2024-09-11T01:36:46.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fe0ad640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:46.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fe0ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:36:46.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fe0ad640 1 -- 172.21.15.12:0/1613602353 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88e4003520 con 0x7f88f813e4a0 2024-09-11T01:36:46.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fe0ad640 10 monclient: _finish_auth 0 2024-09-11T01:36:46.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fe0ad640 10 monclient: _check_auth_tickets 2024-09-11T01:36:46.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fe0ad640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:46.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.995+0000 7f88fd0ab640 1 --2- 172.21.15.12:0/1613602353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88e4004800 0x7f88e4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:36:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.998+0000 7f88fe0ad640 1 -- 172.21.15.12:0/1613602353 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88f813e4a0 msgr2=0x7f88e400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.998+0000 7f88fe0ad640 1 --2- 172.21.15.12:0/1613602353 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88f813e4a0 0x7f88e400b9c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f88f0009610 tx=0x7f88e4005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e79c7c4606e0faf server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.998+0000 7f88fc8aa640 1 -- 172.21.15.12:0/1613602353 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f0033070 con 0x7f88f813e4a0 2024-09-11T01:36:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.998+0000 7f88fc8aa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.998+0000 7f88fc8aa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.998+0000 7f88fc8aa640 10 monclient: dump: 2024-09-11T01:36:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:46.011 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:46.012 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.998+0000 7f88fc8aa640 1 -- 172.21.15.12:0/1613602353 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88f002f860 con 0x7f88f813e4a0 2024-09-11T01:36:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.998+0000 7f88ffb10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:36:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.998+0000 7f88fc8aa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.998+0000 7f88fc8aa640 1 -- 172.21.15.12:0/1613602353 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f0009bd0 con 0x7f88f813e4a0 2024-09-11T01:36:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.998+0000 7f88ffb10640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.998+0000 7f88ffb10640 4 set_mon_vals no callback set 2024-09-11T01:36:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.998+0000 7f88fc8aa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.998+0000 7f88fc8aa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.998+0000 7f88fc8aa640 10 monclient: dump: 2024-09-11T01:36:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:46.012 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:46.013 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.999+0000 7f88ffb10640 10 monclient: shutdown 2024-09-11T01:36:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.999+0000 7f88ffb10640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.999+0000 7f88ffb10640 1 -- 172.21.15.12:0/1613602353 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88f813e4a0 msgr2=0x7f88e400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.999+0000 7f88ffb10640 1 --2- 172.21.15.12:0/1613602353 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88f813e4a0 0x7f88e400b9c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f88f0009610 tx=0x7f88e4005830 comp rx=0 tx=0).stop 2024-09-11T01:36:46.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.999+0000 7f88fe0ad640 1 -- 172.21.15.12:0/1613602353 reap_dead start 2024-09-11T01:36:46.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.999+0000 7f88ffb10640 1 -- 172.21.15.12:0/1613602353 shutdown_connections 2024-09-11T01:36:46.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.999+0000 7f88ffb10640 1 -- 172.21.15.12:0/1613602353 >> 172.21.15.12:0/1613602353 conn(0x7f88f80a8490 msgr2=0x7f88f80a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:46.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.999+0000 7f88ffb10640 1 -- 172.21.15.12:0/1613602353 shutdown_connections 2024-09-11T01:36:46.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:45.999+0000 7f88ffb10640 1 -- 172.21.15.12:0/1613602353 wait complete. 2024-09-11T01:36:46.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.000+0000 7f88ffb10640 10 monclient: build_initial_monmap 2024-09-11T01:36:46.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.000+0000 7f88ffb10640 10 monclient: monmap: 2024-09-11T01:36:46.014 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:46.014 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:46.014 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:46.014 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:46.014 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:46.014 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:46.014 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:46.014 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:46.015 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:46.015 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:46.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.000+0000 7f88ffb10640 1 Processor -- start 2024-09-11T01:36:46.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.000+0000 7f88ffb10640 1 -- start start 2024-09-11T01:36:46.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.000+0000 7f88ffb10640 10 monclient: init 2024-09-11T01:36:46.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88ffb10640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:46.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88ffb10640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:36:46.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88ffb10640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88f81ccfe0 0x7f88f81cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:46.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88ffb10640 10 monclient(hunting): picked mon.noname-a con 0x7f88f81ccfe0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:46.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88ffb10640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88f81cd900 0x7f88f8136020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:46.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88ffb10640 10 monclient(hunting): picked mon.noname-c con 0x7f88f81cd900 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:46.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88ffb10640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88f8136560 0x7f88f81369a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:46.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88ffb10640 10 monclient(hunting): picked mon.noname-b con 0x7f88f8136560 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:46.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88ffb10640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:46.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88ffb10640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f88f813ea00 con 0x7f88f81ccfe0 2024-09-11T01:36:46.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88ffb10640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:46.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88ffb10640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f88f813e880 con 0x7f88f8136560 2024-09-11T01:36:46.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88ffb10640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:46.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88ffb10640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f88f813eb80 con 0x7f88f81cd900 2024-09-11T01:36:46.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88ffb10640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:46.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88ffb10640 10 monclient(hunting): authenticate will time out at 1212.443237s 2024-09-11T01:36:46.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd8ac640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88f81ccfe0 0x7f88f81cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:46.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd8ac640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88f81ccfe0 0x7f88f81cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45120/0 (socket says 172.21.15.12:45120) 2024-09-11T01:36:46.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd0ab640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88f81cd900 0x7f88f8136020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:46.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd8ac640 1 -- 172.21.15.12:0/547080552 learned_addr learned my addr 172.21.15.12:0/547080552 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:46.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd8ac640 10 monclient(hunting): get_auth_request con 0x7f88f81ccfe0 auth_method 0 2024-09-11T01:36:46.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd8ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:46.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd8ac640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:46.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd8ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:46.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd0ab640 10 monclient(hunting): get_auth_request con 0x7f88f81cd900 auth_method 0 2024-09-11T01:36:46.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd0ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:46.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd0ab640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:46.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd0ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:46.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd8ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:46.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd8ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:46.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd8ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:46.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd0ab640 1 -- 172.21.15.12:0/547080552 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88f81cd900 msgr2=0x7f88f8136020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:36:46.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd0ab640 1 -- 172.21.15.12:0/547080552 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88f81cd900 msgr2=0x7f88f8136020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:36:46.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd0ab640 1 --2- 172.21.15.12:0/547080552 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88f81cd900 0x7f88f8136020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:46.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd0ab640 1 --2- 172.21.15.12:0/547080552 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88f81cd900 0x7f88f8136020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:46.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fe0ad640 1 --2- 172.21.15.12:0/547080552 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88f8136560 0x7f88f81369a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:46.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd8ac640 10 monclient(hunting): handle_auth_done global_id 54369 payload 824 2024-09-11T01:36:46.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd8ac640 1 -- 172.21.15.12:0/547080552 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88f81cd900 msgr2=0x7f88f8136020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:36:46.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd8ac640 1 --2- 172.21.15.12:0/547080552 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88f81cd900 0x7f88f8136020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:46.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd8ac640 1 -- 172.21.15.12:0/547080552 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88f8136560 msgr2=0x7f88f81369a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:46.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd8ac640 1 --2- 172.21.15.12:0/547080552 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88f8136560 0x7f88f81369a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:46.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd8ac640 10 monclient: _finish_hunting 0 2024-09-11T01:36:46.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd8ac640 1 monclient: found mon.noname-a 2024-09-11T01:36:46.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd8ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:46.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd8ac640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:46.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fd8ac640 1 -- 172.21.15.12:0/547080552 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f813c560 con 0x7f88f81ccfe0 2024-09-11T01:36:46.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fe0ad640 10 monclient: get_auth_request con 0x7f88f8136560 auth_method 0 2024-09-11T01:36:46.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.001+0000 7f88fe0ad640 1 --2- 172.21.15.12:0/547080552 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88f8136560 0x7f88f81369a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:36:46.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.003+0000 7f88fd8ac640 1 -- 172.21.15.12:0/547080552 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88f81ccfe0 msgr2=0x7f88f81cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:46.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.003+0000 7f88fd8ac640 1 --2- 172.21.15.12:0/547080552 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88f81ccfe0 0x7f88f81cd3c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f88f4001b00 tx=0x7f88f4007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:46.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.003+0000 7f88ee7fc640 1 -- 172.21.15.12:0/547080552 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f400fb70 con 0x7f88f81ccfe0 2024-09-11T01:36:46.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.004+0000 7f88ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:46.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.004+0000 7f88ee7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:46.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.004+0000 7f88ee7fc640 10 monclient: dump: 2024-09-11T01:36:46.022 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:46.023 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:46.023 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:46.023 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:46.023 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:46.023 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:46.023 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:46.023 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:46.023 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:46.023 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:46.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.004+0000 7f88ee7fc640 10 monclient: _finish_auth 0 2024-09-11T01:36:46.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.004+0000 7f88ee7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:36:46.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.004+0000 7f88ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:46.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.004+0000 7f88ffb10640 5 monclient: authenticate success, global_id 54369 2024-09-11T01:36:46.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.004+0000 7f88ee7fc640 1 -- 172.21.15.12:0/547080552 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88f4018070 con 0x7f88f81ccfe0 2024-09-11T01:36:46.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.004+0000 7f88ffb10640 10 monclient: _renew_subs 2024-09-11T01:36:46.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.004+0000 7f88ffb10640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:46.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.004+0000 7f88ffb10640 1 -- 172.21.15.12:0/547080552 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88f813afa0 con 0x7f88f81ccfe0 2024-09-11T01:36:46.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.004+0000 7f88ee7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:46.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.004+0000 7f88ee7fc640 1 -- 172.21.15.12:0/547080552 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f40133f0 con 0x7f88f81ccfe0 2024-09-11T01:36:46.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.004+0000 7f88ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:46.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.004+0000 7f88ee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:46.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.004+0000 7f88ee7fc640 10 monclient: dump: 2024-09-11T01:36:46.024 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:46.024 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:46.024 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:46.024 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:46.024 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:46.025 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:46.025 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:46.025 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:46.025 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:46.025 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:46.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.004+0000 7f88effff640 4 set_mon_vals no callback set 2024-09-11T01:36:46.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.004+0000 7f88ffb10640 10 monclient: _renew_subs 2024-09-11T01:36:46.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.004+0000 7f88ffb10640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:46.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.004+0000 7f88ffb10640 1 -- 172.21.15.12:0/547080552 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88f809f680 con 0x7f88f81ccfe0 2024-09-11T01:36:46.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.005+0000 7f88ee7fc640 1 -- 172.21.15.12:0/547080552 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f88f4013590 con 0x7f88f81ccfe0 2024-09-11T01:36:46.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.005+0000 7f88ffb10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:46.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.005+0000 7f88ffb10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:46.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.005+0000 7f88ffb10640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:46.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.005+0000 7f88ffb10640 1 -- 172.21.15.12:0/547080552 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88f8074ac0 con 0x7f88f81ccfe0 2024-09-11T01:36:46.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.005+0000 7f88ee7fc640 1 --2- 172.21.15.12:0/547080552 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f88dc03e240 0x7f88dc040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:46.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.005+0000 7f88ee7fc640 1 -- 172.21.15.12:0/547080552 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(245..245 src has 186..245) ==== 4755+0+0 (secure 0 0 0) 0x7f88f4052990 con 0x7f88f81ccfe0 2024-09-11T01:36:46.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.005+0000 7f88fd0ab640 1 --2- 172.21.15.12:0/547080552 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f88dc03e240 0x7f88dc040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:46.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.005+0000 7f88fd0ab640 10 monclient: get_auth_request con 0x7f88dc03e240 auth_method 0 2024-09-11T01:36:46.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.008+0000 7f88fd0ab640 1 --2- 172.21.15.12:0/547080552 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f88dc03e240 0x7f88dc040700 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f88f80a2410 tx=0x7f88e8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:46.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.010+0000 7f88ee7fc640 1 -- 172.21.15.12:0/547080552 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f88f8074ac0 con 0x7f88f81ccfe0 2024-09-11T01:36:46.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.010+0000 7f88ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:46.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.010+0000 7f88ee7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:46.164 INFO:tasks.rados.rados.0.smithi071.stdout:1054: done (0 left) 2024-09-11T01:36:46.164 INFO:tasks.rados.rados.0.smithi071.stdout:1055: write oid 34 current snap is 115 2024-09-11T01:36:46.164 INFO:tasks.rados.rados.0.smithi071.stdout:1055: seq_num 392 ranges {588344=631611,1689059=741123} 2024-09-11T01:36:46.171 INFO:tasks.rados.rados.0.smithi071.stdout:1055: writing smithi07134777-34 from 588344 to 1219955 tid 1 2024-09-11T01:36:46.177 INFO:tasks.rados.rados.0.smithi071.stdout:1055: writing smithi07134777-34 from 1689059 to 2430182 tid 2 2024-09-11T01:36:46.178 INFO:tasks.rados.rados.0.smithi071.stdout:1056: delete oid 45 current snap is 115 2024-09-11T01:36:46.181 INFO:tasks.rados.rados.0.smithi071.stdout:1055: finishing write tid 1 to smithi07134777-34 2024-09-11T01:36:46.182 INFO:tasks.rados.rados.0.smithi071.stdout:1055: finishing write tid 2 to smithi07134777-34 2024-09-11T01:36:46.182 INFO:tasks.rados.rados.0.smithi071.stdout:1055: finishing write tid 3 to smithi07134777-34 2024-09-11T01:36:46.182 INFO:tasks.rados.rados.0.smithi071.stdout:1055: finishing write tid 4 to smithi07134777-34 2024-09-11T01:36:46.183 INFO:tasks.rados.rados.0.smithi071.stdout:1055: oid 34 updating version 0 to 419 2024-09-11T01:36:46.183 INFO:tasks.rados.rados.0.smithi071.stdout:1055: oid 34 updating version 419 to 422 2024-09-11T01:36:46.183 INFO:tasks.rados.rados.0.smithi071.stdout:1055: oid 34 updating version 422 to 423 2024-09-11T01:36:46.183 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 423 (ObjNum 392 snap 115 seq_num 392) dirty exists 2024-09-11T01:36:46.183 INFO:tasks.rados.rados.0.smithi071.stdout:1055: left oid 34 (ObjNum 392 snap 115 seq_num 392) 2024-09-11T01:36:46.183 INFO:tasks.rados.rados.0.smithi071.stdout:1055: done (1 left) 2024-09-11T01:36:46.183 INFO:tasks.rados.rados.0.smithi071.stdout:1056: done (0 left) 2024-09-11T01:36:46.184 INFO:tasks.rados.rados.0.smithi071.stdout:1057: delete oid 48 current snap is 115 2024-09-11T01:36:46.185 INFO:tasks.rados.rados.0.smithi071.stdout:1057: done (0 left) 2024-09-11T01:36:46.185 INFO:tasks.rados.rados.0.smithi071.stdout:1058: snap_remove snap 96 2024-09-11T01:36:46.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.274+0000 7f88ffb10640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:36:46.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.274+0000 7f88ffb10640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:36:46.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.274+0000 7f88ffb10640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:46.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.274+0000 7f88ffb10640 1 -- 172.21.15.12:0/547080552 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f88f806dc60 con 0x7f88f81ccfe0 2024-09-11T01:36:46.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.274+0000 7f88ee7fc640 1 -- 172.21.15.12:0/547080552 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f88f401ae10 con 0x7f88f81ccfe0 2024-09-11T01:36:46.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.274+0000 7f88ee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:36:46.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.274+0000 7f88ee7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:46.276 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":78,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:36:46.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.277+0000 7f88ffb10640 1 -- 172.21.15.12:0/547080552 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f88dc03e240 msgr2=0x7f88dc040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:46.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.277+0000 7f88ffb10640 1 --2- 172.21.15.12:0/547080552 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f88dc03e240 0x7f88dc040700 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f88f80a2410 tx=0x7f88e8009040 comp rx=0 tx=0).stop 2024-09-11T01:36:46.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.277+0000 7f88ffb10640 10 monclient: shutdown 2024-09-11T01:36:46.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.277+0000 7f88ffb10640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:46.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.277+0000 7f88ffb10640 1 -- 172.21.15.12:0/547080552 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88f81ccfe0 msgr2=0x7f88f81cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:46.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.277+0000 7f88ffb10640 1 --2- 172.21.15.12:0/547080552 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88f81ccfe0 0x7f88f81cd3c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f88f4001b00 tx=0x7f88f4007720 comp rx=0 tx=0).stop 2024-09-11T01:36:46.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.277+0000 7f88ffb10640 1 -- 172.21.15.12:0/547080552 shutdown_connections 2024-09-11T01:36:46.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.277+0000 7f88ffb10640 1 --2- 172.21.15.12:0/547080552 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88f8136560 0x7f88f81369a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:46.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.277+0000 7f88ffb10640 1 --2- 172.21.15.12:0/547080552 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88f81cd900 0x7f88f8136020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:46.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.277+0000 7f88ffb10640 1 --2- 172.21.15.12:0/547080552 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f88dc03e240 0x7f88dc040700 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:46.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.277+0000 7f88ffb10640 1 --2- 172.21.15.12:0/547080552 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88f81ccfe0 0x7f88f81cd3c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:46.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.277+0000 7f88ffb10640 1 -- 172.21.15.12:0/547080552 >> 172.21.15.12:0/547080552 conn(0x7f88f80a8490 msgr2=0x7f88f813da70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:46.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.277+0000 7f88ffb10640 1 -- 172.21.15.12:0/547080552 shutdown_connections 2024-09-11T01:36:46.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:46.277+0000 7f88ffb10640 1 -- 172.21.15.12:0/547080552 wait complete. 2024-09-11T01:36:47.172 INFO:tasks.rados.rados.0.smithi071.stdout:1058: done (0 left) 2024-09-11T01:36:47.172 INFO:tasks.rados.rados.0.smithi071.stdout:1059: write (excl) oid 48 current snap is 115 2024-09-11T01:36:47.172 INFO:tasks.rados.rados.0.smithi071.stdout:1059: seq_num 395 ranges {638765=457358,1748152=604803} 2024-09-11T01:36:47.178 INFO:tasks.rados.rados.0.smithi071.stdout:1059: writing smithi07134777-48 from 638765 to 1096123 tid 1 2024-09-11T01:36:47.186 INFO:tasks.rados.rados.0.smithi071.stdout:1059: writing smithi07134777-48 from 1748152 to 2352955 tid 2 2024-09-11T01:36:47.186 INFO:tasks.rados.rados.0.smithi071.stdout:1060: snap_remove snap 98 2024-09-11T01:36:48.160 INFO:tasks.rados.rados.0.smithi071.stdout:1060: done (1 left) 2024-09-11T01:36:48.160 INFO:tasks.rados.rados.0.smithi071.stdout:1061: copy_from oid 33 from oid 14 current snap is 115 2024-09-11T01:36:48.160 INFO:tasks.rados.rados.0.smithi071.stdout:1059: finishing write tid 1 to smithi07134777-48 2024-09-11T01:36:48.160 INFO:tasks.rados.rados.0.smithi071.stdout:1059: finishing write tid 2 to smithi07134777-48 2024-09-11T01:36:48.160 INFO:tasks.rados.rados.0.smithi071.stdout:1059: finishing write tid 3 to smithi07134777-48 2024-09-11T01:36:48.160 INFO:tasks.rados.rados.0.smithi071.stdout:1059: finishing write tid 4 to smithi07134777-48 2024-09-11T01:36:48.160 INFO:tasks.rados.rados.0.smithi071.stdout:1059: oid 48 updating version 0 to 526 2024-09-11T01:36:48.160 INFO:tasks.rados.rados.0.smithi071.stdout:1059: oid 48 updating version 526 to 531 2024-09-11T01:36:48.160 INFO:tasks.rados.rados.0.smithi071.stdout:1059: oid 48 updating version 531 to 532 2024-09-11T01:36:48.160 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 532 (ObjNum 395 snap 115 seq_num 395) dirty exists 2024-09-11T01:36:48.161 INFO:tasks.rados.rados.0.smithi071.stdout:1059: left oid 48 (ObjNum 395 snap 115 seq_num 395) 2024-09-11T01:36:48.161 INFO:tasks.rados.rados.0.smithi071.stdout:1059: done (1 left) 2024-09-11T01:36:48.161 INFO:tasks.rados.rados.0.smithi071.stdout:1062: read oid 35 snap -1 2024-09-11T01:36:48.161 INFO:tasks.rados.rados.0.smithi071.stdout:1062: expect (ObjNum 312 snap 87 seq_num 312) 2024-09-11T01:36:48.161 INFO:tasks.rados.rados.0.smithi071.stdout:1063: delete oid 46 current snap is 115 2024-09-11T01:36:48.169 INFO:tasks.rados.rados.0.smithi071.stdout:1062: expect (ObjNum 312 snap 87 seq_num 312) 2024-09-11T01:36:48.251 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2344764 2024-09-11T01:36:48.264 INFO:tasks.rados.rados.0.smithi071.stdout:1061: finishing copy_from racing read to smithi07134777-33 2024-09-11T01:36:48.264 INFO:tasks.rados.rados.0.smithi071.stdout:1061: finishing copy_from to smithi07134777-33 2024-09-11T01:36:48.264 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 464 (ObjNum 334 snap 96 seq_num 334) dirty exists 2024-09-11T01:36:48.264 INFO:tasks.rados.rados.0.smithi071.stdout:1061: done (2 left) 2024-09-11T01:36:48.265 INFO:tasks.rados.rados.0.smithi071.stdout:1062: done (1 left) 2024-09-11T01:36:48.265 INFO:tasks.rados.rados.0.smithi071.stdout:1063: done (0 left) 2024-09-11T01:36:48.265 INFO:tasks.rados.rados.0.smithi071.stdout:1064: write (excl) oid 21 current snap is 115 2024-09-11T01:36:48.265 INFO:tasks.rados.rados.0.smithi071.stdout:1064: seq_num 397 ranges {672379=724209,1865116=608472} 2024-09-11T01:36:48.271 INFO:tasks.rados.rados.0.smithi071.stdout:1064: writing smithi07134777-21 from 672379 to 1396588 tid 1 2024-09-11T01:36:48.278 INFO:tasks.rados.rados.0.smithi071.stdout:1064: writing smithi07134777-21 from 1865116 to 2473588 tid 2 2024-09-11T01:36:48.278 INFO:tasks.rados.rados.0.smithi071.stdout:1065: write oid 10 current snap is 115 2024-09-11T01:36:48.278 INFO:tasks.rados.rados.0.smithi071.stdout:1065: seq_num 398 ranges {689186=799458,1923598=752130} 2024-09-11T01:36:48.286 INFO:tasks.rados.rados.0.smithi071.stdout:1065: writing smithi07134777-10 from 689186 to 1488644 tid 1 2024-09-11T01:36:48.293 INFO:tasks.rados.rados.0.smithi071.stdout:1065: writing smithi07134777-10 from 1923598 to 2675728 tid 2 2024-09-11T01:36:48.293 INFO:tasks.rados.rados.0.smithi071.stdout:1066: read oid 44 snap 112 2024-09-11T01:36:48.293 INFO:tasks.rados.rados.0.smithi071.stdout:1066: expect deleted 2024-09-11T01:36:48.293 INFO:tasks.rados.rados.0.smithi071.stdout:1067: snap_remove snap 97 2024-09-11T01:36:49.181 INFO:tasks.rados.rados.0.smithi071.stdout:1067: done (3 left) 2024-09-11T01:36:49.181 INFO:tasks.rados.rados.0.smithi071.stdout:1068: write oid 18 current snap is 115 2024-09-11T01:36:49.181 INFO:tasks.rados.rados.0.smithi071.stdout:1068: seq_num 399 ranges {705993=474707,1582080=612141} 2024-09-11T01:36:49.187 INFO:tasks.rados.rados.0.smithi071.stdout:1068: writing smithi07134777-18 from 705993 to 1180700 tid 1 2024-09-11T01:36:49.193 INFO:tasks.rados.rados.0.smithi071.stdout:1068: writing smithi07134777-18 from 1582080 to 2194221 tid 2 2024-09-11T01:36:49.193 INFO:tasks.rados.rados.0.smithi071.stdout:1069: write (excl) oid 42 current snap is 115 2024-09-11T01:36:49.193 INFO:tasks.rados.rados.0.smithi071.stdout:1069: seq_num 400 ranges {722800=549956,2040562=682238} 2024-09-11T01:36:49.200 INFO:tasks.rados.rados.0.smithi071.stdout:1069: writing smithi07134777-42 from 722800 to 1272756 tid 1 2024-09-11T01:36:49.209 INFO:tasks.rados.rados.0.smithi071.stdout:1069: writing smithi07134777-42 from 2040562 to 2722800 tid 2 2024-09-11T01:36:49.209 INFO:tasks.rados.rados.0.smithi071.stdout:1070: write (excl) oid 12 current snap is 115 2024-09-11T01:36:49.209 INFO:tasks.rados.rados.0.smithi071.stdout:1070: seq_num 401 ranges {739607=625205,1982691=499457} 2024-09-11T01:36:49.217 INFO:tasks.rados.rados.0.smithi071.stdout:1070: writing smithi07134777-12 from 739607 to 1364812 tid 1 2024-09-11T01:36:49.223 INFO:tasks.rados.rados.0.smithi071.stdout:1070: writing smithi07134777-12 from 1982691 to 2482148 tid 2 2024-09-11T01:36:49.224 INFO:tasks.rados.rados.0.smithi071.stdout:1071: read oid 33 snap 103 2024-09-11T01:36:49.224 INFO:tasks.rados.rados.0.smithi071.stdout:1071: expect (ObjNum 352 snap 101 seq_num 352) 2024-09-11T01:36:49.224 INFO:tasks.rados.rados.0.smithi071.stdout:1072: write (excl) oid 49 current snap is 115 2024-09-11T01:36:49.224 INFO:tasks.rados.rados.0.smithi071.stdout:1064: finishing write tid 1 to smithi07134777-21 2024-09-11T01:36:49.224 INFO:tasks.rados.rados.0.smithi071.stdout:1072: seq_num 402 ranges {756414=700454,2041173=715241} 2024-09-11T01:36:49.231 INFO:tasks.rados.rados.0.smithi071.stdout:1072: writing smithi07134777-49 from 756414 to 1456868 tid 1 2024-09-11T01:36:49.240 INFO:tasks.rados.rados.0.smithi071.stdout:1072: writing smithi07134777-49 from 2041173 to 2756414 tid 2 2024-09-11T01:36:49.240 INFO:tasks.rados.rados.0.smithi071.stdout:1073: read oid 47 snap -1 2024-09-11T01:36:49.240 INFO:tasks.rados.rados.0.smithi071.stdout:1073: expect deleted 2024-09-11T01:36:49.240 INFO:tasks.rados.rados.0.smithi071.stdout:1074: read oid 5 snap -1 2024-09-11T01:36:49.240 INFO:tasks.rados.rados.0.smithi071.stdout:1074: expect deleted 2024-09-11T01:36:49.241 INFO:tasks.rados.rados.0.smithi071.stdout:1075: read oid 13 snap -1 2024-09-11T01:36:49.241 INFO:tasks.rados.rados.0.smithi071.stdout:1075: expect (ObjNum 386 snap 111 seq_num 386) 2024-09-11T01:36:49.241 INFO:tasks.rados.rados.0.smithi071.stdout:1076: rollback oid 48 current snap is 115 2024-09-11T01:36:49.241 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 48 to 114 2024-09-11T01:36:49.241 INFO:tasks.rados.rados.0.smithi071.stdout:1064: finishing write tid 2 to smithi07134777-21 2024-09-11T01:36:49.241 INFO:tasks.rados.rados.0.smithi071.stdout:1064: finishing write tid 3 to smithi07134777-21 2024-09-11T01:36:49.241 INFO:tasks.rados.rados.0.smithi071.stdout:1064: finishing write tid 4 to smithi07134777-21 2024-09-11T01:36:49.241 INFO:tasks.rados.rados.0.smithi071.stdout:1064: oid 21 updating version 0 to 402 2024-09-11T01:36:49.241 INFO:tasks.rados.rados.0.smithi071.stdout:1064: oid 21 updating version 402 to 403 2024-09-11T01:36:49.242 INFO:tasks.rados.rados.0.smithi071.stdout:1064: oid 21 updating version 403 to 404 2024-09-11T01:36:49.242 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 404 (ObjNum 397 snap 115 seq_num 397) dirty exists 2024-09-11T01:36:49.242 INFO:tasks.rados.rados.0.smithi071.stdout:1064: left oid 21 (ObjNum 397 snap 115 seq_num 397) 2024-09-11T01:36:49.242 INFO:tasks.rados.rados.0.smithi071.stdout:1065: finishing write tid 1 to smithi07134777-10 2024-09-11T01:36:49.242 INFO:tasks.rados.rados.0.smithi071.stdout:1064: done (11 left) 2024-09-11T01:36:49.242 INFO:tasks.rados.rados.0.smithi071.stdout:1077: copy_from oid 2 from oid 4 current snap is 115 2024-09-11T01:36:49.242 INFO:tasks.rados.rados.0.smithi071.stdout:1065: finishing write tid 2 to smithi07134777-10 2024-09-11T01:36:49.242 INFO:tasks.rados.rados.0.smithi071.stdout:1065: finishing write tid 3 to smithi07134777-10 2024-09-11T01:36:49.242 INFO:tasks.rados.rados.0.smithi071.stdout:1065: finishing write tid 4 to smithi07134777-10 2024-09-11T01:36:49.243 INFO:tasks.rados.rados.0.smithi071.stdout:1065: oid 10 updating version 0 to 332 2024-09-11T01:36:49.243 INFO:tasks.rados.rados.0.smithi071.stdout:1065: oid 10 updating version 332 to 334 2024-09-11T01:36:49.243 INFO:tasks.rados.rados.0.smithi071.stdout:1065: oid 10 version 334 is already newer than 333 2024-09-11T01:36:49.243 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 334 (ObjNum 398 snap 115 seq_num 398) dirty exists 2024-09-11T01:36:49.243 INFO:tasks.rados.rados.0.smithi071.stdout:1065: left oid 10 (ObjNum 398 snap 115 seq_num 398) 2024-09-11T01:36:49.243 INFO:tasks.rados.rados.0.smithi071.stdout:1068: finishing write tid 1 to smithi07134777-18 2024-09-11T01:36:49.243 INFO:tasks.rados.rados.0.smithi071.stdout:1068: finishing write tid 2 to smithi07134777-18 2024-09-11T01:36:49.243 INFO:tasks.rados.rados.0.smithi071.stdout:1065: done (11 left) 2024-09-11T01:36:49.243 INFO:tasks.rados.rados.0.smithi071.stdout:1066: done (10 left) 2024-09-11T01:36:49.243 INFO:tasks.rados.rados.0.smithi071.stdout:1078: snap_remove snap 108 2024-09-11T01:36:49.290 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:36:49.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.393+0000 7f15837da640 10 monclient: get_monmap_and_config 2024-09-11T01:36:49.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.393+0000 7f15837da640 10 monclient: build_initial_monmap 2024-09-11T01:36:49.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.394+0000 7f15837da640 10 monclient: monmap: 2024-09-11T01:36:49.396 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:49.396 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:49.396 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:49.394725+0000 2024-09-11T01:36:49.396 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:49.394725+0000 2024-09-11T01:36:49.396 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:49.396 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:49.396 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:49.396 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:49.396 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:49.396 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:49.396 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:49.397 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:49.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.394+0000 7f15837da640 1 Processor -- start 2024-09-11T01:36:49.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.394+0000 7f15837da640 1 -- start start 2024-09-11T01:36:49.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.394+0000 7f15837da640 10 monclient: init 2024-09-11T01:36:49.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f15837da640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:49.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f15837da640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-11T01:36:49.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f15837da640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f157c0a03a0 0x7f157c0a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:49.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f15837da640 10 monclient(hunting): picked mon.noname-a con 0x7f157c0a03a0 addr v2:172.21.15.12:3300/0 2024-09-11T01:36:49.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f15837da640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f157c0a11d0 addr v1:172.21.15.71:6789/0 2024-09-11T01:36:49.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f15837da640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f157c0a1600 addr v1:172.21.15.12:6789/0 2024-09-11T01:36:49.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f15837da640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f157c0a1a90 con 0x7f157c0a1600 2024-09-11T01:36:49.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f15837da640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f157c09fa00 con 0x7f157c0a11d0 2024-09-11T01:36:49.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f15837da640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:49.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f15837da640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f157c0a0780 con 0x7f157c0a03a0 2024-09-11T01:36:49.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f15837da640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:49.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f15837da640 10 monclient(hunting): authenticate will time out at 1215.837474s 2024-09-11T01:36:49.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f1581576640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f157c0a03a0 0x7f157c0a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:49.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f1581d77640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f157c0a1600 0x7f157c13c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:48586/0 (socket says 172.21.15.12:48586) 2024-09-11T01:36:49.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f1581d77640 1 -- 172.21.15.12:0/3825023781 learned_addr learned my addr 172.21.15.12:0/3825023781 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:36:49.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f1581576640 10 monclient(hunting): get_auth_request con 0x7f157c0a03a0 auth_method 0 2024-09-11T01:36:49.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f1581576640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:49.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f1581576640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:49.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f1581576640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:49.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f1581576640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:49.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f1581576640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:49.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f1581576640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:49.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f1581576640 10 monclient(hunting): handle_auth_done global_id 54372 payload 274 2024-09-11T01:36:49.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f1581576640 1 -- 172.21.15.12:0/3825023781 >> v1:172.21.15.71:6789/0 conn(0x7f157c0a11d0 legacy=0x7f157c138a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:49.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f1581576640 1 -- 172.21.15.12:0/3825023781 >> v1:172.21.15.12:6789/0 conn(0x7f157c0a1600 legacy=0x7f157c13c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:49.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f1581576640 10 monclient: _finish_hunting 0 2024-09-11T01:36:49.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f1581576640 1 monclient: found mon.noname-a 2024-09-11T01:36:49.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f1581576640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:49.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f1581576640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:49.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.395+0000 7f1581576640 1 -- 172.21.15.12:0/3825023781 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f157c0a29b0 con 0x7f157c0a03a0 2024-09-11T01:36:49.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.398+0000 7f1581576640 1 -- 172.21.15.12:0/3825023781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f157c0a03a0 msgr2=0x7f157c0a0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:49.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.398+0000 7f1581576640 1 --2- 172.21.15.12:0/3825023781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f157c0a03a0 0x7f157c0a0c90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f15780097a0 tx=0x7f1578029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42a263285baf6c82 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:49.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.399+0000 7f1573fff640 1 -- 172.21.15.12:0/3825023781 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1578037070 con 0x7f157c0a03a0 2024-09-11T01:36:49.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.399+0000 7f1573fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:49.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.399+0000 7f1573fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:49.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.399+0000 7f1573fff640 10 monclient: dump: 2024-09-11T01:36:49.411 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:49.411 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:49.411 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:49.411 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:49.411 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:49.411 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:49.411 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:49.411 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:49.412 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:49.412 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:49.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.399+0000 7f1573fff640 10 monclient: _finish_auth 0 2024-09-11T01:36:49.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.399+0000 7f1573fff640 10 monclient: _check_auth_tickets 2024-09-11T01:36:49.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.399+0000 7f1573fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:49.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.399+0000 7f1573fff640 1 -- 172.21.15.12:0/3825023781 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1578032640 con 0x7f157c0a03a0 2024-09-11T01:36:49.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.399+0000 7f15837da640 5 monclient: authenticate success, global_id 54372 2024-09-11T01:36:49.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.399+0000 7f15837da640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:36:49.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.399+0000 7f1573fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:49.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.399+0000 7f1573fff640 1 -- 172.21.15.12:0/3825023781 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1578032a10 con 0x7f157c0a03a0 2024-09-11T01:36:49.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.399+0000 7f15837da640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:49.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.399+0000 7f15837da640 4 set_mon_vals no callback set 2024-09-11T01:36:49.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.399+0000 7f1573fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:49.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.399+0000 7f1573fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:49.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.399+0000 7f1573fff640 10 monclient: dump: 2024-09-11T01:36:49.413 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:49.413 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:49.413 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:49.413 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:49.413 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:49.414 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:49.414 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:49.414 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:49.414 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:49.415 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:49.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.399+0000 7f15837da640 10 monclient: shutdown 2024-09-11T01:36:49.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.399+0000 7f15837da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:49.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.399+0000 7f15837da640 1 -- 172.21.15.12:0/3825023781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f157c0a03a0 msgr2=0x7f157c0a0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:49.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.399+0000 7f15837da640 1 --2- 172.21.15.12:0/3825023781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f157c0a03a0 0x7f157c0a0c90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f15780097a0 tx=0x7f1578029fd0 comp rx=0 tx=0).stop 2024-09-11T01:36:49.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.399+0000 7f15837da640 1 -- 172.21.15.12:0/3825023781 shutdown_connections 2024-09-11T01:36:49.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.399+0000 7f15837da640 1 --2- 172.21.15.12:0/3825023781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f157c0a03a0 0x7f157c0a0c90 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:49.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.399+0000 7f15837da640 1 -- 172.21.15.12:0/3825023781 >> 172.21.15.12:0/3825023781 conn(0x7f157c0a2d90 msgr2=0x7f157c0a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:49.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.399+0000 7f15837da640 1 -- 172.21.15.12:0/3825023781 shutdown_connections 2024-09-11T01:36:49.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.399+0000 7f15837da640 1 -- 172.21.15.12:0/3825023781 wait complete. 2024-09-11T01:36:49.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.400+0000 7f15837da640 10 monclient: build_initial_monmap 2024-09-11T01:36:49.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.400+0000 7f15837da640 10 monclient: monmap: 2024-09-11T01:36:49.416 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:49.416 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:49.416 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:49.416 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:49.416 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:49.416 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:49.416 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:49.416 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:49.417 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:49.417 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:49.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.400+0000 7f15837da640 1 Processor -- start 2024-09-11T01:36:49.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.400+0000 7f15837da640 1 -- start start 2024-09-11T01:36:49.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.400+0000 7f15837da640 10 monclient: init 2024-09-11T01:36:49.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f15837da640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:49.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f15837da640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:36:49.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f15837da640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f157c0a03a0 0x7f157c1c90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:49.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f15837da640 10 monclient(hunting): picked mon.noname-c con 0x7f157c0a03a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:49.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f15837da640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f157c0a11d0 0x7f157c1a8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:49.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f15837da640 10 monclient(hunting): picked mon.noname-a con 0x7f157c0a11d0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:49.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f15837da640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f157c0a1600 0x7f157c1c2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:49.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f15837da640 10 monclient(hunting): picked mon.noname-b con 0x7f157c0a1600 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:49.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f15837da640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:49.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f15837da640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f157c09f1d0 con 0x7f157c0a11d0 2024-09-11T01:36:49.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f15837da640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:49.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f15837da640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f157c0a26f0 con 0x7f157c0a1600 2024-09-11T01:36:49.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f15837da640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:49.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f15837da640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f157c09f350 con 0x7f157c0a03a0 2024-09-11T01:36:49.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f15837da640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:49.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f15837da640 10 monclient(hunting): authenticate will time out at 1215.843387s 2024-09-11T01:36:49.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1581576640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f157c0a03a0 0x7f157c1c90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:49.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1580d75640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f157c0a11d0 0x7f157c1a8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:49.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1581576640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f157c0a03a0 0x7f157c1c90f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52806/0 (socket says 172.21.15.12:52806) 2024-09-11T01:36:49.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1580d75640 10 monclient(hunting): get_auth_request con 0x7f157c0a11d0 auth_method 0 2024-09-11T01:36:49.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1581576640 1 -- 172.21.15.12:0/765543702 learned_addr learned my addr 172.21.15.12:0/765543702 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:49.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1580d75640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:49.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1580d75640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:49.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1580d75640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:49.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1581576640 10 monclient(hunting): get_auth_request con 0x7f157c0a03a0 auth_method 0 2024-09-11T01:36:49.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1581576640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:49.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1581576640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:49.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1581576640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:49.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1581d77640 1 --2- 172.21.15.12:0/765543702 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f157c0a1600 0x7f157c1c2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:49.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1581d77640 10 monclient(hunting): get_auth_request con 0x7f157c0a1600 auth_method 0 2024-09-11T01:36:49.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1581d77640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:49.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1581d77640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:49.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1581d77640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:49.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1580d75640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:49.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1580d75640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:49.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1580d75640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:49.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1581576640 1 -- 172.21.15.12:0/765543702 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f157c0a03a0 msgr2=0x7f157c1c90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:36:49.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1581576640 1 -- 172.21.15.12:0/765543702 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f157c0a03a0 msgr2=0x7f157c1c90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:36:49.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1581576640 1 --2- 172.21.15.12:0/765543702 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f157c0a03a0 0x7f157c1c90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:49.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1581d77640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:49.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1581d77640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:49.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1581576640 1 --2- 172.21.15.12:0/765543702 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f157c0a03a0 0x7f157c1c90f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:49.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1581d77640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:49.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1580d75640 10 monclient(hunting): handle_auth_done global_id 54375 payload 824 2024-09-11T01:36:49.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1580d75640 1 -- 172.21.15.12:0/765543702 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f157c0a03a0 msgr2=0x7f157c1c90f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:36:49.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1580d75640 1 --2- 172.21.15.12:0/765543702 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f157c0a03a0 0x7f157c1c90f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:49.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1580d75640 1 -- 172.21.15.12:0/765543702 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f157c0a1600 msgr2=0x7f157c1c2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:49.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1580d75640 1 --2- 172.21.15.12:0/765543702 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f157c0a1600 0x7f157c1c2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:49.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1580d75640 10 monclient: _finish_hunting 0 2024-09-11T01:36:49.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1580d75640 1 monclient: found mon.noname-a 2024-09-11T01:36:49.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1580d75640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:49.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1580d75640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:49.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.401+0000 7f1580d75640 1 -- 172.21.15.12:0/765543702 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f157c1a9690 con 0x7f157c0a11d0 2024-09-11T01:36:49.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.404+0000 7f1580d75640 1 -- 172.21.15.12:0/765543702 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f157c0a11d0 msgr2=0x7f157c1a8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:49.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.404+0000 7f1580d75640 1 --2- 172.21.15.12:0/765543702 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f157c0a11d0 0x7f157c1a8d00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f156c00c440 tx=0x7f156c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:49.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.404+0000 7f1571ffb640 1 -- 172.21.15.12:0/765543702 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f156c0119a0 con 0x7f157c0a11d0 2024-09-11T01:36:49.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.404+0000 7f1571ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:49.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.404+0000 7f1571ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:49.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.404+0000 7f1571ffb640 10 monclient: dump: 2024-09-11T01:36:49.423 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:49.424 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:49.424 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:49.424 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:49.424 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:49.424 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:49.424 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:49.424 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:49.424 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:49.424 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:49.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.404+0000 7f1571ffb640 10 monclient: _finish_auth 0 2024-09-11T01:36:49.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.404+0000 7f1571ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:36:49.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.404+0000 7f1571ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:49.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.404+0000 7f1571ffb640 1 -- 172.21.15.12:0/765543702 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f156c017070 con 0x7f157c0a11d0 2024-09-11T01:36:49.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.404+0000 7f15837da640 5 monclient: authenticate success, global_id 54375 2024-09-11T01:36:49.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.404+0000 7f1571ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:49.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.404+0000 7f1571ffb640 1 -- 172.21.15.12:0/765543702 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f156c011d80 con 0x7f157c0a11d0 2024-09-11T01:36:49.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.404+0000 7f1571ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:49.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.404+0000 7f1571ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:49.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.404+0000 7f1571ffb640 10 monclient: dump: 2024-09-11T01:36:49.425 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:49.425 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:49.425 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:49.425 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:49.425 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:49.425 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:49.425 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:49.426 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:49.426 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:49.426 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:49.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.404+0000 7f15737fe640 4 set_mon_vals no callback set 2024-09-11T01:36:49.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.404+0000 7f15837da640 10 monclient: _renew_subs 2024-09-11T01:36:49.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.404+0000 7f15837da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:49.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.404+0000 7f15837da640 1 -- 172.21.15.12:0/765543702 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f157c1a7b90 con 0x7f157c0a11d0 2024-09-11T01:36:49.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.404+0000 7f15837da640 10 monclient: _renew_subs 2024-09-11T01:36:49.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.404+0000 7f15837da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:49.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.404+0000 7f15837da640 1 -- 172.21.15.12:0/765543702 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f157c1a9c60 con 0x7f157c0a11d0 2024-09-11T01:36:49.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.405+0000 7f1571ffb640 1 -- 172.21.15.12:0/765543702 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f156c01c460 con 0x7f157c0a11d0 2024-09-11T01:36:49.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.405+0000 7f15837da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:49.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.405+0000 7f15837da640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:49.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.405+0000 7f15837da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:49.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.405+0000 7f15837da640 1 -- 172.21.15.12:0/765543702 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f157c074cc0 con 0x7f157c0a11d0 2024-09-11T01:36:49.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.405+0000 7f1571ffb640 1 --2- 172.21.15.12:0/765543702 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f156003e260 0x7f1560040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:49.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.405+0000 7f1571ffb640 1 -- 172.21.15.12:0/765543702 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(249..249 src has 186..249) ==== 4799+0+0 (secure 0 0 0) 0x7f156c052ac0 con 0x7f157c0a11d0 2024-09-11T01:36:49.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.405+0000 7f1581576640 1 --2- 172.21.15.12:0/765543702 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f156003e260 0x7f1560040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:49.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.405+0000 7f1581576640 10 monclient: get_auth_request con 0x7f156003e260 auth_method 0 2024-09-11T01:36:49.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.408+0000 7f1581576640 1 --2- 172.21.15.12:0/765543702 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f156003e260 0x7f1560040720 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1578009560 tx=0x7f157802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:49.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.410+0000 7f1571ffb640 1 -- 172.21.15.12:0/765543702 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f156c0514b0 con 0x7f157c0a11d0 2024-09-11T01:36:49.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.410+0000 7f1571ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:49.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.410+0000 7f1571ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:49.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.667+0000 7f15837da640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:36:49.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.667+0000 7f15837da640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:36:49.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.667+0000 7f15837da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:49.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.667+0000 7f15837da640 1 -- 172.21.15.12:0/765543702 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f157c073cb0 con 0x7f157c0a11d0 2024-09-11T01:36:49.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.667+0000 7f1571ffb640 1 -- 172.21.15.12:0/765543702 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f156c012420 con 0x7f157c0a11d0 2024-09-11T01:36:49.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.667+0000 7f1571ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:36:49.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.667+0000 7f1571ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:49.670 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":78,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:36:49.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.669+0000 7f15837da640 1 -- 172.21.15.12:0/765543702 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f156003e260 msgr2=0x7f1560040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:49.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.669+0000 7f15837da640 1 --2- 172.21.15.12:0/765543702 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f156003e260 0x7f1560040720 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1578009560 tx=0x7f157802e000 comp rx=0 tx=0).stop 2024-09-11T01:36:49.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.669+0000 7f15837da640 10 monclient: shutdown 2024-09-11T01:36:49.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.669+0000 7f15837da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:49.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.669+0000 7f15837da640 1 -- 172.21.15.12:0/765543702 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f157c0a11d0 msgr2=0x7f157c1a8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:49.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.669+0000 7f15837da640 1 --2- 172.21.15.12:0/765543702 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f157c0a11d0 0x7f157c1a8d00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f156c00c440 tx=0x7f156c00c900 comp rx=0 tx=0).stop 2024-09-11T01:36:49.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.669+0000 7f15837da640 1 -- 172.21.15.12:0/765543702 shutdown_connections 2024-09-11T01:36:49.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.669+0000 7f15837da640 1 --2- 172.21.15.12:0/765543702 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f157c0a1600 0x7f157c1c2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:49.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.669+0000 7f15837da640 1 --2- 172.21.15.12:0/765543702 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f156003e260 0x7f1560040720 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:49.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.669+0000 7f15837da640 1 --2- 172.21.15.12:0/765543702 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f157c0a11d0 0x7f157c1a8d00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:49.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.669+0000 7f15837da640 1 --2- 172.21.15.12:0/765543702 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f157c0a03a0 0x7f157c1c90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:49.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.669+0000 7f15837da640 1 -- 172.21.15.12:0/765543702 >> 172.21.15.12:0/765543702 conn(0x7f157c0a2d90 msgr2=0x7f157c06f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:49.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.669+0000 7f15837da640 1 -- 172.21.15.12:0/765543702 shutdown_connections 2024-09-11T01:36:49.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:49.669+0000 7f15837da640 1 -- 172.21.15.12:0/765543702 wait complete. 2024-09-11T01:36:50.185 INFO:tasks.rados.rados.0.smithi071.stdout:1078: done (10 left) 2024-09-11T01:36:50.185 INFO:tasks.rados.rados.0.smithi071.stdout:1079: write (excl) oid 43 current snap is 115 2024-09-11T01:36:50.185 INFO:tasks.rados.rados.0.smithi071.stdout:1068: finishing write tid 3 to smithi07134777-18 2024-09-11T01:36:50.186 INFO:tasks.rados.rados.0.smithi071.stdout:1079: seq_num 403 ranges {773221=492056,1816008=503126} 2024-09-11T01:36:50.191 INFO:tasks.rados.rados.0.smithi071.stdout:1079: writing smithi07134777-43 from 773221 to 1265277 tid 1 2024-09-11T01:36:50.198 INFO:tasks.rados.rados.0.smithi071.stdout:1079: writing smithi07134777-43 from 1816008 to 2319134 tid 2 2024-09-11T01:36:50.198 INFO:tasks.rados.rados.0.smithi071.stdout:1080: delete oid 15 current snap is 115 2024-09-11T01:36:50.198 INFO:tasks.rados.rados.0.smithi071.stdout:1068: finishing write tid 4 to smithi07134777-18 2024-09-11T01:36:50.198 INFO:tasks.rados.rados.0.smithi071.stdout:1068: oid 18 updating version 0 to 409 2024-09-11T01:36:50.198 INFO:tasks.rados.rados.0.smithi071.stdout:1068: oid 18 version 409 is already newer than 408 2024-09-11T01:36:50.198 INFO:tasks.rados.rados.0.smithi071.stdout:1068: oid 18 updating version 409 to 410 2024-09-11T01:36:50.198 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 410 (ObjNum 399 snap 115 seq_num 399) dirty exists 2024-09-11T01:36:50.199 INFO:tasks.rados.rados.0.smithi071.stdout:1068: left oid 18 (ObjNum 399 snap 115 seq_num 399) 2024-09-11T01:36:50.199 INFO:tasks.rados.rados.0.smithi071.stdout:1069: finishing write tid 1 to smithi07134777-42 2024-09-11T01:36:50.199 INFO:tasks.rados.rados.0.smithi071.stdout:1069: finishing write tid 2 to smithi07134777-42 2024-09-11T01:36:50.199 INFO:tasks.rados.rados.0.smithi071.stdout:1069: finishing write tid 3 to smithi07134777-42 2024-09-11T01:36:50.199 INFO:tasks.rados.rados.0.smithi071.stdout:1069: finishing write tid 4 to smithi07134777-42 2024-09-11T01:36:50.199 INFO:tasks.rados.rados.0.smithi071.stdout:1069: oid 42 updating version 0 to 151 2024-09-11T01:36:50.199 INFO:tasks.rados.rados.0.smithi071.stdout:1069: oid 42 version 151 is already newer than 149 2024-09-11T01:36:50.199 INFO:tasks.rados.rados.0.smithi071.stdout:1069: oid 42 version 151 is already newer than 150 2024-09-11T01:36:50.199 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 151 (ObjNum 400 snap 115 seq_num 400) dirty exists 2024-09-11T01:36:50.199 INFO:tasks.rados.rados.0.smithi071.stdout:1069: left oid 42 (ObjNum 400 snap 115 seq_num 400) 2024-09-11T01:36:50.199 INFO:tasks.rados.rados.0.smithi071.stdout:1070: finishing write tid 1 to smithi07134777-12 2024-09-11T01:36:50.199 INFO:tasks.rados.rados.0.smithi071.stdout:1070: finishing write tid 2 to smithi07134777-12 2024-09-11T01:36:50.200 INFO:tasks.rados.rados.0.smithi071.stdout:1070: finishing write tid 3 to smithi07134777-12 2024-09-11T01:36:50.200 INFO:tasks.rados.rados.0.smithi071.stdout:1070: finishing write tid 4 to smithi07134777-12 2024-09-11T01:36:50.200 INFO:tasks.rados.rados.0.smithi071.stdout:1070: oid 12 updating version 0 to 467 2024-09-11T01:36:50.200 INFO:tasks.rados.rados.0.smithi071.stdout:1070: oid 12 version 467 is already newer than 465 2024-09-11T01:36:50.200 INFO:tasks.rados.rados.0.smithi071.stdout:1070: oid 12 version 467 is already newer than 466 2024-09-11T01:36:50.200 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 467 (ObjNum 401 snap 115 seq_num 401) dirty exists 2024-09-11T01:36:50.200 INFO:tasks.rados.rados.0.smithi071.stdout:1070: left oid 12 (ObjNum 401 snap 115 seq_num 401) 2024-09-11T01:36:50.200 INFO:tasks.rados.rados.0.smithi071.stdout:1071: expect (ObjNum 352 snap 101 seq_num 352) 2024-09-11T01:36:50.201 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 535728 2024-09-11T01:36:50.417 INFO:tasks.rados.rados.0.smithi071.stdout:1072: finishing write tid 1 to smithi07134777-49 2024-09-11T01:36:50.418 INFO:tasks.rados.rados.0.smithi071.stdout:1076: finishing rollback tid 0 to smithi07134777-48 2024-09-11T01:36:50.418 INFO:tasks.rados.rados.0.smithi071.stdout:1076: finishing rollback tid 1 to smithi07134777-48 2024-09-11T01:36:50.418 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 562 (ObjNum 129 snap 0 seq_num 2371418944) dirty dne 2024-09-11T01:36:50.418 INFO:tasks.rados.rados.0.smithi071.stdout:1075: expect (ObjNum 386 snap 111 seq_num 386) 2024-09-11T01:36:50.490 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:36:50.527 INFO:tasks.rados.rados.0.smithi071.stdout:1072: finishing write tid 2 to smithi07134777-49 2024-09-11T01:36:50.527 INFO:tasks.rados.rados.0.smithi071.stdout:1072: finishing write tid 3 to smithi07134777-49 2024-09-11T01:36:50.527 INFO:tasks.rados.rados.0.smithi071.stdout:1072: finishing write tid 4 to smithi07134777-49 2024-09-11T01:36:50.527 INFO:tasks.rados.rados.0.smithi071.stdout:1072: oid 49 updating version 0 to 229 2024-09-11T01:36:50.528 INFO:tasks.rados.rados.0.smithi071.stdout:1072: oid 49 version 229 is already newer than 228 2024-09-11T01:36:50.528 INFO:tasks.rados.rados.0.smithi071.stdout:1072: oid 49 version 229 is already newer than 227 2024-09-11T01:36:50.528 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 229 (ObjNum 402 snap 115 seq_num 402) dirty exists 2024-09-11T01:36:50.528 INFO:tasks.rados.rados.0.smithi071.stdout:1072: left oid 49 (ObjNum 402 snap 115 seq_num 402) 2024-09-11T01:36:50.528 INFO:tasks.rados.rados.0.smithi071.stdout:1077: finishing copy_from racing read to smithi07134777-2 2024-09-11T01:36:50.528 INFO:tasks.rados.rados.0.smithi071.stdout:1077: finishing copy_from to smithi07134777-2 2024-09-11T01:36:50.528 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 404 (ObjNum 337 snap 96 seq_num 337) dirty exists 2024-09-11T01:36:50.528 INFO:tasks.rados.rados.0.smithi071.stdout:1079: finishing write tid 1 to smithi07134777-43 2024-09-11T01:36:50.528 INFO:tasks.rados.rados.0.smithi071.stdout:1079: finishing write tid 2 to smithi07134777-43 2024-09-11T01:36:50.528 INFO:tasks.rados.rados.0.smithi071.stdout:1079: finishing write tid 3 to smithi07134777-43 2024-09-11T01:36:50.528 INFO:tasks.rados.rados.0.smithi071.stdout:1068: done (11 left) 2024-09-11T01:36:50.529 INFO:tasks.rados.rados.0.smithi071.stdout:1069: done (10 left) 2024-09-11T01:36:50.529 INFO:tasks.rados.rados.0.smithi071.stdout:1070: done (9 left) 2024-09-11T01:36:50.529 INFO:tasks.rados.rados.0.smithi071.stdout:1071: done (8 left) 2024-09-11T01:36:50.529 INFO:tasks.rados.rados.0.smithi071.stdout:1072: done (7 left) 2024-09-11T01:36:50.529 INFO:tasks.rados.rados.0.smithi071.stdout:1073: done (6 left) 2024-09-11T01:36:50.529 INFO:tasks.rados.rados.0.smithi071.stdout:1074: done (5 left) 2024-09-11T01:36:50.529 INFO:tasks.rados.rados.0.smithi071.stdout:1075: done (4 left) 2024-09-11T01:36:50.529 INFO:tasks.rados.rados.0.smithi071.stdout:1076: done (3 left) 2024-09-11T01:36:50.529 INFO:tasks.rados.rados.0.smithi071.stdout:1077: done (2 left) 2024-09-11T01:36:50.529 INFO:tasks.rados.rados.0.smithi071.stdout:1080: done (1 left) 2024-09-11T01:36:50.529 INFO:tasks.rados.rados.0.smithi071.stdout:1081: rollback oid 19 current snap is 115 2024-09-11T01:36:50.529 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 19 to 110 2024-09-11T01:36:50.529 INFO:tasks.rados.rados.0.smithi071.stdout:1079: finishing write tid 4 to smithi07134777-43 2024-09-11T01:36:50.530 INFO:tasks.rados.rados.0.smithi071.stdout:1079: oid 43 updating version 0 to 468 2024-09-11T01:36:50.530 INFO:tasks.rados.rados.0.smithi071.stdout:1079: oid 43 updating version 468 to 470 2024-09-11T01:36:50.530 INFO:tasks.rados.rados.0.smithi071.stdout:1079: oid 43 version 470 is already newer than 469 2024-09-11T01:36:50.530 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 470 (ObjNum 403 snap 115 seq_num 403) dirty exists 2024-09-11T01:36:50.530 INFO:tasks.rados.rados.0.smithi071.stdout:1079: left oid 43 (ObjNum 403 snap 115 seq_num 403) 2024-09-11T01:36:50.530 INFO:tasks.rados.rados.0.smithi071.stdout:1079: done (1 left) 2024-09-11T01:36:50.530 INFO:tasks.rados.rados.0.smithi071.stdout:1082: delete oid 46 current snap is 115 2024-09-11T01:36:50.530 INFO:tasks.rados.rados.0.smithi071.stdout:1082: done (1 left) 2024-09-11T01:36:50.531 INFO:tasks.rados.rados.0.smithi071.stdout:1083: snap_remove snap 103 2024-09-11T01:36:51.190 INFO:tasks.rados.rados.0.smithi071.stdout:1083: done (1 left) 2024-09-11T01:36:51.190 INFO:tasks.rados.rados.0.smithi071.stdout:1084: write oid 46 current snap is 115 2024-09-11T01:36:51.190 INFO:tasks.rados.rados.0.smithi071.stdout:1081: finishing rollback tid 0 to smithi07134777-19 2024-09-11T01:36:51.190 INFO:tasks.rados.rados.0.smithi071.stdout:1081: finishing rollback tid 1 to smithi07134777-19 2024-09-11T01:36:51.190 INFO:tasks.rados.rados.0.smithi071.stdout:1081: finishing rollback tid 2 to smithi07134777-19 2024-09-11T01:36:51.190 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 212 (ObjNum 380 snap 110 seq_num 380) dirty exists 2024-09-11T01:36:51.190 INFO:tasks.rados.rados.0.smithi071.stdout:1084: seq_num 406 ranges {423642=717803,1875101=766806} 2024-09-11T01:36:51.196 INFO:tasks.rados.rados.0.smithi071.stdout:1084: writing smithi07134777-46 from 423642 to 1141445 tid 1 2024-09-11T01:36:51.205 INFO:tasks.rados.rados.0.smithi071.stdout:1084: writing smithi07134777-46 from 1875101 to 2641907 tid 2 2024-09-11T01:36:51.205 INFO:tasks.rados.rados.0.smithi071.stdout:1081: done (1 left) 2024-09-11T01:36:51.205 INFO:tasks.rados.rados.0.smithi071.stdout:1085: write oid 4 current snap is 115 2024-09-11T01:36:51.205 INFO:tasks.rados.rados.0.smithi071.stdout:1084: finishing write tid 1 to smithi07134777-46 2024-09-11T01:36:51.205 INFO:tasks.rados.rados.0.smithi071.stdout:1085: seq_num 407 ranges {440449=793052,1933583=510464} 2024-09-11T01:36:51.211 INFO:tasks.rados.rados.0.smithi071.stdout:1085: writing smithi07134777-4 from 440449 to 1233501 tid 1 2024-09-11T01:36:51.218 INFO:tasks.rados.rados.0.smithi071.stdout:1085: writing smithi07134777-4 from 1933583 to 2444047 tid 2 2024-09-11T01:36:51.218 INFO:tasks.rados.rados.0.smithi071.stdout:1086: write oid 47 current snap is 115 2024-09-11T01:36:51.218 INFO:tasks.rados.rados.0.smithi071.stdout:1084: finishing write tid 2 to smithi07134777-46 2024-09-11T01:36:51.218 INFO:tasks.rados.rados.0.smithi071.stdout:1084: finishing write tid 3 to smithi07134777-46 2024-09-11T01:36:51.218 INFO:tasks.rados.rados.0.smithi071.stdout:1084: finishing write tid 4 to smithi07134777-46 2024-09-11T01:36:51.219 INFO:tasks.rados.rados.0.smithi071.stdout:1084: oid 46 updating version 0 to 222 2024-09-11T01:36:51.219 INFO:tasks.rados.rados.0.smithi071.stdout:1084: oid 46 updating version 222 to 224 2024-09-11T01:36:51.219 INFO:tasks.rados.rados.0.smithi071.stdout:1084: oid 46 version 224 is already newer than 223 2024-09-11T01:36:51.219 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 224 (ObjNum 406 snap 115 seq_num 406) dirty exists 2024-09-11T01:36:51.219 INFO:tasks.rados.rados.0.smithi071.stdout:1084: left oid 46 (ObjNum 406 snap 115 seq_num 406) 2024-09-11T01:36:51.219 INFO:tasks.rados.rados.0.smithi071.stdout:1086: seq_num 408 ranges {457256=468301,1592065=770475,2854581=2675} 2024-09-11T01:36:51.223 INFO:tasks.rados.rados.0.smithi071.stdout:1086: writing smithi07134777-47 from 457256 to 925557 tid 1 2024-09-11T01:36:51.231 INFO:tasks.rados.rados.0.smithi071.stdout:1086: writing smithi07134777-47 from 1592065 to 2362540 tid 2 2024-09-11T01:36:51.234 INFO:tasks.rados.rados.0.smithi071.stdout:1086: writing smithi07134777-47 from 2854581 to 2857256 tid 3 2024-09-11T01:36:51.234 INFO:tasks.rados.rados.0.smithi071.stdout:1084: done (2 left) 2024-09-11T01:36:51.234 INFO:tasks.rados.rados.0.smithi071.stdout:1087: read oid 34 snap -1 2024-09-11T01:36:51.234 INFO:tasks.rados.rados.0.smithi071.stdout:1087: expect (ObjNum 392 snap 115 seq_num 392) 2024-09-11T01:36:51.234 INFO:tasks.rados.rados.0.smithi071.stdout:1088: snap_remove snap 101 2024-09-11T01:36:52.195 INFO:tasks.rados.rados.0.smithi071.stdout:1088: done (3 left) 2024-09-11T01:36:52.195 INFO:tasks.rados.rados.0.smithi071.stdout:1089: write (excl) oid 13 current snap is 115 2024-09-11T01:36:52.195 INFO:tasks.rados.rados.0.smithi071.stdout:1085: finishing write tid 1 to smithi07134777-4 2024-09-11T01:36:52.195 INFO:tasks.rados.rados.0.smithi071.stdout:1089: seq_num 409 ranges {474063=543550,1650547=514133,2765651=108412} 2024-09-11T01:36:52.200 INFO:tasks.rados.rados.0.smithi071.stdout:1089: writing smithi07134777-13 from 474063 to 1017613 tid 1 2024-09-11T01:36:52.207 INFO:tasks.rados.rados.0.smithi071.stdout:1089: writing smithi07134777-13 from 1650547 to 2164680 tid 2 2024-09-11T01:36:52.212 INFO:tasks.rados.rados.0.smithi071.stdout:1089: writing smithi07134777-13 from 2765651 to 2874063 tid 3 2024-09-11T01:36:52.212 INFO:tasks.rados.rados.0.smithi071.stdout:1090: delete oid 32 current snap is 115 2024-09-11T01:36:52.212 INFO:tasks.rados.rados.0.smithi071.stdout:1085: finishing write tid 2 to smithi07134777-4 2024-09-11T01:36:52.212 INFO:tasks.rados.rados.0.smithi071.stdout:1085: finishing write tid 3 to smithi07134777-4 2024-09-11T01:36:52.212 INFO:tasks.rados.rados.0.smithi071.stdout:1085: finishing write tid 4 to smithi07134777-4 2024-09-11T01:36:52.212 INFO:tasks.rados.rados.0.smithi071.stdout:1085: oid 4 updating version 0 to 413 2024-09-11T01:36:52.212 INFO:tasks.rados.rados.0.smithi071.stdout:1085: oid 4 version 413 is already newer than 411 2024-09-11T01:36:52.212 INFO:tasks.rados.rados.0.smithi071.stdout:1085: oid 4 version 413 is already newer than 412 2024-09-11T01:36:52.212 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 413 (ObjNum 407 snap 115 seq_num 407) dirty exists 2024-09-11T01:36:52.212 INFO:tasks.rados.rados.0.smithi071.stdout:1085: left oid 4 (ObjNum 407 snap 115 seq_num 407) 2024-09-11T01:36:52.212 INFO:tasks.rados.rados.0.smithi071.stdout:1086: finishing write tid 1 to smithi07134777-47 2024-09-11T01:36:52.212 INFO:tasks.rados.rados.0.smithi071.stdout:1086: finishing write tid 2 to smithi07134777-47 2024-09-11T01:36:52.213 INFO:tasks.rados.rados.0.smithi071.stdout:1086: finishing write tid 3 to smithi07134777-47 2024-09-11T01:36:52.213 INFO:tasks.rados.rados.0.smithi071.stdout:1086: finishing write tid 4 to smithi07134777-47 2024-09-11T01:36:52.213 INFO:tasks.rados.rados.0.smithi071.stdout:1086: finishing write tid 5 to smithi07134777-47 2024-09-11T01:36:52.213 INFO:tasks.rados.rados.0.smithi071.stdout:1086: oid 47 updating version 0 to 425 2024-09-11T01:36:52.213 INFO:tasks.rados.rados.0.smithi071.stdout:1086: oid 47 version 425 is already newer than 422 2024-09-11T01:36:52.213 INFO:tasks.rados.rados.0.smithi071.stdout:1086: oid 47 version 425 is already newer than 424 2024-09-11T01:36:52.213 INFO:tasks.rados.rados.0.smithi071.stdout:1086: oid 47 version 425 is already newer than 423 2024-09-11T01:36:52.213 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 425 (ObjNum 408 snap 115 seq_num 408) dirty exists 2024-09-11T01:36:52.213 INFO:tasks.rados.rados.0.smithi071.stdout:1086: left oid 47 (ObjNum 408 snap 115 seq_num 408) 2024-09-11T01:36:52.213 INFO:tasks.rados.rados.0.smithi071.stdout:1087: expect (ObjNum 392 snap 115 seq_num 392) 2024-09-11T01:36:52.242 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:36:52.287 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1500073 2024-09-11T01:36:52.313 INFO:tasks.rados.rados.0.smithi071.stdout:1089: finishing write tid 1 to smithi07134777-13 2024-09-11T01:36:52.313 INFO:tasks.rados.rados.0.smithi071.stdout:1089: finishing write tid 2 to smithi07134777-13 2024-09-11T01:36:52.313 INFO:tasks.rados.rados.0.smithi071.stdout:1089: finishing write tid 3 to smithi07134777-13 2024-09-11T01:36:52.313 INFO:tasks.rados.rados.0.smithi071.stdout:1089: finishing write tid 4 to smithi07134777-13 2024-09-11T01:36:52.313 INFO:tasks.rados.rados.0.smithi071.stdout:1089: finishing write tid 5 to smithi07134777-13 2024-09-11T01:36:52.314 INFO:tasks.rados.rados.0.smithi071.stdout:1089: oid 13 updating version 0 to 426 2024-09-11T01:36:52.314 INFO:tasks.rados.rados.0.smithi071.stdout:1089: oid 13 updating version 426 to 427 2024-09-11T01:36:52.314 INFO:tasks.rados.rados.0.smithi071.stdout:1089: oid 13 updating version 427 to 428 2024-09-11T01:36:52.314 INFO:tasks.rados.rados.0.smithi071.stdout:1089: oid 13 updating version 428 to 429 2024-09-11T01:36:52.314 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 429 (ObjNum 409 snap 115 seq_num 409) dirty exists 2024-09-11T01:36:52.314 INFO:tasks.rados.rados.0.smithi071.stdout:1089: left oid 13 (ObjNum 409 snap 115 seq_num 409) 2024-09-11T01:36:52.314 INFO:tasks.rados.rados.0.smithi071.stdout:1085: done (4 left) 2024-09-11T01:36:52.314 INFO:tasks.rados.rados.0.smithi071.stdout:1086: done (3 left) 2024-09-11T01:36:52.314 INFO:tasks.rados.rados.0.smithi071.stdout:1087: done (2 left) 2024-09-11T01:36:52.314 INFO:tasks.rados.rados.0.smithi071.stdout:1089: done (1 left) 2024-09-11T01:36:52.314 INFO:tasks.rados.rados.0.smithi071.stdout:1090: done (0 left) 2024-09-11T01:36:52.314 INFO:tasks.rados.rados.0.smithi071.stdout:1091: write oid 16 current snap is 115 2024-09-11T01:36:52.314 INFO:tasks.rados.rados.0.smithi071.stdout:1091: seq_num 411 ranges {507677=410401,1367511=517802,2420497=487180} 2024-09-11T01:36:52.317 INFO:tasks.rados.rados.0.smithi071.stdout:1091: writing smithi07134777-16 from 507677 to 918078 tid 1 2024-09-11T01:36:52.323 INFO:tasks.rados.rados.0.smithi071.stdout:1091: writing smithi07134777-16 from 1367511 to 1885313 tid 2 2024-09-11T01:36:52.330 INFO:tasks.rados.rados.0.smithi071.stdout:1091: writing smithi07134777-16 from 2420497 to 2907677 tid 3 2024-09-11T01:36:52.330 INFO:tasks.rados.rados.0.smithi071.stdout:1092: read oid 37 snap -1 2024-09-11T01:36:52.330 INFO:tasks.rados.rados.0.smithi071.stdout:1092: expect (ObjNum 347 snap 98 seq_num 347) 2024-09-11T01:36:52.330 INFO:tasks.rados.rados.0.smithi071.stdout:1093: copy_from oid 8 from oid 13 current snap is 115 2024-09-11T01:36:52.330 INFO:tasks.rados.rados.0.smithi071.stdout:1091: finishing write tid 1 to smithi07134777-16 2024-09-11T01:36:52.330 INFO:tasks.rados.rados.0.smithi071.stdout:1091: finishing write tid 2 to smithi07134777-16 2024-09-11T01:36:52.330 INFO:tasks.rados.rados.0.smithi071.stdout:1094: rollback oid 36 current snap is 115 2024-09-11T01:36:52.330 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 36 to 114 2024-09-11T01:36:52.330 INFO:tasks.rados.rados.0.smithi071.stdout:1095: read oid 9 snap 112 2024-09-11T01:36:52.331 INFO:tasks.rados.rados.0.smithi071.stdout:1095: expect (ObjNum 336 snap 96 seq_num 336) 2024-09-11T01:36:52.331 INFO:tasks.rados.rados.0.smithi071.stdout:1096: snap_create 2024-09-11T01:36:52.335 INFO:tasks.rados.rados.0.smithi071.stdout:1094: finishing rollback tid 0 to smithi07134777-36 2024-09-11T01:36:52.335 INFO:tasks.rados.rados.0.smithi071.stdout:1094: finishing rollback tid 1 to smithi07134777-36 2024-09-11T01:36:52.335 INFO:tasks.rados.rados.0.smithi071.stdout:1095: expect (ObjNum 336 snap 96 seq_num 336) 2024-09-11T01:36:52.383 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2402409 2024-09-11T01:36:52.412 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2078662 2024-09-11T01:36:52.429 INFO:tasks.rados.rados.0.smithi071.stdout:1094: finishing rollback tid 2 to smithi07134777-36 2024-09-11T01:36:52.429 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 224 (ObjNum 376 snap 108 seq_num 376) dirty exists 2024-09-11T01:36:52.429 INFO:tasks.rados.rados.0.smithi071.stdout:1091: finishing write tid 3 to smithi07134777-16 2024-09-11T01:36:52.429 INFO:tasks.rados.rados.0.smithi071.stdout:1091: finishing write tid 4 to smithi07134777-16 2024-09-11T01:36:52.429 INFO:tasks.rados.rados.0.smithi071.stdout:1091: finishing write tid 5 to smithi07134777-16 2024-09-11T01:36:52.429 INFO:tasks.rados.rados.0.smithi071.stdout:1091: oid 16 updating version 0 to 440 2024-09-11T01:36:52.429 INFO:tasks.rados.rados.0.smithi071.stdout:1091: oid 16 updating version 440 to 441 2024-09-11T01:36:52.430 INFO:tasks.rados.rados.0.smithi071.stdout:1091: oid 16 version 441 is already newer than 439 2024-09-11T01:36:52.430 INFO:tasks.rados.rados.0.smithi071.stdout:1091: oid 16 version 441 is already newer than 438 2024-09-11T01:36:52.430 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 441 (ObjNum 411 snap 115 seq_num 411) dirty exists 2024-09-11T01:36:52.430 INFO:tasks.rados.rados.0.smithi071.stdout:1091: left oid 16 (ObjNum 411 snap 115 seq_num 411) 2024-09-11T01:36:52.430 INFO:tasks.rados.rados.0.smithi071.stdout:1092: expect (ObjNum 347 snap 98 seq_num 347) 2024-09-11T01:36:52.544 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:36:52.600 INFO:tasks.rados.rados.0.smithi071.stdout:1093: finishing copy_from racing read to smithi07134777-8 2024-09-11T01:36:52.600 INFO:tasks.rados.rados.0.smithi071.stdout:1093: finishing copy_from to smithi07134777-8 2024-09-11T01:36:52.600 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 429 (ObjNum 409 snap 115 seq_num 409) dirty exists 2024-09-11T01:36:52.682 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:36:52.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.792+0000 7f5921332640 10 monclient: get_monmap_and_config 2024-09-11T01:36:52.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.792+0000 7f5921332640 10 monclient: build_initial_monmap 2024-09-11T01:36:52.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.792+0000 7f5921332640 10 monclient: monmap: 2024-09-11T01:36:52.794 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:52.794 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:52.794 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:52.793062+0000 2024-09-11T01:36:52.794 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:52.793062+0000 2024-09-11T01:36:52.794 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:52.794 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:52.794 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:52.794 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:52.794 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:52.794 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:52.794 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:52.795 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:52.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.792+0000 7f5921332640 1 Processor -- start 2024-09-11T01:36:52.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f5921332640 1 -- start start 2024-09-11T01:36:52.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f5921332640 10 monclient: init 2024-09-11T01:36:52.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f5921332640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:52.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f5921332640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-11T01:36:52.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f5921332640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f591c09d790 0x7f591c09db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f5921332640 10 monclient(hunting): picked mon.noname-a con 0x7f591c09d790 addr v2:172.21.15.12:3300/0 2024-09-11T01:36:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f5921332640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f591c09e0b0 0x7f591c09e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f5921332640 10 monclient(hunting): picked mon.noname-c con 0x7f591c09e0b0 addr v2:172.21.15.71:3300/0 2024-09-11T01:36:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f5921332640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f591c134a60 addr v1:172.21.15.71:6789/0 2024-09-11T01:36:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f5921332640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f591c0a1da0 con 0x7f591c134a60 2024-09-11T01:36:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f5921332640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f5921332640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f591c0a0760 con 0x7f591c09d790 2024-09-11T01:36:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f5921332640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f5921332640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f591c0a1fa0 con 0x7f591c09e0b0 2024-09-11T01:36:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f5921332640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f5921332640 10 monclient(hunting): authenticate will time out at 1219.235675s 2024-09-11T01:36:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f591affd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f591c09d790 0x7f591c09db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f591affd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f591c09d790 0x7f591c09db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49232/0 (socket says 172.21.15.12:49232) 2024-09-11T01:36:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f591affd640 1 -- 172.21.15.12:0/911754962 learned_addr learned my addr 172.21.15.12:0/911754962 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f591affd640 10 monclient(hunting): get_auth_request con 0x7f591c09d790 auth_method 0 2024-09-11T01:36:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f591affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f591affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f591affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f591affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f591affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f591affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f591a7fc640 1 --2- 172.21.15.12:0/911754962 >> v2:172.21.15.71:3300/0 conn(0x7f591c09e0b0 0x7f591c09e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f591a7fc640 10 monclient(hunting): get_auth_request con 0x7f591c09e0b0 auth_method 0 2024-09-11T01:36:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f591a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f591a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f591a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.793+0000 7f591affd640 10 monclient(hunting): handle_auth_done global_id 54378 payload 274 2024-09-11T01:36:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.794+0000 7f591affd640 1 -- 172.21.15.12:0/911754962 >> v2:172.21.15.71:3300/0 conn(0x7f591c09e0b0 msgr2=0x7f591c09e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.794+0000 7f591affd640 1 --2- 172.21.15.12:0/911754962 >> v2:172.21.15.71:3300/0 conn(0x7f591c09e0b0 0x7f591c09e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.794+0000 7f591affd640 1 -- 172.21.15.12:0/911754962 >> v1:172.21.15.71:6789/0 conn(0x7f591c134a60 legacy=0x7f591c134ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.794+0000 7f591affd640 10 monclient: _finish_hunting 0 2024-09-11T01:36:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.794+0000 7f591affd640 1 monclient: found mon.noname-a 2024-09-11T01:36:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.794+0000 7f591affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.794+0000 7f591affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.794+0000 7f591affd640 1 -- 172.21.15.12:0/911754962 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f591c09eda0 con 0x7f591c09d790 2024-09-11T01:36:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.796+0000 7f591affd640 1 -- 172.21.15.12:0/911754962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f591c09d790 msgr2=0x7f591c09db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.796+0000 7f591affd640 1 --2- 172.21.15.12:0/911754962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f591c09d790 0x7f591c09db70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f591c09eca0 tx=0x7f590c02a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aec6e83a4e68ff6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5919ffb640 1 -- 172.21.15.12:0/911754962 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590c036070 con 0x7f591c09d790 2024-09-11T01:36:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5919ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5919ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5919ffb640 10 monclient: dump: 2024-09-11T01:36:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:52.806 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:52.806 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:52.806 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:52.806 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:52.806 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:52.806 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:52.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5919ffb640 10 monclient: _finish_auth 0 2024-09-11T01:36:52.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5919ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:36:52.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5919ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:52.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5919ffb640 1 -- 172.21.15.12:0/911754962 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f590c007800 con 0x7f591c09d790 2024-09-11T01:36:52.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5921332640 5 monclient: authenticate success, global_id 54378 2024-09-11T01:36:52.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5921332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:36:52.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5919ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:52.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5919ffb640 1 -- 172.21.15.12:0/911754962 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590c007bd0 con 0x7f591c09d790 2024-09-11T01:36:52.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5921332640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:52.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5921332640 4 set_mon_vals no callback set 2024-09-11T01:36:52.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5919ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:52.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5919ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:52.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5919ffb640 10 monclient: dump: 2024-09-11T01:36:52.807 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:52.807 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:52.807 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:52.807 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:52.807 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:52.807 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:52.808 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:52.808 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:52.808 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:52.808 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:52.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5921332640 10 monclient: shutdown 2024-09-11T01:36:52.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5921332640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:52.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5921332640 1 -- 172.21.15.12:0/911754962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f591c09d790 msgr2=0x7f591c09db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:52.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5921332640 1 --2- 172.21.15.12:0/911754962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f591c09d790 0x7f591c09db70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f591c09eca0 tx=0x7f590c02a570 comp rx=0 tx=0).stop 2024-09-11T01:36:52.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5921332640 1 -- 172.21.15.12:0/911754962 shutdown_connections 2024-09-11T01:36:52.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5921332640 1 --2- 172.21.15.12:0/911754962 >> v2:172.21.15.71:3300/0 conn(0x7f591c09e0b0 0x7f591c09e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:52.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5921332640 1 --2- 172.21.15.12:0/911754962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f591c09d790 0x7f591c09db70 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:52.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5921332640 1 -- 172.21.15.12:0/911754962 >> 172.21.15.12:0/911754962 conn(0x7f591c0a2d90 msgr2=0x7f591c0a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:52.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5921332640 1 -- 172.21.15.12:0/911754962 shutdown_connections 2024-09-11T01:36:52.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.797+0000 7f5921332640 1 -- 172.21.15.12:0/911754962 wait complete. 2024-09-11T01:36:52.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.798+0000 7f5921332640 10 monclient: build_initial_monmap 2024-09-11T01:36:52.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.798+0000 7f5921332640 10 monclient: monmap: 2024-09-11T01:36:52.809 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:52.809 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:52.809 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:52.809 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:52.809 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:52.809 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:52.809 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:52.809 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:52.809 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:52.809 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:52.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.798+0000 7f5921332640 1 Processor -- start 2024-09-11T01:36:52.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f5921332640 1 -- start start 2024-09-11T01:36:52.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f5921332640 10 monclient: init 2024-09-11T01:36:52.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f5921332640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:52.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f5921332640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:36:52.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f5921332640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f591c09d790 0x7f591c06f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:52.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f5921332640 10 monclient(hunting): picked mon.noname-b con 0x7f591c09d790 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:52.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f5921332640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f591c09e0b0 0x7f591c06fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:52.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f5921332640 10 monclient(hunting): picked mon.noname-a con 0x7f591c09e0b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:52.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f5921332640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f591c134a60 0x7f591c1ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:52.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f5921332640 10 monclient(hunting): picked mon.noname-c con 0x7f591c134a60 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:52.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f5921332640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:52.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f5921332640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f591c09ce20 con 0x7f591c09e0b0 2024-09-11T01:36:52.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f5921332640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:52.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f5921332640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f591c09cca0 con 0x7f591c09d790 2024-09-11T01:36:52.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f5921332640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:52.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f5921332640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f591c1355d0 con 0x7f591c134a60 2024-09-11T01:36:52.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f5921332640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:52.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f5921332640 10 monclient(hunting): authenticate will time out at 1219.241625s 2024-09-11T01:36:52.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591a7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f591c09e0b0 0x7f591c06fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:52.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f591c134a60 0x7f591c1ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:52.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591a7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f591c09e0b0 0x7f591c06fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49242/0 (socket says 172.21.15.12:49242) 2024-09-11T01:36:52.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591a7fc640 1 -- 172.21.15.12:0/2821191441 learned_addr learned my addr 172.21.15.12:0/2821191441 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:52.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591a7fc640 10 monclient(hunting): get_auth_request con 0x7f591c09e0b0 auth_method 0 2024-09-11T01:36:52.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:52.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591affd640 1 --2- 172.21.15.12:0/2821191441 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f591c09d790 0x7f591c06f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:52.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:52.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:52.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591b7fe640 10 monclient(hunting): get_auth_request con 0x7f591c134a60 auth_method 0 2024-09-11T01:36:52.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:52.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:52.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:52.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591affd640 10 monclient(hunting): get_auth_request con 0x7f591c09d790 auth_method 0 2024-09-11T01:36:52.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:52.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:52.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:52.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:52.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:52.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591b7fe640 1 -- 172.21.15.12:0/2821191441 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f591c134a60 msgr2=0x7f591c1ccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:36:52.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591b7fe640 1 -- 172.21.15.12:0/2821191441 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f591c134a60 msgr2=0x7f591c1ccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:36:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591b7fe640 1 --2- 172.21.15.12:0/2821191441 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f591c134a60 0x7f591c1ccda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591b7fe640 1 --2- 172.21.15.12:0/2821191441 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f591c134a60 0x7f591c1ccda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.799+0000 7f591affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.800+0000 7f591a7fc640 10 monclient(hunting): handle_auth_done global_id 54381 payload 824 2024-09-11T01:36:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.800+0000 7f591a7fc640 1 -- 172.21.15.12:0/2821191441 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f591c134a60 msgr2=0x7f591c1ccda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:36:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.800+0000 7f591a7fc640 1 --2- 172.21.15.12:0/2821191441 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f591c134a60 0x7f591c1ccda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.800+0000 7f591a7fc640 1 -- 172.21.15.12:0/2821191441 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f591c09d790 msgr2=0x7f591c06f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.800+0000 7f591a7fc640 1 --2- 172.21.15.12:0/2821191441 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f591c09d790 0x7f591c06f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.800+0000 7f591a7fc640 10 monclient: _finish_hunting 0 2024-09-11T01:36:52.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.800+0000 7f591a7fc640 1 monclient: found mon.noname-a 2024-09-11T01:36:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.800+0000 7f591a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.800+0000 7f591a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.800+0000 7f591a7fc640 1 -- 172.21.15.12:0/2821191441 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f591c128490 con 0x7f591c09e0b0 2024-09-11T01:36:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.800+0000 7f591affd640 1 --2- 172.21.15.12:0/2821191441 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f591c09d790 0x7f591c06f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:36:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.802+0000 7f591a7fc640 1 -- 172.21.15.12:0/2821191441 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f591c09e0b0 msgr2=0x7f591c06fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:52.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.802+0000 7f591a7fc640 1 --2- 172.21.15.12:0/2821191441 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f591c09e0b0 0x7f591c06fd50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f590400d800 tx=0x7f590400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.804+0000 7f59137fe640 1 -- 172.21.15.12:0/2821191441 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590400b8d0 con 0x7f591c09e0b0 2024-09-11T01:36:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.804+0000 7f59137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.804+0000 7f59137fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.804+0000 7f59137fe640 10 monclient: dump: 2024-09-11T01:36:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:52.816 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:52.817 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.804+0000 7f59137fe640 10 monclient: _finish_auth 0 2024-09-11T01:36:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.804+0000 7f59137fe640 10 monclient: _check_auth_tickets 2024-09-11T01:36:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.804+0000 7f59137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.804+0000 7f59137fe640 1 -- 172.21.15.12:0/2821191441 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5904019070 con 0x7f591c09e0b0 2024-09-11T01:36:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.804+0000 7f5921332640 5 monclient: authenticate success, global_id 54381 2024-09-11T01:36:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.804+0000 7f5921332640 10 monclient: _renew_subs 2024-09-11T01:36:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.804+0000 7f5921332640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:52.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.804+0000 7f5921332640 1 -- 172.21.15.12:0/2821191441 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f591c0a4840 con 0x7f591c09e0b0 2024-09-11T01:36:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.804+0000 7f59137fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.804+0000 7f59137fe640 1 -- 172.21.15.12:0/2821191441 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590400bc90 con 0x7f591c09e0b0 2024-09-11T01:36:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.804+0000 7f59137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.804+0000 7f59137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.804+0000 7f59137fe640 10 monclient: dump: 2024-09-11T01:36:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:52.818 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:52.819 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:52.819 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:52.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.804+0000 7f5921332640 10 monclient: _renew_subs 2024-09-11T01:36:52.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.804+0000 7f59197fa640 4 set_mon_vals no callback set 2024-09-11T01:36:52.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.804+0000 7f5921332640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:52.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.804+0000 7f5921332640 1 -- 172.21.15.12:0/2821191441 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f591c1281b0 con 0x7f591c09e0b0 2024-09-11T01:36:52.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.805+0000 7f59137fe640 1 -- 172.21.15.12:0/2821191441 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f5904013070 con 0x7f591c09e0b0 2024-09-11T01:36:52.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.805+0000 7f5921332640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:52.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.805+0000 7f5921332640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:52.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.805+0000 7f5921332640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:52.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.805+0000 7f5921332640 1 -- 172.21.15.12:0/2821191441 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f591c09db70 con 0x7f591c09e0b0 2024-09-11T01:36:52.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.805+0000 7f59137fe640 1 --2- 172.21.15.12:0/2821191441 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f58fc03e2c0 0x7f58fc040780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:52.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.806+0000 7f59137fe640 1 -- 172.21.15.12:0/2821191441 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(252..252 src has 186..252) ==== 4847+0+0 (secure 0 0 0) 0x7f5904052e50 con 0x7f591c09e0b0 2024-09-11T01:36:52.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.806+0000 7f591affd640 1 --2- 172.21.15.12:0/2821191441 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f58fc03e2c0 0x7f58fc040780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:52.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.806+0000 7f591affd640 10 monclient: get_auth_request con 0x7f58fc03e2c0 auth_method 0 2024-09-11T01:36:52.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.808+0000 7f591affd640 1 --2- 172.21.15.12:0/2821191441 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f58fc03e2c0 0x7f58fc040780 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f590c033000 tx=0x7f590c009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:52.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.810+0000 7f59137fe640 1 -- 172.21.15.12:0/2821191441 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5904051810 con 0x7f591c09e0b0 2024-09-11T01:36:52.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.810+0000 7f59137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:52.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:52.810+0000 7f59137fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:53.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.068+0000 7f5921332640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:36:53.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.068+0000 7f5921332640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:36:53.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.068+0000 7f5921332640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:53.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.068+0000 7f5921332640 1 -- 172.21.15.12:0/2821191441 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f591c1344d0 con 0x7f591c09e0b0 2024-09-11T01:36:53.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.068+0000 7f59137fe640 1 -- 172.21.15.12:0/2821191441 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f590401ec10 con 0x7f591c09e0b0 2024-09-11T01:36:53.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.068+0000 7f59137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:36:53.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.068+0000 7f59137fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:53.071 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:36:53.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.071+0000 7f5921332640 1 -- 172.21.15.12:0/2821191441 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f58fc03e2c0 msgr2=0x7f58fc040780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:53.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.071+0000 7f5921332640 1 --2- 172.21.15.12:0/2821191441 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f58fc03e2c0 0x7f58fc040780 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f590c033000 tx=0x7f590c009000 comp rx=0 tx=0).stop 2024-09-11T01:36:53.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.071+0000 7f5921332640 10 monclient: shutdown 2024-09-11T01:36:53.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.071+0000 7f5921332640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:53.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.071+0000 7f5921332640 1 -- 172.21.15.12:0/2821191441 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f591c09e0b0 msgr2=0x7f591c06fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:53.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.071+0000 7f5921332640 1 --2- 172.21.15.12:0/2821191441 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f591c09e0b0 0x7f591c06fd50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f590400d800 tx=0x7f590400dcc0 comp rx=0 tx=0).stop 2024-09-11T01:36:53.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.071+0000 7f5921332640 1 -- 172.21.15.12:0/2821191441 shutdown_connections 2024-09-11T01:36:53.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.071+0000 7f5921332640 1 --2- 172.21.15.12:0/2821191441 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f591c134a60 0x7f591c1ccda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:53.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.071+0000 7f5921332640 1 --2- 172.21.15.12:0/2821191441 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f58fc03e2c0 0x7f58fc040780 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:53.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.071+0000 7f5921332640 1 --2- 172.21.15.12:0/2821191441 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f591c09e0b0 0x7f591c06fd50 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:53.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.071+0000 7f5921332640 1 --2- 172.21.15.12:0/2821191441 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f591c09d790 0x7f591c06f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:53.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.071+0000 7f5921332640 1 -- 172.21.15.12:0/2821191441 >> 172.21.15.12:0/2821191441 conn(0x7f591c0a2d90 msgr2=0x7f591c0a0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:53.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.071+0000 7f5921332640 1 -- 172.21.15.12:0/2821191441 shutdown_connections 2024-09-11T01:36:53.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.071+0000 7f5921332640 1 -- 172.21.15.12:0/2821191441 wait complete. 2024-09-11T01:36:53.083 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:36:53.083 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:36:53.083 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:36:53.162 INFO:tasks.rados.rados.0.smithi071.stdout:1091: done (5 left) 2024-09-11T01:36:53.162 INFO:tasks.rados.rados.0.smithi071.stdout:1092: done (4 left) 2024-09-11T01:36:53.162 INFO:tasks.rados.rados.0.smithi071.stdout:1093: done (3 left) 2024-09-11T01:36:53.162 INFO:tasks.rados.rados.0.smithi071.stdout:1094: done (2 left) 2024-09-11T01:36:53.162 INFO:tasks.rados.rados.0.smithi071.stdout:1095: done (1 left) 2024-09-11T01:36:53.162 INFO:tasks.rados.rados.0.smithi071.stdout:1096: done (0 left) 2024-09-11T01:36:53.162 INFO:tasks.rados.rados.0.smithi071.stdout:1097: delete oid 46 current snap is 116 2024-09-11T01:36:53.164 INFO:tasks.rados.rados.0.smithi071.stdout:1097: done (0 left) 2024-09-11T01:36:53.164 INFO:tasks.rados.rados.0.smithi071.stdout:1098: snap_remove snap 112 2024-09-11T01:36:53.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.184+0000 7f18e782a640 10 monclient: get_monmap_and_config 2024-09-11T01:36:53.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.184+0000 7f18e782a640 10 monclient: build_initial_monmap 2024-09-11T01:36:53.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.184+0000 7f18e782a640 10 monclient: monmap: 2024-09-11T01:36:53.186 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:53.186 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:53.186 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:53.185401+0000 2024-09-11T01:36:53.186 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:53.185401+0000 2024-09-11T01:36:53.186 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:53.186 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:53.186 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:53.187 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:53.187 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:53.187 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:53.187 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:53.187 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:53.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.185+0000 7f18e782a640 1 Processor -- start 2024-09-11T01:36:53.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.185+0000 7f18e782a640 1 -- start start 2024-09-11T01:36:53.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.185+0000 7f18e782a640 10 monclient: init 2024-09-11T01:36:53.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.185+0000 7f18e782a640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:53.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.185+0000 7f18e782a640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-11T01:36:53.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.185+0000 7f18e782a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f18e0131710 addr v1:172.21.15.71:6789/0 2024-09-11T01:36:53.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.186+0000 7f18e782a640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f18e00a0d70 0x7f18e013adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:53.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.186+0000 7f18e782a640 10 monclient(hunting): picked mon.noname-c con 0x7f18e00a0d70 addr v2:172.21.15.71:3300/0 2024-09-11T01:36:53.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.186+0000 7f18e782a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f18e013b2f0 addr v1:172.21.15.12:6789/0 2024-09-11T01:36:53.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.186+0000 7f18e782a640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18e0074a70 con 0x7f18e013b2f0 2024-09-11T01:36:53.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.186+0000 7f18e782a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18e0073af0 con 0x7f18e0131710 2024-09-11T01:36:53.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.186+0000 7f18e782a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:53.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.186+0000 7f18e782a640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f18e0073d90 con 0x7f18e00a0d70 2024-09-11T01:36:53.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.186+0000 7f18e782a640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:53.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.186+0000 7f18e782a640 10 monclient(hunting): authenticate will time out at 1219.628136s 2024-09-11T01:36:53.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.186+0000 7f18e5dc7640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f18e013b2f0 0x7f18e013d6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:48590/0 (socket says 172.21.15.12:48590) 2024-09-11T01:36:53.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.186+0000 7f18e5dc7640 1 -- 172.21.15.12:0/1780113415 learned_addr learned my addr 172.21.15.12:0/1780113415 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:36:53.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.186+0000 7f18e4dc5640 1 --2- 172.21.15.12:0/1780113415 >> v2:172.21.15.71:3300/0 conn(0x7f18e00a0d70 0x7f18e013adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:53.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.186+0000 7f18e4dc5640 10 monclient(hunting): get_auth_request con 0x7f18e00a0d70 auth_method 0 2024-09-11T01:36:53.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.186+0000 7f18e4dc5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:53.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.186+0000 7f18e4dc5640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:53.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.186+0000 7f18e4dc5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:53.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.186+0000 7f18e4dc5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:53.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.186+0000 7f18e4dc5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:53.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.186+0000 7f18e4dc5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:53.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.188+0000 7f18e5dc7640 1 -- 172.21.15.12:0/1780113415 >> v1:172.21.15.12:6789/0 conn(0x7f18e013b2f0 legacy=0x7f18e013d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:53.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.188+0000 7f18d7fff640 1 -- 172.21.15.12:0/1780113415 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3959263596 0 0) 0x7f18d80017d0 con 0x7f18e013b2f0 2024-09-11T01:36:53.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.188+0000 7f18d7fff640 10 monclient(hunting): _negotiate 2024-09-11T01:36:53.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.188+0000 7f18d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:53.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.188+0000 7f18d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:53.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.188+0000 7f18d7fff640 10 monclient(hunting): my global_id is 54384 2024-09-11T01:36:53.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.188+0000 7f18e55c6640 1 -- 172.21.15.12:0/1780113415 >> v1:172.21.15.71:6789/0 conn(0x7f18e0131710 legacy=0x7f18e0131af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:53.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.188+0000 7f18d7fff640 1 -- 172.21.15.12:0/1780113415 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18cc003520 con 0x7f18e013b2f0 2024-09-11T01:36:53.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.188+0000 7f18e4dc5640 10 monclient(hunting): handle_auth_done global_id 74185 payload 274 2024-09-11T01:36:53.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.188+0000 7f18e4dc5640 1 -- 172.21.15.12:0/1780113415 >> v1:172.21.15.71:6789/0 conn(0x7f18e0131710 legacy=0x7f18e0131af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:53.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.188+0000 7f18e4dc5640 1 -- 172.21.15.12:0/1780113415 >> v1:172.21.15.12:6789/0 conn(0x7f18e013b2f0 legacy=0x7f18e013d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:53.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.188+0000 7f18e4dc5640 10 monclient: _finish_hunting 0 2024-09-11T01:36:53.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.188+0000 7f18e4dc5640 1 monclient: found mon.noname-c 2024-09-11T01:36:53.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.188+0000 7f18e4dc5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:53.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.188+0000 7f18e4dc5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:36:53.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.188+0000 7f18e4dc5640 1 -- 172.21.15.12:0/1780113415 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18e0074070 con 0x7f18e00a0d70 2024-09-11T01:36:53.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.191+0000 7f18e4dc5640 1 -- 172.21.15.12:0/1780113415 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18e00a0d70 msgr2=0x7f18e013adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:53.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.191+0000 7f18e4dc5640 1 --2- 172.21.15.12:0/1780113415 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18e00a0d70 0x7f18e013adb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f18d00098a0 tx=0x7f18d002a320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=107a63eee48482a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:53.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.191+0000 7f18d7fff640 1 -- 172.21.15.12:0/1780113415 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d0037070 con 0x7f18e00a0d70 2024-09-11T01:36:53.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.191+0000 7f18d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:53.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.191+0000 7f18d7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:36:53.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.191+0000 7f18d7fff640 10 monclient: dump: 2024-09-11T01:36:53.197 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:53.197 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:53.197 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:53.197 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:53.197 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:53.197 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:53.197 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:53.197 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:53.197 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:53.198 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:53.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.191+0000 7f18d7fff640 10 monclient: _finish_auth 0 2024-09-11T01:36:53.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.191+0000 7f18d7fff640 10 monclient: _check_auth_tickets 2024-09-11T01:36:53.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.191+0000 7f18d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:53.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.191+0000 7f18d7fff640 1 -- 172.21.15.12:0/1780113415 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18d0032640 con 0x7f18e00a0d70 2024-09-11T01:36:53.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.191+0000 7f18e782a640 5 monclient: authenticate success, global_id 74185 2024-09-11T01:36:53.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.191+0000 7f18e782a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:36:53.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.191+0000 7f18d7fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:53.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.191+0000 7f18e782a640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:53.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.191+0000 7f18e782a640 4 set_mon_vals no callback set 2024-09-11T01:36:53.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.191+0000 7f18d7fff640 1 -- 172.21.15.12:0/1780113415 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d0032a10 con 0x7f18e00a0d70 2024-09-11T01:36:53.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.191+0000 7f18d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:53.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.191+0000 7f18d7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:36:53.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.191+0000 7f18d7fff640 10 monclient: dump: 2024-09-11T01:36:53.199 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:53.199 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:53.199 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:53.199 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:53.199 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:53.199 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:53.199 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:53.199 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:53.199 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:53.199 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:53.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.192+0000 7f18e782a640 10 monclient: shutdown 2024-09-11T01:36:53.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.192+0000 7f18e782a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:53.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.192+0000 7f18e782a640 1 -- 172.21.15.12:0/1780113415 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18e00a0d70 msgr2=0x7f18e013adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:53.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.192+0000 7f18e782a640 1 --2- 172.21.15.12:0/1780113415 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18e00a0d70 0x7f18e013adb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f18d00098a0 tx=0x7f18d002a320 comp rx=0 tx=0).stop 2024-09-11T01:36:53.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.192+0000 7f18e782a640 1 -- 172.21.15.12:0/1780113415 shutdown_connections 2024-09-11T01:36:53.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.192+0000 7f18e782a640 1 --2- 172.21.15.12:0/1780113415 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18e00a0d70 0x7f18e013adb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:53.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.192+0000 7f18e782a640 1 -- 172.21.15.12:0/1780113415 >> 172.21.15.12:0/1780113415 conn(0x7f18e00a36c0 msgr2=0x7f18e00a3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:53.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.192+0000 7f18e782a640 1 -- 172.21.15.12:0/1780113415 shutdown_connections 2024-09-11T01:36:53.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.192+0000 7f18e782a640 1 -- 172.21.15.12:0/1780113415 wait complete. 2024-09-11T01:36:53.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.193+0000 7f18e782a640 10 monclient: build_initial_monmap 2024-09-11T01:36:53.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.193+0000 7f18e782a640 10 monclient: monmap: 2024-09-11T01:36:53.200 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:53.201 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:53.201 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:53.201 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:53.201 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:53.201 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:53.201 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:53.201 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:53.201 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:53.201 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:53.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.193+0000 7f18e782a640 1 Processor -- start 2024-09-11T01:36:53.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.193+0000 7f18e782a640 1 -- start start 2024-09-11T01:36:53.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.193+0000 7f18e782a640 10 monclient: init 2024-09-11T01:36:53.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e782a640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:53.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e782a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:36:53.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e782a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18e00a0d70 0x7f18e0130790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:53.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e782a640 10 monclient(hunting): picked mon.noname-b con 0x7f18e00a0d70 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:53.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e782a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18e0131710 0x7f18e0130cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:53.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e782a640 10 monclient(hunting): picked mon.noname-c con 0x7f18e0131710 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e782a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18e013b2f0 0x7f18e01cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e782a640 10 monclient(hunting): picked mon.noname-a con 0x7f18e013b2f0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e782a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e782a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f18e0073350 con 0x7f18e013b2f0 2024-09-11T01:36:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e782a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e782a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f18e00731d0 con 0x7f18e00a0d70 2024-09-11T01:36:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e782a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e782a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f18e013ee00 con 0x7f18e0131710 2024-09-11T01:36:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e782a640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e782a640 10 monclient(hunting): authenticate will time out at 1219.636151s 2024-09-11T01:36:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e4dc5640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18e0131710 0x7f18e0130cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e4dc5640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18e0131710 0x7f18e0130cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52838/0 (socket says 172.21.15.12:52838) 2024-09-11T01:36:53.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e4dc5640 1 -- 172.21.15.12:0/1565881893 learned_addr learned my addr 172.21.15.12:0/1565881893 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:53.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e4dc5640 10 monclient(hunting): get_auth_request con 0x7f18e0131710 auth_method 0 2024-09-11T01:36:53.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e4dc5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:53.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e4dc5640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:53.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e4dc5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:53.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e5dc7640 1 --2- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18e013b2f0 0x7f18e01cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:53.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e5dc7640 10 monclient(hunting): get_auth_request con 0x7f18e013b2f0 auth_method 0 2024-09-11T01:36:53.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e5dc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:53.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e5dc7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:53.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e5dc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:53.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e4dc5640 1 -- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18e0131710 msgr2=0x7f18e0130cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:36:53.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e4dc5640 1 -- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18e0131710 msgr2=0x7f18e0130cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:36:53.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e4dc5640 1 --2- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18e0131710 0x7f18e0130cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:53.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e55c6640 1 --2- 172.21.15.12:0/1565881893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18e00a0d70 0x7f18e0130790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:53.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e4dc5640 1 --2- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18e0131710 0x7f18e0130cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:53.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e55c6640 10 monclient(hunting): get_auth_request con 0x7f18e00a0d70 auth_method 0 2024-09-11T01:36:53.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e55c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:53.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e55c6640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:53.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e55c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:53.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e5dc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:53.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e5dc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:53.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e5dc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:53.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e55c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:53.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e55c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:53.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e55c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:53.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e5dc7640 10 monclient(hunting): handle_auth_done global_id 54387 payload 824 2024-09-11T01:36:53.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e5dc7640 1 -- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18e0131710 msgr2=0x7f18e0130cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:36:53.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e5dc7640 1 --2- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f18e0131710 0x7f18e0130cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:53.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e5dc7640 1 -- 172.21.15.12:0/1565881893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18e00a0d70 msgr2=0x7f18e0130790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:53.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e5dc7640 1 --2- 172.21.15.12:0/1565881893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18e00a0d70 0x7f18e0130790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:53.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e5dc7640 10 monclient: _finish_hunting 0 2024-09-11T01:36:53.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e5dc7640 1 monclient: found mon.noname-a 2024-09-11T01:36:53.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e5dc7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:53.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e5dc7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:53.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.194+0000 7f18e5dc7640 1 -- 172.21.15.12:0/1565881893 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18e0139650 con 0x7f18e013b2f0 2024-09-11T01:36:53.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.196+0000 7f18e5dc7640 1 -- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18e013b2f0 msgr2=0x7f18e01cd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:53.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.196+0000 7f18e5dc7640 1 --2- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18e013b2f0 0x7f18e01cd020 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f18d8001a20 tx=0x7f18d800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:53.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.196+0000 7f18d5ffb640 1 -- 172.21.15.12:0/1565881893 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d800ad40 con 0x7f18e013b2f0 2024-09-11T01:36:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.196+0000 7f18d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.196+0000 7f18d5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.196+0000 7f18d5ffb640 10 monclient: dump: 2024-09-11T01:36:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:53.211 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:53.212 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:53.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.196+0000 7f18d5ffb640 10 monclient: _finish_auth 0 2024-09-11T01:36:53.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.196+0000 7f18d5ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:36:53.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.196+0000 7f18d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:53.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.196+0000 7f18d5ffb640 1 -- 172.21.15.12:0/1565881893 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18d8012400 con 0x7f18e013b2f0 2024-09-11T01:36:53.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.196+0000 7f18e782a640 5 monclient: authenticate success, global_id 54387 2024-09-11T01:36:53.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.196+0000 7f18e782a640 10 monclient: _renew_subs 2024-09-11T01:36:53.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.196+0000 7f18e782a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:53.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.196+0000 7f18e782a640 1 -- 172.21.15.12:0/1565881893 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18e013ac10 con 0x7f18e013b2f0 2024-09-11T01:36:53.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.196+0000 7f18d5ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:53.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.196+0000 7f18d5ffb640 1 -- 172.21.15.12:0/1565881893 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d80127f0 con 0x7f18e013b2f0 2024-09-11T01:36:53.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.196+0000 7f18d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:53.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.196+0000 7f18d5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:53.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.197+0000 7f18d5ffb640 10 monclient: dump: 2024-09-11T01:36:53.213 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:53.213 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:53.213 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:53.213 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:53.213 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:53.213 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:53.213 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:53.213 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:53.213 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:53.213 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:53.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.196+0000 7f18d77fe640 4 set_mon_vals no callback set 2024-09-11T01:36:53.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.197+0000 7f18e782a640 10 monclient: _renew_subs 2024-09-11T01:36:53.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.197+0000 7f18e782a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:53.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.197+0000 7f18e782a640 1 -- 172.21.15.12:0/1565881893 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18e0072a60 con 0x7f18e013b2f0 2024-09-11T01:36:53.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.197+0000 7f18e782a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:53.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.197+0000 7f18e782a640 1 --2- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18e0139de0 0x7f18e002d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:53.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.197+0000 7f18e782a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:53.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.197+0000 7f18e782a640 1 -- 172.21.15.12:0/1565881893 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f18e0131af0 con 0x7f18e0139de0 2024-09-11T01:36:53.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.198+0000 7f18e55c6640 1 --2- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18e0139de0 0x7f18e002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:53.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.198+0000 7f18e55c6640 10 monclient: get_auth_request con 0x7f18e0139de0 auth_method 0 2024-09-11T01:36:53.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.198+0000 7f18e55c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:53.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.198+0000 7f18e55c6640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:53.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.198+0000 7f18e55c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:53.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.198+0000 7f18d5ffb640 1 -- 172.21.15.12:0/1565881893 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f18d8012990 con 0x7f18e013b2f0 2024-09-11T01:36:53.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.198+0000 7f18e55c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:53.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.198+0000 7f18e55c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:53.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.198+0000 7f18e55c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:53.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.198+0000 7f18e55c6640 10 monclient(hunting): handle_auth_done global_id 54390 payload 824 2024-09-11T01:36:53.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.198+0000 7f18e55c6640 1 -- 172.21.15.12:0/1565881893 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f18e009d130 con 0x7f18e0139de0 2024-09-11T01:36:53.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.198+0000 7f18d5ffb640 1 --2- 172.21.15.12:0/1565881893 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f18c403e210 0x7f18c40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:53.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.198+0000 7f18d5ffb640 1 -- 172.21.15.12:0/1565881893 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(253..253 src has 186..253) ==== 4847+0+0 (secure 0 0 0) 0x7f18d8053330 con 0x7f18e013b2f0 2024-09-11T01:36:53.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.198+0000 7f18e4dc5640 1 --2- 172.21.15.12:0/1565881893 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f18c403e210 0x7f18c40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:53.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.198+0000 7f18e4dc5640 10 monclient: get_auth_request con 0x7f18c403e210 auth_method 0 2024-09-11T01:36:53.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.200+0000 7f18e55c6640 1 -- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18e0139de0 msgr2=0x7f18e002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:53.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.200+0000 7f18e55c6640 1 --2- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18e0139de0 0x7f18e002d440 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f18dc00f560 tx=0x7f18dc00fa20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:53.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.200+0000 7f18d5ffb640 1 -- 172.21.15.12:0/1565881893 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18dc0074f0 con 0x7f18e0139de0 2024-09-11T01:36:53.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.200+0000 7f18d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:53.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.200+0000 7f18d5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:53.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.200+0000 7f18d5ffb640 10 monclient: dump: 2024-09-11T01:36:53.215 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:53.215 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:53.216 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:53.216 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:53.216 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:53.216 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:53.216 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:53.216 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:53.216 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:53.216 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:53.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.201+0000 7f18d5ffb640 1 -- 172.21.15.12:0/1565881893 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f18dc018070 con 0x7f18e0139de0 2024-09-11T01:36:53.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.201+0000 7f18d5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:53.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.201+0000 7f18d5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:53.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.201+0000 7f18d5ffb640 1 -- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18e0139de0 msgr2=0x7f18e002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:53.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.201+0000 7f18d5ffb640 1 --2- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18e0139de0 0x7f18e002d440 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f18dc00f560 tx=0x7f18dc00fa20 comp rx=0 tx=0).stop 2024-09-11T01:36:53.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.201+0000 7f18d5ffb640 1 -- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18e0139de0 msgr2=0x7f18e002d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:53.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.201+0000 7f18d5ffb640 1 --2- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18e0139de0 0x7f18e002d440 secure :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f18dc00f560 tx=0x7f18dc00fa20 comp rx=0 tx=0).stop 2024-09-11T01:36:53.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.201+0000 7f18e4dc5640 1 --2- 172.21.15.12:0/1565881893 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f18c403e210 0x7f18c40406d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f18d002a360 tx=0x7f18d002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:53.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.207+0000 7f18e782a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:36:53.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.207+0000 7f18e782a640 1 --2- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18e002d980 0x7f18e002dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:53.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.207+0000 7f18e782a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:53.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.207+0000 7f18e782a640 1 -- 172.21.15.12:0/1565881893 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f18e0137e50 con 0x7f18e002d980 2024-09-11T01:36:53.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.207+0000 7f18e55c6640 1 --2- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18e002d980 0x7f18e002dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:53.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.207+0000 7f18e55c6640 10 monclient: get_auth_request con 0x7f18e002d980 auth_method 0 2024-09-11T01:36:53.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.207+0000 7f18e55c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:53.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.207+0000 7f18e55c6640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:53.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.207+0000 7f18e55c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:53.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.207+0000 7f18e55c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:53.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.207+0000 7f18e55c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:53.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.207+0000 7f18e55c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:53.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.207+0000 7f18e55c6640 10 monclient(hunting): handle_auth_done global_id 54393 payload 824 2024-09-11T01:36:53.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.207+0000 7f18e55c6640 1 -- 172.21.15.12:0/1565881893 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f18e006dc80 con 0x7f18e002d980 2024-09-11T01:36:53.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.209+0000 7f18e55c6640 1 -- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18e002d980 msgr2=0x7f18e002dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:53.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.209+0000 7f18e55c6640 1 --2- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18e002d980 0x7f18e002dd60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f18dc00f560 tx=0x7f18dc000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:53.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.209+0000 7f18d5ffb640 1 -- 172.21.15.12:0/1565881893 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18dc005cc0 con 0x7f18e002d980 2024-09-11T01:36:53.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.209+0000 7f18d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:53.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.209+0000 7f18d5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:53.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.209+0000 7f18d5ffb640 10 monclient: dump: 2024-09-11T01:36:53.220 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:53.220 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:53.220 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:53.220 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:53.220 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:53.220 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:53.221 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:53.221 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:53.221 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:53.221 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:53.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.210+0000 7f18d5ffb640 1 -- 172.21.15.12:0/1565881893 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4877 (secure 0 0 0) 0x7f18dc005cc0 con 0x7f18e002d980 2024-09-11T01:36:53.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.210+0000 7f18d5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:36:53.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.210+0000 7f18d5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:53.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.210+0000 7f18d5ffb640 1 -- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18e002d980 msgr2=0x7f18e002dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:53.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.210+0000 7f18d5ffb640 1 --2- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18e002d980 0x7f18e002dd60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f18dc00f560 tx=0x7f18dc000ed0 comp rx=0 tx=0).stop 2024-09-11T01:36:53.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.210+0000 7f18d5ffb640 1 -- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18e002d980 msgr2=0x7f18e002dd60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:53.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.210+0000 7f18d5ffb640 1 --2- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18e002d980 0x7f18e002dd60 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:53.222 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:36:53.222 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:36:53.222 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:36:53.222 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:36:53.222 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 80, 2024-09-11T01:36:53.222 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 203190, 2024-09-11T01:36:53.222 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:36:53.222 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:36:53.222 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:36:53.222 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:36:53.222 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:53.222 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:36:53.223 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:53.223 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:36:53.223 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:36:53.223 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:53.223 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:53.223 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:53.223 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:53.223 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:53.223 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:53.223 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:53.223 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:53.223 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:53.223 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:53.224 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:53.224 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:53.224 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:36:53.224 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:36:53.224 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:53.224 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:53.224 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:53.224 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:53.224 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:53.224 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:53.224 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:53.224 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:53.224 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:53.224 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:53.225 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:53.225 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:53.225 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:53.225 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:36:53.225 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:36:53.225 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:36:53.225 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:36:53.225 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:36:53.225 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:36:53.225 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:53.225 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:53.225 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:36:53.225 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:36:53.226 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:36:53.226 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:36:53.226 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:36:53.226 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:36:53.226 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:36:53.226 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:53.226 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:36:53.226 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:53.226 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:53.226 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:53.226 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:53.226 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:53.226 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:53.227 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:53.227 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:53.227 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:53.227 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:53.227 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:53.227 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:53.227 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:36:53.227 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:53.227 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:36:53.227 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:53.227 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:36:53.227 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:36:53.227 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:53.228 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:53.228 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:53.228 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:53.228 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:36:53.228 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:53.228 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:53.228 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:53.228 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:53.228 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:36:53.228 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:53.229 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:53.229 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:53.229 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:53.229 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:36:53.229 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:36:53.229 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:53.229 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:53.229 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:53.229 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:53.229 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:53.229 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:36:53.229 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:36:53.229 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:53.230 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:53.230 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:53.230 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:53.230 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:36:53.230 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:53.230 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:53.230 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:53.230 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:53.230 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:36:53.230 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:53.230 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:53.230 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:53.230 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:53.230 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:36:53.231 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:36:53.231 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:53.231 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:53.231 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:53.231 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:53.231 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:53.231 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:36:53.231 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:36:53.231 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:53.231 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:53.231 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:53.231 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:53.232 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:36:53.232 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:53.232 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:53.232 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:53.232 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:53.232 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:36:53.232 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:53.232 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:53.232 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:53.232 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:53.232 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:36:53.232 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:36:53.232 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:53.232 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:53.233 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:53.233 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:53.233 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:53.233 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:53.233 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:36:53.233 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:36:53.233 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:53.233 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:53.233 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:53.233 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:36:53.233 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:53.233 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:53.233 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:36:53.234 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:53.234 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:53.234 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:53.234 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:36:53.234 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:53.234 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:53.234 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:36:53.234 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:53.234 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:53.234 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:53.234 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:36:53.234 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:53.234 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:53.234 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:36:53.235 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:53.235 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:53.235 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:53.235 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:36:53.235 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:53.235 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:53.235 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:53.235 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:36:53.235 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:36:53.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.210+0000 7f18e782a640 1 -- 172.21.15.12:0/1565881893 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f18c403e210 msgr2=0x7f18c40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:53.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.210+0000 7f18e782a640 1 --2- 172.21.15.12:0/1565881893 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f18c403e210 0x7f18c40406d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f18d002a360 tx=0x7f18d002e000 comp rx=0 tx=0).stop 2024-09-11T01:36:53.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.210+0000 7f18e782a640 10 monclient: shutdown 2024-09-11T01:36:53.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.210+0000 7f18e782a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:53.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.210+0000 7f18e782a640 1 -- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18e013b2f0 msgr2=0x7f18e01cd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:53.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.210+0000 7f18e782a640 1 --2- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18e013b2f0 0x7f18e01cd020 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f18d8001a20 tx=0x7f18d800a430 comp rx=0 tx=0).stop 2024-09-11T01:36:53.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.210+0000 7f18e5dc7640 1 -- 172.21.15.12:0/1565881893 reap_dead start 2024-09-11T01:36:53.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.210+0000 7f18e782a640 1 -- 172.21.15.12:0/1565881893 shutdown_connections 2024-09-11T01:36:53.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.210+0000 7f18e782a640 1 --2- 172.21.15.12:0/1565881893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f18e013b2f0 0x7f18e01cd020 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:53.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.210+0000 7f18e782a640 1 -- 172.21.15.12:0/1565881893 >> 172.21.15.12:0/1565881893 conn(0x7f18e00a36c0 msgr2=0x7f18e013dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:53.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.210+0000 7f18e782a640 1 -- 172.21.15.12:0/1565881893 shutdown_connections 2024-09-11T01:36:53.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.210+0000 7f18e782a640 1 -- 172.21.15.12:0/1565881893 wait complete. 2024-09-11T01:36:53.238 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:36:53.238 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:36:53.238 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:36:53.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.338+0000 7ff6f62e4640 10 monclient: get_monmap_and_config 2024-09-11T01:36:53.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.338+0000 7ff6f62e4640 10 monclient: build_initial_monmap 2024-09-11T01:36:53.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.338+0000 7ff6f62e4640 10 monclient: monmap: 2024-09-11T01:36:53.340 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:53.340 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:53.340 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:53.339390+0000 2024-09-11T01:36:53.340 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:53.339390+0000 2024-09-11T01:36:53.341 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:53.341 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:53.341 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:53.341 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:53.341 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:53.341 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:53.341 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:53.341 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:53.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.339+0000 7ff6f62e4640 1 Processor -- start 2024-09-11T01:36:53.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.339+0000 7ff6f62e4640 1 -- start start 2024-09-11T01:36:53.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.339+0000 7ff6f62e4640 10 monclient: init 2024-09-11T01:36:53.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.339+0000 7ff6f62e4640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:53.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.339+0000 7ff6f62e4640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-11T01:36:53.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.339+0000 7ff6f62e4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff6f0134ad0 addr v1:172.21.15.12:6789/0 2024-09-11T01:36:53.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.339+0000 7ff6f62e4640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff6f00a2200 0x7ff6f013a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:53.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.339+0000 7ff6f62e4640 10 monclient(hunting): picked mon.noname-c con 0x7ff6f00a2200 addr v2:172.21.15.71:3300/0 2024-09-11T01:36:53.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.339+0000 7ff6f62e4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff6f01355c0 0x7ff6f013c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:53.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.339+0000 7ff6f62e4640 10 monclient(hunting): picked mon.noname-a con 0x7ff6f01355c0 addr v2:172.21.15.12:3300/0 2024-09-11T01:36:53.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.339+0000 7ff6f62e4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6f009ed30 con 0x7ff6f0134ad0 2024-09-11T01:36:53.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.339+0000 7ff6f62e4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:53.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.339+0000 7ff6f62e4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff6f009f5d0 con 0x7ff6f01355c0 2024-09-11T01:36:53.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.339+0000 7ff6f62e4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:53.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.339+0000 7ff6f62e4640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7ff6f009f010 con 0x7ff6f00a2200 2024-09-11T01:36:53.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.339+0000 7ff6f62e4640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:53.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.339+0000 7ff6f62e4640 10 monclient(hunting): authenticate will time out at 1219.781954s 2024-09-11T01:36:53.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6f4881640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff6f01355c0 0x7ff6f013c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:53.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6f4881640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff6f01355c0 0x7ff6f013c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49280/0 (socket says 172.21.15.12:49280) 2024-09-11T01:36:53.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6effff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff6f0134ad0 0x7ff6f0134eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:48594/0 (socket says 172.21.15.12:48594) 2024-09-11T01:36:53.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6f4881640 1 -- 172.21.15.12:0/2936757248 learned_addr learned my addr 172.21.15.12:0/2936757248 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:53.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6f4881640 10 monclient(hunting): get_auth_request con 0x7ff6f01355c0 auth_method 0 2024-09-11T01:36:53.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6f4881640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:53.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6f4881640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:53.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6f4881640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:53.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6f4881640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:53.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6f4881640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:53.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6ef7fe640 1 --2- 172.21.15.12:0/2936757248 >> v2:172.21.15.71:3300/0 conn(0x7ff6f00a2200 0x7ff6f013a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:53.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6f4881640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:53.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6ef7fe640 10 monclient(hunting): get_auth_request con 0x7ff6f00a2200 auth_method 0 2024-09-11T01:36:53.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:53.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:53.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:53.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:53.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:53.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:53.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6f4881640 10 monclient(hunting): handle_auth_done global_id 54396 payload 274 2024-09-11T01:36:53.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6f4881640 1 -- 172.21.15.12:0/2936757248 >> v2:172.21.15.71:3300/0 conn(0x7ff6f00a2200 msgr2=0x7ff6f013a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:53.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6f4881640 1 --2- 172.21.15.12:0/2936757248 >> v2:172.21.15.71:3300/0 conn(0x7ff6f00a2200 0x7ff6f013a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:53.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6f4881640 1 -- 172.21.15.12:0/2936757248 >> v1:172.21.15.12:6789/0 conn(0x7ff6f0134ad0 legacy=0x7ff6f0134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:53.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6f4881640 10 monclient: _finish_hunting 0 2024-09-11T01:36:53.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6f4881640 1 monclient: found mon.noname-a 2024-09-11T01:36:53.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6f4881640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:53.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6f4881640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:53.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.340+0000 7ff6f4881640 1 -- 172.21.15.12:0/2936757248 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6f009f2f0 con 0x7ff6f01355c0 2024-09-11T01:36:53.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.343+0000 7ff6f4881640 1 -- 172.21.15.12:0/2936757248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6f01355c0 msgr2=0x7ff6f013c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:53.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.343+0000 7ff6f4881640 1 --2- 172.21.15.12:0/2936757248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6f01355c0 0x7ff6f013c550 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8009e50 tx=0x7ff6e802a5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e64878b8a6bd0d2e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:53.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.343+0000 7ff6eeffd640 1 -- 172.21.15.12:0/2936757248 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8036070 con 0x7ff6f01355c0 2024-09-11T01:36:53.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.343+0000 7ff6eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:53.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.343+0000 7ff6eeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:53.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.343+0000 7ff6eeffd640 10 monclient: dump: 2024-09-11T01:36:53.357 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:53.357 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:53.357 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:53.357 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:53.357 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:53.357 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:53.357 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:53.357 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:53.357 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:53.357 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:53.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.343+0000 7ff6eeffd640 10 monclient: _finish_auth 0 2024-09-11T01:36:53.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.343+0000 7ff6eeffd640 10 monclient: _check_auth_tickets 2024-09-11T01:36:53.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.343+0000 7ff6eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:53.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.343+0000 7ff6f62e4640 5 monclient: authenticate success, global_id 54396 2024-09-11T01:36:53.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.343+0000 7ff6f62e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:36:53.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.343+0000 7ff6eeffd640 1 -- 172.21.15.12:0/2936757248 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6e8007800 con 0x7ff6f01355c0 2024-09-11T01:36:53.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.343+0000 7ff6eeffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:53.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.343+0000 7ff6f62e4640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:53.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.343+0000 7ff6f62e4640 4 set_mon_vals no callback set 2024-09-11T01:36:53.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.343+0000 7ff6eeffd640 1 -- 172.21.15.12:0/2936757248 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8007bd0 con 0x7ff6f01355c0 2024-09-11T01:36:53.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.343+0000 7ff6eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:53.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.343+0000 7ff6eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:53.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.343+0000 7ff6eeffd640 10 monclient: dump: 2024-09-11T01:36:53.359 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:53.359 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:53.359 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:53.359 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:53.359 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:53.359 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:53.359 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:53.359 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:53.359 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:53.359 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:53.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.343+0000 7ff6f62e4640 10 monclient: shutdown 2024-09-11T01:36:53.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.343+0000 7ff6f62e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:53.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.343+0000 7ff6f62e4640 1 -- 172.21.15.12:0/2936757248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6f01355c0 msgr2=0x7ff6f013c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:53.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.343+0000 7ff6f62e4640 1 --2- 172.21.15.12:0/2936757248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6f01355c0 0x7ff6f013c550 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8009e50 tx=0x7ff6e802a5b0 comp rx=0 tx=0).stop 2024-09-11T01:36:53.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.344+0000 7ff6f62e4640 1 -- 172.21.15.12:0/2936757248 shutdown_connections 2024-09-11T01:36:53.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.344+0000 7ff6f62e4640 1 --2- 172.21.15.12:0/2936757248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6f01355c0 0x7ff6f013c550 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:53.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.344+0000 7ff6f62e4640 1 --2- 172.21.15.12:0/2936757248 >> v2:172.21.15.71:3300/0 conn(0x7ff6f00a2200 0x7ff6f013a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:53.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.344+0000 7ff6f62e4640 1 -- 172.21.15.12:0/2936757248 >> 172.21.15.12:0/2936757248 conn(0x7ff6f00a2f70 msgr2=0x7ff6f00a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:53.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.344+0000 7ff6f62e4640 1 -- 172.21.15.12:0/2936757248 shutdown_connections 2024-09-11T01:36:53.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.344+0000 7ff6f62e4640 1 -- 172.21.15.12:0/2936757248 wait complete. 2024-09-11T01:36:53.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.344+0000 7ff6f62e4640 10 monclient: build_initial_monmap 2024-09-11T01:36:53.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.344+0000 7ff6f62e4640 10 monclient: monmap: 2024-09-11T01:36:53.360 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:53.360 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:53.361 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:53.361 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:53.361 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:53.361 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:53.361 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:53.361 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:53.361 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:53.361 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:53.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.344+0000 7ff6f62e4640 1 Processor -- start 2024-09-11T01:36:53.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6f62e4640 1 -- start start 2024-09-11T01:36:53.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6f62e4640 10 monclient: init 2024-09-11T01:36:53.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6f62e4640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:53.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6f62e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:36:53.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6f62e4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f00a2200 0x7ff6f01ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:53.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6f62e4640 10 monclient(hunting): picked mon.noname-c con 0x7ff6f00a2200 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:53.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6f62e4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6f0134ad0 0x7ff6f0130fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:53.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6f62e4640 10 monclient(hunting): picked mon.noname-b con 0x7ff6f0134ad0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:53.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6f62e4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6f01355c0 0x7ff6f01c9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:53.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6f62e4640 10 monclient(hunting): picked mon.noname-a con 0x7ff6f01355c0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:53.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6f62e4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:53.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6f62e4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f00a0150 con 0x7ff6f01355c0 2024-09-11T01:36:53.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6f62e4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:53.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6f62e4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f009ffd0 con 0x7ff6f0134ad0 2024-09-11T01:36:53.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6f62e4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:53.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6f62e4640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff6f00a0c90 con 0x7ff6f00a2200 2024-09-11T01:36:53.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6f62e4640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:53.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6f62e4640 10 monclient(hunting): authenticate will time out at 1219.787747s 2024-09-11T01:36:53.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6effff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f00a2200 0x7ff6f01ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6effff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f00a2200 0x7ff6f01ccfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52842/0 (socket says 172.21.15.12:52842) 2024-09-11T01:36:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6effff640 1 -- 172.21.15.12:0/680372511 learned_addr learned my addr 172.21.15.12:0/680372511 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6effff640 10 monclient(hunting): get_auth_request con 0x7ff6f00a2200 auth_method 0 2024-09-11T01:36:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6effff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6f4881640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6f01355c0 0x7ff6f01c9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6f4881640 10 monclient(hunting): get_auth_request con 0x7ff6f01355c0 auth_method 0 2024-09-11T01:36:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6f4881640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6f4881640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6f4881640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6effff640 1 -- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f00a2200 msgr2=0x7ff6f01ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:36:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6effff640 1 -- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f00a2200 msgr2=0x7ff6f01ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:36:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6effff640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f00a2200 0x7ff6f01ccfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6effff640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f00a2200 0x7ff6f01ccfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6f4881640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6f4881640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.345+0000 7ff6f4881640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.346+0000 7ff6ef7fe640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6f0134ad0 0x7ff6f0130fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.346+0000 7ff6ef7fe640 10 monclient(hunting): get_auth_request con 0x7ff6f0134ad0 auth_method 0 2024-09-11T01:36:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.346+0000 7ff6ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.346+0000 7ff6ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.346+0000 7ff6ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.346+0000 7ff6f4881640 10 monclient(hunting): handle_auth_done global_id 54399 payload 824 2024-09-11T01:36:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.346+0000 7ff6f4881640 1 -- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f00a2200 msgr2=0x7ff6f01ccfe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:36:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.346+0000 7ff6f4881640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f00a2200 0x7ff6f01ccfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.346+0000 7ff6f4881640 1 -- 172.21.15.12:0/680372511 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6f0134ad0 msgr2=0x7ff6f0130fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.346+0000 7ff6f4881640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6f0134ad0 0x7ff6f0130fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.346+0000 7ff6f4881640 10 monclient: _finish_hunting 0 2024-09-11T01:36:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.346+0000 7ff6f4881640 1 monclient: found mon.noname-a 2024-09-11T01:36:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.346+0000 7ff6f4881640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.346+0000 7ff6f4881640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.346+0000 7ff6f4881640 1 -- 172.21.15.12:0/680372511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6f013e570 con 0x7ff6f01355c0 2024-09-11T01:36:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.348+0000 7ff6f4881640 1 -- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6f01355c0 msgr2=0x7ff6f01c9a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.348+0000 7ff6f4881640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6f01355c0 0x7ff6f01c9a80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8030000 tx=0x7ff6e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.348+0000 7ff6ecff9640 1 -- 172.21.15.12:0/680372511 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8036050 con 0x7ff6f01355c0 2024-09-11T01:36:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.348+0000 7ff6ecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.348+0000 7ff6ecff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.348+0000 7ff6ecff9640 10 monclient: dump: 2024-09-11T01:36:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:53.366 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.348+0000 7ff6ecff9640 10 monclient: _finish_auth 0 2024-09-11T01:36:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.348+0000 7ff6ecff9640 10 monclient: _check_auth_tickets 2024-09-11T01:36:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.348+0000 7ff6ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.348+0000 7ff6f62e4640 5 monclient: authenticate success, global_id 54399 2024-09-11T01:36:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.348+0000 7ff6f62e4640 10 monclient: _renew_subs 2024-09-11T01:36:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.348+0000 7ff6ecff9640 1 -- 172.21.15.12:0/680372511 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6e8042070 con 0x7ff6f01355c0 2024-09-11T01:36:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.348+0000 7ff6f62e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.348+0000 7ff6f62e4640 1 -- 172.21.15.12:0/680372511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6f011c980 con 0x7ff6f01355c0 2024-09-11T01:36:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.348+0000 7ff6ecff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.348+0000 7ff6ecff9640 1 -- 172.21.15.12:0/680372511 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8031d30 con 0x7ff6f01355c0 2024-09-11T01:36:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.348+0000 7ff6ecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.348+0000 7ff6ecff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.348+0000 7ff6ecff9640 10 monclient: dump: 2024-09-11T01:36:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:53.368 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.348+0000 7ff6f62e4640 10 monclient: _renew_subs 2024-09-11T01:36:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.348+0000 7ff6f62e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.348+0000 7ff6f62e4640 1 -- 172.21.15.12:0/680372511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6f013c4f0 con 0x7ff6f01355c0 2024-09-11T01:36:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.348+0000 7ff6ee7fc640 4 set_mon_vals no callback set 2024-09-11T01:36:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.349+0000 7ff6ecff9640 1 -- 172.21.15.12:0/680372511 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff6e8047460 con 0x7ff6f01355c0 2024-09-11T01:36:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.349+0000 7ff6f62e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.349+0000 7ff6f62e4640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 0x7ff6f01c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.349+0000 7ff6f62e4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.349+0000 7ff6f62e4640 1 -- 172.21.15.12:0/680372511 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff6f013bfa0 con 0x7ff6f013d050 2024-09-11T01:36:53.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.349+0000 7ff6effff640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 0x7ff6f01c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:53.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.349+0000 7ff6effff640 10 monclient: get_auth_request con 0x7ff6f013d050 auth_method 0 2024-09-11T01:36:53.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.349+0000 7ff6effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:53.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.349+0000 7ff6effff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:53.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.349+0000 7ff6effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:53.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.349+0000 7ff6ecff9640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff6dc03e240 0x7ff6dc040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:53.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.349+0000 7ff6effff640 1 -- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 msgr2=0x7ff6f01c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:36:53.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.349+0000 7ff6effff640 1 -- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 msgr2=0x7ff6f01c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:36:53.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.349+0000 7ff6effff640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 0x7ff6f01c9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:53.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.349+0000 7ff6effff640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 0x7ff6f01c9fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:53.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.349+0000 7ff6ecff9640 1 -- 172.21.15.12:0/680372511 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(253..253 src has 186..253) ==== 4847+0+0 (secure 0 0 0) 0x7ff6e8073d00 con 0x7ff6f01355c0 2024-09-11T01:36:53.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.350+0000 7ff6ef7fe640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff6dc03e240 0x7ff6dc040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:53.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.350+0000 7ff6ef7fe640 10 monclient: get_auth_request con 0x7ff6dc03e240 auth_method 0 2024-09-11T01:36:53.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.352+0000 7ff6ef7fe640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff6dc03e240 0x7ff6dc040700 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff6f00749e0 tx=0x7ff6e4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:53.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.550+0000 7ff6effff640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 0x7ff6f01c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:53.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.550+0000 7ff6effff640 10 monclient: get_auth_request con 0x7ff6f013d050 auth_method 0 2024-09-11T01:36:53.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.550+0000 7ff6effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:53.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.550+0000 7ff6effff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:53.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.550+0000 7ff6effff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:36:53.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.550+0000 7ff6effff640 1 -- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 msgr2=0x7ff6f01c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:36:53.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.550+0000 7ff6effff640 1 -- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 msgr2=0x7ff6f01c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:36:53.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.550+0000 7ff6effff640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 0x7ff6f01c9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:53.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.550+0000 7ff6effff640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 0x7ff6f01c9fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:36:53.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.951+0000 7ff6effff640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 0x7ff6f01c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:53.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.951+0000 7ff6effff640 10 monclient: get_auth_request con 0x7ff6f013d050 auth_method 0 2024-09-11T01:36:53.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.951+0000 7ff6effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:53.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.951+0000 7ff6effff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:53.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.951+0000 7ff6effff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:36:53.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.951+0000 7ff6effff640 1 -- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 msgr2=0x7ff6f01c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:36:53.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.951+0000 7ff6effff640 1 -- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 msgr2=0x7ff6f01c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:36:53.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.951+0000 7ff6effff640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 0x7ff6f01c9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:53.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:53.951+0000 7ff6effff640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 0x7ff6f01c9fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:36:54.163 INFO:tasks.rados.rados.0.smithi071.stdout:1098: done (0 left) 2024-09-11T01:36:54.163 INFO:tasks.rados.rados.0.smithi071.stdout:1099: copy_from oid 34 from oid 8 current snap is 116 2024-09-11T01:36:54.164 INFO:tasks.rados.rados.0.smithi071.stdout:1100: rollback oid 9 current snap is 116 2024-09-11T01:36:54.164 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 9 to 110 2024-09-11T01:36:54.164 INFO:tasks.rados.rados.0.smithi071.stdout:1101: write (excl) oid 39 current snap is 116 2024-09-11T01:36:54.164 INFO:tasks.rados.rados.0.smithi071.stdout:1101: seq_num 413 ranges {541291=560899,1884475=521471} 2024-09-11T01:36:54.169 INFO:tasks.rados.rados.0.smithi071.stdout:1101: writing smithi07134777-39 from 541291 to 1102190 tid 1 2024-09-11T01:36:54.177 INFO:tasks.rados.rados.0.smithi071.stdout:1101: writing smithi07134777-39 from 1884475 to 2405946 tid 2 2024-09-11T01:36:54.177 INFO:tasks.rados.rados.0.smithi071.stdout:1102: rollback oid 49 current snap is 116 2024-09-11T01:36:54.177 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 49 to 115 2024-09-11T01:36:54.177 INFO:tasks.rados.rados.0.smithi071.stdout:1100: finishing rollback tid 0 to smithi07134777-9 2024-09-11T01:36:54.177 INFO:tasks.rados.rados.0.smithi071.stdout:1100: finishing rollback tid 1 to smithi07134777-9 2024-09-11T01:36:54.177 INFO:tasks.rados.rados.0.smithi071.stdout:1100: finishing rollback tid 2 to smithi07134777-9 2024-09-11T01:36:54.177 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 136 (ObjNum 336 snap 96 seq_num 336) dirty exists 2024-09-11T01:36:54.177 INFO:tasks.rados.rados.0.smithi071.stdout:1100: done (3 left) 2024-09-11T01:36:54.177 INFO:tasks.rados.rados.0.smithi071.stdout:1103: copy_from oid 28 from oid 35 current snap is 116 2024-09-11T01:36:54.177 INFO:tasks.rados.rados.0.smithi071.stdout:1104: write oid 42 current snap is 116 2024-09-11T01:36:54.178 INFO:tasks.rados.rados.0.smithi071.stdout:1104: seq_num 414 ranges {558098=636148,1826604=665129} 2024-09-11T01:36:54.184 INFO:tasks.rados.rados.0.smithi071.stdout:1104: writing smithi07134777-42 from 558098 to 1194246 tid 1 2024-09-11T01:36:54.192 INFO:tasks.rados.rados.0.smithi071.stdout:1104: writing smithi07134777-42 from 1826604 to 2491733 tid 2 2024-09-11T01:36:54.193 INFO:tasks.rados.rados.0.smithi071.stdout:1105: read oid 18 snap 115 2024-09-11T01:36:54.193 INFO:tasks.rados.rados.0.smithi071.stdout:1105: expect (ObjNum 399 snap 115 seq_num 399) 2024-09-11T01:36:54.193 INFO:tasks.rados.rados.0.smithi071.stdout:1106: snap_create 2024-09-11T01:36:54.193 INFO:tasks.rados.rados.0.smithi071.stdout:1101: finishing write tid 1 to smithi07134777-39 2024-09-11T01:36:54.193 INFO:tasks.rados.rados.0.smithi071.stdout:1102: finishing rollback tid 0 to smithi07134777-49 2024-09-11T01:36:54.193 INFO:tasks.rados.rados.0.smithi071.stdout:1102: finishing rollback tid 1 to smithi07134777-49 2024-09-11T01:36:54.193 INFO:tasks.rados.rados.0.smithi071.stdout:1102: finishing rollback tid 2 to smithi07134777-49 2024-09-11T01:36:54.193 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 232 (ObjNum 402 snap 115 seq_num 402) dirty exists 2024-09-11T01:36:54.193 INFO:tasks.rados.rados.0.smithi071.stdout:1101: finishing write tid 2 to smithi07134777-39 2024-09-11T01:36:54.193 INFO:tasks.rados.rados.0.smithi071.stdout:1101: finishing write tid 3 to smithi07134777-39 2024-09-11T01:36:54.193 INFO:tasks.rados.rados.0.smithi071.stdout:1101: finishing write tid 4 to smithi07134777-39 2024-09-11T01:36:54.193 INFO:tasks.rados.rados.0.smithi071.stdout:1101: oid 39 updating version 0 to 305 2024-09-11T01:36:54.193 INFO:tasks.rados.rados.0.smithi071.stdout:1101: oid 39 updating version 305 to 307 2024-09-11T01:36:54.194 INFO:tasks.rados.rados.0.smithi071.stdout:1101: oid 39 version 307 is already newer than 306 2024-09-11T01:36:54.194 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 307 (ObjNum 413 snap 116 seq_num 413) dirty exists 2024-09-11T01:36:54.194 INFO:tasks.rados.rados.0.smithi071.stdout:1101: left oid 39 (ObjNum 413 snap 116 seq_num 413) 2024-09-11T01:36:54.194 INFO:tasks.rados.rados.0.smithi071.stdout:1099: finishing copy_from racing read to smithi07134777-34 2024-09-11T01:36:54.194 INFO:tasks.rados.rados.0.smithi071.stdout:1099: finishing copy_from to smithi07134777-34 2024-09-11T01:36:54.194 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 429 (ObjNum 409 snap 115 seq_num 409) dirty exists 2024-09-11T01:36:54.196 INFO:tasks.rados.rados.0.smithi071.stdout:1103: finishing copy_from racing read to smithi07134777-28 2024-09-11T01:36:54.199 INFO:tasks.rados.rados.0.smithi071.stdout:1103: finishing copy_from to smithi07134777-28 2024-09-11T01:36:54.200 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 218 (ObjNum 312 snap 87 seq_num 312) dirty exists 2024-09-11T01:36:54.201 INFO:tasks.rados.rados.0.smithi071.stdout:1104: finishing write tid 1 to smithi07134777-42 2024-09-11T01:36:54.203 INFO:tasks.rados.rados.0.smithi071.stdout:1104: finishing write tid 2 to smithi07134777-42 2024-09-11T01:36:54.203 INFO:tasks.rados.rados.0.smithi071.stdout:1104: finishing write tid 3 to smithi07134777-42 2024-09-11T01:36:54.203 INFO:tasks.rados.rados.0.smithi071.stdout:1104: finishing write tid 4 to smithi07134777-42 2024-09-11T01:36:54.204 INFO:tasks.rados.rados.0.smithi071.stdout:1104: oid 42 updating version 0 to 405 2024-09-11T01:36:54.204 INFO:tasks.rados.rados.0.smithi071.stdout:1104: oid 42 updating version 405 to 407 2024-09-11T01:36:54.204 INFO:tasks.rados.rados.0.smithi071.stdout:1104: oid 42 version 407 is already newer than 406 2024-09-11T01:36:54.204 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 407 (ObjNum 414 snap 116 seq_num 414) dirty exists 2024-09-11T01:36:54.204 INFO:tasks.rados.rados.0.smithi071.stdout:1104: left oid 42 (ObjNum 414 snap 116 seq_num 414) 2024-09-11T01:36:54.207 INFO:tasks.rados.rados.0.smithi071.stdout:1105: expect (ObjNum 399 snap 115 seq_num 399) 2024-09-11T01:36:54.217 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1888660 2024-09-11T01:36:54.236 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1851621 2024-09-11T01:36:54.246 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:36:54.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:54.752+0000 7ff6effff640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 0x7ff6f01c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:54.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:54.752+0000 7ff6effff640 10 monclient: get_auth_request con 0x7ff6f013d050 auth_method 0 2024-09-11T01:36:54.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:54.752+0000 7ff6effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:54.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:54.752+0000 7ff6effff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:54.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:54.752+0000 7ff6effff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:36:54.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:54.752+0000 7ff6effff640 1 -- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 msgr2=0x7ff6f01c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:36:54.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:54.752+0000 7ff6effff640 1 -- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 msgr2=0x7ff6f01c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:36:54.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:54.752+0000 7ff6effff640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 0x7ff6f01c9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:54.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:54.752+0000 7ff6effff640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 0x7ff6f01c9fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:36:55.163 INFO:tasks.rados.rados.0.smithi071.stdout:1099: done (6 left) 2024-09-11T01:36:55.163 INFO:tasks.rados.rados.0.smithi071.stdout:1101: done (5 left) 2024-09-11T01:36:55.163 INFO:tasks.rados.rados.0.smithi071.stdout:1102: done (4 left) 2024-09-11T01:36:55.163 INFO:tasks.rados.rados.0.smithi071.stdout:1103: done (3 left) 2024-09-11T01:36:55.163 INFO:tasks.rados.rados.0.smithi071.stdout:1104: done (2 left) 2024-09-11T01:36:55.163 INFO:tasks.rados.rados.0.smithi071.stdout:1105: done (1 left) 2024-09-11T01:36:55.163 INFO:tasks.rados.rados.0.smithi071.stdout:1106: done (0 left) 2024-09-11T01:36:55.163 INFO:tasks.rados.rados.0.smithi071.stdout:1107: write (excl) oid 14 current snap is 117 2024-09-11T01:36:55.163 INFO:tasks.rados.rados.0.smithi071.stdout:1107: seq_num 415 ranges {574905=711397,1885086=525140,2930189=44716} 2024-09-11T01:36:55.169 INFO:tasks.rados.rados.0.smithi071.stdout:1107: writing smithi07134777-14 from 574905 to 1286302 tid 1 2024-09-11T01:36:55.176 INFO:tasks.rados.rados.0.smithi071.stdout:1107: writing smithi07134777-14 from 1885086 to 2410226 tid 2 2024-09-11T01:36:55.179 INFO:tasks.rados.rados.0.smithi071.stdout:1107: writing smithi07134777-14 from 2930189 to 2974905 tid 3 2024-09-11T01:36:55.179 INFO:tasks.rados.rados.0.smithi071.stdout:1108: write oid 7 current snap is 117 2024-09-11T01:36:55.179 INFO:tasks.rados.rados.0.smithi071.stdout:1108: seq_num 416 ranges {591712=786646,1943568=668798} 2024-09-11T01:36:55.187 INFO:tasks.rados.rados.0.smithi071.stdout:1108: writing smithi07134777-7 from 591712 to 1378358 tid 1 2024-09-11T01:36:55.194 INFO:tasks.rados.rados.0.smithi071.stdout:1108: writing smithi07134777-7 from 1943568 to 2612366 tid 2 2024-09-11T01:36:55.194 INFO:tasks.rados.rados.0.smithi071.stdout:1109: write (excl) oid 21 current snap is 117 2024-09-11T01:36:55.194 INFO:tasks.rados.rados.0.smithi071.stdout:1109: seq_num 417 ranges {608519=461895,1602050=528809,2585035=423484} 2024-09-11T01:36:55.200 INFO:tasks.rados.rados.0.smithi071.stdout:1109: writing smithi07134777-21 from 608519 to 1070414 tid 1 2024-09-11T01:36:55.206 INFO:tasks.rados.rados.0.smithi071.stdout:1109: writing smithi07134777-21 from 1602050 to 2130859 tid 2 2024-09-11T01:36:55.213 INFO:tasks.rados.rados.0.smithi071.stdout:1109: writing smithi07134777-21 from 2585035 to 3008519 tid 3 2024-09-11T01:36:55.214 INFO:tasks.rados.rados.0.smithi071.stdout:1110: copy_from oid 26 from oid 29 current snap is 117 2024-09-11T01:36:55.214 INFO:tasks.rados.rados.0.smithi071.stdout:1107: finishing write tid 1 to smithi07134777-14 2024-09-11T01:36:55.214 INFO:tasks.rados.rados.0.smithi071.stdout:1107: finishing write tid 2 to smithi07134777-14 2024-09-11T01:36:55.214 INFO:tasks.rados.rados.0.smithi071.stdout:1107: finishing write tid 3 to smithi07134777-14 2024-09-11T01:36:55.214 INFO:tasks.rados.rados.0.smithi071.stdout:1107: finishing write tid 4 to smithi07134777-14 2024-09-11T01:36:55.215 INFO:tasks.rados.rados.0.smithi071.stdout:1107: finishing write tid 5 to smithi07134777-14 2024-09-11T01:36:55.215 INFO:tasks.rados.rados.0.smithi071.stdout:1107: oid 14 updating version 0 to 584 2024-09-11T01:36:55.215 INFO:tasks.rados.rados.0.smithi071.stdout:1107: oid 14 updating version 584 to 585 2024-09-11T01:36:55.215 INFO:tasks.rados.rados.0.smithi071.stdout:1107: oid 14 updating version 585 to 587 2024-09-11T01:36:55.215 INFO:tasks.rados.rados.0.smithi071.stdout:1107: oid 14 version 587 is already newer than 586 2024-09-11T01:36:55.215 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 587 (ObjNum 415 snap 117 seq_num 415) dirty exists 2024-09-11T01:36:55.216 INFO:tasks.rados.rados.0.smithi071.stdout:1107: left oid 14 (ObjNum 415 snap 117 seq_num 415) 2024-09-11T01:36:55.216 INFO:tasks.rados.rados.0.smithi071.stdout:1108: finishing write tid 1 to smithi07134777-7 2024-09-11T01:36:55.216 INFO:tasks.rados.rados.0.smithi071.stdout:1108: finishing write tid 2 to smithi07134777-7 2024-09-11T01:36:55.216 INFO:tasks.rados.rados.0.smithi071.stdout:1108: finishing write tid 3 to smithi07134777-7 2024-09-11T01:36:55.216 INFO:tasks.rados.rados.0.smithi071.stdout:1108: finishing write tid 4 to smithi07134777-7 2024-09-11T01:36:55.216 INFO:tasks.rados.rados.0.smithi071.stdout:1108: oid 7 updating version 0 to 590 2024-09-11T01:36:55.217 INFO:tasks.rados.rados.0.smithi071.stdout:1108: oid 7 version 590 is already newer than 588 2024-09-11T01:36:55.217 INFO:tasks.rados.rados.0.smithi071.stdout:1108: oid 7 version 590 is already newer than 589 2024-09-11T01:36:55.217 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 590 (ObjNum 416 snap 117 seq_num 416) dirty exists 2024-09-11T01:36:55.217 INFO:tasks.rados.rados.0.smithi071.stdout:1108: left oid 7 (ObjNum 416 snap 117 seq_num 416) 2024-09-11T01:36:55.217 INFO:tasks.rados.rados.0.smithi071.stdout:1109: finishing write tid 1 to smithi07134777-21 2024-09-11T01:36:55.218 INFO:tasks.rados.rados.0.smithi071.stdout:1109: finishing write tid 2 to smithi07134777-21 2024-09-11T01:36:55.218 INFO:tasks.rados.rados.0.smithi071.stdout:1107: done (3 left) 2024-09-11T01:36:55.218 INFO:tasks.rados.rados.0.smithi071.stdout:1108: done (2 left) 2024-09-11T01:36:55.218 INFO:tasks.rados.rados.0.smithi071.stdout:1111: snap_remove snap 115 2024-09-11T01:36:56.166 INFO:tasks.rados.rados.0.smithi071.stdout:1111: done (2 left) 2024-09-11T01:36:56.167 INFO:tasks.rados.rados.0.smithi071.stdout:1112: write (excl) oid 28 current snap is 117 2024-09-11T01:36:56.167 INFO:tasks.rados.rados.0.smithi071.stdout:1109: finishing write tid 3 to smithi07134777-21 2024-09-11T01:36:56.167 INFO:tasks.rados.rados.0.smithi071.stdout:1112: seq_num 418 ranges {625326=537144,1944179=672467} 2024-09-11T01:36:56.173 INFO:tasks.rados.rados.0.smithi071.stdout:1112: writing smithi07134777-28 from 625326 to 1162470 tid 1 2024-09-11T01:36:56.181 INFO:tasks.rados.rados.0.smithi071.stdout:1112: writing smithi07134777-28 from 1944179 to 2616646 tid 2 2024-09-11T01:36:56.181 INFO:tasks.rados.rados.0.smithi071.stdout:1113: snap_remove snap 110 2024-09-11T01:36:56.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.346+0000 7ff6d77fe640 10 monclient: tick 2024-09-11T01:36:56.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.346+0000 7ff6d77fe640 10 monclient: _check_auth_tickets 2024-09-11T01:36:56.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.346+0000 7ff6d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:56.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.346+0000 7ff6d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:56.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.354+0000 7ff6effff640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 0x7ff6f01c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:56.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.354+0000 7ff6effff640 10 monclient: get_auth_request con 0x7ff6f013d050 auth_method 0 2024-09-11T01:36:56.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.354+0000 7ff6effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:56.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.354+0000 7ff6effff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:56.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.354+0000 7ff6effff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:36:56.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.354+0000 7ff6effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:56.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.354+0000 7ff6effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:56.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.354+0000 7ff6effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:56.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.354+0000 7ff6effff640 10 monclient(hunting): handle_auth_done global_id 84101 payload 824 2024-09-11T01:36:56.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.354+0000 7ff6effff640 1 -- 172.21.15.12:0/680372511 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff6f00a26d0 con 0x7ff6f013d050 2024-09-11T01:36:56.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.357+0000 7ff6effff640 1 -- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 msgr2=0x7ff6f01c9fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:56.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.357+0000 7ff6effff640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 0x7ff6f01c9fc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff6e00128c0 tx=0x7ff6e0012d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:56.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.357+0000 7ff6ecff9640 1 -- 172.21.15.12:0/680372511 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e001c400 con 0x7ff6f013d050 2024-09-11T01:36:56.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.357+0000 7ff6ecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:56.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.357+0000 7ff6ecff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:56.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.357+0000 7ff6ecff9640 10 monclient: dump: 2024-09-11T01:36:56.359 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:56.359 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:56.359 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:56.359 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:56.359 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:56.359 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:56.360 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:56.360 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:56.360 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:56.360 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:56.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.357+0000 7ff6ecff9640 1 -- 172.21.15.12:0/680372511 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff6e0014400 con 0x7ff6f013d050 2024-09-11T01:36:56.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.357+0000 7ff6ecff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:56.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.357+0000 7ff6ecff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:56.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.357+0000 7ff6ecff9640 1 -- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 msgr2=0x7ff6f01c9fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:56.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.357+0000 7ff6ecff9640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 0x7ff6f01c9fc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff6e00128c0 tx=0x7ff6e0012d80 comp rx=0 tx=0).stop 2024-09-11T01:36:56.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.357+0000 7ff6ecff9640 1 -- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 msgr2=0x7ff6f01c9fc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:56.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.357+0000 7ff6ecff9640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f013d050 0x7ff6f01c9fc0 secure :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff6e00128c0 tx=0x7ff6e0012d80 comp rx=0 tx=0).stop 2024-09-11T01:36:56.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.364+0000 7ff6f62e4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:36:56.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.364+0000 7ff6f62e4640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f0131500 0x7ff6f01ca500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:56.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.364+0000 7ff6f62e4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:56.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.364+0000 7ff6f62e4640 1 -- 172.21.15.12:0/680372511 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff6f006d480 con 0x7ff6f0131500 2024-09-11T01:36:56.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.364+0000 7ff6effff640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f0131500 0x7ff6f01ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:56.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.365+0000 7ff6effff640 10 monclient: get_auth_request con 0x7ff6f0131500 auth_method 0 2024-09-11T01:36:56.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.365+0000 7ff6effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:56.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.365+0000 7ff6effff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:56.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.365+0000 7ff6effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:56.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.365+0000 7ff6effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:56.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.365+0000 7ff6effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:56.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.365+0000 7ff6effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:56.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.365+0000 7ff6effff640 10 monclient(hunting): handle_auth_done global_id 84104 payload 824 2024-09-11T01:36:56.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.365+0000 7ff6effff640 1 -- 172.21.15.12:0/680372511 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff6f013e070 con 0x7ff6f0131500 2024-09-11T01:36:56.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.367+0000 7ff6effff640 1 -- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f0131500 msgr2=0x7ff6f01ca500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:56.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.367+0000 7ff6effff640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f0131500 0x7ff6f01ca500 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff6e001b000 tx=0x7ff6e0004930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:56.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.367+0000 7ff6ecff9640 1 -- 172.21.15.12:0/680372511 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e0021980 con 0x7ff6f0131500 2024-09-11T01:36:56.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.367+0000 7ff6ecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:56.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.367+0000 7ff6ecff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:56.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.367+0000 7ff6ecff9640 10 monclient: dump: 2024-09-11T01:36:56.370 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:56.370 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:56.370 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:56.370 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:56.370 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:56.371 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:56.371 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:56.371 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:56.371 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:56.371 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:56.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.368+0000 7ff6ecff9640 1 -- 172.21.15.12:0/680372511 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4540 (secure 0 0 0) 0x7ff6e0029410 con 0x7ff6f0131500 2024-09-11T01:36:56.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.368+0000 7ff6ecff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:36:56.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.368+0000 7ff6ecff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:56.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.368+0000 7ff6ecff9640 1 -- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f0131500 msgr2=0x7ff6f01ca500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:56.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.368+0000 7ff6ecff9640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f0131500 0x7ff6f01ca500 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff6e001b000 tx=0x7ff6e0004930 comp rx=0 tx=0).stop 2024-09-11T01:36:56.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.368+0000 7ff6ecff9640 1 -- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f0131500 msgr2=0x7ff6f01ca500 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:56.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.368+0000 7ff6ecff9640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6f0131500 0x7ff6f01ca500 secure :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff6e001b000 tx=0x7ff6e0004930 comp rx=0 tx=0).stop 2024-09-11T01:36:56.372 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:36:56.372 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:36:56.372 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:36:56.372 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:36:56.372 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 80, 2024-09-11T01:36:56.372 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 10390, 2024-09-11T01:36:56.373 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:36:56.373 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:36:56.373 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:36:56.373 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:36:56.373 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:56.373 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 4, 2024-09-11T01:36:56.373 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:56.373 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:36:56.373 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:36:56.373 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:56.373 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:56.374 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:56.374 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:56.374 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:56.374 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:56.374 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:56.374 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:56.374 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:56.374 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:56.374 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:56.374 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:56.374 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:36:56.375 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:36:56.375 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:56.375 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:56.375 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:56.375 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:56.375 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:56.375 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:56.375 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:56.375 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:56.375 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:56.375 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:56.376 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:56.376 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:56.376 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:56.376 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:36:56.376 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:36:56.376 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:36:56.376 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:36:56.376 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:36:56.376 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:36:56.376 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:56.377 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:56.377 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:36:56.377 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:36:56.377 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:36:56.377 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:36:56.377 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:36:56.377 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:36:56.377 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:36:56.377 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:56.377 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:36:56.377 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:56.378 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:56.378 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:56.378 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:56.378 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:56.378 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:56.378 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:56.378 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:56.378 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:56.378 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:56.378 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:56.378 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:56.378 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:36:56.379 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:56.379 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:36:56.379 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:56.379 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:36:56.379 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:36:56.379 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:56.379 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:56.379 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:56.379 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:56.379 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:36:56.379 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:56.379 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:56.380 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:56.380 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:56.380 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:36:56.380 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:56.380 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:56.380 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:56.380 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:56.380 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:36:56.380 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:36:56.380 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:56.380 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:56.380 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:56.381 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:56.381 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:56.381 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:36:56.381 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:36:56.381 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:56.381 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:56.381 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:56.381 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:56.381 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:36:56.381 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:56.382 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:56.382 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:56.382 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:56.382 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:36:56.382 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:56.382 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:56.382 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:56.382 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:56.382 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:36:56.382 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:36:56.382 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:56.383 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:56.383 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:56.383 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:56.383 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:56.383 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:36:56.383 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:36:56.383 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:56.383 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:56.383 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:56.383 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:56.383 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:36:56.384 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:56.384 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:56.384 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:56.384 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:56.384 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:36:56.384 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:56.384 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:56.384 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:56.384 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:56.384 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:36:56.384 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:36:56.384 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:56.385 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:56.385 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:56.385 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:56.385 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:56.385 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:56.385 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:36:56.385 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:36:56.385 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:56.385 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:56.385 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:56.385 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:36:56.386 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:56.386 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:56.386 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:36:56.386 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:56.386 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:56.386 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:56.386 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:36:56.386 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:56.386 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:56.386 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:56.386 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:36:56.387 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:36:56.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.368+0000 7ff6f62e4640 1 -- 172.21.15.12:0/680372511 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff6dc03e240 msgr2=0x7ff6dc040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:56.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.368+0000 7ff6f62e4640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff6dc03e240 0x7ff6dc040700 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff6f00749e0 tx=0x7ff6e4008040 comp rx=0 tx=0).stop 2024-09-11T01:36:56.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.368+0000 7ff6f62e4640 10 monclient: shutdown 2024-09-11T01:36:56.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.368+0000 7ff6f62e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:56.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.368+0000 7ff6f62e4640 1 -- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6f01355c0 msgr2=0x7ff6f01c9a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:56.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.368+0000 7ff6f62e4640 1 --2- 172.21.15.12:0/680372511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6f01355c0 0x7ff6f01c9a80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8030000 tx=0x7ff6e8004a40 comp rx=0 tx=0).stop 2024-09-11T01:36:56.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.368+0000 7ff6f4881640 1 -- 172.21.15.12:0/680372511 reap_dead start 2024-09-11T01:36:56.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.368+0000 7ff6f4881640 1 -- 172.21.15.12:0/680372511 reap_dead start 2024-09-11T01:36:56.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.368+0000 7ff6f62e4640 1 -- 172.21.15.12:0/680372511 shutdown_connections 2024-09-11T01:36:56.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.368+0000 7ff6f62e4640 1 -- 172.21.15.12:0/680372511 >> 172.21.15.12:0/680372511 conn(0x7ff6f00a2f70 msgr2=0x7ff6f009d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:56.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.368+0000 7ff6f62e4640 1 -- 172.21.15.12:0/680372511 shutdown_connections 2024-09-11T01:36:56.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.368+0000 7ff6f62e4640 1 -- 172.21.15.12:0/680372511 wait complete. 2024-09-11T01:36:56.389 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:36:56.390 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:36:56.390 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:36:56.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.499+0000 7efc3658a640 10 monclient: get_monmap_and_config 2024-09-11T01:36:56.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.499+0000 7efc3658a640 10 monclient: build_initial_monmap 2024-09-11T01:36:56.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.499+0000 7efc3658a640 10 monclient: monmap: 2024-09-11T01:36:56.501 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:56.501 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:56.501 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:56.500409+0000 2024-09-11T01:36:56.502 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:56.500409+0000 2024-09-11T01:36:56.502 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:56.502 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:56.502 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:56.502 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:56.502 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:56.502 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:56.502 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:56.502 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:56.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.500+0000 7efc3658a640 1 Processor -- start 2024-09-11T01:36:56.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.500+0000 7efc3658a640 1 -- start start 2024-09-11T01:36:56.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.500+0000 7efc3658a640 10 monclient: init 2024-09-11T01:36:56.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.500+0000 7efc3658a640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:56.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.500+0000 7efc3658a640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-11T01:36:56.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc3658a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efc300a2e20 addr v1:172.21.15.71:6789/0 2024-09-11T01:36:56.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc3658a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc300a4970 0x7efc3006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:56.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc3658a640 10 monclient(hunting): picked mon.noname-b con 0x7efc300a4970 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:56.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc3658a640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7efc3006f910 0x7efc3013f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:56.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc3658a640 10 monclient(hunting): picked mon.noname-c con 0x7efc3006f910 addr v2:172.21.15.71:3300/0 2024-09-11T01:36:56.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc3658a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc3006f4f0 con 0x7efc300a2e20 2024-09-11T01:36:56.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc3658a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:56.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc3658a640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7efc3006ef30 con 0x7efc3006f910 2024-09-11T01:36:56.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc3658a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:56.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc3658a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7efc3006ec50 con 0x7efc300a4970 2024-09-11T01:36:56.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc3658a640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:56.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc3658a640 10 monclient(hunting): authenticate will time out at 1222.943086s 2024-09-11T01:36:56.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc2f7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc300a4970 0x7efc3006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:56.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc2f7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc300a4970 0x7efc3006ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:34560/0 (socket says 172.21.15.12:34560) 2024-09-11T01:36:56.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc2f7fe640 1 -- 172.21.15.12:0/2752615146 learned_addr learned my addr 172.21.15.12:0/2752615146 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:56.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc2f7fe640 10 monclient(hunting): get_auth_request con 0x7efc300a4970 auth_method 0 2024-09-11T01:36:56.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:56.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:56.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:56.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:56.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:56.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:56.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc2f7fe640 10 monclient(hunting): handle_auth_done global_id 84107 payload 274 2024-09-11T01:36:56.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc34b27640 1 --2- 172.21.15.12:0/2752615146 >> v2:172.21.15.71:3300/0 conn(0x7efc3006f910 0x7efc3013f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:56.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc2f7fe640 1 -- 172.21.15.12:0/2752615146 >> v2:172.21.15.71:3300/0 conn(0x7efc3006f910 msgr2=0x7efc3013f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:56.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc2f7fe640 1 --2- 172.21.15.12:0/2752615146 >> v2:172.21.15.71:3300/0 conn(0x7efc3006f910 0x7efc3013f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:56.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc2f7fe640 1 -- 172.21.15.12:0/2752615146 >> v1:172.21.15.71:6789/0 conn(0x7efc300a2e20 legacy=0x7efc300a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:56.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc2f7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:36:56.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc2f7fe640 1 monclient: found mon.noname-b 2024-09-11T01:36:56.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:56.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc2f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:36:56.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc2f7fe640 1 -- 172.21.15.12:0/2752615146 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc300a3910 con 0x7efc300a4970 2024-09-11T01:36:56.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc34b27640 10 monclient: get_auth_request con 0x7efc3006f910 auth_method 0 2024-09-11T01:36:56.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.501+0000 7efc34b27640 1 --2- 172.21.15.12:0/2752615146 >> v2:172.21.15.71:3300/0 conn(0x7efc3006f910 0x7efc3013f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:36:56.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.504+0000 7efc2f7fe640 1 -- 172.21.15.12:0/2752615146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc300a4970 msgr2=0x7efc3006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:56.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.504+0000 7efc2f7fe640 1 --2- 172.21.15.12:0/2752615146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc300a4970 0x7efc3006ded0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efc24009870 tx=0x7efc2402a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6db300034ec18448 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:56.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.504+0000 7efc2effd640 1 -- 172.21.15.12:0/2752615146 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc24037070 con 0x7efc300a4970 2024-09-11T01:36:56.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.504+0000 7efc2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:56.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.504+0000 7efc2effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:36:56.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.504+0000 7efc2effd640 10 monclient: dump: 2024-09-11T01:36:56.521 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:56.521 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:56.521 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:56.521 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:56.521 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:56.521 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:56.521 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:56.521 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:56.521 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:56.521 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:56.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.504+0000 7efc2effd640 10 monclient: _finish_auth 0 2024-09-11T01:36:56.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.504+0000 7efc2effd640 10 monclient: _check_auth_tickets 2024-09-11T01:36:56.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.504+0000 7efc2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:56.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.504+0000 7efc2effd640 1 -- 172.21.15.12:0/2752615146 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc24032640 con 0x7efc300a4970 2024-09-11T01:36:56.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.504+0000 7efc3658a640 5 monclient: authenticate success, global_id 84107 2024-09-11T01:36:56.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.504+0000 7efc3658a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:36:56.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.504+0000 7efc2effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:56.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.504+0000 7efc2effd640 1 -- 172.21.15.12:0/2752615146 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc24032a10 con 0x7efc300a4970 2024-09-11T01:36:56.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.504+0000 7efc3658a640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:56.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.504+0000 7efc3658a640 4 set_mon_vals no callback set 2024-09-11T01:36:56.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.504+0000 7efc2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:56.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.504+0000 7efc2effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:56.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.504+0000 7efc2effd640 10 monclient: dump: 2024-09-11T01:36:56.523 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:56.523 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:56.523 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:56.523 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:56.523 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:56.523 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:56.523 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:56.523 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:56.523 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:56.523 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:56.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.505+0000 7efc3658a640 10 monclient: shutdown 2024-09-11T01:36:56.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.505+0000 7efc3658a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:56.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.505+0000 7efc3658a640 1 -- 172.21.15.12:0/2752615146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc300a4970 msgr2=0x7efc3006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:56.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.505+0000 7efc3658a640 1 --2- 172.21.15.12:0/2752615146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc300a4970 0x7efc3006ded0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efc24009870 tx=0x7efc2402a0a0 comp rx=0 tx=0).stop 2024-09-11T01:36:56.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.505+0000 7efc3658a640 1 -- 172.21.15.12:0/2752615146 shutdown_connections 2024-09-11T01:36:56.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.505+0000 7efc3658a640 1 --2- 172.21.15.12:0/2752615146 >> v2:172.21.15.71:3300/0 conn(0x7efc3006f910 0x7efc3013f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:56.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.505+0000 7efc3658a640 1 --2- 172.21.15.12:0/2752615146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc300a4970 0x7efc3006ded0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:56.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.505+0000 7efc3658a640 1 -- 172.21.15.12:0/2752615146 >> 172.21.15.12:0/2752615146 conn(0x7efc300a8600 msgr2=0x7efc300a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:56.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.505+0000 7efc3658a640 1 -- 172.21.15.12:0/2752615146 shutdown_connections 2024-09-11T01:36:56.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.505+0000 7efc3658a640 1 -- 172.21.15.12:0/2752615146 wait complete. 2024-09-11T01:36:56.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.505+0000 7efc3658a640 10 monclient: build_initial_monmap 2024-09-11T01:36:56.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.505+0000 7efc3658a640 10 monclient: monmap: 2024-09-11T01:36:56.525 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:56.525 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:56.525 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:56.525 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:56.525 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:56.525 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:56.525 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:56.525 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:56.525 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:56.526 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:56.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.506+0000 7efc3658a640 1 Processor -- start 2024-09-11T01:36:56.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.506+0000 7efc3658a640 1 -- start start 2024-09-11T01:36:56.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.506+0000 7efc3658a640 10 monclient: init 2024-09-11T01:36:56.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.506+0000 7efc3658a640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:56.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.506+0000 7efc3658a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:36:56.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.506+0000 7efc3658a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc3006f910 0x7efc3013b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:56.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.506+0000 7efc3658a640 10 monclient(hunting): picked mon.noname-c con 0x7efc3006f910 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:56.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.506+0000 7efc3658a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc300a2e20 0x7efc3013b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:56.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.506+0000 7efc3658a640 10 monclient(hunting): picked mon.noname-b con 0x7efc300a2e20 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:56.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.506+0000 7efc3658a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc300a4970 0x7efc3013bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:56.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.506+0000 7efc3658a640 10 monclient(hunting): picked mon.noname-a con 0x7efc300a4970 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:56.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.506+0000 7efc3658a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:56.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.506+0000 7efc3658a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7efc3013fd00 con 0x7efc300a4970 2024-09-11T01:36:56.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.506+0000 7efc3658a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:56.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.506+0000 7efc3658a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7efc3013fb80 con 0x7efc300a2e20 2024-09-11T01:36:56.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.506+0000 7efc3658a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:56.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.506+0000 7efc3658a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7efc3013fe80 con 0x7efc3006f910 2024-09-11T01:36:56.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.506+0000 7efc3658a640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:56.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.506+0000 7efc3658a640 10 monclient(hunting): authenticate will time out at 1222.948988s 2024-09-11T01:36:56.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc2ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc3006f910 0x7efc3013b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:56.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc2ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc3006f910 0x7efc3013b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:34572/0 (socket says 172.21.15.12:34572) 2024-09-11T01:36:56.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc2ffff640 1 -- 172.21.15.12:0/488371829 learned_addr learned my addr 172.21.15.12:0/488371829 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:56.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc2ffff640 10 monclient(hunting): get_auth_request con 0x7efc3006f910 auth_method 0 2024-09-11T01:36:56.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:56.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:56.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:56.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc34b27640 1 --2- 172.21.15.12:0/488371829 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc300a4970 0x7efc3013bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:56.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc2f7fe640 1 --2- 172.21.15.12:0/488371829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc300a2e20 0x7efc3013b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:56.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:56.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:56.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:56.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc34b27640 10 monclient(hunting): get_auth_request con 0x7efc300a4970 auth_method 0 2024-09-11T01:36:56.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc34b27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:56.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc34b27640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:56.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc34b27640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:56.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc2f7fe640 10 monclient(hunting): get_auth_request con 0x7efc300a2e20 auth_method 0 2024-09-11T01:36:56.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:56.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:56.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:56.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc2ffff640 1 -- 172.21.15.12:0/488371829 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc3006f910 msgr2=0x7efc3013b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:36:56.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc2ffff640 1 -- 172.21.15.12:0/488371829 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc3006f910 msgr2=0x7efc3013b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:36:56.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc2ffff640 1 --2- 172.21.15.12:0/488371829 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc3006f910 0x7efc3013b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:36:56.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc2ffff640 1 --2- 172.21.15.12:0/488371829 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc3006f910 0x7efc3013b260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:56.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc34b27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:56.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc34b27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:56.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc34b27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:56.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:56.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:56.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:56.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc34b27640 10 monclient(hunting): handle_auth_done global_id 54402 payload 824 2024-09-11T01:36:56.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc34b27640 1 -- 172.21.15.12:0/488371829 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc3006f910 msgr2=0x7efc3013b260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:36:56.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc34b27640 1 --2- 172.21.15.12:0/488371829 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc3006f910 0x7efc3013b260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:56.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc34b27640 1 -- 172.21.15.12:0/488371829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc300a2e20 msgr2=0x7efc3013b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:56.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc34b27640 1 --2- 172.21.15.12:0/488371829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc300a2e20 0x7efc3013b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:56.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc34b27640 10 monclient: _finish_hunting 0 2024-09-11T01:36:56.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc34b27640 1 monclient: found mon.noname-a 2024-09-11T01:36:56.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc34b27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:56.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc34b27640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:56.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.507+0000 7efc34b27640 1 -- 172.21.15.12:0/488371829 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc3009f610 con 0x7efc300a4970 2024-09-11T01:36:56.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.509+0000 7efc34b27640 1 -- 172.21.15.12:0/488371829 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc300a4970 msgr2=0x7efc3013bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:56.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.509+0000 7efc34b27640 1 --2- 172.21.15.12:0/488371829 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc300a4970 0x7efc3013bce0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7efc28007fd0 tx=0x7efc2800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:56.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.509+0000 7efc2cff9640 1 -- 172.21.15.12:0/488371829 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc28003b90 con 0x7efc300a4970 2024-09-11T01:36:56.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.509+0000 7efc2cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:56.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.509+0000 7efc2cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:56.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.509+0000 7efc2cff9640 10 monclient: dump: 2024-09-11T01:36:56.531 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:56.532 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:56.532 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:56.532 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:56.532 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:56.532 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:56.532 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:56.532 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:56.532 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:56.532 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:56.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.509+0000 7efc2cff9640 10 monclient: _finish_auth 0 2024-09-11T01:36:56.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.509+0000 7efc2cff9640 10 monclient: _check_auth_tickets 2024-09-11T01:36:56.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.509+0000 7efc2cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:56.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.509+0000 7efc2cff9640 1 -- 172.21.15.12:0/488371829 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc28003d50 con 0x7efc300a4970 2024-09-11T01:36:56.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.509+0000 7efc2cff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:56.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.509+0000 7efc2cff9640 1 -- 172.21.15.12:0/488371829 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc2800da90 con 0x7efc300a4970 2024-09-11T01:36:56.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.509+0000 7efc3658a640 5 monclient: authenticate success, global_id 54402 2024-09-11T01:36:56.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.509+0000 7efc3658a640 10 monclient: _renew_subs 2024-09-11T01:36:56.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.509+0000 7efc3658a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:56.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.509+0000 7efc3658a640 1 -- 172.21.15.12:0/488371829 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc3013ed10 con 0x7efc300a4970 2024-09-11T01:36:56.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.510+0000 7efc2cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:56.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.510+0000 7efc2cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:56.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.510+0000 7efc2cff9640 10 monclient: dump: 2024-09-11T01:36:56.534 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:56.534 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:56.534 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:56.534 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:56.534 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:56.534 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:56.534 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:56.534 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:56.534 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:56.534 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:56.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.509+0000 7efc2dffb640 4 set_mon_vals no callback set 2024-09-11T01:36:56.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.510+0000 7efc3658a640 10 monclient: _renew_subs 2024-09-11T01:36:56.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.510+0000 7efc3658a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:56.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.510+0000 7efc3658a640 1 -- 172.21.15.12:0/488371829 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc300a6340 con 0x7efc300a4970 2024-09-11T01:36:56.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.510+0000 7efc3658a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:56.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.510+0000 7efc3658a640 1 --2- 172.21.15.12:0/488371829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc3013e900 0x7efc301ccf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:56.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.510+0000 7efc3658a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:56.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.510+0000 7efc3658a640 1 -- 172.21.15.12:0/488371829 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7efc3006a060 con 0x7efc3013e900 2024-09-11T01:36:56.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.510+0000 7efc2cff9640 1 -- 172.21.15.12:0/488371829 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7efc28003f00 con 0x7efc300a4970 2024-09-11T01:36:56.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.511+0000 7efc2cff9640 1 --2- 172.21.15.12:0/488371829 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efc1c03e260 0x7efc1c040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:56.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.511+0000 7efc2ffff640 1 --2- 172.21.15.12:0/488371829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc3013e900 0x7efc301ccf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:56.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.511+0000 7efc2cff9640 1 -- 172.21.15.12:0/488371829 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(256..256 src has 186..256) ==== 4847+0+0 (secure 0 0 0) 0x7efc280537b0 con 0x7efc300a4970 2024-09-11T01:36:56.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.511+0000 7efc2ffff640 10 monclient: get_auth_request con 0x7efc3013e900 auth_method 0 2024-09-11T01:36:56.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.511+0000 7efc2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:56.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.511+0000 7efc2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:56.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.511+0000 7efc2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:56.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.511+0000 7efc2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:56.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.511+0000 7efc2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:56.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.511+0000 7efc2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:56.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.511+0000 7efc2f7fe640 1 --2- 172.21.15.12:0/488371829 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efc1c03e260 0x7efc1c040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:56.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.511+0000 7efc2f7fe640 10 monclient: get_auth_request con 0x7efc1c03e260 auth_method 0 2024-09-11T01:36:56.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.511+0000 7efc2ffff640 10 monclient(hunting): handle_auth_done global_id 74203 payload 824 2024-09-11T01:36:56.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.511+0000 7efc2ffff640 1 -- 172.21.15.12:0/488371829 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efc3006ded0 con 0x7efc3013e900 2024-09-11T01:36:56.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.513+0000 7efc2ffff640 1 -- 172.21.15.12:0/488371829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc3013e900 msgr2=0x7efc301ccf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:56.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.513+0000 7efc2ffff640 1 --2- 172.21.15.12:0/488371829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc3013e900 0x7efc301ccf00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efc2000f770 tx=0x7efc2000fc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:56.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.514+0000 7efc2f7fe640 1 --2- 172.21.15.12:0/488371829 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efc1c03e260 0x7efc1c040720 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efc24034000 tx=0x7efc2402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:56.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.514+0000 7efc2cff9640 1 -- 172.21.15.12:0/488371829 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc200052e0 con 0x7efc3013e900 2024-09-11T01:36:56.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.514+0000 7efc2cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:56.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.514+0000 7efc2cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:36:56.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.514+0000 7efc2cff9640 10 monclient: dump: 2024-09-11T01:36:56.537 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:56.537 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:56.537 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:56.538 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:56.538 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:56.538 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:56.538 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:56.538 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:56.538 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:56.538 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:56.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.514+0000 7efc2cff9640 1 -- 172.21.15.12:0/488371829 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7efc20018070 con 0x7efc3013e900 2024-09-11T01:36:56.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.514+0000 7efc2cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:56.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.514+0000 7efc2cff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:56.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.514+0000 7efc2cff9640 1 -- 172.21.15.12:0/488371829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc3013e900 msgr2=0x7efc301ccf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:56.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.514+0000 7efc2cff9640 1 --2- 172.21.15.12:0/488371829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc3013e900 0x7efc301ccf00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efc2000f770 tx=0x7efc2000fc30 comp rx=0 tx=0).stop 2024-09-11T01:36:56.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.514+0000 7efc2cff9640 1 -- 172.21.15.12:0/488371829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc3013e900 msgr2=0x7efc301ccf00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:56.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.514+0000 7efc2cff9640 1 --2- 172.21.15.12:0/488371829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc3013e900 0x7efc301ccf00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:56.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.520+0000 7efc3658a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:36:56.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.520+0000 7efc3658a640 1 --2- 172.21.15.12:0/488371829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc301cd440 0x7efc301cd820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:56.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.520+0000 7efc3658a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:56.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.520+0000 7efc3658a640 1 -- 172.21.15.12:0/488371829 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7efc30138cc0 con 0x7efc301cd440 2024-09-11T01:36:56.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.521+0000 7efc2ffff640 1 --2- 172.21.15.12:0/488371829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc301cd440 0x7efc301cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:56.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.521+0000 7efc2ffff640 10 monclient: get_auth_request con 0x7efc301cd440 auth_method 0 2024-09-11T01:36:56.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.521+0000 7efc2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:56.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.521+0000 7efc2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:56.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.521+0000 7efc2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:56.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.521+0000 7efc2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:56.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.521+0000 7efc2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:56.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.521+0000 7efc2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:56.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.521+0000 7efc2ffff640 10 monclient(hunting): handle_auth_done global_id 74206 payload 824 2024-09-11T01:36:56.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.521+0000 7efc2ffff640 1 -- 172.21.15.12:0/488371829 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efc30069950 con 0x7efc301cd440 2024-09-11T01:36:56.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.523+0000 7efc2ffff640 1 -- 172.21.15.12:0/488371829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc301cd440 msgr2=0x7efc301cd820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:56.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.523+0000 7efc2ffff640 1 --2- 172.21.15.12:0/488371829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc301cd440 0x7efc301cd820 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efc2000f770 tx=0x7efc20001020 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:56.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.524+0000 7efc2cff9640 1 -- 172.21.15.12:0/488371829 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc20007610 con 0x7efc301cd440 2024-09-11T01:36:56.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.524+0000 7efc2cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:56.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.524+0000 7efc2cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:36:56.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.524+0000 7efc2cff9640 10 monclient: dump: 2024-09-11T01:36:56.543 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:56.543 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:56.543 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:56.543 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:56.543 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:56.543 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:56.543 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:56.543 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:56.543 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:56.543 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:56.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.524+0000 7efc2cff9640 1 -- 172.21.15.12:0/488371829 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4707 (secure 0 0 0) 0x7efc20007610 con 0x7efc301cd440 2024-09-11T01:36:56.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.524+0000 7efc2cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:36:56.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.524+0000 7efc2cff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:56.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.524+0000 7efc2cff9640 1 -- 172.21.15.12:0/488371829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc301cd440 msgr2=0x7efc301cd820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:56.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.524+0000 7efc2cff9640 1 --2- 172.21.15.12:0/488371829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc301cd440 0x7efc301cd820 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efc2000f770 tx=0x7efc20001020 comp rx=0 tx=0).stop 2024-09-11T01:36:56.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.524+0000 7efc2cff9640 1 -- 172.21.15.12:0/488371829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc301cd440 msgr2=0x7efc301cd820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:56.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.524+0000 7efc2cff9640 1 --2- 172.21.15.12:0/488371829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc301cd440 0x7efc301cd820 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efc2000f770 tx=0x7efc20001020 comp rx=0 tx=0).stop 2024-09-11T01:36:56.544 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:36:56.544 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:36:56.544 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:36:56.545 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:36:56.545 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 80, 2024-09-11T01:36:56.545 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 68302, 2024-09-11T01:36:56.545 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:36:56.545 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:36:56.545 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:36:56.545 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:36:56.545 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:56.545 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 4, 2024-09-11T01:36:56.545 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:56.545 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:36:56.546 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:36:56.546 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:56.546 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:56.546 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:56.546 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:56.546 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:56.546 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:56.546 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:56.546 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:56.546 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:56.546 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:56.547 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:56.547 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:56.547 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:36:56.547 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:36:56.547 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:56.547 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:56.547 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:56.547 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:56.547 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:56.547 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:56.547 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:56.548 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:56.548 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:56.548 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:56.548 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:56.548 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:56.548 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:56.548 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:36:56.548 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:36:56.548 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:36:56.548 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:36:56.548 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:36:56.549 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:36:56.549 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:56.549 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:56.549 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:36:56.549 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:36:56.549 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:36:56.549 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:36:56.549 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:36:56.549 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:36:56.549 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:36:56.549 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:56.549 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:36:56.550 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:56.550 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:56.550 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:56.550 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:56.550 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:56.550 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:56.550 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:56.550 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:56.550 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:56.550 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:56.550 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:56.550 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:56.551 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:36:56.551 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:56.551 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:36:56.551 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:56.551 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:36:56.551 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:36:56.551 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:56.551 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:56.551 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:56.551 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:56.551 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:36:56.552 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:56.552 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:56.552 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:56.552 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:56.552 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:36:56.552 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:56.552 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:56.552 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:56.552 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:56.553 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:36:56.553 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:36:56.553 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:56.553 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:56.553 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:56.553 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:56.553 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:56.553 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:36:56.553 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:36:56.553 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:56.554 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:56.554 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:56.554 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:56.554 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:36:56.554 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:56.554 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:56.554 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:56.554 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:56.554 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:36:56.554 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:56.555 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:56.555 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:56.555 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:56.555 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:36:56.555 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:36:56.555 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:56.555 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:56.555 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:56.555 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:56.555 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:56.555 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:36:56.556 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:36:56.556 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:56.556 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:56.556 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:56.556 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:56.556 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:36:56.556 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:56.556 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:56.556 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:56.556 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:56.557 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:36:56.557 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:56.557 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:56.557 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:56.557 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:56.557 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:36:56.557 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:36:56.557 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:56.557 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:56.557 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:56.557 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:56.558 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:56.558 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:56.558 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:36:56.558 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:36:56.558 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:56.558 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:56.558 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:56.558 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:36:56.558 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:56.558 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:56.558 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:36:56.559 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:56.559 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:56.559 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:56.559 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:36:56.559 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:56.559 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:56.559 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:36:56.559 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:56.559 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:56.559 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:56.560 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:36:56.560 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:56.560 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:56.560 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:56.560 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:36:56.560 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:36:56.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.524+0000 7efc3658a640 1 -- 172.21.15.12:0/488371829 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efc1c03e260 msgr2=0x7efc1c040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:56.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.524+0000 7efc3658a640 1 --2- 172.21.15.12:0/488371829 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efc1c03e260 0x7efc1c040720 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efc24034000 tx=0x7efc2402e000 comp rx=0 tx=0).stop 2024-09-11T01:36:56.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.524+0000 7efc3658a640 10 monclient: shutdown 2024-09-11T01:36:56.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.524+0000 7efc3658a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:56.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.524+0000 7efc3658a640 1 -- 172.21.15.12:0/488371829 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc300a4970 msgr2=0x7efc3013bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:56.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.524+0000 7efc3658a640 1 --2- 172.21.15.12:0/488371829 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc300a4970 0x7efc3013bce0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7efc28007fd0 tx=0x7efc2800a710 comp rx=0 tx=0).stop 2024-09-11T01:36:56.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.524+0000 7efc34b27640 1 -- 172.21.15.12:0/488371829 reap_dead start 2024-09-11T01:36:56.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.524+0000 7efc34b27640 1 -- 172.21.15.12:0/488371829 reap_dead start 2024-09-11T01:36:56.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.524+0000 7efc3658a640 1 -- 172.21.15.12:0/488371829 shutdown_connections 2024-09-11T01:36:56.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.524+0000 7efc3658a640 1 -- 172.21.15.12:0/488371829 >> 172.21.15.12:0/488371829 conn(0x7efc300a8600 msgr2=0x7efc30072e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:56.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.524+0000 7efc3658a640 1 -- 172.21.15.12:0/488371829 shutdown_connections 2024-09-11T01:36:56.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.524+0000 7efc3658a640 1 -- 172.21.15.12:0/488371829 wait complete. 2024-09-11T01:36:56.563 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:36:56.563 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:36:56.563 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:36:56.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.663+0000 7feb87271640 10 monclient: get_monmap_and_config 2024-09-11T01:36:56.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.663+0000 7feb87271640 10 monclient: build_initial_monmap 2024-09-11T01:36:56.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.663+0000 7feb87271640 10 monclient: monmap: 2024-09-11T01:36:56.665 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:56.665 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:56.665 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:56.663894+0000 2024-09-11T01:36:56.665 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:56.663894+0000 2024-09-11T01:36:56.665 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:56.665 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:56.666 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:56.666 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:56.666 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:56.666 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:56.666 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:56.666 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:56.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.663+0000 7feb87271640 1 Processor -- start 2024-09-11T01:36:56.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.663+0000 7feb87271640 1 -- start start 2024-09-11T01:36:56.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.663+0000 7feb87271640 10 monclient: init 2024-09-11T01:36:56.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb87271640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:56.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb87271640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-11T01:36:56.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb87271640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feb80137d50 addr v1:172.21.15.12:6789/0 2024-09-11T01:36:56.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb87271640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feb80138130 0x7feb80138580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:56.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb87271640 10 monclient(hunting): picked mon.noname-a con 0x7feb80138130 addr v2:172.21.15.12:3300/0 2024-09-11T01:36:56.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb87271640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb8013d280 0x7feb8013f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:56.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb87271640 10 monclient(hunting): picked mon.noname-b con 0x7feb8013d280 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:56.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb87271640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb8009d820 con 0x7feb80137d50 2024-09-11T01:36:56.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb87271640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:56.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb87271640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7feb8009c650 con 0x7feb80138130 2024-09-11T01:36:56.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb87271640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:56.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb87271640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7feb8009ba20 con 0x7feb8013d280 2024-09-11T01:36:56.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb87271640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:56.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb87271640 10 monclient(hunting): authenticate will time out at 1223.106483s 2024-09-11T01:36:56.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8480c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feb80138130 0x7feb80138580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:56.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8500d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7feb80137d50 0x7feb8009d110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:48624/0 (socket says 172.21.15.12:48624) 2024-09-11T01:36:56.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8500d640 1 -- 172.21.15.12:0/2796487491 learned_addr learned my addr 172.21.15.12:0/2796487491 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:36:56.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8480c640 10 monclient(hunting): get_auth_request con 0x7feb80138130 auth_method 0 2024-09-11T01:36:56.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8580e640 1 --2- 172.21.15.12:0/2796487491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb8013d280 0x7feb8013f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:56.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8480c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:56.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8480c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:56.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8480c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:56.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8580e640 10 monclient(hunting): get_auth_request con 0x7feb8013d280 auth_method 0 2024-09-11T01:36:56.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8580e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:56.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8580e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:56.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8580e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:56.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8480c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:56.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8480c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:56.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8480c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:56.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8580e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:56.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8580e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:56.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8580e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:56.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8480c640 10 monclient(hunting): handle_auth_done global_id 54405 payload 274 2024-09-11T01:36:56.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8480c640 1 -- 172.21.15.12:0/2796487491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb8013d280 msgr2=0x7feb8013f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:56.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8480c640 1 --2- 172.21.15.12:0/2796487491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb8013d280 0x7feb8013f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:56.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8480c640 1 -- 172.21.15.12:0/2796487491 >> v1:172.21.15.12:6789/0 conn(0x7feb80137d50 legacy=0x7feb8009d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:56.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8480c640 10 monclient: _finish_hunting 0 2024-09-11T01:36:56.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8480c640 1 monclient: found mon.noname-a 2024-09-11T01:36:56.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8480c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:56.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8480c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:56.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.664+0000 7feb8480c640 1 -- 172.21.15.12:0/2796487491 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb8009c370 con 0x7feb80138130 2024-09-11T01:36:56.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.667+0000 7feb8480c640 1 -- 172.21.15.12:0/2796487491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb80138130 msgr2=0x7feb80138580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:56.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.667+0000 7feb8480c640 1 --2- 172.21.15.12:0/2796487491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb80138130 0x7feb80138580 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7feb7c0097a0 tx=0x7feb7c02a050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=218e9362a8a2a7b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:56.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.667+0000 7feb73fff640 1 -- 172.21.15.12:0/2796487491 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb7c037070 con 0x7feb80138130 2024-09-11T01:36:56.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.667+0000 7feb73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:56.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.667+0000 7feb73fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:56.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.667+0000 7feb73fff640 10 monclient: dump: 2024-09-11T01:36:56.675 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:56.675 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:56.675 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:56.675 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:56.675 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:56.675 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:56.675 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:56.675 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:56.675 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:56.675 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:56.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.667+0000 7feb73fff640 10 monclient: _finish_auth 0 2024-09-11T01:36:56.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.667+0000 7feb73fff640 10 monclient: _check_auth_tickets 2024-09-11T01:36:56.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.667+0000 7feb73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:56.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.667+0000 7feb87271640 5 monclient: authenticate success, global_id 54405 2024-09-11T01:36:56.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.667+0000 7feb73fff640 1 -- 172.21.15.12:0/2796487491 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb7c032440 con 0x7feb80138130 2024-09-11T01:36:56.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.667+0000 7feb87271640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:36:56.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.667+0000 7feb73fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:56.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.667+0000 7feb87271640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:56.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.667+0000 7feb73fff640 1 -- 172.21.15.12:0/2796487491 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb7c032810 con 0x7feb80138130 2024-09-11T01:36:56.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.667+0000 7feb73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.667+0000 7feb87271640 4 set_mon_vals no callback set 2024-09-11T01:36:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.667+0000 7feb73fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.667+0000 7feb73fff640 10 monclient: dump: 2024-09-11T01:36:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:56.677 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:56.678 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:56.678 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:56.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.668+0000 7feb87271640 10 monclient: shutdown 2024-09-11T01:36:56.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.668+0000 7feb87271640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:56.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.668+0000 7feb87271640 1 -- 172.21.15.12:0/2796487491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb80138130 msgr2=0x7feb80138580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:56.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.668+0000 7feb87271640 1 --2- 172.21.15.12:0/2796487491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb80138130 0x7feb80138580 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7feb7c0097a0 tx=0x7feb7c02a050 comp rx=0 tx=0).stop 2024-09-11T01:36:56.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.668+0000 7feb87271640 1 -- 172.21.15.12:0/2796487491 shutdown_connections 2024-09-11T01:36:56.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.668+0000 7feb87271640 1 --2- 172.21.15.12:0/2796487491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb8013d280 0x7feb8013f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:56.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.668+0000 7feb87271640 1 --2- 172.21.15.12:0/2796487491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb80138130 0x7feb80138580 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:56.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.668+0000 7feb87271640 1 -- 172.21.15.12:0/2796487491 >> 172.21.15.12:0/2796487491 conn(0x7feb800a7020 msgr2=0x7feb800a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:56.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.668+0000 7feb87271640 1 -- 172.21.15.12:0/2796487491 shutdown_connections 2024-09-11T01:36:56.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.668+0000 7feb87271640 1 -- 172.21.15.12:0/2796487491 wait complete. 2024-09-11T01:36:56.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.669+0000 7feb87271640 10 monclient: build_initial_monmap 2024-09-11T01:36:56.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.669+0000 7feb87271640 10 monclient: monmap: 2024-09-11T01:36:56.679 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:56.679 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:56.679 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:56.679 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:56.679 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:56.679 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:56.679 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:56.679 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:56.679 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:56.679 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:56.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.669+0000 7feb87271640 1 Processor -- start 2024-09-11T01:36:56.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.669+0000 7feb87271640 1 -- start start 2024-09-11T01:36:56.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.669+0000 7feb87271640 10 monclient: init 2024-09-11T01:36:56.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.669+0000 7feb87271640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:56.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.669+0000 7feb87271640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:36:56.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.669+0000 7feb87271640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb80137d50 0x7feb801ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:56.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.669+0000 7feb87271640 10 monclient(hunting): picked mon.noname-a con 0x7feb80137d50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:56.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.669+0000 7feb87271640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb80138130 0x7feb80136e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:56.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.669+0000 7feb87271640 10 monclient(hunting): picked mon.noname-b con 0x7feb80138130 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:56.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.669+0000 7feb87271640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb8013d280 0x7feb8013a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:56.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.669+0000 7feb87271640 10 monclient(hunting): picked mon.noname-c con 0x7feb8013d280 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:56.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.669+0000 7feb87271640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:56.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.669+0000 7feb87271640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7feb8006f850 con 0x7feb80137d50 2024-09-11T01:36:56.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.669+0000 7feb87271640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:56.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.669+0000 7feb87271640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7feb8006f6d0 con 0x7feb80138130 2024-09-11T01:36:56.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.669+0000 7feb87271640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:56.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.669+0000 7feb87271640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7feb8013fbb0 con 0x7feb8013d280 2024-09-11T01:36:56.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.669+0000 7feb87271640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:56.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.669+0000 7feb87271640 10 monclient(hunting): authenticate will time out at 1223.111901s 2024-09-11T01:36:56.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.669+0000 7feb8500d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb80137d50 0x7feb801ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:56.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8580e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb8013d280 0x7feb8013a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:56.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8500d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb80137d50 0x7feb801ccee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36388/0 (socket says 172.21.15.12:36388) 2024-09-11T01:36:56.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8500d640 1 -- 172.21.15.12:0/777813919 learned_addr learned my addr 172.21.15.12:0/777813919 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:56.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8580e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb8013d280 0x7feb8013a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:34590/0 (socket says 172.21.15.12:34590) 2024-09-11T01:36:56.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8500d640 10 monclient(hunting): get_auth_request con 0x7feb80137d50 auth_method 0 2024-09-11T01:36:56.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8500d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:56.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8500d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:56.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8500d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:56.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8580e640 10 monclient(hunting): get_auth_request con 0x7feb8013d280 auth_method 0 2024-09-11T01:36:56.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8580e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:56.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8580e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:56.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8580e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:56.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8500d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:56.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8500d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:56.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8500d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:56.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8580e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:56.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8580e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:56.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8580e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:56.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8480c640 1 --2- 172.21.15.12:0/777813919 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb80138130 0x7feb80136e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:56.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8480c640 10 monclient(hunting): get_auth_request con 0x7feb80138130 auth_method 0 2024-09-11T01:36:56.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8480c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:56.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8480c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:56.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8480c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:56.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8500d640 10 monclient(hunting): handle_auth_done global_id 54408 payload 824 2024-09-11T01:36:56.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8500d640 1 -- 172.21.15.12:0/777813919 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb8013d280 msgr2=0x7feb8013a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:56.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8500d640 1 --2- 172.21.15.12:0/777813919 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb8013d280 0x7feb8013a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:56.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8500d640 1 -- 172.21.15.12:0/777813919 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb80138130 msgr2=0x7feb80136e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8500d640 1 --2- 172.21.15.12:0/777813919 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb80138130 0x7feb80136e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8500d640 10 monclient: _finish_hunting 0 2024-09-11T01:36:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8500d640 1 monclient: found mon.noname-a 2024-09-11T01:36:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8500d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8500d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.670+0000 7feb8500d640 1 -- 172.21.15.12:0/777813919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb800a1710 con 0x7feb80137d50 2024-09-11T01:36:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.672+0000 7feb8500d640 1 -- 172.21.15.12:0/777813919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb80137d50 msgr2=0x7feb801ccee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.672+0000 7feb8500d640 1 --2- 172.21.15.12:0/777813919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb80137d50 0x7feb801ccee0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7feb7400c460 tx=0x7feb7400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.672+0000 7feb71ffb640 1 -- 172.21.15.12:0/777813919 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb74007070 con 0x7feb80137d50 2024-09-11T01:36:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.672+0000 7feb71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:56.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.672+0000 7feb71ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.672+0000 7feb71ffb640 10 monclient: dump: 2024-09-11T01:36:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:56.689 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:56.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.672+0000 7feb71ffb640 10 monclient: _finish_auth 0 2024-09-11T01:36:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.672+0000 7feb71ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:36:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.672+0000 7feb71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.672+0000 7feb71ffb640 1 -- 172.21.15.12:0/777813919 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb74007230 con 0x7feb80137d50 2024-09-11T01:36:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.672+0000 7feb71ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.672+0000 7feb71ffb640 1 -- 172.21.15.12:0/777813919 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb74007620 con 0x7feb80137d50 2024-09-11T01:36:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.672+0000 7feb71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.672+0000 7feb71ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.672+0000 7feb71ffb640 10 monclient: dump: 2024-09-11T01:36:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:56.690 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:56.691 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:56.691 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:56.691 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:56.691 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:56.691 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:56.691 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:56.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.672+0000 7feb87271640 5 monclient: authenticate success, global_id 54408 2024-09-11T01:36:56.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.672+0000 7feb87271640 10 monclient: _renew_subs 2024-09-11T01:36:56.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.672+0000 7feb87271640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:56.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.672+0000 7feb87271640 1 -- 172.21.15.12:0/777813919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb8013bcb0 con 0x7feb80137d50 2024-09-11T01:36:56.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.672+0000 7feb737fe640 4 set_mon_vals no callback set 2024-09-11T01:36:56.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.672+0000 7feb87271640 10 monclient: _renew_subs 2024-09-11T01:36:56.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.672+0000 7feb87271640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:56.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.672+0000 7feb87271640 1 -- 172.21.15.12:0/777813919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb80121060 con 0x7feb80137d50 2024-09-11T01:36:56.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.673+0000 7feb71ffb640 1 -- 172.21.15.12:0/777813919 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7feb7401b400 con 0x7feb80137d50 2024-09-11T01:36:56.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.673+0000 7feb87271640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:56.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.673+0000 7feb87271640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:56.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.673+0000 7feb87271640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:56.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.673+0000 7feb87271640 1 -- 172.21.15.12:0/777813919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb8013c8d0 con 0x7feb80137d50 2024-09-11T01:36:56.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.673+0000 7feb71ffb640 1 --2- 172.21.15.12:0/777813919 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7feb6403e210 0x7feb640406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:56.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.674+0000 7feb71ffb640 1 -- 172.21.15.12:0/777813919 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(256..256 src has 186..256) ==== 4847+0+0 (secure 0 0 0) 0x7feb74052a40 con 0x7feb80137d50 2024-09-11T01:36:56.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.674+0000 7feb8480c640 1 --2- 172.21.15.12:0/777813919 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7feb6403e210 0x7feb640406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:56.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.674+0000 7feb8480c640 10 monclient: get_auth_request con 0x7feb6403e210 auth_method 0 2024-09-11T01:36:56.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.676+0000 7feb8480c640 1 --2- 172.21.15.12:0/777813919 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7feb6403e210 0x7feb640406d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7feb8006d4f0 tx=0x7feb7c02e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:56.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.678+0000 7feb71ffb640 1 -- 172.21.15.12:0/777813919 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7feb74051400 con 0x7feb80137d50 2024-09-11T01:36:56.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.678+0000 7feb71ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:56.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.678+0000 7feb71ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:56.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.938+0000 7feb87271640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:36:56.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.938+0000 7feb87271640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:36:56.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.938+0000 7feb87271640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:56.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.938+0000 7feb87271640 1 -- 172.21.15.12:0/777813919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7feb80138580 con 0x7feb80137d50 2024-09-11T01:36:56.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.939+0000 7feb71ffb640 1 -- 172.21.15.12:0/777813919 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7feb74010d10 con 0x7feb80137d50 2024-09-11T01:36:56.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.939+0000 7feb71ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:36:56.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.939+0000 7feb71ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:56.941 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.939+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.941 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.939+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=12,logm=60} crc {auth=4222107487,config=2586702516,health=1068221574,kv=3263697420,logm=2074627709}) 2024-09-11T01:36:56.942 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.939+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=12,config=2,health=10,kv=12,logm=64} crc {auth=2946665252,config=2586702516,health=4177358965,kv=3263697420,logm=1573691383}) 2024-09-11T01:36:56.942 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.939+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.942 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.939+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=12,logm=60} crc {auth=4222107487,config=2586702516,health=1068221574,kv=3263697420,logm=2074627709}) 2024-09-11T01:36:56.942 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.939+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=15,config=2,health=11,kv=10,logm=62} crc {auth=1465848206,config=2586702516,health=3820421734,kv=3209825898,logm=912245108}) 2024-09-11T01:36:56.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.940+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.940+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1783769686}) 2024-09-11T01:36:56.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.940+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1243280107}) 2024-09-11T01:36:56.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.940+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.940+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1783769686}) 2024-09-11T01:36:56.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.940+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2398494614}) 2024-09-11T01:36:56.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.941+0000 7feb87271640 1 -- 172.21.15.12:0/777813919 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7feb6403e210 msgr2=0x7feb640406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:56.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.941+0000 7feb87271640 1 --2- 172.21.15.12:0/777813919 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7feb6403e210 0x7feb640406d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7feb8006d4f0 tx=0x7feb7c02e000 comp rx=0 tx=0).stop 2024-09-11T01:36:56.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.941+0000 7feb87271640 10 monclient: shutdown 2024-09-11T01:36:56.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.941+0000 7feb87271640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:56.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.941+0000 7feb87271640 1 -- 172.21.15.12:0/777813919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb80137d50 msgr2=0x7feb801ccee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:56.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.941+0000 7feb87271640 1 --2- 172.21.15.12:0/777813919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb80137d50 0x7feb801ccee0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7feb7400c460 tx=0x7feb7400c920 comp rx=0 tx=0).stop 2024-09-11T01:36:56.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.941+0000 7feb87271640 1 -- 172.21.15.12:0/777813919 shutdown_connections 2024-09-11T01:36:56.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.941+0000 7feb87271640 1 --2- 172.21.15.12:0/777813919 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb8013d280 0x7feb8013a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:56.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.941+0000 7feb87271640 1 --2- 172.21.15.12:0/777813919 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb80138130 0x7feb80136e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:56.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.941+0000 7feb87271640 1 --2- 172.21.15.12:0/777813919 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7feb6403e210 0x7feb640406d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:56.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.941+0000 7feb87271640 1 --2- 172.21.15.12:0/777813919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb80137d50 0x7feb801ccee0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:56.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.941+0000 7feb87271640 1 -- 172.21.15.12:0/777813919 >> 172.21.15.12:0/777813919 conn(0x7feb800a7020 msgr2=0x7feb8013e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:56.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.941+0000 7feb87271640 1 -- 172.21.15.12:0/777813919 shutdown_connections 2024-09-11T01:36:56.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:56.941+0000 7feb87271640 1 -- 172.21.15.12:0/777813919 wait complete. 2024-09-11T01:36:56.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.941+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.941+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2411488818}) 2024-09-11T01:36:56.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.941+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3137603384}) 2024-09-11T01:36:56.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.941+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.941+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2411488818}) 2024-09-11T01:36:56.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.941+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1562673188}) 2024-09-11T01:36:56.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.942+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.942+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1580940945}) 2024-09-11T01:36:56.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.942+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=131109356}) 2024-09-11T01:36:56.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.942+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.942+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1580940945}) 2024-09-11T01:36:56.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.942+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3668128119}) 2024-09-11T01:36:56.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.943+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.943+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=810583590}) 2024-09-11T01:36:56.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.943+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3557080804}) 2024-09-11T01:36:56.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.943+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.943+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=810583590}) 2024-09-11T01:36:56.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.943+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3178447660}) 2024-09-11T01:36:56.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.944+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.944+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3391951192}) 2024-09-11T01:36:56.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.944+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1091183187}) 2024-09-11T01:36:56.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.944+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.944+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3391951192}) 2024-09-11T01:36:56.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.944+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2161513492}) 2024-09-11T01:36:56.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.945+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.945+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3845742076}) 2024-09-11T01:36:56.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.945+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1270868420}) 2024-09-11T01:36:56.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.945+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.945+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3845742076}) 2024-09-11T01:36:56.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.945+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=186947248}) 2024-09-11T01:36:56.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.946+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.946+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=382960014}) 2024-09-11T01:36:56.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.946+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1256019242}) 2024-09-11T01:36:56.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.946+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.946+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=382960014}) 2024-09-11T01:36:56.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.946+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2970349618}) 2024-09-11T01:36:56.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.947+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.947+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1404359399}) 2024-09-11T01:36:56.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.947+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1122676394}) 2024-09-11T01:36:56.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.947+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.947+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1404359399}) 2024-09-11T01:36:56.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.947+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1817722231}) 2024-09-11T01:36:56.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.948+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.948+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3192846366}) 2024-09-11T01:36:56.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.948+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2237656962}) 2024-09-11T01:36:56.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.948+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.948+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3192846366}) 2024-09-11T01:36:56.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.948+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2028205155}) 2024-09-11T01:36:56.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.949+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.949+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2461831712}) 2024-09-11T01:36:56.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.949+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1494852728}) 2024-09-11T01:36:56.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.949+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.949+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2461831712}) 2024-09-11T01:36:56.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.949+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2296193071}) 2024-09-11T01:36:56.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.950+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.950+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=808789663}) 2024-09-11T01:36:56.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.950+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3436486645}) 2024-09-11T01:36:56.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.950+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.950+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=808789663}) 2024-09-11T01:36:56.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.950+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3490840792}) 2024-09-11T01:36:56.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.951+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.951+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1584299212}) 2024-09-11T01:36:56.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.951+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=861020757}) 2024-09-11T01:36:56.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.951+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.951+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1584299212}) 2024-09-11T01:36:56.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.951+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=10513726}) 2024-09-11T01:36:56.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.952+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.952+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1952650785}) 2024-09-11T01:36:56.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.952+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1586934412}) 2024-09-11T01:36:56.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.952+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.952+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1952650785}) 2024-09-11T01:36:56.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.952+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=594770401}) 2024-09-11T01:36:56.956 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.953+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.956 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.953+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4228535565}) 2024-09-11T01:36:56.956 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.953+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2750120875}) 2024-09-11T01:36:56.956 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.953+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.956 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.953+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4228535565}) 2024-09-11T01:36:56.956 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.953+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2404228660}) 2024-09-11T01:36:56.956 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.954+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.956 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.954+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799983907}) 2024-09-11T01:36:56.956 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.954+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=871012927}) 2024-09-11T01:36:56.957 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.954+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.957 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.954+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799983907}) 2024-09-11T01:36:56.957 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.954+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1476455017}) 2024-09-11T01:36:56.958 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:36:56.959 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.955+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.955+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4286792981}) 2024-09-11T01:36:56.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.955+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=524815043}) 2024-09-11T01:36:56.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.955+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.955+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4286792981}) 2024-09-11T01:36:56.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.955+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1814582767}) 2024-09-11T01:36:56.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.955+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.955+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848051450}) 2024-09-11T01:36:56.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.955+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3938045697}) 2024-09-11T01:36:56.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.955+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.955+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848051450}) 2024-09-11T01:36:56.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.955+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1231994708}) 2024-09-11T01:36:56.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.956+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.961 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.956+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4129334765}) 2024-09-11T01:36:56.961 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.956+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2513640172}) 2024-09-11T01:36:56.961 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.956+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.961 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.956+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4129334765}) 2024-09-11T01:36:56.961 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.956+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2652757485}) 2024-09-11T01:36:56.961 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.957+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.961 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.957+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=46,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2} crc {logm=2696016863,mdsmap=3140514009,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2182305728,mon_config_key=2466682877,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=501920436,osd_snap=4089070047}) 2024-09-11T01:36:56.961 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.957+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=44,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=3} crc {logm=2816260941,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=86463339,mon_config_key=2466682877,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=501920436,osd_snap=966303594}) 2024-09-11T01:36:56.961 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.957+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.961 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.957+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=46,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2} crc {logm=2696016863,mdsmap=3140514009,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2182305728,mon_config_key=2466682877,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=501920436,osd_snap=4089070047}) 2024-09-11T01:36:56.961 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.957+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=46,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=1} crc {logm=434990734,mdsmap=3760429986,mgr=4080174641,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1123815832,mon_config_key=3330371543,monmap=2933275274,nvmeofgw=598337686,osd_metadata=950107459,osd_pg_creating=501920436,osd_snap=3718248}) 2024-09-11T01:36:56.961 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.958+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.962 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.958+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=49,osdmap=51} crc {osd_snap=1247721791,osdmap=3415583329}) 2024-09-11T01:36:56.962 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.958+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=51,osdmap=49} crc {osd_snap=3750495396,osdmap=2023537443}) 2024-09-11T01:36:56.962 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.958+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.962 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.958+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=49,osdmap=51} crc {osd_snap=1247721791,osdmap=3415583329}) 2024-09-11T01:36:56.962 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.958+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=51,osdmap=49} crc {osd_snap=638407565,osdmap=1004432484}) 2024-09-11T01:36:56.963 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.962+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.963 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.962+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=67} crc {osdmap=1461972097}) 2024-09-11T01:36:56.964 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.962+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=64} crc {osdmap=46825046}) 2024-09-11T01:36:56.964 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.962+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:36:56.964 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.962+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=67} crc {osdmap=1461972097}) 2024-09-11T01:36:56.964 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:36:56.962+0000 7f04d5bd0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=62} crc {osdmap=4111016262}) 2024-09-11T01:36:57.178 INFO:tasks.rados.rados.0.smithi071.stdout:1113: done (3 left) 2024-09-11T01:36:57.179 INFO:tasks.rados.rados.0.smithi071.stdout:1114: delete oid 45 current snap is 117 2024-09-11T01:36:57.179 INFO:tasks.rados.rados.0.smithi071.stdout:1109: finishing write tid 4 to smithi07134777-21 2024-09-11T01:36:57.179 INFO:tasks.rados.rados.0.smithi071.stdout:1109: finishing write tid 5 to smithi07134777-21 2024-09-11T01:36:57.179 INFO:tasks.rados.rados.0.smithi071.stdout:1109: oid 21 updating version 0 to 451 2024-09-11T01:36:57.179 INFO:tasks.rados.rados.0.smithi071.stdout:1109: oid 21 updating version 451 to 454 2024-09-11T01:36:57.179 INFO:tasks.rados.rados.0.smithi071.stdout:1109: oid 21 version 454 is already newer than 452 2024-09-11T01:36:57.179 INFO:tasks.rados.rados.0.smithi071.stdout:1109: oid 21 version 454 is already newer than 453 2024-09-11T01:36:57.179 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 454 (ObjNum 417 snap 117 seq_num 417) dirty exists 2024-09-11T01:36:57.179 INFO:tasks.rados.rados.0.smithi071.stdout:1109: left oid 21 (ObjNum 417 snap 117 seq_num 417) 2024-09-11T01:36:57.179 INFO:tasks.rados.rados.0.smithi071.stdout:1110: finishing copy_from racing read to smithi07134777-26 2024-09-11T01:36:57.179 INFO:tasks.rados.rados.0.smithi071.stdout:1110: finishing copy_from to smithi07134777-26 2024-09-11T01:36:57.179 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 488 (ObjNum 226 snap 55 seq_num 226) dirty exists 2024-09-11T01:36:57.179 INFO:tasks.rados.rados.0.smithi071.stdout:1112: finishing write tid 1 to smithi07134777-28 2024-09-11T01:36:57.180 INFO:tasks.rados.rados.0.smithi071.stdout:1112: finishing write tid 2 to smithi07134777-28 2024-09-11T01:36:57.180 INFO:tasks.rados.rados.0.smithi071.stdout:1112: finishing write tid 3 to smithi07134777-28 2024-09-11T01:36:57.180 INFO:tasks.rados.rados.0.smithi071.stdout:1112: finishing write tid 4 to smithi07134777-28 2024-09-11T01:36:57.180 INFO:tasks.rados.rados.0.smithi071.stdout:1112: oid 28 updating version 0 to 462 2024-09-11T01:36:57.180 INFO:tasks.rados.rados.0.smithi071.stdout:1112: oid 28 version 462 is already newer than 461 2024-09-11T01:36:57.180 INFO:tasks.rados.rados.0.smithi071.stdout:1112: oid 28 version 462 is already newer than 460 2024-09-11T01:36:57.180 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 462 (ObjNum 418 snap 117 seq_num 418) dirty exists 2024-09-11T01:36:57.180 INFO:tasks.rados.rados.0.smithi071.stdout:1112: left oid 28 (ObjNum 418 snap 117 seq_num 418) 2024-09-11T01:36:57.184 INFO:tasks.rados.rados.0.smithi071.stdout:1109: done (3 left) 2024-09-11T01:36:57.184 INFO:tasks.rados.rados.0.smithi071.stdout:1110: done (2 left) 2024-09-11T01:36:57.184 INFO:tasks.rados.rados.0.smithi071.stdout:1112: done (1 left) 2024-09-11T01:36:57.184 INFO:tasks.rados.rados.0.smithi071.stdout:1114: done (0 left) 2024-09-11T01:36:57.185 INFO:tasks.rados.rados.0.smithi071.stdout:1115: write oid 22 current snap is 117 2024-09-11T01:36:57.185 INFO:tasks.rados.rados.0.smithi071.stdout:1115: seq_num 420 ranges {658940=403995,1777496=676136,2950951=107989} 2024-09-11T01:36:57.190 INFO:tasks.rados.rados.0.smithi071.stdout:1115: writing smithi07134777-22 from 658940 to 1062935 tid 1 2024-09-11T01:36:57.198 INFO:tasks.rados.rados.0.smithi071.stdout:1115: writing smithi07134777-22 from 1777496 to 2453632 tid 2 2024-09-11T01:36:57.202 INFO:tasks.rados.rados.0.smithi071.stdout:1115: writing smithi07134777-22 from 2950951 to 3058940 tid 3 2024-09-11T01:36:57.202 INFO:tasks.rados.rados.0.smithi071.stdout:1116: copy_from oid 47 from oid 38 current snap is 117 2024-09-11T01:36:57.202 INFO:tasks.rados.rados.0.smithi071.stdout:1115: finishing write tid 1 to smithi07134777-22 2024-09-11T01:36:57.202 INFO:tasks.rados.rados.0.smithi071.stdout:1117: read oid 10 snap 106 2024-09-11T01:36:57.203 INFO:tasks.rados.rados.0.smithi071.stdout:1117: expect (ObjNum 370 snap 104 seq_num 370) 2024-09-11T01:36:57.203 INFO:tasks.rados.rados.0.smithi071.stdout:1118: write (excl) oid 49 current snap is 117 2024-09-11T01:36:57.203 INFO:tasks.rados.rados.0.smithi071.stdout:1118: seq_num 421 ranges {675747=479244,1835978=536147} 2024-09-11T01:36:57.209 INFO:tasks.rados.rados.0.smithi071.stdout:1118: writing smithi07134777-49 from 675747 to 1154991 tid 1 2024-09-11T01:36:57.216 INFO:tasks.rados.rados.0.smithi071.stdout:1118: writing smithi07134777-49 from 1835978 to 2372125 tid 2 2024-09-11T01:36:57.216 INFO:tasks.rados.rados.0.smithi071.stdout:1119: read oid 42 snap -1 2024-09-11T01:36:57.216 INFO:tasks.rados.rados.0.smithi071.stdout:1119: expect (ObjNum 414 snap 116 seq_num 414) 2024-09-11T01:36:57.216 INFO:tasks.rados.rados.0.smithi071.stdout:1120: snap_remove snap 116 2024-09-11T01:36:57.958 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:36:57.959 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:36:58.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.065+0000 7fb3fb3e8640 10 monclient: get_monmap_and_config 2024-09-11T01:36:58.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.065+0000 7fb3fb3e8640 10 monclient: build_initial_monmap 2024-09-11T01:36:58.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.065+0000 7fb3fb3e8640 10 monclient: monmap: 2024-09-11T01:36:58.066 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:58.066 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:58.067 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:58.065843+0000 2024-09-11T01:36:58.067 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:58.065843+0000 2024-09-11T01:36:58.067 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:58.067 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.067 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:58.067 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:58.067 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:58.067 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:58.067 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:58.067 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.065+0000 7fb3fb3e8640 1 Processor -- start 2024-09-11T01:36:58.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3fb3e8640 1 -- start start 2024-09-11T01:36:58.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3fb3e8640 10 monclient: init 2024-09-11T01:36:58.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3fb3e8640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:58.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3fb3e8640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-11T01:36:58.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3fb3e8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb3f4138230 0x7fb3f4138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3fb3e8640 10 monclient(hunting): picked mon.noname-b con 0x7fb3f4138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:58.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3fb3e8640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb3f409d030 0x7fb3f40a5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3fb3e8640 10 monclient(hunting): picked mon.noname-c con 0x7fb3f409d030 addr v2:172.21.15.71:3300/0 2024-09-11T01:36:58.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3fb3e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb3f409d430 addr v1:172.21.15.71:6789/0 2024-09-11T01:36:58.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3fb3e8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3f40a6e30 con 0x7fb3f409d430 2024-09-11T01:36:58.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3fb3e8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:58.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3fb3e8640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fb3f409c9c0 con 0x7fb3f409d030 2024-09-11T01:36:58.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3fb3e8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:58.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3fb3e8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb3f409ba20 con 0x7fb3f4138230 2024-09-11T01:36:58.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3fb3e8640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:58.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3fb3e8640 10 monclient(hunting): authenticate will time out at 1224.508664s 2024-09-11T01:36:58.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3f9184640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb3f4138230 0x7fb3f4138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3f9184640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb3f4138230 0x7fb3f4138610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:34596/0 (socket says 172.21.15.12:34596) 2024-09-11T01:36:58.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3f9184640 1 -- 172.21.15.12:0/2978084400 learned_addr learned my addr 172.21.15.12:0/2978084400 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:58.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3f9184640 10 monclient(hunting): get_auth_request con 0x7fb3f4138230 auth_method 0 2024-09-11T01:36:58.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3f9184640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:58.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3f9184640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:58.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3f9184640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:58.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3f9184640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:58.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3f9184640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:58.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3f9184640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:58.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3f9184640 10 monclient(hunting): handle_auth_done global_id 84119 payload 274 2024-09-11T01:36:58.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3f9184640 1 -- 172.21.15.12:0/2978084400 >> v2:172.21.15.71:3300/0 conn(0x7fb3f409d030 msgr2=0x7fb3f40a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:58.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3f9184640 1 --2- 172.21.15.12:0/2978084400 >> v2:172.21.15.71:3300/0 conn(0x7fb3f409d030 0x7fb3f40a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3f9184640 1 -- 172.21.15.12:0/2978084400 >> v1:172.21.15.71:6789/0 conn(0x7fb3f409d430 legacy=0x7fb3f40a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:58.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3f9184640 10 monclient: _finish_hunting 0 2024-09-11T01:36:58.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3f9184640 1 monclient: found mon.noname-b 2024-09-11T01:36:58.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3f9184640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:58.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3f9184640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:36:58.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.066+0000 7fb3f9184640 1 -- 172.21.15.12:0/2978084400 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3f40a66a0 con 0x7fb3f4138230 2024-09-11T01:36:58.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.069+0000 7fb3f9184640 1 -- 172.21.15.12:0/2978084400 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb3f4138230 msgr2=0x7fb3f4138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.069+0000 7fb3f9184640 1 --2- 172.21.15.12:0/2978084400 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb3f4138230 0x7fb3f4138610 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb3f406d650 tx=0x7fb3f002a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2052b34929a6d28e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.070+0000 7fb3ebfff640 1 -- 172.21.15.12:0/2978084400 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0037070 con 0x7fb3f4138230 2024-09-11T01:36:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.070+0000 7fb3ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.070+0000 7fb3ebfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:36:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.070+0000 7fb3ebfff640 10 monclient: dump: 2024-09-11T01:36:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:58.079 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.070+0000 7fb3ebfff640 10 monclient: _finish_auth 0 2024-09-11T01:36:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.070+0000 7fb3ebfff640 10 monclient: _check_auth_tickets 2024-09-11T01:36:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.070+0000 7fb3ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.070+0000 7fb3ebfff640 1 -- 172.21.15.12:0/2978084400 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3f0032640 con 0x7fb3f4138230 2024-09-11T01:36:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.070+0000 7fb3ebfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.070+0000 7fb3ebfff640 1 -- 172.21.15.12:0/2978084400 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0032a10 con 0x7fb3f4138230 2024-09-11T01:36:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.070+0000 7fb3fb3e8640 5 monclient: authenticate success, global_id 84119 2024-09-11T01:36:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.070+0000 7fb3fb3e8640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.070+0000 7fb3fb3e8640 4 set_mon_vals no callback set 2024-09-11T01:36:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.070+0000 7fb3ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.070+0000 7fb3ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.070+0000 7fb3ebfff640 10 monclient: dump: 2024-09-11T01:36:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.080 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.080 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:58.080 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.080 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:58.080 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:58.080 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:58.080 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.070+0000 7fb3fb3e8640 10 monclient: shutdown 2024-09-11T01:36:58.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.070+0000 7fb3fb3e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:58.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.070+0000 7fb3fb3e8640 1 -- 172.21.15.12:0/2978084400 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb3f4138230 msgr2=0x7fb3f4138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:58.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.070+0000 7fb3fb3e8640 1 --2- 172.21.15.12:0/2978084400 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb3f4138230 0x7fb3f4138610 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb3f406d650 tx=0x7fb3f002a080 comp rx=0 tx=0).stop 2024-09-11T01:36:58.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.070+0000 7fb3fb3e8640 1 -- 172.21.15.12:0/2978084400 shutdown_connections 2024-09-11T01:36:58.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.070+0000 7fb3fb3e8640 1 --2- 172.21.15.12:0/2978084400 >> v2:172.21.15.71:3300/0 conn(0x7fb3f409d030 0x7fb3f40a5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.070+0000 7fb3fb3e8640 1 --2- 172.21.15.12:0/2978084400 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb3f4138230 0x7fb3f4138610 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.070+0000 7fb3fb3e8640 1 -- 172.21.15.12:0/2978084400 >> 172.21.15.12:0/2978084400 conn(0x7fb3f40a7120 msgr2=0x7fb3f40a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:58.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.070+0000 7fb3fb3e8640 1 -- 172.21.15.12:0/2978084400 shutdown_connections 2024-09-11T01:36:58.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.071+0000 7fb3fb3e8640 1 -- 172.21.15.12:0/2978084400 wait complete. 2024-09-11T01:36:58.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.071+0000 7fb3fb3e8640 10 monclient: build_initial_monmap 2024-09-11T01:36:58.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.071+0000 7fb3fb3e8640 10 monclient: monmap: 2024-09-11T01:36:58.081 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:58.081 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:58.081 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:58.081 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:58.081 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:58.081 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.081 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:58.082 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:58.082 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:58.082 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.071+0000 7fb3fb3e8640 1 Processor -- start 2024-09-11T01:36:58.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.071+0000 7fb3fb3e8640 1 -- start start 2024-09-11T01:36:58.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.071+0000 7fb3fb3e8640 10 monclient: init 2024-09-11T01:36:58.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3fb3e8640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:58.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3fb3e8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:36:58.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3fb3e8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3f409d030 0x7fb3f4136d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3fb3e8640 10 monclient(hunting): picked mon.noname-a con 0x7fb3f409d030 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:58.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3fb3e8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb3f409d430 0x7fb3f413cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3fb3e8640 10 monclient(hunting): picked mon.noname-c con 0x7fb3f409d430 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:58.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3fb3e8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb3f4138230 0x7fb3f413d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3fb3e8640 10 monclient(hunting): picked mon.noname-b con 0x7fb3f4138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:58.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3fb3e8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:58.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3fb3e8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f406f970 con 0x7fb3f409d030 2024-09-11T01:36:58.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3fb3e8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:58.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3fb3e8640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f406f7f0 con 0x7fb3f4138230 2024-09-11T01:36:58.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3fb3e8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:58.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3fb3e8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb3f40a5c70 con 0x7fb3f409d430 2024-09-11T01:36:58.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3fb3e8640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:58.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3fb3e8640 10 monclient(hunting): authenticate will time out at 1224.514402s 2024-09-11T01:36:58.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f8983640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb3f409d430 0x7fb3f413cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f9184640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3f409d030 0x7fb3f4136d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f8983640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb3f409d430 0x7fb3f413cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:34612/0 (socket says 172.21.15.12:34612) 2024-09-11T01:36:58.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f8983640 1 -- 172.21.15.12:0/4180461207 learned_addr learned my addr 172.21.15.12:0/4180461207 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:58.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f9184640 10 monclient(hunting): get_auth_request con 0x7fb3f409d030 auth_method 0 2024-09-11T01:36:58.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f9184640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:58.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f9184640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:58.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f9184640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:58.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f8983640 10 monclient(hunting): get_auth_request con 0x7fb3f409d430 auth_method 0 2024-09-11T01:36:58.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f8983640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:58.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f8983640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:58.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f8983640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:58.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f9184640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:58.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f9985640 1 --2- 172.21.15.12:0/4180461207 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb3f4138230 0x7fb3f413d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f9184640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f9184640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f8983640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f8983640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f8983640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f9985640 10 monclient(hunting): get_auth_request con 0x7fb3f4138230 auth_method 0 2024-09-11T01:36:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f9985640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f9985640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f9985640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f9985640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f9985640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f9985640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f9184640 10 monclient(hunting): handle_auth_done global_id 54411 payload 824 2024-09-11T01:36:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f9184640 1 -- 172.21.15.12:0/4180461207 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb3f409d430 msgr2=0x7fb3f413cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f9184640 1 --2- 172.21.15.12:0/4180461207 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb3f409d430 0x7fb3f413cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f9184640 1 -- 172.21.15.12:0/4180461207 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb3f4138230 msgr2=0x7fb3f413d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f9184640 1 --2- 172.21.15.12:0/4180461207 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb3f4138230 0x7fb3f413d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f9184640 10 monclient: _finish_hunting 0 2024-09-11T01:36:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f9184640 1 monclient: found mon.noname-a 2024-09-11T01:36:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f9184640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f9184640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.072+0000 7fb3f9184640 1 -- 172.21.15.12:0/4180461207 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3f41217d0 con 0x7fb3f409d030 2024-09-11T01:36:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.075+0000 7fb3f9184640 1 -- 172.21.15.12:0/4180461207 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3f409d030 msgr2=0x7fb3f4136d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.075+0000 7fb3f9184640 1 --2- 172.21.15.12:0/4180461207 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3f409d030 0x7fb3f4136d70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0031000 tx=0x7fb3f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.075+0000 7fb3e9ffb640 1 -- 172.21.15.12:0/4180461207 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0041070 con 0x7fb3f409d030 2024-09-11T01:36:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.075+0000 7fb3e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.075+0000 7fb3e9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.075+0000 7fb3e9ffb640 10 monclient: dump: 2024-09-11T01:36:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:58.093 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.075+0000 7fb3e9ffb640 10 monclient: _finish_auth 0 2024-09-11T01:36:58.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.075+0000 7fb3e9ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:36:58.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.075+0000 7fb3e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:58.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.075+0000 7fb3e9ffb640 1 -- 172.21.15.12:0/4180461207 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3f003c880 con 0x7fb3f409d030 2024-09-11T01:36:58.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.075+0000 7fb3fb3e8640 5 monclient: authenticate success, global_id 54411 2024-09-11T01:36:58.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.075+0000 7fb3fb3e8640 10 monclient: _renew_subs 2024-09-11T01:36:58.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.075+0000 7fb3fb3e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:58.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.075+0000 7fb3fb3e8640 1 -- 172.21.15.12:0/4180461207 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3f413f320 con 0x7fb3f409d030 2024-09-11T01:36:58.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.075+0000 7fb3e9ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:58.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.075+0000 7fb3e9ffb640 1 -- 172.21.15.12:0/4180461207 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0037040 con 0x7fb3f409d030 2024-09-11T01:36:58.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.075+0000 7fb3e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:58.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.075+0000 7fb3e9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:58.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.075+0000 7fb3e9ffb640 10 monclient: dump: 2024-09-11T01:36:58.095 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:58.096 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:58.096 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.096 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.096 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:58.096 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.096 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:58.096 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:58.096 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:58.097 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.075+0000 7fb3eb7fe640 4 set_mon_vals no callback set 2024-09-11T01:36:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.075+0000 7fb3fb3e8640 10 monclient: _renew_subs 2024-09-11T01:36:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.075+0000 7fb3fb3e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.075+0000 7fb3fb3e8640 1 -- 172.21.15.12:0/4180461207 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3f413e630 con 0x7fb3f409d030 2024-09-11T01:36:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.076+0000 7fb3e9ffb640 1 -- 172.21.15.12:0/4180461207 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fb3f003cc50 con 0x7fb3f409d030 2024-09-11T01:36:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.076+0000 7fb3fb3e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.076+0000 7fb3fb3e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.076+0000 7fb3fb3e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.076+0000 7fb3fb3e8640 1 -- 172.21.15.12:0/4180461207 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3f4138610 con 0x7fb3f409d030 2024-09-11T01:36:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.076+0000 7fb3e9ffb640 1 --2- 172.21.15.12:0/4180461207 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb3d803e210 0x7fb3d80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.076+0000 7fb3e9ffb640 1 -- 172.21.15.12:0/4180461207 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(257..257 src has 186..257) ==== 4863+0+0 (secure 0 0 0) 0x7fb3f00738e0 con 0x7fb3f409d030 2024-09-11T01:36:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.077+0000 7fb3f8983640 1 --2- 172.21.15.12:0/4180461207 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb3d803e210 0x7fb3d80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.077+0000 7fb3f8983640 10 monclient: get_auth_request con 0x7fb3d803e210 auth_method 0 2024-09-11T01:36:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.079+0000 7fb3f8983640 1 --2- 172.21.15.12:0/4180461207 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb3d803e210 0x7fb3d80406d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009a40 tx=0x7fb3e40091c0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.081+0000 7fb3e9ffb640 1 -- 172.21.15.12:0/4180461207 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb3f0072290 con 0x7fb3f409d030 2024-09-11T01:36:58.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.081+0000 7fb3e9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:58.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.081+0000 7fb3e9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:58.160 INFO:tasks.rados.rados.0.smithi071.stdout:1120: done (5 left) 2024-09-11T01:36:58.160 INFO:tasks.rados.rados.0.smithi071.stdout:1121: snap_create 2024-09-11T01:36:58.160 INFO:tasks.rados.rados.0.smithi071.stdout:1115: finishing write tid 2 to smithi07134777-22 2024-09-11T01:36:58.161 INFO:tasks.rados.rados.0.smithi071.stdout:1115: finishing write tid 3 to smithi07134777-22 2024-09-11T01:36:58.161 INFO:tasks.rados.rados.0.smithi071.stdout:1115: finishing write tid 4 to smithi07134777-22 2024-09-11T01:36:58.161 INFO:tasks.rados.rados.0.smithi071.stdout:1115: finishing write tid 5 to smithi07134777-22 2024-09-11T01:36:58.161 INFO:tasks.rados.rados.0.smithi071.stdout:1115: oid 22 updating version 0 to 246 2024-09-11T01:36:58.161 INFO:tasks.rados.rados.0.smithi071.stdout:1115: oid 22 version 246 is already newer than 244 2024-09-11T01:36:58.161 INFO:tasks.rados.rados.0.smithi071.stdout:1115: oid 22 updating version 246 to 247 2024-09-11T01:36:58.161 INFO:tasks.rados.rados.0.smithi071.stdout:1115: oid 22 version 247 is already newer than 245 2024-09-11T01:36:58.161 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 247 (ObjNum 420 snap 117 seq_num 420) dirty exists 2024-09-11T01:36:58.161 INFO:tasks.rados.rados.0.smithi071.stdout:1115: left oid 22 (ObjNum 420 snap 117 seq_num 420) 2024-09-11T01:36:58.161 INFO:tasks.rados.rados.0.smithi071.stdout:1117: expect (ObjNum 370 snap 104 seq_num 370) 2024-09-11T01:36:58.290 INFO:tasks.rados.rados.0.smithi071.stdout:1118: finishing write tid 1 to smithi07134777-49 2024-09-11T01:36:58.290 INFO:tasks.rados.rados.0.smithi071.stdout:1116: finishing copy_from racing read to smithi07134777-47 2024-09-11T01:36:58.290 INFO:tasks.rados.rados.0.smithi071.stdout:1116: finishing copy_from to smithi07134777-47 2024-09-11T01:36:58.290 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 333 (ObjNum 269 snap 71 seq_num 269) dirty exists 2024-09-11T01:36:58.290 INFO:tasks.rados.rados.0.smithi071.stdout:1118: finishing write tid 2 to smithi07134777-49 2024-09-11T01:36:58.291 INFO:tasks.rados.rados.0.smithi071.stdout:1118: finishing write tid 3 to smithi07134777-49 2024-09-11T01:36:58.291 INFO:tasks.rados.rados.0.smithi071.stdout:1118: finishing write tid 4 to smithi07134777-49 2024-09-11T01:36:58.291 INFO:tasks.rados.rados.0.smithi071.stdout:1118: oid 49 updating version 0 to 234 2024-09-11T01:36:58.291 INFO:tasks.rados.rados.0.smithi071.stdout:1118: oid 49 updating version 234 to 235 2024-09-11T01:36:58.291 INFO:tasks.rados.rados.0.smithi071.stdout:1118: oid 49 version 235 is already newer than 233 2024-09-11T01:36:58.291 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 235 (ObjNum 421 snap 117 seq_num 421) dirty exists 2024-09-11T01:36:58.291 INFO:tasks.rados.rados.0.smithi071.stdout:1118: left oid 49 (ObjNum 421 snap 117 seq_num 421) 2024-09-11T01:36:58.291 INFO:tasks.rados.rados.0.smithi071.stdout:1119: expect (ObjNum 414 snap 116 seq_num 414) 2024-09-11T01:36:58.324 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:36:58.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.346+0000 7fb3fb3e8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:36:58.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.346+0000 7fb3fb3e8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:36:58.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.346+0000 7fb3fb3e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:58.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.346+0000 7fb3fb3e8640 1 -- 172.21.15.12:0/4180461207 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb3f40698e0 con 0x7fb3f409d030 2024-09-11T01:36:58.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.347+0000 7fb3e9ffb640 1 -- 172.21.15.12:0/4180461207 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fb3f003bd80 con 0x7fb3f409d030 2024-09-11T01:36:58.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.347+0000 7fb3e9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:36:58.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.347+0000 7fb3e9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:58.349 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:36:58.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.349+0000 7fb3fb3e8640 1 -- 172.21.15.12:0/4180461207 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb3d803e210 msgr2=0x7fb3d80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:58.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.349+0000 7fb3fb3e8640 1 --2- 172.21.15.12:0/4180461207 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb3d803e210 0x7fb3d80406d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009a40 tx=0x7fb3e40091c0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.349+0000 7fb3fb3e8640 10 monclient: shutdown 2024-09-11T01:36:58.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.349+0000 7fb3fb3e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:58.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.349+0000 7fb3fb3e8640 1 -- 172.21.15.12:0/4180461207 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3f409d030 msgr2=0x7fb3f4136d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:58.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.349+0000 7fb3fb3e8640 1 --2- 172.21.15.12:0/4180461207 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3f409d030 0x7fb3f4136d70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0031000 tx=0x7fb3f00046c0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.349+0000 7fb3fb3e8640 1 -- 172.21.15.12:0/4180461207 shutdown_connections 2024-09-11T01:36:58.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.349+0000 7fb3fb3e8640 1 --2- 172.21.15.12:0/4180461207 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb3f4138230 0x7fb3f413d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.349+0000 7fb3fb3e8640 1 --2- 172.21.15.12:0/4180461207 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb3f409d430 0x7fb3f413cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.349+0000 7fb3fb3e8640 1 --2- 172.21.15.12:0/4180461207 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb3d803e210 0x7fb3d80406d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.349+0000 7fb3fb3e8640 1 --2- 172.21.15.12:0/4180461207 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb3f409d030 0x7fb3f4136d70 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.349+0000 7fb3fb3e8640 1 -- 172.21.15.12:0/4180461207 >> 172.21.15.12:0/4180461207 conn(0x7fb3f40a7120 msgr2=0x7fb3f40a3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:58.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.349+0000 7fb3fb3e8640 1 -- 172.21.15.12:0/4180461207 shutdown_connections 2024-09-11T01:36:58.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.349+0000 7fb3fb3e8640 1 -- 172.21.15.12:0/4180461207 wait complete. 2024-09-11T01:36:58.361 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:36:58.361 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:36:58.361 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:36:58.361 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:36:58.368 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:36:58.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.461+0000 7fcb4c408640 10 monclient: get_monmap_and_config 2024-09-11T01:36:58.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.461+0000 7fcb4c408640 10 monclient: build_initial_monmap 2024-09-11T01:36:58.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.461+0000 7fcb4c408640 10 monclient: monmap: 2024-09-11T01:36:58.482 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:58.482 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:58.482 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:58.462484+0000 2024-09-11T01:36:58.482 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:58.462484+0000 2024-09-11T01:36:58.483 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:58.483 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.483 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:58.483 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:58.483 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:58.483 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:58.483 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:58.483 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.462+0000 7fcb4c408640 1 Processor -- start 2024-09-11T01:36:58.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.462+0000 7fcb4c408640 1 -- start start 2024-09-11T01:36:58.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.462+0000 7fcb4c408640 10 monclient: init 2024-09-11T01:36:58.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.462+0000 7fcb4c408640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:58.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.462+0000 7fcb4c408640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-11T01:36:58.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.462+0000 7fcb4c408640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcb440a2e20 addr v1:172.21.15.71:6789/0 2024-09-11T01:36:58.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.462+0000 7fcb4c408640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcb440a4970 0x7fcb4406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.462+0000 7fcb4c408640 10 monclient(hunting): picked mon.noname-a con 0x7fcb440a4970 addr v2:172.21.15.12:3300/0 2024-09-11T01:36:58.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4c408640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcb4406f910 0x7fcb4413f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4c408640 10 monclient(hunting): picked mon.noname-b con 0x7fcb4406f910 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:58.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4c408640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb4406f4f0 con 0x7fcb440a2e20 2024-09-11T01:36:58.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4c408640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:58.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4c408640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fcb4406ef30 con 0x7fcb440a4970 2024-09-11T01:36:58.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4c408640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:58.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4c408640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcb4406ec50 con 0x7fcb4406f910 2024-09-11T01:36:58.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4c408640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:58.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4c408640 10 monclient(hunting): authenticate will time out at 1224.905042s 2024-09-11T01:36:58.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb499a3640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcb440a4970 0x7fcb4406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4a9a5640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcb4406f910 0x7fcb4413f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4a9a5640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcb4406f910 0x7fcb4413f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:34616/0 (socket says 172.21.15.12:34616) 2024-09-11T01:36:58.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb499a3640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcb440a4970 0x7fcb4406ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36406/0 (socket says 172.21.15.12:36406) 2024-09-11T01:36:58.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4a9a5640 1 -- 172.21.15.12:0/1527400496 learned_addr learned my addr 172.21.15.12:0/1527400496 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:58.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4a9a5640 10 monclient(hunting): get_auth_request con 0x7fcb4406f910 auth_method 0 2024-09-11T01:36:58.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4a9a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:58.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4a9a5640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:58.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4a9a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:58.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb499a3640 10 monclient(hunting): get_auth_request con 0x7fcb440a4970 auth_method 0 2024-09-11T01:36:58.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb499a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:58.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb499a3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:58.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb499a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:58.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4a9a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:58.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4a9a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:58.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4a9a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:58.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb499a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:58.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb499a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:58.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb499a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:58.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4a9a5640 10 monclient(hunting): handle_auth_done global_id 84125 payload 274 2024-09-11T01:36:58.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4a9a5640 1 -- 172.21.15.12:0/1527400496 >> v2:172.21.15.12:3300/0 conn(0x7fcb440a4970 msgr2=0x7fcb4406ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:58.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4a9a5640 1 --2- 172.21.15.12:0/1527400496 >> v2:172.21.15.12:3300/0 conn(0x7fcb440a4970 0x7fcb4406ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4a9a5640 1 -- 172.21.15.12:0/1527400496 >> v1:172.21.15.71:6789/0 conn(0x7fcb440a2e20 legacy=0x7fcb440a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:58.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4a9a5640 10 monclient: _finish_hunting 0 2024-09-11T01:36:58.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4a9a5640 1 monclient: found mon.noname-b 2024-09-11T01:36:58.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4a9a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:58.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4a9a5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:36:58.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.463+0000 7fcb4a9a5640 1 -- 172.21.15.12:0/1527400496 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb440a3910 con 0x7fcb4406f910 2024-09-11T01:36:58.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb4a9a5640 1 -- 172.21.15.12:0/1527400496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcb4406f910 msgr2=0x7fcb4413f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:58.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb4a9a5640 1 --2- 172.21.15.12:0/1527400496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcb4406f910 0x7fcb4413f430 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c009e50 tx=0x7fcb3c02a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7bc1440babd9563 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:58.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb491a2640 1 -- 172.21.15.12:0/1527400496 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c036070 con 0x7fcb4406f910 2024-09-11T01:36:58.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb491a2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:58.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb491a2640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:36:58.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb491a2640 10 monclient: dump: 2024-09-11T01:36:58.487 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:58.487 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:58.487 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.487 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.487 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:58.487 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.487 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:58.488 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:58.488 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:58.488 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb491a2640 10 monclient: _finish_auth 0 2024-09-11T01:36:58.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb491a2640 10 monclient: _check_auth_tickets 2024-09-11T01:36:58.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb491a2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:58.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb491a2640 1 -- 172.21.15.12:0/1527400496 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb3c007800 con 0x7fcb4406f910 2024-09-11T01:36:58.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb4c408640 5 monclient: authenticate success, global_id 84125 2024-09-11T01:36:58.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb4c408640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:36:58.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb491a2640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:58.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb491a2640 1 -- 172.21.15.12:0/1527400496 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c007bd0 con 0x7fcb4406f910 2024-09-11T01:36:58.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb491a2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:58.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb491a2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:58.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb491a2640 10 monclient: dump: 2024-09-11T01:36:58.489 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:58.489 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:58.489 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.489 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.489 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:58.489 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.489 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:58.489 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:58.489 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:58.489 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb4c408640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:58.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb4c408640 4 set_mon_vals no callback set 2024-09-11T01:36:58.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb4c408640 10 monclient: shutdown 2024-09-11T01:36:58.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb4c408640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:58.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb4c408640 1 -- 172.21.15.12:0/1527400496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcb4406f910 msgr2=0x7fcb4413f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:58.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb4c408640 1 --2- 172.21.15.12:0/1527400496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcb4406f910 0x7fcb4413f430 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c009e50 tx=0x7fcb3c02a560 comp rx=0 tx=0).stop 2024-09-11T01:36:58.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb4c408640 1 -- 172.21.15.12:0/1527400496 shutdown_connections 2024-09-11T01:36:58.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb4c408640 1 --2- 172.21.15.12:0/1527400496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcb4406f910 0x7fcb4413f430 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb4c408640 1 --2- 172.21.15.12:0/1527400496 >> v2:172.21.15.12:3300/0 conn(0x7fcb440a4970 0x7fcb4406ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.466+0000 7fcb4c408640 1 -- 172.21.15.12:0/1527400496 >> 172.21.15.12:0/1527400496 conn(0x7fcb440a8600 msgr2=0x7fcb440a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:58.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.467+0000 7fcb4c408640 1 -- 172.21.15.12:0/1527400496 shutdown_connections 2024-09-11T01:36:58.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.467+0000 7fcb4c408640 1 -- 172.21.15.12:0/1527400496 wait complete. 2024-09-11T01:36:58.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.467+0000 7fcb4c408640 10 monclient: build_initial_monmap 2024-09-11T01:36:58.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.467+0000 7fcb4c408640 10 monclient: monmap: 2024-09-11T01:36:58.490 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:58.491 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:58.491 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:58.491 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:58.491 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:58.491 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.491 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:58.491 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:58.491 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:58.491 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.467+0000 7fcb4c408640 1 Processor -- start 2024-09-11T01:36:58.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.467+0000 7fcb4c408640 1 -- start start 2024-09-11T01:36:58.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.467+0000 7fcb4c408640 10 monclient: init 2024-09-11T01:36:58.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4c408640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:58.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4c408640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:36:58.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4c408640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb4406f910 0x7fcb4409e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4c408640 10 monclient(hunting): picked mon.noname-a con 0x7fcb4406f910 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:58.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4c408640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcb440a2e20 0x7fcb4413e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4c408640 10 monclient(hunting): picked mon.noname-b con 0x7fcb440a2e20 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:58.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4c408640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcb440a4970 0x7fcb4413ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4c408640 10 monclient(hunting): picked mon.noname-c con 0x7fcb440a4970 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:58.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4c408640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:58.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4c408640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4413fd00 con 0x7fcb4406f910 2024-09-11T01:36:58.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4c408640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:58.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4c408640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4413fb80 con 0x7fcb440a2e20 2024-09-11T01:36:58.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4c408640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:58.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4c408640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcb4413fe80 con 0x7fcb440a4970 2024-09-11T01:36:58.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4c408640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:58.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4c408640 10 monclient(hunting): authenticate will time out at 1224.910423s 2024-09-11T01:36:58.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a1a4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb4406f910 0x7fcb4409e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a9a5640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcb440a4970 0x7fcb4413ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a1a4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb4406f910 0x7fcb4409e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36416/0 (socket says 172.21.15.12:36416) 2024-09-11T01:36:58.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a9a5640 10 monclient(hunting): get_auth_request con 0x7fcb440a4970 auth_method 0 2024-09-11T01:36:58.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a1a4640 1 -- 172.21.15.12:0/3898665120 learned_addr learned my addr 172.21.15.12:0/3898665120 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:58.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a9a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:58.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a9a5640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:58.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a9a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:58.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a1a4640 10 monclient(hunting): get_auth_request con 0x7fcb4406f910 auth_method 0 2024-09-11T01:36:58.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a1a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:58.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a1a4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:58.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a1a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:58.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb499a3640 1 --2- 172.21.15.12:0/3898665120 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcb440a2e20 0x7fcb4413e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a9a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:58.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a9a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:58.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a9a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:58.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a1a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:58.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a1a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:58.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a1a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:58.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb499a3640 10 monclient(hunting): get_auth_request con 0x7fcb440a2e20 auth_method 0 2024-09-11T01:36:58.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb499a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:58.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb499a3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:58.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb499a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:58.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb499a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:58.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb499a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:58.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb499a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:58.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a9a5640 10 monclient(hunting): handle_auth_done global_id 84128 payload 824 2024-09-11T01:36:58.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a9a5640 1 -- 172.21.15.12:0/3898665120 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcb440a2e20 msgr2=0x7fcb4413e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:58.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a9a5640 1 --2- 172.21.15.12:0/3898665120 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcb440a2e20 0x7fcb4413e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a9a5640 1 -- 172.21.15.12:0/3898665120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb4406f910 msgr2=0x7fcb4409e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:58.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a9a5640 1 --2- 172.21.15.12:0/3898665120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb4406f910 0x7fcb4409e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a9a5640 10 monclient: _finish_hunting 0 2024-09-11T01:36:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a9a5640 1 monclient: found mon.noname-c 2024-09-11T01:36:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a9a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a9a5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:36:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a9a5640 1 -- 172.21.15.12:0/3898665120 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb4413c170 con 0x7fcb440a4970 2024-09-11T01:36:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.468+0000 7fcb4a1a4640 1 --2- 172.21.15.12:0/3898665120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb4406f910 0x7fcb4409e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:36:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.471+0000 7fcb4a9a5640 1 -- 172.21.15.12:0/3898665120 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcb440a4970 msgr2=0x7fcb4413ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.471+0000 7fcb4a9a5640 1 --2- 172.21.15.12:0/3898665120 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcb440a4970 0x7fcb4413ebf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c030000 tx=0x7fcb3c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.471+0000 7fcb3affd640 1 -- 172.21.15.12:0/3898665120 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c036050 con 0x7fcb440a4970 2024-09-11T01:36:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.471+0000 7fcb3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.471+0000 7fcb3affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:36:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.471+0000 7fcb3affd640 10 monclient: dump: 2024-09-11T01:36:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:58.496 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.497 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.497 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:58.497 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.497 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:58.497 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:58.497 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:58.497 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.471+0000 7fcb3affd640 10 monclient: _finish_auth 0 2024-09-11T01:36:58.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.471+0000 7fcb3affd640 10 monclient: _check_auth_tickets 2024-09-11T01:36:58.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.471+0000 7fcb3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:58.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.471+0000 7fcb3affd640 1 -- 172.21.15.12:0/3898665120 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb3c042070 con 0x7fcb440a4970 2024-09-11T01:36:58.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.471+0000 7fcb4c408640 5 monclient: authenticate success, global_id 84128 2024-09-11T01:36:58.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.471+0000 7fcb4c408640 10 monclient: _renew_subs 2024-09-11T01:36:58.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.471+0000 7fcb4c408640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:36:58.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.471+0000 7fcb4c408640 1 -- 172.21.15.12:0/3898665120 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb44139a90 con 0x7fcb440a4970 2024-09-11T01:36:58.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.471+0000 7fcb3affd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:58.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.471+0000 7fcb3affd640 1 -- 172.21.15.12:0/3898665120 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c031e10 con 0x7fcb440a4970 2024-09-11T01:36:58.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.471+0000 7fcb3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:58.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.471+0000 7fcb3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:58.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.471+0000 7fcb3affd640 10 monclient: dump: 2024-09-11T01:36:58.498 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:58.498 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:58.498 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.498 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.498 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:58.498 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.498 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:58.499 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:58.499 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:58.499 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.471+0000 7fcb489a1640 4 set_mon_vals no callback set 2024-09-11T01:36:58.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.471+0000 7fcb4c408640 10 monclient: _renew_subs 2024-09-11T01:36:58.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.471+0000 7fcb4c408640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:36:58.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.471+0000 7fcb4c408640 1 -- 172.21.15.12:0/3898665120 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb4409eee0 con 0x7fcb440a4970 2024-09-11T01:36:58.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.472+0000 7fcb4c408640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:58.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.472+0000 7fcb4c408640 1 --2- 172.21.15.12:0/3898665120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb441cd8f0 0x7fcb4402d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.472+0000 7fcb4c408640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:58.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.472+0000 7fcb4c408640 1 -- 172.21.15.12:0/3898665120 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4406a060 con 0x7fcb441cd8f0 2024-09-11T01:36:58.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.472+0000 7fcb4a9a5640 1 --2- 172.21.15.12:0/3898665120 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcb440a4970 0x7fcb4413ebf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c030000 tx=0x7fcb3c004a40 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7fcb3c073db0 osd_map(258..258 src has 186..258) 2024-09-11T01:36:58.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.472+0000 7fcb4a1a4640 1 --2- 172.21.15.12:0/3898665120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb441cd8f0 0x7fcb4402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.472+0000 7fcb3affd640 1 -- 172.21.15.12:0/3898665120 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fcb3c047460 con 0x7fcb440a4970 2024-09-11T01:36:58.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.472+0000 7fcb4a1a4640 10 monclient: get_auth_request con 0x7fcb441cd8f0 auth_method 0 2024-09-11T01:36:58.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.472+0000 7fcb4a1a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:58.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.472+0000 7fcb4a1a4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:58.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.472+0000 7fcb4a1a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:58.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.472+0000 7fcb4a1a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:58.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.472+0000 7fcb4a1a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:58.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.472+0000 7fcb4a1a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:58.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.472+0000 7fcb4a1a4640 10 monclient(hunting): handle_auth_done global_id 54420 payload 824 2024-09-11T01:36:58.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.472+0000 7fcb4a1a4640 1 -- 172.21.15.12:0/3898665120 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcb4406e4d0 con 0x7fcb441cd8f0 2024-09-11T01:36:58.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.472+0000 7fcb3affd640 1 --2- 172.21.15.12:0/3898665120 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcb2803e240 0x7fcb28040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.472+0000 7fcb499a3640 1 --2- 172.21.15.12:0/3898665120 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcb2803e240 0x7fcb28040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.472+0000 7fcb499a3640 10 monclient: get_auth_request con 0x7fcb2803e240 auth_method 0 2024-09-11T01:36:58.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.474+0000 7fcb4a1a4640 1 -- 172.21.15.12:0/3898665120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb441cd8f0 msgr2=0x7fcb4402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:58.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.474+0000 7fcb4a1a4640 1 --2- 172.21.15.12:0/3898665120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb441cd8f0 0x7fcb4402d440 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcb4000fb10 tx=0x7fcb40004580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:58.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.474+0000 7fcb3affd640 1 -- 172.21.15.12:0/3898665120 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb40018020 con 0x7fcb441cd8f0 2024-09-11T01:36:58.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.474+0000 7fcb3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:58.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.474+0000 7fcb3affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:58.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.474+0000 7fcb3affd640 10 monclient: dump: 2024-09-11T01:36:58.501 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:58.501 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:58.501 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.501 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.501 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:58.501 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.501 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:58.501 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:58.501 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:58.502 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.475+0000 7fcb3affd640 1 -- 172.21.15.12:0/3898665120 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcb4000de50 con 0x7fcb441cd8f0 2024-09-11T01:36:58.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.475+0000 7fcb3affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:58.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.475+0000 7fcb3affd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:58.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.475+0000 7fcb3affd640 1 -- 172.21.15.12:0/3898665120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb441cd8f0 msgr2=0x7fcb4402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:58.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.475+0000 7fcb3affd640 1 --2- 172.21.15.12:0/3898665120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb441cd8f0 0x7fcb4402d440 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcb4000fb10 tx=0x7fcb40004580 comp rx=0 tx=0).stop 2024-09-11T01:36:58.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.475+0000 7fcb3affd640 1 -- 172.21.15.12:0/3898665120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb441cd8f0 msgr2=0x7fcb4402d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:58.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.475+0000 7fcb3affd640 1 --2- 172.21.15.12:0/3898665120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb441cd8f0 0x7fcb4402d440 secure :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcb4000fb10 tx=0x7fcb40004580 comp rx=0 tx=0).stop 2024-09-11T01:36:58.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.475+0000 7fcb499a3640 1 --2- 172.21.15.12:0/3898665120 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcb2803e240 0x7fcb28040700 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcb441ce1f0 tx=0x7fcb34009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:58.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.481+0000 7fcb4c408640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:36:58.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.481+0000 7fcb4c408640 1 --2- 172.21.15.12:0/3898665120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb441cdcd0 0x7fcb4402d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.481+0000 7fcb4c408640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:58.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.481+0000 7fcb4c408640 1 -- 172.21.15.12:0/3898665120 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4406ded0 con 0x7fcb441cdcd0 2024-09-11T01:36:58.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.481+0000 7fcb4a1a4640 1 --2- 172.21.15.12:0/3898665120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb441cdcd0 0x7fcb4402d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.481+0000 7fcb4a1a4640 10 monclient: get_auth_request con 0x7fcb441cdcd0 auth_method 0 2024-09-11T01:36:58.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.481+0000 7fcb4a1a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:58.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.481+0000 7fcb4a1a4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:58.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.481+0000 7fcb4a1a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:58.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.481+0000 7fcb4a1a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:58.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.481+0000 7fcb4a1a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:58.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.481+0000 7fcb4a1a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:58.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.481+0000 7fcb4a1a4640 10 monclient(hunting): handle_auth_done global_id 54423 payload 824 2024-09-11T01:36:58.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.481+0000 7fcb4a1a4640 1 -- 172.21.15.12:0/3898665120 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcb4413b8d0 con 0x7fcb441cdcd0 2024-09-11T01:36:58.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.483+0000 7fcb4a1a4640 1 -- 172.21.15.12:0/3898665120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb441cdcd0 msgr2=0x7fcb4402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:58.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.483+0000 7fcb4a1a4640 1 --2- 172.21.15.12:0/3898665120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb441cdcd0 0x7fcb4402d980 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcb40019000 tx=0x7fcb400036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:58.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.484+0000 7fcb3affd640 1 -- 172.21.15.12:0/3898665120 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb40018030 con 0x7fcb441cdcd0 2024-09-11T01:36:58.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.484+0000 7fcb3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:58.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.484+0000 7fcb3affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:58.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.484+0000 7fcb3affd640 10 monclient: dump: 2024-09-11T01:36:58.506 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:58.506 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:58.506 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.506 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.506 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:58.506 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.507 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:58.507 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:58.507 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:58.507 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.484+0000 7fcb3affd640 1 -- 172.21.15.12:0/3898665120 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4877 (secure 0 0 0) 0x7fcb40014b80 con 0x7fcb441cdcd0 2024-09-11T01:36:58.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.484+0000 7fcb3affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:36:58.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.484+0000 7fcb3affd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:58.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.484+0000 7fcb3affd640 1 -- 172.21.15.12:0/3898665120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb441cdcd0 msgr2=0x7fcb4402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:58.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.484+0000 7fcb3affd640 1 --2- 172.21.15.12:0/3898665120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb441cdcd0 0x7fcb4402d980 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcb40019000 tx=0x7fcb400036f0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.484+0000 7fcb3affd640 1 -- 172.21.15.12:0/3898665120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb441cdcd0 msgr2=0x7fcb4402d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:58.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.484+0000 7fcb3affd640 1 --2- 172.21.15.12:0/3898665120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcb441cdcd0 0x7fcb4402d980 secure :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcb40019000 tx=0x7fcb400036f0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.508 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:36:58.509 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:36:58.509 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:36:58.509 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:36:58.509 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 80, 2024-09-11T01:36:58.509 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 208465, 2024-09-11T01:36:58.509 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:36:58.509 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:36:58.509 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:36:58.509 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:36:58.509 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:58.509 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 6, 2024-09-11T01:36:58.509 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:58.509 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:36:58.510 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:36:58.510 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:58.510 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:58.510 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:58.510 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:58.510 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:58.510 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:58.510 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:58.510 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:58.510 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:58.510 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:58.511 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:58.511 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:58.511 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:36:58.511 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:36:58.511 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:58.511 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:58.511 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:58.511 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:58.511 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:58.511 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:58.511 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:58.511 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:58.511 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:58.511 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:58.512 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:58.512 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:58.512 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.512 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:36:58.512 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:36:58.512 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:36:58.512 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:36:58.512 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:36:58.512 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:36:58.512 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:58.512 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:58.512 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:36:58.512 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:36:58.513 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:36:58.513 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:36:58.513 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:36:58.513 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:36:58.513 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:36:58.513 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:58.513 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:36:58.513 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:58.513 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:58.513 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:58.513 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:58.513 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:58.513 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:58.514 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:58.514 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:58.514 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:58.514 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:58.514 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:58.514 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:58.514 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:36:58.514 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.514 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:36:58.514 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.514 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:36:58.514 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:36:58.514 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:58.514 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:58.515 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.515 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:58.515 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:36:58.515 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:58.515 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.515 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.515 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:58.515 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:36:58.515 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:58.515 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:58.515 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:58.515 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.515 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:36:58.516 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:36:58.516 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:58.516 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:58.516 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:58.516 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.516 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.516 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:36:58.516 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:36:58.516 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:58.516 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:58.516 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.516 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:58.517 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:36:58.517 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:58.517 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.517 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.517 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:58.517 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:36:58.517 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:58.517 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:58.517 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:58.517 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.517 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:36:58.517 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:36:58.517 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:58.517 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:58.517 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:58.518 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.518 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.518 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:36:58.518 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:36:58.518 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:58.518 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:58.518 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.518 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:58.518 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:36:58.518 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:58.518 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.518 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.518 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:58.518 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:36:58.519 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:58.519 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:58.519 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:58.519 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.519 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:36:58.519 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:36:58.519 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:58.519 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:58.519 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:58.519 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:58.519 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:58.519 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.519 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:36:58.519 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:36:58.520 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.520 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:58.520 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:58.520 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:36:58.520 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:58.520 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:58.520 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:36:58.520 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.520 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:58.520 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:58.520 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:36:58.520 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:58.520 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:58.521 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:36:58.521 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.521 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:58.521 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:58.521 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:36:58.521 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:58.521 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:58.521 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:36:58.521 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.521 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:58.521 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:58.521 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:36:58.521 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:58.521 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:58.522 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.522 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:36:58.522 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:36:58.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.484+0000 7fcb4c408640 1 -- 172.21.15.12:0/3898665120 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcb2803e240 msgr2=0x7fcb28040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:58.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.484+0000 7fcb4c408640 1 --2- 172.21.15.12:0/3898665120 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcb2803e240 0x7fcb28040700 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcb441ce1f0 tx=0x7fcb34009190 comp rx=0 tx=0).stop 2024-09-11T01:36:58.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.484+0000 7fcb4c408640 10 monclient: shutdown 2024-09-11T01:36:58.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.484+0000 7fcb4c408640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:58.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.484+0000 7fcb4c408640 1 -- 172.21.15.12:0/3898665120 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcb440a4970 msgr2=0x7fcb4413ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:58.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.484+0000 7fcb4a9a5640 1 -- 172.21.15.12:0/3898665120 reap_dead start 2024-09-11T01:36:58.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.484+0000 7fcb4c408640 1 --2- 172.21.15.12:0/3898665120 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcb440a4970 0x7fcb4413ebf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c030000 tx=0x7fcb3c004a40 comp rx=0 tx=0).stop 2024-09-11T01:36:58.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.484+0000 7fcb3affd640 1 -- 172.21.15.12:0/3898665120 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(258..258 src has 186..258) ==== 4875+0+0 (unknown 0 0 0) 0x7fcb3c073db0 con 0x7fcb440a4970 2024-09-11T01:36:58.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.484+0000 7fcb4c408640 1 -- 172.21.15.12:0/3898665120 shutdown_connections 2024-09-11T01:36:58.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.484+0000 7fcb4c408640 1 --2- 172.21.15.12:0/3898665120 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcb440a4970 0x7fcb4413ebf0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.484+0000 7fcb4c408640 1 -- 172.21.15.12:0/3898665120 >> 172.21.15.12:0/3898665120 conn(0x7fcb440a8600 msgr2=0x7fcb440a1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:58.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.484+0000 7fcb4c408640 1 -- 172.21.15.12:0/3898665120 shutdown_connections 2024-09-11T01:36:58.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.484+0000 7fcb4c408640 1 -- 172.21.15.12:0/3898665120 wait complete. 2024-09-11T01:36:58.524 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:36:58.524 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:36:58.524 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:36:58.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.624+0000 7f798415d640 10 monclient: get_monmap_and_config 2024-09-11T01:36:58.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.624+0000 7f798415d640 10 monclient: build_initial_monmap 2024-09-11T01:36:58.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.624+0000 7f798415d640 10 monclient: monmap: 2024-09-11T01:36:58.626 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:58.626 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:58.625617+0000 2024-09-11T01:36:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:58.625617+0000 2024-09-11T01:36:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:58.627 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:58.627 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.625+0000 7f798415d640 1 Processor -- start 2024-09-11T01:36:58.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.625+0000 7f798415d640 1 -- start start 2024-09-11T01:36:58.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.625+0000 7f798415d640 10 monclient: init 2024-09-11T01:36:58.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f798415d640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:58.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f798415d640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-11T01:36:58.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f798415d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c0a2e20 0x7f797c06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f798415d640 10 monclient(hunting): picked mon.noname-b con 0x7f797c0a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:58.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f798415d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f797c0a4970 addr v1:172.21.15.71:6789/0 2024-09-11T01:36:58.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f798415d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f797c06f910 0x7f797c13f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f798415d640 10 monclient(hunting): picked mon.noname-c con 0x7f797c06f910 addr v2:172.21.15.71:3300/0 2024-09-11T01:36:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f798415d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f797c06f4f0 con 0x7f797c0a4970 2024-09-11T01:36:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f798415d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f798415d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f797c06ef30 con 0x7f797c06f910 2024-09-11T01:36:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f798415d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f798415d640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f797c06ec50 con 0x7f797c0a2e20 2024-09-11T01:36:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f798415d640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f798415d640 10 monclient(hunting): authenticate will time out at 1225.068232s 2024-09-11T01:36:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f7981ef9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c0a2e20 0x7f797c06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f7981ef9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c0a2e20 0x7f797c06ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:34630/0 (socket says 172.21.15.12:34630) 2024-09-11T01:36:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f7981ef9640 1 -- 172.21.15.12:0/859190099 learned_addr learned my addr 172.21.15.12:0/859190099 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f7981ef9640 10 monclient(hunting): get_auth_request con 0x7f797c0a2e20 auth_method 0 2024-09-11T01:36:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f7981ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:58.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f7981ef9640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f7981ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f79826fa640 1 --2- 172.21.15.12:0/859190099 >> v2:172.21.15.71:3300/0 conn(0x7f797c06f910 0x7f797c13f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f7981ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f7981ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f7981ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f79826fa640 10 monclient(hunting): get_auth_request con 0x7f797c06f910 auth_method 0 2024-09-11T01:36:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f79826fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f79826fa640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f79826fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f7981ef9640 10 monclient(hunting): handle_auth_done global_id 84131 payload 274 2024-09-11T01:36:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f7981ef9640 1 -- 172.21.15.12:0/859190099 >> v2:172.21.15.71:3300/0 conn(0x7f797c06f910 msgr2=0x7f797c13f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f7981ef9640 1 --2- 172.21.15.12:0/859190099 >> v2:172.21.15.71:3300/0 conn(0x7f797c06f910 0x7f797c13f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f7981ef9640 1 -- 172.21.15.12:0/859190099 >> v1:172.21.15.71:6789/0 conn(0x7f797c0a4970 legacy=0x7f797c0a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:58.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f7981ef9640 10 monclient: _finish_hunting 0 2024-09-11T01:36:58.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f7981ef9640 1 monclient: found mon.noname-b 2024-09-11T01:36:58.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f7981ef9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:58.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f7981ef9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:36:58.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.626+0000 7f7981ef9640 1 -- 172.21.15.12:0/859190099 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f797c0a3910 con 0x7f797c0a2e20 2024-09-11T01:36:58.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.629+0000 7f7981ef9640 1 -- 172.21.15.12:0/859190099 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c0a2e20 msgr2=0x7f797c06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:58.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.629+0000 7f7981ef9640 1 --2- 172.21.15.12:0/859190099 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c0a2e20 0x7f797c06ded0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7978009870 tx=0x7f797802a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83285bcb90a680c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:58.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.629+0000 7f7980ef7640 1 -- 172.21.15.12:0/859190099 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7978037070 con 0x7f797c0a2e20 2024-09-11T01:36:58.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.629+0000 7f7980ef7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:58.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.629+0000 7f7980ef7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:36:58.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.629+0000 7f7980ef7640 10 monclient: dump: 2024-09-11T01:36:58.645 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:58.645 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:58.645 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.645 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.646 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:58.646 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.646 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:58.646 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:58.646 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:58.647 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.629+0000 7f7980ef7640 10 monclient: _finish_auth 0 2024-09-11T01:36:58.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.629+0000 7f7980ef7640 10 monclient: _check_auth_tickets 2024-09-11T01:36:58.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.629+0000 7f7980ef7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:58.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.629+0000 7f7980ef7640 1 -- 172.21.15.12:0/859190099 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7978032440 con 0x7f797c0a2e20 2024-09-11T01:36:58.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.629+0000 7f798415d640 5 monclient: authenticate success, global_id 84131 2024-09-11T01:36:58.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.629+0000 7f798415d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:36:58.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.629+0000 7f7980ef7640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:58.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.629+0000 7f798415d640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:58.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.629+0000 7f7980ef7640 1 -- 172.21.15.12:0/859190099 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7978032810 con 0x7f797c0a2e20 2024-09-11T01:36:58.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.629+0000 7f7980ef7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:58.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.629+0000 7f798415d640 4 set_mon_vals no callback set 2024-09-11T01:36:58.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.629+0000 7f7980ef7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:58.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.629+0000 7f7980ef7640 10 monclient: dump: 2024-09-11T01:36:58.649 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:58.649 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:58.649 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.650 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.650 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:58.650 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.650 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:58.650 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:58.651 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:58.651 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.630+0000 7f798415d640 10 monclient: shutdown 2024-09-11T01:36:58.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.630+0000 7f798415d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:58.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.630+0000 7f798415d640 1 -- 172.21.15.12:0/859190099 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c0a2e20 msgr2=0x7f797c06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:58.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.630+0000 7f798415d640 1 --2- 172.21.15.12:0/859190099 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c0a2e20 0x7f797c06ded0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7978009870 tx=0x7f797802a0a0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.630+0000 7f798415d640 1 -- 172.21.15.12:0/859190099 shutdown_connections 2024-09-11T01:36:58.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.630+0000 7f798415d640 1 --2- 172.21.15.12:0/859190099 >> v2:172.21.15.71:3300/0 conn(0x7f797c06f910 0x7f797c13f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.630+0000 7f798415d640 1 --2- 172.21.15.12:0/859190099 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c0a2e20 0x7f797c06ded0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.630+0000 7f798415d640 1 -- 172.21.15.12:0/859190099 >> 172.21.15.12:0/859190099 conn(0x7f797c0a8600 msgr2=0x7f797c0a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:58.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.630+0000 7f798415d640 1 -- 172.21.15.12:0/859190099 shutdown_connections 2024-09-11T01:36:58.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.630+0000 7f798415d640 1 -- 172.21.15.12:0/859190099 wait complete. 2024-09-11T01:36:58.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.630+0000 7f798415d640 10 monclient: build_initial_monmap 2024-09-11T01:36:58.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.630+0000 7f798415d640 10 monclient: monmap: 2024-09-11T01:36:58.653 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:58.653 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:58.653 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:58.653 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:58.653 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:58.653 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.654 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:58.654 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:58.654 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:58.654 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f798415d640 1 Processor -- start 2024-09-11T01:36:58.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f798415d640 1 -- start start 2024-09-11T01:36:58.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f798415d640 10 monclient: init 2024-09-11T01:36:58.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f798415d640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:58.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f798415d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:36:58.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f798415d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f797c06f910 0x7f797c09e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f798415d640 10 monclient(hunting): picked mon.noname-b con 0x7f797c06f910 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:58.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f798415d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f797c0a2e20 0x7f797c13e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f798415d640 10 monclient(hunting): picked mon.noname-a con 0x7f797c0a2e20 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:58.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f798415d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c0a4970 0x7f797c13ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f798415d640 10 monclient(hunting): picked mon.noname-c con 0x7f797c0a4970 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:58.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f798415d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:58.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f798415d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f797c13fd00 con 0x7f797c0a2e20 2024-09-11T01:36:58.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f798415d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:58.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f798415d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f797c13fb80 con 0x7f797c06f910 2024-09-11T01:36:58.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f798415d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:58.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f798415d640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f797c13fe80 con 0x7f797c0a4970 2024-09-11T01:36:58.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f798415d640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:58.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f798415d640 10 monclient(hunting): authenticate will time out at 1225.073599s 2024-09-11T01:36:58.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f79826fa640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c0a4970 0x7f797c13ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f79816f8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f797c0a2e20 0x7f797c13e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f79826fa640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c0a4970 0x7f797c13ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:34638/0 (socket says 172.21.15.12:34638) 2024-09-11T01:36:58.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f79826fa640 1 -- 172.21.15.12:0/151326417 learned_addr learned my addr 172.21.15.12:0/151326417 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:58.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f79826fa640 10 monclient(hunting): get_auth_request con 0x7f797c0a4970 auth_method 0 2024-09-11T01:36:58.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f79826fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:58.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f79826fa640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:58.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f79826fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:58.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f79816f8640 10 monclient(hunting): get_auth_request con 0x7f797c0a2e20 auth_method 0 2024-09-11T01:36:58.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f79816f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:58.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f79816f8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:58.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f79816f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:58.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f7981ef9640 1 --2- 172.21.15.12:0/151326417 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f797c06f910 0x7f797c09e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f7981ef9640 10 monclient(hunting): get_auth_request con 0x7f797c06f910 auth_method 0 2024-09-11T01:36:58.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f7981ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:58.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f7981ef9640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:58.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f7981ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:58.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f79826fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:58.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f79826fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:58.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f79826fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:58.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f79816f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:58.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f79816f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:58.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f79816f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:58.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f7981ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:58.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f7981ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:58.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.631+0000 7f7981ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:58.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.632+0000 7f79826fa640 10 monclient(hunting): handle_auth_done global_id 84134 payload 824 2024-09-11T01:36:58.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.632+0000 7f79826fa640 1 -- 172.21.15.12:0/151326417 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f797c06f910 msgr2=0x7f797c09e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:58.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.632+0000 7f79826fa640 1 --2- 172.21.15.12:0/151326417 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f797c06f910 0x7f797c09e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.632+0000 7f79826fa640 1 -- 172.21.15.12:0/151326417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f797c0a2e20 msgr2=0x7f797c13e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:58.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.632+0000 7f79826fa640 1 --2- 172.21.15.12:0/151326417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f797c0a2e20 0x7f797c13e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.632+0000 7f79826fa640 10 monclient: _finish_hunting 0 2024-09-11T01:36:58.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.632+0000 7f79826fa640 1 monclient: found mon.noname-c 2024-09-11T01:36:58.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.632+0000 7f79826fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:58.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.632+0000 7f79826fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:36:58.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.632+0000 7f79826fa640 1 -- 172.21.15.12:0/151326417 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f797c09eee0 con 0x7f797c0a4970 2024-09-11T01:36:58.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.632+0000 7f79816f8640 1 --2- 172.21.15.12:0/151326417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f797c0a2e20 0x7f797c13e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:36:58.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.634+0000 7f79826fa640 1 -- 172.21.15.12:0/151326417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c0a4970 msgr2=0x7f797c13ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:58.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.634+0000 7f79826fa640 1 --2- 172.21.15.12:0/151326417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c0a4970 0x7f797c13ebd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f797400d5d0 tx=0x7f797400da90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:58.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.634+0000 7f79727fc640 1 -- 172.21.15.12:0/151326417 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7974012800 con 0x7f797c0a4970 2024-09-11T01:36:58.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.634+0000 7f79727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:58.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.634+0000 7f79727fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:36:58.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.634+0000 7f79727fc640 10 monclient: dump: 2024-09-11T01:36:58.659 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:58.659 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:58.659 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.659 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.659 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:58.659 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.659 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:58.659 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:58.659 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:58.659 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.634+0000 7f79727fc640 10 monclient: _finish_auth 0 2024-09-11T01:36:58.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.634+0000 7f79727fc640 10 monclient: _check_auth_tickets 2024-09-11T01:36:58.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.634+0000 7f79727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:58.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.634+0000 7f798415d640 5 monclient: authenticate success, global_id 84134 2024-09-11T01:36:58.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.634+0000 7f79727fc640 1 -- 172.21.15.12:0/151326417 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7974019070 con 0x7f797c0a4970 2024-09-11T01:36:58.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.634+0000 7f79727fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:58.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.634+0000 7f79727fc640 1 -- 172.21.15.12:0/151326417 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7974012be0 con 0x7f797c0a4970 2024-09-11T01:36:58.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.634+0000 7f79727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:58.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.634+0000 7f79727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:58.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.634+0000 7f79727fc640 10 monclient: dump: 2024-09-11T01:36:58.660 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:58.660 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:58.660 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.660 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.661 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:58.661 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.661 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:58.661 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:58.661 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:58.661 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.634+0000 7f798415d640 10 monclient: _renew_subs 2024-09-11T01:36:58.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.634+0000 7f798415d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:36:58.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.634+0000 7f798415d640 1 -- 172.21.15.12:0/151326417 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f797c13a970 con 0x7f797c0a4970 2024-09-11T01:36:58.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.634+0000 7f7973fff640 4 set_mon_vals no callback set 2024-09-11T01:36:58.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.634+0000 7f798415d640 10 monclient: _renew_subs 2024-09-11T01:36:58.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.634+0000 7f798415d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:36:58.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.634+0000 7f798415d640 1 -- 172.21.15.12:0/151326417 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f797c13b8d0 con 0x7f797c0a4970 2024-09-11T01:36:58.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.635+0000 7f798415d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:58.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.635+0000 7f798415d640 1 --2- 172.21.15.12:0/151326417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c13baa0 0x7f797c1ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.635+0000 7f798415d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:58.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.635+0000 7f798415d640 1 -- 172.21.15.12:0/151326417 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f797c06a060 con 0x7f797c13baa0 2024-09-11T01:36:58.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.635+0000 7f7981ef9640 1 --2- 172.21.15.12:0/151326417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c13baa0 0x7f797c1ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.635+0000 7f7981ef9640 10 monclient: get_auth_request con 0x7f797c13baa0 auth_method 0 2024-09-11T01:36:58.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.635+0000 7f7981ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:58.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.635+0000 7f7981ef9640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:58.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.635+0000 7f7981ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:58.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.635+0000 7f7981ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:58.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.635+0000 7f7981ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:58.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.635+0000 7f7981ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:58.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.635+0000 7f79727fc640 1 -- 172.21.15.12:0/151326417 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f7974013070 con 0x7f797c0a4970 2024-09-11T01:36:58.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.635+0000 7f7981ef9640 10 monclient(hunting): handle_auth_done global_id 84137 payload 824 2024-09-11T01:36:58.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.635+0000 7f7981ef9640 1 -- 172.21.15.12:0/151326417 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f797c06ded0 con 0x7f797c13baa0 2024-09-11T01:36:58.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.635+0000 7f79727fc640 1 --2- 172.21.15.12:0/151326417 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f796803e210 0x7f79680406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.635+0000 7f79727fc640 1 -- 172.21.15.12:0/151326417 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(258..258 src has 186..258) ==== 4875+0+0 (secure 0 0 0) 0x7f7974052ec0 con 0x7f797c0a4970 2024-09-11T01:36:58.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.635+0000 7f79816f8640 1 --2- 172.21.15.12:0/151326417 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f796803e210 0x7f79680406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.635+0000 7f79816f8640 10 monclient: get_auth_request con 0x7f796803e210 auth_method 0 2024-09-11T01:36:58.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.637+0000 7f7981ef9640 1 -- 172.21.15.12:0/151326417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c13baa0 msgr2=0x7f797c1ccfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:58.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.637+0000 7f7981ef9640 1 --2- 172.21.15.12:0/151326417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c13baa0 0x7f797c1ccfe0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7978034000 tx=0x7f79780046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:58.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.638+0000 7f79727fc640 1 -- 172.21.15.12:0/151326417 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7978043070 con 0x7f797c13baa0 2024-09-11T01:36:58.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.638+0000 7f79727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:58.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.638+0000 7f79727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:58.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.638+0000 7f79727fc640 10 monclient: dump: 2024-09-11T01:36:58.665 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:58.665 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:58.665 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.665 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.665 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:58.665 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.665 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:58.665 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:58.666 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:58.666 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.638+0000 7f79727fc640 1 -- 172.21.15.12:0/151326417 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f797803c650 con 0x7f797c13baa0 2024-09-11T01:36:58.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.638+0000 7f79727fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:58.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.638+0000 7f79727fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:58.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.638+0000 7f79727fc640 1 -- 172.21.15.12:0/151326417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c13baa0 msgr2=0x7f797c1ccfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:58.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.638+0000 7f79727fc640 1 --2- 172.21.15.12:0/151326417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c13baa0 0x7f797c1ccfe0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7978034000 tx=0x7f79780046c0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.638+0000 7f79727fc640 1 -- 172.21.15.12:0/151326417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c13baa0 msgr2=0x7f797c1ccfe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:58.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.638+0000 7f79727fc640 1 --2- 172.21.15.12:0/151326417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c13baa0 0x7f797c1ccfe0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7978034000 tx=0x7f79780046c0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.638+0000 7f79816f8640 1 --2- 172.21.15.12:0/151326417 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f796803e210 0x7f79680406d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f797c13a900 tx=0x7f796c003d90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:58.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.644+0000 7f798415d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:36:58.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.644+0000 7f798415d640 1 --2- 172.21.15.12:0/151326417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c1cd520 0x7f797c1cd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.644+0000 7f798415d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:58.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.644+0000 7f798415d640 1 -- 172.21.15.12:0/151326417 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f797c1cde70 con 0x7f797c1cd520 2024-09-11T01:36:58.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.644+0000 7f7981ef9640 1 --2- 172.21.15.12:0/151326417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c1cd520 0x7f797c1cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.644+0000 7f7981ef9640 10 monclient: get_auth_request con 0x7f797c1cd520 auth_method 0 2024-09-11T01:36:58.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.644+0000 7f7981ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:58.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.644+0000 7f7981ef9640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:58.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.644+0000 7f7981ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:58.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.644+0000 7f7981ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:58.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.644+0000 7f7981ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:58.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.644+0000 7f7981ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:58.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.644+0000 7f7981ef9640 10 monclient(hunting): handle_auth_done global_id 84140 payload 824 2024-09-11T01:36:58.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.644+0000 7f7981ef9640 1 -- 172.21.15.12:0/151326417 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f797c069950 con 0x7f797c1cd520 2024-09-11T01:36:58.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.647+0000 7f7981ef9640 1 -- 172.21.15.12:0/151326417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c1cd520 msgr2=0x7f797c1cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:58.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.647+0000 7f7981ef9640 1 --2- 172.21.15.12:0/151326417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c1cd520 0x7f797c1cd930 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f797c0a4e30 tx=0x7f797803c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:58.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.647+0000 7f79727fc640 1 -- 172.21.15.12:0/151326417 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7978037040 con 0x7f797c1cd520 2024-09-11T01:36:58.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.647+0000 7f79727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:58.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.647+0000 7f79727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:58.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.647+0000 7f79727fc640 10 monclient: dump: 2024-09-11T01:36:58.670 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:58.670 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:58.670 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.670 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.670 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:58.671 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.671 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:58.671 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:58.671 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:58.671 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.647+0000 7f79727fc640 1 -- 172.21.15.12:0/151326417 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4540 (secure 0 0 0) 0x7f7978048b70 con 0x7f797c1cd520 2024-09-11T01:36:58.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.647+0000 7f79727fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:36:58.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.647+0000 7f79727fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:58.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.647+0000 7f79727fc640 1 -- 172.21.15.12:0/151326417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c1cd520 msgr2=0x7f797c1cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:58.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.647+0000 7f79727fc640 1 --2- 172.21.15.12:0/151326417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c1cd520 0x7f797c1cd930 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f797c0a4e30 tx=0x7f797803c000 comp rx=0 tx=0).stop 2024-09-11T01:36:58.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.647+0000 7f79727fc640 1 -- 172.21.15.12:0/151326417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c1cd520 msgr2=0x7f797c1cd930 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:58.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.647+0000 7f79727fc640 1 --2- 172.21.15.12:0/151326417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c1cd520 0x7f797c1cd930 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f797c0a4e30 tx=0x7f797803c000 comp rx=0 tx=0).stop 2024-09-11T01:36:58.672 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:36:58.672 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:36:58.672 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:36:58.672 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:36:58.672 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 80, 2024-09-11T01:36:58.672 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 12670, 2024-09-11T01:36:58.672 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:36:58.672 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:36:58.673 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:36:58.673 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:36:58.673 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:58.673 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 6, 2024-09-11T01:36:58.673 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:58.673 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:36:58.673 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:36:58.673 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:58.673 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:58.673 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:58.673 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:58.673 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:58.673 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:58.674 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:58.674 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:58.674 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:58.674 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:58.674 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:58.674 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:58.674 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:36:58.674 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:36:58.674 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:58.674 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:58.674 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:58.674 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:58.674 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:58.674 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:58.675 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:58.675 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:58.675 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:58.675 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:58.675 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:58.675 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:58.675 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.675 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:36:58.675 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:36:58.675 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:36:58.675 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:36:58.675 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:36:58.676 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:36:58.676 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:58.676 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:58.676 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:36:58.676 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:36:58.676 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:36:58.676 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:36:58.676 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:36:58.676 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:36:58.677 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:36:58.677 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:58.677 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:36:58.677 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:58.677 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:58.677 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:58.677 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:58.677 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:58.677 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:58.677 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:58.677 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:58.677 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:58.678 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:58.678 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:58.678 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:58.678 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:36:58.678 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.678 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:36:58.678 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.678 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:36:58.678 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:36:58.678 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:58.678 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:58.678 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.678 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:58.678 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:36:58.678 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:58.679 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.679 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.679 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:58.679 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:36:58.679 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:58.679 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:58.679 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:58.679 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.679 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:36:58.679 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:36:58.679 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:58.679 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:58.679 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:58.680 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.680 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.680 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:36:58.680 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:36:58.680 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:58.680 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:58.680 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.680 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:58.680 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:36:58.680 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:58.680 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.680 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.680 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:58.680 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:36:58.681 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:58.681 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:58.681 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:58.681 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.681 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:36:58.681 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:36:58.681 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:58.681 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:58.681 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:58.681 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.681 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.681 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:36:58.681 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:36:58.682 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:58.682 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:58.682 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.682 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:58.682 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:36:58.682 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:58.682 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.682 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.682 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:58.682 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:36:58.682 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:58.682 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:58.682 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:58.682 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.683 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:36:58.683 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:36:58.683 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:58.683 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:58.683 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:58.683 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:58.683 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:58.683 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.683 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:36:58.683 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:36:58.683 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.683 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:58.683 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:58.683 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:36:58.683 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:58.684 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:58.684 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:36:58.684 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.684 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:58.684 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:58.684 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:36:58.684 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:58.684 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:58.684 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.684 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:36:58.684 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:36:58.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.647+0000 7f798415d640 1 -- 172.21.15.12:0/151326417 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f796803e210 msgr2=0x7f79680406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:58.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.647+0000 7f798415d640 1 --2- 172.21.15.12:0/151326417 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f796803e210 0x7f79680406d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f797c13a900 tx=0x7f796c003d90 comp rx=0 tx=0).stop 2024-09-11T01:36:58.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.647+0000 7f798415d640 10 monclient: shutdown 2024-09-11T01:36:58.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.647+0000 7f798415d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:58.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.647+0000 7f798415d640 1 -- 172.21.15.12:0/151326417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c0a4970 msgr2=0x7f797c13ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:58.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.647+0000 7f798415d640 1 --2- 172.21.15.12:0/151326417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f797c0a4970 0x7f797c13ebd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f797400d5d0 tx=0x7f797400da90 comp rx=0 tx=0).stop 2024-09-11T01:36:58.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.647+0000 7f79826fa640 1 -- 172.21.15.12:0/151326417 reap_dead start 2024-09-11T01:36:58.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.647+0000 7f79826fa640 1 -- 172.21.15.12:0/151326417 reap_dead start 2024-09-11T01:36:58.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.647+0000 7f798415d640 1 -- 172.21.15.12:0/151326417 shutdown_connections 2024-09-11T01:36:58.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.647+0000 7f798415d640 1 -- 172.21.15.12:0/151326417 >> 172.21.15.12:0/151326417 conn(0x7f797c0a8600 msgr2=0x7f797c1382f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:58.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.647+0000 7f798415d640 1 -- 172.21.15.12:0/151326417 shutdown_connections 2024-09-11T01:36:58.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.647+0000 7f798415d640 1 -- 172.21.15.12:0/151326417 wait complete. 2024-09-11T01:36:58.687 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:36:58.687 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:36:58.687 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:36:58.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.787+0000 7ff3c3b94640 10 monclient: get_monmap_and_config 2024-09-11T01:36:58.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.787+0000 7ff3c3b94640 10 monclient: build_initial_monmap 2024-09-11T01:36:58.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.787+0000 7ff3c3b94640 10 monclient: monmap: 2024-09-11T01:36:58.789 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:58.789 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:58.789 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:58.788085+0000 2024-09-11T01:36:58.789 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:58.788085+0000 2024-09-11T01:36:58.789 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:58.789 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.790 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:58.790 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:58.790 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:58.790 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:58.790 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:58.790 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.787+0000 7ff3c3b94640 1 Processor -- start 2024-09-11T01:36:58.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c3b94640 1 -- start start 2024-09-11T01:36:58.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c3b94640 10 monclient: init 2024-09-11T01:36:58.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c3b94640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:58.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c3b94640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-11T01:36:58.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c3b94640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff3bc0a03a0 0x7ff3bc09f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c3b94640 10 monclient(hunting): picked mon.noname-a con 0x7ff3bc0a03a0 addr v2:172.21.15.12:3300/0 2024-09-11T01:36:58.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c3b94640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff3bc0a2530 addr v1:172.21.15.71:6789/0 2024-09-11T01:36:58.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c3b94640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3bc0a13a0 0x7ff3bc13baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c3b94640 10 monclient(hunting): picked mon.noname-b con 0x7ff3bc0a13a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:58.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c3b94640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3bc09e510 con 0x7ff3bc0a2530 2024-09-11T01:36:58.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c3b94640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:58.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c3b94640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff3bc0a29b0 con 0x7ff3bc0a03a0 2024-09-11T01:36:58.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c3b94640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:58.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c3b94640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff3bc0a0780 con 0x7ff3bc0a13a0 2024-09-11T01:36:58.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c3b94640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:58.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c3b94640 10 monclient(hunting): authenticate will time out at 1225.230658s 2024-09-11T01:36:58.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c1930640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff3bc0a03a0 0x7ff3bc09f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c2131640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3bc0a13a0 0x7ff3bc13baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c1930640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff3bc0a03a0 0x7ff3bc09f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36450/0 (socket says 172.21.15.12:36450) 2024-09-11T01:36:58.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c1930640 1 -- 172.21.15.12:0/238810234 learned_addr learned my addr 172.21.15.12:0/238810234 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:58.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c1930640 10 monclient(hunting): get_auth_request con 0x7ff3bc0a03a0 auth_method 0 2024-09-11T01:36:58.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c1930640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:58.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c1930640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:58.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c1930640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:58.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c2131640 10 monclient(hunting): get_auth_request con 0x7ff3bc0a13a0 auth_method 0 2024-09-11T01:36:58.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c2131640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:58.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c2131640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:58.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c2131640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:58.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c1930640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:58.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c1930640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:58.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c1930640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:58.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c2131640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:58.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c2131640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:58.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.788+0000 7ff3c2131640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:58.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.789+0000 7ff3c1930640 10 monclient(hunting): handle_auth_done global_id 54429 payload 274 2024-09-11T01:36:58.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.789+0000 7ff3c1930640 1 -- 172.21.15.12:0/238810234 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3bc0a13a0 msgr2=0x7ff3bc13baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:58.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.789+0000 7ff3c1930640 1 --2- 172.21.15.12:0/238810234 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3bc0a13a0 0x7ff3bc13baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.789+0000 7ff3c1930640 1 -- 172.21.15.12:0/238810234 >> v1:172.21.15.71:6789/0 conn(0x7ff3bc0a2530 legacy=0x7ff3bc0a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:58.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.789+0000 7ff3c1930640 10 monclient: _finish_hunting 0 2024-09-11T01:36:58.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.789+0000 7ff3c1930640 1 monclient: found mon.noname-a 2024-09-11T01:36:58.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.789+0000 7ff3c1930640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:58.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.789+0000 7ff3c1930640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:58.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.789+0000 7ff3c1930640 1 -- 172.21.15.12:0/238810234 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3bc09edb0 con 0x7ff3bc0a03a0 2024-09-11T01:36:58.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.789+0000 7ff3c2131640 1 --2- 172.21.15.12:0/238810234 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3bc0a13a0 0x7ff3bc13baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:36:58.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.791+0000 7ff3c1930640 1 -- 172.21.15.12:0/238810234 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3bc0a03a0 msgr2=0x7ff3bc09f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:58.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.791+0000 7ff3c1930640 1 --2- 172.21.15.12:0/238810234 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3bc0a03a0 0x7ff3bc09f1d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8009870 tx=0x7ff3b802a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db5b7f28ba882e59 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:58.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.792+0000 7ff3c092e640 1 -- 172.21.15.12:0/238810234 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8037070 con 0x7ff3bc0a03a0 2024-09-11T01:36:58.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.792+0000 7ff3c092e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:58.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.792+0000 7ff3c092e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:58.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.792+0000 7ff3c092e640 10 monclient: dump: 2024-09-11T01:36:58.803 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:58.803 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:58.803 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:58.804 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.792+0000 7ff3c092e640 10 monclient: _finish_auth 0 2024-09-11T01:36:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.792+0000 7ff3c092e640 10 monclient: _check_auth_tickets 2024-09-11T01:36:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.792+0000 7ff3c092e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.792+0000 7ff3c092e640 1 -- 172.21.15.12:0/238810234 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3b8032440 con 0x7ff3bc0a03a0 2024-09-11T01:36:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.792+0000 7ff3c3b94640 5 monclient: authenticate success, global_id 54429 2024-09-11T01:36:58.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.792+0000 7ff3c092e640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.792+0000 7ff3c3b94640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.792+0000 7ff3c092e640 1 -- 172.21.15.12:0/238810234 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8032810 con 0x7ff3bc0a03a0 2024-09-11T01:36:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.792+0000 7ff3c092e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.792+0000 7ff3c3b94640 4 set_mon_vals no callback set 2024-09-11T01:36:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.792+0000 7ff3c092e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.792+0000 7ff3c092e640 10 monclient: dump: 2024-09-11T01:36:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:58.805 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.806 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:58.806 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:58.806 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:58.806 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.792+0000 7ff3c3b94640 10 monclient: shutdown 2024-09-11T01:36:58.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.792+0000 7ff3c3b94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:58.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.792+0000 7ff3c3b94640 1 -- 172.21.15.12:0/238810234 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3bc0a03a0 msgr2=0x7ff3bc09f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:58.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.792+0000 7ff3c3b94640 1 --2- 172.21.15.12:0/238810234 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3bc0a03a0 0x7ff3bc09f1d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8009870 tx=0x7ff3b802a0a0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.792+0000 7ff3c3b94640 1 -- 172.21.15.12:0/238810234 shutdown_connections 2024-09-11T01:36:58.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.792+0000 7ff3c3b94640 1 --2- 172.21.15.12:0/238810234 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3bc0a13a0 0x7ff3bc13baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.792+0000 7ff3c3b94640 1 --2- 172.21.15.12:0/238810234 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3bc0a03a0 0x7ff3bc09f1d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.792+0000 7ff3c3b94640 1 -- 172.21.15.12:0/238810234 >> 172.21.15.12:0/238810234 conn(0x7ff3bc0a2f70 msgr2=0x7ff3bc0a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:58.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.792+0000 7ff3c3b94640 1 -- 172.21.15.12:0/238810234 shutdown_connections 2024-09-11T01:36:58.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.792+0000 7ff3c3b94640 1 -- 172.21.15.12:0/238810234 wait complete. 2024-09-11T01:36:58.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.793+0000 7ff3c3b94640 10 monclient: build_initial_monmap 2024-09-11T01:36:58.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.793+0000 7ff3c3b94640 10 monclient: monmap: 2024-09-11T01:36:58.807 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:58.807 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:58.807 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:58.807 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:58.807 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:58.807 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.807 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:58.808 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:58.808 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:58.808 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.793+0000 7ff3c3b94640 1 Processor -- start 2024-09-11T01:36:58.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.793+0000 7ff3c3b94640 1 -- start start 2024-09-11T01:36:58.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.793+0000 7ff3c3b94640 10 monclient: init 2024-09-11T01:36:58.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c3b94640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:58.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c3b94640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:36:58.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c3b94640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3bc0a03a0 0x7ff3bc1a8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c3b94640 10 monclient(hunting): picked mon.noname-a con 0x7ff3bc0a03a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:58.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c3b94640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3bc0a13a0 0x7ff3bc1c2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c3b94640 10 monclient(hunting): picked mon.noname-c con 0x7ff3bc0a13a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:58.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c3b94640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3bc0a2530 0x7ff3bc1c5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c3b94640 10 monclient(hunting): picked mon.noname-b con 0x7ff3bc0a2530 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:58.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c3b94640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:58.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c3b94640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff3bc09ca30 con 0x7ff3bc0a03a0 2024-09-11T01:36:58.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c3b94640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:58.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c3b94640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff3bc0a1a30 con 0x7ff3bc0a2530 2024-09-11T01:36:58.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c3b94640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:58.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c3b94640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff3bc09cbb0 con 0x7ff3bc0a13a0 2024-09-11T01:36:58.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c3b94640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:58.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c3b94640 10 monclient(hunting): authenticate will time out at 1225.236239s 2024-09-11T01:36:58.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c1930640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3bc0a03a0 0x7ff3bc1a8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c112f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3bc0a13a0 0x7ff3bc1c2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c1930640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3bc0a03a0 0x7ff3bc1a8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36456/0 (socket says 172.21.15.12:36456) 2024-09-11T01:36:58.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c1930640 1 -- 172.21.15.12:0/2778480132 learned_addr learned my addr 172.21.15.12:0/2778480132 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:58.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c1930640 10 monclient(hunting): get_auth_request con 0x7ff3bc0a03a0 auth_method 0 2024-09-11T01:36:58.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c1930640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:58.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c1930640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:58.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c1930640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:58.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c112f640 10 monclient(hunting): get_auth_request con 0x7ff3bc0a13a0 auth_method 0 2024-09-11T01:36:58.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c112f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:58.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c112f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:58.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c112f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:58.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c2131640 1 --2- 172.21.15.12:0/2778480132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3bc0a2530 0x7ff3bc1c5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c1930640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:58.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c1930640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:58.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c1930640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:58.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c2131640 10 monclient(hunting): get_auth_request con 0x7ff3bc0a2530 auth_method 0 2024-09-11T01:36:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c2131640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c2131640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c2131640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c112f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c112f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c112f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c2131640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c2131640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c2131640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c1930640 10 monclient(hunting): handle_auth_done global_id 54432 payload 824 2024-09-11T01:36:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c1930640 1 -- 172.21.15.12:0/2778480132 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3bc0a13a0 msgr2=0x7ff3bc1c2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:58.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c1930640 1 --2- 172.21.15.12:0/2778480132 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3bc0a13a0 0x7ff3bc1c2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c1930640 1 -- 172.21.15.12:0/2778480132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3bc0a2530 msgr2=0x7ff3bc1c5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c1930640 1 --2- 172.21.15.12:0/2778480132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3bc0a2530 0x7ff3bc1c5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c1930640 10 monclient: _finish_hunting 0 2024-09-11T01:36:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c1930640 1 monclient: found mon.noname-a 2024-09-11T01:36:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c1930640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c1930640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:36:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c1930640 1 -- 172.21.15.12:0/2778480132 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3bc1a9810 con 0x7ff3bc0a03a0 2024-09-11T01:36:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.794+0000 7ff3c112f640 1 --2- 172.21.15.12:0/2778480132 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3bc0a13a0 0x7ff3bc1c2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:36:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.796+0000 7ff3c1930640 1 -- 172.21.15.12:0/2778480132 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3bc0a03a0 msgr2=0x7ff3bc1a8e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.796+0000 7ff3c1930640 1 --2- 172.21.15.12:0/2778480132 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3bc0a03a0 0x7ff3bc1a8e80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8009870 tx=0x7ff3b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.797+0000 7ff3b27fc640 1 -- 172.21.15.12:0/2778480132 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8037040 con 0x7ff3bc0a03a0 2024-09-11T01:36:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.797+0000 7ff3b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:58.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.797+0000 7ff3b27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:36:58.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.797+0000 7ff3b27fc640 10 monclient: dump: 2024-09-11T01:36:58.813 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:58.813 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:58.813 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.813 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.813 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:58.813 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.813 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:58.813 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:58.813 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:58.814 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.797+0000 7ff3b27fc640 10 monclient: _finish_auth 0 2024-09-11T01:36:58.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.797+0000 7ff3b27fc640 10 monclient: _check_auth_tickets 2024-09-11T01:36:58.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.797+0000 7ff3b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:58.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.797+0000 7ff3b27fc640 1 -- 172.21.15.12:0/2778480132 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3b8042070 con 0x7ff3bc0a03a0 2024-09-11T01:36:58.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.797+0000 7ff3c3b94640 5 monclient: authenticate success, global_id 54432 2024-09-11T01:36:58.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.797+0000 7ff3c3b94640 10 monclient: _renew_subs 2024-09-11T01:36:58.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.797+0000 7ff3c3b94640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:58.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.797+0000 7ff3c3b94640 1 -- 172.21.15.12:0/2778480132 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3bc1c95e0 con 0x7ff3bc0a03a0 2024-09-11T01:36:58.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.797+0000 7ff3b27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:58.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.797+0000 7ff3b27fc640 1 -- 172.21.15.12:0/2778480132 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b803b840 con 0x7ff3bc0a03a0 2024-09-11T01:36:58.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.797+0000 7ff3b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:58.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.797+0000 7ff3b27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:36:58.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.797+0000 7ff3b27fc640 10 monclient: dump: 2024-09-11T01:36:58.815 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:58.815 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:58.815 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.815 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.815 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:58.815 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.815 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:58.815 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:58.815 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:58.815 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.797+0000 7ff3c3b94640 10 monclient: _renew_subs 2024-09-11T01:36:58.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.797+0000 7ff3c3b94640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:36:58.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.797+0000 7ff3b3fff640 4 set_mon_vals no callback set 2024-09-11T01:36:58.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.797+0000 7ff3c3b94640 1 -- 172.21.15.12:0/2778480132 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3bc1aa680 con 0x7ff3bc0a03a0 2024-09-11T01:36:58.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.797+0000 7ff3c3b94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:58.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.797+0000 7ff3c3b94640 1 --2- 172.21.15.12:0/2778480132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3bc1c3160 0x7ff3bc02d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.797+0000 7ff3c3b94640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:58.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.797+0000 7ff3c3b94640 1 -- 172.21.15.12:0/2778480132 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff3bc09d480 con 0x7ff3bc1c3160 2024-09-11T01:36:58.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.798+0000 7ff3c1930640 1 --2- 172.21.15.12:0/2778480132 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3bc0a03a0 0x7ff3bc1a8e80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8009870 tx=0x7ff3b80046c0 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7ff3b8072f50 osd_map(258..258 src has 186..258) 2024-09-11T01:36:58.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.798+0000 7ff3b27fc640 1 -- 172.21.15.12:0/2778480132 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff3b803a030 con 0x7ff3bc0a03a0 2024-09-11T01:36:58.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.798+0000 7ff3c112f640 1 --2- 172.21.15.12:0/2778480132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3bc1c3160 0x7ff3bc02d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.798+0000 7ff3c112f640 10 monclient: get_auth_request con 0x7ff3bc1c3160 auth_method 0 2024-09-11T01:36:58.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.798+0000 7ff3c112f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:58.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.798+0000 7ff3c112f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:58.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.798+0000 7ff3c112f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:58.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.798+0000 7ff3c112f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:58.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.798+0000 7ff3c112f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:58.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.798+0000 7ff3c112f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:58.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.798+0000 7ff3b27fc640 1 --2- 172.21.15.12:0/2778480132 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff3a003e260 0x7ff3a0040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.798+0000 7ff3c112f640 10 monclient(hunting): handle_auth_done global_id 74227 payload 824 2024-09-11T01:36:58.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.798+0000 7ff3c112f640 1 -- 172.21.15.12:0/2778480132 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff3bc09d010 con 0x7ff3bc1c3160 2024-09-11T01:36:58.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.798+0000 7ff3c2131640 1 --2- 172.21.15.12:0/2778480132 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff3a003e260 0x7ff3a0040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.798+0000 7ff3c2131640 10 monclient: get_auth_request con 0x7ff3a003e260 auth_method 0 2024-09-11T01:36:58.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.800+0000 7ff3c112f640 1 -- 172.21.15.12:0/2778480132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3bc1c3160 msgr2=0x7ff3bc02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:58.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.800+0000 7ff3c112f640 1 --2- 172.21.15.12:0/2778480132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3bc1c3160 0x7ff3bc02d440 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc09dd20 tx=0x7ff3ac00ff70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:58.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.800+0000 7ff3b27fc640 1 -- 172.21.15.12:0/2778480132 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac00dd60 con 0x7ff3bc1c3160 2024-09-11T01:36:58.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.800+0000 7ff3b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:58.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.800+0000 7ff3b27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:36:58.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.800+0000 7ff3b27fc640 10 monclient: dump: 2024-09-11T01:36:58.819 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:58.819 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:58.819 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.819 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.819 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:58.820 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.820 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:58.820 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:58.820 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:58.820 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.800+0000 7ff3c2131640 1 --2- 172.21.15.12:0/2778480132 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff3a003e260 0x7ff3a0040720 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4011420 tx=0x7ff3b400b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:58.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.801+0000 7ff3b27fc640 1 -- 172.21.15.12:0/2778480132 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff3ac00dd60 con 0x7ff3bc1c3160 2024-09-11T01:36:58.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.801+0000 7ff3b27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:58.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.801+0000 7ff3b27fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:58.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.801+0000 7ff3b27fc640 1 -- 172.21.15.12:0/2778480132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3bc1c3160 msgr2=0x7ff3bc02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:58.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.801+0000 7ff3b27fc640 1 --2- 172.21.15.12:0/2778480132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3bc1c3160 0x7ff3bc02d440 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc09dd20 tx=0x7ff3ac00ff70 comp rx=0 tx=0).stop 2024-09-11T01:36:58.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.801+0000 7ff3b27fc640 1 -- 172.21.15.12:0/2778480132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3bc1c3160 msgr2=0x7ff3bc02d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:58.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.801+0000 7ff3b27fc640 1 --2- 172.21.15.12:0/2778480132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3bc1c3160 0x7ff3bc02d440 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc09dd20 tx=0x7ff3ac00ff70 comp rx=0 tx=0).stop 2024-09-11T01:36:58.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.807+0000 7ff3c3b94640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:36:58.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.807+0000 7ff3c3b94640 1 --2- 172.21.15.12:0/2778480132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3bc02d980 0x7ff3bc02dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:58.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.807+0000 7ff3c3b94640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:58.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.807+0000 7ff3c3b94640 1 -- 172.21.15.12:0/2778480132 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff3bc1a2d00 con 0x7ff3bc02d980 2024-09-11T01:36:58.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.807+0000 7ff3c112f640 1 --2- 172.21.15.12:0/2778480132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3bc02d980 0x7ff3bc02dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:58.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.807+0000 7ff3c112f640 10 monclient: get_auth_request con 0x7ff3bc02d980 auth_method 0 2024-09-11T01:36:58.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.807+0000 7ff3c112f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:58.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.807+0000 7ff3c112f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:58.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.807+0000 7ff3c112f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:58.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.807+0000 7ff3c112f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:58.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.807+0000 7ff3c112f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:58.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.807+0000 7ff3c112f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:58.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.807+0000 7ff3c112f640 10 monclient(hunting): handle_auth_done global_id 74230 payload 824 2024-09-11T01:36:58.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.807+0000 7ff3c112f640 1 -- 172.21.15.12:0/2778480132 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff3bc1a9e40 con 0x7ff3bc02d980 2024-09-11T01:36:58.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.810+0000 7ff3c112f640 1 -- 172.21.15.12:0/2778480132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3bc02d980 msgr2=0x7ff3bc02dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:58.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.810+0000 7ff3c112f640 1 --2- 172.21.15.12:0/2778480132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3bc02d980 0x7ff3bc02dd60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac018000 tx=0x7ff3ac003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:58.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.810+0000 7ff3b27fc640 1 -- 172.21.15.12:0/2778480132 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac007020 con 0x7ff3bc02d980 2024-09-11T01:36:58.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.810+0000 7ff3b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:58.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.810+0000 7ff3b27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:36:58.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.810+0000 7ff3b27fc640 10 monclient: dump: 2024-09-11T01:36:58.825 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:58.825 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:58.826 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.826 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:58.826 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:58.826 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:58.826 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:58.826 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:58.826 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:58.826 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:58.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.810+0000 7ff3b27fc640 1 -- 172.21.15.12:0/2778480132 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4707 (secure 0 0 0) 0x7ff3ac007020 con 0x7ff3bc02d980 2024-09-11T01:36:58.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.810+0000 7ff3b27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:36:58.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.810+0000 7ff3b27fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:58.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.810+0000 7ff3b27fc640 1 -- 172.21.15.12:0/2778480132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3bc02d980 msgr2=0x7ff3bc02dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:58.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.810+0000 7ff3b27fc640 1 --2- 172.21.15.12:0/2778480132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3bc02d980 0x7ff3bc02dd60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac018000 tx=0x7ff3ac003760 comp rx=0 tx=0).stop 2024-09-11T01:36:58.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.810+0000 7ff3b27fc640 1 -- 172.21.15.12:0/2778480132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3bc02d980 msgr2=0x7ff3bc02dd60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:36:58.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.810+0000 7ff3b27fc640 1 --2- 172.21.15.12:0/2778480132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3bc02d980 0x7ff3bc02dd60 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac018000 tx=0x7ff3ac003760 comp rx=0 tx=0).stop 2024-09-11T01:36:58.827 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:36:58.827 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:36:58.827 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:36:58.827 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:36:58.827 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 80, 2024-09-11T01:36:58.827 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 70589, 2024-09-11T01:36:58.827 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:36:58.828 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:36:58.828 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:36:58.828 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:36:58.828 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:58.828 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 7, 2024-09-11T01:36:58.828 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:58.828 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:36:58.828 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:36:58.828 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:58.828 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:58.828 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:58.828 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:58.828 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:58.829 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:58.829 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:58.829 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:58.829 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:58.829 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:58.829 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:58.829 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:58.829 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:36:58.829 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:36:58.830 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:58.830 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:58.830 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:58.830 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:58.830 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:58.830 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:58.830 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:58.830 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:58.830 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:58.830 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:58.830 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:58.830 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:58.830 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.831 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:36:58.831 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:36:58.831 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:36:58.831 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:36:58.831 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:36:58.831 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:36:58.831 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:58.831 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:36:58.831 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:36:58.831 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:36:58.831 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:36:58.832 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:36:58.832 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:36:58.832 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:36:58.832 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:36:58.832 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:36:58.832 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:36:58.832 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:36:58.832 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:36:58.832 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:36:58.832 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:36:58.833 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:36:58.833 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:36:58.833 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:36:58.833 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:36:58.833 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:36:58.833 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:36:58.833 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:36:58.833 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:58.833 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:36:58.833 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.833 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:36:58.833 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.834 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:36:58.834 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:36:58.834 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:58.834 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:58.834 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.834 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:58.834 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:36:58.834 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:58.834 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.834 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.834 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:58.834 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:36:58.834 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:58.834 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:58.835 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:58.835 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.835 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:36:58.835 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:36:58.835 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:58.835 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:58.835 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:58.835 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.835 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.835 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:36:58.835 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:36:58.835 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:58.835 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:58.836 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.836 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:58.836 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:36:58.836 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:58.836 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.836 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.837 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:58.837 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:36:58.837 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:58.837 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:58.837 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:58.837 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.838 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:36:58.838 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:36:58.838 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:58.838 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:58.838 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:58.838 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.838 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.838 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:36:58.838 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:36:58.839 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:36:58.839 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:36:58.839 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.839 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:36:58.839 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:36:58.839 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:58.839 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.839 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.839 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:36:58.839 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:36:58.839 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:36:58.839 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:58.839 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:58.839 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.839 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:36:58.840 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:36:58.840 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:36:58.840 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:36:58.840 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:36:58.840 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:58.840 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:58.840 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.840 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:36:58.840 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:36:58.840 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.840 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:58.840 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:58.841 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:36:58.841 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:58.841 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:58.841 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:36:58.841 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.841 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:58.841 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:58.842 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:36:58.842 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:58.842 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:36:58.842 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:36:58.842 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:36:58.842 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:36:58.843 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:36:58.843 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:36:58.843 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:36:58.843 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:36:58.843 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:36:58.843 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:36:58.843 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:36:58.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.810+0000 7ff3c3b94640 1 -- 172.21.15.12:0/2778480132 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff3a003e260 msgr2=0x7ff3a0040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:58.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.810+0000 7ff3c3b94640 1 --2- 172.21.15.12:0/2778480132 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff3a003e260 0x7ff3a0040720 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4011420 tx=0x7ff3b400b040 comp rx=0 tx=0).stop 2024-09-11T01:36:58.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.810+0000 7ff3c3b94640 10 monclient: shutdown 2024-09-11T01:36:58.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.810+0000 7ff3c3b94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:58.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.810+0000 7ff3c3b94640 1 -- 172.21.15.12:0/2778480132 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3bc0a03a0 msgr2=0x7ff3bc1a8e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:58.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.810+0000 7ff3c2131640 1 -- 172.21.15.12:0/2778480132 reap_dead start 2024-09-11T01:36:58.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.810+0000 7ff3c3b94640 1 --2- 172.21.15.12:0/2778480132 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3bc0a03a0 0x7ff3bc1a8e80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8009870 tx=0x7ff3b80046c0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.811+0000 7ff3b27fc640 1 -- 172.21.15.12:0/2778480132 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(258..258 src has 186..258) ==== 4875+0+0 (unknown 0 0 0) 0x7ff3b8072f50 con 0x7ff3bc0a03a0 2024-09-11T01:36:58.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.811+0000 7ff3c3b94640 1 -- 172.21.15.12:0/2778480132 shutdown_connections 2024-09-11T01:36:58.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.811+0000 7ff3c3b94640 1 --2- 172.21.15.12:0/2778480132 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3bc0a03a0 0x7ff3bc1a8e80 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:58.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.811+0000 7ff3c3b94640 1 -- 172.21.15.12:0/2778480132 >> 172.21.15.12:0/2778480132 conn(0x7ff3bc0a2f70 msgr2=0x7ff3bc0a5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:58.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.811+0000 7ff3c3b94640 1 -- 172.21.15.12:0/2778480132 shutdown_connections 2024-09-11T01:36:58.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.811+0000 7ff3c3b94640 1 -- 172.21.15.12:0/2778480132 wait complete. 2024-09-11T01:36:58.846 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:36:58.846 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-11T01:36:58.846 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T01:36:58.846 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-11T01:36:58.846 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-11T01:36:58.847 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T01:36:58.847 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:36:58.859 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T01:36:58.859 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:36:58.859 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:36:58.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.998+0000 7f5f8c0b0640 10 monclient: get_monmap_and_config 2024-09-11T01:36:59.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.998+0000 7f5f8c0b0640 10 monclient: build_initial_monmap 2024-09-11T01:36:59.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.998+0000 7f5f8c0b0640 10 monclient: monmap: 2024-09-11T01:36:59.000 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:59.000 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:59.000 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:36:58.999133+0000 2024-09-11T01:36:59.000 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:36:58.999133+0000 2024-09-11T01:36:59.000 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:59.000 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:59.000 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:36:59.000 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:36:59.001 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:36:59.001 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:36:59.001 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:36:59.001 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:59.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.998+0000 7f5f8c0b0640 1 Processor -- start 2024-09-11T01:36:59.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f8c0b0640 1 -- start start 2024-09-11T01:36:59.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f8c0b0640 10 monclient: init 2024-09-11T01:36:59.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f8c0b0640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:59.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f8c0b0640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-11T01:36:59.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f8c0b0640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5f840a2e90 0x7f5f8406dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:59.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f8c0b0640 10 monclient(hunting): picked mon.noname-a con 0x7f5f840a2e90 addr v2:172.21.15.12:3300/0 2024-09-11T01:36:59.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f8c0b0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f840a3270 addr v1:172.21.15.71:6789/0 2024-09-11T01:36:59.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f8c0b0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f8413d490 0x7f5f8413f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:59.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f8c0b0640 10 monclient(hunting): picked mon.noname-b con 0x7f5f8413d490 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:59.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f8c0b0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f840a46c0 con 0x7f5f840a3270 2024-09-11T01:36:59.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f8c0b0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:59.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f8c0b0640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f5f840a4900 con 0x7f5f840a2e90 2024-09-11T01:36:59.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f8c0b0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:59.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f8c0b0640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5f8406ee50 con 0x7f5f8413d490 2024-09-11T01:36:59.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f8c0b0640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:59.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f8c0b0640 10 monclient(hunting): authenticate will time out at 1225.441708s 2024-09-11T01:36:59.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f89e4c640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7f5f840a2e90 msgr2=0x7f5f8406dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:36:59.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f89e4c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5f840a2e90 0x7f5f8406dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:59.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f8a64d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f8413d490 0x7f5f8413f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:59.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f8a64d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f8413d490 0x7f5f8413f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:34684/0 (socket says 172.21.15.12:34684) 2024-09-11T01:36:59.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f8a64d640 1 -- 172.21.15.12:0/303412223 learned_addr learned my addr 172.21.15.12:0/303412223 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:59.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f8a64d640 10 monclient(hunting): get_auth_request con 0x7f5f8413d490 auth_method 0 2024-09-11T01:36:59.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f8a64d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:59.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f8a64d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:59.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f8a64d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:59.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f8a64d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:59.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f8a64d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:59.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:58.999+0000 7f5f8a64d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:59.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.000+0000 7f5f8a64d640 10 monclient(hunting): handle_auth_done global_id 84149 payload 274 2024-09-11T01:36:59.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.000+0000 7f5f8a64d640 1 -- 172.21.15.12:0/303412223 >> v2:172.21.15.12:3300/0 conn(0x7f5f840a2e90 msgr2=0x7f5f8406dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:36:59.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.000+0000 7f5f8a64d640 1 --2- 172.21.15.12:0/303412223 >> v2:172.21.15.12:3300/0 conn(0x7f5f840a2e90 0x7f5f8406dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:59.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.000+0000 7f5f8a64d640 1 -- 172.21.15.12:0/303412223 >> v1:172.21.15.71:6789/0 conn(0x7f5f840a3270 legacy=0x7f5f840a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:36:59.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.000+0000 7f5f8a64d640 10 monclient: _finish_hunting 0 2024-09-11T01:36:59.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.000+0000 7f5f8a64d640 1 monclient: found mon.noname-b 2024-09-11T01:36:59.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.000+0000 7f5f8a64d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:59.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.000+0000 7f5f8a64d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:36:59.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.000+0000 7f5f8a64d640 1 -- 172.21.15.12:0/303412223 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f8406eb70 con 0x7f5f8413d490 2024-09-11T01:36:59.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.002+0000 7f5f8a64d640 1 -- 172.21.15.12:0/303412223 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f8413d490 msgr2=0x7f5f8413f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:59.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.002+0000 7f5f8a64d640 1 --2- 172.21.15.12:0/303412223 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f8413d490 0x7f5f8413f880 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5f84071140 tx=0x7f5f7c02a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3f9c2e8d4f35ebf server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:59.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f88e4a640 1 -- 172.21.15.12:0/303412223 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c036070 con 0x7f5f8413d490 2024-09-11T01:36:59.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f88e4a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:59.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f88e4a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:36:59.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f88e4a640 10 monclient: dump: 2024-09-11T01:36:59.011 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:59.012 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:59.012 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:59.012 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:59.012 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:59.012 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:59.012 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:59.012 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:59.012 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:59.012 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:59.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f88e4a640 10 monclient: _finish_auth 0 2024-09-11T01:36:59.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f88e4a640 10 monclient: _check_auth_tickets 2024-09-11T01:36:59.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f88e4a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:59.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f88e4a640 1 -- 172.21.15.12:0/303412223 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f7c007800 con 0x7f5f8413d490 2024-09-11T01:36:59.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f8c0b0640 5 monclient: authenticate success, global_id 84149 2024-09-11T01:36:59.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f8c0b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:36:59.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f88e4a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:59.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f8c0b0640 10 monclient: get_monmap_and_config success 2024-09-11T01:36:59.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f8c0b0640 4 set_mon_vals no callback set 2024-09-11T01:36:59.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f88e4a640 1 -- 172.21.15.12:0/303412223 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c007bd0 con 0x7f5f8413d490 2024-09-11T01:36:59.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f88e4a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:59.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f88e4a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:59.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f88e4a640 10 monclient: dump: 2024-09-11T01:36:59.013 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:59.013 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:59.013 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:59.013 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:59.014 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:59.014 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:59.014 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:59.014 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:59.014 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:59.014 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:59.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f8c0b0640 10 monclient: shutdown 2024-09-11T01:36:59.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f8c0b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:59.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f8c0b0640 1 -- 172.21.15.12:0/303412223 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f8413d490 msgr2=0x7f5f8413f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:59.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f8c0b0640 1 --2- 172.21.15.12:0/303412223 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f8413d490 0x7f5f8413f880 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5f84071140 tx=0x7f5f7c02a560 comp rx=0 tx=0).stop 2024-09-11T01:36:59.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f8c0b0640 1 -- 172.21.15.12:0/303412223 shutdown_connections 2024-09-11T01:36:59.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f8c0b0640 1 --2- 172.21.15.12:0/303412223 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f8413d490 0x7f5f8413f880 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:59.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f8c0b0640 1 --2- 172.21.15.12:0/303412223 >> v2:172.21.15.12:3300/0 conn(0x7f5f840a2e90 0x7f5f8406dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:59.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f8c0b0640 1 -- 172.21.15.12:0/303412223 >> 172.21.15.12:0/303412223 conn(0x7f5f840a8490 msgr2=0x7f5f840a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:59.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f8c0b0640 1 -- 172.21.15.12:0/303412223 shutdown_connections 2024-09-11T01:36:59.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.003+0000 7f5f8c0b0640 1 -- 172.21.15.12:0/303412223 wait complete. 2024-09-11T01:36:59.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.004+0000 7f5f8c0b0640 10 monclient: build_initial_monmap 2024-09-11T01:36:59.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.004+0000 7f5f8c0b0640 10 monclient: monmap: 2024-09-11T01:36:59.015 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:36:59.015 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:36:59.015 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:36:59.015 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:36:59.015 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:36:59.015 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:59.015 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:36:59.015 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:36:59.015 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:36:59.016 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:59.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.004+0000 7f5f8c0b0640 1 Processor -- start 2024-09-11T01:36:59.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.004+0000 7f5f8c0b0640 1 -- start start 2024-09-11T01:36:59.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.004+0000 7f5f8c0b0640 10 monclient: init 2024-09-11T01:36:59.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.004+0000 7f5f8c0b0640 10 monclient: _reopen_session rank -1 2024-09-11T01:36:59.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8c0b0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:36:59.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8c0b0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f840a2e90 0x7f5f8409dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:59.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8c0b0640 10 monclient(hunting): picked mon.noname-a con 0x7f5f840a2e90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:36:59.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8c0b0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f840a3270 0x7f5f84138370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:59.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8c0b0640 10 monclient(hunting): picked mon.noname-c con 0x7f5f840a3270 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:36:59.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8c0b0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5f8413d490 0x7f5f8409c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:59.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8c0b0640 10 monclient(hunting): picked mon.noname-b con 0x7f5f8413d490 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:36:59.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8c0b0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:59.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8c0b0640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5f8406fd20 con 0x7f5f840a2e90 2024-09-11T01:36:59.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8c0b0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:59.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8c0b0640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5f8406fba0 con 0x7f5f8413d490 2024-09-11T01:36:59.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8c0b0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:36:59.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8c0b0640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5f8413fdc0 con 0x7f5f840a3270 2024-09-11T01:36:59.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8c0b0640 10 monclient(hunting): _renew_subs 2024-09-11T01:36:59.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8c0b0640 10 monclient(hunting): authenticate will time out at 1225.447076s 2024-09-11T01:36:59.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f89e4c640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f840a2e90 msgr2=0x7f5f8409dbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:36:59.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f89e4c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f840a2e90 0x7f5f8409dbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:36:59.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8964b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f840a3270 0x7f5f84138370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:59.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8964b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f840a3270 0x7f5f84138370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:34700/0 (socket says 172.21.15.12:34700) 2024-09-11T01:36:59.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8964b640 1 -- 172.21.15.12:0/1636857028 learned_addr learned my addr 172.21.15.12:0/1636857028 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:36:59.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8964b640 10 monclient(hunting): get_auth_request con 0x7f5f840a3270 auth_method 0 2024-09-11T01:36:59.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8964b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:36:59.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8964b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:36:59.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8964b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:36:59.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8964b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:36:59.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8964b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:36:59.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8964b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:36:59.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8964b640 10 monclient(hunting): handle_auth_done global_id 84152 payload 824 2024-09-11T01:36:59.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8964b640 1 -- 172.21.15.12:0/1636857028 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5f8413d490 msgr2=0x7f5f8409c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:59.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8964b640 1 --2- 172.21.15.12:0/1636857028 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5f8413d490 0x7f5f8409c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:59.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8964b640 1 -- 172.21.15.12:0/1636857028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f840a2e90 msgr2=0x7f5f8409dbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:36:59.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8964b640 1 --2- 172.21.15.12:0/1636857028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f840a2e90 0x7f5f8409dbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:59.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8964b640 10 monclient: _finish_hunting 0 2024-09-11T01:36:59.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8964b640 1 monclient: found mon.noname-c 2024-09-11T01:36:59.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8964b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:36:59.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8964b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:36:59.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.005+0000 7f5f8964b640 1 -- 172.21.15.12:0/1636857028 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f8409f0d0 con 0x7f5f840a3270 2024-09-11T01:36:59.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.007+0000 7f5f8964b640 1 -- 172.21.15.12:0/1636857028 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f840a3270 msgr2=0x7f5f84138370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:36:59.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.007+0000 7f5f8964b640 1 --2- 172.21.15.12:0/1636857028 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f840a3270 0x7f5f84138370 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5f7400c410 tx=0x7f5f7400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:59.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.007+0000 7f5f7a7fc640 1 -- 172.21.15.12:0/1636857028 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f74017070 con 0x7f5f840a3270 2024-09-11T01:36:59.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.007+0000 7f5f7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:59.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.007+0000 7f5f7a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:36:59.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.007+0000 7f5f7a7fc640 10 monclient: dump: 2024-09-11T01:36:59.020 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:59.020 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:59.020 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:59.020 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:59.020 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:59.020 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:59.020 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:59.020 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:59.020 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:59.020 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:59.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.007+0000 7f5f7a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:36:59.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.007+0000 7f5f7a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:36:59.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.007+0000 7f5f7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:36:59.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.007+0000 7f5f7a7fc640 1 -- 172.21.15.12:0/1636857028 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f74003810 con 0x7f5f840a3270 2024-09-11T01:36:59.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.007+0000 7f5f8c0b0640 5 monclient: authenticate success, global_id 84152 2024-09-11T01:36:59.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.007+0000 7f5f8c0b0640 10 monclient: _renew_subs 2024-09-11T01:36:59.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.007+0000 7f5f8c0b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:36:59.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.007+0000 7f5f8c0b0640 1 -- 172.21.15.12:0/1636857028 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f84139010 con 0x7f5f840a3270 2024-09-11T01:36:59.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.007+0000 7f5f7a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:36:59.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.007+0000 7f5f7a7fc640 1 -- 172.21.15.12:0/1636857028 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f74003c00 con 0x7f5f840a3270 2024-09-11T01:36:59.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.007+0000 7f5f7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:36:59.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.007+0000 7f5f7a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:36:59.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.007+0000 7f5f7a7fc640 10 monclient: dump: 2024-09-11T01:36:59.021 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:36:59.021 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:36:59.021 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:59.021 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:36:59.022 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:36:59.022 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:36:59.022 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:36:59.022 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:36:59.022 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:36:59.022 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:36:59.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.007+0000 7f5f8c0b0640 10 monclient: _renew_subs 2024-09-11T01:36:59.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.007+0000 7f5f7bfff640 4 set_mon_vals no callback set 2024-09-11T01:36:59.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.007+0000 7f5f8c0b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:36:59.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.007+0000 7f5f8c0b0640 1 -- 172.21.15.12:0/1636857028 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f841cddb0 con 0x7f5f840a3270 2024-09-11T01:36:59.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.008+0000 7f5f7a7fc640 1 -- 172.21.15.12:0/1636857028 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f5f7401b3b0 con 0x7f5f840a3270 2024-09-11T01:36:59.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.008+0000 7f5f8c0b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:59.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.008+0000 7f5f8c0b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:59.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.008+0000 7f5f8c0b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:36:59.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.008+0000 7f5f8c0b0640 1 -- 172.21.15.12:0/1636857028 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f840a3650 con 0x7f5f840a3270 2024-09-11T01:36:59.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.009+0000 7f5f7a7fc640 1 --2- 172.21.15.12:0/1636857028 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5f6803e1c0 0x7f5f68040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:36:59.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.009+0000 7f5f7a7fc640 1 -- 172.21.15.12:0/1636857028 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(258..258 src has 186..258) ==== 4875+0+0 (secure 0 0 0) 0x7f5f740525c0 con 0x7f5f840a3270 2024-09-11T01:36:59.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.009+0000 7f5f89e4c640 1 --2- 172.21.15.12:0/1636857028 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5f6803e1c0 0x7f5f68040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:36:59.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.009+0000 7f5f89e4c640 10 monclient: get_auth_request con 0x7f5f6803e1c0 auth_method 0 2024-09-11T01:36:59.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.011+0000 7f5f89e4c640 1 --2- 172.21.15.12:0/1636857028 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5f6803e1c0 0x7f5f68040680 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5f80003980 tx=0x7f5f80006d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:36:59.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.013+0000 7f5f7a7fc640 1 -- 172.21.15.12:0/1636857028 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5f840a3650 con 0x7f5f840a3270 2024-09-11T01:36:59.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.013+0000 7f5f7a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:36:59.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.013+0000 7f5f7a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:36:59.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.270+0000 7f5f8c0b0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:36:59.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.270+0000 7f5f8c0b0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:36:59.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.270+0000 7f5f8c0b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:36:59.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.270+0000 7f5f8c0b0640 1 -- 172.21.15.12:0/1636857028 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5f8406dc60 con 0x7f5f840a3270 2024-09-11T01:36:59.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.271+0000 7f5f7a7fc640 1 -- 172.21.15.12:0/1636857028 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f5f74050f60 con 0x7f5f840a3270 2024-09-11T01:36:59.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.271+0000 7f5f7a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:36:59.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.271+0000 7f5f7a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:36:59.273 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:36:59.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.273+0000 7f5f8c0b0640 1 -- 172.21.15.12:0/1636857028 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5f6803e1c0 msgr2=0x7f5f68040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:59.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.273+0000 7f5f8c0b0640 1 --2- 172.21.15.12:0/1636857028 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5f6803e1c0 0x7f5f68040680 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5f80003980 tx=0x7f5f80006d30 comp rx=0 tx=0).stop 2024-09-11T01:36:59.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.273+0000 7f5f8c0b0640 10 monclient: shutdown 2024-09-11T01:36:59.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.273+0000 7f5f8c0b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:36:59.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.273+0000 7f5f8c0b0640 1 -- 172.21.15.12:0/1636857028 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f840a3270 msgr2=0x7f5f84138370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:36:59.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.273+0000 7f5f8c0b0640 1 --2- 172.21.15.12:0/1636857028 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f840a3270 0x7f5f84138370 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5f7400c410 tx=0x7f5f7400c8d0 comp rx=0 tx=0).stop 2024-09-11T01:36:59.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.273+0000 7f5f8c0b0640 1 -- 172.21.15.12:0/1636857028 shutdown_connections 2024-09-11T01:36:59.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.273+0000 7f5f8c0b0640 1 --2- 172.21.15.12:0/1636857028 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5f8413d490 0x7f5f8409c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:59.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.273+0000 7f5f8c0b0640 1 --2- 172.21.15.12:0/1636857028 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f840a3270 0x7f5f84138370 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:59.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.273+0000 7f5f8c0b0640 1 --2- 172.21.15.12:0/1636857028 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5f6803e1c0 0x7f5f68040680 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:59.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.273+0000 7f5f8c0b0640 1 --2- 172.21.15.12:0/1636857028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f840a2e90 0x7f5f8409dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:36:59.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.273+0000 7f5f8c0b0640 1 -- 172.21.15.12:0/1636857028 >> 172.21.15.12:0/1636857028 conn(0x7f5f840a8490 msgr2=0x7f5f8413c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:36:59.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.273+0000 7f5f8c0b0640 1 -- 172.21.15.12:0/1636857028 shutdown_connections 2024-09-11T01:36:59.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:36:59.273+0000 7f5f8c0b0640 1 -- 172.21.15.12:0/1636857028 wait complete. 2024-09-11T01:37:02.287 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:37:02.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.399+0000 7f8c2eb6b640 10 monclient: get_monmap_and_config 2024-09-11T01:37:02.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.399+0000 7f8c2eb6b640 10 monclient: build_initial_monmap 2024-09-11T01:37:02.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.399+0000 7f8c2eb6b640 10 monclient: monmap: 2024-09-11T01:37:02.400 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:02.401 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:02.401 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:37:02.399799+0000 2024-09-11T01:37:02.401 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:37:02.399799+0000 2024-09-11T01:37:02.401 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:02.401 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:02.401 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:37:02.401 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:37:02.401 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:37:02.401 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:37:02.401 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:37:02.401 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:02.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.399+0000 7f8c2eb6b640 1 Processor -- start 2024-09-11T01:37:02.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.399+0000 7f8c2eb6b640 1 -- start start 2024-09-11T01:37:02.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.399+0000 7f8c2eb6b640 10 monclient: init 2024-09-11T01:37:02.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.400+0000 7f8c2eb6b640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:02.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.400+0000 7f8c2eb6b640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-11T01:37:02.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.400+0000 7f8c2eb6b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8c28138230 0x7f8c28138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:02.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.400+0000 7f8c2eb6b640 10 monclient(hunting): picked mon.noname-a con 0x7f8c28138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:37:02.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.400+0000 7f8c2eb6b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8c2809d030 addr v1:172.21.15.12:6789/0 2024-09-11T01:37:02.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.400+0000 7f8c2eb6b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8c2809d460 addr v1:172.21.15.71:6789/0 2024-09-11T01:37:02.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.400+0000 7f8c2eb6b640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c280a5e40 con 0x7f8c2809d030 2024-09-11T01:37:02.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.400+0000 7f8c2eb6b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c280a6e30 con 0x7f8c2809d460 2024-09-11T01:37:02.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.400+0000 7f8c2eb6b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:02.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.400+0000 7f8c2eb6b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f8c2809c9c0 con 0x7f8c28138230 2024-09-11T01:37:02.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.400+0000 7f8c2eb6b640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:02.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.400+0000 7f8c2eb6b640 10 monclient(hunting): authenticate will time out at 1228.842541s 2024-09-11T01:37:02.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.400+0000 7f8c2c907640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7f8c28138230 msgr2=0x7f8c28138610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:37:02.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.400+0000 7f8c2c907640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8c28138230 0x7f8c28138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:37:02.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.400+0000 7f8c27fff640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7f8c2809d030 legacy=0x7f8c280a5020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:37:02.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.400+0000 7f8c2d108640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f8c2809d460 0x7f8c280a5730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.12:41108/0 (socket says 172.21.15.12:41108) 2024-09-11T01:37:02.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.400+0000 7f8c2d108640 1 -- 172.21.15.12:0/3953837102 learned_addr learned my addr 172.21.15.12:0/3953837102 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:37:02.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.403+0000 7f8c2d108640 1 -- 172.21.15.12:0/3953837102 >> v1:172.21.15.71:6789/0 conn(0x7f8c2809d460 legacy=0x7f8c280a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:02.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.403+0000 7f8c277fe640 1 -- 172.21.15.12:0/3953837102 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 948037735 0 0) 0x7f8c280a6e30 con 0x7f8c2809d460 2024-09-11T01:37:02.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.403+0000 7f8c277fe640 10 monclient(hunting): _negotiate 2024-09-11T01:37:02.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.403+0000 7f8c277fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:02.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.403+0000 7f8c277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:02.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.403+0000 7f8c277fe640 10 monclient(hunting): my global_id is 74233 2024-09-11T01:37:02.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.403+0000 7f8c277fe640 1 -- 172.21.15.12:0/3953837102 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8c18003520 con 0x7f8c2809d460 2024-09-11T01:37:02.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.403+0000 7f8c277fe640 1 -- 172.21.15.12:0/3953837102 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3184000073 0 0) 0x7f8c18003520 con 0x7f8c2809d460 2024-09-11T01:37:02.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 1 -- 172.21.15.12:0/3953837102 >> v2:172.21.15.12:3300/0 conn(0x7f8c28138230 msgr2=0x7f8c28138610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:37:02.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 1 --2- 172.21.15.12:0/3953837102 >> v2:172.21.15.12:3300/0 conn(0x7f8c28138230 0x7f8c28138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:02.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 1 -- 172.21.15.12:0/3953837102 >> v1:172.21.15.12:6789/0 conn(0x7f8c2809d030 legacy=0x7f8c280a5020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:37:02.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 10 monclient: _finish_hunting 0 2024-09-11T01:37:02.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 1 monclient: found mon.noname-c-legacy 2024-09-11T01:37:02.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:02.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-09-11T01:37:02.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 1 -- 172.21.15.12:0/3953837102 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c2809ba20 con 0x7f8c2809d460 2024-09-11T01:37:02.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 10 monclient: _finish_auth 0 2024-09-11T01:37:02.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 10 monclient: _check_auth_tickets 2024-09-11T01:37:02.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:02.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 1 -- 172.21.15.12:0/3953837102 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f8c20001de0 con 0x7f8c2809d460 2024-09-11T01:37:02.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c2eb6b640 5 monclient: authenticate success, global_id 74233 2024-09-11T01:37:02.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c2eb6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:37:02.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:02.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-11T01:37:02.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 10 monclient: dump: 2024-09-11T01:37:02.410 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:02.410 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:02.410 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:02.410 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:02.410 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:02.410 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:02.410 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:02.410 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:02.410 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:02.410 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:02.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-09-11T01:37:02.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:02.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 1 -- 172.21.15.12:0/3953837102 >> v1:172.21.15.71:6789/0 conn(0x7f8c2809d460 legacy=0x7f8c280a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:02.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:37:02.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 1 --2- 172.21.15.12:0/3953837102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c180039b0 0x7f8c18003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:02.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 10 monclient(hunting): picked mon.a con 0x7f8c180039b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:37:02.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 1 --2- 172.21.15.12:0/3953837102 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c18004800 0x7f8c18004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:02.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 10 monclient(hunting): picked mon.b con 0x7f8c18004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:37:02.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 1 --2- 172.21.15.12:0/3953837102 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8c180053c0 0x7f8c1800ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:02.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 10 monclient(hunting): picked mon.c con 0x7f8c180053c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:02.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:02.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 1 -- 172.21.15.12:0/3953837102 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2809c9c0 con 0x7f8c180039b0 2024-09-11T01:37:02.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:02.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 1 -- 172.21.15.12:0/3953837102 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8c1800c300 con 0x7f8c18004800 2024-09-11T01:37:02.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:02.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 1 -- 172.21.15.12:0/3953837102 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8c1800c500 con 0x7f8c180053c0 2024-09-11T01:37:02.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:02.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c2c907640 1 -- 172.21.15.12:0/3953837102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c180039b0 msgr2=0x7f8c18003db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:37:02.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c277fe640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:37:02.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c2eb6b640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:37:02.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c2c907640 1 --2- 172.21.15.12:0/3953837102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c180039b0 0x7f8c18003db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:37:02.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c2d108640 1 --2- 172.21.15.12:0/3953837102 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8c180053c0 0x7f8c1800ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:02.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c2d108640 10 monclient(hunting): get_auth_request con 0x7f8c180053c0 auth_method 0 2024-09-11T01:37:02.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c2d108640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:02.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c2d108640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:02.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c2d108640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:37:02.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c27fff640 1 --2- 172.21.15.12:0/3953837102 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c18004800 0x7f8c18004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:02.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c27fff640 10 monclient(hunting): get_auth_request con 0x7f8c18004800 auth_method 0 2024-09-11T01:37:02.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:02.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c27fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:02.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c27fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:37:02.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c2d108640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:02.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c2d108640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:02.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c2d108640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:37:02.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:02.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:02.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c27fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:37:02.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c2d108640 10 monclient(hunting): handle_auth_done global_id 74233 payload 293 2024-09-11T01:37:02.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c2d108640 1 -- 172.21.15.12:0/3953837102 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c18004800 msgr2=0x7f8c18004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:02.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c2d108640 1 --2- 172.21.15.12:0/3953837102 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c18004800 0x7f8c18004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:02.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c2d108640 1 -- 172.21.15.12:0/3953837102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c180039b0 msgr2=0x7f8c18003db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:37:02.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c2d108640 1 --2- 172.21.15.12:0/3953837102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c180039b0 0x7f8c18003db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:02.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c2d108640 10 monclient: _finish_hunting 0 2024-09-11T01:37:02.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c2d108640 1 monclient: found mon.c 2024-09-11T01:37:02.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c2d108640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:02.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c2d108640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:02.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c2d108640 1 -- 172.21.15.12:0/3953837102 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c18003520 con 0x7f8c180053c0 2024-09-11T01:37:02.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c2d108640 10 monclient: _finish_auth 0 2024-09-11T01:37:02.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c2d108640 10 monclient: _check_auth_tickets 2024-09-11T01:37:02.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.404+0000 7f8c2d108640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:02.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.405+0000 7f8c2d108640 1 -- 172.21.15.12:0/3953837102 reap_dead start 2024-09-11T01:37:02.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.407+0000 7f8c2d108640 1 -- 172.21.15.12:0/3953837102 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8c180053c0 msgr2=0x7f8c1800ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:02.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.407+0000 7f8c2d108640 1 --2- 172.21.15.12:0/3953837102 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8c180053c0 0x7f8c1800ba20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8c2809d860 tx=0x7f8c18005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f18b69b02993d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:02.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.407+0000 7f8c277fe640 1 -- 172.21.15.12:0/3953837102 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c2000a7a0 con 0x7f8c180053c0 2024-09-11T01:37:02.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.407+0000 7f8c277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:02.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.407+0000 7f8c277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:02.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.407+0000 7f8c277fe640 10 monclient: dump: 2024-09-11T01:37:02.420 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:02.420 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:02.420 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:02.420 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:02.421 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:02.421 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:02.421 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:02.421 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:02.421 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:02.421 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:02.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.407+0000 7f8c277fe640 1 -- 172.21.15.12:0/3953837102 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c2002f070 con 0x7f8c180053c0 2024-09-11T01:37:02.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.407+0000 7f8c2eb6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:37:02.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.407+0000 7f8c277fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:02.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.407+0000 7f8c2eb6b640 10 monclient: get_monmap_and_config success 2024-09-11T01:37:02.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.407+0000 7f8c277fe640 1 -- 172.21.15.12:0/3953837102 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c2002bd10 con 0x7f8c180053c0 2024-09-11T01:37:02.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.407+0000 7f8c277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:02.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.407+0000 7f8c2eb6b640 4 set_mon_vals no callback set 2024-09-11T01:37:02.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.407+0000 7f8c277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:02.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.407+0000 7f8c277fe640 10 monclient: dump: 2024-09-11T01:37:02.422 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:02.422 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:02.422 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:02.422 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:02.422 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:02.422 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:02.422 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:02.422 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:02.422 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:02.422 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:02.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.408+0000 7f8c2eb6b640 10 monclient: shutdown 2024-09-11T01:37:02.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.408+0000 7f8c2eb6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:02.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.408+0000 7f8c2eb6b640 1 -- 172.21.15.12:0/3953837102 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8c180053c0 msgr2=0x7f8c1800ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:02.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.408+0000 7f8c2eb6b640 1 --2- 172.21.15.12:0/3953837102 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8c180053c0 0x7f8c1800ba20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8c2809d860 tx=0x7f8c18005830 comp rx=0 tx=0).stop 2024-09-11T01:37:02.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.408+0000 7f8c2eb6b640 1 -- 172.21.15.12:0/3953837102 shutdown_connections 2024-09-11T01:37:02.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.408+0000 7f8c2eb6b640 1 --2- 172.21.15.12:0/3953837102 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8c180053c0 0x7f8c1800ba20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:02.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.408+0000 7f8c2eb6b640 1 -- 172.21.15.12:0/3953837102 >> 172.21.15.12:0/3953837102 conn(0x7f8c280a7120 msgr2=0x7f8c280a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:02.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.408+0000 7f8c2eb6b640 1 -- 172.21.15.12:0/3953837102 shutdown_connections 2024-09-11T01:37:02.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.408+0000 7f8c2eb6b640 1 -- 172.21.15.12:0/3953837102 wait complete. 2024-09-11T01:37:02.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.408+0000 7f8c2eb6b640 10 monclient: build_initial_monmap 2024-09-11T01:37:02.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.408+0000 7f8c2eb6b640 10 monclient: monmap: 2024-09-11T01:37:02.423 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:02.423 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:37:02.423 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:37:02.423 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:37:02.424 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:02.424 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:02.424 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:37:02.424 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:37:02.424 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:37:02.424 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:02.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.409+0000 7f8c2eb6b640 1 Processor -- start 2024-09-11T01:37:02.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.409+0000 7f8c2eb6b640 1 -- start start 2024-09-11T01:37:02.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.409+0000 7f8c2eb6b640 10 monclient: init 2024-09-11T01:37:02.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.409+0000 7f8c2eb6b640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:02.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.409+0000 7f8c2eb6b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:37:02.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.409+0000 7f8c2eb6b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8c180053c0 0x7f8c28136fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:02.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.409+0000 7f8c2eb6b640 10 monclient(hunting): picked mon.noname-c con 0x7f8c180053c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:02.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.409+0000 7f8c2eb6b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c281ccd90 0x7f8c28137510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:02.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.409+0000 7f8c2eb6b640 10 monclient(hunting): picked mon.noname-a con 0x7f8c281ccd90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:37:02.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.409+0000 7f8c2eb6b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c28124c00 0x7f8c28125060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:02.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.409+0000 7f8c2eb6b640 10 monclient(hunting): picked mon.noname-b con 0x7f8c28124c00 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:37:02.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.409+0000 7f8c2eb6b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:02.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.409+0000 7f8c2eb6b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2806f970 con 0x7f8c281ccd90 2024-09-11T01:37:02.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.409+0000 7f8c2eb6b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:02.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.409+0000 7f8c2eb6b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2806f7f0 con 0x7f8c28124c00 2024-09-11T01:37:02.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.409+0000 7f8c2eb6b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:02.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.409+0000 7f8c2eb6b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8c28138610 con 0x7f8c180053c0 2024-09-11T01:37:02.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.409+0000 7f8c2eb6b640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:02.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.409+0000 7f8c2eb6b640 10 monclient(hunting): authenticate will time out at 1228.851934s 2024-09-11T01:37:02.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.409+0000 7f8c27fff640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c281ccd90 msgr2=0x7f8c28137510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:37:02.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.409+0000 7f8c27fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c281ccd90 0x7f8c28137510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:37:02.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.409+0000 7f8c257fa640 1 -- DispatchQueue::entry inject delay of 0.002000 2024-09-11T01:37:02.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2c907640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8c180053c0 0x7f8c28136fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:02.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2c907640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8c180053c0 0x7f8c28136fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:34718/0 (socket says 172.21.15.12:34718) 2024-09-11T01:37:02.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2c907640 1 -- 172.21.15.12:0/2831551424 learned_addr learned my addr 172.21.15.12:0/2831551424 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:02.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2c907640 10 monclient(hunting): get_auth_request con 0x7f8c180053c0 auth_method 0 2024-09-11T01:37:02.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2c907640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:02.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2c907640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:02.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2c907640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:02.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2c907640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:02.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2c907640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:02.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2d108640 1 --2- 172.21.15.12:0/2831551424 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c28124c00 0x7f8c28125060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:02.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2c907640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:02.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2d108640 10 monclient(hunting): get_auth_request con 0x7f8c28124c00 auth_method 0 2024-09-11T01:37:02.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2d108640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:02.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2d108640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:02.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2d108640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:02.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2d108640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:02.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2d108640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:02.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2d108640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:02.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2c907640 10 monclient(hunting): handle_auth_done global_id 84155 payload 824 2024-09-11T01:37:02.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2c907640 1 -- 172.21.15.12:0/2831551424 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c28124c00 msgr2=0x7f8c28125060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:02.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2c907640 1 --2- 172.21.15.12:0/2831551424 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c28124c00 0x7f8c28125060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:02.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2c907640 1 -- 172.21.15.12:0/2831551424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c281ccd90 msgr2=0x7f8c28137510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:37:02.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2c907640 1 --2- 172.21.15.12:0/2831551424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c281ccd90 0x7f8c28137510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:02.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2c907640 10 monclient: _finish_hunting 0 2024-09-11T01:37:02.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2c907640 1 monclient: found mon.noname-c 2024-09-11T01:37:02.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2c907640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:02.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2c907640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:37:02.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.410+0000 7f8c2c907640 1 -- 172.21.15.12:0/2831551424 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c281cda40 con 0x7f8c180053c0 2024-09-11T01:37:02.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.412+0000 7f8c2c907640 1 -- 172.21.15.12:0/2831551424 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8c180053c0 msgr2=0x7f8c28136fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:02.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.412+0000 7f8c2c907640 1 --2- 172.21.15.12:0/2831551424 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8c180053c0 0x7f8c28136fd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c00b580 tx=0x7f8c1c00ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:02.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.412+0000 7f8c257fa640 1 -- 172.21.15.12:0/2831551424 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c1c0050f0 con 0x7f8c180053c0 2024-09-11T01:37:02.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.412+0000 7f8c257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:02.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.412+0000 7f8c257fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:37:02.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.412+0000 7f8c257fa640 10 monclient: dump: 2024-09-11T01:37:02.428 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:02.428 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:02.428 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:02.429 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:02.429 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:02.429 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:02.429 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:02.429 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:02.429 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:02.429 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:02.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.412+0000 7f8c257fa640 10 monclient: _finish_auth 0 2024-09-11T01:37:02.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.412+0000 7f8c257fa640 10 monclient: _check_auth_tickets 2024-09-11T01:37:02.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.412+0000 7f8c257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:02.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.412+0000 7f8c257fa640 1 -- 172.21.15.12:0/2831551424 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c1c015070 con 0x7f8c180053c0 2024-09-11T01:37:02.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.412+0000 7f8c2eb6b640 5 monclient: authenticate success, global_id 84155 2024-09-11T01:37:02.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.412+0000 7f8c2eb6b640 10 monclient: _renew_subs 2024-09-11T01:37:02.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.412+0000 7f8c2eb6b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:02.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.412+0000 7f8c2eb6b640 1 -- 172.21.15.12:0/2831551424 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c280a1810 con 0x7f8c180053c0 2024-09-11T01:37:02.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.412+0000 7f8c257fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:02.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.412+0000 7f8c257fa640 1 -- 172.21.15.12:0/2831551424 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c1c0054d0 con 0x7f8c180053c0 2024-09-11T01:37:02.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.412+0000 7f8c257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:02.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.412+0000 7f8c257fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:02.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.412+0000 7f8c257fa640 10 monclient: dump: 2024-09-11T01:37:02.430 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:02.430 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:02.430 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:02.431 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:02.431 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:02.431 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:02.431 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:02.431 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:02.431 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:02.431 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:02.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.412+0000 7f8c26ffd640 4 set_mon_vals no callback set 2024-09-11T01:37:02.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.412+0000 7f8c2eb6b640 10 monclient: _renew_subs 2024-09-11T01:37:02.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.412+0000 7f8c2eb6b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:02.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.412+0000 7f8c2eb6b640 1 -- 172.21.15.12:0/2831551424 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c28137a50 con 0x7f8c180053c0 2024-09-11T01:37:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.413+0000 7f8c257fa640 1 -- 172.21.15.12:0/2831551424 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f8c1c0056b0 con 0x7f8c180053c0 2024-09-11T01:37:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.413+0000 7f8c2eb6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.413+0000 7f8c2eb6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.413+0000 7f8c2eb6b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.413+0000 7f8c2eb6b640 1 -- 172.21.15.12:0/2831551424 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c281389a0 con 0x7f8c180053c0 2024-09-11T01:37:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.413+0000 7f8c257fa640 1 --2- 172.21.15.12:0/2831551424 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8c0c03e1c0 0x7f8c0c040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.414+0000 7f8c257fa640 1 -- 172.21.15.12:0/2831551424 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(258..258 src has 186..258) ==== 4875+0+0 (secure 0 0 0) 0x7f8c1c052ba0 con 0x7f8c180053c0 2024-09-11T01:37:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.414+0000 7f8c27fff640 1 --2- 172.21.15.12:0/2831551424 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8c0c03e1c0 0x7f8c0c040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.414+0000 7f8c27fff640 10 monclient: get_auth_request con 0x7f8c0c03e1c0 auth_method 0 2024-09-11T01:37:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.416+0000 7f8c27fff640 1 --2- 172.21.15.12:0/2831551424 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8c0c03e1c0 0x7f8c0c040680 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8c28127160 tx=0x7f8c14008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.418+0000 7f8c257fa640 1 -- 172.21.15.12:0/2831551424 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8c280a1810 con 0x7f8c180053c0 2024-09-11T01:37:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.418+0000 7f8c257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:02.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.418+0000 7f8c257fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:37:02.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.673+0000 7f8c2eb6b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:37:02.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.673+0000 7f8c2eb6b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:37:02.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.673+0000 7f8c2eb6b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:02.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.673+0000 7f8c2eb6b640 1 -- 172.21.15.12:0/2831551424 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8c28127580 con 0x7f8c180053c0 2024-09-11T01:37:02.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.674+0000 7f8c257fa640 1 -- 172.21.15.12:0/2831551424 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f8c1c051540 con 0x7f8c180053c0 2024-09-11T01:37:02.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.674+0000 7f8c257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:37:02.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.674+0000 7f8c257fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:37:02.676 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:37:02.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.676+0000 7f8c2eb6b640 1 -- 172.21.15.12:0/2831551424 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8c0c03e1c0 msgr2=0x7f8c0c040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:02.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.676+0000 7f8c2eb6b640 1 --2- 172.21.15.12:0/2831551424 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8c0c03e1c0 0x7f8c0c040680 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8c28127160 tx=0x7f8c14008040 comp rx=0 tx=0).stop 2024-09-11T01:37:02.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.676+0000 7f8c2eb6b640 10 monclient: shutdown 2024-09-11T01:37:02.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.676+0000 7f8c2eb6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:02.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.676+0000 7f8c2eb6b640 1 -- 172.21.15.12:0/2831551424 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8c180053c0 msgr2=0x7f8c28136fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:02.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.676+0000 7f8c2eb6b640 1 --2- 172.21.15.12:0/2831551424 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8c180053c0 0x7f8c28136fd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c00b580 tx=0x7f8c1c00ba40 comp rx=0 tx=0).stop 2024-09-11T01:37:02.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.676+0000 7f8c2eb6b640 1 -- 172.21.15.12:0/2831551424 shutdown_connections 2024-09-11T01:37:02.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.676+0000 7f8c2eb6b640 1 --2- 172.21.15.12:0/2831551424 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c28124c00 0x7f8c28125060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:02.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.676+0000 7f8c2eb6b640 1 --2- 172.21.15.12:0/2831551424 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8c0c03e1c0 0x7f8c0c040680 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:02.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.676+0000 7f8c2eb6b640 1 --2- 172.21.15.12:0/2831551424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c281ccd90 0x7f8c28137510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:02.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.676+0000 7f8c2eb6b640 1 --2- 172.21.15.12:0/2831551424 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8c180053c0 0x7f8c28136fd0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:02.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.676+0000 7f8c2eb6b640 1 -- 172.21.15.12:0/2831551424 >> 172.21.15.12:0/2831551424 conn(0x7f8c280a7120 msgr2=0x7f8c280a3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:02.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.676+0000 7f8c2eb6b640 1 -- 172.21.15.12:0/2831551424 shutdown_connections 2024-09-11T01:37:02.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:02.676+0000 7f8c2eb6b640 1 -- 172.21.15.12:0/2831551424 wait complete. 2024-09-11T01:37:05.689 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:37:05.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.800+0000 7f5071ff3640 10 monclient: get_monmap_and_config 2024-09-11T01:37:05.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.800+0000 7f5071ff3640 10 monclient: build_initial_monmap 2024-09-11T01:37:05.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.800+0000 7f5071ff3640 10 monclient: monmap: 2024-09-11T01:37:05.802 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:05.802 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:05.802 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:37:05.801406+0000 2024-09-11T01:37:05.802 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:37:05.801406+0000 2024-09-11T01:37:05.802 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:05.802 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:05.802 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:37:05.803 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:37:05.803 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:37:05.803 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:37:05.803 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:37:05.803 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:05.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.801+0000 7f5071ff3640 1 Processor -- start 2024-09-11T01:37:05.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.801+0000 7f5071ff3640 1 -- start start 2024-09-11T01:37:05.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.801+0000 7f5071ff3640 10 monclient: init 2024-09-11T01:37:05.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.801+0000 7f5071ff3640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:05.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.801+0000 7f5071ff3640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-11T01:37:05.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.801+0000 7f5071ff3640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f506c0a2db0 0x7f506c06cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:05.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f5071ff3640 10 monclient(hunting): picked mon.noname-b con 0x7f506c0a2db0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:05.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f5071ff3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f506c09fbd0 addr v1:172.21.15.12:6789/0 2024-09-11T01:37:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f5071ff3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f506c13d550 addr v1:172.21.15.71:6789/0 2024-09-11T01:37:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f5071ff3640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f506c09eea0 con 0x7f506c09fbd0 2024-09-11T01:37:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f5071ff3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f506c09f180 con 0x7f506c13d550 2024-09-11T01:37:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f5071ff3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f5071ff3640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f506c0a3850 con 0x7f506c0a2db0 2024-09-11T01:37:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f5071ff3640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f5071ff3640 10 monclient(hunting): authenticate will time out at 1232.244077s 2024-09-11T01:37:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f506affd640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7f506c09fbd0 legacy=0x7f506c06d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:37:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f506b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f506c0a2db0 0x7f506c06cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f506b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f506c0a2db0 0x7f506c06cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:34722/0 (socket says 172.21.15.12:34722) 2024-09-11T01:37:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f506b7fe640 1 -- 172.21.15.12:0/3730469356 learned_addr learned my addr 172.21.15.12:0/3730469356 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f506b7fe640 10 monclient(hunting): get_auth_request con 0x7f506c0a2db0 auth_method 0 2024-09-11T01:37:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f506b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:05.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f506b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:05.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f506b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:05.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f506b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:05.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f506b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:05.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f506b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:05.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f506b7fe640 10 monclient(hunting): handle_auth_done global_id 84158 payload 274 2024-09-11T01:37:05.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f506b7fe640 1 -- 172.21.15.12:0/3730469356 >> v1:172.21.15.71:6789/0 conn(0x7f506c13d550 legacy=0x7f506c13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:05.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f506b7fe640 1 -- 172.21.15.12:0/3730469356 >> v1:172.21.15.12:6789/0 conn(0x7f506c09fbd0 legacy=0x7f506c06d090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:37:05.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f506b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:37:05.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f506b7fe640 1 monclient: found mon.noname-b 2024-09-11T01:37:05.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f506b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:05.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f506b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:37:05.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.802+0000 7f506b7fe640 1 -- 172.21.15.12:0/3730469356 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f506c0a4ce0 con 0x7f506c0a2db0 2024-09-11T01:37:05.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.805+0000 7f506b7fe640 1 -- 172.21.15.12:0/3730469356 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f506c0a2db0 msgr2=0x7f506c06cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:05.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.805+0000 7f506b7fe640 1 --2- 172.21.15.12:0/3730469356 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f506c0a2db0 0x7f506c06cb50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f505c0098a0 tx=0x7f505c02a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26681990df2cfff6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:05.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.805+0000 7f506a7fc640 1 -- 172.21.15.12:0/3730469356 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f505c037070 con 0x7f506c0a2db0 2024-09-11T01:37:05.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.805+0000 7f506a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:05.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.805+0000 7f506a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:37:05.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.805+0000 7f506a7fc640 10 monclient: dump: 2024-09-11T01:37:05.812 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:05.812 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:05.813 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:05.813 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:05.813 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:05.813 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:05.813 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:05.813 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:05.813 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:05.813 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:05.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.805+0000 7f506a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:37:05.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.805+0000 7f506a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:37:05.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.805+0000 7f506a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:05.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.805+0000 7f506a7fc640 1 -- 172.21.15.12:0/3730469356 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f505c032640 con 0x7f506c0a2db0 2024-09-11T01:37:05.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.805+0000 7f5071ff3640 5 monclient: authenticate success, global_id 84158 2024-09-11T01:37:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.805+0000 7f5071ff3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:37:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.805+0000 7f506a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.805+0000 7f5071ff3640 10 monclient: get_monmap_and_config success 2024-09-11T01:37:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.805+0000 7f506a7fc640 1 -- 172.21.15.12:0/3730469356 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f505c032a10 con 0x7f506c0a2db0 2024-09-11T01:37:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.805+0000 7f5071ff3640 4 set_mon_vals no callback set 2024-09-11T01:37:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.805+0000 7f506a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.805+0000 7f506a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.805+0000 7f506a7fc640 10 monclient: dump: 2024-09-11T01:37:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:05.814 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:05.815 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:05.815 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:05.815 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:05.815 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:05.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.805+0000 7f5071ff3640 10 monclient: shutdown 2024-09-11T01:37:05.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.805+0000 7f5071ff3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:05.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.805+0000 7f5071ff3640 1 -- 172.21.15.12:0/3730469356 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f506c0a2db0 msgr2=0x7f506c06cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:05.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.805+0000 7f5071ff3640 1 --2- 172.21.15.12:0/3730469356 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f506c0a2db0 0x7f506c06cb50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f505c0098a0 tx=0x7f505c02a080 comp rx=0 tx=0).stop 2024-09-11T01:37:05.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.805+0000 7f5071ff3640 1 -- 172.21.15.12:0/3730469356 shutdown_connections 2024-09-11T01:37:05.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.805+0000 7f5071ff3640 1 --2- 172.21.15.12:0/3730469356 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f506c0a2db0 0x7f506c06cb50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:05.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.805+0000 7f5071ff3640 1 -- 172.21.15.12:0/3730469356 >> 172.21.15.12:0/3730469356 conn(0x7f506c0a3ea0 msgr2=0x7f506c0a42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:05.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.806+0000 7f5071ff3640 1 -- 172.21.15.12:0/3730469356 shutdown_connections 2024-09-11T01:37:05.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.806+0000 7f5071ff3640 1 -- 172.21.15.12:0/3730469356 wait complete. 2024-09-11T01:37:05.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.806+0000 7f5071ff3640 10 monclient: build_initial_monmap 2024-09-11T01:37:05.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.806+0000 7f5071ff3640 10 monclient: monmap: 2024-09-11T01:37:05.816 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:05.816 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:37:05.816 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:37:05.816 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:37:05.816 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:05.816 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:05.816 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:37:05.816 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:37:05.816 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:37:05.816 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:05.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.806+0000 7f5071ff3640 1 Processor -- start 2024-09-11T01:37:05.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f5071ff3640 1 -- start start 2024-09-11T01:37:05.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f5071ff3640 10 monclient: init 2024-09-11T01:37:05.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f5071ff3640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:05.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f5071ff3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:37:05.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f5071ff3640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f506c09fbd0 0x7f506c13ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:05.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f5071ff3640 10 monclient(hunting): picked mon.noname-c con 0x7f506c09fbd0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:05.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f5071ff3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f506c0a2db0 0x7f506c13cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:05.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f5071ff3640 10 monclient(hunting): picked mon.noname-a con 0x7f506c0a2db0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:37:05.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f5071ff3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f506c13d550 0x7f506c0913c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:05.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f5071ff3640 10 monclient(hunting): picked mon.noname-b con 0x7f506c13d550 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:37:05.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f5071ff3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:05.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f5071ff3640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f506c06d7a0 con 0x7f506c0a2db0 2024-09-11T01:37:05.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f5071ff3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:05.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f5071ff3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f506c0a3190 con 0x7f506c13d550 2024-09-11T01:37:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f5071ff3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f5071ff3640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f506c06d920 con 0x7f506c09fbd0 2024-09-11T01:37:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f5071ff3640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f5071ff3640 10 monclient(hunting): authenticate will time out at 1232.249614s 2024-09-11T01:37:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f506affd640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f506c0a2db0 msgr2=0x7f506c13cf90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:37:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f506affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f506c0a2db0 0x7f506c13cf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:37:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f506b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f506c09fbd0 0x7f506c13ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f506b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f506c09fbd0 0x7f506c13ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:34734/0 (socket says 172.21.15.12:34734) 2024-09-11T01:37:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f506b7fe640 1 -- 172.21.15.12:0/1313861084 learned_addr learned my addr 172.21.15.12:0/1313861084 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f506b7fe640 10 monclient(hunting): get_auth_request con 0x7f506c09fbd0 auth_method 0 2024-09-11T01:37:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f506b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f506b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f506b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f506b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:05.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f506b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:05.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f506b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:05.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f506bfff640 1 --2- 172.21.15.12:0/1313861084 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f506c13d550 0x7f506c0913c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:05.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f506b7fe640 10 monclient(hunting): handle_auth_done global_id 84161 payload 824 2024-09-11T01:37:05.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f506b7fe640 1 -- 172.21.15.12:0/1313861084 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f506c13d550 msgr2=0x7f506c0913c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:05.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f506b7fe640 1 --2- 172.21.15.12:0/1313861084 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f506c13d550 0x7f506c0913c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:05.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.807+0000 7f506b7fe640 1 -- 172.21.15.12:0/1313861084 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f506c0a2db0 msgr2=0x7f506c13cf90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:37:05.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.808+0000 7f506b7fe640 1 --2- 172.21.15.12:0/1313861084 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f506c0a2db0 0x7f506c13cf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:05.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.808+0000 7f506b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:37:05.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.808+0000 7f506b7fe640 1 monclient: found mon.noname-c 2024-09-11T01:37:05.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.808+0000 7f506b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:05.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.808+0000 7f506b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:37:05.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.808+0000 7f506b7fe640 1 -- 172.21.15.12:0/1313861084 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f506c0935e0 con 0x7f506c09fbd0 2024-09-11T01:37:05.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.808+0000 7f506bfff640 10 monclient: get_auth_request con 0x7f506c13d550 auth_method 0 2024-09-11T01:37:05.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.808+0000 7f506bfff640 1 --2- 172.21.15.12:0/1313861084 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f506c13d550 0x7f506c0913c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:37:05.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.810+0000 7f506b7fe640 1 -- 172.21.15.12:0/1313861084 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f506c09fbd0 msgr2=0x7f506c13ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:05.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.810+0000 7f506b7fe640 1 --2- 172.21.15.12:0/1313861084 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f506c09fbd0 0x7f506c13ca50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f505c0098a0 tx=0x7f505c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:05.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.810+0000 7f5053fff640 1 -- 172.21.15.12:0/1313861084 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f505c041070 con 0x7f506c09fbd0 2024-09-11T01:37:05.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.810+0000 7f5053fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:05.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.810+0000 7f5053fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:37:05.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.810+0000 7f5053fff640 10 monclient: dump: 2024-09-11T01:37:05.820 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:05.820 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:05.820 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:05.820 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:05.820 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:05.821 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:05.821 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:05.821 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:05.821 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:05.821 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:05.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.810+0000 7f5053fff640 10 monclient: _finish_auth 0 2024-09-11T01:37:05.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.810+0000 7f5053fff640 10 monclient: _check_auth_tickets 2024-09-11T01:37:05.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.810+0000 7f5053fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:05.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.810+0000 7f5053fff640 1 -- 172.21.15.12:0/1313861084 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f505c03ca80 con 0x7f506c09fbd0 2024-09-11T01:37:05.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.810+0000 7f5071ff3640 5 monclient: authenticate success, global_id 84161 2024-09-11T01:37:05.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.810+0000 7f5071ff3640 10 monclient: _renew_subs 2024-09-11T01:37:05.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.810+0000 7f5071ff3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:05.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.810+0000 7f5071ff3640 1 -- 172.21.15.12:0/1313861084 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f506c09e590 con 0x7f506c09fbd0 2024-09-11T01:37:05.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.810+0000 7f5053fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:05.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.810+0000 7f5053fff640 1 -- 172.21.15.12:0/1313861084 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f505c037040 con 0x7f506c09fbd0 2024-09-11T01:37:05.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.810+0000 7f5053fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:05.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.810+0000 7f5053fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:05.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.810+0000 7f5053fff640 10 monclient: dump: 2024-09-11T01:37:05.822 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:05.822 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:05.822 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:05.822 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:05.822 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:05.822 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:05.822 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:05.823 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:05.823 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:05.823 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:05.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.810+0000 7f5071ff3640 10 monclient: _renew_subs 2024-09-11T01:37:05.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.810+0000 7f5071ff3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:05.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.810+0000 7f5071ff3640 1 -- 172.21.15.12:0/1313861084 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f506c073fb0 con 0x7f506c09fbd0 2024-09-11T01:37:05.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.810+0000 7f5069ffb640 4 set_mon_vals no callback set 2024-09-11T01:37:05.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.811+0000 7f5053fff640 1 -- 172.21.15.12:0/1313861084 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f505c03d3f0 con 0x7f506c09fbd0 2024-09-11T01:37:05.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.811+0000 7f5071ff3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:05.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.811+0000 7f5071ff3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:05.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.811+0000 7f5071ff3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:05.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.811+0000 7f5071ff3640 1 -- 172.21.15.12:0/1313861084 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f506c06cbd0 con 0x7f506c09fbd0 2024-09-11T01:37:05.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.811+0000 7f5053fff640 1 --2- 172.21.15.12:0/1313861084 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f505803e1c0 0x7f5058040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:05.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.811+0000 7f5053fff640 1 -- 172.21.15.12:0/1313861084 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(258..258 src has 186..258) ==== 4875+0+0 (secure 0 0 0) 0x7f505c073a30 con 0x7f506c09fbd0 2024-09-11T01:37:05.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.812+0000 7f506affd640 1 --2- 172.21.15.12:0/1313861084 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f505803e1c0 0x7f5058040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:05.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.812+0000 7f506affd640 10 monclient: get_auth_request con 0x7f505803e1c0 auth_method 0 2024-09-11T01:37:05.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.814+0000 7f506affd640 1 --2- 172.21.15.12:0/1313861084 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f505803e1c0 0x7f5058040680 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5060006fd0 tx=0x7f5060006d00 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:05.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.816+0000 7f5053fff640 1 -- 172.21.15.12:0/1313861084 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f506c0935e0 con 0x7f506c09fbd0 2024-09-11T01:37:05.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.816+0000 7f5053fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:05.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:05.816+0000 7f5053fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:37:06.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:06.072+0000 7f5071ff3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:37:06.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:06.072+0000 7f5071ff3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:37:06.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:06.072+0000 7f5071ff3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:06.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:06.072+0000 7f5071ff3640 1 -- 172.21.15.12:0/1313861084 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f506c06e570 con 0x7f506c09fbd0 2024-09-11T01:37:06.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:06.073+0000 7f5053fff640 1 -- 172.21.15.12:0/1313861084 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f505c03de20 con 0x7f506c09fbd0 2024-09-11T01:37:06.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:06.073+0000 7f5053fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:37:06.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:06.073+0000 7f5053fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:37:06.075 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:37:06.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:06.075+0000 7f5071ff3640 1 -- 172.21.15.12:0/1313861084 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f505803e1c0 msgr2=0x7f5058040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:06.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:06.075+0000 7f5071ff3640 1 --2- 172.21.15.12:0/1313861084 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f505803e1c0 0x7f5058040680 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5060006fd0 tx=0x7f5060006d00 comp rx=0 tx=0).stop 2024-09-11T01:37:06.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:06.075+0000 7f5071ff3640 10 monclient: shutdown 2024-09-11T01:37:06.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:06.075+0000 7f5071ff3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:06.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:06.075+0000 7f5071ff3640 1 -- 172.21.15.12:0/1313861084 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f506c09fbd0 msgr2=0x7f506c13ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:06.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:06.075+0000 7f5071ff3640 1 --2- 172.21.15.12:0/1313861084 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f506c09fbd0 0x7f506c13ca50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f505c0098a0 tx=0x7f505c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:37:06.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:06.075+0000 7f5071ff3640 1 -- 172.21.15.12:0/1313861084 shutdown_connections 2024-09-11T01:37:06.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:06.075+0000 7f5071ff3640 1 --2- 172.21.15.12:0/1313861084 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f506c13d550 0x7f506c0913c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:06.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:06.075+0000 7f5071ff3640 1 --2- 172.21.15.12:0/1313861084 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f505803e1c0 0x7f5058040680 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:06.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:06.075+0000 7f5071ff3640 1 --2- 172.21.15.12:0/1313861084 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f506c0a2db0 0x7f506c13cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:06.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:06.075+0000 7f5071ff3640 1 --2- 172.21.15.12:0/1313861084 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f506c09fbd0 0x7f506c13ca50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:06.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:06.075+0000 7f5071ff3640 1 -- 172.21.15.12:0/1313861084 >> 172.21.15.12:0/1313861084 conn(0x7f506c0a3ea0 msgr2=0x7f506c0a00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:06.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:06.075+0000 7f5071ff3640 1 -- 172.21.15.12:0/1313861084 shutdown_connections 2024-09-11T01:37:06.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:06.075+0000 7f5071ff3640 1 -- 172.21.15.12:0/1313861084 wait complete. 2024-09-11T01:37:09.089 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:37:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.199+0000 7f0a0af7d640 10 monclient: get_monmap_and_config 2024-09-11T01:37:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.199+0000 7f0a0af7d640 10 monclient: build_initial_monmap 2024-09-11T01:37:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.199+0000 7f0a0af7d640 10 monclient: monmap: 2024-09-11T01:37:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:09.201 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:37:09.200679+0000 2024-09-11T01:37:09.202 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:37:09.200679+0000 2024-09-11T01:37:09.202 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:09.202 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:09.202 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:37:09.202 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:37:09.202 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:37:09.202 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:37:09.202 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:37:09.202 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:09.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.200+0000 7f0a0af7d640 1 Processor -- start 2024-09-11T01:37:09.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.200+0000 7f0a0af7d640 1 -- start start 2024-09-11T01:37:09.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.200+0000 7f0a0af7d640 10 monclient: init 2024-09-11T01:37:09.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a0af7d640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:09.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a0af7d640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-11T01:37:09.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a0af7d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0a04132b30 0x7f0a04132f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:09.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a0af7d640 10 monclient(hunting): picked mon.noname-a con 0x7f0a04132b30 addr v2:172.21.15.12:3300/0 2024-09-11T01:37:09.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a0af7d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0a04070310 addr v1:172.21.15.12:6789/0 2024-09-11T01:37:09.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a0af7d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0a04136160 0x7f0a0413ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:09.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a0af7d640 10 monclient(hunting): picked mon.noname-c con 0x7f0a04136160 addr v2:172.21.15.71:3300/0 2024-09-11T01:37:09.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a0af7d640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a040738f0 con 0x7f0a04070310 2024-09-11T01:37:09.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a0af7d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a0af7d640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f0a04073610 con 0x7f0a04132b30 2024-09-11T01:37:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a0af7d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a0af7d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f0a04073bd0 con 0x7f0a04136160 2024-09-11T01:37:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a0af7d640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a0af7d640 10 monclient(hunting): authenticate will time out at 1235.643423s 2024-09-11T01:37:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a08d19640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7f0a04132b30 msgr2=0x7f0a04132f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:37:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a03fff640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7f0a04070310 legacy=0x7f0a04135a50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:37:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a08d19640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0a04132b30 0x7f0a04132f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:37:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a0951a640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0a04136160 0x7f0a0413ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a0951a640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0a04136160 0x7f0a0413ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:52180/0 (socket says 172.21.15.12:52180) 2024-09-11T01:37:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a0951a640 1 -- 172.21.15.12:0/827971020 learned_addr learned my addr 172.21.15.12:0/827971020 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a0951a640 10 monclient(hunting): get_auth_request con 0x7f0a04136160 auth_method 0 2024-09-11T01:37:09.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a0951a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a0951a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a0951a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a0951a640 1 -- 172.21.15.12:0/827971020 >> v2:172.21.15.71:3300/0 conn(0x7f0a04136160 msgr2=0x7f0a0413ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:37:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a0951a640 1 -- 172.21.15.12:0/827971020 >> v2:172.21.15.71:3300/0 conn(0x7f0a04136160 msgr2=0x7f0a0413ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:37:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a0951a640 1 --2- 172.21.15.12:0/827971020 >> v2:172.21.15.71:3300/0 conn(0x7f0a04136160 0x7f0a0413ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:37:09.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.201+0000 7f0a0951a640 1 --2- 172.21.15.12:0/827971020 >> v2:172.21.15.71:3300/0 conn(0x7f0a04136160 0x7f0a0413ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:37:09.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.402+0000 7f0a0951a640 1 --2- 172.21.15.12:0/827971020 >> v2:172.21.15.71:3300/0 conn(0x7f0a04136160 0x7f0a0413ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:09.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.402+0000 7f0a0951a640 10 monclient(hunting): get_auth_request con 0x7f0a04136160 auth_method 0 2024-09-11T01:37:09.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.402+0000 7f0a0951a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:09.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.402+0000 7f0a0951a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:09.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.402+0000 7f0a0951a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:37:09.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.402+0000 7f0a0951a640 1 -- 172.21.15.12:0/827971020 >> v2:172.21.15.71:3300/0 conn(0x7f0a04136160 msgr2=0x7f0a0413ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:37:09.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.402+0000 7f0a0951a640 1 -- 172.21.15.12:0/827971020 >> v2:172.21.15.71:3300/0 conn(0x7f0a04136160 msgr2=0x7f0a0413ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:37:09.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.402+0000 7f0a0951a640 1 --2- 172.21.15.12:0/827971020 >> v2:172.21.15.71:3300/0 conn(0x7f0a04136160 0x7f0a0413ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:37:09.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.402+0000 7f0a0951a640 1 --2- 172.21.15.12:0/827971020 >> v2:172.21.15.71:3300/0 conn(0x7f0a04136160 0x7f0a0413ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:37:09.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.402+0000 7f0a03fff640 1 -- 172.21.15.12:0/827971020 >> v1:172.21.15.12:6789/0 conn(0x7f0a04070310 legacy=0x7f0a04135a50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:37:09.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.402+0000 7f0a08d19640 1 -- 172.21.15.12:0/827971020 >> v2:172.21.15.12:3300/0 conn(0x7f0a04132b30 msgr2=0x7f0a04132f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:37:09.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.402+0000 7f0a08d19640 1 --2- 172.21.15.12:0/827971020 >> v2:172.21.15.12:3300/0 conn(0x7f0a04132b30 0x7f0a04132f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:37:09.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.803+0000 7f0a03fff640 1 -- 172.21.15.12:0/827971020 >> v1:172.21.15.12:6789/0 conn(0x7f0a04070310 legacy=0x7f0a04135a50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:37:09.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.803+0000 7f0a08d19640 1 -- 172.21.15.12:0/827971020 >> v2:172.21.15.12:3300/0 conn(0x7f0a04132b30 msgr2=0x7f0a04132f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:37:09.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.803+0000 7f0a08d19640 1 --2- 172.21.15.12:0/827971020 >> v2:172.21.15.12:3300/0 conn(0x7f0a04132b30 0x7f0a04132f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:37:09.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.803+0000 7f0a0951a640 1 --2- 172.21.15.12:0/827971020 >> v2:172.21.15.71:3300/0 conn(0x7f0a04136160 0x7f0a0413ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:09.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.803+0000 7f0a0951a640 10 monclient(hunting): get_auth_request con 0x7f0a04136160 auth_method 0 2024-09-11T01:37:09.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.803+0000 7f0a0951a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:09.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.803+0000 7f0a0951a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:09.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.803+0000 7f0a0951a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:37:09.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.803+0000 7f0a0951a640 1 -- 172.21.15.12:0/827971020 >> v2:172.21.15.71:3300/0 conn(0x7f0a04136160 msgr2=0x7f0a0413ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:37:09.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.803+0000 7f0a0951a640 1 -- 172.21.15.12:0/827971020 >> v2:172.21.15.71:3300/0 conn(0x7f0a04136160 msgr2=0x7f0a0413ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:37:09.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.803+0000 7f0a0951a640 1 --2- 172.21.15.12:0/827971020 >> v2:172.21.15.71:3300/0 conn(0x7f0a04136160 0x7f0a0413ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:37:09.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:09.803+0000 7f0a0951a640 1 --2- 172.21.15.12:0/827971020 >> v2:172.21.15.71:3300/0 conn(0x7f0a04136160 0x7f0a0413ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:37:10.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:10.604+0000 7f0a03fff640 1 -- 172.21.15.12:0/827971020 >> v1:172.21.15.12:6789/0 conn(0x7f0a04070310 legacy=0x7f0a04135a50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:37:10.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:10.604+0000 7f0a08d19640 1 -- 172.21.15.12:0/827971020 >> v2:172.21.15.12:3300/0 conn(0x7f0a04132b30 msgr2=0x7f0a04132f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:37:10.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:10.604+0000 7f0a08d19640 1 --2- 172.21.15.12:0/827971020 >> v2:172.21.15.12:3300/0 conn(0x7f0a04132b30 0x7f0a04132f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:37:10.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:10.604+0000 7f0a0951a640 1 --2- 172.21.15.12:0/827971020 >> v2:172.21.15.71:3300/0 conn(0x7f0a04136160 0x7f0a0413ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:10.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:10.604+0000 7f0a0951a640 10 monclient(hunting): get_auth_request con 0x7f0a04136160 auth_method 0 2024-09-11T01:37:10.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:10.604+0000 7f0a0951a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:10.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:10.604+0000 7f0a0951a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:10.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:10.604+0000 7f0a0951a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:37:10.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:10.605+0000 7f0a0951a640 1 -- 172.21.15.12:0/827971020 >> v2:172.21.15.71:3300/0 conn(0x7f0a04136160 msgr2=0x7f0a0413ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:37:10.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:10.605+0000 7f0a0951a640 1 -- 172.21.15.12:0/827971020 >> v2:172.21.15.71:3300/0 conn(0x7f0a04136160 msgr2=0x7f0a0413ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:37:10.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:10.605+0000 7f0a0951a640 1 --2- 172.21.15.12:0/827971020 >> v2:172.21.15.71:3300/0 conn(0x7f0a04136160 0x7f0a0413ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:37:10.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:10.605+0000 7f0a0951a640 1 --2- 172.21.15.12:0/827971020 >> v2:172.21.15.71:3300/0 conn(0x7f0a04136160 0x7f0a0413ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:37:12.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.201+0000 7f0a027fc640 10 monclient(hunting): tick 2024-09-11T01:37:12.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.201+0000 7f0a027fc640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:37:12.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.201+0000 7f0a027fc640 1 monclient(hunting): continuing hunt 2024-09-11T01:37:12.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.201+0000 7f0a027fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:37:12.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.201+0000 7f0a027fc640 1 -- 172.21.15.12:0/827971020 >> v2:172.21.15.71:3300/0 conn(0x7f0a04136160 msgr2=0x7f0a0413ddf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:37:12.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.201+0000 7f0a027fc640 1 --2- 172.21.15.12:0/827971020 >> v2:172.21.15.71:3300/0 conn(0x7f0a04136160 0x7f0a0413ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:12.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.201+0000 7f0a027fc640 1 -- 172.21.15.12:0/827971020 >> v2:172.21.15.12:3300/0 conn(0x7f0a04132b30 msgr2=0x7f0a04132f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:37:12.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.201+0000 7f0a027fc640 1 --2- 172.21.15.12:0/827971020 >> v2:172.21.15.12:3300/0 conn(0x7f0a04132b30 0x7f0a04132f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:12.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.201+0000 7f0a027fc640 1 -- 172.21.15.12:0/827971020 >> v1:172.21.15.12:6789/0 conn(0x7f0a04070310 legacy=0x7f0a04135a50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:37:12.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.201+0000 7f0a027fc640 10 monclient: _add_conns ranks=[1,2] 2024-09-11T01:37:12.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.201+0000 7f0a027fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f09e8003670 addr v1:172.21.15.71:6789/0 2024-09-11T01:37:12.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.201+0000 7f0a027fc640 1 --2- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09e8007360 0x7f09e8009820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:12.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.201+0000 7f0a027fc640 10 monclient(hunting): picked mon.noname-b con 0x7f09e8007360 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:12.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.201+0000 7f0a027fc640 1 -- 172.21.15.12:0/827971020 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a040a43e0 con 0x7f09e8003670 2024-09-11T01:37:12.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.201+0000 7f0a027fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:12.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.201+0000 7f0a027fc640 1 -- 172.21.15.12:0/827971020 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f0a04073610 con 0x7f09e8007360 2024-09-11T01:37:12.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.201+0000 7f0a027fc640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:12.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.201+0000 7f0a03fff640 1 --2- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09e8007360 0x7f09e8009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:12.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.201+0000 7f0a03fff640 10 monclient(hunting): get_auth_request con 0x7f09e8007360 auth_method 0 2024-09-11T01:37:12.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.201+0000 7f0a03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:12.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.201+0000 7f0a03fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:12.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.201+0000 7f0a03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:12.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.202+0000 7f0a03fff640 1 -- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09e8007360 msgr2=0x7f09e8009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:37:12.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.202+0000 7f0a03fff640 1 -- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09e8007360 msgr2=0x7f09e8009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:37:12.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.202+0000 7f0a03fff640 1 --2- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09e8007360 0x7f09e8009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:37:12.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.202+0000 7f0a03fff640 1 --2- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09e8007360 0x7f09e8009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:37:12.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.204+0000 7f0a08d19640 1 -- 172.21.15.12:0/827971020 >> v1:172.21.15.71:6789/0 conn(0x7f09e8003670 legacy=0x7f09e8005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:12.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.403+0000 7f0a03fff640 1 --2- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09e8007360 0x7f09e8009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:12.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.403+0000 7f0a03fff640 10 monclient(hunting): get_auth_request con 0x7f09e8007360 auth_method 0 2024-09-11T01:37:12.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.403+0000 7f0a03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:12.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.403+0000 7f0a03fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:12.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.403+0000 7f0a03fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:37:12.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.403+0000 7f0a03fff640 1 -- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09e8007360 msgr2=0x7f09e8009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:37:12.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.403+0000 7f0a03fff640 1 -- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09e8007360 msgr2=0x7f09e8009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:37:12.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.403+0000 7f0a03fff640 1 --2- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09e8007360 0x7f09e8009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:37:12.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.403+0000 7f0a03fff640 1 --2- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09e8007360 0x7f09e8009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:37:12.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.804+0000 7f0a03fff640 1 --2- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09e8007360 0x7f09e8009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:12.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.804+0000 7f0a03fff640 10 monclient(hunting): get_auth_request con 0x7f09e8007360 auth_method 0 2024-09-11T01:37:12.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.804+0000 7f0a03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:12.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.804+0000 7f0a03fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:12.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.804+0000 7f0a03fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:37:12.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.804+0000 7f0a03fff640 1 -- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09e8007360 msgr2=0x7f09e8009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:37:12.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.804+0000 7f0a03fff640 1 -- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09e8007360 msgr2=0x7f09e8009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:37:12.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.804+0000 7f0a03fff640 1 --2- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09e8007360 0x7f09e8009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:37:12.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:12.804+0000 7f0a03fff640 1 --2- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09e8007360 0x7f09e8009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:37:13.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.189+0000 7f0a037fe640 1 -- 172.21.15.12:0/827971020 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 206388779 0 0) 0x7f0a040a43e0 con 0x7f09e8003670 2024-09-11T01:37:13.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.189+0000 7f0a037fe640 10 monclient(hunting): _negotiate 2024-09-11T01:37:13.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.189+0000 7f0a037fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:13.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.189+0000 7f0a037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:13.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.189+0000 7f0a037fe640 10 monclient(hunting): my global_id is 74239 2024-09-11T01:37:13.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.189+0000 7f0a037fe640 1 -- 172.21.15.12:0/827971020 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09f4003520 con 0x7f09e8003670 2024-09-11T01:37:13.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.192+0000 7f0a037fe640 1 -- 172.21.15.12:0/827971020 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 362122813 0 0) 0x7f09f4003520 con 0x7f09e8003670 2024-09-11T01:37:13.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.192+0000 7f0a037fe640 1 -- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09e8007360 msgr2=0x7f09e8009820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:37:13.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.192+0000 7f0a037fe640 1 --2- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09e8007360 0x7f09e8009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.192+0000 7f0a037fe640 10 monclient: _finish_hunting 0 2024-09-11T01:37:13.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.192+0000 7f0a037fe640 1 monclient: found mon.noname-c-legacy 2024-09-11T01:37:13.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.192+0000 7f0a037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:13.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.192+0000 7f0a037fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-09-11T01:37:13.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.192+0000 7f0a037fe640 1 -- 172.21.15.12:0/827971020 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a040738f0 con 0x7f09e8003670 2024-09-11T01:37:13.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.192+0000 7f0a037fe640 10 monclient: _finish_auth 0 2024-09-11T01:37:13.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.192+0000 7f0a037fe640 10 monclient: _check_auth_tickets 2024-09-11T01:37:13.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.192+0000 7f0a037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:13.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.192+0000 7f0a037fe640 1 -- 172.21.15.12:0/827971020 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f09f80043f0 con 0x7f09e8003670 2024-09-11T01:37:13.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.192+0000 7f0a037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:13.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.192+0000 7f0a037fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-11T01:37:13.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.192+0000 7f0a037fe640 10 monclient: dump: 2024-09-11T01:37:13.197 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:13.197 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:13.197 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.197 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.197 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:13.197 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:13.197 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:13.197 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:13.197 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:13.197 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:13.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.192+0000 7f0a037fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-09-11T01:37:13.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.192+0000 7f0a037fe640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:13.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.192+0000 7f0a037fe640 1 -- 172.21.15.12:0/827971020 >> v1:172.21.15.71:6789/0 conn(0x7f09e8003670 legacy=0x7f09e8005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:13.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.192+0000 7f0a037fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:37:13.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.192+0000 7f0a0951a640 1 -- 172.21.15.12:0/827971020 reap_dead start 2024-09-11T01:37:13.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.192+0000 7f0a037fe640 1 --2- 172.21.15.12:0/827971020 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09f40039b0 0x7f09f4003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:13.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.192+0000 7f0a037fe640 10 monclient(hunting): picked mon.b con 0x7f09f40039b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:37:13.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.192+0000 7f0a037fe640 1 --2- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09f4004800 0x7f09f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:13.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a037fe640 10 monclient(hunting): picked mon.a con 0x7f09f4004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:37:13.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a037fe640 1 --2- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09f40052e0 0x7f09f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:13.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a037fe640 10 monclient(hunting): picked mon.c con 0x7f09f40052e0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:13.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a037fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:13.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a037fe640 1 -- 172.21.15.12:0/827971020 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0a04073610 con 0x7f09f4004800 2024-09-11T01:37:13.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a037fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:13.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a037fe640 1 -- 172.21.15.12:0/827971020 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f09f400c190 con 0x7f09f40039b0 2024-09-11T01:37:13.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a037fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:13.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a037fe640 1 -- 172.21.15.12:0/827971020 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f09f400c390 con 0x7f09f40052e0 2024-09-11T01:37:13.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a037fe640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:13.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a037fe640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:37:13.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a03fff640 1 -- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09f4004800 msgr2=0x7f09f4004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:37:13.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a03fff640 1 --2- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09f4004800 0x7f09f4004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:37:13.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a08d19640 1 --2- 172.21.15.12:0/827971020 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09f40039b0 0x7f09f4003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:13.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a08d19640 10 monclient(hunting): get_auth_request con 0x7f09f40039b0 auth_method 0 2024-09-11T01:37:13.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a08d19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:13.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a08d19640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:13.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a08d19640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:37:13.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a0951a640 1 --2- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09f40052e0 0x7f09f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:13.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a0951a640 10 monclient(hunting): get_auth_request con 0x7f09f40052e0 auth_method 0 2024-09-11T01:37:13.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a0951a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:13.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a0951a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:13.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a0951a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:37:13.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a08d19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:13.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a08d19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:13.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a08d19640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:37:13.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a0951a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:13.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a0951a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:13.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a0951a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:37:13.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a08d19640 10 monclient(hunting): handle_auth_done global_id 74239 payload 293 2024-09-11T01:37:13.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a08d19640 1 -- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09f40052e0 msgr2=0x7f09f400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:13.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a08d19640 1 --2- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09f40052e0 0x7f09f400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a08d19640 1 -- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09f4004800 msgr2=0x7f09f4004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:37:13.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a08d19640 1 --2- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09f4004800 0x7f09f4004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a08d19640 10 monclient: _finish_hunting 0 2024-09-11T01:37:13.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a08d19640 1 monclient: found mon.b 2024-09-11T01:37:13.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a08d19640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:13.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a08d19640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:37:13.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a08d19640 1 -- 172.21.15.12:0/827971020 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f4003520 con 0x7f09f40039b0 2024-09-11T01:37:13.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a08d19640 10 monclient: _finish_auth 0 2024-09-11T01:37:13.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a08d19640 10 monclient: _check_auth_tickets 2024-09-11T01:37:13.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a08d19640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:13.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a0af7d640 5 monclient: authenticate success, global_id 74239 2024-09-11T01:37:13.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a0af7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:37:13.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.193+0000 7f0a0951a640 1 --2- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09f40052e0 0x7f09f400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:37:13.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.196+0000 7f0a08d19640 1 -- 172.21.15.12:0/827971020 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09f40039b0 msgr2=0x7f09f4003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.196+0000 7f0a08d19640 1 --2- 172.21.15.12:0/827971020 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09f40039b0 0x7f09f4003db0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f09f8004610 tx=0x7f09f4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77eef38f71b6993f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.197+0000 7f0a037fe640 1 -- 172.21.15.12:0/827971020 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f800c0e0 con 0x7f09f40039b0 2024-09-11T01:37:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.197+0000 7f0a037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.197+0000 7f0a037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:37:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.197+0000 7f0a037fe640 10 monclient: dump: 2024-09-11T01:37:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:13.208 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:13.209 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:13.209 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:13.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.197+0000 7f0a037fe640 1 -- 172.21.15.12:0/827971020 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09f8006070 con 0x7f09f40039b0 2024-09-11T01:37:13.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.197+0000 7f0a0af7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:37:13.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.197+0000 7f0a037fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:13.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.197+0000 7f0a0af7d640 10 monclient: get_monmap_and_config success 2024-09-11T01:37:13.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.197+0000 7f0a0af7d640 4 set_mon_vals no callback set 2024-09-11T01:37:13.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.197+0000 7f0a037fe640 1 -- 172.21.15.12:0/827971020 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f8006440 con 0x7f09f40039b0 2024-09-11T01:37:13.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.197+0000 7f0a037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:13.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.197+0000 7f0a037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:37:13.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.197+0000 7f0a037fe640 10 monclient: dump: 2024-09-11T01:37:13.209 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:13.209 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:13.210 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.210 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.210 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:13.210 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:13.210 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:13.210 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:13.210 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:13.210 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:13.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.197+0000 7f0a0af7d640 10 monclient: shutdown 2024-09-11T01:37:13.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.197+0000 7f0a0af7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:13.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.197+0000 7f0a0af7d640 1 -- 172.21.15.12:0/827971020 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09f40039b0 msgr2=0x7f09f4003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:13.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.197+0000 7f0a0af7d640 1 --2- 172.21.15.12:0/827971020 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09f40039b0 0x7f09f4003db0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f09f8004610 tx=0x7f09f4005830 comp rx=0 tx=0).stop 2024-09-11T01:37:13.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.197+0000 7f0a0af7d640 1 -- 172.21.15.12:0/827971020 shutdown_connections 2024-09-11T01:37:13.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.197+0000 7f0a0af7d640 1 --2- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09f40052e0 0x7f09f400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.197+0000 7f0a0af7d640 1 --2- 172.21.15.12:0/827971020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09f4004800 0x7f09f4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.197+0000 7f0a0af7d640 1 --2- 172.21.15.12:0/827971020 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09f40039b0 0x7f09f4003db0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.197+0000 7f0a0af7d640 1 -- 172.21.15.12:0/827971020 >> 172.21.15.12:0/827971020 conn(0x7f0a040a3650 msgr2=0x7f0a040a3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.198+0000 7f0a0af7d640 1 -- 172.21.15.12:0/827971020 shutdown_connections 2024-09-11T01:37:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.198+0000 7f0a0af7d640 1 -- 172.21.15.12:0/827971020 wait complete. 2024-09-11T01:37:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.198+0000 7f0a0af7d640 10 monclient: build_initial_monmap 2024-09-11T01:37:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.198+0000 7f0a0af7d640 10 monclient: monmap: 2024-09-11T01:37:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:37:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:37:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:37:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:13.211 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:37:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:37:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:37:13.212 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.199+0000 7f0a0af7d640 1 Processor -- start 2024-09-11T01:37:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.199+0000 7f0a0af7d640 1 -- start start 2024-09-11T01:37:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.199+0000 7f0a0af7d640 10 monclient: init 2024-09-11T01:37:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0af7d640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0af7d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:37:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0af7d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09f40039b0 0x7f0a041ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0af7d640 10 monclient(hunting): picked mon.noname-b con 0x7f09f40039b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:37:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0af7d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09f4004800 0x7f0a0412f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0af7d640 10 monclient(hunting): picked mon.noname-a con 0x7f09f4004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:37:13.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0af7d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09f40052e0 0x7f0a0412f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:13.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0af7d640 10 monclient(hunting): picked mon.noname-c con 0x7f09f40052e0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:13.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0af7d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:13.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0af7d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0a041365b0 con 0x7f09f4004800 2024-09-11T01:37:13.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0af7d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:13.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0af7d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0a04136730 con 0x7f09f40039b0 2024-09-11T01:37:13.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0af7d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:13.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0af7d640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f0a041368b0 con 0x7f09f40052e0 2024-09-11T01:37:13.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0af7d640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:13.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0af7d640 10 monclient(hunting): authenticate will time out at 1239.642233s 2024-09-11T01:37:13.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a03fff640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09f4004800 msgr2=0x7f0a0412f2e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:37:13.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a03fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09f4004800 0x7f0a0412f2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:37:13.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0951a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09f40052e0 0x7f0a0412f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:13.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0951a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09f40052e0 0x7f0a0412f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41348/0 (socket says 172.21.15.12:41348) 2024-09-11T01:37:13.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0951a640 1 -- 172.21.15.12:0/1625163643 learned_addr learned my addr 172.21.15.12:0/1625163643 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:13.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0951a640 10 monclient(hunting): get_auth_request con 0x7f09f40052e0 auth_method 0 2024-09-11T01:37:13.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0951a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:13.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0951a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:13.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0951a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:13.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0951a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:13.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0951a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:13.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0951a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:13.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a08d19640 1 --2- 172.21.15.12:0/1625163643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09f40039b0 0x7f0a041ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:13.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a08d19640 10 monclient(hunting): get_auth_request con 0x7f09f40039b0 auth_method 0 2024-09-11T01:37:13.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a08d19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:13.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a08d19640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:13.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a08d19640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:13.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0951a640 10 monclient(hunting): handle_auth_done global_id 84164 payload 824 2024-09-11T01:37:13.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0951a640 1 -- 172.21.15.12:0/1625163643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09f40039b0 msgr2=0x7f0a041ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:13.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0951a640 1 --2- 172.21.15.12:0/1625163643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09f40039b0 0x7f0a041ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0951a640 1 -- 172.21.15.12:0/1625163643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09f4004800 msgr2=0x7f0a0412f2e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:37:13.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0951a640 1 --2- 172.21.15.12:0/1625163643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09f4004800 0x7f0a0412f2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0951a640 10 monclient: _finish_hunting 0 2024-09-11T01:37:13.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0951a640 1 monclient: found mon.noname-c 2024-09-11T01:37:13.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0951a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:13.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0951a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:37:13.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.200+0000 7f0a0951a640 1 -- 172.21.15.12:0/1625163643 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a041c0e30 con 0x7f09f40052e0 2024-09-11T01:37:13.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.203+0000 7f0a0951a640 1 -- 172.21.15.12:0/1625163643 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09f40052e0 msgr2=0x7f0a0412f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:13.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.203+0000 7f0a0951a640 1 --2- 172.21.15.12:0/1625163643 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09f40052e0 0x7f0a0412f820 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f09e8007330 tx=0x7f09fc013680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:13.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.203+0000 7f0a017fa640 1 -- 172.21.15.12:0/1625163643 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09fc01c070 con 0x7f09f40052e0 2024-09-11T01:37:13.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.203+0000 7f0a017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:13.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.203+0000 7f0a017fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:37:13.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.203+0000 7f0a017fa640 10 monclient: dump: 2024-09-11T01:37:13.216 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:13.216 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:13.216 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.216 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.216 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:13.216 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:13.216 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:13.216 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:13.216 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:13.216 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:13.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.203+0000 7f0a017fa640 10 monclient: _finish_auth 0 2024-09-11T01:37:13.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.203+0000 7f0a017fa640 10 monclient: _check_auth_tickets 2024-09-11T01:37:13.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.203+0000 7f0a017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:13.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.203+0000 7f0a0af7d640 5 monclient: authenticate success, global_id 84164 2024-09-11T01:37:13.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.203+0000 7f0a017fa640 1 -- 172.21.15.12:0/1625163643 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09fc003990 con 0x7f09f40052e0 2024-09-11T01:37:13.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.203+0000 7f0a0af7d640 10 monclient: _renew_subs 2024-09-11T01:37:13.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.203+0000 7f0a0af7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:13.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.203+0000 7f0a0af7d640 1 -- 172.21.15.12:0/1625163643 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a041c1110 con 0x7f09f40052e0 2024-09-11T01:37:13.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.203+0000 7f0a017fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:13.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.203+0000 7f0a017fa640 1 -- 172.21.15.12:0/1625163643 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09fc003d60 con 0x7f09f40052e0 2024-09-11T01:37:13.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.203+0000 7f0a017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:13.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.203+0000 7f0a017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:13.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.203+0000 7f0a017fa640 10 monclient: dump: 2024-09-11T01:37:13.217 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:13.217 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:13.218 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.218 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.218 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:13.218 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:13.218 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:13.218 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:13.218 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:13.218 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:13.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.203+0000 7f0a0af7d640 10 monclient: _renew_subs 2024-09-11T01:37:13.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.203+0000 7f0a02ffd640 4 set_mon_vals no callback set 2024-09-11T01:37:13.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.203+0000 7f0a0af7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:13.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.203+0000 7f0a0af7d640 1 -- 172.21.15.12:0/1625163643 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a041bf090 con 0x7f09f40052e0 2024-09-11T01:37:13.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.204+0000 7f0a017fa640 1 -- 172.21.15.12:0/1625163643 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f09fc029060 con 0x7f09f40052e0 2024-09-11T01:37:13.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.204+0000 7f0a0af7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:13.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.204+0000 7f0a0af7d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:13.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.204+0000 7f0a0af7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:13.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.204+0000 7f0a0af7d640 1 -- 172.21.15.12:0/1625163643 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a04132f10 con 0x7f09f40052e0 2024-09-11T01:37:13.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.204+0000 7f0a017fa640 1 --2- 172.21.15.12:0/1625163643 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f09e803ea10 0x7f09e8040ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:13.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.204+0000 7f0a017fa640 1 -- 172.21.15.12:0/1625163643 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(258..258 src has 186..258) ==== 4875+0+0 (secure 0 0 0) 0x7f09fc0539a0 con 0x7f09f40052e0 2024-09-11T01:37:13.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.205+0000 7f0a08d19640 1 --2- 172.21.15.12:0/1625163643 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f09e803ea10 0x7f09e8040ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:13.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.205+0000 7f0a08d19640 10 monclient: get_auth_request con 0x7f09e803ea10 auth_method 0 2024-09-11T01:37:13.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.207+0000 7f0a08d19640 1 --2- 172.21.15.12:0/1625163643 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f09e803ea10 0x7f09e8040ed0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f09f8030000 tx=0x7f09f802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:13.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.209+0000 7f0a017fa640 1 -- 172.21.15.12:0/1625163643 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f09fc0203b0 con 0x7f09f40052e0 2024-09-11T01:37:13.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.210+0000 7f0a017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:13.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.210+0000 7f0a017fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:37:13.246 INFO:tasks.rados.rados.0.smithi071.stdout:1115: done (5 left) 2024-09-11T01:37:13.246 INFO:tasks.rados.rados.0.smithi071.stdout:1116: done (4 left) 2024-09-11T01:37:13.247 INFO:tasks.rados.rados.0.smithi071.stdout:1117: done (3 left) 2024-09-11T01:37:13.247 INFO:tasks.rados.rados.0.smithi071.stdout:1118: done (2 left) 2024-09-11T01:37:13.247 INFO:tasks.rados.rados.0.smithi071.stdout:1119: done (1 left) 2024-09-11T01:37:13.247 INFO:tasks.rados.rados.0.smithi071.stdout:1121: done (0 left) 2024-09-11T01:37:13.247 INFO:tasks.rados.rados.0.smithi071.stdout:1122: delete oid 10 current snap is 118 2024-09-11T01:37:13.249 INFO:tasks.rados.rados.0.smithi071.stdout:1122: done (0 left) 2024-09-11T01:37:13.249 INFO:tasks.rados.rados.0.smithi071.stdout:1123: write (excl) oid 42 current snap is 118 2024-09-11T01:37:13.249 INFO:tasks.rados.rados.0.smithi071.stdout:1123: seq_num 423 ranges {709361=629742,1836589=539816,3033220=76141} 2024-09-11T01:37:13.256 INFO:tasks.rados.rados.0.smithi071.stdout:1123: writing smithi07134777-42 from 709361 to 1339103 tid 1 2024-09-11T01:37:13.263 INFO:tasks.rados.rados.0.smithi071.stdout:1123: writing smithi07134777-42 from 1836589 to 2376405 tid 2 2024-09-11T01:37:13.266 INFO:tasks.rados.rados.0.smithi071.stdout:1123: writing smithi07134777-42 from 3033220 to 3109361 tid 3 2024-09-11T01:37:13.266 INFO:tasks.rados.rados.0.smithi071.stdout:1124: delete oid 7 current snap is 118 2024-09-11T01:37:13.266 INFO:tasks.rados.rados.0.smithi071.stdout:1123: finishing write tid 1 to smithi07134777-42 2024-09-11T01:37:13.266 INFO:tasks.rados.rados.0.smithi071.stdout:1123: finishing write tid 2 to smithi07134777-42 2024-09-11T01:37:13.268 INFO:tasks.rados.rados.0.smithi071.stdout:1123: finishing write tid 3 to smithi07134777-42 2024-09-11T01:37:13.270 INFO:tasks.rados.rados.0.smithi071.stdout:1123: finishing write tid 4 to smithi07134777-42 2024-09-11T01:37:13.271 INFO:tasks.rados.rados.0.smithi071.stdout:1123: finishing write tid 5 to smithi07134777-42 2024-09-11T01:37:13.271 INFO:tasks.rados.rados.0.smithi071.stdout:1123: oid 42 updating version 0 to 409 2024-09-11T01:37:13.271 INFO:tasks.rados.rados.0.smithi071.stdout:1123: oid 42 updating version 409 to 410 2024-09-11T01:37:13.271 INFO:tasks.rados.rados.0.smithi071.stdout:1123: oid 42 version 410 is already newer than 408 2024-09-11T01:37:13.272 INFO:tasks.rados.rados.0.smithi071.stdout:1123: oid 42 updating version 410 to 411 2024-09-11T01:37:13.272 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 411 (ObjNum 423 snap 118 seq_num 423) dirty exists 2024-09-11T01:37:13.272 INFO:tasks.rados.rados.0.smithi071.stdout:1123: left oid 42 (ObjNum 423 snap 118 seq_num 423) 2024-09-11T01:37:13.274 INFO:tasks.rados.rados.0.smithi071.stdout:1123: done (1 left) 2024-09-11T01:37:13.274 INFO:tasks.rados.rados.0.smithi071.stdout:1124: done (0 left) 2024-09-11T01:37:13.274 INFO:tasks.rados.rados.0.smithi071.stdout:1125: read oid 33 snap -1 2024-09-11T01:37:13.274 INFO:tasks.rados.rados.0.smithi071.stdout:1125: expect (ObjNum 334 snap 96 seq_num 334) 2024-09-11T01:37:13.274 INFO:tasks.rados.rados.0.smithi071.stdout:1126: write oid 12 current snap is 118 2024-09-11T01:37:13.274 INFO:tasks.rados.rados.0.smithi071.stdout:1126: seq_num 425 ranges {742975=780240,1953553=427132,2971713=171262} 2024-09-11T01:37:13.283 INFO:tasks.rados.rados.0.smithi071.stdout:1126: writing smithi07134777-12 from 742975 to 1523215 tid 1 2024-09-11T01:37:13.288 INFO:tasks.rados.rados.0.smithi071.stdout:1126: writing smithi07134777-12 from 1953553 to 2380685 tid 2 2024-09-11T01:37:13.292 INFO:tasks.rados.rados.0.smithi071.stdout:1126: writing smithi07134777-12 from 2971713 to 3142975 tid 3 2024-09-11T01:37:13.292 INFO:tasks.rados.rados.0.smithi071.stdout:1127: copy_from oid 42 from oid 3 current snap is 118 2024-09-11T01:37:13.292 INFO:tasks.rados.rados.0.smithi071.stdout:1125: expect (ObjNum 334 snap 96 seq_num 334) 2024-09-11T01:37:13.346 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:37:13.412 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1584424 2024-09-11T01:37:13.445 INFO:tasks.rados.rados.0.smithi071.stdout:1126: finishing write tid 1 to smithi07134777-12 2024-09-11T01:37:13.445 INFO:tasks.rados.rados.0.smithi071.stdout:1126: finishing write tid 2 to smithi07134777-12 2024-09-11T01:37:13.445 INFO:tasks.rados.rados.0.smithi071.stdout:1126: finishing write tid 3 to smithi07134777-12 2024-09-11T01:37:13.446 INFO:tasks.rados.rados.0.smithi071.stdout:1126: finishing write tid 4 to smithi07134777-12 2024-09-11T01:37:13.446 INFO:tasks.rados.rados.0.smithi071.stdout:1126: finishing write tid 5 to smithi07134777-12 2024-09-11T01:37:13.446 INFO:tasks.rados.rados.0.smithi071.stdout:1126: oid 12 updating version 0 to 503 2024-09-11T01:37:13.446 INFO:tasks.rados.rados.0.smithi071.stdout:1126: oid 12 updating version 503 to 504 2024-09-11T01:37:13.446 INFO:tasks.rados.rados.0.smithi071.stdout:1126: oid 12 version 504 is already newer than 502 2024-09-11T01:37:13.446 INFO:tasks.rados.rados.0.smithi071.stdout:1126: oid 12 updating version 504 to 505 2024-09-11T01:37:13.446 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 505 (ObjNum 425 snap 118 seq_num 425) dirty exists 2024-09-11T01:37:13.446 INFO:tasks.rados.rados.0.smithi071.stdout:1126: left oid 12 (ObjNum 425 snap 118 seq_num 425) 2024-09-11T01:37:13.446 INFO:tasks.rados.rados.0.smithi071.stdout:1127: finishing copy_from racing read to smithi07134777-42 2024-09-11T01:37:13.446 INFO:tasks.rados.rados.0.smithi071.stdout:1127: finishing copy_from to smithi07134777-42 2024-09-11T01:37:13.446 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 243 (ObjNum 383 snap 111 seq_num 383) dirty exists 2024-09-11T01:37:13.446 INFO:tasks.rados.rados.0.smithi071.stdout:1125: done (2 left) 2024-09-11T01:37:13.446 INFO:tasks.rados.rados.0.smithi071.stdout:1126: done (1 left) 2024-09-11T01:37:13.446 INFO:tasks.rados.rados.0.smithi071.stdout:1127: done (0 left) 2024-09-11T01:37:13.447 INFO:tasks.rados.rados.0.smithi071.stdout:1128: write oid 22 current snap is 118 2024-09-11T01:37:13.447 INFO:tasks.rados.rados.0.smithi071.stdout:1128: seq_num 426 ranges {759782=571842,2012035=687143} 2024-09-11T01:37:13.452 INFO:tasks.rados.rados.0.smithi071.stdout:1128: writing smithi07134777-22 from 759782 to 1331624 tid 1 2024-09-11T01:37:13.460 INFO:tasks.rados.rados.0.smithi071.stdout:1128: writing smithi07134777-22 from 2012035 to 2699178 tid 2 2024-09-11T01:37:13.460 INFO:tasks.rados.rados.0.smithi071.stdout:1129: write (excl) oid 10 current snap is 118 2024-09-11T01:37:13.460 INFO:tasks.rados.rados.0.smithi071.stdout:1129: seq_num 427 ranges {776589=647091,2070517=430801,3026559=150030} 2024-09-11T01:37:13.468 INFO:tasks.rados.rados.0.smithi071.stdout:1129: writing smithi07134777-10 from 776589 to 1423680 tid 1 2024-09-11T01:37:13.473 INFO:tasks.rados.rados.0.smithi071.stdout:1129: writing smithi07134777-10 from 2070517 to 2501318 tid 2 2024-09-11T01:37:13.478 INFO:tasks.rados.rados.0.smithi071.stdout:1129: writing smithi07134777-10 from 3026559 to 3176589 tid 3 2024-09-11T01:37:13.478 INFO:tasks.rados.rados.0.smithi071.stdout:1130: rollback oid 1 current snap is 118 2024-09-11T01:37:13.478 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 1 to 106 2024-09-11T01:37:13.478 INFO:tasks.rados.rados.0.smithi071.stdout:1128: finishing write tid 1 to smithi07134777-22 2024-09-11T01:37:13.478 INFO:tasks.rados.rados.0.smithi071.stdout:1128: finishing write tid 2 to smithi07134777-22 2024-09-11T01:37:13.479 INFO:tasks.rados.rados.0.smithi071.stdout:1128: finishing write tid 3 to smithi07134777-22 2024-09-11T01:37:13.479 INFO:tasks.rados.rados.0.smithi071.stdout:1128: finishing write tid 4 to smithi07134777-22 2024-09-11T01:37:13.479 INFO:tasks.rados.rados.0.smithi071.stdout:1128: oid 22 updating version 0 to 250 2024-09-11T01:37:13.479 INFO:tasks.rados.rados.0.smithi071.stdout:1128: oid 22 version 250 is already newer than 248 2024-09-11T01:37:13.479 INFO:tasks.rados.rados.0.smithi071.stdout:1128: oid 22 version 250 is already newer than 249 2024-09-11T01:37:13.479 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 250 (ObjNum 426 snap 118 seq_num 426) dirty exists 2024-09-11T01:37:13.479 INFO:tasks.rados.rados.0.smithi071.stdout:1128: left oid 22 (ObjNum 426 snap 118 seq_num 426) 2024-09-11T01:37:13.479 INFO:tasks.rados.rados.0.smithi071.stdout:1129: finishing write tid 1 to smithi07134777-10 2024-09-11T01:37:13.479 INFO:tasks.rados.rados.0.smithi071.stdout:1129: finishing write tid 2 to smithi07134777-10 2024-09-11T01:37:13.480 INFO:tasks.rados.rados.0.smithi071.stdout:1128: done (2 left) 2024-09-11T01:37:13.480 INFO:tasks.rados.rados.0.smithi071.stdout:1131: copy_from oid 31 from oid 17 current snap is 118 2024-09-11T01:37:13.480 INFO:tasks.rados.rados.0.smithi071.stdout:1132: copy_from oid 5 from oid 15 current snap is 118 2024-09-11T01:37:13.480 INFO:tasks.rados.rados.0.smithi071.stdout:1133: write (excl) oid 26 current snap is 118 2024-09-11T01:37:13.480 INFO:tasks.rados.rados.0.smithi071.stdout:1133: seq_num 428 ranges {793396=722340,2128999=690812} 2024-09-11T01:37:13.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.478+0000 7f0a0af7d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:37:13.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.478+0000 7f0a0af7d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:37:13.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.478+0000 7f0a0af7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:13.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.478+0000 7f0a0af7d640 1 -- 172.21.15.12:0/1625163643 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0a040700b0 con 0x7f09f40052e0 2024-09-11T01:37:13.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.479+0000 7f0a017fa640 1 -- 172.21.15.12:0/1625163643 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f09fc052340 con 0x7f09f40052e0 2024-09-11T01:37:13.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.479+0000 7f0a017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:37:13.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.479+0000 7f0a017fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:37:13.481 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":84,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:37:13.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.481+0000 7f0a0af7d640 1 -- 172.21.15.12:0/1625163643 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f09e803ea10 msgr2=0x7f09e8040ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:13.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.481+0000 7f0a0af7d640 1 --2- 172.21.15.12:0/1625163643 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f09e803ea10 0x7f09e8040ed0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f09f8030000 tx=0x7f09f802e000 comp rx=0 tx=0).stop 2024-09-11T01:37:13.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.481+0000 7f0a0af7d640 10 monclient: shutdown 2024-09-11T01:37:13.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.481+0000 7f0a0af7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:13.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.481+0000 7f0a0af7d640 1 -- 172.21.15.12:0/1625163643 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09f40052e0 msgr2=0x7f0a0412f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:13.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.481+0000 7f0a0af7d640 1 --2- 172.21.15.12:0/1625163643 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09f40052e0 0x7f0a0412f820 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f09e8007330 tx=0x7f09fc013680 comp rx=0 tx=0).stop 2024-09-11T01:37:13.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.481+0000 7f0a0af7d640 1 -- 172.21.15.12:0/1625163643 shutdown_connections 2024-09-11T01:37:13.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.481+0000 7f0a0af7d640 1 --2- 172.21.15.12:0/1625163643 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09f40052e0 0x7f0a0412f820 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.481+0000 7f0a0af7d640 1 --2- 172.21.15.12:0/1625163643 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f09e803ea10 0x7f09e8040ed0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.481+0000 7f0a0af7d640 1 --2- 172.21.15.12:0/1625163643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09f4004800 0x7f0a0412f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.481+0000 7f0a0af7d640 1 --2- 172.21.15.12:0/1625163643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09f40039b0 0x7f0a041ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.481+0000 7f0a0af7d640 1 -- 172.21.15.12:0/1625163643 >> 172.21.15.12:0/1625163643 conn(0x7f0a040a3650 msgr2=0x7f0a041303d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:13.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.481+0000 7f0a0af7d640 1 -- 172.21.15.12:0/1625163643 shutdown_connections 2024-09-11T01:37:13.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.481+0000 7f0a0af7d640 1 -- 172.21.15.12:0/1625163643 wait complete. 2024-09-11T01:37:13.486 INFO:tasks.rados.rados.0.smithi071.stdout:1133: writing smithi07134777-26 from 793396 to 1515736 tid 1 2024-09-11T01:37:13.493 INFO:tasks.rados.rados.0.smithi071.stdout:1133: writing smithi07134777-26 from 2128999 to 2819811 tid 2 2024-09-11T01:37:13.494 INFO:tasks.rados.rados.0.smithi071.stdout:1134: snap_remove snap 117 2024-09-11T01:37:13.496 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:37:13.496 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T01:37:13.496 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:37:13.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.597+0000 7f486192b640 10 monclient: get_monmap_and_config 2024-09-11T01:37:13.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.597+0000 7f486192b640 10 monclient: build_initial_monmap 2024-09-11T01:37:13.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.597+0000 7f486192b640 10 monclient: monmap: 2024-09-11T01:37:13.600 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:13.600 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:13.600 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:37:13.598645+0000 2024-09-11T01:37:13.600 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:37:13.598645+0000 2024-09-11T01:37:13.600 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:13.600 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:13.601 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:37:13.601 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:37:13.601 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:37:13.601 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:37:13.601 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:37:13.601 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:13.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.598+0000 7f486192b640 1 Processor -- start 2024-09-11T01:37:13.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.598+0000 7f486192b640 1 -- start start 2024-09-11T01:37:13.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.598+0000 7f486192b640 10 monclient: init 2024-09-11T01:37:13.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.598+0000 7f486192b640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:13.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f486192b640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-11T01:37:13.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f486192b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f485c134ad0 0x7f485c134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:13.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f486192b640 10 monclient(hunting): picked mon.noname-c con 0x7f485c134ad0 addr v2:172.21.15.71:3300/0 2024-09-11T01:37:13.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f486192b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f485c0a2200 addr v1:172.21.15.71:6789/0 2024-09-11T01:37:13.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f486192b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c1353f0 0x7f485c134160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:13.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f486192b640 10 monclient(hunting): picked mon.noname-b con 0x7f485c1353f0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:13.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f486192b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f485c09ed30 con 0x7f485c0a2200 2024-09-11T01:37:13.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f486192b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:13.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f486192b640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f485c09f5d0 con 0x7f485c134ad0 2024-09-11T01:37:13.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f486192b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:13.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f486192b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f485c09f010 con 0x7f485c1353f0 2024-09-11T01:37:13.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f486192b640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:13.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f486192b640 10 monclient(hunting): authenticate will time out at 1240.041101s 2024-09-11T01:37:13.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c1353f0 0x7f485c134160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:13.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c1353f0 0x7f485c134160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41350/0 (socket says 172.21.15.12:41350) 2024-09-11T01:37:13.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485b7fe640 1 -- 172.21.15.12:0/3050585862 learned_addr learned my addr 172.21.15.12:0/3050585862 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:13.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485b7fe640 10 monclient(hunting): get_auth_request con 0x7f485c1353f0 auth_method 0 2024-09-11T01:37:13.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:13.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:13.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:13.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:13.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:13.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:13.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485affd640 1 --2- 172.21.15.12:0/3050585862 >> v2:172.21.15.71:3300/0 conn(0x7f485c134ad0 0x7f485c134eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:13.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485affd640 10 monclient(hunting): get_auth_request con 0x7f485c134ad0 auth_method 0 2024-09-11T01:37:13.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:13.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:13.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:13.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:13.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:13.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:13.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485b7fe640 10 monclient(hunting): handle_auth_done global_id 84167 payload 274 2024-09-11T01:37:13.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485b7fe640 1 -- 172.21.15.12:0/3050585862 >> v2:172.21.15.71:3300/0 conn(0x7f485c134ad0 msgr2=0x7f485c134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:13.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485b7fe640 1 --2- 172.21.15.12:0/3050585862 >> v2:172.21.15.71:3300/0 conn(0x7f485c134ad0 0x7f485c134eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485b7fe640 1 -- 172.21.15.12:0/3050585862 >> v1:172.21.15.71:6789/0 conn(0x7f485c0a2200 legacy=0x7f485c133a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:13.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:37:13.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485b7fe640 1 monclient: found mon.noname-b 2024-09-11T01:37:13.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:13.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:37:13.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.599+0000 7f485b7fe640 1 -- 172.21.15.12:0/3050585862 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f485c09f2f0 con 0x7f485c1353f0 2024-09-11T01:37:13.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.602+0000 7f485b7fe640 1 -- 172.21.15.12:0/3050585862 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c1353f0 msgr2=0x7f485c134160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:13.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.602+0000 7f485b7fe640 1 --2- 172.21.15.12:0/3050585862 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c1353f0 0x7f485c134160 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4854009e80 tx=0x7f485402a580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2768788a59e2219c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:13.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.602+0000 7f4859ffb640 1 -- 172.21.15.12:0/3050585862 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4854036070 con 0x7f485c1353f0 2024-09-11T01:37:13.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.602+0000 7f4859ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:13.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.602+0000 7f4859ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:37:13.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.602+0000 7f4859ffb640 10 monclient: dump: 2024-09-11T01:37:13.612 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:13.612 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:13.613 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.613 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.613 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:13.613 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:13.613 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:13.613 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:13.613 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:13.614 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:13.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.602+0000 7f4859ffb640 10 monclient: _finish_auth 0 2024-09-11T01:37:13.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.602+0000 7f4859ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:37:13.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.602+0000 7f4859ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:13.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.602+0000 7f486192b640 5 monclient: authenticate success, global_id 84167 2024-09-11T01:37:13.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.602+0000 7f486192b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:37:13.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.602+0000 7f4859ffb640 1 -- 172.21.15.12:0/3050585862 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4854007800 con 0x7f485c1353f0 2024-09-11T01:37:13.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.602+0000 7f4859ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:13.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.602+0000 7f486192b640 10 monclient: get_monmap_and_config success 2024-09-11T01:37:13.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.602+0000 7f486192b640 4 set_mon_vals no callback set 2024-09-11T01:37:13.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.602+0000 7f4859ffb640 1 -- 172.21.15.12:0/3050585862 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4854007bd0 con 0x7f485c1353f0 2024-09-11T01:37:13.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.602+0000 7f4859ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:13.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.602+0000 7f4859ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:13.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.602+0000 7f4859ffb640 10 monclient: dump: 2024-09-11T01:37:13.615 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:13.615 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:13.615 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.615 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.615 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:13.615 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:13.615 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:13.615 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:13.615 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:13.616 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:13.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.602+0000 7f486192b640 10 monclient: shutdown 2024-09-11T01:37:13.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.602+0000 7f486192b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:13.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.602+0000 7f486192b640 1 -- 172.21.15.12:0/3050585862 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c1353f0 msgr2=0x7f485c134160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:13.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.602+0000 7f486192b640 1 --2- 172.21.15.12:0/3050585862 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c1353f0 0x7f485c134160 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4854009e80 tx=0x7f485402a580 comp rx=0 tx=0).stop 2024-09-11T01:37:13.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.603+0000 7f486192b640 1 -- 172.21.15.12:0/3050585862 shutdown_connections 2024-09-11T01:37:13.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.603+0000 7f486192b640 1 --2- 172.21.15.12:0/3050585862 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c1353f0 0x7f485c134160 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.603+0000 7f486192b640 1 --2- 172.21.15.12:0/3050585862 >> v2:172.21.15.71:3300/0 conn(0x7f485c134ad0 0x7f485c134eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.603+0000 7f486192b640 1 -- 172.21.15.12:0/3050585862 >> 172.21.15.12:0/3050585862 conn(0x7f485c0a2f70 msgr2=0x7f485c0a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:13.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.603+0000 7f486192b640 1 -- 172.21.15.12:0/3050585862 shutdown_connections 2024-09-11T01:37:13.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.603+0000 7f486192b640 1 -- 172.21.15.12:0/3050585862 wait complete. 2024-09-11T01:37:13.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.603+0000 7f486192b640 10 monclient: build_initial_monmap 2024-09-11T01:37:13.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.603+0000 7f486192b640 10 monclient: monmap: 2024-09-11T01:37:13.617 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:13.617 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:37:13.617 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:37:13.617 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:37:13.617 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:13.617 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:13.617 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:37:13.617 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:37:13.617 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:37:13.617 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:13.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.603+0000 7f486192b640 1 Processor -- start 2024-09-11T01:37:13.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f486192b640 1 -- start start 2024-09-11T01:37:13.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f486192b640 10 monclient: init 2024-09-11T01:37:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f486192b640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f486192b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:37:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f486192b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f485c0a2200 0x7f485c131d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f486192b640 10 monclient(hunting): picked mon.noname-a con 0x7f485c0a2200 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:37:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f486192b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f485c134ad0 0x7f485c132250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f486192b640 10 monclient(hunting): picked mon.noname-b con 0x7f485c134ad0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:37:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f486192b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c1353f0 0x7f485c11e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f486192b640 10 monclient(hunting): picked mon.noname-c con 0x7f485c1353f0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f486192b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f486192b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f485c134820 con 0x7f485c0a2200 2024-09-11T01:37:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f486192b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f486192b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f485c1346a0 con 0x7f485c134ad0 2024-09-11T01:37:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f486192b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:13.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f486192b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f485c09ffd0 con 0x7f485c1353f0 2024-09-11T01:37:13.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f486192b640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:13.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f486192b640 10 monclient(hunting): authenticate will time out at 1240.046618s 2024-09-11T01:37:13.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f485affd640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f485c0a2200 msgr2=0x7f485c131d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:37:13.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f485affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f485c0a2200 0x7f485c131d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:37:13.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f485b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c1353f0 0x7f485c11e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:13.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f485b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c1353f0 0x7f485c11e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41360/0 (socket says 172.21.15.12:41360) 2024-09-11T01:37:13.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f485b7fe640 1 -- 172.21.15.12:0/1562551125 learned_addr learned my addr 172.21.15.12:0/1562551125 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:13.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f485b7fe640 10 monclient(hunting): get_auth_request con 0x7f485c1353f0 auth_method 0 2024-09-11T01:37:13.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f485b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:13.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f485b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:13.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f485b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:13.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f485a7fc640 1 --2- 172.21.15.12:0/1562551125 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f485c134ad0 0x7f485c132250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:13.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f485b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f485b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f485b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f485a7fc640 10 monclient(hunting): get_auth_request con 0x7f485c134ad0 auth_method 0 2024-09-11T01:37:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f485a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f485a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f485a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f485a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f485a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.604+0000 7f485a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.605+0000 7f485b7fe640 10 monclient(hunting): handle_auth_done global_id 84170 payload 824 2024-09-11T01:37:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.605+0000 7f485b7fe640 1 -- 172.21.15.12:0/1562551125 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f485c134ad0 msgr2=0x7f485c132250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.605+0000 7f485b7fe640 1 --2- 172.21.15.12:0/1562551125 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f485c134ad0 0x7f485c132250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.605+0000 7f485b7fe640 1 -- 172.21.15.12:0/1562551125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f485c0a2200 msgr2=0x7f485c131d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:37:13.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.605+0000 7f485b7fe640 1 --2- 172.21.15.12:0/1562551125 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f485c0a2200 0x7f485c131d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.605+0000 7f485b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:37:13.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.605+0000 7f485b7fe640 1 monclient: found mon.noname-c 2024-09-11T01:37:13.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.605+0000 7f485b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:13.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.605+0000 7f485b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:37:13.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.605+0000 7f485b7fe640 1 -- 172.21.15.12:0/1562551125 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f485c132790 con 0x7f485c1353f0 2024-09-11T01:37:13.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.607+0000 7f485b7fe640 1 -- 172.21.15.12:0/1562551125 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c1353f0 msgr2=0x7f485c11e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:13.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.607+0000 7f485b7fe640 1 --2- 172.21.15.12:0/1562551125 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c1353f0 0x7f485c11e8e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4854009e80 tx=0x7f4854004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:13.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.607+0000 7f4843fff640 1 -- 172.21.15.12:0/1562551125 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4854041020 con 0x7f485c1353f0 2024-09-11T01:37:13.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.607+0000 7f4843fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:13.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.607+0000 7f4843fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:37:13.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.607+0000 7f4843fff640 10 monclient: dump: 2024-09-11T01:37:13.626 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:13.626 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:13.626 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.626 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.626 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:13.626 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:13.626 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:13.626 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:13.627 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:13.627 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:13.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.607+0000 7f4843fff640 10 monclient: _finish_auth 0 2024-09-11T01:37:13.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.607+0000 7f4843fff640 10 monclient: _check_auth_tickets 2024-09-11T01:37:13.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.607+0000 7f4843fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:13.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.607+0000 7f4843fff640 1 -- 172.21.15.12:0/1562551125 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4854031940 con 0x7f485c1353f0 2024-09-11T01:37:13.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.607+0000 7f4843fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:13.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.607+0000 7f4843fff640 1 -- 172.21.15.12:0/1562551125 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4854036040 con 0x7f485c1353f0 2024-09-11T01:37:13.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.607+0000 7f4843fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:13.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.607+0000 7f4843fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:13.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.607+0000 7f4843fff640 10 monclient: dump: 2024-09-11T01:37:13.627 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:13.627 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:13.627 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.628 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.628 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:13.628 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:13.628 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:13.628 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:13.628 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:13.628 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:13.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.607+0000 7f486192b640 5 monclient: authenticate success, global_id 84170 2024-09-11T01:37:13.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.607+0000 7f486192b640 10 monclient: _renew_subs 2024-09-11T01:37:13.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.607+0000 7f486192b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:13.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.607+0000 7f486192b640 1 -- 172.21.15.12:0/1562551125 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f485c132cd0 con 0x7f485c1353f0 2024-09-11T01:37:13.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.607+0000 7f48597fa640 4 set_mon_vals no callback set 2024-09-11T01:37:13.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.607+0000 7f486192b640 10 monclient: _renew_subs 2024-09-11T01:37:13.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.607+0000 7f486192b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:13.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.607+0000 7f486192b640 1 -- 172.21.15.12:0/1562551125 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f485c122700 con 0x7f485c1353f0 2024-09-11T01:37:13.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.608+0000 7f486192b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:13.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.608+0000 7f486192b640 1 --2- 172.21.15.12:0/1562551125 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c11ee20 0x7f485c11f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:13.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.608+0000 7f486192b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:13.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.608+0000 7f486192b640 1 -- 172.21.15.12:0/1562551125 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f485c09d480 con 0x7f485c11ee20 2024-09-11T01:37:13.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.608+0000 7f485affd640 1 --2- 172.21.15.12:0/1562551125 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c11ee20 0x7f485c11f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:13.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.608+0000 7f485affd640 10 monclient: get_auth_request con 0x7f485c11ee20 auth_method 0 2024-09-11T01:37:13.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.608+0000 7f485affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:13.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.608+0000 7f485affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:13.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.608+0000 7f485affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:13.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.608+0000 7f4843fff640 1 -- 172.21.15.12:0/1562551125 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f485403c470 con 0x7f485c1353f0 2024-09-11T01:37:13.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.608+0000 7f485affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:13.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.608+0000 7f485affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:13.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.608+0000 7f485affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:13.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.608+0000 7f485affd640 10 monclient(hunting): handle_auth_done global_id 84173 payload 824 2024-09-11T01:37:13.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.608+0000 7f485affd640 1 -- 172.21.15.12:0/1562551125 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f485c1222d0 con 0x7f485c11ee20 2024-09-11T01:37:13.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.608+0000 7f4843fff640 1 --2- 172.21.15.12:0/1562551125 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f484803e240 0x7f4848040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:13.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.608+0000 7f4843fff640 1 -- 172.21.15.12:0/1562551125 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(259..259 src has 186..259) ==== 4771+0+0 (secure 0 0 0) 0x7f4854073ab0 con 0x7f485c1353f0 2024-09-11T01:37:13.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.609+0000 7f485a7fc640 1 --2- 172.21.15.12:0/1562551125 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f484803e240 0x7f4848040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:13.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.609+0000 7f485a7fc640 10 monclient: get_auth_request con 0x7f484803e240 auth_method 0 2024-09-11T01:37:13.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.611+0000 7f485affd640 1 -- 172.21.15.12:0/1562551125 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c11ee20 msgr2=0x7f485c11f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:13.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.611+0000 7f485affd640 1 --2- 172.21.15.12:0/1562551125 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c11ee20 0x7f485c11f200 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f484c009fd0 tx=0x7f484c00ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:13.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.611+0000 7f4843fff640 1 -- 172.21.15.12:0/1562551125 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484c017070 con 0x7f485c11ee20 2024-09-11T01:37:13.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.611+0000 7f4843fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:13.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.611+0000 7f4843fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:13.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.611+0000 7f4843fff640 10 monclient: dump: 2024-09-11T01:37:13.631 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:13.631 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:13.631 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.631 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.631 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:13.631 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:13.631 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:13.631 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:13.631 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:13.631 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.611+0000 7f4843fff640 1 -- 172.21.15.12:0/1562551125 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f484c00cbc0 con 0x7f485c11ee20 2024-09-11T01:37:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.611+0000 7f4843fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.611+0000 7f4843fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:37:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.611+0000 7f4843fff640 1 -- 172.21.15.12:0/1562551125 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c11ee20 msgr2=0x7f485c11f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.611+0000 7f4843fff640 1 --2- 172.21.15.12:0/1562551125 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c11ee20 0x7f485c11f200 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f484c009fd0 tx=0x7f484c00ec70 comp rx=0 tx=0).stop 2024-09-11T01:37:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.611+0000 7f4843fff640 1 -- 172.21.15.12:0/1562551125 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c11ee20 msgr2=0x7f485c11f200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:37:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.611+0000 7f4843fff640 1 --2- 172.21.15.12:0/1562551125 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c11ee20 0x7f485c11f200 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f484c009fd0 tx=0x7f484c00ec70 comp rx=0 tx=0).stop 2024-09-11T01:37:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.611+0000 7f485a7fc640 1 --2- 172.21.15.12:0/1562551125 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f484803e240 0x7f4848040700 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f485c134360 tx=0x7f4850007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.617+0000 7f486192b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:37:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.617+0000 7f486192b640 1 --2- 172.21.15.12:0/1562551125 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c11f740 0x7f485c0726b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.617+0000 7f486192b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.617+0000 7f486192b640 1 -- 172.21.15.12:0/1562551125 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f485c134eb0 con 0x7f485c11f740 2024-09-11T01:37:13.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.617+0000 7f485affd640 1 --2- 172.21.15.12:0/1562551125 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c11f740 0x7f485c0726b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:13.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.617+0000 7f485affd640 10 monclient: get_auth_request con 0x7f485c11f740 auth_method 0 2024-09-11T01:37:13.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.617+0000 7f485affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:13.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.617+0000 7f485affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:13.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.617+0000 7f485affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:13.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.617+0000 7f485affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:13.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.617+0000 7f485affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:13.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.617+0000 7f485affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:13.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.617+0000 7f485affd640 10 monclient(hunting): handle_auth_done global_id 84176 payload 824 2024-09-11T01:37:13.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.617+0000 7f485affd640 1 -- 172.21.15.12:0/1562551125 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f485c121b30 con 0x7f485c11f740 2024-09-11T01:37:13.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.620+0000 7f485affd640 1 -- 172.21.15.12:0/1562551125 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c11f740 msgr2=0x7f485c0726b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:13.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.620+0000 7f485affd640 1 --2- 172.21.15.12:0/1562551125 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c11f740 0x7f485c0726b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f484c018000 tx=0x7f484c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:13.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.620+0000 7f4843fff640 1 -- 172.21.15.12:0/1562551125 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484c017030 con 0x7f485c11f740 2024-09-11T01:37:13.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.620+0000 7f4843fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:13.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.620+0000 7f4843fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:13.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.620+0000 7f4843fff640 10 monclient: dump: 2024-09-11T01:37:13.634 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:13.634 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:13.634 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.634 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.634 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:13.634 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:13.634 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:13.634 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:13.634 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:13.635 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:13.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.620+0000 7f4843fff640 1 -- 172.21.15.12:0/1562551125 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4696 (secure 0 0 0) 0x7f484c005db0 con 0x7f485c11f740 2024-09-11T01:37:13.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.620+0000 7f4843fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:37:13.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.620+0000 7f4843fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:37:13.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.620+0000 7f4843fff640 1 -- 172.21.15.12:0/1562551125 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c11f740 msgr2=0x7f485c0726b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:13.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.620+0000 7f4843fff640 1 --2- 172.21.15.12:0/1562551125 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c11f740 0x7f485c0726b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f484c018000 tx=0x7f484c004730 comp rx=0 tx=0).stop 2024-09-11T01:37:13.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.620+0000 7f4843fff640 1 -- 172.21.15.12:0/1562551125 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c11f740 msgr2=0x7f485c0726b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:37:13.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.620+0000 7f4843fff640 1 --2- 172.21.15.12:0/1562551125 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c11f740 0x7f485c0726b0 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f484c018000 tx=0x7f484c004730 comp rx=0 tx=0).stop 2024-09-11T01:37:13.635 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:37:13.636 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:37:13.636 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:37:13.636 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:37:13.636 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 84, 2024-09-11T01:37:13.636 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 27643, 2024-09-11T01:37:13.636 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:37:13.636 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:37:13.636 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:37:13.636 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:13.636 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:37:13.636 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:37:13.636 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:37:13.637 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:37:13.637 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:13.637 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:13.637 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:13.637 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:13.637 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:13.637 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:13.637 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:13.637 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:13.637 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:13.637 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:13.637 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:13.637 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:13.637 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:37:13.638 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:37:13.638 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:13.638 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:13.638 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:13.638 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:13.638 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:13.638 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:13.638 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:13.638 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:13.638 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:13.638 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:13.638 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:13.638 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:13.639 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:13.639 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:37:13.639 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:37:13.639 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:37:13.639 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:37:13.639 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:37:13.639 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:37:13.639 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:37:13.639 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:37:13.639 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:37:13.639 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:37:13.639 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:37:13.639 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:37:13.639 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:37:13.640 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:37:13.640 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:37:13.640 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:37:13.640 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:37:13.640 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:13.640 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:13.640 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:13.640 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:13.640 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:13.640 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:13.640 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:13.640 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:13.640 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:13.641 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:13.641 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:13.641 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:13.641 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:37:13.641 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:13.641 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:37:13.641 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:13.641 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:37:13.641 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:37:13.641 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:13.641 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:13.641 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:13.642 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:13.642 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:37:13.642 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:13.642 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:13.642 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:13.642 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:13.642 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:37:13.642 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:13.642 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:13.642 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:13.642 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:13.642 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:37:13.642 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:37:13.643 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:13.643 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:13.643 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:13.643 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:13.643 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:13.643 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:37:13.643 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:37:13.643 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:13.643 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:13.643 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:13.643 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:13.643 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:37:13.643 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:13.644 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:13.644 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:13.644 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:13.644 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:37:13.644 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:13.644 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:13.644 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:13.644 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:13.644 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:37:13.644 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:37:13.644 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:13.644 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:13.644 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:13.645 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:13.645 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:13.645 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:37:13.645 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:37:13.645 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:13.645 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:13.645 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:13.645 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:13.645 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:37:13.645 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:13.645 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:13.645 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:13.645 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:13.646 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:37:13.646 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:13.646 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:13.646 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:13.646 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:13.646 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:37:13.646 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:37:13.646 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:13.646 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:13.646 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:13.646 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:13.646 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:13.646 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:13.646 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:37:13.647 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:37:13.647 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:13.647 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:13.647 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:13.647 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:37:13.647 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:13.647 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:13.647 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:37:13.647 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:13.647 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:13.647 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:13.647 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:37:13.647 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:13.648 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:13.648 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:37:13.648 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:13.648 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:13.648 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:13.648 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:37:13.648 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:13.648 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:13.648 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:13.648 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:37:13.648 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:37:13.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.620+0000 7f486192b640 1 -- 172.21.15.12:0/1562551125 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f484803e240 msgr2=0x7f4848040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:13.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.620+0000 7f486192b640 1 --2- 172.21.15.12:0/1562551125 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f484803e240 0x7f4848040700 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f485c134360 tx=0x7f4850007e30 comp rx=0 tx=0).stop 2024-09-11T01:37:13.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.620+0000 7f486192b640 10 monclient: shutdown 2024-09-11T01:37:13.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.620+0000 7f486192b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:13.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.620+0000 7f486192b640 1 -- 172.21.15.12:0/1562551125 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c1353f0 msgr2=0x7f485c11e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:13.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.620+0000 7f485b7fe640 1 -- 172.21.15.12:0/1562551125 reap_dead start 2024-09-11T01:37:13.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.620+0000 7f486192b640 1 --2- 172.21.15.12:0/1562551125 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c1353f0 0x7f485c11e8e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4854009e80 tx=0x7f4854004a40 comp rx=0 tx=0).stop 2024-09-11T01:37:13.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.620+0000 7f486192b640 1 -- 172.21.15.12:0/1562551125 shutdown_connections 2024-09-11T01:37:13.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.620+0000 7f486192b640 1 --2- 172.21.15.12:0/1562551125 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f485c1353f0 0x7f485c11e8e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.620+0000 7f486192b640 1 -- 172.21.15.12:0/1562551125 >> 172.21.15.12:0/1562551125 conn(0x7f485c0a2f70 msgr2=0x7f485c0a0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:13.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.621+0000 7f486192b640 1 -- 172.21.15.12:0/1562551125 shutdown_connections 2024-09-11T01:37:13.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.621+0000 7f486192b640 1 -- 172.21.15.12:0/1562551125 wait complete. 2024-09-11T01:37:13.652 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:37:13.652 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T01:37:13.652 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:37:13.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.753+0000 7f7a5f6bd640 10 monclient: get_monmap_and_config 2024-09-11T01:37:13.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.753+0000 7f7a5f6bd640 10 monclient: build_initial_monmap 2024-09-11T01:37:13.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.753+0000 7f7a5f6bd640 10 monclient: monmap: 2024-09-11T01:37:13.755 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:13.755 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:13.755 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:37:13.753970+0000 2024-09-11T01:37:13.756 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:37:13.753970+0000 2024-09-11T01:37:13.756 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:13.756 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:13.756 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:37:13.756 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:37:13.756 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:37:13.756 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:37:13.756 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:37:13.756 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:13.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.753+0000 7f7a5f6bd640 1 Processor -- start 2024-09-11T01:37:13.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5f6bd640 1 -- start start 2024-09-11T01:37:13.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5f6bd640 10 monclient: init 2024-09-11T01:37:13.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5f6bd640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:13.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5f6bd640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-11T01:37:13.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5f6bd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7a58137d50 0x7f7a5809d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:13.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5f6bd640 10 monclient(hunting): picked mon.noname-a con 0x7f7a58137d50 addr v2:172.21.15.12:3300/0 2024-09-11T01:37:13.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5f6bd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7a58138130 0x7f7a58138580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:13.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5f6bd640 10 monclient(hunting): picked mon.noname-c con 0x7f7a58138130 addr v2:172.21.15.71:3300/0 2024-09-11T01:37:13.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5f6bd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a58136cd0 0x7f7a581370f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:13.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5f6bd640 10 monclient(hunting): picked mon.noname-b con 0x7f7a58136cd0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:13.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5f6bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:13.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5f6bd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f7a5809c650 con 0x7f7a58137d50 2024-09-11T01:37:13.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5f6bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:13.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5f6bd640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f7a5809ba20 con 0x7f7a58138130 2024-09-11T01:37:13.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5f6bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:13.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5f6bd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7a5809c370 con 0x7f7a58136cd0 2024-09-11T01:37:13.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5f6bd640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:13.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5f6bd640 10 monclient(hunting): authenticate will time out at 1240.196566s 2024-09-11T01:37:13.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5d459640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7f7a58137d50 msgr2=0x7f7a5809d110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:37:13.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5d459640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7a58137d50 0x7f7a5809d110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:37:13.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5dc5a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a58136cd0 0x7f7a581370f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:13.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5dc5a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a58136cd0 0x7f7a581370f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41388/0 (socket says 172.21.15.12:41388) 2024-09-11T01:37:13.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5dc5a640 1 -- 172.21.15.12:0/23651179 learned_addr learned my addr 172.21.15.12:0/23651179 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:13.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5dc5a640 10 monclient(hunting): get_auth_request con 0x7f7a58136cd0 auth_method 0 2024-09-11T01:37:13.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5dc5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:13.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5dc5a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:13.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5dc5a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:13.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5dc5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:13.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5dc5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:13.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5dc5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:13.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5cc58640 1 --2- 172.21.15.12:0/23651179 >> v2:172.21.15.71:3300/0 conn(0x7f7a58138130 0x7f7a58138580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:13.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5cc58640 10 monclient(hunting): get_auth_request con 0x7f7a58138130 auth_method 0 2024-09-11T01:37:13.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5cc58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:13.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5cc58640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:13.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5cc58640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:13.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5dc5a640 10 monclient(hunting): handle_auth_done global_id 84179 payload 274 2024-09-11T01:37:13.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5dc5a640 1 -- 172.21.15.12:0/23651179 >> v2:172.21.15.71:3300/0 conn(0x7f7a58138130 msgr2=0x7f7a58138580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:13.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5dc5a640 1 --2- 172.21.15.12:0/23651179 >> v2:172.21.15.71:3300/0 conn(0x7f7a58138130 0x7f7a58138580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5dc5a640 1 -- 172.21.15.12:0/23651179 >> v2:172.21.15.12:3300/0 conn(0x7f7a58137d50 msgr2=0x7f7a5809d110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:37:13.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5dc5a640 1 --2- 172.21.15.12:0/23651179 >> v2:172.21.15.12:3300/0 conn(0x7f7a58137d50 0x7f7a5809d110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5dc5a640 10 monclient: _finish_hunting 0 2024-09-11T01:37:13.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5dc5a640 1 monclient: found mon.noname-b 2024-09-11T01:37:13.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5dc5a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:13.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5dc5a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:37:13.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.754+0000 7f7a5dc5a640 1 -- 172.21.15.12:0/23651179 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a580a6ce0 con 0x7f7a58136cd0 2024-09-11T01:37:13.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.757+0000 7f7a5dc5a640 1 -- 172.21.15.12:0/23651179 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a58136cd0 msgr2=0x7f7a581370f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:13.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.757+0000 7f7a5dc5a640 1 --2- 172.21.15.12:0/23651179 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a58136cd0 0x7f7a581370f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7a5806fce0 tx=0x7f7a5002a630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6eedebc1d0a65c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:13.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.757+0000 7f7a4bfff640 1 -- 172.21.15.12:0/23651179 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a50036070 con 0x7f7a58136cd0 2024-09-11T01:37:13.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.757+0000 7f7a4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:13.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.757+0000 7f7a4bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:37:13.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.757+0000 7f7a4bfff640 10 monclient: dump: 2024-09-11T01:37:13.767 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:13.767 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:13.767 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.768 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.768 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:13.768 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:13.768 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:13.768 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:13.768 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:13.768 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:13.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.757+0000 7f7a4bfff640 10 monclient: _finish_auth 0 2024-09-11T01:37:13.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.757+0000 7f7a4bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:37:13.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.757+0000 7f7a4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:13.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.757+0000 7f7a4bfff640 1 -- 172.21.15.12:0/23651179 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a5002ae50 con 0x7f7a58136cd0 2024-09-11T01:37:13.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.757+0000 7f7a5f6bd640 5 monclient: authenticate success, global_id 84179 2024-09-11T01:37:13.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.757+0000 7f7a5f6bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:37:13.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.757+0000 7f7a4bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:13.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.757+0000 7f7a5f6bd640 10 monclient: get_monmap_and_config success 2024-09-11T01:37:13.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.757+0000 7f7a4bfff640 1 -- 172.21.15.12:0/23651179 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a50007a20 con 0x7f7a58136cd0 2024-09-11T01:37:13.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.757+0000 7f7a4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:13.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.757+0000 7f7a5f6bd640 4 set_mon_vals no callback set 2024-09-11T01:37:13.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.757+0000 7f7a4bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:13.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.757+0000 7f7a4bfff640 10 monclient: dump: 2024-09-11T01:37:13.769 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:13.769 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:13.769 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.769 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.769 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:13.769 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:13.769 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:13.770 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:13.770 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:13.770 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:13.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.758+0000 7f7a5f6bd640 10 monclient: shutdown 2024-09-11T01:37:13.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.758+0000 7f7a5f6bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:13.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.758+0000 7f7a5f6bd640 1 -- 172.21.15.12:0/23651179 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a58136cd0 msgr2=0x7f7a581370f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:13.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.758+0000 7f7a5f6bd640 1 --2- 172.21.15.12:0/23651179 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a58136cd0 0x7f7a581370f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7a5806fce0 tx=0x7f7a5002a630 comp rx=0 tx=0).stop 2024-09-11T01:37:13.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.758+0000 7f7a5f6bd640 1 -- 172.21.15.12:0/23651179 shutdown_connections 2024-09-11T01:37:13.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.758+0000 7f7a5f6bd640 1 --2- 172.21.15.12:0/23651179 >> v2:172.21.15.71:3300/0 conn(0x7f7a58138130 0x7f7a58138580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.758+0000 7f7a5f6bd640 1 --2- 172.21.15.12:0/23651179 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a58136cd0 0x7f7a581370f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.758+0000 7f7a5f6bd640 1 --2- 172.21.15.12:0/23651179 >> v2:172.21.15.12:3300/0 conn(0x7f7a58137d50 0x7f7a5809d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.758+0000 7f7a5f6bd640 1 -- 172.21.15.12:0/23651179 >> 172.21.15.12:0/23651179 conn(0x7f7a580a7020 msgr2=0x7f7a580a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:13.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.758+0000 7f7a5f6bd640 1 -- 172.21.15.12:0/23651179 shutdown_connections 2024-09-11T01:37:13.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.758+0000 7f7a5f6bd640 1 -- 172.21.15.12:0/23651179 wait complete. 2024-09-11T01:37:13.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.759+0000 7f7a5f6bd640 10 monclient: build_initial_monmap 2024-09-11T01:37:13.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.759+0000 7f7a5f6bd640 10 monclient: monmap: 2024-09-11T01:37:13.771 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:13.771 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:37:13.771 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:37:13.771 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:37:13.771 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:13.771 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:13.771 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:37:13.771 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:37:13.771 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:37:13.771 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:13.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.759+0000 7f7a5f6bd640 1 Processor -- start 2024-09-11T01:37:13.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.759+0000 7f7a5f6bd640 1 -- start start 2024-09-11T01:37:13.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.759+0000 7f7a5f6bd640 10 monclient: init 2024-09-11T01:37:13.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.759+0000 7f7a5f6bd640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:13.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.759+0000 7f7a5f6bd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:37:13.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.759+0000 7f7a5f6bd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a58136cd0 0x7f7a58125980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:13.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.759+0000 7f7a5f6bd640 10 monclient(hunting): picked mon.noname-b con 0x7f7a58136cd0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:37:13.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5f6bd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a58137d50 0x7f7a581233a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:13.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5f6bd640 10 monclient(hunting): picked mon.noname-a con 0x7f7a58137d50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:37:13.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5f6bd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a58138130 0x7f7a581238e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:13.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5f6bd640 10 monclient(hunting): picked mon.noname-c con 0x7f7a58138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:13.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5f6bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:13.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5f6bd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7a5809d650 con 0x7f7a58137d50 2024-09-11T01:37:13.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5f6bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:13.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5f6bd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f7a58138b70 con 0x7f7a58136cd0 2024-09-11T01:37:13.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5f6bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:13.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5f6bd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7a5809d7d0 con 0x7f7a58138130 2024-09-11T01:37:13.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5f6bd640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:13.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5f6bd640 10 monclient(hunting): authenticate will time out at 1240.202047s 2024-09-11T01:37:13.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5cc58640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a58137d50 msgr2=0x7f7a581233a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:37:13.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5cc58640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a58137d50 0x7f7a581233a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:37:13.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5dc5a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a58138130 0x7f7a581238e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:13.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5dc5a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a58138130 0x7f7a581238e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41392/0 (socket says 172.21.15.12:41392) 2024-09-11T01:37:13.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5dc5a640 1 -- 172.21.15.12:0/4236946772 learned_addr learned my addr 172.21.15.12:0/4236946772 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:13.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5dc5a640 10 monclient(hunting): get_auth_request con 0x7f7a58138130 auth_method 0 2024-09-11T01:37:13.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5dc5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:13.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5dc5a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:13.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5dc5a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:13.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5d459640 1 --2- 172.21.15.12:0/4236946772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a58136cd0 0x7f7a58125980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:13.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5dc5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:13.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5dc5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:13.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5dc5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:13.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5d459640 10 monclient(hunting): get_auth_request con 0x7f7a58136cd0 auth_method 0 2024-09-11T01:37:13.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5d459640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:13.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5d459640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:13.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5d459640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:13.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5d459640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:13.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5d459640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:13.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5d459640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:13.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5dc5a640 10 monclient(hunting): handle_auth_done global_id 84182 payload 824 2024-09-11T01:37:13.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5dc5a640 1 -- 172.21.15.12:0/4236946772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a58136cd0 msgr2=0x7f7a58125980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:13.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5dc5a640 1 --2- 172.21.15.12:0/4236946772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a58136cd0 0x7f7a58125980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5dc5a640 1 -- 172.21.15.12:0/4236946772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a58137d50 msgr2=0x7f7a581233a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:37:13.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5dc5a640 1 --2- 172.21.15.12:0/4236946772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a58137d50 0x7f7a581233a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5dc5a640 10 monclient: _finish_hunting 0 2024-09-11T01:37:13.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5dc5a640 1 monclient: found mon.noname-c 2024-09-11T01:37:13.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5dc5a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:13.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5dc5a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:37:13.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.760+0000 7f7a5dc5a640 1 -- 172.21.15.12:0/4236946772 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a58126ea0 con 0x7f7a58138130 2024-09-11T01:37:13.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.762+0000 7f7a5dc5a640 1 -- 172.21.15.12:0/4236946772 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a58138130 msgr2=0x7f7a581238e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:13.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.762+0000 7f7a5dc5a640 1 --2- 172.21.15.12:0/4236946772 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a58138130 0x7f7a581238e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7a5002ac20 tx=0x7f7a50004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:13.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.762+0000 7f7a49ffb640 1 -- 172.21.15.12:0/4236946772 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a50036050 con 0x7f7a58138130 2024-09-11T01:37:13.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.762+0000 7f7a49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:13.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.762+0000 7f7a49ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:37:13.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.762+0000 7f7a49ffb640 10 monclient: dump: 2024-09-11T01:37:13.781 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:13.781 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:13.781 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.781 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.781 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:13.781 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:13.781 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:13.781 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:13.781 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:13.781 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:13.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.762+0000 7f7a49ffb640 10 monclient: _finish_auth 0 2024-09-11T01:37:13.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.762+0000 7f7a49ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:37:13.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.762+0000 7f7a49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:13.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.762+0000 7f7a49ffb640 1 -- 172.21.15.12:0/4236946772 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a50042070 con 0x7f7a58138130 2024-09-11T01:37:13.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.762+0000 7f7a49ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:13.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.762+0000 7f7a49ffb640 1 -- 172.21.15.12:0/4236946772 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a5003bc80 con 0x7f7a58138130 2024-09-11T01:37:13.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.762+0000 7f7a49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:13.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.762+0000 7f7a49ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:13.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.762+0000 7f7a49ffb640 10 monclient: dump: 2024-09-11T01:37:13.782 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:13.782 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:13.782 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.782 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.782 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:13.783 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:13.783 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:13.783 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:13.783 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:13.783 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:13.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.762+0000 7f7a5f6bd640 5 monclient: authenticate success, global_id 84182 2024-09-11T01:37:13.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.762+0000 7f7a5f6bd640 10 monclient: _renew_subs 2024-09-11T01:37:13.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.762+0000 7f7a4b7fe640 4 set_mon_vals no callback set 2024-09-11T01:37:13.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.762+0000 7f7a5f6bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:13.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.762+0000 7f7a5f6bd640 1 -- 172.21.15.12:0/4236946772 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a5806f100 con 0x7f7a58138130 2024-09-11T01:37:13.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.763+0000 7f7a5f6bd640 10 monclient: _renew_subs 2024-09-11T01:37:13.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.763+0000 7f7a5f6bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:13.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.763+0000 7f7a5f6bd640 1 -- 172.21.15.12:0/4236946772 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a58126bc0 con 0x7f7a58138130 2024-09-11T01:37:13.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.763+0000 7f7a5f6bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:13.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.763+0000 7f7a5f6bd640 1 --2- 172.21.15.12:0/4236946772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a58123e20 0x7f7a581cd060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:13.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.763+0000 7f7a5f6bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:13.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.763+0000 7f7a5f6bd640 1 -- 172.21.15.12:0/4236946772 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f7a58124e50 con 0x7f7a58123e20 2024-09-11T01:37:13.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.763+0000 7f7a49ffb640 1 -- 172.21.15.12:0/4236946772 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f7a5003d450 con 0x7f7a58138130 2024-09-11T01:37:13.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.764+0000 7f7a5d459640 1 --2- 172.21.15.12:0/4236946772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a58123e20 0x7f7a581cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:13.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.764+0000 7f7a5d459640 10 monclient: get_auth_request con 0x7f7a58123e20 auth_method 0 2024-09-11T01:37:13.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.764+0000 7f7a5d459640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:13.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.764+0000 7f7a5d459640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:13.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.764+0000 7f7a5d459640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:13.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.764+0000 7f7a49ffb640 1 --2- 172.21.15.12:0/4236946772 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7a3c03e210 0x7f7a3c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:13.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.764+0000 7f7a5d459640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:13.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.764+0000 7f7a5d459640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:13.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.764+0000 7f7a5d459640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:13.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.764+0000 7f7a49ffb640 1 -- 172.21.15.12:0/4236946772 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(259..259 src has 186..259) ==== 4771+0+0 (secure 0 0 0) 0x7f7a50073480 con 0x7f7a58138130 2024-09-11T01:37:13.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.764+0000 7f7a5d459640 10 monclient(hunting): handle_auth_done global_id 74257 payload 824 2024-09-11T01:37:13.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.764+0000 7f7a5d459640 1 -- 172.21.15.12:0/4236946772 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7a58127680 con 0x7f7a58123e20 2024-09-11T01:37:13.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.764+0000 7f7a5cc58640 1 --2- 172.21.15.12:0/4236946772 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7a3c03e210 0x7f7a3c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:13.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.764+0000 7f7a5cc58640 10 monclient: get_auth_request con 0x7f7a3c03e210 auth_method 0 2024-09-11T01:37:13.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.766+0000 7f7a5d459640 1 -- 172.21.15.12:0/4236946772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a58123e20 msgr2=0x7f7a581cd060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:13.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.766+0000 7f7a5d459640 1 --2- 172.21.15.12:0/4236946772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a58123e20 0x7f7a581cd060 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7a54009610 tx=0x7f7a54009ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:13.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.766+0000 7f7a49ffb640 1 -- 172.21.15.12:0/4236946772 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a54016070 con 0x7f7a58123e20 2024-09-11T01:37:13.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.766+0000 7f7a49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:13.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.766+0000 7f7a49ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:37:13.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.766+0000 7f7a5cc58640 1 --2- 172.21.15.12:0/4236946772 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7a3c03e210 0x7f7a3c0406d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7a58125910 tx=0x7f7a4c0092e0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:13.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.766+0000 7f7a49ffb640 10 monclient: dump: 2024-09-11T01:37:13.786 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:13.786 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:13.786 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.786 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.786 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:13.786 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:13.786 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:13.786 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:13.786 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:13.786 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:13.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.767+0000 7f7a49ffb640 1 -- 172.21.15.12:0/4236946772 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7a54005a40 con 0x7f7a58123e20 2024-09-11T01:37:13.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.767+0000 7f7a49ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:13.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.767+0000 7f7a49ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:37:13.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.767+0000 7f7a49ffb640 1 -- 172.21.15.12:0/4236946772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a58123e20 msgr2=0x7f7a581cd060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:13.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.767+0000 7f7a49ffb640 1 --2- 172.21.15.12:0/4236946772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a58123e20 0x7f7a581cd060 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7a54009610 tx=0x7f7a54009ad0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.767+0000 7f7a49ffb640 1 -- 172.21.15.12:0/4236946772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a58123e20 msgr2=0x7f7a581cd060 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:37:13.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.767+0000 7f7a49ffb640 1 --2- 172.21.15.12:0/4236946772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a58123e20 0x7f7a581cd060 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7a54009610 tx=0x7f7a54009ad0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.773+0000 7f7a5f6bd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:37:13.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.773+0000 7f7a5f6bd640 1 --2- 172.21.15.12:0/4236946772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a581cd5a0 0x7f7a581cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:13.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.773+0000 7f7a5f6bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:13.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.773+0000 7f7a5f6bd640 1 -- 172.21.15.12:0/4236946772 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f7a581cdef0 con 0x7f7a581cd5a0 2024-09-11T01:37:13.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.773+0000 7f7a5d459640 1 --2- 172.21.15.12:0/4236946772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a581cd5a0 0x7f7a581cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:13.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.773+0000 7f7a5d459640 10 monclient: get_auth_request con 0x7f7a581cd5a0 auth_method 0 2024-09-11T01:37:13.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.773+0000 7f7a5d459640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:13.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.773+0000 7f7a5d459640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:13.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.773+0000 7f7a5d459640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:13.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.773+0000 7f7a5d459640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:13.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.773+0000 7f7a5d459640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:13.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.773+0000 7f7a5d459640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:13.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.773+0000 7f7a5d459640 10 monclient(hunting): handle_auth_done global_id 74260 payload 824 2024-09-11T01:37:13.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.773+0000 7f7a5d459640 1 -- 172.21.15.12:0/4236946772 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7a581377d0 con 0x7f7a581cd5a0 2024-09-11T01:37:13.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.775+0000 7f7a5d459640 1 -- 172.21.15.12:0/4236946772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a581cd5a0 msgr2=0x7f7a581cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:13.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.775+0000 7f7a5d459640 1 --2- 172.21.15.12:0/4236946772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a581cd5a0 0x7f7a581cd9b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7a54017000 tx=0x7f7a540059d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:13.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.776+0000 7f7a49ffb640 1 -- 172.21.15.12:0/4236946772 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a54016050 con 0x7f7a581cd5a0 2024-09-11T01:37:13.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.776+0000 7f7a49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:13.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.776+0000 7f7a49ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:37:13.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.776+0000 7f7a49ffb640 10 monclient: dump: 2024-09-11T01:37:13.788 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:13.788 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:13.788 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.788 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:13.789 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:13.789 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:13.789 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:13.789 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:13.789 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:13.789 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:13.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.776+0000 7f7a49ffb640 1 -- 172.21.15.12:0/4236946772 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4865 (secure 0 0 0) 0x7f7a5400e3f0 con 0x7f7a581cd5a0 2024-09-11T01:37:13.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.776+0000 7f7a49ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:37:13.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.776+0000 7f7a49ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:37:13.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.776+0000 7f7a49ffb640 1 -- 172.21.15.12:0/4236946772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a581cd5a0 msgr2=0x7f7a581cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:13.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.776+0000 7f7a49ffb640 1 --2- 172.21.15.12:0/4236946772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a581cd5a0 0x7f7a581cd9b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7a54017000 tx=0x7f7a540059d0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.776+0000 7f7a49ffb640 1 -- 172.21.15.12:0/4236946772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a581cd5a0 msgr2=0x7f7a581cd9b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:37:13.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.776+0000 7f7a49ffb640 1 --2- 172.21.15.12:0/4236946772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a581cd5a0 0x7f7a581cd9b0 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7a54017000 tx=0x7f7a540059d0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.790 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:37:13.790 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:37:13.790 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:37:13.790 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:37:13.790 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 84, 2024-09-11T01:37:13.790 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 85555, 2024-09-11T01:37:13.790 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:37:13.790 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:37:13.790 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:37:13.790 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:13.791 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:37:13.791 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:37:13.791 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:37:13.791 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:37:13.791 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:13.791 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:13.791 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:13.791 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:13.791 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:13.792 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:13.792 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:13.792 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:13.792 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:13.792 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:13.792 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:13.792 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:13.792 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:37:13.792 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:37:13.792 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:13.792 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:13.792 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:13.792 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:13.793 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:13.793 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:13.793 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:13.793 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:13.793 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:13.793 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:13.793 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:13.793 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:13.793 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:13.793 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:37:13.793 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:37:13.793 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:37:13.793 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:37:13.794 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:37:13.794 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:37:13.794 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:37:13.794 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:37:13.794 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:37:13.794 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:37:13.794 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:37:13.794 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:37:13.794 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:37:13.794 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:37:13.794 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:37:13.794 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:37:13.794 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:37:13.794 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:13.795 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:13.795 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:13.795 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:13.795 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:13.795 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:13.795 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:13.795 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:13.795 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:13.795 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:13.795 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:13.795 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:13.795 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:37:13.795 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:13.796 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:37:13.796 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:13.796 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:37:13.796 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:37:13.796 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:13.796 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:13.796 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:13.796 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:13.796 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:37:13.796 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:13.796 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:13.796 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:13.796 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:13.797 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:37:13.797 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:13.797 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:13.797 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:13.797 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:13.797 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:37:13.797 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:37:13.797 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:13.797 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:13.797 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:13.797 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:13.797 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:13.797 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:37:13.797 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:37:13.798 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:13.798 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:13.798 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:13.798 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:13.798 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:37:13.798 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:13.798 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:13.798 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:13.798 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:13.799 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:37:13.799 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:13.799 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:13.799 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:13.799 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:13.799 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:37:13.799 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:37:13.799 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:13.799 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:13.799 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:13.799 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:13.799 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:13.799 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:37:13.800 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:37:13.800 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:13.800 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:13.800 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:13.800 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:13.800 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:37:13.800 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:13.800 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:13.800 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:13.800 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:13.800 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:37:13.800 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:13.800 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:13.801 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:13.801 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:13.801 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:37:13.801 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:37:13.801 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:13.801 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:13.801 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:13.801 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:13.801 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:13.801 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:13.801 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:37:13.801 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:37:13.802 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:13.802 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:13.802 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:13.802 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:37:13.802 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:13.802 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:13.802 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:37:13.802 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:13.802 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:13.802 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:13.802 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:37:13.802 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:13.802 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:13.803 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:37:13.803 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:13.803 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:13.803 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:13.803 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:37:13.803 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:13.803 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:13.803 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:37:13.803 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:13.803 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:13.803 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:13.803 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:37:13.803 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:13.803 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:13.803 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:13.804 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:37:13.804 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:37:13.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.776+0000 7f7a5f6bd640 1 -- 172.21.15.12:0/4236946772 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7a3c03e210 msgr2=0x7f7a3c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:13.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.777+0000 7f7a5f6bd640 1 --2- 172.21.15.12:0/4236946772 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7a3c03e210 0x7f7a3c0406d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7a58125910 tx=0x7f7a4c0092e0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.777+0000 7f7a5f6bd640 10 monclient: shutdown 2024-09-11T01:37:13.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.777+0000 7f7a5f6bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:13.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.777+0000 7f7a5f6bd640 1 -- 172.21.15.12:0/4236946772 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a58138130 msgr2=0x7f7a581238e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:13.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.777+0000 7f7a5f6bd640 1 --2- 172.21.15.12:0/4236946772 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a58138130 0x7f7a581238e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7a5002ac20 tx=0x7f7a50004a40 comp rx=0 tx=0).stop 2024-09-11T01:37:13.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.777+0000 7f7a5dc5a640 1 -- 172.21.15.12:0/4236946772 reap_dead start 2024-09-11T01:37:13.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.777+0000 7f7a5f6bd640 1 -- 172.21.15.12:0/4236946772 shutdown_connections 2024-09-11T01:37:13.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.777+0000 7f7a5f6bd640 1 --2- 172.21.15.12:0/4236946772 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a58138130 0x7f7a581238e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:13.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.777+0000 7f7a5f6bd640 1 -- 172.21.15.12:0/4236946772 >> 172.21.15.12:0/4236946772 conn(0x7f7a580a7020 msgr2=0x7f7a580a0050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:13.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.777+0000 7f7a5f6bd640 1 -- 172.21.15.12:0/4236946772 shutdown_connections 2024-09-11T01:37:13.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:13.777+0000 7f7a5f6bd640 1 -- 172.21.15.12:0/4236946772 wait complete. 2024-09-11T01:37:13.806 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:37:13.807 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:37:14.249 INFO:tasks.rados.rados.0.smithi071.stdout:1134: done (5 left) 2024-09-11T01:37:14.249 INFO:tasks.rados.rados.0.smithi071.stdout:1135: delete oid 21 current snap is 118 2024-09-11T01:37:14.250 INFO:tasks.rados.rados.0.smithi071.stdout:1132: finishing copy_from to smithi07134777-5 2024-09-11T01:37:14.250 INFO:tasks.rados.rados.0.smithi071.stdout:1132: got expected ENOENT (src dne) 2024-09-11T01:37:14.250 INFO:tasks.rados.rados.0.smithi071.stdout:1132: finishing copy_from racing read to smithi07134777-5 2024-09-11T01:37:14.250 INFO:tasks.rados.rados.0.smithi071.stdout:1129: finishing write tid 3 to smithi07134777-10 2024-09-11T01:37:14.250 INFO:tasks.rados.rados.0.smithi071.stdout:1129: finishing write tid 4 to smithi07134777-10 2024-09-11T01:37:14.250 INFO:tasks.rados.rados.0.smithi071.stdout:1129: finishing write tid 5 to smithi07134777-10 2024-09-11T01:37:14.250 INFO:tasks.rados.rados.0.smithi071.stdout:1129: oid 10 updating version 0 to 337 2024-09-11T01:37:14.250 INFO:tasks.rados.rados.0.smithi071.stdout:1129: oid 10 version 337 is already newer than 336 2024-09-11T01:37:14.250 INFO:tasks.rados.rados.0.smithi071.stdout:1129: oid 10 updating version 337 to 338 2024-09-11T01:37:14.250 INFO:tasks.rados.rados.0.smithi071.stdout:1129: oid 10 updating version 338 to 339 2024-09-11T01:37:14.250 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 339 (ObjNum 427 snap 118 seq_num 427) dirty exists 2024-09-11T01:37:14.250 INFO:tasks.rados.rados.0.smithi071.stdout:1129: left oid 10 (ObjNum 427 snap 118 seq_num 427) 2024-09-11T01:37:14.250 INFO:tasks.rados.rados.0.smithi071.stdout:1130: finishing rollback tid 0 to smithi07134777-1 2024-09-11T01:37:14.250 INFO:tasks.rados.rados.0.smithi071.stdout:1130: finishing rollback tid 1 to smithi07134777-1 2024-09-11T01:37:14.251 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 431 (ObjNum 705 snap 0 seq_num 3923164255) dirty dne 2024-09-11T01:37:14.251 INFO:tasks.rados.rados.0.smithi071.stdout:1131: finishing copy_from racing read to smithi07134777-31 2024-09-11T01:37:14.251 INFO:tasks.rados.rados.0.smithi071.stdout:1131: finishing copy_from to smithi07134777-31 2024-09-11T01:37:14.251 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 418 (ObjNum 250 snap 63 seq_num 250) dirty exists 2024-09-11T01:37:14.251 INFO:tasks.rados.rados.0.smithi071.stdout:1133: finishing write tid 1 to smithi07134777-26 2024-09-11T01:37:14.251 INFO:tasks.rados.rados.0.smithi071.stdout:1133: finishing write tid 2 to smithi07134777-26 2024-09-11T01:37:14.251 INFO:tasks.rados.rados.0.smithi071.stdout:1133: finishing write tid 3 to smithi07134777-26 2024-09-11T01:37:14.251 INFO:tasks.rados.rados.0.smithi071.stdout:1133: finishing write tid 4 to smithi07134777-26 2024-09-11T01:37:14.251 INFO:tasks.rados.rados.0.smithi071.stdout:1133: oid 26 updating version 0 to 508 2024-09-11T01:37:14.251 INFO:tasks.rados.rados.0.smithi071.stdout:1133: oid 26 updating version 508 to 509 2024-09-11T01:37:14.251 INFO:tasks.rados.rados.0.smithi071.stdout:1133: oid 26 version 509 is already newer than 507 2024-09-11T01:37:14.251 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 509 (ObjNum 428 snap 118 seq_num 428) dirty exists 2024-09-11T01:37:14.251 INFO:tasks.rados.rados.0.smithi071.stdout:1133: left oid 26 (ObjNum 428 snap 118 seq_num 428) 2024-09-11T01:37:14.253 INFO:tasks.rados.rados.0.smithi071.stdout:1129: done (5 left) 2024-09-11T01:37:14.253 INFO:tasks.rados.rados.0.smithi071.stdout:1130: done (4 left) 2024-09-11T01:37:14.253 INFO:tasks.rados.rados.0.smithi071.stdout:1131: done (3 left) 2024-09-11T01:37:14.253 INFO:tasks.rados.rados.0.smithi071.stdout:1132: done (2 left) 2024-09-11T01:37:14.254 INFO:tasks.rados.rados.0.smithi071.stdout:1133: done (1 left) 2024-09-11T01:37:14.254 INFO:tasks.rados.rados.0.smithi071.stdout:1135: done (0 left) 2024-09-11T01:37:14.254 INFO:tasks.rados.rados.0.smithi071.stdout:1136: snap_create 2024-09-11T01:37:15.264 INFO:tasks.rados.rados.0.smithi071.stdout:1136: done (0 left) 2024-09-11T01:37:15.264 INFO:tasks.rados.rados.0.smithi071.stdout:1137: read oid 15 snap -1 2024-09-11T01:37:15.264 INFO:tasks.rados.rados.0.smithi071.stdout:1137: expect deleted 2024-09-11T01:37:15.264 INFO:tasks.rados.rados.0.smithi071.stdout:1138: write (excl) oid 24 current snap is 119 2024-09-11T01:37:15.264 INFO:tasks.rados.rados.0.smithi071.stdout:1138: seq_num 430 ranges {427010=472838,1329610=694481,2708828=518182} 2024-09-11T01:37:15.268 INFO:tasks.rados.rados.0.smithi071.stdout:1138: writing smithi07134777-24 from 427010 to 899848 tid 1 2024-09-11T01:37:15.275 INFO:tasks.rados.rados.0.smithi071.stdout:1138: writing smithi07134777-24 from 1329610 to 2024091 tid 2 2024-09-11T01:37:15.281 INFO:tasks.rados.rados.0.smithi071.stdout:1138: writing smithi07134777-24 from 2708828 to 3227010 tid 3 2024-09-11T01:37:15.282 INFO:tasks.rados.rados.0.smithi071.stdout:1139: write oid 30 current snap is 119 2024-09-11T01:37:15.282 INFO:tasks.rados.rados.0.smithi071.stdout:1139: seq_num 431 ranges {443817=548087,1788092=438139,3019898=223919} 2024-09-11T01:37:15.287 INFO:tasks.rados.rados.0.smithi071.stdout:1139: writing smithi07134777-30 from 443817 to 991904 tid 1 2024-09-11T01:37:15.294 INFO:tasks.rados.rados.0.smithi071.stdout:1139: writing smithi07134777-30 from 1788092 to 2226231 tid 2 2024-09-11T01:37:15.299 INFO:tasks.rados.rados.0.smithi071.stdout:1139: writing smithi07134777-30 from 3019898 to 3243817 tid 3 2024-09-11T01:37:15.300 INFO:tasks.rados.rados.0.smithi071.stdout:1140: write oid 10 current snap is 119 2024-09-11T01:37:15.300 INFO:tasks.rados.rados.0.smithi071.stdout:1140: seq_num 432 ranges {460624=623336,1846574=698150,3163674=96950} 2024-09-11T01:37:15.305 INFO:tasks.rados.rados.0.smithi071.stdout:1140: writing smithi07134777-10 from 460624 to 1083960 tid 1 2024-09-11T01:37:15.313 INFO:tasks.rados.rados.0.smithi071.stdout:1140: writing smithi07134777-10 from 1846574 to 2544724 tid 2 2024-09-11T01:37:15.317 INFO:tasks.rados.rados.0.smithi071.stdout:1140: writing smithi07134777-10 from 3163674 to 3260624 tid 3 2024-09-11T01:37:15.317 INFO:tasks.rados.rados.0.smithi071.stdout:1141: write oid 46 current snap is 119 2024-09-11T01:37:15.317 INFO:tasks.rados.rados.0.smithi071.stdout:1138: finishing write tid 1 to smithi07134777-24 2024-09-11T01:37:15.317 INFO:tasks.rados.rados.0.smithi071.stdout:1138: finishing write tid 2 to smithi07134777-24 2024-09-11T01:37:15.317 INFO:tasks.rados.rados.0.smithi071.stdout:1141: seq_num 433 ranges {477431=698585,1905056=441808,3074744=202687} 2024-09-11T01:37:15.324 INFO:tasks.rados.rados.0.smithi071.stdout:1141: writing smithi07134777-46 from 477431 to 1176016 tid 1 2024-09-11T01:37:15.330 INFO:tasks.rados.rados.0.smithi071.stdout:1141: writing smithi07134777-46 from 1905056 to 2346864 tid 2 2024-09-11T01:37:15.335 INFO:tasks.rados.rados.0.smithi071.stdout:1141: writing smithi07134777-46 from 3074744 to 3277431 tid 3 2024-09-11T01:37:15.336 INFO:tasks.rados.rados.0.smithi071.stdout:1137: done (4 left) 2024-09-11T01:37:15.336 INFO:tasks.rados.rados.0.smithi071.stdout:1142: snap_remove snap 106 2024-09-11T01:37:16.266 INFO:tasks.rados.rados.0.smithi071.stdout:1142: done (4 left) 2024-09-11T01:37:16.266 INFO:tasks.rados.rados.0.smithi071.stdout:1143: delete oid 25 current snap is 119 2024-09-11T01:37:16.266 INFO:tasks.rados.rados.0.smithi071.stdout:1138: finishing write tid 3 to smithi07134777-24 2024-09-11T01:37:16.266 INFO:tasks.rados.rados.0.smithi071.stdout:1138: finishing write tid 4 to smithi07134777-24 2024-09-11T01:37:16.266 INFO:tasks.rados.rados.0.smithi071.stdout:1138: finishing write tid 5 to smithi07134777-24 2024-09-11T01:37:16.266 INFO:tasks.rados.rados.0.smithi071.stdout:1138: oid 24 updating version 0 to 483 2024-09-11T01:37:16.266 INFO:tasks.rados.rados.0.smithi071.stdout:1138: oid 24 version 483 is already newer than 481 2024-09-11T01:37:16.266 INFO:tasks.rados.rados.0.smithi071.stdout:1138: oid 24 version 483 is already newer than 482 2024-09-11T01:37:16.266 INFO:tasks.rados.rados.0.smithi071.stdout:1138: oid 24 version 483 is already newer than 480 2024-09-11T01:37:16.266 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 483 (ObjNum 430 snap 119 seq_num 430) dirty exists 2024-09-11T01:37:16.267 INFO:tasks.rados.rados.0.smithi071.stdout:1138: left oid 24 (ObjNum 430 snap 119 seq_num 430) 2024-09-11T01:37:16.267 INFO:tasks.rados.rados.0.smithi071.stdout:1139: finishing write tid 1 to smithi07134777-30 2024-09-11T01:37:16.267 INFO:tasks.rados.rados.0.smithi071.stdout:1139: finishing write tid 2 to smithi07134777-30 2024-09-11T01:37:16.267 INFO:tasks.rados.rados.0.smithi071.stdout:1139: finishing write tid 3 to smithi07134777-30 2024-09-11T01:37:16.267 INFO:tasks.rados.rados.0.smithi071.stdout:1139: finishing write tid 4 to smithi07134777-30 2024-09-11T01:37:16.267 INFO:tasks.rados.rados.0.smithi071.stdout:1139: finishing write tid 5 to smithi07134777-30 2024-09-11T01:37:16.267 INFO:tasks.rados.rados.0.smithi071.stdout:1139: oid 30 updating version 0 to 414 2024-09-11T01:37:16.267 INFO:tasks.rados.rados.0.smithi071.stdout:1139: oid 30 updating version 414 to 416 2024-09-11T01:37:16.267 INFO:tasks.rados.rados.0.smithi071.stdout:1139: oid 30 updating version 416 to 417 2024-09-11T01:37:16.267 INFO:tasks.rados.rados.0.smithi071.stdout:1139: oid 30 version 417 is already newer than 415 2024-09-11T01:37:16.267 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 417 (ObjNum 431 snap 119 seq_num 431) dirty exists 2024-09-11T01:37:16.267 INFO:tasks.rados.rados.0.smithi071.stdout:1139: left oid 30 (ObjNum 431 snap 119 seq_num 431) 2024-09-11T01:37:16.267 INFO:tasks.rados.rados.0.smithi071.stdout:1140: finishing write tid 1 to smithi07134777-10 2024-09-11T01:37:16.267 INFO:tasks.rados.rados.0.smithi071.stdout:1140: finishing write tid 2 to smithi07134777-10 2024-09-11T01:37:16.268 INFO:tasks.rados.rados.0.smithi071.stdout:1140: finishing write tid 3 to smithi07134777-10 2024-09-11T01:37:16.268 INFO:tasks.rados.rados.0.smithi071.stdout:1140: finishing write tid 4 to smithi07134777-10 2024-09-11T01:37:16.268 INFO:tasks.rados.rados.0.smithi071.stdout:1140: finishing write tid 5 to smithi07134777-10 2024-09-11T01:37:16.268 INFO:tasks.rados.rados.0.smithi071.stdout:1140: oid 10 updating version 0 to 340 2024-09-11T01:37:16.268 INFO:tasks.rados.rados.0.smithi071.stdout:1140: oid 10 updating version 340 to 342 2024-09-11T01:37:16.268 INFO:tasks.rados.rados.0.smithi071.stdout:1140: oid 10 version 342 is already newer than 341 2024-09-11T01:37:16.268 INFO:tasks.rados.rados.0.smithi071.stdout:1140: oid 10 updating version 342 to 343 2024-09-11T01:37:16.268 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 343 (ObjNum 432 snap 119 seq_num 432) dirty exists 2024-09-11T01:37:16.268 INFO:tasks.rados.rados.0.smithi071.stdout:1140: left oid 10 (ObjNum 432 snap 119 seq_num 432) 2024-09-11T01:37:16.268 INFO:tasks.rados.rados.0.smithi071.stdout:1141: finishing write tid 1 to smithi07134777-46 2024-09-11T01:37:16.268 INFO:tasks.rados.rados.0.smithi071.stdout:1141: finishing write tid 2 to smithi07134777-46 2024-09-11T01:37:16.268 INFO:tasks.rados.rados.0.smithi071.stdout:1141: finishing write tid 3 to smithi07134777-46 2024-09-11T01:37:16.268 INFO:tasks.rados.rados.0.smithi071.stdout:1141: finishing write tid 4 to smithi07134777-46 2024-09-11T01:37:16.268 INFO:tasks.rados.rados.0.smithi071.stdout:1141: finishing write tid 5 to smithi07134777-46 2024-09-11T01:37:16.269 INFO:tasks.rados.rados.0.smithi071.stdout:1141: oid 46 updating version 0 to 227 2024-09-11T01:37:16.269 INFO:tasks.rados.rados.0.smithi071.stdout:1141: oid 46 updating version 227 to 228 2024-09-11T01:37:16.269 INFO:tasks.rados.rados.0.smithi071.stdout:1141: oid 46 updating version 228 to 229 2024-09-11T01:37:16.269 INFO:tasks.rados.rados.0.smithi071.stdout:1141: oid 46 version 229 is already newer than 226 2024-09-11T01:37:16.269 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 229 (ObjNum 433 snap 119 seq_num 433) dirty exists 2024-09-11T01:37:16.269 INFO:tasks.rados.rados.0.smithi071.stdout:1141: left oid 46 (ObjNum 433 snap 119 seq_num 433) 2024-09-11T01:37:16.274 INFO:tasks.rados.rados.0.smithi071.stdout:1138: done (4 left) 2024-09-11T01:37:16.275 INFO:tasks.rados.rados.0.smithi071.stdout:1139: done (3 left) 2024-09-11T01:37:16.275 INFO:tasks.rados.rados.0.smithi071.stdout:1140: done (2 left) 2024-09-11T01:37:16.275 INFO:tasks.rados.rados.0.smithi071.stdout:1141: done (1 left) 2024-09-11T01:37:16.275 INFO:tasks.rados.rados.0.smithi071.stdout:1143: done (0 left) 2024-09-11T01:37:16.275 INFO:tasks.rados.rados.0.smithi071.stdout:1144: rollback oid 1 current snap is 119 2024-09-11T01:37:16.275 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 1 to 118 2024-09-11T01:37:16.275 INFO:tasks.rados.rados.0.smithi071.stdout:1145: write oid 4 current snap is 119 2024-09-11T01:37:16.275 INFO:tasks.rados.rados.0.smithi071.stdout:1145: seq_num 435 ranges {511045=565436,1622020=445477,2729590=556855} 2024-09-11T01:37:16.281 INFO:tasks.rados.rados.0.smithi071.stdout:1145: writing smithi07134777-4 from 511045 to 1076481 tid 1 2024-09-11T01:37:16.286 INFO:tasks.rados.rados.0.smithi071.stdout:1145: writing smithi07134777-4 from 1622020 to 2067497 tid 2 2024-09-11T01:37:16.294 INFO:tasks.rados.rados.0.smithi071.stdout:1145: writing smithi07134777-4 from 2729590 to 3286445 tid 3 2024-09-11T01:37:16.294 INFO:tasks.rados.rados.0.smithi071.stdout:1146: read oid 11 snap -1 2024-09-11T01:37:16.294 INFO:tasks.rados.rados.0.smithi071.stdout:1146: expect (ObjNum 381 snap 110 seq_num 381) 2024-09-11T01:37:16.294 INFO:tasks.rados.rados.0.smithi071.stdout:1147: delete oid 37 current snap is 119 2024-09-11T01:37:16.294 INFO:tasks.rados.rados.0.smithi071.stdout:1144: finishing rollback tid 1 to smithi07134777-1 2024-09-11T01:37:16.294 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 431 (ObjNum 129 snap 0 seq_num 2370716288) dirty dne 2024-09-11T01:37:16.294 INFO:tasks.rados.rados.0.smithi071.stdout:1145: finishing write tid 1 to smithi07134777-4 2024-09-11T01:37:16.294 INFO:tasks.rados.rados.0.smithi071.stdout:1145: finishing write tid 2 to smithi07134777-4 2024-09-11T01:37:16.300 INFO:tasks.rados.rados.0.smithi071.stdout:1144: done (3 left) 2024-09-11T01:37:16.300 INFO:tasks.rados.rados.0.smithi071.stdout:1147: done (2 left) 2024-09-11T01:37:16.300 INFO:tasks.rados.rados.0.smithi071.stdout:1148: write oid 45 current snap is 119 2024-09-11T01:37:16.300 INFO:tasks.rados.rados.0.smithi071.stdout:1148: seq_num 437 ranges {544659=715934,1738984=449146,2784436=560223} 2024-09-11T01:37:16.306 INFO:tasks.rados.rados.0.smithi071.stdout:1148: writing smithi07134777-45 from 544659 to 1260593 tid 1 2024-09-11T01:37:16.312 INFO:tasks.rados.rados.0.smithi071.stdout:1148: writing smithi07134777-45 from 1738984 to 2188130 tid 2 2024-09-11T01:37:16.318 INFO:tasks.rados.rados.0.smithi071.stdout:1148: writing smithi07134777-45 from 2784436 to 3344659 tid 3 2024-09-11T01:37:16.318 INFO:tasks.rados.rados.0.smithi071.stdout:1149: write oid 27 current snap is 119 2024-09-11T01:37:16.318 INFO:tasks.rados.rados.0.smithi071.stdout:1146: expect (ObjNum 381 snap 110 seq_num 381) 2024-09-11T01:37:16.407 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 571835 2024-09-11T01:37:16.450 INFO:tasks.rados.rados.0.smithi071.stdout:1145: finishing write tid 3 to smithi07134777-4 2024-09-11T01:37:16.451 INFO:tasks.rados.rados.0.smithi071.stdout:1145: finishing write tid 4 to smithi07134777-4 2024-09-11T01:37:16.451 INFO:tasks.rados.rados.0.smithi071.stdout:1145: finishing write tid 5 to smithi07134777-4 2024-09-11T01:37:16.451 INFO:tasks.rados.rados.0.smithi071.stdout:1145: oid 4 updating version 0 to 420 2024-09-11T01:37:16.451 INFO:tasks.rados.rados.0.smithi071.stdout:1145: oid 4 version 420 is already newer than 419 2024-09-11T01:37:16.451 INFO:tasks.rados.rados.0.smithi071.stdout:1145: oid 4 updating version 420 to 421 2024-09-11T01:37:16.451 INFO:tasks.rados.rados.0.smithi071.stdout:1145: oid 4 version 421 is already newer than 418 2024-09-11T01:37:16.451 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 421 (ObjNum 435 snap 119 seq_num 435) dirty exists 2024-09-11T01:37:16.451 INFO:tasks.rados.rados.0.smithi071.stdout:1145: left oid 4 (ObjNum 435 snap 119 seq_num 435) 2024-09-11T01:37:16.451 INFO:tasks.rados.rados.0.smithi071.stdout:1148: finishing write tid 1 to smithi07134777-45 2024-09-11T01:37:16.451 INFO:tasks.rados.rados.0.smithi071.stdout:1148: finishing write tid 2 to smithi07134777-45 2024-09-11T01:37:16.451 INFO:tasks.rados.rados.0.smithi071.stdout:1148: finishing write tid 3 to smithi07134777-45 2024-09-11T01:37:16.451 INFO:tasks.rados.rados.0.smithi071.stdout:1148: finishing write tid 4 to smithi07134777-45 2024-09-11T01:37:16.452 INFO:tasks.rados.rados.0.smithi071.stdout:1148: finishing write tid 5 to smithi07134777-45 2024-09-11T01:37:16.452 INFO:tasks.rados.rados.0.smithi071.stdout:1148: oid 45 updating version 0 to 495 2024-09-11T01:37:16.452 INFO:tasks.rados.rados.0.smithi071.stdout:1148: oid 45 updating version 495 to 496 2024-09-11T01:37:16.452 INFO:tasks.rados.rados.0.smithi071.stdout:1148: oid 45 updating version 496 to 497 2024-09-11T01:37:16.452 INFO:tasks.rados.rados.0.smithi071.stdout:1148: oid 45 version 497 is already newer than 494 2024-09-11T01:37:16.452 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 497 (ObjNum 437 snap 119 seq_num 437) dirty exists 2024-09-11T01:37:16.452 INFO:tasks.rados.rados.0.smithi071.stdout:1148: left oid 45 (ObjNum 437 snap 119 seq_num 437) 2024-09-11T01:37:16.452 INFO:tasks.rados.rados.0.smithi071.stdout:1149: seq_num 438 ranges {561466=791183,2081113=592804,3095506=265960} 2024-09-11T01:37:16.457 INFO:tasks.rados.rados.0.smithi071.stdout:1149: writing smithi07134777-27 from 561466 to 1352649 tid 1 2024-09-11T01:37:16.464 INFO:tasks.rados.rados.0.smithi071.stdout:1149: writing smithi07134777-27 from 2081113 to 2673917 tid 2 2024-09-11T01:37:16.468 INFO:tasks.rados.rados.0.smithi071.stdout:1149: writing smithi07134777-27 from 3095506 to 3361466 tid 3 2024-09-11T01:37:16.468 INFO:tasks.rados.rados.0.smithi071.stdout:1145: done (3 left) 2024-09-11T01:37:16.468 INFO:tasks.rados.rados.0.smithi071.stdout:1146: done (2 left) 2024-09-11T01:37:16.468 INFO:tasks.rados.rados.0.smithi071.stdout:1148: done (1 left) 2024-09-11T01:37:16.469 INFO:tasks.rados.rados.0.smithi071.stdout:1150: snap_remove snap 118 2024-09-11T01:37:17.273 INFO:tasks.rados.rados.0.smithi071.stdout:1150: done (1 left) 2024-09-11T01:37:17.273 INFO:tasks.rados.rados.0.smithi071.stdout:1151: snap_create 2024-09-11T01:37:17.273 INFO:tasks.rados.rados.0.smithi071.stdout:1149: finishing write tid 1 to smithi07134777-27 2024-09-11T01:37:17.273 INFO:tasks.rados.rados.0.smithi071.stdout:1149: finishing write tid 2 to smithi07134777-27 2024-09-11T01:37:17.274 INFO:tasks.rados.rados.0.smithi071.stdout:1149: finishing write tid 3 to smithi07134777-27 2024-09-11T01:37:17.274 INFO:tasks.rados.rados.0.smithi071.stdout:1149: finishing write tid 4 to smithi07134777-27 2024-09-11T01:37:17.274 INFO:tasks.rados.rados.0.smithi071.stdout:1149: finishing write tid 5 to smithi07134777-27 2024-09-11T01:37:17.274 INFO:tasks.rados.rados.0.smithi071.stdout:1149: oid 27 updating version 0 to 526 2024-09-11T01:37:17.274 INFO:tasks.rados.rados.0.smithi071.stdout:1149: oid 27 version 526 is already newer than 524 2024-09-11T01:37:17.274 INFO:tasks.rados.rados.0.smithi071.stdout:1149: oid 27 version 526 is already newer than 525 2024-09-11T01:37:17.274 INFO:tasks.rados.rados.0.smithi071.stdout:1149: oid 27 updating version 526 to 527 2024-09-11T01:37:17.274 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 527 (ObjNum 438 snap 119 seq_num 438) dirty exists 2024-09-11T01:37:17.274 INFO:tasks.rados.rados.0.smithi071.stdout:1149: left oid 27 (ObjNum 438 snap 119 seq_num 438) 2024-09-11T01:37:18.238 INFO:tasks.rados.rados.0.smithi071.stdout:1149: done (1 left) 2024-09-11T01:37:18.238 INFO:tasks.rados.rados.0.smithi071.stdout:1151: done (0 left) 2024-09-11T01:37:18.238 INFO:tasks.rados.rados.0.smithi071.stdout:1152: read oid 11 snap -1 2024-09-11T01:37:18.238 INFO:tasks.rados.rados.0.smithi071.stdout:1152: expect (ObjNum 381 snap 110 seq_num 381) 2024-09-11T01:37:18.239 INFO:tasks.rados.rados.0.smithi071.stdout:1153: snap_create 2024-09-11T01:37:18.246 INFO:tasks.rados.rados.0.smithi071.stdout:1152: expect (ObjNum 381 snap 110 seq_num 381) 2024-09-11T01:37:18.320 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1353638 2024-09-11T01:37:19.240 INFO:tasks.rados.rados.0.smithi071.stdout:1152: done (1 left) 2024-09-11T01:37:19.240 INFO:tasks.rados.rados.0.smithi071.stdout:1153: done (0 left) 2024-09-11T01:37:19.240 INFO:tasks.rados.rados.0.smithi071.stdout:1154: read oid 10 snap -1 2024-09-11T01:37:19.241 INFO:tasks.rados.rados.0.smithi071.stdout:1154: expect (ObjNum 432 snap 119 seq_num 432) 2024-09-11T01:37:19.241 INFO:tasks.rados.rados.0.smithi071.stdout:1155: write (excl) oid 42 current snap is 121 2024-09-11T01:37:19.241 INFO:tasks.rados.rados.0.smithi071.stdout:1155: seq_num 439 ranges {578273=466432,1739595=452815,2722929=655344} 2024-09-11T01:37:19.245 INFO:tasks.rados.rados.0.smithi071.stdout:1155: writing smithi07134777-42 from 578273 to 1044705 tid 1 2024-09-11T01:37:19.252 INFO:tasks.rados.rados.0.smithi071.stdout:1155: writing smithi07134777-42 from 1739595 to 2192410 tid 2 2024-09-11T01:37:19.259 INFO:tasks.rados.rados.0.smithi071.stdout:1155: writing smithi07134777-42 from 2722929 to 3378273 tid 3 2024-09-11T01:37:19.259 INFO:tasks.rados.rados.0.smithi071.stdout:1156: read oid 21 snap 114 2024-09-11T01:37:19.259 INFO:tasks.rados.rados.0.smithi071.stdout:1156: expect (ObjNum 362 snap 101 seq_num 362) 2024-09-11T01:37:19.259 INFO:tasks.rados.rados.0.smithi071.stdout:1157: read oid 40 snap -1 2024-09-11T01:37:19.259 INFO:tasks.rados.rados.0.smithi071.stdout:1157: expect (ObjNum 341 snap 96 seq_num 341) 2024-09-11T01:37:19.259 INFO:tasks.rados.rados.0.smithi071.stdout:1158: rollback oid 31 current snap is 121 2024-09-11T01:37:19.259 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 31 to 119 2024-09-11T01:37:19.259 INFO:tasks.rados.rados.0.smithi071.stdout:1154: expect (ObjNum 432 snap 119 seq_num 432) 2024-09-11T01:37:19.276 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1870030 2024-09-11T01:37:19.303 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 523135 2024-09-11T01:37:19.396 INFO:tasks.rados.rados.0.smithi071.stdout:1155: finishing write tid 1 to smithi07134777-42 2024-09-11T01:37:19.396 INFO:tasks.rados.rados.0.smithi071.stdout:1155: finishing write tid 2 to smithi07134777-42 2024-09-11T01:37:19.396 INFO:tasks.rados.rados.0.smithi071.stdout:1158: finishing rollback tid 0 to smithi07134777-31 2024-09-11T01:37:19.396 INFO:tasks.rados.rados.0.smithi071.stdout:1157: expect (ObjNum 341 snap 96 seq_num 341) 2024-09-11T01:37:19.435 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:37:19.509 INFO:tasks.rados.rados.0.smithi071.stdout:1155: finishing write tid 3 to smithi07134777-42 2024-09-11T01:37:19.510 INFO:tasks.rados.rados.0.smithi071.stdout:1155: finishing write tid 4 to smithi07134777-42 2024-09-11T01:37:19.510 INFO:tasks.rados.rados.0.smithi071.stdout:1158: finishing rollback tid 1 to smithi07134777-31 2024-09-11T01:37:19.510 INFO:tasks.rados.rados.0.smithi071.stdout:1155: finishing write tid 5 to smithi07134777-42 2024-09-11T01:37:19.510 INFO:tasks.rados.rados.0.smithi071.stdout:1155: oid 42 updating version 0 to 414 2024-09-11T01:37:19.510 INFO:tasks.rados.rados.0.smithi071.stdout:1155: oid 42 version 414 is already newer than 412 2024-09-11T01:37:19.510 INFO:tasks.rados.rados.0.smithi071.stdout:1155: oid 42 updating version 414 to 415 2024-09-11T01:37:19.510 INFO:tasks.rados.rados.0.smithi071.stdout:1155: oid 42 version 415 is already newer than 413 2024-09-11T01:37:19.510 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 415 (ObjNum 439 snap 121 seq_num 439) dirty exists 2024-09-11T01:37:19.510 INFO:tasks.rados.rados.0.smithi071.stdout:1155: left oid 42 (ObjNum 439 snap 121 seq_num 439) 2024-09-11T01:37:19.510 INFO:tasks.rados.rados.0.smithi071.stdout:1154: done (4 left) 2024-09-11T01:37:19.510 INFO:tasks.rados.rados.0.smithi071.stdout:1155: done (3 left) 2024-09-11T01:37:19.510 INFO:tasks.rados.rados.0.smithi071.stdout:1157: done (2 left) 2024-09-11T01:37:19.510 INFO:tasks.rados.rados.0.smithi071.stdout:1159: snap_remove snap 120 2024-09-11T01:37:20.277 INFO:tasks.rados.rados.0.smithi071.stdout:1159: done (2 left) 2024-09-11T01:37:20.278 INFO:tasks.rados.rados.0.smithi071.stdout:1160: delete oid 7 current snap is 121 2024-09-11T01:37:20.278 INFO:tasks.rados.rados.0.smithi071.stdout:1156: expect (ObjNum 362 snap 101 seq_num 362) 2024-09-11T01:37:20.317 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 424422 2024-09-11T01:37:20.391 INFO:tasks.rados.rados.0.smithi071.stdout:1158: finishing rollback tid 2 to smithi07134777-31 2024-09-11T01:37:20.391 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 434 (ObjNum 250 snap 63 seq_num 250) dirty exists 2024-09-11T01:37:20.391 INFO:tasks.rados.rados.0.smithi071.stdout:1156: done (2 left) 2024-09-11T01:37:20.391 INFO:tasks.rados.rados.0.smithi071.stdout:1158: done (1 left) 2024-09-11T01:37:20.391 INFO:tasks.rados.rados.0.smithi071.stdout:1160: done (0 left) 2024-09-11T01:37:20.391 INFO:tasks.rados.rados.0.smithi071.stdout:1161: copy_from oid 39 from oid 30 current snap is 121 2024-09-11T01:37:20.391 INFO:tasks.rados.rados.0.smithi071.stdout:1162: write oid 41 current snap is 121 2024-09-11T01:37:20.391 INFO:tasks.rados.rados.0.smithi071.stdout:1162: seq_num 441 ranges {611887=733283,1972912=456484,2894128=517759} 2024-09-11T01:37:20.398 INFO:tasks.rados.rados.0.smithi071.stdout:1162: writing smithi07134777-41 from 611887 to 1345170 tid 1 2024-09-11T01:37:20.404 INFO:tasks.rados.rados.0.smithi071.stdout:1162: writing smithi07134777-41 from 1972912 to 2429396 tid 2 2024-09-11T01:37:20.409 INFO:tasks.rados.rados.0.smithi071.stdout:1162: writing smithi07134777-41 from 2894128 to 3411887 tid 3 2024-09-11T01:37:20.410 INFO:tasks.rados.rados.0.smithi071.stdout:1163: snap_remove snap 114 2024-09-11T01:37:21.284 INFO:tasks.rados.rados.0.smithi071.stdout:1163: done (2 left) 2024-09-11T01:37:21.284 INFO:tasks.rados.rados.0.smithi071.stdout:1164: delete oid 19 current snap is 121 2024-09-11T01:37:21.284 INFO:tasks.rados.rados.0.smithi071.stdout:1162: finishing write tid 1 to smithi07134777-41 2024-09-11T01:37:21.284 INFO:tasks.rados.rados.0.smithi071.stdout:1162: finishing write tid 2 to smithi07134777-41 2024-09-11T01:37:21.284 INFO:tasks.rados.rados.0.smithi071.stdout:1161: finishing copy_from racing read to smithi07134777-39 2024-09-11T01:37:21.284 INFO:tasks.rados.rados.0.smithi071.stdout:1161: finishing copy_from to smithi07134777-39 2024-09-11T01:37:21.284 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 417 (ObjNum 431 snap 119 seq_num 431) dirty exists 2024-09-11T01:37:21.284 INFO:tasks.rados.rados.0.smithi071.stdout:1162: finishing write tid 3 to smithi07134777-41 2024-09-11T01:37:21.285 INFO:tasks.rados.rados.0.smithi071.stdout:1162: finishing write tid 4 to smithi07134777-41 2024-09-11T01:37:21.285 INFO:tasks.rados.rados.0.smithi071.stdout:1162: finishing write tid 5 to smithi07134777-41 2024-09-11T01:37:21.285 INFO:tasks.rados.rados.0.smithi071.stdout:1162: oid 41 updating version 0 to 346 2024-09-11T01:37:21.285 INFO:tasks.rados.rados.0.smithi071.stdout:1162: oid 41 version 346 is already newer than 345 2024-09-11T01:37:21.285 INFO:tasks.rados.rados.0.smithi071.stdout:1162: oid 41 updating version 346 to 347 2024-09-11T01:37:21.285 INFO:tasks.rados.rados.0.smithi071.stdout:1162: oid 41 version 347 is already newer than 344 2024-09-11T01:37:21.285 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 347 (ObjNum 441 snap 121 seq_num 441) dirty exists 2024-09-11T01:37:21.285 INFO:tasks.rados.rados.0.smithi071.stdout:1162: left oid 41 (ObjNum 441 snap 121 seq_num 441) 2024-09-11T01:37:21.299 INFO:tasks.rados.rados.0.smithi071.stdout:1161: done (2 left) 2024-09-11T01:37:21.299 INFO:tasks.rados.rados.0.smithi071.stdout:1162: done (1 left) 2024-09-11T01:37:21.299 INFO:tasks.rados.rados.0.smithi071.stdout:1164: done (0 left) 2024-09-11T01:37:21.299 INFO:tasks.rados.rados.0.smithi071.stdout:1165: write oid 34 current snap is 121 2024-09-11T01:37:21.299 INFO:tasks.rados.rados.0.smithi071.stdout:1165: seq_num 443 ranges {645501=483781,1573523=460153,2548974=763384} 2024-09-11T01:37:21.305 INFO:tasks.rados.rados.0.smithi071.stdout:1165: writing smithi07134777-34 from 645501 to 1129282 tid 1 2024-09-11T01:37:21.311 INFO:tasks.rados.rados.0.smithi071.stdout:1165: writing smithi07134777-34 from 1573523 to 2033676 tid 2 2024-09-11T01:37:21.318 INFO:tasks.rados.rados.0.smithi071.stdout:1165: writing smithi07134777-34 from 2548974 to 3312358 tid 3 2024-09-11T01:37:21.318 INFO:tasks.rados.rados.0.smithi071.stdout:1166: read oid 24 snap -1 2024-09-11T01:37:21.318 INFO:tasks.rados.rados.0.smithi071.stdout:1166: expect (ObjNum 430 snap 119 seq_num 430) 2024-09-11T01:37:21.318 INFO:tasks.rados.rados.0.smithi071.stdout:1167: write oid 33 current snap is 121 2024-09-11T01:37:21.318 INFO:tasks.rados.rados.0.smithi071.stdout:1167: seq_num 444 ranges {662308=559030,1632005=603811,2860044=602264} 2024-09-11T01:37:21.325 INFO:tasks.rados.rados.0.smithi071.stdout:1167: writing smithi07134777-33 from 662308 to 1221338 tid 1 2024-09-11T01:37:21.330 INFO:tasks.rados.rados.0.smithi071.stdout:1167: writing smithi07134777-33 from 1632005 to 2235816 tid 2 2024-09-11T01:37:21.337 INFO:tasks.rados.rados.0.smithi071.stdout:1167: writing smithi07134777-33 from 2860044 to 3462308 tid 3 2024-09-11T01:37:21.337 INFO:tasks.rados.rados.0.smithi071.stdout:1168: write (excl) oid 9 current snap is 121 2024-09-11T01:37:21.338 INFO:tasks.rados.rados.0.smithi071.stdout:1165: finishing write tid 1 to smithi07134777-34 2024-09-11T01:37:21.338 INFO:tasks.rados.rados.0.smithi071.stdout:1168: seq_num 445 ranges {679115=634279,2090487=463822,3003820=475295} 2024-09-11T01:37:21.345 INFO:tasks.rados.rados.0.smithi071.stdout:1168: writing smithi07134777-9 from 679115 to 1313394 tid 1 2024-09-11T01:37:21.352 INFO:tasks.rados.rados.0.smithi071.stdout:1168: writing smithi07134777-9 from 2090487 to 2554309 tid 2 2024-09-11T01:37:21.357 INFO:tasks.rados.rados.0.smithi071.stdout:1168: writing smithi07134777-9 from 3003820 to 3479115 tid 3 2024-09-11T01:37:21.357 INFO:tasks.rados.rados.0.smithi071.stdout:1169: read oid 29 snap -1 2024-09-11T01:37:21.357 INFO:tasks.rados.rados.0.smithi071.stdout:1169: expect (ObjNum 226 snap 55 seq_num 226) 2024-09-11T01:37:21.357 INFO:tasks.rados.rados.0.smithi071.stdout:1170: write oid 27 current snap is 121 2024-09-11T01:37:21.357 INFO:tasks.rados.rados.0.smithi071.stdout:1165: finishing write tid 2 to smithi07134777-34 2024-09-11T01:37:21.357 INFO:tasks.rados.rados.0.smithi071.stdout:1170: seq_num 446 ranges {695922=709528,2032616=607480,3198537=297385} 2024-09-11T01:37:21.364 INFO:tasks.rados.rados.0.smithi071.stdout:1170: writing smithi07134777-27 from 695922 to 1405450 tid 1 2024-09-11T01:37:21.371 INFO:tasks.rados.rados.0.smithi071.stdout:1170: writing smithi07134777-27 from 2032616 to 2640096 tid 2 2024-09-11T01:37:21.376 INFO:tasks.rados.rados.0.smithi071.stdout:1170: writing smithi07134777-27 from 3198537 to 3495922 tid 3 2024-09-11T01:37:21.376 INFO:tasks.rados.rados.0.smithi071.stdout:1171: read oid 48 snap -1 2024-09-11T01:37:21.376 INFO:tasks.rados.rados.0.smithi071.stdout:1171: expect deleted 2024-09-11T01:37:21.376 INFO:tasks.rados.rados.0.smithi071.stdout:1172: write oid 42 current snap is 121 2024-09-11T01:37:21.376 INFO:tasks.rados.rados.0.smithi071.stdout:1172: seq_num 447 ranges {712729=784777,2091098=467491,3342313=170416} 2024-09-11T01:37:21.384 INFO:tasks.rados.rados.0.smithi071.stdout:1172: writing smithi07134777-42 from 712729 to 1497506 tid 1 2024-09-11T01:37:21.390 INFO:tasks.rados.rados.0.smithi071.stdout:1172: writing smithi07134777-42 from 2091098 to 2558589 tid 2 2024-09-11T01:37:21.395 INFO:tasks.rados.rados.0.smithi071.stdout:1172: writing smithi07134777-42 from 3342313 to 3512729 tid 3 2024-09-11T01:37:21.395 INFO:tasks.rados.rados.0.smithi071.stdout:1173: delete oid 12 current snap is 121 2024-09-11T01:37:21.396 INFO:tasks.rados.rados.0.smithi071.stdout:1165: finishing write tid 3 to smithi07134777-34 2024-09-11T01:37:21.396 INFO:tasks.rados.rados.0.smithi071.stdout:1165: finishing write tid 4 to smithi07134777-34 2024-09-11T01:37:21.396 INFO:tasks.rados.rados.0.smithi071.stdout:1165: finishing write tid 5 to smithi07134777-34 2024-09-11T01:37:21.396 INFO:tasks.rados.rados.0.smithi071.stdout:1165: oid 34 updating version 0 to 540 2024-09-11T01:37:21.396 INFO:tasks.rados.rados.0.smithi071.stdout:1165: oid 34 version 540 is already newer than 539 2024-09-11T01:37:21.396 INFO:tasks.rados.rados.0.smithi071.stdout:1165: oid 34 version 540 is already newer than 533 2024-09-11T01:37:21.396 INFO:tasks.rados.rados.0.smithi071.stdout:1165: oid 34 version 540 is already newer than 534 2024-09-11T01:37:21.396 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 540 (ObjNum 443 snap 121 seq_num 443) dirty exists 2024-09-11T01:37:21.396 INFO:tasks.rados.rados.0.smithi071.stdout:1165: left oid 34 (ObjNum 443 snap 121 seq_num 443) 2024-09-11T01:37:21.396 INFO:tasks.rados.rados.0.smithi071.stdout:1167: finishing write tid 1 to smithi07134777-33 2024-09-11T01:37:21.396 INFO:tasks.rados.rados.0.smithi071.stdout:1166: expect (ObjNum 430 snap 119 seq_num 430) 2024-09-11T01:37:21.403 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 933636 2024-09-11T01:37:21.447 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:37:21.503 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:37:21.559 INFO:tasks.rados.rados.0.smithi071.stdout:1167: finishing write tid 2 to smithi07134777-33 2024-09-11T01:37:21.560 INFO:tasks.rados.rados.0.smithi071.stdout:1167: finishing write tid 3 to smithi07134777-33 2024-09-11T01:37:21.560 INFO:tasks.rados.rados.0.smithi071.stdout:1167: finishing write tid 4 to smithi07134777-33 2024-09-11T01:37:21.560 INFO:tasks.rados.rados.0.smithi071.stdout:1167: finishing write tid 5 to smithi07134777-33 2024-09-11T01:37:21.560 INFO:tasks.rados.rados.0.smithi071.stdout:1167: oid 33 updating version 0 to 544 2024-09-11T01:37:21.560 INFO:tasks.rados.rados.0.smithi071.stdout:1167: oid 33 updating version 544 to 545 2024-09-11T01:37:21.560 INFO:tasks.rados.rados.0.smithi071.stdout:1167: oid 33 version 545 is already newer than 542 2024-09-11T01:37:21.560 INFO:tasks.rados.rados.0.smithi071.stdout:1167: oid 33 version 545 is already newer than 543 2024-09-11T01:37:21.560 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 545 (ObjNum 444 snap 121 seq_num 444) dirty exists 2024-09-11T01:37:21.560 INFO:tasks.rados.rados.0.smithi071.stdout:1167: left oid 33 (ObjNum 444 snap 121 seq_num 444) 2024-09-11T01:37:21.560 INFO:tasks.rados.rados.0.smithi071.stdout:1168: finishing write tid 1 to smithi07134777-9 2024-09-11T01:37:21.560 INFO:tasks.rados.rados.0.smithi071.stdout:1168: finishing write tid 2 to smithi07134777-9 2024-09-11T01:37:21.561 INFO:tasks.rados.rados.0.smithi071.stdout:1165: done (8 left) 2024-09-11T01:37:21.561 INFO:tasks.rados.rados.0.smithi071.stdout:1166: done (7 left) 2024-09-11T01:37:21.561 INFO:tasks.rados.rados.0.smithi071.stdout:1167: done (6 left) 2024-09-11T01:37:21.561 INFO:tasks.rados.rados.0.smithi071.stdout:1173: done (5 left) 2024-09-11T01:37:21.561 INFO:tasks.rados.rados.0.smithi071.stdout:1174: write oid 36 current snap is 121 2024-09-11T01:37:21.561 INFO:tasks.rados.rados.0.smithi071.stdout:1169: expect (ObjNum 226 snap 55 seq_num 226) 2024-09-11T01:37:21.563 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 771929 2024-09-11T01:37:21.594 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:37:21.669 INFO:tasks.rados.rados.0.smithi071.stdout:1168: finishing write tid 3 to smithi07134777-9 2024-09-11T01:37:21.669 INFO:tasks.rados.rados.0.smithi071.stdout:1168: finishing write tid 4 to smithi07134777-9 2024-09-11T01:37:21.669 INFO:tasks.rados.rados.0.smithi071.stdout:1168: finishing write tid 5 to smithi07134777-9 2024-09-11T01:37:21.669 INFO:tasks.rados.rados.0.smithi071.stdout:1168: oid 9 updating version 0 to 152 2024-09-11T01:37:21.669 INFO:tasks.rados.rados.0.smithi071.stdout:1168: oid 9 updating version 152 to 155 2024-09-11T01:37:21.669 INFO:tasks.rados.rados.0.smithi071.stdout:1168: oid 9 version 155 is already newer than 153 2024-09-11T01:37:21.669 INFO:tasks.rados.rados.0.smithi071.stdout:1168: oid 9 version 155 is already newer than 154 2024-09-11T01:37:21.669 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 155 (ObjNum 445 snap 121 seq_num 445) dirty exists 2024-09-11T01:37:21.669 INFO:tasks.rados.rados.0.smithi071.stdout:1168: left oid 9 (ObjNum 445 snap 121 seq_num 445) 2024-09-11T01:37:21.669 INFO:tasks.rados.rados.0.smithi071.stdout:1170: finishing write tid 1 to smithi07134777-27 2024-09-11T01:37:21.669 INFO:tasks.rados.rados.0.smithi071.stdout:1170: finishing write tid 2 to smithi07134777-27 2024-09-11T01:37:21.670 INFO:tasks.rados.rados.0.smithi071.stdout:1174: seq_num 449 ranges {746343=651628,1924415=754807,3397159=149184} 2024-09-11T01:37:21.676 INFO:tasks.rados.rados.0.smithi071.stdout:1174: writing smithi07134777-36 from 746343 to 1397971 tid 1 2024-09-11T01:37:21.683 INFO:tasks.rados.rados.0.smithi071.stdout:1174: writing smithi07134777-36 from 1924415 to 2679222 tid 2 2024-09-11T01:37:21.688 INFO:tasks.rados.rados.0.smithi071.stdout:1174: writing smithi07134777-36 from 3397159 to 3546343 tid 3 2024-09-11T01:37:21.688 INFO:tasks.rados.rados.0.smithi071.stdout:1168: done (5 left) 2024-09-11T01:37:21.688 INFO:tasks.rados.rados.0.smithi071.stdout:1169: done (4 left) 2024-09-11T01:37:21.688 INFO:tasks.rados.rados.0.smithi071.stdout:1175: copy_from oid 16 from oid 6 current snap is 121 2024-09-11T01:37:21.688 INFO:tasks.rados.rados.0.smithi071.stdout:1170: finishing write tid 3 to smithi07134777-27 2024-09-11T01:37:21.688 INFO:tasks.rados.rados.0.smithi071.stdout:1170: finishing write tid 4 to smithi07134777-27 2024-09-11T01:37:21.688 INFO:tasks.rados.rados.0.smithi071.stdout:1170: finishing write tid 5 to smithi07134777-27 2024-09-11T01:37:21.689 INFO:tasks.rados.rados.0.smithi071.stdout:1170: oid 27 updating version 0 to 550 2024-09-11T01:37:21.689 INFO:tasks.rados.rados.0.smithi071.stdout:1170: oid 27 version 550 is already newer than 548 2024-09-11T01:37:21.689 INFO:tasks.rados.rados.0.smithi071.stdout:1170: oid 27 version 550 is already newer than 547 2024-09-11T01:37:21.689 INFO:tasks.rados.rados.0.smithi071.stdout:1170: oid 27 version 550 is already newer than 549 2024-09-11T01:37:21.689 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 550 (ObjNum 446 snap 121 seq_num 446) dirty exists 2024-09-11T01:37:21.689 INFO:tasks.rados.rados.0.smithi071.stdout:1170: left oid 27 (ObjNum 446 snap 121 seq_num 446) 2024-09-11T01:37:21.689 INFO:tasks.rados.rados.0.smithi071.stdout:1172: finishing write tid 1 to smithi07134777-42 2024-09-11T01:37:21.689 INFO:tasks.rados.rados.0.smithi071.stdout:1172: finishing write tid 2 to smithi07134777-42 2024-09-11T01:37:21.689 INFO:tasks.rados.rados.0.smithi071.stdout:1172: finishing write tid 3 to smithi07134777-42 2024-09-11T01:37:21.689 INFO:tasks.rados.rados.0.smithi071.stdout:1172: finishing write tid 4 to smithi07134777-42 2024-09-11T01:37:21.689 INFO:tasks.rados.rados.0.smithi071.stdout:1172: finishing write tid 5 to smithi07134777-42 2024-09-11T01:37:21.689 INFO:tasks.rados.rados.0.smithi071.stdout:1172: oid 42 updating version 0 to 419 2024-09-11T01:37:21.689 INFO:tasks.rados.rados.0.smithi071.stdout:1172: oid 42 version 419 is already newer than 417 2024-09-11T01:37:21.690 INFO:tasks.rados.rados.0.smithi071.stdout:1172: oid 42 version 419 is already newer than 416 2024-09-11T01:37:21.690 INFO:tasks.rados.rados.0.smithi071.stdout:1172: oid 42 version 419 is already newer than 418 2024-09-11T01:37:21.690 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 419 (ObjNum 447 snap 121 seq_num 447) dirty exists 2024-09-11T01:37:21.690 INFO:tasks.rados.rados.0.smithi071.stdout:1172: left oid 42 (ObjNum 447 snap 121 seq_num 447) 2024-09-11T01:37:21.690 INFO:tasks.rados.rados.0.smithi071.stdout:1174: finishing write tid 1 to smithi07134777-36 2024-09-11T01:37:21.690 INFO:tasks.rados.rados.0.smithi071.stdout:1170: done (4 left) 2024-09-11T01:37:21.690 INFO:tasks.rados.rados.0.smithi071.stdout:1171: done (3 left) 2024-09-11T01:37:21.690 INFO:tasks.rados.rados.0.smithi071.stdout:1172: done (2 left) 2024-09-11T01:37:21.690 INFO:tasks.rados.rados.0.smithi071.stdout:1176: snap_remove snap 119 2024-09-11T01:37:22.283 INFO:tasks.rados.rados.0.smithi071.stdout:1175: finishing copy_from to smithi07134777-16 2024-09-11T01:37:22.284 INFO:tasks.rados.rados.0.smithi071.stdout:1175: got expected ENOENT (src dne) 2024-09-11T01:37:22.284 INFO:tasks.rados.rados.0.smithi071.stdout:1176: done (2 left) 2024-09-11T01:37:22.284 INFO:tasks.rados.rados.0.smithi071.stdout:1177: copy_from oid 21 from oid 7 current snap is 121 2024-09-11T01:37:22.284 INFO:tasks.rados.rados.0.smithi071.stdout:1175: finishing copy_from racing read to smithi07134777-16 2024-09-11T01:37:22.284 INFO:tasks.rados.rados.0.smithi071.stdout:1174: finishing write tid 2 to smithi07134777-36 2024-09-11T01:37:22.284 INFO:tasks.rados.rados.0.smithi071.stdout:1174: finishing write tid 3 to smithi07134777-36 2024-09-11T01:37:22.284 INFO:tasks.rados.rados.0.smithi071.stdout:1174: finishing write tid 4 to smithi07134777-36 2024-09-11T01:37:22.284 INFO:tasks.rados.rados.0.smithi071.stdout:1174: finishing write tid 5 to smithi07134777-36 2024-09-11T01:37:22.284 INFO:tasks.rados.rados.0.smithi071.stdout:1174: oid 36 updating version 0 to 431 2024-09-11T01:37:22.284 INFO:tasks.rados.rados.0.smithi071.stdout:1174: oid 36 updating version 431 to 432 2024-09-11T01:37:22.284 INFO:tasks.rados.rados.0.smithi071.stdout:1174: oid 36 updating version 432 to 434 2024-09-11T01:37:22.284 INFO:tasks.rados.rados.0.smithi071.stdout:1174: oid 36 version 434 is already newer than 433 2024-09-11T01:37:22.284 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 434 (ObjNum 449 snap 121 seq_num 449) dirty exists 2024-09-11T01:37:22.285 INFO:tasks.rados.rados.0.smithi071.stdout:1174: left oid 36 (ObjNum 449 snap 121 seq_num 449) 2024-09-11T01:37:22.285 INFO:tasks.rados.rados.0.smithi071.stdout:1174: done (2 left) 2024-09-11T01:37:22.285 INFO:tasks.rados.rados.0.smithi071.stdout:1175: done (1 left) 2024-09-11T01:37:22.285 INFO:tasks.rados.rados.0.smithi071.stdout:1178: delete oid 3 current snap is 121 2024-09-11T01:37:22.291 INFO:tasks.rados.rados.0.smithi071.stdout:1177: finishing copy_from to smithi07134777-21 2024-09-11T01:37:22.292 INFO:tasks.rados.rados.0.smithi071.stdout:1177: got expected ENOENT (src dne) 2024-09-11T01:37:22.292 INFO:tasks.rados.rados.0.smithi071.stdout:1177: finishing copy_from racing read to smithi07134777-21 2024-09-11T01:37:22.300 INFO:tasks.rados.rados.0.smithi071.stdout:1177: done (1 left) 2024-09-11T01:37:22.301 INFO:tasks.rados.rados.0.smithi071.stdout:1178: done (0 left) 2024-09-11T01:37:22.301 INFO:tasks.rados.rados.0.smithi071.stdout:1179: read oid 31 snap -1 2024-09-11T01:37:22.301 INFO:tasks.rados.rados.0.smithi071.stdout:1179: expect (ObjNum 250 snap 63 seq_num 250) 2024-09-11T01:37:22.301 INFO:tasks.rados.rados.0.smithi071.stdout:1180: read oid 25 snap -1 2024-09-11T01:37:22.301 INFO:tasks.rados.rados.0.smithi071.stdout:1180: expect deleted 2024-09-11T01:37:22.301 INFO:tasks.rados.rados.0.smithi071.stdout:1181: write (excl) oid 13 current snap is 121 2024-09-11T01:37:22.301 INFO:tasks.rados.rados.0.smithi071.stdout:1181: seq_num 451 ranges {779957=402126,1925026=758476,3335652=244305} 2024-09-11T01:37:22.307 INFO:tasks.rados.rados.0.smithi071.stdout:1181: writing smithi07134777-13 from 779957 to 1182083 tid 1 2024-09-11T01:37:22.315 INFO:tasks.rados.rados.0.smithi071.stdout:1181: writing smithi07134777-13 from 1925026 to 2683502 tid 2 2024-09-11T01:37:22.320 INFO:tasks.rados.rados.0.smithi071.stdout:1181: writing smithi07134777-13 from 3335652 to 3579957 tid 3 2024-09-11T01:37:22.320 INFO:tasks.rados.rados.0.smithi071.stdout:1182: read oid 19 snap -1 2024-09-11T01:37:22.320 INFO:tasks.rados.rados.0.smithi071.stdout:1182: expect deleted 2024-09-11T01:37:22.321 INFO:tasks.rados.rados.0.smithi071.stdout:1183: write (excl) oid 2 current snap is 121 2024-09-11T01:37:22.321 INFO:tasks.rados.rados.0.smithi071.stdout:1183: seq_num 452 ranges {796764=477375,1983508=618487,3363075=233689} 2024-09-11T01:37:22.327 INFO:tasks.rados.rados.0.smithi071.stdout:1183: writing smithi07134777-2 from 796764 to 1274139 tid 1 2024-09-11T01:37:22.334 INFO:tasks.rados.rados.0.smithi071.stdout:1183: writing smithi07134777-2 from 1983508 to 2601995 tid 2 2024-09-11T01:37:22.340 INFO:tasks.rados.rados.0.smithi071.stdout:1183: writing smithi07134777-2 from 3363075 to 3596764 tid 3 2024-09-11T01:37:22.340 INFO:tasks.rados.rados.0.smithi071.stdout:1184: write oid 32 current snap is 121 2024-09-11T01:37:22.340 INFO:tasks.rados.rados.0.smithi071.stdout:1184: seq_num 453 ranges {413571=552624,1641990=762145,2990498=425163} 2024-09-11T01:37:22.345 INFO:tasks.rados.rados.0.smithi071.stdout:1184: writing smithi07134777-32 from 413571 to 966195 tid 1 2024-09-11T01:37:22.353 INFO:tasks.rados.rados.0.smithi071.stdout:1184: writing smithi07134777-32 from 1641990 to 2404135 tid 2 2024-09-11T01:37:22.359 INFO:tasks.rados.rados.0.smithi071.stdout:1184: writing smithi07134777-32 from 2990498 to 3415661 tid 3 2024-09-11T01:37:22.359 INFO:tasks.rados.rados.0.smithi071.stdout:1185: write oid 34 current snap is 121 2024-09-11T01:37:22.359 INFO:tasks.rados.rados.0.smithi071.stdout:1179: expect (ObjNum 250 snap 63 seq_num 250) 2024-09-11T01:37:22.359 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:37:22.486 INFO:tasks.rados.rados.0.smithi071.stdout:1181: finishing write tid 1 to smithi07134777-13 2024-09-11T01:37:22.486 INFO:tasks.rados.rados.0.smithi071.stdout:1181: finishing write tid 2 to smithi07134777-13 2024-09-11T01:37:22.486 INFO:tasks.rados.rados.0.smithi071.stdout:1185: seq_num 454 ranges {430378=627873,1700472=622156,3017921=612457} 2024-09-11T01:37:22.491 INFO:tasks.rados.rados.0.smithi071.stdout:1185: writing smithi07134777-34 from 430378 to 1058251 tid 1 2024-09-11T01:37:22.498 INFO:tasks.rados.rados.0.smithi071.stdout:1185: writing smithi07134777-34 from 1700472 to 2322628 tid 2 2024-09-11T01:37:22.505 INFO:tasks.rados.rados.0.smithi071.stdout:1185: writing smithi07134777-34 from 3017921 to 3630378 tid 3 2024-09-11T01:37:22.505 INFO:tasks.rados.rados.0.smithi071.stdout:1179: done (6 left) 2024-09-11T01:37:22.505 INFO:tasks.rados.rados.0.smithi071.stdout:1180: done (5 left) 2024-09-11T01:37:22.505 INFO:tasks.rados.rados.0.smithi071.stdout:1186: write oid 15 current snap is 121 2024-09-11T01:37:22.505 INFO:tasks.rados.rados.0.smithi071.stdout:1181: finishing write tid 3 to smithi07134777-13 2024-09-11T01:37:22.506 INFO:tasks.rados.rados.0.smithi071.stdout:1181: finishing write tid 4 to smithi07134777-13 2024-09-11T01:37:22.506 INFO:tasks.rados.rados.0.smithi071.stdout:1186: seq_num 455 ranges {447185=703122,1642601=765814,2928991=447707} 2024-09-11T01:37:22.511 INFO:tasks.rados.rados.0.smithi071.stdout:1186: writing smithi07134777-15 from 447185 to 1150307 tid 1 2024-09-11T01:37:22.518 INFO:tasks.rados.rados.0.smithi071.stdout:1186: writing smithi07134777-15 from 1642601 to 2408415 tid 2 2024-09-11T01:37:22.524 INFO:tasks.rados.rados.0.smithi071.stdout:1186: writing smithi07134777-15 from 2928991 to 3376698 tid 3 2024-09-11T01:37:22.524 INFO:tasks.rados.rados.0.smithi071.stdout:1182: done (5 left) 2024-09-11T01:37:22.524 INFO:tasks.rados.rados.0.smithi071.stdout:1187: delete oid 33 current snap is 121 2024-09-11T01:37:22.524 INFO:tasks.rados.rados.0.smithi071.stdout:1181: finishing write tid 5 to smithi07134777-13 2024-09-11T01:37:22.525 INFO:tasks.rados.rados.0.smithi071.stdout:1181: oid 13 updating version 0 to 439 2024-09-11T01:37:22.525 INFO:tasks.rados.rados.0.smithi071.stdout:1181: oid 13 version 439 is already newer than 438 2024-09-11T01:37:22.525 INFO:tasks.rados.rados.0.smithi071.stdout:1181: oid 13 version 439 is already newer than 437 2024-09-11T01:37:22.525 INFO:tasks.rados.rados.0.smithi071.stdout:1181: oid 13 updating version 439 to 440 2024-09-11T01:37:22.525 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 440 (ObjNum 451 snap 121 seq_num 451) dirty exists 2024-09-11T01:37:22.525 INFO:tasks.rados.rados.0.smithi071.stdout:1181: left oid 13 (ObjNum 451 snap 121 seq_num 451) 2024-09-11T01:37:22.525 INFO:tasks.rados.rados.0.smithi071.stdout:1183: finishing write tid 1 to smithi07134777-2 2024-09-11T01:37:22.525 INFO:tasks.rados.rados.0.smithi071.stdout:1183: finishing write tid 2 to smithi07134777-2 2024-09-11T01:37:22.525 INFO:tasks.rados.rados.0.smithi071.stdout:1183: finishing write tid 3 to smithi07134777-2 2024-09-11T01:37:22.525 INFO:tasks.rados.rados.0.smithi071.stdout:1183: finishing write tid 4 to smithi07134777-2 2024-09-11T01:37:22.525 INFO:tasks.rados.rados.0.smithi071.stdout:1183: finishing write tid 5 to smithi07134777-2 2024-09-11T01:37:22.525 INFO:tasks.rados.rados.0.smithi071.stdout:1183: oid 2 updating version 0 to 422 2024-09-11T01:37:22.525 INFO:tasks.rados.rados.0.smithi071.stdout:1183: oid 2 updating version 422 to 423 2024-09-11T01:37:22.525 INFO:tasks.rados.rados.0.smithi071.stdout:1183: oid 2 version 423 is already newer than 421 2024-09-11T01:37:22.526 INFO:tasks.rados.rados.0.smithi071.stdout:1183: oid 2 version 423 is already newer than 420 2024-09-11T01:37:22.526 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 423 (ObjNum 452 snap 121 seq_num 452) dirty exists 2024-09-11T01:37:22.526 INFO:tasks.rados.rados.0.smithi071.stdout:1183: left oid 2 (ObjNum 452 snap 121 seq_num 452) 2024-09-11T01:37:22.526 INFO:tasks.rados.rados.0.smithi071.stdout:1184: finishing write tid 1 to smithi07134777-32 2024-09-11T01:37:22.526 INFO:tasks.rados.rados.0.smithi071.stdout:1184: finishing write tid 2 to smithi07134777-32 2024-09-11T01:37:22.526 INFO:tasks.rados.rados.0.smithi071.stdout:1184: finishing write tid 3 to smithi07134777-32 2024-09-11T01:37:22.526 INFO:tasks.rados.rados.0.smithi071.stdout:1184: finishing write tid 4 to smithi07134777-32 2024-09-11T01:37:22.526 INFO:tasks.rados.rados.0.smithi071.stdout:1184: finishing write tid 5 to smithi07134777-32 2024-09-11T01:37:22.526 INFO:tasks.rados.rados.0.smithi071.stdout:1184: oid 32 updating version 0 to 421 2024-09-11T01:37:22.526 INFO:tasks.rados.rados.0.smithi071.stdout:1184: oid 32 version 421 is already newer than 418 2024-09-11T01:37:22.526 INFO:tasks.rados.rados.0.smithi071.stdout:1184: oid 32 version 421 is already newer than 419 2024-09-11T01:37:22.526 INFO:tasks.rados.rados.0.smithi071.stdout:1184: oid 32 version 421 is already newer than 420 2024-09-11T01:37:22.526 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 421 (ObjNum 453 snap 121 seq_num 453) dirty exists 2024-09-11T01:37:22.527 INFO:tasks.rados.rados.0.smithi071.stdout:1184: left oid 32 (ObjNum 453 snap 121 seq_num 453) 2024-09-11T01:37:22.527 INFO:tasks.rados.rados.0.smithi071.stdout:1185: finishing write tid 1 to smithi07134777-34 2024-09-11T01:37:22.527 INFO:tasks.rados.rados.0.smithi071.stdout:1185: finishing write tid 2 to smithi07134777-34 2024-09-11T01:37:22.527 INFO:tasks.rados.rados.0.smithi071.stdout:1185: finishing write tid 3 to smithi07134777-34 2024-09-11T01:37:22.527 INFO:tasks.rados.rados.0.smithi071.stdout:1185: finishing write tid 4 to smithi07134777-34 2024-09-11T01:37:22.527 INFO:tasks.rados.rados.0.smithi071.stdout:1185: finishing write tid 5 to smithi07134777-34 2024-09-11T01:37:22.527 INFO:tasks.rados.rados.0.smithi071.stdout:1185: oid 34 updating version 0 to 561 2024-09-11T01:37:22.527 INFO:tasks.rados.rados.0.smithi071.stdout:1185: oid 34 updating version 561 to 563 2024-09-11T01:37:22.527 INFO:tasks.rados.rados.0.smithi071.stdout:1185: oid 34 version 563 is already newer than 562 2024-09-11T01:37:22.527 INFO:tasks.rados.rados.0.smithi071.stdout:1185: oid 34 updating version 563 to 564 2024-09-11T01:37:22.527 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 564 (ObjNum 454 snap 121 seq_num 454) dirty exists 2024-09-11T01:37:22.527 INFO:tasks.rados.rados.0.smithi071.stdout:1185: left oid 34 (ObjNum 454 snap 121 seq_num 454) 2024-09-11T01:37:22.527 INFO:tasks.rados.rados.0.smithi071.stdout:1186: finishing write tid 1 to smithi07134777-15 2024-09-11T01:37:22.527 INFO:tasks.rados.rados.0.smithi071.stdout:1186: finishing write tid 2 to smithi07134777-15 2024-09-11T01:37:22.528 INFO:tasks.rados.rados.0.smithi071.stdout:1181: done (5 left) 2024-09-11T01:37:22.528 INFO:tasks.rados.rados.0.smithi071.stdout:1183: done (4 left) 2024-09-11T01:37:22.528 INFO:tasks.rados.rados.0.smithi071.stdout:1184: done (3 left) 2024-09-11T01:37:22.528 INFO:tasks.rados.rados.0.smithi071.stdout:1185: done (2 left) 2024-09-11T01:37:22.528 INFO:tasks.rados.rados.0.smithi071.stdout:1187: done (1 left) 2024-09-11T01:37:22.528 INFO:tasks.rados.rados.0.smithi071.stdout:1188: write oid 13 current snap is 121 2024-09-11T01:37:22.528 INFO:tasks.rados.rados.0.smithi071.stdout:1188: seq_num 457 ranges {480799=569973,1475918=769483,2700190=586604} 2024-09-11T01:37:22.531 INFO:tasks.rados.rados.0.smithi071.stdout:1188: writing smithi07134777-13 from 480799 to 1050772 tid 1 2024-09-11T01:37:22.538 INFO:tasks.rados.rados.0.smithi071.stdout:1188: writing smithi07134777-13 from 1475918 to 2245401 tid 2 2024-09-11T01:37:22.545 INFO:tasks.rados.rados.0.smithi071.stdout:1188: writing smithi07134777-13 from 2700190 to 3286794 tid 3 2024-09-11T01:37:22.545 INFO:tasks.rados.rados.0.smithi071.stdout:1186: finishing write tid 3 to smithi07134777-15 2024-09-11T01:37:22.545 INFO:tasks.rados.rados.0.smithi071.stdout:1189: read oid 35 snap -1 2024-09-11T01:37:22.545 INFO:tasks.rados.rados.0.smithi071.stdout:1189: expect (ObjNum 312 snap 87 seq_num 312) 2024-09-11T01:37:22.545 INFO:tasks.rados.rados.0.smithi071.stdout:1190: read oid 29 snap -1 2024-09-11T01:37:22.545 INFO:tasks.rados.rados.0.smithi071.stdout:1190: expect (ObjNum 226 snap 55 seq_num 226) 2024-09-11T01:37:22.545 INFO:tasks.rados.rados.0.smithi071.stdout:1191: copy_from oid 19 from oid 1 current snap is 121 2024-09-11T01:37:22.545 INFO:tasks.rados.rados.0.smithi071.stdout:1186: finishing write tid 4 to smithi07134777-15 2024-09-11T01:37:22.545 INFO:tasks.rados.rados.0.smithi071.stdout:1186: finishing write tid 5 to smithi07134777-15 2024-09-11T01:37:22.545 INFO:tasks.rados.rados.0.smithi071.stdout:1186: oid 15 updating version 0 to 617 2024-09-11T01:37:22.545 INFO:tasks.rados.rados.0.smithi071.stdout:1186: oid 15 updating version 617 to 619 2024-09-11T01:37:22.546 INFO:tasks.rados.rados.0.smithi071.stdout:1186: oid 15 version 619 is already newer than 618 2024-09-11T01:37:22.546 INFO:tasks.rados.rados.0.smithi071.stdout:1186: oid 15 version 619 is already newer than 616 2024-09-11T01:37:22.546 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 619 (ObjNum 455 snap 121 seq_num 455) dirty exists 2024-09-11T01:37:22.546 INFO:tasks.rados.rados.0.smithi071.stdout:1186: left oid 15 (ObjNum 455 snap 121 seq_num 455) 2024-09-11T01:37:22.546 INFO:tasks.rados.rados.0.smithi071.stdout:1188: finishing write tid 1 to smithi07134777-13 2024-09-11T01:37:22.546 INFO:tasks.rados.rados.0.smithi071.stdout:1188: finishing write tid 2 to smithi07134777-13 2024-09-11T01:37:22.546 INFO:tasks.rados.rados.0.smithi071.stdout:1186: done (4 left) 2024-09-11T01:37:22.546 INFO:tasks.rados.rados.0.smithi071.stdout:1192: write oid 42 current snap is 121 2024-09-11T01:37:22.546 INFO:tasks.rados.rados.0.smithi071.stdout:1192: seq_num 458 ranges {497606=645222,1934400=629494,3243966=453640} 2024-09-11T01:37:22.552 INFO:tasks.rados.rados.0.smithi071.stdout:1192: writing smithi07134777-42 from 497606 to 1142828 tid 1 2024-09-11T01:37:22.558 INFO:tasks.rados.rados.0.smithi071.stdout:1192: writing smithi07134777-42 from 1934400 to 2563894 tid 2 2024-09-11T01:37:22.565 INFO:tasks.rados.rados.0.smithi071.stdout:1192: writing smithi07134777-42 from 3243966 to 3697606 tid 3 2024-09-11T01:37:22.565 INFO:tasks.rados.rados.0.smithi071.stdout:1193: snap_create 2024-09-11T01:37:22.565 INFO:tasks.rados.rados.0.smithi071.stdout:1190: expect (ObjNum 226 snap 55 seq_num 226) 2024-09-11T01:37:22.569 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 838989 2024-09-11T01:37:22.642 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1983268 2024-09-11T01:37:22.668 INFO:tasks.rados.rados.0.smithi071.stdout:1188: finishing write tid 3 to smithi07134777-13 2024-09-11T01:37:22.668 INFO:tasks.rados.rados.0.smithi071.stdout:1191: finishing copy_from to smithi07134777-19 2024-09-11T01:37:22.668 INFO:tasks.rados.rados.0.smithi071.stdout:1191: got expected ENOENT (src dne) 2024-09-11T01:37:22.669 INFO:tasks.rados.rados.0.smithi071.stdout:1191: finishing copy_from racing read to smithi07134777-19 2024-09-11T01:37:22.669 INFO:tasks.rados.rados.0.smithi071.stdout:1188: finishing write tid 4 to smithi07134777-13 2024-09-11T01:37:22.669 INFO:tasks.rados.rados.0.smithi071.stdout:1188: finishing write tid 5 to smithi07134777-13 2024-09-11T01:37:22.669 INFO:tasks.rados.rados.0.smithi071.stdout:1188: oid 13 updating version 0 to 443 2024-09-11T01:37:22.669 INFO:tasks.rados.rados.0.smithi071.stdout:1188: oid 13 version 443 is already newer than 442 2024-09-11T01:37:22.669 INFO:tasks.rados.rados.0.smithi071.stdout:1188: oid 13 updating version 443 to 444 2024-09-11T01:37:22.669 INFO:tasks.rados.rados.0.smithi071.stdout:1188: oid 13 version 444 is already newer than 441 2024-09-11T01:37:22.669 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 444 (ObjNum 457 snap 121 seq_num 457) dirty exists 2024-09-11T01:37:22.670 INFO:tasks.rados.rados.0.smithi071.stdout:1188: left oid 13 (ObjNum 457 snap 121 seq_num 457) 2024-09-11T01:37:22.670 INFO:tasks.rados.rados.0.smithi071.stdout:1189: expect (ObjNum 312 snap 87 seq_num 312) 2024-09-11T01:37:22.736 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1290854 2024-09-11T01:37:22.761 INFO:tasks.rados.rados.0.smithi071.stdout:1192: finishing write tid 1 to smithi07134777-42 2024-09-11T01:37:22.761 INFO:tasks.rados.rados.0.smithi071.stdout:1192: finishing write tid 2 to smithi07134777-42 2024-09-11T01:37:22.761 INFO:tasks.rados.rados.0.smithi071.stdout:1192: finishing write tid 3 to smithi07134777-42 2024-09-11T01:37:22.761 INFO:tasks.rados.rados.0.smithi071.stdout:1192: finishing write tid 4 to smithi07134777-42 2024-09-11T01:37:22.761 INFO:tasks.rados.rados.0.smithi071.stdout:1192: finishing write tid 5 to smithi07134777-42 2024-09-11T01:37:22.761 INFO:tasks.rados.rados.0.smithi071.stdout:1192: oid 42 updating version 0 to 424 2024-09-11T01:37:22.761 INFO:tasks.rados.rados.0.smithi071.stdout:1192: oid 42 updating version 424 to 427 2024-09-11T01:37:22.761 INFO:tasks.rados.rados.0.smithi071.stdout:1192: oid 42 version 427 is already newer than 425 2024-09-11T01:37:22.761 INFO:tasks.rados.rados.0.smithi071.stdout:1192: oid 42 version 427 is already newer than 426 2024-09-11T01:37:22.761 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 427 (ObjNum 458 snap 121 seq_num 458) dirty exists 2024-09-11T01:37:22.762 INFO:tasks.rados.rados.0.smithi071.stdout:1192: left oid 42 (ObjNum 458 snap 121 seq_num 458) 2024-09-11T01:37:23.239 INFO:tasks.rados.rados.0.smithi071.stdout:1188: done (5 left) 2024-09-11T01:37:23.240 INFO:tasks.rados.rados.0.smithi071.stdout:1189: done (4 left) 2024-09-11T01:37:23.240 INFO:tasks.rados.rados.0.smithi071.stdout:1190: done (3 left) 2024-09-11T01:37:23.240 INFO:tasks.rados.rados.0.smithi071.stdout:1191: done (2 left) 2024-09-11T01:37:23.240 INFO:tasks.rados.rados.0.smithi071.stdout:1192: done (1 left) 2024-09-11T01:37:23.240 INFO:tasks.rados.rados.0.smithi071.stdout:1193: done (0 left) 2024-09-11T01:37:23.240 INFO:tasks.rados.rados.0.smithi071.stdout:1194: write (excl) oid 40 current snap is 122 2024-09-11T01:37:23.240 INFO:tasks.rados.rados.0.smithi071.stdout:1194: seq_num 459 ranges {514413=720471,1876529=773152,3438683=275730} 2024-09-11T01:37:23.246 INFO:tasks.rados.rados.0.smithi071.stdout:1194: writing smithi07134777-40 from 514413 to 1234884 tid 1 2024-09-11T01:37:23.254 INFO:tasks.rados.rados.0.smithi071.stdout:1194: writing smithi07134777-40 from 1876529 to 2649681 tid 2 2024-09-11T01:37:23.260 INFO:tasks.rados.rados.0.smithi071.stdout:1194: writing smithi07134777-40 from 3438683 to 3714413 tid 3 2024-09-11T01:37:23.260 INFO:tasks.rados.rados.0.smithi071.stdout:1195: snap_remove snap 121 2024-09-11T01:37:24.247 INFO:tasks.rados.rados.0.smithi071.stdout:1195: done (1 left) 2024-09-11T01:37:24.247 INFO:tasks.rados.rados.0.smithi071.stdout:1196: write (excl) oid 10 current snap is 122 2024-09-11T01:37:24.247 INFO:tasks.rados.rados.0.smithi071.stdout:1194: finishing write tid 1 to smithi07134777-40 2024-09-11T01:37:24.247 INFO:tasks.rados.rados.0.smithi071.stdout:1196: seq_num 460 ranges {531220=795720,1935011=516810,3066106=420420} 2024-09-11T01:37:24.253 INFO:tasks.rados.rados.0.smithi071.stdout:1196: writing smithi07134777-10 from 531220 to 1326940 tid 1 2024-09-11T01:37:24.259 INFO:tasks.rados.rados.0.smithi071.stdout:1196: writing smithi07134777-10 from 1935011 to 2451821 tid 2 2024-09-11T01:37:24.266 INFO:tasks.rados.rados.0.smithi071.stdout:1196: writing smithi07134777-10 from 3066106 to 3486526 tid 3 2024-09-11T01:37:24.266 INFO:tasks.rados.rados.0.smithi071.stdout:1197: read oid 15 snap -1 2024-09-11T01:37:24.266 INFO:tasks.rados.rados.0.smithi071.stdout:1197: expect (ObjNum 455 snap 121 seq_num 455) 2024-09-11T01:37:24.266 INFO:tasks.rados.rados.0.smithi071.stdout:1198: delete oid 4 current snap is 122 2024-09-11T01:37:24.266 INFO:tasks.rados.rados.0.smithi071.stdout:1194: finishing write tid 2 to smithi07134777-40 2024-09-11T01:37:24.266 INFO:tasks.rados.rados.0.smithi071.stdout:1194: finishing write tid 3 to smithi07134777-40 2024-09-11T01:37:24.266 INFO:tasks.rados.rados.0.smithi071.stdout:1194: finishing write tid 4 to smithi07134777-40 2024-09-11T01:37:24.266 INFO:tasks.rados.rados.0.smithi071.stdout:1194: finishing write tid 5 to smithi07134777-40 2024-09-11T01:37:24.266 INFO:tasks.rados.rados.0.smithi071.stdout:1194: oid 40 updating version 0 to 231 2024-09-11T01:37:24.266 INFO:tasks.rados.rados.0.smithi071.stdout:1194: oid 40 updating version 231 to 232 2024-09-11T01:37:24.266 INFO:tasks.rados.rados.0.smithi071.stdout:1194: oid 40 version 232 is already newer than 230 2024-09-11T01:37:24.266 INFO:tasks.rados.rados.0.smithi071.stdout:1194: oid 40 updating version 232 to 233 2024-09-11T01:37:24.267 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 233 (ObjNum 459 snap 122 seq_num 459) dirty exists 2024-09-11T01:37:24.267 INFO:tasks.rados.rados.0.smithi071.stdout:1194: left oid 40 (ObjNum 459 snap 122 seq_num 459) 2024-09-11T01:37:24.267 INFO:tasks.rados.rados.0.smithi071.stdout:1196: finishing write tid 1 to smithi07134777-10 2024-09-11T01:37:24.267 INFO:tasks.rados.rados.0.smithi071.stdout:1196: finishing write tid 2 to smithi07134777-10 2024-09-11T01:37:24.269 INFO:tasks.rados.rados.0.smithi071.stdout:1194: done (3 left) 2024-09-11T01:37:24.269 INFO:tasks.rados.rados.0.smithi071.stdout:1198: done (2 left) 2024-09-11T01:37:24.269 INFO:tasks.rados.rados.0.smithi071.stdout:1199: copy_from oid 39 from oid 34 current snap is 122 2024-09-11T01:37:24.269 INFO:tasks.rados.rados.0.smithi071.stdout:1200: write (excl) oid 4 current snap is 122 2024-09-11T01:37:24.269 INFO:tasks.rados.rados.0.smithi071.stdout:1200: seq_num 462 ranges {564834=546218,1651975=520479,2720952=559317} 2024-09-11T01:37:24.275 INFO:tasks.rados.rados.0.smithi071.stdout:1200: writing smithi07134777-4 from 564834 to 1111052 tid 1 2024-09-11T01:37:24.281 INFO:tasks.rados.rados.0.smithi071.stdout:1200: writing smithi07134777-4 from 1651975 to 2172454 tid 2 2024-09-11T01:37:24.287 INFO:tasks.rados.rados.0.smithi071.stdout:1200: writing smithi07134777-4 from 2720952 to 3280269 tid 3 2024-09-11T01:37:24.287 INFO:tasks.rados.rados.0.smithi071.stdout:1201: snap_create 2024-09-11T01:37:24.288 INFO:tasks.rados.rados.0.smithi071.stdout:1196: finishing write tid 3 to smithi07134777-10 2024-09-11T01:37:24.288 INFO:tasks.rados.rados.0.smithi071.stdout:1196: finishing write tid 4 to smithi07134777-10 2024-09-11T01:37:24.288 INFO:tasks.rados.rados.0.smithi071.stdout:1196: finishing write tid 5 to smithi07134777-10 2024-09-11T01:37:24.288 INFO:tasks.rados.rados.0.smithi071.stdout:1196: oid 10 updating version 0 to 348 2024-09-11T01:37:24.288 INFO:tasks.rados.rados.0.smithi071.stdout:1196: oid 10 updating version 348 to 350 2024-09-11T01:37:24.288 INFO:tasks.rados.rados.0.smithi071.stdout:1196: oid 10 version 350 is already newer than 349 2024-09-11T01:37:24.288 INFO:tasks.rados.rados.0.smithi071.stdout:1196: oid 10 updating version 350 to 351 2024-09-11T01:37:24.288 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 351 (ObjNum 460 snap 122 seq_num 460) dirty exists 2024-09-11T01:37:24.288 INFO:tasks.rados.rados.0.smithi071.stdout:1196: left oid 10 (ObjNum 460 snap 122 seq_num 460) 2024-09-11T01:37:24.288 INFO:tasks.rados.rados.0.smithi071.stdout:1197: expect (ObjNum 455 snap 121 seq_num 455) 2024-09-11T01:37:24.343 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3304942 2024-09-11T01:37:24.385 INFO:tasks.rados.rados.0.smithi071.stdout:1200: finishing write tid 1 to smithi07134777-4 2024-09-11T01:37:24.385 INFO:tasks.rados.rados.0.smithi071.stdout:1200: finishing write tid 2 to smithi07134777-4 2024-09-11T01:37:24.385 INFO:tasks.rados.rados.0.smithi071.stdout:1199: finishing copy_from racing read to smithi07134777-39 2024-09-11T01:37:24.386 INFO:tasks.rados.rados.0.smithi071.stdout:1199: finishing copy_from to smithi07134777-39 2024-09-11T01:37:24.386 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 564 (ObjNum 454 snap 121 seq_num 454) dirty exists 2024-09-11T01:37:24.386 INFO:tasks.rados.rados.0.smithi071.stdout:1200: finishing write tid 3 to smithi07134777-4 2024-09-11T01:37:24.386 INFO:tasks.rados.rados.0.smithi071.stdout:1200: finishing write tid 4 to smithi07134777-4 2024-09-11T01:37:24.386 INFO:tasks.rados.rados.0.smithi071.stdout:1200: finishing write tid 5 to smithi07134777-4 2024-09-11T01:37:24.386 INFO:tasks.rados.rados.0.smithi071.stdout:1200: oid 4 updating version 0 to 426 2024-09-11T01:37:24.386 INFO:tasks.rados.rados.0.smithi071.stdout:1200: oid 4 version 426 is already newer than 425 2024-09-11T01:37:24.386 INFO:tasks.rados.rados.0.smithi071.stdout:1200: oid 4 updating version 426 to 428 2024-09-11T01:37:24.386 INFO:tasks.rados.rados.0.smithi071.stdout:1200: oid 4 version 428 is already newer than 427 2024-09-11T01:37:24.386 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 428 (ObjNum 462 snap 122 seq_num 462) dirty exists 2024-09-11T01:37:24.386 INFO:tasks.rados.rados.0.smithi071.stdout:1200: left oid 4 (ObjNum 462 snap 122 seq_num 462) 2024-09-11T01:37:25.248 INFO:tasks.rados.rados.0.smithi071.stdout:1196: done (4 left) 2024-09-11T01:37:25.248 INFO:tasks.rados.rados.0.smithi071.stdout:1197: done (3 left) 2024-09-11T01:37:25.248 INFO:tasks.rados.rados.0.smithi071.stdout:1199: done (2 left) 2024-09-11T01:37:25.248 INFO:tasks.rados.rados.0.smithi071.stdout:1200: done (1 left) 2024-09-11T01:37:25.248 INFO:tasks.rados.rados.0.smithi071.stdout:1201: done (0 left) 2024-09-11T01:37:25.248 INFO:tasks.rados.rados.0.smithi071.stdout:1202: delete oid 49 current snap is 123 2024-09-11T01:37:25.253 INFO:tasks.rados.rados.0.smithi071.stdout:1202: done (0 left) 2024-09-11T01:37:25.253 INFO:tasks.rados.rados.0.smithi071.stdout:1203: copy_from oid 8 from oid 31 current snap is 123 2024-09-11T01:37:25.253 INFO:tasks.rados.rados.0.smithi071.stdout:1204: delete oid 6 current snap is 123 2024-09-11T01:37:25.256 INFO:tasks.rados.rados.0.smithi071.stdout:1204: done (1 left) 2024-09-11T01:37:25.256 INFO:tasks.rados.rados.0.smithi071.stdout:1205: snap_remove snap 122 2024-09-11T01:37:26.254 INFO:tasks.rados.rados.0.smithi071.stdout:1205: done (1 left) 2024-09-11T01:37:26.254 INFO:tasks.rados.rados.0.smithi071.stdout:1206: snap_create 2024-09-11T01:37:26.254 INFO:tasks.rados.rados.0.smithi071.stdout:1203: finishing copy_from racing read to smithi07134777-8 2024-09-11T01:37:26.254 INFO:tasks.rados.rados.0.smithi071.stdout:1203: finishing copy_from to smithi07134777-8 2024-09-11T01:37:26.254 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 434 (ObjNum 250 snap 63 seq_num 250) dirty exists 2024-09-11T01:37:27.256 INFO:tasks.rados.rados.0.smithi071.stdout:1203: done (1 left) 2024-09-11T01:37:27.256 INFO:tasks.rados.rados.0.smithi071.stdout:1206: done (0 left) 2024-09-11T01:37:27.256 INFO:tasks.rados.rados.0.smithi071.stdout:1207: delete oid 6 current snap is 124 2024-09-11T01:37:27.260 INFO:tasks.rados.rados.0.smithi071.stdout:1207: done (0 left) 2024-09-11T01:37:27.260 INFO:tasks.rados.rados.0.smithi071.stdout:1208: copy_from oid 5 from oid 29 current snap is 124 2024-09-11T01:37:27.260 INFO:tasks.rados.rados.0.smithi071.stdout:1209: rollback oid 20 current snap is 124 2024-09-11T01:37:27.260 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 20 to 123 2024-09-11T01:37:27.260 INFO:tasks.rados.rados.0.smithi071.stdout:1210: delete oid 14 current snap is 124 2024-09-11T01:37:27.268 INFO:tasks.rados.rados.0.smithi071.stdout:1210: done (2 left) 2024-09-11T01:37:27.268 INFO:tasks.rados.rados.0.smithi071.stdout:1211: read oid 36 snap 123 2024-09-11T01:37:27.268 INFO:tasks.rados.rados.0.smithi071.stdout:1211: expect (ObjNum 449 snap 121 seq_num 449) 2024-09-11T01:37:27.269 INFO:tasks.rados.rados.0.smithi071.stdout:1212: delete oid 49 current snap is 124 2024-09-11T01:37:27.269 INFO:tasks.rados.rados.0.smithi071.stdout:1209: finishing rollback tid 0 to smithi07134777-20 2024-09-11T01:37:27.269 INFO:tasks.rados.rados.0.smithi071.stdout:1209: finishing rollback tid 1 to smithi07134777-20 2024-09-11T01:37:27.269 INFO:tasks.rados.rados.0.smithi071.stdout:1209: finishing rollback tid 2 to smithi07134777-20 2024-09-11T01:37:27.269 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 248 (ObjNum 323 snap 92 seq_num 323) dirty exists 2024-09-11T01:37:27.270 INFO:tasks.rados.rados.0.smithi071.stdout:1209: done (3 left) 2024-09-11T01:37:27.270 INFO:tasks.rados.rados.0.smithi071.stdout:1212: done (2 left) 2024-09-11T01:37:27.271 INFO:tasks.rados.rados.0.smithi071.stdout:1213: write oid 43 current snap is 124 2024-09-11T01:37:27.271 INFO:tasks.rados.rados.0.smithi071.stdout:1213: seq_num 468 ranges {665676=714065,1886514=531486,3169137=696539} 2024-09-11T01:37:27.279 INFO:tasks.rados.rados.0.smithi071.stdout:1213: writing smithi07134777-43 from 665676 to 1379741 tid 1 2024-09-11T01:37:27.285 INFO:tasks.rados.rados.0.smithi071.stdout:1213: writing smithi07134777-43 from 1886514 to 2418000 tid 2 2024-09-11T01:37:27.293 INFO:tasks.rados.rados.0.smithi071.stdout:1213: writing smithi07134777-43 from 3169137 to 3865676 tid 3 2024-09-11T01:37:27.293 INFO:tasks.rados.rados.0.smithi071.stdout:1214: read oid 3 snap -1 2024-09-11T01:37:27.293 INFO:tasks.rados.rados.0.smithi071.stdout:1214: expect deleted 2024-09-11T01:37:27.293 INFO:tasks.rados.rados.0.smithi071.stdout:1215: copy_from oid 14 from oid 35 current snap is 124 2024-09-11T01:37:27.293 INFO:tasks.rados.rados.0.smithi071.stdout:1208: finishing copy_from racing read to smithi07134777-5 2024-09-11T01:37:27.294 INFO:tasks.rados.rados.0.smithi071.stdout:1211: expect (ObjNum 449 snap 121 seq_num 449) 2024-09-11T01:37:27.294 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:37:27.436 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1305963 2024-09-11T01:37:27.477 INFO:tasks.rados.rados.0.smithi071.stdout:1208: finishing copy_from to smithi07134777-5 2024-09-11T01:37:27.477 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 488 (ObjNum 226 snap 55 seq_num 226) dirty exists 2024-09-11T01:37:27.477 INFO:tasks.rados.rados.0.smithi071.stdout:1213: finishing write tid 1 to smithi07134777-43 2024-09-11T01:37:27.478 INFO:tasks.rados.rados.0.smithi071.stdout:1213: finishing write tid 2 to smithi07134777-43 2024-09-11T01:37:27.478 INFO:tasks.rados.rados.0.smithi071.stdout:1213: finishing write tid 3 to smithi07134777-43 2024-09-11T01:37:27.478 INFO:tasks.rados.rados.0.smithi071.stdout:1213: finishing write tid 4 to smithi07134777-43 2024-09-11T01:37:27.478 INFO:tasks.rados.rados.0.smithi071.stdout:1213: finishing write tid 5 to smithi07134777-43 2024-09-11T01:37:27.478 INFO:tasks.rados.rados.0.smithi071.stdout:1213: oid 43 updating version 0 to 569 2024-09-11T01:37:27.478 INFO:tasks.rados.rados.0.smithi071.stdout:1213: oid 43 updating version 569 to 571 2024-09-11T01:37:27.478 INFO:tasks.rados.rados.0.smithi071.stdout:1213: oid 43 updating version 571 to 572 2024-09-11T01:37:27.478 INFO:tasks.rados.rados.0.smithi071.stdout:1213: oid 43 version 572 is already newer than 570 2024-09-11T01:37:27.478 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 572 (ObjNum 468 snap 124 seq_num 468) dirty exists 2024-09-11T01:37:27.478 INFO:tasks.rados.rados.0.smithi071.stdout:1213: left oid 43 (ObjNum 468 snap 124 seq_num 468) 2024-09-11T01:37:27.478 INFO:tasks.rados.rados.0.smithi071.stdout:1215: finishing copy_from racing read to smithi07134777-14 2024-09-11T01:37:27.478 INFO:tasks.rados.rados.0.smithi071.stdout:1208: done (4 left) 2024-09-11T01:37:27.478 INFO:tasks.rados.rados.0.smithi071.stdout:1211: done (3 left) 2024-09-11T01:37:27.478 INFO:tasks.rados.rados.0.smithi071.stdout:1213: done (2 left) 2024-09-11T01:37:27.479 INFO:tasks.rados.rados.0.smithi071.stdout:1214: done (1 left) 2024-09-11T01:37:27.479 INFO:tasks.rados.rados.0.smithi071.stdout:1216: write oid 41 current snap is 124 2024-09-11T01:37:27.479 INFO:tasks.rados.rados.0.smithi071.stdout:1215: finishing copy_from to smithi07134777-14 2024-09-11T01:37:27.479 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 218 (ObjNum 312 snap 87 seq_num 312) dirty exists 2024-09-11T01:37:27.479 INFO:tasks.rados.rados.0.smithi071.stdout:1216: seq_num 469 ranges {682483=789314,1944996=791497,3196560=554574} 2024-09-11T01:37:27.484 INFO:tasks.rados.rados.0.smithi071.stdout:1216: writing smithi07134777-41 from 682483 to 1471797 tid 1 2024-09-11T01:37:27.491 INFO:tasks.rados.rados.0.smithi071.stdout:1216: writing smithi07134777-41 from 1944996 to 2736493 tid 2 2024-09-11T01:37:27.497 INFO:tasks.rados.rados.0.smithi071.stdout:1216: writing smithi07134777-41 from 3196560 to 3751134 tid 3 2024-09-11T01:37:27.497 INFO:tasks.rados.rados.0.smithi071.stdout:1215: done (1 left) 2024-09-11T01:37:27.497 INFO:tasks.rados.rados.0.smithi071.stdout:1217: read oid 2 snap -1 2024-09-11T01:37:27.497 INFO:tasks.rados.rados.0.smithi071.stdout:1217: expect (ObjNum 452 snap 121 seq_num 452) 2024-09-11T01:37:27.497 INFO:tasks.rados.rados.0.smithi071.stdout:1218: read oid 9 snap -1 2024-09-11T01:37:27.497 INFO:tasks.rados.rados.0.smithi071.stdout:1218: expect (ObjNum 445 snap 121 seq_num 445) 2024-09-11T01:37:27.497 INFO:tasks.rados.rados.0.smithi071.stdout:1219: snap_create 2024-09-11T01:37:27.497 INFO:tasks.rados.rados.0.smithi071.stdout:1216: finishing write tid 1 to smithi07134777-41 2024-09-11T01:37:27.498 INFO:tasks.rados.rados.0.smithi071.stdout:1216: finishing write tid 2 to smithi07134777-41 2024-09-11T01:37:27.500 INFO:tasks.rados.rados.0.smithi071.stdout:1216: finishing write tid 3 to smithi07134777-41 2024-09-11T01:37:27.501 INFO:tasks.rados.rados.0.smithi071.stdout:1216: finishing write tid 4 to smithi07134777-41 2024-09-11T01:37:27.501 INFO:tasks.rados.rados.0.smithi071.stdout:1216: finishing write tid 5 to smithi07134777-41 2024-09-11T01:37:27.501 INFO:tasks.rados.rados.0.smithi071.stdout:1216: oid 41 updating version 0 to 352 2024-09-11T01:37:27.501 INFO:tasks.rados.rados.0.smithi071.stdout:1216: oid 41 updating version 352 to 355 2024-09-11T01:37:27.501 INFO:tasks.rados.rados.0.smithi071.stdout:1216: oid 41 version 355 is already newer than 353 2024-09-11T01:37:27.501 INFO:tasks.rados.rados.0.smithi071.stdout:1216: oid 41 version 355 is already newer than 354 2024-09-11T01:37:27.501 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 355 (ObjNum 469 snap 124 seq_num 469) dirty exists 2024-09-11T01:37:27.501 INFO:tasks.rados.rados.0.smithi071.stdout:1216: left oid 41 (ObjNum 469 snap 124 seq_num 469) 2024-09-11T01:37:27.505 INFO:tasks.rados.rados.0.smithi071.stdout:1217: expect (ObjNum 452 snap 121 seq_num 452) 2024-09-11T01:37:27.581 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:37:27.661 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:37:27.740 INFO:tasks.rados.rados.0.smithi071.stdout:1218: expect (ObjNum 445 snap 121 seq_num 445) 2024-09-11T01:37:27.823 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 419047 2024-09-11T01:37:28.241 INFO:tasks.rados.rados.0.smithi071.stdout:1216: done (3 left) 2024-09-11T01:37:28.241 INFO:tasks.rados.rados.0.smithi071.stdout:1217: done (2 left) 2024-09-11T01:37:28.241 INFO:tasks.rados.rados.0.smithi071.stdout:1218: done (1 left) 2024-09-11T01:37:28.241 INFO:tasks.rados.rados.0.smithi071.stdout:1219: done (0 left) 2024-09-11T01:37:28.241 INFO:tasks.rados.rados.0.smithi071.stdout:1220: read oid 25 snap 123 2024-09-11T01:37:28.241 INFO:tasks.rados.rados.0.smithi071.stdout:1220: expect deleted 2024-09-11T01:37:28.241 INFO:tasks.rados.rados.0.smithi071.stdout:1221: read oid 36 snap -1 2024-09-11T01:37:28.241 INFO:tasks.rados.rados.0.smithi071.stdout:1221: expect (ObjNum 449 snap 121 seq_num 449) 2024-09-11T01:37:28.241 INFO:tasks.rados.rados.0.smithi071.stdout:1222: rollback oid 26 current snap is 125 2024-09-11T01:37:28.241 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 26 to 123 2024-09-11T01:37:28.241 INFO:tasks.rados.rados.0.smithi071.stdout:1223: write (excl) oid 34 current snap is 125 2024-09-11T01:37:28.241 INFO:tasks.rados.rados.0.smithi071.stdout:1223: seq_num 470 ranges {699290=464563,1603478=535155,2823983=765846} 2024-09-11T01:37:28.246 INFO:tasks.rados.rados.0.smithi071.stdout:1223: writing smithi07134777-34 from 699290 to 1163853 tid 1 2024-09-11T01:37:28.252 INFO:tasks.rados.rados.0.smithi071.stdout:1223: writing smithi07134777-34 from 1603478 to 2138633 tid 2 2024-09-11T01:37:28.260 INFO:tasks.rados.rados.0.smithi071.stdout:1223: writing smithi07134777-34 from 2823983 to 3589829 tid 3 2024-09-11T01:37:28.260 INFO:tasks.rados.rados.0.smithi071.stdout:1224: read oid 30 snap 124 2024-09-11T01:37:28.261 INFO:tasks.rados.rados.0.smithi071.stdout:1224: expect (ObjNum 431 snap 119 seq_num 431) 2024-09-11T01:37:28.261 INFO:tasks.rados.rados.0.smithi071.stdout:1225: delete oid 19 current snap is 125 2024-09-11T01:37:28.261 INFO:tasks.rados.rados.0.smithi071.stdout:1222: finishing rollback tid 0 to smithi07134777-26 2024-09-11T01:37:28.261 INFO:tasks.rados.rados.0.smithi071.stdout:1222: finishing rollback tid 1 to smithi07134777-26 2024-09-11T01:37:28.261 INFO:tasks.rados.rados.0.smithi071.stdout:1221: expect (ObjNum 449 snap 121 seq_num 449) 2024-09-11T01:37:28.261 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:37:28.449 INFO:tasks.rados.rados.0.smithi071.stdout:1222: finishing rollback tid 2 to smithi07134777-26 2024-09-11T01:37:28.449 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 576 (ObjNum 428 snap 118 seq_num 428) dirty exists 2024-09-11T01:37:28.449 INFO:tasks.rados.rados.0.smithi071.stdout:1223: finishing write tid 1 to smithi07134777-34 2024-09-11T01:37:28.449 INFO:tasks.rados.rados.0.smithi071.stdout:1223: finishing write tid 2 to smithi07134777-34 2024-09-11T01:37:28.449 INFO:tasks.rados.rados.0.smithi071.stdout:1223: finishing write tid 3 to smithi07134777-34 2024-09-11T01:37:28.449 INFO:tasks.rados.rados.0.smithi071.stdout:1223: finishing write tid 4 to smithi07134777-34 2024-09-11T01:37:28.450 INFO:tasks.rados.rados.0.smithi071.stdout:1223: finishing write tid 5 to smithi07134777-34 2024-09-11T01:37:28.450 INFO:tasks.rados.rados.0.smithi071.stdout:1223: oid 34 updating version 0 to 578 2024-09-11T01:37:28.450 INFO:tasks.rados.rados.0.smithi071.stdout:1223: oid 34 updating version 578 to 581 2024-09-11T01:37:28.450 INFO:tasks.rados.rados.0.smithi071.stdout:1223: oid 34 version 581 is already newer than 579 2024-09-11T01:37:28.450 INFO:tasks.rados.rados.0.smithi071.stdout:1223: oid 34 version 581 is already newer than 580 2024-09-11T01:37:28.450 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 581 (ObjNum 470 snap 125 seq_num 470) dirty exists 2024-09-11T01:37:28.450 INFO:tasks.rados.rados.0.smithi071.stdout:1223: left oid 34 (ObjNum 470 snap 125 seq_num 470) 2024-09-11T01:37:28.450 INFO:tasks.rados.rados.0.smithi071.stdout:1224: expect (ObjNum 431 snap 119 seq_num 431) 2024-09-11T01:37:28.450 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:37:28.485 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:37:28.551 INFO:tasks.rados.rados.0.smithi071.stdout:1220: done (5 left) 2024-09-11T01:37:28.551 INFO:tasks.rados.rados.0.smithi071.stdout:1221: done (4 left) 2024-09-11T01:37:28.551 INFO:tasks.rados.rados.0.smithi071.stdout:1222: done (3 left) 2024-09-11T01:37:28.551 INFO:tasks.rados.rados.0.smithi071.stdout:1223: done (2 left) 2024-09-11T01:37:28.552 INFO:tasks.rados.rados.0.smithi071.stdout:1224: done (1 left) 2024-09-11T01:37:28.552 INFO:tasks.rados.rados.0.smithi071.stdout:1225: done (0 left) 2024-09-11T01:37:28.552 INFO:tasks.rados.rados.0.smithi071.stdout:1226: rollback oid 40 current snap is 125 2024-09-11T01:37:28.552 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 40 to 123 2024-09-11T01:37:28.552 INFO:tasks.rados.rados.0.smithi071.stdout:1227: write oid 36 current snap is 125 2024-09-11T01:37:28.552 INFO:tasks.rados.rados.0.smithi071.stdout:1227: seq_num 472 ranges {732904=731414,2120442=538824,3278829=504743} 2024-09-11T01:37:28.559 INFO:tasks.rados.rados.0.smithi071.stdout:1227: writing smithi07134777-36 from 732904 to 1464318 tid 1 2024-09-11T01:37:28.566 INFO:tasks.rados.rados.0.smithi071.stdout:1227: writing smithi07134777-36 from 2120442 to 2659266 tid 2 2024-09-11T01:37:28.572 INFO:tasks.rados.rados.0.smithi071.stdout:1227: writing smithi07134777-36 from 3278829 to 3783572 tid 3 2024-09-11T01:37:28.572 INFO:tasks.rados.rados.0.smithi071.stdout:1228: read oid 44 snap -1 2024-09-11T01:37:28.573 INFO:tasks.rados.rados.0.smithi071.stdout:1228: expect deleted 2024-09-11T01:37:28.573 INFO:tasks.rados.rados.0.smithi071.stdout:1229: delete oid 2 current snap is 125 2024-09-11T01:37:28.573 INFO:tasks.rados.rados.0.smithi071.stdout:1226: finishing rollback tid 0 to smithi07134777-40 2024-09-11T01:37:28.573 INFO:tasks.rados.rados.0.smithi071.stdout:1226: finishing rollback tid 1 to smithi07134777-40 2024-09-11T01:37:28.573 INFO:tasks.rados.rados.0.smithi071.stdout:1226: finishing rollback tid 2 to smithi07134777-40 2024-09-11T01:37:28.573 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 236 (ObjNum 459 snap 122 seq_num 459) dirty exists 2024-09-11T01:37:28.573 INFO:tasks.rados.rados.0.smithi071.stdout:1227: finishing write tid 1 to smithi07134777-36 2024-09-11T01:37:28.573 INFO:tasks.rados.rados.0.smithi071.stdout:1227: finishing write tid 2 to smithi07134777-36 2024-09-11T01:37:28.575 INFO:tasks.rados.rados.0.smithi071.stdout:1226: done (3 left) 2024-09-11T01:37:28.575 INFO:tasks.rados.rados.0.smithi071.stdout:1228: done (2 left) 2024-09-11T01:37:28.575 INFO:tasks.rados.rados.0.smithi071.stdout:1229: done (1 left) 2024-09-11T01:37:28.576 INFO:tasks.rados.rados.0.smithi071.stdout:1230: read oid 34 snap -1 2024-09-11T01:37:28.576 INFO:tasks.rados.rados.0.smithi071.stdout:1230: expect (ObjNum 470 snap 125 seq_num 470) 2024-09-11T01:37:28.576 INFO:tasks.rados.rados.0.smithi071.stdout:1231: delete oid 32 current snap is 125 2024-09-11T01:37:28.578 INFO:tasks.rados.rados.0.smithi071.stdout:1227: finishing write tid 3 to smithi07134777-36 2024-09-11T01:37:28.579 INFO:tasks.rados.rados.0.smithi071.stdout:1231: done (2 left) 2024-09-11T01:37:28.579 INFO:tasks.rados.rados.0.smithi071.stdout:1232: write oid 9 current snap is 125 2024-09-11T01:37:28.579 INFO:tasks.rados.rados.0.smithi071.stdout:1232: seq_num 475 ranges {783325=557161,1779535=686151,3244745=454912} 2024-09-11T01:37:28.586 INFO:tasks.rados.rados.0.smithi071.stdout:1232: writing smithi07134777-9 from 783325 to 1340486 tid 1 2024-09-11T01:37:28.593 INFO:tasks.rados.rados.0.smithi071.stdout:1232: writing smithi07134777-9 from 1779535 to 2465686 tid 2 2024-09-11T01:37:28.600 INFO:tasks.rados.rados.0.smithi071.stdout:1232: writing smithi07134777-9 from 3244745 to 3699657 tid 3 2024-09-11T01:37:28.600 INFO:tasks.rados.rados.0.smithi071.stdout:1233: rollback oid 50 current snap is 125 2024-09-11T01:37:28.600 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 50 to 124 2024-09-11T01:37:28.600 INFO:tasks.rados.rados.0.smithi071.stdout:1227: finishing write tid 4 to smithi07134777-36 2024-09-11T01:37:28.600 INFO:tasks.rados.rados.0.smithi071.stdout:1227: finishing write tid 5 to smithi07134777-36 2024-09-11T01:37:28.600 INFO:tasks.rados.rados.0.smithi071.stdout:1227: oid 36 updating version 0 to 436 2024-09-11T01:37:28.600 INFO:tasks.rados.rados.0.smithi071.stdout:1227: oid 36 version 436 is already newer than 435 2024-09-11T01:37:28.600 INFO:tasks.rados.rados.0.smithi071.stdout:1227: oid 36 updating version 436 to 437 2024-09-11T01:37:28.601 INFO:tasks.rados.rados.0.smithi071.stdout:1227: oid 36 updating version 437 to 438 2024-09-11T01:37:28.601 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 438 (ObjNum 472 snap 125 seq_num 472) dirty exists 2024-09-11T01:37:28.601 INFO:tasks.rados.rados.0.smithi071.stdout:1227: left oid 36 (ObjNum 472 snap 125 seq_num 472) 2024-09-11T01:37:28.601 INFO:tasks.rados.rados.0.smithi071.stdout:1230: expect (ObjNum 470 snap 125 seq_num 470) 2024-09-11T01:37:28.627 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2092096 2024-09-11T01:37:28.785 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 962397 2024-09-11T01:37:28.869 INFO:tasks.rados.rados.0.smithi071.stdout:1232: finishing write tid 1 to smithi07134777-9 2024-09-11T01:37:28.869 INFO:tasks.rados.rados.0.smithi071.stdout:1232: finishing write tid 2 to smithi07134777-9 2024-09-11T01:37:28.869 INFO:tasks.rados.rados.0.smithi071.stdout:1233: finishing rollback tid 1 to smithi07134777-50 2024-09-11T01:37:28.869 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 0 (ObjNum 929 snap 0 seq_num 2370780304) dirty dne 2024-09-11T01:37:28.869 INFO:tasks.rados.rados.0.smithi071.stdout:1232: finishing write tid 3 to smithi07134777-9 2024-09-11T01:37:28.869 INFO:tasks.rados.rados.0.smithi071.stdout:1232: finishing write tid 4 to smithi07134777-9 2024-09-11T01:37:28.870 INFO:tasks.rados.rados.0.smithi071.stdout:1232: finishing write tid 5 to smithi07134777-9 2024-09-11T01:37:28.870 INFO:tasks.rados.rados.0.smithi071.stdout:1232: oid 9 updating version 0 to 159 2024-09-11T01:37:28.870 INFO:tasks.rados.rados.0.smithi071.stdout:1232: oid 9 updating version 159 to 160 2024-09-11T01:37:28.870 INFO:tasks.rados.rados.0.smithi071.stdout:1232: oid 9 updating version 160 to 161 2024-09-11T01:37:28.870 INFO:tasks.rados.rados.0.smithi071.stdout:1232: oid 9 updating version 161 to 162 2024-09-11T01:37:28.870 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 162 (ObjNum 475 snap 125 seq_num 475) dirty exists 2024-09-11T01:37:28.870 INFO:tasks.rados.rados.0.smithi071.stdout:1232: left oid 9 (ObjNum 475 snap 125 seq_num 475) 2024-09-11T01:37:28.870 INFO:tasks.rados.rados.0.smithi071.stdout:1227: done (3 left) 2024-09-11T01:37:28.870 INFO:tasks.rados.rados.0.smithi071.stdout:1230: done (2 left) 2024-09-11T01:37:28.870 INFO:tasks.rados.rados.0.smithi071.stdout:1232: done (1 left) 2024-09-11T01:37:28.870 INFO:tasks.rados.rados.0.smithi071.stdout:1233: done (0 left) 2024-09-11T01:37:28.870 INFO:tasks.rados.rados.0.smithi071.stdout:1234: read oid 39 snap -1 2024-09-11T01:37:28.870 INFO:tasks.rados.rados.0.smithi071.stdout:1234: expect (ObjNum 454 snap 121 seq_num 454) 2024-09-11T01:37:28.870 INFO:tasks.rados.rados.0.smithi071.stdout:1235: read oid 24 snap -1 2024-09-11T01:37:28.870 INFO:tasks.rados.rados.0.smithi071.stdout:1235: expect (ObjNum 430 snap 119 seq_num 430) 2024-09-11T01:37:28.871 INFO:tasks.rados.rados.0.smithi071.stdout:1236: write (excl) oid 15 current snap is 125 2024-09-11T01:37:28.871 INFO:tasks.rados.rados.0.smithi071.stdout:1236: seq_num 476 ranges {400132=632410,1438017=546162} 2024-09-11T01:37:28.874 INFO:tasks.rados.rados.0.smithi071.stdout:1236: writing smithi07134777-15 from 400132 to 1032542 tid 1 2024-09-11T01:37:28.881 INFO:tasks.rados.rados.0.smithi071.stdout:1236: writing smithi07134777-15 from 1438017 to 1984179 tid 2 2024-09-11T01:37:28.881 INFO:tasks.rados.rados.0.smithi071.stdout:1237: write (excl) oid 44 current snap is 125 2024-09-11T01:37:28.881 INFO:tasks.rados.rados.0.smithi071.stdout:1237: seq_num 477 ranges {416939=707659,1896499=120440} 2024-09-11T01:37:28.887 INFO:tasks.rados.rados.0.smithi071.stdout:1237: writing smithi07134777-44 from 416939 to 1124598 tid 1 2024-09-11T01:37:28.892 INFO:tasks.rados.rados.0.smithi071.stdout:1237: writing smithi07134777-44 from 1896499 to 2016939 tid 2 2024-09-11T01:37:28.893 INFO:tasks.rados.rados.0.smithi071.stdout:1238: snap_create 2024-09-11T01:37:28.893 INFO:tasks.rados.rados.0.smithi071.stdout:1236: finishing write tid 1 to smithi07134777-15 2024-09-11T01:37:28.893 INFO:tasks.rados.rados.0.smithi071.stdout:1235: expect (ObjNum 430 snap 119 seq_num 430) 2024-09-11T01:37:28.893 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:37:29.049 INFO:tasks.rados.rados.0.smithi071.stdout:1236: finishing write tid 2 to smithi07134777-15 2024-09-11T01:37:29.049 INFO:tasks.rados.rados.0.smithi071.stdout:1236: finishing write tid 3 to smithi07134777-15 2024-09-11T01:37:29.049 INFO:tasks.rados.rados.0.smithi071.stdout:1234: expect (ObjNum 454 snap 121 seq_num 454) 2024-09-11T01:37:29.049 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:37:29.192 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 172474 2024-09-11T01:37:29.265 INFO:tasks.rados.rados.0.smithi071.stdout:1236: finishing write tid 4 to smithi07134777-15 2024-09-11T01:37:29.265 INFO:tasks.rados.rados.0.smithi071.stdout:1236: oid 15 updating version 0 to 624 2024-09-11T01:37:29.265 INFO:tasks.rados.rados.0.smithi071.stdout:1236: oid 15 updating version 624 to 625 2024-09-11T01:37:29.265 INFO:tasks.rados.rados.0.smithi071.stdout:1236: oid 15 updating version 625 to 626 2024-09-11T01:37:29.266 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 626 (ObjNum 476 snap 125 seq_num 476) dirty exists 2024-09-11T01:37:29.266 INFO:tasks.rados.rados.0.smithi071.stdout:1236: left oid 15 (ObjNum 476 snap 125 seq_num 476) 2024-09-11T01:37:29.266 INFO:tasks.rados.rados.0.smithi071.stdout:1237: finishing write tid 1 to smithi07134777-44 2024-09-11T01:37:29.266 INFO:tasks.rados.rados.0.smithi071.stdout:1237: finishing write tid 2 to smithi07134777-44 2024-09-11T01:37:29.266 INFO:tasks.rados.rados.0.smithi071.stdout:1237: finishing write tid 3 to smithi07134777-44 2024-09-11T01:37:29.266 INFO:tasks.rados.rados.0.smithi071.stdout:1237: finishing write tid 4 to smithi07134777-44 2024-09-11T01:37:29.266 INFO:tasks.rados.rados.0.smithi071.stdout:1237: oid 44 updating version 0 to 249 2024-09-11T01:37:29.266 INFO:tasks.rados.rados.0.smithi071.stdout:1237: oid 44 updating version 249 to 251 2024-09-11T01:37:29.266 INFO:tasks.rados.rados.0.smithi071.stdout:1237: oid 44 version 251 is already newer than 250 2024-09-11T01:37:29.266 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 251 (ObjNum 477 snap 125 seq_num 477) dirty exists 2024-09-11T01:37:29.266 INFO:tasks.rados.rados.0.smithi071.stdout:1237: left oid 44 (ObjNum 477 snap 125 seq_num 477) 2024-09-11T01:37:29.275 INFO:tasks.rados.rados.0.smithi071.stdout:1234: done (4 left) 2024-09-11T01:37:29.275 INFO:tasks.rados.rados.0.smithi071.stdout:1235: done (3 left) 2024-09-11T01:37:29.275 INFO:tasks.rados.rados.0.smithi071.stdout:1236: done (2 left) 2024-09-11T01:37:29.275 INFO:tasks.rados.rados.0.smithi071.stdout:1237: done (1 left) 2024-09-11T01:37:29.276 INFO:tasks.rados.rados.0.smithi071.stdout:1238: done (0 left) 2024-09-11T01:37:29.276 INFO:tasks.rados.rados.0.smithi071.stdout:1239: write (excl) oid 46 current snap is 126 2024-09-11T01:37:29.276 INFO:tasks.rados.rados.0.smithi071.stdout:1239: seq_num 478 ranges {433746=782908,1954981=78765} 2024-09-11T01:37:29.281 INFO:tasks.rados.rados.0.smithi071.stdout:1239: writing smithi07134777-46 from 433746 to 1216654 tid 1 2024-09-11T01:37:29.286 INFO:tasks.rados.rados.0.smithi071.stdout:1239: writing smithi07134777-46 from 1954981 to 2033746 tid 2 2024-09-11T01:37:29.286 INFO:tasks.rados.rados.0.smithi071.stdout:1240: snap_create 2024-09-11T01:37:29.295 INFO:tasks.rados.rados.0.smithi071.stdout:1239: finishing write tid 1 to smithi07134777-46 2024-09-11T01:37:29.295 INFO:tasks.rados.rados.0.smithi071.stdout:1239: finishing write tid 2 to smithi07134777-46 2024-09-11T01:37:29.295 INFO:tasks.rados.rados.0.smithi071.stdout:1239: finishing write tid 3 to smithi07134777-46 2024-09-11T01:37:29.295 INFO:tasks.rados.rados.0.smithi071.stdout:1239: finishing write tid 4 to smithi07134777-46 2024-09-11T01:37:29.295 INFO:tasks.rados.rados.0.smithi071.stdout:1239: oid 46 updating version 0 to 239 2024-09-11T01:37:29.295 INFO:tasks.rados.rados.0.smithi071.stdout:1239: oid 46 version 239 is already newer than 237 2024-09-11T01:37:29.295 INFO:tasks.rados.rados.0.smithi071.stdout:1239: oid 46 version 239 is already newer than 238 2024-09-11T01:37:29.295 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 239 (ObjNum 478 snap 126 seq_num 478) dirty exists 2024-09-11T01:37:29.295 INFO:tasks.rados.rados.0.smithi071.stdout:1239: left oid 46 (ObjNum 478 snap 126 seq_num 478) 2024-09-11T01:37:30.277 INFO:tasks.rados.rados.0.smithi071.stdout:1239: done (1 left) 2024-09-11T01:37:30.277 INFO:tasks.rados.rados.0.smithi071.stdout:1240: done (0 left) 2024-09-11T01:37:30.277 INFO:tasks.rados.rados.0.smithi071.stdout:1241: snap_create 2024-09-11T01:37:31.278 INFO:tasks.rados.rados.0.smithi071.stdout:1241: done (0 left) 2024-09-11T01:37:31.279 INFO:tasks.rados.rados.0.smithi071.stdout:1242: snap_remove snap 124 2024-09-11T01:37:32.278 INFO:tasks.rados.rados.0.smithi071.stdout:1242: done (0 left) 2024-09-11T01:37:32.279 INFO:tasks.rados.rados.0.smithi071.stdout:1243: snap_remove snap 125 2024-09-11T01:37:33.241 INFO:tasks.rados.rados.0.smithi071.stdout:1243: done (0 left) 2024-09-11T01:37:33.242 INFO:tasks.rados.rados.0.smithi071.stdout:1244: write oid 19 current snap is 128 2024-09-11T01:37:33.242 INFO:tasks.rados.rados.0.smithi071.stdout:1244: seq_num 479 ranges {450553=574510,1613463=437090} 2024-09-11T01:37:33.247 INFO:tasks.rados.rados.0.smithi071.stdout:1244: writing smithi07134777-19 from 450553 to 1025063 tid 1 2024-09-11T01:37:33.253 INFO:tasks.rados.rados.0.smithi071.stdout:1244: writing smithi07134777-19 from 1613463 to 2050553 tid 2 2024-09-11T01:37:33.253 INFO:tasks.rados.rados.0.smithi071.stdout:1245: copy_from oid 12 from oid 41 current snap is 128 2024-09-11T01:37:33.253 INFO:tasks.rados.rados.0.smithi071.stdout:1246: snap_remove snap 127 2024-09-11T01:37:33.807 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-11T01:37:33.808 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T01:37:33.808 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T01:37:33.814 INFO:tasks.ceph.mon.a:Started 2024-09-11T01:37:33.814 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:37:33.814 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:37:33.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.925+0000 7f4e12eb6640 10 monclient: get_monmap_and_config 2024-09-11T01:37:33.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.925+0000 7f4e12eb6640 10 monclient: build_initial_monmap 2024-09-11T01:37:33.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.925+0000 7f4e12eb6640 10 monclient: monmap: 2024-09-11T01:37:33.928 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:33.928 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:33.928 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:37:33.926916+0000 2024-09-11T01:37:33.928 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:37:33.926916+0000 2024-09-11T01:37:33.928 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:33.928 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:33.928 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:37:33.928 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:37:33.928 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:37:33.928 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:37:33.928 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:37:33.928 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:33.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.925+0000 7f4e12eb6640 1 Processor -- start 2024-09-11T01:37:33.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.926+0000 7f4e12eb6640 1 -- start start 2024-09-11T01:37:33.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.926+0000 7f4e12eb6640 10 monclient: init 2024-09-11T01:37:33.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.926+0000 7f4e12eb6640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:33.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.926+0000 7f4e12eb6640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-11T01:37:33.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.926+0000 7f4e12eb6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4e0c138230 addr v1:172.21.15.71:6789/0 2024-09-11T01:37:33.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.926+0000 7f4e12eb6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4e0c09d030 addr v1:172.21.15.12:6789/0 2024-09-11T01:37:33.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.926+0000 7f4e12eb6640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4e0c09d460 0x7f4e0c1407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.926+0000 7f4e12eb6640 10 monclient(hunting): picked mon.noname-a con 0x7f4e0c09d460 addr v2:172.21.15.12:3300/0 2024-09-11T01:37:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.926+0000 7f4e12eb6640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e0c0a6e30 con 0x7f4e0c09d030 2024-09-11T01:37:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.926+0000 7f4e12eb6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e0c09ba20 con 0x7f4e0c138230 2024-09-11T01:37:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.926+0000 7f4e12eb6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.926+0000 7f4e12eb6640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f4e0c09c9c0 con 0x7f4e0c09d460 2024-09-11T01:37:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.926+0000 7f4e12eb6640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.926+0000 7f4e12eb6640 10 monclient(hunting): authenticate will time out at 1260.369583s 2024-09-11T01:37:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.926+0000 7f4e11453640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4e0c09d460 0x7f4e0c1407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.926+0000 7f4e03fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4e0c09d030 0x7f4e0c13d090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60802/0 (socket says 172.21.15.12:60802) 2024-09-11T01:37:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.926+0000 7f4e03fff640 1 -- 172.21.15.12:0/664351387 learned_addr learned my addr 172.21.15.12:0/664351387 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:37:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.926+0000 7f4e11453640 10 monclient(hunting): get_auth_request con 0x7f4e0c09d460 auth_method 0 2024-09-11T01:37:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.926+0000 7f4e11453640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:33.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.926+0000 7f4e11453640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:33.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.926+0000 7f4e11453640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:33.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.926+0000 7f4e11453640 1 -- 172.21.15.12:0/664351387 >> v2:172.21.15.12:3300/0 conn(0x7f4e0c09d460 msgr2=0x7f4e0c1407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:37:33.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.926+0000 7f4e11453640 1 -- 172.21.15.12:0/664351387 >> v2:172.21.15.12:3300/0 conn(0x7f4e0c09d460 msgr2=0x7f4e0c1407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:37:33.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.926+0000 7f4e11453640 1 --2- 172.21.15.12:0/664351387 >> v2:172.21.15.12:3300/0 conn(0x7f4e0c09d460 0x7f4e0c1407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:37:33.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.926+0000 7f4e11453640 1 --2- 172.21.15.12:0/664351387 >> v2:172.21.15.12:3300/0 conn(0x7f4e0c09d460 0x7f4e0c1407c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:37:33.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.928+0000 7f4e03fff640 1 -- 172.21.15.12:0/664351387 >> v1:172.21.15.12:6789/0 conn(0x7f4e0c09d030 legacy=0x7f4e0c13d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:33.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e10c52640 1 -- 172.21.15.12:0/664351387 >> v1:172.21.15.71:6789/0 conn(0x7f4e0c138230 legacy=0x7f4e0c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:33.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 1 -- 172.21.15.12:0/664351387 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 550607339 0 0) 0x7f4e0c09ba20 con 0x7f4e0c138230 2024-09-11T01:37:33.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 10 monclient(hunting): _negotiate 2024-09-11T01:37:33.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:33.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:33.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 10 monclient(hunting): my global_id is 74263 2024-09-11T01:37:33.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 1 -- 172.21.15.12:0/664351387 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4df4003520 con 0x7f4e0c138230 2024-09-11T01:37:33.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 1 -- 172.21.15.12:0/664351387 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2014774663 0 0) 0x7f4df4003520 con 0x7f4e0c138230 2024-09-11T01:37:33.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 1 -- 172.21.15.12:0/664351387 >> v2:172.21.15.12:3300/0 conn(0x7f4e0c09d460 msgr2=0x7f4e0c1407c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:37:33.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 1 --2- 172.21.15.12:0/664351387 >> v2:172.21.15.12:3300/0 conn(0x7f4e0c09d460 0x7f4e0c1407c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:33.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 1 -- 172.21.15.12:0/664351387 >> v1:172.21.15.12:6789/0 conn(0x7f4e0c09d030 legacy=0x7f4e0c13d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:33.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 10 monclient: _finish_hunting 0 2024-09-11T01:37:33.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 1 monclient: found mon.noname-c-legacy 2024-09-11T01:37:33.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:33.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-09-11T01:37:33.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 1 -- 172.21.15.12:0/664351387 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e0c0a66a0 con 0x7f4e0c138230 2024-09-11T01:37:33.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 10 monclient: _finish_auth 0 2024-09-11T01:37:33.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 10 monclient: _check_auth_tickets 2024-09-11T01:37:33.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:33.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 1 -- 172.21.15.12:0/664351387 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f4e080019f0 con 0x7f4e0c138230 2024-09-11T01:37:33.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e12eb6640 5 monclient: authenticate success, global_id 74263 2024-09-11T01:37:33.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e12eb6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:37:33.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:33.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-11T01:37:33.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 10 monclient: dump: 2024-09-11T01:37:33.938 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:33.938 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:33.938 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:33.939 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:33.939 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:33.939 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:33.939 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:33.939 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:33.939 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:33.939 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:33.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-09-11T01:37:33.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:33.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 1 -- 172.21.15.12:0/664351387 >> v1:172.21.15.71:6789/0 conn(0x7f4e0c138230 legacy=0x7f4e0c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:33.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:37:33.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 1 --2- 172.21.15.12:0/664351387 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4df4003b20 0x7f4df4003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:33.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 10 monclient(hunting): picked mon.c con 0x7f4df4003b20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:33.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 1 --2- 172.21.15.12:0/664351387 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4df4004800 0x7f4df4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:33.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 10 monclient(hunting): picked mon.a con 0x7f4df4004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:37:33.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 1 --2- 172.21.15.12:0/664351387 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4df40099b0 0x7f4df400bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:33.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 10 monclient(hunting): picked mon.b con 0x7f4df40099b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:37:33.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:33.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 1 -- 172.21.15.12:0/664351387 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0c09c9c0 con 0x7f4df4004800 2024-09-11T01:37:33.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:33.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 1 -- 172.21.15.12:0/664351387 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4df400c3c0 con 0x7f4df40099b0 2024-09-11T01:37:33.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:33.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 1 -- 172.21.15.12:0/664351387 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4df400c5c0 con 0x7f4df4003b20 2024-09-11T01:37:33.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:33.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e037fe640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:37:33.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e12eb6640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:37:33.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e10c52640 1 --2- 172.21.15.12:0/664351387 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4df4003b20 0x7f4df4003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:33.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e03fff640 1 --2- 172.21.15.12:0/664351387 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4df4004800 0x7f4df4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:33.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e10c52640 10 monclient(hunting): get_auth_request con 0x7f4df4003b20 auth_method 0 2024-09-11T01:37:33.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e10c52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:33.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e10c52640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:33.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e10c52640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:37:33.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e03fff640 10 monclient(hunting): get_auth_request con 0x7f4df4004800 auth_method 0 2024-09-11T01:37:33.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:33.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e03fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:33.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e03fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:37:33.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e11453640 1 --2- 172.21.15.12:0/664351387 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4df40099b0 0x7f4df400bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:33.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e11453640 10 monclient(hunting): get_auth_request con 0x7f4df40099b0 auth_method 0 2024-09-11T01:37:33.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e11453640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:33.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e11453640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:33.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e11453640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:37:33.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:33.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:33.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e03fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:37:33.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e10c52640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:33.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e10c52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:33.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e10c52640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:37:33.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e11453640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:33.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e11453640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:33.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.929+0000 7f4e11453640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:37:33.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.930+0000 7f4e03fff640 10 monclient(hunting): handle_auth_done global_id 74263 payload 293 2024-09-11T01:37:33.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.930+0000 7f4e03fff640 1 -- 172.21.15.12:0/664351387 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4df4003b20 msgr2=0x7f4df4003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:33.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.930+0000 7f4e03fff640 1 --2- 172.21.15.12:0/664351387 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4df4003b20 0x7f4df4003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:33.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.930+0000 7f4e03fff640 1 -- 172.21.15.12:0/664351387 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4df40099b0 msgr2=0x7f4df400bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:33.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.930+0000 7f4e03fff640 1 --2- 172.21.15.12:0/664351387 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4df40099b0 0x7f4df400bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:33.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.930+0000 7f4e03fff640 10 monclient: _finish_hunting 0 2024-09-11T01:37:33.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.930+0000 7f4e03fff640 1 monclient: found mon.a 2024-09-11T01:37:33.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.930+0000 7f4e03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:33.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.930+0000 7f4e03fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:33.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.930+0000 7f4e03fff640 1 -- 172.21.15.12:0/664351387 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df4003520 con 0x7f4df4004800 2024-09-11T01:37:33.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.930+0000 7f4e03fff640 10 monclient: _finish_auth 0 2024-09-11T01:37:33.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.930+0000 7f4e03fff640 10 monclient: _check_auth_tickets 2024-09-11T01:37:33.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.930+0000 7f4e03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:33.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.930+0000 7f4e11453640 1 -- 172.21.15.12:0/664351387 reap_dead start 2024-09-11T01:37:33.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.932+0000 7f4e03fff640 1 -- 172.21.15.12:0/664351387 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4df4004800 msgr2=0x7f4df4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:33.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.932+0000 7f4e03fff640 1 --2- 172.21.15.12:0/664351387 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4df4004800 0x7f4df4004c80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc001a30 tx=0x7f4df4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9baa6283afac7160 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:33.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.933+0000 7f4e037fe640 1 -- 172.21.15.12:0/664351387 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc001760 con 0x7f4df4004800 2024-09-11T01:37:33.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.933+0000 7f4e037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:33.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.933+0000 7f4e037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:33.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.933+0000 7f4e037fe640 10 monclient: dump: 2024-09-11T01:37:33.944 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:33.944 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:33.944 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:33.944 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:33.944 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:33.945 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:33.945 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:33.945 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:33.945 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:33.945 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:33.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.933+0000 7f4e12eb6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:37:33.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.933+0000 7f4e037fe640 1 -- 172.21.15.12:0/664351387 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dfc02e070 con 0x7f4df4004800 2024-09-11T01:37:33.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.933+0000 7f4e037fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:33.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.933+0000 7f4e12eb6640 10 monclient: get_monmap_and_config success 2024-09-11T01:37:33.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.933+0000 7f4e12eb6640 4 set_mon_vals no callback set 2024-09-11T01:37:33.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.933+0000 7f4e037fe640 1 -- 172.21.15.12:0/664351387 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc0048a0 con 0x7f4df4004800 2024-09-11T01:37:33.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.933+0000 7f4e037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:33.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.933+0000 7f4e037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:33.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.933+0000 7f4e037fe640 10 monclient: dump: 2024-09-11T01:37:33.946 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:33.946 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:33.946 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:33.946 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:33.946 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:33.946 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:33.946 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:33.946 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:33.946 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:33.946 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:33.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.933+0000 7f4e12eb6640 10 monclient: shutdown 2024-09-11T01:37:33.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.933+0000 7f4e12eb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:33.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.933+0000 7f4e12eb6640 1 -- 172.21.15.12:0/664351387 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4df4004800 msgr2=0x7f4df4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:33.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.933+0000 7f4e12eb6640 1 --2- 172.21.15.12:0/664351387 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4df4004800 0x7f4df4004c80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc001a30 tx=0x7f4df4005830 comp rx=0 tx=0).stop 2024-09-11T01:37:33.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.933+0000 7f4e12eb6640 1 -- 172.21.15.12:0/664351387 shutdown_connections 2024-09-11T01:37:33.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.933+0000 7f4e12eb6640 1 --2- 172.21.15.12:0/664351387 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4df4004800 0x7f4df4004c80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:33.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.933+0000 7f4e12eb6640 1 -- 172.21.15.12:0/664351387 >> 172.21.15.12:0/664351387 conn(0x7f4e0c0a7120 msgr2=0x7f4e0c0a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:33.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.933+0000 7f4e12eb6640 1 -- 172.21.15.12:0/664351387 shutdown_connections 2024-09-11T01:37:33.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.933+0000 7f4e12eb6640 1 -- 172.21.15.12:0/664351387 wait complete. 2024-09-11T01:37:33.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.934+0000 7f4e12eb6640 10 monclient: build_initial_monmap 2024-09-11T01:37:33.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.934+0000 7f4e12eb6640 10 monclient: monmap: 2024-09-11T01:37:33.951 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:33.951 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:37:33.951 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:37:33.951 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:37:33.951 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:33.951 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:33.951 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:37:33.951 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:37:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:37:33.952 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.934+0000 7f4e12eb6640 1 Processor -- start 2024-09-11T01:37:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.934+0000 7f4e12eb6640 1 -- start start 2024-09-11T01:37:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.934+0000 7f4e12eb6640 10 monclient: init 2024-09-11T01:37:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e12eb6640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e12eb6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:37:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e12eb6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4df4004800 0x7f4e0c1ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e12eb6640 10 monclient(hunting): picked mon.noname-b con 0x7f4df4004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:37:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e12eb6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e0c137050 0x7f4e0c1374d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e12eb6640 10 monclient(hunting): picked mon.noname-a con 0x7f4e0c137050 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:37:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e12eb6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4e0c120c70 0x7f4e0c1210e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e12eb6640 10 monclient(hunting): picked mon.noname-c con 0x7f4e0c120c70 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:33.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e12eb6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e12eb6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0c138790 con 0x7f4e0c137050 2024-09-11T01:37:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e12eb6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e12eb6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0c138610 con 0x7f4df4004800 2024-09-11T01:37:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e12eb6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e12eb6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4e0c138910 con 0x7f4e0c120c70 2024-09-11T01:37:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e12eb6640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e12eb6640 10 monclient(hunting): authenticate will time out at 1260.378491s 2024-09-11T01:37:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e11453640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4e0c120c70 0x7f4e0c1210e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e03fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e0c137050 0x7f4e0c1374d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e11453640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4e0c120c70 0x7f4e0c1210e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:32886/0 (socket says 172.21.15.12:32886) 2024-09-11T01:37:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e03fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e0c137050 0x7f4e0c1374d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40614/0 (socket says 172.21.15.12:40614) 2024-09-11T01:37:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e11453640 1 -- 172.21.15.12:0/2676575732 learned_addr learned my addr 172.21.15.12:0/2676575732 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:33.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e11453640 10 monclient(hunting): get_auth_request con 0x7f4e0c120c70 auth_method 0 2024-09-11T01:37:33.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e11453640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:33.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e11453640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:33.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e11453640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:33.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e03fff640 10 monclient(hunting): get_auth_request con 0x7f4e0c137050 auth_method 0 2024-09-11T01:37:33.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:33.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e03fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:33.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:33.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e10c52640 1 --2- 172.21.15.12:0/2676575732 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4df4004800 0x7f4e0c1ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:33.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e11453640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:33.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e11453640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:33.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e11453640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:33.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e10c52640 10 monclient(hunting): get_auth_request con 0x7f4df4004800 auth_method 0 2024-09-11T01:37:33.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e10c52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:33.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e10c52640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:33.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e10c52640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:33.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e03fff640 1 -- 172.21.15.12:0/2676575732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e0c137050 msgr2=0x7f4e0c1374d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:37:33.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e03fff640 1 -- 172.21.15.12:0/2676575732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e0c137050 msgr2=0x7f4e0c1374d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:37:33.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e03fff640 1 --2- 172.21.15.12:0/2676575732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e0c137050 0x7f4e0c1374d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:37:33.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e03fff640 1 --2- 172.21.15.12:0/2676575732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e0c137050 0x7f4e0c1374d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:37:33.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e10c52640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:33.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e10c52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:33.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e10c52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:33.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e11453640 10 monclient(hunting): handle_auth_done global_id 84185 payload 824 2024-09-11T01:37:33.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e11453640 1 -- 172.21.15.12:0/2676575732 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4df4004800 msgr2=0x7f4e0c1ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:33.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e11453640 1 --2- 172.21.15.12:0/2676575732 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4df4004800 0x7f4e0c1ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:33.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e11453640 1 -- 172.21.15.12:0/2676575732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e0c137050 msgr2=0x7f4e0c1374d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:37:33.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e11453640 1 --2- 172.21.15.12:0/2676575732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e0c137050 0x7f4e0c1374d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:33.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e11453640 10 monclient: _finish_hunting 0 2024-09-11T01:37:33.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e11453640 1 monclient: found mon.noname-c 2024-09-11T01:37:33.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e11453640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:33.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e11453640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:37:33.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.935+0000 7f4e11453640 1 -- 172.21.15.12:0/2676575732 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e0c137a10 con 0x7f4e0c120c70 2024-09-11T01:37:33.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.938+0000 7f4e11453640 1 -- 172.21.15.12:0/2676575732 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4e0c120c70 msgr2=0x7f4e0c1210e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:33.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.938+0000 7f4e11453640 1 --2- 172.21.15.12:0/2676575732 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4e0c120c70 0x7f4e0c1210e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c09d860 tx=0x7f4e04007700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:33.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.938+0000 7f4e017fa640 1 -- 172.21.15.12:0/2676575732 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e04013780 con 0x7f4e0c120c70 2024-09-11T01:37:33.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.938+0000 7f4e017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:33.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.938+0000 7f4e017fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:37:33.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.938+0000 7f4e017fa640 10 monclient: dump: 2024-09-11T01:37:33.956 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:33.957 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:33.957 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:33.957 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:33.957 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:33.957 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:33.957 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:33.957 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:33.957 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:33.957 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:33.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.938+0000 7f4e017fa640 10 monclient: _finish_auth 0 2024-09-11T01:37:33.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.938+0000 7f4e017fa640 10 monclient: _check_auth_tickets 2024-09-11T01:37:33.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.938+0000 7f4e017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:33.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.938+0000 7f4e017fa640 1 -- 172.21.15.12:0/2676575732 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e04019070 con 0x7f4e0c120c70 2024-09-11T01:37:33.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.938+0000 7f4e12eb6640 5 monclient: authenticate success, global_id 84185 2024-09-11T01:37:33.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.938+0000 7f4e12eb6640 10 monclient: _renew_subs 2024-09-11T01:37:33.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.938+0000 7f4e12eb6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:33.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.938+0000 7f4e12eb6640 1 -- 172.21.15.12:0/2676575732 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e0c1226b0 con 0x7f4e0c120c70 2024-09-11T01:37:33.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.938+0000 7f4e017fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:33.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.938+0000 7f4e017fa640 1 -- 172.21.15.12:0/2676575732 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e04013b60 con 0x7f4e0c120c70 2024-09-11T01:37:33.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.938+0000 7f4e017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:33.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.938+0000 7f4e017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:33.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.938+0000 7f4e017fa640 10 monclient: dump: 2024-09-11T01:37:33.958 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:33.958 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:33.958 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:33.958 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:33.958 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:33.959 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.938+0000 7f4e12eb6640 10 monclient: _renew_subs 2024-09-11T01:37:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.938+0000 7f4e12eb6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.938+0000 7f4e12eb6640 1 -- 172.21.15.12:0/2676575732 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e0c0a1810 con 0x7f4e0c120c70 2024-09-11T01:37:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.938+0000 7f4e02ffd640 4 set_mon_vals no callback set 2024-09-11T01:37:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.939+0000 7f4e017fa640 1 -- 172.21.15.12:0/2676575732 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f4e0401d410 con 0x7f4e0c120c70 2024-09-11T01:37:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.939+0000 7f4e12eb6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.939+0000 7f4e12eb6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.939+0000 7f4e12eb6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.939+0000 7f4e12eb6640 1 -- 172.21.15.12:0/2676575732 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e0c1243b0 con 0x7f4e0c120c70 2024-09-11T01:37:33.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.939+0000 7f4e017fa640 1 --2- 172.21.15.12:0/2676575732 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4df803e210 0x7f4df80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:33.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.939+0000 7f4e017fa640 1 -- 172.21.15.12:0/2676575732 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(279..279 src has 186..279) ==== 4795+0+0 (secure 0 0 0) 0x7f4e04053520 con 0x7f4e0c120c70 2024-09-11T01:37:33.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.939+0000 7f4e10c52640 1 --2- 172.21.15.12:0/2676575732 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4df803e210 0x7f4df80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:33.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.939+0000 7f4e10c52640 10 monclient: get_auth_request con 0x7f4df803e210 auth_method 0 2024-09-11T01:37:33.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.940+0000 7f4e10c52640 1 -- 172.21.15.12:0/2676575732 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4df803e210 msgr2=0x7f4df80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:37:33.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.940+0000 7f4e10c52640 1 -- 172.21.15.12:0/2676575732 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4df803e210 msgr2=0x7f4df80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:37:33.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.940+0000 7f4e10c52640 1 --2- 172.21.15.12:0/2676575732 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4df803e210 0x7f4df80406d0 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c09de10 tx=0x7f4e08009040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:37:33.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.940+0000 7f4e10c52640 1 --2- 172.21.15.12:0/2676575732 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4df803e210 0x7f4df80406d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:37:33.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.944+0000 7f4e017fa640 1 -- 172.21.15.12:0/2676575732 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4e04051f10 con 0x7f4e0c120c70 2024-09-11T01:37:33.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.944+0000 7f4e017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:33.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:33.944+0000 7f4e017fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:37:34.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:34.141+0000 7f4e10c52640 1 --2- 172.21.15.12:0/2676575732 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4df803e210 0x7f4df80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:34.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:34.141+0000 7f4e10c52640 10 monclient: get_auth_request con 0x7f4df803e210 auth_method 0 2024-09-11T01:37:34.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:34.144+0000 7f4e10c52640 1 --2- 172.21.15.12:0/2676575732 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4df803e210 0x7f4df80406d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4e08007fd0 tx=0x7f4e08015000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:34.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:34.204+0000 7f4e12eb6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:37:34.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:34.204+0000 7f4e12eb6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:37:34.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:34.204+0000 7f4e12eb6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:34.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:34.204+0000 7f4e12eb6640 1 -- 172.21.15.12:0/2676575732 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4e0c0a5020 con 0x7f4e0c120c70 2024-09-11T01:37:34.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:34.205+0000 7f4e017fa640 1 -- 172.21.15.12:0/2676575732 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f4e040556d0 con 0x7f4e0c120c70 2024-09-11T01:37:34.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:34.205+0000 7f4e017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:37:34.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:34.205+0000 7f4e017fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:37:34.208 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":84,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:37:34.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:34.207+0000 7f4e12eb6640 1 -- 172.21.15.12:0/2676575732 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4df803e210 msgr2=0x7f4df80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:34.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:34.207+0000 7f4e12eb6640 1 --2- 172.21.15.12:0/2676575732 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4df803e210 0x7f4df80406d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4e08007fd0 tx=0x7f4e08015000 comp rx=0 tx=0).stop 2024-09-11T01:37:34.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:34.207+0000 7f4e12eb6640 10 monclient: shutdown 2024-09-11T01:37:34.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:34.207+0000 7f4e12eb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:34.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:34.207+0000 7f4e12eb6640 1 -- 172.21.15.12:0/2676575732 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4e0c120c70 msgr2=0x7f4e0c1210e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:34.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:34.207+0000 7f4e12eb6640 1 --2- 172.21.15.12:0/2676575732 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4e0c120c70 0x7f4e0c1210e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c09d860 tx=0x7f4e04007700 comp rx=0 tx=0).stop 2024-09-11T01:37:34.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:34.207+0000 7f4e12eb6640 1 -- 172.21.15.12:0/2676575732 shutdown_connections 2024-09-11T01:37:34.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:34.207+0000 7f4e12eb6640 1 --2- 172.21.15.12:0/2676575732 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4e0c120c70 0x7f4e0c1210e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:34.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:34.207+0000 7f4e12eb6640 1 --2- 172.21.15.12:0/2676575732 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4df803e210 0x7f4df80406d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:34.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:34.207+0000 7f4e12eb6640 1 --2- 172.21.15.12:0/2676575732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e0c137050 0x7f4e0c1374d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:34.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:34.207+0000 7f4e12eb6640 1 --2- 172.21.15.12:0/2676575732 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4df4004800 0x7f4e0c1ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:34.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:34.207+0000 7f4e12eb6640 1 -- 172.21.15.12:0/2676575732 >> 172.21.15.12:0/2676575732 conn(0x7f4e0c0a7120 msgr2=0x7f4e0c0a2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:34.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:34.207+0000 7f4e12eb6640 1 -- 172.21.15.12:0/2676575732 shutdown_connections 2024-09-11T01:37:34.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:34.207+0000 7f4e12eb6640 1 -- 172.21.15.12:0/2676575732 wait complete. 2024-09-11T01:37:34.251 INFO:tasks.rados.rados.0.smithi071.stdout:1246: done (2 left) 2024-09-11T01:37:34.251 INFO:tasks.rados.rados.0.smithi071.stdout:1247: write oid 21 current snap is 128 2024-09-11T01:37:34.251 INFO:tasks.rados.rados.0.smithi071.stdout:1247: seq_num 480 ranges {467360=649759,1671945=395415} 2024-09-11T01:37:34.257 INFO:tasks.rados.rados.0.smithi071.stdout:1247: writing smithi07134777-21 from 467360 to 1117119 tid 1 2024-09-11T01:37:34.263 INFO:tasks.rados.rados.0.smithi071.stdout:1247: writing smithi07134777-21 from 1671945 to 2067360 tid 2 2024-09-11T01:37:34.263 INFO:tasks.rados.rados.0.smithi071.stdout:1248: read oid 2 snap -1 2024-09-11T01:37:34.263 INFO:tasks.rados.rados.0.smithi071.stdout:1248: expect deleted 2024-09-11T01:37:34.263 INFO:tasks.rados.rados.0.smithi071.stdout:1249: snap_create 2024-09-11T01:37:34.264 INFO:tasks.rados.rados.0.smithi071.stdout:1244: finishing write tid 1 to smithi07134777-19 2024-09-11T01:37:34.264 INFO:tasks.rados.rados.0.smithi071.stdout:1244: finishing write tid 2 to smithi07134777-19 2024-09-11T01:37:34.264 INFO:tasks.rados.rados.0.smithi071.stdout:1244: finishing write tid 3 to smithi07134777-19 2024-09-11T01:37:34.264 INFO:tasks.rados.rados.0.smithi071.stdout:1244: finishing write tid 4 to smithi07134777-19 2024-09-11T01:37:34.264 INFO:tasks.rados.rados.0.smithi071.stdout:1244: oid 19 updating version 0 to 441 2024-09-11T01:37:34.264 INFO:tasks.rados.rados.0.smithi071.stdout:1244: oid 19 version 441 is already newer than 440 2024-09-11T01:37:34.264 INFO:tasks.rados.rados.0.smithi071.stdout:1244: oid 19 version 441 is already newer than 439 2024-09-11T01:37:34.264 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 441 (ObjNum 479 snap 128 seq_num 479) dirty exists 2024-09-11T01:37:34.264 INFO:tasks.rados.rados.0.smithi071.stdout:1244: left oid 19 (ObjNum 479 snap 128 seq_num 479) 2024-09-11T01:37:34.264 INFO:tasks.rados.rados.0.smithi071.stdout:1245: finishing copy_from racing read to smithi07134777-12 2024-09-11T01:37:34.264 INFO:tasks.rados.rados.0.smithi071.stdout:1245: finishing copy_from to smithi07134777-12 2024-09-11T01:37:34.264 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 355 (ObjNum 469 snap 124 seq_num 469) dirty exists 2024-09-11T01:37:34.265 INFO:tasks.rados.rados.0.smithi071.stdout:1247: finishing write tid 1 to smithi07134777-21 2024-09-11T01:37:34.268 INFO:tasks.rados.rados.0.smithi071.stdout:1247: finishing write tid 2 to smithi07134777-21 2024-09-11T01:37:34.269 INFO:tasks.rados.rados.0.smithi071.stdout:1247: finishing write tid 3 to smithi07134777-21 2024-09-11T01:37:34.269 INFO:tasks.rados.rados.0.smithi071.stdout:1247: finishing write tid 4 to smithi07134777-21 2024-09-11T01:37:34.269 INFO:tasks.rados.rados.0.smithi071.stdout:1247: oid 21 updating version 0 to 510 2024-09-11T01:37:34.269 INFO:tasks.rados.rados.0.smithi071.stdout:1247: oid 21 updating version 510 to 512 2024-09-11T01:37:34.269 INFO:tasks.rados.rados.0.smithi071.stdout:1247: oid 21 version 512 is already newer than 511 2024-09-11T01:37:34.269 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 512 (ObjNum 480 snap 128 seq_num 480) dirty exists 2024-09-11T01:37:34.269 INFO:tasks.rados.rados.0.smithi071.stdout:1247: left oid 21 (ObjNum 480 snap 128 seq_num 480) 2024-09-11T01:37:35.254 INFO:tasks.rados.rados.0.smithi071.stdout:1244: done (4 left) 2024-09-11T01:37:35.254 INFO:tasks.rados.rados.0.smithi071.stdout:1245: done (3 left) 2024-09-11T01:37:35.254 INFO:tasks.rados.rados.0.smithi071.stdout:1247: done (2 left) 2024-09-11T01:37:35.254 INFO:tasks.rados.rados.0.smithi071.stdout:1248: done (1 left) 2024-09-11T01:37:35.254 INFO:tasks.rados.rados.0.smithi071.stdout:1249: done (0 left) 2024-09-11T01:37:35.254 INFO:tasks.rados.rados.0.smithi071.stdout:1250: read oid 16 snap -1 2024-09-11T01:37:35.254 INFO:tasks.rados.rados.0.smithi071.stdout:1250: expect (ObjNum 411 snap 115 seq_num 411) 2024-09-11T01:37:35.254 INFO:tasks.rados.rados.0.smithi071.stdout:1251: rollback oid 43 current snap is 129 2024-09-11T01:37:35.255 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 43 to 128 2024-09-11T01:37:35.255 INFO:tasks.rados.rados.0.smithi071.stdout:1252: write (excl) oid 15 current snap is 129 2024-09-11T01:37:35.255 INFO:tasks.rados.rados.0.smithi071.stdout:1252: seq_num 481 ranges {484167=725008,1730427=353740} 2024-09-11T01:37:35.260 INFO:tasks.rados.rados.0.smithi071.stdout:1252: writing smithi07134777-15 from 484167 to 1209175 tid 1 2024-09-11T01:37:35.266 INFO:tasks.rados.rados.0.smithi071.stdout:1252: writing smithi07134777-15 from 1730427 to 2084167 tid 2 2024-09-11T01:37:35.266 INFO:tasks.rados.rados.0.smithi071.stdout:1253: write oid 20 current snap is 129 2024-09-11T01:37:35.266 INFO:tasks.rados.rados.0.smithi071.stdout:1253: seq_num 482 ranges {500974=400257,1388909=557169} 2024-09-11T01:37:35.271 INFO:tasks.rados.rados.0.smithi071.stdout:1253: writing smithi07134777-20 from 500974 to 901231 tid 1 2024-09-11T01:37:35.277 INFO:tasks.rados.rados.0.smithi071.stdout:1253: writing smithi07134777-20 from 1388909 to 1946078 tid 2 2024-09-11T01:37:35.277 INFO:tasks.rados.rados.0.smithi071.stdout:1254: read oid 32 snap -1 2024-09-11T01:37:35.277 INFO:tasks.rados.rados.0.smithi071.stdout:1254: expect deleted 2024-09-11T01:37:35.277 INFO:tasks.rados.rados.0.smithi071.stdout:1255: snap_create 2024-09-11T01:37:35.278 INFO:tasks.rados.rados.0.smithi071.stdout:1251: finishing rollback tid 0 to smithi07134777-43 2024-09-11T01:37:35.278 INFO:tasks.rados.rados.0.smithi071.stdout:1251: finishing rollback tid 1 to smithi07134777-43 2024-09-11T01:37:35.278 INFO:tasks.rados.rados.0.smithi071.stdout:1251: finishing rollback tid 2 to smithi07134777-43 2024-09-11T01:37:35.278 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 590 (ObjNum 468 snap 124 seq_num 468) dirty exists 2024-09-11T01:37:35.278 INFO:tasks.rados.rados.0.smithi071.stdout:1250: expect (ObjNum 411 snap 115 seq_num 411) 2024-09-11T01:37:35.344 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2262010 2024-09-11T01:37:35.406 INFO:tasks.rados.rados.0.smithi071.stdout:1252: finishing write tid 1 to smithi07134777-15 2024-09-11T01:37:35.406 INFO:tasks.rados.rados.0.smithi071.stdout:1252: finishing write tid 2 to smithi07134777-15 2024-09-11T01:37:35.406 INFO:tasks.rados.rados.0.smithi071.stdout:1252: finishing write tid 3 to smithi07134777-15 2024-09-11T01:37:35.406 INFO:tasks.rados.rados.0.smithi071.stdout:1252: finishing write tid 4 to smithi07134777-15 2024-09-11T01:37:35.406 INFO:tasks.rados.rados.0.smithi071.stdout:1252: oid 15 updating version 0 to 631 2024-09-11T01:37:35.407 INFO:tasks.rados.rados.0.smithi071.stdout:1252: oid 15 version 631 is already newer than 630 2024-09-11T01:37:35.407 INFO:tasks.rados.rados.0.smithi071.stdout:1252: oid 15 updating version 631 to 632 2024-09-11T01:37:35.407 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 632 (ObjNum 481 snap 129 seq_num 481) dirty exists 2024-09-11T01:37:35.407 INFO:tasks.rados.rados.0.smithi071.stdout:1252: left oid 15 (ObjNum 481 snap 129 seq_num 481) 2024-09-11T01:37:35.407 INFO:tasks.rados.rados.0.smithi071.stdout:1253: finishing write tid 1 to smithi07134777-20 2024-09-11T01:37:35.407 INFO:tasks.rados.rados.0.smithi071.stdout:1253: finishing write tid 2 to smithi07134777-20 2024-09-11T01:37:35.407 INFO:tasks.rados.rados.0.smithi071.stdout:1253: finishing write tid 3 to smithi07134777-20 2024-09-11T01:37:35.407 INFO:tasks.rados.rados.0.smithi071.stdout:1253: finishing write tid 4 to smithi07134777-20 2024-09-11T01:37:35.407 INFO:tasks.rados.rados.0.smithi071.stdout:1253: oid 20 updating version 0 to 252 2024-09-11T01:37:35.407 INFO:tasks.rados.rados.0.smithi071.stdout:1253: oid 20 updating version 252 to 254 2024-09-11T01:37:35.407 INFO:tasks.rados.rados.0.smithi071.stdout:1253: oid 20 version 254 is already newer than 253 2024-09-11T01:37:35.407 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 254 (ObjNum 482 snap 129 seq_num 482) dirty exists 2024-09-11T01:37:35.407 INFO:tasks.rados.rados.0.smithi071.stdout:1253: left oid 20 (ObjNum 482 snap 129 seq_num 482) 2024-09-11T01:37:36.256 INFO:tasks.rados.rados.0.smithi071.stdout:1250: done (5 left) 2024-09-11T01:37:36.256 INFO:tasks.rados.rados.0.smithi071.stdout:1251: done (4 left) 2024-09-11T01:37:36.256 INFO:tasks.rados.rados.0.smithi071.stdout:1252: done (3 left) 2024-09-11T01:37:36.256 INFO:tasks.rados.rados.0.smithi071.stdout:1253: done (2 left) 2024-09-11T01:37:36.256 INFO:tasks.rados.rados.0.smithi071.stdout:1254: done (1 left) 2024-09-11T01:37:36.256 INFO:tasks.rados.rados.0.smithi071.stdout:1255: done (0 left) 2024-09-11T01:37:36.257 INFO:tasks.rados.rados.0.smithi071.stdout:1256: read oid 5 snap -1 2024-09-11T01:37:36.257 INFO:tasks.rados.rados.0.smithi071.stdout:1256: expect (ObjNum 226 snap 55 seq_num 226) 2024-09-11T01:37:36.257 INFO:tasks.rados.rados.0.smithi071.stdout:1257: read oid 49 snap 128 2024-09-11T01:37:36.257 INFO:tasks.rados.rados.0.smithi071.stdout:1257: expect deleted 2024-09-11T01:37:36.257 INFO:tasks.rados.rados.0.smithi071.stdout:1258: rollback oid 28 current snap is 130 2024-09-11T01:37:36.257 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 28 to 126 2024-09-11T01:37:36.257 INFO:tasks.rados.rados.0.smithi071.stdout:1259: read oid 10 snap -1 2024-09-11T01:37:36.257 INFO:tasks.rados.rados.0.smithi071.stdout:1259: expect (ObjNum 460 snap 122 seq_num 460) 2024-09-11T01:37:36.257 INFO:tasks.rados.rados.0.smithi071.stdout:1260: write (excl) oid 37 current snap is 130 2024-09-11T01:37:36.257 INFO:tasks.rados.rados.0.smithi071.stdout:1260: seq_num 483 ranges {517781=475506,1731038=386743} 2024-09-11T01:37:36.261 INFO:tasks.rados.rados.0.smithi071.stdout:1260: writing smithi07134777-37 from 517781 to 993287 tid 1 2024-09-11T01:37:36.268 INFO:tasks.rados.rados.0.smithi071.stdout:1260: writing smithi07134777-37 from 1731038 to 2117781 tid 2 2024-09-11T01:37:36.268 INFO:tasks.rados.rados.0.smithi071.stdout:1261: rollback oid 1 current snap is 130 2024-09-11T01:37:36.268 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 1 to 128 2024-09-11T01:37:36.268 INFO:tasks.rados.rados.0.smithi071.stdout:1258: finishing rollback tid 0 to smithi07134777-28 2024-09-11T01:37:36.269 INFO:tasks.rados.rados.0.smithi071.stdout:1258: finishing rollback tid 1 to smithi07134777-28 2024-09-11T01:37:36.269 INFO:tasks.rados.rados.0.smithi071.stdout:1258: finishing rollback tid 2 to smithi07134777-28 2024-09-11T01:37:36.269 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 516 (ObjNum 418 snap 117 seq_num 418) dirty exists 2024-09-11T01:37:36.269 INFO:tasks.rados.rados.0.smithi071.stdout:1256: expect (ObjNum 226 snap 55 seq_num 226) 2024-09-11T01:37:36.287 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3438732 2024-09-11T01:37:36.353 INFO:tasks.rados.rados.0.smithi071.stdout:1259: expect (ObjNum 460 snap 122 seq_num 460) 2024-09-11T01:37:36.353 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 276600 2024-09-11T01:37:36.423 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:37:36.556 INFO:tasks.rados.rados.0.smithi071.stdout:1261: finishing rollback tid 1 to smithi07134777-1 2024-09-11T01:37:36.556 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 0 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-11T01:37:36.556 INFO:tasks.rados.rados.0.smithi071.stdout:1260: finishing write tid 1 to smithi07134777-37 2024-09-11T01:37:36.556 INFO:tasks.rados.rados.0.smithi071.stdout:1260: finishing write tid 2 to smithi07134777-37 2024-09-11T01:37:36.556 INFO:tasks.rados.rados.0.smithi071.stdout:1260: finishing write tid 3 to smithi07134777-37 2024-09-11T01:37:36.556 INFO:tasks.rados.rados.0.smithi071.stdout:1260: finishing write tid 4 to smithi07134777-37 2024-09-11T01:37:36.556 INFO:tasks.rados.rados.0.smithi071.stdout:1260: oid 37 updating version 0 to 566 2024-09-11T01:37:36.557 INFO:tasks.rados.rados.0.smithi071.stdout:1260: oid 37 updating version 566 to 568 2024-09-11T01:37:36.557 INFO:tasks.rados.rados.0.smithi071.stdout:1260: oid 37 version 568 is already newer than 567 2024-09-11T01:37:36.557 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 568 (ObjNum 483 snap 130 seq_num 483) dirty exists 2024-09-11T01:37:36.557 INFO:tasks.rados.rados.0.smithi071.stdout:1260: left oid 37 (ObjNum 483 snap 130 seq_num 483) 2024-09-11T01:37:36.557 INFO:tasks.rados.rados.0.smithi071.stdout:1256: done (5 left) 2024-09-11T01:37:36.557 INFO:tasks.rados.rados.0.smithi071.stdout:1257: done (4 left) 2024-09-11T01:37:36.557 INFO:tasks.rados.rados.0.smithi071.stdout:1258: done (3 left) 2024-09-11T01:37:36.557 INFO:tasks.rados.rados.0.smithi071.stdout:1259: done (2 left) 2024-09-11T01:37:36.557 INFO:tasks.rados.rados.0.smithi071.stdout:1260: done (1 left) 2024-09-11T01:37:36.557 INFO:tasks.rados.rados.0.smithi071.stdout:1261: done (0 left) 2024-09-11T01:37:36.557 INFO:tasks.rados.rados.0.smithi071.stdout:1262: read oid 33 snap -1 2024-09-11T01:37:36.557 INFO:tasks.rados.rados.0.smithi071.stdout:1262: expect deleted 2024-09-11T01:37:36.557 INFO:tasks.rados.rados.0.smithi071.stdout:1263: delete oid 13 current snap is 130 2024-09-11T01:37:36.560 INFO:tasks.rados.rados.0.smithi071.stdout:1262: done (1 left) 2024-09-11T01:37:36.560 INFO:tasks.rados.rados.0.smithi071.stdout:1263: done (0 left) 2024-09-11T01:37:36.560 INFO:tasks.rados.rados.0.smithi071.stdout:1264: snap_remove snap 129 2024-09-11T01:37:37.222 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:37:37.264 INFO:tasks.rados.rados.0.smithi071.stdout:1264: done (0 left) 2024-09-11T01:37:37.264 INFO:tasks.rados.rados.0.smithi071.stdout:1265: snap_create 2024-09-11T01:37:37.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.334+0000 7f122dbe8640 10 monclient: get_monmap_and_config 2024-09-11T01:37:37.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.334+0000 7f122dbe8640 10 monclient: build_initial_monmap 2024-09-11T01:37:37.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.334+0000 7f122dbe8640 10 monclient: monmap: 2024-09-11T01:37:37.337 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:37.337 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:37.337 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:37:37.336474+0000 2024-09-11T01:37:37.337 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:37:37.336474+0000 2024-09-11T01:37:37.337 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:37.337 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:37.338 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:37:37.338 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:37:37.338 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:37:37.338 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:37:37.338 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:37:37.338 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:37.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.335+0000 7f122dbe8640 1 Processor -- start 2024-09-11T01:37:37.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.335+0000 7f122dbe8640 1 -- start start 2024-09-11T01:37:37.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.335+0000 7f122dbe8640 10 monclient: init 2024-09-11T01:37:37.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f122dbe8640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:37.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f122dbe8640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-11T01:37:37.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f122dbe8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1228070130 addr v1:172.21.15.71:6789/0 2024-09-11T01:37:37.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f122dbe8640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1228136630 0x7f122813ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:37.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f122dbe8640 10 monclient(hunting): picked mon.noname-c con 0x7f1228136630 addr v2:172.21.15.71:3300/0 2024-09-11T01:37:37.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f122dbe8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f12280741b0 addr v1:172.21.15.12:6789/0 2024-09-11T01:37:37.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f122dbe8640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1228135ad0 con 0x7f12280741b0 2024-09-11T01:37:37.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f122dbe8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1228073440 con 0x7f1228070130 2024-09-11T01:37:37.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f122dbe8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:37.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f122dbe8640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f12280699b0 con 0x7f1228136630 2024-09-11T01:37:37.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f122dbe8640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:37.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f122dbe8640 10 monclient(hunting): authenticate will time out at 1263.779356s 2024-09-11T01:37:37.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f1227fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f12280741b0 0x7f122813d290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55880/0 (socket says 172.21.15.12:55880) 2024-09-11T01:37:37.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f1227fff640 1 -- 172.21.15.12:0/1016662869 learned_addr learned my addr 172.21.15.12:0/1016662869 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:37:37.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f121effd640 1 --2- 172.21.15.12:0/1016662869 >> v2:172.21.15.71:3300/0 conn(0x7f1228136630 0x7f122813ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:37.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f121effd640 10 monclient(hunting): get_auth_request con 0x7f1228136630 auth_method 0 2024-09-11T01:37:37.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f121effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:37.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f121effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:37.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f121effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:37.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f121effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:37.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f121effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:37.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f121effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:37.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f121effd640 10 monclient(hunting): handle_auth_done global_id 74269 payload 274 2024-09-11T01:37:37.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f121effd640 1 -- 172.21.15.12:0/1016662869 >> v1:172.21.15.71:6789/0 conn(0x7f1228070130 legacy=0x7f1228135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:37.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f121effd640 1 -- 172.21.15.12:0/1016662869 >> v1:172.21.15.12:6789/0 conn(0x7f12280741b0 legacy=0x7f122813d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:37.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f121effd640 10 monclient: _finish_hunting 0 2024-09-11T01:37:37.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f121effd640 1 monclient: found mon.noname-c 2024-09-11T01:37:37.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f121effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:37.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f121effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:37:37.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.336+0000 7f121effd640 1 -- 172.21.15.12:0/1016662869 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1228073720 con 0x7f1228136630 2024-09-11T01:37:37.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.339+0000 7f121effd640 1 -- 172.21.15.12:0/1016662869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1228136630 msgr2=0x7f122813ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:37.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.339+0000 7f121effd640 1 --2- 172.21.15.12:0/1016662869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1228136630 0x7f122813ad40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f12100097a0 tx=0x7f1210029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4138291c977de3b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:37.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.340+0000 7f1226ffd640 1 -- 172.21.15.12:0/1016662869 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1210037070 con 0x7f1228136630 2024-09-11T01:37:37.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.340+0000 7f1226ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:37.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.340+0000 7f1226ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:37:37.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.340+0000 7f1226ffd640 10 monclient: dump: 2024-09-11T01:37:37.347 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:37.347 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:37.347 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:37.348 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:37.348 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:37.348 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:37.348 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:37.348 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:37.348 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:37.348 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:37.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.340+0000 7f1226ffd640 10 monclient: _finish_auth 0 2024-09-11T01:37:37.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.340+0000 7f1226ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:37:37.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.340+0000 7f1226ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:37.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.340+0000 7f1226ffd640 1 -- 172.21.15.12:0/1016662869 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1210032640 con 0x7f1228136630 2024-09-11T01:37:37.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.340+0000 7f1226ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:37.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.340+0000 7f1226ffd640 1 -- 172.21.15.12:0/1016662869 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1210032a10 con 0x7f1228136630 2024-09-11T01:37:37.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.340+0000 7f1226ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:37.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.340+0000 7f1226ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:37:37.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.340+0000 7f1226ffd640 10 monclient: dump: 2024-09-11T01:37:37.349 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:37.349 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:37.349 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:37.349 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:37.349 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:37.349 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:37.349 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:37.349 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:37.349 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:37.349 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:37.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.340+0000 7f122dbe8640 5 monclient: authenticate success, global_id 74269 2024-09-11T01:37:37.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.340+0000 7f122dbe8640 10 monclient: get_monmap_and_config success 2024-09-11T01:37:37.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.340+0000 7f122dbe8640 4 set_mon_vals no callback set 2024-09-11T01:37:37.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.340+0000 7f122dbe8640 10 monclient: shutdown 2024-09-11T01:37:37.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.340+0000 7f122dbe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:37.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.340+0000 7f122dbe8640 1 -- 172.21.15.12:0/1016662869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1228136630 msgr2=0x7f122813ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:37.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.340+0000 7f122dbe8640 1 --2- 172.21.15.12:0/1016662869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1228136630 0x7f122813ad40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f12100097a0 tx=0x7f1210029fd0 comp rx=0 tx=0).stop 2024-09-11T01:37:37.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.341+0000 7f122dbe8640 1 -- 172.21.15.12:0/1016662869 shutdown_connections 2024-09-11T01:37:37.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.341+0000 7f122dbe8640 1 --2- 172.21.15.12:0/1016662869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1228136630 0x7f122813ad40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:37.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.341+0000 7f122dbe8640 1 -- 172.21.15.12:0/1016662869 >> 172.21.15.12:0/1016662869 conn(0x7f12280a3de0 msgr2=0x7f1228068950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:37.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.341+0000 7f122dbe8640 1 -- 172.21.15.12:0/1016662869 shutdown_connections 2024-09-11T01:37:37.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.341+0000 7f122dbe8640 1 -- 172.21.15.12:0/1016662869 wait complete. 2024-09-11T01:37:37.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.341+0000 7f122dbe8640 10 monclient: build_initial_monmap 2024-09-11T01:37:37.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.341+0000 7f122dbe8640 10 monclient: monmap: 2024-09-11T01:37:37.351 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:37.351 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:37:37.351 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:37:37.351 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:37:37.351 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:37.351 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:37.351 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:37:37.351 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:37:37.351 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:37:37.351 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:37.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f122dbe8640 1 Processor -- start 2024-09-11T01:37:37.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f122dbe8640 1 -- start start 2024-09-11T01:37:37.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f122dbe8640 10 monclient: init 2024-09-11T01:37:37.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f122dbe8640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:37.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f122dbe8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:37:37.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f122dbe8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1228070130 0x7f122812f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:37.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f122dbe8640 10 monclient(hunting): picked mon.noname-b con 0x7f1228070130 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:37:37.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f122dbe8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f12280741b0 0x7f122812f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:37.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f122dbe8640 10 monclient(hunting): picked mon.noname-c con 0x7f12280741b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:37.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f122dbe8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1228136630 0x7f1228136ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:37.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f122dbe8640 10 monclient(hunting): picked mon.noname-a con 0x7f1228136630 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:37:37.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f122dbe8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:37.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f122dbe8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f122813e9b0 con 0x7f1228136630 2024-09-11T01:37:37.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f122dbe8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:37.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f122dbe8640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1228070510 con 0x7f1228070130 2024-09-11T01:37:37.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f122dbe8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:37.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f122dbe8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f122813eb30 con 0x7f12280741b0 2024-09-11T01:37:37.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f122dbe8640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:37.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f122dbe8640 10 monclient(hunting): authenticate will time out at 1263.785819s 2024-09-11T01:37:37.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f1227fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1228136630 0x7f1228136ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:37.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f121effd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f12280741b0 0x7f122812f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:37.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f1227fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1228136630 0x7f1228136ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53732/0 (socket says 172.21.15.12:53732) 2024-09-11T01:37:37.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f1227fff640 1 -- 172.21.15.12:0/2396586056 learned_addr learned my addr 172.21.15.12:0/2396586056 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:37.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f1227fff640 10 monclient(hunting): get_auth_request con 0x7f1228136630 auth_method 0 2024-09-11T01:37:37.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f1227fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:37.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f1227fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:37.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f1227fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:37.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f121effd640 10 monclient(hunting): get_auth_request con 0x7f12280741b0 auth_method 0 2024-09-11T01:37:37.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f121effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:37.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f121effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:37.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f121effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:37.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.342+0000 7f12277fe640 1 --2- 172.21.15.12:0/2396586056 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1228070130 0x7f122812f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:37.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.343+0000 7f12277fe640 10 monclient(hunting): get_auth_request con 0x7f1228070130 auth_method 0 2024-09-11T01:37:37.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.343+0000 7f12277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:37.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.343+0000 7f12277fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:37.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.343+0000 7f12277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:37.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.343+0000 7f1227fff640 1 -- 172.21.15.12:0/2396586056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1228136630 msgr2=0x7f1228136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:37:37.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.343+0000 7f1227fff640 1 -- 172.21.15.12:0/2396586056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1228136630 msgr2=0x7f1228136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:37:37.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.343+0000 7f1227fff640 1 --2- 172.21.15.12:0/2396586056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1228136630 0x7f1228136ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:37:37.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.343+0000 7f121effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:37.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.343+0000 7f121effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:37.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.343+0000 7f121effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:37.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.343+0000 7f1227fff640 1 --2- 172.21.15.12:0/2396586056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1228136630 0x7f1228136ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:37:37.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.343+0000 7f12277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:37.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.343+0000 7f12277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:37.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.343+0000 7f12277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:37.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.343+0000 7f121effd640 10 monclient(hunting): handle_auth_done global_id 84188 payload 824 2024-09-11T01:37:37.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.343+0000 7f121effd640 1 -- 172.21.15.12:0/2396586056 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1228070130 msgr2=0x7f122812f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:37.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.343+0000 7f121effd640 1 --2- 172.21.15.12:0/2396586056 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1228070130 0x7f122812f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:37.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.343+0000 7f121effd640 1 -- 172.21.15.12:0/2396586056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1228136630 msgr2=0x7f1228136ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:37:37.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.343+0000 7f121effd640 1 --2- 172.21.15.12:0/2396586056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1228136630 0x7f1228136ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:37.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.343+0000 7f121effd640 10 monclient: _finish_hunting 0 2024-09-11T01:37:37.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.343+0000 7f121effd640 1 monclient: found mon.noname-c 2024-09-11T01:37:37.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.343+0000 7f121effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:37.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.343+0000 7f121effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:37:37.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.343+0000 7f121effd640 1 -- 172.21.15.12:0/2396586056 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122813a230 con 0x7f12280741b0 2024-09-11T01:37:37.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.343+0000 7f12277fe640 1 --2- 172.21.15.12:0/2396586056 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1228070130 0x7f122812f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:37:37.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.345+0000 7f121effd640 1 -- 172.21.15.12:0/2396586056 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f12280741b0 msgr2=0x7f122812f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:37.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.345+0000 7f121effd640 1 --2- 172.21.15.12:0/2396586056 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f12280741b0 0x7f122812f710 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1210031000 tx=0x7f12100046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:37.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.345+0000 7f1224ff9640 1 -- 172.21.15.12:0/2396586056 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1210041070 con 0x7f12280741b0 2024-09-11T01:37:37.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.345+0000 7f1224ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:37.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.345+0000 7f1224ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:37:37.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.345+0000 7f1224ff9640 10 monclient: dump: 2024-09-11T01:37:37.358 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:37.358 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:37.358 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:37.358 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:37.358 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:37.358 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:37.358 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:37.358 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:37.358 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:37.359 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:37.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.345+0000 7f1224ff9640 10 monclient: _finish_auth 0 2024-09-11T01:37:37.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.345+0000 7f1224ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:37:37.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.345+0000 7f1224ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:37.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.345+0000 7f122dbe8640 5 monclient: authenticate success, global_id 84188 2024-09-11T01:37:37.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.345+0000 7f122dbe8640 10 monclient: _renew_subs 2024-09-11T01:37:37.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.345+0000 7f1224ff9640 1 -- 172.21.15.12:0/2396586056 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f121003cac0 con 0x7f12280741b0 2024-09-11T01:37:37.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.345+0000 7f122dbe8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:37.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.345+0000 7f122dbe8640 1 -- 172.21.15.12:0/2396586056 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1228074c20 con 0x7f12280741b0 2024-09-11T01:37:37.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.345+0000 7f1224ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:37.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.345+0000 7f1224ff9640 1 -- 172.21.15.12:0/2396586056 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1210037040 con 0x7f12280741b0 2024-09-11T01:37:37.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.345+0000 7f1224ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:37.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.345+0000 7f1224ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:37.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.345+0000 7f1224ff9640 10 monclient: dump: 2024-09-11T01:37:37.360 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:37.360 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:37.360 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:37.360 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:37.360 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:37.360 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:37.360 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:37.360 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:37.360 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:37.360 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:37.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.345+0000 7f12267fc640 4 set_mon_vals no callback set 2024-09-11T01:37:37.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.346+0000 7f122dbe8640 10 monclient: _renew_subs 2024-09-11T01:37:37.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.346+0000 7f122dbe8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:37.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.346+0000 7f122dbe8640 1 -- 172.21.15.12:0/2396586056 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f122812fc50 con 0x7f12280741b0 2024-09-11T01:37:37.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.347+0000 7f1224ff9640 1 -- 172.21.15.12:0/2396586056 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f121003d3f0 con 0x7f12280741b0 2024-09-11T01:37:37.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.347+0000 7f122dbe8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:37.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.347+0000 7f122dbe8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:37.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.347+0000 7f122dbe8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:37.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.347+0000 7f122dbe8640 1 -- 172.21.15.12:0/2396586056 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1228135fa0 con 0x7f12280741b0 2024-09-11T01:37:37.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.347+0000 7f1224ff9640 1 --2- 172.21.15.12:0/2396586056 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f120803e240 0x7f1208040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:37.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.347+0000 7f1224ff9640 1 -- 172.21.15.12:0/2396586056 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(283..283 src has 186..283) ==== 4799+0+0 (secure 0 0 0) 0x7f1210073760 con 0x7f12280741b0 2024-09-11T01:37:37.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.347+0000 7f12277fe640 1 --2- 172.21.15.12:0/2396586056 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f120803e240 0x7f1208040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:37.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.347+0000 7f12277fe640 10 monclient: get_auth_request con 0x7f120803e240 auth_method 0 2024-09-11T01:37:37.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.350+0000 7f12277fe640 1 --2- 172.21.15.12:0/2396586056 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f120803e240 0x7f1208040700 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1218006fd0 tx=0x7f1218009060 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:37.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.352+0000 7f1224ff9640 1 -- 172.21.15.12:0/2396586056 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f121004a020 con 0x7f12280741b0 2024-09-11T01:37:37.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.352+0000 7f1224ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:37.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.352+0000 7f1224ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:37:37.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.605+0000 7f122dbe8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:37:37.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.605+0000 7f122dbe8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:37:37.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.605+0000 7f122dbe8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:37.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.605+0000 7f122dbe8640 1 -- 172.21.15.12:0/2396586056 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f122806da60 con 0x7f12280741b0 2024-09-11T01:37:37.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.606+0000 7f1224ff9640 1 -- 172.21.15.12:0/2396586056 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f1210072150 con 0x7f12280741b0 2024-09-11T01:37:37.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.606+0000 7f1224ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:37:37.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.606+0000 7f1224ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:37:37.609 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":84,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:37:37.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.608+0000 7f122dbe8640 1 -- 172.21.15.12:0/2396586056 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f120803e240 msgr2=0x7f1208040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:37.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.608+0000 7f122dbe8640 1 --2- 172.21.15.12:0/2396586056 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f120803e240 0x7f1208040700 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1218006fd0 tx=0x7f1218009060 comp rx=0 tx=0).stop 2024-09-11T01:37:37.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.608+0000 7f122dbe8640 10 monclient: shutdown 2024-09-11T01:37:37.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.608+0000 7f122dbe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:37.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.608+0000 7f122dbe8640 1 -- 172.21.15.12:0/2396586056 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f12280741b0 msgr2=0x7f122812f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:37.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.608+0000 7f122dbe8640 1 --2- 172.21.15.12:0/2396586056 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f12280741b0 0x7f122812f710 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1210031000 tx=0x7f12100046c0 comp rx=0 tx=0).stop 2024-09-11T01:37:37.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.608+0000 7f122dbe8640 1 -- 172.21.15.12:0/2396586056 shutdown_connections 2024-09-11T01:37:37.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.608+0000 7f122dbe8640 1 --2- 172.21.15.12:0/2396586056 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f120803e240 0x7f1208040700 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:37.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.608+0000 7f122dbe8640 1 --2- 172.21.15.12:0/2396586056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1228136630 0x7f1228136ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:37.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.608+0000 7f122dbe8640 1 --2- 172.21.15.12:0/2396586056 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f12280741b0 0x7f122812f710 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:37.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.608+0000 7f122dbe8640 1 --2- 172.21.15.12:0/2396586056 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1228070130 0x7f122812f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:37.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.608+0000 7f122dbe8640 1 -- 172.21.15.12:0/2396586056 >> 172.21.15.12:0/2396586056 conn(0x7f12280a3de0 msgr2=0x7f1228130360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:37.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.608+0000 7f122dbe8640 1 -- 172.21.15.12:0/2396586056 shutdown_connections 2024-09-11T01:37:37.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:37.608+0000 7f122dbe8640 1 -- 172.21.15.12:0/2396586056 wait complete. 2024-09-11T01:37:38.242 INFO:tasks.rados.rados.0.smithi071.stdout:1265: done (0 left) 2024-09-11T01:37:38.242 INFO:tasks.rados.rados.0.smithi071.stdout:1266: read oid 28 snap -1 2024-09-11T01:37:38.242 INFO:tasks.rados.rados.0.smithi071.stdout:1266: expect (ObjNum 418 snap 117 seq_num 418) 2024-09-11T01:37:38.242 INFO:tasks.rados.rados.0.smithi071.stdout:1267: read oid 14 snap -1 2024-09-11T01:37:38.242 INFO:tasks.rados.rados.0.smithi071.stdout:1267: expect (ObjNum 312 snap 87 seq_num 312) 2024-09-11T01:37:38.243 INFO:tasks.rados.rados.0.smithi071.stdout:1268: delete oid 2 current snap is 131 2024-09-11T01:37:38.247 INFO:tasks.rados.rados.0.smithi071.stdout:1268: done (2 left) 2024-09-11T01:37:38.247 INFO:tasks.rados.rados.0.smithi071.stdout:1269: snap_create 2024-09-11T01:37:38.253 INFO:tasks.rados.rados.0.smithi071.stdout:1267: expect (ObjNum 312 snap 87 seq_num 312) 2024-09-11T01:37:38.263 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1912765 2024-09-11T01:37:38.277 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 55106 2024-09-11T01:37:38.344 INFO:tasks.rados.rados.0.smithi071.stdout:1266: expect (ObjNum 418 snap 117 seq_num 418) 2024-09-11T01:37:38.390 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:37:38.455 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2802080 2024-09-11T01:37:39.243 INFO:tasks.rados.rados.0.smithi071.stdout:1266: done (2 left) 2024-09-11T01:37:39.243 INFO:tasks.rados.rados.0.smithi071.stdout:1267: done (1 left) 2024-09-11T01:37:39.244 INFO:tasks.rados.rados.0.smithi071.stdout:1269: done (0 left) 2024-09-11T01:37:39.244 INFO:tasks.rados.rados.0.smithi071.stdout:1270: write (excl) oid 45 current snap is 132 2024-09-11T01:37:39.244 INFO:tasks.rados.rados.0.smithi071.stdout:1270: seq_num 486 ranges {568202=701253,1906484=261718} 2024-09-11T01:37:39.250 INFO:tasks.rados.rados.0.smithi071.stdout:1270: writing smithi07134777-45 from 568202 to 1269455 tid 1 2024-09-11T01:37:39.255 INFO:tasks.rados.rados.0.smithi071.stdout:1270: writing smithi07134777-45 from 1906484 to 2168202 tid 2 2024-09-11T01:37:39.255 INFO:tasks.rados.rados.0.smithi071.stdout:1271: delete oid 7 current snap is 132 2024-09-11T01:37:39.257 INFO:tasks.rados.rados.0.smithi071.stdout:1271: done (1 left) 2024-09-11T01:37:39.257 INFO:tasks.rados.rados.0.smithi071.stdout:1272: read oid 2 snap 131 2024-09-11T01:37:39.257 INFO:tasks.rados.rados.0.smithi071.stdout:1272: expect deleted 2024-09-11T01:37:39.257 INFO:tasks.rados.rados.0.smithi071.stdout:1273: snap_remove snap 130 2024-09-11T01:37:40.622 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:37:40.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.732+0000 7f5687332640 10 monclient: get_monmap_and_config 2024-09-11T01:37:40.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.732+0000 7f5687332640 10 monclient: build_initial_monmap 2024-09-11T01:37:40.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.732+0000 7f5687332640 10 monclient: monmap: 2024-09-11T01:37:40.735 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:40.735 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:40.735 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:37:40.734364+0000 2024-09-11T01:37:40.735 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:37:40.734364+0000 2024-09-11T01:37:40.735 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:40.735 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:40.735 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:37:40.735 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:37:40.736 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:37:40.736 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:37:40.736 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:37:40.736 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:40.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.733+0000 7f5687332640 1 Processor -- start 2024-09-11T01:37:40.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.733+0000 7f5687332640 1 -- start start 2024-09-11T01:37:40.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.733+0000 7f5687332640 10 monclient: init 2024-09-11T01:37:40.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.733+0000 7f5687332640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:40.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.733+0000 7f5687332640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-11T01:37:40.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.733+0000 7f5687332640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f56800a2e90 addr v1:172.21.15.12:6789/0 2024-09-11T01:37:40.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.733+0000 7f5687332640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56800a3270 0x7f56800a3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:40.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.733+0000 7f5687332640 10 monclient(hunting): picked mon.noname-b con 0x7f56800a3270 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:40.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f5687332640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f568013d490 0x7f568013f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:40.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f5687332640 10 monclient(hunting): picked mon.noname-a con 0x7f568013d490 addr v2:172.21.15.12:3300/0 2024-09-11T01:37:40.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f5687332640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56800a46c0 con 0x7f56800a2e90 2024-09-11T01:37:40.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f5687332640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:40.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f5687332640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f56800a4900 con 0x7f568013d490 2024-09-11T01:37:40.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f5687332640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:40.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f5687332640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f568006ee50 con 0x7f56800a3270 2024-09-11T01:37:40.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f5687332640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:40.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f5687332640 10 monclient(hunting): authenticate will time out at 1267.177040s 2024-09-11T01:37:40.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56848cd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56800a3270 0x7f56800a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:40.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56858cf640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f568013d490 0x7f568013f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:40.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56850ce640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f56800a2e90 0x7f568006dc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55888/0 (socket says 172.21.15.12:55888) 2024-09-11T01:37:40.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56850ce640 1 -- 172.21.15.12:0/1620740429 learned_addr learned my addr 172.21.15.12:0/1620740429 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:37:40.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56848cd640 10 monclient(hunting): get_auth_request con 0x7f56800a3270 auth_method 0 2024-09-11T01:37:40.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56848cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:40.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56848cd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:40.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56848cd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:40.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56858cf640 10 monclient(hunting): get_auth_request con 0x7f568013d490 auth_method 0 2024-09-11T01:37:40.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56858cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:40.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56858cf640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:40.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56858cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:40.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56848cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:40.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56848cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:40.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56848cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:40.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56858cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:40.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56858cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:40.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56858cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:40.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56858cf640 10 monclient(hunting): handle_auth_done global_id 94098 payload 274 2024-09-11T01:37:40.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56858cf640 1 -- 172.21.15.12:0/1620740429 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56800a3270 msgr2=0x7f56800a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:40.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56858cf640 1 --2- 172.21.15.12:0/1620740429 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56800a3270 0x7f56800a3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:40.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56858cf640 1 -- 172.21.15.12:0/1620740429 >> v1:172.21.15.12:6789/0 conn(0x7f56800a2e90 legacy=0x7f568006dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:40.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56858cf640 10 monclient: _finish_hunting 0 2024-09-11T01:37:40.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56858cf640 1 monclient: found mon.noname-a 2024-09-11T01:37:40.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56858cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:40.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56858cf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:37:40.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56858cf640 1 -- 172.21.15.12:0/1620740429 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f568006eb70 con 0x7f568013d490 2024-09-11T01:37:40.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.734+0000 7f56848cd640 1 --2- 172.21.15.12:0/1620740429 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56800a3270 0x7f56800a3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:37:40.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.737+0000 7f56858cf640 1 -- 172.21.15.12:0/1620740429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f568013d490 msgr2=0x7f568013f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:40.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.737+0000 7f56858cf640 1 --2- 172.21.15.12:0/1620740429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f568013d490 0x7f568013f880 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f56800a4a80 tx=0x7f567802a460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a1802c5745c8c11 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:40.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.737+0000 7f5673fff640 1 -- 172.21.15.12:0/1620740429 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5678007950 con 0x7f568013d490 2024-09-11T01:37:40.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.737+0000 7f5673fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:40.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.737+0000 7f5673fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:37:40.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.737+0000 7f5673fff640 10 monclient: dump: 2024-09-11T01:37:40.748 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:40.748 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:40.748 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:40.748 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:40.748 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:40.748 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:40.748 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:40.748 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:40.748 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:40.749 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:40.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.737+0000 7f5673fff640 10 monclient: _finish_auth 0 2024-09-11T01:37:40.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.737+0000 7f5673fff640 10 monclient: _check_auth_tickets 2024-09-11T01:37:40.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.737+0000 7f5673fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:40.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.737+0000 7f5687332640 5 monclient: authenticate success, global_id 94098 2024-09-11T01:37:40.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.737+0000 7f5673fff640 1 -- 172.21.15.12:0/1620740429 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5678007af0 con 0x7f568013d490 2024-09-11T01:37:40.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.737+0000 7f5687332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:37:40.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.737+0000 7f5673fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:40.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.737+0000 7f5687332640 10 monclient: get_monmap_and_config success 2024-09-11T01:37:40.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.737+0000 7f5687332640 4 set_mon_vals no callback set 2024-09-11T01:37:40.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.737+0000 7f5673fff640 1 -- 172.21.15.12:0/1620740429 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5678031400 con 0x7f568013d490 2024-09-11T01:37:40.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.737+0000 7f5673fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:40.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.737+0000 7f5673fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:40.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.737+0000 7f5673fff640 10 monclient: dump: 2024-09-11T01:37:40.750 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:40.750 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:40.750 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:40.750 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:40.750 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:40.750 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:40.750 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:40.750 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:40.750 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:40.750 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:40.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.738+0000 7f5687332640 10 monclient: shutdown 2024-09-11T01:37:40.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.738+0000 7f5687332640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:40.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.738+0000 7f5687332640 1 -- 172.21.15.12:0/1620740429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f568013d490 msgr2=0x7f568013f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:40.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.738+0000 7f5687332640 1 --2- 172.21.15.12:0/1620740429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f568013d490 0x7f568013f880 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f56800a4a80 tx=0x7f567802a460 comp rx=0 tx=0).stop 2024-09-11T01:37:40.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.738+0000 7f5687332640 1 -- 172.21.15.12:0/1620740429 shutdown_connections 2024-09-11T01:37:40.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.738+0000 7f5687332640 1 --2- 172.21.15.12:0/1620740429 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f568013d490 0x7f568013f880 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:40.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.738+0000 7f5687332640 1 --2- 172.21.15.12:0/1620740429 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56800a3270 0x7f56800a3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:40.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.738+0000 7f5687332640 1 -- 172.21.15.12:0/1620740429 >> 172.21.15.12:0/1620740429 conn(0x7f56800a8490 msgr2=0x7f56800a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:40.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.738+0000 7f5687332640 1 -- 172.21.15.12:0/1620740429 shutdown_connections 2024-09-11T01:37:40.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.738+0000 7f5687332640 1 -- 172.21.15.12:0/1620740429 wait complete. 2024-09-11T01:37:40.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.738+0000 7f5687332640 10 monclient: build_initial_monmap 2024-09-11T01:37:40.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.738+0000 7f5687332640 10 monclient: monmap: 2024-09-11T01:37:40.751 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:40.751 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:37:40.752 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:37:40.752 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:37:40.752 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:40.752 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:40.752 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:37:40.752 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:37:40.752 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:37:40.752 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:40.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f5687332640 1 Processor -- start 2024-09-11T01:37:40.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f5687332640 1 -- start start 2024-09-11T01:37:40.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f5687332640 10 monclient: init 2024-09-11T01:37:40.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f5687332640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:40.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f5687332640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:37:40.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f5687332640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56800a2e90 0x7f568013b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f5687332640 10 monclient(hunting): picked mon.noname-a con 0x7f56800a2e90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:37:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f5687332640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56800a3270 0x7f56801381f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f5687332640 10 monclient(hunting): picked mon.noname-b con 0x7f56800a3270 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:37:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f5687332640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f568013d490 0x7f5680139470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f5687332640 10 monclient(hunting): picked mon.noname-c con 0x7f568013d490 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f5687332640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f5687332640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f568006fd20 con 0x7f56800a2e90 2024-09-11T01:37:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f5687332640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f5687332640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f568006fba0 con 0x7f56800a3270 2024-09-11T01:37:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f5687332640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f5687332640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f568013fdc0 con 0x7f568013d490 2024-09-11T01:37:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f5687332640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f5687332640 10 monclient(hunting): authenticate will time out at 1267.182634s 2024-09-11T01:37:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f56850ce640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56800a2e90 0x7f568013b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f56858cf640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f568013d490 0x7f5680139470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f56850ce640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56800a2e90 0x7f568013b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53770/0 (socket says 172.21.15.12:53770) 2024-09-11T01:37:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f56850ce640 1 -- 172.21.15.12:0/1535411182 learned_addr learned my addr 172.21.15.12:0/1535411182 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f56850ce640 10 monclient(hunting): get_auth_request con 0x7f56800a2e90 auth_method 0 2024-09-11T01:37:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f56850ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f56850ce640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f56850ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f56858cf640 10 monclient(hunting): get_auth_request con 0x7f568013d490 auth_method 0 2024-09-11T01:37:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f56858cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f56858cf640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f56858cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f56850ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f56850ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f56850ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f56858cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f56858cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f56848cd640 1 --2- 172.21.15.12:0/1535411182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56800a3270 0x7f56801381f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f56858cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f56848cd640 10 monclient(hunting): get_auth_request con 0x7f56800a3270 auth_method 0 2024-09-11T01:37:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f56848cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f56848cd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.739+0000 7f56848cd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.740+0000 7f56848cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.740+0000 7f56848cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.740+0000 7f56848cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.740+0000 7f56850ce640 10 monclient(hunting): handle_auth_done global_id 94101 payload 824 2024-09-11T01:37:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.740+0000 7f56850ce640 1 -- 172.21.15.12:0/1535411182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f568013d490 msgr2=0x7f5680139470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.740+0000 7f56850ce640 1 --2- 172.21.15.12:0/1535411182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f568013d490 0x7f5680139470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.740+0000 7f56850ce640 1 -- 172.21.15.12:0/1535411182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56800a3270 msgr2=0x7f56801381f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.740+0000 7f56850ce640 1 --2- 172.21.15.12:0/1535411182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56800a3270 0x7f56801381f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.740+0000 7f56850ce640 10 monclient: _finish_hunting 0 2024-09-11T01:37:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.740+0000 7f56850ce640 1 monclient: found mon.noname-a 2024-09-11T01:37:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.740+0000 7f56850ce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.740+0000 7f56850ce640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:37:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.740+0000 7f56850ce640 1 -- 172.21.15.12:0/1535411182 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f568009f680 con 0x7f56800a2e90 2024-09-11T01:37:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.740+0000 7f56858cf640 1 --2- 172.21.15.12:0/1535411182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f568013d490 0x7f5680139470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:37:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.742+0000 7f56850ce640 1 -- 172.21.15.12:0/1535411182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56800a2e90 msgr2=0x7f568013b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.742+0000 7f56850ce640 1 --2- 172.21.15.12:0/1535411182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56800a2e90 0x7f568013b920 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f567c00c460 tx=0x7f567c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.742+0000 7f5671ffb640 1 -- 172.21.15.12:0/1535411182 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567c007070 con 0x7f56800a2e90 2024-09-11T01:37:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.742+0000 7f5671ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:40.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.742+0000 7f5671ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:37:40.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.742+0000 7f5671ffb640 10 monclient: dump: 2024-09-11T01:37:40.757 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:40.757 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:40.757 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:40.757 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:40.757 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:40.757 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:40.757 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:40.757 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:40.757 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:40.757 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:40.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.742+0000 7f5671ffb640 10 monclient: _finish_auth 0 2024-09-11T01:37:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.742+0000 7f5671ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:37:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.742+0000 7f5671ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.742+0000 7f5671ffb640 1 -- 172.21.15.12:0/1535411182 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f567c007230 con 0x7f56800a2e90 2024-09-11T01:37:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.742+0000 7f5671ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.742+0000 7f5671ffb640 1 -- 172.21.15.12:0/1535411182 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567c007620 con 0x7f56800a2e90 2024-09-11T01:37:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.742+0000 7f5671ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.742+0000 7f5671ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.742+0000 7f5671ffb640 10 monclient: dump: 2024-09-11T01:37:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:40.759 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.742+0000 7f5687332640 5 monclient: authenticate success, global_id 94101 2024-09-11T01:37:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.742+0000 7f5687332640 10 monclient: _renew_subs 2024-09-11T01:37:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.742+0000 7f5687332640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.742+0000 7f5687332640 1 -- 172.21.15.12:0/1535411182 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f568013d120 con 0x7f56800a2e90 2024-09-11T01:37:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.742+0000 7f56737fe640 4 set_mon_vals no callback set 2024-09-11T01:37:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.742+0000 7f5687332640 10 monclient: _renew_subs 2024-09-11T01:37:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.742+0000 7f5687332640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.742+0000 7f5687332640 1 -- 172.21.15.12:0/1535411182 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f568013adc0 con 0x7f56800a2e90 2024-09-11T01:37:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.743+0000 7f5671ffb640 1 -- 172.21.15.12:0/1535411182 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f567c01b400 con 0x7f56800a2e90 2024-09-11T01:37:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.743+0000 7f5687332640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.743+0000 7f5687332640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.743+0000 7f5687332640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.743+0000 7f5671ffb640 1 --2- 172.21.15.12:0/1535411182 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f566403e260 0x7f5664040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.743+0000 7f5687332640 1 -- 172.21.15.12:0/1535411182 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56800a3680 con 0x7f56800a2e90 2024-09-11T01:37:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.743+0000 7f5671ffb640 1 -- 172.21.15.12:0/1535411182 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(285..285 src has 186..285) ==== 4771+0+0 (secure 0 0 0) 0x7f567c052930 con 0x7f56800a2e90 2024-09-11T01:37:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.744+0000 7f56848cd640 1 --2- 172.21.15.12:0/1535411182 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f566403e260 0x7f5664040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.744+0000 7f56848cd640 10 monclient: get_auth_request con 0x7f566403e260 auth_method 0 2024-09-11T01:37:40.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.746+0000 7f56848cd640 1 --2- 172.21.15.12:0/1535411182 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f566403e260 0x7f5664040720 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5674009560 tx=0x7f5674009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:40.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.748+0000 7f5671ffb640 1 -- 172.21.15.12:0/1535411182 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f568009f680 con 0x7f56800a2e90 2024-09-11T01:37:40.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.748+0000 7f5671ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:40.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:40.748+0000 7f5671ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:37:41.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.004+0000 7f5687332640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:37:41.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.004+0000 7f5687332640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:37:41.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.004+0000 7f5687332640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:41.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.004+0000 7f5687332640 1 -- 172.21.15.12:0/1535411182 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f568013cf00 con 0x7f56800a2e90 2024-09-11T01:37:41.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.005+0000 7f5671ffb640 1 -- 172.21.15.12:0/1535411182 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f567c010cf0 con 0x7f56800a2e90 2024-09-11T01:37:41.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.005+0000 7f5671ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:37:41.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.005+0000 7f5671ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:37:41.008 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:37:41.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.007+0000 7f5687332640 1 -- 172.21.15.12:0/1535411182 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f566403e260 msgr2=0x7f5664040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.007+0000 7f5687332640 1 --2- 172.21.15.12:0/1535411182 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f566403e260 0x7f5664040720 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5674009560 tx=0x7f5674009190 comp rx=0 tx=0).stop 2024-09-11T01:37:41.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.007+0000 7f5687332640 10 monclient: shutdown 2024-09-11T01:37:41.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.007+0000 7f5687332640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:41.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.007+0000 7f5687332640 1 -- 172.21.15.12:0/1535411182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56800a2e90 msgr2=0x7f568013b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.007+0000 7f5687332640 1 --2- 172.21.15.12:0/1535411182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56800a2e90 0x7f568013b920 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f567c00c460 tx=0x7f567c00c920 comp rx=0 tx=0).stop 2024-09-11T01:37:41.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.007+0000 7f5687332640 1 -- 172.21.15.12:0/1535411182 shutdown_connections 2024-09-11T01:37:41.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.007+0000 7f5687332640 1 --2- 172.21.15.12:0/1535411182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f568013d490 0x7f5680139470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.007+0000 7f5687332640 1 --2- 172.21.15.12:0/1535411182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56800a3270 0x7f56801381f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.007+0000 7f5687332640 1 --2- 172.21.15.12:0/1535411182 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f566403e260 0x7f5664040720 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.007+0000 7f5687332640 1 --2- 172.21.15.12:0/1535411182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56800a2e90 0x7f568013b920 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.007+0000 7f5687332640 1 -- 172.21.15.12:0/1535411182 >> 172.21.15.12:0/1535411182 conn(0x7f56800a8490 msgr2=0x7f56800a6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:41.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.007+0000 7f5687332640 1 -- 172.21.15.12:0/1535411182 shutdown_connections 2024-09-11T01:37:41.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.007+0000 7f5687332640 1 -- 172.21.15.12:0/1535411182 wait complete. 2024-09-11T01:37:41.020 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:37:41.020 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:37:41.020 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:37:41.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.119+0000 7ffbc078a640 10 monclient: get_monmap_and_config 2024-09-11T01:37:41.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.119+0000 7ffbc078a640 10 monclient: build_initial_monmap 2024-09-11T01:37:41.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.120+0000 7ffbc078a640 10 monclient: monmap: 2024-09-11T01:37:41.122 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:41.122 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.122 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:37:41.121728+0000 2024-09-11T01:37:41.122 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:37:41.121728+0000 2024-09-11T01:37:41.123 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:41.123 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.123 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:37:41.123 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:37:41.123 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:37:41.123 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:37:41.123 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:37:41.123 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.120+0000 7ffbc078a640 1 Processor -- start 2024-09-11T01:37:41.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.120+0000 7ffbc078a640 1 -- start start 2024-09-11T01:37:41.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.120+0000 7ffbc078a640 10 monclient: init 2024-09-11T01:37:41.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbc078a640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:41.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbc078a640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-11T01:37:41.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbc078a640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ffbb80a2d40 0x7ffbb806cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbc078a640 10 monclient(hunting): picked mon.noname-c con 0x7ffbb80a2d40 addr v2:172.21.15.71:3300/0 2024-09-11T01:37:41.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbc078a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffbb806d090 0x7ffbb806d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbc078a640 10 monclient(hunting): picked mon.noname-b con 0x7ffbb806d090 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:41.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbc078a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffbb813c4d0 addr v1:172.21.15.71:6789/0 2024-09-11T01:37:41.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbc078a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbb809aed0 con 0x7ffbb813c4d0 2024-09-11T01:37:41.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbc078a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbc078a640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7ffbb806dd10 con 0x7ffbb80a2d40 2024-09-11T01:37:41.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbc078a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbc078a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ffbb809bcf0 con 0x7ffbb806d090 2024-09-11T01:37:41.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbc078a640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:41.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbc078a640 10 monclient(hunting): authenticate will time out at 1267.564260s 2024-09-11T01:37:41.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbbdd25640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffbb806d090 0x7ffbb806d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbbdd25640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffbb806d090 0x7ffbb806d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42310/0 (socket says 172.21.15.12:42310) 2024-09-11T01:37:41.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbbdd25640 1 -- 172.21.15.12:0/1008082328 learned_addr learned my addr 172.21.15.12:0/1008082328 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:41.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbbdd25640 10 monclient(hunting): get_auth_request con 0x7ffbb806d090 auth_method 0 2024-09-11T01:37:41.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbbdd25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:41.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbbdd25640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:41.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbbdd25640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:41.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbbdd25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:41.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbbdd25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:41.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbbdd25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:41.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbbe526640 1 --2- 172.21.15.12:0/1008082328 >> v2:172.21.15.71:3300/0 conn(0x7ffbb80a2d40 0x7ffbb806cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbbe526640 10 monclient(hunting): get_auth_request con 0x7ffbb80a2d40 auth_method 0 2024-09-11T01:37:41.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbbe526640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:41.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbbe526640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:41.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbbe526640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:41.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbbdd25640 10 monclient(hunting): handle_auth_done global_id 84197 payload 274 2024-09-11T01:37:41.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbbdd25640 1 -- 172.21.15.12:0/1008082328 >> v2:172.21.15.71:3300/0 conn(0x7ffbb80a2d40 msgr2=0x7ffbb806cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:41.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbbdd25640 1 --2- 172.21.15.12:0/1008082328 >> v2:172.21.15.71:3300/0 conn(0x7ffbb80a2d40 0x7ffbb806cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbbdd25640 1 -- 172.21.15.12:0/1008082328 >> v1:172.21.15.71:6789/0 conn(0x7ffbb813c4d0 legacy=0x7ffbb813e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:41.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbbdd25640 10 monclient: _finish_hunting 0 2024-09-11T01:37:41.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbbdd25640 1 monclient: found mon.noname-b 2024-09-11T01:37:41.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbbdd25640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:41.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbbdd25640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:37:41.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.121+0000 7ffbbdd25640 1 -- 172.21.15.12:0/1008082328 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbb80a3920 con 0x7ffbb806d090 2024-09-11T01:37:41.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.124+0000 7ffbbdd25640 1 -- 172.21.15.12:0/1008082328 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffbb806d090 msgr2=0x7ffbb806d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:41.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.124+0000 7ffbbdd25640 1 --2- 172.21.15.12:0/1008082328 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffbb806d090 0x7ffbb806d470 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4009870 tx=0x7ffbb402a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=623e3ee8c030d5b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:41.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.124+0000 7ffbbd524640 1 -- 172.21.15.12:0/1008082328 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4037070 con 0x7ffbb806d090 2024-09-11T01:37:41.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.124+0000 7ffbbd524640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:41.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.124+0000 7ffbbd524640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:37:41.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.124+0000 7ffbbd524640 10 monclient: dump: 2024-09-11T01:37:41.135 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:41.135 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.135 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.135 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.135 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:41.135 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.136 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:41.136 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:41.136 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:41.136 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.124+0000 7ffbbd524640 10 monclient: _finish_auth 0 2024-09-11T01:37:41.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.124+0000 7ffbbd524640 10 monclient: _check_auth_tickets 2024-09-11T01:37:41.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.124+0000 7ffbbd524640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:41.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.124+0000 7ffbc078a640 5 monclient: authenticate success, global_id 84197 2024-09-11T01:37:41.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.124+0000 7ffbc078a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:37:41.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.124+0000 7ffbbd524640 1 -- 172.21.15.12:0/1008082328 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbb4032440 con 0x7ffbb806d090 2024-09-11T01:37:41.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.124+0000 7ffbbd524640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:41.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.124+0000 7ffbc078a640 10 monclient: get_monmap_and_config success 2024-09-11T01:37:41.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.124+0000 7ffbc078a640 4 set_mon_vals no callback set 2024-09-11T01:37:41.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.124+0000 7ffbbd524640 1 -- 172.21.15.12:0/1008082328 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4032810 con 0x7ffbb806d090 2024-09-11T01:37:41.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.124+0000 7ffbbd524640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:41.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.124+0000 7ffbbd524640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:41.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.124+0000 7ffbbd524640 10 monclient: dump: 2024-09-11T01:37:41.137 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:41.137 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.137 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.137 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.137 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:41.137 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.137 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:41.137 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:41.137 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:41.137 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.125+0000 7ffbc078a640 10 monclient: shutdown 2024-09-11T01:37:41.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.125+0000 7ffbc078a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:41.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.125+0000 7ffbc078a640 1 -- 172.21.15.12:0/1008082328 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffbb806d090 msgr2=0x7ffbb806d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.125+0000 7ffbc078a640 1 --2- 172.21.15.12:0/1008082328 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffbb806d090 0x7ffbb806d470 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4009870 tx=0x7ffbb402a0a0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.125+0000 7ffbc078a640 1 -- 172.21.15.12:0/1008082328 shutdown_connections 2024-09-11T01:37:41.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.125+0000 7ffbc078a640 1 --2- 172.21.15.12:0/1008082328 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffbb806d090 0x7ffbb806d470 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.125+0000 7ffbc078a640 1 --2- 172.21.15.12:0/1008082328 >> v2:172.21.15.71:3300/0 conn(0x7ffbb80a2d40 0x7ffbb806cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.125+0000 7ffbc078a640 1 -- 172.21.15.12:0/1008082328 >> 172.21.15.12:0/1008082328 conn(0x7ffbb80a3e30 msgr2=0x7ffbb80a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:41.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.125+0000 7ffbc078a640 1 -- 172.21.15.12:0/1008082328 shutdown_connections 2024-09-11T01:37:41.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.125+0000 7ffbc078a640 1 -- 172.21.15.12:0/1008082328 wait complete. 2024-09-11T01:37:41.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.126+0000 7ffbc078a640 10 monclient: build_initial_monmap 2024-09-11T01:37:41.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.126+0000 7ffbc078a640 10 monclient: monmap: 2024-09-11T01:37:41.138 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:41.138 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:37:41.138 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:37:41.138 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:37:41.139 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:41.139 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.139 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:37:41.139 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:37:41.139 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:37:41.139 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.126+0000 7ffbc078a640 1 Processor -- start 2024-09-11T01:37:41.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.126+0000 7ffbc078a640 1 -- start start 2024-09-11T01:37:41.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.126+0000 7ffbc078a640 10 monclient: init 2024-09-11T01:37:41.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.126+0000 7ffbc078a640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:41.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.126+0000 7ffbc078a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:37:41.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.126+0000 7ffbc078a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffbb806d090 0x7ffbb8099c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.126+0000 7ffbc078a640 10 monclient(hunting): picked mon.noname-c con 0x7ffbb806d090 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:41.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.126+0000 7ffbc078a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffbb80a2d40 0x7ffbb80913c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.126+0000 7ffbc078a640 10 monclient(hunting): picked mon.noname-b con 0x7ffbb80a2d40 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:37:41.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.126+0000 7ffbc078a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbb813c4d0 0x7ffbb8091900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.126+0000 7ffbc078a640 10 monclient(hunting): picked mon.noname-a con 0x7ffbb813c4d0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:37:41.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.126+0000 7ffbc078a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.126+0000 7ffbc078a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb809ba10 con 0x7ffbb813c4d0 2024-09-11T01:37:41.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.126+0000 7ffbc078a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.126+0000 7ffbc078a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb80a3120 con 0x7ffbb80a2d40 2024-09-11T01:37:41.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.126+0000 7ffbc078a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.126+0000 7ffbc078a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ffbb809b3a0 con 0x7ffbb806d090 2024-09-11T01:37:41.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.126+0000 7ffbc078a640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:41.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.126+0000 7ffbc078a640 10 monclient(hunting): authenticate will time out at 1267.569907s 2024-09-11T01:37:41.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.126+0000 7ffbbe526640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffbb806d090 0x7ffbb8099c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbe526640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffbb806d090 0x7ffbb8099c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42324/0 (socket says 172.21.15.12:42324) 2024-09-11T01:37:41.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbe526640 1 -- 172.21.15.12:0/3155797145 learned_addr learned my addr 172.21.15.12:0/3155797145 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:41.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbe526640 10 monclient(hunting): get_auth_request con 0x7ffbb806d090 auth_method 0 2024-09-11T01:37:41.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbe526640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:41.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbe526640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:41.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbed27640 1 --2- 172.21.15.12:0/3155797145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbb813c4d0 0x7ffbb8091900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbe526640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:41.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbed27640 10 monclient(hunting): get_auth_request con 0x7ffbb813c4d0 auth_method 0 2024-09-11T01:37:41.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbed27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:41.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbed27640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:41.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbed27640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:41.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbe526640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:41.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbe526640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:41.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbe526640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:41.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbed27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:41.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbed27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:41.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbed27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:41.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbe526640 10 monclient(hunting): handle_auth_done global_id 84200 payload 824 2024-09-11T01:37:41.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbdd25640 1 --2- 172.21.15.12:0/3155797145 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffbb80a2d40 0x7ffbb80913c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbe526640 1 -- 172.21.15.12:0/3155797145 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffbb80a2d40 msgr2=0x7ffbb80913c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbe526640 1 --2- 172.21.15.12:0/3155797145 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffbb80a2d40 0x7ffbb80913c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbe526640 1 -- 172.21.15.12:0/3155797145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbb813c4d0 msgr2=0x7ffbb8091900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbe526640 1 --2- 172.21.15.12:0/3155797145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbb813c4d0 0x7ffbb8091900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbe526640 10 monclient: _finish_hunting 0 2024-09-11T01:37:41.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbe526640 1 monclient: found mon.noname-c 2024-09-11T01:37:41.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbe526640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:41.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbe526640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:37:41.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.127+0000 7ffbbe526640 1 -- 172.21.15.12:0/3155797145 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbb809a170 con 0x7ffbb806d090 2024-09-11T01:37:41.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.129+0000 7ffbbe526640 1 -- 172.21.15.12:0/3155797145 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffbb806d090 msgr2=0x7ffbb8099c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:41.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.129+0000 7ffbbe526640 1 --2- 172.21.15.12:0/3155797145 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffbb806d090 0x7ffbb8099c30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ffbac00d800 tx=0x7ffbac00dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:41.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.129+0000 7ffbaaffd640 1 -- 172.21.15.12:0/3155797145 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbac00b8d0 con 0x7ffbb806d090 2024-09-11T01:37:41.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.129+0000 7ffbaaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:41.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.129+0000 7ffbaaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:37:41.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.129+0000 7ffbaaffd640 10 monclient: dump: 2024-09-11T01:37:41.143 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:41.143 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.143 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.143 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.143 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:41.143 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.144 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:41.144 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:41.144 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:41.144 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.129+0000 7ffbaaffd640 10 monclient: _finish_auth 0 2024-09-11T01:37:41.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.129+0000 7ffbaaffd640 10 monclient: _check_auth_tickets 2024-09-11T01:37:41.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.129+0000 7ffbaaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:41.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.129+0000 7ffbaaffd640 1 -- 172.21.15.12:0/3155797145 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbac019070 con 0x7ffbb806d090 2024-09-11T01:37:41.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.129+0000 7ffbc078a640 5 monclient: authenticate success, global_id 84200 2024-09-11T01:37:41.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.129+0000 7ffbc078a640 10 monclient: _renew_subs 2024-09-11T01:37:41.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.129+0000 7ffbc078a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:41.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.129+0000 7ffbc078a640 1 -- 172.21.15.12:0/3155797145 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbb813f8e0 con 0x7ffbb806d090 2024-09-11T01:37:41.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.129+0000 7ffbaaffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:41.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.129+0000 7ffbaaffd640 1 -- 172.21.15.12:0/3155797145 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbac00bc90 con 0x7ffbb806d090 2024-09-11T01:37:41.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.129+0000 7ffbaaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:41.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.129+0000 7ffbaaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:41.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.129+0000 7ffbaaffd640 10 monclient: dump: 2024-09-11T01:37:41.145 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:41.145 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.145 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.145 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.145 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:41.145 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.145 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:41.145 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:41.145 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:41.146 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.129+0000 7ffbc078a640 10 monclient: _renew_subs 2024-09-11T01:37:41.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.129+0000 7ffbbcd23640 4 set_mon_vals no callback set 2024-09-11T01:37:41.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.129+0000 7ffbc078a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:41.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.129+0000 7ffbc078a640 1 -- 172.21.15.12:0/3155797145 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbb8074a40 con 0x7ffbb806d090 2024-09-11T01:37:41.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.130+0000 7ffbc078a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:41.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.130+0000 7ffbc078a640 1 --2- 172.21.15.12:0/3155797145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbb8091e40 0x7ffbb809f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.130+0000 7ffbc078a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.130+0000 7ffbc078a640 1 -- 172.21.15.12:0/3155797145 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb8092220 con 0x7ffbb8091e40 2024-09-11T01:37:41.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.130+0000 7ffbbdd25640 1 --2- 172.21.15.12:0/3155797145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbb8091e40 0x7ffbb809f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.130+0000 7ffbbdd25640 10 monclient: get_auth_request con 0x7ffbb8091e40 auth_method 0 2024-09-11T01:37:41.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.130+0000 7ffbbdd25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:41.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.130+0000 7ffbbdd25640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:41.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.130+0000 7ffbbdd25640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:41.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.130+0000 7ffbbdd25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:41.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.130+0000 7ffbbdd25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:41.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.130+0000 7ffbbdd25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:41.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.130+0000 7ffbaaffd640 1 -- 172.21.15.12:0/3155797145 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ffbac013070 con 0x7ffbb806d090 2024-09-11T01:37:41.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.130+0000 7ffbbdd25640 10 monclient(hunting): handle_auth_done global_id 94107 payload 824 2024-09-11T01:37:41.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.130+0000 7ffbbdd25640 1 -- 172.21.15.12:0/3155797145 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffbb806d470 con 0x7ffbb8091e40 2024-09-11T01:37:41.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.131+0000 7ffbaaffd640 1 --2- 172.21.15.12:0/3155797145 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffb9c03e260 0x7ffb9c040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.131+0000 7ffbaaffd640 1 -- 172.21.15.12:0/3155797145 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(285..285 src has 186..285) ==== 4771+0+0 (secure 0 0 0) 0x7ffbac052e50 con 0x7ffbb806d090 2024-09-11T01:37:41.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.131+0000 7ffbbed27640 1 --2- 172.21.15.12:0/3155797145 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffb9c03e260 0x7ffb9c040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.131+0000 7ffbbed27640 10 monclient: get_auth_request con 0x7ffb9c03e260 auth_method 0 2024-09-11T01:37:41.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.133+0000 7ffbbdd25640 1 -- 172.21.15.12:0/3155797145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbb8091e40 msgr2=0x7ffbb809f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:41.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.133+0000 7ffbbdd25640 1 --2- 172.21.15.12:0/3155797145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbb8091e40 0x7ffbb809f1b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4034000 tx=0x7ffbb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:41.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.133+0000 7ffbaaffd640 1 -- 172.21.15.12:0/3155797145 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4041070 con 0x7ffbb8091e40 2024-09-11T01:37:41.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.133+0000 7ffbaaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:41.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.133+0000 7ffbaaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.133+0000 7ffbaaffd640 10 monclient: dump: 2024-09-11T01:37:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:41.150 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.133+0000 7ffbaaffd640 1 -- 172.21.15.12:0/3155797145 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ffbb4009490 con 0x7ffbb8091e40 2024-09-11T01:37:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.133+0000 7ffbaaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.133+0000 7ffbaaffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:37:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.133+0000 7ffbaaffd640 1 -- 172.21.15.12:0/3155797145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbb8091e40 msgr2=0x7ffbb809f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.133+0000 7ffbaaffd640 1 --2- 172.21.15.12:0/3155797145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbb8091e40 0x7ffbb809f1b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4034000 tx=0x7ffbb40046c0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.133+0000 7ffbaaffd640 1 -- 172.21.15.12:0/3155797145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbb8091e40 msgr2=0x7ffbb809f1b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:37:41.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.133+0000 7ffbaaffd640 1 --2- 172.21.15.12:0/3155797145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbb8091e40 0x7ffbb809f1b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.133+0000 7ffbbed27640 1 --2- 172.21.15.12:0/3155797145 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffb9c03e260 0x7ffb9c040720 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8093000 tx=0x7ffbb0003d10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:41.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.139+0000 7ffbc078a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:37:41.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.139+0000 7ffbc078a640 1 --2- 172.21.15.12:0/3155797145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbb809f6f0 0x7ffbb809fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.139+0000 7ffbc078a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.139+0000 7ffbc078a640 1 -- 172.21.15.12:0/3155797145 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb806cbc0 con 0x7ffbb809f6f0 2024-09-11T01:37:41.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.140+0000 7ffbbdd25640 1 --2- 172.21.15.12:0/3155797145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbb809f6f0 0x7ffbb809fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.140+0000 7ffbbdd25640 10 monclient: get_auth_request con 0x7ffbb809f6f0 auth_method 0 2024-09-11T01:37:41.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.140+0000 7ffbbdd25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:41.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.140+0000 7ffbbdd25640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:41.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.140+0000 7ffbbdd25640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:41.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.140+0000 7ffbbdd25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:41.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.140+0000 7ffbbdd25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:41.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.140+0000 7ffbbdd25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:41.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.140+0000 7ffbbdd25640 10 monclient(hunting): handle_auth_done global_id 94110 payload 824 2024-09-11T01:37:41.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.140+0000 7ffbbdd25640 1 -- 172.21.15.12:0/3155797145 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ffbb80941d0 con 0x7ffbb809f6f0 2024-09-11T01:37:41.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.142+0000 7ffbbdd25640 1 -- 172.21.15.12:0/3155797145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbb809f6f0 msgr2=0x7ffbb809fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:41.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.142+0000 7ffbbdd25640 1 --2- 172.21.15.12:0/3155797145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbb809f6f0 0x7ffbb809fad0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ffbb402a0e0 tx=0x7ffbb4009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:41.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.142+0000 7ffbaaffd640 1 -- 172.21.15.12:0/3155797145 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4037040 con 0x7ffbb809f6f0 2024-09-11T01:37:41.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.142+0000 7ffbaaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:41.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.142+0000 7ffbaaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:41.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.142+0000 7ffbaaffd640 10 monclient: dump: 2024-09-11T01:37:41.152 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:41.152 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.152 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.152 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.152 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:41.152 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.152 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:41.153 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:41.153 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:41.153 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.142+0000 7ffbaaffd640 1 -- 172.21.15.12:0/3155797145 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7ffbb403c670 con 0x7ffbb809f6f0 2024-09-11T01:37:41.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.142+0000 7ffbaaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:37:41.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.142+0000 7ffbaaffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:37:41.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.142+0000 7ffbaaffd640 1 -- 172.21.15.12:0/3155797145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbb809f6f0 msgr2=0x7ffbb809fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.142+0000 7ffbaaffd640 1 --2- 172.21.15.12:0/3155797145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbb809f6f0 0x7ffbb809fad0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ffbb402a0e0 tx=0x7ffbb4009240 comp rx=0 tx=0).stop 2024-09-11T01:37:41.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.142+0000 7ffbaaffd640 1 -- 172.21.15.12:0/3155797145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbb809f6f0 msgr2=0x7ffbb809fad0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:37:41.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.142+0000 7ffbaaffd640 1 --2- 172.21.15.12:0/3155797145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffbb809f6f0 0x7ffbb809fad0 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ffbb402a0e0 tx=0x7ffbb4009240 comp rx=0 tx=0).stop 2024-09-11T01:37:41.153 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:37:41.154 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:37:41.154 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:37:41.154 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:37:41.154 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 88, 2024-09-11T01:37:41.154 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 7238, 2024-09-11T01:37:41.154 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:37:41.154 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:37:41.154 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:37:41.154 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:37:41.154 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:41.154 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:37:41.154 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:37:41.154 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:37:41.154 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:37:41.154 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:41.154 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:41.155 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:41.155 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:41.155 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:41.155 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:41.155 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:41.155 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:41.155 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:41.155 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:41.155 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:41.155 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:41.155 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:37:41.155 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:37:41.155 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:41.156 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:41.156 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:41.156 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:41.156 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:41.156 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:41.156 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:41.156 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:41.156 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:41.156 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:41.156 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:41.156 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:41.156 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.156 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:37:41.157 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:37:41.157 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:37:41.157 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:37:41.157 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:37:41.157 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:37:41.157 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:37:41.157 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:37:41.157 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:37:41.157 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:37:41.157 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:37:41.157 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:37:41.157 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:37:41.157 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:37:41.157 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:37:41.157 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:37:41.158 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:37:41.158 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:41.158 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:41.158 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:41.158 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:41.158 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:41.158 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:41.158 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:41.158 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:41.158 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:41.158 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:41.158 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:41.158 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:41.158 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:37:41.158 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.159 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:37:41.159 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.159 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:37:41.159 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:37:41.159 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:41.159 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:41.159 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.159 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:41.159 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:37:41.159 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:41.159 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.159 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.159 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:41.159 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:37:41.160 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:41.160 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:41.160 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:41.160 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.160 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:37:41.160 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:37:41.160 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:41.160 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:41.160 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:41.160 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.160 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.160 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:37:41.160 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:37:41.160 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:41.160 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:41.161 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.161 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:41.161 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:37:41.161 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:41.161 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.161 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.161 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:41.161 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:37:41.161 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:41.161 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:41.161 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:41.161 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.161 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:37:41.162 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:37:41.162 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:41.162 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:41.162 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:41.162 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.162 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.162 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:37:41.162 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:37:41.162 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:41.162 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:41.162 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.162 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:41.162 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:37:41.163 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:41.163 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.163 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.163 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:41.163 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:37:41.163 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:41.163 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:41.163 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:41.163 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.163 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:37:41.163 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:37:41.163 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:41.163 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:41.164 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:41.164 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:41.164 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:41.164 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.164 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:37:41.164 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:37:41.164 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.164 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:41.164 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:41.164 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:37:41.164 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:41.164 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:41.164 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:37:41.164 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.165 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:41.165 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:41.165 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:37:41.165 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:41.165 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:41.165 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.165 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:37:41.165 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:37:41.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.143+0000 7ffbc078a640 1 -- 172.21.15.12:0/3155797145 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffb9c03e260 msgr2=0x7ffb9c040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.143+0000 7ffbc078a640 1 --2- 172.21.15.12:0/3155797145 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffb9c03e260 0x7ffb9c040720 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8093000 tx=0x7ffbb0003d10 comp rx=0 tx=0).stop 2024-09-11T01:37:41.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.143+0000 7ffbc078a640 10 monclient: shutdown 2024-09-11T01:37:41.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.143+0000 7ffbc078a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:41.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.143+0000 7ffbc078a640 1 -- 172.21.15.12:0/3155797145 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffbb806d090 msgr2=0x7ffbb8099c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.143+0000 7ffbc078a640 1 --2- 172.21.15.12:0/3155797145 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffbb806d090 0x7ffbb8099c30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ffbac00d800 tx=0x7ffbac00dcc0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.143+0000 7ffbbed27640 1 -- 172.21.15.12:0/3155797145 reap_dead start 2024-09-11T01:37:41.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.143+0000 7ffbbed27640 1 -- 172.21.15.12:0/3155797145 reap_dead start 2024-09-11T01:37:41.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.143+0000 7ffbc078a640 1 -- 172.21.15.12:0/3155797145 shutdown_connections 2024-09-11T01:37:41.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.143+0000 7ffbc078a640 1 -- 172.21.15.12:0/3155797145 >> 172.21.15.12:0/3155797145 conn(0x7ffbb80a3e30 msgr2=0x7ffbb813e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:41.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.143+0000 7ffbc078a640 1 -- 172.21.15.12:0/3155797145 shutdown_connections 2024-09-11T01:37:41.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.143+0000 7ffbc078a640 1 -- 172.21.15.12:0/3155797145 wait complete. 2024-09-11T01:37:41.167 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:37:41.168 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:37:41.168 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:37:41.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.266+0000 7f42c06b8640 10 monclient: get_monmap_and_config 2024-09-11T01:37:41.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.266+0000 7f42c06b8640 10 monclient: build_initial_monmap 2024-09-11T01:37:41.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.267+0000 7f42c06b8640 10 monclient: monmap: 2024-09-11T01:37:41.269 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:41.269 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.270 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:37:41.268721+0000 2024-09-11T01:37:41.270 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:37:41.268721+0000 2024-09-11T01:37:41.270 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:41.270 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.270 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:37:41.270 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:37:41.270 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:37:41.270 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:37:41.270 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:37:41.270 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.267+0000 7f42c06b8640 1 Processor -- start 2024-09-11T01:37:41.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.267+0000 7f42c06b8640 1 -- start start 2024-09-11T01:37:41.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.267+0000 7f42c06b8640 10 monclient: init 2024-09-11T01:37:41.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.268+0000 7f42c06b8640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:41.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.268+0000 7f42c06b8640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-11T01:37:41.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.268+0000 7f42c06b8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f42b8134ad0 addr v1:172.21.15.12:6789/0 2024-09-11T01:37:41.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.268+0000 7f42c06b8640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f42b80a2200 0x7f42b813a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.268+0000 7f42c06b8640 10 monclient(hunting): picked mon.noname-c con 0x7f42b80a2200 addr v2:172.21.15.71:3300/0 2024-09-11T01:37:41.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.268+0000 7f42c06b8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f42b81355c0 addr v1:172.21.15.71:6789/0 2024-09-11T01:37:41.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.268+0000 7f42c06b8640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42b809ed30 con 0x7f42b8134ad0 2024-09-11T01:37:41.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.268+0000 7f42c06b8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42b809f010 con 0x7f42b81355c0 2024-09-11T01:37:41.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.268+0000 7f42c06b8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.268+0000 7f42c06b8640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f42b809f5d0 con 0x7f42b80a2200 2024-09-11T01:37:41.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.268+0000 7f42c06b8640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:41.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.268+0000 7f42c06b8640 10 monclient(hunting): authenticate will time out at 1267.711269s 2024-09-11T01:37:41.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.268+0000 7f42be454640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f42b8134ad0 0x7f42b8134eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55900/0 (socket says 172.21.15.12:55900) 2024-09-11T01:37:41.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.268+0000 7f42be454640 1 -- 172.21.15.12:0/447913820 learned_addr learned my addr 172.21.15.12:0/447913820 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:37:41.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.268+0000 7f42bdc53640 1 --2- 172.21.15.12:0/447913820 >> v2:172.21.15.71:3300/0 conn(0x7f42b80a2200 0x7f42b813a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.268+0000 7f42bdc53640 10 monclient(hunting): get_auth_request con 0x7f42b80a2200 auth_method 0 2024-09-11T01:37:41.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.268+0000 7f42bdc53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:41.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.268+0000 7f42bdc53640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:41.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.268+0000 7f42bdc53640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:41.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.268+0000 7f42bdc53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:41.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.268+0000 7f42bdc53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:41.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.268+0000 7f42bdc53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:41.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.270+0000 7f42be454640 1 -- 172.21.15.12:0/447913820 >> v1:172.21.15.12:6789/0 conn(0x7f42b8134ad0 legacy=0x7f42b8134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:41.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.270+0000 7f42bd452640 1 -- 172.21.15.12:0/447913820 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2822456412 0 0) 0x7f42b809ed30 con 0x7f42b8134ad0 2024-09-11T01:37:41.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.270+0000 7f42bd452640 10 monclient(hunting): _negotiate 2024-09-11T01:37:41.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.270+0000 7f42bd452640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:41.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.270+0000 7f42bd452640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.270+0000 7f42bd452640 10 monclient(hunting): my global_id is 94113 2024-09-11T01:37:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.270+0000 7f42bd452640 1 -- 172.21.15.12:0/447913820 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42a0003520 con 0x7f42b8134ad0 2024-09-11T01:37:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.270+0000 7f42bd452640 1 -- 172.21.15.12:0/447913820 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 844814177 0 0) 0x7f42a0003520 con 0x7f42b8134ad0 2024-09-11T01:37:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.270+0000 7f42bd452640 1 -- 172.21.15.12:0/447913820 >> v2:172.21.15.71:3300/0 conn(0x7f42b80a2200 msgr2=0x7f42b813a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.270+0000 7f42bd452640 1 --2- 172.21.15.12:0/447913820 >> v2:172.21.15.71:3300/0 conn(0x7f42b80a2200 0x7f42b813a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.270+0000 7f42bd452640 1 -- 172.21.15.12:0/447913820 >> v1:172.21.15.71:6789/0 conn(0x7f42b81355c0 legacy=0x7f42b813c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.270+0000 7f42bd452640 10 monclient: _finish_hunting 0 2024-09-11T01:37:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.270+0000 7f42bd452640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:37:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.270+0000 7f42bd452640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.270+0000 7f42bd452640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:37:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.270+0000 7f42bd452640 1 -- 172.21.15.12:0/447913820 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42b809f2f0 con 0x7f42b8134ad0 2024-09-11T01:37:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.270+0000 7f42bd452640 10 monclient: _finish_auth 0 2024-09-11T01:37:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.270+0000 7f42bd452640 10 monclient: _check_auth_tickets 2024-09-11T01:37:41.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.270+0000 7f42bd452640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:41.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.270+0000 7f42bd452640 1 -- 172.21.15.12:0/447913820 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f42a8001a60 con 0x7f42b8134ad0 2024-09-11T01:37:41.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.270+0000 7f42c06b8640 5 monclient: authenticate success, global_id 94113 2024-09-11T01:37:41.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.270+0000 7f42c06b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:37:41.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.270+0000 7f42bd452640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:41.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bd452640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:37:41.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bd452640 10 monclient: dump: 2024-09-11T01:37:41.286 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:41.286 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.286 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.286 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.286 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:41.286 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:41.287 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bd452640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:37:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bd452640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bd452640 1 -- 172.21.15.12:0/447913820 >> v1:172.21.15.12:6789/0 conn(0x7f42b8134ad0 legacy=0x7f42b8134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bd452640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:37:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bd452640 1 --2- 172.21.15.12:0/447913820 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42a0003a30 0x7f42a0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bd452640 10 monclient(hunting): picked mon.a con 0x7f42a0003a30 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:37:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bd452640 1 --2- 172.21.15.12:0/447913820 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f42a0004800 0x7f42a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bd452640 10 monclient(hunting): picked mon.b con 0x7f42a0004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:37:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bd452640 1 --2- 172.21.15.12:0/447913820 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42a00052e0 0x7f42a000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bd452640 10 monclient(hunting): picked mon.c con 0x7f42a00052e0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bd452640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bd452640 1 -- 172.21.15.12:0/447913820 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f42b809f5d0 con 0x7f42a0003a30 2024-09-11T01:37:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bd452640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bd452640 1 -- 172.21.15.12:0/447913820 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f42a000e440 con 0x7f42a0004800 2024-09-11T01:37:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bd452640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bd452640 1 -- 172.21.15.12:0/447913820 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f42a000e640 con 0x7f42a00052e0 2024-09-11T01:37:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bd452640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bd452640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:37:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42c06b8640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:37:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42be454640 1 --2- 172.21.15.12:0/447913820 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42a0003a30 0x7f42a0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42be454640 10 monclient(hunting): get_auth_request con 0x7f42a0003a30 auth_method 0 2024-09-11T01:37:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42be454640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42be454640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42be454640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:37:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bec55640 1 --2- 172.21.15.12:0/447913820 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42a00052e0 0x7f42a000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42be454640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42be454640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42be454640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:37:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bec55640 10 monclient(hunting): get_auth_request con 0x7f42a00052e0 auth_method 0 2024-09-11T01:37:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bec55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bec55640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bec55640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:37:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bdc53640 1 --2- 172.21.15.12:0/447913820 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f42a0004800 0x7f42a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bdc53640 10 monclient(hunting): get_auth_request con 0x7f42a0004800 auth_method 0 2024-09-11T01:37:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bdc53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bdc53640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bdc53640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:37:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bec55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bec55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bec55640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:37:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42be454640 10 monclient(hunting): handle_auth_done global_id 94113 payload 293 2024-09-11T01:37:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42be454640 1 -- 172.21.15.12:0/447913820 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42a00052e0 msgr2=0x7f42a000dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42be454640 1 --2- 172.21.15.12:0/447913820 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42a00052e0 0x7f42a000dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42be454640 1 -- 172.21.15.12:0/447913820 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f42a0004800 msgr2=0x7f42a0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42be454640 1 --2- 172.21.15.12:0/447913820 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f42a0004800 0x7f42a0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42be454640 10 monclient: _finish_hunting 0 2024-09-11T01:37:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42be454640 1 monclient: found mon.a 2024-09-11T01:37:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42be454640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42be454640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bec55640 1 -- 172.21.15.12:0/447913820 reap_dead start 2024-09-11T01:37:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42be454640 1 -- 172.21.15.12:0/447913820 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42a0003520 con 0x7f42a0003a30 2024-09-11T01:37:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42be454640 10 monclient: _finish_auth 0 2024-09-11T01:37:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42be454640 10 monclient: _check_auth_tickets 2024-09-11T01:37:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42be454640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.271+0000 7f42bdc53640 1 --2- 172.21.15.12:0/447913820 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f42a0004800 0x7f42a0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:37:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.274+0000 7f42be454640 1 -- 172.21.15.12:0/447913820 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42a0003a30 msgr2=0x7f42a0003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.274+0000 7f42be454640 1 --2- 172.21.15.12:0/447913820 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42a0003a30 0x7f42a0003e50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f42a8001eb0 tx=0x7f42a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cde1319f3f8018a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.274+0000 7f42bd452640 1 -- 172.21.15.12:0/447913820 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a8035070 con 0x7f42a0003a30 2024-09-11T01:37:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.274+0000 7f42bd452640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.274+0000 7f42bd452640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.274+0000 7f42bd452640 10 monclient: dump: 2024-09-11T01:37:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:41.292 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.274+0000 7f42c06b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:37:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.274+0000 7f42bd452640 1 -- 172.21.15.12:0/447913820 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42a8005af0 con 0x7f42a0003a30 2024-09-11T01:37:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.274+0000 7f42bd452640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.274+0000 7f42c06b8640 10 monclient: get_monmap_and_config success 2024-09-11T01:37:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.274+0000 7f42c06b8640 4 set_mon_vals no callback set 2024-09-11T01:37:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.274+0000 7f42bd452640 1 -- 172.21.15.12:0/447913820 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a800c140 con 0x7f42a0003a30 2024-09-11T01:37:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.274+0000 7f42bd452640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.274+0000 7f42bd452640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.274+0000 7f42bd452640 10 monclient: dump: 2024-09-11T01:37:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:41.294 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.274+0000 7f42c06b8640 10 monclient: shutdown 2024-09-11T01:37:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.274+0000 7f42c06b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.274+0000 7f42c06b8640 1 -- 172.21.15.12:0/447913820 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42a0003a30 msgr2=0x7f42a0003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.274+0000 7f42c06b8640 1 --2- 172.21.15.12:0/447913820 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42a0003a30 0x7f42a0003e50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f42a8001eb0 tx=0x7f42a0005830 comp rx=0 tx=0).stop 2024-09-11T01:37:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.275+0000 7f42c06b8640 1 -- 172.21.15.12:0/447913820 shutdown_connections 2024-09-11T01:37:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.275+0000 7f42c06b8640 1 --2- 172.21.15.12:0/447913820 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42a0003a30 0x7f42a0003e50 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.275+0000 7f42c06b8640 1 -- 172.21.15.12:0/447913820 >> 172.21.15.12:0/447913820 conn(0x7f42b80a2f70 msgr2=0x7f42b80a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.275+0000 7f42c06b8640 1 -- 172.21.15.12:0/447913820 shutdown_connections 2024-09-11T01:37:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.275+0000 7f42c06b8640 1 -- 172.21.15.12:0/447913820 wait complete. 2024-09-11T01:37:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.275+0000 7f42c06b8640 10 monclient: build_initial_monmap 2024-09-11T01:37:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.275+0000 7f42c06b8640 10 monclient: monmap: 2024-09-11T01:37:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:37:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:37:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:37:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:37:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:37:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:37:41.295 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.275+0000 7f42c06b8640 1 Processor -- start 2024-09-11T01:37:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42c06b8640 1 -- start start 2024-09-11T01:37:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42c06b8640 10 monclient: init 2024-09-11T01:37:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42c06b8640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42c06b8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:37:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42c06b8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42a0003a30 0x7f42b8132670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42c06b8640 10 monclient(hunting): picked mon.noname-c con 0x7f42a0003a30 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42c06b8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f42b811df00 0x7f42b811e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42c06b8640 10 monclient(hunting): picked mon.noname-b con 0x7f42b811df00 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:37:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42c06b8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42b811e8c0 0x7f42b811ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42c06b8640 10 monclient(hunting): picked mon.noname-a con 0x7f42b811e8c0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:37:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42c06b8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42c06b8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f42b80a0150 con 0x7f42b811e8c0 2024-09-11T01:37:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42c06b8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42c06b8640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f42b809ffd0 con 0x7f42b811df00 2024-09-11T01:37:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42c06b8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42c06b8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f42b8134eb0 con 0x7f42a0003a30 2024-09-11T01:37:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42c06b8640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42c06b8640 10 monclient(hunting): authenticate will time out at 1267.719478s 2024-09-11T01:37:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42be454640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42a0003a30 0x7f42b8132670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42be454640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42a0003a30 0x7f42b8132670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42342/0 (socket says 172.21.15.12:42342) 2024-09-11T01:37:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42be454640 1 -- 172.21.15.12:0/21243243 learned_addr learned my addr 172.21.15.12:0/21243243 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42be454640 10 monclient(hunting): get_auth_request con 0x7f42a0003a30 auth_method 0 2024-09-11T01:37:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42be454640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42be454640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42be454640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42bec55640 1 --2- 172.21.15.12:0/21243243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42b811e8c0 0x7f42b811ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42bec55640 10 monclient(hunting): get_auth_request con 0x7f42b811e8c0 auth_method 0 2024-09-11T01:37:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42bec55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42bec55640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42bec55640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42bec55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42bec55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42bec55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42be454640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42be454640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42be454640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42bdc53640 1 --2- 172.21.15.12:0/21243243 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f42b811df00 0x7f42b811e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42bec55640 10 monclient(hunting): handle_auth_done global_id 94116 payload 824 2024-09-11T01:37:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42bec55640 1 -- 172.21.15.12:0/21243243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42a0003a30 msgr2=0x7f42b8132670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42bec55640 1 --2- 172.21.15.12:0/21243243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42a0003a30 0x7f42b8132670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42bec55640 1 -- 172.21.15.12:0/21243243 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f42b811df00 msgr2=0x7f42b811e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42bec55640 1 --2- 172.21.15.12:0/21243243 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f42b811df00 0x7f42b811e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42bec55640 10 monclient: _finish_hunting 0 2024-09-11T01:37:41.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42bec55640 1 monclient: found mon.noname-a 2024-09-11T01:37:41.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42bec55640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:41.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42bec55640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:37:41.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42bec55640 1 -- 172.21.15.12:0/21243243 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42b809d480 con 0x7f42b811e8c0 2024-09-11T01:37:41.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42be454640 1 --2- 172.21.15.12:0/21243243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42a0003a30 0x7f42b8132670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:37:41.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42bdc53640 10 monclient: get_auth_request con 0x7f42b811df00 auth_method 0 2024-09-11T01:37:41.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.276+0000 7f42bdc53640 1 --2- 172.21.15.12:0/21243243 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f42b811df00 0x7f42b811e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:37:41.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.279+0000 7f42bec55640 1 -- 172.21.15.12:0/21243243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42b811e8c0 msgr2=0x7f42b811ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:41.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.279+0000 7f42bec55640 1 --2- 172.21.15.12:0/21243243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42b811e8c0 0x7f42b811ed50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f42b809d0a0 tx=0x7f42b000f9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:41.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.279+0000 7f42b6ffd640 1 -- 172.21.15.12:0/21243243 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b0018030 con 0x7f42b811e8c0 2024-09-11T01:37:41.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.279+0000 7f42b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:41.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.279+0000 7f42b6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:37:41.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.279+0000 7f42b6ffd640 10 monclient: dump: 2024-09-11T01:37:41.300 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:41.300 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.300 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.300 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.300 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:41.300 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.300 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:41.300 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:41.300 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:41.300 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.279+0000 7f42b6ffd640 10 monclient: _finish_auth 0 2024-09-11T01:37:41.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.279+0000 7f42b6ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:37:41.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.279+0000 7f42b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:41.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.279+0000 7f42b6ffd640 1 -- 172.21.15.12:0/21243243 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42b0014890 con 0x7f42b811e8c0 2024-09-11T01:37:41.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.279+0000 7f42b6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:41.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.279+0000 7f42b6ffd640 1 -- 172.21.15.12:0/21243243 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b0014a80 con 0x7f42b811e8c0 2024-09-11T01:37:41.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.279+0000 7f42b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:41.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.279+0000 7f42b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:41.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.279+0000 7f42b6ffd640 10 monclient: dump: 2024-09-11T01:37:41.301 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:41.301 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.301 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.302 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.302 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:41.302 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.302 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:41.302 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:41.302 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:41.302 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.279+0000 7f42c06b8640 5 monclient: authenticate success, global_id 94116 2024-09-11T01:37:41.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.279+0000 7f42c06b8640 10 monclient: _renew_subs 2024-09-11T01:37:41.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.279+0000 7f42c06b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:41.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.279+0000 7f42c06b8640 1 -- 172.21.15.12:0/21243243 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42b8120eb0 con 0x7f42b811e8c0 2024-09-11T01:37:41.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.279+0000 7f42bcc51640 4 set_mon_vals no callback set 2024-09-11T01:37:41.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.279+0000 7f42c06b8640 10 monclient: _renew_subs 2024-09-11T01:37:41.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.279+0000 7f42c06b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:41.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.279+0000 7f42c06b8640 1 -- 172.21.15.12:0/21243243 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42b8121480 con 0x7f42b811e8c0 2024-09-11T01:37:41.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.280+0000 7f42c06b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:41.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.280+0000 7f42c06b8640 1 --2- 172.21.15.12:0/21243243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42b80716a0 0x7f42b8071a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.280+0000 7f42c06b8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.280+0000 7f42c06b8640 1 -- 172.21.15.12:0/21243243 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f42b8135240 con 0x7f42b80716a0 2024-09-11T01:37:41.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.280+0000 7f42be454640 1 --2- 172.21.15.12:0/21243243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42b80716a0 0x7f42b8071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.280+0000 7f42b6ffd640 1 -- 172.21.15.12:0/21243243 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f42b0014c60 con 0x7f42b811e8c0 2024-09-11T01:37:41.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.280+0000 7f42be454640 10 monclient: get_auth_request con 0x7f42b80716a0 auth_method 0 2024-09-11T01:37:41.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.280+0000 7f42be454640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:41.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.280+0000 7f42be454640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:41.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.280+0000 7f42be454640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:41.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.280+0000 7f42be454640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:41.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.280+0000 7f42be454640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:41.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.280+0000 7f42be454640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:41.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.280+0000 7f42be454640 10 monclient(hunting): handle_auth_done global_id 84206 payload 824 2024-09-11T01:37:41.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.280+0000 7f42b6ffd640 1 --2- 172.21.15.12:0/21243243 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f42a403e210 0x7f42a40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.280+0000 7f42be454640 1 -- 172.21.15.12:0/21243243 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f42b811f9c0 con 0x7f42b80716a0 2024-09-11T01:37:41.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.280+0000 7f42b6ffd640 1 -- 172.21.15.12:0/21243243 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(285..285 src has 186..285) ==== 4771+0+0 (secure 0 0 0) 0x7f42b0052970 con 0x7f42b811e8c0 2024-09-11T01:37:41.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.281+0000 7f42bdc53640 1 --2- 172.21.15.12:0/21243243 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f42a403e210 0x7f42a40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.281+0000 7f42bdc53640 10 monclient: get_auth_request con 0x7f42a403e210 auth_method 0 2024-09-11T01:37:41.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.282+0000 7f42be454640 1 -- 172.21.15.12:0/21243243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42b80716a0 msgr2=0x7f42b8071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:41.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.282+0000 7f42be454640 1 --2- 172.21.15.12:0/21243243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42b80716a0 0x7f42b8071a80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f42a8000c00 tx=0x7f42b809ef00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:41.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.283+0000 7f42b6ffd640 1 -- 172.21.15.12:0/21243243 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a8035030 con 0x7f42b80716a0 2024-09-11T01:37:41.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.283+0000 7f42b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:41.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.283+0000 7f42b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:41.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.283+0000 7f42b6ffd640 10 monclient: dump: 2024-09-11T01:37:41.306 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:41.306 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.307 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.307 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.307 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:41.307 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.307 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:41.307 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:41.307 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:41.307 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.283+0000 7f42b6ffd640 1 -- 172.21.15.12:0/21243243 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f42a8001320 con 0x7f42b80716a0 2024-09-11T01:37:41.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.283+0000 7f42b6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:41.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.283+0000 7f42b6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:37:41.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.283+0000 7f42b6ffd640 1 -- 172.21.15.12:0/21243243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42b80716a0 msgr2=0x7f42b8071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.283+0000 7f42b6ffd640 1 --2- 172.21.15.12:0/21243243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42b80716a0 0x7f42b8071a80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f42a8000c00 tx=0x7f42b809ef00 comp rx=0 tx=0).stop 2024-09-11T01:37:41.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.283+0000 7f42b6ffd640 1 -- 172.21.15.12:0/21243243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42b80716a0 msgr2=0x7f42b8071a80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:37:41.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.283+0000 7f42b6ffd640 1 --2- 172.21.15.12:0/21243243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42b80716a0 0x7f42b8071a80 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f42a8000c00 tx=0x7f42b809ef00 comp rx=0 tx=0).stop 2024-09-11T01:37:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.283+0000 7f42bdc53640 1 --2- 172.21.15.12:0/21243243 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f42a403e210 0x7f42a40406d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f42b8132410 tx=0x7f42ac00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.289+0000 7f42c06b8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:37:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.289+0000 7f42c06b8640 1 --2- 172.21.15.12:0/21243243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42b8071fc0 0x7f42b81d2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.289+0000 7f42c06b8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.289+0000 7f42c06b8640 1 -- 172.21.15.12:0/21243243 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f42b80723d0 con 0x7f42b8071fc0 2024-09-11T01:37:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.289+0000 7f42be454640 1 --2- 172.21.15.12:0/21243243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42b8071fc0 0x7f42b81d2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.289+0000 7f42be454640 10 monclient: get_auth_request con 0x7f42b8071fc0 auth_method 0 2024-09-11T01:37:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.289+0000 7f42be454640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.289+0000 7f42be454640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.289+0000 7f42be454640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.289+0000 7f42be454640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.289+0000 7f42be454640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.289+0000 7f42be454640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.290+0000 7f42be454640 10 monclient(hunting): handle_auth_done global_id 84209 payload 824 2024-09-11T01:37:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.290+0000 7f42be454640 1 -- 172.21.15.12:0/21243243 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f42b8072580 con 0x7f42b8071fc0 2024-09-11T01:37:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.292+0000 7f42be454640 1 -- 172.21.15.12:0/21243243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42b8071fc0 msgr2=0x7f42b81d2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.292+0000 7f42be454640 1 --2- 172.21.15.12:0/21243243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42b8071fc0 0x7f42b81d2e20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f42a8031000 tx=0x7f42a802e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.292+0000 7f42b6ffd640 1 -- 172.21.15.12:0/21243243 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a803e890 con 0x7f42b8071fc0 2024-09-11T01:37:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.292+0000 7f42b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.292+0000 7f42b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.292+0000 7f42b6ffd640 10 monclient: dump: 2024-09-11T01:37:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:41.310 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:41.310 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:41.310 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.292+0000 7f42b6ffd640 1 -- 172.21.15.12:0/21243243 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4707 (secure 0 0 0) 0x7f42a803e890 con 0x7f42b8071fc0 2024-09-11T01:37:41.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.292+0000 7f42b6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:37:41.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.292+0000 7f42b6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:37:41.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.292+0000 7f42b6ffd640 1 -- 172.21.15.12:0/21243243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42b8071fc0 msgr2=0x7f42b81d2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.292+0000 7f42b6ffd640 1 --2- 172.21.15.12:0/21243243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42b8071fc0 0x7f42b81d2e20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f42a8031000 tx=0x7f42a802e000 comp rx=0 tx=0).stop 2024-09-11T01:37:41.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.292+0000 7f42b6ffd640 1 -- 172.21.15.12:0/21243243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42b8071fc0 msgr2=0x7f42b81d2e20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:37:41.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.292+0000 7f42b6ffd640 1 --2- 172.21.15.12:0/21243243 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42b8071fc0 0x7f42b81d2e20 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f42a8031000 tx=0x7f42a802e000 comp rx=0 tx=0).stop 2024-09-11T01:37:41.310 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:37:41.310 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:37:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:37:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:37:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 88, 2024-09-11T01:37:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 55316, 2024-09-11T01:37:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:37:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:37:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:37:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:37:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:37:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:37:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:37:41.311 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:37:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:41.312 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:37:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:37:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.313 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:37:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:37:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:37:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:37:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:37:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:37:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:37:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:37:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:37:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:37:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:37:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:37:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:37:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:37:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:37:41.314 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:37:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:37:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:37:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:37:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:37:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:37:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:37:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.316 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:37:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:37:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:37:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:37:41.317 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:37:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:37:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:37:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:41.318 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:37:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:37:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:37:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:37:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:37:41.319 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:37:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:37:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:37:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:41.320 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:37:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:37:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:37:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:37:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:41.321 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:37:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:37:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:37:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.322 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:37:41.322 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:37:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.292+0000 7f42c06b8640 1 -- 172.21.15.12:0/21243243 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f42a403e210 msgr2=0x7f42a40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.292+0000 7f42c06b8640 1 --2- 172.21.15.12:0/21243243 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f42a403e210 0x7f42a40406d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f42b8132410 tx=0x7f42ac00b040 comp rx=0 tx=0).stop 2024-09-11T01:37:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.292+0000 7f42c06b8640 10 monclient: shutdown 2024-09-11T01:37:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.292+0000 7f42c06b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.292+0000 7f42c06b8640 1 -- 172.21.15.12:0/21243243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42b811e8c0 msgr2=0x7f42b811ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.292+0000 7f42bec55640 1 -- 172.21.15.12:0/21243243 reap_dead start 2024-09-11T01:37:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.292+0000 7f42c06b8640 1 --2- 172.21.15.12:0/21243243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42b811e8c0 0x7f42b811ed50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f42b809d0a0 tx=0x7f42b000f9d0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.293+0000 7f42c06b8640 1 -- 172.21.15.12:0/21243243 shutdown_connections 2024-09-11T01:37:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.293+0000 7f42c06b8640 1 --2- 172.21.15.12:0/21243243 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42b811e8c0 0x7f42b811ed50 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.293+0000 7f42c06b8640 1 -- 172.21.15.12:0/21243243 >> 172.21.15.12:0/21243243 conn(0x7f42b80a2f70 msgr2=0x7f42b80a16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.293+0000 7f42c06b8640 1 -- 172.21.15.12:0/21243243 shutdown_connections 2024-09-11T01:37:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.293+0000 7f42c06b8640 1 -- 172.21.15.12:0/21243243 wait complete. 2024-09-11T01:37:41.324 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:37:41.324 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:37:41.325 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:37:41.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.423+0000 7ff773a72640 10 monclient: get_monmap_and_config 2024-09-11T01:37:41.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.423+0000 7ff773a72640 10 monclient: build_initial_monmap 2024-09-11T01:37:41.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.423+0000 7ff773a72640 10 monclient: monmap: 2024-09-11T01:37:41.426 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:41.426 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.426 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:37:41.425591+0000 2024-09-11T01:37:41.426 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:37:41.425591+0000 2024-09-11T01:37:41.426 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:37:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:37:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:37:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:37:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:37:41.427 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.424+0000 7ff773a72640 1 Processor -- start 2024-09-11T01:37:41.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.424+0000 7ff773a72640 1 -- start start 2024-09-11T01:37:41.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.424+0000 7ff773a72640 10 monclient: init 2024-09-11T01:37:41.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff773a72640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:41.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff773a72640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-11T01:37:41.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff773a72640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff76c134ad0 0x7ff76c134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff773a72640 10 monclient(hunting): picked mon.noname-b con 0x7ff76c134ad0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:41.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff773a72640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff76c0a2200 0x7ff76c1353f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff773a72640 10 monclient(hunting): picked mon.noname-a con 0x7ff76c0a2200 addr v2:172.21.15.12:3300/0 2024-09-11T01:37:41.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff773a72640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff76c133a50 0x7ff76c133f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff773a72640 10 monclient(hunting): picked mon.noname-c con 0x7ff76c133a50 addr v2:172.21.15.71:3300/0 2024-09-11T01:37:41.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff773a72640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff773a72640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff76c09f5d0 con 0x7ff76c0a2200 2024-09-11T01:37:41.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff773a72640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff773a72640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7ff76c135930 con 0x7ff76c133a50 2024-09-11T01:37:41.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff773a72640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff773a72640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff76c09ed30 con 0x7ff76c134ad0 2024-09-11T01:37:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff773a72640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff773a72640 10 monclient(hunting): authenticate will time out at 1267.868177s 2024-09-11T01:37:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77180e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff76c134ad0 0x7ff76c134eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77100d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff76c0a2200 0x7ff76c1353f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77180e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff76c134ad0 0x7ff76c134eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42372/0 (socket says 172.21.15.12:42372) 2024-09-11T01:37:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77180e640 1 -- 172.21.15.12:0/1041265596 learned_addr learned my addr 172.21.15.12:0/1041265596 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77100d640 10 monclient(hunting): get_auth_request con 0x7ff76c0a2200 auth_method 0 2024-09-11T01:37:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77100d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77100d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77100d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77180e640 10 monclient(hunting): get_auth_request con 0x7ff76c134ad0 auth_method 0 2024-09-11T01:37:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77180e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77180e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:41.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77180e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:41.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77100d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:41.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77100d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:41.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77100d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:41.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77180e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:41.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77180e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:41.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77180e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:41.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77200f640 1 -- 172.21.15.12:0/1041265596 >> v2:172.21.15.71:3300/0 conn(0x7ff76c133a50 msgr2=0x7ff76c133f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:37:41.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77200f640 1 -- 172.21.15.12:0/1041265596 >> v2:172.21.15.71:3300/0 conn(0x7ff76c133a50 msgr2=0x7ff76c133f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:37:41.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77200f640 1 --2- 172.21.15.12:0/1041265596 >> v2:172.21.15.71:3300/0 conn(0x7ff76c133a50 0x7ff76c133f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2024-09-11T01:37:41.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77100d640 10 monclient(hunting): handle_auth_done global_id 94119 payload 274 2024-09-11T01:37:41.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77100d640 1 -- 172.21.15.12:0/1041265596 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff76c134ad0 msgr2=0x7ff76c134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:41.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77200f640 1 --2- 172.21.15.12:0/1041265596 >> v2:172.21.15.71:3300/0 conn(0x7ff76c133a50 0x7ff76c133f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:37:41.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77100d640 1 --2- 172.21.15.12:0/1041265596 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff76c134ad0 0x7ff76c134eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77100d640 1 -- 172.21.15.12:0/1041265596 >> v2:172.21.15.71:3300/0 conn(0x7ff76c133a50 msgr2=0x7ff76c133f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:37:41.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77100d640 1 --2- 172.21.15.12:0/1041265596 >> v2:172.21.15.71:3300/0 conn(0x7ff76c133a50 0x7ff76c133f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77100d640 10 monclient: _finish_hunting 0 2024-09-11T01:37:41.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77100d640 1 monclient: found mon.noname-a 2024-09-11T01:37:41.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77100d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:41.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77100d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:37:41.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77100d640 1 -- 172.21.15.12:0/1041265596 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff76c09f010 con 0x7ff76c0a2200 2024-09-11T01:37:41.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.425+0000 7ff77180e640 1 --2- 172.21.15.12:0/1041265596 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff76c134ad0 0x7ff76c134eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:37:41.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.428+0000 7ff77100d640 1 -- 172.21.15.12:0/1041265596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76c0a2200 msgr2=0x7ff76c1353f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:41.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.428+0000 7ff77100d640 1 --2- 172.21.15.12:0/1041265596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76c0a2200 0x7ff76c1353f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff76c0a2630 tx=0x7ff760029fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b025b78c37d67457 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:41.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.428+0000 7ff77080c640 1 -- 172.21.15.12:0/1041265596 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff760037070 con 0x7ff76c0a2200 2024-09-11T01:37:41.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.428+0000 7ff77080c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:41.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.428+0000 7ff77080c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:37:41.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.428+0000 7ff77080c640 10 monclient: dump: 2024-09-11T01:37:41.439 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:41.440 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.440 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.440 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.440 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:41.440 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.440 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:41.440 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:41.440 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:41.440 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.428+0000 7ff77080c640 10 monclient: _finish_auth 0 2024-09-11T01:37:41.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.428+0000 7ff77080c640 10 monclient: _check_auth_tickets 2024-09-11T01:37:41.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.428+0000 7ff77080c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:41.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.428+0000 7ff77080c640 1 -- 172.21.15.12:0/1041265596 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff760032440 con 0x7ff76c0a2200 2024-09-11T01:37:41.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.428+0000 7ff773a72640 5 monclient: authenticate success, global_id 94119 2024-09-11T01:37:41.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.428+0000 7ff773a72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:37:41.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.428+0000 7ff77080c640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:41.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.428+0000 7ff773a72640 10 monclient: get_monmap_and_config success 2024-09-11T01:37:41.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.428+0000 7ff77080c640 1 -- 172.21.15.12:0/1041265596 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff760032810 con 0x7ff76c0a2200 2024-09-11T01:37:41.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.428+0000 7ff77080c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:41.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.428+0000 7ff773a72640 4 set_mon_vals no callback set 2024-09-11T01:37:41.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.428+0000 7ff77080c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:41.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.428+0000 7ff77080c640 10 monclient: dump: 2024-09-11T01:37:41.441 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:41.441 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.441 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.441 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:41.442 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.429+0000 7ff773a72640 10 monclient: shutdown 2024-09-11T01:37:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.429+0000 7ff773a72640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.429+0000 7ff773a72640 1 -- 172.21.15.12:0/1041265596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76c0a2200 msgr2=0x7ff76c1353f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.429+0000 7ff773a72640 1 --2- 172.21.15.12:0/1041265596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76c0a2200 0x7ff76c1353f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff76c0a2630 tx=0x7ff760029fe0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.429+0000 7ff773a72640 1 -- 172.21.15.12:0/1041265596 shutdown_connections 2024-09-11T01:37:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.429+0000 7ff773a72640 1 --2- 172.21.15.12:0/1041265596 >> v2:172.21.15.71:3300/0 conn(0x7ff76c133a50 0x7ff76c133f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.429+0000 7ff773a72640 1 --2- 172.21.15.12:0/1041265596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76c0a2200 0x7ff76c1353f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.429+0000 7ff773a72640 1 --2- 172.21.15.12:0/1041265596 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff76c134ad0 0x7ff76c134eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.429+0000 7ff773a72640 1 -- 172.21.15.12:0/1041265596 >> 172.21.15.12:0/1041265596 conn(0x7ff76c0a2f70 msgr2=0x7ff76c0a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.429+0000 7ff773a72640 1 -- 172.21.15.12:0/1041265596 shutdown_connections 2024-09-11T01:37:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.429+0000 7ff773a72640 1 -- 172.21.15.12:0/1041265596 wait complete. 2024-09-11T01:37:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.429+0000 7ff773a72640 10 monclient: build_initial_monmap 2024-09-11T01:37:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.429+0000 7ff773a72640 10 monclient: monmap: 2024-09-11T01:37:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:37:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:37:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:37:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:37:41.444 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:37:41.444 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:37:41.444 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff773a72640 1 Processor -- start 2024-09-11T01:37:41.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff773a72640 1 -- start start 2024-09-11T01:37:41.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff773a72640 10 monclient: init 2024-09-11T01:37:41.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff773a72640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:41.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff773a72640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:37:41.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff773a72640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76c0a2200 0x7ff76c13b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff773a72640 10 monclient(hunting): picked mon.noname-a con 0x7ff76c0a2200 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:37:41.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff773a72640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff76c133a50 0x7ff76c1cb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff773a72640 10 monclient(hunting): picked mon.noname-c con 0x7ff76c133a50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:41.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff773a72640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff76c134ad0 0x7ff76c138b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff773a72640 10 monclient(hunting): picked mon.noname-b con 0x7ff76c134ad0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:37:41.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff773a72640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff773a72640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff76c0a0150 con 0x7ff76c0a2200 2024-09-11T01:37:41.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff773a72640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff773a72640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff76c09ffd0 con 0x7ff76c134ad0 2024-09-11T01:37:41.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff773a72640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff773a72640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff76c134440 con 0x7ff76c133a50 2024-09-11T01:37:41.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff773a72640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:41.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff773a72640 10 monclient(hunting): authenticate will time out at 1267.873637s 2024-09-11T01:37:41.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff77180e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76c0a2200 0x7ff76c13b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff77100d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff76c133a50 0x7ff76c1cb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff77180e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76c0a2200 0x7ff76c13b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53822/0 (socket says 172.21.15.12:53822) 2024-09-11T01:37:41.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff77100d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff76c133a50 0x7ff76c1cb580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42382/0 (socket says 172.21.15.12:42382) 2024-09-11T01:37:41.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff77180e640 1 -- 172.21.15.12:0/1240620184 learned_addr learned my addr 172.21.15.12:0/1240620184 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff77180e640 10 monclient(hunting): get_auth_request con 0x7ff76c0a2200 auth_method 0 2024-09-11T01:37:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff77180e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff77180e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff77180e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff77100d640 10 monclient(hunting): get_auth_request con 0x7ff76c133a50 auth_method 0 2024-09-11T01:37:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff77100d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff77100d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff77100d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff77180e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff77180e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff77180e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff77100d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff77100d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff77100d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff77200f640 1 --2- 172.21.15.12:0/1240620184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff76c134ad0 0x7ff76c138b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff77200f640 10 monclient(hunting): get_auth_request con 0x7ff76c134ad0 auth_method 0 2024-09-11T01:37:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff77200f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff77200f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.430+0000 7ff77200f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.431+0000 7ff77180e640 10 monclient(hunting): handle_auth_done global_id 94122 payload 824 2024-09-11T01:37:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.431+0000 7ff77180e640 1 -- 172.21.15.12:0/1240620184 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff76c133a50 msgr2=0x7ff76c1cb580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.431+0000 7ff77180e640 1 --2- 172.21.15.12:0/1240620184 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff76c133a50 0x7ff76c1cb580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.431+0000 7ff77180e640 1 -- 172.21.15.12:0/1240620184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff76c134ad0 msgr2=0x7ff76c138b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.431+0000 7ff77180e640 1 --2- 172.21.15.12:0/1240620184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff76c134ad0 0x7ff76c138b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.431+0000 7ff77180e640 10 monclient: _finish_hunting 0 2024-09-11T01:37:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.431+0000 7ff77180e640 1 monclient: found mon.noname-a 2024-09-11T01:37:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.431+0000 7ff77180e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:41.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.431+0000 7ff77180e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:37:41.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.431+0000 7ff77180e640 1 -- 172.21.15.12:0/1240620184 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff76c11d240 con 0x7ff76c0a2200 2024-09-11T01:37:41.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.431+0000 7ff77100d640 1 --2- 172.21.15.12:0/1240620184 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff76c133a50 0x7ff76c1cb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:37:41.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.431+0000 7ff77200f640 1 --2- 172.21.15.12:0/1240620184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff76c134ad0 0x7ff76c138b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:37:41.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.433+0000 7ff77180e640 1 -- 172.21.15.12:0/1240620184 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76c0a2200 msgr2=0x7ff76c13b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:41.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.433+0000 7ff77180e640 1 --2- 172.21.15.12:0/1240620184 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76c0a2200 0x7ff76c13b160 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff76800e9d0 tx=0x7ff76800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:41.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.433+0000 7ff75e7fc640 1 -- 172.21.15.12:0/1240620184 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff768017070 con 0x7ff76c0a2200 2024-09-11T01:37:41.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.433+0000 7ff75e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:41.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.433+0000 7ff75e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:37:41.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.433+0000 7ff75e7fc640 10 monclient: dump: 2024-09-11T01:37:41.448 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:41.448 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.448 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.448 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.449 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:41.449 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.449 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:41.449 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:41.449 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:41.449 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.433+0000 7ff75e7fc640 10 monclient: _finish_auth 0 2024-09-11T01:37:41.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.433+0000 7ff75e7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:37:41.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.433+0000 7ff75e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:41.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.433+0000 7ff773a72640 5 monclient: authenticate success, global_id 94122 2024-09-11T01:37:41.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.433+0000 7ff75e7fc640 1 -- 172.21.15.12:0/1240620184 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff76800cc70 con 0x7ff76c0a2200 2024-09-11T01:37:41.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.433+0000 7ff75e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:41.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.433+0000 7ff75e7fc640 1 -- 172.21.15.12:0/1240620184 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff768013670 con 0x7ff76c0a2200 2024-09-11T01:37:41.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.433+0000 7ff75e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:41.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.433+0000 7ff75e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:41.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.433+0000 7ff75e7fc640 10 monclient: dump: 2024-09-11T01:37:41.450 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:41.450 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.450 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.450 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.450 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:41.450 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.450 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:41.450 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:41.450 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:41.450 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.433+0000 7ff75f7fe640 4 set_mon_vals no callback set 2024-09-11T01:37:41.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.433+0000 7ff773a72640 10 monclient: _renew_subs 2024-09-11T01:37:41.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.433+0000 7ff773a72640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:41.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.433+0000 7ff773a72640 1 -- 172.21.15.12:0/1240620184 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff76c13baf0 con 0x7ff76c0a2200 2024-09-11T01:37:41.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.433+0000 7ff773a72640 10 monclient: _renew_subs 2024-09-11T01:37:41.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.433+0000 7ff773a72640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:41.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.433+0000 7ff773a72640 1 -- 172.21.15.12:0/1240620184 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff76c11cf60 con 0x7ff76c0a2200 2024-09-11T01:37:41.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.434+0000 7ff773a72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:41.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.434+0000 7ff773a72640 1 --2- 172.21.15.12:0/1240620184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff76c1cbac0 0x7ff76c139080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.434+0000 7ff773a72640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.434+0000 7ff773a72640 1 -- 172.21.15.12:0/1240620184 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff76c11cb30 con 0x7ff76c1cbac0 2024-09-11T01:37:41.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.434+0000 7ff75e7fc640 1 -- 172.21.15.12:0/1240620184 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff768024060 con 0x7ff76c0a2200 2024-09-11T01:37:41.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.434+0000 7ff77100d640 1 --2- 172.21.15.12:0/1240620184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff76c1cbac0 0x7ff76c139080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.434+0000 7ff77100d640 10 monclient: get_auth_request con 0x7ff76c1cbac0 auth_method 0 2024-09-11T01:37:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.434+0000 7ff77100d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.434+0000 7ff77100d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.434+0000 7ff77100d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.434+0000 7ff75e7fc640 1 --2- 172.21.15.12:0/1240620184 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff75003e210 0x7ff7500406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.434+0000 7ff75e7fc640 1 -- 172.21.15.12:0/1240620184 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(285..285 src has 186..285) ==== 4771+0+0 (secure 0 0 0) 0x7ff768052ce0 con 0x7ff76c0a2200 2024-09-11T01:37:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.434+0000 7ff77100d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.434+0000 7ff77100d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.434+0000 7ff77100d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.434+0000 7ff77200f640 1 --2- 172.21.15.12:0/1240620184 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff75003e210 0x7ff7500406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.434+0000 7ff77200f640 10 monclient: get_auth_request con 0x7ff75003e210 auth_method 0 2024-09-11T01:37:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.435+0000 7ff77100d640 10 monclient(hunting): handle_auth_done global_id 74287 payload 824 2024-09-11T01:37:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.435+0000 7ff77100d640 1 -- 172.21.15.12:0/1240620184 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff76c133f00 con 0x7ff76c1cbac0 2024-09-11T01:37:41.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.437+0000 7ff77100d640 1 -- 172.21.15.12:0/1240620184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff76c1cbac0 msgr2=0x7ff76c139080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:41.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.437+0000 7ff77100d640 1 --2- 172.21.15.12:0/1240620184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff76c1cbac0 0x7ff76c139080 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff760034000 tx=0x7ff7600046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:41.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.437+0000 7ff77200f640 1 --2- 172.21.15.12:0/1240620184 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff75003e210 0x7ff7500406d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff764005d00 tx=0x7ff764009470 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:41.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.437+0000 7ff75e7fc640 1 -- 172.21.15.12:0/1240620184 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff760037040 con 0x7ff76c1cbac0 2024-09-11T01:37:41.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.437+0000 7ff75e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:41.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.437+0000 7ff75e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:37:41.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.437+0000 7ff75e7fc640 10 monclient: dump: 2024-09-11T01:37:41.455 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:41.455 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.455 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.455 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.455 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:41.456 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.456 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:41.456 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:41.456 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:41.456 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.437+0000 7ff75e7fc640 1 -- 172.21.15.12:0/1240620184 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff76003ca60 con 0x7ff76c1cbac0 2024-09-11T01:37:41.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.437+0000 7ff75e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:41.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.437+0000 7ff75e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:37:41.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.437+0000 7ff75e7fc640 1 -- 172.21.15.12:0/1240620184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff76c1cbac0 msgr2=0x7ff76c139080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.437+0000 7ff75e7fc640 1 --2- 172.21.15.12:0/1240620184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff76c1cbac0 0x7ff76c139080 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff760034000 tx=0x7ff7600046c0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.438+0000 7ff75e7fc640 1 -- 172.21.15.12:0/1240620184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff76c1cbac0 msgr2=0x7ff76c139080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:37:41.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.438+0000 7ff75e7fc640 1 --2- 172.21.15.12:0/1240620184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff76c1cbac0 0x7ff76c139080 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff760034000 tx=0x7ff7600046c0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.444+0000 7ff773a72640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:37:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.444+0000 7ff773a72640 1 --2- 172.21.15.12:0/1240620184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff76c1395c0 0x7ff76c0726b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.444+0000 7ff773a72640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.444+0000 7ff773a72640 1 -- 172.21.15.12:0/1240620184 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff76c1399a0 con 0x7ff76c1395c0 2024-09-11T01:37:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.444+0000 7ff77100d640 1 --2- 172.21.15.12:0/1240620184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff76c1395c0 0x7ff76c0726b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.444+0000 7ff77100d640 10 monclient: get_auth_request con 0x7ff76c1395c0 auth_method 0 2024-09-11T01:37:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.444+0000 7ff77100d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.444+0000 7ff77100d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.444+0000 7ff77100d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.444+0000 7ff77100d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.444+0000 7ff77100d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.444+0000 7ff77100d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.444+0000 7ff77100d640 10 monclient(hunting): handle_auth_done global_id 74290 payload 824 2024-09-11T01:37:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.444+0000 7ff77100d640 1 -- 172.21.15.12:0/1240620184 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff76c132f60 con 0x7ff76c1395c0 2024-09-11T01:37:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.447+0000 7ff77100d640 1 -- 172.21.15.12:0/1240620184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff76c1395c0 msgr2=0x7ff76c0726b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.447+0000 7ff77100d640 1 --2- 172.21.15.12:0/1240620184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff76c1395c0 0x7ff76c0726b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff76002a020 tx=0x7ff760009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.447+0000 7ff75e7fc640 1 -- 172.21.15.12:0/1240620184 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76004f070 con 0x7ff76c1395c0 2024-09-11T01:37:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.447+0000 7ff75e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.447+0000 7ff75e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:37:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.447+0000 7ff75e7fc640 10 monclient: dump: 2024-09-11T01:37:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:41.458 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.447+0000 7ff75e7fc640 1 -- 172.21.15.12:0/1240620184 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7ff76004d070 con 0x7ff76c1395c0 2024-09-11T01:37:41.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.447+0000 7ff75e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:37:41.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.447+0000 7ff75e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:37:41.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.447+0000 7ff75e7fc640 1 -- 172.21.15.12:0/1240620184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff76c1395c0 msgr2=0x7ff76c0726b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.447+0000 7ff75e7fc640 1 --2- 172.21.15.12:0/1240620184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff76c1395c0 0x7ff76c0726b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff76002a020 tx=0x7ff760009190 comp rx=0 tx=0).stop 2024-09-11T01:37:41.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.447+0000 7ff75e7fc640 1 -- 172.21.15.12:0/1240620184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff76c1395c0 msgr2=0x7ff76c0726b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:37:41.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.447+0000 7ff75e7fc640 1 --2- 172.21.15.12:0/1240620184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff76c1395c0 0x7ff76c0726b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.459 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:37:41.459 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:37:41.459 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:37:41.459 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:37:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 88, 2024-09-11T01:37:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 113226, 2024-09-11T01:37:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:37:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:37:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:37:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:37:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:37:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:37:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:37:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:37:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:41.460 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:37:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:37:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:41.461 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:37:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:37:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:37:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:37:41.462 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:37:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:37:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:37:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:37:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:37:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:37:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:37:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:37:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:37:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:37:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:37:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:37:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:37:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:41.463 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:37:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:37:41.464 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:37:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:37:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:37:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:37:41.465 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:41.466 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:41.466 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:41.466 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.466 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:37:41.466 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:37:41.466 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:41.466 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:41.466 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:41.466 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.466 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.466 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:37:41.466 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:37:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:37:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:37:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:41.467 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:37:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:37:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:37:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:37:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:41.468 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:37:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:37:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:37:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:37:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:41.469 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:37:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:37:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:37:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:37:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.470 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:37:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:37:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:37:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:37:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:41.471 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:41.472 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:41.472 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:37:41.472 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:41.472 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:41.472 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:41.472 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:37:41.472 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:37:41.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.447+0000 7ff773a72640 1 -- 172.21.15.12:0/1240620184 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff75003e210 msgr2=0x7ff7500406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.447+0000 7ff773a72640 1 --2- 172.21.15.12:0/1240620184 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff75003e210 0x7ff7500406d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff764005d00 tx=0x7ff764009470 comp rx=0 tx=0).stop 2024-09-11T01:37:41.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.447+0000 7ff773a72640 10 monclient: shutdown 2024-09-11T01:37:41.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.447+0000 7ff773a72640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:41.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.447+0000 7ff773a72640 1 -- 172.21.15.12:0/1240620184 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76c0a2200 msgr2=0x7ff76c13b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.447+0000 7ff773a72640 1 --2- 172.21.15.12:0/1240620184 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76c0a2200 0x7ff76c13b160 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff76800e9d0 tx=0x7ff76800ee90 comp rx=0 tx=0).stop 2024-09-11T01:37:41.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.447+0000 7ff77200f640 1 -- 172.21.15.12:0/1240620184 reap_dead start 2024-09-11T01:37:41.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.447+0000 7ff77200f640 1 -- 172.21.15.12:0/1240620184 reap_dead start 2024-09-11T01:37:41.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.447+0000 7ff773a72640 1 -- 172.21.15.12:0/1240620184 shutdown_connections 2024-09-11T01:37:41.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.447+0000 7ff773a72640 1 -- 172.21.15.12:0/1240620184 >> 172.21.15.12:0/1240620184 conn(0x7ff76c0a2f70 msgr2=0x7ff76c09d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:41.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.447+0000 7ff773a72640 1 -- 172.21.15.12:0/1240620184 shutdown_connections 2024-09-11T01:37:41.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.447+0000 7ff773a72640 1 -- 172.21.15.12:0/1240620184 wait complete. 2024-09-11T01:37:41.474 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:37:41.474 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:37:41.475 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:37:41.485 INFO:tasks.rados.rados.0.smithi071.stdout:1273: done (2 left) 2024-09-11T01:37:41.486 INFO:tasks.rados.rados.0.smithi071.stdout:1274: rollback oid 22 current snap is 132 2024-09-11T01:37:41.486 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 22 to 123 2024-09-11T01:37:41.486 INFO:tasks.rados.rados.0.smithi071.stdout:1270: finishing write tid 1 to smithi07134777-45 2024-09-11T01:37:41.486 INFO:tasks.rados.rados.0.smithi071.stdout:1270: finishing write tid 2 to smithi07134777-45 2024-09-11T01:37:41.486 INFO:tasks.rados.rados.0.smithi071.stdout:1270: finishing write tid 3 to smithi07134777-45 2024-09-11T01:37:41.486 INFO:tasks.rados.rados.0.smithi071.stdout:1270: finishing write tid 4 to smithi07134777-45 2024-09-11T01:37:41.486 INFO:tasks.rados.rados.0.smithi071.stdout:1270: oid 45 updating version 0 to 520 2024-09-11T01:37:41.486 INFO:tasks.rados.rados.0.smithi071.stdout:1270: oid 45 updating version 520 to 521 2024-09-11T01:37:41.486 INFO:tasks.rados.rados.0.smithi071.stdout:1270: oid 45 updating version 521 to 522 2024-09-11T01:37:41.486 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 522 (ObjNum 486 snap 132 seq_num 486) dirty exists 2024-09-11T01:37:41.486 INFO:tasks.rados.rados.0.smithi071.stdout:1270: left oid 45 (ObjNum 486 snap 132 seq_num 486) 2024-09-11T01:37:41.486 INFO:tasks.rados.rados.0.smithi071.stdout:1270: done (2 left) 2024-09-11T01:37:41.486 INFO:tasks.rados.rados.0.smithi071.stdout:1272: done (1 left) 2024-09-11T01:37:41.487 INFO:tasks.rados.rados.0.smithi071.stdout:1275: write (excl) oid 42 current snap is 132 2024-09-11T01:37:41.487 INFO:tasks.rados.rados.0.smithi071.stdout:1275: seq_num 488 ranges {601816=568104,1623448=451823} 2024-09-11T01:37:41.492 INFO:tasks.rados.rados.0.smithi071.stdout:1275: writing smithi07134777-42 from 601816 to 1169920 tid 1 2024-09-11T01:37:41.498 INFO:tasks.rados.rados.0.smithi071.stdout:1275: writing smithi07134777-42 from 1623448 to 2075271 tid 2 2024-09-11T01:37:41.498 INFO:tasks.rados.rados.0.smithi071.stdout:1276: rollback oid 4 current snap is 132 2024-09-11T01:37:41.498 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 4 to 131 2024-09-11T01:37:41.498 INFO:tasks.rados.rados.0.smithi071.stdout:1274: finishing rollback tid 0 to smithi07134777-22 2024-09-11T01:37:41.498 INFO:tasks.rados.rados.0.smithi071.stdout:1274: finishing rollback tid 1 to smithi07134777-22 2024-09-11T01:37:41.498 INFO:tasks.rados.rados.0.smithi071.stdout:1274: finishing rollback tid 2 to smithi07134777-22 2024-09-11T01:37:41.498 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 254 (ObjNum 426 snap 118 seq_num 426) dirty exists 2024-09-11T01:37:41.498 INFO:tasks.rados.rados.0.smithi071.stdout:1275: finishing write tid 1 to smithi07134777-42 2024-09-11T01:37:41.498 INFO:tasks.rados.rados.0.smithi071.stdout:1274: done (2 left) 2024-09-11T01:37:41.499 INFO:tasks.rados.rados.0.smithi071.stdout:1277: rollback oid 2 current snap is 132 2024-09-11T01:37:41.499 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 2 to 126 2024-09-11T01:37:41.499 INFO:tasks.rados.rados.0.smithi071.stdout:1278: write oid 19 current snap is 132 2024-09-11T01:37:41.499 INFO:tasks.rados.rados.0.smithi071.stdout:1278: seq_num 489 ranges {618623=643353,1681930=536693} 2024-09-11T01:37:41.505 INFO:tasks.rados.rados.0.smithi071.stdout:1278: writing smithi07134777-19 from 618623 to 1261976 tid 1 2024-09-11T01:37:41.510 INFO:tasks.rados.rados.0.smithi071.stdout:1278: writing smithi07134777-19 from 1681930 to 2218623 tid 2 2024-09-11T01:37:41.510 INFO:tasks.rados.rados.0.smithi071.stdout:1279: rollback oid 45 current snap is 132 2024-09-11T01:37:41.511 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 45 to 128 2024-09-11T01:37:41.511 INFO:tasks.rados.rados.0.smithi071.stdout:1275: finishing write tid 2 to smithi07134777-42 2024-09-11T01:37:41.511 INFO:tasks.rados.rados.0.smithi071.stdout:1275: finishing write tid 3 to smithi07134777-42 2024-09-11T01:37:41.511 INFO:tasks.rados.rados.0.smithi071.stdout:1275: finishing write tid 4 to smithi07134777-42 2024-09-11T01:37:41.511 INFO:tasks.rados.rados.0.smithi071.stdout:1275: oid 42 updating version 0 to 430 2024-09-11T01:37:41.511 INFO:tasks.rados.rados.0.smithi071.stdout:1275: oid 42 updating version 430 to 431 2024-09-11T01:37:41.511 INFO:tasks.rados.rados.0.smithi071.stdout:1275: oid 42 version 431 is already newer than 429 2024-09-11T01:37:41.511 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 431 (ObjNum 488 snap 132 seq_num 488) dirty exists 2024-09-11T01:37:41.511 INFO:tasks.rados.rados.0.smithi071.stdout:1275: left oid 42 (ObjNum 488 snap 132 seq_num 488) 2024-09-11T01:37:41.511 INFO:tasks.rados.rados.0.smithi071.stdout:1277: finishing rollback tid 1 to smithi07134777-2 2024-09-11T01:37:41.511 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 428 (ObjNum 929 snap 0 seq_num 2090715384) dirty dne 2024-09-11T01:37:41.511 INFO:tasks.rados.rados.0.smithi071.stdout:1275: done (4 left) 2024-09-11T01:37:41.511 INFO:tasks.rados.rados.0.smithi071.stdout:1277: done (3 left) 2024-09-11T01:37:41.511 INFO:tasks.rados.rados.0.smithi071.stdout:1280: copy_from oid 38 from oid 7 current snap is 132 2024-09-11T01:37:41.512 INFO:tasks.rados.rados.0.smithi071.stdout:1281: write (excl) oid 30 current snap is 132 2024-09-11T01:37:41.512 INFO:tasks.rados.rados.0.smithi071.stdout:1281: seq_num 490 ranges {635430=718602,2140412=95018} 2024-09-11T01:37:41.517 INFO:tasks.rados.rados.0.smithi071.stdout:1281: writing smithi07134777-30 from 635430 to 1354032 tid 1 2024-09-11T01:37:41.522 INFO:tasks.rados.rados.0.smithi071.stdout:1281: writing smithi07134777-30 from 2140412 to 2235430 tid 2 2024-09-11T01:37:41.522 INFO:tasks.rados.rados.0.smithi071.stdout:1282: snap_remove snap 123 2024-09-11T01:37:41.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.581+0000 7ff5b28a9640 10 monclient: get_monmap_and_config 2024-09-11T01:37:41.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.581+0000 7ff5b28a9640 10 monclient: build_initial_monmap 2024-09-11T01:37:41.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.581+0000 7ff5b28a9640 10 monclient: monmap: 2024-09-11T01:37:41.584 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:41.584 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.584 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:37:41.583417+0000 2024-09-11T01:37:41.584 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:37:41.583417+0000 2024-09-11T01:37:41.585 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:41.585 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.585 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:37:41.585 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:37:41.585 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:37:41.585 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:37:41.585 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:37:41.585 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.582+0000 7ff5b28a9640 1 Processor -- start 2024-09-11T01:37:41.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.582+0000 7ff5b28a9640 1 -- start start 2024-09-11T01:37:41.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.582+0000 7ff5b28a9640 10 monclient: init 2024-09-11T01:37:41.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.582+0000 7ff5b28a9640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:41.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.582+0000 7ff5b28a9640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-11T01:37:41.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.582+0000 7ff5b28a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff5ac134ad0 addr v1:172.21.15.12:6789/0 2024-09-11T01:37:41.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.582+0000 7ff5b28a9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff5ac0a2200 0x7ff5ac13a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.582+0000 7ff5b28a9640 10 monclient(hunting): picked mon.noname-a con 0x7ff5ac0a2200 addr v2:172.21.15.12:3300/0 2024-09-11T01:37:41.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.582+0000 7ff5b28a9640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff5ac1355c0 0x7ff5ac13c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.582+0000 7ff5b28a9640 10 monclient(hunting): picked mon.noname-c con 0x7ff5ac1355c0 addr v2:172.21.15.71:3300/0 2024-09-11T01:37:41.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.582+0000 7ff5b28a9640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5ac09ed30 con 0x7ff5ac134ad0 2024-09-11T01:37:41.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.582+0000 7ff5b28a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.582+0000 7ff5b28a9640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff5ac09f5d0 con 0x7ff5ac0a2200 2024-09-11T01:37:41.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.582+0000 7ff5b28a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.582+0000 7ff5b28a9640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7ff5ac09f010 con 0x7ff5ac1355c0 2024-09-11T01:37:41.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.582+0000 7ff5b28a9640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:41.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.582+0000 7ff5b28a9640 10 monclient(hunting): authenticate will time out at 1268.025989s 2024-09-11T01:37:41.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.583+0000 7ff5ab7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff5ac0a2200 0x7ff5ac13a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.583+0000 7ff5abfff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff5ac134ad0 0x7ff5ac134eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55916/0 (socket says 172.21.15.12:55916) 2024-09-11T01:37:41.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.583+0000 7ff5abfff640 1 -- 172.21.15.12:0/849844498 learned_addr learned my addr 172.21.15.12:0/849844498 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:37:41.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.583+0000 7ff5ab7fe640 10 monclient(hunting): get_auth_request con 0x7ff5ac0a2200 auth_method 0 2024-09-11T01:37:41.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.583+0000 7ff5ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:41.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.583+0000 7ff5ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:41.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.583+0000 7ff5ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:41.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.583+0000 7ff5ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:41.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.583+0000 7ff5ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:41.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.583+0000 7ff5ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:41.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.583+0000 7ff5b0e46640 1 --2- 172.21.15.12:0/849844498 >> v2:172.21.15.71:3300/0 conn(0x7ff5ac1355c0 0x7ff5ac13c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.583+0000 7ff5ab7fe640 10 monclient(hunting): handle_auth_done global_id 94125 payload 274 2024-09-11T01:37:41.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.583+0000 7ff5ab7fe640 1 -- 172.21.15.12:0/849844498 >> v2:172.21.15.71:3300/0 conn(0x7ff5ac1355c0 msgr2=0x7ff5ac13c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:41.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.583+0000 7ff5ab7fe640 1 --2- 172.21.15.12:0/849844498 >> v2:172.21.15.71:3300/0 conn(0x7ff5ac1355c0 0x7ff5ac13c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.583+0000 7ff5ab7fe640 1 -- 172.21.15.12:0/849844498 >> v1:172.21.15.12:6789/0 conn(0x7ff5ac134ad0 legacy=0x7ff5ac134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:41.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.583+0000 7ff5ab7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:37:41.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.583+0000 7ff5ab7fe640 1 monclient: found mon.noname-a 2024-09-11T01:37:41.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.583+0000 7ff5ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:41.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.583+0000 7ff5ab7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:37:41.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.583+0000 7ff5ab7fe640 1 -- 172.21.15.12:0/849844498 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5ac09f2f0 con 0x7ff5ac0a2200 2024-09-11T01:37:41.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.583+0000 7ff5b0e46640 10 monclient: get_auth_request con 0x7ff5ac1355c0 auth_method 0 2024-09-11T01:37:41.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.583+0000 7ff5b0e46640 1 --2- 172.21.15.12:0/849844498 >> v2:172.21.15.71:3300/0 conn(0x7ff5ac1355c0 0x7ff5ac13c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:37:41.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.586+0000 7ff5ab7fe640 1 -- 172.21.15.12:0/849844498 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5ac0a2200 msgr2=0x7ff5ac13a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:41.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.586+0000 7ff5ab7fe640 1 --2- 172.21.15.12:0/849844498 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5ac0a2200 0x7ff5ac13a000 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff5a00097a0 tx=0x7ff5a002a010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f60b1734cc18a2d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:41.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.586+0000 7ff5aaffd640 1 -- 172.21.15.12:0/849844498 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a0037070 con 0x7ff5ac0a2200 2024-09-11T01:37:41.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.586+0000 7ff5aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:41.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.586+0000 7ff5aaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:37:41.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.586+0000 7ff5aaffd640 10 monclient: dump: 2024-09-11T01:37:41.595 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:41.595 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.595 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.596 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.596 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:41.596 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.596 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:41.596 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:41.596 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:41.596 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.586+0000 7ff5aaffd640 10 monclient: _finish_auth 0 2024-09-11T01:37:41.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.586+0000 7ff5aaffd640 10 monclient: _check_auth_tickets 2024-09-11T01:37:41.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.586+0000 7ff5aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:41.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.586+0000 7ff5aaffd640 1 -- 172.21.15.12:0/849844498 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5a0032640 con 0x7ff5ac0a2200 2024-09-11T01:37:41.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.586+0000 7ff5b28a9640 5 monclient: authenticate success, global_id 94125 2024-09-11T01:37:41.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.586+0000 7ff5b28a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:37:41.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.586+0000 7ff5aaffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:41.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.586+0000 7ff5b28a9640 10 monclient: get_monmap_and_config success 2024-09-11T01:37:41.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.586+0000 7ff5aaffd640 1 -- 172.21.15.12:0/849844498 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a0032a10 con 0x7ff5ac0a2200 2024-09-11T01:37:41.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.586+0000 7ff5b28a9640 4 set_mon_vals no callback set 2024-09-11T01:37:41.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.586+0000 7ff5aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:41.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.586+0000 7ff5aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:41.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.586+0000 7ff5aaffd640 10 monclient: dump: 2024-09-11T01:37:41.597 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:41.597 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.597 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.597 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.597 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:41.598 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.598 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:41.598 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:41.598 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:41.598 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.586+0000 7ff5b28a9640 10 monclient: shutdown 2024-09-11T01:37:41.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.586+0000 7ff5b28a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:41.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.586+0000 7ff5b28a9640 1 -- 172.21.15.12:0/849844498 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5ac0a2200 msgr2=0x7ff5ac13a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.586+0000 7ff5b28a9640 1 --2- 172.21.15.12:0/849844498 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5ac0a2200 0x7ff5ac13a000 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff5a00097a0 tx=0x7ff5a002a010 comp rx=0 tx=0).stop 2024-09-11T01:37:41.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.586+0000 7ff5b28a9640 1 -- 172.21.15.12:0/849844498 shutdown_connections 2024-09-11T01:37:41.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.586+0000 7ff5b28a9640 1 --2- 172.21.15.12:0/849844498 >> v2:172.21.15.71:3300/0 conn(0x7ff5ac1355c0 0x7ff5ac13c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.586+0000 7ff5b28a9640 1 --2- 172.21.15.12:0/849844498 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5ac0a2200 0x7ff5ac13a000 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.587+0000 7ff5b28a9640 1 -- 172.21.15.12:0/849844498 >> 172.21.15.12:0/849844498 conn(0x7ff5ac0a2f70 msgr2=0x7ff5ac0a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:41.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.587+0000 7ff5b28a9640 1 -- 172.21.15.12:0/849844498 shutdown_connections 2024-09-11T01:37:41.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.587+0000 7ff5b28a9640 1 -- 172.21.15.12:0/849844498 wait complete. 2024-09-11T01:37:41.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.587+0000 7ff5b28a9640 10 monclient: build_initial_monmap 2024-09-11T01:37:41.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.587+0000 7ff5b28a9640 10 monclient: monmap: 2024-09-11T01:37:41.599 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:41.599 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:37:41.599 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:37:41.599 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:37:41.599 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:41.599 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.599 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:37:41.599 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:37:41.599 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:37:41.600 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.587+0000 7ff5b28a9640 1 Processor -- start 2024-09-11T01:37:41.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5b28a9640 1 -- start start 2024-09-11T01:37:41.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5b28a9640 10 monclient: init 2024-09-11T01:37:41.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5b28a9640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:41.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5b28a9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:37:41.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5b28a9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5ac0a2200 0x7ff5ac133330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5b28a9640 10 monclient(hunting): picked mon.noname-a con 0x7ff5ac0a2200 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:37:41.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5b28a9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5ac134ad0 0x7ff5ac133870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5b28a9640 10 monclient(hunting): picked mon.noname-b con 0x7ff5ac134ad0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:37:41.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5b28a9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff5ac1355c0 0x7ff5ac139df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5b28a9640 10 monclient(hunting): picked mon.noname-c con 0x7ff5ac1355c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:41.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5b28a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5b28a9640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff5ac0a0150 con 0x7ff5ac0a2200 2024-09-11T01:37:41.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5b28a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5b28a9640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff5ac09ffd0 con 0x7ff5ac134ad0 2024-09-11T01:37:41.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5b28a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:41.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5b28a9640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff5ac13ca90 con 0x7ff5ac1355c0 2024-09-11T01:37:41.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5b28a9640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:41.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5b28a9640 10 monclient(hunting): authenticate will time out at 1268.031696s 2024-09-11T01:37:41.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5abfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5ac0a2200 0x7ff5ac133330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5b0e46640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff5ac1355c0 0x7ff5ac139df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5abfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5ac0a2200 0x7ff5ac133330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53838/0 (socket says 172.21.15.12:53838) 2024-09-11T01:37:41.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5abfff640 1 -- 172.21.15.12:0/367032987 learned_addr learned my addr 172.21.15.12:0/367032987 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:41.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5abfff640 10 monclient(hunting): get_auth_request con 0x7ff5ac0a2200 auth_method 0 2024-09-11T01:37:41.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:41.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5abfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:41.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:41.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5b0e46640 10 monclient(hunting): get_auth_request con 0x7ff5ac1355c0 auth_method 0 2024-09-11T01:37:41.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5b0e46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:41.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5b0e46640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:41.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5b0e46640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:41.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:41.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:41.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:41.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5b0e46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:41.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5b0e46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:41.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5b0e46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:41.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5ab7fe640 1 --2- 172.21.15.12:0/367032987 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5ac134ad0 0x7ff5ac133870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5ab7fe640 10 monclient(hunting): get_auth_request con 0x7ff5ac134ad0 auth_method 0 2024-09-11T01:37:41.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:41.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:41.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.588+0000 7ff5ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:41.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.589+0000 7ff5abfff640 10 monclient(hunting): handle_auth_done global_id 94128 payload 824 2024-09-11T01:37:41.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.589+0000 7ff5abfff640 1 -- 172.21.15.12:0/367032987 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff5ac1355c0 msgr2=0x7ff5ac139df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.589+0000 7ff5abfff640 1 --2- 172.21.15.12:0/367032987 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff5ac1355c0 0x7ff5ac139df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.589+0000 7ff5abfff640 1 -- 172.21.15.12:0/367032987 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5ac134ad0 msgr2=0x7ff5ac133870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.589+0000 7ff5abfff640 1 --2- 172.21.15.12:0/367032987 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5ac134ad0 0x7ff5ac133870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.589+0000 7ff5abfff640 10 monclient: _finish_hunting 0 2024-09-11T01:37:41.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.589+0000 7ff5abfff640 1 monclient: found mon.noname-a 2024-09-11T01:37:41.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.589+0000 7ff5abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:41.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.589+0000 7ff5abfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:37:41.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.589+0000 7ff5abfff640 1 -- 172.21.15.12:0/367032987 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5ac11ce60 con 0x7ff5ac0a2200 2024-09-11T01:37:41.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.589+0000 7ff5ab7fe640 1 --2- 172.21.15.12:0/367032987 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5ac134ad0 0x7ff5ac133870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:37:41.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.591+0000 7ff5abfff640 1 -- 172.21.15.12:0/367032987 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5ac0a2200 msgr2=0x7ff5ac133330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:41.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.591+0000 7ff5abfff640 1 --2- 172.21.15.12:0/367032987 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5ac0a2200 0x7ff5ac133330 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff59c00c460 tx=0x7ff59c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:41.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.591+0000 7ff5a8ff9640 1 -- 172.21.15.12:0/367032987 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59c007070 con 0x7ff5ac0a2200 2024-09-11T01:37:41.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.591+0000 7ff5a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:41.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.591+0000 7ff5a8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:37:41.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.591+0000 7ff5a8ff9640 10 monclient: dump: 2024-09-11T01:37:41.604 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:41.604 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.604 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.604 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.604 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:41.604 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.605 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:41.605 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:41.605 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:41.605 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.591+0000 7ff5a8ff9640 10 monclient: _finish_auth 0 2024-09-11T01:37:41.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.591+0000 7ff5a8ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:37:41.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.591+0000 7ff5a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:41.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.591+0000 7ff5a8ff9640 1 -- 172.21.15.12:0/367032987 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff59c007230 con 0x7ff5ac0a2200 2024-09-11T01:37:41.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.591+0000 7ff5b28a9640 5 monclient: authenticate success, global_id 94128 2024-09-11T01:37:41.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.591+0000 7ff5b28a9640 10 monclient: _renew_subs 2024-09-11T01:37:41.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.591+0000 7ff5b28a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:41.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.591+0000 7ff5b28a9640 1 -- 172.21.15.12:0/367032987 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5ac11e020 con 0x7ff5ac0a2200 2024-09-11T01:37:41.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.591+0000 7ff5a8ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:41.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.591+0000 7ff5a8ff9640 1 -- 172.21.15.12:0/367032987 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59c007620 con 0x7ff5ac0a2200 2024-09-11T01:37:41.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.591+0000 7ff5a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:41.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.591+0000 7ff5a8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:41.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.591+0000 7ff5a8ff9640 10 monclient: dump: 2024-09-11T01:37:41.606 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:41.606 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:41.606 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.606 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:41.606 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:41.606 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:41.606 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:41.606 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:41.606 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:41.606 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:41.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.591+0000 7ff5b28a9640 10 monclient: _renew_subs 2024-09-11T01:37:41.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.591+0000 7ff5aa7fc640 4 set_mon_vals no callback set 2024-09-11T01:37:41.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.591+0000 7ff5b28a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:41.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.591+0000 7ff5b28a9640 1 -- 172.21.15.12:0/367032987 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5ac0a51a0 con 0x7ff5ac0a2200 2024-09-11T01:37:41.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.592+0000 7ff5a8ff9640 1 -- 172.21.15.12:0/367032987 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff59c01b400 con 0x7ff5ac0a2200 2024-09-11T01:37:41.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.592+0000 7ff5b28a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:41.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.592+0000 7ff5b28a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:41.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.592+0000 7ff5b28a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:41.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.592+0000 7ff5b28a9640 1 -- 172.21.15.12:0/367032987 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5ac134eb0 con 0x7ff5ac0a2200 2024-09-11T01:37:41.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.592+0000 7ff5a8ff9640 1 --2- 172.21.15.12:0/367032987 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff59803e210 0x7ff5980406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:41.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.592+0000 7ff5a8ff9640 1 -- 172.21.15.12:0/367032987 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(286..286 src has 186..286) ==== 4799+0+0 (secure 0 0 0) 0x7ff59c052a10 con 0x7ff5ac0a2200 2024-09-11T01:37:41.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.592+0000 7ff5ab7fe640 1 --2- 172.21.15.12:0/367032987 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff59803e210 0x7ff5980406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:41.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.592+0000 7ff5ab7fe640 10 monclient: get_auth_request con 0x7ff59803e210 auth_method 0 2024-09-11T01:37:41.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.595+0000 7ff5ab7fe640 1 --2- 172.21.15.12:0/367032987 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff59803e210 0x7ff5980406d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac0a72b0 tx=0x7ff5a002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:41.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.597+0000 7ff5a8ff9640 1 -- 172.21.15.12:0/367032987 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff59c051400 con 0x7ff5ac0a2200 2024-09-11T01:37:41.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.597+0000 7ff5a8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:41.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.597+0000 7ff5a8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:37:41.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.857+0000 7ff5b28a9640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:37:41.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.857+0000 7ff5b28a9640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:37:41.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.857+0000 7ff5b28a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:41.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.857+0000 7ff5b28a9640 1 -- 172.21.15.12:0/367032987 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7ff5ac06d480 con 0x7ff5ac0a2200 2024-09-11T01:37:41.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.858+0000 7ff5a8ff9640 1 -- 172.21.15.12:0/367032987 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7ff59c010d10 con 0x7ff5ac0a2200 2024-09-11T01:37:41.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.858+0000 7ff5a8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:37:41.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.858+0000 7ff5a8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:37:41.861 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.858+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.861 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.858+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=12,logm=58} crc {auth=1992314713,config=2586702516,health=4058548626,kv=3263697420,logm=568495260}) 2024-09-11T01:37:41.861 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.858+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=16,config=2,health=12,kv=12,logm=58} crc {auth=1992314713,config=2586702516,health=4058548626,kv=3263697420,logm=1148744594}) 2024-09-11T01:37:41.861 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.858+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.862 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.858+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=12,logm=58} crc {auth=1992314713,config=2586702516,health=4058548626,kv=3263697420,logm=568495260}) 2024-09-11T01:37:41.862 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.858+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=15,config=2,health=10,kv=12,logm=61} crc {auth=193983216,config=2586702516,health=1062321089,kv=3263697420,logm=793056177}) 2024-09-11T01:37:41.862 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.859+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.862 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.859+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334059707}) 2024-09-11T01:37:41.862 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.859+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1410791112}) 2024-09-11T01:37:41.862 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.859+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.862 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.859+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334059707}) 2024-09-11T01:37:41.862 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.859+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3235490790}) 2024-09-11T01:37:41.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.860+0000 7ff5b28a9640 1 -- 172.21.15.12:0/367032987 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff59803e210 msgr2=0x7ff5980406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.860+0000 7ff5b28a9640 1 --2- 172.21.15.12:0/367032987 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff59803e210 0x7ff5980406d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac0a72b0 tx=0x7ff5a002e000 comp rx=0 tx=0).stop 2024-09-11T01:37:41.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.860+0000 7ff5b28a9640 10 monclient: shutdown 2024-09-11T01:37:41.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.860+0000 7ff5b28a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:41.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.860+0000 7ff5b28a9640 1 -- 172.21.15.12:0/367032987 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5ac0a2200 msgr2=0x7ff5ac133330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:41.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.860+0000 7ff5b28a9640 1 --2- 172.21.15.12:0/367032987 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5ac0a2200 0x7ff5ac133330 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff59c00c460 tx=0x7ff59c00c920 comp rx=0 tx=0).stop 2024-09-11T01:37:41.864 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.860+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.864 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.860+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918203770}) 2024-09-11T01:37:41.864 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.860+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1749800699}) 2024-09-11T01:37:41.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.860+0000 7ff5b28a9640 1 -- 172.21.15.12:0/367032987 shutdown_connections 2024-09-11T01:37:41.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.860+0000 7ff5b28a9640 1 --2- 172.21.15.12:0/367032987 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff5ac1355c0 0x7ff5ac139df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.865 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.860+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.860+0000 7ff5b28a9640 1 --2- 172.21.15.12:0/367032987 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5ac134ad0 0x7ff5ac133870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.860+0000 7ff5b28a9640 1 --2- 172.21.15.12:0/367032987 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff59803e210 0x7ff5980406d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.860+0000 7ff5b28a9640 1 --2- 172.21.15.12:0/367032987 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff5ac0a2200 0x7ff5ac133330 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:41.866 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.860+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918203770}) 2024-09-11T01:37:41.866 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.860+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1033158793}) 2024-09-11T01:37:41.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.860+0000 7ff5b28a9640 1 -- 172.21.15.12:0/367032987 >> 172.21.15.12:0/367032987 conn(0x7ff5ac0a2f70 msgr2=0x7ff5ac0a4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:41.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.860+0000 7ff5b28a9640 1 -- 172.21.15.12:0/367032987 shutdown_connections 2024-09-11T01:37:41.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:41.860+0000 7ff5b28a9640 1 -- 172.21.15.12:0/367032987 wait complete. 2024-09-11T01:37:41.869 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.861+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.869 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.861+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1517766145}) 2024-09-11T01:37:41.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.861+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3967662236}) 2024-09-11T01:37:41.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.861+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.861+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1517766145}) 2024-09-11T01:37:41.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.861+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1182530520}) 2024-09-11T01:37:41.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.862+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.862+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2450132816}) 2024-09-11T01:37:41.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.862+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2052874897}) 2024-09-11T01:37:41.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.862+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.862+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2450132816}) 2024-09-11T01:37:41.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.862+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3327253316}) 2024-09-11T01:37:41.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.863+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.863+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2703048921}) 2024-09-11T01:37:41.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.863+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3583787038}) 2024-09-11T01:37:41.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.863+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.863+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2703048921}) 2024-09-11T01:37:41.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.863+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2928710877}) 2024-09-11T01:37:41.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.864+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.864+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3363994776}) 2024-09-11T01:37:41.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.864+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2174527915}) 2024-09-11T01:37:41.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.864+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.864+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3363994776}) 2024-09-11T01:37:41.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.864+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1296116832}) 2024-09-11T01:37:41.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.864+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.864+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439899688}) 2024-09-11T01:37:41.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.864+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3492300436}) 2024-09-11T01:37:41.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.864+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.864+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439899688}) 2024-09-11T01:37:41.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.864+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2422450524}) 2024-09-11T01:37:41.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.865+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.865+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511657295}) 2024-09-11T01:37:41.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.865+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=835146656}) 2024-09-11T01:37:41.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.865+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.865+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511657295}) 2024-09-11T01:37:41.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.865+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2234267444}) 2024-09-11T01:37:41.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.866+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.866+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=93803119}) 2024-09-11T01:37:41.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.866+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3693864321}) 2024-09-11T01:37:41.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.866+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.866+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=93803119}) 2024-09-11T01:37:41.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.866+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3227185430}) 2024-09-11T01:37:41.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.867+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.867+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3176237704}) 2024-09-11T01:37:41.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.867+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=597819921}) 2024-09-11T01:37:41.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.867+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.867+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3176237704}) 2024-09-11T01:37:41.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.867+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2328613662}) 2024-09-11T01:37:41.875 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.868+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.875 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.868+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938101076}) 2024-09-11T01:37:41.875 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.868+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=413490940}) 2024-09-11T01:37:41.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.868+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.868+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938101076}) 2024-09-11T01:37:41.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.868+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2571629043}) 2024-09-11T01:37:41.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.868+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.868+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3739612498}) 2024-09-11T01:37:41.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.868+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3324898141}) 2024-09-11T01:37:41.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.868+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.868+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3739612498}) 2024-09-11T01:37:41.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.868+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4276677539}) 2024-09-11T01:37:41.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.869+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.869+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3363285730}) 2024-09-11T01:37:41.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.869+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2608277939}) 2024-09-11T01:37:41.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.869+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.869+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3363285730}) 2024-09-11T01:37:41.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.869+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1911086643}) 2024-09-11T01:37:41.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.870+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.870+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1514984439}) 2024-09-11T01:37:41.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.870+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2394085763}) 2024-09-11T01:37:41.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.870+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.870+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1514984439}) 2024-09-11T01:37:41.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.870+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3376274783}) 2024-09-11T01:37:41.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.871+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.871+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701730162}) 2024-09-11T01:37:41.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.871+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3635748309}) 2024-09-11T01:37:41.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.871+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.871+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701730162}) 2024-09-11T01:37:41.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.871+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1505655165}) 2024-09-11T01:37:41.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.872+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.872+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2678931087}) 2024-09-11T01:37:41.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.872+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2887968493}) 2024-09-11T01:37:41.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.872+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.872+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2678931087}) 2024-09-11T01:37:41.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.872+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3663037841}) 2024-09-11T01:37:41.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.872+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.872+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3511046558}) 2024-09-11T01:37:41.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.872+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=282653992}) 2024-09-11T01:37:41.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.872+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.872+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3511046558}) 2024-09-11T01:37:41.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.873+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=152651727}) 2024-09-11T01:37:41.880 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:37:41.882 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.873+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.882 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.873+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3574316069}) 2024-09-11T01:37:41.882 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.873+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1024736140}) 2024-09-11T01:37:41.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.873+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.874+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3574316069}) 2024-09-11T01:37:41.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.874+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=961837512}) 2024-09-11T01:37:41.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.874+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.874+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1536826432}) 2024-09-11T01:37:41.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.874+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1834213073}) 2024-09-11T01:37:41.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.874+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.874+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1536826432}) 2024-09-11T01:37:41.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.875+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2943579945}) 2024-09-11T01:37:41.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.875+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.875+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=232721143}) 2024-09-11T01:37:41.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.875+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=733717482}) 2024-09-11T01:37:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.875+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.875+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=232721143}) 2024-09-11T01:37:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.875+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=792045308}) 2024-09-11T01:37:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.876+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.876+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=863915848}) 2024-09-11T01:37:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.876+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2514088891}) 2024-09-11T01:37:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.876+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.876+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=863915848}) 2024-09-11T01:37:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.876+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1360892000}) 2024-09-11T01:37:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.877+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.877+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=8,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=15,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=40} crc {logm=4199938604,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=875795559,mon_config_key=1309540934,monmap=2933275274,nvmeofgw=3140514009,osd_metadata=2942852742,osd_pg_creating=3730616862,osd_snap=361506722}) 2024-09-11T01:37:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.877+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=8,mdsmap=3,mgr=6,mgr_command_descs=1,mgrstat=12,mon_config_key=16,monmap=2,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=43} crc {logm=4199938604,mdsmap=3760429986,mgr=1066993574,mgr_command_descs=3192186153,mgrstat=875795559,mon_config_key=1809882168,monmap=1324671969,nvmeofgw=598337686,osd_metadata=1404192910,osd_pg_creating=3730616862,osd_snap=507755817}) 2024-09-11T01:37:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.877+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.877+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=8,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=15,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=40} crc {logm=4199938604,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=875795559,mon_config_key=1309540934,monmap=2933275274,nvmeofgw=3140514009,osd_metadata=2942852742,osd_pg_creating=3730616862,osd_snap=361506722}) 2024-09-11T01:37:41.885 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.877+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=8,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=38} crc {logm=4199938604,mdsmap=3760429986,mgr=2161719361,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=485785394,mon_config_key=3552379192,monmap=1972551139,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=3730616862,osd_snap=1555504065}) 2024-09-11T01:37:41.885 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.878+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.885 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.878+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=12,osdmap=88} crc {osd_snap=3421378055,osdmap=3977314604}) 2024-09-11T01:37:41.885 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.878+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=12,osdmap=88} crc {osd_snap=1656768257,osdmap=2656954284}) 2024-09-11T01:37:41.885 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.878+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.885 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.878+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=12,osdmap=88} crc {osd_snap=3421378055,osdmap=3977314604}) 2024-09-11T01:37:41.885 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.878+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=12,osdmap=88} crc {osd_snap=2619164757,osdmap=1592170529}) 2024-09-11T01:37:41.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.883+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.883+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=73} crc {osdmap=3795808427}) 2024-09-11T01:37:41.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.883+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=69} crc {osdmap=1021907461}) 2024-09-11T01:37:41.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.883+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:37:41.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.883+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=73} crc {osdmap=3795808427}) 2024-09-11T01:37:41.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:37:41.883+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=70} crc {osdmap=946903660}) 2024-09-11T01:37:42.494 INFO:tasks.rados.rados.0.smithi071.stdout:1282: done (5 left) 2024-09-11T01:37:42.495 INFO:tasks.rados.rados.0.smithi071.stdout:1283: write (excl) oid 43 current snap is 132 2024-09-11T01:37:42.495 INFO:tasks.rados.rados.0.smithi071.stdout:1280: finishing copy_from to smithi07134777-38 2024-09-11T01:37:42.495 INFO:tasks.rados.rados.0.smithi071.stdout:1280: got expected ENOENT (src dne) 2024-09-11T01:37:42.495 INFO:tasks.rados.rados.0.smithi071.stdout:1283: seq_num 491 ranges {652237=793851,2198894=53343} 2024-09-11T01:37:42.503 INFO:tasks.rados.rados.0.smithi071.stdout:1283: writing smithi07134777-43 from 652237 to 1446088 tid 1 2024-09-11T01:37:42.508 INFO:tasks.rados.rados.0.smithi071.stdout:1283: writing smithi07134777-43 from 2198894 to 2252237 tid 2 2024-09-11T01:37:42.508 INFO:tasks.rados.rados.0.smithi071.stdout:1284: snap_remove snap 126 2024-09-11T01:37:42.880 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:37:42.881 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:37:42.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.992+0000 7f1f82652640 10 monclient: get_monmap_and_config 2024-09-11T01:37:42.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.992+0000 7f1f82652640 10 monclient: build_initial_monmap 2024-09-11T01:37:42.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.992+0000 7f1f82652640 10 monclient: monmap: 2024-09-11T01:37:42.994 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:42.995 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:42.995 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:37:42.993805+0000 2024-09-11T01:37:42.995 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:37:42.993805+0000 2024-09-11T01:37:42.995 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:42.995 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:42.995 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:37:42.995 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:37:42.995 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:37:42.995 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:37:42.995 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:37:42.995 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:42.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.992+0000 7f1f82652640 1 Processor -- start 2024-09-11T01:37:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f82652640 1 -- start start 2024-09-11T01:37:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f82652640 10 monclient: init 2024-09-11T01:37:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f82652640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f82652640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-11T01:37:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f82652640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1f7c138230 addr v1:172.21.15.71:6789/0 2024-09-11T01:37:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f82652640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1f7c09d030 0x7f1f7c13d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f82652640 10 monclient(hunting): picked mon.noname-a con 0x7f1f7c09d030 addr v2:172.21.15.12:3300/0 2024-09-11T01:37:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f82652640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1f7c09d460 addr v1:172.21.15.12:6789/0 2024-09-11T01:37:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f82652640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f7c0a6e30 con 0x7f1f7c09d460 2024-09-11T01:37:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f82652640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f7c09ba20 con 0x7f1f7c138230 2024-09-11T01:37:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f82652640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f82652640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f1f7c09c9c0 con 0x7f1f7c09d030 2024-09-11T01:37:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f82652640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f82652640 10 monclient(hunting): authenticate will time out at 1269.436577s 2024-09-11T01:37:42.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f7b7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1f7c09d030 0x7f1f7c13d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:42.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f7b7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1f7c09d030 0x7f1f7c13d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53850/0 (socket says 172.21.15.12:53850) 2024-09-11T01:37:42.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f7b7fe640 1 -- 172.21.15.12:0/1463296828 learned_addr learned my addr 172.21.15.12:0/1463296828 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:42.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f7b7fe640 10 monclient(hunting): get_auth_request con 0x7f1f7c09d030 auth_method 0 2024-09-11T01:37:42.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f80bef640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1f7c09d460 0x7f1f7c13f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55922/0 (socket says 172.21.15.12:55922) 2024-09-11T01:37:42.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:42.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:42.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:42.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:42.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:42.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:42.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f7b7fe640 10 monclient(hunting): handle_auth_done global_id 94131 payload 274 2024-09-11T01:37:42.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f7b7fe640 1 -- 172.21.15.12:0/1463296828 >> v1:172.21.15.71:6789/0 conn(0x7f1f7c138230 legacy=0x7f1f7c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:43.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.993+0000 7f1f7b7fe640 1 -- 172.21.15.12:0/1463296828 >> v1:172.21.15.12:6789/0 conn(0x7f1f7c09d460 legacy=0x7f1f7c13f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:43.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.994+0000 7f1f7b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:37:43.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.994+0000 7f1f7b7fe640 1 monclient: found mon.noname-a 2024-09-11T01:37:43.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.994+0000 7f1f7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:43.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.994+0000 7f1f7b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:37:43.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.994+0000 7f1f7b7fe640 1 -- 172.21.15.12:0/1463296828 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f7c0a66a0 con 0x7f1f7c09d030 2024-09-11T01:37:43.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.996+0000 7f1f7b7fe640 1 -- 172.21.15.12:0/1463296828 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f7c09d030 msgr2=0x7f1f7c13d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:43.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.996+0000 7f1f7b7fe640 1 --2- 172.21.15.12:0/1463296828 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f7c09d030 0x7f1f7c13d090 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1f70009870 tx=0x7f1f7002a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c42cf597b6b6d12 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:43.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.997+0000 7f1f7affd640 1 -- 172.21.15.12:0/1463296828 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f70037070 con 0x7f1f7c09d030 2024-09-11T01:37:43.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.997+0000 7f1f7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:43.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.997+0000 7f1f7affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:37:43.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.997+0000 7f1f7affd640 10 monclient: dump: 2024-09-11T01:37:43.007 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:43.007 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.007 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.007 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.007 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:43.008 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.008 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:43.008 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:43.008 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:43.008 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.997+0000 7f1f7affd640 10 monclient: _finish_auth 0 2024-09-11T01:37:43.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.997+0000 7f1f7affd640 10 monclient: _check_auth_tickets 2024-09-11T01:37:43.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.997+0000 7f1f7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:43.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.997+0000 7f1f7affd640 1 -- 172.21.15.12:0/1463296828 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f70032640 con 0x7f1f7c09d030 2024-09-11T01:37:43.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.997+0000 7f1f7affd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:43.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.997+0000 7f1f7affd640 1 -- 172.21.15.12:0/1463296828 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f70032a10 con 0x7f1f7c09d030 2024-09-11T01:37:43.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.997+0000 7f1f82652640 5 monclient: authenticate success, global_id 94131 2024-09-11T01:37:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.997+0000 7f1f82652640 10 monclient: get_monmap_and_config success 2024-09-11T01:37:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.997+0000 7f1f82652640 4 set_mon_vals no callback set 2024-09-11T01:37:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.997+0000 7f1f7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.997+0000 7f1f7affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.997+0000 7f1f7affd640 10 monclient: dump: 2024-09-11T01:37:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:43.011 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.997+0000 7f1f82652640 10 monclient: shutdown 2024-09-11T01:37:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.997+0000 7f1f82652640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.997+0000 7f1f82652640 1 -- 172.21.15.12:0/1463296828 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f7c09d030 msgr2=0x7f1f7c13d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.997+0000 7f1f82652640 1 --2- 172.21.15.12:0/1463296828 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f7c09d030 0x7f1f7c13d090 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1f70009870 tx=0x7f1f7002a0a0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.997+0000 7f1f82652640 1 -- 172.21.15.12:0/1463296828 shutdown_connections 2024-09-11T01:37:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.997+0000 7f1f82652640 1 --2- 172.21.15.12:0/1463296828 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f7c09d030 0x7f1f7c13d090 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.997+0000 7f1f82652640 1 -- 172.21.15.12:0/1463296828 >> 172.21.15.12:0/1463296828 conn(0x7f1f7c0a7120 msgr2=0x7f1f7c0a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.997+0000 7f1f82652640 1 -- 172.21.15.12:0/1463296828 shutdown_connections 2024-09-11T01:37:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.997+0000 7f1f82652640 1 -- 172.21.15.12:0/1463296828 wait complete. 2024-09-11T01:37:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.998+0000 7f1f82652640 10 monclient: build_initial_monmap 2024-09-11T01:37:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.998+0000 7f1f82652640 10 monclient: monmap: 2024-09-11T01:37:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:37:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:37:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:37:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:37:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:37:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:37:43.012 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.998+0000 7f1f82652640 1 Processor -- start 2024-09-11T01:37:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.998+0000 7f1f82652640 1 -- start start 2024-09-11T01:37:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.998+0000 7f1f82652640 10 monclient: init 2024-09-11T01:37:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f82652640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:43.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f82652640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:37:43.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f82652640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f7c09d030 0x7f1f7c13f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f82652640 10 monclient(hunting): picked mon.noname-b con 0x7f1f7c09d030 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:37:43.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f82652640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1f7c09d460 0x7f1f7c136ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f82652640 10 monclient(hunting): picked mon.noname-c con 0x7f1f7c09d460 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:43.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f82652640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f7c138230 0x7f1f7c1cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f82652640 10 monclient(hunting): picked mon.noname-a con 0x7f1f7c138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:37:43.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f82652640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f82652640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7c140e80 con 0x7f1f7c138230 2024-09-11T01:37:43.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f82652640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f82652640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7c140d00 con 0x7f1f7c09d030 2024-09-11T01:37:43.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f82652640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f82652640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1f7c141000 con 0x7f1f7c09d460 2024-09-11T01:37:43.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f82652640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:43.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f82652640 10 monclient(hunting): authenticate will time out at 1269.442509s 2024-09-11T01:37:43.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f80bef640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f7c138230 0x7f1f7c1cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f7b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1f7c09d460 0x7f1f7c136ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f7b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1f7c09d460 0x7f1f7c136ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42398/0 (socket says 172.21.15.12:42398) 2024-09-11T01:37:43.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f7b7fe640 1 -- 172.21.15.12:0/3456236660 learned_addr learned my addr 172.21.15.12:0/3456236660 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:43.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f80bef640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f7c138230 0x7f1f7c1cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53862/0 (socket says 172.21.15.12:53862) 2024-09-11T01:37:43.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f7b7fe640 10 monclient(hunting): get_auth_request con 0x7f1f7c09d460 auth_method 0 2024-09-11T01:37:43.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:43.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:43.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f80bef640 10 monclient(hunting): get_auth_request con 0x7f1f7c138230 auth_method 0 2024-09-11T01:37:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f80bef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f80bef640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f80bef640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f7bfff640 1 --2- 172.21.15.12:0/3456236660 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f7c09d030 0x7f1f7c13f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f80bef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f80bef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f80bef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f7bfff640 10 monclient(hunting): get_auth_request con 0x7f1f7c09d030 auth_method 0 2024-09-11T01:37:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f80bef640 10 monclient(hunting): handle_auth_done global_id 94134 payload 824 2024-09-11T01:37:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f80bef640 1 -- 172.21.15.12:0/3456236660 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1f7c09d460 msgr2=0x7f1f7c136ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:42.999+0000 7f1f80bef640 1 --2- 172.21.15.12:0/3456236660 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1f7c09d460 0x7f1f7c136ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.000+0000 7f1f80bef640 1 -- 172.21.15.12:0/3456236660 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f7c09d030 msgr2=0x7f1f7c13f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.000+0000 7f1f80bef640 1 --2- 172.21.15.12:0/3456236660 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f7c09d030 0x7f1f7c13f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.000+0000 7f1f80bef640 10 monclient: _finish_hunting 0 2024-09-11T01:37:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.000+0000 7f1f80bef640 1 monclient: found mon.noname-a 2024-09-11T01:37:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.000+0000 7f1f80bef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.000+0000 7f1f80bef640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:37:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.000+0000 7f1f80bef640 1 -- 172.21.15.12:0/3456236660 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f7c0a9f90 con 0x7f1f7c138230 2024-09-11T01:37:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.000+0000 7f1f7b7fe640 1 --2- 172.21.15.12:0/3456236660 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1f7c09d460 0x7f1f7c136ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:37:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.002+0000 7f1f80bef640 1 -- 172.21.15.12:0/3456236660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f7c138230 msgr2=0x7f1f7c1cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.002+0000 7f1f80bef640 1 --2- 172.21.15.12:0/3456236660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f7c138230 0x7f1f7c1cd1e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c09cfc0 tx=0x7f1f7400cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.002+0000 7f1f78ff9640 1 -- 172.21.15.12:0/3456236660 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f74017070 con 0x7f1f7c138230 2024-09-11T01:37:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.002+0000 7f1f78ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.002+0000 7f1f78ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:37:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.002+0000 7f1f78ff9640 10 monclient: dump: 2024-09-11T01:37:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:43.018 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.002+0000 7f1f78ff9640 10 monclient: _finish_auth 0 2024-09-11T01:37:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.002+0000 7f1f78ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:37:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.002+0000 7f1f78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.002+0000 7f1f82652640 5 monclient: authenticate success, global_id 94134 2024-09-11T01:37:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.002+0000 7f1f78ff9640 1 -- 172.21.15.12:0/3456236660 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f7400a640 con 0x7f1f7c138230 2024-09-11T01:37:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.002+0000 7f1f78ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.002+0000 7f1f78ff9640 1 -- 172.21.15.12:0/3456236660 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7400aa30 con 0x7f1f7c138230 2024-09-11T01:37:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.002+0000 7f1f78ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.002+0000 7f1f78ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.002+0000 7f1f78ff9640 10 monclient: dump: 2024-09-11T01:37:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:43.020 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.020 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:43.020 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:43.020 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:43.020 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.002+0000 7f1f82652640 10 monclient: _renew_subs 2024-09-11T01:37:43.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.002+0000 7f1f7a7fc640 4 set_mon_vals no callback set 2024-09-11T01:37:43.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.002+0000 7f1f82652640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:43.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.002+0000 7f1f82652640 1 -- 172.21.15.12:0/3456236660 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f7c13e6a0 con 0x7f1f7c138230 2024-09-11T01:37:43.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.002+0000 7f1f82652640 10 monclient: _renew_subs 2024-09-11T01:37:43.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.002+0000 7f1f82652640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:43.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.002+0000 7f1f82652640 1 -- 172.21.15.12:0/3456236660 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f7c140440 con 0x7f1f7c138230 2024-09-11T01:37:43.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.003+0000 7f1f82652640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:43.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.003+0000 7f1f82652640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:43.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.003+0000 7f1f82652640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:43.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.003+0000 7f1f78ff9640 1 -- 172.21.15.12:0/3456236660 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f1f7400ac10 con 0x7f1f7c138230 2024-09-11T01:37:43.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.003+0000 7f1f82652640 1 -- 172.21.15.12:0/3456236660 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f7c09db10 con 0x7f1f7c138230 2024-09-11T01:37:43.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.004+0000 7f1f78ff9640 1 --2- 172.21.15.12:0/3456236660 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1f6803e210 0x7f1f680406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.004+0000 7f1f78ff9640 1 -- 172.21.15.12:0/3456236660 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(287..287 src has 186..287) ==== 4815+0+0 (secure 0 0 0) 0x7f1f74052630 con 0x7f1f7c138230 2024-09-11T01:37:43.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.004+0000 7f1f7bfff640 1 --2- 172.21.15.12:0/3456236660 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1f6803e210 0x7f1f680406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.004+0000 7f1f7bfff640 10 monclient: get_auth_request con 0x7f1f6803e210 auth_method 0 2024-09-11T01:37:43.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.007+0000 7f1f7bfff640 1 --2- 172.21.15.12:0/3456236660 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1f6803e210 0x7f1f680406d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c004f60 tx=0x7f1f6c007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:43.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.008+0000 7f1f78ff9640 1 -- 172.21.15.12:0/3456236660 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1f7c09db10 con 0x7f1f7c138230 2024-09-11T01:37:43.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.008+0000 7f1f78ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:43.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.008+0000 7f1f78ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:37:43.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.266+0000 7f1f82652640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:37:43.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.266+0000 7f1f82652640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:37:43.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.266+0000 7f1f82652640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:43.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.266+0000 7f1f82652640 1 -- 172.21.15.12:0/3456236660 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1f7c138610 con 0x7f1f7c138230 2024-09-11T01:37:43.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.266+0000 7f1f78ff9640 1 -- 172.21.15.12:0/3456236660 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f1f74051010 con 0x7f1f7c138230 2024-09-11T01:37:43.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.266+0000 7f1f78ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:37:43.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.266+0000 7f1f78ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:37:43.269 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:37:43.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.268+0000 7f1f82652640 1 -- 172.21.15.12:0/3456236660 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1f6803e210 msgr2=0x7f1f680406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.268+0000 7f1f82652640 1 --2- 172.21.15.12:0/3456236660 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1f6803e210 0x7f1f680406d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c004f60 tx=0x7f1f6c007e30 comp rx=0 tx=0).stop 2024-09-11T01:37:43.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.268+0000 7f1f82652640 10 monclient: shutdown 2024-09-11T01:37:43.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.268+0000 7f1f82652640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:43.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.268+0000 7f1f82652640 1 -- 172.21.15.12:0/3456236660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f7c138230 msgr2=0x7f1f7c1cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.268+0000 7f1f82652640 1 --2- 172.21.15.12:0/3456236660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f7c138230 0x7f1f7c1cd1e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c09cfc0 tx=0x7f1f7400cbe0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.268+0000 7f1f82652640 1 -- 172.21.15.12:0/3456236660 shutdown_connections 2024-09-11T01:37:43.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.268+0000 7f1f82652640 1 --2- 172.21.15.12:0/3456236660 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1f6803e210 0x7f1f680406d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.268+0000 7f1f82652640 1 --2- 172.21.15.12:0/3456236660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f7c138230 0x7f1f7c1cd1e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.268+0000 7f1f82652640 1 --2- 172.21.15.12:0/3456236660 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1f7c09d460 0x7f1f7c136ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.268+0000 7f1f82652640 1 --2- 172.21.15.12:0/3456236660 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f7c09d030 0x7f1f7c13f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.268+0000 7f1f82652640 1 -- 172.21.15.12:0/3456236660 >> 172.21.15.12:0/3456236660 conn(0x7f1f7c0a7120 msgr2=0x7f1f7c0a3160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:43.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.268+0000 7f1f82652640 1 -- 172.21.15.12:0/3456236660 shutdown_connections 2024-09-11T01:37:43.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.268+0000 7f1f82652640 1 -- 172.21.15.12:0/3456236660 wait complete. 2024-09-11T01:37:43.281 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:37:43.281 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:37:43.281 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:37:43.281 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:37:43.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.381+0000 7fca54bb4640 10 monclient: get_monmap_and_config 2024-09-11T01:37:43.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.381+0000 7fca54bb4640 10 monclient: build_initial_monmap 2024-09-11T01:37:43.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.381+0000 7fca54bb4640 10 monclient: monmap: 2024-09-11T01:37:43.384 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:43.384 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.384 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:37:43.383061+0000 2024-09-11T01:37:43.384 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:37:43.383061+0000 2024-09-11T01:37:43.384 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:43.384 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.384 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:37:43.384 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:37:43.384 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:37:43.385 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:37:43.385 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:37:43.385 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca54bb4640 1 Processor -- start 2024-09-11T01:37:43.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca54bb4640 1 -- start start 2024-09-11T01:37:43.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca54bb4640 10 monclient: init 2024-09-11T01:37:43.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca54bb4640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:43.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca54bb4640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-11T01:37:43.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca54bb4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fca50074160 0x7fca50074540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca54bb4640 10 monclient(hunting): picked mon.noname-a con 0x7fca50074160 addr v2:172.21.15.12:3300/0 2024-09-11T01:37:43.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca54bb4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fca500a0540 0x7fca50137c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca54bb4640 10 monclient(hunting): picked mon.noname-b con 0x7fca500a0540 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:43.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca54bb4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fca500a2ab0 addr v1:172.21.15.71:6789/0 2024-09-11T01:37:43.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca54bb4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca500a34f0 con 0x7fca500a2ab0 2024-09-11T01:37:43.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca54bb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca54bb4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fca500a0230 con 0x7fca50074160 2024-09-11T01:37:43.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca54bb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca54bb4640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fca50070160 con 0x7fca500a0540 2024-09-11T01:37:43.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca54bb4640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:43.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca54bb4640 10 monclient(hunting): authenticate will time out at 1269.825672s 2024-09-11T01:37:43.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca4e59c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fca50074160 0x7fca50074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca47fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fca500a0540 0x7fca50137c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca4e59c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fca50074160 0x7fca50074540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53874/0 (socket says 172.21.15.12:53874) 2024-09-11T01:37:43.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca4e59c640 1 -- 172.21.15.12:0/3378735593 learned_addr learned my addr 172.21.15.12:0/3378735593 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:43.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca4e59c640 10 monclient(hunting): get_auth_request con 0x7fca50074160 auth_method 0 2024-09-11T01:37:43.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca47fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fca500a0540 0x7fca50137c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42408/0 (socket says 172.21.15.12:42408) 2024-09-11T01:37:43.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca4e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:43.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca4e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:43.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca4e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:43.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca47fff640 10 monclient(hunting): get_auth_request con 0x7fca500a0540 auth_method 0 2024-09-11T01:37:43.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca47fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca4e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca4e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca4e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.382+0000 7fca47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.383+0000 7fca4e59c640 10 monclient(hunting): handle_auth_done global_id 94137 payload 274 2024-09-11T01:37:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.383+0000 7fca4e59c640 1 -- 172.21.15.12:0/3378735593 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fca500a0540 msgr2=0x7fca50137c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.383+0000 7fca4e59c640 1 --2- 172.21.15.12:0/3378735593 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fca500a0540 0x7fca50137c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.383+0000 7fca4e59c640 1 -- 172.21.15.12:0/3378735593 >> v1:172.21.15.71:6789/0 conn(0x7fca500a2ab0 legacy=0x7fca50138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.383+0000 7fca4e59c640 10 monclient: _finish_hunting 0 2024-09-11T01:37:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.383+0000 7fca4e59c640 1 monclient: found mon.noname-a 2024-09-11T01:37:43.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.383+0000 7fca4e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:43.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.383+0000 7fca4e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:37:43.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.383+0000 7fca4e59c640 1 -- 172.21.15.12:0/3378735593 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca50070440 con 0x7fca50074160 2024-09-11T01:37:43.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.383+0000 7fca47fff640 1 --2- 172.21.15.12:0/3378735593 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fca500a0540 0x7fca50137c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:37:43.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.385+0000 7fca4e59c640 1 -- 172.21.15.12:0/3378735593 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca50074160 msgr2=0x7fca50074540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:43.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.385+0000 7fca4e59c640 1 --2- 172.21.15.12:0/3378735593 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca50074160 0x7fca50074540 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fca40009870 tx=0x7fca4002a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e79dcde1b3fba766 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:43.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca4dd9b640 1 -- 172.21.15.12:0/3378735593 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca40037070 con 0x7fca50074160 2024-09-11T01:37:43.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca4dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:43.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca4dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:37:43.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca4dd9b640 10 monclient: dump: 2024-09-11T01:37:43.398 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:43.398 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.398 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.398 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.398 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:43.398 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.398 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:43.398 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:43.398 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:43.398 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca4dd9b640 10 monclient: _finish_auth 0 2024-09-11T01:37:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca4dd9b640 10 monclient: _check_auth_tickets 2024-09-11T01:37:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca4dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca54bb4640 5 monclient: authenticate success, global_id 94137 2024-09-11T01:37:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca4dd9b640 1 -- 172.21.15.12:0/3378735593 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca40032440 con 0x7fca50074160 2024-09-11T01:37:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca54bb4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:37:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca4dd9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca54bb4640 10 monclient: get_monmap_and_config success 2024-09-11T01:37:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca54bb4640 4 set_mon_vals no callback set 2024-09-11T01:37:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca4dd9b640 1 -- 172.21.15.12:0/3378735593 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca40032810 con 0x7fca50074160 2024-09-11T01:37:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca4dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca4dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca4dd9b640 10 monclient: dump: 2024-09-11T01:37:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:43.400 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca54bb4640 10 monclient: shutdown 2024-09-11T01:37:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca54bb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca54bb4640 1 -- 172.21.15.12:0/3378735593 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca50074160 msgr2=0x7fca50074540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca54bb4640 1 --2- 172.21.15.12:0/3378735593 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca50074160 0x7fca50074540 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fca40009870 tx=0x7fca4002a0a0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca54bb4640 1 -- 172.21.15.12:0/3378735593 shutdown_connections 2024-09-11T01:37:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca54bb4640 1 --2- 172.21.15.12:0/3378735593 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fca500a0540 0x7fca50137c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca54bb4640 1 --2- 172.21.15.12:0/3378735593 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca50074160 0x7fca50074540 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca54bb4640 1 -- 172.21.15.12:0/3378735593 >> 172.21.15.12:0/3378735593 conn(0x7fca500a36c0 msgr2=0x7fca500a3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca54bb4640 1 -- 172.21.15.12:0/3378735593 shutdown_connections 2024-09-11T01:37:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.386+0000 7fca54bb4640 1 -- 172.21.15.12:0/3378735593 wait complete. 2024-09-11T01:37:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.387+0000 7fca54bb4640 10 monclient: build_initial_monmap 2024-09-11T01:37:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.387+0000 7fca54bb4640 10 monclient: monmap: 2024-09-11T01:37:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:37:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:37:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:37:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:37:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:37:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:37:43.402 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.387+0000 7fca54bb4640 1 Processor -- start 2024-09-11T01:37:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.387+0000 7fca54bb4640 1 -- start start 2024-09-11T01:37:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.387+0000 7fca54bb4640 10 monclient: init 2024-09-11T01:37:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca54bb4640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca54bb4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:37:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca54bb4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fca50074160 0x7fca5013f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca54bb4640 10 monclient(hunting): picked mon.noname-c con 0x7fca50074160 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca54bb4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca500a0540 0x7fca5013f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca54bb4640 10 monclient(hunting): picked mon.noname-a con 0x7fca500a0540 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:37:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca54bb4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca500a2ab0 0x7fca501cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca54bb4640 10 monclient(hunting): picked mon.noname-b con 0x7fca500a2ab0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:37:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca54bb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca54bb4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fca50138870 con 0x7fca500a0540 2024-09-11T01:37:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca54bb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca54bb4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fca50068c20 con 0x7fca500a2ab0 2024-09-11T01:37:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca54bb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca54bb4640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fca501389f0 con 0x7fca50074160 2024-09-11T01:37:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca54bb4640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca54bb4640 10 monclient(hunting): authenticate will time out at 1269.831409s 2024-09-11T01:37:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca4e59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fca50074160 0x7fca5013f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca4e59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fca50074160 0x7fca5013f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42414/0 (socket says 172.21.15.12:42414) 2024-09-11T01:37:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca4e59c640 1 -- 172.21.15.12:0/201381758 learned_addr learned my addr 172.21.15.12:0/201381758 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca4e59c640 10 monclient(hunting): get_auth_request con 0x7fca50074160 auth_method 0 2024-09-11T01:37:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca4e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca4e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca4e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca47fff640 1 --2- 172.21.15.12:0/201381758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca500a0540 0x7fca5013f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca47fff640 10 monclient(hunting): get_auth_request con 0x7fca500a0540 auth_method 0 2024-09-11T01:37:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca47fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca4e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca4e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca4e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca4ed9d640 1 --2- 172.21.15.12:0/201381758 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca500a2ab0 0x7fca501cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca4e59c640 10 monclient(hunting): handle_auth_done global_id 84227 payload 824 2024-09-11T01:37:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca4e59c640 1 -- 172.21.15.12:0/201381758 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca500a2ab0 msgr2=0x7fca501cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca4e59c640 1 --2- 172.21.15.12:0/201381758 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca500a2ab0 0x7fca501cd0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca4e59c640 1 -- 172.21.15.12:0/201381758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca500a0540 msgr2=0x7fca5013f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca4e59c640 1 --2- 172.21.15.12:0/201381758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca500a0540 0x7fca5013f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca4e59c640 10 monclient: _finish_hunting 0 2024-09-11T01:37:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca4e59c640 1 monclient: found mon.noname-c 2024-09-11T01:37:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca4e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca4e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:37:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca4e59c640 1 -- 172.21.15.12:0/201381758 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca501354c0 con 0x7fca50074160 2024-09-11T01:37:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca4ed9d640 10 monclient: get_auth_request con 0x7fca500a2ab0 auth_method 0 2024-09-11T01:37:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.388+0000 7fca4ed9d640 1 --2- 172.21.15.12:0/201381758 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca500a2ab0 0x7fca501cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:37:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.391+0000 7fca4e59c640 1 -- 172.21.15.12:0/201381758 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fca50074160 msgr2=0x7fca5013f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.391+0000 7fca4e59c640 1 --2- 172.21.15.12:0/201381758 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fca50074160 0x7fca5013f1e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fca400090b0 tx=0x7fca400046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.391+0000 7fca46ffd640 1 -- 172.21.15.12:0/201381758 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca40037040 con 0x7fca50074160 2024-09-11T01:37:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.391+0000 7fca46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.391+0000 7fca46ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:37:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.391+0000 7fca46ffd640 10 monclient: dump: 2024-09-11T01:37:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:43.407 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.391+0000 7fca46ffd640 10 monclient: _finish_auth 0 2024-09-11T01:37:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.391+0000 7fca46ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:37:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.391+0000 7fca46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.391+0000 7fca46ffd640 1 -- 172.21.15.12:0/201381758 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca40042070 con 0x7fca50074160 2024-09-11T01:37:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.391+0000 7fca54bb4640 5 monclient: authenticate success, global_id 84227 2024-09-11T01:37:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.391+0000 7fca54bb4640 10 monclient: _renew_subs 2024-09-11T01:37:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.391+0000 7fca54bb4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.391+0000 7fca54bb4640 1 -- 172.21.15.12:0/201381758 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca50075000 con 0x7fca50074160 2024-09-11T01:37:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.391+0000 7fca46ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.391+0000 7fca46ffd640 1 -- 172.21.15.12:0/201381758 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4003b820 con 0x7fca50074160 2024-09-11T01:37:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.391+0000 7fca46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.391+0000 7fca46ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.391+0000 7fca46ffd640 10 monclient: dump: 2024-09-11T01:37:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.409 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:43.409 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:43.409 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:43.409 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.391+0000 7fca4d59a640 4 set_mon_vals no callback set 2024-09-11T01:37:43.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.391+0000 7fca54bb4640 10 monclient: _renew_subs 2024-09-11T01:37:43.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.391+0000 7fca54bb4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:43.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.391+0000 7fca54bb4640 1 -- 172.21.15.12:0/201381758 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fca5013bc90 con 0x7fca50074160 2024-09-11T01:37:43.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.392+0000 7fca54bb4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:43.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.392+0000 7fca54bb4640 1 --2- 172.21.15.12:0/201381758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca501cd610 0x7fca501cd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.392+0000 7fca54bb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.392+0000 7fca54bb4640 1 -- 172.21.15.12:0/201381758 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fca501cdf30 con 0x7fca501cd610 2024-09-11T01:37:43.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.392+0000 7fca47fff640 1 --2- 172.21.15.12:0/201381758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca501cd610 0x7fca501cd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.392+0000 7fca47fff640 10 monclient: get_auth_request con 0x7fca501cd610 auth_method 0 2024-09-11T01:37:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.392+0000 7fca47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.392+0000 7fca47fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.392+0000 7fca47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.392+0000 7fca46ffd640 1 -- 172.21.15.12:0/201381758 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fca4003a030 con 0x7fca50074160 2024-09-11T01:37:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.392+0000 7fca47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.392+0000 7fca47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.392+0000 7fca47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.392+0000 7fca47fff640 10 monclient(hunting): handle_auth_done global_id 94143 payload 824 2024-09-11T01:37:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.392+0000 7fca46ffd640 1 --2- 172.21.15.12:0/201381758 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fca3003e210 0x7fca300406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.392+0000 7fca47fff640 1 -- 172.21.15.12:0/201381758 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fca50074540 con 0x7fca501cd610 2024-09-11T01:37:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.392+0000 7fca46ffd640 1 -- 172.21.15.12:0/201381758 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(287..287 src has 186..287) ==== 4815+0+0 (secure 0 0 0) 0x7fca40072ff0 con 0x7fca50074160 2024-09-11T01:37:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.392+0000 7fca4ed9d640 1 --2- 172.21.15.12:0/201381758 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fca3003e210 0x7fca300406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.392+0000 7fca4ed9d640 10 monclient: get_auth_request con 0x7fca3003e210 auth_method 0 2024-09-11T01:37:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.394+0000 7fca47fff640 1 -- 172.21.15.12:0/201381758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca501cd610 msgr2=0x7fca501cd9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.394+0000 7fca47fff640 1 --2- 172.21.15.12:0/201381758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca501cd610 0x7fca501cd9f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fca501ce1b0 tx=0x7fca38011d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.394+0000 7fca46ffd640 1 -- 172.21.15.12:0/201381758 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca38019070 con 0x7fca501cd610 2024-09-11T01:37:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.394+0000 7fca46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.394+0000 7fca46ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.394+0000 7fca46ffd640 10 monclient: dump: 2024-09-11T01:37:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:43.412 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.395+0000 7fca46ffd640 1 -- 172.21.15.12:0/201381758 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fca3800f890 con 0x7fca501cd610 2024-09-11T01:37:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.395+0000 7fca46ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.395+0000 7fca46ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:37:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.395+0000 7fca46ffd640 1 -- 172.21.15.12:0/201381758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca501cd610 msgr2=0x7fca501cd9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.395+0000 7fca46ffd640 1 --2- 172.21.15.12:0/201381758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca501cd610 0x7fca501cd9f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fca501ce1b0 tx=0x7fca38011d40 comp rx=0 tx=0).stop 2024-09-11T01:37:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.395+0000 7fca46ffd640 1 -- 172.21.15.12:0/201381758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca501cd610 msgr2=0x7fca501cd9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:37:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.395+0000 7fca46ffd640 1 --2- 172.21.15.12:0/201381758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca501cd610 0x7fca501cd9f0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fca501ce1b0 tx=0x7fca38011d40 comp rx=0 tx=0).stop 2024-09-11T01:37:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.395+0000 7fca4ed9d640 1 --2- 172.21.15.12:0/201381758 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fca3003e210 0x7fca300406d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fca5013ab90 tx=0x7fca48009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.401+0000 7fca54bb4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:37:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.401+0000 7fca54bb4640 1 --2- 172.21.15.12:0/201381758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca5012f540 0x7fca5012f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.401+0000 7fca54bb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.401+0000 7fca54bb4640 1 -- 172.21.15.12:0/201381758 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fca5013b7d0 con 0x7fca5012f540 2024-09-11T01:37:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.401+0000 7fca47fff640 1 --2- 172.21.15.12:0/201381758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca5012f540 0x7fca5012f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.401+0000 7fca47fff640 10 monclient: get_auth_request con 0x7fca5012f540 auth_method 0 2024-09-11T01:37:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.401+0000 7fca47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.401+0000 7fca47fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.401+0000 7fca47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.401+0000 7fca47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.401+0000 7fca47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.401+0000 7fca47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.401+0000 7fca47fff640 10 monclient(hunting): handle_auth_done global_id 94146 payload 824 2024-09-11T01:37:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.401+0000 7fca47fff640 1 -- 172.21.15.12:0/201381758 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fca5013a490 con 0x7fca5012f540 2024-09-11T01:37:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.403+0000 7fca47fff640 1 -- 172.21.15.12:0/201381758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca5012f540 msgr2=0x7fca5012f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.403+0000 7fca47fff640 1 --2- 172.21.15.12:0/201381758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca5012f540 0x7fca5012f920 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fca501ce1b0 tx=0x7fca380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.404+0000 7fca46ffd640 1 -- 172.21.15.12:0/201381758 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca38019040 con 0x7fca5012f540 2024-09-11T01:37:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.404+0000 7fca46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.404+0000 7fca46ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.404+0000 7fca46ffd640 10 monclient: dump: 2024-09-11T01:37:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:43.416 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.404+0000 7fca46ffd640 1 -- 172.21.15.12:0/201381758 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7fca38003d90 con 0x7fca5012f540 2024-09-11T01:37:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.404+0000 7fca46ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:37:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.404+0000 7fca46ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:37:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.404+0000 7fca46ffd640 1 -- 172.21.15.12:0/201381758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca5012f540 msgr2=0x7fca5012f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.404+0000 7fca46ffd640 1 --2- 172.21.15.12:0/201381758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca5012f540 0x7fca5012f920 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fca501ce1b0 tx=0x7fca380046c0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.404+0000 7fca46ffd640 1 -- 172.21.15.12:0/201381758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca5012f540 msgr2=0x7fca5012f920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:37:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.404+0000 7fca46ffd640 1 --2- 172.21.15.12:0/201381758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fca5012f540 0x7fca5012f920 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fca501ce1b0 tx=0x7fca380046c0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.417 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:37:43.417 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:37:43.417 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:37:43.417 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:37:43.417 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 88, 2024-09-11T01:37:43.417 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 9500, 2024-09-11T01:37:43.417 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:37:43.417 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:37:43.417 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:37:43.417 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:37:43.417 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:43.417 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:37:43.418 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:37:43.418 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:37:43.418 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:37:43.418 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:43.418 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:43.418 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:43.418 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:43.418 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:43.418 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:43.418 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:43.418 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:43.418 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:43.418 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:43.419 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:43.419 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:43.419 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:37:43.419 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:37:43.419 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:43.419 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:43.419 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:43.419 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:43.419 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:43.419 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:43.419 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:43.419 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:43.419 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:43.419 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:43.420 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:43.420 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:43.420 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.420 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:37:43.420 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:37:43.420 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:37:43.420 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:37:43.420 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:37:43.420 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:37:43.420 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:37:43.420 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:37:43.420 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:37:43.421 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:37:43.421 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:37:43.421 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:37:43.421 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:37:43.421 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:37:43.421 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:37:43.421 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:37:43.421 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:37:43.421 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:43.421 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:43.421 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:43.421 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:43.421 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:43.422 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:43.422 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:43.422 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:43.422 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:43.422 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:43.422 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:43.422 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:43.422 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:37:43.422 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.422 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:37:43.422 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.422 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:37:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:37:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:37:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:37:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:43.424 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.424 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:37:43.424 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:37:43.424 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:43.424 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:43.424 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:43.424 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.424 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.424 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:37:43.424 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:37:43.424 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:43.424 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:43.424 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.425 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:43.425 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:37:43.425 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:43.425 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.425 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.425 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:43.425 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:37:43.425 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:43.425 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:43.425 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:43.425 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.425 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:37:43.426 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:37:43.426 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:43.426 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:43.426 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:43.426 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.426 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.426 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:37:43.426 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:37:43.426 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:43.426 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:43.426 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.426 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:43.426 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:37:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:37:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:37:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:37:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:43.428 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:43.428 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:43.428 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.428 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:37:43.428 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:37:43.428 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.428 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:43.428 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:43.428 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:37:43.428 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:43.428 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:43.428 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:37:43.428 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.429 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:43.429 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:43.429 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:37:43.429 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:43.429 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:43.429 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.429 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:37:43.429 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:37:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.404+0000 7fca54bb4640 1 -- 172.21.15.12:0/201381758 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fca3003e210 msgr2=0x7fca300406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.404+0000 7fca54bb4640 1 --2- 172.21.15.12:0/201381758 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fca3003e210 0x7fca300406d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fca5013ab90 tx=0x7fca48009040 comp rx=0 tx=0).stop 2024-09-11T01:37:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.404+0000 7fca54bb4640 10 monclient: shutdown 2024-09-11T01:37:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.404+0000 7fca54bb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.404+0000 7fca54bb4640 1 -- 172.21.15.12:0/201381758 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fca50074160 msgr2=0x7fca5013f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.404+0000 7fca4ed9d640 1 -- 172.21.15.12:0/201381758 reap_dead start 2024-09-11T01:37:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.404+0000 7fca54bb4640 1 --2- 172.21.15.12:0/201381758 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fca50074160 0x7fca5013f1e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fca400090b0 tx=0x7fca400046c0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.404+0000 7fca54bb4640 1 -- 172.21.15.12:0/201381758 shutdown_connections 2024-09-11T01:37:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.404+0000 7fca54bb4640 1 --2- 172.21.15.12:0/201381758 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fca50074160 0x7fca5013f1e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.404+0000 7fca54bb4640 1 -- 172.21.15.12:0/201381758 >> 172.21.15.12:0/201381758 conn(0x7fca500a36c0 msgr2=0x7fca50136880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.404+0000 7fca54bb4640 1 -- 172.21.15.12:0/201381758 shutdown_connections 2024-09-11T01:37:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.404+0000 7fca54bb4640 1 -- 172.21.15.12:0/201381758 wait complete. 2024-09-11T01:37:43.432 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:37:43.432 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:37:43.432 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:37:43.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.531+0000 7f0282cc6640 10 monclient: get_monmap_and_config 2024-09-11T01:37:43.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.531+0000 7f0282cc6640 10 monclient: build_initial_monmap 2024-09-11T01:37:43.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.531+0000 7f0282cc6640 10 monclient: monmap: 2024-09-11T01:37:43.534 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:43.534 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.534 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:37:43.533264+0000 2024-09-11T01:37:43.534 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:37:43.533264+0000 2024-09-11T01:37:43.534 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:43.534 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.534 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:37:43.535 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:37:43.535 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:37:43.535 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:37:43.535 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:37:43.535 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.532+0000 7f0282cc6640 1 Processor -- start 2024-09-11T01:37:43.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.532+0000 7f0282cc6640 1 -- start start 2024-09-11T01:37:43.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.532+0000 7f0282cc6640 10 monclient: init 2024-09-11T01:37:43.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.532+0000 7f0282cc6640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.532+0000 7f0282cc6640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-11T01:37:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.532+0000 7f0282cc6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f027c0a03a0 addr v1:172.21.15.71:6789/0 2024-09-11T01:37:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.532+0000 7f0282cc6640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f027c0a2530 0x7f027c09f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.532+0000 7f0282cc6640 10 monclient(hunting): picked mon.noname-a con 0x7f027c0a2530 addr v2:172.21.15.12:3300/0 2024-09-11T01:37:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.532+0000 7f0282cc6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c0a13a0 0x7f027c13baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.532+0000 7f0282cc6640 10 monclient(hunting): picked mon.noname-b con 0x7f027c0a13a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.532+0000 7f0282cc6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f027c09e510 con 0x7f027c0a03a0 2024-09-11T01:37:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.532+0000 7f0282cc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.532+0000 7f0282cc6640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f027c0a29b0 con 0x7f027c0a2530 2024-09-11T01:37:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.532+0000 7f0282cc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.532+0000 7f0282cc6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f027c0a0780 con 0x7f027c0a13a0 2024-09-11T01:37:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.532+0000 7f0282cc6640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.532+0000 7f0282cc6640 10 monclient(hunting): authenticate will time out at 1269.975891s 2024-09-11T01:37:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.532+0000 7f027bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f027c0a2530 0x7f027c09f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f0281263640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c0a13a0 0x7f027c13baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f027bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f027c0a2530 0x7f027c09f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53904/0 (socket says 172.21.15.12:53904) 2024-09-11T01:37:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f027bfff640 1 -- 172.21.15.12:0/112016618 learned_addr learned my addr 172.21.15.12:0/112016618 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f027bfff640 10 monclient(hunting): get_auth_request con 0x7f027c0a2530 auth_method 0 2024-09-11T01:37:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f027bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f027bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f027bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f0281263640 10 monclient(hunting): get_auth_request con 0x7f027c0a13a0 auth_method 0 2024-09-11T01:37:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f0281263640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f0281263640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f0281263640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f027bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:43.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f027bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:43.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f027bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:43.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f0281263640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:43.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f0281263640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:43.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f0281263640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:43.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f0281263640 10 monclient(hunting): handle_auth_done global_id 84230 payload 274 2024-09-11T01:37:43.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f0281263640 1 -- 172.21.15.12:0/112016618 >> v2:172.21.15.12:3300/0 conn(0x7f027c0a2530 msgr2=0x7f027c09f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:43.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f0281263640 1 --2- 172.21.15.12:0/112016618 >> v2:172.21.15.12:3300/0 conn(0x7f027c0a2530 0x7f027c09f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f0281263640 1 -- 172.21.15.12:0/112016618 >> v1:172.21.15.71:6789/0 conn(0x7f027c0a03a0 legacy=0x7f027c0a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:43.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f0281263640 10 monclient: _finish_hunting 0 2024-09-11T01:37:43.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f0281263640 1 monclient: found mon.noname-b 2024-09-11T01:37:43.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f0281263640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:43.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f0281263640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:37:43.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f0281263640 1 -- 172.21.15.12:0/112016618 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f027c09edb0 con 0x7f027c0a13a0 2024-09-11T01:37:43.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.533+0000 7f027bfff640 1 --2- 172.21.15.12:0/112016618 >> v2:172.21.15.12:3300/0 conn(0x7f027c0a2530 0x7f027c09f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:37:43.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.536+0000 7f0281263640 1 -- 172.21.15.12:0/112016618 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c0a13a0 msgr2=0x7f027c13baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:43.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.536+0000 7f0281263640 1 --2- 172.21.15.12:0/112016618 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c0a13a0 0x7f027c13baa0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f02740097a0 tx=0x7f0274029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94a24f03055bc3df server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:43.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.536+0000 7f027b7fe640 1 -- 172.21.15.12:0/112016618 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0274037070 con 0x7f027c0a13a0 2024-09-11T01:37:43.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.536+0000 7f027b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:43.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.536+0000 7f027b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:37:43.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.536+0000 7f027b7fe640 10 monclient: dump: 2024-09-11T01:37:43.547 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:43.547 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.547 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.547 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.547 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:43.547 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.547 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:43.547 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:43.547 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:43.547 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.536+0000 7f027b7fe640 10 monclient: _finish_auth 0 2024-09-11T01:37:43.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.536+0000 7f027b7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:37:43.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.536+0000 7f027b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:43.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.536+0000 7f027b7fe640 1 -- 172.21.15.12:0/112016618 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0274032440 con 0x7f027c0a13a0 2024-09-11T01:37:43.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.536+0000 7f0282cc6640 5 monclient: authenticate success, global_id 84230 2024-09-11T01:37:43.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.536+0000 7f0282cc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:37:43.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.536+0000 7f027b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:43.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.536+0000 7f0282cc6640 10 monclient: get_monmap_and_config success 2024-09-11T01:37:43.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.536+0000 7f027b7fe640 1 -- 172.21.15.12:0/112016618 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0274032810 con 0x7f027c0a13a0 2024-09-11T01:37:43.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.536+0000 7f027b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:43.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.536+0000 7f0282cc6640 4 set_mon_vals no callback set 2024-09-11T01:37:43.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.536+0000 7f027b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:43.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.536+0000 7f027b7fe640 10 monclient: dump: 2024-09-11T01:37:43.549 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:43.549 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.549 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.549 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.549 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:43.549 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.549 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:43.549 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:43.549 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:43.549 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.536+0000 7f0282cc6640 10 monclient: shutdown 2024-09-11T01:37:43.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.536+0000 7f0282cc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:43.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.536+0000 7f0282cc6640 1 -- 172.21.15.12:0/112016618 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c0a13a0 msgr2=0x7f027c13baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.536+0000 7f0282cc6640 1 --2- 172.21.15.12:0/112016618 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c0a13a0 0x7f027c13baa0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f02740097a0 tx=0x7f0274029fd0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.537+0000 7f0282cc6640 1 -- 172.21.15.12:0/112016618 shutdown_connections 2024-09-11T01:37:43.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.537+0000 7f0282cc6640 1 --2- 172.21.15.12:0/112016618 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c0a13a0 0x7f027c13baa0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.537+0000 7f0282cc6640 1 --2- 172.21.15.12:0/112016618 >> v2:172.21.15.12:3300/0 conn(0x7f027c0a2530 0x7f027c09f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.537+0000 7f0282cc6640 1 -- 172.21.15.12:0/112016618 >> 172.21.15.12:0/112016618 conn(0x7f027c0a2f70 msgr2=0x7f027c0a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:43.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.537+0000 7f0282cc6640 1 -- 172.21.15.12:0/112016618 shutdown_connections 2024-09-11T01:37:43.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.537+0000 7f0282cc6640 1 -- 172.21.15.12:0/112016618 wait complete. 2024-09-11T01:37:43.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.537+0000 7f0282cc6640 10 monclient: build_initial_monmap 2024-09-11T01:37:43.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.537+0000 7f0282cc6640 10 monclient: monmap: 2024-09-11T01:37:43.550 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:43.550 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:37:43.550 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:37:43.551 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:37:43.551 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:43.551 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.551 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:37:43.551 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:37:43.551 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:37:43.551 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0282cc6640 1 Processor -- start 2024-09-11T01:37:43.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0282cc6640 1 -- start start 2024-09-11T01:37:43.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0282cc6640 10 monclient: init 2024-09-11T01:37:43.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0282cc6640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:43.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0282cc6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:37:43.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0282cc6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c0a03a0 0x7f027c1cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0282cc6640 10 monclient(hunting): picked mon.noname-c con 0x7f027c0a03a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:43.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0282cc6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f027c0a13a0 0x7f027c1bcb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0282cc6640 10 monclient(hunting): picked mon.noname-b con 0x7f027c0a13a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:37:43.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0282cc6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f027c0a2530 0x7f027c1bd060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0282cc6640 10 monclient(hunting): picked mon.noname-a con 0x7f027c0a2530 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:37:43.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0282cc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0282cc6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f027c09ca30 con 0x7f027c0a2530 2024-09-11T01:37:43.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0282cc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0282cc6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f027c0a1a30 con 0x7f027c0a13a0 2024-09-11T01:37:43.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0282cc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0282cc6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f027c09cbb0 con 0x7f027c0a03a0 2024-09-11T01:37:43.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0282cc6640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:43.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0282cc6640 10 monclient(hunting): authenticate will time out at 1269.981814s 2024-09-11T01:37:43.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0280a62640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c0a03a0 0x7f027c1cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0281263640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f027c0a2530 0x7f027c1bd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0280a62640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c0a03a0 0x7f027c1cd0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42428/0 (socket says 172.21.15.12:42428) 2024-09-11T01:37:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0280a62640 1 -- 172.21.15.12:0/2580107170 learned_addr learned my addr 172.21.15.12:0/2580107170 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0280a62640 10 monclient(hunting): get_auth_request con 0x7f027c0a03a0 auth_method 0 2024-09-11T01:37:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0280a62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0280a62640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0280a62640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0281263640 10 monclient(hunting): get_auth_request con 0x7f027c0a2530 auth_method 0 2024-09-11T01:37:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0281263640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0281263640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0281263640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.538+0000 7f0280a62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.539+0000 7f0280a62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.539+0000 7f0280a62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.539+0000 7f0281263640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.539+0000 7f0281263640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.539+0000 7f0281263640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.539+0000 7f027bfff640 1 --2- 172.21.15.12:0/2580107170 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f027c0a13a0 0x7f027c1bcb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.539+0000 7f027bfff640 10 monclient(hunting): get_auth_request con 0x7f027c0a13a0 auth_method 0 2024-09-11T01:37:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.539+0000 7f027bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.539+0000 7f027bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.539+0000 7f027bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.539+0000 7f0280a62640 10 monclient(hunting): handle_auth_done global_id 84233 payload 824 2024-09-11T01:37:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.539+0000 7f0280a62640 1 -- 172.21.15.12:0/2580107170 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f027c0a13a0 msgr2=0x7f027c1bcb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.539+0000 7f0280a62640 1 --2- 172.21.15.12:0/2580107170 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f027c0a13a0 0x7f027c1bcb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.539+0000 7f0280a62640 1 -- 172.21.15.12:0/2580107170 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f027c0a2530 msgr2=0x7f027c1bd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.539+0000 7f0280a62640 1 --2- 172.21.15.12:0/2580107170 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f027c0a2530 0x7f027c1bd060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.539+0000 7f0280a62640 10 monclient: _finish_hunting 0 2024-09-11T01:37:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.539+0000 7f0280a62640 1 monclient: found mon.noname-c 2024-09-11T01:37:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.539+0000 7f0280a62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.539+0000 7f0280a62640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:37:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.539+0000 7f0280a62640 1 -- 172.21.15.12:0/2580107170 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f027c1ae240 con 0x7f027c0a03a0 2024-09-11T01:37:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.541+0000 7f0280a62640 1 -- 172.21.15.12:0/2580107170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c0a03a0 msgr2=0x7f027c1cd0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.541+0000 7f0280a62640 1 --2- 172.21.15.12:0/2580107170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c0a03a0 0x7f027c1cd0a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f026c00c430 tx=0x7f026c00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.541+0000 7f02797fa640 1 -- 172.21.15.12:0/2580107170 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026c0119a0 con 0x7f027c0a03a0 2024-09-11T01:37:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.541+0000 7f02797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.541+0000 7f02797fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:37:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.541+0000 7f02797fa640 10 monclient: dump: 2024-09-11T01:37:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:43.556 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.541+0000 7f02797fa640 10 monclient: _finish_auth 0 2024-09-11T01:37:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.541+0000 7f02797fa640 10 monclient: _check_auth_tickets 2024-09-11T01:37:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.541+0000 7f02797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.541+0000 7f02797fa640 1 -- 172.21.15.12:0/2580107170 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f026c017070 con 0x7f027c0a03a0 2024-09-11T01:37:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.541+0000 7f02797fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.541+0000 7f02797fa640 1 -- 172.21.15.12:0/2580107170 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026c011d80 con 0x7f027c0a03a0 2024-09-11T01:37:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.541+0000 7f0282cc6640 5 monclient: authenticate success, global_id 84233 2024-09-11T01:37:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.541+0000 7f02797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.541+0000 7f02797fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.541+0000 7f02797fa640 10 monclient: dump: 2024-09-11T01:37:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:43.557 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.541+0000 7f027a7fc640 4 set_mon_vals no callback set 2024-09-11T01:37:43.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.541+0000 7f0282cc6640 10 monclient: _renew_subs 2024-09-11T01:37:43.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.541+0000 7f0282cc6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:43.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.541+0000 7f0282cc6640 1 -- 172.21.15.12:0/2580107170 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f027c1acd20 con 0x7f027c0a03a0 2024-09-11T01:37:43.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.541+0000 7f0282cc6640 10 monclient: _renew_subs 2024-09-11T01:37:43.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.541+0000 7f0282cc6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:43.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.541+0000 7f0282cc6640 1 -- 172.21.15.12:0/2580107170 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f027c1ad300 con 0x7f027c0a03a0 2024-09-11T01:37:43.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.542+0000 7f0282cc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:43.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.542+0000 7f0282cc6640 1 --2- 172.21.15.12:0/2580107170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c1adc80 0x7f027c1aaec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.542+0000 7f0282cc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.542+0000 7f0282cc6640 1 -- 172.21.15.12:0/2580107170 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f027c09cfc0 con 0x7f027c1adc80 2024-09-11T01:37:43.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.542+0000 7f027bfff640 1 --2- 172.21.15.12:0/2580107170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c1adc80 0x7f027c1aaec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.542+0000 7f027bfff640 10 monclient: get_auth_request con 0x7f027c1adc80 auth_method 0 2024-09-11T01:37:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.542+0000 7f027bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.542+0000 7f027bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.542+0000 7f027bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.542+0000 7f027bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.542+0000 7f027bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.542+0000 7f02797fa640 1 -- 172.21.15.12:0/2580107170 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f026c01c460 con 0x7f027c0a03a0 2024-09-11T01:37:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.542+0000 7f027bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.542+0000 7f027bfff640 10 monclient(hunting): handle_auth_done global_id 84236 payload 824 2024-09-11T01:37:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.542+0000 7f027bfff640 1 -- 172.21.15.12:0/2580107170 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f027c1ac1c0 con 0x7f027c1adc80 2024-09-11T01:37:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.543+0000 7f02797fa640 1 --2- 172.21.15.12:0/2580107170 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f025c03e210 0x7f025c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.543+0000 7f02797fa640 1 -- 172.21.15.12:0/2580107170 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(287..287 src has 186..287) ==== 4815+0+0 (secure 0 0 0) 0x7f026c052bf0 con 0x7f027c0a03a0 2024-09-11T01:37:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.543+0000 7f0281263640 1 --2- 172.21.15.12:0/2580107170 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f025c03e210 0x7f025c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.543+0000 7f0281263640 10 monclient: get_auth_request con 0x7f025c03e210 auth_method 0 2024-09-11T01:37:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.545+0000 7f027bfff640 1 -- 172.21.15.12:0/2580107170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c1adc80 msgr2=0x7f027c1aaec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.545+0000 7f027bfff640 1 --2- 172.21.15.12:0/2580107170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c1adc80 0x7f027c1aaec0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f027c1adbb0 tx=0x7f02640075c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.545+0000 7f02797fa640 1 -- 172.21.15.12:0/2580107170 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0264007ea0 con 0x7f027c1adc80 2024-09-11T01:37:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.545+0000 7f02797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.545+0000 7f02797fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.545+0000 7f02797fa640 10 monclient: dump: 2024-09-11T01:37:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.563 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:43.563 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:43.563 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:43.563 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.545+0000 7f02797fa640 1 -- 172.21.15.12:0/2580107170 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0264003990 con 0x7f027c1adc80 2024-09-11T01:37:43.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.545+0000 7f02797fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:43.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.545+0000 7f02797fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:37:43.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.545+0000 7f02797fa640 1 -- 172.21.15.12:0/2580107170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c1adc80 msgr2=0x7f027c1aaec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.545+0000 7f02797fa640 1 --2- 172.21.15.12:0/2580107170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c1adc80 0x7f027c1aaec0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f027c1adbb0 tx=0x7f02640075c0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.545+0000 7f02797fa640 1 -- 172.21.15.12:0/2580107170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c1adc80 msgr2=0x7f027c1aaec0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:37:43.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.545+0000 7f02797fa640 1 --2- 172.21.15.12:0/2580107170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c1adc80 0x7f027c1aaec0 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f027c1adbb0 tx=0x7f02640075c0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.545+0000 7f0281263640 1 --2- 172.21.15.12:0/2580107170 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f025c03e210 0x7f025c0406d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f027c06cd50 tx=0x7f027402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:43.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.552+0000 7f0282cc6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:37:43.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.552+0000 7f0282cc6640 1 --2- 172.21.15.12:0/2580107170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c02d440 0x7f027c02d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.552+0000 7f0282cc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.552+0000 7f0282cc6640 1 -- 172.21.15.12:0/2580107170 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f027c1a2d00 con 0x7f027c02d440 2024-09-11T01:37:43.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.552+0000 7f027bfff640 1 --2- 172.21.15.12:0/2580107170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c02d440 0x7f027c02d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.552+0000 7f027bfff640 10 monclient: get_auth_request con 0x7f027c02d440 auth_method 0 2024-09-11T01:37:43.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.552+0000 7f027bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:43.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.552+0000 7f027bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:43.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.552+0000 7f027bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:43.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.552+0000 7f027bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:43.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.552+0000 7f027bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:43.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.552+0000 7f027bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:43.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.552+0000 7f027bfff640 10 monclient(hunting): handle_auth_done global_id 84239 payload 824 2024-09-11T01:37:43.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.552+0000 7f027bfff640 1 -- 172.21.15.12:0/2580107170 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f027c06d480 con 0x7f027c02d440 2024-09-11T01:37:43.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.554+0000 7f027bfff640 1 -- 172.21.15.12:0/2580107170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c02d440 msgr2=0x7f027c02d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:43.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.554+0000 7f027bfff640 1 --2- 172.21.15.12:0/2580107170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c02d440 0x7f027c02d820 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0264015000 tx=0x7f0264004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:43.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.554+0000 7f02797fa640 1 -- 172.21.15.12:0/2580107170 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026401ebd0 con 0x7f027c02d440 2024-09-11T01:37:43.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.554+0000 7f02797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:43.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.554+0000 7f02797fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:43.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.554+0000 7f02797fa640 10 monclient: dump: 2024-09-11T01:37:43.565 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:43.565 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.565 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.565 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.565 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:43.565 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.565 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:43.565 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:43.566 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.555+0000 7f02797fa640 1 -- 172.21.15.12:0/2580107170 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4707 (secure 0 0 0) 0x7f0264009600 con 0x7f027c02d440 2024-09-11T01:37:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.555+0000 7f02797fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:37:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.555+0000 7f02797fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:37:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.555+0000 7f02797fa640 1 -- 172.21.15.12:0/2580107170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c02d440 msgr2=0x7f027c02d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.555+0000 7f02797fa640 1 --2- 172.21.15.12:0/2580107170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c02d440 0x7f027c02d820 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0264015000 tx=0x7f0264004a40 comp rx=0 tx=0).stop 2024-09-11T01:37:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.555+0000 7f02797fa640 1 -- 172.21.15.12:0/2580107170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c02d440 msgr2=0x7f027c02d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:37:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.555+0000 7f02797fa640 1 --2- 172.21.15.12:0/2580107170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c02d440 0x7f027c02d820 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0264015000 tx=0x7f0264004a40 comp rx=0 tx=0).stop 2024-09-11T01:37:43.566 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:37:43.566 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:37:43.567 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:37:43.567 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:37:43.567 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 88, 2024-09-11T01:37:43.567 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 57578, 2024-09-11T01:37:43.567 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:37:43.567 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:37:43.567 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:37:43.567 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:37:43.567 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:43.567 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:37:43.567 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:37:43.567 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:37:43.567 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:37:43.567 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:43.568 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:43.568 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:43.568 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:43.568 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:43.568 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:43.568 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:43.568 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:43.568 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:43.568 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:43.568 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:43.568 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:43.568 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:37:43.568 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:37:43.569 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:43.569 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:43.569 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:43.569 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:43.569 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:43.569 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:43.569 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:43.569 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:43.569 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:43.569 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:43.569 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:43.569 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:43.569 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.569 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:37:43.570 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:37:43.570 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:37:43.570 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:37:43.570 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:37:43.570 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:37:43.570 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:37:43.570 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:37:43.570 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:37:43.570 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:37:43.570 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:37:43.570 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:37:43.570 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:37:43.570 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:37:43.571 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:37:43.571 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:37:43.571 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:37:43.571 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:43.571 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:43.571 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:43.571 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:43.571 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:43.571 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:43.571 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:43.571 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:43.571 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:43.571 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:43.571 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:43.571 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:43.572 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:37:43.572 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.572 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:37:43.572 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.572 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:37:43.572 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:37:43.572 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:43.572 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:43.572 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.572 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:43.572 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:37:43.572 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:43.572 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.573 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.573 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:43.573 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:37:43.573 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:43.573 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:43.573 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:43.573 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.573 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:37:43.573 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:37:43.573 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:43.573 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:43.573 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:43.573 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:37:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:37:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:37:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:37:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:37:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:37:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:37:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:37:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:37:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:37:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:37:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:37:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:37:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:37:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:37:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:37:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:37:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:37:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:43.579 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:37:43.579 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:43.579 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:43.579 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.579 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:37:43.579 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:37:43.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.555+0000 7f0282cc6640 1 -- 172.21.15.12:0/2580107170 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f025c03e210 msgr2=0x7f025c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.555+0000 7f0282cc6640 1 --2- 172.21.15.12:0/2580107170 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f025c03e210 0x7f025c0406d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f027c06cd50 tx=0x7f027402e000 comp rx=0 tx=0).stop 2024-09-11T01:37:43.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.555+0000 7f0282cc6640 10 monclient: shutdown 2024-09-11T01:37:43.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.555+0000 7f0282cc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:43.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.555+0000 7f0282cc6640 1 -- 172.21.15.12:0/2580107170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c0a03a0 msgr2=0x7f027c1cd0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.555+0000 7f0282cc6640 1 --2- 172.21.15.12:0/2580107170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f027c0a03a0 0x7f027c1cd0a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f026c00c430 tx=0x7f026c00c8f0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.555+0000 7f0281263640 1 -- 172.21.15.12:0/2580107170 reap_dead start 2024-09-11T01:37:43.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.555+0000 7f0281263640 1 -- 172.21.15.12:0/2580107170 reap_dead start 2024-09-11T01:37:43.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.555+0000 7f0282cc6640 1 -- 172.21.15.12:0/2580107170 shutdown_connections 2024-09-11T01:37:43.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.555+0000 7f0282cc6640 1 -- 172.21.15.12:0/2580107170 >> 172.21.15.12:0/2580107170 conn(0x7f027c0a2f70 msgr2=0x7f027c0a58b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:43.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.555+0000 7f0282cc6640 1 -- 172.21.15.12:0/2580107170 shutdown_connections 2024-09-11T01:37:43.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.555+0000 7f0282cc6640 1 -- 172.21.15.12:0/2580107170 wait complete. 2024-09-11T01:37:43.581 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:37:43.582 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:37:43.582 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:37:43.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.681+0000 7f47ec233640 10 monclient: get_monmap_and_config 2024-09-11T01:37:43.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.681+0000 7f47ec233640 10 monclient: build_initial_monmap 2024-09-11T01:37:43.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.681+0000 7f47ec233640 10 monclient: monmap: 2024-09-11T01:37:43.684 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:43.684 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.684 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:37:43.682919+0000 2024-09-11T01:37:43.684 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:37:43.682919+0000 2024-09-11T01:37:43.684 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:43.684 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.684 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:37:43.684 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:37:43.684 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:37:43.684 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:37:43.684 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:37:43.685 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.681+0000 7f47ec233640 1 Processor -- start 2024-09-11T01:37:43.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ec233640 1 -- start start 2024-09-11T01:37:43.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ec233640 10 monclient: init 2024-09-11T01:37:43.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ec233640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:43.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ec233640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-11T01:37:43.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ec233640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f47e4137e50 addr v1:172.21.15.71:6789/0 2024-09-11T01:37:43.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ec233640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f47e4138230 addr v1:172.21.15.12:6789/0 2024-09-11T01:37:43.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ec233640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f47e413e390 0x7f47e4140780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ec233640 10 monclient(hunting): picked mon.noname-a con 0x7f47e413e390 addr v2:172.21.15.12:3300/0 2024-09-11T01:37:43.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ec233640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47e40a6580 con 0x7f47e4138230 2024-09-11T01:37:43.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ec233640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47e40a62d0 con 0x7f47e4137e50 2024-09-11T01:37:43.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ec233640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ec233640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f47e409bb40 con 0x7f47e413e390 2024-09-11T01:37:43.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ec233640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:43.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ec233640 10 monclient(hunting): authenticate will time out at 1270.125611s 2024-09-11T01:37:43.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47e97ce640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f47e4138230 0x7f47e4138640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55938/0 (socket says 172.21.15.12:55938) 2024-09-11T01:37:43.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47e97ce640 1 -- 172.21.15.12:0/3438881517 learned_addr learned my addr 172.21.15.12:0/3438881517 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:37:43.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ea7d0640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f47e413e390 0x7f47e4140780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ea7d0640 10 monclient(hunting): get_auth_request con 0x7f47e413e390 auth_method 0 2024-09-11T01:37:43.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ea7d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:43.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ea7d0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:43.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ea7d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:43.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ea7d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:43.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ea7d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:43.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ea7d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:43.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ea7d0640 10 monclient(hunting): handle_auth_done global_id 94155 payload 274 2024-09-11T01:37:43.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ea7d0640 1 -- 172.21.15.12:0/3438881517 >> v1:172.21.15.71:6789/0 conn(0x7f47e4137e50 legacy=0x7f47e409d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:43.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ea7d0640 1 -- 172.21.15.12:0/3438881517 >> v1:172.21.15.12:6789/0 conn(0x7f47e4138230 legacy=0x7f47e4138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:43.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ea7d0640 10 monclient: _finish_hunting 0 2024-09-11T01:37:43.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ea7d0640 1 monclient: found mon.noname-a 2024-09-11T01:37:43.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ea7d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:43.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ea7d0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:37:43.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.682+0000 7f47ea7d0640 1 -- 172.21.15.12:0/3438881517 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47e409c4a0 con 0x7f47e413e390 2024-09-11T01:37:43.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.685+0000 7f47ea7d0640 1 -- 172.21.15.12:0/3438881517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47e413e390 msgr2=0x7f47e4140780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:43.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.685+0000 7f47ea7d0640 1 --2- 172.21.15.12:0/3438881517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47e413e390 0x7f47e4140780 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f47e409bd20 tx=0x7f47dc02a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cc772a7e15588ec server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:43.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.685+0000 7f47e8fcd640 1 -- 172.21.15.12:0/3438881517 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47dc007a30 con 0x7f47e413e390 2024-09-11T01:37:43.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.685+0000 7f47e8fcd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:43.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.685+0000 7f47e8fcd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:37:43.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.685+0000 7f47e8fcd640 10 monclient: dump: 2024-09-11T01:37:43.694 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:43.694 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.694 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.694 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.694 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:43.695 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.695 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:43.695 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:43.695 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:43.695 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.685+0000 7f47e8fcd640 10 monclient: _finish_auth 0 2024-09-11T01:37:43.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.685+0000 7f47e8fcd640 10 monclient: _check_auth_tickets 2024-09-11T01:37:43.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.685+0000 7f47e8fcd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:43.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.685+0000 7f47e8fcd640 1 -- 172.21.15.12:0/3438881517 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47dc007bd0 con 0x7f47e413e390 2024-09-11T01:37:43.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.685+0000 7f47ec233640 5 monclient: authenticate success, global_id 94155 2024-09-11T01:37:43.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.685+0000 7f47ec233640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:37:43.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.685+0000 7f47e8fcd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:43.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.685+0000 7f47ec233640 10 monclient: get_monmap_and_config success 2024-09-11T01:37:43.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.685+0000 7f47ec233640 4 set_mon_vals no callback set 2024-09-11T01:37:43.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.685+0000 7f47e8fcd640 1 -- 172.21.15.12:0/3438881517 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47dc031400 con 0x7f47e413e390 2024-09-11T01:37:43.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.685+0000 7f47e8fcd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:43.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.685+0000 7f47e8fcd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:43.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.685+0000 7f47e8fcd640 10 monclient: dump: 2024-09-11T01:37:43.696 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:43.696 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.696 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.696 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.696 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:43.696 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.696 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:43.697 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:43.697 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:43.697 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.686+0000 7f47ec233640 10 monclient: shutdown 2024-09-11T01:37:43.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.686+0000 7f47ec233640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:43.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.686+0000 7f47ec233640 1 -- 172.21.15.12:0/3438881517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47e413e390 msgr2=0x7f47e4140780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.686+0000 7f47ec233640 1 --2- 172.21.15.12:0/3438881517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47e413e390 0x7f47e4140780 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f47e409bd20 tx=0x7f47dc02a480 comp rx=0 tx=0).stop 2024-09-11T01:37:43.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.686+0000 7f47ec233640 1 -- 172.21.15.12:0/3438881517 shutdown_connections 2024-09-11T01:37:43.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.686+0000 7f47ec233640 1 --2- 172.21.15.12:0/3438881517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47e413e390 0x7f47e4140780 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.686+0000 7f47ec233640 1 -- 172.21.15.12:0/3438881517 >> 172.21.15.12:0/3438881517 conn(0x7f47e40a7020 msgr2=0x7f47e40a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:43.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.686+0000 7f47ec233640 1 -- 172.21.15.12:0/3438881517 shutdown_connections 2024-09-11T01:37:43.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.686+0000 7f47ec233640 1 -- 172.21.15.12:0/3438881517 wait complete. 2024-09-11T01:37:43.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.687+0000 7f47ec233640 10 monclient: build_initial_monmap 2024-09-11T01:37:43.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.687+0000 7f47ec233640 10 monclient: monmap: 2024-09-11T01:37:43.698 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:43.698 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:37:43.698 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:37:43.698 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:37:43.698 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:43.698 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.698 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:37:43.698 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:37:43.698 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:37:43.698 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.687+0000 7f47ec233640 1 Processor -- start 2024-09-11T01:37:43.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.687+0000 7f47ec233640 1 -- start start 2024-09-11T01:37:43.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.687+0000 7f47ec233640 10 monclient: init 2024-09-11T01:37:43.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.687+0000 7f47ec233640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:43.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47ec233640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:37:43.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47ec233640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f47e4137e50 0x7f47e413b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47ec233640 10 monclient(hunting): picked mon.noname-c con 0x7f47e4137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:43.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47ec233640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47e4138230 0x7f47e4136d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47ec233640 10 monclient(hunting): picked mon.noname-a con 0x7f47e4138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:37:43.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47ec233640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e413e390 0x7f47e4139360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47ec233640 10 monclient(hunting): picked mon.noname-b con 0x7f47e413e390 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:37:43.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47ec233640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47ec233640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f47e4140e40 con 0x7f47e4138230 2024-09-11T01:37:43.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47ec233640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47ec233640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f47e4140cc0 con 0x7f47e413e390 2024-09-11T01:37:43.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47ec233640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47ec233640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f47e4140fc0 con 0x7f47e4137e50 2024-09-11T01:37:43.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47ec233640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:43.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47ec233640 10 monclient(hunting): authenticate will time out at 1270.131119s 2024-09-11T01:37:43.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e9fcf640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f47e4137e50 0x7f47e413b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e97ce640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47e4138230 0x7f47e4136d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e9fcf640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f47e4137e50 0x7f47e413b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42454/0 (socket says 172.21.15.12:42454) 2024-09-11T01:37:43.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e97ce640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47e4138230 0x7f47e4136d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53934/0 (socket says 172.21.15.12:53934) 2024-09-11T01:37:43.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e9fcf640 1 -- 172.21.15.12:0/808556950 learned_addr learned my addr 172.21.15.12:0/808556950 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:43.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e9fcf640 10 monclient(hunting): get_auth_request con 0x7f47e4137e50 auth_method 0 2024-09-11T01:37:43.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e9fcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:43.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e9fcf640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:43.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e9fcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:43.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e97ce640 10 monclient(hunting): get_auth_request con 0x7f47e4138230 auth_method 0 2024-09-11T01:37:43.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e97ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:43.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e97ce640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:43.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e97ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:43.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e97ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:43.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e97ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:43.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e97ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:43.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e9fcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:43.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e9fcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:43.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e9fcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:43.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47ea7d0640 1 --2- 172.21.15.12:0/808556950 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e413e390 0x7f47e4139360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47ea7d0640 10 monclient(hunting): get_auth_request con 0x7f47e413e390 auth_method 0 2024-09-11T01:37:43.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47ea7d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:43.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47ea7d0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:43.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47ea7d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:43.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e97ce640 10 monclient(hunting): handle_auth_done global_id 94158 payload 824 2024-09-11T01:37:43.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e97ce640 1 -- 172.21.15.12:0/808556950 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f47e4137e50 msgr2=0x7f47e413b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e97ce640 1 --2- 172.21.15.12:0/808556950 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f47e4137e50 0x7f47e413b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e97ce640 1 -- 172.21.15.12:0/808556950 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e413e390 msgr2=0x7f47e4139360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e97ce640 1 --2- 172.21.15.12:0/808556950 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e413e390 0x7f47e4139360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e97ce640 10 monclient: _finish_hunting 0 2024-09-11T01:37:43.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e97ce640 1 monclient: found mon.noname-a 2024-09-11T01:37:43.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e97ce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:43.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e97ce640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:37:43.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.688+0000 7f47e97ce640 1 -- 172.21.15.12:0/808556950 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47e413c060 con 0x7f47e4138230 2024-09-11T01:37:43.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.690+0000 7f47e97ce640 1 -- 172.21.15.12:0/808556950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47e4138230 msgr2=0x7f47e4136d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:43.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.690+0000 7f47e97ce640 1 --2- 172.21.15.12:0/808556950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47e4138230 0x7f47e4136d60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f47d400c3e0 tx=0x7f47d400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:43.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.690+0000 7f47da7fc640 1 -- 172.21.15.12:0/808556950 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d4017070 con 0x7f47e4138230 2024-09-11T01:37:43.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.690+0000 7f47da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:43.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.690+0000 7f47da7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:37:43.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.690+0000 7f47da7fc640 10 monclient: dump: 2024-09-11T01:37:43.703 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:43.703 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.703 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.703 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.703 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:43.703 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.703 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:43.703 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:43.703 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:43.703 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.690+0000 7f47da7fc640 10 monclient: _finish_auth 0 2024-09-11T01:37:43.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.690+0000 7f47da7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:37:43.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.690+0000 7f47da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:43.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.690+0000 7f47da7fc640 1 -- 172.21.15.12:0/808556950 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47d40070d0 con 0x7f47e4138230 2024-09-11T01:37:43.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.690+0000 7f47da7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:43.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.690+0000 7f47da7fc640 1 -- 172.21.15.12:0/808556950 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d40074c0 con 0x7f47e4138230 2024-09-11T01:37:43.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.690+0000 7f47ec233640 5 monclient: authenticate success, global_id 94158 2024-09-11T01:37:43.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.690+0000 7f47ec233640 10 monclient: _renew_subs 2024-09-11T01:37:43.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.690+0000 7f47ec233640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:43.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.690+0000 7f47ec233640 1 -- 172.21.15.12:0/808556950 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47e413ced0 con 0x7f47e4138230 2024-09-11T01:37:43.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.690+0000 7f47dbfff640 4 set_mon_vals no callback set 2024-09-11T01:37:43.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.690+0000 7f47da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:43.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.690+0000 7f47da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:43.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.690+0000 7f47da7fc640 10 monclient: dump: 2024-09-11T01:37:43.705 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:43.705 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.705 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.705 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.705 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:43.705 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.705 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:43.705 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:43.705 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:43.705 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.691+0000 7f47ec233640 10 monclient: _renew_subs 2024-09-11T01:37:43.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.691+0000 7f47ec233640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:43.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.691+0000 7f47ec233640 1 -- 172.21.15.12:0/808556950 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47e413ab70 con 0x7f47e4138230 2024-09-11T01:37:43.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.691+0000 7f47da7fc640 1 -- 172.21.15.12:0/808556950 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f47d401b3b0 con 0x7f47e4138230 2024-09-11T01:37:43.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.691+0000 7f47ec233640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:43.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.691+0000 7f47ec233640 1 --2- 172.21.15.12:0/808556950 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e41398a0 0x7f47e4139c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.691+0000 7f47ec233640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.691+0000 7f47ec233640 1 -- 172.21.15.12:0/808556950 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f47e413a1c0 con 0x7f47e41398a0 2024-09-11T01:37:43.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.692+0000 7f47da7fc640 1 --2- 172.21.15.12:0/808556950 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f47c803e260 0x7f47c8040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.692+0000 7f47e9fcf640 1 --2- 172.21.15.12:0/808556950 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e41398a0 0x7f47e4139c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.692+0000 7f47da7fc640 1 -- 172.21.15.12:0/808556950 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(287..287 src has 186..287) ==== 4815+0+0 (secure 0 0 0) 0x7f47d40524e0 con 0x7f47e4138230 2024-09-11T01:37:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.692+0000 7f47e9fcf640 10 monclient: get_auth_request con 0x7f47e41398a0 auth_method 0 2024-09-11T01:37:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.692+0000 7f47e9fcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.692+0000 7f47e9fcf640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.692+0000 7f47e9fcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.692+0000 7f47e9fcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.692+0000 7f47e9fcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.692+0000 7f47e9fcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.692+0000 7f47ea7d0640 1 --2- 172.21.15.12:0/808556950 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f47c803e260 0x7f47c8040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.692+0000 7f47ea7d0640 10 monclient: get_auth_request con 0x7f47c803e260 auth_method 0 2024-09-11T01:37:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.692+0000 7f47e9fcf640 10 monclient(hunting): handle_auth_done global_id 74305 payload 824 2024-09-11T01:37:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.692+0000 7f47e9fcf640 1 -- 172.21.15.12:0/808556950 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f47e409d160 con 0x7f47e41398a0 2024-09-11T01:37:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.694+0000 7f47e9fcf640 1 -- 172.21.15.12:0/808556950 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e41398a0 msgr2=0x7f47e4139c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.694+0000 7f47e9fcf640 1 --2- 172.21.15.12:0/808556950 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e41398a0 0x7f47e4139c80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f47e000f7a0 tx=0x7f47e000fc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.694+0000 7f47ea7d0640 1 --2- 172.21.15.12:0/808556950 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f47c803e260 0x7f47c8040720 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f47e409cf30 tx=0x7f47dc02f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.695+0000 7f47da7fc640 1 -- 172.21.15.12:0/808556950 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e0018030 con 0x7f47e41398a0 2024-09-11T01:37:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.695+0000 7f47da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.695+0000 7f47da7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:37:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.695+0000 7f47da7fc640 10 monclient: dump: 2024-09-11T01:37:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:43.712 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.695+0000 7f47da7fc640 1 -- 172.21.15.12:0/808556950 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f47e000fe70 con 0x7f47e41398a0 2024-09-11T01:37:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.695+0000 7f47da7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.695+0000 7f47da7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:37:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.695+0000 7f47da7fc640 1 -- 172.21.15.12:0/808556950 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e41398a0 msgr2=0x7f47e4139c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.695+0000 7f47da7fc640 1 --2- 172.21.15.12:0/808556950 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e41398a0 0x7f47e4139c80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f47e000f7a0 tx=0x7f47e000fc60 comp rx=0 tx=0).stop 2024-09-11T01:37:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.695+0000 7f47da7fc640 1 -- 172.21.15.12:0/808556950 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e41398a0 msgr2=0x7f47e4139c80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:37:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.695+0000 7f47da7fc640 1 --2- 172.21.15.12:0/808556950 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e41398a0 0x7f47e4139c80 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f47e000f7a0 tx=0x7f47e000fc60 comp rx=0 tx=0).stop 2024-09-11T01:37:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.701+0000 7f47ec233640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:37:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.701+0000 7f47ec233640 1 --2- 172.21.15.12:0/808556950 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e40a0430 0x7f47e40a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.701+0000 7f47ec233640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.701+0000 7f47ec233640 1 -- 172.21.15.12:0/808556950 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f47e413c710 con 0x7f47e40a0430 2024-09-11T01:37:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.701+0000 7f47e9fcf640 1 --2- 172.21.15.12:0/808556950 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e40a0430 0x7f47e40a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.701+0000 7f47e9fcf640 10 monclient: get_auth_request con 0x7f47e40a0430 auth_method 0 2024-09-11T01:37:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.701+0000 7f47e9fcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.701+0000 7f47e9fcf640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.701+0000 7f47e9fcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.701+0000 7f47e9fcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.701+0000 7f47e9fcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.701+0000 7f47e9fcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.702+0000 7f47e9fcf640 10 monclient(hunting): handle_auth_done global_id 74308 payload 824 2024-09-11T01:37:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.702+0000 7f47e9fcf640 1 -- 172.21.15.12:0/808556950 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f47e4069950 con 0x7f47e40a0430 2024-09-11T01:37:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.704+0000 7f47e9fcf640 1 -- 172.21.15.12:0/808556950 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e40a0430 msgr2=0x7f47e40a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.704+0000 7f47e9fcf640 1 --2- 172.21.15.12:0/808556950 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e40a0430 0x7f47e40a0810 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f47e000f7a0 tx=0x7f47e0003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.704+0000 7f47da7fc640 1 -- 172.21.15.12:0/808556950 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e001dd80 con 0x7f47e40a0430 2024-09-11T01:37:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.704+0000 7f47da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.704+0000 7f47da7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:37:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.704+0000 7f47da7fc640 10 monclient: dump: 2024-09-11T01:37:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:43.715 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.705+0000 7f47da7fc640 1 -- 172.21.15.12:0/808556950 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7f47e0003f70 con 0x7f47e40a0430 2024-09-11T01:37:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.705+0000 7f47da7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:37:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.705+0000 7f47da7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:37:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.705+0000 7f47da7fc640 1 -- 172.21.15.12:0/808556950 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e40a0430 msgr2=0x7f47e40a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.705+0000 7f47da7fc640 1 --2- 172.21.15.12:0/808556950 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e40a0430 0x7f47e40a0810 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f47e000f7a0 tx=0x7f47e0003760 comp rx=0 tx=0).stop 2024-09-11T01:37:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.705+0000 7f47da7fc640 1 -- 172.21.15.12:0/808556950 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e40a0430 msgr2=0x7f47e40a0810 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:37:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.705+0000 7f47da7fc640 1 --2- 172.21.15.12:0/808556950 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e40a0430 0x7f47e40a0810 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f47e000f7a0 tx=0x7f47e0003760 comp rx=0 tx=0).stop 2024-09-11T01:37:43.715 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:37:43.715 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:37:43.716 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:37:43.716 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:37:43.716 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 88, 2024-09-11T01:37:43.716 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 115484, 2024-09-11T01:37:43.716 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:37:43.716 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:37:43.716 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:37:43.716 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:37:43.716 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:43.716 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:37:43.716 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:37:43.716 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:37:43.716 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:37:43.717 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:43.717 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:43.717 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:43.717 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:43.717 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:43.717 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:43.717 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:43.717 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:43.717 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:43.717 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:43.717 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:43.717 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:43.717 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:37:43.718 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:37:43.718 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:43.718 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:43.718 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:43.718 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:43.718 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:43.718 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:43.718 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:43.718 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:43.718 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:43.718 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:43.718 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:43.718 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:43.719 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.719 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:37:43.719 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:37:43.719 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:37:43.719 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:37:43.719 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:37:43.719 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:37:43.719 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:37:43.719 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:37:43.719 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:37:43.719 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:37:43.719 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:37:43.719 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:37:43.720 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:37:43.720 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:37:43.720 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:37:43.720 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:37:43.720 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:37:43.720 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:43.720 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:43.720 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:43.720 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:43.720 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:43.720 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:43.720 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:43.720 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:43.720 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:43.721 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:43.721 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:43.721 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:43.721 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:37:43.721 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.721 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:37:43.721 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.721 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:37:43.721 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:37:43.721 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:43.721 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:43.721 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.722 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:43.722 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:37:43.722 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:43.722 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.722 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.722 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:43.722 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:37:43.722 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:43.722 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:43.722 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:43.722 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.722 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:37:43.722 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:37:43.722 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:43.723 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:43.723 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:43.723 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.723 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.723 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:37:43.723 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:37:43.723 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:43.723 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:43.723 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.723 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:43.723 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:37:43.723 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:43.723 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.724 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.724 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:43.724 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:37:43.724 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:43.724 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:43.724 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:43.724 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.724 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:37:43.724 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:37:43.724 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:43.724 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:43.724 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:43.724 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.725 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.725 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:37:43.725 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:37:43.725 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:43.725 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:43.725 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.725 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:43.725 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:37:43.725 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:43.725 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.725 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.725 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:43.725 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:37:43.726 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:43.726 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:43.726 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:43.726 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.726 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:37:43.726 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:37:43.726 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:43.726 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:43.726 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:43.726 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:43.726 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:43.726 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.726 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:37:43.727 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:37:43.727 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.727 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:43.727 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:43.727 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:37:43.727 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:43.727 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:43.727 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:37:43.727 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.727 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:43.727 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:43.727 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:37:43.728 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:43.728 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:43.728 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:37:43.728 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.728 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:43.728 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:43.728 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:37:43.728 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:43.728 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:43.728 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:37:43.728 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:43.728 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:43.729 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:43.729 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:37:43.729 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:43.729 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:43.729 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:43.729 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:37:43.729 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:37:43.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.705+0000 7f47ec233640 1 -- 172.21.15.12:0/808556950 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f47c803e260 msgr2=0x7f47c8040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.705+0000 7f47ec233640 1 --2- 172.21.15.12:0/808556950 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f47c803e260 0x7f47c8040720 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f47e409cf30 tx=0x7f47dc02f000 comp rx=0 tx=0).stop 2024-09-11T01:37:43.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.705+0000 7f47ec233640 10 monclient: shutdown 2024-09-11T01:37:43.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.705+0000 7f47ec233640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:43.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.705+0000 7f47ec233640 1 -- 172.21.15.12:0/808556950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47e4138230 msgr2=0x7f47e4136d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.705+0000 7f47ec233640 1 --2- 172.21.15.12:0/808556950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f47e4138230 0x7f47e4136d60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f47d400c3e0 tx=0x7f47d400c8a0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.705+0000 7f47ea7d0640 1 -- 172.21.15.12:0/808556950 reap_dead start 2024-09-11T01:37:43.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.705+0000 7f47ea7d0640 1 -- 172.21.15.12:0/808556950 reap_dead start 2024-09-11T01:37:43.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.705+0000 7f47ec233640 1 -- 172.21.15.12:0/808556950 shutdown_connections 2024-09-11T01:37:43.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.705+0000 7f47ec233640 1 -- 172.21.15.12:0/808556950 >> 172.21.15.12:0/808556950 conn(0x7f47e40a7020 msgr2=0x7f47e413f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:43.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.705+0000 7f47ec233640 1 -- 172.21.15.12:0/808556950 shutdown_connections 2024-09-11T01:37:43.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.705+0000 7f47ec233640 1 -- 172.21.15.12:0/808556950 wait complete. 2024-09-11T01:37:43.731 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:37:43.732 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-11T01:37:43.732 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T01:37:43.732 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-11T01:37:43.732 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-11T01:37:43.732 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-11T01:37:43.732 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:37:43.748 INFO:tasks.ceph.mon.b:Stopped 2024-09-11T01:37:43.748 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:37:43.748 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:37:43.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.846+0000 7ff75aff4640 10 monclient: get_monmap_and_config 2024-09-11T01:37:43.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.846+0000 7ff75aff4640 10 monclient: build_initial_monmap 2024-09-11T01:37:43.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.846+0000 7ff75aff4640 10 monclient: monmap: 2024-09-11T01:37:43.849 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:43.849 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.849 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:37:43.847770+0000 2024-09-11T01:37:43.849 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:37:43.847770+0000 2024-09-11T01:37:43.849 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:43.849 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.849 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:37:43.849 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:37:43.849 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:37:43.849 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:37:43.849 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:37:43.849 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.846+0000 7ff75aff4640 1 Processor -- start 2024-09-11T01:37:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.846+0000 7ff75aff4640 1 -- start start 2024-09-11T01:37:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.846+0000 7ff75aff4640 10 monclient: init 2024-09-11T01:37:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff75aff4640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff75aff4640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-11T01:37:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff75aff4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff754138230 0x7ff754138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff75aff4640 10 monclient(hunting): picked mon.noname-b con 0x7ff754138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff75aff4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff75409d030 addr v1:172.21.15.12:6789/0 2024-09-11T01:37:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff75aff4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff75409d430 addr v1:172.21.15.71:6789/0 2024-09-11T01:37:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff75aff4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7540a5e40 con 0x7ff75409d030 2024-09-11T01:37:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff75aff4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7540a6e30 con 0x7ff75409d430 2024-09-11T01:37:43.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff75aff4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff75aff4640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff75409c9c0 con 0x7ff754138230 2024-09-11T01:37:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff75aff4640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff75aff4640 10 monclient(hunting): authenticate will time out at 1270.290310s 2024-09-11T01:37:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff759591640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7ff75409d430 legacy=0x7ff7540a5730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:37:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff758d90640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff754138230 0x7ff754138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff753fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff75409d030 0x7ff7540a5020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55952/0 (socket says 172.21.15.12:55952) 2024-09-11T01:37:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff758d90640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff754138230 0x7ff754138610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42466/0 (socket says 172.21.15.12:42466) 2024-09-11T01:37:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff753fff640 1 -- 172.21.15.12:0/2042849516 learned_addr learned my addr 172.21.15.12:0/2042849516 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:37:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff758d90640 10 monclient(hunting): get_auth_request con 0x7ff754138230 auth_method 0 2024-09-11T01:37:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff758d90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff758d90640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:43.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff758d90640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:43.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff758d90640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:43.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff758d90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:43.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff758d90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:43.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff758d90640 10 monclient(hunting): handle_auth_done global_id 84245 payload 274 2024-09-11T01:37:43.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff758d90640 1 -- 172.21.15.12:0/2042849516 >> v1:172.21.15.71:6789/0 conn(0x7ff75409d430 legacy=0x7ff7540a5730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:37:43.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff758d90640 1 -- 172.21.15.12:0/2042849516 >> v1:172.21.15.12:6789/0 conn(0x7ff75409d030 legacy=0x7ff7540a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:43.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff758d90640 10 monclient: _finish_hunting 0 2024-09-11T01:37:43.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff758d90640 1 monclient: found mon.noname-b 2024-09-11T01:37:43.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff758d90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:43.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff758d90640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:37:43.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.847+0000 7ff758d90640 1 -- 172.21.15.12:0/2042849516 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff75409ba20 con 0x7ff754138230 2024-09-11T01:37:43.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.850+0000 7ff758d90640 1 -- 172.21.15.12:0/2042849516 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff754138230 msgr2=0x7ff754138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:43.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.850+0000 7ff758d90640 1 --2- 172.21.15.12:0/2042849516 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff754138230 0x7ff754138610 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff7480097d0 tx=0x7ff748029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=557c1508b4a30de3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:43.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.850+0000 7ff7537fe640 1 -- 172.21.15.12:0/2042849516 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff748037070 con 0x7ff754138230 2024-09-11T01:37:43.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.850+0000 7ff7537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:43.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.850+0000 7ff7537fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:37:43.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.850+0000 7ff7537fe640 10 monclient: dump: 2024-09-11T01:37:43.860 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:43.860 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.860 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.860 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.860 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:43.860 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.860 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:43.860 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:43.861 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:43.861 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.850+0000 7ff7537fe640 10 monclient: _finish_auth 0 2024-09-11T01:37:43.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.850+0000 7ff7537fe640 10 monclient: _check_auth_tickets 2024-09-11T01:37:43.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.850+0000 7ff7537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:43.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.850+0000 7ff75aff4640 5 monclient: authenticate success, global_id 84245 2024-09-11T01:37:43.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.850+0000 7ff7537fe640 1 -- 172.21.15.12:0/2042849516 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff748032440 con 0x7ff754138230 2024-09-11T01:37:43.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.850+0000 7ff7537fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:43.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.850+0000 7ff7537fe640 1 -- 172.21.15.12:0/2042849516 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff748032810 con 0x7ff754138230 2024-09-11T01:37:43.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.850+0000 7ff75aff4640 10 monclient: get_monmap_and_config success 2024-09-11T01:37:43.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.850+0000 7ff75aff4640 4 set_mon_vals no callback set 2024-09-11T01:37:43.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.850+0000 7ff7537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:43.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.850+0000 7ff7537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:43.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.850+0000 7ff7537fe640 10 monclient: dump: 2024-09-11T01:37:43.862 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:43.862 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.862 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.862 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.862 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:43.862 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.862 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:43.862 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:43.862 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:43.862 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.851+0000 7ff75aff4640 10 monclient: shutdown 2024-09-11T01:37:43.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.851+0000 7ff75aff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:43.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.851+0000 7ff75aff4640 1 -- 172.21.15.12:0/2042849516 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff754138230 msgr2=0x7ff754138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.851+0000 7ff75aff4640 1 --2- 172.21.15.12:0/2042849516 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff754138230 0x7ff754138610 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff7480097d0 tx=0x7ff748029fb0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.851+0000 7ff75aff4640 1 -- 172.21.15.12:0/2042849516 shutdown_connections 2024-09-11T01:37:43.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.851+0000 7ff75aff4640 1 --2- 172.21.15.12:0/2042849516 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff754138230 0x7ff754138610 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.851+0000 7ff75aff4640 1 -- 172.21.15.12:0/2042849516 >> 172.21.15.12:0/2042849516 conn(0x7ff7540a7120 msgr2=0x7ff7540a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:43.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.851+0000 7ff75aff4640 1 -- 172.21.15.12:0/2042849516 shutdown_connections 2024-09-11T01:37:43.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.851+0000 7ff75aff4640 1 -- 172.21.15.12:0/2042849516 wait complete. 2024-09-11T01:37:43.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.851+0000 7ff75aff4640 10 monclient: build_initial_monmap 2024-09-11T01:37:43.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.851+0000 7ff75aff4640 10 monclient: monmap: 2024-09-11T01:37:43.863 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:43.863 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:37:43.863 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:37:43.863 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:37:43.864 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:43.864 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.864 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:37:43.864 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:37:43.864 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:37:43.864 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.852+0000 7ff75aff4640 1 Processor -- start 2024-09-11T01:37:43.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.852+0000 7ff75aff4640 1 -- start start 2024-09-11T01:37:43.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.852+0000 7ff75aff4640 10 monclient: init 2024-09-11T01:37:43.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.852+0000 7ff75aff4640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:43.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.852+0000 7ff75aff4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:37:43.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.852+0000 7ff75aff4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff75409d030 0x7ff7541cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.852+0000 7ff75aff4640 10 monclient(hunting): picked mon.noname-c con 0x7ff75409d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:43.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.852+0000 7ff75aff4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff75409d430 0x7ff754140110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.852+0000 7ff75aff4640 10 monclient(hunting): picked mon.noname-a con 0x7ff75409d430 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:37:43.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.852+0000 7ff75aff4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff754138230 0x7ff754140650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.852+0000 7ff75aff4640 10 monclient(hunting): picked mon.noname-b con 0x7ff754138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:37:43.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.852+0000 7ff75aff4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.852+0000 7ff75aff4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff75406f7f0 con 0x7ff75409d430 2024-09-11T01:37:43.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.852+0000 7ff75aff4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.852+0000 7ff75aff4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff75406f970 con 0x7ff754138230 2024-09-11T01:37:43.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.852+0000 7ff75aff4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:43.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.852+0000 7ff75aff4640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff7541410c0 con 0x7ff75409d030 2024-09-11T01:37:43.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.852+0000 7ff75aff4640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:43.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.852+0000 7ff75aff4640 10 monclient(hunting): authenticate will time out at 1270.295927s 2024-09-11T01:37:43.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.852+0000 7ff758d90640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff75409d030 0x7ff7541cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.852+0000 7ff753fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff75409d430 0x7ff754140110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff758d90640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff75409d030 0x7ff7541cd530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42478/0 (socket says 172.21.15.12:42478) 2024-09-11T01:37:43.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff758d90640 1 -- 172.21.15.12:0/428456822 learned_addr learned my addr 172.21.15.12:0/428456822 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:43.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff753fff640 10 monclient(hunting): get_auth_request con 0x7ff75409d430 auth_method 0 2024-09-11T01:37:43.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff753fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:43.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff753fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:43.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff753fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:43.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff758d90640 10 monclient(hunting): get_auth_request con 0x7ff75409d030 auth_method 0 2024-09-11T01:37:43.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff758d90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:43.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff758d90640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:43.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff758d90640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:43.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff759591640 1 -- 172.21.15.12:0/428456822 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff754138230 msgr2=0x7ff754140650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:37:43.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff759591640 1 --2- 172.21.15.12:0/428456822 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff754138230 0x7ff754140650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:37:43.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff758d90640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:43.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff758d90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:43.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff758d90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:43.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff753fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:43.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff753fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:43.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff753fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:43.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff758d90640 10 monclient(hunting): handle_auth_done global_id 84248 payload 824 2024-09-11T01:37:43.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff758d90640 1 -- 172.21.15.12:0/428456822 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff754138230 msgr2=0x7ff754140650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:37:43.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff758d90640 1 --2- 172.21.15.12:0/428456822 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff754138230 0x7ff754140650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff758d90640 1 -- 172.21.15.12:0/428456822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff75409d430 msgr2=0x7ff754140110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:43.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff758d90640 1 --2- 172.21.15.12:0/428456822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff75409d430 0x7ff754140110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:43.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff758d90640 10 monclient: _finish_hunting 0 2024-09-11T01:37:43.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff758d90640 1 monclient: found mon.noname-c 2024-09-11T01:37:43.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff758d90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:43.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff758d90640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:37:43.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff758d90640 1 -- 172.21.15.12:0/428456822 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff75413a860 con 0x7ff75409d030 2024-09-11T01:37:43.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.853+0000 7ff753fff640 1 --2- 172.21.15.12:0/428456822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff75409d430 0x7ff754140110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:37:43.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.855+0000 7ff758d90640 1 -- 172.21.15.12:0/428456822 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff75409d030 msgr2=0x7ff7541cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:43.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.855+0000 7ff758d90640 1 --2- 172.21.15.12:0/428456822 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff75409d030 0x7ff7541cd530 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff7480097d0 tx=0x7ff7480046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:43.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.855+0000 7ff7517fa640 1 -- 172.21.15.12:0/428456822 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff748041070 con 0x7ff75409d030 2024-09-11T01:37:43.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.855+0000 7ff7517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:43.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.855+0000 7ff7517fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:37:43.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.855+0000 7ff7517fa640 10 monclient: dump: 2024-09-11T01:37:43.868 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:43.868 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.868 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.868 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.868 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:43.868 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.868 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:43.868 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:43.869 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:43.869 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.855+0000 7ff7517fa640 10 monclient: _finish_auth 0 2024-09-11T01:37:43.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.855+0000 7ff7517fa640 10 monclient: _check_auth_tickets 2024-09-11T01:37:43.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.855+0000 7ff7517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:43.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.855+0000 7ff7517fa640 1 -- 172.21.15.12:0/428456822 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff74803c660 con 0x7ff75409d030 2024-09-11T01:37:43.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.855+0000 7ff75aff4640 5 monclient: authenticate success, global_id 84248 2024-09-11T01:37:43.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.855+0000 7ff75aff4640 10 monclient: _renew_subs 2024-09-11T01:37:43.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.855+0000 7ff75aff4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:43.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.855+0000 7ff75aff4640 1 -- 172.21.15.12:0/428456822 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7540a15c0 con 0x7ff75409d030 2024-09-11T01:37:43.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.855+0000 7ff7517fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:43.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.855+0000 7ff7517fa640 1 -- 172.21.15.12:0/428456822 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff748037040 con 0x7ff75409d030 2024-09-11T01:37:43.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.855+0000 7ff7517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:43.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.855+0000 7ff7517fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:43.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.855+0000 7ff7517fa640 10 monclient: dump: 2024-09-11T01:37:43.870 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:43.870 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:43.870 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.870 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:43.870 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:43.870 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:43.870 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:43.870 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:43.870 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:43.870 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:43.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.855+0000 7ff752ffd640 4 set_mon_vals no callback set 2024-09-11T01:37:43.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.855+0000 7ff75aff4640 10 monclient: _renew_subs 2024-09-11T01:37:43.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.855+0000 7ff75aff4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:43.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.855+0000 7ff75aff4640 1 -- 172.21.15.12:0/428456822 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7540a4a50 con 0x7ff75409d030 2024-09-11T01:37:43.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.856+0000 7ff7517fa640 1 -- 172.21.15.12:0/428456822 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff74803ca50 con 0x7ff75409d030 2024-09-11T01:37:43.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.856+0000 7ff75aff4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:43.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.856+0000 7ff75aff4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:43.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.856+0000 7ff75aff4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:43.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.856+0000 7ff75aff4640 1 -- 172.21.15.12:0/428456822 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7540a1810 con 0x7ff75409d030 2024-09-11T01:37:43.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.857+0000 7ff7517fa640 1 --2- 172.21.15.12:0/428456822 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff73803e210 0x7ff7380406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:43.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.857+0000 7ff7517fa640 1 -- 172.21.15.12:0/428456822 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(287..287 src has 186..287) ==== 4815+0+0 (secure 0 0 0) 0x7ff748073390 con 0x7ff75409d030 2024-09-11T01:37:43.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.857+0000 7ff753fff640 1 --2- 172.21.15.12:0/428456822 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff73803e210 0x7ff7380406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:43.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.857+0000 7ff753fff640 10 monclient: get_auth_request con 0x7ff73803e210 auth_method 0 2024-09-11T01:37:43.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.860+0000 7ff753fff640 1 --2- 172.21.15.12:0/428456822 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff73803e210 0x7ff7380406d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff740007c90 tx=0x7ff740007be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:43.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.862+0000 7ff7517fa640 1 -- 172.21.15.12:0/428456822 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff7540a15c0 con 0x7ff75409d030 2024-09-11T01:37:43.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.862+0000 7ff7517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:43.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:43.862+0000 7ff7517fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:37:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:44.119+0000 7ff75aff4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:37:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:44.119+0000 7ff75aff4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:37:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:44.119+0000 7ff75aff4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:44.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:44.119+0000 7ff75aff4640 1 -- 172.21.15.12:0/428456822 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff754138610 con 0x7ff75409d030 2024-09-11T01:37:44.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:44.119+0000 7ff7517fa640 1 -- 172.21.15.12:0/428456822 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7ff748071d70 con 0x7ff75409d030 2024-09-11T01:37:44.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:44.119+0000 7ff7517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:37:44.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:44.119+0000 7ff7517fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:37:44.122 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:37:44.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:44.121+0000 7ff75aff4640 1 -- 172.21.15.12:0/428456822 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff73803e210 msgr2=0x7ff7380406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:44.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:44.121+0000 7ff75aff4640 1 --2- 172.21.15.12:0/428456822 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff73803e210 0x7ff7380406d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff740007c90 tx=0x7ff740007be0 comp rx=0 tx=0).stop 2024-09-11T01:37:44.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:44.121+0000 7ff75aff4640 10 monclient: shutdown 2024-09-11T01:37:44.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:44.121+0000 7ff75aff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:44.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:44.121+0000 7ff75aff4640 1 -- 172.21.15.12:0/428456822 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff75409d030 msgr2=0x7ff7541cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:44.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:44.121+0000 7ff75aff4640 1 --2- 172.21.15.12:0/428456822 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff75409d030 0x7ff7541cd530 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff7480097d0 tx=0x7ff7480046c0 comp rx=0 tx=0).stop 2024-09-11T01:37:44.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:44.121+0000 7ff75aff4640 1 -- 172.21.15.12:0/428456822 shutdown_connections 2024-09-11T01:37:44.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:44.121+0000 7ff75aff4640 1 --2- 172.21.15.12:0/428456822 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff754138230 0x7ff754140650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:44.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:44.121+0000 7ff75aff4640 1 --2- 172.21.15.12:0/428456822 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff73803e210 0x7ff7380406d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:44.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:44.121+0000 7ff75aff4640 1 --2- 172.21.15.12:0/428456822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff75409d430 0x7ff754140110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:44.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:44.121+0000 7ff75aff4640 1 --2- 172.21.15.12:0/428456822 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff75409d030 0x7ff7541cd530 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:44.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:44.121+0000 7ff75aff4640 1 -- 172.21.15.12:0/428456822 >> 172.21.15.12:0/428456822 conn(0x7ff7540a7120 msgr2=0x7ff7540a3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:44.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:44.121+0000 7ff75aff4640 1 -- 172.21.15.12:0/428456822 shutdown_connections 2024-09-11T01:37:44.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:44.121+0000 7ff75aff4640 1 -- 172.21.15.12:0/428456822 wait complete. 2024-09-11T01:37:47.135 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:37:47.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.246+0000 7fc58de39640 10 monclient: get_monmap_and_config 2024-09-11T01:37:47.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.246+0000 7fc58de39640 10 monclient: build_initial_monmap 2024-09-11T01:37:47.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.246+0000 7fc58de39640 10 monclient: monmap: 2024-09-11T01:37:47.249 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:47.249 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:47.249 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:37:47.248072+0000 2024-09-11T01:37:47.249 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:37:47.248072+0000 2024-09-11T01:37:47.249 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:47.249 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:47.250 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:37:47.250 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:37:47.250 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:37:47.250 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:37:47.250 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:37:47.250 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.247+0000 7fc58de39640 1 Processor -- start 2024-09-11T01:37:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.247+0000 7fc58de39640 1 -- start start 2024-09-11T01:37:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.247+0000 7fc58de39640 10 monclient: init 2024-09-11T01:37:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.247+0000 7fc58de39640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.247+0000 7fc58de39640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-11T01:37:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.247+0000 7fc58de39640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc5880a2e90 0x7fc58806dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.247+0000 7fc58de39640 10 monclient(hunting): picked mon.noname-c con 0x7fc5880a2e90 addr v2:172.21.15.71:3300/0 2024-09-11T01:37:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.247+0000 7fc58de39640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5880a3270 0x7fc5880a3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.247+0000 7fc58de39640 10 monclient(hunting): picked mon.noname-b con 0x7fc5880a3270 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.247+0000 7fc58de39640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc588138040 addr v1:172.21.15.12:6789/0 2024-09-11T01:37:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.247+0000 7fc58de39640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5880a46c0 con 0x7fc588138040 2024-09-11T01:37:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.247+0000 7fc58de39640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.247+0000 7fc58de39640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fc5880a4900 con 0x7fc5880a2e90 2024-09-11T01:37:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.247+0000 7fc58de39640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.247+0000 7fc58de39640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc58806ee50 con 0x7fc5880a3270 2024-09-11T01:37:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.247+0000 7fc58de39640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.247+0000 7fc58de39640 10 monclient(hunting): authenticate will time out at 1273.690750s 2024-09-11T01:37:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.247+0000 7fc586ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5880a3270 0x7fc5880a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.247+0000 7fc5877fe640 1 -- >> v2:172.21.15.71:3300/0 conn(0x7fc5880a2e90 msgr2=0x7fc58806dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:37:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.247+0000 7fc587fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc588138040 0x7fc588138490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33358/0 (socket says 172.21.15.12:33358) 2024-09-11T01:37:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.247+0000 7fc587fff640 1 -- 172.21.15.12:0/3033181740 learned_addr learned my addr 172.21.15.12:0/3033181740 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:37:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.247+0000 7fc5877fe640 1 --2- 172.21.15.12:0/3033181740 >> v2:172.21.15.71:3300/0 conn(0x7fc5880a2e90 0x7fc58806dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:37:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.247+0000 7fc586ffd640 10 monclient(hunting): get_auth_request con 0x7fc5880a3270 auth_method 0 2024-09-11T01:37:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.247+0000 7fc586ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.247+0000 7fc586ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.247+0000 7fc586ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.248+0000 7fc586ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.248+0000 7fc586ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.248+0000 7fc586ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.248+0000 7fc586ffd640 10 monclient(hunting): handle_auth_done global_id 84251 payload 274 2024-09-11T01:37:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.248+0000 7fc586ffd640 1 -- 172.21.15.12:0/3033181740 >> v2:172.21.15.71:3300/0 conn(0x7fc5880a2e90 msgr2=0x7fc58806dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:37:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.248+0000 7fc586ffd640 1 --2- 172.21.15.12:0/3033181740 >> v2:172.21.15.71:3300/0 conn(0x7fc5880a2e90 0x7fc58806dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.248+0000 7fc586ffd640 1 -- 172.21.15.12:0/3033181740 >> v1:172.21.15.12:6789/0 conn(0x7fc588138040 legacy=0x7fc588138490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.248+0000 7fc586ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:37:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.248+0000 7fc586ffd640 1 monclient: found mon.noname-b 2024-09-11T01:37:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.248+0000 7fc586ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.248+0000 7fc586ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:37:47.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.248+0000 7fc586ffd640 1 -- 172.21.15.12:0/3033181740 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc58806eb70 con 0x7fc5880a3270 2024-09-11T01:37:47.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.250+0000 7fc586ffd640 1 -- 172.21.15.12:0/3033181740 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5880a3270 msgr2=0x7fc5880a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:47.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.250+0000 7fc586ffd640 1 --2- 172.21.15.12:0/3033181740 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5880a3270 0x7fc5880a3680 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc5700097a0 tx=0x7fc570029fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb0f91bf0a18b1df server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:47.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc5867fc640 1 -- 172.21.15.12:0/3033181740 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc570037070 con 0x7fc5880a3270 2024-09-11T01:37:47.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc5867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:47.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc5867fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:37:47.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc5867fc640 10 monclient: dump: 2024-09-11T01:37:47.260 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:47.260 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:47.261 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:47.261 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:47.261 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:47.261 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:47.261 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:47.261 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:47.261 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:47.261 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:47.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc5867fc640 10 monclient: _finish_auth 0 2024-09-11T01:37:47.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc5867fc640 10 monclient: _check_auth_tickets 2024-09-11T01:37:47.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc5867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:47.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc58de39640 5 monclient: authenticate success, global_id 84251 2024-09-11T01:37:47.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc5867fc640 1 -- 172.21.15.12:0/3033181740 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc570032640 con 0x7fc5880a3270 2024-09-11T01:37:47.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc58de39640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:37:47.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc5867fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:47.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc58de39640 10 monclient: get_monmap_and_config success 2024-09-11T01:37:47.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc58de39640 4 set_mon_vals no callback set 2024-09-11T01:37:47.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc5867fc640 1 -- 172.21.15.12:0/3033181740 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc570032a10 con 0x7fc5880a3270 2024-09-11T01:37:47.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc5867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:47.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc5867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:47.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc5867fc640 10 monclient: dump: 2024-09-11T01:37:47.262 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:47.262 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:47.262 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:47.262 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:47.263 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:47.263 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:47.263 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:47.263 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:47.263 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:47.263 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:47.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc58de39640 10 monclient: shutdown 2024-09-11T01:37:47.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc58de39640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:47.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc58de39640 1 -- 172.21.15.12:0/3033181740 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5880a3270 msgr2=0x7fc5880a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:47.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc58de39640 1 --2- 172.21.15.12:0/3033181740 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5880a3270 0x7fc5880a3680 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc5700097a0 tx=0x7fc570029fe0 comp rx=0 tx=0).stop 2024-09-11T01:37:47.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc58de39640 1 -- 172.21.15.12:0/3033181740 shutdown_connections 2024-09-11T01:37:47.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc58de39640 1 --2- 172.21.15.12:0/3033181740 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5880a3270 0x7fc5880a3680 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:47.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc58de39640 1 --2- 172.21.15.12:0/3033181740 >> v2:172.21.15.71:3300/0 conn(0x7fc5880a2e90 0x7fc58806dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:47.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc58de39640 1 -- 172.21.15.12:0/3033181740 >> 172.21.15.12:0/3033181740 conn(0x7fc5880a8490 msgr2=0x7fc5880a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:47.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc58de39640 1 -- 172.21.15.12:0/3033181740 shutdown_connections 2024-09-11T01:37:47.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.251+0000 7fc58de39640 1 -- 172.21.15.12:0/3033181740 wait complete. 2024-09-11T01:37:47.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.252+0000 7fc58de39640 10 monclient: build_initial_monmap 2024-09-11T01:37:47.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.252+0000 7fc58de39640 10 monclient: monmap: 2024-09-11T01:37:47.264 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:47.264 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:37:47.264 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:37:47.264 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:37:47.264 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:47.264 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:47.264 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:37:47.264 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:37:47.264 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:37:47.265 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:47.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.252+0000 7fc58de39640 1 Processor -- start 2024-09-11T01:37:47.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc58de39640 1 -- start start 2024-09-11T01:37:47.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc58de39640 10 monclient: init 2024-09-11T01:37:47.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc58de39640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:47.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc58de39640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:37:47.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc58de39640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5880a2e90 0x7fc5881cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:47.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc58de39640 10 monclient(hunting): picked mon.noname-b con 0x7fc5880a2e90 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:37:47.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc58de39640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5880a3270 0x7fc5881cda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:47.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc58de39640 10 monclient(hunting): picked mon.noname-c con 0x7fc5880a3270 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:47.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc58de39640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc588138040 0x7fc588124c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:47.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc58de39640 10 monclient(hunting): picked mon.noname-a con 0x7fc588138040 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:37:47.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc58de39640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:47.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc58de39640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc58806fba0 con 0x7fc588138040 2024-09-11T01:37:47.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc58de39640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:47.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc58de39640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc5880a3bc0 con 0x7fc5880a2e90 2024-09-11T01:37:47.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc58de39640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:47.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc58de39640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc58806fd20 con 0x7fc5880a3270 2024-09-11T01:37:47.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc58de39640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:47.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc58de39640 10 monclient(hunting): authenticate will time out at 1273.696474s 2024-09-11T01:37:47.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc586ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5880a3270 0x7fc5881cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:47.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc587fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc588138040 0x7fc588124c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:47.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc586ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5880a3270 0x7fc5881cda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41128/0 (socket says 172.21.15.12:41128) 2024-09-11T01:37:47.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc5877fe640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5880a2e90 msgr2=0x7fc5881cd4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:37:47.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc586ffd640 1 -- 172.21.15.12:0/2571242252 learned_addr learned my addr 172.21.15.12:0/2571242252 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:47.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc586ffd640 10 monclient(hunting): get_auth_request con 0x7fc5880a3270 auth_method 0 2024-09-11T01:37:47.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc586ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:47.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc586ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:47.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc586ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:47.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc5877fe640 1 --2- 172.21.15.12:0/2571242252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5880a2e90 0x7fc5881cd4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:37:47.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc587fff640 10 monclient(hunting): get_auth_request con 0x7fc588138040 auth_method 0 2024-09-11T01:37:47.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc587fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:47.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc587fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:47.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc587fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:47.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc586ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:47.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc586ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:47.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc586ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:47.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc587fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:47.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc587fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc587fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc586ffd640 10 monclient(hunting): handle_auth_done global_id 84254 payload 824 2024-09-11T01:37:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc586ffd640 1 -- 172.21.15.12:0/2571242252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5880a2e90 msgr2=0x7fc5881cd4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:37:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc586ffd640 1 --2- 172.21.15.12:0/2571242252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5880a2e90 0x7fc5881cd4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc586ffd640 1 -- 172.21.15.12:0/2571242252 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc588138040 msgr2=0x7fc588124c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc586ffd640 1 --2- 172.21.15.12:0/2571242252 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc588138040 0x7fc588124c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc586ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:37:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc586ffd640 1 monclient: found mon.noname-c 2024-09-11T01:37:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc586ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc586ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:37:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.253+0000 7fc586ffd640 1 -- 172.21.15.12:0/2571242252 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc588073d90 con 0x7fc5880a3270 2024-09-11T01:37:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.256+0000 7fc586ffd640 1 -- 172.21.15.12:0/2571242252 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5880a3270 msgr2=0x7fc5881cda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.256+0000 7fc586ffd640 1 --2- 172.21.15.12:0/2571242252 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5880a3270 0x7fc5881cda20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc5700098c0 tx=0x7fc5700046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:47.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.256+0000 7fc57ffff640 1 -- 172.21.15.12:0/2571242252 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc570042020 con 0x7fc5880a3270 2024-09-11T01:37:47.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.256+0000 7fc57ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:47.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.256+0000 7fc57ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:37:47.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.256+0000 7fc57ffff640 10 monclient: dump: 2024-09-11T01:37:47.269 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:47.269 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:47.269 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:47.269 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:47.269 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:47.269 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:47.269 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:47.269 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:47.270 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:47.270 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:47.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.256+0000 7fc57ffff640 10 monclient: _finish_auth 0 2024-09-11T01:37:47.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.256+0000 7fc57ffff640 10 monclient: _check_auth_tickets 2024-09-11T01:37:47.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.256+0000 7fc57ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:47.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.256+0000 7fc57ffff640 1 -- 172.21.15.12:0/2571242252 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc570030880 con 0x7fc5880a3270 2024-09-11T01:37:47.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.256+0000 7fc57ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:47.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.256+0000 7fc57ffff640 1 -- 172.21.15.12:0/2571242252 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc570037040 con 0x7fc5880a3270 2024-09-11T01:37:47.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.256+0000 7fc57ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:47.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.256+0000 7fc57ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:47.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.256+0000 7fc57ffff640 10 monclient: dump: 2024-09-11T01:37:47.270 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:47.270 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:47.270 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:47.271 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:47.271 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:47.271 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:47.271 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:47.271 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:47.271 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:47.271 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:47.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.256+0000 7fc58de39640 5 monclient: authenticate success, global_id 84254 2024-09-11T01:37:47.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.256+0000 7fc585ffb640 4 set_mon_vals no callback set 2024-09-11T01:37:47.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.256+0000 7fc58de39640 10 monclient: _renew_subs 2024-09-11T01:37:47.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.256+0000 7fc58de39640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:47.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.256+0000 7fc58de39640 1 -- 172.21.15.12:0/2571242252 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc58809f680 con 0x7fc5880a3270 2024-09-11T01:37:47.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.256+0000 7fc58de39640 10 monclient: _renew_subs 2024-09-11T01:37:47.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.256+0000 7fc58de39640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:47.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.256+0000 7fc58de39640 1 -- 172.21.15.12:0/2571242252 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5881cdf60 con 0x7fc5880a3270 2024-09-11T01:37:47.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.257+0000 7fc57ffff640 1 -- 172.21.15.12:0/2571242252 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fc570030c70 con 0x7fc5880a3270 2024-09-11T01:37:47.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.257+0000 7fc58de39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:47.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.257+0000 7fc58de39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:47.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.257+0000 7fc58de39640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:47.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.257+0000 7fc58de39640 1 -- 172.21.15.12:0/2571242252 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc588074570 con 0x7fc5880a3270 2024-09-11T01:37:47.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.257+0000 7fc57ffff640 1 --2- 172.21.15.12:0/2571242252 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc56803e210 0x7fc5680406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:47.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.257+0000 7fc57ffff640 1 -- 172.21.15.12:0/2571242252 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(287..287 src has 186..287) ==== 4815+0+0 (secure 0 0 0) 0x7fc5700739c0 con 0x7fc5880a3270 2024-09-11T01:37:47.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.257+0000 7fc5877fe640 1 --2- 172.21.15.12:0/2571242252 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc56803e210 0x7fc5680406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:47.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.258+0000 7fc5877fe640 10 monclient: get_auth_request con 0x7fc56803e210 auth_method 0 2024-09-11T01:37:47.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.260+0000 7fc5877fe640 1 --2- 172.21.15.12:0/2571242252 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc56803e210 0x7fc5680406d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc578003cb0 tx=0x7fc578009270 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:47.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.262+0000 7fc57ffff640 1 -- 172.21.15.12:0/2571242252 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc588074570 con 0x7fc5880a3270 2024-09-11T01:37:47.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.262+0000 7fc57ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:47.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.262+0000 7fc57ffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:37:47.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.522+0000 7fc58de39640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:37:47.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.522+0000 7fc58de39640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:37:47.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.522+0000 7fc58de39640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:37:47.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.522+0000 7fc58de39640 1 -- 172.21.15.12:0/2571242252 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc588138c30 con 0x7fc5880a3270 2024-09-11T01:37:47.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.523+0000 7fc57ffff640 1 -- 172.21.15.12:0/2571242252 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fc57004c030 con 0x7fc5880a3270 2024-09-11T01:37:47.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.523+0000 7fc57ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:37:47.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.523+0000 7fc57ffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:37:47.526 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:37:47.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.525+0000 7fc58de39640 1 -- 172.21.15.12:0/2571242252 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc56803e210 msgr2=0x7fc5680406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:47.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.525+0000 7fc58de39640 1 --2- 172.21.15.12:0/2571242252 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc56803e210 0x7fc5680406d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc578003cb0 tx=0x7fc578009270 comp rx=0 tx=0).stop 2024-09-11T01:37:47.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.525+0000 7fc58de39640 10 monclient: shutdown 2024-09-11T01:37:47.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.525+0000 7fc58de39640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:47.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.525+0000 7fc58de39640 1 -- 172.21.15.12:0/2571242252 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5880a3270 msgr2=0x7fc5881cda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:47.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.525+0000 7fc58de39640 1 --2- 172.21.15.12:0/2571242252 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5880a3270 0x7fc5881cda20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc5700098c0 tx=0x7fc5700046c0 comp rx=0 tx=0).stop 2024-09-11T01:37:47.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.525+0000 7fc58de39640 1 -- 172.21.15.12:0/2571242252 shutdown_connections 2024-09-11T01:37:47.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.525+0000 7fc58de39640 1 --2- 172.21.15.12:0/2571242252 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc56803e210 0x7fc5680406d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:47.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.525+0000 7fc58de39640 1 --2- 172.21.15.12:0/2571242252 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc588138040 0x7fc588124c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:47.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.525+0000 7fc58de39640 1 --2- 172.21.15.12:0/2571242252 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5880a3270 0x7fc5881cda20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:47.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.525+0000 7fc58de39640 1 --2- 172.21.15.12:0/2571242252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5880a2e90 0x7fc5881cd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:47.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.525+0000 7fc58de39640 1 -- 172.21.15.12:0/2571242252 >> 172.21.15.12:0/2571242252 conn(0x7fc5880a8490 msgr2=0x7fc588072090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:47.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.525+0000 7fc58de39640 1 -- 172.21.15.12:0/2571242252 shutdown_connections 2024-09-11T01:37:47.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:47.525+0000 7fc58de39640 1 -- 172.21.15.12:0/2571242252 wait complete. 2024-09-11T01:37:50.539 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:37:50.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.649+0000 7f4f91efe640 10 monclient: get_monmap_and_config 2024-09-11T01:37:50.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.649+0000 7f4f91efe640 10 monclient: build_initial_monmap 2024-09-11T01:37:50.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.649+0000 7f4f91efe640 10 monclient: monmap: 2024-09-11T01:37:50.652 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:50.652 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:50.652 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:37:50.651423+0000 2024-09-11T01:37:50.652 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:37:50.651423+0000 2024-09-11T01:37:50.652 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:50.652 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:50.653 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:37:50.653 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:37:50.653 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:37:50.653 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:37:50.653 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:37:50.653 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:50.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.650+0000 7f4f91efe640 1 Processor -- start 2024-09-11T01:37:50.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.650+0000 7f4f91efe640 1 -- start start 2024-09-11T01:37:50.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.650+0000 7f4f91efe640 10 monclient: init 2024-09-11T01:37:50.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.650+0000 7f4f91efe640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.650+0000 7f4f91efe640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-11T01:37:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f91efe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4f8c09d790 addr v1:172.21.15.12:6789/0 2024-09-11T01:37:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f91efe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4f8c09e280 0x7f4f8c139f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f91efe640 10 monclient(hunting): picked mon.noname-a con 0x7f4f8c09e280 addr v2:172.21.15.12:3300/0 2024-09-11T01:37:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f91efe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f8c09e6d0 0x7f4f8c13c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f91efe640 10 monclient(hunting): picked mon.noname-b con 0x7f4f8c09e6d0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f91efe640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f8c0a1da0 con 0x7f4f8c09d790 2024-09-11T01:37:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f91efe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f91efe640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f4f8c0a0760 con 0x7f4f8c09e280 2024-09-11T01:37:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f91efe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f91efe640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4f8c0a1fa0 con 0x7f4f8c09e6d0 2024-09-11T01:37:50.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f91efe640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:50.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f91efe640 10 monclient(hunting): authenticate will time out at 1277.094092s 2024-09-11T01:37:50.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8b7fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4f8c09d790 0x7f4f8c09db70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33366/0 (socket says 172.21.15.12:33366) 2024-09-11T01:37:50.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8b7fe640 1 -- 172.21.15.12:0/3647765460 learned_addr learned my addr 172.21.15.12:0/3647765460 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:37:50.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8bfff640 1 --2- 172.21.15.12:0/3647765460 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f8c09e6d0 0x7f4f8c13c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:50.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8affd640 1 --2- 172.21.15.12:0/3647765460 >> v2:172.21.15.12:3300/0 conn(0x7f4f8c09e280 0x7f4f8c139f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:50.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8affd640 10 monclient(hunting): get_auth_request con 0x7f4f8c09e280 auth_method 0 2024-09-11T01:37:50.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:50.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:50.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:50.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8bfff640 10 monclient(hunting): get_auth_request con 0x7f4f8c09e6d0 auth_method 0 2024-09-11T01:37:50.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:50.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:50.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:50.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:50.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:50.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:50.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:50.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:50.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:50.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8affd640 10 monclient(hunting): handle_auth_done global_id 94167 payload 274 2024-09-11T01:37:50.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8affd640 1 -- 172.21.15.12:0/3647765460 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f8c09e6d0 msgr2=0x7f4f8c13c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:50.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8affd640 1 --2- 172.21.15.12:0/3647765460 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f8c09e6d0 0x7f4f8c13c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:50.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8affd640 1 -- 172.21.15.12:0/3647765460 >> v1:172.21.15.12:6789/0 conn(0x7f4f8c09d790 legacy=0x7f4f8c09db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:50.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8affd640 10 monclient: _finish_hunting 0 2024-09-11T01:37:50.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8affd640 1 monclient: found mon.noname-a 2024-09-11T01:37:50.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:50.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:37:50.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.651+0000 7f4f8affd640 1 -- 172.21.15.12:0/3647765460 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f8c09eda0 con 0x7f4f8c09e280 2024-09-11T01:37:50.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.654+0000 7f4f8affd640 1 -- 172.21.15.12:0/3647765460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f8c09e280 msgr2=0x7f4f8c139f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:50.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.654+0000 7f4f8affd640 1 --2- 172.21.15.12:0/3647765460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f8c09e280 0x7f4f8c139f20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4f800097a0 tx=0x7f4f8002a050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=711740e696f07260 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:50.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.654+0000 7f4f8a7fc640 1 -- 172.21.15.12:0/3647765460 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f80037070 con 0x7f4f8c09e280 2024-09-11T01:37:50.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.654+0000 7f4f8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:50.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.654+0000 7f4f8a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:37:50.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.654+0000 7f4f8a7fc640 10 monclient: dump: 2024-09-11T01:37:50.664 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:50.664 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:50.664 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:50.665 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:50.665 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:50.665 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:50.665 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:50.665 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:50.665 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:50.665 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:50.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.654+0000 7f4f8a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:37:50.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.654+0000 7f4f8a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:37:50.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.654+0000 7f4f8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:50.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.654+0000 7f4f8a7fc640 1 -- 172.21.15.12:0/3647765460 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f80032440 con 0x7f4f8c09e280 2024-09-11T01:37:50.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.654+0000 7f4f8a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:50.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.654+0000 7f4f8a7fc640 1 -- 172.21.15.12:0/3647765460 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f80032810 con 0x7f4f8c09e280 2024-09-11T01:37:50.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.654+0000 7f4f91efe640 5 monclient: authenticate success, global_id 94167 2024-09-11T01:37:50.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.654+0000 7f4f91efe640 10 monclient: get_monmap_and_config success 2024-09-11T01:37:50.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.654+0000 7f4f91efe640 4 set_mon_vals no callback set 2024-09-11T01:37:50.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.654+0000 7f4f8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:50.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.654+0000 7f4f8a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:50.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.654+0000 7f4f8a7fc640 10 monclient: dump: 2024-09-11T01:37:50.666 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:50.666 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:50.666 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:50.666 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:50.666 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:50.666 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:50.666 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:50.667 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:50.667 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:50.667 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:50.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.655+0000 7f4f91efe640 10 monclient: shutdown 2024-09-11T01:37:50.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.655+0000 7f4f91efe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:50.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.655+0000 7f4f91efe640 1 -- 172.21.15.12:0/3647765460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f8c09e280 msgr2=0x7f4f8c139f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:50.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.655+0000 7f4f91efe640 1 --2- 172.21.15.12:0/3647765460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f8c09e280 0x7f4f8c139f20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4f800097a0 tx=0x7f4f8002a050 comp rx=0 tx=0).stop 2024-09-11T01:37:50.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.655+0000 7f4f91efe640 1 -- 172.21.15.12:0/3647765460 shutdown_connections 2024-09-11T01:37:50.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.655+0000 7f4f91efe640 1 --2- 172.21.15.12:0/3647765460 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f8c09e6d0 0x7f4f8c13c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:50.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.655+0000 7f4f91efe640 1 --2- 172.21.15.12:0/3647765460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f8c09e280 0x7f4f8c139f20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:50.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.655+0000 7f4f91efe640 1 -- 172.21.15.12:0/3647765460 >> 172.21.15.12:0/3647765460 conn(0x7f4f8c0a2d90 msgr2=0x7f4f8c0a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:50.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.655+0000 7f4f91efe640 1 -- 172.21.15.12:0/3647765460 shutdown_connections 2024-09-11T01:37:50.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.655+0000 7f4f91efe640 1 -- 172.21.15.12:0/3647765460 wait complete. 2024-09-11T01:37:50.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.655+0000 7f4f91efe640 10 monclient: build_initial_monmap 2024-09-11T01:37:50.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.655+0000 7f4f91efe640 10 monclient: monmap: 2024-09-11T01:37:50.668 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:50.668 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:37:50.668 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:37:50.668 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:37:50.668 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:50.668 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:50.668 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:37:50.668 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:37:50.668 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:37:50.668 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:50.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.656+0000 7f4f91efe640 1 Processor -- start 2024-09-11T01:37:50.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.656+0000 7f4f91efe640 1 -- start start 2024-09-11T01:37:50.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.656+0000 7f4f91efe640 10 monclient: init 2024-09-11T01:37:50.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.656+0000 7f4f91efe640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:50.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.656+0000 7f4f91efe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:37:50.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.656+0000 7f4f91efe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f8c09d790 0x7f4f8c1cac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:50.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.656+0000 7f4f91efe640 10 monclient(hunting): picked mon.noname-c con 0x7f4f8c09d790 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:50.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.656+0000 7f4f91efe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f8c09e280 0x7f4f8c1cb170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:50.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.656+0000 7f4f91efe640 10 monclient(hunting): picked mon.noname-a con 0x7f4f8c09e280 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:37:50.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.656+0000 7f4f91efe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f8c09e6d0 0x7f4f8c1cb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:50.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.656+0000 7f4f91efe640 10 monclient(hunting): picked mon.noname-b con 0x7f4f8c09e6d0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:37:50.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.656+0000 7f4f91efe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:50.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.656+0000 7f4f91efe640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8c09ce20 con 0x7f4f8c09e280 2024-09-11T01:37:50.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.656+0000 7f4f91efe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:50.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.656+0000 7f4f91efe640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8c09cca0 con 0x7f4f8c09e6d0 2024-09-11T01:37:50.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.656+0000 7f4f91efe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:50.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.656+0000 7f4f91efe640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4f8c13c9b0 con 0x7f4f8c09d790 2024-09-11T01:37:50.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.656+0000 7f4f91efe640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:50.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.656+0000 7f4f91efe640 10 monclient(hunting): authenticate will time out at 1277.099975s 2024-09-11T01:37:50.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f8c09d790 0x7f4f8c1cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:50.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f8c09e280 0x7f4f8c1cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:50.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8bfff640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f8c09e6d0 msgr2=0x7f4f8c1cb6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:37:50.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f8c09e280 0x7f4f8c1cb170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58112/0 (socket says 172.21.15.12:58112) 2024-09-11T01:37:50.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f8c09d790 0x7f4f8c1cac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41138/0 (socket says 172.21.15.12:41138) 2024-09-11T01:37:50.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8affd640 1 -- 172.21.15.12:0/395143322 learned_addr learned my addr 172.21.15.12:0/395143322 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:50.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8affd640 10 monclient(hunting): get_auth_request con 0x7f4f8c09e280 auth_method 0 2024-09-11T01:37:50.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:50.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8bfff640 1 --2- 172.21.15.12:0/395143322 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f8c09e6d0 0x7f4f8c1cb6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:37:50.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:50.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:50.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8b7fe640 10 monclient(hunting): get_auth_request con 0x7f4f8c09d790 auth_method 0 2024-09-11T01:37:50.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:50.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:50.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:50.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:50.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:50.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:50.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:50.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:50.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:50.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8affd640 10 monclient(hunting): handle_auth_done global_id 94170 payload 824 2024-09-11T01:37:50.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8affd640 1 -- 172.21.15.12:0/395143322 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f8c09d790 msgr2=0x7f4f8c1cac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:50.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8affd640 1 --2- 172.21.15.12:0/395143322 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f8c09d790 0x7f4f8c1cac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:50.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8affd640 1 -- 172.21.15.12:0/395143322 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f8c09e6d0 msgr2=0x7f4f8c1cb6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:37:50.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8affd640 1 --2- 172.21.15.12:0/395143322 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f8c09e6d0 0x7f4f8c1cb6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:50.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8affd640 10 monclient: _finish_hunting 0 2024-09-11T01:37:50.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8affd640 1 monclient: found mon.noname-a 2024-09-11T01:37:50.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:50.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:37:50.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8affd640 1 -- 172.21.15.12:0/395143322 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f8c1328c0 con 0x7f4f8c09e280 2024-09-11T01:37:50.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.657+0000 7f4f8b7fe640 1 --2- 172.21.15.12:0/395143322 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f8c09d790 0x7f4f8c1cac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:37:50.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.659+0000 7f4f8affd640 1 -- 172.21.15.12:0/395143322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f8c09e280 msgr2=0x7f4f8c1cb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:50.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.659+0000 7f4f8affd640 1 --2- 172.21.15.12:0/395143322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f8c09e280 0x7f4f8c1cb170 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4f800098c0 tx=0x7f4f800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:50.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.659+0000 7f4f73fff640 1 -- 172.21.15.12:0/395143322 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f80040020 con 0x7f4f8c09e280 2024-09-11T01:37:50.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.659+0000 7f4f73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:50.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.659+0000 7f4f73fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:37:50.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.659+0000 7f4f73fff640 10 monclient: dump: 2024-09-11T01:37:50.673 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:50.673 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:50.673 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:50.673 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:50.673 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:50.673 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:50.673 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:50.674 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:50.674 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:50.674 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:50.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.659+0000 7f4f73fff640 10 monclient: _finish_auth 0 2024-09-11T01:37:50.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.659+0000 7f4f73fff640 10 monclient: _check_auth_tickets 2024-09-11T01:37:50.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.659+0000 7f4f73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:50.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.659+0000 7f4f73fff640 1 -- 172.21.15.12:0/395143322 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f80038040 con 0x7f4f8c09e280 2024-09-11T01:37:50.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.659+0000 7f4f91efe640 5 monclient: authenticate success, global_id 94170 2024-09-11T01:37:50.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.659+0000 7f4f73fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:50.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.659+0000 7f4f73fff640 1 -- 172.21.15.12:0/395143322 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f80037070 con 0x7f4f8c09e280 2024-09-11T01:37:50.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.659+0000 7f4f91efe640 10 monclient: _renew_subs 2024-09-11T01:37:50.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.659+0000 7f4f91efe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:50.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.659+0000 7f4f91efe640 1 -- 172.21.15.12:0/395143322 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f8c134c00 con 0x7f4f8c09e280 2024-09-11T01:37:50.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.659+0000 7f4f73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:50.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.659+0000 7f4f73fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:50.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.659+0000 7f4f73fff640 10 monclient: dump: 2024-09-11T01:37:50.675 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:50.675 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:50.675 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:50.675 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:50.675 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:50.675 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:50.675 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:50.675 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:50.675 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:50.675 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:50.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.659+0000 7f4f89ffb640 4 set_mon_vals no callback set 2024-09-11T01:37:50.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.660+0000 7f4f91efe640 10 monclient: _renew_subs 2024-09-11T01:37:50.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.660+0000 7f4f91efe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:50.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.660+0000 7f4f91efe640 1 -- 172.21.15.12:0/395143322 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f8c133060 con 0x7f4f8c09e280 2024-09-11T01:37:50.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.661+0000 7f4f73fff640 1 -- 172.21.15.12:0/395143322 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f4f80045410 con 0x7f4f8c09e280 2024-09-11T01:37:50.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.661+0000 7f4f91efe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:50.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.661+0000 7f4f91efe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:50.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.661+0000 7f4f91efe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:50.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.661+0000 7f4f91efe640 1 -- 172.21.15.12:0/395143322 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f8c09db70 con 0x7f4f8c09e280 2024-09-11T01:37:50.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.661+0000 7f4f73fff640 1 --2- 172.21.15.12:0/395143322 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4f7803e1c0 0x7f4f78040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:50.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.661+0000 7f4f73fff640 1 -- 172.21.15.12:0/395143322 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(287..287 src has 186..287) ==== 4815+0+0 (secure 0 0 0) 0x7f4f80073070 con 0x7f4f8c09e280 2024-09-11T01:37:50.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.661+0000 7f4f8b7fe640 1 --2- 172.21.15.12:0/395143322 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4f7803e1c0 0x7f4f78040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:50.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.661+0000 7f4f8b7fe640 10 monclient: get_auth_request con 0x7f4f7803e1c0 auth_method 0 2024-09-11T01:37:50.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.664+0000 7f4f8b7fe640 1 --2- 172.21.15.12:0/395143322 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4f7803e1c0 0x7f4f78040680 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c007c90 tx=0x7f4f7c007be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:50.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.665+0000 7f4f73fff640 1 -- 172.21.15.12:0/395143322 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4f8004a070 con 0x7f4f8c09e280 2024-09-11T01:37:50.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.665+0000 7f4f73fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:50.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.665+0000 7f4f73fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:37:50.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.923+0000 7f4f91efe640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:37:50.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.923+0000 7f4f91efe640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:37:50.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.923+0000 7f4f91efe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:50.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.923+0000 7f4f91efe640 1 -- 172.21.15.12:0/395143322 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4f8c06d460 con 0x7f4f8c09e280 2024-09-11T01:37:50.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.924+0000 7f4f73fff640 1 -- 172.21.15.12:0/395143322 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f4f80071a50 con 0x7f4f8c09e280 2024-09-11T01:37:50.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.924+0000 7f4f73fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:37:50.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.924+0000 7f4f73fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:37:50.927 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:37:50.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.926+0000 7f4f91efe640 1 -- 172.21.15.12:0/395143322 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4f7803e1c0 msgr2=0x7f4f78040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:50.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.926+0000 7f4f91efe640 1 --2- 172.21.15.12:0/395143322 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4f7803e1c0 0x7f4f78040680 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c007c90 tx=0x7f4f7c007be0 comp rx=0 tx=0).stop 2024-09-11T01:37:50.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.926+0000 7f4f91efe640 10 monclient: shutdown 2024-09-11T01:37:50.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.926+0000 7f4f91efe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:50.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.926+0000 7f4f91efe640 1 -- 172.21.15.12:0/395143322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f8c09e280 msgr2=0x7f4f8c1cb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:50.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.926+0000 7f4f91efe640 1 --2- 172.21.15.12:0/395143322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f8c09e280 0x7f4f8c1cb170 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4f800098c0 tx=0x7f4f800046c0 comp rx=0 tx=0).stop 2024-09-11T01:37:50.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.926+0000 7f4f91efe640 1 -- 172.21.15.12:0/395143322 shutdown_connections 2024-09-11T01:37:50.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.926+0000 7f4f91efe640 1 --2- 172.21.15.12:0/395143322 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f8c09e6d0 0x7f4f8c1cb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:50.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.926+0000 7f4f91efe640 1 --2- 172.21.15.12:0/395143322 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4f7803e1c0 0x7f4f78040680 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:50.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.926+0000 7f4f91efe640 1 --2- 172.21.15.12:0/395143322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f8c09e280 0x7f4f8c1cb170 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:50.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.926+0000 7f4f91efe640 1 --2- 172.21.15.12:0/395143322 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f8c09d790 0x7f4f8c1cac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:50.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.926+0000 7f4f91efe640 1 -- 172.21.15.12:0/395143322 >> 172.21.15.12:0/395143322 conn(0x7f4f8c0a2d90 msgr2=0x7f4f8c0a0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:50.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.926+0000 7f4f91efe640 1 -- 172.21.15.12:0/395143322 shutdown_connections 2024-09-11T01:37:50.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:50.926+0000 7f4f91efe640 1 -- 172.21.15.12:0/395143322 wait complete. 2024-09-11T01:37:53.940 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:37:54.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.050+0000 7f5e531cd640 10 monclient: get_monmap_and_config 2024-09-11T01:37:54.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.050+0000 7f5e531cd640 10 monclient: build_initial_monmap 2024-09-11T01:37:54.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.050+0000 7f5e531cd640 10 monclient: monmap: 2024-09-11T01:37:54.053 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:54.053 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:54.053 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:37:54.052571+0000 2024-09-11T01:37:54.054 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:37:54.052571+0000 2024-09-11T01:37:54.054 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:54.054 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:54.054 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:37:54.054 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:37:54.054 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:37:54.054 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:37:54.054 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:37:54.054 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:54.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.051+0000 7f5e531cd640 1 Processor -- start 2024-09-11T01:37:54.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.051+0000 7f5e531cd640 1 -- start start 2024-09-11T01:37:54.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.051+0000 7f5e531cd640 10 monclient: init 2024-09-11T01:37:54.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e531cd640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:54.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e531cd640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-11T01:37:54.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e531cd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e4c09d790 0x7f5e4c09db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:54.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e531cd640 10 monclient(hunting): picked mon.noname-b con 0x7f5e4c09d790 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:54.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e531cd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e4c09e0b0 addr v1:172.21.15.71:6789/0 2024-09-11T01:37:54.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e531cd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5e4c09e4d0 0x7f5e4c135170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:54.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e531cd640 10 monclient(hunting): picked mon.noname-c con 0x7f5e4c09e4d0 addr v2:172.21.15.71:3300/0 2024-09-11T01:37:54.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e531cd640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e4c0a1da0 con 0x7f5e4c09e0b0 2024-09-11T01:37:54.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e531cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:54.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e531cd640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f5e4c0a0760 con 0x7f5e4c09e4d0 2024-09-11T01:37:54.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e531cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:54.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e531cd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5e4c09e8f0 con 0x7f5e4c09d790 2024-09-11T01:37:54.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e531cd640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:54.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e531cd640 10 monclient(hunting): authenticate will time out at 1280.495257s 2024-09-11T01:37:54.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e50f69640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e4c09d790 0x7f5e4c09db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:54.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e4bfff640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f5e4c09e0b0 legacy=0x7f5e4c134a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:37:54.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e5176a640 1 -- >> v2:172.21.15.71:3300/0 conn(0x7f5e4c09e4d0 msgr2=0x7f5e4c135170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:37:54.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e5176a640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5e4c09e4d0 0x7f5e4c135170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:37:54.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e50f69640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e4c09d790 0x7f5e4c09db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41140/0 (socket says 172.21.15.12:41140) 2024-09-11T01:37:54.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e50f69640 1 -- 172.21.15.12:0/2964363294 learned_addr learned my addr 172.21.15.12:0/2964363294 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:54.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e50f69640 10 monclient(hunting): get_auth_request con 0x7f5e4c09d790 auth_method 0 2024-09-11T01:37:54.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e50f69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:54.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e50f69640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:54.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e50f69640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:54.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e50f69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:54.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e50f69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:54.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e50f69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:54.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e50f69640 10 monclient(hunting): handle_auth_done global_id 84263 payload 274 2024-09-11T01:37:54.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e50f69640 1 -- 172.21.15.12:0/2964363294 >> v2:172.21.15.71:3300/0 conn(0x7f5e4c09e4d0 msgr2=0x7f5e4c135170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:37:54.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e50f69640 1 --2- 172.21.15.12:0/2964363294 >> v2:172.21.15.71:3300/0 conn(0x7f5e4c09e4d0 0x7f5e4c135170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:54.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e50f69640 1 -- 172.21.15.12:0/2964363294 >> v1:172.21.15.71:6789/0 conn(0x7f5e4c09e0b0 legacy=0x7f5e4c134a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:37:54.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e50f69640 10 monclient: _finish_hunting 0 2024-09-11T01:37:54.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e50f69640 1 monclient: found mon.noname-b 2024-09-11T01:37:54.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e50f69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:54.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e50f69640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:37:54.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.052+0000 7f5e50f69640 1 -- 172.21.15.12:0/2964363294 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e4c0a1fa0 con 0x7f5e4c09d790 2024-09-11T01:37:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.055+0000 7f5e50f69640 1 -- 172.21.15.12:0/2964363294 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e4c09d790 msgr2=0x7f5e4c09db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.055+0000 7f5e50f69640 1 --2- 172.21.15.12:0/2964363294 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e4c09d790 0x7f5e4c09db70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c009870 tx=0x7f5e3c02a0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25bf8027e7efebef server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.055+0000 7f5e4b7fe640 1 -- 172.21.15.12:0/2964363294 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c037070 con 0x7f5e4c09d790 2024-09-11T01:37:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.055+0000 7f5e4b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.055+0000 7f5e4b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:37:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.055+0000 7f5e4b7fe640 10 monclient: dump: 2024-09-11T01:37:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:54.065 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:54.066 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:54.066 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:54.066 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:54.066 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:54.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.055+0000 7f5e4b7fe640 10 monclient: _finish_auth 0 2024-09-11T01:37:54.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.055+0000 7f5e4b7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:37:54.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.055+0000 7f5e4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:54.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.055+0000 7f5e4b7fe640 1 -- 172.21.15.12:0/2964363294 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e3c032640 con 0x7f5e4c09d790 2024-09-11T01:37:54.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.055+0000 7f5e531cd640 5 monclient: authenticate success, global_id 84263 2024-09-11T01:37:54.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.055+0000 7f5e531cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:37:54.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.055+0000 7f5e4b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:54.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.055+0000 7f5e531cd640 10 monclient: get_monmap_and_config success 2024-09-11T01:37:54.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.055+0000 7f5e4b7fe640 1 -- 172.21.15.12:0/2964363294 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c032a10 con 0x7f5e4c09d790 2024-09-11T01:37:54.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.055+0000 7f5e531cd640 4 set_mon_vals no callback set 2024-09-11T01:37:54.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.055+0000 7f5e4b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:54.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.055+0000 7f5e4b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:54.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.055+0000 7f5e4b7fe640 10 monclient: dump: 2024-09-11T01:37:54.067 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:54.067 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:54.067 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:54.067 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:54.067 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:54.067 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:54.067 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:54.067 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:54.068 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:54.068 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:54.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.056+0000 7f5e531cd640 10 monclient: shutdown 2024-09-11T01:37:54.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.056+0000 7f5e531cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:54.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.056+0000 7f5e531cd640 1 -- 172.21.15.12:0/2964363294 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e4c09d790 msgr2=0x7f5e4c09db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:54.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.056+0000 7f5e531cd640 1 --2- 172.21.15.12:0/2964363294 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e4c09d790 0x7f5e4c09db70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c009870 tx=0x7f5e3c02a0d0 comp rx=0 tx=0).stop 2024-09-11T01:37:54.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.056+0000 7f5e531cd640 1 -- 172.21.15.12:0/2964363294 shutdown_connections 2024-09-11T01:37:54.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.056+0000 7f5e531cd640 1 --2- 172.21.15.12:0/2964363294 >> v2:172.21.15.71:3300/0 conn(0x7f5e4c09e4d0 0x7f5e4c135170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:54.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.056+0000 7f5e531cd640 1 --2- 172.21.15.12:0/2964363294 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e4c09d790 0x7f5e4c09db70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:54.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.056+0000 7f5e531cd640 1 -- 172.21.15.12:0/2964363294 >> 172.21.15.12:0/2964363294 conn(0x7f5e4c0a2d90 msgr2=0x7f5e4c0a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:54.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.056+0000 7f5e531cd640 1 -- 172.21.15.12:0/2964363294 shutdown_connections 2024-09-11T01:37:54.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.056+0000 7f5e531cd640 1 -- 172.21.15.12:0/2964363294 wait complete. 2024-09-11T01:37:54.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.057+0000 7f5e531cd640 10 monclient: build_initial_monmap 2024-09-11T01:37:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.057+0000 7f5e531cd640 10 monclient: monmap: 2024-09-11T01:37:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:37:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:37:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:37:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:37:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:37:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:37:54.069 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.057+0000 7f5e531cd640 1 Processor -- start 2024-09-11T01:37:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.057+0000 7f5e531cd640 1 -- start start 2024-09-11T01:37:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.057+0000 7f5e531cd640 10 monclient: init 2024-09-11T01:37:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e531cd640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e531cd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:37:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e531cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e4c09d790 0x7f5e4c13bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e531cd640 10 monclient(hunting): picked mon.noname-a con 0x7f5e4c09d790 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:37:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e531cd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e4c09e0b0 0x7f5e4c1cb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e531cd640 10 monclient(hunting): picked mon.noname-c con 0x7f5e4c09e0b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e531cd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e4c09e4d0 0x7f5e4c139720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e531cd640 10 monclient(hunting): picked mon.noname-b con 0x7f5e4c09e4d0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:37:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e531cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e531cd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5e4c0a23a0 con 0x7f5e4c09d790 2024-09-11T01:37:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e531cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e531cd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5e4c0a2220 con 0x7f5e4c09e4d0 2024-09-11T01:37:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e531cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e531cd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5e4c09cca0 con 0x7f5e4c09e0b0 2024-09-11T01:37:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e531cd640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e531cd640 10 monclient(hunting): authenticate will time out at 1280.501222s 2024-09-11T01:37:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e50f69640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e4c09d790 0x7f5e4c13bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e4bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e4c09e0b0 0x7f5e4c1cb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e50f69640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e4c09d790 0x7f5e4c13bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58128/0 (socket says 172.21.15.12:58128) 2024-09-11T01:37:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e50f69640 1 -- 172.21.15.12:0/3386072236 learned_addr learned my addr 172.21.15.12:0/3386072236 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e50f69640 10 monclient(hunting): get_auth_request con 0x7f5e4c09d790 auth_method 0 2024-09-11T01:37:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e50f69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e50f69640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e50f69640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e4bfff640 10 monclient(hunting): get_auth_request con 0x7f5e4c09e0b0 auth_method 0 2024-09-11T01:37:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e5176a640 1 -- 172.21.15.12:0/3386072236 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e4c09e4d0 msgr2=0x7f5e4c139720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:37:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e5176a640 1 --2- 172.21.15.12:0/3386072236 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e4c09e4d0 0x7f5e4c139720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:37:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e50f69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e50f69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e50f69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e50f69640 10 monclient(hunting): handle_auth_done global_id 94173 payload 824 2024-09-11T01:37:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e50f69640 1 -- 172.21.15.12:0/3386072236 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e4c09e0b0 msgr2=0x7f5e4c1cb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e50f69640 1 --2- 172.21.15.12:0/3386072236 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e4c09e0b0 0x7f5e4c1cb5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e50f69640 1 -- 172.21.15.12:0/3386072236 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e4c09e4d0 msgr2=0x7f5e4c139720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:37:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e50f69640 1 --2- 172.21.15.12:0/3386072236 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e4c09e4d0 0x7f5e4c139720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:54.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e50f69640 10 monclient: _finish_hunting 0 2024-09-11T01:37:54.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e50f69640 1 monclient: found mon.noname-a 2024-09-11T01:37:54.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e50f69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:54.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e50f69640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:37:54.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.058+0000 7f5e50f69640 1 -- 172.21.15.12:0/3386072236 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e4c0a5450 con 0x7f5e4c09d790 2024-09-11T01:37:54.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.060+0000 7f5e50f69640 1 -- 172.21.15.12:0/3386072236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e4c09d790 msgr2=0x7f5e4c13bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:54.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.060+0000 7f5e50f69640 1 --2- 172.21.15.12:0/3386072236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e4c09d790 0x7f5e4c13bca0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c009870 tx=0x7f5e3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:54.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.061+0000 7f5e497fa640 1 -- 172.21.15.12:0/3386072236 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c041070 con 0x7f5e4c09d790 2024-09-11T01:37:54.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.061+0000 7f5e497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:54.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.061+0000 7f5e497fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:37:54.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.061+0000 7f5e497fa640 10 monclient: dump: 2024-09-11T01:37:54.073 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:54.074 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:54.074 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:54.074 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:54.074 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:54.074 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:54.074 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:54.074 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:54.074 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:54.074 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:54.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.061+0000 7f5e497fa640 10 monclient: _finish_auth 0 2024-09-11T01:37:54.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.061+0000 7f5e497fa640 10 monclient: _check_auth_tickets 2024-09-11T01:37:54.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.061+0000 7f5e497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:54.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.061+0000 7f5e531cd640 5 monclient: authenticate success, global_id 94173 2024-09-11T01:37:54.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.061+0000 7f5e531cd640 10 monclient: _renew_subs 2024-09-11T01:37:54.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.061+0000 7f5e497fa640 1 -- 172.21.15.12:0/3386072236 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e3c03c880 con 0x7f5e4c09d790 2024-09-11T01:37:54.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.061+0000 7f5e531cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:54.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.061+0000 7f5e531cd640 1 -- 172.21.15.12:0/3386072236 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e4c11ce10 con 0x7f5e4c09d790 2024-09-11T01:37:54.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.061+0000 7f5e497fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:54.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.061+0000 7f5e497fa640 1 -- 172.21.15.12:0/3386072236 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c037040 con 0x7f5e4c09d790 2024-09-11T01:37:54.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.061+0000 7f5e497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:54.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.061+0000 7f5e497fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:54.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.061+0000 7f5e497fa640 10 monclient: dump: 2024-09-11T01:37:54.075 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:54.075 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:54.075 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:54.075 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:54.076 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:54.076 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:54.076 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:54.076 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:54.076 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:54.076 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:54.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.061+0000 7f5e531cd640 10 monclient: _renew_subs 2024-09-11T01:37:54.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.061+0000 7f5e531cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:54.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.061+0000 7f5e531cd640 1 -- 172.21.15.12:0/3386072236 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e4c13c630 con 0x7f5e4c09d790 2024-09-11T01:37:54.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.061+0000 7f5e4affd640 4 set_mon_vals no callback set 2024-09-11T01:37:54.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.062+0000 7f5e497fa640 1 -- 172.21.15.12:0/3386072236 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f5e3c03cc50 con 0x7f5e4c09d790 2024-09-11T01:37:54.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.062+0000 7f5e531cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:54.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.062+0000 7f5e497fa640 1 --2- 172.21.15.12:0/3386072236 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5e3803e240 0x7f5e38040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:54.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.062+0000 7f5e531cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:54.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.062+0000 7f5e531cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:54.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.062+0000 7f5e531cd640 1 -- 172.21.15.12:0/3386072236 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e4c135840 con 0x7f5e4c09d790 2024-09-11T01:37:54.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.062+0000 7f5e497fa640 1 -- 172.21.15.12:0/3386072236 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(287..287 src has 186..287) ==== 4815+0+0 (secure 0 0 0) 0x7f5e3c0730c0 con 0x7f5e4c09d790 2024-09-11T01:37:54.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.062+0000 7f5e4bfff640 1 --2- 172.21.15.12:0/3386072236 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5e3803e240 0x7f5e38040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:54.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.062+0000 7f5e4bfff640 10 monclient: get_auth_request con 0x7f5e3803e240 auth_method 0 2024-09-11T01:37:54.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.065+0000 7f5e4bfff640 1 --2- 172.21.15.12:0/3386072236 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5e3803e240 0x7f5e38040700 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5e40009370 tx=0x7f5e40009300 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:54.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.067+0000 7f5e497fa640 1 -- 172.21.15.12:0/3386072236 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5e3c04b020 con 0x7f5e4c09d790 2024-09-11T01:37:54.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.067+0000 7f5e497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:54.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.067+0000 7f5e497fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:37:54.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.324+0000 7f5e531cd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:37:54.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.324+0000 7f5e531cd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:37:54.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.324+0000 7f5e531cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:54.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:54.324+0000 7f5e531cd640 1 -- 172.21.15.12:0/3386072236 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5e4c06d460 con 0x7f5e4c09d790 2024-09-11T01:37:57.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:57.058+0000 7f5e33fff640 10 monclient: tick 2024-09-11T01:37:57.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:57.058+0000 7f5e33fff640 10 monclient: _check_auth_tickets 2024-09-11T01:37:57.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:57.058+0000 7f5e33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:57.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:57.058+0000 7f5e33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:58.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:58.059+0000 7f5e33fff640 10 monclient: tick 2024-09-11T01:37:58.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:58.059+0000 7f5e33fff640 10 monclient: _check_auth_tickets 2024-09-11T01:37:58.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:58.059+0000 7f5e33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:59.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.059+0000 7f5e33fff640 10 monclient: tick 2024-09-11T01:37:59.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.059+0000 7f5e33fff640 10 monclient: _check_auth_tickets 2024-09-11T01:37:59.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.059+0000 7f5e33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:59.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.077+0000 7f5e497fa640 1 -- 172.21.15.12:0/3386072236 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f5e4c0a5450 con 0x7f5e4c09d790 2024-09-11T01:37:59.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.077+0000 7f5e497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:37:59.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.077+0000 7f5e497fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:37:59.080 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":90,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:37:59.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.080+0000 7f5e531cd640 1 -- 172.21.15.12:0/3386072236 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5e3803e240 msgr2=0x7f5e38040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:59.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.080+0000 7f5e531cd640 1 --2- 172.21.15.12:0/3386072236 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5e3803e240 0x7f5e38040700 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5e40009370 tx=0x7f5e40009300 comp rx=0 tx=0).stop 2024-09-11T01:37:59.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.080+0000 7f5e531cd640 10 monclient: shutdown 2024-09-11T01:37:59.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.080+0000 7f5e531cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:59.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.080+0000 7f5e531cd640 1 -- 172.21.15.12:0/3386072236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e4c09d790 msgr2=0x7f5e4c13bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:59.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.080+0000 7f5e531cd640 1 --2- 172.21.15.12:0/3386072236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e4c09d790 0x7f5e4c13bca0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c009870 tx=0x7f5e3c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:37:59.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.080+0000 7f5e531cd640 1 -- 172.21.15.12:0/3386072236 shutdown_connections 2024-09-11T01:37:59.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.080+0000 7f5e531cd640 1 --2- 172.21.15.12:0/3386072236 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e4c09e4d0 0x7f5e4c139720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:59.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.080+0000 7f5e531cd640 1 --2- 172.21.15.12:0/3386072236 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e4c09e0b0 0x7f5e4c1cb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:59.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.080+0000 7f5e531cd640 1 --2- 172.21.15.12:0/3386072236 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5e3803e240 0x7f5e38040700 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:59.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.080+0000 7f5e531cd640 1 --2- 172.21.15.12:0/3386072236 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e4c09d790 0x7f5e4c13bca0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:59.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.080+0000 7f5e531cd640 1 -- 172.21.15.12:0/3386072236 >> 172.21.15.12:0/3386072236 conn(0x7f5e4c0a2d90 msgr2=0x7f5e4c0a4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:59.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.080+0000 7f5e531cd640 1 -- 172.21.15.12:0/3386072236 shutdown_connections 2024-09-11T01:37:59.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.080+0000 7f5e531cd640 1 -- 172.21.15.12:0/3386072236 wait complete. 2024-09-11T01:37:59.098 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:37:59.098 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-11T01:37:59.099 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:37:59.101 INFO:tasks.rados.rados.0.smithi071.stdout:1284: done (6 left) 2024-09-11T01:37:59.101 INFO:tasks.rados.rados.0.smithi071.stdout:1285: write oid 23 current snap is 132 2024-09-11T01:37:59.102 INFO:tasks.rados.rados.0.smithi071.stdout:1280: finishing copy_from racing read to smithi07134777-38 2024-09-11T01:37:59.102 INFO:tasks.rados.rados.0.smithi071.stdout:1279: finishing rollback tid 0 to smithi07134777-45 2024-09-11T01:37:59.102 INFO:tasks.rados.rados.0.smithi071.stdout:1276: finishing rollback tid 0 to smithi07134777-4 2024-09-11T01:37:59.102 INFO:tasks.rados.rados.0.smithi071.stdout:1276: finishing rollback tid 1 to smithi07134777-4 2024-09-11T01:37:59.102 INFO:tasks.rados.rados.0.smithi071.stdout:1276: finishing rollback tid 2 to smithi07134777-4 2024-09-11T01:37:59.102 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 432 (ObjNum 462 snap 122 seq_num 462) dirty exists 2024-09-11T01:37:59.102 INFO:tasks.rados.rados.0.smithi071.stdout:1278: finishing write tid 1 to smithi07134777-19 2024-09-11T01:37:59.102 INFO:tasks.rados.rados.0.smithi071.stdout:1279: finishing rollback tid 1 to smithi07134777-45 2024-09-11T01:37:59.102 INFO:tasks.rados.rados.0.smithi071.stdout:1285: seq_num 492 ranges {669044=469100,1741023=459161} 2024-09-11T01:37:59.107 INFO:tasks.rados.rados.0.smithi071.stdout:1285: writing smithi07134777-23 from 669044 to 1138144 tid 1 2024-09-11T01:37:59.114 INFO:tasks.rados.rados.0.smithi071.stdout:1285: writing smithi07134777-23 from 1741023 to 2200184 tid 2 2024-09-11T01:37:59.114 INFO:tasks.rados.rados.0.smithi071.stdout:1276: done (6 left) 2024-09-11T01:37:59.114 INFO:tasks.rados.rados.0.smithi071.stdout:1280: done (5 left) 2024-09-11T01:37:59.114 INFO:tasks.rados.rados.0.smithi071.stdout:1286: write oid 34 current snap is 132 2024-09-11T01:37:59.114 INFO:tasks.rados.rados.0.smithi071.stdout:1279: finishing rollback tid 2 to smithi07134777-45 2024-09-11T01:37:59.114 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 525 (ObjNum 437 snap 119 seq_num 437) dirty exists 2024-09-11T01:37:59.114 INFO:tasks.rados.rados.0.smithi071.stdout:1281: finishing write tid 1 to smithi07134777-30 2024-09-11T01:37:59.114 INFO:tasks.rados.rados.0.smithi071.stdout:1278: finishing write tid 2 to smithi07134777-19 2024-09-11T01:37:59.114 INFO:tasks.rados.rados.0.smithi071.stdout:1281: finishing write tid 2 to smithi07134777-30 2024-09-11T01:37:59.114 INFO:tasks.rados.rados.0.smithi071.stdout:1278: finishing write tid 3 to smithi07134777-19 2024-09-11T01:37:59.114 INFO:tasks.rados.rados.0.smithi071.stdout:1286: seq_num 493 ranges {685851=544349,1799505=486346} 2024-09-11T01:37:59.121 INFO:tasks.rados.rados.0.smithi071.stdout:1286: writing smithi07134777-34 from 685851 to 1230200 tid 1 2024-09-11T01:37:59.127 INFO:tasks.rados.rados.0.smithi071.stdout:1286: writing smithi07134777-34 from 1799505 to 2285851 tid 2 2024-09-11T01:37:59.127 INFO:tasks.rados.rados.0.smithi071.stdout:1279: done (5 left) 2024-09-11T01:37:59.127 INFO:tasks.rados.rados.0.smithi071.stdout:1287: snap_remove snap 131 2024-09-11T01:37:59.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.208+0000 7f88703c2640 10 monclient: get_monmap_and_config 2024-09-11T01:37:59.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.208+0000 7f88703c2640 10 monclient: build_initial_monmap 2024-09-11T01:37:59.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.208+0000 7f88703c2640 10 monclient: monmap: 2024-09-11T01:37:59.211 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:59.211 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:59.211 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:37:59.210283+0000 2024-09-11T01:37:59.211 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:37:59.210283+0000 2024-09-11T01:37:59.211 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:59.211 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:59.211 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:37:59.211 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:37:59.212 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:37:59.212 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:37:59.212 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:37:59.212 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:59.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.209+0000 7f88703c2640 1 Processor -- start 2024-09-11T01:37:59.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.209+0000 7f88703c2640 1 -- start start 2024-09-11T01:37:59.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.209+0000 7f88703c2640 10 monclient: init 2024-09-11T01:37:59.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.209+0000 7f88703c2640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:59.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.209+0000 7f88703c2640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-11T01:37:59.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.209+0000 7f88703c2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88680a2e20 0x7f886806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:59.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.209+0000 7f88703c2640 10 monclient(hunting): picked mon.noname-b con 0x7f88680a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:59.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.209+0000 7f88703c2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f88680a4970 addr v1:172.21.15.12:6789/0 2024-09-11T01:37:59.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.209+0000 7f88703c2640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f886806f910 0x7f886813f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:59.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.209+0000 7f88703c2640 10 monclient(hunting): picked mon.noname-a con 0x7f886806f910 addr v2:172.21.15.12:3300/0 2024-09-11T01:37:59.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.209+0000 7f88703c2640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f886806f4f0 con 0x7f88680a4970 2024-09-11T01:37:59.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.209+0000 7f88703c2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:59.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.209+0000 7f88703c2640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f886806ef30 con 0x7f886806f910 2024-09-11T01:37:59.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.209+0000 7f88703c2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:59.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.209+0000 7f88703c2640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f886806ec50 con 0x7f88680a2e20 2024-09-11T01:37:59.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.209+0000 7f88703c2640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:59.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.209+0000 7f88703c2640 10 monclient(hunting): authenticate will time out at 1285.652873s 2024-09-11T01:37:59.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.209+0000 7f886e15e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88680a2e20 0x7f886806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:59.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886e95f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f886806f910 0x7f886813f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:59.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886d95d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f88680a4970 0x7f88680a3200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35122/0 (socket says 172.21.15.12:35122) 2024-09-11T01:37:59.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886e15e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88680a2e20 0x7f886806ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:59710/0 (socket says 172.21.15.12:59710) 2024-09-11T01:37:59.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886d95d640 1 -- 172.21.15.12:0/1229956620 learned_addr learned my addr 172.21.15.12:0/1229956620 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:37:59.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886e15e640 10 monclient(hunting): get_auth_request con 0x7f88680a2e20 auth_method 0 2024-09-11T01:37:59.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886e15e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:59.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886e15e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:59.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886e15e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:59.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886e95f640 10 monclient(hunting): get_auth_request con 0x7f886806f910 auth_method 0 2024-09-11T01:37:59.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886e95f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:59.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886e95f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:59.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886e95f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:59.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886e15e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:59.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886e15e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:59.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886e15e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:59.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886e95f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:59.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886e95f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:59.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886e95f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:59.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886e95f640 10 monclient(hunting): handle_auth_done global_id 94176 payload 274 2024-09-11T01:37:59.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886e95f640 1 -- 172.21.15.12:0/1229956620 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88680a2e20 msgr2=0x7f886806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:59.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886e95f640 1 --2- 172.21.15.12:0/1229956620 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88680a2e20 0x7f886806ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:59.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886e95f640 1 -- 172.21.15.12:0/1229956620 >> v1:172.21.15.12:6789/0 conn(0x7f88680a4970 legacy=0x7f88680a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:59.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886e95f640 10 monclient: _finish_hunting 0 2024-09-11T01:37:59.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886e95f640 1 monclient: found mon.noname-a 2024-09-11T01:37:59.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886e95f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:59.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886e95f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:37:59.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886e95f640 1 -- 172.21.15.12:0/1229956620 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88680a3910 con 0x7f886806f910 2024-09-11T01:37:59.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.210+0000 7f886e15e640 1 --2- 172.21.15.12:0/1229956620 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88680a2e20 0x7f886806ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:37:59.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.213+0000 7f886e95f640 1 -- 172.21.15.12:0/1229956620 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886806f910 msgr2=0x7f886813f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:59.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.213+0000 7f886e95f640 1 --2- 172.21.15.12:0/1229956620 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886806f910 0x7f886813f430 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8860009d90 tx=0x7f886002a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa61ce9e4d034fa1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:59.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.213+0000 7f886d15c640 1 -- 172.21.15.12:0/1229956620 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8860007950 con 0x7f886806f910 2024-09-11T01:37:59.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.213+0000 7f886d15c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:59.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.213+0000 7f886d15c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:37:59.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.213+0000 7f886d15c640 10 monclient: dump: 2024-09-11T01:37:59.225 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:59.225 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:59.225 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:59.225 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:59.225 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:59.225 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:59.225 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:59.226 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:59.226 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:59.226 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:59.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.213+0000 7f886d15c640 10 monclient: _finish_auth 0 2024-09-11T01:37:59.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.213+0000 7f886d15c640 10 monclient: _check_auth_tickets 2024-09-11T01:37:59.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.213+0000 7f886d15c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:59.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.213+0000 7f886d15c640 1 -- 172.21.15.12:0/1229956620 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8860007af0 con 0x7f886806f910 2024-09-11T01:37:59.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.213+0000 7f886d15c640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:59.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.213+0000 7f886d15c640 1 -- 172.21.15.12:0/1229956620 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8860031400 con 0x7f886806f910 2024-09-11T01:37:59.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.213+0000 7f88703c2640 5 monclient: authenticate success, global_id 94176 2024-09-11T01:37:59.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.213+0000 7f88703c2640 10 monclient: get_monmap_and_config success 2024-09-11T01:37:59.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.213+0000 7f88703c2640 4 set_mon_vals no callback set 2024-09-11T01:37:59.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.213+0000 7f886d15c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:59.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.213+0000 7f886d15c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:59.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.213+0000 7f886d15c640 10 monclient: dump: 2024-09-11T01:37:59.227 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:59.227 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:59.227 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:59.227 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:59.227 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:59.227 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:59.227 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:59.227 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:59.227 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:59.227 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:59.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.213+0000 7f88703c2640 10 monclient: shutdown 2024-09-11T01:37:59.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.213+0000 7f88703c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:59.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.213+0000 7f88703c2640 1 -- 172.21.15.12:0/1229956620 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886806f910 msgr2=0x7f886813f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:59.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.213+0000 7f88703c2640 1 --2- 172.21.15.12:0/1229956620 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886806f910 0x7f886813f430 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8860009d90 tx=0x7f886002a490 comp rx=0 tx=0).stop 2024-09-11T01:37:59.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.213+0000 7f88703c2640 1 -- 172.21.15.12:0/1229956620 shutdown_connections 2024-09-11T01:37:59.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.213+0000 7f88703c2640 1 --2- 172.21.15.12:0/1229956620 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886806f910 0x7f886813f430 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:59.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.213+0000 7f88703c2640 1 --2- 172.21.15.12:0/1229956620 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88680a2e20 0x7f886806ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:59.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.213+0000 7f88703c2640 1 -- 172.21.15.12:0/1229956620 >> 172.21.15.12:0/1229956620 conn(0x7f88680a8600 msgr2=0x7f88680a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:59.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.214+0000 7f88703c2640 1 -- 172.21.15.12:0/1229956620 shutdown_connections 2024-09-11T01:37:59.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.214+0000 7f88703c2640 1 -- 172.21.15.12:0/1229956620 wait complete. 2024-09-11T01:37:59.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.214+0000 7f88703c2640 10 monclient: build_initial_monmap 2024-09-11T01:37:59.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.214+0000 7f88703c2640 10 monclient: monmap: 2024-09-11T01:37:59.228 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:59.228 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:37:59.228 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:37:59.229 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:37:59.229 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:59.229 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:59.229 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:37:59.229 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:37:59.229 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:37:59.229 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:59.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.214+0000 7f88703c2640 1 Processor -- start 2024-09-11T01:37:59.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.214+0000 7f88703c2640 1 -- start start 2024-09-11T01:37:59.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f88703c2640 10 monclient: init 2024-09-11T01:37:59.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f88703c2640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:59.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f88703c2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:37:59.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f88703c2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f886806f910 0x7f886813dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:59.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f88703c2640 10 monclient(hunting): picked mon.noname-b con 0x7f886806f910 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:37:59.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f88703c2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88680a2e20 0x7f886813bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:59.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f88703c2640 10 monclient(hunting): picked mon.noname-c con 0x7f88680a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:59.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f88703c2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88680a4970 0x7f8868138460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:59.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f88703c2640 10 monclient(hunting): picked mon.noname-a con 0x7f88680a4970 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:37:59.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f88703c2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:59.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f88703c2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f886813fd00 con 0x7f88680a4970 2024-09-11T01:37:59.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f88703c2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:59.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f88703c2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f886813fb80 con 0x7f886806f910 2024-09-11T01:37:59.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f88703c2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:59.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f88703c2640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f886813fe80 con 0x7f88680a2e20 2024-09-11T01:37:59.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f88703c2640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:59.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f88703c2640 10 monclient(hunting): authenticate will time out at 1285.658510s 2024-09-11T01:37:59.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886d95d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88680a2e20 0x7f886813bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:59.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886e95f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88680a4970 0x7f8868138460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:59.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886e15e640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f886806f910 msgr2=0x7f886813dbd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:37:59.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886d95d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88680a2e20 0x7f886813bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:59712/0 (socket says 172.21.15.12:59712) 2024-09-11T01:37:59.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886e95f640 10 monclient(hunting): get_auth_request con 0x7f88680a4970 auth_method 0 2024-09-11T01:37:59.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886d95d640 1 -- 172.21.15.12:0/3928511038 learned_addr learned my addr 172.21.15.12:0/3928511038 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:59.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886e15e640 1 --2- 172.21.15.12:0/3928511038 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f886806f910 0x7f886813dbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:37:59.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886e95f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:59.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886e95f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:59.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886e95f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:59.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886d95d640 10 monclient(hunting): get_auth_request con 0x7f88680a2e20 auth_method 0 2024-09-11T01:37:59.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886d95d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:59.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886d95d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:59.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886d95d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:59.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886e95f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:59.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886e95f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:59.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886e95f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:59.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886d95d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:59.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886d95d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:59.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886d95d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:59.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886e95f640 10 monclient(hunting): handle_auth_done global_id 94179 payload 824 2024-09-11T01:37:59.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886e95f640 1 -- 172.21.15.12:0/3928511038 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88680a2e20 msgr2=0x7f886813bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:59.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886e95f640 1 --2- 172.21.15.12:0/3928511038 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88680a2e20 0x7f886813bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:59.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886e95f640 1 -- 172.21.15.12:0/3928511038 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f886806f910 msgr2=0x7f886813dbd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:37:59.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886e95f640 1 --2- 172.21.15.12:0/3928511038 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f886806f910 0x7f886813dbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:59.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886e95f640 10 monclient: _finish_hunting 0 2024-09-11T01:37:59.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886e95f640 1 monclient: found mon.noname-a 2024-09-11T01:37:59.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886e95f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:59.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886e95f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:37:59.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.215+0000 7f886e95f640 1 -- 172.21.15.12:0/3928511038 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f886813d830 con 0x7f88680a4970 2024-09-11T01:37:59.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.218+0000 7f886e95f640 1 -- 172.21.15.12:0/3928511038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88680a4970 msgr2=0x7f8868138460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:59.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.218+0000 7f886e95f640 1 --2- 172.21.15.12:0/3928511038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88680a4970 0x7f8868138460 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8860031000 tx=0x7f8860004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:59.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.218+0000 7f885effd640 1 -- 172.21.15.12:0/3928511038 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8860040070 con 0x7f88680a4970 2024-09-11T01:37:59.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.218+0000 7f885effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:59.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.218+0000 7f885effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:37:59.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.218+0000 7f885effd640 10 monclient: dump: 2024-09-11T01:37:59.234 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:59.234 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:59.234 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:59.234 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:59.234 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:59.234 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:59.234 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:59.235 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:59.235 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:59.235 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:59.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.218+0000 7f885effd640 10 monclient: _finish_auth 0 2024-09-11T01:37:59.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.218+0000 7f885effd640 10 monclient: _check_auth_tickets 2024-09-11T01:37:59.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.218+0000 7f885effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:59.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.218+0000 7f885effd640 1 -- 172.21.15.12:0/3928511038 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8860045460 con 0x7f88680a4970 2024-09-11T01:37:59.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.218+0000 7f88703c2640 5 monclient: authenticate success, global_id 94179 2024-09-11T01:37:59.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.218+0000 7f88703c2640 10 monclient: _renew_subs 2024-09-11T01:37:59.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.218+0000 7f88703c2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:59.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.218+0000 7f88703c2640 1 -- 172.21.15.12:0/3928511038 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88680a6570 con 0x7f88680a4970 2024-09-11T01:37:59.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.218+0000 7f885effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:59.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.218+0000 7f885effd640 1 -- 172.21.15.12:0/3928511038 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8860045850 con 0x7f88680a4970 2024-09-11T01:37:59.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.218+0000 7f885effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:59.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.218+0000 7f885effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:59.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.218+0000 7f885effd640 10 monclient: dump: 2024-09-11T01:37:59.236 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:59.236 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:59.236 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:59.236 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:59.236 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:59.236 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:59.236 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:59.236 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:59.236 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:59.236 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:59.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.218+0000 7f886c95b640 4 set_mon_vals no callback set 2024-09-11T01:37:59.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.218+0000 7f88703c2640 10 monclient: _renew_subs 2024-09-11T01:37:59.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.218+0000 7f88703c2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:59.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.218+0000 7f88703c2640 1 -- 172.21.15.12:0/3928511038 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f886813d070 con 0x7f88680a4970 2024-09-11T01:37:59.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.219+0000 7f885effd640 1 -- 172.21.15.12:0/3928511038 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f886004d030 con 0x7f88680a4970 2024-09-11T01:37:59.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.219+0000 7f88703c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:59.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.219+0000 7f88703c2640 1 --2- 172.21.15.12:0/3928511038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88681389a0 0x7f88681cd960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:59.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.219+0000 7f88703c2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:59.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.219+0000 7f88703c2640 1 -- 172.21.15.12:0/3928511038 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f886806a060 con 0x7f88681389a0 2024-09-11T01:37:59.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.219+0000 7f885effd640 1 --2- 172.21.15.12:0/3928511038 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f884c03e210 0x7f884c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:59.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.219+0000 7f886e15e640 1 --2- 172.21.15.12:0/3928511038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88681389a0 0x7f88681cd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:59.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.219+0000 7f886e15e640 10 monclient: get_auth_request con 0x7f88681389a0 auth_method 0 2024-09-11T01:37:59.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.219+0000 7f885effd640 1 -- 172.21.15.12:0/3928511038 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(288..288 src has 186..288) ==== 4799+0+0 (secure 0 0 0) 0x7f8860074590 con 0x7f88680a4970 2024-09-11T01:37:59.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.219+0000 7f886e15e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:59.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.219+0000 7f886e15e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:59.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.219+0000 7f886e15e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:59.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.219+0000 7f886e15e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:59.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.219+0000 7f886e15e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:59.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.219+0000 7f886e15e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:59.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.219+0000 7f886e15e640 10 monclient(hunting): handle_auth_done global_id 94182 payload 824 2024-09-11T01:37:59.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.219+0000 7f886e15e640 1 -- 172.21.15.12:0/3928511038 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f886806ded0 con 0x7f88681389a0 2024-09-11T01:37:59.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.219+0000 7f886d95d640 1 --2- 172.21.15.12:0/3928511038 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f884c03e210 0x7f884c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:59.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.219+0000 7f886d95d640 10 monclient: get_auth_request con 0x7f884c03e210 auth_method 0 2024-09-11T01:37:59.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.222+0000 7f886e15e640 1 -- 172.21.15.12:0/3928511038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88681389a0 msgr2=0x7f88681cd960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:59.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.222+0000 7f886e15e640 1 --2- 172.21.15.12:0/3928511038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88681389a0 0x7f88681cd960 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8864009630 tx=0x7f886400eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:59.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.222+0000 7f885effd640 1 -- 172.21.15.12:0/3928511038 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886400cbc0 con 0x7f88681389a0 2024-09-11T01:37:59.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.222+0000 7f885effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:59.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.222+0000 7f885effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:59.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.222+0000 7f885effd640 10 monclient: dump: 2024-09-11T01:37:59.239 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:59.239 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:59.239 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:59.239 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:59.239 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:59.239 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:59.239 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:59.239 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:59.239 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:59.239 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:59.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.222+0000 7f885effd640 1 -- 172.21.15.12:0/3928511038 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f886400cbc0 con 0x7f88681389a0 2024-09-11T01:37:59.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.222+0000 7f885effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:59.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.222+0000 7f885effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:37:59.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.222+0000 7f885effd640 1 -- 172.21.15.12:0/3928511038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88681389a0 msgr2=0x7f88681cd960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:59.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.222+0000 7f885effd640 1 --2- 172.21.15.12:0/3928511038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88681389a0 0x7f88681cd960 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8864009630 tx=0x7f886400eb90 comp rx=0 tx=0).stop 2024-09-11T01:37:59.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.222+0000 7f885effd640 1 -- 172.21.15.12:0/3928511038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88681389a0 msgr2=0x7f88681cd960 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:37:59.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.222+0000 7f885effd640 1 --2- 172.21.15.12:0/3928511038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88681389a0 0x7f88681cd960 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8864009630 tx=0x7f886400eb90 comp rx=0 tx=0).stop 2024-09-11T01:37:59.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.222+0000 7f886d95d640 1 --2- 172.21.15.12:0/3928511038 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f884c03e210 0x7f884c0406d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8858007d50 tx=0x7f8858007ca0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:59.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.228+0000 7f88703c2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:37:59.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.228+0000 7f88703c2640 1 --2- 172.21.15.12:0/3928511038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886802d440 0x7f886802d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:59.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.228+0000 7f88703c2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:59.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.228+0000 7f88703c2640 1 -- 172.21.15.12:0/3928511038 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f88680a2830 con 0x7f886802d440 2024-09-11T01:37:59.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.228+0000 7f886e15e640 1 --2- 172.21.15.12:0/3928511038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886802d440 0x7f886802d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:59.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.228+0000 7f886e15e640 10 monclient: get_auth_request con 0x7f886802d440 auth_method 0 2024-09-11T01:37:59.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.228+0000 7f886e15e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:59.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.228+0000 7f886e15e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:59.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.228+0000 7f886e15e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:59.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.228+0000 7f886e15e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:59.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.228+0000 7f886e15e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:59.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.228+0000 7f886e15e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:59.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.228+0000 7f886e15e640 10 monclient(hunting): handle_auth_done global_id 94185 payload 824 2024-09-11T01:37:59.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.228+0000 7f886e15e640 1 -- 172.21.15.12:0/3928511038 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f886809da30 con 0x7f886802d440 2024-09-11T01:37:59.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.231+0000 7f886e15e640 1 -- 172.21.15.12:0/3928511038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886802d440 msgr2=0x7f886802d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:59.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.231+0000 7f886e15e640 1 --2- 172.21.15.12:0/3928511038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886802d440 0x7f886802d820 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8864018000 tx=0x7f8864004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:59.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.231+0000 7f885effd640 1 -- 172.21.15.12:0/3928511038 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8864003820 con 0x7f886802d440 2024-09-11T01:37:59.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.231+0000 7f885effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:59.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.231+0000 7f885effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:59.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.231+0000 7f885effd640 10 monclient: dump: 2024-09-11T01:37:59.243 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:59.243 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:59.244 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:59.244 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:59.244 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:59.244 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:59.244 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:59.244 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:59.244 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:59.244 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:59.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.231+0000 7f885effd640 1 -- 172.21.15.12:0/3928511038 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4698 (secure 0 0 0) 0x7f8864003820 con 0x7f886802d440 2024-09-11T01:37:59.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.231+0000 7f885effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:37:59.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.231+0000 7f885effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:37:59.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.231+0000 7f885effd640 1 -- 172.21.15.12:0/3928511038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886802d440 msgr2=0x7f886802d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:59.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.231+0000 7f885effd640 1 --2- 172.21.15.12:0/3928511038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886802d440 0x7f886802d820 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8864018000 tx=0x7f8864004730 comp rx=0 tx=0).stop 2024-09-11T01:37:59.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.231+0000 7f885effd640 1 -- 172.21.15.12:0/3928511038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886802d440 msgr2=0x7f886802d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:37:59.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.231+0000 7f885effd640 1 --2- 172.21.15.12:0/3928511038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886802d440 0x7f886802d820 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8864018000 tx=0x7f8864004730 comp rx=0 tx=0).stop 2024-09-11T01:37:59.245 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:37:59.245 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:37:59.245 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:37:59.245 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:37:59.245 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 90, 2024-09-11T01:37:59.245 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 25327, 2024-09-11T01:37:59.245 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:37:59.246 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:37:59.246 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:37:59.246 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:59.246 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:37:59.246 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:37:59.246 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:37:59.246 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:37:59.246 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:59.246 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:59.246 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:59.246 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:59.246 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:59.246 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:59.247 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:59.247 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:59.247 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:59.247 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:59.247 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:59.247 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:59.247 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:37:59.247 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:37:59.247 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:59.247 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:59.247 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:59.247 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:59.247 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:59.248 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:59.248 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:59.248 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:59.248 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:59.248 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:59.248 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:59.248 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:59.248 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:59.248 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:37:59.248 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:37:59.248 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:37:59.248 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:37:59.248 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:37:59.248 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:37:59.249 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:37:59.249 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:37:59.249 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:37:59.249 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:37:59.249 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:37:59.249 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:37:59.249 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:37:59.249 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:37:59.249 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:37:59.249 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:37:59.249 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:37:59.249 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:59.249 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:37:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:37:59.250 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:37:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:37:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:37:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:37:59.251 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:37:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:37:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:37:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:37:59.252 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:37:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:37:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:59.253 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:59.254 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:37:59.254 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:37:59.254 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:59.254 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:59.254 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:59.254 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:59.254 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:59.254 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:37:59.254 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:37:59.254 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:59.254 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:59.254 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:59.254 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:37:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:37:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:37:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:37:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:59.255 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:37:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:37:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:37:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:59.256 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:37:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:37:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:37:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:37:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:59.257 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:59.258 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:37:59.258 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:37:59.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.231+0000 7f88703c2640 1 -- 172.21.15.12:0/3928511038 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f884c03e210 msgr2=0x7f884c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:59.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.231+0000 7f88703c2640 1 --2- 172.21.15.12:0/3928511038 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f884c03e210 0x7f884c0406d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8858007d50 tx=0x7f8858007ca0 comp rx=0 tx=0).stop 2024-09-11T01:37:59.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.231+0000 7f88703c2640 10 monclient: shutdown 2024-09-11T01:37:59.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.231+0000 7f88703c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:59.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.231+0000 7f88703c2640 1 -- 172.21.15.12:0/3928511038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88680a4970 msgr2=0x7f8868138460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:59.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.231+0000 7f886e95f640 1 -- 172.21.15.12:0/3928511038 reap_dead start 2024-09-11T01:37:59.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.231+0000 7f88703c2640 1 --2- 172.21.15.12:0/3928511038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88680a4970 0x7f8868138460 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8860031000 tx=0x7f8860004a40 comp rx=0 tx=0).stop 2024-09-11T01:37:59.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.231+0000 7f88703c2640 1 -- 172.21.15.12:0/3928511038 shutdown_connections 2024-09-11T01:37:59.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.231+0000 7f88703c2640 1 --2- 172.21.15.12:0/3928511038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88680a4970 0x7f8868138460 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:59.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.232+0000 7f88703c2640 1 -- 172.21.15.12:0/3928511038 >> 172.21.15.12:0/3928511038 conn(0x7f88680a8600 msgr2=0x7f88680a1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:59.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.232+0000 7f88703c2640 1 -- 172.21.15.12:0/3928511038 shutdown_connections 2024-09-11T01:37:59.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.232+0000 7f88703c2640 1 -- 172.21.15.12:0/3928511038 wait complete. 2024-09-11T01:37:59.260 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:37:59.260 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T01:37:59.260 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:37:59.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.359+0000 7f8b3a2bd640 10 monclient: get_monmap_and_config 2024-09-11T01:37:59.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.359+0000 7f8b3a2bd640 10 monclient: build_initial_monmap 2024-09-11T01:37:59.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.359+0000 7f8b3a2bd640 10 monclient: monmap: 2024-09-11T01:37:59.362 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:59.362 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:59.362 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:37:59.361280+0000 2024-09-11T01:37:59.362 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:37:59.361280+0000 2024-09-11T01:37:59.362 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:59.362 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:59.362 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:37:59.362 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:37:59.363 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:37:59.363 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:37:59.363 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:37:59.363 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.360+0000 7f8b3a2bd640 1 Processor -- start 2024-09-11T01:37:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.360+0000 7f8b3a2bd640 1 -- start start 2024-09-11T01:37:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.360+0000 7f8b3a2bd640 10 monclient: init 2024-09-11T01:37:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.360+0000 7f8b3a2bd640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.360+0000 7f8b3a2bd640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-11T01:37:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.360+0000 7f8b3a2bd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8b340a2e20 0x7f8b3406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.360+0000 7f8b3a2bd640 10 monclient(hunting): picked mon.noname-a con 0x7f8b340a2e20 addr v2:172.21.15.12:3300/0 2024-09-11T01:37:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.360+0000 7f8b3a2bd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b340a4970 0x7f8b340a3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.360+0000 7f8b3a2bd640 10 monclient(hunting): picked mon.noname-b con 0x7f8b340a4970 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.360+0000 7f8b3a2bd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f8b3406f910 0x7f8b340a3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.360+0000 7f8b3a2bd640 10 monclient(hunting): picked mon.noname-c con 0x7f8b3406f910 addr v2:172.21.15.71:3300/0 2024-09-11T01:37:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.360+0000 7f8b3a2bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.360+0000 7f8b3a2bd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f8b3406ef30 con 0x7f8b340a2e20 2024-09-11T01:37:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.360+0000 7f8b3a2bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.360+0000 7f8b3a2bd640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f8b340a3c80 con 0x7f8b3406f910 2024-09-11T01:37:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.360+0000 7f8b3a2bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.360+0000 7f8b3a2bd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8b3406ec50 con 0x7f8b340a4970 2024-09-11T01:37:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.360+0000 7f8b3a2bd640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.360+0000 7f8b3a2bd640 10 monclient(hunting): authenticate will time out at 1285.803853s 2024-09-11T01:37:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.360+0000 7f8b33fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8b340a2e20 0x7f8b3406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.360+0000 7f8b33fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8b340a2e20 0x7f8b3406ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38762/0 (socket says 172.21.15.12:38762) 2024-09-11T01:37:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b33fff640 1 -- 172.21.15.12:0/2348483861 learned_addr learned my addr 172.21.15.12:0/2348483861 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b33fff640 10 monclient(hunting): get_auth_request con 0x7f8b340a2e20 auth_method 0 2024-09-11T01:37:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b33fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b3885a640 1 -- 172.21.15.12:0/2348483861 >> v2:172.21.15.71:3300/0 conn(0x7f8b3406f910 msgr2=0x7f8b340a3740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:37:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b337fe640 1 --2- 172.21.15.12:0/2348483861 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b340a4970 0x7f8b340a3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b3885a640 1 --2- 172.21.15.12:0/2348483861 >> v2:172.21.15.71:3300/0 conn(0x7f8b3406f910 0x7f8b340a3740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:37:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b337fe640 10 monclient(hunting): get_auth_request con 0x7f8b340a4970 auth_method 0 2024-09-11T01:37:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b337fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b33fff640 10 monclient(hunting): handle_auth_done global_id 94188 payload 274 2024-09-11T01:37:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b33fff640 1 -- 172.21.15.12:0/2348483861 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b340a4970 msgr2=0x7f8b340a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:37:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b33fff640 1 --2- 172.21.15.12:0/2348483861 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b340a4970 0x7f8b340a3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b33fff640 1 -- 172.21.15.12:0/2348483861 >> v2:172.21.15.71:3300/0 conn(0x7f8b3406f910 msgr2=0x7f8b340a3740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:37:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b33fff640 1 --2- 172.21.15.12:0/2348483861 >> v2:172.21.15.71:3300/0 conn(0x7f8b3406f910 0x7f8b340a3740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b33fff640 10 monclient: _finish_hunting 0 2024-09-11T01:37:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b33fff640 1 monclient: found mon.noname-a 2024-09-11T01:37:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b33fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:37:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.361+0000 7f8b33fff640 1 -- 172.21.15.12:0/2348483861 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b3406f4f0 con 0x7f8b340a2e20 2024-09-11T01:37:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b33fff640 1 -- 172.21.15.12:0/2348483861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b340a2e20 msgr2=0x7f8b3406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b33fff640 1 --2- 172.21.15.12:0/2348483861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b340a2e20 0x7f8b3406ded0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8b34074f40 tx=0x7f8b2402a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ee23808903bb03 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b32ffd640 1 -- 172.21.15.12:0/2348483861 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b24037070 con 0x7f8b340a2e20 2024-09-11T01:37:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b32ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:37:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b32ffd640 10 monclient: dump: 2024-09-11T01:37:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:59.375 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b32ffd640 10 monclient: _finish_auth 0 2024-09-11T01:37:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b32ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:37:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b32ffd640 1 -- 172.21.15.12:0/2348483861 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b24032640 con 0x7f8b340a2e20 2024-09-11T01:37:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b3a2bd640 5 monclient: authenticate success, global_id 94188 2024-09-11T01:37:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b3a2bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:37:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b32ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b3a2bd640 10 monclient: get_monmap_and_config success 2024-09-11T01:37:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b32ffd640 1 -- 172.21.15.12:0/2348483861 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b24032a10 con 0x7f8b340a2e20 2024-09-11T01:37:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b3a2bd640 4 set_mon_vals no callback set 2024-09-11T01:37:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b32ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b32ffd640 10 monclient: dump: 2024-09-11T01:37:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:59.377 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b3a2bd640 10 monclient: shutdown 2024-09-11T01:37:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b3a2bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b3a2bd640 1 -- 172.21.15.12:0/2348483861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b340a2e20 msgr2=0x7f8b3406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b3a2bd640 1 --2- 172.21.15.12:0/2348483861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b340a2e20 0x7f8b3406ded0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8b34074f40 tx=0x7f8b2402a0d0 comp rx=0 tx=0).stop 2024-09-11T01:37:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b3a2bd640 1 -- 172.21.15.12:0/2348483861 shutdown_connections 2024-09-11T01:37:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b3a2bd640 1 --2- 172.21.15.12:0/2348483861 >> v2:172.21.15.71:3300/0 conn(0x7f8b3406f910 0x7f8b340a3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b3a2bd640 1 --2- 172.21.15.12:0/2348483861 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b340a4970 0x7f8b340a3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b3a2bd640 1 --2- 172.21.15.12:0/2348483861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b340a2e20 0x7f8b3406ded0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b3a2bd640 1 -- 172.21.15.12:0/2348483861 >> 172.21.15.12:0/2348483861 conn(0x7f8b340a8600 msgr2=0x7f8b340a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b3a2bd640 1 -- 172.21.15.12:0/2348483861 shutdown_connections 2024-09-11T01:37:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.364+0000 7f8b3a2bd640 1 -- 172.21.15.12:0/2348483861 wait complete. 2024-09-11T01:37:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.365+0000 7f8b3a2bd640 10 monclient: build_initial_monmap 2024-09-11T01:37:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.365+0000 7f8b3a2bd640 10 monclient: monmap: 2024-09-11T01:37:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:37:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:37:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:37:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:37:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:37:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:37:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:37:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:37:59.378 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.365+0000 7f8b3a2bd640 1 Processor -- start 2024-09-11T01:37:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.365+0000 7f8b3a2bd640 1 -- start start 2024-09-11T01:37:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.365+0000 7f8b3a2bd640 10 monclient: init 2024-09-11T01:37:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b3a2bd640 10 monclient: _reopen_session rank -1 2024-09-11T01:37:59.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b3a2bd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:37:59.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b3a2bd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b3406f910 0x7f8b341cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:59.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b3a2bd640 10 monclient(hunting): picked mon.noname-a con 0x7f8b3406f910 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:37:59.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b3a2bd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b340a2e20 0x7f8b341cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:59.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b3a2bd640 10 monclient(hunting): picked mon.noname-b con 0x7f8b340a2e20 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:37:59.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b3a2bd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b340a4970 0x7f8b341249f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:59.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b3a2bd640 10 monclient(hunting): picked mon.noname-c con 0x7f8b340a4970 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:37:59.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b3a2bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:59.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b3a2bd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8b34138440 con 0x7f8b3406f910 2024-09-11T01:37:59.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b3a2bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:59.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b3a2bd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8b341382c0 con 0x7f8b340a2e20 2024-09-11T01:37:59.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b3a2bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:59.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b3a2bd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8b341385c0 con 0x7f8b340a4970 2024-09-11T01:37:59.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b3a2bd640 10 monclient(hunting): _renew_subs 2024-09-11T01:37:59.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b3a2bd640 10 monclient(hunting): authenticate will time out at 1285.809394s 2024-09-11T01:37:59.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b33fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b3406f910 0x7f8b341cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:59.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b337fe640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b340a2e20 msgr2=0x7f8b341cd9b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:37:59.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b33fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b3406f910 0x7f8b341cd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38768/0 (socket says 172.21.15.12:38768) 2024-09-11T01:37:59.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b337fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b340a2e20 0x7f8b341cd9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:37:59.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b33fff640 1 -- 172.21.15.12:0/4019220118 learned_addr learned my addr 172.21.15.12:0/4019220118 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:37:59.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b33fff640 10 monclient(hunting): get_auth_request con 0x7f8b3406f910 auth_method 0 2024-09-11T01:37:59.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:59.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b33fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:59.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:59.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b3885a640 1 --2- 172.21.15.12:0/4019220118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b340a4970 0x7f8b341249f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:59.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b3885a640 10 monclient(hunting): get_auth_request con 0x7f8b340a4970 auth_method 0 2024-09-11T01:37:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b3885a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b3885a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b3885a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b3885a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b3885a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b3885a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b33fff640 10 monclient(hunting): handle_auth_done global_id 94191 payload 824 2024-09-11T01:37:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b33fff640 1 -- 172.21.15.12:0/4019220118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b340a4970 msgr2=0x7f8b341249f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:59.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b33fff640 1 --2- 172.21.15.12:0/4019220118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b340a4970 0x7f8b341249f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:59.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b33fff640 1 -- 172.21.15.12:0/4019220118 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b340a2e20 msgr2=0x7f8b341cd9b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:37:59.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b33fff640 1 --2- 172.21.15.12:0/4019220118 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b340a2e20 0x7f8b341cd9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:37:59.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b33fff640 10 monclient: _finish_hunting 0 2024-09-11T01:37:59.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b33fff640 1 monclient: found mon.noname-a 2024-09-11T01:37:59.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:37:59.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b33fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:37:59.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.366+0000 7f8b33fff640 1 -- 172.21.15.12:0/4019220118 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b340738a0 con 0x7f8b3406f910 2024-09-11T01:37:59.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.369+0000 7f8b33fff640 1 -- 172.21.15.12:0/4019220118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b3406f910 msgr2=0x7f8b341cd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:59.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.369+0000 7f8b33fff640 1 --2- 172.21.15.12:0/4019220118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b3406f910 0x7f8b341cd470 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8b24009960 tx=0x7f8b240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:59.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.369+0000 7f8b30ff9640 1 -- 172.21.15.12:0/4019220118 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b24041070 con 0x7f8b3406f910 2024-09-11T01:37:59.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.369+0000 7f8b30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:59.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.369+0000 7f8b30ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:37:59.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.369+0000 7f8b30ff9640 10 monclient: dump: 2024-09-11T01:37:59.382 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:59.383 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:59.383 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:59.383 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:59.383 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:59.383 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:59.383 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:59.383 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:59.383 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:59.383 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:59.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.369+0000 7f8b30ff9640 10 monclient: _finish_auth 0 2024-09-11T01:37:59.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.369+0000 7f8b30ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:37:59.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.369+0000 7f8b30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:37:59.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.369+0000 7f8b30ff9640 1 -- 172.21.15.12:0/4019220118 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b2403c880 con 0x7f8b3406f910 2024-09-11T01:37:59.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.369+0000 7f8b3a2bd640 5 monclient: authenticate success, global_id 94191 2024-09-11T01:37:59.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.369+0000 7f8b30ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:37:59.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.369+0000 7f8b30ff9640 1 -- 172.21.15.12:0/4019220118 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b24037040 con 0x7f8b3406f910 2024-09-11T01:37:59.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.369+0000 7f8b3a2bd640 10 monclient: _renew_subs 2024-09-11T01:37:59.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.369+0000 7f8b3a2bd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:59.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.369+0000 7f8b3a2bd640 1 -- 172.21.15.12:0/4019220118 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b34073df0 con 0x7f8b3406f910 2024-09-11T01:37:59.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.369+0000 7f8b30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:59.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.369+0000 7f8b30ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:37:59.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.369+0000 7f8b30ff9640 10 monclient: dump: 2024-09-11T01:37:59.384 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:59.384 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:59.384 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:59.385 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:59.385 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:59.385 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:59.385 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:59.385 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:59.385 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:59.385 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:59.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.369+0000 7f8b327fc640 4 set_mon_vals no callback set 2024-09-11T01:37:59.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.369+0000 7f8b3a2bd640 10 monclient: _renew_subs 2024-09-11T01:37:59.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.369+0000 7f8b3a2bd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:37:59.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.369+0000 7f8b3a2bd640 1 -- 172.21.15.12:0/4019220118 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b34128810 con 0x7f8b3406f910 2024-09-11T01:37:59.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.370+0000 7f8b3a2bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:59.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.370+0000 7f8b3a2bd640 1 --2- 172.21.15.12:0/4019220118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b341256f0 0x7f8b3402d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:59.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.370+0000 7f8b3a2bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:59.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.370+0000 7f8b3a2bd640 1 -- 172.21.15.12:0/4019220118 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8b3406a060 con 0x7f8b341256f0 2024-09-11T01:37:59.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.370+0000 7f8b30ff9640 1 -- 172.21.15.12:0/4019220118 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f8b2403cc50 con 0x7f8b3406f910 2024-09-11T01:37:59.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.370+0000 7f8b337fe640 1 --2- 172.21.15.12:0/4019220118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b341256f0 0x7f8b3402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:59.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.370+0000 7f8b337fe640 10 monclient: get_auth_request con 0x7f8b341256f0 auth_method 0 2024-09-11T01:37:59.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.370+0000 7f8b337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:59.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.370+0000 7f8b337fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:59.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.370+0000 7f8b337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:59.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.370+0000 7f8b337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:59.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.370+0000 7f8b337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:59.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.370+0000 7f8b337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:59.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.370+0000 7f8b337fe640 10 monclient(hunting): handle_auth_done global_id 84281 payload 824 2024-09-11T01:37:59.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.370+0000 7f8b30ff9640 1 --2- 172.21.15.12:0/4019220118 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8b2003e210 0x7f8b200406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:59.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.370+0000 7f8b337fe640 1 -- 172.21.15.12:0/4019220118 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8b3406ded0 con 0x7f8b341256f0 2024-09-11T01:37:59.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.370+0000 7f8b30ff9640 1 -- 172.21.15.12:0/4019220118 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(288..288 src has 186..288) ==== 4799+0+0 (secure 0 0 0) 0x7f8b240733e0 con 0x7f8b3406f910 2024-09-11T01:37:59.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.370+0000 7f8b3885a640 1 --2- 172.21.15.12:0/4019220118 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8b2003e210 0x7f8b200406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:59.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.370+0000 7f8b3885a640 10 monclient: get_auth_request con 0x7f8b2003e210 auth_method 0 2024-09-11T01:37:59.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.372+0000 7f8b337fe640 1 -- 172.21.15.12:0/4019220118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b341256f0 msgr2=0x7f8b3402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:59.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.372+0000 7f8b337fe640 1 --2- 172.21.15.12:0/4019220118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b341256f0 0x7f8b3402d440 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8b280094a0 tx=0x7f8b2800ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:59.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.372+0000 7f8b30ff9640 1 -- 172.21.15.12:0/4019220118 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b28017070 con 0x7f8b341256f0 2024-09-11T01:37:59.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.372+0000 7f8b30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:59.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.372+0000 7f8b30ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:59.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.372+0000 7f8b30ff9640 10 monclient: dump: 2024-09-11T01:37:59.390 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:59.390 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:59.390 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:59.390 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:59.390 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:59.390 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:59.390 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:59.390 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:59.390 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:59.390 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:59.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.373+0000 7f8b30ff9640 1 -- 172.21.15.12:0/4019220118 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8b2800cb90 con 0x7f8b341256f0 2024-09-11T01:37:59.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.373+0000 7f8b30ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:37:59.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.373+0000 7f8b30ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:37:59.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.373+0000 7f8b30ff9640 1 -- 172.21.15.12:0/4019220118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b341256f0 msgr2=0x7f8b3402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:59.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.373+0000 7f8b30ff9640 1 --2- 172.21.15.12:0/4019220118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b341256f0 0x7f8b3402d440 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8b280094a0 tx=0x7f8b2800ec70 comp rx=0 tx=0).stop 2024-09-11T01:37:59.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.373+0000 7f8b30ff9640 1 -- 172.21.15.12:0/4019220118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b341256f0 msgr2=0x7f8b3402d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:37:59.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.373+0000 7f8b30ff9640 1 --2- 172.21.15.12:0/4019220118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b341256f0 0x7f8b3402d440 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8b280094a0 tx=0x7f8b2800ec70 comp rx=0 tx=0).stop 2024-09-11T01:37:59.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.373+0000 7f8b3885a640 1 --2- 172.21.15.12:0/4019220118 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8b2003e210 0x7f8b200406d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8b34126da0 tx=0x7f8b2c0095f0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:59.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.379+0000 7f8b3a2bd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:37:59.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.379+0000 7f8b3a2bd640 1 --2- 172.21.15.12:0/4019220118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b34125ad0 0x7f8b3402d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:37:59.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.379+0000 7f8b3a2bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:37:59.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.379+0000 7f8b3a2bd640 1 -- 172.21.15.12:0/4019220118 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8b3409da30 con 0x7f8b34125ad0 2024-09-11T01:37:59.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.379+0000 7f8b337fe640 1 --2- 172.21.15.12:0/4019220118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b34125ad0 0x7f8b3402d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:37:59.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.379+0000 7f8b337fe640 10 monclient: get_auth_request con 0x7f8b34125ad0 auth_method 0 2024-09-11T01:37:59.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.379+0000 7f8b337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:37:59.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.379+0000 7f8b337fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:37:59.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.379+0000 7f8b337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:37:59.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.379+0000 7f8b337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:37:59.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.379+0000 7f8b337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:37:59.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.379+0000 7f8b337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:37:59.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.379+0000 7f8b337fe640 10 monclient(hunting): handle_auth_done global_id 84284 payload 824 2024-09-11T01:37:59.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.379+0000 7f8b337fe640 1 -- 172.21.15.12:0/4019220118 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8b34069950 con 0x7f8b34125ad0 2024-09-11T01:37:59.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.381+0000 7f8b337fe640 1 -- 172.21.15.12:0/4019220118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b34125ad0 msgr2=0x7f8b3402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:37:59.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.381+0000 7f8b337fe640 1 --2- 172.21.15.12:0/4019220118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b34125ad0 0x7f8b3402d980 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8b28018000 tx=0x7f8b28004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:37:59.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.381+0000 7f8b30ff9640 1 -- 172.21.15.12:0/4019220118 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b28017070 con 0x7f8b34125ad0 2024-09-11T01:37:59.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.381+0000 7f8b30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:37:59.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.381+0000 7f8b30ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:37:59.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.381+0000 7f8b30ff9640 10 monclient: dump: 2024-09-11T01:37:59.393 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:37:59.393 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:37:59.393 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:59.393 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:37:59.393 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:37:59.393 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:37:59.393 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:37:59.393 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:37:59.393 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:37:59.393 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:37:59.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.382+0000 7f8b30ff9640 1 -- 172.21.15.12:0/4019220118 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4863 (secure 0 0 0) 0x7f8b28003610 con 0x7f8b34125ad0 2024-09-11T01:37:59.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.382+0000 7f8b30ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:37:59.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.382+0000 7f8b30ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:37:59.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.382+0000 7f8b30ff9640 1 -- 172.21.15.12:0/4019220118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b34125ad0 msgr2=0x7f8b3402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:59.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.382+0000 7f8b30ff9640 1 --2- 172.21.15.12:0/4019220118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b34125ad0 0x7f8b3402d980 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8b28018000 tx=0x7f8b28004730 comp rx=0 tx=0).stop 2024-09-11T01:37:59.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.382+0000 7f8b30ff9640 1 -- 172.21.15.12:0/4019220118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b34125ad0 msgr2=0x7f8b3402d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:37:59.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.382+0000 7f8b30ff9640 1 --2- 172.21.15.12:0/4019220118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b34125ad0 0x7f8b3402d980 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8b28018000 tx=0x7f8b28004730 comp rx=0 tx=0).stop 2024-09-11T01:37:59.397 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:37:59.397 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:37:59.397 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:37:59.397 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:37:59.397 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 90, 2024-09-11T01:37:59.397 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 73405, 2024-09-11T01:37:59.397 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:37:59.397 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:37:59.397 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:37:59.397 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:59.398 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:37:59.398 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:37:59.398 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:37:59.398 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:37:59.398 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:59.398 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:59.398 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:59.398 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:59.398 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:59.398 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:59.398 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:59.398 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:59.398 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:59.399 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:59.399 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:59.399 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:59.399 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:37:59.399 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:37:59.399 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:59.399 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:59.399 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:59.399 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:59.399 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:59.399 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:59.399 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:59.400 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:59.400 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:59.400 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:59.400 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:59.400 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:59.400 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:59.400 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:37:59.400 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:37:59.400 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:37:59.400 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:37:59.400 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:37:59.400 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:37:59.400 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:37:59.401 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:37:59.401 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:37:59.401 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:37:59.401 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:37:59.401 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:37:59.401 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:37:59.401 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:37:59.401 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:37:59.401 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:37:59.401 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:37:59.401 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:37:59.401 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:37:59.401 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:37:59.402 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:37:59.402 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:37:59.402 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:37:59.402 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:37:59.402 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:37:59.402 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:37:59.402 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:37:59.402 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:37:59.402 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:59.402 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:37:59.402 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:59.402 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:37:59.402 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:59.403 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:37:59.403 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:37:59.403 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:59.403 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:59.403 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:59.403 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:59.403 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:37:59.403 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:59.403 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:59.403 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:59.403 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:59.403 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:37:59.403 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:59.404 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:59.404 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:59.404 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:59.404 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:37:59.404 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:37:59.404 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:59.404 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:59.404 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:59.404 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:59.404 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:59.404 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:37:59.404 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:37:59.404 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:59.404 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:59.405 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:59.405 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:59.405 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:37:59.405 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:59.405 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:59.405 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:59.405 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:59.405 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:37:59.405 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:59.405 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:59.405 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:59.405 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:59.405 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:37:59.406 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:37:59.406 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:59.406 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:59.406 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:59.406 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:59.406 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:59.406 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:37:59.406 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:37:59.406 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:37:59.406 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:37:59.406 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:59.406 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:37:59.406 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:37:59.407 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:59.407 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:59.407 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:59.407 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:37:59.407 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:37:59.407 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:37:59.407 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:59.407 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:59.407 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:59.407 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:37:59.407 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:37:59.407 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:37:59.407 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:37:59.407 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:37:59.408 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:59.408 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:59.408 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:59.408 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:37:59.408 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:37:59.408 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:59.408 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:59.408 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:59.408 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:37:59.408 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:59.408 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:59.408 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:37:59.408 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:59.408 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:59.408 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:59.409 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:37:59.409 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:59.409 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:59.409 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:37:59.409 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:59.409 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:59.409 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:59.409 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:37:59.409 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:59.409 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:37:59.409 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:37:59.409 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:37:59.409 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:37:59.410 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:37:59.410 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:37:59.410 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:37:59.410 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:37:59.410 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:37:59.410 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:37:59.410 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:37:59.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.382+0000 7f8b3a2bd640 1 -- 172.21.15.12:0/4019220118 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8b2003e210 msgr2=0x7f8b200406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:59.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.382+0000 7f8b3a2bd640 1 --2- 172.21.15.12:0/4019220118 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8b2003e210 0x7f8b200406d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8b34126da0 tx=0x7f8b2c0095f0 comp rx=0 tx=0).stop 2024-09-11T01:37:59.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.382+0000 7f8b3a2bd640 10 monclient: shutdown 2024-09-11T01:37:59.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.382+0000 7f8b3a2bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:37:59.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.382+0000 7f8b3a2bd640 1 -- 172.21.15.12:0/4019220118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b3406f910 msgr2=0x7f8b341cd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:37:59.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.382+0000 7f8b3a2bd640 1 --2- 172.21.15.12:0/4019220118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b3406f910 0x7f8b341cd470 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8b24009960 tx=0x7f8b240046c0 comp rx=0 tx=0).stop 2024-09-11T01:37:59.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.382+0000 7f8b3885a640 1 -- 172.21.15.12:0/4019220118 reap_dead start 2024-09-11T01:37:59.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.382+0000 7f8b3885a640 1 -- 172.21.15.12:0/4019220118 reap_dead start 2024-09-11T01:37:59.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.382+0000 7f8b3a2bd640 1 -- 172.21.15.12:0/4019220118 shutdown_connections 2024-09-11T01:37:59.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.382+0000 7f8b3a2bd640 1 -- 172.21.15.12:0/4019220118 >> 172.21.15.12:0/4019220118 conn(0x7f8b340a8600 msgr2=0x7f8b340a1830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:37:59.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.382+0000 7f8b3a2bd640 1 -- 172.21.15.12:0/4019220118 shutdown_connections 2024-09-11T01:37:59.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:37:59.382+0000 7f8b3a2bd640 1 -- 172.21.15.12:0/4019220118 wait complete. 2024-09-11T01:37:59.413 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:37:59.413 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:38:00.107 INFO:tasks.rados.rados.0.smithi071.stdout:1287: done (5 left) 2024-09-11T01:38:00.108 INFO:tasks.rados.rados.0.smithi071.stdout:1288: write oid 48 current snap is 132 2024-09-11T01:38:00.108 INFO:tasks.rados.rados.0.smithi071.stdout:1288: seq_num 494 ranges {702658=619598,1857987=444671} 2024-09-11T01:38:00.114 INFO:tasks.rados.rados.0.smithi071.stdout:1288: writing smithi07134777-48 from 702658 to 1322256 tid 1 2024-09-11T01:38:00.121 INFO:tasks.rados.rados.0.smithi071.stdout:1288: writing smithi07134777-48 from 1857987 to 2302658 tid 2 2024-09-11T01:38:00.121 INFO:tasks.rados.rados.0.smithi071.stdout:1289: read oid 36 snap -1 2024-09-11T01:38:00.121 INFO:tasks.rados.rados.0.smithi071.stdout:1289: expect (ObjNum 472 snap 125 seq_num 472) 2024-09-11T01:38:00.121 INFO:tasks.rados.rados.0.smithi071.stdout:1290: write oid 31 current snap is 132 2024-09-11T01:38:00.121 INFO:tasks.rados.rados.0.smithi071.stdout:1290: seq_num 495 ranges {719465=411200,1632822=686643} 2024-09-11T01:38:00.127 INFO:tasks.rados.rados.0.smithi071.stdout:1290: writing smithi07134777-31 from 719465 to 1130665 tid 1 2024-09-11T01:38:00.134 INFO:tasks.rados.rados.0.smithi071.stdout:1290: writing smithi07134777-31 from 1632822 to 2319465 tid 2 2024-09-11T01:38:00.134 INFO:tasks.rados.rados.0.smithi071.stdout:1291: snap_remove snap 128 2024-09-11T01:38:01.112 INFO:tasks.rados.rados.0.smithi071.stdout:1291: done (8 left) 2024-09-11T01:38:01.113 INFO:tasks.rados.rados.0.smithi071.stdout:1292: delete oid 14 current snap is 132 2024-09-11T01:38:01.113 INFO:tasks.rados.rados.0.smithi071.stdout:1281: finishing write tid 3 to smithi07134777-30 2024-09-11T01:38:01.113 INFO:tasks.rados.rados.0.smithi071.stdout:1278: finishing write tid 4 to smithi07134777-19 2024-09-11T01:38:01.113 INFO:tasks.rados.rados.0.smithi071.stdout:1278: oid 19 updating version 0 to 444 2024-09-11T01:38:01.113 INFO:tasks.rados.rados.0.smithi071.stdout:1278: oid 19 version 444 is already newer than 442 2024-09-11T01:38:01.113 INFO:tasks.rados.rados.0.smithi071.stdout:1278: oid 19 version 444 is already newer than 443 2024-09-11T01:38:01.113 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 444 (ObjNum 489 snap 132 seq_num 489) dirty exists 2024-09-11T01:38:01.113 INFO:tasks.rados.rados.0.smithi071.stdout:1278: left oid 19 (ObjNum 489 snap 132 seq_num 489) 2024-09-11T01:38:01.113 INFO:tasks.rados.rados.0.smithi071.stdout:1281: finishing write tid 4 to smithi07134777-30 2024-09-11T01:38:01.113 INFO:tasks.rados.rados.0.smithi071.stdout:1281: oid 30 updating version 0 to 435 2024-09-11T01:38:01.113 INFO:tasks.rados.rados.0.smithi071.stdout:1281: oid 30 version 435 is already newer than 434 2024-09-11T01:38:01.113 INFO:tasks.rados.rados.0.smithi071.stdout:1281: oid 30 updating version 435 to 436 2024-09-11T01:38:01.113 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 436 (ObjNum 490 snap 132 seq_num 490) dirty exists 2024-09-11T01:38:01.114 INFO:tasks.rados.rados.0.smithi071.stdout:1281: left oid 30 (ObjNum 490 snap 132 seq_num 490) 2024-09-11T01:38:01.114 INFO:tasks.rados.rados.0.smithi071.stdout:1283: finishing write tid 1 to smithi07134777-43 2024-09-11T01:38:01.114 INFO:tasks.rados.rados.0.smithi071.stdout:1283: finishing write tid 2 to smithi07134777-43 2024-09-11T01:38:01.114 INFO:tasks.rados.rados.0.smithi071.stdout:1283: finishing write tid 3 to smithi07134777-43 2024-09-11T01:38:01.114 INFO:tasks.rados.rados.0.smithi071.stdout:1283: finishing write tid 4 to smithi07134777-43 2024-09-11T01:38:01.114 INFO:tasks.rados.rados.0.smithi071.stdout:1283: oid 43 updating version 0 to 598 2024-09-11T01:38:01.114 INFO:tasks.rados.rados.0.smithi071.stdout:1283: oid 43 updating version 598 to 600 2024-09-11T01:38:01.114 INFO:tasks.rados.rados.0.smithi071.stdout:1283: oid 43 version 600 is already newer than 599 2024-09-11T01:38:01.114 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 600 (ObjNum 491 snap 132 seq_num 491) dirty exists 2024-09-11T01:38:01.114 INFO:tasks.rados.rados.0.smithi071.stdout:1283: left oid 43 (ObjNum 491 snap 132 seq_num 491) 2024-09-11T01:38:01.114 INFO:tasks.rados.rados.0.smithi071.stdout:1285: finishing write tid 1 to smithi07134777-23 2024-09-11T01:38:01.115 INFO:tasks.rados.rados.0.smithi071.stdout:1285: finishing write tid 2 to smithi07134777-23 2024-09-11T01:38:01.115 INFO:tasks.rados.rados.0.smithi071.stdout:1285: finishing write tid 3 to smithi07134777-23 2024-09-11T01:38:01.115 INFO:tasks.rados.rados.0.smithi071.stdout:1285: finishing write tid 4 to smithi07134777-23 2024-09-11T01:38:01.115 INFO:tasks.rados.rados.0.smithi071.stdout:1285: oid 23 updating version 0 to 255 2024-09-11T01:38:01.115 INFO:tasks.rados.rados.0.smithi071.stdout:1285: oid 23 updating version 255 to 257 2024-09-11T01:38:01.115 INFO:tasks.rados.rados.0.smithi071.stdout:1285: oid 23 version 257 is already newer than 256 2024-09-11T01:38:01.115 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 257 (ObjNum 492 snap 132 seq_num 492) dirty exists 2024-09-11T01:38:01.115 INFO:tasks.rados.rados.0.smithi071.stdout:1285: left oid 23 (ObjNum 492 snap 132 seq_num 492) 2024-09-11T01:38:01.115 INFO:tasks.rados.rados.0.smithi071.stdout:1286: finishing write tid 1 to smithi07134777-34 2024-09-11T01:38:01.115 INFO:tasks.rados.rados.0.smithi071.stdout:1286: finishing write tid 2 to smithi07134777-34 2024-09-11T01:38:01.115 INFO:tasks.rados.rados.0.smithi071.stdout:1286: finishing write tid 3 to smithi07134777-34 2024-09-11T01:38:01.115 INFO:tasks.rados.rados.0.smithi071.stdout:1286: finishing write tid 4 to smithi07134777-34 2024-09-11T01:38:01.116 INFO:tasks.rados.rados.0.smithi071.stdout:1286: oid 34 updating version 0 to 604 2024-09-11T01:38:01.116 INFO:tasks.rados.rados.0.smithi071.stdout:1286: oid 34 updating version 604 to 606 2024-09-11T01:38:01.116 INFO:tasks.rados.rados.0.smithi071.stdout:1286: oid 34 version 606 is already newer than 605 2024-09-11T01:38:01.116 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 606 (ObjNum 493 snap 132 seq_num 493) dirty exists 2024-09-11T01:38:01.116 INFO:tasks.rados.rados.0.smithi071.stdout:1286: left oid 34 (ObjNum 493 snap 132 seq_num 493) 2024-09-11T01:38:01.116 INFO:tasks.rados.rados.0.smithi071.stdout:1288: finishing write tid 1 to smithi07134777-48 2024-09-11T01:38:01.116 INFO:tasks.rados.rados.0.smithi071.stdout:1288: finishing write tid 2 to smithi07134777-48 2024-09-11T01:38:01.116 INFO:tasks.rados.rados.0.smithi071.stdout:1288: finishing write tid 3 to smithi07134777-48 2024-09-11T01:38:01.116 INFO:tasks.rados.rados.0.smithi071.stdout:1288: finishing write tid 4 to smithi07134777-48 2024-09-11T01:38:01.116 INFO:tasks.rados.rados.0.smithi071.stdout:1288: oid 48 updating version 0 to 642 2024-09-11T01:38:01.116 INFO:tasks.rados.rados.0.smithi071.stdout:1288: oid 48 version 642 is already newer than 640 2024-09-11T01:38:01.116 INFO:tasks.rados.rados.0.smithi071.stdout:1288: oid 48 version 642 is already newer than 641 2024-09-11T01:38:01.117 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 642 (ObjNum 494 snap 132 seq_num 494) dirty exists 2024-09-11T01:38:01.117 INFO:tasks.rados.rados.0.smithi071.stdout:1288: left oid 48 (ObjNum 494 snap 132 seq_num 494) 2024-09-11T01:38:01.117 INFO:tasks.rados.rados.0.smithi071.stdout:1289: expect (ObjNum 472 snap 125 seq_num 472) 2024-09-11T01:38:01.202 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:38:01.371 INFO:tasks.rados.rados.0.smithi071.stdout:1290: finishing write tid 1 to smithi07134777-31 2024-09-11T01:38:01.371 INFO:tasks.rados.rados.0.smithi071.stdout:1290: finishing write tid 2 to smithi07134777-31 2024-09-11T01:38:01.372 INFO:tasks.rados.rados.0.smithi071.stdout:1290: finishing write tid 3 to smithi07134777-31 2024-09-11T01:38:01.372 INFO:tasks.rados.rados.0.smithi071.stdout:1290: finishing write tid 4 to smithi07134777-31 2024-09-11T01:38:01.372 INFO:tasks.rados.rados.0.smithi071.stdout:1290: oid 31 updating version 0 to 456 2024-09-11T01:38:01.372 INFO:tasks.rados.rados.0.smithi071.stdout:1290: oid 31 version 456 is already newer than 455 2024-09-11T01:38:01.372 INFO:tasks.rados.rados.0.smithi071.stdout:1290: oid 31 version 456 is already newer than 454 2024-09-11T01:38:01.372 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 456 (ObjNum 495 snap 132 seq_num 495) dirty exists 2024-09-11T01:38:01.372 INFO:tasks.rados.rados.0.smithi071.stdout:1290: left oid 31 (ObjNum 495 snap 132 seq_num 495) 2024-09-11T01:38:01.374 INFO:tasks.rados.rados.0.smithi071.stdout:1278: done (8 left) 2024-09-11T01:38:01.374 INFO:tasks.rados.rados.0.smithi071.stdout:1281: done (7 left) 2024-09-11T01:38:01.374 INFO:tasks.rados.rados.0.smithi071.stdout:1283: done (6 left) 2024-09-11T01:38:01.374 INFO:tasks.rados.rados.0.smithi071.stdout:1285: done (5 left) 2024-09-11T01:38:01.374 INFO:tasks.rados.rados.0.smithi071.stdout:1286: done (4 left) 2024-09-11T01:38:01.374 INFO:tasks.rados.rados.0.smithi071.stdout:1288: done (3 left) 2024-09-11T01:38:01.374 INFO:tasks.rados.rados.0.smithi071.stdout:1289: done (2 left) 2024-09-11T01:38:01.374 INFO:tasks.rados.rados.0.smithi071.stdout:1290: done (1 left) 2024-09-11T01:38:01.374 INFO:tasks.rados.rados.0.smithi071.stdout:1292: done (0 left) 2024-09-11T01:38:01.374 INFO:tasks.rados.rados.0.smithi071.stdout:1293: read oid 19 snap -1 2024-09-11T01:38:01.374 INFO:tasks.rados.rados.0.smithi071.stdout:1293: expect (ObjNum 489 snap 132 seq_num 489) 2024-09-11T01:38:01.374 INFO:tasks.rados.rados.0.smithi071.stdout:1294: copy_from oid 34 from oid 28 current snap is 132 2024-09-11T01:38:01.374 INFO:tasks.rados.rados.0.smithi071.stdout:1295: read oid 43 snap -1 2024-09-11T01:38:01.375 INFO:tasks.rados.rados.0.smithi071.stdout:1295: expect (ObjNum 491 snap 132 seq_num 491) 2024-09-11T01:38:01.375 INFO:tasks.rados.rados.0.smithi071.stdout:1296: snap_create 2024-09-11T01:38:01.383 INFO:tasks.rados.rados.0.smithi071.stdout:1293: expect (ObjNum 489 snap 132 seq_num 489) 2024-09-11T01:38:01.475 INFO:tasks.rados.rados.0.smithi071.stdout:1295: expect (ObjNum 491 snap 132 seq_num 491) 2024-09-11T01:38:01.475 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:38:01.519 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 631710 2024-09-11T01:38:01.603 INFO:tasks.rados.rados.0.smithi071.stdout:1294: finishing copy_from racing read to smithi07134777-34 2024-09-11T01:38:01.603 INFO:tasks.rados.rados.0.smithi071.stdout:1294: finishing copy_from to smithi07134777-34 2024-09-11T01:38:01.603 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 516 (ObjNum 418 snap 117 seq_num 418) dirty exists 2024-09-11T01:38:02.116 INFO:tasks.rados.rados.0.smithi071.stdout:1293: done (3 left) 2024-09-11T01:38:02.116 INFO:tasks.rados.rados.0.smithi071.stdout:1294: done (2 left) 2024-09-11T01:38:02.117 INFO:tasks.rados.rados.0.smithi071.stdout:1295: done (1 left) 2024-09-11T01:38:02.117 INFO:tasks.rados.rados.0.smithi071.stdout:1296: done (0 left) 2024-09-11T01:38:02.117 INFO:tasks.rados.rados.0.smithi071.stdout:1297: write (excl) oid 3 current snap is 133 2024-09-11T01:38:02.117 INFO:tasks.rados.rados.0.smithi071.stdout:1297: seq_num 497 ranges {753079=561698,2033433=319646} 2024-09-11T01:38:02.123 INFO:tasks.rados.rados.0.smithi071.stdout:1297: writing smithi07134777-3 from 753079 to 1314777 tid 1 2024-09-11T01:38:02.129 INFO:tasks.rados.rados.0.smithi071.stdout:1297: writing smithi07134777-3 from 2033433 to 2353079 tid 2 2024-09-11T01:38:02.129 INFO:tasks.rados.rados.0.smithi071.stdout:1298: read oid 20 snap -1 2024-09-11T01:38:02.129 INFO:tasks.rados.rados.0.smithi071.stdout:1298: expect (ObjNum 482 snap 129 seq_num 482) 2024-09-11T01:38:02.130 INFO:tasks.rados.rados.0.smithi071.stdout:1299: read oid 29 snap -1 2024-09-11T01:38:02.130 INFO:tasks.rados.rados.0.smithi071.stdout:1299: expect (ObjNum 226 snap 55 seq_num 226) 2024-09-11T01:38:02.130 INFO:tasks.rados.rados.0.smithi071.stdout:1300: snap_create 2024-09-11T01:38:02.131 INFO:tasks.rados.rados.0.smithi071.stdout:1297: finishing write tid 1 to smithi07134777-3 2024-09-11T01:38:02.132 INFO:tasks.rados.rados.0.smithi071.stdout:1297: finishing write tid 2 to smithi07134777-3 2024-09-11T01:38:02.132 INFO:tasks.rados.rados.0.smithi071.stdout:1297: finishing write tid 3 to smithi07134777-3 2024-09-11T01:38:02.132 INFO:tasks.rados.rados.0.smithi071.stdout:1297: finishing write tid 4 to smithi07134777-3 2024-09-11T01:38:02.132 INFO:tasks.rados.rados.0.smithi071.stdout:1297: oid 3 updating version 0 to 255 2024-09-11T01:38:02.132 INFO:tasks.rados.rados.0.smithi071.stdout:1297: oid 3 updating version 255 to 257 2024-09-11T01:38:02.133 INFO:tasks.rados.rados.0.smithi071.stdout:1297: oid 3 version 257 is already newer than 256 2024-09-11T01:38:02.133 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 257 (ObjNum 497 snap 133 seq_num 497) dirty exists 2024-09-11T01:38:02.133 INFO:tasks.rados.rados.0.smithi071.stdout:1297: left oid 3 (ObjNum 497 snap 133 seq_num 497) 2024-09-11T01:38:02.135 INFO:tasks.rados.rados.0.smithi071.stdout:1298: expect (ObjNum 482 snap 129 seq_num 482) 2024-09-11T01:38:02.135 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 271495 2024-09-11T01:38:02.184 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1489032 2024-09-11T01:38:02.237 INFO:tasks.rados.rados.0.smithi071.stdout:1299: expect (ObjNum 226 snap 55 seq_num 226) 2024-09-11T01:38:02.249 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2236513 2024-09-11T01:38:02.265 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:38:02.305 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:38:03.129 INFO:tasks.rados.rados.0.smithi071.stdout:1297: done (3 left) 2024-09-11T01:38:03.129 INFO:tasks.rados.rados.0.smithi071.stdout:1298: done (2 left) 2024-09-11T01:38:03.129 INFO:tasks.rados.rados.0.smithi071.stdout:1299: done (1 left) 2024-09-11T01:38:03.129 INFO:tasks.rados.rados.0.smithi071.stdout:1300: done (0 left) 2024-09-11T01:38:03.129 INFO:tasks.rados.rados.0.smithi071.stdout:1301: read oid 39 snap 133 2024-09-11T01:38:03.129 INFO:tasks.rados.rados.0.smithi071.stdout:1301: expect (ObjNum 454 snap 121 seq_num 454) 2024-09-11T01:38:03.129 INFO:tasks.rados.rados.0.smithi071.stdout:1302: read oid 30 snap -1 2024-09-11T01:38:03.129 INFO:tasks.rados.rados.0.smithi071.stdout:1302: expect (ObjNum 490 snap 132 seq_num 490) 2024-09-11T01:38:03.129 INFO:tasks.rados.rados.0.smithi071.stdout:1303: read oid 2 snap -1 2024-09-11T01:38:03.130 INFO:tasks.rados.rados.0.smithi071.stdout:1303: expect deleted 2024-09-11T01:38:03.130 INFO:tasks.rados.rados.0.smithi071.stdout:1304: write oid 20 current snap is 134 2024-09-11T01:38:03.130 INFO:tasks.rados.rados.0.smithi071.stdout:1304: seq_num 498 ranges {769886=636947,2091915=277971} 2024-09-11T01:38:03.138 INFO:tasks.rados.rados.0.smithi071.stdout:1304: writing smithi07134777-20 from 769886 to 1406833 tid 1 2024-09-11T01:38:03.144 INFO:tasks.rados.rados.0.smithi071.stdout:1304: writing smithi07134777-20 from 2091915 to 2369886 tid 2 2024-09-11T01:38:03.144 INFO:tasks.rados.rados.0.smithi071.stdout:1305: copy_from oid 21 from oid 42 current snap is 134 2024-09-11T01:38:03.144 INFO:tasks.rados.rados.0.smithi071.stdout:1302: expect (ObjNum 490 snap 132 seq_num 490) 2024-09-11T01:38:03.221 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1893416 2024-09-11T01:38:03.231 INFO:tasks.rados.rados.0.smithi071.stdout:1301: expect (ObjNum 454 snap 121 seq_num 454) 2024-09-11T01:38:03.231 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:38:03.354 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2712016 2024-09-11T01:38:03.443 INFO:tasks.rados.rados.0.smithi071.stdout:1304: finishing write tid 1 to smithi07134777-20 2024-09-11T01:38:03.443 INFO:tasks.rados.rados.0.smithi071.stdout:1304: finishing write tid 2 to smithi07134777-20 2024-09-11T01:38:03.443 INFO:tasks.rados.rados.0.smithi071.stdout:1304: finishing write tid 3 to smithi07134777-20 2024-09-11T01:38:03.443 INFO:tasks.rados.rados.0.smithi071.stdout:1304: finishing write tid 4 to smithi07134777-20 2024-09-11T01:38:03.444 INFO:tasks.rados.rados.0.smithi071.stdout:1304: oid 20 updating version 0 to 259 2024-09-11T01:38:03.444 INFO:tasks.rados.rados.0.smithi071.stdout:1304: oid 20 version 259 is already newer than 258 2024-09-11T01:38:03.444 INFO:tasks.rados.rados.0.smithi071.stdout:1304: oid 20 updating version 259 to 260 2024-09-11T01:38:03.444 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 260 (ObjNum 498 snap 134 seq_num 498) dirty exists 2024-09-11T01:38:03.444 INFO:tasks.rados.rados.0.smithi071.stdout:1304: left oid 20 (ObjNum 498 snap 134 seq_num 498) 2024-09-11T01:38:03.444 INFO:tasks.rados.rados.0.smithi071.stdout:1305: finishing copy_from racing read to smithi07134777-21 2024-09-11T01:38:03.444 INFO:tasks.rados.rados.0.smithi071.stdout:1305: finishing copy_from to smithi07134777-21 2024-09-11T01:38:03.444 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 431 (ObjNum 488 snap 132 seq_num 488) dirty exists 2024-09-11T01:38:03.444 INFO:tasks.rados.rados.0.smithi071.stdout:1301: done (4 left) 2024-09-11T01:38:03.444 INFO:tasks.rados.rados.0.smithi071.stdout:1302: done (3 left) 2024-09-11T01:38:03.444 INFO:tasks.rados.rados.0.smithi071.stdout:1303: done (2 left) 2024-09-11T01:38:03.444 INFO:tasks.rados.rados.0.smithi071.stdout:1304: done (1 left) 2024-09-11T01:38:03.444 INFO:tasks.rados.rados.0.smithi071.stdout:1305: done (0 left) 2024-09-11T01:38:03.444 INFO:tasks.rados.rados.0.smithi071.stdout:1306: read oid 43 snap -1 2024-09-11T01:38:03.444 INFO:tasks.rados.rados.0.smithi071.stdout:1306: expect (ObjNum 491 snap 132 seq_num 491) 2024-09-11T01:38:03.445 INFO:tasks.rados.rados.0.smithi071.stdout:1307: rollback oid 25 current snap is 134 2024-09-11T01:38:03.445 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 25 to 132 2024-09-11T01:38:03.445 INFO:tasks.rados.rados.0.smithi071.stdout:1308: read oid 12 snap -1 2024-09-11T01:38:03.445 INFO:tasks.rados.rados.0.smithi071.stdout:1308: expect (ObjNum 469 snap 124 seq_num 469) 2024-09-11T01:38:03.445 INFO:tasks.rados.rados.0.smithi071.stdout:1309: read oid 46 snap 133 2024-09-11T01:38:03.445 INFO:tasks.rados.rados.0.smithi071.stdout:1309: expect (ObjNum 478 snap 126 seq_num 478) 2024-09-11T01:38:03.445 INFO:tasks.rados.rados.0.smithi071.stdout:1310: snap_create 2024-09-11T01:38:03.445 INFO:tasks.rados.rados.0.smithi071.stdout:1307: finishing rollback tid 1 to smithi07134777-25 2024-09-11T01:38:03.445 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-11T01:38:03.445 INFO:tasks.rados.rados.0.smithi071.stdout:1309: expect (ObjNum 478 snap 126 seq_num 478) 2024-09-11T01:38:03.446 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:38:03.542 INFO:tasks.rados.rados.0.smithi071.stdout:1306: expect (ObjNum 491 snap 132 seq_num 491) 2024-09-11T01:38:03.586 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 79852 2024-09-11T01:38:03.674 INFO:tasks.rados.rados.0.smithi071.stdout:1308: expect (ObjNum 469 snap 124 seq_num 469) 2024-09-11T01:38:03.716 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:38:03.923 INFO:tasks.rados.rados.0.smithi071.stdout:1306: done (4 left) 2024-09-11T01:38:03.924 INFO:tasks.rados.rados.0.smithi071.stdout:1307: done (3 left) 2024-09-11T01:38:03.924 INFO:tasks.rados.rados.0.smithi071.stdout:1308: done (2 left) 2024-09-11T01:38:03.924 INFO:tasks.rados.rados.0.smithi071.stdout:1309: done (1 left) 2024-09-11T01:38:03.924 INFO:tasks.rados.rados.0.smithi071.stdout:1310: done (0 left) 2024-09-11T01:38:03.924 INFO:tasks.rados.rados.0.smithi071.stdout:1311: read oid 12 snap -1 2024-09-11T01:38:03.924 INFO:tasks.rados.rados.0.smithi071.stdout:1311: expect (ObjNum 469 snap 124 seq_num 469) 2024-09-11T01:38:03.924 INFO:tasks.rados.rados.0.smithi071.stdout:1312: rollback oid 15 current snap is 135 2024-09-11T01:38:03.924 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 15 to 132 2024-09-11T01:38:03.924 INFO:tasks.rados.rados.0.smithi071.stdout:1313: delete oid 38 current snap is 135 2024-09-11T01:38:03.938 INFO:tasks.rados.rados.0.smithi071.stdout:1312: finishing rollback tid 0 to smithi07134777-15 2024-09-11T01:38:03.939 INFO:tasks.rados.rados.0.smithi071.stdout:1312: finishing rollback tid 1 to smithi07134777-15 2024-09-11T01:38:03.939 INFO:tasks.rados.rados.0.smithi071.stdout:1312: finishing rollback tid 2 to smithi07134777-15 2024-09-11T01:38:03.939 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 649 (ObjNum 481 snap 129 seq_num 481) dirty exists 2024-09-11T01:38:03.939 INFO:tasks.rados.rados.0.smithi071.stdout:1312: done (2 left) 2024-09-11T01:38:03.939 INFO:tasks.rados.rados.0.smithi071.stdout:1313: done (1 left) 2024-09-11T01:38:03.939 INFO:tasks.rados.rados.0.smithi071.stdout:1314: write oid 15 current snap is 135 2024-09-11T01:38:03.939 INFO:tasks.rados.rados.0.smithi071.stdout:1314: seq_num 500 ranges {403500=787445,1692526=473837} 2024-09-11T01:38:03.945 INFO:tasks.rados.rados.0.smithi071.stdout:1314: writing smithi07134777-15 from 403500 to 1190945 tid 1 2024-09-11T01:38:03.951 INFO:tasks.rados.rados.0.smithi071.stdout:1314: writing smithi07134777-15 from 1692526 to 2166363 tid 2 2024-09-11T01:38:03.951 INFO:tasks.rados.rados.0.smithi071.stdout:1315: delete oid 28 current snap is 135 2024-09-11T01:38:03.951 INFO:tasks.rados.rados.0.smithi071.stdout:1311: expect (ObjNum 469 snap 124 seq_num 469) 2024-09-11T01:38:03.951 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:38:04.010 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:38:04.112 INFO:tasks.rados.rados.0.smithi071.stdout:1314: finishing write tid 1 to smithi07134777-15 2024-09-11T01:38:04.112 INFO:tasks.rados.rados.0.smithi071.stdout:1314: finishing write tid 2 to smithi07134777-15 2024-09-11T01:38:04.113 INFO:tasks.rados.rados.0.smithi071.stdout:1314: finishing write tid 3 to smithi07134777-15 2024-09-11T01:38:04.113 INFO:tasks.rados.rados.0.smithi071.stdout:1314: finishing write tid 4 to smithi07134777-15 2024-09-11T01:38:04.113 INFO:tasks.rados.rados.0.smithi071.stdout:1314: oid 15 updating version 0 to 654 2024-09-11T01:38:04.113 INFO:tasks.rados.rados.0.smithi071.stdout:1314: oid 15 version 654 is already newer than 652 2024-09-11T01:38:04.113 INFO:tasks.rados.rados.0.smithi071.stdout:1314: oid 15 version 654 is already newer than 653 2024-09-11T01:38:04.113 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 654 (ObjNum 500 snap 135 seq_num 500) dirty exists 2024-09-11T01:38:04.113 INFO:tasks.rados.rados.0.smithi071.stdout:1314: left oid 15 (ObjNum 500 snap 135 seq_num 500) 2024-09-11T01:38:04.113 INFO:tasks.rados.rados.0.smithi071.stdout:1311: done (2 left) 2024-09-11T01:38:04.113 INFO:tasks.rados.rados.0.smithi071.stdout:1314: done (1 left) 2024-09-11T01:38:04.113 INFO:tasks.rados.rados.0.smithi071.stdout:1315: done (0 left) 2024-09-11T01:38:04.114 INFO:tasks.rados.rados.0.smithi071.stdout:1316: copy_from oid 30 from oid 45 current snap is 135 2024-09-11T01:38:04.114 INFO:tasks.rados.rados.0.smithi071.stdout:1317: delete oid 29 current snap is 135 2024-09-11T01:38:04.114 INFO:tasks.rados.rados.0.smithi071.stdout:1317: done (1 left) 2024-09-11T01:38:04.114 INFO:tasks.rados.rados.0.smithi071.stdout:1318: snap_remove snap 132 2024-09-11T01:38:04.934 INFO:tasks.rados.rados.0.smithi071.stdout:1318: done (1 left) 2024-09-11T01:38:04.934 INFO:tasks.rados.rados.0.smithi071.stdout:1319: write oid 2 current snap is 135 2024-09-11T01:38:04.934 INFO:tasks.rados.rados.0.smithi071.stdout:1316: finishing copy_from racing read to smithi07134777-30 2024-09-11T01:38:04.934 INFO:tasks.rados.rados.0.smithi071.stdout:1319: seq_num 503 ranges {453921=729545,1584325=621164} 2024-09-11T01:38:04.941 INFO:tasks.rados.rados.0.smithi071.stdout:1319: writing smithi07134777-2 from 453921 to 1183466 tid 1 2024-09-11T01:38:04.947 INFO:tasks.rados.rados.0.smithi071.stdout:1319: writing smithi07134777-2 from 1584325 to 2205489 tid 2 2024-09-11T01:38:04.947 INFO:tasks.rados.rados.0.smithi071.stdout:1320: snap_create 2024-09-11T01:38:04.947 INFO:tasks.rados.rados.0.smithi071.stdout:1316: finishing copy_from to smithi07134777-30 2024-09-11T01:38:04.947 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 525 (ObjNum 437 snap 119 seq_num 437) dirty exists 2024-09-11T01:38:04.947 INFO:tasks.rados.rados.0.smithi071.stdout:1319: finishing write tid 1 to smithi07134777-2 2024-09-11T01:38:04.951 INFO:tasks.rados.rados.0.smithi071.stdout:1319: finishing write tid 2 to smithi07134777-2 2024-09-11T01:38:04.951 INFO:tasks.rados.rados.0.smithi071.stdout:1319: finishing write tid 3 to smithi07134777-2 2024-09-11T01:38:04.951 INFO:tasks.rados.rados.0.smithi071.stdout:1319: finishing write tid 4 to smithi07134777-2 2024-09-11T01:38:04.951 INFO:tasks.rados.rados.0.smithi071.stdout:1319: oid 2 updating version 0 to 432 2024-09-11T01:38:04.951 INFO:tasks.rados.rados.0.smithi071.stdout:1319: oid 2 updating version 432 to 434 2024-09-11T01:38:04.951 INFO:tasks.rados.rados.0.smithi071.stdout:1319: oid 2 version 434 is already newer than 433 2024-09-11T01:38:04.951 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 434 (ObjNum 503 snap 135 seq_num 503) dirty exists 2024-09-11T01:38:04.951 INFO:tasks.rados.rados.0.smithi071.stdout:1319: left oid 2 (ObjNum 503 snap 135 seq_num 503) 2024-09-11T01:38:05.934 INFO:tasks.rados.rados.0.smithi071.stdout:1316: done (2 left) 2024-09-11T01:38:05.935 INFO:tasks.rados.rados.0.smithi071.stdout:1319: done (1 left) 2024-09-11T01:38:05.935 INFO:tasks.rados.rados.0.smithi071.stdout:1320: done (0 left) 2024-09-11T01:38:05.935 INFO:tasks.rados.rados.0.smithi071.stdout:1321: read oid 22 snap 135 2024-09-11T01:38:05.935 INFO:tasks.rados.rados.0.smithi071.stdout:1321: expect (ObjNum 426 snap 118 seq_num 426) 2024-09-11T01:38:05.935 INFO:tasks.rados.rados.0.smithi071.stdout:1322: snap_create 2024-09-11T01:38:05.943 INFO:tasks.rados.rados.0.smithi071.stdout:1321: expect (ObjNum 426 snap 118 seq_num 426) 2024-09-11T01:38:06.093 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:38:06.937 INFO:tasks.rados.rados.0.smithi071.stdout:1321: done (1 left) 2024-09-11T01:38:06.937 INFO:tasks.rados.rados.0.smithi071.stdout:1322: done (0 left) 2024-09-11T01:38:06.938 INFO:tasks.rados.rados.0.smithi071.stdout:1323: snap_create 2024-09-11T01:38:07.950 INFO:tasks.rados.rados.0.smithi071.stdout:1323: done (0 left) 2024-09-11T01:38:07.950 INFO:tasks.rados.rados.0.smithi071.stdout:1324: delete oid 34 current snap is 138 2024-09-11T01:38:07.956 INFO:tasks.rados.rados.0.smithi071.stdout:1324: done (0 left) 2024-09-11T01:38:07.956 INFO:tasks.rados.rados.0.smithi071.stdout:1325: snap_remove snap 137 2024-09-11T01:38:08.922 INFO:tasks.rados.rados.0.smithi071.stdout:1325: done (0 left) 2024-09-11T01:38:08.922 INFO:tasks.rados.rados.0.smithi071.stdout:1326: write (excl) oid 29 current snap is 138 2024-09-11T01:38:08.922 INFO:tasks.rados.rados.0.smithi071.stdout:1326: seq_num 505 ranges {487535=480043,1584936=624833} 2024-09-11T01:38:08.927 INFO:tasks.rados.rados.0.smithi071.stdout:1326: writing smithi07134777-29 from 487535 to 967578 tid 1 2024-09-11T01:38:08.934 INFO:tasks.rados.rados.0.smithi071.stdout:1326: writing smithi07134777-29 from 1584936 to 2209769 tid 2 2024-09-11T01:38:08.934 INFO:tasks.rados.rados.0.smithi071.stdout:1327: delete oid 41 current snap is 138 2024-09-11T01:38:08.934 INFO:tasks.rados.rados.0.smithi071.stdout:1326: finishing write tid 1 to smithi07134777-29 2024-09-11T01:38:08.938 INFO:tasks.rados.rados.0.smithi071.stdout:1326: finishing write tid 2 to smithi07134777-29 2024-09-11T01:38:08.939 INFO:tasks.rados.rados.0.smithi071.stdout:1326: finishing write tid 3 to smithi07134777-29 2024-09-11T01:38:08.939 INFO:tasks.rados.rados.0.smithi071.stdout:1326: finishing write tid 4 to smithi07134777-29 2024-09-11T01:38:08.939 INFO:tasks.rados.rados.0.smithi071.stdout:1326: oid 29 updating version 0 to 664 2024-09-11T01:38:08.939 INFO:tasks.rados.rados.0.smithi071.stdout:1326: oid 29 updating version 664 to 665 2024-09-11T01:38:08.939 INFO:tasks.rados.rados.0.smithi071.stdout:1326: oid 29 version 665 is already newer than 663 2024-09-11T01:38:08.939 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 665 (ObjNum 505 snap 138 seq_num 505) dirty exists 2024-09-11T01:38:08.939 INFO:tasks.rados.rados.0.smithi071.stdout:1326: left oid 29 (ObjNum 505 snap 138 seq_num 505) 2024-09-11T01:38:08.940 INFO:tasks.rados.rados.0.smithi071.stdout:1326: done (1 left) 2024-09-11T01:38:08.940 INFO:tasks.rados.rados.0.smithi071.stdout:1327: done (0 left) 2024-09-11T01:38:08.940 INFO:tasks.rados.rados.0.smithi071.stdout:1328: write oid 4 current snap is 138 2024-09-11T01:38:08.940 INFO:tasks.rados.rados.0.smithi071.stdout:1328: seq_num 507 ranges {521149=630541,1701900=628502} 2024-09-11T01:38:08.945 INFO:tasks.rados.rados.0.smithi071.stdout:1328: writing smithi07134777-4 from 521149 to 1151690 tid 1 2024-09-11T01:38:08.952 INFO:tasks.rados.rados.0.smithi071.stdout:1328: writing smithi07134777-4 from 1701900 to 2330402 tid 2 2024-09-11T01:38:08.952 INFO:tasks.rados.rados.0.smithi071.stdout:1329: rollback oid 47 current snap is 138 2024-09-11T01:38:08.952 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 47 to 133 2024-09-11T01:38:08.952 INFO:tasks.rados.rados.0.smithi071.stdout:1330: rollback oid 35 current snap is 138 2024-09-11T01:38:08.952 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 35 to 135 2024-09-11T01:38:08.952 INFO:tasks.rados.rados.0.smithi071.stdout:1331: read oid 10 snap -1 2024-09-11T01:38:08.952 INFO:tasks.rados.rados.0.smithi071.stdout:1331: expect (ObjNum 460 snap 122 seq_num 460) 2024-09-11T01:38:08.952 INFO:tasks.rados.rados.0.smithi071.stdout:1332: snap_remove snap 134 2024-09-11T01:38:09.957 INFO:tasks.rados.rados.0.smithi071.stdout:1332: done (4 left) 2024-09-11T01:38:09.957 INFO:tasks.rados.rados.0.smithi071.stdout:1333: rollback oid 17 current snap is 138 2024-09-11T01:38:09.957 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 17 to 135 2024-09-11T01:38:09.957 INFO:tasks.rados.rados.0.smithi071.stdout:1331: expect (ObjNum 460 snap 122 seq_num 460) 2024-09-11T01:38:10.162 INFO:tasks.rados.rados.0.smithi071.stdout:1329: finishing rollback tid 0 to smithi07134777-47 2024-09-11T01:38:10.162 INFO:tasks.rados.rados.0.smithi071.stdout:1329: finishing rollback tid 1 to smithi07134777-47 2024-09-11T01:38:10.162 INFO:tasks.rados.rados.0.smithi071.stdout:1329: finishing rollback tid 2 to smithi07134777-47 2024-09-11T01:38:10.162 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 464 (ObjNum 269 snap 71 seq_num 269) dirty exists 2024-09-11T01:38:10.162 INFO:tasks.rados.rados.0.smithi071.stdout:1328: finishing write tid 1 to smithi07134777-4 2024-09-11T01:38:10.162 INFO:tasks.rados.rados.0.smithi071.stdout:1330: finishing rollback tid 0 to smithi07134777-35 2024-09-11T01:38:10.162 INFO:tasks.rados.rados.0.smithi071.stdout:1330: finishing rollback tid 1 to smithi07134777-35 2024-09-11T01:38:10.162 INFO:tasks.rados.rados.0.smithi071.stdout:1330: finishing rollback tid 2 to smithi07134777-35 2024-09-11T01:38:10.162 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 571 (ObjNum 312 snap 87 seq_num 312) dirty exists 2024-09-11T01:38:10.162 INFO:tasks.rados.rados.0.smithi071.stdout:1328: finishing write tid 2 to smithi07134777-4 2024-09-11T01:38:10.162 INFO:tasks.rados.rados.0.smithi071.stdout:1328: finishing write tid 3 to smithi07134777-4 2024-09-11T01:38:10.163 INFO:tasks.rados.rados.0.smithi071.stdout:1328: finishing write tid 4 to smithi07134777-4 2024-09-11T01:38:10.163 INFO:tasks.rados.rados.0.smithi071.stdout:1328: oid 4 updating version 0 to 527 2024-09-11T01:38:10.163 INFO:tasks.rados.rados.0.smithi071.stdout:1328: oid 4 version 527 is already newer than 526 2024-09-11T01:38:10.163 INFO:tasks.rados.rados.0.smithi071.stdout:1328: oid 4 updating version 527 to 528 2024-09-11T01:38:10.163 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 528 (ObjNum 507 snap 138 seq_num 507) dirty exists 2024-09-11T01:38:10.163 INFO:tasks.rados.rados.0.smithi071.stdout:1328: left oid 4 (ObjNum 507 snap 138 seq_num 507) 2024-09-11T01:38:10.163 INFO:tasks.rados.rados.0.smithi071.stdout:1333: finishing rollback tid 0 to smithi07134777-17 2024-09-11T01:38:10.163 INFO:tasks.rados.rados.0.smithi071.stdout:1333: finishing rollback tid 1 to smithi07134777-17 2024-09-11T01:38:10.163 INFO:tasks.rados.rados.0.smithi071.stdout:1333: finishing rollback tid 2 to smithi07134777-17 2024-09-11T01:38:10.163 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 468 (ObjNum 250 snap 63 seq_num 250) dirty exists 2024-09-11T01:38:10.163 INFO:tasks.rados.rados.0.smithi071.stdout:1328: done (4 left) 2024-09-11T01:38:10.163 INFO:tasks.rados.rados.0.smithi071.stdout:1329: done (3 left) 2024-09-11T01:38:10.163 INFO:tasks.rados.rados.0.smithi071.stdout:1330: done (2 left) 2024-09-11T01:38:10.163 INFO:tasks.rados.rados.0.smithi071.stdout:1331: done (1 left) 2024-09-11T01:38:10.163 INFO:tasks.rados.rados.0.smithi071.stdout:1333: done (0 left) 2024-09-11T01:38:10.164 INFO:tasks.rados.rados.0.smithi071.stdout:1334: write oid 8 current snap is 138 2024-09-11T01:38:10.164 INFO:tasks.rados.rados.0.smithi071.stdout:1334: seq_num 508 ranges {537956=705790,1644029=772160} 2024-09-11T01:38:10.168 INFO:tasks.rados.rados.0.smithi071.stdout:1334: writing smithi07134777-8 from 537956 to 1243746 tid 1 2024-09-11T01:38:10.174 INFO:tasks.rados.rados.0.smithi071.stdout:1334: writing smithi07134777-8 from 1644029 to 2416189 tid 2 2024-09-11T01:38:10.174 INFO:tasks.rados.rados.0.smithi071.stdout:1335: snap_create 2024-09-11T01:38:10.178 INFO:tasks.rados.rados.0.smithi071.stdout:1334: finishing write tid 1 to smithi07134777-8 2024-09-11T01:38:10.181 INFO:tasks.rados.rados.0.smithi071.stdout:1334: finishing write tid 2 to smithi07134777-8 2024-09-11T01:38:10.181 INFO:tasks.rados.rados.0.smithi071.stdout:1334: finishing write tid 3 to smithi07134777-8 2024-09-11T01:38:10.181 INFO:tasks.rados.rados.0.smithi071.stdout:1334: finishing write tid 4 to smithi07134777-8 2024-09-11T01:38:10.181 INFO:tasks.rados.rados.0.smithi071.stdout:1334: oid 8 updating version 0 to 446 2024-09-11T01:38:10.181 INFO:tasks.rados.rados.0.smithi071.stdout:1334: oid 8 updating version 446 to 447 2024-09-11T01:38:10.182 INFO:tasks.rados.rados.0.smithi071.stdout:1334: oid 8 version 447 is already newer than 445 2024-09-11T01:38:10.182 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 447 (ObjNum 508 snap 138 seq_num 508) dirty exists 2024-09-11T01:38:10.182 INFO:tasks.rados.rados.0.smithi071.stdout:1334: left oid 8 (ObjNum 508 snap 138 seq_num 508) 2024-09-11T01:38:10.966 INFO:tasks.rados.rados.0.smithi071.stdout:1334: done (1 left) 2024-09-11T01:38:10.966 INFO:tasks.rados.rados.0.smithi071.stdout:1335: done (0 left) 2024-09-11T01:38:10.966 INFO:tasks.rados.rados.0.smithi071.stdout:1336: write (excl) oid 31 current snap is 139 2024-09-11T01:38:10.966 INFO:tasks.rados.rados.0.smithi071.stdout:1336: seq_num 509 ranges {554763=781039,2102511=452252} 2024-09-11T01:38:10.973 INFO:tasks.rados.rados.0.smithi071.stdout:1336: writing smithi07134777-31 from 554763 to 1335802 tid 1 2024-09-11T01:38:10.981 INFO:tasks.rados.rados.0.smithi071.stdout:1336: writing smithi07134777-31 from 2102511 to 2554763 tid 2 2024-09-11T01:38:10.981 INFO:tasks.rados.rados.0.smithi071.stdout:1337: read oid 29 snap -1 2024-09-11T01:38:10.981 INFO:tasks.rados.rados.0.smithi071.stdout:1337: expect (ObjNum 505 snap 138 seq_num 505) 2024-09-11T01:38:10.981 INFO:tasks.rados.rados.0.smithi071.stdout:1338: rollback oid 30 current snap is 139 2024-09-11T01:38:10.981 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 30 to 136 2024-09-11T01:38:10.981 INFO:tasks.rados.rados.0.smithi071.stdout:1339: write (excl) oid 39 current snap is 139 2024-09-11T01:38:10.981 INFO:tasks.rados.rados.0.smithi071.stdout:1339: seq_num 510 ranges {571570=572641,1877346=694224} 2024-09-11T01:38:10.987 INFO:tasks.rados.rados.0.smithi071.stdout:1339: writing smithi07134777-39 from 571570 to 1144211 tid 1 2024-09-11T01:38:10.995 INFO:tasks.rados.rados.0.smithi071.stdout:1339: writing smithi07134777-39 from 1877346 to 2571570 tid 2 2024-09-11T01:38:10.995 INFO:tasks.rados.rados.0.smithi071.stdout:1340: snap_create 2024-09-11T01:38:10.996 INFO:tasks.rados.rados.0.smithi071.stdout:1336: finishing write tid 1 to smithi07134777-31 2024-09-11T01:38:10.996 INFO:tasks.rados.rados.0.smithi071.stdout:1337: expect (ObjNum 505 snap 138 seq_num 505) 2024-09-11T01:38:11.025 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1198146 2024-09-11T01:38:11.054 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2182547 2024-09-11T01:38:11.058 INFO:tasks.rados.rados.0.smithi071.stdout:1336: finishing write tid 2 to smithi07134777-31 2024-09-11T01:38:11.058 INFO:tasks.rados.rados.0.smithi071.stdout:1336: finishing write tid 3 to smithi07134777-31 2024-09-11T01:38:11.058 INFO:tasks.rados.rados.0.smithi071.stdout:1336: finishing write tid 4 to smithi07134777-31 2024-09-11T01:38:11.058 INFO:tasks.rados.rados.0.smithi071.stdout:1336: oid 31 updating version 0 to 472 2024-09-11T01:38:11.058 INFO:tasks.rados.rados.0.smithi071.stdout:1336: oid 31 updating version 472 to 474 2024-09-11T01:38:11.058 INFO:tasks.rados.rados.0.smithi071.stdout:1336: oid 31 version 474 is already newer than 473 2024-09-11T01:38:11.059 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 474 (ObjNum 509 snap 139 seq_num 509) dirty exists 2024-09-11T01:38:11.060 INFO:tasks.rados.rados.0.smithi071.stdout:1336: left oid 31 (ObjNum 509 snap 139 seq_num 509) 2024-09-11T01:38:11.060 INFO:tasks.rados.rados.0.smithi071.stdout:1338: finishing rollback tid 0 to smithi07134777-30 2024-09-11T01:38:11.060 INFO:tasks.rados.rados.0.smithi071.stdout:1338: finishing rollback tid 1 to smithi07134777-30 2024-09-11T01:38:11.060 INFO:tasks.rados.rados.0.smithi071.stdout:1338: finishing rollback tid 2 to smithi07134777-30 2024-09-11T01:38:11.060 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 531 (ObjNum 437 snap 119 seq_num 437) dirty exists 2024-09-11T01:38:11.060 INFO:tasks.rados.rados.0.smithi071.stdout:1339: finishing write tid 1 to smithi07134777-39 2024-09-11T01:38:11.060 INFO:tasks.rados.rados.0.smithi071.stdout:1339: finishing write tid 2 to smithi07134777-39 2024-09-11T01:38:11.060 INFO:tasks.rados.rados.0.smithi071.stdout:1339: finishing write tid 3 to smithi07134777-39 2024-09-11T01:38:11.060 INFO:tasks.rados.rados.0.smithi071.stdout:1339: finishing write tid 4 to smithi07134777-39 2024-09-11T01:38:11.060 INFO:tasks.rados.rados.0.smithi071.stdout:1339: oid 39 updating version 0 to 572 2024-09-11T01:38:11.060 INFO:tasks.rados.rados.0.smithi071.stdout:1339: oid 39 updating version 572 to 573 2024-09-11T01:38:11.060 INFO:tasks.rados.rados.0.smithi071.stdout:1339: oid 39 updating version 573 to 574 2024-09-11T01:38:11.060 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 574 (ObjNum 510 snap 139 seq_num 510) dirty exists 2024-09-11T01:38:11.061 INFO:tasks.rados.rados.0.smithi071.stdout:1339: left oid 39 (ObjNum 510 snap 139 seq_num 510) 2024-09-11T01:38:11.967 INFO:tasks.rados.rados.0.smithi071.stdout:1336: done (4 left) 2024-09-11T01:38:11.967 INFO:tasks.rados.rados.0.smithi071.stdout:1337: done (3 left) 2024-09-11T01:38:11.967 INFO:tasks.rados.rados.0.smithi071.stdout:1338: done (2 left) 2024-09-11T01:38:11.967 INFO:tasks.rados.rados.0.smithi071.stdout:1339: done (1 left) 2024-09-11T01:38:11.967 INFO:tasks.rados.rados.0.smithi071.stdout:1340: done (0 left) 2024-09-11T01:38:11.967 INFO:tasks.rados.rados.0.smithi071.stdout:1341: read oid 23 snap 133 2024-09-11T01:38:11.967 INFO:tasks.rados.rados.0.smithi071.stdout:1341: expect (ObjNum 492 snap 132 seq_num 492) 2024-09-11T01:38:11.968 INFO:tasks.rados.rados.0.smithi071.stdout:1342: snap_remove snap 138 2024-09-11T01:38:13.017 INFO:tasks.rados.rados.0.smithi071.stdout:1342: done (1 left) 2024-09-11T01:38:13.018 INFO:tasks.rados.rados.0.smithi071.stdout:1343: read oid 37 snap -1 2024-09-11T01:38:13.018 INFO:tasks.rados.rados.0.smithi071.stdout:1343: expect (ObjNum 483 snap 130 seq_num 483) 2024-09-11T01:38:13.018 INFO:tasks.rados.rados.0.smithi071.stdout:1344: delete oid 30 current snap is 140 2024-09-11T01:38:13.018 INFO:tasks.rados.rados.0.smithi071.stdout:1341: expect (ObjNum 492 snap 132 seq_num 492) 2024-09-11T01:38:13.018 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:38:13.046 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1405238 2024-09-11T01:38:13.079 INFO:tasks.rados.rados.0.smithi071.stdout:1343: expect (ObjNum 483 snap 130 seq_num 483) 2024-09-11T01:38:13.079 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:38:13.136 INFO:tasks.rados.rados.0.smithi071.stdout:1341: done (2 left) 2024-09-11T01:38:13.137 INFO:tasks.rados.rados.0.smithi071.stdout:1343: done (1 left) 2024-09-11T01:38:13.137 INFO:tasks.rados.rados.0.smithi071.stdout:1344: done (0 left) 2024-09-11T01:38:13.137 INFO:tasks.rados.rados.0.smithi071.stdout:1345: delete oid 10 current snap is 140 2024-09-11T01:38:13.140 INFO:tasks.rados.rados.0.smithi071.stdout:1345: done (0 left) 2024-09-11T01:38:13.140 INFO:tasks.rados.rados.0.smithi071.stdout:1346: read oid 24 snap -1 2024-09-11T01:38:13.140 INFO:tasks.rados.rados.0.smithi071.stdout:1346: expect (ObjNum 430 snap 119 seq_num 430) 2024-09-11T01:38:13.140 INFO:tasks.rados.rados.0.smithi071.stdout:1347: snap_create 2024-09-11T01:38:13.149 INFO:tasks.rados.rados.0.smithi071.stdout:1346: expect (ObjNum 430 snap 119 seq_num 430) 2024-09-11T01:38:13.149 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 277341 2024-09-11T01:38:13.212 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1105159 2024-09-11T01:38:13.254 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:38:13.924 INFO:tasks.rados.rados.0.smithi071.stdout:1346: done (1 left) 2024-09-11T01:38:13.924 INFO:tasks.rados.rados.0.smithi071.stdout:1347: done (0 left) 2024-09-11T01:38:13.924 INFO:tasks.rados.rados.0.smithi071.stdout:1348: copy_from oid 15 from oid 27 current snap is 141 2024-09-11T01:38:13.924 INFO:tasks.rados.rados.0.smithi071.stdout:1349: copy_from oid 36 from oid 12 current snap is 141 2024-09-11T01:38:13.924 INFO:tasks.rados.rados.0.smithi071.stdout:1350: rollback oid 49 current snap is 141 2024-09-11T01:38:13.924 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 49 to 133 2024-09-11T01:38:13.924 INFO:tasks.rados.rados.0.smithi071.stdout:1351: write (excl) oid 20 current snap is 141 2024-09-11T01:38:13.924 INFO:tasks.rados.rados.0.smithi071.stdout:1351: seq_num 513 ranges {621991=798388,1936439=639509} 2024-09-11T01:38:13.931 INFO:tasks.rados.rados.0.smithi071.stdout:1351: writing smithi07134777-20 from 621991 to 1420379 tid 1 2024-09-11T01:38:13.939 INFO:tasks.rados.rados.0.smithi071.stdout:1351: writing smithi07134777-20 from 1936439 to 2575948 tid 2 2024-09-11T01:38:13.939 INFO:tasks.rados.rados.0.smithi071.stdout:1352: delete oid 19 current snap is 141 2024-09-11T01:38:13.939 INFO:tasks.rados.rados.0.smithi071.stdout:1350: finishing rollback tid 1 to smithi07134777-49 2024-09-11T01:38:13.939 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 0 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-11T01:38:13.939 INFO:tasks.rados.rados.0.smithi071.stdout:1351: finishing write tid 1 to smithi07134777-20 2024-09-11T01:38:13.942 INFO:tasks.rados.rados.0.smithi071.stdout:1351: finishing write tid 2 to smithi07134777-20 2024-09-11T01:38:13.942 INFO:tasks.rados.rados.0.smithi071.stdout:1351: finishing write tid 3 to smithi07134777-20 2024-09-11T01:38:13.943 INFO:tasks.rados.rados.0.smithi071.stdout:1351: finishing write tid 4 to smithi07134777-20 2024-09-11T01:38:13.943 INFO:tasks.rados.rados.0.smithi071.stdout:1351: oid 20 updating version 0 to 263 2024-09-11T01:38:13.943 INFO:tasks.rados.rados.0.smithi071.stdout:1351: oid 20 version 263 is already newer than 262 2024-09-11T01:38:13.943 INFO:tasks.rados.rados.0.smithi071.stdout:1351: oid 20 version 263 is already newer than 261 2024-09-11T01:38:13.943 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 263 (ObjNum 513 snap 141 seq_num 513) dirty exists 2024-09-11T01:38:13.943 INFO:tasks.rados.rados.0.smithi071.stdout:1351: left oid 20 (ObjNum 513 snap 141 seq_num 513) 2024-09-11T01:38:13.945 INFO:tasks.rados.rados.0.smithi071.stdout:1349: finishing copy_from racing read to smithi07134777-36 2024-09-11T01:38:13.945 INFO:tasks.rados.rados.0.smithi071.stdout:1348: finishing copy_from racing read to smithi07134777-15 2024-09-11T01:38:13.951 INFO:tasks.rados.rados.0.smithi071.stdout:1348: finishing copy_from to smithi07134777-15 2024-09-11T01:38:13.951 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 550 (ObjNum 446 snap 121 seq_num 446) dirty exists 2024-09-11T01:38:13.957 INFO:tasks.rados.rados.0.smithi071.stdout:1349: finishing copy_from to smithi07134777-36 2024-09-11T01:38:13.958 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 355 (ObjNum 469 snap 124 seq_num 469) dirty exists 2024-09-11T01:38:13.960 INFO:tasks.rados.rados.0.smithi071.stdout:1348: done (4 left) 2024-09-11T01:38:13.960 INFO:tasks.rados.rados.0.smithi071.stdout:1349: done (3 left) 2024-09-11T01:38:13.960 INFO:tasks.rados.rados.0.smithi071.stdout:1350: done (2 left) 2024-09-11T01:38:13.961 INFO:tasks.rados.rados.0.smithi071.stdout:1351: done (1 left) 2024-09-11T01:38:13.961 INFO:tasks.rados.rados.0.smithi071.stdout:1352: done (0 left) 2024-09-11T01:38:13.961 INFO:tasks.rados.rados.0.smithi071.stdout:1353: copy_from oid 7 from oid 22 current snap is 141 2024-09-11T01:38:13.961 INFO:tasks.rados.rados.0.smithi071.stdout:1354: write oid 19 current snap is 141 2024-09-11T01:38:13.961 INFO:tasks.rados.rados.0.smithi071.stdout:1354: seq_num 515 ranges {655605=548886,1653403=643178} 2024-09-11T01:38:13.967 INFO:tasks.rados.rados.0.smithi071.stdout:1354: writing smithi07134777-19 from 655605 to 1204491 tid 1 2024-09-11T01:38:13.974 INFO:tasks.rados.rados.0.smithi071.stdout:1354: writing smithi07134777-19 from 1653403 to 2296581 tid 2 2024-09-11T01:38:13.974 INFO:tasks.rados.rados.0.smithi071.stdout:1355: write (excl) oid 50 current snap is 141 2024-09-11T01:38:13.974 INFO:tasks.rados.rados.0.smithi071.stdout:1353: finishing copy_from racing read to smithi07134777-7 2024-09-11T01:38:13.974 INFO:tasks.rados.rados.0.smithi071.stdout:1353: finishing copy_from to smithi07134777-7 2024-09-11T01:38:13.974 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 254 (ObjNum 426 snap 118 seq_num 426) dirty exists 2024-09-11T01:38:13.974 INFO:tasks.rados.rados.0.smithi071.stdout:1354: finishing write tid 1 to smithi07134777-19 2024-09-11T01:38:13.974 INFO:tasks.rados.rados.0.smithi071.stdout:1355: seq_num 516 ranges {672412=624135,1995532=676880} 2024-09-11T01:38:13.980 INFO:tasks.rados.rados.0.smithi071.stdout:1355: writing smithi07134777-50 from 672412 to 1296547 tid 1 2024-09-11T01:38:13.988 INFO:tasks.rados.rados.0.smithi071.stdout:1355: writing smithi07134777-50 from 1995532 to 2672412 tid 2 2024-09-11T01:38:13.988 INFO:tasks.rados.rados.0.smithi071.stdout:1353: done (2 left) 2024-09-11T01:38:13.988 INFO:tasks.rados.rados.0.smithi071.stdout:1356: write oid 38 current snap is 141 2024-09-11T01:38:13.988 INFO:tasks.rados.rados.0.smithi071.stdout:1356: seq_num 517 ranges {689219=415737,1770367=646847} 2024-09-11T01:38:13.994 INFO:tasks.rados.rados.0.smithi071.stdout:1356: writing smithi07134777-38 from 689219 to 1104956 tid 1 2024-09-11T01:38:14.002 INFO:tasks.rados.rados.0.smithi071.stdout:1356: writing smithi07134777-38 from 1770367 to 2417214 tid 2 2024-09-11T01:38:14.002 INFO:tasks.rados.rados.0.smithi071.stdout:1357: read oid 42 snap -1 2024-09-11T01:38:14.002 INFO:tasks.rados.rados.0.smithi071.stdout:1357: expect (ObjNum 488 snap 132 seq_num 488) 2024-09-11T01:38:14.002 INFO:tasks.rados.rados.0.smithi071.stdout:1358: read oid 9 snap -1 2024-09-11T01:38:14.002 INFO:tasks.rados.rados.0.smithi071.stdout:1358: expect (ObjNum 475 snap 125 seq_num 475) 2024-09-11T01:38:14.002 INFO:tasks.rados.rados.0.smithi071.stdout:1359: write oid 47 current snap is 141 2024-09-11T01:38:14.002 INFO:tasks.rados.rados.0.smithi071.stdout:1354: finishing write tid 2 to smithi07134777-19 2024-09-11T01:38:14.002 INFO:tasks.rados.rados.0.smithi071.stdout:1359: seq_num 518 ranges {706026=490986,1828849=790505} 2024-09-11T01:38:14.008 INFO:tasks.rados.rados.0.smithi071.stdout:1359: writing smithi07134777-47 from 706026 to 1197012 tid 1 2024-09-11T01:38:14.016 INFO:tasks.rados.rados.0.smithi071.stdout:1359: writing smithi07134777-47 from 1828849 to 2619354 tid 2 2024-09-11T01:38:14.016 INFO:tasks.rados.rados.0.smithi071.stdout:1360: snap_remove snap 136 2024-09-11T01:38:15.018 INFO:tasks.rados.rados.0.smithi071.stdout:1360: done (6 left) 2024-09-11T01:38:15.018 INFO:tasks.rados.rados.0.smithi071.stdout:1361: snap_create 2024-09-11T01:38:15.019 INFO:tasks.rados.rados.0.smithi071.stdout:1354: finishing write tid 3 to smithi07134777-19 2024-09-11T01:38:15.019 INFO:tasks.rados.rados.0.smithi071.stdout:1354: finishing write tid 4 to smithi07134777-19 2024-09-11T01:38:15.019 INFO:tasks.rados.rados.0.smithi071.stdout:1354: oid 19 updating version 0 to 450 2024-09-11T01:38:15.019 INFO:tasks.rados.rados.0.smithi071.stdout:1354: oid 19 updating version 450 to 451 2024-09-11T01:38:15.019 INFO:tasks.rados.rados.0.smithi071.stdout:1354: oid 19 version 451 is already newer than 449 2024-09-11T01:38:15.019 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 451 (ObjNum 515 snap 141 seq_num 515) dirty exists 2024-09-11T01:38:15.019 INFO:tasks.rados.rados.0.smithi071.stdout:1354: left oid 19 (ObjNum 515 snap 141 seq_num 515) 2024-09-11T01:38:15.019 INFO:tasks.rados.rados.0.smithi071.stdout:1355: finishing write tid 1 to smithi07134777-50 2024-09-11T01:38:15.019 INFO:tasks.rados.rados.0.smithi071.stdout:1355: finishing write tid 2 to smithi07134777-50 2024-09-11T01:38:15.019 INFO:tasks.rados.rados.0.smithi071.stdout:1355: finishing write tid 3 to smithi07134777-50 2024-09-11T01:38:15.019 INFO:tasks.rados.rados.0.smithi071.stdout:1355: finishing write tid 4 to smithi07134777-50 2024-09-11T01:38:15.019 INFO:tasks.rados.rados.0.smithi071.stdout:1355: oid 50 updating version 0 to 676 2024-09-11T01:38:15.019 INFO:tasks.rados.rados.0.smithi071.stdout:1355: oid 50 updating version 676 to 677 2024-09-11T01:38:15.019 INFO:tasks.rados.rados.0.smithi071.stdout:1355: oid 50 version 677 is already newer than 675 2024-09-11T01:38:15.020 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 677 (ObjNum 516 snap 141 seq_num 516) dirty exists 2024-09-11T01:38:15.020 INFO:tasks.rados.rados.0.smithi071.stdout:1355: left oid 50 (ObjNum 516 snap 141 seq_num 516) 2024-09-11T01:38:15.020 INFO:tasks.rados.rados.0.smithi071.stdout:1356: finishing write tid 1 to smithi07134777-38 2024-09-11T01:38:15.020 INFO:tasks.rados.rados.0.smithi071.stdout:1357: expect (ObjNum 488 snap 132 seq_num 488) 2024-09-11T01:38:15.060 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 424123 2024-09-11T01:38:15.132 INFO:tasks.rados.rados.0.smithi071.stdout:1356: finishing write tid 2 to smithi07134777-38 2024-09-11T01:38:15.132 INFO:tasks.rados.rados.0.smithi071.stdout:1356: finishing write tid 3 to smithi07134777-38 2024-09-11T01:38:15.132 INFO:tasks.rados.rados.0.smithi071.stdout:1356: finishing write tid 4 to smithi07134777-38 2024-09-11T01:38:15.132 INFO:tasks.rados.rados.0.smithi071.stdout:1356: oid 38 updating version 0 to 678 2024-09-11T01:38:15.132 INFO:tasks.rados.rados.0.smithi071.stdout:1356: oid 38 updating version 678 to 679 2024-09-11T01:38:15.132 INFO:tasks.rados.rados.0.smithi071.stdout:1356: oid 38 updating version 679 to 680 2024-09-11T01:38:15.132 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 680 (ObjNum 517 snap 141 seq_num 517) dirty exists 2024-09-11T01:38:15.132 INFO:tasks.rados.rados.0.smithi071.stdout:1356: left oid 38 (ObjNum 517 snap 141 seq_num 517) 2024-09-11T01:38:15.132 INFO:tasks.rados.rados.0.smithi071.stdout:1358: expect (ObjNum 475 snap 125 seq_num 475) 2024-09-11T01:38:15.326 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3315493 2024-09-11T01:38:15.336 INFO:tasks.rados.rados.0.smithi071.stdout:1359: finishing write tid 1 to smithi07134777-47 2024-09-11T01:38:15.336 INFO:tasks.rados.rados.0.smithi071.stdout:1359: finishing write tid 2 to smithi07134777-47 2024-09-11T01:38:15.336 INFO:tasks.rados.rados.0.smithi071.stdout:1359: finishing write tid 3 to smithi07134777-47 2024-09-11T01:38:15.336 INFO:tasks.rados.rados.0.smithi071.stdout:1359: finishing write tid 4 to smithi07134777-47 2024-09-11T01:38:15.336 INFO:tasks.rados.rados.0.smithi071.stdout:1359: oid 47 updating version 0 to 479 2024-09-11T01:38:15.336 INFO:tasks.rados.rados.0.smithi071.stdout:1359: oid 47 updating version 479 to 480 2024-09-11T01:38:15.336 INFO:tasks.rados.rados.0.smithi071.stdout:1359: oid 47 version 480 is already newer than 478 2024-09-11T01:38:15.336 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 480 (ObjNum 518 snap 141 seq_num 518) dirty exists 2024-09-11T01:38:15.336 INFO:tasks.rados.rados.0.smithi071.stdout:1359: left oid 47 (ObjNum 518 snap 141 seq_num 518) 2024-09-11T01:38:16.021 INFO:tasks.rados.rados.0.smithi071.stdout:1354: done (6 left) 2024-09-11T01:38:16.022 INFO:tasks.rados.rados.0.smithi071.stdout:1355: done (5 left) 2024-09-11T01:38:16.022 INFO:tasks.rados.rados.0.smithi071.stdout:1356: done (4 left) 2024-09-11T01:38:16.022 INFO:tasks.rados.rados.0.smithi071.stdout:1357: done (3 left) 2024-09-11T01:38:16.022 INFO:tasks.rados.rados.0.smithi071.stdout:1358: done (2 left) 2024-09-11T01:38:16.022 INFO:tasks.rados.rados.0.smithi071.stdout:1359: done (1 left) 2024-09-11T01:38:16.022 INFO:tasks.rados.rados.0.smithi071.stdout:1361: done (0 left) 2024-09-11T01:38:16.022 INFO:tasks.rados.rados.0.smithi071.stdout:1362: copy_from oid 33 from oid 17 current snap is 142 2024-09-11T01:38:16.022 INFO:tasks.rados.rados.0.smithi071.stdout:1363: snap_create 2024-09-11T01:38:16.040 INFO:tasks.rados.rados.0.smithi071.stdout:1362: finishing copy_from racing read to smithi07134777-33 2024-09-11T01:38:16.041 INFO:tasks.rados.rados.0.smithi071.stdout:1362: finishing copy_from to smithi07134777-33 2024-09-11T01:38:16.041 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 468 (ObjNum 250 snap 63 seq_num 250) dirty exists 2024-09-11T01:38:17.033 INFO:tasks.rados.rados.0.smithi071.stdout:1362: done (1 left) 2024-09-11T01:38:17.033 INFO:tasks.rados.rados.0.smithi071.stdout:1363: done (0 left) 2024-09-11T01:38:17.033 INFO:tasks.rados.rados.0.smithi071.stdout:1364: copy_from oid 38 from oid 39 current snap is 143 2024-09-11T01:38:17.033 INFO:tasks.rados.rados.0.smithi071.stdout:1365: rollback oid 32 current snap is 143 2024-09-11T01:38:17.033 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 32 to 140 2024-09-11T01:38:17.033 INFO:tasks.rados.rados.0.smithi071.stdout:1366: read oid 29 snap -1 2024-09-11T01:38:17.033 INFO:tasks.rados.rados.0.smithi071.stdout:1366: expect (ObjNum 505 snap 138 seq_num 505) 2024-09-11T01:38:17.033 INFO:tasks.rados.rados.0.smithi071.stdout:1367: write (excl) oid 34 current snap is 143 2024-09-11T01:38:17.033 INFO:tasks.rados.rados.0.smithi071.stdout:1367: seq_num 519 ranges {722833=566235,1887331=650516} 2024-09-11T01:38:17.040 INFO:tasks.rados.rados.0.smithi071.stdout:1367: writing smithi07134777-34 from 722833 to 1289068 tid 1 2024-09-11T01:38:17.047 INFO:tasks.rados.rados.0.smithi071.stdout:1367: writing smithi07134777-34 from 1887331 to 2537847 tid 2 2024-09-11T01:38:17.048 INFO:tasks.rados.rados.0.smithi071.stdout:1368: write (excl) oid 10 current snap is 143 2024-09-11T01:38:17.048 INFO:tasks.rados.rados.0.smithi071.stdout:1365: finishing rollback tid 1 to smithi07134777-32 2024-09-11T01:38:17.048 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 0 (ObjNum 225 snap 0 seq_num 2370852928) dirty dne 2024-09-11T01:38:17.048 INFO:tasks.rados.rados.0.smithi071.stdout:1368: seq_num 520 ranges {739640=641484,1829460=794174} 2024-09-11T01:38:17.054 INFO:tasks.rados.rados.0.smithi071.stdout:1368: writing smithi07134777-10 from 739640 to 1381124 tid 1 2024-09-11T01:38:17.062 INFO:tasks.rados.rados.0.smithi071.stdout:1368: writing smithi07134777-10 from 1829460 to 2623634 tid 2 2024-09-11T01:38:17.062 INFO:tasks.rados.rados.0.smithi071.stdout:1365: done (4 left) 2024-09-11T01:38:17.062 INFO:tasks.rados.rados.0.smithi071.stdout:1369: write oid 15 current snap is 143 2024-09-11T01:38:17.062 INFO:tasks.rados.rados.0.smithi071.stdout:1366: expect (ObjNum 505 snap 138 seq_num 505) 2024-09-11T01:38:17.062 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:38:17.089 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 136399 2024-09-11T01:38:17.120 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 942654 2024-09-11T01:38:17.137 INFO:tasks.rados.rados.0.smithi071.stdout:1367: finishing write tid 1 to smithi07134777-34 2024-09-11T01:38:17.137 INFO:tasks.rados.rados.0.smithi071.stdout:1364: finishing copy_from racing read to smithi07134777-38 2024-09-11T01:38:17.137 INFO:tasks.rados.rados.0.smithi071.stdout:1364: finishing copy_from to smithi07134777-38 2024-09-11T01:38:17.137 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 574 (ObjNum 510 snap 139 seq_num 510) dirty exists 2024-09-11T01:38:17.137 INFO:tasks.rados.rados.0.smithi071.stdout:1367: finishing write tid 2 to smithi07134777-34 2024-09-11T01:38:17.137 INFO:tasks.rados.rados.0.smithi071.stdout:1367: finishing write tid 3 to smithi07134777-34 2024-09-11T01:38:17.137 INFO:tasks.rados.rados.0.smithi071.stdout:1367: finishing write tid 4 to smithi07134777-34 2024-09-11T01:38:17.138 INFO:tasks.rados.rados.0.smithi071.stdout:1367: oid 34 updating version 0 to 625 2024-09-11T01:38:17.138 INFO:tasks.rados.rados.0.smithi071.stdout:1367: oid 34 updating version 625 to 627 2024-09-11T01:38:17.138 INFO:tasks.rados.rados.0.smithi071.stdout:1367: oid 34 version 627 is already newer than 626 2024-09-11T01:38:17.138 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 627 (ObjNum 519 snap 143 seq_num 519) dirty exists 2024-09-11T01:38:17.138 INFO:tasks.rados.rados.0.smithi071.stdout:1367: left oid 34 (ObjNum 519 snap 143 seq_num 519) 2024-09-11T01:38:17.138 INFO:tasks.rados.rados.0.smithi071.stdout:1368: finishing write tid 1 to smithi07134777-10 2024-09-11T01:38:17.138 INFO:tasks.rados.rados.0.smithi071.stdout:1368: finishing write tid 2 to smithi07134777-10 2024-09-11T01:38:17.138 INFO:tasks.rados.rados.0.smithi071.stdout:1368: finishing write tid 3 to smithi07134777-10 2024-09-11T01:38:17.138 INFO:tasks.rados.rados.0.smithi071.stdout:1368: finishing write tid 4 to smithi07134777-10 2024-09-11T01:38:17.138 INFO:tasks.rados.rados.0.smithi071.stdout:1368: oid 10 updating version 0 to 359 2024-09-11T01:38:17.138 INFO:tasks.rados.rados.0.smithi071.stdout:1368: oid 10 version 359 is already newer than 358 2024-09-11T01:38:17.138 INFO:tasks.rados.rados.0.smithi071.stdout:1368: oid 10 updating version 359 to 360 2024-09-11T01:38:17.138 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 360 (ObjNum 520 snap 143 seq_num 520) dirty exists 2024-09-11T01:38:17.138 INFO:tasks.rados.rados.0.smithi071.stdout:1368: left oid 10 (ObjNum 520 snap 143 seq_num 520) 2024-09-11T01:38:17.139 INFO:tasks.rados.rados.0.smithi071.stdout:1369: seq_num 521 ranges {756447=716733,1887942=537832} 2024-09-11T01:38:17.144 INFO:tasks.rados.rados.0.smithi071.stdout:1369: writing smithi07134777-15 from 756447 to 1473180 tid 1 2024-09-11T01:38:17.150 INFO:tasks.rados.rados.0.smithi071.stdout:1369: writing smithi07134777-15 from 1887942 to 2425774 tid 2 2024-09-11T01:38:17.150 INFO:tasks.rados.rados.0.smithi071.stdout:1364: done (4 left) 2024-09-11T01:38:17.150 INFO:tasks.rados.rados.0.smithi071.stdout:1366: done (3 left) 2024-09-11T01:38:17.150 INFO:tasks.rados.rados.0.smithi071.stdout:1367: done (2 left) 2024-09-11T01:38:17.151 INFO:tasks.rados.rados.0.smithi071.stdout:1368: done (1 left) 2024-09-11T01:38:17.151 INFO:tasks.rados.rados.0.smithi071.stdout:1370: rollback oid 48 current snap is 143 2024-09-11T01:38:17.151 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 48 to 135 2024-09-11T01:38:17.151 INFO:tasks.rados.rados.0.smithi071.stdout:1369: finishing write tid 1 to smithi07134777-15 2024-09-11T01:38:17.151 INFO:tasks.rados.rados.0.smithi071.stdout:1371: snap_remove snap 133 2024-09-11T01:38:18.043 INFO:tasks.rados.rados.0.smithi071.stdout:1371: done (2 left) 2024-09-11T01:38:18.043 INFO:tasks.rados.rados.0.smithi071.stdout:1372: rollback oid 45 current snap is 143 2024-09-11T01:38:18.043 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 45 to 135 2024-09-11T01:38:18.043 INFO:tasks.rados.rados.0.smithi071.stdout:1369: finishing write tid 2 to smithi07134777-15 2024-09-11T01:38:18.043 INFO:tasks.rados.rados.0.smithi071.stdout:1369: finishing write tid 3 to smithi07134777-15 2024-09-11T01:38:18.043 INFO:tasks.rados.rados.0.smithi071.stdout:1369: finishing write tid 4 to smithi07134777-15 2024-09-11T01:38:18.044 INFO:tasks.rados.rados.0.smithi071.stdout:1369: oid 15 updating version 0 to 687 2024-09-11T01:38:18.044 INFO:tasks.rados.rados.0.smithi071.stdout:1369: oid 15 version 687 is already newer than 686 2024-09-11T01:38:18.044 INFO:tasks.rados.rados.0.smithi071.stdout:1369: oid 15 updating version 687 to 688 2024-09-11T01:38:18.044 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 688 (ObjNum 521 snap 143 seq_num 521) dirty exists 2024-09-11T01:38:18.044 INFO:tasks.rados.rados.0.smithi071.stdout:1369: left oid 15 (ObjNum 521 snap 143 seq_num 521) 2024-09-11T01:38:18.044 INFO:tasks.rados.rados.0.smithi071.stdout:1370: finishing rollback tid 0 to smithi07134777-48 2024-09-11T01:38:18.044 INFO:tasks.rados.rados.0.smithi071.stdout:1370: finishing rollback tid 1 to smithi07134777-48 2024-09-11T01:38:18.044 INFO:tasks.rados.rados.0.smithi071.stdout:1370: finishing rollback tid 2 to smithi07134777-48 2024-09-11T01:38:18.044 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 692 (ObjNum 494 snap 132 seq_num 494) dirty exists 2024-09-11T01:38:18.044 INFO:tasks.rados.rados.0.smithi071.stdout:1369: done (2 left) 2024-09-11T01:38:18.044 INFO:tasks.rados.rados.0.smithi071.stdout:1370: done (1 left) 2024-09-11T01:38:18.044 INFO:tasks.rados.rados.0.smithi071.stdout:1373: copy_from oid 18 from oid 34 current snap is 143 2024-09-11T01:38:18.044 INFO:tasks.rados.rados.0.smithi071.stdout:1374: read oid 30 snap -1 2024-09-11T01:38:18.044 INFO:tasks.rados.rados.0.smithi071.stdout:1374: expect deleted 2024-09-11T01:38:18.045 INFO:tasks.rados.rados.0.smithi071.stdout:1375: read oid 9 snap -1 2024-09-11T01:38:18.045 INFO:tasks.rados.rados.0.smithi071.stdout:1375: expect (ObjNum 475 snap 125 seq_num 475) 2024-09-11T01:38:18.045 INFO:tasks.rados.rados.0.smithi071.stdout:1376: snap_create 2024-09-11T01:38:18.063 INFO:tasks.rados.rados.0.smithi071.stdout:1375: expect (ObjNum 475 snap 125 seq_num 475) 2024-09-11T01:38:18.122 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 456574 2024-09-11T01:38:18.189 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:38:18.261 INFO:tasks.rados.rados.0.smithi071.stdout:1372: finishing rollback tid 0 to smithi07134777-45 2024-09-11T01:38:18.261 INFO:tasks.rados.rados.0.smithi071.stdout:1372: finishing rollback tid 1 to smithi07134777-45 2024-09-11T01:38:18.261 INFO:tasks.rados.rados.0.smithi071.stdout:1372: finishing rollback tid 2 to smithi07134777-45 2024-09-11T01:38:18.261 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 555 (ObjNum 437 snap 119 seq_num 437) dirty exists 2024-09-11T01:38:18.261 INFO:tasks.rados.rados.0.smithi071.stdout:1373: finishing copy_from racing read to smithi07134777-18 2024-09-11T01:38:18.261 INFO:tasks.rados.rados.0.smithi071.stdout:1373: finishing copy_from to smithi07134777-18 2024-09-11T01:38:18.261 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 627 (ObjNum 519 snap 143 seq_num 519) dirty exists 2024-09-11T01:38:18.922 INFO:tasks.rados.rados.0.smithi071.stdout:1372: done (4 left) 2024-09-11T01:38:18.922 INFO:tasks.rados.rados.0.smithi071.stdout:1373: done (3 left) 2024-09-11T01:38:18.922 INFO:tasks.rados.rados.0.smithi071.stdout:1374: done (2 left) 2024-09-11T01:38:18.922 INFO:tasks.rados.rados.0.smithi071.stdout:1375: done (1 left) 2024-09-11T01:38:18.922 INFO:tasks.rados.rados.0.smithi071.stdout:1376: done (0 left) 2024-09-11T01:38:18.922 INFO:tasks.rados.rados.0.smithi071.stdout:1377: write oid 14 current snap is 144 2024-09-11T01:38:18.922 INFO:tasks.rados.rados.0.smithi071.stdout:1377: seq_num 522 ranges {773254=791982,2346424=426830} 2024-09-11T01:38:18.930 INFO:tasks.rados.rados.0.smithi071.stdout:1377: writing smithi07134777-14 from 773254 to 1565236 tid 1 2024-09-11T01:38:18.937 INFO:tasks.rados.rados.0.smithi071.stdout:1377: writing smithi07134777-14 from 2346424 to 2773254 tid 2 2024-09-11T01:38:18.937 INFO:tasks.rados.rados.0.smithi071.stdout:1378: write (excl) oid 43 current snap is 144 2024-09-11T01:38:18.937 INFO:tasks.rados.rados.0.smithi071.stdout:1378: seq_num 523 ranges {790061=467231,2004906=541501} 2024-09-11T01:38:18.944 INFO:tasks.rados.rados.0.smithi071.stdout:1378: writing smithi07134777-43 from 790061 to 1257292 tid 1 2024-09-11T01:38:18.951 INFO:tasks.rados.rados.0.smithi071.stdout:1378: writing smithi07134777-43 from 2004906 to 2546407 tid 2 2024-09-11T01:38:18.951 INFO:tasks.rados.rados.0.smithi071.stdout:1379: snap_remove snap 135 2024-09-11T01:38:19.414 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-11T01:38:19.414 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-11T01:38:19.414 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-11T01:38:19.417 INFO:tasks.ceph.mon.b:Started 2024-09-11T01:38:19.417 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:38:19.417 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:38:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.529+0000 7fd98f784640 10 monclient: get_monmap_and_config 2024-09-11T01:38:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.529+0000 7fd98f784640 10 monclient: build_initial_monmap 2024-09-11T01:38:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.529+0000 7fd98f784640 10 monclient: monmap: 2024-09-11T01:38:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:38:19.530352+0000 2024-09-11T01:38:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:38:19.530352+0000 2024-09-11T01:38:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:38:19.531 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:38:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:38:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:38:19.532 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:38:19.532 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:19.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.530+0000 7fd98f784640 1 Processor -- start 2024-09-11T01:38:19.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.530+0000 7fd98f784640 1 -- start start 2024-09-11T01:38:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.530+0000 7fd98f784640 10 monclient: init 2024-09-11T01:38:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.530+0000 7fd98f784640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.530+0000 7fd98f784640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-11T01:38:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.530+0000 7fd98f784640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd9880a1380 0x7fd9880a1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.530+0000 7fd98f784640 10 monclient(hunting): picked mon.noname-b con 0x7fd9880a1380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.530+0000 7fd98f784640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd9880a0c90 0x7fd988135ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.530+0000 7fd98f784640 10 monclient(hunting): picked mon.noname-a con 0x7fd9880a0c90 addr v2:172.21.15.12:3300/0 2024-09-11T01:38:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.530+0000 7fd98f784640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd988136020 0x7fd988136470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.530+0000 7fd98f784640 10 monclient(hunting): picked mon.noname-c con 0x7fd988136020 addr v2:172.21.15.71:3300/0 2024-09-11T01:38:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.530+0000 7fd98f784640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.530+0000 7fd98f784640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fd9880a2b40 con 0x7fd9880a0c90 2024-09-11T01:38:19.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.530+0000 7fd98f784640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.530+0000 7fd98f784640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fd98809f9a0 con 0x7fd988136020 2024-09-11T01:38:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.530+0000 7fd98f784640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.530+0000 7fd98f784640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd9880a2230 con 0x7fd9880a1380 2024-09-11T01:38:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.530+0000 7fd98f784640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.530+0000 7fd98f784640 10 monclient(hunting): authenticate will time out at 1305.972974s 2024-09-11T01:38:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98d520640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd9880a1380 0x7fd9880a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98cd1f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd9880a0c90 0x7fd988135ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98d520640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd9880a1380 0x7fd9880a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45000/0 (socket says 172.21.15.12:45000) 2024-09-11T01:38:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98d520640 1 -- 172.21.15.12:0/443320532 learned_addr learned my addr 172.21.15.12:0/443320532 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:38:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98d520640 10 monclient(hunting): get_auth_request con 0x7fd9880a1380 auth_method 0 2024-09-11T01:38:19.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98d520640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98d520640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98d520640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98cd1f640 10 monclient(hunting): get_auth_request con 0x7fd9880a0c90 auth_method 0 2024-09-11T01:38:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98cd1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98cd1f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98cd1f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98d520640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98d520640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98d520640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98cd1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98cd1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98cd1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:19.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98dd21640 1 --2- 172.21.15.12:0/443320532 >> v2:172.21.15.71:3300/0 conn(0x7fd988136020 0x7fd988136470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98cd1f640 10 monclient(hunting): handle_auth_done global_id 94194 payload 274 2024-09-11T01:38:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98cd1f640 1 -- 172.21.15.12:0/443320532 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd9880a1380 msgr2=0x7fd9880a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:38:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98cd1f640 1 --2- 172.21.15.12:0/443320532 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd9880a1380 0x7fd9880a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98cd1f640 1 -- 172.21.15.12:0/443320532 >> v2:172.21.15.71:3300/0 conn(0x7fd988136020 msgr2=0x7fd988136470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:38:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98cd1f640 1 --2- 172.21.15.12:0/443320532 >> v2:172.21.15.71:3300/0 conn(0x7fd988136020 0x7fd988136470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98cd1f640 10 monclient: _finish_hunting 0 2024-09-11T01:38:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98cd1f640 1 monclient: found mon.noname-a 2024-09-11T01:38:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98cd1f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98cd1f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:38:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98cd1f640 1 -- 172.21.15.12:0/443320532 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9880a2510 con 0x7fd9880a0c90 2024-09-11T01:38:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98dd21640 10 monclient: get_auth_request con 0x7fd988136020 auth_method 0 2024-09-11T01:38:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98d520640 1 --2- 172.21.15.12:0/443320532 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd9880a1380 0x7fd9880a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:38:19.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.531+0000 7fd98dd21640 1 --2- 172.21.15.12:0/443320532 >> v2:172.21.15.71:3300/0 conn(0x7fd988136020 0x7fd988136470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:38:19.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.534+0000 7fd98cd1f640 1 -- 172.21.15.12:0/443320532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9880a0c90 msgr2=0x7fd988135ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:19.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.534+0000 7fd98cd1f640 1 --2- 172.21.15.12:0/443320532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9880a0c90 0x7fd988135ae0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd98806eb90 tx=0x7fd978029fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66d2d3e39b605bfa server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:19.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.534+0000 7fd97ffff640 1 -- 172.21.15.12:0/443320532 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd978037070 con 0x7fd9880a0c90 2024-09-11T01:38:19.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.534+0000 7fd97ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:19.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.534+0000 7fd97ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:38:19.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.534+0000 7fd97ffff640 10 monclient: dump: 2024-09-11T01:38:19.545 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:19.545 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:19.545 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:19.545 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:19.545 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:19.545 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:19.545 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:19.545 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:19.545 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:19.545 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:19.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.534+0000 7fd97ffff640 10 monclient: _finish_auth 0 2024-09-11T01:38:19.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.534+0000 7fd97ffff640 10 monclient: _check_auth_tickets 2024-09-11T01:38:19.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.534+0000 7fd97ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:19.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.534+0000 7fd98f784640 5 monclient: authenticate success, global_id 94194 2024-09-11T01:38:19.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.534+0000 7fd97ffff640 1 -- 172.21.15.12:0/443320532 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd978032440 con 0x7fd9880a0c90 2024-09-11T01:38:19.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.534+0000 7fd98f784640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:38:19.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.534+0000 7fd97ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:19.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.534+0000 7fd98f784640 10 monclient: get_monmap_and_config success 2024-09-11T01:38:19.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.534+0000 7fd98f784640 4 set_mon_vals no callback set 2024-09-11T01:38:19.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.534+0000 7fd97ffff640 1 -- 172.21.15.12:0/443320532 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd978032810 con 0x7fd9880a0c90 2024-09-11T01:38:19.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.534+0000 7fd97ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:19.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.534+0000 7fd97ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:38:19.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.534+0000 7fd97ffff640 10 monclient: dump: 2024-09-11T01:38:19.546 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:19.546 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:19.546 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:19.546 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:19.547 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:19.547 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:19.547 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:19.547 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:19.547 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:19.547 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:19.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.534+0000 7fd98f784640 10 monclient: shutdown 2024-09-11T01:38:19.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.534+0000 7fd98f784640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:19.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.534+0000 7fd98f784640 1 -- 172.21.15.12:0/443320532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9880a0c90 msgr2=0x7fd988135ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:19.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.534+0000 7fd98f784640 1 --2- 172.21.15.12:0/443320532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9880a0c90 0x7fd988135ae0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd98806eb90 tx=0x7fd978029fe0 comp rx=0 tx=0).stop 2024-09-11T01:38:19.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.535+0000 7fd98f784640 1 -- 172.21.15.12:0/443320532 shutdown_connections 2024-09-11T01:38:19.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.535+0000 7fd98f784640 1 --2- 172.21.15.12:0/443320532 >> v2:172.21.15.71:3300/0 conn(0x7fd988136020 0x7fd988136470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:19.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.535+0000 7fd98f784640 1 --2- 172.21.15.12:0/443320532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9880a0c90 0x7fd988135ae0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:19.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.535+0000 7fd98f784640 1 --2- 172.21.15.12:0/443320532 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd9880a1380 0x7fd9880a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:19.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.535+0000 7fd98f784640 1 -- 172.21.15.12:0/443320532 >> 172.21.15.12:0/443320532 conn(0x7fd9880a2d90 msgr2=0x7fd9880a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:19.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.535+0000 7fd98f784640 1 -- 172.21.15.12:0/443320532 shutdown_connections 2024-09-11T01:38:19.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.535+0000 7fd98f784640 1 -- 172.21.15.12:0/443320532 wait complete. 2024-09-11T01:38:19.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.535+0000 7fd98f784640 10 monclient: build_initial_monmap 2024-09-11T01:38:19.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.535+0000 7fd98f784640 10 monclient: monmap: 2024-09-11T01:38:19.548 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:19.548 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:38:19.548 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:38:19.548 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:38:19.548 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:19.548 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:19.548 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:38:19.549 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:38:19.549 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:38:19.549 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:19.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98f784640 1 Processor -- start 2024-09-11T01:38:19.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98f784640 1 -- start start 2024-09-11T01:38:19.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98f784640 10 monclient: init 2024-09-11T01:38:19.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98f784640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:19.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98f784640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:38:19.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98f784640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9880a0c90 0x7fd9881c7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:19.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98f784640 10 monclient(hunting): picked mon.noname-a con 0x7fd9880a0c90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:38:19.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98f784640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd9880a1380 0x7fd988135050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:19.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98f784640 10 monclient(hunting): picked mon.noname-c con 0x7fd9880a1380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:19.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98f784640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd988136020 0x7fd988135590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:19.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98f784640 10 monclient(hunting): picked mon.noname-b con 0x7fd988136020 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:38:19.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98f784640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:19.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98f784640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd98809ce20 con 0x7fd9880a0c90 2024-09-11T01:38:19.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98f784640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:19.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98f784640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd98809cca0 con 0x7fd988136020 2024-09-11T01:38:19.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98f784640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:19.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98f784640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd9880a0390 con 0x7fd9880a1380 2024-09-11T01:38:19.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98f784640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:19.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98f784640 10 monclient(hunting): authenticate will time out at 1305.978711s 2024-09-11T01:38:19.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98d520640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9880a0c90 0x7fd9881c7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:19.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98d520640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9880a0c90 0x7fd9881c7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39148/0 (socket says 172.21.15.12:39148) 2024-09-11T01:38:19.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98d520640 1 -- 172.21.15.12:0/3472664972 learned_addr learned my addr 172.21.15.12:0/3472664972 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:38:19.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98d520640 10 monclient(hunting): get_auth_request con 0x7fd9880a0c90 auth_method 0 2024-09-11T01:38:19.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98d520640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:19.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98d520640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:19.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98d520640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:19.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98cd1f640 1 --2- 172.21.15.12:0/3472664972 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd9880a1380 0x7fd988135050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:19.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98cd1f640 10 monclient(hunting): get_auth_request con 0x7fd9880a1380 auth_method 0 2024-09-11T01:38:19.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98cd1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:19.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98cd1f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:19.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98cd1f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:19.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98d520640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:19.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98d520640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:19.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98d520640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:19.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98cd1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:19.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98cd1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:19.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.536+0000 7fd98cd1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:19.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.537+0000 7fd98dd21640 1 --2- 172.21.15.12:0/3472664972 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd988136020 0x7fd988135590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:19.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.537+0000 7fd98dd21640 10 monclient(hunting): get_auth_request con 0x7fd988136020 auth_method 0 2024-09-11T01:38:19.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.537+0000 7fd98dd21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:19.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.537+0000 7fd98dd21640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:19.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.537+0000 7fd98dd21640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:19.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.537+0000 7fd98d520640 10 monclient(hunting): handle_auth_done global_id 94197 payload 824 2024-09-11T01:38:19.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.537+0000 7fd98d520640 1 -- 172.21.15.12:0/3472664972 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd9880a1380 msgr2=0x7fd988135050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:19.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.537+0000 7fd98d520640 1 --2- 172.21.15.12:0/3472664972 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd9880a1380 0x7fd988135050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:19.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.537+0000 7fd98d520640 1 -- 172.21.15.12:0/3472664972 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd988136020 msgr2=0x7fd988135590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:19.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.537+0000 7fd98d520640 1 --2- 172.21.15.12:0/3472664972 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd988136020 0x7fd988135590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:19.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.537+0000 7fd98d520640 10 monclient: _finish_hunting 0 2024-09-11T01:38:19.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.537+0000 7fd98d520640 1 monclient: found mon.noname-a 2024-09-11T01:38:19.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.537+0000 7fd98d520640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:19.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.537+0000 7fd98d520640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:38:19.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.537+0000 7fd98d520640 1 -- 172.21.15.12:0/3472664972 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd98809d480 con 0x7fd9880a0c90 2024-09-11T01:38:19.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.539+0000 7fd98d520640 1 -- 172.21.15.12:0/3472664972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9880a0c90 msgr2=0x7fd9881c7580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:19.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.539+0000 7fd98d520640 1 --2- 172.21.15.12:0/3472664972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9880a0c90 0x7fd9881c7580 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd98400ea00 tx=0x7fd98400eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:19.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.539+0000 7fd97dffb640 1 -- 172.21.15.12:0/3472664972 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd984017070 con 0x7fd9880a0c90 2024-09-11T01:38:19.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.539+0000 7fd97dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:19.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.539+0000 7fd97dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:38:19.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.539+0000 7fd97dffb640 10 monclient: dump: 2024-09-11T01:38:19.553 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:19.553 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:19.554 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:19.554 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:19.554 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:19.554 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:19.554 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:19.554 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:19.554 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:19.554 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:19.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.539+0000 7fd97dffb640 10 monclient: _finish_auth 0 2024-09-11T01:38:19.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.539+0000 7fd97dffb640 10 monclient: _check_auth_tickets 2024-09-11T01:38:19.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.539+0000 7fd97dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:19.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.539+0000 7fd97dffb640 1 -- 172.21.15.12:0/3472664972 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd98400cc70 con 0x7fd9880a0c90 2024-09-11T01:38:19.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.539+0000 7fd97dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:19.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.539+0000 7fd97dffb640 1 -- 172.21.15.12:0/3472664972 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd984013470 con 0x7fd9880a0c90 2024-09-11T01:38:19.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.539+0000 7fd98f784640 5 monclient: authenticate success, global_id 94197 2024-09-11T01:38:19.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.539+0000 7fd98f784640 10 monclient: _renew_subs 2024-09-11T01:38:19.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.539+0000 7fd98f784640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:38:19.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.539+0000 7fd98f784640 1 -- 172.21.15.12:0/3472664972 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd98806f970 con 0x7fd9880a0c90 2024-09-11T01:38:19.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.539+0000 7fd97f7fe640 4 set_mon_vals no callback set 2024-09-11T01:38:19.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.539+0000 7fd97dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:19.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.539+0000 7fd97dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:38:19.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.539+0000 7fd97dffb640 10 monclient: dump: 2024-09-11T01:38:19.555 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:19.555 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:19.555 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:19.555 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:19.555 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:19.555 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:19.555 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:19.556 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:19.556 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:19.556 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:19.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.539+0000 7fd98f784640 10 monclient: _renew_subs 2024-09-11T01:38:19.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.539+0000 7fd98f784640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:38:19.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.539+0000 7fd98f784640 1 -- 172.21.15.12:0/3472664972 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd988123a20 con 0x7fd9880a0c90 2024-09-11T01:38:19.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.540+0000 7fd98f784640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:19.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.540+0000 7fd98f784640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:19.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.540+0000 7fd98f784640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:38:19.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.540+0000 7fd98f784640 1 -- 172.21.15.12:0/3472664972 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd988135ae0 con 0x7fd9880a0c90 2024-09-11T01:38:19.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.541+0000 7fd97dffb640 1 -- 172.21.15.12:0/3472664972 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fd984024060 con 0x7fd9880a0c90 2024-09-11T01:38:19.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.541+0000 7fd97dffb640 1 --2- 172.21.15.12:0/3472664972 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd96c03e210 0x7fd96c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:19.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.541+0000 7fd97dffb640 1 -- 172.21.15.12:0/3472664972 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(308..308 src has 186..308) ==== 4799+0+0 (secure 0 0 0) 0x7fd984052cc0 con 0x7fd9880a0c90 2024-09-11T01:38:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.541+0000 7fd98cd1f640 1 --2- 172.21.15.12:0/3472664972 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd96c03e210 0x7fd96c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.541+0000 7fd98cd1f640 10 monclient: get_auth_request con 0x7fd96c03e210 auth_method 0 2024-09-11T01:38:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.544+0000 7fd98cd1f640 1 --2- 172.21.15.12:0/3472664972 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd96c03e210 0x7fd96c0406d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd98806ea30 tx=0x7fd97802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.545+0000 7fd97dffb640 1 -- 172.21.15.12:0/3472664972 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd984023d00 con 0x7fd9880a0c90 2024-09-11T01:38:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.545+0000 7fd97dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:19.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.545+0000 7fd97dffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:38:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.802+0000 7fd98f784640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:38:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.802+0000 7fd98f784640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:38:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.802+0000 7fd98f784640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:38:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.802+0000 7fd98f784640 1 -- 172.21.15.12:0/3472664972 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd988122220 con 0x7fd9880a0c90 2024-09-11T01:38:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.802+0000 7fd97dffb640 1 -- 172.21.15.12:0/3472664972 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fd98401b3b0 con 0x7fd9880a0c90 2024-09-11T01:38:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.802+0000 7fd97dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:38:19.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.802+0000 7fd97dffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:38:19.804 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":90,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:38:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.805+0000 7fd98f784640 1 -- 172.21.15.12:0/3472664972 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd96c03e210 msgr2=0x7fd96c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.805+0000 7fd98f784640 1 --2- 172.21.15.12:0/3472664972 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd96c03e210 0x7fd96c0406d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd98806ea30 tx=0x7fd97802e000 comp rx=0 tx=0).stop 2024-09-11T01:38:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.805+0000 7fd98f784640 10 monclient: shutdown 2024-09-11T01:38:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.805+0000 7fd98f784640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.805+0000 7fd98f784640 1 -- 172.21.15.12:0/3472664972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9880a0c90 msgr2=0x7fd9881c7580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.805+0000 7fd98f784640 1 --2- 172.21.15.12:0/3472664972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9880a0c90 0x7fd9881c7580 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd98400ea00 tx=0x7fd98400eec0 comp rx=0 tx=0).stop 2024-09-11T01:38:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.805+0000 7fd98f784640 1 -- 172.21.15.12:0/3472664972 shutdown_connections 2024-09-11T01:38:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.805+0000 7fd98f784640 1 --2- 172.21.15.12:0/3472664972 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd988136020 0x7fd988135590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:19.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.805+0000 7fd98f784640 1 --2- 172.21.15.12:0/3472664972 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd9880a1380 0x7fd988135050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.805+0000 7fd98f784640 1 --2- 172.21.15.12:0/3472664972 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd96c03e210 0x7fd96c0406d0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.805+0000 7fd98f784640 1 --2- 172.21.15.12:0/3472664972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9880a0c90 0x7fd9881c7580 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.805+0000 7fd98f784640 1 -- 172.21.15.12:0/3472664972 >> 172.21.15.12:0/3472664972 conn(0x7fd9880a2d90 msgr2=0x7fd9880a5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.805+0000 7fd98f784640 1 -- 172.21.15.12:0/3472664972 shutdown_connections 2024-09-11T01:38:19.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:19.805+0000 7fd98f784640 1 -- 172.21.15.12:0/3472664972 wait complete. 2024-09-11T01:38:19.927 INFO:tasks.rados.rados.0.smithi071.stdout:1379: done (2 left) 2024-09-11T01:38:19.927 INFO:tasks.rados.rados.0.smithi071.stdout:1380: write (excl) oid 11 current snap is 144 2024-09-11T01:38:19.927 INFO:tasks.rados.rados.0.smithi071.stdout:1380: seq_num 524 ranges {406868=542480,1547035=401512,2690354=116514} 2024-09-11T01:38:19.932 INFO:tasks.rados.rados.0.smithi071.stdout:1380: writing smithi07134777-11 from 406868 to 949348 tid 1 2024-09-11T01:38:19.938 INFO:tasks.rados.rados.0.smithi071.stdout:1380: writing smithi07134777-11 from 1547035 to 1948547 tid 2 2024-09-11T01:38:19.944 INFO:tasks.rados.rados.0.smithi071.stdout:1380: writing smithi07134777-11 from 2690354 to 2806868 tid 3 2024-09-11T01:38:19.944 INFO:tasks.rados.rados.0.smithi071.stdout:1381: rollback oid 32 current snap is 144 2024-09-11T01:38:19.944 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 32 to 142 2024-09-11T01:38:19.944 INFO:tasks.rados.rados.0.smithi071.stdout:1377: finishing write tid 1 to smithi07134777-14 2024-09-11T01:38:19.944 INFO:tasks.rados.rados.0.smithi071.stdout:1377: finishing write tid 2 to smithi07134777-14 2024-09-11T01:38:19.944 INFO:tasks.rados.rados.0.smithi071.stdout:1377: finishing write tid 3 to smithi07134777-14 2024-09-11T01:38:19.944 INFO:tasks.rados.rados.0.smithi071.stdout:1377: finishing write tid 4 to smithi07134777-14 2024-09-11T01:38:19.944 INFO:tasks.rados.rados.0.smithi071.stdout:1377: oid 14 updating version 0 to 703 2024-09-11T01:38:19.944 INFO:tasks.rados.rados.0.smithi071.stdout:1377: oid 14 version 703 is already newer than 702 2024-09-11T01:38:19.944 INFO:tasks.rados.rados.0.smithi071.stdout:1377: oid 14 version 703 is already newer than 701 2024-09-11T01:38:19.945 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 703 (ObjNum 522 snap 144 seq_num 522) dirty exists 2024-09-11T01:38:19.945 INFO:tasks.rados.rados.0.smithi071.stdout:1377: left oid 14 (ObjNum 522 snap 144 seq_num 522) 2024-09-11T01:38:19.945 INFO:tasks.rados.rados.0.smithi071.stdout:1378: finishing write tid 1 to smithi07134777-43 2024-09-11T01:38:19.945 INFO:tasks.rados.rados.0.smithi071.stdout:1378: finishing write tid 2 to smithi07134777-43 2024-09-11T01:38:19.945 INFO:tasks.rados.rados.0.smithi071.stdout:1378: finishing write tid 3 to smithi07134777-43 2024-09-11T01:38:19.945 INFO:tasks.rados.rados.0.smithi071.stdout:1378: finishing write tid 4 to smithi07134777-43 2024-09-11T01:38:19.945 INFO:tasks.rados.rados.0.smithi071.stdout:1378: oid 43 updating version 0 to 633 2024-09-11T01:38:19.945 INFO:tasks.rados.rados.0.smithi071.stdout:1378: oid 43 version 633 is already newer than 632 2024-09-11T01:38:19.945 INFO:tasks.rados.rados.0.smithi071.stdout:1378: oid 43 version 633 is already newer than 631 2024-09-11T01:38:19.945 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 633 (ObjNum 523 snap 144 seq_num 523) dirty exists 2024-09-11T01:38:19.945 INFO:tasks.rados.rados.0.smithi071.stdout:1378: left oid 43 (ObjNum 523 snap 144 seq_num 523) 2024-09-11T01:38:19.945 INFO:tasks.rados.rados.0.smithi071.stdout:1380: finishing write tid 1 to smithi07134777-11 2024-09-11T01:38:19.945 INFO:tasks.rados.rados.0.smithi071.stdout:1380: finishing write tid 2 to smithi07134777-11 2024-09-11T01:38:19.945 INFO:tasks.rados.rados.0.smithi071.stdout:1377: done (3 left) 2024-09-11T01:38:19.946 INFO:tasks.rados.rados.0.smithi071.stdout:1378: done (2 left) 2024-09-11T01:38:19.946 INFO:tasks.rados.rados.0.smithi071.stdout:1382: rollback oid 1 current snap is 144 2024-09-11T01:38:19.946 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 1 to 142 2024-09-11T01:38:19.946 INFO:tasks.rados.rados.0.smithi071.stdout:1383: delete oid 50 current snap is 144 2024-09-11T01:38:19.947 INFO:tasks.rados.rados.0.smithi071.stdout:1382: finishing rollback tid 1 to smithi07134777-1 2024-09-11T01:38:19.947 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 0 (ObjNum 289 snap 0 seq_num 3922329442) dirty dne 2024-09-11T01:38:19.947 INFO:tasks.rados.rados.0.smithi071.stdout:1380: finishing write tid 3 to smithi07134777-11 2024-09-11T01:38:19.947 INFO:tasks.rados.rados.0.smithi071.stdout:1380: finishing write tid 4 to smithi07134777-11 2024-09-11T01:38:19.947 INFO:tasks.rados.rados.0.smithi071.stdout:1380: finishing write tid 5 to smithi07134777-11 2024-09-11T01:38:19.947 INFO:tasks.rados.rados.0.smithi071.stdout:1380: oid 11 updating version 0 to 629 2024-09-11T01:38:19.947 INFO:tasks.rados.rados.0.smithi071.stdout:1380: oid 11 updating version 629 to 630 2024-09-11T01:38:19.947 INFO:tasks.rados.rados.0.smithi071.stdout:1380: oid 11 updating version 630 to 631 2024-09-11T01:38:19.947 INFO:tasks.rados.rados.0.smithi071.stdout:1380: oid 11 version 631 is already newer than 628 2024-09-11T01:38:19.947 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 631 (ObjNum 524 snap 144 seq_num 524) dirty exists 2024-09-11T01:38:19.948 INFO:tasks.rados.rados.0.smithi071.stdout:1380: left oid 11 (ObjNum 524 snap 144 seq_num 524) 2024-09-11T01:38:19.948 INFO:tasks.rados.rados.0.smithi071.stdout:1380: done (3 left) 2024-09-11T01:38:19.948 INFO:tasks.rados.rados.0.smithi071.stdout:1382: done (2 left) 2024-09-11T01:38:19.948 INFO:tasks.rados.rados.0.smithi071.stdout:1383: done (1 left) 2024-09-11T01:38:19.948 INFO:tasks.rados.rados.0.smithi071.stdout:1384: snap_remove snap 139 2024-09-11T01:38:20.926 INFO:tasks.rados.rados.0.smithi071.stdout:1384: done (1 left) 2024-09-11T01:38:20.927 INFO:tasks.rados.rados.0.smithi071.stdout:1385: read oid 39 snap -1 2024-09-11T01:38:20.927 INFO:tasks.rados.rados.0.smithi071.stdout:1385: expect (ObjNum 510 snap 139 seq_num 510) 2024-09-11T01:38:20.927 INFO:tasks.rados.rados.0.smithi071.stdout:1386: copy_from oid 21 from oid 47 current snap is 144 2024-09-11T01:38:20.927 INFO:tasks.rados.rados.0.smithi071.stdout:1381: finishing rollback tid 1 to smithi07134777-32 2024-09-11T01:38:20.927 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 0 (ObjNum 513 snap 0 seq_num 2370826128) dirty dne 2024-09-11T01:38:20.927 INFO:tasks.rados.rados.0.smithi071.stdout:1381: done (2 left) 2024-09-11T01:38:20.927 INFO:tasks.rados.rados.0.smithi071.stdout:1387: snap_create 2024-09-11T01:38:20.936 INFO:tasks.rados.rados.0.smithi071.stdout:1385: expect (ObjNum 510 snap 139 seq_num 510) 2024-09-11T01:38:21.068 INFO:tasks.rados.rados.0.smithi071.stdout:1386: finishing copy_from racing read to smithi07134777-21 2024-09-11T01:38:21.068 INFO:tasks.rados.rados.0.smithi071.stdout:1386: finishing copy_from to smithi07134777-21 2024-09-11T01:38:21.068 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 480 (ObjNum 518 snap 141 seq_num 518) dirty exists 2024-09-11T01:38:21.936 INFO:tasks.rados.rados.0.smithi071.stdout:1385: done (2 left) 2024-09-11T01:38:21.936 INFO:tasks.rados.rados.0.smithi071.stdout:1386: done (1 left) 2024-09-11T01:38:21.936 INFO:tasks.rados.rados.0.smithi071.stdout:1387: done (0 left) 2024-09-11T01:38:21.936 INFO:tasks.rados.rados.0.smithi071.stdout:1388: write (excl) oid 32 current snap is 145 2024-09-11T01:38:21.936 INFO:tasks.rados.rados.0.smithi071.stdout:1388: seq_num 526 ranges {440482=409331,1380352=405181,2461553=378929} 2024-09-11T01:38:21.940 INFO:tasks.rados.rados.0.smithi071.stdout:1388: writing smithi07134777-32 from 440482 to 849813 tid 1 2024-09-11T01:38:21.945 INFO:tasks.rados.rados.0.smithi071.stdout:1388: writing smithi07134777-32 from 1380352 to 1785533 tid 2 2024-09-11T01:38:21.951 INFO:tasks.rados.rados.0.smithi071.stdout:1388: writing smithi07134777-32 from 2461553 to 2840482 tid 3 2024-09-11T01:38:21.951 INFO:tasks.rados.rados.0.smithi071.stdout:1389: snap_create 2024-09-11T01:38:21.951 INFO:tasks.rados.rados.0.smithi071.stdout:1388: finishing write tid 1 to smithi07134777-32 2024-09-11T01:38:21.951 INFO:tasks.rados.rados.0.smithi071.stdout:1388: finishing write tid 2 to smithi07134777-32 2024-09-11T01:38:21.954 INFO:tasks.rados.rados.0.smithi071.stdout:1388: finishing write tid 3 to smithi07134777-32 2024-09-11T01:38:21.955 INFO:tasks.rados.rados.0.smithi071.stdout:1388: finishing write tid 4 to smithi07134777-32 2024-09-11T01:38:21.955 INFO:tasks.rados.rados.0.smithi071.stdout:1388: finishing write tid 5 to smithi07134777-32 2024-09-11T01:38:21.955 INFO:tasks.rados.rados.0.smithi071.stdout:1388: oid 32 updating version 0 to 578 2024-09-11T01:38:21.955 INFO:tasks.rados.rados.0.smithi071.stdout:1388: oid 32 version 578 is already newer than 577 2024-09-11T01:38:21.955 INFO:tasks.rados.rados.0.smithi071.stdout:1388: oid 32 version 578 is already newer than 575 2024-09-11T01:38:21.955 INFO:tasks.rados.rados.0.smithi071.stdout:1388: oid 32 version 578 is already newer than 576 2024-09-11T01:38:21.955 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 578 (ObjNum 526 snap 145 seq_num 526) dirty exists 2024-09-11T01:38:21.955 INFO:tasks.rados.rados.0.smithi071.stdout:1388: left oid 32 (ObjNum 526 snap 145 seq_num 526) 2024-09-11T01:38:22.818 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:38:22.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.930+0000 7f5add945640 10 monclient: get_monmap_and_config 2024-09-11T01:38:22.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.930+0000 7f5add945640 10 monclient: build_initial_monmap 2024-09-11T01:38:22.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.930+0000 7f5add945640 10 monclient: monmap: 2024-09-11T01:38:22.932 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:22.932 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:22.932 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:38:22.931203+0000 2024-09-11T01:38:22.932 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:38:22.931203+0000 2024-09-11T01:38:22.932 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:22.932 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:22.932 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:38:22.933 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:38:22.933 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:38:22.933 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:38:22.933 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:38:22.933 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:22.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.931+0000 7f5add945640 1 Processor -- start 2024-09-11T01:38:22.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.931+0000 7f5add945640 1 -- start start 2024-09-11T01:38:22.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.931+0000 7f5add945640 10 monclient: init 2024-09-11T01:38:22.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.931+0000 7f5add945640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:22.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.931+0000 7f5add945640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-11T01:38:22.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.931+0000 7f5add945640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5ad8138230 addr v1:172.21.15.71:6789/0 2024-09-11T01:38:22.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.931+0000 7f5add945640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ad809d030 0x7f5ad813d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:22.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.931+0000 7f5add945640 10 monclient(hunting): picked mon.noname-b con 0x7f5ad809d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:22.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.931+0000 7f5add945640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5ad809d460 0x7f5ad813f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:22.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.931+0000 7f5add945640 10 monclient(hunting): picked mon.noname-c con 0x7f5ad809d460 addr v2:172.21.15.71:3300/0 2024-09-11T01:38:22.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.931+0000 7f5add945640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ad80a6e30 con 0x7f5ad8138230 2024-09-11T01:38:22.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.931+0000 7f5add945640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:22.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.931+0000 7f5add945640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f5ad809c9c0 con 0x7f5ad809d460 2024-09-11T01:38:22.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.931+0000 7f5add945640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:22.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.931+0000 7f5add945640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5ad809ba20 con 0x7f5ad809d030 2024-09-11T01:38:22.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.931+0000 7f5add945640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:22.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.931+0000 7f5add945640 10 monclient(hunting): authenticate will time out at 1309.373793s 2024-09-11T01:38:22.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.931+0000 7f5ad67fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ad809d030 0x7f5ad813d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:22.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.931+0000 7f5ad67fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ad809d030 0x7f5ad813d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45032/0 (socket says 172.21.15.12:45032) 2024-09-11T01:38:22.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.931+0000 7f5ad67fc640 1 -- 172.21.15.12:0/1365441144 learned_addr learned my addr 172.21.15.12:0/1365441144 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:38:22.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.931+0000 7f5ad67fc640 10 monclient(hunting): get_auth_request con 0x7f5ad809d030 auth_method 0 2024-09-11T01:38:22.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.931+0000 7f5ad67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:22.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.931+0000 7f5ad67fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:22.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.931+0000 7f5ad67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:22.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.931+0000 7f5ad77fe640 1 --2- 172.21.15.12:0/1365441144 >> v2:172.21.15.71:3300/0 conn(0x7f5ad809d460 0x7f5ad813f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:22.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.932+0000 7f5ad77fe640 10 monclient(hunting): get_auth_request con 0x7f5ad809d460 auth_method 0 2024-09-11T01:38:22.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.932+0000 7f5ad77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:22.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.932+0000 7f5ad77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:22.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.932+0000 7f5ad77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:22.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.932+0000 7f5ad67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:22.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.932+0000 7f5ad67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:22.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.932+0000 7f5ad77fe640 1 -- 172.21.15.12:0/1365441144 >> v2:172.21.15.71:3300/0 conn(0x7f5ad809d460 msgr2=0x7f5ad813f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:38:22.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.932+0000 7f5ad77fe640 1 -- 172.21.15.12:0/1365441144 >> v2:172.21.15.71:3300/0 conn(0x7f5ad809d460 msgr2=0x7f5ad813f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:38:22.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.932+0000 7f5ad77fe640 1 --2- 172.21.15.12:0/1365441144 >> v2:172.21.15.71:3300/0 conn(0x7f5ad809d460 0x7f5ad813f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:38:22.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.932+0000 7f5ad67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:22.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.932+0000 7f5ad77fe640 1 --2- 172.21.15.12:0/1365441144 >> v2:172.21.15.71:3300/0 conn(0x7f5ad809d460 0x7f5ad813f5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:38:22.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.932+0000 7f5ad67fc640 10 monclient(hunting): handle_auth_done global_id 84293 payload 274 2024-09-11T01:38:22.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.932+0000 7f5ad67fc640 1 -- 172.21.15.12:0/1365441144 >> v2:172.21.15.71:3300/0 conn(0x7f5ad809d460 msgr2=0x7f5ad813f5e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:38:22.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.932+0000 7f5ad67fc640 1 --2- 172.21.15.12:0/1365441144 >> v2:172.21.15.71:3300/0 conn(0x7f5ad809d460 0x7f5ad813f5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:22.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.932+0000 7f5ad67fc640 1 -- 172.21.15.12:0/1365441144 >> v1:172.21.15.71:6789/0 conn(0x7f5ad8138230 legacy=0x7f5ad8138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:38:22.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.932+0000 7f5ad67fc640 10 monclient: _finish_hunting 0 2024-09-11T01:38:22.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.932+0000 7f5ad67fc640 1 monclient: found mon.noname-b 2024-09-11T01:38:22.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.932+0000 7f5ad67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:22.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.932+0000 7f5ad67fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:38:22.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.932+0000 7f5ad67fc640 1 -- 172.21.15.12:0/1365441144 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad80a66a0 con 0x7f5ad809d030 2024-09-11T01:38:22.939 INFO:tasks.rados.rados.0.smithi071.stdout:1388: done (1 left) 2024-09-11T01:38:22.940 INFO:tasks.rados.rados.0.smithi071.stdout:1389: done (0 left) 2024-09-11T01:38:22.940 INFO:tasks.rados.rados.0.smithi071.stdout:1390: read oid 29 snap 145 2024-09-11T01:38:22.940 INFO:tasks.rados.rados.0.smithi071.stdout:1390: expect (ObjNum 505 snap 138 seq_num 505) 2024-09-11T01:38:22.940 INFO:tasks.rados.rados.0.smithi071.stdout:1391: snap_create 2024-09-11T01:38:22.940 INFO:tasks.rados.rados.0.smithi071.stdout:1390: expect (ObjNum 505 snap 138 seq_num 505) 2024-09-11T01:38:22.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.935+0000 7f5ad67fc640 1 -- 172.21.15.12:0/1365441144 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ad809d030 msgr2=0x7f5ad813d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:22.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.935+0000 7f5ad67fc640 1 --2- 172.21.15.12:0/1365441144 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ad809d030 0x7f5ad813d090 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5ad806e500 tx=0x7f5acc02a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b394d9c11499d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:22.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.937+0000 7f5ad5ffb640 1 -- 172.21.15.12:0/1365441144 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5acc037070 con 0x7f5ad809d030 2024-09-11T01:38:22.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.937+0000 7f5ad5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:22.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.937+0000 7f5ad5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:38:22.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.937+0000 7f5ad5ffb640 10 monclient: dump: 2024-09-11T01:38:22.948 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:22.948 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:22.948 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:22.948 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:22.948 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:22.948 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:22.948 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:22.949 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:22.949 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:22.949 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:22.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.937+0000 7f5ad5ffb640 10 monclient: _finish_auth 0 2024-09-11T01:38:22.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.937+0000 7f5ad5ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:38:22.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.937+0000 7f5ad5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:22.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.937+0000 7f5ad5ffb640 1 -- 172.21.15.12:0/1365441144 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5acc032440 con 0x7f5ad809d030 2024-09-11T01:38:22.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.937+0000 7f5add945640 5 monclient: authenticate success, global_id 84293 2024-09-11T01:38:22.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.937+0000 7f5add945640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:38:22.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.937+0000 7f5ad5ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:22.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.937+0000 7f5ad5ffb640 1 -- 172.21.15.12:0/1365441144 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5acc032810 con 0x7f5ad809d030 2024-09-11T01:38:22.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.937+0000 7f5ad5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:22.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.937+0000 7f5ad5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:38:22.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.937+0000 7f5ad5ffb640 10 monclient: dump: 2024-09-11T01:38:22.950 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:22.950 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:22.950 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:22.950 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:22.950 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:22.950 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:22.950 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:22.950 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:22.950 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:22.950 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:22.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.937+0000 7f5add945640 10 monclient: get_monmap_and_config success 2024-09-11T01:38:22.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.937+0000 7f5add945640 4 set_mon_vals no callback set 2024-09-11T01:38:22.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.937+0000 7f5add945640 10 monclient: shutdown 2024-09-11T01:38:22.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.937+0000 7f5add945640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:22.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.937+0000 7f5add945640 1 -- 172.21.15.12:0/1365441144 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ad809d030 msgr2=0x7f5ad813d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:22.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.937+0000 7f5add945640 1 --2- 172.21.15.12:0/1365441144 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ad809d030 0x7f5ad813d090 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5ad806e500 tx=0x7f5acc02a060 comp rx=0 tx=0).stop 2024-09-11T01:38:22.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.938+0000 7f5add945640 1 -- 172.21.15.12:0/1365441144 shutdown_connections 2024-09-11T01:38:22.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.938+0000 7f5add945640 1 --2- 172.21.15.12:0/1365441144 >> v2:172.21.15.71:3300/0 conn(0x7f5ad809d460 0x7f5ad813f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:22.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.938+0000 7f5add945640 1 --2- 172.21.15.12:0/1365441144 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ad809d030 0x7f5ad813d090 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:22.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.938+0000 7f5add945640 1 -- 172.21.15.12:0/1365441144 >> 172.21.15.12:0/1365441144 conn(0x7f5ad80a7120 msgr2=0x7f5ad80a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:22.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.938+0000 7f5add945640 1 -- 172.21.15.12:0/1365441144 shutdown_connections 2024-09-11T01:38:22.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.938+0000 7f5add945640 1 -- 172.21.15.12:0/1365441144 wait complete. 2024-09-11T01:38:22.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.938+0000 7f5add945640 10 monclient: build_initial_monmap 2024-09-11T01:38:22.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.938+0000 7f5add945640 10 monclient: monmap: 2024-09-11T01:38:22.951 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:22.951 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:38:22.951 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:38:22.952 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:38:22.952 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:22.952 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:22.952 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:38:22.952 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:38:22.952 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:38:22.952 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:22.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5add945640 1 Processor -- start 2024-09-11T01:38:22.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5add945640 1 -- start start 2024-09-11T01:38:22.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5add945640 10 monclient: init 2024-09-11T01:38:22.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5add945640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:22.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5add945640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:38:22.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5add945640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ad809d030 0x7f5ad813f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5add945640 10 monclient(hunting): picked mon.noname-b con 0x7f5ad809d030 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:38:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5add945640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ad809d460 0x7f5ad81cd890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5add945640 10 monclient(hunting): picked mon.noname-a con 0x7f5ad809d460 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:38:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5add945640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ad8138230 0x7f5ad8137470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5add945640 10 monclient(hunting): picked mon.noname-c con 0x7f5ad8138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5add945640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5add945640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad813fca0 con 0x7f5ad809d460 2024-09-11T01:38:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5add945640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5add945640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad813fb20 con 0x7f5ad809d030 2024-09-11T01:38:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5add945640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5add945640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5ad813fe20 con 0x7f5ad8138230 2024-09-11T01:38:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5add945640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:22.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5add945640 10 monclient(hunting): authenticate will time out at 1309.381782s 2024-09-11T01:38:22.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5ad77fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ad8138230 0x7f5ad8137470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:22.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5ad67fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ad809d460 0x7f5ad81cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:22.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5ad77fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ad8138230 0x7f5ad8137470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45038/0 (socket says 172.21.15.12:45038) 2024-09-11T01:38:22.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5ad77fe640 1 -- 172.21.15.12:0/464704570 learned_addr learned my addr 172.21.15.12:0/464704570 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:38:22.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5ad67fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ad809d460 0x7f5ad81cd890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39162/0 (socket says 172.21.15.12:39162) 2024-09-11T01:38:22.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5ad77fe640 10 monclient(hunting): get_auth_request con 0x7f5ad8138230 auth_method 0 2024-09-11T01:38:22.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5ad77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:22.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5ad77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:22.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5ad77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:22.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5ad67fc640 10 monclient(hunting): get_auth_request con 0x7f5ad809d460 auth_method 0 2024-09-11T01:38:22.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5ad67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:22.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5ad67fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:22.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.939+0000 7f5ad67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:22.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.940+0000 7f5ad77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:22.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.940+0000 7f5ad77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:22.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.940+0000 7f5ad77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:22.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.940+0000 7f5ad67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:22.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.940+0000 7f5ad67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:22.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.940+0000 7f5ad6ffd640 1 --2- 172.21.15.12:0/464704570 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ad809d030 0x7f5ad813f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:22.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.940+0000 7f5ad67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:22.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.940+0000 7f5ad6ffd640 10 monclient(hunting): get_auth_request con 0x7f5ad809d030 auth_method 0 2024-09-11T01:38:22.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.940+0000 7f5ad6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:22.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.940+0000 7f5ad6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:22.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.940+0000 7f5ad6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:22.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.940+0000 7f5ad6ffd640 1 -- 172.21.15.12:0/464704570 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ad809d030 msgr2=0x7f5ad813f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:38:22.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.940+0000 7f5ad6ffd640 1 -- 172.21.15.12:0/464704570 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ad809d030 msgr2=0x7f5ad813f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:38:22.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.940+0000 7f5ad6ffd640 1 --2- 172.21.15.12:0/464704570 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ad809d030 0x7f5ad813f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:38:22.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.940+0000 7f5ad77fe640 10 monclient(hunting): handle_auth_done global_id 84296 payload 824 2024-09-11T01:38:22.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.940+0000 7f5ad6ffd640 1 --2- 172.21.15.12:0/464704570 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ad809d030 0x7f5ad813f4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:38:22.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.940+0000 7f5ad77fe640 1 -- 172.21.15.12:0/464704570 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ad809d030 msgr2=0x7f5ad813f4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:38:22.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.940+0000 7f5ad77fe640 1 --2- 172.21.15.12:0/464704570 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ad809d030 0x7f5ad813f4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:22.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.940+0000 7f5ad77fe640 1 -- 172.21.15.12:0/464704570 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ad809d460 msgr2=0x7f5ad81cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:22.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.940+0000 7f5ad77fe640 1 --2- 172.21.15.12:0/464704570 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ad809d460 0x7f5ad81cd890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:22.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.940+0000 7f5ad77fe640 10 monclient: _finish_hunting 0 2024-09-11T01:38:22.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.940+0000 7f5ad77fe640 1 monclient: found mon.noname-c 2024-09-11T01:38:22.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.940+0000 7f5ad77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:22.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.940+0000 7f5ad77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:38:22.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.940+0000 7f5ad77fe640 1 -- 172.21.15.12:0/464704570 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad80a1810 con 0x7f5ad8138230 2024-09-11T01:38:22.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.940+0000 7f5ad67fc640 1 --2- 172.21.15.12:0/464704570 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ad809d460 0x7f5ad81cd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:38:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.942+0000 7f5ad77fe640 1 -- 172.21.15.12:0/464704570 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ad8138230 msgr2=0x7f5ad8137470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.942+0000 7f5ad77fe640 1 --2- 172.21.15.12:0/464704570 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ad8138230 0x7f5ad8137470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5ad00070b0 tx=0x7f5ad000e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.942+0000 7f5abffff640 1 -- 172.21.15.12:0/464704570 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0014620 con 0x7f5ad8138230 2024-09-11T01:38:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.942+0000 7f5abffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.942+0000 7f5abffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:38:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.942+0000 7f5abffff640 10 monclient: dump: 2024-09-11T01:38:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:22.957 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:22.958 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:22.958 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:22.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.942+0000 7f5abffff640 10 monclient: _finish_auth 0 2024-09-11T01:38:22.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.942+0000 7f5abffff640 10 monclient: _check_auth_tickets 2024-09-11T01:38:22.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.942+0000 7f5abffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:22.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.942+0000 7f5abffff640 1 -- 172.21.15.12:0/464704570 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ad0019070 con 0x7f5ad8138230 2024-09-11T01:38:22.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.942+0000 7f5add945640 5 monclient: authenticate success, global_id 84296 2024-09-11T01:38:22.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.942+0000 7f5add945640 10 monclient: _renew_subs 2024-09-11T01:38:22.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.942+0000 7f5add945640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:38:22.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.942+0000 7f5add945640 1 -- 172.21.15.12:0/464704570 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ad809e060 con 0x7f5ad8138230 2024-09-11T01:38:22.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.942+0000 7f5abffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:22.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.942+0000 7f5abffff640 1 -- 172.21.15.12:0/464704570 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0014a20 con 0x7f5ad8138230 2024-09-11T01:38:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.942+0000 7f5abffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.942+0000 7f5abffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:38:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.942+0000 7f5abffff640 10 monclient: dump: 2024-09-11T01:38:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:22.959 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:22.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.942+0000 7f5add945640 10 monclient: _renew_subs 2024-09-11T01:38:22.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.942+0000 7f5ad57fa640 4 set_mon_vals no callback set 2024-09-11T01:38:22.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.942+0000 7f5add945640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:38:22.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.942+0000 7f5add945640 1 -- 172.21.15.12:0/464704570 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ad813e950 con 0x7f5ad8138230 2024-09-11T01:38:22.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.944+0000 7f5abffff640 1 -- 172.21.15.12:0/464704570 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f5ad0014c00 con 0x7f5ad8138230 2024-09-11T01:38:22.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.944+0000 7f5add945640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:22.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.944+0000 7f5add945640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:22.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.944+0000 7f5add945640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:38:22.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.944+0000 7f5add945640 1 -- 172.21.15.12:0/464704570 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ad8138610 con 0x7f5ad8138230 2024-09-11T01:38:22.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.944+0000 7f5abffff640 1 --2- 172.21.15.12:0/464704570 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5ac403e290 0x7f5ac4040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:22.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.944+0000 7f5abffff640 1 -- 172.21.15.12:0/464704570 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(312..312 src has 186..312) ==== 4771+0+0 (secure 0 0 0) 0x7f5ad00534a0 con 0x7f5ad8138230 2024-09-11T01:38:22.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.944+0000 7f5ad6ffd640 1 --2- 172.21.15.12:0/464704570 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5ac403e290 0x7f5ac4040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:22.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.944+0000 7f5ad6ffd640 10 monclient: get_auth_request con 0x7f5ac403e290 auth_method 0 2024-09-11T01:38:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.947+0000 7f5ad6ffd640 1 --2- 172.21.15.12:0/464704570 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5ac403e290 0x7f5ac4040750 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8007c50 tx=0x7f5ac8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.949+0000 7f5abffff640 1 -- 172.21.15.12:0/464704570 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5ad0051ea0 con 0x7f5ad8138230 2024-09-11T01:38:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.949+0000 7f5abffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:22.949+0000 7f5abffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:38:22.991 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1020309 2024-09-11T01:38:23.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:23.205+0000 7f5add945640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:38:23.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:23.205+0000 7f5add945640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:38:23.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:23.205+0000 7f5add945640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:38:23.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:23.205+0000 7f5add945640 1 -- 172.21.15.12:0/464704570 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5ad80a15c0 con 0x7f5ad8138230 2024-09-11T01:38:23.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:23.206+0000 7f5abffff640 1 -- 172.21.15.12:0/464704570 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f5ad00531d0 con 0x7f5ad8138230 2024-09-11T01:38:23.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:23.206+0000 7f5abffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:38:23.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:23.206+0000 7f5abffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:38:23.208 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":90,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:38:23.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:23.208+0000 7f5add945640 1 -- 172.21.15.12:0/464704570 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5ac403e290 msgr2=0x7f5ac4040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:23.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:23.208+0000 7f5add945640 1 --2- 172.21.15.12:0/464704570 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5ac403e290 0x7f5ac4040750 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8007c50 tx=0x7f5ac8009040 comp rx=0 tx=0).stop 2024-09-11T01:38:23.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:23.208+0000 7f5add945640 10 monclient: shutdown 2024-09-11T01:38:23.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:23.208+0000 7f5add945640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:23.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:23.208+0000 7f5add945640 1 -- 172.21.15.12:0/464704570 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ad8138230 msgr2=0x7f5ad8137470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:23.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:23.208+0000 7f5add945640 1 --2- 172.21.15.12:0/464704570 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ad8138230 0x7f5ad8137470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5ad00070b0 tx=0x7f5ad000e4b0 comp rx=0 tx=0).stop 2024-09-11T01:38:23.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:23.208+0000 7f5add945640 1 -- 172.21.15.12:0/464704570 shutdown_connections 2024-09-11T01:38:23.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:23.208+0000 7f5add945640 1 --2- 172.21.15.12:0/464704570 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ad8138230 0x7f5ad8137470 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:23.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:23.208+0000 7f5add945640 1 --2- 172.21.15.12:0/464704570 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5ac403e290 0x7f5ac4040750 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:23.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:23.208+0000 7f5add945640 1 --2- 172.21.15.12:0/464704570 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ad809d460 0x7f5ad81cd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:23.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:23.208+0000 7f5add945640 1 --2- 172.21.15.12:0/464704570 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ad809d030 0x7f5ad813f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:23.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:23.208+0000 7f5add945640 1 -- 172.21.15.12:0/464704570 >> 172.21.15.12:0/464704570 conn(0x7f5ad80a7120 msgr2=0x7f5ad80a3140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:23.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:23.208+0000 7f5add945640 1 -- 172.21.15.12:0/464704570 shutdown_connections 2024-09-11T01:38:23.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:23.208+0000 7f5add945640 1 -- 172.21.15.12:0/464704570 wait complete. 2024-09-11T01:38:23.923 INFO:tasks.rados.rados.0.smithi071.stdout:1390: done (1 left) 2024-09-11T01:38:23.923 INFO:tasks.rados.rados.0.smithi071.stdout:1391: done (0 left) 2024-09-11T01:38:23.923 INFO:tasks.rados.rados.0.smithi071.stdout:1392: snap_create 2024-09-11T01:38:25.322 INFO:tasks.rados.rados.0.smithi071.stdout:1392: done (0 left) 2024-09-11T01:38:25.323 INFO:tasks.rados.rados.0.smithi071.stdout:1393: write oid 27 current snap is 148 2024-09-11T01:38:25.323 INFO:tasks.rados.rados.0.smithi071.stdout:1393: seq_num 527 ranges {457289=484580,1438834=548839,2772623=84666} 2024-09-11T01:38:25.327 INFO:tasks.rados.rados.0.smithi071.stdout:1393: writing smithi07134777-27 from 457289 to 941869 tid 1 2024-09-11T01:38:25.333 INFO:tasks.rados.rados.0.smithi071.stdout:1393: writing smithi07134777-27 from 1438834 to 1987673 tid 2 2024-09-11T01:38:25.339 INFO:tasks.rados.rados.0.smithi071.stdout:1393: writing smithi07134777-27 from 2772623 to 2857289 tid 3 2024-09-11T01:38:25.339 INFO:tasks.rados.rados.0.smithi071.stdout:1394: read oid 46 snap 143 2024-09-11T01:38:25.339 INFO:tasks.rados.rados.0.smithi071.stdout:1394: expect (ObjNum 478 snap 126 seq_num 478) 2024-09-11T01:38:25.339 INFO:tasks.rados.rados.0.smithi071.stdout:1395: rollback oid 39 current snap is 148 2024-09-11T01:38:25.339 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 39 to 146 2024-09-11T01:38:25.339 INFO:tasks.rados.rados.0.smithi071.stdout:1393: finishing write tid 1 to smithi07134777-27 2024-09-11T01:38:25.339 INFO:tasks.rados.rados.0.smithi071.stdout:1393: finishing write tid 2 to smithi07134777-27 2024-09-11T01:38:25.339 INFO:tasks.rados.rados.0.smithi071.stdout:1396: read oid 9 snap -1 2024-09-11T01:38:25.339 INFO:tasks.rados.rados.0.smithi071.stdout:1396: expect (ObjNum 475 snap 125 seq_num 475) 2024-09-11T01:38:25.339 INFO:tasks.rados.rados.0.smithi071.stdout:1397: copy_from oid 37 from oid 10 current snap is 148 2024-09-11T01:38:25.339 INFO:tasks.rados.rados.0.smithi071.stdout:1398: delete oid 26 current snap is 148 2024-09-11T01:38:25.340 INFO:tasks.rados.rados.0.smithi071.stdout:1394: expect (ObjNum 478 snap 126 seq_num 478) 2024-09-11T01:38:25.393 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1990614 2024-09-11T01:38:25.416 INFO:tasks.rados.rados.0.smithi071.stdout:1395: finishing rollback tid 0 to smithi07134777-39 2024-09-11T01:38:25.416 INFO:tasks.rados.rados.0.smithi071.stdout:1395: finishing rollback tid 1 to smithi07134777-39 2024-09-11T01:38:25.416 INFO:tasks.rados.rados.0.smithi071.stdout:1396: expect (ObjNum 475 snap 125 seq_num 475) 2024-09-11T01:38:25.545 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 39316 2024-09-11T01:38:25.618 INFO:tasks.rados.rados.0.smithi071.stdout:1393: finishing write tid 3 to smithi07134777-27 2024-09-11T01:38:25.618 INFO:tasks.rados.rados.0.smithi071.stdout:1393: finishing write tid 4 to smithi07134777-27 2024-09-11T01:38:25.618 INFO:tasks.rados.rados.0.smithi071.stdout:1393: finishing write tid 5 to smithi07134777-27 2024-09-11T01:38:25.618 INFO:tasks.rados.rados.0.smithi071.stdout:1393: oid 27 updating version 0 to 644 2024-09-11T01:38:25.619 INFO:tasks.rados.rados.0.smithi071.stdout:1393: oid 27 version 644 is already newer than 641 2024-09-11T01:38:25.619 INFO:tasks.rados.rados.0.smithi071.stdout:1393: oid 27 version 644 is already newer than 643 2024-09-11T01:38:25.619 INFO:tasks.rados.rados.0.smithi071.stdout:1393: oid 27 version 644 is already newer than 642 2024-09-11T01:38:25.619 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 644 (ObjNum 527 snap 148 seq_num 527) dirty exists 2024-09-11T01:38:25.619 INFO:tasks.rados.rados.0.smithi071.stdout:1393: left oid 27 (ObjNum 527 snap 148 seq_num 527) 2024-09-11T01:38:25.619 INFO:tasks.rados.rados.0.smithi071.stdout:1395: finishing rollback tid 2 to smithi07134777-39 2024-09-11T01:38:25.619 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 581 (ObjNum 510 snap 139 seq_num 510) dirty exists 2024-09-11T01:38:25.619 INFO:tasks.rados.rados.0.smithi071.stdout:1397: finishing copy_from racing read to smithi07134777-37 2024-09-11T01:38:25.619 INFO:tasks.rados.rados.0.smithi071.stdout:1393: done (5 left) 2024-09-11T01:38:25.619 INFO:tasks.rados.rados.0.smithi071.stdout:1394: done (4 left) 2024-09-11T01:38:25.619 INFO:tasks.rados.rados.0.smithi071.stdout:1395: done (3 left) 2024-09-11T01:38:25.619 INFO:tasks.rados.rados.0.smithi071.stdout:1396: done (2 left) 2024-09-11T01:38:25.619 INFO:tasks.rados.rados.0.smithi071.stdout:1398: done (1 left) 2024-09-11T01:38:25.619 INFO:tasks.rados.rados.0.smithi071.stdout:1399: delete oid 7 current snap is 148 2024-09-11T01:38:25.619 INFO:tasks.rados.rados.0.smithi071.stdout:1397: finishing copy_from to smithi07134777-37 2024-09-11T01:38:25.620 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 360 (ObjNum 520 snap 143 seq_num 520) dirty exists 2024-09-11T01:38:25.626 INFO:tasks.rados.rados.0.smithi071.stdout:1397: done (1 left) 2024-09-11T01:38:25.626 INFO:tasks.rados.rados.0.smithi071.stdout:1399: done (0 left) 2024-09-11T01:38:25.626 INFO:tasks.rados.rados.0.smithi071.stdout:1400: snap_remove snap 140 2024-09-11T01:38:26.221 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:38:26.334 INFO:tasks.rados.rados.0.smithi071.stdout:1400: done (0 left) 2024-09-11T01:38:26.334 INFO:tasks.rados.rados.0.smithi071.stdout:1401: snap_remove snap 141 2024-09-11T01:38:26.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.343+0000 7efe0d671640 10 monclient: get_monmap_and_config 2024-09-11T01:38:26.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.343+0000 7efe0d671640 10 monclient: build_initial_monmap 2024-09-11T01:38:26.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.343+0000 7efe0d671640 10 monclient: monmap: 2024-09-11T01:38:26.345 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:26.345 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:26.345 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:38:26.343996+0000 2024-09-11T01:38:26.345 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:38:26.343996+0000 2024-09-11T01:38:26.345 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:26.345 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:26.345 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:38:26.345 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:38:26.345 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:38:26.345 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:38:26.345 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:38:26.346 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:26.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.344+0000 7efe0d671640 1 Processor -- start 2024-09-11T01:38:26.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.344+0000 7efe0d671640 1 -- start start 2024-09-11T01:38:26.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.344+0000 7efe0d671640 10 monclient: init 2024-09-11T01:38:26.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.344+0000 7efe0d671640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:26.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.344+0000 7efe0d671640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-11T01:38:26.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.344+0000 7efe0d671640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efe080a2e90 addr v1:172.21.15.71:6789/0 2024-09-11T01:38:26.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.344+0000 7efe0d671640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efe080a3270 addr v1:172.21.15.12:6789/0 2024-09-11T01:38:26.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.344+0000 7efe0d671640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7efe0813c0b0 0x7efe0813e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:26.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.344+0000 7efe0d671640 10 monclient(hunting): picked mon.noname-c con 0x7efe0813c0b0 addr v2:172.21.15.71:3300/0 2024-09-11T01:38:26.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.344+0000 7efe0d671640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe080a4870 con 0x7efe080a3270 2024-09-11T01:38:26.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.344+0000 7efe0d671640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe080a21b0 con 0x7efe080a2e90 2024-09-11T01:38:26.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.344+0000 7efe0d671640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:26.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.344+0000 7efe0d671640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7efe080a2970 con 0x7efe0813c0b0 2024-09-11T01:38:26.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.344+0000 7efe0d671640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:26.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.344+0000 7efe0d671640 10 monclient(hunting): authenticate will time out at 1312.786882s 2024-09-11T01:38:26.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.345+0000 7efe067fc640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7efe080a3270 0x7efe080a3680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:51900/0 (socket says 172.21.15.12:51900) 2024-09-11T01:38:26.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.345+0000 7efe067fc640 1 -- 172.21.15.12:0/307514080 learned_addr learned my addr 172.21.15.12:0/307514080 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:38:26.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.345+0000 7efe077fe640 1 --2- 172.21.15.12:0/307514080 >> v2:172.21.15.71:3300/0 conn(0x7efe0813c0b0 0x7efe0813e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:26.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.345+0000 7efe077fe640 10 monclient(hunting): get_auth_request con 0x7efe0813c0b0 auth_method 0 2024-09-11T01:38:26.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.345+0000 7efe077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:26.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.345+0000 7efe077fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:26.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.345+0000 7efe077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:26.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe067fc640 1 -- 172.21.15.12:0/307514080 >> v1:172.21.15.12:6789/0 conn(0x7efe080a3270 legacy=0x7efe080a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:26.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe06ffd640 1 -- 172.21.15.12:0/307514080 >> v1:172.21.15.71:6789/0 conn(0x7efe080a2e90 legacy=0x7efe080a1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:26.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe077fe640 1 -- 172.21.15.12:0/307514080 >> v2:172.21.15.71:3300/0 conn(0x7efe0813c0b0 msgr2=0x7efe0813e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:38:26.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe077fe640 1 -- 172.21.15.12:0/307514080 >> v2:172.21.15.71:3300/0 conn(0x7efe0813c0b0 msgr2=0x7efe0813e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:38:26.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe077fe640 1 --2- 172.21.15.12:0/307514080 >> v2:172.21.15.71:3300/0 conn(0x7efe0813c0b0 0x7efe0813e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:38:26.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 1 -- 172.21.15.12:0/307514080 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2331668804 0 0) 0x7efe080a4870 con 0x7efe080a3270 2024-09-11T01:38:26.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 10 monclient(hunting): _negotiate 2024-09-11T01:38:26.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:26.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:26.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 10 monclient(hunting): my global_id is 94203 2024-09-11T01:38:26.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe077fe640 1 --2- 172.21.15.12:0/307514080 >> v2:172.21.15.71:3300/0 conn(0x7efe0813c0b0 0x7efe0813e4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:38:26.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 1 -- 172.21.15.12:0/307514080 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efdf0003520 con 0x7efe080a3270 2024-09-11T01:38:26.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 1 -- 172.21.15.12:0/307514080 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1326227679 0 0) 0x7efdf0003520 con 0x7efe080a3270 2024-09-11T01:38:26.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 1 -- 172.21.15.12:0/307514080 >> v2:172.21.15.71:3300/0 conn(0x7efe0813c0b0 msgr2=0x7efe0813e4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:38:26.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 1 --2- 172.21.15.12:0/307514080 >> v2:172.21.15.71:3300/0 conn(0x7efe0813c0b0 0x7efe0813e4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 1 -- 172.21.15.12:0/307514080 >> v1:172.21.15.71:6789/0 conn(0x7efe080a2e90 legacy=0x7efe080a1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:26.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 10 monclient: _finish_hunting 0 2024-09-11T01:38:26.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:38:26.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:26.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:38:26.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 1 -- 172.21.15.12:0/307514080 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe080a0a70 con 0x7efe080a3270 2024-09-11T01:38:26.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 10 monclient: _finish_auth 0 2024-09-11T01:38:26.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:38:26.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:26.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 1 -- 172.21.15.12:0/307514080 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7efdfc0019a0 con 0x7efe080a3270 2024-09-11T01:38:26.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe0d671640 5 monclient: authenticate success, global_id 94203 2024-09-11T01:38:26.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe0d671640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:38:26.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:26.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:38:26.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 10 monclient: dump: 2024-09-11T01:38:26.355 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:26.355 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:26.355 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.355 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.355 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:26.355 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:26.356 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:26.356 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:26.356 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:26.356 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:26.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:38:26.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:26.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 1 -- 172.21.15.12:0/307514080 >> v1:172.21.15.12:6789/0 conn(0x7efe080a3270 legacy=0x7efe080a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:26.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:38:26.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 1 --2- 172.21.15.12:0/307514080 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efdf0003b20 0x7efdf0003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:26.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 10 monclient(hunting): picked mon.c con 0x7efdf0003b20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:26.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 1 --2- 172.21.15.12:0/307514080 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efdf0004800 0x7efdf0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:26.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 10 monclient(hunting): picked mon.b con 0x7efdf0004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:38:26.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 1 --2- 172.21.15.12:0/307514080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efdf00099b0 0x7efdf000bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:26.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 10 monclient(hunting): picked mon.a con 0x7efdf00099b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:38:26.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:26.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 1 -- 172.21.15.12:0/307514080 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7efe080a2970 con 0x7efdf00099b0 2024-09-11T01:38:26.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:26.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 1 -- 172.21.15.12:0/307514080 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7efdf000c3c0 con 0x7efdf0004800 2024-09-11T01:38:26.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:26.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 1 -- 172.21.15.12:0/307514080 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7efdf000c5c0 con 0x7efdf0003b20 2024-09-11T01:38:26.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:26.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe05ffb640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:38:26.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe0d671640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:38:26.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe06ffd640 1 --2- 172.21.15.12:0/307514080 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efdf0003b20 0x7efdf0003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:26.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe067fc640 1 --2- 172.21.15.12:0/307514080 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efdf0004800 0x7efdf0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:26.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe06ffd640 10 monclient(hunting): get_auth_request con 0x7efdf0003b20 auth_method 0 2024-09-11T01:38:26.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:26.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:26.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe06ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:38:26.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe067fc640 10 monclient(hunting): get_auth_request con 0x7efdf0004800 auth_method 0 2024-09-11T01:38:26.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:26.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe067fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:26.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe067fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:38:26.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe077fe640 1 --2- 172.21.15.12:0/307514080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efdf00099b0 0x7efdf000bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:26.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe077fe640 10 monclient(hunting): get_auth_request con 0x7efdf00099b0 auth_method 0 2024-09-11T01:38:26.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:26.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe077fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:26.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.347+0000 7efe077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:38:26.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.348+0000 7efe06ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:26.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.348+0000 7efe06ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:26.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.348+0000 7efe06ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:38:26.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.348+0000 7efe067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:26.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.348+0000 7efe067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:26.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.348+0000 7efe067fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:38:26.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.348+0000 7efe077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:26.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.348+0000 7efe077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:26.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.348+0000 7efe077fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:38:26.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.348+0000 7efe06ffd640 10 monclient(hunting): handle_auth_done global_id 94203 payload 293 2024-09-11T01:38:26.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.348+0000 7efe06ffd640 1 -- 172.21.15.12:0/307514080 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efdf0004800 msgr2=0x7efdf0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:38:26.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.348+0000 7efe06ffd640 1 --2- 172.21.15.12:0/307514080 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efdf0004800 0x7efdf0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.348+0000 7efe06ffd640 1 -- 172.21.15.12:0/307514080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efdf00099b0 msgr2=0x7efdf000bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:38:26.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.348+0000 7efe06ffd640 1 --2- 172.21.15.12:0/307514080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efdf00099b0 0x7efdf000bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.348+0000 7efe06ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:38:26.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.348+0000 7efe06ffd640 1 monclient: found mon.c 2024-09-11T01:38:26.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.348+0000 7efe06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:26.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.348+0000 7efe06ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:38:26.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.348+0000 7efe06ffd640 1 -- 172.21.15.12:0/307514080 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdf0003520 con 0x7efdf0003b20 2024-09-11T01:38:26.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.348+0000 7efe06ffd640 10 monclient: _finish_auth 0 2024-09-11T01:38:26.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.348+0000 7efe06ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:38:26.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.348+0000 7efe06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:26.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.348+0000 7efe077fe640 1 --2- 172.21.15.12:0/307514080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efdf00099b0 0x7efdf000bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:38:26.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.348+0000 7efe077fe640 1 -- 172.21.15.12:0/307514080 reap_dead start 2024-09-11T01:38:26.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.351+0000 7efe06ffd640 1 -- 172.21.15.12:0/307514080 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efdf0003b20 msgr2=0x7efdf0003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:26.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.351+0000 7efe06ffd640 1 --2- 172.21.15.12:0/307514080 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efdf0003b20 0x7efdf0003f40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efdf8001990 tx=0x7efdf0005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c382b24a6506c06c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:26.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.351+0000 7efe05ffb640 1 -- 172.21.15.12:0/307514080 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf8001690 con 0x7efdf0003b20 2024-09-11T01:38:26.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.351+0000 7efe05ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:26.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.351+0000 7efe05ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:38:26.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.351+0000 7efe05ffb640 10 monclient: dump: 2024-09-11T01:38:26.361 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:26.361 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:26.361 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.361 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.361 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:26.361 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:26.361 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:26.361 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:26.362 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:26.362 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:26.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.351+0000 7efe05ffb640 1 -- 172.21.15.12:0/307514080 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdf802e070 con 0x7efdf0003b20 2024-09-11T01:38:26.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.351+0000 7efe0d671640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:38:26.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.351+0000 7efe05ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:26.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.351+0000 7efe0d671640 10 monclient: get_monmap_and_config success 2024-09-11T01:38:26.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.351+0000 7efe05ffb640 1 -- 172.21.15.12:0/307514080 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf8007070 con 0x7efdf0003b20 2024-09-11T01:38:26.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.351+0000 7efe05ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:26.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.351+0000 7efe0d671640 4 set_mon_vals no callback set 2024-09-11T01:38:26.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.351+0000 7efe05ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:38:26.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.351+0000 7efe05ffb640 10 monclient: dump: 2024-09-11T01:38:26.362 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:26.362 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:26.362 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.363 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.363 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:26.363 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:26.363 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:26.363 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:26.363 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:26.363 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:26.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.352+0000 7efe0d671640 10 monclient: shutdown 2024-09-11T01:38:26.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.352+0000 7efe0d671640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:26.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.352+0000 7efe0d671640 1 -- 172.21.15.12:0/307514080 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efdf0003b20 msgr2=0x7efdf0003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:26.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.352+0000 7efe0d671640 1 --2- 172.21.15.12:0/307514080 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efdf0003b20 0x7efdf0003f40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efdf8001990 tx=0x7efdf0005830 comp rx=0 tx=0).stop 2024-09-11T01:38:26.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.352+0000 7efe0d671640 1 -- 172.21.15.12:0/307514080 shutdown_connections 2024-09-11T01:38:26.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.352+0000 7efe0d671640 1 --2- 172.21.15.12:0/307514080 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efdf0003b20 0x7efdf0003f40 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.352+0000 7efe0d671640 1 -- 172.21.15.12:0/307514080 >> 172.21.15.12:0/307514080 conn(0x7efe080a8670 msgr2=0x7efe080a8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:26.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.352+0000 7efe0d671640 1 -- 172.21.15.12:0/307514080 shutdown_connections 2024-09-11T01:38:26.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.352+0000 7efe0d671640 1 -- 172.21.15.12:0/307514080 wait complete. 2024-09-11T01:38:26.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.352+0000 7efe0d671640 10 monclient: build_initial_monmap 2024-09-11T01:38:26.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.352+0000 7efe0d671640 10 monclient: monmap: 2024-09-11T01:38:26.371 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:26.371 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:38:26.371 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:38:26.371 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:38:26.371 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:26.371 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:26.371 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:38:26.371 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:38:26.371 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:38:26.371 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:26.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.353+0000 7efe0d671640 1 Processor -- start 2024-09-11T01:38:26.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.353+0000 7efe0d671640 1 -- start start 2024-09-11T01:38:26.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.353+0000 7efe0d671640 10 monclient: init 2024-09-11T01:38:26.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe0d671640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:26.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe0d671640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:38:26.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe0d671640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efdf0003b20 0x7efe0806dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:26.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe0d671640 10 monclient(hunting): picked mon.noname-a con 0x7efdf0003b20 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:38:26.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe0d671640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efe0806e1a0 0x7efe0806e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:26.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe0d671640 10 monclient(hunting): picked mon.noname-b con 0x7efe0806e1a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:38:26.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe0d671640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efe081cd260 0x7efe081cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:26.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe0d671640 10 monclient(hunting): picked mon.noname-c con 0x7efe081cd260 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:26.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe0d671640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:26.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe0d671640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7efe080a1c20 con 0x7efdf0003b20 2024-09-11T01:38:26.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe0d671640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:26.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe0d671640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7efe080a1aa0 con 0x7efe0806e1a0 2024-09-11T01:38:26.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe0d671640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:26.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe0d671640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7efe080a1da0 con 0x7efe081cd260 2024-09-11T01:38:26.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe0d671640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:26.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe0d671640 10 monclient(hunting): authenticate will time out at 1312.796130s 2024-09-11T01:38:26.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe06ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efdf0003b20 0x7efe0806dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:26.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe06ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efdf0003b20 0x7efe0806dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57694/0 (socket says 172.21.15.12:57694) 2024-09-11T01:38:26.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe077fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efe081cd260 0x7efe081cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:26.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe06ffd640 1 -- 172.21.15.12:0/4266402722 learned_addr learned my addr 172.21.15.12:0/4266402722 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:38:26.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe06ffd640 10 monclient(hunting): get_auth_request con 0x7efdf0003b20 auth_method 0 2024-09-11T01:38:26.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:26.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:26.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:26.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe077fe640 10 monclient(hunting): get_auth_request con 0x7efe081cd260 auth_method 0 2024-09-11T01:38:26.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:26.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe077fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:26.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:26.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe06ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:26.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe06ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:26.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe067fc640 1 --2- 172.21.15.12:0/4266402722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efe0806e1a0 0x7efe0806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:26.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe06ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:26.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:26.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:26.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:26.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe067fc640 10 monclient(hunting): get_auth_request con 0x7efe0806e1a0 auth_method 0 2024-09-11T01:38:26.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:26.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe067fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe067fc640 1 -- 172.21.15.12:0/4266402722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efe0806e1a0 msgr2=0x7efe0806e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:38:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe067fc640 1 -- 172.21.15.12:0/4266402722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efe0806e1a0 msgr2=0x7efe0806e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:38:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe067fc640 1 --2- 172.21.15.12:0/4266402722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efe0806e1a0 0x7efe0806e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:38:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe067fc640 1 --2- 172.21.15.12:0/4266402722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efe0806e1a0 0x7efe0806e620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:38:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe06ffd640 10 monclient(hunting): handle_auth_done global_id 94206 payload 824 2024-09-11T01:38:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe06ffd640 1 -- 172.21.15.12:0/4266402722 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efe081cd260 msgr2=0x7efe081cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe06ffd640 1 --2- 172.21.15.12:0/4266402722 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efe081cd260 0x7efe081cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe06ffd640 1 -- 172.21.15.12:0/4266402722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efe0806e1a0 msgr2=0x7efe0806e620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:38:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe06ffd640 1 --2- 172.21.15.12:0/4266402722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efe0806e1a0 0x7efe0806e620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe06ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:38:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe06ffd640 1 monclient: found mon.noname-a 2024-09-11T01:38:26.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:26.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe06ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:38:26.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe06ffd640 1 -- 172.21.15.12:0/4266402722 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe08123a50 con 0x7efdf0003b20 2024-09-11T01:38:26.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.354+0000 7efe077fe640 1 --2- 172.21.15.12:0/4266402722 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efe081cd260 0x7efe081cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:38:26.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.356+0000 7efe06ffd640 1 -- 172.21.15.12:0/4266402722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efdf0003b20 msgr2=0x7efe0806dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:26.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.356+0000 7efe06ffd640 1 --2- 172.21.15.12:0/4266402722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efdf0003b20 0x7efe0806dc60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efdf8001ab0 tx=0x7efdf8001520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:26.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.356+0000 7efdef7fe640 1 -- 172.21.15.12:0/4266402722 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf8001690 con 0x7efdf0003b20 2024-09-11T01:38:26.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.357+0000 7efdef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:26.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.357+0000 7efdef7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:38:26.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.357+0000 7efdef7fe640 10 monclient: dump: 2024-09-11T01:38:26.377 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:26.377 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:26.377 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.377 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.377 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:26.377 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:26.377 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:26.377 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:26.377 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:26.377 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:26.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.357+0000 7efdef7fe640 10 monclient: _finish_auth 0 2024-09-11T01:38:26.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.357+0000 7efdef7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:38:26.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.357+0000 7efdef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:26.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.357+0000 7efdef7fe640 1 -- 172.21.15.12:0/4266402722 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdf80059c0 con 0x7efdf0003b20 2024-09-11T01:38:26.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.357+0000 7efdef7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:26.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.357+0000 7efdef7fe640 1 -- 172.21.15.12:0/4266402722 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf8005d90 con 0x7efdf0003b20 2024-09-11T01:38:26.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.357+0000 7efdef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:26.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.357+0000 7efdef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:38:26.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.357+0000 7efdef7fe640 10 monclient: dump: 2024-09-11T01:38:26.378 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:26.378 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:26.378 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.378 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.378 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:26.379 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:26.379 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:26.379 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:26.379 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:26.379 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:26.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.357+0000 7efe0d671640 5 monclient: authenticate success, global_id 94206 2024-09-11T01:38:26.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.357+0000 7efe0d671640 10 monclient: _renew_subs 2024-09-11T01:38:26.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.357+0000 7efe0d671640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:38:26.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.357+0000 7efe0d671640 1 -- 172.21.15.12:0/4266402722 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe08123d30 con 0x7efdf0003b20 2024-09-11T01:38:26.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.357+0000 7efe057fa640 4 set_mon_vals no callback set 2024-09-11T01:38:26.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.357+0000 7efe0d671640 10 monclient: _renew_subs 2024-09-11T01:38:26.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.357+0000 7efe0d671640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:38:26.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.357+0000 7efe0d671640 1 -- 172.21.15.12:0/4266402722 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe08074270 con 0x7efdf0003b20 2024-09-11T01:38:26.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.357+0000 7efdef7fe640 1 -- 172.21.15.12:0/4266402722 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7efdf8033470 con 0x7efdf0003b20 2024-09-11T01:38:26.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.358+0000 7efe0d671640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:26.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.358+0000 7efe0d671640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:26.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.358+0000 7efe0d671640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:38:26.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.358+0000 7efe0d671640 1 -- 172.21.15.12:0/4266402722 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe08072540 con 0x7efdf0003b20 2024-09-11T01:38:26.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.358+0000 7efdef7fe640 1 --2- 172.21.15.12:0/4266402722 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efdf403e210 0x7efdf40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:26.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.358+0000 7efdef7fe640 1 -- 172.21.15.12:0/4266402722 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(315..315 src has 186..315) ==== 4799+0+0 (secure 0 0 0) 0x7efdf8072960 con 0x7efdf0003b20 2024-09-11T01:38:26.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.358+0000 7efe067fc640 1 --2- 172.21.15.12:0/4266402722 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efdf403e210 0x7efdf40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:26.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.358+0000 7efe067fc640 10 monclient: get_auth_request con 0x7efdf403e210 auth_method 0 2024-09-11T01:38:26.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.360+0000 7efe067fc640 1 --2- 172.21.15.12:0/4266402722 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efdf403e210 0x7efdf40406d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7efdfc0010e0 tx=0x7efdfc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:26.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.362+0000 7efdef7fe640 1 -- 172.21.15.12:0/4266402722 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7efdf8071350 con 0x7efdf0003b20 2024-09-11T01:38:26.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.362+0000 7efdef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:26.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.362+0000 7efdef7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:38:26.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.625+0000 7efe0d671640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:38:26.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.625+0000 7efe0d671640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:38:26.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.625+0000 7efe0d671640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:38:26.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.625+0000 7efe0d671640 1 -- 172.21.15.12:0/4266402722 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7efe080a3680 con 0x7efdf0003b20 2024-09-11T01:38:26.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.626+0000 7efdef7fe640 1 -- 172.21.15.12:0/4266402722 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7efdf803f430 con 0x7efdf0003b20 2024-09-11T01:38:26.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.626+0000 7efdef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:38:26.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.626+0000 7efdef7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:38:26.628 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:38:26.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.628+0000 7efe0d671640 1 -- 172.21.15.12:0/4266402722 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efdf403e210 msgr2=0x7efdf40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:26.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.628+0000 7efe0d671640 1 --2- 172.21.15.12:0/4266402722 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efdf403e210 0x7efdf40406d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7efdfc0010e0 tx=0x7efdfc009040 comp rx=0 tx=0).stop 2024-09-11T01:38:26.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.628+0000 7efe0d671640 10 monclient: shutdown 2024-09-11T01:38:26.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.628+0000 7efe0d671640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:26.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.628+0000 7efe0d671640 1 -- 172.21.15.12:0/4266402722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efdf0003b20 msgr2=0x7efe0806dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:26.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.628+0000 7efe0d671640 1 --2- 172.21.15.12:0/4266402722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efdf0003b20 0x7efe0806dc60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efdf8001ab0 tx=0x7efdf8001520 comp rx=0 tx=0).stop 2024-09-11T01:38:26.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.628+0000 7efe0d671640 1 -- 172.21.15.12:0/4266402722 shutdown_connections 2024-09-11T01:38:26.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.628+0000 7efe0d671640 1 --2- 172.21.15.12:0/4266402722 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efe081cd260 0x7efe081cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.628+0000 7efe0d671640 1 --2- 172.21.15.12:0/4266402722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efe0806e1a0 0x7efe0806e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.628+0000 7efe0d671640 1 --2- 172.21.15.12:0/4266402722 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efdf403e210 0x7efdf40406d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.628+0000 7efe0d671640 1 --2- 172.21.15.12:0/4266402722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efdf0003b20 0x7efe0806dc60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.628+0000 7efe0d671640 1 -- 172.21.15.12:0/4266402722 >> 172.21.15.12:0/4266402722 conn(0x7efe080a8670 msgr2=0x7efe08074ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:26.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.628+0000 7efe0d671640 1 -- 172.21.15.12:0/4266402722 shutdown_connections 2024-09-11T01:38:26.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.628+0000 7efe0d671640 1 -- 172.21.15.12:0/4266402722 wait complete. 2024-09-11T01:38:26.640 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:38:26.640 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:38:26.640 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:38:26.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.740+0000 7f4e8caa9640 10 monclient: get_monmap_and_config 2024-09-11T01:38:26.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.740+0000 7f4e8caa9640 10 monclient: build_initial_monmap 2024-09-11T01:38:26.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.740+0000 7f4e8caa9640 10 monclient: monmap: 2024-09-11T01:38:26.742 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:26.742 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:26.742 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:38:26.741508+0000 2024-09-11T01:38:26.742 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:38:26.741508+0000 2024-09-11T01:38:26.742 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:26.742 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:26.743 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:38:26.743 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:38:26.743 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:38:26.743 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:38:26.743 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:38:26.743 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:26.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.741+0000 7f4e8caa9640 1 Processor -- start 2024-09-11T01:38:26.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.741+0000 7f4e8caa9640 1 -- start start 2024-09-11T01:38:26.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.741+0000 7f4e8caa9640 10 monclient: init 2024-09-11T01:38:26.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.741+0000 7f4e8caa9640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:26.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.741+0000 7f4e8caa9640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-11T01:38:26.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e8caa9640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4e880a03a0 0x7f4e8809f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:26.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e8caa9640 10 monclient(hunting): picked mon.noname-c con 0x7f4e880a03a0 addr v2:172.21.15.71:3300/0 2024-09-11T01:38:26.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e8caa9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4e880a2530 0x7f4e880a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:26.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e8caa9640 10 monclient(hunting): picked mon.noname-b con 0x7f4e880a2530 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:26.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e8caa9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4e880a11d0 0x7f4e880a1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:26.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e8caa9640 10 monclient(hunting): picked mon.noname-a con 0x7f4e880a11d0 addr v2:172.21.15.12:3300/0 2024-09-11T01:38:26.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e8caa9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:26.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e8caa9640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f4e880a29b0 con 0x7f4e880a11d0 2024-09-11T01:38:26.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e8caa9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:26.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e8caa9640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f4e880a0780 con 0x7f4e880a03a0 2024-09-11T01:38:26.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e8caa9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:26.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e8caa9640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4e8809e510 con 0x7f4e880a2530 2024-09-11T01:38:26.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e8caa9640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:26.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e8caa9640 10 monclient(hunting): authenticate will time out at 1313.184085s 2024-09-11T01:38:26.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e85d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4e880a2530 0x7f4e880a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:26.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e86d9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4e880a11d0 0x7f4e880a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:26.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e85d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4e880a2530 0x7f4e880a0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:50842/0 (socket says 172.21.15.12:50842) 2024-09-11T01:38:26.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e86d9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4e880a11d0 0x7f4e880a1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57708/0 (socket says 172.21.15.12:57708) 2024-09-11T01:38:26.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e85d9b640 1 -- 172.21.15.12:0/1854749414 learned_addr learned my addr 172.21.15.12:0/1854749414 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:38:26.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e85d9b640 10 monclient(hunting): get_auth_request con 0x7f4e880a2530 auth_method 0 2024-09-11T01:38:26.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e85d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:26.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e85d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:26.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e85d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:26.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e86d9d640 10 monclient(hunting): get_auth_request con 0x7f4e880a11d0 auth_method 0 2024-09-11T01:38:26.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e86d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:26.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e86d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:26.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e86d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:26.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e85d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:26.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e85d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:26.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e85d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:26.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e86d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:26.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e86d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:26.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e86d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:26.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e8659c640 1 --2- 172.21.15.12:0/1854749414 >> v2:172.21.15.71:3300/0 conn(0x7f4e880a03a0 0x7f4e8809f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:26.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e8659c640 10 monclient(hunting): get_auth_request con 0x7f4e880a03a0 auth_method 0 2024-09-11T01:38:26.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e8659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:26.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e8659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:26.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e8659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:26.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e85d9b640 10 monclient(hunting): handle_auth_done global_id 84302 payload 274 2024-09-11T01:38:26.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e85d9b640 1 -- 172.21.15.12:0/1854749414 >> v2:172.21.15.71:3300/0 conn(0x7f4e880a03a0 msgr2=0x7f4e8809f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:38:26.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e85d9b640 1 --2- 172.21.15.12:0/1854749414 >> v2:172.21.15.71:3300/0 conn(0x7f4e880a03a0 0x7f4e8809f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e85d9b640 1 -- 172.21.15.12:0/1854749414 >> v2:172.21.15.12:3300/0 conn(0x7f4e880a11d0 msgr2=0x7f4e880a1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:38:26.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e85d9b640 1 --2- 172.21.15.12:0/1854749414 >> v2:172.21.15.12:3300/0 conn(0x7f4e880a11d0 0x7f4e880a1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e85d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:38:26.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e85d9b640 1 monclient: found mon.noname-b 2024-09-11T01:38:26.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e85d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:26.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e85d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:38:26.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e85d9b640 1 -- 172.21.15.12:0/1854749414 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e8809edb0 con 0x7f4e880a2530 2024-09-11T01:38:26.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.742+0000 7f4e86d9d640 1 --2- 172.21.15.12:0/1854749414 >> v2:172.21.15.12:3300/0 conn(0x7f4e880a11d0 0x7f4e880a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:38:26.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.745+0000 7f4e85d9b640 1 -- 172.21.15.12:0/1854749414 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4e880a2530 msgr2=0x7f4e880a0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:26.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.745+0000 7f4e85d9b640 1 --2- 172.21.15.12:0/1854749414 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4e880a2530 0x7f4e880a0c90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4e8806cd50 tx=0x7f4e7002a0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7cf6ac22599e1fe server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:26.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.745+0000 7f4e8559a640 1 -- 172.21.15.12:0/1854749414 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e70037070 con 0x7f4e880a2530 2024-09-11T01:38:26.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.745+0000 7f4e8559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:26.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.745+0000 7f4e8559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:38:26.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.745+0000 7f4e8559a640 10 monclient: dump: 2024-09-11T01:38:26.752 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:26.753 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:26.753 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.753 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.753 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:26.753 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:26.753 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:26.753 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:26.753 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:26.753 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:26.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.745+0000 7f4e8559a640 10 monclient: _finish_auth 0 2024-09-11T01:38:26.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.745+0000 7f4e8559a640 10 monclient: _check_auth_tickets 2024-09-11T01:38:26.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.745+0000 7f4e8559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:26.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.745+0000 7f4e8559a640 1 -- 172.21.15.12:0/1854749414 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e70032640 con 0x7f4e880a2530 2024-09-11T01:38:26.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.745+0000 7f4e8caa9640 5 monclient: authenticate success, global_id 84302 2024-09-11T01:38:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.745+0000 7f4e8caa9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:38:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.745+0000 7f4e8559a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.745+0000 7f4e8caa9640 10 monclient: get_monmap_and_config success 2024-09-11T01:38:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.745+0000 7f4e8caa9640 4 set_mon_vals no callback set 2024-09-11T01:38:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.745+0000 7f4e8559a640 1 -- 172.21.15.12:0/1854749414 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e70032a10 con 0x7f4e880a2530 2024-09-11T01:38:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.745+0000 7f4e8559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.745+0000 7f4e8559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:38:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.745+0000 7f4e8559a640 10 monclient: dump: 2024-09-11T01:38:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:26.754 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:26.755 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:26.755 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:26.755 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:26.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.746+0000 7f4e8caa9640 10 monclient: shutdown 2024-09-11T01:38:26.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.746+0000 7f4e8caa9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:26.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.746+0000 7f4e8caa9640 1 -- 172.21.15.12:0/1854749414 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4e880a2530 msgr2=0x7f4e880a0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:26.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.746+0000 7f4e8caa9640 1 --2- 172.21.15.12:0/1854749414 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4e880a2530 0x7f4e880a0c90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4e8806cd50 tx=0x7f4e7002a0f0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.746+0000 7f4e8caa9640 1 -- 172.21.15.12:0/1854749414 shutdown_connections 2024-09-11T01:38:26.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.746+0000 7f4e8caa9640 1 --2- 172.21.15.12:0/1854749414 >> v2:172.21.15.12:3300/0 conn(0x7f4e880a11d0 0x7f4e880a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.746+0000 7f4e8caa9640 1 --2- 172.21.15.12:0/1854749414 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4e880a2530 0x7f4e880a0c90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.746+0000 7f4e8caa9640 1 --2- 172.21.15.12:0/1854749414 >> v2:172.21.15.71:3300/0 conn(0x7f4e880a03a0 0x7f4e8809f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.746+0000 7f4e8caa9640 1 -- 172.21.15.12:0/1854749414 >> 172.21.15.12:0/1854749414 conn(0x7f4e880a2f70 msgr2=0x7f4e880a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:26.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.746+0000 7f4e8caa9640 1 -- 172.21.15.12:0/1854749414 shutdown_connections 2024-09-11T01:38:26.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.746+0000 7f4e8caa9640 1 -- 172.21.15.12:0/1854749414 wait complete. 2024-09-11T01:38:26.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.746+0000 7f4e8caa9640 10 monclient: build_initial_monmap 2024-09-11T01:38:26.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.746+0000 7f4e8caa9640 10 monclient: monmap: 2024-09-11T01:38:26.756 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:26.756 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:38:26.756 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:38:26.756 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:38:26.756 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:26.756 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:26.756 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:38:26.756 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:38:26.756 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:38:26.756 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:26.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e8caa9640 1 Processor -- start 2024-09-11T01:38:26.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e8caa9640 1 -- start start 2024-09-11T01:38:26.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e8caa9640 10 monclient: init 2024-09-11T01:38:26.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e8caa9640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:26.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e8caa9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:38:26.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e8caa9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e880a03a0 0x7f4e88128080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:26.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e8caa9640 10 monclient(hunting): picked mon.noname-b con 0x7f4e880a03a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:38:26.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e8caa9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e880a11d0 0x7f4e88125f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:26.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e8caa9640 10 monclient(hunting): picked mon.noname-a con 0x7f4e880a11d0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:38:26.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e8caa9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4e880a2530 0x7f4e8813ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:26.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e8caa9640 10 monclient(hunting): picked mon.noname-c con 0x7f4e880a2530 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:26.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e8caa9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:26.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e8caa9640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8809cdc0 con 0x7f4e880a11d0 2024-09-11T01:38:26.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e8caa9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:26.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e8caa9640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8809cc40 con 0x7f4e880a03a0 2024-09-11T01:38:26.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e8caa9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:26.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e8caa9640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4e8809cf40 con 0x7f4e880a2530 2024-09-11T01:38:26.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e8caa9640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:26.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e8caa9640 10 monclient(hunting): authenticate will time out at 1313.189769s 2024-09-11T01:38:26.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e85d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e880a11d0 0x7f4e88125f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:26.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e86d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4e880a2530 0x7f4e8813ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:26.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e85d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e880a11d0 0x7f4e88125f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57722/0 (socket says 172.21.15.12:57722) 2024-09-11T01:38:26.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e86d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4e880a2530 0x7f4e8813ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:50854/0 (socket says 172.21.15.12:50854) 2024-09-11T01:38:26.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e85d9b640 1 -- 172.21.15.12:0/3771134141 learned_addr learned my addr 172.21.15.12:0/3771134141 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:38:26.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e85d9b640 10 monclient(hunting): get_auth_request con 0x7f4e880a11d0 auth_method 0 2024-09-11T01:38:26.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e85d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:26.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e85d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:26.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e85d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:26.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e8659c640 1 --2- 172.21.15.12:0/3771134141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e880a03a0 0x7f4e88128080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:26.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e86d9d640 10 monclient(hunting): get_auth_request con 0x7f4e880a2530 auth_method 0 2024-09-11T01:38:26.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e86d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:26.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e86d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:26.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e86d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:26.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e8659c640 10 monclient(hunting): get_auth_request con 0x7f4e880a03a0 auth_method 0 2024-09-11T01:38:26.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e8659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:26.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e8659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:26.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e8659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:26.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e85d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:26.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.747+0000 7f4e85d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:26.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.748+0000 7f4e85d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:26.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.748+0000 7f4e86d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:26.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.748+0000 7f4e86d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:26.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.748+0000 7f4e86d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:26.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.748+0000 7f4e8659c640 1 -- 172.21.15.12:0/3771134141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e880a03a0 msgr2=0x7f4e88128080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:38:26.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.748+0000 7f4e8659c640 1 -- 172.21.15.12:0/3771134141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e880a03a0 msgr2=0x7f4e88128080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:38:26.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.748+0000 7f4e8659c640 1 --2- 172.21.15.12:0/3771134141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e880a03a0 0x7f4e88128080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:38:26.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.748+0000 7f4e8659c640 1 --2- 172.21.15.12:0/3771134141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e880a03a0 0x7f4e88128080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:38:26.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.748+0000 7f4e86d9d640 10 monclient(hunting): handle_auth_done global_id 84305 payload 824 2024-09-11T01:38:26.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.748+0000 7f4e86d9d640 1 -- 172.21.15.12:0/3771134141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e880a03a0 msgr2=0x7f4e88128080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:38:26.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.748+0000 7f4e86d9d640 1 --2- 172.21.15.12:0/3771134141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e880a03a0 0x7f4e88128080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.748+0000 7f4e86d9d640 1 -- 172.21.15.12:0/3771134141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e880a11d0 msgr2=0x7f4e88125f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:26.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.748+0000 7f4e86d9d640 1 --2- 172.21.15.12:0/3771134141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e880a11d0 0x7f4e88125f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.748+0000 7f4e86d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:38:26.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.748+0000 7f4e86d9d640 1 monclient: found mon.noname-c 2024-09-11T01:38:26.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.748+0000 7f4e86d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:26.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.748+0000 7f4e86d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:38:26.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.748+0000 7f4e86d9d640 1 -- 172.21.15.12:0/3771134141 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e8809d480 con 0x7f4e880a2530 2024-09-11T01:38:26.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.748+0000 7f4e85d9b640 1 --2- 172.21.15.12:0/3771134141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e880a11d0 0x7f4e88125f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:38:26.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.750+0000 7f4e86d9d640 1 -- 172.21.15.12:0/3771134141 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4e880a2530 msgr2=0x7f4e8813ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:26.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.750+0000 7f4e86d9d640 1 --2- 172.21.15.12:0/3771134141 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4e880a2530 0x7f4e8813ced0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4e8000efc0 tx=0x7f4e8000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:26.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.750+0000 7f4e7effd640 1 -- 172.21.15.12:0/3771134141 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e80018030 con 0x7f4e880a2530 2024-09-11T01:38:26.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.750+0000 7f4e7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:26.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.750+0000 7f4e7effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:38:26.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.750+0000 7f4e7effd640 10 monclient: dump: 2024-09-11T01:38:26.767 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:26.767 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:26.767 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.767 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.767 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:26.767 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:26.767 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:26.768 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:26.768 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:26.768 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:26.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.750+0000 7f4e7effd640 10 monclient: _finish_auth 0 2024-09-11T01:38:26.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.750+0000 7f4e7effd640 10 monclient: _check_auth_tickets 2024-09-11T01:38:26.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.750+0000 7f4e7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:26.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.750+0000 7f4e7effd640 1 -- 172.21.15.12:0/3771134141 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e80003990 con 0x7f4e880a2530 2024-09-11T01:38:26.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.750+0000 7f4e8caa9640 5 monclient: authenticate success, global_id 84305 2024-09-11T01:38:26.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.750+0000 7f4e7effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:26.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.750+0000 7f4e7effd640 1 -- 172.21.15.12:0/3771134141 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e80003d60 con 0x7f4e880a2530 2024-09-11T01:38:26.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.750+0000 7f4e8caa9640 10 monclient: _renew_subs 2024-09-11T01:38:26.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.750+0000 7f4e8caa9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:38:26.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.750+0000 7f4e8caa9640 1 -- 172.21.15.12:0/3771134141 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e88129880 con 0x7f4e880a2530 2024-09-11T01:38:26.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.750+0000 7f4e7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:26.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.750+0000 7f4e84d99640 4 set_mon_vals no callback set 2024-09-11T01:38:26.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.750+0000 7f4e7effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:38:26.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.750+0000 7f4e7effd640 10 monclient: dump: 2024-09-11T01:38:26.769 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:26.769 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:26.769 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.769 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.769 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:26.769 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:26.770 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:26.770 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:26.770 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:26.770 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:26.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.750+0000 7f4e8caa9640 10 monclient: _renew_subs 2024-09-11T01:38:26.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.750+0000 7f4e8caa9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:38:26.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.750+0000 7f4e8caa9640 1 -- 172.21.15.12:0/3771134141 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e88128fe0 con 0x7f4e880a2530 2024-09-11T01:38:26.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.751+0000 7f4e8caa9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:26.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.751+0000 7f4e8caa9640 1 --2- 172.21.15.12:0/3771134141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8813d470 0x7f4e881cd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:26.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.751+0000 7f4e8caa9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:26.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.751+0000 7f4e8caa9640 1 -- 172.21.15.12:0/3771134141 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4e880a1600 con 0x7f4e8813d470 2024-09-11T01:38:26.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.751+0000 7f4e86d9d640 1 --2- 172.21.15.12:0/3771134141 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4e880a2530 0x7f4e8813ced0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4e8000efc0 tx=0x7f4e8000c490 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7f4e80003f40 mgrmap(e 5) 2024-09-11T01:38:26.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.751+0000 7f4e8659c640 1 --2- 172.21.15.12:0/3771134141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8813d470 0x7f4e881cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:26.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.751+0000 7f4e7effd640 1 -- 172.21.15.12:0/3771134141 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f4e80003f40 con 0x7f4e880a2530 2024-09-11T01:38:26.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.751+0000 7f4e8659c640 10 monclient: get_auth_request con 0x7f4e8813d470 auth_method 0 2024-09-11T01:38:26.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.751+0000 7f4e8659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:26.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.751+0000 7f4e8659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:26.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.751+0000 7f4e8659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:26.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.751+0000 7f4e8659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:26.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.751+0000 7f4e8659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:26.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.751+0000 7f4e8659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:26.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.751+0000 7f4e8659c640 10 monclient(hunting): handle_auth_done global_id 94215 payload 824 2024-09-11T01:38:26.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.751+0000 7f4e8659c640 1 -- 172.21.15.12:0/3771134141 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4e880008d0 con 0x7f4e8813d470 2024-09-11T01:38:26.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.751+0000 7f4e7effd640 1 --2- 172.21.15.12:0/3771134141 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4e6803e210 0x7f4e680406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:26.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.752+0000 7f4e85d9b640 1 --2- 172.21.15.12:0/3771134141 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4e6803e210 0x7f4e680406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:26.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.752+0000 7f4e85d9b640 10 monclient: get_auth_request con 0x7f4e6803e210 auth_method 0 2024-09-11T01:38:26.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.754+0000 7f4e8659c640 1 -- 172.21.15.12:0/3771134141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8813d470 msgr2=0x7f4e881cd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:26.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.754+0000 7f4e8659c640 1 --2- 172.21.15.12:0/3771134141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8813d470 0x7f4e881cd490 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4e78006fd0 tx=0x7f4e7800f910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:26.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.754+0000 7f4e7effd640 1 -- 172.21.15.12:0/3771134141 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e78019020 con 0x7f4e8813d470 2024-09-11T01:38:26.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.754+0000 7f4e7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:26.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.754+0000 7f4e7effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:38:26.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.754+0000 7f4e7effd640 10 monclient: dump: 2024-09-11T01:38:26.772 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:26.772 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:26.772 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.772 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.772 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:26.772 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:26.772 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:26.772 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:26.773 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:26.773 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:26.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.754+0000 7f4e7effd640 1 -- 172.21.15.12:0/3771134141 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4e78014a80 con 0x7f4e8813d470 2024-09-11T01:38:26.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.754+0000 7f4e7effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:26.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.754+0000 7f4e7effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:38:26.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.754+0000 7f4e7effd640 1 -- 172.21.15.12:0/3771134141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8813d470 msgr2=0x7f4e881cd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:26.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.754+0000 7f4e7effd640 1 --2- 172.21.15.12:0/3771134141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8813d470 0x7f4e881cd490 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4e78006fd0 tx=0x7f4e7800f910 comp rx=0 tx=0).stop 2024-09-11T01:38:26.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.754+0000 7f4e7effd640 1 -- 172.21.15.12:0/3771134141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8813d470 msgr2=0x7f4e881cd490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:38:26.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.754+0000 7f4e7effd640 1 --2- 172.21.15.12:0/3771134141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8813d470 0x7f4e881cd490 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4e78006fd0 tx=0x7f4e7800f910 comp rx=0 tx=0).stop 2024-09-11T01:38:26.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.754+0000 7f4e85d9b640 1 --2- 172.21.15.12:0/3771134141 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4e6803e210 0x7f4e680406d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4e700097f0 tx=0x7f4e7002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:26.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.760+0000 7f4e8caa9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:38:26.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.760+0000 7f4e8caa9640 1 --2- 172.21.15.12:0/3771134141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8802d440 0x7f4e8802d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:26.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.760+0000 7f4e8caa9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:26.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.760+0000 7f4e8caa9640 1 -- 172.21.15.12:0/3771134141 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4e88129430 con 0x7f4e8802d440 2024-09-11T01:38:26.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.760+0000 7f4e8659c640 1 --2- 172.21.15.12:0/3771134141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8802d440 0x7f4e8802d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:26.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.760+0000 7f4e8659c640 10 monclient: get_auth_request con 0x7f4e8802d440 auth_method 0 2024-09-11T01:38:26.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.760+0000 7f4e8659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:26.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.760+0000 7f4e8659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:26.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.760+0000 7f4e8659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:26.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.760+0000 7f4e8659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:26.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.760+0000 7f4e8659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:26.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.760+0000 7f4e8659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:26.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.761+0000 7f4e8659c640 10 monclient(hunting): handle_auth_done global_id 94218 payload 824 2024-09-11T01:38:26.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.761+0000 7f4e8659c640 1 -- 172.21.15.12:0/3771134141 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4e8811dff0 con 0x7f4e8802d440 2024-09-11T01:38:26.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.763+0000 7f4e8659c640 1 -- 172.21.15.12:0/3771134141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8802d440 msgr2=0x7f4e8802d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:26.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.763+0000 7f4e8659c640 1 --2- 172.21.15.12:0/3771134141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8802d440 0x7f4e8802d850 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4e7801a000 tx=0x7f4e780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:26.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.763+0000 7f4e7effd640 1 -- 172.21.15.12:0/3771134141 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e78019040 con 0x7f4e8802d440 2024-09-11T01:38:26.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.763+0000 7f4e7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:26.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.763+0000 7f4e7effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:38:26.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.763+0000 7f4e7effd640 10 monclient: dump: 2024-09-11T01:38:26.776 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:26.776 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:26.776 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.776 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.776 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:26.776 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:26.776 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:26.776 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:26.776 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:26.776 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:26.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.763+0000 7f4e7effd640 1 -- 172.21.15.12:0/3771134141 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7f4e7801e630 con 0x7f4e8802d440 2024-09-11T01:38:26.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.763+0000 7f4e7effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:38:26.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.763+0000 7f4e7effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:38:26.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.763+0000 7f4e7effd640 1 -- 172.21.15.12:0/3771134141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8802d440 msgr2=0x7f4e8802d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:26.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.763+0000 7f4e7effd640 1 --2- 172.21.15.12:0/3771134141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8802d440 0x7f4e8802d850 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4e7801a000 tx=0x7f4e780046c0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.763+0000 7f4e7effd640 1 -- 172.21.15.12:0/3771134141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8802d440 msgr2=0x7f4e8802d850 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:38:26.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.763+0000 7f4e7effd640 1 --2- 172.21.15.12:0/3771134141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e8802d440 0x7f4e8802d850 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.777 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:38:26.777 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:38:26.777 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:38:26.777 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:38:26.777 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 92, 2024-09-11T01:38:26.777 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 52859, 2024-09-11T01:38:26.778 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:38:26.778 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:38:26.778 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:38:26.778 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:38:26.778 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:26.778 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:38:26.778 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:38:26.778 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:38:26.778 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:38:26.778 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:38:26.778 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:38:26.778 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:38:26.778 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:38:26.779 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:38:26.779 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:38:26.779 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:38:26.779 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:38:26.779 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:38:26.779 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:38:26.779 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:38:26.779 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:26.779 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:38:26.779 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:38:26.779 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:38:26.779 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:38:26.779 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:38:26.779 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:38:26.780 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:38:26.780 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:38:26.780 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:38:26.780 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:38:26.780 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:38:26.780 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:38:26.780 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:38:26.780 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:26.780 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:26.780 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:38:26.780 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:38:26.780 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:38:26.780 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:38:26.780 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:38:26.781 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:38:26.781 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:38:26.781 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:38:26.781 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:38:26.781 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:38:26.781 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:38:26.781 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:38:26.781 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:38:26.781 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:38:26.781 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:38:26.781 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:38:26.781 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:38:26.781 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:38:26.782 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:38:26.782 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:38:26.782 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:38:26.782 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:38:26.782 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:38:26.782 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:38:26.782 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:38:26.782 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:38:26.782 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:38:26.782 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:38:26.782 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:26.782 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:38:26.782 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:26.783 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:38:26.783 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:26.783 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:38:26.783 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:38:26.783 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:38:26.783 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:38:26.783 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:26.783 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:38:26.783 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:38:26.783 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:26.783 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:26.783 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:26.783 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:38:26.783 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:38:26.783 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:26.784 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:26.784 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:26.784 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:26.784 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:38:26.784 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:38:26.784 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:38:26.784 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:38:26.784 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:38:26.784 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:26.784 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:26.784 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:38:26.784 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:38:26.784 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:38:26.785 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:38:26.785 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:26.785 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:38:26.785 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:38:26.785 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:26.785 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:26.785 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:26.785 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:38:26.785 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:38:26.785 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:26.785 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:26.785 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:26.785 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:26.785 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:38:26.785 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:38:26.786 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:38:26.786 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:38:26.786 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:38:26.786 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:26.786 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:26.786 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:38:26.786 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:38:26.786 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:38:26.786 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:38:26.786 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:26.786 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:38:26.786 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:38:26.786 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:26.786 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:26.787 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:26.787 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:38:26.787 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:38:26.787 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:26.787 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:26.787 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:26.787 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:26.787 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:38:26.787 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:38:26.787 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:38:26.787 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:38:26.787 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:38:26.787 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:26.787 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:26.787 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:26.788 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:38:26.788 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:38:26.788 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:26.788 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:38:26.788 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:38:26.788 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:38:26.788 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:26.788 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:26.788 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:38:26.788 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:26.788 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:38:26.788 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:38:26.788 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:38:26.788 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:26.789 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:26.789 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:38:26.789 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:26.789 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:38:26.789 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:38:26.789 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:38:26.789 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:26.789 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:26.789 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:26.789 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:38:26.789 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:38:26.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.763+0000 7f4e8caa9640 1 -- 172.21.15.12:0/3771134141 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4e6803e210 msgr2=0x7f4e680406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:26.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.763+0000 7f4e8caa9640 1 --2- 172.21.15.12:0/3771134141 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4e6803e210 0x7f4e680406d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4e700097f0 tx=0x7f4e7002e000 comp rx=0 tx=0).stop 2024-09-11T01:38:26.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.763+0000 7f4e8caa9640 10 monclient: shutdown 2024-09-11T01:38:26.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.763+0000 7f4e8caa9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:26.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.763+0000 7f4e8caa9640 1 -- 172.21.15.12:0/3771134141 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4e880a2530 msgr2=0x7f4e8813ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:26.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.763+0000 7f4e8caa9640 1 --2- 172.21.15.12:0/3771134141 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4e880a2530 0x7f4e8813ced0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4e8000efc0 tx=0x7f4e8000c490 comp rx=0 tx=0).stop 2024-09-11T01:38:26.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.763+0000 7f4e86d9d640 1 -- 172.21.15.12:0/3771134141 reap_dead start 2024-09-11T01:38:26.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.763+0000 7f4e86d9d640 1 -- 172.21.15.12:0/3771134141 reap_dead start 2024-09-11T01:38:26.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.763+0000 7f4e7effd640 1 -- 172.21.15.12:0/3771134141 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(315..315 src has 186..315) ==== 4799+0+0 (unknown 0 0 0) 0x7f4e80053aa0 con 0x7f4e880a2530 2024-09-11T01:38:26.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.763+0000 7f4e8caa9640 1 -- 172.21.15.12:0/3771134141 shutdown_connections 2024-09-11T01:38:26.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.763+0000 7f4e8caa9640 1 -- 172.21.15.12:0/3771134141 >> 172.21.15.12:0/3771134141 conn(0x7f4e880a2f70 msgr2=0x7f4e880a5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:26.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.764+0000 7f4e8caa9640 1 -- 172.21.15.12:0/3771134141 shutdown_connections 2024-09-11T01:38:26.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.764+0000 7f4e8caa9640 1 -- 172.21.15.12:0/3771134141 wait complete. 2024-09-11T01:38:26.792 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:38:26.792 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:38:26.792 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:38:26.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.892+0000 7fc567c7f640 10 monclient: get_monmap_and_config 2024-09-11T01:38:26.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.892+0000 7fc567c7f640 10 monclient: build_initial_monmap 2024-09-11T01:38:26.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.892+0000 7fc567c7f640 10 monclient: monmap: 2024-09-11T01:38:26.894 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:26.894 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:26.894 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:38:26.893241+0000 2024-09-11T01:38:26.894 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:38:26.893241+0000 2024-09-11T01:38:26.894 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:26.894 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:26.894 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:38:26.894 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:38:26.894 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:38:26.895 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:38:26.895 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:38:26.895 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:26.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.893+0000 7fc567c7f640 1 Processor -- start 2024-09-11T01:38:26.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.893+0000 7fc567c7f640 1 -- start start 2024-09-11T01:38:26.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.893+0000 7fc567c7f640 10 monclient: init 2024-09-11T01:38:26.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.893+0000 7fc567c7f640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:26.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.893+0000 7fc567c7f640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-11T01:38:26.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.893+0000 7fc567c7f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5600a2e20 0x7fc56006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:26.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.893+0000 7fc567c7f640 10 monclient(hunting): picked mon.noname-b con 0x7fc5600a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:26.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.893+0000 7fc567c7f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc5600a4970 0x7fc5600a3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:26.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.893+0000 7fc567c7f640 10 monclient(hunting): picked mon.noname-c con 0x7fc5600a4970 addr v2:172.21.15.71:3300/0 2024-09-11T01:38:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.893+0000 7fc567c7f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc56006f910 addr v1:172.21.15.71:6789/0 2024-09-11T01:38:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.893+0000 7fc567c7f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc56006f4f0 con 0x7fc56006f910 2024-09-11T01:38:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.893+0000 7fc567c7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.893+0000 7fc567c7f640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fc56006ef30 con 0x7fc5600a4970 2024-09-11T01:38:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.893+0000 7fc567c7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.893+0000 7fc567c7f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc56006ec50 con 0x7fc5600a2e20 2024-09-11T01:38:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.893+0000 7fc567c7f640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.893+0000 7fc567c7f640 10 monclient(hunting): authenticate will time out at 1313.335825s 2024-09-11T01:38:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.893+0000 7fc565a1b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5600a2e20 0x7fc56006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.893+0000 7fc565a1b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5600a2e20 0x7fc56006ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:50864/0 (socket says 172.21.15.12:50864) 2024-09-11T01:38:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.893+0000 7fc565a1b640 1 -- 172.21.15.12:0/1757760042 learned_addr learned my addr 172.21.15.12:0/1757760042 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:38:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.894+0000 7fc565a1b640 10 monclient(hunting): get_auth_request con 0x7fc5600a2e20 auth_method 0 2024-09-11T01:38:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.894+0000 7fc565a1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.894+0000 7fc565a1b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:26.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.894+0000 7fc565a1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:26.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.894+0000 7fc565a1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:26.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.894+0000 7fc565a1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:26.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.894+0000 7fc56521a640 1 --2- 172.21.15.12:0/1757760042 >> v2:172.21.15.71:3300/0 conn(0x7fc5600a4970 0x7fc5600a3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:26.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.894+0000 7fc565a1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:26.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.894+0000 7fc56521a640 10 monclient(hunting): get_auth_request con 0x7fc5600a4970 auth_method 0 2024-09-11T01:38:26.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.894+0000 7fc56521a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:26.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.894+0000 7fc56521a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:26.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.894+0000 7fc56521a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:26.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.894+0000 7fc56521a640 1 -- 172.21.15.12:0/1757760042 >> v2:172.21.15.71:3300/0 conn(0x7fc5600a4970 msgr2=0x7fc5600a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:38:26.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.894+0000 7fc56521a640 1 -- 172.21.15.12:0/1757760042 >> v2:172.21.15.71:3300/0 conn(0x7fc5600a4970 msgr2=0x7fc5600a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:38:26.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.894+0000 7fc56521a640 1 --2- 172.21.15.12:0/1757760042 >> v2:172.21.15.71:3300/0 conn(0x7fc5600a4970 0x7fc5600a3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:38:26.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.894+0000 7fc56521a640 1 --2- 172.21.15.12:0/1757760042 >> v2:172.21.15.71:3300/0 conn(0x7fc5600a4970 0x7fc5600a3200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:38:26.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.894+0000 7fc565a1b640 10 monclient(hunting): handle_auth_done global_id 84308 payload 274 2024-09-11T01:38:26.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.894+0000 7fc565a1b640 1 -- 172.21.15.12:0/1757760042 >> v2:172.21.15.71:3300/0 conn(0x7fc5600a4970 msgr2=0x7fc5600a3200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:38:26.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.894+0000 7fc565a1b640 1 --2- 172.21.15.12:0/1757760042 >> v2:172.21.15.71:3300/0 conn(0x7fc5600a4970 0x7fc5600a3200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.894+0000 7fc565a1b640 1 -- 172.21.15.12:0/1757760042 >> v1:172.21.15.71:6789/0 conn(0x7fc56006f910 legacy=0x7fc560137fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:38:26.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.894+0000 7fc565a1b640 10 monclient: _finish_hunting 0 2024-09-11T01:38:26.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.894+0000 7fc565a1b640 1 monclient: found mon.noname-b 2024-09-11T01:38:26.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.894+0000 7fc565a1b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:26.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.894+0000 7fc565a1b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:38:26.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.894+0000 7fc565a1b640 1 -- 172.21.15.12:0/1757760042 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5600a3740 con 0x7fc5600a2e20 2024-09-11T01:38:26.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc565a1b640 1 -- 172.21.15.12:0/1757760042 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5600a2e20 msgr2=0x7fc56006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:26.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc565a1b640 1 --2- 172.21.15.12:0/1757760042 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5600a2e20 0x7fc56006ded0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc55c009870 tx=0x7fc55c02a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b03cd2549138cbe4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:26.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc564a19640 1 -- 172.21.15.12:0/1757760042 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55c037070 con 0x7fc5600a2e20 2024-09-11T01:38:26.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc564a19640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:26.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc564a19640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:38:26.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc564a19640 10 monclient: dump: 2024-09-11T01:38:26.907 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:26.907 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:26.907 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.907 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.907 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:26.907 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:26.907 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:26.907 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:26.907 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:26.907 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:26.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc564a19640 10 monclient: _finish_auth 0 2024-09-11T01:38:26.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc564a19640 10 monclient: _check_auth_tickets 2024-09-11T01:38:26.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc564a19640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:26.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc564a19640 1 -- 172.21.15.12:0/1757760042 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc55c032440 con 0x7fc5600a2e20 2024-09-11T01:38:26.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc567c7f640 5 monclient: authenticate success, global_id 84308 2024-09-11T01:38:26.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc567c7f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:38:26.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc564a19640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:26.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc567c7f640 10 monclient: get_monmap_and_config success 2024-09-11T01:38:26.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc564a19640 1 -- 172.21.15.12:0/1757760042 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55c032810 con 0x7fc5600a2e20 2024-09-11T01:38:26.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc567c7f640 4 set_mon_vals no callback set 2024-09-11T01:38:26.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc564a19640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:26.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc564a19640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:38:26.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc564a19640 10 monclient: dump: 2024-09-11T01:38:26.908 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:26.908 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:26.908 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.908 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.909 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:26.909 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:26.909 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:26.909 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:26.909 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:26.909 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:26.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc567c7f640 10 monclient: shutdown 2024-09-11T01:38:26.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc567c7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:26.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc567c7f640 1 -- 172.21.15.12:0/1757760042 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5600a2e20 msgr2=0x7fc56006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:26.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc567c7f640 1 --2- 172.21.15.12:0/1757760042 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5600a2e20 0x7fc56006ded0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc55c009870 tx=0x7fc55c02a0a0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc567c7f640 1 -- 172.21.15.12:0/1757760042 shutdown_connections 2024-09-11T01:38:26.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc567c7f640 1 --2- 172.21.15.12:0/1757760042 >> v2:172.21.15.71:3300/0 conn(0x7fc5600a4970 0x7fc5600a3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc567c7f640 1 --2- 172.21.15.12:0/1757760042 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5600a2e20 0x7fc56006ded0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc567c7f640 1 -- 172.21.15.12:0/1757760042 >> 172.21.15.12:0/1757760042 conn(0x7fc5600a8600 msgr2=0x7fc5600a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:26.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc567c7f640 1 -- 172.21.15.12:0/1757760042 shutdown_connections 2024-09-11T01:38:26.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.897+0000 7fc567c7f640 1 -- 172.21.15.12:0/1757760042 wait complete. 2024-09-11T01:38:26.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.898+0000 7fc567c7f640 10 monclient: build_initial_monmap 2024-09-11T01:38:26.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.898+0000 7fc567c7f640 10 monclient: monmap: 2024-09-11T01:38:26.910 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:26.910 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:38:26.910 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:38:26.910 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:38:26.910 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:26.910 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:26.910 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:38:26.910 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:38:26.910 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:38:26.910 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:26.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.898+0000 7fc567c7f640 1 Processor -- start 2024-09-11T01:38:26.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.898+0000 7fc567c7f640 1 -- start start 2024-09-11T01:38:26.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.898+0000 7fc567c7f640 10 monclient: init 2024-09-11T01:38:26.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc567c7f640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:26.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc567c7f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:38:26.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc567c7f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc56006f910 0x7fc5601cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:26.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc567c7f640 10 monclient(hunting): picked mon.noname-a con 0x7fc56006f910 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:38:26.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc567c7f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5600a2e20 0x7fc5601cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:26.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc567c7f640 10 monclient(hunting): picked mon.noname-c con 0x7fc5600a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:26.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc567c7f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5600a4970 0x7fc560124b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:26.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc567c7f640 10 monclient(hunting): picked mon.noname-b con 0x7fc5600a4970 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:38:26.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc567c7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:26.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc567c7f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc560138860 con 0x7fc56006f910 2024-09-11T01:38:26.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc567c7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:26.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc567c7f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc5601386e0 con 0x7fc5600a4970 2024-09-11T01:38:26.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc567c7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:26.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc567c7f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc5601389e0 con 0x7fc5600a2e20 2024-09-11T01:38:26.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc567c7f640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:26.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc567c7f640 10 monclient(hunting): authenticate will time out at 1313.341263s 2024-09-11T01:38:26.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc56521a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5600a2e20 0x7fc5601cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:26.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc565a1b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc56006f910 0x7fc5601cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:26.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc56521a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5600a2e20 0x7fc5601cd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:50872/0 (socket says 172.21.15.12:50872) 2024-09-11T01:38:26.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc56521a640 1 -- 172.21.15.12:0/3361470318 learned_addr learned my addr 172.21.15.12:0/3361470318 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:38:26.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc565a1b640 10 monclient(hunting): get_auth_request con 0x7fc56006f910 auth_method 0 2024-09-11T01:38:26.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc565a1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:26.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc565a1b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:26.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc565a1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:26.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc56521a640 10 monclient(hunting): get_auth_request con 0x7fc5600a2e20 auth_method 0 2024-09-11T01:38:26.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc56521a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:26.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc56521a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:26.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc56521a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:26.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc565a1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:26.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc565a1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:26.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc565a1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:26.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc56521a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:26.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc56521a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:26.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc56521a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:26.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc565a1b640 10 monclient(hunting): handle_auth_done global_id 94221 payload 824 2024-09-11T01:38:26.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc565a1b640 1 -- 172.21.15.12:0/3361470318 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5600a2e20 msgr2=0x7fc5601cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:26.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc565a1b640 1 --2- 172.21.15.12:0/3361470318 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5600a2e20 0x7fc5601cd9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc565a1b640 1 -- 172.21.15.12:0/3361470318 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5600a4970 msgr2=0x7fc560124b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:26.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc56621c640 1 --2- 172.21.15.12:0/3361470318 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5600a4970 0x7fc560124b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:26.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc565a1b640 1 --2- 172.21.15.12:0/3361470318 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5600a4970 0x7fc560124b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc565a1b640 10 monclient: _finish_hunting 0 2024-09-11T01:38:26.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc565a1b640 1 monclient: found mon.noname-a 2024-09-11T01:38:26.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc565a1b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:26.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc565a1b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:38:26.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc565a1b640 1 -- 172.21.15.12:0/3361470318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc560126650 con 0x7fc56006f910 2024-09-11T01:38:26.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc56621c640 10 monclient: get_auth_request con 0x7fc5600a4970 auth_method 0 2024-09-11T01:38:26.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc56521a640 1 --2- 172.21.15.12:0/3361470318 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5600a2e20 0x7fc5601cd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:38:26.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.899+0000 7fc56621c640 1 --2- 172.21.15.12:0/3361470318 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5600a4970 0x7fc560124b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:38:26.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.902+0000 7fc565a1b640 1 -- 172.21.15.12:0/3361470318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc56006f910 msgr2=0x7fc5601cd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:26.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.902+0000 7fc565a1b640 1 --2- 172.21.15.12:0/3361470318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc56006f910 0x7fc5601cd470 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc55c031000 tx=0x7fc55c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:26.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.902+0000 7fc5567fc640 1 -- 172.21.15.12:0/3361470318 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55c041030 con 0x7fc56006f910 2024-09-11T01:38:26.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.902+0000 7fc5567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:26.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.902+0000 7fc5567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:38:26.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.902+0000 7fc5567fc640 10 monclient: dump: 2024-09-11T01:38:26.915 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:26.915 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:26.915 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.915 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.915 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:26.915 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:26.915 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:26.915 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:26.916 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:26.916 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:26.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.902+0000 7fc5567fc640 10 monclient: _finish_auth 0 2024-09-11T01:38:26.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.902+0000 7fc5567fc640 10 monclient: _check_auth_tickets 2024-09-11T01:38:26.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.902+0000 7fc5567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:26.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.902+0000 7fc567c7f640 5 monclient: authenticate success, global_id 94221 2024-09-11T01:38:26.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.902+0000 7fc5567fc640 1 -- 172.21.15.12:0/3361470318 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc55c03d6f0 con 0x7fc56006f910 2024-09-11T01:38:26.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.902+0000 7fc5567fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:26.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.902+0000 7fc5567fc640 1 -- 172.21.15.12:0/3361470318 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55c037040 con 0x7fc56006f910 2024-09-11T01:38:26.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.902+0000 7fc567c7f640 10 monclient: _renew_subs 2024-09-11T01:38:26.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.902+0000 7fc567c7f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:38:26.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.902+0000 7fc567c7f640 1 -- 172.21.15.12:0/3361470318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc560128110 con 0x7fc56006f910 2024-09-11T01:38:26.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.902+0000 7fc5567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:26.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.902+0000 7fc5567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:38:26.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.902+0000 7fc5567fc640 10 monclient: dump: 2024-09-11T01:38:26.917 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:26.917 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:26.917 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.917 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.917 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:26.917 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:26.917 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:26.917 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:26.917 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:26.917 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:26.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.902+0000 7fc557fff640 4 set_mon_vals no callback set 2024-09-11T01:38:26.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.902+0000 7fc567c7f640 10 monclient: _renew_subs 2024-09-11T01:38:26.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.902+0000 7fc567c7f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:38:26.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.902+0000 7fc567c7f640 1 -- 172.21.15.12:0/3361470318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc560137f70 con 0x7fc56006f910 2024-09-11T01:38:26.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.903+0000 7fc5567fc640 1 -- 172.21.15.12:0/3361470318 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fc55c03a030 con 0x7fc56006f910 2024-09-11T01:38:26.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.903+0000 7fc567c7f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:26.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.903+0000 7fc567c7f640 1 --2- 172.21.15.12:0/3361470318 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5601250d0 0x7fc5601254b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:26.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.903+0000 7fc567c7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:26.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.903+0000 7fc567c7f640 1 -- 172.21.15.12:0/3361470318 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc56006a060 con 0x7fc5601250d0 2024-09-11T01:38:26.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.903+0000 7fc56521a640 1 --2- 172.21.15.12:0/3361470318 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5601250d0 0x7fc5601254b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:26.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.903+0000 7fc56521a640 10 monclient: get_auth_request con 0x7fc5601250d0 auth_method 0 2024-09-11T01:38:26.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.903+0000 7fc56521a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:26.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.903+0000 7fc56521a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:26.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.903+0000 7fc56521a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:26.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.903+0000 7fc56521a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:26.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.903+0000 7fc56521a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:26.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.903+0000 7fc5567fc640 1 --2- 172.21.15.12:0/3361470318 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc54403e240 0x7fc544040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:26.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.903+0000 7fc56521a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:26.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.903+0000 7fc5567fc640 1 -- 172.21.15.12:0/3361470318 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(315..315 src has 186..315) ==== 4799+0+0 (secure 0 0 0) 0x7fc55c072bd0 con 0x7fc56006f910 2024-09-11T01:38:26.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.903+0000 7fc56521a640 10 monclient(hunting): handle_auth_done global_id 84314 payload 824 2024-09-11T01:38:26.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.903+0000 7fc56521a640 1 -- 172.21.15.12:0/3361470318 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc560128860 con 0x7fc5601250d0 2024-09-11T01:38:26.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.903+0000 7fc56621c640 1 --2- 172.21.15.12:0/3361470318 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc54403e240 0x7fc544040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:26.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.903+0000 7fc56621c640 10 monclient: get_auth_request con 0x7fc54403e240 auth_method 0 2024-09-11T01:38:26.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.905+0000 7fc56521a640 1 -- 172.21.15.12:0/3361470318 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5601250d0 msgr2=0x7fc5601254b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:26.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.905+0000 7fc56521a640 1 --2- 172.21.15.12:0/3361470318 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5601250d0 0x7fc5601254b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc550006e80 tx=0x7fc55000e5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:26.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.905+0000 7fc5567fc640 1 -- 172.21.15.12:0/3361470318 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc550018070 con 0x7fc5601250d0 2024-09-11T01:38:26.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.905+0000 7fc5567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:26.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.905+0000 7fc5567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:38:26.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.905+0000 7fc5567fc640 10 monclient: dump: 2024-09-11T01:38:26.921 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:26.921 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:26.922 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.922 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.922 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:26.922 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:26.922 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:26.922 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:26.922 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:26.922 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:26.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.906+0000 7fc5567fc640 1 -- 172.21.15.12:0/3361470318 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc550014640 con 0x7fc5601250d0 2024-09-11T01:38:26.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.906+0000 7fc5567fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:26.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.906+0000 7fc5567fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:38:26.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.906+0000 7fc5567fc640 1 -- 172.21.15.12:0/3361470318 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5601250d0 msgr2=0x7fc5601254b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:26.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.906+0000 7fc5567fc640 1 --2- 172.21.15.12:0/3361470318 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5601250d0 0x7fc5601254b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc550006e80 tx=0x7fc55000e5a0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.906+0000 7fc5567fc640 1 -- 172.21.15.12:0/3361470318 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5601250d0 msgr2=0x7fc5601254b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:38:26.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.906+0000 7fc5567fc640 1 --2- 172.21.15.12:0/3361470318 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5601250d0 0x7fc5601254b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.906+0000 7fc56621c640 1 --2- 172.21.15.12:0/3361470318 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc54403e240 0x7fc544040700 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc560128600 tx=0x7fc558009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:26.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.912+0000 7fc567c7f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:38:26.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.912+0000 7fc567c7f640 1 --2- 172.21.15.12:0/3361470318 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5601259f0 0x7fc5600a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:26.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.912+0000 7fc567c7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:26.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.912+0000 7fc567c7f640 1 -- 172.21.15.12:0/3361470318 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc56006ded0 con 0x7fc5601259f0 2024-09-11T01:38:26.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.912+0000 7fc56521a640 1 --2- 172.21.15.12:0/3361470318 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5601259f0 0x7fc5600a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:26.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.912+0000 7fc56521a640 10 monclient: get_auth_request con 0x7fc5601259f0 auth_method 0 2024-09-11T01:38:26.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.912+0000 7fc56521a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:26.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.912+0000 7fc56521a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:26.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.912+0000 7fc56521a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:26.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.912+0000 7fc56521a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:26.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.912+0000 7fc56521a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:26.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.912+0000 7fc56521a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:26.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.912+0000 7fc56521a640 10 monclient(hunting): handle_auth_done global_id 84317 payload 824 2024-09-11T01:38:26.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.912+0000 7fc56521a640 1 -- 172.21.15.12:0/3361470318 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc56009f590 con 0x7fc5601259f0 2024-09-11T01:38:26.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.914+0000 7fc56521a640 1 -- 172.21.15.12:0/3361470318 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5601259f0 msgr2=0x7fc5600a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:26.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.915+0000 7fc56521a640 1 --2- 172.21.15.12:0/3361470318 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5601259f0 0x7fc5600a0830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc55001a000 tx=0x7fc5500046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:26.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.915+0000 7fc5567fc640 1 -- 172.21.15.12:0/3361470318 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55002b020 con 0x7fc5601259f0 2024-09-11T01:38:26.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.915+0000 7fc5567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:26.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.915+0000 7fc5567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:38:26.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.915+0000 7fc5567fc640 10 monclient: dump: 2024-09-11T01:38:26.924 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:26.924 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:26.924 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.924 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:26.924 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:26.924 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:26.924 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:26.925 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:26.925 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:26.925 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:26.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.915+0000 7fc5567fc640 1 -- 172.21.15.12:0/3361470318 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7fc5500273f0 con 0x7fc5601259f0 2024-09-11T01:38:26.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.915+0000 7fc5567fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:38:26.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.915+0000 7fc5567fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:38:26.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.915+0000 7fc5567fc640 1 -- 172.21.15.12:0/3361470318 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5601259f0 msgr2=0x7fc5600a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:26.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.915+0000 7fc5567fc640 1 --2- 172.21.15.12:0/3361470318 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5601259f0 0x7fc5600a0830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc55001a000 tx=0x7fc5500046c0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.915+0000 7fc5567fc640 1 -- 172.21.15.12:0/3361470318 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5601259f0 msgr2=0x7fc5600a0830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:38:26.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.915+0000 7fc5567fc640 1 --2- 172.21.15.12:0/3361470318 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc5601259f0 0x7fc5600a0830 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc55001a000 tx=0x7fc5500046c0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.925 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:38:26.926 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:38:26.926 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:38:26.926 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:38:26.926 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 92, 2024-09-11T01:38:26.926 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 100939, 2024-09-11T01:38:26.926 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:38:26.926 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:38:26.926 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:38:26.926 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:38:26.926 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:26.926 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:38:26.926 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:38:26.927 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:38:26.927 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:38:26.927 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:38:26.927 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:38:26.927 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:38:26.927 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:38:26.927 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:38:26.927 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:38:26.927 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:38:26.927 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:38:26.927 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:38:26.927 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:38:26.927 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:38:26.927 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:26.927 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:38:26.928 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:38:26.928 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:38:26.928 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:38:26.928 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:38:26.928 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:38:26.928 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:38:26.928 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:38:26.928 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:38:26.928 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:38:26.928 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:38:26.928 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:38:26.928 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:38:26.928 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:26.928 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:26.929 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:38:26.929 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:38:26.929 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:38:26.929 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:38:26.929 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:38:26.929 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:38:26.929 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:38:26.929 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:38:26.929 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:38:26.929 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:38:26.929 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:38:26.929 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:38:26.929 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:38:26.929 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:38:26.930 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:38:26.930 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:38:26.930 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:38:26.930 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:38:26.930 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:38:26.930 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:38:26.930 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:38:26.930 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:38:26.930 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:38:26.930 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:38:26.930 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:38:26.930 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:38:26.930 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:38:26.930 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:38:26.931 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:26.931 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:38:26.931 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:26.931 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:38:26.931 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:26.931 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:38:26.931 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:38:26.931 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:38:26.931 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:38:26.931 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:26.931 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:38:26.931 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:38:26.931 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:26.931 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:26.932 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:26.932 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:38:26.932 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:38:26.932 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:26.932 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:26.932 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:26.932 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:26.932 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:38:26.932 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:38:26.932 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:38:26.932 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:38:26.932 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:38:26.932 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:26.932 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:26.933 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:38:26.933 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:38:26.933 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:38:26.933 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:38:26.933 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:26.933 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:38:26.933 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:38:26.933 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:26.933 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:26.933 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:26.933 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:38:26.933 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:38:26.933 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:26.933 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:26.934 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:26.934 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:26.934 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:38:26.934 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:38:26.934 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:38:26.934 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:38:26.934 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:38:26.934 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:26.934 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:26.934 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:38:26.934 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:38:26.934 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:38:26.935 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:38:26.935 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:26.935 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:38:26.935 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:38:26.935 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:26.935 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:26.935 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:26.935 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:38:26.935 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:38:26.935 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:26.935 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:26.935 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:26.936 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:26.936 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:38:26.936 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:38:26.936 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:38:26.936 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:38:26.936 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:38:26.936 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:26.936 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:26.936 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:26.936 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:38:26.936 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:38:26.937 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:26.937 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:38:26.937 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:38:26.937 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:38:26.937 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:26.937 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:26.937 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:38:26.937 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:26.937 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:38:26.937 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:38:26.937 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:38:26.937 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:26.937 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:26.937 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:38:26.938 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:26.938 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:38:26.938 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:38:26.938 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:38:26.938 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:26.938 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:26.939 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:38:26.939 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:26.939 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:38:26.939 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:38:26.939 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:38:26.939 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:26.939 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:26.939 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:26.939 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:38:26.939 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:38:26.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.915+0000 7fc567c7f640 1 -- 172.21.15.12:0/3361470318 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc54403e240 msgr2=0x7fc544040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:26.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.915+0000 7fc567c7f640 1 --2- 172.21.15.12:0/3361470318 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc54403e240 0x7fc544040700 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc560128600 tx=0x7fc558009040 comp rx=0 tx=0).stop 2024-09-11T01:38:26.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.915+0000 7fc567c7f640 10 monclient: shutdown 2024-09-11T01:38:26.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.915+0000 7fc567c7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:26.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.915+0000 7fc567c7f640 1 -- 172.21.15.12:0/3361470318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc56006f910 msgr2=0x7fc5601cd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:26.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.915+0000 7fc567c7f640 1 --2- 172.21.15.12:0/3361470318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc56006f910 0x7fc5601cd470 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc55c031000 tx=0x7fc55c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.915+0000 7fc56621c640 1 -- 172.21.15.12:0/3361470318 reap_dead start 2024-09-11T01:38:26.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.915+0000 7fc567c7f640 1 -- 172.21.15.12:0/3361470318 shutdown_connections 2024-09-11T01:38:26.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.915+0000 7fc567c7f640 1 --2- 172.21.15.12:0/3361470318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc56006f910 0x7fc5601cd470 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:26.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.915+0000 7fc567c7f640 1 -- 172.21.15.12:0/3361470318 >> 172.21.15.12:0/3361470318 conn(0x7fc5600a8600 msgr2=0x7fc560072e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:26.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.915+0000 7fc567c7f640 1 -- 172.21.15.12:0/3361470318 shutdown_connections 2024-09-11T01:38:26.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:26.915+0000 7fc567c7f640 1 -- 172.21.15.12:0/3361470318 wait complete. 2024-09-11T01:38:26.942 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:38:26.942 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:38:26.942 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:38:27.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.042+0000 7fd5af03c640 10 monclient: get_monmap_and_config 2024-09-11T01:38:27.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.042+0000 7fd5af03c640 10 monclient: build_initial_monmap 2024-09-11T01:38:27.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.042+0000 7fd5af03c640 10 monclient: monmap: 2024-09-11T01:38:27.044 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:27.044 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:27.044 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:38:27.043427+0000 2024-09-11T01:38:27.044 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:38:27.043427+0000 2024-09-11T01:38:27.044 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:27.044 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:27.045 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:38:27.045 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:38:27.045 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:38:27.045 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:38:27.045 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:38:27.045 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:27.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.043+0000 7fd5af03c640 1 Processor -- start 2024-09-11T01:38:27.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.043+0000 7fd5af03c640 1 -- start start 2024-09-11T01:38:27.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.043+0000 7fd5af03c640 10 monclient: init 2024-09-11T01:38:27.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.043+0000 7fd5af03c640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:27.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.043+0000 7fd5af03c640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-11T01:38:27.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.043+0000 7fd5af03c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd5a8134ad0 0x7fd5a8134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:27.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.043+0000 7fd5af03c640 10 monclient(hunting): picked mon.noname-c con 0x7fd5a8134ad0 addr v2:172.21.15.71:3300/0 2024-09-11T01:38:27.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.043+0000 7fd5af03c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5a80a2200 0x7fd5a81353f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:27.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.043+0000 7fd5af03c640 10 monclient(hunting): picked mon.noname-b con 0x7fd5a80a2200 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:27.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.043+0000 7fd5af03c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd5a8133a50 addr v1:172.21.15.71:6789/0 2024-09-11T01:38:27.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.043+0000 7fd5af03c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5a809ed30 con 0x7fd5a8133a50 2024-09-11T01:38:27.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.043+0000 7fd5af03c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:27.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.043+0000 7fd5af03c640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fd5a809f5d0 con 0x7fd5a8134ad0 2024-09-11T01:38:27.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.043+0000 7fd5af03c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:27.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.043+0000 7fd5af03c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd5a8135930 con 0x7fd5a80a2200 2024-09-11T01:38:27.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.043+0000 7fd5af03c640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:27.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.043+0000 7fd5af03c640 10 monclient(hunting): authenticate will time out at 1313.485942s 2024-09-11T01:38:27.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.044+0000 7fd5a7fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5a80a2200 0x7fd5a81353f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:27.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.044+0000 7fd5a7fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5a80a2200 0x7fd5a81353f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:50904/0 (socket says 172.21.15.12:50904) 2024-09-11T01:38:27.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.044+0000 7fd5a7fff640 1 -- 172.21.15.12:0/3567812733 learned_addr learned my addr 172.21.15.12:0/3567812733 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:38:27.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.044+0000 7fd5a7fff640 10 monclient(hunting): get_auth_request con 0x7fd5a80a2200 auth_method 0 2024-09-11T01:38:27.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.044+0000 7fd5a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:27.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.044+0000 7fd5a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:27.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.044+0000 7fd5a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:27.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.044+0000 7fd5a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:27.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.044+0000 7fd5a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:27.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.044+0000 7fd5a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:27.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.044+0000 7fd5acdd8640 1 --2- 172.21.15.12:0/3567812733 >> v2:172.21.15.71:3300/0 conn(0x7fd5a8134ad0 0x7fd5a8134eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:27.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.044+0000 7fd5acdd8640 10 monclient(hunting): get_auth_request con 0x7fd5a8134ad0 auth_method 0 2024-09-11T01:38:27.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.044+0000 7fd5acdd8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:27.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.044+0000 7fd5acdd8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:27.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.044+0000 7fd5acdd8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:27.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.044+0000 7fd5a7fff640 10 monclient(hunting): handle_auth_done global_id 84320 payload 274 2024-09-11T01:38:27.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.044+0000 7fd5a7fff640 1 -- 172.21.15.12:0/3567812733 >> v2:172.21.15.71:3300/0 conn(0x7fd5a8134ad0 msgr2=0x7fd5a8134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:38:27.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.044+0000 7fd5a7fff640 1 --2- 172.21.15.12:0/3567812733 >> v2:172.21.15.71:3300/0 conn(0x7fd5a8134ad0 0x7fd5a8134eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:27.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.044+0000 7fd5a7fff640 1 -- 172.21.15.12:0/3567812733 >> v1:172.21.15.71:6789/0 conn(0x7fd5a8133a50 legacy=0x7fd5a8133ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:38:27.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.044+0000 7fd5a7fff640 10 monclient: _finish_hunting 0 2024-09-11T01:38:27.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.044+0000 7fd5a7fff640 1 monclient: found mon.noname-b 2024-09-11T01:38:27.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.044+0000 7fd5a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:27.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.044+0000 7fd5a7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:38:27.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.044+0000 7fd5a7fff640 1 -- 172.21.15.12:0/3567812733 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a809f010 con 0x7fd5a80a2200 2024-09-11T01:38:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5a7fff640 1 -- 172.21.15.12:0/3567812733 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5a80a2200 msgr2=0x7fd5a81353f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5a7fff640 1 --2- 172.21.15.12:0/3567812733 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5a80a2200 0x7fd5a81353f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd59400bb10 tx=0x7fd59402c750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d713f3a1617a6ad server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5a77fe640 1 -- 172.21.15.12:0/3567812733 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd594031920 con 0x7fd5a80a2200 2024-09-11T01:38:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:27.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5a77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:38:27.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5a77fe640 10 monclient: dump: 2024-09-11T01:38:27.057 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:27.057 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:27.057 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:27.057 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:27.057 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:27.057 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:27.057 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:27.057 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:27.057 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:27.057 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:27.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5a77fe640 10 monclient: _finish_auth 0 2024-09-11T01:38:27.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5a77fe640 10 monclient: _check_auth_tickets 2024-09-11T01:38:27.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:27.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5a77fe640 1 -- 172.21.15.12:0/3567812733 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd594031ac0 con 0x7fd5a80a2200 2024-09-11T01:38:27.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5af03c640 5 monclient: authenticate success, global_id 84320 2024-09-11T01:38:27.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5af03c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:38:27.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5a77fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:27.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5af03c640 10 monclient: get_monmap_and_config success 2024-09-11T01:38:27.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5a77fe640 1 -- 172.21.15.12:0/3567812733 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd594033400 con 0x7fd5a80a2200 2024-09-11T01:38:27.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5af03c640 4 set_mon_vals no callback set 2024-09-11T01:38:27.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:27.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:38:27.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5a77fe640 10 monclient: dump: 2024-09-11T01:38:27.058 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:27.058 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:27.058 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:27.058 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:27.059 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:27.059 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:27.059 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:27.059 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:27.059 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:27.059 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:27.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5af03c640 10 monclient: shutdown 2024-09-11T01:38:27.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5af03c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:27.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5af03c640 1 -- 172.21.15.12:0/3567812733 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5a80a2200 msgr2=0x7fd5a81353f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:27.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5af03c640 1 --2- 172.21.15.12:0/3567812733 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5a80a2200 0x7fd5a81353f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd59400bb10 tx=0x7fd59402c750 comp rx=0 tx=0).stop 2024-09-11T01:38:27.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5af03c640 1 -- 172.21.15.12:0/3567812733 shutdown_connections 2024-09-11T01:38:27.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5af03c640 1 --2- 172.21.15.12:0/3567812733 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5a80a2200 0x7fd5a81353f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:27.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5af03c640 1 --2- 172.21.15.12:0/3567812733 >> v2:172.21.15.71:3300/0 conn(0x7fd5a8134ad0 0x7fd5a8134eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:27.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.047+0000 7fd5af03c640 1 -- 172.21.15.12:0/3567812733 >> 172.21.15.12:0/3567812733 conn(0x7fd5a80a2f70 msgr2=0x7fd5a80a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:27.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.048+0000 7fd5af03c640 1 -- 172.21.15.12:0/3567812733 shutdown_connections 2024-09-11T01:38:27.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.048+0000 7fd5af03c640 1 -- 172.21.15.12:0/3567812733 wait complete. 2024-09-11T01:38:27.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.048+0000 7fd5af03c640 10 monclient: build_initial_monmap 2024-09-11T01:38:27.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.048+0000 7fd5af03c640 10 monclient: monmap: 2024-09-11T01:38:27.060 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:27.060 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:38:27.060 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:38:27.060 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:38:27.060 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:27.060 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:27.060 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:38:27.060 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:38:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:38:27.061 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.048+0000 7fd5af03c640 1 Processor -- start 2024-09-11T01:38:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.048+0000 7fd5af03c640 1 -- start start 2024-09-11T01:38:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.048+0000 7fd5af03c640 10 monclient: init 2024-09-11T01:38:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5af03c640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5af03c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:38:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5af03c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a80a2200 0x7fd5a81cd300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5af03c640 10 monclient(hunting): picked mon.noname-b con 0x7fd5a80a2200 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:38:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5af03c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5a8133a50 0x7fd5a8132f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5af03c640 10 monclient(hunting): picked mon.noname-a con 0x7fd5a8133a50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:38:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5af03c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5a8134ad0 0x7fd5a8133480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5af03c640 10 monclient(hunting): picked mon.noname-c con 0x7fd5a8134ad0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:27.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5af03c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:27.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5af03c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a80a0150 con 0x7fd5a8133a50 2024-09-11T01:38:27.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5af03c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:27.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5af03c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a809ffd0 con 0x7fd5a80a2200 2024-09-11T01:38:27.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5af03c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:27.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5af03c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd5a81345f0 con 0x7fd5a8134ad0 2024-09-11T01:38:27.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5af03c640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:27.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5af03c640 10 monclient(hunting): authenticate will time out at 1313.491537s 2024-09-11T01:38:27.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5a7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5a8133a50 0x7fd5a8132f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:27.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5ad5d9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5a8134ad0 0x7fd5a8133480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:27.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5a7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5a8133a50 0x7fd5a8132f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57748/0 (socket says 172.21.15.12:57748) 2024-09-11T01:38:27.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5a7fff640 1 -- 172.21.15.12:0/2874081558 learned_addr learned my addr 172.21.15.12:0/2874081558 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:38:27.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5ad5d9640 10 monclient(hunting): get_auth_request con 0x7fd5a8134ad0 auth_method 0 2024-09-11T01:38:27.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5ad5d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:27.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5ad5d9640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:27.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5ad5d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:27.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5a7fff640 10 monclient(hunting): get_auth_request con 0x7fd5a8133a50 auth_method 0 2024-09-11T01:38:27.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:27.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:27.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:27.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5ad5d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:27.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5ad5d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:27.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5ad5d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:27.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:27.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:27.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:27.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5acdd8640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a80a2200 0x7fd5a81cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:27.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5acdd8640 10 monclient(hunting): get_auth_request con 0x7fd5a80a2200 auth_method 0 2024-09-11T01:38:27.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5acdd8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:27.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5acdd8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:27.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5acdd8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:27.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5ad5d9640 10 monclient(hunting): handle_auth_done global_id 84323 payload 824 2024-09-11T01:38:27.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5ad5d9640 1 -- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a80a2200 msgr2=0x7fd5a81cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:27.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5ad5d9640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a80a2200 0x7fd5a81cd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:27.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5ad5d9640 1 -- 172.21.15.12:0/2874081558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5a8133a50 msgr2=0x7fd5a8132f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:27.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5ad5d9640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5a8133a50 0x7fd5a8132f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:27.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5ad5d9640 10 monclient: _finish_hunting 0 2024-09-11T01:38:27.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5ad5d9640 1 monclient: found mon.noname-c 2024-09-11T01:38:27.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5ad5d9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:27.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5ad5d9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:38:27.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.049+0000 7fd5ad5d9640 1 -- 172.21.15.12:0/2874081558 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a8127690 con 0x7fd5a8134ad0 2024-09-11T01:38:27.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.050+0000 7fd5a7fff640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5a8133a50 0x7fd5a8132f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:38:27.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.052+0000 7fd5ad5d9640 1 -- 172.21.15.12:0/2874081558 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5a8134ad0 msgr2=0x7fd5a8133480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:27.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.052+0000 7fd5ad5d9640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5a8134ad0 0x7fd5a8133480 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd5a000c7b0 tx=0x7fd5a000cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:27.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.052+0000 7fd5a57fa640 1 -- 172.21.15.12:0/2874081558 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a0016020 con 0x7fd5a8134ad0 2024-09-11T01:38:27.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.052+0000 7fd5a57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:27.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.052+0000 7fd5a57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:38:27.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.052+0000 7fd5a57fa640 10 monclient: dump: 2024-09-11T01:38:27.065 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:27.065 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:27.065 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:27.065 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:27.065 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:27.065 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:27.066 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:27.066 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:27.066 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:27.066 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:27.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.052+0000 7fd5a57fa640 10 monclient: _finish_auth 0 2024-09-11T01:38:27.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.052+0000 7fd5a57fa640 10 monclient: _check_auth_tickets 2024-09-11T01:38:27.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.052+0000 7fd5a57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:27.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.052+0000 7fd5a57fa640 1 -- 172.21.15.12:0/2874081558 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5a000ac60 con 0x7fd5a8134ad0 2024-09-11T01:38:27.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.052+0000 7fd5af03c640 5 monclient: authenticate success, global_id 84323 2024-09-11T01:38:27.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.052+0000 7fd5af03c640 10 monclient: _renew_subs 2024-09-11T01:38:27.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.052+0000 7fd5af03c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:38:27.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.052+0000 7fd5af03c640 1 -- 172.21.15.12:0/2874081558 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5a8127970 con 0x7fd5a8134ad0 2024-09-11T01:38:27.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.052+0000 7fd5a57fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:27.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.052+0000 7fd5a57fa640 1 -- 172.21.15.12:0/2874081558 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a000ce60 con 0x7fd5a8134ad0 2024-09-11T01:38:27.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.052+0000 7fd5a57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:27.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.052+0000 7fd5a57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:38:27.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.052+0000 7fd5a57fa640 10 monclient: dump: 2024-09-11T01:38:27.067 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:27.067 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:27.067 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:27.067 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:27.067 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:27.067 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:27.067 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:27.067 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:27.067 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:27.067 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:27.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.052+0000 7fd5a6ffd640 4 set_mon_vals no callback set 2024-09-11T01:38:27.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.052+0000 7fd5af03c640 10 monclient: _renew_subs 2024-09-11T01:38:27.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.052+0000 7fd5af03c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:38:27.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.052+0000 7fd5af03c640 1 -- 172.21.15.12:0/2874081558 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5a80a5510 con 0x7fd5a8134ad0 2024-09-11T01:38:27.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.053+0000 7fd5af03c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:27.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.053+0000 7fd5af03c640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 0x7fd5a8123f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:27.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.053+0000 7fd5af03c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:27.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.053+0000 7fd5af03c640 1 -- 172.21.15.12:0/2874081558 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a80a4140 con 0x7fd5a8123b50 2024-09-11T01:38:27.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.053+0000 7fd5a57fa640 1 -- 172.21.15.12:0/2874081558 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fd5a00126e0 con 0x7fd5a8134ad0 2024-09-11T01:38:27.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.053+0000 7fd5acdd8640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 0x7fd5a8123f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:27.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.053+0000 7fd5acdd8640 10 monclient: get_auth_request con 0x7fd5a8123b50 auth_method 0 2024-09-11T01:38:27.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.053+0000 7fd5acdd8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:27.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.053+0000 7fd5acdd8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:27.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.053+0000 7fd5acdd8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:27.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.053+0000 7fd5a57fa640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd58c03e210 0x7fd58c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:27.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.053+0000 7fd5acdd8640 1 -- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 msgr2=0x7fd5a8123f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:38:27.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.053+0000 7fd5acdd8640 1 -- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 msgr2=0x7fd5a8123f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:38:27.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.053+0000 7fd5acdd8640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 0x7fd5a8123f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:38:27.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.053+0000 7fd5a57fa640 1 -- 172.21.15.12:0/2874081558 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(315..315 src has 186..315) ==== 4799+0+0 (secure 0 0 0) 0x7fd5a0053290 con 0x7fd5a8134ad0 2024-09-11T01:38:27.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.053+0000 7fd5acdd8640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 0x7fd5a8123f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:38:27.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.053+0000 7fd5a7fff640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd58c03e210 0x7fd58c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:27.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.053+0000 7fd5a7fff640 10 monclient: get_auth_request con 0x7fd58c03e210 auth_method 0 2024-09-11T01:38:27.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.056+0000 7fd5a7fff640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd58c03e210 0x7fd58c0406d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd59402c790 tx=0x7fd594008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:27.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.254+0000 7fd5acdd8640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 0x7fd5a8123f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:27.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.254+0000 7fd5acdd8640 10 monclient: get_auth_request con 0x7fd5a8123b50 auth_method 0 2024-09-11T01:38:27.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.254+0000 7fd5acdd8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:27.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.254+0000 7fd5acdd8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:27.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.254+0000 7fd5acdd8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:38:27.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.254+0000 7fd5acdd8640 1 -- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 msgr2=0x7fd5a8123f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:38:27.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.254+0000 7fd5acdd8640 1 -- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 msgr2=0x7fd5a8123f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:38:27.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.254+0000 7fd5acdd8640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 0x7fd5a8123f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:38:27.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.254+0000 7fd5acdd8640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 0x7fd5a8123f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:38:27.344 INFO:tasks.rados.rados.0.smithi071.stdout:1401: done (0 left) 2024-09-11T01:38:27.344 INFO:tasks.rados.rados.0.smithi071.stdout:1402: write (excl) oid 7 current snap is 148 2024-09-11T01:38:27.344 INFO:tasks.rados.rados.0.smithi071.stdout:1402: seq_num 530 ranges {507710=710327,1897927=412519,2854892=52818} 2024-09-11T01:38:27.350 INFO:tasks.rados.rados.0.smithi071.stdout:1402: writing smithi07134777-7 from 507710 to 1218037 tid 1 2024-09-11T01:38:27.356 INFO:tasks.rados.rados.0.smithi071.stdout:1402: writing smithi07134777-7 from 1897927 to 2310446 tid 2 2024-09-11T01:38:27.360 INFO:tasks.rados.rados.0.smithi071.stdout:1402: writing smithi07134777-7 from 2854892 to 2907710 tid 3 2024-09-11T01:38:27.360 INFO:tasks.rados.rados.0.smithi071.stdout:1403: snap_remove snap 142 2024-09-11T01:38:27.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.655+0000 7fd5acdd8640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 0x7fd5a8123f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:27.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.655+0000 7fd5acdd8640 10 monclient: get_auth_request con 0x7fd5a8123b50 auth_method 0 2024-09-11T01:38:27.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.655+0000 7fd5acdd8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:27.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.655+0000 7fd5acdd8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:27.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.655+0000 7fd5acdd8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:38:27.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.655+0000 7fd5acdd8640 1 -- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 msgr2=0x7fd5a8123f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:38:27.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.655+0000 7fd5acdd8640 1 -- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 msgr2=0x7fd5a8123f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:38:27.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.655+0000 7fd5acdd8640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 0x7fd5a8123f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:38:27.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:27.655+0000 7fd5acdd8640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 0x7fd5a8123f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:38:28.351 INFO:tasks.rados.rados.0.smithi071.stdout:1403: done (1 left) 2024-09-11T01:38:28.351 INFO:tasks.rados.rados.0.smithi071.stdout:1404: read oid 23 snap -1 2024-09-11T01:38:28.351 INFO:tasks.rados.rados.0.smithi071.stdout:1404: expect (ObjNum 492 snap 132 seq_num 492) 2024-09-11T01:38:28.351 INFO:tasks.rados.rados.0.smithi071.stdout:1405: delete oid 3 current snap is 148 2024-09-11T01:38:28.351 INFO:tasks.rados.rados.0.smithi071.stdout:1402: finishing write tid 1 to smithi07134777-7 2024-09-11T01:38:28.351 INFO:tasks.rados.rados.0.smithi071.stdout:1402: finishing write tid 2 to smithi07134777-7 2024-09-11T01:38:28.351 INFO:tasks.rados.rados.0.smithi071.stdout:1402: finishing write tid 3 to smithi07134777-7 2024-09-11T01:38:28.351 INFO:tasks.rados.rados.0.smithi071.stdout:1402: finishing write tid 4 to smithi07134777-7 2024-09-11T01:38:28.351 INFO:tasks.rados.rados.0.smithi071.stdout:1402: finishing write tid 5 to smithi07134777-7 2024-09-11T01:38:28.351 INFO:tasks.rados.rados.0.smithi071.stdout:1402: oid 7 updating version 0 to 725 2024-09-11T01:38:28.352 INFO:tasks.rados.rados.0.smithi071.stdout:1402: oid 7 updating version 725 to 727 2024-09-11T01:38:28.352 INFO:tasks.rados.rados.0.smithi071.stdout:1402: oid 7 version 727 is already newer than 726 2024-09-11T01:38:28.352 INFO:tasks.rados.rados.0.smithi071.stdout:1402: oid 7 version 727 is already newer than 720 2024-09-11T01:38:28.352 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 727 (ObjNum 530 snap 148 seq_num 530) dirty exists 2024-09-11T01:38:28.352 INFO:tasks.rados.rados.0.smithi071.stdout:1402: left oid 7 (ObjNum 530 snap 148 seq_num 530) 2024-09-11T01:38:28.359 INFO:tasks.rados.rados.0.smithi071.stdout:1402: done (2 left) 2024-09-11T01:38:28.359 INFO:tasks.rados.rados.0.smithi071.stdout:1405: done (1 left) 2024-09-11T01:38:28.359 INFO:tasks.rados.rados.0.smithi071.stdout:1406: rollback oid 30 current snap is 148 2024-09-11T01:38:28.359 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 30 to 145 2024-09-11T01:38:28.359 INFO:tasks.rados.rados.0.smithi071.stdout:1407: snap_remove snap 147 2024-09-11T01:38:28.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:28.457+0000 7fd5acdd8640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 0x7fd5a8123f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:28.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:28.457+0000 7fd5acdd8640 10 monclient: get_auth_request con 0x7fd5a8123b50 auth_method 0 2024-09-11T01:38:28.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:28.457+0000 7fd5acdd8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:28.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:28.457+0000 7fd5acdd8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:28.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:28.457+0000 7fd5acdd8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:38:28.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:28.457+0000 7fd5acdd8640 1 -- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 msgr2=0x7fd5a8123f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:38:28.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:28.457+0000 7fd5acdd8640 1 -- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 msgr2=0x7fd5a8123f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:38:28.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:28.457+0000 7fd5acdd8640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 0x7fd5a8123f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:38:28.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:28.457+0000 7fd5acdd8640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 0x7fd5a8123f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:38:28.925 INFO:tasks.rados.rados.0.smithi071.stdout:1406: finishing rollback tid 1 to smithi07134777-30 2024-09-11T01:38:28.925 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 0 (ObjNum 97 snap 0 seq_num 3921755115) dirty dne 2024-09-11T01:38:28.925 INFO:tasks.rados.rados.0.smithi071.stdout:1404: expect (ObjNum 492 snap 132 seq_num 492) 2024-09-11T01:38:28.925 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:38:28.952 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:38:28.978 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:38:29.003 INFO:tasks.rados.rados.0.smithi071.stdout:1404: done (2 left) 2024-09-11T01:38:29.003 INFO:tasks.rados.rados.0.smithi071.stdout:1406: done (1 left) 2024-09-11T01:38:29.003 INFO:tasks.rados.rados.0.smithi071.stdout:1407: done (0 left) 2024-09-11T01:38:29.003 INFO:tasks.rados.rados.0.smithi071.stdout:1408: delete oid 47 current snap is 148 2024-09-11T01:38:29.006 INFO:tasks.rados.rados.0.smithi071.stdout:1408: done (0 left) 2024-09-11T01:38:29.006 INFO:tasks.rados.rados.0.smithi071.stdout:1409: read oid 14 snap -1 2024-09-11T01:38:29.006 INFO:tasks.rados.rados.0.smithi071.stdout:1409: expect (ObjNum 522 snap 144 seq_num 522) 2024-09-11T01:38:29.006 INFO:tasks.rados.rados.0.smithi071.stdout:1410: snap_remove snap 144 2024-09-11T01:38:29.935 INFO:tasks.rados.rados.0.smithi071.stdout:1409: expect (ObjNum 522 snap 144 seq_num 522) 2024-09-11T01:38:30.029 INFO:tasks.rados.rados.0.smithi071.stdout:1409: done (1 left) 2024-09-11T01:38:30.029 INFO:tasks.rados.rados.0.smithi071.stdout:1410: done (0 left) 2024-09-11T01:38:30.029 INFO:tasks.rados.rados.0.smithi071.stdout:1411: rollback oid 42 current snap is 148 2024-09-11T01:38:30.029 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 42 to 145 2024-09-11T01:38:30.029 INFO:tasks.rados.rados.0.smithi071.stdout:1412: copy_from oid 22 from oid 2 current snap is 148 2024-09-11T01:38:30.029 INFO:tasks.rados.rados.0.smithi071.stdout:1413: delete oid 39 current snap is 148 2024-09-11T01:38:30.032 INFO:tasks.rados.rados.0.smithi071.stdout:1413: done (2 left) 2024-09-11T01:38:30.033 INFO:tasks.rados.rados.0.smithi071.stdout:1414: write oid 43 current snap is 148 2024-09-11T01:38:30.033 INFO:tasks.rados.rados.0.smithi071.stdout:1414: seq_num 534 ranges {574938=727676,1731855=703504,2848231=126707} 2024-09-11T01:38:30.040 INFO:tasks.rados.rados.0.smithi071.stdout:1414: writing smithi07134777-43 from 574938 to 1302614 tid 1 2024-09-11T01:38:30.046 INFO:tasks.rados.rados.0.smithi071.stdout:1414: writing smithi07134777-43 from 1731855 to 2435359 tid 2 2024-09-11T01:38:30.049 INFO:tasks.rados.rados.0.smithi071.stdout:1414: writing smithi07134777-43 from 2848231 to 2974938 tid 3 2024-09-11T01:38:30.049 INFO:tasks.rados.rados.0.smithi071.stdout:1415: snap_remove snap 143 2024-09-11T01:38:30.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.049+0000 7fd593fff640 10 monclient: tick 2024-09-11T01:38:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.049+0000 7fd593fff640 10 monclient: _check_auth_tickets 2024-09-11T01:38:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.049+0000 7fd593fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.049+0000 7fd593fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.059+0000 7fd5acdd8640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 0x7fd5a8123f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.059+0000 7fd5acdd8640 10 monclient: get_auth_request con 0x7fd5a8123b50 auth_method 0 2024-09-11T01:38:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.059+0000 7fd5acdd8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.059+0000 7fd5acdd8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.059+0000 7fd5acdd8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:38:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.059+0000 7fd5acdd8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.059+0000 7fd5acdd8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.059+0000 7fd5acdd8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.060+0000 7fd5acdd8640 10 monclient(hunting): handle_auth_done global_id 104098 payload 824 2024-09-11T01:38:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.060+0000 7fd5acdd8640 1 -- 172.21.15.12:0/2874081558 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd5a80699d0 con 0x7fd5a8123b50 2024-09-11T01:38:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.062+0000 7fd5acdd8640 1 -- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 msgr2=0x7fd5a8123f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.062+0000 7fd5acdd8640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 0x7fd5a8123f30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd59c017980 tx=0x7fd59c017e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.062+0000 7fd5a57fa640 1 -- 172.21.15.12:0/2874081558 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59c004450 con 0x7fd5a8123b50 2024-09-11T01:38:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.062+0000 7fd5a57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.062+0000 7fd5a57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:38:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.062+0000 7fd5a57fa640 10 monclient: dump: 2024-09-11T01:38:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:30.065 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:30.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.063+0000 7fd5a57fa640 1 -- 172.21.15.12:0/2874081558 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd59c01c070 con 0x7fd5a8123b50 2024-09-11T01:38:30.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.063+0000 7fd5a57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:30.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.063+0000 7fd5a57fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:38:30.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.063+0000 7fd5a57fa640 1 -- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 msgr2=0x7fd5a8123f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:30.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.063+0000 7fd5a57fa640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 0x7fd5a8123f30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd59c017980 tx=0x7fd59c017e40 comp rx=0 tx=0).stop 2024-09-11T01:38:30.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.063+0000 7fd5a57fa640 1 -- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 msgr2=0x7fd5a8123f30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:38:30.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.063+0000 7fd5a57fa640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8123b50 0x7fd5a8123f30 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd59c017980 tx=0x7fd59c017e40 comp rx=0 tx=0).stop 2024-09-11T01:38:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.070+0000 7fd5af03c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:38:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.070+0000 7fd5af03c640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8124470 0x7fd5a80726b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.070+0000 7fd5af03c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.070+0000 7fd5af03c640 1 -- 172.21.15.12:0/2874081558 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a80a16f0 con 0x7fd5a8124470 2024-09-11T01:38:30.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.070+0000 7fd5acdd8640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8124470 0x7fd5a80726b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:30.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.070+0000 7fd5acdd8640 10 monclient: get_auth_request con 0x7fd5a8124470 auth_method 0 2024-09-11T01:38:30.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.070+0000 7fd5acdd8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:30.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.070+0000 7fd5acdd8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:30.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.070+0000 7fd5acdd8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:30.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.070+0000 7fd5acdd8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:30.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.070+0000 7fd5acdd8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:30.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.070+0000 7fd5acdd8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:30.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.070+0000 7fd5acdd8640 10 monclient(hunting): handle_auth_done global_id 104101 payload 824 2024-09-11T01:38:30.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.070+0000 7fd5acdd8640 1 -- 172.21.15.12:0/2874081558 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd5a8133ee0 con 0x7fd5a8124470 2024-09-11T01:38:30.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.073+0000 7fd5acdd8640 1 -- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8124470 msgr2=0x7fd5a80726b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:30.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.073+0000 7fd5acdd8640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8124470 0x7fd5a80726b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd59c017980 tx=0x7fd59c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:30.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.073+0000 7fd5a57fa640 1 -- 172.21.15.12:0/2874081558 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59c003a20 con 0x7fd5a8124470 2024-09-11T01:38:30.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.073+0000 7fd5a57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:30.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.073+0000 7fd5a57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:38:30.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.073+0000 7fd5a57fa640 10 monclient: dump: 2024-09-11T01:38:30.075 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:30.075 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:30.075 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:30.075 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:30.075 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:30.075 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:30.076 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:30.076 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:30.076 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:30.076 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:30.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.073+0000 7fd5a57fa640 1 -- 172.21.15.12:0/2874081558 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4540 (secure 0 0 0) 0x7fd59c003a20 con 0x7fd5a8124470 2024-09-11T01:38:30.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.073+0000 7fd5a57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:38:30.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.073+0000 7fd5a57fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:38:30.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.073+0000 7fd5a57fa640 1 -- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8124470 msgr2=0x7fd5a80726b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:30.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.073+0000 7fd5a57fa640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8124470 0x7fd5a80726b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd59c017980 tx=0x7fd59c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:38:30.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.073+0000 7fd5a57fa640 1 -- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8124470 msgr2=0x7fd5a80726b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:38:30.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.073+0000 7fd5a57fa640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5a8124470 0x7fd5a80726b0 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd59c017980 tx=0x7fd59c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:38:30.077 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:38:30.077 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:38:30.077 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:38:30.077 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:38:30.077 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 92, 2024-09-11T01:38:30.077 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 10568, 2024-09-11T01:38:30.077 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:38:30.077 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:38:30.077 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:38:30.077 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:38:30.077 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:30.078 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 5, 2024-09-11T01:38:30.078 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:38:30.078 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:38:30.078 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:38:30.078 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:38:30.078 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:38:30.078 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:38:30.078 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:38:30.078 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:38:30.078 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:38:30.078 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:38:30.078 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:38:30.078 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:38:30.079 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:38:30.079 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:38:30.079 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:30.079 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:38:30.079 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:38:30.079 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:38:30.079 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:38:30.079 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:38:30.079 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:38:30.079 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:38:30.079 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:38:30.079 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:38:30.079 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:38:30.080 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:38:30.080 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:38:30.080 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:38:30.080 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:30.080 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:30.080 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:38:30.080 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:38:30.080 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:38:30.080 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:38:30.080 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:38:30.080 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:38:30.080 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:38:30.080 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:38:30.081 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:38:30.081 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:38:30.081 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:38:30.081 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:38:30.081 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:38:30.081 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:38:30.081 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:38:30.081 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:38:30.081 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:38:30.081 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:38:30.081 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:38:30.081 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:38:30.081 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:38:30.081 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:38:30.082 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:38:30.082 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:38:30.082 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:38:30.082 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:38:30.082 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:38:30.082 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:38:30.082 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:30.082 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:38:30.082 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:30.082 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:38:30.082 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:30.083 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:38:30.083 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:38:30.083 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:38:30.083 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:38:30.083 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:30.083 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:38:30.083 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:38:30.083 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:30.083 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:30.083 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:30.083 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:38:30.083 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:38:30.083 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:30.084 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:30.084 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:30.084 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:30.084 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:38:30.084 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:38:30.084 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:38:30.084 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:38:30.084 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:38:30.084 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:30.084 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:30.084 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:38:30.084 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:38:30.084 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:38:30.085 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:38:30.085 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:30.085 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:38:30.085 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:38:30.085 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:30.085 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:30.085 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:30.085 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:38:30.085 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:38:30.085 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:30.085 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:30.085 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:30.085 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:38:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:38:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:38:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:38:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:38:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:38:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:38:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:38:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:38:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:38:30.087 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:38:30.087 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:30.087 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:30.087 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:30.087 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:38:30.087 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:38:30.087 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:30.087 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:30.087 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:30.087 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:30.087 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:38:30.087 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:38:30.087 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:38:30.087 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:38:30.088 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:38:30.088 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:30.088 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:30.088 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:30.088 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:38:30.088 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:38:30.088 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:30.088 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:38:30.088 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:38:30.088 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:38:30.088 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:30.088 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:30.088 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:38:30.089 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:30.089 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:38:30.089 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:38:30.089 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:38:30.089 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:30.089 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:30.089 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:30.089 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:38:30.089 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:38:30.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.074+0000 7fd5af03c640 1 -- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd58c03e210 msgr2=0x7fd58c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:30.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.074+0000 7fd5af03c640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd58c03e210 0x7fd58c0406d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd59402c790 tx=0x7fd594008040 comp rx=0 tx=0).stop 2024-09-11T01:38:30.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.074+0000 7fd5af03c640 10 monclient: shutdown 2024-09-11T01:38:30.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.074+0000 7fd5af03c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:30.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.074+0000 7fd5af03c640 1 -- 172.21.15.12:0/2874081558 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5a8134ad0 msgr2=0x7fd5a8133480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:30.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.074+0000 7fd5af03c640 1 --2- 172.21.15.12:0/2874081558 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5a8134ad0 0x7fd5a8133480 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd5a000c7b0 tx=0x7fd5a000cc70 comp rx=0 tx=0).stop 2024-09-11T01:38:30.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.074+0000 7fd5ad5d9640 1 -- 172.21.15.12:0/2874081558 reap_dead start 2024-09-11T01:38:30.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.074+0000 7fd5ad5d9640 1 -- 172.21.15.12:0/2874081558 reap_dead start 2024-09-11T01:38:30.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.074+0000 7fd5af03c640 1 -- 172.21.15.12:0/2874081558 shutdown_connections 2024-09-11T01:38:30.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.074+0000 7fd5af03c640 1 -- 172.21.15.12:0/2874081558 >> 172.21.15.12:0/2874081558 conn(0x7fd5a80a2f70 msgr2=0x7fd5a809d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:30.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.074+0000 7fd5af03c640 1 -- 172.21.15.12:0/2874081558 shutdown_connections 2024-09-11T01:38:30.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.074+0000 7fd5af03c640 1 -- 172.21.15.12:0/2874081558 wait complete. 2024-09-11T01:38:30.092 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:38:30.092 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:38:30.092 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:38:30.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.198+0000 7fceb48b0640 10 monclient: get_monmap_and_config 2024-09-11T01:38:30.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.198+0000 7fceb48b0640 10 monclient: build_initial_monmap 2024-09-11T01:38:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.198+0000 7fceb48b0640 10 monclient: monmap: 2024-09-11T01:38:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:38:30.199064+0000 2024-09-11T01:38:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:38:30.199064+0000 2024-09-11T01:38:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:38:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:38:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:38:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:38:30.200 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:38:30.201 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:30.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.198+0000 7fceb48b0640 1 Processor -- start 2024-09-11T01:38:30.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.199+0000 7fceb48b0640 1 -- start start 2024-09-11T01:38:30.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.199+0000 7fceb48b0640 10 monclient: init 2024-09-11T01:38:30.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.199+0000 7fceb48b0640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:30.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.199+0000 7fceb48b0640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-11T01:38:30.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.199+0000 7fceb48b0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fceb00a2e20 addr v1:172.21.15.12:6789/0 2024-09-11T01:38:30.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.199+0000 7fceb48b0640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fceb00a4970 0x7fceb006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:30.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.199+0000 7fceb48b0640 10 monclient(hunting): picked mon.noname-c con 0x7fceb00a4970 addr v2:172.21.15.71:3300/0 2024-09-11T01:38:30.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.199+0000 7fceb48b0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb006f910 0x7fceb013f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:30.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.199+0000 7fceb48b0640 10 monclient(hunting): picked mon.noname-b con 0x7fceb006f910 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:30.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.199+0000 7fceb48b0640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fceb006f4f0 con 0x7fceb00a2e20 2024-09-11T01:38:30.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.199+0000 7fceb48b0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:30.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.199+0000 7fceb48b0640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fceb006ef30 con 0x7fceb00a4970 2024-09-11T01:38:30.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.199+0000 7fceb48b0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:30.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.199+0000 7fceb48b0640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fceb006ec50 con 0x7fceb006f910 2024-09-11T01:38:30.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.199+0000 7fceb48b0640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:30.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.199+0000 7fceb48b0640 10 monclient(hunting): authenticate will time out at 1316.641733s 2024-09-11T01:38:30.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.199+0000 7fceaed9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb006f910 0x7fceb013f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:30.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.199+0000 7fceae59c640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fceb00a2e20 0x7fceb00a3200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:51916/0 (socket says 172.21.15.12:51916) 2024-09-11T01:38:30.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.199+0000 7fceae59c640 1 -- 172.21.15.12:0/2169313732 learned_addr learned my addr 172.21.15.12:0/2169313732 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:38:30.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.199+0000 7fceaed9d640 10 monclient(hunting): get_auth_request con 0x7fceb006f910 auth_method 0 2024-09-11T01:38:30.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.199+0000 7fceaed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:30.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.199+0000 7fceaed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:30.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.199+0000 7fceaed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:30.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.199+0000 7fceaed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:30.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.199+0000 7fceaed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:30.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.200+0000 7fceaed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:30.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.200+0000 7fceadd9b640 1 --2- 172.21.15.12:0/2169313732 >> v2:172.21.15.71:3300/0 conn(0x7fceb00a4970 0x7fceb006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:30.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.200+0000 7fceadd9b640 10 monclient(hunting): get_auth_request con 0x7fceb00a4970 auth_method 0 2024-09-11T01:38:30.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.200+0000 7fceadd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:30.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.200+0000 7fceadd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:30.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.200+0000 7fceadd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:30.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.200+0000 7fceaed9d640 10 monclient(hunting): handle_auth_done global_id 84326 payload 274 2024-09-11T01:38:30.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.200+0000 7fceaed9d640 1 -- 172.21.15.12:0/2169313732 >> v2:172.21.15.71:3300/0 conn(0x7fceb00a4970 msgr2=0x7fceb006ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:38:30.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.200+0000 7fceaed9d640 1 --2- 172.21.15.12:0/2169313732 >> v2:172.21.15.71:3300/0 conn(0x7fceb00a4970 0x7fceb006ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:30.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.200+0000 7fceaed9d640 1 -- 172.21.15.12:0/2169313732 >> v1:172.21.15.12:6789/0 conn(0x7fceb00a2e20 legacy=0x7fceb00a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:38:30.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.200+0000 7fceaed9d640 10 monclient: _finish_hunting 0 2024-09-11T01:38:30.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.200+0000 7fceaed9d640 1 monclient: found mon.noname-b 2024-09-11T01:38:30.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.200+0000 7fceaed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:30.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.200+0000 7fceaed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:38:30.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.200+0000 7fceaed9d640 1 -- 172.21.15.12:0/2169313732 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceb00a3910 con 0x7fceb006f910 2024-09-11T01:38:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.202+0000 7fceaed9d640 1 -- 172.21.15.12:0/2169313732 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb006f910 msgr2=0x7fceb013f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.202+0000 7fceaed9d640 1 --2- 172.21.15.12:0/2169313732 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb006f910 0x7fceb013f430 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcea8009d90 tx=0x7fcea802a490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e499caba47358463 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fcead59a640 1 -- 172.21.15.12:0/2169313732 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea8007950 con 0x7fceb006f910 2024-09-11T01:38:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fcead59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fcead59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:38:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fcead59a640 10 monclient: dump: 2024-09-11T01:38:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:30.212 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:30.213 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fcead59a640 10 monclient: _finish_auth 0 2024-09-11T01:38:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fcead59a640 10 monclient: _check_auth_tickets 2024-09-11T01:38:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fcead59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fceb48b0640 5 monclient: authenticate success, global_id 84326 2024-09-11T01:38:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fceb48b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:38:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fcead59a640 1 -- 172.21.15.12:0/2169313732 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcea8007af0 con 0x7fceb006f910 2024-09-11T01:38:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fcead59a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fceb48b0640 10 monclient: get_monmap_and_config success 2024-09-11T01:38:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fceb48b0640 4 set_mon_vals no callback set 2024-09-11T01:38:30.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fcead59a640 1 -- 172.21.15.12:0/2169313732 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea8031400 con 0x7fceb006f910 2024-09-11T01:38:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fcead59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fcead59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:38:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fcead59a640 10 monclient: dump: 2024-09-11T01:38:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:30.214 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fceb48b0640 10 monclient: shutdown 2024-09-11T01:38:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fceb48b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fceb48b0640 1 -- 172.21.15.12:0/2169313732 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb006f910 msgr2=0x7fceb013f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fceb48b0640 1 --2- 172.21.15.12:0/2169313732 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb006f910 0x7fceb013f430 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcea8009d90 tx=0x7fcea802a490 comp rx=0 tx=0).stop 2024-09-11T01:38:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fceb48b0640 1 -- 172.21.15.12:0/2169313732 shutdown_connections 2024-09-11T01:38:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fceb48b0640 1 --2- 172.21.15.12:0/2169313732 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb006f910 0x7fceb013f430 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fceb48b0640 1 --2- 172.21.15.12:0/2169313732 >> v2:172.21.15.71:3300/0 conn(0x7fceb00a4970 0x7fceb006ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fceb48b0640 1 -- 172.21.15.12:0/2169313732 >> 172.21.15.12:0/2169313732 conn(0x7fceb00a8600 msgr2=0x7fceb00a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fceb48b0640 1 -- 172.21.15.12:0/2169313732 shutdown_connections 2024-09-11T01:38:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.203+0000 7fceb48b0640 1 -- 172.21.15.12:0/2169313732 wait complete. 2024-09-11T01:38:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.204+0000 7fceb48b0640 10 monclient: build_initial_monmap 2024-09-11T01:38:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.204+0000 7fceb48b0640 10 monclient: monmap: 2024-09-11T01:38:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:38:30.216 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:38:30.216 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:38:30.216 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:30.216 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:30.216 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:38:30.216 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:38:30.216 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:38:30.216 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:30.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.204+0000 7fceb48b0640 1 Processor -- start 2024-09-11T01:38:30.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.204+0000 7fceb48b0640 1 -- start start 2024-09-11T01:38:30.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceb48b0640 10 monclient: init 2024-09-11T01:38:30.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceb48b0640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:30.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceb48b0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:38:30.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceb48b0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb006f910 0x7fceb01cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceb48b0640 10 monclient(hunting): picked mon.noname-c con 0x7fceb006f910 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceb48b0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fceb00a2e20 0x7fceb01cd920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceb48b0640 10 monclient(hunting): picked mon.noname-b con 0x7fceb00a2e20 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:38:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceb48b0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceb00a4970 0x7fceb0139400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceb48b0640 10 monclient(hunting): picked mon.noname-a con 0x7fceb00a4970 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:38:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceb48b0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceb48b0640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fceb013fd00 con 0x7fceb00a4970 2024-09-11T01:38:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceb48b0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceb48b0640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fceb013fb80 con 0x7fceb00a2e20 2024-09-11T01:38:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceb48b0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceb48b0640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fceb013fe80 con 0x7fceb006f910 2024-09-11T01:38:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceb48b0640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:30.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceb48b0640 10 monclient(hunting): authenticate will time out at 1316.647465s 2024-09-11T01:38:30.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceae59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb006f910 0x7fceb01cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:30.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceae59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb006f910 0x7fceb01cd3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:50938/0 (socket says 172.21.15.12:50938) 2024-09-11T01:38:30.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceae59c640 1 -- 172.21.15.12:0/1833772246 learned_addr learned my addr 172.21.15.12:0/1833772246 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:38:30.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceae59c640 10 monclient(hunting): get_auth_request con 0x7fceb006f910 auth_method 0 2024-09-11T01:38:30.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:30.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceae59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:30.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:30.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceaed9d640 1 --2- 172.21.15.12:0/1833772246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceb00a4970 0x7fceb0139400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:30.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceaed9d640 10 monclient(hunting): get_auth_request con 0x7fceb00a4970 auth_method 0 2024-09-11T01:38:30.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceaed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:30.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceaed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:30.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceaed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:30.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceadd9b640 1 --2- 172.21.15.12:0/1833772246 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fceb00a2e20 0x7fceb01cd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:30.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceadd9b640 10 monclient(hunting): get_auth_request con 0x7fceb00a2e20 auth_method 0 2024-09-11T01:38:30.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceadd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:30.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceadd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:30.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceadd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:30.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:30.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:30.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:30.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceaed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:30.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceaed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:30.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceaed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:30.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceadd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:30.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceadd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:30.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceadd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:30.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceae59c640 10 monclient(hunting): handle_auth_done global_id 84329 payload 824 2024-09-11T01:38:30.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceae59c640 1 -- 172.21.15.12:0/1833772246 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fceb00a2e20 msgr2=0x7fceb01cd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceae59c640 1 --2- 172.21.15.12:0/1833772246 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fceb00a2e20 0x7fceb01cd920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceae59c640 1 -- 172.21.15.12:0/1833772246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceb00a4970 msgr2=0x7fceb0139400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceae59c640 1 --2- 172.21.15.12:0/1833772246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceb00a4970 0x7fceb0139400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceae59c640 10 monclient: _finish_hunting 0 2024-09-11T01:38:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceae59c640 1 monclient: found mon.noname-c 2024-09-11T01:38:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceae59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceae59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:38:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.205+0000 7fceae59c640 1 -- 172.21.15.12:0/1833772246 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceb013bed0 con 0x7fceb006f910 2024-09-11T01:38:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.208+0000 7fceae59c640 1 -- 172.21.15.12:0/1833772246 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb006f910 msgr2=0x7fceb01cd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.208+0000 7fceae59c640 1 --2- 172.21.15.12:0/1833772246 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb006f910 0x7fceb01cd3e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcea000c460 tx=0x7fcea000c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.208+0000 7fcea6ffd640 1 -- 172.21.15.12:0/1833772246 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea0007070 con 0x7fceb006f910 2024-09-11T01:38:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.208+0000 7fcea6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.208+0000 7fcea6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:38:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.208+0000 7fcea6ffd640 10 monclient: dump: 2024-09-11T01:38:30.221 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:30.221 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:30.221 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:30.221 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:30.221 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:30.221 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:30.221 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:30.221 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:30.221 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:30.221 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:30.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.208+0000 7fcea6ffd640 10 monclient: _finish_auth 0 2024-09-11T01:38:30.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.208+0000 7fcea6ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:38:30.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.208+0000 7fcea6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.208+0000 7fceb48b0640 5 monclient: authenticate success, global_id 84329 2024-09-11T01:38:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.208+0000 7fcea6ffd640 1 -- 172.21.15.12:0/1833772246 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcea0007230 con 0x7fceb006f910 2024-09-11T01:38:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.208+0000 7fceb48b0640 10 monclient: _renew_subs 2024-09-11T01:38:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.208+0000 7fceb48b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:38:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.208+0000 7fceb48b0640 1 -- 172.21.15.12:0/1833772246 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fceb01cde60 con 0x7fceb006f910 2024-09-11T01:38:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.208+0000 7fcea6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.208+0000 7fcea6ffd640 1 -- 172.21.15.12:0/1833772246 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea0007620 con 0x7fceb006f910 2024-09-11T01:38:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.208+0000 7fcea6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.208+0000 7fcea6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:38:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.208+0000 7fcea6ffd640 10 monclient: dump: 2024-09-11T01:38:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:30.223 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.208+0000 7fceb48b0640 10 monclient: _renew_subs 2024-09-11T01:38:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.208+0000 7fceacd99640 4 set_mon_vals no callback set 2024-09-11T01:38:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.208+0000 7fceb48b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:38:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.208+0000 7fceb48b0640 1 -- 172.21.15.12:0/1833772246 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fceb00a6590 con 0x7fceb006f910 2024-09-11T01:38:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.209+0000 7fcea6ffd640 1 -- 172.21.15.12:0/1833772246 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fcea001b400 con 0x7fceb006f910 2024-09-11T01:38:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.209+0000 7fceb48b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:30.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.209+0000 7fceb48b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:30.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.209+0000 7fceb48b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:38:30.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.209+0000 7fceb48b0640 1 -- 172.21.15.12:0/1833772246 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fceb006ded0 con 0x7fceb006f910 2024-09-11T01:38:30.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.209+0000 7fcea6ffd640 1 --2- 172.21.15.12:0/1833772246 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fce9003e240 0x7fce90040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:30.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.209+0000 7fcea6ffd640 1 -- 172.21.15.12:0/1833772246 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(319..319 src has 186..319) ==== 4799+0+0 (secure 0 0 0) 0x7fcea00529e0 con 0x7fceb006f910 2024-09-11T01:38:30.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.209+0000 7fceadd9b640 1 --2- 172.21.15.12:0/1833772246 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fce9003e240 0x7fce90040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:30.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.210+0000 7fceadd9b640 10 monclient: get_auth_request con 0x7fce9003e240 auth_method 0 2024-09-11T01:38:30.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.212+0000 7fceadd9b640 1 --2- 172.21.15.12:0/1833772246 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fce9003e240 0x7fce90040700 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fceb0074830 tx=0x7fce98008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:30.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.214+0000 7fcea6ffd640 1 -- 172.21.15.12:0/1833772246 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fcea00513d0 con 0x7fceb006f910 2024-09-11T01:38:30.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.214+0000 7fcea6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:30.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.214+0000 7fcea6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:38:30.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.469+0000 7fceb48b0640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:38:30.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.469+0000 7fceb48b0640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:38:30.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.469+0000 7fceb48b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:38:30.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.469+0000 7fceb48b0640 1 -- 172.21.15.12:0/1833772246 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fceb0069950 con 0x7fceb006f910 2024-09-11T01:38:30.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.471+0000 7fcea6ffd640 1 -- 172.21.15.12:0/1833772246 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fcea0010d10 con 0x7fceb006f910 2024-09-11T01:38:30.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.471+0000 7fcea6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:38:30.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.471+0000 7fcea6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:38:30.473 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.471+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:30.473 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.471+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=12,logm=58} crc {auth=82590416,config=2586702516,health=1767452812,kv=3263697420,logm=1639492890}) 2024-09-11T01:38:30.473 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.471+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=18,config=2,health=12,kv=12,logm=56} crc {auth=3751219687,config=2586702516,health=1970285960,kv=2313072560,logm=4029860190}) 2024-09-11T01:38:30.473 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.471+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:30.473 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.471+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=12,logm=58} crc {auth=82590416,config=2586702516,health=1767452812,kv=3263697420,logm=1639492890}) 2024-09-11T01:38:30.473 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.471+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=17,config=2,health=12,kv=11,logm=58} crc {auth=3594262328,config=2586702516,health=1970285960,kv=54124508,logm=254288234}) 2024-09-11T01:38:30.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.472+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:30.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.472+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1992503536}) 2024-09-11T01:38:30.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.472+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2744436692}) 2024-09-11T01:38:30.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.472+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:30.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.472+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1992503536}) 2024-09-11T01:38:30.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.472+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2021179477}) 2024-09-11T01:38:30.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.473+0000 7fceb48b0640 1 -- 172.21.15.12:0/1833772246 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fce9003e240 msgr2=0x7fce90040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:30.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.473+0000 7fceb48b0640 1 --2- 172.21.15.12:0/1833772246 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fce9003e240 0x7fce90040700 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fceb0074830 tx=0x7fce98008040 comp rx=0 tx=0).stop 2024-09-11T01:38:30.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.473+0000 7fceb48b0640 10 monclient: shutdown 2024-09-11T01:38:30.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.473+0000 7fceb48b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:30.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.473+0000 7fceb48b0640 1 -- 172.21.15.12:0/1833772246 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb006f910 msgr2=0x7fceb01cd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:30.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.473+0000 7fceb48b0640 1 --2- 172.21.15.12:0/1833772246 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb006f910 0x7fceb01cd3e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcea000c460 tx=0x7fcea000c920 comp rx=0 tx=0).stop 2024-09-11T01:38:30.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.473+0000 7fceb48b0640 1 -- 172.21.15.12:0/1833772246 shutdown_connections 2024-09-11T01:38:30.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.473+0000 7fceb48b0640 1 --2- 172.21.15.12:0/1833772246 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fce9003e240 0x7fce90040700 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:30.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.473+0000 7fceb48b0640 1 --2- 172.21.15.12:0/1833772246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceb00a4970 0x7fceb0139400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:30.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.473+0000 7fceb48b0640 1 --2- 172.21.15.12:0/1833772246 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fceb00a2e20 0x7fceb01cd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:30.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.473+0000 7fceb48b0640 1 --2- 172.21.15.12:0/1833772246 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb006f910 0x7fceb01cd3e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:30.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.473+0000 7fceb48b0640 1 -- 172.21.15.12:0/1833772246 >> 172.21.15.12:0/1833772246 conn(0x7fceb00a8600 msgr2=0x7fceb00a1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:30.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.473+0000 7fceb48b0640 1 -- 172.21.15.12:0/1833772246 shutdown_connections 2024-09-11T01:38:30.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:30.473+0000 7fceb48b0640 1 -- 172.21.15.12:0/1833772246 wait complete. 2024-09-11T01:38:30.485 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:38:30.500 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.499+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:30.501 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.499+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2095328090}) 2024-09-11T01:38:30.501 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.499+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1446281422}) 2024-09-11T01:38:30.501 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.499+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:30.501 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.499+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2095328090}) 2024-09-11T01:38:30.501 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.499+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1552734012}) 2024-09-11T01:38:30.516 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.512+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:30.516 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.512+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3989807961}) 2024-09-11T01:38:30.517 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.513+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3527556023}) 2024-09-11T01:38:30.517 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.513+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:30.517 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.513+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3989807961}) 2024-09-11T01:38:30.517 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.513+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4078707179}) 2024-09-11T01:38:30.519 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.517+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:30.519 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.517+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1449881116}) 2024-09-11T01:38:30.519 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.517+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2490959937}) 2024-09-11T01:38:30.519 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.517+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:30.519 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.518+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1449881116}) 2024-09-11T01:38:30.519 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.518+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=96828243}) 2024-09-11T01:38:30.934 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.932+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:30.934 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.932+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3198647422}) 2024-09-11T01:38:30.934 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.932+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2293952767}) 2024-09-11T01:38:30.934 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.932+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:30.934 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.932+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3198647422}) 2024-09-11T01:38:30.934 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:30.932+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2336120061}) 2024-09-11T01:38:31.334 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.332+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.334 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.332+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1316170337}) 2024-09-11T01:38:31.334 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.332+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3698999576}) 2024-09-11T01:38:31.334 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.332+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.334 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.332+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1316170337}) 2024-09-11T01:38:31.334 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.332+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1327351050}) 2024-09-11T01:38:31.335 INFO:tasks.rados.rados.0.smithi071.stdout:1415: done (3 left) 2024-09-11T01:38:31.335 INFO:tasks.rados.rados.0.smithi071.stdout:1416: snap_remove snap 146 2024-09-11T01:38:31.486 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:38:31.486 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:38:31.501 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.500+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.501 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.500+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3197065192}) 2024-09-11T01:38:31.501 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.500+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2001027188}) 2024-09-11T01:38:31.501 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.500+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.502 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.500+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3197065192}) 2024-09-11T01:38:31.502 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.500+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=607312763}) 2024-09-11T01:38:31.517 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.515+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.517 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.515+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3242301}) 2024-09-11T01:38:31.517 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.515+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2097393390}) 2024-09-11T01:38:31.517 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.515+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.517 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.515+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3242301}) 2024-09-11T01:38:31.517 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.515+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2383720615}) 2024-09-11T01:38:31.522 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.521+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.522 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.521+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2963273228}) 2024-09-11T01:38:31.522 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.521+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3109692127}) 2024-09-11T01:38:31.522 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.521+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.522 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.521+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2963273228}) 2024-09-11T01:38:31.523 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.521+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=192430026}) 2024-09-11T01:38:31.576 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.575+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.575+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4134194693}) 2024-09-11T01:38:31.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.575+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4192662501}) 2024-09-11T01:38:31.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.575+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.575+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4134194693}) 2024-09-11T01:38:31.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.575+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4199486474}) 2024-09-11T01:38:31.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.576+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.576+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183582230}) 2024-09-11T01:38:31.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.576+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4090251877}) 2024-09-11T01:38:31.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.576+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.576+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183582230}) 2024-09-11T01:38:31.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.576+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2483806570}) 2024-09-11T01:38:31.579 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.577+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.579 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.577+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2916243677}) 2024-09-11T01:38:31.579 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.577+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2261538955}) 2024-09-11T01:38:31.579 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.577+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.579 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.577+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2916243677}) 2024-09-11T01:38:31.579 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.577+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1571421855}) 2024-09-11T01:38:31.579 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.578+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.580 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.578+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=603169708}) 2024-09-11T01:38:31.580 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.578+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2502030596}) 2024-09-11T01:38:31.580 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.578+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.580 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.578+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=603169708}) 2024-09-11T01:38:31.580 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.578+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3932641862}) 2024-09-11T01:38:31.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.578+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.578+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3893038091}) 2024-09-11T01:38:31.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.578+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1933916848}) 2024-09-11T01:38:31.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.578+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.578+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3893038091}) 2024-09-11T01:38:31.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.578+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1971018173}) 2024-09-11T01:38:31.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.579+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.579+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1170042041}) 2024-09-11T01:38:31.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.579+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4109863780}) 2024-09-11T01:38:31.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.579+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.582 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.579+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1170042041}) 2024-09-11T01:38:31.582 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.579+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3435744456}) 2024-09-11T01:38:31.582 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.580+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.580+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=19620472}) 2024-09-11T01:38:31.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.580+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1547298840}) 2024-09-11T01:38:31.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.580+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.580+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=19620472}) 2024-09-11T01:38:31.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.580+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2350012372}) 2024-09-11T01:38:31.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.581+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.581+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=598682913}) 2024-09-11T01:38:31.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.581+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=386442153}) 2024-09-11T01:38:31.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.581+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.581+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=598682913}) 2024-09-11T01:38:31.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.581+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2207102120}) 2024-09-11T01:38:31.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.582+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.582+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1944736126}) 2024-09-11T01:38:31.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.582+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1008303661}) 2024-09-11T01:38:31.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.582+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.582+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1944736126}) 2024-09-11T01:38:31.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.582+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=775234610}) 2024-09-11T01:38:31.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.583+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.583+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2680899880}) 2024-09-11T01:38:31.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.583+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3575939894}) 2024-09-11T01:38:31.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.583+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.583+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2680899880}) 2024-09-11T01:38:31.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.583+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1649224125}) 2024-09-11T01:38:31.586 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.584+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.586 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.584+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2383346813}) 2024-09-11T01:38:31.586 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.584+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=749328892}) 2024-09-11T01:38:31.586 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.584+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.586 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.584+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2383346813}) 2024-09-11T01:38:31.587 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.584+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=35623794}) 2024-09-11T01:38:31.587 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.585+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.587 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.585+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=890128499}) 2024-09-11T01:38:31.587 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.585+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=514785098}) 2024-09-11T01:38:31.587 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.585+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.587 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.585+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=890128499}) 2024-09-11T01:38:31.587 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.585+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1788612300}) 2024-09-11T01:38:31.589 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.586+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.589 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.586+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1701904617}) 2024-09-11T01:38:31.589 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.586+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1517087184}) 2024-09-11T01:38:31.589 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.586+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.589 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.586+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1701904617}) 2024-09-11T01:38:31.589 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.586+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2187049655}) 2024-09-11T01:38:31.589 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.586+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.589 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.586+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1420239427}) 2024-09-11T01:38:31.589 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.586+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=315221759}) 2024-09-11T01:38:31.589 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.586+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.589 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.586+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1420239427}) 2024-09-11T01:38:31.589 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.586+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2020737579}) 2024-09-11T01:38:31.590 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.587+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.590 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.587+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3065940096}) 2024-09-11T01:38:31.590 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.587+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3525532039}) 2024-09-11T01:38:31.590 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.587+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.590 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.587+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3065940096}) 2024-09-11T01:38:31.590 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.587+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3158745921}) 2024-09-11T01:38:31.591 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.588+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.591 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.588+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=36,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=15} crc {logm=1875770140,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2235474613,mon_config_key=3715565326,monmap=2933275274,nvmeofgw=598337686,osd_metadata=1246036448,osd_pg_creating=2812414034,osd_snap=1637040764}) 2024-09-11T01:38:31.591 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.588+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=35,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_snap=15} crc {logm=1121192971,mdsmap=3760429986,mgr=3526296668,mgr_command_descs=3192186154,mgr_metadata=3338432716,mgrstat=2235474613,mon_config_key=3330371543,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_snap=1637040764}) 2024-09-11T01:38:31.591 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.588+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.591 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.588+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=36,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=15} crc {logm=1875770140,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2235474613,mon_config_key=3715565326,monmap=2933275274,nvmeofgw=598337686,osd_metadata=1246036448,osd_pg_creating=2812414034,osd_snap=1637040764}) 2024-09-11T01:38:31.592 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.588+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=37,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=14} crc {logm=4078051102,mdsmap=3760429986,mgr=3428337343,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2235474613,mon_config_key=1069619752,monmap=2933275274,nvmeofgw=598337686,osd_metadata=118787097,osd_pg_creating=2812414034,osd_snap=1748918631}) 2024-09-11T01:38:31.592 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.589+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.592 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.589+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=43,osdmap=57} crc {osd_snap=3771262222,osdmap=266031905}) 2024-09-11T01:38:31.592 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.589+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=44,osdmap=56} crc {osd_snap=2552776307,osdmap=477490979}) 2024-09-11T01:38:31.592 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.589+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.592 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.589+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=43,osdmap=57} crc {osd_snap=3771262222,osdmap=266031905}) 2024-09-11T01:38:31.592 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.589+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=45,osdmap=55} crc {osd_snap=949498773,osdmap=2029219031}) 2024-09-11T01:38:31.593 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.590+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.593 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.590+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4276202895}) 2024-09-11T01:38:31.593 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.590+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2245127795}) 2024-09-11T01:38:31.593 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.591+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.593 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.591+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4276202895}) 2024-09-11T01:38:31.593 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.591+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1448173485}) 2024-09-11T01:38:31.596 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.594+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.596 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.594+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=50} crc {osdmap=3300534979}) 2024-09-11T01:38:31.596 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.594+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=53} crc {osdmap=1453072642}) 2024-09-11T01:38:31.596 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.594+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:38:31.596 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.594+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=50} crc {osdmap=3300534979}) 2024-09-11T01:38:31.596 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:38:31.594+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=51} crc {osdmap=703092984}) 2024-09-11T01:38:31.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.600+0000 7f3400a8c640 10 monclient: get_monmap_and_config 2024-09-11T01:38:31.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.600+0000 7f3400a8c640 10 monclient: build_initial_monmap 2024-09-11T01:38:31.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.600+0000 7f3400a8c640 10 monclient: monmap: 2024-09-11T01:38:31.602 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:31.602 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:31.602 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:38:31.601280+0000 2024-09-11T01:38:31.602 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:38:31.601280+0000 2024-09-11T01:38:31.602 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:31.602 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:31.602 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:38:31.602 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:38:31.602 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:38:31.602 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:38:31.602 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:38:31.602 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:31.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.601+0000 7f3400a8c640 1 Processor -- start 2024-09-11T01:38:31.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.601+0000 7f3400a8c640 1 -- start start 2024-09-11T01:38:31.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.601+0000 7f3400a8c640 10 monclient: init 2024-09-11T01:38:31.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.601+0000 7f3400a8c640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:31.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.601+0000 7f3400a8c640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-11T01:38:31.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.601+0000 7f3400a8c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f33fc0a4170 addr v1:172.21.15.71:6789/0 2024-09-11T01:38:31.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.601+0000 7f3400a8c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f33fc13af80 addr v1:172.21.15.12:6789/0 2024-09-11T01:38:31.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.601+0000 7f3400a8c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f33fc13ea90 0x7f33fc140e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:31.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.601+0000 7f3400a8c640 10 monclient(hunting): picked mon.noname-b con 0x7f33fc13ea90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:31.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.601+0000 7f3400a8c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33fc1380d0 con 0x7f33fc13af80 2024-09-11T01:38:31.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.601+0000 7f3400a8c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33fc09fff0 con 0x7f33fc0a4170 2024-09-11T01:38:31.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.601+0000 7f3400a8c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:31.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.601+0000 7f3400a8c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f33fc138350 con 0x7f33fc13ea90 2024-09-11T01:38:31.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.601+0000 7f3400a8c640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:31.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.601+0000 7f3400a8c640 10 monclient(hunting): authenticate will time out at 1318.043938s 2024-09-11T01:38:31.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.602+0000 7f33f3fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f33fc13af80 0x7f33fc13d370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:51924/0 (socket says 172.21.15.12:51924) 2024-09-11T01:38:31.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.602+0000 7f33f3fff640 1 -- 172.21.15.12:0/583410213 learned_addr learned my addr 172.21.15.12:0/583410213 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:38:31.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.602+0000 7f33fad9d640 1 --2- 172.21.15.12:0/583410213 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f33fc13ea90 0x7f33fc140e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:31.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.602+0000 7f33fad9d640 10 monclient(hunting): get_auth_request con 0x7f33fc13ea90 auth_method 0 2024-09-11T01:38:31.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.602+0000 7f33fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:31.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.602+0000 7f33fad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:31.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.602+0000 7f33fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:31.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.602+0000 7f33fad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:31.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.602+0000 7f33fad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:31.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.602+0000 7f33fad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:31.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.602+0000 7f33fad9d640 10 monclient(hunting): handle_auth_done global_id 84332 payload 274 2024-09-11T01:38:31.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.602+0000 7f33fad9d640 1 -- 172.21.15.12:0/583410213 >> v1:172.21.15.71:6789/0 conn(0x7f33fc0a4170 legacy=0x7f33fc073fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:38:31.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.602+0000 7f33fad9d640 1 -- 172.21.15.12:0/583410213 >> v1:172.21.15.12:6789/0 conn(0x7f33fc13af80 legacy=0x7f33fc13d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:38:31.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.602+0000 7f33fad9d640 10 monclient: _finish_hunting 0 2024-09-11T01:38:31.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.602+0000 7f33fad9d640 1 monclient: found mon.noname-b 2024-09-11T01:38:31.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.602+0000 7f33fad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:31.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.602+0000 7f33fad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:38:31.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.602+0000 7f33fad9d640 1 -- 172.21.15.12:0/583410213 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33fc070280 con 0x7f33fc13ea90 2024-09-11T01:38:31.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.605+0000 7f33fad9d640 1 -- 172.21.15.12:0/583410213 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f33fc13ea90 msgr2=0x7f33fc140e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:31.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.605+0000 7f33fad9d640 1 --2- 172.21.15.12:0/583410213 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f33fc13ea90 0x7f33fc140e80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f33f4009dc0 tx=0x7f33f402a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42bba542cfd017c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:31.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.605+0000 7f33f9d9b640 1 -- 172.21.15.12:0/583410213 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f4007a30 con 0x7f33fc13ea90 2024-09-11T01:38:31.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.605+0000 7f33f9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:31.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.605+0000 7f33f9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:38:31.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.605+0000 7f33f9d9b640 10 monclient: dump: 2024-09-11T01:38:31.612 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:31.612 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:31.612 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:31.613 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:31.613 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:31.613 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:31.613 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:31.613 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:31.613 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:31.613 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:31.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.605+0000 7f33f9d9b640 10 monclient: _finish_auth 0 2024-09-11T01:38:31.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.605+0000 7f33f9d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:38:31.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.605+0000 7f33f9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:31.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.605+0000 7f3400a8c640 5 monclient: authenticate success, global_id 84332 2024-09-11T01:38:31.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.605+0000 7f33f9d9b640 1 -- 172.21.15.12:0/583410213 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33f4007bd0 con 0x7f33fc13ea90 2024-09-11T01:38:31.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.605+0000 7f3400a8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:38:31.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.605+0000 7f33f9d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:31.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.605+0000 7f3400a8c640 10 monclient: get_monmap_and_config success 2024-09-11T01:38:31.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.605+0000 7f3400a8c640 4 set_mon_vals no callback set 2024-09-11T01:38:31.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.605+0000 7f33f9d9b640 1 -- 172.21.15.12:0/583410213 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f4031400 con 0x7f33fc13ea90 2024-09-11T01:38:31.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.605+0000 7f33f9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:31.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.605+0000 7f33f9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:38:31.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.605+0000 7f33f9d9b640 10 monclient: dump: 2024-09-11T01:38:31.614 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:31.614 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:31.614 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:31.614 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:31.614 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:31.614 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:31.614 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:31.615 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:31.615 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:31.615 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:31.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.605+0000 7f3400a8c640 10 monclient: shutdown 2024-09-11T01:38:31.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.605+0000 7f3400a8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:31.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.605+0000 7f3400a8c640 1 -- 172.21.15.12:0/583410213 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f33fc13ea90 msgr2=0x7f33fc140e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:31.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.605+0000 7f3400a8c640 1 --2- 172.21.15.12:0/583410213 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f33fc13ea90 0x7f33fc140e80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f33f4009dc0 tx=0x7f33f402a480 comp rx=0 tx=0).stop 2024-09-11T01:38:31.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.605+0000 7f3400a8c640 1 -- 172.21.15.12:0/583410213 shutdown_connections 2024-09-11T01:38:31.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.605+0000 7f3400a8c640 1 --2- 172.21.15.12:0/583410213 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f33fc13ea90 0x7f33fc140e80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:31.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.605+0000 7f3400a8c640 1 -- 172.21.15.12:0/583410213 >> 172.21.15.12:0/583410213 conn(0x7f33fc0a3650 msgr2=0x7f33fc0a3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:31.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.606+0000 7f3400a8c640 1 -- 172.21.15.12:0/583410213 shutdown_connections 2024-09-11T01:38:31.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.606+0000 7f3400a8c640 1 -- 172.21.15.12:0/583410213 wait complete. 2024-09-11T01:38:31.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.606+0000 7f3400a8c640 10 monclient: build_initial_monmap 2024-09-11T01:38:31.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.606+0000 7f3400a8c640 10 monclient: monmap: 2024-09-11T01:38:31.616 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:31.616 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:38:31.616 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:38:31.616 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:38:31.616 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:31.616 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:31.616 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:38:31.616 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:38:31.616 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:38:31.616 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:31.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f3400a8c640 1 Processor -- start 2024-09-11T01:38:31.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f3400a8c640 1 -- start start 2024-09-11T01:38:31.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f3400a8c640 10 monclient: init 2024-09-11T01:38:31.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f3400a8c640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:31.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f3400a8c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:38:31.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f3400a8c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33fc0a4170 0x7f33fc131680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:31.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f3400a8c640 10 monclient(hunting): picked mon.noname-a con 0x7f33fc0a4170 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:38:31.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f3400a8c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33fc13af80 0x7f33fc131bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:31.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f3400a8c640 10 monclient(hunting): picked mon.noname-b con 0x7f33fc13af80 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:38:31.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f3400a8c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f33fc13ea90 0x7f33fc13b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:31.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f3400a8c640 10 monclient(hunting): picked mon.noname-c con 0x7f33fc13ea90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:31.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f3400a8c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:31.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f3400a8c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f33fc074850 con 0x7f33fc0a4170 2024-09-11T01:38:31.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f3400a8c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:31.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f3400a8c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f33fc0746d0 con 0x7f33fc13af80 2024-09-11T01:38:31.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f3400a8c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:31.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f3400a8c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f33fc0a0660 con 0x7f33fc13ea90 2024-09-11T01:38:31.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f3400a8c640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:31.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f3400a8c640 10 monclient(hunting): authenticate will time out at 1318.049676s 2024-09-11T01:38:31.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f33fa59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33fc0a4170 0x7f33fc131680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:31.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f33fad9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f33fc13ea90 0x7f33fc13b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:31.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f33fa59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33fc0a4170 0x7f33fc131680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57776/0 (socket says 172.21.15.12:57776) 2024-09-11T01:38:31.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f33fa59c640 1 -- 172.21.15.12:0/403107067 learned_addr learned my addr 172.21.15.12:0/403107067 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:38:31.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f33fa59c640 10 monclient(hunting): get_auth_request con 0x7f33fc0a4170 auth_method 0 2024-09-11T01:38:31.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f33fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:31.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f33fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:31.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f33fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:31.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f33fad9d640 10 monclient(hunting): get_auth_request con 0x7f33fc13ea90 auth_method 0 2024-09-11T01:38:31.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f33fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:31.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f33fad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:31.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f33fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:31.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f33fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:31.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f33fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:31.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f33fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:31.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f33fad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:31.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f33fad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:31.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f33f3fff640 1 --2- 172.21.15.12:0/403107067 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33fc13af80 0x7f33fc131bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:31.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f33fad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:31.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f33f3fff640 10 monclient(hunting): get_auth_request con 0x7f33fc13af80 auth_method 0 2024-09-11T01:38:31.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f33f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:31.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f33f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:31.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.607+0000 7f33f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:31.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.608+0000 7f33f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:31.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.608+0000 7f33f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:31.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.608+0000 7f33f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:31.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.608+0000 7f33fad9d640 10 monclient(hunting): handle_auth_done global_id 84335 payload 824 2024-09-11T01:38:31.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.608+0000 7f33fad9d640 1 -- 172.21.15.12:0/403107067 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33fc13af80 msgr2=0x7f33fc131bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:31.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.608+0000 7f33fad9d640 1 --2- 172.21.15.12:0/403107067 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33fc13af80 0x7f33fc131bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:31.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.608+0000 7f33fad9d640 1 -- 172.21.15.12:0/403107067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33fc0a4170 msgr2=0x7f33fc131680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:31.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.608+0000 7f33fad9d640 1 --2- 172.21.15.12:0/403107067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33fc0a4170 0x7f33fc131680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:31.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.608+0000 7f33fad9d640 10 monclient: _finish_hunting 0 2024-09-11T01:38:31.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.608+0000 7f33fad9d640 1 monclient: found mon.noname-c 2024-09-11T01:38:31.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.608+0000 7f33fad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:31.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.608+0000 7f33fad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:38:31.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.608+0000 7f33fad9d640 1 -- 172.21.15.12:0/403107067 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33fc11d2d0 con 0x7f33fc13ea90 2024-09-11T01:38:31.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.608+0000 7f33fa59c640 1 --2- 172.21.15.12:0/403107067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33fc0a4170 0x7f33fc131680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:38:31.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.610+0000 7f33fad9d640 1 -- 172.21.15.12:0/403107067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f33fc13ea90 msgr2=0x7f33fc13b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:31.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.610+0000 7f33fad9d640 1 --2- 172.21.15.12:0/403107067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f33fc13ea90 0x7f33fc13b360 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f33f4031000 tx=0x7f33f4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:31.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.610+0000 7f33f2ffd640 1 -- 172.21.15.12:0/403107067 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f4040070 con 0x7f33fc13ea90 2024-09-11T01:38:31.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.610+0000 7f33f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:31.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.610+0000 7f33f2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:38:31.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.610+0000 7f33f2ffd640 10 monclient: dump: 2024-09-11T01:38:31.621 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:31.621 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:31.621 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:31.621 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:31.621 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:31.621 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:31.621 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:31.621 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:31.621 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:31.621 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:31.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.610+0000 7f33f2ffd640 10 monclient: _finish_auth 0 2024-09-11T01:38:31.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.610+0000 7f33f2ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:38:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.610+0000 7f33f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.610+0000 7f33f2ffd640 1 -- 172.21.15.12:0/403107067 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33f4045460 con 0x7f33fc13ea90 2024-09-11T01:38:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.610+0000 7f33f2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.610+0000 7f33f2ffd640 1 -- 172.21.15.12:0/403107067 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f4007a30 con 0x7f33fc13ea90 2024-09-11T01:38:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.610+0000 7f3400a8c640 5 monclient: authenticate success, global_id 84335 2024-09-11T01:38:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.610+0000 7f3400a8c640 10 monclient: _renew_subs 2024-09-11T01:38:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.610+0000 7f3400a8c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:38:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.610+0000 7f3400a8c640 1 -- 172.21.15.12:0/403107067 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33fc13e720 con 0x7f33fc13ea90 2024-09-11T01:38:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.610+0000 7f33f959a640 4 set_mon_vals no callback set 2024-09-11T01:38:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.610+0000 7f33f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.610+0000 7f33f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:38:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.610+0000 7f33f2ffd640 10 monclient: dump: 2024-09-11T01:38:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:31.622 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:31.623 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:31.623 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:31.623 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:31.623 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:31.623 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:31.623 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:31.623 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:31.623 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:31.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.610+0000 7f3400a8c640 10 monclient: _renew_subs 2024-09-11T01:38:31.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.610+0000 7f3400a8c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:38:31.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.610+0000 7f3400a8c640 1 -- 172.21.15.12:0/403107067 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f33fc13c8a0 con 0x7f33fc13ea90 2024-09-11T01:38:31.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.611+0000 7f33f2ffd640 1 -- 172.21.15.12:0/403107067 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f33f403c420 con 0x7f33fc13ea90 2024-09-11T01:38:31.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.611+0000 7f3400a8c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:31.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.611+0000 7f3400a8c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:31.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.611+0000 7f3400a8c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:38:31.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.611+0000 7f3400a8c640 1 -- 172.21.15.12:0/403107067 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33fc073fc0 con 0x7f33fc13ea90 2024-09-11T01:38:31.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.611+0000 7f33f2ffd640 1 --2- 172.21.15.12:0/403107067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f33dc03e260 0x7f33dc040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:31.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.612+0000 7f33f2ffd640 1 -- 172.21.15.12:0/403107067 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(320..320 src has 186..320) ==== 4783+0+0 (secure 0 0 0) 0x7f33f4074ac0 con 0x7f33fc13ea90 2024-09-11T01:38:31.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.612+0000 7f33fa59c640 1 --2- 172.21.15.12:0/403107067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f33dc03e260 0x7f33dc040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:31.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.612+0000 7f33fa59c640 10 monclient: get_auth_request con 0x7f33dc03e260 auth_method 0 2024-09-11T01:38:31.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.615+0000 7f33fa59c640 1 --2- 172.21.15.12:0/403107067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f33dc03e260 0x7f33dc040720 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f33ec009d80 tx=0x7f33ec009110 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:31.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.616+0000 7f33f2ffd640 1 -- 172.21.15.12:0/403107067 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f33f404a060 con 0x7f33fc13ea90 2024-09-11T01:38:31.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.616+0000 7f33f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:31.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.616+0000 7f33f2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:38:31.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.872+0000 7f3400a8c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:38:31.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.872+0000 7f3400a8c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:38:31.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.872+0000 7f3400a8c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:38:31.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.872+0000 7f3400a8c640 1 -- 172.21.15.12:0/403107067 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f33fc132100 con 0x7f33fc13ea90 2024-09-11T01:38:31.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.872+0000 7f33f2ffd640 1 -- 172.21.15.12:0/403107067 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f33fc1380d0 con 0x7f33fc13ea90 2024-09-11T01:38:31.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.872+0000 7f33f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:38:31.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.872+0000 7f33f2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:38:31.874 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:38:31.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.874+0000 7f3400a8c640 1 -- 172.21.15.12:0/403107067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f33dc03e260 msgr2=0x7f33dc040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:31.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.874+0000 7f3400a8c640 1 --2- 172.21.15.12:0/403107067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f33dc03e260 0x7f33dc040720 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f33ec009d80 tx=0x7f33ec009110 comp rx=0 tx=0).stop 2024-09-11T01:38:31.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.874+0000 7f3400a8c640 10 monclient: shutdown 2024-09-11T01:38:31.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.874+0000 7f3400a8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:31.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.874+0000 7f3400a8c640 1 -- 172.21.15.12:0/403107067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f33fc13ea90 msgr2=0x7f33fc13b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:31.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.874+0000 7f3400a8c640 1 --2- 172.21.15.12:0/403107067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f33fc13ea90 0x7f33fc13b360 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f33f4031000 tx=0x7f33f4004a40 comp rx=0 tx=0).stop 2024-09-11T01:38:31.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.874+0000 7f3400a8c640 1 -- 172.21.15.12:0/403107067 shutdown_connections 2024-09-11T01:38:31.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.874+0000 7f3400a8c640 1 --2- 172.21.15.12:0/403107067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f33fc13ea90 0x7f33fc13b360 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:31.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.874+0000 7f3400a8c640 1 --2- 172.21.15.12:0/403107067 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33fc13af80 0x7f33fc131bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:31.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.874+0000 7f3400a8c640 1 --2- 172.21.15.12:0/403107067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f33dc03e260 0x7f33dc040720 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:31.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.874+0000 7f3400a8c640 1 --2- 172.21.15.12:0/403107067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f33fc0a4170 0x7f33fc131680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:31.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.874+0000 7f3400a8c640 1 -- 172.21.15.12:0/403107067 >> 172.21.15.12:0/403107067 conn(0x7f33fc0a3650 msgr2=0x7f33fc136550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:31.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.874+0000 7f3400a8c640 1 -- 172.21.15.12:0/403107067 shutdown_connections 2024-09-11T01:38:31.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:31.874+0000 7f3400a8c640 1 -- 172.21.15.12:0/403107067 wait complete. 2024-09-11T01:38:31.886 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:38:31.886 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:38:31.887 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:38:31.887 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:38:32.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.027+0000 7fbb823da640 10 monclient: get_monmap_and_config 2024-09-11T01:38:32.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.027+0000 7fbb823da640 10 monclient: build_initial_monmap 2024-09-11T01:38:32.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.027+0000 7fbb823da640 10 monclient: monmap: 2024-09-11T01:38:32.028 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:32.029 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:32.029 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:38:32.027909+0000 2024-09-11T01:38:32.029 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:38:32.027909+0000 2024-09-11T01:38:32.029 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:32.029 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:32.029 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:38:32.029 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:38:32.029 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:38:32.029 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:38:32.029 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:38:32.029 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:32.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.027+0000 7fbb823da640 1 Processor -- start 2024-09-11T01:38:32.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.027+0000 7fbb823da640 1 -- start start 2024-09-11T01:38:32.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.027+0000 7fbb823da640 10 monclient: init 2024-09-11T01:38:32.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb823da640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:32.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb823da640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-11T01:38:32.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb823da640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbb7c137e50 addr v1:172.21.15.12:6789/0 2024-09-11T01:38:32.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb823da640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbb7c138230 0x7fbb7c138640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:32.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb823da640 10 monclient(hunting): picked mon.noname-c con 0x7fbb7c138230 addr v2:172.21.15.71:3300/0 2024-09-11T01:38:32.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb823da640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbb7c13d380 0x7fbb7c13f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:32.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb823da640 10 monclient(hunting): picked mon.noname-a con 0x7fbb7c13d380 addr v2:172.21.15.12:3300/0 2024-09-11T01:38:32.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb823da640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb7c0a62d0 con 0x7fbb7c137e50 2024-09-11T01:38:32.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb823da640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:32.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb823da640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fbb7c09bb40 con 0x7fbb7c13d380 2024-09-11T01:38:32.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb823da640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:32.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb823da640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fbb7c09c4a0 con 0x7fbb7c138230 2024-09-11T01:38:32.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb823da640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:32.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb823da640 10 monclient(hunting): authenticate will time out at 1318.470518s 2024-09-11T01:38:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb7bfff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbb7c137e50 0x7fbb7c09d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:51938/0 (socket says 172.21.15.12:51938) 2024-09-11T01:38:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb7bfff640 1 -- 172.21.15.12:0/3799741745 learned_addr learned my addr 172.21.15.12:0/3799741745 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:38:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb80977640 1 --2- 172.21.15.12:0/3799741745 >> v2:172.21.15.12:3300/0 conn(0x7fbb7c13d380 0x7fbb7c13f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb80977640 10 monclient(hunting): get_auth_request con 0x7fbb7c13d380 auth_method 0 2024-09-11T01:38:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb80977640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb80977640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb80977640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb7b7fe640 1 --2- 172.21.15.12:0/3799741745 >> v2:172.21.15.71:3300/0 conn(0x7fbb7c138230 0x7fbb7c138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb7b7fe640 10 monclient(hunting): get_auth_request con 0x7fbb7c138230 auth_method 0 2024-09-11T01:38:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:32.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.028+0000 7fbb7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:32.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.029+0000 7fbb7b7fe640 10 monclient(hunting): handle_auth_done global_id 104113 payload 274 2024-09-11T01:38:32.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.029+0000 7fbb7b7fe640 1 -- 172.21.15.12:0/3799741745 >> v2:172.21.15.12:3300/0 conn(0x7fbb7c13d380 msgr2=0x7fbb7c13f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:38:32.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.029+0000 7fbb7b7fe640 1 --2- 172.21.15.12:0/3799741745 >> v2:172.21.15.12:3300/0 conn(0x7fbb7c13d380 0x7fbb7c13f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:32.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.029+0000 7fbb7b7fe640 1 -- 172.21.15.12:0/3799741745 >> v1:172.21.15.12:6789/0 conn(0x7fbb7c137e50 legacy=0x7fbb7c09d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:38:32.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.029+0000 7fbb7b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:38:32.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.029+0000 7fbb7b7fe640 1 monclient: found mon.noname-c 2024-09-11T01:38:32.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.029+0000 7fbb7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:32.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.029+0000 7fbb7b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:38:32.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.029+0000 7fbb7b7fe640 1 -- 172.21.15.12:0/3799741745 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb7c09c780 con 0x7fbb7c138230 2024-09-11T01:38:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.032+0000 7fbb7b7fe640 1 -- 172.21.15.12:0/3799741745 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb7c138230 msgr2=0x7fbb7c138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.032+0000 7fbb7b7fe640 1 --2- 172.21.15.12:0/3799741745 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb7c138230 0x7fbb7c138640 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c0a6f80 tx=0x7fbb70029fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50e62c11ab170b23 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.032+0000 7fbb7affd640 1 -- 172.21.15.12:0/3799741745 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb70037070 con 0x7fbb7c138230 2024-09-11T01:38:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.032+0000 7fbb7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.032+0000 7fbb7affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:38:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.032+0000 7fbb7affd640 10 monclient: dump: 2024-09-11T01:38:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:32.038 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.032+0000 7fbb7affd640 10 monclient: _finish_auth 0 2024-09-11T01:38:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.032+0000 7fbb7affd640 10 monclient: _check_auth_tickets 2024-09-11T01:38:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.032+0000 7fbb7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.032+0000 7fbb7affd640 1 -- 172.21.15.12:0/3799741745 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb70032440 con 0x7fbb7c138230 2024-09-11T01:38:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.032+0000 7fbb823da640 5 monclient: authenticate success, global_id 104113 2024-09-11T01:38:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.032+0000 7fbb823da640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:38:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.032+0000 7fbb7affd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.032+0000 7fbb823da640 10 monclient: get_monmap_and_config success 2024-09-11T01:38:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.032+0000 7fbb823da640 4 set_mon_vals no callback set 2024-09-11T01:38:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.032+0000 7fbb7affd640 1 -- 172.21.15.12:0/3799741745 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb70032810 con 0x7fbb7c138230 2024-09-11T01:38:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.032+0000 7fbb7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.032+0000 7fbb7affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:38:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.032+0000 7fbb7affd640 10 monclient: dump: 2024-09-11T01:38:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:32.040 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.033+0000 7fbb823da640 10 monclient: shutdown 2024-09-11T01:38:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.033+0000 7fbb823da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.033+0000 7fbb823da640 1 -- 172.21.15.12:0/3799741745 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb7c138230 msgr2=0x7fbb7c138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.033+0000 7fbb823da640 1 --2- 172.21.15.12:0/3799741745 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb7c138230 0x7fbb7c138640 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c0a6f80 tx=0x7fbb70029fb0 comp rx=0 tx=0).stop 2024-09-11T01:38:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.033+0000 7fbb823da640 1 -- 172.21.15.12:0/3799741745 shutdown_connections 2024-09-11T01:38:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.033+0000 7fbb823da640 1 --2- 172.21.15.12:0/3799741745 >> v2:172.21.15.12:3300/0 conn(0x7fbb7c13d380 0x7fbb7c13f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.033+0000 7fbb823da640 1 --2- 172.21.15.12:0/3799741745 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb7c138230 0x7fbb7c138640 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.033+0000 7fbb823da640 1 -- 172.21.15.12:0/3799741745 >> 172.21.15.12:0/3799741745 conn(0x7fbb7c0a7020 msgr2=0x7fbb7c0a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.033+0000 7fbb823da640 1 -- 172.21.15.12:0/3799741745 shutdown_connections 2024-09-11T01:38:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.033+0000 7fbb823da640 1 -- 172.21.15.12:0/3799741745 wait complete. 2024-09-11T01:38:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.033+0000 7fbb823da640 10 monclient: build_initial_monmap 2024-09-11T01:38:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.033+0000 7fbb823da640 10 monclient: monmap: 2024-09-11T01:38:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:38:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:38:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:38:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:38:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:38:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:38:32.042 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb823da640 1 Processor -- start 2024-09-11T01:38:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb823da640 1 -- start start 2024-09-11T01:38:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb823da640 10 monclient: init 2024-09-11T01:38:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb823da640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb823da640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:38:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb823da640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb7c137e50 0x7fbb7c1cd000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb823da640 10 monclient(hunting): picked mon.noname-b con 0x7fbb7c137e50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:38:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb823da640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbb7c138230 0x7fbb7c137050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb823da640 10 monclient(hunting): picked mon.noname-c con 0x7fbb7c138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb823da640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb7c13d380 0x7fbb7c0a1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb823da640 10 monclient(hunting): picked mon.noname-a con 0x7fbb7c13d380 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:38:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb823da640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb823da640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c06f890 con 0x7fbb7c13d380 2024-09-11T01:38:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb823da640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb823da640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c06f710 con 0x7fbb7c137e50 2024-09-11T01:38:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb823da640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb823da640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fbb7c13fcb0 con 0x7fbb7c138230 2024-09-11T01:38:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb823da640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb823da640 10 monclient(hunting): authenticate will time out at 1318.476788s 2024-09-11T01:38:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb80977640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb7c13d380 0x7fbb7c0a1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb7b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbb7c138230 0x7fbb7c137050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb80977640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb7c13d380 0x7fbb7c0a1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57792/0 (socket says 172.21.15.12:57792) 2024-09-11T01:38:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb80977640 1 -- 172.21.15.12:0/4150204487 learned_addr learned my addr 172.21.15.12:0/4150204487 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:38:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb7b7fe640 10 monclient(hunting): get_auth_request con 0x7fbb7c138230 auth_method 0 2024-09-11T01:38:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb80977640 10 monclient(hunting): get_auth_request con 0x7fbb7c13d380 auth_method 0 2024-09-11T01:38:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb80977640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb80977640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb80977640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.034+0000 7fbb7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.035+0000 7fbb7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.035+0000 7fbb7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.035+0000 7fbb80977640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.035+0000 7fbb80977640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.035+0000 7fbb80977640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.035+0000 7fbb7bfff640 1 --2- 172.21.15.12:0/4150204487 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb7c137e50 0x7fbb7c1cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.035+0000 7fbb7bfff640 10 monclient(hunting): get_auth_request con 0x7fbb7c137e50 auth_method 0 2024-09-11T01:38:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.035+0000 7fbb7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.035+0000 7fbb7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.035+0000 7fbb7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.035+0000 7fbb7b7fe640 10 monclient(hunting): handle_auth_done global_id 84338 payload 824 2024-09-11T01:38:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.035+0000 7fbb7b7fe640 1 -- 172.21.15.12:0/4150204487 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb7c137e50 msgr2=0x7fbb7c1cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.035+0000 7fbb7b7fe640 1 --2- 172.21.15.12:0/4150204487 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb7c137e50 0x7fbb7c1cd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.035+0000 7fbb7b7fe640 1 -- 172.21.15.12:0/4150204487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb7c13d380 msgr2=0x7fbb7c0a1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.035+0000 7fbb7b7fe640 1 --2- 172.21.15.12:0/4150204487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb7c13d380 0x7fbb7c0a1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.035+0000 7fbb7b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:38:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.035+0000 7fbb7b7fe640 1 monclient: found mon.noname-c 2024-09-11T01:38:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.035+0000 7fbb7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.035+0000 7fbb7b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:38:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.035+0000 7fbb7b7fe640 1 -- 172.21.15.12:0/4150204487 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb7c0a1710 con 0x7fbb7c138230 2024-09-11T01:38:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.035+0000 7fbb80977640 1 --2- 172.21.15.12:0/4150204487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb7c13d380 0x7fbb7c0a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:38:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.035+0000 7fbb7bfff640 1 --2- 172.21.15.12:0/4150204487 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb7c137e50 0x7fbb7c1cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:38:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.037+0000 7fbb7b7fe640 1 -- 172.21.15.12:0/4150204487 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbb7c138230 msgr2=0x7fbb7c137050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.037+0000 7fbb7b7fe640 1 --2- 172.21.15.12:0/4150204487 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbb7c138230 0x7fbb7c137050 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbb70031000 tx=0x7fbb700046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.037+0000 7fbb78ff9640 1 -- 172.21.15.12:0/4150204487 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb70037040 con 0x7fbb7c138230 2024-09-11T01:38:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.037+0000 7fbb78ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.037+0000 7fbb78ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:38:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.037+0000 7fbb78ff9640 10 monclient: dump: 2024-09-11T01:38:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:32.050 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.037+0000 7fbb78ff9640 10 monclient: _finish_auth 0 2024-09-11T01:38:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.037+0000 7fbb78ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:38:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.037+0000 7fbb78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.037+0000 7fbb78ff9640 1 -- 172.21.15.12:0/4150204487 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb7003b6c0 con 0x7fbb7c138230 2024-09-11T01:38:32.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.037+0000 7fbb823da640 5 monclient: authenticate success, global_id 84338 2024-09-11T01:38:32.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.037+0000 7fbb823da640 10 monclient: _renew_subs 2024-09-11T01:38:32.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.037+0000 7fbb823da640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:38:32.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.037+0000 7fbb823da640 1 -- 172.21.15.12:0/4150204487 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb7c13fa60 con 0x7fbb7c138230 2024-09-11T01:38:32.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.037+0000 7fbb78ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:32.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.037+0000 7fbb78ff9640 1 -- 172.21.15.12:0/4150204487 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7003ba90 con 0x7fbb7c138230 2024-09-11T01:38:32.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.037+0000 7fbb78ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:32.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.037+0000 7fbb78ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:38:32.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.037+0000 7fbb78ff9640 10 monclient: dump: 2024-09-11T01:38:32.051 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:32.051 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:32.051 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.051 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.051 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:32.052 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.037+0000 7fbb823da640 10 monclient: _renew_subs 2024-09-11T01:38:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.037+0000 7fbb823da640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:38:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.037+0000 7fbb7a7fc640 4 set_mon_vals no callback set 2024-09-11T01:38:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.037+0000 7fbb823da640 1 -- 172.21.15.12:0/4150204487 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb7c1cde90 con 0x7fbb7c138230 2024-09-11T01:38:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.038+0000 7fbb823da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.038+0000 7fbb823da640 1 --2- 172.21.15.12:0/4150204487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb7c0a22a0 0x7fbb7c0a2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.038+0000 7fbb823da640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.038+0000 7fbb823da640 1 -- 172.21.15.12:0/4150204487 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c06a060 con 0x7fbb7c0a22a0 2024-09-11T01:38:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.038+0000 7fbb7bfff640 1 --2- 172.21.15.12:0/4150204487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb7c0a22a0 0x7fbb7c0a2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.038+0000 7fbb7bfff640 10 monclient: get_auth_request con 0x7fbb7c0a22a0 auth_method 0 2024-09-11T01:38:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.038+0000 7fbb7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.038+0000 7fbb7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.038+0000 7fbb7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.038+0000 7fbb78ff9640 1 -- 172.21.15.12:0/4150204487 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fbb7003a030 con 0x7fbb7c138230 2024-09-11T01:38:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.038+0000 7fbb7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.038+0000 7fbb7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.038+0000 7fbb7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.038+0000 7fbb7bfff640 10 monclient(hunting): handle_auth_done global_id 94239 payload 824 2024-09-11T01:38:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.038+0000 7fbb7bfff640 1 -- 172.21.15.12:0/4150204487 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbb7c0a2bc0 con 0x7fbb7c0a22a0 2024-09-11T01:38:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.038+0000 7fbb78ff9640 1 --2- 172.21.15.12:0/4150204487 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbb6803e240 0x7fbb68040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.039+0000 7fbb78ff9640 1 -- 172.21.15.12:0/4150204487 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(320..320 src has 186..320) ==== 4783+0+0 (secure 0 0 0) 0x7fbb70072e80 con 0x7fbb7c138230 2024-09-11T01:38:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.039+0000 7fbb80977640 1 --2- 172.21.15.12:0/4150204487 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbb6803e240 0x7fbb68040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.039+0000 7fbb80977640 10 monclient: get_auth_request con 0x7fbb6803e240 auth_method 0 2024-09-11T01:38:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.041+0000 7fbb7bfff640 1 -- 172.21.15.12:0/4150204487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb7c0a22a0 msgr2=0x7fbb7c0a2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.041+0000 7fbb7bfff640 1 --2- 172.21.15.12:0/4150204487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb7c0a22a0 0x7fbb7c0a2680 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c00e520 tx=0x7fbb6c00e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.041+0000 7fbb78ff9640 1 -- 172.21.15.12:0/4150204487 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c005cf0 con 0x7fbb7c0a22a0 2024-09-11T01:38:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.041+0000 7fbb78ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.041+0000 7fbb78ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:38:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.041+0000 7fbb78ff9640 10 monclient: dump: 2024-09-11T01:38:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:32.055 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.041+0000 7fbb78ff9640 1 -- 172.21.15.12:0/4150204487 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbb6c018070 con 0x7fbb7c0a22a0 2024-09-11T01:38:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.041+0000 7fbb78ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.041+0000 7fbb78ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:38:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.041+0000 7fbb78ff9640 1 -- 172.21.15.12:0/4150204487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb7c0a22a0 msgr2=0x7fbb7c0a2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.041+0000 7fbb78ff9640 1 --2- 172.21.15.12:0/4150204487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb7c0a22a0 0x7fbb7c0a2680 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c00e520 tx=0x7fbb6c00e9e0 comp rx=0 tx=0).stop 2024-09-11T01:38:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.041+0000 7fbb78ff9640 1 -- 172.21.15.12:0/4150204487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb7c0a22a0 msgr2=0x7fbb7c0a2680 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:38:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.041+0000 7fbb78ff9640 1 --2- 172.21.15.12:0/4150204487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb7c0a22a0 0x7fbb7c0a2680 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.041+0000 7fbb80977640 1 --2- 172.21.15.12:0/4150204487 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbb6803e240 0x7fbb68040700 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbb74007880 tx=0x7fbb740098d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.047+0000 7fbb823da640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:38:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.047+0000 7fbb823da640 1 --2- 172.21.15.12:0/4150204487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb7c121380 0x7fbb7c09e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.047+0000 7fbb823da640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.047+0000 7fbb823da640 1 -- 172.21.15.12:0/4150204487 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c138640 con 0x7fbb7c121380 2024-09-11T01:38:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.047+0000 7fbb7bfff640 1 --2- 172.21.15.12:0/4150204487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb7c121380 0x7fbb7c09e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.047+0000 7fbb7bfff640 10 monclient: get_auth_request con 0x7fbb7c121380 auth_method 0 2024-09-11T01:38:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.047+0000 7fbb7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.047+0000 7fbb7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.047+0000 7fbb7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.047+0000 7fbb7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.047+0000 7fbb7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.047+0000 7fbb7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.048+0000 7fbb7bfff640 10 monclient(hunting): handle_auth_done global_id 94242 payload 824 2024-09-11T01:38:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.048+0000 7fbb7bfff640 1 -- 172.21.15.12:0/4150204487 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbb7c137b40 con 0x7fbb7c121380 2024-09-11T01:38:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.050+0000 7fbb7bfff640 1 -- 172.21.15.12:0/4150204487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb7c121380 msgr2=0x7fbb7c09e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.050+0000 7fbb7bfff640 1 --2- 172.21.15.12:0/4150204487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb7c121380 0x7fbb7c09e420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c017000 tx=0x7fbb6c004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.050+0000 7fbb78ff9640 1 -- 172.21.15.12:0/4150204487 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c0074f0 con 0x7fbb7c121380 2024-09-11T01:38:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.050+0000 7fbb78ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.050+0000 7fbb78ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:38:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.050+0000 7fbb78ff9640 10 monclient: dump: 2024-09-11T01:38:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:32.058 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.050+0000 7fbb78ff9640 1 -- 172.21.15.12:0/4150204487 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7fbb6c0074f0 con 0x7fbb7c121380 2024-09-11T01:38:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.050+0000 7fbb78ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:38:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.050+0000 7fbb78ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:38:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.050+0000 7fbb78ff9640 1 -- 172.21.15.12:0/4150204487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb7c121380 msgr2=0x7fbb7c09e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.050+0000 7fbb78ff9640 1 --2- 172.21.15.12:0/4150204487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb7c121380 0x7fbb7c09e420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c017000 tx=0x7fbb6c004940 comp rx=0 tx=0).stop 2024-09-11T01:38:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.050+0000 7fbb78ff9640 1 -- 172.21.15.12:0/4150204487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb7c121380 msgr2=0x7fbb7c09e420 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:38:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.050+0000 7fbb78ff9640 1 --2- 172.21.15.12:0/4150204487 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb7c121380 0x7fbb7c09e420 secure :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c017000 tx=0x7fbb6c004940 comp rx=0 tx=0).stop 2024-09-11T01:38:32.059 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:38:32.059 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:38:32.059 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:38:32.059 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:38:32.059 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 92, 2024-09-11T01:38:32.059 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 58146, 2024-09-11T01:38:32.060 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:38:32.060 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:38:32.060 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:38:32.060 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:38:32.060 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:32.060 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 7, 2024-09-11T01:38:32.060 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:38:32.060 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:38:32.060 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:38:32.060 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:38:32.060 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:38:32.060 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:38:32.060 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:38:32.061 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:38:32.061 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:38:32.061 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:38:32.061 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:38:32.061 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:38:32.061 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:38:32.061 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:38:32.061 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:32.061 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:38:32.061 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:38:32.061 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:38:32.061 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:38:32.061 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:38:32.062 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:38:32.062 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:38:32.062 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:38:32.062 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:38:32.062 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:38:32.062 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:38:32.062 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:38:32.062 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:38:32.062 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:32.062 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.062 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:38:32.062 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:38:32.062 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:38:32.062 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:38:32.063 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:38:32.063 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:38:32.063 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:38:32.063 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:38:32.063 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:38:32.063 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:38:32.063 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:38:32.063 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:38:32.063 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:38:32.063 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:38:32.063 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:38:32.063 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:38:32.063 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:38:32.063 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:38:32.064 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:38:32.064 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:38:32.064 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:38:32.064 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:38:32.064 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:38:32.064 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:38:32.064 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:38:32.064 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:38:32.064 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:38:32.064 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:38:32.064 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:32.064 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:38:32.064 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.064 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:38:32.065 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.065 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:38:32.065 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:38:32.065 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:38:32.065 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:38:32.065 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.065 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:38:32.065 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:38:32.065 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:32.065 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.065 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.065 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:38:32.065 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:38:32.065 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:32.066 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:32.066 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:32.066 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.066 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:38:32.066 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:38:32.066 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:38:32.066 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:38:32.066 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:38:32.066 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.066 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.066 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:38:32.066 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:38:32.066 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:38:32.067 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:38:32.067 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.067 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:38:32.067 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:38:32.067 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:32.067 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.067 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.067 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:38:32.067 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:38:32.067 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:32.067 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:32.067 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:32.067 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.068 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:38:32.068 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:38:32.068 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:38:32.068 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:38:32.068 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:38:32.068 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.068 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.068 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:38:32.068 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:38:32.068 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:38:32.068 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:38:32.068 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.068 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:38:32.068 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:38:32.069 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:32.069 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.069 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.069 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:38:32.069 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:38:32.069 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:32.069 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:32.069 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:32.069 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.069 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:38:32.069 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:38:32.069 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:38:32.069 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:38:32.069 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:38:32.069 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:32.070 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:32.070 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.070 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:38:32.070 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:38:32.070 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.070 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:38:32.070 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:38:32.070 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:38:32.070 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:32.070 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:32.070 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:38:32.070 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.070 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:38:32.071 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:38:32.071 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:38:32.071 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:32.071 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:32.071 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:38:32.071 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.071 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:38:32.071 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:38:32.071 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:38:32.071 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:32.071 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:32.071 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.071 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:38:32.071 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:38:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.051+0000 7fbb823da640 1 -- 172.21.15.12:0/4150204487 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbb6803e240 msgr2=0x7fbb68040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.051+0000 7fbb823da640 1 --2- 172.21.15.12:0/4150204487 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbb6803e240 0x7fbb68040700 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbb74007880 tx=0x7fbb740098d0 comp rx=0 tx=0).stop 2024-09-11T01:38:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.051+0000 7fbb823da640 10 monclient: shutdown 2024-09-11T01:38:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.051+0000 7fbb823da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.051+0000 7fbb823da640 1 -- 172.21.15.12:0/4150204487 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbb7c138230 msgr2=0x7fbb7c137050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.051+0000 7fbb823da640 1 --2- 172.21.15.12:0/4150204487 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbb7c138230 0x7fbb7c137050 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbb70031000 tx=0x7fbb700046c0 comp rx=0 tx=0).stop 2024-09-11T01:38:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.051+0000 7fbb80977640 1 -- 172.21.15.12:0/4150204487 reap_dead start 2024-09-11T01:38:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.051+0000 7fbb80977640 1 -- 172.21.15.12:0/4150204487 reap_dead start 2024-09-11T01:38:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.051+0000 7fbb823da640 1 -- 172.21.15.12:0/4150204487 shutdown_connections 2024-09-11T01:38:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.051+0000 7fbb823da640 1 -- 172.21.15.12:0/4150204487 >> 172.21.15.12:0/4150204487 conn(0x7fbb7c0a7020 msgr2=0x7fbb7c13e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.051+0000 7fbb823da640 1 -- 172.21.15.12:0/4150204487 shutdown_connections 2024-09-11T01:38:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.051+0000 7fbb823da640 1 -- 172.21.15.12:0/4150204487 wait complete. 2024-09-11T01:38:32.074 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:38:32.074 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:38:32.074 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:38:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.175+0000 7f7759c16640 10 monclient: get_monmap_and_config 2024-09-11T01:38:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.175+0000 7f7759c16640 10 monclient: build_initial_monmap 2024-09-11T01:38:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.175+0000 7f7759c16640 10 monclient: monmap: 2024-09-11T01:38:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:38:32.175920+0000 2024-09-11T01:38:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:38:32.175920+0000 2024-09-11T01:38:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:38:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:38:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:38:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:38:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:38:32.177 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.175+0000 7f7759c16640 1 Processor -- start 2024-09-11T01:38:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.175+0000 7f7759c16640 1 -- start start 2024-09-11T01:38:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.175+0000 7f7759c16640 10 monclient: init 2024-09-11T01:38:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.176+0000 7f7759c16640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.176+0000 7f7759c16640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-11T01:38:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.176+0000 7f7759c16640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7754137e50 addr v1:172.21.15.71:6789/0 2024-09-11T01:38:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.176+0000 7f7759c16640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7754138230 addr v1:172.21.15.12:6789/0 2024-09-11T01:38:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.176+0000 7f7759c16640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f775413e390 0x7f7754140780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.176+0000 7f7759c16640 10 monclient(hunting): picked mon.noname-c con 0x7f775413e390 addr v2:172.21.15.71:3300/0 2024-09-11T01:38:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.176+0000 7f7759c16640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77540a6580 con 0x7f7754138230 2024-09-11T01:38:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.176+0000 7f7759c16640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77540a62d0 con 0x7f7754137e50 2024-09-11T01:38:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.176+0000 7f7759c16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.176+0000 7f7759c16640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f775409bb40 con 0x7f775413e390 2024-09-11T01:38:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.176+0000 7f7759c16640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.176+0000 7f7759c16640 10 monclient(hunting): authenticate will time out at 1318.618507s 2024-09-11T01:38:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.176+0000 7f7752ffd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7754138230 0x7f7754138640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:51946/0 (socket says 172.21.15.12:51946) 2024-09-11T01:38:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.176+0000 7f7752ffd640 1 -- 172.21.15.12:0/3848544916 learned_addr learned my addr 172.21.15.12:0/3848544916 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:38:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.176+0000 7f7753fff640 1 --2- 172.21.15.12:0/3848544916 >> v2:172.21.15.71:3300/0 conn(0x7f775413e390 0x7f7754140780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.176+0000 7f7753fff640 10 monclient(hunting): get_auth_request con 0x7f775413e390 auth_method 0 2024-09-11T01:38:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.176+0000 7f7753fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.176+0000 7f7753fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.176+0000 7f7753fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:32.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.178+0000 7f7752ffd640 1 -- 172.21.15.12:0/3848544916 >> v1:172.21.15.12:6789/0 conn(0x7f7754138230 legacy=0x7f7754138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:32.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77537fe640 1 -- 172.21.15.12:0/3848544916 >> v1:172.21.15.71:6789/0 conn(0x7f7754137e50 legacy=0x7f775409d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:32.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f7753fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:32.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f7753fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:32.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 1 -- 172.21.15.12:0/3848544916 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3719173356 0 0) 0x7f77540a6580 con 0x7f7754138230 2024-09-11T01:38:32.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f7753fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:32.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 10 monclient(hunting): _negotiate 2024-09-11T01:38:32.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:32.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:32.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 10 monclient(hunting): my global_id is 94245 2024-09-11T01:38:32.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 1 -- 172.21.15.12:0/3848544916 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f773c003520 con 0x7f7754138230 2024-09-11T01:38:32.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77537fe640 1 -- 172.21.15.12:0/3848544916 >> v1:172.21.15.71:6789/0 conn(0x7f7754137e50 legacy=0x7f775409d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-11T01:38:32.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77537fe640 1 -- 172.21.15.12:0/3848544916 >> v1:172.21.15.71:6789/0 conn(0x7f7754137e50 legacy=0x7f775409d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:38:32.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77537fe640 1 --1- 172.21.15.12:0/3848544916 >> v1:172.21.15.71:6789/0 conn(0x7f7754137e50 0x7f775409d160 :-1 s=OPENED pgs=34 cs=1 l=1).handle_message read tag failed 2024-09-11T01:38:32.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77537fe640 1 --1- 172.21.15.12:0/3848544916 >> v1:172.21.15.71:6789/0 conn(0x7f7754137e50 0x7f775409d160 :-1 s=OPENED pgs=34 cs=1 l=1).fault on lossy channel, failing 2024-09-11T01:38:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.71:6789/0 2024-09-11T01:38:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 1 -- 172.21.15.12:0/3848544916 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 442253957 0 0) 0x7f773c003520 con 0x7f7754138230 2024-09-11T01:38:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 1 -- 172.21.15.12:0/3848544916 >> v2:172.21.15.71:3300/0 conn(0x7f775413e390 msgr2=0x7f7754140780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:38:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 1 --2- 172.21.15.12:0/3848544916 >> v2:172.21.15.71:3300/0 conn(0x7f775413e390 0x7f7754140780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 1 -- 172.21.15.12:0/3848544916 >> v1:172.21.15.71:6789/0 conn(0x7f7754137e50 legacy=0x7f775409d160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:38:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 10 monclient: _finish_hunting 0 2024-09-11T01:38:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:38:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:38:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 1 -- 172.21.15.12:0/3848544916 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f775409c4a0 con 0x7f7754138230 2024-09-11T01:38:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 10 monclient: _finish_auth 0 2024-09-11T01:38:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 10 monclient: _check_auth_tickets 2024-09-11T01:38:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 1 -- 172.21.15.12:0/3848544916 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f77480019a0 con 0x7f7754138230 2024-09-11T01:38:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:38:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 10 monclient: dump: 2024-09-11T01:38:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:32.186 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:38:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 1 -- 172.21.15.12:0/3848544916 >> v1:172.21.15.12:6789/0 conn(0x7f7754138230 legacy=0x7f7754138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:38:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 1 --2- 172.21.15.12:0/3848544916 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f773c003a50 0x7f773c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 10 monclient(hunting): picked mon.c con 0x7f773c003a50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 1 --2- 172.21.15.12:0/3848544916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f773c004800 0x7f773c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 10 monclient(hunting): picked mon.a con 0x7f773c004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:38:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 1 --2- 172.21.15.12:0/3848544916 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f773c0053c0 0x7f773c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 10 monclient(hunting): picked mon.b con 0x7f773c0053c0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:38:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 1 -- 172.21.15.12:0/3848544916 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f775409bb40 con 0x7f773c004800 2024-09-11T01:38:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 1 -- 172.21.15.12:0/3848544916 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f773c00c230 con 0x7f773c0053c0 2024-09-11T01:38:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 1 -- 172.21.15.12:0/3848544916 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f773c00c430 con 0x7f773c003a50 2024-09-11T01:38:32.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:32.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77527fc640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:38:32.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f7752ffd640 1 --2- 172.21.15.12:0/3848544916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f773c004800 0x7f773c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:32.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77537fe640 1 --2- 172.21.15.12:0/3848544916 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f773c003a50 0x7f773c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:32.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f7752ffd640 10 monclient(hunting): get_auth_request con 0x7f773c004800 auth_method 0 2024-09-11T01:38:32.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f7752ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:32.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f7752ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:32.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f7752ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:38:32.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77537fe640 10 monclient(hunting): get_auth_request con 0x7f773c003a50 auth_method 0 2024-09-11T01:38:32.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:32.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77537fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:32.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77537fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:38:32.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f7752ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:32.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f7752ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:32.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f7753fff640 1 --2- 172.21.15.12:0/3848544916 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f773c0053c0 0x7f773c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:32.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f7752ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:38:32.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:32.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:32.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77537fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:38:32.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f7753fff640 10 monclient(hunting): get_auth_request con 0x7f773c0053c0 auth_method 0 2024-09-11T01:38:32.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f7753fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:32.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f7753fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:32.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f7753fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:38:32.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f7753fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:32.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f7753fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:32.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f7753fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:38:32.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77537fe640 10 monclient(hunting): handle_auth_done global_id 94245 payload 293 2024-09-11T01:38:32.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77537fe640 1 -- 172.21.15.12:0/3848544916 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f773c0053c0 msgr2=0x7f773c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:38:32.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77537fe640 1 --2- 172.21.15.12:0/3848544916 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f773c0053c0 0x7f773c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:32.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77537fe640 1 -- 172.21.15.12:0/3848544916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f773c004800 msgr2=0x7f773c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:38:32.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77537fe640 1 --2- 172.21.15.12:0/3848544916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f773c004800 0x7f773c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:32.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77537fe640 10 monclient: _finish_hunting 0 2024-09-11T01:38:32.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77537fe640 1 monclient: found mon.c 2024-09-11T01:38:32.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:32.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77537fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:38:32.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77537fe640 1 -- 172.21.15.12:0/3848544916 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f773c003520 con 0x7f773c003a50 2024-09-11T01:38:32.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77537fe640 10 monclient: _finish_auth 0 2024-09-11T01:38:32.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77537fe640 10 monclient: _check_auth_tickets 2024-09-11T01:38:32.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f77537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:32.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f7753fff640 1 -- 172.21.15.12:0/3848544916 reap_dead start 2024-09-11T01:38:32.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f7759c16640 5 monclient: authenticate success, global_id 94245 2024-09-11T01:38:32.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.179+0000 7f7752ffd640 1 --2- 172.21.15.12:0/3848544916 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f773c004800 0x7f773c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:38:32.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.180+0000 7f7759c16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:38:32.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.182+0000 7f77537fe640 1 -- 172.21.15.12:0/3848544916 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f773c003a50 msgr2=0x7f773c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:32.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.182+0000 7f77537fe640 1 --2- 172.21.15.12:0/3848544916 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f773c003a50 0x7f773c003e70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7744007ca0 tx=0x7f773c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d145cf82b7612a5f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:32.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.182+0000 7f77527fc640 1 -- 172.21.15.12:0/3848544916 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77440079a0 con 0x7f773c003a50 2024-09-11T01:38:32.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.182+0000 7f77527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:32.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.182+0000 7f77527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:38:32.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.182+0000 7f77527fc640 10 monclient: dump: 2024-09-11T01:38:32.196 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:32.197 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:32.197 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.197 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.197 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:32.197 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:32.197 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:32.197 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:32.197 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:32.197 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:32.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.182+0000 7f77527fc640 1 -- 172.21.15.12:0/3848544916 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f774402e070 con 0x7f773c003a50 2024-09-11T01:38:32.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.182+0000 7f7759c16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:38:32.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.182+0000 7f77527fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:32.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.182+0000 7f77527fc640 1 -- 172.21.15.12:0/3848544916 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77440072b0 con 0x7f773c003a50 2024-09-11T01:38:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.182+0000 7f7759c16640 10 monclient: get_monmap_and_config success 2024-09-11T01:38:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.182+0000 7f7759c16640 4 set_mon_vals no callback set 2024-09-11T01:38:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.182+0000 7f77527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.182+0000 7f77527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:38:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.182+0000 7f77527fc640 10 monclient: dump: 2024-09-11T01:38:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:32.199 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:32.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.183+0000 7f7759c16640 10 monclient: shutdown 2024-09-11T01:38:32.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.183+0000 7f7759c16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:32.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.183+0000 7f7759c16640 1 -- 172.21.15.12:0/3848544916 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f773c003a50 msgr2=0x7f773c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:32.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.183+0000 7f7759c16640 1 --2- 172.21.15.12:0/3848544916 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f773c003a50 0x7f773c003e70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7744007ca0 tx=0x7f773c005830 comp rx=0 tx=0).stop 2024-09-11T01:38:32.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.183+0000 7f7759c16640 1 -- 172.21.15.12:0/3848544916 shutdown_connections 2024-09-11T01:38:32.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.183+0000 7f7759c16640 1 --2- 172.21.15.12:0/3848544916 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f773c003a50 0x7f773c003e70 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:32.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.183+0000 7f7759c16640 1 -- 172.21.15.12:0/3848544916 >> 172.21.15.12:0/3848544916 conn(0x7f77540a7020 msgr2=0x7f77540a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:32.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.183+0000 7f7759c16640 1 -- 172.21.15.12:0/3848544916 shutdown_connections 2024-09-11T01:38:32.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.183+0000 7f7759c16640 1 -- 172.21.15.12:0/3848544916 wait complete. 2024-09-11T01:38:32.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.184+0000 7f7759c16640 10 monclient: build_initial_monmap 2024-09-11T01:38:32.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.184+0000 7f7759c16640 10 monclient: monmap: 2024-09-11T01:38:32.199 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:32.200 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:38:32.200 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:38:32.200 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:38:32.200 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:32.200 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:32.200 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:38:32.200 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:38:32.200 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:38:32.200 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:32.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.184+0000 7f7759c16640 1 Processor -- start 2024-09-11T01:38:32.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.184+0000 7f7759c16640 1 -- start start 2024-09-11T01:38:32.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.184+0000 7f7759c16640 10 monclient: init 2024-09-11T01:38:32.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7759c16640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:32.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7759c16640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:38:32.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7759c16640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f773c003a50 0x7f7754120b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:32.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7759c16640 10 monclient(hunting): picked mon.noname-b con 0x7f773c003a50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:38:32.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7759c16640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f77541210b0 0x7f77541214d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:32.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7759c16640 10 monclient(hunting): picked mon.noname-c con 0x7f77541210b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:32.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7759c16640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77541cd520 0x7f77541cd980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:32.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7759c16640 10 monclient(hunting): picked mon.noname-a con 0x7f77541cd520 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:38:32.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7759c16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:32.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7759c16640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f775409d160 con 0x7f77541cd520 2024-09-11T01:38:32.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7759c16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:32.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7759c16640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f775409c840 con 0x7f773c003a50 2024-09-11T01:38:32.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7759c16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:32.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7759c16640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f775409d2e0 con 0x7f77541210b0 2024-09-11T01:38:32.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7759c16640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:32.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7759c16640 10 monclient(hunting): authenticate will time out at 1318.627274s 2024-09-11T01:38:32.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7752ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f77541210b0 0x7f77541214d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:32.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7753fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77541cd520 0x7f77541cd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:32.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7752ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f77541210b0 0x7f77541214d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:50998/0 (socket says 172.21.15.12:50998) 2024-09-11T01:38:32.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7753fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77541cd520 0x7f77541cd980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57842/0 (socket says 172.21.15.12:57842) 2024-09-11T01:38:32.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7752ffd640 1 -- 172.21.15.12:0/1344961455 learned_addr learned my addr 172.21.15.12:0/1344961455 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:38:32.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7752ffd640 10 monclient(hunting): get_auth_request con 0x7f77541210b0 auth_method 0 2024-09-11T01:38:32.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7752ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:32.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7752ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:32.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7752ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:32.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7753fff640 10 monclient(hunting): get_auth_request con 0x7f77541cd520 auth_method 0 2024-09-11T01:38:32.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7753fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:32.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7753fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:32.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7753fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:32.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7753fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:32.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7753fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:32.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7753fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:32.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7752ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:32.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7752ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:32.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7752ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:32.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f77537fe640 1 --2- 172.21.15.12:0/1344961455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f773c003a50 0x7f7754120b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:32.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f77537fe640 10 monclient(hunting): get_auth_request con 0x7f773c003a50 auth_method 0 2024-09-11T01:38:32.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f77537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:32.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f77537fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:32.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f77537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:32.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7753fff640 10 monclient(hunting): handle_auth_done global_id 94248 payload 824 2024-09-11T01:38:32.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7753fff640 1 -- 172.21.15.12:0/1344961455 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f77541210b0 msgr2=0x7f77541214d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:32.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7753fff640 1 --2- 172.21.15.12:0/1344961455 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f77541210b0 0x7f77541214d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7753fff640 1 -- 172.21.15.12:0/1344961455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f773c003a50 msgr2=0x7f7754120b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7753fff640 1 --2- 172.21.15.12:0/1344961455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f773c003a50 0x7f7754120b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7753fff640 10 monclient: _finish_hunting 0 2024-09-11T01:38:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7753fff640 1 monclient: found mon.noname-a 2024-09-11T01:38:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7753fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7753fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:38:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f7753fff640 1 -- 172.21.15.12:0/1344961455 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7754123430 con 0x7f77541cd520 2024-09-11T01:38:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.185+0000 7f77537fe640 1 --2- 172.21.15.12:0/1344961455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f773c003a50 0x7f7754120b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:38:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.187+0000 7f7753fff640 1 -- 172.21.15.12:0/1344961455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77541cd520 msgr2=0x7f77541cd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.187+0000 7f7753fff640 1 --2- 172.21.15.12:0/1344961455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77541cd520 0x7f77541cd980 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7754138dd0 tx=0x7f774c00f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.188+0000 7f773bfff640 1 -- 172.21.15.12:0/1344961455 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f774c015430 con 0x7f77541cd520 2024-09-11T01:38:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.188+0000 7f773bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.188+0000 7f773bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:38:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.188+0000 7f773bfff640 10 monclient: dump: 2024-09-11T01:38:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:32.205 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:32.205 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.205 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.205 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:32.205 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:32.205 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:32.205 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:32.205 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:32.205 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:32.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.188+0000 7f773bfff640 10 monclient: _finish_auth 0 2024-09-11T01:38:32.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.188+0000 7f773bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:38:32.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.188+0000 7f773bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:32.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.188+0000 7f7759c16640 5 monclient: authenticate success, global_id 94248 2024-09-11T01:38:32.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.188+0000 7f773bfff640 1 -- 172.21.15.12:0/1344961455 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f774c0155d0 con 0x7f77541cd520 2024-09-11T01:38:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.188+0000 7f773bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.188+0000 7f7759c16640 10 monclient: _renew_subs 2024-09-11T01:38:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.188+0000 7f773bfff640 1 -- 172.21.15.12:0/1344961455 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f774c0159c0 con 0x7f77541cd520 2024-09-11T01:38:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.188+0000 7f7759c16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:38:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.188+0000 7f7759c16640 1 -- 172.21.15.12:0/1344961455 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77540a1710 con 0x7f77541cd520 2024-09-11T01:38:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.188+0000 7f773bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.188+0000 7f773bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:38:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.188+0000 7f773bfff640 10 monclient: dump: 2024-09-11T01:38:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:32.207 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.188+0000 7f7751ffb640 4 set_mon_vals no callback set 2024-09-11T01:38:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.188+0000 7f7759c16640 10 monclient: _renew_subs 2024-09-11T01:38:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.188+0000 7f7759c16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:38:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.188+0000 7f7759c16640 1 -- 172.21.15.12:0/1344961455 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7754123150 con 0x7f77541cd520 2024-09-11T01:38:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.188+0000 7f7759c16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.188+0000 7f7759c16640 1 --2- 172.21.15.12:0/1344961455 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f77541cdec0 0x7f775409f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.188+0000 7f7759c16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.188+0000 7f7759c16640 1 -- 172.21.15.12:0/1344961455 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7754077410 con 0x7f77541cdec0 2024-09-11T01:38:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.189+0000 7f77537fe640 1 --2- 172.21.15.12:0/1344961455 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f77541cdec0 0x7f775409f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.189+0000 7f77537fe640 10 monclient: get_auth_request con 0x7f77541cdec0 auth_method 0 2024-09-11T01:38:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.189+0000 7f77537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.189+0000 7f77537fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.189+0000 7f77537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.189+0000 7f773bfff640 1 -- 172.21.15.12:0/1344961455 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f774c015b60 con 0x7f77541cd520 2024-09-11T01:38:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.189+0000 7f77537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.189+0000 7f77537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.189+0000 7f77537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.189+0000 7f77537fe640 10 monclient(hunting): handle_auth_done global_id 84344 payload 824 2024-09-11T01:38:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.189+0000 7f77537fe640 1 -- 172.21.15.12:0/1344961455 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7754138640 con 0x7f77541cdec0 2024-09-11T01:38:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.189+0000 7f773bfff640 1 --2- 172.21.15.12:0/1344961455 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f774003e210 0x7f77400406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.189+0000 7f773bfff640 1 -- 172.21.15.12:0/1344961455 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(320..320 src has 186..320) ==== 4783+0+0 (secure 0 0 0) 0x7f774c053c60 con 0x7f77541cd520 2024-09-11T01:38:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.189+0000 7f7752ffd640 1 --2- 172.21.15.12:0/1344961455 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f774003e210 0x7f77400406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.189+0000 7f7752ffd640 10 monclient: get_auth_request con 0x7f774003e210 auth_method 0 2024-09-11T01:38:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.191+0000 7f77537fe640 1 -- 172.21.15.12:0/1344961455 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f77541cdec0 msgr2=0x7f775409f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.191+0000 7f77537fe640 1 --2- 172.21.15.12:0/1344961455 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f77541cdec0 0x7f775409f2f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7744007c70 tx=0x7f7744001690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:32.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.191+0000 7f773bfff640 1 -- 172.21.15.12:0/1344961455 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7744043070 con 0x7f77541cdec0 2024-09-11T01:38:32.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.191+0000 7f773bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:32.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.191+0000 7f773bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:38:32.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.191+0000 7f773bfff640 10 monclient: dump: 2024-09-11T01:38:32.209 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:32.209 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:32.209 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.209 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.209 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:32.209 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:32.209 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:32.209 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:32.209 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:32.209 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:32.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.191+0000 7f773bfff640 1 -- 172.21.15.12:0/1344961455 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f774402e040 con 0x7f77541cdec0 2024-09-11T01:38:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.191+0000 7f773bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.191+0000 7f773bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:38:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.192+0000 7f773bfff640 1 -- 172.21.15.12:0/1344961455 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f77541cdec0 msgr2=0x7f775409f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.192+0000 7f773bfff640 1 --2- 172.21.15.12:0/1344961455 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f77541cdec0 0x7f775409f2f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7744007c70 tx=0x7f7744001690 comp rx=0 tx=0).stop 2024-09-11T01:38:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.192+0000 7f773bfff640 1 -- 172.21.15.12:0/1344961455 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f77541cdec0 msgr2=0x7f775409f2f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:38:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.192+0000 7f773bfff640 1 --2- 172.21.15.12:0/1344961455 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f77541cdec0 0x7f775409f2f0 secure :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7744007c70 tx=0x7f7744001690 comp rx=0 tx=0).stop 2024-09-11T01:38:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.192+0000 7f7752ffd640 1 --2- 172.21.15.12:0/1344961455 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f774003e210 0x7f77400406d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f77480014a0 tx=0x7f7748001e40 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.198+0000 7f7759c16640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:38:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.198+0000 7f7759c16640 1 --2- 172.21.15.12:0/1344961455 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f775409f830 0x7f775409fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.198+0000 7f7759c16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.198+0000 7f7759c16640 1 -- 172.21.15.12:0/1344961455 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7754121f90 con 0x7f775409f830 2024-09-11T01:38:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.198+0000 7f77537fe640 1 --2- 172.21.15.12:0/1344961455 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f775409f830 0x7f775409fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.198+0000 7f77537fe640 10 monclient: get_auth_request con 0x7f775409f830 auth_method 0 2024-09-11T01:38:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.198+0000 7f77537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.198+0000 7f77537fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.198+0000 7f77537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.198+0000 7f77537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.198+0000 7f77537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.198+0000 7f77537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.198+0000 7f77537fe640 10 monclient(hunting): handle_auth_done global_id 84347 payload 824 2024-09-11T01:38:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.198+0000 7f77537fe640 1 -- 172.21.15.12:0/1344961455 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f77541ce2a0 con 0x7f775409f830 2024-09-11T01:38:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.200+0000 7f77537fe640 1 -- 172.21.15.12:0/1344961455 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f775409f830 msgr2=0x7f775409fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.200+0000 7f77537fe640 1 --2- 172.21.15.12:0/1344961455 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f775409f830 0x7f775409fc40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f77540a5db0 tx=0x7f7744001620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.200+0000 7f773bfff640 1 -- 172.21.15.12:0/1344961455 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7744043050 con 0x7f775409f830 2024-09-11T01:38:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.200+0000 7f773bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.200+0000 7f773bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:38:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.200+0000 7f773bfff640 10 monclient: dump: 2024-09-11T01:38:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:32.214 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.201+0000 7f773bfff640 1 -- 172.21.15.12:0/1344961455 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7f7744044040 con 0x7f775409f830 2024-09-11T01:38:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.201+0000 7f773bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:38:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.201+0000 7f773bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:38:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.201+0000 7f773bfff640 1 -- 172.21.15.12:0/1344961455 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f775409f830 msgr2=0x7f775409fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.201+0000 7f773bfff640 1 --2- 172.21.15.12:0/1344961455 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f775409f830 0x7f775409fc40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f77540a5db0 tx=0x7f7744001620 comp rx=0 tx=0).stop 2024-09-11T01:38:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.201+0000 7f773bfff640 1 -- 172.21.15.12:0/1344961455 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f775409f830 msgr2=0x7f775409fc40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:38:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.201+0000 7f773bfff640 1 --2- 172.21.15.12:0/1344961455 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f775409f830 0x7f775409fc40 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f77540a5db0 tx=0x7f7744001620 comp rx=0 tx=0).stop 2024-09-11T01:38:32.215 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:38:32.215 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:38:32.215 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:38:32.215 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:38:32.215 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 92, 2024-09-11T01:38:32.215 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 106224, 2024-09-11T01:38:32.215 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:38:32.215 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:38:32.215 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:38:32.215 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:38:32.215 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:32.215 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 7, 2024-09-11T01:38:32.215 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:38:32.215 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:38:32.216 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:38:32.216 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:38:32.216 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:38:32.216 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:38:32.216 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:38:32.216 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:38:32.216 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:38:32.216 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:38:32.216 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:38:32.216 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:38:32.216 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:38:32.216 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:38:32.216 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:32.216 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:38:32.216 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:38:32.217 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:38:32.217 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:38:32.217 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:38:32.217 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:38:32.217 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:38:32.217 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:38:32.217 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:38:32.217 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:38:32.217 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:38:32.217 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:38:32.217 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:38:32.217 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:32.217 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.217 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:38:32.218 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:38:32.218 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:38:32.218 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:38:32.218 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:38:32.218 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:38:32.218 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:38:32.218 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:38:32.218 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:38:32.218 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:38:32.218 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:38:32.218 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:38:32.218 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:38:32.218 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:38:32.218 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:38:32.219 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:38:32.219 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:38:32.219 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:38:32.219 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:38:32.219 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:38:32.219 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:38:32.219 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:38:32.219 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:38:32.219 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:38:32.219 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:38:32.219 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:38:32.219 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:38:32.219 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:38:32.219 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:32.219 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:38:32.220 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.220 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:38:32.220 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.220 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:38:32.220 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:38:32.220 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:38:32.220 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:38:32.220 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.220 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:38:32.220 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:38:32.220 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:32.220 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.220 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.220 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:38:32.220 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:38:32.221 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:32.221 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:32.221 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:32.221 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.221 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:38:32.221 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:38:32.221 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:38:32.221 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:38:32.221 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:38:32.221 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.221 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.221 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:38:32.221 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:38:32.221 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:38:32.221 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:38:32.222 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.222 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:38:32.222 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:38:32.222 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:32.222 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.222 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.222 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:38:32.222 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:38:32.222 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:32.222 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:32.222 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:32.222 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.222 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:38:32.222 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:38:32.222 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:38:32.223 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:38:32.223 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:38:32.223 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.223 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.223 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:38:32.223 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:38:32.223 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:38:32.223 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:38:32.223 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.223 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:38:32.223 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:38:32.223 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:32.223 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.223 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.223 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:38:32.224 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:38:32.224 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:32.224 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:32.224 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:32.224 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.224 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:38:32.224 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:38:32.224 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:38:32.224 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:38:32.224 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:38:32.224 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:32.224 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:32.224 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.224 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:38:32.225 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:38:32.225 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.225 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:38:32.225 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:38:32.225 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:38:32.225 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:32.225 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:32.225 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:38:32.225 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.225 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:38:32.225 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:38:32.225 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:38:32.225 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:32.225 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:38:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:38:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:38:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:38:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:38:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:38:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:38:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:38:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.227 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:38:32.227 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:38:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.201+0000 7f7759c16640 1 -- 172.21.15.12:0/1344961455 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f774003e210 msgr2=0x7f77400406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.201+0000 7f7759c16640 1 --2- 172.21.15.12:0/1344961455 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f774003e210 0x7f77400406d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f77480014a0 tx=0x7f7748001e40 comp rx=0 tx=0).stop 2024-09-11T01:38:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.201+0000 7f7759c16640 10 monclient: shutdown 2024-09-11T01:38:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.201+0000 7f7759c16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.201+0000 7f7759c16640 1 -- 172.21.15.12:0/1344961455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77541cd520 msgr2=0x7f77541cd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.201+0000 7f7759c16640 1 --2- 172.21.15.12:0/1344961455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77541cd520 0x7f77541cd980 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7754138dd0 tx=0x7f774c00f1b0 comp rx=0 tx=0).stop 2024-09-11T01:38:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.201+0000 7f7753fff640 1 -- 172.21.15.12:0/1344961455 reap_dead start 2024-09-11T01:38:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.201+0000 7f7759c16640 1 -- 172.21.15.12:0/1344961455 shutdown_connections 2024-09-11T01:38:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.201+0000 7f7759c16640 1 --2- 172.21.15.12:0/1344961455 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f77541cd520 0x7f77541cd980 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.201+0000 7f7759c16640 1 -- 172.21.15.12:0/1344961455 >> 172.21.15.12:0/1344961455 conn(0x7f77540a7020 msgr2=0x7f77540a8420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.201+0000 7f7759c16640 1 -- 172.21.15.12:0/1344961455 shutdown_connections 2024-09-11T01:38:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.201+0000 7f7759c16640 1 -- 172.21.15.12:0/1344961455 wait complete. 2024-09-11T01:38:32.229 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:38:32.229 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:38:32.229 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:38:32.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.329+0000 7fc0236c6640 10 monclient: get_monmap_and_config 2024-09-11T01:38:32.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.329+0000 7fc0236c6640 10 monclient: build_initial_monmap 2024-09-11T01:38:32.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.329+0000 7fc0236c6640 10 monclient: monmap: 2024-09-11T01:38:32.331 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:32.331 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:32.331 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:38:32.330464+0000 2024-09-11T01:38:32.331 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:38:32.330464+0000 2024-09-11T01:38:32.331 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:32.331 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:32.331 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:38:32.331 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:38:32.331 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:38:32.331 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:38:32.332 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:38:32.332 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.330+0000 7fc0236c6640 1 Processor -- start 2024-09-11T01:38:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.330+0000 7fc0236c6640 1 -- start start 2024-09-11T01:38:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.330+0000 7fc0236c6640 10 monclient: init 2024-09-11T01:38:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc0236c6640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc0236c6640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-11T01:38:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc0236c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc01c137e50 addr v1:172.21.15.12:6789/0 2024-09-11T01:38:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc0236c6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc01c138230 addr v1:172.21.15.71:6789/0 2024-09-11T01:38:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc0236c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc01c13e390 0x7fc01c140780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc0236c6640 10 monclient(hunting): picked mon.noname-b con 0x7fc01c13e390 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc0236c6640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc01c0a6580 con 0x7fc01c137e50 2024-09-11T01:38:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc0236c6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc01c0a62d0 con 0x7fc01c138230 2024-09-11T01:38:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc0236c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc0236c6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc01c09bb40 con 0x7fc01c13e390 2024-09-11T01:38:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc0236c6640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc0236c6640 10 monclient(hunting): authenticate will time out at 1318.773250s 2024-09-11T01:38:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc021462640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc01c137e50 0x7fc01c09d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:51950/0 (socket says 172.21.15.12:51950) 2024-09-11T01:38:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc021462640 1 -- 172.21.15.12:0/3247608848 learned_addr learned my addr 172.21.15.12:0/3247608848 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:38:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc021c63640 1 --2- 172.21.15.12:0/3247608848 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc01c13e390 0x7fc01c140780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc021c63640 10 monclient(hunting): get_auth_request con 0x7fc01c13e390 auth_method 0 2024-09-11T01:38:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc021c63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc021c63640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc021c63640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc021c63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc021c63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc021c63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc021c63640 10 monclient(hunting): handle_auth_done global_id 84350 payload 274 2024-09-11T01:38:32.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc021c63640 1 -- 172.21.15.12:0/3247608848 >> v1:172.21.15.71:6789/0 conn(0x7fc01c138230 legacy=0x7fc01c138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:38:32.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc021c63640 1 -- 172.21.15.12:0/3247608848 >> v1:172.21.15.12:6789/0 conn(0x7fc01c137e50 legacy=0x7fc01c09d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:38:32.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc021c63640 10 monclient: _finish_hunting 0 2024-09-11T01:38:32.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc021c63640 1 monclient: found mon.noname-b 2024-09-11T01:38:32.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc021c63640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:32.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc021c63640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:38:32.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.331+0000 7fc021c63640 1 -- 172.21.15.12:0/3247608848 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc01c09c4a0 con 0x7fc01c13e390 2024-09-11T01:38:32.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.334+0000 7fc021c63640 1 -- 172.21.15.12:0/3247608848 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc01c13e390 msgr2=0x7fc01c140780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:32.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.334+0000 7fc021c63640 1 --2- 172.21.15.12:0/3247608848 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc01c13e390 0x7fc01c140780 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc014009dc0 tx=0x7fc01402a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=450f1eccb322da0c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:32.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.334+0000 7fc013fff640 1 -- 172.21.15.12:0/3247608848 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc014007a30 con 0x7fc01c13e390 2024-09-11T01:38:32.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.334+0000 7fc013fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:32.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.334+0000 7fc013fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:38:32.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.334+0000 7fc013fff640 10 monclient: dump: 2024-09-11T01:38:32.337 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:32.337 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:32.337 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.337 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.337 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:32.337 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:32.337 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:32.338 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:32.338 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:32.338 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:32.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.334+0000 7fc013fff640 10 monclient: _finish_auth 0 2024-09-11T01:38:32.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.334+0000 7fc013fff640 10 monclient: _check_auth_tickets 2024-09-11T01:38:32.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.334+0000 7fc013fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:32.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.334+0000 7fc0236c6640 5 monclient: authenticate success, global_id 84350 2024-09-11T01:38:32.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.334+0000 7fc0236c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:38:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.337+0000 7fc013fff640 1 -- 172.21.15.12:0/3247608848 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc014007bd0 con 0x7fc01c13e390 2024-09-11T01:38:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.337+0000 7fc013fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.337+0000 7fc013fff640 1 -- 172.21.15.12:0/3247608848 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc014031400 con 0x7fc01c13e390 2024-09-11T01:38:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.337+0000 7fc013fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.337+0000 7fc013fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:38:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.337+0000 7fc013fff640 10 monclient: dump: 2024-09-11T01:38:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.339 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:32.340 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.337+0000 7fc0236c6640 10 monclient: get_monmap_and_config success 2024-09-11T01:38:32.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.337+0000 7fc0236c6640 4 set_mon_vals no callback set 2024-09-11T01:38:32.341 INFO:tasks.rados.rados.0.smithi071.stdout:1416: done (3 left) 2024-09-11T01:38:32.341 INFO:tasks.rados.rados.0.smithi071.stdout:1417: read oid 41 snap -1 2024-09-11T01:38:32.341 INFO:tasks.rados.rados.0.smithi071.stdout:1417: expect deleted 2024-09-11T01:38:32.341 INFO:tasks.rados.rados.0.smithi071.stdout:1411: finishing rollback tid 0 to smithi07134777-42 2024-09-11T01:38:32.341 INFO:tasks.rados.rados.0.smithi071.stdout:1411: finishing rollback tid 1 to smithi07134777-42 2024-09-11T01:38:32.341 INFO:tasks.rados.rados.0.smithi071.stdout:1411: finishing rollback tid 2 to smithi07134777-42 2024-09-11T01:38:32.341 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 437 (ObjNum 488 snap 132 seq_num 488) dirty exists 2024-09-11T01:38:32.341 INFO:tasks.rados.rados.0.smithi071.stdout:1412: finishing copy_from racing read to smithi07134777-22 2024-09-11T01:38:32.342 INFO:tasks.rados.rados.0.smithi071.stdout:1412: finishing copy_from to smithi07134777-22 2024-09-11T01:38:32.342 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 434 (ObjNum 503 snap 135 seq_num 503) dirty exists 2024-09-11T01:38:32.342 INFO:tasks.rados.rados.0.smithi071.stdout:1414: finishing write tid 1 to smithi07134777-43 2024-09-11T01:38:32.342 INFO:tasks.rados.rados.0.smithi071.stdout:1414: finishing write tid 2 to smithi07134777-43 2024-09-11T01:38:32.342 INFO:tasks.rados.rados.0.smithi071.stdout:1414: finishing write tid 3 to smithi07134777-43 2024-09-11T01:38:32.342 INFO:tasks.rados.rados.0.smithi071.stdout:1414: finishing write tid 4 to smithi07134777-43 2024-09-11T01:38:32.342 INFO:tasks.rados.rados.0.smithi071.stdout:1414: finishing write tid 5 to smithi07134777-43 2024-09-11T01:38:32.342 INFO:tasks.rados.rados.0.smithi071.stdout:1414: oid 43 updating version 0 to 674 2024-09-11T01:38:32.342 INFO:tasks.rados.rados.0.smithi071.stdout:1414: oid 43 updating version 674 to 675 2024-09-11T01:38:32.342 INFO:tasks.rados.rados.0.smithi071.stdout:1414: oid 43 updating version 675 to 676 2024-09-11T01:38:32.342 INFO:tasks.rados.rados.0.smithi071.stdout:1414: oid 43 updating version 676 to 677 2024-09-11T01:38:32.342 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 677 (ObjNum 534 snap 148 seq_num 534) dirty exists 2024-09-11T01:38:32.342 INFO:tasks.rados.rados.0.smithi071.stdout:1414: left oid 43 (ObjNum 534 snap 148 seq_num 534) 2024-09-11T01:38:32.342 INFO:tasks.rados.rados.0.smithi071.stdout:1411: done (3 left) 2024-09-11T01:38:32.342 INFO:tasks.rados.rados.0.smithi071.stdout:1412: done (2 left) 2024-09-11T01:38:32.343 INFO:tasks.rados.rados.0.smithi071.stdout:1414: done (1 left) 2024-09-11T01:38:32.343 INFO:tasks.rados.rados.0.smithi071.stdout:1418: snap_create 2024-09-11T01:38:32.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.338+0000 7fc0236c6640 10 monclient: shutdown 2024-09-11T01:38:32.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.338+0000 7fc0236c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:32.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.338+0000 7fc0236c6640 1 -- 172.21.15.12:0/3247608848 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc01c13e390 msgr2=0x7fc01c140780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:32.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.338+0000 7fc0236c6640 1 --2- 172.21.15.12:0/3247608848 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc01c13e390 0x7fc01c140780 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc014009dc0 tx=0x7fc01402a480 comp rx=0 tx=0).stop 2024-09-11T01:38:32.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.338+0000 7fc0236c6640 1 -- 172.21.15.12:0/3247608848 shutdown_connections 2024-09-11T01:38:32.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.338+0000 7fc0236c6640 1 --2- 172.21.15.12:0/3247608848 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc01c13e390 0x7fc01c140780 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:32.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.338+0000 7fc0236c6640 1 -- 172.21.15.12:0/3247608848 >> 172.21.15.12:0/3247608848 conn(0x7fc01c0a7020 msgr2=0x7fc01c0a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:32.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.339+0000 7fc0236c6640 1 -- 172.21.15.12:0/3247608848 shutdown_connections 2024-09-11T01:38:32.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.339+0000 7fc0236c6640 1 -- 172.21.15.12:0/3247608848 wait complete. 2024-09-11T01:38:32.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.340+0000 7fc0236c6640 10 monclient: build_initial_monmap 2024-09-11T01:38:32.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.340+0000 7fc0236c6640 10 monclient: monmap: 2024-09-11T01:38:32.346 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:32.346 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:38:32.346 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:38:32.346 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:38:32.346 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:32.346 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:32.346 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:38:32.346 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:38:32.346 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:38:32.346 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:32.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.340+0000 7fc0236c6640 1 Processor -- start 2024-09-11T01:38:32.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.340+0000 7fc0236c6640 1 -- start start 2024-09-11T01:38:32.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.340+0000 7fc0236c6640 10 monclient: init 2024-09-11T01:38:32.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc0236c6640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:32.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc0236c6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:38:32.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc0236c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc01c137e50 0x7fc01c137020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:32.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc0236c6640 10 monclient(hunting): picked mon.noname-c con 0x7fc01c137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:32.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc0236c6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc01c138230 0x7fc01c13a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:32.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc0236c6640 10 monclient(hunting): picked mon.noname-a con 0x7fc01c138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:38:32.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc0236c6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c13e390 0x7fc01c13a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:32.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc0236c6640 10 monclient(hunting): picked mon.noname-b con 0x7fc01c13e390 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:38:32.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc0236c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:32.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc0236c6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc01c140e40 con 0x7fc01c138230 2024-09-11T01:38:32.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc0236c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:32.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc0236c6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc01c140cc0 con 0x7fc01c13e390 2024-09-11T01:38:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc0236c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc0236c6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc01c140fc0 con 0x7fc01c137e50 2024-09-11T01:38:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc0236c6640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc0236c6640 10 monclient(hunting): authenticate will time out at 1318.783577s 2024-09-11T01:38:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc021462640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc01c137e50 0x7fc01c137020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc021462640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc01c137e50 0x7fc01c137020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51034/0 (socket says 172.21.15.12:51034) 2024-09-11T01:38:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc021462640 1 -- 172.21.15.12:0/1336595666 learned_addr learned my addr 172.21.15.12:0/1336595666 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:38:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc021462640 10 monclient(hunting): get_auth_request con 0x7fc01c137e50 auth_method 0 2024-09-11T01:38:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc021462640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc021462640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc021462640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc021462640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc021462640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc021462640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc021c63640 1 --2- 172.21.15.12:0/1336595666 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c13e390 0x7fc01c13a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc021c63640 10 monclient(hunting): get_auth_request con 0x7fc01c13e390 auth_method 0 2024-09-11T01:38:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc021c63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:32.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc021c63640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:32.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.341+0000 7fc021c63640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:32.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.342+0000 7fc021c63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:32.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.342+0000 7fc021c63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:32.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.342+0000 7fc021c63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:32.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.342+0000 7fc021462640 10 monclient(hunting): handle_auth_done global_id 84353 payload 824 2024-09-11T01:38:32.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.342+0000 7fc021462640 1 -- 172.21.15.12:0/1336595666 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c13e390 msgr2=0x7fc01c13a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:32.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.342+0000 7fc021462640 1 --2- 172.21.15.12:0/1336595666 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c13e390 0x7fc01c13a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:32.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.342+0000 7fc021462640 1 -- 172.21.15.12:0/1336595666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc01c138230 msgr2=0x7fc01c13a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:32.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.342+0000 7fc021462640 1 --2- 172.21.15.12:0/1336595666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc01c138230 0x7fc01c13a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:32.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.342+0000 7fc021462640 10 monclient: _finish_hunting 0 2024-09-11T01:38:32.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.342+0000 7fc021462640 1 monclient: found mon.noname-c 2024-09-11T01:38:32.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.342+0000 7fc021462640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:32.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.342+0000 7fc021462640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:38:32.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.342+0000 7fc021462640 1 -- 172.21.15.12:0/1336595666 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc01c0a1710 con 0x7fc01c137e50 2024-09-11T01:38:32.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.344+0000 7fc021462640 1 -- 172.21.15.12:0/1336595666 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc01c137e50 msgr2=0x7fc01c137020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:32.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.344+0000 7fc021462640 1 --2- 172.21.15.12:0/1336595666 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc01c137e50 0x7fc01c137020 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc01800c490 tx=0x7fc01800c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:32.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.344+0000 7fc011ffb640 1 -- 172.21.15.12:0/1336595666 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc018007070 con 0x7fc01c137e50 2024-09-11T01:38:32.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.344+0000 7fc011ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:32.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.344+0000 7fc011ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:38:32.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.344+0000 7fc011ffb640 10 monclient: dump: 2024-09-11T01:38:32.356 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:32.356 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:32.356 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.356 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:32.357 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.344+0000 7fc011ffb640 10 monclient: _finish_auth 0 2024-09-11T01:38:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.344+0000 7fc011ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:38:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.344+0000 7fc011ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.344+0000 7fc011ffb640 1 -- 172.21.15.12:0/1336595666 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc018007230 con 0x7fc01c137e50 2024-09-11T01:38:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.344+0000 7fc0236c6640 5 monclient: authenticate success, global_id 84353 2024-09-11T01:38:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.344+0000 7fc0236c6640 10 monclient: _renew_subs 2024-09-11T01:38:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.344+0000 7fc0236c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:38:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.344+0000 7fc0236c6640 1 -- 172.21.15.12:0/1336595666 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc01c13bbe0 con 0x7fc01c137e50 2024-09-11T01:38:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.344+0000 7fc011ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.344+0000 7fc011ffb640 1 -- 172.21.15.12:0/1336595666 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc018007620 con 0x7fc01c137e50 2024-09-11T01:38:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.344+0000 7fc011ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.344+0000 7fc011ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:38:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.344+0000 7fc011ffb640 10 monclient: dump: 2024-09-11T01:38:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:32.359 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:32.359 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:32.359 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:32.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.344+0000 7fc0137fe640 4 set_mon_vals no callback set 2024-09-11T01:38:32.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.345+0000 7fc0236c6640 10 monclient: _renew_subs 2024-09-11T01:38:32.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.345+0000 7fc0236c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:38:32.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.345+0000 7fc0236c6640 1 -- 172.21.15.12:0/1336595666 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc01c13cb80 con 0x7fc01c137e50 2024-09-11T01:38:32.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.345+0000 7fc0236c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:32.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.345+0000 7fc0236c6640 1 --2- 172.21.15.12:0/1336595666 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c13c740 0x7fc01c13aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:32.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.345+0000 7fc0236c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:32.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.345+0000 7fc0236c6640 1 -- 172.21.15.12:0/1336595666 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc01c1c64a0 con 0x7fc01c13c740 2024-09-11T01:38:32.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.346+0000 7fc011ffb640 1 -- 172.21.15.12:0/1336595666 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fc01801b400 con 0x7fc01c137e50 2024-09-11T01:38:32.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.346+0000 7fc011ffb640 1 --2- 172.21.15.12:0/1336595666 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc00003e260 0x7fc000040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:32.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.346+0000 7fc020c61640 1 --2- 172.21.15.12:0/1336595666 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c13c740 0x7fc01c13aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.346+0000 7fc011ffb640 1 -- 172.21.15.12:0/1336595666 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(321..321 src has 186..321) ==== 4783+0+0 (secure 0 0 0) 0x7fc0180529b0 con 0x7fc01c137e50 2024-09-11T01:38:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.346+0000 7fc020c61640 10 monclient: get_auth_request con 0x7fc01c13c740 auth_method 0 2024-09-11T01:38:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.346+0000 7fc020c61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.346+0000 7fc020c61640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.346+0000 7fc020c61640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.346+0000 7fc020c61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.346+0000 7fc020c61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.346+0000 7fc020c61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.346+0000 7fc021c63640 1 --2- 172.21.15.12:0/1336595666 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc00003e260 0x7fc000040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.346+0000 7fc021c63640 10 monclient: get_auth_request con 0x7fc00003e260 auth_method 0 2024-09-11T01:38:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.346+0000 7fc020c61640 10 monclient(hunting): handle_auth_done global_id 104128 payload 824 2024-09-11T01:38:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.346+0000 7fc020c61640 1 -- 172.21.15.12:0/1336595666 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc01c138610 con 0x7fc01c13c740 2024-09-11T01:38:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.348+0000 7fc020c61640 1 -- 172.21.15.12:0/1336595666 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c13c740 msgr2=0x7fc01c13aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.349+0000 7fc020c61640 1 --2- 172.21.15.12:0/1336595666 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c13c740 0x7fc01c13aba0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc00c00c310 tx=0x7fc00c00c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.349+0000 7fc021c63640 1 --2- 172.21.15.12:0/1336595666 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc00003e260 0x7fc000040720 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc01c0ab580 tx=0x7fc01402f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.349+0000 7fc011ffb640 1 -- 172.21.15.12:0/1336595666 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc00c017070 con 0x7fc01c13c740 2024-09-11T01:38:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.349+0000 7fc011ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.349+0000 7fc011ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:38:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.349+0000 7fc011ffb640 10 monclient: dump: 2024-09-11T01:38:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:32.362 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.349+0000 7fc011ffb640 1 -- 172.21.15.12:0/1336595666 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc00c0047d0 con 0x7fc01c13c740 2024-09-11T01:38:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.349+0000 7fc011ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.349+0000 7fc011ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:38:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.349+0000 7fc011ffb640 1 -- 172.21.15.12:0/1336595666 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c13c740 msgr2=0x7fc01c13aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.349+0000 7fc011ffb640 1 --2- 172.21.15.12:0/1336595666 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c13c740 0x7fc01c13aba0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc00c00c310 tx=0x7fc00c00c7d0 comp rx=0 tx=0).stop 2024-09-11T01:38:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.349+0000 7fc011ffb640 1 -- 172.21.15.12:0/1336595666 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c13c740 msgr2=0x7fc01c13aba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:38:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.349+0000 7fc011ffb640 1 --2- 172.21.15.12:0/1336595666 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c13c740 0x7fc01c13aba0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:32.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.355+0000 7fc0236c6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:38:32.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.355+0000 7fc0236c6640 1 --2- 172.21.15.12:0/1336595666 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c13b0e0 0x7fc01c0a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:32.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.355+0000 7fc0236c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:32.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.355+0000 7fc0236c6640 1 -- 172.21.15.12:0/1336595666 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc01c13d310 con 0x7fc01c13b0e0 2024-09-11T01:38:32.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.356+0000 7fc020c61640 1 --2- 172.21.15.12:0/1336595666 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c13b0e0 0x7fc01c0a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:32.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.356+0000 7fc020c61640 10 monclient: get_auth_request con 0x7fc01c13b0e0 auth_method 0 2024-09-11T01:38:32.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.356+0000 7fc020c61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:32.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.356+0000 7fc020c61640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:32.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.356+0000 7fc020c61640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:32.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.356+0000 7fc020c61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.356+0000 7fc020c61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.356+0000 7fc020c61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.356+0000 7fc020c61640 10 monclient(hunting): handle_auth_done global_id 104131 payload 824 2024-09-11T01:38:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.356+0000 7fc020c61640 1 -- 172.21.15.12:0/1336595666 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc01c069950 con 0x7fc01c13b0e0 2024-09-11T01:38:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.358+0000 7fc020c61640 1 -- 172.21.15.12:0/1336595666 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c13b0e0 msgr2=0x7fc01c0a18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.358+0000 7fc020c61640 1 --2- 172.21.15.12:0/1336595666 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c13b0e0 0x7fc01c0a18e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc00c00c310 tx=0x7fc00c003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.358+0000 7fc011ffb640 1 -- 172.21.15.12:0/1336595666 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc00c017030 con 0x7fc01c13b0e0 2024-09-11T01:38:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.358+0000 7fc011ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.358+0000 7fc011ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:38:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.358+0000 7fc011ffb640 10 monclient: dump: 2024-09-11T01:38:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:32.366 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.359+0000 7fc011ffb640 1 -- 172.21.15.12:0/1336595666 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4540 (secure 0 0 0) 0x7fc00c0223d0 con 0x7fc01c13b0e0 2024-09-11T01:38:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.359+0000 7fc011ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:38:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.359+0000 7fc011ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:38:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.359+0000 7fc011ffb640 1 -- 172.21.15.12:0/1336595666 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c13b0e0 msgr2=0x7fc01c0a18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.359+0000 7fc011ffb640 1 --2- 172.21.15.12:0/1336595666 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c13b0e0 0x7fc01c0a18e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc00c00c310 tx=0x7fc00c003760 comp rx=0 tx=0).stop 2024-09-11T01:38:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.359+0000 7fc011ffb640 1 -- 172.21.15.12:0/1336595666 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c13b0e0 msgr2=0x7fc01c0a18e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:38:32.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.359+0000 7fc011ffb640 1 --2- 172.21.15.12:0/1336595666 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c13b0e0 0x7fc01c0a18e0 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc00c00c310 tx=0x7fc00c003760 comp rx=0 tx=0).stop 2024-09-11T01:38:32.367 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:38:32.367 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:38:32.367 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:38:32.367 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:38:32.367 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 92, 2024-09-11T01:38:32.367 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 12853, 2024-09-11T01:38:32.367 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:38:32.367 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:38:32.367 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:38:32.367 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:38:32.368 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:32.368 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 7, 2024-09-11T01:38:32.368 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:38:32.368 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:38:32.368 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:38:32.368 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:38:32.368 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:38:32.368 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:38:32.368 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:38:32.368 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:38:32.368 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:38:32.368 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:38:32.369 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:38:32.369 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:38:32.369 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:38:32.369 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:38:32.369 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:32.369 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:38:32.369 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:38:32.369 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:38:32.369 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:38:32.369 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:38:32.369 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:38:32.369 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:38:32.369 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:38:32.369 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:38:32.370 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:38:32.370 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:38:32.370 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:38:32.370 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:38:32.370 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:32.370 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.370 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:38:32.370 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:38:32.370 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:38:32.370 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:38:32.370 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:38:32.370 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:38:32.370 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:38:32.370 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:38:32.371 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:38:32.371 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:38:32.371 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:38:32.371 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:38:32.371 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:38:32.371 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:38:32.371 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:38:32.371 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:38:32.371 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:38:32.371 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:38:32.371 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:38:32.371 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:38:32.371 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:38:32.371 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:38:32.371 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:38:32.372 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:38:32.372 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:38:32.372 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:38:32.372 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:38:32.372 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:38:32.372 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:32.372 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:38:32.372 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.372 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:38:32.372 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.372 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:38:32.372 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:38:32.372 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:38:32.373 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:38:32.373 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.373 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:38:32.373 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:38:32.373 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:32.373 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.373 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.373 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:38:32.373 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:38:32.373 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:32.373 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:32.373 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:32.373 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.373 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:38:32.374 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:38:32.374 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:38:32.374 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:38:32.374 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:38:32.374 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.374 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.374 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:38:32.374 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:38:32.374 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:38:32.374 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:38:32.374 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.374 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:38:32.374 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:38:32.374 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:32.375 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.375 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.375 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:38:32.375 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:38:32.375 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:32.375 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:32.375 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:32.375 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.375 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:38:32.375 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:38:32.375 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:38:32.375 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:38:32.375 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:38:32.375 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.375 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.376 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:38:32.376 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:38:32.376 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:38:32.376 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:38:32.376 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.376 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:38:32.376 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:38:32.376 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:32.376 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.376 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.376 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:38:32.376 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:38:32.376 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:32.377 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:32.377 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:32.377 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.377 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:38:32.377 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:38:32.377 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:38:32.377 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:38:32.377 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:38:32.377 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:32.377 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:32.377 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.377 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:38:32.377 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:38:32.378 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.378 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:38:32.378 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:38:32.378 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:38:32.378 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:32.378 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:32.378 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:38:32.378 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:32.378 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:38:32.378 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:38:32.378 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:38:32.378 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:32.378 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:32.378 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:32.379 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:38:32.379 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:38:32.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.359+0000 7fc0236c6640 1 -- 172.21.15.12:0/1336595666 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc00003e260 msgr2=0x7fc000040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:32.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.359+0000 7fc0236c6640 1 --2- 172.21.15.12:0/1336595666 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc00003e260 0x7fc000040720 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc01c0ab580 tx=0x7fc01402f000 comp rx=0 tx=0).stop 2024-09-11T01:38:32.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.359+0000 7fc0236c6640 10 monclient: shutdown 2024-09-11T01:38:32.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.359+0000 7fc0236c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:32.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.359+0000 7fc0236c6640 1 -- 172.21.15.12:0/1336595666 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc01c137e50 msgr2=0x7fc01c137020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:32.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.359+0000 7fc0236c6640 1 --2- 172.21.15.12:0/1336595666 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc01c137e50 0x7fc01c137020 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc01800c490 tx=0x7fc01800c950 comp rx=0 tx=0).stop 2024-09-11T01:38:32.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.359+0000 7fc021c63640 1 -- 172.21.15.12:0/1336595666 reap_dead start 2024-09-11T01:38:32.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.359+0000 7fc0236c6640 1 -- 172.21.15.12:0/1336595666 shutdown_connections 2024-09-11T01:38:32.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.359+0000 7fc0236c6640 1 --2- 172.21.15.12:0/1336595666 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc01c137e50 0x7fc01c137020 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:32.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.359+0000 7fc0236c6640 1 -- 172.21.15.12:0/1336595666 >> 172.21.15.12:0/1336595666 conn(0x7fc01c0a7020 msgr2=0x7fc01c13f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:32.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.359+0000 7fc0236c6640 1 -- 172.21.15.12:0/1336595666 shutdown_connections 2024-09-11T01:38:32.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:32.359+0000 7fc0236c6640 1 -- 172.21.15.12:0/1336595666 wait complete. 2024-09-11T01:38:32.381 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:38:32.381 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-11T01:38:32.381 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-11T01:38:32.381 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-11T01:38:32.381 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-11T01:38:32.381 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-11T01:38:32.382 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:38:32.393 INFO:tasks.ceph.mon.c:Stopped 2024-09-11T01:38:32.393 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-11T01:38:32.393 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-11T01:38:32.394 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T01:38:47.394 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-11T01:38:47.395 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-11T01:38:47.395 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:38:47.395 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:38:47.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.507+0000 7f192c272640 10 monclient: get_monmap_and_config 2024-09-11T01:38:47.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.507+0000 7f192c272640 10 monclient: build_initial_monmap 2024-09-11T01:38:47.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.507+0000 7f192c272640 10 monclient: monmap: 2024-09-11T01:38:47.508 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:47.509 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:47.509 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:38:47.507823+0000 2024-09-11T01:38:47.509 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:38:47.507823+0000 2024-09-11T01:38:47.509 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:47.509 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:47.509 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:38:47.509 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:38:47.509 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:38:47.509 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:38:47.509 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:38:47.509 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:47.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.507+0000 7f192c272640 1 Processor -- start 2024-09-11T01:38:47.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.507+0000 7f192c272640 1 -- start start 2024-09-11T01:38:47.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.507+0000 7f192c272640 10 monclient: init 2024-09-11T01:38:47.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.508+0000 7f192c272640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:47.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.508+0000 7f192c272640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-11T01:38:47.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.508+0000 7f192c272640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1924138230 0x7f1924138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:47.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.508+0000 7f192c272640 10 monclient(hunting): picked mon.noname-c con 0x7f1924138230 addr v2:172.21.15.71:3300/0 2024-09-11T01:38:47.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.508+0000 7f192c272640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f192409d030 addr v1:172.21.15.12:6789/0 2024-09-11T01:38:47.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.508+0000 7f192c272640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f192409d460 addr v1:172.21.15.71:6789/0 2024-09-11T01:38:47.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.508+0000 7f192c272640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19240a5e40 con 0x7f192409d030 2024-09-11T01:38:47.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.508+0000 7f192c272640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19240a6e30 con 0x7f192409d460 2024-09-11T01:38:47.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.508+0000 7f192c272640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:47.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.508+0000 7f192c272640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f192409c9c0 con 0x7f1924138230 2024-09-11T01:38:47.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.508+0000 7f192c272640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:47.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.508+0000 7f192c272640 10 monclient(hunting): authenticate will time out at 1333.950567s 2024-09-11T01:38:47.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.508+0000 7f192980d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f192409d030 0x7f19240a5020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50828/0 (socket says 172.21.15.12:50828) 2024-09-11T01:38:47.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.508+0000 7f192980d640 1 -- 172.21.15.12:0/299324182 learned_addr learned my addr 172.21.15.12:0/299324182 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:38:47.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.508+0000 7f192a00e640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1924138230 0x7f1924138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:47.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.509+0000 7f192a00e640 10 monclient(hunting): get_auth_request con 0x7f1924138230 auth_method 0 2024-09-11T01:38:47.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.509+0000 7f192a00e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:47.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.509+0000 7f192a00e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:47.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.509+0000 7f192a00e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:47.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.509+0000 7f192a00e640 1 -- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1924138230 msgr2=0x7f1924138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:38:47.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.509+0000 7f192a00e640 1 -- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1924138230 msgr2=0x7f1924138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:38:47.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.509+0000 7f192a00e640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1924138230 0x7f1924138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:38:47.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.509+0000 7f192a00e640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1924138230 0x7f1924138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:38:47.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.510+0000 7f192980d640 1 -- 172.21.15.12:0/299324182 >> v1:172.21.15.12:6789/0 conn(0x7f192409d030 legacy=0x7f19240a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:47.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.511+0000 7f192a80f640 1 -- 172.21.15.12:0/299324182 >> v1:172.21.15.71:6789/0 conn(0x7f192409d460 legacy=0x7f19240a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:47.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.710+0000 7f192a00e640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1924138230 0x7f1924138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:47.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.710+0000 7f192a00e640 10 monclient(hunting): get_auth_request con 0x7f1924138230 auth_method 0 2024-09-11T01:38:47.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.710+0000 7f192a00e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:47.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.710+0000 7f192a00e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:47.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.710+0000 7f192a00e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:38:47.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.711+0000 7f192a00e640 1 -- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1924138230 msgr2=0x7f1924138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:38:47.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.711+0000 7f192a00e640 1 -- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1924138230 msgr2=0x7f1924138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:38:47.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.711+0000 7f192a00e640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1924138230 0x7f1924138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:38:47.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:47.711+0000 7f192a00e640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1924138230 0x7f1924138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:38:48.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:48.113+0000 7f192a00e640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1924138230 0x7f1924138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:48.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:48.113+0000 7f192a00e640 10 monclient(hunting): get_auth_request con 0x7f1924138230 auth_method 0 2024-09-11T01:38:48.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:48.113+0000 7f192a00e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:48.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:48.113+0000 7f192a00e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:48.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:48.113+0000 7f192a00e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:38:48.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:48.113+0000 7f192a00e640 1 -- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1924138230 msgr2=0x7f1924138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:38:48.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:48.113+0000 7f192a00e640 1 -- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1924138230 msgr2=0x7f1924138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:38:48.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:48.113+0000 7f192a00e640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1924138230 0x7f1924138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:38:48.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:48.113+0000 7f192a00e640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1924138230 0x7f1924138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:38:48.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:48.914+0000 7f192a00e640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1924138230 0x7f1924138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:48.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:48.914+0000 7f192a00e640 10 monclient(hunting): get_auth_request con 0x7f1924138230 auth_method 0 2024-09-11T01:38:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:48.914+0000 7f192a00e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:48.914+0000 7f192a00e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:48.914+0000 7f192a00e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:38:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:48.914+0000 7f192a00e640 1 -- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1924138230 msgr2=0x7f1924138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:38:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:48.914+0000 7f192a00e640 1 -- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1924138230 msgr2=0x7f1924138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:38:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:48.914+0000 7f192a00e640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1924138230 0x7f1924138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:38:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:48.914+0000 7f192a00e640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1924138230 0x7f1924138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:38:50.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.508+0000 7f1923fff640 10 monclient(hunting): tick 2024-09-11T01:38:50.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.508+0000 7f1923fff640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:38:50.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.508+0000 7f1923fff640 1 monclient(hunting): continuing hunt 2024-09-11T01:38:50.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.508+0000 7f1923fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:38:50.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.508+0000 7f1923fff640 1 -- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1924138230 msgr2=0x7f1924138610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:38:50.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.508+0000 7f1923fff640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1924138230 0x7f1924138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:50.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.508+0000 7f1923fff640 1 -- 172.21.15.12:0/299324182 >> v1:172.21.15.71:6789/0 conn(0x7f192409d460 legacy=0x7f19240a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:50.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.508+0000 7f1923fff640 1 -- 172.21.15.12:0/299324182 >> v1:172.21.15.12:6789/0 conn(0x7f192409d030 legacy=0x7f19240a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:50.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.508+0000 7f1923fff640 10 monclient: _add_conns ranks=[2,3] 2024-09-11T01:38:50.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.508+0000 7f1923fff640 1 --2- 172.21.15.12:0/299324182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1910003830 0x7f1910005c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:50.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.508+0000 7f1923fff640 10 monclient(hunting): picked mon.noname-b con 0x7f1910003830 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:50.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.508+0000 7f1923fff640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.12:3300/0 conn(0x7f1910006390 0x7f1910008850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:50.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.508+0000 7f1923fff640 10 monclient(hunting): picked mon.noname-a con 0x7f1910006390 addr v2:172.21.15.12:3300/0 2024-09-11T01:38:50.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.508+0000 7f1923fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:50.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.508+0000 7f1923fff640 1 -- 172.21.15.12:0/299324182 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f192409c9c0 con 0x7f1910006390 2024-09-11T01:38:50.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.508+0000 7f1923fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:50.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.508+0000 7f1923fff640 1 -- 172.21.15.12:0/299324182 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1910008f10 con 0x7f1910003830 2024-09-11T01:38:50.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.508+0000 7f1923fff640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:50.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.508+0000 7f192a00e640 1 -- 172.21.15.12:0/299324182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1910003830 msgr2=0x7f1910005c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:38:50.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.508+0000 7f192a00e640 1 --2- 172.21.15.12:0/299324182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1910003830 0x7f1910005c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:38:50.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.508+0000 7f192980d640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.12:3300/0 conn(0x7f1910006390 0x7f1910008850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:50.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.508+0000 7f192980d640 10 monclient(hunting): get_auth_request con 0x7f1910006390 auth_method 0 2024-09-11T01:38:50.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.508+0000 7f192980d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:50.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.508+0000 7f192980d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:50.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.508+0000 7f192980d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:50.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.509+0000 7f192980d640 1 -- 172.21.15.12:0/299324182 >> v2:172.21.15.12:3300/0 conn(0x7f1910006390 msgr2=0x7f1910008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:38:50.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.509+0000 7f192980d640 1 -- 172.21.15.12:0/299324182 >> v2:172.21.15.12:3300/0 conn(0x7f1910006390 msgr2=0x7f1910008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:38:50.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.509+0000 7f192980d640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.12:3300/0 conn(0x7f1910006390 0x7f1910008850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:38:50.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.509+0000 7f192980d640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.12:3300/0 conn(0x7f1910006390 0x7f1910008850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:38:50.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.709+0000 7f192a00e640 1 -- 172.21.15.12:0/299324182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1910003830 msgr2=0x7f1910005c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:38:50.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.709+0000 7f192a00e640 1 --2- 172.21.15.12:0/299324182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1910003830 0x7f1910005c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:38:50.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.710+0000 7f192980d640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.12:3300/0 conn(0x7f1910006390 0x7f1910008850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:50.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.710+0000 7f192980d640 10 monclient(hunting): get_auth_request con 0x7f1910006390 auth_method 0 2024-09-11T01:38:50.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.710+0000 7f192980d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:50.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.710+0000 7f192980d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:50.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.710+0000 7f192980d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:38:50.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.710+0000 7f192980d640 1 -- 172.21.15.12:0/299324182 >> v2:172.21.15.12:3300/0 conn(0x7f1910006390 msgr2=0x7f1910008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:38:50.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.710+0000 7f192980d640 1 -- 172.21.15.12:0/299324182 >> v2:172.21.15.12:3300/0 conn(0x7f1910006390 msgr2=0x7f1910008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:38:50.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.710+0000 7f192980d640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.12:3300/0 conn(0x7f1910006390 0x7f1910008850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:38:50.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:50.710+0000 7f192980d640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.12:3300/0 conn(0x7f1910006390 0x7f1910008850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:38:51.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:51.110+0000 7f192a00e640 1 -- 172.21.15.12:0/299324182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1910003830 msgr2=0x7f1910005c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:38:51.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:51.111+0000 7f192a00e640 1 --2- 172.21.15.12:0/299324182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1910003830 0x7f1910005c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:38:51.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:51.111+0000 7f192980d640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.12:3300/0 conn(0x7f1910006390 0x7f1910008850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:51.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:51.111+0000 7f192980d640 10 monclient(hunting): get_auth_request con 0x7f1910006390 auth_method 0 2024-09-11T01:38:51.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:51.111+0000 7f192980d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:51.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:51.111+0000 7f192980d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:51.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:51.111+0000 7f192980d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:38:51.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:51.111+0000 7f192980d640 1 -- 172.21.15.12:0/299324182 >> v2:172.21.15.12:3300/0 conn(0x7f1910006390 msgr2=0x7f1910008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:38:51.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:51.111+0000 7f192980d640 1 -- 172.21.15.12:0/299324182 >> v2:172.21.15.12:3300/0 conn(0x7f1910006390 msgr2=0x7f1910008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:38:51.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:51.111+0000 7f192980d640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.12:3300/0 conn(0x7f1910006390 0x7f1910008850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:38:51.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:51.111+0000 7f192980d640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.12:3300/0 conn(0x7f1910006390 0x7f1910008850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:38:51.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:51.912+0000 7f192a00e640 1 -- 172.21.15.12:0/299324182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1910003830 msgr2=0x7f1910005c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:38:51.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:51.912+0000 7f192a00e640 1 --2- 172.21.15.12:0/299324182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1910003830 0x7f1910005c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:38:51.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:51.912+0000 7f192980d640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.12:3300/0 conn(0x7f1910006390 0x7f1910008850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:51.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:51.912+0000 7f192980d640 10 monclient(hunting): get_auth_request con 0x7f1910006390 auth_method 0 2024-09-11T01:38:51.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:51.912+0000 7f192980d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:51.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:51.912+0000 7f192980d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:51.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:51.912+0000 7f192980d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:38:51.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:51.912+0000 7f192980d640 1 -- 172.21.15.12:0/299324182 >> v2:172.21.15.12:3300/0 conn(0x7f1910006390 msgr2=0x7f1910008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:38:51.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:51.912+0000 7f192980d640 1 -- 172.21.15.12:0/299324182 >> v2:172.21.15.12:3300/0 conn(0x7f1910006390 msgr2=0x7f1910008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:38:51.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:51.912+0000 7f192980d640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.12:3300/0 conn(0x7f1910006390 0x7f1910008850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:38:51.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:51.912+0000 7f192980d640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.12:3300/0 conn(0x7f1910006390 0x7f1910008850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:38:53.410 INFO:tasks.rados.rados.0.smithi071.stdout:1417: done (1 left) 2024-09-11T01:38:53.410 INFO:tasks.rados.rados.0.smithi071.stdout:1418: done (0 left) 2024-09-11T01:38:53.410 INFO:tasks.rados.rados.0.smithi071.stdout:1419: read oid 22 snap -1 2024-09-11T01:38:53.410 INFO:tasks.rados.rados.0.smithi071.stdout:1419: expect (ObjNum 503 snap 135 seq_num 503) 2024-09-11T01:38:53.410 INFO:tasks.rados.rados.0.smithi071.stdout:1420: read oid 37 snap 145 2024-09-11T01:38:53.410 INFO:tasks.rados.rados.0.smithi071.stdout:1420: expect (ObjNum 483 snap 130 seq_num 483) 2024-09-11T01:38:53.410 INFO:tasks.rados.rados.0.smithi071.stdout:1421: read oid 27 snap -1 2024-09-11T01:38:53.410 INFO:tasks.rados.rados.0.smithi071.stdout:1421: expect (ObjNum 527 snap 148 seq_num 527) 2024-09-11T01:38:53.411 INFO:tasks.rados.rados.0.smithi071.stdout:1422: read oid 21 snap 145 2024-09-11T01:38:53.411 INFO:tasks.rados.rados.0.smithi071.stdout:1422: expect (ObjNum 518 snap 141 seq_num 518) 2024-09-11T01:38:53.411 INFO:tasks.rados.rados.0.smithi071.stdout:1423: snap_create 2024-09-11T01:38:53.417 INFO:tasks.rados.rados.0.smithi071.stdout:1419: expect (ObjNum 503 snap 135 seq_num 503) 2024-09-11T01:38:53.417 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 149210 2024-09-11T01:38:53.472 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 952892 2024-09-11T01:38:53.487 INFO:tasks.rados.rados.0.smithi071.stdout:1421: expect (ObjNum 527 snap 148 seq_num 527) 2024-09-11T01:38:53.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f1923fff640 10 monclient(hunting): tick 2024-09-11T01:38:53.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f1923fff640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:38:53.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f1923fff640 1 monclient(hunting): continuing hunt 2024-09-11T01:38:53.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f1923fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:38:53.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f1923fff640 1 -- 172.21.15.12:0/299324182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1910003830 msgr2=0x7f1910005c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:38:53.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f1923fff640 1 --2- 172.21.15.12:0/299324182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1910003830 0x7f1910005c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:53.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f1923fff640 1 -- 172.21.15.12:0/299324182 >> v2:172.21.15.12:3300/0 conn(0x7f1910006390 msgr2=0x7f1910008850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:38:53.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f1923fff640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.12:3300/0 conn(0x7f1910006390 0x7f1910008850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:53.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f192a80f640 1 -- 172.21.15.12:0/299324182 reap_dead start 2024-09-11T01:38:53.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f1923fff640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-11T01:38:53.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f1923fff640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.12:3300/0 conn(0x7f1910003c50 0x7f1910006840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:53.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f1923fff640 10 monclient(hunting): picked mon.noname-a con 0x7f1910003c50 addr v2:172.21.15.12:3300/0 2024-09-11T01:38:53.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f1923fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1910006d80 addr v1:172.21.15.71:6789/0 2024-09-11T01:38:53.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f1923fff640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1910008900 0x7f191000d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:53.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f1923fff640 10 monclient(hunting): picked mon.noname-c con 0x7f1910008900 addr v2:172.21.15.71:3300/0 2024-09-11T01:38:53.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f1923fff640 1 -- 172.21.15.12:0/299324182 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f192409ba20 con 0x7f1910006d80 2024-09-11T01:38:53.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f1923fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:53.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f1923fff640 1 -- 172.21.15.12:0/299324182 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f192409c9c0 con 0x7f1910003c50 2024-09-11T01:38:53.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f1923fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:53.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f1923fff640 1 -- 172.21.15.12:0/299324182 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f1910008f10 con 0x7f1910008900 2024-09-11T01:38:53.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f1923fff640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:53.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f192a00e640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.12:3300/0 conn(0x7f1910003c50 0x7f1910006840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:53.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f192a00e640 10 monclient(hunting): get_auth_request con 0x7f1910003c50 auth_method 0 2024-09-11T01:38:53.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f192a00e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:53.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f192a00e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:53.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f192a00e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:53.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f192a80f640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1910008900 0x7f191000d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:53.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f192a00e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:53.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f192a00e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:53.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f192a00e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:53.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f192a80f640 10 monclient(hunting): get_auth_request con 0x7f1910008900 auth_method 0 2024-09-11T01:38:53.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f192a80f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:53.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f192a80f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:53.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f192a80f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:53.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f192a80f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:53.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f192a80f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:53.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f192a80f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f192a00e640 10 monclient(hunting): handle_auth_done global_id 94251 payload 274 2024-09-11T01:38:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f192a00e640 1 -- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1910008900 msgr2=0x7f191000d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:38:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f192a00e640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1910008900 0x7f191000d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f192a00e640 1 -- 172.21.15.12:0/299324182 >> v1:172.21.15.71:6789/0 conn(0x7f1910006d80 legacy=0x7f1910007180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:38:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f192a00e640 10 monclient: _finish_hunting 0 2024-09-11T01:38:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f192a00e640 1 monclient: found mon.noname-a 2024-09-11T01:38:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f192a00e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f192a00e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:38:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.509+0000 7f192a00e640 1 -- 172.21.15.12:0/299324182 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f191000eb10 con 0x7f1910003c50 2024-09-11T01:38:53.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.512+0000 7f192a00e640 1 -- 172.21.15.12:0/299324182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1910003c50 msgr2=0x7f1910006840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:53.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.512+0000 7f192a00e640 1 --2- 172.21.15.12:0/299324182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1910003c50 0x7f1910006840 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f19100088d0 tx=0x7f191402b370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72d330b4c2fa7159 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.513+0000 7f192900c640 1 -- 172.21.15.12:0/299324182 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1914031070 con 0x7f1910003c50 2024-09-11T01:38:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.513+0000 7f192900c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.513+0000 7f192900c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:38:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.513+0000 7f192900c640 10 monclient: dump: 2024-09-11T01:38:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:53.521 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.513+0000 7f192900c640 10 monclient: _finish_auth 0 2024-09-11T01:38:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.513+0000 7f192900c640 10 monclient: _check_auth_tickets 2024-09-11T01:38:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.513+0000 7f192900c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.513+0000 7f192900c640 1 -- 172.21.15.12:0/299324182 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1914031640 con 0x7f1910003c50 2024-09-11T01:38:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.513+0000 7f192c272640 5 monclient: authenticate success, global_id 94251 2024-09-11T01:38:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.513+0000 7f192c272640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:38:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.513+0000 7f192900c640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.513+0000 7f192c272640 10 monclient: get_monmap_and_config success 2024-09-11T01:38:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.513+0000 7f192c272640 4 set_mon_vals no callback set 2024-09-11T01:38:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.513+0000 7f192900c640 1 -- 172.21.15.12:0/299324182 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1914031a30 con 0x7f1910003c50 2024-09-11T01:38:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.513+0000 7f192900c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.513+0000 7f192900c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:38:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.513+0000 7f192900c640 10 monclient: dump: 2024-09-11T01:38:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:53.523 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.513+0000 7f192c272640 10 monclient: shutdown 2024-09-11T01:38:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.513+0000 7f192c272640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.513+0000 7f192c272640 1 -- 172.21.15.12:0/299324182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1910003c50 msgr2=0x7f1910006840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.513+0000 7f192c272640 1 --2- 172.21.15.12:0/299324182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1910003c50 0x7f1910006840 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f19100088d0 tx=0x7f191402b370 comp rx=0 tx=0).stop 2024-09-11T01:38:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.514+0000 7f192c272640 1 -- 172.21.15.12:0/299324182 shutdown_connections 2024-09-11T01:38:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.514+0000 7f192c272640 1 --2- 172.21.15.12:0/299324182 >> v2:172.21.15.71:3300/0 conn(0x7f1910008900 0x7f191000d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.514+0000 7f192c272640 1 --2- 172.21.15.12:0/299324182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1910003c50 0x7f1910006840 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.514+0000 7f192c272640 1 -- 172.21.15.12:0/299324182 >> 172.21.15.12:0/299324182 conn(0x7f19240a7120 msgr2=0x7f19240a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.514+0000 7f192c272640 1 -- 172.21.15.12:0/299324182 shutdown_connections 2024-09-11T01:38:53.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.514+0000 7f192c272640 1 -- 172.21.15.12:0/299324182 wait complete. 2024-09-11T01:38:53.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.514+0000 7f192c272640 10 monclient: build_initial_monmap 2024-09-11T01:38:53.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.514+0000 7f192c272640 10 monclient: monmap: 2024-09-11T01:38:53.524 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:53.524 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:38:53.524 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:38:53.524 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:38:53.524 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:53.524 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:53.524 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:38:53.524 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:38:53.524 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:38:53.524 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:53.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.515+0000 7f192c272640 1 Processor -- start 2024-09-11T01:38:53.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.515+0000 7f192c272640 1 -- start start 2024-09-11T01:38:53.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.515+0000 7f192c272640 10 monclient: init 2024-09-11T01:38:53.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.515+0000 7f192c272640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:53.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.515+0000 7f192c272640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:38:53.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192c272640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1910003c50 0x7f1924136fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:53.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192c272640 10 monclient(hunting): picked mon.noname-b con 0x7f1910003c50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:38:53.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192c272640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1910006d80 0x7f19241374e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:53.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192c272640 10 monclient(hunting): picked mon.noname-c con 0x7f1910006d80 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:53.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192c272640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1910008900 0x7f1924124cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:53.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192c272640 10 monclient(hunting): picked mon.noname-a con 0x7f1910008900 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:38:53.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192c272640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:53.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192c272640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f192406f970 con 0x7f1910008900 2024-09-11T01:38:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192c272640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192c272640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f192406f7f0 con 0x7f1910003c50 2024-09-11T01:38:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192c272640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192c272640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1924138610 con 0x7f1910006d80 2024-09-11T01:38:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192c272640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192980d640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1910006d80 msgr2=0x7f19241374e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:38:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192c272640 10 monclient(hunting): authenticate will time out at 1339.958120s 2024-09-11T01:38:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192980d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1910006d80 0x7f19241374e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:38:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a00e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1910003c50 0x7f1924136fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a80f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1910008900 0x7f1924124cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a00e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1910003c50 0x7f1924136fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:55686/0 (socket says 172.21.15.12:55686) 2024-09-11T01:38:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a00e640 1 -- 172.21.15.12:0/3107930330 learned_addr learned my addr 172.21.15.12:0/3107930330 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:38:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a80f640 10 monclient(hunting): get_auth_request con 0x7f1910008900 auth_method 0 2024-09-11T01:38:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a80f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a80f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a80f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a00e640 10 monclient(hunting): get_auth_request con 0x7f1910003c50 auth_method 0 2024-09-11T01:38:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a00e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a00e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a00e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a80f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a80f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a80f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a00e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a00e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:53.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a00e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:53.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a80f640 10 monclient(hunting): handle_auth_done global_id 94254 payload 824 2024-09-11T01:38:53.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a80f640 1 -- 172.21.15.12:0/3107930330 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1910006d80 msgr2=0x7f19241374e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:38:53.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a80f640 1 --2- 172.21.15.12:0/3107930330 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1910006d80 0x7f19241374e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:53.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a80f640 1 -- 172.21.15.12:0/3107930330 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1910003c50 msgr2=0x7f1924136fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:53.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a80f640 1 --2- 172.21.15.12:0/3107930330 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1910003c50 0x7f1924136fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:53.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a80f640 10 monclient: _finish_hunting 0 2024-09-11T01:38:53.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a80f640 1 monclient: found mon.noname-a 2024-09-11T01:38:53.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a80f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:53.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a80f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:38:53.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a80f640 1 -- 172.21.15.12:0/3107930330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1924126730 con 0x7f1910008900 2024-09-11T01:38:53.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.516+0000 7f192a00e640 1 --2- 172.21.15.12:0/3107930330 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1910003c50 0x7f1924136fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:38:53.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.518+0000 7f192a80f640 1 -- 172.21.15.12:0/3107930330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1910008900 msgr2=0x7f1924124cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:53.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.518+0000 7f192a80f640 1 --2- 172.21.15.12:0/3107930330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1910008900 0x7f1924124cb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f19240a6330 tx=0x7f191c001960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:53.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.519+0000 7f1922ffd640 1 -- 172.21.15.12:0/3107930330 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191c018070 con 0x7f1910008900 2024-09-11T01:38:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.519+0000 7f1922ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.519+0000 7f1922ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:38:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.519+0000 7f1922ffd640 10 monclient: dump: 2024-09-11T01:38:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:53.531 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:53.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.519+0000 7f1922ffd640 10 monclient: _finish_auth 0 2024-09-11T01:38:53.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.519+0000 7f1922ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:38:53.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.519+0000 7f1922ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:53.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.519+0000 7f1922ffd640 1 -- 172.21.15.12:0/3107930330 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f191c012770 con 0x7f1910008900 2024-09-11T01:38:53.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.519+0000 7f1922ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:53.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.519+0000 7f1922ffd640 1 -- 172.21.15.12:0/3107930330 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191c012b20 con 0x7f1910008900 2024-09-11T01:38:53.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.519+0000 7f192c272640 5 monclient: authenticate success, global_id 94254 2024-09-11T01:38:53.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.519+0000 7f1922ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:53.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.519+0000 7f1922ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:38:53.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.519+0000 7f1922ffd640 10 monclient: dump: 2024-09-11T01:38:53.532 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:53.532 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:53.533 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:53.533 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:53.533 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:53.533 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:53.533 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:53.533 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:53.533 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:53.533 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:53.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.519+0000 7f192c272640 10 monclient: _renew_subs 2024-09-11T01:38:53.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.519+0000 7f192c272640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:38:53.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.519+0000 7f192c272640 1 -- 172.21.15.12:0/3107930330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1924137a90 con 0x7f1910008900 2024-09-11T01:38:53.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.519+0000 7f192880b640 4 set_mon_vals no callback set 2024-09-11T01:38:53.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.519+0000 7f192c272640 10 monclient: _renew_subs 2024-09-11T01:38:53.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.519+0000 7f192c272640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:38:53.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.519+0000 7f192c272640 1 -- 172.21.15.12:0/3107930330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19240a1810 con 0x7f1910008900 2024-09-11T01:38:53.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.520+0000 7f1922ffd640 1 -- 172.21.15.12:0/3107930330 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f191c012cc0 con 0x7f1910008900 2024-09-11T01:38:53.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.520+0000 7f192c272640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:53.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.520+0000 7f192c272640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:53.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.520+0000 7f192c272640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:38:53.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.520+0000 7f192c272640 1 -- 172.21.15.12:0/3107930330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1924138a40 con 0x7f1910008900 2024-09-11T01:38:53.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.520+0000 7f1922ffd640 1 --2- 172.21.15.12:0/3107930330 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1910040140 0x7f1910042600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:53.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.520+0000 7f1922ffd640 1 -- 172.21.15.12:0/3107930330 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(322..322 src has 186..322) ==== 4755+0+0 (secure 0 0 0) 0x7f191c0521e0 con 0x7f1910008900 2024-09-11T01:38:53.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.521+0000 7f192a00e640 1 --2- 172.21.15.12:0/3107930330 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1910040140 0x7f1910042600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:53.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.521+0000 7f192a00e640 10 monclient: get_auth_request con 0x7f1910040140 auth_method 0 2024-09-11T01:38:53.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.523+0000 7f192a00e640 1 --2- 172.21.15.12:0/3107930330 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1910040140 0x7f1910042600 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1914037040 tx=0x7f1914031000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:53.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.526+0000 7f1922ffd640 1 -- 172.21.15.12:0/3107930330 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f191c050bf0 con 0x7f1910008900 2024-09-11T01:38:53.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.526+0000 7f1922ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:53.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.526+0000 7f1922ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:38:53.628 INFO:tasks.rados.rados.0.smithi071.stdout:1420: expect (ObjNum 483 snap 130 seq_num 483) 2024-09-11T01:38:53.698 INFO:tasks.rados.rados.0.smithi071.stdout:1422: expect (ObjNum 518 snap 141 seq_num 518) 2024-09-11T01:38:53.707 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1517413 2024-09-11T01:38:53.771 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:38:53.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.784+0000 7f192c272640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:38:53.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.784+0000 7f192c272640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:38:53.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.784+0000 7f192c272640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:38:53.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.784+0000 7f192c272640 1 -- 172.21.15.12:0/3107930330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1924128710 con 0x7f1910008900 2024-09-11T01:38:53.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.784+0000 7f1922ffd640 1 -- 172.21.15.12:0/3107930330 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f191c01cb60 con 0x7f1910008900 2024-09-11T01:38:53.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.784+0000 7f1922ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:38:53.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.784+0000 7f1922ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:38:53.787 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":94,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:38:53.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.787+0000 7f192c272640 1 -- 172.21.15.12:0/3107930330 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1910040140 msgr2=0x7f1910042600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:53.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.787+0000 7f192c272640 1 --2- 172.21.15.12:0/3107930330 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1910040140 0x7f1910042600 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1914037040 tx=0x7f1914031000 comp rx=0 tx=0).stop 2024-09-11T01:38:53.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.787+0000 7f192c272640 10 monclient: shutdown 2024-09-11T01:38:53.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.787+0000 7f192c272640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:53.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.787+0000 7f192c272640 1 -- 172.21.15.12:0/3107930330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1910008900 msgr2=0x7f1924124cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:53.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.787+0000 7f192c272640 1 --2- 172.21.15.12:0/3107930330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1910008900 0x7f1924124cb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f19240a6330 tx=0x7f191c001960 comp rx=0 tx=0).stop 2024-09-11T01:38:53.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.787+0000 7f192c272640 1 -- 172.21.15.12:0/3107930330 shutdown_connections 2024-09-11T01:38:53.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.787+0000 7f192c272640 1 --2- 172.21.15.12:0/3107930330 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1910040140 0x7f1910042600 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:53.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.787+0000 7f192c272640 1 --2- 172.21.15.12:0/3107930330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1910008900 0x7f1924124cb0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:53.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.787+0000 7f192c272640 1 --2- 172.21.15.12:0/3107930330 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1910006d80 0x7f19241374e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:53.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.787+0000 7f192c272640 1 --2- 172.21.15.12:0/3107930330 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1910003c50 0x7f1924136fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:53.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.787+0000 7f192c272640 1 -- 172.21.15.12:0/3107930330 >> 172.21.15.12:0/3107930330 conn(0x7f19240a7120 msgr2=0x7f19240a3220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:53.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.787+0000 7f192c272640 1 -- 172.21.15.12:0/3107930330 shutdown_connections 2024-09-11T01:38:53.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.787+0000 7f192c272640 1 -- 172.21.15.12:0/3107930330 wait complete. 2024-09-11T01:38:53.799 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:38:53.799 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-11T01:38:53.799 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:38:53.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.900+0000 7fb17f52b640 10 monclient: get_monmap_and_config 2024-09-11T01:38:53.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.900+0000 7fb17f52b640 10 monclient: build_initial_monmap 2024-09-11T01:38:53.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.900+0000 7fb17f52b640 10 monclient: monmap: 2024-09-11T01:38:53.901 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:53.902 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:53.902 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:38:53.900788+0000 2024-09-11T01:38:53.902 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:38:53.900788+0000 2024-09-11T01:38:53.902 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:53.902 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:53.902 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:38:53.902 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:38:53.902 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:38:53.902 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:38:53.902 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:38:53.902 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:53.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.900+0000 7fb17f52b640 1 Processor -- start 2024-09-11T01:38:53.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.900+0000 7fb17f52b640 1 -- start start 2024-09-11T01:38:53.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.900+0000 7fb17f52b640 10 monclient: init 2024-09-11T01:38:53.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17f52b640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:53.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17f52b640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-11T01:38:53.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17f52b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb178137e50 0x7fb17809d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:53.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17f52b640 10 monclient(hunting): picked mon.noname-b con 0x7fb178137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:53.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17f52b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb178138230 0x7fb178138630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:53.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17f52b640 10 monclient(hunting): picked mon.noname-c con 0x7fb178138230 addr v2:172.21.15.71:3300/0 2024-09-11T01:38:53.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17f52b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb178134cb0 addr v1:172.21.15.71:6789/0 2024-09-11T01:38:53.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17f52b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1780a62d0 con 0x7fb178134cb0 2024-09-11T01:38:53.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17f52b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:53.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17f52b640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fb1780a65b0 con 0x7fb178138230 2024-09-11T01:38:53.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17f52b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:53.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17f52b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb17809bb90 con 0x7fb178137e50 2024-09-11T01:38:53.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17f52b640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:53.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17f52b640 10 monclient(hunting): authenticate will time out at 1340.343383s 2024-09-11T01:38:53.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17d2c7640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb178137e50 msgr2=0x7fb17809d160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:38:53.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17d2c7640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb178137e50 0x7fb17809d160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:38:53.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17cac6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb178138230 0x7fb178138630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:53.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17dac8640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fb178134cb0 0x7fb178135120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.12:56344/0 (socket says 172.21.15.12:56344) 2024-09-11T01:38:53.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17cac6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb178138230 0x7fb178138630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:55690/0 (socket says 172.21.15.12:55690) 2024-09-11T01:38:53.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17dac8640 1 -- 172.21.15.12:0/1946126939 learned_addr learned my addr 172.21.15.12:0/1946126939 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:38:53.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17cac6640 10 monclient(hunting): get_auth_request con 0x7fb178138230 auth_method 0 2024-09-11T01:38:53.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17cac6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:53.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17cac6640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:53.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17cac6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:53.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17cac6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:53.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17cac6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:53.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17cac6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:53.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17cac6640 10 monclient(hunting): handle_auth_done global_id 104140 payload 274 2024-09-11T01:38:53.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17cac6640 1 -- 172.21.15.12:0/1946126939 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb178137e50 msgr2=0x7fb17809d160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:38:53.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17cac6640 1 --2- 172.21.15.12:0/1946126939 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb178137e50 0x7fb17809d160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:53.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17cac6640 1 -- 172.21.15.12:0/1946126939 >> v1:172.21.15.71:6789/0 conn(0x7fb178134cb0 legacy=0x7fb178135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:38:53.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17cac6640 10 monclient: _finish_hunting 0 2024-09-11T01:38:53.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17cac6640 1 monclient: found mon.noname-c 2024-09-11T01:38:53.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17cac6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:53.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17cac6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:38:53.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.901+0000 7fb17cac6640 1 -- 172.21.15.12:0/1946126939 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb17809c4f0 con 0x7fb178138230 2024-09-11T01:38:53.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.904+0000 7fb17cac6640 1 -- 172.21.15.12:0/1946126939 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb178138230 msgr2=0x7fb178138630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:53.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.904+0000 7fb17cac6640 1 --2- 172.21.15.12:0/1946126939 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb178138230 0x7fb178138630 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb1680097a0 tx=0x7fb168029fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed0630f9aedd99e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:53.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb16ffff640 1 -- 172.21.15.12:0/1946126939 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb168037070 con 0x7fb178138230 2024-09-11T01:38:53.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb16ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:53.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb16ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:38:53.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb16ffff640 10 monclient: dump: 2024-09-11T01:38:53.916 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:53.916 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:53.916 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:53.916 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:53.916 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:53.916 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:53.916 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:53.916 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:53.917 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:53.917 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:53.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb16ffff640 10 monclient: _finish_auth 0 2024-09-11T01:38:53.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb16ffff640 10 monclient: _check_auth_tickets 2024-09-11T01:38:53.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb16ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:53.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb17f52b640 5 monclient: authenticate success, global_id 104140 2024-09-11T01:38:53.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb16ffff640 1 -- 172.21.15.12:0/1946126939 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb168032640 con 0x7fb178138230 2024-09-11T01:38:53.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb17f52b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:38:53.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb16ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:53.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb17f52b640 10 monclient: get_monmap_and_config success 2024-09-11T01:38:53.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb17f52b640 4 set_mon_vals no callback set 2024-09-11T01:38:53.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb16ffff640 1 -- 172.21.15.12:0/1946126939 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb168032a10 con 0x7fb178138230 2024-09-11T01:38:53.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb16ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:53.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb16ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:38:53.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb16ffff640 10 monclient: dump: 2024-09-11T01:38:53.918 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:53.918 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:53.918 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:53.918 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:53.918 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:53.918 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:53.918 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:53.918 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:53.918 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:53.918 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:53.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb17f52b640 10 monclient: shutdown 2024-09-11T01:38:53.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb17f52b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:53.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb17f52b640 1 -- 172.21.15.12:0/1946126939 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb178138230 msgr2=0x7fb178138630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:53.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb17f52b640 1 --2- 172.21.15.12:0/1946126939 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb178138230 0x7fb178138630 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb1680097a0 tx=0x7fb168029fe0 comp rx=0 tx=0).stop 2024-09-11T01:38:53.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb17f52b640 1 -- 172.21.15.12:0/1946126939 shutdown_connections 2024-09-11T01:38:53.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb17f52b640 1 --2- 172.21.15.12:0/1946126939 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb178138230 0x7fb178138630 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:53.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb17f52b640 1 --2- 172.21.15.12:0/1946126939 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb178137e50 0x7fb17809d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:53.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb17f52b640 1 -- 172.21.15.12:0/1946126939 >> 172.21.15.12:0/1946126939 conn(0x7fb1780a7020 msgr2=0x7fb1780a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:53.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb17f52b640 1 -- 172.21.15.12:0/1946126939 shutdown_connections 2024-09-11T01:38:53.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.905+0000 7fb17f52b640 1 -- 172.21.15.12:0/1946126939 wait complete. 2024-09-11T01:38:53.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.906+0000 7fb17f52b640 10 monclient: build_initial_monmap 2024-09-11T01:38:53.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.906+0000 7fb17f52b640 10 monclient: monmap: 2024-09-11T01:38:53.919 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:53.919 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:38:53.919 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:38:53.920 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:38:53.920 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:53.920 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:53.920 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:38:53.920 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:38:53.920 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:38:53.920 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:53.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.906+0000 7fb17f52b640 1 Processor -- start 2024-09-11T01:38:53.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.906+0000 7fb17f52b640 1 -- start start 2024-09-11T01:38:53.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.906+0000 7fb17f52b640 10 monclient: init 2024-09-11T01:38:53.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17f52b640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:53.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17f52b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:38:53.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17f52b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb178134cb0 0x7fb1781cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17f52b640 10 monclient(hunting): picked mon.noname-b con 0x7fb178134cb0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:38:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17f52b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb178137e50 0x7fb178133ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17f52b640 10 monclient(hunting): picked mon.noname-c con 0x7fb178137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17f52b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb178138230 0x7fb1781341e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17f52b640 10 monclient(hunting): picked mon.noname-a con 0x7fb178138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:38:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17f52b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17f52b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb17806f710 con 0x7fb178138230 2024-09-11T01:38:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17f52b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17f52b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb178138b70 con 0x7fb178134cb0 2024-09-11T01:38:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17f52b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17f52b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb17806f890 con 0x7fb178137e50 2024-09-11T01:38:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17f52b640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:53.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17f52b640 10 monclient(hunting): authenticate will time out at 1340.349201s 2024-09-11T01:38:53.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17cac6640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb178137e50 msgr2=0x7fb178133ca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:38:53.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17cac6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb178137e50 0x7fb178133ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:38:53.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17dac8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb178138230 0x7fb1781341e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:53.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17dac8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb178138230 0x7fb1781341e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33038/0 (socket says 172.21.15.12:33038) 2024-09-11T01:38:53.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17dac8640 1 -- 172.21.15.12:0/1057093081 learned_addr learned my addr 172.21.15.12:0/1057093081 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:38:53.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17dac8640 10 monclient(hunting): get_auth_request con 0x7fb178138230 auth_method 0 2024-09-11T01:38:53.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17dac8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:53.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17dac8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:53.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17dac8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:53.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17dac8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:53.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17dac8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:53.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17dac8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:53.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17d2c7640 1 --2- 172.21.15.12:0/1057093081 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb178134cb0 0x7fb1781cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:53.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17d2c7640 10 monclient(hunting): get_auth_request con 0x7fb178134cb0 auth_method 0 2024-09-11T01:38:53.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17d2c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:53.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17d2c7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:53.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17d2c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:53.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17dac8640 10 monclient(hunting): handle_auth_done global_id 94257 payload 824 2024-09-11T01:38:53.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17dac8640 1 -- 172.21.15.12:0/1057093081 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb178137e50 msgr2=0x7fb178133ca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:38:53.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17dac8640 1 --2- 172.21.15.12:0/1057093081 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb178137e50 0x7fb178133ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:53.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17dac8640 1 -- 172.21.15.12:0/1057093081 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb178134cb0 msgr2=0x7fb1781cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:53.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17dac8640 1 --2- 172.21.15.12:0/1057093081 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb178134cb0 0x7fb1781cd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:53.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17dac8640 10 monclient: _finish_hunting 0 2024-09-11T01:38:53.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17dac8640 1 monclient: found mon.noname-a 2024-09-11T01:38:53.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17dac8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:53.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17dac8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:38:53.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17dac8640 1 -- 172.21.15.12:0/1057093081 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1780a1710 con 0x7fb178138230 2024-09-11T01:38:53.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.907+0000 7fb17d2c7640 1 --2- 172.21.15.12:0/1057093081 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb178134cb0 0x7fb1781cd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:38:53.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.909+0000 7fb17dac8640 1 -- 172.21.15.12:0/1057093081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb178138230 msgr2=0x7fb1781341e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:53.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.909+0000 7fb17dac8640 1 --2- 172.21.15.12:0/1057093081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb178138230 0x7fb1781341e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb17000caf0 tx=0x7fb17000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:53.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.910+0000 7fb16dffb640 1 -- 172.21.15.12:0/1057093081 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb170012400 con 0x7fb178138230 2024-09-11T01:38:53.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.910+0000 7fb16dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:53.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.910+0000 7fb16dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:38:53.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.910+0000 7fb16dffb640 10 monclient: dump: 2024-09-11T01:38:53.924 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:53.924 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:53.924 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:53.924 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:53.924 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:53.925 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:53.925 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:53.925 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:53.925 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:53.925 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:53.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.910+0000 7fb16dffb640 10 monclient: _finish_auth 0 2024-09-11T01:38:53.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.910+0000 7fb16dffb640 10 monclient: _check_auth_tickets 2024-09-11T01:38:53.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.910+0000 7fb16dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:53.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.910+0000 7fb17f52b640 5 monclient: authenticate success, global_id 94257 2024-09-11T01:38:53.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.910+0000 7fb17f52b640 10 monclient: _renew_subs 2024-09-11T01:38:53.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.910+0000 7fb17f52b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:38:53.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.910+0000 7fb17f52b640 1 -- 172.21.15.12:0/1057093081 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1781256a0 con 0x7fb178138230 2024-09-11T01:38:53.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.910+0000 7fb16dffb640 1 -- 172.21.15.12:0/1057093081 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1700125c0 con 0x7fb178138230 2024-09-11T01:38:53.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.910+0000 7fb16dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:53.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.910+0000 7fb16dffb640 1 -- 172.21.15.12:0/1057093081 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1700129b0 con 0x7fb178138230 2024-09-11T01:38:53.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.910+0000 7fb16dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:53.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.910+0000 7fb16dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:38:53.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.910+0000 7fb16dffb640 10 monclient: dump: 2024-09-11T01:38:53.926 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:53.926 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:53.926 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:53.926 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:53.926 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:53.926 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:53.926 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:53.926 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:53.926 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:53.927 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:53.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.910+0000 7fb16f7fe640 4 set_mon_vals no callback set 2024-09-11T01:38:53.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.910+0000 7fb17f52b640 10 monclient: _renew_subs 2024-09-11T01:38:53.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.910+0000 7fb17f52b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:38:53.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.910+0000 7fb17f52b640 1 -- 172.21.15.12:0/1057093081 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb178124e00 con 0x7fb178138230 2024-09-11T01:38:53.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.911+0000 7fb17f52b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:53.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.911+0000 7fb17f52b640 1 --2- 172.21.15.12:0/1057093081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb178121880 0x7fb178121c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:53.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.911+0000 7fb17f52b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:53.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.911+0000 7fb17f52b640 1 -- 172.21.15.12:0/1057093081 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb1781359a0 con 0x7fb178121880 2024-09-11T01:38:53.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.911+0000 7fb16dffb640 1 -- 172.21.15.12:0/1057093081 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fb170012b50 con 0x7fb178138230 2024-09-11T01:38:53.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.911+0000 7fb17d2c7640 1 --2- 172.21.15.12:0/1057093081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb178121880 0x7fb178121c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:53.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.911+0000 7fb17d2c7640 10 monclient: get_auth_request con 0x7fb178121880 auth_method 0 2024-09-11T01:38:53.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.911+0000 7fb17d2c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:53.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.911+0000 7fb17d2c7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:53.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.911+0000 7fb17d2c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:53.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.911+0000 7fb17d2c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:53.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.911+0000 7fb17d2c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:53.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.911+0000 7fb17d2c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:53.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.911+0000 7fb16dffb640 1 --2- 172.21.15.12:0/1057093081 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb15c03e1c0 0x7fb15c040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:53.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.911+0000 7fb17d2c7640 10 monclient(hunting): handle_auth_done global_id 94260 payload 824 2024-09-11T01:38:53.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.911+0000 7fb17d2c7640 1 -- 172.21.15.12:0/1057093081 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb1781251f0 con 0x7fb178121880 2024-09-11T01:38:53.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.911+0000 7fb16dffb640 1 -- 172.21.15.12:0/1057093081 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(322..322 src has 186..322) ==== 4755+0+0 (secure 0 0 0) 0x7fb1700533f0 con 0x7fb178138230 2024-09-11T01:38:53.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.911+0000 7fb17cac6640 1 --2- 172.21.15.12:0/1057093081 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb15c03e1c0 0x7fb15c040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:53.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.911+0000 7fb17cac6640 10 monclient: get_auth_request con 0x7fb15c03e1c0 auth_method 0 2024-09-11T01:38:53.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.913+0000 7fb17d2c7640 1 -- 172.21.15.12:0/1057093081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb178121880 msgr2=0x7fb178121c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:53.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.913+0000 7fb17d2c7640 1 --2- 172.21.15.12:0/1057093081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb178121880 0x7fb178121c60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb17400d700 tx=0x7fb17400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:53.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.913+0000 7fb16dffb640 1 -- 172.21.15.12:0/1057093081 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb174003bc0 con 0x7fb178121880 2024-09-11T01:38:53.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.913+0000 7fb16dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:53.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.913+0000 7fb16dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:38:53.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.913+0000 7fb16dffb640 10 monclient: dump: 2024-09-11T01:38:53.929 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:53.929 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:53.929 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:53.929 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:53.929 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:53.929 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:53.929 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:53.929 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:53.929 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:53.929 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:53.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.914+0000 7fb17cac6640 1 --2- 172.21.15.12:0/1057093081 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb15c03e1c0 0x7fb15c040680 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb168034000 tx=0x7fb16802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:53.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.914+0000 7fb16dffb640 1 -- 172.21.15.12:0/1057093081 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb174003bc0 con 0x7fb178121880 2024-09-11T01:38:53.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.914+0000 7fb16dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:53.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.914+0000 7fb16dffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:38:53.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.914+0000 7fb16dffb640 1 -- 172.21.15.12:0/1057093081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb178121880 msgr2=0x7fb178121c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:53.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.914+0000 7fb16dffb640 1 --2- 172.21.15.12:0/1057093081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb178121880 0x7fb178121c60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb17400d700 tx=0x7fb17400dbc0 comp rx=0 tx=0).stop 2024-09-11T01:38:53.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.914+0000 7fb16dffb640 1 -- 172.21.15.12:0/1057093081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb178121880 msgr2=0x7fb178121c60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:38:53.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.914+0000 7fb16dffb640 1 --2- 172.21.15.12:0/1057093081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb178121880 0x7fb178121c60 secure :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb17400d700 tx=0x7fb17400dbc0 comp rx=0 tx=0).stop 2024-09-11T01:38:53.931 INFO:tasks.rados.rados.0.smithi071.stdout:1419: done (4 left) 2024-09-11T01:38:53.931 INFO:tasks.rados.rados.0.smithi071.stdout:1420: done (3 left) 2024-09-11T01:38:53.931 INFO:tasks.rados.rados.0.smithi071.stdout:1421: done (2 left) 2024-09-11T01:38:53.931 INFO:tasks.rados.rados.0.smithi071.stdout:1422: done (1 left) 2024-09-11T01:38:53.931 INFO:tasks.rados.rados.0.smithi071.stdout:1423: done (0 left) 2024-09-11T01:38:53.931 INFO:tasks.rados.rados.0.smithi071.stdout:1424: snap_remove snap 149 2024-09-11T01:38:53.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.920+0000 7fb17f52b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:38:53.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.920+0000 7fb17f52b640 1 --2- 172.21.15.12:0/1057093081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1781221a0 0x7fb1780a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:53.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.920+0000 7fb17f52b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:53.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.920+0000 7fb17f52b640 1 -- 172.21.15.12:0/1057093081 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb178069950 con 0x7fb1781221a0 2024-09-11T01:38:53.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.920+0000 7fb17d2c7640 1 --2- 172.21.15.12:0/1057093081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1781221a0 0x7fb1780a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:53.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.920+0000 7fb17d2c7640 10 monclient: get_auth_request con 0x7fb1781221a0 auth_method 0 2024-09-11T01:38:53.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.920+0000 7fb17d2c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:53.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.920+0000 7fb17d2c7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:53.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.920+0000 7fb17d2c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:53.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.920+0000 7fb17d2c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:53.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.920+0000 7fb17d2c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:53.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.920+0000 7fb17d2c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:53.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.920+0000 7fb17d2c7640 10 monclient(hunting): handle_auth_done global_id 94263 payload 824 2024-09-11T01:38:53.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.920+0000 7fb17d2c7640 1 -- 172.21.15.12:0/1057093081 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb178122580 con 0x7fb1781221a0 2024-09-11T01:38:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.922+0000 7fb17d2c7640 1 -- 172.21.15.12:0/1057093081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1781221a0 msgr2=0x7fb1780a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.922+0000 7fb17d2c7640 1 --2- 172.21.15.12:0/1057093081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1781221a0 0x7fb1780a1920 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb17400d700 tx=0x7fb1740059d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.923+0000 7fb16dffb640 1 -- 172.21.15.12:0/1057093081 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17401b970 con 0x7fb1781221a0 2024-09-11T01:38:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.923+0000 7fb16dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.923+0000 7fb16dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:38:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.923+0000 7fb16dffb640 10 monclient: dump: 2024-09-11T01:38:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:53.934 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:53.935 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:53.935 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:53.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.923+0000 7fb16dffb640 1 -- 172.21.15.12:0/1057093081 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4698 (secure 0 0 0) 0x7fb17401b970 con 0x7fb1781221a0 2024-09-11T01:38:53.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.923+0000 7fb16dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:38:53.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.923+0000 7fb16dffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:38:53.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.923+0000 7fb16dffb640 1 -- 172.21.15.12:0/1057093081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1781221a0 msgr2=0x7fb1780a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:53.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.923+0000 7fb16dffb640 1 --2- 172.21.15.12:0/1057093081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1781221a0 0x7fb1780a1920 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb17400d700 tx=0x7fb1740059d0 comp rx=0 tx=0).stop 2024-09-11T01:38:53.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.923+0000 7fb16dffb640 1 -- 172.21.15.12:0/1057093081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1781221a0 msgr2=0x7fb1780a1920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:38:53.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.923+0000 7fb16dffb640 1 --2- 172.21.15.12:0/1057093081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1781221a0 0x7fb1780a1920 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb17400d700 tx=0x7fb1740059d0 comp rx=0 tx=0).stop 2024-09-11T01:38:53.935 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:38:53.936 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:38:53.936 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:38:53.936 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:38:53.936 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 94, 2024-09-11T01:38:53.936 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 80019, 2024-09-11T01:38:53.936 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:38:53.936 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:38:53.936 INFO:teuthology.orchestra.run.smithi012.stdout: 1 2024-09-11T01:38:53.936 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:53.936 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:38:53.936 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:38:53.936 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:38:53.937 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:38:53.937 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:38:53.937 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:38:53.937 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:38:53.937 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:38:53.937 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:38:53.937 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:38:53.937 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:38:53.937 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:38:53.937 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:38:53.937 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:38:53.937 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:38:53.937 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:53.937 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:38:53.938 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:38:53.938 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:38:53.938 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:38:53.938 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:38:53.938 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:38:53.938 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:38:53.938 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:38:53.938 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:38:53.938 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:38:53.938 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:38:53.938 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:38:53.938 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:38:53.938 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:53.939 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:53.939 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:38:53.939 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:38:53.939 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:38:53.939 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:38:53.939 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:38:53.939 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:38:53.939 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:38:53.939 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:38:53.939 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:38:53.939 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:38:53.939 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:38:53.939 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:38:53.939 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:38:53.940 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:38:53.940 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:38:53.940 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:38:53.940 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:38:53.940 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:38:53.940 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:38:53.940 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:38:53.940 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:38:53.940 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:38:53.940 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:38:53.940 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:38:53.940 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:38:53.940 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:38:53.940 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:38:53.941 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:38:53.941 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:53.941 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:38:53.941 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:53.941 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:38:53.941 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:53.941 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:38:53.941 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:38:53.941 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:38:53.941 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:38:53.941 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:53.941 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:38:53.941 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:38:53.942 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:53.942 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:53.942 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:53.942 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:38:53.942 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:38:53.942 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:53.942 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:53.942 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:53.942 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:53.942 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:38:53.942 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:38:53.942 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:38:53.942 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:38:53.942 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:38:53.943 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:53.943 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:53.943 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:38:53.943 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:38:53.943 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:38:53.943 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:38:53.943 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:53.943 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:38:53.943 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:38:53.943 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:53.943 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:53.943 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:53.944 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:38:53.944 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:38:53.944 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:53.944 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:53.944 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:53.944 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:53.944 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:38:53.944 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:38:53.944 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:38:53.944 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:38:53.944 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:38:53.944 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:53.944 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:53.945 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:38:53.945 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:38:53.945 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:38:53.945 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:38:53.945 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:53.945 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:38:53.945 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:38:53.945 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:53.945 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:53.945 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:53.945 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:38:53.945 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:38:53.945 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:53.946 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:53.946 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:53.946 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:53.946 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:38:53.946 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:38:53.946 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:38:53.946 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:38:53.946 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:38:53.946 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:53.946 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:53.946 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:53.946 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:38:53.946 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:38:53.946 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:53.946 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:38:53.947 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:38:53.947 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:38:53.947 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:53.947 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:53.947 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:38:53.947 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:53.947 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:38:53.947 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:38:53.947 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:38:53.947 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:53.947 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:53.947 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:38:53.947 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:53.947 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:38:53.948 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:38:53.948 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 7 2024-09-11T01:38:53.948 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:53.948 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:53.948 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:53.948 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:38:53.948 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:38:53.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.923+0000 7fb17f52b640 1 -- 172.21.15.12:0/1057093081 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb15c03e1c0 msgr2=0x7fb15c040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:53.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.923+0000 7fb17f52b640 1 --2- 172.21.15.12:0/1057093081 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb15c03e1c0 0x7fb15c040680 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb168034000 tx=0x7fb16802e000 comp rx=0 tx=0).stop 2024-09-11T01:38:53.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.923+0000 7fb17f52b640 10 monclient: shutdown 2024-09-11T01:38:53.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.923+0000 7fb17f52b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:53.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.923+0000 7fb17f52b640 1 -- 172.21.15.12:0/1057093081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb178138230 msgr2=0x7fb1781341e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:53.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.923+0000 7fb17dac8640 1 -- 172.21.15.12:0/1057093081 reap_dead start 2024-09-11T01:38:53.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.923+0000 7fb17f52b640 1 --2- 172.21.15.12:0/1057093081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb178138230 0x7fb1781341e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb17000caf0 tx=0x7fb17000a060 comp rx=0 tx=0).stop 2024-09-11T01:38:53.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.923+0000 7fb17f52b640 1 -- 172.21.15.12:0/1057093081 shutdown_connections 2024-09-11T01:38:53.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.923+0000 7fb17f52b640 1 --2- 172.21.15.12:0/1057093081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb178138230 0x7fb1781341e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:53.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.923+0000 7fb17f52b640 1 -- 172.21.15.12:0/1057093081 >> 172.21.15.12:0/1057093081 conn(0x7fb1780a7020 msgr2=0x7fb1781361c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:53.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.923+0000 7fb17f52b640 1 -- 172.21.15.12:0/1057093081 shutdown_connections 2024-09-11T01:38:53.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:53.923+0000 7fb17f52b640 1 -- 172.21.15.12:0/1057093081 wait complete. 2024-09-11T01:38:53.950 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:38:53.951 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T01:38:53.951 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:38:54.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.051+0000 7f9602332640 10 monclient: get_monmap_and_config 2024-09-11T01:38:54.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.051+0000 7f9602332640 10 monclient: build_initial_monmap 2024-09-11T01:38:54.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.051+0000 7f9602332640 10 monclient: monmap: 2024-09-11T01:38:54.053 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:54.053 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:54.053 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:38:54.052299+0000 2024-09-11T01:38:54.053 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:38:54.052299+0000 2024-09-11T01:38:54.053 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:54.053 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:54.053 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:38:54.054 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:38:54.054 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:38:54.054 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:38:54.054 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:38:54.054 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:54.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.052+0000 7f9602332640 1 Processor -- start 2024-09-11T01:38:54.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.052+0000 7f9602332640 1 -- start start 2024-09-11T01:38:54.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.052+0000 7f9602332640 10 monclient: init 2024-09-11T01:38:54.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.052+0000 7f9602332640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:54.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.052+0000 7f9602332640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-11T01:38:54.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.052+0000 7f9602332640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f95fc074180 addr v1:172.21.15.12:6789/0 2024-09-11T01:38:54.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.052+0000 7f9602332640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f95fc0a06d0 addr v1:172.21.15.71:6789/0 2024-09-11T01:38:54.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.052+0000 7f9602332640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f95fc134ac0 0x7f95fc1405f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:54.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.052+0000 7f9602332640 10 monclient(hunting): picked mon.noname-b con 0x7f95fc134ac0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:54.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.052+0000 7f9602332640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95fc070160 con 0x7f95fc074180 2024-09-11T01:38:54.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.052+0000 7f9602332640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95fc070440 con 0x7f95fc0a06d0 2024-09-11T01:38:54.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.052+0000 7f9602332640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:54.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.052+0000 7f9602332640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f95fc135560 con 0x7f95fc134ac0 2024-09-11T01:38:54.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.052+0000 7f9602332640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:54.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.052+0000 7f9602332640 10 monclient(hunting): authenticate will time out at 1340.494902s 2024-09-11T01:38:54.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.052+0000 7f96008cf640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f95fc134ac0 msgr2=0x7f95fc1405f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:38:54.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.052+0000 7f96008cf640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f95fc134ac0 0x7f95fc1405f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:38:54.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.052+0000 7f95fbfff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f95fc074180 0x7f95fc074560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50834/0 (socket says 172.21.15.12:50834) 2024-09-11T01:38:54.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.053+0000 7f95fbfff640 1 -- 172.21.15.12:0/406554321 learned_addr learned my addr 172.21.15.12:0/406554321 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:38:54.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95fbfff640 1 -- 172.21.15.12:0/406554321 >> v1:172.21.15.12:6789/0 conn(0x7f95fc074180 legacy=0x7f95fc074560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:54.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95fb7fe640 1 -- 172.21.15.12:0/406554321 >> v1:172.21.15.71:6789/0 conn(0x7f95fc0a06d0 legacy=0x7f95fc13cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:54.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 1 -- 172.21.15.12:0/406554321 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1522233472 0 0) 0x7f95fc070160 con 0x7f95fc074180 2024-09-11T01:38:54.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 10 monclient(hunting): _negotiate 2024-09-11T01:38:54.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:54.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:54.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 10 monclient(hunting): my global_id is 94266 2024-09-11T01:38:54.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 1 -- 172.21.15.12:0/406554321 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95e8003520 con 0x7f95fc074180 2024-09-11T01:38:54.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 1 -- 172.21.15.12:0/406554321 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1710827887 0 0) 0x7f95fc070440 con 0x7f95fc0a06d0 2024-09-11T01:38:54.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 10 monclient(hunting): _negotiate 2024-09-11T01:38:54.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:54.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:54.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 10 monclient(hunting): my global_id is 104146 2024-09-11T01:38:54.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 1 -- 172.21.15.12:0/406554321 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95fc070160 con 0x7f95fc0a06d0 2024-09-11T01:38:54.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 1 -- 172.21.15.12:0/406554321 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3542815143 0 0) 0x7f95e8003520 con 0x7f95fc074180 2024-09-11T01:38:54.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 1 -- 172.21.15.12:0/406554321 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f95fc134ac0 msgr2=0x7f95fc1405f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:38:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 1 --2- 172.21.15.12:0/406554321 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f95fc134ac0 0x7f95fc1405f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 1 -- 172.21.15.12:0/406554321 >> v1:172.21.15.71:6789/0 conn(0x7f95fc0a06d0 legacy=0x7f95fc13cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 10 monclient: _finish_hunting 0 2024-09-11T01:38:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:38:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:38:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 1 -- 172.21.15.12:0/406554321 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95fc068bc0 con 0x7f95fc074180 2024-09-11T01:38:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 10 monclient: _finish_auth 0 2024-09-11T01:38:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 10 monclient: _check_auth_tickets 2024-09-11T01:38:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 1 -- 172.21.15.12:0/406554321 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f95ec0019a0 con 0x7f95fc074180 2024-09-11T01:38:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f9602332640 5 monclient: authenticate success, global_id 94266 2024-09-11T01:38:54.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f9602332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:38:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:38:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 10 monclient: dump: 2024-09-11T01:38:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:54.070 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:54.070 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:38:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 1 -- 172.21.15.12:0/406554321 >> v1:172.21.15.12:6789/0 conn(0x7f95fc074180 legacy=0x7f95fc074560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:38:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 1 --2- 172.21.15.12:0/406554321 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f95fc134ac0 0x7f95e8003c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 10 monclient(hunting): picked mon.c con 0x7f95fc134ac0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 1 --2- 172.21.15.12:0/406554321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95e80041e0 0x7f95e8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 10 monclient(hunting): picked mon.a con 0x7f95e80041e0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:38:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 1 --2- 172.21.15.12:0/406554321 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95e8004f90 0x7f95e800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 10 monclient(hunting): picked mon.b con 0x7f95e8004f90 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:38:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95fbfff640 1 -- 172.21.15.12:0/406554321 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f95fc134ac0 msgr2=0x7f95e8003c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:38:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 1 -- 172.21.15.12:0/406554321 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f95e800bf00 con 0x7f95e80041e0 2024-09-11T01:38:54.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 1 -- 172.21.15.12:0/406554321 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f95e800c080 con 0x7f95e8004f90 2024-09-11T01:38:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 1 -- 172.21.15.12:0/406554321 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f95e800c200 con 0x7f95fc134ac0 2024-09-11T01:38:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95fbfff640 1 --2- 172.21.15.12:0/406554321 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f95fc134ac0 0x7f95e8003c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:38:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95faffd640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:38:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f9602332640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:38:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.055+0000 7f95fb7fe640 1 --2- 172.21.15.12:0/406554321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95e80041e0 0x7f95e8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f95fb7fe640 10 monclient(hunting): get_auth_request con 0x7f95e80041e0 auth_method 0 2024-09-11T01:38:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f95fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f95fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f95fb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:38:54.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f96008cf640 1 --2- 172.21.15.12:0/406554321 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95e8004f90 0x7f95e800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:54.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f96008cf640 10 monclient(hunting): get_auth_request con 0x7f95e8004f90 auth_method 0 2024-09-11T01:38:54.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f96008cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:54.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f96008cf640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:54.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f96008cf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:38:54.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f95fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:54.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f95fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:54.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f95fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:38:54.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f96008cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:54.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f96008cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:54.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f96008cf640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:38:54.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f95fb7fe640 10 monclient(hunting): handle_auth_done global_id 94266 payload 293 2024-09-11T01:38:54.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f95fb7fe640 1 -- 172.21.15.12:0/406554321 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f95fc134ac0 msgr2=0x7f95e8003c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:38:54.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f95fb7fe640 1 --2- 172.21.15.12:0/406554321 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f95fc134ac0 0x7f95e8003c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:54.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f95fb7fe640 1 -- 172.21.15.12:0/406554321 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95e8004f90 msgr2=0x7f95e800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:38:54.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f95fb7fe640 1 --2- 172.21.15.12:0/406554321 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95e8004f90 0x7f95e800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:54.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f95fb7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:38:54.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f95fb7fe640 1 monclient: found mon.a 2024-09-11T01:38:54.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f95fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:54.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f95fb7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:38:54.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f95fb7fe640 1 -- 172.21.15.12:0/406554321 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e8003520 con 0x7f95e80041e0 2024-09-11T01:38:54.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f95fb7fe640 10 monclient: _finish_auth 0 2024-09-11T01:38:54.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f95fb7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:38:54.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.056+0000 7f95fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:54.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.058+0000 7f95fb7fe640 1 -- 172.21.15.12:0/406554321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95e80041e0 msgr2=0x7f95e8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:54.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.058+0000 7f95fb7fe640 1 --2- 172.21.15.12:0/406554321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95e80041e0 0x7f95e8004800 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f95e4007bf0 tx=0x7f95e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c81a1485300d5b30 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:54.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.059+0000 7f95faffd640 1 -- 172.21.15.12:0/406554321 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e4007920 con 0x7f95e80041e0 2024-09-11T01:38:54.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.059+0000 7f95faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:54.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.059+0000 7f95faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:38:54.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.059+0000 7f95faffd640 10 monclient: dump: 2024-09-11T01:38:54.075 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:54.075 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:54.075 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:54.075 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:54.075 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:54.075 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:54.075 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:54.075 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:54.075 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:54.076 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:54.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.059+0000 7f9602332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:38:54.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.059+0000 7f95faffd640 1 -- 172.21.15.12:0/406554321 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95e402e070 con 0x7f95e80041e0 2024-09-11T01:38:54.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.059+0000 7f95faffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:54.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.059+0000 7f9602332640 10 monclient: get_monmap_and_config success 2024-09-11T01:38:54.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.059+0000 7f9602332640 4 set_mon_vals no callback set 2024-09-11T01:38:54.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.059+0000 7f95faffd640 1 -- 172.21.15.12:0/406554321 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e4005a00 con 0x7f95e80041e0 2024-09-11T01:38:54.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.059+0000 7f95faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:54.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.059+0000 7f95faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:38:54.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.059+0000 7f95faffd640 10 monclient: dump: 2024-09-11T01:38:54.076 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:54.076 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:54.076 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:54.076 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:54.077 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:54.077 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:54.077 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:54.077 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:54.077 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:54.077 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:54.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.059+0000 7f9602332640 10 monclient: shutdown 2024-09-11T01:38:54.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.059+0000 7f9602332640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:54.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.059+0000 7f9602332640 1 -- 172.21.15.12:0/406554321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95e80041e0 msgr2=0x7f95e8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:54.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.059+0000 7f9602332640 1 --2- 172.21.15.12:0/406554321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95e80041e0 0x7f95e8004800 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f95e4007bf0 tx=0x7f95e8005830 comp rx=0 tx=0).stop 2024-09-11T01:38:54.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.059+0000 7f96008cf640 1 -- 172.21.15.12:0/406554321 reap_dead start 2024-09-11T01:38:54.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.059+0000 7f9602332640 1 -- 172.21.15.12:0/406554321 shutdown_connections 2024-09-11T01:38:54.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.059+0000 7f9602332640 1 -- 172.21.15.12:0/406554321 >> 172.21.15.12:0/406554321 conn(0x7f95fc0a36c0 msgr2=0x7f95fc0a3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:54.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.059+0000 7f9602332640 1 -- 172.21.15.12:0/406554321 shutdown_connections 2024-09-11T01:38:54.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.059+0000 7f9602332640 1 -- 172.21.15.12:0/406554321 wait complete. 2024-09-11T01:38:54.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.060+0000 7f9602332640 10 monclient: build_initial_monmap 2024-09-11T01:38:54.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.060+0000 7f9602332640 10 monclient: monmap: 2024-09-11T01:38:54.078 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:38:54.078 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:38:54.078 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:38:54.078 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:38:54.078 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:38:54.078 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:54.078 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:38:54.078 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:38:54.078 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:38:54.079 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:54.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.060+0000 7f9602332640 1 Processor -- start 2024-09-11T01:38:54.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.060+0000 7f9602332640 1 -- start start 2024-09-11T01:38:54.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.060+0000 7f9602332640 10 monclient: init 2024-09-11T01:38:54.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f9602332640 10 monclient: _reopen_session rank -1 2024-09-11T01:38:54.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f9602332640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:38:54.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f9602332640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95e80041e0 0x7f95fc13a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:54.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f9602332640 10 monclient(hunting): picked mon.noname-a con 0x7f95e80041e0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:38:54.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f9602332640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95fc131d80 0x7f95fc132200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:54.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f9602332640 10 monclient(hunting): picked mon.noname-b con 0x7f95fc131d80 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:38:54.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f9602332640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f95fc1cccd0 0x7f95fc1cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:54.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f9602332640 10 monclient(hunting): picked mon.noname-c con 0x7f95fc1cccd0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:38:54.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f9602332640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:54.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f9602332640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc070740 con 0x7f95e80041e0 2024-09-11T01:38:54.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f9602332640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:54.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f9602332640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc13e7f0 con 0x7f95fc131d80 2024-09-11T01:38:54.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f9602332640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:54.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f9602332640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f95fc13e970 con 0x7f95fc1cccd0 2024-09-11T01:38:54.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f9602332640 10 monclient(hunting): _renew_subs 2024-09-11T01:38:54.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f9602332640 10 monclient(hunting): authenticate will time out at 1340.503336s 2024-09-11T01:38:54.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f96008cf640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f95fc1cccd0 msgr2=0x7f95fc1cd100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:38:54.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f96008cf640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f95fc1cccd0 0x7f95fc1cd100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:38:54.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95e80041e0 0x7f95fc13a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:54.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95e80041e0 0x7f95fc13a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33070/0 (socket says 172.21.15.12:33070) 2024-09-11T01:38:54.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fbfff640 1 -- 172.21.15.12:0/4194857289 learned_addr learned my addr 172.21.15.12:0/4194857289 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:38:54.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fbfff640 10 monclient(hunting): get_auth_request con 0x7f95e80041e0 auth_method 0 2024-09-11T01:38:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fb7fe640 1 --2- 172.21.15.12:0/4194857289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95fc131d80 0x7f95fc132200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fb7fe640 10 monclient(hunting): get_auth_request con 0x7f95fc131d80 auth_method 0 2024-09-11T01:38:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:54.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:54.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:54.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fbfff640 10 monclient(hunting): handle_auth_done global_id 94269 payload 824 2024-09-11T01:38:54.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fbfff640 1 -- 172.21.15.12:0/4194857289 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f95fc1cccd0 msgr2=0x7f95fc1cd100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:38:54.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fbfff640 1 --2- 172.21.15.12:0/4194857289 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f95fc1cccd0 0x7f95fc1cd100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:54.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fbfff640 1 -- 172.21.15.12:0/4194857289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95fc131d80 msgr2=0x7f95fc132200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:54.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fbfff640 1 --2- 172.21.15.12:0/4194857289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95fc131d80 0x7f95fc132200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:38:54.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fbfff640 10 monclient: _finish_hunting 0 2024-09-11T01:38:54.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fbfff640 1 monclient: found mon.noname-a 2024-09-11T01:38:54.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:38:54.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:38:54.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.061+0000 7f95fbfff640 1 -- 172.21.15.12:0/4194857289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95fc13b940 con 0x7f95e80041e0 2024-09-11T01:38:54.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.063+0000 7f95fbfff640 1 -- 172.21.15.12:0/4194857289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95e80041e0 msgr2=0x7f95fc13a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:54.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.063+0000 7f95fbfff640 1 --2- 172.21.15.12:0/4194857289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95e80041e0 0x7f95fc13a3c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f95ec000f60 tx=0x7f95ec00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.064+0000 7f95f8ff9640 1 -- 172.21.15.12:0/4194857289 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ec007560 con 0x7f95e80041e0 2024-09-11T01:38:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.064+0000 7f95f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.064+0000 7f95f8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:38:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.064+0000 7f95f8ff9640 10 monclient: dump: 2024-09-11T01:38:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:54.083 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:54.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.064+0000 7f95f8ff9640 10 monclient: _finish_auth 0 2024-09-11T01:38:54.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.064+0000 7f95f8ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:38:54.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.064+0000 7f95f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:38:54.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.064+0000 7f9602332640 5 monclient: authenticate success, global_id 94269 2024-09-11T01:38:54.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.064+0000 7f9602332640 10 monclient: _renew_subs 2024-09-11T01:38:54.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.064+0000 7f95f8ff9640 1 -- 172.21.15.12:0/4194857289 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95ec0047e0 con 0x7f95e80041e0 2024-09-11T01:38:54.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.064+0000 7f9602332640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:38:54.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.064+0000 7f9602332640 1 -- 172.21.15.12:0/4194857289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95fc0a6510 con 0x7f95e80041e0 2024-09-11T01:38:54.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.064+0000 7f95f8ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:38:54.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.064+0000 7f95f8ff9640 1 -- 172.21.15.12:0/4194857289 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ec004bd0 con 0x7f95e80041e0 2024-09-11T01:38:54.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.064+0000 7f95f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:54.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.064+0000 7f95f8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:38:54.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.064+0000 7f95f8ff9640 10 monclient: dump: 2024-09-11T01:38:54.084 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:54.085 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:54.085 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:54.085 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:54.085 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:54.085 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:54.085 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:54.085 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:54.085 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:54.085 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:54.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.064+0000 7f9602332640 10 monclient: _renew_subs 2024-09-11T01:38:54.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.064+0000 7f9602332640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:38:54.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.064+0000 7f95fa7fc640 4 set_mon_vals no callback set 2024-09-11T01:38:54.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.064+0000 7f9602332640 1 -- 172.21.15.12:0/4194857289 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95fc13a900 con 0x7f95e80041e0 2024-09-11T01:38:54.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.065+0000 7f9602332640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:54.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.065+0000 7f9602332640 1 --2- 172.21.15.12:0/4194857289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95fc1cd640 0x7f95fc135710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:54.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.065+0000 7f9602332640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:54.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.065+0000 7f9602332640 1 -- 172.21.15.12:0/4194857289 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc0a0490 con 0x7f95fc1cd640 2024-09-11T01:38:54.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.065+0000 7f95f8ff9640 1 -- 172.21.15.12:0/4194857289 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f95ec00a640 con 0x7f95e80041e0 2024-09-11T01:38:54.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.065+0000 7f95fb7fe640 1 --2- 172.21.15.12:0/4194857289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95fc1cd640 0x7f95fc135710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:54.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.065+0000 7f95f8ff9640 1 --2- 172.21.15.12:0/4194857289 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f95dc03e1f0 0x7f95dc0406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:54.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.065+0000 7f95fb7fe640 10 monclient: get_auth_request con 0x7f95fc1cd640 auth_method 0 2024-09-11T01:38:54.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.065+0000 7f95fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:54.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.065+0000 7f95fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:54.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.065+0000 7f95fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:54.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.065+0000 7f95f8ff9640 1 -- 172.21.15.12:0/4194857289 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(323..323 src has 186..323) ==== 4755+0+0 (secure 0 0 0) 0x7f95ec052be0 con 0x7f95e80041e0 2024-09-11T01:38:54.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.065+0000 7f95fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.065+0000 7f95fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.065+0000 7f95fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.065+0000 7f96008cf640 1 --2- 172.21.15.12:0/4194857289 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f95dc03e1f0 0x7f95dc0406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.065+0000 7f96008cf640 10 monclient: get_auth_request con 0x7f95dc03e1f0 auth_method 0 2024-09-11T01:38:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.065+0000 7f95fb7fe640 10 monclient(hunting): handle_auth_done global_id 104152 payload 824 2024-09-11T01:38:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.065+0000 7f95fb7fe640 1 -- 172.21.15.12:0/4194857289 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f95fc074fe0 con 0x7f95fc1cd640 2024-09-11T01:38:54.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.068+0000 7f95fb7fe640 1 -- 172.21.15.12:0/4194857289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95fc1cd640 msgr2=0x7f95fc135710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:54.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.068+0000 7f95fb7fe640 1 --2- 172.21.15.12:0/4194857289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95fc1cd640 0x7f95fc135710 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f95e4032000 tx=0x7f95e40016b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:54.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.068+0000 7f96008cf640 1 --2- 172.21.15.12:0/4194857289 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f95dc03e1f0 0x7f95dc0406b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f95e800c3d0 tx=0x7f95f4009540 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:54.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.068+0000 7f95f8ff9640 1 -- 172.21.15.12:0/4194857289 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e4042020 con 0x7f95fc1cd640 2024-09-11T01:38:54.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.068+0000 7f95f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:54.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.068+0000 7f95f8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:38:54.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.068+0000 7f95f8ff9640 10 monclient: dump: 2024-09-11T01:38:54.090 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:54.090 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:54.090 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:54.090 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:54.090 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:54.090 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:54.090 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:54.090 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:54.090 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:54.090 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:54.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.068+0000 7f95f8ff9640 1 -- 172.21.15.12:0/4194857289 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f95e402e070 con 0x7f95fc1cd640 2024-09-11T01:38:54.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.068+0000 7f95f8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:38:54.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.068+0000 7f95f8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:38:54.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.068+0000 7f95f8ff9640 1 -- 172.21.15.12:0/4194857289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95fc1cd640 msgr2=0x7f95fc135710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:54.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.068+0000 7f95f8ff9640 1 --2- 172.21.15.12:0/4194857289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95fc1cd640 0x7f95fc135710 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f95e4032000 tx=0x7f95e40016b0 comp rx=0 tx=0).stop 2024-09-11T01:38:54.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.068+0000 7f95f8ff9640 1 -- 172.21.15.12:0/4194857289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95fc1cd640 msgr2=0x7f95fc135710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:38:54.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.068+0000 7f95f8ff9640 1 --2- 172.21.15.12:0/4194857289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95fc1cd640 0x7f95fc135710 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f95e4032000 tx=0x7f95e40016b0 comp rx=0 tx=0).stop 2024-09-11T01:38:54.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.074+0000 7f9602332640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:38:54.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.074+0000 7f9602332640 1 --2- 172.21.15.12:0/4194857289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95fc135c50 0x7f95fc136060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:38:54.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.074+0000 7f9602332640 10 monclient(hunting): start opening mon connection 2024-09-11T01:38:54.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.074+0000 7f9602332640 1 -- 172.21.15.12:0/4194857289 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc13b3f0 con 0x7f95fc135c50 2024-09-11T01:38:54.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.074+0000 7f95fb7fe640 1 --2- 172.21.15.12:0/4194857289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95fc135c50 0x7f95fc136060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:38:54.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.074+0000 7f95fb7fe640 10 monclient: get_auth_request con 0x7f95fc135c50 auth_method 0 2024-09-11T01:38:54.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.074+0000 7f95fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:38:54.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.074+0000 7f95fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:38:54.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.074+0000 7f95fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:38:54.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.075+0000 7f95fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:38:54.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.075+0000 7f95fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:38:54.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.075+0000 7f95fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:38:54.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.075+0000 7f95fb7fe640 10 monclient(hunting): handle_auth_done global_id 104155 payload 824 2024-09-11T01:38:54.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.075+0000 7f95fb7fe640 1 -- 172.21.15.12:0/4194857289 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f95fc074560 con 0x7f95fc135c50 2024-09-11T01:38:54.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.077+0000 7f95fb7fe640 1 -- 172.21.15.12:0/4194857289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95fc135c50 msgr2=0x7f95fc136060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:38:54.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.077+0000 7f95fb7fe640 1 --2- 172.21.15.12:0/4194857289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95fc135c50 0x7f95fc136060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f95e4000c00 tx=0x7f95e4009030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:38:54.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.077+0000 7f95f8ff9640 1 -- 172.21.15.12:0/4194857289 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e4042030 con 0x7f95fc135c50 2024-09-11T01:38:54.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.077+0000 7f95f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:38:54.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.077+0000 7f95f8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:38:54.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.077+0000 7f95f8ff9640 10 monclient: dump: 2024-09-11T01:38:54.092 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:38:54.092 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:38:54.093 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:54.093 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:38:54.093 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:38:54.093 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:38:54.093 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:38:54.093 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:38:54.093 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:38:54.093 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:38:54.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.077+0000 7f95f8ff9640 1 -- 172.21.15.12:0/4194857289 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4863 (secure 0 0 0) 0x7f95e4049e50 con 0x7f95fc135c50 2024-09-11T01:38:54.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.077+0000 7f95f8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:38:54.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.077+0000 7f95f8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:38:54.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.077+0000 7f95f8ff9640 1 -- 172.21.15.12:0/4194857289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95fc135c50 msgr2=0x7f95fc136060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:54.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.077+0000 7f95f8ff9640 1 --2- 172.21.15.12:0/4194857289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95fc135c50 0x7f95fc136060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f95e4000c00 tx=0x7f95e4009030 comp rx=0 tx=0).stop 2024-09-11T01:38:54.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.077+0000 7f95f8ff9640 1 -- 172.21.15.12:0/4194857289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95fc135c50 msgr2=0x7f95fc136060 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:38:54.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.077+0000 7f95f8ff9640 1 --2- 172.21.15.12:0/4194857289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95fc135c50 0x7f95fc136060 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f95e4000c00 tx=0x7f95e4009030 comp rx=0 tx=0).stop 2024-09-11T01:38:54.094 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:38:54.094 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:38:54.094 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:38:54.094 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:38:54.094 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 94, 2024-09-11T01:38:54.094 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 34572, 2024-09-11T01:38:54.094 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:38:54.094 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:38:54.095 INFO:teuthology.orchestra.run.smithi012.stdout: 1 2024-09-11T01:38:54.095 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:54.095 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:38:54.095 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:38:54.095 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:38:54.095 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:38:54.095 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:38:54.095 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:38:54.095 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:38:54.095 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:38:54.095 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:38:54.095 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:38:54.095 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:38:54.096 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:38:54.096 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:38:54.096 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:38:54.096 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:38:54.096 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:54.096 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:38:54.096 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:38:54.096 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:38:54.096 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:38:54.096 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:38:54.096 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:38:54.096 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:38:54.096 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:38:54.097 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:38:54.097 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:38:54.097 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:38:54.097 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:38:54.097 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:38:54.097 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:54.097 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:54.097 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:38:54.097 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:38:54.097 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:38:54.097 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:38:54.097 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:38:54.098 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:38:54.098 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:38:54.098 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:38:54.098 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:38:54.098 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:38:54.098 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:38:54.098 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:38:54.098 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:38:54.098 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:38:54.098 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:38:54.098 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:38:54.098 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:38:54.098 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:38:54.098 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:38:54.099 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:38:54.099 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:38:54.099 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:38:54.099 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:38:54.099 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:38:54.099 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:38:54.099 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:38:54.099 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:38:54.099 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:38:54.099 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:54.099 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:38:54.099 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:54.099 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:38:54.100 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:54.100 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:38:54.100 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:38:54.100 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:38:54.100 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:38:54.100 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:54.100 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:38:54.100 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:38:54.100 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:54.100 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:54.100 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:54.100 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:38:54.100 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:38:54.101 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:54.101 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:54.101 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:54.101 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:54.101 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:38:54.101 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:38:54.101 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:38:54.101 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:38:54.101 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:38:54.101 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:54.101 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:54.101 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:38:54.101 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:38:54.102 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:38:54.102 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:38:54.102 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:54.102 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:38:54.102 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:38:54.102 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:54.102 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:54.102 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:54.102 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:38:54.102 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:38:54.102 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:54.102 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:54.102 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:54.103 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:54.103 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:38:54.103 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:38:54.103 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:38:54.103 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:38:54.103 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:38:54.103 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:54.103 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:54.103 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:38:54.103 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:38:54.103 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:38:54.103 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:38:54.103 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:54.104 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:38:54.104 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:38:54.104 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:54.104 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:54.104 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:54.104 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:38:54.104 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:38:54.104 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:38:54.104 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:54.104 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:54.104 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:54.104 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:38:54.104 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:38:54.104 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:38:54.105 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:38:54.105 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:38:54.105 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:54.105 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:54.105 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:54.105 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:38:54.105 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:38:54.105 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:54.105 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:38:54.105 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:38:54.105 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:38:54.105 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:54.105 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:54.106 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:38:54.106 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:54.106 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:38:54.106 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:38:54.106 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:38:54.106 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:54.106 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:54.106 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:38:54.106 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:54.106 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:38:54.106 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:38:54.106 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:38:54.106 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:54.107 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:38:54.107 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:38:54.107 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:38:54.107 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:38:54.107 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:38:54.107 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:38:54.107 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:38:54.107 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:38:54.107 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:38:54.107 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:38:54.107 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:38:54.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.078+0000 7f9602332640 1 -- 172.21.15.12:0/4194857289 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f95dc03e1f0 msgr2=0x7f95dc0406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:54.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.078+0000 7f9602332640 1 --2- 172.21.15.12:0/4194857289 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f95dc03e1f0 0x7f95dc0406b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f95e800c3d0 tx=0x7f95f4009540 comp rx=0 tx=0).stop 2024-09-11T01:38:54.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.078+0000 7f9602332640 10 monclient: shutdown 2024-09-11T01:38:54.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.078+0000 7f9602332640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:38:54.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.078+0000 7f9602332640 1 -- 172.21.15.12:0/4194857289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95e80041e0 msgr2=0x7f95fc13a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:38:54.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.078+0000 7f9602332640 1 --2- 172.21.15.12:0/4194857289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f95e80041e0 0x7f95fc13a3c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f95ec000f60 tx=0x7f95ec00cb90 comp rx=0 tx=0).stop 2024-09-11T01:38:54.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.078+0000 7f96008cf640 1 -- 172.21.15.12:0/4194857289 reap_dead start 2024-09-11T01:38:54.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.078+0000 7f96008cf640 1 -- 172.21.15.12:0/4194857289 reap_dead start 2024-09-11T01:38:54.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.078+0000 7f9602332640 1 -- 172.21.15.12:0/4194857289 shutdown_connections 2024-09-11T01:38:54.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.078+0000 7f9602332640 1 -- 172.21.15.12:0/4194857289 >> 172.21.15.12:0/4194857289 conn(0x7f95fc0a36c0 msgr2=0x7f95fc13ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:38:54.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.078+0000 7f9602332640 1 -- 172.21.15.12:0/4194857289 shutdown_connections 2024-09-11T01:38:54.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:38:54.078+0000 7f9602332640 1 -- 172.21.15.12:0/4194857289 wait complete. 2024-09-11T01:38:54.110 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:38:54.110 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:38:54.929 INFO:tasks.rados.rados.0.smithi071.stdout:1424: done (0 left) 2024-09-11T01:38:54.929 INFO:tasks.rados.rados.0.smithi071.stdout:1425: rollback oid 17 current snap is 150 2024-09-11T01:38:54.929 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 17 to 145 2024-09-11T01:38:54.929 INFO:tasks.rados.rados.0.smithi071.stdout:1426: copy_from oid 29 from oid 36 current snap is 150 2024-09-11T01:38:54.930 INFO:tasks.rados.rados.0.smithi071.stdout:1427: read oid 6 snap -1 2024-09-11T01:38:54.930 INFO:tasks.rados.rados.0.smithi071.stdout:1427: expect deleted 2024-09-11T01:38:54.930 INFO:tasks.rados.rados.0.smithi071.stdout:1428: snap_remove snap 148 2024-09-11T01:38:56.409 INFO:tasks.rados.rados.0.smithi071.stdout:1428: done (3 left) 2024-09-11T01:38:56.409 INFO:tasks.rados.rados.0.smithi071.stdout:1429: copy_from oid 9 from oid 4 current snap is 150 2024-09-11T01:38:56.409 INFO:tasks.rados.rados.0.smithi071.stdout:1425: finishing rollback tid 0 to smithi07134777-17 2024-09-11T01:38:56.409 INFO:tasks.rados.rados.0.smithi071.stdout:1425: finishing rollback tid 1 to smithi07134777-17 2024-09-11T01:38:56.409 INFO:tasks.rados.rados.0.smithi071.stdout:1425: finishing rollback tid 2 to smithi07134777-17 2024-09-11T01:38:56.409 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 515 (ObjNum 250 snap 63 seq_num 250) dirty exists 2024-09-11T01:38:56.409 INFO:tasks.rados.rados.0.smithi071.stdout:1426: finishing copy_from racing read to smithi07134777-29 2024-09-11T01:38:56.409 INFO:tasks.rados.rados.0.smithi071.stdout:1426: finishing copy_from to smithi07134777-29 2024-09-11T01:38:56.410 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 355 (ObjNum 469 snap 124 seq_num 469) dirty exists 2024-09-11T01:38:56.410 INFO:tasks.rados.rados.0.smithi071.stdout:1425: done (3 left) 2024-09-11T01:38:56.410 INFO:tasks.rados.rados.0.smithi071.stdout:1426: done (2 left) 2024-09-11T01:38:56.410 INFO:tasks.rados.rados.0.smithi071.stdout:1427: done (1 left) 2024-09-11T01:38:56.410 INFO:tasks.rados.rados.0.smithi071.stdout:1430: write oid 38 current snap is 150 2024-09-11T01:38:56.410 INFO:tasks.rados.rados.0.smithi071.stdout:1430: seq_num 535 ranges {591745=402925,1790337=563515} 2024-09-11T01:38:56.414 INFO:tasks.rados.rados.0.smithi071.stdout:1430: writing smithi07134777-38 from 591745 to 994670 tid 1 2024-09-11T01:38:56.422 INFO:tasks.rados.rados.0.smithi071.stdout:1430: writing smithi07134777-38 from 1790337 to 2353852 tid 2 2024-09-11T01:38:56.422 INFO:tasks.rados.rados.0.smithi071.stdout:1431: snap_create 2024-09-11T01:38:56.422 INFO:tasks.rados.rados.0.smithi071.stdout:1430: finishing write tid 1 to smithi07134777-38 2024-09-11T01:38:56.424 INFO:tasks.rados.rados.0.smithi071.stdout:1429: finishing copy_from racing read to smithi07134777-9 2024-09-11T01:38:56.428 INFO:tasks.rados.rados.0.smithi071.stdout:1429: finishing copy_from to smithi07134777-9 2024-09-11T01:38:56.428 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 528 (ObjNum 507 snap 138 seq_num 507) dirty exists 2024-09-11T01:38:56.428 INFO:tasks.rados.rados.0.smithi071.stdout:1430: finishing write tid 2 to smithi07134777-38 2024-09-11T01:38:56.431 INFO:tasks.rados.rados.0.smithi071.stdout:1430: finishing write tid 3 to smithi07134777-38 2024-09-11T01:38:56.431 INFO:tasks.rados.rados.0.smithi071.stdout:1430: finishing write tid 4 to smithi07134777-38 2024-09-11T01:38:56.431 INFO:tasks.rados.rados.0.smithi071.stdout:1430: oid 38 updating version 0 to 757 2024-09-11T01:38:56.431 INFO:tasks.rados.rados.0.smithi071.stdout:1430: oid 38 updating version 757 to 761 2024-09-11T01:38:56.431 INFO:tasks.rados.rados.0.smithi071.stdout:1430: oid 38 version 761 is already newer than 760 2024-09-11T01:38:56.432 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 761 (ObjNum 535 snap 150 seq_num 535) dirty exists 2024-09-11T01:38:56.432 INFO:tasks.rados.rados.0.smithi071.stdout:1430: left oid 38 (ObjNum 535 snap 150 seq_num 535) 2024-09-11T01:38:57.407 INFO:tasks.rados.rados.0.smithi071.stdout:1429: done (2 left) 2024-09-11T01:38:57.407 INFO:tasks.rados.rados.0.smithi071.stdout:1430: done (1 left) 2024-09-11T01:38:57.408 INFO:tasks.rados.rados.0.smithi071.stdout:1431: done (0 left) 2024-09-11T01:38:57.408 INFO:tasks.rados.rados.0.smithi071.stdout:1432: copy_from oid 34 from oid 11 current snap is 151 2024-09-11T01:38:57.408 INFO:tasks.rados.rados.0.smithi071.stdout:1433: delete oid 6 current snap is 151 2024-09-11T01:38:57.412 INFO:tasks.rados.rados.0.smithi071.stdout:1433: done (1 left) 2024-09-11T01:38:57.412 INFO:tasks.rados.rados.0.smithi071.stdout:1434: read oid 2 snap -1 2024-09-11T01:38:57.412 INFO:tasks.rados.rados.0.smithi071.stdout:1434: expect (ObjNum 503 snap 135 seq_num 503) 2024-09-11T01:38:57.412 INFO:tasks.rados.rados.0.smithi071.stdout:1435: copy_from oid 47 from oid 20 current snap is 151 2024-09-11T01:38:57.412 INFO:tasks.rados.rados.0.smithi071.stdout:1436: snap_create 2024-09-11T01:38:57.418 INFO:tasks.rados.rados.0.smithi071.stdout:1434: expect (ObjNum 503 snap 135 seq_num 503) 2024-09-11T01:38:57.422 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 694790 2024-09-11T01:38:57.442 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:38:57.472 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 940022 2024-09-11T01:38:57.488 INFO:tasks.rados.rados.0.smithi071.stdout:1432: finishing copy_from racing read to smithi07134777-34 2024-09-11T01:38:57.488 INFO:tasks.rados.rados.0.smithi071.stdout:1432: finishing copy_from to smithi07134777-34 2024-09-11T01:38:57.488 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 631 (ObjNum 524 snap 144 seq_num 524) dirty exists 2024-09-11T01:38:57.488 INFO:tasks.rados.rados.0.smithi071.stdout:1435: finishing copy_from racing read to smithi07134777-47 2024-09-11T01:38:57.488 INFO:tasks.rados.rados.0.smithi071.stdout:1435: finishing copy_from to smithi07134777-47 2024-09-11T01:38:57.488 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 263 (ObjNum 513 snap 141 seq_num 513) dirty exists 2024-09-11T01:38:58.420 INFO:tasks.rados.rados.0.smithi071.stdout:1432: done (3 left) 2024-09-11T01:38:58.420 INFO:tasks.rados.rados.0.smithi071.stdout:1434: done (2 left) 2024-09-11T01:38:58.420 INFO:tasks.rados.rados.0.smithi071.stdout:1435: done (1 left) 2024-09-11T01:38:58.420 INFO:tasks.rados.rados.0.smithi071.stdout:1436: done (0 left) 2024-09-11T01:38:58.420 INFO:tasks.rados.rados.0.smithi071.stdout:1437: write oid 3 current snap is 152 2024-09-11T01:38:58.420 INFO:tasks.rados.rados.0.smithi071.stdout:1437: seq_num 537 ranges {625359=553423,1790948=567184,2930500=94859} 2024-09-11T01:38:58.426 INFO:tasks.rados.rados.0.smithi071.stdout:1437: writing smithi07134777-3 from 625359 to 1178782 tid 1 2024-09-11T01:38:58.432 INFO:tasks.rados.rados.0.smithi071.stdout:1437: writing smithi07134777-3 from 1790948 to 2358132 tid 2 2024-09-11T01:38:58.437 INFO:tasks.rados.rados.0.smithi071.stdout:1437: writing smithi07134777-3 from 2930500 to 3025359 tid 3 2024-09-11T01:38:58.437 INFO:tasks.rados.rados.0.smithi071.stdout:1438: read oid 44 snap -1 2024-09-11T01:38:58.437 INFO:tasks.rados.rados.0.smithi071.stdout:1438: expect (ObjNum 477 snap 125 seq_num 477) 2024-09-11T01:38:58.437 INFO:tasks.rados.rados.0.smithi071.stdout:1439: write (excl) oid 27 current snap is 152 2024-09-11T01:38:58.437 INFO:tasks.rados.rados.0.smithi071.stdout:1437: finishing write tid 1 to smithi07134777-3 2024-09-11T01:38:58.437 INFO:tasks.rados.rados.0.smithi071.stdout:1437: finishing write tid 2 to smithi07134777-3 2024-09-11T01:38:58.437 INFO:tasks.rados.rados.0.smithi071.stdout:1439: seq_num 538 ranges {642166=628672,1849430=710842} 2024-09-11T01:38:58.443 INFO:tasks.rados.rados.0.smithi071.stdout:1439: writing smithi07134777-27 from 642166 to 1270838 tid 1 2024-09-11T01:38:58.451 INFO:tasks.rados.rados.0.smithi071.stdout:1439: writing smithi07134777-27 from 1849430 to 2560272 tid 2 2024-09-11T01:38:58.451 INFO:tasks.rados.rados.0.smithi071.stdout:1440: snap_create 2024-09-11T01:38:58.451 INFO:tasks.rados.rados.0.smithi071.stdout:1437: finishing write tid 3 to smithi07134777-3 2024-09-11T01:38:58.451 INFO:tasks.rados.rados.0.smithi071.stdout:1438: expect (ObjNum 477 snap 125 seq_num 477) 2024-09-11T01:38:58.451 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:38:58.473 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:38:58.497 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1816926 2024-09-11T01:38:58.507 INFO:tasks.rados.rados.0.smithi071.stdout:1437: finishing write tid 4 to smithi07134777-3 2024-09-11T01:38:58.507 INFO:tasks.rados.rados.0.smithi071.stdout:1437: finishing write tid 5 to smithi07134777-3 2024-09-11T01:38:58.507 INFO:tasks.rados.rados.0.smithi071.stdout:1437: oid 3 updating version 0 to 435 2024-09-11T01:38:58.507 INFO:tasks.rados.rados.0.smithi071.stdout:1437: oid 3 updating version 435 to 437 2024-09-11T01:38:58.507 INFO:tasks.rados.rados.0.smithi071.stdout:1437: oid 3 updating version 437 to 438 2024-09-11T01:38:58.507 INFO:tasks.rados.rados.0.smithi071.stdout:1437: oid 3 version 438 is already newer than 436 2024-09-11T01:38:58.507 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 438 (ObjNum 537 snap 152 seq_num 537) dirty exists 2024-09-11T01:38:58.507 INFO:tasks.rados.rados.0.smithi071.stdout:1437: left oid 3 (ObjNum 537 snap 152 seq_num 537) 2024-09-11T01:38:58.507 INFO:tasks.rados.rados.0.smithi071.stdout:1439: finishing write tid 1 to smithi07134777-27 2024-09-11T01:38:58.507 INFO:tasks.rados.rados.0.smithi071.stdout:1439: finishing write tid 2 to smithi07134777-27 2024-09-11T01:38:58.507 INFO:tasks.rados.rados.0.smithi071.stdout:1439: finishing write tid 3 to smithi07134777-27 2024-09-11T01:38:58.507 INFO:tasks.rados.rados.0.smithi071.stdout:1439: finishing write tid 4 to smithi07134777-27 2024-09-11T01:38:58.508 INFO:tasks.rados.rados.0.smithi071.stdout:1439: oid 27 updating version 0 to 695 2024-09-11T01:38:58.508 INFO:tasks.rados.rados.0.smithi071.stdout:1439: oid 27 version 695 is already newer than 694 2024-09-11T01:38:58.508 INFO:tasks.rados.rados.0.smithi071.stdout:1439: oid 27 updating version 695 to 696 2024-09-11T01:38:58.508 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 696 (ObjNum 538 snap 152 seq_num 538) dirty exists 2024-09-11T01:38:58.508 INFO:tasks.rados.rados.0.smithi071.stdout:1439: left oid 27 (ObjNum 538 snap 152 seq_num 538) 2024-09-11T01:38:58.928 INFO:tasks.rados.rados.0.smithi071.stdout:1437: done (3 left) 2024-09-11T01:38:58.929 INFO:tasks.rados.rados.0.smithi071.stdout:1438: done (2 left) 2024-09-11T01:38:58.929 INFO:tasks.rados.rados.0.smithi071.stdout:1439: done (1 left) 2024-09-11T01:38:58.929 INFO:tasks.rados.rados.0.smithi071.stdout:1440: done (0 left) 2024-09-11T01:38:58.929 INFO:tasks.rados.rados.0.smithi071.stdout:1441: delete oid 9 current snap is 153 2024-09-11T01:38:58.936 INFO:tasks.rados.rados.0.smithi071.stdout:1441: done (0 left) 2024-09-11T01:38:58.936 INFO:tasks.rados.rados.0.smithi071.stdout:1442: copy_from oid 49 from oid 12 current snap is 153 2024-09-11T01:38:58.936 INFO:tasks.rados.rados.0.smithi071.stdout:1443: read oid 33 snap -1 2024-09-11T01:38:58.936 INFO:tasks.rados.rados.0.smithi071.stdout:1443: expect (ObjNum 250 snap 63 seq_num 250) 2024-09-11T01:38:58.936 INFO:tasks.rados.rados.0.smithi071.stdout:1444: write oid 17 current snap is 153 2024-09-11T01:38:58.936 INFO:tasks.rados.rados.0.smithi071.stdout:1444: seq_num 540 ranges {675780=495523,1682747=714511,3012769=63011} 2024-09-11T01:38:58.942 INFO:tasks.rados.rados.0.smithi071.stdout:1444: writing smithi07134777-17 from 675780 to 1171303 tid 1 2024-09-11T01:38:58.949 INFO:tasks.rados.rados.0.smithi071.stdout:1444: writing smithi07134777-17 from 1682747 to 2397258 tid 2 2024-09-11T01:38:58.953 INFO:tasks.rados.rados.0.smithi071.stdout:1444: writing smithi07134777-17 from 3012769 to 3075780 tid 3 2024-09-11T01:38:58.953 INFO:tasks.rados.rados.0.smithi071.stdout:1445: snap_create 2024-09-11T01:38:58.953 INFO:tasks.rados.rados.0.smithi071.stdout:1443: expect (ObjNum 250 snap 63 seq_num 250) 2024-09-11T01:38:59.002 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1406885 2024-09-11T01:38:59.060 INFO:tasks.rados.rados.0.smithi071.stdout:1442: finishing copy_from racing read to smithi07134777-49 2024-09-11T01:38:59.060 INFO:tasks.rados.rados.0.smithi071.stdout:1442: finishing copy_from to smithi07134777-49 2024-09-11T01:38:59.060 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 355 (ObjNum 469 snap 124 seq_num 469) dirty exists 2024-09-11T01:38:59.061 INFO:tasks.rados.rados.0.smithi071.stdout:1444: finishing write tid 1 to smithi07134777-17 2024-09-11T01:38:59.061 INFO:tasks.rados.rados.0.smithi071.stdout:1444: finishing write tid 2 to smithi07134777-17 2024-09-11T01:38:59.061 INFO:tasks.rados.rados.0.smithi071.stdout:1444: finishing write tid 3 to smithi07134777-17 2024-09-11T01:38:59.061 INFO:tasks.rados.rados.0.smithi071.stdout:1444: finishing write tid 4 to smithi07134777-17 2024-09-11T01:38:59.061 INFO:tasks.rados.rados.0.smithi071.stdout:1444: finishing write tid 5 to smithi07134777-17 2024-09-11T01:38:59.061 INFO:tasks.rados.rados.0.smithi071.stdout:1444: oid 17 updating version 0 to 523 2024-09-11T01:38:59.061 INFO:tasks.rados.rados.0.smithi071.stdout:1444: oid 17 version 523 is already newer than 522 2024-09-11T01:38:59.061 INFO:tasks.rados.rados.0.smithi071.stdout:1444: oid 17 version 523 is already newer than 521 2024-09-11T01:38:59.061 INFO:tasks.rados.rados.0.smithi071.stdout:1444: oid 17 version 523 is already newer than 520 2024-09-11T01:38:59.061 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 523 (ObjNum 540 snap 153 seq_num 540) dirty exists 2024-09-11T01:38:59.061 INFO:tasks.rados.rados.0.smithi071.stdout:1444: left oid 17 (ObjNum 540 snap 153 seq_num 540) 2024-09-11T01:38:59.937 INFO:tasks.rados.rados.0.smithi071.stdout:1442: done (3 left) 2024-09-11T01:38:59.937 INFO:tasks.rados.rados.0.smithi071.stdout:1443: done (2 left) 2024-09-11T01:38:59.937 INFO:tasks.rados.rados.0.smithi071.stdout:1444: done (1 left) 2024-09-11T01:38:59.937 INFO:tasks.rados.rados.0.smithi071.stdout:1445: done (0 left) 2024-09-11T01:38:59.937 INFO:tasks.rados.rados.0.smithi071.stdout:1446: snap_remove snap 145 2024-09-11T01:39:00.942 INFO:tasks.rados.rados.0.smithi071.stdout:1446: done (0 left) 2024-09-11T01:39:00.942 INFO:tasks.rados.rados.0.smithi071.stdout:1447: write oid 47 current snap is 154 2024-09-11T01:39:00.942 INFO:tasks.rados.rados.0.smithi071.stdout:1447: seq_num 541 ranges {692587=570772,2024876=574522,3040192=52395} 2024-09-11T01:39:00.949 INFO:tasks.rados.rados.0.smithi071.stdout:1447: writing smithi07134777-47 from 692587 to 1263359 tid 1 2024-09-11T01:39:00.957 INFO:tasks.rados.rados.0.smithi071.stdout:1447: writing smithi07134777-47 from 2024876 to 2599398 tid 2 2024-09-11T01:39:00.960 INFO:tasks.rados.rados.0.smithi071.stdout:1447: writing smithi07134777-47 from 3040192 to 3092587 tid 3 2024-09-11T01:39:00.960 INFO:tasks.rados.rados.0.smithi071.stdout:1448: read oid 9 snap -1 2024-09-11T01:39:00.960 INFO:tasks.rados.rados.0.smithi071.stdout:1448: expect deleted 2024-09-11T01:39:00.960 INFO:tasks.rados.rados.0.smithi071.stdout:1449: write (excl) oid 21 current snap is 154 2024-09-11T01:39:00.960 INFO:tasks.rados.rados.0.smithi071.stdout:1449: seq_num 542 ranges {709394=646021,2083358=718180} 2024-09-11T01:39:00.968 INFO:tasks.rados.rados.0.smithi071.stdout:1449: writing smithi07134777-21 from 709394 to 1355415 tid 1 2024-09-11T01:39:00.977 INFO:tasks.rados.rados.0.smithi071.stdout:1449: writing smithi07134777-21 from 2083358 to 2801538 tid 2 2024-09-11T01:39:00.977 INFO:tasks.rados.rados.0.smithi071.stdout:1450: write oid 14 current snap is 154 2024-09-11T01:39:00.977 INFO:tasks.rados.rados.0.smithi071.stdout:1450: seq_num 543 ranges {726201=721270,2141840=578191} 2024-09-11T01:39:00.984 INFO:tasks.rados.rados.0.smithi071.stdout:1450: writing smithi07134777-14 from 726201 to 1447471 tid 1 2024-09-11T01:39:00.992 INFO:tasks.rados.rados.0.smithi071.stdout:1450: writing smithi07134777-14 from 2141840 to 2720031 tid 2 2024-09-11T01:39:00.992 INFO:tasks.rados.rados.0.smithi071.stdout:1451: read oid 3 snap -1 2024-09-11T01:39:00.992 INFO:tasks.rados.rados.0.smithi071.stdout:1451: expect (ObjNum 537 snap 152 seq_num 537) 2024-09-11T01:39:00.992 INFO:tasks.rados.rados.0.smithi071.stdout:1452: write oid 34 current snap is 154 2024-09-11T01:39:00.993 INFO:tasks.rados.rados.0.smithi071.stdout:1447: finishing write tid 1 to smithi07134777-47 2024-09-11T01:39:00.993 INFO:tasks.rados.rados.0.smithi071.stdout:1447: finishing write tid 2 to smithi07134777-47 2024-09-11T01:39:00.993 INFO:tasks.rados.rados.0.smithi071.stdout:1447: finishing write tid 3 to smithi07134777-47 2024-09-11T01:39:00.993 INFO:tasks.rados.rados.0.smithi071.stdout:1447: finishing write tid 4 to smithi07134777-47 2024-09-11T01:39:00.993 INFO:tasks.rados.rados.0.smithi071.stdout:1452: seq_num 544 ranges {743008=796519,2200322=721849} 2024-09-11T01:39:01.000 INFO:tasks.rados.rados.0.smithi071.stdout:1452: writing smithi07134777-34 from 743008 to 1539527 tid 1 2024-09-11T01:39:01.009 INFO:tasks.rados.rados.0.smithi071.stdout:1452: writing smithi07134777-34 from 2200322 to 2922171 tid 2 2024-09-11T01:39:01.009 INFO:tasks.rados.rados.0.smithi071.stdout:1448: done (5 left) 2024-09-11T01:39:01.009 INFO:tasks.rados.rados.0.smithi071.stdout:1453: copy_from oid 46 from oid 48 current snap is 154 2024-09-11T01:39:01.009 INFO:tasks.rados.rados.0.smithi071.stdout:1447: finishing write tid 5 to smithi07134777-47 2024-09-11T01:39:01.009 INFO:tasks.rados.rados.0.smithi071.stdout:1447: oid 47 updating version 0 to 527 2024-09-11T01:39:01.009 INFO:tasks.rados.rados.0.smithi071.stdout:1447: oid 47 version 527 is already newer than 525 2024-09-11T01:39:01.009 INFO:tasks.rados.rados.0.smithi071.stdout:1447: oid 47 version 527 is already newer than 526 2024-09-11T01:39:01.010 INFO:tasks.rados.rados.0.smithi071.stdout:1447: oid 47 updating version 527 to 528 2024-09-11T01:39:01.010 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 528 (ObjNum 541 snap 154 seq_num 541) dirty exists 2024-09-11T01:39:01.010 INFO:tasks.rados.rados.0.smithi071.stdout:1447: left oid 47 (ObjNum 541 snap 154 seq_num 541) 2024-09-11T01:39:01.010 INFO:tasks.rados.rados.0.smithi071.stdout:1449: finishing write tid 1 to smithi07134777-21 2024-09-11T01:39:01.010 INFO:tasks.rados.rados.0.smithi071.stdout:1447: done (5 left) 2024-09-11T01:39:01.010 INFO:tasks.rados.rados.0.smithi071.stdout:1454: delete oid 23 current snap is 154 2024-09-11T01:39:01.010 INFO:tasks.rados.rados.0.smithi071.stdout:1449: finishing write tid 2 to smithi07134777-21 2024-09-11T01:39:01.010 INFO:tasks.rados.rados.0.smithi071.stdout:1449: finishing write tid 3 to smithi07134777-21 2024-09-11T01:39:01.010 INFO:tasks.rados.rados.0.smithi071.stdout:1449: finishing write tid 4 to smithi07134777-21 2024-09-11T01:39:01.010 INFO:tasks.rados.rados.0.smithi071.stdout:1449: oid 21 updating version 0 to 583 2024-09-11T01:39:01.010 INFO:tasks.rados.rados.0.smithi071.stdout:1449: oid 21 version 583 is already newer than 582 2024-09-11T01:39:01.010 INFO:tasks.rados.rados.0.smithi071.stdout:1449: oid 21 version 583 is already newer than 581 2024-09-11T01:39:01.010 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 583 (ObjNum 542 snap 154 seq_num 542) dirty exists 2024-09-11T01:39:01.010 INFO:tasks.rados.rados.0.smithi071.stdout:1449: left oid 21 (ObjNum 542 snap 154 seq_num 542) 2024-09-11T01:39:01.011 INFO:tasks.rados.rados.0.smithi071.stdout:1450: finishing write tid 1 to smithi07134777-14 2024-09-11T01:39:01.011 INFO:tasks.rados.rados.0.smithi071.stdout:1450: finishing write tid 2 to smithi07134777-14 2024-09-11T01:39:01.011 INFO:tasks.rados.rados.0.smithi071.stdout:1450: finishing write tid 3 to smithi07134777-14 2024-09-11T01:39:01.011 INFO:tasks.rados.rados.0.smithi071.stdout:1450: finishing write tid 4 to smithi07134777-14 2024-09-11T01:39:01.011 INFO:tasks.rados.rados.0.smithi071.stdout:1450: oid 14 updating version 0 to 771 2024-09-11T01:39:01.011 INFO:tasks.rados.rados.0.smithi071.stdout:1450: oid 14 version 771 is already newer than 770 2024-09-11T01:39:01.011 INFO:tasks.rados.rados.0.smithi071.stdout:1450: oid 14 version 771 is already newer than 769 2024-09-11T01:39:01.011 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 771 (ObjNum 543 snap 154 seq_num 543) dirty exists 2024-09-11T01:39:01.011 INFO:tasks.rados.rados.0.smithi071.stdout:1450: left oid 14 (ObjNum 543 snap 154 seq_num 543) 2024-09-11T01:39:01.011 INFO:tasks.rados.rados.0.smithi071.stdout:1451: expect (ObjNum 537 snap 152 seq_num 537) 2024-09-11T01:39:01.011 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:39:01.043 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:39:01.086 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2801406 2024-09-11T01:39:01.094 INFO:tasks.rados.rados.0.smithi071.stdout:1452: finishing write tid 1 to smithi07134777-34 2024-09-11T01:39:01.094 INFO:tasks.rados.rados.0.smithi071.stdout:1452: finishing write tid 2 to smithi07134777-34 2024-09-11T01:39:01.095 INFO:tasks.rados.rados.0.smithi071.stdout:1452: finishing write tid 3 to smithi07134777-34 2024-09-11T01:39:01.095 INFO:tasks.rados.rados.0.smithi071.stdout:1453: finishing copy_from racing read to smithi07134777-46 2024-09-11T01:39:01.095 INFO:tasks.rados.rados.0.smithi071.stdout:1452: finishing write tid 4 to smithi07134777-34 2024-09-11T01:39:01.095 INFO:tasks.rados.rados.0.smithi071.stdout:1452: oid 34 updating version 0 to 708 2024-09-11T01:39:01.095 INFO:tasks.rados.rados.0.smithi071.stdout:1452: oid 34 version 708 is already newer than 707 2024-09-11T01:39:01.095 INFO:tasks.rados.rados.0.smithi071.stdout:1452: oid 34 version 708 is already newer than 706 2024-09-11T01:39:01.095 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 708 (ObjNum 544 snap 154 seq_num 544) dirty exists 2024-09-11T01:39:01.095 INFO:tasks.rados.rados.0.smithi071.stdout:1452: left oid 34 (ObjNum 544 snap 154 seq_num 544) 2024-09-11T01:39:01.095 INFO:tasks.rados.rados.0.smithi071.stdout:1453: finishing copy_from to smithi07134777-46 2024-09-11T01:39:01.095 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 692 (ObjNum 494 snap 132 seq_num 494) dirty exists 2024-09-11T01:39:01.095 INFO:tasks.rados.rados.0.smithi071.stdout:1449: done (5 left) 2024-09-11T01:39:01.095 INFO:tasks.rados.rados.0.smithi071.stdout:1450: done (4 left) 2024-09-11T01:39:01.095 INFO:tasks.rados.rados.0.smithi071.stdout:1451: done (3 left) 2024-09-11T01:39:01.095 INFO:tasks.rados.rados.0.smithi071.stdout:1452: done (2 left) 2024-09-11T01:39:01.096 INFO:tasks.rados.rados.0.smithi071.stdout:1453: done (1 left) 2024-09-11T01:39:01.096 INFO:tasks.rados.rados.0.smithi071.stdout:1454: done (0 left) 2024-09-11T01:39:01.096 INFO:tasks.rados.rados.0.smithi071.stdout:1455: rollback oid 3 current snap is 154 2024-09-11T01:39:01.096 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 3 to 151 2024-09-11T01:39:01.096 INFO:tasks.rados.rados.0.smithi071.stdout:1456: read oid 10 snap -1 2024-09-11T01:39:01.096 INFO:tasks.rados.rados.0.smithi071.stdout:1456: expect (ObjNum 520 snap 143 seq_num 520) 2024-09-11T01:39:01.096 INFO:tasks.rados.rados.0.smithi071.stdout:1457: write (excl) oid 29 current snap is 154 2024-09-11T01:39:01.096 INFO:tasks.rados.rados.0.smithi071.stdout:1457: seq_num 546 ranges {776622=547017,1800933=725518,2944601=232021} 2024-09-11T01:39:01.101 INFO:tasks.rados.rados.0.smithi071.stdout:1457: writing smithi07134777-29 from 776622 to 1323639 tid 1 2024-09-11T01:39:01.108 INFO:tasks.rados.rados.0.smithi071.stdout:1457: writing smithi07134777-29 from 1800933 to 2526451 tid 2 2024-09-11T01:39:01.112 INFO:tasks.rados.rados.0.smithi071.stdout:1457: writing smithi07134777-29 from 2944601 to 3176622 tid 3 2024-09-11T01:39:01.112 INFO:tasks.rados.rados.0.smithi071.stdout:1458: snap_create 2024-09-11T01:39:01.112 INFO:tasks.rados.rados.0.smithi071.stdout:1456: expect (ObjNum 520 snap 143 seq_num 520) 2024-09-11T01:39:01.147 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2253332 2024-09-11T01:39:01.169 INFO:tasks.rados.rados.0.smithi071.stdout:1455: finishing rollback tid 0 to smithi07134777-3 2024-09-11T01:39:01.169 INFO:tasks.rados.rados.0.smithi071.stdout:1455: finishing rollback tid 1 to smithi07134777-3 2024-09-11T01:39:01.169 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 440 (ObjNum 705 snap 0 seq_num 348296349) dirty dne 2024-09-11T01:39:01.169 INFO:tasks.rados.rados.0.smithi071.stdout:1457: finishing write tid 1 to smithi07134777-29 2024-09-11T01:39:01.170 INFO:tasks.rados.rados.0.smithi071.stdout:1457: finishing write tid 2 to smithi07134777-29 2024-09-11T01:39:01.170 INFO:tasks.rados.rados.0.smithi071.stdout:1457: finishing write tid 3 to smithi07134777-29 2024-09-11T01:39:01.170 INFO:tasks.rados.rados.0.smithi071.stdout:1457: finishing write tid 4 to smithi07134777-29 2024-09-11T01:39:01.170 INFO:tasks.rados.rados.0.smithi071.stdout:1457: finishing write tid 5 to smithi07134777-29 2024-09-11T01:39:01.170 INFO:tasks.rados.rados.0.smithi071.stdout:1457: oid 29 updating version 0 to 774 2024-09-11T01:39:01.170 INFO:tasks.rados.rados.0.smithi071.stdout:1457: oid 29 updating version 774 to 776 2024-09-11T01:39:01.170 INFO:tasks.rados.rados.0.smithi071.stdout:1457: oid 29 version 776 is already newer than 775 2024-09-11T01:39:01.170 INFO:tasks.rados.rados.0.smithi071.stdout:1457: oid 29 version 776 is already newer than 773 2024-09-11T01:39:01.170 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 776 (ObjNum 546 snap 154 seq_num 546) dirty exists 2024-09-11T01:39:01.170 INFO:tasks.rados.rados.0.smithi071.stdout:1457: left oid 29 (ObjNum 546 snap 154 seq_num 546) 2024-09-11T01:39:01.949 INFO:tasks.rados.rados.0.smithi071.stdout:1455: done (3 left) 2024-09-11T01:39:01.950 INFO:tasks.rados.rados.0.smithi071.stdout:1456: done (2 left) 2024-09-11T01:39:01.950 INFO:tasks.rados.rados.0.smithi071.stdout:1457: done (1 left) 2024-09-11T01:39:01.950 INFO:tasks.rados.rados.0.smithi071.stdout:1458: done (0 left) 2024-09-11T01:39:01.950 INFO:tasks.rados.rados.0.smithi071.stdout:1459: copy_from oid 32 from oid 9 current snap is 155 2024-09-11T01:39:01.950 INFO:tasks.rados.rados.0.smithi071.stdout:1460: read oid 2 snap 151 2024-09-11T01:39:01.950 INFO:tasks.rados.rados.0.smithi071.stdout:1460: expect (ObjNum 503 snap 135 seq_num 503) 2024-09-11T01:39:01.950 INFO:tasks.rados.rados.0.smithi071.stdout:1461: snap_create 2024-09-11T01:39:01.952 INFO:tasks.rados.rados.0.smithi071.stdout:1459: finishing copy_from to smithi07134777-32 2024-09-11T01:39:01.952 INFO:tasks.rados.rados.0.smithi071.stdout:1459: got expected ENOENT (src dne) 2024-09-11T01:39:01.952 INFO:tasks.rados.rados.0.smithi071.stdout:1459: finishing copy_from racing read to smithi07134777-32 2024-09-11T01:39:01.953 INFO:tasks.rados.rados.0.smithi071.stdout:1460: expect (ObjNum 503 snap 135 seq_num 503) 2024-09-11T01:39:02.949 INFO:tasks.rados.rados.0.smithi071.stdout:1459: done (2 left) 2024-09-11T01:39:02.949 INFO:tasks.rados.rados.0.smithi071.stdout:1460: done (1 left) 2024-09-11T01:39:02.950 INFO:tasks.rados.rados.0.smithi071.stdout:1461: done (0 left) 2024-09-11T01:39:02.950 INFO:tasks.rados.rados.0.smithi071.stdout:1462: write (excl) oid 3 current snap is 156 2024-09-11T01:39:02.950 INFO:tasks.rados.rados.0.smithi071.stdout:1462: seq_num 547 ranges {793429=622266,1859415=469176,2972024=221405} 2024-09-11T01:39:02.957 INFO:tasks.rados.rados.0.smithi071.stdout:1462: writing smithi07134777-3 from 793429 to 1415695 tid 1 2024-09-11T01:39:02.962 INFO:tasks.rados.rados.0.smithi071.stdout:1462: writing smithi07134777-3 from 1859415 to 2328591 tid 2 2024-09-11T01:39:02.967 INFO:tasks.rados.rados.0.smithi071.stdout:1462: writing smithi07134777-3 from 2972024 to 3193429 tid 3 2024-09-11T01:39:02.967 INFO:tasks.rados.rados.0.smithi071.stdout:1463: delete oid 30 current snap is 156 2024-09-11T01:39:02.967 INFO:tasks.rados.rados.0.smithi071.stdout:1462: finishing write tid 1 to smithi07134777-3 2024-09-11T01:39:02.967 INFO:tasks.rados.rados.0.smithi071.stdout:1462: finishing write tid 2 to smithi07134777-3 2024-09-11T01:39:02.969 INFO:tasks.rados.rados.0.smithi071.stdout:1462: finishing write tid 3 to smithi07134777-3 2024-09-11T01:39:02.969 INFO:tasks.rados.rados.0.smithi071.stdout:1463: done (1 left) 2024-09-11T01:39:02.969 INFO:tasks.rados.rados.0.smithi071.stdout:1464: delete oid 23 current snap is 156 2024-09-11T01:39:02.970 INFO:tasks.rados.rados.0.smithi071.stdout:1462: finishing write tid 4 to smithi07134777-3 2024-09-11T01:39:02.970 INFO:tasks.rados.rados.0.smithi071.stdout:1462: finishing write tid 5 to smithi07134777-3 2024-09-11T01:39:02.970 INFO:tasks.rados.rados.0.smithi071.stdout:1462: oid 3 updating version 0 to 443 2024-09-11T01:39:02.970 INFO:tasks.rados.rados.0.smithi071.stdout:1462: oid 3 version 443 is already newer than 442 2024-09-11T01:39:02.970 INFO:tasks.rados.rados.0.smithi071.stdout:1462: oid 3 version 443 is already newer than 441 2024-09-11T01:39:02.970 INFO:tasks.rados.rados.0.smithi071.stdout:1462: oid 3 updating version 443 to 444 2024-09-11T01:39:02.970 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 444 (ObjNum 547 snap 156 seq_num 547) dirty exists 2024-09-11T01:39:02.970 INFO:tasks.rados.rados.0.smithi071.stdout:1462: left oid 3 (ObjNum 547 snap 156 seq_num 547) 2024-09-11T01:39:02.971 INFO:tasks.rados.rados.0.smithi071.stdout:1462: done (1 left) 2024-09-11T01:39:02.972 INFO:tasks.rados.rados.0.smithi071.stdout:1464: done (0 left) 2024-09-11T01:39:02.972 INFO:tasks.rados.rados.0.smithi071.stdout:1465: rollback oid 33 current snap is 156 2024-09-11T01:39:02.972 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 33 to 154 2024-09-11T01:39:02.972 INFO:tasks.rados.rados.0.smithi071.stdout:1466: rollback oid 37 current snap is 156 2024-09-11T01:39:02.972 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 37 to 152 2024-09-11T01:39:02.972 INFO:tasks.rados.rados.0.smithi071.stdout:1467: snap_remove snap 153 2024-09-11T01:39:03.933 INFO:tasks.rados.rados.0.smithi071.stdout:1467: done (2 left) 2024-09-11T01:39:03.933 INFO:tasks.rados.rados.0.smithi071.stdout:1468: snap_create 2024-09-11T01:39:03.934 INFO:tasks.rados.rados.0.smithi071.stdout:1466: finishing rollback tid 0 to smithi07134777-37 2024-09-11T01:39:03.934 INFO:tasks.rados.rados.0.smithi071.stdout:1465: finishing rollback tid 0 to smithi07134777-33 2024-09-11T01:39:03.934 INFO:tasks.rados.rados.0.smithi071.stdout:1466: finishing rollback tid 1 to smithi07134777-37 2024-09-11T01:39:03.934 INFO:tasks.rados.rados.0.smithi071.stdout:1466: finishing rollback tid 2 to smithi07134777-37 2024-09-11T01:39:03.934 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 585 (ObjNum 520 snap 143 seq_num 520) dirty exists 2024-09-11T01:39:03.934 INFO:tasks.rados.rados.0.smithi071.stdout:1465: finishing rollback tid 1 to smithi07134777-33 2024-09-11T01:39:03.934 INFO:tasks.rados.rados.0.smithi071.stdout:1465: finishing rollback tid 2 to smithi07134777-33 2024-09-11T01:39:03.934 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 712 (ObjNum 250 snap 63 seq_num 250) dirty exists 2024-09-11T01:39:04.930 INFO:tasks.rados.rados.0.smithi071.stdout:1465: done (2 left) 2024-09-11T01:39:04.930 INFO:tasks.rados.rados.0.smithi071.stdout:1466: done (1 left) 2024-09-11T01:39:04.931 INFO:tasks.rados.rados.0.smithi071.stdout:1468: done (0 left) 2024-09-11T01:39:04.931 INFO:tasks.rados.rados.0.smithi071.stdout:1469: delete oid 17 current snap is 157 2024-09-11T01:39:04.936 INFO:tasks.rados.rados.0.smithi071.stdout:1469: done (0 left) 2024-09-11T01:39:04.936 INFO:tasks.rados.rados.0.smithi071.stdout:1470: snap_remove snap 154 2024-09-11T01:39:05.934 INFO:tasks.rados.rados.0.smithi071.stdout:1470: done (0 left) 2024-09-11T01:39:05.935 INFO:tasks.rados.rados.0.smithi071.stdout:1471: read oid 3 snap -1 2024-09-11T01:39:05.935 INFO:tasks.rados.rados.0.smithi071.stdout:1471: expect (ObjNum 547 snap 156 seq_num 547) 2024-09-11T01:39:05.935 INFO:tasks.rados.rados.0.smithi071.stdout:1472: write (excl) oid 50 current snap is 157 2024-09-11T01:39:05.935 INFO:tasks.rados.rados.0.smithi071.stdout:1472: seq_num 551 ranges {460657=639615,1693343=476514,2681716=578941} 2024-09-11T01:39:05.941 INFO:tasks.rados.rados.0.smithi071.stdout:1472: writing smithi07134777-50 from 460657 to 1100272 tid 1 2024-09-11T01:39:05.947 INFO:tasks.rados.rados.0.smithi071.stdout:1472: writing smithi07134777-50 from 1693343 to 2169857 tid 2 2024-09-11T01:39:05.954 INFO:tasks.rados.rados.0.smithi071.stdout:1472: writing smithi07134777-50 from 2681716 to 3260657 tid 3 2024-09-11T01:39:05.954 INFO:tasks.rados.rados.0.smithi071.stdout:1473: write (excl) oid 41 current snap is 157 2024-09-11T01:39:05.954 INFO:tasks.rados.rados.0.smithi071.stdout:1473: seq_num 552 ranges {477464=714864,1751825=736525,3225492=51972} 2024-09-11T01:39:05.960 INFO:tasks.rados.rados.0.smithi071.stdout:1473: writing smithi07134777-41 from 477464 to 1192328 tid 1 2024-09-11T01:39:05.967 INFO:tasks.rados.rados.0.smithi071.stdout:1473: writing smithi07134777-41 from 1751825 to 2488350 tid 2 2024-09-11T01:39:05.972 INFO:tasks.rados.rados.0.smithi071.stdout:1473: writing smithi07134777-41 from 3225492 to 3277464 tid 3 2024-09-11T01:39:05.972 INFO:tasks.rados.rados.0.smithi071.stdout:1474: delete oid 43 current snap is 157 2024-09-11T01:39:05.972 INFO:tasks.rados.rados.0.smithi071.stdout:1471: expect (ObjNum 547 snap 156 seq_num 547) 2024-09-11T01:39:06.008 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1118068 2024-09-11T01:39:06.053 INFO:tasks.rados.rados.0.smithi071.stdout:1472: finishing write tid 1 to smithi07134777-50 2024-09-11T01:39:06.053 INFO:tasks.rados.rados.0.smithi071.stdout:1472: finishing write tid 2 to smithi07134777-50 2024-09-11T01:39:06.053 INFO:tasks.rados.rados.0.smithi071.stdout:1472: finishing write tid 3 to smithi07134777-50 2024-09-11T01:39:06.053 INFO:tasks.rados.rados.0.smithi071.stdout:1472: finishing write tid 4 to smithi07134777-50 2024-09-11T01:39:06.053 INFO:tasks.rados.rados.0.smithi071.stdout:1472: finishing write tid 5 to smithi07134777-50 2024-09-11T01:39:06.053 INFO:tasks.rados.rados.0.smithi071.stdout:1472: oid 50 updating version 0 to 783 2024-09-11T01:39:06.054 INFO:tasks.rados.rados.0.smithi071.stdout:1472: oid 50 version 783 is already newer than 781 2024-09-11T01:39:06.054 INFO:tasks.rados.rados.0.smithi071.stdout:1472: oid 50 version 783 is already newer than 782 2024-09-11T01:39:06.054 INFO:tasks.rados.rados.0.smithi071.stdout:1472: oid 50 updating version 783 to 784 2024-09-11T01:39:06.054 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 784 (ObjNum 551 snap 157 seq_num 551) dirty exists 2024-09-11T01:39:06.054 INFO:tasks.rados.rados.0.smithi071.stdout:1472: left oid 50 (ObjNum 551 snap 157 seq_num 551) 2024-09-11T01:39:06.054 INFO:tasks.rados.rados.0.smithi071.stdout:1473: finishing write tid 1 to smithi07134777-41 2024-09-11T01:39:06.054 INFO:tasks.rados.rados.0.smithi071.stdout:1473: finishing write tid 2 to smithi07134777-41 2024-09-11T01:39:06.054 INFO:tasks.rados.rados.0.smithi071.stdout:1471: done (3 left) 2024-09-11T01:39:06.054 INFO:tasks.rados.rados.0.smithi071.stdout:1472: done (2 left) 2024-09-11T01:39:06.054 INFO:tasks.rados.rados.0.smithi071.stdout:1474: done (1 left) 2024-09-11T01:39:06.054 INFO:tasks.rados.rados.0.smithi071.stdout:1475: write (excl) oid 37 current snap is 157 2024-09-11T01:39:06.054 INFO:tasks.rados.rados.0.smithi071.stdout:1473: finishing write tid 3 to smithi07134777-41 2024-09-11T01:39:06.054 INFO:tasks.rados.rados.0.smithi071.stdout:1475: seq_num 554 ranges {511078=465362,1752436=740194,3163985=147093} 2024-09-11T01:39:06.058 INFO:tasks.rados.rados.0.smithi071.stdout:1475: writing smithi07134777-37 from 511078 to 976440 tid 1 2024-09-11T01:39:06.066 INFO:tasks.rados.rados.0.smithi071.stdout:1475: writing smithi07134777-37 from 1752436 to 2492630 tid 2 2024-09-11T01:39:06.071 INFO:tasks.rados.rados.0.smithi071.stdout:1475: writing smithi07134777-37 from 3163985 to 3311078 tid 3 2024-09-11T01:39:06.071 INFO:tasks.rados.rados.0.smithi071.stdout:1476: read oid 17 snap -1 2024-09-11T01:39:06.071 INFO:tasks.rados.rados.0.smithi071.stdout:1476: expect deleted 2024-09-11T01:39:06.071 INFO:tasks.rados.rados.0.smithi071.stdout:1477: delete oid 44 current snap is 157 2024-09-11T01:39:06.071 INFO:tasks.rados.rados.0.smithi071.stdout:1473: finishing write tid 4 to smithi07134777-41 2024-09-11T01:39:06.071 INFO:tasks.rados.rados.0.smithi071.stdout:1473: finishing write tid 5 to smithi07134777-41 2024-09-11T01:39:06.071 INFO:tasks.rados.rados.0.smithi071.stdout:1473: oid 41 updating version 0 to 364 2024-09-11T01:39:06.071 INFO:tasks.rados.rados.0.smithi071.stdout:1473: oid 41 version 364 is already newer than 361 2024-09-11T01:39:06.072 INFO:tasks.rados.rados.0.smithi071.stdout:1473: oid 41 version 364 is already newer than 362 2024-09-11T01:39:06.072 INFO:tasks.rados.rados.0.smithi071.stdout:1473: oid 41 version 364 is already newer than 363 2024-09-11T01:39:06.072 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 364 (ObjNum 552 snap 157 seq_num 552) dirty exists 2024-09-11T01:39:06.072 INFO:tasks.rados.rados.0.smithi071.stdout:1473: left oid 41 (ObjNum 552 snap 157 seq_num 552) 2024-09-11T01:39:06.072 INFO:tasks.rados.rados.0.smithi071.stdout:1475: finishing write tid 1 to smithi07134777-37 2024-09-11T01:39:06.072 INFO:tasks.rados.rados.0.smithi071.stdout:1475: finishing write tid 2 to smithi07134777-37 2024-09-11T01:39:06.072 INFO:tasks.rados.rados.0.smithi071.stdout:1473: done (3 left) 2024-09-11T01:39:06.072 INFO:tasks.rados.rados.0.smithi071.stdout:1476: done (2 left) 2024-09-11T01:39:06.072 INFO:tasks.rados.rados.0.smithi071.stdout:1477: done (1 left) 2024-09-11T01:39:06.073 INFO:tasks.rados.rados.0.smithi071.stdout:1478: delete oid 38 current snap is 157 2024-09-11T01:39:06.073 INFO:tasks.rados.rados.0.smithi071.stdout:1475: finishing write tid 3 to smithi07134777-37 2024-09-11T01:39:06.074 INFO:tasks.rados.rados.0.smithi071.stdout:1478: done (1 left) 2024-09-11T01:39:06.075 INFO:tasks.rados.rados.0.smithi071.stdout:1479: write oid 35 current snap is 157 2024-09-11T01:39:06.075 INFO:tasks.rados.rados.0.smithi071.stdout:1479: seq_num 557 ranges {561499=407462,1527882=487521,2729901=557217} 2024-09-11T01:39:06.080 INFO:tasks.rados.rados.0.smithi071.stdout:1479: writing smithi07134777-35 from 561499 to 968961 tid 1 2024-09-11T01:39:06.085 INFO:tasks.rados.rados.0.smithi071.stdout:1479: writing smithi07134777-35 from 1527882 to 2015403 tid 2 2024-09-11T01:39:06.093 INFO:tasks.rados.rados.0.smithi071.stdout:1479: writing smithi07134777-35 from 2729901 to 3287118 tid 3 2024-09-11T01:39:06.093 INFO:tasks.rados.rados.0.smithi071.stdout:1480: copy_from oid 25 from oid 42 current snap is 157 2024-09-11T01:39:06.093 INFO:tasks.rados.rados.0.smithi071.stdout:1475: finishing write tid 4 to smithi07134777-37 2024-09-11T01:39:06.094 INFO:tasks.rados.rados.0.smithi071.stdout:1475: finishing write tid 5 to smithi07134777-37 2024-09-11T01:39:06.094 INFO:tasks.rados.rados.0.smithi071.stdout:1475: oid 37 updating version 0 to 589 2024-09-11T01:39:06.094 INFO:tasks.rados.rados.0.smithi071.stdout:1475: oid 37 version 589 is already newer than 586 2024-09-11T01:39:06.094 INFO:tasks.rados.rados.0.smithi071.stdout:1475: oid 37 version 589 is already newer than 588 2024-09-11T01:39:06.094 INFO:tasks.rados.rados.0.smithi071.stdout:1475: oid 37 version 589 is already newer than 587 2024-09-11T01:39:06.094 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 589 (ObjNum 554 snap 157 seq_num 554) dirty exists 2024-09-11T01:39:06.094 INFO:tasks.rados.rados.0.smithi071.stdout:1475: left oid 37 (ObjNum 554 snap 157 seq_num 554) 2024-09-11T01:39:06.094 INFO:tasks.rados.rados.0.smithi071.stdout:1479: finishing write tid 1 to smithi07134777-35 2024-09-11T01:39:06.094 INFO:tasks.rados.rados.0.smithi071.stdout:1479: finishing write tid 2 to smithi07134777-35 2024-09-11T01:39:06.094 INFO:tasks.rados.rados.0.smithi071.stdout:1475: done (2 left) 2024-09-11T01:39:06.094 INFO:tasks.rados.rados.0.smithi071.stdout:1481: rollback oid 29 current snap is 157 2024-09-11T01:39:06.094 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 29 to 151 2024-09-11T01:39:06.095 INFO:tasks.rados.rados.0.smithi071.stdout:1482: delete oid 37 current snap is 157 2024-09-11T01:39:06.097 INFO:tasks.rados.rados.0.smithi071.stdout:1479: finishing write tid 3 to smithi07134777-35 2024-09-11T01:39:06.097 INFO:tasks.rados.rados.0.smithi071.stdout:1481: finishing rollback tid 0 to smithi07134777-29 2024-09-11T01:39:06.097 INFO:tasks.rados.rados.0.smithi071.stdout:1481: finishing rollback tid 1 to smithi07134777-29 2024-09-11T01:39:06.097 INFO:tasks.rados.rados.0.smithi071.stdout:1481: finishing rollback tid 2 to smithi07134777-29 2024-09-11T01:39:06.097 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 790 (ObjNum 469 snap 124 seq_num 469) dirty exists 2024-09-11T01:39:06.099 INFO:tasks.rados.rados.0.smithi071.stdout:1479: finishing write tid 4 to smithi07134777-35 2024-09-11T01:39:06.099 INFO:tasks.rados.rados.0.smithi071.stdout:1479: finishing write tid 5 to smithi07134777-35 2024-09-11T01:39:06.099 INFO:tasks.rados.rados.0.smithi071.stdout:1479: oid 35 updating version 0 to 593 2024-09-11T01:39:06.099 INFO:tasks.rados.rados.0.smithi071.stdout:1479: oid 35 version 593 is already newer than 591 2024-09-11T01:39:06.099 INFO:tasks.rados.rados.0.smithi071.stdout:1479: oid 35 version 593 is already newer than 592 2024-09-11T01:39:06.099 INFO:tasks.rados.rados.0.smithi071.stdout:1479: oid 35 version 593 is already newer than 590 2024-09-11T01:39:06.099 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 593 (ObjNum 557 snap 157 seq_num 557) dirty exists 2024-09-11T01:39:06.099 INFO:tasks.rados.rados.0.smithi071.stdout:1479: left oid 35 (ObjNum 557 snap 157 seq_num 557) 2024-09-11T01:39:06.101 INFO:tasks.rados.rados.0.smithi071.stdout:1479: done (3 left) 2024-09-11T01:39:06.101 INFO:tasks.rados.rados.0.smithi071.stdout:1481: done (2 left) 2024-09-11T01:39:06.101 INFO:tasks.rados.rados.0.smithi071.stdout:1482: done (1 left) 2024-09-11T01:39:06.101 INFO:tasks.rados.rados.0.smithi071.stdout:1483: snap_remove snap 155 2024-09-11T01:39:06.939 INFO:tasks.rados.rados.0.smithi071.stdout:1483: done (1 left) 2024-09-11T01:39:06.939 INFO:tasks.rados.rados.0.smithi071.stdout:1484: read oid 49 snap 152 2024-09-11T01:39:06.939 INFO:tasks.rados.rados.0.smithi071.stdout:1484: expect deleted 2024-09-11T01:39:06.939 INFO:tasks.rados.rados.0.smithi071.stdout:1485: read oid 29 snap -1 2024-09-11T01:39:06.940 INFO:tasks.rados.rados.0.smithi071.stdout:1485: expect (ObjNum 469 snap 124 seq_num 469) 2024-09-11T01:39:06.940 INFO:tasks.rados.rados.0.smithi071.stdout:1486: snap_create 2024-09-11T01:39:06.940 INFO:tasks.rados.rados.0.smithi071.stdout:1480: finishing copy_from racing read to smithi07134777-25 2024-09-11T01:39:06.940 INFO:tasks.rados.rados.0.smithi071.stdout:1480: finishing copy_from to smithi07134777-25 2024-09-11T01:39:06.940 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 437 (ObjNum 488 snap 132 seq_num 488) dirty exists 2024-09-11T01:39:06.952 INFO:tasks.rados.rados.0.smithi071.stdout:1485: expect (ObjNum 469 snap 124 seq_num 469) 2024-09-11T01:39:06.952 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:39:07.069 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:39:07.944 INFO:tasks.rados.rados.0.smithi071.stdout:1480: done (3 left) 2024-09-11T01:39:07.944 INFO:tasks.rados.rados.0.smithi071.stdout:1484: done (2 left) 2024-09-11T01:39:07.944 INFO:tasks.rados.rados.0.smithi071.stdout:1485: done (1 left) 2024-09-11T01:39:07.944 INFO:tasks.rados.rados.0.smithi071.stdout:1486: done (0 left) 2024-09-11T01:39:07.944 INFO:tasks.rados.rados.0.smithi071.stdout:1487: delete oid 7 current snap is 158 2024-09-11T01:39:07.948 INFO:tasks.rados.rados.0.smithi071.stdout:1487: done (0 left) 2024-09-11T01:39:07.948 INFO:tasks.rados.rados.0.smithi071.stdout:1488: delete oid 47 current snap is 158 2024-09-11T01:39:07.951 INFO:tasks.rados.rados.0.smithi071.stdout:1488: done (0 left) 2024-09-11T01:39:07.951 INFO:tasks.rados.rados.0.smithi071.stdout:1489: rollback oid 2 current snap is 158 2024-09-11T01:39:07.951 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 2 to 157 2024-09-11T01:39:07.952 INFO:tasks.rados.rados.0.smithi071.stdout:1490: read oid 31 snap 156 2024-09-11T01:39:07.952 INFO:tasks.rados.rados.0.smithi071.stdout:1490: expect (ObjNum 509 snap 139 seq_num 509) 2024-09-11T01:39:07.952 INFO:tasks.rados.rados.0.smithi071.stdout:1491: write (excl) oid 29 current snap is 158 2024-09-11T01:39:07.952 INFO:tasks.rados.rados.0.smithi071.stdout:1491: seq_num 561 ranges {628727=708458,2045457=494859,3123240=305487} 2024-09-11T01:39:07.959 INFO:tasks.rados.rados.0.smithi071.stdout:1491: writing smithi07134777-29 from 628727 to 1337185 tid 1 2024-09-11T01:39:07.968 INFO:tasks.rados.rados.0.smithi071.stdout:1491: writing smithi07134777-29 from 2045457 to 2540316 tid 2 2024-09-11T01:39:07.970 INFO:tasks.rados.rados.0.smithi071.stdout:1491: writing smithi07134777-29 from 3123240 to 3428727 tid 3 2024-09-11T01:39:07.970 INFO:tasks.rados.rados.0.smithi071.stdout:1492: snap_remove snap 151 2024-09-11T01:39:08.930 INFO:tasks.rados.rados.0.smithi071.stdout:1492: done (3 left) 2024-09-11T01:39:08.930 INFO:tasks.rados.rados.0.smithi071.stdout:1493: write (excl) oid 17 current snap is 158 2024-09-11T01:39:08.930 INFO:tasks.rados.rados.0.smithi071.stdout:1493: seq_num 562 ranges {645534=783707,2103939=638517,3150663=294871} 2024-09-11T01:39:08.938 INFO:tasks.rados.rados.0.smithi071.stdout:1493: writing smithi07134777-17 from 645534 to 1429241 tid 1 2024-09-11T01:39:08.946 INFO:tasks.rados.rados.0.smithi071.stdout:1493: writing smithi07134777-17 from 2103939 to 2742456 tid 2 2024-09-11T01:39:08.950 INFO:tasks.rados.rados.0.smithi071.stdout:1493: writing smithi07134777-17 from 3150663 to 3445534 tid 3 2024-09-11T01:39:08.950 INFO:tasks.rados.rados.0.smithi071.stdout:1494: read oid 40 snap -1 2024-09-11T01:39:08.950 INFO:tasks.rados.rados.0.smithi071.stdout:1494: expect (ObjNum 459 snap 122 seq_num 459) 2024-09-11T01:39:08.950 INFO:tasks.rados.rados.0.smithi071.stdout:1495: read oid 3 snap -1 2024-09-11T01:39:08.950 INFO:tasks.rados.rados.0.smithi071.stdout:1495: expect (ObjNum 547 snap 156 seq_num 547) 2024-09-11T01:39:08.950 INFO:tasks.rados.rados.0.smithi071.stdout:1496: read oid 26 snap -1 2024-09-11T01:39:08.950 INFO:tasks.rados.rados.0.smithi071.stdout:1496: expect deleted 2024-09-11T01:39:08.950 INFO:tasks.rados.rados.0.smithi071.stdout:1497: snap_remove snap 152 2024-09-11T01:39:09.957 INFO:tasks.rados.rados.0.smithi071.stdout:1497: done (7 left) 2024-09-11T01:39:09.957 INFO:tasks.rados.rados.0.smithi071.stdout:1498: copy_from oid 42 from oid 13 current snap is 158 2024-09-11T01:39:09.957 INFO:tasks.rados.rados.0.smithi071.stdout:1489: finishing rollback tid 0 to smithi07134777-2 2024-09-11T01:39:09.957 INFO:tasks.rados.rados.0.smithi071.stdout:1489: finishing rollback tid 1 to smithi07134777-2 2024-09-11T01:39:09.957 INFO:tasks.rados.rados.0.smithi071.stdout:1489: finishing rollback tid 2 to smithi07134777-2 2024-09-11T01:39:09.957 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 440 (ObjNum 503 snap 135 seq_num 503) dirty exists 2024-09-11T01:39:09.958 INFO:tasks.rados.rados.0.smithi071.stdout:1490: expect (ObjNum 509 snap 139 seq_num 509) 2024-09-11T01:39:09.958 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:39:10.051 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:39:10.096 INFO:tasks.rados.rados.0.smithi071.stdout:1491: finishing write tid 1 to smithi07134777-29 2024-09-11T01:39:10.096 INFO:tasks.rados.rados.0.smithi071.stdout:1491: finishing write tid 2 to smithi07134777-29 2024-09-11T01:39:10.097 INFO:tasks.rados.rados.0.smithi071.stdout:1491: finishing write tid 3 to smithi07134777-29 2024-09-11T01:39:10.097 INFO:tasks.rados.rados.0.smithi071.stdout:1491: finishing write tid 4 to smithi07134777-29 2024-09-11T01:39:10.097 INFO:tasks.rados.rados.0.smithi071.stdout:1491: finishing write tid 5 to smithi07134777-29 2024-09-11T01:39:10.097 INFO:tasks.rados.rados.0.smithi071.stdout:1491: oid 29 updating version 0 to 799 2024-09-11T01:39:10.097 INFO:tasks.rados.rados.0.smithi071.stdout:1491: oid 29 updating version 799 to 800 2024-09-11T01:39:10.097 INFO:tasks.rados.rados.0.smithi071.stdout:1491: oid 29 updating version 800 to 801 2024-09-11T01:39:10.097 INFO:tasks.rados.rados.0.smithi071.stdout:1491: oid 29 version 801 is already newer than 798 2024-09-11T01:39:10.097 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 801 (ObjNum 561 snap 158 seq_num 561) dirty exists 2024-09-11T01:39:10.097 INFO:tasks.rados.rados.0.smithi071.stdout:1491: left oid 29 (ObjNum 561 snap 158 seq_num 561) 2024-09-11T01:39:10.097 INFO:tasks.rados.rados.0.smithi071.stdout:1494: expect (ObjNum 459 snap 122 seq_num 459) 2024-09-11T01:39:10.169 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2591021 2024-09-11T01:39:10.234 INFO:tasks.rados.rados.0.smithi071.stdout:1495: expect (ObjNum 547 snap 156 seq_num 547) 2024-09-11T01:39:10.238 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 908043 2024-09-11T01:39:10.297 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:39:10.332 INFO:tasks.rados.rados.0.smithi071.stdout:1493: finishing write tid 1 to smithi07134777-17 2024-09-11T01:39:10.332 INFO:tasks.rados.rados.0.smithi071.stdout:1493: finishing write tid 2 to smithi07134777-17 2024-09-11T01:39:10.332 INFO:tasks.rados.rados.0.smithi071.stdout:1493: finishing write tid 3 to smithi07134777-17 2024-09-11T01:39:10.332 INFO:tasks.rados.rados.0.smithi071.stdout:1493: finishing write tid 4 to smithi07134777-17 2024-09-11T01:39:10.332 INFO:tasks.rados.rados.0.smithi071.stdout:1493: finishing write tid 5 to smithi07134777-17 2024-09-11T01:39:10.332 INFO:tasks.rados.rados.0.smithi071.stdout:1493: oid 17 updating version 0 to 543 2024-09-11T01:39:10.332 INFO:tasks.rados.rados.0.smithi071.stdout:1493: oid 17 updating version 543 to 549 2024-09-11T01:39:10.332 INFO:tasks.rados.rados.0.smithi071.stdout:1493: oid 17 version 549 is already newer than 548 2024-09-11T01:39:10.332 INFO:tasks.rados.rados.0.smithi071.stdout:1493: oid 17 updating version 549 to 550 2024-09-11T01:39:10.332 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 550 (ObjNum 562 snap 158 seq_num 562) dirty exists 2024-09-11T01:39:10.333 INFO:tasks.rados.rados.0.smithi071.stdout:1493: left oid 17 (ObjNum 562 snap 158 seq_num 562) 2024-09-11T01:39:10.333 INFO:tasks.rados.rados.0.smithi071.stdout:1498: finishing copy_from to smithi07134777-42 2024-09-11T01:39:10.333 INFO:tasks.rados.rados.0.smithi071.stdout:1498: got expected ENOENT (src dne) 2024-09-11T01:39:10.333 INFO:tasks.rados.rados.0.smithi071.stdout:1498: finishing copy_from racing read to smithi07134777-42 2024-09-11T01:39:10.333 INFO:tasks.rados.rados.0.smithi071.stdout:1489: done (7 left) 2024-09-11T01:39:10.333 INFO:tasks.rados.rados.0.smithi071.stdout:1490: done (6 left) 2024-09-11T01:39:10.333 INFO:tasks.rados.rados.0.smithi071.stdout:1491: done (5 left) 2024-09-11T01:39:10.333 INFO:tasks.rados.rados.0.smithi071.stdout:1493: done (4 left) 2024-09-11T01:39:10.333 INFO:tasks.rados.rados.0.smithi071.stdout:1494: done (3 left) 2024-09-11T01:39:10.333 INFO:tasks.rados.rados.0.smithi071.stdout:1495: done (2 left) 2024-09-11T01:39:10.333 INFO:tasks.rados.rados.0.smithi071.stdout:1496: done (1 left) 2024-09-11T01:39:10.333 INFO:tasks.rados.rados.0.smithi071.stdout:1498: done (0 left) 2024-09-11T01:39:10.333 INFO:tasks.rados.rados.0.smithi071.stdout:1499: write oid 29 current snap is 158 2024-09-11T01:39:10.333 INFO:tasks.rados.rados.0.smithi071.stdout:1499: seq_num 563 ranges {662341=575309,1878774=498528,2894439=457555} 2024-09-11T01:39:10.338 INFO:tasks.rados.rados.0.smithi071.stdout:1499: writing smithi07134777-29 from 662341 to 1237650 tid 1 2024-09-11T01:39:10.344 INFO:tasks.rados.rados.0.smithi071.stdout:1499: writing smithi07134777-29 from 1878774 to 2377302 tid 2 2024-09-11T01:39:10.350 INFO:tasks.rados.rados.0.smithi071.stdout:1499: writing smithi07134777-29 from 2894439 to 3351994 tid 3 2024-09-11T01:39:10.350 INFO:tasks.rados.rados.0.smithi071.stdout:1500: copy_from oid 32 from oid 7 current snap is 158 2024-09-11T01:39:10.350 INFO:tasks.rados.rados.0.smithi071.stdout:1499: finishing write tid 1 to smithi07134777-29 2024-09-11T01:39:10.350 INFO:tasks.rados.rados.0.smithi071.stdout:1499: finishing write tid 2 to smithi07134777-29 2024-09-11T01:39:10.350 INFO:tasks.rados.rados.0.smithi071.stdout:1501: snap_create 2024-09-11T01:39:10.353 INFO:tasks.rados.rados.0.smithi071.stdout:1499: finishing write tid 3 to smithi07134777-29 2024-09-11T01:39:10.353 INFO:tasks.rados.rados.0.smithi071.stdout:1499: finishing write tid 4 to smithi07134777-29 2024-09-11T01:39:10.353 INFO:tasks.rados.rados.0.smithi071.stdout:1499: finishing write tid 5 to smithi07134777-29 2024-09-11T01:39:10.353 INFO:tasks.rados.rados.0.smithi071.stdout:1499: oid 29 updating version 0 to 821 2024-09-11T01:39:10.353 INFO:tasks.rados.rados.0.smithi071.stdout:1499: oid 29 version 821 is already newer than 818 2024-09-11T01:39:10.353 INFO:tasks.rados.rados.0.smithi071.stdout:1499: oid 29 version 821 is already newer than 820 2024-09-11T01:39:10.353 INFO:tasks.rados.rados.0.smithi071.stdout:1499: oid 29 version 821 is already newer than 819 2024-09-11T01:39:10.353 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 821 (ObjNum 563 snap 158 seq_num 563) dirty exists 2024-09-11T01:39:10.354 INFO:tasks.rados.rados.0.smithi071.stdout:1499: left oid 29 (ObjNum 563 snap 158 seq_num 563) 2024-09-11T01:39:10.356 INFO:tasks.rados.rados.0.smithi071.stdout:1500: finishing copy_from to smithi07134777-32 2024-09-11T01:39:10.356 INFO:tasks.rados.rados.0.smithi071.stdout:1500: got expected ENOENT (src dne) 2024-09-11T01:39:10.356 INFO:tasks.rados.rados.0.smithi071.stdout:1500: finishing copy_from racing read to smithi07134777-32 2024-09-11T01:39:10.962 INFO:tasks.rados.rados.0.smithi071.stdout:1499: done (2 left) 2024-09-11T01:39:10.963 INFO:tasks.rados.rados.0.smithi071.stdout:1500: done (1 left) 2024-09-11T01:39:10.963 INFO:tasks.rados.rados.0.smithi071.stdout:1501: done (0 left) 2024-09-11T01:39:10.963 INFO:tasks.rados.rados.0.smithi071.stdout:1502: snap_create 2024-09-11T01:39:11.968 INFO:tasks.rados.rados.0.smithi071.stdout:1502: done (0 left) 2024-09-11T01:39:11.968 INFO:tasks.rados.rados.0.smithi071.stdout:1503: rollback oid 2 current snap is 160 2024-09-11T01:39:11.968 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 2 to 159 2024-09-11T01:39:11.968 INFO:tasks.rados.rados.0.smithi071.stdout:1504: copy_from oid 3 from oid 19 current snap is 160 2024-09-11T01:39:11.968 INFO:tasks.rados.rados.0.smithi071.stdout:1505: write (excl) oid 41 current snap is 160 2024-09-11T01:39:11.968 INFO:tasks.rados.rados.0.smithi071.stdout:1505: seq_num 564 ranges {679148=650558,1937256=642186,3321862=157286} 2024-09-11T01:39:11.975 INFO:tasks.rados.rados.0.smithi071.stdout:1505: writing smithi07134777-41 from 679148 to 1329706 tid 1 2024-09-11T01:39:11.982 INFO:tasks.rados.rados.0.smithi071.stdout:1505: writing smithi07134777-41 from 1937256 to 2579442 tid 2 2024-09-11T01:39:11.988 INFO:tasks.rados.rados.0.smithi071.stdout:1505: writing smithi07134777-41 from 3321862 to 3479148 tid 3 2024-09-11T01:39:11.988 INFO:tasks.rados.rados.0.smithi071.stdout:1506: write (excl) oid 15 current snap is 160 2024-09-11T01:39:11.988 INFO:tasks.rados.rados.0.smithi071.stdout:1506: seq_num 565 ranges {695955=725807,1879385=502197,2949285=480099} 2024-09-11T01:39:11.996 INFO:tasks.rados.rados.0.smithi071.stdout:1506: writing smithi07134777-15 from 695955 to 1421762 tid 1 2024-09-11T01:39:12.001 INFO:tasks.rados.rados.0.smithi071.stdout:1506: writing smithi07134777-15 from 1879385 to 2381582 tid 2 2024-09-11T01:39:12.007 INFO:tasks.rados.rados.0.smithi071.stdout:1506: writing smithi07134777-15 from 2949285 to 3429384 tid 3 2024-09-11T01:39:12.007 INFO:tasks.rados.rados.0.smithi071.stdout:1507: write oid 14 current snap is 160 2024-09-11T01:39:12.008 INFO:tasks.rados.rados.0.smithi071.stdout:1503: finishing rollback tid 0 to smithi07134777-2 2024-09-11T01:39:12.008 INFO:tasks.rados.rados.0.smithi071.stdout:1507: seq_num 566 ranges {712762=401056,1537867=645855,2860355=652407} 2024-09-11T01:39:12.013 INFO:tasks.rados.rados.0.smithi071.stdout:1507: writing smithi07134777-14 from 712762 to 1113818 tid 1 2024-09-11T01:39:12.019 INFO:tasks.rados.rados.0.smithi071.stdout:1507: writing smithi07134777-14 from 1537867 to 2183722 tid 2 2024-09-11T01:39:12.027 INFO:tasks.rados.rados.0.smithi071.stdout:1507: writing smithi07134777-14 from 2860355 to 3512762 tid 3 2024-09-11T01:39:12.027 INFO:tasks.rados.rados.0.smithi071.stdout:1508: write oid 13 current snap is 160 2024-09-11T01:39:12.027 INFO:tasks.rados.rados.0.smithi071.stdout:1503: finishing rollback tid 1 to smithi07134777-2 2024-09-11T01:39:12.028 INFO:tasks.rados.rados.0.smithi071.stdout:1503: finishing rollback tid 2 to smithi07134777-2 2024-09-11T01:39:12.028 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 443 (ObjNum 503 snap 135 seq_num 503) dirty exists 2024-09-11T01:39:12.028 INFO:tasks.rados.rados.0.smithi071.stdout:1505: finishing write tid 1 to smithi07134777-41 2024-09-11T01:39:12.028 INFO:tasks.rados.rados.0.smithi071.stdout:1508: seq_num 567 ranges {729569=476305,1996349=505866,3004131=525438} 2024-09-11T01:39:12.033 INFO:tasks.rados.rados.0.smithi071.stdout:1508: writing smithi07134777-13 from 729569 to 1205874 tid 1 2024-09-11T01:39:12.040 INFO:tasks.rados.rados.0.smithi071.stdout:1508: writing smithi07134777-13 from 1996349 to 2502215 tid 2 2024-09-11T01:39:12.047 INFO:tasks.rados.rados.0.smithi071.stdout:1508: writing smithi07134777-13 from 3004131 to 3529569 tid 3 2024-09-11T01:39:12.047 INFO:tasks.rados.rados.0.smithi071.stdout:1503: done (5 left) 2024-09-11T01:39:12.047 INFO:tasks.rados.rados.0.smithi071.stdout:1509: read oid 20 snap -1 2024-09-11T01:39:12.047 INFO:tasks.rados.rados.0.smithi071.stdout:1509: expect (ObjNum 513 snap 141 seq_num 513) 2024-09-11T01:39:12.047 INFO:tasks.rados.rados.0.smithi071.stdout:1510: rollback oid 6 current snap is 160 2024-09-11T01:39:12.047 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 6 to 150 2024-09-11T01:39:12.047 INFO:tasks.rados.rados.0.smithi071.stdout:1505: finishing write tid 2 to smithi07134777-41 2024-09-11T01:39:12.047 INFO:tasks.rados.rados.0.smithi071.stdout:1505: finishing write tid 3 to smithi07134777-41 2024-09-11T01:39:12.048 INFO:tasks.rados.rados.0.smithi071.stdout:1504: finishing copy_from racing read to smithi07134777-3 2024-09-11T01:39:12.048 INFO:tasks.rados.rados.0.smithi071.stdout:1505: finishing write tid 4 to smithi07134777-41 2024-09-11T01:39:12.048 INFO:tasks.rados.rados.0.smithi071.stdout:1505: finishing write tid 5 to smithi07134777-41 2024-09-11T01:39:12.048 INFO:tasks.rados.rados.0.smithi071.stdout:1505: oid 41 updating version 0 to 366 2024-09-11T01:39:12.048 INFO:tasks.rados.rados.0.smithi071.stdout:1505: oid 41 version 366 is already newer than 365 2024-09-11T01:39:12.048 INFO:tasks.rados.rados.0.smithi071.stdout:1505: oid 41 updating version 366 to 368 2024-09-11T01:39:12.048 INFO:tasks.rados.rados.0.smithi071.stdout:1505: oid 41 version 368 is already newer than 367 2024-09-11T01:39:12.048 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 368 (ObjNum 564 snap 160 seq_num 564) dirty exists 2024-09-11T01:39:12.048 INFO:tasks.rados.rados.0.smithi071.stdout:1505: left oid 41 (ObjNum 564 snap 160 seq_num 564) 2024-09-11T01:39:12.048 INFO:tasks.rados.rados.0.smithi071.stdout:1504: finishing copy_from to smithi07134777-3 2024-09-11T01:39:12.048 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 451 (ObjNum 515 snap 141 seq_num 515) dirty exists 2024-09-11T01:39:12.048 INFO:tasks.rados.rados.0.smithi071.stdout:1506: finishing write tid 1 to smithi07134777-15 2024-09-11T01:39:12.048 INFO:tasks.rados.rados.0.smithi071.stdout:1506: finishing write tid 2 to smithi07134777-15 2024-09-11T01:39:12.049 INFO:tasks.rados.rados.0.smithi071.stdout:1504: done (6 left) 2024-09-11T01:39:12.049 INFO:tasks.rados.rados.0.smithi071.stdout:1505: done (5 left) 2024-09-11T01:39:12.049 INFO:tasks.rados.rados.0.smithi071.stdout:1511: write (excl) oid 1 current snap is 160 2024-09-11T01:39:12.049 INFO:tasks.rados.rados.0.smithi071.stdout:1506: finishing write tid 3 to smithi07134777-15 2024-09-11T01:39:12.049 INFO:tasks.rados.rados.0.smithi071.stdout:1506: finishing write tid 4 to smithi07134777-15 2024-09-11T01:39:12.049 INFO:tasks.rados.rados.0.smithi071.stdout:1511: seq_num 568 ranges {746376=551554,2054831=649524,3315201=231175} 2024-09-11T01:39:12.054 INFO:tasks.rados.rados.0.smithi071.stdout:1511: writing smithi07134777-1 from 746376 to 1297930 tid 1 2024-09-11T01:39:12.061 INFO:tasks.rados.rados.0.smithi071.stdout:1511: writing smithi07134777-1 from 2054831 to 2704355 tid 2 2024-09-11T01:39:12.067 INFO:tasks.rados.rados.0.smithi071.stdout:1511: writing smithi07134777-1 from 3315201 to 3546376 tid 3 2024-09-11T01:39:12.067 INFO:tasks.rados.rados.0.smithi071.stdout:1512: write oid 23 current snap is 160 2024-09-11T01:39:12.067 INFO:tasks.rados.rados.0.smithi071.stdout:1506: finishing write tid 5 to smithi07134777-15 2024-09-11T01:39:12.067 INFO:tasks.rados.rados.0.smithi071.stdout:1506: oid 15 updating version 0 to 823 2024-09-11T01:39:12.067 INFO:tasks.rados.rados.0.smithi071.stdout:1506: oid 15 updating version 823 to 825 2024-09-11T01:39:12.067 INFO:tasks.rados.rados.0.smithi071.stdout:1506: oid 15 updating version 825 to 826 2024-09-11T01:39:12.067 INFO:tasks.rados.rados.0.smithi071.stdout:1506: oid 15 version 826 is already newer than 824 2024-09-11T01:39:12.067 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 826 (ObjNum 565 snap 160 seq_num 565) dirty exists 2024-09-11T01:39:12.067 INFO:tasks.rados.rados.0.smithi071.stdout:1506: left oid 15 (ObjNum 565 snap 160 seq_num 565) 2024-09-11T01:39:12.067 INFO:tasks.rados.rados.0.smithi071.stdout:1507: finishing write tid 1 to smithi07134777-14 2024-09-11T01:39:12.067 INFO:tasks.rados.rados.0.smithi071.stdout:1507: finishing write tid 2 to smithi07134777-14 2024-09-11T01:39:12.067 INFO:tasks.rados.rados.0.smithi071.stdout:1507: finishing write tid 3 to smithi07134777-14 2024-09-11T01:39:12.068 INFO:tasks.rados.rados.0.smithi071.stdout:1512: seq_num 569 ranges {763183=626803,1996960=793182,3226271=336912} 2024-09-11T01:39:12.074 INFO:tasks.rados.rados.0.smithi071.stdout:1512: writing smithi07134777-23 from 763183 to 1389986 tid 1 2024-09-11T01:39:12.084 INFO:tasks.rados.rados.0.smithi071.stdout:1512: writing smithi07134777-23 from 1996960 to 2790142 tid 2 2024-09-11T01:39:12.087 INFO:tasks.rados.rados.0.smithi071.stdout:1512: writing smithi07134777-23 from 3226271 to 3563183 tid 3 2024-09-11T01:39:12.087 INFO:tasks.rados.rados.0.smithi071.stdout:1506: done (6 left) 2024-09-11T01:39:12.087 INFO:tasks.rados.rados.0.smithi071.stdout:1513: copy_from oid 12 from oid 47 current snap is 160 2024-09-11T01:39:12.087 INFO:tasks.rados.rados.0.smithi071.stdout:1507: finishing write tid 4 to smithi07134777-14 2024-09-11T01:39:12.087 INFO:tasks.rados.rados.0.smithi071.stdout:1507: finishing write tid 5 to smithi07134777-14 2024-09-11T01:39:12.087 INFO:tasks.rados.rados.0.smithi071.stdout:1507: oid 14 updating version 0 to 831 2024-09-11T01:39:12.087 INFO:tasks.rados.rados.0.smithi071.stdout:1507: oid 14 version 831 is already newer than 830 2024-09-11T01:39:12.087 INFO:tasks.rados.rados.0.smithi071.stdout:1507: oid 14 version 831 is already newer than 829 2024-09-11T01:39:12.087 INFO:tasks.rados.rados.0.smithi071.stdout:1507: oid 14 version 831 is already newer than 828 2024-09-11T01:39:12.087 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 831 (ObjNum 566 snap 160 seq_num 566) dirty exists 2024-09-11T01:39:12.087 INFO:tasks.rados.rados.0.smithi071.stdout:1507: left oid 14 (ObjNum 566 snap 160 seq_num 566) 2024-09-11T01:39:12.088 INFO:tasks.rados.rados.0.smithi071.stdout:1508: finishing write tid 1 to smithi07134777-13 2024-09-11T01:39:12.088 INFO:tasks.rados.rados.0.smithi071.stdout:1508: finishing write tid 2 to smithi07134777-13 2024-09-11T01:39:12.088 INFO:tasks.rados.rados.0.smithi071.stdout:1507: done (6 left) 2024-09-11T01:39:12.088 INFO:tasks.rados.rados.0.smithi071.stdout:1514: read oid 28 snap -1 2024-09-11T01:39:12.088 INFO:tasks.rados.rados.0.smithi071.stdout:1514: expect deleted 2024-09-11T01:39:12.088 INFO:tasks.rados.rados.0.smithi071.stdout:1515: snap_create 2024-09-11T01:39:12.088 INFO:tasks.rados.rados.0.smithi071.stdout:1510: finishing rollback tid 1 to smithi07134777-6 2024-09-11T01:39:12.088 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-11T01:39:12.088 INFO:tasks.rados.rados.0.smithi071.stdout:1509: expect (ObjNum 513 snap 141 seq_num 513) 2024-09-11T01:39:12.088 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:39:12.146 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1568546 2024-09-11T01:39:12.217 INFO:tasks.rados.rados.0.smithi071.stdout:1508: finishing write tid 3 to smithi07134777-13 2024-09-11T01:39:12.217 INFO:tasks.rados.rados.0.smithi071.stdout:1508: finishing write tid 4 to smithi07134777-13 2024-09-11T01:39:12.217 INFO:tasks.rados.rados.0.smithi071.stdout:1508: finishing write tid 5 to smithi07134777-13 2024-09-11T01:39:12.217 INFO:tasks.rados.rados.0.smithi071.stdout:1508: oid 13 updating version 0 to 557 2024-09-11T01:39:12.217 INFO:tasks.rados.rados.0.smithi071.stdout:1508: oid 13 version 557 is already newer than 555 2024-09-11T01:39:12.217 INFO:tasks.rados.rados.0.smithi071.stdout:1508: oid 13 updating version 557 to 558 2024-09-11T01:39:12.218 INFO:tasks.rados.rados.0.smithi071.stdout:1508: oid 13 version 558 is already newer than 556 2024-09-11T01:39:12.218 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 558 (ObjNum 567 snap 160 seq_num 567) dirty exists 2024-09-11T01:39:12.218 INFO:tasks.rados.rados.0.smithi071.stdout:1508: left oid 13 (ObjNum 567 snap 160 seq_num 567) 2024-09-11T01:39:12.218 INFO:tasks.rados.rados.0.smithi071.stdout:1511: finishing write tid 1 to smithi07134777-1 2024-09-11T01:39:12.218 INFO:tasks.rados.rados.0.smithi071.stdout:1511: finishing write tid 2 to smithi07134777-1 2024-09-11T01:39:12.218 INFO:tasks.rados.rados.0.smithi071.stdout:1511: finishing write tid 3 to smithi07134777-1 2024-09-11T01:39:12.218 INFO:tasks.rados.rados.0.smithi071.stdout:1511: finishing write tid 4 to smithi07134777-1 2024-09-11T01:39:12.218 INFO:tasks.rados.rados.0.smithi071.stdout:1511: finishing write tid 5 to smithi07134777-1 2024-09-11T01:39:12.218 INFO:tasks.rados.rados.0.smithi071.stdout:1511: oid 1 updating version 0 to 560 2024-09-11T01:39:12.218 INFO:tasks.rados.rados.0.smithi071.stdout:1511: oid 1 version 560 is already newer than 559 2024-09-11T01:39:12.218 INFO:tasks.rados.rados.0.smithi071.stdout:1511: oid 1 updating version 560 to 561 2024-09-11T01:39:12.218 INFO:tasks.rados.rados.0.smithi071.stdout:1511: oid 1 updating version 561 to 562 2024-09-11T01:39:12.218 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 562 (ObjNum 568 snap 160 seq_num 568) dirty exists 2024-09-11T01:39:12.218 INFO:tasks.rados.rados.0.smithi071.stdout:1511: left oid 1 (ObjNum 568 snap 160 seq_num 568) 2024-09-11T01:39:12.219 INFO:tasks.rados.rados.0.smithi071.stdout:1512: finishing write tid 1 to smithi07134777-23 2024-09-11T01:39:12.219 INFO:tasks.rados.rados.0.smithi071.stdout:1513: finishing copy_from to smithi07134777-12 2024-09-11T01:39:12.219 INFO:tasks.rados.rados.0.smithi071.stdout:1513: got expected ENOENT (src dne) 2024-09-11T01:39:12.219 INFO:tasks.rados.rados.0.smithi071.stdout:1513: finishing copy_from racing read to smithi07134777-12 2024-09-11T01:39:12.219 INFO:tasks.rados.rados.0.smithi071.stdout:1512: finishing write tid 2 to smithi07134777-23 2024-09-11T01:39:12.219 INFO:tasks.rados.rados.0.smithi071.stdout:1512: finishing write tid 3 to smithi07134777-23 2024-09-11T01:39:12.219 INFO:tasks.rados.rados.0.smithi071.stdout:1512: finishing write tid 4 to smithi07134777-23 2024-09-11T01:39:12.219 INFO:tasks.rados.rados.0.smithi071.stdout:1512: finishing write tid 5 to smithi07134777-23 2024-09-11T01:39:12.219 INFO:tasks.rados.rados.0.smithi071.stdout:1512: oid 23 updating version 0 to 282 2024-09-11T01:39:12.219 INFO:tasks.rados.rados.0.smithi071.stdout:1512: oid 23 updating version 282 to 284 2024-09-11T01:39:12.219 INFO:tasks.rados.rados.0.smithi071.stdout:1512: oid 23 version 284 is already newer than 283 2024-09-11T01:39:12.219 INFO:tasks.rados.rados.0.smithi071.stdout:1512: oid 23 updating version 284 to 285 2024-09-11T01:39:12.219 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 285 (ObjNum 569 snap 160 seq_num 569) dirty exists 2024-09-11T01:39:12.219 INFO:tasks.rados.rados.0.smithi071.stdout:1512: left oid 23 (ObjNum 569 snap 160 seq_num 569) 2024-09-11T01:39:12.972 INFO:tasks.rados.rados.0.smithi071.stdout:1508: done (7 left) 2024-09-11T01:39:12.972 INFO:tasks.rados.rados.0.smithi071.stdout:1509: done (6 left) 2024-09-11T01:39:12.972 INFO:tasks.rados.rados.0.smithi071.stdout:1510: done (5 left) 2024-09-11T01:39:12.972 INFO:tasks.rados.rados.0.smithi071.stdout:1511: done (4 left) 2024-09-11T01:39:12.972 INFO:tasks.rados.rados.0.smithi071.stdout:1512: done (3 left) 2024-09-11T01:39:12.972 INFO:tasks.rados.rados.0.smithi071.stdout:1513: done (2 left) 2024-09-11T01:39:12.972 INFO:tasks.rados.rados.0.smithi071.stdout:1514: done (1 left) 2024-09-11T01:39:12.972 INFO:tasks.rados.rados.0.smithi071.stdout:1515: done (0 left) 2024-09-11T01:39:12.973 INFO:tasks.rados.rados.0.smithi071.stdout:1516: copy_from oid 17 from oid 28 current snap is 161 2024-09-11T01:39:12.973 INFO:tasks.rados.rados.0.smithi071.stdout:1517: write oid 29 current snap is 161 2024-09-11T01:39:12.973 INFO:tasks.rados.rados.0.smithi071.stdout:1517: seq_num 570 ranges {779990=702052,2055442=653193,3253694=326296} 2024-09-11T01:39:12.979 INFO:tasks.rados.rados.0.smithi071.stdout:1517: writing smithi07134777-29 from 779990 to 1482042 tid 1 2024-09-11T01:39:12.987 INFO:tasks.rados.rados.0.smithi071.stdout:1517: writing smithi07134777-29 from 2055442 to 2708635 tid 2 2024-09-11T01:39:12.992 INFO:tasks.rados.rados.0.smithi071.stdout:1517: writing smithi07134777-29 from 3253694 to 3579990 tid 3 2024-09-11T01:39:12.992 INFO:tasks.rados.rados.0.smithi071.stdout:1518: snap_create 2024-09-11T01:39:12.992 INFO:tasks.rados.rados.0.smithi071.stdout:1516: finishing copy_from to smithi07134777-17 2024-09-11T01:39:12.992 INFO:tasks.rados.rados.0.smithi071.stdout:1516: got expected ENOENT (src dne) 2024-09-11T01:39:12.992 INFO:tasks.rados.rados.0.smithi071.stdout:1516: finishing copy_from racing read to smithi07134777-17 2024-09-11T01:39:12.992 INFO:tasks.rados.rados.0.smithi071.stdout:1517: finishing write tid 1 to smithi07134777-29 2024-09-11T01:39:12.994 INFO:tasks.rados.rados.0.smithi071.stdout:1517: finishing write tid 2 to smithi07134777-29 2024-09-11T01:39:12.995 INFO:tasks.rados.rados.0.smithi071.stdout:1517: finishing write tid 3 to smithi07134777-29 2024-09-11T01:39:12.995 INFO:tasks.rados.rados.0.smithi071.stdout:1517: finishing write tid 4 to smithi07134777-29 2024-09-11T01:39:12.995 INFO:tasks.rados.rados.0.smithi071.stdout:1517: finishing write tid 5 to smithi07134777-29 2024-09-11T01:39:12.995 INFO:tasks.rados.rados.0.smithi071.stdout:1517: oid 29 updating version 0 to 835 2024-09-11T01:39:12.995 INFO:tasks.rados.rados.0.smithi071.stdout:1517: oid 29 version 835 is already newer than 834 2024-09-11T01:39:12.995 INFO:tasks.rados.rados.0.smithi071.stdout:1517: oid 29 updating version 835 to 836 2024-09-11T01:39:12.996 INFO:tasks.rados.rados.0.smithi071.stdout:1517: oid 29 version 836 is already newer than 833 2024-09-11T01:39:12.996 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 836 (ObjNum 570 snap 161 seq_num 570) dirty exists 2024-09-11T01:39:12.996 INFO:tasks.rados.rados.0.smithi071.stdout:1517: left oid 29 (ObjNum 570 snap 161 seq_num 570) 2024-09-11T01:39:14.111 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-11T01:39:14.111 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-11T01:39:14.111 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-11T01:39:14.114 INFO:tasks.ceph.mon.c:Started 2024-09-11T01:39:14.114 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-11T01:39:14.115 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-11T01:39:14.115 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T01:39:14.116 INFO:tasks.rados.rados.0.smithi071.stdout:1516: done (2 left) 2024-09-11T01:39:14.116 INFO:tasks.rados.rados.0.smithi071.stdout:1517: done (1 left) 2024-09-11T01:39:14.116 INFO:tasks.rados.rados.0.smithi071.stdout:1518: done (0 left) 2024-09-11T01:39:14.116 INFO:tasks.rados.rados.0.smithi071.stdout:1519: read oid 22 snap -1 2024-09-11T01:39:14.116 INFO:tasks.rados.rados.0.smithi071.stdout:1519: expect (ObjNum 503 snap 135 seq_num 503) 2024-09-11T01:39:14.116 INFO:tasks.rados.rados.0.smithi071.stdout:1520: snap_remove snap 159 2024-09-11T01:39:29.116 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-11T01:39:29.117 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-11T01:39:29.117 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:39:29.117 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:39:29.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.228+0000 7fb119c16640 10 monclient: get_monmap_and_config 2024-09-11T01:39:29.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.228+0000 7fb119c16640 10 monclient: build_initial_monmap 2024-09-11T01:39:29.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.228+0000 7fb119c16640 10 monclient: monmap: 2024-09-11T01:39:29.230 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:29.230 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:29.230 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:39:29.229633+0000 2024-09-11T01:39:29.230 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:39:29.229633+0000 2024-09-11T01:39:29.230 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:29.231 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:29.231 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:39:29.231 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:39:29.231 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:39:29.231 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:39:29.231 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:39:29.231 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:29.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.229+0000 7fb119c16640 1 Processor -- start 2024-09-11T01:39:29.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.229+0000 7fb119c16640 1 -- start start 2024-09-11T01:39:29.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.229+0000 7fb119c16640 10 monclient: init 2024-09-11T01:39:29.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.230+0000 7fb119c16640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:29.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.230+0000 7fb119c16640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-11T01:39:29.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.230+0000 7fb119c16640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb1140a03a0 addr v1:172.21.15.12:6789/0 2024-09-11T01:39:29.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.230+0000 7fb119c16640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb1140a13a0 0x7fb114139a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:29.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.230+0000 7fb119c16640 10 monclient(hunting): picked mon.noname-a con 0x7fb1140a13a0 addr v2:172.21.15.12:3300/0 2024-09-11T01:39:29.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.230+0000 7fb119c16640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb1140a17d0 addr v1:172.21.15.71:6789/0 2024-09-11T01:39:29.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.230+0000 7fb119c16640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb11409fa00 con 0x7fb1140a03a0 2024-09-11T01:39:29.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.230+0000 7fb119c16640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1140a29b0 con 0x7fb1140a17d0 2024-09-11T01:39:29.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.230+0000 7fb119c16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:29.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.230+0000 7fb119c16640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fb1140a0780 con 0x7fb1140a13a0 2024-09-11T01:39:29.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.230+0000 7fb119c16640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:29.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.230+0000 7fb119c16640 10 monclient(hunting): authenticate will time out at 1375.672340s 2024-09-11T01:39:29.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.230+0000 7fb112ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb1140a13a0 0x7fb114139a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:29.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.230+0000 7fb1137fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb1140a03a0 0x7fb1140a0c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37842/0 (socket says 172.21.15.12:37842) 2024-09-11T01:39:29.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.230+0000 7fb1137fe640 1 -- 172.21.15.12:0/2542814178 learned_addr learned my addr 172.21.15.12:0/2542814178 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:39:29.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.230+0000 7fb112ffd640 10 monclient(hunting): get_auth_request con 0x7fb1140a13a0 auth_method 0 2024-09-11T01:39:29.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.230+0000 7fb112ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:29.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.230+0000 7fb112ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:29.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.230+0000 7fb112ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:29.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.230+0000 7fb112ffd640 1 -- 172.21.15.12:0/2542814178 >> v2:172.21.15.12:3300/0 conn(0x7fb1140a13a0 msgr2=0x7fb114139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:39:29.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.230+0000 7fb112ffd640 1 -- 172.21.15.12:0/2542814178 >> v2:172.21.15.12:3300/0 conn(0x7fb1140a13a0 msgr2=0x7fb114139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:39:29.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.230+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.12:3300/0 conn(0x7fb1140a13a0 0x7fb114139a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:39:29.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.230+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.12:3300/0 conn(0x7fb1140a13a0 0x7fb114139a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:39:29.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.232+0000 7fb1137fe640 1 -- 172.21.15.12:0/2542814178 >> v1:172.21.15.12:6789/0 conn(0x7fb1140a03a0 legacy=0x7fb1140a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:29.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.232+0000 7fb113fff640 1 -- 172.21.15.12:0/2542814178 >> v1:172.21.15.71:6789/0 conn(0x7fb1140a17d0 legacy=0x7fb11413bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:29.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.431+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.12:3300/0 conn(0x7fb1140a13a0 0x7fb114139a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:29.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.431+0000 7fb112ffd640 10 monclient(hunting): get_auth_request con 0x7fb1140a13a0 auth_method 0 2024-09-11T01:39:29.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.431+0000 7fb112ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:29.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.431+0000 7fb112ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:29.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.431+0000 7fb112ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:39:29.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.431+0000 7fb112ffd640 1 -- 172.21.15.12:0/2542814178 >> v2:172.21.15.12:3300/0 conn(0x7fb1140a13a0 msgr2=0x7fb114139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:39:29.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.431+0000 7fb112ffd640 1 -- 172.21.15.12:0/2542814178 >> v2:172.21.15.12:3300/0 conn(0x7fb1140a13a0 msgr2=0x7fb114139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:39:29.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.431+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.12:3300/0 conn(0x7fb1140a13a0 0x7fb114139a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:39:29.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.431+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.12:3300/0 conn(0x7fb1140a13a0 0x7fb114139a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:39:29.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.833+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.12:3300/0 conn(0x7fb1140a13a0 0x7fb114139a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:29.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.833+0000 7fb112ffd640 10 monclient(hunting): get_auth_request con 0x7fb1140a13a0 auth_method 0 2024-09-11T01:39:29.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.833+0000 7fb112ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:29.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.833+0000 7fb112ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:29.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.833+0000 7fb112ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:39:29.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.833+0000 7fb112ffd640 1 -- 172.21.15.12:0/2542814178 >> v2:172.21.15.12:3300/0 conn(0x7fb1140a13a0 msgr2=0x7fb114139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:39:29.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.833+0000 7fb112ffd640 1 -- 172.21.15.12:0/2542814178 >> v2:172.21.15.12:3300/0 conn(0x7fb1140a13a0 msgr2=0x7fb114139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:39:29.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.833+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.12:3300/0 conn(0x7fb1140a13a0 0x7fb114139a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:39:29.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:29.833+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.12:3300/0 conn(0x7fb1140a13a0 0x7fb114139a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:39:30.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:30.634+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.12:3300/0 conn(0x7fb1140a13a0 0x7fb114139a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:30.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:30.634+0000 7fb112ffd640 10 monclient(hunting): get_auth_request con 0x7fb1140a13a0 auth_method 0 2024-09-11T01:39:30.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:30.634+0000 7fb112ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:30.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:30.634+0000 7fb112ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:30.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:30.634+0000 7fb112ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:39:30.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:30.634+0000 7fb112ffd640 1 -- 172.21.15.12:0/2542814178 >> v2:172.21.15.12:3300/0 conn(0x7fb1140a13a0 msgr2=0x7fb114139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:39:30.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:30.634+0000 7fb112ffd640 1 -- 172.21.15.12:0/2542814178 >> v2:172.21.15.12:3300/0 conn(0x7fb1140a13a0 msgr2=0x7fb114139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:39:30.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:30.634+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.12:3300/0 conn(0x7fb1140a13a0 0x7fb114139a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:39:30.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:30.634+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.12:3300/0 conn(0x7fb1140a13a0 0x7fb114139a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:39:32.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.230+0000 7fb1117fa640 10 monclient(hunting): tick 2024-09-11T01:39:32.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.230+0000 7fb1117fa640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:39:32.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.230+0000 7fb1117fa640 1 monclient(hunting): continuing hunt 2024-09-11T01:39:32.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.230+0000 7fb1117fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:39:32.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.230+0000 7fb1117fa640 1 -- 172.21.15.12:0/2542814178 >> v2:172.21.15.12:3300/0 conn(0x7fb1140a13a0 msgr2=0x7fb114139a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:39:32.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.230+0000 7fb1117fa640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.12:3300/0 conn(0x7fb1140a13a0 0x7fb114139a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:32.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.230+0000 7fb1117fa640 1 -- 172.21.15.12:0/2542814178 >> v1:172.21.15.71:6789/0 conn(0x7fb1140a17d0 legacy=0x7fb11413bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:32.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.230+0000 7fb1117fa640 1 -- 172.21.15.12:0/2542814178 >> v1:172.21.15.12:6789/0 conn(0x7fb1140a03a0 legacy=0x7fb1140a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:32.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.230+0000 7fb1117fa640 10 monclient: _add_conns ranks=[2,4] 2024-09-11T01:39:32.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.230+0000 7fb1117fa640 1 --2- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 0x7fb100005cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:32.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.230+0000 7fb1117fa640 10 monclient(hunting): picked mon.noname-b con 0x7fb100003860 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:32.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.230+0000 7fb1117fa640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 0x7fb100008860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:32.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.230+0000 7fb1117fa640 10 monclient(hunting): picked mon.noname-c con 0x7fb1000063e0 addr v2:172.21.15.71:3300/0 2024-09-11T01:39:32.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.230+0000 7fb1117fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:32.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.230+0000 7fb1117fa640 1 -- 172.21.15.12:0/2542814178 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fb1140a0780 con 0x7fb1000063e0 2024-09-11T01:39:32.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.230+0000 7fb1117fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:32.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.230+0000 7fb1117fa640 1 -- 172.21.15.12:0/2542814178 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb100008f20 con 0x7fb100003860 2024-09-11T01:39:32.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.230+0000 7fb1117fa640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:32.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.230+0000 7fb1137fe640 1 --2- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 0x7fb100005cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:32.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.230+0000 7fb1137fe640 10 monclient(hunting): get_auth_request con 0x7fb100003860 auth_method 0 2024-09-11T01:39:32.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.230+0000 7fb1137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:32.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.230+0000 7fb1137fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:32.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.230+0000 7fb1137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.231+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 0x7fb100008860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.231+0000 7fb1137fe640 1 -- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 msgr2=0x7fb100005cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:39:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.231+0000 7fb1137fe640 1 -- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 msgr2=0x7fb100005cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:39:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.231+0000 7fb1137fe640 1 --2- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 0x7fb100005cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:39:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.231+0000 7fb112ffd640 10 monclient(hunting): get_auth_request con 0x7fb1000063e0 auth_method 0 2024-09-11T01:39:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.231+0000 7fb112ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.231+0000 7fb112ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.231+0000 7fb112ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.231+0000 7fb1137fe640 1 --2- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 0x7fb100005cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:39:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.231+0000 7fb112ffd640 1 -- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 msgr2=0x7fb100008860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:39:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.231+0000 7fb112ffd640 1 -- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 msgr2=0x7fb100008860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:39:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.231+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 0x7fb100008860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:39:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.231+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 0x7fb100008860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:39:32.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.432+0000 7fb1137fe640 1 --2- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 0x7fb100005cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:32.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.432+0000 7fb1137fe640 10 monclient(hunting): get_auth_request con 0x7fb100003860 auth_method 0 2024-09-11T01:39:32.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.432+0000 7fb1137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:32.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.432+0000 7fb1137fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:32.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.432+0000 7fb1137fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:39:32.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.432+0000 7fb1137fe640 1 -- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 msgr2=0x7fb100005cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:39:32.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.432+0000 7fb1137fe640 1 -- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 msgr2=0x7fb100005cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:39:32.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.432+0000 7fb1137fe640 1 --2- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 0x7fb100005cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:39:32.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.432+0000 7fb1137fe640 1 --2- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 0x7fb100005cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:39:32.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.432+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 0x7fb100008860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:32.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.432+0000 7fb112ffd640 10 monclient(hunting): get_auth_request con 0x7fb1000063e0 auth_method 0 2024-09-11T01:39:32.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.432+0000 7fb112ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:32.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.432+0000 7fb112ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:32.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.432+0000 7fb112ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:39:32.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.433+0000 7fb112ffd640 1 -- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 msgr2=0x7fb100008860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:39:32.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.433+0000 7fb112ffd640 1 -- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 msgr2=0x7fb100008860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:39:32.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.433+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 0x7fb100008860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:39:32.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.433+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 0x7fb100008860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:39:32.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.833+0000 7fb1137fe640 1 --2- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 0x7fb100005cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:32.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.833+0000 7fb1137fe640 10 monclient(hunting): get_auth_request con 0x7fb100003860 auth_method 0 2024-09-11T01:39:32.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.833+0000 7fb1137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:32.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.833+0000 7fb1137fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:32.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.833+0000 7fb1137fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:39:32.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.833+0000 7fb1137fe640 1 -- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 msgr2=0x7fb100005cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:39:32.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.833+0000 7fb1137fe640 1 -- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 msgr2=0x7fb100005cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:39:32.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.833+0000 7fb1137fe640 1 --2- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 0x7fb100005cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:39:32.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.833+0000 7fb1137fe640 1 --2- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 0x7fb100005cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:39:32.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.834+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 0x7fb100008860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:32.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.834+0000 7fb112ffd640 10 monclient(hunting): get_auth_request con 0x7fb1000063e0 auth_method 0 2024-09-11T01:39:32.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.834+0000 7fb112ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:32.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.834+0000 7fb112ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:32.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.834+0000 7fb112ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:39:32.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.835+0000 7fb112ffd640 1 -- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 msgr2=0x7fb100008860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:39:32.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.835+0000 7fb112ffd640 1 -- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 msgr2=0x7fb100008860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:39:32.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.835+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 0x7fb100008860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:39:32.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:32.835+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 0x7fb100008860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:39:33.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:33.635+0000 7fb1137fe640 1 --2- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 0x7fb100005cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:33.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:33.635+0000 7fb1137fe640 10 monclient(hunting): get_auth_request con 0x7fb100003860 auth_method 0 2024-09-11T01:39:33.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:33.635+0000 7fb1137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:33.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:33.635+0000 7fb1137fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:33.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:33.635+0000 7fb1137fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:39:33.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:33.635+0000 7fb1137fe640 1 -- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 msgr2=0x7fb100005cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:39:33.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:33.635+0000 7fb1137fe640 1 -- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 msgr2=0x7fb100005cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:39:33.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:33.635+0000 7fb1137fe640 1 --2- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 0x7fb100005cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:39:33.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:33.635+0000 7fb1137fe640 1 --2- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 0x7fb100005cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:39:33.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:33.636+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 0x7fb100008860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:33.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:33.636+0000 7fb112ffd640 10 monclient(hunting): get_auth_request con 0x7fb1000063e0 auth_method 0 2024-09-11T01:39:33.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:33.636+0000 7fb112ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:33.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:33.636+0000 7fb112ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:33.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:33.636+0000 7fb112ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:39:33.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:33.636+0000 7fb112ffd640 1 -- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 msgr2=0x7fb100008860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:39:33.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:33.636+0000 7fb112ffd640 1 -- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 msgr2=0x7fb100008860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:39:33.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:33.636+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 0x7fb100008860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:39:33.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:33.636+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 0x7fb100008860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:39:33.934 INFO:tasks.rados.rados.0.smithi071.stdout:1520: done (1 left) 2024-09-11T01:39:33.934 INFO:tasks.rados.rados.0.smithi071.stdout:1521: snap_create 2024-09-11T01:39:33.934 INFO:tasks.rados.rados.0.smithi071.stdout:1519: expect (ObjNum 503 snap 135 seq_num 503) 2024-09-11T01:39:33.934 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:39:35.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1117fa640 10 monclient(hunting): tick 2024-09-11T01:39:35.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1117fa640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:39:35.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1117fa640 1 monclient(hunting): continuing hunt 2024-09-11T01:39:35.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1117fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:39:35.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1117fa640 1 -- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 msgr2=0x7fb100005cb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:39:35.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1117fa640 1 --2- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 0x7fb100005cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1117fa640 1 -- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 msgr2=0x7fb100008860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:39:35.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1117fa640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 0x7fb100008860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1117fa640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-11T01:39:35.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb113fff640 1 -- 172.21.15.12:0/2542814178 reap_dead start 2024-09-11T01:39:35.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1117fa640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.12:3300/0 conn(0x7fb1140a13a0 0x7fb10000b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1117fa640 10 monclient(hunting): picked mon.noname-a con 0x7fb1140a13a0 addr v2:172.21.15.12:3300/0 2024-09-11T01:39:35.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1117fa640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 0x7fb100008860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1117fa640 10 monclient(hunting): picked mon.noname-c con 0x7fb1000063e0 addr v2:172.21.15.71:3300/0 2024-09-11T01:39:35.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1117fa640 1 --2- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 0x7fb100005cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1117fa640 10 monclient(hunting): picked mon.noname-b con 0x7fb100003860 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:35.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1117fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:35.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1117fa640 1 -- 172.21.15.12:0/2542814178 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fb1140a0780 con 0x7fb1140a13a0 2024-09-11T01:39:35.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1117fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:35.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1117fa640 1 -- 172.21.15.12:0/2542814178 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fb100008f20 con 0x7fb1000063e0 2024-09-11T01:39:35.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1117fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:35.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1117fa640 1 -- 172.21.15.12:0/2542814178 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb10000b8f0 con 0x7fb100003860 2024-09-11T01:39:35.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1117fa640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:35.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1137fe640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.12:3300/0 conn(0x7fb1140a13a0 0x7fb10000b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1137fe640 10 monclient(hunting): get_auth_request con 0x7fb1140a13a0 auth_method 0 2024-09-11T01:39:35.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:35.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1137fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:35.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb113fff640 1 --2- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 0x7fb100005cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:35.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb113fff640 10 monclient(hunting): get_auth_request con 0x7fb100003860 auth_method 0 2024-09-11T01:39:35.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb113fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:35.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb113fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:35.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb113fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:35.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:35.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:35.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:35.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb113fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:35.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb113fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:35.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb113fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:35.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 0x7fb100008860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb112ffd640 10 monclient(hunting): get_auth_request con 0x7fb1000063e0 auth_method 0 2024-09-11T01:39:35.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb112ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb112ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb112ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1137fe640 10 monclient(hunting): handle_auth_done global_id 94272 payload 274 2024-09-11T01:39:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1137fe640 1 -- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 msgr2=0x7fb100005cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:39:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1137fe640 1 --2- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 0x7fb100005cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1137fe640 1 -- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 msgr2=0x7fb100008860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:39:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1137fe640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 0x7fb100008860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1137fe640 10 monclient: _finish_hunting 0 2024-09-11T01:39:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1137fe640 1 monclient: found mon.noname-a 2024-09-11T01:39:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1137fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:39:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb1137fe640 1 -- 172.21.15.12:0/2542814178 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb11409e510 con 0x7fb1140a13a0 2024-09-11T01:39:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.231+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 0x7fb100008860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:39:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.234+0000 7fb1137fe640 1 -- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1140a13a0 msgr2=0x7fb10000b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.234+0000 7fb1137fe640 1 --2- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1140a13a0 0x7fb10000b390 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb1140a3cd0 tx=0x7fb10402adf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a8081e18b9da52c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.234+0000 7fb1127fc640 1 -- 172.21.15.12:0/2542814178 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb104031070 con 0x7fb1140a13a0 2024-09-11T01:39:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.234+0000 7fb1127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.234+0000 7fb1127fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:39:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.234+0000 7fb1127fc640 10 monclient: dump: 2024-09-11T01:39:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:35.242 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.234+0000 7fb1127fc640 10 monclient: _finish_auth 0 2024-09-11T01:39:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.234+0000 7fb1127fc640 10 monclient: _check_auth_tickets 2024-09-11T01:39:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.234+0000 7fb1127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.234+0000 7fb1127fc640 1 -- 172.21.15.12:0/2542814178 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb10402b8f0 con 0x7fb1140a13a0 2024-09-11T01:39:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.234+0000 7fb1127fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.234+0000 7fb1127fc640 1 -- 172.21.15.12:0/2542814178 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10402bce0 con 0x7fb1140a13a0 2024-09-11T01:39:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.234+0000 7fb119c16640 5 monclient: authenticate success, global_id 94272 2024-09-11T01:39:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.234+0000 7fb119c16640 10 monclient: get_monmap_and_config success 2024-09-11T01:39:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.234+0000 7fb119c16640 4 set_mon_vals no callback set 2024-09-11T01:39:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.234+0000 7fb1127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.234+0000 7fb1127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:39:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.234+0000 7fb1127fc640 10 monclient: dump: 2024-09-11T01:39:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:35.243 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.235+0000 7fb119c16640 10 monclient: shutdown 2024-09-11T01:39:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.235+0000 7fb119c16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.235+0000 7fb119c16640 1 -- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1140a13a0 msgr2=0x7fb10000b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.235+0000 7fb119c16640 1 --2- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1140a13a0 0x7fb10000b390 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb1140a3cd0 tx=0x7fb10402adf0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.235+0000 7fb119c16640 1 -- 172.21.15.12:0/2542814178 shutdown_connections 2024-09-11T01:39:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.235+0000 7fb119c16640 1 --2- 172.21.15.12:0/2542814178 >> v2:172.21.15.71:3300/0 conn(0x7fb1000063e0 0x7fb100008860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.235+0000 7fb119c16640 1 --2- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 0x7fb100005cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.235+0000 7fb119c16640 1 --2- 172.21.15.12:0/2542814178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1140a13a0 0x7fb10000b390 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.235+0000 7fb119c16640 1 -- 172.21.15.12:0/2542814178 >> 172.21.15.12:0/2542814178 conn(0x7fb1140a2d90 msgr2=0x7fb1140a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.235+0000 7fb119c16640 1 -- 172.21.15.12:0/2542814178 shutdown_connections 2024-09-11T01:39:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.235+0000 7fb119c16640 1 -- 172.21.15.12:0/2542814178 wait complete. 2024-09-11T01:39:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.236+0000 7fb119c16640 10 monclient: build_initial_monmap 2024-09-11T01:39:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.236+0000 7fb119c16640 10 monclient: monmap: 2024-09-11T01:39:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:39:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:39:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:39:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:39:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:39:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:39:35.245 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.237+0000 7fb119c16640 1 Processor -- start 2024-09-11T01:39:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.237+0000 7fb119c16640 1 -- start start 2024-09-11T01:39:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.237+0000 7fb119c16640 10 monclient: init 2024-09-11T01:39:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.237+0000 7fb119c16640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.237+0000 7fb119c16640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:39:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb119c16640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 0x7fb114133a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb119c16640 10 monclient(hunting): picked mon.noname-c con 0x7fb100003860 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb119c16640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1000063e0 0x7fb114133fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb119c16640 10 monclient(hunting): picked mon.noname-b con 0x7fb1000063e0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:39:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb119c16640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1140a13a0 0x7fb1141217d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb119c16640 10 monclient(hunting): picked mon.noname-a con 0x7fb1140a13a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:39:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb119c16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb119c16640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb11409f1d0 con 0x7fb1140a13a0 2024-09-11T01:39:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb119c16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb119c16640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb1140a26f0 con 0x7fb1000063e0 2024-09-11T01:39:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb119c16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb119c16640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb11409f350 con 0x7fb100003860 2024-09-11T01:39:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb119c16640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb119c16640 10 monclient(hunting): authenticate will time out at 1381.680079s 2024-09-11T01:39:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb1137fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 0x7fb114133a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb113fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1140a13a0 0x7fb1141217d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb1137fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 0x7fb114133a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:44564/0 (socket says 172.21.15.12:44564) 2024-09-11T01:39:35.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb1137fe640 1 -- 172.21.15.12:0/2062768257 learned_addr learned my addr 172.21.15.12:0/2062768257 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:39:35.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb113fff640 10 monclient(hunting): get_auth_request con 0x7fb1140a13a0 auth_method 0 2024-09-11T01:39:35.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb113fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:35.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb113fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:35.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb113fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:35.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb1137fe640 10 monclient(hunting): get_auth_request con 0x7fb100003860 auth_method 0 2024-09-11T01:39:35.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb1137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:35.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb1137fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:35.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb1137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:35.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb113fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:35.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb113fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:35.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb113fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb1137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb1137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb1137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2062768257 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1000063e0 0x7fb114133fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb112ffd640 10 monclient(hunting): get_auth_request con 0x7fb1000063e0 auth_method 0 2024-09-11T01:39:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb112ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb112ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb112ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb113fff640 10 monclient(hunting): handle_auth_done global_id 94275 payload 824 2024-09-11T01:39:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb113fff640 1 -- 172.21.15.12:0/2062768257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 msgr2=0x7fb114133a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb113fff640 1 --2- 172.21.15.12:0/2062768257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 0x7fb114133a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb113fff640 1 -- 172.21.15.12:0/2062768257 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1000063e0 msgr2=0x7fb114133fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb113fff640 1 --2- 172.21.15.12:0/2062768257 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1000063e0 0x7fb114133fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb113fff640 10 monclient: _finish_hunting 0 2024-09-11T01:39:35.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb113fff640 1 monclient: found mon.noname-a 2024-09-11T01:39:35.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb113fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:35.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb113fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:39:35.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb113fff640 1 -- 172.21.15.12:0/2062768257 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb11409d480 con 0x7fb1140a13a0 2024-09-11T01:39:35.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.238+0000 7fb112ffd640 1 --2- 172.21.15.12:0/2062768257 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1000063e0 0x7fb114133fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:39:35.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.240+0000 7fb113fff640 1 -- 172.21.15.12:0/2062768257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1140a13a0 msgr2=0x7fb1141217d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:35.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.240+0000 7fb113fff640 1 --2- 172.21.15.12:0/2062768257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1140a13a0 0x7fb1141217d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb1140a23d0 tx=0x7fb10c00fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:35.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.241+0000 7fb0fbfff640 1 -- 172.21.15.12:0/2062768257 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10c00d700 con 0x7fb1140a13a0 2024-09-11T01:39:35.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.241+0000 7fb0fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:35.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.241+0000 7fb0fbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:39:35.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.241+0000 7fb0fbfff640 10 monclient: dump: 2024-09-11T01:39:35.251 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:35.251 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:35.252 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.252 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.252 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:35.252 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.252 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:35.252 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:35.252 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:35.252 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.241+0000 7fb0fbfff640 10 monclient: _finish_auth 0 2024-09-11T01:39:35.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.241+0000 7fb0fbfff640 10 monclient: _check_auth_tickets 2024-09-11T01:39:35.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.241+0000 7fb0fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:35.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.241+0000 7fb0fbfff640 1 -- 172.21.15.12:0/2062768257 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb10c018070 con 0x7fb1140a13a0 2024-09-11T01:39:35.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.241+0000 7fb0fbfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:35.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.241+0000 7fb0fbfff640 1 -- 172.21.15.12:0/2062768257 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10c00dae0 con 0x7fb1140a13a0 2024-09-11T01:39:35.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.241+0000 7fb119c16640 5 monclient: authenticate success, global_id 94275 2024-09-11T01:39:35.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.241+0000 7fb119c16640 10 monclient: _renew_subs 2024-09-11T01:39:35.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.241+0000 7fb119c16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:35.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.241+0000 7fb119c16640 1 -- 172.21.15.12:0/2062768257 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1140a5d00 con 0x7fb1140a13a0 2024-09-11T01:39:35.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.241+0000 7fb0fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:35.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.241+0000 7fb0fbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:39:35.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.241+0000 7fb0fbfff640 10 monclient: dump: 2024-09-11T01:39:35.253 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:35.253 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:35.253 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.253 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.253 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:35.253 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.254 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:35.254 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:35.254 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:35.254 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.241+0000 7fb111ffb640 4 set_mon_vals no callback set 2024-09-11T01:39:35.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.241+0000 7fb119c16640 10 monclient: _renew_subs 2024-09-11T01:39:35.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.241+0000 7fb119c16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:35.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.241+0000 7fb119c16640 1 -- 172.21.15.12:0/2062768257 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1141ccf50 con 0x7fb1140a13a0 2024-09-11T01:39:35.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.242+0000 7fb0fbfff640 1 -- 172.21.15.12:0/2062768257 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fb10c00dd00 con 0x7fb1140a13a0 2024-09-11T01:39:35.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.242+0000 7fb119c16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:35.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.242+0000 7fb119c16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:35.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.242+0000 7fb119c16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:35.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.242+0000 7fb119c16640 1 -- 172.21.15.12:0/2062768257 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb114074d10 con 0x7fb1140a13a0 2024-09-11T01:39:35.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.242+0000 7fb0fbfff640 1 --2- 172.21.15.12:0/2062768257 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb10003f3e0 0x7fb1000418a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.242+0000 7fb0fbfff640 1 -- 172.21.15.12:0/2062768257 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(344..344 src has 186..344) ==== 4811+0+0 (secure 0 0 0) 0x7fb10c052c20 con 0x7fb1140a13a0 2024-09-11T01:39:35.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.243+0000 7fb1137fe640 1 --2- 172.21.15.12:0/2062768257 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb10003f3e0 0x7fb1000418a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.243+0000 7fb1137fe640 10 monclient: get_auth_request con 0x7fb10003f3e0 auth_method 0 2024-09-11T01:39:35.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.245+0000 7fb1137fe640 1 --2- 172.21.15.12:0/2062768257 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb10003f3e0 0x7fb1000418a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb1040010e0 tx=0x7fb104010040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:35.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.247+0000 7fb0fbfff640 1 -- 172.21.15.12:0/2062768257 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb10c051600 con 0x7fb1140a13a0 2024-09-11T01:39:35.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.247+0000 7fb0fbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:35.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.247+0000 7fb0fbfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:39:35.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.505+0000 7fb119c16640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:39:35.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.505+0000 7fb119c16640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:39:35.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.505+0000 7fb119c16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:35.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.505+0000 7fb119c16640 1 -- 172.21.15.12:0/2062768257 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb11406d460 con 0x7fb1140a13a0 2024-09-11T01:39:35.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.506+0000 7fb0fbfff640 1 -- 172.21.15.12:0/2062768257 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fb10c01d460 con 0x7fb1140a13a0 2024-09-11T01:39:35.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.506+0000 7fb0fbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:39:35.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.506+0000 7fb0fbfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:39:35.508 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":100,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:39:35.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.508+0000 7fb119c16640 1 -- 172.21.15.12:0/2062768257 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb10003f3e0 msgr2=0x7fb1000418a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:35.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.508+0000 7fb119c16640 1 --2- 172.21.15.12:0/2062768257 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb10003f3e0 0x7fb1000418a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb1040010e0 tx=0x7fb104010040 comp rx=0 tx=0).stop 2024-09-11T01:39:35.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.508+0000 7fb119c16640 10 monclient: shutdown 2024-09-11T01:39:35.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.508+0000 7fb119c16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:35.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.508+0000 7fb119c16640 1 -- 172.21.15.12:0/2062768257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1140a13a0 msgr2=0x7fb1141217d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:35.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.508+0000 7fb119c16640 1 --2- 172.21.15.12:0/2062768257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1140a13a0 0x7fb1141217d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb1140a23d0 tx=0x7fb10c00fd40 comp rx=0 tx=0).stop 2024-09-11T01:39:35.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.508+0000 7fb119c16640 1 -- 172.21.15.12:0/2062768257 shutdown_connections 2024-09-11T01:39:35.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.508+0000 7fb119c16640 1 --2- 172.21.15.12:0/2062768257 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb10003f3e0 0x7fb1000418a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.508+0000 7fb119c16640 1 --2- 172.21.15.12:0/2062768257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb1140a13a0 0x7fb1141217d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.508+0000 7fb119c16640 1 --2- 172.21.15.12:0/2062768257 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1000063e0 0x7fb114133fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.508+0000 7fb119c16640 1 --2- 172.21.15.12:0/2062768257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb100003860 0x7fb114133a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.508+0000 7fb119c16640 1 -- 172.21.15.12:0/2062768257 >> 172.21.15.12:0/2062768257 conn(0x7fb1140a2d90 msgr2=0x7fb114074200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:35.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.508+0000 7fb119c16640 1 -- 172.21.15.12:0/2062768257 shutdown_connections 2024-09-11T01:39:35.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.508+0000 7fb119c16640 1 -- 172.21.15.12:0/2062768257 wait complete. 2024-09-11T01:39:35.520 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:39:35.520 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:39:35.520 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:39:35.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.620+0000 7f03b5ba2640 10 monclient: get_monmap_and_config 2024-09-11T01:39:35.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.620+0000 7f03b5ba2640 10 monclient: build_initial_monmap 2024-09-11T01:39:35.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.620+0000 7f03b5ba2640 10 monclient: monmap: 2024-09-11T01:39:35.622 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:35.622 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:35.622 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:39:35.621484+0000 2024-09-11T01:39:35.622 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:39:35.621484+0000 2024-09-11T01:39:35.622 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:35.622 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.623 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:39:35.623 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:39:35.623 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:39:35.623 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:39:35.623 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:39:35.623 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.621+0000 7f03b5ba2640 1 Processor -- start 2024-09-11T01:39:35.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.621+0000 7f03b5ba2640 1 -- start start 2024-09-11T01:39:35.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.621+0000 7f03b5ba2640 10 monclient: init 2024-09-11T01:39:35.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.621+0000 7f03b5ba2640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.621+0000 7f03b5ba2640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-11T01:39:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03b5ba2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03b0073f50 0x7f03b0074330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03b5ba2640 10 monclient(hunting): picked mon.noname-b con 0x7f03b0073f50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03b5ba2640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f03b0074870 0x7f03b00a25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03b5ba2640 10 monclient(hunting): picked mon.noname-c con 0x7f03b0074870 addr v2:172.21.15.71:3300/0 2024-09-11T01:39:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03b5ba2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f03b00a0810 addr v1:172.21.15.71:6789/0 2024-09-11T01:39:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03b5ba2640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03b0070160 con 0x7f03b00a0810 2024-09-11T01:39:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03b5ba2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03b5ba2640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f03b00a0440 con 0x7f03b0074870 2024-09-11T01:39:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03b5ba2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03b5ba2640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f03b0070440 con 0x7f03b0073f50 2024-09-11T01:39:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03b5ba2640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03b5ba2640 10 monclient(hunting): authenticate will time out at 1382.064086s 2024-09-11T01:39:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03af7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03b0073f50 0x7f03b0074330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03af7fe640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03b0073f50 msgr2=0x7f03b0074330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:39:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03af7fe640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03b0073f50 msgr2=0x7f03b0074330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:39:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03af7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03b0073f50 0x7f03b0074330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:39:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03af7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03b0073f50 0x7f03b0074330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:39:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03aeffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f03b0074870 0x7f03b00a25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03affff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f03b00a0810 0x7f03b00a2b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.12:43242/0 (socket says 172.21.15.12:43242) 2024-09-11T01:39:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03affff640 1 -- 172.21.15.12:0/1500422333 learned_addr learned my addr 172.21.15.12:0/1500422333 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:39:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03aeffd640 10 monclient(hunting): get_auth_request con 0x7f03b0074870 auth_method 0 2024-09-11T01:39:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:35.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:35.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03aeffd640 10 monclient(hunting): handle_auth_done global_id 104164 payload 274 2024-09-11T01:39:35.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03aeffd640 1 -- 172.21.15.12:0/1500422333 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03b0073f50 msgr2=0x7f03b0074330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:39:35.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03aeffd640 1 --2- 172.21.15.12:0/1500422333 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03b0073f50 0x7f03b0074330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03aeffd640 1 -- 172.21.15.12:0/1500422333 >> v1:172.21.15.71:6789/0 conn(0x7f03b00a0810 legacy=0x7f03b00a2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:39:35.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03aeffd640 10 monclient: _finish_hunting 0 2024-09-11T01:39:35.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03aeffd640 1 monclient: found mon.noname-c 2024-09-11T01:39:35.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:35.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03aeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:39:35.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.622+0000 7f03aeffd640 1 -- 172.21.15.12:0/1500422333 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03b0070720 con 0x7f03b0074870 2024-09-11T01:39:35.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.625+0000 7f03aeffd640 1 -- 172.21.15.12:0/1500422333 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03b0074870 msgr2=0x7f03b00a25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:35.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.625+0000 7f03aeffd640 1 --2- 172.21.15.12:0/1500422333 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03b0074870 0x7f03b00a25d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f03a40097a0 tx=0x7f03a402a0e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc3810d718d6f9a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:35.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.626+0000 7f03ae7fc640 1 -- 172.21.15.12:0/1500422333 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a4037070 con 0x7f03b0074870 2024-09-11T01:39:35.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.626+0000 7f03ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:35.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.626+0000 7f03ae7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:39:35.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.626+0000 7f03ae7fc640 10 monclient: dump: 2024-09-11T01:39:35.633 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:35.633 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:35.633 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.633 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.633 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:35.633 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.633 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:35.633 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:35.633 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:35.633 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.626+0000 7f03ae7fc640 10 monclient: _finish_auth 0 2024-09-11T01:39:35.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.626+0000 7f03ae7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:39:35.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.626+0000 7f03ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:35.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.626+0000 7f03ae7fc640 1 -- 172.21.15.12:0/1500422333 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03a4032640 con 0x7f03b0074870 2024-09-11T01:39:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.626+0000 7f03b5ba2640 5 monclient: authenticate success, global_id 104164 2024-09-11T01:39:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.626+0000 7f03b5ba2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:39:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.626+0000 7f03ae7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.626+0000 7f03ae7fc640 1 -- 172.21.15.12:0/1500422333 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a4032a10 con 0x7f03b0074870 2024-09-11T01:39:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.626+0000 7f03b5ba2640 10 monclient: get_monmap_and_config success 2024-09-11T01:39:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.626+0000 7f03b5ba2640 4 set_mon_vals no callback set 2024-09-11T01:39:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.626+0000 7f03ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.626+0000 7f03ae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:39:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.626+0000 7f03ae7fc640 10 monclient: dump: 2024-09-11T01:39:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.635 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:35.635 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:35.635 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:35.635 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.626+0000 7f03b5ba2640 10 monclient: shutdown 2024-09-11T01:39:35.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.626+0000 7f03b5ba2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:35.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.626+0000 7f03b5ba2640 1 -- 172.21.15.12:0/1500422333 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03b0074870 msgr2=0x7f03b00a25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:35.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.626+0000 7f03b5ba2640 1 --2- 172.21.15.12:0/1500422333 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03b0074870 0x7f03b00a25d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f03a40097a0 tx=0x7f03a402a0e0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.626+0000 7f03b5ba2640 1 -- 172.21.15.12:0/1500422333 shutdown_connections 2024-09-11T01:39:35.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.626+0000 7f03b5ba2640 1 --2- 172.21.15.12:0/1500422333 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03b0074870 0x7f03b00a25d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.626+0000 7f03b5ba2640 1 --2- 172.21.15.12:0/1500422333 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03b0073f50 0x7f03b0074330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.626+0000 7f03b5ba2640 1 -- 172.21.15.12:0/1500422333 >> 172.21.15.12:0/1500422333 conn(0x7f03b00a36c0 msgr2=0x7f03b00a3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:35.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.626+0000 7f03b5ba2640 1 -- 172.21.15.12:0/1500422333 shutdown_connections 2024-09-11T01:39:35.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.627+0000 7f03b5ba2640 1 -- 172.21.15.12:0/1500422333 wait complete. 2024-09-11T01:39:35.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.627+0000 7f03b5ba2640 10 monclient: build_initial_monmap 2024-09-11T01:39:35.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.627+0000 7f03b5ba2640 10 monclient: monmap: 2024-09-11T01:39:35.636 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:35.636 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:39:35.636 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:39:35.636 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:39:35.636 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:35.636 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.636 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:39:35.636 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:39:35.636 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:39:35.636 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.627+0000 7f03b5ba2640 1 Processor -- start 2024-09-11T01:39:35.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03b5ba2640 1 -- start start 2024-09-11T01:39:35.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03b5ba2640 10 monclient: init 2024-09-11T01:39:35.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03b5ba2640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:35.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03b5ba2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:39:35.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03b5ba2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03b0073f50 0x7f03b01cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03b5ba2640 10 monclient(hunting): picked mon.noname-a con 0x7f03b0073f50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:39:35.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03b5ba2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03b0074870 0x7f03b0131860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03b5ba2640 10 monclient(hunting): picked mon.noname-c con 0x7f03b0074870 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:35.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03b5ba2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03b00a0810 0x7f03b0131da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03b5ba2640 10 monclient(hunting): picked mon.noname-b con 0x7f03b00a0810 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:39:35.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03b5ba2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:35.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03b5ba2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f03b00a33a0 con 0x7f03b0073f50 2024-09-11T01:39:35.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03b5ba2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:35.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03b5ba2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f03b00a3220 con 0x7f03b00a0810 2024-09-11T01:39:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03b5ba2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03b5ba2640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f03b0134bc0 con 0x7f03b0074870 2024-09-11T01:39:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03b5ba2640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03b5ba2640 10 monclient(hunting): authenticate will time out at 1382.070648s 2024-09-11T01:39:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03aeffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03b0074870 0x7f03b0131860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03af7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03b0073f50 0x7f03b01cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03af7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03b0073f50 0x7f03b01cd0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40024/0 (socket says 172.21.15.12:40024) 2024-09-11T01:39:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03af7fe640 1 -- 172.21.15.12:0/3381344360 learned_addr learned my addr 172.21.15.12:0/3381344360 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:39:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03af7fe640 10 monclient(hunting): get_auth_request con 0x7f03b0073f50 auth_method 0 2024-09-11T01:39:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03aeffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03b0074870 0x7f03b0131860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:44576/0 (socket says 172.21.15.12:44576) 2024-09-11T01:39:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03aeffd640 10 monclient(hunting): get_auth_request con 0x7f03b0074870 auth_method 0 2024-09-11T01:39:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.628+0000 7f03affff640 1 --2- 172.21.15.12:0/3381344360 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03b00a0810 0x7f03b0131da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.629+0000 7f03affff640 10 monclient(hunting): get_auth_request con 0x7f03b00a0810 auth_method 0 2024-09-11T01:39:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.629+0000 7f03affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:35.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.629+0000 7f03affff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:35.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.629+0000 7f03affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:35.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.629+0000 7f03af7fe640 10 monclient(hunting): handle_auth_done global_id 94278 payload 824 2024-09-11T01:39:35.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.629+0000 7f03af7fe640 1 -- 172.21.15.12:0/3381344360 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03b0074870 msgr2=0x7f03b0131860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:35.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.629+0000 7f03af7fe640 1 --2- 172.21.15.12:0/3381344360 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03b0074870 0x7f03b0131860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.629+0000 7f03af7fe640 1 -- 172.21.15.12:0/3381344360 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03b00a0810 msgr2=0x7f03b0131da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:35.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.629+0000 7f03af7fe640 1 --2- 172.21.15.12:0/3381344360 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03b00a0810 0x7f03b0131da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.629+0000 7f03af7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:39:35.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.629+0000 7f03af7fe640 1 monclient: found mon.noname-a 2024-09-11T01:39:35.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.629+0000 7f03af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:35.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.629+0000 7f03af7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:39:35.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.629+0000 7f03af7fe640 1 -- 172.21.15.12:0/3381344360 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03b0134080 con 0x7f03b0073f50 2024-09-11T01:39:35.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.629+0000 7f03aeffd640 1 --2- 172.21.15.12:0/3381344360 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03b0074870 0x7f03b0131860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:39:35.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.631+0000 7f03af7fe640 1 -- 172.21.15.12:0/3381344360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03b0073f50 msgr2=0x7f03b01cd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:35.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.631+0000 7f03af7fe640 1 --2- 172.21.15.12:0/3381344360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03b0073f50 0x7f03b01cd0d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f03a000c7f0 tx=0x7f03a000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:35.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.631+0000 7f0397fff640 1 -- 172.21.15.12:0/3381344360 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a0015070 con 0x7f03b0073f50 2024-09-11T01:39:35.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.631+0000 7f0397fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.631+0000 7f0397fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:39:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.631+0000 7f0397fff640 10 monclient: dump: 2024-09-11T01:39:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:35.645 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.631+0000 7f0397fff640 10 monclient: _finish_auth 0 2024-09-11T01:39:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.631+0000 7f0397fff640 10 monclient: _check_auth_tickets 2024-09-11T01:39:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.631+0000 7f0397fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.631+0000 7f0397fff640 1 -- 172.21.15.12:0/3381344360 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03a0007a50 con 0x7f03b0073f50 2024-09-11T01:39:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.631+0000 7f0397fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.631+0000 7f0397fff640 1 -- 172.21.15.12:0/3381344360 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a00113f0 con 0x7f03b0073f50 2024-09-11T01:39:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.631+0000 7f0397fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.631+0000 7f0397fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:39:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.631+0000 7f0397fff640 10 monclient: dump: 2024-09-11T01:39:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:35.647 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:35.647 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:35.647 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.631+0000 7f03b5ba2640 5 monclient: authenticate success, global_id 94278 2024-09-11T01:39:35.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.631+0000 7f03b5ba2640 10 monclient: _renew_subs 2024-09-11T01:39:35.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.631+0000 7f03b5ba2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:35.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.631+0000 7f03b5ba2640 1 -- 172.21.15.12:0/3381344360 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03b00a6740 con 0x7f03b0073f50 2024-09-11T01:39:35.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.631+0000 7f03ad7fa640 4 set_mon_vals no callback set 2024-09-11T01:39:35.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.631+0000 7f03b5ba2640 10 monclient: _renew_subs 2024-09-11T01:39:35.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.631+0000 7f03b5ba2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:35.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.631+0000 7f03b5ba2640 1 -- 172.21.15.12:0/3381344360 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03b01cd610 con 0x7f03b0073f50 2024-09-11T01:39:35.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.632+0000 7f03b5ba2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:35.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.632+0000 7f03b5ba2640 1 --2- 172.21.15.12:0/3381344360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03b01c0340 0x7f03b002d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.632+0000 7f03b5ba2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.632+0000 7f03b5ba2640 1 -- 172.21.15.12:0/3381344360 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f03b0074fe0 con 0x7f03b01c0340 2024-09-11T01:39:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.632+0000 7f03aeffd640 1 --2- 172.21.15.12:0/3381344360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03b01c0340 0x7f03b002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.632+0000 7f03aeffd640 10 monclient: get_auth_request con 0x7f03b01c0340 auth_method 0 2024-09-11T01:39:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.632+0000 7f03aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.632+0000 7f03aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.632+0000 7f03aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.632+0000 7f03aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.632+0000 7f03aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.632+0000 7f03aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.632+0000 7f03af7fe640 1 --2- 172.21.15.12:0/3381344360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03b0073f50 0x7f03b01cd0d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f03a000c7f0 tx=0x7f03a000ccb0 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7f03a0052a80 osd_map(344..344 src has 186..344) 2024-09-11T01:39:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.632+0000 7f0397fff640 1 -- 172.21.15.12:0/3381344360 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f03a0022060 con 0x7f03b0073f50 2024-09-11T01:39:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.632+0000 7f03aeffd640 10 monclient(hunting): handle_auth_done global_id 94281 payload 824 2024-09-11T01:39:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.632+0000 7f03aeffd640 1 -- 172.21.15.12:0/3381344360 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f03b0073740 con 0x7f03b01c0340 2024-09-11T01:39:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.632+0000 7f0397fff640 1 --2- 172.21.15.12:0/3381344360 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f039c03e240 0x7f039c040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.633+0000 7f03affff640 1 --2- 172.21.15.12:0/3381344360 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f039c03e240 0x7f039c040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.633+0000 7f03affff640 10 monclient: get_auth_request con 0x7f039c03e240 auth_method 0 2024-09-11T01:39:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.635+0000 7f03aeffd640 1 -- 172.21.15.12:0/3381344360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03b01c0340 msgr2=0x7f03b002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.635+0000 7f03aeffd640 1 --2- 172.21.15.12:0/3381344360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03b01c0340 0x7f03b002d440 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f03b0070e50 tx=0x7f03a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.635+0000 7f0397fff640 1 -- 172.21.15.12:0/3381344360 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a4037040 con 0x7f03b01c0340 2024-09-11T01:39:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.635+0000 7f0397fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.635+0000 7f0397fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:39:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.635+0000 7f0397fff640 10 monclient: dump: 2024-09-11T01:39:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.650 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:35.650 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:35.650 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:35.650 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.635+0000 7f0397fff640 1 -- 172.21.15.12:0/3381344360 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f03a4038040 con 0x7f03b01c0340 2024-09-11T01:39:35.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.635+0000 7f0397fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:35.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.635+0000 7f0397fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:39:35.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.635+0000 7f0397fff640 1 -- 172.21.15.12:0/3381344360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03b01c0340 msgr2=0x7f03b002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:35.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.635+0000 7f0397fff640 1 --2- 172.21.15.12:0/3381344360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03b01c0340 0x7f03b002d440 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f03b0070e50 tx=0x7f03a40046c0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.635+0000 7f0397fff640 1 -- 172.21.15.12:0/3381344360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03b01c0340 msgr2=0x7f03b002d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:39:35.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.635+0000 7f0397fff640 1 --2- 172.21.15.12:0/3381344360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03b01c0340 0x7f03b002d440 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f03b0070e50 tx=0x7f03a40046c0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.636+0000 7f03affff640 1 --2- 172.21.15.12:0/3381344360 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f039c03e240 0x7f039c040700 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f03a8007ae0 tx=0x7f03a8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:35.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.641+0000 7f03b5ba2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:39:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.641+0000 7f03b5ba2640 1 --2- 172.21.15.12:0/3381344360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03b01c0720 0x7f03b002d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.641+0000 7f03b5ba2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.641+0000 7f03b5ba2640 1 -- 172.21.15.12:0/3381344360 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f03b0074330 con 0x7f03b01c0720 2024-09-11T01:39:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.641+0000 7f03aeffd640 1 --2- 172.21.15.12:0/3381344360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03b01c0720 0x7f03b002d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.641+0000 7f03aeffd640 10 monclient: get_auth_request con 0x7f03b01c0720 auth_method 0 2024-09-11T01:39:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.641+0000 7f03aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.641+0000 7f03aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.641+0000 7f03aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.641+0000 7f03aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.641+0000 7f03aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.641+0000 7f03aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.642+0000 7f03aeffd640 10 monclient(hunting): handle_auth_done global_id 94284 payload 824 2024-09-11T01:39:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.642+0000 7f03aeffd640 1 -- 172.21.15.12:0/3381344360 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f03b01bbe20 con 0x7f03b01c0720 2024-09-11T01:39:35.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.644+0000 7f03aeffd640 1 -- 172.21.15.12:0/3381344360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03b01c0720 msgr2=0x7f03b002d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:35.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.644+0000 7f03aeffd640 1 --2- 172.21.15.12:0/3381344360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03b01c0720 0x7f03b002d980 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f03a402a120 tx=0x7f03a4008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.644+0000 7f0397fff640 1 -- 172.21.15.12:0/3381344360 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a4037040 con 0x7f03b01c0720 2024-09-11T01:39:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.644+0000 7f0397fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.644+0000 7f0397fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:39:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.644+0000 7f0397fff640 10 monclient: dump: 2024-09-11T01:39:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:35.654 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:35.654 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.644+0000 7f0397fff640 1 -- 172.21.15.12:0/3381344360 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f03a4049e50 con 0x7f03b01c0720 2024-09-11T01:39:35.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.644+0000 7f0397fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:39:35.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.644+0000 7f0397fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:39:35.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.644+0000 7f0397fff640 1 -- 172.21.15.12:0/3381344360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03b01c0720 msgr2=0x7f03b002d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:35.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.644+0000 7f0397fff640 1 --2- 172.21.15.12:0/3381344360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03b01c0720 0x7f03b002d980 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f03a402a120 tx=0x7f03a4008030 comp rx=0 tx=0).stop 2024-09-11T01:39:35.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.644+0000 7f0397fff640 1 -- 172.21.15.12:0/3381344360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03b01c0720 msgr2=0x7f03b002d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:39:35.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.644+0000 7f0397fff640 1 --2- 172.21.15.12:0/3381344360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03b01c0720 0x7f03b002d980 secure :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f03a402a120 tx=0x7f03a4008030 comp rx=0 tx=0).stop 2024-09-11T01:39:35.654 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:39:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:39:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:39:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:39:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 100, 2024-09-11T01:39:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 121740, 2024-09-11T01:39:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:39:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:39:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:39:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:39:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:39:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:39:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:39:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:39:35.656 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:39:35.656 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:39:35.656 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:39:35.656 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:39:35.656 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:39:35.656 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:39:35.656 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:39:35.656 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:39:35.656 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:39:35.656 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:39:35.656 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:39:35.656 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:35.656 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:39:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:39:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:39:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:39:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:39:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:39:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:39:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:39:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:39:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:39:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:39:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:39:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:39:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:39:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:39:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:39:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:39:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:39:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:39:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:39:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:39:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:39:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:39:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:39:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:39:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:39:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:39:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:39:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:39:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:39:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:39:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:39:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:39:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:39:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:39:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:39:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:39:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:39:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:39:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:39:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:39:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:39:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:39:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:39:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:39:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:39:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:39:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:39:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:39:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:39:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:39:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:39:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:39:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:39:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:39:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:39:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:39:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:39:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:39:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:39:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:39:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:39:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:39:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:39:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:39:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:39:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:39:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:39:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:39:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:39:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:39:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:39:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:39:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:39:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:39:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:39:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:39:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:39:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:39:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:39:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:39:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:39:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:39:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:39:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:39:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:39:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:39:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:39:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:39:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:39:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:39:35.666 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:35.666 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:35.666 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:39:35.666 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.666 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:39:35.666 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:39:35.666 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 7 2024-09-11T01:39:35.666 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:35.666 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:35.666 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.666 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:39:35.666 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:39:35.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.645+0000 7f03b5ba2640 1 -- 172.21.15.12:0/3381344360 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f039c03e240 msgr2=0x7f039c040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:35.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.645+0000 7f03b5ba2640 1 --2- 172.21.15.12:0/3381344360 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f039c03e240 0x7f039c040700 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f03a8007ae0 tx=0x7f03a8009040 comp rx=0 tx=0).stop 2024-09-11T01:39:35.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.645+0000 7f03b5ba2640 10 monclient: shutdown 2024-09-11T01:39:35.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.645+0000 7f03b5ba2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:35.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.645+0000 7f03b5ba2640 1 -- 172.21.15.12:0/3381344360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03b0073f50 msgr2=0x7f03b01cd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:35.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.645+0000 7f03b5ba2640 1 --2- 172.21.15.12:0/3381344360 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03b0073f50 0x7f03b01cd0d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f03a000c7f0 tx=0x7f03a000ccb0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.645+0000 7f03affff640 1 -- 172.21.15.12:0/3381344360 reap_dead start 2024-09-11T01:39:35.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.645+0000 7f0397fff640 1 -- 172.21.15.12:0/3381344360 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(344..344 src has 186..344) ==== 4811+0+0 (unknown 0 0 0) 0x7f03a0052a80 con 0x7f03b0073f50 2024-09-11T01:39:35.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.645+0000 7f03affff640 1 -- 172.21.15.12:0/3381344360 reap_dead start 2024-09-11T01:39:35.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.645+0000 7f03b5ba2640 1 -- 172.21.15.12:0/3381344360 shutdown_connections 2024-09-11T01:39:35.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.645+0000 7f03b5ba2640 1 -- 172.21.15.12:0/3381344360 >> 172.21.15.12:0/3381344360 conn(0x7f03b00a36c0 msgr2=0x7f03b0073030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:35.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.645+0000 7f03b5ba2640 1 -- 172.21.15.12:0/3381344360 shutdown_connections 2024-09-11T01:39:35.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.645+0000 7f03b5ba2640 1 -- 172.21.15.12:0/3381344360 wait complete. 2024-09-11T01:39:35.669 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:39:35.669 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:39:35.669 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:39:35.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.769+0000 7f5a459e2640 10 monclient: get_monmap_and_config 2024-09-11T01:39:35.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.769+0000 7f5a459e2640 10 monclient: build_initial_monmap 2024-09-11T01:39:35.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.769+0000 7f5a459e2640 10 monclient: monmap: 2024-09-11T01:39:35.771 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:35.771 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:35.771 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:39:35.770643+0000 2024-09-11T01:39:35.771 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:39:35.770643+0000 2024-09-11T01:39:35.771 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:35.772 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.772 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:39:35.772 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:39:35.772 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:39:35.772 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:39:35.772 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:39:35.772 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.770+0000 7f5a459e2640 1 Processor -- start 2024-09-11T01:39:35.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.770+0000 7f5a459e2640 1 -- start start 2024-09-11T01:39:35.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.770+0000 7f5a459e2640 10 monclient: init 2024-09-11T01:39:35.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.770+0000 7f5a459e2640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:35.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.770+0000 7f5a459e2640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-11T01:39:35.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a459e2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5a40134ad0 addr v1:172.21.15.71:6789/0 2024-09-11T01:39:35.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a459e2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5a400a2200 addr v1:172.21.15.12:6789/0 2024-09-11T01:39:35.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a459e2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a401355c0 0x7f5a4013d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a459e2640 10 monclient(hunting): picked mon.noname-b con 0x7f5a401355c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:35.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a459e2640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a4009ed30 con 0x7f5a400a2200 2024-09-11T01:39:35.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a459e2640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a4009f010 con 0x7f5a40134ad0 2024-09-11T01:39:35.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a459e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:35.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a459e2640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5a4009f5d0 con 0x7f5a401355c0 2024-09-11T01:39:35.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a459e2640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:35.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a459e2640 10 monclient(hunting): authenticate will time out at 1382.213073s 2024-09-11T01:39:35.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a3f7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a401355c0 0x7f5a4013d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a3f7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a401355c0 0x7f5a4013d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:44590/0 (socket says 172.21.15.12:44590) 2024-09-11T01:39:35.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a3f7fe640 1 -- 172.21.15.12:0/3455091965 learned_addr learned my addr 172.21.15.12:0/3455091965 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:39:35.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a3f7fe640 10 monclient(hunting): get_auth_request con 0x7f5a401355c0 auth_method 0 2024-09-11T01:39:35.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:35.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:35.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:35.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:35.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:35.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:35.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a3f7fe640 10 monclient(hunting): handle_auth_done global_id 114110 payload 274 2024-09-11T01:39:35.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a3f7fe640 1 -- 172.21.15.12:0/3455091965 >> v1:172.21.15.71:6789/0 conn(0x7f5a40134ad0 legacy=0x7f5a40134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:39:35.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a3f7fe640 1 -- 172.21.15.12:0/3455091965 >> v1:172.21.15.12:6789/0 conn(0x7f5a400a2200 legacy=0x7f5a4013a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:39:35.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a3f7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:39:35.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a3f7fe640 1 monclient: found mon.noname-b 2024-09-11T01:39:35.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:35.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a3f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:39:35.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.771+0000 7f5a3f7fe640 1 -- 172.21.15.12:0/3455091965 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a4009f2f0 con 0x7f5a401355c0 2024-09-11T01:39:35.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.774+0000 7f5a3f7fe640 1 -- 172.21.15.12:0/3455091965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a401355c0 msgr2=0x7f5a4013d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:35.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.774+0000 7f5a3f7fe640 1 --2- 172.21.15.12:0/3455091965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a401355c0 0x7f5a4013d730 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5a38009eb0 tx=0x7f5a3802a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f5e557e77062c8c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:35.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.774+0000 7f5a3dffb640 1 -- 172.21.15.12:0/3455091965 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a38036070 con 0x7f5a401355c0 2024-09-11T01:39:35.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.774+0000 7f5a3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:35.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.774+0000 7f5a3dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:39:35.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.774+0000 7f5a3dffb640 10 monclient: dump: 2024-09-11T01:39:35.780 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:35.780 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:35.780 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.780 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.780 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:35.780 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:35.781 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.774+0000 7f5a3dffb640 10 monclient: _finish_auth 0 2024-09-11T01:39:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.774+0000 7f5a3dffb640 10 monclient: _check_auth_tickets 2024-09-11T01:39:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.774+0000 7f5a3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.774+0000 7f5a459e2640 5 monclient: authenticate success, global_id 114110 2024-09-11T01:39:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.774+0000 7f5a3dffb640 1 -- 172.21.15.12:0/3455091965 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a38007800 con 0x7f5a401355c0 2024-09-11T01:39:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.774+0000 7f5a459e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:39:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.774+0000 7f5a3dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.774+0000 7f5a459e2640 10 monclient: get_monmap_and_config success 2024-09-11T01:39:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.774+0000 7f5a3dffb640 1 -- 172.21.15.12:0/3455091965 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a38007bd0 con 0x7f5a401355c0 2024-09-11T01:39:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.774+0000 7f5a459e2640 4 set_mon_vals no callback set 2024-09-11T01:39:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.774+0000 7f5a3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.774+0000 7f5a3dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:39:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.774+0000 7f5a3dffb640 10 monclient: dump: 2024-09-11T01:39:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:35.782 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.775+0000 7f5a459e2640 10 monclient: shutdown 2024-09-11T01:39:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.775+0000 7f5a459e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.775+0000 7f5a459e2640 1 -- 172.21.15.12:0/3455091965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a401355c0 msgr2=0x7f5a4013d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.775+0000 7f5a459e2640 1 --2- 172.21.15.12:0/3455091965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a401355c0 0x7f5a4013d730 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5a38009eb0 tx=0x7f5a3802a560 comp rx=0 tx=0).stop 2024-09-11T01:39:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.775+0000 7f5a459e2640 1 -- 172.21.15.12:0/3455091965 shutdown_connections 2024-09-11T01:39:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.775+0000 7f5a459e2640 1 --2- 172.21.15.12:0/3455091965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a401355c0 0x7f5a4013d730 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.775+0000 7f5a459e2640 1 -- 172.21.15.12:0/3455091965 >> 172.21.15.12:0/3455091965 conn(0x7f5a400a2f70 msgr2=0x7f5a400a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.775+0000 7f5a459e2640 1 -- 172.21.15.12:0/3455091965 shutdown_connections 2024-09-11T01:39:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.775+0000 7f5a459e2640 1 -- 172.21.15.12:0/3455091965 wait complete. 2024-09-11T01:39:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.775+0000 7f5a459e2640 10 monclient: build_initial_monmap 2024-09-11T01:39:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.775+0000 7f5a459e2640 10 monclient: monmap: 2024-09-11T01:39:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:39:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:39:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:39:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.784 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:39:35.784 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:39:35.784 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:39:35.784 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.775+0000 7f5a459e2640 1 Processor -- start 2024-09-11T01:39:35.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a459e2640 1 -- start start 2024-09-11T01:39:35.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a459e2640 10 monclient: init 2024-09-11T01:39:35.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a459e2640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:35.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a459e2640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:39:35.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a459e2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a400a2200 0x7f5a401cd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a459e2640 10 monclient(hunting): picked mon.noname-b con 0x7f5a400a2200 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:39:35.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a459e2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a40134ad0 0x7f5a40133cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a459e2640 10 monclient(hunting): picked mon.noname-a con 0x7f5a40134ad0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:39:35.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a459e2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a401355c0 0x7f5a401341f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a459e2640 10 monclient(hunting): picked mon.noname-c con 0x7f5a401355c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:35.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a459e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:35.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a459e2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5a4009cde0 con 0x7f5a40134ad0 2024-09-11T01:39:35.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a459e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:35.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a459e2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5a400a01e0 con 0x7f5a400a2200 2024-09-11T01:39:35.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a459e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:35.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a459e2640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5a400a0c90 con 0x7f5a401355c0 2024-09-11T01:39:35.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a459e2640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:35.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a459e2640 10 monclient(hunting): authenticate will time out at 1382.218695s 2024-09-11T01:39:35.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a3f7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a401355c0 0x7f5a401341f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a3e7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a40134ad0 0x7f5a40133cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a3f7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a401355c0 0x7f5a401341f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:44604/0 (socket says 172.21.15.12:44604) 2024-09-11T01:39:35.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a3f7fe640 1 -- 172.21.15.12:0/143804787 learned_addr learned my addr 172.21.15.12:0/143804787 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:39:35.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a3f7fe640 10 monclient(hunting): get_auth_request con 0x7f5a401355c0 auth_method 0 2024-09-11T01:39:35.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:35.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:35.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:35.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a3e7fc640 10 monclient(hunting): get_auth_request con 0x7f5a40134ad0 auth_method 0 2024-09-11T01:39:35.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a3e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:35.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a3e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:35.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:35.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a3effd640 1 --2- 172.21.15.12:0/143804787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a400a2200 0x7f5a401cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:35.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:35.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:35.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a3effd640 10 monclient(hunting): get_auth_request con 0x7f5a400a2200 auth_method 0 2024-09-11T01:39:35.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:35.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a3effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:35.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:35.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a3e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:35.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a3e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:35.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.776+0000 7f5a3e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:35.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.777+0000 7f5a3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:35.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.777+0000 7f5a3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:35.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.777+0000 7f5a3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:35.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.777+0000 7f5a3f7fe640 10 monclient(hunting): handle_auth_done global_id 114113 payload 824 2024-09-11T01:39:35.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.777+0000 7f5a3f7fe640 1 -- 172.21.15.12:0/143804787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a400a2200 msgr2=0x7f5a401cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:35.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.777+0000 7f5a3f7fe640 1 --2- 172.21.15.12:0/143804787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a400a2200 0x7f5a401cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.777+0000 7f5a3f7fe640 1 -- 172.21.15.12:0/143804787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a40134ad0 msgr2=0x7f5a40133cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:35.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.777+0000 7f5a3f7fe640 1 --2- 172.21.15.12:0/143804787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a40134ad0 0x7f5a40133cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.777+0000 7f5a3f7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:39:35.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.777+0000 7f5a3f7fe640 1 monclient: found mon.noname-c 2024-09-11T01:39:35.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.777+0000 7f5a3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:35.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.777+0000 7f5a3f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:39:35.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.777+0000 7f5a3f7fe640 1 -- 172.21.15.12:0/143804787 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a400a4a40 con 0x7f5a401355c0 2024-09-11T01:39:35.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.777+0000 7f5a3e7fc640 1 --2- 172.21.15.12:0/143804787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5a40134ad0 0x7f5a40133cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:39:35.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.779+0000 7f5a3f7fe640 1 -- 172.21.15.12:0/143804787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a401355c0 msgr2=0x7f5a401341f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:35.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.779+0000 7f5a3f7fe640 1 --2- 172.21.15.12:0/143804787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a401355c0 0x7f5a401341f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5a38009eb0 tx=0x7f5a38004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:35.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.779+0000 7f5a27fff640 1 -- 172.21.15.12:0/143804787 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a38041070 con 0x7f5a401355c0 2024-09-11T01:39:35.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.779+0000 7f5a27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:35.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.779+0000 7f5a27fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:39:35.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.779+0000 7f5a27fff640 10 monclient: dump: 2024-09-11T01:39:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:35.793 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.779+0000 7f5a27fff640 10 monclient: _finish_auth 0 2024-09-11T01:39:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.779+0000 7f5a27fff640 10 monclient: _check_auth_tickets 2024-09-11T01:39:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.779+0000 7f5a27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.779+0000 7f5a27fff640 1 -- 172.21.15.12:0/143804787 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a3803c860 con 0x7f5a401355c0 2024-09-11T01:39:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.779+0000 7f5a459e2640 5 monclient: authenticate success, global_id 114113 2024-09-11T01:39:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.779+0000 7f5a459e2640 10 monclient: _renew_subs 2024-09-11T01:39:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.779+0000 7f5a459e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:39:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.779+0000 7f5a459e2640 1 -- 172.21.15.12:0/143804787 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a40128700 con 0x7f5a401355c0 2024-09-11T01:39:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.779+0000 7f5a27fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.779+0000 7f5a27fff640 1 -- 172.21.15.12:0/143804787 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a38036040 con 0x7f5a401355c0 2024-09-11T01:39:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.779+0000 7f5a27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.779+0000 7f5a27fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:39:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.779+0000 7f5a27fff640 10 monclient: dump: 2024-09-11T01:39:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:35.795 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.779+0000 7f5a459e2640 10 monclient: _renew_subs 2024-09-11T01:39:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.779+0000 7f5a459e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:39:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.779+0000 7f5a459e2640 1 -- 172.21.15.12:0/143804787 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a401263a0 con 0x7f5a401355c0 2024-09-11T01:39:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.779+0000 7f5a3d7fa640 4 set_mon_vals no callback set 2024-09-11T01:39:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.780+0000 7f5a27fff640 1 -- 172.21.15.12:0/143804787 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f5a3803cc70 con 0x7f5a401355c0 2024-09-11T01:39:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.780+0000 7f5a459e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.780+0000 7f5a459e2640 1 --2- 172.21.15.12:0/143804787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a40128280 0x7f5a401248e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.780+0000 7f5a459e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.780+0000 7f5a459e2640 1 -- 172.21.15.12:0/143804787 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5a401280a0 con 0x7f5a40128280 2024-09-11T01:39:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.780+0000 7f5a27fff640 1 --2- 172.21.15.12:0/143804787 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5a2c03e270 0x7f5a2c040730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.780+0000 7f5a3effd640 1 --2- 172.21.15.12:0/143804787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a40128280 0x7f5a401248e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.780+0000 7f5a3effd640 10 monclient: get_auth_request con 0x7f5a40128280 auth_method 0 2024-09-11T01:39:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.780+0000 7f5a3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.780+0000 7f5a3effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.780+0000 7f5a3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.780+0000 7f5a27fff640 1 -- 172.21.15.12:0/143804787 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(344..344 src has 186..344) ==== 4811+0+0 (secure 0 0 0) 0x7f5a380739f0 con 0x7f5a401355c0 2024-09-11T01:39:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.780+0000 7f5a3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.780+0000 7f5a3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.780+0000 7f5a3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.780+0000 7f5a3effd640 10 monclient(hunting): handle_auth_done global_id 114116 payload 824 2024-09-11T01:39:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.780+0000 7f5a3effd640 1 -- 172.21.15.12:0/143804787 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5a40134eb0 con 0x7f5a40128280 2024-09-11T01:39:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.781+0000 7f5a3e7fc640 1 --2- 172.21.15.12:0/143804787 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5a2c03e270 0x7f5a2c040730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.781+0000 7f5a3e7fc640 10 monclient: get_auth_request con 0x7f5a2c03e270 auth_method 0 2024-09-11T01:39:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.783+0000 7f5a3effd640 1 -- 172.21.15.12:0/143804787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a40128280 msgr2=0x7f5a401248e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.783+0000 7f5a3effd640 1 --2- 172.21.15.12:0/143804787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a40128280 0x7f5a401248e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5a3000f7a0 tx=0x7f5a3000fc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.783+0000 7f5a27fff640 1 -- 172.21.15.12:0/143804787 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a30018030 con 0x7f5a40128280 2024-09-11T01:39:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.783+0000 7f5a27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.783+0000 7f5a27fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:39:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.783+0000 7f5a27fff640 10 monclient: dump: 2024-09-11T01:39:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:35.798 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.783+0000 7f5a3e7fc640 1 --2- 172.21.15.12:0/143804787 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5a2c03e270 0x7f5a2c040730 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5a40127900 tx=0x7f5a34007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.783+0000 7f5a27fff640 1 -- 172.21.15.12:0/143804787 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5a3000d9b0 con 0x7f5a40128280 2024-09-11T01:39:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.783+0000 7f5a27fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.783+0000 7f5a27fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:39:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.783+0000 7f5a27fff640 1 -- 172.21.15.12:0/143804787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a40128280 msgr2=0x7f5a401248e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.783+0000 7f5a27fff640 1 --2- 172.21.15.12:0/143804787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a40128280 0x7f5a401248e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5a3000f7a0 tx=0x7f5a3000fc60 comp rx=0 tx=0).stop 2024-09-11T01:39:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.783+0000 7f5a27fff640 1 -- 172.21.15.12:0/143804787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a40128280 msgr2=0x7f5a401248e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:39:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.783+0000 7f5a27fff640 1 --2- 172.21.15.12:0/143804787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a40128280 0x7f5a401248e0 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5a3000f7a0 tx=0x7f5a3000fc60 comp rx=0 tx=0).stop 2024-09-11T01:39:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.789+0000 7f5a459e2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:39:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.789+0000 7f5a459e2640 1 --2- 172.21.15.12:0/143804787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a40124e20 0x7f5a40125200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.789+0000 7f5a459e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:35.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.789+0000 7f5a459e2640 1 -- 172.21.15.12:0/143804787 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5a40125740 con 0x7f5a40124e20 2024-09-11T01:39:35.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.789+0000 7f5a3effd640 1 --2- 172.21.15.12:0/143804787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a40124e20 0x7f5a40125200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.789+0000 7f5a3effd640 10 monclient: get_auth_request con 0x7f5a40124e20 auth_method 0 2024-09-11T01:39:35.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.789+0000 7f5a3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:35.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.789+0000 7f5a3effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:35.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.789+0000 7f5a3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:35.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.789+0000 7f5a3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:35.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.789+0000 7f5a3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:35.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.789+0000 7f5a3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:35.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.790+0000 7f5a3effd640 10 monclient(hunting): handle_auth_done global_id 114119 payload 824 2024-09-11T01:39:35.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.790+0000 7f5a3effd640 1 -- 172.21.15.12:0/143804787 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5a4006d480 con 0x7f5a40124e20 2024-09-11T01:39:35.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.792+0000 7f5a3effd640 1 -- 172.21.15.12:0/143804787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a40124e20 msgr2=0x7f5a40125200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:35.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.792+0000 7f5a3effd640 1 --2- 172.21.15.12:0/143804787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a40124e20 0x7f5a40125200 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5a3000f7a0 tx=0x7f5a30003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:35.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.792+0000 7f5a27fff640 1 -- 172.21.15.12:0/143804787 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3001dbe0 con 0x7f5a40124e20 2024-09-11T01:39:35.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.792+0000 7f5a27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:35.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.792+0000 7f5a27fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:39:35.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.792+0000 7f5a27fff640 10 monclient: dump: 2024-09-11T01:39:35.801 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:35.801 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:35.801 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.801 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.801 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:35.801 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.801 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:35.801 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:35.801 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:35.801 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.792+0000 7f5a27fff640 1 -- 172.21.15.12:0/143804787 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7f5a3001dbe0 con 0x7f5a40124e20 2024-09-11T01:39:35.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.792+0000 7f5a27fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:39:35.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.792+0000 7f5a27fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:39:35.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.792+0000 7f5a27fff640 1 -- 172.21.15.12:0/143804787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a40124e20 msgr2=0x7f5a40125200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:35.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.792+0000 7f5a27fff640 1 --2- 172.21.15.12:0/143804787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a40124e20 0x7f5a40125200 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5a3000f7a0 tx=0x7f5a30003760 comp rx=0 tx=0).stop 2024-09-11T01:39:35.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.792+0000 7f5a27fff640 1 -- 172.21.15.12:0/143804787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a40124e20 msgr2=0x7f5a40125200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:39:35.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.792+0000 7f5a27fff640 1 --2- 172.21.15.12:0/143804787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a40124e20 0x7f5a40125200 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5a3000f7a0 tx=0x7f5a30003760 comp rx=0 tx=0).stop 2024-09-11T01:39:35.802 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:39:35.802 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:39:35.802 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:39:35.802 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:39:35.802 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 100, 2024-09-11T01:39:35.802 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 21588, 2024-09-11T01:39:35.803 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:39:35.803 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:39:35.803 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:39:35.803 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:39:35.803 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:35.803 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:39:35.803 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:39:35.803 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:39:35.803 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:39:35.803 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:39:35.803 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:39:35.803 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:39:35.804 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:39:35.804 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:39:35.804 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:39:35.804 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:39:35.804 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:39:35.804 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:39:35.804 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:39:35.804 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:39:35.804 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:35.804 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:39:35.804 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:39:35.804 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:39:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:39:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:39:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:39:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:39:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:39:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:39:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:39:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:39:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:39:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:39:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:39:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:39:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:39:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:39:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:39:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:39:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:39:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:39:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:39:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:39:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:39:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:39:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:39:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:39:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:39:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:39:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:39:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:39:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:39:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:39:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:39:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:39:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:39:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:39:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:39:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:39:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:39:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:39:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:39:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:39:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:39:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:39:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:39:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:39:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:39:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:39:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:39:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:39:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:39:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:39:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:39:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:39:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:39:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:39:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:39:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:39:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:39:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:39:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:39:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:39:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:39:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:39:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:39:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:39:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:39:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:39:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:39:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:39:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:39:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:39:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:39:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:39:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.812 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:39:35.812 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:39:35.812 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:35.812 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:35.812 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:35.812 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.812 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:39:35.812 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:39:35.812 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:39:35.812 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:39:35.812 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:39:35.812 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:35.812 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:39:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:39:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:39:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:39:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:39:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:39:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:39:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:39:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:39:35.814 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:35.814 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:35.814 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.814 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:39:35.814 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:39:35.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.793+0000 7f5a459e2640 1 -- 172.21.15.12:0/143804787 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5a2c03e270 msgr2=0x7f5a2c040730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:35.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.793+0000 7f5a459e2640 1 --2- 172.21.15.12:0/143804787 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5a2c03e270 0x7f5a2c040730 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5a40127900 tx=0x7f5a34007e30 comp rx=0 tx=0).stop 2024-09-11T01:39:35.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.793+0000 7f5a459e2640 10 monclient: shutdown 2024-09-11T01:39:35.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.793+0000 7f5a459e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:35.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.793+0000 7f5a459e2640 1 -- 172.21.15.12:0/143804787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a401355c0 msgr2=0x7f5a401341f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:35.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.793+0000 7f5a3f7fe640 1 -- 172.21.15.12:0/143804787 reap_dead start 2024-09-11T01:39:35.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.793+0000 7f5a459e2640 1 --2- 172.21.15.12:0/143804787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a401355c0 0x7f5a401341f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5a38009eb0 tx=0x7f5a38004a40 comp rx=0 tx=0).stop 2024-09-11T01:39:35.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.793+0000 7f5a459e2640 1 -- 172.21.15.12:0/143804787 shutdown_connections 2024-09-11T01:39:35.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.793+0000 7f5a459e2640 1 --2- 172.21.15.12:0/143804787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5a401355c0 0x7f5a401341f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.793+0000 7f5a459e2640 1 -- 172.21.15.12:0/143804787 >> 172.21.15.12:0/143804787 conn(0x7f5a400a2f70 msgr2=0x7f5a400a4e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:35.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.793+0000 7f5a459e2640 1 -- 172.21.15.12:0/143804787 shutdown_connections 2024-09-11T01:39:35.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.793+0000 7f5a459e2640 1 -- 172.21.15.12:0/143804787 wait complete. 2024-09-11T01:39:35.816 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:39:35.816 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:39:35.816 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:39:35.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.916+0000 7f5c5056b640 10 monclient: get_monmap_and_config 2024-09-11T01:39:35.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.916+0000 7f5c5056b640 10 monclient: build_initial_monmap 2024-09-11T01:39:35.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.916+0000 7f5c5056b640 10 monclient: monmap: 2024-09-11T01:39:35.918 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:35.918 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:35.918 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:39:35.917610+0000 2024-09-11T01:39:35.918 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:39:35.917610+0000 2024-09-11T01:39:35.918 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:35.919 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.919 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:39:35.919 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:39:35.919 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:39:35.919 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:39:35.919 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:39:35.919 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.917+0000 7f5c5056b640 1 Processor -- start 2024-09-11T01:39:35.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.917+0000 7f5c5056b640 1 -- start start 2024-09-11T01:39:35.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.917+0000 7f5c5056b640 10 monclient: init 2024-09-11T01:39:35.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c5056b640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:35.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c5056b640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-11T01:39:35.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c5056b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5c480a2e20 0x7f5c4806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c5056b640 10 monclient(hunting): picked mon.noname-c con 0x7f5c480a2e20 addr v2:172.21.15.71:3300/0 2024-09-11T01:39:35.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c5056b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5c480a4970 0x7f5c480a3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c5056b640 10 monclient(hunting): picked mon.noname-b con 0x7f5c480a4970 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:35.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c5056b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5c4806f910 addr v1:172.21.15.71:6789/0 2024-09-11T01:39:35.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c5056b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c4806f4f0 con 0x7f5c4806f910 2024-09-11T01:39:35.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c5056b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:35.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c5056b640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f5c4806ef30 con 0x7f5c480a2e20 2024-09-11T01:39:35.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c5056b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c5056b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5c4806ec50 con 0x7f5c480a4970 2024-09-11T01:39:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c5056b640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c5056b640 10 monclient(hunting): authenticate will time out at 1382.360245s 2024-09-11T01:39:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c4db06640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5c480a4970 0x7f5c480a3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c4db06640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5c480a4970 0x7f5c480a3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:44616/0 (socket says 172.21.15.12:44616) 2024-09-11T01:39:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c4db06640 1 -- 172.21.15.12:0/1327384174 learned_addr learned my addr 172.21.15.12:0/1327384174 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:39:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c4db06640 10 monclient(hunting): get_auth_request con 0x7f5c480a4970 auth_method 0 2024-09-11T01:39:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c4db06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c4db06640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c4db06640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c4db06640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c4db06640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c4db06640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c4e307640 1 --2- 172.21.15.12:0/1327384174 >> v2:172.21.15.71:3300/0 conn(0x7f5c480a2e20 0x7f5c4806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c4e307640 10 monclient(hunting): get_auth_request con 0x7f5c480a2e20 auth_method 0 2024-09-11T01:39:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c4e307640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c4e307640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c4e307640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c4db06640 10 monclient(hunting): handle_auth_done global_id 114122 payload 274 2024-09-11T01:39:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c4db06640 1 -- 172.21.15.12:0/1327384174 >> v2:172.21.15.71:3300/0 conn(0x7f5c480a2e20 msgr2=0x7f5c4806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:39:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c4db06640 1 --2- 172.21.15.12:0/1327384174 >> v2:172.21.15.71:3300/0 conn(0x7f5c480a2e20 0x7f5c4806ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c4db06640 1 -- 172.21.15.12:0/1327384174 >> v1:172.21.15.71:6789/0 conn(0x7f5c4806f910 legacy=0x7f5c48137fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:39:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c4db06640 10 monclient: _finish_hunting 0 2024-09-11T01:39:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c4db06640 1 monclient: found mon.noname-b 2024-09-11T01:39:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c4db06640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c4db06640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:39:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c4db06640 1 -- 172.21.15.12:0/1327384174 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c480a3740 con 0x7f5c480a4970 2024-09-11T01:39:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.918+0000 7f5c4e307640 1 --2- 172.21.15.12:0/1327384174 >> v2:172.21.15.71:3300/0 conn(0x7f5c480a2e20 0x7f5c4806ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:39:35.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.921+0000 7f5c4db06640 1 -- 172.21.15.12:0/1327384174 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5c480a4970 msgr2=0x7f5c480a3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:35.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.921+0000 7f5c4db06640 1 --2- 172.21.15.12:0/1327384174 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5c480a4970 0x7f5c480a3200 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5c38009870 tx=0x7f5c3802a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d66df1283e32dba6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:35.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.921+0000 7f5c4d305640 1 -- 172.21.15.12:0/1327384174 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c38037070 con 0x7f5c480a4970 2024-09-11T01:39:35.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.921+0000 7f5c4d305640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:35.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.921+0000 7f5c4d305640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:39:35.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.921+0000 7f5c4d305640 10 monclient: dump: 2024-09-11T01:39:35.931 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:35.931 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:35.931 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.931 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.931 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:35.931 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.931 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:35.931 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:35.931 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:35.931 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.921+0000 7f5c4d305640 10 monclient: _finish_auth 0 2024-09-11T01:39:35.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.921+0000 7f5c4d305640 10 monclient: _check_auth_tickets 2024-09-11T01:39:35.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.921+0000 7f5c4d305640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:35.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.921+0000 7f5c5056b640 5 monclient: authenticate success, global_id 114122 2024-09-11T01:39:35.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.921+0000 7f5c5056b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:39:35.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.921+0000 7f5c4d305640 1 -- 172.21.15.12:0/1327384174 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c38032440 con 0x7f5c480a4970 2024-09-11T01:39:35.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.921+0000 7f5c4d305640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:35.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.921+0000 7f5c5056b640 10 monclient: get_monmap_and_config success 2024-09-11T01:39:35.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.921+0000 7f5c5056b640 4 set_mon_vals no callback set 2024-09-11T01:39:35.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.921+0000 7f5c4d305640 1 -- 172.21.15.12:0/1327384174 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c38032810 con 0x7f5c480a4970 2024-09-11T01:39:35.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.921+0000 7f5c4d305640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:35.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.921+0000 7f5c4d305640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:39:35.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.921+0000 7f5c4d305640 10 monclient: dump: 2024-09-11T01:39:35.932 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:35.932 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:35.932 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:35.933 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.922+0000 7f5c5056b640 10 monclient: shutdown 2024-09-11T01:39:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.922+0000 7f5c5056b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.922+0000 7f5c5056b640 1 -- 172.21.15.12:0/1327384174 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5c480a4970 msgr2=0x7f5c480a3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.922+0000 7f5c5056b640 1 --2- 172.21.15.12:0/1327384174 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5c480a4970 0x7f5c480a3200 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5c38009870 tx=0x7f5c3802a0a0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.922+0000 7f5c5056b640 1 -- 172.21.15.12:0/1327384174 shutdown_connections 2024-09-11T01:39:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.922+0000 7f5c5056b640 1 --2- 172.21.15.12:0/1327384174 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5c480a4970 0x7f5c480a3200 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.922+0000 7f5c5056b640 1 --2- 172.21.15.12:0/1327384174 >> v2:172.21.15.71:3300/0 conn(0x7f5c480a2e20 0x7f5c4806ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.922+0000 7f5c5056b640 1 -- 172.21.15.12:0/1327384174 >> 172.21.15.12:0/1327384174 conn(0x7f5c480a8600 msgr2=0x7f5c480a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:35.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.922+0000 7f5c5056b640 1 -- 172.21.15.12:0/1327384174 shutdown_connections 2024-09-11T01:39:35.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.922+0000 7f5c5056b640 1 -- 172.21.15.12:0/1327384174 wait complete. 2024-09-11T01:39:35.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.922+0000 7f5c5056b640 10 monclient: build_initial_monmap 2024-09-11T01:39:35.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.922+0000 7f5c5056b640 10 monclient: monmap: 2024-09-11T01:39:35.934 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:35.934 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:39:35.934 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:39:35.934 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:39:35.934 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:35.934 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.934 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:39:35.934 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:39:35.934 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:39:35.934 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c5056b640 1 Processor -- start 2024-09-11T01:39:35.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c5056b640 1 -- start start 2024-09-11T01:39:35.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c5056b640 10 monclient: init 2024-09-11T01:39:35.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c5056b640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:35.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c5056b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:39:35.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c5056b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c4806f910 0x7f5c481cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c5056b640 10 monclient(hunting): picked mon.noname-a con 0x7f5c4806f910 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:39:35.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c5056b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5c480a2e20 0x7f5c481cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c5056b640 10 monclient(hunting): picked mon.noname-c con 0x7f5c480a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:35.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c5056b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c480a4970 0x7f5c48124d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c5056b640 10 monclient(hunting): picked mon.noname-b con 0x7f5c480a4970 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:39:35.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c5056b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:35.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c5056b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5c48138860 con 0x7f5c4806f910 2024-09-11T01:39:35.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c5056b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:35.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c5056b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5c481386e0 con 0x7f5c480a4970 2024-09-11T01:39:35.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c5056b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:35.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c5056b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5c481389e0 con 0x7f5c480a2e20 2024-09-11T01:39:35.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c5056b640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:35.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c5056b640 10 monclient(hunting): authenticate will time out at 1382.365681s 2024-09-11T01:39:35.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c4e307640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c4806f910 0x7f5c481cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c4db06640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5c480a2e20 0x7f5c481cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c4e307640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c4806f910 0x7f5c481cd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40056/0 (socket says 172.21.15.12:40056) 2024-09-11T01:39:35.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c4e307640 1 -- 172.21.15.12:0/4066010190 learned_addr learned my addr 172.21.15.12:0/4066010190 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:39:35.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c4e307640 10 monclient(hunting): get_auth_request con 0x7f5c4806f910 auth_method 0 2024-09-11T01:39:35.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c4e307640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:35.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c4e307640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:35.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c4e307640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:35.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c4db06640 10 monclient(hunting): get_auth_request con 0x7f5c480a2e20 auth_method 0 2024-09-11T01:39:35.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c4db06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:35.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c4db06640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:35.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c4db06640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:35.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c4db06640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:35.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c4db06640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:35.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c4db06640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:35.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c4e307640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:35.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c4e307640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:35.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c4e307640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:35.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c4eb08640 1 --2- 172.21.15.12:0/4066010190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c480a4970 0x7f5c48124d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c4eb08640 10 monclient(hunting): get_auth_request con 0x7f5c480a4970 auth_method 0 2024-09-11T01:39:35.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c4eb08640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:35.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c4eb08640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:35.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.923+0000 7f5c4eb08640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:35.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.924+0000 7f5c4eb08640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:35.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.924+0000 7f5c4eb08640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:35.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.924+0000 7f5c4eb08640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:35.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.924+0000 7f5c4db06640 10 monclient(hunting): handle_auth_done global_id 114125 payload 824 2024-09-11T01:39:35.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.924+0000 7f5c4db06640 1 -- 172.21.15.12:0/4066010190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c480a4970 msgr2=0x7f5c48124d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:35.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.924+0000 7f5c4db06640 1 --2- 172.21.15.12:0/4066010190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c480a4970 0x7f5c48124d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.924+0000 7f5c4db06640 1 -- 172.21.15.12:0/4066010190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c4806f910 msgr2=0x7f5c481cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:35.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.924+0000 7f5c4db06640 1 --2- 172.21.15.12:0/4066010190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c4806f910 0x7f5c481cd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.924+0000 7f5c4db06640 10 monclient: _finish_hunting 0 2024-09-11T01:39:35.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.924+0000 7f5c4db06640 1 monclient: found mon.noname-c 2024-09-11T01:39:35.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.924+0000 7f5c4db06640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:35.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.924+0000 7f5c4db06640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:39:35.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.924+0000 7f5c4db06640 1 -- 172.21.15.12:0/4066010190 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c481267d0 con 0x7f5c480a2e20 2024-09-11T01:39:35.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.924+0000 7f5c4e307640 1 --2- 172.21.15.12:0/4066010190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5c4806f910 0x7f5c481cd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:39:35.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.926+0000 7f5c4db06640 1 -- 172.21.15.12:0/4066010190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5c480a2e20 msgr2=0x7f5c481cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:35.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.926+0000 7f5c4db06640 1 --2- 172.21.15.12:0/4066010190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5c480a2e20 0x7f5c481cd9b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5c480a4d80 tx=0x7f5c380046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:35.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.926+0000 7f5c3effd640 1 -- 172.21.15.12:0/4066010190 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c38037040 con 0x7f5c480a2e20 2024-09-11T01:39:35.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.926+0000 7f5c3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:35.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.926+0000 7f5c3effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:39:35.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.926+0000 7f5c3effd640 10 monclient: dump: 2024-09-11T01:39:35.939 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:35.939 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:35.939 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.939 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.939 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:35.939 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.939 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:35.939 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:35.939 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:35.939 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.926+0000 7f5c3effd640 10 monclient: _finish_auth 0 2024-09-11T01:39:35.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.926+0000 7f5c3effd640 10 monclient: _check_auth_tickets 2024-09-11T01:39:35.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.926+0000 7f5c3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:35.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.926+0000 7f5c3effd640 1 -- 172.21.15.12:0/4066010190 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c38042070 con 0x7f5c480a2e20 2024-09-11T01:39:35.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.926+0000 7f5c5056b640 5 monclient: authenticate success, global_id 114125 2024-09-11T01:39:35.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.926+0000 7f5c5056b640 10 monclient: _renew_subs 2024-09-11T01:39:35.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.926+0000 7f5c5056b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:39:35.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.926+0000 7f5c5056b640 1 -- 172.21.15.12:0/4066010190 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c480a6220 con 0x7f5c480a2e20 2024-09-11T01:39:35.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.926+0000 7f5c3effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:35.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.926+0000 7f5c3effd640 1 -- 172.21.15.12:0/4066010190 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3803b960 con 0x7f5c480a2e20 2024-09-11T01:39:35.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.926+0000 7f5c3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:35.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.926+0000 7f5c3effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:39:35.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.926+0000 7f5c3effd640 10 monclient: dump: 2024-09-11T01:39:35.940 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:35.940 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:35.941 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.941 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.941 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:35.941 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.941 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:35.941 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:35.941 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:35.941 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.926+0000 7f5c5056b640 10 monclient: _renew_subs 2024-09-11T01:39:35.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.926+0000 7f5c4cb04640 4 set_mon_vals no callback set 2024-09-11T01:39:35.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.926+0000 7f5c5056b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:39:35.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.926+0000 7f5c5056b640 1 -- 172.21.15.12:0/4066010190 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c48128b30 con 0x7f5c480a2e20 2024-09-11T01:39:35.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.927+0000 7f5c5056b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:35.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.927+0000 7f5c5056b640 1 --2- 172.21.15.12:0/4066010190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c48125250 0x7f5c48125630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.927+0000 7f5c5056b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:35.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.927+0000 7f5c5056b640 1 -- 172.21.15.12:0/4066010190 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4806a060 con 0x7f5c48125250 2024-09-11T01:39:35.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.927+0000 7f5c3effd640 1 -- 172.21.15.12:0/4066010190 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f5c3803a070 con 0x7f5c480a2e20 2024-09-11T01:39:35.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.927+0000 7f5c4e307640 1 --2- 172.21.15.12:0/4066010190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c48125250 0x7f5c48125630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.927+0000 7f5c4e307640 10 monclient: get_auth_request con 0x7f5c48125250 auth_method 0 2024-09-11T01:39:35.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.927+0000 7f5c4e307640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:35.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.927+0000 7f5c4e307640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:35.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.927+0000 7f5c4e307640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:35.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.927+0000 7f5c4e307640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:35.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.927+0000 7f5c4e307640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:35.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.927+0000 7f5c4e307640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:35.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.927+0000 7f5c3effd640 1 --2- 172.21.15.12:0/4066010190 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5c2c03e210 0x7f5c2c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.927+0000 7f5c3effd640 1 -- 172.21.15.12:0/4066010190 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(344..344 src has 186..344) ==== 4811+0+0 (secure 0 0 0) 0x7f5c38072f30 con 0x7f5c480a2e20 2024-09-11T01:39:35.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.927+0000 7f5c4e307640 10 monclient(hunting): handle_auth_done global_id 104179 payload 824 2024-09-11T01:39:35.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.927+0000 7f5c4e307640 1 -- 172.21.15.12:0/4066010190 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5c48074530 con 0x7f5c48125250 2024-09-11T01:39:35.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.928+0000 7f5c4eb08640 1 --2- 172.21.15.12:0/4066010190 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5c2c03e210 0x7f5c2c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.928+0000 7f5c4eb08640 10 monclient: get_auth_request con 0x7f5c2c03e210 auth_method 0 2024-09-11T01:39:35.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.930+0000 7f5c4e307640 1 -- 172.21.15.12:0/4066010190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c48125250 msgr2=0x7f5c48125630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:35.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.930+0000 7f5c4e307640 1 --2- 172.21.15.12:0/4066010190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c48125250 0x7f5c48125630 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5c44003f20 tx=0x7f5c44004410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:35.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.930+0000 7f5c3effd640 1 -- 172.21.15.12:0/4066010190 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c44019020 con 0x7f5c48125250 2024-09-11T01:39:35.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.930+0000 7f5c3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:35.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.930+0000 7f5c3effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:39:35.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.930+0000 7f5c3effd640 10 monclient: dump: 2024-09-11T01:39:35.945 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:35.945 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:35.945 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.945 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.945 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:35.945 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.946 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:35.946 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:35.946 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:35.946 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.930+0000 7f5c4eb08640 1 --2- 172.21.15.12:0/4066010190 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5c2c03e210 0x7f5c2c0406d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5c4806fd60 tx=0x7f5c40007db0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:35.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.930+0000 7f5c3effd640 1 -- 172.21.15.12:0/4066010190 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5c44015b40 con 0x7f5c48125250 2024-09-11T01:39:35.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.930+0000 7f5c3effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:35.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.930+0000 7f5c3effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:39:35.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.930+0000 7f5c3effd640 1 -- 172.21.15.12:0/4066010190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c48125250 msgr2=0x7f5c48125630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:35.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.930+0000 7f5c3effd640 1 --2- 172.21.15.12:0/4066010190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c48125250 0x7f5c48125630 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5c44003f20 tx=0x7f5c44004410 comp rx=0 tx=0).stop 2024-09-11T01:39:35.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.930+0000 7f5c3effd640 1 -- 172.21.15.12:0/4066010190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c48125250 msgr2=0x7f5c48125630 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:39:35.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.930+0000 7f5c3effd640 1 --2- 172.21.15.12:0/4066010190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c48125250 0x7f5c48125630 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5c44003f20 tx=0x7f5c44004410 comp rx=0 tx=0).stop 2024-09-11T01:39:35.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.936+0000 7f5c5056b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:39:35.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.936+0000 7f5c5056b640 1 --2- 172.21.15.12:0/4066010190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c48125b70 0x7f5c480a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:35.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.936+0000 7f5c5056b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:35.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.936+0000 7f5c5056b640 1 -- 172.21.15.12:0/4066010190 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4806ded0 con 0x7f5c48125b70 2024-09-11T01:39:35.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.937+0000 7f5c4e307640 1 --2- 172.21.15.12:0/4066010190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c48125b70 0x7f5c480a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:35.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.937+0000 7f5c4e307640 10 monclient: get_auth_request con 0x7f5c48125b70 auth_method 0 2024-09-11T01:39:35.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.937+0000 7f5c4e307640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:35.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.937+0000 7f5c4e307640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:35.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.937+0000 7f5c4e307640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:35.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.937+0000 7f5c4e307640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:35.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.937+0000 7f5c4e307640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:35.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.937+0000 7f5c4e307640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:35.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.937+0000 7f5c4e307640 10 monclient(hunting): handle_auth_done global_id 104182 payload 824 2024-09-11T01:39:35.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.937+0000 7f5c4e307640 1 -- 172.21.15.12:0/4066010190 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5c48069950 con 0x7f5c48125b70 2024-09-11T01:39:35.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.940+0000 7f5c4e307640 1 -- 172.21.15.12:0/4066010190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c48125b70 msgr2=0x7f5c480a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:35.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.940+0000 7f5c4e307640 1 --2- 172.21.15.12:0/4066010190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c48125b70 0x7f5c480a0830 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5c44003f20 tx=0x7f5c44004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:35.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.940+0000 7f5c3effd640 1 -- 172.21.15.12:0/4066010190 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c44019030 con 0x7f5c48125b70 2024-09-11T01:39:35.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.940+0000 7f5c3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:35.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.940+0000 7f5c3effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:39:35.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.940+0000 7f5c3effd640 10 monclient: dump: 2024-09-11T01:39:35.948 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:35.948 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:35.948 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.948 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:35.948 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:35.948 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:35.948 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:35.948 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:35.948 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:35.948 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:35.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.940+0000 7f5c3effd640 1 -- 172.21.15.12:0/4066010190 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7f5c44020220 con 0x7f5c48125b70 2024-09-11T01:39:35.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.940+0000 7f5c3effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:39:35.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.940+0000 7f5c3effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:39:35.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.940+0000 7f5c3effd640 1 -- 172.21.15.12:0/4066010190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c48125b70 msgr2=0x7f5c480a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:35.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.940+0000 7f5c3effd640 1 --2- 172.21.15.12:0/4066010190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c48125b70 0x7f5c480a0830 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5c44003f20 tx=0x7f5c44004a40 comp rx=0 tx=0).stop 2024-09-11T01:39:35.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.940+0000 7f5c3effd640 1 -- 172.21.15.12:0/4066010190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c48125b70 msgr2=0x7f5c480a0830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:39:35.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.940+0000 7f5c3effd640 1 --2- 172.21.15.12:0/4066010190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c48125b70 0x7f5c480a0830 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5c44003f20 tx=0x7f5c44004a40 comp rx=0 tx=0).stop 2024-09-11T01:39:35.949 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:39:35.949 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:39:35.949 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:39:35.949 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:39:35.949 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 100, 2024-09-11T01:39:35.950 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 76434, 2024-09-11T01:39:35.950 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:39:35.950 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:39:35.950 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:39:35.950 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:39:35.950 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:35.950 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:39:35.950 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:39:35.950 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:39:35.950 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:39:35.950 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:39:35.950 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:39:35.950 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:39:35.950 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:39:35.950 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:39:35.951 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:39:35.951 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:39:35.951 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:39:35.951 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:39:35.951 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:39:35.951 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:39:35.951 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:35.951 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:39:35.951 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:39:35.951 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:39:35.951 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:39:35.951 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:39:35.951 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:39:35.951 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:39:35.951 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:39:35.951 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:39:35.952 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:39:35.952 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:39:35.952 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:39:35.952 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:39:35.952 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:35.952 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.952 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:39:35.952 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:39:35.952 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:39:35.952 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:39:35.952 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:39:35.952 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:39:35.952 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:39:35.952 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:39:35.952 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:39:35.953 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:39:35.953 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:39:35.953 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:39:35.953 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:39:35.953 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:39:35.953 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:39:35.953 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:39:35.953 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:39:35.953 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:39:35.953 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:39:35.953 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:39:35.953 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:39:35.953 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:39:35.953 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:39:35.953 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:39:35.954 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:39:35.954 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:39:35.954 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:39:35.954 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:39:35.954 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:35.954 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:39:35.954 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.954 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:39:35.954 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.954 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:39:35.954 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:39:35.954 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:39:35.954 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:39:35.954 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.954 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:39:35.955 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:39:35.955 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:35.955 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.955 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.955 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:39:35.955 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:39:35.955 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:35.955 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:35.955 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:35.955 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.955 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:39:35.955 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:39:35.956 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:39:35.956 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:39:35.956 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:39:35.956 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.956 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.956 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:39:35.956 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:39:35.956 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:39:35.956 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:39:35.956 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.956 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:39:35.956 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:39:35.956 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:35.956 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.956 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.957 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:39:35.957 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:39:35.957 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:35.957 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:35.957 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:35.957 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.957 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:39:35.957 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:39:35.957 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:39:35.957 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:39:35.957 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:39:35.957 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.957 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.957 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:39:35.958 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:39:35.958 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:39:35.958 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:39:35.958 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.958 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:39:35.958 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:39:35.958 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:35.958 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.958 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.958 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:39:35.958 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:39:35.958 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:35.958 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:35.958 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:35.958 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.959 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:39:35.959 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:39:35.959 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:39:35.959 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:39:35.959 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:39:35.959 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:35.959 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:35.959 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.959 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:39:35.959 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:39:35.959 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.959 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:39:35.959 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:39:35.960 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:39:35.960 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:35.960 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:35.960 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:39:35.960 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.960 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:39:35.960 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:39:35.960 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:39:35.960 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:35.960 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:35.960 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:39:35.960 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.960 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:39:35.960 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:39:35.961 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:39:35.961 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:35.961 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:35.961 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:39:35.961 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:35.961 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:39:35.961 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:39:35.961 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:39:35.961 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:35.961 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:35.961 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:35.961 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:39:35.961 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:39:35.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.940+0000 7f5c5056b640 1 -- 172.21.15.12:0/4066010190 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5c2c03e210 msgr2=0x7f5c2c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:35.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.940+0000 7f5c5056b640 1 --2- 172.21.15.12:0/4066010190 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5c2c03e210 0x7f5c2c0406d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5c4806fd60 tx=0x7f5c40007db0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.940+0000 7f5c5056b640 10 monclient: shutdown 2024-09-11T01:39:35.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.940+0000 7f5c5056b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:35.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.940+0000 7f5c5056b640 1 -- 172.21.15.12:0/4066010190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5c480a2e20 msgr2=0x7f5c481cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:35.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.940+0000 7f5c5056b640 1 --2- 172.21.15.12:0/4066010190 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5c480a2e20 0x7f5c481cd9b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5c480a4d80 tx=0x7f5c380046c0 comp rx=0 tx=0).stop 2024-09-11T01:39:35.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.940+0000 7f5c4eb08640 1 -- 172.21.15.12:0/4066010190 reap_dead start 2024-09-11T01:39:35.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.940+0000 7f5c4eb08640 1 -- 172.21.15.12:0/4066010190 reap_dead start 2024-09-11T01:39:35.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.940+0000 7f5c5056b640 1 -- 172.21.15.12:0/4066010190 shutdown_connections 2024-09-11T01:39:35.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.940+0000 7f5c5056b640 1 -- 172.21.15.12:0/4066010190 >> 172.21.15.12:0/4066010190 conn(0x7f5c480a8600 msgr2=0x7f5c48072e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:35.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.940+0000 7f5c5056b640 1 -- 172.21.15.12:0/4066010190 shutdown_connections 2024-09-11T01:39:35.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:35.940+0000 7f5c5056b640 1 -- 172.21.15.12:0/4066010190 wait complete. 2024-09-11T01:39:35.964 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:39:35.964 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:39:35.964 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:39:36.043 INFO:tasks.rados.rados.0.smithi071.stdout:1519: done (1 left) 2024-09-11T01:39:36.043 INFO:tasks.rados.rados.0.smithi071.stdout:1521: done (0 left) 2024-09-11T01:39:36.043 INFO:tasks.rados.rados.0.smithi071.stdout:1522: read oid 9 snap -1 2024-09-11T01:39:36.044 INFO:tasks.rados.rados.0.smithi071.stdout:1522: expect deleted 2024-09-11T01:39:36.044 INFO:tasks.rados.rados.0.smithi071.stdout:1523: delete oid 35 current snap is 163 2024-09-11T01:39:36.049 INFO:tasks.rados.rados.0.smithi071.stdout:1522: done (1 left) 2024-09-11T01:39:36.049 INFO:tasks.rados.rados.0.smithi071.stdout:1523: done (0 left) 2024-09-11T01:39:36.049 INFO:tasks.rados.rados.0.smithi071.stdout:1524: delete oid 49 current snap is 163 2024-09-11T01:39:36.054 INFO:tasks.rados.rados.0.smithi071.stdout:1524: done (0 left) 2024-09-11T01:39:36.054 INFO:tasks.rados.rados.0.smithi071.stdout:1525: snap_create 2024-09-11T01:39:36.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.065+0000 7f5e0f5ad640 10 monclient: get_monmap_and_config 2024-09-11T01:39:36.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.065+0000 7f5e0f5ad640 10 monclient: build_initial_monmap 2024-09-11T01:39:36.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.065+0000 7f5e0f5ad640 10 monclient: monmap: 2024-09-11T01:39:36.067 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:36.067 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:36.067 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:39:36.066695+0000 2024-09-11T01:39:36.067 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:39:36.066695+0000 2024-09-11T01:39:36.067 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:36.067 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:36.068 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:39:36.068 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:39:36.068 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:39:36.068 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:39:36.068 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:39:36.068 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:36.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.066+0000 7f5e0f5ad640 1 Processor -- start 2024-09-11T01:39:36.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.066+0000 7f5e0f5ad640 1 -- start start 2024-09-11T01:39:36.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.066+0000 7f5e0f5ad640 10 monclient: init 2024-09-11T01:39:36.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0f5ad640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:36.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0f5ad640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-11T01:39:36.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0f5ad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e08131850 addr v1:172.21.15.71:6789/0 2024-09-11T01:39:36.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0f5ad640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e080a2fb0 addr v1:172.21.15.12:6789/0 2024-09-11T01:39:36.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0f5ad640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e0809df40 0x7f5e0813d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:36.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0f5ad640 10 monclient(hunting): picked mon.noname-b con 0x7f5e0809df40 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:36.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0f5ad640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e0809cbe0 con 0x7f5e080a2fb0 2024-09-11T01:39:36.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0f5ad640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e0809cec0 con 0x7f5e08131850 2024-09-11T01:39:36.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0f5ad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:36.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0f5ad640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5e08132340 con 0x7f5e0809df40 2024-09-11T01:39:36.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0f5ad640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:36.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0f5ad640 10 monclient(hunting): authenticate will time out at 1382.509295s 2024-09-11T01:39:36.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0cb48640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5e080a2fb0 0x7f5e08139d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37854/0 (socket says 172.21.15.12:37854) 2024-09-11T01:39:36.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0cb48640 1 -- 172.21.15.12:0/1576862879 learned_addr learned my addr 172.21.15.12:0/1576862879 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:39:36.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0db4a640 1 --2- 172.21.15.12:0/1576862879 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e0809df40 0x7f5e0813d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:36.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0db4a640 10 monclient(hunting): get_auth_request con 0x7f5e0809df40 auth_method 0 2024-09-11T01:39:36.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0db4a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:36.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0db4a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:36.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0db4a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:36.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0db4a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:36.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0db4a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:36.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0db4a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:36.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0db4a640 10 monclient(hunting): handle_auth_done global_id 114128 payload 274 2024-09-11T01:39:36.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0db4a640 1 -- 172.21.15.12:0/1576862879 >> v1:172.21.15.71:6789/0 conn(0x7f5e08131850 legacy=0x7f5e08131c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:39:36.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0db4a640 1 -- 172.21.15.12:0/1576862879 >> v1:172.21.15.12:6789/0 conn(0x7f5e080a2fb0 legacy=0x7f5e08139d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:39:36.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0db4a640 10 monclient: _finish_hunting 0 2024-09-11T01:39:36.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0db4a640 1 monclient: found mon.noname-b 2024-09-11T01:39:36.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0db4a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:36.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0db4a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:39:36.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.067+0000 7f5e0db4a640 1 -- 172.21.15.12:0/1576862879 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e08131520 con 0x7f5e0809df40 2024-09-11T01:39:36.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.070+0000 7f5e0db4a640 1 -- 172.21.15.12:0/1576862879 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e0809df40 msgr2=0x7f5e0813d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:36.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.070+0000 7f5e0db4a640 1 --2- 172.21.15.12:0/1576862879 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e0809df40 0x7f5e0813d470 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5e00009dc0 tx=0x7f5e0002a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7ceae9ac51a417a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:36.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.070+0000 7f5dfffff640 1 -- 172.21.15.12:0/1576862879 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e00007a30 con 0x7f5e0809df40 2024-09-11T01:39:36.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.070+0000 7f5dfffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:36.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.070+0000 7f5dfffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:39:36.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.070+0000 7f5dfffff640 10 monclient: dump: 2024-09-11T01:39:36.076 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:36.077 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:36.077 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:36.077 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:36.077 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:36.077 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:36.077 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:36.077 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:36.077 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:36.077 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:36.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.070+0000 7f5dfffff640 10 monclient: _finish_auth 0 2024-09-11T01:39:36.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.070+0000 7f5dfffff640 10 monclient: _check_auth_tickets 2024-09-11T01:39:36.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.070+0000 7f5dfffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:36.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.070+0000 7f5dfffff640 1 -- 172.21.15.12:0/1576862879 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e00007bd0 con 0x7f5e0809df40 2024-09-11T01:39:36.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.070+0000 7f5dfffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:36.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.070+0000 7f5dfffff640 1 -- 172.21.15.12:0/1576862879 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e00031400 con 0x7f5e0809df40 2024-09-11T01:39:36.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.070+0000 7f5e0f5ad640 5 monclient: authenticate success, global_id 114128 2024-09-11T01:39:36.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.070+0000 7f5e0f5ad640 10 monclient: get_monmap_and_config success 2024-09-11T01:39:36.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.070+0000 7f5e0f5ad640 4 set_mon_vals no callback set 2024-09-11T01:39:36.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.070+0000 7f5dfffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:36.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.070+0000 7f5dfffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:39:36.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.070+0000 7f5dfffff640 10 monclient: dump: 2024-09-11T01:39:36.078 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:36.078 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:36.078 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:36.078 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:36.078 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:36.078 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:36.078 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:36.079 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:36.079 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:36.079 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:36.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.071+0000 7f5e0f5ad640 10 monclient: shutdown 2024-09-11T01:39:36.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.071+0000 7f5e0f5ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:36.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.071+0000 7f5e0f5ad640 1 -- 172.21.15.12:0/1576862879 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e0809df40 msgr2=0x7f5e0813d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:36.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.071+0000 7f5e0f5ad640 1 --2- 172.21.15.12:0/1576862879 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e0809df40 0x7f5e0813d470 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5e00009dc0 tx=0x7f5e0002a480 comp rx=0 tx=0).stop 2024-09-11T01:39:36.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.071+0000 7f5e0f5ad640 1 -- 172.21.15.12:0/1576862879 shutdown_connections 2024-09-11T01:39:36.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.071+0000 7f5e0f5ad640 1 --2- 172.21.15.12:0/1576862879 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e0809df40 0x7f5e0813d470 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:36.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.071+0000 7f5e0f5ad640 1 -- 172.21.15.12:0/1576862879 >> 172.21.15.12:0/1576862879 conn(0x7f5e080a3e30 msgr2=0x7f5e080a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:36.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.071+0000 7f5e0f5ad640 1 -- 172.21.15.12:0/1576862879 shutdown_connections 2024-09-11T01:39:36.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.071+0000 7f5e0f5ad640 1 -- 172.21.15.12:0/1576862879 wait complete. 2024-09-11T01:39:36.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.071+0000 7f5e0f5ad640 10 monclient: build_initial_monmap 2024-09-11T01:39:36.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.071+0000 7f5e0f5ad640 10 monclient: monmap: 2024-09-11T01:39:36.079 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:39:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:39:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:39:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:39:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:39:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:39:36.080 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0f5ad640 1 Processor -- start 2024-09-11T01:39:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0f5ad640 1 -- start start 2024-09-11T01:39:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0f5ad640 10 monclient: init 2024-09-11T01:39:36.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0f5ad640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0f5ad640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:39:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0f5ad640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e0809df40 0x7f5e0809a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0f5ad640 10 monclient(hunting): picked mon.noname-a con 0x7f5e0809df40 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:39:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0f5ad640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e080a2fb0 0x7f5e0809a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0f5ad640 10 monclient(hunting): picked mon.noname-b con 0x7f5e080a2fb0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:39:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0f5ad640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e08131850 0x7f5e08132610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0f5ad640 10 monclient(hunting): picked mon.noname-c con 0x7f5e08131850 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0f5ad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0f5ad640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0813dd40 con 0x7f5e0809df40 2024-09-11T01:39:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0f5ad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0f5ad640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0813dbc0 con 0x7f5e080a2fb0 2024-09-11T01:39:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0f5ad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:36.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0f5ad640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5e0813dec0 con 0x7f5e08131850 2024-09-11T01:39:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0f5ad640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0f5ad640 10 monclient(hunting): authenticate will time out at 1382.514848s 2024-09-11T01:39:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0db4a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e08131850 0x7f5e08132610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0d349640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e0809df40 0x7f5e0809a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0d349640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e0809df40 0x7f5e0809a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40062/0 (socket says 172.21.15.12:40062) 2024-09-11T01:39:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0db4a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e08131850 0x7f5e08132610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:44654/0 (socket says 172.21.15.12:44654) 2024-09-11T01:39:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0d349640 1 -- 172.21.15.12:0/3210396349 learned_addr learned my addr 172.21.15.12:0/3210396349 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:39:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0d349640 10 monclient(hunting): get_auth_request con 0x7f5e0809df40 auth_method 0 2024-09-11T01:39:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0d349640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0d349640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0d349640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0db4a640 10 monclient(hunting): get_auth_request con 0x7f5e08131850 auth_method 0 2024-09-11T01:39:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.072+0000 7f5e0db4a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:36.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.073+0000 7f5e0db4a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:36.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.073+0000 7f5e0db4a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:36.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.073+0000 7f5e0cb48640 1 --2- 172.21.15.12:0/3210396349 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e080a2fb0 0x7f5e0809a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:36.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.073+0000 7f5e0d349640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:36.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.073+0000 7f5e0d349640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:36.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.073+0000 7f5e0d349640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:36.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.073+0000 7f5e0cb48640 10 monclient(hunting): get_auth_request con 0x7f5e080a2fb0 auth_method 0 2024-09-11T01:39:36.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.073+0000 7f5e0cb48640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:36.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.073+0000 7f5e0cb48640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:36.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.073+0000 7f5e0cb48640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:36.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.073+0000 7f5e0db4a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:36.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.073+0000 7f5e0db4a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:36.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.073+0000 7f5e0db4a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:36.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.073+0000 7f5e0cb48640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:36.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.073+0000 7f5e0cb48640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:36.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.073+0000 7f5e0cb48640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:36.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.073+0000 7f5e0d349640 10 monclient(hunting): handle_auth_done global_id 94293 payload 824 2024-09-11T01:39:36.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.073+0000 7f5e0d349640 1 -- 172.21.15.12:0/3210396349 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e08131850 msgr2=0x7f5e08132610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:36.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.073+0000 7f5e0d349640 1 --2- 172.21.15.12:0/3210396349 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e08131850 0x7f5e08132610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:36.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.073+0000 7f5e0d349640 1 -- 172.21.15.12:0/3210396349 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e080a2fb0 msgr2=0x7f5e0809a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:36.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.073+0000 7f5e0d349640 1 --2- 172.21.15.12:0/3210396349 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e080a2fb0 0x7f5e0809a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:36.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.073+0000 7f5e0d349640 10 monclient: _finish_hunting 0 2024-09-11T01:39:36.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.073+0000 7f5e0d349640 1 monclient: found mon.noname-a 2024-09-11T01:39:36.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.073+0000 7f5e0d349640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:36.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.073+0000 7f5e0d349640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:39:36.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.073+0000 7f5e0d349640 1 -- 172.21.15.12:0/3210396349 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e081352e0 con 0x7f5e0809df40 2024-09-11T01:39:36.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.075+0000 7f5e0d349640 1 -- 172.21.15.12:0/3210396349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e0809df40 msgr2=0x7f5e0809a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:36.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.075+0000 7f5e0d349640 1 --2- 172.21.15.12:0/3210396349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e0809df40 0x7f5e0809a3d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5e0400c430 tx=0x7f5e0400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:36.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.075+0000 7f5dfdffb640 1 -- 172.21.15.12:0/3210396349 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e040119a0 con 0x7f5e0809df40 2024-09-11T01:39:36.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.075+0000 7f5dfdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:36.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.075+0000 7f5dfdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:39:36.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.075+0000 7f5dfdffb640 10 monclient: dump: 2024-09-11T01:39:36.087 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:36.087 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:36.087 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:36.087 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:36.087 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:36.087 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:36.087 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:36.087 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:36.087 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:36.087 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:36.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.075+0000 7f5dfdffb640 10 monclient: _finish_auth 0 2024-09-11T01:39:36.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.075+0000 7f5dfdffb640 10 monclient: _check_auth_tickets 2024-09-11T01:39:36.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.075+0000 7f5dfdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:36.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.075+0000 7f5dfdffb640 1 -- 172.21.15.12:0/3210396349 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e04017070 con 0x7f5e0809df40 2024-09-11T01:39:36.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.075+0000 7f5e0f5ad640 5 monclient: authenticate success, global_id 94293 2024-09-11T01:39:36.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.075+0000 7f5e0f5ad640 10 monclient: _renew_subs 2024-09-11T01:39:36.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.075+0000 7f5e0f5ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:36.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.075+0000 7f5e0f5ad640 1 -- 172.21.15.12:0/3210396349 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e0809e5e0 con 0x7f5e0809df40 2024-09-11T01:39:36.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.075+0000 7f5dfdffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:36.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.075+0000 7f5dfdffb640 1 -- 172.21.15.12:0/3210396349 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e04011d80 con 0x7f5e0809df40 2024-09-11T01:39:36.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.075+0000 7f5dfdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:36.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.075+0000 7f5dfdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:39:36.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.075+0000 7f5dfdffb640 10 monclient: dump: 2024-09-11T01:39:36.089 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:36.089 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:36.089 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:36.089 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:36.089 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:36.089 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:36.089 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:36.089 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:36.089 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:36.089 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:36.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.075+0000 7f5dff7fe640 4 set_mon_vals no callback set 2024-09-11T01:39:36.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.076+0000 7f5e0f5ad640 10 monclient: _renew_subs 2024-09-11T01:39:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.076+0000 7f5e0f5ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.076+0000 7f5e0f5ad640 1 -- 172.21.15.12:0/3210396349 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e0809eb30 con 0x7f5e0809df40 2024-09-11T01:39:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.076+0000 7f5dfdffb640 1 -- 172.21.15.12:0/3210396349 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f5e0401c460 con 0x7f5e0809df40 2024-09-11T01:39:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.076+0000 7f5dfdffb640 1 --2- 172.21.15.12:0/3210396349 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5dec03e240 0x7f5dec040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.076+0000 7f5dfdffb640 1 -- 172.21.15.12:0/3210396349 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(345..345 src has 186..345) ==== 4755+0+0 (secure 0 0 0) 0x7f5e04052910 con 0x7f5e0809df40 2024-09-11T01:39:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.077+0000 7f5e0f5ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.077+0000 7f5e0f5ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.077+0000 7f5e0f5ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.077+0000 7f5e0f5ad640 1 -- 172.21.15.12:0/3210396349 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e08131cb0 con 0x7f5e0809df40 2024-09-11T01:39:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.077+0000 7f5e0cb48640 1 --2- 172.21.15.12:0/3210396349 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5dec03e240 0x7f5dec040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.077+0000 7f5e0cb48640 10 monclient: get_auth_request con 0x7f5dec03e240 auth_method 0 2024-09-11T01:39:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.079+0000 7f5e0cb48640 1 --2- 172.21.15.12:0/3210396349 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5dec03e240 0x7f5dec040700 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5df8007c90 tx=0x7f5df8007be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:36.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.081+0000 7f5dfdffb640 1 -- 172.21.15.12:0/3210396349 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5e04021020 con 0x7f5e0809df40 2024-09-11T01:39:36.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.081+0000 7f5dfdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:36.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.081+0000 7f5dfdffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:39:36.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.343+0000 7f5e0f5ad640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:39:36.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.343+0000 7f5e0f5ad640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:39:36.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.343+0000 7f5e0f5ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:36.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.343+0000 7f5e0f5ad640 1 -- 172.21.15.12:0/3210396349 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f5e08130e50 con 0x7f5e0809df40 2024-09-11T01:39:36.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.344+0000 7f5dfdffb640 1 -- 172.21.15.12:0/3210396349 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f5e0401ac60 con 0x7f5e0809df40 2024-09-11T01:39:36.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.344+0000 7f5dfdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:39:36.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.344+0000 7f5dfdffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:39:36.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.345+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.345+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=11,kv=11,logm=59} crc {auth=3687803472,config=2586702516,health=1902400218,kv=1918975338,logm=3433096565}) 2024-09-11T01:39:36.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.345+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=18,config=2,health=12,kv=11,logm=57} crc {auth=2068495492,config=2586702517,health=3286790080,kv=1703855226,logm=2807535579}) 2024-09-11T01:39:36.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.345+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.345+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=11,kv=11,logm=59} crc {auth=3687803472,config=2586702516,health=1902400218,kv=1918975338,logm=3433096565}) 2024-09-11T01:39:36.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.345+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=18,config=2,health=12,kv=12,logm=56} crc {auth=3889384721,config=2586702516,health=3286790080,kv=3263697420,logm=2393851868}) 2024-09-11T01:39:36.348 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.346+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.348 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.346+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1886915774}) 2024-09-11T01:39:36.348 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.346+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3675360930}) 2024-09-11T01:39:36.348 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.346+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.348 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.346+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1886915774}) 2024-09-11T01:39:36.348 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.346+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1263131460}) 2024-09-11T01:39:36.349 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.347+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.349 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.347+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1447460612}) 2024-09-11T01:39:36.349 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.347+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1129701878}) 2024-09-11T01:39:36.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.347+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.347+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1447460612}) 2024-09-11T01:39:36.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.347+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2666726145}) 2024-09-11T01:39:36.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.348+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.348+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2840690884}) 2024-09-11T01:39:36.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.348+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2098746924}) 2024-09-11T01:39:36.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.348+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.348+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2840690884}) 2024-09-11T01:39:36.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.348+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1321074687}) 2024-09-11T01:39:36.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.348+0000 7f5e0f5ad640 1 -- 172.21.15.12:0/3210396349 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5dec03e240 msgr2=0x7f5dec040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:36.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.348+0000 7f5e0f5ad640 1 --2- 172.21.15.12:0/3210396349 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5dec03e240 0x7f5dec040700 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5df8007c90 tx=0x7f5df8007be0 comp rx=0 tx=0).stop 2024-09-11T01:39:36.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.348+0000 7f5e0f5ad640 10 monclient: shutdown 2024-09-11T01:39:36.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.348+0000 7f5e0f5ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:36.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.348+0000 7f5e0f5ad640 1 -- 172.21.15.12:0/3210396349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e0809df40 msgr2=0x7f5e0809a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:36.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.348+0000 7f5e0f5ad640 1 --2- 172.21.15.12:0/3210396349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e0809df40 0x7f5e0809a3d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5e0400c430 tx=0x7f5e0400c8f0 comp rx=0 tx=0).stop 2024-09-11T01:39:36.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.348+0000 7f5e0f5ad640 1 -- 172.21.15.12:0/3210396349 shutdown_connections 2024-09-11T01:39:36.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.348+0000 7f5e0f5ad640 1 --2- 172.21.15.12:0/3210396349 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e08131850 0x7f5e08132610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.348+0000 7f5e0f5ad640 1 --2- 172.21.15.12:0/3210396349 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e080a2fb0 0x7f5e0809a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.348+0000 7f5e0f5ad640 1 --2- 172.21.15.12:0/3210396349 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5dec03e240 0x7f5dec040700 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.348+0000 7f5e0f5ad640 1 --2- 172.21.15.12:0/3210396349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e0809df40 0x7f5e0809a3d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.348+0000 7f5e0f5ad640 1 -- 172.21.15.12:0/3210396349 >> 172.21.15.12:0/3210396349 conn(0x7f5e080a3e30 msgr2=0x7f5e08072f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.348+0000 7f5e0f5ad640 1 -- 172.21.15.12:0/3210396349 shutdown_connections 2024-09-11T01:39:36.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:36.348+0000 7f5e0f5ad640 1 -- 172.21.15.12:0/3210396349 wait complete. 2024-09-11T01:39:36.353 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.349+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.353 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.349+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3234110464}) 2024-09-11T01:39:36.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.349+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=787528914}) 2024-09-11T01:39:36.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.349+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.349+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3234110464}) 2024-09-11T01:39:36.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.349+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1934128875}) 2024-09-11T01:39:36.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.349+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.349+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=626453089}) 2024-09-11T01:39:36.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.349+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4216618128}) 2024-09-11T01:39:36.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.349+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.349+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=626453089}) 2024-09-11T01:39:36.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.349+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3798926506}) 2024-09-11T01:39:36.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.350+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.350+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1920145546}) 2024-09-11T01:39:36.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.350+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1994312739}) 2024-09-11T01:39:36.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.350+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.350+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1920145546}) 2024-09-11T01:39:36.355 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.350+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=156367849}) 2024-09-11T01:39:36.355 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.351+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.355 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.351+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938290056}) 2024-09-11T01:39:36.355 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.351+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3597806601}) 2024-09-11T01:39:36.355 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.351+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.355 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.351+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938290056}) 2024-09-11T01:39:36.355 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.351+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3490436197}) 2024-09-11T01:39:36.356 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.352+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.356 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.353+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1693873978}) 2024-09-11T01:39:36.356 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.353+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1720464682}) 2024-09-11T01:39:36.356 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.353+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.356 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.353+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1693873978}) 2024-09-11T01:39:36.356 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.353+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3793824578}) 2024-09-11T01:39:36.361 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:39:36.364 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.363+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.364 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.363+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2463025872}) 2024-09-11T01:39:36.365 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.363+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4172836424}) 2024-09-11T01:39:36.365 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.363+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.365 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.363+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2463025872}) 2024-09-11T01:39:36.365 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.363+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2975382623}) 2024-09-11T01:39:36.365 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.363+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.365 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.363+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2693320906}) 2024-09-11T01:39:36.365 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.363+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2435783680}) 2024-09-11T01:39:36.366 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.364+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.366 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.364+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2693320906}) 2024-09-11T01:39:36.366 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.364+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1094907270}) 2024-09-11T01:39:36.367 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.364+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.367 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.364+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=166779518}) 2024-09-11T01:39:36.367 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.364+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3855521347}) 2024-09-11T01:39:36.367 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.364+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.367 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.364+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=166779518}) 2024-09-11T01:39:36.367 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.364+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2418782477}) 2024-09-11T01:39:36.367 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.365+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.367 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.365+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2663856020}) 2024-09-11T01:39:36.367 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.365+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3449905236}) 2024-09-11T01:39:36.367 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.365+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.367 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.365+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2663856020}) 2024-09-11T01:39:36.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.365+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=59523174}) 2024-09-11T01:39:36.369 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.366+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.369 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.366+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2310031562}) 2024-09-11T01:39:36.369 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.366+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1466438033}) 2024-09-11T01:39:36.369 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.366+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.370 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.366+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2310031562}) 2024-09-11T01:39:36.370 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.366+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=615150599}) 2024-09-11T01:39:36.370 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.366+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.370 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.366+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4025449823}) 2024-09-11T01:39:36.370 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.366+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3067814972}) 2024-09-11T01:39:36.370 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.366+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.370 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.366+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4025449823}) 2024-09-11T01:39:36.370 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.366+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=232847844}) 2024-09-11T01:39:36.370 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.367+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.370 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.367+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1449334564}) 2024-09-11T01:39:36.370 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.367+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=129748969}) 2024-09-11T01:39:36.370 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.367+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.370 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.367+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1449334564}) 2024-09-11T01:39:36.371 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.367+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2953930791}) 2024-09-11T01:39:36.371 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.368+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.371 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.368+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3830689505}) 2024-09-11T01:39:36.371 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.368+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3965116501}) 2024-09-11T01:39:36.371 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.368+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.371 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.368+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3830689505}) 2024-09-11T01:39:36.371 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.368+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=818848628}) 2024-09-11T01:39:36.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.872+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.872+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2077721255}) 2024-09-11T01:39:36.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.872+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3729246796}) 2024-09-11T01:39:36.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.872+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.875 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.872+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2077721255}) 2024-09-11T01:39:36.875 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.872+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4021602603}) 2024-09-11T01:39:36.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.873+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.873+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4028822129}) 2024-09-11T01:39:36.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.873+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1689691653}) 2024-09-11T01:39:36.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.873+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.873+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4028822129}) 2024-09-11T01:39:36.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.873+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1564247062}) 2024-09-11T01:39:36.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.874+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.874+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1112372474}) 2024-09-11T01:39:36.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.874+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1917585136}) 2024-09-11T01:39:36.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.874+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.874+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1112372474}) 2024-09-11T01:39:36.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.874+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3222777017}) 2024-09-11T01:39:36.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.875+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.875+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=257884937}) 2024-09-11T01:39:36.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.875+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3385032371}) 2024-09-11T01:39:36.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.875+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.875+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=257884937}) 2024-09-11T01:39:36.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.875+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1193993617}) 2024-09-11T01:39:36.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.876+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.876+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=288895266}) 2024-09-11T01:39:36.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.876+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3918921549}) 2024-09-11T01:39:36.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.876+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.876+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=288895266}) 2024-09-11T01:39:36.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.876+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1728381614}) 2024-09-11T01:39:36.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.877+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.877+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2180223716}) 2024-09-11T01:39:36.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.877+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4021663373}) 2024-09-11T01:39:36.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.877+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.877+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2180223716}) 2024-09-11T01:39:36.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.877+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1469736820}) 2024-09-11T01:39:36.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.877+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.877+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3710933586}) 2024-09-11T01:39:36.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.877+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=817874955}) 2024-09-11T01:39:36.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.877+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.877+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3710933586}) 2024-09-11T01:39:36.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.877+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1255898579}) 2024-09-11T01:39:36.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.878+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.878+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=586249443}) 2024-09-11T01:39:36.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.878+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1310756330}) 2024-09-11T01:39:36.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.878+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.878+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=586249443}) 2024-09-11T01:39:36.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.878+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=841784728}) 2024-09-11T01:39:36.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.879+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.879+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821320435}) 2024-09-11T01:39:36.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.879+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2738176252}) 2024-09-11T01:39:36.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.879+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.879+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821320435}) 2024-09-11T01:39:36.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.879+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2795825896}) 2024-09-11T01:39:36.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.880+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.880+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4146356279}) 2024-09-11T01:39:36.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.880+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=64503161}) 2024-09-11T01:39:36.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.880+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.880+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4146356279}) 2024-09-11T01:39:36.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.880+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2088098816}) 2024-09-11T01:39:36.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.880+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.880+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=94,mdsmap=2,mgr=4} crc {logm=3487746569,mdsmap=161958762,mgr=1944970001}) 2024-09-11T01:39:36.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.880+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=1839990004,mdsmap=3760429986,mgr=1944970001}) 2024-09-11T01:39:36.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.880+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.880+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=94,mdsmap=2,mgr=4} crc {logm=3487746569,mdsmap=161958762,mgr=1944970001}) 2024-09-11T01:39:36.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.880+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=95,mdsmap=3,mgr=2} crc {logm=4085897401,mdsmap=3760429986,mgr=2517664975}) 2024-09-11T01:39:36.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.881+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.881+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=2,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=56} crc {mgr=2655145342,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1863092138,mon_config_key=3715565326,monmap=1324671969,nvmeofgw=1596038039,osd_metadata=2942852742,osd_pg_creating=4032943021,osd_snap=2261667168}) 2024-09-11T01:39:36.885 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.881+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=54} crc {mgr=1417461417,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1756176861,mon_config_key=3330371543,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=4032943021,osd_snap=4116144077}) 2024-09-11T01:39:36.885 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.881+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.885 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.881+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=2,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=56} crc {mgr=2655145342,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1863092138,mon_config_key=3715565326,monmap=1324671969,nvmeofgw=1596038039,osd_metadata=2942852742,osd_pg_creating=4032943021,osd_snap=2261667168}) 2024-09-11T01:39:36.885 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.881+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=55} crc {mgr=1417461417,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=3209233126,mon_config_key=3330371543,monmap=1324671969,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=4032943021,osd_snap=1431806768}) 2024-09-11T01:39:36.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.882+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.882+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=9,osdmap=91} crc {osd_snap=1014424353,osdmap=4161319571}) 2024-09-11T01:39:36.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.882+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=12,osdmap=88} crc {osd_snap=2476221015,osdmap=3683670602}) 2024-09-11T01:39:36.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.882+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.882+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=9,osdmap=91} crc {osd_snap=1014424353,osdmap=4161319571}) 2024-09-11T01:39:36.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.882+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=11,osdmap=89} crc {osd_snap=1336238632,osdmap=3404578522}) 2024-09-11T01:39:36.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.884+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.884+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1066461817}) 2024-09-11T01:39:36.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.884+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1646449122}) 2024-09-11T01:39:36.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.884+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.884+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1066461817}) 2024-09-11T01:39:36.887 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.884+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3913835248}) 2024-09-11T01:39:36.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.888+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.888+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=56} crc {osdmap=1046948201}) 2024-09-11T01:39:36.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.888+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=53} crc {osdmap=3106470480}) 2024-09-11T01:39:36.890 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.888+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:39:36.890 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.888+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=56} crc {osdmap=1046948201}) 2024-09-11T01:39:36.890 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:39:36.888+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=56} crc {osdmap=2588097798}) 2024-09-11T01:39:37.045 INFO:tasks.rados.rados.0.smithi071.stdout:1525: done (0 left) 2024-09-11T01:39:37.045 INFO:tasks.rados.rados.0.smithi071.stdout:1526: delete oid 11 current snap is 164 2024-09-11T01:39:37.049 INFO:tasks.rados.rados.0.smithi071.stdout:1526: done (0 left) 2024-09-11T01:39:37.049 INFO:tasks.rados.rados.0.smithi071.stdout:1527: write oid 39 current snap is 164 2024-09-11T01:39:37.049 INFO:tasks.rados.rados.0.smithi071.stdout:1527: seq_num 574 ranges {447218=719401,1889370=660531,2963386=614253} 2024-09-11T01:39:37.055 INFO:tasks.rados.rados.0.smithi071.stdout:1527: writing smithi07134777-39 from 447218 to 1166619 tid 1 2024-09-11T01:39:37.062 INFO:tasks.rados.rados.0.smithi071.stdout:1527: writing smithi07134777-39 from 1889370 to 2549901 tid 2 2024-09-11T01:39:37.068 INFO:tasks.rados.rados.0.smithi071.stdout:1527: writing smithi07134777-39 from 2963386 to 3577639 tid 3 2024-09-11T01:39:37.068 INFO:tasks.rados.rados.0.smithi071.stdout:1528: snap_remove snap 160 2024-09-11T01:39:37.362 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:39:37.363 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:39:37.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.475+0000 7fb10d1bb640 10 monclient: get_monmap_and_config 2024-09-11T01:39:37.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.475+0000 7fb10d1bb640 10 monclient: build_initial_monmap 2024-09-11T01:39:37.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.475+0000 7fb10d1bb640 10 monclient: monmap: 2024-09-11T01:39:37.477 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:37.477 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:37.477 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:39:37.475910+0000 2024-09-11T01:39:37.477 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:39:37.475910+0000 2024-09-11T01:39:37.477 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:37.477 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:37.477 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:39:37.477 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:39:37.477 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:39:37.477 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:39:37.477 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:39:37.477 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:37.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.475+0000 7fb10d1bb640 1 Processor -- start 2024-09-11T01:39:37.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.475+0000 7fb10d1bb640 1 -- start start 2024-09-11T01:39:37.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.475+0000 7fb10d1bb640 10 monclient: init 2024-09-11T01:39:37.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10d1bb640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:37.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10d1bb640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-11T01:39:37.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10d1bb640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb108138230 0x7fb108138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:37.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10d1bb640 10 monclient(hunting): picked mon.noname-c con 0x7fb108138230 addr v2:172.21.15.71:3300/0 2024-09-11T01:39:37.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10d1bb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb10809d030 addr v1:172.21.15.12:6789/0 2024-09-11T01:39:37.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10d1bb640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb10809d460 0x7fb1080a5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:37.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10d1bb640 10 monclient(hunting): picked mon.noname-b con 0x7fb10809d460 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:37.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10d1bb640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1080a6e30 con 0x7fb10809d030 2024-09-11T01:39:37.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10d1bb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:37.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10d1bb640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fb10809c9c0 con 0x7fb108138230 2024-09-11T01:39:37.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10d1bb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:37.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10d1bb640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb10809ba20 con 0x7fb10809d460 2024-09-11T01:39:37.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10d1bb640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:37.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10d1bb640 10 monclient(hunting): authenticate will time out at 1383.918555s 2024-09-11T01:39:37.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10759e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb10809d460 0x7fb1080a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:37.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10659c640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb10809d030 0x7fb1080a5020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:41894/0 (socket says 172.21.15.12:41894) 2024-09-11T01:39:37.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10759e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb10809d460 0x7fb1080a5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:47724/0 (socket says 172.21.15.12:47724) 2024-09-11T01:39:37.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10659c640 1 -- 172.21.15.12:0/2948332647 learned_addr learned my addr 172.21.15.12:0/2948332647 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:39:37.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10759e640 10 monclient(hunting): get_auth_request con 0x7fb10809d460 auth_method 0 2024-09-11T01:39:37.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:37.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:37.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:37.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:37.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:37.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb106d9d640 1 --2- 172.21.15.12:0/2948332647 >> v2:172.21.15.71:3300/0 conn(0x7fb108138230 0x7fb108138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:37.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:37.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb106d9d640 10 monclient(hunting): get_auth_request con 0x7fb108138230 auth_method 0 2024-09-11T01:39:37.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb106d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:37.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb106d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:37.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb106d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:37.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb106d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:37.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb106d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:37.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb106d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:37.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10759e640 10 monclient(hunting): handle_auth_done global_id 114134 payload 274 2024-09-11T01:39:37.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10759e640 1 -- 172.21.15.12:0/2948332647 >> v2:172.21.15.71:3300/0 conn(0x7fb108138230 msgr2=0x7fb108138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:39:37.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10759e640 1 --2- 172.21.15.12:0/2948332647 >> v2:172.21.15.71:3300/0 conn(0x7fb108138230 0x7fb108138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:37.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10759e640 1 -- 172.21.15.12:0/2948332647 >> v1:172.21.15.12:6789/0 conn(0x7fb10809d030 legacy=0x7fb1080a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:39:37.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10759e640 10 monclient: _finish_hunting 0 2024-09-11T01:39:37.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10759e640 1 monclient: found mon.noname-b 2024-09-11T01:39:37.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:37.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10759e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:39:37.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.476+0000 7fb10759e640 1 -- 172.21.15.12:0/2948332647 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1080a66a0 con 0x7fb10809d460 2024-09-11T01:39:37.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.479+0000 7fb10759e640 1 -- 172.21.15.12:0/2948332647 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb10809d460 msgr2=0x7fb1080a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:37.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.479+0000 7fb10759e640 1 --2- 172.21.15.12:0/2948332647 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb10809d460 0x7fb1080a5730 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb1080a17e0 tx=0x7fb10002a460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb821b7f47b0fee2 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:37.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.480+0000 7fb105d9b640 1 -- 172.21.15.12:0/2948332647 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb100007950 con 0x7fb10809d460 2024-09-11T01:39:37.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.480+0000 7fb105d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:37.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.480+0000 7fb105d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:39:37.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.480+0000 7fb105d9b640 10 monclient: dump: 2024-09-11T01:39:37.489 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:37.489 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:37.489 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:37.489 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:37.489 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:37.489 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:37.489 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:37.489 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:37.489 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:37.490 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:37.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.480+0000 7fb105d9b640 10 monclient: _finish_auth 0 2024-09-11T01:39:37.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.480+0000 7fb105d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:39:37.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.480+0000 7fb105d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:37.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.480+0000 7fb10d1bb640 5 monclient: authenticate success, global_id 114134 2024-09-11T01:39:37.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.480+0000 7fb105d9b640 1 -- 172.21.15.12:0/2948332647 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb100007af0 con 0x7fb10809d460 2024-09-11T01:39:37.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.480+0000 7fb105d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:37.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.480+0000 7fb10d1bb640 10 monclient: get_monmap_and_config success 2024-09-11T01:39:37.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.480+0000 7fb10d1bb640 4 set_mon_vals no callback set 2024-09-11T01:39:37.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.480+0000 7fb105d9b640 1 -- 172.21.15.12:0/2948332647 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb100031400 con 0x7fb10809d460 2024-09-11T01:39:37.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.480+0000 7fb105d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:37.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.480+0000 7fb105d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:39:37.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.480+0000 7fb105d9b640 10 monclient: dump: 2024-09-11T01:39:37.491 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:37.491 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:37.491 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:37.491 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:37.491 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:37.491 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:37.491 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:37.491 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:37.491 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:37.491 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:37.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.480+0000 7fb10d1bb640 10 monclient: shutdown 2024-09-11T01:39:37.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.480+0000 7fb10d1bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:37.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.480+0000 7fb10d1bb640 1 -- 172.21.15.12:0/2948332647 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb10809d460 msgr2=0x7fb1080a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:37.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.480+0000 7fb10d1bb640 1 --2- 172.21.15.12:0/2948332647 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb10809d460 0x7fb1080a5730 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb1080a17e0 tx=0x7fb10002a460 comp rx=0 tx=0).stop 2024-09-11T01:39:37.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.480+0000 7fb10d1bb640 1 -- 172.21.15.12:0/2948332647 shutdown_connections 2024-09-11T01:39:37.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.480+0000 7fb10d1bb640 1 --2- 172.21.15.12:0/2948332647 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb10809d460 0x7fb1080a5730 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:37.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.480+0000 7fb10d1bb640 1 --2- 172.21.15.12:0/2948332647 >> v2:172.21.15.71:3300/0 conn(0x7fb108138230 0x7fb108138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:37.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.480+0000 7fb10d1bb640 1 -- 172.21.15.12:0/2948332647 >> 172.21.15.12:0/2948332647 conn(0x7fb1080a7120 msgr2=0x7fb1080a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:37.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.480+0000 7fb10d1bb640 1 -- 172.21.15.12:0/2948332647 shutdown_connections 2024-09-11T01:39:37.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.480+0000 7fb10d1bb640 1 -- 172.21.15.12:0/2948332647 wait complete. 2024-09-11T01:39:37.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.481+0000 7fb10d1bb640 10 monclient: build_initial_monmap 2024-09-11T01:39:37.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.481+0000 7fb10d1bb640 10 monclient: monmap: 2024-09-11T01:39:37.492 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:37.492 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:39:37.492 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:39:37.493 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:39:37.493 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:37.493 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:37.493 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:39:37.493 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:39:37.493 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:39:37.493 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:37.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.481+0000 7fb10d1bb640 1 Processor -- start 2024-09-11T01:39:37.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10d1bb640 1 -- start start 2024-09-11T01:39:37.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10d1bb640 10 monclient: init 2024-09-11T01:39:37.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10d1bb640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:37.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10d1bb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:39:37.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10d1bb640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb10809d030 0x7fb108136d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:37.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10d1bb640 10 monclient(hunting): picked mon.noname-c con 0x7fb10809d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:37.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10d1bb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb10809d460 0x7fb108137260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:37.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10d1bb640 10 monclient(hunting): picked mon.noname-a con 0x7fb10809d460 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:39:37.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10d1bb640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb108138230 0x7fb108120c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:37.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10d1bb640 10 monclient(hunting): picked mon.noname-b con 0x7fb108138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:39:37.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10d1bb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:37.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10d1bb640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb10806f970 con 0x7fb10809d460 2024-09-11T01:39:37.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10d1bb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:37.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10d1bb640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb10806f7f0 con 0x7fb108138230 2024-09-11T01:39:37.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10d1bb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:37.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10d1bb640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb1080a5c70 con 0x7fb10809d030 2024-09-11T01:39:37.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10d1bb640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:37.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10d1bb640 10 monclient(hunting): authenticate will time out at 1383.924498s 2024-09-11T01:39:37.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb106d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb10809d030 0x7fb108136d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:37.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb10809d460 0x7fb108137260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:37.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb106d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb10809d030 0x7fb108136d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:47732/0 (socket says 172.21.15.12:47732) 2024-09-11T01:39:37.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb106d9d640 1 -- 172.21.15.12:0/1352329707 learned_addr learned my addr 172.21.15.12:0/1352329707 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:39:37.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb106d9d640 10 monclient(hunting): get_auth_request con 0x7fb10809d030 auth_method 0 2024-09-11T01:39:37.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb106d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:37.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb106d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:37.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb106d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:37.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10659c640 10 monclient(hunting): get_auth_request con 0x7fb10809d460 auth_method 0 2024-09-11T01:39:37.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:37.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:37.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:37.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb106d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:37.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb106d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:37.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb106d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:37.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:37.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:37.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:37.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10759e640 1 --2- 172.21.15.12:0/1352329707 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb108138230 0x7fb108120c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:37.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10759e640 10 monclient(hunting): get_auth_request con 0x7fb108138230 auth_method 0 2024-09-11T01:39:37.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:37.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:37.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:37.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:37.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:37.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb10759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:37.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb106d9d640 10 monclient(hunting): handle_auth_done global_id 114137 payload 824 2024-09-11T01:39:37.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb106d9d640 1 -- 172.21.15.12:0/1352329707 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb108138230 msgr2=0x7fb108120c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:37.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb106d9d640 1 --2- 172.21.15.12:0/1352329707 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb108138230 0x7fb108120c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:37.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb106d9d640 1 -- 172.21.15.12:0/1352329707 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb10809d460 msgr2=0x7fb108137260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:37.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb106d9d640 1 --2- 172.21.15.12:0/1352329707 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb10809d460 0x7fb108137260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:37.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb106d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:39:37.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb106d9d640 1 monclient: found mon.noname-c 2024-09-11T01:39:37.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb106d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:37.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb106d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:39:37.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.482+0000 7fb106d9d640 1 -- 172.21.15.12:0/1352329707 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1080a12a0 con 0x7fb10809d030 2024-09-11T01:39:37.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.485+0000 7fb106d9d640 1 -- 172.21.15.12:0/1352329707 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb10809d030 msgr2=0x7fb108136d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:37.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.485+0000 7fb106d9d640 1 --2- 172.21.15.12:0/1352329707 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb10809d030 0x7fb108136d20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb0f800e9b0 tx=0x7fb0f800ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:37.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.485+0000 7fb0ff7fe640 1 -- 172.21.15.12:0/1352329707 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f8017070 con 0x7fb10809d030 2024-09-11T01:39:37.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.485+0000 7fb0ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:37.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.485+0000 7fb0ff7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:39:37.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.485+0000 7fb0ff7fe640 10 monclient: dump: 2024-09-11T01:39:37.498 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:37.498 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:37.498 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:37.498 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:37.498 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:37.498 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:37.498 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:37.498 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:37.498 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:37.498 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:37.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.485+0000 7fb0ff7fe640 10 monclient: _finish_auth 0 2024-09-11T01:39:37.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.485+0000 7fb0ff7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:39:37.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.485+0000 7fb0ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:37.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.485+0000 7fb0ff7fe640 1 -- 172.21.15.12:0/1352329707 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0f800cbb0 con 0x7fb10809d030 2024-09-11T01:39:37.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.485+0000 7fb0ff7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:37.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.485+0000 7fb0ff7fe640 1 -- 172.21.15.12:0/1352329707 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f8013470 con 0x7fb10809d030 2024-09-11T01:39:37.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.485+0000 7fb10d1bb640 5 monclient: authenticate success, global_id 114137 2024-09-11T01:39:37.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.485+0000 7fb10d1bb640 10 monclient: _renew_subs 2024-09-11T01:39:37.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.485+0000 7fb10d1bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:39:37.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.485+0000 7fb10d1bb640 1 -- 172.21.15.12:0/1352329707 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb108122380 con 0x7fb10809d030 2024-09-11T01:39:37.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.485+0000 7fb0ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:37.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.485+0000 7fb0ff7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:39:37.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.485+0000 7fb10559a640 4 set_mon_vals no callback set 2024-09-11T01:39:37.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.485+0000 7fb0ff7fe640 10 monclient: dump: 2024-09-11T01:39:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:37.500 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.485+0000 7fb10d1bb640 10 monclient: _renew_subs 2024-09-11T01:39:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.485+0000 7fb10d1bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:39:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.485+0000 7fb10d1bb640 1 -- 172.21.15.12:0/1352329707 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb108122ee0 con 0x7fb10809d030 2024-09-11T01:39:37.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.486+0000 7fb0ff7fe640 1 -- 172.21.15.12:0/1352329707 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fb0f8024060 con 0x7fb10809d030 2024-09-11T01:39:37.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.486+0000 7fb10d1bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:37.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.486+0000 7fb10d1bb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:37.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.486+0000 7fb10d1bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:39:37.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.486+0000 7fb10d1bb640 1 -- 172.21.15.12:0/1352329707 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb108138610 con 0x7fb10809d030 2024-09-11T01:39:37.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.486+0000 7fb0ff7fe640 1 --2- 172.21.15.12:0/1352329707 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb0e803e210 0x7fb0e80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:37.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.486+0000 7fb0ff7fe640 1 -- 172.21.15.12:0/1352329707 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(346..346 src has 186..346) ==== 4755+0+0 (secure 0 0 0) 0x7fb0f8052da0 con 0x7fb10809d030 2024-09-11T01:39:37.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.487+0000 7fb10659c640 1 --2- 172.21.15.12:0/1352329707 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb0e803e210 0x7fb0e80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:37.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.487+0000 7fb10659c640 10 monclient: get_auth_request con 0x7fb0e803e210 auth_method 0 2024-09-11T01:39:37.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.489+0000 7fb10659c640 1 --2- 172.21.15.12:0/1352329707 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb0e803e210 0x7fb0e80406d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0007c90 tx=0x7fb0f0007be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:37.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.492+0000 7fb0ff7fe640 1 -- 172.21.15.12:0/1352329707 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb1080a12a0 con 0x7fb10809d030 2024-09-11T01:39:37.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.492+0000 7fb0ff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:37.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.492+0000 7fb0ff7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:39:37.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.753+0000 7fb10d1bb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:39:37.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.753+0000 7fb10d1bb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:39:37.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.753+0000 7fb10d1bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:39:37.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.753+0000 7fb10d1bb640 1 -- 172.21.15.12:0/1352329707 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb1080698e0 con 0x7fb10809d030 2024-09-11T01:39:37.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.753+0000 7fb0ff7fe640 1 -- 172.21.15.12:0/1352329707 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fb0f80517b0 con 0x7fb10809d030 2024-09-11T01:39:37.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.753+0000 7fb0ff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:39:37.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.753+0000 7fb0ff7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:39:37.755 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":100,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:39:37.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.756+0000 7fb10d1bb640 1 -- 172.21.15.12:0/1352329707 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb0e803e210 msgr2=0x7fb0e80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:37.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.756+0000 7fb10d1bb640 1 --2- 172.21.15.12:0/1352329707 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb0e803e210 0x7fb0e80406d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0007c90 tx=0x7fb0f0007be0 comp rx=0 tx=0).stop 2024-09-11T01:39:37.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.756+0000 7fb10d1bb640 10 monclient: shutdown 2024-09-11T01:39:37.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.756+0000 7fb10d1bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:37.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.756+0000 7fb10d1bb640 1 -- 172.21.15.12:0/1352329707 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb10809d030 msgr2=0x7fb108136d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:37.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.756+0000 7fb10d1bb640 1 --2- 172.21.15.12:0/1352329707 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb10809d030 0x7fb108136d20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb0f800e9b0 tx=0x7fb0f800ee70 comp rx=0 tx=0).stop 2024-09-11T01:39:37.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.756+0000 7fb10d1bb640 1 -- 172.21.15.12:0/1352329707 shutdown_connections 2024-09-11T01:39:37.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.756+0000 7fb10d1bb640 1 --2- 172.21.15.12:0/1352329707 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb108138230 0x7fb108120c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:37.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.756+0000 7fb10d1bb640 1 --2- 172.21.15.12:0/1352329707 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb0e803e210 0x7fb0e80406d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:37.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.756+0000 7fb10d1bb640 1 --2- 172.21.15.12:0/1352329707 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb10809d460 0x7fb108137260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:37.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.756+0000 7fb10d1bb640 1 --2- 172.21.15.12:0/1352329707 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb10809d030 0x7fb108136d20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:37.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.756+0000 7fb10d1bb640 1 -- 172.21.15.12:0/1352329707 >> 172.21.15.12:0/1352329707 conn(0x7fb1080a7120 msgr2=0x7fb1080a3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:37.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.756+0000 7fb10d1bb640 1 -- 172.21.15.12:0/1352329707 shutdown_connections 2024-09-11T01:39:37.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.756+0000 7fb10d1bb640 1 -- 172.21.15.12:0/1352329707 wait complete. 2024-09-11T01:39:37.768 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:39:37.768 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:39:37.768 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:39:37.768 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:39:37.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.869+0000 7f976ae18640 10 monclient: get_monmap_and_config 2024-09-11T01:39:37.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.869+0000 7f976ae18640 10 monclient: build_initial_monmap 2024-09-11T01:39:37.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.869+0000 7f976ae18640 10 monclient: monmap: 2024-09-11T01:39:37.871 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:37.871 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:37.871 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:39:37.870020+0000 2024-09-11T01:39:37.871 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:39:37.870020+0000 2024-09-11T01:39:37.871 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:37.871 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:37.871 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:39:37.871 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:39:37.871 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:39:37.871 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:39:37.871 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:39:37.871 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:37.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.869+0000 7f976ae18640 1 Processor -- start 2024-09-11T01:39:37.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f976ae18640 1 -- start start 2024-09-11T01:39:37.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f976ae18640 10 monclient: init 2024-09-11T01:39:37.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f976ae18640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:37.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f976ae18640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-11T01:39:37.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f976ae18640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f97640a03a0 0x7f976409f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:37.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f976ae18640 10 monclient(hunting): picked mon.noname-a con 0x7f97640a03a0 addr v2:172.21.15.12:3300/0 2024-09-11T01:39:37.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f976ae18640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f97640a2530 addr v1:172.21.15.71:6789/0 2024-09-11T01:39:37.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f976ae18640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f97640a13a0 addr v1:172.21.15.12:6789/0 2024-09-11T01:39:37.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f976ae18640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f976409e510 con 0x7f97640a13a0 2024-09-11T01:39:37.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f976ae18640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f976409edb0 con 0x7f97640a2530 2024-09-11T01:39:37.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f976ae18640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:37.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f976ae18640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f97640a29b0 con 0x7f97640a03a0 2024-09-11T01:39:37.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f976ae18640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:37.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f976ae18640 10 monclient(hunting): authenticate will time out at 1384.312585s 2024-09-11T01:39:37.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f9768bb4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f97640a03a0 0x7f976409f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:37.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f97693b5640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f97640a13a0 0x7f976413baa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:41910/0 (socket says 172.21.15.12:41910) 2024-09-11T01:39:37.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f9768bb4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f97640a03a0 0x7f976409f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43548/0 (socket says 172.21.15.12:43548) 2024-09-11T01:39:37.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f97693b5640 1 -- 172.21.15.12:0/3934197824 learned_addr learned my addr 172.21.15.12:0/3934197824 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:39:37.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f9768bb4640 10 monclient(hunting): get_auth_request con 0x7f97640a03a0 auth_method 0 2024-09-11T01:39:37.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f9768bb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:37.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f9768bb4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:37.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f9768bb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:37.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f9768bb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:37.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f9768bb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:37.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f9768bb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:37.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f9768bb4640 10 monclient(hunting): handle_auth_done global_id 94299 payload 274 2024-09-11T01:39:37.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f9768bb4640 1 -- 172.21.15.12:0/3934197824 >> v1:172.21.15.71:6789/0 conn(0x7f97640a2530 legacy=0x7f97640a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:39:37.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f9768bb4640 1 -- 172.21.15.12:0/3934197824 >> v1:172.21.15.12:6789/0 conn(0x7f97640a13a0 legacy=0x7f976413baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:39:37.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f9768bb4640 10 monclient: _finish_hunting 0 2024-09-11T01:39:37.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f9768bb4640 1 monclient: found mon.noname-a 2024-09-11T01:39:37.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f9768bb4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:37.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f9768bb4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:39:37.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.870+0000 7f9768bb4640 1 -- 172.21.15.12:0/3934197824 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97640a17f0 con 0x7f97640a03a0 2024-09-11T01:39:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.873+0000 7f9768bb4640 1 -- 172.21.15.12:0/3934197824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97640a03a0 msgr2=0x7f976409f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.873+0000 7f9768bb4640 1 --2- 172.21.15.12:0/3934197824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97640a03a0 0x7f976409f1d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f97540097a0 tx=0x7f9754029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ede3040eeeb69f7c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.874+0000 7f97637fe640 1 -- 172.21.15.12:0/3934197824 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9754037070 con 0x7f97640a03a0 2024-09-11T01:39:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.874+0000 7f97637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.874+0000 7f97637fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:39:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.874+0000 7f97637fe640 10 monclient: dump: 2024-09-11T01:39:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:37.884 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.874+0000 7f97637fe640 10 monclient: _finish_auth 0 2024-09-11T01:39:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.874+0000 7f97637fe640 10 monclient: _check_auth_tickets 2024-09-11T01:39:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.874+0000 7f97637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.874+0000 7f97637fe640 1 -- 172.21.15.12:0/3934197824 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9754032640 con 0x7f97640a03a0 2024-09-11T01:39:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.874+0000 7f976ae18640 5 monclient: authenticate success, global_id 94299 2024-09-11T01:39:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.874+0000 7f976ae18640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:39:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.874+0000 7f97637fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.874+0000 7f976ae18640 10 monclient: get_monmap_and_config success 2024-09-11T01:39:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.874+0000 7f97637fe640 1 -- 172.21.15.12:0/3934197824 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9754032a10 con 0x7f97640a03a0 2024-09-11T01:39:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.874+0000 7f976ae18640 4 set_mon_vals no callback set 2024-09-11T01:39:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.874+0000 7f97637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.874+0000 7f97637fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:39:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.874+0000 7f97637fe640 10 monclient: dump: 2024-09-11T01:39:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:37.886 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:37.886 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:37.886 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:37.886 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:37.886 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:37.886 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:37.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.874+0000 7f976ae18640 10 monclient: shutdown 2024-09-11T01:39:37.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.874+0000 7f976ae18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:37.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.874+0000 7f976ae18640 1 -- 172.21.15.12:0/3934197824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97640a03a0 msgr2=0x7f976409f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:37.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.874+0000 7f976ae18640 1 --2- 172.21.15.12:0/3934197824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97640a03a0 0x7f976409f1d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f97540097a0 tx=0x7f9754029fd0 comp rx=0 tx=0).stop 2024-09-11T01:39:37.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.874+0000 7f976ae18640 1 -- 172.21.15.12:0/3934197824 shutdown_connections 2024-09-11T01:39:37.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.874+0000 7f976ae18640 1 --2- 172.21.15.12:0/3934197824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97640a03a0 0x7f976409f1d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:37.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.874+0000 7f976ae18640 1 -- 172.21.15.12:0/3934197824 >> 172.21.15.12:0/3934197824 conn(0x7f97640a2f70 msgr2=0x7f97640a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:37.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.874+0000 7f976ae18640 1 -- 172.21.15.12:0/3934197824 shutdown_connections 2024-09-11T01:39:37.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.874+0000 7f976ae18640 1 -- 172.21.15.12:0/3934197824 wait complete. 2024-09-11T01:39:37.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.875+0000 7f976ae18640 10 monclient: build_initial_monmap 2024-09-11T01:39:37.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.875+0000 7f976ae18640 10 monclient: monmap: 2024-09-11T01:39:37.887 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:37.887 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:39:37.887 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:39:37.887 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:39:37.887 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:37.887 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:37.887 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:39:37.887 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:39:37.887 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:39:37.888 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:37.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.875+0000 7f976ae18640 1 Processor -- start 2024-09-11T01:39:37.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.875+0000 7f976ae18640 1 -- start start 2024-09-11T01:39:37.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.875+0000 7f976ae18640 10 monclient: init 2024-09-11T01:39:37.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f976ae18640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:37.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f976ae18640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:39:37.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f976ae18640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97640a03a0 0x7f97641236c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:37.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f976ae18640 10 monclient(hunting): picked mon.noname-b con 0x7f97640a03a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:39:37.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f976ae18640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97640a13a0 0x7f9764123c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:37.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f976ae18640 10 monclient(hunting): picked mon.noname-a con 0x7f97640a13a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:39:37.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f976ae18640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97640a2530 0x7f9764124140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:37.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f976ae18640 10 monclient(hunting): picked mon.noname-c con 0x7f97640a2530 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:37.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f976ae18640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:37.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f976ae18640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f976409ca30 con 0x7f97640a13a0 2024-09-11T01:39:37.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f976ae18640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:37.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f976ae18640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f97640a0780 con 0x7f97640a03a0 2024-09-11T01:39:37.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f976ae18640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:37.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f976ae18640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f976409cbb0 con 0x7f97640a2530 2024-09-11T01:39:37.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f976ae18640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:37.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f976ae18640 10 monclient(hunting): authenticate will time out at 1384.318434s 2024-09-11T01:39:37.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9763fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97640a13a0 0x7f9764123c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:37.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f97693b5640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97640a2530 0x7f9764124140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:37.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9763fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97640a13a0 0x7f9764123c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43560/0 (socket says 172.21.15.12:43560) 2024-09-11T01:39:37.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9763fff640 1 -- 172.21.15.12:0/2299654520 learned_addr learned my addr 172.21.15.12:0/2299654520 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:39:37.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9763fff640 10 monclient(hunting): get_auth_request con 0x7f97640a13a0 auth_method 0 2024-09-11T01:39:37.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9763fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:37.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9763fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:37.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9763fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:37.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9768bb4640 1 --2- 172.21.15.12:0/2299654520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97640a03a0 0x7f97641236c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:37.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f97693b5640 10 monclient(hunting): get_auth_request con 0x7f97640a2530 auth_method 0 2024-09-11T01:39:37.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f97693b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:37.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f97693b5640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:37.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f97693b5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:37.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9768bb4640 10 monclient(hunting): get_auth_request con 0x7f97640a03a0 auth_method 0 2024-09-11T01:39:37.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9768bb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:37.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9768bb4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:37.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9768bb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:37.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9763fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:37.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9763fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:37.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9763fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:37.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f97693b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:37.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f97693b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:37.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f97693b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:37.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9768bb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:37.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9768bb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:37.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9768bb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:37.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9763fff640 10 monclient(hunting): handle_auth_done global_id 94302 payload 824 2024-09-11T01:39:37.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9763fff640 1 -- 172.21.15.12:0/2299654520 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97640a2530 msgr2=0x7f9764124140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:37.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9763fff640 1 --2- 172.21.15.12:0/2299654520 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97640a2530 0x7f9764124140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:37.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9763fff640 1 -- 172.21.15.12:0/2299654520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97640a03a0 msgr2=0x7f97641236c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:37.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9763fff640 1 --2- 172.21.15.12:0/2299654520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97640a03a0 0x7f97641236c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:37.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9763fff640 10 monclient: _finish_hunting 0 2024-09-11T01:39:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9763fff640 1 monclient: found mon.noname-a 2024-09-11T01:39:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9763fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9763fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:39:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9763fff640 1 -- 172.21.15.12:0/2299654520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f976413fc90 con 0x7f97640a13a0 2024-09-11T01:39:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f97693b5640 1 --2- 172.21.15.12:0/2299654520 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97640a2530 0x7f9764124140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:39:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.876+0000 7f9768bb4640 1 --2- 172.21.15.12:0/2299654520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97640a03a0 0x7f97641236c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:39:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.879+0000 7f9763fff640 1 -- 172.21.15.12:0/2299654520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97640a13a0 msgr2=0x7f9764123c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.879+0000 7f9763fff640 1 --2- 172.21.15.12:0/2299654520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97640a13a0 0x7f9764123c00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f975800c7f0 tx=0x7f975800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.879+0000 7f97617fa640 1 -- 172.21.15.12:0/2299654520 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9758017070 con 0x7f97640a13a0 2024-09-11T01:39:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.879+0000 7f97617fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.879+0000 7f97617fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:39:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.879+0000 7f97617fa640 10 monclient: dump: 2024-09-11T01:39:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:37.893 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.879+0000 7f97617fa640 10 monclient: _finish_auth 0 2024-09-11T01:39:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.879+0000 7f97617fa640 10 monclient: _check_auth_tickets 2024-09-11T01:39:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.879+0000 7f97617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.879+0000 7f976ae18640 5 monclient: authenticate success, global_id 94302 2024-09-11T01:39:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.879+0000 7f976ae18640 10 monclient: _renew_subs 2024-09-11T01:39:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.879+0000 7f97617fa640 1 -- 172.21.15.12:0/2299654520 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9758005070 con 0x7f97640a13a0 2024-09-11T01:39:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.879+0000 7f976ae18640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.879+0000 7f976ae18640 1 -- 172.21.15.12:0/2299654520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9764126400 con 0x7f97640a13a0 2024-09-11T01:39:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.879+0000 7f97617fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.879+0000 7f97617fa640 1 -- 172.21.15.12:0/2299654520 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9758005460 con 0x7f97640a13a0 2024-09-11T01:39:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.879+0000 7f97617fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.879+0000 7f97617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:39:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.879+0000 7f97617fa640 10 monclient: dump: 2024-09-11T01:39:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:37.895 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.879+0000 7f976ae18640 10 monclient: _renew_subs 2024-09-11T01:39:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.879+0000 7f976ae18640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.879+0000 7f9762ffd640 4 set_mon_vals no callback set 2024-09-11T01:39:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.879+0000 7f976ae18640 1 -- 172.21.15.12:0/2299654520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f976409d480 con 0x7f97640a13a0 2024-09-11T01:39:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.880+0000 7f976ae18640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.880+0000 7f976ae18640 1 --2- 172.21.15.12:0/2299654520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f976413d990 0x7f976402d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.880+0000 7f976ae18640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.880+0000 7f976ae18640 1 -- 172.21.15.12:0/2299654520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9764125bc0 con 0x7f976413d990 2024-09-11T01:39:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.880+0000 7f97617fa640 1 -- 172.21.15.12:0/2299654520 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f975800a5e0 con 0x7f97640a13a0 2024-09-11T01:39:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.880+0000 7f9768bb4640 1 --2- 172.21.15.12:0/2299654520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f976413d990 0x7f976402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.880+0000 7f9768bb4640 10 monclient: get_auth_request con 0x7f976413d990 auth_method 0 2024-09-11T01:39:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.880+0000 7f9768bb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.880+0000 7f9768bb4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.880+0000 7f9768bb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.880+0000 7f9768bb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.880+0000 7f9768bb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.880+0000 7f9768bb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.880+0000 7f97617fa640 1 --2- 172.21.15.12:0/2299654520 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f975003e210 0x7f97500406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.880+0000 7f9768bb4640 10 monclient(hunting): handle_auth_done global_id 94305 payload 824 2024-09-11T01:39:37.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.880+0000 7f9768bb4640 1 -- 172.21.15.12:0/2299654520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f976409cfe0 con 0x7f976413d990 2024-09-11T01:39:37.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.880+0000 7f97617fa640 1 -- 172.21.15.12:0/2299654520 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(346..346 src has 186..346) ==== 4755+0+0 (secure 0 0 0) 0x7f9758052710 con 0x7f97640a13a0 2024-09-11T01:39:37.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.881+0000 7f97693b5640 1 --2- 172.21.15.12:0/2299654520 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f975003e210 0x7f97500406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:37.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.881+0000 7f97693b5640 10 monclient: get_auth_request con 0x7f975003e210 auth_method 0 2024-09-11T01:39:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.883+0000 7f9768bb4640 1 -- 172.21.15.12:0/2299654520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f976413d990 msgr2=0x7f976402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.883+0000 7f9768bb4640 1 --2- 172.21.15.12:0/2299654520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f976413d990 0x7f976402d440 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f976409e880 tx=0x7f97540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.883+0000 7f97617fa640 1 -- 172.21.15.12:0/2299654520 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9754037040 con 0x7f976413d990 2024-09-11T01:39:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.883+0000 7f97617fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.883+0000 7f97617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:39:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.883+0000 7f97617fa640 10 monclient: dump: 2024-09-11T01:39:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:37.900 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.883+0000 7f97617fa640 1 -- 172.21.15.12:0/2299654520 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f975403cb10 con 0x7f976413d990 2024-09-11T01:39:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.883+0000 7f97617fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.883+0000 7f97617fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:39:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.883+0000 7f97617fa640 1 -- 172.21.15.12:0/2299654520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f976413d990 msgr2=0x7f976402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.883+0000 7f97617fa640 1 --2- 172.21.15.12:0/2299654520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f976413d990 0x7f976402d440 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f976409e880 tx=0x7f97540046c0 comp rx=0 tx=0).stop 2024-09-11T01:39:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.883+0000 7f97617fa640 1 -- 172.21.15.12:0/2299654520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f976413d990 msgr2=0x7f976402d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:39:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.883+0000 7f97617fa640 1 --2- 172.21.15.12:0/2299654520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f976413d990 0x7f976402d440 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.883+0000 7f97693b5640 1 --2- 172.21.15.12:0/2299654520 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f975003e210 0x7f97500406d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f975c007c90 tx=0x7f975c007be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.889+0000 7f976ae18640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:39:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.889+0000 7f976ae18640 1 --2- 172.21.15.12:0/2299654520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f976413dd70 0x7f976402d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.889+0000 7f976ae18640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.889+0000 7f976ae18640 1 -- 172.21.15.12:0/2299654520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f976411dff0 con 0x7f976413dd70 2024-09-11T01:39:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.889+0000 7f9768bb4640 1 --2- 172.21.15.12:0/2299654520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f976413dd70 0x7f976402d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.889+0000 7f9768bb4640 10 monclient: get_auth_request con 0x7f976413dd70 auth_method 0 2024-09-11T01:39:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.889+0000 7f9768bb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.889+0000 7f9768bb4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.889+0000 7f9768bb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.889+0000 7f9768bb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.889+0000 7f9768bb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.889+0000 7f9768bb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.889+0000 7f9768bb4640 10 monclient(hunting): handle_auth_done global_id 94308 payload 824 2024-09-11T01:39:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.889+0000 7f9768bb4640 1 -- 172.21.15.12:0/2299654520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f976413ea80 con 0x7f976413dd70 2024-09-11T01:39:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.892+0000 7f9768bb4640 1 -- 172.21.15.12:0/2299654520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f976413dd70 msgr2=0x7f976402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.892+0000 7f9768bb4640 1 --2- 172.21.15.12:0/2299654520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f976413dd70 0x7f976402d980 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9754000c00 tx=0x7f9754008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.892+0000 7f97617fa640 1 -- 172.21.15.12:0/2299654520 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975404f070 con 0x7f976413dd70 2024-09-11T01:39:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.892+0000 7f97617fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.892+0000 7f97617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:39:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.892+0000 7f97617fa640 10 monclient: dump: 2024-09-11T01:39:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:37.903 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.892+0000 7f97617fa640 1 -- 172.21.15.12:0/2299654520 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f9754003e60 con 0x7f976413dd70 2024-09-11T01:39:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.892+0000 7f97617fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:39:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.892+0000 7f97617fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:39:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.892+0000 7f97617fa640 1 -- 172.21.15.12:0/2299654520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f976413dd70 msgr2=0x7f976402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.892+0000 7f97617fa640 1 --2- 172.21.15.12:0/2299654520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f976413dd70 0x7f976402d980 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9754000c00 tx=0x7f9754008030 comp rx=0 tx=0).stop 2024-09-11T01:39:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.892+0000 7f97617fa640 1 -- 172.21.15.12:0/2299654520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f976413dd70 msgr2=0x7f976402d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:39:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.892+0000 7f97617fa640 1 --2- 172.21.15.12:0/2299654520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f976413dd70 0x7f976402d980 secure :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9754000c00 tx=0x7f9754008030 comp rx=0 tx=0).stop 2024-09-11T01:39:37.904 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:39:37.904 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:39:37.904 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:39:37.904 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:39:37.904 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 100, 2024-09-11T01:39:37.904 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 123988, 2024-09-11T01:39:37.904 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:39:37.904 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:39:37.904 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:39:37.904 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:39:37.905 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:37.905 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:39:37.905 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:39:37.905 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:39:37.905 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:39:37.905 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:39:37.905 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:39:37.905 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:39:37.905 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:39:37.905 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:39:37.905 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:39:37.905 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:39:37.905 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:39:37.905 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:39:37.906 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:39:37.906 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:39:37.906 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:37.906 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:39:37.906 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:39:37.906 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:39:37.906 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:39:37.906 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:39:37.906 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:39:37.906 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:39:37.906 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:39:37.906 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:39:37.906 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:39:37.907 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:39:37.907 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:39:37.907 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:39:37.907 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:37.907 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:37.907 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:39:37.907 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:39:37.907 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:39:37.907 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:39:37.907 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:39:37.907 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:39:37.907 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:39:37.907 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:39:37.907 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:39:37.908 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:39:37.908 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:39:37.908 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:39:37.908 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:39:37.908 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:39:37.908 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:39:37.908 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:39:37.908 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:39:37.908 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:39:37.908 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:39:37.908 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:39:37.908 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:39:37.908 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:39:37.909 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:39:37.909 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:39:37.909 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:39:37.909 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:39:37.909 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:39:37.909 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:39:37.909 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:37.909 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:39:37.909 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:37.909 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:39:37.909 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:37.909 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:39:37.909 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:39:37.910 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:39:37.910 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:39:37.910 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:37.910 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:39:37.910 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:39:37.910 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:37.910 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:37.910 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:37.910 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:39:37.910 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:39:37.910 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:37.910 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:37.911 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:37.911 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:37.911 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:39:37.911 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:39:37.911 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:39:37.911 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:39:37.911 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:39:37.911 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:37.911 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:37.911 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:39:37.911 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:39:37.911 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:39:37.911 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:39:37.912 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:37.912 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:39:37.912 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:39:37.912 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:37.912 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:37.912 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:37.912 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:39:37.912 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:39:37.912 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:37.912 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:37.912 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:37.912 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:37.912 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:39:37.912 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:39:37.913 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:39:37.913 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:39:37.913 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:39:37.913 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:37.913 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:37.913 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:39:37.913 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:39:37.913 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:39:37.913 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:39:37.913 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:37.913 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:39:37.913 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:39:37.913 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:37.913 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:37.914 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:37.914 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:39:37.914 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:39:37.914 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:37.914 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:37.914 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:37.914 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:37.914 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:39:37.914 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:39:37.914 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:39:37.914 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:39:37.914 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:39:37.914 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:37.914 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:37.915 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:37.915 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:39:37.915 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:39:37.915 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:37.915 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:39:37.915 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:39:37.915 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:39:37.915 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:37.915 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:37.915 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:39:37.915 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:37.915 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:39:37.916 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:39:37.916 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:39:37.916 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:37.916 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:37.916 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:39:37.916 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:37.916 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:39:37.916 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:39:37.916 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 7 2024-09-11T01:39:37.916 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:37.916 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:37.916 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:37.916 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:39:37.917 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:39:37.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.892+0000 7f976ae18640 1 -- 172.21.15.12:0/2299654520 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f975003e210 msgr2=0x7f97500406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:37.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.892+0000 7f976ae18640 1 --2- 172.21.15.12:0/2299654520 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f975003e210 0x7f97500406d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f975c007c90 tx=0x7f975c007be0 comp rx=0 tx=0).stop 2024-09-11T01:39:37.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.892+0000 7f976ae18640 10 monclient: shutdown 2024-09-11T01:39:37.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.892+0000 7f976ae18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:37.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.892+0000 7f976ae18640 1 -- 172.21.15.12:0/2299654520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97640a13a0 msgr2=0x7f9764123c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:37.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.892+0000 7f976ae18640 1 --2- 172.21.15.12:0/2299654520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97640a13a0 0x7f9764123c00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f975800c7f0 tx=0x7f975800ccb0 comp rx=0 tx=0).stop 2024-09-11T01:39:37.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.892+0000 7f97693b5640 1 -- 172.21.15.12:0/2299654520 reap_dead start 2024-09-11T01:39:37.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.893+0000 7f976ae18640 1 -- 172.21.15.12:0/2299654520 shutdown_connections 2024-09-11T01:39:37.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.893+0000 7f976ae18640 1 --2- 172.21.15.12:0/2299654520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97640a13a0 0x7f9764123c00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:37.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.893+0000 7f976ae18640 1 -- 172.21.15.12:0/2299654520 >> 172.21.15.12:0/2299654520 conn(0x7f97640a2f70 msgr2=0x7f97640a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:37.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.893+0000 7f976ae18640 1 -- 172.21.15.12:0/2299654520 shutdown_connections 2024-09-11T01:39:37.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:37.893+0000 7f976ae18640 1 -- 172.21.15.12:0/2299654520 wait complete. 2024-09-11T01:39:37.919 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:39:37.919 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:39:37.919 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:39:38.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.019+0000 7f3a12ff8640 10 monclient: get_monmap_and_config 2024-09-11T01:39:38.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.019+0000 7f3a12ff8640 10 monclient: build_initial_monmap 2024-09-11T01:39:38.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.019+0000 7f3a12ff8640 10 monclient: monmap: 2024-09-11T01:39:38.021 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:38.021 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:38.021 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:39:38.020424+0000 2024-09-11T01:39:38.021 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:39:38.020424+0000 2024-09-11T01:39:38.021 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:38.021 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:38.021 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:39:38.021 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:39:38.022 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:39:38.022 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:39:38.022 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:39:38.022 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:38.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.020+0000 7f3a12ff8640 1 Processor -- start 2024-09-11T01:39:38.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.020+0000 7f3a12ff8640 1 -- start start 2024-09-11T01:39:38.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.020+0000 7f3a12ff8640 10 monclient: init 2024-09-11T01:39:38.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.020+0000 7f3a12ff8640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:38.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.020+0000 7f3a12ff8640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-11T01:39:38.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.020+0000 7f3a12ff8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a0c134ad0 addr v1:172.21.15.12:6789/0 2024-09-11T01:39:38.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.020+0000 7f3a12ff8640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3a0c0a2200 0x7f3a0c13a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:38.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.020+0000 7f3a12ff8640 10 monclient(hunting): picked mon.noname-c con 0x7f3a0c0a2200 addr v2:172.21.15.71:3300/0 2024-09-11T01:39:38.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.020+0000 7f3a12ff8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3a0c1355c0 addr v1:172.21.15.71:6789/0 2024-09-11T01:39:38.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.020+0000 7f3a12ff8640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a0c09ed30 con 0x7f3a0c134ad0 2024-09-11T01:39:38.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.020+0000 7f3a12ff8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a0c09f010 con 0x7f3a0c1355c0 2024-09-11T01:39:38.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.020+0000 7f3a12ff8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:38.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.020+0000 7f3a12ff8640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f3a0c09f5d0 con 0x7f3a0c0a2200 2024-09-11T01:39:38.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.020+0000 7f3a12ff8640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:38.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.020+0000 7f3a12ff8640 10 monclient(hunting): authenticate will time out at 1384.462975s 2024-09-11T01:39:38.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.021+0000 7f3a10d94640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3a0c134ad0 0x7f3a0c134eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:41912/0 (socket says 172.21.15.12:41912) 2024-09-11T01:39:38.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.021+0000 7f3a10d94640 1 -- 172.21.15.12:0/570624512 learned_addr learned my addr 172.21.15.12:0/570624512 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:39:38.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.021+0000 7f3a0bfff640 1 --2- 172.21.15.12:0/570624512 >> v2:172.21.15.71:3300/0 conn(0x7f3a0c0a2200 0x7f3a0c13a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:38.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.021+0000 7f3a0bfff640 10 monclient(hunting): get_auth_request con 0x7f3a0c0a2200 auth_method 0 2024-09-11T01:39:38.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.021+0000 7f3a0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:38.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.021+0000 7f3a0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:38.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.021+0000 7f3a0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:38.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.021+0000 7f3a0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:38.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.021+0000 7f3a0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:38.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.021+0000 7f3a0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:38.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.021+0000 7f3a0bfff640 10 monclient(hunting): handle_auth_done global_id 104197 payload 274 2024-09-11T01:39:38.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.021+0000 7f3a0bfff640 1 -- 172.21.15.12:0/570624512 >> v1:172.21.15.71:6789/0 conn(0x7f3a0c1355c0 legacy=0x7f3a0c13c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:39:38.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.021+0000 7f3a0bfff640 1 -- 172.21.15.12:0/570624512 >> v1:172.21.15.12:6789/0 conn(0x7f3a0c134ad0 legacy=0x7f3a0c134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:39:38.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.021+0000 7f3a0bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:39:38.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.021+0000 7f3a0bfff640 1 monclient: found mon.noname-c 2024-09-11T01:39:38.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.021+0000 7f3a0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:38.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.021+0000 7f3a0bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:39:38.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.021+0000 7f3a0bfff640 1 -- 172.21.15.12:0/570624512 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a0c09f2f0 con 0x7f3a0c0a2200 2024-09-11T01:39:38.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.024+0000 7f3a0bfff640 1 -- 172.21.15.12:0/570624512 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3a0c0a2200 msgr2=0x7f3a0c13a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:38.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.024+0000 7f3a0bfff640 1 --2- 172.21.15.12:0/570624512 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3a0c0a2200 0x7f3a0c13a000 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c09f750 tx=0x7f39f8029fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82552be142152345 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:38.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.024+0000 7f3a0b7fe640 1 -- 172.21.15.12:0/570624512 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f8037070 con 0x7f3a0c0a2200 2024-09-11T01:39:38.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.024+0000 7f3a0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:38.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.024+0000 7f3a0b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:39:38.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.024+0000 7f3a0b7fe640 10 monclient: dump: 2024-09-11T01:39:38.030 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:38.030 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:38.030 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.030 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.031 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:38.031 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:38.031 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:38.031 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:38.031 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:38.031 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:38.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.024+0000 7f3a0b7fe640 10 monclient: _finish_auth 0 2024-09-11T01:39:38.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.024+0000 7f3a0b7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:39:38.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.024+0000 7f3a0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:38.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.024+0000 7f3a0b7fe640 1 -- 172.21.15.12:0/570624512 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39f8032440 con 0x7f3a0c0a2200 2024-09-11T01:39:38.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.024+0000 7f3a0b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:38.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.024+0000 7f3a0b7fe640 1 -- 172.21.15.12:0/570624512 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f80327f0 con 0x7f3a0c0a2200 2024-09-11T01:39:38.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.024+0000 7f3a0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:38.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.024+0000 7f3a0b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:39:38.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.024+0000 7f3a0b7fe640 10 monclient: dump: 2024-09-11T01:39:38.032 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:38.032 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:38.032 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.032 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.032 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:38.032 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:38.032 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:38.032 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:38.032 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:38.032 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:38.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.024+0000 7f3a12ff8640 5 monclient: authenticate success, global_id 104197 2024-09-11T01:39:38.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.024+0000 7f3a12ff8640 10 monclient: get_monmap_and_config success 2024-09-11T01:39:38.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.024+0000 7f3a12ff8640 4 set_mon_vals no callback set 2024-09-11T01:39:38.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.025+0000 7f3a12ff8640 10 monclient: shutdown 2024-09-11T01:39:38.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.025+0000 7f3a12ff8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:38.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.025+0000 7f3a12ff8640 1 -- 172.21.15.12:0/570624512 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3a0c0a2200 msgr2=0x7f3a0c13a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:38.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.025+0000 7f3a12ff8640 1 --2- 172.21.15.12:0/570624512 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3a0c0a2200 0x7f3a0c13a000 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c09f750 tx=0x7f39f8029fe0 comp rx=0 tx=0).stop 2024-09-11T01:39:38.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.025+0000 7f3a12ff8640 1 -- 172.21.15.12:0/570624512 shutdown_connections 2024-09-11T01:39:38.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.025+0000 7f3a12ff8640 1 --2- 172.21.15.12:0/570624512 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3a0c0a2200 0x7f3a0c13a000 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:38.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.025+0000 7f3a12ff8640 1 -- 172.21.15.12:0/570624512 >> 172.21.15.12:0/570624512 conn(0x7f3a0c0a2f70 msgr2=0x7f3a0c0a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:38.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.025+0000 7f3a12ff8640 1 -- 172.21.15.12:0/570624512 shutdown_connections 2024-09-11T01:39:38.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.025+0000 7f3a12ff8640 1 -- 172.21.15.12:0/570624512 wait complete. 2024-09-11T01:39:38.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.025+0000 7f3a12ff8640 10 monclient: build_initial_monmap 2024-09-11T01:39:38.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.025+0000 7f3a12ff8640 10 monclient: monmap: 2024-09-11T01:39:38.033 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:38.034 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:39:38.034 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:39:38.034 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:39:38.034 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:38.034 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:38.034 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:39:38.034 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:39:38.034 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:39:38.034 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:38.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.026+0000 7f3a12ff8640 1 Processor -- start 2024-09-11T01:39:38.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.026+0000 7f3a12ff8640 1 -- start start 2024-09-11T01:39:38.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.026+0000 7f3a12ff8640 10 monclient: init 2024-09-11T01:39:38.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.026+0000 7f3a12ff8640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:38.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.026+0000 7f3a12ff8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:39:38.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.026+0000 7f3a12ff8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3a0c0a2200 0x7f3a0c133870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:38.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.026+0000 7f3a12ff8640 10 monclient(hunting): picked mon.noname-c con 0x7f3a0c0a2200 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:38.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.026+0000 7f3a12ff8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3a0c134ad0 0x7f3a0c133db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:38.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.026+0000 7f3a12ff8640 10 monclient(hunting): picked mon.noname-b con 0x7f3a0c134ad0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:39:38.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.026+0000 7f3a12ff8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a0c1355c0 0x7f3a0c120490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:38.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.026+0000 7f3a12ff8640 10 monclient(hunting): picked mon.noname-a con 0x7f3a0c1355c0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:39:38.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.026+0000 7f3a12ff8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:38.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.026+0000 7f3a12ff8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0c0a0150 con 0x7f3a0c1355c0 2024-09-11T01:39:38.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.026+0000 7f3a12ff8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:38.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.026+0000 7f3a12ff8640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0c09ffd0 con 0x7f3a0c134ad0 2024-09-11T01:39:38.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.026+0000 7f3a12ff8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:38.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.026+0000 7f3a12ff8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f3a0c0a0c90 con 0x7f3a0c0a2200 2024-09-11T01:39:38.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.026+0000 7f3a12ff8640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:38.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.026+0000 7f3a12ff8640 10 monclient(hunting): authenticate will time out at 1384.468854s 2024-09-11T01:39:38.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.026+0000 7f3a11595640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a0c1355c0 0x7f3a0c120490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:38.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.026+0000 7f3a10d94640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3a0c0a2200 0x7f3a0c133870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:38.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.026+0000 7f3a11595640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a0c1355c0 0x7f3a0c120490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43596/0 (socket says 172.21.15.12:43596) 2024-09-11T01:39:38.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.026+0000 7f3a11595640 1 -- 172.21.15.12:0/948534311 learned_addr learned my addr 172.21.15.12:0/948534311 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:39:38.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.026+0000 7f3a11595640 10 monclient(hunting): get_auth_request con 0x7f3a0c1355c0 auth_method 0 2024-09-11T01:39:38.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.026+0000 7f3a11595640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:38.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.026+0000 7f3a11595640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:38.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a11595640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:38.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a10d94640 10 monclient(hunting): get_auth_request con 0x7f3a0c0a2200 auth_method 0 2024-09-11T01:39:38.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a10d94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:38.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a10d94640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:38.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a10d94640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:38.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a0bfff640 1 --2- 172.21.15.12:0/948534311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3a0c134ad0 0x7f3a0c133db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:38.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a0bfff640 10 monclient(hunting): get_auth_request con 0x7f3a0c134ad0 auth_method 0 2024-09-11T01:39:38.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:38.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:38.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:38.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a11595640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:38.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a11595640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:38.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a11595640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:38.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a10d94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:38.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a10d94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:38.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a10d94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:38.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:38.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:38.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:38.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a11595640 10 monclient(hunting): handle_auth_done global_id 94311 payload 824 2024-09-11T01:39:38.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a11595640 1 -- 172.21.15.12:0/948534311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3a0c0a2200 msgr2=0x7f3a0c133870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:38.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a11595640 1 --2- 172.21.15.12:0/948534311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3a0c0a2200 0x7f3a0c133870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:38.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a11595640 1 -- 172.21.15.12:0/948534311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3a0c134ad0 msgr2=0x7f3a0c133db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:38.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a11595640 1 --2- 172.21.15.12:0/948534311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3a0c134ad0 0x7f3a0c133db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:38.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a11595640 10 monclient: _finish_hunting 0 2024-09-11T01:39:38.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a11595640 1 monclient: found mon.noname-a 2024-09-11T01:39:38.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a11595640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:38.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a11595640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:39:38.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a11595640 1 -- 172.21.15.12:0/948534311 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a0c0a4a10 con 0x7f3a0c1355c0 2024-09-11T01:39:38.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a10d94640 1 --2- 172.21.15.12:0/948534311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3a0c0a2200 0x7f3a0c133870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:39:38.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.027+0000 7f3a0bfff640 1 --2- 172.21.15.12:0/948534311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3a0c134ad0 0x7f3a0c133db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:39:38.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.029+0000 7f3a11595640 1 -- 172.21.15.12:0/948534311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a0c1355c0 msgr2=0x7f3a0c120490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:38.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.029+0000 7f3a11595640 1 --2- 172.21.15.12:0/948534311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a0c1355c0 0x7f3a0c120490 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400c780 tx=0x7f3a0400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:38.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.029+0000 7f3a097fa640 1 -- 172.21.15.12:0/948534311 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0400ac60 con 0x7f3a0c1355c0 2024-09-11T01:39:38.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.029+0000 7f3a097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:38.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.029+0000 7f3a097fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:39:38.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.029+0000 7f3a097fa640 10 monclient: dump: 2024-09-11T01:39:38.042 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:38.042 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:38.042 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.042 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.042 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:38.042 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:38.042 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:38.043 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:38.043 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:38.043 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:38.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.029+0000 7f3a097fa640 10 monclient: _finish_auth 0 2024-09-11T01:39:38.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.029+0000 7f3a097fa640 10 monclient: _check_auth_tickets 2024-09-11T01:39:38.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.029+0000 7f3a097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:38.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.029+0000 7f3a12ff8640 5 monclient: authenticate success, global_id 94311 2024-09-11T01:39:38.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.029+0000 7f3a097fa640 1 -- 172.21.15.12:0/948534311 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a0400ae00 con 0x7f3a0c1355c0 2024-09-11T01:39:38.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.029+0000 7f3a12ff8640 10 monclient: _renew_subs 2024-09-11T01:39:38.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.029+0000 7f3a12ff8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:38.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.029+0000 7f3a12ff8640 1 -- 172.21.15.12:0/948534311 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a0c09d480 con 0x7f3a0c1355c0 2024-09-11T01:39:38.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.029+0000 7f3a097fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:38.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.029+0000 7f3a097fa640 1 -- 172.21.15.12:0/948534311 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a04004ba0 con 0x7f3a0c1355c0 2024-09-11T01:39:38.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.029+0000 7f3a097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:38.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.029+0000 7f3a097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:39:38.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.029+0000 7f3a097fa640 10 monclient: dump: 2024-09-11T01:39:38.044 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:38.044 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:38.044 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.044 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.044 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:38.044 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:38.044 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:38.044 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:38.044 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:38.044 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:38.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.029+0000 7f3a0affd640 4 set_mon_vals no callback set 2024-09-11T01:39:38.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.029+0000 7f3a12ff8640 10 monclient: _renew_subs 2024-09-11T01:39:38.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.029+0000 7f3a12ff8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:38.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.029+0000 7f3a12ff8640 1 -- 172.21.15.12:0/948534311 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a0c134830 con 0x7f3a0c1355c0 2024-09-11T01:39:38.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.030+0000 7f3a12ff8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:38.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.030+0000 7f3a12ff8640 1 --2- 172.21.15.12:0/948534311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3a0c123eb0 0x7f3a0c1209d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:38.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.030+0000 7f3a12ff8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:38.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.030+0000 7f3a12ff8640 1 -- 172.21.15.12:0/948534311 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f3a0c134eb0 con 0x7f3a0c123eb0 2024-09-11T01:39:38.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.030+0000 7f3a097fa640 1 -- 172.21.15.12:0/948534311 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f3a04005030 con 0x7f3a0c1355c0 2024-09-11T01:39:38.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.030+0000 7f3a10d94640 1 --2- 172.21.15.12:0/948534311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3a0c123eb0 0x7f3a0c1209d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:38.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.030+0000 7f3a10d94640 10 monclient: get_auth_request con 0x7f3a0c123eb0 auth_method 0 2024-09-11T01:39:38.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.030+0000 7f3a10d94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:38.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.030+0000 7f3a10d94640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:38.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.030+0000 7f3a10d94640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:38.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.030+0000 7f3a10d94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:38.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.030+0000 7f3a10d94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:38.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.030+0000 7f3a10d94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:38.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.031+0000 7f3a097fa640 1 --2- 172.21.15.12:0/948534311 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f39f003e240 0x7f39f0040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:38.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.031+0000 7f3a10d94640 10 monclient(hunting): handle_auth_done global_id 114146 payload 824 2024-09-11T01:39:38.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.031+0000 7f3a10d94640 1 -- 172.21.15.12:0/948534311 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3a0c0008d0 con 0x7f3a0c123eb0 2024-09-11T01:39:38.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.031+0000 7f3a097fa640 1 -- 172.21.15.12:0/948534311 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(346..346 src has 186..346) ==== 4755+0+0 (secure 0 0 0) 0x7f3a04052f90 con 0x7f3a0c1355c0 2024-09-11T01:39:38.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.031+0000 7f3a0bfff640 1 --2- 172.21.15.12:0/948534311 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f39f003e240 0x7f39f0040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:38.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.031+0000 7f3a0bfff640 10 monclient: get_auth_request con 0x7f39f003e240 auth_method 0 2024-09-11T01:39:38.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.033+0000 7f3a10d94640 1 -- 172.21.15.12:0/948534311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3a0c123eb0 msgr2=0x7f3a0c1209d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:38.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.033+0000 7f3a10d94640 1 --2- 172.21.15.12:0/948534311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3a0c123eb0 0x7f3a0c1209d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3a00007fd0 tx=0x7f3a00007700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:38.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.033+0000 7f3a097fa640 1 -- 172.21.15.12:0/948534311 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a00017030 con 0x7f3a0c123eb0 2024-09-11T01:39:38.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.033+0000 7f3a097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:38.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.033+0000 7f3a097fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:39:38.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.033+0000 7f3a097fa640 10 monclient: dump: 2024-09-11T01:39:38.046 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:38.046 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:38.047 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.047 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.047 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:38.047 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:38.047 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:38.047 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:38.047 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:38.047 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:38.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.033+0000 7f3a097fa640 1 -- 172.21.15.12:0/948534311 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3a0000fc50 con 0x7f3a0c123eb0 2024-09-11T01:39:38.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.033+0000 7f3a097fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:38.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.033+0000 7f3a097fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:39:38.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.033+0000 7f3a097fa640 1 -- 172.21.15.12:0/948534311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3a0c123eb0 msgr2=0x7f3a0c1209d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:38.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.033+0000 7f3a097fa640 1 --2- 172.21.15.12:0/948534311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3a0c123eb0 0x7f3a0c1209d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3a00007fd0 tx=0x7f3a00007700 comp rx=0 tx=0).stop 2024-09-11T01:39:38.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.033+0000 7f3a097fa640 1 -- 172.21.15.12:0/948534311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3a0c123eb0 msgr2=0x7f3a0c1209d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:39:38.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.033+0000 7f3a097fa640 1 --2- 172.21.15.12:0/948534311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3a0c123eb0 0x7f3a0c1209d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:38.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.033+0000 7f3a0bfff640 1 --2- 172.21.15.12:0/948534311 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f39f003e240 0x7f39f0040700 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c0a72b0 tx=0x7f39f802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:38.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.039+0000 7f3a12ff8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:39:38.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.039+0000 7f3a12ff8640 1 --2- 172.21.15.12:0/948534311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3a0c120f10 0x7f3a0c121320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:38.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.039+0000 7f3a12ff8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:38.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.039+0000 7f3a12ff8640 1 -- 172.21.15.12:0/948534311 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f3a0c1243b0 con 0x7f3a0c120f10 2024-09-11T01:39:38.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.040+0000 7f3a10d94640 1 --2- 172.21.15.12:0/948534311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3a0c120f10 0x7f3a0c121320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:38.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.040+0000 7f3a10d94640 10 monclient: get_auth_request con 0x7f3a0c120f10 auth_method 0 2024-09-11T01:39:38.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.040+0000 7f3a10d94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:38.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.040+0000 7f3a10d94640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:38.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.040+0000 7f3a10d94640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:38.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.040+0000 7f3a10d94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:38.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.040+0000 7f3a10d94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:38.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.040+0000 7f3a10d94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:38.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.040+0000 7f3a10d94640 10 monclient(hunting): handle_auth_done global_id 114149 payload 824 2024-09-11T01:39:38.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.040+0000 7f3a10d94640 1 -- 172.21.15.12:0/948534311 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3a0c121860 con 0x7f3a0c120f10 2024-09-11T01:39:38.050 INFO:tasks.rados.rados.0.smithi071.stdout:1528: done (1 left) 2024-09-11T01:39:38.050 INFO:tasks.rados.rados.0.smithi071.stdout:1529: write (excl) oid 38 current snap is 164 2024-09-11T01:39:38.050 INFO:tasks.rados.rados.0.smithi071.stdout:1529: seq_num 575 ranges {464025=794650,1947852=404189,2990809=425525} 2024-09-11T01:39:38.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.042+0000 7f3a10d94640 1 -- 172.21.15.12:0/948534311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3a0c120f10 msgr2=0x7f3a0c121320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:38.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.042+0000 7f3a10d94640 1 --2- 172.21.15.12:0/948534311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3a0c120f10 0x7f3a0c121320 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3a00017000 tx=0x7f3a00004940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:38.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.045+0000 7f3a097fa640 1 -- 172.21.15.12:0/948534311 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a00006230 con 0x7f3a0c120f10 2024-09-11T01:39:38.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.045+0000 7f3a097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:38.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.045+0000 7f3a097fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:39:38.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.045+0000 7f3a097fa640 10 monclient: dump: 2024-09-11T01:39:38.051 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:38.051 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:38.051 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.051 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.051 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:38.051 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:38.051 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:38.051 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:38.051 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:38.051 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:38.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.045+0000 7f3a097fa640 1 -- 172.21.15.12:0/948534311 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7f3a00006230 con 0x7f3a0c120f10 2024-09-11T01:39:38.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.045+0000 7f3a097fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:39:38.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.045+0000 7f3a097fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:39:38.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.045+0000 7f3a097fa640 1 -- 172.21.15.12:0/948534311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3a0c120f10 msgr2=0x7f3a0c121320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:38.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.045+0000 7f3a097fa640 1 --2- 172.21.15.12:0/948534311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3a0c120f10 0x7f3a0c121320 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3a00017000 tx=0x7f3a00004940 comp rx=0 tx=0).stop 2024-09-11T01:39:38.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.045+0000 7f3a097fa640 1 -- 172.21.15.12:0/948534311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3a0c120f10 msgr2=0x7f3a0c121320 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:39:38.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.045+0000 7f3a097fa640 1 --2- 172.21.15.12:0/948534311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3a0c120f10 0x7f3a0c121320 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3a00017000 tx=0x7f3a00004940 comp rx=0 tx=0).stop 2024-09-11T01:39:38.052 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:39:38.052 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:39:38.052 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:39:38.052 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:39:38.053 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 100, 2024-09-11T01:39:38.053 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 23841, 2024-09-11T01:39:38.053 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:39:38.053 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:39:38.053 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:39:38.053 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:39:38.053 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:38.053 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:39:38.053 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:39:38.053 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:39:38.053 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:39:38.053 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:39:38.053 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:39:38.053 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:39:38.054 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:39:38.054 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:39:38.054 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:39:38.054 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:39:38.054 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:39:38.054 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:39:38.054 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:39:38.054 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:39:38.054 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:38.054 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:39:38.054 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:39:38.054 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:39:38.054 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:39:38.054 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:39:38.055 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:39:38.055 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:39:38.055 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:39:38.055 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:39:38.055 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:39:38.055 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:39:38.055 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:39:38.055 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:39:38.055 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:38.055 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:38.055 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:39:38.055 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:39:38.055 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:39:38.055 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:39:38.056 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:39:38.056 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:39:38.056 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:39:38.056 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:39:38.056 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:39:38.056 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:39:38.056 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:39:38.056 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:39:38.056 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:39:38.056 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:39:38.056 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:39:38.056 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:39:38.056 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:39:38.057 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:39:38.057 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:39:38.057 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:39:38.057 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:39:38.057 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:39:38.057 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:39:38.057 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:39:38.057 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:39:38.057 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:39:38.057 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:39:38.057 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:39:38.057 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:38.057 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:39:38.058 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:38.058 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:39:38.058 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:38.058 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:39:38.058 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:39:38.058 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:39:38.058 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:39:38.058 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:38.058 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:39:38.058 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:39:38.058 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:38.058 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:38.058 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:38.059 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:39:38.059 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:39:38.059 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:38.059 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:38.059 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:38.059 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:38.059 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:39:38.059 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:39:38.059 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:39:38.059 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:39:38.059 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:39:38.059 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:38.060 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:38.060 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:39:38.060 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:39:38.060 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:39:38.060 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:39:38.060 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:38.060 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:39:38.060 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:39:38.060 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:38.060 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:38.060 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:38.060 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:39:38.060 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:39:38.061 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:38.061 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:38.061 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:38.061 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:38.061 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:39:38.061 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:39:38.061 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:39:38.061 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:39:38.061 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:39:38.061 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:38.061 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:38.061 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:39:38.062 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:39:38.062 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:39:38.062 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:39:38.062 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:38.062 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:39:38.062 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:39:38.062 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:38.062 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:38.062 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:38.062 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:39:38.062 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:39:38.062 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:38.062 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:38.063 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:38.063 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:38.063 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:39:38.063 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:39:38.063 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:39:38.063 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:39:38.063 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:39:38.063 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:38.063 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:38.063 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:38.063 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:39:38.063 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:39:38.063 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:38.063 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:39:38.064 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:39:38.064 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:39:38.064 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:38.064 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:38.064 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:39:38.064 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:38.064 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:39:38.064 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:39:38.064 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:39:38.064 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:38.064 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:38.064 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:38.064 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:39:38.065 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:39:38.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.045+0000 7f3a12ff8640 1 -- 172.21.15.12:0/948534311 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f39f003e240 msgr2=0x7f39f0040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:38.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.045+0000 7f3a12ff8640 1 --2- 172.21.15.12:0/948534311 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f39f003e240 0x7f39f0040700 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c0a72b0 tx=0x7f39f802e000 comp rx=0 tx=0).stop 2024-09-11T01:39:38.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.045+0000 7f3a12ff8640 10 monclient: shutdown 2024-09-11T01:39:38.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.045+0000 7f3a12ff8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:38.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.045+0000 7f3a12ff8640 1 -- 172.21.15.12:0/948534311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a0c1355c0 msgr2=0x7f3a0c120490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:38.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.045+0000 7f3a12ff8640 1 --2- 172.21.15.12:0/948534311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a0c1355c0 0x7f3a0c120490 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400c780 tx=0x7f3a0400cc40 comp rx=0 tx=0).stop 2024-09-11T01:39:38.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.045+0000 7f3a11595640 1 -- 172.21.15.12:0/948534311 reap_dead start 2024-09-11T01:39:38.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.045+0000 7f3a12ff8640 1 -- 172.21.15.12:0/948534311 shutdown_connections 2024-09-11T01:39:38.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.045+0000 7f3a12ff8640 1 --2- 172.21.15.12:0/948534311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3a0c1355c0 0x7f3a0c120490 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:38.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.045+0000 7f3a12ff8640 1 -- 172.21.15.12:0/948534311 >> 172.21.15.12:0/948534311 conn(0x7f3a0c0a2f70 msgr2=0x7f3a0c09d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:38.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.045+0000 7f3a12ff8640 1 -- 172.21.15.12:0/948534311 shutdown_connections 2024-09-11T01:39:38.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.046+0000 7f3a12ff8640 1 -- 172.21.15.12:0/948534311 wait complete. 2024-09-11T01:39:38.067 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:39:38.067 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:39:38.067 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:39:38.068 INFO:tasks.rados.rados.0.smithi071.stdout:1529: writing smithi07134777-38 from 464025 to 1258675 tid 1 2024-09-11T01:39:38.068 INFO:tasks.rados.rados.0.smithi071.stdout:1529: writing smithi07134777-38 from 1947852 to 2352041 tid 2 2024-09-11T01:39:38.068 INFO:tasks.rados.rados.0.smithi071.stdout:1529: writing smithi07134777-38 from 2990809 to 3416334 tid 3 2024-09-11T01:39:38.068 INFO:tasks.rados.rados.0.smithi071.stdout:1530: snap_remove snap 161 2024-09-11T01:39:38.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.168+0000 7f9facc7f640 10 monclient: get_monmap_and_config 2024-09-11T01:39:38.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.168+0000 7f9facc7f640 10 monclient: build_initial_monmap 2024-09-11T01:39:38.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.168+0000 7f9facc7f640 10 monclient: monmap: 2024-09-11T01:39:38.170 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:38.170 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:38.170 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:39:38.169611+0000 2024-09-11T01:39:38.170 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:39:38.169611+0000 2024-09-11T01:39:38.170 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:38.170 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:38.171 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:39:38.171 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:39:38.171 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:39:38.171 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:39:38.171 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:39:38.171 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:38.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.169+0000 7f9facc7f640 1 Processor -- start 2024-09-11T01:39:38.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.169+0000 7f9facc7f640 1 -- start start 2024-09-11T01:39:38.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.169+0000 7f9facc7f640 10 monclient: init 2024-09-11T01:39:38.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9facc7f640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:38.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9facc7f640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-11T01:39:38.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9facc7f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fa80a2db0 0x7f9fa806f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:38.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9facc7f640 10 monclient(hunting): picked mon.noname-b con 0x7f9fa80a2db0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:38.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9facc7f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9fa806ded0 0x7f9fa80a3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:38.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9facc7f640 10 monclient(hunting): picked mon.noname-a con 0x7f9fa806ded0 addr v2:172.21.15.12:3300/0 2024-09-11T01:39:38.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9facc7f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9fa80a36d0 addr v1:172.21.15.71:6789/0 2024-09-11T01:39:38.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9facc7f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fa80a4650 con 0x7f9fa80a36d0 2024-09-11T01:39:38.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9facc7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:38.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9facc7f640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f9fa806f1a0 con 0x7f9fa806ded0 2024-09-11T01:39:38.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9facc7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:38.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9facc7f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f9fa806eec0 con 0x7f9fa80a2db0 2024-09-11T01:39:38.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9facc7f640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:38.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9facc7f640 10 monclient(hunting): authenticate will time out at 1384.612208s 2024-09-11T01:39:38.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fa80a2db0 0x7f9fa806f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa5d9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9fa806ded0 0x7f9fa80a3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa5d9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9fa806ded0 0x7f9fa80a3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43608/0 (socket says 172.21.15.12:43608) 2024-09-11T01:39:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fa80a2db0 0x7f9fa806f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:47766/0 (socket says 172.21.15.12:47766) 2024-09-11T01:39:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa5d9b640 1 -- 172.21.15.12:0/3469034573 learned_addr learned my addr 172.21.15.12:0/3469034573 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:39:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa5d9b640 10 monclient(hunting): get_auth_request con 0x7f9fa806ded0 auth_method 0 2024-09-11T01:39:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa659c640 10 monclient(hunting): get_auth_request con 0x7f9fa80a2db0 auth_method 0 2024-09-11T01:39:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:38.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:38.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:38.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:38.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:38.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa659c640 10 monclient(hunting): handle_auth_done global_id 114152 payload 274 2024-09-11T01:39:38.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa659c640 1 -- 172.21.15.12:0/3469034573 >> v2:172.21.15.12:3300/0 conn(0x7f9fa806ded0 msgr2=0x7f9fa80a3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:39:38.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa659c640 1 --2- 172.21.15.12:0/3469034573 >> v2:172.21.15.12:3300/0 conn(0x7f9fa806ded0 0x7f9fa80a3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:38.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa659c640 1 -- 172.21.15.12:0/3469034573 >> v1:172.21.15.71:6789/0 conn(0x7f9fa80a36d0 legacy=0x7f9fa8137f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:39:38.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa659c640 10 monclient: _finish_hunting 0 2024-09-11T01:39:38.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa659c640 1 monclient: found mon.noname-b 2024-09-11T01:39:38.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:38.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:39:38.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa659c640 1 -- 172.21.15.12:0/3469034573 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fa806ebe0 con 0x7f9fa80a2db0 2024-09-11T01:39:38.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.170+0000 7f9fa5d9b640 1 --2- 172.21.15.12:0/3469034573 >> v2:172.21.15.12:3300/0 conn(0x7f9fa806ded0 0x7f9fa80a3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:39:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.173+0000 7f9fa659c640 1 -- 172.21.15.12:0/3469034573 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fa80a2db0 msgr2=0x7f9fa806f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.173+0000 7f9fa659c640 1 --2- 172.21.15.12:0/3469034573 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fa80a2db0 0x7f9fa806f8a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9f980097a0 tx=0x7f9f9802a050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8846e6162c3e1bb0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.173+0000 7f9fa559a640 1 -- 172.21.15.12:0/3469034573 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f98037070 con 0x7f9fa80a2db0 2024-09-11T01:39:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.173+0000 7f9fa559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.173+0000 7f9fa559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:39:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.173+0000 7f9fa559a640 10 monclient: dump: 2024-09-11T01:39:38.182 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:38.183 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.173+0000 7f9fa559a640 10 monclient: _finish_auth 0 2024-09-11T01:39:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.173+0000 7f9fa559a640 10 monclient: _check_auth_tickets 2024-09-11T01:39:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.173+0000 7f9fa559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.173+0000 7f9fa559a640 1 -- 172.21.15.12:0/3469034573 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f98032440 con 0x7f9fa80a2db0 2024-09-11T01:39:38.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.173+0000 7f9facc7f640 5 monclient: authenticate success, global_id 114152 2024-09-11T01:39:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.173+0000 7f9facc7f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:39:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.173+0000 7f9fa559a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.173+0000 7f9facc7f640 10 monclient: get_monmap_and_config success 2024-09-11T01:39:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.173+0000 7f9fa559a640 1 -- 172.21.15.12:0/3469034573 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f98032810 con 0x7f9fa80a2db0 2024-09-11T01:39:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.173+0000 7f9fa559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.173+0000 7f9facc7f640 4 set_mon_vals no callback set 2024-09-11T01:39:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.173+0000 7f9fa559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:39:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.173+0000 7f9fa559a640 10 monclient: dump: 2024-09-11T01:39:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:38.184 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:38.185 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.174+0000 7f9facc7f640 10 monclient: shutdown 2024-09-11T01:39:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.174+0000 7f9facc7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.174+0000 7f9facc7f640 1 -- 172.21.15.12:0/3469034573 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fa80a2db0 msgr2=0x7f9fa806f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.174+0000 7f9facc7f640 1 --2- 172.21.15.12:0/3469034573 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fa80a2db0 0x7f9fa806f8a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9f980097a0 tx=0x7f9f9802a050 comp rx=0 tx=0).stop 2024-09-11T01:39:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.174+0000 7f9facc7f640 1 -- 172.21.15.12:0/3469034573 shutdown_connections 2024-09-11T01:39:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.174+0000 7f9facc7f640 1 --2- 172.21.15.12:0/3469034573 >> v2:172.21.15.12:3300/0 conn(0x7f9fa806ded0 0x7f9fa80a3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.174+0000 7f9facc7f640 1 --2- 172.21.15.12:0/3469034573 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fa80a2db0 0x7f9fa806f8a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.174+0000 7f9facc7f640 1 -- 172.21.15.12:0/3469034573 >> 172.21.15.12:0/3469034573 conn(0x7f9fa80a83b0 msgr2=0x7f9fa80a87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.174+0000 7f9facc7f640 1 -- 172.21.15.12:0/3469034573 shutdown_connections 2024-09-11T01:39:38.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.174+0000 7f9facc7f640 1 -- 172.21.15.12:0/3469034573 wait complete. 2024-09-11T01:39:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.174+0000 7f9facc7f640 10 monclient: build_initial_monmap 2024-09-11T01:39:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.174+0000 7f9facc7f640 10 monclient: monmap: 2024-09-11T01:39:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:39:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:39:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:39:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:39:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:39:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:39:38.186 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9facc7f640 1 Processor -- start 2024-09-11T01:39:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9facc7f640 1 -- start start 2024-09-11T01:39:38.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9facc7f640 10 monclient: init 2024-09-11T01:39:38.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9facc7f640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:38.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9facc7f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:39:38.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9facc7f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fa806ded0 0x7f9fa81cd400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:38.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9facc7f640 10 monclient(hunting): picked mon.noname-c con 0x7f9fa806ded0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:38.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9facc7f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fa80a2db0 0x7f9fa81cd940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:38.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9facc7f640 10 monclient(hunting): picked mon.noname-a con 0x7f9fa80a2db0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:39:38.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9facc7f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fa80a36d0 0x7f9fa8125020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:38.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9facc7f640 10 monclient(hunting): picked mon.noname-b con 0x7f9fa80a36d0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:39:38.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9facc7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:38.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9facc7f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa81387f0 con 0x7f9fa80a2db0 2024-09-11T01:39:38.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9facc7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:38.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9facc7f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa8138670 con 0x7f9fa80a36d0 2024-09-11T01:39:38.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9facc7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:38.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9facc7f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f9fa8138970 con 0x7f9fa806ded0 2024-09-11T01:39:38.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9facc7f640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:38.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9facc7f640 10 monclient(hunting): authenticate will time out at 1384.617801s 2024-09-11T01:39:38.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9fa5d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fa80a2db0 0x7f9fa81cd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:38.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9fa659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fa806ded0 0x7f9fa81cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:38.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9fa5d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fa80a2db0 0x7f9fa81cd940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43612/0 (socket says 172.21.15.12:43612) 2024-09-11T01:39:38.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9fa5d9b640 1 -- 172.21.15.12:0/4194047423 learned_addr learned my addr 172.21.15.12:0/4194047423 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:39:38.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9fa5d9b640 10 monclient(hunting): get_auth_request con 0x7f9fa80a2db0 auth_method 0 2024-09-11T01:39:38.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9fa5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:38.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9fa659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fa806ded0 0x7f9fa81cd400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:47776/0 (socket says 172.21.15.12:47776) 2024-09-11T01:39:38.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9fa5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:38.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9fa5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:38.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9fa659c640 10 monclient(hunting): get_auth_request con 0x7f9fa806ded0 auth_method 0 2024-09-11T01:39:38.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9fa659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9fa659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9fa659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9fa5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.175+0000 7f9fa5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.176+0000 7f9fa5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.176+0000 7f9fa659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.176+0000 7f9fa659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.176+0000 7f9fa659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.176+0000 7f9fa5d9b640 10 monclient(hunting): handle_auth_done global_id 94317 payload 824 2024-09-11T01:39:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.176+0000 7f9fa6d9d640 1 --2- 172.21.15.12:0/4194047423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fa80a36d0 0x7f9fa8125020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.176+0000 7f9fa5d9b640 1 -- 172.21.15.12:0/4194047423 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fa806ded0 msgr2=0x7f9fa81cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.176+0000 7f9fa5d9b640 1 --2- 172.21.15.12:0/4194047423 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fa806ded0 0x7f9fa81cd400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.176+0000 7f9fa5d9b640 1 -- 172.21.15.12:0/4194047423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fa80a36d0 msgr2=0x7f9fa8125020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.176+0000 7f9fa5d9b640 1 --2- 172.21.15.12:0/4194047423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fa80a36d0 0x7f9fa8125020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:38.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.176+0000 7f9fa5d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:39:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.176+0000 7f9fa5d9b640 1 monclient: found mon.noname-a 2024-09-11T01:39:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.176+0000 7f9fa5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.176+0000 7f9fa5d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:39:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.176+0000 7f9fa5d9b640 1 -- 172.21.15.12:0/4194047423 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fa80a5820 con 0x7f9fa80a2db0 2024-09-11T01:39:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.176+0000 7f9fa659c640 1 --2- 172.21.15.12:0/4194047423 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fa806ded0 0x7f9fa81cd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:39:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.178+0000 7f9fa5d9b640 1 -- 172.21.15.12:0/4194047423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fa80a2db0 msgr2=0x7f9fa81cd940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.178+0000 7f9fa5d9b640 1 --2- 172.21.15.12:0/4194047423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fa80a2db0 0x7f9fa81cd940 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9f9000ea00 tx=0x7f9f9000eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.178+0000 7f9f9effd640 1 -- 172.21.15.12:0/4194047423 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f90017070 con 0x7f9fa80a2db0 2024-09-11T01:39:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.178+0000 7f9f9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.178+0000 7f9f9effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:39:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.178+0000 7f9f9effd640 10 monclient: dump: 2024-09-11T01:39:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:38.190 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.191 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.191 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:38.191 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:38.191 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:38.191 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:38.191 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:38.191 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:38.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.178+0000 7f9f9effd640 10 monclient: _finish_auth 0 2024-09-11T01:39:38.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.178+0000 7f9f9effd640 10 monclient: _check_auth_tickets 2024-09-11T01:39:38.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.178+0000 7f9f9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:38.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.178+0000 7f9facc7f640 5 monclient: authenticate success, global_id 94317 2024-09-11T01:39:38.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.178+0000 7f9facc7f640 10 monclient: _renew_subs 2024-09-11T01:39:38.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.178+0000 7f9f9effd640 1 -- 172.21.15.12:0/4194047423 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f9000cc10 con 0x7f9fa80a2db0 2024-09-11T01:39:38.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.178+0000 7f9facc7f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:38.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.178+0000 7f9facc7f640 1 -- 172.21.15.12:0/4194047423 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fa809f5a0 con 0x7f9fa80a2db0 2024-09-11T01:39:38.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.178+0000 7f9f9effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:38.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.178+0000 7f9f9effd640 1 -- 172.21.15.12:0/4194047423 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f90013470 con 0x7f9fa80a2db0 2024-09-11T01:39:38.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.178+0000 7f9f9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:38.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.178+0000 7f9f9effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:39:38.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.178+0000 7f9f9effd640 10 monclient: dump: 2024-09-11T01:39:38.192 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:38.192 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:38.192 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.192 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.192 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:38.192 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:38.192 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:38.192 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:38.193 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:38.193 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:38.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.178+0000 7f9facc7f640 10 monclient: _renew_subs 2024-09-11T01:39:38.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.178+0000 7f9facc7f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:38.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.178+0000 7f9facc7f640 1 -- 172.21.15.12:0/4194047423 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fa8128df0 con 0x7f9fa80a2db0 2024-09-11T01:39:38.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.178+0000 7f9fa4d99640 4 set_mon_vals no callback set 2024-09-11T01:39:38.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.179+0000 7f9facc7f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:38.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.179+0000 7f9facc7f640 1 --2- 172.21.15.12:0/4194047423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fa8125d20 0x7f9fa802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:38.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.179+0000 7f9facc7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:38.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.179+0000 7f9facc7f640 1 -- 172.21.15.12:0/4194047423 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa806a060 con 0x7f9fa8125d20 2024-09-11T01:39:38.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.179+0000 7f9f9effd640 1 -- 172.21.15.12:0/4194047423 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f9f90024060 con 0x7f9fa80a2db0 2024-09-11T01:39:38.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.179+0000 7f9f9effd640 1 --2- 172.21.15.12:0/4194047423 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9f8803e210 0x7f9f880406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:38.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.179+0000 7f9fa659c640 1 --2- 172.21.15.12:0/4194047423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fa8125d20 0x7f9fa802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:38.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.179+0000 7f9fa659c640 10 monclient: get_auth_request con 0x7f9fa8125d20 auth_method 0 2024-09-11T01:39:38.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.179+0000 7f9fa659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:38.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.179+0000 7f9fa659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:38.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.179+0000 7f9fa659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:38.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.179+0000 7f9f9effd640 1 -- 172.21.15.12:0/4194047423 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(347..347 src has 186..347) ==== 4783+0+0 (secure 0 0 0) 0x7f9f90052cc0 con 0x7f9fa80a2db0 2024-09-11T01:39:38.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.179+0000 7f9fa659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:38.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.179+0000 7f9fa659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:38.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.179+0000 7f9fa659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:38.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.180+0000 7f9fa6d9d640 1 --2- 172.21.15.12:0/4194047423 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9f8803e210 0x7f9f880406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:38.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.180+0000 7f9fa6d9d640 10 monclient: get_auth_request con 0x7f9f8803e210 auth_method 0 2024-09-11T01:39:38.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.180+0000 7f9fa659c640 10 monclient(hunting): handle_auth_done global_id 104203 payload 824 2024-09-11T01:39:38.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.180+0000 7f9fa659c640 1 -- 172.21.15.12:0/4194047423 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9fa8127650 con 0x7f9fa8125d20 2024-09-11T01:39:38.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.182+0000 7f9fa659c640 1 -- 172.21.15.12:0/4194047423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fa8125d20 msgr2=0x7f9fa802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.182+0000 7f9fa659c640 1 --2- 172.21.15.12:0/4194047423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fa8125d20 0x7f9fa802d440 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9fa806e860 tx=0x7f9f980046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.182+0000 7f9fa6d9d640 1 --2- 172.21.15.12:0/4194047423 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9f8803e210 0x7f9f880406d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9fa0007fd0 tx=0x7f9fa0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.182+0000 7f9f9effd640 1 -- 172.21.15.12:0/4194047423 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f98043070 con 0x7f9fa8125d20 2024-09-11T01:39:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.182+0000 7f9f9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.182+0000 7f9f9effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:39:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.182+0000 7f9f9effd640 10 monclient: dump: 2024-09-11T01:39:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:38.197 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:38.198 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.183+0000 7f9f9effd640 1 -- 172.21.15.12:0/4194047423 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9f9803ca60 con 0x7f9fa8125d20 2024-09-11T01:39:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.183+0000 7f9f9effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.183+0000 7f9f9effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:39:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.183+0000 7f9f9effd640 1 -- 172.21.15.12:0/4194047423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fa8125d20 msgr2=0x7f9fa802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.183+0000 7f9f9effd640 1 --2- 172.21.15.12:0/4194047423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fa8125d20 0x7f9fa802d440 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9fa806e860 tx=0x7f9f980046c0 comp rx=0 tx=0).stop 2024-09-11T01:39:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.183+0000 7f9f9effd640 1 -- 172.21.15.12:0/4194047423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fa8125d20 msgr2=0x7f9fa802d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:39:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.183+0000 7f9f9effd640 1 --2- 172.21.15.12:0/4194047423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fa8125d20 0x7f9fa802d440 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.189+0000 7f9facc7f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:39:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.189+0000 7f9facc7f640 1 --2- 172.21.15.12:0/4194047423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fa8126100 0x7f9fa802d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.189+0000 7f9facc7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.189+0000 7f9facc7f640 1 -- 172.21.15.12:0/4194047423 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa8138d40 con 0x7f9fa8126100 2024-09-11T01:39:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.189+0000 7f9fa659c640 1 --2- 172.21.15.12:0/4194047423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fa8126100 0x7f9fa802d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.189+0000 7f9fa659c640 10 monclient: get_auth_request con 0x7f9fa8126100 auth_method 0 2024-09-11T01:39:38.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.189+0000 7f9fa659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:38.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.189+0000 7f9fa659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:38.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.189+0000 7f9fa659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:38.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.189+0000 7f9fa659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:38.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.189+0000 7f9fa659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:38.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.189+0000 7f9fa659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:38.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.189+0000 7f9fa659c640 10 monclient(hunting): handle_auth_done global_id 104206 payload 824 2024-09-11T01:39:38.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.189+0000 7f9fa659c640 1 -- 172.21.15.12:0/4194047423 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9fa809d9c0 con 0x7f9fa8126100 2024-09-11T01:39:38.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.192+0000 7f9fa659c640 1 -- 172.21.15.12:0/4194047423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fa8126100 msgr2=0x7f9fa802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:38.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.192+0000 7f9fa659c640 1 --2- 172.21.15.12:0/4194047423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fa8126100 0x7f9fa802d980 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9f9802a090 tx=0x7f9f98008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:38.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.192+0000 7f9f9effd640 1 -- 172.21.15.12:0/4194047423 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f98054020 con 0x7f9fa8126100 2024-09-11T01:39:38.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.192+0000 7f9f9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:38.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.192+0000 7f9f9effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:39:38.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.192+0000 7f9f9effd640 10 monclient: dump: 2024-09-11T01:39:38.200 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:38.200 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:38.200 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.200 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.200 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:38.200 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:38.200 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:38.200 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:38.200 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:38.200 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:38.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.192+0000 7f9f9effd640 1 -- 172.21.15.12:0/4194047423 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7f9f9803b7c0 con 0x7f9fa8126100 2024-09-11T01:39:38.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.192+0000 7f9f9effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:39:38.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.192+0000 7f9f9effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:39:38.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.192+0000 7f9f9effd640 1 -- 172.21.15.12:0/4194047423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fa8126100 msgr2=0x7f9fa802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:38.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.192+0000 7f9f9effd640 1 --2- 172.21.15.12:0/4194047423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fa8126100 0x7f9fa802d980 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9f9802a090 tx=0x7f9f98008030 comp rx=0 tx=0).stop 2024-09-11T01:39:38.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.192+0000 7f9f9effd640 1 -- 172.21.15.12:0/4194047423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fa8126100 msgr2=0x7f9fa802d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:39:38.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.192+0000 7f9f9effd640 1 --2- 172.21.15.12:0/4194047423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fa8126100 0x7f9fa802d980 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9f9802a090 tx=0x7f9f98008030 comp rx=0 tx=0).stop 2024-09-11T01:39:38.201 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:39:38.201 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:39:38.201 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:39:38.201 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:39:38.201 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 100, 2024-09-11T01:39:38.201 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 78687, 2024-09-11T01:39:38.201 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:39:38.201 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:39:38.201 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:39:38.202 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:39:38.202 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:38.202 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 4, 2024-09-11T01:39:38.202 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:39:38.202 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:39:38.202 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:39:38.202 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:39:38.202 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:39:38.202 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:39:38.202 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:39:38.202 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:39:38.202 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:39:38.202 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:39:38.202 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:39:38.202 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:39:38.203 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:39:38.203 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:39:38.203 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:38.203 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:39:38.203 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:39:38.203 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:39:38.203 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:39:38.203 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:39:38.203 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:39:38.203 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:39:38.203 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:39:38.203 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:39:38.203 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:39:38.203 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:39:38.203 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:39:38.204 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:39:38.204 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:38.204 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:38.204 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:39:38.204 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:39:38.204 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:39:38.204 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:39:38.204 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:39:38.204 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:39:38.204 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:39:38.204 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:39:38.204 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:39:38.204 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:39:38.204 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:39:38.204 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:39:38.205 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:39:38.205 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:39:38.205 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:39:38.205 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:39:38.205 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:39:38.205 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:39:38.205 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:39:38.205 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:39:38.205 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:39:38.205 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:39:38.205 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:39:38.208 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:39:38.208 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:39:38.208 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:39:38.208 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:39:38.208 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:39:38.208 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:38.209 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:39:38.209 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:38.209 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:39:38.209 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:38.209 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:39:38.209 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:39:38.209 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:39:38.209 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:39:38.209 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:38.209 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:39:38.209 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:39:38.209 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:38.209 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:38.210 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:38.210 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:39:38.210 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:39:38.210 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:38.210 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:38.210 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:38.210 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:38.210 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:39:38.210 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:39:38.210 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:39:38.210 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:39:38.210 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:39:38.210 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:38.210 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:38.211 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:39:38.211 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:39:38.211 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:39:38.211 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:39:38.211 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:38.211 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:39:38.211 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:39:38.211 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:38.211 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:38.211 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:38.211 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:39:38.211 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:39:38.211 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:38.211 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:38.212 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:38.212 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:38.212 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:39:38.212 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:39:38.212 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:39:38.212 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:39:38.212 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:39:38.212 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:38.212 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:38.212 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:39:38.212 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:39:38.212 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:39:38.212 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:39:38.213 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:38.213 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:39:38.213 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:39:38.213 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:38.213 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:38.213 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:38.213 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:39:38.213 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:39:38.213 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:38.213 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:38.213 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:38.213 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:38.213 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:39:38.213 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:39:38.213 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:39:38.214 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:39:38.214 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:39:38.214 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:38.214 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:38.214 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:38.214 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:39:38.214 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:39:38.214 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:38.214 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:39:38.214 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:39:38.214 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:39:38.214 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:38.214 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:38.214 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:39:38.214 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:38.214 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:39:38.215 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:39:38.215 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:39:38.215 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:38.215 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:38.215 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:39:38.215 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:38.215 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:39:38.215 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:39:38.215 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:39:38.215 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:38.215 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:38.215 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:39:38.215 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:38.215 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:39:38.216 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:39:38.216 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:39:38.216 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:38.216 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:38.216 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:38.216 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:39:38.216 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:39:38.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.192+0000 7f9facc7f640 1 -- 172.21.15.12:0/4194047423 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9f8803e210 msgr2=0x7f9f880406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:38.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.192+0000 7f9facc7f640 1 --2- 172.21.15.12:0/4194047423 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9f8803e210 0x7f9f880406d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9fa0007fd0 tx=0x7f9fa0009040 comp rx=0 tx=0).stop 2024-09-11T01:39:38.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.192+0000 7f9facc7f640 10 monclient: shutdown 2024-09-11T01:39:38.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.192+0000 7f9facc7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:38.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.192+0000 7f9facc7f640 1 -- 172.21.15.12:0/4194047423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fa80a2db0 msgr2=0x7f9fa81cd940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:38.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.192+0000 7f9fa6d9d640 1 -- 172.21.15.12:0/4194047423 reap_dead start 2024-09-11T01:39:38.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.192+0000 7f9facc7f640 1 --2- 172.21.15.12:0/4194047423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fa80a2db0 0x7f9fa81cd940 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9f9000ea00 tx=0x7f9f9000eec0 comp rx=0 tx=0).stop 2024-09-11T01:39:38.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.192+0000 7f9facc7f640 1 -- 172.21.15.12:0/4194047423 shutdown_connections 2024-09-11T01:39:38.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.192+0000 7f9facc7f640 1 --2- 172.21.15.12:0/4194047423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fa80a2db0 0x7f9fa81cd940 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:38.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.193+0000 7f9facc7f640 1 -- 172.21.15.12:0/4194047423 >> 172.21.15.12:0/4194047423 conn(0x7f9fa80a83b0 msgr2=0x7f9fa80a17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:38.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.193+0000 7f9facc7f640 1 -- 172.21.15.12:0/4194047423 shutdown_connections 2024-09-11T01:39:38.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.193+0000 7f9facc7f640 1 -- 172.21.15.12:0/4194047423 wait complete. 2024-09-11T01:39:38.218 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:39:38.219 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-11T01:39:38.219 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T01:39:38.219 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-11T01:39:38.219 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-11T01:39:38.219 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-11T01:39:38.219 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:39:38.233 INFO:tasks.ceph.mon.b:Stopped 2024-09-11T01:39:38.233 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:39:38.233 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:39:38.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.331+0000 7f60e334b640 10 monclient: get_monmap_and_config 2024-09-11T01:39:38.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.331+0000 7f60e334b640 10 monclient: build_initial_monmap 2024-09-11T01:39:38.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.331+0000 7f60e334b640 10 monclient: monmap: 2024-09-11T01:39:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:39:38.332012+0000 2024-09-11T01:39:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:39:38.332012+0000 2024-09-11T01:39:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:39:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:39:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:39:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:39:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:39:38.333 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:38.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.331+0000 7f60e334b640 1 Processor -- start 2024-09-11T01:39:38.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e334b640 1 -- start start 2024-09-11T01:39:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e334b640 10 monclient: init 2024-09-11T01:39:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e334b640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e334b640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-11T01:39:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e334b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f60dc138230 0x7f60dc138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e334b640 10 monclient(hunting): picked mon.noname-c con 0x7f60dc138230 addr v2:172.21.15.71:3300/0 2024-09-11T01:39:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e334b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f60dc09d030 addr v1:172.21.15.12:6789/0 2024-09-11T01:39:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e334b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f60dc09d460 0x7f60dc0a5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e334b640 10 monclient(hunting): picked mon.noname-a con 0x7f60dc09d460 addr v2:172.21.15.12:3300/0 2024-09-11T01:39:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e334b640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60dc0a6e30 con 0x7f60dc09d030 2024-09-11T01:39:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e334b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e334b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f60dc09c9c0 con 0x7f60dc09d460 2024-09-11T01:39:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e334b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e334b640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f60dc09ba20 con 0x7f60dc138230 2024-09-11T01:39:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e334b640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e334b640 10 monclient(hunting): authenticate will time out at 1384.774600s 2024-09-11T01:39:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e10e7640 1 -- >> v2:172.21.15.71:3300/0 conn(0x7f60dc138230 msgr2=0x7f60dc138610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:39:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e10e7640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f60dc138230 0x7f60dc138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:39:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e18e8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f60dc09d460 0x7f60dc0a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e08e6640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f60dc09d030 0x7f60dc0a5020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:41924/0 (socket says 172.21.15.12:41924) 2024-09-11T01:39:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e08e6640 1 -- 172.21.15.12:0/1690354907 learned_addr learned my addr 172.21.15.12:0/1690354907 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:39:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e18e8640 10 monclient(hunting): get_auth_request con 0x7f60dc09d460 auth_method 0 2024-09-11T01:39:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e18e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e18e8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e18e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e18e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:38.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e18e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:38.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.332+0000 7f60e18e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:38.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.333+0000 7f60e18e8640 10 monclient(hunting): handle_auth_done global_id 94320 payload 274 2024-09-11T01:39:38.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.333+0000 7f60e18e8640 1 -- 172.21.15.12:0/1690354907 >> v2:172.21.15.71:3300/0 conn(0x7f60dc138230 msgr2=0x7f60dc138610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:39:38.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.333+0000 7f60e18e8640 1 --2- 172.21.15.12:0/1690354907 >> v2:172.21.15.71:3300/0 conn(0x7f60dc138230 0x7f60dc138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:38.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.333+0000 7f60e18e8640 1 -- 172.21.15.12:0/1690354907 >> v1:172.21.15.12:6789/0 conn(0x7f60dc09d030 legacy=0x7f60dc0a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:39:38.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.333+0000 7f60e18e8640 10 monclient: _finish_hunting 0 2024-09-11T01:39:38.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.333+0000 7f60e18e8640 1 monclient: found mon.noname-a 2024-09-11T01:39:38.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.333+0000 7f60e18e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:38.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.333+0000 7f60e18e8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:39:38.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.333+0000 7f60e18e8640 1 -- 172.21.15.12:0/1690354907 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60dc0a66a0 con 0x7f60dc09d460 2024-09-11T01:39:38.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.335+0000 7f60e18e8640 1 -- 172.21.15.12:0/1690354907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60dc09d460 msgr2=0x7f60dc0a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:38.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.335+0000 7f60e18e8640 1 --2- 172.21.15.12:0/1690354907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60dc09d460 0x7f60dc0a5730 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f60dc0a17e0 tx=0x7f60d402a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d30b32dfbc60d8b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:38.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.336+0000 7f60dbfff640 1 -- 172.21.15.12:0/1690354907 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d4037070 con 0x7f60dc09d460 2024-09-11T01:39:38.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.336+0000 7f60dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:38.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.336+0000 7f60dbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:39:38.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.336+0000 7f60dbfff640 10 monclient: dump: 2024-09-11T01:39:38.343 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:38.343 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:38.343 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.343 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.343 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:38.343 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:38.343 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:38.344 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:38.344 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:38.344 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:38.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.336+0000 7f60dbfff640 10 monclient: _finish_auth 0 2024-09-11T01:39:38.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.336+0000 7f60dbfff640 10 monclient: _check_auth_tickets 2024-09-11T01:39:38.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.336+0000 7f60dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:38.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.336+0000 7f60dbfff640 1 -- 172.21.15.12:0/1690354907 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60d4032640 con 0x7f60dc09d460 2024-09-11T01:39:38.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.336+0000 7f60e334b640 5 monclient: authenticate success, global_id 94320 2024-09-11T01:39:38.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.336+0000 7f60dbfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:38.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.336+0000 7f60dbfff640 1 -- 172.21.15.12:0/1690354907 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d4032a10 con 0x7f60dc09d460 2024-09-11T01:39:38.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.336+0000 7f60dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:38.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.336+0000 7f60dbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:39:38.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.336+0000 7f60dbfff640 10 monclient: dump: 2024-09-11T01:39:38.345 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:38.345 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:38.345 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.345 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.345 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:38.345 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:38.345 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:38.345 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:38.345 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:38.345 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:38.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.336+0000 7f60e334b640 10 monclient: get_monmap_and_config success 2024-09-11T01:39:38.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.336+0000 7f60e334b640 4 set_mon_vals no callback set 2024-09-11T01:39:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.336+0000 7f60e334b640 10 monclient: shutdown 2024-09-11T01:39:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.336+0000 7f60e334b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.336+0000 7f60e334b640 1 -- 172.21.15.12:0/1690354907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60dc09d460 msgr2=0x7f60dc0a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.336+0000 7f60e334b640 1 --2- 172.21.15.12:0/1690354907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60dc09d460 0x7f60dc0a5730 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f60dc0a17e0 tx=0x7f60d402a000 comp rx=0 tx=0).stop 2024-09-11T01:39:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.336+0000 7f60e334b640 1 -- 172.21.15.12:0/1690354907 shutdown_connections 2024-09-11T01:39:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.336+0000 7f60e334b640 1 --2- 172.21.15.12:0/1690354907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60dc09d460 0x7f60dc0a5730 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.336+0000 7f60e334b640 1 --2- 172.21.15.12:0/1690354907 >> v2:172.21.15.71:3300/0 conn(0x7f60dc138230 0x7f60dc138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.336+0000 7f60e334b640 1 -- 172.21.15.12:0/1690354907 >> 172.21.15.12:0/1690354907 conn(0x7f60dc0a7120 msgr2=0x7f60dc0a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.336+0000 7f60e334b640 1 -- 172.21.15.12:0/1690354907 shutdown_connections 2024-09-11T01:39:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.336+0000 7f60e334b640 1 -- 172.21.15.12:0/1690354907 wait complete. 2024-09-11T01:39:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.337+0000 7f60e334b640 10 monclient: build_initial_monmap 2024-09-11T01:39:38.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.337+0000 7f60e334b640 10 monclient: monmap: 2024-09-11T01:39:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:39:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:39:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:39:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:39:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:39:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:39:38.347 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.337+0000 7f60e334b640 1 Processor -- start 2024-09-11T01:39:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.337+0000 7f60e334b640 1 -- start start 2024-09-11T01:39:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.337+0000 7f60e334b640 10 monclient: init 2024-09-11T01:39:38.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e334b640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:38.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e334b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:39:38.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e334b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60dc09d030 0x7f60dc1cce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:38.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e334b640 10 monclient(hunting): picked mon.noname-a con 0x7f60dc09d030 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:39:38.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e334b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60dc09d460 0x7f60dc136dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:38.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e334b640 10 monclient(hunting): picked mon.noname-b con 0x7f60dc09d460 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:39:38.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e334b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f60dc138230 0x7f60dc137310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:38.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e334b640 10 monclient(hunting): picked mon.noname-c con 0x7f60dc138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:38.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e334b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:38.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e334b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f60dc06f970 con 0x7f60dc09d030 2024-09-11T01:39:38.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e334b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:38.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e334b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f60dc06f7f0 con 0x7f60dc09d460 2024-09-11T01:39:38.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e334b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:38.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e334b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f60dc0a5c70 con 0x7f60dc138230 2024-09-11T01:39:38.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e334b640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:38.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e334b640 10 monclient(hunting): authenticate will time out at 1384.780154s 2024-09-11T01:39:38.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e10e7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60dc09d030 0x7f60dc1cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:38.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e10e7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60dc09d030 0x7f60dc1cce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43650/0 (socket says 172.21.15.12:43650) 2024-09-11T01:39:38.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e10e7640 1 -- 172.21.15.12:0/1704265782 learned_addr learned my addr 172.21.15.12:0/1704265782 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:39:38.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e10e7640 10 monclient(hunting): get_auth_request con 0x7f60dc09d030 auth_method 0 2024-09-11T01:39:38.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e10e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:38.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e10e7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:38.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e10e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:38.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e18e8640 1 --2- 172.21.15.12:0/1704265782 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f60dc138230 0x7f60dc137310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:38.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e08e6640 1 -- 172.21.15.12:0/1704265782 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60dc09d460 msgr2=0x7f60dc136dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:39:38.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e18e8640 10 monclient(hunting): get_auth_request con 0x7f60dc138230 auth_method 0 2024-09-11T01:39:38.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e18e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e18e8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e18e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e10e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e10e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e08e6640 1 --2- 172.21.15.12:0/1704265782 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60dc09d460 0x7f60dc136dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:39:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e10e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e18e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e18e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e18e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e10e7640 10 monclient(hunting): handle_auth_done global_id 94323 payload 824 2024-09-11T01:39:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e10e7640 1 -- 172.21.15.12:0/1704265782 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f60dc138230 msgr2=0x7f60dc137310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e10e7640 1 --2- 172.21.15.12:0/1704265782 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f60dc138230 0x7f60dc137310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e10e7640 1 -- 172.21.15.12:0/1704265782 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60dc09d460 msgr2=0x7f60dc136dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:39:38.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e10e7640 1 --2- 172.21.15.12:0/1704265782 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60dc09d460 0x7f60dc136dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e10e7640 10 monclient: _finish_hunting 0 2024-09-11T01:39:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e10e7640 1 monclient: found mon.noname-a 2024-09-11T01:39:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e10e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e10e7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:39:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.338+0000 7f60e10e7640 1 -- 172.21.15.12:0/1704265782 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60dc122430 con 0x7f60dc09d030 2024-09-11T01:39:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.340+0000 7f60e10e7640 1 -- 172.21.15.12:0/1704265782 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60dc09d030 msgr2=0x7f60dc1cce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.340+0000 7f60e10e7640 1 --2- 172.21.15.12:0/1704265782 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60dc09d030 0x7f60dc1cce70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00b9e0 tx=0x7f60cc00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.340+0000 7f60d9ffb640 1 -- 172.21.15.12:0/1704265782 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc00cd50 con 0x7f60dc09d030 2024-09-11T01:39:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.340+0000 7f60d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.340+0000 7f60d9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:39:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.340+0000 7f60d9ffb640 10 monclient: dump: 2024-09-11T01:39:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:38.351 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:38.352 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.352 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.352 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:38.352 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:38.352 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:38.352 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:38.352 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:38.352 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:38.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.340+0000 7f60d9ffb640 10 monclient: _finish_auth 0 2024-09-11T01:39:38.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.340+0000 7f60d9ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:39:38.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.340+0000 7f60d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:38.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.340+0000 7f60d9ffb640 1 -- 172.21.15.12:0/1704265782 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60cc010070 con 0x7f60dc09d030 2024-09-11T01:39:38.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.340+0000 7f60e334b640 5 monclient: authenticate success, global_id 94323 2024-09-11T01:39:38.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.340+0000 7f60d9ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:38.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.341+0000 7f60d9ffb640 1 -- 172.21.15.12:0/1704265782 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc0156b0 con 0x7f60dc09d030 2024-09-11T01:39:38.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.341+0000 7f60d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:38.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.341+0000 7f60d9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:39:38.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.341+0000 7f60d9ffb640 10 monclient: dump: 2024-09-11T01:39:38.353 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:38.353 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:38.353 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.353 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:38.353 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:38.353 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:38.353 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:38.353 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:38.353 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:38.354 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:38.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.341+0000 7f60e334b640 10 monclient: _renew_subs 2024-09-11T01:39:38.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.341+0000 7f60db7fe640 4 set_mon_vals no callback set 2024-09-11T01:39:38.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.341+0000 7f60e334b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:38.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.341+0000 7f60e334b640 1 -- 172.21.15.12:0/1704265782 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60dc123ef0 con 0x7f60dc09d030 2024-09-11T01:39:38.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.341+0000 7f60e334b640 10 monclient: _renew_subs 2024-09-11T01:39:38.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.341+0000 7f60e334b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:38.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.341+0000 7f60e334b640 1 -- 172.21.15.12:0/1704265782 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60dc122f90 con 0x7f60dc09d030 2024-09-11T01:39:38.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.342+0000 7f60d9ffb640 1 -- 172.21.15.12:0/1704265782 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f60cc013370 con 0x7f60dc09d030 2024-09-11T01:39:38.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.342+0000 7f60e334b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:38.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.342+0000 7f60e334b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:38.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.342+0000 7f60e334b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:38.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.342+0000 7f60e334b640 1 -- 172.21.15.12:0/1704265782 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60dc138610 con 0x7f60dc09d030 2024-09-11T01:39:38.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.342+0000 7f60d9ffb640 1 --2- 172.21.15.12:0/1704265782 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f60c803e220 0x7f60c80406e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:38.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.342+0000 7f60d9ffb640 1 -- 172.21.15.12:0/1704265782 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(347..347 src has 186..347) ==== 4783+0+0 (secure 0 0 0) 0x7f60cc052c80 con 0x7f60dc09d030 2024-09-11T01:39:38.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.343+0000 7f60e08e6640 1 --2- 172.21.15.12:0/1704265782 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f60c803e220 0x7f60c80406e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:38.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.343+0000 7f60e08e6640 10 monclient: get_auth_request con 0x7f60c803e220 auth_method 0 2024-09-11T01:39:38.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.345+0000 7f60e08e6640 1 --2- 172.21.15.12:0/1704265782 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f60c803e220 0x7f60c80406e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f60d0007fd0 tx=0x7f60d0007b20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:38.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.347+0000 7f60d9ffb640 1 -- 172.21.15.12:0/1704265782 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f60cc051680 con 0x7f60dc09d030 2024-09-11T01:39:38.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.347+0000 7f60d9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:38.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.347+0000 7f60d9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:39:38.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.601+0000 7f60e334b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:39:38.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.601+0000 7f60e334b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:39:38.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.601+0000 7f60e334b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:38.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.601+0000 7f60e334b640 1 -- 172.21.15.12:0/1704265782 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f60dc0698e0 con 0x7f60dc09d030 2024-09-11T01:39:38.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.601+0000 7f60d9ffb640 1 -- 172.21.15.12:0/1704265782 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f60cc01cd10 con 0x7f60dc09d030 2024-09-11T01:39:38.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.601+0000 7f60d9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:39:38.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.601+0000 7f60d9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:39:38.603 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":100,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:39:38.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.603+0000 7f60e334b640 1 -- 172.21.15.12:0/1704265782 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f60c803e220 msgr2=0x7f60c80406e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:38.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.603+0000 7f60e334b640 1 --2- 172.21.15.12:0/1704265782 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f60c803e220 0x7f60c80406e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f60d0007fd0 tx=0x7f60d0007b20 comp rx=0 tx=0).stop 2024-09-11T01:39:38.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.603+0000 7f60e334b640 10 monclient: shutdown 2024-09-11T01:39:38.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.603+0000 7f60e334b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:38.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.603+0000 7f60e334b640 1 -- 172.21.15.12:0/1704265782 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60dc09d030 msgr2=0x7f60dc1cce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:38.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.603+0000 7f60e334b640 1 --2- 172.21.15.12:0/1704265782 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60dc09d030 0x7f60dc1cce70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00b9e0 tx=0x7f60cc00bea0 comp rx=0 tx=0).stop 2024-09-11T01:39:38.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.603+0000 7f60e334b640 1 -- 172.21.15.12:0/1704265782 shutdown_connections 2024-09-11T01:39:38.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.603+0000 7f60e334b640 1 --2- 172.21.15.12:0/1704265782 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f60dc138230 0x7f60dc137310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:38.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.603+0000 7f60e334b640 1 --2- 172.21.15.12:0/1704265782 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60dc09d460 0x7f60dc136dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:38.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.603+0000 7f60e334b640 1 --2- 172.21.15.12:0/1704265782 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f60c803e220 0x7f60c80406e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:38.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.603+0000 7f60e334b640 1 --2- 172.21.15.12:0/1704265782 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60dc09d030 0x7f60dc1cce70 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:38.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.603+0000 7f60e334b640 1 -- 172.21.15.12:0/1704265782 >> 172.21.15.12:0/1704265782 conn(0x7f60dc0a7120 msgr2=0x7f60dc0a3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:38.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.603+0000 7f60e334b640 1 -- 172.21.15.12:0/1704265782 shutdown_connections 2024-09-11T01:39:38.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:38.603+0000 7f60e334b640 1 -- 172.21.15.12:0/1704265782 wait complete. 2024-09-11T01:39:41.616 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:39:41.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.728+0000 7f6ce306e640 10 monclient: get_monmap_and_config 2024-09-11T01:39:41.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.728+0000 7f6ce306e640 10 monclient: build_initial_monmap 2024-09-11T01:39:41.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.728+0000 7f6ce306e640 10 monclient: monmap: 2024-09-11T01:39:41.730 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:41.730 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:41.730 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:39:41.728820+0000 2024-09-11T01:39:41.730 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:39:41.728820+0000 2024-09-11T01:39:41.730 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:41.730 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:41.730 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:39:41.730 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:39:41.730 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:39:41.730 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:39:41.731 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:39:41.731 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:41.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.728+0000 7f6ce306e640 1 Processor -- start 2024-09-11T01:39:41.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.728+0000 7f6ce306e640 1 -- start start 2024-09-11T01:39:41.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.728+0000 7f6ce306e640 10 monclient: init 2024-09-11T01:39:41.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce306e640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:41.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce306e640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-11T01:39:41.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce306e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6cdc138210 0x7f6cdc1385f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:41.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce306e640 10 monclient(hunting): picked mon.noname-a con 0x7f6cdc138210 addr v2:172.21.15.12:3300/0 2024-09-11T01:39:41.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce306e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6cdc0a6810 addr v1:172.21.15.71:6789/0 2024-09-11T01:39:41.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce306e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6cdc0a57a0 addr v1:172.21.15.12:6789/0 2024-09-11T01:39:41.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce306e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cdc09ba20 con 0x7f6cdc0a57a0 2024-09-11T01:39:41.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce306e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cdc0a6590 con 0x7f6cdc0a6810 2024-09-11T01:39:41.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce306e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:41.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce306e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f6cdc09c4c0 con 0x7f6cdc138210 2024-09-11T01:39:41.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce306e640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:41.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce306e640 10 monclient(hunting): authenticate will time out at 1388.171482s 2024-09-11T01:39:41.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce0e0a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6cdc138210 0x7f6cdc1385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:41.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6cdbfff640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f6cdc0a6810 legacy=0x7f6cdc0a5090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:39:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce0e0a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6cdc138210 0x7f6cdc1385f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43658/0 (socket says 172.21.15.12:43658) 2024-09-11T01:39:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce0e0a640 1 -- 172.21.15.12:0/3893928673 learned_addr learned my addr 172.21.15.12:0/3893928673 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:39:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce0e0a640 10 monclient(hunting): get_auth_request con 0x7f6cdc138210 auth_method 0 2024-09-11T01:39:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce0e0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce160b640 1 --1- 172.21.15.12:0/3893928673 >> v1:172.21.15.12:6789/0 conn(0x7f6cdc0a57a0 0x7f6cdc140120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:41936/0 (socket says 172.21.15.12:41936) 2024-09-11T01:39:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce0e0a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce0e0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce0e0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce0e0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce0e0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce0e0a640 10 monclient(hunting): handle_auth_done global_id 94326 payload 274 2024-09-11T01:39:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce0e0a640 1 -- 172.21.15.12:0/3893928673 >> v1:172.21.15.71:6789/0 conn(0x7f6cdc0a6810 legacy=0x7f6cdc0a5090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:39:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce0e0a640 1 -- 172.21.15.12:0/3893928673 >> v1:172.21.15.12:6789/0 conn(0x7f6cdc0a57a0 legacy=0x7f6cdc140120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:39:41.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce0e0a640 10 monclient: _finish_hunting 0 2024-09-11T01:39:41.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce0e0a640 1 monclient: found mon.noname-a 2024-09-11T01:39:41.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce0e0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:41.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce0e0a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:39:41.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.729+0000 7f6ce0e0a640 1 -- 172.21.15.12:0/3893928673 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cdc09c1e0 con 0x7f6cdc138210 2024-09-11T01:39:41.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.732+0000 7f6ce0e0a640 1 -- 172.21.15.12:0/3893928673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cdc138210 msgr2=0x7f6cdc1385f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:41.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.732+0000 7f6ce0e0a640 1 --2- 172.21.15.12:0/3893928673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cdc138210 0x7f6cdc1385f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc009870 tx=0x7f6ccc02a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59ea8e140482b09c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:41.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.733+0000 7f6cdb7fe640 1 -- 172.21.15.12:0/3893928673 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc037070 con 0x7f6cdc138210 2024-09-11T01:39:41.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.733+0000 7f6cdb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:41.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.733+0000 7f6cdb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:39:41.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.733+0000 7f6cdb7fe640 10 monclient: dump: 2024-09-11T01:39:41.742 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:41.742 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:41.742 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:41.742 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:41.742 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:41.742 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:41.742 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:41.742 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:41.742 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:41.743 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:41.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.733+0000 7f6cdb7fe640 10 monclient: _finish_auth 0 2024-09-11T01:39:41.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.733+0000 7f6cdb7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:39:41.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.733+0000 7f6cdb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:41.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.733+0000 7f6cdb7fe640 1 -- 172.21.15.12:0/3893928673 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ccc032640 con 0x7f6cdc138210 2024-09-11T01:39:41.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.733+0000 7f6ce306e640 5 monclient: authenticate success, global_id 94326 2024-09-11T01:39:41.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.733+0000 7f6ce306e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:39:41.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.733+0000 7f6cdb7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:41.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.733+0000 7f6ce306e640 10 monclient: get_monmap_and_config success 2024-09-11T01:39:41.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.733+0000 7f6cdb7fe640 1 -- 172.21.15.12:0/3893928673 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc032a10 con 0x7f6cdc138210 2024-09-11T01:39:41.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.733+0000 7f6cdb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:41.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.733+0000 7f6ce306e640 4 set_mon_vals no callback set 2024-09-11T01:39:41.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.733+0000 7f6cdb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:39:41.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.733+0000 7f6cdb7fe640 10 monclient: dump: 2024-09-11T01:39:41.744 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:41.744 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:41.744 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:41.744 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:41.744 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:41.744 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:41.744 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:41.744 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:41.744 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:41.744 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:41.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.733+0000 7f6ce306e640 10 monclient: shutdown 2024-09-11T01:39:41.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.733+0000 7f6ce306e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:41.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.733+0000 7f6ce306e640 1 -- 172.21.15.12:0/3893928673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cdc138210 msgr2=0x7f6cdc1385f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:41.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.733+0000 7f6ce306e640 1 --2- 172.21.15.12:0/3893928673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cdc138210 0x7f6cdc1385f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc009870 tx=0x7f6ccc02a0a0 comp rx=0 tx=0).stop 2024-09-11T01:39:41.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.733+0000 7f6ce306e640 1 -- 172.21.15.12:0/3893928673 shutdown_connections 2024-09-11T01:39:41.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.733+0000 7f6ce306e640 1 --2- 172.21.15.12:0/3893928673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cdc138210 0x7f6cdc1385f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:41.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.733+0000 7f6ce306e640 1 -- 172.21.15.12:0/3893928673 >> 172.21.15.12:0/3893928673 conn(0x7f6cdc0aaa80 msgr2=0x7f6cdc0aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:41.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.733+0000 7f6ce306e640 1 -- 172.21.15.12:0/3893928673 shutdown_connections 2024-09-11T01:39:41.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.733+0000 7f6ce306e640 1 -- 172.21.15.12:0/3893928673 wait complete. 2024-09-11T01:39:41.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.734+0000 7f6ce306e640 10 monclient: build_initial_monmap 2024-09-11T01:39:41.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.734+0000 7f6ce306e640 10 monclient: monmap: 2024-09-11T01:39:41.745 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:41.745 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:39:41.745 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:39:41.745 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:39:41.745 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:41.746 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:41.746 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:39:41.746 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:39:41.746 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:39:41.746 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:41.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.734+0000 7f6ce306e640 1 Processor -- start 2024-09-11T01:39:41.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.734+0000 7f6ce306e640 1 -- start start 2024-09-11T01:39:41.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.734+0000 7f6ce306e640 10 monclient: init 2024-09-11T01:39:41.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce306e640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:41.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce306e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:39:41.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce306e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cdc0a57a0 0x7f6cdc136f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:41.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce306e640 10 monclient(hunting): picked mon.noname-b con 0x7f6cdc0a57a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:39:41.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce306e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6cdc0a6810 0x7f6cdc120ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:41.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce306e640 10 monclient(hunting): picked mon.noname-c con 0x7f6cdc0a6810 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:41.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce306e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cdc138210 0x7f6cdc121220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:41.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce306e640 10 monclient(hunting): picked mon.noname-a con 0x7f6cdc138210 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:39:41.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce306e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:41.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce306e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc138b30 con 0x7f6cdc138210 2024-09-11T01:39:41.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce306e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:41.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce306e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc09d6e0 con 0x7f6cdc0a57a0 2024-09-11T01:39:41.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce306e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:41.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce306e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6cdc138cb0 con 0x7f6cdc0a6810 2024-09-11T01:39:41.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce306e640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:41.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce306e640 10 monclient(hunting): authenticate will time out at 1388.177519s 2024-09-11T01:39:41.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce0e0a640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cdc0a57a0 msgr2=0x7f6cdc136f70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:39:41.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce160b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cdc138210 0x7f6cdc121220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:41.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce0e0a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cdc0a57a0 0x7f6cdc136f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:39:41.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6cdbfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6cdc0a6810 0x7f6cdc120ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:41.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce160b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cdc138210 0x7f6cdc121220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43660/0 (socket says 172.21.15.12:43660) 2024-09-11T01:39:41.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce160b640 1 -- 172.21.15.12:0/3690788784 learned_addr learned my addr 172.21.15.12:0/3690788784 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:39:41.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce160b640 10 monclient(hunting): get_auth_request con 0x7f6cdc138210 auth_method 0 2024-09-11T01:39:41.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce160b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:41.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce160b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:41.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce160b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:41.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6cdbfff640 10 monclient(hunting): get_auth_request con 0x7f6cdc0a6810 auth_method 0 2024-09-11T01:39:41.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6cdbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:41.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6cdbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:41.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6cdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:41.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce160b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:41.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce160b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:41.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce160b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:41.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6cdbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:41.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6cdbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:41.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6cdbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:41.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce160b640 10 monclient(hunting): handle_auth_done global_id 94329 payload 824 2024-09-11T01:39:41.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce160b640 1 -- 172.21.15.12:0/3690788784 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6cdc0a6810 msgr2=0x7f6cdc120ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:41.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce160b640 1 --2- 172.21.15.12:0/3690788784 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6cdc0a6810 0x7f6cdc120ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:41.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce160b640 1 -- 172.21.15.12:0/3690788784 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cdc0a57a0 msgr2=0x7f6cdc136f70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:39:41.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce160b640 1 --2- 172.21.15.12:0/3690788784 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cdc0a57a0 0x7f6cdc136f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:41.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce160b640 10 monclient: _finish_hunting 0 2024-09-11T01:39:41.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce160b640 1 monclient: found mon.noname-a 2024-09-11T01:39:41.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce160b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:41.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce160b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:39:41.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.735+0000 7f6ce160b640 1 -- 172.21.15.12:0/3690788784 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cdc124670 con 0x7f6cdc138210 2024-09-11T01:39:41.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.738+0000 7f6ce160b640 1 -- 172.21.15.12:0/3690788784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cdc138210 msgr2=0x7f6cdc121220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:41.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.738+0000 7f6ce160b640 1 --2- 172.21.15.12:0/3690788784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cdc138210 0x7f6cdc121220 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6cd400c750 tx=0x7f6cd400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:41.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.738+0000 7f6cd97fa640 1 -- 172.21.15.12:0/3690788784 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd4017070 con 0x7f6cdc138210 2024-09-11T01:39:41.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.738+0000 7f6cd97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:41.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.738+0000 7f6cd97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:39:41.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.738+0000 7f6cd97fa640 10 monclient: dump: 2024-09-11T01:39:41.750 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:41.750 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:41.750 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:41.750 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:41.751 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:41.751 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:41.751 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:41.751 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:41.751 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:41.751 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:41.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.738+0000 7f6cd97fa640 10 monclient: _finish_auth 0 2024-09-11T01:39:41.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.738+0000 7f6cd97fa640 10 monclient: _check_auth_tickets 2024-09-11T01:39:41.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.738+0000 7f6cd97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:41.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.738+0000 7f6cd97fa640 1 -- 172.21.15.12:0/3690788784 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cd400a860 con 0x7f6cdc138210 2024-09-11T01:39:41.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.738+0000 7f6cd97fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:41.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.738+0000 7f6cd97fa640 1 -- 172.21.15.12:0/3690788784 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd400ce60 con 0x7f6cdc138210 2024-09-11T01:39:41.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.738+0000 7f6cd97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:41.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.738+0000 7f6cd97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:39:41.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.738+0000 7f6cd97fa640 10 monclient: dump: 2024-09-11T01:39:41.752 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:41.752 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:41.752 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:41.752 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:41.752 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:41.752 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:41.752 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:41.752 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:41.752 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:41.752 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:41.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.738+0000 7f6ce306e640 5 monclient: authenticate success, global_id 94329 2024-09-11T01:39:41.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.738+0000 7f6ce306e640 10 monclient: _renew_subs 2024-09-11T01:39:41.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.738+0000 7f6ce306e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:41.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.738+0000 7f6ce306e640 1 -- 172.21.15.12:0/3690788784 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cdc0a1880 con 0x7f6cdc138210 2024-09-11T01:39:41.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.738+0000 7f6cdaffd640 4 set_mon_vals no callback set 2024-09-11T01:39:41.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.738+0000 7f6ce306e640 10 monclient: _renew_subs 2024-09-11T01:39:41.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.738+0000 7f6ce306e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:41.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.738+0000 7f6ce306e640 1 -- 172.21.15.12:0/3690788784 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cdc1240b0 con 0x7f6cdc138210 2024-09-11T01:39:41.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.739+0000 7f6ce306e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:41.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.739+0000 7f6ce306e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:41.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.739+0000 7f6ce306e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:41.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.739+0000 7f6cd97fa640 1 -- 172.21.15.12:0/3690788784 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f6cd400ac50 con 0x7f6cdc138210 2024-09-11T01:39:41.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.739+0000 7f6ce306e640 1 -- 172.21.15.12:0/3690788784 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cdc0a5d20 con 0x7f6cdc138210 2024-09-11T01:39:41.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.739+0000 7f6cd97fa640 1 --2- 172.21.15.12:0/3690788784 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6cc803e1c0 0x7f6cc8040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:41.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.739+0000 7f6cd97fa640 1 -- 172.21.15.12:0/3690788784 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(347..347 src has 186..347) ==== 4783+0+0 (secure 0 0 0) 0x7f6cd40527a0 con 0x7f6cdc138210 2024-09-11T01:39:41.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.740+0000 7f6ce0e0a640 1 --2- 172.21.15.12:0/3690788784 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6cc803e1c0 0x7f6cc8040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:41.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.740+0000 7f6ce0e0a640 10 monclient: get_auth_request con 0x7f6cc803e1c0 auth_method 0 2024-09-11T01:39:41.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.742+0000 7f6ce0e0a640 1 --2- 172.21.15.12:0/3690788784 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6cc803e1c0 0x7f6cc8040680 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc09cf00 tx=0x7f6ccc02e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:41.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.744+0000 7f6cd97fa640 1 -- 172.21.15.12:0/3690788784 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6cd40511a0 con 0x7f6cdc138210 2024-09-11T01:39:41.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.744+0000 7f6cd97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:41.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.744+0000 7f6cd97fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:39:42.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.998+0000 7f6ce306e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:39:42.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.999+0000 7f6ce306e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:39:42.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.999+0000 7f6ce306e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:42.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.999+0000 7f6ce306e640 1 -- 172.21.15.12:0/3690788784 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6cdc074b50 con 0x7f6cdc138210 2024-09-11T01:39:42.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.999+0000 7f6cd97fa640 1 -- 172.21.15.12:0/3690788784 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f6cd401b3b0 con 0x7f6cdc138210 2024-09-11T01:39:42.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.999+0000 7f6cd97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:39:42.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:41.999+0000 7f6cd97fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:39:42.001 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":100,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:39:42.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:42.001+0000 7f6ce306e640 1 -- 172.21.15.12:0/3690788784 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6cc803e1c0 msgr2=0x7f6cc8040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:42.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:42.001+0000 7f6ce306e640 1 --2- 172.21.15.12:0/3690788784 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6cc803e1c0 0x7f6cc8040680 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc09cf00 tx=0x7f6ccc02e000 comp rx=0 tx=0).stop 2024-09-11T01:39:42.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:42.001+0000 7f6ce306e640 10 monclient: shutdown 2024-09-11T01:39:42.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:42.001+0000 7f6ce306e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:42.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:42.001+0000 7f6ce306e640 1 -- 172.21.15.12:0/3690788784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cdc138210 msgr2=0x7f6cdc121220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:42.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:42.001+0000 7f6ce306e640 1 --2- 172.21.15.12:0/3690788784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cdc138210 0x7f6cdc121220 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6cd400c750 tx=0x7f6cd400cc10 comp rx=0 tx=0).stop 2024-09-11T01:39:42.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:42.001+0000 7f6ce306e640 1 -- 172.21.15.12:0/3690788784 shutdown_connections 2024-09-11T01:39:42.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:42.001+0000 7f6ce306e640 1 --2- 172.21.15.12:0/3690788784 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6cc803e1c0 0x7f6cc8040680 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:42.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:42.001+0000 7f6ce306e640 1 --2- 172.21.15.12:0/3690788784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cdc138210 0x7f6cdc121220 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:42.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:42.001+0000 7f6ce306e640 1 --2- 172.21.15.12:0/3690788784 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6cdc0a6810 0x7f6cdc120ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:42.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:42.001+0000 7f6ce306e640 1 --2- 172.21.15.12:0/3690788784 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cdc0a57a0 0x7f6cdc136f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:42.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:42.001+0000 7f6ce306e640 1 -- 172.21.15.12:0/3690788784 >> 172.21.15.12:0/3690788784 conn(0x7f6cdc0aaa80 msgr2=0x7f6cdc09ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:42.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:42.001+0000 7f6ce306e640 1 -- 172.21.15.12:0/3690788784 shutdown_connections 2024-09-11T01:39:42.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:42.001+0000 7f6ce306e640 1 -- 172.21.15.12:0/3690788784 wait complete. 2024-09-11T01:39:45.014 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:39:45.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.124+0000 7fa46e9bb640 10 monclient: get_monmap_and_config 2024-09-11T01:39:45.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.124+0000 7fa46e9bb640 10 monclient: build_initial_monmap 2024-09-11T01:39:45.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.124+0000 7fa46e9bb640 10 monclient: monmap: 2024-09-11T01:39:45.127 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:45.127 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:45.127 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:39:45.126508+0000 2024-09-11T01:39:45.127 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:39:45.126508+0000 2024-09-11T01:39:45.127 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:45.128 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:45.128 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:39:45.128 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:39:45.128 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:39:45.128 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:39:45.128 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:39:45.128 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:45.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.125+0000 7fa46e9bb640 1 Processor -- start 2024-09-11T01:39:45.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.125+0000 7fa46e9bb640 1 -- start start 2024-09-11T01:39:45.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.125+0000 7fa46e9bb640 10 monclient: init 2024-09-11T01:39:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46e9bb640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46e9bb640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-11T01:39:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46e9bb640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4680a2db0 0x7fa46806cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46e9bb640 10 monclient(hunting): picked mon.noname-b con 0x7fa4680a2db0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46e9bb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa46809fbd0 addr v1:172.21.15.12:6789/0 2024-09-11T01:39:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46e9bb640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa46813d550 0x7fa46813f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46e9bb640 10 monclient(hunting): picked mon.noname-a con 0x7fa46813d550 addr v2:172.21.15.12:3300/0 2024-09-11T01:39:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46e9bb640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa46809eea0 con 0x7fa46809fbd0 2024-09-11T01:39:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46e9bb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46e9bb640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fa4680a3850 con 0x7fa46813d550 2024-09-11T01:39:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46e9bb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46e9bb640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa4680a3190 con 0x7fa4680a2db0 2024-09-11T01:39:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46e9bb640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46e9bb640 10 monclient(hunting): authenticate will time out at 1391.569376s 2024-09-11T01:39:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa467fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4680a2db0 0x7fa46806cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46cf58640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa46813d550 0x7fa46813f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa467fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4680a2db0 0x7fa46806cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:47818/0 (socket says 172.21.15.12:47818) 2024-09-11T01:39:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa467fff640 1 -- 172.21.15.12:0/1767681734 learned_addr learned my addr 172.21.15.12:0/1767681734 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:39:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa467fff640 10 monclient(hunting): get_auth_request con 0x7fa4680a2db0 auth_method 0 2024-09-11T01:39:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa467fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa467fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa467fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46cf58640 10 monclient(hunting): get_auth_request con 0x7fa46813d550 auth_method 0 2024-09-11T01:39:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46cf58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46cf58640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46cf58640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa467fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa467fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa467fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46cf58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46cf58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46cf58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46cf58640 10 monclient(hunting): handle_auth_done global_id 94332 payload 274 2024-09-11T01:39:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46cf58640 1 -- 172.21.15.12:0/1767681734 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4680a2db0 msgr2=0x7fa46806cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:39:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46cf58640 1 --2- 172.21.15.12:0/1767681734 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4680a2db0 0x7fa46806cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46cf58640 1 -- 172.21.15.12:0/1767681734 >> v1:172.21.15.12:6789/0 conn(0x7fa46809fbd0 legacy=0x7fa46806d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:39:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46cf58640 10 monclient: _finish_hunting 0 2024-09-11T01:39:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46cf58640 1 monclient: found mon.noname-a 2024-09-11T01:39:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46cf58640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46cf58640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:39:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa46cf58640 1 -- 172.21.15.12:0/1767681734 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa46809f180 con 0x7fa46813d550 2024-09-11T01:39:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.126+0000 7fa467fff640 1 --2- 172.21.15.12:0/1767681734 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4680a2db0 0x7fa46806cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:39:45.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.129+0000 7fa46cf58640 1 -- 172.21.15.12:0/1767681734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa46813d550 msgr2=0x7fa46813f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:45.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.129+0000 7fa46cf58640 1 --2- 172.21.15.12:0/1767681734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa46813d550 0x7fa46813f940 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa460009dc0 tx=0x7fa46002a4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88f3d729dd99082a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.129+0000 7fa466ffd640 1 -- 172.21.15.12:0/1767681734 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa460007950 con 0x7fa46813d550 2024-09-11T01:39:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.129+0000 7fa466ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.129+0000 7fa466ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:39:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.129+0000 7fa466ffd640 10 monclient: dump: 2024-09-11T01:39:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:45.141 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.129+0000 7fa466ffd640 10 monclient: _finish_auth 0 2024-09-11T01:39:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.129+0000 7fa466ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:39:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.129+0000 7fa466ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.129+0000 7fa466ffd640 1 -- 172.21.15.12:0/1767681734 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa460007af0 con 0x7fa46813d550 2024-09-11T01:39:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.129+0000 7fa46e9bb640 5 monclient: authenticate success, global_id 94332 2024-09-11T01:39:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.129+0000 7fa46e9bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:39:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.129+0000 7fa466ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.129+0000 7fa46e9bb640 10 monclient: get_monmap_and_config success 2024-09-11T01:39:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.129+0000 7fa466ffd640 1 -- 172.21.15.12:0/1767681734 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa460031400 con 0x7fa46813d550 2024-09-11T01:39:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.129+0000 7fa46e9bb640 4 set_mon_vals no callback set 2024-09-11T01:39:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.129+0000 7fa466ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.129+0000 7fa466ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:39:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.129+0000 7fa466ffd640 10 monclient: dump: 2024-09-11T01:39:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:45.142 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.130+0000 7fa46e9bb640 10 monclient: shutdown 2024-09-11T01:39:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.130+0000 7fa46e9bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.130+0000 7fa46e9bb640 1 -- 172.21.15.12:0/1767681734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa46813d550 msgr2=0x7fa46813f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.130+0000 7fa46e9bb640 1 --2- 172.21.15.12:0/1767681734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa46813d550 0x7fa46813f940 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa460009dc0 tx=0x7fa46002a4d0 comp rx=0 tx=0).stop 2024-09-11T01:39:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.130+0000 7fa46e9bb640 1 -- 172.21.15.12:0/1767681734 shutdown_connections 2024-09-11T01:39:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.130+0000 7fa46e9bb640 1 --2- 172.21.15.12:0/1767681734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa46813d550 0x7fa46813f940 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.130+0000 7fa46e9bb640 1 --2- 172.21.15.12:0/1767681734 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4680a2db0 0x7fa46806cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.130+0000 7fa46e9bb640 1 -- 172.21.15.12:0/1767681734 >> 172.21.15.12:0/1767681734 conn(0x7fa4680a3ea0 msgr2=0x7fa4680a42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.130+0000 7fa46e9bb640 1 -- 172.21.15.12:0/1767681734 shutdown_connections 2024-09-11T01:39:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.130+0000 7fa46e9bb640 1 -- 172.21.15.12:0/1767681734 wait complete. 2024-09-11T01:39:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.131+0000 7fa46e9bb640 10 monclient: build_initial_monmap 2024-09-11T01:39:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.131+0000 7fa46e9bb640 10 monclient: monmap: 2024-09-11T01:39:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:39:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:39:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:39:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:39:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:39:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:39:45.144 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.131+0000 7fa46e9bb640 1 Processor -- start 2024-09-11T01:39:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.131+0000 7fa46e9bb640 1 -- start start 2024-09-11T01:39:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.131+0000 7fa46e9bb640 10 monclient: init 2024-09-11T01:39:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46e9bb640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46e9bb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:39:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46e9bb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa46809fbd0 0x7fa46813cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46e9bb640 10 monclient(hunting): picked mon.noname-a con 0x7fa46809fbd0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:39:45.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46e9bb640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4680a2db0 0x7fa468099da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:45.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46e9bb640 10 monclient(hunting): picked mon.noname-b con 0x7fa4680a2db0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:39:45.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46e9bb640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa46813d550 0x7fa4680913f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:45.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46e9bb640 10 monclient(hunting): picked mon.noname-c con 0x7fa46813d550 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:45.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46e9bb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:45.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46e9bb640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa46809ea20 con 0x7fa46809fbd0 2024-09-11T01:39:45.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46e9bb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:45.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46e9bb640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa46809e8a0 con 0x7fa4680a2db0 2024-09-11T01:39:45.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46e9bb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:45.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46e9bb640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa46806d7a0 con 0x7fa46813d550 2024-09-11T01:39:45.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46e9bb640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:45.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46e9bb640 10 monclient(hunting): authenticate will time out at 1391.575357s 2024-09-11T01:39:45.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa467fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa46809fbd0 0x7fa46813cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:45.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa4677fe640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4680a2db0 msgr2=0x7fa468099da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:39:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa467fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa46809fbd0 0x7fa46813cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43686/0 (socket says 172.21.15.12:43686) 2024-09-11T01:39:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa467fff640 1 -- 172.21.15.12:0/2913914389 learned_addr learned my addr 172.21.15.12:0/2913914389 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:39:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa467fff640 10 monclient(hunting): get_auth_request con 0x7fa46809fbd0 auth_method 0 2024-09-11T01:39:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa467fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa467fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa467fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46cf58640 1 --2- 172.21.15.12:0/2913914389 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa46813d550 0x7fa4680913f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa4677fe640 1 --2- 172.21.15.12:0/2913914389 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4680a2db0 0x7fa468099da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:39:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46cf58640 10 monclient(hunting): get_auth_request con 0x7fa46813d550 auth_method 0 2024-09-11T01:39:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46cf58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46cf58640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46cf58640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa467fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa467fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:45.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa467fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:45.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46cf58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:45.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46cf58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:45.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46cf58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:45.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46cf58640 10 monclient(hunting): handle_auth_done global_id 114167 payload 824 2024-09-11T01:39:45.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46cf58640 1 -- 172.21.15.12:0/2913914389 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4680a2db0 msgr2=0x7fa468099da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:39:45.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46cf58640 1 --2- 172.21.15.12:0/2913914389 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4680a2db0 0x7fa468099da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:45.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46cf58640 1 -- 172.21.15.12:0/2913914389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa46809fbd0 msgr2=0x7fa46813cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:45.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46cf58640 1 --2- 172.21.15.12:0/2913914389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa46809fbd0 0x7fa46813cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:45.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46cf58640 10 monclient: _finish_hunting 0 2024-09-11T01:39:45.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46cf58640 1 monclient: found mon.noname-c 2024-09-11T01:39:45.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46cf58640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:45.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46cf58640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:39:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa46cf58640 1 -- 172.21.15.12:0/2913914389 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa468094ee0 con 0x7fa46813d550 2024-09-11T01:39:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.132+0000 7fa467fff640 1 --2- 172.21.15.12:0/2913914389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa46809fbd0 0x7fa46813cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:39:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.135+0000 7fa46cf58640 1 -- 172.21.15.12:0/2913914389 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa46813d550 msgr2=0x7fa4680913f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.135+0000 7fa46cf58640 1 --2- 172.21.15.12:0/2913914389 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa46813d550 0x7fa4680913f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa46806f440 tx=0x7fa460004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.135+0000 7fa464ff9640 1 -- 172.21.15.12:0/2913914389 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa460040070 con 0x7fa46813d550 2024-09-11T01:39:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.135+0000 7fa464ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.135+0000 7fa464ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:39:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.135+0000 7fa464ff9640 10 monclient: dump: 2024-09-11T01:39:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:45.149 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:45.149 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:45.149 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:45.149 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:45.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.135+0000 7fa464ff9640 10 monclient: _finish_auth 0 2024-09-11T01:39:45.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.135+0000 7fa464ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:39:45.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.135+0000 7fa464ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:45.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.135+0000 7fa464ff9640 1 -- 172.21.15.12:0/2913914389 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa460045460 con 0x7fa46813d550 2024-09-11T01:39:45.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.135+0000 7fa464ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:45.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.135+0000 7fa464ff9640 1 -- 172.21.15.12:0/2913914389 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa460007950 con 0x7fa46813d550 2024-09-11T01:39:45.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.135+0000 7fa46e9bb640 5 monclient: authenticate success, global_id 114167 2024-09-11T01:39:45.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.135+0000 7fa464ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:45.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.135+0000 7fa464ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:39:45.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.135+0000 7fa464ff9640 10 monclient: dump: 2024-09-11T01:39:45.150 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:45.150 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:45.150 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:45.150 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:45.150 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:45.150 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:45.150 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:45.150 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:45.150 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:45.150 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:45.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.135+0000 7fa46e9bb640 10 monclient: _renew_subs 2024-09-11T01:39:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.135+0000 7fa46e9bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:39:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.135+0000 7fa4667fc640 4 set_mon_vals no callback set 2024-09-11T01:39:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.135+0000 7fa46e9bb640 1 -- 172.21.15.12:0/2913914389 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4680939c0 con 0x7fa46813d550 2024-09-11T01:39:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.135+0000 7fa46e9bb640 10 monclient: _renew_subs 2024-09-11T01:39:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.135+0000 7fa46e9bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:39:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.135+0000 7fa46e9bb640 1 -- 172.21.15.12:0/2913914389 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa468092e60 con 0x7fa46813d550 2024-09-11T01:39:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.136+0000 7fa464ff9640 1 -- 172.21.15.12:0/2913914389 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fa46003c420 con 0x7fa46813d550 2024-09-11T01:39:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.136+0000 7fa46e9bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.136+0000 7fa46e9bb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.136+0000 7fa46e9bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:39:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.136+0000 7fa46e9bb640 1 -- 172.21.15.12:0/2913914389 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa46806cb50 con 0x7fa46813d550 2024-09-11T01:39:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.136+0000 7fa464ff9640 1 --2- 172.21.15.12:0/2913914389 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa45403e1c0 0x7fa454040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.136+0000 7fa464ff9640 1 -- 172.21.15.12:0/2913914389 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(347..347 src has 186..347) ==== 4783+0+0 (secure 0 0 0) 0x7fa4600748d0 con 0x7fa46813d550 2024-09-11T01:39:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.136+0000 7fa467fff640 1 --2- 172.21.15.12:0/2913914389 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa45403e1c0 0x7fa454040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:45.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.136+0000 7fa467fff640 10 monclient: get_auth_request con 0x7fa45403e1c0 auth_method 0 2024-09-11T01:39:45.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.139+0000 7fa467fff640 1 --2- 172.21.15.12:0/2913914389 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa45403e1c0 0x7fa454040680 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa4580113d0 tx=0x7fa458009240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:45.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.141+0000 7fa464ff9640 1 -- 172.21.15.12:0/2913914389 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa4600732d0 con 0x7fa46813d550 2024-09-11T01:39:45.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.141+0000 7fa464ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:45.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.141+0000 7fa464ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:39:45.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.399+0000 7fa46e9bb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:39:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.399+0000 7fa46e9bb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:39:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.399+0000 7fa46e9bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:39:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.399+0000 7fa46e9bb640 1 -- 172.21.15.12:0/2913914389 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa46806e570 con 0x7fa46813d550 2024-09-11T01:39:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.399+0000 7fa464ff9640 1 -- 172.21.15.12:0/2913914389 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7fa46003cac0 con 0x7fa46813d550 2024-09-11T01:39:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.399+0000 7fa464ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:39:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.399+0000 7fa464ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:39:45.402 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":100,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:39:45.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.401+0000 7fa46e9bb640 1 -- 172.21.15.12:0/2913914389 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa45403e1c0 msgr2=0x7fa454040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:45.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.401+0000 7fa46e9bb640 1 --2- 172.21.15.12:0/2913914389 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa45403e1c0 0x7fa454040680 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa4580113d0 tx=0x7fa458009240 comp rx=0 tx=0).stop 2024-09-11T01:39:45.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.401+0000 7fa46e9bb640 10 monclient: shutdown 2024-09-11T01:39:45.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.401+0000 7fa46e9bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:45.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.401+0000 7fa46e9bb640 1 -- 172.21.15.12:0/2913914389 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa46813d550 msgr2=0x7fa4680913f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:45.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.401+0000 7fa46e9bb640 1 --2- 172.21.15.12:0/2913914389 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa46813d550 0x7fa4680913f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa46806f440 tx=0x7fa460004a40 comp rx=0 tx=0).stop 2024-09-11T01:39:45.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.401+0000 7fa46e9bb640 1 -- 172.21.15.12:0/2913914389 shutdown_connections 2024-09-11T01:39:45.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.401+0000 7fa46e9bb640 1 --2- 172.21.15.12:0/2913914389 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa46813d550 0x7fa4680913f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:45.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.401+0000 7fa46e9bb640 1 --2- 172.21.15.12:0/2913914389 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4680a2db0 0x7fa468099da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:45.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.401+0000 7fa46e9bb640 1 --2- 172.21.15.12:0/2913914389 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa45403e1c0 0x7fa454040680 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:45.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.401+0000 7fa46e9bb640 1 --2- 172.21.15.12:0/2913914389 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa46809fbd0 0x7fa46813cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:45.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.401+0000 7fa46e9bb640 1 -- 172.21.15.12:0/2913914389 >> 172.21.15.12:0/2913914389 conn(0x7fa4680a3ea0 msgr2=0x7fa46813e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:45.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.402+0000 7fa46e9bb640 1 -- 172.21.15.12:0/2913914389 shutdown_connections 2024-09-11T01:39:45.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:45.402+0000 7fa46e9bb640 1 -- 172.21.15.12:0/2913914389 wait complete. 2024-09-11T01:39:48.416 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:39:48.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.526+0000 7f019890e640 10 monclient: get_monmap_and_config 2024-09-11T01:39:48.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.526+0000 7f019890e640 10 monclient: build_initial_monmap 2024-09-11T01:39:48.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.526+0000 7f019890e640 10 monclient: monmap: 2024-09-11T01:39:48.529 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:48.529 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:48.529 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:39:48.528240+0000 2024-09-11T01:39:48.529 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:39:48.528240+0000 2024-09-11T01:39:48.529 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:48.529 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:48.530 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:39:48.530 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:39:48.530 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:39:48.530 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:39:48.530 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:39:48.530 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:48.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.527+0000 7f019890e640 1 Processor -- start 2024-09-11T01:39:48.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.527+0000 7f019890e640 1 -- start start 2024-09-11T01:39:48.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.527+0000 7f019890e640 10 monclient: init 2024-09-11T01:39:48.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.527+0000 7f019890e640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:48.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.527+0000 7f019890e640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-11T01:39:48.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.527+0000 7f019890e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f01940a4170 0x7f0194073fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:48.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.527+0000 7f019890e640 10 monclient(hunting): picked mon.noname-c con 0x7f01940a4170 addr v2:172.21.15.71:3300/0 2024-09-11T01:39:48.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.527+0000 7f019890e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0194074500 addr v1:172.21.15.12:6789/0 2024-09-11T01:39:48.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.527+0000 7f019890e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01940a2c70 0x7f019413ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:48.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.527+0000 7f019890e640 10 monclient(hunting): picked mon.noname-b con 0x7f01940a2c70 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:48.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.528+0000 7f019890e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01941380d0 con 0x7f0194074500 2024-09-11T01:39:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.528+0000 7f019890e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.528+0000 7f019890e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f01940a4550 con 0x7f01940a4170 2024-09-11T01:39:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.528+0000 7f019890e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.528+0000 7f019890e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f019409fff0 con 0x7f01940a2c70 2024-09-11T01:39:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.528+0000 7f019890e640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.528+0000 7f019890e640 10 monclient(hunting): authenticate will time out at 1394.971035s 2024-09-11T01:39:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.528+0000 7f019259c640 1 -- >> v2:172.21.15.71:3300/0 conn(0x7f01940a4170 msgr2=0x7f0194073fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:39:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.528+0000 7f0191d9b640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0194074500 0x7f01940a2560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35120/0 (socket says 172.21.15.12:35120) 2024-09-11T01:39:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.528+0000 7f0192d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01940a2c70 0x7f019413ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.528+0000 7f0191d9b640 1 -- 172.21.15.12:0/2840641965 learned_addr learned my addr 172.21.15.12:0/2840641965 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:39:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.528+0000 7f019259c640 1 --2- 172.21.15.12:0/2840641965 >> v2:172.21.15.71:3300/0 conn(0x7f01940a4170 0x7f0194073fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:39:48.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.528+0000 7f0192d9d640 10 monclient(hunting): get_auth_request con 0x7f01940a2c70 auth_method 0 2024-09-11T01:39:48.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.528+0000 7f0192d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:48.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.528+0000 7f0192d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:48.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.528+0000 7f0192d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:48.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.528+0000 7f0192d9d640 1 -- 172.21.15.12:0/2840641965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01940a2c70 msgr2=0x7f019413ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:39:48.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.528+0000 7f0192d9d640 1 -- 172.21.15.12:0/2840641965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01940a2c70 msgr2=0x7f019413ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:39:48.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.528+0000 7f0192d9d640 1 --2- 172.21.15.12:0/2840641965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01940a2c70 0x7f019413ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:39:48.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.528+0000 7f0192d9d640 1 --2- 172.21.15.12:0/2840641965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01940a2c70 0x7f019413ffb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:39:48.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.530+0000 7f0191d9b640 1 -- 172.21.15.12:0/2840641965 >> v1:172.21.15.12:6789/0 conn(0x7f0194074500 legacy=0x7f01940a2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:48.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.729+0000 7f019259c640 1 -- 172.21.15.12:0/2840641965 >> v2:172.21.15.71:3300/0 conn(0x7f01940a4170 msgr2=0x7f0194073fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:39:48.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.729+0000 7f019259c640 1 --2- 172.21.15.12:0/2840641965 >> v2:172.21.15.71:3300/0 conn(0x7f01940a4170 0x7f0194073fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:39:48.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.729+0000 7f0192d9d640 1 --2- 172.21.15.12:0/2840641965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01940a2c70 0x7f019413ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:48.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.729+0000 7f0192d9d640 10 monclient(hunting): get_auth_request con 0x7f01940a2c70 auth_method 0 2024-09-11T01:39:48.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.729+0000 7f0192d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:48.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.729+0000 7f0192d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:48.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.729+0000 7f0192d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:39:48.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.729+0000 7f0192d9d640 1 -- 172.21.15.12:0/2840641965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01940a2c70 msgr2=0x7f019413ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:39:48.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.729+0000 7f0192d9d640 1 -- 172.21.15.12:0/2840641965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01940a2c70 msgr2=0x7f019413ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:39:48.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.729+0000 7f0192d9d640 1 --2- 172.21.15.12:0/2840641965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01940a2c70 0x7f019413ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:39:48.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:48.729+0000 7f0192d9d640 1 --2- 172.21.15.12:0/2840641965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01940a2c70 0x7f019413ffb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:39:49.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:49.130+0000 7f019259c640 1 -- 172.21.15.12:0/2840641965 >> v2:172.21.15.71:3300/0 conn(0x7f01940a4170 msgr2=0x7f0194073fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:39:49.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:49.130+0000 7f019259c640 1 --2- 172.21.15.12:0/2840641965 >> v2:172.21.15.71:3300/0 conn(0x7f01940a4170 0x7f0194073fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:39:49.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:49.130+0000 7f0192d9d640 1 --2- 172.21.15.12:0/2840641965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01940a2c70 0x7f019413ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:49.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:49.130+0000 7f0192d9d640 10 monclient(hunting): get_auth_request con 0x7f01940a2c70 auth_method 0 2024-09-11T01:39:49.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:49.130+0000 7f0192d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:49.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:49.130+0000 7f0192d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:49.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:49.130+0000 7f0192d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:39:49.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:49.130+0000 7f0192d9d640 1 -- 172.21.15.12:0/2840641965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01940a2c70 msgr2=0x7f019413ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:39:49.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:49.130+0000 7f0192d9d640 1 -- 172.21.15.12:0/2840641965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01940a2c70 msgr2=0x7f019413ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:39:49.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:49.130+0000 7f0192d9d640 1 --2- 172.21.15.12:0/2840641965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01940a2c70 0x7f019413ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:39:49.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:49.130+0000 7f0192d9d640 1 --2- 172.21.15.12:0/2840641965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01940a2c70 0x7f019413ffb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:39:49.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:49.932+0000 7f019259c640 1 -- 172.21.15.12:0/2840641965 >> v2:172.21.15.71:3300/0 conn(0x7f01940a4170 msgr2=0x7f0194073fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:39:49.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:49.932+0000 7f019259c640 1 --2- 172.21.15.12:0/2840641965 >> v2:172.21.15.71:3300/0 conn(0x7f01940a4170 0x7f0194073fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:39:49.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:49.932+0000 7f0192d9d640 1 --2- 172.21.15.12:0/2840641965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01940a2c70 0x7f019413ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:49.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:49.932+0000 7f0192d9d640 10 monclient(hunting): get_auth_request con 0x7f01940a2c70 auth_method 0 2024-09-11T01:39:49.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:49.932+0000 7f0192d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:49.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:49.932+0000 7f0192d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:49.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:49.932+0000 7f0192d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:39:49.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:49.932+0000 7f0192d9d640 1 -- 172.21.15.12:0/2840641965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01940a2c70 msgr2=0x7f019413ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:39:49.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:49.932+0000 7f0192d9d640 1 -- 172.21.15.12:0/2840641965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01940a2c70 msgr2=0x7f019413ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:39:49.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:49.932+0000 7f0192d9d640 1 --2- 172.21.15.12:0/2840641965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01940a2c70 0x7f019413ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:39:49.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:49.932+0000 7f0192d9d640 1 --2- 172.21.15.12:0/2840641965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01940a2c70 0x7f019413ffb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:39:51.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f0183fff640 10 monclient(hunting): tick 2024-09-11T01:39:51.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f0183fff640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:39:51.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f0183fff640 1 monclient(hunting): continuing hunt 2024-09-11T01:39:51.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f0183fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:39:51.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f0183fff640 1 -- 172.21.15.12:0/2840641965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01940a2c70 msgr2=0x7f019413ffb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:39:51.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f0183fff640 1 --2- 172.21.15.12:0/2840641965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01940a2c70 0x7f019413ffb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:51.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f0183fff640 1 -- 172.21.15.12:0/2840641965 >> v2:172.21.15.71:3300/0 conn(0x7f01940a4170 msgr2=0x7f0194073fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:39:51.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f0183fff640 1 --2- 172.21.15.12:0/2840641965 >> v2:172.21.15.71:3300/0 conn(0x7f01940a4170 0x7f0194073fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:51.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f0183fff640 1 -- 172.21.15.12:0/2840641965 >> v1:172.21.15.12:6789/0 conn(0x7f0194074500 legacy=0x7f01940a2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:51.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f0183fff640 10 monclient: _add_conns ranks=[3,1] 2024-09-11T01:39:51.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f0183fff640 1 --2- 172.21.15.12:0/2840641965 >> v2:172.21.15.12:3300/0 conn(0x7f0174003740 0x7f0174005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:51.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f0183fff640 10 monclient(hunting): picked mon.noname-a con 0x7f0174003740 addr v2:172.21.15.12:3300/0 2024-09-11T01:39:51.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f0183fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f01740062b0 addr v1:172.21.15.71:6789/0 2024-09-11T01:39:51.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f0183fff640 1 -- 172.21.15.12:0/2840641965 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0194138350 con 0x7f01740062b0 2024-09-11T01:39:51.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f0183fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:51.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f0183fff640 1 -- 172.21.15.12:0/2840641965 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f01940a4550 con 0x7f0174003740 2024-09-11T01:39:51.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f0183fff640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:51.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f019259c640 1 --2- 172.21.15.12:0/2840641965 >> v2:172.21.15.12:3300/0 conn(0x7f0174003740 0x7f0174005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:51.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f0191d9b640 1 -- 172.21.15.12:0/2840641965 >> v1:172.21.15.71:6789/0 conn(0x7f01740062b0 legacy=0x7f0174008770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:39:51.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f019259c640 10 monclient(hunting): get_auth_request con 0x7f0174003740 auth_method 0 2024-09-11T01:39:51.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f019259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:51.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f019259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:51.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f019259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:51.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f019259c640 1 -- 172.21.15.12:0/2840641965 >> v2:172.21.15.12:3300/0 conn(0x7f0174003740 msgr2=0x7f0174005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:39:51.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f019259c640 1 -- 172.21.15.12:0/2840641965 >> v2:172.21.15.12:3300/0 conn(0x7f0174003740 msgr2=0x7f0174005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:39:51.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f019259c640 1 --2- 172.21.15.12:0/2840641965 >> v2:172.21.15.12:3300/0 conn(0x7f0174003740 0x7f0174005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:39:51.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.528+0000 7f019259c640 1 --2- 172.21.15.12:0/2840641965 >> v2:172.21.15.12:3300/0 conn(0x7f0174003740 0x7f0174005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:39:51.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.729+0000 7f0191d9b640 1 -- 172.21.15.12:0/2840641965 >> v1:172.21.15.71:6789/0 conn(0x7f01740062b0 legacy=0x7f0174008770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:39:51.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.729+0000 7f019259c640 1 --2- 172.21.15.12:0/2840641965 >> v2:172.21.15.12:3300/0 conn(0x7f0174003740 0x7f0174005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:51.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.729+0000 7f019259c640 10 monclient(hunting): get_auth_request con 0x7f0174003740 auth_method 0 2024-09-11T01:39:51.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.729+0000 7f019259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:51.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.729+0000 7f019259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:51.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.729+0000 7f019259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:39:51.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.729+0000 7f019259c640 1 -- 172.21.15.12:0/2840641965 >> v2:172.21.15.12:3300/0 conn(0x7f0174003740 msgr2=0x7f0174005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:39:51.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.729+0000 7f019259c640 1 -- 172.21.15.12:0/2840641965 >> v2:172.21.15.12:3300/0 conn(0x7f0174003740 msgr2=0x7f0174005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:39:51.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.729+0000 7f019259c640 1 --2- 172.21.15.12:0/2840641965 >> v2:172.21.15.12:3300/0 conn(0x7f0174003740 0x7f0174005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:39:51.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:51.729+0000 7f019259c640 1 --2- 172.21.15.12:0/2840641965 >> v2:172.21.15.12:3300/0 conn(0x7f0174003740 0x7f0174005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:39:52.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:52.130+0000 7f0191d9b640 1 -- 172.21.15.12:0/2840641965 >> v1:172.21.15.71:6789/0 conn(0x7f01740062b0 legacy=0x7f0174008770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:39:52.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:52.130+0000 7f019259c640 1 --2- 172.21.15.12:0/2840641965 >> v2:172.21.15.12:3300/0 conn(0x7f0174003740 0x7f0174005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:52.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:52.130+0000 7f019259c640 10 monclient(hunting): get_auth_request con 0x7f0174003740 auth_method 0 2024-09-11T01:39:52.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:52.130+0000 7f019259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:52.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:52.130+0000 7f019259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:52.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:52.130+0000 7f019259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:39:52.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:52.130+0000 7f019259c640 1 -- 172.21.15.12:0/2840641965 >> v2:172.21.15.12:3300/0 conn(0x7f0174003740 msgr2=0x7f0174005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:39:52.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:52.130+0000 7f019259c640 1 -- 172.21.15.12:0/2840641965 >> v2:172.21.15.12:3300/0 conn(0x7f0174003740 msgr2=0x7f0174005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:39:52.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:52.130+0000 7f019259c640 1 --2- 172.21.15.12:0/2840641965 >> v2:172.21.15.12:3300/0 conn(0x7f0174003740 0x7f0174005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:39:52.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:52.130+0000 7f019259c640 1 --2- 172.21.15.12:0/2840641965 >> v2:172.21.15.12:3300/0 conn(0x7f0174003740 0x7f0174005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:39:52.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:52.931+0000 7f0191d9b640 1 -- 172.21.15.12:0/2840641965 >> v1:172.21.15.71:6789/0 conn(0x7f01740062b0 legacy=0x7f0174008770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:39:52.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:52.932+0000 7f019259c640 1 --2- 172.21.15.12:0/2840641965 >> v2:172.21.15.12:3300/0 conn(0x7f0174003740 0x7f0174005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:52.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:52.932+0000 7f019259c640 10 monclient(hunting): get_auth_request con 0x7f0174003740 auth_method 0 2024-09-11T01:39:52.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:52.932+0000 7f019259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:52.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:52.932+0000 7f019259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:52.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:52.932+0000 7f019259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:39:52.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:52.932+0000 7f019259c640 1 -- 172.21.15.12:0/2840641965 >> v2:172.21.15.12:3300/0 conn(0x7f0174003740 msgr2=0x7f0174005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:39:52.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:52.932+0000 7f019259c640 1 -- 172.21.15.12:0/2840641965 >> v2:172.21.15.12:3300/0 conn(0x7f0174003740 msgr2=0x7f0174005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:39:52.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:52.932+0000 7f019259c640 1 --2- 172.21.15.12:0/2840641965 >> v2:172.21.15.12:3300/0 conn(0x7f0174003740 0x7f0174005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:39:52.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:52.932+0000 7f019259c640 1 --2- 172.21.15.12:0/2840641965 >> v2:172.21.15.12:3300/0 conn(0x7f0174003740 0x7f0174005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:39:53.146 INFO:tasks.rados.rados.0.smithi071.stdout:1530: done (2 left) 2024-09-11T01:39:53.146 INFO:tasks.rados.rados.0.smithi071.stdout:1531: write (excl) oid 41 current snap is 164 2024-09-11T01:39:53.146 INFO:tasks.rados.rados.0.smithi071.stdout:1531: seq_num 576 ranges {480832=469899,1606334=664200,3018232=662600} 2024-09-11T01:39:53.151 INFO:tasks.rados.rados.0.smithi071.stdout:1531: writing smithi07134777-41 from 480832 to 950731 tid 1 2024-09-11T01:39:53.158 INFO:tasks.rados.rados.0.smithi071.stdout:1531: writing smithi07134777-41 from 1606334 to 2270534 tid 2 2024-09-11T01:39:53.167 INFO:tasks.rados.rados.0.smithi071.stdout:1531: writing smithi07134777-41 from 3018232 to 3680832 tid 3 2024-09-11T01:39:53.167 INFO:tasks.rados.rados.0.smithi071.stdout:1532: snap_create 2024-09-11T01:39:53.167 INFO:tasks.rados.rados.0.smithi071.stdout:1527: finishing write tid 1 to smithi07134777-39 2024-09-11T01:39:53.167 INFO:tasks.rados.rados.0.smithi071.stdout:1527: finishing write tid 2 to smithi07134777-39 2024-09-11T01:39:53.167 INFO:tasks.rados.rados.0.smithi071.stdout:1527: finishing write tid 3 to smithi07134777-39 2024-09-11T01:39:53.167 INFO:tasks.rados.rados.0.smithi071.stdout:1527: finishing write tid 4 to smithi07134777-39 2024-09-11T01:39:53.167 INFO:tasks.rados.rados.0.smithi071.stdout:1527: finishing write tid 5 to smithi07134777-39 2024-09-11T01:39:53.168 INFO:tasks.rados.rados.0.smithi071.stdout:1527: oid 39 updating version 0 to 596 2024-09-11T01:39:53.168 INFO:tasks.rados.rados.0.smithi071.stdout:1527: oid 39 updating version 596 to 598 2024-09-11T01:39:53.168 INFO:tasks.rados.rados.0.smithi071.stdout:1527: oid 39 version 598 is already newer than 597 2024-09-11T01:39:53.168 INFO:tasks.rados.rados.0.smithi071.stdout:1527: oid 39 updating version 598 to 599 2024-09-11T01:39:53.168 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 599 (ObjNum 574 snap 164 seq_num 574) dirty exists 2024-09-11T01:39:53.168 INFO:tasks.rados.rados.0.smithi071.stdout:1527: left oid 39 (ObjNum 574 snap 164 seq_num 574) 2024-09-11T01:39:53.168 INFO:tasks.rados.rados.0.smithi071.stdout:1529: finishing write tid 1 to smithi07134777-38 2024-09-11T01:39:53.168 INFO:tasks.rados.rados.0.smithi071.stdout:1529: finishing write tid 2 to smithi07134777-38 2024-09-11T01:39:53.168 INFO:tasks.rados.rados.0.smithi071.stdout:1529: finishing write tid 3 to smithi07134777-38 2024-09-11T01:39:53.168 INFO:tasks.rados.rados.0.smithi071.stdout:1529: finishing write tid 4 to smithi07134777-38 2024-09-11T01:39:53.168 INFO:tasks.rados.rados.0.smithi071.stdout:1529: finishing write tid 5 to smithi07134777-38 2024-09-11T01:39:53.168 INFO:tasks.rados.rados.0.smithi071.stdout:1529: oid 38 updating version 0 to 848 2024-09-11T01:39:53.168 INFO:tasks.rados.rados.0.smithi071.stdout:1529: oid 38 version 848 is already newer than 846 2024-09-11T01:39:53.168 INFO:tasks.rados.rados.0.smithi071.stdout:1529: oid 38 version 848 is already newer than 843 2024-09-11T01:39:53.168 INFO:tasks.rados.rados.0.smithi071.stdout:1529: oid 38 version 848 is already newer than 847 2024-09-11T01:39:53.169 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 848 (ObjNum 575 snap 164 seq_num 575) dirty exists 2024-09-11T01:39:53.169 INFO:tasks.rados.rados.0.smithi071.stdout:1529: left oid 38 (ObjNum 575 snap 164 seq_num 575) 2024-09-11T01:39:53.169 INFO:tasks.rados.rados.0.smithi071.stdout:1531: finishing write tid 1 to smithi07134777-41 2024-09-11T01:39:53.169 INFO:tasks.rados.rados.0.smithi071.stdout:1531: finishing write tid 2 to smithi07134777-41 2024-09-11T01:39:53.172 INFO:tasks.rados.rados.0.smithi071.stdout:1531: finishing write tid 3 to smithi07134777-41 2024-09-11T01:39:53.172 INFO:tasks.rados.rados.0.smithi071.stdout:1531: finishing write tid 4 to smithi07134777-41 2024-09-11T01:39:53.173 INFO:tasks.rados.rados.0.smithi071.stdout:1531: finishing write tid 5 to smithi07134777-41 2024-09-11T01:39:53.173 INFO:tasks.rados.rados.0.smithi071.stdout:1531: oid 41 updating version 0 to 371 2024-09-11T01:39:53.173 INFO:tasks.rados.rados.0.smithi071.stdout:1531: oid 41 updating version 371 to 372 2024-09-11T01:39:53.173 INFO:tasks.rados.rados.0.smithi071.stdout:1531: oid 41 version 372 is already newer than 369 2024-09-11T01:39:53.173 INFO:tasks.rados.rados.0.smithi071.stdout:1531: oid 41 version 372 is already newer than 370 2024-09-11T01:39:53.173 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 372 (ObjNum 576 snap 164 seq_num 576) dirty exists 2024-09-11T01:39:53.173 INFO:tasks.rados.rados.0.smithi071.stdout:1531: left oid 41 (ObjNum 576 snap 164 seq_num 576) 2024-09-11T01:39:54.163 INFO:tasks.rados.rados.0.smithi071.stdout:1527: done (3 left) 2024-09-11T01:39:54.163 INFO:tasks.rados.rados.0.smithi071.stdout:1529: done (2 left) 2024-09-11T01:39:54.163 INFO:tasks.rados.rados.0.smithi071.stdout:1531: done (1 left) 2024-09-11T01:39:54.163 INFO:tasks.rados.rados.0.smithi071.stdout:1532: done (0 left) 2024-09-11T01:39:54.163 INFO:tasks.rados.rados.0.smithi071.stdout:1533: write (excl) oid 37 current snap is 165 2024-09-11T01:39:54.163 INFO:tasks.rados.rados.0.smithi071.stdout:1533: seq_num 577 ranges {497639=545148,1548463=407858,2529302=564422,3528143=169496} 2024-09-11T01:39:54.168 INFO:tasks.rados.rados.0.smithi071.stdout:1533: writing smithi07134777-37 from 497639 to 1042787 tid 1 2024-09-11T01:39:54.173 INFO:tasks.rados.rados.0.smithi071.stdout:1533: writing smithi07134777-37 from 1548463 to 1956321 tid 2 2024-09-11T01:39:54.179 INFO:tasks.rados.rados.0.smithi071.stdout:1533: writing smithi07134777-37 from 2529302 to 3093724 tid 3 2024-09-11T01:39:54.185 INFO:tasks.rados.rados.0.smithi071.stdout:1533: writing smithi07134777-37 from 3528143 to 3697639 tid 4 2024-09-11T01:39:54.185 INFO:tasks.rados.rados.0.smithi071.stdout:1534: read oid 2 snap 164 2024-09-11T01:39:54.185 INFO:tasks.rados.rados.0.smithi071.stdout:1534: expect (ObjNum 503 snap 135 seq_num 503) 2024-09-11T01:39:54.185 INFO:tasks.rados.rados.0.smithi071.stdout:1535: rollback oid 10 current snap is 165 2024-09-11T01:39:54.185 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 10 to 163 2024-09-11T01:39:54.185 INFO:tasks.rados.rados.0.smithi071.stdout:1533: finishing write tid 1 to smithi07134777-37 2024-09-11T01:39:54.185 INFO:tasks.rados.rados.0.smithi071.stdout:1533: finishing write tid 2 to smithi07134777-37 2024-09-11T01:39:54.185 INFO:tasks.rados.rados.0.smithi071.stdout:1536: snap_create 2024-09-11T01:39:54.186 INFO:tasks.rados.rados.0.smithi071.stdout:1533: finishing write tid 3 to smithi07134777-37 2024-09-11T01:39:54.186 INFO:tasks.rados.rados.0.smithi071.stdout:1533: finishing write tid 4 to smithi07134777-37 2024-09-11T01:39:54.186 INFO:tasks.rados.rados.0.smithi071.stdout:1533: finishing write tid 5 to smithi07134777-37 2024-09-11T01:39:54.187 INFO:tasks.rados.rados.0.smithi071.stdout:1533: finishing write tid 6 to smithi07134777-37 2024-09-11T01:39:54.187 INFO:tasks.rados.rados.0.smithi071.stdout:1533: oid 37 updating version 0 to 603 2024-09-11T01:39:54.187 INFO:tasks.rados.rados.0.smithi071.stdout:1533: oid 37 version 603 is already newer than 602 2024-09-11T01:39:54.187 INFO:tasks.rados.rados.0.smithi071.stdout:1533: oid 37 version 603 is already newer than 600 2024-09-11T01:39:54.187 INFO:tasks.rados.rados.0.smithi071.stdout:1533: oid 37 version 603 is already newer than 601 2024-09-11T01:39:54.187 INFO:tasks.rados.rados.0.smithi071.stdout:1533: oid 37 updating version 603 to 604 2024-09-11T01:39:54.187 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 604 (ObjNum 577 snap 165 seq_num 577) dirty exists 2024-09-11T01:39:54.187 INFO:tasks.rados.rados.0.smithi071.stdout:1533: left oid 37 (ObjNum 577 snap 165 seq_num 577) 2024-09-11T01:39:54.190 INFO:tasks.rados.rados.0.smithi071.stdout:1534: expect (ObjNum 503 snap 135 seq_num 503) 2024-09-11T01:39:54.201 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2103737 2024-09-11T01:39:54.205 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 126543 2024-09-11T01:39:54.230 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 86570 2024-09-11T01:39:54.255 INFO:tasks.rados.rados.0.smithi071.stdout:1535: finishing rollback tid 0 to smithi07134777-10 2024-09-11T01:39:54.255 INFO:tasks.rados.rados.0.smithi071.stdout:1535: finishing rollback tid 1 to smithi07134777-10 2024-09-11T01:39:54.255 INFO:tasks.rados.rados.0.smithi071.stdout:1535: finishing rollback tid 2 to smithi07134777-10 2024-09-11T01:39:54.255 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 375 (ObjNum 520 snap 143 seq_num 520) dirty exists 2024-09-11T01:39:54.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0183fff640 10 monclient(hunting): tick 2024-09-11T01:39:54.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0183fff640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:39:54.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0183fff640 1 monclient(hunting): continuing hunt 2024-09-11T01:39:54.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0183fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:39:54.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0183fff640 1 -- 172.21.15.12:0/2840641965 >> v2:172.21.15.12:3300/0 conn(0x7f0174003740 msgr2=0x7f0174005bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:39:54.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0183fff640 1 --2- 172.21.15.12:0/2840641965 >> v2:172.21.15.12:3300/0 conn(0x7f0174003740 0x7f0174005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:54.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0183fff640 1 -- 172.21.15.12:0/2840641965 >> v1:172.21.15.71:6789/0 conn(0x7f01740062b0 legacy=0x7f0174008770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:39:54.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0192d9d640 1 -- 172.21.15.12:0/2840641965 reap_dead start 2024-09-11T01:39:54.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0183fff640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-11T01:39:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0183fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0174008f40 addr v1:172.21.15.12:6789/0 2024-09-11T01:39:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0183fff640 1 --2- 172.21.15.12:0/2840641965 >> v2:172.21.15.12:3300/0 conn(0x7f0174005c60 0x7f0174009a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0183fff640 10 monclient(hunting): picked mon.noname-a con 0x7f0174005c60 addr v2:172.21.15.12:3300/0 2024-09-11T01:39:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0183fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0174008800 addr v1:172.21.15.71:6789/0 2024-09-11T01:39:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0183fff640 1 -- 172.21.15.12:0/2840641965 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f017400b020 con 0x7f0174008f40 2024-09-11T01:39:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0183fff640 1 -- 172.21.15.12:0/2840641965 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0194138350 con 0x7f0174008800 2024-09-11T01:39:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0183fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0183fff640 1 -- 172.21.15.12:0/2840641965 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f01940a4550 con 0x7f0174005c60 2024-09-11T01:39:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0183fff640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0192d9d640 1 -- 172.21.15.12:0/2840641965 >> v1:172.21.15.71:6789/0 conn(0x7f0174008800 legacy=0x7f017400d440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:39:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0191d9b640 1 --2- 172.21.15.12:0/2840641965 >> v2:172.21.15.12:3300/0 conn(0x7f0174005c60 0x7f0174009a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0191d9b640 10 monclient(hunting): get_auth_request con 0x7f0174005c60 auth_method 0 2024-09-11T01:39:54.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0191d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0191d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0191d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0191d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0191d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0191d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0191d9b640 10 monclient(hunting): handle_auth_done global_id 94338 payload 274 2024-09-11T01:39:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0191d9b640 1 -- 172.21.15.12:0/2840641965 >> v1:172.21.15.71:6789/0 conn(0x7f0174008800 legacy=0x7f017400d440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:39:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0191d9b640 1 -- 172.21.15.12:0/2840641965 >> v1:172.21.15.12:6789/0 conn(0x7f0174008f40 legacy=0x7f0174009320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:39:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0191d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:39:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0191d9b640 1 monclient: found mon.noname-a 2024-09-11T01:39:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0191d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0191d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:39:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.529+0000 7f0191d9b640 1 -- 172.21.15.12:0/2840641965 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0174010c20 con 0x7f0174005c60 2024-09-11T01:39:54.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.532+0000 7f0191d9b640 1 -- 172.21.15.12:0/2840641965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0174005c60 msgr2=0x7f0174009a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:54.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.532+0000 7f0191d9b640 1 --2- 172.21.15.12:0/2840641965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0174005c60 0x7f0174009a30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0188007fd0 tx=0x7f018802a4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=982ad5cc67cbda3e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:54.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.532+0000 7f019159a640 1 -- 172.21.15.12:0/2840641965 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0188005bb0 con 0x7f0174005c60 2024-09-11T01:39:54.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.532+0000 7f019159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:54.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.532+0000 7f019159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:39:54.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.532+0000 7f019159a640 10 monclient: dump: 2024-09-11T01:39:54.536 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:54.536 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:54.536 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:54.536 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:54.536 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:54.537 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:54.537 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:54.537 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:54.537 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:54.537 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:54.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.532+0000 7f019159a640 10 monclient: _finish_auth 0 2024-09-11T01:39:54.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.532+0000 7f019159a640 10 monclient: _check_auth_tickets 2024-09-11T01:39:54.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.532+0000 7f019159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:54.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.532+0000 7f019159a640 1 -- 172.21.15.12:0/2840641965 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0188005d50 con 0x7f0174005c60 2024-09-11T01:39:54.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.532+0000 7f019890e640 5 monclient: authenticate success, global_id 94338 2024-09-11T01:39:54.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.532+0000 7f019890e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:39:54.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.532+0000 7f019159a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:54.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.532+0000 7f019890e640 10 monclient: get_monmap_and_config success 2024-09-11T01:39:54.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.532+0000 7f019890e640 4 set_mon_vals no callback set 2024-09-11T01:39:54.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.532+0000 7f019159a640 1 -- 172.21.15.12:0/2840641965 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0188007290 con 0x7f0174005c60 2024-09-11T01:39:54.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.532+0000 7f019159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:54.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.532+0000 7f019159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:39:54.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.532+0000 7f019159a640 10 monclient: dump: 2024-09-11T01:39:54.538 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:54.538 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:54.538 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:54.538 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:54.538 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:54.538 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:54.538 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:54.538 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:54.538 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:54.538 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:54.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.533+0000 7f019890e640 10 monclient: shutdown 2024-09-11T01:39:54.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.533+0000 7f019890e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:54.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.533+0000 7f019890e640 1 -- 172.21.15.12:0/2840641965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0174005c60 msgr2=0x7f0174009a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:54.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.533+0000 7f019890e640 1 --2- 172.21.15.12:0/2840641965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0174005c60 0x7f0174009a30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0188007fd0 tx=0x7f018802a4d0 comp rx=0 tx=0).stop 2024-09-11T01:39:54.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.533+0000 7f019890e640 1 -- 172.21.15.12:0/2840641965 shutdown_connections 2024-09-11T01:39:54.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.533+0000 7f019890e640 1 --2- 172.21.15.12:0/2840641965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0174005c60 0x7f0174009a30 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:54.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.533+0000 7f019890e640 1 -- 172.21.15.12:0/2840641965 >> 172.21.15.12:0/2840641965 conn(0x7f01940a3650 msgr2=0x7f01940a3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:54.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.533+0000 7f019890e640 1 -- 172.21.15.12:0/2840641965 shutdown_connections 2024-09-11T01:39:54.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.533+0000 7f019890e640 1 -- 172.21.15.12:0/2840641965 wait complete. 2024-09-11T01:39:54.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.534+0000 7f019890e640 10 monclient: build_initial_monmap 2024-09-11T01:39:54.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.534+0000 7f019890e640 10 monclient: monmap: 2024-09-11T01:39:54.543 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:54.543 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:39:54.543 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:39:54.543 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:39:54.543 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:54.543 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:54.543 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:39:54.543 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:39:54.543 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:39:54.543 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:54.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.535+0000 7f019890e640 1 Processor -- start 2024-09-11T01:39:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.535+0000 7f019890e640 1 -- start start 2024-09-11T01:39:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.535+0000 7f019890e640 10 monclient: init 2024-09-11T01:39:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.535+0000 7f019890e640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.535+0000 7f019890e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:39:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.535+0000 7f019890e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0174005c60 0x7f01941ccf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.535+0000 7f019890e640 10 monclient(hunting): picked mon.noname-c con 0x7f0174005c60 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.535+0000 7f019890e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0174008800 0x7f0194131450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.535+0000 7f019890e640 10 monclient(hunting): picked mon.noname-a con 0x7f0174008800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:39:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.535+0000 7f019890e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0174008f40 0x7f0194131990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.535+0000 7f019890e640 10 monclient(hunting): picked mon.noname-b con 0x7f0174008f40 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:39:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.535+0000 7f019890e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.535+0000 7f019890e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0194070840 con 0x7f0174008800 2024-09-11T01:39:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.535+0000 7f019890e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.535+0000 7f019890e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0194070340 con 0x7f0174008f40 2024-09-11T01:39:54.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.535+0000 7f019890e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:54.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.535+0000 7f019890e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f01940a0660 con 0x7f0174005c60 2024-09-11T01:39:54.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.535+0000 7f019890e640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:54.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.535+0000 7f019890e640 10 monclient(hunting): authenticate will time out at 1400.978822s 2024-09-11T01:39:54.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.535+0000 7f0191d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0174008800 0x7f0194131450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:54.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.535+0000 7f019259c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0174005c60 0x7f01941ccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:54.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.535+0000 7f0191d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0174008800 0x7f0194131450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52584/0 (socket says 172.21.15.12:52584) 2024-09-11T01:39:54.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.536+0000 7f0191d9b640 1 -- 172.21.15.12:0/1917485482 learned_addr learned my addr 172.21.15.12:0/1917485482 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:39:54.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.535+0000 7f0192d9d640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0174008f40 msgr2=0x7f0194131990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:39:54.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.536+0000 7f0191d9b640 10 monclient(hunting): get_auth_request con 0x7f0174008800 auth_method 0 2024-09-11T01:39:54.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.536+0000 7f0191d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:54.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.536+0000 7f0191d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:54.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.536+0000 7f0191d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:54.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.536+0000 7f019259c640 10 monclient(hunting): get_auth_request con 0x7f0174005c60 auth_method 0 2024-09-11T01:39:54.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.536+0000 7f019259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:54.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.536+0000 7f019259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:54.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.536+0000 7f019259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:54.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.536+0000 7f0192d9d640 1 --2- 172.21.15.12:0/1917485482 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0174008f40 0x7f0194131990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:39:54.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.536+0000 7f0191d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:54.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.536+0000 7f0191d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:54.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.536+0000 7f0191d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:54.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.536+0000 7f019259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:54.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.536+0000 7f019259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:54.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.536+0000 7f019259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:54.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.536+0000 7f0191d9b640 10 monclient(hunting): handle_auth_done global_id 94341 payload 824 2024-09-11T01:39:54.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.536+0000 7f0191d9b640 1 -- 172.21.15.12:0/1917485482 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0174005c60 msgr2=0x7f01941ccf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:54.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.536+0000 7f0191d9b640 1 --2- 172.21.15.12:0/1917485482 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0174005c60 0x7f01941ccf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:54.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.536+0000 7f0191d9b640 1 -- 172.21.15.12:0/1917485482 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0174008f40 msgr2=0x7f0194131990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:39:54.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.536+0000 7f0191d9b640 1 --2- 172.21.15.12:0/1917485482 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0174008f40 0x7f0194131990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:54.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.536+0000 7f0191d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:39:54.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.536+0000 7f0191d9b640 1 monclient: found mon.noname-a 2024-09-11T01:39:54.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.536+0000 7f0191d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:54.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.536+0000 7f0191d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:39:54.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.536+0000 7f0191d9b640 1 -- 172.21.15.12:0/1917485482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0194120af0 con 0x7f0174008800 2024-09-11T01:39:54.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.538+0000 7f0191d9b640 1 -- 172.21.15.12:0/1917485482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0174008800 msgr2=0x7f0194131450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:54.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.538+0000 7f0191d9b640 1 --2- 172.21.15.12:0/1917485482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0174008800 0x7f0194131450 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0188009000 tx=0x7f0188001510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:54.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.539+0000 7f0182ffd640 1 -- 172.21.15.12:0/1917485482 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0188040020 con 0x7f0174008800 2024-09-11T01:39:54.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.539+0000 7f0182ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:54.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.539+0000 7f0182ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:39:54.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.539+0000 7f0182ffd640 10 monclient: dump: 2024-09-11T01:39:54.547 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:54.547 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:54.548 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:54.548 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:54.548 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:54.548 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:54.548 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:54.548 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:54.548 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:54.548 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:54.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.539+0000 7f0182ffd640 10 monclient: _finish_auth 0 2024-09-11T01:39:54.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.539+0000 7f0182ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:39:54.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.539+0000 7f0182ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:54.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.539+0000 7f0182ffd640 1 -- 172.21.15.12:0/1917485482 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0188007480 con 0x7f0174008800 2024-09-11T01:39:54.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.539+0000 7f019890e640 5 monclient: authenticate success, global_id 94341 2024-09-11T01:39:54.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.539+0000 7f019890e640 10 monclient: _renew_subs 2024-09-11T01:39:54.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.539+0000 7f019890e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:54.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.539+0000 7f019890e640 1 -- 172.21.15.12:0/1917485482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0194074fd0 con 0x7f0174008800 2024-09-11T01:39:54.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.539+0000 7f0182ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:54.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.539+0000 7f0182ffd640 1 -- 172.21.15.12:0/1917485482 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0188037040 con 0x7f0174008800 2024-09-11T01:39:54.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.539+0000 7f0182ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:54.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.539+0000 7f0182ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:39:54.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.539+0000 7f0182ffd640 10 monclient: dump: 2024-09-11T01:39:54.549 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:54.549 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:54.549 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:54.549 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:54.549 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:54.549 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:54.549 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:54.550 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:54.550 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:54.550 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:54.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.539+0000 7f019890e640 10 monclient: _renew_subs 2024-09-11T01:39:54.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.539+0000 7f019890e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:54.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.539+0000 7f019890e640 1 -- 172.21.15.12:0/1917485482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0194122e50 con 0x7f0174008800 2024-09-11T01:39:54.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.539+0000 7f0183fff640 4 set_mon_vals no callback set 2024-09-11T01:39:54.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.540+0000 7f0182ffd640 1 -- 172.21.15.12:0/1917485482 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f0188032430 con 0x7f0174008800 2024-09-11T01:39:54.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.540+0000 7f019890e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:54.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.540+0000 7f019890e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:54.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.540+0000 7f019890e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:54.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.540+0000 7f019890e640 1 -- 172.21.15.12:0/1917485482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01940a3100 con 0x7f0174008800 2024-09-11T01:39:54.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.540+0000 7f0182ffd640 1 --2- 172.21.15.12:0/1917485482 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0174040160 0x7f0174042620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:54.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.540+0000 7f0182ffd640 1 -- 172.21.15.12:0/1917485482 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(349..349 src has 285..349) ==== 4739+0+0 (secure 0 0 0) 0x7f01880731b0 con 0x7f0174008800 2024-09-11T01:39:54.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.541+0000 7f019259c640 1 --2- 172.21.15.12:0/1917485482 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0174040160 0x7f0174042620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:54.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.541+0000 7f019259c640 10 monclient: get_auth_request con 0x7f0174040160 auth_method 0 2024-09-11T01:39:54.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.543+0000 7f019259c640 1 --2- 172.21.15.12:0/1917485482 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0174040160 0x7f0174042620 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0184005010 tx=0x7f018400d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:54.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.545+0000 7f0182ffd640 1 -- 172.21.15.12:0/1917485482 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f018804a030 con 0x7f0174008800 2024-09-11T01:39:54.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.545+0000 7f0182ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:54.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.545+0000 7f0182ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:39:54.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.803+0000 7f019890e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:39:54.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.803+0000 7f019890e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:39:54.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.803+0000 7f019890e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:54.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.803+0000 7f019890e640 1 -- 172.21.15.12:0/1917485482 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f01941379f0 con 0x7f0174008800 2024-09-11T01:39:54.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.804+0000 7f0182ffd640 1 -- 172.21.15.12:0/1917485482 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f01941380d0 con 0x7f0174008800 2024-09-11T01:39:54.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.804+0000 7f0182ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:39:54.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.804+0000 7f0182ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:39:54.807 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":104,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:39:54.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.805+0000 7f019890e640 1 -- 172.21.15.12:0/1917485482 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0174040160 msgr2=0x7f0174042620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:54.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.805+0000 7f019890e640 1 --2- 172.21.15.12:0/1917485482 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0174040160 0x7f0174042620 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0184005010 tx=0x7f018400d040 comp rx=0 tx=0).stop 2024-09-11T01:39:54.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.805+0000 7f019890e640 10 monclient: shutdown 2024-09-11T01:39:54.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.805+0000 7f019890e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:54.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.805+0000 7f019890e640 1 -- 172.21.15.12:0/1917485482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0174008800 msgr2=0x7f0194131450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:54.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.805+0000 7f019890e640 1 --2- 172.21.15.12:0/1917485482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0174008800 0x7f0194131450 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0188009000 tx=0x7f0188001510 comp rx=0 tx=0).stop 2024-09-11T01:39:54.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.806+0000 7f019890e640 1 -- 172.21.15.12:0/1917485482 shutdown_connections 2024-09-11T01:39:54.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.806+0000 7f019890e640 1 --2- 172.21.15.12:0/1917485482 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0174008f40 0x7f0194131990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:54.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.806+0000 7f019890e640 1 --2- 172.21.15.12:0/1917485482 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0174040160 0x7f0174042620 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:54.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.806+0000 7f019890e640 1 --2- 172.21.15.12:0/1917485482 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0174008800 0x7f0194131450 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:54.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.806+0000 7f019890e640 1 --2- 172.21.15.12:0/1917485482 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0174005c60 0x7f01941ccf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:54.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.806+0000 7f019890e640 1 -- 172.21.15.12:0/1917485482 >> 172.21.15.12:0/1917485482 conn(0x7f01940a3650 msgr2=0x7f0194135bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:54.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.806+0000 7f019890e640 1 -- 172.21.15.12:0/1917485482 shutdown_connections 2024-09-11T01:39:54.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.806+0000 7f019890e640 1 -- 172.21.15.12:0/1917485482 wait complete. 2024-09-11T01:39:54.819 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:39:54.819 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-11T01:39:54.819 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:39:54.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.918+0000 7f090c317640 10 monclient: get_monmap_and_config 2024-09-11T01:39:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.918+0000 7f090c317640 10 monclient: build_initial_monmap 2024-09-11T01:39:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.918+0000 7f090c317640 10 monclient: monmap: 2024-09-11T01:39:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:39:54.920254+0000 2024-09-11T01:39:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:39:54.920254+0000 2024-09-11T01:39:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:39:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:39:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:39:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:39:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:39:54.922 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:54.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.919+0000 7f090c317640 1 Processor -- start 2024-09-11T01:39:54.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.919+0000 7f090c317640 1 -- start start 2024-09-11T01:39:54.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.919+0000 7f090c317640 10 monclient: init 2024-09-11T01:39:54.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.919+0000 7f090c317640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:54.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.919+0000 7f090c317640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-11T01:39:54.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.919+0000 7f090c317640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f09040a2e20 0x7f090406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:54.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.919+0000 7f090c317640 10 monclient(hunting): picked mon.noname-a con 0x7f09040a2e20 addr v2:172.21.15.12:3300/0 2024-09-11T01:39:54.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.919+0000 7f090c317640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09040a4970 0x7f09040a3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:54.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.919+0000 7f090c317640 10 monclient(hunting): picked mon.noname-b con 0x7f09040a4970 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:54.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.919+0000 7f090c317640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f090406f910 addr v1:172.21.15.12:6789/0 2024-09-11T01:39:54.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.919+0000 7f090c317640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f090406f4f0 con 0x7f090406f910 2024-09-11T01:39:54.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.919+0000 7f090c317640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:54.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.919+0000 7f090c317640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f090406ef30 con 0x7f09040a2e20 2024-09-11T01:39:54.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.919+0000 7f090c317640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:54.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.919+0000 7f090c317640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f090406ec50 con 0x7f09040a4970 2024-09-11T01:39:54.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.919+0000 7f090c317640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:54.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.919+0000 7f090c317640 10 monclient(hunting): authenticate will time out at 1401.362963s 2024-09-11T01:39:54.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f090a0b3640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f09040a2e20 0x7f090406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:54.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f090a8b4640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f090406f910 0x7f0904137fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35140/0 (socket says 172.21.15.12:35140) 2024-09-11T01:39:54.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f09098b2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09040a4970 0x7f09040a3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:54.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f090a8b4640 1 -- 172.21.15.12:0/3525004674 learned_addr learned my addr 172.21.15.12:0/3525004674 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:39:54.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f090a0b3640 10 monclient(hunting): get_auth_request con 0x7f09040a2e20 auth_method 0 2024-09-11T01:39:54.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f090a0b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:54.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f090a0b3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:54.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f090a0b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:54.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f09098b2640 10 monclient(hunting): get_auth_request con 0x7f09040a4970 auth_method 0 2024-09-11T01:39:54.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f09098b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:54.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f09098b2640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:54.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f09098b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:54.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f090a0b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:54.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f090a0b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:54.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f090a0b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:54.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f09098b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:54.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f09098b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:54.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f09098b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:54.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f090a0b3640 10 monclient(hunting): handle_auth_done global_id 94344 payload 274 2024-09-11T01:39:54.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f090a0b3640 1 -- 172.21.15.12:0/3525004674 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09040a4970 msgr2=0x7f09040a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:39:54.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f090a0b3640 1 --2- 172.21.15.12:0/3525004674 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09040a4970 0x7f09040a3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:54.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f090a0b3640 1 -- 172.21.15.12:0/3525004674 >> v1:172.21.15.12:6789/0 conn(0x7f090406f910 legacy=0x7f0904137fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:39:54.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f090a0b3640 10 monclient: _finish_hunting 0 2024-09-11T01:39:54.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f090a0b3640 1 monclient: found mon.noname-a 2024-09-11T01:39:54.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f090a0b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:54.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f090a0b3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:39:54.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f090a0b3640 1 -- 172.21.15.12:0/3525004674 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09040a3740 con 0x7f09040a2e20 2024-09-11T01:39:54.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.920+0000 7f09098b2640 1 --2- 172.21.15.12:0/3525004674 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09040a4970 0x7f09040a3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:39:54.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f090a0b3640 1 -- 172.21.15.12:0/3525004674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09040a2e20 msgr2=0x7f090406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:54.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f090a0b3640 1 --2- 172.21.15.12:0/3525004674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09040a2e20 0x7f090406ded0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f09000097a0 tx=0x7f0900029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae5b026b788e759c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:54.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f09090b1640 1 -- 172.21.15.12:0/3525004674 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0900037070 con 0x7f09040a2e20 2024-09-11T01:39:54.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f09090b1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:54.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f09090b1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:39:54.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f09090b1640 10 monclient: dump: 2024-09-11T01:39:54.933 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:54.933 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:54.933 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:54.933 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:54.933 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:54.934 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:54.934 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:54.934 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:54.934 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:54.934 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:54.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f09090b1640 10 monclient: _finish_auth 0 2024-09-11T01:39:54.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f09090b1640 10 monclient: _check_auth_tickets 2024-09-11T01:39:54.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f09090b1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:54.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f090c317640 5 monclient: authenticate success, global_id 94344 2024-09-11T01:39:54.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f09090b1640 1 -- 172.21.15.12:0/3525004674 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0900032440 con 0x7f09040a2e20 2024-09-11T01:39:54.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f090c317640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:39:54.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f09090b1640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:54.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f090c317640 10 monclient: get_monmap_and_config success 2024-09-11T01:39:54.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f09090b1640 1 -- 172.21.15.12:0/3525004674 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0900032810 con 0x7f09040a2e20 2024-09-11T01:39:54.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f090c317640 4 set_mon_vals no callback set 2024-09-11T01:39:54.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f09090b1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:54.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f09090b1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:39:54.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f09090b1640 10 monclient: dump: 2024-09-11T01:39:54.935 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:54.935 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:54.935 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:54.935 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:54.935 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:54.935 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:54.935 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:54.935 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:54.935 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:54.935 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:54.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f090c317640 10 monclient: shutdown 2024-09-11T01:39:54.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f090c317640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:54.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f090c317640 1 -- 172.21.15.12:0/3525004674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09040a2e20 msgr2=0x7f090406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:54.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f090c317640 1 --2- 172.21.15.12:0/3525004674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09040a2e20 0x7f090406ded0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f09000097a0 tx=0x7f0900029fd0 comp rx=0 tx=0).stop 2024-09-11T01:39:54.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f090c317640 1 -- 172.21.15.12:0/3525004674 shutdown_connections 2024-09-11T01:39:54.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f090c317640 1 --2- 172.21.15.12:0/3525004674 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09040a4970 0x7f09040a3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:54.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f090c317640 1 --2- 172.21.15.12:0/3525004674 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09040a2e20 0x7f090406ded0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:54.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.923+0000 7f090c317640 1 -- 172.21.15.12:0/3525004674 >> 172.21.15.12:0/3525004674 conn(0x7f09040a8600 msgr2=0x7f09040a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:54.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.924+0000 7f090c317640 1 -- 172.21.15.12:0/3525004674 shutdown_connections 2024-09-11T01:39:54.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.924+0000 7f090c317640 1 -- 172.21.15.12:0/3525004674 wait complete. 2024-09-11T01:39:54.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.924+0000 7f090c317640 10 monclient: build_initial_monmap 2024-09-11T01:39:54.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.924+0000 7f090c317640 10 monclient: monmap: 2024-09-11T01:39:54.936 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:54.936 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:39:54.936 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:39:54.936 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:39:54.937 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:54.937 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:54.937 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:39:54.937 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:39:54.937 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:39:54.937 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:54.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.924+0000 7f090c317640 1 Processor -- start 2024-09-11T01:39:54.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090c317640 1 -- start start 2024-09-11T01:39:54.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090c317640 10 monclient: init 2024-09-11T01:39:54.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090c317640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:54.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090c317640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:39:54.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090c317640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f090406f910 0x7f09041cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:54.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090c317640 10 monclient(hunting): picked mon.noname-a con 0x7f090406f910 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:39:54.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090c317640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09040a2e20 0x7f09041cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:54.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090c317640 10 monclient(hunting): picked mon.noname-c con 0x7f09040a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:54.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090c317640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09040a4970 0x7f0904124eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:54.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090c317640 10 monclient(hunting): picked mon.noname-b con 0x7f09040a4970 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:39:54.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090c317640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:54.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090c317640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0904138860 con 0x7f090406f910 2024-09-11T01:39:54.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090c317640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:54.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090c317640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f09041386e0 con 0x7f09040a4970 2024-09-11T01:39:54.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090c317640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:54.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090c317640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f09041389e0 con 0x7f09040a2e20 2024-09-11T01:39:54.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090c317640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:54.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090c317640 10 monclient(hunting): authenticate will time out at 1401.368523s 2024-09-11T01:39:54.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090a0b3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f090406f910 0x7f09041cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:54.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090a0b3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f090406f910 0x7f09041cd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52594/0 (socket says 172.21.15.12:52594) 2024-09-11T01:39:54.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f09098b2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09040a2e20 0x7f09041cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:54.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090a0b3640 1 -- 172.21.15.12:0/745354880 learned_addr learned my addr 172.21.15.12:0/745354880 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:39:54.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090a0b3640 10 monclient(hunting): get_auth_request con 0x7f090406f910 auth_method 0 2024-09-11T01:39:54.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090a0b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:54.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090a0b3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:54.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090a0b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:54.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f09098b2640 10 monclient(hunting): get_auth_request con 0x7f09040a2e20 auth_method 0 2024-09-11T01:39:54.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f09098b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:54.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f09098b2640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:54.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f09098b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:54.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090a8b4640 1 -- 172.21.15.12:0/745354880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09040a4970 msgr2=0x7f0904124eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:39:54.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090a0b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:54.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090a0b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:54.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090a0b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:54.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090a8b4640 1 --2- 172.21.15.12:0/745354880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09040a4970 0x7f0904124eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:39:54.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f09098b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:54.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f09098b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:54.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f09098b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:54.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090a0b3640 10 monclient(hunting): handle_auth_done global_id 94347 payload 824 2024-09-11T01:39:54.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090a0b3640 1 -- 172.21.15.12:0/745354880 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09040a2e20 msgr2=0x7f09041cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:54.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090a0b3640 1 --2- 172.21.15.12:0/745354880 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09040a2e20 0x7f09041cd9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:54.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090a0b3640 1 -- 172.21.15.12:0/745354880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09040a4970 msgr2=0x7f0904124eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:39:54.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090a0b3640 1 --2- 172.21.15.12:0/745354880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09040a4970 0x7f0904124eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:54.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090a0b3640 10 monclient: _finish_hunting 0 2024-09-11T01:39:54.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090a0b3640 1 monclient: found mon.noname-a 2024-09-11T01:39:54.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090a0b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:54.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090a0b3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:39:54.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.925+0000 7f090a0b3640 1 -- 172.21.15.12:0/745354880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f090409f610 con 0x7f090406f910 2024-09-11T01:39:54.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.928+0000 7f090a0b3640 1 -- 172.21.15.12:0/745354880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f090406f910 msgr2=0x7f09041cd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:54.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.928+0000 7f090a0b3640 1 --2- 172.21.15.12:0/745354880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f090406f910 0x7f09041cd470 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0900000c00 tx=0x7f09000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:54.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.928+0000 7f08faffd640 1 -- 172.21.15.12:0/745354880 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0900037040 con 0x7f090406f910 2024-09-11T01:39:54.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.928+0000 7f08faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:54.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.928+0000 7f08faffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:39:54.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.928+0000 7f08faffd640 10 monclient: dump: 2024-09-11T01:39:54.941 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:54.941 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:54.941 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:54.941 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:54.941 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:54.942 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:54.942 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:54.942 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:54.942 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:54.942 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:54.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.928+0000 7f08faffd640 10 monclient: _finish_auth 0 2024-09-11T01:39:54.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.928+0000 7f08faffd640 10 monclient: _check_auth_tickets 2024-09-11T01:39:54.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.928+0000 7f08faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:54.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.928+0000 7f08faffd640 1 -- 172.21.15.12:0/745354880 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0900042070 con 0x7f090406f910 2024-09-11T01:39:54.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.928+0000 7f08faffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:54.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.928+0000 7f08faffd640 1 -- 172.21.15.12:0/745354880 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090003bb00 con 0x7f090406f910 2024-09-11T01:39:54.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.928+0000 7f090c317640 5 monclient: authenticate success, global_id 94347 2024-09-11T01:39:54.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.928+0000 7f090c317640 10 monclient: _renew_subs 2024-09-11T01:39:54.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.928+0000 7f090c317640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:54.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.928+0000 7f090c317640 1 -- 172.21.15.12:0/745354880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0904126970 con 0x7f090406f910 2024-09-11T01:39:54.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.928+0000 7f08faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:54.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.928+0000 7f08faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:39:54.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.928+0000 7f08faffd640 10 monclient: dump: 2024-09-11T01:39:54.943 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:54.943 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:54.943 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:54.943 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:54.943 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:54.943 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:54.943 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:54.943 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:54.943 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:54.943 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:54.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.928+0000 7f09088b0640 4 set_mon_vals no callback set 2024-09-11T01:39:54.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.928+0000 7f090c317640 10 monclient: _renew_subs 2024-09-11T01:39:54.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.928+0000 7f090c317640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:39:54.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.928+0000 7f090c317640 1 -- 172.21.15.12:0/745354880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09041296b0 con 0x7f090406f910 2024-09-11T01:39:54.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.929+0000 7f090c317640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:54.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.929+0000 7f090c317640 1 --2- 172.21.15.12:0/745354880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09041253f0 0x7f09041257d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:54.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.929+0000 7f090c317640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:54.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.929+0000 7f090c317640 1 -- 172.21.15.12:0/745354880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f090406a060 con 0x7f09041253f0 2024-09-11T01:39:54.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.929+0000 7f09098b2640 1 --2- 172.21.15.12:0/745354880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09041253f0 0x7f09041257d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:54.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.929+0000 7f09098b2640 10 monclient: get_auth_request con 0x7f09041253f0 auth_method 0 2024-09-11T01:39:54.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.929+0000 7f09098b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:54.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.929+0000 7f09098b2640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:54.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.929+0000 7f09098b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:54.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.929+0000 7f09098b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:54.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.929+0000 7f09098b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:54.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.929+0000 7f09098b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:54.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.929+0000 7f08faffd640 1 -- 172.21.15.12:0/745354880 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f090003a030 con 0x7f090406f910 2024-09-11T01:39:54.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.929+0000 7f09098b2640 10 monclient(hunting): handle_auth_done global_id 94350 payload 824 2024-09-11T01:39:54.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.929+0000 7f09098b2640 1 -- 172.21.15.12:0/745354880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f09040a6220 con 0x7f09041253f0 2024-09-11T01:39:54.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.929+0000 7f08faffd640 1 --2- 172.21.15.12:0/745354880 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f08e803e1f0 0x7f08e80406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:54.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.929+0000 7f08faffd640 1 -- 172.21.15.12:0/745354880 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(349..349 src has 285..349) ==== 4739+0+0 (secure 0 0 0) 0x7f09000731e0 con 0x7f090406f910 2024-09-11T01:39:54.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.929+0000 7f090a8b4640 1 --2- 172.21.15.12:0/745354880 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f08e803e1f0 0x7f08e80406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:54.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.929+0000 7f090a8b4640 10 monclient: get_auth_request con 0x7f08e803e1f0 auth_method 0 2024-09-11T01:39:54.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.931+0000 7f09098b2640 1 -- 172.21.15.12:0/745354880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09041253f0 msgr2=0x7f09041257d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:54.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.931+0000 7f09098b2640 1 --2- 172.21.15.12:0/745354880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09041253f0 0x7f09041257d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f08f40118b0 tx=0x7f08f4011d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:54.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.931+0000 7f08faffd640 1 -- 172.21.15.12:0/745354880 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f400fe50 con 0x7f09041253f0 2024-09-11T01:39:54.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.931+0000 7f08faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:54.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.931+0000 7f08faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:39:54.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.931+0000 7f08faffd640 10 monclient: dump: 2024-09-11T01:39:54.947 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:54.948 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:54.948 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:54.948 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:54.948 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:54.948 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:54.948 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:54.948 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:54.948 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:54.948 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:54.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.932+0000 7f08faffd640 1 -- 172.21.15.12:0/745354880 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f08f400fe50 con 0x7f09041253f0 2024-09-11T01:39:54.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.932+0000 7f08faffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:54.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.932+0000 7f08faffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:39:54.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.932+0000 7f08faffd640 1 -- 172.21.15.12:0/745354880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09041253f0 msgr2=0x7f09041257d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:54.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.932+0000 7f08faffd640 1 --2- 172.21.15.12:0/745354880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09041253f0 0x7f09041257d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f08f40118b0 tx=0x7f08f4011d70 comp rx=0 tx=0).stop 2024-09-11T01:39:54.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.932+0000 7f08faffd640 1 -- 172.21.15.12:0/745354880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09041253f0 msgr2=0x7f09041257d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:39:54.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.932+0000 7f08faffd640 1 --2- 172.21.15.12:0/745354880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09041253f0 0x7f09041257d0 secure :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f08f40118b0 tx=0x7f08f4011d70 comp rx=0 tx=0).stop 2024-09-11T01:39:54.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.932+0000 7f090a8b4640 1 --2- 172.21.15.12:0/745354880 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f08e803e1f0 0x7f08e80406b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f090406fd60 tx=0x7f08fc004fd0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:54.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.938+0000 7f090c317640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:39:54.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.938+0000 7f090c317640 1 --2- 172.21.15.12:0/745354880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0904125d10 0x7f09040a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:54.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.938+0000 7f090c317640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:54.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.938+0000 7f090c317640 1 -- 172.21.15.12:0/745354880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f09041288a0 con 0x7f0904125d10 2024-09-11T01:39:54.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.938+0000 7f09098b2640 1 --2- 172.21.15.12:0/745354880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0904125d10 0x7f09040a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:54.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.938+0000 7f09098b2640 10 monclient: get_auth_request con 0x7f0904125d10 auth_method 0 2024-09-11T01:39:54.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.938+0000 7f09098b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:54.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.938+0000 7f09098b2640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:54.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.938+0000 7f09098b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:54.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.938+0000 7f09098b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:54.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.938+0000 7f09098b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:54.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.938+0000 7f09098b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:54.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.938+0000 7f09098b2640 10 monclient(hunting): handle_auth_done global_id 94353 payload 824 2024-09-11T01:39:54.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.938+0000 7f09098b2640 1 -- 172.21.15.12:0/745354880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f090406ded0 con 0x7f0904125d10 2024-09-11T01:39:54.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.940+0000 7f09098b2640 1 -- 172.21.15.12:0/745354880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0904125d10 msgr2=0x7f09040a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:54.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.940+0000 7f09098b2640 1 --2- 172.21.15.12:0/745354880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0904125d10 0x7f09040a0830 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f08f40118b0 tx=0x7f08f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:54.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.940+0000 7f08faffd640 1 -- 172.21.15.12:0/745354880 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f40222e0 con 0x7f0904125d10 2024-09-11T01:39:54.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.940+0000 7f08faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:54.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.940+0000 7f08faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:39:54.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.940+0000 7f08faffd640 10 monclient: dump: 2024-09-11T01:39:54.950 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:54.950 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:54.950 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:54.951 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:54.951 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:54.951 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:54.951 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:54.951 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:54.951 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:54.951 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:54.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.941+0000 7f08faffd640 1 -- 172.21.15.12:0/745354880 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4700 (secure 0 0 0) 0x7f08f402c440 con 0x7f0904125d10 2024-09-11T01:39:54.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.941+0000 7f08faffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:39:54.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.941+0000 7f08faffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:39:54.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.941+0000 7f08faffd640 1 -- 172.21.15.12:0/745354880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0904125d10 msgr2=0x7f09040a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:54.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.941+0000 7f08faffd640 1 --2- 172.21.15.12:0/745354880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0904125d10 0x7f09040a0830 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f08f40118b0 tx=0x7f08f40046c0 comp rx=0 tx=0).stop 2024-09-11T01:39:54.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.941+0000 7f08faffd640 1 -- 172.21.15.12:0/745354880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0904125d10 msgr2=0x7f09040a0830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:39:54.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.941+0000 7f08faffd640 1 --2- 172.21.15.12:0/745354880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0904125d10 0x7f09040a0830 secure :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f08f40118b0 tx=0x7f08f40046c0 comp rx=0 tx=0).stop 2024-09-11T01:39:54.952 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:39:54.952 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:39:54.952 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:39:54.952 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:39:54.952 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 104, 2024-09-11T01:39:54.952 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 141036, 2024-09-11T01:39:54.952 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:39:54.952 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:39:54.952 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:39:54.952 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:54.952 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:39:54.953 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:39:54.953 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:39:54.953 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:39:54.953 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:39:54.953 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:39:54.953 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:39:54.953 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:39:54.953 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:39:54.953 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:39:54.953 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:39:54.953 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:39:54.953 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:39:54.953 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:39:54.953 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:39:54.954 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:54.954 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:39:54.954 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:39:54.954 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:39:54.954 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:39:54.954 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:39:54.954 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:39:54.954 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:39:54.954 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:39:54.954 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:39:54.954 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:39:54.954 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:39:54.955 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:39:54.955 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:39:54.955 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:54.955 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:54.955 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:39:54.955 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:39:54.955 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:39:54.955 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:39:54.955 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:39:54.955 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:39:54.955 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:39:54.955 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:39:54.955 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:39:54.955 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:39:54.955 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:39:54.956 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:39:54.956 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:39:54.956 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:39:54.956 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:39:54.956 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:39:54.956 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:39:54.956 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:39:54.956 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:39:54.956 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:39:54.956 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:39:54.956 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:39:54.956 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:39:54.956 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:39:54.956 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:39:54.957 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:39:54.957 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:39:54.957 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:39:54.957 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:54.957 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:39:54.957 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:54.957 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:39:54.957 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:54.957 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:39:54.957 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:39:54.957 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:39:54.957 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:39:54.957 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:54.957 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:39:54.958 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:39:54.958 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:54.958 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:54.958 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:54.958 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:39:54.958 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:39:54.958 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:54.958 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:54.958 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:54.958 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:54.958 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:39:54.958 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:39:54.958 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:39:54.958 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:39:54.958 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:39:54.959 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:54.959 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:54.959 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:39:54.959 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:39:54.959 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:39:54.959 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:39:54.959 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:54.959 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:39:54.959 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:39:54.959 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:54.959 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:54.959 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:54.959 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:39:54.959 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:39:54.959 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:54.960 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:54.960 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:54.960 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:54.960 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:39:54.960 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:39:54.960 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:39:54.960 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:39:54.960 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:39:54.960 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:54.960 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:54.960 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:39:54.960 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:39:54.960 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:39:54.961 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:39:54.961 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:54.961 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:39:54.961 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:39:54.961 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:54.961 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:54.961 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:54.961 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:39:54.961 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:39:54.961 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:54.962 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:54.962 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:54.962 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:54.962 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:39:54.962 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:39:54.962 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:39:54.962 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:39:54.962 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:39:54.962 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:54.962 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:54.962 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:54.962 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:39:54.962 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:39:54.963 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:54.963 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:39:54.963 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:39:54.963 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:39:54.963 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:54.963 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:54.963 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:39:54.963 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:54.963 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:39:54.963 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:39:54.963 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:39:54.963 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:54.963 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:54.963 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:39:54.964 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:54.964 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:39:54.964 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:39:54.964 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 7 2024-09-11T01:39:54.964 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:54.964 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:54.964 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:54.964 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:39:54.964 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:39:54.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.941+0000 7f090c317640 1 -- 172.21.15.12:0/745354880 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f08e803e1f0 msgr2=0x7f08e80406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:54.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.941+0000 7f090c317640 1 --2- 172.21.15.12:0/745354880 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f08e803e1f0 0x7f08e80406b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f090406fd60 tx=0x7f08fc004fd0 comp rx=0 tx=0).stop 2024-09-11T01:39:54.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.941+0000 7f090c317640 10 monclient: shutdown 2024-09-11T01:39:54.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.941+0000 7f090c317640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:54.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.941+0000 7f090c317640 1 -- 172.21.15.12:0/745354880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f090406f910 msgr2=0x7f09041cd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:54.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.941+0000 7f090c317640 1 --2- 172.21.15.12:0/745354880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f090406f910 0x7f09041cd470 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0900000c00 tx=0x7f09000046c0 comp rx=0 tx=0).stop 2024-09-11T01:39:54.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.941+0000 7f090a8b4640 1 -- 172.21.15.12:0/745354880 reap_dead start 2024-09-11T01:39:54.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.941+0000 7f090a8b4640 1 -- 172.21.15.12:0/745354880 reap_dead start 2024-09-11T01:39:54.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.941+0000 7f090c317640 1 -- 172.21.15.12:0/745354880 shutdown_connections 2024-09-11T01:39:54.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.941+0000 7f090c317640 1 -- 172.21.15.12:0/745354880 >> 172.21.15.12:0/745354880 conn(0x7f09040a8600 msgr2=0x7f0904072e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:54.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.941+0000 7f090c317640 1 -- 172.21.15.12:0/745354880 shutdown_connections 2024-09-11T01:39:54.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:54.941+0000 7f090c317640 1 -- 172.21.15.12:0/745354880 wait complete. 2024-09-11T01:39:54.966 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:39:54.967 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T01:39:54.967 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:39:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.067+0000 7fcca9f17640 10 monclient: get_monmap_and_config 2024-09-11T01:39:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.067+0000 7fcca9f17640 10 monclient: build_initial_monmap 2024-09-11T01:39:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.067+0000 7fcca9f17640 10 monclient: monmap: 2024-09-11T01:39:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:39:55.069307+0000 2024-09-11T01:39:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:39:55.069307+0000 2024-09-11T01:39:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:39:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:39:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:39:55.071 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:39:55.071 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:39:55.071 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:55.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.068+0000 7fcca9f17640 1 Processor -- start 2024-09-11T01:39:55.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.068+0000 7fcca9f17640 1 -- start start 2024-09-11T01:39:55.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.068+0000 7fcca9f17640 10 monclient: init 2024-09-11T01:39:55.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.068+0000 7fcca9f17640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:55.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.068+0000 7fcca9f17640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-11T01:39:55.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.068+0000 7fcca9f17640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca40a1380 0x7fcca40a1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:55.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.068+0000 7fcca9f17640 10 monclient(hunting): picked mon.noname-b con 0x7fcca40a1380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:55.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.068+0000 7fcca9f17640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcca40a0370 addr v1:172.21.15.12:6789/0 2024-09-11T01:39:55.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.068+0000 7fcca9f17640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcca40a0c90 0x7fcca41361f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:55.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.068+0000 7fcca9f17640 10 monclient(hunting): picked mon.noname-c con 0x7fcca40a0c90 addr v2:172.21.15.71:3300/0 2024-09-11T01:39:55.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca9f17640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcca40a1ca0 con 0x7fcca40a0370 2024-09-11T01:39:55.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca9f17640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca9f17640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fcca40a21c0 con 0x7fcca40a0c90 2024-09-11T01:39:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca9f17640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca9f17640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcca40a24a0 con 0x7fcca40a1380 2024-09-11T01:39:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca9f17640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca9f17640 10 monclient(hunting): authenticate will time out at 1401.512077s 2024-09-11T01:39:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca37fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca40a1380 0x7fcca40a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca2ffd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fcca40a0370 0x7fcca4135ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35152/0 (socket says 172.21.15.12:35152) 2024-09-11T01:39:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca2ffd640 1 -- 172.21.15.12:0/2090658024 learned_addr learned my addr 172.21.15.12:0/2090658024 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:39:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca37fe640 10 monclient(hunting): get_auth_request con 0x7fcca40a1380 auth_method 0 2024-09-11T01:39:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca3fff640 1 -- 172.21.15.12:0/2090658024 >> v2:172.21.15.71:3300/0 conn(0x7fcca40a0c90 msgr2=0x7fcca41361f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:39:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca3fff640 1 --2- 172.21.15.12:0/2090658024 >> v2:172.21.15.71:3300/0 conn(0x7fcca40a0c90 0x7fcca41361f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:39:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca37fe640 10 monclient(hunting): handle_auth_done global_id 114179 payload 274 2024-09-11T01:39:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca37fe640 1 -- 172.21.15.12:0/2090658024 >> v2:172.21.15.71:3300/0 conn(0x7fcca40a0c90 msgr2=0x7fcca41361f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:39:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca37fe640 1 --2- 172.21.15.12:0/2090658024 >> v2:172.21.15.71:3300/0 conn(0x7fcca40a0c90 0x7fcca41361f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca37fe640 1 -- 172.21.15.12:0/2090658024 >> v1:172.21.15.12:6789/0 conn(0x7fcca40a0370 legacy=0x7fcca4135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:39:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca37fe640 10 monclient: _finish_hunting 0 2024-09-11T01:39:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca37fe640 1 monclient: found mon.noname-b 2024-09-11T01:39:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:39:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.069+0000 7fcca37fe640 1 -- 172.21.15.12:0/2090658024 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcca40a2780 con 0x7fcca40a1380 2024-09-11T01:39:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca37fe640 1 -- 172.21.15.12:0/2090658024 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca40a1380 msgr2=0x7fcca40a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca37fe640 1 --2- 172.21.15.12:0/2090658024 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca40a1380 0x7fcca40a1760 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcc940097a0 tx=0x7fcc94029fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c251c00600e7eb8c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca27fc640 1 -- 172.21.15.12:0/2090658024 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc94037070 con 0x7fcca40a1380 2024-09-11T01:39:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:39:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca27fc640 10 monclient: dump: 2024-09-11T01:39:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:55.079 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca27fc640 10 monclient: _finish_auth 0 2024-09-11T01:39:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca27fc640 10 monclient: _check_auth_tickets 2024-09-11T01:39:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca27fc640 1 -- 172.21.15.12:0/2090658024 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc94032640 con 0x7fcca40a1380 2024-09-11T01:39:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca9f17640 5 monclient: authenticate success, global_id 114179 2024-09-11T01:39:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca9f17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:39:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca9f17640 10 monclient: get_monmap_and_config success 2024-09-11T01:39:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca27fc640 1 -- 172.21.15.12:0/2090658024 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc94032a10 con 0x7fcca40a1380 2024-09-11T01:39:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca9f17640 4 set_mon_vals no callback set 2024-09-11T01:39:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:39:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca27fc640 10 monclient: dump: 2024-09-11T01:39:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:55.081 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca9f17640 10 monclient: shutdown 2024-09-11T01:39:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca9f17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca9f17640 1 -- 172.21.15.12:0/2090658024 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca40a1380 msgr2=0x7fcca40a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca9f17640 1 --2- 172.21.15.12:0/2090658024 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca40a1380 0x7fcca40a1760 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcc940097a0 tx=0x7fcc94029fe0 comp rx=0 tx=0).stop 2024-09-11T01:39:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca9f17640 1 -- 172.21.15.12:0/2090658024 shutdown_connections 2024-09-11T01:39:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca9f17640 1 --2- 172.21.15.12:0/2090658024 >> v2:172.21.15.71:3300/0 conn(0x7fcca40a0c90 0x7fcca41361f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca9f17640 1 --2- 172.21.15.12:0/2090658024 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca40a1380 0x7fcca40a1760 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca9f17640 1 -- 172.21.15.12:0/2090658024 >> 172.21.15.12:0/2090658024 conn(0x7fcca40a2f70 msgr2=0x7fcca40a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.072+0000 7fcca9f17640 1 -- 172.21.15.12:0/2090658024 shutdown_connections 2024-09-11T01:39:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.073+0000 7fcca9f17640 1 -- 172.21.15.12:0/2090658024 wait complete. 2024-09-11T01:39:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.073+0000 7fcca9f17640 10 monclient: build_initial_monmap 2024-09-11T01:39:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.073+0000 7fcca9f17640 10 monclient: monmap: 2024-09-11T01:39:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:39:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:39:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:39:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:39:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:39:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:39:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:39:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:39:55.083 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.073+0000 7fcca9f17640 1 Processor -- start 2024-09-11T01:39:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca9f17640 1 -- start start 2024-09-11T01:39:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca9f17640 10 monclient: init 2024-09-11T01:39:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca9f17640 10 monclient: _reopen_session rank -1 2024-09-11T01:39:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca9f17640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:39:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca9f17640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcca40a0370 0x7fcca41cad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca9f17640 10 monclient(hunting): picked mon.noname-b con 0x7fcca40a0370 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:39:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca9f17640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca40a0c90 0x7fcca4135c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca9f17640 10 monclient(hunting): picked mon.noname-c con 0x7fcca40a0c90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:39:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca9f17640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcca40a1380 0x7fcca4136170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca9f17640 10 monclient(hunting): picked mon.noname-a con 0x7fcca40a1380 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:39:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca9f17640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca9f17640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcca409ce20 con 0x7fcca40a1380 2024-09-11T01:39:55.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca9f17640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:55.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca9f17640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fcca409cca0 con 0x7fcca40a0370 2024-09-11T01:39:55.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca9f17640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:55.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca9f17640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcca4136730 con 0x7fcca40a0c90 2024-09-11T01:39:55.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca9f17640 10 monclient(hunting): _renew_subs 2024-09-11T01:39:55.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca9f17640 10 monclient(hunting): authenticate will time out at 1401.517519s 2024-09-11T01:39:55.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca2ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca40a0c90 0x7fcca4135c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:55.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcca40a1380 0x7fcca4136170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:55.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca37fe640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcca40a0370 msgr2=0x7fcca41cad90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:39:55.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca37fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcca40a0370 0x7fcca41cad90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:39:55.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca2ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca40a0c90 0x7fcca4135c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:49558/0 (socket says 172.21.15.12:49558) 2024-09-11T01:39:55.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca2ffd640 1 -- 172.21.15.12:0/1620341067 learned_addr learned my addr 172.21.15.12:0/1620341067 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:39:55.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcca40a1380 0x7fcca4136170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52616/0 (socket says 172.21.15.12:52616) 2024-09-11T01:39:55.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca2ffd640 10 monclient(hunting): get_auth_request con 0x7fcca40a0c90 auth_method 0 2024-09-11T01:39:55.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:55.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:55.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:55.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca3fff640 10 monclient(hunting): get_auth_request con 0x7fcca40a1380 auth_method 0 2024-09-11T01:39:55.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:55.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:55.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:55.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:55.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:55.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:55.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:55.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:55.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:55.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca2ffd640 10 monclient(hunting): handle_auth_done global_id 114182 payload 824 2024-09-11T01:39:55.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca2ffd640 1 -- 172.21.15.12:0/1620341067 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcca40a0370 msgr2=0x7fcca41cad90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:39:55.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca2ffd640 1 --2- 172.21.15.12:0/1620341067 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcca40a0370 0x7fcca41cad90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:55.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca2ffd640 1 -- 172.21.15.12:0/1620341067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcca40a1380 msgr2=0x7fcca4136170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:55.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca2ffd640 1 --2- 172.21.15.12:0/1620341067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcca40a1380 0x7fcca4136170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:55.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca2ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:39:55.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca2ffd640 1 monclient: found mon.noname-c 2024-09-11T01:39:55.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:39:55.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:39:55.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca2ffd640 1 -- 172.21.15.12:0/1620341067 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcca4127350 con 0x7fcca40a0c90 2024-09-11T01:39:55.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.074+0000 7fcca3fff640 1 --2- 172.21.15.12:0/1620341067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcca40a1380 0x7fcca4136170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:39:55.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.077+0000 7fcca2ffd640 1 -- 172.21.15.12:0/1620341067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca40a0c90 msgr2=0x7fcca4135c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:55.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.077+0000 7fcca2ffd640 1 --2- 172.21.15.12:0/1620341067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca40a0c90 0x7fcca4135c30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcc9800c460 tx=0x7fcc9800c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:55.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.077+0000 7fcc8bfff640 1 -- 172.21.15.12:0/1620341067 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc98007070 con 0x7fcca40a0c90 2024-09-11T01:39:55.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.077+0000 7fcc8bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:55.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.077+0000 7fcc8bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:39:55.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.077+0000 7fcc8bfff640 10 monclient: dump: 2024-09-11T01:39:55.091 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:55.091 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:55.091 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:55.091 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:55.091 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:55.091 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:55.091 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:55.092 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:55.092 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:55.092 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:55.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.077+0000 7fcc8bfff640 10 monclient: _finish_auth 0 2024-09-11T01:39:55.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.077+0000 7fcc8bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:39:55.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.077+0000 7fcc8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:39:55.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.077+0000 7fcca9f17640 5 monclient: authenticate success, global_id 114182 2024-09-11T01:39:55.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.077+0000 7fcc8bfff640 1 -- 172.21.15.12:0/1620341067 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc98007230 con 0x7fcca40a0c90 2024-09-11T01:39:55.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.077+0000 7fcc8bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:39:55.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.077+0000 7fcc8bfff640 1 -- 172.21.15.12:0/1620341067 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc98007620 con 0x7fcca40a0c90 2024-09-11T01:39:55.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.077+0000 7fcca9f17640 10 monclient: _renew_subs 2024-09-11T01:39:55.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.077+0000 7fcca9f17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:39:55.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.077+0000 7fcca9f17640 1 -- 172.21.15.12:0/1620341067 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcca4127630 con 0x7fcca40a0c90 2024-09-11T01:39:55.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.077+0000 7fcc8bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:55.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.077+0000 7fcc8bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:39:55.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.077+0000 7fcca1ffb640 4 set_mon_vals no callback set 2024-09-11T01:39:55.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.077+0000 7fcc8bfff640 10 monclient: dump: 2024-09-11T01:39:55.093 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:55.093 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:55.093 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:55.093 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:55.093 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:55.093 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:55.093 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:55.093 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:55.093 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:55.094 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:55.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.077+0000 7fcca9f17640 10 monclient: _renew_subs 2024-09-11T01:39:55.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.077+0000 7fcca9f17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:39:55.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.077+0000 7fcca9f17640 1 -- 172.21.15.12:0/1620341067 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcca406ff60 con 0x7fcca40a0c90 2024-09-11T01:39:55.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.078+0000 7fcca9f17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:55.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.078+0000 7fcca9f17640 1 --2- 172.21.15.12:0/1620341067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca4123fd0 0x7fcca402d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:55.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.078+0000 7fcca9f17640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:55.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.078+0000 7fcca9f17640 1 -- 172.21.15.12:0/1620341067 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcca409f7a0 con 0x7fcca4123fd0 2024-09-11T01:39:55.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.078+0000 7fcca37fe640 1 --2- 172.21.15.12:0/1620341067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca4123fd0 0x7fcca402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:55.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.078+0000 7fcc8bfff640 1 -- 172.21.15.12:0/1620341067 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fcc9801b400 con 0x7fcca40a0c90 2024-09-11T01:39:55.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.078+0000 7fcca37fe640 10 monclient: get_auth_request con 0x7fcca4123fd0 auth_method 0 2024-09-11T01:39:55.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.078+0000 7fcca37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:55.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.078+0000 7fcca37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:55.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.078+0000 7fcca37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:55.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.078+0000 7fcca37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:55.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.078+0000 7fcca37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:55.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.078+0000 7fcca37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:55.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.078+0000 7fcca37fe640 10 monclient(hunting): handle_auth_done global_id 114185 payload 824 2024-09-11T01:39:55.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.078+0000 7fcca37fe640 1 -- 172.21.15.12:0/1620341067 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcca4073620 con 0x7fcca4123fd0 2024-09-11T01:39:55.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.078+0000 7fcc8bfff640 1 --2- 172.21.15.12:0/1620341067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcc9003e240 0x7fcc90040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:55.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.078+0000 7fcc8bfff640 1 -- 172.21.15.12:0/1620341067 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(349..349 src has 285..349) ==== 4739+0+0 (secure 0 0 0) 0x7fcc98052920 con 0x7fcca40a0c90 2024-09-11T01:39:55.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.078+0000 7fcca3fff640 1 --2- 172.21.15.12:0/1620341067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcc9003e240 0x7fcc90040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:55.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.079+0000 7fcca3fff640 10 monclient: get_auth_request con 0x7fcc9003e240 auth_method 0 2024-09-11T01:39:55.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.080+0000 7fcca37fe640 1 -- 172.21.15.12:0/1620341067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca4123fd0 msgr2=0x7fcca402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:55.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.080+0000 7fcca37fe640 1 --2- 172.21.15.12:0/1620341067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca4123fd0 0x7fcca402d440 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcca406ce00 tx=0x7fcc940046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:55.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.081+0000 7fcc8bfff640 1 -- 172.21.15.12:0/1620341067 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc94042070 con 0x7fcca4123fd0 2024-09-11T01:39:55.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.081+0000 7fcc8bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:55.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.081+0000 7fcc8bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:39:55.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.081+0000 7fcc8bfff640 10 monclient: dump: 2024-09-11T01:39:55.096 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:55.096 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:55.096 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:55.096 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:55.096 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:55.096 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:55.096 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:55.096 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:55.096 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:55.096 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:55.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.081+0000 7fcca3fff640 1 --2- 172.21.15.12:0/1620341067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcc9003e240 0x7fcc90040700 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcca40a1120 tx=0x7fcc9c0095f0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:55.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.081+0000 7fcc8bfff640 1 -- 172.21.15.12:0/1620341067 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcc9402ae20 con 0x7fcca4123fd0 2024-09-11T01:39:55.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.081+0000 7fcc8bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:39:55.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.081+0000 7fcc8bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:39:55.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.081+0000 7fcc8bfff640 1 -- 172.21.15.12:0/1620341067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca4123fd0 msgr2=0x7fcca402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:55.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.081+0000 7fcc8bfff640 1 --2- 172.21.15.12:0/1620341067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca4123fd0 0x7fcca402d440 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcca406ce00 tx=0x7fcc940046c0 comp rx=0 tx=0).stop 2024-09-11T01:39:55.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.081+0000 7fcc8bfff640 1 -- 172.21.15.12:0/1620341067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca4123fd0 msgr2=0x7fcca402d440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:39:55.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.081+0000 7fcc8bfff640 1 --2- 172.21.15.12:0/1620341067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca4123fd0 0x7fcca402d440 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:55.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.087+0000 7fcca9f17640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:39:55.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.087+0000 7fcca9f17640 1 --2- 172.21.15.12:0/1620341067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca41243b0 0x7fcca402d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:39:55.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.087+0000 7fcca9f17640 10 monclient(hunting): start opening mon connection 2024-09-11T01:39:55.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.087+0000 7fcca9f17640 1 -- 172.21.15.12:0/1620341067 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcca4126f20 con 0x7fcca41243b0 2024-09-11T01:39:55.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.087+0000 7fcca37fe640 1 --2- 172.21.15.12:0/1620341067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca41243b0 0x7fcca402d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:39:55.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.087+0000 7fcca37fe640 10 monclient: get_auth_request con 0x7fcca41243b0 auth_method 0 2024-09-11T01:39:55.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.087+0000 7fcca37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:39:55.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.087+0000 7fcca37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:39:55.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.087+0000 7fcca37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:39:55.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.087+0000 7fcca37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:39:55.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.087+0000 7fcca37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:39:55.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.087+0000 7fcca37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:39:55.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.088+0000 7fcca37fe640 10 monclient(hunting): handle_auth_done global_id 114188 payload 824 2024-09-11T01:39:55.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.088+0000 7fcca37fe640 1 -- 172.21.15.12:0/1620341067 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcca409d480 con 0x7fcca41243b0 2024-09-11T01:39:55.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.090+0000 7fcca37fe640 1 -- 172.21.15.12:0/1620341067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca41243b0 msgr2=0x7fcca402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:39:55.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.090+0000 7fcca37fe640 1 --2- 172.21.15.12:0/1620341067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca41243b0 0x7fcca402d980 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcc94000c00 tx=0x7fcc94009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:39:55.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.090+0000 7fcc8bfff640 1 -- 172.21.15.12:0/1620341067 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc94000e60 con 0x7fcca41243b0 2024-09-11T01:39:55.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.090+0000 7fcc8bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:39:55.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.090+0000 7fcc8bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:39:55.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.090+0000 7fcc8bfff640 10 monclient: dump: 2024-09-11T01:39:55.099 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:39:55.099 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:39:55.099 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:55.099 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:39:55.100 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:39:55.100 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:39:55.100 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:39:55.100 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:39:55.100 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:39:55.100 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:39:55.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.090+0000 7fcc8bfff640 1 -- 172.21.15.12:0/1620341067 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4864 (secure 0 0 0) 0x7fcc94003a20 con 0x7fcca41243b0 2024-09-11T01:39:55.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.090+0000 7fcc8bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:39:55.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.090+0000 7fcc8bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:39:55.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.090+0000 7fcc8bfff640 1 -- 172.21.15.12:0/1620341067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca41243b0 msgr2=0x7fcca402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:55.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.090+0000 7fcc8bfff640 1 --2- 172.21.15.12:0/1620341067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca41243b0 0x7fcca402d980 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcc94000c00 tx=0x7fcc94009190 comp rx=0 tx=0).stop 2024-09-11T01:39:55.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.090+0000 7fcc8bfff640 1 -- 172.21.15.12:0/1620341067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca41243b0 msgr2=0x7fcca402d980 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:39:55.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.090+0000 7fcc8bfff640 1 --2- 172.21.15.12:0/1620341067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca41243b0 0x7fcca402d980 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:39:55.101 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:39:55.101 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:39:55.101 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:39:55.101 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:39:55.101 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 104, 2024-09-11T01:39:55.101 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 40887, 2024-09-11T01:39:55.101 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:39:55.101 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:39:55.101 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:39:55.101 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:55.101 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:39:55.102 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:39:55.102 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:39:55.102 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:39:55.102 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:39:55.102 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:39:55.102 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:39:55.102 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:39:55.102 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:39:55.102 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:39:55.102 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:39:55.102 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:39:55.102 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:39:55.102 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:39:55.102 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:39:55.103 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:55.103 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:39:55.103 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:39:55.103 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:39:55.103 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:39:55.103 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:39:55.103 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:39:55.103 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:39:55.103 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:39:55.103 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:39:55.103 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:39:55.103 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:39:55.103 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:39:55.103 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:39:55.103 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:55.104 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:55.104 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:39:55.104 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:39:55.104 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:39:55.104 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:39:55.104 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:39:55.104 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:39:55.104 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:39:55.104 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:39:55.104 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:39:55.104 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:39:55.104 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:39:55.104 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:39:55.104 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:39:55.105 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:39:55.105 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:39:55.105 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:39:55.105 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:39:55.105 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:39:55.105 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:39:55.105 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:39:55.105 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:39:55.105 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:39:55.105 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:39:55.105 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:39:55.105 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:39:55.105 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:39:55.105 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:39:55.105 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:39:55.106 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:55.106 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:39:55.106 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:55.106 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:39:55.106 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:55.106 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:39:55.106 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:39:55.106 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:39:55.106 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:39:55.106 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:55.106 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:39:55.106 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:39:55.106 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:55.106 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:55.106 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:55.106 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:39:55.107 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:39:55.107 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:55.107 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:55.107 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:55.107 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:55.107 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:39:55.107 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:39:55.107 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:39:55.107 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:39:55.107 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:39:55.107 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:55.107 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:55.107 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:39:55.107 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:39:55.107 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:39:55.108 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:39:55.108 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:55.108 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:39:55.108 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:39:55.108 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:55.108 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:55.108 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:55.108 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:39:55.108 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:39:55.108 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:55.108 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:55.108 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:55.108 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:55.108 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:39:55.108 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:39:55.109 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:39:55.109 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:39:55.109 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:39:55.109 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:55.109 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:55.109 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:39:55.109 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:39:55.109 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:39:55.109 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:39:55.109 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:55.109 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:39:55.109 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:39:55.109 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:55.109 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:55.109 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:55.110 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:39:55.110 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:39:55.110 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:39:55.110 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:55.110 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:55.110 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:55.110 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:39:55.110 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:39:55.110 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:39:55.110 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:39:55.110 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:39:55.110 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:55.110 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:55.110 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:55.110 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:39:55.110 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:39:55.111 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:55.111 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:39:55.111 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:39:55.111 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:39:55.111 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:55.111 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:55.111 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:39:55.111 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:55.111 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:39:55.111 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:39:55.111 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:39:55.111 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:55.111 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:55.111 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:39:55.111 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:55.112 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:39:55.112 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:39:55.112 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:39:55.112 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:55.112 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:39:55.112 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:39:55.112 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:39:55.112 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:39:55.112 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:39:55.112 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:39:55.112 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:39:55.112 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:39:55.112 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:39:55.112 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:39:55.112 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:39:55.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.091+0000 7fcca9f17640 1 -- 172.21.15.12:0/1620341067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcc9003e240 msgr2=0x7fcc90040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:55.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.091+0000 7fcca9f17640 1 --2- 172.21.15.12:0/1620341067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcc9003e240 0x7fcc90040700 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcca40a1120 tx=0x7fcc9c0095f0 comp rx=0 tx=0).stop 2024-09-11T01:39:55.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.091+0000 7fcca9f17640 10 monclient: shutdown 2024-09-11T01:39:55.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.091+0000 7fcca9f17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:39:55.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.091+0000 7fcca9f17640 1 -- 172.21.15.12:0/1620341067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca40a0c90 msgr2=0x7fcca4135c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:39:55.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.091+0000 7fcca9f17640 1 --2- 172.21.15.12:0/1620341067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcca40a0c90 0x7fcca4135c30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcc9800c460 tx=0x7fcc9800c920 comp rx=0 tx=0).stop 2024-09-11T01:39:55.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.091+0000 7fcca3fff640 1 -- 172.21.15.12:0/1620341067 reap_dead start 2024-09-11T01:39:55.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.091+0000 7fcca3fff640 1 -- 172.21.15.12:0/1620341067 reap_dead start 2024-09-11T01:39:55.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.091+0000 7fcca9f17640 1 -- 172.21.15.12:0/1620341067 shutdown_connections 2024-09-11T01:39:55.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.091+0000 7fcca9f17640 1 -- 172.21.15.12:0/1620341067 >> 172.21.15.12:0/1620341067 conn(0x7fcca40a2f70 msgr2=0x7fcca4071d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:39:55.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.091+0000 7fcca9f17640 1 -- 172.21.15.12:0/1620341067 shutdown_connections 2024-09-11T01:39:55.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:39:55.091+0000 7fcca9f17640 1 -- 172.21.15.12:0/1620341067 wait complete. 2024-09-11T01:39:55.115 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:39:55.115 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:39:55.172 INFO:tasks.rados.rados.0.smithi071.stdout:1533: done (3 left) 2024-09-11T01:39:55.172 INFO:tasks.rados.rados.0.smithi071.stdout:1534: done (2 left) 2024-09-11T01:39:55.172 INFO:tasks.rados.rados.0.smithi071.stdout:1535: done (1 left) 2024-09-11T01:39:55.172 INFO:tasks.rados.rados.0.smithi071.stdout:1536: done (0 left) 2024-09-11T01:39:55.172 INFO:tasks.rados.rados.0.smithi071.stdout:1537: copy_from oid 45 from oid 49 current snap is 166 2024-09-11T01:39:55.172 INFO:tasks.rados.rados.0.smithi071.stdout:1538: snap_remove snap 157 2024-09-11T01:39:56.173 INFO:tasks.rados.rados.0.smithi071.stdout:1538: done (1 left) 2024-09-11T01:39:56.173 INFO:tasks.rados.rados.0.smithi071.stdout:1539: read oid 24 snap 150 2024-09-11T01:39:56.174 INFO:tasks.rados.rados.0.smithi071.stdout:1539: expect (ObjNum 430 snap 119 seq_num 430) 2024-09-11T01:39:56.174 INFO:tasks.rados.rados.0.smithi071.stdout:1540: copy_from oid 11 from oid 37 current snap is 166 2024-09-11T01:39:56.174 INFO:tasks.rados.rados.0.smithi071.stdout:1537: finishing copy_from to smithi07134777-45 2024-09-11T01:39:56.174 INFO:tasks.rados.rados.0.smithi071.stdout:1537: got expected ENOENT (src dne) 2024-09-11T01:39:56.174 INFO:tasks.rados.rados.0.smithi071.stdout:1537: finishing copy_from racing read to smithi07134777-45 2024-09-11T01:39:56.174 INFO:tasks.rados.rados.0.smithi071.stdout:1537: done (2 left) 2024-09-11T01:39:56.174 INFO:tasks.rados.rados.0.smithi071.stdout:1541: snap_remove snap 158 2024-09-11T01:39:57.175 INFO:tasks.rados.rados.0.smithi071.stdout:1541: done (2 left) 2024-09-11T01:39:57.175 INFO:tasks.rados.rados.0.smithi071.stdout:1542: write (excl) oid 32 current snap is 166 2024-09-11T01:39:57.175 INFO:tasks.rados.rados.0.smithi071.stdout:1542: seq_num 578 ranges {514446=736750,1723298=667869,3073078=641368} 2024-09-11T01:39:57.181 INFO:tasks.rados.rados.0.smithi071.stdout:1542: writing smithi07134777-32 from 514446 to 1251196 tid 1 2024-09-11T01:39:57.187 INFO:tasks.rados.rados.0.smithi071.stdout:1542: writing smithi07134777-32 from 1723298 to 2391167 tid 2 2024-09-11T01:39:57.195 INFO:tasks.rados.rados.0.smithi071.stdout:1542: writing smithi07134777-32 from 3073078 to 3714446 tid 3 2024-09-11T01:39:57.195 INFO:tasks.rados.rados.0.smithi071.stdout:1543: write oid 39 current snap is 166 2024-09-11T01:39:57.195 INFO:tasks.rados.rados.0.smithi071.stdout:1543: seq_num 579 ranges {531253=411999,1381780=411527,2300501=586966,3376974=354279} 2024-09-11T01:39:57.199 INFO:tasks.rados.rados.0.smithi071.stdout:1543: writing smithi07134777-39 from 531253 to 943252 tid 1 2024-09-11T01:39:57.204 INFO:tasks.rados.rados.0.smithi071.stdout:1543: writing smithi07134777-39 from 1381780 to 1793307 tid 2 2024-09-11T01:39:57.210 INFO:tasks.rados.rados.0.smithi071.stdout:1543: writing smithi07134777-39 from 2300501 to 2887467 tid 3 2024-09-11T01:39:57.215 INFO:tasks.rados.rados.0.smithi071.stdout:1543: writing smithi07134777-39 from 3376974 to 3731253 tid 4 2024-09-11T01:39:57.215 INFO:tasks.rados.rados.0.smithi071.stdout:1544: copy_from oid 34 from oid 14 current snap is 166 2024-09-11T01:39:57.215 INFO:tasks.rados.rados.0.smithi071.stdout:1539: expect (ObjNum 430 snap 119 seq_num 430) 2024-09-11T01:39:57.236 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1928066 2024-09-11T01:39:57.369 INFO:tasks.rados.rados.0.smithi071.stdout:1540: finishing copy_from racing read to smithi07134777-11 2024-09-11T01:39:57.370 INFO:tasks.rados.rados.0.smithi071.stdout:1540: finishing copy_from to smithi07134777-11 2024-09-11T01:39:57.370 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 604 (ObjNum 577 snap 165 seq_num 577) dirty exists 2024-09-11T01:39:57.370 INFO:tasks.rados.rados.0.smithi071.stdout:1542: finishing write tid 1 to smithi07134777-32 2024-09-11T01:39:57.370 INFO:tasks.rados.rados.0.smithi071.stdout:1542: finishing write tid 2 to smithi07134777-32 2024-09-11T01:39:57.370 INFO:tasks.rados.rados.0.smithi071.stdout:1542: finishing write tid 3 to smithi07134777-32 2024-09-11T01:39:57.370 INFO:tasks.rados.rados.0.smithi071.stdout:1542: finishing write tid 4 to smithi07134777-32 2024-09-11T01:39:57.370 INFO:tasks.rados.rados.0.smithi071.stdout:1542: finishing write tid 5 to smithi07134777-32 2024-09-11T01:39:57.370 INFO:tasks.rados.rados.0.smithi071.stdout:1542: oid 32 updating version 0 to 607 2024-09-11T01:39:57.370 INFO:tasks.rados.rados.0.smithi071.stdout:1542: oid 32 version 607 is already newer than 606 2024-09-11T01:39:57.370 INFO:tasks.rados.rados.0.smithi071.stdout:1542: oid 32 updating version 607 to 608 2024-09-11T01:39:57.370 INFO:tasks.rados.rados.0.smithi071.stdout:1542: oid 32 version 608 is already newer than 605 2024-09-11T01:39:57.370 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 608 (ObjNum 578 snap 166 seq_num 578) dirty exists 2024-09-11T01:39:57.370 INFO:tasks.rados.rados.0.smithi071.stdout:1542: left oid 32 (ObjNum 578 snap 166 seq_num 578) 2024-09-11T01:39:57.370 INFO:tasks.rados.rados.0.smithi071.stdout:1543: finishing write tid 1 to smithi07134777-39 2024-09-11T01:39:57.371 INFO:tasks.rados.rados.0.smithi071.stdout:1543: finishing write tid 2 to smithi07134777-39 2024-09-11T01:39:57.371 INFO:tasks.rados.rados.0.smithi071.stdout:1543: finishing write tid 3 to smithi07134777-39 2024-09-11T01:39:57.371 INFO:tasks.rados.rados.0.smithi071.stdout:1543: finishing write tid 4 to smithi07134777-39 2024-09-11T01:39:57.371 INFO:tasks.rados.rados.0.smithi071.stdout:1543: finishing write tid 5 to smithi07134777-39 2024-09-11T01:39:57.371 INFO:tasks.rados.rados.0.smithi071.stdout:1543: finishing write tid 6 to smithi07134777-39 2024-09-11T01:39:57.371 INFO:tasks.rados.rados.0.smithi071.stdout:1543: oid 39 updating version 0 to 610 2024-09-11T01:39:57.371 INFO:tasks.rados.rados.0.smithi071.stdout:1543: oid 39 updating version 610 to 613 2024-09-11T01:39:57.371 INFO:tasks.rados.rados.0.smithi071.stdout:1543: oid 39 version 613 is already newer than 612 2024-09-11T01:39:57.371 INFO:tasks.rados.rados.0.smithi071.stdout:1543: oid 39 version 613 is already newer than 609 2024-09-11T01:39:57.371 INFO:tasks.rados.rados.0.smithi071.stdout:1543: oid 39 version 613 is already newer than 611 2024-09-11T01:39:57.371 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 613 (ObjNum 579 snap 166 seq_num 579) dirty exists 2024-09-11T01:39:57.371 INFO:tasks.rados.rados.0.smithi071.stdout:1543: left oid 39 (ObjNum 579 snap 166 seq_num 579) 2024-09-11T01:39:57.371 INFO:tasks.rados.rados.0.smithi071.stdout:1539: done (4 left) 2024-09-11T01:39:57.371 INFO:tasks.rados.rados.0.smithi071.stdout:1540: done (3 left) 2024-09-11T01:39:57.371 INFO:tasks.rados.rados.0.smithi071.stdout:1542: done (2 left) 2024-09-11T01:39:57.372 INFO:tasks.rados.rados.0.smithi071.stdout:1543: done (1 left) 2024-09-11T01:39:57.372 INFO:tasks.rados.rados.0.smithi071.stdout:1545: delete oid 41 current snap is 166 2024-09-11T01:39:57.383 INFO:tasks.rados.rados.0.smithi071.stdout:1545: done (1 left) 2024-09-11T01:39:57.383 INFO:tasks.rados.rados.0.smithi071.stdout:1546: snap_remove snap 163 2024-09-11T01:39:58.192 INFO:tasks.rados.rados.0.smithi071.stdout:1546: done (1 left) 2024-09-11T01:39:58.192 INFO:tasks.rados.rados.0.smithi071.stdout:1547: read oid 15 snap -1 2024-09-11T01:39:58.192 INFO:tasks.rados.rados.0.smithi071.stdout:1547: expect (ObjNum 565 snap 160 seq_num 565) 2024-09-11T01:39:58.192 INFO:tasks.rados.rados.0.smithi071.stdout:1548: read oid 2 snap -1 2024-09-11T01:39:58.192 INFO:tasks.rados.rados.0.smithi071.stdout:1548: expect (ObjNum 503 snap 135 seq_num 503) 2024-09-11T01:39:58.192 INFO:tasks.rados.rados.0.smithi071.stdout:1544: finishing copy_from racing read to smithi07134777-34 2024-09-11T01:39:58.192 INFO:tasks.rados.rados.0.smithi071.stdout:1544: finishing copy_from to smithi07134777-34 2024-09-11T01:39:58.192 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 831 (ObjNum 566 snap 160 seq_num 566) dirty exists 2024-09-11T01:39:58.192 INFO:tasks.rados.rados.0.smithi071.stdout:1549: read oid 42 snap 165 2024-09-11T01:39:58.192 INFO:tasks.rados.rados.0.smithi071.stdout:1549: expect (ObjNum 488 snap 132 seq_num 488) 2024-09-11T01:39:58.193 INFO:tasks.rados.rados.0.smithi071.stdout:1544: done (3 left) 2024-09-11T01:39:58.193 INFO:tasks.rados.rados.0.smithi071.stdout:1550: write (excl) oid 7 current snap is 166 2024-09-11T01:39:58.193 INFO:tasks.rados.rados.0.smithi071.stdout:1550: seq_num 581 ranges {564867=562497,1898744=415196,2755347=725863} 2024-09-11T01:39:58.198 INFO:tasks.rados.rados.0.smithi071.stdout:1550: writing smithi07134777-7 from 564867 to 1127364 tid 1 2024-09-11T01:39:58.205 INFO:tasks.rados.rados.0.smithi071.stdout:1550: writing smithi07134777-7 from 1898744 to 2313940 tid 2 2024-09-11T01:39:58.212 INFO:tasks.rados.rados.0.smithi071.stdout:1550: writing smithi07134777-7 from 2755347 to 3481210 tid 3 2024-09-11T01:39:58.212 INFO:tasks.rados.rados.0.smithi071.stdout:1551: rollback oid 45 current snap is 166 2024-09-11T01:39:58.212 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 45 to 165 2024-09-11T01:39:58.212 INFO:tasks.rados.rados.0.smithi071.stdout:1548: expect (ObjNum 503 snap 135 seq_num 503) 2024-09-11T01:39:58.221 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1690397 2024-09-11T01:39:58.255 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:39:58.280 INFO:tasks.rados.rados.0.smithi071.stdout:1549: expect (ObjNum 488 snap 132 seq_num 488) 2024-09-11T01:39:58.280 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:39:58.323 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:39:58.365 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 179650 2024-09-11T01:39:58.406 INFO:tasks.rados.rados.0.smithi071.stdout:1547: expect (ObjNum 565 snap 160 seq_num 565) 2024-09-11T01:39:58.553 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:39:58.634 INFO:tasks.rados.rados.0.smithi071.stdout:1550: finishing write tid 1 to smithi07134777-7 2024-09-11T01:39:58.634 INFO:tasks.rados.rados.0.smithi071.stdout:1550: finishing write tid 2 to smithi07134777-7 2024-09-11T01:39:58.634 INFO:tasks.rados.rados.0.smithi071.stdout:1550: finishing write tid 3 to smithi07134777-7 2024-09-11T01:39:58.634 INFO:tasks.rados.rados.0.smithi071.stdout:1550: finishing write tid 4 to smithi07134777-7 2024-09-11T01:39:58.634 INFO:tasks.rados.rados.0.smithi071.stdout:1551: finishing rollback tid 0 to smithi07134777-45 2024-09-11T01:39:58.634 INFO:tasks.rados.rados.0.smithi071.stdout:1550: finishing write tid 5 to smithi07134777-7 2024-09-11T01:39:58.634 INFO:tasks.rados.rados.0.smithi071.stdout:1550: oid 7 updating version 0 to 864 2024-09-11T01:39:58.634 INFO:tasks.rados.rados.0.smithi071.stdout:1550: oid 7 updating version 864 to 865 2024-09-11T01:39:58.635 INFO:tasks.rados.rados.0.smithi071.stdout:1550: oid 7 version 865 is already newer than 863 2024-09-11T01:39:58.635 INFO:tasks.rados.rados.0.smithi071.stdout:1550: oid 7 updating version 865 to 866 2024-09-11T01:39:58.635 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 866 (ObjNum 581 snap 166 seq_num 581) dirty exists 2024-09-11T01:39:58.635 INFO:tasks.rados.rados.0.smithi071.stdout:1550: left oid 7 (ObjNum 581 snap 166 seq_num 581) 2024-09-11T01:39:58.635 INFO:tasks.rados.rados.0.smithi071.stdout:1551: finishing rollback tid 1 to smithi07134777-45 2024-09-11T01:39:58.635 INFO:tasks.rados.rados.0.smithi071.stdout:1547: done (4 left) 2024-09-11T01:39:58.635 INFO:tasks.rados.rados.0.smithi071.stdout:1548: done (3 left) 2024-09-11T01:39:58.635 INFO:tasks.rados.rados.0.smithi071.stdout:1549: done (2 left) 2024-09-11T01:39:58.635 INFO:tasks.rados.rados.0.smithi071.stdout:1550: done (1 left) 2024-09-11T01:39:58.635 INFO:tasks.rados.rados.0.smithi071.stdout:1552: snap_remove snap 150 2024-09-11T01:39:58.939 INFO:tasks.rados.rados.0.smithi071.stdout:1552: done (1 left) 2024-09-11T01:39:58.939 INFO:tasks.rados.rados.0.smithi071.stdout:1553: snap_remove snap 162 2024-09-11T01:39:59.945 INFO:tasks.rados.rados.0.smithi071.stdout:1553: done (1 left) 2024-09-11T01:39:59.945 INFO:tasks.rados.rados.0.smithi071.stdout:1554: read oid 24 snap -1 2024-09-11T01:39:59.946 INFO:tasks.rados.rados.0.smithi071.stdout:1554: expect (ObjNum 430 snap 119 seq_num 430) 2024-09-11T01:39:59.946 INFO:tasks.rados.rados.0.smithi071.stdout:1555: read oid 34 snap -1 2024-09-11T01:39:59.946 INFO:tasks.rados.rados.0.smithi071.stdout:1555: expect (ObjNum 566 snap 160 seq_num 566) 2024-09-11T01:39:59.946 INFO:tasks.rados.rados.0.smithi071.stdout:1556: read oid 9 snap -1 2024-09-11T01:39:59.946 INFO:tasks.rados.rados.0.smithi071.stdout:1556: expect deleted 2024-09-11T01:39:59.946 INFO:tasks.rados.rados.0.smithi071.stdout:1557: read oid 33 snap 165 2024-09-11T01:39:59.946 INFO:tasks.rados.rados.0.smithi071.stdout:1557: expect (ObjNum 250 snap 63 seq_num 250) 2024-09-11T01:39:59.946 INFO:tasks.rados.rados.0.smithi071.stdout:1558: read oid 31 snap -1 2024-09-11T01:39:59.946 INFO:tasks.rados.rados.0.smithi071.stdout:1558: expect (ObjNum 509 snap 139 seq_num 509) 2024-09-11T01:39:59.946 INFO:tasks.rados.rados.0.smithi071.stdout:1559: read oid 36 snap 156 2024-09-11T01:39:59.946 INFO:tasks.rados.rados.0.smithi071.stdout:1559: expect (ObjNum 469 snap 124 seq_num 469) 2024-09-11T01:39:59.946 INFO:tasks.rados.rados.0.smithi071.stdout:1560: snap_create 2024-09-11T01:39:59.946 INFO:tasks.rados.rados.0.smithi071.stdout:1551: finishing rollback tid 2 to smithi07134777-45 2024-09-11T01:39:59.946 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 593 (ObjNum 437 snap 119 seq_num 437) dirty exists 2024-09-11T01:39:59.949 INFO:tasks.rados.rados.0.smithi071.stdout:1558: expect (ObjNum 509 snap 139 seq_num 509) 2024-09-11T01:39:59.960 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1270133 2024-09-11T01:40:00.040 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:40:00.085 INFO:tasks.rados.rados.0.smithi071.stdout:1554: expect (ObjNum 430 snap 119 seq_num 430) 2024-09-11T01:40:00.138 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 445939 2024-09-11T01:40:00.189 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:40:00.245 INFO:tasks.rados.rados.0.smithi071.stdout:1555: expect (ObjNum 566 snap 160 seq_num 566) 2024-09-11T01:40:00.303 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:40:00.394 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2548413 2024-09-11T01:40:00.416 INFO:tasks.rados.rados.0.smithi071.stdout:1557: expect (ObjNum 250 snap 63 seq_num 250) 2024-09-11T01:40:00.533 INFO:tasks.rados.rados.0.smithi071.stdout:1559: expect (ObjNum 469 snap 124 seq_num 469) 2024-09-11T01:40:00.533 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:40:00.649 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:40:00.947 INFO:tasks.rados.rados.0.smithi071.stdout:1551: done (7 left) 2024-09-11T01:40:00.948 INFO:tasks.rados.rados.0.smithi071.stdout:1554: done (6 left) 2024-09-11T01:40:00.948 INFO:tasks.rados.rados.0.smithi071.stdout:1555: done (5 left) 2024-09-11T01:40:00.948 INFO:tasks.rados.rados.0.smithi071.stdout:1556: done (4 left) 2024-09-11T01:40:00.948 INFO:tasks.rados.rados.0.smithi071.stdout:1557: done (3 left) 2024-09-11T01:40:00.948 INFO:tasks.rados.rados.0.smithi071.stdout:1558: done (2 left) 2024-09-11T01:40:00.948 INFO:tasks.rados.rados.0.smithi071.stdout:1559: done (1 left) 2024-09-11T01:40:00.948 INFO:tasks.rados.rados.0.smithi071.stdout:1560: done (0 left) 2024-09-11T01:40:00.948 INFO:tasks.rados.rados.0.smithi071.stdout:1561: snap_remove snap 156 2024-09-11T01:40:01.957 INFO:tasks.rados.rados.0.smithi071.stdout:1561: done (0 left) 2024-09-11T01:40:01.957 INFO:tasks.rados.rados.0.smithi071.stdout:1562: copy_from oid 40 from oid 49 current snap is 167 2024-09-11T01:40:01.957 INFO:tasks.rados.rados.0.smithi071.stdout:1563: rollback oid 16 current snap is 167 2024-09-11T01:40:01.957 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 16 to 166 2024-09-11T01:40:01.958 INFO:tasks.rados.rados.0.smithi071.stdout:1564: copy_from oid 43 from oid 30 current snap is 167 2024-09-11T01:40:01.958 INFO:tasks.rados.rados.0.smithi071.stdout:1565: snap_remove snap 165 2024-09-11T01:40:02.960 INFO:tasks.rados.rados.0.smithi071.stdout:1565: done (3 left) 2024-09-11T01:40:02.960 INFO:tasks.rados.rados.0.smithi071.stdout:1566: copy_from oid 13 from oid 50 current snap is 167 2024-09-11T01:40:02.960 INFO:tasks.rados.rados.0.smithi071.stdout:1562: finishing copy_from to smithi07134777-40 2024-09-11T01:40:02.960 INFO:tasks.rados.rados.0.smithi071.stdout:1562: got expected ENOENT (src dne) 2024-09-11T01:40:02.960 INFO:tasks.rados.rados.0.smithi071.stdout:1562: finishing copy_from racing read to smithi07134777-40 2024-09-11T01:40:02.960 INFO:tasks.rados.rados.0.smithi071.stdout:1563: finishing rollback tid 0 to smithi07134777-16 2024-09-11T01:40:02.960 INFO:tasks.rados.rados.0.smithi071.stdout:1563: finishing rollback tid 1 to smithi07134777-16 2024-09-11T01:40:02.960 INFO:tasks.rados.rados.0.smithi071.stdout:1563: finishing rollback tid 2 to smithi07134777-16 2024-09-11T01:40:02.961 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 603 (ObjNum 411 snap 115 seq_num 411) dirty exists 2024-09-11T01:40:02.961 INFO:tasks.rados.rados.0.smithi071.stdout:1564: finishing copy_from to smithi07134777-43 2024-09-11T01:40:02.965 INFO:tasks.rados.rados.0.smithi071.stdout:1564: got expected ENOENT (src dne) 2024-09-11T01:40:02.965 INFO:tasks.rados.rados.0.smithi071.stdout:1564: finishing copy_from racing read to smithi07134777-43 2024-09-11T01:40:02.965 INFO:tasks.rados.rados.0.smithi071.stdout:1562: done (3 left) 2024-09-11T01:40:02.966 INFO:tasks.rados.rados.0.smithi071.stdout:1563: done (2 left) 2024-09-11T01:40:02.966 INFO:tasks.rados.rados.0.smithi071.stdout:1564: done (1 left) 2024-09-11T01:40:02.966 INFO:tasks.rados.rados.0.smithi071.stdout:1567: write oid 38 current snap is 167 2024-09-11T01:40:02.966 INFO:tasks.rados.rados.0.smithi071.stdout:1567: seq_num 582 ranges {581674=637746,1840873=558854,3066417=537135} 2024-09-11T01:40:02.966 INFO:tasks.rados.rados.0.smithi071.stdout:1567: writing smithi07134777-38 from 581674 to 1219420 tid 1 2024-09-11T01:40:02.973 INFO:tasks.rados.rados.0.smithi071.stdout:1567: writing smithi07134777-38 from 1840873 to 2399727 tid 2 2024-09-11T01:40:02.980 INFO:tasks.rados.rados.0.smithi071.stdout:1567: writing smithi07134777-38 from 3066417 to 3603552 tid 3 2024-09-11T01:40:02.980 INFO:tasks.rados.rados.0.smithi071.stdout:1568: write oid 2 current snap is 167 2024-09-11T01:40:02.981 INFO:tasks.rados.rados.0.smithi071.stdout:1568: seq_num 583 ranges {598481=712995,1899355=418865,3093840=704641} 2024-09-11T01:40:02.987 INFO:tasks.rados.rados.0.smithi071.stdout:1568: writing smithi07134777-2 from 598481 to 1311476 tid 1 2024-09-11T01:40:02.993 INFO:tasks.rados.rados.0.smithi071.stdout:1568: writing smithi07134777-2 from 1899355 to 2318220 tid 2 2024-09-11T01:40:03.002 INFO:tasks.rados.rados.0.smithi071.stdout:1568: writing smithi07134777-2 from 3093840 to 3798481 tid 3 2024-09-11T01:40:03.002 INFO:tasks.rados.rados.0.smithi071.stdout:1569: read oid 41 snap -1 2024-09-11T01:40:03.002 INFO:tasks.rados.rados.0.smithi071.stdout:1569: expect deleted 2024-09-11T01:40:03.002 INFO:tasks.rados.rados.0.smithi071.stdout:1570: delete oid 6 current snap is 167 2024-09-11T01:40:03.002 INFO:tasks.rados.rados.0.smithi071.stdout:1567: finishing write tid 1 to smithi07134777-38 2024-09-11T01:40:03.002 INFO:tasks.rados.rados.0.smithi071.stdout:1566: finishing copy_from racing read to smithi07134777-13 2024-09-11T01:40:03.002 INFO:tasks.rados.rados.0.smithi071.stdout:1566: finishing copy_from to smithi07134777-13 2024-09-11T01:40:03.003 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 784 (ObjNum 551 snap 157 seq_num 551) dirty exists 2024-09-11T01:40:03.003 INFO:tasks.rados.rados.0.smithi071.stdout:1567: finishing write tid 2 to smithi07134777-38 2024-09-11T01:40:03.003 INFO:tasks.rados.rados.0.smithi071.stdout:1567: finishing write tid 3 to smithi07134777-38 2024-09-11T01:40:03.003 INFO:tasks.rados.rados.0.smithi071.stdout:1567: finishing write tid 4 to smithi07134777-38 2024-09-11T01:40:03.003 INFO:tasks.rados.rados.0.smithi071.stdout:1567: finishing write tid 5 to smithi07134777-38 2024-09-11T01:40:03.003 INFO:tasks.rados.rados.0.smithi071.stdout:1567: oid 38 updating version 0 to 891 2024-09-11T01:40:03.003 INFO:tasks.rados.rados.0.smithi071.stdout:1567: oid 38 version 891 is already newer than 888 2024-09-11T01:40:03.003 INFO:tasks.rados.rados.0.smithi071.stdout:1567: oid 38 version 891 is already newer than 890 2024-09-11T01:40:03.003 INFO:tasks.rados.rados.0.smithi071.stdout:1567: oid 38 version 891 is already newer than 889 2024-09-11T01:40:03.003 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 891 (ObjNum 582 snap 167 seq_num 582) dirty exists 2024-09-11T01:40:03.003 INFO:tasks.rados.rados.0.smithi071.stdout:1567: left oid 38 (ObjNum 582 snap 167 seq_num 582) 2024-09-11T01:40:03.003 INFO:tasks.rados.rados.0.smithi071.stdout:1568: finishing write tid 1 to smithi07134777-2 2024-09-11T01:40:03.003 INFO:tasks.rados.rados.0.smithi071.stdout:1568: finishing write tid 2 to smithi07134777-2 2024-09-11T01:40:03.004 INFO:tasks.rados.rados.0.smithi071.stdout:1566: done (4 left) 2024-09-11T01:40:03.004 INFO:tasks.rados.rados.0.smithi071.stdout:1567: done (3 left) 2024-09-11T01:40:03.004 INFO:tasks.rados.rados.0.smithi071.stdout:1569: done (2 left) 2024-09-11T01:40:03.004 INFO:tasks.rados.rados.0.smithi071.stdout:1570: done (1 left) 2024-09-11T01:40:03.004 INFO:tasks.rados.rados.0.smithi071.stdout:1571: snap_create 2024-09-11T01:40:03.005 INFO:tasks.rados.rados.0.smithi071.stdout:1568: finishing write tid 3 to smithi07134777-2 2024-09-11T01:40:03.007 INFO:tasks.rados.rados.0.smithi071.stdout:1568: finishing write tid 4 to smithi07134777-2 2024-09-11T01:40:03.007 INFO:tasks.rados.rados.0.smithi071.stdout:1568: finishing write tid 5 to smithi07134777-2 2024-09-11T01:40:03.007 INFO:tasks.rados.rados.0.smithi071.stdout:1568: oid 2 updating version 0 to 444 2024-09-11T01:40:03.007 INFO:tasks.rados.rados.0.smithi071.stdout:1568: oid 2 updating version 444 to 447 2024-09-11T01:40:03.007 INFO:tasks.rados.rados.0.smithi071.stdout:1568: oid 2 version 447 is already newer than 446 2024-09-11T01:40:03.007 INFO:tasks.rados.rados.0.smithi071.stdout:1568: oid 2 version 447 is already newer than 445 2024-09-11T01:40:03.007 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 447 (ObjNum 583 snap 167 seq_num 583) dirty exists 2024-09-11T01:40:03.007 INFO:tasks.rados.rados.0.smithi071.stdout:1568: left oid 2 (ObjNum 583 snap 167 seq_num 583) 2024-09-11T01:40:03.946 INFO:tasks.rados.rados.0.smithi071.stdout:1568: done (1 left) 2024-09-11T01:40:03.946 INFO:tasks.rados.rados.0.smithi071.stdout:1571: done (0 left) 2024-09-11T01:40:03.947 INFO:tasks.rados.rados.0.smithi071.stdout:1572: write oid 12 current snap is 168 2024-09-11T01:40:03.947 INFO:tasks.rados.rados.0.smithi071.stdout:1572: seq_num 585 ranges {632095=463493,1616319=422534,2748686=487304} 2024-09-11T01:40:03.952 INFO:tasks.rados.rados.0.smithi071.stdout:1572: writing smithi07134777-12 from 632095 to 1095588 tid 1 2024-09-11T01:40:03.957 INFO:tasks.rados.rados.0.smithi071.stdout:1572: writing smithi07134777-12 from 1616319 to 2038853 tid 2 2024-09-11T01:40:03.964 INFO:tasks.rados.rados.0.smithi071.stdout:1572: writing smithi07134777-12 from 2748686 to 3235990 tid 3 2024-09-11T01:40:03.964 INFO:tasks.rados.rados.0.smithi071.stdout:1573: copy_from oid 24 from oid 25 current snap is 168 2024-09-11T01:40:03.964 INFO:tasks.rados.rados.0.smithi071.stdout:1574: rollback oid 36 current snap is 168 2024-09-11T01:40:03.964 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 36 to 167 2024-09-11T01:40:03.964 INFO:tasks.rados.rados.0.smithi071.stdout:1575: write oid 48 current snap is 168 2024-09-11T01:40:03.964 INFO:tasks.rados.rados.0.smithi071.stdout:1575: seq_num 586 ranges {648902=655095,2074801=566192,3176109=672793} 2024-09-11T01:40:03.970 INFO:tasks.rados.rados.0.smithi071.stdout:1575: writing smithi07134777-48 from 648902 to 1303997 tid 1 2024-09-11T01:40:03.978 INFO:tasks.rados.rados.0.smithi071.stdout:1575: writing smithi07134777-48 from 2074801 to 2640993 tid 2 2024-09-11T01:40:03.985 INFO:tasks.rados.rados.0.smithi071.stdout:1575: writing smithi07134777-48 from 3176109 to 3848902 tid 3 2024-09-11T01:40:03.985 INFO:tasks.rados.rados.0.smithi071.stdout:1576: write (excl) oid 5 current snap is 168 2024-09-11T01:40:03.985 INFO:tasks.rados.rados.0.smithi071.stdout:1576: seq_num 587 ranges {665709=730344,2133283=426203,3203532=509848} 2024-09-11T01:40:03.993 INFO:tasks.rados.rados.0.smithi071.stdout:1576: writing smithi07134777-5 from 665709 to 1396053 tid 1 2024-09-11T01:40:03.999 INFO:tasks.rados.rados.0.smithi071.stdout:1576: writing smithi07134777-5 from 2133283 to 2559486 tid 2 2024-09-11T01:40:04.006 INFO:tasks.rados.rados.0.smithi071.stdout:1576: writing smithi07134777-5 from 3203532 to 3713380 tid 3 2024-09-11T01:40:04.006 INFO:tasks.rados.rados.0.smithi071.stdout:1577: rollback oid 33 current snap is 168 2024-09-11T01:40:04.006 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 33 to 166 2024-09-11T01:40:04.007 INFO:tasks.rados.rados.0.smithi071.stdout:1572: finishing write tid 1 to smithi07134777-12 2024-09-11T01:40:04.007 INFO:tasks.rados.rados.0.smithi071.stdout:1572: finishing write tid 2 to smithi07134777-12 2024-09-11T01:40:04.007 INFO:tasks.rados.rados.0.smithi071.stdout:1572: finishing write tid 3 to smithi07134777-12 2024-09-11T01:40:04.007 INFO:tasks.rados.rados.0.smithi071.stdout:1574: finishing rollback tid 0 to smithi07134777-36 2024-09-11T01:40:04.007 INFO:tasks.rados.rados.0.smithi071.stdout:1573: finishing copy_from racing read to smithi07134777-24 2024-09-11T01:40:04.007 INFO:tasks.rados.rados.0.smithi071.stdout:1572: finishing write tid 4 to smithi07134777-12 2024-09-11T01:40:04.007 INFO:tasks.rados.rados.0.smithi071.stdout:1574: finishing rollback tid 1 to smithi07134777-36 2024-09-11T01:40:04.007 INFO:tasks.rados.rados.0.smithi071.stdout:1572: finishing write tid 5 to smithi07134777-12 2024-09-11T01:40:04.007 INFO:tasks.rados.rados.0.smithi071.stdout:1572: oid 12 updating version 0 to 833 2024-09-11T01:40:04.007 INFO:tasks.rados.rados.0.smithi071.stdout:1572: oid 12 version 833 is already newer than 832 2024-09-11T01:40:04.007 INFO:tasks.rados.rados.0.smithi071.stdout:1572: oid 12 updating version 833 to 834 2024-09-11T01:40:04.007 INFO:tasks.rados.rados.0.smithi071.stdout:1572: oid 12 updating version 834 to 835 2024-09-11T01:40:04.007 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 835 (ObjNum 585 snap 168 seq_num 585) dirty exists 2024-09-11T01:40:04.008 INFO:tasks.rados.rados.0.smithi071.stdout:1572: left oid 12 (ObjNum 585 snap 168 seq_num 585) 2024-09-11T01:40:04.008 INFO:tasks.rados.rados.0.smithi071.stdout:1574: finishing rollback tid 2 to smithi07134777-36 2024-09-11T01:40:04.008 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 454 (ObjNum 469 snap 124 seq_num 469) dirty exists 2024-09-11T01:40:04.008 INFO:tasks.rados.rados.0.smithi071.stdout:1575: finishing write tid 1 to smithi07134777-48 2024-09-11T01:40:04.008 INFO:tasks.rados.rados.0.smithi071.stdout:1572: done (5 left) 2024-09-11T01:40:04.008 INFO:tasks.rados.rados.0.smithi071.stdout:1574: done (4 left) 2024-09-11T01:40:04.008 INFO:tasks.rados.rados.0.smithi071.stdout:1578: read oid 26 snap -1 2024-09-11T01:40:04.008 INFO:tasks.rados.rados.0.smithi071.stdout:1578: expect deleted 2024-09-11T01:40:04.008 INFO:tasks.rados.rados.0.smithi071.stdout:1579: snap_create 2024-09-11T01:40:04.008 INFO:tasks.rados.rados.0.smithi071.stdout:1575: finishing write tid 2 to smithi07134777-48 2024-09-11T01:40:04.008 INFO:tasks.rados.rados.0.smithi071.stdout:1573: finishing copy_from to smithi07134777-24 2024-09-11T01:40:04.008 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 437 (ObjNum 488 snap 132 seq_num 488) dirty exists 2024-09-11T01:40:04.008 INFO:tasks.rados.rados.0.smithi071.stdout:1575: finishing write tid 3 to smithi07134777-48 2024-09-11T01:40:04.008 INFO:tasks.rados.rados.0.smithi071.stdout:1575: finishing write tid 4 to smithi07134777-48 2024-09-11T01:40:04.008 INFO:tasks.rados.rados.0.smithi071.stdout:1575: finishing write tid 5 to smithi07134777-48 2024-09-11T01:40:04.009 INFO:tasks.rados.rados.0.smithi071.stdout:1575: oid 48 updating version 0 to 894 2024-09-11T01:40:04.009 INFO:tasks.rados.rados.0.smithi071.stdout:1575: oid 48 updating version 894 to 896 2024-09-11T01:40:04.009 INFO:tasks.rados.rados.0.smithi071.stdout:1575: oid 48 version 896 is already newer than 895 2024-09-11T01:40:04.009 INFO:tasks.rados.rados.0.smithi071.stdout:1575: oid 48 version 896 is already newer than 893 2024-09-11T01:40:04.009 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 896 (ObjNum 586 snap 168 seq_num 586) dirty exists 2024-09-11T01:40:04.009 INFO:tasks.rados.rados.0.smithi071.stdout:1575: left oid 48 (ObjNum 586 snap 168 seq_num 586) 2024-09-11T01:40:04.009 INFO:tasks.rados.rados.0.smithi071.stdout:1576: finishing write tid 1 to smithi07134777-5 2024-09-11T01:40:04.009 INFO:tasks.rados.rados.0.smithi071.stdout:1576: finishing write tid 2 to smithi07134777-5 2024-09-11T01:40:04.010 INFO:tasks.rados.rados.0.smithi071.stdout:1576: finishing write tid 3 to smithi07134777-5 2024-09-11T01:40:04.010 INFO:tasks.rados.rados.0.smithi071.stdout:1576: finishing write tid 4 to smithi07134777-5 2024-09-11T01:40:04.010 INFO:tasks.rados.rados.0.smithi071.stdout:1577: finishing rollback tid 0 to smithi07134777-33 2024-09-11T01:40:04.010 INFO:tasks.rados.rados.0.smithi071.stdout:1576: finishing write tid 5 to smithi07134777-5 2024-09-11T01:40:04.010 INFO:tasks.rados.rados.0.smithi071.stdout:1576: oid 5 updating version 0 to 492 2024-09-11T01:40:04.010 INFO:tasks.rados.rados.0.smithi071.stdout:1576: oid 5 version 492 is already newer than 490 2024-09-11T01:40:04.010 INFO:tasks.rados.rados.0.smithi071.stdout:1576: oid 5 version 492 is already newer than 489 2024-09-11T01:40:04.010 INFO:tasks.rados.rados.0.smithi071.stdout:1576: oid 5 version 492 is already newer than 491 2024-09-11T01:40:04.011 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 492 (ObjNum 587 snap 168 seq_num 587) dirty exists 2024-09-11T01:40:04.011 INFO:tasks.rados.rados.0.smithi071.stdout:1576: left oid 5 (ObjNum 587 snap 168 seq_num 587) 2024-09-11T01:40:04.011 INFO:tasks.rados.rados.0.smithi071.stdout:1577: finishing rollback tid 1 to smithi07134777-33 2024-09-11T01:40:04.011 INFO:tasks.rados.rados.0.smithi071.stdout:1577: finishing rollback tid 2 to smithi07134777-33 2024-09-11T01:40:04.012 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 838 (ObjNum 250 snap 63 seq_num 250) dirty exists 2024-09-11T01:40:04.974 INFO:tasks.rados.rados.0.smithi071.stdout:1573: done (5 left) 2024-09-11T01:40:04.974 INFO:tasks.rados.rados.0.smithi071.stdout:1575: done (4 left) 2024-09-11T01:40:04.974 INFO:tasks.rados.rados.0.smithi071.stdout:1576: done (3 left) 2024-09-11T01:40:04.974 INFO:tasks.rados.rados.0.smithi071.stdout:1577: done (2 left) 2024-09-11T01:40:04.974 INFO:tasks.rados.rados.0.smithi071.stdout:1578: done (1 left) 2024-09-11T01:40:04.974 INFO:tasks.rados.rados.0.smithi071.stdout:1579: done (0 left) 2024-09-11T01:40:04.974 INFO:tasks.rados.rados.0.smithi071.stdout:1580: write (excl) oid 30 current snap is 169 2024-09-11T01:40:04.974 INFO:tasks.rados.rados.0.smithi071.stdout:1580: seq_num 588 ranges {682516=405593,1791765=569861,2830955=721120} 2024-09-11T01:40:04.979 INFO:tasks.rados.rados.0.smithi071.stdout:1580: writing smithi07134777-30 from 682516 to 1088109 tid 1 2024-09-11T01:40:04.986 INFO:tasks.rados.rados.0.smithi071.stdout:1580: writing smithi07134777-30 from 1791765 to 2361626 tid 2 2024-09-11T01:40:04.993 INFO:tasks.rados.rados.0.smithi071.stdout:1580: writing smithi07134777-30 from 2830955 to 3552075 tid 3 2024-09-11T01:40:04.993 INFO:tasks.rados.rados.0.smithi071.stdout:1581: delete oid 22 current snap is 169 2024-09-11T01:40:04.993 INFO:tasks.rados.rados.0.smithi071.stdout:1580: finishing write tid 1 to smithi07134777-30 2024-09-11T01:40:04.993 INFO:tasks.rados.rados.0.smithi071.stdout:1580: finishing write tid 2 to smithi07134777-30 2024-09-11T01:40:04.997 INFO:tasks.rados.rados.0.smithi071.stdout:1580: finishing write tid 3 to smithi07134777-30 2024-09-11T01:40:04.997 INFO:tasks.rados.rados.0.smithi071.stdout:1580: finishing write tid 4 to smithi07134777-30 2024-09-11T01:40:04.998 INFO:tasks.rados.rados.0.smithi071.stdout:1580: finishing write tid 5 to smithi07134777-30 2024-09-11T01:40:04.998 INFO:tasks.rados.rados.0.smithi071.stdout:1580: oid 30 updating version 0 to 636 2024-09-11T01:40:04.998 INFO:tasks.rados.rados.0.smithi071.stdout:1580: oid 30 version 636 is already newer than 635 2024-09-11T01:40:04.998 INFO:tasks.rados.rados.0.smithi071.stdout:1580: oid 30 version 636 is already newer than 634 2024-09-11T01:40:04.998 INFO:tasks.rados.rados.0.smithi071.stdout:1580: oid 30 version 636 is already newer than 633 2024-09-11T01:40:04.998 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 636 (ObjNum 588 snap 169 seq_num 588) dirty exists 2024-09-11T01:40:04.999 INFO:tasks.rados.rados.0.smithi071.stdout:1580: left oid 30 (ObjNum 588 snap 169 seq_num 588) 2024-09-11T01:40:04.999 INFO:tasks.rados.rados.0.smithi071.stdout:1580: done (1 left) 2024-09-11T01:40:04.999 INFO:tasks.rados.rados.0.smithi071.stdout:1581: done (0 left) 2024-09-11T01:40:04.999 INFO:tasks.rados.rados.0.smithi071.stdout:1582: rollback oid 27 current snap is 169 2024-09-11T01:40:04.999 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 27 to 167 2024-09-11T01:40:04.999 INFO:tasks.rados.rados.0.smithi071.stdout:1583: delete oid 24 current snap is 169 2024-09-11T01:40:05.002 INFO:tasks.rados.rados.0.smithi071.stdout:1583: done (1 left) 2024-09-11T01:40:05.002 INFO:tasks.rados.rados.0.smithi071.stdout:1584: copy_from oid 32 from oid 38 current snap is 169 2024-09-11T01:40:05.002 INFO:tasks.rados.rados.0.smithi071.stdout:1585: read oid 21 snap -1 2024-09-11T01:40:05.002 INFO:tasks.rados.rados.0.smithi071.stdout:1585: expect (ObjNum 542 snap 154 seq_num 542) 2024-09-11T01:40:05.002 INFO:tasks.rados.rados.0.smithi071.stdout:1586: write oid 35 current snap is 169 2024-09-11T01:40:05.002 INFO:tasks.rados.rados.0.smithi071.stdout:1586: seq_num 591 ranges {732937=631340,1850858=433541,2796871=671289,3888195=44742} 2024-09-11T01:40:05.009 INFO:tasks.rados.rados.0.smithi071.stdout:1586: writing smithi07134777-35 from 732937 to 1364277 tid 1 2024-09-11T01:40:05.014 INFO:tasks.rados.rados.0.smithi071.stdout:1586: writing smithi07134777-35 from 1850858 to 2284399 tid 2 2024-09-11T01:40:05.021 INFO:tasks.rados.rados.0.smithi071.stdout:1586: writing smithi07134777-35 from 2796871 to 3468160 tid 3 2024-09-11T01:40:05.023 INFO:tasks.rados.rados.0.smithi071.stdout:1586: writing smithi07134777-35 from 3888195 to 3932937 tid 4 2024-09-11T01:40:05.024 INFO:tasks.rados.rados.0.smithi071.stdout:1587: delete oid 49 current snap is 169 2024-09-11T01:40:05.024 INFO:tasks.rados.rados.0.smithi071.stdout:1582: finishing rollback tid 0 to smithi07134777-27 2024-09-11T01:40:05.024 INFO:tasks.rados.rados.0.smithi071.stdout:1582: finishing rollback tid 1 to smithi07134777-27 2024-09-11T01:40:05.024 INFO:tasks.rados.rados.0.smithi071.stdout:1582: finishing rollback tid 2 to smithi07134777-27 2024-09-11T01:40:05.024 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 841 (ObjNum 538 snap 152 seq_num 538) dirty exists 2024-09-11T01:40:05.024 INFO:tasks.rados.rados.0.smithi071.stdout:1585: expect (ObjNum 542 snap 154 seq_num 542) 2024-09-11T01:40:05.072 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:40:05.162 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2985379 2024-09-11T01:40:05.168 INFO:tasks.rados.rados.0.smithi071.stdout:1584: finishing copy_from racing read to smithi07134777-32 2024-09-11T01:40:05.168 INFO:tasks.rados.rados.0.smithi071.stdout:1586: finishing write tid 1 to smithi07134777-35 2024-09-11T01:40:05.168 INFO:tasks.rados.rados.0.smithi071.stdout:1584: finishing copy_from to smithi07134777-32 2024-09-11T01:40:05.169 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 891 (ObjNum 582 snap 167 seq_num 582) dirty exists 2024-09-11T01:40:05.169 INFO:tasks.rados.rados.0.smithi071.stdout:1586: finishing write tid 2 to smithi07134777-35 2024-09-11T01:40:05.169 INFO:tasks.rados.rados.0.smithi071.stdout:1586: finishing write tid 3 to smithi07134777-35 2024-09-11T01:40:05.169 INFO:tasks.rados.rados.0.smithi071.stdout:1586: finishing write tid 4 to smithi07134777-35 2024-09-11T01:40:05.169 INFO:tasks.rados.rados.0.smithi071.stdout:1586: finishing write tid 5 to smithi07134777-35 2024-09-11T01:40:05.169 INFO:tasks.rados.rados.0.smithi071.stdout:1586: finishing write tid 6 to smithi07134777-35 2024-09-11T01:40:05.169 INFO:tasks.rados.rados.0.smithi071.stdout:1586: oid 35 updating version 0 to 894 2024-09-11T01:40:05.169 INFO:tasks.rados.rados.0.smithi071.stdout:1586: oid 35 updating version 894 to 895 2024-09-11T01:40:05.169 INFO:tasks.rados.rados.0.smithi071.stdout:1586: oid 35 version 895 is already newer than 614 2024-09-11T01:40:05.169 INFO:tasks.rados.rados.0.smithi071.stdout:1586: oid 35 version 895 is already newer than 893 2024-09-11T01:40:05.169 INFO:tasks.rados.rados.0.smithi071.stdout:1586: oid 35 version 895 is already newer than 892 2024-09-11T01:40:05.169 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 895 (ObjNum 591 snap 169 seq_num 591) dirty exists 2024-09-11T01:40:05.169 INFO:tasks.rados.rados.0.smithi071.stdout:1586: left oid 35 (ObjNum 591 snap 169 seq_num 591) 2024-09-11T01:40:05.170 INFO:tasks.rados.rados.0.smithi071.stdout:1582: done (4 left) 2024-09-11T01:40:05.170 INFO:tasks.rados.rados.0.smithi071.stdout:1584: done (3 left) 2024-09-11T01:40:05.170 INFO:tasks.rados.rados.0.smithi071.stdout:1585: done (2 left) 2024-09-11T01:40:05.170 INFO:tasks.rados.rados.0.smithi071.stdout:1586: done (1 left) 2024-09-11T01:40:05.170 INFO:tasks.rados.rados.0.smithi071.stdout:1587: done (0 left) 2024-09-11T01:40:05.170 INFO:tasks.rados.rados.0.smithi071.stdout:1588: snap_create 2024-09-11T01:40:05.973 INFO:tasks.rados.rados.0.smithi071.stdout:1588: done (0 left) 2024-09-11T01:40:05.973 INFO:tasks.rados.rados.0.smithi071.stdout:1589: read oid 14 snap -1 2024-09-11T01:40:05.973 INFO:tasks.rados.rados.0.smithi071.stdout:1589: expect (ObjNum 566 snap 160 seq_num 566) 2024-09-11T01:40:05.973 INFO:tasks.rados.rados.0.smithi071.stdout:1590: snap_remove snap 169 2024-09-11T01:40:06.976 INFO:tasks.rados.rados.0.smithi071.stdout:1590: done (1 left) 2024-09-11T01:40:06.976 INFO:tasks.rados.rados.0.smithi071.stdout:1591: read oid 17 snap -1 2024-09-11T01:40:06.976 INFO:tasks.rados.rados.0.smithi071.stdout:1591: expect (ObjNum 562 snap 158 seq_num 562) 2024-09-11T01:40:06.976 INFO:tasks.rados.rados.0.smithi071.stdout:1592: delete oid 32 current snap is 170 2024-09-11T01:40:06.976 INFO:tasks.rados.rados.0.smithi071.stdout:1589: expect (ObjNum 566 snap 160 seq_num 566) 2024-09-11T01:40:06.976 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 132182 2024-09-11T01:40:07.100 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 208269 2024-09-11T01:40:07.159 INFO:tasks.rados.rados.0.smithi071.stdout:1591: expect (ObjNum 562 snap 158 seq_num 562) 2024-09-11T01:40:07.163 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1772001 2024-09-11T01:40:07.186 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:40:07.259 INFO:tasks.rados.rados.0.smithi071.stdout:1589: done (2 left) 2024-09-11T01:40:07.259 INFO:tasks.rados.rados.0.smithi071.stdout:1591: done (1 left) 2024-09-11T01:40:07.259 INFO:tasks.rados.rados.0.smithi071.stdout:1592: done (0 left) 2024-09-11T01:40:07.259 INFO:tasks.rados.rados.0.smithi071.stdout:1593: read oid 9 snap 164 2024-09-11T01:40:07.260 INFO:tasks.rados.rados.0.smithi071.stdout:1593: expect deleted 2024-09-11T01:40:07.260 INFO:tasks.rados.rados.0.smithi071.stdout:1594: delete oid 17 current snap is 170 2024-09-11T01:40:07.262 INFO:tasks.rados.rados.0.smithi071.stdout:1593: done (1 left) 2024-09-11T01:40:07.262 INFO:tasks.rados.rados.0.smithi071.stdout:1594: done (0 left) 2024-09-11T01:40:07.262 INFO:tasks.rados.rados.0.smithi071.stdout:1595: rollback oid 22 current snap is 170 2024-09-11T01:40:07.262 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 22 to 168 2024-09-11T01:40:07.262 INFO:tasks.rados.rados.0.smithi071.stdout:1596: read oid 13 snap 164 2024-09-11T01:40:07.262 INFO:tasks.rados.rados.0.smithi071.stdout:1596: expect (ObjNum 567 snap 160 seq_num 567) 2024-09-11T01:40:07.263 INFO:tasks.rados.rados.0.smithi071.stdout:1597: delete oid 40 current snap is 170 2024-09-11T01:40:07.266 INFO:tasks.rados.rados.0.smithi071.stdout:1597: done (2 left) 2024-09-11T01:40:07.266 INFO:tasks.rados.rados.0.smithi071.stdout:1598: rollback oid 24 current snap is 170 2024-09-11T01:40:07.266 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 24 to 167 2024-09-11T01:40:07.266 INFO:tasks.rados.rados.0.smithi071.stdout:1599: delete oid 4 current snap is 170 2024-09-11T01:40:07.266 INFO:tasks.rados.rados.0.smithi071.stdout:1595: finishing rollback tid 1 to smithi07134777-22 2024-09-11T01:40:07.266 INFO:tasks.rados.rados.0.smithi071.stdout:1595: finishing rollback tid 2 to smithi07134777-22 2024-09-11T01:40:07.266 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 454 (ObjNum 503 snap 135 seq_num 503) dirty exists 2024-09-11T01:40:07.269 INFO:tasks.rados.rados.0.smithi071.stdout:1596: expect (ObjNum 567 snap 160 seq_num 567) 2024-09-11T01:40:07.280 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2120571 2024-09-11T01:40:07.318 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:40:07.356 INFO:tasks.rados.rados.0.smithi071.stdout:1598: finishing rollback tid 1 to smithi07134777-24 2024-09-11T01:40:07.356 INFO:tasks.rados.rados.0.smithi071.stdout:1598: finishing rollback tid 2 to smithi07134777-24 2024-09-11T01:40:07.356 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 615 (ObjNum 430 snap 119 seq_num 430) dirty exists 2024-09-11T01:40:07.357 INFO:tasks.rados.rados.0.smithi071.stdout:1595: done (3 left) 2024-09-11T01:40:07.357 INFO:tasks.rados.rados.0.smithi071.stdout:1596: done (2 left) 2024-09-11T01:40:07.357 INFO:tasks.rados.rados.0.smithi071.stdout:1598: done (1 left) 2024-09-11T01:40:07.357 INFO:tasks.rados.rados.0.smithi071.stdout:1599: done (0 left) 2024-09-11T01:40:07.357 INFO:tasks.rados.rados.0.smithi071.stdout:1600: write (excl) oid 11 current snap is 170 2024-09-11T01:40:07.357 INFO:tasks.rados.rados.0.smithi071.stdout:1600: seq_num 597 ranges {433779=799187,1801750=232029} 2024-09-11T01:40:07.362 INFO:tasks.rados.rados.0.smithi071.stdout:1600: writing smithi07134777-11 from 433779 to 1232966 tid 1 2024-09-11T01:40:07.367 INFO:tasks.rados.rados.0.smithi071.stdout:1600: writing smithi07134777-11 from 1801750 to 2033779 tid 2 2024-09-11T01:40:07.367 INFO:tasks.rados.rados.0.smithi071.stdout:1601: read oid 44 snap -1 2024-09-11T01:40:07.367 INFO:tasks.rados.rados.0.smithi071.stdout:1601: expect deleted 2024-09-11T01:40:07.367 INFO:tasks.rados.rados.0.smithi071.stdout:1602: copy_from oid 26 from oid 37 current snap is 170 2024-09-11T01:40:07.367 INFO:tasks.rados.rados.0.smithi071.stdout:1603: write oid 19 current snap is 170 2024-09-11T01:40:07.368 INFO:tasks.rados.rados.0.smithi071.stdout:1603: seq_num 598 ranges {450586=474436,1343879=588206} 2024-09-11T01:40:07.372 INFO:tasks.rados.rados.0.smithi071.stdout:1603: writing smithi07134777-19 from 450586 to 925022 tid 1 2024-09-11T01:40:07.377 INFO:tasks.rados.rados.0.smithi071.stdout:1603: writing smithi07134777-19 from 1343879 to 1932085 tid 2 2024-09-11T01:40:07.377 INFO:tasks.rados.rados.0.smithi071.stdout:1604: snap_remove snap 164 2024-09-11T01:40:07.987 INFO:tasks.rados.rados.0.smithi071.stdout:1604: done (4 left) 2024-09-11T01:40:07.987 INFO:tasks.rados.rados.0.smithi071.stdout:1605: snap_create 2024-09-11T01:40:07.987 INFO:tasks.rados.rados.0.smithi071.stdout:1600: finishing write tid 1 to smithi07134777-11 2024-09-11T01:40:07.987 INFO:tasks.rados.rados.0.smithi071.stdout:1600: finishing write tid 2 to smithi07134777-11 2024-09-11T01:40:07.987 INFO:tasks.rados.rados.0.smithi071.stdout:1600: finishing write tid 3 to smithi07134777-11 2024-09-11T01:40:07.988 INFO:tasks.rados.rados.0.smithi071.stdout:1600: finishing write tid 4 to smithi07134777-11 2024-09-11T01:40:07.988 INFO:tasks.rados.rados.0.smithi071.stdout:1600: oid 11 updating version 0 to 639 2024-09-11T01:40:07.988 INFO:tasks.rados.rados.0.smithi071.stdout:1600: oid 11 version 639 is already newer than 638 2024-09-11T01:40:07.988 INFO:tasks.rados.rados.0.smithi071.stdout:1600: oid 11 updating version 639 to 640 2024-09-11T01:40:07.988 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 640 (ObjNum 597 snap 170 seq_num 597) dirty exists 2024-09-11T01:40:07.988 INFO:tasks.rados.rados.0.smithi071.stdout:1600: left oid 11 (ObjNum 597 snap 170 seq_num 597) 2024-09-11T01:40:07.988 INFO:tasks.rados.rados.0.smithi071.stdout:1602: finishing copy_from racing read to smithi07134777-26 2024-09-11T01:40:07.988 INFO:tasks.rados.rados.0.smithi071.stdout:1603: finishing write tid 1 to smithi07134777-19 2024-09-11T01:40:07.988 INFO:tasks.rados.rados.0.smithi071.stdout:1602: finishing copy_from to smithi07134777-26 2024-09-11T01:40:07.988 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 604 (ObjNum 577 snap 165 seq_num 577) dirty exists 2024-09-11T01:40:07.988 INFO:tasks.rados.rados.0.smithi071.stdout:1603: finishing write tid 2 to smithi07134777-19 2024-09-11T01:40:07.989 INFO:tasks.rados.rados.0.smithi071.stdout:1603: finishing write tid 3 to smithi07134777-19 2024-09-11T01:40:07.989 INFO:tasks.rados.rados.0.smithi071.stdout:1603: finishing write tid 4 to smithi07134777-19 2024-09-11T01:40:07.989 INFO:tasks.rados.rados.0.smithi071.stdout:1603: oid 19 updating version 0 to 455 2024-09-11T01:40:07.989 INFO:tasks.rados.rados.0.smithi071.stdout:1603: oid 19 updating version 455 to 456 2024-09-11T01:40:07.989 INFO:tasks.rados.rados.0.smithi071.stdout:1603: oid 19 updating version 456 to 457 2024-09-11T01:40:07.989 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 457 (ObjNum 598 snap 170 seq_num 598) dirty exists 2024-09-11T01:40:07.989 INFO:tasks.rados.rados.0.smithi071.stdout:1603: left oid 19 (ObjNum 598 snap 170 seq_num 598) 2024-09-11T01:40:08.944 INFO:tasks.rados.rados.0.smithi071.stdout:1600: done (4 left) 2024-09-11T01:40:08.944 INFO:tasks.rados.rados.0.smithi071.stdout:1601: done (3 left) 2024-09-11T01:40:08.944 INFO:tasks.rados.rados.0.smithi071.stdout:1602: done (2 left) 2024-09-11T01:40:08.944 INFO:tasks.rados.rados.0.smithi071.stdout:1603: done (1 left) 2024-09-11T01:40:08.944 INFO:tasks.rados.rados.0.smithi071.stdout:1605: done (0 left) 2024-09-11T01:40:08.945 INFO:tasks.rados.rados.0.smithi071.stdout:1606: write oid 47 current snap is 171 2024-09-11T01:40:08.945 INFO:tasks.rados.rados.0.smithi071.stdout:1606: seq_num 599 ranges {467393=549685,1802361=265032} 2024-09-11T01:40:08.949 INFO:tasks.rados.rados.0.smithi071.stdout:1606: writing smithi07134777-47 from 467393 to 1017078 tid 1 2024-09-11T01:40:08.956 INFO:tasks.rados.rados.0.smithi071.stdout:1606: writing smithi07134777-47 from 1802361 to 2067393 tid 2 2024-09-11T01:40:08.956 INFO:tasks.rados.rados.0.smithi071.stdout:1607: rollback oid 48 current snap is 171 2024-09-11T01:40:08.956 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 48 to 168 2024-09-11T01:40:08.956 INFO:tasks.rados.rados.0.smithi071.stdout:1606: finishing write tid 1 to smithi07134777-47 2024-09-11T01:40:08.956 INFO:tasks.rados.rados.0.smithi071.stdout:1608: write (excl) oid 24 current snap is 171 2024-09-11T01:40:08.956 INFO:tasks.rados.rados.0.smithi071.stdout:1608: seq_num 600 ranges {484200=624934,1860843=223357} 2024-09-11T01:40:08.962 INFO:tasks.rados.rados.0.smithi071.stdout:1608: writing smithi07134777-24 from 484200 to 1109134 tid 1 2024-09-11T01:40:08.968 INFO:tasks.rados.rados.0.smithi071.stdout:1608: writing smithi07134777-24 from 1860843 to 2084200 tid 2 2024-09-11T01:40:08.968 INFO:tasks.rados.rados.0.smithi071.stdout:1609: rollback oid 46 current snap is 171 2024-09-11T01:40:08.968 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 46 to 166 2024-09-11T01:40:08.968 INFO:tasks.rados.rados.0.smithi071.stdout:1606: finishing write tid 2 to smithi07134777-47 2024-09-11T01:40:08.968 INFO:tasks.rados.rados.0.smithi071.stdout:1606: finishing write tid 3 to smithi07134777-47 2024-09-11T01:40:08.968 INFO:tasks.rados.rados.0.smithi071.stdout:1606: finishing write tid 4 to smithi07134777-47 2024-09-11T01:40:08.969 INFO:tasks.rados.rados.0.smithi071.stdout:1606: oid 47 updating version 0 to 788 2024-09-11T01:40:08.969 INFO:tasks.rados.rados.0.smithi071.stdout:1606: oid 47 version 788 is already newer than 787 2024-09-11T01:40:08.969 INFO:tasks.rados.rados.0.smithi071.stdout:1606: oid 47 version 788 is already newer than 786 2024-09-11T01:40:08.969 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 788 (ObjNum 599 snap 171 seq_num 599) dirty exists 2024-09-11T01:40:08.969 INFO:tasks.rados.rados.0.smithi071.stdout:1606: left oid 47 (ObjNum 599 snap 171 seq_num 599) 2024-09-11T01:40:08.969 INFO:tasks.rados.rados.0.smithi071.stdout:1607: finishing rollback tid 0 to smithi07134777-48 2024-09-11T01:40:08.969 INFO:tasks.rados.rados.0.smithi071.stdout:1607: finishing rollback tid 1 to smithi07134777-48 2024-09-11T01:40:08.969 INFO:tasks.rados.rados.0.smithi071.stdout:1607: finishing rollback tid 2 to smithi07134777-48 2024-09-11T01:40:08.969 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 904 (ObjNum 586 snap 168 seq_num 586) dirty exists 2024-09-11T01:40:08.969 INFO:tasks.rados.rados.0.smithi071.stdout:1606: done (3 left) 2024-09-11T01:40:08.969 INFO:tasks.rados.rados.0.smithi071.stdout:1607: done (2 left) 2024-09-11T01:40:08.969 INFO:tasks.rados.rados.0.smithi071.stdout:1610: snap_create 2024-09-11T01:40:08.972 INFO:tasks.rados.rados.0.smithi071.stdout:1608: finishing write tid 1 to smithi07134777-24 2024-09-11T01:40:08.972 INFO:tasks.rados.rados.0.smithi071.stdout:1609: finishing rollback tid 0 to smithi07134777-46 2024-09-11T01:40:08.973 INFO:tasks.rados.rados.0.smithi071.stdout:1608: finishing write tid 2 to smithi07134777-24 2024-09-11T01:40:08.973 INFO:tasks.rados.rados.0.smithi071.stdout:1608: finishing write tid 3 to smithi07134777-24 2024-09-11T01:40:08.973 INFO:tasks.rados.rados.0.smithi071.stdout:1608: finishing write tid 4 to smithi07134777-24 2024-09-11T01:40:08.974 INFO:tasks.rados.rados.0.smithi071.stdout:1608: oid 24 updating version 0 to 624 2024-09-11T01:40:08.974 INFO:tasks.rados.rados.0.smithi071.stdout:1608: oid 24 updating version 624 to 625 2024-09-11T01:40:08.974 INFO:tasks.rados.rados.0.smithi071.stdout:1608: oid 24 version 625 is already newer than 623 2024-09-11T01:40:08.974 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 625 (ObjNum 600 snap 171 seq_num 600) dirty exists 2024-09-11T01:40:08.974 INFO:tasks.rados.rados.0.smithi071.stdout:1608: left oid 24 (ObjNum 600 snap 171 seq_num 600) 2024-09-11T01:40:08.976 INFO:tasks.rados.rados.0.smithi071.stdout:1609: finishing rollback tid 1 to smithi07134777-46 2024-09-11T01:40:08.976 INFO:tasks.rados.rados.0.smithi071.stdout:1609: finishing rollback tid 2 to smithi07134777-46 2024-09-11T01:40:08.976 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 696 (ObjNum 494 snap 132 seq_num 494) dirty exists 2024-09-11T01:40:09.943 INFO:tasks.rados.rados.0.smithi071.stdout:1608: done (2 left) 2024-09-11T01:40:09.943 INFO:tasks.rados.rados.0.smithi071.stdout:1609: done (1 left) 2024-09-11T01:40:09.943 INFO:tasks.rados.rados.0.smithi071.stdout:1610: done (0 left) 2024-09-11T01:40:09.943 INFO:tasks.rados.rados.0.smithi071.stdout:1611: write oid 10 current snap is 172 2024-09-11T01:40:09.943 INFO:tasks.rados.rados.0.smithi071.stdout:1611: seq_num 601 ranges {501007=416536,1635678=465329} 2024-09-11T01:40:09.947 INFO:tasks.rados.rados.0.smithi071.stdout:1611: writing smithi07134777-10 from 501007 to 917543 tid 1 2024-09-11T01:40:09.954 INFO:tasks.rados.rados.0.smithi071.stdout:1611: writing smithi07134777-10 from 1635678 to 2101007 tid 2 2024-09-11T01:40:09.954 INFO:tasks.rados.rados.0.smithi071.stdout:1612: snap_create 2024-09-11T01:40:09.954 INFO:tasks.rados.rados.0.smithi071.stdout:1611: finishing write tid 1 to smithi07134777-10 2024-09-11T01:40:09.958 INFO:tasks.rados.rados.0.smithi071.stdout:1611: finishing write tid 2 to smithi07134777-10 2024-09-11T01:40:09.960 INFO:tasks.rados.rados.0.smithi071.stdout:1611: finishing write tid 3 to smithi07134777-10 2024-09-11T01:40:09.960 INFO:tasks.rados.rados.0.smithi071.stdout:1611: finishing write tid 4 to smithi07134777-10 2024-09-11T01:40:09.960 INFO:tasks.rados.rados.0.smithi071.stdout:1611: oid 10 updating version 0 to 377 2024-09-11T01:40:09.960 INFO:tasks.rados.rados.0.smithi071.stdout:1611: oid 10 updating version 377 to 378 2024-09-11T01:40:09.960 INFO:tasks.rados.rados.0.smithi071.stdout:1611: oid 10 updating version 378 to 379 2024-09-11T01:40:09.960 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 379 (ObjNum 601 snap 172 seq_num 601) dirty exists 2024-09-11T01:40:09.960 INFO:tasks.rados.rados.0.smithi071.stdout:1611: left oid 10 (ObjNum 601 snap 172 seq_num 601) 2024-09-11T01:40:10.945 INFO:tasks.rados.rados.0.smithi071.stdout:1611: done (1 left) 2024-09-11T01:40:10.945 INFO:tasks.rados.rados.0.smithi071.stdout:1612: done (0 left) 2024-09-11T01:40:10.945 INFO:tasks.rados.rados.0.smithi071.stdout:1613: snap_remove snap 168 2024-09-11T01:40:11.951 INFO:tasks.rados.rados.0.smithi071.stdout:1613: done (0 left) 2024-09-11T01:40:11.951 INFO:tasks.rados.rados.0.smithi071.stdout:1614: delete oid 36 current snap is 173 2024-09-11T01:40:11.955 INFO:tasks.rados.rados.0.smithi071.stdout:1614: done (0 left) 2024-09-11T01:40:11.955 INFO:tasks.rados.rados.0.smithi071.stdout:1615: delete oid 22 current snap is 173 2024-09-11T01:40:11.961 INFO:tasks.rados.rados.0.smithi071.stdout:1615: done (0 left) 2024-09-11T01:40:11.962 INFO:tasks.rados.rados.0.smithi071.stdout:1616: write (excl) oid 1 current snap is 173 2024-09-11T01:40:11.962 INFO:tasks.rados.rados.0.smithi071.stdout:1616: seq_num 604 ranges {551428=642283,1694771=456657} 2024-09-11T01:40:11.968 INFO:tasks.rados.rados.0.smithi071.stdout:1616: writing smithi07134777-1 from 551428 to 1193711 tid 1 2024-09-11T01:40:11.973 INFO:tasks.rados.rados.0.smithi071.stdout:1616: writing smithi07134777-1 from 1694771 to 2151428 tid 2 2024-09-11T01:40:11.974 INFO:tasks.rados.rados.0.smithi071.stdout:1617: delete oid 39 current snap is 173 2024-09-11T01:40:11.976 INFO:tasks.rados.rados.0.smithi071.stdout:1616: finishing write tid 1 to smithi07134777-1 2024-09-11T01:40:11.977 INFO:tasks.rados.rados.0.smithi071.stdout:1616: finishing write tid 2 to smithi07134777-1 2024-09-11T01:40:11.978 INFO:tasks.rados.rados.0.smithi071.stdout:1616: finishing write tid 3 to smithi07134777-1 2024-09-11T01:40:11.978 INFO:tasks.rados.rados.0.smithi071.stdout:1616: finishing write tid 4 to smithi07134777-1 2024-09-11T01:40:11.978 INFO:tasks.rados.rados.0.smithi071.stdout:1616: oid 1 updating version 0 to 789 2024-09-11T01:40:11.978 INFO:tasks.rados.rados.0.smithi071.stdout:1616: oid 1 updating version 789 to 790 2024-09-11T01:40:11.978 INFO:tasks.rados.rados.0.smithi071.stdout:1616: oid 1 updating version 790 to 791 2024-09-11T01:40:11.978 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 791 (ObjNum 604 snap 173 seq_num 604) dirty exists 2024-09-11T01:40:11.978 INFO:tasks.rados.rados.0.smithi071.stdout:1616: left oid 1 (ObjNum 604 snap 173 seq_num 604) 2024-09-11T01:40:11.979 INFO:tasks.rados.rados.0.smithi071.stdout:1616: done (1 left) 2024-09-11T01:40:11.979 INFO:tasks.rados.rados.0.smithi071.stdout:1617: done (0 left) 2024-09-11T01:40:11.979 INFO:tasks.rados.rados.0.smithi071.stdout:1618: rollback oid 26 current snap is 173 2024-09-11T01:40:11.979 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 26 to 172 2024-09-11T01:40:11.979 INFO:tasks.rados.rados.0.smithi071.stdout:1619: snap_create 2024-09-11T01:40:11.982 INFO:tasks.rados.rados.0.smithi071.stdout:1618: finishing rollback tid 0 to smithi07134777-26 2024-09-11T01:40:11.982 INFO:tasks.rados.rados.0.smithi071.stdout:1618: finishing rollback tid 1 to smithi07134777-26 2024-09-11T01:40:11.983 INFO:tasks.rados.rados.0.smithi071.stdout:1618: finishing rollback tid 2 to smithi07134777-26 2024-09-11T01:40:11.983 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 844 (ObjNum 577 snap 165 seq_num 577) dirty exists 2024-09-11T01:40:12.956 INFO:tasks.rados.rados.0.smithi071.stdout:1618: done (1 left) 2024-09-11T01:40:12.956 INFO:tasks.rados.rados.0.smithi071.stdout:1619: done (0 left) 2024-09-11T01:40:12.956 INFO:tasks.rados.rados.0.smithi071.stdout:1620: snap_create 2024-09-11T01:40:13.945 INFO:tasks.rados.rados.0.smithi071.stdout:1620: done (0 left) 2024-09-11T01:40:13.946 INFO:tasks.rados.rados.0.smithi071.stdout:1621: snap_remove snap 166 2024-09-11T01:40:14.945 INFO:tasks.rados.rados.0.smithi071.stdout:1621: done (0 left) 2024-09-11T01:40:14.945 INFO:tasks.rados.rados.0.smithi071.stdout:1622: write (excl) oid 31 current snap is 175 2024-09-11T01:40:14.945 INFO:tasks.rados.rados.0.smithi071.stdout:1622: seq_num 606 ranges {585042=792781,2095382=89660} 2024-09-11T01:40:14.952 INFO:tasks.rados.rados.0.smithi071.stdout:1622: writing smithi07134777-31 from 585042 to 1377823 tid 1 2024-09-11T01:40:14.957 INFO:tasks.rados.rados.0.smithi071.stdout:1622: writing smithi07134777-31 from 2095382 to 2185042 tid 2 2024-09-11T01:40:14.957 INFO:tasks.rados.rados.0.smithi071.stdout:1623: read oid 9 snap -1 2024-09-11T01:40:14.957 INFO:tasks.rados.rados.0.smithi071.stdout:1623: expect deleted 2024-09-11T01:40:14.958 INFO:tasks.rados.rados.0.smithi071.stdout:1624: copy_from oid 44 from oid 4 current snap is 175 2024-09-11T01:40:14.958 INFO:tasks.rados.rados.0.smithi071.stdout:1625: delete oid 8 current snap is 175 2024-09-11T01:40:14.959 INFO:tasks.rados.rados.0.smithi071.stdout:1624: finishing copy_from to smithi07134777-44 2024-09-11T01:40:14.959 INFO:tasks.rados.rados.0.smithi071.stdout:1624: got expected ENOENT (src dne) 2024-09-11T01:40:14.959 INFO:tasks.rados.rados.0.smithi071.stdout:1624: finishing copy_from racing read to smithi07134777-44 2024-09-11T01:40:14.963 INFO:tasks.rados.rados.0.smithi071.stdout:1623: done (3 left) 2024-09-11T01:40:14.963 INFO:tasks.rados.rados.0.smithi071.stdout:1624: done (2 left) 2024-09-11T01:40:14.963 INFO:tasks.rados.rados.0.smithi071.stdout:1625: done (1 left) 2024-09-11T01:40:14.963 INFO:tasks.rados.rados.0.smithi071.stdout:1626: snap_remove snap 170 2024-09-11T01:40:15.115 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-11T01:40:15.115 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-11T01:40:15.116 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-11T01:40:15.119 INFO:tasks.ceph.mon.b:Started 2024-09-11T01:40:15.119 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:40:15.119 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:40:15.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.231+0000 7f50aad00640 10 monclient: get_monmap_and_config 2024-09-11T01:40:15.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.231+0000 7f50aad00640 10 monclient: build_initial_monmap 2024-09-11T01:40:15.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.231+0000 7f50aad00640 10 monclient: monmap: 2024-09-11T01:40:15.234 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:15.234 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:15.234 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:40:15.233429+0000 2024-09-11T01:40:15.234 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:40:15.233429+0000 2024-09-11T01:40:15.234 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:15.234 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:15.234 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:40:15.235 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:40:15.235 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:40:15.235 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:40:15.235 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:40:15.235 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:15.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.232+0000 7f50aad00640 1 Processor -- start 2024-09-11T01:40:15.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.232+0000 7f50aad00640 1 -- start start 2024-09-11T01:40:15.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.232+0000 7f50aad00640 10 monclient: init 2024-09-11T01:40:15.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.232+0000 7f50aad00640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:15.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.232+0000 7f50aad00640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-11T01:40:15.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.232+0000 7f50aad00640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f50a409d790 0x7f50a409db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:15.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50aad00640 10 monclient(hunting): picked mon.noname-a con 0x7f50a409d790 addr v2:172.21.15.12:3300/0 2024-09-11T01:40:15.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50aad00640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f50a409e0b0 addr v1:172.21.15.71:6789/0 2024-09-11T01:40:15.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50aad00640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f50a409e500 0x7f50a4135170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:15.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50aad00640 10 monclient(hunting): picked mon.noname-c con 0x7f50a409e500 addr v2:172.21.15.71:3300/0 2024-09-11T01:40:15.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50aad00640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50a40a1da0 con 0x7f50a409e0b0 2024-09-11T01:40:15.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50aad00640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:15.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50aad00640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f50a409e960 con 0x7f50a409d790 2024-09-11T01:40:15.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50aad00640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:15.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50aad00640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f50a40a0760 con 0x7f50a409e500 2024-09-11T01:40:15.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50aad00640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:15.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50aad00640 10 monclient(hunting): authenticate will time out at 1421.676059s 2024-09-11T01:40:15.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50a8a9c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f50a409d790 0x7f50a409db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:15.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50a8a9c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f50a409d790 0x7f50a409db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48244/0 (socket says 172.21.15.12:48244) 2024-09-11T01:40:15.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50a8a9c640 1 -- 172.21.15.12:0/653361305 learned_addr learned my addr 172.21.15.12:0/653361305 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:40:15.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50a8a9c640 10 monclient(hunting): get_auth_request con 0x7f50a409d790 auth_method 0 2024-09-11T01:40:15.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50a8a9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:15.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50a8a9c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:15.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50a8a9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:15.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50a8a9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:15.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50a8a9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:15.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50a8a9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:15.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50a929d640 1 --2- 172.21.15.12:0/653361305 >> v2:172.21.15.71:3300/0 conn(0x7f50a409e500 0x7f50a4135170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:15.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50a929d640 10 monclient(hunting): get_auth_request con 0x7f50a409e500 auth_method 0 2024-09-11T01:40:15.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50a929d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:15.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50a929d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:15.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50a929d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:15.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50a8a9c640 10 monclient(hunting): handle_auth_done global_id 94359 payload 274 2024-09-11T01:40:15.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50a8a9c640 1 -- 172.21.15.12:0/653361305 >> v2:172.21.15.71:3300/0 conn(0x7f50a409e500 msgr2=0x7f50a4135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:40:15.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50a8a9c640 1 --2- 172.21.15.12:0/653361305 >> v2:172.21.15.71:3300/0 conn(0x7f50a409e500 0x7f50a4135170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:15.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50a8a9c640 1 -- 172.21.15.12:0/653361305 >> v1:172.21.15.71:6789/0 conn(0x7f50a409e0b0 legacy=0x7f50a4134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:40:15.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50a8a9c640 10 monclient: _finish_hunting 0 2024-09-11T01:40:15.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50a8a9c640 1 monclient: found mon.noname-a 2024-09-11T01:40:15.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50a8a9c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:15.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50a8a9c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:40:15.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.233+0000 7f50a8a9c640 1 -- 172.21.15.12:0/653361305 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50a40a1fa0 con 0x7f50a409d790 2024-09-11T01:40:15.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.236+0000 7f50a8a9c640 1 -- 172.21.15.12:0/653361305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50a409d790 msgr2=0x7f50a409db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:15.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.236+0000 7f50a8a9c640 1 --2- 172.21.15.12:0/653361305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50a409d790 0x7f50a409db70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f50a4074f40 tx=0x7f50a002a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5076a34df813d53 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:15.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.236+0000 7f509b7fe640 1 -- 172.21.15.12:0/653361305 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a0037070 con 0x7f50a409d790 2024-09-11T01:40:15.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.236+0000 7f509b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:15.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.236+0000 7f509b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:40:15.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.236+0000 7f509b7fe640 10 monclient: dump: 2024-09-11T01:40:15.246 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:15.246 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:15.246 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:15.246 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:15.246 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:15.246 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:15.246 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:15.246 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:15.246 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:15.246 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:15.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.236+0000 7f509b7fe640 10 monclient: _finish_auth 0 2024-09-11T01:40:15.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.236+0000 7f509b7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:40:15.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.236+0000 7f509b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:15.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.236+0000 7f509b7fe640 1 -- 172.21.15.12:0/653361305 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50a0032640 con 0x7f50a409d790 2024-09-11T01:40:15.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.236+0000 7f50aad00640 5 monclient: authenticate success, global_id 94359 2024-09-11T01:40:15.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.236+0000 7f50aad00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:40:15.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.236+0000 7f509b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:15.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.236+0000 7f50aad00640 10 monclient: get_monmap_and_config success 2024-09-11T01:40:15.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.236+0000 7f509b7fe640 1 -- 172.21.15.12:0/653361305 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a0032a10 con 0x7f50a409d790 2024-09-11T01:40:15.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.236+0000 7f509b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:15.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.236+0000 7f50aad00640 4 set_mon_vals no callback set 2024-09-11T01:40:15.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.236+0000 7f509b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:40:15.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.236+0000 7f509b7fe640 10 monclient: dump: 2024-09-11T01:40:15.247 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:15.247 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:15.248 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:15.248 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:15.248 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:15.248 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:15.248 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:15.248 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:15.248 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:15.248 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:15.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.237+0000 7f50aad00640 10 monclient: shutdown 2024-09-11T01:40:15.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.237+0000 7f50aad00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:15.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.237+0000 7f50aad00640 1 -- 172.21.15.12:0/653361305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50a409d790 msgr2=0x7f50a409db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:15.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.237+0000 7f50aad00640 1 --2- 172.21.15.12:0/653361305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50a409d790 0x7f50a409db70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f50a4074f40 tx=0x7f50a002a0b0 comp rx=0 tx=0).stop 2024-09-11T01:40:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.237+0000 7f50aad00640 1 -- 172.21.15.12:0/653361305 shutdown_connections 2024-09-11T01:40:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.237+0000 7f50aad00640 1 --2- 172.21.15.12:0/653361305 >> v2:172.21.15.71:3300/0 conn(0x7f50a409e500 0x7f50a4135170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.237+0000 7f50aad00640 1 --2- 172.21.15.12:0/653361305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50a409d790 0x7f50a409db70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.237+0000 7f50aad00640 1 -- 172.21.15.12:0/653361305 >> 172.21.15.12:0/653361305 conn(0x7f50a40a2d90 msgr2=0x7f50a40a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.237+0000 7f50aad00640 1 -- 172.21.15.12:0/653361305 shutdown_connections 2024-09-11T01:40:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.237+0000 7f50aad00640 1 -- 172.21.15.12:0/653361305 wait complete. 2024-09-11T01:40:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.237+0000 7f50aad00640 10 monclient: build_initial_monmap 2024-09-11T01:40:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.237+0000 7f50aad00640 10 monclient: monmap: 2024-09-11T01:40:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:40:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:40:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:40:15.249 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:40:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:40:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:40:15.250 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50aad00640 1 Processor -- start 2024-09-11T01:40:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50aad00640 1 -- start start 2024-09-11T01:40:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50aad00640 10 monclient: init 2024-09-11T01:40:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50aad00640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50aad00640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:40:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50aad00640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50a409d790 0x7f50a412bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50aad00640 10 monclient(hunting): picked mon.noname-a con 0x7f50a409d790 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:40:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50aad00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f50a409e0b0 0x7f50a412c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50aad00640 10 monclient(hunting): picked mon.noname-b con 0x7f50a409e0b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:40:15.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50aad00640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f50a409e500 0x7f50a413df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:15.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50aad00640 10 monclient(hunting): picked mon.noname-c con 0x7f50a409e500 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:40:15.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50aad00640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:15.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50aad00640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f50a40a23a0 con 0x7f50a409d790 2024-09-11T01:40:15.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50aad00640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:15.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50aad00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f50a40a2220 con 0x7f50a409e0b0 2024-09-11T01:40:15.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50aad00640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:15.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50aad00640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f50a409cca0 con 0x7f50a409e500 2024-09-11T01:40:15.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50aad00640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:15.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50aad00640 10 monclient(hunting): authenticate will time out at 1421.681772s 2024-09-11T01:40:15.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50a8a9c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50a409d790 0x7f50a412bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:15.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50a929d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f50a409e500 0x7f50a413df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:15.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50a8a9c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50a409d790 0x7f50a412bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48258/0 (socket says 172.21.15.12:48258) 2024-09-11T01:40:15.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50a8a9c640 1 -- 172.21.15.12:0/2233839435 learned_addr learned my addr 172.21.15.12:0/2233839435 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:40:15.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50a8a9c640 10 monclient(hunting): get_auth_request con 0x7f50a409d790 auth_method 0 2024-09-11T01:40:15.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50a8a9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:15.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50a8a9c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:15.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50a8a9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:15.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50a929d640 10 monclient(hunting): get_auth_request con 0x7f50a409e500 auth_method 0 2024-09-11T01:40:15.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50a929d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:15.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50a929d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:15.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50a929d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:15.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50a8a9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:15.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.238+0000 7f50a8a9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:15.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.239+0000 7f50a8a9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:15.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.239+0000 7f509bfff640 1 --2- 172.21.15.12:0/2233839435 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f50a409e0b0 0x7f50a412c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:15.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.239+0000 7f50a929d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:15.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.239+0000 7f50a929d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:15.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.239+0000 7f50a929d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.239+0000 7f509bfff640 10 monclient(hunting): get_auth_request con 0x7f50a409e0b0 auth_method 0 2024-09-11T01:40:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.239+0000 7f509bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.239+0000 7f509bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.239+0000 7f509bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.239+0000 7f509bfff640 1 -- 172.21.15.12:0/2233839435 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f50a409e0b0 msgr2=0x7f50a412c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:40:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.239+0000 7f509bfff640 1 -- 172.21.15.12:0/2233839435 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f50a409e0b0 msgr2=0x7f50a412c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:40:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.239+0000 7f509bfff640 1 --2- 172.21.15.12:0/2233839435 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f50a409e0b0 0x7f50a412c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:40:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.239+0000 7f509bfff640 1 --2- 172.21.15.12:0/2233839435 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f50a409e0b0 0x7f50a412c2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:40:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.239+0000 7f50a8a9c640 10 monclient(hunting): handle_auth_done global_id 94362 payload 824 2024-09-11T01:40:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.239+0000 7f50a8a9c640 1 -- 172.21.15.12:0/2233839435 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f50a409e500 msgr2=0x7f50a413df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.239+0000 7f50a8a9c640 1 --2- 172.21.15.12:0/2233839435 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f50a409e500 0x7f50a413df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.239+0000 7f50a8a9c640 1 -- 172.21.15.12:0/2233839435 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f50a409e0b0 msgr2=0x7f50a412c2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:40:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.239+0000 7f50a8a9c640 1 --2- 172.21.15.12:0/2233839435 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f50a409e0b0 0x7f50a412c2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:15.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.239+0000 7f50a8a9c640 10 monclient: _finish_hunting 0 2024-09-11T01:40:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.239+0000 7f50a8a9c640 1 monclient: found mon.noname-a 2024-09-11T01:40:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.239+0000 7f50a8a9c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.239+0000 7f50a8a9c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:40:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.239+0000 7f50a8a9c640 1 -- 172.21.15.12:0/2233839435 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50a413f010 con 0x7f50a409d790 2024-09-11T01:40:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.239+0000 7f50a929d640 1 --2- 172.21.15.12:0/2233839435 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f50a409e500 0x7f50a413df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:40:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.241+0000 7f50a8a9c640 1 -- 172.21.15.12:0/2233839435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50a409d790 msgr2=0x7f50a412bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.241+0000 7f50a8a9c640 1 --2- 172.21.15.12:0/2233839435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50a409d790 0x7f50a412bd90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f50a00090b0 tx=0x7f50a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.241+0000 7f50997fa640 1 -- 172.21.15.12:0/2233839435 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a0041020 con 0x7f50a409d790 2024-09-11T01:40:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.241+0000 7f50997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.241+0000 7f50997fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:40:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.241+0000 7f50997fa640 10 monclient: dump: 2024-09-11T01:40:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:15.254 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:15.255 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:15.255 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:15.255 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:15.255 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:15.255 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:15.255 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:15.255 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:15.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.241+0000 7f50997fa640 10 monclient: _finish_auth 0 2024-09-11T01:40:15.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.241+0000 7f50997fa640 10 monclient: _check_auth_tickets 2024-09-11T01:40:15.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.241+0000 7f50997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:15.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.241+0000 7f50997fa640 1 -- 172.21.15.12:0/2233839435 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50a003d850 con 0x7f50a409d790 2024-09-11T01:40:15.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.241+0000 7f50aad00640 5 monclient: authenticate success, global_id 94362 2024-09-11T01:40:15.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.241+0000 7f50997fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:15.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.241+0000 7f50aad00640 10 monclient: _renew_subs 2024-09-11T01:40:15.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.241+0000 7f50997fa640 1 -- 172.21.15.12:0/2233839435 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a0037040 con 0x7f50a409d790 2024-09-11T01:40:15.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.241+0000 7f50aad00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:40:15.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.241+0000 7f50aad00640 1 -- 172.21.15.12:0/2233839435 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50a412c810 con 0x7f50a409d790 2024-09-11T01:40:15.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.241+0000 7f50997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:15.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.241+0000 7f50997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:40:15.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.241+0000 7f50997fa640 10 monclient: dump: 2024-09-11T01:40:15.256 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:15.256 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:15.256 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:15.256 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:15.256 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:15.256 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:15.256 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:15.256 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:15.257 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:15.257 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:15.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.241+0000 7f509affd640 4 set_mon_vals no callback set 2024-09-11T01:40:15.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.241+0000 7f50aad00640 10 monclient: _renew_subs 2024-09-11T01:40:15.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.241+0000 7f50aad00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:40:15.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.241+0000 7f50aad00640 1 -- 172.21.15.12:0/2233839435 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50a41364a0 con 0x7f50a409d790 2024-09-11T01:40:15.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.242+0000 7f50997fa640 1 -- 172.21.15.12:0/2233839435 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f50a003a030 con 0x7f50a409d790 2024-09-11T01:40:15.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.242+0000 7f50aad00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:15.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.242+0000 7f50aad00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:15.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.242+0000 7f50aad00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:40:15.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.242+0000 7f50aad00640 1 -- 172.21.15.12:0/2233839435 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50a4135740 con 0x7f50a409d790 2024-09-11T01:40:15.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.243+0000 7f50997fa640 1 --2- 172.21.15.12:0/2233839435 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f509003e260 0x7f5090040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:15.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.243+0000 7f50997fa640 1 -- 172.21.15.12:0/2233839435 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(370..370 src has 285..370) ==== 4799+0+0 (secure 0 0 0) 0x7f50a00732a0 con 0x7f50a409d790 2024-09-11T01:40:15.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.243+0000 7f509bfff640 1 --2- 172.21.15.12:0/2233839435 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f509003e260 0x7f5090040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:15.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.243+0000 7f509bfff640 10 monclient: get_auth_request con 0x7f509003e260 auth_method 0 2024-09-11T01:40:15.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.245+0000 7f509bfff640 1 --2- 172.21.15.12:0/2233839435 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f509003e260 0x7f5090040720 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f50a4136020 tx=0x7f5094009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:15.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.247+0000 7f50997fa640 1 -- 172.21.15.12:0/2233839435 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f50a004b030 con 0x7f50a409d790 2024-09-11T01:40:15.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.247+0000 7f50997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:15.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.247+0000 7f50997fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:40:15.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.504+0000 7f50aad00640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:40:15.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.504+0000 7f50aad00640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:40:15.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.504+0000 7f50aad00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:40:15.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.504+0000 7f50aad00640 1 -- 172.21.15.12:0/2233839435 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f50a413f210 con 0x7f50a409d790 2024-09-11T01:40:15.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.504+0000 7f50997fa640 1 -- 172.21.15.12:0/2233839435 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f50a003a350 con 0x7f50a409d790 2024-09-11T01:40:15.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.504+0000 7f50997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:40:15.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.504+0000 7f50997fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:40:15.507 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":104,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:40:15.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.506+0000 7f50aad00640 1 -- 172.21.15.12:0/2233839435 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f509003e260 msgr2=0x7f5090040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:15.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.506+0000 7f50aad00640 1 --2- 172.21.15.12:0/2233839435 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f509003e260 0x7f5090040720 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f50a4136020 tx=0x7f5094009040 comp rx=0 tx=0).stop 2024-09-11T01:40:15.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.506+0000 7f50aad00640 10 monclient: shutdown 2024-09-11T01:40:15.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.506+0000 7f50aad00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:15.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.506+0000 7f50aad00640 1 -- 172.21.15.12:0/2233839435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50a409d790 msgr2=0x7f50a412bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:15.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.506+0000 7f50aad00640 1 --2- 172.21.15.12:0/2233839435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50a409d790 0x7f50a412bd90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f50a00090b0 tx=0x7f50a00046c0 comp rx=0 tx=0).stop 2024-09-11T01:40:15.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.506+0000 7f50aad00640 1 -- 172.21.15.12:0/2233839435 shutdown_connections 2024-09-11T01:40:15.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.506+0000 7f50aad00640 1 --2- 172.21.15.12:0/2233839435 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f50a409e500 0x7f50a413df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:15.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.506+0000 7f50aad00640 1 --2- 172.21.15.12:0/2233839435 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f50a409e0b0 0x7f50a412c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:15.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.506+0000 7f50aad00640 1 --2- 172.21.15.12:0/2233839435 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f509003e260 0x7f5090040720 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:15.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.506+0000 7f50aad00640 1 --2- 172.21.15.12:0/2233839435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f50a409d790 0x7f50a412bd90 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:15.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.506+0000 7f50aad00640 1 -- 172.21.15.12:0/2233839435 >> 172.21.15.12:0/2233839435 conn(0x7f50a40a2d90 msgr2=0x7f50a40a4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:15.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.506+0000 7f50aad00640 1 -- 172.21.15.12:0/2233839435 shutdown_connections 2024-09-11T01:40:15.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:15.506+0000 7f50aad00640 1 -- 172.21.15.12:0/2233839435 wait complete. 2024-09-11T01:40:15.956 INFO:tasks.rados.rados.0.smithi071.stdout:1626: done (1 left) 2024-09-11T01:40:15.956 INFO:tasks.rados.rados.0.smithi071.stdout:1627: snap_create 2024-09-11T01:40:15.956 INFO:tasks.rados.rados.0.smithi071.stdout:1622: finishing write tid 1 to smithi07134777-31 2024-09-11T01:40:15.956 INFO:tasks.rados.rados.0.smithi071.stdout:1622: finishing write tid 2 to smithi07134777-31 2024-09-11T01:40:15.956 INFO:tasks.rados.rados.0.smithi071.stdout:1622: finishing write tid 3 to smithi07134777-31 2024-09-11T01:40:15.956 INFO:tasks.rados.rados.0.smithi071.stdout:1622: finishing write tid 4 to smithi07134777-31 2024-09-11T01:40:15.956 INFO:tasks.rados.rados.0.smithi071.stdout:1622: oid 31 updating version 0 to 794 2024-09-11T01:40:15.956 INFO:tasks.rados.rados.0.smithi071.stdout:1622: oid 31 version 794 is already newer than 793 2024-09-11T01:40:15.956 INFO:tasks.rados.rados.0.smithi071.stdout:1622: oid 31 version 794 is already newer than 792 2024-09-11T01:40:15.956 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 794 (ObjNum 606 snap 175 seq_num 606) dirty exists 2024-09-11T01:40:15.956 INFO:tasks.rados.rados.0.smithi071.stdout:1622: left oid 31 (ObjNum 606 snap 175 seq_num 606) 2024-09-11T01:40:16.960 INFO:tasks.rados.rados.0.smithi071.stdout:1622: done (1 left) 2024-09-11T01:40:16.960 INFO:tasks.rados.rados.0.smithi071.stdout:1627: done (0 left) 2024-09-11T01:40:16.960 INFO:tasks.rados.rados.0.smithi071.stdout:1628: write (excl) oid 31 current snap is 176 2024-09-11T01:40:16.960 INFO:tasks.rados.rados.0.smithi071.stdout:1628: seq_num 608 ranges {618656=543279,1812346=406310} 2024-09-11T01:40:16.966 INFO:tasks.rados.rados.0.smithi071.stdout:1628: writing smithi07134777-31 from 618656 to 1161935 tid 1 2024-09-11T01:40:16.972 INFO:tasks.rados.rados.0.smithi071.stdout:1628: writing smithi07134777-31 from 1812346 to 2218656 tid 2 2024-09-11T01:40:16.973 INFO:tasks.rados.rados.0.smithi071.stdout:1629: rollback oid 47 current snap is 176 2024-09-11T01:40:16.973 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 47 to 173 2024-09-11T01:40:16.973 INFO:tasks.rados.rados.0.smithi071.stdout:1630: write (excl) oid 43 current snap is 176 2024-09-11T01:40:16.973 INFO:tasks.rados.rados.0.smithi071.stdout:1630: seq_num 609 ranges {635463=734881,1987181=248282} 2024-09-11T01:40:16.980 INFO:tasks.rados.rados.0.smithi071.stdout:1630: writing smithi07134777-43 from 635463 to 1370344 tid 1 2024-09-11T01:40:16.986 INFO:tasks.rados.rados.0.smithi071.stdout:1630: writing smithi07134777-43 from 1987181 to 2235463 tid 2 2024-09-11T01:40:16.986 INFO:tasks.rados.rados.0.smithi071.stdout:1631: copy_from oid 38 from oid 37 current snap is 176 2024-09-11T01:40:16.986 INFO:tasks.rados.rados.0.smithi071.stdout:1628: finishing write tid 1 to smithi07134777-31 2024-09-11T01:40:16.986 INFO:tasks.rados.rados.0.smithi071.stdout:1628: finishing write tid 2 to smithi07134777-31 2024-09-11T01:40:16.986 INFO:tasks.rados.rados.0.smithi071.stdout:1628: finishing write tid 3 to smithi07134777-31 2024-09-11T01:40:16.986 INFO:tasks.rados.rados.0.smithi071.stdout:1628: finishing write tid 4 to smithi07134777-31 2024-09-11T01:40:16.987 INFO:tasks.rados.rados.0.smithi071.stdout:1628: oid 31 updating version 0 to 796 2024-09-11T01:40:16.987 INFO:tasks.rados.rados.0.smithi071.stdout:1628: oid 31 updating version 796 to 797 2024-09-11T01:40:16.987 INFO:tasks.rados.rados.0.smithi071.stdout:1628: oid 31 version 797 is already newer than 795 2024-09-11T01:40:16.987 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 797 (ObjNum 608 snap 176 seq_num 608) dirty exists 2024-09-11T01:40:16.987 INFO:tasks.rados.rados.0.smithi071.stdout:1628: left oid 31 (ObjNum 608 snap 176 seq_num 608) 2024-09-11T01:40:16.987 INFO:tasks.rados.rados.0.smithi071.stdout:1629: finishing rollback tid 0 to smithi07134777-47 2024-09-11T01:40:16.987 INFO:tasks.rados.rados.0.smithi071.stdout:1629: finishing rollback tid 1 to smithi07134777-47 2024-09-11T01:40:16.987 INFO:tasks.rados.rados.0.smithi071.stdout:1629: finishing rollback tid 2 to smithi07134777-47 2024-09-11T01:40:16.987 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 800 (ObjNum 599 snap 171 seq_num 599) dirty exists 2024-09-11T01:40:16.987 INFO:tasks.rados.rados.0.smithi071.stdout:1630: finishing write tid 1 to smithi07134777-43 2024-09-11T01:40:16.987 INFO:tasks.rados.rados.0.smithi071.stdout:1628: done (3 left) 2024-09-11T01:40:16.987 INFO:tasks.rados.rados.0.smithi071.stdout:1629: done (2 left) 2024-09-11T01:40:16.987 INFO:tasks.rados.rados.0.smithi071.stdout:1632: read oid 18 snap -1 2024-09-11T01:40:16.987 INFO:tasks.rados.rados.0.smithi071.stdout:1632: expect (ObjNum 519 snap 143 seq_num 519) 2024-09-11T01:40:16.988 INFO:tasks.rados.rados.0.smithi071.stdout:1633: read oid 1 snap -1 2024-09-11T01:40:16.988 INFO:tasks.rados.rados.0.smithi071.stdout:1633: expect (ObjNum 604 snap 173 seq_num 604) 2024-09-11T01:40:16.988 INFO:tasks.rados.rados.0.smithi071.stdout:1634: read oid 35 snap -1 2024-09-11T01:40:16.988 INFO:tasks.rados.rados.0.smithi071.stdout:1634: expect (ObjNum 591 snap 169 seq_num 591) 2024-09-11T01:40:16.988 INFO:tasks.rados.rados.0.smithi071.stdout:1635: rollback oid 47 current snap is 176 2024-09-11T01:40:16.988 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 47 to 173 2024-09-11T01:40:16.988 INFO:tasks.rados.rados.0.smithi071.stdout:1636: copy_from oid 8 from oid 49 current snap is 176 2024-09-11T01:40:16.988 INFO:tasks.rados.rados.0.smithi071.stdout:1637: snap_remove snap 167 2024-09-11T01:40:17.966 INFO:tasks.rados.rados.0.smithi071.stdout:1637: done (7 left) 2024-09-11T01:40:17.966 INFO:tasks.rados.rados.0.smithi071.stdout:1638: write oid 2 current snap is 176 2024-09-11T01:40:17.966 INFO:tasks.rados.rados.0.smithi071.stdout:1638: seq_num 610 ranges {652270=410130,1529310=493867} 2024-09-11T01:40:17.972 INFO:tasks.rados.rados.0.smithi071.stdout:1638: writing smithi07134777-2 from 652270 to 1062400 tid 1 2024-09-11T01:40:17.978 INFO:tasks.rados.rados.0.smithi071.stdout:1638: writing smithi07134777-2 from 1529310 to 2023177 tid 2 2024-09-11T01:40:17.978 INFO:tasks.rados.rados.0.smithi071.stdout:1639: read oid 48 snap -1 2024-09-11T01:40:17.978 INFO:tasks.rados.rados.0.smithi071.stdout:1639: expect (ObjNum 586 snap 168 seq_num 586) 2024-09-11T01:40:17.979 INFO:tasks.rados.rados.0.smithi071.stdout:1640: copy_from oid 17 from oid 13 current snap is 176 2024-09-11T01:40:17.979 INFO:tasks.rados.rados.0.smithi071.stdout:1636: finishing copy_from to smithi07134777-8 2024-09-11T01:40:17.979 INFO:tasks.rados.rados.0.smithi071.stdout:1636: got expected ENOENT (src dne) 2024-09-11T01:40:17.979 INFO:tasks.rados.rados.0.smithi071.stdout:1636: finishing copy_from racing read to smithi07134777-8 2024-09-11T01:40:17.979 INFO:tasks.rados.rados.0.smithi071.stdout:1630: finishing write tid 2 to smithi07134777-43 2024-09-11T01:40:17.979 INFO:tasks.rados.rados.0.smithi071.stdout:1630: finishing write tid 3 to smithi07134777-43 2024-09-11T01:40:17.979 INFO:tasks.rados.rados.0.smithi071.stdout:1630: finishing write tid 4 to smithi07134777-43 2024-09-11T01:40:17.979 INFO:tasks.rados.rados.0.smithi071.stdout:1630: oid 43 updating version 0 to 847 2024-09-11T01:40:17.979 INFO:tasks.rados.rados.0.smithi071.stdout:1630: oid 43 version 847 is already newer than 845 2024-09-11T01:40:17.979 INFO:tasks.rados.rados.0.smithi071.stdout:1630: oid 43 version 847 is already newer than 846 2024-09-11T01:40:17.979 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 847 (ObjNum 609 snap 176 seq_num 609) dirty exists 2024-09-11T01:40:17.979 INFO:tasks.rados.rados.0.smithi071.stdout:1630: left oid 43 (ObjNum 609 snap 176 seq_num 609) 2024-09-11T01:40:17.979 INFO:tasks.rados.rados.0.smithi071.stdout:1633: expect (ObjNum 604 snap 173 seq_num 604) 2024-09-11T01:40:17.979 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:40:18.062 INFO:tasks.rados.rados.0.smithi071.stdout:1635: finishing rollback tid 0 to smithi07134777-47 2024-09-11T01:40:18.062 INFO:tasks.rados.rados.0.smithi071.stdout:1635: finishing rollback tid 1 to smithi07134777-47 2024-09-11T01:40:18.062 INFO:tasks.rados.rados.0.smithi071.stdout:1630: done (9 left) 2024-09-11T01:40:18.062 INFO:tasks.rados.rados.0.smithi071.stdout:1633: done (8 left) 2024-09-11T01:40:18.062 INFO:tasks.rados.rados.0.smithi071.stdout:1636: done (7 left) 2024-09-11T01:40:18.062 INFO:tasks.rados.rados.0.smithi071.stdout:1641: snap_create 2024-09-11T01:40:18.062 INFO:tasks.rados.rados.0.smithi071.stdout:1635: finishing rollback tid 2 to smithi07134777-47 2024-09-11T01:40:18.063 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 803 (ObjNum 599 snap 171 seq_num 599) dirty exists 2024-09-11T01:40:18.063 INFO:tasks.rados.rados.0.smithi071.stdout:1632: expect (ObjNum 519 snap 143 seq_num 519) 2024-09-11T01:40:18.072 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2595484 2024-09-11T01:40:18.106 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 49861 2024-09-11T01:40:18.135 INFO:tasks.rados.rados.0.smithi071.stdout:1634: expect (ObjNum 591 snap 169 seq_num 591) 2024-09-11T01:40:18.207 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:40:18.250 INFO:tasks.rados.rados.0.smithi071.stdout:1631: finishing copy_from racing read to smithi07134777-38 2024-09-11T01:40:18.250 INFO:tasks.rados.rados.0.smithi071.stdout:1631: finishing copy_from to smithi07134777-38 2024-09-11T01:40:18.250 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 604 (ObjNum 577 snap 165 seq_num 577) dirty exists 2024-09-11T01:40:18.250 INFO:tasks.rados.rados.0.smithi071.stdout:1638: finishing write tid 1 to smithi07134777-2 2024-09-11T01:40:18.250 INFO:tasks.rados.rados.0.smithi071.stdout:1638: finishing write tid 2 to smithi07134777-2 2024-09-11T01:40:18.250 INFO:tasks.rados.rados.0.smithi071.stdout:1638: finishing write tid 3 to smithi07134777-2 2024-09-11T01:40:18.250 INFO:tasks.rados.rados.0.smithi071.stdout:1638: finishing write tid 4 to smithi07134777-2 2024-09-11T01:40:18.251 INFO:tasks.rados.rados.0.smithi071.stdout:1638: oid 2 updating version 0 to 448 2024-09-11T01:40:18.251 INFO:tasks.rados.rados.0.smithi071.stdout:1638: oid 2 updating version 448 to 450 2024-09-11T01:40:18.251 INFO:tasks.rados.rados.0.smithi071.stdout:1638: oid 2 version 450 is already newer than 449 2024-09-11T01:40:18.251 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 450 (ObjNum 610 snap 176 seq_num 610) dirty exists 2024-09-11T01:40:18.251 INFO:tasks.rados.rados.0.smithi071.stdout:1638: left oid 2 (ObjNum 610 snap 176 seq_num 610) 2024-09-11T01:40:18.251 INFO:tasks.rados.rados.0.smithi071.stdout:1639: expect (ObjNum 586 snap 168 seq_num 586) 2024-09-11T01:40:18.251 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:40:18.329 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3004023 2024-09-11T01:40:18.394 INFO:tasks.rados.rados.0.smithi071.stdout:1640: finishing copy_from racing read to smithi07134777-17 2024-09-11T01:40:18.394 INFO:tasks.rados.rados.0.smithi071.stdout:1640: finishing copy_from to smithi07134777-17 2024-09-11T01:40:18.395 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 784 (ObjNum 551 snap 157 seq_num 551) dirty exists 2024-09-11T01:40:18.521 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:40:18.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.632+0000 7f11f112c640 10 monclient: get_monmap_and_config 2024-09-11T01:40:18.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.632+0000 7f11f112c640 10 monclient: build_initial_monmap 2024-09-11T01:40:18.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.632+0000 7f11f112c640 10 monclient: monmap: 2024-09-11T01:40:18.634 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:18.634 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:18.634 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:40:18.633394+0000 2024-09-11T01:40:18.634 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:40:18.633394+0000 2024-09-11T01:40:18.634 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:18.634 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:18.634 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:40:18.635 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:40:18.635 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:40:18.635 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:40:18.635 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:40:18.635 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:18.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.632+0000 7f11f112c640 1 Processor -- start 2024-09-11T01:40:18.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.632+0000 7f11f112c640 1 -- start start 2024-09-11T01:40:18.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.632+0000 7f11f112c640 10 monclient: init 2024-09-11T01:40:18.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.632+0000 7f11f112c640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:18.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.632+0000 7f11f112c640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-11T01:40:18.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.632+0000 7f11f112c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11ec0a2e90 0x7f11ec06dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:18.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.632+0000 7f11f112c640 10 monclient(hunting): picked mon.noname-b con 0x7f11ec0a2e90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:40:18.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.632+0000 7f11f112c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f11ec0a3270 addr v1:172.21.15.71:6789/0 2024-09-11T01:40:18.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.632+0000 7f11f112c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f11ec13d490 0x7f11ec13f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:18.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.632+0000 7f11f112c640 10 monclient(hunting): picked mon.noname-a con 0x7f11ec13d490 addr v2:172.21.15.12:3300/0 2024-09-11T01:40:18.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11f112c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11ec0a46c0 con 0x7f11ec0a3270 2024-09-11T01:40:18.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11f112c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:18.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11f112c640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f11ec0a4900 con 0x7f11ec13d490 2024-09-11T01:40:18.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11f112c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:18.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11f112c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f11ec06ee50 con 0x7f11ec0a2e90 2024-09-11T01:40:18.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11f112c640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:18.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11f112c640 10 monclient(hunting): authenticate will time out at 1425.076043s 2024-09-11T01:40:18.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11eb59e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f11ec13d490 0x7f11ec13f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:18.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11eb59e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f11ec13d490 0x7f11ec13f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60868/0 (socket says 172.21.15.12:60868) 2024-09-11T01:40:18.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11eb59e640 1 -- 172.21.15.12:0/4054082815 learned_addr learned my addr 172.21.15.12:0/4054082815 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:40:18.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11eb59e640 10 monclient(hunting): get_auth_request con 0x7f11ec13d490 auth_method 0 2024-09-11T01:40:18.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11eb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:18.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11eb59e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:18.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11eb59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:18.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11ead9d640 1 --2- 172.21.15.12:0/4054082815 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11ec0a2e90 0x7f11ec06dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:18.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11ead9d640 10 monclient(hunting): get_auth_request con 0x7f11ec0a2e90 auth_method 0 2024-09-11T01:40:18.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:18.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11ead9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:18.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11ead9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:18.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11eb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:18.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11eb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:18.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11eb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:18.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11ead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:18.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11ead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:18.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11ead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:18.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11eb59e640 10 monclient(hunting): handle_auth_done global_id 94365 payload 274 2024-09-11T01:40:18.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11eb59e640 1 -- 172.21.15.12:0/4054082815 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11ec0a2e90 msgr2=0x7f11ec06dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:40:18.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11eb59e640 1 --2- 172.21.15.12:0/4054082815 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11ec0a2e90 0x7f11ec06dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:18.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11eb59e640 1 -- 172.21.15.12:0/4054082815 >> v1:172.21.15.71:6789/0 conn(0x7f11ec0a3270 legacy=0x7f11ec0a3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:40:18.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11eb59e640 10 monclient: _finish_hunting 0 2024-09-11T01:40:18.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11eb59e640 1 monclient: found mon.noname-a 2024-09-11T01:40:18.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11eb59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:18.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11eb59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:40:18.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.633+0000 7f11eb59e640 1 -- 172.21.15.12:0/4054082815 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11ec06eb70 con 0x7f11ec13d490 2024-09-11T01:40:18.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11eb59e640 1 -- 172.21.15.12:0/4054082815 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11ec13d490 msgr2=0x7f11ec13f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:18.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11eb59e640 1 --2- 172.21.15.12:0/4054082815 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11ec13d490 0x7f11ec13f880 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f11ec0a4a80 tx=0x7f11e402a540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d644be1f1d2c3e3c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:18.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11ea59c640 1 -- 172.21.15.12:0/4054082815 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e4036070 con 0x7f11ec13d490 2024-09-11T01:40:18.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11ea59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:18.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11ea59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:40:18.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11ea59c640 10 monclient: dump: 2024-09-11T01:40:18.646 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:18.646 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:18.646 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:18.646 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:18.647 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:18.647 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:18.647 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:18.647 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:18.647 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:18.647 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:18.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11ea59c640 10 monclient: _finish_auth 0 2024-09-11T01:40:18.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11ea59c640 10 monclient: _check_auth_tickets 2024-09-11T01:40:18.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11ea59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:18.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11f112c640 5 monclient: authenticate success, global_id 94365 2024-09-11T01:40:18.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11ea59c640 1 -- 172.21.15.12:0/4054082815 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11e4007800 con 0x7f11ec13d490 2024-09-11T01:40:18.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11f112c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:40:18.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11ea59c640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:18.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11f112c640 10 monclient: get_monmap_and_config success 2024-09-11T01:40:18.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11ea59c640 1 -- 172.21.15.12:0/4054082815 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e4007bd0 con 0x7f11ec13d490 2024-09-11T01:40:18.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11f112c640 4 set_mon_vals no callback set 2024-09-11T01:40:18.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11ea59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:18.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11ea59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:40:18.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11ea59c640 10 monclient: dump: 2024-09-11T01:40:18.648 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:18.648 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:18.649 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:18.649 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:18.649 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:18.649 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:18.649 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:18.649 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:18.649 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:18.649 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:18.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11f112c640 10 monclient: shutdown 2024-09-11T01:40:18.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11f112c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:18.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11f112c640 1 -- 172.21.15.12:0/4054082815 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11ec13d490 msgr2=0x7f11ec13f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:18.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11f112c640 1 --2- 172.21.15.12:0/4054082815 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11ec13d490 0x7f11ec13f880 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f11ec0a4a80 tx=0x7f11e402a540 comp rx=0 tx=0).stop 2024-09-11T01:40:18.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11f112c640 1 -- 172.21.15.12:0/4054082815 shutdown_connections 2024-09-11T01:40:18.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11f112c640 1 --2- 172.21.15.12:0/4054082815 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11ec13d490 0x7f11ec13f880 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:18.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11f112c640 1 --2- 172.21.15.12:0/4054082815 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11ec0a2e90 0x7f11ec06dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:18.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11f112c640 1 -- 172.21.15.12:0/4054082815 >> 172.21.15.12:0/4054082815 conn(0x7f11ec0a8490 msgr2=0x7f11ec0a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:18.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11f112c640 1 -- 172.21.15.12:0/4054082815 shutdown_connections 2024-09-11T01:40:18.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11f112c640 1 -- 172.21.15.12:0/4054082815 wait complete. 2024-09-11T01:40:18.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11f112c640 10 monclient: build_initial_monmap 2024-09-11T01:40:18.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.637+0000 7f11f112c640 10 monclient: monmap: 2024-09-11T01:40:18.650 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:18.650 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:40:18.650 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:40:18.650 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:40:18.650 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:18.650 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:18.650 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:40:18.650 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:40:18.651 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:40:18.651 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:18.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.638+0000 7f11f112c640 1 Processor -- start 2024-09-11T01:40:18.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.638+0000 7f11f112c640 1 -- start start 2024-09-11T01:40:18.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.638+0000 7f11f112c640 10 monclient: init 2024-09-11T01:40:18.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.638+0000 7f11f112c640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:18.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.638+0000 7f11f112c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:40:18.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.638+0000 7f11f112c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11ec0a2e90 0x7f11ec13b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:18.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.638+0000 7f11f112c640 10 monclient(hunting): picked mon.noname-a con 0x7f11ec0a2e90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:40:18.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.638+0000 7f11f112c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f11ec0a3270 0x7f11ec09f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:18.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.638+0000 7f11f112c640 10 monclient(hunting): picked mon.noname-b con 0x7f11ec0a3270 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:40:18.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.638+0000 7f11f112c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11ec13d490 0x7f11ec09fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:18.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.638+0000 7f11f112c640 10 monclient(hunting): picked mon.noname-c con 0x7f11ec13d490 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:40:18.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.638+0000 7f11f112c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:18.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.638+0000 7f11f112c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec06fd20 con 0x7f11ec0a2e90 2024-09-11T01:40:18.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.638+0000 7f11f112c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:18.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.638+0000 7f11f112c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec06fba0 con 0x7f11ec0a3270 2024-09-11T01:40:18.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.638+0000 7f11f112c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:18.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.638+0000 7f11f112c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f11ec13fdc0 con 0x7f11ec13d490 2024-09-11T01:40:18.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.638+0000 7f11f112c640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:18.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.638+0000 7f11f112c640 10 monclient(hunting): authenticate will time out at 1425.081914s 2024-09-11T01:40:18.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.638+0000 7f11ead9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11ec0a2e90 0x7f11ec13b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:18.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11ead9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11ec0a2e90 0x7f11ec13b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60878/0 (socket says 172.21.15.12:60878) 2024-09-11T01:40:18.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11eb59e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11ec13d490 0x7f11ec09fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:18.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11ead9d640 1 -- 172.21.15.12:0/2361781809 learned_addr learned my addr 172.21.15.12:0/2361781809 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:40:18.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11ead9d640 10 monclient(hunting): get_auth_request con 0x7f11ec0a2e90 auth_method 0 2024-09-11T01:40:18.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:18.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11ead9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:18.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11ead9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:18.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11eb59e640 10 monclient(hunting): get_auth_request con 0x7f11ec13d490 auth_method 0 2024-09-11T01:40:18.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11eb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:18.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11eb59e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:18.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11eb59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:18.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11ead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:18.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11ead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:18.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11ead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:18.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11eb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:18.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11eb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:18.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11eb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:18.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11e259c640 1 --2- 172.21.15.12:0/2361781809 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f11ec0a3270 0x7f11ec09f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:18.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11e259c640 10 monclient(hunting): get_auth_request con 0x7f11ec0a3270 auth_method 0 2024-09-11T01:40:18.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:18.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11e259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:18.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11e259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:18.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11eb59e640 10 monclient(hunting): handle_auth_done global_id 114197 payload 824 2024-09-11T01:40:18.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11eb59e640 1 -- 172.21.15.12:0/2361781809 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f11ec0a3270 msgr2=0x7f11ec09f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:18.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11eb59e640 1 --2- 172.21.15.12:0/2361781809 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f11ec0a3270 0x7f11ec09f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:18.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11eb59e640 1 -- 172.21.15.12:0/2361781809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11ec0a2e90 msgr2=0x7f11ec13b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:18.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11eb59e640 1 --2- 172.21.15.12:0/2361781809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11ec0a2e90 0x7f11ec13b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:18.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11eb59e640 10 monclient: _finish_hunting 0 2024-09-11T01:40:18.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11eb59e640 1 monclient: found mon.noname-c 2024-09-11T01:40:18.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11eb59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:18.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11eb59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:40:18.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11eb59e640 1 -- 172.21.15.12:0/2361781809 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11ec0a65d0 con 0x7f11ec13d490 2024-09-11T01:40:18.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.639+0000 7f11ead9d640 1 --2- 172.21.15.12:0/2361781809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11ec0a2e90 0x7f11ec13b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:40:18.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.641+0000 7f11eb59e640 1 -- 172.21.15.12:0/2361781809 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11ec13d490 msgr2=0x7f11ec09fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:18.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.641+0000 7f11eb59e640 1 --2- 172.21.15.12:0/2361781809 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11ec13d490 0x7f11ec09fbc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f11e4033000 tx=0x7f11e4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:18.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.641+0000 7f11e3fff640 1 -- 172.21.15.12:0/2361781809 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e4041020 con 0x7f11ec13d490 2024-09-11T01:40:18.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.641+0000 7f11e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:18.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.641+0000 7f11e3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:40:18.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.641+0000 7f11e3fff640 10 monclient: dump: 2024-09-11T01:40:18.655 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:18.655 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:18.655 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:18.656 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:18.656 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:18.656 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:18.656 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:18.656 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:18.656 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:18.656 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:18.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.641+0000 7f11e3fff640 10 monclient: _finish_auth 0 2024-09-11T01:40:18.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.641+0000 7f11e3fff640 10 monclient: _check_auth_tickets 2024-09-11T01:40:18.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.641+0000 7f11e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:18.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.641+0000 7f11e3fff640 1 -- 172.21.15.12:0/2361781809 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11e4032950 con 0x7f11ec13d490 2024-09-11T01:40:18.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.641+0000 7f11e3fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:18.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.641+0000 7f11e3fff640 1 -- 172.21.15.12:0/2361781809 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e4036040 con 0x7f11ec13d490 2024-09-11T01:40:18.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.641+0000 7f11e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:18.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.641+0000 7f11e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:40:18.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.641+0000 7f11e3fff640 10 monclient: dump: 2024-09-11T01:40:18.657 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:18.657 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:18.657 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:18.657 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:18.657 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:18.657 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:18.657 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:18.657 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:18.657 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:18.657 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:18.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.641+0000 7f11f112c640 5 monclient: authenticate success, global_id 114197 2024-09-11T01:40:18.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.641+0000 7f11f112c640 10 monclient: _renew_subs 2024-09-11T01:40:18.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.641+0000 7f11f112c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:40:18.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.641+0000 7f11f112c640 1 -- 172.21.15.12:0/2361781809 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11ec0a59e0 con 0x7f11ec13d490 2024-09-11T01:40:18.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.641+0000 7f11e9d9b640 4 set_mon_vals no callback set 2024-09-11T01:40:18.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.642+0000 7f11f112c640 10 monclient: _renew_subs 2024-09-11T01:40:18.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.642+0000 7f11f112c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:40:18.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.642+0000 7f11f112c640 1 -- 172.21.15.12:0/2361781809 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f11ec0a2720 con 0x7f11ec13d490 2024-09-11T01:40:18.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.642+0000 7f11e3fff640 1 -- 172.21.15.12:0/2361781809 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f11e403c470 con 0x7f11ec13d490 2024-09-11T01:40:18.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.643+0000 7f11f112c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:18.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.643+0000 7f11f112c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:18.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.643+0000 7f11f112c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:40:18.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.643+0000 7f11f112c640 1 -- 172.21.15.12:0/2361781809 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11ec06dc60 con 0x7f11ec13d490 2024-09-11T01:40:18.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.643+0000 7f11e3fff640 1 --2- 172.21.15.12:0/2361781809 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f11cc03e260 0x7f11cc040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:18.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.643+0000 7f11e3fff640 1 -- 172.21.15.12:0/2361781809 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(373..373 src has 285..373) ==== 4799+0+0 (secure 0 0 0) 0x7f11e4073a20 con 0x7f11ec13d490 2024-09-11T01:40:18.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.643+0000 7f11ead9d640 1 --2- 172.21.15.12:0/2361781809 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f11cc03e260 0x7f11cc040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:18.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.643+0000 7f11ead9d640 10 monclient: get_auth_request con 0x7f11cc03e260 auth_method 0 2024-09-11T01:40:18.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.645+0000 7f11ead9d640 1 --2- 172.21.15.12:0/2361781809 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f11cc03e260 0x7f11cc040720 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f11dc009f70 tx=0x7f11dc009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:18.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.648+0000 7f11e3fff640 1 -- 172.21.15.12:0/2361781809 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f11e404b070 con 0x7f11ec13d490 2024-09-11T01:40:18.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.648+0000 7f11e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:18.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.648+0000 7f11e3fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:40:18.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.904+0000 7f11f112c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:40:18.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.904+0000 7f11f112c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:40:18.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.904+0000 7f11f112c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:40:18.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.904+0000 7f11f112c640 1 -- 172.21.15.12:0/2361781809 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f11ec0a0f20 con 0x7f11ec13d490 2024-09-11T01:40:18.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.904+0000 7f11e3fff640 1 -- 172.21.15.12:0/2361781809 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f11e4072410 con 0x7f11ec13d490 2024-09-11T01:40:18.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.905+0000 7f11e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:40:18.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.905+0000 7f11e3fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:40:18.908 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":104,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:40:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.907+0000 7f11f112c640 1 -- 172.21.15.12:0/2361781809 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f11cc03e260 msgr2=0x7f11cc040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.907+0000 7f11f112c640 1 --2- 172.21.15.12:0/2361781809 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f11cc03e260 0x7f11cc040720 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f11dc009f70 tx=0x7f11dc009190 comp rx=0 tx=0).stop 2024-09-11T01:40:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.907+0000 7f11f112c640 10 monclient: shutdown 2024-09-11T01:40:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.907+0000 7f11f112c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.907+0000 7f11f112c640 1 -- 172.21.15.12:0/2361781809 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11ec13d490 msgr2=0x7f11ec09fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.907+0000 7f11f112c640 1 --2- 172.21.15.12:0/2361781809 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11ec13d490 0x7f11ec09fbc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f11e4033000 tx=0x7f11e4004a40 comp rx=0 tx=0).stop 2024-09-11T01:40:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.907+0000 7f11f112c640 1 -- 172.21.15.12:0/2361781809 shutdown_connections 2024-09-11T01:40:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.907+0000 7f11f112c640 1 --2- 172.21.15.12:0/2361781809 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11ec13d490 0x7f11ec09fbc0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.907+0000 7f11f112c640 1 --2- 172.21.15.12:0/2361781809 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f11ec0a3270 0x7f11ec09f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.907+0000 7f11f112c640 1 --2- 172.21.15.12:0/2361781809 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f11cc03e260 0x7f11cc040720 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.907+0000 7f11f112c640 1 --2- 172.21.15.12:0/2361781809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11ec0a2e90 0x7f11ec13b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:18.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.907+0000 7f11f112c640 1 -- 172.21.15.12:0/2361781809 >> 172.21.15.12:0/2361781809 conn(0x7f11ec0a8490 msgr2=0x7f11ec13c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:18.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.907+0000 7f11f112c640 1 -- 172.21.15.12:0/2361781809 shutdown_connections 2024-09-11T01:40:18.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:18.907+0000 7f11f112c640 1 -- 172.21.15.12:0/2361781809 wait complete. 2024-09-11T01:40:18.945 INFO:tasks.rados.rados.0.smithi071.stdout:1631: done (7 left) 2024-09-11T01:40:18.946 INFO:tasks.rados.rados.0.smithi071.stdout:1632: done (6 left) 2024-09-11T01:40:18.946 INFO:tasks.rados.rados.0.smithi071.stdout:1634: done (5 left) 2024-09-11T01:40:18.946 INFO:tasks.rados.rados.0.smithi071.stdout:1635: done (4 left) 2024-09-11T01:40:18.946 INFO:tasks.rados.rados.0.smithi071.stdout:1638: done (3 left) 2024-09-11T01:40:18.946 INFO:tasks.rados.rados.0.smithi071.stdout:1639: done (2 left) 2024-09-11T01:40:18.946 INFO:tasks.rados.rados.0.smithi071.stdout:1640: done (1 left) 2024-09-11T01:40:18.946 INFO:tasks.rados.rados.0.smithi071.stdout:1641: done (0 left) 2024-09-11T01:40:18.946 INFO:tasks.rados.rados.0.smithi071.stdout:1642: read oid 36 snap -1 2024-09-11T01:40:18.946 INFO:tasks.rados.rados.0.smithi071.stdout:1642: expect deleted 2024-09-11T01:40:18.946 INFO:tasks.rados.rados.0.smithi071.stdout:1643: read oid 28 snap -1 2024-09-11T01:40:18.946 INFO:tasks.rados.rados.0.smithi071.stdout:1643: expect deleted 2024-09-11T01:40:18.946 INFO:tasks.rados.rados.0.smithi071.stdout:1644: snap_create 2024-09-11T01:40:19.971 INFO:tasks.rados.rados.0.smithi071.stdout:1642: done (2 left) 2024-09-11T01:40:19.971 INFO:tasks.rados.rados.0.smithi071.stdout:1643: done (1 left) 2024-09-11T01:40:19.971 INFO:tasks.rados.rados.0.smithi071.stdout:1644: done (0 left) 2024-09-11T01:40:19.971 INFO:tasks.rados.rados.0.smithi071.stdout:1645: rollback oid 36 current snap is 178 2024-09-11T01:40:19.971 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 36 to 175 2024-09-11T01:40:19.971 INFO:tasks.rados.rados.0.smithi071.stdout:1646: snap_create 2024-09-11T01:40:19.972 INFO:tasks.rados.rados.0.smithi071.stdout:1645: finishing rollback tid 1 to smithi07134777-36 2024-09-11T01:40:19.972 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 458 (ObjNum 129 snap 0 seq_num 2371368520) dirty dne 2024-09-11T01:40:20.972 INFO:tasks.rados.rados.0.smithi071.stdout:1645: done (1 left) 2024-09-11T01:40:20.972 INFO:tasks.rados.rados.0.smithi071.stdout:1646: done (0 left) 2024-09-11T01:40:20.972 INFO:tasks.rados.rados.0.smithi071.stdout:1647: read oid 45 snap 171 2024-09-11T01:40:20.972 INFO:tasks.rados.rados.0.smithi071.stdout:1647: expect (ObjNum 437 snap 119 seq_num 437) 2024-09-11T01:40:20.972 INFO:tasks.rados.rados.0.smithi071.stdout:1648: read oid 21 snap -1 2024-09-11T01:40:20.972 INFO:tasks.rados.rados.0.smithi071.stdout:1648: expect (ObjNum 542 snap 154 seq_num 542) 2024-09-11T01:40:20.972 INFO:tasks.rados.rados.0.smithi071.stdout:1649: read oid 39 snap -1 2024-09-11T01:40:20.973 INFO:tasks.rados.rados.0.smithi071.stdout:1649: expect deleted 2024-09-11T01:40:20.973 INFO:tasks.rados.rados.0.smithi071.stdout:1650: rollback oid 15 current snap is 179 2024-09-11T01:40:20.973 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 15 to 176 2024-09-11T01:40:20.973 INFO:tasks.rados.rados.0.smithi071.stdout:1651: snap_create 2024-09-11T01:40:20.977 INFO:tasks.rados.rados.0.smithi071.stdout:1647: expect (ObjNum 437 snap 119 seq_num 437) 2024-09-11T01:40:21.004 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2684468 2024-09-11T01:40:21.034 INFO:tasks.rados.rados.0.smithi071.stdout:1650: finishing rollback tid 0 to smithi07134777-15 2024-09-11T01:40:21.034 INFO:tasks.rados.rados.0.smithi071.stdout:1650: finishing rollback tid 1 to smithi07134777-15 2024-09-11T01:40:21.034 INFO:tasks.rados.rados.0.smithi071.stdout:1648: expect (ObjNum 542 snap 154 seq_num 542) 2024-09-11T01:40:21.200 INFO:tasks.rados.rados.0.smithi071.stdout:1650: finishing rollback tid 2 to smithi07134777-15 2024-09-11T01:40:21.200 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 922 (ObjNum 565 snap 160 seq_num 565) dirty exists 2024-09-11T01:40:21.920 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:40:21.973 INFO:tasks.rados.rados.0.smithi071.stdout:1647: done (4 left) 2024-09-11T01:40:21.973 INFO:tasks.rados.rados.0.smithi071.stdout:1648: done (3 left) 2024-09-11T01:40:21.974 INFO:tasks.rados.rados.0.smithi071.stdout:1649: done (2 left) 2024-09-11T01:40:21.974 INFO:tasks.rados.rados.0.smithi071.stdout:1650: done (1 left) 2024-09-11T01:40:21.974 INFO:tasks.rados.rados.0.smithi071.stdout:1651: done (0 left) 2024-09-11T01:40:21.974 INFO:tasks.rados.rados.0.smithi071.stdout:1652: snap_remove snap 176 2024-09-11T01:40:22.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.035+0000 7f1b7cad6640 10 monclient: get_monmap_and_config 2024-09-11T01:40:22.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.035+0000 7f1b7cad6640 10 monclient: build_initial_monmap 2024-09-11T01:40:22.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.035+0000 7f1b7cad6640 10 monclient: monmap: 2024-09-11T01:40:22.037 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:22.037 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:22.037 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:40:22.036353+0000 2024-09-11T01:40:22.037 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:40:22.036353+0000 2024-09-11T01:40:22.037 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:22.037 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.037 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:40:22.037 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:40:22.038 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:40:22.038 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:40:22.038 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:40:22.038 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.036+0000 7f1b7cad6640 1 Processor -- start 2024-09-11T01:40:22.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.036+0000 7f1b7cad6640 1 -- start start 2024-09-11T01:40:22.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.036+0000 7f1b7cad6640 10 monclient: init 2024-09-11T01:40:22.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.036+0000 7f1b7cad6640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:22.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.036+0000 7f1b7cad6640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-11T01:40:22.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.036+0000 7f1b7cad6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1b780a2db0 addr v1:172.21.15.12:6789/0 2024-09-11T01:40:22.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.036+0000 7f1b7cad6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1b7809fbd0 addr v1:172.21.15.71:6789/0 2024-09-11T01:40:22.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.036+0000 7f1b7cad6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b7813e560 0x7f1b78140950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.036+0000 7f1b7cad6640 10 monclient(hunting): picked mon.noname-b con 0x7f1b7813e560 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:40:22.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.036+0000 7f1b7cad6640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b7806d970 con 0x7f1b780a2db0 2024-09-11T01:40:22.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.036+0000 7f1b7cad6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b7809eea0 con 0x7f1b7809fbd0 2024-09-11T01:40:22.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.036+0000 7f1b7cad6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:22.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.036+0000 7f1b7cad6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1b780a3850 con 0x7f1b7813e560 2024-09-11T01:40:22.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.036+0000 7f1b7cad6640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:22.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.036+0000 7f1b7cad6640 10 monclient(hunting): authenticate will time out at 1428.478967s 2024-09-11T01:40:22.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.037+0000 7f1b7659c640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1b780a2db0 0x7f1b7806cb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43288/0 (socket says 172.21.15.12:43288) 2024-09-11T01:40:22.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.037+0000 7f1b7659c640 1 -- 172.21.15.12:0/1482487813 learned_addr learned my addr 172.21.15.12:0/1482487813 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:40:22.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.037+0000 7f1b76d9d640 1 --2- 172.21.15.12:0/1482487813 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b7813e560 0x7f1b78140950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.037+0000 7f1b76d9d640 10 monclient(hunting): get_auth_request con 0x7f1b7813e560 auth_method 0 2024-09-11T01:40:22.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.037+0000 7f1b76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:22.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.037+0000 7f1b76d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:22.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.037+0000 7f1b76d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:22.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.037+0000 7f1b76d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:22.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.037+0000 7f1b76d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:22.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.037+0000 7f1b76d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:22.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.037+0000 7f1b76d9d640 10 monclient(hunting): handle_auth_done global_id 114200 payload 274 2024-09-11T01:40:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.037+0000 7f1b76d9d640 1 -- 172.21.15.12:0/1482487813 >> v1:172.21.15.71:6789/0 conn(0x7f1b7809fbd0 legacy=0x7f1b7806d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:40:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.037+0000 7f1b76d9d640 1 -- 172.21.15.12:0/1482487813 >> v1:172.21.15.12:6789/0 conn(0x7f1b780a2db0 legacy=0x7f1b7806cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:40:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.037+0000 7f1b76d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:40:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.037+0000 7f1b76d9d640 1 monclient: found mon.noname-b 2024-09-11T01:40:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.037+0000 7f1b76d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.037+0000 7f1b76d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:40:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.037+0000 7f1b76d9d640 1 -- 172.21.15.12:0/1482487813 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b7809f180 con 0x7f1b7813e560 2024-09-11T01:40:22.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.040+0000 7f1b76d9d640 1 -- 172.21.15.12:0/1482487813 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b7813e560 msgr2=0x7f1b78140950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:22.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.040+0000 7f1b76d9d640 1 --2- 172.21.15.12:0/1482487813 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b7813e560 0x7f1b78140950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1b70009dc0 tx=0x7f1b7002a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbccf0bca8711b76 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:22.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.040+0000 7f1b7559a640 1 -- 172.21.15.12:0/1482487813 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b70007a30 con 0x7f1b7813e560 2024-09-11T01:40:22.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.040+0000 7f1b7559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:22.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.040+0000 7f1b7559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:40:22.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.040+0000 7f1b7559a640 10 monclient: dump: 2024-09-11T01:40:22.047 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:22.048 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:22.048 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.048 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.048 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:22.048 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.048 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:22.048 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:22.048 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:22.048 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.040+0000 7f1b7559a640 10 monclient: _finish_auth 0 2024-09-11T01:40:22.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.040+0000 7f1b7559a640 10 monclient: _check_auth_tickets 2024-09-11T01:40:22.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.040+0000 7f1b7559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:22.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.040+0000 7f1b7559a640 1 -- 172.21.15.12:0/1482487813 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b70007bd0 con 0x7f1b7813e560 2024-09-11T01:40:22.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.040+0000 7f1b7559a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:22.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.040+0000 7f1b7559a640 1 -- 172.21.15.12:0/1482487813 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b70031400 con 0x7f1b7813e560 2024-09-11T01:40:22.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.040+0000 7f1b7559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:22.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.040+0000 7f1b7559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:40:22.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.040+0000 7f1b7559a640 10 monclient: dump: 2024-09-11T01:40:22.049 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:22.049 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:22.049 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.049 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.049 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:22.049 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.049 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:22.049 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:22.050 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.040+0000 7f1b7cad6640 5 monclient: authenticate success, global_id 114200 2024-09-11T01:40:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.040+0000 7f1b7cad6640 10 monclient: get_monmap_and_config success 2024-09-11T01:40:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.040+0000 7f1b7cad6640 4 set_mon_vals no callback set 2024-09-11T01:40:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.040+0000 7f1b7cad6640 10 monclient: shutdown 2024-09-11T01:40:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.040+0000 7f1b7cad6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.040+0000 7f1b7cad6640 1 -- 172.21.15.12:0/1482487813 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b7813e560 msgr2=0x7f1b78140950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.040+0000 7f1b7cad6640 1 --2- 172.21.15.12:0/1482487813 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b7813e560 0x7f1b78140950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1b70009dc0 tx=0x7f1b7002a480 comp rx=0 tx=0).stop 2024-09-11T01:40:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.041+0000 7f1b7cad6640 1 -- 172.21.15.12:0/1482487813 shutdown_connections 2024-09-11T01:40:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.041+0000 7f1b7cad6640 1 --2- 172.21.15.12:0/1482487813 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b7813e560 0x7f1b78140950 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.041+0000 7f1b7cad6640 1 -- 172.21.15.12:0/1482487813 >> 172.21.15.12:0/1482487813 conn(0x7f1b780a3ea0 msgr2=0x7f1b780a42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.041+0000 7f1b7cad6640 1 -- 172.21.15.12:0/1482487813 shutdown_connections 2024-09-11T01:40:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.041+0000 7f1b7cad6640 1 -- 172.21.15.12:0/1482487813 wait complete. 2024-09-11T01:40:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.041+0000 7f1b7cad6640 10 monclient: build_initial_monmap 2024-09-11T01:40:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.041+0000 7f1b7cad6640 10 monclient: monmap: 2024-09-11T01:40:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:40:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:40:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:40:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:40:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:40:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:40:22.051 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7cad6640 1 Processor -- start 2024-09-11T01:40:22.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7cad6640 1 -- start start 2024-09-11T01:40:22.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7cad6640 10 monclient: init 2024-09-11T01:40:22.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7cad6640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:22.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7cad6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:40:22.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7cad6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b7809fbd0 0x7f1b780994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7cad6640 10 monclient(hunting): picked mon.noname-a con 0x7f1b7809fbd0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:40:22.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7cad6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b780a2db0 0x7f1b780913c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7cad6640 10 monclient(hunting): picked mon.noname-b con 0x7f1b780a2db0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:40:22.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7cad6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b7813e560 0x7f1b78091900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7cad6640 10 monclient(hunting): picked mon.noname-c con 0x7f1b7813e560 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:40:22.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7cad6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:22.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7cad6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1b7809e8a0 con 0x7f1b7809fbd0 2024-09-11T01:40:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7cad6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7cad6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1b780a3190 con 0x7f1b780a2db0 2024-09-11T01:40:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7cad6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7cad6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1b7809ea20 con 0x7f1b7813e560 2024-09-11T01:40:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7cad6640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7cad6640 10 monclient(hunting): authenticate will time out at 1428.484782s 2024-09-11T01:40:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b7809fbd0 0x7f1b780994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b76d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b7813e560 0x7f1b78091900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b7809fbd0 0x7f1b780994a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60880/0 (socket says 172.21.15.12:60880) 2024-09-11T01:40:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7659c640 1 -- 172.21.15.12:0/77356551 learned_addr learned my addr 172.21.15.12:0/77356551 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:40:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b76d9d640 10 monclient(hunting): get_auth_request con 0x7f1b7813e560 auth_method 0 2024-09-11T01:40:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b76d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:22.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b76d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:22.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7659c640 10 monclient(hunting): get_auth_request con 0x7f1b7809fbd0 auth_method 0 2024-09-11T01:40:22.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:22.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:22.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:22.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:22.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.042+0000 7f1b7659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:22.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.043+0000 7f1b7659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:22.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.043+0000 7f1b76d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:22.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.043+0000 7f1b76d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:22.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.043+0000 7f1b76d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:22.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.043+0000 7f1b75d9b640 1 --2- 172.21.15.12:0/77356551 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b780a2db0 0x7f1b780913c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.043+0000 7f1b75d9b640 10 monclient(hunting): get_auth_request con 0x7f1b780a2db0 auth_method 0 2024-09-11T01:40:22.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.043+0000 7f1b75d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:22.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.043+0000 7f1b75d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:22.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.043+0000 7f1b75d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:22.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.043+0000 7f1b75d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:22.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.043+0000 7f1b75d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:22.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.043+0000 7f1b75d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:22.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.043+0000 7f1b7659c640 10 monclient(hunting): handle_auth_done global_id 94371 payload 824 2024-09-11T01:40:22.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.043+0000 7f1b7659c640 1 -- 172.21.15.12:0/77356551 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b7813e560 msgr2=0x7f1b78091900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:22.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.043+0000 7f1b7659c640 1 --2- 172.21.15.12:0/77356551 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b7813e560 0x7f1b78091900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.043+0000 7f1b7659c640 1 -- 172.21.15.12:0/77356551 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b780a2db0 msgr2=0x7f1b780913c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:22.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.043+0000 7f1b7659c640 1 --2- 172.21.15.12:0/77356551 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b780a2db0 0x7f1b780913c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.043+0000 7f1b7659c640 10 monclient: _finish_hunting 0 2024-09-11T01:40:22.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.043+0000 7f1b7659c640 1 monclient: found mon.noname-a 2024-09-11T01:40:22.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.043+0000 7f1b7659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:22.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.043+0000 7f1b7659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:40:22.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.043+0000 7f1b7659c640 1 -- 172.21.15.12:0/77356551 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b78094c80 con 0x7f1b7809fbd0 2024-09-11T01:40:22.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.045+0000 7f1b7659c640 1 -- 172.21.15.12:0/77356551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b7809fbd0 msgr2=0x7f1b780994a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:22.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.045+0000 7f1b7659c640 1 --2- 172.21.15.12:0/77356551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b7809fbd0 0x7f1b780994a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1b6800c460 tx=0x7f1b6800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:22.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.045+0000 7f1b66ffd640 1 -- 172.21.15.12:0/77356551 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b68007070 con 0x7f1b7809fbd0 2024-09-11T01:40:22.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.045+0000 7f1b66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:22.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.045+0000 7f1b66ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:40:22.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.045+0000 7f1b66ffd640 10 monclient: dump: 2024-09-11T01:40:22.056 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:22.056 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:22.056 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:22.057 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.045+0000 7f1b66ffd640 10 monclient: _finish_auth 0 2024-09-11T01:40:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.045+0000 7f1b66ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:40:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.045+0000 7f1b66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.045+0000 7f1b66ffd640 1 -- 172.21.15.12:0/77356551 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b68007230 con 0x7f1b7809fbd0 2024-09-11T01:40:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.045+0000 7f1b7cad6640 5 monclient: authenticate success, global_id 94371 2024-09-11T01:40:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.045+0000 7f1b7cad6640 10 monclient: _renew_subs 2024-09-11T01:40:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.045+0000 7f1b7cad6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:40:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.045+0000 7f1b7cad6640 1 -- 172.21.15.12:0/77356551 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b78094f60 con 0x7f1b7809fbd0 2024-09-11T01:40:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.045+0000 7f1b66ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.045+0000 7f1b66ffd640 1 -- 172.21.15.12:0/77356551 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b68007620 con 0x7f1b7809fbd0 2024-09-11T01:40:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.045+0000 7f1b66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.045+0000 7f1b66ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:40:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.045+0000 7f1b66ffd640 10 monclient: dump: 2024-09-11T01:40:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:22.059 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.045+0000 7f1b67fff640 4 set_mon_vals no callback set 2024-09-11T01:40:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.045+0000 7f1b7cad6640 10 monclient: _renew_subs 2024-09-11T01:40:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.045+0000 7f1b7cad6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:40:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.045+0000 7f1b7cad6640 1 -- 172.21.15.12:0/77356551 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b780949a0 con 0x7f1b7809fbd0 2024-09-11T01:40:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.046+0000 7f1b66ffd640 1 -- 172.21.15.12:0/77356551 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f1b6801b400 con 0x7f1b7809fbd0 2024-09-11T01:40:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.046+0000 7f1b7cad6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.046+0000 7f1b7cad6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.046+0000 7f1b7cad6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:40:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.046+0000 7f1b7cad6640 1 -- 172.21.15.12:0/77356551 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b7806cbd0 con 0x7f1b7809fbd0 2024-09-11T01:40:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.046+0000 7f1b66ffd640 1 --2- 172.21.15.12:0/77356551 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1b5803e260 0x7f1b58040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.046+0000 7f1b66ffd640 1 -- 172.21.15.12:0/77356551 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(377..377 src has 285..377) ==== 4755+0+0 (secure 0 0 0) 0x7f1b68052970 con 0x7f1b7809fbd0 2024-09-11T01:40:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.047+0000 7f1b75d9b640 1 --2- 172.21.15.12:0/77356551 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1b5803e260 0x7f1b58040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.047+0000 7f1b75d9b640 10 monclient: get_auth_request con 0x7f1b5803e260 auth_method 0 2024-09-11T01:40:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.049+0000 7f1b75d9b640 1 --2- 172.21.15.12:0/77356551 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1b5803e260 0x7f1b58040720 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1b78074f90 tx=0x7f1b6c007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.051+0000 7f1b66ffd640 1 -- 172.21.15.12:0/77356551 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1b68051380 con 0x7f1b7809fbd0 2024-09-11T01:40:22.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.051+0000 7f1b66ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:22.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.051+0000 7f1b66ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:40:22.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.304+0000 7f1b7cad6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:40:22.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.304+0000 7f1b7cad6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:40:22.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.304+0000 7f1b7cad6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:40:22.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.304+0000 7f1b7cad6640 1 -- 172.21.15.12:0/77356551 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1b78140870 con 0x7f1b7809fbd0 2024-09-11T01:40:22.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.304+0000 7f1b66ffd640 1 -- 172.21.15.12:0/77356551 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f1b68010d10 con 0x7f1b7809fbd0 2024-09-11T01:40:22.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.304+0000 7f1b66ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:40:22.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.304+0000 7f1b66ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:40:22.307 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":106,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:40:22.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.306+0000 7f1b7cad6640 1 -- 172.21.15.12:0/77356551 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1b5803e260 msgr2=0x7f1b58040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:22.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.306+0000 7f1b7cad6640 1 --2- 172.21.15.12:0/77356551 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1b5803e260 0x7f1b58040720 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1b78074f90 tx=0x7f1b6c007e30 comp rx=0 tx=0).stop 2024-09-11T01:40:22.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.306+0000 7f1b7cad6640 10 monclient: shutdown 2024-09-11T01:40:22.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.306+0000 7f1b7cad6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:22.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.306+0000 7f1b7cad6640 1 -- 172.21.15.12:0/77356551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b7809fbd0 msgr2=0x7f1b780994a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:22.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.306+0000 7f1b7cad6640 1 --2- 172.21.15.12:0/77356551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b7809fbd0 0x7f1b780994a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1b6800c460 tx=0x7f1b6800c920 comp rx=0 tx=0).stop 2024-09-11T01:40:22.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.306+0000 7f1b7cad6640 1 -- 172.21.15.12:0/77356551 shutdown_connections 2024-09-11T01:40:22.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.306+0000 7f1b7cad6640 1 --2- 172.21.15.12:0/77356551 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b7813e560 0x7f1b78091900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.306+0000 7f1b7cad6640 1 --2- 172.21.15.12:0/77356551 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b780a2db0 0x7f1b780913c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.306+0000 7f1b7cad6640 1 --2- 172.21.15.12:0/77356551 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1b5803e260 0x7f1b58040720 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.306+0000 7f1b7cad6640 1 --2- 172.21.15.12:0/77356551 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b7809fbd0 0x7f1b780994a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.306+0000 7f1b7cad6640 1 -- 172.21.15.12:0/77356551 >> 172.21.15.12:0/77356551 conn(0x7f1b780a3ea0 msgr2=0x7f1b7813edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:22.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.306+0000 7f1b7cad6640 1 -- 172.21.15.12:0/77356551 shutdown_connections 2024-09-11T01:40:22.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.306+0000 7f1b7cad6640 1 -- 172.21.15.12:0/77356551 wait complete. 2024-09-11T01:40:22.318 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:40:22.318 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:40:22.318 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:40:22.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.418+0000 7f3cf2ad4640 10 monclient: get_monmap_and_config 2024-09-11T01:40:22.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.419+0000 7f3cf2ad4640 10 monclient: build_initial_monmap 2024-09-11T01:40:22.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.419+0000 7f3cf2ad4640 10 monclient: monmap: 2024-09-11T01:40:22.420 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:22.420 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:22.420 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:40:22.419755+0000 2024-09-11T01:40:22.420 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:40:22.419755+0000 2024-09-11T01:40:22.420 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:22.421 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.421 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:40:22.421 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:40:22.421 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:40:22.421 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:40:22.421 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:40:22.421 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.419+0000 7f3cf2ad4640 1 Processor -- start 2024-09-11T01:40:22.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.419+0000 7f3cf2ad4640 1 -- start start 2024-09-11T01:40:22.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.419+0000 7f3cf2ad4640 10 monclient: init 2024-09-11T01:40:22.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cf2ad4640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:22.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cf2ad4640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-11T01:40:22.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cf2ad4640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3cec1381c0 0x7f3cec1385a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cf2ad4640 10 monclient(hunting): picked mon.noname-c con 0x7f3cec1381c0 addr v2:172.21.15.71:3300/0 2024-09-11T01:40:22.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cf2ad4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3cec0a6aa0 0x7f3cec09cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cf2ad4640 10 monclient(hunting): picked mon.noname-b con 0x7f3cec0a6aa0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:40:22.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cf2ad4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3cec0a4f20 addr v1:172.21.15.12:6789/0 2024-09-11T01:40:22.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cf2ad4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cec09c310 con 0x7f3cec0a4f20 2024-09-11T01:40:22.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cf2ad4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:22.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cf2ad4640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f3cec06fc90 con 0x7f3cec1381c0 2024-09-11T01:40:22.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cf2ad4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:22.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cf2ad4640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f3cec09c5f0 con 0x7f3cec0a6aa0 2024-09-11T01:40:22.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cf2ad4640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:22.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cf2ad4640 10 monclient(hunting): authenticate will time out at 1428.862343s 2024-09-11T01:40:22.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cf1071640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3cec0a4f20 0x7f3cec0a5390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43292/0 (socket says 172.21.15.12:43292) 2024-09-11T01:40:22.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cf1071640 1 -- 172.21.15.12:0/2651472460 learned_addr learned my addr 172.21.15.12:0/2651472460 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:40:22.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cebfff640 1 --2- 172.21.15.12:0/2651472460 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3cec0a6aa0 0x7f3cec09cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cebfff640 10 monclient(hunting): get_auth_request con 0x7f3cec0a6aa0 auth_method 0 2024-09-11T01:40:22.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:22.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cebfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:22.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:22.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cf0870640 1 --2- 172.21.15.12:0/2651472460 >> v2:172.21.15.71:3300/0 conn(0x7f3cec1381c0 0x7f3cec1385a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cf0870640 10 monclient(hunting): get_auth_request con 0x7f3cec1381c0 auth_method 0 2024-09-11T01:40:22.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cf0870640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:22.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cf0870640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:22.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cf0870640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:22.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:22.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:22.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:22.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cf0870640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:22.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cf0870640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:22.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cf0870640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:22.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cebfff640 10 monclient(hunting): handle_auth_done global_id 114206 payload 274 2024-09-11T01:40:22.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cebfff640 1 -- 172.21.15.12:0/2651472460 >> v2:172.21.15.71:3300/0 conn(0x7f3cec1381c0 msgr2=0x7f3cec1385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:40:22.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cebfff640 1 --2- 172.21.15.12:0/2651472460 >> v2:172.21.15.71:3300/0 conn(0x7f3cec1381c0 0x7f3cec1385a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cebfff640 1 -- 172.21.15.12:0/2651472460 >> v1:172.21.15.12:6789/0 conn(0x7f3cec0a4f20 legacy=0x7f3cec0a5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:40:22.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cebfff640 10 monclient: _finish_hunting 0 2024-09-11T01:40:22.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cebfff640 1 monclient: found mon.noname-b 2024-09-11T01:40:22.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:22.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cebfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:40:22.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.420+0000 7f3cebfff640 1 -- 172.21.15.12:0/2651472460 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cec09c8d0 con 0x7f3cec0a6aa0 2024-09-11T01:40:22.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.423+0000 7f3cebfff640 1 -- 172.21.15.12:0/2651472460 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3cec0a6aa0 msgr2=0x7f3cec09cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:22.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.423+0000 7f3cebfff640 1 --2- 172.21.15.12:0/2651472460 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3cec0a6aa0 0x7f3cec09cfd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3cd40097a0 tx=0x7f3cd402a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b03b0b8f28186e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:22.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.423+0000 7f3ceb7fe640 1 -- 172.21.15.12:0/2651472460 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd4037070 con 0x7f3cec0a6aa0 2024-09-11T01:40:22.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.423+0000 7f3ceb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:22.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.423+0000 7f3ceb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:40:22.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.423+0000 7f3ceb7fe640 10 monclient: dump: 2024-09-11T01:40:22.433 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:22.433 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:22.433 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.433 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.433 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:22.434 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.434 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:22.434 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:22.434 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:22.434 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.423+0000 7f3ceb7fe640 10 monclient: _finish_auth 0 2024-09-11T01:40:22.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.423+0000 7f3ceb7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:40:22.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.423+0000 7f3ceb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:22.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.423+0000 7f3ceb7fe640 1 -- 172.21.15.12:0/2651472460 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cd4032440 con 0x7f3cec0a6aa0 2024-09-11T01:40:22.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.423+0000 7f3cf2ad4640 5 monclient: authenticate success, global_id 114206 2024-09-11T01:40:22.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.423+0000 7f3cf2ad4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:40:22.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.423+0000 7f3ceb7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:22.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.423+0000 7f3cf2ad4640 10 monclient: get_monmap_and_config success 2024-09-11T01:40:22.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.423+0000 7f3ceb7fe640 1 -- 172.21.15.12:0/2651472460 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd4032810 con 0x7f3cec0a6aa0 2024-09-11T01:40:22.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.423+0000 7f3ceb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:22.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.423+0000 7f3cf2ad4640 4 set_mon_vals no callback set 2024-09-11T01:40:22.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.423+0000 7f3ceb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:40:22.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.423+0000 7f3ceb7fe640 10 monclient: dump: 2024-09-11T01:40:22.435 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:22.435 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:22.435 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.435 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.435 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:22.435 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.435 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:22.435 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:22.435 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:22.436 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.424+0000 7f3cf2ad4640 10 monclient: shutdown 2024-09-11T01:40:22.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.424+0000 7f3cf2ad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:22.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.424+0000 7f3cf2ad4640 1 -- 172.21.15.12:0/2651472460 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3cec0a6aa0 msgr2=0x7f3cec09cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:22.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.424+0000 7f3cf2ad4640 1 --2- 172.21.15.12:0/2651472460 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3cec0a6aa0 0x7f3cec09cfd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3cd40097a0 tx=0x7f3cd402a040 comp rx=0 tx=0).stop 2024-09-11T01:40:22.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.424+0000 7f3cf2ad4640 1 -- 172.21.15.12:0/2651472460 shutdown_connections 2024-09-11T01:40:22.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.424+0000 7f3cf2ad4640 1 --2- 172.21.15.12:0/2651472460 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3cec0a6aa0 0x7f3cec09cfd0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.424+0000 7f3cf2ad4640 1 --2- 172.21.15.12:0/2651472460 >> v2:172.21.15.71:3300/0 conn(0x7f3cec1381c0 0x7f3cec1385a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.424+0000 7f3cf2ad4640 1 -- 172.21.15.12:0/2651472460 >> 172.21.15.12:0/2651472460 conn(0x7f3cec0a7440 msgr2=0x7f3cec0a7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:22.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.424+0000 7f3cf2ad4640 1 -- 172.21.15.12:0/2651472460 shutdown_connections 2024-09-11T01:40:22.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.424+0000 7f3cf2ad4640 1 -- 172.21.15.12:0/2651472460 wait complete. 2024-09-11T01:40:22.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.424+0000 7f3cf2ad4640 10 monclient: build_initial_monmap 2024-09-11T01:40:22.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.424+0000 7f3cf2ad4640 10 monclient: monmap: 2024-09-11T01:40:22.437 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:22.437 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:40:22.437 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:40:22.437 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:40:22.437 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:22.437 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.437 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:40:22.437 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:40:22.437 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:40:22.437 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.425+0000 7f3cf2ad4640 1 Processor -- start 2024-09-11T01:40:22.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.425+0000 7f3cf2ad4640 1 -- start start 2024-09-11T01:40:22.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.425+0000 7f3cf2ad4640 10 monclient: init 2024-09-11T01:40:22.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.425+0000 7f3cf2ad4640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:22.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.425+0000 7f3cf2ad4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:40:22.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.425+0000 7f3cf2ad4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cec0a4f20 0x7f3cec135f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.425+0000 7f3cf2ad4640 10 monclient(hunting): picked mon.noname-a con 0x7f3cec0a4f20 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:40:22.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.425+0000 7f3cf2ad4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cec0a6aa0 0x7f3cec1364a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.425+0000 7f3cf2ad4640 10 monclient(hunting): picked mon.noname-b con 0x7f3cec0a6aa0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:40:22.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.425+0000 7f3cf2ad4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3cec1381c0 0x7f3cec123af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.425+0000 7f3cf2ad4640 10 monclient(hunting): picked mon.noname-c con 0x7f3cec1381c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:40:22.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.425+0000 7f3cf2ad4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:22.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.425+0000 7f3cf2ad4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3cec09d690 con 0x7f3cec0a4f20 2024-09-11T01:40:22.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.425+0000 7f3cf2ad4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:22.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.425+0000 7f3cf2ad4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3cec09d510 con 0x7f3cec0a6aa0 2024-09-11T01:40:22.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.425+0000 7f3cf2ad4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:22.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.425+0000 7f3cf2ad4640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f3cec0a5aa0 con 0x7f3cec1381c0 2024-09-11T01:40:22.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.425+0000 7f3cf2ad4640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:22.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.425+0000 7f3cf2ad4640 10 monclient(hunting): authenticate will time out at 1428.867953s 2024-09-11T01:40:22.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf0870640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cec0a4f20 0x7f3cec135f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf0870640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cec0a4f20 0x7f3cec135f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60896/0 (socket says 172.21.15.12:60896) 2024-09-11T01:40:22.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf0870640 1 -- 172.21.15.12:0/205036421 learned_addr learned my addr 172.21.15.12:0/205036421 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:40:22.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf0870640 10 monclient(hunting): get_auth_request con 0x7f3cec0a4f20 auth_method 0 2024-09-11T01:40:22.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf0870640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:22.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf0870640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:22.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf0870640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:22.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf1071640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3cec1381c0 0x7f3cec123af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf1071640 10 monclient(hunting): get_auth_request con 0x7f3cec1381c0 auth_method 0 2024-09-11T01:40:22.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf1071640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:22.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf1071640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:22.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf1071640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:22.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf0870640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:22.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf0870640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:22.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf0870640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:22.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf1071640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:22.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf1071640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:22.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf1071640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:22.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cebfff640 1 --2- 172.21.15.12:0/205036421 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cec0a6aa0 0x7f3cec1364a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf1071640 10 monclient(hunting): handle_auth_done global_id 114209 payload 824 2024-09-11T01:40:22.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf1071640 1 -- 172.21.15.12:0/205036421 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cec0a6aa0 msgr2=0x7f3cec1364a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:22.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf1071640 1 --2- 172.21.15.12:0/205036421 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cec0a6aa0 0x7f3cec1364a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf1071640 1 -- 172.21.15.12:0/205036421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cec0a4f20 msgr2=0x7f3cec135f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:22.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf1071640 1 --2- 172.21.15.12:0/205036421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cec0a4f20 0x7f3cec135f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf1071640 10 monclient: _finish_hunting 0 2024-09-11T01:40:22.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf1071640 1 monclient: found mon.noname-c 2024-09-11T01:40:22.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf1071640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:22.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf1071640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:40:22.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf1071640 1 -- 172.21.15.12:0/205036421 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cec0a8420 con 0x7f3cec1381c0 2024-09-11T01:40:22.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cebfff640 10 monclient: get_auth_request con 0x7f3cec0a6aa0 auth_method 0 2024-09-11T01:40:22.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cebfff640 1 --2- 172.21.15.12:0/205036421 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cec0a6aa0 0x7f3cec1364a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:40:22.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.426+0000 7f3cf0870640 1 --2- 172.21.15.12:0/205036421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cec0a4f20 0x7f3cec135f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:40:22.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.428+0000 7f3cf1071640 1 -- 172.21.15.12:0/205036421 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3cec1381c0 msgr2=0x7f3cec123af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:22.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.428+0000 7f3cf1071640 1 --2- 172.21.15.12:0/205036421 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3cec1381c0 0x7f3cec123af0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3ce400cb20 tx=0x7f3ce4005ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:22.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.428+0000 7f3ce97fa640 1 -- 172.21.15.12:0/205036421 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce400ad80 con 0x7f3cec1381c0 2024-09-11T01:40:22.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.428+0000 7f3ce97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:22.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.428+0000 7f3ce97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:40:22.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.428+0000 7f3ce97fa640 10 monclient: dump: 2024-09-11T01:40:22.442 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:22.442 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:22.442 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.442 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.442 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:22.442 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.443 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:22.443 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:22.443 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:22.443 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.428+0000 7f3ce97fa640 10 monclient: _finish_auth 0 2024-09-11T01:40:22.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.428+0000 7f3ce97fa640 10 monclient: _check_auth_tickets 2024-09-11T01:40:22.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.428+0000 7f3ce97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:22.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.428+0000 7f3cf2ad4640 5 monclient: authenticate success, global_id 114209 2024-09-11T01:40:22.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.428+0000 7f3ce97fa640 1 -- 172.21.15.12:0/205036421 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ce4012400 con 0x7f3cec1381c0 2024-09-11T01:40:22.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.428+0000 7f3ce97fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:22.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.428+0000 7f3ce97fa640 1 -- 172.21.15.12:0/205036421 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce40127f0 con 0x7f3cec1381c0 2024-09-11T01:40:22.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.428+0000 7f3cf2ad4640 10 monclient: _renew_subs 2024-09-11T01:40:22.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.428+0000 7f3cf2ad4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:40:22.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.428+0000 7f3cf2ad4640 1 -- 172.21.15.12:0/205036421 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cec1369e0 con 0x7f3cec1381c0 2024-09-11T01:40:22.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.428+0000 7f3ceaffd640 4 set_mon_vals no callback set 2024-09-11T01:40:22.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.428+0000 7f3ce97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:22.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.428+0000 7f3ce97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:40:22.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.428+0000 7f3ce97fa640 10 monclient: dump: 2024-09-11T01:40:22.444 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:22.444 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:22.444 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.444 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.444 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:22.444 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.444 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:22.444 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:22.445 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:22.445 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.428+0000 7f3cf2ad4640 10 monclient: _renew_subs 2024-09-11T01:40:22.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.428+0000 7f3cf2ad4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:40:22.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.428+0000 7f3cf2ad4640 1 -- 172.21.15.12:0/205036421 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cec127680 con 0x7f3cec1381c0 2024-09-11T01:40:22.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.429+0000 7f3cf2ad4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:22.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.429+0000 7f3cf2ad4640 1 --2- 172.21.15.12:0/205036421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cec1cd080 0x7f3cec124030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.429+0000 7f3cf2ad4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:22.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.429+0000 7f3cf2ad4640 1 -- 172.21.15.12:0/205036421 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3cec077510 con 0x7f3cec1cd080 2024-09-11T01:40:22.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.429+0000 7f3ce97fa640 1 -- 172.21.15.12:0/205036421 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f3ce4012a50 con 0x7f3cec1381c0 2024-09-11T01:40:22.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.429+0000 7f3cf0870640 1 --2- 172.21.15.12:0/205036421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cec1cd080 0x7f3cec124030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.429+0000 7f3cf0870640 10 monclient: get_auth_request con 0x7f3cec1cd080 auth_method 0 2024-09-11T01:40:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.429+0000 7f3cf0870640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.429+0000 7f3cf0870640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.429+0000 7f3cf0870640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.429+0000 7f3cf0870640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.429+0000 7f3cf0870640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.429+0000 7f3cf0870640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.430+0000 7f3ce97fa640 1 --2- 172.21.15.12:0/205036421 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3ccc03e240 0x7f3ccc040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.430+0000 7f3cf0870640 10 monclient(hunting): handle_auth_done global_id 94377 payload 824 2024-09-11T01:40:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.430+0000 7f3ce97fa640 1 -- 172.21.15.12:0/205036421 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(377..377 src has 285..377) ==== 4755+0+0 (secure 0 0 0) 0x7f3ce40532d0 con 0x7f3cec1381c0 2024-09-11T01:40:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.430+0000 7f3cf0870640 1 -- 172.21.15.12:0/205036421 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3cec125d40 con 0x7f3cec1cd080 2024-09-11T01:40:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.430+0000 7f3cebfff640 1 --2- 172.21.15.12:0/205036421 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3ccc03e240 0x7f3ccc040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.430+0000 7f3cebfff640 10 monclient: get_auth_request con 0x7f3ccc03e240 auth_method 0 2024-09-11T01:40:22.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.432+0000 7f3cf0870640 1 -- 172.21.15.12:0/205036421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cec1cd080 msgr2=0x7f3cec124030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:22.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.432+0000 7f3cf0870640 1 --2- 172.21.15.12:0/205036421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cec1cd080 0x7f3cec124030 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc011db0 tx=0x7f3cdc00f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:22.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.432+0000 7f3ce97fa640 1 -- 172.21.15.12:0/205036421 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc00fe70 con 0x7f3cec1cd080 2024-09-11T01:40:22.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.432+0000 7f3ce97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:22.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.432+0000 7f3ce97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:40:22.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.432+0000 7f3ce97fa640 10 monclient: dump: 2024-09-11T01:40:22.449 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:22.449 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:22.449 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.449 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.449 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:22.449 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.450 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:22.450 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:22.450 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:22.450 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.432+0000 7f3ce97fa640 1 -- 172.21.15.12:0/205036421 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3cdc00fe70 con 0x7f3cec1cd080 2024-09-11T01:40:22.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.432+0000 7f3ce97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:22.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.432+0000 7f3ce97fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:40:22.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.432+0000 7f3cebfff640 1 --2- 172.21.15.12:0/205036421 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3ccc03e240 0x7f3ccc040700 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3cec0a4be0 tx=0x7f3cd402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:22.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.432+0000 7f3ce97fa640 1 -- 172.21.15.12:0/205036421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cec1cd080 msgr2=0x7f3cec124030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:22.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.432+0000 7f3ce97fa640 1 --2- 172.21.15.12:0/205036421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cec1cd080 0x7f3cec124030 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc011db0 tx=0x7f3cdc00f610 comp rx=0 tx=0).stop 2024-09-11T01:40:22.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.432+0000 7f3ce97fa640 1 -- 172.21.15.12:0/205036421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cec1cd080 msgr2=0x7f3cec124030 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:40:22.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.432+0000 7f3ce97fa640 1 --2- 172.21.15.12:0/205036421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cec1cd080 0x7f3cec124030 secure :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc011db0 tx=0x7f3cdc00f610 comp rx=0 tx=0).stop 2024-09-11T01:40:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.438+0000 7f3cf2ad4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:40:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.438+0000 7f3cf2ad4640 1 --2- 172.21.15.12:0/205036421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cec124570 0x7f3cec0a3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.438+0000 7f3cf2ad4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.438+0000 7f3cf2ad4640 1 -- 172.21.15.12:0/205036421 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3cec124980 con 0x7f3cec124570 2024-09-11T01:40:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.439+0000 7f3cf0870640 1 --2- 172.21.15.12:0/205036421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cec124570 0x7f3cec0a3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.439+0000 7f3cf0870640 10 monclient: get_auth_request con 0x7f3cec124570 auth_method 0 2024-09-11T01:40:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.439+0000 7f3cf0870640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.439+0000 7f3cf0870640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.439+0000 7f3cf0870640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.439+0000 7f3cf0870640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.439+0000 7f3cf0870640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.439+0000 7f3cf0870640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:22.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.439+0000 7f3cf0870640 10 monclient(hunting): handle_auth_done global_id 94380 payload 824 2024-09-11T01:40:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.439+0000 7f3cf0870640 1 -- 172.21.15.12:0/205036421 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3cec09cfd0 con 0x7f3cec124570 2024-09-11T01:40:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.441+0000 7f3cf0870640 1 -- 172.21.15.12:0/205036421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cec124570 msgr2=0x7f3cec0a3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.441+0000 7f3cf0870640 1 --2- 172.21.15.12:0/205036421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cec124570 0x7f3cec0a3230 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc01b000 tx=0x7f3cdc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.441+0000 7f3ce97fa640 1 -- 172.21.15.12:0/205036421 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc003a20 con 0x7f3cec124570 2024-09-11T01:40:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.441+0000 7f3ce97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.441+0000 7f3ce97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:40:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.441+0000 7f3ce97fa640 10 monclient: dump: 2024-09-11T01:40:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.452 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:22.453 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.453 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:22.453 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:22.453 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:22.453 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.441+0000 7f3ce97fa640 1 -- 172.21.15.12:0/205036421 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f3cdc003a20 con 0x7f3cec124570 2024-09-11T01:40:22.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.441+0000 7f3ce97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:40:22.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.441+0000 7f3ce97fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:40:22.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.441+0000 7f3ce97fa640 1 -- 172.21.15.12:0/205036421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cec124570 msgr2=0x7f3cec0a3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:22.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.441+0000 7f3ce97fa640 1 --2- 172.21.15.12:0/205036421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cec124570 0x7f3cec0a3230 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc01b000 tx=0x7f3cdc0046c0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.441+0000 7f3ce97fa640 1 -- 172.21.15.12:0/205036421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cec124570 msgr2=0x7f3cec0a3230 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:40:22.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.441+0000 7f3ce97fa640 1 --2- 172.21.15.12:0/205036421 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cec124570 0x7f3cec0a3230 secure :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc01b000 tx=0x7f3cdc0046c0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.454 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:40:22.454 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:40:22.454 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:40:22.454 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:40:22.454 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 106, 2024-09-11T01:40:22.454 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 168537, 2024-09-11T01:40:22.454 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:40:22.454 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:40:22.454 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:40:22.454 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:40:22.454 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:22.454 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:40:22.455 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:40:22.455 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:40:22.455 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:40:22.455 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:40:22.455 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:40:22.455 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:40:22.455 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:40:22.455 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:40:22.455 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:40:22.455 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:40:22.455 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:40:22.455 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:40:22.455 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:40:22.455 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:40:22.456 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:22.456 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:40:22.456 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:40:22.456 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:40:22.456 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:40:22.456 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:40:22.456 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:40:22.456 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:40:22.456 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:40:22.456 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:40:22.456 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:40:22.456 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:40:22.457 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:40:22.457 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:40:22.457 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:22.457 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.457 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:40:22.457 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:40:22.457 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:40:22.457 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:40:22.457 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:40:22.457 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:40:22.457 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:40:22.457 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:40:22.457 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:40:22.458 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:40:22.458 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:40:22.458 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:40:22.458 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:40:22.458 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:40:22.458 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:40:22.458 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:40:22.458 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:40:22.458 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:40:22.458 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:40:22.458 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:40:22.458 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:40:22.458 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:40:22.458 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:40:22.459 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:40:22.459 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:40:22.459 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:40:22.459 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:40:22.459 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:40:22.459 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:22.459 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:40:22.459 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.459 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:40:22.459 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.459 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:40:22.459 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:40:22.459 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:40:22.459 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:40:22.459 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.460 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:40:22.460 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:40:22.460 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:22.460 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.460 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.460 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:40:22.460 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:40:22.460 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:22.460 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:22.460 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:22.460 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.460 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:40:22.460 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:40:22.460 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:40:22.461 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:40:22.461 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:40:22.461 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.461 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.461 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:40:22.461 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:40:22.461 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:40:22.461 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:40:22.461 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.461 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:40:22.461 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:40:22.461 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:22.461 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.461 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.462 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:40:22.462 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:40:22.462 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:22.462 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:22.462 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:22.462 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.462 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:40:22.462 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:40:22.462 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:40:22.462 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:40:22.462 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:40:22.462 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.462 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.463 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:40:22.463 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:40:22.463 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:40:22.463 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:40:22.463 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.463 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:40:22.463 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:40:22.463 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:22.463 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.463 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.463 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:40:22.463 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:40:22.463 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:22.464 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:22.464 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:22.464 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.464 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:40:22.464 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:40:22.464 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:40:22.464 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:40:22.464 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:40:22.464 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:22.464 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:22.464 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.464 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:40:22.464 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:40:22.464 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.465 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:40:22.465 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:40:22.465 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:40:22.465 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:22.465 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:22.465 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:40:22.465 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.465 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:40:22.465 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:40:22.465 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:40:22.465 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:22.465 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:22.465 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:40:22.465 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.465 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:40:22.466 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:40:22.466 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:40:22.466 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:22.466 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:22.466 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.466 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:40:22.466 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:40:22.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.442+0000 7f3cf2ad4640 1 -- 172.21.15.12:0/205036421 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3ccc03e240 msgr2=0x7f3ccc040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:22.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.442+0000 7f3cf2ad4640 1 --2- 172.21.15.12:0/205036421 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3ccc03e240 0x7f3ccc040700 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3cec0a4be0 tx=0x7f3cd402e000 comp rx=0 tx=0).stop 2024-09-11T01:40:22.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.442+0000 7f3cf2ad4640 10 monclient: shutdown 2024-09-11T01:40:22.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.442+0000 7f3cf2ad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:22.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.442+0000 7f3cf2ad4640 1 -- 172.21.15.12:0/205036421 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3cec1381c0 msgr2=0x7f3cec123af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:22.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.442+0000 7f3cf1071640 1 -- 172.21.15.12:0/205036421 reap_dead start 2024-09-11T01:40:22.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.442+0000 7f3cf2ad4640 1 --2- 172.21.15.12:0/205036421 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3cec1381c0 0x7f3cec123af0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3ce400cb20 tx=0x7f3ce4005ac0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.442+0000 7f3cf2ad4640 1 -- 172.21.15.12:0/205036421 shutdown_connections 2024-09-11T01:40:22.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.442+0000 7f3cf2ad4640 1 --2- 172.21.15.12:0/205036421 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3cec1381c0 0x7f3cec123af0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.442+0000 7f3cf2ad4640 1 -- 172.21.15.12:0/205036421 >> 172.21.15.12:0/205036421 conn(0x7f3cec0a7440 msgr2=0x7f3cec09ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:22.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.442+0000 7f3cf2ad4640 1 -- 172.21.15.12:0/205036421 shutdown_connections 2024-09-11T01:40:22.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.442+0000 7f3cf2ad4640 1 -- 172.21.15.12:0/205036421 wait complete. 2024-09-11T01:40:22.469 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:40:22.469 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:40:22.469 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:40:22.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.569+0000 7faa2cacf640 10 monclient: get_monmap_and_config 2024-09-11T01:40:22.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.569+0000 7faa2cacf640 10 monclient: build_initial_monmap 2024-09-11T01:40:22.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.569+0000 7faa2cacf640 10 monclient: monmap: 2024-09-11T01:40:22.571 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:22.571 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:22.571 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:40:22.570016+0000 2024-09-11T01:40:22.571 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:40:22.570016+0000 2024-09-11T01:40:22.571 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:22.571 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.571 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:40:22.571 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:40:22.571 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:40:22.571 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:40:22.572 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:40:22.572 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.569+0000 7faa2cacf640 1 Processor -- start 2024-09-11T01:40:22.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.569+0000 7faa2cacf640 1 -- start start 2024-09-11T01:40:22.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.569+0000 7faa2cacf640 10 monclient: init 2024-09-11T01:40:22.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa2cacf640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:22.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa2cacf640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-11T01:40:22.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa2cacf640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7faa28137e50 0x7faa2809d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa2cacf640 10 monclient(hunting): picked mon.noname-c con 0x7faa28137e50 addr v2:172.21.15.71:3300/0 2024-09-11T01:40:22.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa2cacf640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7faa28138230 0x7faa28138640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa2cacf640 10 monclient(hunting): picked mon.noname-a con 0x7faa28138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:40:22.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa2cacf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faa28134cb0 addr v1:172.21.15.71:6789/0 2024-09-11T01:40:22.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa2cacf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa280a62d0 con 0x7faa28134cb0 2024-09-11T01:40:22.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa2cacf640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:22.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa2cacf640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7faa2809bb40 con 0x7faa28138230 2024-09-11T01:40:22.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa2cacf640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:22.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa2cacf640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7faa2809c4a0 con 0x7faa28137e50 2024-09-11T01:40:22.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa2cacf640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:22.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa2cacf640 10 monclient(hunting): authenticate will time out at 1429.012604s 2024-09-11T01:40:22.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa25d9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7faa28138230 0x7faa28138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa25d9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7faa28138230 0x7faa28138640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60922/0 (socket says 172.21.15.12:60922) 2024-09-11T01:40:22.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa25d9b640 1 -- 172.21.15.12:0/1315062035 learned_addr learned my addr 172.21.15.12:0/1315062035 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:40:22.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa25d9b640 10 monclient(hunting): get_auth_request con 0x7faa28138230 auth_method 0 2024-09-11T01:40:22.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa25d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:22.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa25d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:22.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa25d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:22.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa25d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:22.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa25d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:22.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa25d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:22.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa2659c640 1 --2- 172.21.15.12:0/1315062035 >> v2:172.21.15.71:3300/0 conn(0x7faa28137e50 0x7faa2809d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa2659c640 10 monclient(hunting): get_auth_request con 0x7faa28137e50 auth_method 0 2024-09-11T01:40:22.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:22.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa2659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:22.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.570+0000 7faa2659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:22.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.571+0000 7faa2659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:22.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.571+0000 7faa2659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:22.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.571+0000 7faa2659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:22.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.571+0000 7faa25d9b640 10 monclient(hunting): handle_auth_done global_id 94383 payload 274 2024-09-11T01:40:22.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.571+0000 7faa25d9b640 1 -- 172.21.15.12:0/1315062035 >> v2:172.21.15.71:3300/0 conn(0x7faa28137e50 msgr2=0x7faa2809d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:40:22.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.571+0000 7faa25d9b640 1 --2- 172.21.15.12:0/1315062035 >> v2:172.21.15.71:3300/0 conn(0x7faa28137e50 0x7faa2809d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.571+0000 7faa25d9b640 1 -- 172.21.15.12:0/1315062035 >> v1:172.21.15.71:6789/0 conn(0x7faa28134cb0 legacy=0x7faa28135160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:40:22.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.571+0000 7faa25d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:40:22.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.571+0000 7faa25d9b640 1 monclient: found mon.noname-a 2024-09-11T01:40:22.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.571+0000 7faa25d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:22.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.571+0000 7faa25d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:40:22.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.571+0000 7faa25d9b640 1 -- 172.21.15.12:0/1315062035 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa2809c780 con 0x7faa28138230 2024-09-11T01:40:22.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.573+0000 7faa25d9b640 1 -- 172.21.15.12:0/1315062035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa28138230 msgr2=0x7faa28138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:22.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.573+0000 7faa25d9b640 1 --2- 172.21.15.12:0/1315062035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa28138230 0x7faa28138640 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7faa2809cf30 tx=0x7faa1c02a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e8f1058aab9bcbb server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:22.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2559a640 1 -- 172.21.15.12:0/1315062035 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c037070 con 0x7faa28138230 2024-09-11T01:40:22.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:22.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:40:22.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2559a640 10 monclient: dump: 2024-09-11T01:40:22.583 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:22.583 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:22.583 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.583 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.583 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:22.583 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:22.584 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2559a640 10 monclient: _finish_auth 0 2024-09-11T01:40:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2559a640 10 monclient: _check_auth_tickets 2024-09-11T01:40:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2cacf640 5 monclient: authenticate success, global_id 94383 2024-09-11T01:40:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2559a640 1 -- 172.21.15.12:0/1315062035 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa1c032440 con 0x7faa28138230 2024-09-11T01:40:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2cacf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:40:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2559a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2cacf640 10 monclient: get_monmap_and_config success 2024-09-11T01:40:22.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2559a640 1 -- 172.21.15.12:0/1315062035 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c0327f0 con 0x7faa28138230 2024-09-11T01:40:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2cacf640 4 set_mon_vals no callback set 2024-09-11T01:40:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:40:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2559a640 10 monclient: dump: 2024-09-11T01:40:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:22.585 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:22.585 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2cacf640 10 monclient: shutdown 2024-09-11T01:40:22.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2cacf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:22.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2cacf640 1 -- 172.21.15.12:0/1315062035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa28138230 msgr2=0x7faa28138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:22.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2cacf640 1 --2- 172.21.15.12:0/1315062035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa28138230 0x7faa28138640 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7faa2809cf30 tx=0x7faa1c02a080 comp rx=0 tx=0).stop 2024-09-11T01:40:22.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2cacf640 1 -- 172.21.15.12:0/1315062035 shutdown_connections 2024-09-11T01:40:22.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2cacf640 1 --2- 172.21.15.12:0/1315062035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa28138230 0x7faa28138640 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2cacf640 1 --2- 172.21.15.12:0/1315062035 >> v2:172.21.15.71:3300/0 conn(0x7faa28137e50 0x7faa2809d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2cacf640 1 -- 172.21.15.12:0/1315062035 >> 172.21.15.12:0/1315062035 conn(0x7faa280a7020 msgr2=0x7faa280a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:22.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2cacf640 1 -- 172.21.15.12:0/1315062035 shutdown_connections 2024-09-11T01:40:22.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.574+0000 7faa2cacf640 1 -- 172.21.15.12:0/1315062035 wait complete. 2024-09-11T01:40:22.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.575+0000 7faa2cacf640 10 monclient: build_initial_monmap 2024-09-11T01:40:22.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.575+0000 7faa2cacf640 10 monclient: monmap: 2024-09-11T01:40:22.586 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:22.587 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:40:22.587 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:40:22.587 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:40:22.587 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:22.587 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.587 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:40:22.587 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:40:22.587 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:40:22.587 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.575+0000 7faa2cacf640 1 Processor -- start 2024-09-11T01:40:22.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.575+0000 7faa2cacf640 1 -- start start 2024-09-11T01:40:22.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.575+0000 7faa2cacf640 10 monclient: init 2024-09-11T01:40:22.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa2cacf640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:22.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa2cacf640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:40:22.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa2cacf640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa28134cb0 0x7faa281cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa2cacf640 10 monclient(hunting): picked mon.noname-c con 0x7faa28134cb0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:40:22.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa2cacf640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa28137e50 0x7faa281cd910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa2cacf640 10 monclient(hunting): picked mon.noname-a con 0x7faa28137e50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:40:22.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa2cacf640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa28138230 0x7faa281216d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa2cacf640 10 monclient(hunting): picked mon.noname-b con 0x7faa28138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:40:22.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa2cacf640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:22.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa2cacf640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7faa2806f710 con 0x7faa28137e50 2024-09-11T01:40:22.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa2cacf640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:22.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa2cacf640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7faa28138b80 con 0x7faa28138230 2024-09-11T01:40:22.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa2cacf640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:22.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa2cacf640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7faa2806f890 con 0x7faa28134cb0 2024-09-11T01:40:22.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa2cacf640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:22.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa2cacf640 10 monclient(hunting): authenticate will time out at 1429.018262s 2024-09-11T01:40:22.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa25d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa28137e50 0x7faa281cd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa2659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa28134cb0 0x7faa281cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa25d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa28137e50 0x7faa281cd910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60926/0 (socket says 172.21.15.12:60926) 2024-09-11T01:40:22.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa25d9b640 1 -- 172.21.15.12:0/2424507174 learned_addr learned my addr 172.21.15.12:0/2424507174 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:40:22.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa25d9b640 10 monclient(hunting): get_auth_request con 0x7faa28137e50 auth_method 0 2024-09-11T01:40:22.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa25d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:22.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa25d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:22.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa25d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:22.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa2659c640 10 monclient(hunting): get_auth_request con 0x7faa28134cb0 auth_method 0 2024-09-11T01:40:22.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:22.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa2659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:22.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa2659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:22.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa25d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:22.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa25d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:22.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa25d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:22.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa2659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:22.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa2659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:22.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa2659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:22.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa26d9d640 1 --2- 172.21.15.12:0/2424507174 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa28138230 0x7faa281216d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa26d9d640 10 monclient(hunting): get_auth_request con 0x7faa28138230 auth_method 0 2024-09-11T01:40:22.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa26d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:22.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa26d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:22.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa26d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:22.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa25d9b640 10 monclient(hunting): handle_auth_done global_id 94386 payload 824 2024-09-11T01:40:22.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa25d9b640 1 -- 172.21.15.12:0/2424507174 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa28134cb0 msgr2=0x7faa281cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:22.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa25d9b640 1 --2- 172.21.15.12:0/2424507174 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa28134cb0 0x7faa281cd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa25d9b640 1 -- 172.21.15.12:0/2424507174 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa28138230 msgr2=0x7faa281216d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:22.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa25d9b640 1 --2- 172.21.15.12:0/2424507174 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa28138230 0x7faa281216d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa25d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:40:22.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa25d9b640 1 monclient: found mon.noname-a 2024-09-11T01:40:22.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa25d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:22.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa25d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:40:22.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa25d9b640 1 -- 172.21.15.12:0/2424507174 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa280a8e90 con 0x7faa28137e50 2024-09-11T01:40:22.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.576+0000 7faa26d9d640 1 --2- 172.21.15.12:0/2424507174 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa28138230 0x7faa281216d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:40:22.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.578+0000 7faa25d9b640 1 -- 172.21.15.12:0/2424507174 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa28137e50 msgr2=0x7faa281cd910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:22.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.578+0000 7faa25d9b640 1 --2- 172.21.15.12:0/2424507174 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa28137e50 0x7faa281cd910 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7faa1c0090b0 tx=0x7faa1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:22.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.579+0000 7faa16ffd640 1 -- 172.21.15.12:0/2424507174 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c037040 con 0x7faa28137e50 2024-09-11T01:40:22.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.579+0000 7faa16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:22.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.579+0000 7faa16ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:40:22.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.579+0000 7faa16ffd640 10 monclient: dump: 2024-09-11T01:40:22.592 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:22.592 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:22.592 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.592 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.592 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:22.592 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.592 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:22.592 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:22.592 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:22.592 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.579+0000 7faa16ffd640 10 monclient: _finish_auth 0 2024-09-11T01:40:22.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.579+0000 7faa16ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:40:22.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.579+0000 7faa16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:22.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.579+0000 7faa16ffd640 1 -- 172.21.15.12:0/2424507174 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa1c042070 con 0x7faa28137e50 2024-09-11T01:40:22.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.579+0000 7faa2cacf640 5 monclient: authenticate success, global_id 94386 2024-09-11T01:40:22.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.579+0000 7faa16ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:22.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.579+0000 7faa16ffd640 1 -- 172.21.15.12:0/2424507174 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c03ba60 con 0x7faa28137e50 2024-09-11T01:40:22.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.579+0000 7faa16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:22.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.579+0000 7faa16ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:40:22.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.579+0000 7faa16ffd640 10 monclient: dump: 2024-09-11T01:40:22.593 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:22.593 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:22.593 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.593 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.593 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:22.593 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.594 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:22.594 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:22.594 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:22.594 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.579+0000 7faa2cacf640 10 monclient: _renew_subs 2024-09-11T01:40:22.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.579+0000 7faa24d99640 4 set_mon_vals no callback set 2024-09-11T01:40:22.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.579+0000 7faa2cacf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:40:22.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.579+0000 7faa2cacf640 1 -- 172.21.15.12:0/2424507174 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa28137280 con 0x7faa28137e50 2024-09-11T01:40:22.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.579+0000 7faa2cacf640 10 monclient: _renew_subs 2024-09-11T01:40:22.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.579+0000 7faa2cacf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:40:22.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.579+0000 7faa2cacf640 1 -- 172.21.15.12:0/2424507174 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa281254f0 con 0x7faa28137e50 2024-09-11T01:40:22.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.579+0000 7faa16ffd640 1 -- 172.21.15.12:0/2424507174 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7faa1c03a030 con 0x7faa28137e50 2024-09-11T01:40:22.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.580+0000 7faa2cacf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.580+0000 7faa2cacf640 1 --2- 172.21.15.12:0/2424507174 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa28121c10 0x7faa28121ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.580+0000 7faa2cacf640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.580+0000 7faa2cacf640 1 -- 172.21.15.12:0/2424507174 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7faa2806a060 con 0x7faa28121c10 2024-09-11T01:40:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.580+0000 7faa2659c640 1 --2- 172.21.15.12:0/2424507174 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa28121c10 0x7faa28121ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.580+0000 7faa2659c640 10 monclient: get_auth_request con 0x7faa28121c10 auth_method 0 2024-09-11T01:40:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.580+0000 7faa2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.580+0000 7faa2659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.580+0000 7faa2659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.580+0000 7faa16ffd640 1 --2- 172.21.15.12:0/2424507174 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7faa0803e210 0x7faa080406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.580+0000 7faa2659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.580+0000 7faa2659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.580+0000 7faa2659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.580+0000 7faa16ffd640 1 -- 172.21.15.12:0/2424507174 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(377..377 src has 285..377) ==== 4755+0+0 (secure 0 0 0) 0x7faa1c073180 con 0x7faa28137e50 2024-09-11T01:40:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.580+0000 7faa2659c640 10 monclient(hunting): handle_auth_done global_id 114215 payload 824 2024-09-11T01:40:22.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.580+0000 7faa2659c640 1 -- 172.21.15.12:0/2424507174 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faa28135900 con 0x7faa28121c10 2024-09-11T01:40:22.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.580+0000 7faa26d9d640 1 --2- 172.21.15.12:0/2424507174 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7faa0803e210 0x7faa080406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.580+0000 7faa26d9d640 10 monclient: get_auth_request con 0x7faa0803e210 auth_method 0 2024-09-11T01:40:22.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.582+0000 7faa2659c640 1 -- 172.21.15.12:0/2424507174 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa28121c10 msgr2=0x7faa28121ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:22.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.582+0000 7faa2659c640 1 --2- 172.21.15.12:0/2424507174 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa28121c10 0x7faa28121ff0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7faa180118b0 tx=0x7faa18011d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:22.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.582+0000 7faa16ffd640 1 -- 172.21.15.12:0/2424507174 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa18019070 con 0x7faa28121c10 2024-09-11T01:40:22.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.582+0000 7faa16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:22.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.582+0000 7faa16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:40:22.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.582+0000 7faa16ffd640 10 monclient: dump: 2024-09-11T01:40:22.598 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:22.598 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:22.599 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.599 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.599 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:22.599 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.599 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:22.599 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:22.599 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:22.599 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.583+0000 7faa16ffd640 1 -- 172.21.15.12:0/2424507174 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7faa1800f890 con 0x7faa28121c10 2024-09-11T01:40:22.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.583+0000 7faa16ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:22.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.583+0000 7faa16ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:40:22.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.583+0000 7faa16ffd640 1 -- 172.21.15.12:0/2424507174 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa28121c10 msgr2=0x7faa28121ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:22.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.583+0000 7faa16ffd640 1 --2- 172.21.15.12:0/2424507174 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa28121c10 0x7faa28121ff0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7faa180118b0 tx=0x7faa18011d70 comp rx=0 tx=0).stop 2024-09-11T01:40:22.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.583+0000 7faa16ffd640 1 -- 172.21.15.12:0/2424507174 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa28121c10 msgr2=0x7faa28121ff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:40:22.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.583+0000 7faa16ffd640 1 --2- 172.21.15.12:0/2424507174 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa28121c10 0x7faa28121ff0 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7faa180118b0 tx=0x7faa18011d70 comp rx=0 tx=0).stop 2024-09-11T01:40:22.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.583+0000 7faa26d9d640 1 --2- 172.21.15.12:0/2424507174 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7faa0803e210 0x7faa080406d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7faa28123a80 tx=0x7faa20009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:22.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.589+0000 7faa2cacf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:40:22.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.589+0000 7faa2cacf640 1 --2- 172.21.15.12:0/2424507174 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa28122530 0x7faa280a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.589+0000 7faa2cacf640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:22.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.589+0000 7faa2cacf640 1 -- 172.21.15.12:0/2424507174 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7faa28123cf0 con 0x7faa28122530 2024-09-11T01:40:22.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.589+0000 7faa2659c640 1 --2- 172.21.15.12:0/2424507174 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa28122530 0x7faa280a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.589+0000 7faa2659c640 10 monclient: get_auth_request con 0x7faa28122530 auth_method 0 2024-09-11T01:40:22.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.589+0000 7faa2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:22.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.589+0000 7faa2659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:22.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.589+0000 7faa2659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:22.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.589+0000 7faa2659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:22.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.589+0000 7faa2659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:22.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.589+0000 7faa2659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:22.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.589+0000 7faa2659c640 10 monclient(hunting): handle_auth_done global_id 114218 payload 824 2024-09-11T01:40:22.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.589+0000 7faa2659c640 1 -- 172.21.15.12:0/2424507174 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faa28069950 con 0x7faa28122530 2024-09-11T01:40:22.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.591+0000 7faa2659c640 1 -- 172.21.15.12:0/2424507174 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa28122530 msgr2=0x7faa280a0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:22.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.591+0000 7faa2659c640 1 --2- 172.21.15.12:0/2424507174 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa28122530 0x7faa280a0430 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7faa1801b000 tx=0x7faa180046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:22.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.591+0000 7faa16ffd640 1 -- 172.21.15.12:0/2424507174 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa18019040 con 0x7faa28122530 2024-09-11T01:40:22.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.591+0000 7faa16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:22.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.591+0000 7faa16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:40:22.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.591+0000 7faa16ffd640 10 monclient: dump: 2024-09-11T01:40:22.601 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:22.601 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:22.601 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.601 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.601 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:22.601 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.601 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:22.602 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:22.602 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:22.602 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.592+0000 7faa16ffd640 1 -- 172.21.15.12:0/2424507174 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7faa1801e660 con 0x7faa28122530 2024-09-11T01:40:22.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.592+0000 7faa16ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:40:22.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.592+0000 7faa16ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:40:22.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.592+0000 7faa16ffd640 1 -- 172.21.15.12:0/2424507174 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa28122530 msgr2=0x7faa280a0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:22.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.592+0000 7faa16ffd640 1 --2- 172.21.15.12:0/2424507174 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa28122530 0x7faa280a0430 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7faa1801b000 tx=0x7faa180046c0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.592+0000 7faa16ffd640 1 -- 172.21.15.12:0/2424507174 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa28122530 msgr2=0x7faa280a0430 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:40:22.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.592+0000 7faa16ffd640 1 --2- 172.21.15.12:0/2424507174 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa28122530 0x7faa280a0430 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7faa1801b000 tx=0x7faa180046c0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.602 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:40:22.602 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:40:22.603 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:40:22.603 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:40:22.603 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 106, 2024-09-11T01:40:22.603 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 68388, 2024-09-11T01:40:22.603 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:40:22.603 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:40:22.603 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:40:22.603 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:40:22.603 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:22.603 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:40:22.603 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:40:22.603 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:40:22.603 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:40:22.603 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:40:22.603 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:40:22.604 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:40:22.604 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:40:22.604 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:40:22.604 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:40:22.604 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:40:22.604 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:40:22.604 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:40:22.604 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:40:22.604 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:40:22.604 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:22.604 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:40:22.604 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:40:22.604 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:40:22.604 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:40:22.604 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:40:22.605 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:40:22.605 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:40:22.605 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:40:22.605 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:40:22.605 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:40:22.605 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:40:22.605 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:40:22.605 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:40:22.605 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:22.605 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.605 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:40:22.605 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:40:22.605 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:40:22.605 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:40:22.605 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:40:22.606 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:40:22.606 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:40:22.606 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:40:22.606 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:40:22.606 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:40:22.606 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:40:22.606 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:40:22.606 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:40:22.606 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:40:22.606 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:40:22.606 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:40:22.606 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:40:22.606 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:40:22.607 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:40:22.607 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:40:22.607 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:40:22.607 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:40:22.607 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:40:22.607 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:40:22.607 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:40:22.607 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:40:22.607 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:40:22.607 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:40:22.607 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:22.607 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:40:22.607 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.608 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:40:22.608 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.608 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:40:22.608 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:40:22.608 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:40:22.608 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:40:22.608 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.608 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:40:22.608 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:40:22.608 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:22.608 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.608 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.608 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:40:22.608 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:40:22.609 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:22.609 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:22.609 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:22.609 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.609 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:40:22.609 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:40:22.609 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:40:22.609 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:40:22.609 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:40:22.609 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.609 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.609 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:40:22.609 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:40:22.609 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:40:22.610 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:40:22.610 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.610 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:40:22.610 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:40:22.610 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:22.610 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.610 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.610 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:40:22.610 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:40:22.610 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:22.610 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:22.610 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:22.610 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.610 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:40:22.610 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:40:22.611 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:40:22.611 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:40:22.611 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:40:22.611 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.611 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.611 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:40:22.611 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:40:22.611 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:40:22.611 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:40:22.611 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.611 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:40:22.611 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:40:22.611 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:22.612 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.612 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.612 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:40:22.612 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:40:22.612 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:22.612 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:22.612 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:22.612 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.612 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:40:22.612 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:40:22.612 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:40:22.612 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:40:22.612 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:40:22.612 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:22.612 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:22.613 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.613 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:40:22.613 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:40:22.613 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.613 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:40:22.613 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:40:22.613 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:40:22.613 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:22.613 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:22.613 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:40:22.613 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.613 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:40:22.613 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:40:22.613 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:40:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:40:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:40:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:40:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:40:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:40:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:40:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:40:22.614 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:40:22.615 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:22.615 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:22.615 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.615 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:40:22.615 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:40:22.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.592+0000 7faa2cacf640 1 -- 172.21.15.12:0/2424507174 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7faa0803e210 msgr2=0x7faa080406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:22.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.592+0000 7faa2cacf640 1 --2- 172.21.15.12:0/2424507174 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7faa0803e210 0x7faa080406d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7faa28123a80 tx=0x7faa20009040 comp rx=0 tx=0).stop 2024-09-11T01:40:22.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.592+0000 7faa2cacf640 10 monclient: shutdown 2024-09-11T01:40:22.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.592+0000 7faa2cacf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:22.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.592+0000 7faa2cacf640 1 -- 172.21.15.12:0/2424507174 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa28137e50 msgr2=0x7faa281cd910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:22.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.592+0000 7faa2cacf640 1 --2- 172.21.15.12:0/2424507174 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa28137e50 0x7faa281cd910 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7faa1c0090b0 tx=0x7faa1c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.592+0000 7faa26d9d640 1 -- 172.21.15.12:0/2424507174 reap_dead start 2024-09-11T01:40:22.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.592+0000 7faa26d9d640 1 -- 172.21.15.12:0/2424507174 reap_dead start 2024-09-11T01:40:22.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.592+0000 7faa2cacf640 1 -- 172.21.15.12:0/2424507174 shutdown_connections 2024-09-11T01:40:22.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.592+0000 7faa2cacf640 1 -- 172.21.15.12:0/2424507174 >> 172.21.15.12:0/2424507174 conn(0x7faa280a7020 msgr2=0x7faa28137620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:22.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.592+0000 7faa2cacf640 1 -- 172.21.15.12:0/2424507174 shutdown_connections 2024-09-11T01:40:22.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.592+0000 7faa2cacf640 1 -- 172.21.15.12:0/2424507174 wait complete. 2024-09-11T01:40:22.617 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:40:22.618 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:40:22.618 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:40:22.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.718+0000 7fc0ee35a640 10 monclient: get_monmap_and_config 2024-09-11T01:40:22.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.718+0000 7fc0ee35a640 10 monclient: build_initial_monmap 2024-09-11T01:40:22.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.718+0000 7fc0ee35a640 10 monclient: monmap: 2024-09-11T01:40:22.720 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:22.720 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:22.720 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:40:22.719118+0000 2024-09-11T01:40:22.720 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:40:22.719118+0000 2024-09-11T01:40:22.720 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:22.720 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.721 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:40:22.721 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:40:22.721 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:40:22.721 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:40:22.721 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:40:22.721 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ee35a640 1 Processor -- start 2024-09-11T01:40:22.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ee35a640 1 -- start start 2024-09-11T01:40:22.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ee35a640 10 monclient: init 2024-09-11T01:40:22.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ee35a640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:22.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ee35a640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-11T01:40:22.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ee35a640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc0e80a2db0 0x7fc0e806f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ee35a640 10 monclient(hunting): picked mon.noname-c con 0x7fc0e80a2db0 addr v2:172.21.15.71:3300/0 2024-09-11T01:40:22.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ee35a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc0e806ded0 addr v1:172.21.15.71:6789/0 2024-09-11T01:40:22.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ee35a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0e80a38a0 0x7fc0e813f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ee35a640 10 monclient(hunting): picked mon.noname-b con 0x7fc0e80a38a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:40:22.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ee35a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0e80a4650 con 0x7fc0e806ded0 2024-09-11T01:40:22.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ee35a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:22.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ee35a640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fc0e806f1a0 con 0x7fc0e80a2db0 2024-09-11T01:40:22.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ee35a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:22.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ee35a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc0e806eec0 con 0x7fc0e80a38a0 2024-09-11T01:40:22.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ee35a640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:22.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ee35a640 10 monclient(hunting): authenticate will time out at 1429.161712s 2024-09-11T01:40:22.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ec8f7640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0e80a38a0 0x7fc0e813f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ec8f7640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0e80a38a0 0x7fc0e813f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:49278/0 (socket says 172.21.15.12:49278) 2024-09-11T01:40:22.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ec8f7640 1 -- 172.21.15.12:0/2386275431 learned_addr learned my addr 172.21.15.12:0/2386275431 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:40:22.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ec8f7640 10 monclient(hunting): get_auth_request con 0x7fc0e80a38a0 auth_method 0 2024-09-11T01:40:22.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ec8f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:22.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ec8f7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:22.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ec8f7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:22.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ec8f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:22.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ec8f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:22.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.719+0000 7fc0ec8f7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:22.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.720+0000 7fc0ec8f7640 10 monclient(hunting): handle_auth_done global_id 114221 payload 274 2024-09-11T01:40:22.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.720+0000 7fc0e7fff640 1 --2- 172.21.15.12:0/2386275431 >> v2:172.21.15.71:3300/0 conn(0x7fc0e80a2db0 0x7fc0e806f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.720+0000 7fc0ec8f7640 1 -- 172.21.15.12:0/2386275431 >> v2:172.21.15.71:3300/0 conn(0x7fc0e80a2db0 msgr2=0x7fc0e806f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:40:22.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.720+0000 7fc0ec8f7640 1 --2- 172.21.15.12:0/2386275431 >> v2:172.21.15.71:3300/0 conn(0x7fc0e80a2db0 0x7fc0e806f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.720+0000 7fc0ec8f7640 1 -- 172.21.15.12:0/2386275431 >> v1:172.21.15.71:6789/0 conn(0x7fc0e806ded0 legacy=0x7fc0e80a3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:40:22.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.720+0000 7fc0ec8f7640 10 monclient: _finish_hunting 0 2024-09-11T01:40:22.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.720+0000 7fc0ec8f7640 1 monclient: found mon.noname-b 2024-09-11T01:40:22.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.720+0000 7fc0ec8f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:22.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.720+0000 7fc0ec8f7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:40:22.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.720+0000 7fc0ec8f7640 1 -- 172.21.15.12:0/2386275431 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e806e2e0 con 0x7fc0e80a38a0 2024-09-11T01:40:22.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.720+0000 7fc0e7fff640 10 monclient: get_auth_request con 0x7fc0e80a2db0 auth_method 0 2024-09-11T01:40:22.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.720+0000 7fc0e7fff640 1 --2- 172.21.15.12:0/2386275431 >> v2:172.21.15.71:3300/0 conn(0x7fc0e80a2db0 0x7fc0e806f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:40:22.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0ec8f7640 1 -- 172.21.15.12:0/2386275431 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0e80a38a0 msgr2=0x7fc0e813f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:22.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0ec8f7640 1 --2- 172.21.15.12:0/2386275431 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0e80a38a0 0x7fc0e813f3c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0009e50 tx=0x7fc0e002a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b62d41aad1b2447d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:22.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0e6ffd640 1 -- 172.21.15.12:0/2386275431 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e0036070 con 0x7fc0e80a38a0 2024-09-11T01:40:22.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:22.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0e6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:40:22.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0e6ffd640 10 monclient: dump: 2024-09-11T01:40:22.732 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:22.732 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:22.732 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.732 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.732 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:22.732 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.733 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:22.733 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:22.733 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:22.733 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0e6ffd640 10 monclient: _finish_auth 0 2024-09-11T01:40:22.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0e6ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:40:22.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:22.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0e6ffd640 1 -- 172.21.15.12:0/2386275431 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0e0007800 con 0x7fc0e80a38a0 2024-09-11T01:40:22.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0ee35a640 5 monclient: authenticate success, global_id 114221 2024-09-11T01:40:22.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0ee35a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:40:22.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0e6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:22.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0e6ffd640 1 -- 172.21.15.12:0/2386275431 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e0007bd0 con 0x7fc0e80a38a0 2024-09-11T01:40:22.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0ee35a640 10 monclient: get_monmap_and_config success 2024-09-11T01:40:22.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0ee35a640 4 set_mon_vals no callback set 2024-09-11T01:40:22.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:22.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:40:22.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0e6ffd640 10 monclient: dump: 2024-09-11T01:40:22.734 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:22.734 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:22.734 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.734 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.734 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:22.734 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.734 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:22.734 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:22.734 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:22.734 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0ee35a640 10 monclient: shutdown 2024-09-11T01:40:22.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0ee35a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:22.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0ee35a640 1 -- 172.21.15.12:0/2386275431 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0e80a38a0 msgr2=0x7fc0e813f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:22.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0ee35a640 1 --2- 172.21.15.12:0/2386275431 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0e80a38a0 0x7fc0e813f3c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0009e50 tx=0x7fc0e002a5f0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0ee35a640 1 -- 172.21.15.12:0/2386275431 shutdown_connections 2024-09-11T01:40:22.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0ee35a640 1 --2- 172.21.15.12:0/2386275431 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0e80a38a0 0x7fc0e813f3c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0ee35a640 1 --2- 172.21.15.12:0/2386275431 >> v2:172.21.15.71:3300/0 conn(0x7fc0e80a2db0 0x7fc0e806f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.723+0000 7fc0ee35a640 1 -- 172.21.15.12:0/2386275431 >> 172.21.15.12:0/2386275431 conn(0x7fc0e80a83b0 msgr2=0x7fc0e80a87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:22.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.724+0000 7fc0ee35a640 1 -- 172.21.15.12:0/2386275431 shutdown_connections 2024-09-11T01:40:22.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.724+0000 7fc0ee35a640 1 -- 172.21.15.12:0/2386275431 wait complete. 2024-09-11T01:40:22.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.724+0000 7fc0ee35a640 10 monclient: build_initial_monmap 2024-09-11T01:40:22.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.724+0000 7fc0ee35a640 10 monclient: monmap: 2024-09-11T01:40:22.735 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:22.735 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:40:22.736 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:40:22.736 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:40:22.736 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:22.736 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.736 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:40:22.736 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:40:22.736 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:40:22.736 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.724+0000 7fc0ee35a640 1 Processor -- start 2024-09-11T01:40:22.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0ee35a640 1 -- start start 2024-09-11T01:40:22.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0ee35a640 10 monclient: init 2024-09-11T01:40:22.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0ee35a640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:22.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0ee35a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:40:22.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0ee35a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0e806ded0 0x7fc0e8137ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0ee35a640 10 monclient(hunting): picked mon.noname-c con 0x7fc0e806ded0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:40:22.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0ee35a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e80a2db0 0x7fc0e8138420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0ee35a640 10 monclient(hunting): picked mon.noname-b con 0x7fc0e80a2db0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:40:22.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0ee35a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0e80a38a0 0x7fc0e813b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0ee35a640 10 monclient(hunting): picked mon.noname-a con 0x7fc0e80a38a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:40:22.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0ee35a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:22.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0ee35a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e813fc90 con 0x7fc0e80a38a0 2024-09-11T01:40:22.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0ee35a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:22.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0ee35a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e813fb10 con 0x7fc0e80a2db0 2024-09-11T01:40:22.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0ee35a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:22.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0ee35a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc0e813fe10 con 0x7fc0e806ded0 2024-09-11T01:40:22.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0ee35a640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:22.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0ee35a640 10 monclient(hunting): authenticate will time out at 1429.167582s 2024-09-11T01:40:22.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0e7fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0e806ded0 0x7fc0e8137ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0e7fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0e806ded0 0x7fc0e8137ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:49292/0 (socket says 172.21.15.12:49292) 2024-09-11T01:40:22.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0ec8f7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0e80a38a0 0x7fc0e813b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0e7fff640 1 -- 172.21.15.12:0/2857857419 learned_addr learned my addr 172.21.15.12:0/2857857419 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:40:22.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0e7fff640 10 monclient(hunting): get_auth_request con 0x7fc0e806ded0 auth_method 0 2024-09-11T01:40:22.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:22.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:22.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:22.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0ec8f7640 10 monclient(hunting): get_auth_request con 0x7fc0e80a38a0 auth_method 0 2024-09-11T01:40:22.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0ec8f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:22.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0ec8f7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:22.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0ec8f7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:22.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0e77fe640 1 --2- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e80a2db0 0x7fc0e8138420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:22.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:22.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:22.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0e77fe640 10 monclient(hunting): get_auth_request con 0x7fc0e80a2db0 auth_method 0 2024-09-11T01:40:22.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:22.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:22.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:22.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0ec8f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:22.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0ec8f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:22.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0ec8f7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:22.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:22.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:22.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:22.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0e7fff640 10 monclient(hunting): handle_auth_done global_id 114224 payload 824 2024-09-11T01:40:22.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0e7fff640 1 -- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e80a2db0 msgr2=0x7fc0e8138420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:22.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.725+0000 7fc0e7fff640 1 --2- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e80a2db0 0x7fc0e8138420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.726+0000 7fc0e7fff640 1 -- 172.21.15.12:0/2857857419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0e80a38a0 msgr2=0x7fc0e813b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:22.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.726+0000 7fc0e7fff640 1 --2- 172.21.15.12:0/2857857419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0e80a38a0 0x7fc0e813b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:22.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.726+0000 7fc0e7fff640 10 monclient: _finish_hunting 0 2024-09-11T01:40:22.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.726+0000 7fc0e7fff640 1 monclient: found mon.noname-c 2024-09-11T01:40:22.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.726+0000 7fc0e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:22.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.726+0000 7fc0e7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:40:22.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.726+0000 7fc0e7fff640 1 -- 172.21.15.12:0/2857857419 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e813ee00 con 0x7fc0e806ded0 2024-09-11T01:40:22.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.726+0000 7fc0ec8f7640 1 --2- 172.21.15.12:0/2857857419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0e80a38a0 0x7fc0e813b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:40:22.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.728+0000 7fc0e7fff640 1 -- 172.21.15.12:0/2857857419 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0e806ded0 msgr2=0x7fc0e8137ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:22.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.728+0000 7fc0e7fff640 1 --2- 172.21.15.12:0/2857857419 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0e806ded0 0x7fc0e8137ee0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc0d800c820 tx=0x7fc0d800cc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:22.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.728+0000 7fc0e4ff9640 1 -- 172.21.15.12:0/2857857419 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d8015070 con 0x7fc0e806ded0 2024-09-11T01:40:22.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.728+0000 7fc0e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:22.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.728+0000 7fc0e4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:40:22.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.728+0000 7fc0e4ff9640 10 monclient: dump: 2024-09-11T01:40:22.741 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:22.741 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:22.741 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.741 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.741 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:22.741 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.741 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:22.741 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:22.741 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:22.741 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.728+0000 7fc0e4ff9640 10 monclient: _finish_auth 0 2024-09-11T01:40:22.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.728+0000 7fc0e4ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:40:22.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.728+0000 7fc0e4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:22.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.728+0000 7fc0e4ff9640 1 -- 172.21.15.12:0/2857857419 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0d8007c40 con 0x7fc0e806ded0 2024-09-11T01:40:22.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.728+0000 7fc0ee35a640 5 monclient: authenticate success, global_id 114224 2024-09-11T01:40:22.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.728+0000 7fc0ee35a640 10 monclient: _renew_subs 2024-09-11T01:40:22.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.728+0000 7fc0ee35a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:40:22.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.728+0000 7fc0ee35a640 1 -- 172.21.15.12:0/2857857419 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0e813f0e0 con 0x7fc0e806ded0 2024-09-11T01:40:22.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.728+0000 7fc0e4ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:22.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.728+0000 7fc0e4ff9640 1 -- 172.21.15.12:0/2857857419 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d80113f0 con 0x7fc0e806ded0 2024-09-11T01:40:22.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.728+0000 7fc0e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:22.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.728+0000 7fc0e4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:40:22.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.728+0000 7fc0e4ff9640 10 monclient: dump: 2024-09-11T01:40:22.742 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:22.742 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:22.742 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.742 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.742 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:22.743 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.728+0000 7fc0ee35a640 10 monclient: _renew_subs 2024-09-11T01:40:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.728+0000 7fc0ee35a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:40:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.728+0000 7fc0ee35a640 1 -- 172.21.15.12:0/2857857419 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0e809f5a0 con 0x7fc0e806ded0 2024-09-11T01:40:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.728+0000 7fc0e67fc640 4 set_mon_vals no callback set 2024-09-11T01:40:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.729+0000 7fc0ee35a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.729+0000 7fc0ee35a640 1 --2- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e813c0b0 0x7fc0e802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.729+0000 7fc0ee35a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.729+0000 7fc0ee35a640 1 -- 172.21.15.12:0/2857857419 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e806a060 con 0x7fc0e813c0b0 2024-09-11T01:40:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.729+0000 7fc0e4ff9640 1 -- 172.21.15.12:0/2857857419 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fc0d8011590 con 0x7fc0e806ded0 2024-09-11T01:40:22.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.729+0000 7fc0e4ff9640 1 --2- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc0d403e260 0x7fc0d4040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.729+0000 7fc0e77fe640 1 --2- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e813c0b0 0x7fc0e802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.729+0000 7fc0e77fe640 10 monclient: get_auth_request con 0x7fc0e813c0b0 auth_method 0 2024-09-11T01:40:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.729+0000 7fc0e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.729+0000 7fc0e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.729+0000 7fc0e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.729+0000 7fc0e4ff9640 1 -- 172.21.15.12:0/2857857419 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(377..377 src has 285..377) ==== 4755+0+0 (secure 0 0 0) 0x7fc0d8053120 con 0x7fc0e806ded0 2024-09-11T01:40:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.729+0000 7fc0e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.729+0000 7fc0e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.729+0000 7fc0e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.729+0000 7fc0ec8f7640 1 --2- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc0d403e260 0x7fc0d4040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.729+0000 7fc0ec8f7640 10 monclient: get_auth_request con 0x7fc0d403e260 auth_method 0 2024-09-11T01:40:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.729+0000 7fc0e77fe640 10 monclient(hunting): handle_auth_done global_id 124114 payload 824 2024-09-11T01:40:22.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.729+0000 7fc0e77fe640 1 -- 172.21.15.12:0/2857857419 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc0e806f8a0 con 0x7fc0e813c0b0 2024-09-11T01:40:22.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.729+0000 7fc0e77fe640 1 -- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e813c0b0 msgr2=0x7fc0e802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:40:22.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.729+0000 7fc0e77fe640 1 -- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e813c0b0 msgr2=0x7fc0e802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:40:22.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.729+0000 7fc0e77fe640 1 --2- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e813c0b0 0x7fc0e802d440 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc00f520 tx=0x7fc0dc00f9e0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:40:22.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.730+0000 7fc0e77fe640 1 --2- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e813c0b0 0x7fc0e802d440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:40:22.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.732+0000 7fc0ec8f7640 1 --2- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc0d403e260 0x7fc0d4040720 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0033000 tx=0x7fc0e0009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:22.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.931+0000 7fc0e77fe640 1 --2- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e813c0b0 0x7fc0e802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.931+0000 7fc0e77fe640 10 monclient: get_auth_request con 0x7fc0e813c0b0 auth_method 0 2024-09-11T01:40:22.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.931+0000 7fc0e77fe640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:22.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.931+0000 7fc0e77fe640 10 monclient: _init_auth method 2 2024-09-11T01:40:22.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.931+0000 7fc0e77fe640 10 monclient: _init_auth already have auth, reseting 2024-09-11T01:40:22.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.931+0000 7fc0e77fe640 10 monclient: handle_auth_reply_more payload 9 2024-09-11T01:40:22.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.931+0000 7fc0e77fe640 10 monclient: handle_auth_reply_more payload_len 9 2024-09-11T01:40:22.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.931+0000 7fc0e77fe640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-09-11T01:40:22.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.931+0000 7fc0e77fe640 10 monclient: handle_auth_done global_id 124114 payload 293 2024-09-11T01:40:22.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.934+0000 7fc0e77fe640 1 -- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e813c0b0 msgr2=0x7fc0e802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:22.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.934+0000 7fc0e77fe640 1 --2- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e813c0b0 0x7fc0e802d440 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc00f520 tx=0x7fc0dc009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:22.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.934+0000 7fc0e4ff9640 1 -- 172.21.15.12:0/2857857419 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc00d5c0 con 0x7fc0e813c0b0 2024-09-11T01:40:22.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.934+0000 7fc0e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:22.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.934+0000 7fc0e4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:40:22.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.934+0000 7fc0e4ff9640 10 monclient: dump: 2024-09-11T01:40:22.936 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:22.936 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:22.936 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.936 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.936 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:22.936 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.936 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:22.936 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:22.936 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:22.936 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.935+0000 7fc0e4ff9640 1 -- 172.21.15.12:0/2857857419 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc0dc019070 con 0x7fc0e813c0b0 2024-09-11T01:40:22.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.935+0000 7fc0e4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:22.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.935+0000 7fc0e4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:40:22.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.935+0000 7fc0e4ff9640 1 -- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e813c0b0 msgr2=0x7fc0e802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:22.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.935+0000 7fc0e4ff9640 1 --2- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e813c0b0 0x7fc0e802d440 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc00f520 tx=0x7fc0dc009040 comp rx=0 tx=0).stop 2024-09-11T01:40:22.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.935+0000 7fc0e4ff9640 1 -- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e813c0b0 msgr2=0x7fc0e802d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:40:22.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.935+0000 7fc0e4ff9640 1 --2- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e813c0b0 0x7fc0e802d440 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc00f520 tx=0x7fc0dc009040 comp rx=0 tx=0).stop 2024-09-11T01:40:22.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.941+0000 7fc0ee35a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:40:22.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.941+0000 7fc0ee35a640 1 --2- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e813c490 0x7fc0e802d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:22.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.941+0000 7fc0ee35a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:22.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.941+0000 7fc0ee35a640 1 -- 172.21.15.12:0/2857857419 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e813dbb0 con 0x7fc0e813c490 2024-09-11T01:40:22.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.941+0000 7fc0e77fe640 1 --2- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e813c490 0x7fc0e802d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:22.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.941+0000 7fc0e77fe640 10 monclient: get_auth_request con 0x7fc0e813c490 auth_method 0 2024-09-11T01:40:22.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.941+0000 7fc0e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:22.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.941+0000 7fc0e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:22.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.941+0000 7fc0e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:22.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.941+0000 7fc0e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:22.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.941+0000 7fc0e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:22.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.941+0000 7fc0e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:22.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.941+0000 7fc0e77fe640 10 monclient(hunting): handle_auth_done global_id 124117 payload 824 2024-09-11T01:40:22.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.941+0000 7fc0e77fe640 1 -- 172.21.15.12:0/2857857419 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc0e809d9c0 con 0x7fc0e813c490 2024-09-11T01:40:22.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.944+0000 7fc0e77fe640 1 -- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e813c490 msgr2=0x7fc0e802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:22.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.944+0000 7fc0e77fe640 1 --2- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e813c490 0x7fc0e802d980 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc00f520 tx=0x7fc0dc001020 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:22.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.944+0000 7fc0e4ff9640 1 -- 172.21.15.12:0/2857857419 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc006990 con 0x7fc0e813c490 2024-09-11T01:40:22.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.944+0000 7fc0e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:22.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.944+0000 7fc0e4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:40:22.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.944+0000 7fc0e4ff9640 10 monclient: dump: 2024-09-11T01:40:22.946 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:22.946 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:22.946 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.946 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:22.946 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:22.947 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:22.947 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:22.947 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:22.947 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:22.947 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:22.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.944+0000 7fc0e4ff9640 1 -- 172.21.15.12:0/2857857419 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4540 (secure 0 0 0) 0x7fc0dc006990 con 0x7fc0e813c490 2024-09-11T01:40:22.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.944+0000 7fc0e4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:40:22.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.944+0000 7fc0e4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:40:22.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.944+0000 7fc0e4ff9640 1 -- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e813c490 msgr2=0x7fc0e802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:22.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.944+0000 7fc0e4ff9640 1 --2- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e813c490 0x7fc0e802d980 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc00f520 tx=0x7fc0dc001020 comp rx=0 tx=0).stop 2024-09-11T01:40:22.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.944+0000 7fc0e4ff9640 1 -- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e813c490 msgr2=0x7fc0e802d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:40:22.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.944+0000 7fc0e4ff9640 1 --2- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e813c490 0x7fc0e802d980 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc00f520 tx=0x7fc0dc001020 comp rx=0 tx=0).stop 2024-09-11T01:40:22.948 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:40:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:40:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:40:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:40:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 106, 2024-09-11T01:40:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 7737, 2024-09-11T01:40:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:40:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:40:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:40:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:40:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:40:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:40:22.948 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:40:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:40:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:40:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:40:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:40:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:40:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:40:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:40:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:40:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:40:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:40:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:40:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:40:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:22.949 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:40:22.950 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:40:22.950 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:40:22.950 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:40:22.950 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:40:22.950 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:40:22.950 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:40:22.950 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:40:22.950 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:40:22.950 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:40:22.950 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:40:22.950 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:40:22.950 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:40:22.950 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:22.950 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.950 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:40:22.951 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:40:22.951 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:40:22.951 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:40:22.951 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:40:22.951 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:40:22.951 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:40:22.951 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:40:22.951 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:40:22.951 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:40:22.951 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:40:22.951 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:40:22.951 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:40:22.951 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:40:22.951 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:40:22.951 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:40:22.952 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:40:22.952 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:40:22.952 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:40:22.952 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:40:22.952 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:40:22.952 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:40:22.952 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:40:22.952 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:40:22.952 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:40:22.952 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:40:22.952 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:40:22.953 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:40:22.953 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:22.953 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:40:22.953 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.953 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:40:22.953 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.953 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:40:22.953 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:40:22.953 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:40:22.953 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:40:22.953 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.953 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:40:22.953 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:40:22.954 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:22.954 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.954 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.954 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:40:22.954 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:40:22.954 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:22.954 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:22.954 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:22.954 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.954 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:40:22.954 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:40:22.954 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:40:22.954 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:40:22.954 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:40:22.955 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.955 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.955 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:40:22.955 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:40:22.955 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:40:22.955 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:40:22.955 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.955 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:40:22.955 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:40:22.955 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:22.955 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.955 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.955 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:40:22.955 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:40:22.956 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:22.956 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:22.956 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:22.956 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.956 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:40:22.956 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:40:22.956 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:40:22.956 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:40:22.956 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:40:22.956 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.956 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.956 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:40:22.956 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:40:22.956 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:40:22.956 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:40:22.957 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.957 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:40:22.957 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:40:22.957 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:22.957 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.957 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.957 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:40:22.957 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:40:22.957 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:22.957 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:22.957 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:22.957 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.957 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:40:22.957 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:40:22.957 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:40:22.958 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:40:22.958 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:40:22.958 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:22.958 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:22.958 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.958 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:40:22.958 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:40:22.958 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.958 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:40:22.958 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:40:22.958 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:40:22.958 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:22.959 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:22.959 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:40:22.959 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:22.959 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:40:22.959 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:40:22.959 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:40:22.959 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:22.959 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:22.959 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:22.959 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:40:22.959 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:40:22.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.945+0000 7fc0ee35a640 1 -- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc0d403e260 msgr2=0x7fc0d4040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.945+0000 7fc0ee35a640 1 --2- 172.21.15.12:0/2857857419 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc0d403e260 0x7fc0d4040720 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0033000 tx=0x7fc0e0009000 comp rx=0 tx=0).stop 2024-09-11T01:40:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.945+0000 7fc0ee35a640 10 monclient: shutdown 2024-09-11T01:40:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.945+0000 7fc0ee35a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.945+0000 7fc0ee35a640 1 -- 172.21.15.12:0/2857857419 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0e806ded0 msgr2=0x7fc0e8137ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.945+0000 7fc0ee35a640 1 --2- 172.21.15.12:0/2857857419 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0e806ded0 0x7fc0e8137ee0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc0d800c820 tx=0x7fc0d800cc20 comp rx=0 tx=0).stop 2024-09-11T01:40:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.945+0000 7fc0ec8f7640 1 -- 172.21.15.12:0/2857857419 reap_dead start 2024-09-11T01:40:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.945+0000 7fc0ec8f7640 1 -- 172.21.15.12:0/2857857419 reap_dead start 2024-09-11T01:40:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.945+0000 7fc0ee35a640 1 -- 172.21.15.12:0/2857857419 shutdown_connections 2024-09-11T01:40:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.945+0000 7fc0ee35a640 1 -- 172.21.15.12:0/2857857419 >> 172.21.15.12:0/2857857419 conn(0x7fc0e80a83b0 msgr2=0x7fc0e80a1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.945+0000 7fc0ee35a640 1 -- 172.21.15.12:0/2857857419 shutdown_connections 2024-09-11T01:40:22.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:22.945+0000 7fc0ee35a640 1 -- 172.21.15.12:0/2857857419 wait complete. 2024-09-11T01:40:22.961 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:40:22.962 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:40:22.962 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:40:22.975 INFO:tasks.rados.rados.0.smithi071.stdout:1652: done (0 left) 2024-09-11T01:40:22.975 INFO:tasks.rados.rados.0.smithi071.stdout:1653: rollback oid 49 current snap is 180 2024-09-11T01:40:22.975 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 49 to 178 2024-09-11T01:40:22.975 INFO:tasks.rados.rados.0.smithi071.stdout:1654: snap_create 2024-09-11T01:40:22.976 INFO:tasks.rados.rados.0.smithi071.stdout:1653: finishing rollback tid 1 to smithi07134777-49 2024-09-11T01:40:22.976 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 0 (ObjNum 33 snap 0 seq_num 2369891536) dirty dne 2024-09-11T01:40:23.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.064+0000 7fc31260c640 10 monclient: get_monmap_and_config 2024-09-11T01:40:23.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.064+0000 7fc31260c640 10 monclient: build_initial_monmap 2024-09-11T01:40:23.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.064+0000 7fc31260c640 10 monclient: monmap: 2024-09-11T01:40:23.066 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:23.066 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:23.066 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:40:23.064795+0000 2024-09-11T01:40:23.066 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:40:23.064795+0000 2024-09-11T01:40:23.066 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:23.066 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:23.066 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:40:23.066 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:40:23.066 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:40:23.066 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:40:23.066 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:40:23.066 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:23.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.064+0000 7fc31260c640 1 Processor -- start 2024-09-11T01:40:23.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.064+0000 7fc31260c640 1 -- start start 2024-09-11T01:40:23.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.064+0000 7fc31260c640 10 monclient: init 2024-09-11T01:40:23.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc31260c640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:23.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc31260c640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-11T01:40:23.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc31260c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc30c137e50 0x7fc30c09d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:23.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc31260c640 10 monclient(hunting): picked mon.noname-a con 0x7fc30c137e50 addr v2:172.21.15.12:3300/0 2024-09-11T01:40:23.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc31260c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc30c138230 addr v1:172.21.15.71:6789/0 2024-09-11T01:40:23.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc31260c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc30c13d380 0x7fc30c13f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:23.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc31260c640 10 monclient(hunting): picked mon.noname-c con 0x7fc30c13d380 addr v2:172.21.15.71:3300/0 2024-09-11T01:40:23.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc31260c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc30c0a62d0 con 0x7fc30c138230 2024-09-11T01:40:23.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc31260c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:23.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc31260c640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc30c09bb40 con 0x7fc30c137e50 2024-09-11T01:40:23.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc31260c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:23.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc31260c640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fc30c09c4a0 con 0x7fc30c13d380 2024-09-11T01:40:23.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc31260c640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:23.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc31260c640 10 monclient(hunting): authenticate will time out at 1429.507541s 2024-09-11T01:40:23.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc30bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc30c137e50 0x7fc30c09d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:23.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc30bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc30c137e50 0x7fc30c09d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60932/0 (socket says 172.21.15.12:60932) 2024-09-11T01:40:23.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc30bfff640 1 -- 172.21.15.12:0/3564233975 learned_addr learned my addr 172.21.15.12:0/3564233975 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:40:23.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc30bfff640 10 monclient(hunting): get_auth_request con 0x7fc30c137e50 auth_method 0 2024-09-11T01:40:23.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc30bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:23.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc30bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:23.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc30bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:23.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc30bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:23.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc30bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:23.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc30bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:23.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc30bfff640 10 monclient(hunting): handle_auth_done global_id 94392 payload 274 2024-09-11T01:40:23.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc30bfff640 1 -- 172.21.15.12:0/3564233975 >> v2:172.21.15.71:3300/0 conn(0x7fc30c13d380 msgr2=0x7fc30c13f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:40:23.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc30bfff640 1 --2- 172.21.15.12:0/3564233975 >> v2:172.21.15.71:3300/0 conn(0x7fc30c13d380 0x7fc30c13f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:23.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc30bfff640 1 -- 172.21.15.12:0/3564233975 >> v1:172.21.15.71:6789/0 conn(0x7fc30c138230 legacy=0x7fc30c138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:40:23.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc30bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:40:23.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc30bfff640 1 monclient: found mon.noname-a 2024-09-11T01:40:23.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc30bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:23.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc30bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:40:23.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.065+0000 7fc30bfff640 1 -- 172.21.15.12:0/3564233975 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30c09c780 con 0x7fc30c137e50 2024-09-11T01:40:23.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.068+0000 7fc30bfff640 1 -- 172.21.15.12:0/3564233975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc30c137e50 msgr2=0x7fc30c09d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:23.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.068+0000 7fc30bfff640 1 --2- 172.21.15.12:0/3564233975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc30c137e50 0x7fc30c09d160 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc30c0ab580 tx=0x7fc2fc02a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fff88bd8ad6c506d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:23.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.068+0000 7fc30affd640 1 -- 172.21.15.12:0/3564233975 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc037070 con 0x7fc30c137e50 2024-09-11T01:40:23.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.068+0000 7fc30affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:23.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.068+0000 7fc30affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:40:23.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.068+0000 7fc30affd640 10 monclient: dump: 2024-09-11T01:40:23.078 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:23.078 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:23.078 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:23.078 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:23.078 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:23.078 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:23.078 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:23.078 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:23.078 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:23.078 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:23.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.068+0000 7fc30affd640 10 monclient: _finish_auth 0 2024-09-11T01:40:23.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.068+0000 7fc30affd640 10 monclient: _check_auth_tickets 2024-09-11T01:40:23.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.068+0000 7fc30affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:23.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.068+0000 7fc30affd640 1 -- 172.21.15.12:0/3564233975 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2fc032640 con 0x7fc30c137e50 2024-09-11T01:40:23.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.068+0000 7fc30affd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:23.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.068+0000 7fc30affd640 1 -- 172.21.15.12:0/3564233975 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc032a10 con 0x7fc30c137e50 2024-09-11T01:40:23.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.068+0000 7fc30affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:23.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.068+0000 7fc30affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:40:23.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.068+0000 7fc30affd640 10 monclient: dump: 2024-09-11T01:40:23.079 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:23.079 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:23.079 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:23.079 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:23.079 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:23.079 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:23.079 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:23.079 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:23.080 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:23.080 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:23.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.068+0000 7fc31260c640 5 monclient: authenticate success, global_id 94392 2024-09-11T01:40:23.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.068+0000 7fc31260c640 10 monclient: get_monmap_and_config success 2024-09-11T01:40:23.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.068+0000 7fc31260c640 4 set_mon_vals no callback set 2024-09-11T01:40:23.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.069+0000 7fc31260c640 10 monclient: shutdown 2024-09-11T01:40:23.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.069+0000 7fc31260c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:23.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.069+0000 7fc31260c640 1 -- 172.21.15.12:0/3564233975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc30c137e50 msgr2=0x7fc30c09d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:23.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.069+0000 7fc31260c640 1 --2- 172.21.15.12:0/3564233975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc30c137e50 0x7fc30c09d160 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc30c0ab580 tx=0x7fc2fc02a0a0 comp rx=0 tx=0).stop 2024-09-11T01:40:23.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.069+0000 7fc31260c640 1 -- 172.21.15.12:0/3564233975 shutdown_connections 2024-09-11T01:40:23.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.069+0000 7fc31260c640 1 --2- 172.21.15.12:0/3564233975 >> v2:172.21.15.71:3300/0 conn(0x7fc30c13d380 0x7fc30c13f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:23.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.069+0000 7fc31260c640 1 --2- 172.21.15.12:0/3564233975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc30c137e50 0x7fc30c09d160 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:23.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.069+0000 7fc31260c640 1 -- 172.21.15.12:0/3564233975 >> 172.21.15.12:0/3564233975 conn(0x7fc30c0a7020 msgr2=0x7fc30c0a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:23.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.069+0000 7fc31260c640 1 -- 172.21.15.12:0/3564233975 shutdown_connections 2024-09-11T01:40:23.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.069+0000 7fc31260c640 1 -- 172.21.15.12:0/3564233975 wait complete. 2024-09-11T01:40:23.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.070+0000 7fc31260c640 10 monclient: build_initial_monmap 2024-09-11T01:40:23.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.070+0000 7fc31260c640 10 monclient: monmap: 2024-09-11T01:40:23.081 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:23.081 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:40:23.081 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:40:23.081 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:40:23.081 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:23.081 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:23.081 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:40:23.081 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:40:23.081 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:40:23.081 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:23.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.070+0000 7fc31260c640 1 Processor -- start 2024-09-11T01:40:23.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.070+0000 7fc31260c640 1 -- start start 2024-09-11T01:40:23.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.070+0000 7fc31260c640 10 monclient: init 2024-09-11T01:40:23.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc31260c640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:23.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc31260c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:40:23.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc31260c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc30c137e50 0x7fc30c0a2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:23.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc31260c640 10 monclient(hunting): picked mon.noname-a con 0x7fc30c137e50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:40:23.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc31260c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc30c138230 0x7fc30c0a25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:23.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc31260c640 10 monclient(hunting): picked mon.noname-c con 0x7fc30c138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:40:23.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc31260c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc30c13d380 0x7fc30c1cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:23.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc31260c640 10 monclient(hunting): picked mon.noname-b con 0x7fc30c13d380 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:40:23.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc31260c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:23.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc31260c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c06f890 con 0x7fc30c137e50 2024-09-11T01:40:23.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc31260c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:23.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc31260c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c06f710 con 0x7fc30c13d380 2024-09-11T01:40:23.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc31260c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:23.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc31260c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc30c13fcb0 con 0x7fc30c138230 2024-09-11T01:40:23.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc31260c640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:23.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc31260c640 10 monclient(hunting): authenticate will time out at 1429.513113s 2024-09-11T01:40:23.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc30c137e50 0x7fc30c0a2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:23.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc30c137e50 0x7fc30c0a2070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60944/0 (socket says 172.21.15.12:60944) 2024-09-11T01:40:23.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc30c138230 0x7fc30c0a25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:23.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30bfff640 1 -- 172.21.15.12:0/29026320 learned_addr learned my addr 172.21.15.12:0/29026320 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:40:23.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30bfff640 10 monclient(hunting): get_auth_request con 0x7fc30c137e50 auth_method 0 2024-09-11T01:40:23.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:23.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:23.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:23.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30b7fe640 10 monclient(hunting): get_auth_request con 0x7fc30c138230 auth_method 0 2024-09-11T01:40:23.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:23.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:23.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:23.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:23.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:23.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:23.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:23.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:23.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:23.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30bfff640 10 monclient(hunting): handle_auth_done global_id 94395 payload 824 2024-09-11T01:40:23.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc310ba9640 1 --2- 172.21.15.12:0/29026320 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc30c13d380 0x7fc30c1cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:23.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30bfff640 1 -- 172.21.15.12:0/29026320 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc30c138230 msgr2=0x7fc30c0a25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:23.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30bfff640 1 --2- 172.21.15.12:0/29026320 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc30c138230 0x7fc30c0a25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:23.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30bfff640 1 -- 172.21.15.12:0/29026320 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc30c13d380 msgr2=0x7fc30c1cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:23.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30bfff640 1 --2- 172.21.15.12:0/29026320 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc30c13d380 0x7fc30c1cd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:23.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:40:23.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30bfff640 1 monclient: found mon.noname-a 2024-09-11T01:40:23.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:23.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:40:23.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30bfff640 1 -- 172.21.15.12:0/29026320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30c0a32e0 con 0x7fc30c137e50 2024-09-11T01:40:23.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc310ba9640 10 monclient: get_auth_request con 0x7fc30c13d380 auth_method 0 2024-09-11T01:40:23.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc30b7fe640 1 --2- 172.21.15.12:0/29026320 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc30c138230 0x7fc30c0a25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:40:23.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.071+0000 7fc310ba9640 1 --2- 172.21.15.12:0/29026320 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc30c13d380 0x7fc30c1cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:40:23.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.073+0000 7fc30bfff640 1 -- 172.21.15.12:0/29026320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc30c137e50 msgr2=0x7fc30c0a2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:23.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.073+0000 7fc30bfff640 1 --2- 172.21.15.12:0/29026320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc30c137e50 0x7fc30c0a2070 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc000c00 tx=0x7fc2fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:23.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.073+0000 7fc308ff9640 1 -- 172.21.15.12:0/29026320 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc041070 con 0x7fc30c137e50 2024-09-11T01:40:23.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.073+0000 7fc308ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:23.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.073+0000 7fc308ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:40:23.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.073+0000 7fc308ff9640 10 monclient: dump: 2024-09-11T01:40:23.086 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:23.086 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:23.086 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:23.086 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:23.086 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:23.086 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:23.086 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:23.086 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:23.086 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:23.086 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:23.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.074+0000 7fc308ff9640 10 monclient: _finish_auth 0 2024-09-11T01:40:23.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.074+0000 7fc308ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:40:23.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.074+0000 7fc308ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:23.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.074+0000 7fc308ff9640 1 -- 172.21.15.12:0/29026320 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2fc03c880 con 0x7fc30c137e50 2024-09-11T01:40:23.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.074+0000 7fc31260c640 5 monclient: authenticate success, global_id 94395 2024-09-11T01:40:23.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.074+0000 7fc31260c640 10 monclient: _renew_subs 2024-09-11T01:40:23.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.074+0000 7fc31260c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:40:23.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.074+0000 7fc31260c640 1 -- 172.21.15.12:0/29026320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc30c0a96c0 con 0x7fc30c137e50 2024-09-11T01:40:23.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.074+0000 7fc308ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:23.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.074+0000 7fc308ff9640 1 -- 172.21.15.12:0/29026320 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc037040 con 0x7fc30c137e50 2024-09-11T01:40:23.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.074+0000 7fc308ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:23.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.074+0000 7fc308ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:40:23.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.074+0000 7fc308ff9640 10 monclient: dump: 2024-09-11T01:40:23.087 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:23.087 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:23.087 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:23.087 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:23.087 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:23.088 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:23.088 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:23.088 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:23.088 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:23.088 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:23.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.074+0000 7fc31260c640 10 monclient: _renew_subs 2024-09-11T01:40:23.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.074+0000 7fc31260c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:40:23.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.074+0000 7fc31260c640 1 -- 172.21.15.12:0/29026320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc30c0a4520 con 0x7fc30c137e50 2024-09-11T01:40:23.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.074+0000 7fc309ffb640 4 set_mon_vals no callback set 2024-09-11T01:40:23.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.074+0000 7fc31260c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:23.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.074+0000 7fc31260c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:23.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.075+0000 7fc31260c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:40:23.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.075+0000 7fc31260c640 1 -- 172.21.15.12:0/29026320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc30c09d160 con 0x7fc30c137e50 2024-09-11T01:40:23.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.075+0000 7fc308ff9640 1 -- 172.21.15.12:0/29026320 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fc2fc03cc50 con 0x7fc30c137e50 2024-09-11T01:40:23.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.075+0000 7fc308ff9640 1 --2- 172.21.15.12:0/29026320 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc2f803e260 0x7fc2f8040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:23.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.075+0000 7fc308ff9640 1 -- 172.21.15.12:0/29026320 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(378..378 src has 285..378) ==== 4799+0+0 (secure 0 0 0) 0x7fc2fc073370 con 0x7fc30c137e50 2024-09-11T01:40:23.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.075+0000 7fc30b7fe640 1 --2- 172.21.15.12:0/29026320 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc2f803e260 0x7fc2f8040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:23.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.075+0000 7fc30b7fe640 10 monclient: get_auth_request con 0x7fc2f803e260 auth_method 0 2024-09-11T01:40:23.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.078+0000 7fc30b7fe640 1 --2- 172.21.15.12:0/29026320 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc2f803e260 0x7fc2f8040720 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc30c133ce0 tx=0x7fc300009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:23.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.079+0000 7fc308ff9640 1 -- 172.21.15.12:0/29026320 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc2fc03dba0 con 0x7fc30c137e50 2024-09-11T01:40:23.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.079+0000 7fc308ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:23.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.079+0000 7fc308ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:40:23.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.336+0000 7fc31260c640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:40:23.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.336+0000 7fc31260c640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:40:23.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.336+0000 7fc31260c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:40:23.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.336+0000 7fc31260c640 1 -- 172.21.15.12:0/29026320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fc30c0a1bd0 con 0x7fc30c137e50 2024-09-11T01:40:23.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.337+0000 7fc308ff9640 1 -- 172.21.15.12:0/29026320 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fc2fc0730d0 con 0x7fc30c137e50 2024-09-11T01:40:23.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.337+0000 7fc308ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:40:23.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.337+0000 7fc308ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:40:23.339 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.337+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.339 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.337+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=12,kv=12,logm=56} crc {auth=3691721166,config=2586702516,health=2872352161,kv=3263697420,logm=3583451019}) 2024-09-11T01:40:23.339 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.337+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=12,kv=11,logm=56} crc {auth=1251944625,config=2586702516,health=2872352161,kv=2198455534,logm=3728136010}) 2024-09-11T01:40:23.339 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.337+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.339 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.337+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=12,kv=12,logm=56} crc {auth=3691721166,config=2586702516,health=2872352161,kv=3263697420,logm=3583451019}) 2024-09-11T01:40:23.339 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.337+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=12,kv=9,logm=57} crc {auth=597684464,config=2586702516,health=2872352161,kv=395677583,logm=2560580797}) 2024-09-11T01:40:23.340 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.338+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.340 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.339+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3291283218}) 2024-09-11T01:40:23.340 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.339+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=625094074}) 2024-09-11T01:40:23.340 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.339+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.340 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.339+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3291283218}) 2024-09-11T01:40:23.340 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.339+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1649105540}) 2024-09-11T01:40:23.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.339+0000 7fc31260c640 1 -- 172.21.15.12:0/29026320 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc2f803e260 msgr2=0x7fc2f8040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:23.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.339+0000 7fc31260c640 1 --2- 172.21.15.12:0/29026320 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc2f803e260 0x7fc2f8040720 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc30c133ce0 tx=0x7fc300009040 comp rx=0 tx=0).stop 2024-09-11T01:40:23.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.339+0000 7fc31260c640 10 monclient: shutdown 2024-09-11T01:40:23.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.339+0000 7fc31260c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:23.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.339+0000 7fc31260c640 1 -- 172.21.15.12:0/29026320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc30c137e50 msgr2=0x7fc30c0a2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:23.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.339+0000 7fc31260c640 1 --2- 172.21.15.12:0/29026320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc30c137e50 0x7fc30c0a2070 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc000c00 tx=0x7fc2fc0046c0 comp rx=0 tx=0).stop 2024-09-11T01:40:23.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.339+0000 7fc31260c640 1 -- 172.21.15.12:0/29026320 shutdown_connections 2024-09-11T01:40:23.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.339+0000 7fc31260c640 1 --2- 172.21.15.12:0/29026320 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc30c13d380 0x7fc30c1cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:23.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.339+0000 7fc31260c640 1 --2- 172.21.15.12:0/29026320 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc30c138230 0x7fc30c0a25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:23.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.339+0000 7fc31260c640 1 --2- 172.21.15.12:0/29026320 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc2f803e260 0x7fc2f8040720 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:23.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.339+0000 7fc31260c640 1 --2- 172.21.15.12:0/29026320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc30c137e50 0x7fc30c0a2070 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:23.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.339+0000 7fc31260c640 1 -- 172.21.15.12:0/29026320 >> 172.21.15.12:0/29026320 conn(0x7fc30c0a7020 msgr2=0x7fc30c137080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:23.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.340+0000 7fc31260c640 1 -- 172.21.15.12:0/29026320 shutdown_connections 2024-09-11T01:40:23.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:23.340+0000 7fc31260c640 1 -- 172.21.15.12:0/29026320 wait complete. 2024-09-11T01:40:23.343 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.340+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.343 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.340+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3223326516}) 2024-09-11T01:40:23.344 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.340+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3409296717}) 2024-09-11T01:40:23.344 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.340+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.344 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.340+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3223326516}) 2024-09-11T01:40:23.344 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.340+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3610380261}) 2024-09-11T01:40:23.344 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.341+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.344 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.341+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4081394179}) 2024-09-11T01:40:23.344 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.341+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3538992550}) 2024-09-11T01:40:23.344 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.341+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.344 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.341+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4081394179}) 2024-09-11T01:40:23.344 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.341+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=304504263}) 2024-09-11T01:40:23.344 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.342+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.344 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.342+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1727656665}) 2024-09-11T01:40:23.344 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.342+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=580707980}) 2024-09-11T01:40:23.344 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.342+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.345 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.342+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1727656665}) 2024-09-11T01:40:23.345 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.342+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1780749422}) 2024-09-11T01:40:23.346 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.342+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.346 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.342+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3001468695}) 2024-09-11T01:40:23.346 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.342+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3579219563}) 2024-09-11T01:40:23.346 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.342+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.346 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.342+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3001468695}) 2024-09-11T01:40:23.346 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.342+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4002839486}) 2024-09-11T01:40:23.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.343+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.343+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3458840603}) 2024-09-11T01:40:23.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.343+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1969195750}) 2024-09-11T01:40:23.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.343+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.343+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3458840603}) 2024-09-11T01:40:23.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.343+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3155879749}) 2024-09-11T01:40:23.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.344+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.344+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1553126336}) 2024-09-11T01:40:23.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.344+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=730440520}) 2024-09-11T01:40:23.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.344+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.344+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1553126336}) 2024-09-11T01:40:23.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.344+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=175415267}) 2024-09-11T01:40:23.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.345+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.345+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2882351195}) 2024-09-11T01:40:23.348 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.345+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1536331893}) 2024-09-11T01:40:23.348 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.345+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.348 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.345+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2882351195}) 2024-09-11T01:40:23.348 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.345+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1907722221}) 2024-09-11T01:40:23.349 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.345+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.349 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.345+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3668902144}) 2024-09-11T01:40:23.349 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.345+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1206227438}) 2024-09-11T01:40:23.349 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.345+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.349 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.345+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3668902144}) 2024-09-11T01:40:23.349 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.345+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1705564327}) 2024-09-11T01:40:23.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.346+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.346+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3040652052}) 2024-09-11T01:40:23.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.346+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1608086439}) 2024-09-11T01:40:23.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.346+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.346+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3040652052}) 2024-09-11T01:40:23.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.346+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=571765038}) 2024-09-11T01:40:23.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.347+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.347+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4287748237}) 2024-09-11T01:40:23.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.347+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2490779467}) 2024-09-11T01:40:23.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.347+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.347+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4287748237}) 2024-09-11T01:40:23.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.347+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3460164956}) 2024-09-11T01:40:23.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.348+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.351 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.348+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=668878532}) 2024-09-11T01:40:23.351 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.348+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3567318541}) 2024-09-11T01:40:23.351 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.348+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.351 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.348+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=668878532}) 2024-09-11T01:40:23.351 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.348+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2887854884}) 2024-09-11T01:40:23.352 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.348+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.353 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.348+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=259652700}) 2024-09-11T01:40:23.353 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.348+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1462277661}) 2024-09-11T01:40:23.353 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.348+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.353 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.348+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=259652700}) 2024-09-11T01:40:23.353 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.348+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3801373628}) 2024-09-11T01:40:23.353 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.349+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.353 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.349+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2161243262}) 2024-09-11T01:40:23.353 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.349+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2447103755}) 2024-09-11T01:40:23.353 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.349+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.353 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.349+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2161243262}) 2024-09-11T01:40:23.353 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.349+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=162230097}) 2024-09-11T01:40:23.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.350+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.350+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3404033929}) 2024-09-11T01:40:23.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.350+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=528735229}) 2024-09-11T01:40:23.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.350+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.350+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3404033929}) 2024-09-11T01:40:23.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.350+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2363888640}) 2024-09-11T01:40:23.355 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:40:23.358 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.351+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.358 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.351+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2202870}) 2024-09-11T01:40:23.358 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.351+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2603034756}) 2024-09-11T01:40:23.358 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.351+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.358 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.351+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2202870}) 2024-09-11T01:40:23.358 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.351+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=897462083}) 2024-09-11T01:40:23.358 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.352+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.359 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.352+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=207164486}) 2024-09-11T01:40:23.359 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.352+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4126476722}) 2024-09-11T01:40:23.359 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.352+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.359 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.352+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=207164486}) 2024-09-11T01:40:23.359 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.352+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3086010556}) 2024-09-11T01:40:23.359 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.353+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.359 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.353+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3093299918}) 2024-09-11T01:40:23.359 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.353+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1123038494}) 2024-09-11T01:40:23.359 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.353+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.359 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.353+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3093299918}) 2024-09-11T01:40:23.359 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.353+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3045059244}) 2024-09-11T01:40:23.359 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.354+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.359 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.354+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1477134970}) 2024-09-11T01:40:23.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.354+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3811774006}) 2024-09-11T01:40:23.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.354+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.354+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1477134970}) 2024-09-11T01:40:23.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.354+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=606212902}) 2024-09-11T01:40:23.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.355+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.355+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3131576931}) 2024-09-11T01:40:23.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.355+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2801054359}) 2024-09-11T01:40:23.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.355+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.355+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3131576931}) 2024-09-11T01:40:23.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.355+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4122891752}) 2024-09-11T01:40:23.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.356+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.356+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1275929276}) 2024-09-11T01:40:23.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.356+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3485845502}) 2024-09-11T01:40:23.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.356+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.361 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.356+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1275929276}) 2024-09-11T01:40:23.361 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.356+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1868501549}) 2024-09-11T01:40:23.363 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.357+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.363 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.357+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4163656364}) 2024-09-11T01:40:23.363 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.357+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4186262765}) 2024-09-11T01:40:23.363 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.357+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.364 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.357+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4163656364}) 2024-09-11T01:40:23.364 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.357+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2061707461}) 2024-09-11T01:40:23.364 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.358+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.364 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.358+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1955942701}) 2024-09-11T01:40:23.364 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.358+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3768394095}) 2024-09-11T01:40:23.364 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.358+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.364 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.358+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1955942701}) 2024-09-11T01:40:23.364 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.358+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3824319134}) 2024-09-11T01:40:23.364 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.359+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.364 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.359+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1059218889}) 2024-09-11T01:40:23.364 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.359+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1720350597}) 2024-09-11T01:40:23.364 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.359+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.364 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.359+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1059218889}) 2024-09-11T01:40:23.364 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.359+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1047138168}) 2024-09-11T01:40:23.364 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.360+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.365 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.360+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=200103352}) 2024-09-11T01:40:23.365 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.360+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=877695613}) 2024-09-11T01:40:23.365 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.360+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.365 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.360+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=200103352}) 2024-09-11T01:40:23.365 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.360+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2434861841}) 2024-09-11T01:40:23.365 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.361+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.365 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.361+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3439167538}) 2024-09-11T01:40:23.365 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.361+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1931475339}) 2024-09-11T01:40:23.365 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.361+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.365 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.361+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3439167538}) 2024-09-11T01:40:23.365 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.361+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4025311156}) 2024-09-11T01:40:23.367 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.362+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.367 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.362+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2824306313}) 2024-09-11T01:40:23.367 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.362+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2120075638}) 2024-09-11T01:40:23.367 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.362+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.362+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2824306313}) 2024-09-11T01:40:23.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.362+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3918621130}) 2024-09-11T01:40:23.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.363+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.363+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3778882669}) 2024-09-11T01:40:23.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.363+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2227446569}) 2024-09-11T01:40:23.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.363+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.363+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3778882669}) 2024-09-11T01:40:23.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.363+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4243200152}) 2024-09-11T01:40:23.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.364+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.364+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=564466391}) 2024-09-11T01:40:23.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.364+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3704542803}) 2024-09-11T01:40:23.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.364+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.364+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=564466391}) 2024-09-11T01:40:23.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.364+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2944832604}) 2024-09-11T01:40:23.369 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.365+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.369 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.365+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2087784373}) 2024-09-11T01:40:23.369 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.365+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=97624321}) 2024-09-11T01:40:23.369 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.365+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.369 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.365+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2087784373}) 2024-09-11T01:40:23.369 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.365+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=201630218}) 2024-09-11T01:40:23.370 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.367+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.370 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.367+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=8,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=39} crc {logm=1995804332,mdsmap=3140514009,mgr=2672916156,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2503311804,mon_config_key=804823608,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=3929219181,osd_snap=3010994219}) 2024-09-11T01:40:23.370 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.367+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=8,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=37} crc {logm=1995804332,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2503311804,mon_config_key=3330371543,monmap=2933275274,nvmeofgw=3517310312,osd_metadata=642414809,osd_pg_creating=3929219181,osd_snap=116012273}) 2024-09-11T01:40:23.370 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.367+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.370 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.367+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=8,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=39} crc {logm=1995804332,mdsmap=3140514009,mgr=2672916156,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2503311804,mon_config_key=804823608,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=3929219181,osd_snap=3010994219}) 2024-09-11T01:40:23.370 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.367+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=8,mdsmap=1,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=5,osd_snap=45} crc {logm=1995804332,mdsmap=1228700967,mgr=2161719361,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=170241621,mon_config_key=3584078491,monmap=2933275274,nvmeofgw=598337686,osd_metadata=118787097,osd_snap=1736638981}) 2024-09-11T01:40:23.371 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.367+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.371 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.367+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=33,osdmap=67} crc {osd_snap=2248234252,osdmap=1899963331}) 2024-09-11T01:40:23.371 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.367+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=33,osdmap=67} crc {osd_snap=2524265008,osdmap=3705120162}) 2024-09-11T01:40:23.371 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.367+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.371 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.367+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=33,osdmap=67} crc {osd_snap=2248234252,osdmap=1899963331}) 2024-09-11T01:40:23.371 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.367+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=32,osdmap=68} crc {osd_snap=582923675,osdmap=3568773663}) 2024-09-11T01:40:23.374 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.372+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.374 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.372+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=88} crc {osdmap=3315004710}) 2024-09-11T01:40:23.374 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.372+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=87} crc {osdmap=727188661}) 2024-09-11T01:40:23.374 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.372+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:40:23.374 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.372+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=88} crc {osdmap=3315004710}) 2024-09-11T01:40:23.374 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:40:23.372+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=86} crc {osdmap=487696950}) 2024-09-11T01:40:23.945 INFO:tasks.rados.rados.0.smithi071.stdout:1653: done (1 left) 2024-09-11T01:40:23.945 INFO:tasks.rados.rados.0.smithi071.stdout:1654: done (0 left) 2024-09-11T01:40:23.946 INFO:tasks.rados.rados.0.smithi071.stdout:1655: read oid 45 snap -1 2024-09-11T01:40:23.946 INFO:tasks.rados.rados.0.smithi071.stdout:1655: expect (ObjNum 437 snap 119 seq_num 437) 2024-09-11T01:40:23.946 INFO:tasks.rados.rados.0.smithi071.stdout:1656: snap_remove snap 177 2024-09-11T01:40:24.355 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:40:24.356 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:40:24.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.467+0000 7fb6e20be640 10 monclient: get_monmap_and_config 2024-09-11T01:40:24.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.467+0000 7fb6e20be640 10 monclient: build_initial_monmap 2024-09-11T01:40:24.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.467+0000 7fb6e20be640 10 monclient: monmap: 2024-09-11T01:40:24.469 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:24.469 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:24.469 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:40:24.468055+0000 2024-09-11T01:40:24.469 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:40:24.468055+0000 2024-09-11T01:40:24.469 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:24.469 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:24.469 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:40:24.469 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:40:24.469 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:40:24.469 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:40:24.469 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:40:24.469 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:24.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6e20be640 1 Processor -- start 2024-09-11T01:40:24.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6e20be640 1 -- start start 2024-09-11T01:40:24.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6e20be640 10 monclient: init 2024-09-11T01:40:24.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6e20be640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:24.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6e20be640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-11T01:40:24.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6e20be640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb6dc09d790 0x7fb6dc09db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:24.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6e20be640 10 monclient(hunting): picked mon.noname-a con 0x7fb6dc09d790 addr v2:172.21.15.12:3300/0 2024-09-11T01:40:24.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6e20be640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb6dc09e0b0 addr v1:172.21.15.12:6789/0 2024-09-11T01:40:24.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6e20be640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb6dc09e500 0x7fb6dc135170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:24.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6e20be640 10 monclient(hunting): picked mon.noname-b con 0x7fb6dc09e500 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:40:24.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6e20be640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6dc0a1da0 con 0x7fb6dc09e0b0 2024-09-11T01:40:24.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6e20be640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:24.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6e20be640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fb6dc09e960 con 0x7fb6dc09d790 2024-09-11T01:40:24.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6e20be640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:24.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6e20be640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb6dc0a0760 con 0x7fb6dc09e500 2024-09-11T01:40:24.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6e20be640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:24.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6e20be640 10 monclient(hunting): authenticate will time out at 1430.910767s 2024-09-11T01:40:24.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6dbfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb6dc09e500 0x7fb6dc135170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:24.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6db7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb6dc09d790 0x7fb6dc09db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:24.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6daffd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb6dc09e0b0 0x7fb6dc134a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43298/0 (socket says 172.21.15.12:43298) 2024-09-11T01:40:24.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6dbfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb6dc09e500 0x7fb6dc135170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:49312/0 (socket says 172.21.15.12:49312) 2024-09-11T01:40:24.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6daffd640 1 -- 172.21.15.12:0/956357836 learned_addr learned my addr 172.21.15.12:0/956357836 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:40:24.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6dbfff640 10 monclient(hunting): get_auth_request con 0x7fb6dc09e500 auth_method 0 2024-09-11T01:40:24.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:24.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:24.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:24.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6db7fe640 10 monclient(hunting): get_auth_request con 0x7fb6dc09d790 auth_method 0 2024-09-11T01:40:24.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:24.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:24.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.468+0000 7fb6db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:24.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.469+0000 7fb6dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:24.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.469+0000 7fb6dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:24.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.469+0000 7fb6dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:24.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.469+0000 7fb6db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:24.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.469+0000 7fb6db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:24.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.469+0000 7fb6db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:24.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.469+0000 7fb6dbfff640 10 monclient(hunting): handle_auth_done global_id 114230 payload 274 2024-09-11T01:40:24.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.469+0000 7fb6dbfff640 1 -- 172.21.15.12:0/956357836 >> v2:172.21.15.12:3300/0 conn(0x7fb6dc09d790 msgr2=0x7fb6dc09db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:40:24.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.469+0000 7fb6dbfff640 1 --2- 172.21.15.12:0/956357836 >> v2:172.21.15.12:3300/0 conn(0x7fb6dc09d790 0x7fb6dc09db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:24.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.469+0000 7fb6dbfff640 1 -- 172.21.15.12:0/956357836 >> v1:172.21.15.12:6789/0 conn(0x7fb6dc09e0b0 legacy=0x7fb6dc134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:40:24.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.469+0000 7fb6dbfff640 10 monclient: _finish_hunting 0 2024-09-11T01:40:24.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.469+0000 7fb6dbfff640 1 monclient: found mon.noname-b 2024-09-11T01:40:24.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.469+0000 7fb6dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:24.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.469+0000 7fb6dbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:40:24.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.469+0000 7fb6dbfff640 1 -- 172.21.15.12:0/956357836 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6dc0a1fa0 con 0x7fb6dc09e500 2024-09-11T01:40:24.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.469+0000 7fb6db7fe640 1 --2- 172.21.15.12:0/956357836 >> v2:172.21.15.12:3300/0 conn(0x7fb6dc09d790 0x7fb6dc09db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:40:24.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6dbfff640 1 -- 172.21.15.12:0/956357836 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb6dc09e500 msgr2=0x7fb6dc135170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:24.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6dbfff640 1 --2- 172.21.15.12:0/956357836 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb6dc09e500 0x7fb6dc135170 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4009d90 tx=0x7fb6d402a490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47fc0a07c7fe0d1a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:24.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6da7fc640 1 -- 172.21.15.12:0/956357836 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d4007950 con 0x7fb6dc09e500 2024-09-11T01:40:24.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:24.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6da7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:40:24.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6da7fc640 10 monclient: dump: 2024-09-11T01:40:24.481 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:24.481 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:24.481 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:24.481 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:24.481 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:24.481 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:24.481 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:24.481 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:24.481 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:24.481 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:24.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6da7fc640 10 monclient: _finish_auth 0 2024-09-11T01:40:24.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6da7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:40:24.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:24.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6da7fc640 1 -- 172.21.15.12:0/956357836 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6d4007af0 con 0x7fb6dc09e500 2024-09-11T01:40:24.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6da7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:24.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6da7fc640 1 -- 172.21.15.12:0/956357836 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d4031400 con 0x7fb6dc09e500 2024-09-11T01:40:24.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:24.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6da7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:40:24.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6da7fc640 10 monclient: dump: 2024-09-11T01:40:24.482 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:24.482 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:24.482 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:24.482 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:24.482 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:24.482 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:24.482 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:24.483 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6e20be640 5 monclient: authenticate success, global_id 114230 2024-09-11T01:40:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6e20be640 10 monclient: get_monmap_and_config success 2024-09-11T01:40:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6e20be640 4 set_mon_vals no callback set 2024-09-11T01:40:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6e20be640 10 monclient: shutdown 2024-09-11T01:40:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6e20be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6e20be640 1 -- 172.21.15.12:0/956357836 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb6dc09e500 msgr2=0x7fb6dc135170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6e20be640 1 --2- 172.21.15.12:0/956357836 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb6dc09e500 0x7fb6dc135170 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4009d90 tx=0x7fb6d402a490 comp rx=0 tx=0).stop 2024-09-11T01:40:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6e20be640 1 -- 172.21.15.12:0/956357836 shutdown_connections 2024-09-11T01:40:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6e20be640 1 --2- 172.21.15.12:0/956357836 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb6dc09e500 0x7fb6dc135170 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6e20be640 1 --2- 172.21.15.12:0/956357836 >> v2:172.21.15.12:3300/0 conn(0x7fb6dc09d790 0x7fb6dc09db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6e20be640 1 -- 172.21.15.12:0/956357836 >> 172.21.15.12:0/956357836 conn(0x7fb6dc0a2d90 msgr2=0x7fb6dc0a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:24.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.472+0000 7fb6e20be640 1 -- 172.21.15.12:0/956357836 shutdown_connections 2024-09-11T01:40:24.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.473+0000 7fb6e20be640 1 -- 172.21.15.12:0/956357836 wait complete. 2024-09-11T01:40:24.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.473+0000 7fb6e20be640 10 monclient: build_initial_monmap 2024-09-11T01:40:24.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.473+0000 7fb6e20be640 10 monclient: monmap: 2024-09-11T01:40:24.484 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:24.484 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:40:24.484 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:40:24.484 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:40:24.484 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:24.484 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:24.484 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:40:24.484 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:40:24.484 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:40:24.484 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:24.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.473+0000 7fb6e20be640 1 Processor -- start 2024-09-11T01:40:24.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6e20be640 1 -- start start 2024-09-11T01:40:24.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6e20be640 10 monclient: init 2024-09-11T01:40:24.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6e20be640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:24.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6e20be640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:40:24.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6e20be640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb6dc09d790 0x7fb6dc12cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:24.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6e20be640 10 monclient(hunting): picked mon.noname-c con 0x7fb6dc09d790 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:40:24.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6e20be640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6dc09e0b0 0x7fb6dc12d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:24.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6e20be640 10 monclient(hunting): picked mon.noname-a con 0x7fb6dc09e0b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:40:24.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6e20be640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6dc09e500 0x7fb6dc13cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:24.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6e20be640 10 monclient(hunting): picked mon.noname-b con 0x7fb6dc09e500 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:40:24.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6e20be640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:24.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6e20be640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb6dc0a23a0 con 0x7fb6dc09e0b0 2024-09-11T01:40:24.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6e20be640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:24.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6e20be640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb6dc0a2220 con 0x7fb6dc09e500 2024-09-11T01:40:24.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6e20be640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:24.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6e20be640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb6dc09cca0 con 0x7fb6dc09d790 2024-09-11T01:40:24.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6e20be640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:24.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6e20be640 10 monclient(hunting): authenticate will time out at 1430.916704s 2024-09-11T01:40:24.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6db7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb6dc09d790 0x7fb6dc12cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:24.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6daffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6dc09e0b0 0x7fb6dc12d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:24.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6daffd640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6dc09e0b0 msgr2=0x7fb6dc12d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:40:24.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6db7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb6dc09d790 0x7fb6dc12cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:49318/0 (socket says 172.21.15.12:49318) 2024-09-11T01:40:24.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6daffd640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6dc09e0b0 msgr2=0x7fb6dc12d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:40:24.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6db7fe640 1 -- 172.21.15.12:0/581737891 learned_addr learned my addr 172.21.15.12:0/581737891 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:40:24.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6db7fe640 10 monclient(hunting): get_auth_request con 0x7fb6dc09d790 auth_method 0 2024-09-11T01:40:24.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6daffd640 1 --2- 172.21.15.12:0/581737891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6dc09e0b0 0x7fb6dc12d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:40:24.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:24.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:24.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:24.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6daffd640 1 --2- 172.21.15.12:0/581737891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6dc09e0b0 0x7fb6dc12d190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:40:24.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:24.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:24.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.474+0000 7fb6db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:24.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.475+0000 7fb6db7fe640 10 monclient(hunting): handle_auth_done global_id 114233 payload 824 2024-09-11T01:40:24.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.475+0000 7fb6db7fe640 1 -- 172.21.15.12:0/581737891 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6dc09e500 msgr2=0x7fb6dc13cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:24.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.475+0000 7fb6db7fe640 1 --2- 172.21.15.12:0/581737891 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6dc09e500 0x7fb6dc13cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:24.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.475+0000 7fb6db7fe640 1 -- 172.21.15.12:0/581737891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6dc09e0b0 msgr2=0x7fb6dc12d190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:40:24.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.475+0000 7fb6db7fe640 1 --2- 172.21.15.12:0/581737891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6dc09e0b0 0x7fb6dc12d190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:24.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.475+0000 7fb6db7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:40:24.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.475+0000 7fb6db7fe640 1 monclient: found mon.noname-c 2024-09-11T01:40:24.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.475+0000 7fb6db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:24.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.475+0000 7fb6db7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:40:24.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.475+0000 7fb6db7fe640 1 -- 172.21.15.12:0/581737891 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6dc137710 con 0x7fb6dc09d790 2024-09-11T01:40:24.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.477+0000 7fb6db7fe640 1 -- 172.21.15.12:0/581737891 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb6dc09d790 msgr2=0x7fb6dc12cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:24.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.477+0000 7fb6db7fe640 1 --2- 172.21.15.12:0/581737891 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb6dc09d790 0x7fb6dc12cc50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc00eb60 tx=0x7fb6cc00c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:24.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.477+0000 7fb6c3fff640 1 -- 172.21.15.12:0/581737891 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc017070 con 0x7fb6dc09d790 2024-09-11T01:40:24.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.477+0000 7fb6c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:24.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.477+0000 7fb6c3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:40:24.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.477+0000 7fb6c3fff640 10 monclient: dump: 2024-09-11T01:40:24.488 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:24.488 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:24.489 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:24.489 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:24.489 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:24.489 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:24.489 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:24.489 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:24.489 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:24.489 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:24.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.477+0000 7fb6c3fff640 10 monclient: _finish_auth 0 2024-09-11T01:40:24.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.477+0000 7fb6c3fff640 10 monclient: _check_auth_tickets 2024-09-11T01:40:24.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.477+0000 7fb6c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:24.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.477+0000 7fb6e20be640 5 monclient: authenticate success, global_id 114233 2024-09-11T01:40:24.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.477+0000 7fb6c3fff640 1 -- 172.21.15.12:0/581737891 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6cc00ee60 con 0x7fb6dc09d790 2024-09-11T01:40:24.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.477+0000 7fb6e20be640 10 monclient: _renew_subs 2024-09-11T01:40:24.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.477+0000 7fb6e20be640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:40:24.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.477+0000 7fb6e20be640 1 -- 172.21.15.12:0/581737891 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6dc1379f0 con 0x7fb6dc09d790 2024-09-11T01:40:24.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.477+0000 7fb6c3fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:24.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.477+0000 7fb6c3fff640 1 -- 172.21.15.12:0/581737891 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc013470 con 0x7fb6dc09d790 2024-09-11T01:40:24.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.477+0000 7fb6c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:24.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.477+0000 7fb6c3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:40:24.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.477+0000 7fb6c3fff640 10 monclient: dump: 2024-09-11T01:40:24.490 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:24.490 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:24.490 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:24.490 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:24.490 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:24.491 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:24.491 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:24.491 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:24.491 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:24.491 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:24.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.477+0000 7fb6e20be640 10 monclient: _renew_subs 2024-09-11T01:40:24.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.477+0000 7fb6d9ffb640 4 set_mon_vals no callback set 2024-09-11T01:40:24.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.477+0000 7fb6e20be640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:40:24.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.477+0000 7fb6e20be640 1 -- 172.21.15.12:0/581737891 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6dc137430 con 0x7fb6dc09d790 2024-09-11T01:40:24.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.478+0000 7fb6c3fff640 1 -- 172.21.15.12:0/581737891 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fb6cc024060 con 0x7fb6dc09d790 2024-09-11T01:40:24.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.478+0000 7fb6e20be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:24.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.478+0000 7fb6e20be640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:24.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.478+0000 7fb6e20be640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:40:24.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.478+0000 7fb6e20be640 1 -- 172.21.15.12:0/581737891 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6dc135730 con 0x7fb6dc09d790 2024-09-11T01:40:24.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.478+0000 7fb6c3fff640 1 --2- 172.21.15.12:0/581737891 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb6c803e210 0x7fb6c80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:24.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.478+0000 7fb6c3fff640 1 -- 172.21.15.12:0/581737891 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(379..379 src has 285..379) ==== 4799+0+0 (secure 0 0 0) 0x7fb6cc052d60 con 0x7fb6dc09d790 2024-09-11T01:40:24.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.479+0000 7fb6daffd640 1 --2- 172.21.15.12:0/581737891 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb6c803e210 0x7fb6c80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:24.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.479+0000 7fb6daffd640 10 monclient: get_auth_request con 0x7fb6c803e210 auth_method 0 2024-09-11T01:40:24.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.481+0000 7fb6daffd640 1 --2- 172.21.15.12:0/581737891 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb6c803e210 0x7fb6c80406d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc13eef0 tx=0x7fb6d0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:24.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.483+0000 7fb6c3fff640 1 -- 172.21.15.12:0/581737891 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb6cc051750 con 0x7fb6dc09d790 2024-09-11T01:40:24.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.483+0000 7fb6c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:24.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.483+0000 7fb6c3fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:40:24.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.741+0000 7fb6e20be640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:40:24.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.741+0000 7fb6e20be640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:40:24.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.741+0000 7fb6e20be640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:40:24.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.741+0000 7fb6e20be640 1 -- 172.21.15.12:0/581737891 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb6dc06d460 con 0x7fb6dc09d790 2024-09-11T01:40:24.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.741+0000 7fb6c3fff640 1 -- 172.21.15.12:0/581737891 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fb6cc023d00 con 0x7fb6dc09d790 2024-09-11T01:40:24.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.741+0000 7fb6c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:40:24.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.741+0000 7fb6c3fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:40:24.744 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":106,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:40:24.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.743+0000 7fb6e20be640 1 -- 172.21.15.12:0/581737891 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb6c803e210 msgr2=0x7fb6c80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:24.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.743+0000 7fb6e20be640 1 --2- 172.21.15.12:0/581737891 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb6c803e210 0x7fb6c80406d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc13eef0 tx=0x7fb6d0009040 comp rx=0 tx=0).stop 2024-09-11T01:40:24.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.743+0000 7fb6e20be640 10 monclient: shutdown 2024-09-11T01:40:24.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.743+0000 7fb6e20be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:24.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.743+0000 7fb6e20be640 1 -- 172.21.15.12:0/581737891 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb6dc09d790 msgr2=0x7fb6dc12cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:24.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.743+0000 7fb6e20be640 1 --2- 172.21.15.12:0/581737891 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb6dc09d790 0x7fb6dc12cc50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc00eb60 tx=0x7fb6cc00c520 comp rx=0 tx=0).stop 2024-09-11T01:40:24.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.743+0000 7fb6e20be640 1 -- 172.21.15.12:0/581737891 shutdown_connections 2024-09-11T01:40:24.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.743+0000 7fb6e20be640 1 --2- 172.21.15.12:0/581737891 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6dc09e500 0x7fb6dc13cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:24.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.743+0000 7fb6e20be640 1 --2- 172.21.15.12:0/581737891 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb6c803e210 0x7fb6c80406d0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:24.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.743+0000 7fb6e20be640 1 --2- 172.21.15.12:0/581737891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6dc09e0b0 0x7fb6dc12d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:24.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.743+0000 7fb6e20be640 1 --2- 172.21.15.12:0/581737891 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb6dc09d790 0x7fb6dc12cc50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:24.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.743+0000 7fb6e20be640 1 -- 172.21.15.12:0/581737891 >> 172.21.15.12:0/581737891 conn(0x7fb6dc0a2d90 msgr2=0x7fb6dc0a4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:24.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.744+0000 7fb6e20be640 1 -- 172.21.15.12:0/581737891 shutdown_connections 2024-09-11T01:40:24.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.744+0000 7fb6e20be640 1 -- 172.21.15.12:0/581737891 wait complete. 2024-09-11T01:40:24.755 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:40:24.755 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:40:24.755 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:40:24.756 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:40:24.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.856+0000 7febfecf7640 10 monclient: get_monmap_and_config 2024-09-11T01:40:24.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.856+0000 7febfecf7640 10 monclient: build_initial_monmap 2024-09-11T01:40:24.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.856+0000 7febfecf7640 10 monclient: monmap: 2024-09-11T01:40:24.857 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:24.858 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:24.858 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:40:24.856904+0000 2024-09-11T01:40:24.858 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:40:24.856904+0000 2024-09-11T01:40:24.858 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:24.858 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:24.858 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:40:24.858 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:40:24.858 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:40:24.858 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:40:24.858 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:40:24.858 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:24.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.856+0000 7febfecf7640 1 Processor -- start 2024-09-11T01:40:24.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.856+0000 7febfecf7640 1 -- start start 2024-09-11T01:40:24.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.856+0000 7febfecf7640 10 monclient: init 2024-09-11T01:40:24.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febfecf7640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:24.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febfecf7640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-11T01:40:24.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febfecf7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7febf8137e50 addr v1:172.21.15.71:6789/0 2024-09-11T01:40:24.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febfecf7640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7febf8138230 0x7febf8138640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:24.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febfecf7640 10 monclient(hunting): picked mon.noname-a con 0x7febf8138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:40:24.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febfecf7640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7febf813d380 0x7febf813f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:24.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febfecf7640 10 monclient(hunting): picked mon.noname-c con 0x7febf813d380 addr v2:172.21.15.71:3300/0 2024-09-11T01:40:24.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febfecf7640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febf80a62d0 con 0x7febf8137e50 2024-09-11T01:40:24.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febfecf7640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:24.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febfecf7640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7febf809bb40 con 0x7febf8138230 2024-09-11T01:40:24.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febfecf7640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:24.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febfecf7640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7febf809c4a0 con 0x7febf813d380 2024-09-11T01:40:24.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febfecf7640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:24.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febfecf7640 10 monclient(hunting): authenticate will time out at 1431.299466s 2024-09-11T01:40:24.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febf7fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7febf8138230 0x7febf8138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:24.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febf7fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7febf8138230 0x7febf8138640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60958/0 (socket says 172.21.15.12:60958) 2024-09-11T01:40:24.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febf7fff640 1 -- 172.21.15.12:0/872703601 learned_addr learned my addr 172.21.15.12:0/872703601 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:40:24.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febf7fff640 10 monclient(hunting): get_auth_request con 0x7febf8138230 auth_method 0 2024-09-11T01:40:24.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febf7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:24.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febf7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:24.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febf7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:24.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febf7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:24.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febf7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:24.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febf7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:24.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febfd294640 1 --2- 172.21.15.12:0/872703601 >> v2:172.21.15.71:3300/0 conn(0x7febf813d380 0x7febf813f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:24.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febfd294640 10 monclient(hunting): get_auth_request con 0x7febf813d380 auth_method 0 2024-09-11T01:40:24.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febfd294640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:24.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febfd294640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:24.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febfd294640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:24.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febf7fff640 10 monclient(hunting): handle_auth_done global_id 94401 payload 274 2024-09-11T01:40:24.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febf7fff640 1 -- 172.21.15.12:0/872703601 >> v2:172.21.15.71:3300/0 conn(0x7febf813d380 msgr2=0x7febf813f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:40:24.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febf7fff640 1 --2- 172.21.15.12:0/872703601 >> v2:172.21.15.71:3300/0 conn(0x7febf813d380 0x7febf813f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:24.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febf7fff640 1 -- 172.21.15.12:0/872703601 >> v1:172.21.15.71:6789/0 conn(0x7febf8137e50 legacy=0x7febf809d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:40:24.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febf7fff640 10 monclient: _finish_hunting 0 2024-09-11T01:40:24.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febf7fff640 1 monclient: found mon.noname-a 2024-09-11T01:40:24.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febf7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:24.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febf7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:40:24.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.857+0000 7febf7fff640 1 -- 172.21.15.12:0/872703601 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febf809c780 con 0x7febf8138230 2024-09-11T01:40:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.860+0000 7febf7fff640 1 -- 172.21.15.12:0/872703601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf8138230 msgr2=0x7febf8138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.860+0000 7febf7fff640 1 --2- 172.21.15.12:0/872703601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf8138230 0x7febf8138640 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7febf809cf30 tx=0x7febe402a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b99adce465cf9320 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.860+0000 7febf77fe640 1 -- 172.21.15.12:0/872703601 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe4037070 con 0x7febf8138230 2024-09-11T01:40:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.860+0000 7febf77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.860+0000 7febf77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:40:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.860+0000 7febf77fe640 10 monclient: dump: 2024-09-11T01:40:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:24.871 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.860+0000 7febf77fe640 10 monclient: _finish_auth 0 2024-09-11T01:40:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.860+0000 7febf77fe640 10 monclient: _check_auth_tickets 2024-09-11T01:40:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.860+0000 7febf77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.860+0000 7febf77fe640 1 -- 172.21.15.12:0/872703601 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febe4032440 con 0x7febf8138230 2024-09-11T01:40:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.860+0000 7febfecf7640 5 monclient: authenticate success, global_id 94401 2024-09-11T01:40:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.860+0000 7febfecf7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:40:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.860+0000 7febf77fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.860+0000 7febfecf7640 10 monclient: get_monmap_and_config success 2024-09-11T01:40:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.860+0000 7febf77fe640 1 -- 172.21.15.12:0/872703601 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe40327f0 con 0x7febf8138230 2024-09-11T01:40:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.860+0000 7febf77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.860+0000 7febfecf7640 4 set_mon_vals no callback set 2024-09-11T01:40:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.860+0000 7febf77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:40:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.860+0000 7febf77fe640 10 monclient: dump: 2024-09-11T01:40:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:24.872 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:24.873 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:24.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.861+0000 7febfecf7640 10 monclient: shutdown 2024-09-11T01:40:24.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.861+0000 7febfecf7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:24.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.861+0000 7febfecf7640 1 -- 172.21.15.12:0/872703601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf8138230 msgr2=0x7febf8138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:24.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.861+0000 7febfecf7640 1 --2- 172.21.15.12:0/872703601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf8138230 0x7febf8138640 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7febf809cf30 tx=0x7febe402a080 comp rx=0 tx=0).stop 2024-09-11T01:40:24.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.861+0000 7febfecf7640 1 -- 172.21.15.12:0/872703601 shutdown_connections 2024-09-11T01:40:24.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.861+0000 7febfecf7640 1 --2- 172.21.15.12:0/872703601 >> v2:172.21.15.71:3300/0 conn(0x7febf813d380 0x7febf813f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:24.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.861+0000 7febfecf7640 1 --2- 172.21.15.12:0/872703601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf8138230 0x7febf8138640 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:24.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.861+0000 7febfecf7640 1 -- 172.21.15.12:0/872703601 >> 172.21.15.12:0/872703601 conn(0x7febf80a7020 msgr2=0x7febf80a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:24.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.861+0000 7febfecf7640 1 -- 172.21.15.12:0/872703601 shutdown_connections 2024-09-11T01:40:24.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.861+0000 7febfecf7640 1 -- 172.21.15.12:0/872703601 wait complete. 2024-09-11T01:40:24.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.862+0000 7febfecf7640 10 monclient: build_initial_monmap 2024-09-11T01:40:24.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.862+0000 7febfecf7640 10 monclient: monmap: 2024-09-11T01:40:24.874 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:24.874 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:40:24.874 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:40:24.874 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:40:24.874 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:24.874 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:24.874 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:40:24.874 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:40:24.874 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:40:24.874 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:24.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.862+0000 7febfecf7640 1 Processor -- start 2024-09-11T01:40:24.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.862+0000 7febfecf7640 1 -- start start 2024-09-11T01:40:24.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.862+0000 7febfecf7640 10 monclient: init 2024-09-11T01:40:24.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.862+0000 7febfecf7640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:24.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.862+0000 7febfecf7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:40:24.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.862+0000 7febfecf7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf8137e50 0x7febf81ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:24.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.862+0000 7febfecf7640 10 monclient(hunting): picked mon.noname-a con 0x7febf8137e50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:40:24.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febfecf7640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7febf8138230 0x7febf8136f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:24.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febfecf7640 10 monclient(hunting): picked mon.noname-c con 0x7febf8138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:40:24.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febfecf7640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7febf813d380 0x7febf80a1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:24.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febfecf7640 10 monclient(hunting): picked mon.noname-b con 0x7febf813d380 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:40:24.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febfecf7640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:24.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febfecf7640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7febf806f890 con 0x7febf8137e50 2024-09-11T01:40:24.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febfecf7640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:24.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febfecf7640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7febf806f710 con 0x7febf813d380 2024-09-11T01:40:24.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febfecf7640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:24.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febfecf7640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7febf813fcb0 con 0x7febf8138230 2024-09-11T01:40:24.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febfecf7640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:24.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febfecf7640 10 monclient(hunting): authenticate will time out at 1431.305044s 2024-09-11T01:40:24.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febfca93640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf8137e50 0x7febf81ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:24.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febf7fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7febf8138230 0x7febf8136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:24.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febfca93640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf8137e50 0x7febf81ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60966/0 (socket says 172.21.15.12:60966) 2024-09-11T01:40:24.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febf7fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7febf8138230 0x7febf8136f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:49320/0 (socket says 172.21.15.12:49320) 2024-09-11T01:40:24.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febfca93640 1 -- 172.21.15.12:0/173385242 learned_addr learned my addr 172.21.15.12:0/173385242 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:40:24.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febfca93640 10 monclient(hunting): get_auth_request con 0x7febf8137e50 auth_method 0 2024-09-11T01:40:24.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febfca93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:24.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febfca93640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:24.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febfca93640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:24.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febf7fff640 10 monclient(hunting): get_auth_request con 0x7febf8138230 auth_method 0 2024-09-11T01:40:24.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febf7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:24.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febf7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:24.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febf7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:24.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febfca93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:24.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febfca93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:24.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febfca93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:24.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febf7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:24.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febf7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:24.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febf7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:24.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febf7fff640 10 monclient(hunting): handle_auth_done global_id 114236 payload 824 2024-09-11T01:40:24.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febfd294640 1 --2- 172.21.15.12:0/173385242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7febf813d380 0x7febf80a1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:24.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febf7fff640 1 -- 172.21.15.12:0/173385242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7febf813d380 msgr2=0x7febf80a1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:24.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febf7fff640 1 --2- 172.21.15.12:0/173385242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7febf813d380 0x7febf80a1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:24.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febf7fff640 1 -- 172.21.15.12:0/173385242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf8137e50 msgr2=0x7febf81ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:24.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febf7fff640 1 --2- 172.21.15.12:0/173385242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf8137e50 0x7febf81ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:24.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febf7fff640 10 monclient: _finish_hunting 0 2024-09-11T01:40:24.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febf7fff640 1 monclient: found mon.noname-c 2024-09-11T01:40:24.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febf7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:24.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febf7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:40:24.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febf7fff640 1 -- 172.21.15.12:0/173385242 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febf81cde80 con 0x7febf8138230 2024-09-11T01:40:24.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febfd294640 10 monclient: get_auth_request con 0x7febf813d380 auth_method 0 2024-09-11T01:40:24.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febfca93640 1 --2- 172.21.15.12:0/173385242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf8137e50 0x7febf81ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:40:24.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.863+0000 7febfd294640 1 --2- 172.21.15.12:0/173385242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7febf813d380 0x7febf80a1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:40:24.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.865+0000 7febf7fff640 1 -- 172.21.15.12:0/173385242 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7febf8138230 msgr2=0x7febf8136f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.865+0000 7febf7fff640 1 --2- 172.21.15.12:0/173385242 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7febf8138230 0x7febf8136f40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7febf806d570 tx=0x7febe40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.865+0000 7febf57fa640 1 -- 172.21.15.12:0/173385242 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe4041020 con 0x7febf8138230 2024-09-11T01:40:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.865+0000 7febf57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.865+0000 7febf57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:40:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.865+0000 7febf57fa640 10 monclient: dump: 2024-09-11T01:40:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:24.881 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:24.881 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:24.881 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:24.881 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:24.881 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:24.881 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:24.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.865+0000 7febf57fa640 10 monclient: _finish_auth 0 2024-09-11T01:40:24.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.865+0000 7febf57fa640 10 monclient: _check_auth_tickets 2024-09-11T01:40:24.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.865+0000 7febf57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:24.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.865+0000 7febf57fa640 1 -- 172.21.15.12:0/173385242 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febe403d850 con 0x7febf8138230 2024-09-11T01:40:24.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.865+0000 7febf57fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:24.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.865+0000 7febf57fa640 1 -- 172.21.15.12:0/173385242 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe4037040 con 0x7febf8138230 2024-09-11T01:40:24.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.865+0000 7febf57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:24.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.865+0000 7febf57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:40:24.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.865+0000 7febf57fa640 10 monclient: dump: 2024-09-11T01:40:24.882 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:24.882 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:24.882 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:24.882 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:24.882 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:24.882 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:24.882 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:24.882 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:24.882 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:24.883 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.865+0000 7febfecf7640 5 monclient: authenticate success, global_id 114236 2024-09-11T01:40:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.865+0000 7febfecf7640 10 monclient: _renew_subs 2024-09-11T01:40:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.865+0000 7febfecf7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:40:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.865+0000 7febfecf7640 1 -- 172.21.15.12:0/173385242 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febf80a35d0 con 0x7febf8138230 2024-09-11T01:40:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.865+0000 7febf6ffd640 4 set_mon_vals no callback set 2024-09-11T01:40:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.866+0000 7febfecf7640 10 monclient: _renew_subs 2024-09-11T01:40:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.866+0000 7febfecf7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:40:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.866+0000 7febfecf7640 1 -- 172.21.15.12:0/173385242 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7febf81218d0 con 0x7febf8138230 2024-09-11T01:40:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.866+0000 7febfecf7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.866+0000 7febfecf7640 1 --2- 172.21.15.12:0/173385242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf8137480 0x7febf80a2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.866+0000 7febfecf7640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.866+0000 7febfecf7640 1 -- 172.21.15.12:0/173385242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7febf806a060 con 0x7febf8137480 2024-09-11T01:40:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.866+0000 7febfca93640 1 --2- 172.21.15.12:0/173385242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf8137480 0x7febf80a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.866+0000 7febf57fa640 1 -- 172.21.15.12:0/173385242 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7febe403a030 con 0x7febf8138230 2024-09-11T01:40:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.866+0000 7febfca93640 10 monclient: get_auth_request con 0x7febf8137480 auth_method 0 2024-09-11T01:40:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.866+0000 7febfca93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.866+0000 7febfca93640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.866+0000 7febfca93640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.866+0000 7febfca93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.866+0000 7febfca93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.866+0000 7febfca93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.867+0000 7febfca93640 10 monclient(hunting): handle_auth_done global_id 94407 payload 824 2024-09-11T01:40:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.867+0000 7febfca93640 1 -- 172.21.15.12:0/173385242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7febf8138640 con 0x7febf8137480 2024-09-11T01:40:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.867+0000 7febf57fa640 1 --2- 172.21.15.12:0/173385242 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7febdc03e210 0x7febdc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.867+0000 7febf57fa640 1 -- 172.21.15.12:0/173385242 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(379..379 src has 285..379) ==== 4799+0+0 (secure 0 0 0) 0x7febe40732c0 con 0x7febf8138230 2024-09-11T01:40:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.867+0000 7febfd294640 1 --2- 172.21.15.12:0/173385242 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7febdc03e210 0x7febdc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.867+0000 7febfd294640 10 monclient: get_auth_request con 0x7febdc03e210 auth_method 0 2024-09-11T01:40:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.869+0000 7febfca93640 1 -- 172.21.15.12:0/173385242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf8137480 msgr2=0x7febf80a2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.869+0000 7febfca93640 1 --2- 172.21.15.12:0/173385242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf8137480 0x7febf80a2150 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7febec00fae0 tx=0x7febec0045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.869+0000 7febf57fa640 1 -- 172.21.15.12:0/173385242 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febec018020 con 0x7febf8137480 2024-09-11T01:40:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.869+0000 7febf57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.869+0000 7febf57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:40:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.869+0000 7febf57fa640 10 monclient: dump: 2024-09-11T01:40:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:24.888 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.869+0000 7febf57fa640 1 -- 172.21.15.12:0/173385242 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7febec00de20 con 0x7febf8137480 2024-09-11T01:40:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.869+0000 7febf57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.869+0000 7febf57fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:40:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.869+0000 7febf57fa640 1 -- 172.21.15.12:0/173385242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf8137480 msgr2=0x7febf80a2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.869+0000 7febf57fa640 1 --2- 172.21.15.12:0/173385242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf8137480 0x7febf80a2150 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7febec00fae0 tx=0x7febec0045a0 comp rx=0 tx=0).stop 2024-09-11T01:40:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.869+0000 7febf57fa640 1 -- 172.21.15.12:0/173385242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf8137480 msgr2=0x7febf80a2150 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:40:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.869+0000 7febf57fa640 1 --2- 172.21.15.12:0/173385242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf8137480 0x7febf80a2150 secure :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7febec00fae0 tx=0x7febec0045a0 comp rx=0 tx=0).stop 2024-09-11T01:40:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.870+0000 7febfd294640 1 --2- 172.21.15.12:0/173385242 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7febdc03e210 0x7febdc0406d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7febf80a3e10 tx=0x7febf000b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.875+0000 7febfecf7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:40:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.875+0000 7febfecf7640 1 --2- 172.21.15.12:0/173385242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf80a2690 0x7febf813dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.876+0000 7febfecf7640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.876+0000 7febfecf7640 1 -- 172.21.15.12:0/173385242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7febf80a2a70 con 0x7febf80a2690 2024-09-11T01:40:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.876+0000 7febfca93640 1 --2- 172.21.15.12:0/173385242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf80a2690 0x7febf813dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:24.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.876+0000 7febfca93640 10 monclient: get_auth_request con 0x7febf80a2690 auth_method 0 2024-09-11T01:40:24.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.876+0000 7febfca93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:24.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.876+0000 7febfca93640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:24.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.876+0000 7febfca93640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:24.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.876+0000 7febfca93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:24.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.876+0000 7febfca93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:24.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.876+0000 7febfca93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:24.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.876+0000 7febfca93640 10 monclient(hunting): handle_auth_done global_id 94410 payload 824 2024-09-11T01:40:24.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.876+0000 7febfca93640 1 -- 172.21.15.12:0/173385242 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7febf80a4130 con 0x7febf80a2690 2024-09-11T01:40:24.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.878+0000 7febfca93640 1 -- 172.21.15.12:0/173385242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf80a2690 msgr2=0x7febf813dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:24.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.878+0000 7febfca93640 1 --2- 172.21.15.12:0/173385242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf80a2690 0x7febf813dd00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7febec019000 tx=0x7febec0036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:24.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.879+0000 7febf57fa640 1 -- 172.21.15.12:0/173385242 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febec018030 con 0x7febf80a2690 2024-09-11T01:40:24.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.879+0000 7febf57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:24.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.879+0000 7febf57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:40:24.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.879+0000 7febf57fa640 10 monclient: dump: 2024-09-11T01:40:24.891 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:24.891 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:24.891 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:24.891 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:24.891 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:24.891 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:24.891 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:24.891 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:24.891 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:24.891 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:24.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.879+0000 7febf57fa640 1 -- 172.21.15.12:0/173385242 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7febec007460 con 0x7febf80a2690 2024-09-11T01:40:24.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.879+0000 7febf57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:40:24.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.879+0000 7febf57fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:40:24.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.879+0000 7febf57fa640 1 -- 172.21.15.12:0/173385242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf80a2690 msgr2=0x7febf813dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:24.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.879+0000 7febf57fa640 1 --2- 172.21.15.12:0/173385242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf80a2690 0x7febf813dd00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7febec019000 tx=0x7febec0036f0 comp rx=0 tx=0).stop 2024-09-11T01:40:24.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.879+0000 7febf57fa640 1 -- 172.21.15.12:0/173385242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf80a2690 msgr2=0x7febf813dd00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:40:24.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.879+0000 7febf57fa640 1 --2- 172.21.15.12:0/173385242 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7febf80a2690 0x7febf813dd00 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:24.892 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:40:24.892 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:40:24.893 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:40:24.893 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:40:24.893 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 106, 2024-09-11T01:40:24.893 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 170975, 2024-09-11T01:40:24.893 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:40:24.893 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:40:24.893 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:40:24.893 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:40:24.893 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:24.893 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 5, 2024-09-11T01:40:24.893 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:40:24.893 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:40:24.894 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:40:24.894 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:40:24.894 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:40:24.894 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:40:24.894 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:40:24.894 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:40:24.894 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:40:24.894 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:40:24.894 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:40:24.894 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:40:24.894 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:40:24.894 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:40:24.894 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:24.895 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:40:24.895 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:40:24.895 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:40:24.895 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:40:24.895 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:40:24.895 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:40:24.895 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:40:24.895 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:40:24.895 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:40:24.895 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:40:24.895 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:40:24.895 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:40:24.895 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:40:24.896 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:24.896 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:24.896 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:40:24.896 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:40:24.896 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:40:24.896 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:40:24.896 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:40:24.896 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:40:24.896 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:40:24.896 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:40:24.896 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:40:24.896 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:40:24.896 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:40:24.897 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:40:24.897 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:40:24.897 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:40:24.897 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:40:24.897 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:40:24.897 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:40:24.897 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:40:24.897 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:40:24.897 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:40:24.897 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:40:24.897 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:40:24.897 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:40:24.898 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:40:24.898 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:40:24.898 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:40:24.898 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:40:24.898 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:40:24.898 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:24.898 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:40:24.898 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:24.898 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:40:24.898 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:24.898 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:40:24.898 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:40:24.898 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:40:24.899 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:40:24.899 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:24.899 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:40:24.899 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:40:24.899 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:24.899 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:24.899 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:24.899 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:40:24.899 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:40:24.899 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:24.899 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:24.899 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:24.900 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:24.900 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:40:24.900 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:40:24.900 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:40:24.900 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:40:24.900 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:40:24.900 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:24.900 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:24.900 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:40:24.900 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:40:24.901 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:40:24.901 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:40:24.901 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:24.901 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:40:24.901 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:40:24.901 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:24.901 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:24.901 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:24.901 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:40:24.901 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:40:24.901 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:24.902 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:24.902 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:24.902 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:24.902 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:40:24.902 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:40:24.902 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:40:24.902 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:40:24.902 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:40:24.902 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:24.902 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:24.902 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:40:24.902 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:40:24.902 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:40:24.902 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:40:24.903 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:24.903 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:40:24.903 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:40:24.903 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:24.903 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:24.903 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:24.903 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:40:24.903 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:40:24.903 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:24.903 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:24.903 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:24.903 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:24.903 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:40:24.903 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:40:24.903 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:40:24.904 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:40:24.904 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:40:24.904 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:24.904 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:24.904 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:24.904 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:40:24.904 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:40:24.904 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:24.904 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:40:24.904 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:40:24.904 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:40:24.905 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:24.905 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:24.905 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:40:24.905 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:24.905 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:40:24.905 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:40:24.905 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:40:24.905 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:24.905 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:24.905 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:40:24.905 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:24.905 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:40:24.905 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:40:24.905 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:40:24.906 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:24.906 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:24.906 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:24.906 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:40:24.906 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:40:24.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.879+0000 7febfecf7640 1 -- 172.21.15.12:0/173385242 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7febdc03e210 msgr2=0x7febdc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:24.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.879+0000 7febfecf7640 1 --2- 172.21.15.12:0/173385242 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7febdc03e210 0x7febdc0406d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7febf80a3e10 tx=0x7febf000b040 comp rx=0 tx=0).stop 2024-09-11T01:40:24.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.879+0000 7febfecf7640 10 monclient: shutdown 2024-09-11T01:40:24.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.879+0000 7febfecf7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:24.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.879+0000 7febfecf7640 1 -- 172.21.15.12:0/173385242 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7febf8138230 msgr2=0x7febf8136f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:24.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.879+0000 7febfecf7640 1 --2- 172.21.15.12:0/173385242 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7febf8138230 0x7febf8136f40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7febf806d570 tx=0x7febe40046c0 comp rx=0 tx=0).stop 2024-09-11T01:40:24.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.879+0000 7febfd294640 1 -- 172.21.15.12:0/173385242 reap_dead start 2024-09-11T01:40:24.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.879+0000 7febfecf7640 1 -- 172.21.15.12:0/173385242 shutdown_connections 2024-09-11T01:40:24.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.879+0000 7febfecf7640 1 --2- 172.21.15.12:0/173385242 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7febf8138230 0x7febf8136f40 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:24.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.879+0000 7febfecf7640 1 -- 172.21.15.12:0/173385242 >> 172.21.15.12:0/173385242 conn(0x7febf80a7020 msgr2=0x7febf80a9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:24.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.879+0000 7febfecf7640 1 -- 172.21.15.12:0/173385242 shutdown_connections 2024-09-11T01:40:24.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:24.879+0000 7febfecf7640 1 -- 172.21.15.12:0/173385242 wait complete. 2024-09-11T01:40:24.908 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:40:24.909 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:40:24.909 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:40:24.948 INFO:tasks.rados.rados.0.smithi071.stdout:1656: done (1 left) 2024-09-11T01:40:24.948 INFO:tasks.rados.rados.0.smithi071.stdout:1657: write (excl) oid 2 current snap is 181 2024-09-11T01:40:24.948 INFO:tasks.rados.rados.0.smithi071.stdout:1657: seq_num 611 ranges {669077=485379,1587792=681285} 2024-09-11T01:40:24.954 INFO:tasks.rados.rados.0.smithi071.stdout:1657: writing smithi07134777-2 from 669077 to 1154456 tid 1 2024-09-11T01:40:24.961 INFO:tasks.rados.rados.0.smithi071.stdout:1657: writing smithi07134777-2 from 1587792 to 2269077 tid 2 2024-09-11T01:40:24.961 INFO:tasks.rados.rados.0.smithi071.stdout:1658: write oid 35 current snap is 181 2024-09-11T01:40:24.961 INFO:tasks.rados.rados.0.smithi071.stdout:1655: expect (ObjNum 437 snap 119 seq_num 437) 2024-09-11T01:40:25.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.015+0000 7fd6dfdb4640 10 monclient: get_monmap_and_config 2024-09-11T01:40:25.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.015+0000 7fd6dfdb4640 10 monclient: build_initial_monmap 2024-09-11T01:40:25.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.015+0000 7fd6dfdb4640 10 monclient: monmap: 2024-09-11T01:40:25.017 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:25.017 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:25.017 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:40:25.016267+0000 2024-09-11T01:40:25.017 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:40:25.016267+0000 2024-09-11T01:40:25.017 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:25.018 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:25.018 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:40:25.018 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:40:25.018 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:40:25.018 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:40:25.018 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:40:25.018 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:25.019 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:40:25.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.016+0000 7fd6dfdb4640 1 Processor -- start 2024-09-11T01:40:25.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.016+0000 7fd6dfdb4640 1 -- start start 2024-09-11T01:40:25.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.016+0000 7fd6dfdb4640 10 monclient: init 2024-09-11T01:40:25.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.016+0000 7fd6dfdb4640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:25.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.016+0000 7fd6dfdb4640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-11T01:40:25.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.016+0000 7fd6dfdb4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d80a1380 0x7fd6d80a1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:25.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.016+0000 7fd6dfdb4640 10 monclient(hunting): picked mon.noname-b con 0x7fd6d80a1380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:40:25.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.016+0000 7fd6dfdb4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6d80a0370 addr v1:172.21.15.12:6789/0 2024-09-11T01:40:25.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.016+0000 7fd6dfdb4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd6d80a0c90 0x7fd6d81361f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:25.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.016+0000 7fd6dfdb4640 10 monclient(hunting): picked mon.noname-a con 0x7fd6d80a0c90 addr v2:172.21.15.12:3300/0 2024-09-11T01:40:25.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.016+0000 7fd6dfdb4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6d80a1ca0 con 0x7fd6d80a0370 2024-09-11T01:40:25.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.016+0000 7fd6dfdb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:25.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.016+0000 7fd6dfdb4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fd6d80a21c0 con 0x7fd6d80a0c90 2024-09-11T01:40:25.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.016+0000 7fd6dfdb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:25.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.016+0000 7fd6dfdb4640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd6d80a24a0 con 0x7fd6d80a1380 2024-09-11T01:40:25.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.016+0000 7fd6dfdb4640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:25.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.016+0000 7fd6dfdb4640 10 monclient(hunting): authenticate will time out at 1431.458882s 2024-09-11T01:40:25.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.016+0000 7fd6ddb50640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d80a1380 0x7fd6d80a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:25.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6de351640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd6d80a0c90 0x7fd6d81361f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:25.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6ddb50640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d80a1380 0x7fd6d80a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:49346/0 (socket says 172.21.15.12:49346) 2024-09-11T01:40:25.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6ddb50640 1 -- 172.21.15.12:0/2852842248 learned_addr learned my addr 172.21.15.12:0/2852842248 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:40:25.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6ddb50640 10 monclient(hunting): get_auth_request con 0x7fd6d80a1380 auth_method 0 2024-09-11T01:40:25.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6ddb50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:25.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6ddb50640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:25.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6ddb50640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:25.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6de351640 10 monclient(hunting): get_auth_request con 0x7fd6d80a0c90 auth_method 0 2024-09-11T01:40:25.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6de351640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:25.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6de351640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:25.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6de351640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:25.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6ddb50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:25.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6ddb50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:25.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6ddb50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:25.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6de351640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:25.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6de351640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:25.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6de351640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:25.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6ddb50640 10 monclient(hunting): handle_auth_done global_id 114239 payload 274 2024-09-11T01:40:25.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6ddb50640 1 -- 172.21.15.12:0/2852842248 >> v2:172.21.15.12:3300/0 conn(0x7fd6d80a0c90 msgr2=0x7fd6d81361f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:40:25.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6ddb50640 1 --2- 172.21.15.12:0/2852842248 >> v2:172.21.15.12:3300/0 conn(0x7fd6d80a0c90 0x7fd6d81361f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:25.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6ddb50640 1 -- 172.21.15.12:0/2852842248 >> v1:172.21.15.12:6789/0 conn(0x7fd6d80a0370 legacy=0x7fd6d8135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:40:25.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6ddb50640 10 monclient: _finish_hunting 0 2024-09-11T01:40:25.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6ddb50640 1 monclient: found mon.noname-b 2024-09-11T01:40:25.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6ddb50640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:25.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6ddb50640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:40:25.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6ddb50640 1 -- 172.21.15.12:0/2852842248 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d80a2780 con 0x7fd6d80a1380 2024-09-11T01:40:25.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.017+0000 7fd6de351640 1 --2- 172.21.15.12:0/2852842248 >> v2:172.21.15.12:3300/0 conn(0x7fd6d80a0c90 0x7fd6d81361f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:40:25.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6ddb50640 1 -- 172.21.15.12:0/2852842248 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d80a1380 msgr2=0x7fd6d80a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:25.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6ddb50640 1 --2- 172.21.15.12:0/2852842248 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d80a1380 0x7fd6d80a1760 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4009870 tx=0x7fd6d402a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40072145a81776dd server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:25.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dcb4e640 1 -- 172.21.15.12:0/2852842248 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4037070 con 0x7fd6d80a1380 2024-09-11T01:40:25.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dcb4e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:25.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dcb4e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:40:25.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dcb4e640 10 monclient: dump: 2024-09-11T01:40:25.027 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:25.027 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:25.027 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:25.027 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:25.027 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:25.027 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:25.027 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:25.027 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:25.028 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:25.028 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:25.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dcb4e640 10 monclient: _finish_auth 0 2024-09-11T01:40:25.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dcb4e640 10 monclient: _check_auth_tickets 2024-09-11T01:40:25.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dcb4e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:25.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dfdb4640 5 monclient: authenticate success, global_id 114239 2024-09-11T01:40:25.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dfdb4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:40:25.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dcb4e640 1 -- 172.21.15.12:0/2852842248 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6d4032440 con 0x7fd6d80a1380 2024-09-11T01:40:25.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dcb4e640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:25.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dfdb4640 10 monclient: get_monmap_and_config success 2024-09-11T01:40:25.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dfdb4640 4 set_mon_vals no callback set 2024-09-11T01:40:25.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dcb4e640 1 -- 172.21.15.12:0/2852842248 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4032810 con 0x7fd6d80a1380 2024-09-11T01:40:25.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dcb4e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:25.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dcb4e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:40:25.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dcb4e640 10 monclient: dump: 2024-09-11T01:40:25.029 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:25.029 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:25.029 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:25.029 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:25.029 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:25.029 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:25.029 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:25.029 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:25.029 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:25.029 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:25.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dfdb4640 10 monclient: shutdown 2024-09-11T01:40:25.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dfdb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:25.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dfdb4640 1 -- 172.21.15.12:0/2852842248 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d80a1380 msgr2=0x7fd6d80a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:25.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dfdb4640 1 --2- 172.21.15.12:0/2852842248 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d80a1380 0x7fd6d80a1760 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4009870 tx=0x7fd6d402a0a0 comp rx=0 tx=0).stop 2024-09-11T01:40:25.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dfdb4640 1 -- 172.21.15.12:0/2852842248 shutdown_connections 2024-09-11T01:40:25.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dfdb4640 1 --2- 172.21.15.12:0/2852842248 >> v2:172.21.15.12:3300/0 conn(0x7fd6d80a0c90 0x7fd6d81361f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:25.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dfdb4640 1 --2- 172.21.15.12:0/2852842248 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d80a1380 0x7fd6d80a1760 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:25.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dfdb4640 1 -- 172.21.15.12:0/2852842248 >> 172.21.15.12:0/2852842248 conn(0x7fd6d80a2f70 msgr2=0x7fd6d80a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:25.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dfdb4640 1 -- 172.21.15.12:0/2852842248 shutdown_connections 2024-09-11T01:40:25.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.020+0000 7fd6dfdb4640 1 -- 172.21.15.12:0/2852842248 wait complete. 2024-09-11T01:40:25.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.021+0000 7fd6dfdb4640 10 monclient: build_initial_monmap 2024-09-11T01:40:25.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.021+0000 7fd6dfdb4640 10 monclient: monmap: 2024-09-11T01:40:25.030 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:25.031 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:40:25.031 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:40:25.031 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:40:25.031 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:25.031 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:25.031 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:40:25.031 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:40:25.031 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:40:25.031 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:25.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.021+0000 7fd6dfdb4640 1 Processor -- start 2024-09-11T01:40:25.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.021+0000 7fd6dfdb4640 1 -- start start 2024-09-11T01:40:25.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.021+0000 7fd6dfdb4640 10 monclient: init 2024-09-11T01:40:25.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dfdb4640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:25.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dfdb4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:40:25.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dfdb4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6d80a0370 0x7fd6d8135b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:25.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dfdb4640 10 monclient(hunting): picked mon.noname-b con 0x7fd6d80a0370 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:40:25.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dfdb4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d80a0c90 0x7fd6d8136070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:25.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dfdb4640 10 monclient(hunting): picked mon.noname-a con 0x7fd6d80a0c90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:40:25.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dfdb4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d80a1380 0x7fd6d8123750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:25.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dfdb4640 10 monclient(hunting): picked mon.noname-c con 0x7fd6d80a1380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:40:25.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dfdb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:25.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dfdb4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d809ce20 con 0x7fd6d80a0c90 2024-09-11T01:40:25.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dfdb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:25.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dfdb4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d809cca0 con 0x7fd6d80a0370 2024-09-11T01:40:25.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dfdb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:25.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dfdb4640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd6d8136730 con 0x7fd6d80a1380 2024-09-11T01:40:25.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dfdb4640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:25.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dfdb4640 10 monclient(hunting): authenticate will time out at 1431.464266s 2024-09-11T01:40:25.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6de351640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d80a1380 0x7fd6d8123750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:25.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dd34f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d80a0c90 0x7fd6d8136070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:25.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dd34f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d80a0c90 0x7fd6d8136070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:32774/0 (socket says 172.21.15.12:32774) 2024-09-11T01:40:25.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6de351640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d80a1380 0x7fd6d8123750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:49350/0 (socket says 172.21.15.12:49350) 2024-09-11T01:40:25.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dd34f640 1 -- 172.21.15.12:0/1095886423 learned_addr learned my addr 172.21.15.12:0/1095886423 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:40:25.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dd34f640 10 monclient(hunting): get_auth_request con 0x7fd6d80a0c90 auth_method 0 2024-09-11T01:40:25.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dd34f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:25.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dd34f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:25.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dd34f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:25.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6de351640 10 monclient(hunting): get_auth_request con 0x7fd6d80a1380 auth_method 0 2024-09-11T01:40:25.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6de351640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:25.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6de351640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:25.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6de351640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:25.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6ddb50640 1 --2- 172.21.15.12:0/1095886423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6d80a0370 0x7fd6d8135b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:25.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dd34f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:25.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dd34f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:25.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dd34f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:25.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6de351640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:25.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6de351640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:25.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6de351640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:25.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6ddb50640 10 monclient(hunting): get_auth_request con 0x7fd6d80a0370 auth_method 0 2024-09-11T01:40:25.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6ddb50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:25.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6ddb50640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:25.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6ddb50640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:25.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6ddb50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:25.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6ddb50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:25.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6ddb50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:25.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6de351640 10 monclient(hunting): handle_auth_done global_id 114242 payload 824 2024-09-11T01:40:25.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6de351640 1 -- 172.21.15.12:0/1095886423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6d80a0370 msgr2=0x7fd6d8135b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:25.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6de351640 1 --2- 172.21.15.12:0/1095886423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6d80a0370 0x7fd6d8135b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:25.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6de351640 1 -- 172.21.15.12:0/1095886423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d80a0c90 msgr2=0x7fd6d8136070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:25.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6de351640 1 --2- 172.21.15.12:0/1095886423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d80a0c90 0x7fd6d8136070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:25.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6de351640 10 monclient: _finish_hunting 0 2024-09-11T01:40:25.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6de351640 1 monclient: found mon.noname-c 2024-09-11T01:40:25.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6de351640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:25.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6de351640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:40:25.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6de351640 1 -- 172.21.15.12:0/1095886423 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d81263a0 con 0x7fd6d80a1380 2024-09-11T01:40:25.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.022+0000 7fd6dd34f640 1 --2- 172.21.15.12:0/1095886423 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6d80a0c90 0x7fd6d8136070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:40:25.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.024+0000 7fd6de351640 1 -- 172.21.15.12:0/1095886423 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d80a1380 msgr2=0x7fd6d8123750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:25.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.024+0000 7fd6de351640 1 --2- 172.21.15.12:0/1095886423 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d80a1380 0x7fd6d8123750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd6d000efc0 tx=0x7fd6d000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:25.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.025+0000 7fd6ce7fc640 1 -- 172.21.15.12:0/1095886423 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d0018020 con 0x7fd6d80a1380 2024-09-11T01:40:25.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.025+0000 7fd6ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:25.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.025+0000 7fd6ce7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:40:25.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.025+0000 7fd6ce7fc640 10 monclient: dump: 2024-09-11T01:40:25.042 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:25.042 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:25.043 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:25.043 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:25.043 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:25.043 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:25.043 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:25.043 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:25.043 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:25.043 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:25.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.025+0000 7fd6ce7fc640 10 monclient: _finish_auth 0 2024-09-11T01:40:25.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.025+0000 7fd6ce7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:40:25.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.025+0000 7fd6ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:25.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.025+0000 7fd6ce7fc640 1 -- 172.21.15.12:0/1095886423 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6d00146a0 con 0x7fd6d80a1380 2024-09-11T01:40:25.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.025+0000 7fd6dfdb4640 5 monclient: authenticate success, global_id 114242 2024-09-11T01:40:25.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.025+0000 7fd6dfdb4640 10 monclient: _renew_subs 2024-09-11T01:40:25.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.025+0000 7fd6dfdb4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:40:25.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.025+0000 7fd6dfdb4640 1 -- 172.21.15.12:0/1095886423 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6d81251d0 con 0x7fd6d80a1380 2024-09-11T01:40:25.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.025+0000 7fd6ce7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:25.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.025+0000 7fd6ce7fc640 1 -- 172.21.15.12:0/1095886423 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d0014a90 con 0x7fd6d80a1380 2024-09-11T01:40:25.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.025+0000 7fd6ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:25.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.025+0000 7fd6ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:40:25.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.025+0000 7fd6ce7fc640 10 monclient: dump: 2024-09-11T01:40:25.044 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:25.044 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:25.044 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:25.044 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:25.044 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:25.044 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:25.045 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:25.045 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:25.045 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:25.045 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:25.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.025+0000 7fd6dfdb4640 10 monclient: _renew_subs 2024-09-11T01:40:25.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.025+0000 7fd6dfdb4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:40:25.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.025+0000 7fd6dfdb4640 1 -- 172.21.15.12:0/1095886423 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6d8127530 con 0x7fd6d80a1380 2024-09-11T01:40:25.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.025+0000 7fd6cffff640 4 set_mon_vals no callback set 2024-09-11T01:40:25.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.025+0000 7fd6dfdb4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:25.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.026+0000 7fd6dfdb4640 1 --2- 172.21.15.12:0/1095886423 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d81caee0 0x7fd6d802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:25.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.026+0000 7fd6dfdb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:25.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.026+0000 7fd6dfdb4640 1 -- 172.21.15.12:0/1095886423 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd6d81271b0 con 0x7fd6d81caee0 2024-09-11T01:40:25.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.026+0000 7fd6ddb50640 1 --2- 172.21.15.12:0/1095886423 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d81caee0 0x7fd6d802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:25.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.026+0000 7fd6ddb50640 10 monclient: get_auth_request con 0x7fd6d81caee0 auth_method 0 2024-09-11T01:40:25.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.026+0000 7fd6ddb50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:25.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.026+0000 7fd6ddb50640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:25.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.026+0000 7fd6ddb50640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:25.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.026+0000 7fd6ce7fc640 1 -- 172.21.15.12:0/1095886423 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fd6d0014c30 con 0x7fd6d80a1380 2024-09-11T01:40:25.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.026+0000 7fd6ddb50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:25.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.026+0000 7fd6ddb50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:25.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.026+0000 7fd6ddb50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:25.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.026+0000 7fd6ddb50640 10 monclient(hunting): handle_auth_done global_id 114245 payload 824 2024-09-11T01:40:25.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.026+0000 7fd6ddb50640 1 -- 172.21.15.12:0/1095886423 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd6d80a1760 con 0x7fd6d81caee0 2024-09-11T01:40:25.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.026+0000 7fd6ce7fc640 1 --2- 172.21.15.12:0/1095886423 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd6bc03e290 0x7fd6bc040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:25.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.026+0000 7fd6ce7fc640 1 -- 172.21.15.12:0/1095886423 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(380..380 src has 285..380) ==== 4783+0+0 (secure 0 0 0) 0x7fd6d0053f00 con 0x7fd6d80a1380 2024-09-11T01:40:25.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.026+0000 7fd6dd34f640 1 --2- 172.21.15.12:0/1095886423 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd6bc03e290 0x7fd6bc040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:25.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.026+0000 7fd6dd34f640 10 monclient: get_auth_request con 0x7fd6bc03e290 auth_method 0 2024-09-11T01:40:25.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.028+0000 7fd6ddb50640 1 -- 172.21.15.12:0/1095886423 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d81caee0 msgr2=0x7fd6d802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:25.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.028+0000 7fd6ddb50640 1 --2- 172.21.15.12:0/1095886423 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d81caee0 0x7fd6d802d440 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd6d806ce00 tx=0x7fd6d40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:25.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.028+0000 7fd6ce7fc640 1 -- 172.21.15.12:0/1095886423 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4043070 con 0x7fd6d81caee0 2024-09-11T01:40:25.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.028+0000 7fd6ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:25.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.028+0000 7fd6ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:40:25.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.028+0000 7fd6ce7fc640 10 monclient: dump: 2024-09-11T01:40:25.048 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:25.048 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:25.048 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:25.048 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:25.048 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:25.048 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:25.048 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:25.048 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:25.048 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:25.048 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:25.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.029+0000 7fd6ce7fc640 1 -- 172.21.15.12:0/1095886423 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd6d403c440 con 0x7fd6d81caee0 2024-09-11T01:40:25.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.029+0000 7fd6ce7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:25.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.029+0000 7fd6ce7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:40:25.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.029+0000 7fd6ce7fc640 1 -- 172.21.15.12:0/1095886423 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d81caee0 msgr2=0x7fd6d802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:25.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.029+0000 7fd6ce7fc640 1 --2- 172.21.15.12:0/1095886423 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d81caee0 0x7fd6d802d440 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd6d806ce00 tx=0x7fd6d40046c0 comp rx=0 tx=0).stop 2024-09-11T01:40:25.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.029+0000 7fd6ce7fc640 1 -- 172.21.15.12:0/1095886423 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d81caee0 msgr2=0x7fd6d802d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:40:25.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.029+0000 7fd6ce7fc640 1 --2- 172.21.15.12:0/1095886423 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d81caee0 0x7fd6d802d440 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd6d806ce00 tx=0x7fd6d40046c0 comp rx=0 tx=0).stop 2024-09-11T01:40:25.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.029+0000 7fd6dd34f640 1 --2- 172.21.15.12:0/1095886423 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd6bc03e290 0x7fd6bc040750 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8126bc0 tx=0x7fd6c8007ef0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:25.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.035+0000 7fd6dfdb4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:40:25.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.035+0000 7fd6dfdb4640 1 --2- 172.21.15.12:0/1095886423 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d802d980 0x7fd6d802dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:25.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.035+0000 7fd6dfdb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:25.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.035+0000 7fd6dfdb4640 1 -- 172.21.15.12:0/1095886423 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd6d8120000 con 0x7fd6d802d980 2024-09-11T01:40:25.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.035+0000 7fd6ddb50640 1 --2- 172.21.15.12:0/1095886423 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d802d980 0x7fd6d802dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:25.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.035+0000 7fd6ddb50640 10 monclient: get_auth_request con 0x7fd6d802d980 auth_method 0 2024-09-11T01:40:25.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.035+0000 7fd6ddb50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:25.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.035+0000 7fd6ddb50640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:25.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.035+0000 7fd6ddb50640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:25.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.035+0000 7fd6ddb50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:25.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.035+0000 7fd6ddb50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:25.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.035+0000 7fd6ddb50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:25.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.035+0000 7fd6ddb50640 10 monclient(hunting): handle_auth_done global_id 114248 payload 824 2024-09-11T01:40:25.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.035+0000 7fd6ddb50640 1 -- 172.21.15.12:0/1095886423 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd6d806d480 con 0x7fd6d802d980 2024-09-11T01:40:25.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.037+0000 7fd6ddb50640 1 -- 172.21.15.12:0/1095886423 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d802d980 msgr2=0x7fd6d802dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:25.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.037+0000 7fd6ddb50640 1 --2- 172.21.15.12:0/1095886423 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d802d980 0x7fd6d802dd90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd6d402a0e0 tx=0x7fd6d4008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:25.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.037+0000 7fd6ce7fc640 1 -- 172.21.15.12:0/1095886423 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4037040 con 0x7fd6d802d980 2024-09-11T01:40:25.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.037+0000 7fd6ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:25.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.037+0000 7fd6ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:40:25.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.037+0000 7fd6ce7fc640 10 monclient: dump: 2024-09-11T01:40:25.052 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:25.052 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:25.052 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:25.052 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:25.052 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:25.052 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:25.052 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:25.052 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:25.052 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:25.052 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:25.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.038+0000 7fd6ce7fc640 1 -- 172.21.15.12:0/1095886423 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7fd6d4049c60 con 0x7fd6d802d980 2024-09-11T01:40:25.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.038+0000 7fd6ce7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:40:25.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.038+0000 7fd6ce7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:40:25.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.038+0000 7fd6ce7fc640 1 -- 172.21.15.12:0/1095886423 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d802d980 msgr2=0x7fd6d802dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:25.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.038+0000 7fd6ce7fc640 1 --2- 172.21.15.12:0/1095886423 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d802d980 0x7fd6d802dd90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd6d402a0e0 tx=0x7fd6d4008030 comp rx=0 tx=0).stop 2024-09-11T01:40:25.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.038+0000 7fd6ce7fc640 1 -- 172.21.15.12:0/1095886423 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d802d980 msgr2=0x7fd6d802dd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:40:25.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.038+0000 7fd6ce7fc640 1 --2- 172.21.15.12:0/1095886423 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d802d980 0x7fd6d802dd90 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd6d402a0e0 tx=0x7fd6d4008030 comp rx=0 tx=0).stop 2024-09-11T01:40:25.053 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:40:25.053 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:40:25.053 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:40:25.053 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:40:25.053 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 106, 2024-09-11T01:40:25.054 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 70834, 2024-09-11T01:40:25.054 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:40:25.054 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:40:25.054 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:40:25.054 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:40:25.054 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:25.054 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 5, 2024-09-11T01:40:25.054 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:40:25.054 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:40:25.054 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:40:25.054 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:40:25.054 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:40:25.054 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:40:25.055 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:40:25.055 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:40:25.055 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:40:25.055 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:40:25.055 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:40:25.055 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:40:25.055 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:40:25.055 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:40:25.055 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:25.055 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:40:25.055 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:40:25.055 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:40:25.056 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:40:25.056 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:40:25.056 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:40:25.056 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:40:25.056 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:40:25.056 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:40:25.056 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:40:25.056 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:40:25.056 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:40:25.056 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:40:25.056 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:25.056 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:25.057 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:40:25.057 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:40:25.057 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:40:25.057 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:40:25.057 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:40:25.057 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:40:25.057 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:40:25.057 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:40:25.057 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:40:25.057 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:40:25.057 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:40:25.057 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:40:25.057 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:40:25.058 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:40:25.058 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:40:25.058 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:40:25.058 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:40:25.058 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:40:25.058 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:40:25.058 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:40:25.058 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:40:25.058 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:40:25.058 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:40:25.058 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:40:25.058 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:40:25.058 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:40:25.058 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:40:25.058 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:40:25.059 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:25.059 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:40:25.060 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:25.060 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:40:25.060 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:25.060 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:40:25.060 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:40:25.060 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:40:25.060 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:40:25.060 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:25.060 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:40:25.060 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:40:25.061 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:25.061 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:25.061 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:25.061 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:40:25.061 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:40:25.061 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:25.061 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:25.061 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:25.061 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:25.062 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:40:25.062 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:40:25.062 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:40:25.062 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:40:25.062 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:40:25.062 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:25.062 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:25.062 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:40:25.062 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:40:25.062 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:40:25.062 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:40:25.062 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:25.062 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:40:25.063 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:40:25.063 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:25.063 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:25.063 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:25.063 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:40:25.063 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:40:25.063 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:25.063 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:25.063 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:25.063 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:25.063 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:40:25.063 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:40:25.063 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:40:25.064 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:40:25.064 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:40:25.064 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:25.064 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:25.064 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:40:25.064 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:40:25.064 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:40:25.064 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:40:25.064 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:25.064 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:40:25.064 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:40:25.064 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:25.064 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:25.065 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:25.065 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:40:25.065 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:40:25.065 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:25.065 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:25.065 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:25.065 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:25.065 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:40:25.065 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:40:25.065 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:40:25.065 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:40:25.066 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:40:25.066 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:25.066 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:25.066 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:25.066 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:40:25.066 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:40:25.066 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:25.066 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:40:25.066 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:40:25.066 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:40:25.066 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:25.066 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:25.067 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:40:25.067 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:25.067 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:40:25.067 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:40:25.067 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:40:25.067 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:25.067 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:25.067 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:40:25.067 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:25.067 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:40:25.067 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:40:25.068 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:40:25.068 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:25.068 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:25.068 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:40:25.068 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:25.068 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:40:25.068 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:40:25.068 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:40:25.068 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:25.068 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:25.068 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:25.068 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:40:25.068 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:40:25.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.038+0000 7fd6dfdb4640 1 -- 172.21.15.12:0/1095886423 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd6bc03e290 msgr2=0x7fd6bc040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:25.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.038+0000 7fd6dfdb4640 1 --2- 172.21.15.12:0/1095886423 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd6bc03e290 0x7fd6bc040750 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8126bc0 tx=0x7fd6c8007ef0 comp rx=0 tx=0).stop 2024-09-11T01:40:25.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.038+0000 7fd6dfdb4640 10 monclient: shutdown 2024-09-11T01:40:25.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.038+0000 7fd6dfdb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:25.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.038+0000 7fd6dfdb4640 1 -- 172.21.15.12:0/1095886423 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d80a1380 msgr2=0x7fd6d8123750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:25.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.038+0000 7fd6de351640 1 -- 172.21.15.12:0/1095886423 reap_dead start 2024-09-11T01:40:25.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.038+0000 7fd6dfdb4640 1 --2- 172.21.15.12:0/1095886423 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d80a1380 0x7fd6d8123750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd6d000efc0 tx=0x7fd6d000c490 comp rx=0 tx=0).stop 2024-09-11T01:40:25.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.038+0000 7fd6dfdb4640 1 -- 172.21.15.12:0/1095886423 shutdown_connections 2024-09-11T01:40:25.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.038+0000 7fd6dfdb4640 1 --2- 172.21.15.12:0/1095886423 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6d80a1380 0x7fd6d8123750 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:25.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.038+0000 7fd6dfdb4640 1 -- 172.21.15.12:0/1095886423 >> 172.21.15.12:0/1095886423 conn(0x7fd6d80a2f70 msgr2=0x7fd6d80a5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:25.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.038+0000 7fd6dfdb4640 1 -- 172.21.15.12:0/1095886423 shutdown_connections 2024-09-11T01:40:25.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.038+0000 7fd6dfdb4640 1 -- 172.21.15.12:0/1095886423 wait complete. 2024-09-11T01:40:25.071 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:40:25.071 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:40:25.071 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:40:25.073 INFO:tasks.rados.rados.0.smithi071.stdout:1657: finishing write tid 1 to smithi07134777-2 2024-09-11T01:40:25.073 INFO:tasks.rados.rados.0.smithi071.stdout:1657: finishing write tid 2 to smithi07134777-2 2024-09-11T01:40:25.073 INFO:tasks.rados.rados.0.smithi071.stdout:1657: finishing write tid 3 to smithi07134777-2 2024-09-11T01:40:25.073 INFO:tasks.rados.rados.0.smithi071.stdout:1657: finishing write tid 4 to smithi07134777-2 2024-09-11T01:40:25.073 INFO:tasks.rados.rados.0.smithi071.stdout:1657: oid 2 updating version 0 to 451 2024-09-11T01:40:25.073 INFO:tasks.rados.rados.0.smithi071.stdout:1657: oid 2 updating version 451 to 453 2024-09-11T01:40:25.073 INFO:tasks.rados.rados.0.smithi071.stdout:1657: oid 2 version 453 is already newer than 452 2024-09-11T01:40:25.073 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 453 (ObjNum 611 snap 181 seq_num 611) dirty exists 2024-09-11T01:40:25.073 INFO:tasks.rados.rados.0.smithi071.stdout:1657: left oid 2 (ObjNum 611 snap 181 seq_num 611) 2024-09-11T01:40:25.073 INFO:tasks.rados.rados.0.smithi071.stdout:1658: seq_num 612 ranges {685884=560628,2046274=239610} 2024-09-11T01:40:25.073 INFO:tasks.rados.rados.0.smithi071.stdout:1658: writing smithi07134777-35 from 685884 to 1246512 tid 1 2024-09-11T01:40:25.074 INFO:tasks.rados.rados.0.smithi071.stdout:1658: writing smithi07134777-35 from 2046274 to 2285884 tid 2 2024-09-11T01:40:25.074 INFO:tasks.rados.rados.0.smithi071.stdout:1655: done (2 left) 2024-09-11T01:40:25.074 INFO:tasks.rados.rados.0.smithi071.stdout:1657: done (1 left) 2024-09-11T01:40:25.074 INFO:tasks.rados.rados.0.smithi071.stdout:1659: write oid 18 current snap is 181 2024-09-11T01:40:25.074 INFO:tasks.rados.rados.0.smithi071.stdout:1659: seq_num 613 ranges {702691=635877,2104756=197935} 2024-09-11T01:40:25.074 INFO:tasks.rados.rados.0.smithi071.stdout:1659: writing smithi07134777-18 from 702691 to 1338568 tid 1 2024-09-11T01:40:25.078 INFO:tasks.rados.rados.0.smithi071.stdout:1659: writing smithi07134777-18 from 2104756 to 2302691 tid 2 2024-09-11T01:40:25.078 INFO:tasks.rados.rados.0.smithi071.stdout:1660: write oid 29 current snap is 181 2024-09-11T01:40:25.078 INFO:tasks.rados.rados.0.smithi071.stdout:1658: finishing write tid 1 to smithi07134777-35 2024-09-11T01:40:25.078 INFO:tasks.rados.rados.0.smithi071.stdout:1658: finishing write tid 2 to smithi07134777-35 2024-09-11T01:40:25.078 INFO:tasks.rados.rados.0.smithi071.stdout:1658: finishing write tid 3 to smithi07134777-35 2024-09-11T01:40:25.078 INFO:tasks.rados.rados.0.smithi071.stdout:1660: seq_num 614 ranges {719498=711126,2046885=272613} 2024-09-11T01:40:25.085 INFO:tasks.rados.rados.0.smithi071.stdout:1660: writing smithi07134777-29 from 719498 to 1430624 tid 1 2024-09-11T01:40:25.090 INFO:tasks.rados.rados.0.smithi071.stdout:1660: writing smithi07134777-29 from 2046885 to 2319498 tid 2 2024-09-11T01:40:25.091 INFO:tasks.rados.rados.0.smithi071.stdout:1661: snap_remove snap 178 2024-09-11T01:40:25.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.175+0000 7f5b4f153640 10 monclient: get_monmap_and_config 2024-09-11T01:40:25.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.175+0000 7f5b4f153640 10 monclient: build_initial_monmap 2024-09-11T01:40:25.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.175+0000 7f5b4f153640 10 monclient: monmap: 2024-09-11T01:40:25.177 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:25.177 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:25.177 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:40:25.176119+0000 2024-09-11T01:40:25.177 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:40:25.176119+0000 2024-09-11T01:40:25.177 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:25.178 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:25.178 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:40:25.178 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:40:25.178 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:40:25.178 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:40:25.178 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:40:25.178 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:25.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.176+0000 7f5b4f153640 1 Processor -- start 2024-09-11T01:40:25.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.176+0000 7f5b4f153640 1 -- start start 2024-09-11T01:40:25.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.176+0000 7f5b4f153640 10 monclient: init 2024-09-11T01:40:25.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.176+0000 7f5b4f153640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:25.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.176+0000 7f5b4f153640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-11T01:40:25.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.176+0000 7f5b4f153640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5b48074180 addr v1:172.21.15.12:6789/0 2024-09-11T01:40:25.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.176+0000 7f5b4f153640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5b480a06d0 addr v1:172.21.15.71:6789/0 2024-09-11T01:40:25.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.176+0000 7f5b4f153640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b48134ac0 0x7f5b481405f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:25.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.176+0000 7f5b4f153640 10 monclient(hunting): picked mon.noname-b con 0x7f5b48134ac0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:40:25.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.176+0000 7f5b4f153640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b48070160 con 0x7f5b48074180 2024-09-11T01:40:25.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.176+0000 7f5b4f153640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b48070440 con 0x7f5b480a06d0 2024-09-11T01:40:25.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.176+0000 7f5b4f153640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:25.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.176+0000 7f5b4f153640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5b48135560 con 0x7f5b48134ac0 2024-09-11T01:40:25.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.176+0000 7f5b4f153640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:25.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.176+0000 7f5b4f153640 10 monclient(hunting): authenticate will time out at 1431.618700s 2024-09-11T01:40:25.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.176+0000 7f5b4ceef640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5b48074180 0x7f5b48074560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43302/0 (socket says 172.21.15.12:43302) 2024-09-11T01:40:25.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.176+0000 7f5b4ceef640 1 -- 172.21.15.12:0/2885189222 learned_addr learned my addr 172.21.15.12:0/2885189222 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:40:25.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.176+0000 7f5b4d6f0640 1 --2- 172.21.15.12:0/2885189222 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b48134ac0 0x7f5b481405f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:25.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.176+0000 7f5b4d6f0640 10 monclient(hunting): get_auth_request con 0x7f5b48134ac0 auth_method 0 2024-09-11T01:40:25.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.176+0000 7f5b4d6f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:25.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.176+0000 7f5b4d6f0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:25.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.176+0000 7f5b4d6f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:25.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.176+0000 7f5b4d6f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:25.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.176+0000 7f5b4d6f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:25.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.176+0000 7f5b4d6f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:25.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.177+0000 7f5b4d6f0640 10 monclient(hunting): handle_auth_done global_id 114251 payload 274 2024-09-11T01:40:25.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.177+0000 7f5b4d6f0640 1 -- 172.21.15.12:0/2885189222 >> v1:172.21.15.71:6789/0 conn(0x7f5b480a06d0 legacy=0x7f5b4813cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:40:25.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.177+0000 7f5b4d6f0640 1 -- 172.21.15.12:0/2885189222 >> v1:172.21.15.12:6789/0 conn(0x7f5b48074180 legacy=0x7f5b48074560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:40:25.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.177+0000 7f5b4d6f0640 10 monclient: _finish_hunting 0 2024-09-11T01:40:25.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.177+0000 7f5b4d6f0640 1 monclient: found mon.noname-b 2024-09-11T01:40:25.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.177+0000 7f5b4d6f0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:25.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.177+0000 7f5b4d6f0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:40:25.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.177+0000 7f5b4d6f0640 1 -- 172.21.15.12:0/2885189222 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b48068bc0 con 0x7f5b48134ac0 2024-09-11T01:40:25.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.179+0000 7f5b4d6f0640 1 -- 172.21.15.12:0/2885189222 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b48134ac0 msgr2=0x7f5b481405f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:25.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.179+0000 7f5b4d6f0640 1 --2- 172.21.15.12:0/2885189222 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b48134ac0 0x7f5b481405f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5b40009dc0 tx=0x7f5b4002a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da823e30cf7c7dca server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:25.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.180+0000 7f5b477fe640 1 -- 172.21.15.12:0/2885189222 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b40007a30 con 0x7f5b48134ac0 2024-09-11T01:40:25.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.180+0000 7f5b477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:25.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.180+0000 7f5b477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:40:25.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.180+0000 7f5b477fe640 10 monclient: dump: 2024-09-11T01:40:25.185 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:25.185 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:25.185 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:25.185 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:25.185 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:25.185 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:25.185 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:25.185 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:25.186 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:25.186 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:25.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.180+0000 7f5b477fe640 10 monclient: _finish_auth 0 2024-09-11T01:40:25.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.180+0000 7f5b477fe640 10 monclient: _check_auth_tickets 2024-09-11T01:40:25.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.180+0000 7f5b477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:25.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.180+0000 7f5b4f153640 5 monclient: authenticate success, global_id 114251 2024-09-11T01:40:25.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.180+0000 7f5b477fe640 1 -- 172.21.15.12:0/2885189222 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b40007bd0 con 0x7f5b48134ac0 2024-09-11T01:40:25.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.180+0000 7f5b4f153640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:40:25.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.180+0000 7f5b477fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:25.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.180+0000 7f5b4f153640 10 monclient: get_monmap_and_config success 2024-09-11T01:40:25.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.180+0000 7f5b4f153640 4 set_mon_vals no callback set 2024-09-11T01:40:25.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.180+0000 7f5b477fe640 1 -- 172.21.15.12:0/2885189222 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b40031400 con 0x7f5b48134ac0 2024-09-11T01:40:25.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.180+0000 7f5b477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:25.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.180+0000 7f5b477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:40:25.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.180+0000 7f5b477fe640 10 monclient: dump: 2024-09-11T01:40:25.187 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:25.187 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:25.187 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:25.187 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:25.187 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:25.187 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:25.187 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:25.187 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:25.187 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:25.187 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:25.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.180+0000 7f5b4f153640 10 monclient: shutdown 2024-09-11T01:40:25.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.180+0000 7f5b4f153640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:25.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.180+0000 7f5b4f153640 1 -- 172.21.15.12:0/2885189222 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b48134ac0 msgr2=0x7f5b481405f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:25.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.180+0000 7f5b4f153640 1 --2- 172.21.15.12:0/2885189222 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b48134ac0 0x7f5b481405f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5b40009dc0 tx=0x7f5b4002a480 comp rx=0 tx=0).stop 2024-09-11T01:40:25.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.180+0000 7f5b4f153640 1 -- 172.21.15.12:0/2885189222 shutdown_connections 2024-09-11T01:40:25.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.180+0000 7f5b4f153640 1 --2- 172.21.15.12:0/2885189222 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b48134ac0 0x7f5b481405f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:25.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.180+0000 7f5b4f153640 1 -- 172.21.15.12:0/2885189222 >> 172.21.15.12:0/2885189222 conn(0x7f5b480a36c0 msgr2=0x7f5b480a3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:25.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.180+0000 7f5b4f153640 1 -- 172.21.15.12:0/2885189222 shutdown_connections 2024-09-11T01:40:25.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.180+0000 7f5b4f153640 1 -- 172.21.15.12:0/2885189222 wait complete. 2024-09-11T01:40:25.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.181+0000 7f5b4f153640 10 monclient: build_initial_monmap 2024-09-11T01:40:25.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.181+0000 7f5b4f153640 10 monclient: monmap: 2024-09-11T01:40:25.194 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:25.194 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:40:25.194 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:40:25.194 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:40:25.194 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:25.194 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:25.194 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:40:25.194 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:40:25.194 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:40:25.194 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:25.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.181+0000 7f5b4f153640 1 Processor -- start 2024-09-11T01:40:25.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.181+0000 7f5b4f153640 1 -- start start 2024-09-11T01:40:25.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.181+0000 7f5b4f153640 10 monclient: init 2024-09-11T01:40:25.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4f153640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:25.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4f153640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:40:25.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4f153640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b48074180 0x7f5b4813b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:25.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4f153640 10 monclient(hunting): picked mon.noname-c con 0x7f5b48074180 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:40:25.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4f153640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b480a06d0 0x7f5b4813b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:25.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4f153640 10 monclient(hunting): picked mon.noname-a con 0x7f5b480a06d0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:40:25.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4f153640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b48134ac0 0x7f5b4813bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:25.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4f153640 10 monclient(hunting): picked mon.noname-b con 0x7f5b48134ac0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:40:25.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4f153640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:25.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4f153640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5b48140d40 con 0x7f5b480a06d0 2024-09-11T01:40:25.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4f153640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:25.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4f153640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5b48070740 con 0x7f5b48134ac0 2024-09-11T01:40:25.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4f153640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:25.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4f153640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5b48140ec0 con 0x7f5b48074180 2024-09-11T01:40:25.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4f153640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:25.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4f153640 10 monclient(hunting): authenticate will time out at 1431.624243s 2024-09-11T01:40:25.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4ceef640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b48074180 0x7f5b4813b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:25.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4ceef640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b48074180 0x7f5b4813b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:49392/0 (socket says 172.21.15.12:49392) 2024-09-11T01:40:25.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4ceef640 1 -- 172.21.15.12:0/2471262632 learned_addr learned my addr 172.21.15.12:0/2471262632 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:40:25.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4ceef640 10 monclient(hunting): get_auth_request con 0x7f5b48074180 auth_method 0 2024-09-11T01:40:25.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4ceef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:25.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4ceef640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:25.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4ceef640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b47fff640 1 --2- 172.21.15.12:0/2471262632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b480a06d0 0x7f5b4813b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b47fff640 10 monclient(hunting): get_auth_request con 0x7f5b480a06d0 auth_method 0 2024-09-11T01:40:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b47fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4ceef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4ceef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4ceef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4d6f0640 1 --2- 172.21.15.12:0/2471262632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b48134ac0 0x7f5b4813bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4d6f0640 10 monclient(hunting): get_auth_request con 0x7f5b48134ac0 auth_method 0 2024-09-11T01:40:25.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4d6f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:25.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4d6f0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:25.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4d6f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:25.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4ceef640 10 monclient(hunting): handle_auth_done global_id 114254 payload 824 2024-09-11T01:40:25.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4ceef640 1 -- 172.21.15.12:0/2471262632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b48134ac0 msgr2=0x7f5b4813bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:25.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4ceef640 1 --2- 172.21.15.12:0/2471262632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b48134ac0 0x7f5b4813bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:25.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4ceef640 1 -- 172.21.15.12:0/2471262632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b480a06d0 msgr2=0x7f5b4813b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:25.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4ceef640 1 --2- 172.21.15.12:0/2471262632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b480a06d0 0x7f5b4813b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:25.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4ceef640 10 monclient: _finish_hunting 0 2024-09-11T01:40:25.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4ceef640 1 monclient: found mon.noname-c 2024-09-11T01:40:25.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4ceef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:25.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4ceef640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:40:25.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4ceef640 1 -- 172.21.15.12:0/2471262632 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b4813ec00 con 0x7f5b48074180 2024-09-11T01:40:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b4d6f0640 1 --2- 172.21.15.12:0/2471262632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b48134ac0 0x7f5b4813bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:40:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.182+0000 7f5b47fff640 1 --2- 172.21.15.12:0/2471262632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b480a06d0 0x7f5b4813b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:40:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.184+0000 7f5b4ceef640 1 -- 172.21.15.12:0/2471262632 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b48074180 msgr2=0x7f5b4813b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.184+0000 7f5b4ceef640 1 --2- 172.21.15.12:0/2471262632 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b48074180 0x7f5b4813b0c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5b3000c460 tx=0x7f5b3000c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.185+0000 7f5b457fa640 1 -- 172.21.15.12:0/2471262632 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b30007070 con 0x7f5b48074180 2024-09-11T01:40:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.185+0000 7f5b457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.185+0000 7f5b457fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:40:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.185+0000 7f5b457fa640 10 monclient: dump: 2024-09-11T01:40:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:25.200 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:25.200 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:25.200 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:25.200 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:25.200 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:25.200 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:25.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.185+0000 7f5b457fa640 10 monclient: _finish_auth 0 2024-09-11T01:40:25.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.185+0000 7f5b457fa640 10 monclient: _check_auth_tickets 2024-09-11T01:40:25.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.185+0000 7f5b457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:25.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.185+0000 7f5b457fa640 1 -- 172.21.15.12:0/2471262632 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b30007230 con 0x7f5b48074180 2024-09-11T01:40:25.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.185+0000 7f5b4f153640 5 monclient: authenticate success, global_id 114254 2024-09-11T01:40:25.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.185+0000 7f5b457fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.185+0000 7f5b457fa640 1 -- 172.21.15.12:0/2471262632 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b30007620 con 0x7f5b48074180 2024-09-11T01:40:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.185+0000 7f5b4f153640 10 monclient: _renew_subs 2024-09-11T01:40:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.185+0000 7f5b4f153640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:40:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.185+0000 7f5b4f153640 1 -- 172.21.15.12:0/2471262632 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b480a54c0 con 0x7f5b48074180 2024-09-11T01:40:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.185+0000 7f5b457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.185+0000 7f5b457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:40:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.185+0000 7f5b457fa640 10 monclient: dump: 2024-09-11T01:40:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:25.202 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:25.202 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:25.202 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:25.202 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:25.202 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:25.202 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:25.202 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:25.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.185+0000 7f5b46ffd640 4 set_mon_vals no callback set 2024-09-11T01:40:25.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.185+0000 7f5b4f153640 10 monclient: _renew_subs 2024-09-11T01:40:25.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.185+0000 7f5b4f153640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:40:25.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.185+0000 7f5b4f153640 1 -- 172.21.15.12:0/2471262632 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b4813e920 con 0x7f5b48074180 2024-09-11T01:40:25.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.185+0000 7f5b4f153640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:25.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.185+0000 7f5b4f153640 1 --2- 172.21.15.12:0/2471262632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b4813f670 0x7f5b481323f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:25.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.185+0000 7f5b4f153640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:25.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.185+0000 7f5b4f153640 1 -- 172.21.15.12:0/2471262632 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5b48074560 con 0x7f5b4813f670 2024-09-11T01:40:25.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.185+0000 7f5b457fa640 1 -- 172.21.15.12:0/2471262632 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f5b3001b400 con 0x7f5b48074180 2024-09-11T01:40:25.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.186+0000 7f5b457fa640 1 --2- 172.21.15.12:0/2471262632 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5b2803e210 0x7f5b280406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:25.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.186+0000 7f5b47fff640 1 --2- 172.21.15.12:0/2471262632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b4813f670 0x7f5b481323f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:25.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.186+0000 7f5b47fff640 10 monclient: get_auth_request con 0x7f5b4813f670 auth_method 0 2024-09-11T01:40:25.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.186+0000 7f5b47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:25.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.186+0000 7f5b47fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:25.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.186+0000 7f5b47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:25.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.186+0000 7f5b457fa640 1 -- 172.21.15.12:0/2471262632 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(380..380 src has 285..380) ==== 4783+0+0 (secure 0 0 0) 0x7f5b30052a00 con 0x7f5b48074180 2024-09-11T01:40:25.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.186+0000 7f5b47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:25.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.186+0000 7f5b47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:25.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.186+0000 7f5b47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:25.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.186+0000 7f5b4d6f0640 1 --2- 172.21.15.12:0/2471262632 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5b2803e210 0x7f5b280406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:25.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.186+0000 7f5b4d6f0640 10 monclient: get_auth_request con 0x7f5b2803e210 auth_method 0 2024-09-11T01:40:25.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.186+0000 7f5b47fff640 10 monclient(hunting): handle_auth_done global_id 124129 payload 824 2024-09-11T01:40:25.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.186+0000 7f5b47fff640 1 -- 172.21.15.12:0/2471262632 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5b4813da90 con 0x7f5b4813f670 2024-09-11T01:40:25.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.188+0000 7f5b47fff640 1 -- 172.21.15.12:0/2471262632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b4813f670 msgr2=0x7f5b481323f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:25.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.188+0000 7f5b47fff640 1 --2- 172.21.15.12:0/2471262632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b4813f670 0x7f5b481323f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800fae0 tx=0x7f5b380044c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:25.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.188+0000 7f5b4d6f0640 1 --2- 172.21.15.12:0/2471262632 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5b2803e210 0x7f5b280406d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5b4806ef60 tx=0x7f5b4002f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:25.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.189+0000 7f5b457fa640 1 -- 172.21.15.12:0/2471262632 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b38018020 con 0x7f5b4813f670 2024-09-11T01:40:25.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.189+0000 7f5b457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:25.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.189+0000 7f5b457fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:40:25.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.189+0000 7f5b457fa640 10 monclient: dump: 2024-09-11T01:40:25.204 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:25.205 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:25.205 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:25.205 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:25.205 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:25.205 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:25.205 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:25.205 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:25.205 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:25.205 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:25.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.189+0000 7f5b457fa640 1 -- 172.21.15.12:0/2471262632 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5b3800dd30 con 0x7f5b4813f670 2024-09-11T01:40:25.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.189+0000 7f5b457fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:25.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.189+0000 7f5b457fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:40:25.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.189+0000 7f5b457fa640 1 -- 172.21.15.12:0/2471262632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b4813f670 msgr2=0x7f5b481323f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:25.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.189+0000 7f5b457fa640 1 --2- 172.21.15.12:0/2471262632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b4813f670 0x7f5b481323f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800fae0 tx=0x7f5b380044c0 comp rx=0 tx=0).stop 2024-09-11T01:40:25.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.189+0000 7f5b457fa640 1 -- 172.21.15.12:0/2471262632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b4813f670 msgr2=0x7f5b481323f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:40:25.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.189+0000 7f5b457fa640 1 --2- 172.21.15.12:0/2471262632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b4813f670 0x7f5b481323f0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800fae0 tx=0x7f5b380044c0 comp rx=0 tx=0).stop 2024-09-11T01:40:25.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.195+0000 7f5b4f153640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:40:25.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.195+0000 7f5b4f153640 1 --2- 172.21.15.12:0/2471262632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b48132930 0x7f5b48132d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:25.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.195+0000 7f5b4f153640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:25.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.195+0000 7f5b4f153640 1 -- 172.21.15.12:0/2471262632 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5b48133250 con 0x7f5b48132930 2024-09-11T01:40:25.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.195+0000 7f5b47fff640 1 --2- 172.21.15.12:0/2471262632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b48132930 0x7f5b48132d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:25.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.195+0000 7f5b47fff640 10 monclient: get_auth_request con 0x7f5b48132930 auth_method 0 2024-09-11T01:40:25.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.195+0000 7f5b47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:25.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.195+0000 7f5b47fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:25.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.195+0000 7f5b47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:25.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.195+0000 7f5b47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:25.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.195+0000 7f5b47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:25.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.195+0000 7f5b47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:25.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.195+0000 7f5b47fff640 10 monclient(hunting): handle_auth_done global_id 124132 payload 824 2024-09-11T01:40:25.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.195+0000 7f5b47fff640 1 -- 172.21.15.12:0/2471262632 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5b4806dc80 con 0x7f5b48132930 2024-09-11T01:40:25.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.198+0000 7f5b47fff640 1 -- 172.21.15.12:0/2471262632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b48132930 msgr2=0x7f5b48132d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:25.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.198+0000 7f5b47fff640 1 --2- 172.21.15.12:0/2471262632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b48132930 0x7f5b48132d10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800fae0 tx=0x7f5b380036f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:25.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.198+0000 7f5b457fa640 1 -- 172.21.15.12:0/2471262632 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b38018030 con 0x7f5b48132930 2024-09-11T01:40:25.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.198+0000 7f5b457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:25.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.198+0000 7f5b457fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:40:25.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.198+0000 7f5b457fa640 10 monclient: dump: 2024-09-11T01:40:25.209 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:25.209 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:25.209 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:25.209 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:25.210 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:25.210 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:25.210 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:25.210 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:25.210 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:25.210 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:25.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.198+0000 7f5b457fa640 1 -- 172.21.15.12:0/2471262632 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4540 (secure 0 0 0) 0x7f5b38014b00 con 0x7f5b48132930 2024-09-11T01:40:25.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.198+0000 7f5b457fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:40:25.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.198+0000 7f5b457fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:40:25.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.198+0000 7f5b457fa640 1 -- 172.21.15.12:0/2471262632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b48132930 msgr2=0x7f5b48132d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:25.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.198+0000 7f5b457fa640 1 --2- 172.21.15.12:0/2471262632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b48132930 0x7f5b48132d10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800fae0 tx=0x7f5b380036f0 comp rx=0 tx=0).stop 2024-09-11T01:40:25.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.198+0000 7f5b457fa640 1 -- 172.21.15.12:0/2471262632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b48132930 msgr2=0x7f5b48132d10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:40:25.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.198+0000 7f5b457fa640 1 --2- 172.21.15.12:0/2471262632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b48132930 0x7f5b48132d10 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800fae0 tx=0x7f5b380036f0 comp rx=0 tx=0).stop 2024-09-11T01:40:25.211 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:40:25.211 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:40:25.211 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:40:25.211 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:40:25.211 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 106, 2024-09-11T01:40:25.211 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 9991, 2024-09-11T01:40:25.212 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:40:25.212 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:40:25.212 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:40:25.212 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:40:25.212 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:25.212 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 5, 2024-09-11T01:40:25.212 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:40:25.212 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:40:25.212 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:40:25.212 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:40:25.212 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:40:25.212 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:40:25.212 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:40:25.212 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:40:25.213 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:40:25.213 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:40:25.213 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:40:25.213 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:40:25.213 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:40:25.213 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:40:25.213 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:25.213 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:40:25.213 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:40:25.213 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:40:25.213 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:40:25.213 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:40:25.213 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:40:25.214 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:40:25.214 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:40:25.214 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:40:25.214 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:40:25.214 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:40:25.214 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:40:25.214 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:40:25.214 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:25.214 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:25.214 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:40:25.214 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:40:25.214 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:40:25.214 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:40:25.214 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:40:25.215 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:40:25.215 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:40:25.215 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:40:25.215 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:40:25.215 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:40:25.215 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:40:25.215 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:40:25.215 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:40:25.215 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:40:25.215 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:40:25.215 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:40:25.215 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:40:25.215 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:40:25.215 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:40:25.216 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:40:25.216 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:40:25.216 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:40:25.216 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:40:25.216 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:40:25.216 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:40:25.216 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:40:25.216 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:40:25.216 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:40:25.216 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:25.216 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:40:25.216 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:25.216 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:40:25.216 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:25.217 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:40:25.217 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:40:25.217 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:40:25.217 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:40:25.217 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:25.217 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:40:25.217 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:40:25.217 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:25.217 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:25.217 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:25.217 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:40:25.217 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:40:25.217 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:25.217 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:25.218 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:25.218 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:25.218 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:40:25.218 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:40:25.218 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:40:25.218 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:40:25.218 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:40:25.218 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:25.218 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:25.218 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:40:25.218 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:40:25.218 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:40:25.218 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:40:25.218 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:25.219 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:40:25.219 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:40:25.219 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:25.219 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:25.219 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:25.219 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:40:25.219 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:40:25.219 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:25.219 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:25.219 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:25.219 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:25.219 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:40:25.219 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:40:25.220 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:40:25.220 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:40:25.220 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:40:25.220 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:25.220 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:25.220 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:40:25.220 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:40:25.220 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:40:25.220 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:40:25.220 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:25.220 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:40:25.220 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:40:25.220 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:25.221 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:25.221 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:25.221 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:40:25.221 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:40:25.221 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:25.221 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:25.221 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:25.221 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:25.221 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:40:25.221 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:40:25.221 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:40:25.221 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:40:25.221 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:40:25.222 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:25.222 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:25.222 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:25.222 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:40:25.222 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:40:25.222 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:25.222 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:40:25.222 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:40:25.222 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:40:25.222 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:25.222 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:25.222 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:40:25.222 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:25.222 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:40:25.223 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:40:25.223 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:40:25.223 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:25.223 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:25.223 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:25.223 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:40:25.223 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:40:25.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.199+0000 7f5b4f153640 1 -- 172.21.15.12:0/2471262632 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5b2803e210 msgr2=0x7f5b280406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:25.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.199+0000 7f5b4f153640 1 --2- 172.21.15.12:0/2471262632 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5b2803e210 0x7f5b280406d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5b4806ef60 tx=0x7f5b4002f000 comp rx=0 tx=0).stop 2024-09-11T01:40:25.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.199+0000 7f5b4f153640 10 monclient: shutdown 2024-09-11T01:40:25.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.199+0000 7f5b4f153640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:25.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.199+0000 7f5b4f153640 1 -- 172.21.15.12:0/2471262632 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b48074180 msgr2=0x7f5b4813b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:25.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.199+0000 7f5b4f153640 1 --2- 172.21.15.12:0/2471262632 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b48074180 0x7f5b4813b0c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5b3000c460 tx=0x7f5b3000c920 comp rx=0 tx=0).stop 2024-09-11T01:40:25.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.199+0000 7f5b4d6f0640 1 -- 172.21.15.12:0/2471262632 reap_dead start 2024-09-11T01:40:25.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.199+0000 7f5b4f153640 1 -- 172.21.15.12:0/2471262632 shutdown_connections 2024-09-11T01:40:25.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.199+0000 7f5b4f153640 1 --2- 172.21.15.12:0/2471262632 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b48074180 0x7f5b4813b0c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:25.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.199+0000 7f5b4f153640 1 -- 172.21.15.12:0/2471262632 >> 172.21.15.12:0/2471262632 conn(0x7f5b480a36c0 msgr2=0x7f5b480a2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:25.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.199+0000 7f5b4f153640 1 -- 172.21.15.12:0/2471262632 shutdown_connections 2024-09-11T01:40:25.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:25.199+0000 7f5b4f153640 1 -- 172.21.15.12:0/2471262632 wait complete. 2024-09-11T01:40:25.225 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:40:25.226 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-11T01:40:25.226 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-11T01:40:25.226 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-11T01:40:25.226 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-11T01:40:25.226 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-11T01:40:25.226 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:40:25.235 INFO:tasks.ceph.mon.b:Stopped 2024-09-11T01:40:25.236 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-11T01:40:25.236 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-11T01:40:25.236 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T01:40:40.237 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-11T01:40:40.238 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-11T01:40:40.238 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:40:40.238 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:40:40.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.349+0000 7f32a1272640 10 monclient: get_monmap_and_config 2024-09-11T01:40:40.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.349+0000 7f32a1272640 10 monclient: build_initial_monmap 2024-09-11T01:40:40.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.349+0000 7f32a1272640 10 monclient: monmap: 2024-09-11T01:40:40.350 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:40.351 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:40.351 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:40:40.349944+0000 2024-09-11T01:40:40.351 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:40:40.349944+0000 2024-09-11T01:40:40.351 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:40.351 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:40.351 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:40:40.351 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:40:40.351 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:40:40.351 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:40:40.351 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:40:40.351 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:40.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f32a1272640 1 Processor -- start 2024-09-11T01:40:40.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f32a1272640 1 -- start start 2024-09-11T01:40:40.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f32a1272640 10 monclient: init 2024-09-11T01:40:40.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f32a1272640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:40.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f32a1272640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-11T01:40:40.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f32a1272640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f329c138230 0x7f329c138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:40.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f32a1272640 10 monclient(hunting): picked mon.noname-b con 0x7f329c138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:40:40.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f32a1272640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f329c09d030 addr v1:172.21.15.71:6789/0 2024-09-11T01:40:40.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f32a1272640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f329c09d430 addr v1:172.21.15.12:6789/0 2024-09-11T01:40:40.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f32a1272640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f329c0a5e40 con 0x7f329c09d430 2024-09-11T01:40:40.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f32a1272640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f329c0a6e30 con 0x7f329c09d030 2024-09-11T01:40:40.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f32a1272640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:40.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f32a1272640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f329c09c9c0 con 0x7f329c138230 2024-09-11T01:40:40.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f32a1272640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:40.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f32a1272640 10 monclient(hunting): authenticate will time out at 1446.792637s 2024-09-11T01:40:40.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f329b7fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f329c09d430 0x7f329c0a5730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36190/0 (socket says 172.21.15.12:36190) 2024-09-11T01:40:40.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f3293fff640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f329c09d030 legacy=0x7f329c0a5020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:40:40.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f329b7fe640 1 -- 172.21.15.12:0/835478496 learned_addr learned my addr 172.21.15.12:0/835478496 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:40:40.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f329affd640 1 --2- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f329c138230 0x7f329c138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:40.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f329affd640 10 monclient(hunting): get_auth_request con 0x7f329c138230 auth_method 0 2024-09-11T01:40:40.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f329affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:40.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f329affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:40.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f329affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:40.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f329affd640 1 -- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f329c138230 msgr2=0x7f329c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:40:40.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f329affd640 1 -- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f329c138230 msgr2=0x7f329c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:40:40.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f329affd640 1 --2- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f329c138230 0x7f329c138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:40:40.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.350+0000 7f329affd640 1 --2- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f329c138230 0x7f329c138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:40:40.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.352+0000 7f329b7fe640 1 -- 172.21.15.12:0/835478496 >> v1:172.21.15.12:6789/0 conn(0x7f329c09d430 legacy=0x7f329c0a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:40.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.551+0000 7f3293fff640 1 -- 172.21.15.12:0/835478496 >> v1:172.21.15.71:6789/0 conn(0x7f329c09d030 legacy=0x7f329c0a5020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:40:40.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.551+0000 7f329affd640 1 --2- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f329c138230 0x7f329c138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:40.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.551+0000 7f329affd640 10 monclient(hunting): get_auth_request con 0x7f329c138230 auth_method 0 2024-09-11T01:40:40.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.551+0000 7f329affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:40.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.551+0000 7f329affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:40.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.551+0000 7f329affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:40:40.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.551+0000 7f329affd640 1 -- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f329c138230 msgr2=0x7f329c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:40:40.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.551+0000 7f329affd640 1 -- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f329c138230 msgr2=0x7f329c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:40:40.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.551+0000 7f329affd640 1 --2- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f329c138230 0x7f329c138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:40:40.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.551+0000 7f329affd640 1 --2- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f329c138230 0x7f329c138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:40:40.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.953+0000 7f3293fff640 1 -- 172.21.15.12:0/835478496 >> v1:172.21.15.71:6789/0 conn(0x7f329c09d030 legacy=0x7f329c0a5020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:40:40.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.953+0000 7f329affd640 1 --2- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f329c138230 0x7f329c138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:40.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.953+0000 7f329affd640 10 monclient(hunting): get_auth_request con 0x7f329c138230 auth_method 0 2024-09-11T01:40:40.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.953+0000 7f329affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:40.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.953+0000 7f329affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:40.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.953+0000 7f329affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:40:40.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.953+0000 7f329affd640 1 -- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f329c138230 msgr2=0x7f329c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:40:40.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.953+0000 7f329affd640 1 -- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f329c138230 msgr2=0x7f329c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:40:40.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.953+0000 7f329affd640 1 --2- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f329c138230 0x7f329c138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:40:40.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:40.953+0000 7f329affd640 1 --2- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f329c138230 0x7f329c138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:40:41.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:41.754+0000 7f3293fff640 1 -- 172.21.15.12:0/835478496 >> v1:172.21.15.71:6789/0 conn(0x7f329c09d030 legacy=0x7f329c0a5020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:40:41.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:41.754+0000 7f329affd640 1 --2- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f329c138230 0x7f329c138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:41.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:41.754+0000 7f329affd640 10 monclient(hunting): get_auth_request con 0x7f329c138230 auth_method 0 2024-09-11T01:40:41.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:41.754+0000 7f329affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:41.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:41.754+0000 7f329affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:41.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:41.754+0000 7f329affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:40:41.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:41.754+0000 7f329affd640 1 -- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f329c138230 msgr2=0x7f329c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:40:41.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:41.754+0000 7f329affd640 1 -- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f329c138230 msgr2=0x7f329c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:40:41.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:41.754+0000 7f329affd640 1 --2- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f329c138230 0x7f329c138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:40:41.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:41.754+0000 7f329affd640 1 --2- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f329c138230 0x7f329c138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:40:43.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.350+0000 7f32997fa640 10 monclient(hunting): tick 2024-09-11T01:40:43.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.350+0000 7f32997fa640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:40:43.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.350+0000 7f32997fa640 1 monclient(hunting): continuing hunt 2024-09-11T01:40:43.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.350+0000 7f32997fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:40:43.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.350+0000 7f32997fa640 1 -- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f329c138230 msgr2=0x7f329c138610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:40:43.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.350+0000 7f32997fa640 1 --2- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f329c138230 0x7f329c138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:43.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.350+0000 7f32997fa640 1 -- 172.21.15.12:0/835478496 >> v1:172.21.15.71:6789/0 conn(0x7f329c09d030 legacy=0x7f329c0a5020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:40:43.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.350+0000 7f32997fa640 1 -- 172.21.15.12:0/835478496 >> v1:172.21.15.12:6789/0 conn(0x7f329c09d430 legacy=0x7f329c0a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:43.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.350+0000 7f32997fa640 10 monclient: _add_conns ranks=[3,4] 2024-09-11T01:40:43.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.350+0000 7f32997fa640 1 --2- 172.21.15.12:0/835478496 >> v2:172.21.15.12:3300/0 conn(0x7f327c003750 0x7f327c005bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:43.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.350+0000 7f32997fa640 10 monclient(hunting): picked mon.noname-a con 0x7f327c003750 addr v2:172.21.15.12:3300/0 2024-09-11T01:40:43.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.350+0000 7f32997fa640 1 --2- 172.21.15.12:0/835478496 >> v2:172.21.15.71:3300/0 conn(0x7f327c0062c0 0x7f327c008780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:43.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.350+0000 7f32997fa640 10 monclient(hunting): picked mon.noname-c con 0x7f327c0062c0 addr v2:172.21.15.71:3300/0 2024-09-11T01:40:43.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.350+0000 7f32997fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:43.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.350+0000 7f32997fa640 1 -- 172.21.15.12:0/835478496 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f329c09c9c0 con 0x7f327c003750 2024-09-11T01:40:43.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.350+0000 7f32997fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:43.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.350+0000 7f32997fa640 1 -- 172.21.15.12:0/835478496 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f327c008e40 con 0x7f327c0062c0 2024-09-11T01:40:43.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.351+0000 7f32997fa640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:43.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.351+0000 7f329affd640 1 --2- 172.21.15.12:0/835478496 >> v2:172.21.15.12:3300/0 conn(0x7f327c003750 0x7f327c005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:43.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.351+0000 7f329affd640 10 monclient(hunting): get_auth_request con 0x7f327c003750 auth_method 0 2024-09-11T01:40:43.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.351+0000 7f329affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:43.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.351+0000 7f329affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:43.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.351+0000 7f329affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:43.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.351+0000 7f3293fff640 1 -- 172.21.15.12:0/835478496 >> v2:172.21.15.71:3300/0 conn(0x7f327c0062c0 msgr2=0x7f327c008780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:40:43.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.351+0000 7f3293fff640 1 --2- 172.21.15.12:0/835478496 >> v2:172.21.15.71:3300/0 conn(0x7f327c0062c0 0x7f327c008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:40:43.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.351+0000 7f329affd640 1 -- 172.21.15.12:0/835478496 >> v2:172.21.15.12:3300/0 conn(0x7f327c003750 msgr2=0x7f327c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:40:43.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.351+0000 7f329affd640 1 -- 172.21.15.12:0/835478496 >> v2:172.21.15.12:3300/0 conn(0x7f327c003750 msgr2=0x7f327c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:40:43.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.351+0000 7f329affd640 1 --2- 172.21.15.12:0/835478496 >> v2:172.21.15.12:3300/0 conn(0x7f327c003750 0x7f327c005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:40:43.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.351+0000 7f329affd640 1 --2- 172.21.15.12:0/835478496 >> v2:172.21.15.12:3300/0 conn(0x7f327c003750 0x7f327c005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:40:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.552+0000 7f3293fff640 1 -- 172.21.15.12:0/835478496 >> v2:172.21.15.71:3300/0 conn(0x7f327c0062c0 msgr2=0x7f327c008780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:40:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.552+0000 7f3293fff640 1 --2- 172.21.15.12:0/835478496 >> v2:172.21.15.71:3300/0 conn(0x7f327c0062c0 0x7f327c008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:40:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.553+0000 7f329affd640 1 --2- 172.21.15.12:0/835478496 >> v2:172.21.15.12:3300/0 conn(0x7f327c003750 0x7f327c005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.553+0000 7f329affd640 10 monclient(hunting): get_auth_request con 0x7f327c003750 auth_method 0 2024-09-11T01:40:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.553+0000 7f329affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.553+0000 7f329affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.553+0000 7f329affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:40:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.553+0000 7f329affd640 1 -- 172.21.15.12:0/835478496 >> v2:172.21.15.12:3300/0 conn(0x7f327c003750 msgr2=0x7f327c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:40:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.553+0000 7f329affd640 1 -- 172.21.15.12:0/835478496 >> v2:172.21.15.12:3300/0 conn(0x7f327c003750 msgr2=0x7f327c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:40:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.553+0000 7f329affd640 1 --2- 172.21.15.12:0/835478496 >> v2:172.21.15.12:3300/0 conn(0x7f327c003750 0x7f327c005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:40:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.553+0000 7f329affd640 1 --2- 172.21.15.12:0/835478496 >> v2:172.21.15.12:3300/0 conn(0x7f327c003750 0x7f327c005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:40:43.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.953+0000 7f3293fff640 1 -- 172.21.15.12:0/835478496 >> v2:172.21.15.71:3300/0 conn(0x7f327c0062c0 msgr2=0x7f327c008780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:40:43.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.953+0000 7f3293fff640 1 --2- 172.21.15.12:0/835478496 >> v2:172.21.15.71:3300/0 conn(0x7f327c0062c0 0x7f327c008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:40:43.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.954+0000 7f329affd640 1 --2- 172.21.15.12:0/835478496 >> v2:172.21.15.12:3300/0 conn(0x7f327c003750 0x7f327c005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:43.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.954+0000 7f329affd640 10 monclient(hunting): get_auth_request con 0x7f327c003750 auth_method 0 2024-09-11T01:40:43.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.954+0000 7f329affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:43.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.954+0000 7f329affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:43.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.954+0000 7f329affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:40:43.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.955+0000 7f329affd640 1 -- 172.21.15.12:0/835478496 >> v2:172.21.15.12:3300/0 conn(0x7f327c003750 msgr2=0x7f327c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:40:43.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.955+0000 7f329affd640 1 -- 172.21.15.12:0/835478496 >> v2:172.21.15.12:3300/0 conn(0x7f327c003750 msgr2=0x7f327c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:40:43.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.955+0000 7f329affd640 1 --2- 172.21.15.12:0/835478496 >> v2:172.21.15.12:3300/0 conn(0x7f327c003750 0x7f327c005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:40:43.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:43.955+0000 7f329affd640 1 --2- 172.21.15.12:0/835478496 >> v2:172.21.15.12:3300/0 conn(0x7f327c003750 0x7f327c005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:40:44.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:44.754+0000 7f3293fff640 1 -- 172.21.15.12:0/835478496 >> v2:172.21.15.71:3300/0 conn(0x7f327c0062c0 msgr2=0x7f327c008780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:40:44.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:44.754+0000 7f3293fff640 1 --2- 172.21.15.12:0/835478496 >> v2:172.21.15.71:3300/0 conn(0x7f327c0062c0 0x7f327c008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:40:44.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:44.757+0000 7f329affd640 1 --2- 172.21.15.12:0/835478496 >> v2:172.21.15.12:3300/0 conn(0x7f327c003750 0x7f327c005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:44.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:44.757+0000 7f329affd640 10 monclient(hunting): get_auth_request con 0x7f327c003750 auth_method 0 2024-09-11T01:40:44.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:44.757+0000 7f329affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:44.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:44.757+0000 7f329affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:44.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:44.757+0000 7f329affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:40:44.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:44.757+0000 7f329affd640 1 -- 172.21.15.12:0/835478496 >> v2:172.21.15.12:3300/0 conn(0x7f327c003750 msgr2=0x7f327c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:40:44.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:44.757+0000 7f329affd640 1 -- 172.21.15.12:0/835478496 >> v2:172.21.15.12:3300/0 conn(0x7f327c003750 msgr2=0x7f327c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:40:44.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:44.757+0000 7f329affd640 1 --2- 172.21.15.12:0/835478496 >> v2:172.21.15.12:3300/0 conn(0x7f327c003750 0x7f327c005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:40:44.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:44.757+0000 7f329affd640 1 --2- 172.21.15.12:0/835478496 >> v2:172.21.15.12:3300/0 conn(0x7f327c003750 0x7f327c005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:40:46.259 INFO:tasks.rados.rados.0.smithi071.stdout:1661: done (3 left) 2024-09-11T01:40:46.259 INFO:tasks.rados.rados.0.smithi071.stdout:1662: rollback oid 41 current snap is 181 2024-09-11T01:40:46.259 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 41 to 175 2024-09-11T01:40:46.259 INFO:tasks.rados.rados.0.smithi071.stdout:1659: finishing write tid 1 to smithi07134777-18 2024-09-11T01:40:46.259 INFO:tasks.rados.rados.0.smithi071.stdout:1658: finishing write tid 4 to smithi07134777-35 2024-09-11T01:40:46.259 INFO:tasks.rados.rados.0.smithi071.stdout:1658: oid 35 updating version 0 to 898 2024-09-11T01:40:46.259 INFO:tasks.rados.rados.0.smithi071.stdout:1658: oid 35 updating version 898 to 899 2024-09-11T01:40:46.260 INFO:tasks.rados.rados.0.smithi071.stdout:1658: oid 35 updating version 899 to 900 2024-09-11T01:40:46.260 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 900 (ObjNum 612 snap 181 seq_num 612) dirty exists 2024-09-11T01:40:46.260 INFO:tasks.rados.rados.0.smithi071.stdout:1658: left oid 35 (ObjNum 612 snap 181 seq_num 612) 2024-09-11T01:40:46.260 INFO:tasks.rados.rados.0.smithi071.stdout:1659: finishing write tid 2 to smithi07134777-18 2024-09-11T01:40:46.260 INFO:tasks.rados.rados.0.smithi071.stdout:1659: finishing write tid 3 to smithi07134777-18 2024-09-11T01:40:46.260 INFO:tasks.rados.rados.0.smithi071.stdout:1659: finishing write tid 4 to smithi07134777-18 2024-09-11T01:40:46.260 INFO:tasks.rados.rados.0.smithi071.stdout:1659: oid 18 updating version 0 to 641 2024-09-11T01:40:46.260 INFO:tasks.rados.rados.0.smithi071.stdout:1659: oid 18 updating version 641 to 643 2024-09-11T01:40:46.260 INFO:tasks.rados.rados.0.smithi071.stdout:1659: oid 18 version 643 is already newer than 642 2024-09-11T01:40:46.260 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 643 (ObjNum 613 snap 181 seq_num 613) dirty exists 2024-09-11T01:40:46.260 INFO:tasks.rados.rados.0.smithi071.stdout:1659: left oid 18 (ObjNum 613 snap 181 seq_num 613) 2024-09-11T01:40:46.260 INFO:tasks.rados.rados.0.smithi071.stdout:1660: finishing write tid 1 to smithi07134777-29 2024-09-11T01:40:46.261 INFO:tasks.rados.rados.0.smithi071.stdout:1660: finishing write tid 2 to smithi07134777-29 2024-09-11T01:40:46.261 INFO:tasks.rados.rados.0.smithi071.stdout:1660: finishing write tid 3 to smithi07134777-29 2024-09-11T01:40:46.261 INFO:tasks.rados.rados.0.smithi071.stdout:1660: finishing write tid 4 to smithi07134777-29 2024-09-11T01:40:46.261 INFO:tasks.rados.rados.0.smithi071.stdout:1660: oid 29 updating version 0 to 928 2024-09-11T01:40:46.261 INFO:tasks.rados.rados.0.smithi071.stdout:1660: oid 29 updating version 928 to 929 2024-09-11T01:40:46.261 INFO:tasks.rados.rados.0.smithi071.stdout:1660: oid 29 updating version 929 to 930 2024-09-11T01:40:46.261 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 930 (ObjNum 614 snap 181 seq_num 614) dirty exists 2024-09-11T01:40:46.261 INFO:tasks.rados.rados.0.smithi071.stdout:1660: left oid 29 (ObjNum 614 snap 181 seq_num 614) 2024-09-11T01:40:46.261 INFO:tasks.rados.rados.0.smithi071.stdout:1658: done (3 left) 2024-09-11T01:40:46.261 INFO:tasks.rados.rados.0.smithi071.stdout:1659: done (2 left) 2024-09-11T01:40:46.261 INFO:tasks.rados.rados.0.smithi071.stdout:1660: done (1 left) 2024-09-11T01:40:46.261 INFO:tasks.rados.rados.0.smithi071.stdout:1663: rollback oid 39 current snap is 181 2024-09-11T01:40:46.261 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 39 to 174 2024-09-11T01:40:46.261 INFO:tasks.rados.rados.0.smithi071.stdout:1664: snap_create 2024-09-11T01:40:46.262 INFO:tasks.rados.rados.0.smithi071.stdout:1662: finishing rollback tid 1 to smithi07134777-41 2024-09-11T01:40:46.262 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-11T01:40:46.263 INFO:tasks.rados.rados.0.smithi071.stdout:1663: finishing rollback tid 1 to smithi07134777-39 2024-09-11T01:40:46.264 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 897 (ObjNum 177 snap 0 seq_num 2370520672) dirty dne 2024-09-11T01:40:46.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f32997fa640 10 monclient(hunting): tick 2024-09-11T01:40:46.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f32997fa640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:40:46.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f32997fa640 1 monclient(hunting): continuing hunt 2024-09-11T01:40:46.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f32997fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:40:46.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f32997fa640 1 -- 172.21.15.12:0/835478496 >> v2:172.21.15.71:3300/0 conn(0x7f327c0062c0 msgr2=0x7f327c008780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:40:46.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f32997fa640 1 --2- 172.21.15.12:0/835478496 >> v2:172.21.15.71:3300/0 conn(0x7f327c0062c0 0x7f327c008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f32997fa640 1 -- 172.21.15.12:0/835478496 >> v2:172.21.15.12:3300/0 conn(0x7f327c003750 msgr2=0x7f327c005bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:40:46.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f32997fa640 1 --2- 172.21.15.12:0/835478496 >> v2:172.21.15.12:3300/0 conn(0x7f327c003750 0x7f327c005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f329b7fe640 1 -- 172.21.15.12:0/835478496 reap_dead start 2024-09-11T01:40:46.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f32997fa640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-11T01:40:46.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f32997fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f327c003bc0 addr v1:172.21.15.12:6789/0 2024-09-11T01:40:46.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f32997fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f327c007e90 addr v1:172.21.15.71:6789/0 2024-09-11T01:40:46.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f32997fa640 1 --2- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f327c00c250 0x7f327c00e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:46.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f32997fa640 10 monclient(hunting): picked mon.noname-b con 0x7f327c00c250 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:40:46.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f32997fa640 1 -- 172.21.15.12:0/835478496 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f329c09ba20 con 0x7f327c003bc0 2024-09-11T01:40:46.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f32997fa640 1 -- 172.21.15.12:0/835478496 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f329c0a6e30 con 0x7f327c007e90 2024-09-11T01:40:46.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f32997fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:46.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f32997fa640 1 -- 172.21.15.12:0/835478496 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f329c09c9c0 con 0x7f327c00c250 2024-09-11T01:40:46.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f32997fa640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:46.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f3293fff640 1 -- 172.21.15.12:0/835478496 >> v1:172.21.15.71:6789/0 conn(0x7f327c007e90 legacy=0x7f327c0082f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:40:46.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f329b7fe640 1 --2- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f327c00c250 0x7f327c00e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:46.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f329b7fe640 10 monclient(hunting): get_auth_request con 0x7f327c00c250 auth_method 0 2024-09-11T01:40:46.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f329b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:46.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f329b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:46.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f329b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:46.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f329b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:46.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f329b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:46.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f329b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:46.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f329b7fe640 10 monclient(hunting): handle_auth_done global_id 114257 payload 274 2024-09-11T01:40:46.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f329b7fe640 1 -- 172.21.15.12:0/835478496 >> v1:172.21.15.71:6789/0 conn(0x7f327c007e90 legacy=0x7f327c0082f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:40:46.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f329b7fe640 1 -- 172.21.15.12:0/835478496 >> v1:172.21.15.12:6789/0 conn(0x7f327c003bc0 legacy=0x7f327c006770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:40:46.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f329b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:40:46.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f329b7fe640 1 monclient: found mon.noname-b 2024-09-11T01:40:46.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f329b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:46.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f329b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:40:46.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.351+0000 7f329b7fe640 1 -- 172.21.15.12:0/835478496 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f327c010bf0 con 0x7f327c00c250 2024-09-11T01:40:46.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.354+0000 7f329b7fe640 1 -- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f327c00c250 msgr2=0x7f327c00e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:46.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.354+0000 7f329b7fe640 1 --2- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f327c00c250 0x7f327c00e640 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f329c0a17e0 tx=0x7f329402a650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82f38d9bfbe014f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:46.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.355+0000 7f329a7fc640 1 -- 172.21.15.12:0/835478496 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3294007350 con 0x7f327c00c250 2024-09-11T01:40:46.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.355+0000 7f329a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:46.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.355+0000 7f329a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:40:46.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.355+0000 7f329a7fc640 10 monclient: dump: 2024-09-11T01:40:46.361 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:46.361 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:46.361 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.362 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.362 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:46.362 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:46.362 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:46.362 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:46.362 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:46.362 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:46.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.355+0000 7f329a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:40:46.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.355+0000 7f329a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:40:46.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.355+0000 7f329a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:46.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.355+0000 7f329a7fc640 1 -- 172.21.15.12:0/835478496 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32940074f0 con 0x7f327c00c250 2024-09-11T01:40:46.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.355+0000 7f32a1272640 5 monclient: authenticate success, global_id 114257 2024-09-11T01:40:46.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.355+0000 7f32a1272640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:40:46.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.355+0000 7f329a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:46.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.355+0000 7f329a7fc640 1 -- 172.21.15.12:0/835478496 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32940078a0 con 0x7f327c00c250 2024-09-11T01:40:46.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.355+0000 7f329a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:46.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.355+0000 7f329a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:40:46.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.355+0000 7f329a7fc640 10 monclient: dump: 2024-09-11T01:40:46.363 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:46.363 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:46.363 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.364 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.364 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:46.364 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:46.364 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:46.364 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:46.364 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:46.364 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:46.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.355+0000 7f32a1272640 10 monclient: get_monmap_and_config success 2024-09-11T01:40:46.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.355+0000 7f32a1272640 4 set_mon_vals no callback set 2024-09-11T01:40:46.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.355+0000 7f32a1272640 10 monclient: shutdown 2024-09-11T01:40:46.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.355+0000 7f32a1272640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:46.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.355+0000 7f32a1272640 1 -- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f327c00c250 msgr2=0x7f327c00e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:46.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.355+0000 7f32a1272640 1 --2- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f327c00c250 0x7f327c00e640 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f329c0a17e0 tx=0x7f329402a650 comp rx=0 tx=0).stop 2024-09-11T01:40:46.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.356+0000 7f32a1272640 1 -- 172.21.15.12:0/835478496 shutdown_connections 2024-09-11T01:40:46.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.356+0000 7f32a1272640 1 --2- 172.21.15.12:0/835478496 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f327c00c250 0x7f327c00e640 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.356+0000 7f32a1272640 1 -- 172.21.15.12:0/835478496 >> 172.21.15.12:0/835478496 conn(0x7f329c0a7120 msgr2=0x7f329c0a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:46.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.356+0000 7f32a1272640 1 -- 172.21.15.12:0/835478496 shutdown_connections 2024-09-11T01:40:46.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.356+0000 7f32a1272640 1 -- 172.21.15.12:0/835478496 wait complete. 2024-09-11T01:40:46.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.356+0000 7f32a1272640 10 monclient: build_initial_monmap 2024-09-11T01:40:46.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.356+0000 7f32a1272640 10 monclient: monmap: 2024-09-11T01:40:46.366 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:46.366 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:40:46.366 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:40:46.366 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:40:46.366 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:46.366 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:46.366 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:40:46.366 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:40:46.366 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:40:46.366 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:46.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.357+0000 7f32a1272640 1 Processor -- start 2024-09-11T01:40:46.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.357+0000 7f32a1272640 1 -- start start 2024-09-11T01:40:46.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.357+0000 7f32a1272640 10 monclient: init 2024-09-11T01:40:46.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.357+0000 7f32a1272640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:46.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.357+0000 7f32a1272640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:40:46.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.357+0000 7f32a1272640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f327c003bc0 0x7f329c1cce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:46.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f32a1272640 10 monclient(hunting): picked mon.noname-b con 0x7f327c003bc0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:40:46.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f32a1272640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f327c007e90 0x7f329c137270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:46.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f32a1272640 10 monclient(hunting): picked mon.noname-c con 0x7f327c007e90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:40:46.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f32a1272640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f327c00c250 0x7f329c1377b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:46.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f32a1272640 10 monclient(hunting): picked mon.noname-a con 0x7f327c00c250 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:40:46.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f32a1272640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:46.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f32a1272640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f329c06f970 con 0x7f327c00c250 2024-09-11T01:40:46.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f32a1272640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:46.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f32a1272640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f329c06f7f0 con 0x7f327c003bc0 2024-09-11T01:40:46.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f32a1272640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:46.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f32a1272640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f329c138610 con 0x7f327c007e90 2024-09-11T01:40:46.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f32a1272640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:46.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f32a1272640 10 monclient(hunting): authenticate will time out at 1452.800081s 2024-09-11T01:40:46.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f329affd640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f327c003bc0 msgr2=0x7f329c1cce80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:40:46.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f329affd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f327c003bc0 0x7f329c1cce80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:40:46.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f3293fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f327c007e90 0x7f329c137270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:46.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f329b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f327c00c250 0x7f329c1377b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:46.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f3293fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f327c007e90 0x7f329c137270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:57408/0 (socket says 172.21.15.12:57408) 2024-09-11T01:40:46.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f3293fff640 1 -- 172.21.15.12:0/1290741954 learned_addr learned my addr 172.21.15.12:0/1290741954 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:40:46.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f3293fff640 10 monclient(hunting): get_auth_request con 0x7f327c007e90 auth_method 0 2024-09-11T01:40:46.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f3293fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:46.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f3293fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:46.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f3293fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:46.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f329b7fe640 10 monclient(hunting): get_auth_request con 0x7f327c00c250 auth_method 0 2024-09-11T01:40:46.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f329b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:46.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f329b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:46.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f329b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:46.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f3293fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:46.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f3293fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:46.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f3293fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:46.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f329b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:46.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f329b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:46.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f329b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:46.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f3293fff640 10 monclient(hunting): handle_auth_done global_id 114260 payload 824 2024-09-11T01:40:46.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f3293fff640 1 -- 172.21.15.12:0/1290741954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f327c003bc0 msgr2=0x7f329c1cce80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:40:46.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f3293fff640 1 --2- 172.21.15.12:0/1290741954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f327c003bc0 0x7f329c1cce80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f3293fff640 1 -- 172.21.15.12:0/1290741954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f327c00c250 msgr2=0x7f329c1377b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:46.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f3293fff640 1 --2- 172.21.15.12:0/1290741954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f327c00c250 0x7f329c1377b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f3293fff640 10 monclient: _finish_hunting 0 2024-09-11T01:40:46.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f3293fff640 1 monclient: found mon.noname-c 2024-09-11T01:40:46.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f3293fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:46.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f3293fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:40:46.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f3293fff640 1 -- 172.21.15.12:0/1290741954 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f329c127b70 con 0x7f327c007e90 2024-09-11T01:40:46.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.358+0000 7f329b7fe640 1 --2- 172.21.15.12:0/1290741954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f327c00c250 0x7f329c1377b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:40:46.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.360+0000 7f3293fff640 1 -- 172.21.15.12:0/1290741954 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f327c007e90 msgr2=0x7f329c137270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:46.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.360+0000 7f3293fff640 1 --2- 172.21.15.12:0/1290741954 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f327c007e90 0x7f329c137270 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f328400b980 tx=0x7f328400bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:46.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.361+0000 7f32937fe640 1 -- 172.21.15.12:0/1290741954 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328400cdc0 con 0x7f327c007e90 2024-09-11T01:40:46.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.361+0000 7f32937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:46.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.361+0000 7f32937fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:40:46.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.361+0000 7f32937fe640 10 monclient: dump: 2024-09-11T01:40:46.373 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:46.373 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:46.373 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.373 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.373 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:46.373 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:46.373 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:46.373 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:46.373 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:46.373 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:46.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.361+0000 7f32937fe640 10 monclient: _finish_auth 0 2024-09-11T01:40:46.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.361+0000 7f32937fe640 10 monclient: _check_auth_tickets 2024-09-11T01:40:46.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.361+0000 7f32937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:46.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.361+0000 7f32a1272640 5 monclient: authenticate success, global_id 114260 2024-09-11T01:40:46.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.361+0000 7f32937fe640 1 -- 172.21.15.12:0/1290741954 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3284010030 con 0x7f327c007e90 2024-09-11T01:40:46.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.361+0000 7f32937fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:46.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.361+0000 7f32937fe640 1 -- 172.21.15.12:0/1290741954 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3284015660 con 0x7f327c007e90 2024-09-11T01:40:46.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.361+0000 7f32a1272640 10 monclient: _renew_subs 2024-09-11T01:40:46.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.361+0000 7f32a1272640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:40:46.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.361+0000 7f32a1272640 1 -- 172.21.15.12:0/1290741954 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f329c126910 con 0x7f327c007e90 2024-09-11T01:40:46.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.361+0000 7f32937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:46.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.361+0000 7f32937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:40:46.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.361+0000 7f32937fe640 10 monclient: dump: 2024-09-11T01:40:46.374 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:46.374 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:46.374 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.374 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.375 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:46.375 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:46.375 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:46.375 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:46.375 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:46.375 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:46.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.361+0000 7f3299ffb640 4 set_mon_vals no callback set 2024-09-11T01:40:46.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.361+0000 7f32a1272640 10 monclient: _renew_subs 2024-09-11T01:40:46.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.361+0000 7f32a1272640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:40:46.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.361+0000 7f32a1272640 1 -- 172.21.15.12:0/1290741954 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f329c0a1810 con 0x7f327c007e90 2024-09-11T01:40:46.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.362+0000 7f32937fe640 1 -- 172.21.15.12:0/1290741954 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f32840143d0 con 0x7f327c007e90 2024-09-11T01:40:46.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.362+0000 7f32a1272640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:46.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.362+0000 7f32a1272640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:46.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.362+0000 7f32a1272640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:40:46.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.362+0000 7f32a1272640 1 -- 172.21.15.12:0/1290741954 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f329c128800 con 0x7f327c007e90 2024-09-11T01:40:46.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.362+0000 7f32937fe640 1 --2- 172.21.15.12:0/1290741954 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f327c03fe00 0x7f327c0422c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:46.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.362+0000 7f32937fe640 1 -- 172.21.15.12:0/1290741954 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(381..381 src has 285..381) ==== 4783+0+0 (secure 0 0 0) 0x7f3284052780 con 0x7f327c007e90 2024-09-11T01:40:46.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.363+0000 7f329affd640 1 --2- 172.21.15.12:0/1290741954 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f327c03fe00 0x7f327c0422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:46.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.363+0000 7f329affd640 10 monclient: get_auth_request con 0x7f327c03fe00 auth_method 0 2024-09-11T01:40:46.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.365+0000 7f329affd640 1 --2- 172.21.15.12:0/1290741954 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f327c03fe00 0x7f327c0422c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f328c006fd0 tx=0x7f328c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:46.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.367+0000 7f32937fe640 1 -- 172.21.15.12:0/1290741954 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3284051180 con 0x7f327c007e90 2024-09-11T01:40:46.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.367+0000 7f32937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:46.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.367+0000 7f32937fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:40:46.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.626+0000 7f32a1272640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:40:46.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.626+0000 7f32a1272640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:40:46.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.626+0000 7f32a1272640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:40:46.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.626+0000 7f32a1272640 1 -- 172.21.15.12:0/1290741954 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f329c1389f0 con 0x7f327c007e90 2024-09-11T01:40:46.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.626+0000 7f3293fff640 1 -- 172.21.15.12:0/1290741954 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f327c007e90 msgr2=0x7f329c137270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:40:46.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.626+0000 7f3293fff640 1 -- 172.21.15.12:0/1290741954 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f327c007e90 msgr2=0x7f329c137270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:40:46.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.626+0000 7f3293fff640 1 --2- 172.21.15.12:0/1290741954 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f327c007e90 0x7f329c137270 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f328400b980 tx=0x7f328400bd80 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:40:46.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.626+0000 7f3293fff640 1 --2- 172.21.15.12:0/1290741954 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f327c007e90 0x7f329c137270 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f328400b980 tx=0x7f328400bd80 comp rx=0 tx=0).stop 2024-09-11T01:40:46.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.626+0000 7f32937fe640 10 monclient: ms_handle_reset current mon [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:40:46.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.626+0000 7f32937fe640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:46.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.626+0000 7f32937fe640 1 -- 172.21.15.12:0/1290741954 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f327c007e90 msgr2=0x7f329c137270 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:40:46.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.626+0000 7f32937fe640 1 --2- 172.21.15.12:0/1290741954 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f327c007e90 0x7f329c137270 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f32937fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:40:46.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f32937fe640 1 --2- 172.21.15.12:0/1290741954 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f327c0462f0 0x7f327c0486e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:46.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f32937fe640 10 monclient(hunting): picked mon.c con 0x7f327c0462f0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:40:46.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f32937fe640 1 --2- 172.21.15.12:0/1290741954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f327c003bc0 0x7f327c04afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:46.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f32937fe640 10 monclient(hunting): picked mon.b con 0x7f327c003bc0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:40:46.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f32937fe640 1 --2- 172.21.15.12:0/1290741954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f327c00c250 0x7f327c04da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:46.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f32937fe640 10 monclient(hunting): picked mon.a con 0x7f327c00c250 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:40:46.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f32937fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:46.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f32937fe640 1 -- 172.21.15.12:0/1290741954 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f327c04e430 con 0x7f327c00c250 2024-09-11T01:40:46.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f32937fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:46.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f32937fe640 1 -- 172.21.15.12:0/1290741954 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f327c04e5b0 con 0x7f327c003bc0 2024-09-11T01:40:46.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f32937fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:46.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f32937fe640 1 -- 172.21.15.12:0/1290741954 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f327c04e730 con 0x7f327c0462f0 2024-09-11T01:40:46.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f32937fe640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:46.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f3293fff640 1 --2- 172.21.15.12:0/1290741954 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f327c0462f0 0x7f327c0486e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:46.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f329affd640 1 --2- 172.21.15.12:0/1290741954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f327c00c250 0x7f327c04da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:46.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f3293fff640 10 monclient(hunting): get_auth_request con 0x7f327c0462f0 auth_method 0 2024-09-11T01:40:46.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f329b7fe640 1 -- 172.21.15.12:0/1290741954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f327c003bc0 msgr2=0x7f327c04afe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:40:46.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f3293fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:46.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f3293fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:46.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f3293fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:40:46.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f329affd640 10 monclient(hunting): get_auth_request con 0x7f327c00c250 auth_method 0 2024-09-11T01:40:46.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f329affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:46.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f329affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:46.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f329affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:40:46.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f3293fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:46.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f3293fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:46.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f329b7fe640 1 --2- 172.21.15.12:0/1290741954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f327c003bc0 0x7f327c04afe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:40:46.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f3293fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:40:46.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f329affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:46.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f329affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:46.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f329affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:40:46.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f3293fff640 10 monclient(hunting): handle_auth_done global_id 114260 payload 293 2024-09-11T01:40:46.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f3293fff640 1 -- 172.21.15.12:0/1290741954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f327c003bc0 msgr2=0x7f327c04afe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:40:46.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f3293fff640 1 --2- 172.21.15.12:0/1290741954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f327c003bc0 0x7f327c04afe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f3293fff640 1 -- 172.21.15.12:0/1290741954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f327c00c250 msgr2=0x7f327c04da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:46.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f3293fff640 1 --2- 172.21.15.12:0/1290741954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f327c00c250 0x7f327c04da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f3293fff640 10 monclient: _finish_hunting 0 2024-09-11T01:40:46.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f3293fff640 1 monclient: found mon.c 2024-09-11T01:40:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f3293fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f3293fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:40:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f3293fff640 1 -- 172.21.15.12:0/1290741954 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) -- 0x7f327c04e900 con 0x7f327c0462f0 2024-09-11T01:40:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f3293fff640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:40:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f3293fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:40:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f3293fff640 1 -- 172.21.15.12:0/1290741954 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f329c1389f0 con 0x7f327c0462f0 2024-09-11T01:40:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f3293fff640 10 monclient: _finish_auth 0 2024-09-11T01:40:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f3293fff640 10 monclient: _check_auth_tickets 2024-09-11T01:40:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f3293fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.627+0000 7f329affd640 1 --2- 172.21.15.12:0/1290741954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f327c00c250 0x7f327c04da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:40:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.629+0000 7f3293fff640 1 -- 172.21.15.12:0/1290741954 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f327c0462f0 msgr2=0x7f327c0486e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.629+0000 7f3293fff640 1 --2- 172.21.15.12:0/1290741954 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f327c0462f0 0x7f327c0486e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3284051070 tx=0x7f3284009480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:46.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.629+0000 7f32937fe640 1 -- 172.21.15.12:0/1290741954 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3284011a60 con 0x7f327c0462f0 2024-09-11T01:40:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.629+0000 7f32937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.629+0000 7f32937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:40:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.629+0000 7f32937fe640 10 monclient: dump: 2024-09-11T01:40:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:46.637 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:46.638 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:46.638 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:46.638 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:46.638 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:46.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.629+0000 7f32937fe640 1 -- 172.21.15.12:0/1290741954 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32840133f0 con 0x7f327c0462f0 2024-09-11T01:40:46.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.630+0000 7f32937fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:46.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.630+0000 7f3299ffb640 4 set_mon_vals no callback set 2024-09-11T01:40:46.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.630+0000 7f32937fe640 1 -- 172.21.15.12:0/1290741954 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f3284011c90 con 0x7f327c0462f0 2024-09-11T01:40:46.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.630+0000 7f32937fe640 1 -- 172.21.15.12:0/1290741954 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f3284046070 con 0x7f327c0462f0 2024-09-11T01:40:46.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.630+0000 7f32937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:40:46.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.630+0000 7f32937fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:40:46.639 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":108,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:40:46.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.632+0000 7f32a1272640 1 -- 172.21.15.12:0/1290741954 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f327c03fe00 msgr2=0x7f327c0422c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:46.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.632+0000 7f32a1272640 1 --2- 172.21.15.12:0/1290741954 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f327c03fe00 0x7f327c0422c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f328c006fd0 tx=0x7f328c008040 comp rx=0 tx=0).stop 2024-09-11T01:40:46.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.632+0000 7f32a1272640 10 monclient: shutdown 2024-09-11T01:40:46.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.632+0000 7f32a1272640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:46.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.632+0000 7f32a1272640 1 -- 172.21.15.12:0/1290741954 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f327c0462f0 msgr2=0x7f327c0486e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:46.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.632+0000 7f32a1272640 1 --2- 172.21.15.12:0/1290741954 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f327c0462f0 0x7f327c0486e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3284051070 tx=0x7f3284009480 comp rx=0 tx=0).stop 2024-09-11T01:40:46.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.632+0000 7f32a1272640 1 -- 172.21.15.12:0/1290741954 shutdown_connections 2024-09-11T01:40:46.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.632+0000 7f32a1272640 1 --2- 172.21.15.12:0/1290741954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f327c003bc0 0x7f327c04afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.632+0000 7f32a1272640 1 --2- 172.21.15.12:0/1290741954 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f327c0462f0 0x7f327c0486e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.632+0000 7f32a1272640 1 --2- 172.21.15.12:0/1290741954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f327c00c250 0x7f327c04da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.632+0000 7f32a1272640 1 --2- 172.21.15.12:0/1290741954 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f327c03fe00 0x7f327c0422c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.632+0000 7f32a1272640 1 -- 172.21.15.12:0/1290741954 >> 172.21.15.12:0/1290741954 conn(0x7f329c0a7120 msgr2=0x7f329c0a4000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:46.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.633+0000 7f32a1272640 1 -- 172.21.15.12:0/1290741954 shutdown_connections 2024-09-11T01:40:46.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.633+0000 7f32a1272640 1 -- 172.21.15.12:0/1290741954 wait complete. 2024-09-11T01:40:46.645 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:40:46.645 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-11T01:40:46.645 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:40:46.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.745+0000 7fc31e110640 10 monclient: get_monmap_and_config 2024-09-11T01:40:46.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.745+0000 7fc31e110640 10 monclient: build_initial_monmap 2024-09-11T01:40:46.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.745+0000 7fc31e110640 10 monclient: monmap: 2024-09-11T01:40:46.747 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:46.748 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:46.748 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:40:46.746697+0000 2024-09-11T01:40:46.748 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:40:46.746697+0000 2024-09-11T01:40:46.748 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:46.748 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:46.748 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:40:46.748 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:40:46.748 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:40:46.748 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:40:46.748 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:40:46.748 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:46.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.746+0000 7fc31e110640 1 Processor -- start 2024-09-11T01:40:46.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.746+0000 7fc31e110640 1 -- start start 2024-09-11T01:40:46.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.746+0000 7fc31e110640 10 monclient: init 2024-09-11T01:40:46.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc31e110640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:46.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc31e110640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-11T01:40:46.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc31e110640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc318134ad0 0x7fc318134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:46.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc31e110640 10 monclient(hunting): picked mon.noname-b con 0x7fc318134ad0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:40:46.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc31e110640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc3180a2200 addr v1:172.21.15.71:6789/0 2024-09-11T01:40:46.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc31e110640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc3181353f0 addr v1:172.21.15.12:6789/0 2024-09-11T01:40:46.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc31e110640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc318134870 con 0x7fc3181353f0 2024-09-11T01:40:46.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc31e110640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc31809ed30 con 0x7fc3180a2200 2024-09-11T01:40:46.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc31e110640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:46.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc31e110640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc31809f5d0 con 0x7fc318134ad0 2024-09-11T01:40:46.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc31e110640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:46.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc31e110640 10 monclient(hunting): authenticate will time out at 1453.189476s 2024-09-11T01:40:46.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc3177fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc318134ad0 0x7fc318134eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:46.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc3177fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc318134ad0 0x7fc318134eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:57422/0 (socket says 172.21.15.12:57422) 2024-09-11T01:40:46.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc3177fe640 1 -- 172.21.15.12:0/2190104817 learned_addr learned my addr 172.21.15.12:0/2190104817 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:40:46.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc3177fe640 10 monclient(hunting): get_auth_request con 0x7fc318134ad0 auth_method 0 2024-09-11T01:40:46.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc3177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:46.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc3177fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:46.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc3177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:46.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc316ffd640 1 -- 172.21.15.12:0/2190104817 >> v1:172.21.15.71:6789/0 conn(0x7fc3180a2200 legacy=0x7fc318133a50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:40:46.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc3177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:46.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc3177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:46.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc3177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:46.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc3177fe640 10 monclient(hunting): handle_auth_done global_id 114263 payload 274 2024-09-11T01:40:46.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc3177fe640 1 -- 172.21.15.12:0/2190104817 >> v1:172.21.15.71:6789/0 conn(0x7fc3180a2200 legacy=0x7fc318133a50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:40:46.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc3177fe640 1 -- 172.21.15.12:0/2190104817 >> v1:172.21.15.12:6789/0 conn(0x7fc3181353f0 legacy=0x7fc318134160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:40:46.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc3177fe640 10 monclient: _finish_hunting 0 2024-09-11T01:40:46.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc3177fe640 1 monclient: found mon.noname-b 2024-09-11T01:40:46.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc3177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:46.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc3177fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:40:46.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.747+0000 7fc3177fe640 1 -- 172.21.15.12:0/2190104817 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc31809f010 con 0x7fc318134ad0 2024-09-11T01:40:46.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.750+0000 7fc3177fe640 1 -- 172.21.15.12:0/2190104817 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc318134ad0 msgr2=0x7fc318134eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:46.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.750+0000 7fc3177fe640 1 --2- 172.21.15.12:0/2190104817 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc318134ad0 0x7fc318134eb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc3080098a0 tx=0x7fc30802a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7de017698ec3cc01 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:46.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.750+0000 7fc3167fc640 1 -- 172.21.15.12:0/2190104817 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc308037070 con 0x7fc318134ad0 2024-09-11T01:40:46.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.750+0000 7fc3167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:46.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.750+0000 7fc3167fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:40:46.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.750+0000 7fc3167fc640 10 monclient: dump: 2024-09-11T01:40:46.761 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:46.762 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:46.762 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.762 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.762 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:46.762 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:46.762 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:46.762 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:46.762 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:46.762 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:46.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.750+0000 7fc3167fc640 10 monclient: _finish_auth 0 2024-09-11T01:40:46.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.750+0000 7fc3167fc640 10 monclient: _check_auth_tickets 2024-09-11T01:40:46.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.750+0000 7fc3167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:46.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.750+0000 7fc3167fc640 1 -- 172.21.15.12:0/2190104817 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc308032640 con 0x7fc318134ad0 2024-09-11T01:40:46.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.750+0000 7fc31e110640 5 monclient: authenticate success, global_id 114263 2024-09-11T01:40:46.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.750+0000 7fc31e110640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:40:46.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.750+0000 7fc3167fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:46.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.750+0000 7fc31e110640 10 monclient: get_monmap_and_config success 2024-09-11T01:40:46.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.750+0000 7fc3167fc640 1 -- 172.21.15.12:0/2190104817 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc308032a10 con 0x7fc318134ad0 2024-09-11T01:40:46.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.750+0000 7fc3167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:46.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.750+0000 7fc31e110640 4 set_mon_vals no callback set 2024-09-11T01:40:46.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.750+0000 7fc3167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:40:46.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.750+0000 7fc3167fc640 10 monclient: dump: 2024-09-11T01:40:46.763 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:46.763 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:46.764 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.764 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.764 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:46.764 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:46.764 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:46.764 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:46.764 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:46.764 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:46.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.751+0000 7fc31e110640 10 monclient: shutdown 2024-09-11T01:40:46.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.751+0000 7fc31e110640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:46.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.751+0000 7fc31e110640 1 -- 172.21.15.12:0/2190104817 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc318134ad0 msgr2=0x7fc318134eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:46.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.751+0000 7fc31e110640 1 --2- 172.21.15.12:0/2190104817 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc318134ad0 0x7fc318134eb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc3080098a0 tx=0x7fc30802a080 comp rx=0 tx=0).stop 2024-09-11T01:40:46.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.751+0000 7fc31e110640 1 -- 172.21.15.12:0/2190104817 shutdown_connections 2024-09-11T01:40:46.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.751+0000 7fc31e110640 1 --2- 172.21.15.12:0/2190104817 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc318134ad0 0x7fc318134eb0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.751+0000 7fc31e110640 1 -- 172.21.15.12:0/2190104817 >> 172.21.15.12:0/2190104817 conn(0x7fc3180a2f70 msgr2=0x7fc3180a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:46.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.751+0000 7fc31e110640 1 -- 172.21.15.12:0/2190104817 shutdown_connections 2024-09-11T01:40:46.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.751+0000 7fc31e110640 1 -- 172.21.15.12:0/2190104817 wait complete. 2024-09-11T01:40:46.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.751+0000 7fc31e110640 10 monclient: build_initial_monmap 2024-09-11T01:40:46.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.751+0000 7fc31e110640 10 monclient: monmap: 2024-09-11T01:40:46.766 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:46.766 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:40:46.766 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:40:46.766 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:40:46.766 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:46.766 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:46.766 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:40:46.766 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:40:46.766 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:40:46.766 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:46.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.752+0000 7fc31e110640 1 Processor -- start 2024-09-11T01:40:46.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.752+0000 7fc31e110640 1 -- start start 2024-09-11T01:40:46.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.752+0000 7fc31e110640 10 monclient: init 2024-09-11T01:40:46.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.752+0000 7fc31e110640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:46.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.752+0000 7fc31e110640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:40:46.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.752+0000 7fc31e110640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc3180a2200 0x7fc3181330c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:46.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.752+0000 7fc31e110640 10 monclient(hunting): picked mon.noname-c con 0x7fc3180a2200 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:40:46.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.752+0000 7fc31e110640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc318134ad0 0x7fc318133600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:46.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.752+0000 7fc31e110640 10 monclient(hunting): picked mon.noname-a con 0x7fc318134ad0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:40:46.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.752+0000 7fc31e110640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc3181353f0 0x7fc31813e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:46.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.752+0000 7fc31e110640 10 monclient(hunting): picked mon.noname-b con 0x7fc3181353f0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:40:46.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.752+0000 7fc31e110640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:46.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.752+0000 7fc31e110640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc31809ce20 con 0x7fc318134ad0 2024-09-11T01:40:46.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.752+0000 7fc31e110640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:46.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.752+0000 7fc31e110640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc31809cca0 con 0x7fc3181353f0 2024-09-11T01:40:46.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.752+0000 7fc31e110640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:46.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.752+0000 7fc31e110640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc31809ffd0 con 0x7fc3180a2200 2024-09-11T01:40:46.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.752+0000 7fc31e110640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:46.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc31e110640 10 monclient(hunting): authenticate will time out at 1453.195009s 2024-09-11T01:40:46.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc3177fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc3180a2200 0x7fc3181330c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:46.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc3177fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc3180a2200 0x7fc3181330c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:57438/0 (socket says 172.21.15.12:57438) 2024-09-11T01:40:46.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc3177fe640 1 -- 172.21.15.12:0/1194214491 learned_addr learned my addr 172.21.15.12:0/1194214491 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:40:46.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc3177fe640 10 monclient(hunting): get_auth_request con 0x7fc3180a2200 auth_method 0 2024-09-11T01:40:46.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc3177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:46.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc3177fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:46.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc3177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:46.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc316ffd640 1 --2- 172.21.15.12:0/1194214491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc318134ad0 0x7fc318133600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:46.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc316ffd640 10 monclient(hunting): get_auth_request con 0x7fc318134ad0 auth_method 0 2024-09-11T01:40:46.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc316ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:46.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc316ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:46.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc316ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:46.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc3177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:46.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc3177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:46.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc317fff640 1 -- 172.21.15.12:0/1194214491 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc3181353f0 msgr2=0x7fc31813e0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:40:46.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc3177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:46.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc317fff640 1 --2- 172.21.15.12:0/1194214491 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc3181353f0 0x7fc31813e0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:40:46.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc316ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:46.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc316ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:46.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc316ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:46.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc3177fe640 10 monclient(hunting): handle_auth_done global_id 114266 payload 824 2024-09-11T01:40:46.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc3177fe640 1 -- 172.21.15.12:0/1194214491 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc3181353f0 msgr2=0x7fc31813e0c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:40:46.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc3177fe640 1 --2- 172.21.15.12:0/1194214491 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc3181353f0 0x7fc31813e0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc3177fe640 1 -- 172.21.15.12:0/1194214491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc318134ad0 msgr2=0x7fc318133600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:46.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc3177fe640 1 --2- 172.21.15.12:0/1194214491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc318134ad0 0x7fc318133600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc3177fe640 10 monclient: _finish_hunting 0 2024-09-11T01:40:46.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc3177fe640 1 monclient: found mon.noname-c 2024-09-11T01:40:46.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc3177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:46.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc3177fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:40:46.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc3177fe640 1 -- 172.21.15.12:0/1194214491 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc31811dae0 con 0x7fc3180a2200 2024-09-11T01:40:46.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.753+0000 7fc316ffd640 1 --2- 172.21.15.12:0/1194214491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc318134ad0 0x7fc318133600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:40:46.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.755+0000 7fc3177fe640 1 -- 172.21.15.12:0/1194214491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc3180a2200 msgr2=0x7fc3181330c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:46.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.755+0000 7fc3177fe640 1 --2- 172.21.15.12:0/1194214491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc3180a2200 0x7fc3181330c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc3080097f0 tx=0x7fc3080046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:46.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.755+0000 7fc2fffff640 1 -- 172.21.15.12:0/1194214491 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc308041070 con 0x7fc3180a2200 2024-09-11T01:40:46.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.755+0000 7fc2fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:46.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.755+0000 7fc2fffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:40:46.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.755+0000 7fc2fffff640 10 monclient: dump: 2024-09-11T01:40:46.770 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:46.771 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:46.771 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.771 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.771 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:46.771 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:46.771 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:46.771 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:46.771 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:46.771 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:46.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.755+0000 7fc2fffff640 10 monclient: _finish_auth 0 2024-09-11T01:40:46.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.755+0000 7fc2fffff640 10 monclient: _check_auth_tickets 2024-09-11T01:40:46.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.755+0000 7fc2fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:46.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.755+0000 7fc2fffff640 1 -- 172.21.15.12:0/1194214491 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc30803c880 con 0x7fc3180a2200 2024-09-11T01:40:46.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.755+0000 7fc31e110640 5 monclient: authenticate success, global_id 114266 2024-09-11T01:40:46.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.755+0000 7fc31e110640 10 monclient: _renew_subs 2024-09-11T01:40:46.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.755+0000 7fc31e110640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:40:46.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.755+0000 7fc31e110640 1 -- 172.21.15.12:0/1194214491 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc31811ddc0 con 0x7fc3180a2200 2024-09-11T01:40:46.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.755+0000 7fc2fffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:46.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.755+0000 7fc2fffff640 1 -- 172.21.15.12:0/1194214491 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc308037040 con 0x7fc3180a2200 2024-09-11T01:40:46.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.755+0000 7fc2fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:46.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.755+0000 7fc2fffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:40:46.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.755+0000 7fc2fffff640 10 monclient: dump: 2024-09-11T01:40:46.772 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:46.772 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:46.772 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.772 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.772 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:46.772 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:46.773 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:46.773 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:46.773 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:46.773 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:46.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.755+0000 7fc31e110640 10 monclient: _renew_subs 2024-09-11T01:40:46.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.755+0000 7fc31e110640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:40:46.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.755+0000 7fc31e110640 1 -- 172.21.15.12:0/1194214491 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc31811d800 con 0x7fc3180a2200 2024-09-11T01:40:46.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.755+0000 7fc315ffb640 4 set_mon_vals no callback set 2024-09-11T01:40:46.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.756+0000 7fc31e110640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:46.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.756+0000 7fc31e110640 1 --2- 172.21.15.12:0/1194214491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc31813edc0 0x7fc31802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:46.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.756+0000 7fc31e110640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:46.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.756+0000 7fc31e110640 1 -- 172.21.15.12:0/1194214491 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc318134eb0 con 0x7fc31813edc0 2024-09-11T01:40:46.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.756+0000 7fc2fffff640 1 -- 172.21.15.12:0/1194214491 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fc30803cc50 con 0x7fc3180a2200 2024-09-11T01:40:46.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.756+0000 7fc316ffd640 1 --2- 172.21.15.12:0/1194214491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc31813edc0 0x7fc31802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.756+0000 7fc316ffd640 10 monclient: get_auth_request con 0x7fc31813edc0 auth_method 0 2024-09-11T01:40:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.756+0000 7fc316ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.756+0000 7fc316ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.756+0000 7fc316ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.757+0000 7fc316ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.757+0000 7fc316ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.757+0000 7fc316ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.757+0000 7fc316ffd640 10 monclient(hunting): handle_auth_done global_id 94428 payload 824 2024-09-11T01:40:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.757+0000 7fc2fffff640 1 --2- 172.21.15.12:0/1194214491 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc30403e1c0 0x7fc304040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.757+0000 7fc316ffd640 1 -- 172.21.15.12:0/1194214491 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc318072db0 con 0x7fc31813edc0 2024-09-11T01:40:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.757+0000 7fc2fffff640 1 -- 172.21.15.12:0/1194214491 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(381..381 src has 285..381) ==== 4783+0+0 (secure 0 0 0) 0x7fc308073840 con 0x7fc3180a2200 2024-09-11T01:40:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.757+0000 7fc317fff640 1 --2- 172.21.15.12:0/1194214491 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc30403e1c0 0x7fc304040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.757+0000 7fc317fff640 10 monclient: get_auth_request con 0x7fc30403e1c0 auth_method 0 2024-09-11T01:40:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.759+0000 7fc316ffd640 1 -- 172.21.15.12:0/1194214491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc31813edc0 msgr2=0x7fc31802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.759+0000 7fc316ffd640 1 --2- 172.21.15.12:0/1194214491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc31813edc0 0x7fc31802d440 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc31811cb20 tx=0x7fc30c0057a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.759+0000 7fc2fffff640 1 -- 172.21.15.12:0/1194214491 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c016070 con 0x7fc31813edc0 2024-09-11T01:40:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.759+0000 7fc2fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.759+0000 7fc2fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:40:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.759+0000 7fc2fffff640 10 monclient: dump: 2024-09-11T01:40:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:46.776 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:46.776 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:46.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.759+0000 7fc2fffff640 1 -- 172.21.15.12:0/1194214491 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc30c00ed30 con 0x7fc31813edc0 2024-09-11T01:40:46.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.759+0000 7fc2fffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:46.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.759+0000 7fc2fffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:40:46.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.759+0000 7fc2fffff640 1 -- 172.21.15.12:0/1194214491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc31813edc0 msgr2=0x7fc31802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:46.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.759+0000 7fc2fffff640 1 --2- 172.21.15.12:0/1194214491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc31813edc0 0x7fc31802d440 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc31811cb20 tx=0x7fc30c0057a0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.759+0000 7fc2fffff640 1 -- 172.21.15.12:0/1194214491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc31813edc0 msgr2=0x7fc31802d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:40:46.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.759+0000 7fc2fffff640 1 --2- 172.21.15.12:0/1194214491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc31813edc0 0x7fc31802d440 secure :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc31811cb20 tx=0x7fc30c0057a0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.759+0000 7fc317fff640 1 --2- 172.21.15.12:0/1194214491 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc30403e1c0 0x7fc304040680 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc310003aa0 tx=0x7fc310007d10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:46.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.766+0000 7fc31e110640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:40:46.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.766+0000 7fc31e110640 1 --2- 172.21.15.12:0/1194214491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc31813f1a0 0x7fc31802d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:46.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.766+0000 7fc31e110640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:46.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.766+0000 7fc31e110640 1 -- 172.21.15.12:0/1194214491 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc31806d480 con 0x7fc31813f1a0 2024-09-11T01:40:46.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.766+0000 7fc316ffd640 1 --2- 172.21.15.12:0/1194214491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc31813f1a0 0x7fc31802d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:46.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.766+0000 7fc316ffd640 10 monclient: get_auth_request con 0x7fc31813f1a0 auth_method 0 2024-09-11T01:40:46.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.766+0000 7fc316ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:46.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.766+0000 7fc316ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:46.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.766+0000 7fc316ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:46.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.766+0000 7fc316ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:46.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.766+0000 7fc316ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:46.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.766+0000 7fc316ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:46.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.766+0000 7fc316ffd640 10 monclient(hunting): handle_auth_done global_id 94431 payload 824 2024-09-11T01:40:46.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.766+0000 7fc316ffd640 1 -- 172.21.15.12:0/1194214491 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc31813fed0 con 0x7fc31813f1a0 2024-09-11T01:40:46.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.768+0000 7fc316ffd640 1 -- 172.21.15.12:0/1194214491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc31813f1a0 msgr2=0x7fc31802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:46.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.768+0000 7fc316ffd640 1 --2- 172.21.15.12:0/1194214491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc31813f1a0 0x7fc31802d980 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc30c018000 tx=0x7fc30c0059c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:46.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.768+0000 7fc2fffff640 1 -- 172.21.15.12:0/1194214491 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c016040 con 0x7fc31813f1a0 2024-09-11T01:40:46.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.768+0000 7fc2fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:46.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.768+0000 7fc2fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:40:46.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.768+0000 7fc2fffff640 10 monclient: dump: 2024-09-11T01:40:46.780 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:46.780 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:46.780 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.780 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.780 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:46.780 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:46.780 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:46.780 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:46.780 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:46.780 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:46.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.769+0000 7fc2fffff640 1 -- 172.21.15.12:0/1194214491 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4700 (secure 0 0 0) 0x7fc30c005c40 con 0x7fc31813f1a0 2024-09-11T01:40:46.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.769+0000 7fc2fffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:40:46.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.769+0000 7fc2fffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:40:46.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.769+0000 7fc2fffff640 1 -- 172.21.15.12:0/1194214491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc31813f1a0 msgr2=0x7fc31802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:46.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.769+0000 7fc2fffff640 1 --2- 172.21.15.12:0/1194214491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc31813f1a0 0x7fc31802d980 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc30c018000 tx=0x7fc30c0059c0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.769+0000 7fc2fffff640 1 -- 172.21.15.12:0/1194214491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc31813f1a0 msgr2=0x7fc31802d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:40:46.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.769+0000 7fc2fffff640 1 --2- 172.21.15.12:0/1194214491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc31813f1a0 0x7fc31802d980 secure :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc30c018000 tx=0x7fc30c0059c0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.781 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:40:46.781 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:40:46.781 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:40:46.781 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:40:46.781 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 108, 2024-09-11T01:40:46.782 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 192864, 2024-09-11T01:40:46.782 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:40:46.782 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:40:46.782 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:40:46.782 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:46.782 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:40:46.782 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:40:46.782 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:40:46.782 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:40:46.782 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:40:46.782 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:40:46.782 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:40:46.782 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:40:46.782 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:40:46.783 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:40:46.783 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:40:46.783 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:40:46.783 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:40:46.783 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:40:46.783 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:40:46.783 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:46.783 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:40:46.783 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:40:46.783 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:40:46.783 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:40:46.783 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:40:46.783 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:40:46.783 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:40:46.784 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:40:46.784 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:40:46.784 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:40:46.784 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:40:46.784 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:40:46.784 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:40:46.784 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:46.784 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:46.784 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:40:46.784 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:40:46.784 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:40:46.784 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:40:46.784 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:40:46.784 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:40:46.785 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:40:46.785 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:40:46.785 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:40:46.785 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:40:46.785 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:40:46.785 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:40:46.785 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:40:46.785 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:40:46.785 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:40:46.785 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:40:46.785 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:40:46.785 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:40:46.785 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:40:46.786 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:40:46.786 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:40:46.786 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:40:46.786 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:40:46.786 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:40:46.786 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:40:46.786 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:40:46.786 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:40:46.786 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:40:46.786 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:46.786 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:40:46.786 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:46.786 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:40:46.786 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:46.787 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:40:46.787 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:40:46.787 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:40:46.787 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:40:46.787 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:46.787 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:40:46.787 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:40:46.787 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:46.787 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:46.787 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:46.787 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:40:46.787 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:40:46.787 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:46.788 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:46.788 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:46.788 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:46.788 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:40:46.788 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:40:46.788 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:40:46.788 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:40:46.788 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:40:46.788 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:46.788 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:46.788 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:40:46.788 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:40:46.788 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:40:46.788 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:40:46.789 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:46.789 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:40:46.789 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:40:46.789 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:46.789 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:46.789 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:46.789 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:40:46.789 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:40:46.789 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:46.789 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:46.789 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:46.789 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:46.789 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:40:46.790 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:40:46.790 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:40:46.790 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:40:46.790 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:40:46.790 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:46.790 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:46.790 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:40:46.790 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:40:46.790 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:40:46.790 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:40:46.790 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:46.790 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:40:46.790 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:40:46.790 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:46.791 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:46.791 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:46.791 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:40:46.791 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:40:46.791 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:46.791 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:46.791 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:46.791 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:46.791 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:40:46.791 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:40:46.791 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:40:46.791 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:40:46.791 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:40:46.791 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:46.792 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:46.792 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:46.792 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:40:46.792 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:40:46.792 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:46.792 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:40:46.792 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:40:46.792 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:40:46.792 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:46.792 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:46.792 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:40:46.792 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:46.792 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:40:46.793 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:40:46.793 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:40:46.793 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:46.793 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:46.793 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:40:46.793 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:46.793 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:40:46.793 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:40:46.793 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:40:46.793 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:46.793 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:46.793 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:46.793 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:40:46.793 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:40:46.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.769+0000 7fc31e110640 1 -- 172.21.15.12:0/1194214491 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc30403e1c0 msgr2=0x7fc304040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:46.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.769+0000 7fc31e110640 1 --2- 172.21.15.12:0/1194214491 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc30403e1c0 0x7fc304040680 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc310003aa0 tx=0x7fc310007d10 comp rx=0 tx=0).stop 2024-09-11T01:40:46.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.769+0000 7fc31e110640 10 monclient: shutdown 2024-09-11T01:40:46.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.769+0000 7fc31e110640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:46.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.769+0000 7fc31e110640 1 -- 172.21.15.12:0/1194214491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc3180a2200 msgr2=0x7fc3181330c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:46.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.769+0000 7fc31e110640 1 --2- 172.21.15.12:0/1194214491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc3180a2200 0x7fc3181330c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc3080097f0 tx=0x7fc3080046c0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.769+0000 7fc317fff640 1 -- 172.21.15.12:0/1194214491 reap_dead start 2024-09-11T01:40:46.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.769+0000 7fc31e110640 1 -- 172.21.15.12:0/1194214491 shutdown_connections 2024-09-11T01:40:46.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.769+0000 7fc31e110640 1 --2- 172.21.15.12:0/1194214491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc3180a2200 0x7fc3181330c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.769+0000 7fc31e110640 1 -- 172.21.15.12:0/1194214491 >> 172.21.15.12:0/1194214491 conn(0x7fc3180a2f70 msgr2=0x7fc3180a4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:46.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.769+0000 7fc31e110640 1 -- 172.21.15.12:0/1194214491 shutdown_connections 2024-09-11T01:40:46.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.769+0000 7fc31e110640 1 -- 172.21.15.12:0/1194214491 wait complete. 2024-09-11T01:40:46.796 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:40:46.796 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T01:40:46.796 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:40:46.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.897+0000 7fe3c27a0640 10 monclient: get_monmap_and_config 2024-09-11T01:40:46.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.897+0000 7fe3c27a0640 10 monclient: build_initial_monmap 2024-09-11T01:40:46.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.897+0000 7fe3c27a0640 10 monclient: monmap: 2024-09-11T01:40:46.899 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:46.899 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:46.899 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:40:46.898263+0000 2024-09-11T01:40:46.899 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:40:46.898263+0000 2024-09-11T01:40:46.899 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:46.899 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:46.899 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:40:46.900 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:40:46.900 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:40:46.900 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:40:46.900 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:40:46.900 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:46.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.898+0000 7fe3c27a0640 1 Processor -- start 2024-09-11T01:40:46.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.898+0000 7fe3c27a0640 1 -- start start 2024-09-11T01:40:46.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.898+0000 7fe3c27a0640 10 monclient: init 2024-09-11T01:40:46.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.898+0000 7fe3c27a0640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:46.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.898+0000 7fe3c27a0640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-11T01:40:46.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.898+0000 7fe3c27a0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe3bc134ad0 addr v1:172.21.15.71:6789/0 2024-09-11T01:40:46.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.898+0000 7fe3c27a0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe3bc0a2200 addr v1:172.21.15.12:6789/0 2024-09-11T01:40:46.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.898+0000 7fe3c27a0640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe3bc1355c0 0x7fe3bc13d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:46.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.898+0000 7fe3c27a0640 10 monclient(hunting): picked mon.noname-a con 0x7fe3bc1355c0 addr v2:172.21.15.12:3300/0 2024-09-11T01:40:46.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.898+0000 7fe3c27a0640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3bc09ed30 con 0x7fe3bc0a2200 2024-09-11T01:40:46.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.898+0000 7fe3c27a0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3bc09f010 con 0x7fe3bc134ad0 2024-09-11T01:40:46.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.898+0000 7fe3c27a0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:46.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.898+0000 7fe3c27a0640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fe3bc09f5d0 con 0x7fe3bc1355c0 2024-09-11T01:40:46.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.898+0000 7fe3c27a0640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:46.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.898+0000 7fe3c27a0640 10 monclient(hunting): authenticate will time out at 1453.340820s 2024-09-11T01:40:46.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.898+0000 7fe3bbfff640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7fe3bc134ad0 legacy=0x7fe3bc134eb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:40:46.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.898+0000 7fe3c0d3d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe3bc1355c0 0x7fe3bc13d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:46.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.898+0000 7fe3bb7fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe3bc0a2200 0x7fe3bc13a000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:45374/0 (socket says 172.21.15.12:45374) 2024-09-11T01:40:46.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.898+0000 7fe3bb7fe640 1 -- 172.21.15.12:0/3151697363 learned_addr learned my addr 172.21.15.12:0/3151697363 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:40:46.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.898+0000 7fe3c0d3d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe3bc1355c0 0x7fe3bc13d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55858/0 (socket says 172.21.15.12:55858) 2024-09-11T01:40:46.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.898+0000 7fe3c0d3d640 10 monclient(hunting): get_auth_request con 0x7fe3bc1355c0 auth_method 0 2024-09-11T01:40:46.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.898+0000 7fe3c0d3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:46.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.898+0000 7fe3c0d3d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:46.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.898+0000 7fe3c0d3d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:46.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.899+0000 7fe3c0d3d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:46.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.899+0000 7fe3c0d3d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:46.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.899+0000 7fe3c0d3d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:46.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.899+0000 7fe3c0d3d640 10 monclient(hunting): handle_auth_done global_id 94434 payload 274 2024-09-11T01:40:46.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.899+0000 7fe3c0d3d640 1 -- 172.21.15.12:0/3151697363 >> v1:172.21.15.71:6789/0 conn(0x7fe3bc134ad0 legacy=0x7fe3bc134eb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:40:46.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.899+0000 7fe3c0d3d640 1 -- 172.21.15.12:0/3151697363 >> v1:172.21.15.12:6789/0 conn(0x7fe3bc0a2200 legacy=0x7fe3bc13a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:40:46.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.899+0000 7fe3c0d3d640 10 monclient: _finish_hunting 0 2024-09-11T01:40:46.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.899+0000 7fe3c0d3d640 1 monclient: found mon.noname-a 2024-09-11T01:40:46.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.899+0000 7fe3c0d3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:46.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.899+0000 7fe3c0d3d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:40:46.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.899+0000 7fe3c0d3d640 1 -- 172.21.15.12:0/3151697363 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3bc09f2f0 con 0x7fe3bc1355c0 2024-09-11T01:40:46.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.902+0000 7fe3c0d3d640 1 -- 172.21.15.12:0/3151697363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3bc1355c0 msgr2=0x7fe3bc13d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:46.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.902+0000 7fe3c0d3d640 1 --2- 172.21.15.12:0/3151697363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3bc1355c0 0x7fe3bc13d730 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc09f750 tx=0x7fe3b402a4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2d670b9cdae3657 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:46.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.902+0000 7fe3baffd640 1 -- 172.21.15.12:0/3151697363 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4007a30 con 0x7fe3bc1355c0 2024-09-11T01:40:46.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.902+0000 7fe3baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:46.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.902+0000 7fe3baffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:40:46.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.902+0000 7fe3baffd640 10 monclient: dump: 2024-09-11T01:40:46.909 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:46.909 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:46.909 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.909 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.909 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:46.909 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:46.910 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:46.910 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:46.910 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:46.910 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:46.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.902+0000 7fe3baffd640 10 monclient: _finish_auth 0 2024-09-11T01:40:46.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.902+0000 7fe3baffd640 10 monclient: _check_auth_tickets 2024-09-11T01:40:46.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.902+0000 7fe3baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:46.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.902+0000 7fe3c27a0640 5 monclient: authenticate success, global_id 94434 2024-09-11T01:40:46.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.902+0000 7fe3c27a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:40:46.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.902+0000 7fe3baffd640 1 -- 172.21.15.12:0/3151697363 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3b4007bd0 con 0x7fe3bc1355c0 2024-09-11T01:40:46.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.902+0000 7fe3baffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:46.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.902+0000 7fe3c27a0640 10 monclient: get_monmap_and_config success 2024-09-11T01:40:46.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.902+0000 7fe3c27a0640 4 set_mon_vals no callback set 2024-09-11T01:40:46.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.902+0000 7fe3baffd640 1 -- 172.21.15.12:0/3151697363 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4031400 con 0x7fe3bc1355c0 2024-09-11T01:40:46.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.902+0000 7fe3baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:46.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.902+0000 7fe3baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:40:46.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.902+0000 7fe3baffd640 10 monclient: dump: 2024-09-11T01:40:46.911 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:46.911 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:46.911 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.911 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.911 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:46.911 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:46.911 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:46.911 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:46.911 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:46.911 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:46.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.902+0000 7fe3c27a0640 10 monclient: shutdown 2024-09-11T01:40:46.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.902+0000 7fe3c27a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:46.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.902+0000 7fe3c27a0640 1 -- 172.21.15.12:0/3151697363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3bc1355c0 msgr2=0x7fe3bc13d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:46.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.902+0000 7fe3c27a0640 1 --2- 172.21.15.12:0/3151697363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3bc1355c0 0x7fe3bc13d730 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc09f750 tx=0x7fe3b402a4b0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.902+0000 7fe3c27a0640 1 -- 172.21.15.12:0/3151697363 shutdown_connections 2024-09-11T01:40:46.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.902+0000 7fe3c27a0640 1 --2- 172.21.15.12:0/3151697363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3bc1355c0 0x7fe3bc13d730 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.902+0000 7fe3c27a0640 1 -- 172.21.15.12:0/3151697363 >> 172.21.15.12:0/3151697363 conn(0x7fe3bc0a2f70 msgr2=0x7fe3bc0a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:46.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.903+0000 7fe3c27a0640 1 -- 172.21.15.12:0/3151697363 shutdown_connections 2024-09-11T01:40:46.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.903+0000 7fe3c27a0640 1 -- 172.21.15.12:0/3151697363 wait complete. 2024-09-11T01:40:46.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.903+0000 7fe3c27a0640 10 monclient: build_initial_monmap 2024-09-11T01:40:46.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.903+0000 7fe3c27a0640 10 monclient: monmap: 2024-09-11T01:40:46.912 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:40:46.912 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:40:46.912 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:40:46.912 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:40:46.912 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:40:46.912 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:46.913 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:40:46.913 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:40:46.913 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:40:46.913 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:46.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.903+0000 7fe3c27a0640 1 Processor -- start 2024-09-11T01:40:46.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c27a0640 1 -- start start 2024-09-11T01:40:46.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c27a0640 10 monclient: init 2024-09-11T01:40:46.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c27a0640 10 monclient: _reopen_session rank -1 2024-09-11T01:40:46.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c27a0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:40:46.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c27a0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc0a2200 0x7fe3bc133410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:46.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c27a0640 10 monclient(hunting): picked mon.noname-c con 0x7fe3bc0a2200 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:40:46.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c27a0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3bc134ad0 0x7fe3bc133950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:46.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c27a0640 10 monclient(hunting): picked mon.noname-b con 0x7fe3bc134ad0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:40:46.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c27a0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3bc1355c0 0x7fe3bc13e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:46.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c27a0640 10 monclient(hunting): picked mon.noname-a con 0x7fe3bc1355c0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:40:46.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c27a0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:46.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c27a0640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe3bc0a0150 con 0x7fe3bc1355c0 2024-09-11T01:40:46.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c27a0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:46.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c27a0640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe3bc09ffd0 con 0x7fe3bc134ad0 2024-09-11T01:40:46.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c27a0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:46.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c27a0640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe3bc13dc70 con 0x7fe3bc0a2200 2024-09-11T01:40:46.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c27a0640 10 monclient(hunting): _renew_subs 2024-09-11T01:40:46.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c27a0640 10 monclient(hunting): authenticate will time out at 1453.346522s 2024-09-11T01:40:46.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c0d3d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3bc1355c0 0x7fe3bc13e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:46.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3bbfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc0a2200 0x7fe3bc133410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:46.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c0d3d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3bc1355c0 0x7fe3bc13e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55868/0 (socket says 172.21.15.12:55868) 2024-09-11T01:40:46.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c0d3d640 1 -- 172.21.15.12:0/732649359 learned_addr learned my addr 172.21.15.12:0/732649359 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:40:46.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c0d3d640 10 monclient(hunting): get_auth_request con 0x7fe3bc1355c0 auth_method 0 2024-09-11T01:40:46.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c0d3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:46.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c0d3d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:46.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c0d3d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:46.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3bb7fe640 1 -- 172.21.15.12:0/732649359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3bc134ad0 msgr2=0x7fe3bc133950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:40:46.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3bbfff640 10 monclient(hunting): get_auth_request con 0x7fe3bc0a2200 auth_method 0 2024-09-11T01:40:46.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:46.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:46.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:46.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3bb7fe640 1 --2- 172.21.15.12:0/732649359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3bc134ad0 0x7fe3bc133950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:40:46.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c0d3d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:46.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c0d3d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:46.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c0d3d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:46.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:46.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:46.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:46.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c0d3d640 10 monclient(hunting): handle_auth_done global_id 94437 payload 824 2024-09-11T01:40:46.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c0d3d640 1 -- 172.21.15.12:0/732649359 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc0a2200 msgr2=0x7fe3bc133410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:46.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c0d3d640 1 --2- 172.21.15.12:0/732649359 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc0a2200 0x7fe3bc133410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c0d3d640 1 -- 172.21.15.12:0/732649359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3bc134ad0 msgr2=0x7fe3bc133950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:40:46.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c0d3d640 1 --2- 172.21.15.12:0/732649359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3bc134ad0 0x7fe3bc133950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c0d3d640 10 monclient: _finish_hunting 0 2024-09-11T01:40:46.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c0d3d640 1 monclient: found mon.noname-a 2024-09-11T01:40:46.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c0d3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:40:46.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c0d3d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:40:46.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3c0d3d640 1 -- 172.21.15.12:0/732649359 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3bc140050 con 0x7fe3bc1355c0 2024-09-11T01:40:46.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.904+0000 7fe3bbfff640 1 --2- 172.21.15.12:0/732649359 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc0a2200 0x7fe3bc133410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:40:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.907+0000 7fe3c0d3d640 1 -- 172.21.15.12:0/732649359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3bc1355c0 msgr2=0x7fe3bc13e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.907+0000 7fe3c0d3d640 1 --2- 172.21.15.12:0/732649359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3bc1355c0 0x7fe3bc13e050 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe3b402aaa0 tx=0x7fe3b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.907+0000 7fe3b8ff9640 1 -- 172.21.15.12:0/732649359 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4040070 con 0x7fe3bc1355c0 2024-09-11T01:40:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.907+0000 7fe3b8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.907+0000 7fe3b8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:40:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.907+0000 7fe3b8ff9640 10 monclient: dump: 2024-09-11T01:40:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:46.918 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:46.918 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:46.918 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:46.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.907+0000 7fe3b8ff9640 10 monclient: _finish_auth 0 2024-09-11T01:40:46.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.907+0000 7fe3b8ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:40:46.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.907+0000 7fe3b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:40:46.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.907+0000 7fe3c27a0640 5 monclient: authenticate success, global_id 94437 2024-09-11T01:40:46.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.907+0000 7fe3b8ff9640 1 -- 172.21.15.12:0/732649359 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3b4045660 con 0x7fe3bc1355c0 2024-09-11T01:40:46.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.907+0000 7fe3b8ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:40:46.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.907+0000 7fe3b8ff9640 1 -- 172.21.15.12:0/732649359 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b40314c0 con 0x7fe3bc1355c0 2024-09-11T01:40:46.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.907+0000 7fe3b8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:46.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.907+0000 7fe3b8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:40:46.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.907+0000 7fe3b8ff9640 10 monclient: dump: 2024-09-11T01:40:46.918 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:46.919 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:46.919 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.919 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.919 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:46.919 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:46.919 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:46.919 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:46.919 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:46.919 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:46.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.907+0000 7fe3ba7fc640 4 set_mon_vals no callback set 2024-09-11T01:40:46.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.907+0000 7fe3c27a0640 10 monclient: _renew_subs 2024-09-11T01:40:46.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.907+0000 7fe3c27a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:40:46.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.907+0000 7fe3c27a0640 1 -- 172.21.15.12:0/732649359 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3bc11dd20 con 0x7fe3bc1355c0 2024-09-11T01:40:46.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.907+0000 7fe3c27a0640 10 monclient: _renew_subs 2024-09-11T01:40:46.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.907+0000 7fe3c27a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:40:46.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.907+0000 7fe3c27a0640 1 -- 172.21.15.12:0/732649359 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3bc11e2e0 con 0x7fe3bc1355c0 2024-09-11T01:40:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.908+0000 7fe3b8ff9640 1 -- 172.21.15.12:0/732649359 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fe3b4007a40 con 0x7fe3bc1355c0 2024-09-11T01:40:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.908+0000 7fe3c27a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.908+0000 7fe3c27a0640 1 --2- 172.21.15.12:0/732649359 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc13e590 0x7fe3bc13e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.908+0000 7fe3c27a0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.908+0000 7fe3c27a0640 1 -- 172.21.15.12:0/732649359 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe3bc134eb0 con 0x7fe3bc13e590 2024-09-11T01:40:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.908+0000 7fe3bbfff640 1 --2- 172.21.15.12:0/732649359 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc13e590 0x7fe3bc13e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:46.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.908+0000 7fe3bbfff640 10 monclient: get_auth_request con 0x7fe3bc13e590 auth_method 0 2024-09-11T01:40:46.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.908+0000 7fe3bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:46.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.908+0000 7fe3bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:46.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.908+0000 7fe3bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:46.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.908+0000 7fe3bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:46.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.908+0000 7fe3bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:46.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.908+0000 7fe3bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:46.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.908+0000 7fe3b8ff9640 1 --2- 172.21.15.12:0/732649359 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe3a803e1f0 0x7fe3a80406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:46.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.908+0000 7fe3b8ff9640 1 -- 172.21.15.12:0/732649359 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(381..381 src has 285..381) ==== 4783+0+0 (secure 0 0 0) 0x7fe3b4074a70 con 0x7fe3bc1355c0 2024-09-11T01:40:46.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.908+0000 7fe3bbfff640 10 monclient(hunting): handle_auth_done global_id 114272 payload 824 2024-09-11T01:40:46.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.908+0000 7fe3bbfff640 1 -- 172.21.15.12:0/732649359 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe3bc0008d0 con 0x7fe3bc13e590 2024-09-11T01:40:46.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.908+0000 7fe3bb7fe640 1 --2- 172.21.15.12:0/732649359 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe3a803e1f0 0x7fe3a80406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:46.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.908+0000 7fe3bb7fe640 10 monclient: get_auth_request con 0x7fe3a803e1f0 auth_method 0 2024-09-11T01:40:46.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.910+0000 7fe3bbfff640 1 -- 172.21.15.12:0/732649359 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc13e590 msgr2=0x7fe3bc13e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:46.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.910+0000 7fe3bbfff640 1 --2- 172.21.15.12:0/732649359 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc13e590 0x7fe3bc13e970 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac003930 tx=0x7fe3ac005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:46.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.911+0000 7fe3b8ff9640 1 -- 172.21.15.12:0/732649359 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac016070 con 0x7fe3bc13e590 2024-09-11T01:40:46.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.911+0000 7fe3b8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:46.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.911+0000 7fe3b8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:40:46.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.911+0000 7fe3b8ff9640 10 monclient: dump: 2024-09-11T01:40:46.925 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:46.925 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:46.925 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.925 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.925 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:46.925 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:46.925 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:46.926 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.911+0000 7fe3b8ff9640 1 -- 172.21.15.12:0/732649359 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe3ac00ed10 con 0x7fe3bc13e590 2024-09-11T01:40:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.911+0000 7fe3b8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:40:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.911+0000 7fe3b8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:40:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.911+0000 7fe3b8ff9640 1 -- 172.21.15.12:0/732649359 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc13e590 msgr2=0x7fe3bc13e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.911+0000 7fe3b8ff9640 1 --2- 172.21.15.12:0/732649359 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc13e590 0x7fe3bc13e970 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac003930 tx=0x7fe3ac005850 comp rx=0 tx=0).stop 2024-09-11T01:40:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.911+0000 7fe3b8ff9640 1 -- 172.21.15.12:0/732649359 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc13e590 msgr2=0x7fe3bc13e970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:40:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.911+0000 7fe3b8ff9640 1 --2- 172.21.15.12:0/732649359 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc13e590 0x7fe3bc13e970 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac003930 tx=0x7fe3ac005850 comp rx=0 tx=0).stop 2024-09-11T01:40:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.911+0000 7fe3bb7fe640 1 --2- 172.21.15.12:0/732649359 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe3a803e1f0 0x7fe3a80406b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0007fd0 tx=0x7fe3b0007b20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.917+0000 7fe3c27a0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:40:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.917+0000 7fe3c27a0640 1 --2- 172.21.15.12:0/732649359 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc13eeb0 0x7fe3bc0726b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:40:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.917+0000 7fe3c27a0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:40:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.917+0000 7fe3c27a0640 1 -- 172.21.15.12:0/732649359 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe3bc11da40 con 0x7fe3bc13eeb0 2024-09-11T01:40:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.917+0000 7fe3bbfff640 1 --2- 172.21.15.12:0/732649359 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc13eeb0 0x7fe3bc0726b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:40:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.917+0000 7fe3bbfff640 10 monclient: get_auth_request con 0x7fe3bc13eeb0 auth_method 0 2024-09-11T01:40:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.917+0000 7fe3bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:40:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.917+0000 7fe3bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:40:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.917+0000 7fe3bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:40:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.917+0000 7fe3bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:40:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.917+0000 7fe3bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:40:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.917+0000 7fe3bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:40:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.917+0000 7fe3bbfff640 10 monclient(hunting): handle_auth_done global_id 114275 payload 824 2024-09-11T01:40:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.917+0000 7fe3bbfff640 1 -- 172.21.15.12:0/732649359 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe3bc133e90 con 0x7fe3bc13eeb0 2024-09-11T01:40:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.920+0000 7fe3bbfff640 1 -- 172.21.15.12:0/732649359 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc13eeb0 msgr2=0x7fe3bc0726b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:40:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.920+0000 7fe3bbfff640 1 --2- 172.21.15.12:0/732649359 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc13eeb0 0x7fe3bc0726b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac018000 tx=0x7fe3ac0059d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:40:46.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.920+0000 7fe3b8ff9640 1 -- 172.21.15.12:0/732649359 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac016040 con 0x7fe3bc13eeb0 2024-09-11T01:40:46.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.920+0000 7fe3b8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:40:46.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.920+0000 7fe3b8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:40:46.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.920+0000 7fe3b8ff9640 10 monclient: dump: 2024-09-11T01:40:46.928 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:40:46.928 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:40:46.928 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.928 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:40:46.928 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:40:46.928 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:40:46.928 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:40:46.928 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:40:46.928 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:40:46.929 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:40:46.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.920+0000 7fe3b8ff9640 1 -- 172.21.15.12:0/732649359 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4864 (secure 0 0 0) 0x7fe3ac005c50 con 0x7fe3bc13eeb0 2024-09-11T01:40:46.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.920+0000 7fe3b8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:40:46.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.920+0000 7fe3b8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:40:46.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.920+0000 7fe3b8ff9640 1 -- 172.21.15.12:0/732649359 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc13eeb0 msgr2=0x7fe3bc0726b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:46.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.920+0000 7fe3b8ff9640 1 --2- 172.21.15.12:0/732649359 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc13eeb0 0x7fe3bc0726b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac018000 tx=0x7fe3ac0059d0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.920+0000 7fe3b8ff9640 1 -- 172.21.15.12:0/732649359 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc13eeb0 msgr2=0x7fe3bc0726b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:40:46.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.920+0000 7fe3b8ff9640 1 --2- 172.21.15.12:0/732649359 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc13eeb0 0x7fe3bc0726b0 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac018000 tx=0x7fe3ac0059d0 comp rx=0 tx=0).stop 2024-09-11T01:40:46.929 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:40:46.929 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:40:46.929 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:40:46.930 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:40:46.930 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 108, 2024-09-11T01:40:46.930 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 92717, 2024-09-11T01:40:46.930 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:40:46.930 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:40:46.930 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:40:46.930 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:46.930 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:40:46.930 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:40:46.930 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:40:46.930 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:40:46.930 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:40:46.930 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:40:46.931 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:40:46.931 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:40:46.931 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:40:46.931 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:40:46.931 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:40:46.931 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:40:46.931 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:40:46.931 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:40:46.931 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:40:46.931 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:46.931 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:40:46.931 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:40:46.931 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:40:46.931 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:40:46.932 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:40:46.932 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:40:46.932 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:40:46.932 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:40:46.932 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:40:46.932 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:40:46.932 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:40:46.932 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:40:46.932 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:40:46.932 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:46.932 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:46.932 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:40:46.932 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:40:46.933 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:40:46.933 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:40:46.933 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:40:46.933 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:40:46.933 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:40:46.933 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:40:46.933 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:40:46.933 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:40:46.933 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:40:46.933 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:40:46.933 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:40:46.933 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:40:46.933 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:40:46.933 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:40:46.934 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:40:46.934 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:40:46.934 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:40:46.934 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:40:46.934 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:40:46.934 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:40:46.934 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:40:46.934 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:40:46.934 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:40:46.934 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:40:46.934 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:40:46.934 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:40:46.934 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:46.935 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:40:46.935 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:46.935 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:40:46.935 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:46.935 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:40:46.935 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:40:46.935 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:40:46.935 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:40:46.935 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:46.935 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:40:46.935 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:40:46.935 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:46.935 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:46.935 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:46.936 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:40:46.936 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:40:46.936 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:46.936 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:46.936 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:46.936 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:46.936 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:40:46.936 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:40:46.936 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:40:46.936 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:40:46.936 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:40:46.936 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:46.936 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:46.936 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:40:46.937 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:40:46.937 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:40:46.937 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:40:46.937 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:46.937 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:40:46.937 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:40:46.937 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:46.937 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:46.937 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:46.937 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:40:46.937 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:40:46.937 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:46.937 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:46.937 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:46.938 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:46.938 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:40:46.938 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:40:46.938 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:40:46.938 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:40:46.938 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:40:46.938 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:46.938 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:46.938 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:40:46.938 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:40:46.938 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:40:46.938 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:40:46.938 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:46.938 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:40:46.939 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:40:46.939 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:46.939 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:46.939 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:46.939 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:40:46.939 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:40:46.939 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:40:46.939 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:46.939 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:46.939 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:46.939 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:40:46.939 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:40:46.939 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:40:46.939 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:40:46.940 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:40:46.940 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:46.940 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:46.940 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:46.940 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:40:46.940 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:40:46.940 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:46.940 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:40:46.940 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:40:46.940 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:40:46.940 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:46.940 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:46.940 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:40:46.941 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:46.941 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:40:46.941 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:40:46.941 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:40:46.941 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:46.941 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:46.941 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:40:46.941 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:46.941 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:40:46.941 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:40:46.941 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:40:46.941 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:46.941 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:40:46.941 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:40:46.941 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:40:46.942 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:40:46.942 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:40:46.942 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:40:46.942 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:40:46.942 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:40:46.942 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:40:46.942 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:40:46.942 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:40:46.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.920+0000 7fe3c27a0640 1 -- 172.21.15.12:0/732649359 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe3a803e1f0 msgr2=0x7fe3a80406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:46.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.920+0000 7fe3c27a0640 1 --2- 172.21.15.12:0/732649359 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe3a803e1f0 0x7fe3a80406b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0007fd0 tx=0x7fe3b0007b20 comp rx=0 tx=0).stop 2024-09-11T01:40:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.920+0000 7fe3c27a0640 10 monclient: shutdown 2024-09-11T01:40:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.920+0000 7fe3c27a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:40:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.920+0000 7fe3c27a0640 1 -- 172.21.15.12:0/732649359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3bc1355c0 msgr2=0x7fe3bc13e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:40:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.920+0000 7fe3c27a0640 1 --2- 172.21.15.12:0/732649359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3bc1355c0 0x7fe3bc13e050 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe3b402aaa0 tx=0x7fe3b4004a40 comp rx=0 tx=0).stop 2024-09-11T01:40:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.920+0000 7fe3c0d3d640 1 -- 172.21.15.12:0/732649359 reap_dead start 2024-09-11T01:40:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.920+0000 7fe3c0d3d640 1 -- 172.21.15.12:0/732649359 reap_dead start 2024-09-11T01:40:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.920+0000 7fe3c27a0640 1 -- 172.21.15.12:0/732649359 shutdown_connections 2024-09-11T01:40:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.920+0000 7fe3c27a0640 1 -- 172.21.15.12:0/732649359 >> 172.21.15.12:0/732649359 conn(0x7fe3bc0a2f70 msgr2=0x7fe3bc0a0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:40:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.920+0000 7fe3c27a0640 1 -- 172.21.15.12:0/732649359 shutdown_connections 2024-09-11T01:40:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:40:46.920+0000 7fe3c27a0640 1 -- 172.21.15.12:0/732649359 wait complete. 2024-09-11T01:40:46.944 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:40:46.945 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:40:47.269 INFO:tasks.rados.rados.0.smithi071.stdout:1662: done (2 left) 2024-09-11T01:40:47.269 INFO:tasks.rados.rados.0.smithi071.stdout:1663: done (1 left) 2024-09-11T01:40:47.269 INFO:tasks.rados.rados.0.smithi071.stdout:1664: done (0 left) 2024-09-11T01:40:47.269 INFO:tasks.rados.rados.0.smithi071.stdout:1665: write oid 18 current snap is 182 2024-09-11T01:40:47.269 INFO:tasks.rados.rados.0.smithi071.stdout:1665: seq_num 615 ranges {736305=786375,2105367=230938} 2024-09-11T01:40:47.277 INFO:tasks.rados.rados.0.smithi071.stdout:1665: writing smithi07134777-18 from 736305 to 1522680 tid 1 2024-09-11T01:40:47.281 INFO:tasks.rados.rados.0.smithi071.stdout:1665: writing smithi07134777-18 from 2105367 to 2336305 tid 2 2024-09-11T01:40:47.281 INFO:tasks.rados.rados.0.smithi071.stdout:1666: read oid 2 snap 179 2024-09-11T01:40:47.281 INFO:tasks.rados.rados.0.smithi071.stdout:1666: expect (ObjNum 610 snap 176 seq_num 610) 2024-09-11T01:40:47.281 INFO:tasks.rados.rados.0.smithi071.stdout:1667: snap_create 2024-09-11T01:40:47.285 INFO:tasks.rados.rados.0.smithi071.stdout:1665: finishing write tid 1 to smithi07134777-18 2024-09-11T01:40:47.286 INFO:tasks.rados.rados.0.smithi071.stdout:1666: expect (ObjNum 610 snap 176 seq_num 610) 2024-09-11T01:40:47.289 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 545401 2024-09-11T01:40:47.329 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 875218 2024-09-11T01:40:47.386 INFO:tasks.rados.rados.0.smithi071.stdout:1665: finishing write tid 2 to smithi07134777-18 2024-09-11T01:40:47.386 INFO:tasks.rados.rados.0.smithi071.stdout:1665: finishing write tid 3 to smithi07134777-18 2024-09-11T01:40:47.386 INFO:tasks.rados.rados.0.smithi071.stdout:1665: finishing write tid 4 to smithi07134777-18 2024-09-11T01:40:47.386 INFO:tasks.rados.rados.0.smithi071.stdout:1665: oid 18 updating version 0 to 644 2024-09-11T01:40:47.386 INFO:tasks.rados.rados.0.smithi071.stdout:1665: oid 18 updating version 644 to 646 2024-09-11T01:40:47.387 INFO:tasks.rados.rados.0.smithi071.stdout:1665: oid 18 version 646 is already newer than 645 2024-09-11T01:40:47.387 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 646 (ObjNum 615 snap 182 seq_num 615) dirty exists 2024-09-11T01:40:47.387 INFO:tasks.rados.rados.0.smithi071.stdout:1665: left oid 18 (ObjNum 615 snap 182 seq_num 615) 2024-09-11T01:40:48.271 INFO:tasks.rados.rados.0.smithi071.stdout:1665: done (2 left) 2024-09-11T01:40:48.271 INFO:tasks.rados.rados.0.smithi071.stdout:1666: done (1 left) 2024-09-11T01:40:48.271 INFO:tasks.rados.rados.0.smithi071.stdout:1667: done (0 left) 2024-09-11T01:40:48.271 INFO:tasks.rados.rados.0.smithi071.stdout:1668: read oid 10 snap -1 2024-09-11T01:40:48.271 INFO:tasks.rados.rados.0.smithi071.stdout:1668: expect (ObjNum 601 snap 172 seq_num 601) 2024-09-11T01:40:48.271 INFO:tasks.rados.rados.0.smithi071.stdout:1669: write oid 27 current snap is 183 2024-09-11T01:40:48.272 INFO:tasks.rados.rados.0.smithi071.stdout:1669: seq_num 616 ranges {753112=577977,1880202=472910} 2024-09-11T01:40:48.278 INFO:tasks.rados.rados.0.smithi071.stdout:1669: writing smithi07134777-27 from 753112 to 1331089 tid 1 2024-09-11T01:40:48.284 INFO:tasks.rados.rados.0.smithi071.stdout:1669: writing smithi07134777-27 from 1880202 to 2353112 tid 2 2024-09-11T01:40:48.284 INFO:tasks.rados.rados.0.smithi071.stdout:1670: write oid 29 current snap is 183 2024-09-11T01:40:48.284 INFO:tasks.rados.rados.0.smithi071.stdout:1670: seq_num 617 ranges {769919=653226,1938684=431235} 2024-09-11T01:40:48.291 INFO:tasks.rados.rados.0.smithi071.stdout:1670: writing smithi07134777-29 from 769919 to 1423145 tid 1 2024-09-11T01:40:48.297 INFO:tasks.rados.rados.0.smithi071.stdout:1670: writing smithi07134777-29 from 1938684 to 2369919 tid 2 2024-09-11T01:40:48.297 INFO:tasks.rados.rados.0.smithi071.stdout:1671: write (excl) oid 18 current snap is 183 2024-09-11T01:40:48.297 INFO:tasks.rados.rados.0.smithi071.stdout:1668: expect (ObjNum 601 snap 172 seq_num 601) 2024-09-11T01:40:48.316 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 506254 2024-09-11T01:40:48.364 INFO:tasks.rados.rados.0.smithi071.stdout:1669: finishing write tid 1 to smithi07134777-27 2024-09-11T01:40:48.364 INFO:tasks.rados.rados.0.smithi071.stdout:1669: finishing write tid 2 to smithi07134777-27 2024-09-11T01:40:48.364 INFO:tasks.rados.rados.0.smithi071.stdout:1669: finishing write tid 3 to smithi07134777-27 2024-09-11T01:40:48.365 INFO:tasks.rados.rados.0.smithi071.stdout:1669: finishing write tid 4 to smithi07134777-27 2024-09-11T01:40:48.365 INFO:tasks.rados.rados.0.smithi071.stdout:1669: oid 27 updating version 0 to 848 2024-09-11T01:40:48.365 INFO:tasks.rados.rados.0.smithi071.stdout:1669: oid 27 updating version 848 to 850 2024-09-11T01:40:48.365 INFO:tasks.rados.rados.0.smithi071.stdout:1669: oid 27 version 850 is already newer than 849 2024-09-11T01:40:48.365 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 850 (ObjNum 616 snap 183 seq_num 616) dirty exists 2024-09-11T01:40:48.365 INFO:tasks.rados.rados.0.smithi071.stdout:1669: left oid 27 (ObjNum 616 snap 183 seq_num 616) 2024-09-11T01:40:48.365 INFO:tasks.rados.rados.0.smithi071.stdout:1670: finishing write tid 1 to smithi07134777-29 2024-09-11T01:40:48.365 INFO:tasks.rados.rados.0.smithi071.stdout:1670: finishing write tid 2 to smithi07134777-29 2024-09-11T01:40:48.365 INFO:tasks.rados.rados.0.smithi071.stdout:1670: finishing write tid 3 to smithi07134777-29 2024-09-11T01:40:48.365 INFO:tasks.rados.rados.0.smithi071.stdout:1671: seq_num 618 ranges {786726=728475,2280813=105913} 2024-09-11T01:40:48.372 INFO:tasks.rados.rados.0.smithi071.stdout:1671: writing smithi07134777-18 from 786726 to 1515201 tid 1 2024-09-11T01:40:48.377 INFO:tasks.rados.rados.0.smithi071.stdout:1671: writing smithi07134777-18 from 2280813 to 2386726 tid 2 2024-09-11T01:40:48.377 INFO:tasks.rados.rados.0.smithi071.stdout:1668: done (3 left) 2024-09-11T01:40:48.377 INFO:tasks.rados.rados.0.smithi071.stdout:1669: done (2 left) 2024-09-11T01:40:48.377 INFO:tasks.rados.rados.0.smithi071.stdout:1672: write (excl) oid 31 current snap is 183 2024-09-11T01:40:48.377 INFO:tasks.rados.rados.0.smithi071.stdout:1672: seq_num 619 ranges {403533=403724,1539295=652201} 2024-09-11T01:40:48.381 INFO:tasks.rados.rados.0.smithi071.stdout:1672: writing smithi07134777-31 from 403533 to 807257 tid 1 2024-09-11T01:40:48.388 INFO:tasks.rados.rados.0.smithi071.stdout:1672: writing smithi07134777-31 from 1539295 to 2191496 tid 2 2024-09-11T01:40:48.388 INFO:tasks.rados.rados.0.smithi071.stdout:1673: snap_remove snap 179 2024-09-11T01:40:48.949 INFO:tasks.rados.rados.0.smithi071.stdout:1673: done (3 left) 2024-09-11T01:40:48.949 INFO:tasks.rados.rados.0.smithi071.stdout:1674: write oid 23 current snap is 183 2024-09-11T01:40:48.949 INFO:tasks.rados.rados.0.smithi071.stdout:1670: finishing write tid 4 to smithi07134777-29 2024-09-11T01:40:48.949 INFO:tasks.rados.rados.0.smithi071.stdout:1670: oid 29 updating version 0 to 936 2024-09-11T01:40:48.949 INFO:tasks.rados.rados.0.smithi071.stdout:1670: oid 29 updating version 936 to 937 2024-09-11T01:40:48.949 INFO:tasks.rados.rados.0.smithi071.stdout:1670: oid 29 updating version 937 to 938 2024-09-11T01:40:48.950 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 938 (ObjNum 617 snap 183 seq_num 617) dirty exists 2024-09-11T01:40:48.950 INFO:tasks.rados.rados.0.smithi071.stdout:1670: left oid 29 (ObjNum 617 snap 183 seq_num 617) 2024-09-11T01:40:48.950 INFO:tasks.rados.rados.0.smithi071.stdout:1674: seq_num 620 ranges {420340=478973,1597777=512212} 2024-09-11T01:40:48.953 INFO:tasks.rados.rados.0.smithi071.stdout:1674: writing smithi07134777-23 from 420340 to 899313 tid 1 2024-09-11T01:40:48.960 INFO:tasks.rados.rados.0.smithi071.stdout:1674: writing smithi07134777-23 from 1597777 to 2109989 tid 2 2024-09-11T01:40:48.961 INFO:tasks.rados.rados.0.smithi071.stdout:1670: done (3 left) 2024-09-11T01:40:48.961 INFO:tasks.rados.rados.0.smithi071.stdout:1675: snap_create 2024-09-11T01:40:48.961 INFO:tasks.rados.rados.0.smithi071.stdout:1671: finishing write tid 1 to smithi07134777-18 2024-09-11T01:40:48.961 INFO:tasks.rados.rados.0.smithi071.stdout:1671: finishing write tid 2 to smithi07134777-18 2024-09-11T01:40:48.961 INFO:tasks.rados.rados.0.smithi071.stdout:1671: finishing write tid 3 to smithi07134777-18 2024-09-11T01:40:48.961 INFO:tasks.rados.rados.0.smithi071.stdout:1671: finishing write tid 4 to smithi07134777-18 2024-09-11T01:40:48.961 INFO:tasks.rados.rados.0.smithi071.stdout:1671: oid 18 updating version 0 to 647 2024-09-11T01:40:48.961 INFO:tasks.rados.rados.0.smithi071.stdout:1671: oid 18 updating version 647 to 649 2024-09-11T01:40:48.961 INFO:tasks.rados.rados.0.smithi071.stdout:1671: oid 18 version 649 is already newer than 648 2024-09-11T01:40:48.961 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 649 (ObjNum 618 snap 183 seq_num 618) dirty exists 2024-09-11T01:40:48.961 INFO:tasks.rados.rados.0.smithi071.stdout:1671: left oid 18 (ObjNum 618 snap 183 seq_num 618) 2024-09-11T01:40:48.961 INFO:tasks.rados.rados.0.smithi071.stdout:1672: finishing write tid 1 to smithi07134777-31 2024-09-11T01:40:48.962 INFO:tasks.rados.rados.0.smithi071.stdout:1672: finishing write tid 2 to smithi07134777-31 2024-09-11T01:40:48.962 INFO:tasks.rados.rados.0.smithi071.stdout:1672: finishing write tid 3 to smithi07134777-31 2024-09-11T01:40:48.962 INFO:tasks.rados.rados.0.smithi071.stdout:1672: finishing write tid 4 to smithi07134777-31 2024-09-11T01:40:48.962 INFO:tasks.rados.rados.0.smithi071.stdout:1672: oid 31 updating version 0 to 806 2024-09-11T01:40:48.962 INFO:tasks.rados.rados.0.smithi071.stdout:1672: oid 31 version 806 is already newer than 804 2024-09-11T01:40:48.962 INFO:tasks.rados.rados.0.smithi071.stdout:1672: oid 31 version 806 is already newer than 805 2024-09-11T01:40:48.962 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 806 (ObjNum 619 snap 183 seq_num 619) dirty exists 2024-09-11T01:40:48.962 INFO:tasks.rados.rados.0.smithi071.stdout:1672: left oid 31 (ObjNum 619 snap 183 seq_num 619) 2024-09-11T01:40:48.965 INFO:tasks.rados.rados.0.smithi071.stdout:1674: finishing write tid 1 to smithi07134777-23 2024-09-11T01:40:48.967 INFO:tasks.rados.rados.0.smithi071.stdout:1674: finishing write tid 2 to smithi07134777-23 2024-09-11T01:40:48.968 INFO:tasks.rados.rados.0.smithi071.stdout:1674: finishing write tid 3 to smithi07134777-23 2024-09-11T01:40:48.968 INFO:tasks.rados.rados.0.smithi071.stdout:1674: finishing write tid 4 to smithi07134777-23 2024-09-11T01:40:48.968 INFO:tasks.rados.rados.0.smithi071.stdout:1674: oid 23 updating version 0 to 295 2024-09-11T01:40:48.968 INFO:tasks.rados.rados.0.smithi071.stdout:1674: oid 23 version 295 is already newer than 294 2024-09-11T01:40:48.968 INFO:tasks.rados.rados.0.smithi071.stdout:1674: oid 23 version 295 is already newer than 293 2024-09-11T01:40:48.968 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 295 (ObjNum 620 snap 183 seq_num 620) dirty exists 2024-09-11T01:40:48.968 INFO:tasks.rados.rados.0.smithi071.stdout:1674: left oid 23 (ObjNum 620 snap 183 seq_num 620) 2024-09-11T01:40:49.954 INFO:tasks.rados.rados.0.smithi071.stdout:1671: done (3 left) 2024-09-11T01:40:49.955 INFO:tasks.rados.rados.0.smithi071.stdout:1672: done (2 left) 2024-09-11T01:40:49.955 INFO:tasks.rados.rados.0.smithi071.stdout:1674: done (1 left) 2024-09-11T01:40:49.955 INFO:tasks.rados.rados.0.smithi071.stdout:1675: done (0 left) 2024-09-11T01:40:49.955 INFO:tasks.rados.rados.0.smithi071.stdout:1676: copy_from oid 29 from oid 36 current snap is 184 2024-09-11T01:40:49.955 INFO:tasks.rados.rados.0.smithi071.stdout:1677: delete oid 8 current snap is 184 2024-09-11T01:40:49.958 INFO:tasks.rados.rados.0.smithi071.stdout:1676: finishing copy_from to smithi07134777-29 2024-09-11T01:40:49.959 INFO:tasks.rados.rados.0.smithi071.stdout:1676: got expected ENOENT (src dne) 2024-09-11T01:40:49.959 INFO:tasks.rados.rados.0.smithi071.stdout:1676: finishing copy_from racing read to smithi07134777-29 2024-09-11T01:40:49.959 INFO:tasks.rados.rados.0.smithi071.stdout:1676: done (1 left) 2024-09-11T01:40:49.959 INFO:tasks.rados.rados.0.smithi071.stdout:1677: done (0 left) 2024-09-11T01:40:49.959 INFO:tasks.rados.rados.0.smithi071.stdout:1678: copy_from oid 45 from oid 35 current snap is 184 2024-09-11T01:40:49.959 INFO:tasks.rados.rados.0.smithi071.stdout:1679: copy_from oid 42 from oid 31 current snap is 184 2024-09-11T01:40:49.959 INFO:tasks.rados.rados.0.smithi071.stdout:1680: snap_create 2024-09-11T01:40:49.969 INFO:tasks.rados.rados.0.smithi071.stdout:1679: finishing copy_from racing read to smithi07134777-42 2024-09-11T01:40:49.970 INFO:tasks.rados.rados.0.smithi071.stdout:1678: finishing copy_from racing read to smithi07134777-45 2024-09-11T01:40:49.975 INFO:tasks.rados.rados.0.smithi071.stdout:1679: finishing copy_from to smithi07134777-42 2024-09-11T01:40:49.975 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 806 (ObjNum 619 snap 183 seq_num 619) dirty exists 2024-09-11T01:40:49.975 INFO:tasks.rados.rados.0.smithi071.stdout:1678: finishing copy_from to smithi07134777-45 2024-09-11T01:40:49.976 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 900 (ObjNum 612 snap 181 seq_num 612) dirty exists 2024-09-11T01:40:50.958 INFO:tasks.rados.rados.0.smithi071.stdout:1678: done (2 left) 2024-09-11T01:40:50.958 INFO:tasks.rados.rados.0.smithi071.stdout:1679: done (1 left) 2024-09-11T01:40:50.959 INFO:tasks.rados.rados.0.smithi071.stdout:1680: done (0 left) 2024-09-11T01:40:50.959 INFO:tasks.rados.rados.0.smithi071.stdout:1681: copy_from oid 14 from oid 21 current snap is 185 2024-09-11T01:40:50.959 INFO:tasks.rados.rados.0.smithi071.stdout:1682: snap_create 2024-09-11T01:40:50.973 INFO:tasks.rados.rados.0.smithi071.stdout:1681: finishing copy_from racing read to smithi07134777-14 2024-09-11T01:40:50.977 INFO:tasks.rados.rados.0.smithi071.stdout:1681: finishing copy_from to smithi07134777-14 2024-09-11T01:40:50.977 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 583 (ObjNum 542 snap 154 seq_num 542) dirty exists 2024-09-11T01:40:51.963 INFO:tasks.rados.rados.0.smithi071.stdout:1681: done (1 left) 2024-09-11T01:40:51.963 INFO:tasks.rados.rados.0.smithi071.stdout:1682: done (0 left) 2024-09-11T01:40:51.963 INFO:tasks.rados.rados.0.smithi071.stdout:1683: snap_create 2024-09-11T01:40:52.964 INFO:tasks.rados.rados.0.smithi071.stdout:1683: done (0 left) 2024-09-11T01:40:52.964 INFO:tasks.rados.rados.0.smithi071.stdout:1684: write oid 2 current snap is 187 2024-09-11T01:40:52.964 INFO:tasks.rados.rados.0.smithi071.stdout:1684: seq_num 622 ranges {453954=629471,1598388=515881} 2024-09-11T01:40:52.969 INFO:tasks.rados.rados.0.smithi071.stdout:1684: writing smithi07134777-2 from 453954 to 1083425 tid 1 2024-09-11T01:40:52.975 INFO:tasks.rados.rados.0.smithi071.stdout:1684: writing smithi07134777-2 from 1598388 to 2114269 tid 2 2024-09-11T01:40:52.976 INFO:tasks.rados.rados.0.smithi071.stdout:1685: write (excl) oid 27 current snap is 187 2024-09-11T01:40:52.976 INFO:tasks.rados.rados.0.smithi071.stdout:1685: seq_num 623 ranges {470761=704720,1656870=659539} 2024-09-11T01:40:52.982 INFO:tasks.rados.rados.0.smithi071.stdout:1685: writing smithi07134777-27 from 470761 to 1175481 tid 1 2024-09-11T01:40:52.989 INFO:tasks.rados.rados.0.smithi071.stdout:1685: writing smithi07134777-27 from 1656870 to 2316409 tid 2 2024-09-11T01:40:52.989 INFO:tasks.rados.rados.0.smithi071.stdout:1686: read oid 50 snap -1 2024-09-11T01:40:52.989 INFO:tasks.rados.rados.0.smithi071.stdout:1686: expect (ObjNum 551 snap 157 seq_num 551) 2024-09-11T01:40:52.989 INFO:tasks.rados.rados.0.smithi071.stdout:1687: delete oid 1 current snap is 187 2024-09-11T01:40:52.989 INFO:tasks.rados.rados.0.smithi071.stdout:1684: finishing write tid 1 to smithi07134777-2 2024-09-11T01:40:52.989 INFO:tasks.rados.rados.0.smithi071.stdout:1684: finishing write tid 2 to smithi07134777-2 2024-09-11T01:40:52.989 INFO:tasks.rados.rados.0.smithi071.stdout:1684: finishing write tid 3 to smithi07134777-2 2024-09-11T01:40:52.989 INFO:tasks.rados.rados.0.smithi071.stdout:1684: finishing write tid 4 to smithi07134777-2 2024-09-11T01:40:52.989 INFO:tasks.rados.rados.0.smithi071.stdout:1684: oid 2 updating version 0 to 807 2024-09-11T01:40:52.990 INFO:tasks.rados.rados.0.smithi071.stdout:1684: oid 2 updating version 807 to 809 2024-09-11T01:40:52.990 INFO:tasks.rados.rados.0.smithi071.stdout:1684: oid 2 version 809 is already newer than 808 2024-09-11T01:40:52.990 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 809 (ObjNum 622 snap 187 seq_num 622) dirty exists 2024-09-11T01:40:52.990 INFO:tasks.rados.rados.0.smithi071.stdout:1684: left oid 2 (ObjNum 622 snap 187 seq_num 622) 2024-09-11T01:40:52.992 INFO:tasks.rados.rados.0.smithi071.stdout:1686: expect (ObjNum 551 snap 157 seq_num 551) 2024-09-11T01:40:53.039 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:40:53.074 INFO:tasks.rados.rados.0.smithi071.stdout:1685: finishing write tid 1 to smithi07134777-27 2024-09-11T01:40:53.074 INFO:tasks.rados.rados.0.smithi071.stdout:1685: finishing write tid 2 to smithi07134777-27 2024-09-11T01:40:53.074 INFO:tasks.rados.rados.0.smithi071.stdout:1684: done (3 left) 2024-09-11T01:40:53.074 INFO:tasks.rados.rados.0.smithi071.stdout:1686: done (2 left) 2024-09-11T01:40:53.074 INFO:tasks.rados.rados.0.smithi071.stdout:1687: done (1 left) 2024-09-11T01:40:53.074 INFO:tasks.rados.rados.0.smithi071.stdout:1688: read oid 30 snap -1 2024-09-11T01:40:53.074 INFO:tasks.rados.rados.0.smithi071.stdout:1688: expect (ObjNum 588 snap 169 seq_num 588) 2024-09-11T01:40:53.074 INFO:tasks.rados.rados.0.smithi071.stdout:1689: snap_remove snap 184 2024-09-11T01:40:53.952 INFO:tasks.rados.rados.0.smithi071.stdout:1689: done (2 left) 2024-09-11T01:40:53.952 INFO:tasks.rados.rados.0.smithi071.stdout:1690: read oid 31 snap -1 2024-09-11T01:40:53.952 INFO:tasks.rados.rados.0.smithi071.stdout:1690: expect (ObjNum 619 snap 183 seq_num 619) 2024-09-11T01:40:53.952 INFO:tasks.rados.rados.0.smithi071.stdout:1691: delete oid 48 current snap is 187 2024-09-11T01:40:53.952 INFO:tasks.rados.rados.0.smithi071.stdout:1685: finishing write tid 3 to smithi07134777-27 2024-09-11T01:40:53.952 INFO:tasks.rados.rados.0.smithi071.stdout:1685: finishing write tid 4 to smithi07134777-27 2024-09-11T01:40:53.952 INFO:tasks.rados.rados.0.smithi071.stdout:1685: oid 27 updating version 0 to 852 2024-09-11T01:40:53.952 INFO:tasks.rados.rados.0.smithi071.stdout:1685: oid 27 version 852 is already newer than 851 2024-09-11T01:40:53.952 INFO:tasks.rados.rados.0.smithi071.stdout:1685: oid 27 updating version 852 to 853 2024-09-11T01:40:53.952 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 853 (ObjNum 623 snap 187 seq_num 623) dirty exists 2024-09-11T01:40:53.952 INFO:tasks.rados.rados.0.smithi071.stdout:1685: left oid 27 (ObjNum 623 snap 187 seq_num 623) 2024-09-11T01:40:53.953 INFO:tasks.rados.rados.0.smithi071.stdout:1688: expect (ObjNum 588 snap 169 seq_num 588) 2024-09-11T01:40:53.992 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2491043 2024-09-11T01:40:54.046 INFO:tasks.rados.rados.0.smithi071.stdout:1690: expect (ObjNum 619 snap 183 seq_num 619) 2024-09-11T01:40:54.072 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1640538 2024-09-11T01:40:54.170 INFO:tasks.rados.rados.0.smithi071.stdout:1685: done (3 left) 2024-09-11T01:40:54.170 INFO:tasks.rados.rados.0.smithi071.stdout:1688: done (2 left) 2024-09-11T01:40:54.170 INFO:tasks.rados.rados.0.smithi071.stdout:1690: done (1 left) 2024-09-11T01:40:54.170 INFO:tasks.rados.rados.0.smithi071.stdout:1691: done (0 left) 2024-09-11T01:40:54.171 INFO:tasks.rados.rados.0.smithi071.stdout:1692: snap_create 2024-09-11T01:40:54.987 INFO:tasks.rados.rados.0.smithi071.stdout:1692: done (0 left) 2024-09-11T01:40:54.987 INFO:tasks.rados.rados.0.smithi071.stdout:1693: read oid 35 snap 187 2024-09-11T01:40:54.988 INFO:tasks.rados.rados.0.smithi071.stdout:1693: expect (ObjNum 612 snap 181 seq_num 612) 2024-09-11T01:40:54.988 INFO:tasks.rados.rados.0.smithi071.stdout:1694: snap_create 2024-09-11T01:40:54.994 INFO:tasks.rados.rados.0.smithi071.stdout:1693: expect (ObjNum 612 snap 181 seq_num 612) 2024-09-11T01:40:54.994 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:40:55.050 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2220553 2024-09-11T01:40:55.986 INFO:tasks.rados.rados.0.smithi071.stdout:1693: done (1 left) 2024-09-11T01:40:55.986 INFO:tasks.rados.rados.0.smithi071.stdout:1694: done (0 left) 2024-09-11T01:40:55.986 INFO:tasks.rados.rados.0.smithi071.stdout:1695: read oid 25 snap -1 2024-09-11T01:40:55.986 INFO:tasks.rados.rados.0.smithi071.stdout:1695: expect (ObjNum 488 snap 132 seq_num 488) 2024-09-11T01:40:55.987 INFO:tasks.rados.rados.0.smithi071.stdout:1696: copy_from oid 40 from oid 36 current snap is 189 2024-09-11T01:40:55.987 INFO:tasks.rados.rados.0.smithi071.stdout:1697: write oid 50 current snap is 189 2024-09-11T01:40:55.987 INFO:tasks.rados.rados.0.smithi071.stdout:1697: seq_num 626 ranges {521182=646820,1948669=523219} 2024-09-11T01:40:55.992 INFO:tasks.rados.rados.0.smithi071.stdout:1697: writing smithi07134777-50 from 521182 to 1168002 tid 1 2024-09-11T01:40:56.000 INFO:tasks.rados.rados.0.smithi071.stdout:1697: writing smithi07134777-50 from 1948669 to 2471888 tid 2 2024-09-11T01:40:56.000 INFO:tasks.rados.rados.0.smithi071.stdout:1698: write oid 23 current snap is 189 2024-09-11T01:40:56.000 INFO:tasks.rados.rados.0.smithi071.stdout:1698: seq_num 627 ranges {537989=722069,1890798=647191} 2024-09-11T01:40:56.007 INFO:tasks.rados.rados.0.smithi071.stdout:1698: writing smithi07134777-23 from 537989 to 1260058 tid 1 2024-09-11T01:40:56.014 INFO:tasks.rados.rados.0.smithi071.stdout:1698: writing smithi07134777-23 from 1890798 to 2537989 tid 2 2024-09-11T01:40:56.015 INFO:tasks.rados.rados.0.smithi071.stdout:1699: read oid 8 snap -1 2024-09-11T01:40:56.015 INFO:tasks.rados.rados.0.smithi071.stdout:1699: expect deleted 2024-09-11T01:40:56.015 INFO:tasks.rados.rados.0.smithi071.stdout:1700: delete oid 43 current snap is 189 2024-09-11T01:40:56.015 INFO:tasks.rados.rados.0.smithi071.stdout:1695: expect (ObjNum 488 snap 132 seq_num 488) 2024-09-11T01:40:56.102 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 647320 2024-09-11T01:40:56.133 INFO:tasks.rados.rados.0.smithi071.stdout:1696: finishing copy_from to smithi07134777-40 2024-09-11T01:40:56.133 INFO:tasks.rados.rados.0.smithi071.stdout:1696: got expected ENOENT (src dne) 2024-09-11T01:40:56.133 INFO:tasks.rados.rados.0.smithi071.stdout:1696: finishing copy_from racing read to smithi07134777-40 2024-09-11T01:40:56.133 INFO:tasks.rados.rados.0.smithi071.stdout:1697: finishing write tid 1 to smithi07134777-50 2024-09-11T01:40:56.133 INFO:tasks.rados.rados.0.smithi071.stdout:1697: finishing write tid 2 to smithi07134777-50 2024-09-11T01:40:56.133 INFO:tasks.rados.rados.0.smithi071.stdout:1697: finishing write tid 3 to smithi07134777-50 2024-09-11T01:40:56.134 INFO:tasks.rados.rados.0.smithi071.stdout:1697: finishing write tid 4 to smithi07134777-50 2024-09-11T01:40:56.134 INFO:tasks.rados.rados.0.smithi071.stdout:1697: oid 50 updating version 0 to 950 2024-09-11T01:40:56.134 INFO:tasks.rados.rados.0.smithi071.stdout:1697: oid 50 version 950 is already newer than 948 2024-09-11T01:40:56.134 INFO:tasks.rados.rados.0.smithi071.stdout:1697: oid 50 version 950 is already newer than 949 2024-09-11T01:40:56.134 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 950 (ObjNum 626 snap 189 seq_num 626) dirty exists 2024-09-11T01:40:56.134 INFO:tasks.rados.rados.0.smithi071.stdout:1697: left oid 50 (ObjNum 626 snap 189 seq_num 626) 2024-09-11T01:40:56.134 INFO:tasks.rados.rados.0.smithi071.stdout:1695: done (5 left) 2024-09-11T01:40:56.134 INFO:tasks.rados.rados.0.smithi071.stdout:1696: done (4 left) 2024-09-11T01:40:56.134 INFO:tasks.rados.rados.0.smithi071.stdout:1697: done (3 left) 2024-09-11T01:40:56.134 INFO:tasks.rados.rados.0.smithi071.stdout:1700: done (2 left) 2024-09-11T01:40:56.134 INFO:tasks.rados.rados.0.smithi071.stdout:1701: copy_from oid 27 from oid 4 current snap is 189 2024-09-11T01:40:56.134 INFO:tasks.rados.rados.0.smithi071.stdout:1698: finishing write tid 1 to smithi07134777-23 2024-09-11T01:40:56.134 INFO:tasks.rados.rados.0.smithi071.stdout:1698: finishing write tid 2 to smithi07134777-23 2024-09-11T01:40:56.134 INFO:tasks.rados.rados.0.smithi071.stdout:1698: finishing write tid 3 to smithi07134777-23 2024-09-11T01:40:56.135 INFO:tasks.rados.rados.0.smithi071.stdout:1698: finishing write tid 4 to smithi07134777-23 2024-09-11T01:40:56.135 INFO:tasks.rados.rados.0.smithi071.stdout:1698: oid 23 updating version 0 to 297 2024-09-11T01:40:56.135 INFO:tasks.rados.rados.0.smithi071.stdout:1698: oid 23 updating version 297 to 299 2024-09-11T01:40:56.135 INFO:tasks.rados.rados.0.smithi071.stdout:1698: oid 23 version 299 is already newer than 298 2024-09-11T01:40:56.135 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 299 (ObjNum 627 snap 189 seq_num 627) dirty exists 2024-09-11T01:40:56.135 INFO:tasks.rados.rados.0.smithi071.stdout:1698: left oid 23 (ObjNum 627 snap 189 seq_num 627) 2024-09-11T01:40:56.135 INFO:tasks.rados.rados.0.smithi071.stdout:1698: done (2 left) 2024-09-11T01:40:56.135 INFO:tasks.rados.rados.0.smithi071.stdout:1699: done (1 left) 2024-09-11T01:40:56.135 INFO:tasks.rados.rados.0.smithi071.stdout:1702: read oid 16 snap -1 2024-09-11T01:40:56.135 INFO:tasks.rados.rados.0.smithi071.stdout:1702: expect (ObjNum 411 snap 115 seq_num 411) 2024-09-11T01:40:56.135 INFO:tasks.rados.rados.0.smithi071.stdout:1703: snap_create 2024-09-11T01:40:56.135 INFO:tasks.rados.rados.0.smithi071.stdout:1701: finishing copy_from to smithi07134777-27 2024-09-11T01:40:56.135 INFO:tasks.rados.rados.0.smithi071.stdout:1701: got expected ENOENT (src dne) 2024-09-11T01:40:56.135 INFO:tasks.rados.rados.0.smithi071.stdout:1701: finishing copy_from racing read to smithi07134777-27 2024-09-11T01:40:56.144 INFO:tasks.rados.rados.0.smithi071.stdout:1702: expect (ObjNum 411 snap 115 seq_num 411) 2024-09-11T01:40:56.193 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:40:56.244 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:40:56.988 INFO:tasks.rados.rados.0.smithi071.stdout:1701: done (2 left) 2024-09-11T01:40:56.988 INFO:tasks.rados.rados.0.smithi071.stdout:1702: done (1 left) 2024-09-11T01:40:56.988 INFO:tasks.rados.rados.0.smithi071.stdout:1703: done (0 left) 2024-09-11T01:40:56.988 INFO:tasks.rados.rados.0.smithi071.stdout:1704: read oid 31 snap 181 2024-09-11T01:40:56.988 INFO:tasks.rados.rados.0.smithi071.stdout:1704: expect (ObjNum 608 snap 176 seq_num 608) 2024-09-11T01:40:56.988 INFO:tasks.rados.rados.0.smithi071.stdout:1705: write oid 4 current snap is 190 2024-09-11T01:40:56.988 INFO:tasks.rados.rados.0.smithi071.stdout:1705: seq_num 629 ranges {571603=472567,1607762=670546} 2024-09-11T01:40:56.993 INFO:tasks.rados.rados.0.smithi071.stdout:1705: writing smithi07134777-4 from 571603 to 1044170 tid 1 2024-09-11T01:40:57.000 INFO:tasks.rados.rados.0.smithi071.stdout:1705: writing smithi07134777-4 from 1607762 to 2278308 tid 2 2024-09-11T01:40:57.000 INFO:tasks.rados.rados.0.smithi071.stdout:1706: snap_remove snap 172 2024-09-11T01:40:57.995 INFO:tasks.rados.rados.0.smithi071.stdout:1706: done (2 left) 2024-09-11T01:40:57.996 INFO:tasks.rados.rados.0.smithi071.stdout:1707: copy_from oid 32 from oid 29 current snap is 190 2024-09-11T01:40:57.996 INFO:tasks.rados.rados.0.smithi071.stdout:1704: expect (ObjNum 608 snap 176 seq_num 608) 2024-09-11T01:40:58.050 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:40:58.152 INFO:tasks.rados.rados.0.smithi071.stdout:1705: finishing write tid 1 to smithi07134777-4 2024-09-11T01:40:58.152 INFO:tasks.rados.rados.0.smithi071.stdout:1705: finishing write tid 2 to smithi07134777-4 2024-09-11T01:40:58.152 INFO:tasks.rados.rados.0.smithi071.stdout:1705: finishing write tid 3 to smithi07134777-4 2024-09-11T01:40:58.152 INFO:tasks.rados.rados.0.smithi071.stdout:1705: finishing write tid 4 to smithi07134777-4 2024-09-11T01:40:58.152 INFO:tasks.rados.rados.0.smithi071.stdout:1705: oid 4 updating version 0 to 651 2024-09-11T01:40:58.152 INFO:tasks.rados.rados.0.smithi071.stdout:1705: oid 4 updating version 651 to 652 2024-09-11T01:40:58.153 INFO:tasks.rados.rados.0.smithi071.stdout:1705: oid 4 version 652 is already newer than 650 2024-09-11T01:40:58.153 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 652 (ObjNum 629 snap 190 seq_num 629) dirty exists 2024-09-11T01:40:58.153 INFO:tasks.rados.rados.0.smithi071.stdout:1705: left oid 4 (ObjNum 629 snap 190 seq_num 629) 2024-09-11T01:40:58.153 INFO:tasks.rados.rados.0.smithi071.stdout:1704: done (2 left) 2024-09-11T01:40:58.153 INFO:tasks.rados.rados.0.smithi071.stdout:1705: done (1 left) 2024-09-11T01:40:58.153 INFO:tasks.rados.rados.0.smithi071.stdout:1708: read oid 18 snap -1 2024-09-11T01:40:58.153 INFO:tasks.rados.rados.0.smithi071.stdout:1708: expect (ObjNum 618 snap 183 seq_num 618) 2024-09-11T01:40:58.153 INFO:tasks.rados.rados.0.smithi071.stdout:1709: delete oid 42 current snap is 190 2024-09-11T01:40:58.153 INFO:tasks.rados.rados.0.smithi071.stdout:1707: finishing copy_from racing read to smithi07134777-32 2024-09-11T01:40:58.153 INFO:tasks.rados.rados.0.smithi071.stdout:1707: finishing copy_from to smithi07134777-32 2024-09-11T01:40:58.153 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 938 (ObjNum 617 snap 183 seq_num 617) dirty exists 2024-09-11T01:40:58.155 INFO:tasks.rados.rados.0.smithi071.stdout:1707: done (2 left) 2024-09-11T01:40:58.155 INFO:tasks.rados.rados.0.smithi071.stdout:1709: done (1 left) 2024-09-11T01:40:58.155 INFO:tasks.rados.rados.0.smithi071.stdout:1710: copy_from oid 36 from oid 35 current snap is 190 2024-09-11T01:40:58.155 INFO:tasks.rados.rados.0.smithi071.stdout:1711: write oid 49 current snap is 190 2024-09-11T01:40:58.155 INFO:tasks.rados.rados.0.smithi071.stdout:1711: seq_num 631 ranges {605217=739418,2124726=480491} 2024-09-11T01:40:58.162 INFO:tasks.rados.rados.0.smithi071.stdout:1711: writing smithi07134777-49 from 605217 to 1344635 tid 1 2024-09-11T01:40:58.169 INFO:tasks.rados.rados.0.smithi071.stdout:1711: writing smithi07134777-49 from 2124726 to 2605217 tid 2 2024-09-11T01:40:58.170 INFO:tasks.rados.rados.0.smithi071.stdout:1712: read oid 21 snap 175 2024-09-11T01:40:58.170 INFO:tasks.rados.rados.0.smithi071.stdout:1712: expect (ObjNum 542 snap 154 seq_num 542) 2024-09-11T01:40:58.170 INFO:tasks.rados.rados.0.smithi071.stdout:1713: rollback oid 50 current snap is 190 2024-09-11T01:40:58.170 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 50 to 182 2024-09-11T01:40:58.170 INFO:tasks.rados.rados.0.smithi071.stdout:1708: expect (ObjNum 618 snap 183 seq_num 618) 2024-09-11T01:40:58.170 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 38419 2024-09-11T01:40:58.338 INFO:tasks.rados.rados.0.smithi071.stdout:1710: finishing copy_from racing read to smithi07134777-36 2024-09-11T01:40:58.339 INFO:tasks.rados.rados.0.smithi071.stdout:1710: finishing copy_from to smithi07134777-36 2024-09-11T01:40:58.339 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 900 (ObjNum 612 snap 181 seq_num 612) dirty exists 2024-09-11T01:40:58.339 INFO:tasks.rados.rados.0.smithi071.stdout:1711: finishing write tid 1 to smithi07134777-49 2024-09-11T01:40:58.339 INFO:tasks.rados.rados.0.smithi071.stdout:1713: finishing rollback tid 0 to smithi07134777-50 2024-09-11T01:40:58.339 INFO:tasks.rados.rados.0.smithi071.stdout:1713: finishing rollback tid 1 to smithi07134777-50 2024-09-11T01:40:58.339 INFO:tasks.rados.rados.0.smithi071.stdout:1713: finishing rollback tid 2 to smithi07134777-50 2024-09-11T01:40:58.339 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 966 (ObjNum 551 snap 157 seq_num 551) dirty exists 2024-09-11T01:40:58.339 INFO:tasks.rados.rados.0.smithi071.stdout:1711: finishing write tid 2 to smithi07134777-49 2024-09-11T01:40:58.339 INFO:tasks.rados.rados.0.smithi071.stdout:1708: done (4 left) 2024-09-11T01:40:58.339 INFO:tasks.rados.rados.0.smithi071.stdout:1710: done (3 left) 2024-09-11T01:40:58.340 INFO:tasks.rados.rados.0.smithi071.stdout:1713: done (2 left) 2024-09-11T01:40:58.340 INFO:tasks.rados.rados.0.smithi071.stdout:1714: snap_remove snap 173 2024-09-11T01:40:58.951 INFO:tasks.rados.rados.0.smithi071.stdout:1714: done (2 left) 2024-09-11T01:40:58.951 INFO:tasks.rados.rados.0.smithi071.stdout:1715: write (excl) oid 13 current snap is 190 2024-09-11T01:40:58.951 INFO:tasks.rados.rados.0.smithi071.stdout:1711: finishing write tid 3 to smithi07134777-49 2024-09-11T01:40:58.951 INFO:tasks.rados.rados.0.smithi071.stdout:1712: expect (ObjNum 542 snap 154 seq_num 542) 2024-09-11T01:40:58.951 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:40:59.009 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 18402 2024-09-11T01:40:59.064 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:40:59.119 INFO:tasks.rados.rados.0.smithi071.stdout:1711: finishing write tid 4 to smithi07134777-49 2024-09-11T01:40:59.120 INFO:tasks.rados.rados.0.smithi071.stdout:1711: oid 49 updating version 0 to 359 2024-09-11T01:40:59.120 INFO:tasks.rados.rados.0.smithi071.stdout:1711: oid 49 version 359 is already newer than 358 2024-09-11T01:40:59.120 INFO:tasks.rados.rados.0.smithi071.stdout:1711: oid 49 version 359 is already newer than 357 2024-09-11T01:40:59.120 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 359 (ObjNum 631 snap 190 seq_num 631) dirty exists 2024-09-11T01:40:59.120 INFO:tasks.rados.rados.0.smithi071.stdout:1711: left oid 49 (ObjNum 631 snap 190 seq_num 631) 2024-09-11T01:40:59.120 INFO:tasks.rados.rados.0.smithi071.stdout:1715: seq_num 632 ranges {622024=414667,1783208=417873} 2024-09-11T01:40:59.125 INFO:tasks.rados.rados.0.smithi071.stdout:1715: writing smithi07134777-13 from 622024 to 1036691 tid 1 2024-09-11T01:40:59.131 INFO:tasks.rados.rados.0.smithi071.stdout:1715: writing smithi07134777-13 from 1783208 to 2201081 tid 2 2024-09-11T01:40:59.131 INFO:tasks.rados.rados.0.smithi071.stdout:1711: done (2 left) 2024-09-11T01:40:59.132 INFO:tasks.rados.rados.0.smithi071.stdout:1712: done (1 left) 2024-09-11T01:40:59.132 INFO:tasks.rados.rados.0.smithi071.stdout:1716: write oid 43 current snap is 190 2024-09-11T01:40:59.132 INFO:tasks.rados.rados.0.smithi071.stdout:1716: seq_num 633 ranges {638831=489916,1841690=677884} 2024-09-11T01:40:59.137 INFO:tasks.rados.rados.0.smithi071.stdout:1716: writing smithi07134777-43 from 638831 to 1128747 tid 1 2024-09-11T01:40:59.146 INFO:tasks.rados.rados.0.smithi071.stdout:1716: writing smithi07134777-43 from 1841690 to 2519574 tid 2 2024-09-11T01:40:59.146 INFO:tasks.rados.rados.0.smithi071.stdout:1717: rollback oid 23 current snap is 190 2024-09-11T01:40:59.146 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 23 to 189 2024-09-11T01:40:59.147 INFO:tasks.rados.rados.0.smithi071.stdout:1715: finishing write tid 1 to smithi07134777-13 2024-09-11T01:40:59.147 INFO:tasks.rados.rados.0.smithi071.stdout:1715: finishing write tid 2 to smithi07134777-13 2024-09-11T01:40:59.147 INFO:tasks.rados.rados.0.smithi071.stdout:1715: finishing write tid 3 to smithi07134777-13 2024-09-11T01:40:59.147 INFO:tasks.rados.rados.0.smithi071.stdout:1715: finishing write tid 4 to smithi07134777-13 2024-09-11T01:40:59.147 INFO:tasks.rados.rados.0.smithi071.stdout:1715: oid 13 updating version 0 to 809 2024-09-11T01:40:59.147 INFO:tasks.rados.rados.0.smithi071.stdout:1715: oid 13 updating version 809 to 810 2024-09-11T01:40:59.147 INFO:tasks.rados.rados.0.smithi071.stdout:1715: oid 13 version 810 is already newer than 808 2024-09-11T01:40:59.147 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 810 (ObjNum 632 snap 190 seq_num 632) dirty exists 2024-09-11T01:40:59.147 INFO:tasks.rados.rados.0.smithi071.stdout:1715: left oid 13 (ObjNum 632 snap 190 seq_num 632) 2024-09-11T01:40:59.147 INFO:tasks.rados.rados.0.smithi071.stdout:1716: finishing write tid 1 to smithi07134777-43 2024-09-11T01:40:59.147 INFO:tasks.rados.rados.0.smithi071.stdout:1715: done (2 left) 2024-09-11T01:40:59.147 INFO:tasks.rados.rados.0.smithi071.stdout:1718: copy_from oid 50 from oid 41 current snap is 190 2024-09-11T01:40:59.147 INFO:tasks.rados.rados.0.smithi071.stdout:1719: read oid 3 snap -1 2024-09-11T01:40:59.148 INFO:tasks.rados.rados.0.smithi071.stdout:1719: expect (ObjNum 515 snap 141 seq_num 515) 2024-09-11T01:40:59.148 INFO:tasks.rados.rados.0.smithi071.stdout:1720: read oid 40 snap -1 2024-09-11T01:40:59.148 INFO:tasks.rados.rados.0.smithi071.stdout:1720: expect deleted 2024-09-11T01:40:59.148 INFO:tasks.rados.rados.0.smithi071.stdout:1721: snap_create 2024-09-11T01:40:59.148 INFO:tasks.rados.rados.0.smithi071.stdout:1718: finishing copy_from to smithi07134777-50 2024-09-11T01:40:59.148 INFO:tasks.rados.rados.0.smithi071.stdout:1718: got expected ENOENT (src dne) 2024-09-11T01:40:59.148 INFO:tasks.rados.rados.0.smithi071.stdout:1718: finishing copy_from racing read to smithi07134777-50 2024-09-11T01:40:59.150 INFO:tasks.rados.rados.0.smithi071.stdout:1716: finishing write tid 2 to smithi07134777-43 2024-09-11T01:40:59.150 INFO:tasks.rados.rados.0.smithi071.stdout:1716: finishing write tid 3 to smithi07134777-43 2024-09-11T01:40:59.150 INFO:tasks.rados.rados.0.smithi071.stdout:1716: finishing write tid 4 to smithi07134777-43 2024-09-11T01:40:59.150 INFO:tasks.rados.rados.0.smithi071.stdout:1716: oid 43 updating version 0 to 857 2024-09-11T01:40:59.150 INFO:tasks.rados.rados.0.smithi071.stdout:1716: oid 43 version 857 is already newer than 855 2024-09-11T01:40:59.151 INFO:tasks.rados.rados.0.smithi071.stdout:1716: oid 43 version 857 is already newer than 856 2024-09-11T01:40:59.151 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 857 (ObjNum 633 snap 190 seq_num 633) dirty exists 2024-09-11T01:40:59.151 INFO:tasks.rados.rados.0.smithi071.stdout:1716: left oid 43 (ObjNum 633 snap 190 seq_num 633) 2024-09-11T01:40:59.151 INFO:tasks.rados.rados.0.smithi071.stdout:1719: expect (ObjNum 515 snap 141 seq_num 515) 2024-09-11T01:40:59.151 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:40:59.179 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 368926 2024-09-11T01:40:59.213 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1671381 2024-09-11T01:40:59.223 INFO:tasks.rados.rados.0.smithi071.stdout:1717: finishing rollback tid 0 to smithi07134777-23 2024-09-11T01:40:59.223 INFO:tasks.rados.rados.0.smithi071.stdout:1717: finishing rollback tid 1 to smithi07134777-23 2024-09-11T01:40:59.223 INFO:tasks.rados.rados.0.smithi071.stdout:1717: finishing rollback tid 2 to smithi07134777-23 2024-09-11T01:40:59.223 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 307 (ObjNum 627 snap 189 seq_num 627) dirty exists 2024-09-11T01:41:00.007 INFO:tasks.rados.rados.0.smithi071.stdout:1716: done (5 left) 2024-09-11T01:41:00.007 INFO:tasks.rados.rados.0.smithi071.stdout:1717: done (4 left) 2024-09-11T01:41:00.007 INFO:tasks.rados.rados.0.smithi071.stdout:1718: done (3 left) 2024-09-11T01:41:00.007 INFO:tasks.rados.rados.0.smithi071.stdout:1719: done (2 left) 2024-09-11T01:41:00.007 INFO:tasks.rados.rados.0.smithi071.stdout:1720: done (1 left) 2024-09-11T01:41:00.007 INFO:tasks.rados.rados.0.smithi071.stdout:1721: done (0 left) 2024-09-11T01:41:00.007 INFO:tasks.rados.rados.0.smithi071.stdout:1722: rollback oid 21 current snap is 191 2024-09-11T01:41:00.007 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 21 to 187 2024-09-11T01:41:00.007 INFO:tasks.rados.rados.0.smithi071.stdout:1723: rollback oid 47 current snap is 191 2024-09-11T01:41:00.008 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 47 to 183 2024-09-11T01:41:00.008 INFO:tasks.rados.rados.0.smithi071.stdout:1724: snap_remove snap 189 2024-09-11T01:41:01.011 INFO:tasks.rados.rados.0.smithi071.stdout:1724: done (2 left) 2024-09-11T01:41:01.011 INFO:tasks.rados.rados.0.smithi071.stdout:1725: write oid 41 current snap is 191 2024-09-11T01:41:01.011 INFO:tasks.rados.rados.0.smithi071.stdout:1725: seq_num 634 ranges {655638=565165,1900172=421542} 2024-09-11T01:41:01.017 INFO:tasks.rados.rados.0.smithi071.stdout:1725: writing smithi07134777-41 from 655638 to 1220803 tid 1 2024-09-11T01:41:01.025 INFO:tasks.rados.rados.0.smithi071.stdout:1725: writing smithi07134777-41 from 1900172 to 2321714 tid 2 2024-09-11T01:41:01.025 INFO:tasks.rados.rados.0.smithi071.stdout:1726: snap_create 2024-09-11T01:41:01.025 INFO:tasks.rados.rados.0.smithi071.stdout:1723: finishing rollback tid 0 to smithi07134777-47 2024-09-11T01:41:01.025 INFO:tasks.rados.rados.0.smithi071.stdout:1722: finishing rollback tid 0 to smithi07134777-21 2024-09-11T01:41:01.025 INFO:tasks.rados.rados.0.smithi071.stdout:1723: finishing rollback tid 1 to smithi07134777-47 2024-09-11T01:41:01.025 INFO:tasks.rados.rados.0.smithi071.stdout:1723: finishing rollback tid 2 to smithi07134777-47 2024-09-11T01:41:01.025 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 813 (ObjNum 599 snap 171 seq_num 599) dirty exists 2024-09-11T01:41:01.025 INFO:tasks.rados.rados.0.smithi071.stdout:1722: finishing rollback tid 1 to smithi07134777-21 2024-09-11T01:41:01.025 INFO:tasks.rados.rados.0.smithi071.stdout:1722: finishing rollback tid 2 to smithi07134777-21 2024-09-11T01:41:01.025 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 903 (ObjNum 542 snap 154 seq_num 542) dirty exists 2024-09-11T01:41:01.025 INFO:tasks.rados.rados.0.smithi071.stdout:1725: finishing write tid 1 to smithi07134777-41 2024-09-11T01:41:01.026 INFO:tasks.rados.rados.0.smithi071.stdout:1725: finishing write tid 2 to smithi07134777-41 2024-09-11T01:41:01.029 INFO:tasks.rados.rados.0.smithi071.stdout:1725: finishing write tid 3 to smithi07134777-41 2024-09-11T01:41:01.029 INFO:tasks.rados.rados.0.smithi071.stdout:1725: finishing write tid 4 to smithi07134777-41 2024-09-11T01:41:01.029 INFO:tasks.rados.rados.0.smithi071.stdout:1725: oid 41 updating version 0 to 382 2024-09-11T01:41:01.030 INFO:tasks.rados.rados.0.smithi071.stdout:1725: oid 41 version 382 is already newer than 381 2024-09-11T01:41:01.030 INFO:tasks.rados.rados.0.smithi071.stdout:1725: oid 41 version 382 is already newer than 380 2024-09-11T01:41:01.030 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 382 (ObjNum 634 snap 191 seq_num 634) dirty exists 2024-09-11T01:41:01.030 INFO:tasks.rados.rados.0.smithi071.stdout:1725: left oid 41 (ObjNum 634 snap 191 seq_num 634) 2024-09-11T01:41:02.010 INFO:tasks.rados.rados.0.smithi071.stdout:1722: done (3 left) 2024-09-11T01:41:02.010 INFO:tasks.rados.rados.0.smithi071.stdout:1723: done (2 left) 2024-09-11T01:41:02.010 INFO:tasks.rados.rados.0.smithi071.stdout:1725: done (1 left) 2024-09-11T01:41:02.010 INFO:tasks.rados.rados.0.smithi071.stdout:1726: done (0 left) 2024-09-11T01:41:02.010 INFO:tasks.rados.rados.0.smithi071.stdout:1727: rollback oid 42 current snap is 192 2024-09-11T01:41:02.011 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 42 to 187 2024-09-11T01:41:02.011 INFO:tasks.rados.rados.0.smithi071.stdout:1728: write (excl) oid 9 current snap is 192 2024-09-11T01:41:02.011 INFO:tasks.rados.rados.0.smithi071.stdout:1728: seq_num 635 ranges {672445=640414,1842301=681553} 2024-09-11T01:41:02.017 INFO:tasks.rados.rados.0.smithi071.stdout:1728: writing smithi07134777-9 from 672445 to 1312859 tid 1 2024-09-11T01:41:02.024 INFO:tasks.rados.rados.0.smithi071.stdout:1728: writing smithi07134777-9 from 1842301 to 2523854 tid 2 2024-09-11T01:41:02.024 INFO:tasks.rados.rados.0.smithi071.stdout:1729: read oid 10 snap -1 2024-09-11T01:41:02.025 INFO:tasks.rados.rados.0.smithi071.stdout:1729: expect (ObjNum 601 snap 172 seq_num 601) 2024-09-11T01:41:02.025 INFO:tasks.rados.rados.0.smithi071.stdout:1730: read oid 32 snap -1 2024-09-11T01:41:02.025 INFO:tasks.rados.rados.0.smithi071.stdout:1730: expect (ObjNum 617 snap 183 seq_num 617) 2024-09-11T01:41:02.025 INFO:tasks.rados.rados.0.smithi071.stdout:1731: copy_from oid 35 from oid 18 current snap is 192 2024-09-11T01:41:02.025 INFO:tasks.rados.rados.0.smithi071.stdout:1727: finishing rollback tid 1 to smithi07134777-42 2024-09-11T01:41:02.025 INFO:tasks.rados.rados.0.smithi071.stdout:1727: finishing rollback tid 2 to smithi07134777-42 2024-09-11T01:41:02.025 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 812 (ObjNum 619 snap 183 seq_num 619) dirty exists 2024-09-11T01:41:02.025 INFO:tasks.rados.rados.0.smithi071.stdout:1728: finishing write tid 1 to smithi07134777-9 2024-09-11T01:41:02.025 INFO:tasks.rados.rados.0.smithi071.stdout:1727: done (4 left) 2024-09-11T01:41:02.025 INFO:tasks.rados.rados.0.smithi071.stdout:1732: snap_create 2024-09-11T01:41:02.028 INFO:tasks.rados.rados.0.smithi071.stdout:1728: finishing write tid 2 to smithi07134777-9 2024-09-11T01:41:02.029 INFO:tasks.rados.rados.0.smithi071.stdout:1728: finishing write tid 3 to smithi07134777-9 2024-09-11T01:41:02.029 INFO:tasks.rados.rados.0.smithi071.stdout:1728: finishing write tid 4 to smithi07134777-9 2024-09-11T01:41:02.029 INFO:tasks.rados.rados.0.smithi071.stdout:1728: oid 9 updating version 0 to 530 2024-09-11T01:41:02.029 INFO:tasks.rados.rados.0.smithi071.stdout:1728: oid 9 updating version 530 to 532 2024-09-11T01:41:02.029 INFO:tasks.rados.rados.0.smithi071.stdout:1728: oid 9 version 532 is already newer than 531 2024-09-11T01:41:02.029 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 532 (ObjNum 635 snap 192 seq_num 635) dirty exists 2024-09-11T01:41:02.029 INFO:tasks.rados.rados.0.smithi071.stdout:1728: left oid 9 (ObjNum 635 snap 192 seq_num 635) 2024-09-11T01:41:02.029 INFO:tasks.rados.rados.0.smithi071.stdout:1729: expect (ObjNum 601 snap 172 seq_num 601) 2024-09-11T01:41:02.057 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:41:02.099 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1833271 2024-09-11T01:41:02.101 INFO:tasks.rados.rados.0.smithi071.stdout:1730: expect (ObjNum 617 snap 183 seq_num 617) 2024-09-11T01:41:02.120 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 929283 2024-09-11T01:41:02.180 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1460607 2024-09-11T01:41:02.255 INFO:tasks.rados.rados.0.smithi071.stdout:1731: finishing copy_from racing read to smithi07134777-35 2024-09-11T01:41:02.255 INFO:tasks.rados.rados.0.smithi071.stdout:1731: finishing copy_from to smithi07134777-35 2024-09-11T01:41:02.255 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 649 (ObjNum 618 snap 183 seq_num 618) dirty exists 2024-09-11T01:41:03.020 INFO:tasks.rados.rados.0.smithi071.stdout:1728: done (4 left) 2024-09-11T01:41:03.020 INFO:tasks.rados.rados.0.smithi071.stdout:1729: done (3 left) 2024-09-11T01:41:03.020 INFO:tasks.rados.rados.0.smithi071.stdout:1730: done (2 left) 2024-09-11T01:41:03.020 INFO:tasks.rados.rados.0.smithi071.stdout:1731: done (1 left) 2024-09-11T01:41:03.020 INFO:tasks.rados.rados.0.smithi071.stdout:1732: done (0 left) 2024-09-11T01:41:03.020 INFO:tasks.rados.rados.0.smithi071.stdout:1733: snap_remove snap 188 2024-09-11T01:41:03.952 INFO:tasks.rados.rados.0.smithi071.stdout:1733: done (0 left) 2024-09-11T01:41:03.952 INFO:tasks.rados.rados.0.smithi071.stdout:1734: rollback oid 2 current snap is 193 2024-09-11T01:41:03.952 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 2 to 185 2024-09-11T01:41:03.952 INFO:tasks.rados.rados.0.smithi071.stdout:1735: write oid 28 current snap is 193 2024-09-11T01:41:03.952 INFO:tasks.rados.rados.0.smithi071.stdout:1735: seq_num 636 ranges {689252=715663,1900783=425211} 2024-09-11T01:41:03.960 INFO:tasks.rados.rados.0.smithi071.stdout:1735: writing smithi07134777-28 from 689252 to 1404915 tid 1 2024-09-11T01:41:03.966 INFO:tasks.rados.rados.0.smithi071.stdout:1735: writing smithi07134777-28 from 1900783 to 2325994 tid 2 2024-09-11T01:41:03.966 INFO:tasks.rados.rados.0.smithi071.stdout:1736: read oid 9 snap -1 2024-09-11T01:41:03.966 INFO:tasks.rados.rados.0.smithi071.stdout:1736: expect (ObjNum 635 snap 192 seq_num 635) 2024-09-11T01:41:03.966 INFO:tasks.rados.rados.0.smithi071.stdout:1737: delete oid 22 current snap is 193 2024-09-11T01:41:03.966 INFO:tasks.rados.rados.0.smithi071.stdout:1734: finishing rollback tid 0 to smithi07134777-2 2024-09-11T01:41:03.966 INFO:tasks.rados.rados.0.smithi071.stdout:1734: finishing rollback tid 1 to smithi07134777-2 2024-09-11T01:41:03.966 INFO:tasks.rados.rados.0.smithi071.stdout:1734: finishing rollback tid 2 to smithi07134777-2 2024-09-11T01:41:03.966 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 815 (ObjNum 611 snap 181 seq_num 611) dirty exists 2024-09-11T01:41:03.969 INFO:tasks.rados.rados.0.smithi071.stdout:1735: finishing write tid 1 to smithi07134777-28 2024-09-11T01:41:03.969 INFO:tasks.rados.rados.0.smithi071.stdout:1735: finishing write tid 2 to smithi07134777-28 2024-09-11T01:41:03.969 INFO:tasks.rados.rados.0.smithi071.stdout:1735: finishing write tid 3 to smithi07134777-28 2024-09-11T01:41:03.969 INFO:tasks.rados.rados.0.smithi071.stdout:1735: finishing write tid 4 to smithi07134777-28 2024-09-11T01:41:03.969 INFO:tasks.rados.rados.0.smithi071.stdout:1735: oid 28 updating version 0 to 905 2024-09-11T01:41:03.969 INFO:tasks.rados.rados.0.smithi071.stdout:1735: oid 28 updating version 905 to 906 2024-09-11T01:41:03.969 INFO:tasks.rados.rados.0.smithi071.stdout:1735: oid 28 version 906 is already newer than 904 2024-09-11T01:41:03.969 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 906 (ObjNum 636 snap 193 seq_num 636) dirty exists 2024-09-11T01:41:03.969 INFO:tasks.rados.rados.0.smithi071.stdout:1735: left oid 28 (ObjNum 636 snap 193 seq_num 636) 2024-09-11T01:41:03.973 INFO:tasks.rados.rados.0.smithi071.stdout:1734: done (3 left) 2024-09-11T01:41:03.974 INFO:tasks.rados.rados.0.smithi071.stdout:1735: done (2 left) 2024-09-11T01:41:03.974 INFO:tasks.rados.rados.0.smithi071.stdout:1737: done (1 left) 2024-09-11T01:41:03.974 INFO:tasks.rados.rados.0.smithi071.stdout:1738: read oid 17 snap -1 2024-09-11T01:41:03.974 INFO:tasks.rados.rados.0.smithi071.stdout:1738: expect (ObjNum 551 snap 157 seq_num 551) 2024-09-11T01:41:03.974 INFO:tasks.rados.rados.0.smithi071.stdout:1739: copy_from oid 21 from oid 40 current snap is 193 2024-09-11T01:41:03.974 INFO:tasks.rados.rados.0.smithi071.stdout:1740: read oid 45 snap -1 2024-09-11T01:41:03.974 INFO:tasks.rados.rados.0.smithi071.stdout:1740: expect (ObjNum 612 snap 181 seq_num 612) 2024-09-11T01:41:03.974 INFO:tasks.rados.rados.0.smithi071.stdout:1741: read oid 41 snap 171 2024-09-11T01:41:03.974 INFO:tasks.rados.rados.0.smithi071.stdout:1741: expect deleted 2024-09-11T01:41:03.974 INFO:tasks.rados.rados.0.smithi071.stdout:1742: snap_remove snap 182 2024-09-11T01:41:04.959 INFO:tasks.rados.rados.0.smithi071.stdout:1742: done (5 left) 2024-09-11T01:41:04.959 INFO:tasks.rados.rados.0.smithi071.stdout:1743: rollback oid 37 current snap is 193 2024-09-11T01:41:04.959 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 37 to 185 2024-09-11T01:41:04.959 INFO:tasks.rados.rados.0.smithi071.stdout:1738: expect (ObjNum 551 snap 157 seq_num 551) 2024-09-11T01:41:04.997 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:41:05.048 INFO:tasks.rados.rados.0.smithi071.stdout:1736: expect (ObjNum 635 snap 192 seq_num 635) 2024-09-11T01:41:05.070 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:41:05.098 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:41:05.126 INFO:tasks.rados.rados.0.smithi071.stdout:1739: finishing copy_from to smithi07134777-21 2024-09-11T01:41:05.127 INFO:tasks.rados.rados.0.smithi071.stdout:1739: got expected ENOENT (src dne) 2024-09-11T01:41:05.127 INFO:tasks.rados.rados.0.smithi071.stdout:1739: finishing copy_from racing read to smithi07134777-21 2024-09-11T01:41:05.127 INFO:tasks.rados.rados.0.smithi071.stdout:1740: expect (ObjNum 612 snap 181 seq_num 612) 2024-09-11T01:41:05.127 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:41:05.188 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:41:05.224 INFO:tasks.rados.rados.0.smithi071.stdout:1743: finishing rollback tid 0 to smithi07134777-37 2024-09-11T01:41:05.224 INFO:tasks.rados.rados.0.smithi071.stdout:1743: finishing rollback tid 1 to smithi07134777-37 2024-09-11T01:41:05.224 INFO:tasks.rados.rados.0.smithi071.stdout:1743: finishing rollback tid 2 to smithi07134777-37 2024-09-11T01:41:05.224 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 941 (ObjNum 577 snap 165 seq_num 577) dirty exists 2024-09-11T01:41:05.224 INFO:tasks.rados.rados.0.smithi071.stdout:1736: done (5 left) 2024-09-11T01:41:05.224 INFO:tasks.rados.rados.0.smithi071.stdout:1738: done (4 left) 2024-09-11T01:41:05.224 INFO:tasks.rados.rados.0.smithi071.stdout:1739: done (3 left) 2024-09-11T01:41:05.224 INFO:tasks.rados.rados.0.smithi071.stdout:1740: done (2 left) 2024-09-11T01:41:05.224 INFO:tasks.rados.rados.0.smithi071.stdout:1741: done (1 left) 2024-09-11T01:41:05.224 INFO:tasks.rados.rados.0.smithi071.stdout:1743: done (0 left) 2024-09-11T01:41:05.224 INFO:tasks.rados.rados.0.smithi071.stdout:1744: delete oid 3 current snap is 193 2024-09-11T01:41:05.226 INFO:tasks.rados.rados.0.smithi071.stdout:1744: done (0 left) 2024-09-11T01:41:05.226 INFO:tasks.rados.rados.0.smithi071.stdout:1745: snap_create 2024-09-11T01:41:05.963 INFO:tasks.rados.rados.0.smithi071.stdout:1745: done (0 left) 2024-09-11T01:41:05.964 INFO:tasks.rados.rados.0.smithi071.stdout:1746: snap_create 2024-09-11T01:41:06.945 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-11T01:41:06.945 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-11T01:41:06.946 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-11T01:41:06.949 INFO:tasks.ceph.mon.b:Started 2024-09-11T01:41:06.949 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-11T01:41:06.950 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-11T01:41:06.950 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T01:41:21.951 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-11T01:41:21.951 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-11T01:41:21.952 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:41:21.952 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:41:22.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.063+0000 7ff63d893640 10 monclient: get_monmap_and_config 2024-09-11T01:41:22.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.063+0000 7ff63d893640 10 monclient: build_initial_monmap 2024-09-11T01:41:22.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.063+0000 7ff63d893640 10 monclient: monmap: 2024-09-11T01:41:22.065 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:22.065 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:22.066 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:41:22.064674+0000 2024-09-11T01:41:22.066 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:41:22.064674+0000 2024-09-11T01:41:22.066 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:22.066 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:22.066 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:41:22.066 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:41:22.066 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:41:22.066 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:41:22.066 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:41:22.066 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:22.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.064+0000 7ff63d893640 1 Processor -- start 2024-09-11T01:41:22.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.064+0000 7ff63d893640 1 -- start start 2024-09-11T01:41:22.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.064+0000 7ff63d893640 10 monclient: init 2024-09-11T01:41:22.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff63d893640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:22.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff63d893640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-11T01:41:22.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff63d893640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff63809d790 0x7ff63809db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:22.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff63d893640 10 monclient(hunting): picked mon.noname-b con 0x7ff63809d790 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:22.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff63d893640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff63809e0b0 addr v1:172.21.15.71:6789/0 2024-09-11T01:41:22.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff63d893640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff63809e4d0 0x7ff638135170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:22.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff63d893640 10 monclient(hunting): picked mon.noname-c con 0x7ff63809e4d0 addr v2:172.21.15.71:3300/0 2024-09-11T01:41:22.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff63d893640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6380a1da0 con 0x7ff63809e0b0 2024-09-11T01:41:22.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff63d893640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:22.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff63d893640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7ff6380a0760 con 0x7ff63809e4d0 2024-09-11T01:41:22.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff63d893640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:22.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff63d893640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff63809e8f0 con 0x7ff63809d790 2024-09-11T01:41:22.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff63d893640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:22.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff63d893640 10 monclient(hunting): authenticate will time out at 1488.507200s 2024-09-11T01:41:22.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff636ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff63809d790 0x7ff63809db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:22.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff636ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff63809d790 0x7ff63809db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45610/0 (socket says 172.21.15.12:45610) 2024-09-11T01:41:22.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff636ffd640 1 -- 172.21.15.12:0/3255899323 learned_addr learned my addr 172.21.15.12:0/3255899323 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:41:22.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff636ffd640 10 monclient(hunting): get_auth_request con 0x7ff63809d790 auth_method 0 2024-09-11T01:41:22.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff636ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:22.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff636ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:22.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff636ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:22.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff636ffd640 1 -- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff63809d790 msgr2=0x7ff63809db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:41:22.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff636ffd640 1 -- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff63809d790 msgr2=0x7ff63809db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:41:22.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff6377fe640 1 --2- 172.21.15.12:0/3255899323 >> v2:172.21.15.71:3300/0 conn(0x7ff63809e4d0 0x7ff638135170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:22.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff636ffd640 1 --2- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff63809d790 0x7ff63809db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:41:22.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff6377fe640 10 monclient(hunting): get_auth_request con 0x7ff63809e4d0 auth_method 0 2024-09-11T01:41:22.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff6377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:22.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff6377fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:22.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff6377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:22.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.065+0000 7ff636ffd640 1 --2- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff63809d790 0x7ff63809db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:41:22.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.067+0000 7ff62e7fc640 1 -- 172.21.15.12:0/3255899323 >> v1:172.21.15.71:6789/0 conn(0x7ff63809e0b0 legacy=0x7ff638134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:22.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.067+0000 7ff6377fe640 1 -- 172.21.15.12:0/3255899323 >> v2:172.21.15.71:3300/0 conn(0x7ff63809e4d0 msgr2=0x7ff638135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:41:22.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.067+0000 7ff6377fe640 1 -- 172.21.15.12:0/3255899323 >> v2:172.21.15.71:3300/0 conn(0x7ff63809e4d0 msgr2=0x7ff638135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:41:22.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.067+0000 7ff6377fe640 1 --2- 172.21.15.12:0/3255899323 >> v2:172.21.15.71:3300/0 conn(0x7ff63809e4d0 0x7ff638135170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:41:22.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.067+0000 7ff6377fe640 1 --2- 172.21.15.12:0/3255899323 >> v2:172.21.15.71:3300/0 conn(0x7ff63809e4d0 0x7ff638135170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:41:22.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.266+0000 7ff636ffd640 1 --2- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff63809d790 0x7ff63809db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:22.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.266+0000 7ff636ffd640 10 monclient(hunting): get_auth_request con 0x7ff63809d790 auth_method 0 2024-09-11T01:41:22.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.266+0000 7ff636ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:22.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.266+0000 7ff636ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:22.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.266+0000 7ff636ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:41:22.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.266+0000 7ff636ffd640 1 -- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff63809d790 msgr2=0x7ff63809db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:41:22.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.266+0000 7ff636ffd640 1 -- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff63809d790 msgr2=0x7ff63809db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:41:22.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.266+0000 7ff636ffd640 1 --2- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff63809d790 0x7ff63809db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:41:22.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.266+0000 7ff636ffd640 1 --2- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff63809d790 0x7ff63809db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:41:22.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.269+0000 7ff6377fe640 1 --2- 172.21.15.12:0/3255899323 >> v2:172.21.15.71:3300/0 conn(0x7ff63809e4d0 0x7ff638135170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:22.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.269+0000 7ff6377fe640 10 monclient(hunting): get_auth_request con 0x7ff63809e4d0 auth_method 0 2024-09-11T01:41:22.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.269+0000 7ff6377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:22.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.269+0000 7ff6377fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:22.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.269+0000 7ff6377fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:41:22.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.269+0000 7ff6377fe640 1 -- 172.21.15.12:0/3255899323 >> v2:172.21.15.71:3300/0 conn(0x7ff63809e4d0 msgr2=0x7ff638135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:41:22.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.269+0000 7ff6377fe640 1 -- 172.21.15.12:0/3255899323 >> v2:172.21.15.71:3300/0 conn(0x7ff63809e4d0 msgr2=0x7ff638135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:41:22.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.269+0000 7ff6377fe640 1 --2- 172.21.15.12:0/3255899323 >> v2:172.21.15.71:3300/0 conn(0x7ff63809e4d0 0x7ff638135170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:41:22.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.269+0000 7ff6377fe640 1 --2- 172.21.15.12:0/3255899323 >> v2:172.21.15.71:3300/0 conn(0x7ff63809e4d0 0x7ff638135170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:41:22.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.668+0000 7ff636ffd640 1 --2- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff63809d790 0x7ff63809db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:22.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.668+0000 7ff636ffd640 10 monclient(hunting): get_auth_request con 0x7ff63809d790 auth_method 0 2024-09-11T01:41:22.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.668+0000 7ff636ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:22.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.668+0000 7ff636ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:22.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.668+0000 7ff636ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:41:22.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.668+0000 7ff636ffd640 1 -- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff63809d790 msgr2=0x7ff63809db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:41:22.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.668+0000 7ff636ffd640 1 -- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff63809d790 msgr2=0x7ff63809db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:41:22.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.668+0000 7ff636ffd640 1 --2- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff63809d790 0x7ff63809db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:41:22.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.668+0000 7ff636ffd640 1 --2- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff63809d790 0x7ff63809db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:41:22.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.671+0000 7ff6377fe640 1 --2- 172.21.15.12:0/3255899323 >> v2:172.21.15.71:3300/0 conn(0x7ff63809e4d0 0x7ff638135170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:22.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.671+0000 7ff6377fe640 10 monclient(hunting): get_auth_request con 0x7ff63809e4d0 auth_method 0 2024-09-11T01:41:22.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.671+0000 7ff6377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:22.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.671+0000 7ff6377fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:22.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.671+0000 7ff6377fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:41:22.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.671+0000 7ff6377fe640 1 -- 172.21.15.12:0/3255899323 >> v2:172.21.15.71:3300/0 conn(0x7ff63809e4d0 msgr2=0x7ff638135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:41:22.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.671+0000 7ff6377fe640 1 -- 172.21.15.12:0/3255899323 >> v2:172.21.15.71:3300/0 conn(0x7ff63809e4d0 msgr2=0x7ff638135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:41:22.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.671+0000 7ff6377fe640 1 --2- 172.21.15.12:0/3255899323 >> v2:172.21.15.71:3300/0 conn(0x7ff63809e4d0 0x7ff638135170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:41:22.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:22.671+0000 7ff6377fe640 1 --2- 172.21.15.12:0/3255899323 >> v2:172.21.15.71:3300/0 conn(0x7ff63809e4d0 0x7ff638135170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:41:23.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:23.469+0000 7ff636ffd640 1 --2- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff63809d790 0x7ff63809db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:23.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:23.469+0000 7ff636ffd640 10 monclient(hunting): get_auth_request con 0x7ff63809d790 auth_method 0 2024-09-11T01:41:23.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:23.469+0000 7ff636ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:23.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:23.469+0000 7ff636ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:23.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:23.469+0000 7ff636ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:41:23.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:23.469+0000 7ff636ffd640 1 -- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff63809d790 msgr2=0x7ff63809db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:41:23.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:23.469+0000 7ff636ffd640 1 -- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff63809d790 msgr2=0x7ff63809db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:41:23.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:23.469+0000 7ff636ffd640 1 --2- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff63809d790 0x7ff63809db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:41:23.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:23.469+0000 7ff636ffd640 1 --2- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff63809d790 0x7ff63809db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:41:23.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:23.472+0000 7ff6377fe640 1 --2- 172.21.15.12:0/3255899323 >> v2:172.21.15.71:3300/0 conn(0x7ff63809e4d0 0x7ff638135170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:23.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:23.472+0000 7ff6377fe640 10 monclient(hunting): get_auth_request con 0x7ff63809e4d0 auth_method 0 2024-09-11T01:41:23.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:23.472+0000 7ff6377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:23.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:23.472+0000 7ff6377fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:23.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:23.472+0000 7ff6377fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:41:23.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:23.472+0000 7ff6377fe640 1 -- 172.21.15.12:0/3255899323 >> v2:172.21.15.71:3300/0 conn(0x7ff63809e4d0 msgr2=0x7ff638135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:41:23.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:23.472+0000 7ff6377fe640 1 -- 172.21.15.12:0/3255899323 >> v2:172.21.15.71:3300/0 conn(0x7ff63809e4d0 msgr2=0x7ff638135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:41:23.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:23.472+0000 7ff6377fe640 1 --2- 172.21.15.12:0/3255899323 >> v2:172.21.15.71:3300/0 conn(0x7ff63809e4d0 0x7ff638135170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:41:23.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:23.472+0000 7ff6377fe640 1 --2- 172.21.15.12:0/3255899323 >> v2:172.21.15.71:3300/0 conn(0x7ff63809e4d0 0x7ff638135170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:41:25.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.065+0000 7ff6357fa640 10 monclient(hunting): tick 2024-09-11T01:41:25.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.065+0000 7ff6357fa640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:41:25.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.065+0000 7ff6357fa640 1 monclient(hunting): continuing hunt 2024-09-11T01:41:25.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.065+0000 7ff6357fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:41:25.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.065+0000 7ff6357fa640 1 -- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff63809d790 msgr2=0x7ff63809db70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:41:25.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.065+0000 7ff6357fa640 1 --2- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff63809d790 0x7ff63809db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:25.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.065+0000 7ff6357fa640 1 -- 172.21.15.12:0/3255899323 >> v2:172.21.15.71:3300/0 conn(0x7ff63809e4d0 msgr2=0x7ff638135170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:41:25.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.065+0000 7ff6357fa640 1 --2- 172.21.15.12:0/3255899323 >> v2:172.21.15.71:3300/0 conn(0x7ff63809e4d0 0x7ff638135170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:25.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.065+0000 7ff6357fa640 1 -- 172.21.15.12:0/3255899323 >> v1:172.21.15.71:6789/0 conn(0x7ff63809e0b0 legacy=0x7ff638134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:25.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.065+0000 7ff6357fa640 10 monclient: _add_conns ranks=[0,3] 2024-09-11T01:41:25.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.065+0000 7ff6357fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff618003740 addr v1:172.21.15.12:6789/0 2024-09-11T01:41:25.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.065+0000 7ff6357fa640 1 --2- 172.21.15.12:0/3255899323 >> v2:172.21.15.12:3300/0 conn(0x7ff618007470 0x7ff618009910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:25.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.065+0000 7ff6357fa640 10 monclient(hunting): picked mon.noname-a con 0x7ff618007470 addr v2:172.21.15.12:3300/0 2024-09-11T01:41:25.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.065+0000 7ff6357fa640 1 -- 172.21.15.12:0/3255899323 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6380a1fa0 con 0x7ff618003740 2024-09-11T01:41:25.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.065+0000 7ff6357fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:25.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.065+0000 7ff6357fa640 1 -- 172.21.15.12:0/3255899323 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff6380a0760 con 0x7ff618007470 2024-09-11T01:41:25.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.065+0000 7ff6357fa640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:25.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.065+0000 7ff62e7fc640 1 --2- 172.21.15.12:0/3255899323 >> v2:172.21.15.12:3300/0 conn(0x7ff618007470 0x7ff618009910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:25.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.065+0000 7ff62e7fc640 10 monclient(hunting): get_auth_request con 0x7ff618007470 auth_method 0 2024-09-11T01:41:25.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.065+0000 7ff62e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:25.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.065+0000 7ff62e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:25.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.065+0000 7ff62e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:25.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.065+0000 7ff62e7fc640 1 -- 172.21.15.12:0/3255899323 >> v2:172.21.15.12:3300/0 conn(0x7ff618007470 msgr2=0x7ff618009910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:41:25.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.065+0000 7ff62e7fc640 1 -- 172.21.15.12:0/3255899323 >> v2:172.21.15.12:3300/0 conn(0x7ff618007470 msgr2=0x7ff618009910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:41:25.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.065+0000 7ff62e7fc640 1 --2- 172.21.15.12:0/3255899323 >> v2:172.21.15.12:3300/0 conn(0x7ff618007470 0x7ff618009910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:41:25.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.065+0000 7ff62e7fc640 1 --2- 172.21.15.12:0/3255899323 >> v2:172.21.15.12:3300/0 conn(0x7ff618007470 0x7ff618009910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:41:25.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.067+0000 7ff636ffd640 1 -- 172.21.15.12:0/3255899323 >> v1:172.21.15.12:6789/0 conn(0x7ff618003740 legacy=0x7ff618005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:25.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.266+0000 7ff62e7fc640 1 --2- 172.21.15.12:0/3255899323 >> v2:172.21.15.12:3300/0 conn(0x7ff618007470 0x7ff618009910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:25.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.266+0000 7ff62e7fc640 10 monclient(hunting): get_auth_request con 0x7ff618007470 auth_method 0 2024-09-11T01:41:25.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.266+0000 7ff62e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:25.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.266+0000 7ff62e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:25.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.266+0000 7ff62e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:41:25.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.266+0000 7ff62e7fc640 1 -- 172.21.15.12:0/3255899323 >> v2:172.21.15.12:3300/0 conn(0x7ff618007470 msgr2=0x7ff618009910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:41:25.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.266+0000 7ff62e7fc640 1 -- 172.21.15.12:0/3255899323 >> v2:172.21.15.12:3300/0 conn(0x7ff618007470 msgr2=0x7ff618009910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:41:25.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.266+0000 7ff62e7fc640 1 --2- 172.21.15.12:0/3255899323 >> v2:172.21.15.12:3300/0 conn(0x7ff618007470 0x7ff618009910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:41:25.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.266+0000 7ff62e7fc640 1 --2- 172.21.15.12:0/3255899323 >> v2:172.21.15.12:3300/0 conn(0x7ff618007470 0x7ff618009910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:41:25.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.667+0000 7ff62e7fc640 1 --2- 172.21.15.12:0/3255899323 >> v2:172.21.15.12:3300/0 conn(0x7ff618007470 0x7ff618009910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:25.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.667+0000 7ff62e7fc640 10 monclient(hunting): get_auth_request con 0x7ff618007470 auth_method 0 2024-09-11T01:41:25.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.667+0000 7ff62e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:25.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.667+0000 7ff62e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:25.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.667+0000 7ff62e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:41:25.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.667+0000 7ff62e7fc640 1 -- 172.21.15.12:0/3255899323 >> v2:172.21.15.12:3300/0 conn(0x7ff618007470 msgr2=0x7ff618009910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:41:25.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.667+0000 7ff62e7fc640 1 -- 172.21.15.12:0/3255899323 >> v2:172.21.15.12:3300/0 conn(0x7ff618007470 msgr2=0x7ff618009910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:41:25.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.667+0000 7ff62e7fc640 1 --2- 172.21.15.12:0/3255899323 >> v2:172.21.15.12:3300/0 conn(0x7ff618007470 0x7ff618009910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:41:25.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.667+0000 7ff62e7fc640 1 --2- 172.21.15.12:0/3255899323 >> v2:172.21.15.12:3300/0 conn(0x7ff618007470 0x7ff618009910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:41:25.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.968+0000 7ff6367fc640 1 -- 172.21.15.12:0/3255899323 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3348609214 0 0) 0x7ff6380a1fa0 con 0x7ff618003740 2024-09-11T01:41:25.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.968+0000 7ff6367fc640 10 monclient(hunting): _negotiate 2024-09-11T01:41:25.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.968+0000 7ff6367fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:25.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.968+0000 7ff6367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:25.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.968+0000 7ff6367fc640 10 monclient(hunting): my global_id is 94440 2024-09-11T01:41:25.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.968+0000 7ff6367fc640 1 -- 172.21.15.12:0/3255899323 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff624003520 con 0x7ff618003740 2024-09-11T01:41:25.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.970+0000 7ff6367fc640 1 -- 172.21.15.12:0/3255899323 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4067176830 0 0) 0x7ff624003520 con 0x7ff618003740 2024-09-11T01:41:25.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.970+0000 7ff6367fc640 1 -- 172.21.15.12:0/3255899323 >> v2:172.21.15.12:3300/0 conn(0x7ff618007470 msgr2=0x7ff618009910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:41:25.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.970+0000 7ff6367fc640 1 --2- 172.21.15.12:0/3255899323 >> v2:172.21.15.12:3300/0 conn(0x7ff618007470 0x7ff618009910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:25.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.970+0000 7ff6367fc640 10 monclient: _finish_hunting 0 2024-09-11T01:41:25.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.970+0000 7ff6367fc640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:41:25.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.970+0000 7ff6367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:25.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.970+0000 7ff6367fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:41:25.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.970+0000 7ff6367fc640 1 -- 172.21.15.12:0/3255899323 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff61800b0a0 con 0x7ff618003740 2024-09-11T01:41:25.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.970+0000 7ff6367fc640 10 monclient: _finish_auth 0 2024-09-11T01:41:25.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.970+0000 7ff6367fc640 10 monclient: _check_auth_tickets 2024-09-11T01:41:25.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.970+0000 7ff6367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:25.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.970+0000 7ff6367fc640 1 -- 172.21.15.12:0/3255899323 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7ff62800f040 con 0x7ff618003740 2024-09-11T01:41:25.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.970+0000 7ff6367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:25.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.970+0000 7ff6367fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:41:25.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.970+0000 7ff6367fc640 10 monclient: dump: 2024-09-11T01:41:25.976 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:25.976 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:25.976 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:25.976 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:25.976 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:25.977 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:25.977 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:25.977 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:25.977 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:25.977 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:25.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.970+0000 7ff6367fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:41:25.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.970+0000 7ff6367fc640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:25.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.970+0000 7ff6367fc640 1 -- 172.21.15.12:0/3255899323 >> v1:172.21.15.12:6789/0 conn(0x7ff618003740 legacy=0x7ff618005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:25.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.970+0000 7ff6377fe640 1 -- 172.21.15.12:0/3255899323 reap_dead start 2024-09-11T01:41:25.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.970+0000 7ff6367fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:41:25.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff6367fc640 1 --2- 172.21.15.12:0/3255899323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff624003aa0 0x7ff624003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:25.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff6367fc640 10 monclient(hunting): picked mon.b con 0x7ff624003aa0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:41:25.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff6367fc640 1 --2- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff624004800 0x7ff624004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:25.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff6367fc640 10 monclient(hunting): picked mon.a con 0x7ff624004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:41:25.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff6367fc640 1 --2- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff624005410 0x7ff62400ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:25.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff6367fc640 10 monclient(hunting): picked mon.c con 0x7ff624005410 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:25.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff6367fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:25.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff6367fc640 1 -- 172.21.15.12:0/3255899323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff6380a0760 con 0x7ff624004800 2024-09-11T01:41:25.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff6367fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:25.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff6367fc640 1 -- 172.21.15.12:0/3255899323 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff62400c320 con 0x7ff624003aa0 2024-09-11T01:41:25.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff6367fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:25.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff6367fc640 1 -- 172.21.15.12:0/3255899323 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff62400c520 con 0x7ff624005410 2024-09-11T01:41:25.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff6367fc640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:25.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff6367fc640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:41:25.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff62e7fc640 1 --2- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff624004800 0x7ff624004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:25.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff62e7fc640 10 monclient(hunting): get_auth_request con 0x7ff624004800 auth_method 0 2024-09-11T01:41:25.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff62e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:25.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff62e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:25.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff62e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:41:25.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff636ffd640 1 --2- 172.21.15.12:0/3255899323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff624003aa0 0x7ff624003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:25.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff6377fe640 1 --2- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff624005410 0x7ff62400ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:25.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff636ffd640 10 monclient(hunting): get_auth_request con 0x7ff624003aa0 auth_method 0 2024-09-11T01:41:25.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff636ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:25.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff636ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:25.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff636ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:41:25.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff6377fe640 10 monclient(hunting): get_auth_request con 0x7ff624005410 auth_method 0 2024-09-11T01:41:25.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff6377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:25.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff6377fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:25.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff6377fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:41:25.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff62e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:25.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff62e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:25.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff62e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:41:25.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff636ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:25.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff636ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:25.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff636ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:41:25.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff6377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:25.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff6377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:25.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff6377fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:41:25.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff62e7fc640 10 monclient(hunting): handle_auth_done global_id 94440 payload 293 2024-09-11T01:41:25.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff62e7fc640 1 -- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff624005410 msgr2=0x7ff62400ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:25.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff62e7fc640 1 --2- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff624005410 0x7ff62400ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:25.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff62e7fc640 1 -- 172.21.15.12:0/3255899323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff624003aa0 msgr2=0x7ff624003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:25.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff62e7fc640 1 --2- 172.21.15.12:0/3255899323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff624003aa0 0x7ff624003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:25.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff62e7fc640 10 monclient: _finish_hunting 0 2024-09-11T01:41:25.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff62e7fc640 1 monclient: found mon.a 2024-09-11T01:41:25.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff62e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:25.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff62e7fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:25.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff62e7fc640 1 -- 172.21.15.12:0/3255899323 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff624003520 con 0x7ff624004800 2024-09-11T01:41:25.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff62e7fc640 10 monclient: _finish_auth 0 2024-09-11T01:41:25.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff62e7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:41:25.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff62e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:25.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff63d893640 5 monclient: authenticate success, global_id 94440 2024-09-11T01:41:25.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.971+0000 7ff63d893640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:41:25.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.974+0000 7ff62e7fc640 1 -- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff624004800 msgr2=0x7ff624004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:25.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.974+0000 7ff62e7fc640 1 --2- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff624004800 0x7ff624004c80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff620001900 tx=0x7ff624005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e0167eb6497fa4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:25.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.974+0000 7ff6367fc640 1 -- 172.21.15.12:0/3255899323 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff620001630 con 0x7ff624004800 2024-09-11T01:41:25.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.974+0000 7ff6367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:25.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.974+0000 7ff6367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:25.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.974+0000 7ff6367fc640 10 monclient: dump: 2024-09-11T01:41:25.987 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:25.987 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:25.987 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:25.988 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:25.988 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:25.988 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:25.988 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:25.988 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:25.988 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:25.988 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:25.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.974+0000 7ff63d893640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:41:25.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.974+0000 7ff6367fc640 1 -- 172.21.15.12:0/3255899323 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6200070d0 con 0x7ff624004800 2024-09-11T01:41:25.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.974+0000 7ff6367fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:25.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.974+0000 7ff63d893640 10 monclient: get_monmap_and_config success 2024-09-11T01:41:25.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.974+0000 7ff6367fc640 1 -- 172.21.15.12:0/3255899323 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6200074a0 con 0x7ff624004800 2024-09-11T01:41:25.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.974+0000 7ff63d893640 4 set_mon_vals no callback set 2024-09-11T01:41:25.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.974+0000 7ff6367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:25.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.974+0000 7ff6367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:25.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.974+0000 7ff6367fc640 10 monclient: dump: 2024-09-11T01:41:25.989 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:25.989 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:25.989 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:25.989 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:25.989 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:25.989 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:25.989 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:25.989 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:25.989 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:25.989 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:25.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.975+0000 7ff63d893640 10 monclient: shutdown 2024-09-11T01:41:25.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.975+0000 7ff63d893640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:25.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.975+0000 7ff63d893640 1 -- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff624004800 msgr2=0x7ff624004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:25.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.975+0000 7ff63d893640 1 --2- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff624004800 0x7ff624004c80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff620001900 tx=0x7ff624005830 comp rx=0 tx=0).stop 2024-09-11T01:41:25.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.975+0000 7ff63d893640 1 -- 172.21.15.12:0/3255899323 shutdown_connections 2024-09-11T01:41:25.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.975+0000 7ff63d893640 1 --2- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff624005410 0x7ff62400ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:25.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.975+0000 7ff63d893640 1 --2- 172.21.15.12:0/3255899323 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff624004800 0x7ff624004c80 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:25.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.975+0000 7ff63d893640 1 --2- 172.21.15.12:0/3255899323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff624003aa0 0x7ff624003ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:25.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.975+0000 7ff63d893640 1 -- 172.21.15.12:0/3255899323 >> 172.21.15.12:0/3255899323 conn(0x7ff6380a2d90 msgr2=0x7ff6380a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:25.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.975+0000 7ff63d893640 1 -- 172.21.15.12:0/3255899323 shutdown_connections 2024-09-11T01:41:25.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.975+0000 7ff63d893640 1 -- 172.21.15.12:0/3255899323 wait complete. 2024-09-11T01:41:25.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.976+0000 7ff63d893640 10 monclient: build_initial_monmap 2024-09-11T01:41:25.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.976+0000 7ff63d893640 10 monclient: monmap: 2024-09-11T01:41:25.990 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:25.990 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:41:25.991 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:41:25.991 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:41:25.991 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:25.991 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:25.991 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:41:25.991 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:41:25.991 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:41:25.991 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:25.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.976+0000 7ff63d893640 1 Processor -- start 2024-09-11T01:41:25.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.977+0000 7ff63d893640 1 -- start start 2024-09-11T01:41:25.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.977+0000 7ff63d893640 10 monclient: init 2024-09-11T01:41:25.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.977+0000 7ff63d893640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:25.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.977+0000 7ff63d893640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:41:25.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.977+0000 7ff63d893640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff624003aa0 0x7ff6381242b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:25.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.977+0000 7ff63d893640 10 monclient(hunting): picked mon.noname-a con 0x7ff624003aa0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:41:25.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.977+0000 7ff63d893640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff624004800 0x7ff6381247f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:25.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.977+0000 7ff63d893640 10 monclient(hunting): picked mon.noname-b con 0x7ff624004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:41:25.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.977+0000 7ff63d893640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff624005410 0x7ff638124d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:25.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff63d893640 10 monclient(hunting): picked mon.noname-c con 0x7ff624005410 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:25.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff63d893640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:25.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff63d893640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff6380a23a0 con 0x7ff624003aa0 2024-09-11T01:41:25.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff63d893640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:25.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff63d893640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff6380a2220 con 0x7ff624004800 2024-09-11T01:41:25.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff63d893640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:25.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff63d893640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff63809cca0 con 0x7ff624005410 2024-09-11T01:41:25.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff63d893640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff63d893640 10 monclient(hunting): authenticate will time out at 1492.420046s 2024-09-11T01:41:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff636ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff624003aa0 0x7ff6381242b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff636ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff624003aa0 0x7ff6381242b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39488/0 (socket says 172.21.15.12:39488) 2024-09-11T01:41:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff636ffd640 1 -- 172.21.15.12:0/883756200 learned_addr learned my addr 172.21.15.12:0/883756200 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:41:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff636ffd640 10 monclient(hunting): get_auth_request con 0x7ff624003aa0 auth_method 0 2024-09-11T01:41:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff636ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff636ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff636ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff6377fe640 1 --2- 172.21.15.12:0/883756200 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff624005410 0x7ff638124d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff6377fe640 10 monclient(hunting): get_auth_request con 0x7ff624005410 auth_method 0 2024-09-11T01:41:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff6377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:25.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff6377fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:25.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff6377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:25.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff636ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:25.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff636ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:25.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff636ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:25.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff6377fe640 1 -- 172.21.15.12:0/883756200 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff624005410 msgr2=0x7ff638124d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:41:25.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff6377fe640 1 -- 172.21.15.12:0/883756200 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff624005410 msgr2=0x7ff638124d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:41:25.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff6377fe640 1 --2- 172.21.15.12:0/883756200 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff624005410 0x7ff638124d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:41:25.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff62e7fc640 1 --2- 172.21.15.12:0/883756200 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff624004800 0x7ff6381247f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:25.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff62e7fc640 10 monclient(hunting): get_auth_request con 0x7ff624004800 auth_method 0 2024-09-11T01:41:25.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff62e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:25.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff62e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:25.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff62e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:25.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff6377fe640 1 --2- 172.21.15.12:0/883756200 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff624005410 0x7ff638124d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:41:25.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff636ffd640 10 monclient(hunting): handle_auth_done global_id 94443 payload 824 2024-09-11T01:41:25.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff62e7fc640 1 -- 172.21.15.12:0/883756200 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff624004800 msgr2=0x7ff6381247f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:41:25.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff62e7fc640 1 -- 172.21.15.12:0/883756200 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff624004800 msgr2=0x7ff6381247f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:41:25.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff62e7fc640 1 --2- 172.21.15.12:0/883756200 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff624004800 0x7ff6381247f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:41:25.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff636ffd640 1 -- 172.21.15.12:0/883756200 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff624005410 msgr2=0x7ff638124d30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:41:25.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff636ffd640 1 --2- 172.21.15.12:0/883756200 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff624005410 0x7ff638124d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:25.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff62e7fc640 1 --2- 172.21.15.12:0/883756200 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff624004800 0x7ff6381247f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:41:25.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff636ffd640 1 -- 172.21.15.12:0/883756200 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff624004800 msgr2=0x7ff6381247f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:41:25.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff636ffd640 1 --2- 172.21.15.12:0/883756200 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff624004800 0x7ff6381247f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:25.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff636ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:41:25.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff636ffd640 1 monclient: found mon.noname-a 2024-09-11T01:41:25.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff636ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:25.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff636ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:41:25.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.978+0000 7ff636ffd640 1 -- 172.21.15.12:0/883756200 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6381280d0 con 0x7ff624003aa0 2024-09-11T01:41:25.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.980+0000 7ff636ffd640 1 -- 172.21.15.12:0/883756200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff624003aa0 msgr2=0x7ff6381242b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:25.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.980+0000 7ff636ffd640 1 --2- 172.21.15.12:0/883756200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff624003aa0 0x7ff6381242b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff628016e70 tx=0x7ff628014720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.981+0000 7ff634ff9640 1 -- 172.21.15.12:0/883756200 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62801c070 con 0x7ff624003aa0 2024-09-11T01:41:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.981+0000 7ff634ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.981+0000 7ff634ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:41:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.981+0000 7ff634ff9640 10 monclient: dump: 2024-09-11T01:41:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:25.996 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:25.997 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:25.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.981+0000 7ff634ff9640 10 monclient: _finish_auth 0 2024-09-11T01:41:25.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.981+0000 7ff634ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:41:25.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.981+0000 7ff634ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:25.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.981+0000 7ff63d893640 5 monclient: authenticate success, global_id 94443 2024-09-11T01:41:25.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.981+0000 7ff634ff9640 1 -- 172.21.15.12:0/883756200 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff628004250 con 0x7ff624003aa0 2024-09-11T01:41:25.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.981+0000 7ff634ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:25.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.981+0000 7ff634ff9640 1 -- 172.21.15.12:0/883756200 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62800f040 con 0x7ff624003aa0 2024-09-11T01:41:25.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.981+0000 7ff63d893640 10 monclient: _renew_subs 2024-09-11T01:41:25.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.981+0000 7ff63d893640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:25.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.981+0000 7ff63d893640 1 -- 172.21.15.12:0/883756200 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff63809d480 con 0x7ff624003aa0 2024-09-11T01:41:25.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.981+0000 7ff634ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:25.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.981+0000 7ff635ffb640 4 set_mon_vals no callback set 2024-09-11T01:41:25.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.981+0000 7ff634ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:25.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.981+0000 7ff634ff9640 10 monclient: dump: 2024-09-11T01:41:25.998 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:25.998 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:25.998 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:25.998 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:25.998 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:25.998 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:25.998 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:25.998 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:25.998 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:25.998 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:25.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.981+0000 7ff63d893640 10 monclient: _renew_subs 2024-09-11T01:41:25.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.981+0000 7ff63d893640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:25.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.981+0000 7ff63d893640 1 -- 172.21.15.12:0/883756200 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff638125d70 con 0x7ff624003aa0 2024-09-11T01:41:25.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.982+0000 7ff634ff9640 1 -- 172.21.15.12:0/883756200 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff628015040 con 0x7ff624003aa0 2024-09-11T01:41:25.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.982+0000 7ff63d893640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:25.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.982+0000 7ff63d893640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:25.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.982+0000 7ff63d893640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:25.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.982+0000 7ff63d893640 1 -- 172.21.15.12:0/883756200 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff63809db70 con 0x7ff624003aa0 2024-09-11T01:41:25.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.982+0000 7ff634ff9640 1 --2- 172.21.15.12:0/883756200 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff61803eb70 0x7ff618041030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:25.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.982+0000 7ff634ff9640 1 -- 172.21.15.12:0/883756200 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(402..402 src has 285..402) ==== 4755+0+0 (secure 0 0 0) 0x7ff628053b10 con 0x7ff624003aa0 2024-09-11T01:41:25.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.983+0000 7ff62e7fc640 1 --2- 172.21.15.12:0/883756200 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff61803eb70 0x7ff618041030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:25.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.983+0000 7ff62e7fc640 10 monclient: get_auth_request con 0x7ff61803eb70 auth_method 0 2024-09-11T01:41:25.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.985+0000 7ff62e7fc640 1 --2- 172.21.15.12:0/883756200 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff61803eb70 0x7ff618041030 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff620031000 tx=0x7ff620007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:26.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.987+0000 7ff634ff9640 1 -- 172.21.15.12:0/883756200 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff628052520 con 0x7ff624003aa0 2024-09-11T01:41:26.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.987+0000 7ff634ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:26.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:25.987+0000 7ff634ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:41:26.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:26.245+0000 7ff63d893640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:41:26.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:26.246+0000 7ff63d893640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:41:26.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:26.246+0000 7ff63d893640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:26.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:26.246+0000 7ff63d893640 1 -- 172.21.15.12:0/883756200 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff63806d460 con 0x7ff624003aa0 2024-09-11T01:41:26.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:26.246+0000 7ff634ff9640 1 -- 172.21.15.12:0/883756200 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7ff628003dc0 con 0x7ff624003aa0 2024-09-11T01:41:26.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:26.246+0000 7ff634ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:41:26.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:26.246+0000 7ff634ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:41:26.248 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":110,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:41:26.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:26.248+0000 7ff63d893640 1 -- 172.21.15.12:0/883756200 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff61803eb70 msgr2=0x7ff618041030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:26.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:26.248+0000 7ff63d893640 1 --2- 172.21.15.12:0/883756200 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff61803eb70 0x7ff618041030 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff620031000 tx=0x7ff620007000 comp rx=0 tx=0).stop 2024-09-11T01:41:26.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:26.248+0000 7ff63d893640 10 monclient: shutdown 2024-09-11T01:41:26.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:26.248+0000 7ff63d893640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:26.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:26.248+0000 7ff63d893640 1 -- 172.21.15.12:0/883756200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff624003aa0 msgr2=0x7ff6381242b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:26.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:26.248+0000 7ff63d893640 1 --2- 172.21.15.12:0/883756200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff624003aa0 0x7ff6381242b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff628016e70 tx=0x7ff628014720 comp rx=0 tx=0).stop 2024-09-11T01:41:26.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:26.248+0000 7ff63d893640 1 -- 172.21.15.12:0/883756200 shutdown_connections 2024-09-11T01:41:26.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:26.248+0000 7ff63d893640 1 --2- 172.21.15.12:0/883756200 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff624005410 0x7ff638124d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:26.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:26.248+0000 7ff63d893640 1 --2- 172.21.15.12:0/883756200 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff624004800 0x7ff6381247f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:26.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:26.248+0000 7ff63d893640 1 --2- 172.21.15.12:0/883756200 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff61803eb70 0x7ff618041030 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:26.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:26.248+0000 7ff63d893640 1 --2- 172.21.15.12:0/883756200 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff624003aa0 0x7ff6381242b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:26.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:26.248+0000 7ff63d893640 1 -- 172.21.15.12:0/883756200 >> 172.21.15.12:0/883756200 conn(0x7ff6380a2d90 msgr2=0x7ff6380a4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:26.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:26.248+0000 7ff63d893640 1 -- 172.21.15.12:0/883756200 shutdown_connections 2024-09-11T01:41:26.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:26.248+0000 7ff63d893640 1 -- 172.21.15.12:0/883756200 wait complete. 2024-09-11T01:41:27.322 INFO:tasks.rados.rados.0.smithi071.stdout:1746: done (0 left) 2024-09-11T01:41:27.322 INFO:tasks.rados.rados.0.smithi071.stdout:1747: read oid 22 snap 191 2024-09-11T01:41:27.322 INFO:tasks.rados.rados.0.smithi071.stdout:1747: expect deleted 2024-09-11T01:41:27.322 INFO:tasks.rados.rados.0.smithi071.stdout:1748: write oid 5 current snap is 195 2024-09-11T01:41:27.322 INFO:tasks.rados.rados.0.smithi071.stdout:1748: seq_num 639 ranges {739673=657763,2076229=663444} 2024-09-11T01:41:27.329 INFO:tasks.rados.rados.0.smithi071.stdout:1748: writing smithi07134777-5 from 739673 to 1397436 tid 1 2024-09-11T01:41:27.337 INFO:tasks.rados.rados.0.smithi071.stdout:1748: writing smithi07134777-5 from 2076229 to 2739673 tid 2 2024-09-11T01:41:27.337 INFO:tasks.rados.rados.0.smithi071.stdout:1749: write (excl) oid 19 current snap is 195 2024-09-11T01:41:27.337 INFO:tasks.rados.rados.0.smithi071.stdout:1749: seq_num 640 ranges {756480=733012,2134711=432549} 2024-09-11T01:41:27.345 INFO:tasks.rados.rados.0.smithi071.stdout:1749: writing smithi07134777-19 from 756480 to 1489492 tid 1 2024-09-11T01:41:27.352 INFO:tasks.rados.rados.0.smithi071.stdout:1749: writing smithi07134777-19 from 2134711 to 2567260 tid 2 2024-09-11T01:41:27.352 INFO:tasks.rados.rados.0.smithi071.stdout:1750: rollback oid 40 current snap is 195 2024-09-11T01:41:27.352 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 40 to 174 2024-09-11T01:41:27.352 INFO:tasks.rados.rados.0.smithi071.stdout:1748: finishing write tid 1 to smithi07134777-5 2024-09-11T01:41:27.352 INFO:tasks.rados.rados.0.smithi071.stdout:1748: finishing write tid 2 to smithi07134777-5 2024-09-11T01:41:27.352 INFO:tasks.rados.rados.0.smithi071.stdout:1748: finishing write tid 3 to smithi07134777-5 2024-09-11T01:41:27.352 INFO:tasks.rados.rados.0.smithi071.stdout:1748: finishing write tid 4 to smithi07134777-5 2024-09-11T01:41:27.352 INFO:tasks.rados.rados.0.smithi071.stdout:1748: oid 5 updating version 0 to 493 2024-09-11T01:41:27.352 INFO:tasks.rados.rados.0.smithi071.stdout:1748: oid 5 updating version 493 to 495 2024-09-11T01:41:27.352 INFO:tasks.rados.rados.0.smithi071.stdout:1748: oid 5 version 495 is already newer than 494 2024-09-11T01:41:27.352 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 495 (ObjNum 639 snap 195 seq_num 639) dirty exists 2024-09-11T01:41:27.352 INFO:tasks.rados.rados.0.smithi071.stdout:1748: left oid 5 (ObjNum 639 snap 195 seq_num 639) 2024-09-11T01:41:27.353 INFO:tasks.rados.rados.0.smithi071.stdout:1749: finishing write tid 1 to smithi07134777-19 2024-09-11T01:41:27.353 INFO:tasks.rados.rados.0.smithi071.stdout:1747: done (3 left) 2024-09-11T01:41:27.353 INFO:tasks.rados.rados.0.smithi071.stdout:1748: done (2 left) 2024-09-11T01:41:27.353 INFO:tasks.rados.rados.0.smithi071.stdout:1751: write oid 1 current snap is 195 2024-09-11T01:41:27.353 INFO:tasks.rados.rados.0.smithi071.stdout:1751: seq_num 641 ranges {773287=408261,1793193=576207} 2024-09-11T01:41:27.358 INFO:tasks.rados.rados.0.smithi071.stdout:1751: writing smithi07134777-1 from 773287 to 1181548 tid 1 2024-09-11T01:41:27.364 INFO:tasks.rados.rados.0.smithi071.stdout:1751: writing smithi07134777-1 from 1793193 to 2369400 tid 2 2024-09-11T01:41:27.364 INFO:tasks.rados.rados.0.smithi071.stdout:1752: snap_remove snap 183 2024-09-11T01:41:28.331 INFO:tasks.rados.rados.0.smithi071.stdout:1752: done (3 left) 2024-09-11T01:41:28.331 INFO:tasks.rados.rados.0.smithi071.stdout:1753: copy_from oid 21 from oid 15 current snap is 195 2024-09-11T01:41:28.331 INFO:tasks.rados.rados.0.smithi071.stdout:1750: finishing rollback tid 1 to smithi07134777-40 2024-09-11T01:41:28.331 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 0 (ObjNum 33 snap 0 seq_num 2342451096) dirty dne 2024-09-11T01:41:28.331 INFO:tasks.rados.rados.0.smithi071.stdout:1749: finishing write tid 2 to smithi07134777-19 2024-09-11T01:41:28.331 INFO:tasks.rados.rados.0.smithi071.stdout:1749: finishing write tid 3 to smithi07134777-19 2024-09-11T01:41:28.332 INFO:tasks.rados.rados.0.smithi071.stdout:1749: finishing write tid 4 to smithi07134777-19 2024-09-11T01:41:28.332 INFO:tasks.rados.rados.0.smithi071.stdout:1749: oid 19 updating version 0 to 901 2024-09-11T01:41:28.332 INFO:tasks.rados.rados.0.smithi071.stdout:1749: oid 19 updating version 901 to 903 2024-09-11T01:41:28.332 INFO:tasks.rados.rados.0.smithi071.stdout:1749: oid 19 version 903 is already newer than 902 2024-09-11T01:41:28.332 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 903 (ObjNum 640 snap 195 seq_num 640) dirty exists 2024-09-11T01:41:28.332 INFO:tasks.rados.rados.0.smithi071.stdout:1749: left oid 19 (ObjNum 640 snap 195 seq_num 640) 2024-09-11T01:41:28.332 INFO:tasks.rados.rados.0.smithi071.stdout:1751: finishing write tid 1 to smithi07134777-1 2024-09-11T01:41:28.332 INFO:tasks.rados.rados.0.smithi071.stdout:1751: finishing write tid 2 to smithi07134777-1 2024-09-11T01:41:28.332 INFO:tasks.rados.rados.0.smithi071.stdout:1751: finishing write tid 3 to smithi07134777-1 2024-09-11T01:41:28.332 INFO:tasks.rados.rados.0.smithi071.stdout:1751: finishing write tid 4 to smithi07134777-1 2024-09-11T01:41:28.332 INFO:tasks.rados.rados.0.smithi071.stdout:1751: oid 1 updating version 0 to 815 2024-09-11T01:41:28.332 INFO:tasks.rados.rados.0.smithi071.stdout:1751: oid 1 version 815 is already newer than 814 2024-09-11T01:41:28.332 INFO:tasks.rados.rados.0.smithi071.stdout:1751: oid 1 updating version 815 to 816 2024-09-11T01:41:28.332 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 816 (ObjNum 641 snap 195 seq_num 641) dirty exists 2024-09-11T01:41:28.333 INFO:tasks.rados.rados.0.smithi071.stdout:1751: left oid 1 (ObjNum 641 snap 195 seq_num 641) 2024-09-11T01:41:28.333 INFO:tasks.rados.rados.0.smithi071.stdout:1749: done (3 left) 2024-09-11T01:41:28.333 INFO:tasks.rados.rados.0.smithi071.stdout:1750: done (2 left) 2024-09-11T01:41:28.333 INFO:tasks.rados.rados.0.smithi071.stdout:1751: done (1 left) 2024-09-11T01:41:28.333 INFO:tasks.rados.rados.0.smithi071.stdout:1754: snap_create 2024-09-11T01:41:28.349 INFO:tasks.rados.rados.0.smithi071.stdout:1753: finishing copy_from racing read to smithi07134777-21 2024-09-11T01:41:28.351 INFO:tasks.rados.rados.0.smithi071.stdout:1753: finishing copy_from to smithi07134777-21 2024-09-11T01:41:28.351 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 922 (ObjNum 565 snap 160 seq_num 565) dirty exists 2024-09-11T01:41:28.956 INFO:tasks.rados.rados.0.smithi071.stdout:1753: done (1 left) 2024-09-11T01:41:28.956 INFO:tasks.rados.rados.0.smithi071.stdout:1754: done (0 left) 2024-09-11T01:41:28.956 INFO:tasks.rados.rados.0.smithi071.stdout:1755: delete oid 42 current snap is 196 2024-09-11T01:41:28.961 INFO:tasks.rados.rados.0.smithi071.stdout:1755: done (0 left) 2024-09-11T01:41:28.962 INFO:tasks.rados.rados.0.smithi071.stdout:1756: write oid 23 current snap is 196 2024-09-11T01:41:28.962 INFO:tasks.rados.rados.0.smithi071.stdout:1756: seq_num 643 ranges {406901=558759,1393804=579876,2608396=198505} 2024-09-11T01:41:28.966 INFO:tasks.rados.rados.0.smithi071.stdout:1756: writing smithi07134777-23 from 406901 to 965660 tid 1 2024-09-11T01:41:28.973 INFO:tasks.rados.rados.0.smithi071.stdout:1756: writing smithi07134777-23 from 1393804 to 1973680 tid 2 2024-09-11T01:41:28.978 INFO:tasks.rados.rados.0.smithi071.stdout:1756: writing smithi07134777-23 from 2608396 to 2806901 tid 3 2024-09-11T01:41:28.978 INFO:tasks.rados.rados.0.smithi071.stdout:1757: write oid 4 current snap is 196 2024-09-11T01:41:28.978 INFO:tasks.rados.rados.0.smithi071.stdout:1757: seq_num 644 ranges {423708=634008,1852286=439887,2752172=71536} 2024-09-11T01:41:28.983 INFO:tasks.rados.rados.0.smithi071.stdout:1757: writing smithi07134777-4 from 423708 to 1057716 tid 1 2024-09-11T01:41:28.990 INFO:tasks.rados.rados.0.smithi071.stdout:1757: writing smithi07134777-4 from 1852286 to 2292173 tid 2 2024-09-11T01:41:28.993 INFO:tasks.rados.rados.0.smithi071.stdout:1757: writing smithi07134777-4 from 2752172 to 2823708 tid 3 2024-09-11T01:41:28.993 INFO:tasks.rados.rados.0.smithi071.stdout:1758: write oid 44 current snap is 196 2024-09-11T01:41:28.993 INFO:tasks.rados.rados.0.smithi071.stdout:1758: seq_num 645 ranges {440515=709257,1910768=583545} 2024-09-11T01:41:28.999 INFO:tasks.rados.rados.0.smithi071.stdout:1758: writing smithi07134777-44 from 440515 to 1149772 tid 1 2024-09-11T01:41:29.007 INFO:tasks.rados.rados.0.smithi071.stdout:1758: writing smithi07134777-44 from 1910768 to 2494313 tid 2 2024-09-11T01:41:29.007 INFO:tasks.rados.rados.0.smithi071.stdout:1759: read oid 11 snap -1 2024-09-11T01:41:29.007 INFO:tasks.rados.rados.0.smithi071.stdout:1759: expect (ObjNum 597 snap 170 seq_num 597) 2024-09-11T01:41:29.007 INFO:tasks.rados.rados.0.smithi071.stdout:1760: read oid 46 snap -1 2024-09-11T01:41:29.007 INFO:tasks.rados.rados.0.smithi071.stdout:1760: expect (ObjNum 494 snap 132 seq_num 494) 2024-09-11T01:41:29.008 INFO:tasks.rados.rados.0.smithi071.stdout:1761: snap_create 2024-09-11T01:41:29.008 INFO:tasks.rados.rados.0.smithi071.stdout:1756: finishing write tid 1 to smithi07134777-23 2024-09-11T01:41:29.008 INFO:tasks.rados.rados.0.smithi071.stdout:1756: finishing write tid 2 to smithi07134777-23 2024-09-11T01:41:29.008 INFO:tasks.rados.rados.0.smithi071.stdout:1756: finishing write tid 3 to smithi07134777-23 2024-09-11T01:41:29.008 INFO:tasks.rados.rados.0.smithi071.stdout:1756: finishing write tid 4 to smithi07134777-23 2024-09-11T01:41:29.008 INFO:tasks.rados.rados.0.smithi071.stdout:1756: finishing write tid 5 to smithi07134777-23 2024-09-11T01:41:29.008 INFO:tasks.rados.rados.0.smithi071.stdout:1756: oid 23 updating version 0 to 319 2024-09-11T01:41:29.008 INFO:tasks.rados.rados.0.smithi071.stdout:1756: oid 23 updating version 319 to 320 2024-09-11T01:41:29.008 INFO:tasks.rados.rados.0.smithi071.stdout:1756: oid 23 version 320 is already newer than 318 2024-09-11T01:41:29.008 INFO:tasks.rados.rados.0.smithi071.stdout:1756: oid 23 version 320 is already newer than 317 2024-09-11T01:41:29.008 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 320 (ObjNum 643 snap 196 seq_num 643) dirty exists 2024-09-11T01:41:29.008 INFO:tasks.rados.rados.0.smithi071.stdout:1756: left oid 23 (ObjNum 643 snap 196 seq_num 643) 2024-09-11T01:41:29.008 INFO:tasks.rados.rados.0.smithi071.stdout:1757: finishing write tid 1 to smithi07134777-4 2024-09-11T01:41:29.008 INFO:tasks.rados.rados.0.smithi071.stdout:1757: finishing write tid 2 to smithi07134777-4 2024-09-11T01:41:29.009 INFO:tasks.rados.rados.0.smithi071.stdout:1757: finishing write tid 3 to smithi07134777-4 2024-09-11T01:41:29.009 INFO:tasks.rados.rados.0.smithi071.stdout:1757: finishing write tid 4 to smithi07134777-4 2024-09-11T01:41:29.009 INFO:tasks.rados.rados.0.smithi071.stdout:1757: finishing write tid 5 to smithi07134777-4 2024-09-11T01:41:29.009 INFO:tasks.rados.rados.0.smithi071.stdout:1757: oid 4 updating version 0 to 653 2024-09-11T01:41:29.009 INFO:tasks.rados.rados.0.smithi071.stdout:1757: oid 4 updating version 653 to 656 2024-09-11T01:41:29.009 INFO:tasks.rados.rados.0.smithi071.stdout:1757: oid 4 version 656 is already newer than 655 2024-09-11T01:41:29.009 INFO:tasks.rados.rados.0.smithi071.stdout:1757: oid 4 version 656 is already newer than 654 2024-09-11T01:41:29.009 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 656 (ObjNum 644 snap 196 seq_num 644) dirty exists 2024-09-11T01:41:29.009 INFO:tasks.rados.rados.0.smithi071.stdout:1757: left oid 4 (ObjNum 644 snap 196 seq_num 644) 2024-09-11T01:41:29.009 INFO:tasks.rados.rados.0.smithi071.stdout:1758: finishing write tid 1 to smithi07134777-44 2024-09-11T01:41:29.010 INFO:tasks.rados.rados.0.smithi071.stdout:1758: finishing write tid 2 to smithi07134777-44 2024-09-11T01:41:29.010 INFO:tasks.rados.rados.0.smithi071.stdout:1758: finishing write tid 3 to smithi07134777-44 2024-09-11T01:41:29.010 INFO:tasks.rados.rados.0.smithi071.stdout:1758: finishing write tid 4 to smithi07134777-44 2024-09-11T01:41:29.010 INFO:tasks.rados.rados.0.smithi071.stdout:1758: oid 44 updating version 0 to 322 2024-09-11T01:41:29.010 INFO:tasks.rados.rados.0.smithi071.stdout:1758: oid 44 version 322 is already newer than 321 2024-09-11T01:41:29.010 INFO:tasks.rados.rados.0.smithi071.stdout:1758: oid 44 updating version 322 to 323 2024-09-11T01:41:29.010 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 323 (ObjNum 645 snap 196 seq_num 645) dirty exists 2024-09-11T01:41:29.010 INFO:tasks.rados.rados.0.smithi071.stdout:1758: left oid 44 (ObjNum 645 snap 196 seq_num 645) 2024-09-11T01:41:29.012 INFO:tasks.rados.rados.0.smithi071.stdout:1760: expect (ObjNum 494 snap 132 seq_num 494) 2024-09-11T01:41:29.012 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:41:29.088 INFO:tasks.rados.rados.0.smithi071.stdout:1759: expect (ObjNum 597 snap 170 seq_num 597) 2024-09-11T01:41:29.088 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:41:29.118 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1503330 2024-09-11T01:41:29.138 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 272466 2024-09-11T01:41:29.261 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:41:29.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.375+0000 7f7f14633640 10 monclient: get_monmap_and_config 2024-09-11T01:41:29.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.375+0000 7f7f14633640 10 monclient: build_initial_monmap 2024-09-11T01:41:29.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.375+0000 7f7f14633640 10 monclient: monmap: 2024-09-11T01:41:29.376 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:29.376 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:29.376 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:41:29.375922+0000 2024-09-11T01:41:29.377 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:41:29.375922+0000 2024-09-11T01:41:29.377 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:29.377 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:29.377 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:41:29.377 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:41:29.377 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:41:29.377 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:41:29.377 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:41:29.377 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:29.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.375+0000 7f7f14633640 1 Processor -- start 2024-09-11T01:41:29.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.375+0000 7f7f14633640 1 -- start start 2024-09-11T01:41:29.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.375+0000 7f7f14633640 10 monclient: init 2024-09-11T01:41:29.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.376+0000 7f7f14633640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:29.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.376+0000 7f7f14633640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-11T01:41:29.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.376+0000 7f7f14633640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7f0c138230 addr v1:172.21.15.71:6789/0 2024-09-11T01:41:29.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.376+0000 7f7f14633640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7f0c09d030 addr v1:172.21.15.12:6789/0 2024-09-11T01:41:29.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.376+0000 7f7f14633640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7f0c09d460 0x7f7f0c1407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:29.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.376+0000 7f7f14633640 10 monclient(hunting): picked mon.noname-a con 0x7f7f0c09d460 addr v2:172.21.15.12:3300/0 2024-09-11T01:41:29.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.376+0000 7f7f14633640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f0c0a6e30 con 0x7f7f0c09d030 2024-09-11T01:41:29.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.376+0000 7f7f14633640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f0c09ba20 con 0x7f7f0c138230 2024-09-11T01:41:29.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.376+0000 7f7f14633640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:29.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.376+0000 7f7f14633640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f7f0c09c9c0 con 0x7f7f0c09d460 2024-09-11T01:41:29.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.376+0000 7f7f14633640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:29.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.376+0000 7f7f14633640 10 monclient(hunting): authenticate will time out at 1495.818604s 2024-09-11T01:41:29.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.376+0000 7f7f11bce640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7f0c09d030 0x7f7f0c13d090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42250/0 (socket says 172.21.15.12:42250) 2024-09-11T01:41:29.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.376+0000 7f7f12bd0640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7f0c09d460 0x7f7f0c1407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:29.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.376+0000 7f7f11bce640 1 -- 172.21.15.12:0/3100912696 learned_addr learned my addr 172.21.15.12:0/3100912696 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:41:29.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.376+0000 7f7f12bd0640 10 monclient(hunting): get_auth_request con 0x7f7f0c09d460 auth_method 0 2024-09-11T01:41:29.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.376+0000 7f7f12bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:29.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.376+0000 7f7f12bd0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:29.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.376+0000 7f7f12bd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:29.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.376+0000 7f7f12bd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:29.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.376+0000 7f7f12bd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:29.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.376+0000 7f7f12bd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:29.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.377+0000 7f7f12bd0640 10 monclient(hunting): handle_auth_done global_id 94446 payload 274 2024-09-11T01:41:29.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.377+0000 7f7f12bd0640 1 -- 172.21.15.12:0/3100912696 >> v1:172.21.15.71:6789/0 conn(0x7f7f0c138230 legacy=0x7f7f0c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:29.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.377+0000 7f7f12bd0640 1 -- 172.21.15.12:0/3100912696 >> v1:172.21.15.12:6789/0 conn(0x7f7f0c09d030 legacy=0x7f7f0c13d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:29.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.377+0000 7f7f12bd0640 10 monclient: _finish_hunting 0 2024-09-11T01:41:29.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.377+0000 7f7f12bd0640 1 monclient: found mon.noname-a 2024-09-11T01:41:29.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.377+0000 7f7f12bd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:29.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.377+0000 7f7f12bd0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:41:29.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.377+0000 7f7f12bd0640 1 -- 172.21.15.12:0/3100912696 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f0c0a66a0 con 0x7f7f0c09d460 2024-09-11T01:41:29.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.379+0000 7f7f12bd0640 1 -- 172.21.15.12:0/3100912696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f0c09d460 msgr2=0x7f7f0c1407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:29.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.380+0000 7f7f12bd0640 1 --2- 172.21.15.12:0/3100912696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f0c09d460 0x7f7f0c1407c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009d90 tx=0x7f7f0402a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae70d9502bd5813f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:29.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.380+0000 7f7f113cd640 1 -- 172.21.15.12:0/3100912696 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f04007a30 con 0x7f7f0c09d460 2024-09-11T01:41:29.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.380+0000 7f7f113cd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:29.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.380+0000 7f7f113cd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:41:29.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.380+0000 7f7f113cd640 10 monclient: dump: 2024-09-11T01:41:29.386 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:29.386 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:29.386 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.386 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.386 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:29.386 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:29.386 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:29.386 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:29.386 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:29.386 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:29.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.380+0000 7f7f113cd640 10 monclient: _finish_auth 0 2024-09-11T01:41:29.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.380+0000 7f7f113cd640 10 monclient: _check_auth_tickets 2024-09-11T01:41:29.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.380+0000 7f7f113cd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:29.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.380+0000 7f7f113cd640 1 -- 172.21.15.12:0/3100912696 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f04007bd0 con 0x7f7f0c09d460 2024-09-11T01:41:29.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.380+0000 7f7f14633640 5 monclient: authenticate success, global_id 94446 2024-09-11T01:41:29.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.380+0000 7f7f14633640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:41:29.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.380+0000 7f7f113cd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:29.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.380+0000 7f7f113cd640 1 -- 172.21.15.12:0/3100912696 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f04031400 con 0x7f7f0c09d460 2024-09-11T01:41:29.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.380+0000 7f7f14633640 10 monclient: get_monmap_and_config success 2024-09-11T01:41:29.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.380+0000 7f7f14633640 4 set_mon_vals no callback set 2024-09-11T01:41:29.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.380+0000 7f7f113cd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:29.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.380+0000 7f7f113cd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:29.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.380+0000 7f7f113cd640 10 monclient: dump: 2024-09-11T01:41:29.387 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:29.387 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:29.388 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.388 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.388 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:29.388 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:29.388 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:29.388 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:29.388 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:29.388 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:29.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.380+0000 7f7f14633640 10 monclient: shutdown 2024-09-11T01:41:29.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.380+0000 7f7f14633640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:29.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.380+0000 7f7f14633640 1 -- 172.21.15.12:0/3100912696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f0c09d460 msgr2=0x7f7f0c1407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:29.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.380+0000 7f7f14633640 1 --2- 172.21.15.12:0/3100912696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f0c09d460 0x7f7f0c1407c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009d90 tx=0x7f7f0402a490 comp rx=0 tx=0).stop 2024-09-11T01:41:29.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.380+0000 7f7f14633640 1 -- 172.21.15.12:0/3100912696 shutdown_connections 2024-09-11T01:41:29.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.380+0000 7f7f14633640 1 --2- 172.21.15.12:0/3100912696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f0c09d460 0x7f7f0c1407c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:29.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.380+0000 7f7f14633640 1 -- 172.21.15.12:0/3100912696 >> 172.21.15.12:0/3100912696 conn(0x7f7f0c0a7120 msgr2=0x7f7f0c0a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:29.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.380+0000 7f7f14633640 1 -- 172.21.15.12:0/3100912696 shutdown_connections 2024-09-11T01:41:29.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.381+0000 7f7f14633640 1 -- 172.21.15.12:0/3100912696 wait complete. 2024-09-11T01:41:29.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.381+0000 7f7f14633640 10 monclient: build_initial_monmap 2024-09-11T01:41:29.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.381+0000 7f7f14633640 10 monclient: monmap: 2024-09-11T01:41:29.389 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:29.389 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:41:29.389 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:41:29.389 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:41:29.389 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:29.389 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:29.389 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:41:29.389 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:41:29.390 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:41:29.390 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:29.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.381+0000 7f7f14633640 1 Processor -- start 2024-09-11T01:41:29.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.381+0000 7f7f14633640 1 -- start start 2024-09-11T01:41:29.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.381+0000 7f7f14633640 10 monclient: init 2024-09-11T01:41:29.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f14633640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:29.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f14633640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:41:29.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f14633640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7f0c09d030 0x7f7f0c136d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:29.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f14633640 10 monclient(hunting): picked mon.noname-c con 0x7f7f0c09d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:29.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f14633640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7f0c09d460 0x7f7f0c13c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:29.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f14633640 10 monclient(hunting): picked mon.noname-b con 0x7f7f0c09d460 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:41:29.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f14633640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f0c138230 0x7f7f0c13ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:29.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f14633640 10 monclient(hunting): picked mon.noname-a con 0x7f7f0c138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:41:29.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f14633640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:29.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f14633640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7f0c140e80 con 0x7f7f0c138230 2024-09-11T01:41:29.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f14633640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:29.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f14633640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f7f0c140d00 con 0x7f7f0c09d460 2024-09-11T01:41:29.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f14633640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:29.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f14633640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7f0c141000 con 0x7f7f0c09d030 2024-09-11T01:41:29.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f14633640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:29.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f14633640 10 monclient(hunting): authenticate will time out at 1495.824470s 2024-09-11T01:41:29.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f123cf640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7f0c09d030 0x7f7f0c136d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:29.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f123cf640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7f0c09d030 0x7f7f0c136d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:56012/0 (socket says 172.21.15.12:56012) 2024-09-11T01:41:29.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f123cf640 1 -- 172.21.15.12:0/914117853 learned_addr learned my addr 172.21.15.12:0/914117853 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:41:29.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f123cf640 10 monclient(hunting): get_auth_request con 0x7f7f0c09d030 auth_method 0 2024-09-11T01:41:29.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f123cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:29.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f123cf640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:29.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f11bce640 1 --2- 172.21.15.12:0/914117853 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7f0c09d460 0x7f7f0c13c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:29.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f123cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:29.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f12bd0640 1 --2- 172.21.15.12:0/914117853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f0c138230 0x7f7f0c13ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:29.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f11bce640 10 monclient(hunting): get_auth_request con 0x7f7f0c09d460 auth_method 0 2024-09-11T01:41:29.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f11bce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:29.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f11bce640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:29.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f11bce640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:29.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f12bd0640 10 monclient(hunting): get_auth_request con 0x7f7f0c138230 auth_method 0 2024-09-11T01:41:29.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f12bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:29.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f12bd0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:29.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f12bd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:29.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f123cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:29.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f123cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:29.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f123cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:29.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f11bce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:29.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f11bce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:29.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f11bce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:29.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f12bd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:29.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f12bd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:29.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f12bd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:29.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f11bce640 10 monclient(hunting): handle_auth_done global_id 134098 payload 824 2024-09-11T01:41:29.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f11bce640 1 -- 172.21.15.12:0/914117853 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7f0c09d030 msgr2=0x7f7f0c136d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:29.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f11bce640 1 --2- 172.21.15.12:0/914117853 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7f0c09d030 0x7f7f0c136d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:29.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f11bce640 1 -- 172.21.15.12:0/914117853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f0c138230 msgr2=0x7f7f0c13ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:29.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f11bce640 1 --2- 172.21.15.12:0/914117853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f0c138230 0x7f7f0c13ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:29.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f11bce640 10 monclient: _finish_hunting 0 2024-09-11T01:41:29.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.382+0000 7f7f11bce640 1 monclient: found mon.noname-b 2024-09-11T01:41:29.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.383+0000 7f7f11bce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:29.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.383+0000 7f7f11bce640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-09-11T01:41:29.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.383+0000 7f7f11bce640 1 -- 172.21.15.12:0/914117853 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f0c13e810 con 0x7f7f0c09d460 2024-09-11T01:41:29.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.383+0000 7f7f12bd0640 1 --2- 172.21.15.12:0/914117853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f0c138230 0x7f7f0c13ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:41:29.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.383+0000 7f7f123cf640 1 --2- 172.21.15.12:0/914117853 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7f0c09d030 0x7f7f0c136d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:41:29.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.385+0000 7f7f11bce640 1 -- 172.21.15.12:0/914117853 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7f0c09d460 msgr2=0x7f7f0c13c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:29.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.385+0000 7f7f11bce640 1 --2- 172.21.15.12:0/914117853 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7f0c09d460 0x7f7f0c13c8f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7f0000c3e0 tx=0x7f7f0000c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:29.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.385+0000 7f7efeffd640 1 -- 172.21.15.12:0/914117853 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f00017070 con 0x7f7f0c09d460 2024-09-11T01:41:29.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.385+0000 7f7efeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:29.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.385+0000 7f7efeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:41:29.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.385+0000 7f7efeffd640 10 monclient: dump: 2024-09-11T01:41:29.396 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:29.396 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:29.396 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.396 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.396 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:29.396 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:29.396 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:29.396 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:29.396 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:29.396 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:29.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.385+0000 7f7efeffd640 10 monclient: _finish_auth 0 2024-09-11T01:41:29.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.385+0000 7f7efeffd640 10 monclient: _check_auth_tickets 2024-09-11T01:41:29.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.385+0000 7f7efeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:29.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.385+0000 7f7efeffd640 1 -- 172.21.15.12:0/914117853 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f000070d0 con 0x7f7f0c09d460 2024-09-11T01:41:29.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.385+0000 7f7efeffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:29.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.385+0000 7f7efeffd640 1 -- 172.21.15.12:0/914117853 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f000074c0 con 0x7f7f0c09d460 2024-09-11T01:41:29.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.385+0000 7f7efeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:29.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.385+0000 7f7efeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:41:29.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.385+0000 7f7efeffd640 10 monclient: dump: 2024-09-11T01:41:29.397 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:29.397 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:29.397 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.397 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.397 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:29.398 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:29.398 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:29.398 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:29.398 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:29.398 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:29.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.385+0000 7f7f14633640 5 monclient: authenticate success, global_id 134098 2024-09-11T01:41:29.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.385+0000 7f7f14633640 10 monclient: _renew_subs 2024-09-11T01:41:29.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.385+0000 7f7f14633640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:41:29.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.385+0000 7f7f14633640 1 -- 172.21.15.12:0/914117853 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f0c13e3b0 con 0x7f7f0c09d460 2024-09-11T01:41:29.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.385+0000 7f7efffff640 4 set_mon_vals no callback set 2024-09-11T01:41:29.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.385+0000 7f7f14633640 10 monclient: _renew_subs 2024-09-11T01:41:29.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.385+0000 7f7f14633640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:41:29.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.385+0000 7f7f14633640 1 -- 172.21.15.12:0/914117853 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f0c13f350 con 0x7f7f0c09d460 2024-09-11T01:41:29.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.386+0000 7f7efeffd640 1 -- 172.21.15.12:0/914117853 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f7f0001b3b0 con 0x7f7f0c09d460 2024-09-11T01:41:29.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.386+0000 7f7f14633640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:29.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.386+0000 7f7f14633640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:29.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.386+0000 7f7f14633640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:41:29.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.386+0000 7f7f14633640 1 -- 172.21.15.12:0/914117853 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f0c138610 con 0x7f7f0c09d460 2024-09-11T01:41:29.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.386+0000 7f7efeffd640 1 --2- 172.21.15.12:0/914117853 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7ef003e260 0x7f7ef0040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:29.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.386+0000 7f7efeffd640 1 -- 172.21.15.12:0/914117853 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(405..405 src has 285..405) ==== 4783+0+0 (secure 0 0 0) 0x7f7f00052500 con 0x7f7f0c09d460 2024-09-11T01:41:29.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.386+0000 7f7f123cf640 1 --2- 172.21.15.12:0/914117853 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7ef003e260 0x7f7ef0040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:29.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.386+0000 7f7f123cf640 10 monclient: get_auth_request con 0x7f7ef003e260 auth_method 0 2024-09-11T01:41:29.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.389+0000 7f7f123cf640 1 --2- 172.21.15.12:0/914117853 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7ef003e260 0x7f7ef0040720 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7f080043e0 tx=0x7f7f08007ef0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:29.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.391+0000 7f7efeffd640 1 -- 172.21.15.12:0/914117853 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7f00050f00 con 0x7f7f0c09d460 2024-09-11T01:41:29.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.391+0000 7f7efeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:29.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.391+0000 7f7efeffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:41:29.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.645+0000 7f7f14633640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:41:29.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.645+0000 7f7f14633640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:41:29.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.646+0000 7f7f14633640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:41:29.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.646+0000 7f7f14633640 1 -- 172.21.15.12:0/914117853 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7f0c140390 con 0x7f7f0c09d460 2024-09-11T01:41:29.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.646+0000 7f7efeffd640 1 -- 172.21.15.12:0/914117853 <== mon.1 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f7f000521e0 con 0x7f7f0c09d460 2024-09-11T01:41:29.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.646+0000 7f7efeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:41:29.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.646+0000 7f7efeffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:41:29.648 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:41:29.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.648+0000 7f7f14633640 1 -- 172.21.15.12:0/914117853 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7ef003e260 msgr2=0x7f7ef0040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:29.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.648+0000 7f7f14633640 1 --2- 172.21.15.12:0/914117853 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7ef003e260 0x7f7ef0040720 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7f080043e0 tx=0x7f7f08007ef0 comp rx=0 tx=0).stop 2024-09-11T01:41:29.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.648+0000 7f7f14633640 10 monclient: shutdown 2024-09-11T01:41:29.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.648+0000 7f7f14633640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:29.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.648+0000 7f7f14633640 1 -- 172.21.15.12:0/914117853 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7f0c09d460 msgr2=0x7f7f0c13c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:29.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.648+0000 7f7f14633640 1 --2- 172.21.15.12:0/914117853 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7f0c09d460 0x7f7f0c13c8f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7f0000c3e0 tx=0x7f7f0000c8a0 comp rx=0 tx=0).stop 2024-09-11T01:41:29.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.648+0000 7f7f14633640 1 -- 172.21.15.12:0/914117853 shutdown_connections 2024-09-11T01:41:29.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.648+0000 7f7f14633640 1 --2- 172.21.15.12:0/914117853 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7ef003e260 0x7f7ef0040720 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:29.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.648+0000 7f7f14633640 1 --2- 172.21.15.12:0/914117853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7f0c138230 0x7f7f0c13ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:29.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.649+0000 7f7f14633640 1 --2- 172.21.15.12:0/914117853 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7f0c09d460 0x7f7f0c13c8f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:29.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.649+0000 7f7f14633640 1 --2- 172.21.15.12:0/914117853 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7f0c09d030 0x7f7f0c136d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:29.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.649+0000 7f7f14633640 1 -- 172.21.15.12:0/914117853 >> 172.21.15.12:0/914117853 conn(0x7f7f0c0a7120 msgr2=0x7f7f0c0a8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:29.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.649+0000 7f7f14633640 1 -- 172.21.15.12:0/914117853 shutdown_connections 2024-09-11T01:41:29.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.649+0000 7f7f14633640 1 -- 172.21.15.12:0/914117853 wait complete. 2024-09-11T01:41:29.660 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:41:29.661 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:41:29.661 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:41:29.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.761+0000 7ff1f0190640 10 monclient: get_monmap_and_config 2024-09-11T01:41:29.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.761+0000 7ff1f0190640 10 monclient: build_initial_monmap 2024-09-11T01:41:29.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.761+0000 7ff1f0190640 10 monclient: monmap: 2024-09-11T01:41:29.763 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:29.763 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:29.763 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:41:29.762647+0000 2024-09-11T01:41:29.764 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:41:29.762647+0000 2024-09-11T01:41:29.764 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:29.764 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:29.764 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:41:29.764 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:41:29.764 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:41:29.764 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:41:29.764 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:41:29.764 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.762+0000 7ff1f0190640 1 Processor -- start 2024-09-11T01:41:29.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.762+0000 7ff1f0190640 1 -- start start 2024-09-11T01:41:29.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.762+0000 7ff1f0190640 10 monclient: init 2024-09-11T01:41:29.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1f0190640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:29.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1f0190640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-11T01:41:29.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1f0190640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff1e8134ad0 0x7ff1e8134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:29.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1f0190640 10 monclient(hunting): picked mon.noname-a con 0x7ff1e8134ad0 addr v2:172.21.15.12:3300/0 2024-09-11T01:41:29.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1f0190640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff1e80a2200 0x7ff1e81353f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:29.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1f0190640 10 monclient(hunting): picked mon.noname-c con 0x7ff1e80a2200 addr v2:172.21.15.71:3300/0 2024-09-11T01:41:29.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1f0190640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff1e8133a50 0x7ff1e8133f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:29.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1f0190640 10 monclient(hunting): picked mon.noname-b con 0x7ff1e8133a50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:29.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1f0190640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:29.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1f0190640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff1e809f5d0 con 0x7ff1e8134ad0 2024-09-11T01:41:29.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1f0190640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:29.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1f0190640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7ff1e8135930 con 0x7ff1e80a2200 2024-09-11T01:41:29.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1f0190640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:29.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1f0190640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff1e809ed30 con 0x7ff1e8133a50 2024-09-11T01:41:29.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1f0190640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:29.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1f0190640 10 monclient(hunting): authenticate will time out at 1496.205260s 2024-09-11T01:41:29.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1edf2c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff1e8134ad0 0x7ff1e8134eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:29.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1ee72d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff1e8133a50 0x7ff1e8133f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:29.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1edf2c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff1e8134ad0 0x7ff1e8134eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58594/0 (socket says 172.21.15.12:58594) 2024-09-11T01:41:29.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1edf2c640 1 -- 172.21.15.12:0/2733543249 learned_addr learned my addr 172.21.15.12:0/2733543249 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:41:29.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1edf2c640 10 monclient(hunting): get_auth_request con 0x7ff1e8134ad0 auth_method 0 2024-09-11T01:41:29.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1edf2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:29.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1edf2c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:29.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1edf2c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:29.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1ee72d640 10 monclient(hunting): get_auth_request con 0x7ff1e8133a50 auth_method 0 2024-09-11T01:41:29.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1ee72d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:29.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1ee72d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:29.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1ee72d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:29.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1ee72d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:29.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1ee72d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:29.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1ee72d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:29.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1edf2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:29.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1edf2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:29.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1edf2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:29.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1ed72b640 1 --2- 172.21.15.12:0/2733543249 >> v2:172.21.15.71:3300/0 conn(0x7ff1e80a2200 0x7ff1e81353f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:29.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1ed72b640 10 monclient(hunting): get_auth_request con 0x7ff1e80a2200 auth_method 0 2024-09-11T01:41:29.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1ed72b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:29.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1ed72b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:29.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1ed72b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:29.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1ee72d640 10 monclient(hunting): handle_auth_done global_id 114281 payload 274 2024-09-11T01:41:29.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1ee72d640 1 -- 172.21.15.12:0/2733543249 >> v2:172.21.15.71:3300/0 conn(0x7ff1e80a2200 msgr2=0x7ff1e81353f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:29.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1ee72d640 1 --2- 172.21.15.12:0/2733543249 >> v2:172.21.15.71:3300/0 conn(0x7ff1e80a2200 0x7ff1e81353f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:29.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1ee72d640 1 -- 172.21.15.12:0/2733543249 >> v2:172.21.15.12:3300/0 conn(0x7ff1e8134ad0 msgr2=0x7ff1e8134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:29.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1ee72d640 1 --2- 172.21.15.12:0/2733543249 >> v2:172.21.15.12:3300/0 conn(0x7ff1e8134ad0 0x7ff1e8134eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:29.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1ee72d640 10 monclient: _finish_hunting 0 2024-09-11T01:41:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1ee72d640 1 monclient: found mon.noname-b 2024-09-11T01:41:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1ee72d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1ee72d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:41:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1ee72d640 1 -- 172.21.15.12:0/2733543249 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1e809f010 con 0x7ff1e8133a50 2024-09-11T01:41:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.763+0000 7ff1edf2c640 1 --2- 172.21.15.12:0/2733543249 >> v2:172.21.15.12:3300/0 conn(0x7ff1e8134ad0 0x7ff1e8134eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:41:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.766+0000 7ff1ee72d640 1 -- 172.21.15.12:0/2733543249 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff1e8133a50 msgr2=0x7ff1e8133f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.766+0000 7ff1ee72d640 1 --2- 172.21.15.12:0/2733543249 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff1e8133a50 0x7ff1e8133f00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff1e00097a0 tx=0x7ff1e002c220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76713791ac2b87f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.766+0000 7ff1ecf2a640 1 -- 172.21.15.12:0/2733543249 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e0039070 con 0x7ff1e8133a50 2024-09-11T01:41:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.766+0000 7ff1ecf2a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.766+0000 7ff1ecf2a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:41:29.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.766+0000 7ff1ecf2a640 10 monclient: dump: 2024-09-11T01:41:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:29.777 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.766+0000 7ff1ecf2a640 10 monclient: _finish_auth 0 2024-09-11T01:41:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.766+0000 7ff1ecf2a640 10 monclient: _check_auth_tickets 2024-09-11T01:41:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.766+0000 7ff1ecf2a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:29.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.766+0000 7ff1ecf2a640 1 -- 172.21.15.12:0/2733543249 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1e0034640 con 0x7ff1e8133a50 2024-09-11T01:41:29.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.766+0000 7ff1f0190640 5 monclient: authenticate success, global_id 114281 2024-09-11T01:41:29.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.766+0000 7ff1f0190640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:41:29.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.766+0000 7ff1ecf2a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:29.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.766+0000 7ff1f0190640 10 monclient: get_monmap_and_config success 2024-09-11T01:41:29.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.766+0000 7ff1ecf2a640 1 -- 172.21.15.12:0/2733543249 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e0034a10 con 0x7ff1e8133a50 2024-09-11T01:41:29.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.766+0000 7ff1ecf2a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:29.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.766+0000 7ff1f0190640 4 set_mon_vals no callback set 2024-09-11T01:41:29.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.766+0000 7ff1ecf2a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:41:29.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.766+0000 7ff1ecf2a640 10 monclient: dump: 2024-09-11T01:41:29.778 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:29.778 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:29.778 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.778 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.779 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:29.779 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:29.779 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:29.779 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:29.779 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:29.779 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:29.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.767+0000 7ff1f0190640 10 monclient: shutdown 2024-09-11T01:41:29.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.767+0000 7ff1f0190640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:29.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.767+0000 7ff1f0190640 1 -- 172.21.15.12:0/2733543249 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff1e8133a50 msgr2=0x7ff1e8133f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:29.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.767+0000 7ff1f0190640 1 --2- 172.21.15.12:0/2733543249 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff1e8133a50 0x7ff1e8133f00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff1e00097a0 tx=0x7ff1e002c220 comp rx=0 tx=0).stop 2024-09-11T01:41:29.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.767+0000 7ff1f0190640 1 -- 172.21.15.12:0/2733543249 shutdown_connections 2024-09-11T01:41:29.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.767+0000 7ff1f0190640 1 --2- 172.21.15.12:0/2733543249 >> v2:172.21.15.71:3300/0 conn(0x7ff1e80a2200 0x7ff1e81353f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:29.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.767+0000 7ff1f0190640 1 --2- 172.21.15.12:0/2733543249 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff1e8133a50 0x7ff1e8133f00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:29.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.767+0000 7ff1f0190640 1 --2- 172.21.15.12:0/2733543249 >> v2:172.21.15.12:3300/0 conn(0x7ff1e8134ad0 0x7ff1e8134eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:29.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.767+0000 7ff1f0190640 1 -- 172.21.15.12:0/2733543249 >> 172.21.15.12:0/2733543249 conn(0x7ff1e80a2f70 msgr2=0x7ff1e80a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:29.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.767+0000 7ff1f0190640 1 -- 172.21.15.12:0/2733543249 shutdown_connections 2024-09-11T01:41:29.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.767+0000 7ff1f0190640 1 -- 172.21.15.12:0/2733543249 wait complete. 2024-09-11T01:41:29.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.767+0000 7ff1f0190640 10 monclient: build_initial_monmap 2024-09-11T01:41:29.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.767+0000 7ff1f0190640 10 monclient: monmap: 2024-09-11T01:41:29.780 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:29.780 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:41:29.780 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:41:29.780 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:41:29.780 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:29.780 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:29.780 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:41:29.780 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:41:29.781 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:41:29.781 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:29.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1f0190640 1 Processor -- start 2024-09-11T01:41:29.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1f0190640 1 -- start start 2024-09-11T01:41:29.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1f0190640 10 monclient: init 2024-09-11T01:41:29.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1f0190640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:29.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1f0190640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:41:29.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1f0190640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1e80a2200 0x7ff1e81cd440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:29.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1f0190640 10 monclient(hunting): picked mon.noname-b con 0x7ff1e80a2200 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:41:29.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1f0190640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1e8133a50 0x7ff1e8123570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:29.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1f0190640 10 monclient(hunting): picked mon.noname-a con 0x7ff1e8133a50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:41:29.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1f0190640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff1e8134ad0 0x7ff1e8123ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:29.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1f0190640 10 monclient(hunting): picked mon.noname-c con 0x7ff1e8134ad0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:29.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1f0190640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:29.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1f0190640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e809ffd0 con 0x7ff1e8133a50 2024-09-11T01:41:29.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1f0190640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:29.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1f0190640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e809f380 con 0x7ff1e80a2200 2024-09-11T01:41:29.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1f0190640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:29.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1f0190640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff1e80a0150 con 0x7ff1e8134ad0 2024-09-11T01:41:29.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1f0190640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:29.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1f0190640 10 monclient(hunting): authenticate will time out at 1496.210735s 2024-09-11T01:41:29.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1ed72b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1e8133a50 0x7ff1e8123570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:29.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1ee72d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff1e8134ad0 0x7ff1e8123ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:29.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1ed72b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1e8133a50 0x7ff1e8123570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58606/0 (socket says 172.21.15.12:58606) 2024-09-11T01:41:29.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1ed72b640 1 -- 172.21.15.12:0/2325820131 learned_addr learned my addr 172.21.15.12:0/2325820131 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:41:29.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1ed72b640 10 monclient(hunting): get_auth_request con 0x7ff1e8133a50 auth_method 0 2024-09-11T01:41:29.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1ed72b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:29.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1ed72b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:29.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1ed72b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:29.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1ee72d640 10 monclient(hunting): get_auth_request con 0x7ff1e8134ad0 auth_method 0 2024-09-11T01:41:29.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1ee72d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:29.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1ee72d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:29.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1ee72d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:29.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1ed72b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:29.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1ed72b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:29.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1ed72b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:29.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1ee72d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:29.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1ee72d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:29.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1edf2c640 1 --2- 172.21.15.12:0/2325820131 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1e80a2200 0x7ff1e81cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:29.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1ee72d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:29.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1edf2c640 10 monclient(hunting): get_auth_request con 0x7ff1e80a2200 auth_method 0 2024-09-11T01:41:29.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1edf2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:29.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1edf2c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:29.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.768+0000 7ff1edf2c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:29.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.769+0000 7ff1edf2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:29.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.769+0000 7ff1edf2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:29.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.769+0000 7ff1edf2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:29.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.769+0000 7ff1ed72b640 10 monclient(hunting): handle_auth_done global_id 94455 payload 824 2024-09-11T01:41:29.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.769+0000 7ff1ed72b640 1 -- 172.21.15.12:0/2325820131 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff1e8134ad0 msgr2=0x7ff1e8123ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:29.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.769+0000 7ff1ed72b640 1 --2- 172.21.15.12:0/2325820131 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff1e8134ad0 0x7ff1e8123ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:29.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.769+0000 7ff1ed72b640 1 -- 172.21.15.12:0/2325820131 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1e80a2200 msgr2=0x7ff1e81cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:29.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.769+0000 7ff1ed72b640 1 --2- 172.21.15.12:0/2325820131 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1e80a2200 0x7ff1e81cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:29.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.769+0000 7ff1ed72b640 10 monclient: _finish_hunting 0 2024-09-11T01:41:29.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.769+0000 7ff1ed72b640 1 monclient: found mon.noname-a 2024-09-11T01:41:29.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.769+0000 7ff1ed72b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:29.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.769+0000 7ff1ed72b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:41:29.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.769+0000 7ff1ed72b640 1 -- 172.21.15.12:0/2325820131 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1e80a1500 con 0x7ff1e8133a50 2024-09-11T01:41:29.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.771+0000 7ff1ed72b640 1 -- 172.21.15.12:0/2325820131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1e8133a50 msgr2=0x7ff1e8123570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:29.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.771+0000 7ff1ed72b640 1 --2- 172.21.15.12:0/2325820131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1e8133a50 0x7ff1e8123570 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc00d7d0 tx=0x7ff1dc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:29.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.771+0000 7ff1e67fc640 1 -- 172.21.15.12:0/2325820131 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc004020 con 0x7ff1e8133a50 2024-09-11T01:41:29.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.771+0000 7ff1e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:29.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.771+0000 7ff1e67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:41:29.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.771+0000 7ff1e67fc640 10 monclient: dump: 2024-09-11T01:41:29.790 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:29.790 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:29.791 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.791 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.791 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:29.791 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:29.791 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:29.791 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:29.791 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:29.791 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:29.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.771+0000 7ff1e67fc640 10 monclient: _finish_auth 0 2024-09-11T01:41:29.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.771+0000 7ff1e67fc640 10 monclient: _check_auth_tickets 2024-09-11T01:41:29.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.771+0000 7ff1e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:29.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.771+0000 7ff1f0190640 5 monclient: authenticate success, global_id 94455 2024-09-11T01:41:29.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.771+0000 7ff1f0190640 10 monclient: _renew_subs 2024-09-11T01:41:29.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.771+0000 7ff1f0190640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:29.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.771+0000 7ff1e67fc640 1 -- 172.21.15.12:0/2325820131 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1dc0041e0 con 0x7ff1e8133a50 2024-09-11T01:41:29.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.771+0000 7ff1f0190640 1 -- 172.21.15.12:0/2325820131 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1e8127d90 con 0x7ff1e8133a50 2024-09-11T01:41:29.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.771+0000 7ff1e67fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:29.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.771+0000 7ff1e67fc640 1 -- 172.21.15.12:0/2325820131 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc0043d0 con 0x7ff1e8133a50 2024-09-11T01:41:29.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.771+0000 7ff1e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:29.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.771+0000 7ff1e67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:29.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.771+0000 7ff1e67fc640 10 monclient: dump: 2024-09-11T01:41:29.794 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:29.794 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:29.794 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.794 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.794 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:29.794 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:29.794 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:29.794 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:29.795 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:29.795 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:29.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.771+0000 7ff1e7fff640 4 set_mon_vals no callback set 2024-09-11T01:41:29.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.771+0000 7ff1f0190640 10 monclient: _renew_subs 2024-09-11T01:41:29.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.771+0000 7ff1f0190640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:29.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.771+0000 7ff1f0190640 1 -- 172.21.15.12:0/2325820131 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1e8127390 con 0x7ff1e8133a50 2024-09-11T01:41:29.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.772+0000 7ff1f0190640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:29.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.772+0000 7ff1f0190640 1 --2- 172.21.15.12:0/2325820131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1e81cd980 0x7ff1e8123ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:29.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.772+0000 7ff1f0190640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:29.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.772+0000 7ff1f0190640 1 -- 172.21.15.12:0/2325820131 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e8134560 con 0x7ff1e81cd980 2024-09-11T01:41:29.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.772+0000 7ff1ed72b640 1 --2- 172.21.15.12:0/2325820131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1e8133a50 0x7ff1e8123570 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc00d7d0 tx=0x7ff1dc00dc90 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7ff1dc013070 mgrmap(e 5) 2024-09-11T01:41:29.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.772+0000 7ff1edf2c640 1 --2- 172.21.15.12:0/2325820131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1e81cd980 0x7ff1e8123ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:29.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.772+0000 7ff1edf2c640 10 monclient: get_auth_request con 0x7ff1e81cd980 auth_method 0 2024-09-11T01:41:29.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.772+0000 7ff1edf2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:29.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.772+0000 7ff1edf2c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:29.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.772+0000 7ff1edf2c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:29.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.772+0000 7ff1e67fc640 1 -- 172.21.15.12:0/2325820131 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff1dc013070 con 0x7ff1e8133a50 2024-09-11T01:41:29.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.772+0000 7ff1edf2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:29.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.772+0000 7ff1edf2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:29.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.772+0000 7ff1edf2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:29.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.772+0000 7ff1edf2c640 10 monclient(hunting): handle_auth_done global_id 94458 payload 824 2024-09-11T01:41:29.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.772+0000 7ff1edf2c640 1 -- 172.21.15.12:0/2325820131 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff1e8126f40 con 0x7ff1e81cd980 2024-09-11T01:41:29.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.772+0000 7ff1e67fc640 1 --2- 172.21.15.12:0/2325820131 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff1d403e210 0x7ff1d40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:29.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.773+0000 7ff1ee72d640 1 --2- 172.21.15.12:0/2325820131 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff1d403e210 0x7ff1d40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:29.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.773+0000 7ff1ee72d640 10 monclient: get_auth_request con 0x7ff1d403e210 auth_method 0 2024-09-11T01:41:29.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.775+0000 7ff1edf2c640 1 -- 172.21.15.12:0/2325820131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1e81cd980 msgr2=0x7ff1e8123ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:29.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.775+0000 7ff1edf2c640 1 --2- 172.21.15.12:0/2325820131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1e81cd980 0x7ff1e8123ff0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8009e60 tx=0x7ff1d800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:29.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.775+0000 7ff1e67fc640 1 -- 172.21.15.12:0/2325820131 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8019070 con 0x7ff1e81cd980 2024-09-11T01:41:29.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.775+0000 7ff1e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:29.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.775+0000 7ff1e67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:29.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.775+0000 7ff1e67fc640 10 monclient: dump: 2024-09-11T01:41:29.798 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:29.799 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:29.799 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.799 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.799 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:29.799 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:29.799 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:29.799 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:29.799 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:29.799 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:29.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.775+0000 7ff1e67fc640 1 -- 172.21.15.12:0/2325820131 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff1d8015630 con 0x7ff1e81cd980 2024-09-11T01:41:29.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.775+0000 7ff1e67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:29.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.775+0000 7ff1e67fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:41:29.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.775+0000 7ff1e67fc640 1 -- 172.21.15.12:0/2325820131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1e81cd980 msgr2=0x7ff1e8123ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:29.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.775+0000 7ff1e67fc640 1 --2- 172.21.15.12:0/2325820131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1e81cd980 0x7ff1e8123ff0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8009e60 tx=0x7ff1d800f580 comp rx=0 tx=0).stop 2024-09-11T01:41:29.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.775+0000 7ff1e67fc640 1 -- 172.21.15.12:0/2325820131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1e81cd980 msgr2=0x7ff1e8123ff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:41:29.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.775+0000 7ff1e67fc640 1 --2- 172.21.15.12:0/2325820131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1e81cd980 0x7ff1e8123ff0 secure :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8009e60 tx=0x7ff1d800f580 comp rx=0 tx=0).stop 2024-09-11T01:41:29.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.775+0000 7ff1ee72d640 1 --2- 172.21.15.12:0/2325820131 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff1d403e210 0x7ff1d40406d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff1e80a72b0 tx=0x7ff1e0030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:29.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.781+0000 7ff1f0190640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:41:29.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.781+0000 7ff1f0190640 1 --2- 172.21.15.12:0/2325820131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1e8124530 0x7ff1e80726b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:29.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.781+0000 7ff1f0190640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:29.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.781+0000 7ff1f0190640 1 -- 172.21.15.12:0/2325820131 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e8124940 con 0x7ff1e8124530 2024-09-11T01:41:29.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.781+0000 7ff1edf2c640 1 --2- 172.21.15.12:0/2325820131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1e8124530 0x7ff1e80726b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:29.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.781+0000 7ff1edf2c640 10 monclient: get_auth_request con 0x7ff1e8124530 auth_method 0 2024-09-11T01:41:29.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.781+0000 7ff1edf2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:29.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.781+0000 7ff1edf2c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:29.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.781+0000 7ff1edf2c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:29.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.781+0000 7ff1edf2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:29.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.781+0000 7ff1edf2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:29.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.781+0000 7ff1edf2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:29.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.781+0000 7ff1edf2c640 10 monclient(hunting): handle_auth_done global_id 94461 payload 824 2024-09-11T01:41:29.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.781+0000 7ff1edf2c640 1 -- 172.21.15.12:0/2325820131 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff1e806d480 con 0x7ff1e8124530 2024-09-11T01:41:29.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.784+0000 7ff1edf2c640 1 -- 172.21.15.12:0/2325820131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1e8124530 msgr2=0x7ff1e80726b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:29.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.784+0000 7ff1edf2c640 1 --2- 172.21.15.12:0/2325820131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1e8124530 0x7ff1e80726b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800b010 tx=0x7ff1d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:29.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.784+0000 7ff1e67fc640 1 -- 172.21.15.12:0/2325820131 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d801e950 con 0x7ff1e8124530 2024-09-11T01:41:29.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.784+0000 7ff1e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:29.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.784+0000 7ff1e67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:29.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.784+0000 7ff1e67fc640 10 monclient: dump: 2024-09-11T01:41:29.803 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:29.803 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:29.803 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.803 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.804 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:29.804 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:29.804 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:29.804 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:29.804 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:29.804 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:29.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.784+0000 7ff1e67fc640 1 -- 172.21.15.12:0/2325820131 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7ff1d80073f0 con 0x7ff1e8124530 2024-09-11T01:41:29.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.784+0000 7ff1e67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:41:29.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.784+0000 7ff1e67fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:41:29.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.784+0000 7ff1e67fc640 1 -- 172.21.15.12:0/2325820131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1e8124530 msgr2=0x7ff1e80726b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:29.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.784+0000 7ff1e67fc640 1 --2- 172.21.15.12:0/2325820131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1e8124530 0x7ff1e80726b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800b010 tx=0x7ff1d8004a40 comp rx=0 tx=0).stop 2024-09-11T01:41:29.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.784+0000 7ff1e67fc640 1 -- 172.21.15.12:0/2325820131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1e8124530 msgr2=0x7ff1e80726b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:41:29.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.784+0000 7ff1e67fc640 1 --2- 172.21.15.12:0/2325820131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1e8124530 0x7ff1e80726b0 secure :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800b010 tx=0x7ff1d8004a40 comp rx=0 tx=0).stop 2024-09-11T01:41:29.805 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:41:29.806 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:41:29.806 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:41:29.806 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:41:29.806 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 114, 2024-09-11T01:41:29.806 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 235880, 2024-09-11T01:41:29.806 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:41:29.806 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:41:29.806 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:41:29.806 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:41:29.807 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:29.807 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:41:29.807 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:41:29.807 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:41:29.807 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:41:29.807 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:41:29.807 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:41:29.807 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:41:29.808 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:41:29.808 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:41:29.808 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:41:29.808 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:41:29.808 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:41:29.808 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:41:29.808 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:41:29.808 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:41:29.808 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:29.808 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:41:29.809 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:41:29.809 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:41:29.809 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:41:29.809 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:41:29.809 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:41:29.809 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:41:29.809 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:41:29.809 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:41:29.809 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:41:29.809 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:41:29.809 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:41:29.809 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:41:29.809 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:29.810 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:29.810 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:41:29.810 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:41:29.810 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:41:29.810 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:41:29.810 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:41:29.810 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:41:29.810 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:41:29.810 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:41:29.810 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:41:29.810 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:41:29.810 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:41:29.810 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:41:29.810 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:41:29.811 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:41:29.811 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:41:29.811 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:41:29.811 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:41:29.811 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:41:29.811 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:41:29.811 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:41:29.811 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:41:29.811 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:41:29.811 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:41:29.811 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:41:29.811 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:41:29.811 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:41:29.811 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:41:29.811 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:41:29.812 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:29.812 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:41:29.812 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:29.812 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:41:29.812 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:29.812 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:41:29.812 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:41:29.812 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:41:29.812 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:41:29.812 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:29.812 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:41:29.812 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:41:29.812 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:29.813 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:29.813 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:29.813 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:41:29.813 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:41:29.813 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:29.813 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:29.813 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:29.813 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:29.813 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:41:29.813 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:41:29.813 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:41:29.813 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:41:29.813 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:41:29.813 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:29.814 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:29.814 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:41:29.814 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:41:29.814 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:41:29.814 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:41:29.814 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:29.814 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:41:29.814 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:41:29.814 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:29.814 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:29.814 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:29.814 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:41:29.814 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:41:29.814 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:29.815 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:29.815 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:29.815 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:29.815 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:41:29.815 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:41:29.815 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:41:29.815 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:41:29.815 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:41:29.815 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:29.815 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:29.815 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:41:29.815 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:41:29.815 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:41:29.816 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:41:29.816 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:29.816 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:41:29.816 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:41:29.816 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:29.816 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:29.816 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:29.816 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:41:29.816 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:41:29.816 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:29.816 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:29.816 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:29.817 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:29.817 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:41:29.817 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:41:29.817 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:41:29.817 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:41:29.817 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:41:29.817 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:29.817 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:29.817 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:29.817 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:41:29.817 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:41:29.817 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:29.817 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:41:29.818 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:41:29.818 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:41:29.818 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:29.818 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:29.818 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:41:29.818 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:29.818 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:41:29.818 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:41:29.818 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:41:29.818 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:29.818 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:29.818 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:41:29.818 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:29.818 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:41:29.819 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:41:29.819 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:41:29.819 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:29.819 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:29.819 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:29.819 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:41:29.819 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:41:29.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.784+0000 7ff1f0190640 1 -- 172.21.15.12:0/2325820131 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff1d403e210 msgr2=0x7ff1d40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:29.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.784+0000 7ff1f0190640 1 --2- 172.21.15.12:0/2325820131 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff1d403e210 0x7ff1d40406d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff1e80a72b0 tx=0x7ff1e0030000 comp rx=0 tx=0).stop 2024-09-11T01:41:29.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.784+0000 7ff1f0190640 10 monclient: shutdown 2024-09-11T01:41:29.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.784+0000 7ff1f0190640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:29.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.784+0000 7ff1f0190640 1 -- 172.21.15.12:0/2325820131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1e8133a50 msgr2=0x7ff1e8123570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:29.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.784+0000 7ff1f0190640 1 --2- 172.21.15.12:0/2325820131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1e8133a50 0x7ff1e8123570 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc00d7d0 tx=0x7ff1dc00dc90 comp rx=0 tx=0).stop 2024-09-11T01:41:29.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.784+0000 7ff1ee72d640 1 -- 172.21.15.12:0/2325820131 reap_dead start 2024-09-11T01:41:29.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.784+0000 7ff1e67fc640 1 -- 172.21.15.12:0/2325820131 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(405..405 src has 285..405) ==== 4783+0+0 (unknown 0 0 0) 0x7ff1dc052b50 con 0x7ff1e8133a50 2024-09-11T01:41:29.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.784+0000 7ff1ee72d640 1 -- 172.21.15.12:0/2325820131 reap_dead start 2024-09-11T01:41:29.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.784+0000 7ff1f0190640 1 -- 172.21.15.12:0/2325820131 shutdown_connections 2024-09-11T01:41:29.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.784+0000 7ff1f0190640 1 -- 172.21.15.12:0/2325820131 >> 172.21.15.12:0/2325820131 conn(0x7ff1e80a2f70 msgr2=0x7ff1e809d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:29.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.784+0000 7ff1f0190640 1 -- 172.21.15.12:0/2325820131 shutdown_connections 2024-09-11T01:41:29.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.784+0000 7ff1f0190640 1 -- 172.21.15.12:0/2325820131 wait complete. 2024-09-11T01:41:29.822 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:41:29.822 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:41:29.822 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:41:29.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.922+0000 7f4133abd640 10 monclient: get_monmap_and_config 2024-09-11T01:41:29.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.922+0000 7f4133abd640 10 monclient: build_initial_monmap 2024-09-11T01:41:29.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.922+0000 7f4133abd640 10 monclient: monmap: 2024-09-11T01:41:29.924 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:29.925 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:29.925 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:41:29.923708+0000 2024-09-11T01:41:29.925 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:41:29.923708+0000 2024-09-11T01:41:29.925 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:29.925 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:29.925 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:41:29.925 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:41:29.925 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:41:29.925 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:41:29.925 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:41:29.925 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:29.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.923+0000 7f4133abd640 1 Processor -- start 2024-09-11T01:41:29.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.923+0000 7f4133abd640 1 -- start start 2024-09-11T01:41:29.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.923+0000 7f4133abd640 10 monclient: init 2024-09-11T01:41:29.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4133abd640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:29.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4133abd640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-11T01:41:29.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4133abd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f412c0a2e20 0x7f412c06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:29.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4133abd640 10 monclient(hunting): picked mon.noname-a con 0x7f412c0a2e20 addr v2:172.21.15.12:3300/0 2024-09-11T01:41:29.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4133abd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f412c0a4970 addr v1:172.21.15.12:6789/0 2024-09-11T01:41:29.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4133abd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c06f910 0x7f412c13f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:29.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4133abd640 10 monclient(hunting): picked mon.noname-b con 0x7f412c06f910 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:29.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4133abd640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f412c06f4f0 con 0x7f412c0a4970 2024-09-11T01:41:29.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4133abd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:29.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4133abd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f412c06ef30 con 0x7f412c0a2e20 2024-09-11T01:41:29.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4133abd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:29.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4133abd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f412c06ec50 con 0x7f412c06f910 2024-09-11T01:41:29.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4133abd640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:29.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4133abd640 10 monclient(hunting): authenticate will time out at 1496.366314s 2024-09-11T01:41:29.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4131859640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f412c0a2e20 0x7f412c06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:29.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4131058640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f412c0a4970 0x7f412c0a3200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42264/0 (socket says 172.21.15.12:42264) 2024-09-11T01:41:29.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4131859640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f412c0a2e20 0x7f412c06ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58636/0 (socket says 172.21.15.12:58636) 2024-09-11T01:41:29.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4131058640 1 -- 172.21.15.12:0/1867761931 learned_addr learned my addr 172.21.15.12:0/1867761931 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:41:29.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f413205a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c06f910 0x7f412c13f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:29.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4131859640 10 monclient(hunting): get_auth_request con 0x7f412c0a2e20 auth_method 0 2024-09-11T01:41:29.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4131859640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:29.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4131859640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:29.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4131859640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:29.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f413205a640 10 monclient(hunting): get_auth_request con 0x7f412c06f910 auth_method 0 2024-09-11T01:41:29.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f413205a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:29.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f413205a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:29.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f413205a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:29.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4131859640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:29.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4131859640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:29.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4131859640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:29.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f413205a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:29.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f413205a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:29.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f413205a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:29.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4131859640 10 monclient(hunting): handle_auth_done global_id 94464 payload 274 2024-09-11T01:41:29.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4131859640 1 -- 172.21.15.12:0/1867761931 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c06f910 msgr2=0x7f412c13f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:29.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4131859640 1 --2- 172.21.15.12:0/1867761931 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c06f910 0x7f412c13f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:29.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4131859640 1 -- 172.21.15.12:0/1867761931 >> v1:172.21.15.12:6789/0 conn(0x7f412c0a4970 legacy=0x7f412c0a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:29.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4131859640 10 monclient: _finish_hunting 0 2024-09-11T01:41:29.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4131859640 1 monclient: found mon.noname-a 2024-09-11T01:41:29.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4131859640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:29.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4131859640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:41:29.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f4131859640 1 -- 172.21.15.12:0/1867761931 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f412c0a3910 con 0x7f412c0a2e20 2024-09-11T01:41:29.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.924+0000 7f413205a640 1 --2- 172.21.15.12:0/1867761931 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c06f910 0x7f412c13f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:41:29.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.927+0000 7f4131859640 1 -- 172.21.15.12:0/1867761931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f412c0a2e20 msgr2=0x7f412c06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:29.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.927+0000 7f4131859640 1 --2- 172.21.15.12:0/1867761931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f412c0a2e20 0x7f412c06ded0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f41280097a0 tx=0x7f4128029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2c9f75207f43ee1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:29.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.927+0000 7f4130857640 1 -- 172.21.15.12:0/1867761931 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4128037070 con 0x7f412c0a2e20 2024-09-11T01:41:29.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.927+0000 7f4130857640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:29.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.927+0000 7f4130857640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:41:29.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.927+0000 7f4130857640 10 monclient: dump: 2024-09-11T01:41:29.934 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:29.934 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:29.934 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.934 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.934 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:29.935 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:29.935 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:29.935 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:29.935 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:29.935 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:29.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.927+0000 7f4130857640 10 monclient: _finish_auth 0 2024-09-11T01:41:29.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.927+0000 7f4130857640 10 monclient: _check_auth_tickets 2024-09-11T01:41:29.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.927+0000 7f4130857640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:29.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.927+0000 7f4130857640 1 -- 172.21.15.12:0/1867761931 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4128032440 con 0x7f412c0a2e20 2024-09-11T01:41:29.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.927+0000 7f4133abd640 5 monclient: authenticate success, global_id 94464 2024-09-11T01:41:29.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.927+0000 7f4133abd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:41:29.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.927+0000 7f4130857640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:29.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.927+0000 7f4133abd640 10 monclient: get_monmap_and_config success 2024-09-11T01:41:29.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.927+0000 7f4133abd640 4 set_mon_vals no callback set 2024-09-11T01:41:29.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.927+0000 7f4130857640 1 -- 172.21.15.12:0/1867761931 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4128032810 con 0x7f412c0a2e20 2024-09-11T01:41:29.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.927+0000 7f4130857640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:29.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.927+0000 7f4130857640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:29.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.927+0000 7f4130857640 10 monclient: dump: 2024-09-11T01:41:29.936 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:29.936 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:29.936 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.936 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.936 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:29.936 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:29.936 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:29.936 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:29.936 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:29.937 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:29.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.928+0000 7f4133abd640 10 monclient: shutdown 2024-09-11T01:41:29.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.928+0000 7f4133abd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:29.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.928+0000 7f4133abd640 1 -- 172.21.15.12:0/1867761931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f412c0a2e20 msgr2=0x7f412c06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:29.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.928+0000 7f4133abd640 1 --2- 172.21.15.12:0/1867761931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f412c0a2e20 0x7f412c06ded0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f41280097a0 tx=0x7f4128029fd0 comp rx=0 tx=0).stop 2024-09-11T01:41:29.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.928+0000 7f4133abd640 1 -- 172.21.15.12:0/1867761931 shutdown_connections 2024-09-11T01:41:29.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.928+0000 7f4133abd640 1 --2- 172.21.15.12:0/1867761931 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c06f910 0x7f412c13f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:29.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.928+0000 7f4133abd640 1 --2- 172.21.15.12:0/1867761931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f412c0a2e20 0x7f412c06ded0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:29.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.928+0000 7f4133abd640 1 -- 172.21.15.12:0/1867761931 >> 172.21.15.12:0/1867761931 conn(0x7f412c0a8600 msgr2=0x7f412c0a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:29.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.928+0000 7f4133abd640 1 -- 172.21.15.12:0/1867761931 shutdown_connections 2024-09-11T01:41:29.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.928+0000 7f4133abd640 1 -- 172.21.15.12:0/1867761931 wait complete. 2024-09-11T01:41:29.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.928+0000 7f4133abd640 10 monclient: build_initial_monmap 2024-09-11T01:41:29.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.928+0000 7f4133abd640 10 monclient: monmap: 2024-09-11T01:41:29.938 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:29.938 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:41:29.938 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:41:29.938 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:41:29.938 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:29.938 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:29.938 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:41:29.938 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:41:29.938 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:41:29.939 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:29.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f4133abd640 1 Processor -- start 2024-09-11T01:41:29.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f4133abd640 1 -- start start 2024-09-11T01:41:29.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f4133abd640 10 monclient: init 2024-09-11T01:41:29.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f4133abd640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:29.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f4133abd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:41:29.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f4133abd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f412c06f910 0x7f412c1398e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:29.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f4133abd640 10 monclient(hunting): picked mon.noname-b con 0x7f412c06f910 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:41:29.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f4133abd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c0a2e20 0x7f412c139e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:29.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f4133abd640 10 monclient(hunting): picked mon.noname-c con 0x7f412c0a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:29.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f4133abd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f412c0a4970 0x7f412c1cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:29.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f4133abd640 10 monclient(hunting): picked mon.noname-a con 0x7f412c0a4970 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:41:29.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f4133abd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:29.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f4133abd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f412c13fd00 con 0x7f412c0a4970 2024-09-11T01:41:29.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f4133abd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:29.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f4133abd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f412c13fb80 con 0x7f412c06f910 2024-09-11T01:41:29.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f4133abd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:29.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f4133abd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f412c13fe80 con 0x7f412c0a2e20 2024-09-11T01:41:29.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f4133abd640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:29.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f4133abd640 10 monclient(hunting): authenticate will time out at 1496.371861s 2024-09-11T01:41:29.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f4131058640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c0a2e20 0x7f412c139e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:29.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f4131058640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c0a2e20 0x7f412c139e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:56046/0 (socket says 172.21.15.12:56046) 2024-09-11T01:41:29.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f4131058640 1 -- 172.21.15.12:0/312806292 learned_addr learned my addr 172.21.15.12:0/312806292 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:41:29.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f4131058640 10 monclient(hunting): get_auth_request con 0x7f412c0a2e20 auth_method 0 2024-09-11T01:41:29.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f4131058640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:29.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f4131058640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:29.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f4131058640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f413205a640 1 --2- 172.21.15.12:0/312806292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f412c0a4970 0x7f412c1cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f413205a640 10 monclient(hunting): get_auth_request con 0x7f412c0a4970 auth_method 0 2024-09-11T01:41:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f413205a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f413205a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.929+0000 7f413205a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.930+0000 7f4131058640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.930+0000 7f4131058640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.930+0000 7f4131058640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.930+0000 7f413205a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.930+0000 7f413205a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.930+0000 7f413205a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.930+0000 7f4131859640 1 --2- 172.21.15.12:0/312806292 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f412c06f910 0x7f412c1398e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.930+0000 7f4131859640 10 monclient(hunting): get_auth_request con 0x7f412c06f910 auth_method 0 2024-09-11T01:41:29.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.930+0000 7f4131859640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:29.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.930+0000 7f4131859640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:29.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.930+0000 7f4131859640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:29.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.930+0000 7f4131859640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:29.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.930+0000 7f4131859640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:29.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.930+0000 7f4131859640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:29.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.930+0000 7f4131058640 10 monclient(hunting): handle_auth_done global_id 114290 payload 824 2024-09-11T01:41:29.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.930+0000 7f4131058640 1 -- 172.21.15.12:0/312806292 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f412c06f910 msgr2=0x7f412c1398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:29.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.930+0000 7f4131058640 1 --2- 172.21.15.12:0/312806292 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f412c06f910 0x7f412c1398e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:29.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.930+0000 7f4131058640 1 -- 172.21.15.12:0/312806292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f412c0a4970 msgr2=0x7f412c1cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:29.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.930+0000 7f4131058640 1 --2- 172.21.15.12:0/312806292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f412c0a4970 0x7f412c1cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:29.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.930+0000 7f4131058640 10 monclient: _finish_hunting 0 2024-09-11T01:41:29.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.930+0000 7f4131058640 1 monclient: found mon.noname-c 2024-09-11T01:41:29.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.930+0000 7f4131058640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:29.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.930+0000 7f4131058640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:41:29.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.930+0000 7f4131058640 1 -- 172.21.15.12:0/312806292 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f412c13c050 con 0x7f412c0a2e20 2024-09-11T01:41:29.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.930+0000 7f413205a640 1 --2- 172.21.15.12:0/312806292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f412c0a4970 0x7f412c1cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:41:29.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.932+0000 7f4131058640 1 -- 172.21.15.12:0/312806292 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c0a2e20 msgr2=0x7f412c139e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:29.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.932+0000 7f4131058640 1 --2- 172.21.15.12:0/312806292 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c0a2e20 0x7f412c139e20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f411c00c460 tx=0x7f411c00c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:29.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.932+0000 7f41227fc640 1 -- 172.21.15.12:0/312806292 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c007070 con 0x7f412c0a2e20 2024-09-11T01:41:29.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.932+0000 7f41227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:29.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.932+0000 7f41227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:41:29.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.932+0000 7f41227fc640 10 monclient: dump: 2024-09-11T01:41:29.950 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:29.950 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:29.950 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.950 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.950 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:29.950 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:29.950 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:29.950 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:29.950 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:29.950 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:29.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.932+0000 7f41227fc640 10 monclient: _finish_auth 0 2024-09-11T01:41:29.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.932+0000 7f41227fc640 10 monclient: _check_auth_tickets 2024-09-11T01:41:29.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.932+0000 7f41227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:29.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.932+0000 7f41227fc640 1 -- 172.21.15.12:0/312806292 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f411c007230 con 0x7f412c0a2e20 2024-09-11T01:41:29.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.932+0000 7f41227fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:29.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.932+0000 7f41227fc640 1 -- 172.21.15.12:0/312806292 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c007620 con 0x7f412c0a2e20 2024-09-11T01:41:29.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.932+0000 7f41227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:29.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.932+0000 7f41227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:41:29.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.932+0000 7f41227fc640 10 monclient: dump: 2024-09-11T01:41:29.951 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:29.951 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:29.951 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.951 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.951 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:29.951 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:29.951 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:29.951 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:29.951 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:29.952 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:29.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.932+0000 7f4133abd640 5 monclient: authenticate success, global_id 114290 2024-09-11T01:41:29.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.932+0000 7f4133abd640 10 monclient: _renew_subs 2024-09-11T01:41:29.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.932+0000 7f4133abd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:41:29.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.932+0000 7f4133abd640 1 -- 172.21.15.12:0/312806292 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f412c072070 con 0x7f412c0a2e20 2024-09-11T01:41:29.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.932+0000 7f4123fff640 4 set_mon_vals no callback set 2024-09-11T01:41:29.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.932+0000 7f4133abd640 10 monclient: _renew_subs 2024-09-11T01:41:29.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.932+0000 7f4133abd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:41:29.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.932+0000 7f4133abd640 1 -- 172.21.15.12:0/312806292 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f412c13bd70 con 0x7f412c0a2e20 2024-09-11T01:41:29.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.933+0000 7f4131058640 1 --2- 172.21.15.12:0/312806292 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c0a2e20 0x7f412c139e20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f411c00c460 tx=0x7f411c00c920 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7f411c052980 osd_map(405..405 src has 285..405) 2024-09-11T01:41:29.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.933+0000 7f41227fc640 1 -- 172.21.15.12:0/312806292 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f411c01b400 con 0x7f412c0a2e20 2024-09-11T01:41:29.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.933+0000 7f4133abd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:29.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.933+0000 7f4133abd640 1 --2- 172.21.15.12:0/312806292 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c1cd570 0x7f412c1cd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:29.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.933+0000 7f4133abd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:29.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.933+0000 7f4133abd640 1 -- 172.21.15.12:0/312806292 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f412c06a060 con 0x7f412c1cd570 2024-09-11T01:41:29.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.933+0000 7f4131859640 1 --2- 172.21.15.12:0/312806292 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c1cd570 0x7f412c1cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:29.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.933+0000 7f4131859640 10 monclient: get_auth_request con 0x7f412c1cd570 auth_method 0 2024-09-11T01:41:29.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.933+0000 7f4131859640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:29.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.933+0000 7f4131859640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:29.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.933+0000 7f4131859640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:29.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.933+0000 7f41227fc640 1 --2- 172.21.15.12:0/312806292 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f411003e260 0x7f4110040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:29.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.933+0000 7f4131859640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:29.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.933+0000 7f4131859640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:29.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.933+0000 7f4131859640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:29.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.934+0000 7f413205a640 1 --2- 172.21.15.12:0/312806292 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f411003e260 0x7f4110040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:29.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.934+0000 7f4131859640 10 monclient(hunting): handle_auth_done global_id 114293 payload 824 2024-09-11T01:41:29.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.934+0000 7f4131859640 1 -- 172.21.15.12:0/312806292 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f412c1cde90 con 0x7f412c1cd570 2024-09-11T01:41:29.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.934+0000 7f413205a640 10 monclient: get_auth_request con 0x7f411003e260 auth_method 0 2024-09-11T01:41:29.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.936+0000 7f4131859640 1 -- 172.21.15.12:0/312806292 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c1cd570 msgr2=0x7f412c1cd950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:29.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.936+0000 7f4131859640 1 --2- 172.21.15.12:0/312806292 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c1cd570 0x7f412c1cd950 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4128034000 tx=0x7f41280046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:29.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.936+0000 7f41227fc640 1 -- 172.21.15.12:0/312806292 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4128043070 con 0x7f412c1cd570 2024-09-11T01:41:29.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.936+0000 7f41227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:29.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.936+0000 7f41227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:41:29.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.936+0000 7f41227fc640 10 monclient: dump: 2024-09-11T01:41:29.954 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:29.954 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:29.954 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.954 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.954 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:29.955 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:29.955 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:29.955 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:29.955 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:29.955 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:29.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.936+0000 7f413205a640 1 --2- 172.21.15.12:0/312806292 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f411003e260 0x7f4110040720 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4124009fd0 tx=0x7f412400b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:29.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.936+0000 7f41227fc640 1 -- 172.21.15.12:0/312806292 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f412803c850 con 0x7f412c1cd570 2024-09-11T01:41:29.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.936+0000 7f41227fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:29.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.936+0000 7f41227fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:41:29.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.936+0000 7f41227fc640 1 -- 172.21.15.12:0/312806292 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c1cd570 msgr2=0x7f412c1cd950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:29.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.936+0000 7f41227fc640 1 --2- 172.21.15.12:0/312806292 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c1cd570 0x7f412c1cd950 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4128034000 tx=0x7f41280046c0 comp rx=0 tx=0).stop 2024-09-11T01:41:29.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.936+0000 7f41227fc640 1 -- 172.21.15.12:0/312806292 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c1cd570 msgr2=0x7f412c1cd950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:41:29.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.936+0000 7f41227fc640 1 --2- 172.21.15.12:0/312806292 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c1cd570 0x7f412c1cd950 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4128034000 tx=0x7f41280046c0 comp rx=0 tx=0).stop 2024-09-11T01:41:29.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.942+0000 7f4133abd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:41:29.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.942+0000 7f4133abd640 1 --2- 172.21.15.12:0/312806292 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c09d330 0x7f412c09d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:29.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.942+0000 7f4133abd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:29.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.942+0000 7f4133abd640 1 -- 172.21.15.12:0/312806292 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f412c13b8d0 con 0x7f412c09d330 2024-09-11T01:41:29.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.942+0000 7f4131859640 1 --2- 172.21.15.12:0/312806292 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c09d330 0x7f412c09d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:29.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.942+0000 7f4131859640 10 monclient: get_auth_request con 0x7f412c09d330 auth_method 0 2024-09-11T01:41:29.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.942+0000 7f4131859640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:29.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.942+0000 7f4131859640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:29.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.942+0000 7f4131859640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:29.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.943+0000 7f4131859640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:29.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.943+0000 7f4131859640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:29.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.943+0000 7f4131859640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:29.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.943+0000 7f4131859640 10 monclient(hunting): handle_auth_done global_id 114296 payload 824 2024-09-11T01:41:29.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.943+0000 7f4131859640 1 -- 172.21.15.12:0/312806292 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f412c13efd0 con 0x7f412c09d330 2024-09-11T01:41:29.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.945+0000 7f4131859640 1 -- 172.21.15.12:0/312806292 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c09d330 msgr2=0x7f412c09d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:29.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.945+0000 7f4131859640 1 --2- 172.21.15.12:0/312806292 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c09d330 0x7f412c09d710 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f412802a010 tx=0x7f4128009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:29.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.945+0000 7f41227fc640 1 -- 172.21.15.12:0/312806292 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4128037040 con 0x7f412c09d330 2024-09-11T01:41:29.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.945+0000 7f41227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:29.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.945+0000 7f41227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:41:29.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.945+0000 7f41227fc640 10 monclient: dump: 2024-09-11T01:41:29.958 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:29.958 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:29.958 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.958 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:29.958 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:29.958 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:29.959 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:29.959 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:29.959 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:29.959 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:29.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.945+0000 7f41227fc640 1 -- 172.21.15.12:0/312806292 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f4128048d60 con 0x7f412c09d330 2024-09-11T01:41:29.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.945+0000 7f41227fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:41:29.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.945+0000 7f41227fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:41:29.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.945+0000 7f41227fc640 1 -- 172.21.15.12:0/312806292 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c09d330 msgr2=0x7f412c09d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:29.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.945+0000 7f41227fc640 1 --2- 172.21.15.12:0/312806292 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c09d330 0x7f412c09d710 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f412802a010 tx=0x7f4128009190 comp rx=0 tx=0).stop 2024-09-11T01:41:29.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.945+0000 7f41227fc640 1 -- 172.21.15.12:0/312806292 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c09d330 msgr2=0x7f412c09d710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:41:29.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.945+0000 7f41227fc640 1 --2- 172.21.15.12:0/312806292 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c09d330 0x7f412c09d710 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f412802a010 tx=0x7f4128009190 comp rx=0 tx=0).stop 2024-09-11T01:41:29.960 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:41:29.960 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:41:29.960 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:41:29.960 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:41:29.960 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 114, 2024-09-11T01:41:29.960 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 135741, 2024-09-11T01:41:29.960 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:41:29.960 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:41:29.960 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:41:29.960 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:41:29.960 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:29.961 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:41:29.961 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:41:29.961 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:41:29.961 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:41:29.961 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:41:29.961 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:41:29.961 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:41:29.961 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:41:29.961 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:41:29.961 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:41:29.961 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:41:29.961 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:41:29.961 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:41:29.962 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:41:29.962 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:41:29.962 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:29.962 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:41:29.962 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:41:29.962 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:41:29.962 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:41:29.962 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:41:29.962 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:41:29.962 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:41:29.962 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:41:29.962 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:41:29.962 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:41:29.963 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:41:29.963 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:41:29.963 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:41:29.963 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:29.963 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:29.963 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:41:29.963 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:41:29.963 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:41:29.964 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:41:29.964 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:41:29.964 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:41:29.964 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:41:29.964 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:41:29.964 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:41:29.964 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:41:29.964 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:41:29.964 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:41:29.964 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:41:29.964 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:41:29.964 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:41:29.965 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:41:29.965 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:41:29.965 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:41:29.965 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:41:29.965 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:41:29.965 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:41:29.965 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:41:29.965 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:41:29.965 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:41:29.965 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:41:29.966 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:41:29.966 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:41:29.966 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:41:29.966 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:29.966 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:41:29.966 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:29.966 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:41:29.966 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:29.966 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:41:29.966 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:41:29.966 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:41:29.966 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:41:29.966 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:29.967 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:41:29.967 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:41:29.967 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:29.967 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:29.967 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:29.967 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:41:29.967 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:41:29.967 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:29.967 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:29.967 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:29.967 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:29.968 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:41:29.968 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:41:29.968 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:41:29.968 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:41:29.968 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:41:29.968 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:29.968 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:29.968 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:41:29.968 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:41:29.968 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:41:29.968 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:41:29.968 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:29.968 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:41:29.969 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:41:29.969 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:29.969 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:29.969 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:29.969 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:41:29.969 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:41:29.969 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:29.970 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:29.970 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:29.970 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:29.970 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:41:29.970 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:41:29.970 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:41:29.970 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:41:29.971 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:41:29.971 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:29.971 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:29.971 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:41:29.971 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:41:29.971 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:41:29.971 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:41:29.971 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:29.971 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:41:29.971 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:41:29.971 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:29.971 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:29.971 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:29.972 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:41:29.972 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:41:29.972 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:29.972 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:29.972 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:29.972 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:29.972 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:41:29.972 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:41:29.972 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:41:29.972 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:41:29.972 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:41:29.972 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:29.972 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:29.972 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:29.973 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:41:29.973 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:41:29.973 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:29.973 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:41:29.973 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:41:29.973 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:41:29.973 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:29.973 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:29.973 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:41:29.973 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:29.973 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:41:29.973 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:41:29.973 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:41:29.973 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:29.974 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:29.974 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:41:29.974 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:29.974 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:41:29.974 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:41:29.974 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:41:29.974 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:29.974 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:29.974 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:41:29.974 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:29.974 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:41:29.974 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:41:29.974 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:41:29.974 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:29.975 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:29.975 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:29.975 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:41:29.975 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:41:29.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.946+0000 7f4133abd640 1 -- 172.21.15.12:0/312806292 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f411003e260 msgr2=0x7f4110040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:29.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.946+0000 7f4133abd640 1 --2- 172.21.15.12:0/312806292 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f411003e260 0x7f4110040720 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4124009fd0 tx=0x7f412400b040 comp rx=0 tx=0).stop 2024-09-11T01:41:29.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.946+0000 7f4133abd640 10 monclient: shutdown 2024-09-11T01:41:29.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.946+0000 7f4133abd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:29.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.946+0000 7f4133abd640 1 -- 172.21.15.12:0/312806292 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c0a2e20 msgr2=0x7f412c139e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.946+0000 7f4133abd640 1 --2- 172.21.15.12:0/312806292 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f412c0a2e20 0x7f412c139e20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f411c00c460 tx=0x7f411c00c920 comp rx=0 tx=0).stop 2024-09-11T01:41:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.946+0000 7f413205a640 1 -- 172.21.15.12:0/312806292 reap_dead start 2024-09-11T01:41:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.946+0000 7f41227fc640 1 -- 172.21.15.12:0/312806292 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(405..405 src has 285..405) ==== 4783+0+0 (unknown 0 0 0) 0x7f411c052980 con 0x7f412c0a2e20 2024-09-11T01:41:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.946+0000 7f413205a640 1 -- 172.21.15.12:0/312806292 reap_dead start 2024-09-11T01:41:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.946+0000 7f4133abd640 1 -- 172.21.15.12:0/312806292 shutdown_connections 2024-09-11T01:41:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.946+0000 7f4133abd640 1 -- 172.21.15.12:0/312806292 >> 172.21.15.12:0/312806292 conn(0x7f412c0a8600 msgr2=0x7f412c1381d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.946+0000 7f4133abd640 1 -- 172.21.15.12:0/312806292 shutdown_connections 2024-09-11T01:41:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:29.946+0000 7f4133abd640 1 -- 172.21.15.12:0/312806292 wait complete. 2024-09-11T01:41:29.977 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:41:29.978 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:41:29.978 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:41:29.979 INFO:tasks.rados.rados.0.smithi071.stdout:1756: done (5 left) 2024-09-11T01:41:29.979 INFO:tasks.rados.rados.0.smithi071.stdout:1757: done (4 left) 2024-09-11T01:41:29.979 INFO:tasks.rados.rados.0.smithi071.stdout:1758: done (3 left) 2024-09-11T01:41:29.979 INFO:tasks.rados.rados.0.smithi071.stdout:1759: done (2 left) 2024-09-11T01:41:29.979 INFO:tasks.rados.rados.0.smithi071.stdout:1760: done (1 left) 2024-09-11T01:41:29.980 INFO:tasks.rados.rados.0.smithi071.stdout:1761: done (0 left) 2024-09-11T01:41:29.980 INFO:tasks.rados.rados.0.smithi071.stdout:1762: delete oid 39 current snap is 197 2024-09-11T01:41:29.980 INFO:tasks.rados.rados.0.smithi071.stdout:1762: done (0 left) 2024-09-11T01:41:29.980 INFO:tasks.rados.rados.0.smithi071.stdout:1763: rollback oid 20 current snap is 197 2024-09-11T01:41:29.980 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 20 to 196 2024-09-11T01:41:29.980 INFO:tasks.rados.rados.0.smithi071.stdout:1764: copy_from oid 16 from oid 35 current snap is 197 2024-09-11T01:41:29.980 INFO:tasks.rados.rados.0.smithi071.stdout:1765: read oid 27 snap 175 2024-09-11T01:41:29.980 INFO:tasks.rados.rados.0.smithi071.stdout:1765: expect (ObjNum 538 snap 152 seq_num 538) 2024-09-11T01:41:29.980 INFO:tasks.rados.rados.0.smithi071.stdout:1766: read oid 22 snap -1 2024-09-11T01:41:29.980 INFO:tasks.rados.rados.0.smithi071.stdout:1766: expect deleted 2024-09-11T01:41:29.981 INFO:tasks.rados.rados.0.smithi071.stdout:1767: read oid 38 snap -1 2024-09-11T01:41:29.981 INFO:tasks.rados.rados.0.smithi071.stdout:1767: expect (ObjNum 577 snap 165 seq_num 577) 2024-09-11T01:41:29.981 INFO:tasks.rados.rados.0.smithi071.stdout:1768: snap_create 2024-09-11T01:41:29.981 INFO:tasks.rados.rados.0.smithi071.stdout:1767: expect (ObjNum 577 snap 165 seq_num 577) 2024-09-11T01:41:30.062 INFO:tasks.rados.rados.0.smithi071.stdout:1763: finishing rollback tid 0 to smithi07134777-20 2024-09-11T01:41:30.062 INFO:tasks.rados.rados.0.smithi071.stdout:1763: finishing rollback tid 1 to smithi07134777-20 2024-09-11T01:41:30.062 INFO:tasks.rados.rados.0.smithi071.stdout:1765: expect (ObjNum 538 snap 152 seq_num 538) 2024-09-11T01:41:30.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.081+0000 7f8146175640 10 monclient: get_monmap_and_config 2024-09-11T01:41:30.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.081+0000 7f8146175640 10 monclient: build_initial_monmap 2024-09-11T01:41:30.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.081+0000 7f8146175640 10 monclient: monmap: 2024-09-11T01:41:30.083 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:30.083 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:30.083 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:41:30.081905+0000 2024-09-11T01:41:30.083 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:41:30.081905+0000 2024-09-11T01:41:30.083 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:30.083 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:30.083 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:41:30.083 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:41:30.083 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:41:30.084 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:41:30.084 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:41:30.084 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:30.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.081+0000 7f8146175640 1 Processor -- start 2024-09-11T01:41:30.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.081+0000 7f8146175640 1 -- start start 2024-09-11T01:41:30.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.081+0000 7f8146175640 10 monclient: init 2024-09-11T01:41:30.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f8146175640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:30.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f8146175640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-11T01:41:30.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f8146175640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8140134ad0 addr v1:172.21.15.71:6789/0 2024-09-11T01:41:30.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f8146175640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f81400a2200 0x7f814013a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:30.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f8146175640 10 monclient(hunting): picked mon.noname-c con 0x7f81400a2200 addr v2:172.21.15.71:3300/0 2024-09-11T01:41:30.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f8146175640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f81401355c0 addr v1:172.21.15.12:6789/0 2024-09-11T01:41:30.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f8146175640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f814009ed30 con 0x7f81401355c0 2024-09-11T01:41:30.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f8146175640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f814009f010 con 0x7f8140134ad0 2024-09-11T01:41:30.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f8146175640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:30.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f8146175640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f814009f5d0 con 0x7f81400a2200 2024-09-11T01:41:30.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f8146175640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:30.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f8146175640 10 monclient(hunting): authenticate will time out at 1496.524467s 2024-09-11T01:41:30.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f813ffff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f81401355c0 0x7f814013c550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42278/0 (socket says 172.21.15.12:42278) 2024-09-11T01:41:30.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f813ffff640 1 -- 172.21.15.12:0/2276299841 learned_addr learned my addr 172.21.15.12:0/2276299841 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:41:30.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f813effd640 1 --2- 172.21.15.12:0/2276299841 >> v2:172.21.15.71:3300/0 conn(0x7f81400a2200 0x7f814013a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:30.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f813effd640 10 monclient(hunting): get_auth_request con 0x7f81400a2200 auth_method 0 2024-09-11T01:41:30.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f813effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:30.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f813effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:30.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f813effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:30.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f813effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:30.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f813effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:30.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f813effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:30.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f813effd640 10 monclient(hunting): handle_auth_done global_id 134110 payload 274 2024-09-11T01:41:30.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f813effd640 1 -- 172.21.15.12:0/2276299841 >> v1:172.21.15.71:6789/0 conn(0x7f8140134ad0 legacy=0x7f8140134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:30.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f813effd640 1 -- 172.21.15.12:0/2276299841 >> v1:172.21.15.12:6789/0 conn(0x7f81401355c0 legacy=0x7f814013c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:30.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f813effd640 10 monclient: _finish_hunting 0 2024-09-11T01:41:30.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f813effd640 1 monclient: found mon.noname-c 2024-09-11T01:41:30.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.082+0000 7f813effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:30.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.083+0000 7f813effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:41:30.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.083+0000 7f813effd640 1 -- 172.21.15.12:0/2276299841 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f814009f2f0 con 0x7f81400a2200 2024-09-11T01:41:30.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.085+0000 7f813effd640 1 -- 172.21.15.12:0/2276299841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81400a2200 msgr2=0x7f814013a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:30.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.085+0000 7f813effd640 1 --2- 172.21.15.12:0/2276299841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81400a2200 0x7f814013a000 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f814009f750 tx=0x7f813402c1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62371d4917ec7f17 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:30.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.086+0000 7f813e7fc640 1 -- 172.21.15.12:0/2276299841 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8134039070 con 0x7f81400a2200 2024-09-11T01:41:30.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.086+0000 7f813e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:30.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.086+0000 7f813e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:41:30.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.086+0000 7f813e7fc640 10 monclient: dump: 2024-09-11T01:41:30.099 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:30.099 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:30.099 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:30.099 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:30.100 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.086+0000 7f813e7fc640 10 monclient: _finish_auth 0 2024-09-11T01:41:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.086+0000 7f813e7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:41:30.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.086+0000 7f813e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:30.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.086+0000 7f813e7fc640 1 -- 172.21.15.12:0/2276299841 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8134034640 con 0x7f81400a2200 2024-09-11T01:41:30.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.086+0000 7f813e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:30.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.086+0000 7f813e7fc640 1 -- 172.21.15.12:0/2276299841 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8134034a10 con 0x7f81400a2200 2024-09-11T01:41:30.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.086+0000 7f813e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:30.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.086+0000 7f813e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:41:30.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.086+0000 7f813e7fc640 10 monclient: dump: 2024-09-11T01:41:30.101 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:30.102 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:30.102 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:30.102 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:30.102 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:30.102 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:30.102 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:30.102 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:30.102 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:30.103 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:30.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.086+0000 7f8146175640 5 monclient: authenticate success, global_id 134110 2024-09-11T01:41:30.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.086+0000 7f8146175640 10 monclient: get_monmap_and_config success 2024-09-11T01:41:30.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.086+0000 7f8146175640 4 set_mon_vals no callback set 2024-09-11T01:41:30.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.086+0000 7f8146175640 10 monclient: shutdown 2024-09-11T01:41:30.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.086+0000 7f8146175640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:30.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.086+0000 7f8146175640 1 -- 172.21.15.12:0/2276299841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81400a2200 msgr2=0x7f814013a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:30.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.086+0000 7f8146175640 1 --2- 172.21.15.12:0/2276299841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81400a2200 0x7f814013a000 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f814009f750 tx=0x7f813402c1f0 comp rx=0 tx=0).stop 2024-09-11T01:41:30.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.086+0000 7f8146175640 1 -- 172.21.15.12:0/2276299841 shutdown_connections 2024-09-11T01:41:30.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.086+0000 7f8146175640 1 --2- 172.21.15.12:0/2276299841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81400a2200 0x7f814013a000 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:30.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.086+0000 7f8146175640 1 -- 172.21.15.12:0/2276299841 >> 172.21.15.12:0/2276299841 conn(0x7f81400a2f70 msgr2=0x7f81400a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:30.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.086+0000 7f8146175640 1 -- 172.21.15.12:0/2276299841 shutdown_connections 2024-09-11T01:41:30.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.086+0000 7f8146175640 1 -- 172.21.15.12:0/2276299841 wait complete. 2024-09-11T01:41:30.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.087+0000 7f8146175640 10 monclient: build_initial_monmap 2024-09-11T01:41:30.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.087+0000 7f8146175640 10 monclient: monmap: 2024-09-11T01:41:30.104 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:30.104 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:41:30.105 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:41:30.105 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:41:30.105 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:30.105 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:30.105 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:41:30.105 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:41:30.105 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:41:30.106 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:30.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.087+0000 7f8146175640 1 Processor -- start 2024-09-11T01:41:30.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.087+0000 7f8146175640 1 -- start start 2024-09-11T01:41:30.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.087+0000 7f8146175640 10 monclient: init 2024-09-11T01:41:30.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f8146175640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:30.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f8146175640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:41:30.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f8146175640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81400a2200 0x7f8140133830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:30.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f8146175640 10 monclient(hunting): picked mon.noname-b con 0x7f81400a2200 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:41:30.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f8146175640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8140134ad0 0x7f8140133d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:30.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f8146175640 10 monclient(hunting): picked mon.noname-a con 0x7f8140134ad0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:41:30.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f8146175640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f81401355c0 0x7f8140120400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:30.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f8146175640 10 monclient(hunting): picked mon.noname-c con 0x7f81401355c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:30.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f8146175640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:30.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f8146175640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f81400a0150 con 0x7f8140134ad0 2024-09-11T01:41:30.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f8146175640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:30.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f8146175640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f814009ffd0 con 0x7f81400a2200 2024-09-11T01:41:30.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f8146175640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:30.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f8146175640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f81400a0c90 con 0x7f81401355c0 2024-09-11T01:41:30.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f8146175640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:30.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f8146175640 10 monclient(hunting): authenticate will time out at 1496.530440s 2024-09-11T01:41:30.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813effd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8140134ad0 0x7f8140133d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:30.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f81401355c0 0x7f8140120400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:30.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813effd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8140134ad0 0x7f8140133d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58652/0 (socket says 172.21.15.12:58652) 2024-09-11T01:41:30.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813effd640 1 -- 172.21.15.12:0/3230871852 learned_addr learned my addr 172.21.15.12:0/3230871852 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:41:30.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813effd640 10 monclient(hunting): get_auth_request con 0x7f8140134ad0 auth_method 0 2024-09-11T01:41:30.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:30.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:30.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:30.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813ffff640 10 monclient(hunting): get_auth_request con 0x7f81401355c0 auth_method 0 2024-09-11T01:41:30.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813f7fe640 1 --2- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81400a2200 0x7f8140133830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:30.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:30.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:30.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:30.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813f7fe640 10 monclient(hunting): get_auth_request con 0x7f81400a2200 auth_method 0 2024-09-11T01:41:30.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:30.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:30.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:30.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:30.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:30.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:30.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:30.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:30.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:30.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:30.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:30.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:30.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813f7fe640 10 monclient(hunting): handle_auth_done global_id 134113 payload 824 2024-09-11T01:41:30.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813f7fe640 1 -- 172.21.15.12:0/3230871852 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f81401355c0 msgr2=0x7f8140120400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:30.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813f7fe640 1 --2- 172.21.15.12:0/3230871852 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f81401355c0 0x7f8140120400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:30.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813f7fe640 1 -- 172.21.15.12:0/3230871852 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8140134ad0 msgr2=0x7f8140133d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:30.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813f7fe640 1 --2- 172.21.15.12:0/3230871852 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8140134ad0 0x7f8140133d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:30.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813f7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:41:30.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813f7fe640 1 monclient: found mon.noname-b 2024-09-11T01:41:30.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:30.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-09-11T01:41:30.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813f7fe640 1 -- 172.21.15.12:0/3230871852 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81400a4950 con 0x7f81400a2200 2024-09-11T01:41:30.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813ffff640 1 --2- 172.21.15.12:0/3230871852 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f81401355c0 0x7f8140120400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:41:30.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.088+0000 7f813effd640 1 --2- 172.21.15.12:0/3230871852 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8140134ad0 0x7f8140133d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:41:30.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.091+0000 7f813f7fe640 1 -- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81400a2200 msgr2=0x7f8140133830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:30.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.091+0000 7f813f7fe640 1 --2- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81400a2200 0x7f8140133830 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f813000c410 tx=0x7f813000c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:30.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.091+0000 7f8127fff640 1 -- 172.21.15.12:0/3230871852 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8130017070 con 0x7f81400a2200 2024-09-11T01:41:30.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.091+0000 7f8127fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:30.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.091+0000 7f8127fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:41:30.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.091+0000 7f8127fff640 10 monclient: dump: 2024-09-11T01:41:30.111 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:30.111 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:30.111 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:30.111 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:30.111 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:30.111 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:30.111 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:30.111 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:30.111 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:30.112 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:30.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.091+0000 7f8127fff640 10 monclient: _finish_auth 0 2024-09-11T01:41:30.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.091+0000 7f8127fff640 10 monclient: _check_auth_tickets 2024-09-11T01:41:30.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.091+0000 7f8127fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:30.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.091+0000 7f8127fff640 1 -- 172.21.15.12:0/3230871852 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8130005230 con 0x7f81400a2200 2024-09-11T01:41:30.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.091+0000 7f8127fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:30.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.091+0000 7f8127fff640 1 -- 172.21.15.12:0/3230871852 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81300117a0 con 0x7f81400a2200 2024-09-11T01:41:30.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.091+0000 7f8127fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:30.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.091+0000 7f8127fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:41:30.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.091+0000 7f8127fff640 10 monclient: dump: 2024-09-11T01:41:30.112 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:30.112 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:30.112 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:30.112 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:30.112 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:30.113 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:30.113 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:30.113 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:30.113 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:30.113 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:30.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.091+0000 7f8146175640 5 monclient: authenticate success, global_id 134113 2024-09-11T01:41:30.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.091+0000 7f8146175640 10 monclient: _renew_subs 2024-09-11T01:41:30.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.091+0000 7f813dffb640 4 set_mon_vals no callback set 2024-09-11T01:41:30.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.091+0000 7f8146175640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:41:30.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.091+0000 7f8146175640 1 -- 172.21.15.12:0/3230871852 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f814009d480 con 0x7f81400a2200 2024-09-11T01:41:30.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.091+0000 7f8146175640 10 monclient: _renew_subs 2024-09-11T01:41:30.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.091+0000 7f8146175640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:41:30.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.091+0000 7f8146175640 1 -- 172.21.15.12:0/3230871852 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8140121ec0 con 0x7f81400a2200 2024-09-11T01:41:30.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.092+0000 7f8127fff640 1 -- 172.21.15.12:0/3230871852 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f81300119c0 con 0x7f81400a2200 2024-09-11T01:41:30.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.092+0000 7f8146175640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:30.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.092+0000 7f8146175640 1 --2- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8140120940 0x7f8140120d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:30.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.092+0000 7f8146175640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:30.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.092+0000 7f8146175640 1 -- 172.21.15.12:0/3230871852 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8140123df0 con 0x7f8140120940 2024-09-11T01:41:30.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.092+0000 7f813effd640 1 --2- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8140120940 0x7f8140120d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:30.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.092+0000 7f813effd640 10 monclient: get_auth_request con 0x7f8140120940 auth_method 0 2024-09-11T01:41:30.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.092+0000 7f813effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:30.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.092+0000 7f813effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:30.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.092+0000 7f813effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:30.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.092+0000 7f8127fff640 1 --2- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f812c03e240 0x7f812c040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:30.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.092+0000 7f813effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:30.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.092+0000 7f813effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:30.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.092+0000 7f813effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:30.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.092+0000 7f8127fff640 1 -- 172.21.15.12:0/3230871852 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(406..406 src has 285..406) ==== 4739+0+0 (secure 0 0 0) 0x7f8130052330 con 0x7f81400a2200 2024-09-11T01:41:30.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.092+0000 7f813ffff640 1 --2- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f812c03e240 0x7f812c040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:30.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.092+0000 7f813ffff640 10 monclient: get_auth_request con 0x7f812c03e240 auth_method 0 2024-09-11T01:41:30.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.092+0000 7f813effd640 10 monclient(hunting): handle_auth_done global_id 134116 payload 824 2024-09-11T01:41:30.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.092+0000 7f813effd640 1 -- 172.21.15.12:0/3230871852 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8140134eb0 con 0x7f8140120940 2024-09-11T01:41:30.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.095+0000 7f813effd640 1 -- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8140120940 msgr2=0x7f8140120d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:30.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.095+0000 7f813effd640 1 --2- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8140120940 0x7f8140120d20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f81401227b0 tx=0x7f81340046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:30.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.095+0000 7f813ffff640 1 --2- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f812c03e240 0x7f812c040700 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8138005c00 tx=0x7f8138009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:30.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.095+0000 7f8127fff640 1 -- 172.21.15.12:0/3230871852 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8134044070 con 0x7f8140120940 2024-09-11T01:41:30.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.095+0000 7f8127fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:30.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.095+0000 7f8127fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:41:30.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.095+0000 7f8127fff640 10 monclient: dump: 2024-09-11T01:41:30.115 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:30.116 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:30.116 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:30.116 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:30.116 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:30.116 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:30.116 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:30.116 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:30.116 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:30.116 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:30.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.095+0000 7f8127fff640 1 -- 172.21.15.12:0/3230871852 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f813403ebd0 con 0x7f8140120940 2024-09-11T01:41:30.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.095+0000 7f8127fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:30.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.095+0000 7f8127fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:41:30.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.095+0000 7f8127fff640 1 -- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8140120940 msgr2=0x7f8140120d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:30.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.095+0000 7f8127fff640 1 --2- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8140120940 0x7f8140120d20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f81401227b0 tx=0x7f81340046c0 comp rx=0 tx=0).stop 2024-09-11T01:41:30.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.095+0000 7f8127fff640 1 -- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8140120940 msgr2=0x7f8140120d20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:41:30.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.095+0000 7f8127fff640 1 --2- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8140120940 0x7f8140120d20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:30.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.101+0000 7f8146175640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:41:30.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.101+0000 7f8146175640 1 --2- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8140121260 0x7f814011c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:30.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.101+0000 7f8146175640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:30.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.101+0000 7f8146175640 1 -- 172.21.15.12:0/3230871852 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8140122a20 con 0x7f8140121260 2024-09-11T01:41:30.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.102+0000 7f813effd640 1 --2- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8140121260 0x7f814011c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:30.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.102+0000 7f813effd640 10 monclient: get_auth_request con 0x7f8140121260 auth_method 0 2024-09-11T01:41:30.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.102+0000 7f813effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:30.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.102+0000 7f813effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:30.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.102+0000 7f813effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:30.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.102+0000 7f813effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:30.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.102+0000 7f813effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:30.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.102+0000 7f813effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:30.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.102+0000 7f813effd640 10 monclient(hunting): handle_auth_done global_id 134119 payload 824 2024-09-11T01:41:30.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.102+0000 7f813effd640 1 -- 172.21.15.12:0/3230871852 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f814006d480 con 0x7f8140121260 2024-09-11T01:41:30.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.104+0000 7f813effd640 1 -- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8140121260 msgr2=0x7f814011c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:30.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.104+0000 7f813effd640 1 --2- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8140121260 0x7f814011c8e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f813402c230 tx=0x7f8134008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:30.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.104+0000 7f8127fff640 1 -- 172.21.15.12:0/3230871852 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8134039040 con 0x7f8140121260 2024-09-11T01:41:30.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.104+0000 7f8127fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:30.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.104+0000 7f8127fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:41:30.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.104+0000 7f8127fff640 10 monclient: dump: 2024-09-11T01:41:30.120 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:30.120 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:30.120 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:30.120 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:30.120 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:30.120 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:30.120 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:30.120 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:30.120 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:30.121 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:30.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.105+0000 7f8127fff640 1 -- 172.21.15.12:0/3230871852 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7f813404fc90 con 0x7f8140121260 2024-09-11T01:41:30.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.105+0000 7f8127fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:41:30.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.105+0000 7f8127fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:41:30.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.105+0000 7f8127fff640 1 -- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8140121260 msgr2=0x7f814011c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:30.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.105+0000 7f8127fff640 1 --2- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8140121260 0x7f814011c8e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f813402c230 tx=0x7f8134008030 comp rx=0 tx=0).stop 2024-09-11T01:41:30.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.105+0000 7f8127fff640 1 -- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8140121260 msgr2=0x7f814011c8e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:41:30.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.105+0000 7f8127fff640 1 --2- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8140121260 0x7f814011c8e0 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f813402c230 tx=0x7f8134008030 comp rx=0 tx=0).stop 2024-09-11T01:41:30.121 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:41:30.121 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:41:30.122 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:41:30.122 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:41:30.122 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 114, 2024-09-11T01:41:30.122 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 23066, 2024-09-11T01:41:30.122 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:41:30.122 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:41:30.122 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:41:30.122 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:41:30.122 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:30.122 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:41:30.122 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:41:30.122 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:41:30.122 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:41:30.122 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:41:30.122 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:41:30.123 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:41:30.123 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:41:30.123 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:41:30.123 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:41:30.123 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:41:30.123 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:41:30.123 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:41:30.123 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:41:30.123 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:41:30.123 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:30.123 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:41:30.123 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:41:30.123 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:41:30.123 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:41:30.124 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:41:30.124 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:41:30.124 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:41:30.124 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:41:30.124 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:41:30.124 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:41:30.124 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:41:30.124 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:41:30.124 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:41:30.124 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:30.124 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:30.124 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:41:30.124 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:41:30.124 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:41:30.125 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:41:30.125 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:41:30.125 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:41:30.125 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:41:30.125 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:41:30.125 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:41:30.125 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:41:30.125 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:41:30.125 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:41:30.125 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:41:30.125 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:41:30.125 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:41:30.125 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:41:30.125 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:41:30.126 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:41:30.126 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:41:30.126 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:41:30.126 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:41:30.126 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:41:30.126 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:41:30.126 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:41:30.126 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:41:30.126 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:41:30.126 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:41:30.126 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:41:30.126 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:30.126 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:41:30.126 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:30.126 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:41:30.127 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:30.127 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:41:30.127 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:41:30.127 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:41:30.127 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:41:30.127 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:30.127 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:41:30.127 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:41:30.127 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:30.127 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:30.127 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:30.127 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:41:30.127 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:41:30.128 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:30.128 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:30.128 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:30.128 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:30.128 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:41:30.128 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:41:30.128 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:41:30.128 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:41:30.128 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:41:30.129 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:30.129 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:30.129 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:41:30.129 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:41:30.129 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:41:30.129 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:41:30.129 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:30.129 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:41:30.129 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:41:30.129 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:30.129 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:30.129 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:30.129 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:41:30.129 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:41:30.130 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:30.130 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:30.130 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:30.130 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:30.130 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:41:30.130 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:41:30.130 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:41:30.130 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:41:30.130 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:41:30.130 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:30.130 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:30.130 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:41:30.130 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:41:30.130 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:41:30.131 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:41:30.131 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:30.131 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:41:30.131 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:41:30.131 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:30.131 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:30.131 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:30.131 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:41:30.131 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:41:30.131 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:30.131 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:30.131 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:30.131 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:30.131 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:41:30.132 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:41:30.132 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:41:30.132 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:41:30.132 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:41:30.132 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:30.132 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:30.132 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:30.132 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:41:30.132 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:41:30.132 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:30.132 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:41:30.132 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:41:30.132 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:41:30.132 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:30.133 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:30.133 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:41:30.133 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:30.133 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:41:30.133 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:41:30.133 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:41:30.133 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:30.133 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:30.133 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:30.133 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:41:30.133 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:41:30.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.105+0000 7f8146175640 1 -- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f812c03e240 msgr2=0x7f812c040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:30.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.105+0000 7f8146175640 1 --2- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f812c03e240 0x7f812c040700 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8138005c00 tx=0x7f8138009040 comp rx=0 tx=0).stop 2024-09-11T01:41:30.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.105+0000 7f8146175640 10 monclient: shutdown 2024-09-11T01:41:30.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.105+0000 7f8146175640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:30.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.105+0000 7f8146175640 1 -- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81400a2200 msgr2=0x7f8140133830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:30.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.105+0000 7f8146175640 1 --2- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81400a2200 0x7f8140133830 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f813000c410 tx=0x7f813000c8d0 comp rx=0 tx=0).stop 2024-09-11T01:41:30.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.105+0000 7f813ffff640 1 -- 172.21.15.12:0/3230871852 reap_dead start 2024-09-11T01:41:30.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.105+0000 7f8146175640 1 -- 172.21.15.12:0/3230871852 shutdown_connections 2024-09-11T01:41:30.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.105+0000 7f8146175640 1 --2- 172.21.15.12:0/3230871852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81400a2200 0x7f8140133830 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:30.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.105+0000 7f8146175640 1 -- 172.21.15.12:0/3230871852 >> 172.21.15.12:0/3230871852 conn(0x7f81400a2f70 msgr2=0x7f814009d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:30.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.105+0000 7f8146175640 1 -- 172.21.15.12:0/3230871852 shutdown_connections 2024-09-11T01:41:30.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.105+0000 7f8146175640 1 -- 172.21.15.12:0/3230871852 wait complete. 2024-09-11T01:41:30.136 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:41:30.136 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:41:30.136 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:41:30.144 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1498439 2024-09-11T01:41:30.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.235+0000 7f5f262c6640 10 monclient: get_monmap_and_config 2024-09-11T01:41:30.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.235+0000 7f5f262c6640 10 monclient: build_initial_monmap 2024-09-11T01:41:30.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.235+0000 7f5f262c6640 10 monclient: monmap: 2024-09-11T01:41:30.237 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:30.237 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:30.238 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:41:30.236041+0000 2024-09-11T01:41:30.238 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:41:30.236041+0000 2024-09-11T01:41:30.238 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:30.238 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:30.238 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:41:30.238 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:41:30.238 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:41:30.238 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:41:30.239 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:41:30.239 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:30.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f262c6640 1 Processor -- start 2024-09-11T01:41:30.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f262c6640 1 -- start start 2024-09-11T01:41:30.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f262c6640 10 monclient: init 2024-09-11T01:41:30.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f262c6640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:30.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f262c6640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-11T01:41:30.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f262c6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5f20137e50 0x7f5f2009d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:30.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f262c6640 10 monclient(hunting): picked mon.noname-c con 0x7f5f20137e50 addr v2:172.21.15.71:3300/0 2024-09-11T01:41:30.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f262c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f20138230 0x7f5f20138660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:30.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f262c6640 10 monclient(hunting): picked mon.noname-b con 0x7f5f20138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:30.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f262c6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f20134cb0 addr v1:172.21.15.71:6789/0 2024-09-11T01:41:30.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f262c6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f200a6580 con 0x7f5f20134cb0 2024-09-11T01:41:30.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f262c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:30.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f262c6640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f5f200a62d0 con 0x7f5f20137e50 2024-09-11T01:41:30.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f262c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:30.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f262c6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5f2009bb40 con 0x7f5f20138230 2024-09-11T01:41:30.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f262c6640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:30.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f262c6640 10 monclient(hunting): authenticate will time out at 1496.678693s 2024-09-11T01:41:30.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f1f7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f20138230 0x7f5f20138660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:30.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f1f7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f20138230 0x7f5f20138660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:56086/0 (socket says 172.21.15.12:56086) 2024-09-11T01:41:30.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f1f7fe640 1 -- 172.21.15.12:0/1242386737 learned_addr learned my addr 172.21.15.12:0/1242386737 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:41:30.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f1f7fe640 10 monclient(hunting): get_auth_request con 0x7f5f20138230 auth_method 0 2024-09-11T01:41:30.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:30.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:30.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:30.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f1ffff640 1 --2- 172.21.15.12:0/1242386737 >> v2:172.21.15.71:3300/0 conn(0x7f5f20137e50 0x7f5f2009d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:30.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f1ffff640 10 monclient(hunting): get_auth_request con 0x7f5f20137e50 auth_method 0 2024-09-11T01:41:30.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:30.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:30.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:30.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:30.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:30.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:30.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:30.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:30.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.236+0000 7f5f1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:30.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.237+0000 7f5f1ffff640 10 monclient(hunting): handle_auth_done global_id 134122 payload 274 2024-09-11T01:41:30.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.237+0000 7f5f1ffff640 1 -- 172.21.15.12:0/1242386737 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f20138230 msgr2=0x7f5f20138660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:30.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.237+0000 7f5f1ffff640 1 --2- 172.21.15.12:0/1242386737 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f20138230 0x7f5f20138660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:30.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.237+0000 7f5f1ffff640 1 -- 172.21.15.12:0/1242386737 >> v1:172.21.15.71:6789/0 conn(0x7f5f20134cb0 legacy=0x7f5f20135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:30.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.237+0000 7f5f1ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:41:30.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.237+0000 7f5f1ffff640 1 monclient: found mon.noname-c 2024-09-11T01:41:30.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.237+0000 7f5f1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:30.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.237+0000 7f5f1ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:41:30.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.237+0000 7f5f1ffff640 1 -- 172.21.15.12:0/1242386737 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f2009c4a0 con 0x7f5f20137e50 2024-09-11T01:41:30.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.237+0000 7f5f1f7fe640 1 --2- 172.21.15.12:0/1242386737 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f20138230 0x7f5f20138660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:41:30.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f1ffff640 1 -- 172.21.15.12:0/1242386737 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5f20137e50 msgr2=0x7f5f2009d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:30.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f1ffff640 1 --2- 172.21.15.12:0/1242386737 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5f20137e50 0x7f5f2009d160 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5f100097a0 tx=0x7f5f10029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51aa52e5c3a902da server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:30.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f1effd640 1 -- 172.21.15.12:0/1242386737 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f10037070 con 0x7f5f20137e50 2024-09-11T01:41:30.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:30.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f1effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:41:30.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f1effd640 10 monclient: dump: 2024-09-11T01:41:30.252 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:30.252 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:30.252 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:30.253 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:30.253 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:30.253 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:30.253 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:30.253 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:30.253 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:30.253 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:30.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f1effd640 10 monclient: _finish_auth 0 2024-09-11T01:41:30.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f1effd640 10 monclient: _check_auth_tickets 2024-09-11T01:41:30.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:30.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f1effd640 1 -- 172.21.15.12:0/1242386737 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f10032440 con 0x7f5f20137e50 2024-09-11T01:41:30.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f1effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:30.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f1effd640 1 -- 172.21.15.12:0/1242386737 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f10032810 con 0x7f5f20137e50 2024-09-11T01:41:30.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f262c6640 5 monclient: authenticate success, global_id 134122 2024-09-11T01:41:30.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f262c6640 10 monclient: get_monmap_and_config success 2024-09-11T01:41:30.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f262c6640 4 set_mon_vals no callback set 2024-09-11T01:41:30.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:30.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f1effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:41:30.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f1effd640 10 monclient: dump: 2024-09-11T01:41:30.254 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:30.254 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:30.254 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:30.254 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:30.254 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:30.254 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:30.254 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:30.254 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:30.254 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:30.255 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:30.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f262c6640 10 monclient: shutdown 2024-09-11T01:41:30.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f262c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:30.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f262c6640 1 -- 172.21.15.12:0/1242386737 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5f20137e50 msgr2=0x7f5f2009d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:30.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f262c6640 1 --2- 172.21.15.12:0/1242386737 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5f20137e50 0x7f5f2009d160 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5f100097a0 tx=0x7f5f10029fd0 comp rx=0 tx=0).stop 2024-09-11T01:41:30.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f262c6640 1 -- 172.21.15.12:0/1242386737 shutdown_connections 2024-09-11T01:41:30.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f262c6640 1 --2- 172.21.15.12:0/1242386737 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f20138230 0x7f5f20138660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:30.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f262c6640 1 --2- 172.21.15.12:0/1242386737 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5f20137e50 0x7f5f2009d160 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:30.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f262c6640 1 -- 172.21.15.12:0/1242386737 >> 172.21.15.12:0/1242386737 conn(0x7f5f200a7020 msgr2=0x7f5f200a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:30.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f262c6640 1 -- 172.21.15.12:0/1242386737 shutdown_connections 2024-09-11T01:41:30.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.240+0000 7f5f262c6640 1 -- 172.21.15.12:0/1242386737 wait complete. 2024-09-11T01:41:30.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.241+0000 7f5f262c6640 10 monclient: build_initial_monmap 2024-09-11T01:41:30.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.241+0000 7f5f262c6640 10 monclient: monmap: 2024-09-11T01:41:30.255 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:30.256 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:41:30.256 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:41:30.256 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:41:30.256 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:30.256 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:30.256 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:41:30.256 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:41:30.256 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:41:30.256 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:30.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.241+0000 7f5f262c6640 1 Processor -- start 2024-09-11T01:41:30.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f262c6640 1 -- start start 2024-09-11T01:41:30.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f262c6640 10 monclient: init 2024-09-11T01:41:30.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f262c6640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:30.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f262c6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:41:30.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f262c6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f20134cb0 0x7f5f20133d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:30.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f262c6640 10 monclient(hunting): picked mon.noname-a con 0x7f5f20134cb0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:41:30.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f262c6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5f20137e50 0x7f5f20134240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:30.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f262c6640 10 monclient(hunting): picked mon.noname-b con 0x7f5f20137e50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:41:30.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f262c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f20138230 0x7f5f201219e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:30.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f262c6640 10 monclient(hunting): picked mon.noname-c con 0x7f5f20138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:30.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f262c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:30.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f262c6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5f2006f710 con 0x7f5f20134cb0 2024-09-11T01:41:30.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f262c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:30.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f262c6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5f20138ba0 con 0x7f5f20137e50 2024-09-11T01:41:30.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f262c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:30.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f262c6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5f2006f890 con 0x7f5f20138230 2024-09-11T01:41:30.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f262c6640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:30.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f262c6640 10 monclient(hunting): authenticate will time out at 1496.684559s 2024-09-11T01:41:30.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f1ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f20134cb0 0x7f5f20133d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:30.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f1f7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5f20137e50 0x7f5f20134240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:30.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f24863640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f20138230 0x7f5f201219e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:30.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f1f7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5f20137e50 0x7f5f20134240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:58906/0 (socket says 172.21.15.12:58906) 2024-09-11T01:41:30.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f1f7fe640 1 -- 172.21.15.12:0/2985136413 learned_addr learned my addr 172.21.15.12:0/2985136413 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:41:30.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f1ffff640 10 monclient(hunting): get_auth_request con 0x7f5f20134cb0 auth_method 0 2024-09-11T01:41:30.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:30.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:30.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:30.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f24863640 10 monclient(hunting): get_auth_request con 0x7f5f20138230 auth_method 0 2024-09-11T01:41:30.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f24863640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:30.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f24863640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:30.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f24863640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:30.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f1f7fe640 10 monclient(hunting): get_auth_request con 0x7f5f20137e50 auth_method 0 2024-09-11T01:41:30.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:30.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:30.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:30.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:30.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:30.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:30.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f24863640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:30.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f24863640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:30.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f24863640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:30.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:30.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:30.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:30.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f24863640 10 monclient(hunting): handle_auth_done global_id 114305 payload 824 2024-09-11T01:41:30.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f24863640 1 -- 172.21.15.12:0/2985136413 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5f20137e50 msgr2=0x7f5f20134240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:30.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.242+0000 7f5f24863640 1 --2- 172.21.15.12:0/2985136413 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5f20137e50 0x7f5f20134240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:30.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.243+0000 7f5f24863640 1 -- 172.21.15.12:0/2985136413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f20134cb0 msgr2=0x7f5f20133d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:30.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.243+0000 7f5f24863640 1 --2- 172.21.15.12:0/2985136413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f20134cb0 0x7f5f20133d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:30.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.243+0000 7f5f24863640 10 monclient: _finish_hunting 0 2024-09-11T01:41:30.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.243+0000 7f5f24863640 1 monclient: found mon.noname-c 2024-09-11T01:41:30.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.243+0000 7f5f24863640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:30.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.243+0000 7f5f24863640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:41:30.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.243+0000 7f5f24863640 1 -- 172.21.15.12:0/2985136413 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f20125720 con 0x7f5f20138230 2024-09-11T01:41:30.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.243+0000 7f5f1f7fe640 1 --2- 172.21.15.12:0/2985136413 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5f20137e50 0x7f5f20134240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:41:30.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.243+0000 7f5f1ffff640 1 --2- 172.21.15.12:0/2985136413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f20134cb0 0x7f5f20133d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:41:30.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.245+0000 7f5f24863640 1 -- 172.21.15.12:0/2985136413 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f20138230 msgr2=0x7f5f201219e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:30.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.245+0000 7f5f24863640 1 --2- 172.21.15.12:0/2985136413 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f20138230 0x7f5f201219e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5f1800c780 tx=0x7f5f1800cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:30.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.245+0000 7f5f1cff9640 1 -- 172.21.15.12:0/2985136413 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f18016020 con 0x7f5f20138230 2024-09-11T01:41:30.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.245+0000 7f5f1cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:30.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.245+0000 7f5f1cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:41:30.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.245+0000 7f5f1cff9640 10 monclient: dump: 2024-09-11T01:41:30.261 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:30.261 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:30.261 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:30.261 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:30.261 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:30.261 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:30.262 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:30.262 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:30.262 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:30.262 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:30.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.245+0000 7f5f1cff9640 10 monclient: _finish_auth 0 2024-09-11T01:41:30.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.245+0000 7f5f1cff9640 10 monclient: _check_auth_tickets 2024-09-11T01:41:30.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.245+0000 7f5f1cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:30.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.245+0000 7f5f262c6640 5 monclient: authenticate success, global_id 114305 2024-09-11T01:41:30.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.245+0000 7f5f1cff9640 1 -- 172.21.15.12:0/2985136413 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f1800ac60 con 0x7f5f20138230 2024-09-11T01:41:30.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.245+0000 7f5f262c6640 10 monclient: _renew_subs 2024-09-11T01:41:30.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.245+0000 7f5f262c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:41:30.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.245+0000 7f5f262c6640 1 -- 172.21.15.12:0/2985136413 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f200a1710 con 0x7f5f20138230 2024-09-11T01:41:30.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.245+0000 7f5f1cff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:30.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.245+0000 7f5f1cff9640 1 -- 172.21.15.12:0/2985136413 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f18012660 con 0x7f5f20138230 2024-09-11T01:41:30.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.245+0000 7f5f1cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:30.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.245+0000 7f5f1cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:41:30.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.245+0000 7f5f1cff9640 10 monclient: dump: 2024-09-11T01:41:30.263 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:30.263 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:30.263 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:30.263 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:30.263 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:30.263 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:30.263 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:30.264 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:30.264 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:30.264 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:30.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.245+0000 7f5f1e7fc640 4 set_mon_vals no callback set 2024-09-11T01:41:30.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.245+0000 7f5f262c6640 10 monclient: _renew_subs 2024-09-11T01:41:30.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.245+0000 7f5f262c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:41:30.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.245+0000 7f5f262c6640 1 -- 172.21.15.12:0/2985136413 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f20134780 con 0x7f5f20138230 2024-09-11T01:41:30.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.246+0000 7f5f1cff9640 1 -- 172.21.15.12:0/2985136413 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f5f18012880 con 0x7f5f20138230 2024-09-11T01:41:30.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.246+0000 7f5f262c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:30.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.246+0000 7f5f262c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:30.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.246+0000 7f5f262c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:41:30.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.246+0000 7f5f262c6640 1 -- 172.21.15.12:0/2985136413 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f20136110 con 0x7f5f20138230 2024-09-11T01:41:30.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.246+0000 7f5f1cff9640 1 --2- 172.21.15.12:0/2985136413 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5f0c03e210 0x7f5f0c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:30.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.246+0000 7f5f1cff9640 1 -- 172.21.15.12:0/2985136413 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(406..406 src has 285..406) ==== 4739+0+0 (secure 0 0 0) 0x7f5f18052ed0 con 0x7f5f20138230 2024-09-11T01:41:30.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.246+0000 7f5f1ffff640 1 --2- 172.21.15.12:0/2985136413 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5f0c03e210 0x7f5f0c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:30.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.246+0000 7f5f1ffff640 10 monclient: get_auth_request con 0x7f5f0c03e210 auth_method 0 2024-09-11T01:41:30.266 INFO:tasks.rados.rados.0.smithi071.stdout:1763: finishing rollback tid 2 to smithi07134777-20 2024-09-11T01:41:30.266 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 327 (ObjNum 513 snap 141 seq_num 513) dirty exists 2024-09-11T01:41:30.266 INFO:tasks.rados.rados.0.smithi071.stdout:1764: finishing copy_from racing read to smithi07134777-16 2024-09-11T01:41:30.266 INFO:tasks.rados.rados.0.smithi071.stdout:1764: finishing copy_from to smithi07134777-16 2024-09-11T01:41:30.266 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 649 (ObjNum 618 snap 183 seq_num 618) dirty exists 2024-09-11T01:41:30.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.249+0000 7f5f1ffff640 1 --2- 172.21.15.12:0/2985136413 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5f0c03e210 0x7f5f0c0406d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5f2009db80 tx=0x7f5f1002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:30.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.251+0000 7f5f1cff9640 1 -- 172.21.15.12:0/2985136413 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5f18020030 con 0x7f5f20138230 2024-09-11T01:41:30.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.251+0000 7f5f1cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:30.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.251+0000 7f5f1cff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:41:30.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.508+0000 7f5f262c6640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:41:30.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.508+0000 7f5f262c6640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:41:30.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.508+0000 7f5f262c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:41:30.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.508+0000 7f5f262c6640 1 -- 172.21.15.12:0/2985136413 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f5f20069950 con 0x7f5f20138230 2024-09-11T01:41:30.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.509+0000 7f5f1cff9640 1 -- 172.21.15.12:0/2985136413 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f5f1800ae10 con 0x7f5f20138230 2024-09-11T01:41:30.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.509+0000 7f5f1cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:41:30.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.509+0000 7f5f1cff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:41:30.511 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.510+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.511 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.510+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=22,config=2,health=11,kv=12,logm=53} crc {auth=1246602878,config=2586702516,health=498449171,kv=3263697421,logm=894278218}) 2024-09-11T01:41:30.512 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.510+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=22,config=2,health=12,kv=11,logm=53} crc {auth=1246602878,config=2586702516,health=3597419013,kv=1918975338,logm=2654211073}) 2024-09-11T01:41:30.512 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.510+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.512 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.510+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=22,config=2,health=11,kv=12,logm=53} crc {auth=1246602878,config=2586702516,health=498449171,kv=3263697421,logm=894278218}) 2024-09-11T01:41:30.512 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.510+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=19,config=2,health=11,kv=10,logm=58} crc {auth=4274796396,config=2586702516,health=766026065,kv=3517404579,logm=1945780409}) 2024-09-11T01:41:30.513 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.510+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.513 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.510+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562790460}) 2024-09-11T01:41:30.513 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.510+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1779790456}) 2024-09-11T01:41:30.513 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.510+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.513 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.510+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562790460}) 2024-09-11T01:41:30.513 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.511+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=455066393}) 2024-09-11T01:41:30.514 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.512+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.514 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.512+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=838305897}) 2024-09-11T01:41:30.514 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.512+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3652038001}) 2024-09-11T01:41:30.514 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.512+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.514 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.512+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=838305897}) 2024-09-11T01:41:30.514 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.512+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3321159504}) 2024-09-11T01:41:30.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.512+0000 7f5f262c6640 1 -- 172.21.15.12:0/2985136413 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5f0c03e210 msgr2=0x7f5f0c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:30.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.512+0000 7f5f262c6640 1 --2- 172.21.15.12:0/2985136413 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5f0c03e210 0x7f5f0c0406d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5f2009db80 tx=0x7f5f1002e000 comp rx=0 tx=0).stop 2024-09-11T01:41:30.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.512+0000 7f5f262c6640 10 monclient: shutdown 2024-09-11T01:41:30.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.512+0000 7f5f262c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:30.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.512+0000 7f5f262c6640 1 -- 172.21.15.12:0/2985136413 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f20138230 msgr2=0x7f5f201219e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:30.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.512+0000 7f5f262c6640 1 --2- 172.21.15.12:0/2985136413 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f20138230 0x7f5f201219e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5f1800c780 tx=0x7f5f1800cc40 comp rx=0 tx=0).stop 2024-09-11T01:41:30.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.512+0000 7f5f262c6640 1 -- 172.21.15.12:0/2985136413 shutdown_connections 2024-09-11T01:41:30.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.512+0000 7f5f262c6640 1 --2- 172.21.15.12:0/2985136413 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5f20138230 0x7f5f201219e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:30.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.512+0000 7f5f262c6640 1 --2- 172.21.15.12:0/2985136413 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5f20137e50 0x7f5f20134240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:30.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.512+0000 7f5f262c6640 1 --2- 172.21.15.12:0/2985136413 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5f0c03e210 0x7f5f0c0406d0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:30.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.512+0000 7f5f262c6640 1 --2- 172.21.15.12:0/2985136413 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5f20134cb0 0x7f5f20133d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:30.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.512+0000 7f5f262c6640 1 -- 172.21.15.12:0/2985136413 >> 172.21.15.12:0/2985136413 conn(0x7f5f200a7020 msgr2=0x7f5f200a94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:30.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.512+0000 7f5f262c6640 1 -- 172.21.15.12:0/2985136413 shutdown_connections 2024-09-11T01:41:30.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:30.512+0000 7f5f262c6640 1 -- 172.21.15.12:0/2985136413 wait complete. 2024-09-11T01:41:30.520 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.513+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.520 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.513+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1652670269}) 2024-09-11T01:41:30.520 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.513+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4134912574}) 2024-09-11T01:41:30.520 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.513+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.520 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.513+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1652670269}) 2024-09-11T01:41:30.520 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.513+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4246138265}) 2024-09-11T01:41:30.520 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.514+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.520 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.514+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3954990401}) 2024-09-11T01:41:30.520 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.514+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1935234226}) 2024-09-11T01:41:30.521 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.514+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.521 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.514+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3954990401}) 2024-09-11T01:41:30.521 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.514+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=6536226}) 2024-09-11T01:41:30.521 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.515+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.521 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.515+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1532265598}) 2024-09-11T01:41:30.521 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.515+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4249547848}) 2024-09-11T01:41:30.521 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.515+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.521 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.515+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1532265598}) 2024-09-11T01:41:30.521 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.515+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2052997288}) 2024-09-11T01:41:30.521 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.516+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.521 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.516+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3973121020}) 2024-09-11T01:41:30.521 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.516+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1606887091}) 2024-09-11T01:41:30.521 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.516+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.521 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.516+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3973121020}) 2024-09-11T01:41:30.522 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.516+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=219936703}) 2024-09-11T01:41:30.522 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.517+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.522 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.517+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1173158555}) 2024-09-11T01:41:30.522 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.517+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1114391187}) 2024-09-11T01:41:30.522 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.517+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.522 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.517+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1173158555}) 2024-09-11T01:41:30.522 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.517+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1739197530}) 2024-09-11T01:41:30.522 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.518+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.522 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.518+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3993995658}) 2024-09-11T01:41:30.522 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.518+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=541448913}) 2024-09-11T01:41:30.523 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.518+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.523 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.518+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3993995658}) 2024-09-11T01:41:30.523 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.518+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3260496758}) 2024-09-11T01:41:30.529 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.519+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.529 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.519+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255090851}) 2024-09-11T01:41:30.529 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.519+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=840415022}) 2024-09-11T01:41:30.529 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.519+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.529 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.519+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255090851}) 2024-09-11T01:41:30.530 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.519+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3985899965}) 2024-09-11T01:41:30.530 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.520+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.530 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.520+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476285723}) 2024-09-11T01:41:30.530 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.520+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=320527953}) 2024-09-11T01:41:30.531 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.520+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.532 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.520+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476285723}) 2024-09-11T01:41:30.532 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.520+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3071383765}) 2024-09-11T01:41:30.532 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.521+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.532 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.521+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2062044983}) 2024-09-11T01:41:30.532 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.521+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1957828600}) 2024-09-11T01:41:30.532 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.521+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.532 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.521+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2062044983}) 2024-09-11T01:41:30.532 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.521+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=98835826}) 2024-09-11T01:41:30.532 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.522+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.532 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.522+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370579837}) 2024-09-11T01:41:30.532 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.522+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=551969917}) 2024-09-11T01:41:30.532 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.522+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.532 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.522+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370579837}) 2024-09-11T01:41:30.533 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.522+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2341230021}) 2024-09-11T01:41:30.533 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.523+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.533 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.523+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3863293620}) 2024-09-11T01:41:30.533 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.523+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2899996821}) 2024-09-11T01:41:30.533 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.523+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.533 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.523+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3863293620}) 2024-09-11T01:41:30.533 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.523+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=575879606}) 2024-09-11T01:41:30.533 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.524+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.533 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.524+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2358583337}) 2024-09-11T01:41:30.533 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.524+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3204410277}) 2024-09-11T01:41:30.533 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.524+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.533 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.524+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2358583337}) 2024-09-11T01:41:30.533 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.524+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1466439493}) 2024-09-11T01:41:30.534 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:41:30.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.777+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.777+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3346142098}) 2024-09-11T01:41:30.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.777+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1806839709}) 2024-09-11T01:41:30.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.777+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.777+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3346142098}) 2024-09-11T01:41:30.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.777+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1555801228}) 2024-09-11T01:41:30.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.777+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.777+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4212336136}) 2024-09-11T01:41:30.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.777+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1445440412}) 2024-09-11T01:41:30.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.777+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.777+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4212336136}) 2024-09-11T01:41:30.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.777+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=658380007}) 2024-09-11T01:41:30.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.778+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.778+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2852874498}) 2024-09-11T01:41:30.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.778+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=266975702}) 2024-09-11T01:41:30.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.778+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.778+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2852874498}) 2024-09-11T01:41:30.782 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.778+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3517166427}) 2024-09-11T01:41:30.782 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.779+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.782 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.779+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1279400227}) 2024-09-11T01:41:30.782 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.779+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2222344693}) 2024-09-11T01:41:30.782 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.779+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.782 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.779+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1279400227}) 2024-09-11T01:41:30.782 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.779+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1285772541}) 2024-09-11T01:41:30.787 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.780+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.787 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.780+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2238763618}) 2024-09-11T01:41:30.787 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.780+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1401769063}) 2024-09-11T01:41:30.787 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.780+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.787 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.780+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2238763618}) 2024-09-11T01:41:30.787 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.780+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=575595461}) 2024-09-11T01:41:30.787 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.780+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.780+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1713375624}) 2024-09-11T01:41:30.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.780+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3522752310}) 2024-09-11T01:41:30.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.780+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.780+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1713375624}) 2024-09-11T01:41:30.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.781+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1527613208}) 2024-09-11T01:41:30.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.781+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.781+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2013652065}) 2024-09-11T01:41:30.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.781+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3649755078}) 2024-09-11T01:41:30.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.781+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.781+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2013652065}) 2024-09-11T01:41:30.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.781+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1759730089}) 2024-09-11T01:41:30.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.782+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.782+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1148655138}) 2024-09-11T01:41:30.789 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.782+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4119700466}) 2024-09-11T01:41:30.789 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.782+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.789 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.782+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1148655138}) 2024-09-11T01:41:30.789 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.782+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1057011977}) 2024-09-11T01:41:30.789 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.783+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.789 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.783+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1693045540}) 2024-09-11T01:41:30.789 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.783+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1665143635}) 2024-09-11T01:41:30.789 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.783+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.789 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.783+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1693045540}) 2024-09-11T01:41:30.789 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.783+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2686671699}) 2024-09-11T01:41:30.789 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.784+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.789 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.784+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2193159682}) 2024-09-11T01:41:30.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.784+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1690060492}) 2024-09-11T01:41:30.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.784+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.784+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2193159682}) 2024-09-11T01:41:30.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.784+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2636776106}) 2024-09-11T01:41:30.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.784+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.784+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=171423381}) 2024-09-11T01:41:30.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.784+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2357711394}) 2024-09-11T01:41:30.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.784+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.784+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=171423381}) 2024-09-11T01:41:30.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.784+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1875301136}) 2024-09-11T01:41:30.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.785+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.785+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1111038042}) 2024-09-11T01:41:30.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.785+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1803031382}) 2024-09-11T01:41:30.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.785+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.791 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.785+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1111038042}) 2024-09-11T01:41:30.791 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.785+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1274662286}) 2024-09-11T01:41:30.797 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.786+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.797 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.786+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3389478069}) 2024-09-11T01:41:30.797 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.786+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2159970236}) 2024-09-11T01:41:30.797 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.786+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.797 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.786+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3389478069}) 2024-09-11T01:41:30.797 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.786+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2815454169}) 2024-09-11T01:41:30.797 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.787+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.797 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.787+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1877692121}) 2024-09-11T01:41:30.797 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.787+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=844491301}) 2024-09-11T01:41:30.797 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.787+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.798 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.787+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1877692121}) 2024-09-11T01:41:30.798 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.787+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2615699657}) 2024-09-11T01:41:30.798 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.788+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.798 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.788+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=457541470}) 2024-09-11T01:41:30.798 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.788+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3004390341}) 2024-09-11T01:41:30.798 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.788+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.798 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.788+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=457541470}) 2024-09-11T01:41:30.798 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.788+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1756177874}) 2024-09-11T01:41:30.798 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.789+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.798 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.789+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354656479}) 2024-09-11T01:41:30.798 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.789+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1037950462}) 2024-09-11T01:41:30.798 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.789+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.798 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.789+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354656479}) 2024-09-11T01:41:30.798 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.789+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=315895131}) 2024-09-11T01:41:30.799 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.790+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.799 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.790+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3883259379}) 2024-09-11T01:41:30.799 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.790+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2484246585}) 2024-09-11T01:41:30.799 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.790+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.799 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.790+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3883259379}) 2024-09-11T01:41:30.799 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.790+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=179820420}) 2024-09-11T01:41:30.799 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.790+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.799 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.790+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3950859687}) 2024-09-11T01:41:30.799 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.790+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1894089859}) 2024-09-11T01:41:30.799 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.790+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.799 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.791+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3950859687}) 2024-09-11T01:41:30.799 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.791+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2202164048}) 2024-09-11T01:41:30.799 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.791+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.800 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.791+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2000076047}) 2024-09-11T01:41:30.800 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.791+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3088250327}) 2024-09-11T01:41:30.800 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.791+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.800 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.791+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2000076047}) 2024-09-11T01:41:30.800 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.791+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1056024013}) 2024-09-11T01:41:30.800 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.792+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.800 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.792+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=42,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=5} crc {logm=881319676,mdsmap=3760429986,mgr=2672916156,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=3924264864,mon_config_key=508202868,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=1011494367,osd_snap=3537452557}) 2024-09-11T01:41:30.800 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.792+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=41,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=6} crc {logm=151831114,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=4279862985,mon_config_key=3884116196,monmap=2933275274,nvmeofgw=598337686,osd_metadata=642414809,osd_pg_creating=1011494367,osd_snap=1703336224}) 2024-09-11T01:41:30.800 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.792+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.800 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.792+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=42,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=5} crc {logm=881319676,mdsmap=3760429986,mgr=2672916156,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=3924264864,mon_config_key=508202868,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=1011494367,osd_snap=3537452557}) 2024-09-11T01:41:30.800 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.792+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=42,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=15,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=7} crc {logm=839742026,mdsmap=3760429986,mgr=4080174641,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2761204552,mon_config_key=1751862279,monmap=2933275274,nvmeofgw=1596038039,osd_metadata=1833811859,osd_pg_creating=1011494367,osd_snap=2052956484}) 2024-09-11T01:41:30.801 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.793+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.801 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.793+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=76,osdmap=24} crc {osd_snap=2836848583,osdmap=2032205574}) 2024-09-11T01:41:30.801 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.793+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=76,osdmap=24} crc {osd_snap=2741478539,osdmap=2127203348}) 2024-09-11T01:41:30.801 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.793+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.801 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.793+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=76,osdmap=24} crc {osd_snap=2836848583,osdmap=2032205574}) 2024-09-11T01:41:30.801 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.793+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=77,osdmap=23} crc {osd_snap=4048733153,osdmap=2844234338}) 2024-09-11T01:41:30.801 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.794+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.801 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.794+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=504794944}) 2024-09-11T01:41:30.801 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.794+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3425361714}) 2024-09-11T01:41:30.801 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.794+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.801 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.794+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=504794944}) 2024-09-11T01:41:30.801 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.794+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3478012823}) 2024-09-11T01:41:30.802 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.799+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.802 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.799+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=62} crc {osdmap=187601748}) 2024-09-11T01:41:30.802 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.799+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=65} crc {osdmap=3046807782}) 2024-09-11T01:41:30.802 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.799+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:41:30.802 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.799+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=62} crc {osdmap=187601748}) 2024-09-11T01:41:30.803 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:41:30.799+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=64} crc {osdmap=4106533119}) 2024-09-11T01:41:30.964 INFO:tasks.rados.rados.0.smithi071.stdout:1763: done (5 left) 2024-09-11T01:41:30.964 INFO:tasks.rados.rados.0.smithi071.stdout:1764: done (4 left) 2024-09-11T01:41:30.964 INFO:tasks.rados.rados.0.smithi071.stdout:1765: done (3 left) 2024-09-11T01:41:30.965 INFO:tasks.rados.rados.0.smithi071.stdout:1766: done (2 left) 2024-09-11T01:41:30.965 INFO:tasks.rados.rados.0.smithi071.stdout:1767: done (1 left) 2024-09-11T01:41:30.965 INFO:tasks.rados.rados.0.smithi071.stdout:1768: done (0 left) 2024-09-11T01:41:30.965 INFO:tasks.rados.rados.0.smithi071.stdout:1769: copy_from oid 6 from oid 29 current snap is 198 2024-09-11T01:41:30.965 INFO:tasks.rados.rados.0.smithi071.stdout:1770: copy_from oid 24 from oid 27 current snap is 198 2024-09-11T01:41:30.965 INFO:tasks.rados.rados.0.smithi071.stdout:1771: snap_create 2024-09-11T01:41:30.974 INFO:tasks.rados.rados.0.smithi071.stdout:1770: finishing copy_from racing read to smithi07134777-24 2024-09-11T01:41:30.977 INFO:tasks.rados.rados.0.smithi071.stdout:1770: finishing copy_from to smithi07134777-24 2024-09-11T01:41:30.978 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 853 (ObjNum 623 snap 187 seq_num 623) dirty exists 2024-09-11T01:41:30.978 INFO:tasks.rados.rados.0.smithi071.stdout:1769: finishing copy_from racing read to smithi07134777-6 2024-09-11T01:41:30.979 INFO:tasks.rados.rados.0.smithi071.stdout:1769: finishing copy_from to smithi07134777-6 2024-09-11T01:41:30.979 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 938 (ObjNum 617 snap 183 seq_num 617) dirty exists 2024-09-11T01:41:31.535 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:41:31.535 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:41:31.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.647+0000 7fd4c7270640 10 monclient: get_monmap_and_config 2024-09-11T01:41:31.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.647+0000 7fd4c7270640 10 monclient: build_initial_monmap 2024-09-11T01:41:31.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.647+0000 7fd4c7270640 10 monclient: monmap: 2024-09-11T01:41:31.648 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:31.649 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:31.649 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:41:31.647788+0000 2024-09-11T01:41:31.649 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:41:31.647788+0000 2024-09-11T01:41:31.649 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:31.649 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:31.649 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:41:31.649 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:41:31.649 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:41:31.649 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:41:31.649 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:41:31.649 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:31.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.647+0000 7fd4c7270640 1 Processor -- start 2024-09-11T01:41:31.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.647+0000 7fd4c7270640 1 -- start start 2024-09-11T01:41:31.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.647+0000 7fd4c7270640 10 monclient: init 2024-09-11T01:41:31.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c7270640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:31.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c7270640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-11T01:41:31.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c7270640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd4c0138210 0x7fd4c01385f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:31.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c7270640 10 monclient(hunting): picked mon.noname-b con 0x7fd4c0138210 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:31.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c7270640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd4c00a6810 0x7fd4c00a5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:31.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c7270640 10 monclient(hunting): picked mon.noname-c con 0x7fd4c00a6810 addr v2:172.21.15.71:3300/0 2024-09-11T01:41:31.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c7270640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd4c00a55d0 addr v1:172.21.15.12:6789/0 2024-09-11T01:41:31.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c7270640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4c009ba20 con 0x7fd4c00a55d0 2024-09-11T01:41:31.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c7270640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:31.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c7270640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fd4c009c7a0 con 0x7fd4c00a6810 2024-09-11T01:41:31.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c7270640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:31.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c7270640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd4c009c4c0 con 0x7fd4c0138210 2024-09-11T01:41:31.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c7270640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:31.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c7270640 10 monclient(hunting): authenticate will time out at 1498.090600s 2024-09-11T01:41:31.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c500c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd4c0138210 0x7fd4c01385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:31.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c500c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd4c0138210 0x7fd4c01385f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:56102/0 (socket says 172.21.15.12:56102) 2024-09-11T01:41:31.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c500c640 1 -- 172.21.15.12:0/4035287256 learned_addr learned my addr 172.21.15.12:0/4035287256 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:41:31.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c500c640 10 monclient(hunting): get_auth_request con 0x7fd4c0138210 auth_method 0 2024-09-11T01:41:31.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c500c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:31.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c500c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:31.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c500c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:31.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c500c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:31.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c500c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:31.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c500c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:31.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c480b640 1 --2- 172.21.15.12:0/4035287256 >> v2:172.21.15.71:3300/0 conn(0x7fd4c00a6810 0x7fd4c00a5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:31.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c480b640 10 monclient(hunting): get_auth_request con 0x7fd4c00a6810 auth_method 0 2024-09-11T01:41:31.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c480b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:31.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c480b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:31.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c480b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:31.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.648+0000 7fd4c500c640 10 monclient(hunting): handle_auth_done global_id 114308 payload 274 2024-09-11T01:41:31.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.649+0000 7fd4c500c640 1 -- 172.21.15.12:0/4035287256 >> v2:172.21.15.71:3300/0 conn(0x7fd4c00a6810 msgr2=0x7fd4c00a5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:31.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.649+0000 7fd4c500c640 1 --2- 172.21.15.12:0/4035287256 >> v2:172.21.15.71:3300/0 conn(0x7fd4c00a6810 0x7fd4c00a5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:31.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.649+0000 7fd4c500c640 1 -- 172.21.15.12:0/4035287256 >> v1:172.21.15.12:6789/0 conn(0x7fd4c00a55d0 legacy=0x7fd4c013f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:31.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.649+0000 7fd4c500c640 10 monclient: _finish_hunting 0 2024-09-11T01:41:31.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.649+0000 7fd4c500c640 1 monclient: found mon.noname-b 2024-09-11T01:41:31.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.649+0000 7fd4c500c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:31.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.649+0000 7fd4c500c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:41:31.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.649+0000 7fd4c500c640 1 -- 172.21.15.12:0/4035287256 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4c00a6590 con 0x7fd4c0138210 2024-09-11T01:41:31.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.649+0000 7fd4c480b640 1 --2- 172.21.15.12:0/4035287256 >> v2:172.21.15.71:3300/0 conn(0x7fd4c00a6810 0x7fd4c00a5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:41:31.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.651+0000 7fd4c500c640 1 -- 172.21.15.12:0/4035287256 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd4c0138210 msgr2=0x7fd4c01385f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:31.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.651+0000 7fd4c500c640 1 --2- 172.21.15.12:0/4035287256 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd4c0138210 0x7fd4c01385f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc009870 tx=0x7fd4bc02a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c943fdd0788c6eb5 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:31.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4b7fff640 1 -- 172.21.15.12:0/4035287256 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4bc037070 con 0x7fd4c0138210 2024-09-11T01:41:31.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:31.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4b7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:41:31.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4b7fff640 10 monclient: dump: 2024-09-11T01:41:31.660 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:31.661 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:31.661 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:31.661 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:31.661 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:31.661 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:31.661 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:31.661 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:31.661 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:31.661 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:31.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4b7fff640 10 monclient: _finish_auth 0 2024-09-11T01:41:31.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4b7fff640 10 monclient: _check_auth_tickets 2024-09-11T01:41:31.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:31.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4c7270640 5 monclient: authenticate success, global_id 114308 2024-09-11T01:41:31.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4c7270640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:41:31.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4b7fff640 1 -- 172.21.15.12:0/4035287256 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4bc032440 con 0x7fd4c0138210 2024-09-11T01:41:31.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4b7fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:31.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4c7270640 10 monclient: get_monmap_and_config success 2024-09-11T01:41:31.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4c7270640 4 set_mon_vals no callback set 2024-09-11T01:41:31.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4b7fff640 1 -- 172.21.15.12:0/4035287256 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4bc032810 con 0x7fd4c0138210 2024-09-11T01:41:31.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:31.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4b7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:41:31.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4b7fff640 10 monclient: dump: 2024-09-11T01:41:31.662 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:31.662 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:31.662 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:31.663 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4c7270640 10 monclient: shutdown 2024-09-11T01:41:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4c7270640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4c7270640 1 -- 172.21.15.12:0/4035287256 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd4c0138210 msgr2=0x7fd4c01385f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4c7270640 1 --2- 172.21.15.12:0/4035287256 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd4c0138210 0x7fd4c01385f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc009870 tx=0x7fd4bc02a0a0 comp rx=0 tx=0).stop 2024-09-11T01:41:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4c7270640 1 -- 172.21.15.12:0/4035287256 shutdown_connections 2024-09-11T01:41:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4c7270640 1 --2- 172.21.15.12:0/4035287256 >> v2:172.21.15.71:3300/0 conn(0x7fd4c00a6810 0x7fd4c00a5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:31.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4c7270640 1 --2- 172.21.15.12:0/4035287256 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd4c0138210 0x7fd4c01385f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4c7270640 1 -- 172.21.15.12:0/4035287256 >> 172.21.15.12:0/4035287256 conn(0x7fd4c00aaa80 msgr2=0x7fd4c00aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4c7270640 1 -- 172.21.15.12:0/4035287256 shutdown_connections 2024-09-11T01:41:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.652+0000 7fd4c7270640 1 -- 172.21.15.12:0/4035287256 wait complete. 2024-09-11T01:41:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.653+0000 7fd4c7270640 10 monclient: build_initial_monmap 2024-09-11T01:41:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.653+0000 7fd4c7270640 10 monclient: monmap: 2024-09-11T01:41:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:41:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:41:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:41:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:41:31.664 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:41:31.665 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:41:31.665 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:31.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.653+0000 7fd4c7270640 1 Processor -- start 2024-09-11T01:41:31.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.653+0000 7fd4c7270640 1 -- start start 2024-09-11T01:41:31.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.653+0000 7fd4c7270640 10 monclient: init 2024-09-11T01:41:31.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c7270640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:31.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c7270640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:41:31.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c7270640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd4c00a55d0 0x7fd4c0137030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:31.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c7270640 10 monclient(hunting): picked mon.noname-b con 0x7fd4c00a55d0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:41:31.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c7270640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd4c00a6810 0x7fd4c0137570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:31.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c7270640 10 monclient(hunting): picked mon.noname-c con 0x7fd4c00a6810 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:31.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c7270640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4c0138210 0x7fd4c0124da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:31.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c7270640 10 monclient(hunting): picked mon.noname-a con 0x7fd4c0138210 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:41:31.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c7270640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:31.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c7270640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c006f670 con 0x7fd4c0138210 2024-09-11T01:41:31.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c7270640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:31.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c7270640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c006f4f0 con 0x7fd4c00a55d0 2024-09-11T01:41:31.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c7270640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:31.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c7270640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd4c0138b30 con 0x7fd4c00a6810 2024-09-11T01:41:31.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c7270640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:31.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c7270640 10 monclient(hunting): authenticate will time out at 1498.096351s 2024-09-11T01:41:31.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c480b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd4c00a6810 0x7fd4c0137570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:31.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c580d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4c0138210 0x7fd4c0124da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:31.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c480b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd4c00a6810 0x7fd4c0137570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:56112/0 (socket says 172.21.15.12:56112) 2024-09-11T01:41:31.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c580d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4c0138210 0x7fd4c0124da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58674/0 (socket says 172.21.15.12:58674) 2024-09-11T01:41:31.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c480b640 1 -- 172.21.15.12:0/4096825402 learned_addr learned my addr 172.21.15.12:0/4096825402 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:41:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c480b640 10 monclient(hunting): get_auth_request con 0x7fd4c00a6810 auth_method 0 2024-09-11T01:41:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c480b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c480b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c480b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c580d640 10 monclient(hunting): get_auth_request con 0x7fd4c0138210 auth_method 0 2024-09-11T01:41:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c580d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c580d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c580d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c500c640 1 --2- 172.21.15.12:0/4096825402 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd4c00a55d0 0x7fd4c0137030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c500c640 10 monclient(hunting): get_auth_request con 0x7fd4c00a55d0 auth_method 0 2024-09-11T01:41:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c500c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c500c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c500c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:31.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c480b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:31.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c480b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:31.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c480b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:31.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c580d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:31.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c580d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:31.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c580d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:31.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c500c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:31.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c500c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:31.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c500c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:31.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c580d640 10 monclient(hunting): handle_auth_done global_id 94476 payload 824 2024-09-11T01:41:31.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c580d640 1 -- 172.21.15.12:0/4096825402 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd4c00a6810 msgr2=0x7fd4c0137570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:31.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c580d640 1 --2- 172.21.15.12:0/4096825402 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd4c00a6810 0x7fd4c0137570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:31.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c580d640 1 -- 172.21.15.12:0/4096825402 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd4c00a55d0 msgr2=0x7fd4c0137030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:31.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c580d640 1 --2- 172.21.15.12:0/4096825402 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd4c00a55d0 0x7fd4c0137030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c580d640 10 monclient: _finish_hunting 0 2024-09-11T01:41:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c580d640 1 monclient: found mon.noname-a 2024-09-11T01:41:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c580d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c580d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:41:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c580d640 1 -- 172.21.15.12:0/4096825402 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4c01267d0 con 0x7fd4c0138210 2024-09-11T01:41:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.654+0000 7fd4c480b640 1 --2- 172.21.15.12:0/4096825402 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd4c00a6810 0x7fd4c0137570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:41:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.657+0000 7fd4c580d640 1 -- 172.21.15.12:0/4096825402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4c0138210 msgr2=0x7fd4c0124da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.657+0000 7fd4c580d640 1 --2- 172.21.15.12:0/4096825402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4c0138210 0x7fd4c0124da0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd4b800c780 tx=0x7fd4b800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.657+0000 7fd4b5ffb640 1 -- 172.21.15.12:0/4096825402 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b8016020 con 0x7fd4c0138210 2024-09-11T01:41:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.657+0000 7fd4b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.657+0000 7fd4b5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:41:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.657+0000 7fd4b5ffb640 10 monclient: dump: 2024-09-11T01:41:31.669 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:31.670 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.657+0000 7fd4b5ffb640 10 monclient: _finish_auth 0 2024-09-11T01:41:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.657+0000 7fd4b5ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:41:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.657+0000 7fd4b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.657+0000 7fd4b5ffb640 1 -- 172.21.15.12:0/4096825402 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4b800ac60 con 0x7fd4c0138210 2024-09-11T01:41:31.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.657+0000 7fd4c7270640 5 monclient: authenticate success, global_id 94476 2024-09-11T01:41:31.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.657+0000 7fd4c7270640 10 monclient: _renew_subs 2024-09-11T01:41:31.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.657+0000 7fd4c7270640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:31.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.657+0000 7fd4c7270640 1 -- 172.21.15.12:0/4096825402 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4c00a1880 con 0x7fd4c0138210 2024-09-11T01:41:31.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.657+0000 7fd4b5ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:31.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.657+0000 7fd4b5ffb640 1 -- 172.21.15.12:0/4096825402 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b8012660 con 0x7fd4c0138210 2024-09-11T01:41:31.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.657+0000 7fd4b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:31.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.657+0000 7fd4b5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:31.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.657+0000 7fd4b5ffb640 10 monclient: dump: 2024-09-11T01:41:31.671 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:31.671 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:31.671 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:31.671 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:31.672 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:31.672 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:31.672 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:31.672 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:31.672 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:31.672 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:31.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.657+0000 7fd4c7270640 10 monclient: _renew_subs 2024-09-11T01:41:31.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.657+0000 7fd4b77fe640 4 set_mon_vals no callback set 2024-09-11T01:41:31.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.657+0000 7fd4c7270640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:31.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.657+0000 7fd4c7270640 1 -- 172.21.15.12:0/4096825402 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4c00a1120 con 0x7fd4c0138210 2024-09-11T01:41:31.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.658+0000 7fd4c7270640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:31.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.658+0000 7fd4c7270640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:31.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.658+0000 7fd4c7270640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.658+0000 7fd4b5ffb640 1 -- 172.21.15.12:0/4096825402 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fd4b80128c0 con 0x7fd4c0138210 2024-09-11T01:41:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.658+0000 7fd4c7270640 1 -- 172.21.15.12:0/4096825402 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4c01283e0 con 0x7fd4c0138210 2024-09-11T01:41:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.658+0000 7fd4b5ffb640 1 --2- 172.21.15.12:0/4096825402 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd4a403e210 0x7fd4a40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.658+0000 7fd4b5ffb640 1 -- 172.21.15.12:0/4096825402 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(407..407 src has 285..407) ==== 4739+0+0 (secure 0 0 0) 0x7fd4b8052f30 con 0x7fd4c0138210 2024-09-11T01:41:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.659+0000 7fd4c500c640 1 --2- 172.21.15.12:0/4096825402 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd4a403e210 0x7fd4a40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.659+0000 7fd4c500c640 10 monclient: get_auth_request con 0x7fd4a403e210 auth_method 0 2024-09-11T01:41:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.661+0000 7fd4c500c640 1 --2- 172.21.15.12:0/4096825402 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd4a403e210 0x7fd4a40406d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd4c009d460 tx=0x7fd4bc02e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.663+0000 7fd4b5ffb640 1 -- 172.21.15.12:0/4096825402 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd4b8020030 con 0x7fd4c0138210 2024-09-11T01:41:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.663+0000 7fd4b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.663+0000 7fd4b5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:41:31.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.919+0000 7fd4c7270640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:41:31.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.919+0000 7fd4c7270640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:41:31.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.919+0000 7fd4c7270640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:31.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.919+0000 7fd4c7270640 1 -- 172.21.15.12:0/4096825402 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd4c00a5090 con 0x7fd4c0138210 2024-09-11T01:41:31.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.919+0000 7fd4b5ffb640 1 -- 172.21.15.12:0/4096825402 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fd4c00a1880 con 0x7fd4c0138210 2024-09-11T01:41:31.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.919+0000 7fd4b5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:41:31.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.919+0000 7fd4b5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:41:31.921 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:41:31.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.921+0000 7fd4c7270640 1 -- 172.21.15.12:0/4096825402 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd4a403e210 msgr2=0x7fd4a40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:31.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.921+0000 7fd4c7270640 1 --2- 172.21.15.12:0/4096825402 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd4a403e210 0x7fd4a40406d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd4c009d460 tx=0x7fd4bc02e000 comp rx=0 tx=0).stop 2024-09-11T01:41:31.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.921+0000 7fd4c7270640 10 monclient: shutdown 2024-09-11T01:41:31.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.921+0000 7fd4c7270640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:31.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.921+0000 7fd4c7270640 1 -- 172.21.15.12:0/4096825402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4c0138210 msgr2=0x7fd4c0124da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:31.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.921+0000 7fd4c7270640 1 --2- 172.21.15.12:0/4096825402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4c0138210 0x7fd4c0124da0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd4b800c780 tx=0x7fd4b800cc40 comp rx=0 tx=0).stop 2024-09-11T01:41:31.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.921+0000 7fd4c7270640 1 -- 172.21.15.12:0/4096825402 shutdown_connections 2024-09-11T01:41:31.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.921+0000 7fd4c7270640 1 --2- 172.21.15.12:0/4096825402 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd4a403e210 0x7fd4a40406d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:31.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.921+0000 7fd4c7270640 1 --2- 172.21.15.12:0/4096825402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd4c0138210 0x7fd4c0124da0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:31.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.921+0000 7fd4c7270640 1 --2- 172.21.15.12:0/4096825402 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd4c00a6810 0x7fd4c0137570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:31.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.921+0000 7fd4c7270640 1 --2- 172.21.15.12:0/4096825402 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd4c00a55d0 0x7fd4c0137030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:31.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.921+0000 7fd4c7270640 1 -- 172.21.15.12:0/4096825402 >> 172.21.15.12:0/4096825402 conn(0x7fd4c00aaa80 msgr2=0x7fd4c00a3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:31.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.921+0000 7fd4c7270640 1 -- 172.21.15.12:0/4096825402 shutdown_connections 2024-09-11T01:41:31.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:31.921+0000 7fd4c7270640 1 -- 172.21.15.12:0/4096825402 wait complete. 2024-09-11T01:41:31.933 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:41:31.933 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:41:31.933 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:41:31.934 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:41:31.971 INFO:tasks.rados.rados.0.smithi071.stdout:1769: done (2 left) 2024-09-11T01:41:31.971 INFO:tasks.rados.rados.0.smithi071.stdout:1770: done (1 left) 2024-09-11T01:41:31.971 INFO:tasks.rados.rados.0.smithi071.stdout:1771: done (0 left) 2024-09-11T01:41:31.971 INFO:tasks.rados.rados.0.smithi071.stdout:1772: write (excl) oid 40 current snap is 199 2024-09-11T01:41:31.971 INFO:tasks.rados.rados.0.smithi071.stdout:1772: seq_num 647 ranges {474129=576108,1627732=587214,2718088=156041} 2024-09-11T01:41:31.977 INFO:tasks.rados.rados.0.smithi071.stdout:1772: writing smithi07134777-40 from 474129 to 1050237 tid 1 2024-09-11T01:41:31.983 INFO:tasks.rados.rados.0.smithi071.stdout:1772: writing smithi07134777-40 from 1627732 to 2214946 tid 2 2024-09-11T01:41:31.987 INFO:tasks.rados.rados.0.smithi071.stdout:1772: writing smithi07134777-40 from 2718088 to 2874129 tid 3 2024-09-11T01:41:31.988 INFO:tasks.rados.rados.0.smithi071.stdout:1773: rollback oid 48 current snap is 199 2024-09-11T01:41:31.988 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 48 to 185 2024-09-11T01:41:31.988 INFO:tasks.rados.rados.0.smithi071.stdout:1772: finishing write tid 1 to smithi07134777-40 2024-09-11T01:41:31.988 INFO:tasks.rados.rados.0.smithi071.stdout:1772: finishing write tid 2 to smithi07134777-40 2024-09-11T01:41:31.988 INFO:tasks.rados.rados.0.smithi071.stdout:1774: write (excl) oid 42 current snap is 199 2024-09-11T01:41:31.988 INFO:tasks.rados.rados.0.smithi071.stdout:1774: seq_num 648 ranges {490936=651357,1686214=447225,2861864=29072} 2024-09-11T01:41:31.994 INFO:tasks.rados.rados.0.smithi071.stdout:1774: writing smithi07134777-42 from 490936 to 1142293 tid 1 2024-09-11T01:41:31.999 INFO:tasks.rados.rados.0.smithi071.stdout:1774: writing smithi07134777-42 from 1686214 to 2133439 tid 2 2024-09-11T01:41:32.004 INFO:tasks.rados.rados.0.smithi071.stdout:1774: writing smithi07134777-42 from 2861864 to 2890936 tid 3 2024-09-11T01:41:32.004 INFO:tasks.rados.rados.0.smithi071.stdout:1775: read oid 18 snap -1 2024-09-11T01:41:32.004 INFO:tasks.rados.rados.0.smithi071.stdout:1775: expect (ObjNum 618 snap 183 seq_num 618) 2024-09-11T01:41:32.004 INFO:tasks.rados.rados.0.smithi071.stdout:1776: write oid 46 current snap is 199 2024-09-11T01:41:32.004 INFO:tasks.rados.rados.0.smithi071.stdout:1772: finishing write tid 3 to smithi07134777-40 2024-09-11T01:41:32.004 INFO:tasks.rados.rados.0.smithi071.stdout:1776: seq_num 649 ranges {507743=726606,1744696=590883,2772934=134809} 2024-09-11T01:41:32.011 INFO:tasks.rados.rados.0.smithi071.stdout:1776: writing smithi07134777-46 from 507743 to 1234349 tid 1 2024-09-11T01:41:32.017 INFO:tasks.rados.rados.0.smithi071.stdout:1776: writing smithi07134777-46 from 1744696 to 2335579 tid 2 2024-09-11T01:41:32.021 INFO:tasks.rados.rados.0.smithi071.stdout:1776: writing smithi07134777-46 from 2772934 to 2907743 tid 3 2024-09-11T01:41:32.021 INFO:tasks.rados.rados.0.smithi071.stdout:1777: delete oid 5 current snap is 199 2024-09-11T01:41:32.021 INFO:tasks.rados.rados.0.smithi071.stdout:1772: finishing write tid 4 to smithi07134777-40 2024-09-11T01:41:32.021 INFO:tasks.rados.rados.0.smithi071.stdout:1772: finishing write tid 5 to smithi07134777-40 2024-09-11T01:41:32.021 INFO:tasks.rados.rados.0.smithi071.stdout:1772: oid 40 updating version 0 to 699 2024-09-11T01:41:32.021 INFO:tasks.rados.rados.0.smithi071.stdout:1772: oid 40 version 699 is already newer than 697 2024-09-11T01:41:32.021 INFO:tasks.rados.rados.0.smithi071.stdout:1772: oid 40 updating version 699 to 700 2024-09-11T01:41:32.021 INFO:tasks.rados.rados.0.smithi071.stdout:1772: oid 40 version 700 is already newer than 698 2024-09-11T01:41:32.021 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 700 (ObjNum 647 snap 199 seq_num 647) dirty exists 2024-09-11T01:41:32.021 INFO:tasks.rados.rados.0.smithi071.stdout:1772: left oid 40 (ObjNum 647 snap 199 seq_num 647) 2024-09-11T01:41:32.022 INFO:tasks.rados.rados.0.smithi071.stdout:1773: finishing rollback tid 1 to smithi07134777-48 2024-09-11T01:41:32.022 INFO:tasks.rados.rados.0.smithi071.stdout:1773: finishing rollback tid 2 to smithi07134777-48 2024-09-11T01:41:32.022 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 998 (ObjNum 586 snap 168 seq_num 586) dirty exists 2024-09-11T01:41:32.022 INFO:tasks.rados.rados.0.smithi071.stdout:1774: finishing write tid 1 to smithi07134777-42 2024-09-11T01:41:32.022 INFO:tasks.rados.rados.0.smithi071.stdout:1774: finishing write tid 2 to smithi07134777-42 2024-09-11T01:41:32.022 INFO:tasks.rados.rados.0.smithi071.stdout:1774: finishing write tid 3 to smithi07134777-42 2024-09-11T01:41:32.022 INFO:tasks.rados.rados.0.smithi071.stdout:1774: finishing write tid 4 to smithi07134777-42 2024-09-11T01:41:32.022 INFO:tasks.rados.rados.0.smithi071.stdout:1774: finishing write tid 5 to smithi07134777-42 2024-09-11T01:41:32.022 INFO:tasks.rados.rados.0.smithi071.stdout:1774: oid 42 updating version 0 to 817 2024-09-11T01:41:32.022 INFO:tasks.rados.rados.0.smithi071.stdout:1774: oid 42 updating version 817 to 818 2024-09-11T01:41:32.022 INFO:tasks.rados.rados.0.smithi071.stdout:1774: oid 42 updating version 818 to 819 2024-09-11T01:41:32.022 INFO:tasks.rados.rados.0.smithi071.stdout:1774: oid 42 updating version 819 to 820 2024-09-11T01:41:32.022 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 820 (ObjNum 648 snap 199 seq_num 648) dirty exists 2024-09-11T01:41:32.022 INFO:tasks.rados.rados.0.smithi071.stdout:1774: left oid 42 (ObjNum 648 snap 199 seq_num 648) 2024-09-11T01:41:32.023 INFO:tasks.rados.rados.0.smithi071.stdout:1775: expect (ObjNum 618 snap 183 seq_num 618) 2024-09-11T01:41:32.033 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 830104 2024-09-11T01:41:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.042+0000 7f89e609b640 10 monclient: get_monmap_and_config 2024-09-11T01:41:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.042+0000 7f89e609b640 10 monclient: build_initial_monmap 2024-09-11T01:41:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.042+0000 7f89e609b640 10 monclient: monmap: 2024-09-11T01:41:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:41:32.042928+0000 2024-09-11T01:41:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:41:32.042928+0000 2024-09-11T01:41:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:41:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:41:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:41:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:41:32.044 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:41:32.044 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.042+0000 7f89e609b640 1 Processor -- start 2024-09-11T01:41:32.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.042+0000 7f89e609b640 1 -- start start 2024-09-11T01:41:32.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.042+0000 7f89e609b640 10 monclient: init 2024-09-11T01:41:32.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.043+0000 7f89e609b640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.043+0000 7f89e609b640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-11T01:41:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.043+0000 7f89e609b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f89e0137e50 0x7f89e009d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.043+0000 7f89e609b640 10 monclient(hunting): picked mon.noname-c con 0x7f89e0137e50 addr v2:172.21.15.71:3300/0 2024-09-11T01:41:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.043+0000 7f89e609b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f89e0138230 addr v1:172.21.15.12:6789/0 2024-09-11T01:41:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.043+0000 7f89e609b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f89e013d380 addr v1:172.21.15.71:6789/0 2024-09-11T01:41:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.043+0000 7f89e609b640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89e00a62d0 con 0x7f89e0138230 2024-09-11T01:41:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.043+0000 7f89e609b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89e009bb40 con 0x7f89e013d380 2024-09-11T01:41:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.043+0000 7f89e609b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.043+0000 7f89e609b640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f89e009c4a0 con 0x7f89e0137e50 2024-09-11T01:41:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.043+0000 7f89e609b640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.043+0000 7f89e609b640 10 monclient(hunting): authenticate will time out at 1498.485526s 2024-09-11T01:41:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.043+0000 7f89deffd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f89e0138230 0x7f89e0138640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42304/0 (socket says 172.21.15.12:42304) 2024-09-11T01:41:32.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.043+0000 7f89deffd640 1 -- 172.21.15.12:0/2093956892 learned_addr learned my addr 172.21.15.12:0/2093956892 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:41:32.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.043+0000 7f89df7fe640 1 --2- 172.21.15.12:0/2093956892 >> v2:172.21.15.71:3300/0 conn(0x7f89e0137e50 0x7f89e009d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:32.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.043+0000 7f89df7fe640 10 monclient(hunting): get_auth_request con 0x7f89e0137e50 auth_method 0 2024-09-11T01:41:32.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.043+0000 7f89df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:32.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.043+0000 7f89df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:32.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.043+0000 7f89df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:32.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.043+0000 7f89df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:32.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.043+0000 7f89df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:32.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.043+0000 7f89df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:32.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.043+0000 7f89df7fe640 10 monclient(hunting): handle_auth_done global_id 134134 payload 274 2024-09-11T01:41:32.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.043+0000 7f89df7fe640 1 -- 172.21.15.12:0/2093956892 >> v1:172.21.15.71:6789/0 conn(0x7f89e013d380 legacy=0x7f89e013f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.043+0000 7f89df7fe640 1 -- 172.21.15.12:0/2093956892 >> v1:172.21.15.12:6789/0 conn(0x7f89e0138230 legacy=0x7f89e0138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.044+0000 7f89df7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:41:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.044+0000 7f89df7fe640 1 monclient: found mon.noname-c 2024-09-11T01:41:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.044+0000 7f89df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.044+0000 7f89df7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:41:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.044+0000 7f89df7fe640 1 -- 172.21.15.12:0/2093956892 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e009c780 con 0x7f89e0137e50 2024-09-11T01:41:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.046+0000 7f89df7fe640 1 -- 172.21.15.12:0/2093956892 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89e0137e50 msgr2=0x7f89e009d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.046+0000 7f89df7fe640 1 --2- 172.21.15.12:0/2093956892 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89e0137e50 0x7f89e009d160 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f89e0138d80 tx=0x7f89d0029fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ab7ea1495f42f32 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.047+0000 7f89de7fc640 1 -- 172.21.15.12:0/2093956892 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d0037070 con 0x7f89e0137e50 2024-09-11T01:41:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.047+0000 7f89de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.047+0000 7f89de7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:41:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.047+0000 7f89de7fc640 10 monclient: dump: 2024-09-11T01:41:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:32.057 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.047+0000 7f89de7fc640 10 monclient: _finish_auth 0 2024-09-11T01:41:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.047+0000 7f89de7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:41:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.047+0000 7f89de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.047+0000 7f89de7fc640 1 -- 172.21.15.12:0/2093956892 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89d0032440 con 0x7f89e0137e50 2024-09-11T01:41:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.047+0000 7f89e609b640 5 monclient: authenticate success, global_id 134134 2024-09-11T01:41:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.047+0000 7f89e609b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:41:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.047+0000 7f89de7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.047+0000 7f89e609b640 10 monclient: get_monmap_and_config success 2024-09-11T01:41:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.047+0000 7f89de7fc640 1 -- 172.21.15.12:0/2093956892 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d0032810 con 0x7f89e0137e50 2024-09-11T01:41:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.047+0000 7f89e609b640 4 set_mon_vals no callback set 2024-09-11T01:41:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.047+0000 7f89de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.047+0000 7f89de7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:41:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.047+0000 7f89de7fc640 10 monclient: dump: 2024-09-11T01:41:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:32.058 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.047+0000 7f89e609b640 10 monclient: shutdown 2024-09-11T01:41:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.047+0000 7f89e609b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.047+0000 7f89e609b640 1 -- 172.21.15.12:0/2093956892 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89e0137e50 msgr2=0x7f89e009d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.047+0000 7f89e609b640 1 --2- 172.21.15.12:0/2093956892 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89e0137e50 0x7f89e009d160 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f89e0138d80 tx=0x7f89d0029fb0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.047+0000 7f89e609b640 1 -- 172.21.15.12:0/2093956892 shutdown_connections 2024-09-11T01:41:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.047+0000 7f89e609b640 1 --2- 172.21.15.12:0/2093956892 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89e0137e50 0x7f89e009d160 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.047+0000 7f89e609b640 1 -- 172.21.15.12:0/2093956892 >> 172.21.15.12:0/2093956892 conn(0x7f89e00a7020 msgr2=0x7f89e00a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.047+0000 7f89e609b640 1 -- 172.21.15.12:0/2093956892 shutdown_connections 2024-09-11T01:41:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.047+0000 7f89e609b640 1 -- 172.21.15.12:0/2093956892 wait complete. 2024-09-11T01:41:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.048+0000 7f89e609b640 10 monclient: build_initial_monmap 2024-09-11T01:41:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.048+0000 7f89e609b640 10 monclient: monmap: 2024-09-11T01:41:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:41:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:41:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:41:32.060 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:32.060 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.060 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:41:32.060 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:41:32.060 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:41:32.060 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.048+0000 7f89e609b640 1 Processor -- start 2024-09-11T01:41:32.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.048+0000 7f89e609b640 1 -- start start 2024-09-11T01:41:32.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.048+0000 7f89e609b640 10 monclient: init 2024-09-11T01:41:32.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89e609b640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:32.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89e609b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:41:32.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89e609b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f89e0137e50 0x7f89e0136a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89e609b640 10 monclient(hunting): picked mon.noname-c con 0x7f89e0137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:32.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89e609b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e0138230 0x7f89e0136f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89e609b640 10 monclient(hunting): picked mon.noname-a con 0x7f89e0138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:41:32.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89e609b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89e013d380 0x7f89e00a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89e609b640 10 monclient(hunting): picked mon.noname-b con 0x7f89e013d380 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:41:32.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89e609b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:32.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89e609b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f89e006f890 con 0x7f89e0138230 2024-09-11T01:41:32.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89e609b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:32.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89e609b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f89e006f710 con 0x7f89e013d380 2024-09-11T01:41:32.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89e609b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:32.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89e609b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f89e0140e90 con 0x7f89e0137e50 2024-09-11T01:41:32.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89e609b640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:32.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89e609b640 10 monclient(hunting): authenticate will time out at 1498.491217s 2024-09-11T01:41:32.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89deffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e0138230 0x7f89e0136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:32.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89df7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f89e0137e50 0x7f89e0136a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:32.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89deffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e0138230 0x7f89e0136f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58688/0 (socket says 172.21.15.12:58688) 2024-09-11T01:41:32.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89df7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f89e0137e50 0x7f89e0136a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:56128/0 (socket says 172.21.15.12:56128) 2024-09-11T01:41:32.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89deffd640 1 -- 172.21.15.12:0/3239561686 learned_addr learned my addr 172.21.15.12:0/3239561686 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:41:32.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89deffd640 10 monclient(hunting): get_auth_request con 0x7f89e0138230 auth_method 0 2024-09-11T01:41:32.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:32.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89deffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:32.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:32.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89df7fe640 10 monclient(hunting): get_auth_request con 0x7f89e0137e50 auth_method 0 2024-09-11T01:41:32.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:32.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:32.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:32.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89dffff640 1 --2- 172.21.15.12:0/3239561686 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89e013d380 0x7f89e00a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:32.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:32.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:32.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:32.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:32.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:32.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:32.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89dffff640 10 monclient(hunting): get_auth_request con 0x7f89e013d380 auth_method 0 2024-09-11T01:41:32.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:32.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89dffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:32.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:32.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:32.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:32.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:32.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89deffd640 10 monclient(hunting): handle_auth_done global_id 94479 payload 824 2024-09-11T01:41:32.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89deffd640 1 -- 172.21.15.12:0/3239561686 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f89e0137e50 msgr2=0x7f89e0136a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:32.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89deffd640 1 --2- 172.21.15.12:0/3239561686 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f89e0137e50 0x7f89e0136a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89deffd640 1 -- 172.21.15.12:0/3239561686 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89e013d380 msgr2=0x7f89e00a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:32.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89deffd640 1 --2- 172.21.15.12:0/3239561686 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89e013d380 0x7f89e00a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89deffd640 10 monclient: _finish_hunting 0 2024-09-11T01:41:32.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89deffd640 1 monclient: found mon.noname-a 2024-09-11T01:41:32.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:32.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89deffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:41:32.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89deffd640 1 -- 172.21.15.12:0/3239561686 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e00a9380 con 0x7f89e0138230 2024-09-11T01:41:32.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.049+0000 7f89df7fe640 1 --2- 172.21.15.12:0/3239561686 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f89e0137e50 0x7f89e0136a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:41:32.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.051+0000 7f89deffd640 1 -- 172.21.15.12:0/3239561686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e0138230 msgr2=0x7f89e0136f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:32.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.051+0000 7f89deffd640 1 --2- 172.21.15.12:0/3239561686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e0138230 0x7f89e0136f40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f89d400c460 tx=0x7f89d400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:32.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.052+0000 7f89c7fff640 1 -- 172.21.15.12:0/3239561686 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d4007070 con 0x7f89e0138230 2024-09-11T01:41:32.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.052+0000 7f89c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:32.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.052+0000 7f89c7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:41:32.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.052+0000 7f89c7fff640 10 monclient: dump: 2024-09-11T01:41:32.065 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:32.065 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:32.065 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.065 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.065 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:32.065 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.065 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:32.065 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:32.065 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:32.066 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.052+0000 7f89c7fff640 10 monclient: _finish_auth 0 2024-09-11T01:41:32.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.052+0000 7f89c7fff640 10 monclient: _check_auth_tickets 2024-09-11T01:41:32.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.052+0000 7f89c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:32.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.052+0000 7f89c7fff640 1 -- 172.21.15.12:0/3239561686 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89d4007230 con 0x7f89e0138230 2024-09-11T01:41:32.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.052+0000 7f89e609b640 5 monclient: authenticate success, global_id 94479 2024-09-11T01:41:32.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.052+0000 7f89e609b640 10 monclient: _renew_subs 2024-09-11T01:41:32.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.052+0000 7f89e609b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:32.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.052+0000 7f89e609b640 1 -- 172.21.15.12:0/3239561686 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89e00a3360 con 0x7f89e0138230 2024-09-11T01:41:32.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.052+0000 7f89c7fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:32.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.052+0000 7f89c7fff640 1 -- 172.21.15.12:0/3239561686 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d4007620 con 0x7f89e0138230 2024-09-11T01:41:32.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.052+0000 7f89c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:32.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.052+0000 7f89c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:32.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.052+0000 7f89c7fff640 10 monclient: dump: 2024-09-11T01:41:32.067 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:32.067 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:32.067 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.067 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.067 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:32.067 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.067 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:32.067 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:32.067 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:32.067 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.052+0000 7f89e609b640 10 monclient: _renew_subs 2024-09-11T01:41:32.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.052+0000 7f89e609b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:32.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.052+0000 7f89e609b640 1 -- 172.21.15.12:0/3239561686 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89e0137480 con 0x7f89e0138230 2024-09-11T01:41:32.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.052+0000 7f89ddffb640 4 set_mon_vals no callback set 2024-09-11T01:41:32.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.053+0000 7f89c7fff640 1 -- 172.21.15.12:0/3239561686 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f89d401b400 con 0x7f89e0138230 2024-09-11T01:41:32.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.053+0000 7f89e609b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:32.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.053+0000 7f89e609b640 1 --2- 172.21.15.12:0/3239561686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e00a1c50 0x7f89e00a2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.053+0000 7f89e609b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:32.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.053+0000 7f89e609b640 1 -- 172.21.15.12:0/3239561686 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f89e009d160 con 0x7f89e00a1c50 2024-09-11T01:41:32.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.053+0000 7f89df7fe640 1 --2- 172.21.15.12:0/3239561686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e00a1c50 0x7f89e00a2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:32.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.053+0000 7f89c7fff640 1 --2- 172.21.15.12:0/3239561686 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f89cc03e260 0x7f89cc040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.053+0000 7f89df7fe640 10 monclient: get_auth_request con 0x7f89e00a1c50 auth_method 0 2024-09-11T01:41:32.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.053+0000 7f89df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:32.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.053+0000 7f89df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:32.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.053+0000 7f89df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:32.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.053+0000 7f89c7fff640 1 -- 172.21.15.12:0/3239561686 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(408..408 src has 285..408) ==== 4739+0+0 (secure 0 0 0) 0x7f89d4052960 con 0x7f89e0138230 2024-09-11T01:41:32.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.053+0000 7f89df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:32.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.053+0000 7f89df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:32.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.053+0000 7f89df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:32.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.053+0000 7f89dffff640 1 --2- 172.21.15.12:0/3239561686 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f89cc03e260 0x7f89cc040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:32.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.053+0000 7f89dffff640 10 monclient: get_auth_request con 0x7f89cc03e260 auth_method 0 2024-09-11T01:41:32.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.053+0000 7f89df7fe640 10 monclient(hunting): handle_auth_done global_id 94482 payload 824 2024-09-11T01:41:32.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.053+0000 7f89df7fe640 1 -- 172.21.15.12:0/3239561686 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f89e00008d0 con 0x7f89e00a1c50 2024-09-11T01:41:32.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.055+0000 7f89df7fe640 1 -- 172.21.15.12:0/3239561686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e00a1c50 msgr2=0x7f89e00a2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.055+0000 7f89df7fe640 1 --2- 172.21.15.12:0/3239561686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e00a1c50 0x7f89e00a2030 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f89d0034000 tx=0x7f89d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.055+0000 7f89dffff640 1 --2- 172.21.15.12:0/3239561686 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f89cc03e260 0x7f89cc040720 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f89e00a3e40 tx=0x7f89d8003d10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.056+0000 7f89c7fff640 1 -- 172.21.15.12:0/3239561686 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d0037040 con 0x7f89e00a1c50 2024-09-11T01:41:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.056+0000 7f89c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.056+0000 7f89c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.056+0000 7f89c7fff640 10 monclient: dump: 2024-09-11T01:41:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.072 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:32.073 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.056+0000 7f89c7fff640 1 -- 172.21.15.12:0/3239561686 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f89d003c820 con 0x7f89e00a1c50 2024-09-11T01:41:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.056+0000 7f89c7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.056+0000 7f89c7fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:41:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.056+0000 7f89c7fff640 1 -- 172.21.15.12:0/3239561686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e00a1c50 msgr2=0x7f89e00a2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.056+0000 7f89c7fff640 1 --2- 172.21.15.12:0/3239561686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e00a1c50 0x7f89e00a2030 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f89d0034000 tx=0x7f89d00046c0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.056+0000 7f89c7fff640 1 -- 172.21.15.12:0/3239561686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e00a1c50 msgr2=0x7f89e00a2030 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:41:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.056+0000 7f89c7fff640 1 --2- 172.21.15.12:0/3239561686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e00a1c50 0x7f89e00a2030 secure :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f89d0034000 tx=0x7f89d00046c0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.062+0000 7f89e609b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:41:32.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.062+0000 7f89e609b640 1 --2- 172.21.15.12:0/3239561686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e00a2570 0x7f89e009f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.062+0000 7f89e609b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:32.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.062+0000 7f89e609b640 1 -- 172.21.15.12:0/3239561686 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f89e00a3010 con 0x7f89e00a2570 2024-09-11T01:41:32.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.062+0000 7f89df7fe640 1 --2- 172.21.15.12:0/3239561686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e00a2570 0x7f89e009f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:32.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.062+0000 7f89df7fe640 10 monclient: get_auth_request con 0x7f89e00a2570 auth_method 0 2024-09-11T01:41:32.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.062+0000 7f89df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:32.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.062+0000 7f89df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:32.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.062+0000 7f89df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:32.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.062+0000 7f89df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:32.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.062+0000 7f89df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:32.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.062+0000 7f89df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:32.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.062+0000 7f89df7fe640 10 monclient(hunting): handle_auth_done global_id 94485 payload 824 2024-09-11T01:41:32.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.062+0000 7f89df7fe640 1 -- 172.21.15.12:0/3239561686 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f89e013b610 con 0x7f89e00a2570 2024-09-11T01:41:32.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.065+0000 7f89df7fe640 1 -- 172.21.15.12:0/3239561686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e00a2570 msgr2=0x7f89e009f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:32.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.065+0000 7f89df7fe640 1 --2- 172.21.15.12:0/3239561686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e00a2570 0x7f89e009f590 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f89d0029ff0 tx=0x7f89d0008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:32.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.065+0000 7f89c7fff640 1 -- 172.21.15.12:0/3239561686 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d0050070 con 0x7f89e00a2570 2024-09-11T01:41:32.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.065+0000 7f89c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:32.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.065+0000 7f89c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:32.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.065+0000 7f89c7fff640 10 monclient: dump: 2024-09-11T01:41:32.075 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:32.075 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:32.075 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.075 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.075 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:32.075 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.075 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:32.075 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:32.075 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:32.075 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.065+0000 7f89c7fff640 1 -- 172.21.15.12:0/3239561686 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f89d004c810 con 0x7f89e00a2570 2024-09-11T01:41:32.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.065+0000 7f89c7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:41:32.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.065+0000 7f89c7fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:41:32.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.065+0000 7f89c7fff640 1 -- 172.21.15.12:0/3239561686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e00a2570 msgr2=0x7f89e009f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:32.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.065+0000 7f89c7fff640 1 --2- 172.21.15.12:0/3239561686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e00a2570 0x7f89e009f590 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f89d0029ff0 tx=0x7f89d0008030 comp rx=0 tx=0).stop 2024-09-11T01:41:32.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.065+0000 7f89c7fff640 1 -- 172.21.15.12:0/3239561686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e00a2570 msgr2=0x7f89e009f590 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:41:32.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.065+0000 7f89c7fff640 1 --2- 172.21.15.12:0/3239561686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e00a2570 0x7f89e009f590 secure :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f89d0029ff0 tx=0x7f89d0008030 comp rx=0 tx=0).stop 2024-09-11T01:41:32.076 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:41:32.076 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:41:32.077 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:41:32.077 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:41:32.077 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 114, 2024-09-11T01:41:32.077 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 238161, 2024-09-11T01:41:32.077 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:41:32.077 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:41:32.077 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:41:32.077 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:41:32.077 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:32.077 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 5, 2024-09-11T01:41:32.077 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:41:32.077 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:41:32.077 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:41:32.078 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:41:32.078 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:41:32.078 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:41:32.078 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:41:32.078 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:41:32.078 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:41:32.078 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:41:32.078 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:41:32.078 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:41:32.078 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:41:32.078 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:41:32.078 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:32.078 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:41:32.078 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:41:32.079 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:41:32.079 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:41:32.079 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:41:32.079 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:41:32.079 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:41:32.079 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:41:32.079 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:41:32.079 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:41:32.079 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:41:32.079 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:41:32.079 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:41:32.079 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:32.079 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.080 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:41:32.080 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:41:32.080 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:41:32.080 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:41:32.080 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:41:32.080 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:41:32.080 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:41:32.080 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:41:32.080 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:41:32.080 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:41:32.080 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:41:32.080 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:41:32.080 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:41:32.080 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:41:32.081 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:41:32.081 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:41:32.081 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:41:32.081 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:41:32.081 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:41:32.081 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:41:32.081 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:41:32.081 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:41:32.081 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:41:32.081 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:41:32.081 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:41:32.081 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:41:32.081 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:41:32.082 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:41:32.082 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:32.082 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:41:32.082 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.082 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:41:32.082 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.082 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:41:32.082 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:41:32.082 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:41:32.082 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:41:32.082 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.082 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:41:32.082 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:41:32.082 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:32.083 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.083 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.083 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:41:32.083 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:41:32.083 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:32.083 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:32.083 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:32.083 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.083 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:41:32.083 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:41:32.083 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:41:32.083 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:41:32.083 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:41:32.083 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.084 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.084 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:41:32.084 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:41:32.084 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:41:32.084 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:41:32.084 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.084 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:41:32.084 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:41:32.084 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:32.084 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.084 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.084 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:41:32.084 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:41:32.084 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:32.085 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:32.085 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:32.085 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.085 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:41:32.085 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:41:32.085 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:41:32.085 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:41:32.085 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:41:32.085 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.085 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.085 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:41:32.085 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:41:32.085 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:41:32.086 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:41:32.086 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.086 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:41:32.086 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:41:32.086 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:32.086 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.086 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.086 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:41:32.086 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:41:32.086 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:32.086 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:32.086 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:32.086 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.087 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:41:32.087 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:41:32.087 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:41:32.087 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:41:32.087 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:41:32.087 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:32.087 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:32.087 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.087 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:41:32.087 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:41:32.087 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.087 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:41:32.087 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:41:32.088 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:41:32.088 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:32.088 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:32.088 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:41:32.088 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.088 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:41:32.088 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:41:32.088 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:41:32.088 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:32.088 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:32.088 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:41:32.088 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.088 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:41:32.088 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:41:32.089 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:41:32.089 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:32.089 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:32.089 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.089 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:41:32.089 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:41:32.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.065+0000 7f89e609b640 1 -- 172.21.15.12:0/3239561686 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f89cc03e260 msgr2=0x7f89cc040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:32.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.065+0000 7f89e609b640 1 --2- 172.21.15.12:0/3239561686 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f89cc03e260 0x7f89cc040720 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f89e00a3e40 tx=0x7f89d8003d10 comp rx=0 tx=0).stop 2024-09-11T01:41:32.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.065+0000 7f89e609b640 10 monclient: shutdown 2024-09-11T01:41:32.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.065+0000 7f89e609b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:32.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.065+0000 7f89e609b640 1 -- 172.21.15.12:0/3239561686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e0138230 msgr2=0x7f89e0136f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:32.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.065+0000 7f89e609b640 1 --2- 172.21.15.12:0/3239561686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e0138230 0x7f89e0136f40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f89d400c460 tx=0x7f89d400c920 comp rx=0 tx=0).stop 2024-09-11T01:41:32.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.065+0000 7f89dffff640 1 -- 172.21.15.12:0/3239561686 reap_dead start 2024-09-11T01:41:32.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.065+0000 7f89e609b640 1 -- 172.21.15.12:0/3239561686 shutdown_connections 2024-09-11T01:41:32.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.065+0000 7f89e609b640 1 --2- 172.21.15.12:0/3239561686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f89e0138230 0x7f89e0136f40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.065+0000 7f89e609b640 1 -- 172.21.15.12:0/3239561686 >> 172.21.15.12:0/3239561686 conn(0x7f89e00a7020 msgr2=0x7f89e013c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:32.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.065+0000 7f89e609b640 1 -- 172.21.15.12:0/3239561686 shutdown_connections 2024-09-11T01:41:32.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.065+0000 7f89e609b640 1 -- 172.21.15.12:0/3239561686 wait complete. 2024-09-11T01:41:32.091 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:41:32.091 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:41:32.091 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:41:32.133 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:41:32.192 INFO:tasks.rados.rados.0.smithi071.stdout:1776: finishing write tid 1 to smithi07134777-46 2024-09-11T01:41:32.192 INFO:tasks.rados.rados.0.smithi071.stdout:1776: finishing write tid 2 to smithi07134777-46 2024-09-11T01:41:32.192 INFO:tasks.rados.rados.0.smithi071.stdout:1772: done (5 left) 2024-09-11T01:41:32.192 INFO:tasks.rados.rados.0.smithi071.stdout:1773: done (4 left) 2024-09-11T01:41:32.192 INFO:tasks.rados.rados.0.smithi071.stdout:1774: done (3 left) 2024-09-11T01:41:32.192 INFO:tasks.rados.rados.0.smithi071.stdout:1775: done (2 left) 2024-09-11T01:41:32.192 INFO:tasks.rados.rados.0.smithi071.stdout:1777: done (1 left) 2024-09-11T01:41:32.192 INFO:tasks.rados.rados.0.smithi071.stdout:1778: snap_remove snap 175 2024-09-11T01:41:32.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.192+0000 7f0ca6039640 10 monclient: get_monmap_and_config 2024-09-11T01:41:32.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.192+0000 7f0ca6039640 10 monclient: build_initial_monmap 2024-09-11T01:41:32.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.192+0000 7f0ca6039640 10 monclient: monmap: 2024-09-11T01:41:32.193 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:32.193 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:32.193 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:41:32.192756+0000 2024-09-11T01:41:32.193 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:41:32.192756+0000 2024-09-11T01:41:32.193 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:32.193 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.193 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:41:32.194 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:41:32.194 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:41:32.194 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:41:32.194 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:41:32.194 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.192+0000 7f0ca6039640 1 Processor -- start 2024-09-11T01:41:32.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.192+0000 7f0ca6039640 1 -- start start 2024-09-11T01:41:32.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.192+0000 7f0ca6039640 10 monclient: init 2024-09-11T01:41:32.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0ca6039640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:32.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0ca6039640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-11T01:41:32.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0ca6039640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0ca006f540 0x7f0ca00a5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0ca6039640 10 monclient(hunting): picked mon.noname-a con 0x7f0ca006f540 addr v2:172.21.15.12:3300/0 2024-09-11T01:41:32.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0ca6039640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ca009cff0 addr v1:172.21.15.12:6789/0 2024-09-11T01:41:32.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0ca6039640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ca013d230 addr v1:172.21.15.71:6789/0 2024-09-11T01:41:32.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0ca6039640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ca00a6ea0 con 0x7f0ca009cff0 2024-09-11T01:41:32.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0ca6039640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ca009d4d0 con 0x7f0ca013d230 2024-09-11T01:41:32.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0ca6039640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:32.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0ca6039640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f0ca006f920 con 0x7f0ca006f540 2024-09-11T01:41:32.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0ca6039640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:32.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0ca6039640 10 monclient(hunting): authenticate will time out at 1498.635355s 2024-09-11T01:41:32.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0c9f7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0ca006f540 0x7f0ca00a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:32.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0c9f7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0ca006f540 0x7f0ca00a5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58706/0 (socket says 172.21.15.12:58706) 2024-09-11T01:41:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0c9effd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0ca009cff0 0x7f0ca00a5560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42306/0 (socket says 172.21.15.12:42306) 2024-09-11T01:41:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0c9f7fe640 1 -- 172.21.15.12:0/4248139376 learned_addr learned my addr 172.21.15.12:0/4248139376 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:41:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0c9f7fe640 10 monclient(hunting): get_auth_request con 0x7f0ca006f540 auth_method 0 2024-09-11T01:41:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0c9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0c9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0c9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0c9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0c9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0c9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0c9f7fe640 10 monclient(hunting): handle_auth_done global_id 94488 payload 274 2024-09-11T01:41:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0c9f7fe640 1 -- 172.21.15.12:0/4248139376 >> v1:172.21.15.71:6789/0 conn(0x7f0ca013d230 legacy=0x7f0ca013f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0c9f7fe640 1 -- 172.21.15.12:0/4248139376 >> v1:172.21.15.12:6789/0 conn(0x7f0ca009cff0 legacy=0x7f0ca00a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:32.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0c9f7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:41:32.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0c9f7fe640 1 monclient: found mon.noname-a 2024-09-11T01:41:32.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0c9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:32.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0c9f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:41:32.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.193+0000 7f0c9f7fe640 1 -- 172.21.15.12:0/4248139376 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ca009c330 con 0x7f0ca006f540 2024-09-11T01:41:32.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.196+0000 7f0c9f7fe640 1 -- 172.21.15.12:0/4248139376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ca006f540 msgr2=0x7f0ca00a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:32.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.196+0000 7f0c9f7fe640 1 --2- 172.21.15.12:0/4248139376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ca006f540 0x7f0ca00a5020 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0c90009870 tx=0x7f0c9002a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eb64f776b5de2ab server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:32.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.196+0000 7f0c9e7fc640 1 -- 172.21.15.12:0/4248139376 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c90037070 con 0x7f0ca006f540 2024-09-11T01:41:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.196+0000 7f0c9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.196+0000 7f0c9e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:41:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.196+0000 7f0c9e7fc640 10 monclient: dump: 2024-09-11T01:41:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:32.204 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:32.205 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.196+0000 7f0c9e7fc640 10 monclient: _finish_auth 0 2024-09-11T01:41:32.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.196+0000 7f0c9e7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:41:32.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.196+0000 7f0c9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:32.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.196+0000 7f0c9e7fc640 1 -- 172.21.15.12:0/4248139376 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c90032640 con 0x7f0ca006f540 2024-09-11T01:41:32.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.196+0000 7f0ca6039640 5 monclient: authenticate success, global_id 94488 2024-09-11T01:41:32.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.196+0000 7f0ca6039640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:41:32.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.196+0000 7f0c9e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:32.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.196+0000 7f0ca6039640 10 monclient: get_monmap_and_config success 2024-09-11T01:41:32.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.196+0000 7f0ca6039640 4 set_mon_vals no callback set 2024-09-11T01:41:32.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.196+0000 7f0c9e7fc640 1 -- 172.21.15.12:0/4248139376 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c90032a10 con 0x7f0ca006f540 2024-09-11T01:41:32.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.196+0000 7f0c9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:32.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.196+0000 7f0c9e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.196+0000 7f0c9e7fc640 10 monclient: dump: 2024-09-11T01:41:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:32.206 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.197+0000 7f0ca6039640 10 monclient: shutdown 2024-09-11T01:41:32.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.197+0000 7f0ca6039640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.197+0000 7f0ca6039640 1 -- 172.21.15.12:0/4248139376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ca006f540 msgr2=0x7f0ca00a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.197+0000 7f0ca6039640 1 --2- 172.21.15.12:0/4248139376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ca006f540 0x7f0ca00a5020 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0c90009870 tx=0x7f0c9002a0a0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.197+0000 7f0ca6039640 1 -- 172.21.15.12:0/4248139376 shutdown_connections 2024-09-11T01:41:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.197+0000 7f0ca6039640 1 --2- 172.21.15.12:0/4248139376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ca006f540 0x7f0ca00a5020 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.197+0000 7f0ca6039640 1 -- 172.21.15.12:0/4248139376 >> 172.21.15.12:0/4248139376 conn(0x7f0ca00a7300 msgr2=0x7f0ca00a7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.197+0000 7f0ca6039640 1 -- 172.21.15.12:0/4248139376 shutdown_connections 2024-09-11T01:41:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.197+0000 7f0ca6039640 1 -- 172.21.15.12:0/4248139376 wait complete. 2024-09-11T01:41:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.198+0000 7f0ca6039640 10 monclient: build_initial_monmap 2024-09-11T01:41:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.198+0000 7f0ca6039640 10 monclient: monmap: 2024-09-11T01:41:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:41:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:41:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:41:32.207 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:41:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:41:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:41:32.208 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.198+0000 7f0ca6039640 1 Processor -- start 2024-09-11T01:41:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.198+0000 7f0ca6039640 1 -- start start 2024-09-11T01:41:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.198+0000 7f0ca6039640 10 monclient: init 2024-09-11T01:41:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.198+0000 7f0ca6039640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.198+0000 7f0ca6039640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:41:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.198+0000 7f0ca6039640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ca006f540 0x7f0ca0137a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.198+0000 7f0ca6039640 10 monclient(hunting): picked mon.noname-c con 0x7f0ca006f540 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:32.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0ca6039640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ca009cff0 0x7f0ca0137f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0ca6039640 10 monclient(hunting): picked mon.noname-a con 0x7f0ca009cff0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:41:32.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0ca6039640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ca013d230 0x7f0ca009e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0ca6039640 10 monclient(hunting): picked mon.noname-b con 0x7f0ca013d230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:41:32.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0ca6039640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:32.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0ca6039640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca0140d40 con 0x7f0ca009cff0 2024-09-11T01:41:32.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0ca6039640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:32.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0ca6039640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca00a5e80 con 0x7f0ca013d230 2024-09-11T01:41:32.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0ca6039640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:32.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0ca6039640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f0ca0140ec0 con 0x7f0ca006f540 2024-09-11T01:41:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0ca6039640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0ca6039640 10 monclient(hunting): authenticate will time out at 1498.641092s 2024-09-11T01:41:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9effd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ca009cff0 0x7f0ca0137f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9f7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ca006f540 0x7f0ca0137a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9effd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ca009cff0 0x7f0ca0137f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58720/0 (socket says 172.21.15.12:58720) 2024-09-11T01:41:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9effd640 1 -- 172.21.15.12:0/3431265638 learned_addr learned my addr 172.21.15.12:0/3431265638 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:41:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9effd640 10 monclient(hunting): get_auth_request con 0x7f0ca009cff0 auth_method 0 2024-09-11T01:41:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9f7fe640 1 --2- 172.21.15.12:0/3431265638 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ca006f540 0x7f0ca0137a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:56144/0 (socket says 172.21.15.12:56144) 2024-09-11T01:41:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9f7fe640 10 monclient(hunting): get_auth_request con 0x7f0ca006f540 auth_method 0 2024-09-11T01:41:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9effd640 10 monclient(hunting): handle_auth_done global_id 94491 payload 824 2024-09-11T01:41:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9effd640 1 -- 172.21.15.12:0/3431265638 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ca006f540 msgr2=0x7f0ca0137a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9ffff640 1 --2- 172.21.15.12:0/3431265638 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ca013d230 0x7f0ca009e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9effd640 1 --2- 172.21.15.12:0/3431265638 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ca006f540 0x7f0ca0137a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9effd640 1 -- 172.21.15.12:0/3431265638 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ca013d230 msgr2=0x7f0ca009e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9effd640 1 --2- 172.21.15.12:0/3431265638 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ca013d230 0x7f0ca009e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9effd640 10 monclient: _finish_hunting 0 2024-09-11T01:41:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9effd640 1 monclient: found mon.noname-a 2024-09-11T01:41:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:41:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9effd640 1 -- 172.21.15.12:0/3431265638 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ca01212f0 con 0x7f0ca009cff0 2024-09-11T01:41:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9ffff640 10 monclient: get_auth_request con 0x7f0ca013d230 auth_method 0 2024-09-11T01:41:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.199+0000 7f0c9ffff640 1 --2- 172.21.15.12:0/3431265638 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ca013d230 0x7f0ca009e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:41:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.201+0000 7f0c9effd640 1 -- 172.21.15.12:0/3431265638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ca009cff0 msgr2=0x7f0ca0137f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.201+0000 7f0c9effd640 1 --2- 172.21.15.12:0/3431265638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ca009cff0 0x7f0ca0137f50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0c9400c450 tx=0x7f0c9400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.202+0000 7f0c87fff640 1 -- 172.21.15.12:0/3431265638 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c94017070 con 0x7f0ca009cff0 2024-09-11T01:41:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.202+0000 7f0c87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.202+0000 7f0c87fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:41:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.202+0000 7f0c87fff640 10 monclient: dump: 2024-09-11T01:41:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:32.213 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:32.213 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.202+0000 7f0c87fff640 10 monclient: _finish_auth 0 2024-09-11T01:41:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.202+0000 7f0c87fff640 10 monclient: _check_auth_tickets 2024-09-11T01:41:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.202+0000 7f0c87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.202+0000 7f0c87fff640 1 -- 172.21.15.12:0/3431265638 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c94005810 con 0x7f0ca009cff0 2024-09-11T01:41:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.202+0000 7f0ca6039640 5 monclient: authenticate success, global_id 94491 2024-09-11T01:41:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.202+0000 7f0ca6039640 10 monclient: _renew_subs 2024-09-11T01:41:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.202+0000 7f0ca6039640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.202+0000 7f0ca6039640 1 -- 172.21.15.12:0/3431265638 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ca01215d0 con 0x7f0ca009cff0 2024-09-11T01:41:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.202+0000 7f0c87fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.202+0000 7f0c87fff640 1 -- 172.21.15.12:0/3431265638 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c94005c00 con 0x7f0ca009cff0 2024-09-11T01:41:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.202+0000 7f0c87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.202+0000 7f0c87fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.202+0000 7f0c87fff640 10 monclient: dump: 2024-09-11T01:41:32.214 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:32.215 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.202+0000 7f0ca6039640 10 monclient: _renew_subs 2024-09-11T01:41:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.202+0000 7f0ca6039640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.202+0000 7f0c9dffb640 4 set_mon_vals no callback set 2024-09-11T01:41:32.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.202+0000 7f0ca6039640 1 -- 172.21.15.12:0/3431265638 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ca0138490 con 0x7f0ca009cff0 2024-09-11T01:41:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.203+0000 7f0c87fff640 1 -- 172.21.15.12:0/3431265638 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f0c9401b3b0 con 0x7f0ca009cff0 2024-09-11T01:41:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.203+0000 7f0ca6039640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.203+0000 7f0ca6039640 1 --2- 172.21.15.12:0/3431265638 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ca009e920 0x7f0ca009ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.203+0000 7f0ca6039640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.203+0000 7f0ca6039640 1 -- 172.21.15.12:0/3431265638 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f0ca00a1b90 con 0x7f0ca009e920 2024-09-11T01:41:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.203+0000 7f0c9f7fe640 1 --2- 172.21.15.12:0/3431265638 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ca009e920 0x7f0ca009ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.203+0000 7f0c9f7fe640 10 monclient: get_auth_request con 0x7f0ca009e920 auth_method 0 2024-09-11T01:41:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.203+0000 7f0c9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.203+0000 7f0c9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.203+0000 7f0c9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.203+0000 7f0c9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.203+0000 7f0c9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.203+0000 7f0c9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.203+0000 7f0c87fff640 1 --2- 172.21.15.12:0/3431265638 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0c8c03e240 0x7f0c8c040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.203+0000 7f0c87fff640 1 -- 172.21.15.12:0/3431265638 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(408..408 src has 285..408) ==== 4739+0+0 (secure 0 0 0) 0x7f0c94052520 con 0x7f0ca009cff0 2024-09-11T01:41:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.203+0000 7f0c9f7fe640 10 monclient(hunting): handle_auth_done global_id 114320 payload 824 2024-09-11T01:41:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.203+0000 7f0c9f7fe640 1 -- 172.21.15.12:0/3431265638 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0ca0138660 con 0x7f0ca009e920 2024-09-11T01:41:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.203+0000 7f0c9ffff640 1 --2- 172.21.15.12:0/3431265638 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0c8c03e240 0x7f0c8c040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.203+0000 7f0c9ffff640 10 monclient: get_auth_request con 0x7f0c8c03e240 auth_method 0 2024-09-11T01:41:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.205+0000 7f0c9f7fe640 1 -- 172.21.15.12:0/3431265638 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ca009e920 msgr2=0x7f0ca009ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.205+0000 7f0c9f7fe640 1 --2- 172.21.15.12:0/3431265638 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ca009e920 0x7f0ca009ed00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0c90034000 tx=0x7f0c900046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.205+0000 7f0c87fff640 1 -- 172.21.15.12:0/3431265638 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c90037040 con 0x7f0ca009e920 2024-09-11T01:41:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.205+0000 7f0c87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.205+0000 7f0c87fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:41:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.205+0000 7f0c87fff640 10 monclient: dump: 2024-09-11T01:41:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:32.222 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.206+0000 7f0c87fff640 1 -- 172.21.15.12:0/3431265638 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0c9003c820 con 0x7f0ca009e920 2024-09-11T01:41:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.206+0000 7f0c87fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.206+0000 7f0c9ffff640 1 --2- 172.21.15.12:0/3431265638 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0c8c03e240 0x7f0c8c040700 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00a0790 tx=0x7f0c98009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.206+0000 7f0c87fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:41:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.206+0000 7f0c87fff640 1 -- 172.21.15.12:0/3431265638 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ca009e920 msgr2=0x7f0ca009ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.206+0000 7f0c87fff640 1 --2- 172.21.15.12:0/3431265638 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ca009e920 0x7f0ca009ed00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0c90034000 tx=0x7f0c900046c0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.206+0000 7f0c87fff640 1 -- 172.21.15.12:0/3431265638 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ca009e920 msgr2=0x7f0ca009ed00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:41:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.206+0000 7f0c87fff640 1 --2- 172.21.15.12:0/3431265638 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ca009e920 0x7f0ca009ed00 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0c90034000 tx=0x7f0c900046c0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.212+0000 7f0ca6039640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:41:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.212+0000 7f0ca6039640 1 --2- 172.21.15.12:0/3431265638 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ca009f240 0x7f0ca0139210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.212+0000 7f0ca6039640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.212+0000 7f0ca6039640 1 -- 172.21.15.12:0/3431265638 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f0ca006a060 con 0x7f0ca009f240 2024-09-11T01:41:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.212+0000 7f0c9f7fe640 1 --2- 172.21.15.12:0/3431265638 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ca009f240 0x7f0ca0139210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.212+0000 7f0c9f7fe640 10 monclient: get_auth_request con 0x7f0ca009f240 auth_method 0 2024-09-11T01:41:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.212+0000 7f0c9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.212+0000 7f0c9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.212+0000 7f0c9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.212+0000 7f0c9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.212+0000 7f0c9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.212+0000 7f0c9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.212+0000 7f0c9f7fe640 10 monclient(hunting): handle_auth_done global_id 114323 payload 824 2024-09-11T01:41:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.212+0000 7f0c9f7fe640 1 -- 172.21.15.12:0/3431265638 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0ca00a5020 con 0x7f0ca009f240 2024-09-11T01:41:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.215+0000 7f0c9f7fe640 1 -- 172.21.15.12:0/3431265638 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ca009f240 msgr2=0x7f0ca0139210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.215+0000 7f0c9f7fe640 1 --2- 172.21.15.12:0/3431265638 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ca009f240 0x7f0ca0139210 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0ca009da30 tx=0x7f0c90008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.215+0000 7f0c87fff640 1 -- 172.21.15.12:0/3431265638 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c90037040 con 0x7f0ca009f240 2024-09-11T01:41:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.215+0000 7f0c87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.215+0000 7f0c87fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:41:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.215+0000 7f0c87fff640 10 monclient: dump: 2024-09-11T01:41:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:32.225 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.215+0000 7f0c87fff640 1 -- 172.21.15.12:0/3431265638 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f0c900044d0 con 0x7f0ca009f240 2024-09-11T01:41:32.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.215+0000 7f0c87fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:41:32.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.215+0000 7f0c87fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:41:32.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.215+0000 7f0c87fff640 1 -- 172.21.15.12:0/3431265638 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ca009f240 msgr2=0x7f0ca0139210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:32.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.215+0000 7f0c87fff640 1 --2- 172.21.15.12:0/3431265638 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ca009f240 0x7f0ca0139210 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0ca009da30 tx=0x7f0c90008030 comp rx=0 tx=0).stop 2024-09-11T01:41:32.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.215+0000 7f0c87fff640 1 -- 172.21.15.12:0/3431265638 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ca009f240 msgr2=0x7f0ca0139210 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:41:32.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.215+0000 7f0c87fff640 1 --2- 172.21.15.12:0/3431265638 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ca009f240 0x7f0ca0139210 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0ca009da30 tx=0x7f0c90008030 comp rx=0 tx=0).stop 2024-09-11T01:41:32.225 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:41:32.225 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:41:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:41:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:41:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 114, 2024-09-11T01:41:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 138011, 2024-09-11T01:41:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:41:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:41:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:41:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:41:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 5, 2024-09-11T01:41:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:41:32.227 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:41:32.227 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:41:32.227 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:41:32.227 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:41:32.227 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:41:32.227 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:41:32.227 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:41:32.227 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:41:32.227 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:41:32.227 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:41:32.227 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:41:32.228 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:41:32.228 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:41:32.228 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:32.228 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:41:32.228 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:41:32.228 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:41:32.228 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:41:32.228 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:41:32.228 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:41:32.228 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:41:32.228 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:41:32.228 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:41:32.228 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:41:32.228 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:41:32.229 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:41:32.229 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:41:32.229 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:32.229 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.229 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:41:32.229 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:41:32.229 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:41:32.229 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:41:32.229 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:41:32.229 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:41:32.229 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:41:32.229 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:41:32.229 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:41:32.230 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:41:32.230 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:41:32.230 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:41:32.230 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:41:32.230 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:41:32.230 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:41:32.230 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:41:32.230 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:41:32.230 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:41:32.230 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:41:32.230 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:41:32.230 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:41:32.230 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:41:32.230 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:41:32.231 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:41:32.231 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:41:32.231 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:41:32.231 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:41:32.231 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:41:32.231 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:32.231 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:41:32.231 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.231 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:41:32.231 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.231 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:41:32.231 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:41:32.231 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:41:32.232 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:41:32.232 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.232 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:41:32.232 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:41:32.232 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:32.232 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.232 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.232 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:41:32.232 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:41:32.232 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:32.232 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:32.232 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:32.232 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.232 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:41:32.233 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:41:32.233 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:41:32.233 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:41:32.233 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:41:32.233 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.233 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.233 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:41:32.233 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:41:32.233 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:41:32.233 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:41:32.233 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.234 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:41:32.234 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:41:32.234 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:32.234 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.234 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.234 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:41:32.234 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:41:32.234 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:32.234 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:32.234 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:32.234 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.234 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:41:32.234 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:41:32.234 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:41:32.235 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:41:32.235 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:41:32.235 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.235 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.235 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:41:32.235 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:41:32.235 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:41:32.235 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:41:32.235 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.235 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:41:32.235 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:41:32.235 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:32.235 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.235 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.235 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:41:32.236 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:41:32.236 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:32.236 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:32.236 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:32.236 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.236 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:41:32.236 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:41:32.236 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:41:32.236 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:41:32.236 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:41:32.237 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:32.237 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:32.237 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.237 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:41:32.237 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:41:32.237 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.237 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:41:32.237 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:41:32.237 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:41:32.237 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:32.237 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:32.237 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:41:32.237 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.238 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:41:32.238 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:41:32.238 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:41:32.238 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:32.238 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:32.238 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:41:32.238 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.238 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:41:32.238 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:41:32.238 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:41:32.238 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:32.239 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:32.239 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:41:32.239 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.239 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:41:32.239 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:41:32.239 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:41:32.239 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:32.239 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:32.239 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.239 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:41:32.239 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:41:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.215+0000 7f0ca6039640 1 -- 172.21.15.12:0/3431265638 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0c8c03e240 msgr2=0x7f0c8c040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.215+0000 7f0ca6039640 1 --2- 172.21.15.12:0/3431265638 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0c8c03e240 0x7f0c8c040700 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00a0790 tx=0x7f0c98009040 comp rx=0 tx=0).stop 2024-09-11T01:41:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.215+0000 7f0ca6039640 10 monclient: shutdown 2024-09-11T01:41:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.215+0000 7f0ca6039640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.215+0000 7f0ca6039640 1 -- 172.21.15.12:0/3431265638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ca009cff0 msgr2=0x7f0ca0137f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.215+0000 7f0ca6039640 1 --2- 172.21.15.12:0/3431265638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ca009cff0 0x7f0ca0137f50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0c9400c450 tx=0x7f0c9400c910 comp rx=0 tx=0).stop 2024-09-11T01:41:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.215+0000 7f0c9ffff640 1 -- 172.21.15.12:0/3431265638 reap_dead start 2024-09-11T01:41:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.215+0000 7f0c9ffff640 1 -- 172.21.15.12:0/3431265638 reap_dead start 2024-09-11T01:41:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.215+0000 7f0ca6039640 1 -- 172.21.15.12:0/3431265638 shutdown_connections 2024-09-11T01:41:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.215+0000 7f0ca6039640 1 -- 172.21.15.12:0/3431265638 >> 172.21.15.12:0/3431265638 conn(0x7f0ca00a7300 msgr2=0x7f0ca00a3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.215+0000 7f0ca6039640 1 -- 172.21.15.12:0/3431265638 shutdown_connections 2024-09-11T01:41:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.215+0000 7f0ca6039640 1 -- 172.21.15.12:0/3431265638 wait complete. 2024-09-11T01:41:32.242 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:41:32.242 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:41:32.242 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:41:32.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.342+0000 7f5ae64cd640 10 monclient: get_monmap_and_config 2024-09-11T01:41:32.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.342+0000 7f5ae64cd640 10 monclient: build_initial_monmap 2024-09-11T01:41:32.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.342+0000 7f5ae64cd640 10 monclient: monmap: 2024-09-11T01:41:32.344 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:32.345 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:32.345 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:41:32.343711+0000 2024-09-11T01:41:32.345 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:41:32.343711+0000 2024-09-11T01:41:32.345 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:32.345 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.345 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:41:32.345 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:41:32.345 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:41:32.345 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:41:32.345 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:41:32.345 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.343+0000 7f5ae64cd640 1 Processor -- start 2024-09-11T01:41:32.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.343+0000 7f5ae64cd640 1 -- start start 2024-09-11T01:41:32.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.343+0000 7f5ae64cd640 10 monclient: init 2024-09-11T01:41:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5ae64cd640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5ae64cd640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-11T01:41:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5ae64cd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ae0137e50 0x7f5ae009d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5ae64cd640 10 monclient(hunting): picked mon.noname-b con 0x7f5ae0137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5ae64cd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5ae0138230 0x7f5ae0138630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5ae64cd640 10 monclient(hunting): picked mon.noname-a con 0x7f5ae0138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:41:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5ae64cd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5ae0134cb0 addr v1:172.21.15.71:6789/0 2024-09-11T01:41:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5ae64cd640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ae00a62d0 con 0x7f5ae0134cb0 2024-09-11T01:41:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5ae64cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5ae64cd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f5ae00a65b0 con 0x7f5ae0138230 2024-09-11T01:41:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5ae64cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5ae64cd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5ae009bb90 con 0x7f5ae0137e50 2024-09-11T01:41:32.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5ae64cd640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5ae64cd640 10 monclient(hunting): authenticate will time out at 1498.786462s 2024-09-11T01:41:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5ad7fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5ae0138230 0x7f5ae0138630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5adffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ae0137e50 0x7f5ae009d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5ad7fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5ae0138230 0x7f5ae0138630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58728/0 (socket says 172.21.15.12:58728) 2024-09-11T01:41:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5ad7fff640 1 -- 172.21.15.12:0/2328263794 learned_addr learned my addr 172.21.15.12:0/2328263794 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:41:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5ad7fff640 10 monclient(hunting): get_auth_request con 0x7f5ae0138230 auth_method 0 2024-09-11T01:41:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5ad7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5ad7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5ad7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5adffff640 10 monclient(hunting): get_auth_request con 0x7f5ae0137e50 auth_method 0 2024-09-11T01:41:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5adffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5adffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:32.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5adffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:32.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5ad7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:32.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5ad7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:32.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5ad7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:32.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5adffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:32.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5adffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:32.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5adffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:32.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5adffff640 10 monclient(hunting): handle_auth_done global_id 114326 payload 274 2024-09-11T01:41:32.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5adffff640 1 -- 172.21.15.12:0/2328263794 >> v2:172.21.15.12:3300/0 conn(0x7f5ae0138230 msgr2=0x7f5ae0138630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:32.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5adffff640 1 --2- 172.21.15.12:0/2328263794 >> v2:172.21.15.12:3300/0 conn(0x7f5ae0138230 0x7f5ae0138630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5adffff640 1 -- 172.21.15.12:0/2328263794 >> v1:172.21.15.71:6789/0 conn(0x7f5ae0134cb0 legacy=0x7f5ae0135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:32.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5adffff640 10 monclient: _finish_hunting 0 2024-09-11T01:41:32.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5adffff640 1 monclient: found mon.noname-b 2024-09-11T01:41:32.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5adffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:32.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5adffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:41:32.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5adffff640 1 -- 172.21.15.12:0/2328263794 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ae009c4f0 con 0x7f5ae0137e50 2024-09-11T01:41:32.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.344+0000 7f5ad7fff640 1 --2- 172.21.15.12:0/2328263794 >> v2:172.21.15.12:3300/0 conn(0x7f5ae0138230 0x7f5ae0138630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:41:32.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.347+0000 7f5adffff640 1 -- 172.21.15.12:0/2328263794 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ae0137e50 msgr2=0x7f5ae009d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.347+0000 7f5adffff640 1 --2- 172.21.15.12:0/2328263794 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ae0137e50 0x7f5ae009d160 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5ad00097a0 tx=0x7f5ad0029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bcfae4930391076 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.347+0000 7f5adf7fe640 1 -- 172.21.15.12:0/2328263794 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0037070 con 0x7f5ae0137e50 2024-09-11T01:41:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.347+0000 7f5adf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.347+0000 7f5adf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:41:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.347+0000 7f5adf7fe640 10 monclient: dump: 2024-09-11T01:41:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:32.357 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:32.358 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.347+0000 7f5adf7fe640 10 monclient: _finish_auth 0 2024-09-11T01:41:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.347+0000 7f5adf7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:41:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.347+0000 7f5adf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.347+0000 7f5adf7fe640 1 -- 172.21.15.12:0/2328263794 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ad0032440 con 0x7f5ae0137e50 2024-09-11T01:41:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.347+0000 7f5ae64cd640 5 monclient: authenticate success, global_id 114326 2024-09-11T01:41:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.347+0000 7f5ae64cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:41:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.347+0000 7f5adf7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.347+0000 7f5adf7fe640 1 -- 172.21.15.12:0/2328263794 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0032810 con 0x7f5ae0137e50 2024-09-11T01:41:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.347+0000 7f5adf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.347+0000 7f5adf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:41:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.347+0000 7f5adf7fe640 10 monclient: dump: 2024-09-11T01:41:32.358 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:32.359 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:32.359 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.359 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.359 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:32.359 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.359 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:32.359 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:32.359 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:32.359 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.347+0000 7f5ae64cd640 10 monclient: get_monmap_and_config success 2024-09-11T01:41:32.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.347+0000 7f5ae64cd640 4 set_mon_vals no callback set 2024-09-11T01:41:32.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.348+0000 7f5ae64cd640 10 monclient: shutdown 2024-09-11T01:41:32.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.348+0000 7f5ae64cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:32.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.348+0000 7f5ae64cd640 1 -- 172.21.15.12:0/2328263794 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ae0137e50 msgr2=0x7f5ae009d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.348+0000 7f5ae64cd640 1 --2- 172.21.15.12:0/2328263794 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ae0137e50 0x7f5ae009d160 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5ad00097a0 tx=0x7f5ad0029fd0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.348+0000 7f5ae64cd640 1 -- 172.21.15.12:0/2328263794 shutdown_connections 2024-09-11T01:41:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.348+0000 7f5ae64cd640 1 --2- 172.21.15.12:0/2328263794 >> v2:172.21.15.12:3300/0 conn(0x7f5ae0138230 0x7f5ae0138630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.348+0000 7f5ae64cd640 1 --2- 172.21.15.12:0/2328263794 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ae0137e50 0x7f5ae009d160 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.348+0000 7f5ae64cd640 1 -- 172.21.15.12:0/2328263794 >> 172.21.15.12:0/2328263794 conn(0x7f5ae00a7020 msgr2=0x7f5ae00a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.348+0000 7f5ae64cd640 1 -- 172.21.15.12:0/2328263794 shutdown_connections 2024-09-11T01:41:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.348+0000 7f5ae64cd640 1 -- 172.21.15.12:0/2328263794 wait complete. 2024-09-11T01:41:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.349+0000 7f5ae64cd640 10 monclient: build_initial_monmap 2024-09-11T01:41:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.349+0000 7f5ae64cd640 10 monclient: monmap: 2024-09-11T01:41:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:41:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:41:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:41:32.360 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:41:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:41:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:41:32.361 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.349+0000 7f5ae64cd640 1 Processor -- start 2024-09-11T01:41:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.349+0000 7f5ae64cd640 1 -- start start 2024-09-11T01:41:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.349+0000 7f5ae64cd640 10 monclient: init 2024-09-11T01:41:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ae64cd640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ae64cd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:41:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ae64cd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ae0134cb0 0x7f5ae0133d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ae64cd640 10 monclient(hunting): picked mon.noname-c con 0x7f5ae0134cb0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ae64cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ae0137e50 0x7f5ae01342d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ae64cd640 10 monclient(hunting): picked mon.noname-a con 0x7f5ae0137e50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:41:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ae64cd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ae0138230 0x7f5ae01219c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ae64cd640 10 monclient(hunting): picked mon.noname-b con 0x7f5ae0138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:41:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ae64cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ae64cd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae006f710 con 0x7f5ae0137e50 2024-09-11T01:41:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ae64cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ae64cd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae0138b70 con 0x7f5ae0138230 2024-09-11T01:41:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ae64cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ae64cd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5ae006f890 con 0x7f5ae0134cb0 2024-09-11T01:41:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ae64cd640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5adffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ae0134cb0 0x7f5ae0133d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ae64cd640 10 monclient(hunting): authenticate will time out at 1498.792200s 2024-09-11T01:41:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ad7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ae0137e50 0x7f5ae01342d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5adffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ae0134cb0 0x7f5ae0133d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:56164/0 (socket says 172.21.15.12:56164) 2024-09-11T01:41:32.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ad7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ae0137e50 0x7f5ae01342d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58740/0 (socket says 172.21.15.12:58740) 2024-09-11T01:41:32.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5adffff640 1 -- 172.21.15.12:0/4050842724 learned_addr learned my addr 172.21.15.12:0/4050842724 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:41:32.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5adffff640 10 monclient(hunting): get_auth_request con 0x7f5ae0134cb0 auth_method 0 2024-09-11T01:41:32.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5adffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:32.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5adffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:32.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5adffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:32.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ad7fff640 10 monclient(hunting): get_auth_request con 0x7f5ae0137e50 auth_method 0 2024-09-11T01:41:32.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ad7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:32.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ad7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:32.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ad7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:32.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ad7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:32.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ad7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:32.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ad7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:32.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5adffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:32.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5adffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:32.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5adffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ad7fff640 10 monclient(hunting): handle_auth_done global_id 94497 payload 824 2024-09-11T01:41:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ad7fff640 1 -- 172.21.15.12:0/4050842724 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ae0134cb0 msgr2=0x7f5ae0133d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ad7fff640 1 --2- 172.21.15.12:0/4050842724 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ae0134cb0 0x7f5ae0133d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ae4a6a640 1 --2- 172.21.15.12:0/4050842724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ae0138230 0x7f5ae01219c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ad7fff640 1 -- 172.21.15.12:0/4050842724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ae0138230 msgr2=0x7f5ae01219c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ad7fff640 1 --2- 172.21.15.12:0/4050842724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ae0138230 0x7f5ae01219c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ad7fff640 10 monclient: _finish_hunting 0 2024-09-11T01:41:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ad7fff640 1 monclient: found mon.noname-a 2024-09-11T01:41:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ad7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ad7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:41:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5ad7fff640 1 -- 172.21.15.12:0/4050842724 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ae01ce1f0 con 0x7f5ae0137e50 2024-09-11T01:41:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.350+0000 7f5adffff640 1 --2- 172.21.15.12:0/4050842724 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5ae0134cb0 0x7f5ae0133d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:41:32.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.352+0000 7f5ad7fff640 1 -- 172.21.15.12:0/4050842724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ae0137e50 msgr2=0x7f5ae01342d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:32.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.352+0000 7f5ad7fff640 1 --2- 172.21.15.12:0/4050842724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ae0137e50 0x7f5ae01342d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800e900 tx=0x7f5ac800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:32.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.352+0000 7f5add7fa640 1 -- 172.21.15.12:0/4050842724 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac800cc10 con 0x7f5ae0137e50 2024-09-11T01:41:32.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.352+0000 7f5add7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:32.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.352+0000 7f5add7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:41:32.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.352+0000 7f5add7fa640 10 monclient: dump: 2024-09-11T01:41:32.367 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:32.368 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:32.368 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.368 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.368 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:32.368 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.368 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:32.369 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:32.369 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:32.369 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.352+0000 7f5add7fa640 10 monclient: _finish_auth 0 2024-09-11T01:41:32.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.352+0000 7f5add7fa640 10 monclient: _check_auth_tickets 2024-09-11T01:41:32.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.352+0000 7f5add7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:32.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.352+0000 7f5add7fa640 1 -- 172.21.15.12:0/4050842724 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ac800cdb0 con 0x7f5ae0137e50 2024-09-11T01:41:32.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.352+0000 7f5ae64cd640 5 monclient: authenticate success, global_id 94497 2024-09-11T01:41:32.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.352+0000 7f5ae64cd640 10 monclient: _renew_subs 2024-09-11T01:41:32.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.352+0000 7f5ae64cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:32.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.352+0000 7f5ae64cd640 1 -- 172.21.15.12:0/4050842724 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ae0125790 con 0x7f5ae0137e50 2024-09-11T01:41:32.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.352+0000 7f5add7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:32.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.352+0000 7f5add7fa640 1 -- 172.21.15.12:0/4050842724 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac8004250 con 0x7f5ae0137e50 2024-09-11T01:41:32.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.352+0000 7f5add7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:32.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.352+0000 7f5add7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:32.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.352+0000 7f5add7fa640 10 monclient: dump: 2024-09-11T01:41:32.371 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:32.371 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:32.372 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.372 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.372 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:32.372 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.372 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:32.372 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:32.373 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:32.373 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.352+0000 7f5adeffd640 4 set_mon_vals no callback set 2024-09-11T01:41:32.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.353+0000 7f5ae64cd640 10 monclient: _renew_subs 2024-09-11T01:41:32.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.353+0000 7f5ae64cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:32.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.353+0000 7f5ae64cd640 1 -- 172.21.15.12:0/4050842724 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ae0134810 con 0x7f5ae0137e50 2024-09-11T01:41:32.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.353+0000 7f5add7fa640 1 -- 172.21.15.12:0/4050842724 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f5ac8023060 con 0x7f5ae0137e50 2024-09-11T01:41:32.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.353+0000 7f5ae64cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:32.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.353+0000 7f5ae64cd640 1 --2- 172.21.15.12:0/4050842724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ae01cd540 0x7f5ae0121f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.353+0000 7f5ae64cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:32.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.353+0000 7f5ae64cd640 1 -- 172.21.15.12:0/4050842724 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae006a060 con 0x7f5ae01cd540 2024-09-11T01:41:32.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.354+0000 7f5add7fa640 1 --2- 172.21.15.12:0/4050842724 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5ac003e210 0x7f5ac00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.354+0000 7f5adffff640 1 --2- 172.21.15.12:0/4050842724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ae01cd540 0x7f5ae0121f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:32.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.354+0000 7f5adffff640 10 monclient: get_auth_request con 0x7f5ae01cd540 auth_method 0 2024-09-11T01:41:32.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.354+0000 7f5adffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:32.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.354+0000 7f5adffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:32.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.354+0000 7f5adffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:32.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.354+0000 7f5add7fa640 1 -- 172.21.15.12:0/4050842724 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(408..408 src has 285..408) ==== 4739+0+0 (secure 0 0 0) 0x7f5ac8052a60 con 0x7f5ae0137e50 2024-09-11T01:41:32.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.354+0000 7f5adffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:32.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.354+0000 7f5adffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:32.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.354+0000 7f5adffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:32.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.354+0000 7f5ae4a6a640 1 --2- 172.21.15.12:0/4050842724 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5ac003e210 0x7f5ac00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:32.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.354+0000 7f5ae4a6a640 10 monclient: get_auth_request con 0x7f5ac003e210 auth_method 0 2024-09-11T01:41:32.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.354+0000 7f5adffff640 10 monclient(hunting): handle_auth_done global_id 134140 payload 824 2024-09-11T01:41:32.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.354+0000 7f5adffff640 1 -- 172.21.15.12:0/4050842724 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5ae01243a0 con 0x7f5ae01cd540 2024-09-11T01:41:32.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.356+0000 7f5adffff640 1 -- 172.21.15.12:0/4050842724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ae01cd540 msgr2=0x7f5ae0121f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:32.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.356+0000 7f5ae4a6a640 1 --2- 172.21.15.12:0/4050842724 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5ac003e210 0x7f5ac00406d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0123860 tx=0x7f5ad8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:32.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.356+0000 7f5adffff640 1 --2- 172.21.15.12:0/4050842724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ae01cd540 0x7f5ae0121f00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5ad00097a0 tx=0x7f5ad00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:32.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.357+0000 7f5add7fa640 1 -- 172.21.15.12:0/4050842724 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0043070 con 0x7f5ae01cd540 2024-09-11T01:41:32.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.357+0000 7f5add7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:32.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.357+0000 7f5add7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:41:32.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.357+0000 7f5add7fa640 10 monclient: dump: 2024-09-11T01:41:32.381 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:32.382 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:32.382 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.382 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.382 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:32.382 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.382 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:32.382 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:32.382 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:32.382 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.357+0000 7f5add7fa640 1 -- 172.21.15.12:0/4050842724 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5ad003ca60 con 0x7f5ae01cd540 2024-09-11T01:41:32.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.357+0000 7f5add7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:32.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.357+0000 7f5add7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:41:32.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.357+0000 7f5add7fa640 1 -- 172.21.15.12:0/4050842724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ae01cd540 msgr2=0x7f5ae0121f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:32.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.357+0000 7f5add7fa640 1 --2- 172.21.15.12:0/4050842724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ae01cd540 0x7f5ae0121f00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5ad00097a0 tx=0x7f5ad00046c0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.357+0000 7f5add7fa640 1 -- 172.21.15.12:0/4050842724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ae01cd540 msgr2=0x7f5ae0121f00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:41:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.357+0000 7f5add7fa640 1 --2- 172.21.15.12:0/4050842724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ae01cd540 0x7f5ae0121f00 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5ad00097a0 tx=0x7f5ad00046c0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.363+0000 7f5ae64cd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:41:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.363+0000 7f5ae64cd640 1 --2- 172.21.15.12:0/4050842724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ae0122440 0x7f5ae00a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.363+0000 7f5ae64cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.363+0000 7f5ae64cd640 1 -- 172.21.15.12:0/4050842724 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae0122850 con 0x7f5ae0122440 2024-09-11T01:41:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.363+0000 7f5adffff640 1 --2- 172.21.15.12:0/4050842724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ae0122440 0x7f5ae00a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.363+0000 7f5adffff640 10 monclient: get_auth_request con 0x7f5ae0122440 auth_method 0 2024-09-11T01:41:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.363+0000 7f5adffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.363+0000 7f5adffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.363+0000 7f5adffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.364+0000 7f5adffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.364+0000 7f5adffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.364+0000 7f5adffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.364+0000 7f5adffff640 10 monclient(hunting): handle_auth_done global_id 134143 payload 824 2024-09-11T01:41:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.364+0000 7f5adffff640 1 -- 172.21.15.12:0/4050842724 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5ae0135950 con 0x7f5ae0122440 2024-09-11T01:41:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.366+0000 7f5adffff640 1 -- 172.21.15.12:0/4050842724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ae0122440 msgr2=0x7f5ae00a18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.366+0000 7f5adffff640 1 --2- 172.21.15.12:0/4050842724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ae0122440 0x7f5ae00a18d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5ae00a9030 tx=0x7f5ad0009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.366+0000 7f5add7fa640 1 -- 172.21.15.12:0/4050842724 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0037040 con 0x7f5ae0122440 2024-09-11T01:41:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.366+0000 7f5add7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.366+0000 7f5add7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:41:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.366+0000 7f5add7fa640 10 monclient: dump: 2024-09-11T01:41:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.385 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:32.385 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:32.385 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:32.385 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.366+0000 7f5add7fa640 1 -- 172.21.15.12:0/4050842724 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7f5ad003b3a0 con 0x7f5ae0122440 2024-09-11T01:41:32.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.366+0000 7f5add7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:41:32.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.366+0000 7f5add7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:41:32.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.366+0000 7f5add7fa640 1 -- 172.21.15.12:0/4050842724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ae0122440 msgr2=0x7f5ae00a18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:32.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.366+0000 7f5add7fa640 1 --2- 172.21.15.12:0/4050842724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ae0122440 0x7f5ae00a18d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5ae00a9030 tx=0x7f5ad0009190 comp rx=0 tx=0).stop 2024-09-11T01:41:32.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.366+0000 7f5add7fa640 1 -- 172.21.15.12:0/4050842724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ae0122440 msgr2=0x7f5ae00a18d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:41:32.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.366+0000 7f5add7fa640 1 --2- 172.21.15.12:0/4050842724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ae0122440 0x7f5ae00a18d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.386 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:41:32.386 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:41:32.386 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:41:32.386 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:41:32.386 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 114, 2024-09-11T01:41:32.386 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 25328, 2024-09-11T01:41:32.386 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:41:32.386 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:41:32.386 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:41:32.386 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:41:32.386 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:32.386 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 5, 2024-09-11T01:41:32.386 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:41:32.387 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:41:32.387 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:41:32.387 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:41:32.387 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:41:32.387 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:41:32.387 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:41:32.387 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:41:32.387 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:41:32.387 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:41:32.388 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:41:32.388 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:41:32.388 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:41:32.388 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:41:32.388 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:32.388 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:41:32.388 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:41:32.388 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:41:32.388 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:41:32.388 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:41:32.388 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:41:32.388 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:41:32.389 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:41:32.389 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:41:32.389 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:41:32.389 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:41:32.389 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:41:32.389 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:41:32.389 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:32.389 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.389 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:41:32.389 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:41:32.389 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:41:32.389 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:41:32.389 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:41:32.389 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:41:32.390 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:41:32.390 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:41:32.390 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:41:32.390 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:41:32.390 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:41:32.390 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:41:32.390 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:41:32.390 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:41:32.390 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:41:32.390 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:41:32.390 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:41:32.390 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:41:32.390 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:41:32.390 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:41:32.390 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:41:32.391 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:41:32.391 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:41:32.391 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:41:32.391 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:41:32.391 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:41:32.391 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:41:32.391 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:41:32.391 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:32.391 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:41:32.391 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.391 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:41:32.391 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.391 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:41:32.392 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:41:32.392 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:41:32.392 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:41:32.392 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.392 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:41:32.392 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:41:32.392 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:32.392 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.392 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.392 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:41:32.392 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:41:32.392 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:32.392 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:32.392 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:32.393 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.393 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:41:32.393 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:41:32.393 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:41:32.393 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:41:32.393 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:41:32.393 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.393 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.393 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:41:32.393 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:41:32.393 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:41:32.393 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:41:32.393 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.394 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:41:32.394 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:41:32.394 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:32.394 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.394 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.394 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:41:32.394 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:41:32.394 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:32.394 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:32.394 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:32.394 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.394 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:41:32.394 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:41:32.394 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:41:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:41:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:41:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:41:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:41:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:41:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:41:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:41:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:41:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:41:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:41:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:41:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:41:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:41:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:41:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:41:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:41:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:41:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:41:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:41:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:41:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:41:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:41:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:41:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:41:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:32.398 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:32.398 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:32.398 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:41:32.398 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:41:32.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.367+0000 7f5ae64cd640 1 -- 172.21.15.12:0/4050842724 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5ac003e210 msgr2=0x7f5ac00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:32.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.367+0000 7f5ae64cd640 1 --2- 172.21.15.12:0/4050842724 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5ac003e210 0x7f5ac00406d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0123860 tx=0x7f5ad8009040 comp rx=0 tx=0).stop 2024-09-11T01:41:32.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.367+0000 7f5ae64cd640 10 monclient: shutdown 2024-09-11T01:41:32.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.367+0000 7f5ae64cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:32.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.367+0000 7f5ae64cd640 1 -- 172.21.15.12:0/4050842724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ae0137e50 msgr2=0x7f5ae01342d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:32.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.367+0000 7f5ae64cd640 1 --2- 172.21.15.12:0/4050842724 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5ae0137e50 0x7f5ae01342d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800e900 tx=0x7f5ac800edc0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.367+0000 7f5ae4a6a640 1 -- 172.21.15.12:0/4050842724 reap_dead start 2024-09-11T01:41:32.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.367+0000 7f5ae4a6a640 1 -- 172.21.15.12:0/4050842724 reap_dead start 2024-09-11T01:41:32.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.367+0000 7f5ae64cd640 1 -- 172.21.15.12:0/4050842724 shutdown_connections 2024-09-11T01:41:32.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.367+0000 7f5ae64cd640 1 -- 172.21.15.12:0/4050842724 >> 172.21.15.12:0/4050842724 conn(0x7f5ae00a7020 msgr2=0x7f5ae0136220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:32.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.367+0000 7f5ae64cd640 1 -- 172.21.15.12:0/4050842724 shutdown_connections 2024-09-11T01:41:32.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.367+0000 7f5ae64cd640 1 -- 172.21.15.12:0/4050842724 wait complete. 2024-09-11T01:41:32.400 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:41:32.401 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-11T01:41:32.401 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T01:41:32.401 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-11T01:41:32.401 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-11T01:41:32.401 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-11T01:41:32.401 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:41:32.413 INFO:tasks.ceph.mon.c:Stopped 2024-09-11T01:41:32.413 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:41:32.413 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:41:32.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.551+0000 7fc1c6cc6640 10 monclient: get_monmap_and_config 2024-09-11T01:41:32.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.551+0000 7fc1c6cc6640 10 monclient: build_initial_monmap 2024-09-11T01:41:32.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.551+0000 7fc1c6cc6640 10 monclient: monmap: 2024-09-11T01:41:32.553 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:32.553 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:32.553 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:41:32.552590+0000 2024-09-11T01:41:32.554 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:41:32.552590+0000 2024-09-11T01:41:32.554 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:32.554 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.554 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:41:32.554 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:41:32.554 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:41:32.554 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:41:32.554 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:41:32.554 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.552+0000 7fc1c6cc6640 1 Processor -- start 2024-09-11T01:41:32.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.552+0000 7fc1c6cc6640 1 -- start start 2024-09-11T01:41:32.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.552+0000 7fc1c6cc6640 10 monclient: init 2024-09-11T01:41:32.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1c6cc6640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:32.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1c6cc6640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-11T01:41:32.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1c6cc6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1c009d790 0x7fc1c009db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1c6cc6640 10 monclient(hunting): picked mon.noname-b con 0x7fc1c009d790 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:32.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1c6cc6640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc1c009e0b0 0x7fc1c009e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1c6cc6640 10 monclient(hunting): picked mon.noname-a con 0x7fc1c009e0b0 addr v2:172.21.15.12:3300/0 2024-09-11T01:41:32.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1c6cc6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc1c0134a60 addr v1:172.21.15.12:6789/0 2024-09-11T01:41:32.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1c6cc6640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1c00a1da0 con 0x7fc1c0134a60 2024-09-11T01:41:32.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1c6cc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:32.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1c6cc6640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc1c00a0760 con 0x7fc1c009e0b0 2024-09-11T01:41:32.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1c6cc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:32.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1c6cc6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc1c00a1fa0 con 0x7fc1c009d790 2024-09-11T01:41:32.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1c6cc6640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:32.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1c6cc6640 10 monclient(hunting): authenticate will time out at 1498.995235s 2024-09-11T01:41:32.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1c4a62640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1c009d790 msgr2=0x7fc1c009db70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:41:32.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1c4a62640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1c009d790 0x7fc1c009db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:41:32.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1bffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc1c009e0b0 0x7fc1c009e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:32.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1c5263640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc1c0134a60 0x7fc1c0134e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42316/0 (socket says 172.21.15.12:42316) 2024-09-11T01:41:32.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1bffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc1c009e0b0 0x7fc1c009e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58750/0 (socket says 172.21.15.12:58750) 2024-09-11T01:41:32.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1c5263640 1 -- 172.21.15.12:0/1933354900 learned_addr learned my addr 172.21.15.12:0/1933354900 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:41:32.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1bffff640 10 monclient(hunting): get_auth_request con 0x7fc1c009e0b0 auth_method 0 2024-09-11T01:41:32.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:32.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1bffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:32.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:32.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:32.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:32.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:32.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1bffff640 10 monclient(hunting): handle_auth_done global_id 94500 payload 274 2024-09-11T01:41:32.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1bffff640 1 -- 172.21.15.12:0/1933354900 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1c009d790 msgr2=0x7fc1c009db70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:41:32.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1bffff640 1 --2- 172.21.15.12:0/1933354900 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1c009d790 0x7fc1c009db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1bffff640 1 -- 172.21.15.12:0/1933354900 >> v1:172.21.15.12:6789/0 conn(0x7fc1c0134a60 legacy=0x7fc1c0134e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:32.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1bffff640 10 monclient: _finish_hunting 0 2024-09-11T01:41:32.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1bffff640 1 monclient: found mon.noname-a 2024-09-11T01:41:32.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:32.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1bffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:41:32.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.553+0000 7fc1bffff640 1 -- 172.21.15.12:0/1933354900 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1c009eda0 con 0x7fc1c009e0b0 2024-09-11T01:41:32.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.556+0000 7fc1bffff640 1 -- 172.21.15.12:0/1933354900 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1c009e0b0 msgr2=0x7fc1c009e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:32.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.556+0000 7fc1bffff640 1 --2- 172.21.15.12:0/1933354900 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1c009e0b0 0x7fc1c009e4d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8009d90 tx=0x7fc1a802a4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d18148fb37baf0f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:32.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.556+0000 7fc1bf7fe640 1 -- 172.21.15.12:0/1933354900 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a8007a30 con 0x7fc1c009e0b0 2024-09-11T01:41:32.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.556+0000 7fc1bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:32.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.556+0000 7fc1bf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:41:32.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.556+0000 7fc1bf7fe640 10 monclient: dump: 2024-09-11T01:41:32.567 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:32.567 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:32.567 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.568 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.568 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:32.568 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.568 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:32.568 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:32.568 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:32.568 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.556+0000 7fc1bf7fe640 10 monclient: _finish_auth 0 2024-09-11T01:41:32.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.556+0000 7fc1bf7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:41:32.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.556+0000 7fc1bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:32.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.556+0000 7fc1bf7fe640 1 -- 172.21.15.12:0/1933354900 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1a8007bd0 con 0x7fc1c009e0b0 2024-09-11T01:41:32.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.556+0000 7fc1bf7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:32.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.556+0000 7fc1bf7fe640 1 -- 172.21.15.12:0/1933354900 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a8031400 con 0x7fc1c009e0b0 2024-09-11T01:41:32.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.556+0000 7fc1c6cc6640 5 monclient: authenticate success, global_id 94500 2024-09-11T01:41:32.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.556+0000 7fc1c6cc6640 10 monclient: get_monmap_and_config success 2024-09-11T01:41:32.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.556+0000 7fc1c6cc6640 4 set_mon_vals no callback set 2024-09-11T01:41:32.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.556+0000 7fc1bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:32.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.556+0000 7fc1bf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:32.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.556+0000 7fc1bf7fe640 10 monclient: dump: 2024-09-11T01:41:32.572 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:32.572 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:32.572 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.572 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.573 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:32.573 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.573 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:32.573 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:32.573 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:32.573 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.557+0000 7fc1c6cc6640 10 monclient: shutdown 2024-09-11T01:41:32.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.557+0000 7fc1c6cc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:32.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.557+0000 7fc1c6cc6640 1 -- 172.21.15.12:0/1933354900 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1c009e0b0 msgr2=0x7fc1c009e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:32.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.557+0000 7fc1c6cc6640 1 --2- 172.21.15.12:0/1933354900 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1c009e0b0 0x7fc1c009e4d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8009d90 tx=0x7fc1a802a4b0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.557+0000 7fc1c6cc6640 1 -- 172.21.15.12:0/1933354900 shutdown_connections 2024-09-11T01:41:32.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.557+0000 7fc1c6cc6640 1 --2- 172.21.15.12:0/1933354900 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1c009e0b0 0x7fc1c009e4d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.557+0000 7fc1c6cc6640 1 --2- 172.21.15.12:0/1933354900 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1c009d790 0x7fc1c009db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.557+0000 7fc1c6cc6640 1 -- 172.21.15.12:0/1933354900 >> 172.21.15.12:0/1933354900 conn(0x7fc1c00a2d90 msgr2=0x7fc1c00a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:32.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.557+0000 7fc1c6cc6640 1 -- 172.21.15.12:0/1933354900 shutdown_connections 2024-09-11T01:41:32.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.557+0000 7fc1c6cc6640 1 -- 172.21.15.12:0/1933354900 wait complete. 2024-09-11T01:41:32.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.557+0000 7fc1c6cc6640 10 monclient: build_initial_monmap 2024-09-11T01:41:32.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.557+0000 7fc1c6cc6640 10 monclient: monmap: 2024-09-11T01:41:32.576 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:32.576 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:41:32.576 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:41:32.576 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:41:32.576 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:32.577 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.577 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:41:32.577 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:41:32.577 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:41:32.577 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.558+0000 7fc1c6cc6640 1 Processor -- start 2024-09-11T01:41:32.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.558+0000 7fc1c6cc6640 1 -- start start 2024-09-11T01:41:32.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.558+0000 7fc1c6cc6640 10 monclient: init 2024-09-11T01:41:32.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.558+0000 7fc1c6cc6640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:32.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.558+0000 7fc1c6cc6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:41:32.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.558+0000 7fc1c6cc6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1c009d790 0x7fc1c0133a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.558+0000 7fc1c6cc6640 10 monclient(hunting): picked mon.noname-c con 0x7fc1c009d790 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:32.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.558+0000 7fc1c6cc6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1c009e0b0 0x7fc1c0133fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.558+0000 7fc1c6cc6640 10 monclient(hunting): picked mon.noname-a con 0x7fc1c009e0b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:41:32.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.558+0000 7fc1c6cc6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc1c0134a60 0x7fc1c0124680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.558+0000 7fc1c6cc6640 10 monclient(hunting): picked mon.noname-b con 0x7fc1c0134a60 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:41:32.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.558+0000 7fc1c6cc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:32.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.558+0000 7fc1c6cc6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c009ce20 con 0x7fc1c009e0b0 2024-09-11T01:41:32.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.558+0000 7fc1c6cc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:32.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.558+0000 7fc1c6cc6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c009cca0 con 0x7fc1c0134a60 2024-09-11T01:41:32.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.558+0000 7fc1c6cc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:32.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.558+0000 7fc1c6cc6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc1c0135590 con 0x7fc1c009d790 2024-09-11T01:41:32.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.558+0000 7fc1c6cc6640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:32.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.558+0000 7fc1c6cc6640 10 monclient(hunting): authenticate will time out at 1499.000939s 2024-09-11T01:41:32.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.558+0000 7fc1c4a62640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1c009d790 msgr2=0x7fc1c0133a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:41:32.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.558+0000 7fc1c4a62640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1c009d790 0x7fc1c0133a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:41:32.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.559+0000 7fc1bffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1c009e0b0 0x7fc1c0133fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:32.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.559+0000 7fc1bffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1c009e0b0 0x7fc1c0133fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58762/0 (socket says 172.21.15.12:58762) 2024-09-11T01:41:32.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.559+0000 7fc1bffff640 1 -- 172.21.15.12:0/1809489037 learned_addr learned my addr 172.21.15.12:0/1809489037 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:41:32.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.559+0000 7fc1bffff640 10 monclient(hunting): get_auth_request con 0x7fc1c009e0b0 auth_method 0 2024-09-11T01:41:32.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.559+0000 7fc1bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:32.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.559+0000 7fc1bffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:32.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.559+0000 7fc1bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:32.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.559+0000 7fc1bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:32.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.559+0000 7fc1bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:32.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.559+0000 7fc1bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:32.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.559+0000 7fc1c5263640 1 --2- 172.21.15.12:0/1809489037 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc1c0134a60 0x7fc1c0124680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:32.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.559+0000 7fc1bffff640 10 monclient(hunting): handle_auth_done global_id 94503 payload 824 2024-09-11T01:41:32.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.559+0000 7fc1bffff640 1 -- 172.21.15.12:0/1809489037 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1c009d790 msgr2=0x7fc1c0133a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:41:32.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.559+0000 7fc1bffff640 1 --2- 172.21.15.12:0/1809489037 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1c009d790 0x7fc1c0133a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.559+0000 7fc1bffff640 1 -- 172.21.15.12:0/1809489037 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc1c0134a60 msgr2=0x7fc1c0124680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:32.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.559+0000 7fc1bffff640 1 --2- 172.21.15.12:0/1809489037 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc1c0134a60 0x7fc1c0124680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.559+0000 7fc1bffff640 10 monclient: _finish_hunting 0 2024-09-11T01:41:32.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.559+0000 7fc1bffff640 1 monclient: found mon.noname-a 2024-09-11T01:41:32.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.559+0000 7fc1bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:32.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.559+0000 7fc1bffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:41:32.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.559+0000 7fc1bffff640 1 -- 172.21.15.12:0/1809489037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1c01cdcc0 con 0x7fc1c009e0b0 2024-09-11T01:41:32.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.559+0000 7fc1c5263640 10 monclient: get_auth_request con 0x7fc1c0134a60 auth_method 0 2024-09-11T01:41:32.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.559+0000 7fc1c5263640 1 --2- 172.21.15.12:0/1809489037 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc1c0134a60 0x7fc1c0124680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:41:32.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.561+0000 7fc1bffff640 1 -- 172.21.15.12:0/1809489037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1c009e0b0 msgr2=0x7fc1c0133fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:32.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.561+0000 7fc1bffff640 1 --2- 172.21.15.12:0/1809489037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1c009e0b0 0x7fc1c0133fb0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8009eb0 tx=0x7fc1a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:32.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.561+0000 7fc1bd7fa640 1 -- 172.21.15.12:0/1809489037 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a8040070 con 0x7fc1c009e0b0 2024-09-11T01:41:32.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.561+0000 7fc1bd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:32.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.561+0000 7fc1bd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:41:32.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.561+0000 7fc1bd7fa640 10 monclient: dump: 2024-09-11T01:41:32.584 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:32.584 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:32.584 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.584 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.584 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:32.584 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.584 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:32.585 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:32.585 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:32.585 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.561+0000 7fc1bd7fa640 10 monclient: _finish_auth 0 2024-09-11T01:41:32.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.561+0000 7fc1bd7fa640 10 monclient: _check_auth_tickets 2024-09-11T01:41:32.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.561+0000 7fc1bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:32.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.561+0000 7fc1bd7fa640 1 -- 172.21.15.12:0/1809489037 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1a8045660 con 0x7fc1c009e0b0 2024-09-11T01:41:32.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.561+0000 7fc1c6cc6640 5 monclient: authenticate success, global_id 94503 2024-09-11T01:41:32.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.561+0000 7fc1bd7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:32.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.561+0000 7fc1bd7fa640 1 -- 172.21.15.12:0/1809489037 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a8045a30 con 0x7fc1c009e0b0 2024-09-11T01:41:32.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.561+0000 7fc1bd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:32.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.561+0000 7fc1bd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:32.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.561+0000 7fc1bd7fa640 10 monclient: dump: 2024-09-11T01:41:32.586 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:32.586 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:32.586 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.586 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:32.586 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:32.586 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:32.586 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:32.586 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:32.586 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:32.586 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:32.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.561+0000 7fc1c6cc6640 10 monclient: _renew_subs 2024-09-11T01:41:32.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.561+0000 7fc1c6cc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:32.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.561+0000 7fc1c6cc6640 1 -- 172.21.15.12:0/1809489037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1c01344f0 con 0x7fc1c009e0b0 2024-09-11T01:41:32.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.561+0000 7fc1beffd640 4 set_mon_vals no callback set 2024-09-11T01:41:32.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.561+0000 7fc1c6cc6640 10 monclient: _renew_subs 2024-09-11T01:41:32.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.561+0000 7fc1c6cc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:32.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.561+0000 7fc1c6cc6640 1 -- 172.21.15.12:0/1809489037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1c01281c0 con 0x7fc1c009e0b0 2024-09-11T01:41:32.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.562+0000 7fc1bd7fa640 1 -- 172.21.15.12:0/1809489037 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fc1a8007a40 con 0x7fc1c009e0b0 2024-09-11T01:41:32.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.562+0000 7fc1c6cc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:32.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.562+0000 7fc1c6cc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:32.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.562+0000 7fc1c6cc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:32.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.562+0000 7fc1c6cc6640 1 -- 172.21.15.12:0/1809489037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1c009db70 con 0x7fc1c009e0b0 2024-09-11T01:41:32.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.563+0000 7fc1bd7fa640 1 --2- 172.21.15.12:0/1809489037 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc1a003e1f0 0x7fc1a00406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:32.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.563+0000 7fc1bd7fa640 1 -- 172.21.15.12:0/1809489037 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(408..408 src has 285..408) ==== 4739+0+0 (secure 0 0 0) 0x7fc1a8074be0 con 0x7fc1c009e0b0 2024-09-11T01:41:32.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.563+0000 7fc1c4a62640 1 --2- 172.21.15.12:0/1809489037 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc1a003e1f0 0x7fc1a00406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:32.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.563+0000 7fc1c4a62640 10 monclient: get_auth_request con 0x7fc1a003e1f0 auth_method 0 2024-09-11T01:41:32.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.565+0000 7fc1c4a62640 1 --2- 172.21.15.12:0/1809489037 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc1a003e1f0 0x7fc1a00406b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0003980 tx=0x7fc1b0006d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:32.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.567+0000 7fc1bd7fa640 1 -- 172.21.15.12:0/1809489037 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc1a8073600 con 0x7fc1c009e0b0 2024-09-11T01:41:32.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.567+0000 7fc1bd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:32.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.567+0000 7fc1bd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:41:32.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.822+0000 7fc1c6cc6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:41:32.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.822+0000 7fc1c6cc6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:41:32.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.822+0000 7fc1c6cc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:32.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.822+0000 7fc1c6cc6640 1 -- 172.21.15.12:0/1809489037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc1c006d460 con 0x7fc1c009e0b0 2024-09-11T01:41:32.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.823+0000 7fc1bd7fa640 1 -- 172.21.15.12:0/1809489037 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fc1a803cd80 con 0x7fc1c009e0b0 2024-09-11T01:41:32.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.823+0000 7fc1bd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:41:32.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.823+0000 7fc1bd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:41:32.825 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:41:32.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.825+0000 7fc1c6cc6640 1 -- 172.21.15.12:0/1809489037 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc1a003e1f0 msgr2=0x7fc1a00406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:32.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.825+0000 7fc1c6cc6640 1 --2- 172.21.15.12:0/1809489037 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc1a003e1f0 0x7fc1a00406b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0003980 tx=0x7fc1b0006d30 comp rx=0 tx=0).stop 2024-09-11T01:41:32.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.825+0000 7fc1c6cc6640 10 monclient: shutdown 2024-09-11T01:41:32.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.825+0000 7fc1c6cc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:32.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.825+0000 7fc1c6cc6640 1 -- 172.21.15.12:0/1809489037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1c009e0b0 msgr2=0x7fc1c0133fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:32.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.825+0000 7fc1c6cc6640 1 --2- 172.21.15.12:0/1809489037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1c009e0b0 0x7fc1c0133fb0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8009eb0 tx=0x7fc1a8004a40 comp rx=0 tx=0).stop 2024-09-11T01:41:32.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.825+0000 7fc1c6cc6640 1 -- 172.21.15.12:0/1809489037 shutdown_connections 2024-09-11T01:41:32.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.825+0000 7fc1c6cc6640 1 --2- 172.21.15.12:0/1809489037 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc1c0134a60 0x7fc1c0124680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.825+0000 7fc1c6cc6640 1 --2- 172.21.15.12:0/1809489037 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc1a003e1f0 0x7fc1a00406b0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.825+0000 7fc1c6cc6640 1 --2- 172.21.15.12:0/1809489037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc1c009e0b0 0x7fc1c0133fb0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.825+0000 7fc1c6cc6640 1 --2- 172.21.15.12:0/1809489037 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1c009d790 0x7fc1c0133a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:32.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.825+0000 7fc1c6cc6640 1 -- 172.21.15.12:0/1809489037 >> 172.21.15.12:0/1809489037 conn(0x7fc1c00a2d90 msgr2=0x7fc1c00a0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:32.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.825+0000 7fc1c6cc6640 1 -- 172.21.15.12:0/1809489037 shutdown_connections 2024-09-11T01:41:32.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:32.825+0000 7fc1c6cc6640 1 -- 172.21.15.12:0/1809489037 wait complete. 2024-09-11T01:41:35.838 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:41:35.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.949+0000 7f695d13d640 10 monclient: get_monmap_and_config 2024-09-11T01:41:35.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.949+0000 7f695d13d640 10 monclient: build_initial_monmap 2024-09-11T01:41:35.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.949+0000 7f695d13d640 10 monclient: monmap: 2024-09-11T01:41:35.952 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:35.952 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:35.952 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:41:35.950501+0000 2024-09-11T01:41:35.952 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:41:35.950501+0000 2024-09-11T01:41:35.952 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:35.952 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:35.952 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:41:35.952 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:41:35.952 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:41:35.952 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:41:35.953 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:41:35.953 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:35.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.950+0000 7f695d13d640 1 Processor -- start 2024-09-11T01:41:35.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.950+0000 7f695d13d640 1 -- start start 2024-09-11T01:41:35.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.950+0000 7f695d13d640 10 monclient: init 2024-09-11T01:41:35.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.950+0000 7f695d13d640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:35.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.950+0000 7f695d13d640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-11T01:41:35.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.950+0000 7f695d13d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f69580a2e90 0x7f695806dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:35.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.950+0000 7f695d13d640 10 monclient(hunting): picked mon.noname-b con 0x7f69580a2e90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:35.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.950+0000 7f695d13d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f69580a3270 0x7f69580a3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:35.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.950+0000 7f695d13d640 10 monclient(hunting): picked mon.noname-c con 0x7f69580a3270 addr v2:172.21.15.71:3300/0 2024-09-11T01:41:35.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.950+0000 7f695d13d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6958138040 addr v1:172.21.15.12:6789/0 2024-09-11T01:41:35.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.950+0000 7f695d13d640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69580a46c0 con 0x7f6958138040 2024-09-11T01:41:35.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.950+0000 7f695d13d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:35.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.950+0000 7f695d13d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f69580a4900 con 0x7f69580a3270 2024-09-11T01:41:35.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.950+0000 7f695d13d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:35.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.950+0000 7f695d13d640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f695806ee50 con 0x7f69580a2e90 2024-09-11T01:41:35.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.950+0000 7f695d13d640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:35.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.950+0000 7f695d13d640 10 monclient(hunting): authenticate will time out at 1502.392998s 2024-09-11T01:41:35.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.951+0000 7f6956d9d640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f69580a2e90 msgr2=0x7f695806dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:41:35.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.951+0000 7f6956d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f69580a2e90 0x7f695806dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:41:35.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.951+0000 7f695759e640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6958138040 0x7f6958138460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42318/0 (socket says 172.21.15.12:42318) 2024-09-11T01:41:35.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.951+0000 7f695759e640 1 -- 172.21.15.12:0/105180733 learned_addr learned my addr 172.21.15.12:0/105180733 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:41:35.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.951+0000 7f695659c640 1 --2- 172.21.15.12:0/105180733 >> v2:172.21.15.71:3300/0 conn(0x7f69580a3270 0x7f69580a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:35.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.951+0000 7f695659c640 10 monclient(hunting): get_auth_request con 0x7f69580a3270 auth_method 0 2024-09-11T01:41:35.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.951+0000 7f695659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:35.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.951+0000 7f695659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:35.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.951+0000 7f695659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:35.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.951+0000 7f695659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:35.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.951+0000 7f695659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:35.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.951+0000 7f695659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:35.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.951+0000 7f695659c640 10 monclient(hunting): handle_auth_done global_id 134146 payload 274 2024-09-11T01:41:35.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.951+0000 7f695659c640 1 -- 172.21.15.12:0/105180733 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f69580a2e90 msgr2=0x7f695806dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:41:35.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.951+0000 7f695659c640 1 --2- 172.21.15.12:0/105180733 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f69580a2e90 0x7f695806dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:35.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.951+0000 7f695659c640 1 -- 172.21.15.12:0/105180733 >> v1:172.21.15.12:6789/0 conn(0x7f6958138040 legacy=0x7f6958138460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:35.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.951+0000 7f695659c640 10 monclient: _finish_hunting 0 2024-09-11T01:41:35.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.951+0000 7f695659c640 1 monclient: found mon.noname-c 2024-09-11T01:41:35.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.951+0000 7f695659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:35.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.951+0000 7f695659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:41:35.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.951+0000 7f695659c640 1 -- 172.21.15.12:0/105180733 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f695806eb70 con 0x7f69580a3270 2024-09-11T01:41:35.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.954+0000 7f695659c640 1 -- 172.21.15.12:0/105180733 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69580a3270 msgr2=0x7f69580a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:35.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.954+0000 7f695659c640 1 --2- 172.21.15.12:0/105180733 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69580a3270 0x7f69580a3650 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f69580a4a80 tx=0x7f6940029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d1ced5a57d29d3c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:35.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.955+0000 7f6955d9b640 1 -- 172.21.15.12:0/105180733 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6940037070 con 0x7f69580a3270 2024-09-11T01:41:35.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.955+0000 7f6955d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:35.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.955+0000 7f6955d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:41:35.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.955+0000 7f6955d9b640 10 monclient: dump: 2024-09-11T01:41:35.966 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:35.967 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:35.967 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:35.967 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:35.967 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:35.967 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:35.967 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:35.967 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:35.967 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:35.967 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:35.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.955+0000 7f6955d9b640 10 monclient: _finish_auth 0 2024-09-11T01:41:35.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.955+0000 7f6955d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:41:35.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.955+0000 7f6955d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:35.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.955+0000 7f6955d9b640 1 -- 172.21.15.12:0/105180733 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6940032640 con 0x7f69580a3270 2024-09-11T01:41:35.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.955+0000 7f6955d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:35.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.955+0000 7f6955d9b640 1 -- 172.21.15.12:0/105180733 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6940032a10 con 0x7f69580a3270 2024-09-11T01:41:35.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.955+0000 7f695d13d640 5 monclient: authenticate success, global_id 134146 2024-09-11T01:41:35.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.955+0000 7f695d13d640 10 monclient: get_monmap_and_config success 2024-09-11T01:41:35.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.955+0000 7f695d13d640 4 set_mon_vals no callback set 2024-09-11T01:41:35.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.955+0000 7f6955d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:35.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.955+0000 7f6955d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:41:35.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.955+0000 7f6955d9b640 10 monclient: dump: 2024-09-11T01:41:35.968 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:35.969 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:35.969 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:35.969 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:35.969 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:35.969 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:35.969 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:35.969 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:35.969 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:35.969 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:35.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.955+0000 7f695d13d640 10 monclient: shutdown 2024-09-11T01:41:35.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.955+0000 7f695d13d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:35.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.955+0000 7f695d13d640 1 -- 172.21.15.12:0/105180733 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69580a3270 msgr2=0x7f69580a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:35.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.955+0000 7f695d13d640 1 --2- 172.21.15.12:0/105180733 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69580a3270 0x7f69580a3650 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f69580a4a80 tx=0x7f6940029fd0 comp rx=0 tx=0).stop 2024-09-11T01:41:35.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.955+0000 7f695d13d640 1 -- 172.21.15.12:0/105180733 shutdown_connections 2024-09-11T01:41:35.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.955+0000 7f695d13d640 1 --2- 172.21.15.12:0/105180733 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69580a3270 0x7f69580a3650 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:35.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.955+0000 7f695d13d640 1 --2- 172.21.15.12:0/105180733 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f69580a2e90 0x7f695806dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:35.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.955+0000 7f695d13d640 1 -- 172.21.15.12:0/105180733 >> 172.21.15.12:0/105180733 conn(0x7f69580a8490 msgr2=0x7f69580a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:35.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.956+0000 7f695d13d640 1 -- 172.21.15.12:0/105180733 shutdown_connections 2024-09-11T01:41:35.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.956+0000 7f695d13d640 1 -- 172.21.15.12:0/105180733 wait complete. 2024-09-11T01:41:35.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.956+0000 7f695d13d640 10 monclient: build_initial_monmap 2024-09-11T01:41:35.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.956+0000 7f695d13d640 10 monclient: monmap: 2024-09-11T01:41:35.971 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:35.971 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:41:35.971 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:41:35.971 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:41:35.971 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:35.971 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:35.971 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:41:35.971 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:41:35.971 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:41:35.971 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:35.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.957+0000 7f695d13d640 1 Processor -- start 2024-09-11T01:41:35.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.957+0000 7f695d13d640 1 -- start start 2024-09-11T01:41:35.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.957+0000 7f695d13d640 10 monclient: init 2024-09-11T01:41:35.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.957+0000 7f695d13d640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:35.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.957+0000 7f695d13d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:41:35.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.957+0000 7f695d13d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69580a2e90 0x7f69581cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:35.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.957+0000 7f695d13d640 10 monclient(hunting): picked mon.noname-b con 0x7f69580a2e90 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:41:35.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.957+0000 7f695d13d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69580a3270 0x7f69581cda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:35.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.957+0000 7f695d13d640 10 monclient(hunting): picked mon.noname-a con 0x7f69580a3270 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:41:35.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.957+0000 7f695d13d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6958138040 0x7f6958124910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:35.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.957+0000 7f695d13d640 10 monclient(hunting): picked mon.noname-c con 0x7f6958138040 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:35.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.957+0000 7f695d13d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:35.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.957+0000 7f695d13d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f69580a3b90 con 0x7f69580a3270 2024-09-11T01:41:35.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.957+0000 7f695d13d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:35.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.957+0000 7f695d13d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f695806fba0 con 0x7f69580a2e90 2024-09-11T01:41:35.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.957+0000 7f695d13d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:35.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.957+0000 7f695d13d640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f695809f680 con 0x7f6958138040 2024-09-11T01:41:35.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.957+0000 7f695d13d640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:35.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.957+0000 7f695d13d640 10 monclient(hunting): authenticate will time out at 1502.399975s 2024-09-11T01:41:35.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f695759e640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6958138040 msgr2=0x7f6958124910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:41:35.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f695759e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6958138040 0x7f6958124910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:41:35.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f695659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69580a3270 0x7f69581cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:35.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f695659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69580a3270 0x7f69581cda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58778/0 (socket says 172.21.15.12:58778) 2024-09-11T01:41:35.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f695659c640 1 -- 172.21.15.12:0/1770182753 learned_addr learned my addr 172.21.15.12:0/1770182753 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:41:35.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f695659c640 10 monclient(hunting): get_auth_request con 0x7f69580a3270 auth_method 0 2024-09-11T01:41:35.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f695659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:35.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f695659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:35.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f695659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:35.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f695659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:35.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f695659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:35.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f695659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:35.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f6956d9d640 1 --2- 172.21.15.12:0/1770182753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69580a2e90 0x7f69581cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:35.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f6956d9d640 10 monclient(hunting): get_auth_request con 0x7f69580a2e90 auth_method 0 2024-09-11T01:41:35.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f6956d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:35.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f6956d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:35.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f6956d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:35.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f6956d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:35.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f6956d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:35.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f6956d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:35.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f695659c640 10 monclient(hunting): handle_auth_done global_id 94506 payload 824 2024-09-11T01:41:35.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f695659c640 1 -- 172.21.15.12:0/1770182753 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6958138040 msgr2=0x7f6958124910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:41:35.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f695659c640 1 --2- 172.21.15.12:0/1770182753 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6958138040 0x7f6958124910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:35.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f695659c640 1 -- 172.21.15.12:0/1770182753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69580a2e90 msgr2=0x7f69581cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:35.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f695659c640 1 --2- 172.21.15.12:0/1770182753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69580a2e90 0x7f69581cd4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:35.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f695659c640 10 monclient: _finish_hunting 0 2024-09-11T01:41:35.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f695659c640 1 monclient: found mon.noname-a 2024-09-11T01:41:35.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f695659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:35.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f695659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:41:35.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.958+0000 7f695659c640 1 -- 172.21.15.12:0/1770182753 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6958128450 con 0x7f69580a3270 2024-09-11T01:41:35.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.960+0000 7f695659c640 1 -- 172.21.15.12:0/1770182753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69580a3270 msgr2=0x7f69581cda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:35.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.960+0000 7f695659c640 1 --2- 172.21.15.12:0/1770182753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69580a3270 0x7f69581cda20 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6940009890 tx=0x7f69400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:35.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.960+0000 7f694f7fe640 1 -- 172.21.15.12:0/1770182753 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6940042020 con 0x7f69580a3270 2024-09-11T01:41:35.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.960+0000 7f694f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:35.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.960+0000 7f694f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:41:35.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.960+0000 7f694f7fe640 10 monclient: dump: 2024-09-11T01:41:35.976 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:35.977 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:35.977 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:35.977 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:35.977 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:35.977 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:35.977 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:35.977 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:35.977 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:35.977 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:35.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.961+0000 7f694f7fe640 10 monclient: _finish_auth 0 2024-09-11T01:41:35.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.961+0000 7f694f7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:41:35.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.961+0000 7f694f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:35.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.961+0000 7f694f7fe640 1 -- 172.21.15.12:0/1770182753 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6940030880 con 0x7f69580a3270 2024-09-11T01:41:35.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.961+0000 7f695d13d640 5 monclient: authenticate success, global_id 94506 2024-09-11T01:41:35.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.961+0000 7f695d13d640 10 monclient: _renew_subs 2024-09-11T01:41:35.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.961+0000 7f695d13d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:35.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.961+0000 7f695d13d640 1 -- 172.21.15.12:0/1770182753 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6958128730 con 0x7f69580a3270 2024-09-11T01:41:35.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.961+0000 7f694f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:35.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.961+0000 7f694f7fe640 1 -- 172.21.15.12:0/1770182753 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6940037040 con 0x7f69580a3270 2024-09-11T01:41:35.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.961+0000 7f694f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:35.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.961+0000 7f694f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:35.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.961+0000 7f694f7fe640 10 monclient: dump: 2024-09-11T01:41:35.978 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:35.979 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:35.979 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:35.979 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:35.979 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:35.979 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:35.979 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:35.979 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:35.979 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:35.979 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:35.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.961+0000 7f695559a640 4 set_mon_vals no callback set 2024-09-11T01:41:35.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.961+0000 7f695d13d640 10 monclient: _renew_subs 2024-09-11T01:41:35.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.961+0000 7f695d13d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:35.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.961+0000 7f695d13d640 1 -- 172.21.15.12:0/1770182753 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6958127b60 con 0x7f69580a3270 2024-09-11T01:41:35.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.962+0000 7f694f7fe640 1 -- 172.21.15.12:0/1770182753 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f6940030c90 con 0x7f69580a3270 2024-09-11T01:41:35.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.962+0000 7f695d13d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:35.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.962+0000 7f695d13d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:35.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.962+0000 7f695d13d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:35.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.962+0000 7f695d13d640 1 -- 172.21.15.12:0/1770182753 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6958138c90 con 0x7f69580a3270 2024-09-11T01:41:35.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.962+0000 7f694f7fe640 1 --2- 172.21.15.12:0/1770182753 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f693803e1c0 0x7f6938040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:35.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.962+0000 7f694f7fe640 1 -- 172.21.15.12:0/1770182753 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(408..408 src has 285..408) ==== 4739+0+0 (secure 0 0 0) 0x7f6940073720 con 0x7f69580a3270 2024-09-11T01:41:35.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.962+0000 7f6956d9d640 1 --2- 172.21.15.12:0/1770182753 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f693803e1c0 0x7f6938040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:35.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.962+0000 7f6956d9d640 10 monclient: get_auth_request con 0x7f693803e1c0 auth_method 0 2024-09-11T01:41:35.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.965+0000 7f694f7fe640 1 -- 172.21.15.12:0/1770182753 DispatchQueue::entry inject delay of 0.002000 2024-09-11T01:41:35.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.965+0000 7f6956d9d640 1 --2- 172.21.15.12:0/1770182753 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f693803e1c0 0x7f6938040680 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6948003d00 tx=0x7f694800d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:35.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.967+0000 7f694f7fe640 1 -- 172.21.15.12:0/1770182753 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f694004c070 con 0x7f69580a3270 2024-09-11T01:41:35.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.967+0000 7f694f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:35.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:35.967+0000 7f694f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:41:36.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:36.222+0000 7f695d13d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:41:36.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:36.222+0000 7f695d13d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:41:36.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:36.222+0000 7f695d13d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:36.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:36.222+0000 7f695d13d640 1 -- 172.21.15.12:0/1770182753 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6958127fe0 con 0x7f69580a3270 2024-09-11T01:41:36.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:36.223+0000 7f694f7fe640 1 -- 172.21.15.12:0/1770182753 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f6940072140 con 0x7f69580a3270 2024-09-11T01:41:36.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:36.223+0000 7f694f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:41:36.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:36.223+0000 7f694f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:41:36.225 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:41:36.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:36.225+0000 7f695d13d640 1 -- 172.21.15.12:0/1770182753 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f693803e1c0 msgr2=0x7f6938040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:36.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:36.225+0000 7f695d13d640 1 --2- 172.21.15.12:0/1770182753 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f693803e1c0 0x7f6938040680 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6948003d00 tx=0x7f694800d040 comp rx=0 tx=0).stop 2024-09-11T01:41:36.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:36.225+0000 7f695d13d640 10 monclient: shutdown 2024-09-11T01:41:36.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:36.225+0000 7f695d13d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:36.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:36.225+0000 7f695d13d640 1 -- 172.21.15.12:0/1770182753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69580a3270 msgr2=0x7f69581cda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:36.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:36.225+0000 7f695d13d640 1 --2- 172.21.15.12:0/1770182753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69580a3270 0x7f69581cda20 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6940009890 tx=0x7f69400046c0 comp rx=0 tx=0).stop 2024-09-11T01:41:36.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:36.225+0000 7f695d13d640 1 -- 172.21.15.12:0/1770182753 shutdown_connections 2024-09-11T01:41:36.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:36.225+0000 7f695d13d640 1 --2- 172.21.15.12:0/1770182753 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6958138040 0x7f6958124910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:36.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:36.225+0000 7f695d13d640 1 --2- 172.21.15.12:0/1770182753 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f693803e1c0 0x7f6938040680 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:36.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:36.225+0000 7f695d13d640 1 --2- 172.21.15.12:0/1770182753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69580a3270 0x7f69581cda20 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:36.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:36.225+0000 7f695d13d640 1 --2- 172.21.15.12:0/1770182753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69580a2e90 0x7f69581cd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:36.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:36.225+0000 7f695d13d640 1 -- 172.21.15.12:0/1770182753 >> 172.21.15.12:0/1770182753 conn(0x7f69580a8490 msgr2=0x7f6958072090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:36.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:36.225+0000 7f695d13d640 1 -- 172.21.15.12:0/1770182753 shutdown_connections 2024-09-11T01:41:36.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:36.225+0000 7f695d13d640 1 -- 172.21.15.12:0/1770182753 wait complete. 2024-09-11T01:41:39.238 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:41:39.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.350+0000 7f84c61cd640 10 monclient: get_monmap_and_config 2024-09-11T01:41:39.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.350+0000 7f84c61cd640 10 monclient: build_initial_monmap 2024-09-11T01:41:39.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.350+0000 7f84c61cd640 10 monclient: monmap: 2024-09-11T01:41:39.351 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:39.352 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:39.352 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:41:39.350849+0000 2024-09-11T01:41:39.352 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:41:39.350849+0000 2024-09-11T01:41:39.352 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:39.352 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:39.352 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:41:39.352 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:41:39.352 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:41:39.352 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:41:39.352 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:41:39.352 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:39.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.350+0000 7f84c61cd640 1 Processor -- start 2024-09-11T01:41:39.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.350+0000 7f84c61cd640 1 -- start start 2024-09-11T01:41:39.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84c61cd640 10 monclient: init 2024-09-11T01:41:39.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84c61cd640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:39.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84c61cd640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-11T01:41:39.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84c61cd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f84c0138230 0x7f84c0138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:39.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84c61cd640 10 monclient(hunting): picked mon.noname-a con 0x7f84c0138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:41:39.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84c61cd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84c009d030 addr v1:172.21.15.12:6789/0 2024-09-11T01:41:39.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84c61cd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f84c009d460 0x7f84c00a5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:39.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84c61cd640 10 monclient(hunting): picked mon.noname-c con 0x7f84c009d460 addr v2:172.21.15.71:3300/0 2024-09-11T01:41:39.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84c61cd640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84c00a6e30 con 0x7f84c009d030 2024-09-11T01:41:39.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84c61cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:39.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84c61cd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f84c009c9c0 con 0x7f84c0138230 2024-09-11T01:41:39.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84c61cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:39.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84c61cd640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f84c009ba20 con 0x7f84c009d460 2024-09-11T01:41:39.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84c61cd640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:39.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84c61cd640 10 monclient(hunting): authenticate will time out at 1505.793617s 2024-09-11T01:41:39.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84bf7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f84c0138230 0x7f84c0138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:39.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84beffd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f84c009d030 0x7f84c00a5020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47230/0 (socket says 172.21.15.12:47230) 2024-09-11T01:41:39.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84beffd640 1 -- 172.21.15.12:0/1768481086 learned_addr learned my addr 172.21.15.12:0/1768481086 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:41:39.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84bf7fe640 10 monclient(hunting): get_auth_request con 0x7f84c0138230 auth_method 0 2024-09-11T01:41:39.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:39.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:39.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:39.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:39.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:39.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:39.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84bffff640 1 --2- 172.21.15.12:0/1768481086 >> v2:172.21.15.71:3300/0 conn(0x7f84c009d460 0x7f84c00a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:39.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84bffff640 10 monclient(hunting): get_auth_request con 0x7f84c009d460 auth_method 0 2024-09-11T01:41:39.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:39.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84bffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:39.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:39.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.351+0000 7f84bf7fe640 10 monclient(hunting): handle_auth_done global_id 94509 payload 274 2024-09-11T01:41:39.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.352+0000 7f84bf7fe640 1 -- 172.21.15.12:0/1768481086 >> v2:172.21.15.71:3300/0 conn(0x7f84c009d460 msgr2=0x7f84c00a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:39.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.352+0000 7f84bf7fe640 1 --2- 172.21.15.12:0/1768481086 >> v2:172.21.15.71:3300/0 conn(0x7f84c009d460 0x7f84c00a5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:39.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.352+0000 7f84bf7fe640 1 -- 172.21.15.12:0/1768481086 >> v1:172.21.15.12:6789/0 conn(0x7f84c009d030 legacy=0x7f84c00a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:39.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.352+0000 7f84bf7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:41:39.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.352+0000 7f84bf7fe640 1 monclient: found mon.noname-a 2024-09-11T01:41:39.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.352+0000 7f84bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:39.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.352+0000 7f84bf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:41:39.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.352+0000 7f84bf7fe640 1 -- 172.21.15.12:0/1768481086 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84c00a66a0 con 0x7f84c0138230 2024-09-11T01:41:39.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.354+0000 7f84bf7fe640 1 -- 172.21.15.12:0/1768481086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84c0138230 msgr2=0x7f84c0138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:39.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.354+0000 7f84bf7fe640 1 --2- 172.21.15.12:0/1768481086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84c0138230 0x7f84c0138610 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f84c009bff0 tx=0x7f84b002a4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=464c5a98e46f8849 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:39.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.355+0000 7f84be7fc640 1 -- 172.21.15.12:0/1768481086 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b0007950 con 0x7f84c0138230 2024-09-11T01:41:39.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.355+0000 7f84be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:39.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.355+0000 7f84be7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:41:39.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.355+0000 7f84be7fc640 10 monclient: dump: 2024-09-11T01:41:39.361 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:39.361 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:39.362 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:39.362 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:39.362 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:39.362 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:39.362 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:39.362 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:39.362 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:39.362 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:39.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.355+0000 7f84be7fc640 10 monclient: _finish_auth 0 2024-09-11T01:41:39.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.355+0000 7f84be7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:41:39.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.355+0000 7f84be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:39.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.355+0000 7f84be7fc640 1 -- 172.21.15.12:0/1768481086 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84b0007af0 con 0x7f84c0138230 2024-09-11T01:41:39.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.355+0000 7f84be7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:39.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.355+0000 7f84be7fc640 1 -- 172.21.15.12:0/1768481086 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b0031400 con 0x7f84c0138230 2024-09-11T01:41:39.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.355+0000 7f84c61cd640 5 monclient: authenticate success, global_id 94509 2024-09-11T01:41:39.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.355+0000 7f84c61cd640 10 monclient: get_monmap_and_config success 2024-09-11T01:41:39.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.355+0000 7f84be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:39.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.355+0000 7f84c61cd640 4 set_mon_vals no callback set 2024-09-11T01:41:39.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.355+0000 7f84be7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:39.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.355+0000 7f84be7fc640 10 monclient: dump: 2024-09-11T01:41:39.363 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:39.363 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:39.363 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:39.363 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:39.363 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:39.364 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:39.364 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:39.364 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:39.364 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:39.364 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:39.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.355+0000 7f84c61cd640 10 monclient: shutdown 2024-09-11T01:41:39.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.355+0000 7f84c61cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:39.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.355+0000 7f84c61cd640 1 -- 172.21.15.12:0/1768481086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84c0138230 msgr2=0x7f84c0138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:39.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.355+0000 7f84c61cd640 1 --2- 172.21.15.12:0/1768481086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84c0138230 0x7f84c0138610 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f84c009bff0 tx=0x7f84b002a4b0 comp rx=0 tx=0).stop 2024-09-11T01:41:39.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.355+0000 7f84c61cd640 1 -- 172.21.15.12:0/1768481086 shutdown_connections 2024-09-11T01:41:39.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.355+0000 7f84c61cd640 1 --2- 172.21.15.12:0/1768481086 >> v2:172.21.15.71:3300/0 conn(0x7f84c009d460 0x7f84c00a5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:39.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.355+0000 7f84c61cd640 1 --2- 172.21.15.12:0/1768481086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84c0138230 0x7f84c0138610 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:39.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.355+0000 7f84c61cd640 1 -- 172.21.15.12:0/1768481086 >> 172.21.15.12:0/1768481086 conn(0x7f84c00a7120 msgr2=0x7f84c00a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:39.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.355+0000 7f84c61cd640 1 -- 172.21.15.12:0/1768481086 shutdown_connections 2024-09-11T01:41:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.355+0000 7f84c61cd640 1 -- 172.21.15.12:0/1768481086 wait complete. 2024-09-11T01:41:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.356+0000 7f84c61cd640 10 monclient: build_initial_monmap 2024-09-11T01:41:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.356+0000 7f84c61cd640 10 monclient: monmap: 2024-09-11T01:41:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:41:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:41:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:41:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:41:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:41:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:41:39.365 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:39.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.356+0000 7f84c61cd640 1 Processor -- start 2024-09-11T01:41:39.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84c61cd640 1 -- start start 2024-09-11T01:41:39.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84c61cd640 10 monclient: init 2024-09-11T01:41:39.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84c61cd640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:39.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84c61cd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:41:39.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84c61cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84c009d030 0x7f84c0136800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:39.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84c61cd640 10 monclient(hunting): picked mon.noname-a con 0x7f84c009d030 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:41:39.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84c61cd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84c009d460 0x7f84c0136d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:39.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84c61cd640 10 monclient(hunting): picked mon.noname-b con 0x7f84c009d460 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:41:39.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84c61cd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f84c0138230 0x7f84c0120c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:39.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84c61cd640 10 monclient(hunting): picked mon.noname-c con 0x7f84c0138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:39.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84c61cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84c61cd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f84c006f970 con 0x7f84c009d030 2024-09-11T01:41:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84c61cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84c61cd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f84c006f7f0 con 0x7f84c009d460 2024-09-11T01:41:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84c61cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84c61cd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f84c00a5c70 con 0x7f84c0138230 2024-09-11T01:41:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84c61cd640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84c61cd640 10 monclient(hunting): authenticate will time out at 1505.799614s 2024-09-11T01:41:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84bffff640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f84c0138230 msgr2=0x7f84c0120c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:41:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84bffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f84c0138230 0x7f84c0120c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:41:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84bf7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84c009d030 0x7f84c0136800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84bf7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84c009d030 0x7f84c0136800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45110/0 (socket says 172.21.15.12:45110) 2024-09-11T01:41:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84bf7fe640 1 -- 172.21.15.12:0/2634530891 learned_addr learned my addr 172.21.15.12:0/2634530891 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:41:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84bf7fe640 10 monclient(hunting): get_auth_request con 0x7f84c009d030 auth_method 0 2024-09-11T01:41:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84beffd640 1 --2- 172.21.15.12:0/2634530891 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84c009d460 0x7f84c0136d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.357+0000 7f84bf7fe640 10 monclient(hunting): handle_auth_done global_id 94512 payload 824 2024-09-11T01:41:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.358+0000 7f84bf7fe640 1 -- 172.21.15.12:0/2634530891 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f84c0138230 msgr2=0x7f84c0120c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:41:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.358+0000 7f84bf7fe640 1 --2- 172.21.15.12:0/2634530891 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f84c0138230 0x7f84c0120c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.358+0000 7f84bf7fe640 1 -- 172.21.15.12:0/2634530891 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84c009d460 msgr2=0x7f84c0136d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.358+0000 7f84bf7fe640 1 --2- 172.21.15.12:0/2634530891 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84c009d460 0x7f84c0136d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.358+0000 7f84bf7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:41:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.358+0000 7f84bf7fe640 1 monclient: found mon.noname-a 2024-09-11T01:41:39.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.358+0000 7f84bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:39.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.358+0000 7f84bf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:41:39.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.358+0000 7f84bf7fe640 1 -- 172.21.15.12:0/2634530891 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84c00a1810 con 0x7f84c009d030 2024-09-11T01:41:39.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.358+0000 7f84beffd640 10 monclient: get_auth_request con 0x7f84c009d460 auth_method 0 2024-09-11T01:41:39.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.358+0000 7f84beffd640 1 --2- 172.21.15.12:0/2634530891 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84c009d460 0x7f84c0136d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:41:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.360+0000 7f84bf7fe640 1 -- 172.21.15.12:0/2634530891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84c009d030 msgr2=0x7f84c0136800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.360+0000 7f84bf7fe640 1 --2- 172.21.15.12:0/2634530891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84c009d030 0x7f84c0136800 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f84b0009e80 tx=0x7f84b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.360+0000 7f84a7fff640 1 -- 172.21.15.12:0/2634530891 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b0040070 con 0x7f84c009d030 2024-09-11T01:41:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.360+0000 7f84a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.360+0000 7f84a7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:41:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.360+0000 7f84a7fff640 10 monclient: dump: 2024-09-11T01:41:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:39.372 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.360+0000 7f84a7fff640 10 monclient: _finish_auth 0 2024-09-11T01:41:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.360+0000 7f84a7fff640 10 monclient: _check_auth_tickets 2024-09-11T01:41:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.360+0000 7f84a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.360+0000 7f84a7fff640 1 -- 172.21.15.12:0/2634530891 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84b0045660 con 0x7f84c009d030 2024-09-11T01:41:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.360+0000 7f84c61cd640 5 monclient: authenticate success, global_id 94512 2024-09-11T01:41:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.360+0000 7f84a7fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.360+0000 7f84c61cd640 10 monclient: _renew_subs 2024-09-11T01:41:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.360+0000 7f84a7fff640 1 -- 172.21.15.12:0/2634530891 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b0045a10 con 0x7f84c009d030 2024-09-11T01:41:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.360+0000 7f84c61cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.360+0000 7f84c61cd640 1 -- 172.21.15.12:0/2634530891 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84c01222f0 con 0x7f84c009d030 2024-09-11T01:41:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.360+0000 7f84a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.360+0000 7f84a7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.360+0000 7f84a7fff640 10 monclient: dump: 2024-09-11T01:41:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:39.374 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.360+0000 7f84bdffb640 4 set_mon_vals no callback set 2024-09-11T01:41:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.360+0000 7f84c61cd640 10 monclient: _renew_subs 2024-09-11T01:41:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.360+0000 7f84c61cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.360+0000 7f84c61cd640 1 -- 172.21.15.12:0/2634530891 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84c0137600 con 0x7f84c009d030 2024-09-11T01:41:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.361+0000 7f84a7fff640 1 -- 172.21.15.12:0/2634530891 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f84b0007960 con 0x7f84c009d030 2024-09-11T01:41:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.361+0000 7f84c61cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.361+0000 7f84c61cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.361+0000 7f84c61cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.361+0000 7f84c61cd640 1 -- 172.21.15.12:0/2634530891 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84c0138610 con 0x7f84c009d030 2024-09-11T01:41:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.361+0000 7f84a7fff640 1 --2- 172.21.15.12:0/2634530891 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f84ac03e1c0 0x7f84ac040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.361+0000 7f84a7fff640 1 -- 172.21.15.12:0/2634530891 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(408..408 src has 285..408) ==== 4739+0+0 (secure 0 0 0) 0x7f84b0074bd0 con 0x7f84c009d030 2024-09-11T01:41:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.362+0000 7f84beffd640 1 --2- 172.21.15.12:0/2634530891 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f84ac03e1c0 0x7f84ac040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.362+0000 7f84beffd640 10 monclient: get_auth_request con 0x7f84ac03e1c0 auth_method 0 2024-09-11T01:41:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.364+0000 7f84beffd640 1 --2- 172.21.15.12:0/2634530891 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f84ac03e1c0 0x7f84ac040680 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f84c00a4720 tx=0x7f84b4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.366+0000 7f84a7fff640 1 -- 172.21.15.12:0/2634530891 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f84c00a1810 con 0x7f84c009d030 2024-09-11T01:41:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.366+0000 7f84a7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.366+0000 7f84a7fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:41:39.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.621+0000 7f84c61cd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:41:39.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.621+0000 7f84c61cd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:41:39.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.621+0000 7f84c61cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:39.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.621+0000 7f84c61cd640 1 -- 172.21.15.12:0/2634530891 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f84c00698e0 con 0x7f84c009d030 2024-09-11T01:41:39.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.622+0000 7f84a7fff640 1 -- 172.21.15.12:0/2634530891 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7f84b00735f0 con 0x7f84c009d030 2024-09-11T01:41:39.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.622+0000 7f84a7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:41:39.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.622+0000 7f84a7fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:41:39.624 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:41:39.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.624+0000 7f84c61cd640 1 -- 172.21.15.12:0/2634530891 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f84ac03e1c0 msgr2=0x7f84ac040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:39.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.624+0000 7f84c61cd640 1 --2- 172.21.15.12:0/2634530891 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f84ac03e1c0 0x7f84ac040680 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f84c00a4720 tx=0x7f84b4009040 comp rx=0 tx=0).stop 2024-09-11T01:41:39.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.624+0000 7f84c61cd640 10 monclient: shutdown 2024-09-11T01:41:39.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.624+0000 7f84c61cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:39.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.624+0000 7f84c61cd640 1 -- 172.21.15.12:0/2634530891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84c009d030 msgr2=0x7f84c0136800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:39.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.624+0000 7f84c61cd640 1 --2- 172.21.15.12:0/2634530891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84c009d030 0x7f84c0136800 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f84b0009e80 tx=0x7f84b0004a40 comp rx=0 tx=0).stop 2024-09-11T01:41:39.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.624+0000 7f84c61cd640 1 -- 172.21.15.12:0/2634530891 shutdown_connections 2024-09-11T01:41:39.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.624+0000 7f84c61cd640 1 --2- 172.21.15.12:0/2634530891 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f84c0138230 0x7f84c0120c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:39.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.624+0000 7f84c61cd640 1 --2- 172.21.15.12:0/2634530891 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84c009d460 0x7f84c0136d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:39.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.624+0000 7f84c61cd640 1 --2- 172.21.15.12:0/2634530891 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f84ac03e1c0 0x7f84ac040680 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:39.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.624+0000 7f84c61cd640 1 --2- 172.21.15.12:0/2634530891 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f84c009d030 0x7f84c0136800 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:39.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.624+0000 7f84c61cd640 1 -- 172.21.15.12:0/2634530891 >> 172.21.15.12:0/2634530891 conn(0x7f84c00a7120 msgr2=0x7f84c00a3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:39.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.624+0000 7f84c61cd640 1 -- 172.21.15.12:0/2634530891 shutdown_connections 2024-09-11T01:41:39.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:39.624+0000 7f84c61cd640 1 -- 172.21.15.12:0/2634530891 wait complete. 2024-09-11T01:41:42.637 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:41:42.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.748+0000 7fa8d47ca640 10 monclient: get_monmap_and_config 2024-09-11T01:41:42.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.748+0000 7fa8d47ca640 10 monclient: build_initial_monmap 2024-09-11T01:41:42.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.748+0000 7fa8d47ca640 10 monclient: monmap: 2024-09-11T01:41:42.750 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:42.750 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:42.750 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:41:42.749501+0000 2024-09-11T01:41:42.750 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:41:42.749501+0000 2024-09-11T01:41:42.750 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:42.750 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:42.751 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:41:42.751 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:41:42.751 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:41:42.751 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:41:42.751 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:41:42.751 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:42.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.749+0000 7fa8d47ca640 1 Processor -- start 2024-09-11T01:41:42.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.749+0000 7fa8d47ca640 1 -- start start 2024-09-11T01:41:42.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.749+0000 7fa8d47ca640 10 monclient: init 2024-09-11T01:41:42.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d47ca640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:42.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d47ca640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-11T01:41:42.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d47ca640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa8cc0a2e90 0x7fa8cc06dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:42.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d47ca640 10 monclient(hunting): picked mon.noname-a con 0x7fa8cc0a2e90 addr v2:172.21.15.12:3300/0 2024-09-11T01:41:42.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d47ca640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa8cc0a3270 0x7fa8cc0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:42.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d47ca640 10 monclient(hunting): picked mon.noname-c con 0x7fa8cc0a3270 addr v2:172.21.15.71:3300/0 2024-09-11T01:41:42.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d47ca640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa8cc138040 addr v1:172.21.15.71:6789/0 2024-09-11T01:41:42.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d47ca640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8cc0a46c0 con 0x7fa8cc138040 2024-09-11T01:41:42.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d47ca640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:42.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d47ca640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fa8cc0a4900 con 0x7fa8cc0a2e90 2024-09-11T01:41:42.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d47ca640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:42.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d47ca640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fa8cc06ee50 con 0x7fa8cc0a3270 2024-09-11T01:41:42.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d47ca640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:42.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d47ca640 10 monclient(hunting): authenticate will time out at 1509.192196s 2024-09-11T01:41:42.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d2566640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa8cc0a2e90 0x7fa8cc06dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:42.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d2566640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa8cc0a2e90 0x7fa8cc06dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45122/0 (socket says 172.21.15.12:45122) 2024-09-11T01:41:42.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d2566640 1 -- 172.21.15.12:0/3434264283 learned_addr learned my addr 172.21.15.12:0/3434264283 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:41:42.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d2566640 10 monclient(hunting): get_auth_request con 0x7fa8cc0a2e90 auth_method 0 2024-09-11T01:41:42.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d2566640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:42.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d2566640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:42.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d2566640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:42.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d2566640 1 -- 172.21.15.12:0/3434264283 >> v2:172.21.15.12:3300/0 conn(0x7fa8cc0a2e90 msgr2=0x7fa8cc06dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:41:42.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d2566640 1 -- 172.21.15.12:0/3434264283 >> v2:172.21.15.12:3300/0 conn(0x7fa8cc0a2e90 msgr2=0x7fa8cc06dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:41:42.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d2566640 1 --2- 172.21.15.12:0/3434264283 >> v2:172.21.15.12:3300/0 conn(0x7fa8cc0a2e90 0x7fa8cc06dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:41:42.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d2566640 1 --2- 172.21.15.12:0/3434264283 >> v2:172.21.15.12:3300/0 conn(0x7fa8cc0a2e90 0x7fa8cc06dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:41:42.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d1d65640 1 --2- 172.21.15.12:0/3434264283 >> v2:172.21.15.71:3300/0 conn(0x7fa8cc0a3270 0x7fa8cc0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:42.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d1d65640 10 monclient(hunting): get_auth_request con 0x7fa8cc0a3270 auth_method 0 2024-09-11T01:41:42.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d1d65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:42.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d1d65640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:42.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.750+0000 7fa8d1d65640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:42.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.752+0000 7fa8d2d67640 1 -- 172.21.15.12:0/3434264283 >> v1:172.21.15.71:6789/0 conn(0x7fa8cc138040 legacy=0x7fa8cc1384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:42.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.752+0000 7fa8d1d65640 1 -- 172.21.15.12:0/3434264283 >> v2:172.21.15.71:3300/0 conn(0x7fa8cc0a3270 msgr2=0x7fa8cc0a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:41:42.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.752+0000 7fa8d1d65640 1 -- 172.21.15.12:0/3434264283 >> v2:172.21.15.71:3300/0 conn(0x7fa8cc0a3270 msgr2=0x7fa8cc0a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:41:42.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.752+0000 7fa8d1d65640 1 --2- 172.21.15.12:0/3434264283 >> v2:172.21.15.71:3300/0 conn(0x7fa8cc0a3270 0x7fa8cc0a3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:41:42.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.752+0000 7fa8d1d65640 1 --2- 172.21.15.12:0/3434264283 >> v2:172.21.15.71:3300/0 conn(0x7fa8cc0a3270 0x7fa8cc0a3680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:41:42.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.952+0000 7fa8d2566640 1 --2- 172.21.15.12:0/3434264283 >> v2:172.21.15.12:3300/0 conn(0x7fa8cc0a2e90 0x7fa8cc06dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:42.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.952+0000 7fa8d2566640 10 monclient(hunting): get_auth_request con 0x7fa8cc0a2e90 auth_method 0 2024-09-11T01:41:42.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.952+0000 7fa8d2566640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:42.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.952+0000 7fa8d2566640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:42.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.952+0000 7fa8d2566640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:41:42.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.952+0000 7fa8d2566640 1 -- 172.21.15.12:0/3434264283 >> v2:172.21.15.12:3300/0 conn(0x7fa8cc0a2e90 msgr2=0x7fa8cc06dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:41:42.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.952+0000 7fa8d2566640 1 -- 172.21.15.12:0/3434264283 >> v2:172.21.15.12:3300/0 conn(0x7fa8cc0a2e90 msgr2=0x7fa8cc06dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:41:42.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.952+0000 7fa8d2566640 1 --2- 172.21.15.12:0/3434264283 >> v2:172.21.15.12:3300/0 conn(0x7fa8cc0a2e90 0x7fa8cc06dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:41:42.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.952+0000 7fa8d2566640 1 --2- 172.21.15.12:0/3434264283 >> v2:172.21.15.12:3300/0 conn(0x7fa8cc0a2e90 0x7fa8cc06dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:41:42.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.953+0000 7fa8d1d65640 1 --2- 172.21.15.12:0/3434264283 >> v2:172.21.15.71:3300/0 conn(0x7fa8cc0a3270 0x7fa8cc0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:42.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.953+0000 7fa8d1d65640 10 monclient(hunting): get_auth_request con 0x7fa8cc0a3270 auth_method 0 2024-09-11T01:41:42.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.953+0000 7fa8d1d65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:42.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.953+0000 7fa8d1d65640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:42.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.953+0000 7fa8d1d65640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:41:42.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.953+0000 7fa8d1d65640 1 -- 172.21.15.12:0/3434264283 >> v2:172.21.15.71:3300/0 conn(0x7fa8cc0a3270 msgr2=0x7fa8cc0a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:41:42.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.953+0000 7fa8d1d65640 1 -- 172.21.15.12:0/3434264283 >> v2:172.21.15.71:3300/0 conn(0x7fa8cc0a3270 msgr2=0x7fa8cc0a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:41:42.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.953+0000 7fa8d1d65640 1 --2- 172.21.15.12:0/3434264283 >> v2:172.21.15.71:3300/0 conn(0x7fa8cc0a3270 0x7fa8cc0a3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:41:42.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:42.953+0000 7fa8d1d65640 1 --2- 172.21.15.12:0/3434264283 >> v2:172.21.15.71:3300/0 conn(0x7fa8cc0a3270 0x7fa8cc0a3680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:41:43.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:43.353+0000 7fa8d2566640 1 --2- 172.21.15.12:0/3434264283 >> v2:172.21.15.12:3300/0 conn(0x7fa8cc0a2e90 0x7fa8cc06dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:43.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:43.353+0000 7fa8d2566640 10 monclient(hunting): get_auth_request con 0x7fa8cc0a2e90 auth_method 0 2024-09-11T01:41:43.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:43.353+0000 7fa8d2566640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:43.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:43.353+0000 7fa8d2566640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:43.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:43.353+0000 7fa8d2566640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:41:43.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:43.353+0000 7fa8d2566640 1 -- 172.21.15.12:0/3434264283 >> v2:172.21.15.12:3300/0 conn(0x7fa8cc0a2e90 msgr2=0x7fa8cc06dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:41:43.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:43.353+0000 7fa8d2566640 1 -- 172.21.15.12:0/3434264283 >> v2:172.21.15.12:3300/0 conn(0x7fa8cc0a2e90 msgr2=0x7fa8cc06dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:41:43.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:43.353+0000 7fa8d2566640 1 --2- 172.21.15.12:0/3434264283 >> v2:172.21.15.12:3300/0 conn(0x7fa8cc0a2e90 0x7fa8cc06dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:41:43.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:43.353+0000 7fa8d2566640 1 --2- 172.21.15.12:0/3434264283 >> v2:172.21.15.12:3300/0 conn(0x7fa8cc0a2e90 0x7fa8cc06dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:41:43.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:43.354+0000 7fa8d1d65640 1 --2- 172.21.15.12:0/3434264283 >> v2:172.21.15.71:3300/0 conn(0x7fa8cc0a3270 0x7fa8cc0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:43.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:43.354+0000 7fa8d1d65640 10 monclient(hunting): get_auth_request con 0x7fa8cc0a3270 auth_method 0 2024-09-11T01:41:43.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:43.354+0000 7fa8d1d65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:43.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:43.354+0000 7fa8d1d65640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:43.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:43.354+0000 7fa8d1d65640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:41:43.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:43.354+0000 7fa8d1d65640 1 -- 172.21.15.12:0/3434264283 >> v2:172.21.15.71:3300/0 conn(0x7fa8cc0a3270 msgr2=0x7fa8cc0a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:41:43.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:43.354+0000 7fa8d1d65640 1 -- 172.21.15.12:0/3434264283 >> v2:172.21.15.71:3300/0 conn(0x7fa8cc0a3270 msgr2=0x7fa8cc0a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:41:43.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:43.354+0000 7fa8d1d65640 1 --2- 172.21.15.12:0/3434264283 >> v2:172.21.15.71:3300/0 conn(0x7fa8cc0a3270 0x7fa8cc0a3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:41:43.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:43.354+0000 7fa8d1d65640 1 --2- 172.21.15.12:0/3434264283 >> v2:172.21.15.71:3300/0 conn(0x7fa8cc0a3270 0x7fa8cc0a3680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:41:44.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:44.154+0000 7fa8d2566640 1 --2- 172.21.15.12:0/3434264283 >> v2:172.21.15.12:3300/0 conn(0x7fa8cc0a2e90 0x7fa8cc06dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:44.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:44.154+0000 7fa8d2566640 10 monclient(hunting): get_auth_request con 0x7fa8cc0a2e90 auth_method 0 2024-09-11T01:41:44.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:44.154+0000 7fa8d2566640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:44.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:44.154+0000 7fa8d2566640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:44.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:44.154+0000 7fa8d2566640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:41:44.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:44.154+0000 7fa8d2566640 1 -- 172.21.15.12:0/3434264283 >> v2:172.21.15.12:3300/0 conn(0x7fa8cc0a2e90 msgr2=0x7fa8cc06dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:41:44.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:44.154+0000 7fa8d2566640 1 -- 172.21.15.12:0/3434264283 >> v2:172.21.15.12:3300/0 conn(0x7fa8cc0a2e90 msgr2=0x7fa8cc06dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:41:44.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:44.154+0000 7fa8d2566640 1 --2- 172.21.15.12:0/3434264283 >> v2:172.21.15.12:3300/0 conn(0x7fa8cc0a2e90 0x7fa8cc06dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:41:44.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:44.154+0000 7fa8d2566640 1 --2- 172.21.15.12:0/3434264283 >> v2:172.21.15.12:3300/0 conn(0x7fa8cc0a2e90 0x7fa8cc06dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:41:44.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:44.155+0000 7fa8d1d65640 1 --2- 172.21.15.12:0/3434264283 >> v2:172.21.15.71:3300/0 conn(0x7fa8cc0a3270 0x7fa8cc0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:44.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:44.155+0000 7fa8d1d65640 10 monclient(hunting): get_auth_request con 0x7fa8cc0a3270 auth_method 0 2024-09-11T01:41:44.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:44.155+0000 7fa8d1d65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:44.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:44.155+0000 7fa8d1d65640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:44.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:44.155+0000 7fa8d1d65640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:41:44.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:44.155+0000 7fa8d1d65640 1 -- 172.21.15.12:0/3434264283 >> v2:172.21.15.71:3300/0 conn(0x7fa8cc0a3270 msgr2=0x7fa8cc0a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:41:44.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:44.155+0000 7fa8d1d65640 1 -- 172.21.15.12:0/3434264283 >> v2:172.21.15.71:3300/0 conn(0x7fa8cc0a3270 msgr2=0x7fa8cc0a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:41:44.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:44.155+0000 7fa8d1d65640 1 --2- 172.21.15.12:0/3434264283 >> v2:172.21.15.71:3300/0 conn(0x7fa8cc0a3270 0x7fa8cc0a3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:41:44.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:44.155+0000 7fa8d1d65640 1 --2- 172.21.15.12:0/3434264283 >> v2:172.21.15.71:3300/0 conn(0x7fa8cc0a3270 0x7fa8cc0a3680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:41:45.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:45.750+0000 7fa8c3fff640 10 monclient(hunting): tick 2024-09-11T01:41:45.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:45.750+0000 7fa8c3fff640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:41:45.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:45.750+0000 7fa8c3fff640 1 monclient(hunting): continuing hunt 2024-09-11T01:41:45.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:45.750+0000 7fa8c3fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:41:45.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:45.750+0000 7fa8c3fff640 1 -- 172.21.15.12:0/3434264283 >> v2:172.21.15.71:3300/0 conn(0x7fa8cc0a3270 msgr2=0x7fa8cc0a3680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:41:45.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:45.750+0000 7fa8c3fff640 1 --2- 172.21.15.12:0/3434264283 >> v2:172.21.15.71:3300/0 conn(0x7fa8cc0a3270 0x7fa8cc0a3680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:45.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:45.750+0000 7fa8c3fff640 1 -- 172.21.15.12:0/3434264283 >> v2:172.21.15.12:3300/0 conn(0x7fa8cc0a2e90 msgr2=0x7fa8cc06dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:41:45.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:45.750+0000 7fa8c3fff640 1 --2- 172.21.15.12:0/3434264283 >> v2:172.21.15.12:3300/0 conn(0x7fa8cc0a2e90 0x7fa8cc06dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:45.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:45.750+0000 7fa8c3fff640 1 -- 172.21.15.12:0/3434264283 >> v1:172.21.15.71:6789/0 conn(0x7fa8cc138040 legacy=0x7fa8cc1384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:45.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:45.750+0000 7fa8c3fff640 10 monclient: _add_conns ranks=[0,2] 2024-09-11T01:41:45.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:45.750+0000 7fa8c3fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa8b0003730 addr v1:172.21.15.12:6789/0 2024-09-11T01:41:45.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:45.750+0000 7fa8c3fff640 1 --2- 172.21.15.12:0/3434264283 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8b00073e0 0x7fa8b0009880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:45.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:45.750+0000 7fa8c3fff640 10 monclient(hunting): picked mon.noname-b con 0x7fa8b00073e0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:45.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:45.750+0000 7fa8c3fff640 1 -- 172.21.15.12:0/3434264283 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8cc06eb70 con 0x7fa8b0003730 2024-09-11T01:41:45.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:45.750+0000 7fa8c3fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:45.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:45.750+0000 7fa8c3fff640 1 -- 172.21.15.12:0/3434264283 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa8cc0a4900 con 0x7fa8b00073e0 2024-09-11T01:41:45.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:45.750+0000 7fa8c3fff640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:45.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:45.750+0000 7fa8d1d65640 1 -- 172.21.15.12:0/3434264283 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8b00073e0 msgr2=0x7fa8b0009880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:41:45.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:45.750+0000 7fa8d1d65640 1 --2- 172.21.15.12:0/3434264283 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8b00073e0 0x7fa8b0009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:41:45.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:45.752+0000 7fa8d2566640 1 -- 172.21.15.12:0/3434264283 >> v1:172.21.15.12:6789/0 conn(0x7fa8b0003730 legacy=0x7fa8b0005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:45.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:45.952+0000 7fa8d1d65640 1 -- 172.21.15.12:0/3434264283 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8b00073e0 msgr2=0x7fa8b0009880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:41:45.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:45.952+0000 7fa8d1d65640 1 --2- 172.21.15.12:0/3434264283 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8b00073e0 0x7fa8b0009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:41:46.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:46.353+0000 7fa8d1d65640 1 -- 172.21.15.12:0/3434264283 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8b00073e0 msgr2=0x7fa8b0009880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:41:46.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:46.353+0000 7fa8d1d65640 1 --2- 172.21.15.12:0/3434264283 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8b00073e0 0x7fa8b0009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:41:47.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.154+0000 7fa8d1d65640 1 -- 172.21.15.12:0/3434264283 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8b00073e0 msgr2=0x7fa8b0009880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:41:47.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.154+0000 7fa8d1d65640 1 --2- 172.21.15.12:0/3434264283 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8b00073e0 0x7fa8b0009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:41:47.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.353+0000 7fa8d1564640 1 -- 172.21.15.12:0/3434264283 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1793749552 0 0) 0x7fa8cc06eb70 con 0x7fa8b0003730 2024-09-11T01:41:47.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.353+0000 7fa8d1564640 10 monclient(hunting): _negotiate 2024-09-11T01:41:47.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.353+0000 7fa8d1564640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:47.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.353+0000 7fa8d1564640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:47.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.353+0000 7fa8d1564640 10 monclient(hunting): my global_id is 94515 2024-09-11T01:41:47.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.353+0000 7fa8d1564640 1 -- 172.21.15.12:0/3434264283 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8b8003520 con 0x7fa8b0003730 2024-09-11T01:41:47.355 INFO:tasks.rados.rados.0.smithi071.stdout:1778: done (1 left) 2024-09-11T01:41:47.355 INFO:tasks.rados.rados.0.smithi071.stdout:1779: read oid 10 snap -1 2024-09-11T01:41:47.356 INFO:tasks.rados.rados.0.smithi071.stdout:1779: expect (ObjNum 601 snap 172 seq_num 601) 2024-09-11T01:41:47.356 INFO:tasks.rados.rados.0.smithi071.stdout:1780: write (excl) oid 25 current snap is 199 2024-09-11T01:41:47.356 INFO:tasks.rados.rados.0.smithi071.stdout:1780: seq_num 651 ranges {541357=477104,1745307=594552} 2024-09-11T01:41:47.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 1 -- 172.21.15.12:0/3434264283 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2821955066 0 0) 0x7fa8b8003520 con 0x7fa8b0003730 2024-09-11T01:41:47.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 1 -- 172.21.15.12:0/3434264283 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8b00073e0 msgr2=0x7fa8b0009880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:41:47.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 1 --2- 172.21.15.12:0/3434264283 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8b00073e0 0x7fa8b0009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 10 monclient: _finish_hunting 0 2024-09-11T01:41:47.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:41:47.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:47.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:41:47.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 1 -- 172.21.15.12:0/3434264283 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8b000b0c0 con 0x7fa8b0003730 2024-09-11T01:41:47.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 10 monclient: _finish_auth 0 2024-09-11T01:41:47.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 10 monclient: _check_auth_tickets 2024-09-11T01:41:47.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:47.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 1 -- 172.21.15.12:0/3434264283 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7fa8c800f040 con 0x7fa8b0003730 2024-09-11T01:41:47.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:47.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:41:47.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 10 monclient: dump: 2024-09-11T01:41:47.359 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:47.359 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:47.359 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.359 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.359 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:47.359 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:47.359 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:47.359 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:47.359 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:47.359 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:47.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:41:47.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:47.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 1 -- 172.21.15.12:0/3434264283 >> v1:172.21.15.12:6789/0 conn(0x7fa8b0003730 legacy=0x7fa8b0005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:47.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:41:47.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d2d67640 1 -- 172.21.15.12:0/3434264283 reap_dead start 2024-09-11T01:41:47.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 1 --2- 172.21.15.12:0/3434264283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8b80039b0 0x7fa8b8003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:47.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 10 monclient(hunting): picked mon.a con 0x7fa8b80039b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:41:47.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 1 --2- 172.21.15.12:0/3434264283 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8b8004800 0x7fa8b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:47.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 10 monclient(hunting): picked mon.c con 0x7fa8b8004800 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:47.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 1 --2- 172.21.15.12:0/3434264283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8b80052e0 0x7fa8b800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:47.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 10 monclient(hunting): picked mon.b con 0x7fa8b80052e0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:41:47.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:47.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 1 -- 172.21.15.12:0/3434264283 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa8cc0a4900 con 0x7fa8b80039b0 2024-09-11T01:41:47.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:47.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 1 -- 172.21.15.12:0/3434264283 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b800c1e0 con 0x7fa8b80052e0 2024-09-11T01:41:47.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:47.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 1 -- 172.21.15.12:0/3434264283 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa8b800c3e0 con 0x7fa8b8004800 2024-09-11T01:41:47.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:47.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1564640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:41:47.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1d65640 1 -- 172.21.15.12:0/3434264283 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8b8004800 msgr2=0x7fa8b8004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:41:47.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d1d65640 1 --2- 172.21.15.12:0/3434264283 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8b8004800 0x7fa8b8004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:41:47.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d2566640 1 --2- 172.21.15.12:0/3434264283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8b80039b0 0x7fa8b8003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:47.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d2d67640 1 --2- 172.21.15.12:0/3434264283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8b80052e0 0x7fa8b800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:47.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d2566640 10 monclient(hunting): get_auth_request con 0x7fa8b80039b0 auth_method 0 2024-09-11T01:41:47.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d2566640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:47.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d2566640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:47.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d2566640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:41:47.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d2d67640 10 monclient(hunting): get_auth_request con 0x7fa8b80052e0 auth_method 0 2024-09-11T01:41:47.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d2d67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:47.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d2d67640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:47.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d2d67640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:41:47.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d2566640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:47.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d2566640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:47.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d2566640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:41:47.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d2d67640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:47.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d2d67640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:47.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.355+0000 7fa8d2d67640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:41:47.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.356+0000 7fa8d2566640 10 monclient(hunting): handle_auth_done global_id 94515 payload 293 2024-09-11T01:41:47.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.356+0000 7fa8d2566640 1 -- 172.21.15.12:0/3434264283 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8b8004800 msgr2=0x7fa8b8004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:41:47.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.356+0000 7fa8d2566640 1 --2- 172.21.15.12:0/3434264283 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8b8004800 0x7fa8b8004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.356+0000 7fa8d2566640 1 -- 172.21.15.12:0/3434264283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8b80052e0 msgr2=0x7fa8b800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:47.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.356+0000 7fa8d2566640 1 --2- 172.21.15.12:0/3434264283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8b80052e0 0x7fa8b800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.356+0000 7fa8d2566640 10 monclient: _finish_hunting 0 2024-09-11T01:41:47.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.356+0000 7fa8d2566640 1 monclient: found mon.a 2024-09-11T01:41:47.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.356+0000 7fa8d2566640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:47.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.356+0000 7fa8d2566640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:47.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.356+0000 7fa8d2566640 1 -- 172.21.15.12:0/3434264283 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8b000b0c0 con 0x7fa8b80039b0 2024-09-11T01:41:47.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.356+0000 7fa8d2566640 10 monclient: _finish_auth 0 2024-09-11T01:41:47.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.356+0000 7fa8d2566640 10 monclient: _check_auth_tickets 2024-09-11T01:41:47.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.356+0000 7fa8d2566640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:47.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.356+0000 7fa8d47ca640 5 monclient: authenticate success, global_id 94515 2024-09-11T01:41:47.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.356+0000 7fa8d47ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:41:47.364 INFO:tasks.rados.rados.0.smithi071.stdout:1780: writing smithi07134777-25 from 541357 to 1018461 tid 1 2024-09-11T01:41:47.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.359+0000 7fa8d2566640 1 -- 172.21.15.12:0/3434264283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8b80039b0 msgr2=0x7fa8b8003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:47.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.359+0000 7fa8d2566640 1 --2- 172.21.15.12:0/3434264283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8b80039b0 0x7fa8b8003db0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0007350 tx=0x7fa8b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86616c1bb27da7d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:47.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.359+0000 7fa8d1564640 1 -- 172.21.15.12:0/3434264283 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c8039070 con 0x7fa8b80039b0 2024-09-11T01:41:47.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.360+0000 7fa8d1564640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:47.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.360+0000 7fa8d1564640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:47.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.360+0000 7fa8d1564640 10 monclient: dump: 2024-09-11T01:41:47.369 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:47.369 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:47.369 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.369 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.369 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:47.369 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:47.369 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:47.369 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:47.370 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:47.370 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:47.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.360+0000 7fa8d1564640 1 -- 172.21.15.12:0/3434264283 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8c8034410 con 0x7fa8b80039b0 2024-09-11T01:41:47.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.360+0000 7fa8d1564640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:47.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.360+0000 7fa8d47ca640 10 monclient: get_monmap_and_config success 2024-09-11T01:41:47.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.360+0000 7fa8d1564640 1 -- 172.21.15.12:0/3434264283 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c800ce10 con 0x7fa8b80039b0 2024-09-11T01:41:47.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.360+0000 7fa8d1564640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:47.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.360+0000 7fa8d47ca640 4 set_mon_vals no callback set 2024-09-11T01:41:47.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.360+0000 7fa8d1564640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:47.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.360+0000 7fa8d1564640 10 monclient: dump: 2024-09-11T01:41:47.370 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:47.370 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:47.370 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.370 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.370 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:47.371 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:47.371 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:47.371 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:47.371 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:47.371 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:47.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.360+0000 7fa8d47ca640 10 monclient: shutdown 2024-09-11T01:41:47.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.360+0000 7fa8d47ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:47.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.360+0000 7fa8d47ca640 1 -- 172.21.15.12:0/3434264283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8b80039b0 msgr2=0x7fa8b8003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:47.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.360+0000 7fa8d47ca640 1 --2- 172.21.15.12:0/3434264283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8b80039b0 0x7fa8b8003db0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0007350 tx=0x7fa8b8005830 comp rx=0 tx=0).stop 2024-09-11T01:41:47.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.360+0000 7fa8d47ca640 1 -- 172.21.15.12:0/3434264283 shutdown_connections 2024-09-11T01:41:47.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.361+0000 7fa8d47ca640 1 --2- 172.21.15.12:0/3434264283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8b80052e0 0x7fa8b800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.361+0000 7fa8d47ca640 1 --2- 172.21.15.12:0/3434264283 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8b8004800 0x7fa8b8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.361+0000 7fa8d47ca640 1 --2- 172.21.15.12:0/3434264283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8b80039b0 0x7fa8b8003db0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.361+0000 7fa8d47ca640 1 -- 172.21.15.12:0/3434264283 >> 172.21.15.12:0/3434264283 conn(0x7fa8cc0a8490 msgr2=0x7fa8cc0a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:47.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.361+0000 7fa8d47ca640 1 -- 172.21.15.12:0/3434264283 shutdown_connections 2024-09-11T01:41:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.361+0000 7fa8d47ca640 1 -- 172.21.15.12:0/3434264283 wait complete. 2024-09-11T01:41:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.362+0000 7fa8d47ca640 10 monclient: build_initial_monmap 2024-09-11T01:41:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.362+0000 7fa8d47ca640 10 monclient: monmap: 2024-09-11T01:41:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:41:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:41:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:41:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:41:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:41:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:41:47.372 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:47.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.362+0000 7fa8d47ca640 1 Processor -- start 2024-09-11T01:41:47.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.362+0000 7fa8d47ca640 1 -- start start 2024-09-11T01:41:47.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.362+0000 7fa8d47ca640 10 monclient: init 2024-09-11T01:41:47.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d47ca640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:47.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d47ca640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:41:47.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d47ca640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8b80039b0 0x7fa8cc1cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:47.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d47ca640 10 monclient(hunting): picked mon.noname-a con 0x7fa8b80039b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:41:47.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d47ca640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8b8004800 0x7fa8cc137190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:47.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d47ca640 10 monclient(hunting): picked mon.noname-b con 0x7fa8b8004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:41:47.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d47ca640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8b80052e0 0x7fa8cc1376d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:47.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d47ca640 10 monclient(hunting): picked mon.noname-c con 0x7fa8b80052e0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:47.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d47ca640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:47.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d47ca640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa8cc06dde0 con 0x7fa8b80039b0 2024-09-11T01:41:47.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d47ca640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:47.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d2d67640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8b80052e0 msgr2=0x7fa8cc1376d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:41:47.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d47ca640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa8cc06dc60 con 0x7fa8b8004800 2024-09-11T01:41:47.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d47ca640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:47.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d47ca640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa8cc06df60 con 0x7fa8b80052e0 2024-09-11T01:41:47.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d47ca640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:47.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d2d67640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8b80052e0 0x7fa8cc1376d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:41:47.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d47ca640 10 monclient(hunting): authenticate will time out at 1513.805682s 2024-09-11T01:41:47.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d2566640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8b80039b0 0x7fa8cc1cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:47.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d2566640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8b80039b0 0x7fa8cc1cc9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48224/0 (socket says 172.21.15.12:48224) 2024-09-11T01:41:47.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d2566640 1 -- 172.21.15.12:0/2226271580 learned_addr learned my addr 172.21.15.12:0/2226271580 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:41:47.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d2566640 10 monclient(hunting): get_auth_request con 0x7fa8b80039b0 auth_method 0 2024-09-11T01:41:47.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d2566640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:47.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d2566640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:47.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d2566640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:47.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d1d65640 1 --2- 172.21.15.12:0/2226271580 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8b8004800 0x7fa8cc137190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:47.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d2566640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:47.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d2566640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:47.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d2566640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:47.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d1d65640 10 monclient(hunting): get_auth_request con 0x7fa8b8004800 auth_method 0 2024-09-11T01:41:47.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d1d65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:47.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d1d65640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:47.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.363+0000 7fa8d1d65640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:47.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.364+0000 7fa8d1d65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:47.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.364+0000 7fa8d1d65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:47.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.364+0000 7fa8d1d65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:47.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.364+0000 7fa8d2566640 10 monclient(hunting): handle_auth_done global_id 94518 payload 824 2024-09-11T01:41:47.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.364+0000 7fa8d2566640 1 -- 172.21.15.12:0/2226271580 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8b80052e0 msgr2=0x7fa8cc1376d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:41:47.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.364+0000 7fa8d2566640 1 --2- 172.21.15.12:0/2226271580 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8b80052e0 0x7fa8cc1376d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.364+0000 7fa8d2566640 1 -- 172.21.15.12:0/2226271580 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8b8004800 msgr2=0x7fa8cc137190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:47.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.364+0000 7fa8d2566640 1 --2- 172.21.15.12:0/2226271580 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8b8004800 0x7fa8cc137190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.364+0000 7fa8d2566640 10 monclient: _finish_hunting 0 2024-09-11T01:41:47.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.364+0000 7fa8d2566640 1 monclient: found mon.noname-a 2024-09-11T01:41:47.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.364+0000 7fa8d2566640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:47.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.364+0000 7fa8d2566640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:41:47.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.364+0000 7fa8d2566640 1 -- 172.21.15.12:0/2226271580 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8cc09ed30 con 0x7fa8b80039b0 2024-09-11T01:41:47.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.366+0000 7fa8d2566640 1 -- 172.21.15.12:0/2226271580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8b80039b0 msgr2=0x7fa8cc1cc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:47.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.366+0000 7fa8d2566640 1 --2- 172.21.15.12:0/2226271580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8b80039b0 0x7fa8cc1cc9f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa8c800c300 tx=0x7fa8c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:47.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.366+0000 7fa8c2ffd640 1 -- 172.21.15.12:0/2226271580 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c8042020 con 0x7fa8b80039b0 2024-09-11T01:41:47.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.366+0000 7fa8c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:47.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.366+0000 7fa8c2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:41:47.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.366+0000 7fa8c2ffd640 10 monclient: dump: 2024-09-11T01:41:47.376 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:47.376 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:47.376 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.376 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.376 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:47.377 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:47.377 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:47.377 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:47.377 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:47.377 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:47.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.366+0000 7fa8c2ffd640 10 monclient: _finish_auth 0 2024-09-11T01:41:47.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.366+0000 7fa8c2ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:41:47.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.366+0000 7fa8c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:47.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.366+0000 7fa8c2ffd640 1 -- 172.21.15.12:0/2226271580 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8c803a040 con 0x7fa8b80039b0 2024-09-11T01:41:47.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.366+0000 7fa8d47ca640 5 monclient: authenticate success, global_id 94518 2024-09-11T01:41:47.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.366+0000 7fa8c2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:47.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.366+0000 7fa8d47ca640 10 monclient: _renew_subs 2024-09-11T01:41:47.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.366+0000 7fa8c2ffd640 1 -- 172.21.15.12:0/2226271580 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c8039070 con 0x7fa8b80039b0 2024-09-11T01:41:47.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.366+0000 7fa8d47ca640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:47.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.367+0000 7fa8d47ca640 1 -- 172.21.15.12:0/2226271580 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8cc0732a0 con 0x7fa8b80039b0 2024-09-11T01:41:47.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.367+0000 7fa8c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:47.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.367+0000 7fa8c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:47.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.367+0000 7fa8c2ffd640 10 monclient: dump: 2024-09-11T01:41:47.378 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:47.378 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:47.378 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.378 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.378 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:47.378 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:47.378 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:47.378 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:47.378 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:47.379 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:47.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.366+0000 7fa8d0d63640 4 set_mon_vals no callback set 2024-09-11T01:41:47.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.367+0000 7fa8d47ca640 10 monclient: _renew_subs 2024-09-11T01:41:47.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.367+0000 7fa8d47ca640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:47.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.367+0000 7fa8d47ca640 1 -- 172.21.15.12:0/2226271580 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8cc124b50 con 0x7fa8b80039b0 2024-09-11T01:41:47.381 INFO:tasks.rados.rados.0.smithi071.stdout:1780: writing smithi07134777-25 from 1745307 to 2339859 tid 2 2024-09-11T01:41:47.381 INFO:tasks.rados.rados.0.smithi071.stdout:1781: copy_from oid 43 from oid 3 current snap is 199 2024-09-11T01:41:47.381 INFO:tasks.rados.rados.0.smithi071.stdout:1776: finishing write tid 3 to smithi07134777-46 2024-09-11T01:41:47.381 INFO:tasks.rados.rados.0.smithi071.stdout:1776: finishing write tid 4 to smithi07134777-46 2024-09-11T01:41:47.381 INFO:tasks.rados.rados.0.smithi071.stdout:1776: finishing write tid 5 to smithi07134777-46 2024-09-11T01:41:47.381 INFO:tasks.rados.rados.0.smithi071.stdout:1776: oid 46 updating version 0 to 704 2024-09-11T01:41:47.381 INFO:tasks.rados.rados.0.smithi071.stdout:1776: oid 46 version 704 is already newer than 701 2024-09-11T01:41:47.381 INFO:tasks.rados.rados.0.smithi071.stdout:1776: oid 46 version 704 is already newer than 703 2024-09-11T01:41:47.382 INFO:tasks.rados.rados.0.smithi071.stdout:1776: oid 46 version 704 is already newer than 702 2024-09-11T01:41:47.382 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 704 (ObjNum 649 snap 199 seq_num 649) dirty exists 2024-09-11T01:41:47.382 INFO:tasks.rados.rados.0.smithi071.stdout:1776: left oid 46 (ObjNum 649 snap 199 seq_num 649) 2024-09-11T01:41:47.382 INFO:tasks.rados.rados.0.smithi071.stdout:1776: done (3 left) 2024-09-11T01:41:47.382 INFO:tasks.rados.rados.0.smithi071.stdout:1782: copy_from oid 11 from oid 28 current snap is 199 2024-09-11T01:41:47.382 INFO:tasks.rados.rados.0.smithi071.stdout:1779: expect (ObjNum 601 snap 172 seq_num 601) 2024-09-11T01:41:47.382 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 291008 2024-09-11T01:41:47.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.368+0000 7fa8c2ffd640 1 -- 172.21.15.12:0/2226271580 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fa8c8015040 con 0x7fa8b80039b0 2024-09-11T01:41:47.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.368+0000 7fa8c2ffd640 1 --2- 172.21.15.12:0/2226271580 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa8b003f030 0x7fa8b00414f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:47.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.368+0000 7fa8d47ca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:47.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.368+0000 7fa8c2ffd640 1 -- 172.21.15.12:0/2226271580 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(409..409 src has 285..409) ==== 4783+0+0 (secure 0 0 0) 0x7fa8c8073800 con 0x7fa8b80039b0 2024-09-11T01:41:47.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.368+0000 7fa8d47ca640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:47.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.368+0000 7fa8d47ca640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:47.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.368+0000 7fa8d47ca640 1 -- 172.21.15.12:0/2226271580 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8cc0a3680 con 0x7fa8b80039b0 2024-09-11T01:41:47.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.369+0000 7fa8d1d65640 1 --2- 172.21.15.12:0/2226271580 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa8b003f030 0x7fa8b00414f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:47.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.369+0000 7fa8d1d65640 10 monclient: get_auth_request con 0x7fa8b003f030 auth_method 0 2024-09-11T01:41:47.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.371+0000 7fa8d1d65640 1 --2- 172.21.15.12:0/2226271580 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa8b003f030 0x7fa8b00414f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc0a2a40 tx=0x7fa8bc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:47.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.376+0000 7fa8c2ffd640 1 -- 172.21.15.12:0/2226271580 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa8c804c030 con 0x7fa8b80039b0 2024-09-11T01:41:47.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.376+0000 7fa8c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:47.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.376+0000 7fa8c2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:41:47.405 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1597099 2024-09-11T01:41:47.438 INFO:tasks.rados.rados.0.smithi071.stdout:1780: finishing write tid 1 to smithi07134777-25 2024-09-11T01:41:47.438 INFO:tasks.rados.rados.0.smithi071.stdout:1781: finishing copy_from to smithi07134777-43 2024-09-11T01:41:47.438 INFO:tasks.rados.rados.0.smithi071.stdout:1781: got expected ENOENT (src dne) 2024-09-11T01:41:47.438 INFO:tasks.rados.rados.0.smithi071.stdout:1781: finishing copy_from racing read to smithi07134777-43 2024-09-11T01:41:47.438 INFO:tasks.rados.rados.0.smithi071.stdout:1780: finishing write tid 2 to smithi07134777-25 2024-09-11T01:41:47.438 INFO:tasks.rados.rados.0.smithi071.stdout:1780: finishing write tid 3 to smithi07134777-25 2024-09-11T01:41:47.438 INFO:tasks.rados.rados.0.smithi071.stdout:1780: finishing write tid 4 to smithi07134777-25 2024-09-11T01:41:47.438 INFO:tasks.rados.rados.0.smithi071.stdout:1780: oid 25 updating version 0 to 535 2024-09-11T01:41:47.438 INFO:tasks.rados.rados.0.smithi071.stdout:1780: oid 25 version 535 is already newer than 533 2024-09-11T01:41:47.438 INFO:tasks.rados.rados.0.smithi071.stdout:1780: oid 25 version 535 is already newer than 534 2024-09-11T01:41:47.438 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 535 (ObjNum 651 snap 199 seq_num 651) dirty exists 2024-09-11T01:41:47.439 INFO:tasks.rados.rados.0.smithi071.stdout:1780: left oid 25 (ObjNum 651 snap 199 seq_num 651) 2024-09-11T01:41:47.439 INFO:tasks.rados.rados.0.smithi071.stdout:1782: finishing copy_from racing read to smithi07134777-11 2024-09-11T01:41:47.439 INFO:tasks.rados.rados.0.smithi071.stdout:1782: finishing copy_from to smithi07134777-11 2024-09-11T01:41:47.439 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 906 (ObjNum 636 snap 193 seq_num 636) dirty exists 2024-09-11T01:41:47.439 INFO:tasks.rados.rados.0.smithi071.stdout:1779: done (3 left) 2024-09-11T01:41:47.439 INFO:tasks.rados.rados.0.smithi071.stdout:1780: done (2 left) 2024-09-11T01:41:47.439 INFO:tasks.rados.rados.0.smithi071.stdout:1781: done (1 left) 2024-09-11T01:41:47.439 INFO:tasks.rados.rados.0.smithi071.stdout:1782: done (0 left) 2024-09-11T01:41:47.439 INFO:tasks.rados.rados.0.smithi071.stdout:1783: copy_from oid 38 from oid 14 current snap is 199 2024-09-11T01:41:47.439 INFO:tasks.rados.rados.0.smithi071.stdout:1784: rollback oid 4 current snap is 199 2024-09-11T01:41:47.439 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 4 to 174 2024-09-11T01:41:47.439 INFO:tasks.rados.rados.0.smithi071.stdout:1785: copy_from oid 27 from oid 48 current snap is 199 2024-09-11T01:41:47.439 INFO:tasks.rados.rados.0.smithi071.stdout:1786: write (excl) oid 23 current snap is 199 2024-09-11T01:41:47.439 INFO:tasks.rados.rados.0.smithi071.stdout:1786: seq_num 652 ranges {558164=552353,1803789=454563,2855203=102961} 2024-09-11T01:41:47.443 INFO:tasks.rados.rados.0.smithi071.stdout:1786: writing smithi07134777-23 from 558164 to 1110517 tid 1 2024-09-11T01:41:47.450 INFO:tasks.rados.rados.0.smithi071.stdout:1786: writing smithi07134777-23 from 1803789 to 2258352 tid 2 2024-09-11T01:41:47.455 INFO:tasks.rados.rados.0.smithi071.stdout:1786: writing smithi07134777-23 from 2855203 to 2958164 tid 3 2024-09-11T01:41:47.455 INFO:tasks.rados.rados.0.smithi071.stdout:1787: snap_create 2024-09-11T01:41:47.455 INFO:tasks.rados.rados.0.smithi071.stdout:1784: finishing rollback tid 0 to smithi07134777-4 2024-09-11T01:41:47.455 INFO:tasks.rados.rados.0.smithi071.stdout:1784: finishing rollback tid 1 to smithi07134777-4 2024-09-11T01:41:47.455 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 908 (ObjNum 129 snap 0 seq_num 2371328008) dirty dne 2024-09-11T01:41:47.455 INFO:tasks.rados.rados.0.smithi071.stdout:1786: finishing write tid 1 to smithi07134777-23 2024-09-11T01:41:47.455 INFO:tasks.rados.rados.0.smithi071.stdout:1783: finishing copy_from racing read to smithi07134777-38 2024-09-11T01:41:47.456 INFO:tasks.rados.rados.0.smithi071.stdout:1785: finishing copy_from racing read to smithi07134777-27 2024-09-11T01:41:47.456 INFO:tasks.rados.rados.0.smithi071.stdout:1783: finishing copy_from to smithi07134777-38 2024-09-11T01:41:47.456 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 583 (ObjNum 542 snap 154 seq_num 542) dirty exists 2024-09-11T01:41:47.456 INFO:tasks.rados.rados.0.smithi071.stdout:1786: finishing write tid 2 to smithi07134777-23 2024-09-11T01:41:47.457 INFO:tasks.rados.rados.0.smithi071.stdout:1786: finishing write tid 3 to smithi07134777-23 2024-09-11T01:41:47.457 INFO:tasks.rados.rados.0.smithi071.stdout:1786: finishing write tid 4 to smithi07134777-23 2024-09-11T01:41:47.457 INFO:tasks.rados.rados.0.smithi071.stdout:1786: finishing write tid 5 to smithi07134777-23 2024-09-11T01:41:47.457 INFO:tasks.rados.rados.0.smithi071.stdout:1786: oid 23 updating version 0 to 334 2024-09-11T01:41:47.458 INFO:tasks.rados.rados.0.smithi071.stdout:1786: oid 23 updating version 334 to 335 2024-09-11T01:41:47.458 INFO:tasks.rados.rados.0.smithi071.stdout:1786: oid 23 version 335 is already newer than 333 2024-09-11T01:41:47.458 INFO:tasks.rados.rados.0.smithi071.stdout:1786: oid 23 updating version 335 to 336 2024-09-11T01:41:47.458 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 336 (ObjNum 652 snap 199 seq_num 652) dirty exists 2024-09-11T01:41:47.458 INFO:tasks.rados.rados.0.smithi071.stdout:1786: left oid 23 (ObjNum 652 snap 199 seq_num 652) 2024-09-11T01:41:47.458 INFO:tasks.rados.rados.0.smithi071.stdout:1785: finishing copy_from to smithi07134777-27 2024-09-11T01:41:47.458 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 998 (ObjNum 586 snap 168 seq_num 586) dirty exists 2024-09-11T01:41:47.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.642+0000 7fa8d47ca640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:41:47.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.642+0000 7fa8d47ca640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:41:47.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.642+0000 7fa8d47ca640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:47.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.642+0000 7fa8d47ca640 1 -- 172.21.15.12:0/2226271580 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa8cc1ccf30 con 0x7fa8b80039b0 2024-09-11T01:41:47.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.643+0000 7fa8c2ffd640 1 -- 172.21.15.12:0/2226271580 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fa8c8032d00 con 0x7fa8b80039b0 2024-09-11T01:41:47.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.643+0000 7fa8c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:41:47.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.643+0000 7fa8c2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:41:47.645 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":118,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:41:47.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.645+0000 7fa8d47ca640 1 -- 172.21.15.12:0/2226271580 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa8b003f030 msgr2=0x7fa8b00414f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:47.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.645+0000 7fa8d47ca640 1 --2- 172.21.15.12:0/2226271580 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa8b003f030 0x7fa8b00414f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc0a2a40 tx=0x7fa8bc008040 comp rx=0 tx=0).stop 2024-09-11T01:41:47.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.645+0000 7fa8d47ca640 10 monclient: shutdown 2024-09-11T01:41:47.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.645+0000 7fa8d47ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:47.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.645+0000 7fa8d47ca640 1 -- 172.21.15.12:0/2226271580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8b80039b0 msgr2=0x7fa8cc1cc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:47.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.645+0000 7fa8d47ca640 1 --2- 172.21.15.12:0/2226271580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8b80039b0 0x7fa8cc1cc9f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa8c800c300 tx=0x7fa8c80046c0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.645+0000 7fa8d47ca640 1 -- 172.21.15.12:0/2226271580 shutdown_connections 2024-09-11T01:41:47.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.645+0000 7fa8d47ca640 1 --2- 172.21.15.12:0/2226271580 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8b80052e0 0x7fa8cc1376d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.645+0000 7fa8d47ca640 1 --2- 172.21.15.12:0/2226271580 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8b8004800 0x7fa8cc137190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.645+0000 7fa8d47ca640 1 --2- 172.21.15.12:0/2226271580 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa8b003f030 0x7fa8b00414f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.645+0000 7fa8d47ca640 1 --2- 172.21.15.12:0/2226271580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa8b80039b0 0x7fa8cc1cc9f0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.645+0000 7fa8d47ca640 1 -- 172.21.15.12:0/2226271580 >> 172.21.15.12:0/2226271580 conn(0x7fa8cc0a8490 msgr2=0x7fa8cc0a6b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:47.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.645+0000 7fa8d47ca640 1 -- 172.21.15.12:0/2226271580 shutdown_connections 2024-09-11T01:41:47.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.645+0000 7fa8d47ca640 1 -- 172.21.15.12:0/2226271580 wait complete. 2024-09-11T01:41:47.657 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:41:47.657 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-11T01:41:47.657 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:41:47.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.758+0000 7fdbe880c640 10 monclient: get_monmap_and_config 2024-09-11T01:41:47.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.758+0000 7fdbe880c640 10 monclient: build_initial_monmap 2024-09-11T01:41:47.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.758+0000 7fdbe880c640 10 monclient: monmap: 2024-09-11T01:41:47.759 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:47.759 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:47.760 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:41:47.758759+0000 2024-09-11T01:41:47.760 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:41:47.758759+0000 2024-09-11T01:41:47.760 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:47.760 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:47.760 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:41:47.760 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:41:47.760 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:41:47.760 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:41:47.760 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:41:47.760 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.758+0000 7fdbe880c640 1 Processor -- start 2024-09-11T01:41:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.758+0000 7fdbe880c640 1 -- start start 2024-09-11T01:41:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.758+0000 7fdbe880c640 10 monclient: init 2024-09-11T01:41:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe880c640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe880c640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-11T01:41:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe880c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdbe40a2db0 addr v1:172.21.15.71:6789/0 2024-09-11T01:41:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe880c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdbe40a38a0 0x7fdbe406f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe880c640 10 monclient(hunting): picked mon.noname-b con 0x7fdbe40a38a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe880c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdbe406ded0 0x7fdbe413f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe880c640 10 monclient(hunting): picked mon.noname-a con 0x7fdbe406ded0 addr v2:172.21.15.12:3300/0 2024-09-11T01:41:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe880c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbe406e320 con 0x7fdbe40a2db0 2024-09-11T01:41:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe880c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe880c640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fdbe406f1a0 con 0x7fdbe406ded0 2024-09-11T01:41:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe880c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe880c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fdbe406eec0 con 0x7fdbe40a38a0 2024-09-11T01:41:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe880c640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe880c640 10 monclient(hunting): authenticate will time out at 1514.201437s 2024-09-11T01:41:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe1d9b640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdbe40a38a0 msgr2=0x7fdbe406f8a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:41:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe1d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdbe40a38a0 0x7fdbe406f8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:41:47.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe2d9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdbe406ded0 0x7fdbe413f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:47.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe2d9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdbe406ded0 0x7fdbe413f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48238/0 (socket says 172.21.15.12:48238) 2024-09-11T01:41:47.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe2d9d640 1 -- 172.21.15.12:0/2268793760 learned_addr learned my addr 172.21.15.12:0/2268793760 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:41:47.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe2d9d640 10 monclient(hunting): get_auth_request con 0x7fdbe406ded0 auth_method 0 2024-09-11T01:41:47.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:47.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:47.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:47.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:47.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:47.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:47.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe2d9d640 10 monclient(hunting): handle_auth_done global_id 94521 payload 274 2024-09-11T01:41:47.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe2d9d640 1 -- 172.21.15.12:0/2268793760 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdbe40a38a0 msgr2=0x7fdbe406f8a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:41:47.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe2d9d640 1 --2- 172.21.15.12:0/2268793760 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdbe40a38a0 0x7fdbe406f8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe2d9d640 1 -- 172.21.15.12:0/2268793760 >> v1:172.21.15.71:6789/0 conn(0x7fdbe40a2db0 legacy=0x7fdbe40a3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe2d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:41:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe2d9d640 1 monclient: found mon.noname-a 2024-09-11T01:41:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:41:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.759+0000 7fdbe2d9d640 1 -- 172.21.15.12:0/2268793760 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbe40a4650 con 0x7fdbe406ded0 2024-09-11T01:41:47.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.762+0000 7fdbe2d9d640 1 -- 172.21.15.12:0/2268793760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe406ded0 msgr2=0x7fdbe413f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:47.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.762+0000 7fdbe2d9d640 1 --2- 172.21.15.12:0/2268793760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe406ded0 0x7fdbe413f3c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc009e50 tx=0x7fdbdc02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98a7394b32aa9a74 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:47.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.762+0000 7fdbe159a640 1 -- 172.21.15.12:0/2268793760 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbdc036070 con 0x7fdbe406ded0 2024-09-11T01:41:47.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.762+0000 7fdbe159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:47.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.762+0000 7fdbe159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:41:47.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.762+0000 7fdbe159a640 10 monclient: dump: 2024-09-11T01:41:47.772 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:47.772 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:47.772 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.772 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.772 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:47.772 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:47.773 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:47.773 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:47.773 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:47.773 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:47.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.762+0000 7fdbe159a640 10 monclient: _finish_auth 0 2024-09-11T01:41:47.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.762+0000 7fdbe159a640 10 monclient: _check_auth_tickets 2024-09-11T01:41:47.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.762+0000 7fdbe159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:47.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.762+0000 7fdbe880c640 5 monclient: authenticate success, global_id 94521 2024-09-11T01:41:47.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.762+0000 7fdbe880c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:41:47.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.762+0000 7fdbe159a640 1 -- 172.21.15.12:0/2268793760 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbdc007800 con 0x7fdbe406ded0 2024-09-11T01:41:47.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.762+0000 7fdbe159a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:47.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.762+0000 7fdbe880c640 10 monclient: get_monmap_and_config success 2024-09-11T01:41:47.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.762+0000 7fdbe880c640 4 set_mon_vals no callback set 2024-09-11T01:41:47.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.762+0000 7fdbe159a640 1 -- 172.21.15.12:0/2268793760 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbdc007bd0 con 0x7fdbe406ded0 2024-09-11T01:41:47.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.762+0000 7fdbe159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:47.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.762+0000 7fdbe159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:47.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.762+0000 7fdbe159a640 10 monclient: dump: 2024-09-11T01:41:47.774 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:47.774 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:47.774 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.774 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.774 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:47.774 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:47.774 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:47.774 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:47.774 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:47.774 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:47.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.763+0000 7fdbe880c640 10 monclient: shutdown 2024-09-11T01:41:47.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.763+0000 7fdbe880c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:47.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.763+0000 7fdbe880c640 1 -- 172.21.15.12:0/2268793760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe406ded0 msgr2=0x7fdbe413f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:47.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.763+0000 7fdbe880c640 1 --2- 172.21.15.12:0/2268793760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe406ded0 0x7fdbe413f3c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc009e50 tx=0x7fdbdc02a610 comp rx=0 tx=0).stop 2024-09-11T01:41:47.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.763+0000 7fdbe880c640 1 -- 172.21.15.12:0/2268793760 shutdown_connections 2024-09-11T01:41:47.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.763+0000 7fdbe880c640 1 --2- 172.21.15.12:0/2268793760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe406ded0 0x7fdbe413f3c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.763+0000 7fdbe880c640 1 --2- 172.21.15.12:0/2268793760 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdbe40a38a0 0x7fdbe406f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.763+0000 7fdbe880c640 1 -- 172.21.15.12:0/2268793760 >> 172.21.15.12:0/2268793760 conn(0x7fdbe40a83b0 msgr2=0x7fdbe40a87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:47.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.763+0000 7fdbe880c640 1 -- 172.21.15.12:0/2268793760 shutdown_connections 2024-09-11T01:41:47.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.763+0000 7fdbe880c640 1 -- 172.21.15.12:0/2268793760 wait complete. 2024-09-11T01:41:47.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.764+0000 7fdbe880c640 10 monclient: build_initial_monmap 2024-09-11T01:41:47.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.764+0000 7fdbe880c640 10 monclient: monmap: 2024-09-11T01:41:47.775 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:47.775 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:41:47.775 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:41:47.776 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:41:47.776 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:47.776 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:47.776 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:41:47.776 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:41:47.776 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:41:47.776 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:47.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.764+0000 7fdbe880c640 1 Processor -- start 2024-09-11T01:41:47.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.764+0000 7fdbe880c640 1 -- start start 2024-09-11T01:41:47.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.764+0000 7fdbe880c640 10 monclient: init 2024-09-11T01:41:47.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe880c640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:47.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe880c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:41:47.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe880c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdbe406ded0 0x7fdbe41ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:47.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe880c640 10 monclient(hunting): picked mon.noname-c con 0x7fdbe406ded0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:47.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe880c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe40a2db0 0x7fdbe41cd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:47.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe880c640 10 monclient(hunting): picked mon.noname-a con 0x7fdbe40a2db0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:41:47.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe880c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdbe40a38a0 0x7fdbe41cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:47.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe880c640 10 monclient(hunting): picked mon.noname-b con 0x7fdbe40a38a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:41:47.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe880c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:47.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe880c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe413fc90 con 0x7fdbe40a2db0 2024-09-11T01:41:47.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe880c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:47.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe880c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe413fb10 con 0x7fdbe40a38a0 2024-09-11T01:41:47.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe880c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:47.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe880c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fdbe413fe10 con 0x7fdbe406ded0 2024-09-11T01:41:47.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe880c640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:47.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe880c640 10 monclient(hunting): authenticate will time out at 1514.207178s 2024-09-11T01:41:47.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe259c640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdbe406ded0 msgr2=0x7fdbe41ccf40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:41:47.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe259c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdbe406ded0 0x7fdbe41ccf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:41:47.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe1d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe40a2db0 0x7fdbe41cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:47.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe1d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe40a2db0 0x7fdbe41cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48246/0 (socket says 172.21.15.12:48246) 2024-09-11T01:41:47.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe1d9b640 1 -- 172.21.15.12:0/1470599075 learned_addr learned my addr 172.21.15.12:0/1470599075 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:41:47.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe1d9b640 10 monclient(hunting): get_auth_request con 0x7fdbe40a2db0 auth_method 0 2024-09-11T01:41:47.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:47.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:47.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:47.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:47.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:47.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:47.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe2d9d640 1 --2- 172.21.15.12:0/1470599075 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdbe40a38a0 0x7fdbe41cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:47.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe2d9d640 10 monclient(hunting): get_auth_request con 0x7fdbe40a38a0 auth_method 0 2024-09-11T01:41:47.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:47.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:47.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:47.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:47.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:47.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:47.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe1d9b640 10 monclient(hunting): handle_auth_done global_id 94524 payload 824 2024-09-11T01:41:47.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe1d9b640 1 -- 172.21.15.12:0/1470599075 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdbe406ded0 msgr2=0x7fdbe41ccf40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:41:47.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe1d9b640 1 --2- 172.21.15.12:0/1470599075 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdbe406ded0 0x7fdbe41ccf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe1d9b640 1 -- 172.21.15.12:0/1470599075 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdbe40a38a0 msgr2=0x7fdbe41cd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:47.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe1d9b640 1 --2- 172.21.15.12:0/1470599075 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdbe40a38a0 0x7fdbe41cd9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe1d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:41:47.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe1d9b640 1 monclient: found mon.noname-a 2024-09-11T01:41:47.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:47.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:41:47.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.765+0000 7fdbe1d9b640 1 -- 172.21.15.12:0/1470599075 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbe413bc90 con 0x7fdbe40a2db0 2024-09-11T01:41:47.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.767+0000 7fdbe1d9b640 1 -- 172.21.15.12:0/1470599075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe40a2db0 msgr2=0x7fdbe41cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:47.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.767+0000 7fdbe1d9b640 1 --2- 172.21.15.12:0/1470599075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe40a2db0 0x7fdbe41cd480 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fdbd800b550 tx=0x7fdbd800ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:47.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.767+0000 7fdbe1d9b640 1 --2- 172.21.15.12:0/1470599075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe40a2db0 0x7fdbe41cd480 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fdbd800b550 tx=0x7fdbd800ba10 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7fdbd8003dd0 mon_map magic: 0 2024-09-11T01:41:47.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.767+0000 7fdbd2ffd640 1 -- 172.21.15.12:0/1470599075 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd8003dd0 con 0x7fdbe40a2db0 2024-09-11T01:41:47.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:47.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbd2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:41:47.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbd2ffd640 10 monclient: dump: 2024-09-11T01:41:47.781 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:47.781 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:47.781 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.781 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.781 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:47.781 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:47.781 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:47.781 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:47.781 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:47.781 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:47.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbd2ffd640 10 monclient: _finish_auth 0 2024-09-11T01:41:47.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbd2ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:41:47.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:47.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbd2ffd640 1 -- 172.21.15.12:0/1470599075 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbd8015070 con 0x7fdbe40a2db0 2024-09-11T01:41:47.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbe880c640 5 monclient: authenticate success, global_id 94524 2024-09-11T01:41:47.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbe880c640 10 monclient: _renew_subs 2024-09-11T01:41:47.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbe880c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:47.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbe880c640 1 -- 172.21.15.12:0/1470599075 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbe413bf70 con 0x7fdbe40a2db0 2024-09-11T01:41:47.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbd2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:47.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbe880c640 10 monclient: _renew_subs 2024-09-11T01:41:47.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbe0d99640 4 set_mon_vals no callback set 2024-09-11T01:41:47.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbe880c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:47.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbe880c640 1 -- 172.21.15.12:0/1470599075 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbe413ee90 con 0x7fdbe40a2db0 2024-09-11T01:41:47.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbe880c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:47.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbe880c640 1 --2- 172.21.15.12:0/1470599075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe413a770 0x7fdbe4134f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:47.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbe880c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:47.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbe880c640 1 -- 172.21.15.12:0/1470599075 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe406a060 con 0x7fdbe413a770 2024-09-11T01:41:47.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbe259c640 1 --2- 172.21.15.12:0/1470599075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe413a770 0x7fdbe4134f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:47.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbd2ffd640 1 -- 172.21.15.12:0/1470599075 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd80052b0 con 0x7fdbe40a2db0 2024-09-11T01:41:47.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:47.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbd2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:47.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbd2ffd640 10 monclient: dump: 2024-09-11T01:41:47.783 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:47.783 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:47.783 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.783 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.784 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:47.784 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:47.784 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:47.784 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:47.784 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:47.784 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:47.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbe259c640 10 monclient: get_auth_request con 0x7fdbe413a770 auth_method 0 2024-09-11T01:41:47.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbd2ffd640 1 -- 172.21.15.12:0/1470599075 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fdbd8005490 con 0x7fdbe40a2db0 2024-09-11T01:41:47.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbe259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:47.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbe259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:47.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.768+0000 7fdbe259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:47.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.769+0000 7fdbe259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:47.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.769+0000 7fdbe259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:47.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.769+0000 7fdbe259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:47.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.769+0000 7fdbe259c640 10 monclient(hunting): handle_auth_done global_id 94527 payload 824 2024-09-11T01:41:47.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.769+0000 7fdbe259c640 1 -- 172.21.15.12:0/1470599075 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdbe406f8a0 con 0x7fdbe413a770 2024-09-11T01:41:47.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.769+0000 7fdbd2ffd640 1 --2- 172.21.15.12:0/1470599075 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdbc403e1c0 0x7fdbc4040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:47.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.769+0000 7fdbe2d9d640 1 --2- 172.21.15.12:0/1470599075 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdbc403e1c0 0x7fdbc4040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:47.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.769+0000 7fdbe2d9d640 10 monclient: get_auth_request con 0x7fdbc403e1c0 auth_method 0 2024-09-11T01:41:47.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.771+0000 7fdbe259c640 1 -- 172.21.15.12:0/1470599075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe413a770 msgr2=0x7fdbe4134f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:47.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.771+0000 7fdbe259c640 1 --2- 172.21.15.12:0/1470599075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe413a770 0x7fdbe4134f30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdbd400c380 tx=0x7fdbd400c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:47.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.771+0000 7fdbd2ffd640 1 -- 172.21.15.12:0/1470599075 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd4017070 con 0x7fdbe413a770 2024-09-11T01:41:47.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.771+0000 7fdbd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:47.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.771+0000 7fdbd2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:47.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.771+0000 7fdbd2ffd640 10 monclient: dump: 2024-09-11T01:41:47.788 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:47.788 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:47.788 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.788 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.788 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:47.788 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:47.788 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:47.788 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:47.788 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:47.788 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:47.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.771+0000 7fdbd2ffd640 1 -- 172.21.15.12:0/1470599075 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdbd40057c0 con 0x7fdbe413a770 2024-09-11T01:41:47.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.771+0000 7fdbd2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:47.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.771+0000 7fdbd2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:41:47.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.771+0000 7fdbd2ffd640 1 -- 172.21.15.12:0/1470599075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe413a770 msgr2=0x7fdbe4134f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:47.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.771+0000 7fdbd2ffd640 1 --2- 172.21.15.12:0/1470599075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe413a770 0x7fdbe4134f30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdbd400c380 tx=0x7fdbd400c840 comp rx=0 tx=0).stop 2024-09-11T01:41:47.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.771+0000 7fdbd2ffd640 1 -- 172.21.15.12:0/1470599075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe413a770 msgr2=0x7fdbe4134f30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:41:47.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.771+0000 7fdbd2ffd640 1 --2- 172.21.15.12:0/1470599075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe413a770 0x7fdbe4134f30 secure :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdbd400c380 tx=0x7fdbd400c840 comp rx=0 tx=0).stop 2024-09-11T01:41:47.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.772+0000 7fdbe2d9d640 1 --2- 172.21.15.12:0/1470599075 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdbc403e1c0 0x7fdbc4040680 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4070b40 tx=0x7fdbdc009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:47.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.777+0000 7fdbe880c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:41:47.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.777+0000 7fdbe880c640 1 --2- 172.21.15.12:0/1470599075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe4135470 0x7fdbe4135880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:47.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.777+0000 7fdbe880c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:47.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.777+0000 7fdbe880c640 1 -- 172.21.15.12:0/1470599075 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe4135dc0 con 0x7fdbe4135470 2024-09-11T01:41:47.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.778+0000 7fdbe259c640 1 --2- 172.21.15.12:0/1470599075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe4135470 0x7fdbe4135880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:47.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.778+0000 7fdbe259c640 10 monclient: get_auth_request con 0x7fdbe4135470 auth_method 0 2024-09-11T01:41:47.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.778+0000 7fdbe259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:47.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.778+0000 7fdbe259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:47.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.778+0000 7fdbe259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:47.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.778+0000 7fdbe259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:47.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.778+0000 7fdbe259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:47.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.778+0000 7fdbe259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:47.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.778+0000 7fdbe259c640 10 monclient(hunting): handle_auth_done global_id 94530 payload 824 2024-09-11T01:41:47.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.778+0000 7fdbe259c640 1 -- 172.21.15.12:0/1470599075 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdbe409e5c0 con 0x7fdbe4135470 2024-09-11T01:41:47.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.780+0000 7fdbe259c640 1 -- 172.21.15.12:0/1470599075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe4135470 msgr2=0x7fdbe4135880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:47.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.780+0000 7fdbe259c640 1 --2- 172.21.15.12:0/1470599075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe4135470 0x7fdbe4135880 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdbd400c380 tx=0x7fdbd40037e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:47.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.780+0000 7fdbd2ffd640 1 -- 172.21.15.12:0/1470599075 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd4017040 con 0x7fdbe4135470 2024-09-11T01:41:47.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.780+0000 7fdbd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:47.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.780+0000 7fdbd2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:47.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.780+0000 7fdbd2ffd640 10 monclient: dump: 2024-09-11T01:41:47.791 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:47.791 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:47.791 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.791 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.791 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:47.791 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:47.792 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:47.792 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:47.792 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:47.792 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:47.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.780+0000 7fdbd2ffd640 1 -- 172.21.15.12:0/1470599075 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4700 (secure 0 0 0) 0x7fdbd4012400 con 0x7fdbe4135470 2024-09-11T01:41:47.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.780+0000 7fdbd2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:41:47.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.780+0000 7fdbd2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:41:47.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.780+0000 7fdbd2ffd640 1 -- 172.21.15.12:0/1470599075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe4135470 msgr2=0x7fdbe4135880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:47.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.780+0000 7fdbd2ffd640 1 --2- 172.21.15.12:0/1470599075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe4135470 0x7fdbe4135880 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdbd400c380 tx=0x7fdbd40037e0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.780+0000 7fdbd2ffd640 1 -- 172.21.15.12:0/1470599075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe4135470 msgr2=0x7fdbe4135880 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:41:47.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.780+0000 7fdbd2ffd640 1 --2- 172.21.15.12:0/1470599075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe4135470 0x7fdbe4135880 secure :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdbd400c380 tx=0x7fdbd40037e0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.793 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:41:47.793 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:41:47.793 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:41:47.793 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:41:47.793 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 118, 2024-09-11T01:41:47.793 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 253876, 2024-09-11T01:41:47.793 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:41:47.793 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:41:47.793 INFO:teuthology.orchestra.run.smithi012.stdout: 1 2024-09-11T01:41:47.793 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:47.793 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:41:47.794 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:41:47.794 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:41:47.794 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:41:47.794 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:41:47.794 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:41:47.794 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:41:47.794 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:41:47.794 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:41:47.794 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:41:47.794 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:41:47.794 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:41:47.794 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:41:47.794 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:41:47.795 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:41:47.795 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:47.795 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:41:47.795 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:41:47.795 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:41:47.795 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:41:47.795 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:41:47.795 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:41:47.795 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:41:47.795 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:41:47.795 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:41:47.795 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:41:47.795 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:41:47.795 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:41:47.796 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:41:47.796 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:47.796 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:47.796 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:41:47.796 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:41:47.796 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:41:47.796 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:41:47.796 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:41:47.796 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:41:47.796 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:41:47.796 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:41:47.796 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:41:47.796 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:41:47.796 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:41:47.797 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:41:47.797 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:41:47.797 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:41:47.797 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:41:47.797 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:41:47.797 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:41:47.797 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:41:47.797 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:41:47.797 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:41:47.797 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:41:47.797 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:41:47.797 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:41:47.797 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:41:47.798 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:41:47.798 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:41:47.798 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:41:47.798 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:41:47.798 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:47.798 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:41:47.798 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:47.798 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:41:47.798 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:47.798 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:41:47.798 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:41:47.798 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:41:47.798 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:41:47.799 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:47.799 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:41:47.799 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:41:47.799 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:47.799 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:47.799 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:47.799 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:41:47.799 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:41:47.799 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:47.799 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:47.799 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:47.799 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:47.799 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:41:47.799 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:41:47.800 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:41:47.800 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:41:47.800 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:41:47.800 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:47.800 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:47.800 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:41:47.800 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:41:47.800 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:41:47.800 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:41:47.800 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:47.800 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:41:47.800 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:41:47.800 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:47.801 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:47.801 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:47.801 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:41:47.801 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:41:47.801 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:47.801 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:47.801 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:47.801 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:47.801 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:41:47.801 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:41:47.801 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:41:47.801 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:41:47.801 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:41:47.801 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:47.802 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:47.802 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:41:47.802 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:41:47.802 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:41:47.802 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:41:47.802 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:47.802 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:41:47.802 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:41:47.802 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:47.802 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:47.802 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:47.802 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:41:47.802 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:41:47.803 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:47.803 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:47.803 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:47.803 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:47.803 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:41:47.803 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:41:47.803 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:41:47.803 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:41:47.803 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:41:47.803 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:47.803 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:47.803 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:47.803 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:41:47.803 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:41:47.803 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:47.804 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:41:47.804 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:41:47.804 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:41:47.804 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:47.804 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:47.804 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:41:47.804 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:47.804 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:41:47.804 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:41:47.804 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:41:47.804 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:47.804 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:47.804 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:41:47.805 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:47.805 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:41:47.805 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:41:47.805 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:41:47.805 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:47.805 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:47.805 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:47.805 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:41:47.805 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:41:47.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.781+0000 7fdbe880c640 1 -- 172.21.15.12:0/1470599075 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdbc403e1c0 msgr2=0x7fdbc4040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:47.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.781+0000 7fdbe880c640 1 --2- 172.21.15.12:0/1470599075 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdbc403e1c0 0x7fdbc4040680 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4070b40 tx=0x7fdbdc009000 comp rx=0 tx=0).stop 2024-09-11T01:41:47.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.781+0000 7fdbe880c640 10 monclient: shutdown 2024-09-11T01:41:47.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.781+0000 7fdbe880c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:47.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.781+0000 7fdbe880c640 1 -- 172.21.15.12:0/1470599075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe40a2db0 msgr2=0x7fdbe41cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:47.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.781+0000 7fdbe880c640 1 --2- 172.21.15.12:0/1470599075 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdbe40a2db0 0x7fdbe41cd480 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fdbd800b550 tx=0x7fdbd800ba10 comp rx=0 tx=0).stop 2024-09-11T01:41:47.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.781+0000 7fdbe2d9d640 1 -- 172.21.15.12:0/1470599075 reap_dead start 2024-09-11T01:41:47.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.781+0000 7fdbd2ffd640 1 -- 172.21.15.12:0/1470599075 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(409..409 src has 285..409) ==== 4783+0+0 (unknown 0 0 0) 0x7fdbd8052730 con 0x7fdbe40a2db0 2024-09-11T01:41:47.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.781+0000 7fdbe2d9d640 1 -- 172.21.15.12:0/1470599075 reap_dead start 2024-09-11T01:41:47.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.781+0000 7fdbe880c640 1 -- 172.21.15.12:0/1470599075 shutdown_connections 2024-09-11T01:41:47.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.781+0000 7fdbe880c640 1 -- 172.21.15.12:0/1470599075 >> 172.21.15.12:0/1470599075 conn(0x7fdbe40a83b0 msgr2=0x7fdbe40a1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:47.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.781+0000 7fdbe880c640 1 -- 172.21.15.12:0/1470599075 shutdown_connections 2024-09-11T01:41:47.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.781+0000 7fdbe880c640 1 -- 172.21.15.12:0/1470599075 wait complete. 2024-09-11T01:41:47.808 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:41:47.808 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T01:41:47.808 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:41:47.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.909+0000 7fda72105640 10 monclient: get_monmap_and_config 2024-09-11T01:41:47.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.909+0000 7fda72105640 10 monclient: build_initial_monmap 2024-09-11T01:41:47.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.909+0000 7fda72105640 10 monclient: monmap: 2024-09-11T01:41:47.910 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:41:47.909771+0000 2024-09-11T01:41:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:41:47.909771+0000 2024-09-11T01:41:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:41:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:41:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:41:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:41:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:41:47.911 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:47.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.909+0000 7fda72105640 1 Processor -- start 2024-09-11T01:41:47.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.909+0000 7fda72105640 1 -- start start 2024-09-11T01:41:47.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.909+0000 7fda72105640 10 monclient: init 2024-09-11T01:41:47.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda72105640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:47.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda72105640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-11T01:41:47.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda72105640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda6c137d50 0x7fda6c09d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:47.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda72105640 10 monclient(hunting): picked mon.noname-b con 0x7fda6c137d50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:47.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda72105640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fda6c138130 0x7fda6c138550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:47.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda72105640 10 monclient(hunting): picked mon.noname-a con 0x7fda6c138130 addr v2:172.21.15.12:3300/0 2024-09-11T01:41:47.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda72105640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fda6c136cd0 0x7fda6c137120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:47.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda72105640 10 monclient(hunting): picked mon.noname-c con 0x7fda6c136cd0 addr v2:172.21.15.71:3300/0 2024-09-11T01:41:47.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda72105640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:47.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda72105640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fda6c09c650 con 0x7fda6c138130 2024-09-11T01:41:47.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda72105640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:47.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda72105640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fda6c09ba20 con 0x7fda6c136cd0 2024-09-11T01:41:47.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda72105640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:47.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda72105640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fda6c09c370 con 0x7fda6c137d50 2024-09-11T01:41:47.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda72105640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:47.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda72105640 10 monclient(hunting): authenticate will time out at 1514.352351s 2024-09-11T01:41:47.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6b7fe640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda6c137d50 msgr2=0x7fda6c09d110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:41:47.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda6c137d50 0x7fda6c09d110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:41:47.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6affd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fda6c138130 0x7fda6c138550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:47.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6affd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fda6c138130 0x7fda6c138550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48274/0 (socket says 172.21.15.12:48274) 2024-09-11T01:41:47.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6affd640 1 -- 172.21.15.12:0/2857390134 learned_addr learned my addr 172.21.15.12:0/2857390134 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:41:47.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6affd640 10 monclient(hunting): get_auth_request con 0x7fda6c138130 auth_method 0 2024-09-11T01:41:47.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:47.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:47.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:47.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:47.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:47.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:47.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6bfff640 1 --2- 172.21.15.12:0/2857390134 >> v2:172.21.15.71:3300/0 conn(0x7fda6c136cd0 0x7fda6c137120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:47.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6bfff640 10 monclient(hunting): get_auth_request con 0x7fda6c136cd0 auth_method 0 2024-09-11T01:41:47.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:47.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:47.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:47.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6affd640 10 monclient(hunting): handle_auth_done global_id 94533 payload 274 2024-09-11T01:41:47.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6affd640 1 -- 172.21.15.12:0/2857390134 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda6c137d50 msgr2=0x7fda6c09d110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:41:47.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6affd640 1 --2- 172.21.15.12:0/2857390134 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda6c137d50 0x7fda6c09d110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6affd640 1 -- 172.21.15.12:0/2857390134 >> v2:172.21.15.71:3300/0 conn(0x7fda6c136cd0 msgr2=0x7fda6c137120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:41:47.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6affd640 1 --2- 172.21.15.12:0/2857390134 >> v2:172.21.15.71:3300/0 conn(0x7fda6c136cd0 0x7fda6c137120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6affd640 10 monclient: _finish_hunting 0 2024-09-11T01:41:47.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6affd640 1 monclient: found mon.noname-a 2024-09-11T01:41:47.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:47.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:41:47.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6affd640 1 -- 172.21.15.12:0/2857390134 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda6c0a6ce0 con 0x7fda6c138130 2024-09-11T01:41:47.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.910+0000 7fda6bfff640 1 --2- 172.21.15.12:0/2857390134 >> v2:172.21.15.71:3300/0 conn(0x7fda6c136cd0 0x7fda6c137120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:41:47.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.913+0000 7fda6affd640 1 -- 172.21.15.12:0/2857390134 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda6c138130 msgr2=0x7fda6c138550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:47.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.913+0000 7fda6affd640 1 --2- 172.21.15.12:0/2857390134 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda6c138130 0x7fda6c138550 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fda6c0a6f10 tx=0x7fda5402a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=631669811c7b7f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:47.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda6a7fc640 1 -- 172.21.15.12:0/2857390134 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda54037070 con 0x7fda6c138130 2024-09-11T01:41:47.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda6a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:47.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda6a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:41:47.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda6a7fc640 10 monclient: dump: 2024-09-11T01:41:47.922 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:47.922 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:47.922 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.922 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.922 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:47.922 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:47.922 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:47.922 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:47.922 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:47.922 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:47.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda6a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:41:47.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda6a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:41:47.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:47.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda6a7fc640 1 -- 172.21.15.12:0/2857390134 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda54032640 con 0x7fda6c138130 2024-09-11T01:41:47.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda6a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:47.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda6a7fc640 1 -- 172.21.15.12:0/2857390134 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda54032a10 con 0x7fda6c138130 2024-09-11T01:41:47.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda72105640 5 monclient: authenticate success, global_id 94533 2024-09-11T01:41:47.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda72105640 10 monclient: get_monmap_and_config success 2024-09-11T01:41:47.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda72105640 4 set_mon_vals no callback set 2024-09-11T01:41:47.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda6a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:47.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda6a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:47.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda6a7fc640 10 monclient: dump: 2024-09-11T01:41:47.923 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:47.923 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:47.924 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.924 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.924 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:47.924 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:47.924 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:47.924 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:47.924 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:47.924 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:47.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda72105640 10 monclient: shutdown 2024-09-11T01:41:47.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda72105640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:47.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda72105640 1 -- 172.21.15.12:0/2857390134 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda6c138130 msgr2=0x7fda6c138550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:47.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda72105640 1 --2- 172.21.15.12:0/2857390134 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda6c138130 0x7fda6c138550 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fda6c0a6f10 tx=0x7fda5402a150 comp rx=0 tx=0).stop 2024-09-11T01:41:47.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda72105640 1 -- 172.21.15.12:0/2857390134 shutdown_connections 2024-09-11T01:41:47.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda72105640 1 --2- 172.21.15.12:0/2857390134 >> v2:172.21.15.71:3300/0 conn(0x7fda6c136cd0 0x7fda6c137120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda72105640 1 --2- 172.21.15.12:0/2857390134 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda6c138130 0x7fda6c138550 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda72105640 1 --2- 172.21.15.12:0/2857390134 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda6c137d50 0x7fda6c09d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda72105640 1 -- 172.21.15.12:0/2857390134 >> 172.21.15.12:0/2857390134 conn(0x7fda6c0a7020 msgr2=0x7fda6c0a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:47.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda72105640 1 -- 172.21.15.12:0/2857390134 shutdown_connections 2024-09-11T01:41:47.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.914+0000 7fda72105640 1 -- 172.21.15.12:0/2857390134 wait complete. 2024-09-11T01:41:47.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.915+0000 7fda72105640 10 monclient: build_initial_monmap 2024-09-11T01:41:47.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.915+0000 7fda72105640 10 monclient: monmap: 2024-09-11T01:41:47.925 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:41:47.925 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:41:47.925 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:41:47.925 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:41:47.925 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:41:47.925 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:47.926 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:41:47.926 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:41:47.926 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:41:47.926 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:47.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.915+0000 7fda72105640 1 Processor -- start 2024-09-11T01:41:47.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.915+0000 7fda72105640 1 -- start start 2024-09-11T01:41:47.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.915+0000 7fda72105640 10 monclient: init 2024-09-11T01:41:47.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda72105640 10 monclient: _reopen_session rank -1 2024-09-11T01:41:47.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda72105640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:41:47.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda72105640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c136cd0 0x7fda6c1cd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:47.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda72105640 10 monclient(hunting): picked mon.noname-b con 0x7fda6c136cd0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:41:47.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda72105640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda6c137d50 0x7fda6c1cd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:47.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda72105640 10 monclient(hunting): picked mon.noname-c con 0x7fda6c137d50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:41:47.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda72105640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda6c138130 0x7fda6c1236e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:47.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda72105640 10 monclient(hunting): picked mon.noname-a con 0x7fda6c138130 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:41:47.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda72105640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:47.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda72105640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fda6c06f850 con 0x7fda6c138130 2024-09-11T01:41:47.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda72105640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:47.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda72105640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fda6c06f6d0 con 0x7fda6c136cd0 2024-09-11T01:41:47.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda72105640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:47.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda72105640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fda6c09d650 con 0x7fda6c137d50 2024-09-11T01:41:47.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda72105640 10 monclient(hunting): _renew_subs 2024-09-11T01:41:47.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda72105640 10 monclient(hunting): authenticate will time out at 1514.358391s 2024-09-11T01:41:47.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6affd640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda6c137d50 msgr2=0x7fda6c1cd810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:41:47.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6affd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda6c137d50 0x7fda6c1cd810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:41:47.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda6c138130 0x7fda6c1236e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:47.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda6c138130 0x7fda6c1236e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48286/0 (socket says 172.21.15.12:48286) 2024-09-11T01:41:47.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6bfff640 1 -- 172.21.15.12:0/1655098520 learned_addr learned my addr 172.21.15.12:0/1655098520 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:41:47.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6bfff640 10 monclient(hunting): get_auth_request con 0x7fda6c138130 auth_method 0 2024-09-11T01:41:47.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:47.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:47.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:47.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6b7fe640 1 --2- 172.21.15.12:0/1655098520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c136cd0 0x7fda6c1cd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:47.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:47.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:47.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:47.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6b7fe640 10 monclient(hunting): get_auth_request con 0x7fda6c136cd0 auth_method 0 2024-09-11T01:41:47.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:47.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:47.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:47.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:47.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:47.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:47.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6bfff640 10 monclient(hunting): handle_auth_done global_id 94536 payload 824 2024-09-11T01:41:47.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6bfff640 1 -- 172.21.15.12:0/1655098520 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda6c137d50 msgr2=0x7fda6c1cd810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:41:47.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6bfff640 1 --2- 172.21.15.12:0/1655098520 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda6c137d50 0x7fda6c1cd810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6bfff640 1 -- 172.21.15.12:0/1655098520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c136cd0 msgr2=0x7fda6c1cd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:47.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6bfff640 1 --2- 172.21.15.12:0/1655098520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c136cd0 0x7fda6c1cd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:41:47.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6bfff640 1 monclient: found mon.noname-a 2024-09-11T01:41:47.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:41:47.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:41:47.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.916+0000 7fda6bfff640 1 -- 172.21.15.12:0/1655098520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda6c0a1710 con 0x7fda6c138130 2024-09-11T01:41:47.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.919+0000 7fda6bfff640 1 -- 172.21.15.12:0/1655098520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda6c138130 msgr2=0x7fda6c1236e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:47.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.919+0000 7fda6bfff640 1 --2- 172.21.15.12:0/1655098520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda6c138130 0x7fda6c1236e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fda5c00e7e0 tx=0x7fda5c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:47.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.919+0000 7fda63fff640 1 -- 172.21.15.12:0/1655098520 DispatchQueue::entry inject delay of 0.002000 2024-09-11T01:41:47.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.921+0000 7fda63fff640 1 -- 172.21.15.12:0/1655098520 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda5c00f040 con 0x7fda6c138130 2024-09-11T01:41:47.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.921+0000 7fda63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:47.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.921+0000 7fda63fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:41:47.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.921+0000 7fda63fff640 10 monclient: dump: 2024-09-11T01:41:47.934 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:47.934 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:47.934 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.934 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.934 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:47.934 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:47.934 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:47.934 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:47.934 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:47.934 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:47.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.921+0000 7fda63fff640 10 monclient: _finish_auth 0 2024-09-11T01:41:47.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.921+0000 7fda63fff640 10 monclient: _check_auth_tickets 2024-09-11T01:41:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.921+0000 7fda63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:41:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.921+0000 7fda63fff640 1 -- 172.21.15.12:0/1655098520 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda5c004230 con 0x7fda6c138130 2024-09-11T01:41:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.921+0000 7fda72105640 5 monclient: authenticate success, global_id 94536 2024-09-11T01:41:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.921+0000 7fda63fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:41:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.921+0000 7fda63fff640 1 -- 172.21.15.12:0/1655098520 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda5c00c6d0 con 0x7fda6c138130 2024-09-11T01:41:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.921+0000 7fda72105640 10 monclient: _renew_subs 2024-09-11T01:41:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.921+0000 7fda72105640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.921+0000 7fda72105640 1 -- 172.21.15.12:0/1655098520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda6c06ee10 con 0x7fda6c138130 2024-09-11T01:41:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.921+0000 7fda69ffb640 4 set_mon_vals no callback set 2024-09-11T01:41:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.921+0000 7fda63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.921+0000 7fda63fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:41:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.921+0000 7fda63fff640 10 monclient: dump: 2024-09-11T01:41:47.935 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:47.936 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:47.936 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.936 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.936 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:47.936 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:47.936 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:47.936 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:47.936 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:47.936 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:47.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.921+0000 7fda72105640 10 monclient: _renew_subs 2024-09-11T01:41:47.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.921+0000 7fda72105640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:41:47.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.921+0000 7fda72105640 1 -- 172.21.15.12:0/1655098520 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda6c127500 con 0x7fda6c138130 2024-09-11T01:41:47.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.922+0000 7fda72105640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:47.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.922+0000 7fda72105640 1 --2- 172.21.15.12:0/1655098520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c123c20 0x7fda6c124000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:47.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.922+0000 7fda72105640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:47.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.922+0000 7fda72105640 1 -- 172.21.15.12:0/1655098520 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fda6c09d110 con 0x7fda6c123c20 2024-09-11T01:41:47.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.922+0000 7fda63fff640 1 -- 172.21.15.12:0/1655098520 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fda5c012050 con 0x7fda6c138130 2024-09-11T01:41:47.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.922+0000 7fda63fff640 1 --2- 172.21.15.12:0/1655098520 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda4c03e240 0x7fda4c040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:47.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.922+0000 7fda6b7fe640 1 --2- 172.21.15.12:0/1655098520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c123c20 0x7fda6c124000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:47.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.922+0000 7fda6b7fe640 10 monclient: get_auth_request con 0x7fda6c123c20 auth_method 0 2024-09-11T01:41:47.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.922+0000 7fda63fff640 1 -- 172.21.15.12:0/1655098520 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(409..409 src has 285..409) ==== 4783+0+0 (secure 0 0 0) 0x7fda5c0527b0 con 0x7fda6c138130 2024-09-11T01:41:47.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.922+0000 7fda6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:47.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.922+0000 7fda6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:47.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.922+0000 7fda6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:47.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.922+0000 7fda6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:47.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.922+0000 7fda6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:47.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.922+0000 7fda6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:47.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.922+0000 7fda6affd640 1 --2- 172.21.15.12:0/1655098520 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda4c03e240 0x7fda4c040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:47.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.922+0000 7fda6affd640 10 monclient: get_auth_request con 0x7fda4c03e240 auth_method 0 2024-09-11T01:41:47.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.922+0000 7fda6b7fe640 10 monclient(hunting): handle_auth_done global_id 134167 payload 824 2024-09-11T01:41:47.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.922+0000 7fda6b7fe640 1 -- 172.21.15.12:0/1655098520 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fda6c0008d0 con 0x7fda6c123c20 2024-09-11T01:41:47.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.925+0000 7fda6b7fe640 1 -- 172.21.15.12:0/1655098520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c123c20 msgr2=0x7fda6c124000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:47.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.925+0000 7fda6b7fe640 1 --2- 172.21.15.12:0/1655098520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c123c20 0x7fda6c124000 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fda64009ed0 tx=0x7fda64007660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:47.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.925+0000 7fda6affd640 1 --2- 172.21.15.12:0/1655098520 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda4c03e240 0x7fda4c040700 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fda6c0a6050 tx=0x7fda5402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:47.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.925+0000 7fda63fff640 1 -- 172.21.15.12:0/1655098520 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6401a070 con 0x7fda6c123c20 2024-09-11T01:41:47.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.925+0000 7fda63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:47.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.925+0000 7fda63fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:41:47.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.925+0000 7fda63fff640 10 monclient: dump: 2024-09-11T01:41:47.939 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:47.939 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:47.939 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.939 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.939 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:47.939 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:47.939 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:47.939 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:47.939 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:47.939 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:47.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.925+0000 7fda63fff640 1 -- 172.21.15.12:0/1655098520 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fda640056f0 con 0x7fda6c123c20 2024-09-11T01:41:47.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.925+0000 7fda63fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:41:47.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.925+0000 7fda63fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:41:47.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.925+0000 7fda63fff640 1 -- 172.21.15.12:0/1655098520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c123c20 msgr2=0x7fda6c124000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:47.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.925+0000 7fda63fff640 1 --2- 172.21.15.12:0/1655098520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c123c20 0x7fda6c124000 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fda64009ed0 tx=0x7fda64007660 comp rx=0 tx=0).stop 2024-09-11T01:41:47.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.925+0000 7fda63fff640 1 -- 172.21.15.12:0/1655098520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c123c20 msgr2=0x7fda6c124000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:41:47.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.925+0000 7fda63fff640 1 --2- 172.21.15.12:0/1655098520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c123c20 0x7fda6c124000 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fda64009ed0 tx=0x7fda64007660 comp rx=0 tx=0).stop 2024-09-11T01:41:47.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.931+0000 7fda72105640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:41:47.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.931+0000 7fda72105640 1 --2- 172.21.15.12:0/1655098520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c124540 0x7fda6c09f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:41:47.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.931+0000 7fda72105640 10 monclient(hunting): start opening mon connection 2024-09-11T01:41:47.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.931+0000 7fda72105640 1 -- 172.21.15.12:0/1655098520 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fda6c125d00 con 0x7fda6c124540 2024-09-11T01:41:47.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.931+0000 7fda6b7fe640 1 --2- 172.21.15.12:0/1655098520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c124540 0x7fda6c09f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:41:47.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.931+0000 7fda6b7fe640 10 monclient: get_auth_request con 0x7fda6c124540 auth_method 0 2024-09-11T01:41:47.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.931+0000 7fda6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:41:47.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.931+0000 7fda6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:41:47.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.931+0000 7fda6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:41:47.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.931+0000 7fda6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:41:47.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.931+0000 7fda6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:41:47.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.932+0000 7fda6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:41:47.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.932+0000 7fda6b7fe640 10 monclient(hunting): handle_auth_done global_id 134170 payload 824 2024-09-11T01:41:47.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.932+0000 7fda6b7fe640 1 -- 172.21.15.12:0/1655098520 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fda6c1cdd50 con 0x7fda6c124540 2024-09-11T01:41:47.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.934+0000 7fda6b7fe640 1 -- 172.21.15.12:0/1655098520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c124540 msgr2=0x7fda6c09f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:41:47.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.934+0000 7fda6b7fe640 1 --2- 172.21.15.12:0/1655098520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c124540 0x7fda6c09f050 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fda64009ed0 tx=0x7fda64005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:41:47.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.935+0000 7fda63fff640 1 -- 172.21.15.12:0/1655098520 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6401a030 con 0x7fda6c124540 2024-09-11T01:41:47.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.935+0000 7fda63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:41:47.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.935+0000 7fda63fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:41:47.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.935+0000 7fda63fff640 10 monclient: dump: 2024-09-11T01:41:47.942 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:41:47.942 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:41:47.942 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.942 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:41:47.942 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:41:47.942 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:41:47.942 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:41:47.942 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:41:47.942 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:41:47.942 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:41:47.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.935+0000 7fda63fff640 1 -- 172.21.15.12:0/1655098520 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4697 (secure 0 0 0) 0x7fda6401b040 con 0x7fda6c124540 2024-09-11T01:41:47.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.935+0000 7fda63fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:41:47.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.935+0000 7fda63fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:41:47.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.935+0000 7fda63fff640 1 -- 172.21.15.12:0/1655098520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c124540 msgr2=0x7fda6c09f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:47.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.935+0000 7fda63fff640 1 --2- 172.21.15.12:0/1655098520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c124540 0x7fda6c09f050 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fda64009ed0 tx=0x7fda64005d50 comp rx=0 tx=0).stop 2024-09-11T01:41:47.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.935+0000 7fda63fff640 1 -- 172.21.15.12:0/1655098520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c124540 msgr2=0x7fda6c09f050 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:41:47.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.935+0000 7fda63fff640 1 --2- 172.21.15.12:0/1655098520 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c124540 0x7fda6c09f050 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fda64009ed0 tx=0x7fda64005d50 comp rx=0 tx=0).stop 2024-09-11T01:41:47.943 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:41:47.943 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:41:47.943 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:41:47.943 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:41:47.944 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 118, 2024-09-11T01:41:47.944 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 40897, 2024-09-11T01:41:47.944 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:41:47.944 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:41:47.944 INFO:teuthology.orchestra.run.smithi012.stdout: 1 2024-09-11T01:41:47.944 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:47.944 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:41:47.944 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:41:47.944 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:41:47.944 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:41:47.944 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:41:47.944 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:41:47.944 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:41:47.944 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:41:47.945 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:41:47.945 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:41:47.945 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:41:47.945 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:41:47.945 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:41:47.945 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:41:47.945 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:41:47.945 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:47.945 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:41:47.945 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:41:47.945 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:41:47.945 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:41:47.945 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:41:47.945 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:41:47.946 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:41:47.946 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:41:47.946 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:41:47.946 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:41:47.946 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:41:47.946 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:41:47.946 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:41:47.946 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:47.946 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:47.946 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:41:47.946 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:41:47.946 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:41:47.946 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:41:47.947 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:41:47.947 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:41:47.947 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:41:47.947 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:41:47.947 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:41:47.947 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:41:47.947 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:41:47.947 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:41:47.947 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:41:47.947 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:41:47.947 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:41:47.947 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:41:47.947 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:41:47.948 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:41:47.948 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:41:47.948 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:41:47.948 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:41:47.948 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:41:47.948 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:41:47.948 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:41:47.948 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:41:47.948 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:41:47.948 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:41:47.948 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:41:47.948 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:47.948 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:41:47.949 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:47.949 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:41:47.949 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:47.949 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:41:47.949 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:41:47.949 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:41:47.949 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:41:47.949 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:47.949 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:41:47.949 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:41:47.949 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:47.949 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:47.949 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:47.949 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:41:47.950 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:41:47.950 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:47.950 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:47.950 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:47.950 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:47.950 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:41:47.950 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:41:47.950 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:41:47.950 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:41:47.950 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:41:47.950 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:47.950 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:47.950 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:41:47.950 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:41:47.951 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:41:47.951 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:41:47.951 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:47.951 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:41:47.951 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:41:47.951 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:47.951 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:47.951 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:47.951 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:41:47.951 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:41:47.951 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:47.951 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:47.951 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:47.952 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:47.952 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:41:47.952 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:41:47.952 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:41:47.952 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:41:47.952 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:41:47.952 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:47.952 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:47.952 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:41:47.952 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:41:47.952 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:41:47.952 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:41:47.952 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:47.953 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:41:47.953 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:41:47.953 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:47.953 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:47.953 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:47.953 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:41:47.953 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:41:47.953 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:41:47.953 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:47.953 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:47.953 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:47.953 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:41:47.953 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:41:47.953 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:41:47.954 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:41:47.954 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:41:47.954 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:47.954 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:47.954 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:47.954 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:41:47.954 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:41:47.954 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:47.954 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:41:47.954 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:41:47.954 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:41:47.954 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:47.954 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:47.955 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:41:47.955 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:47.955 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:41:47.955 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:41:47.955 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:41:47.955 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:47.955 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:41:47.955 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:41:47.955 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:41:47.955 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:41:47.955 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:41:47.955 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:41:47.955 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:41:47.956 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:41:47.956 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:41:47.956 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:41:47.956 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:41:47.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.935+0000 7fda72105640 1 -- 172.21.15.12:0/1655098520 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda4c03e240 msgr2=0x7fda4c040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:47.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.935+0000 7fda72105640 1 --2- 172.21.15.12:0/1655098520 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda4c03e240 0x7fda4c040700 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fda6c0a6050 tx=0x7fda5402e000 comp rx=0 tx=0).stop 2024-09-11T01:41:47.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.935+0000 7fda72105640 10 monclient: shutdown 2024-09-11T01:41:47.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.935+0000 7fda72105640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:41:47.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.935+0000 7fda72105640 1 -- 172.21.15.12:0/1655098520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda6c138130 msgr2=0x7fda6c1236e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:41:47.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.935+0000 7fda72105640 1 --2- 172.21.15.12:0/1655098520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda6c138130 0x7fda6c1236e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fda5c00e7e0 tx=0x7fda5c00eca0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.935+0000 7fda6bfff640 1 -- 172.21.15.12:0/1655098520 reap_dead start 2024-09-11T01:41:47.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.936+0000 7fda72105640 1 -- 172.21.15.12:0/1655098520 shutdown_connections 2024-09-11T01:41:47.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.936+0000 7fda72105640 1 --2- 172.21.15.12:0/1655098520 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda6c138130 0x7fda6c1236e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:41:47.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.936+0000 7fda72105640 1 -- 172.21.15.12:0/1655098520 >> 172.21.15.12:0/1655098520 conn(0x7fda6c0a7020 msgr2=0x7fda6c0a4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:41:47.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.936+0000 7fda72105640 1 -- 172.21.15.12:0/1655098520 shutdown_connections 2024-09-11T01:41:47.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:41:47.936+0000 7fda72105640 1 -- 172.21.15.12:0/1655098520 wait complete. 2024-09-11T01:41:47.958 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:41:47.958 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:41:48.359 INFO:tasks.rados.rados.0.smithi071.stdout:1783: done (4 left) 2024-09-11T01:41:48.359 INFO:tasks.rados.rados.0.smithi071.stdout:1784: done (3 left) 2024-09-11T01:41:48.359 INFO:tasks.rados.rados.0.smithi071.stdout:1785: done (2 left) 2024-09-11T01:41:48.359 INFO:tasks.rados.rados.0.smithi071.stdout:1786: done (1 left) 2024-09-11T01:41:48.359 INFO:tasks.rados.rados.0.smithi071.stdout:1787: done (0 left) 2024-09-11T01:41:48.360 INFO:tasks.rados.rados.0.smithi071.stdout:1788: write (excl) oid 22 current snap is 200 2024-09-11T01:41:48.360 INFO:tasks.rados.rados.0.smithi071.stdout:1788: seq_num 653 ranges {574971=627602,1862271=598221} 2024-09-11T01:41:48.365 INFO:tasks.rados.rados.0.smithi071.stdout:1788: writing smithi07134777-22 from 574971 to 1202573 tid 1 2024-09-11T01:41:48.372 INFO:tasks.rados.rados.0.smithi071.stdout:1788: writing smithi07134777-22 from 1862271 to 2460492 tid 2 2024-09-11T01:41:48.372 INFO:tasks.rados.rados.0.smithi071.stdout:1789: snap_remove snap 190 2024-09-11T01:41:49.746 INFO:tasks.rados.rados.0.smithi071.stdout:1789: done (1 left) 2024-09-11T01:41:49.747 INFO:tasks.rados.rados.0.smithi071.stdout:1790: snap_remove snap 187 2024-09-11T01:41:50.757 INFO:tasks.rados.rados.0.smithi071.stdout:1788: finishing write tid 1 to smithi07134777-22 2024-09-11T01:41:50.757 INFO:tasks.rados.rados.0.smithi071.stdout:1788: finishing write tid 2 to smithi07134777-22 2024-09-11T01:41:50.757 INFO:tasks.rados.rados.0.smithi071.stdout:1788: finishing write tid 3 to smithi07134777-22 2024-09-11T01:41:50.757 INFO:tasks.rados.rados.0.smithi071.stdout:1788: finishing write tid 4 to smithi07134777-22 2024-09-11T01:41:50.757 INFO:tasks.rados.rados.0.smithi071.stdout:1788: oid 22 updating version 0 to 458 2024-09-11T01:41:50.757 INFO:tasks.rados.rados.0.smithi071.stdout:1788: oid 22 updating version 458 to 459 2024-09-11T01:41:50.757 INFO:tasks.rados.rados.0.smithi071.stdout:1788: oid 22 version 459 is already newer than 457 2024-09-11T01:41:50.757 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 459 (ObjNum 653 snap 200 seq_num 653) dirty exists 2024-09-11T01:41:50.758 INFO:tasks.rados.rados.0.smithi071.stdout:1788: left oid 22 (ObjNum 653 snap 200 seq_num 653) 2024-09-11T01:41:50.758 INFO:tasks.rados.rados.0.smithi071.stdout:1788: done (1 left) 2024-09-11T01:41:50.758 INFO:tasks.rados.rados.0.smithi071.stdout:1790: done (0 left) 2024-09-11T01:41:50.758 INFO:tasks.rados.rados.0.smithi071.stdout:1791: snap_remove snap 194 2024-09-11T01:41:51.773 INFO:tasks.rados.rados.0.smithi071.stdout:1791: done (0 left) 2024-09-11T01:41:51.773 INFO:tasks.rados.rados.0.smithi071.stdout:1792: read oid 39 snap -1 2024-09-11T01:41:51.773 INFO:tasks.rados.rados.0.smithi071.stdout:1792: expect deleted 2024-09-11T01:41:51.773 INFO:tasks.rados.rados.0.smithi071.stdout:1793: read oid 6 snap 191 2024-09-11T01:41:51.773 INFO:tasks.rados.rados.0.smithi071.stdout:1793: expect deleted 2024-09-11T01:41:51.773 INFO:tasks.rados.rados.0.smithi071.stdout:1794: rollback oid 36 current snap is 200 2024-09-11T01:41:51.773 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 36 to 197 2024-09-11T01:41:51.773 INFO:tasks.rados.rados.0.smithi071.stdout:1795: write (excl) oid 20 current snap is 200 2024-09-11T01:41:51.773 INFO:tasks.rados.rados.0.smithi071.stdout:1795: seq_num 654 ranges {591778=419204,1637106=741879,2910049=81729} 2024-09-11T01:41:51.778 INFO:tasks.rados.rados.0.smithi071.stdout:1795: writing smithi07134777-20 from 591778 to 1010982 tid 1 2024-09-11T01:41:51.786 INFO:tasks.rados.rados.0.smithi071.stdout:1795: writing smithi07134777-20 from 1637106 to 2378985 tid 2 2024-09-11T01:41:51.790 INFO:tasks.rados.rados.0.smithi071.stdout:1795: writing smithi07134777-20 from 2910049 to 2991778 tid 3 2024-09-11T01:41:51.790 INFO:tasks.rados.rados.0.smithi071.stdout:1796: snap_create 2024-09-11T01:41:51.790 INFO:tasks.rados.rados.0.smithi071.stdout:1795: finishing write tid 1 to smithi07134777-20 2024-09-11T01:41:51.790 INFO:tasks.rados.rados.0.smithi071.stdout:1794: finishing rollback tid 0 to smithi07134777-36 2024-09-11T01:41:51.790 INFO:tasks.rados.rados.0.smithi071.stdout:1794: finishing rollback tid 1 to smithi07134777-36 2024-09-11T01:41:51.790 INFO:tasks.rados.rados.0.smithi071.stdout:1794: finishing rollback tid 2 to smithi07134777-36 2024-09-11T01:41:51.790 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 906 (ObjNum 612 snap 181 seq_num 612) dirty exists 2024-09-11T01:41:51.791 INFO:tasks.rados.rados.0.smithi071.stdout:1795: finishing write tid 2 to smithi07134777-20 2024-09-11T01:41:51.792 INFO:tasks.rados.rados.0.smithi071.stdout:1795: finishing write tid 3 to smithi07134777-20 2024-09-11T01:41:51.793 INFO:tasks.rados.rados.0.smithi071.stdout:1795: finishing write tid 4 to smithi07134777-20 2024-09-11T01:41:51.793 INFO:tasks.rados.rados.0.smithi071.stdout:1795: finishing write tid 5 to smithi07134777-20 2024-09-11T01:41:51.793 INFO:tasks.rados.rados.0.smithi071.stdout:1795: oid 20 updating version 0 to 351 2024-09-11T01:41:51.793 INFO:tasks.rados.rados.0.smithi071.stdout:1795: oid 20 updating version 351 to 352 2024-09-11T01:41:51.794 INFO:tasks.rados.rados.0.smithi071.stdout:1795: oid 20 updating version 352 to 353 2024-09-11T01:41:51.794 INFO:tasks.rados.rados.0.smithi071.stdout:1795: oid 20 version 353 is already newer than 350 2024-09-11T01:41:51.794 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 353 (ObjNum 654 snap 200 seq_num 654) dirty exists 2024-09-11T01:41:51.794 INFO:tasks.rados.rados.0.smithi071.stdout:1795: left oid 20 (ObjNum 654 snap 200 seq_num 654) 2024-09-11T01:41:52.767 INFO:tasks.rados.rados.0.smithi071.stdout:1792: done (4 left) 2024-09-11T01:41:52.767 INFO:tasks.rados.rados.0.smithi071.stdout:1793: done (3 left) 2024-09-11T01:41:52.767 INFO:tasks.rados.rados.0.smithi071.stdout:1794: done (2 left) 2024-09-11T01:41:52.767 INFO:tasks.rados.rados.0.smithi071.stdout:1795: done (1 left) 2024-09-11T01:41:52.767 INFO:tasks.rados.rados.0.smithi071.stdout:1796: done (0 left) 2024-09-11T01:41:52.767 INFO:tasks.rados.rados.0.smithi071.stdout:1797: snap_remove snap 193 2024-09-11T01:41:53.775 INFO:tasks.rados.rados.0.smithi071.stdout:1797: done (0 left) 2024-09-11T01:41:53.775 INFO:tasks.rados.rados.0.smithi071.stdout:1798: read oid 10 snap -1 2024-09-11T01:41:53.775 INFO:tasks.rados.rados.0.smithi071.stdout:1798: expect (ObjNum 601 snap 172 seq_num 601) 2024-09-11T01:41:53.775 INFO:tasks.rados.rados.0.smithi071.stdout:1799: read oid 4 snap 180 2024-09-11T01:41:53.775 INFO:tasks.rados.rados.0.smithi071.stdout:1799: expect deleted 2024-09-11T01:41:53.775 INFO:tasks.rados.rados.0.smithi071.stdout:1800: read oid 28 snap 196 2024-09-11T01:41:53.775 INFO:tasks.rados.rados.0.smithi071.stdout:1800: expect (ObjNum 636 snap 193 seq_num 636) 2024-09-11T01:41:53.775 INFO:tasks.rados.rados.0.smithi071.stdout:1801: delete oid 29 current snap is 201 2024-09-11T01:41:53.780 INFO:tasks.rados.rados.0.smithi071.stdout:1801: done (3 left) 2024-09-11T01:41:53.780 INFO:tasks.rados.rados.0.smithi071.stdout:1802: write oid 20 current snap is 201 2024-09-11T01:41:53.781 INFO:tasks.rados.rados.0.smithi071.stdout:1802: seq_num 656 ranges {625392=569702,1637717=745548,2848542=176850} 2024-09-11T01:41:53.787 INFO:tasks.rados.rados.0.smithi071.stdout:1802: writing smithi07134777-20 from 625392 to 1195094 tid 1 2024-09-11T01:41:53.794 INFO:tasks.rados.rados.0.smithi071.stdout:1802: writing smithi07134777-20 from 1637717 to 2383265 tid 2 2024-09-11T01:41:53.798 INFO:tasks.rados.rados.0.smithi071.stdout:1802: writing smithi07134777-20 from 2848542 to 3025392 tid 3 2024-09-11T01:41:53.798 INFO:tasks.rados.rados.0.smithi071.stdout:1803: read oid 13 snap -1 2024-09-11T01:41:53.798 INFO:tasks.rados.rados.0.smithi071.stdout:1803: expect (ObjNum 632 snap 190 seq_num 632) 2024-09-11T01:41:53.798 INFO:tasks.rados.rados.0.smithi071.stdout:1804: write oid 2 current snap is 201 2024-09-11T01:41:53.798 INFO:tasks.rados.rados.0.smithi071.stdout:1804: seq_num 657 ranges {642199=644951,1696199=605559,2992318=49881} 2024-09-11T01:41:53.805 INFO:tasks.rados.rados.0.smithi071.stdout:1804: writing smithi07134777-2 from 642199 to 1287150 tid 1 2024-09-11T01:41:53.811 INFO:tasks.rados.rados.0.smithi071.stdout:1804: writing smithi07134777-2 from 1696199 to 2301758 tid 2 2024-09-11T01:41:53.816 INFO:tasks.rados.rados.0.smithi071.stdout:1804: writing smithi07134777-2 from 2992318 to 3042199 tid 3 2024-09-11T01:41:53.816 INFO:tasks.rados.rados.0.smithi071.stdout:1805: copy_from oid 8 from oid 27 current snap is 201 2024-09-11T01:41:53.816 INFO:tasks.rados.rados.0.smithi071.stdout:1798: expect (ObjNum 601 snap 172 seq_num 601) 2024-09-11T01:41:53.816 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:41:53.844 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:41:53.880 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1404119 2024-09-11T01:41:53.892 INFO:tasks.rados.rados.0.smithi071.stdout:1798: done (6 left) 2024-09-11T01:41:53.892 INFO:tasks.rados.rados.0.smithi071.stdout:1799: done (5 left) 2024-09-11T01:41:53.892 INFO:tasks.rados.rados.0.smithi071.stdout:1806: write oid 22 current snap is 201 2024-09-11T01:41:53.893 INFO:tasks.rados.rados.0.smithi071.stdout:1806: seq_num 658 ranges {659006=720200,2154681=749217} 2024-09-11T01:41:53.899 INFO:tasks.rados.rados.0.smithi071.stdout:1806: writing smithi07134777-22 from 659006 to 1379206 tid 1 2024-09-11T01:41:53.907 INFO:tasks.rados.rados.0.smithi071.stdout:1806: writing smithi07134777-22 from 2154681 to 2903898 tid 2 2024-09-11T01:41:53.907 INFO:tasks.rados.rados.0.smithi071.stdout:1807: delete oid 1 current snap is 201 2024-09-11T01:41:53.907 INFO:tasks.rados.rados.0.smithi071.stdout:1800: expect (ObjNum 636 snap 193 seq_num 636) 2024-09-11T01:41:53.907 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 127888 2024-09-11T01:41:53.937 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:41:53.985 INFO:tasks.rados.rados.0.smithi071.stdout:1802: finishing write tid 1 to smithi07134777-20 2024-09-11T01:41:53.985 INFO:tasks.rados.rados.0.smithi071.stdout:1802: finishing write tid 2 to smithi07134777-20 2024-09-11T01:41:53.985 INFO:tasks.rados.rados.0.smithi071.stdout:1800: done (6 left) 2024-09-11T01:41:53.985 INFO:tasks.rados.rados.0.smithi071.stdout:1807: done (5 left) 2024-09-11T01:41:53.985 INFO:tasks.rados.rados.0.smithi071.stdout:1808: read oid 16 snap -1 2024-09-11T01:41:53.985 INFO:tasks.rados.rados.0.smithi071.stdout:1808: expect (ObjNum 618 snap 183 seq_num 618) 2024-09-11T01:41:53.986 INFO:tasks.rados.rados.0.smithi071.stdout:1809: delete oid 33 current snap is 201 2024-09-11T01:41:53.986 INFO:tasks.rados.rados.0.smithi071.stdout:1802: finishing write tid 3 to smithi07134777-20 2024-09-11T01:41:53.986 INFO:tasks.rados.rados.0.smithi071.stdout:1802: finishing write tid 4 to smithi07134777-20 2024-09-11T01:41:53.986 INFO:tasks.rados.rados.0.smithi071.stdout:1803: expect (ObjNum 632 snap 190 seq_num 632) 2024-09-11T01:41:53.987 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 463916 2024-09-11T01:41:54.095 INFO:tasks.rados.rados.0.smithi071.stdout:1802: finishing write tid 5 to smithi07134777-20 2024-09-11T01:41:54.095 INFO:tasks.rados.rados.0.smithi071.stdout:1802: oid 20 updating version 0 to 359 2024-09-11T01:41:54.095 INFO:tasks.rados.rados.0.smithi071.stdout:1802: oid 20 updating version 359 to 362 2024-09-11T01:41:54.095 INFO:tasks.rados.rados.0.smithi071.stdout:1802: oid 20 version 362 is already newer than 360 2024-09-11T01:41:54.095 INFO:tasks.rados.rados.0.smithi071.stdout:1802: oid 20 version 362 is already newer than 361 2024-09-11T01:41:54.095 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 362 (ObjNum 656 snap 201 seq_num 656) dirty exists 2024-09-11T01:41:54.095 INFO:tasks.rados.rados.0.smithi071.stdout:1802: left oid 20 (ObjNum 656 snap 201 seq_num 656) 2024-09-11T01:41:54.095 INFO:tasks.rados.rados.0.smithi071.stdout:1804: finishing write tid 1 to smithi07134777-2 2024-09-11T01:41:54.096 INFO:tasks.rados.rados.0.smithi071.stdout:1804: finishing write tid 2 to smithi07134777-2 2024-09-11T01:41:54.096 INFO:tasks.rados.rados.0.smithi071.stdout:1804: finishing write tid 3 to smithi07134777-2 2024-09-11T01:41:54.096 INFO:tasks.rados.rados.0.smithi071.stdout:1804: finishing write tid 4 to smithi07134777-2 2024-09-11T01:41:54.096 INFO:tasks.rados.rados.0.smithi071.stdout:1804: finishing write tid 5 to smithi07134777-2 2024-09-11T01:41:54.096 INFO:tasks.rados.rados.0.smithi071.stdout:1804: oid 2 updating version 0 to 822 2024-09-11T01:41:54.096 INFO:tasks.rados.rados.0.smithi071.stdout:1804: oid 2 updating version 822 to 823 2024-09-11T01:41:54.096 INFO:tasks.rados.rados.0.smithi071.stdout:1804: oid 2 version 823 is already newer than 821 2024-09-11T01:41:54.096 INFO:tasks.rados.rados.0.smithi071.stdout:1804: oid 2 updating version 823 to 824 2024-09-11T01:41:54.096 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 824 (ObjNum 657 snap 201 seq_num 657) dirty exists 2024-09-11T01:41:54.097 INFO:tasks.rados.rados.0.smithi071.stdout:1804: left oid 2 (ObjNum 657 snap 201 seq_num 657) 2024-09-11T01:41:54.097 INFO:tasks.rados.rados.0.smithi071.stdout:1805: finishing copy_from racing read to smithi07134777-8 2024-09-11T01:41:54.097 INFO:tasks.rados.rados.0.smithi071.stdout:1805: finishing copy_from to smithi07134777-8 2024-09-11T01:41:54.097 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 998 (ObjNum 586 snap 168 seq_num 586) dirty exists 2024-09-11T01:41:54.097 INFO:tasks.rados.rados.0.smithi071.stdout:1806: finishing write tid 1 to smithi07134777-22 2024-09-11T01:41:54.097 INFO:tasks.rados.rados.0.smithi071.stdout:1806: finishing write tid 2 to smithi07134777-22 2024-09-11T01:41:54.097 INFO:tasks.rados.rados.0.smithi071.stdout:1806: finishing write tid 3 to smithi07134777-22 2024-09-11T01:41:54.097 INFO:tasks.rados.rados.0.smithi071.stdout:1806: finishing write tid 4 to smithi07134777-22 2024-09-11T01:41:54.098 INFO:tasks.rados.rados.0.smithi071.stdout:1806: oid 22 updating version 0 to 462 2024-09-11T01:41:54.098 INFO:tasks.rados.rados.0.smithi071.stdout:1806: oid 22 version 462 is already newer than 461 2024-09-11T01:41:54.098 INFO:tasks.rados.rados.0.smithi071.stdout:1806: oid 22 version 462 is already newer than 460 2024-09-11T01:41:54.098 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 462 (ObjNum 658 snap 201 seq_num 658) dirty exists 2024-09-11T01:41:54.098 INFO:tasks.rados.rados.0.smithi071.stdout:1806: left oid 22 (ObjNum 658 snap 201 seq_num 658) 2024-09-11T01:41:54.098 INFO:tasks.rados.rados.0.smithi071.stdout:1802: done (6 left) 2024-09-11T01:41:54.098 INFO:tasks.rados.rados.0.smithi071.stdout:1803: done (5 left) 2024-09-11T01:41:54.098 INFO:tasks.rados.rados.0.smithi071.stdout:1804: done (4 left) 2024-09-11T01:41:54.098 INFO:tasks.rados.rados.0.smithi071.stdout:1805: done (3 left) 2024-09-11T01:41:54.099 INFO:tasks.rados.rados.0.smithi071.stdout:1806: done (2 left) 2024-09-11T01:41:54.099 INFO:tasks.rados.rados.0.smithi071.stdout:1809: done (1 left) 2024-09-11T01:41:54.099 INFO:tasks.rados.rados.0.smithi071.stdout:1810: snap_remove snap 192 2024-09-11T01:41:54.971 INFO:tasks.rados.rados.0.smithi071.stdout:1810: done (1 left) 2024-09-11T01:41:54.971 INFO:tasks.rados.rados.0.smithi071.stdout:1811: write (excl) oid 29 current snap is 201 2024-09-11T01:41:54.971 INFO:tasks.rados.rados.0.smithi071.stdout:1811: seq_num 661 ranges {709427=545947,1813774=612897,2985657=123770} 2024-09-11T01:41:54.978 INFO:tasks.rados.rados.0.smithi071.stdout:1811: writing smithi07134777-29 from 709427 to 1255374 tid 1 2024-09-11T01:41:54.984 INFO:tasks.rados.rados.0.smithi071.stdout:1811: writing smithi07134777-29 from 1813774 to 2426671 tid 2 2024-09-11T01:41:54.988 INFO:tasks.rados.rados.0.smithi071.stdout:1811: writing smithi07134777-29 from 2985657 to 3109427 tid 3 2024-09-11T01:41:54.988 INFO:tasks.rados.rados.0.smithi071.stdout:1812: delete oid 2 current snap is 201 2024-09-11T01:41:54.988 INFO:tasks.rados.rados.0.smithi071.stdout:1808: expect (ObjNum 618 snap 183 seq_num 618) 2024-09-11T01:41:55.121 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1886450 2024-09-11T01:41:55.153 INFO:tasks.rados.rados.0.smithi071.stdout:1811: finishing write tid 1 to smithi07134777-29 2024-09-11T01:41:55.153 INFO:tasks.rados.rados.0.smithi071.stdout:1811: finishing write tid 2 to smithi07134777-29 2024-09-11T01:41:55.154 INFO:tasks.rados.rados.0.smithi071.stdout:1811: finishing write tid 3 to smithi07134777-29 2024-09-11T01:41:55.154 INFO:tasks.rados.rados.0.smithi071.stdout:1811: finishing write tid 4 to smithi07134777-29 2024-09-11T01:41:55.154 INFO:tasks.rados.rados.0.smithi071.stdout:1811: finishing write tid 5 to smithi07134777-29 2024-09-11T01:41:55.154 INFO:tasks.rados.rados.0.smithi071.stdout:1811: oid 29 updating version 0 to 1029 2024-09-11T01:41:55.154 INFO:tasks.rados.rados.0.smithi071.stdout:1811: oid 29 updating version 1029 to 1032 2024-09-11T01:41:55.154 INFO:tasks.rados.rados.0.smithi071.stdout:1811: oid 29 version 1032 is already newer than 1030 2024-09-11T01:41:55.154 INFO:tasks.rados.rados.0.smithi071.stdout:1811: oid 29 version 1032 is already newer than 1031 2024-09-11T01:41:55.154 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 1032 (ObjNum 661 snap 201 seq_num 661) dirty exists 2024-09-11T01:41:55.154 INFO:tasks.rados.rados.0.smithi071.stdout:1811: left oid 29 (ObjNum 661 snap 201 seq_num 661) 2024-09-11T01:41:55.154 INFO:tasks.rados.rados.0.smithi071.stdout:1808: done (2 left) 2024-09-11T01:41:55.154 INFO:tasks.rados.rados.0.smithi071.stdout:1811: done (1 left) 2024-09-11T01:41:55.154 INFO:tasks.rados.rados.0.smithi071.stdout:1812: done (0 left) 2024-09-11T01:41:55.154 INFO:tasks.rados.rados.0.smithi071.stdout:1813: snap_remove snap 171 2024-09-11T01:41:55.965 INFO:tasks.rados.rados.0.smithi071.stdout:1813: done (0 left) 2024-09-11T01:41:55.965 INFO:tasks.rados.rados.0.smithi071.stdout:1814: write oid 16 current snap is 201 2024-09-11T01:41:55.965 INFO:tasks.rados.rados.0.smithi071.stdout:1814: seq_num 663 ranges {743041=412798,1930738=616566,3040503=102538} 2024-09-11T01:41:55.972 INFO:tasks.rados.rados.0.smithi071.stdout:1814: writing smithi07134777-16 from 743041 to 1155839 tid 1 2024-09-11T01:41:55.980 INFO:tasks.rados.rados.0.smithi071.stdout:1814: writing smithi07134777-16 from 1930738 to 2547304 tid 2 2024-09-11T01:41:55.984 INFO:tasks.rados.rados.0.smithi071.stdout:1814: writing smithi07134777-16 from 3040503 to 3143041 tid 3 2024-09-11T01:41:55.984 INFO:tasks.rados.rados.0.smithi071.stdout:1815: copy_from oid 46 from oid 29 current snap is 201 2024-09-11T01:41:55.984 INFO:tasks.rados.rados.0.smithi071.stdout:1816: write (excl) oid 2 current snap is 201 2024-09-11T01:41:55.984 INFO:tasks.rados.rados.0.smithi071.stdout:1816: seq_num 664 ranges {759848=488047,1989220=760224} 2024-09-11T01:41:55.991 INFO:tasks.rados.rados.0.smithi071.stdout:1816: writing smithi07134777-2 from 759848 to 1247895 tid 1 2024-09-11T01:41:56.000 INFO:tasks.rados.rados.0.smithi071.stdout:1816: writing smithi07134777-2 from 1989220 to 2749444 tid 2 2024-09-11T01:41:56.000 INFO:tasks.rados.rados.0.smithi071.stdout:1817: write (excl) oid 41 current snap is 201 2024-09-11T01:41:56.000 INFO:tasks.rados.rados.0.smithi071.stdout:1814: finishing write tid 1 to smithi07134777-16 2024-09-11T01:41:56.000 INFO:tasks.rados.rados.0.smithi071.stdout:1814: finishing write tid 2 to smithi07134777-16 2024-09-11T01:41:56.000 INFO:tasks.rados.rados.0.smithi071.stdout:1814: finishing write tid 3 to smithi07134777-16 2024-09-11T01:41:56.000 INFO:tasks.rados.rados.0.smithi071.stdout:1817: seq_num 665 ranges {776655=563296,2047702=620235,3095349=81306} 2024-09-11T01:41:56.007 INFO:tasks.rados.rados.0.smithi071.stdout:1817: writing smithi07134777-41 from 776655 to 1339951 tid 1 2024-09-11T01:41:56.014 INFO:tasks.rados.rados.0.smithi071.stdout:1817: writing smithi07134777-41 from 2047702 to 2667937 tid 2 2024-09-11T01:41:56.018 INFO:tasks.rados.rados.0.smithi071.stdout:1817: writing smithi07134777-41 from 3095349 to 3176655 tid 3 2024-09-11T01:41:56.018 INFO:tasks.rados.rados.0.smithi071.stdout:1818: rollback oid 20 current snap is 201 2024-09-11T01:41:56.018 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 20 to 185 2024-09-11T01:41:56.018 INFO:tasks.rados.rados.0.smithi071.stdout:1814: finishing write tid 4 to smithi07134777-16 2024-09-11T01:41:56.018 INFO:tasks.rados.rados.0.smithi071.stdout:1814: finishing write tid 5 to smithi07134777-16 2024-09-11T01:41:56.018 INFO:tasks.rados.rados.0.smithi071.stdout:1814: oid 16 updating version 0 to 924 2024-09-11T01:41:56.019 INFO:tasks.rados.rados.0.smithi071.stdout:1814: oid 16 updating version 924 to 926 2024-09-11T01:41:56.019 INFO:tasks.rados.rados.0.smithi071.stdout:1814: oid 16 version 926 is already newer than 925 2024-09-11T01:41:56.019 INFO:tasks.rados.rados.0.smithi071.stdout:1814: oid 16 version 926 is already newer than 923 2024-09-11T01:41:56.019 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 926 (ObjNum 663 snap 201 seq_num 663) dirty exists 2024-09-11T01:41:56.019 INFO:tasks.rados.rados.0.smithi071.stdout:1814: left oid 16 (ObjNum 663 snap 201 seq_num 663) 2024-09-11T01:41:56.019 INFO:tasks.rados.rados.0.smithi071.stdout:1816: finishing write tid 1 to smithi07134777-2 2024-09-11T01:41:56.019 INFO:tasks.rados.rados.0.smithi071.stdout:1815: finishing copy_from racing read to smithi07134777-46 2024-09-11T01:41:56.019 INFO:tasks.rados.rados.0.smithi071.stdout:1815: finishing copy_from to smithi07134777-46 2024-09-11T01:41:56.019 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 1032 (ObjNum 661 snap 201 seq_num 661) dirty exists 2024-09-11T01:41:56.019 INFO:tasks.rados.rados.0.smithi071.stdout:1816: finishing write tid 2 to smithi07134777-2 2024-09-11T01:41:56.019 INFO:tasks.rados.rados.0.smithi071.stdout:1816: finishing write tid 3 to smithi07134777-2 2024-09-11T01:41:56.019 INFO:tasks.rados.rados.0.smithi071.stdout:1816: finishing write tid 4 to smithi07134777-2 2024-09-11T01:41:56.019 INFO:tasks.rados.rados.0.smithi071.stdout:1816: oid 2 updating version 0 to 827 2024-09-11T01:41:56.020 INFO:tasks.rados.rados.0.smithi071.stdout:1816: oid 2 version 827 is already newer than 826 2024-09-11T01:41:56.020 INFO:tasks.rados.rados.0.smithi071.stdout:1816: oid 2 updating version 827 to 828 2024-09-11T01:41:56.020 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 828 (ObjNum 664 snap 201 seq_num 664) dirty exists 2024-09-11T01:41:56.020 INFO:tasks.rados.rados.0.smithi071.stdout:1816: left oid 2 (ObjNum 664 snap 201 seq_num 664) 2024-09-11T01:41:56.020 INFO:tasks.rados.rados.0.smithi071.stdout:1817: finishing write tid 1 to smithi07134777-41 2024-09-11T01:41:56.020 INFO:tasks.rados.rados.0.smithi071.stdout:1814: done (4 left) 2024-09-11T01:41:56.020 INFO:tasks.rados.rados.0.smithi071.stdout:1815: done (3 left) 2024-09-11T01:41:56.020 INFO:tasks.rados.rados.0.smithi071.stdout:1816: done (2 left) 2024-09-11T01:41:56.020 INFO:tasks.rados.rados.0.smithi071.stdout:1819: read oid 16 snap -1 2024-09-11T01:41:56.020 INFO:tasks.rados.rados.0.smithi071.stdout:1819: expect (ObjNum 663 snap 201 seq_num 663) 2024-09-11T01:41:56.020 INFO:tasks.rados.rados.0.smithi071.stdout:1820: read oid 43 snap -1 2024-09-11T01:41:56.020 INFO:tasks.rados.rados.0.smithi071.stdout:1820: expect (ObjNum 633 snap 190 seq_num 633) 2024-09-11T01:41:56.020 INFO:tasks.rados.rados.0.smithi071.stdout:1821: snap_remove snap 197 2024-09-11T01:41:56.968 INFO:tasks.rados.rados.0.smithi071.stdout:1821: done (4 left) 2024-09-11T01:41:56.968 INFO:tasks.rados.rados.0.smithi071.stdout:1822: read oid 14 snap -1 2024-09-11T01:41:56.968 INFO:tasks.rados.rados.0.smithi071.stdout:1822: expect (ObjNum 542 snap 154 seq_num 542) 2024-09-11T01:41:56.968 INFO:tasks.rados.rados.0.smithi071.stdout:1823: snap_create 2024-09-11T01:41:56.968 INFO:tasks.rados.rados.0.smithi071.stdout:1817: finishing write tid 2 to smithi07134777-41 2024-09-11T01:41:56.968 INFO:tasks.rados.rados.0.smithi071.stdout:1818: finishing rollback tid 0 to smithi07134777-20 2024-09-11T01:41:56.968 INFO:tasks.rados.rados.0.smithi071.stdout:1817: finishing write tid 3 to smithi07134777-41 2024-09-11T01:41:56.968 INFO:tasks.rados.rados.0.smithi071.stdout:1817: finishing write tid 4 to smithi07134777-41 2024-09-11T01:41:56.968 INFO:tasks.rados.rados.0.smithi071.stdout:1817: finishing write tid 5 to smithi07134777-41 2024-09-11T01:41:56.968 INFO:tasks.rados.rados.0.smithi071.stdout:1817: oid 41 updating version 0 to 386 2024-09-11T01:41:56.969 INFO:tasks.rados.rados.0.smithi071.stdout:1817: oid 41 version 386 is already newer than 383 2024-09-11T01:41:56.969 INFO:tasks.rados.rados.0.smithi071.stdout:1817: oid 41 version 386 is already newer than 385 2024-09-11T01:41:56.969 INFO:tasks.rados.rados.0.smithi071.stdout:1817: oid 41 version 386 is already newer than 384 2024-09-11T01:41:56.969 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 386 (ObjNum 665 snap 201 seq_num 665) dirty exists 2024-09-11T01:41:56.969 INFO:tasks.rados.rados.0.smithi071.stdout:1817: left oid 41 (ObjNum 665 snap 201 seq_num 665) 2024-09-11T01:41:56.969 INFO:tasks.rados.rados.0.smithi071.stdout:1818: finishing rollback tid 1 to smithi07134777-20 2024-09-11T01:41:56.969 INFO:tasks.rados.rados.0.smithi071.stdout:1818: finishing rollback tid 2 to smithi07134777-20 2024-09-11T01:41:56.969 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 373 (ObjNum 513 snap 141 seq_num 513) dirty exists 2024-09-11T01:41:56.969 INFO:tasks.rados.rados.0.smithi071.stdout:1819: expect (ObjNum 663 snap 201 seq_num 663) 2024-09-11T01:41:57.107 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:41:57.175 INFO:tasks.rados.rados.0.smithi071.stdout:1820: expect (ObjNum 633 snap 190 seq_num 633) 2024-09-11T01:41:57.208 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1103835 2024-09-11T01:41:57.225 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:41:57.253 INFO:tasks.rados.rados.0.smithi071.stdout:1822: expect (ObjNum 542 snap 154 seq_num 542) 2024-09-11T01:41:57.285 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2622293 2024-09-11T01:41:57.977 INFO:tasks.rados.rados.0.smithi071.stdout:1817: done (5 left) 2024-09-11T01:41:57.977 INFO:tasks.rados.rados.0.smithi071.stdout:1818: done (4 left) 2024-09-11T01:41:57.977 INFO:tasks.rados.rados.0.smithi071.stdout:1819: done (3 left) 2024-09-11T01:41:57.977 INFO:tasks.rados.rados.0.smithi071.stdout:1820: done (2 left) 2024-09-11T01:41:57.977 INFO:tasks.rados.rados.0.smithi071.stdout:1822: done (1 left) 2024-09-11T01:41:57.977 INFO:tasks.rados.rados.0.smithi071.stdout:1823: done (0 left) 2024-09-11T01:41:57.978 INFO:tasks.rados.rados.0.smithi071.stdout:1824: write oid 34 current snap is 202 2024-09-11T01:41:57.978 INFO:tasks.rados.rados.0.smithi071.stdout:1824: seq_num 666 ranges {793462=638545,2106184=763893} 2024-09-11T01:41:57.984 INFO:tasks.rados.rados.0.smithi071.stdout:1824: writing smithi07134777-34 from 793462 to 1432007 tid 1 2024-09-11T01:41:57.992 INFO:tasks.rados.rados.0.smithi071.stdout:1824: writing smithi07134777-34 from 2106184 to 2870077 tid 2 2024-09-11T01:41:57.992 INFO:tasks.rados.rados.0.smithi071.stdout:1825: write (excl) oid 35 current snap is 202 2024-09-11T01:41:57.993 INFO:tasks.rados.rados.0.smithi071.stdout:1825: seq_num 667 ranges {410269=713794,1648313=507551,2917489=292780} 2024-09-11T01:41:57.998 INFO:tasks.rados.rados.0.smithi071.stdout:1825: writing smithi07134777-35 from 410269 to 1124063 tid 1 2024-09-11T01:41:58.004 INFO:tasks.rados.rados.0.smithi071.stdout:1825: writing smithi07134777-35 from 1648313 to 2155864 tid 2 2024-09-11T01:41:58.009 INFO:tasks.rados.rados.0.smithi071.stdout:1825: writing smithi07134777-35 from 2917489 to 3210269 tid 3 2024-09-11T01:41:58.009 INFO:tasks.rados.rados.0.smithi071.stdout:1826: write (excl) oid 14 current snap is 202 2024-09-11T01:41:58.010 INFO:tasks.rados.rados.0.smithi071.stdout:1826: seq_num 668 ranges {427076=789043,1706795=767562,2944912=282164} 2024-09-11T01:41:58.016 INFO:tasks.rados.rados.0.smithi071.stdout:1826: writing smithi07134777-14 from 427076 to 1216119 tid 1 2024-09-11T01:41:58.023 INFO:tasks.rados.rados.0.smithi071.stdout:1826: writing smithi07134777-14 from 1706795 to 2474357 tid 2 2024-09-11T01:41:58.028 INFO:tasks.rados.rados.0.smithi071.stdout:1826: writing smithi07134777-14 from 2944912 to 3227076 tid 3 2024-09-11T01:41:58.028 INFO:tasks.rados.rados.0.smithi071.stdout:1827: snap_create 2024-09-11T01:41:58.028 INFO:tasks.rados.rados.0.smithi071.stdout:1824: finishing write tid 1 to smithi07134777-34 2024-09-11T01:41:58.028 INFO:tasks.rados.rados.0.smithi071.stdout:1824: finishing write tid 2 to smithi07134777-34 2024-09-11T01:41:58.028 INFO:tasks.rados.rados.0.smithi071.stdout:1824: finishing write tid 3 to smithi07134777-34 2024-09-11T01:41:58.029 INFO:tasks.rados.rados.0.smithi071.stdout:1824: finishing write tid 4 to smithi07134777-34 2024-09-11T01:41:58.029 INFO:tasks.rados.rados.0.smithi071.stdout:1824: oid 34 updating version 0 to 1002 2024-09-11T01:41:58.029 INFO:tasks.rados.rados.0.smithi071.stdout:1824: oid 34 version 1002 is already newer than 1000 2024-09-11T01:41:58.029 INFO:tasks.rados.rados.0.smithi071.stdout:1824: oid 34 version 1002 is already newer than 1001 2024-09-11T01:41:58.029 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 1002 (ObjNum 666 snap 202 seq_num 666) dirty exists 2024-09-11T01:41:58.029 INFO:tasks.rados.rados.0.smithi071.stdout:1824: left oid 34 (ObjNum 666 snap 202 seq_num 666) 2024-09-11T01:41:58.029 INFO:tasks.rados.rados.0.smithi071.stdout:1825: finishing write tid 1 to smithi07134777-35 2024-09-11T01:41:58.029 INFO:tasks.rados.rados.0.smithi071.stdout:1825: finishing write tid 2 to smithi07134777-35 2024-09-11T01:41:58.029 INFO:tasks.rados.rados.0.smithi071.stdout:1825: finishing write tid 3 to smithi07134777-35 2024-09-11T01:41:58.029 INFO:tasks.rados.rados.0.smithi071.stdout:1825: finishing write tid 4 to smithi07134777-35 2024-09-11T01:41:58.029 INFO:tasks.rados.rados.0.smithi071.stdout:1825: finishing write tid 5 to smithi07134777-35 2024-09-11T01:41:58.029 INFO:tasks.rados.rados.0.smithi071.stdout:1825: oid 35 updating version 0 to 945 2024-09-11T01:41:58.029 INFO:tasks.rados.rados.0.smithi071.stdout:1825: oid 35 version 945 is already newer than 942 2024-09-11T01:41:58.030 INFO:tasks.rados.rados.0.smithi071.stdout:1825: oid 35 version 945 is already newer than 944 2024-09-11T01:41:58.030 INFO:tasks.rados.rados.0.smithi071.stdout:1825: oid 35 version 945 is already newer than 943 2024-09-11T01:41:58.030 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 945 (ObjNum 667 snap 202 seq_num 667) dirty exists 2024-09-11T01:41:58.030 INFO:tasks.rados.rados.0.smithi071.stdout:1825: left oid 35 (ObjNum 667 snap 202 seq_num 667) 2024-09-11T01:41:58.030 INFO:tasks.rados.rados.0.smithi071.stdout:1826: finishing write tid 1 to smithi07134777-14 2024-09-11T01:41:58.030 INFO:tasks.rados.rados.0.smithi071.stdout:1826: finishing write tid 2 to smithi07134777-14 2024-09-11T01:41:58.031 INFO:tasks.rados.rados.0.smithi071.stdout:1826: finishing write tid 3 to smithi07134777-14 2024-09-11T01:41:58.033 INFO:tasks.rados.rados.0.smithi071.stdout:1826: finishing write tid 4 to smithi07134777-14 2024-09-11T01:41:58.033 INFO:tasks.rados.rados.0.smithi071.stdout:1826: finishing write tid 5 to smithi07134777-14 2024-09-11T01:41:58.033 INFO:tasks.rados.rados.0.smithi071.stdout:1826: oid 14 updating version 0 to 1053 2024-09-11T01:41:58.033 INFO:tasks.rados.rados.0.smithi071.stdout:1826: oid 14 version 1053 is already newer than 1050 2024-09-11T01:41:58.033 INFO:tasks.rados.rados.0.smithi071.stdout:1826: oid 14 version 1053 is already newer than 1052 2024-09-11T01:41:58.033 INFO:tasks.rados.rados.0.smithi071.stdout:1826: oid 14 version 1053 is already newer than 1051 2024-09-11T01:41:58.034 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 1053 (ObjNum 668 snap 202 seq_num 668) dirty exists 2024-09-11T01:41:58.034 INFO:tasks.rados.rados.0.smithi071.stdout:1826: left oid 14 (ObjNum 668 snap 202 seq_num 668) 2024-09-11T01:41:58.958 INFO:tasks.rados.rados.0.smithi071.stdout:1824: done (3 left) 2024-09-11T01:41:58.958 INFO:tasks.rados.rados.0.smithi071.stdout:1825: done (2 left) 2024-09-11T01:41:58.959 INFO:tasks.rados.rados.0.smithi071.stdout:1826: done (1 left) 2024-09-11T01:41:58.959 INFO:tasks.rados.rados.0.smithi071.stdout:1827: done (0 left) 2024-09-11T01:41:58.959 INFO:tasks.rados.rados.0.smithi071.stdout:1828: rollback oid 17 current snap is 203 2024-09-11T01:41:58.959 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 17 to 198 2024-09-11T01:41:58.959 INFO:tasks.rados.rados.0.smithi071.stdout:1829: snap_remove snap 199 2024-09-11T01:41:59.964 INFO:tasks.rados.rados.0.smithi071.stdout:1829: done (1 left) 2024-09-11T01:41:59.965 INFO:tasks.rados.rados.0.smithi071.stdout:1830: delete oid 14 current snap is 203 2024-09-11T01:41:59.965 INFO:tasks.rados.rados.0.smithi071.stdout:1828: finishing rollback tid 0 to smithi07134777-17 2024-09-11T01:41:59.965 INFO:tasks.rados.rados.0.smithi071.stdout:1828: finishing rollback tid 1 to smithi07134777-17 2024-09-11T01:41:59.965 INFO:tasks.rados.rados.0.smithi071.stdout:1828: finishing rollback tid 2 to smithi07134777-17 2024-09-11T01:41:59.965 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 820 (ObjNum 551 snap 157 seq_num 551) dirty exists 2024-09-11T01:41:59.974 INFO:tasks.rados.rados.0.smithi071.stdout:1828: done (1 left) 2024-09-11T01:41:59.974 INFO:tasks.rados.rados.0.smithi071.stdout:1830: done (0 left) 2024-09-11T01:41:59.974 INFO:tasks.rados.rados.0.smithi071.stdout:1831: read oid 22 snap -1 2024-09-11T01:41:59.974 INFO:tasks.rados.rados.0.smithi071.stdout:1831: expect (ObjNum 658 snap 201 seq_num 658) 2024-09-11T01:41:59.974 INFO:tasks.rados.rados.0.smithi071.stdout:1832: delete oid 8 current snap is 203 2024-09-11T01:41:59.978 INFO:tasks.rados.rados.0.smithi071.stdout:1832: done (1 left) 2024-09-11T01:41:59.978 INFO:tasks.rados.rados.0.smithi071.stdout:1833: write (excl) oid 35 current snap is 203 2024-09-11T01:41:59.979 INFO:tasks.rados.rados.0.smithi071.stdout:1833: seq_num 671 ranges {477497=731143,1998594=514889,3143534=133963} 2024-09-11T01:41:59.985 INFO:tasks.rados.rados.0.smithi071.stdout:1833: writing smithi07134777-35 from 477497 to 1208640 tid 1 2024-09-11T01:41:59.992 INFO:tasks.rados.rados.0.smithi071.stdout:1833: writing smithi07134777-35 from 1998594 to 2513483 tid 2 2024-09-11T01:41:59.996 INFO:tasks.rados.rados.0.smithi071.stdout:1833: writing smithi07134777-35 from 3143534 to 3277497 tid 3 2024-09-11T01:41:59.996 INFO:tasks.rados.rados.0.smithi071.stdout:1834: write (excl) oid 49 current snap is 203 2024-09-11T01:41:59.996 INFO:tasks.rados.rados.0.smithi071.stdout:1834: seq_num 672 ranges {494304=406392,1540723=774900,2770957=523347} 2024-09-11T01:42:00.001 INFO:tasks.rados.rados.0.smithi071.stdout:1834: writing smithi07134777-49 from 494304 to 900696 tid 1 2024-09-11T01:42:00.008 INFO:tasks.rados.rados.0.smithi071.stdout:1834: writing smithi07134777-49 from 1540723 to 2315623 tid 2 2024-09-11T01:42:00.014 INFO:tasks.rados.rados.0.smithi071.stdout:1834: writing smithi07134777-49 from 2770957 to 3294304 tid 3 2024-09-11T01:42:00.014 INFO:tasks.rados.rados.0.smithi071.stdout:1835: snap_create 2024-09-11T01:42:00.014 INFO:tasks.rados.rados.0.smithi071.stdout:1831: expect (ObjNum 658 snap 201 seq_num 658) 2024-09-11T01:42:00.014 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:00.061 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:00.148 INFO:tasks.rados.rados.0.smithi071.stdout:1833: finishing write tid 1 to smithi07134777-35 2024-09-11T01:42:00.148 INFO:tasks.rados.rados.0.smithi071.stdout:1833: finishing write tid 2 to smithi07134777-35 2024-09-11T01:42:00.148 INFO:tasks.rados.rados.0.smithi071.stdout:1833: finishing write tid 3 to smithi07134777-35 2024-09-11T01:42:00.148 INFO:tasks.rados.rados.0.smithi071.stdout:1833: finishing write tid 4 to smithi07134777-35 2024-09-11T01:42:00.148 INFO:tasks.rados.rados.0.smithi071.stdout:1833: finishing write tid 5 to smithi07134777-35 2024-09-11T01:42:00.148 INFO:tasks.rados.rados.0.smithi071.stdout:1833: oid 35 updating version 0 to 949 2024-09-11T01:42:00.148 INFO:tasks.rados.rados.0.smithi071.stdout:1833: oid 35 version 949 is already newer than 946 2024-09-11T01:42:00.148 INFO:tasks.rados.rados.0.smithi071.stdout:1833: oid 35 version 949 is already newer than 948 2024-09-11T01:42:00.148 INFO:tasks.rados.rados.0.smithi071.stdout:1833: oid 35 version 949 is already newer than 947 2024-09-11T01:42:00.148 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 949 (ObjNum 671 snap 203 seq_num 671) dirty exists 2024-09-11T01:42:00.149 INFO:tasks.rados.rados.0.smithi071.stdout:1833: left oid 35 (ObjNum 671 snap 203 seq_num 671) 2024-09-11T01:42:00.149 INFO:tasks.rados.rados.0.smithi071.stdout:1834: finishing write tid 1 to smithi07134777-49 2024-09-11T01:42:00.149 INFO:tasks.rados.rados.0.smithi071.stdout:1834: finishing write tid 2 to smithi07134777-49 2024-09-11T01:42:00.149 INFO:tasks.rados.rados.0.smithi071.stdout:1834: finishing write tid 3 to smithi07134777-49 2024-09-11T01:42:00.149 INFO:tasks.rados.rados.0.smithi071.stdout:1834: finishing write tid 4 to smithi07134777-49 2024-09-11T01:42:00.149 INFO:tasks.rados.rados.0.smithi071.stdout:1834: finishing write tid 5 to smithi07134777-49 2024-09-11T01:42:00.149 INFO:tasks.rados.rados.0.smithi071.stdout:1834: oid 49 updating version 0 to 362 2024-09-11T01:42:00.149 INFO:tasks.rados.rados.0.smithi071.stdout:1834: oid 49 version 362 is already newer than 360 2024-09-11T01:42:00.149 INFO:tasks.rados.rados.0.smithi071.stdout:1834: oid 49 version 362 is already newer than 361 2024-09-11T01:42:00.149 INFO:tasks.rados.rados.0.smithi071.stdout:1834: oid 49 updating version 362 to 363 2024-09-11T01:42:00.149 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 363 (ObjNum 672 snap 203 seq_num 672) dirty exists 2024-09-11T01:42:00.149 INFO:tasks.rados.rados.0.smithi071.stdout:1834: left oid 49 (ObjNum 672 snap 203 seq_num 672) 2024-09-11T01:42:00.983 INFO:tasks.rados.rados.0.smithi071.stdout:1831: done (3 left) 2024-09-11T01:42:00.983 INFO:tasks.rados.rados.0.smithi071.stdout:1833: done (2 left) 2024-09-11T01:42:00.983 INFO:tasks.rados.rados.0.smithi071.stdout:1834: done (1 left) 2024-09-11T01:42:00.983 INFO:tasks.rados.rados.0.smithi071.stdout:1835: done (0 left) 2024-09-11T01:42:00.983 INFO:tasks.rados.rados.0.smithi071.stdout:1836: read oid 21 snap -1 2024-09-11T01:42:00.983 INFO:tasks.rados.rados.0.smithi071.stdout:1836: expect (ObjNum 565 snap 160 seq_num 565) 2024-09-11T01:42:00.983 INFO:tasks.rados.rados.0.smithi071.stdout:1837: write oid 29 current snap is 204 2024-09-11T01:42:00.983 INFO:tasks.rados.rados.0.smithi071.stdout:1837: seq_num 673 ranges {511111=481641,1599205=518558,2682027=489947} 2024-09-11T01:42:00.988 INFO:tasks.rados.rados.0.smithi071.stdout:1837: writing smithi07134777-29 from 511111 to 992752 tid 1 2024-09-11T01:42:00.995 INFO:tasks.rados.rados.0.smithi071.stdout:1837: writing smithi07134777-29 from 1599205 to 2117763 tid 2 2024-09-11T01:42:01.001 INFO:tasks.rados.rados.0.smithi071.stdout:1837: writing smithi07134777-29 from 2682027 to 3171974 tid 3 2024-09-11T01:42:01.001 INFO:tasks.rados.rados.0.smithi071.stdout:1838: snap_remove snap 201 2024-09-11T01:42:01.988 INFO:tasks.rados.rados.0.smithi071.stdout:1838: done (2 left) 2024-09-11T01:42:01.988 INFO:tasks.rados.rados.0.smithi071.stdout:1839: write (excl) oid 7 current snap is 204 2024-09-11T01:42:01.988 INFO:tasks.rados.rados.0.smithi071.stdout:1839: seq_num 674 ranges {527918=556890,1657687=778569,3225803=102115} 2024-09-11T01:42:01.993 INFO:tasks.rados.rados.0.smithi071.stdout:1839: writing smithi07134777-7 from 527918 to 1084808 tid 1 2024-09-11T01:42:02.002 INFO:tasks.rados.rados.0.smithi071.stdout:1839: writing smithi07134777-7 from 1657687 to 2436256 tid 2 2024-09-11T01:42:02.007 INFO:tasks.rados.rados.0.smithi071.stdout:1839: writing smithi07134777-7 from 3225803 to 3327918 tid 3 2024-09-11T01:42:02.007 INFO:tasks.rados.rados.0.smithi071.stdout:1840: delete oid 48 current snap is 204 2024-09-11T01:42:02.007 INFO:tasks.rados.rados.0.smithi071.stdout:1836: expect (ObjNum 565 snap 160 seq_num 565) 2024-09-11T01:42:02.150 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3451441 2024-09-11T01:42:02.227 INFO:tasks.rados.rados.0.smithi071.stdout:1837: finishing write tid 1 to smithi07134777-29 2024-09-11T01:42:02.227 INFO:tasks.rados.rados.0.smithi071.stdout:1837: finishing write tid 2 to smithi07134777-29 2024-09-11T01:42:02.227 INFO:tasks.rados.rados.0.smithi071.stdout:1837: finishing write tid 3 to smithi07134777-29 2024-09-11T01:42:02.227 INFO:tasks.rados.rados.0.smithi071.stdout:1837: finishing write tid 4 to smithi07134777-29 2024-09-11T01:42:02.228 INFO:tasks.rados.rados.0.smithi071.stdout:1837: finishing write tid 5 to smithi07134777-29 2024-09-11T01:42:02.228 INFO:tasks.rados.rados.0.smithi071.stdout:1837: oid 29 updating version 0 to 1063 2024-09-11T01:42:02.228 INFO:tasks.rados.rados.0.smithi071.stdout:1837: oid 29 version 1063 is already newer than 1062 2024-09-11T01:42:02.228 INFO:tasks.rados.rados.0.smithi071.stdout:1837: oid 29 version 1063 is already newer than 1061 2024-09-11T01:42:02.228 INFO:tasks.rados.rados.0.smithi071.stdout:1837: oid 29 updating version 1063 to 1064 2024-09-11T01:42:02.228 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 1064 (ObjNum 673 snap 204 seq_num 673) dirty exists 2024-09-11T01:42:02.228 INFO:tasks.rados.rados.0.smithi071.stdout:1837: left oid 29 (ObjNum 673 snap 204 seq_num 673) 2024-09-11T01:42:02.228 INFO:tasks.rados.rados.0.smithi071.stdout:1839: finishing write tid 1 to smithi07134777-7 2024-09-11T01:42:02.228 INFO:tasks.rados.rados.0.smithi071.stdout:1839: finishing write tid 2 to smithi07134777-7 2024-09-11T01:42:02.228 INFO:tasks.rados.rados.0.smithi071.stdout:1839: finishing write tid 3 to smithi07134777-7 2024-09-11T01:42:02.228 INFO:tasks.rados.rados.0.smithi071.stdout:1839: finishing write tid 4 to smithi07134777-7 2024-09-11T01:42:02.228 INFO:tasks.rados.rados.0.smithi071.stdout:1836: done (3 left) 2024-09-11T01:42:02.229 INFO:tasks.rados.rados.0.smithi071.stdout:1837: done (2 left) 2024-09-11T01:42:02.229 INFO:tasks.rados.rados.0.smithi071.stdout:1840: done (1 left) 2024-09-11T01:42:02.229 INFO:tasks.rados.rados.0.smithi071.stdout:1841: delete oid 49 current snap is 204 2024-09-11T01:42:02.229 INFO:tasks.rados.rados.0.smithi071.stdout:1839: finishing write tid 5 to smithi07134777-7 2024-09-11T01:42:02.229 INFO:tasks.rados.rados.0.smithi071.stdout:1839: oid 7 updating version 0 to 1066 2024-09-11T01:42:02.229 INFO:tasks.rados.rados.0.smithi071.stdout:1839: oid 7 updating version 1066 to 1071 2024-09-11T01:42:02.229 INFO:tasks.rados.rados.0.smithi071.stdout:1839: oid 7 updating version 1071 to 1073 2024-09-11T01:42:02.229 INFO:tasks.rados.rados.0.smithi071.stdout:1839: oid 7 version 1073 is already newer than 1072 2024-09-11T01:42:02.229 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 1073 (ObjNum 674 snap 204 seq_num 674) dirty exists 2024-09-11T01:42:02.229 INFO:tasks.rados.rados.0.smithi071.stdout:1839: left oid 7 (ObjNum 674 snap 204 seq_num 674) 2024-09-11T01:42:02.230 INFO:tasks.rados.rados.0.smithi071.stdout:1839: done (1 left) 2024-09-11T01:42:02.230 INFO:tasks.rados.rados.0.smithi071.stdout:1841: done (0 left) 2024-09-11T01:42:02.230 INFO:tasks.rados.rados.0.smithi071.stdout:1842: write (excl) oid 5 current snap is 204 2024-09-11T01:42:02.230 INFO:tasks.rados.rados.0.smithi071.stdout:1842: seq_num 677 ranges {578339=498990,1833133=525896,2791719=586620} 2024-09-11T01:42:02.236 INFO:tasks.rados.rados.0.smithi071.stdout:1842: writing smithi07134777-5 from 578339 to 1077329 tid 1 2024-09-11T01:42:02.243 INFO:tasks.rados.rados.0.smithi071.stdout:1842: writing smithi07134777-5 from 1833133 to 2359029 tid 2 2024-09-11T01:42:02.248 INFO:tasks.rados.rados.0.smithi071.stdout:1842: writing smithi07134777-5 from 2791719 to 3378339 tid 3 2024-09-11T01:42:02.248 INFO:tasks.rados.rados.0.smithi071.stdout:1843: snap_remove snap 181 2024-09-11T01:42:02.993 INFO:tasks.rados.rados.0.smithi071.stdout:1843: done (1 left) 2024-09-11T01:42:02.993 INFO:tasks.rados.rados.0.smithi071.stdout:1844: rollback oid 35 current snap is 204 2024-09-11T01:42:02.993 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 35 to 180 2024-09-11T01:42:02.993 INFO:tasks.rados.rados.0.smithi071.stdout:1842: finishing write tid 1 to smithi07134777-5 2024-09-11T01:42:02.993 INFO:tasks.rados.rados.0.smithi071.stdout:1842: finishing write tid 2 to smithi07134777-5 2024-09-11T01:42:02.993 INFO:tasks.rados.rados.0.smithi071.stdout:1842: finishing write tid 3 to smithi07134777-5 2024-09-11T01:42:02.993 INFO:tasks.rados.rados.0.smithi071.stdout:1842: finishing write tid 4 to smithi07134777-5 2024-09-11T01:42:02.993 INFO:tasks.rados.rados.0.smithi071.stdout:1842: finishing write tid 5 to smithi07134777-5 2024-09-11T01:42:02.993 INFO:tasks.rados.rados.0.smithi071.stdout:1842: oid 5 updating version 0 to 498 2024-09-11T01:42:02.993 INFO:tasks.rados.rados.0.smithi071.stdout:1842: oid 5 updating version 498 to 499 2024-09-11T01:42:02.994 INFO:tasks.rados.rados.0.smithi071.stdout:1842: oid 5 version 499 is already newer than 497 2024-09-11T01:42:02.994 INFO:tasks.rados.rados.0.smithi071.stdout:1842: oid 5 updating version 499 to 500 2024-09-11T01:42:02.994 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 500 (ObjNum 677 snap 204 seq_num 677) dirty exists 2024-09-11T01:42:02.994 INFO:tasks.rados.rados.0.smithi071.stdout:1842: left oid 5 (ObjNum 677 snap 204 seq_num 677) 2024-09-11T01:42:02.994 INFO:tasks.rados.rados.0.smithi071.stdout:1842: done (1 left) 2024-09-11T01:42:02.994 INFO:tasks.rados.rados.0.smithi071.stdout:1845: rollback oid 7 current snap is 204 2024-09-11T01:42:02.994 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 7 to 191 2024-09-11T01:42:02.994 INFO:tasks.rados.rados.0.smithi071.stdout:1846: read oid 17 snap -1 2024-09-11T01:42:02.994 INFO:tasks.rados.rados.0.smithi071.stdout:1846: expect (ObjNum 551 snap 157 seq_num 551) 2024-09-11T01:42:02.994 INFO:tasks.rados.rados.0.smithi071.stdout:1847: copy_from oid 24 from oid 40 current snap is 204 2024-09-11T01:42:02.994 INFO:tasks.rados.rados.0.smithi071.stdout:1848: read oid 29 snap -1 2024-09-11T01:42:02.994 INFO:tasks.rados.rados.0.smithi071.stdout:1848: expect (ObjNum 673 snap 204 seq_num 673) 2024-09-11T01:42:02.994 INFO:tasks.rados.rados.0.smithi071.stdout:1849: rollback oid 19 current snap is 204 2024-09-11T01:42:02.994 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 19 to 196 2024-09-11T01:42:02.995 INFO:tasks.rados.rados.0.smithi071.stdout:1850: read oid 2 snap 191 2024-09-11T01:42:02.995 INFO:tasks.rados.rados.0.smithi071.stdout:1850: expect (ObjNum 622 snap 187 seq_num 622) 2024-09-11T01:42:02.995 INFO:tasks.rados.rados.0.smithi071.stdout:1851: rollback oid 49 current snap is 204 2024-09-11T01:42:02.995 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 49 to 198 2024-09-11T01:42:02.995 INFO:tasks.rados.rados.0.smithi071.stdout:1852: write oid 3 current snap is 204 2024-09-11T01:42:02.995 INFO:tasks.rados.rados.0.smithi071.stdout:1852: seq_num 678 ranges {595146=574239,1891615=669554,3219142=176004} 2024-09-11T01:42:02.998 INFO:tasks.rados.rados.0.smithi071.stdout:1852: writing smithi07134777-3 from 595146 to 1169385 tid 1 2024-09-11T01:42:03.007 INFO:tasks.rados.rados.0.smithi071.stdout:1852: writing smithi07134777-3 from 1891615 to 2561169 tid 2 2024-09-11T01:42:03.012 INFO:tasks.rados.rados.0.smithi071.stdout:1852: writing smithi07134777-3 from 3219142 to 3395146 tid 3 2024-09-11T01:42:03.013 INFO:tasks.rados.rados.0.smithi071.stdout:1853: rollback oid 9 current snap is 204 2024-09-11T01:42:03.013 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 9 to 198 2024-09-11T01:42:03.013 INFO:tasks.rados.rados.0.smithi071.stdout:1854: write oid 21 current snap is 204 2024-09-11T01:42:03.013 INFO:tasks.rados.rados.0.smithi071.stdout:1854: seq_num 679 ranges {611953=649488,1950097=529565,3246565=165388} 2024-09-11T01:42:03.020 INFO:tasks.rados.rados.0.smithi071.stdout:1854: writing smithi07134777-21 from 611953 to 1261441 tid 1 2024-09-11T01:42:03.027 INFO:tasks.rados.rados.0.smithi071.stdout:1854: writing smithi07134777-21 from 1950097 to 2479662 tid 2 2024-09-11T01:42:03.032 INFO:tasks.rados.rados.0.smithi071.stdout:1854: writing smithi07134777-21 from 3246565 to 3411953 tid 3 2024-09-11T01:42:03.033 INFO:tasks.rados.rados.0.smithi071.stdout:1855: delete oid 27 current snap is 204 2024-09-11T01:42:03.033 INFO:tasks.rados.rados.0.smithi071.stdout:1851: finishing rollback tid 1 to smithi07134777-49 2024-09-11T01:42:03.033 INFO:tasks.rados.rados.0.smithi071.stdout:1852: finishing write tid 1 to smithi07134777-3 2024-09-11T01:42:03.033 INFO:tasks.rados.rados.0.smithi071.stdout:1851: finishing rollback tid 2 to smithi07134777-49 2024-09-11T01:42:03.033 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 366 (ObjNum 631 snap 190 seq_num 631) dirty exists 2024-09-11T01:42:03.034 INFO:tasks.rados.rados.0.smithi071.stdout:1850: expect (ObjNum 622 snap 187 seq_num 622) 2024-09-11T01:42:03.087 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:03.190 INFO:tasks.rados.rados.0.smithi071.stdout:1845: finishing rollback tid 0 to smithi07134777-7 2024-09-11T01:42:03.190 INFO:tasks.rados.rados.0.smithi071.stdout:1845: finishing rollback tid 1 to smithi07134777-7 2024-09-11T01:42:03.190 INFO:tasks.rados.rados.0.smithi071.stdout:1845: finishing rollback tid 2 to smithi07134777-7 2024-09-11T01:42:03.190 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 1078 (ObjNum 581 snap 166 seq_num 581) dirty exists 2024-09-11T01:42:03.190 INFO:tasks.rados.rados.0.smithi071.stdout:1849: finishing rollback tid 0 to smithi07134777-19 2024-09-11T01:42:03.190 INFO:tasks.rados.rados.0.smithi071.stdout:1849: finishing rollback tid 1 to smithi07134777-19 2024-09-11T01:42:03.190 INFO:tasks.rados.rados.0.smithi071.stdout:1849: finishing rollback tid 2 to smithi07134777-19 2024-09-11T01:42:03.191 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 1002 (ObjNum 640 snap 195 seq_num 640) dirty exists 2024-09-11T01:42:03.191 INFO:tasks.rados.rados.0.smithi071.stdout:1844: finishing rollback tid 0 to smithi07134777-35 2024-09-11T01:42:03.191 INFO:tasks.rados.rados.0.smithi071.stdout:1844: finishing rollback tid 1 to smithi07134777-35 2024-09-11T01:42:03.191 INFO:tasks.rados.rados.0.smithi071.stdout:1844: finishing rollback tid 2 to smithi07134777-35 2024-09-11T01:42:03.191 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 952 (ObjNum 591 snap 169 seq_num 591) dirty exists 2024-09-11T01:42:03.191 INFO:tasks.rados.rados.0.smithi071.stdout:1848: expect (ObjNum 673 snap 204 seq_num 673) 2024-09-11T01:42:03.191 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 151723 2024-09-11T01:42:03.323 INFO:tasks.rados.rados.0.smithi071.stdout:1846: expect (ObjNum 551 snap 157 seq_num 551) 2024-09-11T01:42:03.332 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1710299 2024-09-11T01:42:03.347 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:03.390 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2228599 2024-09-11T01:42:03.405 INFO:tasks.rados.rados.0.smithi071.stdout:1852: finishing write tid 2 to smithi07134777-3 2024-09-11T01:42:03.405 INFO:tasks.rados.rados.0.smithi071.stdout:1852: finishing write tid 3 to smithi07134777-3 2024-09-11T01:42:03.406 INFO:tasks.rados.rados.0.smithi071.stdout:1852: finishing write tid 4 to smithi07134777-3 2024-09-11T01:42:03.406 INFO:tasks.rados.rados.0.smithi071.stdout:1847: finishing copy_from racing read to smithi07134777-24 2024-09-11T01:42:03.406 INFO:tasks.rados.rados.0.smithi071.stdout:1844: done (11 left) 2024-09-11T01:42:03.406 INFO:tasks.rados.rados.0.smithi071.stdout:1845: done (10 left) 2024-09-11T01:42:03.406 INFO:tasks.rados.rados.0.smithi071.stdout:1846: done (9 left) 2024-09-11T01:42:03.406 INFO:tasks.rados.rados.0.smithi071.stdout:1848: done (8 left) 2024-09-11T01:42:03.406 INFO:tasks.rados.rados.0.smithi071.stdout:1849: done (7 left) 2024-09-11T01:42:03.406 INFO:tasks.rados.rados.0.smithi071.stdout:1850: done (6 left) 2024-09-11T01:42:03.406 INFO:tasks.rados.rados.0.smithi071.stdout:1851: done (5 left) 2024-09-11T01:42:03.406 INFO:tasks.rados.rados.0.smithi071.stdout:1855: done (4 left) 2024-09-11T01:42:03.406 INFO:tasks.rados.rados.0.smithi071.stdout:1856: write oid 5 current snap is 204 2024-09-11T01:42:03.406 INFO:tasks.rados.rados.0.smithi071.stdout:1852: finishing write tid 5 to smithi07134777-3 2024-09-11T01:42:03.406 INFO:tasks.rados.rados.0.smithi071.stdout:1852: oid 3 updating version 0 to 463 2024-09-11T01:42:03.406 INFO:tasks.rados.rados.0.smithi071.stdout:1852: oid 3 updating version 463 to 464 2024-09-11T01:42:03.407 INFO:tasks.rados.rados.0.smithi071.stdout:1852: oid 3 updating version 464 to 466 2024-09-11T01:42:03.407 INFO:tasks.rados.rados.0.smithi071.stdout:1852: oid 3 version 466 is already newer than 465 2024-09-11T01:42:03.407 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 466 (ObjNum 678 snap 204 seq_num 678) dirty exists 2024-09-11T01:42:03.407 INFO:tasks.rados.rados.0.smithi071.stdout:1852: left oid 3 (ObjNum 678 snap 204 seq_num 678) 2024-09-11T01:42:03.407 INFO:tasks.rados.rados.0.smithi071.stdout:1853: finishing rollback tid 0 to smithi07134777-9 2024-09-11T01:42:03.407 INFO:tasks.rados.rados.0.smithi071.stdout:1853: finishing rollback tid 1 to smithi07134777-9 2024-09-11T01:42:03.407 INFO:tasks.rados.rados.0.smithi071.stdout:1853: finishing rollback tid 2 to smithi07134777-9 2024-09-11T01:42:03.407 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 538 (ObjNum 635 snap 192 seq_num 635) dirty exists 2024-09-11T01:42:03.407 INFO:tasks.rados.rados.0.smithi071.stdout:1856: seq_num 681 ranges {645567=799986,1950708=533234,3185058=260509} 2024-09-11T01:42:03.412 INFO:tasks.rados.rados.0.smithi071.stdout:1856: writing smithi07134777-5 from 645567 to 1445553 tid 1 2024-09-11T01:42:03.418 INFO:tasks.rados.rados.0.smithi071.stdout:1856: writing smithi07134777-5 from 1950708 to 2483942 tid 2 2024-09-11T01:42:03.424 INFO:tasks.rados.rados.0.smithi071.stdout:1856: writing smithi07134777-5 from 3185058 to 3445567 tid 3 2024-09-11T01:42:03.424 INFO:tasks.rados.rados.0.smithi071.stdout:1852: done (4 left) 2024-09-11T01:42:03.424 INFO:tasks.rados.rados.0.smithi071.stdout:1853: done (3 left) 2024-09-11T01:42:03.424 INFO:tasks.rados.rados.0.smithi071.stdout:1857: read oid 6 snap 186 2024-09-11T01:42:03.424 INFO:tasks.rados.rados.0.smithi071.stdout:1857: expect deleted 2024-09-11T01:42:03.424 INFO:tasks.rados.rados.0.smithi071.stdout:1858: write (excl) oid 37 current snap is 204 2024-09-11T01:42:03.424 INFO:tasks.rados.rados.0.smithi071.stdout:1847: finishing copy_from to smithi07134777-24 2024-09-11T01:42:03.424 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 700 (ObjNum 647 snap 199 seq_num 647) dirty exists 2024-09-11T01:42:03.424 INFO:tasks.rados.rados.0.smithi071.stdout:1854: finishing write tid 1 to smithi07134777-21 2024-09-11T01:42:03.424 INFO:tasks.rados.rados.0.smithi071.stdout:1854: finishing write tid 2 to smithi07134777-21 2024-09-11T01:42:03.425 INFO:tasks.rados.rados.0.smithi071.stdout:1854: finishing write tid 3 to smithi07134777-21 2024-09-11T01:42:03.425 INFO:tasks.rados.rados.0.smithi071.stdout:1854: finishing write tid 4 to smithi07134777-21 2024-09-11T01:42:03.425 INFO:tasks.rados.rados.0.smithi071.stdout:1854: finishing write tid 5 to smithi07134777-21 2024-09-11T01:42:03.425 INFO:tasks.rados.rados.0.smithi071.stdout:1854: oid 21 updating version 0 to 929 2024-09-11T01:42:03.425 INFO:tasks.rados.rados.0.smithi071.stdout:1854: oid 21 updating version 929 to 930 2024-09-11T01:42:03.425 INFO:tasks.rados.rados.0.smithi071.stdout:1854: oid 21 version 930 is already newer than 928 2024-09-11T01:42:03.425 INFO:tasks.rados.rados.0.smithi071.stdout:1854: oid 21 version 930 is already newer than 927 2024-09-11T01:42:03.425 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 930 (ObjNum 679 snap 204 seq_num 679) dirty exists 2024-09-11T01:42:03.425 INFO:tasks.rados.rados.0.smithi071.stdout:1854: left oid 21 (ObjNum 679 snap 204 seq_num 679) 2024-09-11T01:42:03.425 INFO:tasks.rados.rados.0.smithi071.stdout:1858: seq_num 682 ranges {662374=475235,1609190=676892,2812481=624101} 2024-09-11T01:42:03.429 INFO:tasks.rados.rados.0.smithi071.stdout:1858: writing smithi07134777-37 from 662374 to 1137609 tid 1 2024-09-11T01:42:03.435 INFO:tasks.rados.rados.0.smithi071.stdout:1858: writing smithi07134777-37 from 1609190 to 2286082 tid 2 2024-09-11T01:42:03.442 INFO:tasks.rados.rados.0.smithi071.stdout:1858: writing smithi07134777-37 from 2812481 to 3436582 tid 3 2024-09-11T01:42:03.442 INFO:tasks.rados.rados.0.smithi071.stdout:1847: done (4 left) 2024-09-11T01:42:03.442 INFO:tasks.rados.rados.0.smithi071.stdout:1854: done (3 left) 2024-09-11T01:42:03.442 INFO:tasks.rados.rados.0.smithi071.stdout:1859: delete oid 36 current snap is 204 2024-09-11T01:42:03.442 INFO:tasks.rados.rados.0.smithi071.stdout:1856: finishing write tid 1 to smithi07134777-5 2024-09-11T01:42:03.442 INFO:tasks.rados.rados.0.smithi071.stdout:1856: finishing write tid 2 to smithi07134777-5 2024-09-11T01:42:03.442 INFO:tasks.rados.rados.0.smithi071.stdout:1856: finishing write tid 3 to smithi07134777-5 2024-09-11T01:42:03.443 INFO:tasks.rados.rados.0.smithi071.stdout:1856: finishing write tid 4 to smithi07134777-5 2024-09-11T01:42:03.443 INFO:tasks.rados.rados.0.smithi071.stdout:1856: finishing write tid 5 to smithi07134777-5 2024-09-11T01:42:03.443 INFO:tasks.rados.rados.0.smithi071.stdout:1856: oid 5 updating version 0 to 504 2024-09-11T01:42:03.443 INFO:tasks.rados.rados.0.smithi071.stdout:1856: oid 5 version 504 is already newer than 502 2024-09-11T01:42:03.443 INFO:tasks.rados.rados.0.smithi071.stdout:1856: oid 5 version 504 is already newer than 501 2024-09-11T01:42:03.443 INFO:tasks.rados.rados.0.smithi071.stdout:1856: oid 5 version 504 is already newer than 503 2024-09-11T01:42:03.443 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 504 (ObjNum 681 snap 204 seq_num 681) dirty exists 2024-09-11T01:42:03.443 INFO:tasks.rados.rados.0.smithi071.stdout:1856: left oid 5 (ObjNum 681 snap 204 seq_num 681) 2024-09-11T01:42:03.443 INFO:tasks.rados.rados.0.smithi071.stdout:1858: finishing write tid 1 to smithi07134777-37 2024-09-11T01:42:03.443 INFO:tasks.rados.rados.0.smithi071.stdout:1858: finishing write tid 2 to smithi07134777-37 2024-09-11T01:42:03.445 INFO:tasks.rados.rados.0.smithi071.stdout:1856: done (3 left) 2024-09-11T01:42:03.445 INFO:tasks.rados.rados.0.smithi071.stdout:1857: done (2 left) 2024-09-11T01:42:03.445 INFO:tasks.rados.rados.0.smithi071.stdout:1859: done (1 left) 2024-09-11T01:42:03.445 INFO:tasks.rados.rados.0.smithi071.stdout:1860: write oid 39 current snap is 204 2024-09-11T01:42:03.445 INFO:tasks.rados.rados.0.smithi071.stdout:1860: seq_num 684 ranges {695988=625733,2009801=680561,3150974=345014} 2024-09-11T01:42:03.452 INFO:tasks.rados.rados.0.smithi071.stdout:1860: writing smithi07134777-39 from 695988 to 1321721 tid 1 2024-09-11T01:42:03.459 INFO:tasks.rados.rados.0.smithi071.stdout:1860: writing smithi07134777-39 from 2009801 to 2690362 tid 2 2024-09-11T01:42:03.463 INFO:tasks.rados.rados.0.smithi071.stdout:1860: writing smithi07134777-39 from 3150974 to 3495988 tid 3 2024-09-11T01:42:03.463 INFO:tasks.rados.rados.0.smithi071.stdout:1861: write oid 3 current snap is 204 2024-09-11T01:42:03.464 INFO:tasks.rados.rados.0.smithi071.stdout:1861: seq_num 685 ranges {712795=417335,1784636=540572,2894750=457917} 2024-09-11T01:42:03.469 INFO:tasks.rados.rados.0.smithi071.stdout:1861: writing smithi07134777-3 from 712795 to 1130130 tid 1 2024-09-11T01:42:03.476 INFO:tasks.rados.rados.0.smithi071.stdout:1861: writing smithi07134777-3 from 1784636 to 2325208 tid 2 2024-09-11T01:42:03.482 INFO:tasks.rados.rados.0.smithi071.stdout:1861: writing smithi07134777-3 from 2894750 to 3352667 tid 3 2024-09-11T01:42:03.482 INFO:tasks.rados.rados.0.smithi071.stdout:1862: read oid 24 snap 202 2024-09-11T01:42:03.482 INFO:tasks.rados.rados.0.smithi071.stdout:1862: expect (ObjNum 623 snap 187 seq_num 623) 2024-09-11T01:42:03.482 INFO:tasks.rados.rados.0.smithi071.stdout:1863: write (excl) oid 2 current snap is 204 2024-09-11T01:42:03.482 INFO:tasks.rados.rados.0.smithi071.stdout:1858: finishing write tid 3 to smithi07134777-37 2024-09-11T01:42:03.482 INFO:tasks.rados.rados.0.smithi071.stdout:1863: seq_num 686 ranges {729602=492584,1843118=684230,3322173=207429} 2024-09-11T01:42:03.489 INFO:tasks.rados.rados.0.smithi071.stdout:1863: writing smithi07134777-2 from 729602 to 1222186 tid 1 2024-09-11T01:42:03.495 INFO:tasks.rados.rados.0.smithi071.stdout:1863: writing smithi07134777-2 from 1843118 to 2527348 tid 2 2024-09-11T01:42:03.502 INFO:tasks.rados.rados.0.smithi071.stdout:1863: writing smithi07134777-2 from 3322173 to 3529602 tid 3 2024-09-11T01:42:03.502 INFO:tasks.rados.rados.0.smithi071.stdout:1864: write oid 36 current snap is 204 2024-09-11T01:42:03.502 INFO:tasks.rados.rados.0.smithi071.stdout:1858: finishing write tid 4 to smithi07134777-37 2024-09-11T01:42:03.502 INFO:tasks.rados.rados.0.smithi071.stdout:1858: finishing write tid 5 to smithi07134777-37 2024-09-11T01:42:03.502 INFO:tasks.rados.rados.0.smithi071.stdout:1858: oid 37 updating version 0 to 956 2024-09-11T01:42:03.502 INFO:tasks.rados.rados.0.smithi071.stdout:1858: oid 37 version 956 is already newer than 954 2024-09-11T01:42:03.502 INFO:tasks.rados.rados.0.smithi071.stdout:1858: oid 37 version 956 is already newer than 953 2024-09-11T01:42:03.502 INFO:tasks.rados.rados.0.smithi071.stdout:1858: oid 37 version 956 is already newer than 955 2024-09-11T01:42:03.502 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 956 (ObjNum 682 snap 204 seq_num 682) dirty exists 2024-09-11T01:42:03.502 INFO:tasks.rados.rados.0.smithi071.stdout:1858: left oid 37 (ObjNum 682 snap 204 seq_num 682) 2024-09-11T01:42:03.502 INFO:tasks.rados.rados.0.smithi071.stdout:1860: finishing write tid 1 to smithi07134777-39 2024-09-11T01:42:03.503 INFO:tasks.rados.rados.0.smithi071.stdout:1864: seq_num 687 ranges {746409=567833,1901600=544241,3065949=480460} 2024-09-11T01:42:03.508 INFO:tasks.rados.rados.0.smithi071.stdout:1864: writing smithi07134777-36 from 746409 to 1314242 tid 1 2024-09-11T01:42:03.514 INFO:tasks.rados.rados.0.smithi071.stdout:1864: writing smithi07134777-36 from 1901600 to 2445841 tid 2 2024-09-11T01:42:03.520 INFO:tasks.rados.rados.0.smithi071.stdout:1864: writing smithi07134777-36 from 3065949 to 3546409 tid 3 2024-09-11T01:42:03.521 INFO:tasks.rados.rados.0.smithi071.stdout:1858: done (5 left) 2024-09-11T01:42:03.521 INFO:tasks.rados.rados.0.smithi071.stdout:1865: write oid 1 current snap is 204 2024-09-11T01:42:03.521 INFO:tasks.rados.rados.0.smithi071.stdout:1860: finishing write tid 2 to smithi07134777-39 2024-09-11T01:42:03.521 INFO:tasks.rados.rados.0.smithi071.stdout:1860: finishing write tid 3 to smithi07134777-39 2024-09-11T01:42:03.521 INFO:tasks.rados.rados.0.smithi071.stdout:1860: finishing write tid 4 to smithi07134777-39 2024-09-11T01:42:03.521 INFO:tasks.rados.rados.0.smithi071.stdout:1860: finishing write tid 5 to smithi07134777-39 2024-09-11T01:42:03.521 INFO:tasks.rados.rados.0.smithi071.stdout:1860: oid 39 updating version 0 to 960 2024-09-11T01:42:03.521 INFO:tasks.rados.rados.0.smithi071.stdout:1860: oid 39 version 960 is already newer than 959 2024-09-11T01:42:03.521 INFO:tasks.rados.rados.0.smithi071.stdout:1860: oid 39 version 960 is already newer than 958 2024-09-11T01:42:03.521 INFO:tasks.rados.rados.0.smithi071.stdout:1860: oid 39 version 960 is already newer than 957 2024-09-11T01:42:03.521 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 960 (ObjNum 684 snap 204 seq_num 684) dirty exists 2024-09-11T01:42:03.521 INFO:tasks.rados.rados.0.smithi071.stdout:1860: left oid 39 (ObjNum 684 snap 204 seq_num 684) 2024-09-11T01:42:03.521 INFO:tasks.rados.rados.0.smithi071.stdout:1861: finishing write tid 1 to smithi07134777-3 2024-09-11T01:42:03.522 INFO:tasks.rados.rados.0.smithi071.stdout:1865: seq_num 688 ranges {763216=643082,1843729=687899,3260666=302550} 2024-09-11T01:42:03.527 INFO:tasks.rados.rados.0.smithi071.stdout:1865: writing smithi07134777-1 from 763216 to 1406298 tid 1 2024-09-11T01:42:03.534 INFO:tasks.rados.rados.0.smithi071.stdout:1865: writing smithi07134777-1 from 1843729 to 2531628 tid 2 2024-09-11T01:42:03.540 INFO:tasks.rados.rados.0.smithi071.stdout:1865: writing smithi07134777-1 from 3260666 to 3563216 tid 3 2024-09-11T01:42:03.540 INFO:tasks.rados.rados.0.smithi071.stdout:1860: done (5 left) 2024-09-11T01:42:03.541 INFO:tasks.rados.rados.0.smithi071.stdout:1866: read oid 44 snap -1 2024-09-11T01:42:03.541 INFO:tasks.rados.rados.0.smithi071.stdout:1866: expect (ObjNum 645 snap 196 seq_num 645) 2024-09-11T01:42:03.541 INFO:tasks.rados.rados.0.smithi071.stdout:1867: rollback oid 47 current snap is 204 2024-09-11T01:42:03.541 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 47 to 203 2024-09-11T01:42:03.541 INFO:tasks.rados.rados.0.smithi071.stdout:1861: finishing write tid 2 to smithi07134777-3 2024-09-11T01:42:03.541 INFO:tasks.rados.rados.0.smithi071.stdout:1861: finishing write tid 3 to smithi07134777-3 2024-09-11T01:42:03.541 INFO:tasks.rados.rados.0.smithi071.stdout:1861: finishing write tid 4 to smithi07134777-3 2024-09-11T01:42:03.541 INFO:tasks.rados.rados.0.smithi071.stdout:1861: finishing write tid 5 to smithi07134777-3 2024-09-11T01:42:03.541 INFO:tasks.rados.rados.0.smithi071.stdout:1861: oid 3 updating version 0 to 468 2024-09-11T01:42:03.541 INFO:tasks.rados.rados.0.smithi071.stdout:1861: oid 3 updating version 468 to 470 2024-09-11T01:42:03.541 INFO:tasks.rados.rados.0.smithi071.stdout:1861: oid 3 version 470 is already newer than 467 2024-09-11T01:42:03.541 INFO:tasks.rados.rados.0.smithi071.stdout:1861: oid 3 version 470 is already newer than 469 2024-09-11T01:42:03.541 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 470 (ObjNum 685 snap 204 seq_num 685) dirty exists 2024-09-11T01:42:03.542 INFO:tasks.rados.rados.0.smithi071.stdout:1861: left oid 3 (ObjNum 685 snap 204 seq_num 685) 2024-09-11T01:42:03.542 INFO:tasks.rados.rados.0.smithi071.stdout:1862: expect (ObjNum 623 snap 187 seq_num 623) 2024-09-11T01:42:03.589 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:03.685 INFO:tasks.rados.rados.0.smithi071.stdout:1863: finishing write tid 1 to smithi07134777-2 2024-09-11T01:42:03.685 INFO:tasks.rados.rados.0.smithi071.stdout:1863: finishing write tid 2 to smithi07134777-2 2024-09-11T01:42:03.686 INFO:tasks.rados.rados.0.smithi071.stdout:1863: finishing write tid 3 to smithi07134777-2 2024-09-11T01:42:03.686 INFO:tasks.rados.rados.0.smithi071.stdout:1863: finishing write tid 4 to smithi07134777-2 2024-09-11T01:42:03.686 INFO:tasks.rados.rados.0.smithi071.stdout:1863: finishing write tid 5 to smithi07134777-2 2024-09-11T01:42:03.686 INFO:tasks.rados.rados.0.smithi071.stdout:1863: oid 2 updating version 0 to 830 2024-09-11T01:42:03.686 INFO:tasks.rados.rados.0.smithi071.stdout:1863: oid 2 version 830 is already newer than 829 2024-09-11T01:42:03.686 INFO:tasks.rados.rados.0.smithi071.stdout:1863: oid 2 updating version 830 to 832 2024-09-11T01:42:03.686 INFO:tasks.rados.rados.0.smithi071.stdout:1863: oid 2 version 832 is already newer than 831 2024-09-11T01:42:03.686 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 832 (ObjNum 686 snap 204 seq_num 686) dirty exists 2024-09-11T01:42:03.686 INFO:tasks.rados.rados.0.smithi071.stdout:1863: left oid 2 (ObjNum 686 snap 204 seq_num 686) 2024-09-11T01:42:03.686 INFO:tasks.rados.rados.0.smithi071.stdout:1864: finishing write tid 1 to smithi07134777-36 2024-09-11T01:42:03.687 INFO:tasks.rados.rados.0.smithi071.stdout:1864: finishing write tid 2 to smithi07134777-36 2024-09-11T01:42:03.687 INFO:tasks.rados.rados.0.smithi071.stdout:1864: finishing write tid 3 to smithi07134777-36 2024-09-11T01:42:03.687 INFO:tasks.rados.rados.0.smithi071.stdout:1864: finishing write tid 4 to smithi07134777-36 2024-09-11T01:42:03.687 INFO:tasks.rados.rados.0.smithi071.stdout:1864: finishing write tid 5 to smithi07134777-36 2024-09-11T01:42:03.687 INFO:tasks.rados.rados.0.smithi071.stdout:1864: oid 36 updating version 0 to 1006 2024-09-11T01:42:03.687 INFO:tasks.rados.rados.0.smithi071.stdout:1864: oid 36 version 1006 is already newer than 1005 2024-09-11T01:42:03.687 INFO:tasks.rados.rados.0.smithi071.stdout:1864: oid 36 updating version 1006 to 1007 2024-09-11T01:42:03.687 INFO:tasks.rados.rados.0.smithi071.stdout:1864: oid 36 version 1007 is already newer than 1004 2024-09-11T01:42:03.687 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 1007 (ObjNum 687 snap 204 seq_num 687) dirty exists 2024-09-11T01:42:03.687 INFO:tasks.rados.rados.0.smithi071.stdout:1864: left oid 36 (ObjNum 687 snap 204 seq_num 687) 2024-09-11T01:42:03.687 INFO:tasks.rados.rados.0.smithi071.stdout:1865: finishing write tid 1 to smithi07134777-1 2024-09-11T01:42:03.687 INFO:tasks.rados.rados.0.smithi071.stdout:1865: finishing write tid 2 to smithi07134777-1 2024-09-11T01:42:03.687 INFO:tasks.rados.rados.0.smithi071.stdout:1861: done (6 left) 2024-09-11T01:42:03.687 INFO:tasks.rados.rados.0.smithi071.stdout:1862: done (5 left) 2024-09-11T01:42:03.688 INFO:tasks.rados.rados.0.smithi071.stdout:1863: done (4 left) 2024-09-11T01:42:03.688 INFO:tasks.rados.rados.0.smithi071.stdout:1864: done (3 left) 2024-09-11T01:42:03.688 INFO:tasks.rados.rados.0.smithi071.stdout:1868: write oid 35 current snap is 204 2024-09-11T01:42:03.688 INFO:tasks.rados.rados.0.smithi071.stdout:1868: seq_num 689 ranges {780023=718331,1902211=547910,3004442=575581} 2024-09-11T01:42:03.692 INFO:tasks.rados.rados.0.smithi071.stdout:1868: writing smithi07134777-35 from 780023 to 1498354 tid 1 2024-09-11T01:42:03.697 INFO:tasks.rados.rados.0.smithi071.stdout:1868: writing smithi07134777-35 from 1902211 to 2450121 tid 2 2024-09-11T01:42:03.704 INFO:tasks.rados.rados.0.smithi071.stdout:1868: writing smithi07134777-35 from 3004442 to 3580023 tid 3 2024-09-11T01:42:03.704 INFO:tasks.rados.rados.0.smithi071.stdout:1869: read oid 2 snap -1 2024-09-11T01:42:03.704 INFO:tasks.rados.rados.0.smithi071.stdout:1869: expect (ObjNum 686 snap 204 seq_num 686) 2024-09-11T01:42:03.704 INFO:tasks.rados.rados.0.smithi071.stdout:1870: write oid 12 current snap is 204 2024-09-11T01:42:03.704 INFO:tasks.rados.rados.0.smithi071.stdout:1870: seq_num 690 ranges {796830=793580,2360693=691568} 2024-09-11T01:42:03.712 INFO:tasks.rados.rados.0.smithi071.stdout:1870: writing smithi07134777-12 from 796830 to 1590410 tid 1 2024-09-11T01:42:03.720 INFO:tasks.rados.rados.0.smithi071.stdout:1870: writing smithi07134777-12 from 2360693 to 3052261 tid 2 2024-09-11T01:42:03.720 INFO:tasks.rados.rados.0.smithi071.stdout:1871: write oid 46 current snap is 204 2024-09-11T01:42:03.720 INFO:tasks.rados.rados.0.smithi071.stdout:1871: seq_num 691 ranges {413637=468829,1619175=435226,2542935=758255} 2024-09-11T01:42:03.725 INFO:tasks.rados.rados.0.smithi071.stdout:1871: writing smithi07134777-46 from 413637 to 882466 tid 1 2024-09-11T01:42:03.731 INFO:tasks.rados.rados.0.smithi071.stdout:1871: writing smithi07134777-46 from 1619175 to 2054401 tid 2 2024-09-11T01:42:03.738 INFO:tasks.rados.rados.0.smithi071.stdout:1871: writing smithi07134777-46 from 2542935 to 3301190 tid 3 2024-09-11T01:42:03.738 INFO:tasks.rados.rados.0.smithi071.stdout:1872: delete oid 30 current snap is 204 2024-09-11T01:42:03.739 INFO:tasks.rados.rados.0.smithi071.stdout:1865: finishing write tid 3 to smithi07134777-1 2024-09-11T01:42:03.739 INFO:tasks.rados.rados.0.smithi071.stdout:1866: expect (ObjNum 645 snap 196 seq_num 645) 2024-09-11T01:42:03.739 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:03.779 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2673286 2024-09-11T01:42:03.817 INFO:tasks.rados.rados.0.smithi071.stdout:1865: finishing write tid 4 to smithi07134777-1 2024-09-11T01:42:03.817 INFO:tasks.rados.rados.0.smithi071.stdout:1866: done (7 left) 2024-09-11T01:42:03.817 INFO:tasks.rados.rados.0.smithi071.stdout:1872: done (6 left) 2024-09-11T01:42:03.817 INFO:tasks.rados.rados.0.smithi071.stdout:1873: snap_create 2024-09-11T01:42:03.817 INFO:tasks.rados.rados.0.smithi071.stdout:1865: finishing write tid 5 to smithi07134777-1 2024-09-11T01:42:03.817 INFO:tasks.rados.rados.0.smithi071.stdout:1865: oid 1 updating version 0 to 822 2024-09-11T01:42:03.817 INFO:tasks.rados.rados.0.smithi071.stdout:1865: oid 1 updating version 822 to 823 2024-09-11T01:42:03.817 INFO:tasks.rados.rados.0.smithi071.stdout:1865: oid 1 version 823 is already newer than 821 2024-09-11T01:42:03.817 INFO:tasks.rados.rados.0.smithi071.stdout:1865: oid 1 updating version 823 to 824 2024-09-11T01:42:03.817 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 824 (ObjNum 688 snap 204 seq_num 688) dirty exists 2024-09-11T01:42:03.817 INFO:tasks.rados.rados.0.smithi071.stdout:1865: left oid 1 (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:42:03.818 INFO:tasks.rados.rados.0.smithi071.stdout:1867: finishing rollback tid 0 to smithi07134777-47 2024-09-11T01:42:03.818 INFO:tasks.rados.rados.0.smithi071.stdout:1867: finishing rollback tid 1 to smithi07134777-47 2024-09-11T01:42:03.818 INFO:tasks.rados.rados.0.smithi071.stdout:1867: finishing rollback tid 2 to smithi07134777-47 2024-09-11T01:42:03.818 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 827 (ObjNum 599 snap 171 seq_num 599) dirty exists 2024-09-11T01:42:03.818 INFO:tasks.rados.rados.0.smithi071.stdout:1868: finishing write tid 1 to smithi07134777-35 2024-09-11T01:42:03.818 INFO:tasks.rados.rados.0.smithi071.stdout:1868: finishing write tid 2 to smithi07134777-35 2024-09-11T01:42:03.818 INFO:tasks.rados.rados.0.smithi071.stdout:1869: expect (ObjNum 686 snap 204 seq_num 686) 2024-09-11T01:42:03.871 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 516624 2024-09-11T01:42:03.969 INFO:tasks.rados.rados.0.smithi071.stdout:1868: finishing write tid 3 to smithi07134777-35 2024-09-11T01:42:03.969 INFO:tasks.rados.rados.0.smithi071.stdout:1868: finishing write tid 4 to smithi07134777-35 2024-09-11T01:42:03.969 INFO:tasks.rados.rados.0.smithi071.stdout:1865: done (6 left) 2024-09-11T01:42:03.970 INFO:tasks.rados.rados.0.smithi071.stdout:1867: done (5 left) 2024-09-11T01:42:03.970 INFO:tasks.rados.rados.0.smithi071.stdout:1869: done (4 left) 2024-09-11T01:42:03.970 INFO:tasks.rados.rados.0.smithi071.stdout:1873: done (3 left) 2024-09-11T01:42:03.970 INFO:tasks.rados.rados.0.smithi071.stdout:1874: copy_from oid 20 from oid 5 current snap is 205 2024-09-11T01:42:03.970 INFO:tasks.rados.rados.0.smithi071.stdout:1868: finishing write tid 5 to smithi07134777-35 2024-09-11T01:42:03.970 INFO:tasks.rados.rados.0.smithi071.stdout:1868: oid 35 updating version 0 to 961 2024-09-11T01:42:03.970 INFO:tasks.rados.rados.0.smithi071.stdout:1868: oid 35 updating version 961 to 964 2024-09-11T01:42:03.970 INFO:tasks.rados.rados.0.smithi071.stdout:1868: oid 35 version 964 is already newer than 962 2024-09-11T01:42:03.970 INFO:tasks.rados.rados.0.smithi071.stdout:1868: oid 35 version 964 is already newer than 963 2024-09-11T01:42:03.970 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 964 (ObjNum 689 snap 204 seq_num 689) dirty exists 2024-09-11T01:42:03.970 INFO:tasks.rados.rados.0.smithi071.stdout:1868: left oid 35 (ObjNum 689 snap 204 seq_num 689) 2024-09-11T01:42:03.970 INFO:tasks.rados.rados.0.smithi071.stdout:1870: finishing write tid 1 to smithi07134777-12 2024-09-11T01:42:03.970 INFO:tasks.rados.rados.0.smithi071.stdout:1870: finishing write tid 2 to smithi07134777-12 2024-09-11T01:42:03.971 INFO:tasks.rados.rados.0.smithi071.stdout:1870: finishing write tid 3 to smithi07134777-12 2024-09-11T01:42:03.971 INFO:tasks.rados.rados.0.smithi071.stdout:1870: finishing write tid 4 to smithi07134777-12 2024-09-11T01:42:03.971 INFO:tasks.rados.rados.0.smithi071.stdout:1870: oid 12 updating version 0 to 1005 2024-09-11T01:42:03.971 INFO:tasks.rados.rados.0.smithi071.stdout:1870: oid 12 updating version 1005 to 1006 2024-09-11T01:42:03.971 INFO:tasks.rados.rados.0.smithi071.stdout:1870: oid 12 version 1006 is already newer than 1004 2024-09-11T01:42:03.971 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 1006 (ObjNum 690 snap 204 seq_num 690) dirty exists 2024-09-11T01:42:03.971 INFO:tasks.rados.rados.0.smithi071.stdout:1870: left oid 12 (ObjNum 690 snap 204 seq_num 690) 2024-09-11T01:42:03.971 INFO:tasks.rados.rados.0.smithi071.stdout:1871: finishing write tid 1 to smithi07134777-46 2024-09-11T01:42:03.971 INFO:tasks.rados.rados.0.smithi071.stdout:1871: finishing write tid 2 to smithi07134777-46 2024-09-11T01:42:03.971 INFO:tasks.rados.rados.0.smithi071.stdout:1871: finishing write tid 3 to smithi07134777-46 2024-09-11T01:42:03.971 INFO:tasks.rados.rados.0.smithi071.stdout:1871: finishing write tid 4 to smithi07134777-46 2024-09-11T01:42:03.972 INFO:tasks.rados.rados.0.smithi071.stdout:1871: finishing write tid 5 to smithi07134777-46 2024-09-11T01:42:03.972 INFO:tasks.rados.rados.0.smithi071.stdout:1871: oid 46 updating version 0 to 1033 2024-09-11T01:42:03.972 INFO:tasks.rados.rados.0.smithi071.stdout:1871: oid 46 updating version 1033 to 1035 2024-09-11T01:42:03.972 INFO:tasks.rados.rados.0.smithi071.stdout:1871: oid 46 version 1035 is already newer than 1034 2024-09-11T01:42:03.972 INFO:tasks.rados.rados.0.smithi071.stdout:1871: oid 46 updating version 1035 to 1036 2024-09-11T01:42:03.972 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 1036 (ObjNum 691 snap 204 seq_num 691) dirty exists 2024-09-11T01:42:03.972 INFO:tasks.rados.rados.0.smithi071.stdout:1871: left oid 46 (ObjNum 691 snap 204 seq_num 691) 2024-09-11T01:42:03.972 INFO:tasks.rados.rados.0.smithi071.stdout:1868: done (3 left) 2024-09-11T01:42:03.972 INFO:tasks.rados.rados.0.smithi071.stdout:1870: done (2 left) 2024-09-11T01:42:03.972 INFO:tasks.rados.rados.0.smithi071.stdout:1871: done (1 left) 2024-09-11T01:42:03.972 INFO:tasks.rados.rados.0.smithi071.stdout:1875: snap_create 2024-09-11T01:42:03.980 INFO:tasks.rados.rados.0.smithi071.stdout:1874: finishing copy_from racing read to smithi07134777-20 2024-09-11T01:42:03.981 INFO:tasks.rados.rados.0.smithi071.stdout:1874: finishing copy_from to smithi07134777-20 2024-09-11T01:42:03.981 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 504 (ObjNum 681 snap 204 seq_num 681) dirty exists 2024-09-11T01:42:04.961 INFO:tasks.rados.rados.0.smithi071.stdout:1874: done (1 left) 2024-09-11T01:42:04.961 INFO:tasks.rados.rados.0.smithi071.stdout:1875: done (0 left) 2024-09-11T01:42:04.961 INFO:tasks.rados.rados.0.smithi071.stdout:1876: read oid 49 snap -1 2024-09-11T01:42:04.961 INFO:tasks.rados.rados.0.smithi071.stdout:1876: expect (ObjNum 631 snap 190 seq_num 631) 2024-09-11T01:42:04.962 INFO:tasks.rados.rados.0.smithi071.stdout:1877: read oid 33 snap -1 2024-09-11T01:42:04.962 INFO:tasks.rados.rados.0.smithi071.stdout:1877: expect deleted 2024-09-11T01:42:04.962 INFO:tasks.rados.rados.0.smithi071.stdout:1878: write oid 44 current snap is 206 2024-09-11T01:42:04.962 INFO:tasks.rados.rados.0.smithi071.stdout:1878: seq_num 693 ranges {447251=735680,1736139=438895,2597781=780799} 2024-09-11T01:42:04.967 INFO:tasks.rados.rados.0.smithi071.stdout:1878: writing smithi07134777-44 from 447251 to 1182931 tid 1 2024-09-11T01:42:04.973 INFO:tasks.rados.rados.0.smithi071.stdout:1878: writing smithi07134777-44 from 1736139 to 2175034 tid 2 2024-09-11T01:42:04.980 INFO:tasks.rados.rados.0.smithi071.stdout:1878: writing smithi07134777-44 from 2597781 to 3378580 tid 3 2024-09-11T01:42:04.980 INFO:tasks.rados.rados.0.smithi071.stdout:1879: delete oid 10 current snap is 206 2024-09-11T01:42:04.980 INFO:tasks.rados.rados.0.smithi071.stdout:1876: expect (ObjNum 631 snap 190 seq_num 631) 2024-09-11T01:42:05.052 INFO:tasks.rados.rados.0.smithi071.stdout:1878: finishing write tid 1 to smithi07134777-44 2024-09-11T01:42:05.052 INFO:tasks.rados.rados.0.smithi071.stdout:1878: finishing write tid 2 to smithi07134777-44 2024-09-11T01:42:05.052 INFO:tasks.rados.rados.0.smithi071.stdout:1878: finishing write tid 3 to smithi07134777-44 2024-09-11T01:42:05.052 INFO:tasks.rados.rados.0.smithi071.stdout:1878: finishing write tid 4 to smithi07134777-44 2024-09-11T01:42:05.052 INFO:tasks.rados.rados.0.smithi071.stdout:1878: finishing write tid 5 to smithi07134777-44 2024-09-11T01:42:05.052 INFO:tasks.rados.rados.0.smithi071.stdout:1878: oid 44 updating version 0 to 506 2024-09-11T01:42:05.052 INFO:tasks.rados.rados.0.smithi071.stdout:1878: oid 44 updating version 506 to 508 2024-09-11T01:42:05.052 INFO:tasks.rados.rados.0.smithi071.stdout:1878: oid 44 version 508 is already newer than 505 2024-09-11T01:42:05.052 INFO:tasks.rados.rados.0.smithi071.stdout:1878: oid 44 version 508 is already newer than 507 2024-09-11T01:42:05.052 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 508 (ObjNum 693 snap 206 seq_num 693) dirty exists 2024-09-11T01:42:05.052 INFO:tasks.rados.rados.0.smithi071.stdout:1878: left oid 44 (ObjNum 693 snap 206 seq_num 693) 2024-09-11T01:42:05.053 INFO:tasks.rados.rados.0.smithi071.stdout:1876: done (3 left) 2024-09-11T01:42:05.053 INFO:tasks.rados.rados.0.smithi071.stdout:1877: done (2 left) 2024-09-11T01:42:05.053 INFO:tasks.rados.rados.0.smithi071.stdout:1878: done (1 left) 2024-09-11T01:42:05.053 INFO:tasks.rados.rados.0.smithi071.stdout:1879: done (0 left) 2024-09-11T01:42:05.053 INFO:tasks.rados.rados.0.smithi071.stdout:1880: copy_from oid 48 from oid 15 current snap is 206 2024-09-11T01:42:05.053 INFO:tasks.rados.rados.0.smithi071.stdout:1881: snap_create 2024-09-11T01:42:05.062 INFO:tasks.rados.rados.0.smithi071.stdout:1880: finishing copy_from racing read to smithi07134777-48 2024-09-11T01:42:05.064 INFO:tasks.rados.rados.0.smithi071.stdout:1880: finishing copy_from to smithi07134777-48 2024-09-11T01:42:05.064 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 922 (ObjNum 565 snap 160 seq_num 565) dirty exists 2024-09-11T01:42:05.963 INFO:tasks.rados.rados.0.smithi071.stdout:1880: done (1 left) 2024-09-11T01:42:05.963 INFO:tasks.rados.rados.0.smithi071.stdout:1881: done (0 left) 2024-09-11T01:42:05.963 INFO:tasks.rados.rados.0.smithi071.stdout:1882: write (excl) oid 43 current snap is 207 2024-09-11T01:42:05.963 INFO:tasks.rados.rados.0.smithi071.stdout:1882: seq_num 695 ranges {480865=486178,1453103=442564,2652627=519696} 2024-09-11T01:42:05.968 INFO:tasks.rados.rados.0.smithi071.stdout:1882: writing smithi07134777-43 from 480865 to 967043 tid 1 2024-09-11T01:42:05.973 INFO:tasks.rados.rados.0.smithi071.stdout:1882: writing smithi07134777-43 from 1453103 to 1895667 tid 2 2024-09-11T01:42:05.980 INFO:tasks.rados.rados.0.smithi071.stdout:1882: writing smithi07134777-43 from 2652627 to 3172323 tid 3 2024-09-11T01:42:05.980 INFO:tasks.rados.rados.0.smithi071.stdout:1883: snap_create 2024-09-11T01:42:05.980 INFO:tasks.rados.rados.0.smithi071.stdout:1882: finishing write tid 1 to smithi07134777-43 2024-09-11T01:42:05.980 INFO:tasks.rados.rados.0.smithi071.stdout:1882: finishing write tid 2 to smithi07134777-43 2024-09-11T01:42:05.984 INFO:tasks.rados.rados.0.smithi071.stdout:1882: finishing write tid 3 to smithi07134777-43 2024-09-11T01:42:05.984 INFO:tasks.rados.rados.0.smithi071.stdout:1882: finishing write tid 4 to smithi07134777-43 2024-09-11T01:42:05.984 INFO:tasks.rados.rados.0.smithi071.stdout:1882: finishing write tid 5 to smithi07134777-43 2024-09-11T01:42:05.984 INFO:tasks.rados.rados.0.smithi071.stdout:1882: oid 43 updating version 0 to 1007 2024-09-11T01:42:05.984 INFO:tasks.rados.rados.0.smithi071.stdout:1882: oid 43 updating version 1007 to 1009 2024-09-11T01:42:05.984 INFO:tasks.rados.rados.0.smithi071.stdout:1882: oid 43 updating version 1009 to 1010 2024-09-11T01:42:05.984 INFO:tasks.rados.rados.0.smithi071.stdout:1882: oid 43 version 1010 is already newer than 1008 2024-09-11T01:42:05.984 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 1010 (ObjNum 695 snap 207 seq_num 695) dirty exists 2024-09-11T01:42:05.984 INFO:tasks.rados.rados.0.smithi071.stdout:1882: left oid 43 (ObjNum 695 snap 207 seq_num 695) 2024-09-11T01:42:06.978 INFO:tasks.rados.rados.0.smithi071.stdout:1882: done (1 left) 2024-09-11T01:42:06.978 INFO:tasks.rados.rados.0.smithi071.stdout:1883: done (0 left) 2024-09-11T01:42:06.978 INFO:tasks.rados.rados.0.smithi071.stdout:1884: read oid 25 snap -1 2024-09-11T01:42:06.978 INFO:tasks.rados.rados.0.smithi071.stdout:1884: expect (ObjNum 651 snap 199 seq_num 651) 2024-09-11T01:42:06.978 INFO:tasks.rados.rados.0.smithi071.stdout:1885: write oid 12 current snap is 208 2024-09-11T01:42:06.978 INFO:tasks.rados.rados.0.smithi071.stdout:1885: seq_num 696 ranges {497672=561427,1795232=702575,2963697=730968} 2024-09-11T01:42:06.983 INFO:tasks.rados.rados.0.smithi071.stdout:1885: writing smithi07134777-12 from 497672 to 1059099 tid 1 2024-09-11T01:42:06.991 INFO:tasks.rados.rados.0.smithi071.stdout:1885: writing smithi07134777-12 from 1795232 to 2497807 tid 2 2024-09-11T01:42:06.998 INFO:tasks.rados.rados.0.smithi071.stdout:1885: writing smithi07134777-12 from 2963697 to 3694665 tid 3 2024-09-11T01:42:06.998 INFO:tasks.rados.rados.0.smithi071.stdout:1886: snap_remove snap 205 2024-09-11T01:42:07.959 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-11T01:42:07.959 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-11T01:42:07.960 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-11T01:42:07.962 INFO:tasks.ceph.mon.c:Started 2024-09-11T01:42:07.963 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:42:07.963 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:42:07.987 INFO:tasks.rados.rados.0.smithi071.stdout:1886: done (2 left) 2024-09-11T01:42:07.987 INFO:tasks.rados.rados.0.smithi071.stdout:1887: snap_create 2024-09-11T01:42:07.987 INFO:tasks.rados.rados.0.smithi071.stdout:1884: expect (ObjNum 651 snap 199 seq_num 651) 2024-09-11T01:42:07.987 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 187995 2024-09-11T01:42:08.041 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:08.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.073+0000 7f4bf4dec640 10 monclient: get_monmap_and_config 2024-09-11T01:42:08.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.073+0000 7f4bf4dec640 10 monclient: build_initial_monmap 2024-09-11T01:42:08.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.073+0000 7f4bf4dec640 10 monclient: monmap: 2024-09-11T01:42:08.076 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:08.076 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:08.076 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:42:08.075280+0000 2024-09-11T01:42:08.076 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:42:08.075280+0000 2024-09-11T01:42:08.076 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:08.076 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:08.076 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:42:08.076 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:42:08.077 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:42:08.077 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:42:08.077 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:42:08.077 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:08.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.074+0000 7f4bf4dec640 1 Processor -- start 2024-09-11T01:42:08.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.074+0000 7f4bf4dec640 1 -- start start 2024-09-11T01:42:08.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.074+0000 7f4bf4dec640 10 monclient: init 2024-09-11T01:42:08.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.074+0000 7f4bf4dec640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:08.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.074+0000 7f4bf4dec640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-11T01:42:08.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.074+0000 7f4bf4dec640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4bf00a1380 0x7f4bf00a1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:08.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.074+0000 7f4bf4dec640 10 monclient(hunting): picked mon.noname-a con 0x7f4bf00a1380 addr v2:172.21.15.12:3300/0 2024-09-11T01:42:08.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.074+0000 7f4bf4dec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4bf00a0c90 addr v1:172.21.15.12:6789/0 2024-09-11T01:42:08.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.074+0000 7f4bf4dec640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4bf01361f0 0x7f4bf013de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:08.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.074+0000 7f4bf4dec640 10 monclient(hunting): picked mon.noname-c con 0x7f4bf01361f0 addr v2:172.21.15.71:3300/0 2024-09-11T01:42:08.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.074+0000 7f4bf4dec640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bf00a2b40 con 0x7f4bf00a0c90 2024-09-11T01:42:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.074+0000 7f4bf4dec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.074+0000 7f4bf4dec640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f4bf009f9a0 con 0x7f4bf00a1380 2024-09-11T01:42:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.074+0000 7f4bf4dec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.074+0000 7f4bf4dec640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f4bf00a2230 con 0x7f4bf01361f0 2024-09-11T01:42:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.074+0000 7f4bf4dec640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.074+0000 7f4bf4dec640 10 monclient(hunting): authenticate will time out at 1534.517898s 2024-09-11T01:42:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.074+0000 7f4bee59c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4bf00a1380 0x7f4bf00a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4bee59c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4bf00a1380 0x7f4bf00a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46448/0 (socket says 172.21.15.12:46448) 2024-09-11T01:42:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4bee59c640 1 -- 172.21.15.12:0/3855369101 learned_addr learned my addr 172.21.15.12:0/3855369101 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4bee59c640 10 monclient(hunting): get_auth_request con 0x7f4bf00a1380 auth_method 0 2024-09-11T01:42:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4bee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4bee59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4bedd9b640 1 --1- 172.21.15.12:0/3855369101 >> v1:172.21.15.12:6789/0 conn(0x7f4bf00a0c90 0x7f4bf0135ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60794/0 (socket says 172.21.15.12:60794) 2024-09-11T01:42:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4bee59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4beed9d640 1 --2- 172.21.15.12:0/3855369101 >> v2:172.21.15.71:3300/0 conn(0x7f4bf01361f0 0x7f4bf013de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4beed9d640 10 monclient(hunting): get_auth_request con 0x7f4bf01361f0 auth_method 0 2024-09-11T01:42:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4beed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4beed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4beed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4bee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4bee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4bee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4beed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4beed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4beed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4bee59c640 10 monclient(hunting): handle_auth_done global_id 94539 payload 274 2024-09-11T01:42:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4bee59c640 1 -- 172.21.15.12:0/3855369101 >> v2:172.21.15.71:3300/0 conn(0x7f4bf01361f0 msgr2=0x7f4bf013de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:08.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4bee59c640 1 --2- 172.21.15.12:0/3855369101 >> v2:172.21.15.71:3300/0 conn(0x7f4bf01361f0 0x7f4bf013de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:08.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4bee59c640 1 -- 172.21.15.12:0/3855369101 >> v1:172.21.15.12:6789/0 conn(0x7f4bf00a0c90 legacy=0x7f4bf0135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:08.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4bee59c640 10 monclient: _finish_hunting 0 2024-09-11T01:42:08.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4bee59c640 1 monclient: found mon.noname-a 2024-09-11T01:42:08.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4bee59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:08.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4bee59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:42:08.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4bee59c640 1 -- 172.21.15.12:0/3855369101 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf00a2510 con 0x7f4bf00a1380 2024-09-11T01:42:08.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.075+0000 7f4beed9d640 1 --2- 172.21.15.12:0/3855369101 >> v2:172.21.15.71:3300/0 conn(0x7f4bf01361f0 0x7f4bf013de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:42:08.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.078+0000 7f4bee59c640 1 -- 172.21.15.12:0/3855369101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bf00a1380 msgr2=0x7f4bf00a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:08.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.078+0000 7f4bee59c640 1 --2- 172.21.15.12:0/3855369101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bf00a1380 0x7f4bf00a1760 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4bf00a10e0 tx=0x7f4be002a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86a31f4464296d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:08.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.078+0000 7f4bed59a640 1 -- 172.21.15.12:0/3855369101 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be0037070 con 0x7f4bf00a1380 2024-09-11T01:42:08.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.078+0000 7f4bed59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:08.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.078+0000 7f4bed59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:42:08.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.078+0000 7f4bed59a640 10 monclient: dump: 2024-09-11T01:42:08.086 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:08.086 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:08.086 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:08.086 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:08.086 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:08.086 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:08.086 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:08.086 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:08.086 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:08.086 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:08.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.078+0000 7f4bed59a640 10 monclient: _finish_auth 0 2024-09-11T01:42:08.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.078+0000 7f4bed59a640 10 monclient: _check_auth_tickets 2024-09-11T01:42:08.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.078+0000 7f4bed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:08.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.078+0000 7f4bed59a640 1 -- 172.21.15.12:0/3855369101 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4be0032640 con 0x7f4bf00a1380 2024-09-11T01:42:08.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.078+0000 7f4bf4dec640 5 monclient: authenticate success, global_id 94539 2024-09-11T01:42:08.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.078+0000 7f4bf4dec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:42:08.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.078+0000 7f4bed59a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:08.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.078+0000 7f4bed59a640 1 -- 172.21.15.12:0/3855369101 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be0032a10 con 0x7f4bf00a1380 2024-09-11T01:42:08.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.078+0000 7f4bf4dec640 10 monclient: get_monmap_and_config success 2024-09-11T01:42:08.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.078+0000 7f4bf4dec640 4 set_mon_vals no callback set 2024-09-11T01:42:08.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.078+0000 7f4bed59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:08.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.078+0000 7f4bed59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:08.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.078+0000 7f4bed59a640 10 monclient: dump: 2024-09-11T01:42:08.087 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:08.087 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:08.087 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:08.088 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:08.088 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:08.088 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:08.088 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:08.088 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:08.088 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:08.088 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:08.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.078+0000 7f4bf4dec640 10 monclient: shutdown 2024-09-11T01:42:08.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.078+0000 7f4bf4dec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:08.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.078+0000 7f4bf4dec640 1 -- 172.21.15.12:0/3855369101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bf00a1380 msgr2=0x7f4bf00a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:08.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.078+0000 7f4bf4dec640 1 --2- 172.21.15.12:0/3855369101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bf00a1380 0x7f4bf00a1760 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4bf00a10e0 tx=0x7f4be002a0b0 comp rx=0 tx=0).stop 2024-09-11T01:42:08.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.079+0000 7f4bf4dec640 1 -- 172.21.15.12:0/3855369101 shutdown_connections 2024-09-11T01:42:08.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.079+0000 7f4bf4dec640 1 --2- 172.21.15.12:0/3855369101 >> v2:172.21.15.71:3300/0 conn(0x7f4bf01361f0 0x7f4bf013de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:08.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.079+0000 7f4bf4dec640 1 --2- 172.21.15.12:0/3855369101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bf00a1380 0x7f4bf00a1760 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:08.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.079+0000 7f4bf4dec640 1 -- 172.21.15.12:0/3855369101 >> 172.21.15.12:0/3855369101 conn(0x7f4bf00a2d90 msgr2=0x7f4bf00a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:08.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.079+0000 7f4bf4dec640 1 -- 172.21.15.12:0/3855369101 shutdown_connections 2024-09-11T01:42:08.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.079+0000 7f4bf4dec640 1 -- 172.21.15.12:0/3855369101 wait complete. 2024-09-11T01:42:08.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.079+0000 7f4bf4dec640 10 monclient: build_initial_monmap 2024-09-11T01:42:08.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.079+0000 7f4bf4dec640 10 monclient: monmap: 2024-09-11T01:42:08.089 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:08.089 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:42:08.089 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:42:08.089 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:42:08.089 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:08.089 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:08.089 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:42:08.089 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:42:08.089 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:42:08.090 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:08.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4bf4dec640 1 Processor -- start 2024-09-11T01:42:08.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4bf4dec640 1 -- start start 2024-09-11T01:42:08.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4bf4dec640 10 monclient: init 2024-09-11T01:42:08.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4bf4dec640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:08.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4bf4dec640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:42:08.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4bf4dec640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4bf00a0c90 0x7f4bf01ccc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:08.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4bf4dec640 10 monclient(hunting): picked mon.noname-b con 0x7f4bf00a0c90 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:42:08.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4bf4dec640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bf00a1380 0x7f4bf0134df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:08.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4bf4dec640 10 monclient(hunting): picked mon.noname-c con 0x7f4bf00a1380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:08.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4bf4dec640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bf01361f0 0x7f4bf009d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:08.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4bf4dec640 10 monclient(hunting): picked mon.noname-a con 0x7f4bf01361f0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:42:08.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4bf4dec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:08.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4bf4dec640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf009ce20 con 0x7f4bf01361f0 2024-09-11T01:42:08.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4bf4dec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:08.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4bf4dec640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf009cca0 con 0x7f4bf00a0c90 2024-09-11T01:42:08.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4bf4dec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:08.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4bf4dec640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4bf00a0390 con 0x7f4bf00a1380 2024-09-11T01:42:08.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4bf4dec640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:08.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4bf4dec640 10 monclient(hunting): authenticate will time out at 1534.523832s 2024-09-11T01:42:08.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4beed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bf01361f0 0x7f4bf009d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:08.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4bedd9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bf00a1380 0x7f4bf0134df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:08.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4beed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bf01361f0 0x7f4bf009d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46452/0 (socket says 172.21.15.12:46452) 2024-09-11T01:42:08.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4beed9d640 1 -- 172.21.15.12:0/3548062706 learned_addr learned my addr 172.21.15.12:0/3548062706 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:08.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4beed9d640 10 monclient(hunting): get_auth_request con 0x7f4bf01361f0 auth_method 0 2024-09-11T01:42:08.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4beed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:08.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4beed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:08.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4beed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:08.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4bee59c640 1 --2- 172.21.15.12:0/3548062706 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4bf00a0c90 0x7f4bf01ccc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:08.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.080+0000 7f4bedd9b640 10 monclient(hunting): get_auth_request con 0x7f4bf00a1380 auth_method 0 2024-09-11T01:42:08.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4bedd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:08.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4bedd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:08.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4bedd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:08.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4bee59c640 10 monclient(hunting): get_auth_request con 0x7f4bf00a0c90 auth_method 0 2024-09-11T01:42:08.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4bee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:08.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4bee59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:08.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4bee59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:08.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4beed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:08.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4beed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:08.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4beed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:08.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4bee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:08.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4bee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:08.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4bee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:08.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4bedd9b640 1 -- 172.21.15.12:0/3548062706 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bf00a1380 msgr2=0x7f4bf0134df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:42:08.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4bedd9b640 1 -- 172.21.15.12:0/3548062706 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bf00a1380 msgr2=0x7f4bf0134df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:42:08.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4bedd9b640 1 --2- 172.21.15.12:0/3548062706 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bf00a1380 0x7f4bf0134df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:42:08.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4bedd9b640 1 --2- 172.21.15.12:0/3548062706 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bf00a1380 0x7f4bf0134df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:08.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4bee59c640 10 monclient(hunting): handle_auth_done global_id 134176 payload 824 2024-09-11T01:42:08.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4bee59c640 1 -- 172.21.15.12:0/3548062706 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bf00a1380 msgr2=0x7f4bf0134df0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:42:08.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4bee59c640 1 --2- 172.21.15.12:0/3548062706 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bf00a1380 0x7f4bf0134df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:08.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4bee59c640 1 -- 172.21.15.12:0/3548062706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bf01361f0 msgr2=0x7f4bf009d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:08.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4bee59c640 1 --2- 172.21.15.12:0/3548062706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bf01361f0 0x7f4bf009d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:08.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4bee59c640 10 monclient: _finish_hunting 0 2024-09-11T01:42:08.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4bee59c640 1 monclient: found mon.noname-b 2024-09-11T01:42:08.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4bee59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:08.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4bee59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-09-11T01:42:08.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.081+0000 7f4bee59c640 1 -- 172.21.15.12:0/3548062706 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf0122ff0 con 0x7f4bf00a0c90 2024-09-11T01:42:08.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.083+0000 7f4bee59c640 1 -- 172.21.15.12:0/3548062706 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4bf00a0c90 msgr2=0x7f4bf01ccc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:08.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.083+0000 7f4bee59c640 1 --2- 172.21.15.12:0/3548062706 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4bf00a0c90 0x7f4bf01ccc60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4be0031000 tx=0x7f4be00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:08.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.083+0000 7f4bdeffd640 1 -- 172.21.15.12:0/3548062706 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be0041020 con 0x7f4bf00a0c90 2024-09-11T01:42:08.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.083+0000 7f4bdeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:08.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.083+0000 7f4bdeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:42:08.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.083+0000 7f4bdeffd640 10 monclient: dump: 2024-09-11T01:42:08.099 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:08.099 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:08.099 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:08.099 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:08.099 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:08.099 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:08.099 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:08.099 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:08.099 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:08.099 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:08.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.083+0000 7f4bdeffd640 10 monclient: _finish_auth 0 2024-09-11T01:42:08.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.083+0000 7f4bdeffd640 10 monclient: _check_auth_tickets 2024-09-11T01:42:08.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.083+0000 7f4bdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:08.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.083+0000 7f4bdeffd640 1 -- 172.21.15.12:0/3548062706 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4be003d830 con 0x7f4bf00a0c90 2024-09-11T01:42:08.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.083+0000 7f4bdeffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:08.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.083+0000 7f4bdeffd640 1 -- 172.21.15.12:0/3548062706 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be0037040 con 0x7f4bf00a0c90 2024-09-11T01:42:08.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.083+0000 7f4bf4dec640 5 monclient: authenticate success, global_id 134176 2024-09-11T01:42:08.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.083+0000 7f4bdeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:08.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.083+0000 7f4bdeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:42:08.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.083+0000 7f4bdeffd640 10 monclient: dump: 2024-09-11T01:42:08.100 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:08.100 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:08.100 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:08.101 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:08.101 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:08.101 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:08.101 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:08.101 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:08.101 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:08.101 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:08.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.083+0000 7f4bf4dec640 10 monclient: _renew_subs 2024-09-11T01:42:08.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.083+0000 7f4bf4dec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:42:08.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.083+0000 7f4becd99640 4 set_mon_vals no callback set 2024-09-11T01:42:08.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.083+0000 7f4bf4dec640 1 -- 172.21.15.12:0/3548062706 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bf009f130 con 0x7f4bf00a0c90 2024-09-11T01:42:08.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.084+0000 7f4bf4dec640 10 monclient: _renew_subs 2024-09-11T01:42:08.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.084+0000 7f4bf4dec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:42:08.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.084+0000 7f4bf4dec640 1 -- 172.21.15.12:0/3548062706 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bf0124510 con 0x7f4bf00a0c90 2024-09-11T01:42:08.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.084+0000 7f4bdeffd640 1 -- 172.21.15.12:0/3548062706 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f4be003a030 con 0x7f4bf00a0c90 2024-09-11T01:42:08.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.084+0000 7f4bf4dec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:08.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.084+0000 7f4bf4dec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:08.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.084+0000 7f4bf4dec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:42:08.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.084+0000 7f4bf4dec640 1 -- 172.21.15.12:0/3548062706 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bf01366a0 con 0x7f4bf00a0c90 2024-09-11T01:42:08.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.085+0000 7f4bdeffd640 1 --2- 172.21.15.12:0/3548062706 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4bd003e210 0x7f4bd00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:08.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.085+0000 7f4bdeffd640 1 -- 172.21.15.12:0/3548062706 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(430..430 src has 285..430) ==== 4815+0+0 (secure 0 0 0) 0x7f4be0072ef0 con 0x7f4bf00a0c90 2024-09-11T01:42:08.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.085+0000 7f4bedd9b640 1 --2- 172.21.15.12:0/3548062706 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4bd003e210 0x7f4bd00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:08.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.085+0000 7f4bedd9b640 10 monclient: get_auth_request con 0x7f4bd003e210 auth_method 0 2024-09-11T01:42:08.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.087+0000 7f4bedd9b640 1 --2- 172.21.15.12:0/3548062706 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4bd003e210 0x7f4bd00406d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4bf009f6b0 tx=0x7f4be4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:08.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.090+0000 7f4bdeffd640 1 -- 172.21.15.12:0/3548062706 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4be004b070 con 0x7f4bf00a0c90 2024-09-11T01:42:08.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.090+0000 7f4bdeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:08.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.090+0000 7f4bdeffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:42:08.142 INFO:tasks.rados.rados.0.smithi071.stdout:1885: finishing write tid 1 to smithi07134777-12 2024-09-11T01:42:08.142 INFO:tasks.rados.rados.0.smithi071.stdout:1885: finishing write tid 2 to smithi07134777-12 2024-09-11T01:42:08.142 INFO:tasks.rados.rados.0.smithi071.stdout:1885: finishing write tid 3 to smithi07134777-12 2024-09-11T01:42:08.142 INFO:tasks.rados.rados.0.smithi071.stdout:1885: finishing write tid 4 to smithi07134777-12 2024-09-11T01:42:08.143 INFO:tasks.rados.rados.0.smithi071.stdout:1885: finishing write tid 5 to smithi07134777-12 2024-09-11T01:42:08.143 INFO:tasks.rados.rados.0.smithi071.stdout:1885: oid 12 updating version 0 to 1012 2024-09-11T01:42:08.143 INFO:tasks.rados.rados.0.smithi071.stdout:1885: oid 12 updating version 1012 to 1014 2024-09-11T01:42:08.143 INFO:tasks.rados.rados.0.smithi071.stdout:1885: oid 12 version 1014 is already newer than 1013 2024-09-11T01:42:08.143 INFO:tasks.rados.rados.0.smithi071.stdout:1885: oid 12 version 1014 is already newer than 1011 2024-09-11T01:42:08.143 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 1014 (ObjNum 696 snap 208 seq_num 696) dirty exists 2024-09-11T01:42:08.143 INFO:tasks.rados.rados.0.smithi071.stdout:1885: left oid 12 (ObjNum 696 snap 208 seq_num 696) 2024-09-11T01:42:08.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.349+0000 7f4bf4dec640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:42:08.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.349+0000 7f4bf4dec640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:42:08.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.349+0000 7f4bf4dec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:42:08.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.349+0000 7f4bf4dec640 1 -- 172.21.15.12:0/3548062706 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4bf0123fb0 con 0x7f4bf00a0c90 2024-09-11T01:42:08.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.350+0000 7f4bdeffd640 1 -- 172.21.15.12:0/3548062706 <== mon.1 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f4be003a350 con 0x7f4bf00a0c90 2024-09-11T01:42:08.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.350+0000 7f4bdeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:42:08.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.350+0000 7f4bdeffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:42:08.353 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":118,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:42:08.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.352+0000 7f4bf4dec640 1 -- 172.21.15.12:0/3548062706 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4bd003e210 msgr2=0x7f4bd00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:08.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.352+0000 7f4bf4dec640 1 --2- 172.21.15.12:0/3548062706 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4bd003e210 0x7f4bd00406d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4bf009f6b0 tx=0x7f4be4009040 comp rx=0 tx=0).stop 2024-09-11T01:42:08.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.352+0000 7f4bf4dec640 10 monclient: shutdown 2024-09-11T01:42:08.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.352+0000 7f4bf4dec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:08.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.352+0000 7f4bf4dec640 1 -- 172.21.15.12:0/3548062706 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4bf00a0c90 msgr2=0x7f4bf01ccc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:08.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.352+0000 7f4bf4dec640 1 --2- 172.21.15.12:0/3548062706 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4bf00a0c90 0x7f4bf01ccc60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4be0031000 tx=0x7f4be00046c0 comp rx=0 tx=0).stop 2024-09-11T01:42:08.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.352+0000 7f4bf4dec640 1 -- 172.21.15.12:0/3548062706 shutdown_connections 2024-09-11T01:42:08.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.352+0000 7f4bf4dec640 1 --2- 172.21.15.12:0/3548062706 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4bd003e210 0x7f4bd00406d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:08.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.352+0000 7f4bf4dec640 1 --2- 172.21.15.12:0/3548062706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bf01361f0 0x7f4bf009d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:08.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.352+0000 7f4bf4dec640 1 --2- 172.21.15.12:0/3548062706 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bf00a1380 0x7f4bf0134df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:08.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.352+0000 7f4bf4dec640 1 --2- 172.21.15.12:0/3548062706 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4bf00a0c90 0x7f4bf01ccc60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:08.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.352+0000 7f4bf4dec640 1 -- 172.21.15.12:0/3548062706 >> 172.21.15.12:0/3548062706 conn(0x7f4bf00a2d90 msgr2=0x7f4bf0070380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:08.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.352+0000 7f4bf4dec640 1 -- 172.21.15.12:0/3548062706 shutdown_connections 2024-09-11T01:42:08.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:08.352+0000 7f4bf4dec640 1 -- 172.21.15.12:0/3548062706 wait complete. 2024-09-11T01:42:08.961 INFO:tasks.rados.rados.0.smithi071.stdout:1884: done (2 left) 2024-09-11T01:42:08.961 INFO:tasks.rados.rados.0.smithi071.stdout:1885: done (1 left) 2024-09-11T01:42:08.961 INFO:tasks.rados.rados.0.smithi071.stdout:1887: done (0 left) 2024-09-11T01:42:08.961 INFO:tasks.rados.rados.0.smithi071.stdout:1888: read oid 3 snap -1 2024-09-11T01:42:08.961 INFO:tasks.rados.rados.0.smithi071.stdout:1888: expect (ObjNum 685 snap 204 seq_num 685) 2024-09-11T01:42:08.961 INFO:tasks.rados.rados.0.smithi071.stdout:1889: read oid 47 snap 195 2024-09-11T01:42:08.961 INFO:tasks.rados.rados.0.smithi071.stdout:1889: expect (ObjNum 599 snap 171 seq_num 599) 2024-09-11T01:42:08.961 INFO:tasks.rados.rados.0.smithi071.stdout:1890: snap_create 2024-09-11T01:42:08.968 INFO:tasks.rados.rados.0.smithi071.stdout:1889: expect (ObjNum 599 snap 171 seq_num 599) 2024-09-11T01:42:09.039 INFO:tasks.rados.rados.0.smithi071.stdout:1888: expect (ObjNum 685 snap 204 seq_num 685) 2024-09-11T01:42:09.103 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1483915 2024-09-11T01:42:09.140 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:09.965 INFO:tasks.rados.rados.0.smithi071.stdout:1888: done (2 left) 2024-09-11T01:42:09.965 INFO:tasks.rados.rados.0.smithi071.stdout:1889: done (1 left) 2024-09-11T01:42:09.965 INFO:tasks.rados.rados.0.smithi071.stdout:1890: done (0 left) 2024-09-11T01:42:09.966 INFO:tasks.rados.rados.0.smithi071.stdout:1891: write (excl) oid 43 current snap is 210 2024-09-11T01:42:09.966 INFO:tasks.rados.rados.0.smithi071.stdout:1891: seq_num 697 ranges {514479=636676,1853714=446233,2991120=542240} 2024-09-11T01:42:09.971 INFO:tasks.rados.rados.0.smithi071.stdout:1891: writing smithi07134777-43 from 514479 to 1151155 tid 1 2024-09-11T01:42:09.977 INFO:tasks.rados.rados.0.smithi071.stdout:1891: writing smithi07134777-43 from 1853714 to 2299947 tid 2 2024-09-11T01:42:09.984 INFO:tasks.rados.rados.0.smithi071.stdout:1891: writing smithi07134777-43 from 2991120 to 3533360 tid 3 2024-09-11T01:42:09.984 INFO:tasks.rados.rados.0.smithi071.stdout:1892: delete oid 37 current snap is 210 2024-09-11T01:42:09.984 INFO:tasks.rados.rados.0.smithi071.stdout:1891: finishing write tid 1 to smithi07134777-43 2024-09-11T01:42:09.984 INFO:tasks.rados.rados.0.smithi071.stdout:1891: finishing write tid 2 to smithi07134777-43 2024-09-11T01:42:09.993 INFO:tasks.rados.rados.0.smithi071.stdout:1891: finishing write tid 3 to smithi07134777-43 2024-09-11T01:42:09.993 INFO:tasks.rados.rados.0.smithi071.stdout:1891: finishing write tid 4 to smithi07134777-43 2024-09-11T01:42:09.993 INFO:tasks.rados.rados.0.smithi071.stdout:1891: finishing write tid 5 to smithi07134777-43 2024-09-11T01:42:09.993 INFO:tasks.rados.rados.0.smithi071.stdout:1891: oid 43 updating version 0 to 1017 2024-09-11T01:42:09.993 INFO:tasks.rados.rados.0.smithi071.stdout:1891: oid 43 version 1017 is already newer than 1016 2024-09-11T01:42:09.994 INFO:tasks.rados.rados.0.smithi071.stdout:1891: oid 43 updating version 1017 to 1018 2024-09-11T01:42:09.994 INFO:tasks.rados.rados.0.smithi071.stdout:1891: oid 43 version 1018 is already newer than 1015 2024-09-11T01:42:09.994 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 1018 (ObjNum 697 snap 210 seq_num 697) dirty exists 2024-09-11T01:42:09.994 INFO:tasks.rados.rados.0.smithi071.stdout:1891: left oid 43 (ObjNum 697 snap 210 seq_num 697) 2024-09-11T01:42:09.994 INFO:tasks.rados.rados.0.smithi071.stdout:1891: done (1 left) 2024-09-11T01:42:09.994 INFO:tasks.rados.rados.0.smithi071.stdout:1892: done (0 left) 2024-09-11T01:42:09.995 INFO:tasks.rados.rados.0.smithi071.stdout:1893: snap_create 2024-09-11T01:42:10.988 INFO:tasks.rados.rados.0.smithi071.stdout:1893: done (0 left) 2024-09-11T01:42:10.988 INFO:tasks.rados.rados.0.smithi071.stdout:1894: delete oid 5 current snap is 211 2024-09-11T01:42:10.994 INFO:tasks.rados.rados.0.smithi071.stdout:1894: done (0 left) 2024-09-11T01:42:10.994 INFO:tasks.rados.rados.0.smithi071.stdout:1895: read oid 30 snap -1 2024-09-11T01:42:10.994 INFO:tasks.rados.rados.0.smithi071.stdout:1895: expect deleted 2024-09-11T01:42:10.994 INFO:tasks.rados.rados.0.smithi071.stdout:1896: write (excl) oid 13 current snap is 211 2024-09-11T01:42:10.995 INFO:tasks.rados.rados.0.smithi071.stdout:1896: seq_num 700 ranges {564900=578776,1629160=709913,3073389=492409} 2024-09-11T01:42:11.001 INFO:tasks.rados.rados.0.smithi071.stdout:1896: writing smithi07134777-13 from 564900 to 1143676 tid 1 2024-09-11T01:42:11.007 INFO:tasks.rados.rados.0.smithi071.stdout:1896: writing smithi07134777-13 from 1629160 to 2339073 tid 2 2024-09-11T01:42:11.015 INFO:tasks.rados.rados.0.smithi071.stdout:1896: writing smithi07134777-13 from 3073389 to 3565798 tid 3 2024-09-11T01:42:11.015 INFO:tasks.rados.rados.0.smithi071.stdout:1897: rollback oid 4 current snap is 211 2024-09-11T01:42:11.015 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 4 to 196 2024-09-11T01:42:11.015 INFO:tasks.rados.rados.0.smithi071.stdout:1896: finishing write tid 1 to smithi07134777-13 2024-09-11T01:42:11.015 INFO:tasks.rados.rados.0.smithi071.stdout:1896: finishing write tid 2 to smithi07134777-13 2024-09-11T01:42:11.015 INFO:tasks.rados.rados.0.smithi071.stdout:1895: done (2 left) 2024-09-11T01:42:11.015 INFO:tasks.rados.rados.0.smithi071.stdout:1898: write oid 39 current snap is 211 2024-09-11T01:42:11.015 INFO:tasks.rados.rados.0.smithi071.stdout:1898: seq_num 701 ranges {581707=654025,1687642=453571,2700812=703681} 2024-09-11T01:42:11.021 INFO:tasks.rados.rados.0.smithi071.stdout:1898: writing smithi07134777-39 from 581707 to 1235732 tid 1 2024-09-11T01:42:11.026 INFO:tasks.rados.rados.0.smithi071.stdout:1898: writing smithi07134777-39 from 1687642 to 2141213 tid 2 2024-09-11T01:42:11.034 INFO:tasks.rados.rados.0.smithi071.stdout:1898: writing smithi07134777-39 from 2700812 to 3404493 tid 3 2024-09-11T01:42:11.034 INFO:tasks.rados.rados.0.smithi071.stdout:1899: read oid 30 snap -1 2024-09-11T01:42:11.034 INFO:tasks.rados.rados.0.smithi071.stdout:1899: expect deleted 2024-09-11T01:42:11.034 INFO:tasks.rados.rados.0.smithi071.stdout:1900: read oid 1 snap -1 2024-09-11T01:42:11.034 INFO:tasks.rados.rados.0.smithi071.stdout:1900: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:42:11.034 INFO:tasks.rados.rados.0.smithi071.stdout:1901: snap_remove snap 195 2024-09-11T01:42:11.366 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:42:11.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.477+0000 7efe18aef640 10 monclient: get_monmap_and_config 2024-09-11T01:42:11.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.477+0000 7efe18aef640 10 monclient: build_initial_monmap 2024-09-11T01:42:11.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.477+0000 7efe18aef640 10 monclient: monmap: 2024-09-11T01:42:11.480 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:11.480 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:11.480 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:42:11.478815+0000 2024-09-11T01:42:11.480 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:42:11.478815+0000 2024-09-11T01:42:11.480 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:11.480 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:11.480 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:42:11.480 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:42:11.480 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:42:11.480 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:42:11.480 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:42:11.480 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:11.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.477+0000 7efe18aef640 1 Processor -- start 2024-09-11T01:42:11.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.477+0000 7efe18aef640 1 -- start start 2024-09-11T01:42:11.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.477+0000 7efe18aef640 10 monclient: init 2024-09-11T01:42:11.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe18aef640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:11.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe18aef640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-11T01:42:11.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe18aef640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7efe14138210 0x7efe141385f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:11.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe18aef640 10 monclient(hunting): picked mon.noname-c con 0x7efe14138210 addr v2:172.21.15.71:3300/0 2024-09-11T01:42:11.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe18aef640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7efe140a6810 0x7efe140a5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:11.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe18aef640 10 monclient(hunting): picked mon.noname-a con 0x7efe140a6810 addr v2:172.21.15.12:3300/0 2024-09-11T01:42:11.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe18aef640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efe140a55d0 0x7efe140a59e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:11.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe18aef640 10 monclient(hunting): picked mon.noname-b con 0x7efe140a55d0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:11.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe18aef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:11.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe18aef640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7efe140a5f20 con 0x7efe140a6810 2024-09-11T01:42:11.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe18aef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:11.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe18aef640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7efe1409c7a0 con 0x7efe14138210 2024-09-11T01:42:11.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe18aef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:11.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe18aef640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7efe1409c4c0 con 0x7efe140a55d0 2024-09-11T01:42:11.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe18aef640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:11.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe18aef640 10 monclient(hunting): authenticate will time out at 1537.921419s 2024-09-11T01:42:11.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe0bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7efe140a6810 0x7efe140a5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:11.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe12d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efe140a55d0 0x7efe140a59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:11.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe0bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7efe140a6810 0x7efe140a5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46502/0 (socket says 172.21.15.12:46502) 2024-09-11T01:42:11.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe0bfff640 1 -- 172.21.15.12:0/34916879 learned_addr learned my addr 172.21.15.12:0/34916879 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:11.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe12d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efe140a55d0 0x7efe140a59e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:43054/0 (socket says 172.21.15.12:43054) 2024-09-11T01:42:11.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe0bfff640 10 monclient(hunting): get_auth_request con 0x7efe140a6810 auth_method 0 2024-09-11T01:42:11.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:11.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:11.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:11.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe12d9d640 10 monclient(hunting): get_auth_request con 0x7efe140a55d0 auth_method 0 2024-09-11T01:42:11.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe12d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:11.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe12d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:11.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe12d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:11.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe1259c640 1 --2- 172.21.15.12:0/34916879 >> v2:172.21.15.71:3300/0 conn(0x7efe14138210 0x7efe141385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:11.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:11.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:11.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:11.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe1259c640 10 monclient(hunting): get_auth_request con 0x7efe14138210 auth_method 0 2024-09-11T01:42:11.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe1259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:11.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe1259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:11.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe1259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:11.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe12d9d640 1 -- 172.21.15.12:0/34916879 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efe140a55d0 msgr2=0x7efe140a59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:42:11.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe12d9d640 1 -- 172.21.15.12:0/34916879 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efe140a55d0 msgr2=0x7efe140a59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:42:11.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe12d9d640 1 --2- 172.21.15.12:0/34916879 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efe140a55d0 0x7efe140a59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:42:11.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe12d9d640 1 --2- 172.21.15.12:0/34916879 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efe140a55d0 0x7efe140a59e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:11.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe1259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:11.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe1259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:11.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe1259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:11.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe0bfff640 10 monclient(hunting): handle_auth_done global_id 94545 payload 274 2024-09-11T01:42:11.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe0bfff640 1 -- 172.21.15.12:0/34916879 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efe140a55d0 msgr2=0x7efe140a59e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:42:11.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe0bfff640 1 --2- 172.21.15.12:0/34916879 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efe140a55d0 0x7efe140a59e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:11.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe0bfff640 1 -- 172.21.15.12:0/34916879 >> v2:172.21.15.71:3300/0 conn(0x7efe14138210 msgr2=0x7efe141385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:11.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe0bfff640 1 --2- 172.21.15.12:0/34916879 >> v2:172.21.15.71:3300/0 conn(0x7efe14138210 0x7efe141385f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:11.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe0bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:42:11.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe0bfff640 1 monclient: found mon.noname-a 2024-09-11T01:42:11.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.478+0000 7efe0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:11.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.479+0000 7efe0bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:42:11.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.479+0000 7efe0bfff640 1 -- 172.21.15.12:0/34916879 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe1409ba20 con 0x7efe140a6810 2024-09-11T01:42:11.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.481+0000 7efe0bfff640 1 -- 172.21.15.12:0/34916879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe140a6810 msgr2=0x7efe140a5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:11.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.481+0000 7efe0bfff640 1 --2- 172.21.15.12:0/34916879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe140a6810 0x7efe140a5090 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7efe140a8230 tx=0x7efdfc02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=277b67480f865e9c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:11.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.481+0000 7efe11d9b640 1 -- 172.21.15.12:0/34916879 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdfc037070 con 0x7efe140a6810 2024-09-11T01:42:11.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.481+0000 7efe11d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:11.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.481+0000 7efe11d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:42:11.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.481+0000 7efe11d9b640 10 monclient: dump: 2024-09-11T01:42:11.493 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:11.493 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:11.493 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:11.493 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:11.493 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:11.493 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:11.493 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:11.493 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:11.494 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.482+0000 7efe11d9b640 10 monclient: _finish_auth 0 2024-09-11T01:42:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.482+0000 7efe11d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:42:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.482+0000 7efe11d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.482+0000 7efe11d9b640 1 -- 172.21.15.12:0/34916879 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdfc032640 con 0x7efe140a6810 2024-09-11T01:42:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.482+0000 7efe18aef640 5 monclient: authenticate success, global_id 94545 2024-09-11T01:42:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.482+0000 7efe18aef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:42:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.482+0000 7efe11d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.482+0000 7efe18aef640 10 monclient: get_monmap_and_config success 2024-09-11T01:42:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.482+0000 7efe11d9b640 1 -- 172.21.15.12:0/34916879 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdfc032a10 con 0x7efe140a6810 2024-09-11T01:42:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.482+0000 7efe11d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.482+0000 7efe18aef640 4 set_mon_vals no callback set 2024-09-11T01:42:11.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.482+0000 7efe11d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:11.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.482+0000 7efe11d9b640 10 monclient: dump: 2024-09-11T01:42:11.495 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:11.495 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:11.495 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:11.495 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:11.495 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:11.495 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:11.495 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:11.495 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:11.495 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:11.495 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:11.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.482+0000 7efe18aef640 10 monclient: shutdown 2024-09-11T01:42:11.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.482+0000 7efe18aef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:11.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.482+0000 7efe18aef640 1 -- 172.21.15.12:0/34916879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe140a6810 msgr2=0x7efe140a5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:11.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.482+0000 7efe18aef640 1 --2- 172.21.15.12:0/34916879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe140a6810 0x7efe140a5090 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7efe140a8230 tx=0x7efdfc02a110 comp rx=0 tx=0).stop 2024-09-11T01:42:11.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.482+0000 7efe18aef640 1 -- 172.21.15.12:0/34916879 shutdown_connections 2024-09-11T01:42:11.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.482+0000 7efe18aef640 1 --2- 172.21.15.12:0/34916879 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efe140a55d0 0x7efe140a59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:11.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.482+0000 7efe18aef640 1 --2- 172.21.15.12:0/34916879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe140a6810 0x7efe140a5090 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:11.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.482+0000 7efe18aef640 1 --2- 172.21.15.12:0/34916879 >> v2:172.21.15.71:3300/0 conn(0x7efe14138210 0x7efe141385f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:11.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.482+0000 7efe18aef640 1 -- 172.21.15.12:0/34916879 >> 172.21.15.12:0/34916879 conn(0x7efe140aaa80 msgr2=0x7efe140aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:11.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.482+0000 7efe18aef640 1 -- 172.21.15.12:0/34916879 shutdown_connections 2024-09-11T01:42:11.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.482+0000 7efe18aef640 1 -- 172.21.15.12:0/34916879 wait complete. 2024-09-11T01:42:11.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.483+0000 7efe18aef640 10 monclient: build_initial_monmap 2024-09-11T01:42:11.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.483+0000 7efe18aef640 10 monclient: monmap: 2024-09-11T01:42:11.496 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:11.497 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:42:11.497 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:42:11.497 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:42:11.497 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:11.497 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:11.497 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:42:11.497 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:42:11.497 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:42:11.497 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:11.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.483+0000 7efe18aef640 1 Processor -- start 2024-09-11T01:42:11.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.483+0000 7efe18aef640 1 -- start start 2024-09-11T01:42:11.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.483+0000 7efe18aef640 10 monclient: init 2024-09-11T01:42:11.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe18aef640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:11.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe18aef640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:42:11.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe18aef640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efe140a55d0 0x7efe14136740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:11.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe18aef640 10 monclient(hunting): picked mon.noname-c con 0x7efe140a55d0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:11.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe18aef640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efe140a6810 0x7efe14136c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:11.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe18aef640 10 monclient(hunting): picked mon.noname-b con 0x7efe140a6810 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:42:11.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe18aef640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe14138210 0x7efe14120ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:11.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe18aef640 10 monclient(hunting): picked mon.noname-a con 0x7efe14138210 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:42:11.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe18aef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:11.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe18aef640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7efe1406f4f0 con 0x7efe14138210 2024-09-11T01:42:11.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe18aef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:11.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe18aef640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7efe1409d620 con 0x7efe140a6810 2024-09-11T01:42:11.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe18aef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:11.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe18aef640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7efe1406f670 con 0x7efe140a55d0 2024-09-11T01:42:11.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe18aef640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:11.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe18aef640 10 monclient(hunting): authenticate will time out at 1537.927301s 2024-09-11T01:42:11.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe1259c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efe140a55d0 0x7efe14136740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:11.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe1259c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efe140a55d0 0x7efe14136740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:43056/0 (socket says 172.21.15.12:43056) 2024-09-11T01:42:11.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe1259c640 1 -- 172.21.15.12:0/3927346831 learned_addr learned my addr 172.21.15.12:0/3927346831 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:11.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe1259c640 10 monclient(hunting): get_auth_request con 0x7efe140a55d0 auth_method 0 2024-09-11T01:42:11.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe1259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:11.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe1259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:11.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe1259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:11.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe12d9d640 1 --2- 172.21.15.12:0/3927346831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe14138210 0x7efe14120ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:11.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe12d9d640 10 monclient(hunting): get_auth_request con 0x7efe14138210 auth_method 0 2024-09-11T01:42:11.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe12d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:11.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe12d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:11.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe12d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:11.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe1259c640 1 -- 172.21.15.12:0/3927346831 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efe140a55d0 msgr2=0x7efe14136740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:42:11.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe1259c640 1 -- 172.21.15.12:0/3927346831 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efe140a55d0 msgr2=0x7efe14136740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:42:11.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe1259c640 1 --2- 172.21.15.12:0/3927346831 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efe140a55d0 0x7efe14136740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:42:11.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe12d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:11.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe12d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:11.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe1259c640 1 --2- 172.21.15.12:0/3927346831 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efe140a55d0 0x7efe14136740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:11.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe12d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:11.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe0bfff640 1 --2- 172.21.15.12:0/3927346831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efe140a6810 0x7efe14136c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:11.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe0bfff640 10 monclient(hunting): get_auth_request con 0x7efe140a6810 auth_method 0 2024-09-11T01:42:11.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:11.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:11.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:11.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe12d9d640 10 monclient(hunting): handle_auth_done global_id 94548 payload 824 2024-09-11T01:42:11.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe12d9d640 1 -- 172.21.15.12:0/3927346831 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efe140a55d0 msgr2=0x7efe14136740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:42:11.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe12d9d640 1 --2- 172.21.15.12:0/3927346831 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efe140a55d0 0x7efe14136740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:11.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe12d9d640 1 -- 172.21.15.12:0/3927346831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efe140a6810 msgr2=0x7efe14136c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:11.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe12d9d640 1 --2- 172.21.15.12:0/3927346831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efe140a6810 0x7efe14136c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:11.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe12d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:42:11.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe12d9d640 1 monclient: found mon.noname-a 2024-09-11T01:42:11.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe12d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:11.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe12d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:42:11.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe12d9d640 1 -- 172.21.15.12:0/3927346831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe14124440 con 0x7efe14138210 2024-09-11T01:42:11.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.484+0000 7efe0bfff640 1 --2- 172.21.15.12:0/3927346831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efe140a6810 0x7efe14136c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:42:11.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.487+0000 7efe12d9d640 1 -- 172.21.15.12:0/3927346831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe14138210 msgr2=0x7efe14120ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:11.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.487+0000 7efe12d9d640 1 --2- 172.21.15.12:0/3927346831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe14138210 0x7efe14120ce0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7efe0c007cf0 tx=0x7efe0c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.487+0000 7efe0affd640 1 -- 172.21.15.12:0/3927346831 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0c003990 con 0x7efe14138210 2024-09-11T01:42:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.487+0000 7efe0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.487+0000 7efe0affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:42:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.487+0000 7efe0affd640 10 monclient: dump: 2024-09-11T01:42:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:11.502 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:11.503 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:11.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.487+0000 7efe0affd640 10 monclient: _finish_auth 0 2024-09-11T01:42:11.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.487+0000 7efe0affd640 10 monclient: _check_auth_tickets 2024-09-11T01:42:11.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.487+0000 7efe0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:11.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.487+0000 7efe0affd640 1 -- 172.21.15.12:0/3927346831 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe0c003b30 con 0x7efe14138210 2024-09-11T01:42:11.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.487+0000 7efe18aef640 5 monclient: authenticate success, global_id 94548 2024-09-11T01:42:11.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.487+0000 7efe18aef640 10 monclient: _renew_subs 2024-09-11T01:42:11.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.487+0000 7efe18aef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:11.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.487+0000 7efe18aef640 1 -- 172.21.15.12:0/3927346831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe14124720 con 0x7efe14138210 2024-09-11T01:42:11.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.487+0000 7efe0affd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:11.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.487+0000 7efe0affd640 1 -- 172.21.15.12:0/3927346831 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0c004020 con 0x7efe14138210 2024-09-11T01:42:11.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.487+0000 7efe0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:11.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.487+0000 7efe0affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:11.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.487+0000 7efe0affd640 10 monclient: dump: 2024-09-11T01:42:11.504 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:11.504 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:11.504 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:11.504 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:11.504 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:11.504 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:11.504 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:11.504 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:11.504 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:11.504 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:11.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.487+0000 7efe18aef640 10 monclient: _renew_subs 2024-09-11T01:42:11.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.487+0000 7efe1159a640 4 set_mon_vals no callback set 2024-09-11T01:42:11.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.487+0000 7efe18aef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:11.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.487+0000 7efe18aef640 1 -- 172.21.15.12:0/3927346831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe140a1150 con 0x7efe14138210 2024-09-11T01:42:11.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.488+0000 7efe0affd640 1 -- 172.21.15.12:0/3927346831 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7efe0c004240 con 0x7efe14138210 2024-09-11T01:42:11.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.488+0000 7efe18aef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:11.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.488+0000 7efe18aef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:11.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.488+0000 7efe18aef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:11.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.488+0000 7efe18aef640 1 -- 172.21.15.12:0/3927346831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe140a5090 con 0x7efe14138210 2024-09-11T01:42:11.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.488+0000 7efe0affd640 1 --2- 172.21.15.12:0/3927346831 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efdf403e240 0x7efdf4040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:11.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.488+0000 7efe0affd640 1 -- 172.21.15.12:0/3927346831 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(433..433 src has 285..433) ==== 4755+0+0 (secure 0 0 0) 0x7efe0c0531d0 con 0x7efe14138210 2024-09-11T01:42:11.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.488+0000 7efe1259c640 1 --2- 172.21.15.12:0/3927346831 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efdf403e240 0x7efdf4040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:11.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.488+0000 7efe1259c640 10 monclient: get_auth_request con 0x7efdf403e240 auth_method 0 2024-09-11T01:42:11.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.491+0000 7efe1259c640 1 --2- 172.21.15.12:0/3927346831 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efdf403e240 0x7efdf4040700 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7efe040095c0 tx=0x7efe04008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:11.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.493+0000 7efe0affd640 1 -- 172.21.15.12:0/3927346831 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7efe0c01c370 con 0x7efe14138210 2024-09-11T01:42:11.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.493+0000 7efe0affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:11.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.493+0000 7efe0affd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:42:11.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.749+0000 7efe18aef640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:42:11.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.749+0000 7efe18aef640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:42:11.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.749+0000 7efe18aef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:11.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.749+0000 7efe18aef640 1 -- 172.21.15.12:0/3927346831 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7efe141241c0 con 0x7efe14138210 2024-09-11T01:42:11.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.750+0000 7efe0affd640 1 -- 172.21.15.12:0/3927346831 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7efe0c051be0 con 0x7efe14138210 2024-09-11T01:42:11.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.750+0000 7efe0affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:42:11.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.750+0000 7efe0affd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:42:11.753 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":118,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:42:11.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.752+0000 7efe18aef640 1 -- 172.21.15.12:0/3927346831 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efdf403e240 msgr2=0x7efdf4040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:11.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.752+0000 7efe18aef640 1 --2- 172.21.15.12:0/3927346831 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efdf403e240 0x7efdf4040700 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7efe040095c0 tx=0x7efe04008040 comp rx=0 tx=0).stop 2024-09-11T01:42:11.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.752+0000 7efe18aef640 10 monclient: shutdown 2024-09-11T01:42:11.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.752+0000 7efe18aef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:11.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.752+0000 7efe18aef640 1 -- 172.21.15.12:0/3927346831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe14138210 msgr2=0x7efe14120ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:11.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.752+0000 7efe18aef640 1 --2- 172.21.15.12:0/3927346831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe14138210 0x7efe14120ce0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7efe0c007cf0 tx=0x7efe0c00e4b0 comp rx=0 tx=0).stop 2024-09-11T01:42:11.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.752+0000 7efe18aef640 1 -- 172.21.15.12:0/3927346831 shutdown_connections 2024-09-11T01:42:11.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.752+0000 7efe18aef640 1 --2- 172.21.15.12:0/3927346831 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efdf403e240 0x7efdf4040700 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:11.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.752+0000 7efe18aef640 1 --2- 172.21.15.12:0/3927346831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe14138210 0x7efe14120ce0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:11.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.752+0000 7efe18aef640 1 --2- 172.21.15.12:0/3927346831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efe140a6810 0x7efe14136c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:11.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.752+0000 7efe18aef640 1 --2- 172.21.15.12:0/3927346831 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efe140a55d0 0x7efe14136740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:11.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.752+0000 7efe18aef640 1 -- 172.21.15.12:0/3927346831 >> 172.21.15.12:0/3927346831 conn(0x7efe140aaa80 msgr2=0x7efe140a04d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:11.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.752+0000 7efe18aef640 1 -- 172.21.15.12:0/3927346831 shutdown_connections 2024-09-11T01:42:11.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:11.752+0000 7efe18aef640 1 -- 172.21.15.12:0/3927346831 wait complete. 2024-09-11T01:42:11.996 INFO:tasks.rados.rados.0.smithi071.stdout:1901: done (5 left) 2024-09-11T01:42:11.996 INFO:tasks.rados.rados.0.smithi071.stdout:1902: read oid 15 snap -1 2024-09-11T01:42:11.996 INFO:tasks.rados.rados.0.smithi071.stdout:1902: expect (ObjNum 565 snap 160 seq_num 565) 2024-09-11T01:42:11.996 INFO:tasks.rados.rados.0.smithi071.stdout:1903: read oid 42 snap -1 2024-09-11T01:42:11.996 INFO:tasks.rados.rados.0.smithi071.stdout:1903: expect (ObjNum 648 snap 199 seq_num 648) 2024-09-11T01:42:11.996 INFO:tasks.rados.rados.0.smithi071.stdout:1896: finishing write tid 3 to smithi07134777-13 2024-09-11T01:42:11.996 INFO:tasks.rados.rados.0.smithi071.stdout:1904: read oid 45 snap -1 2024-09-11T01:42:11.996 INFO:tasks.rados.rados.0.smithi071.stdout:1904: expect (ObjNum 612 snap 181 seq_num 612) 2024-09-11T01:42:11.997 INFO:tasks.rados.rados.0.smithi071.stdout:1905: snap_remove snap 186 2024-09-11T01:42:13.004 INFO:tasks.rados.rados.0.smithi071.stdout:1905: done (8 left) 2024-09-11T01:42:13.004 INFO:tasks.rados.rados.0.smithi071.stdout:1906: read oid 36 snap -1 2024-09-11T01:42:13.004 INFO:tasks.rados.rados.0.smithi071.stdout:1906: expect (ObjNum 687 snap 204 seq_num 687) 2024-09-11T01:42:13.004 INFO:tasks.rados.rados.0.smithi071.stdout:1907: delete oid 17 current snap is 211 2024-09-11T01:42:13.004 INFO:tasks.rados.rados.0.smithi071.stdout:1896: finishing write tid 4 to smithi07134777-13 2024-09-11T01:42:13.004 INFO:tasks.rados.rados.0.smithi071.stdout:1896: finishing write tid 5 to smithi07134777-13 2024-09-11T01:42:13.004 INFO:tasks.rados.rados.0.smithi071.stdout:1896: oid 13 updating version 0 to 829 2024-09-11T01:42:13.004 INFO:tasks.rados.rados.0.smithi071.stdout:1896: oid 13 updating version 829 to 830 2024-09-11T01:42:13.004 INFO:tasks.rados.rados.0.smithi071.stdout:1896: oid 13 updating version 830 to 831 2024-09-11T01:42:13.004 INFO:tasks.rados.rados.0.smithi071.stdout:1896: oid 13 version 831 is already newer than 828 2024-09-11T01:42:13.004 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 831 (ObjNum 700 snap 211 seq_num 700) dirty exists 2024-09-11T01:42:13.004 INFO:tasks.rados.rados.0.smithi071.stdout:1896: left oid 13 (ObjNum 700 snap 211 seq_num 700) 2024-09-11T01:42:13.005 INFO:tasks.rados.rados.0.smithi071.stdout:1897: finishing rollback tid 1 to smithi07134777-4 2024-09-11T01:42:13.005 INFO:tasks.rados.rados.0.smithi071.stdout:1897: finishing rollback tid 2 to smithi07134777-4 2024-09-11T01:42:13.005 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 911 (ObjNum 644 snap 196 seq_num 644) dirty exists 2024-09-11T01:42:13.005 INFO:tasks.rados.rados.0.smithi071.stdout:1898: finishing write tid 1 to smithi07134777-39 2024-09-11T01:42:13.005 INFO:tasks.rados.rados.0.smithi071.stdout:1898: finishing write tid 2 to smithi07134777-39 2024-09-11T01:42:13.005 INFO:tasks.rados.rados.0.smithi071.stdout:1900: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:42:13.005 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 193578 2024-09-11T01:42:13.089 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 872819 2024-09-11T01:42:13.118 INFO:tasks.rados.rados.0.smithi071.stdout:1898: finishing write tid 3 to smithi07134777-39 2024-09-11T01:42:13.118 INFO:tasks.rados.rados.0.smithi071.stdout:1898: finishing write tid 4 to smithi07134777-39 2024-09-11T01:42:13.118 INFO:tasks.rados.rados.0.smithi071.stdout:1898: finishing write tid 5 to smithi07134777-39 2024-09-11T01:42:13.118 INFO:tasks.rados.rados.0.smithi071.stdout:1898: oid 39 updating version 0 to 969 2024-09-11T01:42:13.118 INFO:tasks.rados.rados.0.smithi071.stdout:1898: oid 39 version 969 is already newer than 967 2024-09-11T01:42:13.118 INFO:tasks.rados.rados.0.smithi071.stdout:1898: oid 39 version 969 is already newer than 966 2024-09-11T01:42:13.118 INFO:tasks.rados.rados.0.smithi071.stdout:1898: oid 39 version 969 is already newer than 968 2024-09-11T01:42:13.118 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 969 (ObjNum 701 snap 211 seq_num 701) dirty exists 2024-09-11T01:42:13.118 INFO:tasks.rados.rados.0.smithi071.stdout:1898: left oid 39 (ObjNum 701 snap 211 seq_num 701) 2024-09-11T01:42:13.118 INFO:tasks.rados.rados.0.smithi071.stdout:1896: done (9 left) 2024-09-11T01:42:13.118 INFO:tasks.rados.rados.0.smithi071.stdout:1897: done (8 left) 2024-09-11T01:42:13.118 INFO:tasks.rados.rados.0.smithi071.stdout:1898: done (7 left) 2024-09-11T01:42:13.119 INFO:tasks.rados.rados.0.smithi071.stdout:1899: done (6 left) 2024-09-11T01:42:13.119 INFO:tasks.rados.rados.0.smithi071.stdout:1900: done (5 left) 2024-09-11T01:42:13.119 INFO:tasks.rados.rados.0.smithi071.stdout:1907: done (4 left) 2024-09-11T01:42:13.119 INFO:tasks.rados.rados.0.smithi071.stdout:1908: snap_create 2024-09-11T01:42:13.119 INFO:tasks.rados.rados.0.smithi071.stdout:1902: expect (ObjNum 565 snap 160 seq_num 565) 2024-09-11T01:42:13.136 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1628322 2024-09-11T01:42:13.215 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2286839 2024-09-11T01:42:13.327 INFO:tasks.rados.rados.0.smithi071.stdout:1904: expect (ObjNum 612 snap 181 seq_num 612) 2024-09-11T01:42:13.327 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:13.400 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1174461 2024-09-11T01:42:13.422 INFO:tasks.rados.rados.0.smithi071.stdout:1903: expect (ObjNum 648 snap 199 seq_num 648) 2024-09-11T01:42:13.480 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 34245 2024-09-11T01:42:13.511 INFO:tasks.rados.rados.0.smithi071.stdout:1906: expect (ObjNum 687 snap 204 seq_num 687) 2024-09-11T01:42:13.553 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1734103 2024-09-11T01:42:13.967 INFO:tasks.rados.rados.0.smithi071.stdout:1902: done (4 left) 2024-09-11T01:42:13.967 INFO:tasks.rados.rados.0.smithi071.stdout:1903: done (3 left) 2024-09-11T01:42:13.967 INFO:tasks.rados.rados.0.smithi071.stdout:1904: done (2 left) 2024-09-11T01:42:13.967 INFO:tasks.rados.rados.0.smithi071.stdout:1906: done (1 left) 2024-09-11T01:42:13.967 INFO:tasks.rados.rados.0.smithi071.stdout:1908: done (0 left) 2024-09-11T01:42:13.967 INFO:tasks.rados.rados.0.smithi071.stdout:1909: snap_remove snap 211 2024-09-11T01:42:14.768 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:42:14.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.878+0000 7f2ef0dec640 10 monclient: get_monmap_and_config 2024-09-11T01:42:14.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.878+0000 7f2ef0dec640 10 monclient: build_initial_monmap 2024-09-11T01:42:14.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.878+0000 7f2ef0dec640 10 monclient: monmap: 2024-09-11T01:42:14.881 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:14.881 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:14.881 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:42:14.880284+0000 2024-09-11T01:42:14.881 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:42:14.880284+0000 2024-09-11T01:42:14.881 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:14.881 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:14.881 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:42:14.881 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:42:14.881 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:42:14.882 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:42:14.882 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:42:14.882 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.879+0000 7f2ef0dec640 1 Processor -- start 2024-09-11T01:42:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.879+0000 7f2ef0dec640 1 -- start start 2024-09-11T01:42:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.879+0000 7f2ef0dec640 10 monclient: init 2024-09-11T01:42:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.879+0000 7f2ef0dec640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.879+0000 7f2ef0dec640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-11T01:42:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.879+0000 7f2ef0dec640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2eec138230 0x7f2eec138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.879+0000 7f2ef0dec640 10 monclient(hunting): picked mon.noname-a con 0x7f2eec138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:42:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.879+0000 7f2ef0dec640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2eec09d030 0x7f2eec0a5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.879+0000 7f2ef0dec640 10 monclient(hunting): picked mon.noname-c con 0x7f2eec09d030 addr v2:172.21.15.71:3300/0 2024-09-11T01:42:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.879+0000 7f2ef0dec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2eec09d460 addr v1:172.21.15.71:6789/0 2024-09-11T01:42:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.879+0000 7f2ef0dec640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2eec0a6e30 con 0x7f2eec09d460 2024-09-11T01:42:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.879+0000 7f2ef0dec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.879+0000 7f2ef0dec640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f2eec09c9c0 con 0x7f2eec138230 2024-09-11T01:42:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.879+0000 7f2ef0dec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.879+0000 7f2ef0dec640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f2eec09ba20 con 0x7f2eec09d030 2024-09-11T01:42:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.879+0000 7f2ef0dec640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.879+0000 7f2eea59c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2eec138230 0x7f2eec138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.879+0000 7f2ef0dec640 10 monclient(hunting): authenticate will time out at 1541.322988s 2024-09-11T01:42:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.880+0000 7f2eea59c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2eec138230 0x7f2eec138610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46520/0 (socket says 172.21.15.12:46520) 2024-09-11T01:42:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.880+0000 7f2eea59c640 1 -- 172.21.15.12:0/2055800508 learned_addr learned my addr 172.21.15.12:0/2055800508 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.880+0000 7f2eea59c640 10 monclient(hunting): get_auth_request con 0x7f2eec138230 auth_method 0 2024-09-11T01:42:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.880+0000 7f2eea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.880+0000 7f2eea59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.880+0000 7f2eea59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.880+0000 7f2eea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.880+0000 7f2eea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.880+0000 7f2eea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.880+0000 7f2ee1d9b640 1 --2- 172.21.15.12:0/2055800508 >> v2:172.21.15.71:3300/0 conn(0x7f2eec09d030 0x7f2eec0a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.880+0000 7f2eea59c640 10 monclient(hunting): handle_auth_done global_id 94551 payload 274 2024-09-11T01:42:14.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.880+0000 7f2eea59c640 1 -- 172.21.15.12:0/2055800508 >> v2:172.21.15.71:3300/0 conn(0x7f2eec09d030 msgr2=0x7f2eec0a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:14.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.880+0000 7f2eea59c640 1 --2- 172.21.15.12:0/2055800508 >> v2:172.21.15.71:3300/0 conn(0x7f2eec09d030 0x7f2eec0a5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:14.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.880+0000 7f2eea59c640 1 -- 172.21.15.12:0/2055800508 >> v1:172.21.15.71:6789/0 conn(0x7f2eec09d460 legacy=0x7f2eec0a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:14.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.880+0000 7f2eea59c640 10 monclient: _finish_hunting 0 2024-09-11T01:42:14.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.880+0000 7f2eea59c640 1 monclient: found mon.noname-a 2024-09-11T01:42:14.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.880+0000 7f2eea59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:14.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.880+0000 7f2eea59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:42:14.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.880+0000 7f2eea59c640 1 -- 172.21.15.12:0/2055800508 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eec0a66a0 con 0x7f2eec138230 2024-09-11T01:42:14.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.880+0000 7f2ee1d9b640 10 monclient: get_auth_request con 0x7f2eec09d030 auth_method 0 2024-09-11T01:42:14.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.880+0000 7f2ee1d9b640 1 --2- 172.21.15.12:0/2055800508 >> v2:172.21.15.71:3300/0 conn(0x7f2eec09d030 0x7f2eec0a5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:42:14.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2eea59c640 1 -- 172.21.15.12:0/2055800508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2eec138230 msgr2=0x7f2eec138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:14.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2eea59c640 1 --2- 172.21.15.12:0/2055800508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2eec138230 0x7f2eec138610 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2eec09dae0 tx=0x7f2edc02a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8bfda08740a1a3e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:14.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ee9d9b640 1 -- 172.21.15.12:0/2055800508 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2edc037070 con 0x7f2eec138230 2024-09-11T01:42:14.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ee9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:14.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ee9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:42:14.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ee9d9b640 10 monclient: dump: 2024-09-11T01:42:14.893 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:14.893 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:14.893 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:14.893 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:14.894 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:14.894 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:14.894 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:14.894 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:14.894 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:14.894 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:14.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ee9d9b640 10 monclient: _finish_auth 0 2024-09-11T01:42:14.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ee9d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:42:14.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ee9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:14.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ef0dec640 5 monclient: authenticate success, global_id 94551 2024-09-11T01:42:14.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ef0dec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:42:14.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ee9d9b640 1 -- 172.21.15.12:0/2055800508 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2edc032640 con 0x7f2eec138230 2024-09-11T01:42:14.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ee9d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:14.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ef0dec640 10 monclient: get_monmap_and_config success 2024-09-11T01:42:14.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ef0dec640 4 set_mon_vals no callback set 2024-09-11T01:42:14.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ee9d9b640 1 -- 172.21.15.12:0/2055800508 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2edc032a10 con 0x7f2eec138230 2024-09-11T01:42:14.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ee9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:14.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ee9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:14.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ee9d9b640 10 monclient: dump: 2024-09-11T01:42:14.895 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:14.895 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:14.895 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:14.895 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:14.895 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:14.895 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:14.895 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:14.896 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:14.896 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:14.896 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:14.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ef0dec640 10 monclient: shutdown 2024-09-11T01:42:14.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ef0dec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:14.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ef0dec640 1 -- 172.21.15.12:0/2055800508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2eec138230 msgr2=0x7f2eec138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:14.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ef0dec640 1 --2- 172.21.15.12:0/2055800508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2eec138230 0x7f2eec138610 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2eec09dae0 tx=0x7f2edc02a0b0 comp rx=0 tx=0).stop 2024-09-11T01:42:14.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ef0dec640 1 -- 172.21.15.12:0/2055800508 shutdown_connections 2024-09-11T01:42:14.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ef0dec640 1 --2- 172.21.15.12:0/2055800508 >> v2:172.21.15.71:3300/0 conn(0x7f2eec09d030 0x7f2eec0a5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:14.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ef0dec640 1 --2- 172.21.15.12:0/2055800508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2eec138230 0x7f2eec138610 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:14.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ef0dec640 1 -- 172.21.15.12:0/2055800508 >> 172.21.15.12:0/2055800508 conn(0x7f2eec0a7120 msgr2=0x7f2eec0a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:14.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ef0dec640 1 -- 172.21.15.12:0/2055800508 shutdown_connections 2024-09-11T01:42:14.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.883+0000 7f2ef0dec640 1 -- 172.21.15.12:0/2055800508 wait complete. 2024-09-11T01:42:14.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.884+0000 7f2ef0dec640 10 monclient: build_initial_monmap 2024-09-11T01:42:14.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.884+0000 7f2ef0dec640 10 monclient: monmap: 2024-09-11T01:42:14.897 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:14.897 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:42:14.897 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:42:14.897 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:42:14.897 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:14.897 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:14.897 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:42:14.897 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:42:14.897 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:42:14.897 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:14.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2ef0dec640 1 Processor -- start 2024-09-11T01:42:14.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2ef0dec640 1 -- start start 2024-09-11T01:42:14.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2ef0dec640 10 monclient: init 2024-09-11T01:42:14.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2ef0dec640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:14.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2ef0dec640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:42:14.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2ef0dec640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2eec09d030 0x7f2eec1ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:14.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2ef0dec640 10 monclient(hunting): picked mon.noname-c con 0x7f2eec09d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:14.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2ef0dec640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2eec09d460 0x7f2eec136fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:14.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2ef0dec640 10 monclient(hunting): picked mon.noname-a con 0x7f2eec09d460 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:42:14.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2ef0dec640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2eec138230 0x7f2eec137510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:14.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2ef0dec640 10 monclient(hunting): picked mon.noname-b con 0x7f2eec138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:42:14.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2ef0dec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:14.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2ef0dec640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2eec06f970 con 0x7f2eec09d460 2024-09-11T01:42:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2ef0dec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2ef0dec640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f2eec06f7f0 con 0x7f2eec138230 2024-09-11T01:42:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2ef0dec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2ef0dec640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2eec0a5c70 con 0x7f2eec09d030 2024-09-11T01:42:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2ef0dec640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2ef0dec640 10 monclient(hunting): authenticate will time out at 1541.328784s 2024-09-11T01:42:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2ee1d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2eec09d460 0x7f2eec136fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2eea59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2eec09d030 0x7f2eec1ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2ee1d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2eec09d460 0x7f2eec136fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46524/0 (socket says 172.21.15.12:46524) 2024-09-11T01:42:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2ee1d9b640 1 -- 172.21.15.12:0/1942172279 learned_addr learned my addr 172.21.15.12:0/1942172279 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2ee1d9b640 10 monclient(hunting): get_auth_request con 0x7f2eec09d460 auth_method 0 2024-09-11T01:42:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2ee1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:14.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2ee1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:14.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2ee1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:14.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2eea59c640 10 monclient(hunting): get_auth_request con 0x7f2eec09d030 auth_method 0 2024-09-11T01:42:14.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2eea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:14.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2eea59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:14.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2eea59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:14.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.885+0000 7f2eead9d640 1 --2- 172.21.15.12:0/1942172279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2eec138230 0x7f2eec137510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:14.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.886+0000 7f2ee1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:14.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.886+0000 7f2ee1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:14.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.886+0000 7f2eea59c640 1 -- 172.21.15.12:0/1942172279 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2eec09d030 msgr2=0x7f2eec1ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:42:14.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.886+0000 7f2ee1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:14.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.886+0000 7f2eea59c640 1 -- 172.21.15.12:0/1942172279 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2eec09d030 msgr2=0x7f2eec1ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:42:14.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.886+0000 7f2eead9d640 10 monclient(hunting): get_auth_request con 0x7f2eec138230 auth_method 0 2024-09-11T01:42:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.886+0000 7f2eead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.886+0000 7f2eead9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.886+0000 7f2eead9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.886+0000 7f2eea59c640 1 --2- 172.21.15.12:0/1942172279 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2eec09d030 0x7f2eec1ccdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:42:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.886+0000 7f2eea59c640 1 --2- 172.21.15.12:0/1942172279 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2eec09d030 0x7f2eec1ccdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.886+0000 7f2eead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.886+0000 7f2eead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.886+0000 7f2eead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.886+0000 7f2ee1d9b640 10 monclient(hunting): handle_auth_done global_id 94554 payload 824 2024-09-11T01:42:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.886+0000 7f2ee1d9b640 1 -- 172.21.15.12:0/1942172279 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2eec09d030 msgr2=0x7f2eec1ccdd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:42:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.886+0000 7f2ee1d9b640 1 --2- 172.21.15.12:0/1942172279 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2eec09d030 0x7f2eec1ccdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.886+0000 7f2ee1d9b640 1 -- 172.21.15.12:0/1942172279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2eec138230 msgr2=0x7f2eec137510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.886+0000 7f2ee1d9b640 1 --2- 172.21.15.12:0/1942172279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2eec138230 0x7f2eec137510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:14.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.886+0000 7f2ee1d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:42:14.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.886+0000 7f2ee1d9b640 1 monclient: found mon.noname-a 2024-09-11T01:42:14.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.886+0000 7f2ee1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:14.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.886+0000 7f2ee1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:42:14.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.886+0000 7f2ee1d9b640 1 -- 172.21.15.12:0/1942172279 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eec126670 con 0x7f2eec09d460 2024-09-11T01:42:14.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.888+0000 7f2ee1d9b640 1 -- 172.21.15.12:0/1942172279 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2eec09d460 msgr2=0x7f2eec136fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:14.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.888+0000 7f2ee1d9b640 1 --2- 172.21.15.12:0/1942172279 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2eec09d460 0x7f2eec136fd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2ed400c7f0 tx=0x7f2ed400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:14.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.888+0000 7f2ee37fe640 1 -- 172.21.15.12:0/1942172279 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed4015070 con 0x7f2eec09d460 2024-09-11T01:42:14.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.888+0000 7f2ee37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:14.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.888+0000 7f2ee37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:42:14.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.888+0000 7f2ee37fe640 10 monclient: dump: 2024-09-11T01:42:14.902 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:14.902 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:14.902 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:14.902 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:14.903 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:14.903 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:14.903 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:14.903 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:14.903 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:14.903 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:14.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.888+0000 7f2ee37fe640 10 monclient: _finish_auth 0 2024-09-11T01:42:14.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.888+0000 7f2ee37fe640 10 monclient: _check_auth_tickets 2024-09-11T01:42:14.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.888+0000 7f2ee37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:14.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.888+0000 7f2ee37fe640 1 -- 172.21.15.12:0/1942172279 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ed4007c40 con 0x7f2eec09d460 2024-09-11T01:42:14.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.888+0000 7f2ee37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:14.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.888+0000 7f2ee37fe640 1 -- 172.21.15.12:0/1942172279 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed400ce40 con 0x7f2eec09d460 2024-09-11T01:42:14.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.888+0000 7f2ee37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:14.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.888+0000 7f2ee37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:14.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.888+0000 7f2ee37fe640 10 monclient: dump: 2024-09-11T01:42:14.904 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:14.904 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:14.904 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:14.904 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:14.904 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:14.904 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:14.905 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.888+0000 7f2ef0dec640 5 monclient: authenticate success, global_id 94554 2024-09-11T01:42:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.888+0000 7f2ef0dec640 10 monclient: _renew_subs 2024-09-11T01:42:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.888+0000 7f2ef0dec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.888+0000 7f2ee959a640 4 set_mon_vals no callback set 2024-09-11T01:42:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.888+0000 7f2ef0dec640 1 -- 172.21.15.12:0/1942172279 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eec137a50 con 0x7f2eec09d460 2024-09-11T01:42:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.888+0000 7f2ef0dec640 10 monclient: _renew_subs 2024-09-11T01:42:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.888+0000 7f2ef0dec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.888+0000 7f2ef0dec640 1 -- 172.21.15.12:0/1942172279 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eec1289d0 con 0x7f2eec09d460 2024-09-11T01:42:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.889+0000 7f2ee37fe640 1 -- 172.21.15.12:0/1942172279 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f2ed40113f0 con 0x7f2eec09d460 2024-09-11T01:42:14.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.889+0000 7f2ef0dec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.889+0000 7f2ef0dec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.889+0000 7f2ef0dec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.889+0000 7f2ee37fe640 1 --2- 172.21.15.12:0/1942172279 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2ecc03e210 0x7f2ecc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.889+0000 7f2ef0dec640 1 -- 172.21.15.12:0/1942172279 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eec138610 con 0x7f2eec09d460 2024-09-11T01:42:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.890+0000 7f2ee37fe640 1 -- 172.21.15.12:0/1942172279 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(436..436 src has 285..436) ==== 4799+0+0 (secure 0 0 0) 0x7f2ed4053180 con 0x7f2eec09d460 2024-09-11T01:42:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.890+0000 7f2eea59c640 1 --2- 172.21.15.12:0/1942172279 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2ecc03e210 0x7f2ecc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.890+0000 7f2eea59c640 10 monclient: get_auth_request con 0x7f2ecc03e210 auth_method 0 2024-09-11T01:42:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.892+0000 7f2eea59c640 1 --2- 172.21.15.12:0/1942172279 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2ecc03e210 0x7f2ecc0406d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2eec09bff0 tx=0x7f2edc02e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:14.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.894+0000 7f2ee37fe640 1 -- 172.21.15.12:0/1942172279 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2ed4051b70 con 0x7f2eec09d460 2024-09-11T01:42:14.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.894+0000 7f2ee37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:14.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:14.894+0000 7f2ee37fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:42:14.975 INFO:tasks.rados.rados.0.smithi071.stdout:1909: done (0 left) 2024-09-11T01:42:14.975 INFO:tasks.rados.rados.0.smithi071.stdout:1910: delete oid 16 current snap is 212 2024-09-11T01:42:14.980 INFO:tasks.rados.rados.0.smithi071.stdout:1910: done (0 left) 2024-09-11T01:42:14.980 INFO:tasks.rados.rados.0.smithi071.stdout:1911: snap_create 2024-09-11T01:42:15.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.152+0000 7f2ef0dec640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:42:15.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.152+0000 7f2ef0dec640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:42:15.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.152+0000 7f2ef0dec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:15.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.152+0000 7f2ef0dec640 1 -- 172.21.15.12:0/1942172279 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2eec1271d0 con 0x7f2eec09d460 2024-09-11T01:42:15.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.152+0000 7f2ee37fe640 1 -- 172.21.15.12:0/1942172279 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f2ed4021d10 con 0x7f2eec09d460 2024-09-11T01:42:15.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.152+0000 7f2ee37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:42:15.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.152+0000 7f2ee37fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:42:15.155 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":122,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:42:15.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.154+0000 7f2ef0dec640 1 -- 172.21.15.12:0/1942172279 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2ecc03e210 msgr2=0x7f2ecc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:15.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.154+0000 7f2ef0dec640 1 --2- 172.21.15.12:0/1942172279 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2ecc03e210 0x7f2ecc0406d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2eec09bff0 tx=0x7f2edc02e000 comp rx=0 tx=0).stop 2024-09-11T01:42:15.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.154+0000 7f2ef0dec640 10 monclient: shutdown 2024-09-11T01:42:15.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.154+0000 7f2ef0dec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:15.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.154+0000 7f2ef0dec640 1 -- 172.21.15.12:0/1942172279 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2eec09d460 msgr2=0x7f2eec136fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:15.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.154+0000 7f2ef0dec640 1 --2- 172.21.15.12:0/1942172279 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2eec09d460 0x7f2eec136fd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2ed400c7f0 tx=0x7f2ed400cbf0 comp rx=0 tx=0).stop 2024-09-11T01:42:15.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.154+0000 7f2ef0dec640 1 -- 172.21.15.12:0/1942172279 shutdown_connections 2024-09-11T01:42:15.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.154+0000 7f2ef0dec640 1 --2- 172.21.15.12:0/1942172279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2eec138230 0x7f2eec137510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:15.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.154+0000 7f2ef0dec640 1 --2- 172.21.15.12:0/1942172279 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2ecc03e210 0x7f2ecc0406d0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:15.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.154+0000 7f2ef0dec640 1 --2- 172.21.15.12:0/1942172279 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2eec09d460 0x7f2eec136fd0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:15.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.154+0000 7f2ef0dec640 1 --2- 172.21.15.12:0/1942172279 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2eec09d030 0x7f2eec1ccdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:15.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.154+0000 7f2ef0dec640 1 -- 172.21.15.12:0/1942172279 >> 172.21.15.12:0/1942172279 conn(0x7f2eec0a7120 msgr2=0x7f2eec0a31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:15.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.154+0000 7f2ef0dec640 1 -- 172.21.15.12:0/1942172279 shutdown_connections 2024-09-11T01:42:15.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.154+0000 7f2ef0dec640 1 -- 172.21.15.12:0/1942172279 wait complete. 2024-09-11T01:42:15.167 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:42:15.167 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:42:15.167 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:42:15.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.266+0000 7fa9f0a16640 10 monclient: get_monmap_and_config 2024-09-11T01:42:15.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.266+0000 7fa9f0a16640 10 monclient: build_initial_monmap 2024-09-11T01:42:15.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.266+0000 7fa9f0a16640 10 monclient: monmap: 2024-09-11T01:42:15.269 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:15.269 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:15.269 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:42:15.268476+0000 2024-09-11T01:42:15.269 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:42:15.268476+0000 2024-09-11T01:42:15.270 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:15.270 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:15.270 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:42:15.270 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:42:15.270 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:42:15.270 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:42:15.270 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:42:15.270 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:15.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.267+0000 7fa9f0a16640 1 Processor -- start 2024-09-11T01:42:15.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.267+0000 7fa9f0a16640 1 -- start start 2024-09-11T01:42:15.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.267+0000 7fa9f0a16640 10 monclient: init 2024-09-11T01:42:15.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.267+0000 7fa9f0a16640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:15.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.267+0000 7fa9f0a16640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-11T01:42:15.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.267+0000 7fa9f0a16640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9ec134ad0 0x7fa9ec134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.267+0000 7fa9f0a16640 10 monclient(hunting): picked mon.noname-b con 0x7fa9ec134ad0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9f0a16640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa9ec0a2200 addr v1:172.21.15.71:6789/0 2024-09-11T01:42:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9f0a16640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa9ec1353f0 0x7fa9ec134160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9f0a16640 10 monclient(hunting): picked mon.noname-c con 0x7fa9ec1353f0 addr v2:172.21.15.71:3300/0 2024-09-11T01:42:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9f0a16640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9ec09ed30 con 0x7fa9ec0a2200 2024-09-11T01:42:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9f0a16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9f0a16640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fa9ec09f5d0 con 0x7fa9ec1353f0 2024-09-11T01:42:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9f0a16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9f0a16640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa9ec09f010 con 0x7fa9ec134ad0 2024-09-11T01:42:15.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9f0a16640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:15.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9f0a16640 10 monclient(hunting): authenticate will time out at 1541.711068s 2024-09-11T01:42:15.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9ea59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9ec134ad0 0x7fa9ec134eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:15.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9ea59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9ec134ad0 0x7fa9ec134eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:43088/0 (socket says 172.21.15.12:43088) 2024-09-11T01:42:15.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9ea59c640 1 -- 172.21.15.12:0/286183124 learned_addr learned my addr 172.21.15.12:0/286183124 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:15.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9ea59c640 10 monclient(hunting): get_auth_request con 0x7fa9ec134ad0 auth_method 0 2024-09-11T01:42:15.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:15.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9ea59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:15.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9ea59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:15.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9ea59c640 1 -- 172.21.15.12:0/286183124 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9ec134ad0 msgr2=0x7fa9ec134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:42:15.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9ea59c640 1 -- 172.21.15.12:0/286183124 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9ec134ad0 msgr2=0x7fa9ec134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:42:15.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9ea59c640 1 --2- 172.21.15.12:0/286183124 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9ec134ad0 0x7fa9ec134eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:42:15.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9ea59c640 1 --2- 172.21.15.12:0/286183124 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9ec134ad0 0x7fa9ec134eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:15.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9ead9d640 1 --2- 172.21.15.12:0/286183124 >> v2:172.21.15.71:3300/0 conn(0x7fa9ec1353f0 0x7fa9ec134160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:15.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9ead9d640 10 monclient(hunting): get_auth_request con 0x7fa9ec1353f0 auth_method 0 2024-09-11T01:42:15.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:15.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9ead9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:15.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9ead9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:15.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9ead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:15.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9ead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:15.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.268+0000 7fa9ead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:15.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.270+0000 7fa9e3fff640 1 -- 172.21.15.12:0/286183124 >> v1:172.21.15.71:6789/0 conn(0x7fa9ec0a2200 legacy=0x7fa9ec133a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:15.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.270+0000 7fa9ead9d640 10 monclient(hunting): handle_auth_done global_id 134188 payload 274 2024-09-11T01:42:15.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.270+0000 7fa9ead9d640 1 -- 172.21.15.12:0/286183124 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9ec134ad0 msgr2=0x7fa9ec134eb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:42:15.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.270+0000 7fa9ead9d640 1 --2- 172.21.15.12:0/286183124 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9ec134ad0 0x7fa9ec134eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:15.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.270+0000 7fa9ead9d640 1 -- 172.21.15.12:0/286183124 >> v1:172.21.15.71:6789/0 conn(0x7fa9ec0a2200 legacy=0x7fa9ec133a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:15.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.270+0000 7fa9ead9d640 10 monclient: _finish_hunting 0 2024-09-11T01:42:15.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.270+0000 7fa9ead9d640 1 monclient: found mon.noname-c 2024-09-11T01:42:15.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.270+0000 7fa9ead9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:15.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.270+0000 7fa9ead9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:42:15.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.270+0000 7fa9ead9d640 1 -- 172.21.15.12:0/286183124 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9ec09f2f0 con 0x7fa9ec1353f0 2024-09-11T01:42:15.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.273+0000 7fa9ead9d640 1 -- 172.21.15.12:0/286183124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9ec1353f0 msgr2=0x7fa9ec134160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:15.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.273+0000 7fa9ead9d640 1 --2- 172.21.15.12:0/286183124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9ec1353f0 0x7fa9ec134160 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4009e10 tx=0x7fa9e402a5b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f23365e8d0057aed server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:15.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.274+0000 7fa9e9d9b640 1 -- 172.21.15.12:0/286183124 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e4007950 con 0x7fa9ec1353f0 2024-09-11T01:42:15.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.274+0000 7fa9e9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:15.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.274+0000 7fa9e9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:42:15.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.274+0000 7fa9e9d9b640 10 monclient: dump: 2024-09-11T01:42:15.278 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:15.278 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:15.279 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:15.279 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:15.279 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:15.279 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:15.279 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:15.279 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:15.279 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:15.279 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:15.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.274+0000 7fa9e9d9b640 10 monclient: _finish_auth 0 2024-09-11T01:42:15.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.274+0000 7fa9e9d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:42:15.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.274+0000 7fa9e9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:15.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.274+0000 7fa9e9d9b640 1 -- 172.21.15.12:0/286183124 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9e4007af0 con 0x7fa9ec1353f0 2024-09-11T01:42:15.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.274+0000 7fa9e9d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:15.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.274+0000 7fa9e9d9b640 1 -- 172.21.15.12:0/286183124 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e4031400 con 0x7fa9ec1353f0 2024-09-11T01:42:15.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.274+0000 7fa9f0a16640 5 monclient: authenticate success, global_id 134188 2024-09-11T01:42:15.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.274+0000 7fa9f0a16640 10 monclient: get_monmap_and_config success 2024-09-11T01:42:15.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.274+0000 7fa9f0a16640 4 set_mon_vals no callback set 2024-09-11T01:42:15.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.274+0000 7fa9e9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:15.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.274+0000 7fa9e9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:42:15.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.274+0000 7fa9e9d9b640 10 monclient: dump: 2024-09-11T01:42:15.280 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:15.280 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:15.280 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:15.280 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:15.280 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:15.280 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:15.280 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:15.280 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:15.281 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:15.281 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:15.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.274+0000 7fa9f0a16640 10 monclient: shutdown 2024-09-11T01:42:15.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.274+0000 7fa9f0a16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:15.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.274+0000 7fa9f0a16640 1 -- 172.21.15.12:0/286183124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9ec1353f0 msgr2=0x7fa9ec134160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:15.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.274+0000 7fa9f0a16640 1 --2- 172.21.15.12:0/286183124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9ec1353f0 0x7fa9ec134160 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4009e10 tx=0x7fa9e402a5b0 comp rx=0 tx=0).stop 2024-09-11T01:42:15.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.274+0000 7fa9f0a16640 1 -- 172.21.15.12:0/286183124 shutdown_connections 2024-09-11T01:42:15.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.274+0000 7fa9f0a16640 1 --2- 172.21.15.12:0/286183124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9ec1353f0 0x7fa9ec134160 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:15.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.274+0000 7fa9f0a16640 1 --2- 172.21.15.12:0/286183124 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9ec134ad0 0x7fa9ec134eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:15.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.274+0000 7fa9f0a16640 1 -- 172.21.15.12:0/286183124 >> 172.21.15.12:0/286183124 conn(0x7fa9ec0a2f70 msgr2=0x7fa9ec0a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:15.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.274+0000 7fa9f0a16640 1 -- 172.21.15.12:0/286183124 shutdown_connections 2024-09-11T01:42:15.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.274+0000 7fa9f0a16640 1 -- 172.21.15.12:0/286183124 wait complete. 2024-09-11T01:42:15.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.275+0000 7fa9f0a16640 10 monclient: build_initial_monmap 2024-09-11T01:42:15.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.275+0000 7fa9f0a16640 10 monclient: monmap: 2024-09-11T01:42:15.282 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:15.282 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:42:15.282 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:42:15.282 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:42:15.282 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:15.282 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:15.282 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:42:15.282 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:42:15.282 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:42:15.282 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:15.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.275+0000 7fa9f0a16640 1 Processor -- start 2024-09-11T01:42:15.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.275+0000 7fa9f0a16640 1 -- start start 2024-09-11T01:42:15.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.275+0000 7fa9f0a16640 10 monclient: init 2024-09-11T01:42:15.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9f0a16640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:15.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9f0a16640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:42:15.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9f0a16640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9ec0a2200 0x7fa9ec133340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:15.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9f0a16640 10 monclient(hunting): picked mon.noname-b con 0x7fa9ec0a2200 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:42:15.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9f0a16640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9ec134ad0 0x7fa9ec133880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:15.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9f0a16640 10 monclient(hunting): picked mon.noname-c con 0x7fa9ec134ad0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:15.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9f0a16640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ec1353f0 0x7fa9ec133dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:15.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9f0a16640 10 monclient(hunting): picked mon.noname-a con 0x7fa9ec1353f0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:42:15.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9f0a16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:15.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9f0a16640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ec134820 con 0x7fa9ec1353f0 2024-09-11T01:42:15.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9f0a16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:15.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9f0a16640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ec1346a0 con 0x7fa9ec0a2200 2024-09-11T01:42:15.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9f0a16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:15.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9f0a16640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa9ec09ffd0 con 0x7fa9ec134ad0 2024-09-11T01:42:15.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9f0a16640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:15.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9f0a16640 10 monclient(hunting): authenticate will time out at 1541.719297s 2024-09-11T01:42:15.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9e3fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9ec134ad0 0x7fa9ec133880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:15.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9ead9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ec1353f0 0x7fa9ec133dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:15.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9e3fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9ec134ad0 0x7fa9ec133880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:43100/0 (socket says 172.21.15.12:43100) 2024-09-11T01:42:15.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9e3fff640 1 -- 172.21.15.12:0/226753877 learned_addr learned my addr 172.21.15.12:0/226753877 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:15.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9e3fff640 10 monclient(hunting): get_auth_request con 0x7fa9ec134ad0 auth_method 0 2024-09-11T01:42:15.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:15.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:15.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:15.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9ead9d640 10 monclient(hunting): get_auth_request con 0x7fa9ec1353f0 auth_method 0 2024-09-11T01:42:15.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:15.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9ead9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:15.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9ead9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:15.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9ea59c640 1 --2- 172.21.15.12:0/226753877 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9ec0a2200 0x7fa9ec133340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:15.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9e3fff640 1 -- 172.21.15.12:0/226753877 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9ec134ad0 msgr2=0x7fa9ec133880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:42:15.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9e3fff640 1 -- 172.21.15.12:0/226753877 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9ec134ad0 msgr2=0x7fa9ec133880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:42:15.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9e3fff640 1 --2- 172.21.15.12:0/226753877 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9ec134ad0 0x7fa9ec133880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:42:15.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9ead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:15.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9ead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:15.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9ead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:15.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9e3fff640 1 --2- 172.21.15.12:0/226753877 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9ec134ad0 0x7fa9ec133880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:15.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9ea59c640 10 monclient(hunting): get_auth_request con 0x7fa9ec0a2200 auth_method 0 2024-09-11T01:42:15.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:15.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9ea59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:15.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9ea59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:15.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9ea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:15.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9ea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:15.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9ea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:15.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9ead9d640 10 monclient(hunting): handle_auth_done global_id 94557 payload 824 2024-09-11T01:42:15.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9ead9d640 1 -- 172.21.15.12:0/226753877 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9ec134ad0 msgr2=0x7fa9ec133880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:42:15.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9ead9d640 1 --2- 172.21.15.12:0/226753877 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9ec134ad0 0x7fa9ec133880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:15.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9ead9d640 1 -- 172.21.15.12:0/226753877 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9ec0a2200 msgr2=0x7fa9ec133340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:15.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9ead9d640 1 --2- 172.21.15.12:0/226753877 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9ec0a2200 0x7fa9ec133340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:15.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9ead9d640 10 monclient: _finish_hunting 0 2024-09-11T01:42:15.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9ead9d640 1 monclient: found mon.noname-a 2024-09-11T01:42:15.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9ead9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:15.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9ead9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:42:15.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.276+0000 7fa9ead9d640 1 -- 172.21.15.12:0/226753877 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9ec09d480 con 0x7fa9ec1353f0 2024-09-11T01:42:15.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.279+0000 7fa9ead9d640 1 -- 172.21.15.12:0/226753877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ec1353f0 msgr2=0x7fa9ec133dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:15.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.279+0000 7fa9ead9d640 1 --2- 172.21.15.12:0/226753877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ec1353f0 0x7fa9ec133dc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4031000 tx=0x7fa9e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:15.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.279+0000 7fa9e2ffd640 1 -- 172.21.15.12:0/226753877 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e4040070 con 0x7fa9ec1353f0 2024-09-11T01:42:15.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.279+0000 7fa9e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:15.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.279+0000 7fa9e2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:42:15.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.279+0000 7fa9e2ffd640 10 monclient: dump: 2024-09-11T01:42:15.293 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:15.293 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:15.293 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:15.293 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:15.293 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:15.293 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:15.293 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:15.293 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:15.294 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:15.294 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:15.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.279+0000 7fa9e2ffd640 10 monclient: _finish_auth 0 2024-09-11T01:42:15.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.279+0000 7fa9e2ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:42:15.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.279+0000 7fa9e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:15.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.279+0000 7fa9e2ffd640 1 -- 172.21.15.12:0/226753877 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9e4045680 con 0x7fa9ec1353f0 2024-09-11T01:42:15.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.279+0000 7fa9f0a16640 5 monclient: authenticate success, global_id 94557 2024-09-11T01:42:15.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.279+0000 7fa9f0a16640 10 monclient: _renew_subs 2024-09-11T01:42:15.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.279+0000 7fa9f0a16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:15.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.279+0000 7fa9f0a16640 1 -- 172.21.15.12:0/226753877 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9ec136510 con 0x7fa9ec1353f0 2024-09-11T01:42:15.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.279+0000 7fa9e2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:15.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.279+0000 7fa9e2ffd640 1 -- 172.21.15.12:0/226753877 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e40314c0 con 0x7fa9ec1353f0 2024-09-11T01:42:15.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.279+0000 7fa9e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:15.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.279+0000 7fa9e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:15.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.279+0000 7fa9e2ffd640 10 monclient: dump: 2024-09-11T01:42:15.295 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:15.295 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:15.295 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:15.295 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:15.295 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:15.295 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:15.295 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:15.295 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:15.295 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:15.295 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:15.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.279+0000 7fa9e959a640 4 set_mon_vals no callback set 2024-09-11T01:42:15.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.279+0000 7fa9f0a16640 10 monclient: _renew_subs 2024-09-11T01:42:15.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.279+0000 7fa9f0a16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:15.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.279+0000 7fa9f0a16640 1 -- 172.21.15.12:0/226753877 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9ec073ea0 con 0x7fa9ec1353f0 2024-09-11T01:42:15.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.280+0000 7fa9f0a16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:15.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.280+0000 7fa9f0a16640 1 --2- 172.21.15.12:0/226753877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ec12a9c0 0x7fa9ec12ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:15.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.280+0000 7fa9f0a16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:15.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.280+0000 7fa9f0a16640 1 -- 172.21.15.12:0/226753877 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ec0a4190 con 0x7fa9ec12a9c0 2024-09-11T01:42:15.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.280+0000 7fa9e2ffd640 1 -- 172.21.15.12:0/226753877 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fa9e404d030 con 0x7fa9ec1353f0 2024-09-11T01:42:15.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.280+0000 7fa9ea59c640 1 --2- 172.21.15.12:0/226753877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ec12a9c0 0x7fa9ec12ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:15.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.280+0000 7fa9ea59c640 10 monclient: get_auth_request con 0x7fa9ec12a9c0 auth_method 0 2024-09-11T01:42:15.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.280+0000 7fa9ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:15.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.280+0000 7fa9ea59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:15.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.280+0000 7fa9ea59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:15.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.280+0000 7fa9ea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:15.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.280+0000 7fa9ea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:15.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.280+0000 7fa9ea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:15.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.280+0000 7fa9ea59c640 10 monclient(hunting): handle_auth_done global_id 94560 payload 824 2024-09-11T01:42:15.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.280+0000 7fa9e2ffd640 1 --2- 172.21.15.12:0/226753877 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa9cc03e210 0x7fa9cc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:15.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.280+0000 7fa9ea59c640 1 -- 172.21.15.12:0/226753877 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa9ec0a46e0 con 0x7fa9ec12a9c0 2024-09-11T01:42:15.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.280+0000 7fa9e2ffd640 1 -- 172.21.15.12:0/226753877 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(437..437 src has 285..437) ==== 4815+0+0 (secure 0 0 0) 0x7fa9e40746a0 con 0x7fa9ec1353f0 2024-09-11T01:42:15.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.280+0000 7fa9e3fff640 1 --2- 172.21.15.12:0/226753877 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa9cc03e210 0x7fa9cc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:15.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.280+0000 7fa9e3fff640 10 monclient: get_auth_request con 0x7fa9cc03e210 auth_method 0 2024-09-11T01:42:15.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.282+0000 7fa9ea59c640 1 -- 172.21.15.12:0/226753877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ec12a9c0 msgr2=0x7fa9ec12ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:15.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.282+0000 7fa9ea59c640 1 --2- 172.21.15.12:0/226753877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ec12a9c0 0x7fa9ec12ada0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc00a010 tx=0x7fa9dc011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:15.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.282+0000 7fa9e2ffd640 1 -- 172.21.15.12:0/226753877 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc019070 con 0x7fa9ec12a9c0 2024-09-11T01:42:15.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.282+0000 7fa9e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:15.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.282+0000 7fa9e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:15.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.282+0000 7fa9e2ffd640 10 monclient: dump: 2024-09-11T01:42:15.298 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:15.298 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:15.298 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:15.298 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:15.298 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:15.298 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:15.298 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:15.298 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:15.298 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:15.298 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:15.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.283+0000 7fa9e2ffd640 1 -- 172.21.15.12:0/226753877 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa9dc00fcc0 con 0x7fa9ec12a9c0 2024-09-11T01:42:15.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.283+0000 7fa9e2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:15.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.283+0000 7fa9e2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:42:15.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.283+0000 7fa9e2ffd640 1 -- 172.21.15.12:0/226753877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ec12a9c0 msgr2=0x7fa9ec12ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:15.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.283+0000 7fa9e2ffd640 1 --2- 172.21.15.12:0/226753877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ec12a9c0 0x7fa9ec12ada0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc00a010 tx=0x7fa9dc011a60 comp rx=0 tx=0).stop 2024-09-11T01:42:15.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.283+0000 7fa9e2ffd640 1 -- 172.21.15.12:0/226753877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ec12a9c0 msgr2=0x7fa9ec12ada0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:42:15.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.283+0000 7fa9e2ffd640 1 --2- 172.21.15.12:0/226753877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ec12a9c0 0x7fa9ec12ada0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:15.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.283+0000 7fa9e3fff640 1 --2- 172.21.15.12:0/226753877 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa9cc03e210 0x7fa9cc0406d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec134410 tx=0x7fa9d4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:15.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.289+0000 7fa9f0a16640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:42:15.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.289+0000 7fa9f0a16640 1 --2- 172.21.15.12:0/226753877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ec12b2e0 0x7fa9ec06f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:15.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.289+0000 7fa9f0a16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:15.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.289+0000 7fa9f0a16640 1 -- 172.21.15.12:0/226753877 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ec077150 con 0x7fa9ec12b2e0 2024-09-11T01:42:15.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.289+0000 7fa9ea59c640 1 --2- 172.21.15.12:0/226753877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ec12b2e0 0x7fa9ec06f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:15.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.289+0000 7fa9ea59c640 10 monclient: get_auth_request con 0x7fa9ec12b2e0 auth_method 0 2024-09-11T01:42:15.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.289+0000 7fa9ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:15.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.289+0000 7fa9ea59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:15.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.289+0000 7fa9ea59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:15.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.289+0000 7fa9ea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:15.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.289+0000 7fa9ea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:15.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.289+0000 7fa9ea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:15.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.289+0000 7fa9ea59c640 10 monclient(hunting): handle_auth_done global_id 94563 payload 824 2024-09-11T01:42:15.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.289+0000 7fa9ea59c640 1 -- 172.21.15.12:0/226753877 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa9ec134eb0 con 0x7fa9ec12b2e0 2024-09-11T01:42:15.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.291+0000 7fa9ea59c640 1 -- 172.21.15.12:0/226753877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ec12b2e0 msgr2=0x7fa9ec06f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:15.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.291+0000 7fa9ea59c640 1 --2- 172.21.15.12:0/226753877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ec12b2e0 0x7fa9ec06f690 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc01b000 tx=0x7fa9dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:15.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.291+0000 7fa9e2ffd640 1 -- 172.21.15.12:0/226753877 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc00f040 con 0x7fa9ec12b2e0 2024-09-11T01:42:15.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.291+0000 7fa9e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:15.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.291+0000 7fa9e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:15.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.291+0000 7fa9e2ffd640 10 monclient: dump: 2024-09-11T01:42:15.302 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:15.302 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:15.302 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:15.302 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:15.302 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:15.303 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:15.303 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:15.303 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:15.303 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:15.303 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:15.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.292+0000 7fa9e2ffd640 1 -- 172.21.15.12:0/226753877 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7fa9dc0143e0 con 0x7fa9ec12b2e0 2024-09-11T01:42:15.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.292+0000 7fa9e2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:42:15.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.292+0000 7fa9e2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:42:15.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.292+0000 7fa9e2ffd640 1 -- 172.21.15.12:0/226753877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ec12b2e0 msgr2=0x7fa9ec06f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:15.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.292+0000 7fa9e2ffd640 1 --2- 172.21.15.12:0/226753877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ec12b2e0 0x7fa9ec06f690 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc01b000 tx=0x7fa9dc0046c0 comp rx=0 tx=0).stop 2024-09-11T01:42:15.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.292+0000 7fa9e2ffd640 1 -- 172.21.15.12:0/226753877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ec12b2e0 msgr2=0x7fa9ec06f690 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:42:15.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.292+0000 7fa9e2ffd640 1 --2- 172.21.15.12:0/226753877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ec12b2e0 0x7fa9ec06f690 secure :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc01b000 tx=0x7fa9dc0046c0 comp rx=0 tx=0).stop 2024-09-11T01:42:15.304 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:42:15.304 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:42:15.304 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:42:15.304 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:42:15.304 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 122, 2024-09-11T01:42:15.304 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 281387, 2024-09-11T01:42:15.304 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:42:15.304 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:42:15.304 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:42:15.304 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:42:15.304 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:15.305 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:42:15.305 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:42:15.305 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:42:15.305 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:42:15.305 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:42:15.305 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:42:15.305 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:42:15.305 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:42:15.305 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:42:15.305 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:42:15.305 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:42:15.305 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:42:15.305 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:42:15.305 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:42:15.306 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:42:15.306 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:15.306 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:42:15.306 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:42:15.306 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:42:15.306 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:42:15.306 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:42:15.306 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:42:15.306 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:42:15.306 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:42:15.306 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:42:15.306 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:42:15.306 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:42:15.306 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:42:15.306 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:42:15.307 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:15.307 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:15.307 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:42:15.307 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:42:15.307 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:42:15.307 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:42:15.307 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:42:15.307 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:42:15.307 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:42:15.307 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:42:15.307 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:42:15.307 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:42:15.307 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:42:15.307 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:42:15.308 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:42:15.308 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:42:15.308 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:42:15.308 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:42:15.308 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:42:15.308 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:42:15.308 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:42:15.308 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:42:15.308 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:42:15.308 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:42:15.308 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:42:15.308 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:42:15.308 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:42:15.308 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:42:15.309 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:42:15.309 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:42:15.309 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:15.309 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:42:15.309 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:15.309 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:42:15.309 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:15.309 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:42:15.309 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:42:15.309 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:42:15.309 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:42:15.309 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:15.309 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:42:15.310 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:42:15.310 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:15.310 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:15.310 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:15.310 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:42:15.310 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:42:15.310 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:15.310 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:15.310 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:15.310 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:15.310 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:42:15.310 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:42:15.310 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:42:15.311 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:42:15.311 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:42:15.311 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:15.311 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:15.311 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:42:15.311 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:42:15.311 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:42:15.311 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:42:15.311 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:15.312 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:42:15.312 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:42:15.312 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:15.312 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:15.312 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:15.312 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:42:15.312 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:42:15.312 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:15.312 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:15.312 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:15.312 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:15.312 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:42:15.312 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:42:15.313 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:42:15.313 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:42:15.313 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:42:15.313 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:15.313 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:15.313 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:42:15.313 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:42:15.313 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:42:15.313 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:42:15.313 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:15.313 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:42:15.313 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:42:15.314 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:15.314 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:15.314 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:15.314 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:42:15.314 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:42:15.314 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:15.314 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:15.314 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:15.314 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:15.314 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:42:15.314 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:42:15.314 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:42:15.314 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:42:15.314 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:42:15.314 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:15.315 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:15.315 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:15.315 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:42:15.315 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:42:15.315 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:15.315 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:42:15.315 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:42:15.315 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:42:15.315 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:15.315 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:15.315 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:42:15.315 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:15.315 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:42:15.315 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:42:15.315 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:42:15.315 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:15.316 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:15.316 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:42:15.316 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:15.316 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:42:15.316 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:42:15.316 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:42:15.316 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:15.316 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:15.316 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:15.316 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:42:15.316 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:42:15.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.292+0000 7fa9f0a16640 1 -- 172.21.15.12:0/226753877 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa9cc03e210 msgr2=0x7fa9cc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:15.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.292+0000 7fa9f0a16640 1 --2- 172.21.15.12:0/226753877 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa9cc03e210 0x7fa9cc0406d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec134410 tx=0x7fa9d4009040 comp rx=0 tx=0).stop 2024-09-11T01:42:15.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.292+0000 7fa9f0a16640 10 monclient: shutdown 2024-09-11T01:42:15.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.292+0000 7fa9f0a16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:15.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.292+0000 7fa9f0a16640 1 -- 172.21.15.12:0/226753877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ec1353f0 msgr2=0x7fa9ec133dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:15.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.292+0000 7fa9f0a16640 1 --2- 172.21.15.12:0/226753877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9ec1353f0 0x7fa9ec133dc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4031000 tx=0x7fa9e4004a40 comp rx=0 tx=0).stop 2024-09-11T01:42:15.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.292+0000 7fa9ead9d640 1 -- 172.21.15.12:0/226753877 reap_dead start 2024-09-11T01:42:15.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.292+0000 7fa9ead9d640 1 -- 172.21.15.12:0/226753877 reap_dead start 2024-09-11T01:42:15.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.292+0000 7fa9f0a16640 1 -- 172.21.15.12:0/226753877 shutdown_connections 2024-09-11T01:42:15.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.292+0000 7fa9f0a16640 1 -- 172.21.15.12:0/226753877 >> 172.21.15.12:0/226753877 conn(0x7fa9ec0a2f70 msgr2=0x7fa9ec12dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:15.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.292+0000 7fa9f0a16640 1 -- 172.21.15.12:0/226753877 shutdown_connections 2024-09-11T01:42:15.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.292+0000 7fa9f0a16640 1 -- 172.21.15.12:0/226753877 wait complete. 2024-09-11T01:42:15.319 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:42:15.319 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:42:15.319 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:42:15.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.419+0000 7f72d02bd640 10 monclient: get_monmap_and_config 2024-09-11T01:42:15.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.419+0000 7f72d02bd640 10 monclient: build_initial_monmap 2024-09-11T01:42:15.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.419+0000 7f72d02bd640 10 monclient: monmap: 2024-09-11T01:42:15.422 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:15.422 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:15.422 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:42:15.421262+0000 2024-09-11T01:42:15.422 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:42:15.421262+0000 2024-09-11T01:42:15.422 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:15.422 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:15.422 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:42:15.422 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:42:15.423 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:42:15.423 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:42:15.423 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:42:15.423 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:15.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.420+0000 7f72d02bd640 1 Processor -- start 2024-09-11T01:42:15.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.420+0000 7f72d02bd640 1 -- start start 2024-09-11T01:42:15.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.420+0000 7f72d02bd640 10 monclient: init 2024-09-11T01:42:15.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.420+0000 7f72d02bd640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:15.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.420+0000 7f72d02bd640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-11T01:42:15.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.420+0000 7f72d02bd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f72c80a2db0 0x7f72c806f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:15.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.420+0000 7f72d02bd640 10 monclient(hunting): picked mon.noname-a con 0x7f72c80a2db0 addr v2:172.21.15.12:3300/0 2024-09-11T01:42:15.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.420+0000 7f72d02bd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f72c806ded0 addr v1:172.21.15.71:6789/0 2024-09-11T01:42:15.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.420+0000 7f72d02bd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c80a38a0 0x7f72c813f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:15.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.420+0000 7f72d02bd640 10 monclient(hunting): picked mon.noname-b con 0x7f72c80a38a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:15.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.420+0000 7f72d02bd640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72c80a4650 con 0x7f72c806ded0 2024-09-11T01:42:15.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.420+0000 7f72d02bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:15.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.420+0000 7f72d02bd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f72c806f1a0 con 0x7f72c80a2db0 2024-09-11T01:42:15.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.420+0000 7f72d02bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:15.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.420+0000 7f72d02bd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f72c806eec0 con 0x7f72c80a38a0 2024-09-11T01:42:15.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.420+0000 7f72d02bd640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:15.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.420+0000 7f72d02bd640 10 monclient(hunting): authenticate will time out at 1541.863833s 2024-09-11T01:42:15.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.420+0000 7f72ce059640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f72c80a2db0 0x7f72c806f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:15.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.420+0000 7f72ce059640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f72c80a2db0 0x7f72c806f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46572/0 (socket says 172.21.15.12:46572) 2024-09-11T01:42:15.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.421+0000 7f72ce059640 1 -- 172.21.15.12:0/4198520081 learned_addr learned my addr 172.21.15.12:0/4198520081 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:15.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.421+0000 7f72ce059640 10 monclient(hunting): get_auth_request con 0x7f72c80a2db0 auth_method 0 2024-09-11T01:42:15.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.421+0000 7f72ce059640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:15.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.421+0000 7f72ce059640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:15.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.421+0000 7f72ce059640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:15.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.421+0000 7f72ce85a640 1 --2- 172.21.15.12:0/4198520081 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c80a38a0 0x7f72c813f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:15.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.421+0000 7f72ce85a640 10 monclient(hunting): get_auth_request con 0x7f72c80a38a0 auth_method 0 2024-09-11T01:42:15.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.421+0000 7f72ce85a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:15.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.421+0000 7f72ce85a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:15.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.421+0000 7f72ce85a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:15.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.421+0000 7f72ce059640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:15.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.421+0000 7f72ce059640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:15.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.421+0000 7f72ce059640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:15.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.421+0000 7f72ce85a640 1 -- 172.21.15.12:0/4198520081 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c80a38a0 msgr2=0x7f72c813f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:42:15.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.421+0000 7f72ce85a640 1 -- 172.21.15.12:0/4198520081 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c80a38a0 msgr2=0x7f72c813f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:42:15.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.421+0000 7f72ce85a640 1 --2- 172.21.15.12:0/4198520081 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c80a38a0 0x7f72c813f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:42:15.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.421+0000 7f72ce85a640 1 --2- 172.21.15.12:0/4198520081 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c80a38a0 0x7f72c813f3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:15.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.421+0000 7f72ce059640 10 monclient(hunting): handle_auth_done global_id 94566 payload 274 2024-09-11T01:42:15.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.421+0000 7f72ce059640 1 -- 172.21.15.12:0/4198520081 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c80a38a0 msgr2=0x7f72c813f3c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:42:15.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.421+0000 7f72ce059640 1 --2- 172.21.15.12:0/4198520081 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c80a38a0 0x7f72c813f3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:15.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.421+0000 7f72ce059640 1 -- 172.21.15.12:0/4198520081 >> v1:172.21.15.71:6789/0 conn(0x7f72c806ded0 legacy=0x7f72c80a3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:15.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.421+0000 7f72ce059640 10 monclient: _finish_hunting 0 2024-09-11T01:42:15.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.421+0000 7f72ce059640 1 monclient: found mon.noname-a 2024-09-11T01:42:15.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.421+0000 7f72ce059640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:15.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.421+0000 7f72ce059640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:42:15.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.421+0000 7f72ce059640 1 -- 172.21.15.12:0/4198520081 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72c806e2e0 con 0x7f72c80a2db0 2024-09-11T01:42:15.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72ce059640 1 -- 172.21.15.12:0/4198520081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72c80a2db0 msgr2=0x7f72c806f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:15.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72ce059640 1 --2- 172.21.15.12:0/4198520081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72c80a2db0 0x7f72c806f8a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f72b8009870 tx=0x7f72b802a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=829a24742138370a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:15.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72cd057640 1 -- 172.21.15.12:0/4198520081 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b8037070 con 0x7f72c80a2db0 2024-09-11T01:42:15.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72cd057640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:15.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72cd057640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:42:15.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72cd057640 10 monclient: dump: 2024-09-11T01:42:15.435 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:15.435 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:15.435 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:15.435 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:15.435 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:15.435 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:15.435 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:15.435 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:15.435 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:15.435 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:15.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72cd057640 10 monclient: _finish_auth 0 2024-09-11T01:42:15.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72cd057640 10 monclient: _check_auth_tickets 2024-09-11T01:42:15.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72cd057640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:15.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72cd057640 1 -- 172.21.15.12:0/4198520081 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72b8032640 con 0x7f72c80a2db0 2024-09-11T01:42:15.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72cd057640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:15.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72cd057640 1 -- 172.21.15.12:0/4198520081 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b8032a10 con 0x7f72c80a2db0 2024-09-11T01:42:15.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72cd057640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:15.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72cd057640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:15.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72cd057640 10 monclient: dump: 2024-09-11T01:42:15.436 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:15.436 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:15.436 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:15.436 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:15.436 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:15.436 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:15.436 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:15.436 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:15.436 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:15.437 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:15.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72d02bd640 5 monclient: authenticate success, global_id 94566 2024-09-11T01:42:15.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72d02bd640 10 monclient: get_monmap_and_config success 2024-09-11T01:42:15.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72d02bd640 4 set_mon_vals no callback set 2024-09-11T01:42:15.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72d02bd640 10 monclient: shutdown 2024-09-11T01:42:15.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72d02bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:15.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72d02bd640 1 -- 172.21.15.12:0/4198520081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72c80a2db0 msgr2=0x7f72c806f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:15.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72d02bd640 1 --2- 172.21.15.12:0/4198520081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72c80a2db0 0x7f72c806f8a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f72b8009870 tx=0x7f72b802a130 comp rx=0 tx=0).stop 2024-09-11T01:42:15.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72d02bd640 1 -- 172.21.15.12:0/4198520081 shutdown_connections 2024-09-11T01:42:15.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72d02bd640 1 --2- 172.21.15.12:0/4198520081 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c80a38a0 0x7f72c813f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:15.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72d02bd640 1 --2- 172.21.15.12:0/4198520081 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72c80a2db0 0x7f72c806f8a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:15.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72d02bd640 1 -- 172.21.15.12:0/4198520081 >> 172.21.15.12:0/4198520081 conn(0x7f72c80a83b0 msgr2=0x7f72c80a87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:15.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72d02bd640 1 -- 172.21.15.12:0/4198520081 shutdown_connections 2024-09-11T01:42:15.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.424+0000 7f72d02bd640 1 -- 172.21.15.12:0/4198520081 wait complete. 2024-09-11T01:42:15.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.425+0000 7f72d02bd640 10 monclient: build_initial_monmap 2024-09-11T01:42:15.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.425+0000 7f72d02bd640 10 monclient: monmap: 2024-09-11T01:42:15.438 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:15.438 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:42:15.438 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:42:15.438 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:42:15.438 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:15.438 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:15.438 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:42:15.438 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:42:15.438 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:42:15.438 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:15.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.425+0000 7f72d02bd640 1 Processor -- start 2024-09-11T01:42:15.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.425+0000 7f72d02bd640 1 -- start start 2024-09-11T01:42:15.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.425+0000 7f72d02bd640 10 monclient: init 2024-09-11T01:42:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72d02bd640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72d02bd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:42:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72d02bd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72c806ded0 0x7f72c809d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72d02bd640 10 monclient(hunting): picked mon.noname-b con 0x7f72c806ded0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:42:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72d02bd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c80a2db0 0x7f72c809d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72d02bd640 10 monclient(hunting): picked mon.noname-c con 0x7f72c80a2db0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72d02bd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72c80a38a0 0x7f72c809dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72d02bd640 10 monclient(hunting): picked mon.noname-a con 0x7f72c80a38a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:42:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72d02bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72d02bd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f72c813fc90 con 0x7f72c80a38a0 2024-09-11T01:42:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72d02bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72d02bd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f72c813fb10 con 0x7f72c806ded0 2024-09-11T01:42:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72d02bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72d02bd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f72c813fe10 con 0x7f72c80a2db0 2024-09-11T01:42:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72d02bd640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:15.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72d02bd640 10 monclient(hunting): authenticate will time out at 1541.869281s 2024-09-11T01:42:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce85a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72c80a38a0 0x7f72c809dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72cd858640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c80a2db0 0x7f72c809d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce85a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72c80a38a0 0x7f72c809dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46586/0 (socket says 172.21.15.12:46586) 2024-09-11T01:42:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce85a640 1 -- 172.21.15.12:0/980620166 learned_addr learned my addr 172.21.15.12:0/980620166 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce85a640 10 monclient(hunting): get_auth_request con 0x7f72c80a38a0 auth_method 0 2024-09-11T01:42:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce85a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce85a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce85a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72cd858640 10 monclient(hunting): get_auth_request con 0x7f72c80a2db0 auth_method 0 2024-09-11T01:42:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72cd858640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72cd858640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72cd858640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce85a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:15.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce85a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:15.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce059640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72c806ded0 0x7f72c809d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:15.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce85a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:15.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72cd858640 1 -- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c80a2db0 msgr2=0x7f72c809d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:42:15.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce059640 10 monclient(hunting): get_auth_request con 0x7f72c806ded0 auth_method 0 2024-09-11T01:42:15.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce059640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:15.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72cd858640 1 -- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c80a2db0 msgr2=0x7f72c809d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:42:15.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce059640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:15.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce059640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:15.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72cd858640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c80a2db0 0x7f72c809d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:42:15.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72cd858640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c80a2db0 0x7f72c809d730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:15.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce059640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:15.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce059640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:15.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce059640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:15.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce85a640 10 monclient(hunting): handle_auth_done global_id 94569 payload 824 2024-09-11T01:42:15.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce85a640 1 -- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c80a2db0 msgr2=0x7f72c809d730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:42:15.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce85a640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c80a2db0 0x7f72c809d730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:15.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce85a640 1 -- 172.21.15.12:0/980620166 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72c806ded0 msgr2=0x7f72c809d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:15.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce85a640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72c806ded0 0x7f72c809d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:15.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce85a640 10 monclient: _finish_hunting 0 2024-09-11T01:42:15.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce85a640 1 monclient: found mon.noname-a 2024-09-11T01:42:15.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce85a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:15.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce85a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:42:15.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.426+0000 7f72ce85a640 1 -- 172.21.15.12:0/980620166 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72c80a62b0 con 0x7f72c80a38a0 2024-09-11T01:42:15.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.428+0000 7f72ce85a640 1 -- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72c80a38a0 msgr2=0x7f72c809dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:15.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.428+0000 7f72ce85a640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72c80a38a0 0x7f72c809dc70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f72c0007e30 tx=0x7f72c000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:15.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.429+0000 7f72c6ffd640 1 -- 172.21.15.12:0/980620166 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c0014620 con 0x7f72c80a38a0 2024-09-11T01:42:15.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.429+0000 7f72c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:15.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.429+0000 7f72c6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:42:15.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.429+0000 7f72c6ffd640 10 monclient: dump: 2024-09-11T01:42:15.443 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:15.443 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:15.443 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:15.443 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:15.443 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:15.443 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:15.443 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:15.443 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:15.443 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:15.443 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:15.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.429+0000 7f72c6ffd640 10 monclient: _finish_auth 0 2024-09-11T01:42:15.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.429+0000 7f72c6ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:42:15.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.429+0000 7f72c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:15.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.429+0000 7f72c6ffd640 1 -- 172.21.15.12:0/980620166 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72c0019070 con 0x7f72c80a38a0 2024-09-11T01:42:15.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.429+0000 7f72d02bd640 5 monclient: authenticate success, global_id 94569 2024-09-11T01:42:15.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.429+0000 7f72c6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:15.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.429+0000 7f72c6ffd640 1 -- 172.21.15.12:0/980620166 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c0014a20 con 0x7f72c80a38a0 2024-09-11T01:42:15.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.429+0000 7f72d02bd640 10 monclient: _renew_subs 2024-09-11T01:42:15.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.429+0000 7f72d02bd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:15.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.429+0000 7f72d02bd640 1 -- 172.21.15.12:0/980620166 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72c809f5a0 con 0x7f72c80a38a0 2024-09-11T01:42:15.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.429+0000 7f72c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:15.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.429+0000 7f72c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:15.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.429+0000 7f72c6ffd640 10 monclient: dump: 2024-09-11T01:42:15.444 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:15.444 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:15.444 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:15.444 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:15.444 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:15.445 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:15.445 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:15.445 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:15.445 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:15.445 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:15.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.429+0000 7f72cc856640 4 set_mon_vals no callback set 2024-09-11T01:42:15.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.429+0000 7f72d02bd640 10 monclient: _renew_subs 2024-09-11T01:42:15.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.429+0000 7f72d02bd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:15.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.429+0000 7f72d02bd640 1 -- 172.21.15.12:0/980620166 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72c813af60 con 0x7f72c80a38a0 2024-09-11T01:42:15.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.429+0000 7f72d02bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:15.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.429+0000 7f72d02bd640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 0x7f72c81cd430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:15.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.429+0000 7f72d02bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:15.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.429+0000 7f72d02bd640 1 -- 172.21.15.12:0/980620166 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f72c806a060 con 0x7f72c81cd050 2024-09-11T01:42:15.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.430+0000 7f72ce059640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 0x7f72c81cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:15.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.430+0000 7f72c6ffd640 1 -- 172.21.15.12:0/980620166 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f72c0014bc0 con 0x7f72c80a38a0 2024-09-11T01:42:15.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.430+0000 7f72ce059640 10 monclient: get_auth_request con 0x7f72c81cd050 auth_method 0 2024-09-11T01:42:15.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.430+0000 7f72ce059640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:15.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.430+0000 7f72ce059640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:15.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.430+0000 7f72ce059640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:15.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.430+0000 7f72ce059640 1 -- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 msgr2=0x7f72c81cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:42:15.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.430+0000 7f72ce059640 1 -- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 msgr2=0x7f72c81cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:42:15.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.430+0000 7f72ce059640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 0x7f72c81cd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:42:15.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.430+0000 7f72ce059640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 0x7f72c81cd430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:15.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.430+0000 7f72c6ffd640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f72b403e210 0x7f72b40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:15.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.430+0000 7f72c6ffd640 1 -- 172.21.15.12:0/980620166 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(437..437 src has 285..437) ==== 4815+0+0 (secure 0 0 0) 0x7f72c0053590 con 0x7f72c80a38a0 2024-09-11T01:42:15.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.430+0000 7f72cd858640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f72b403e210 0x7f72b40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:15.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.430+0000 7f72cd858640 10 monclient: get_auth_request con 0x7f72b403e210 auth_method 0 2024-09-11T01:42:15.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.433+0000 7f72cd858640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f72b403e210 0x7f72b40406d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f72c8074880 tx=0x7f72bc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:15.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.631+0000 7f72ce059640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 0x7f72c81cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:15.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.631+0000 7f72ce059640 10 monclient: get_auth_request con 0x7f72c81cd050 auth_method 0 2024-09-11T01:42:15.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.631+0000 7f72ce059640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:15.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.631+0000 7f72ce059640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:15.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.631+0000 7f72ce059640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:42:15.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.631+0000 7f72ce059640 1 -- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 msgr2=0x7f72c81cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:42:15.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.631+0000 7f72ce059640 1 -- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 msgr2=0x7f72c81cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:42:15.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.631+0000 7f72ce059640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 0x7f72c81cd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:42:15.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:15.631+0000 7f72ce059640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 0x7f72c81cd430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:42:15.982 INFO:tasks.rados.rados.0.smithi071.stdout:1911: done (0 left) 2024-09-11T01:42:15.982 INFO:tasks.rados.rados.0.smithi071.stdout:1912: read oid 36 snap -1 2024-09-11T01:42:15.982 INFO:tasks.rados.rados.0.smithi071.stdout:1912: expect (ObjNum 687 snap 204 seq_num 687) 2024-09-11T01:42:15.982 INFO:tasks.rados.rados.0.smithi071.stdout:1913: copy_from oid 50 from oid 49 current snap is 213 2024-09-11T01:42:15.982 INFO:tasks.rados.rados.0.smithi071.stdout:1914: write (excl) oid 47 current snap is 213 2024-09-11T01:42:15.983 INFO:tasks.rados.rados.0.smithi071.stdout:1914: seq_num 704 ranges {632128=479772,1746735=600898,3066728=653850} 2024-09-11T01:42:15.988 INFO:tasks.rados.rados.0.smithi071.stdout:1914: writing smithi07134777-47 from 632128 to 1111900 tid 1 2024-09-11T01:42:15.995 INFO:tasks.rados.rados.0.smithi071.stdout:1914: writing smithi07134777-47 from 1746735 to 2347633 tid 2 2024-09-11T01:42:16.003 INFO:tasks.rados.rados.0.smithi071.stdout:1914: writing smithi07134777-47 from 3066728 to 3720578 tid 3 2024-09-11T01:42:16.003 INFO:tasks.rados.rados.0.smithi071.stdout:1915: snap_remove snap 204 2024-09-11T01:42:16.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:16.032+0000 7f72ce059640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 0x7f72c81cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:16.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:16.032+0000 7f72ce059640 10 monclient: get_auth_request con 0x7f72c81cd050 auth_method 0 2024-09-11T01:42:16.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:16.032+0000 7f72ce059640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:16.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:16.032+0000 7f72ce059640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:16.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:16.032+0000 7f72ce059640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:42:16.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:16.032+0000 7f72ce059640 1 -- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 msgr2=0x7f72c81cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:42:16.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:16.032+0000 7f72ce059640 1 -- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 msgr2=0x7f72c81cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:42:16.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:16.032+0000 7f72ce059640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 0x7f72c81cd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:42:16.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:16.032+0000 7f72ce059640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 0x7f72c81cd430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:42:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:16.833+0000 7f72ce059640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 0x7f72c81cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:16.833+0000 7f72ce059640 10 monclient: get_auth_request con 0x7f72c81cd050 auth_method 0 2024-09-11T01:42:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:16.833+0000 7f72ce059640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:16.833+0000 7f72ce059640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:16.833+0000 7f72ce059640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:42:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:16.833+0000 7f72ce059640 1 -- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 msgr2=0x7f72c81cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:42:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:16.833+0000 7f72ce059640 1 -- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 msgr2=0x7f72c81cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:42:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:16.833+0000 7f72ce059640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 0x7f72c81cd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:42:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:16.833+0000 7f72ce059640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 0x7f72c81cd430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:42:16.980 INFO:tasks.rados.rados.0.smithi071.stdout:1915: done (3 left) 2024-09-11T01:42:16.980 INFO:tasks.rados.rados.0.smithi071.stdout:1916: snap_create 2024-09-11T01:42:16.980 INFO:tasks.rados.rados.0.smithi071.stdout:1914: finishing write tid 1 to smithi07134777-47 2024-09-11T01:42:16.980 INFO:tasks.rados.rados.0.smithi071.stdout:1912: expect (ObjNum 687 snap 204 seq_num 687) 2024-09-11T01:42:17.047 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:17.086 INFO:tasks.rados.rados.0.smithi071.stdout:1914: finishing write tid 2 to smithi07134777-47 2024-09-11T01:42:17.086 INFO:tasks.rados.rados.0.smithi071.stdout:1913: finishing copy_from racing read to smithi07134777-50 2024-09-11T01:42:17.086 INFO:tasks.rados.rados.0.smithi071.stdout:1913: finishing copy_from to smithi07134777-50 2024-09-11T01:42:17.086 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 366 (ObjNum 631 snap 190 seq_num 631) dirty exists 2024-09-11T01:42:17.086 INFO:tasks.rados.rados.0.smithi071.stdout:1914: finishing write tid 3 to smithi07134777-47 2024-09-11T01:42:17.087 INFO:tasks.rados.rados.0.smithi071.stdout:1914: finishing write tid 4 to smithi07134777-47 2024-09-11T01:42:17.087 INFO:tasks.rados.rados.0.smithi071.stdout:1914: finishing write tid 5 to smithi07134777-47 2024-09-11T01:42:17.087 INFO:tasks.rados.rados.0.smithi071.stdout:1914: oid 47 updating version 0 to 834 2024-09-11T01:42:17.087 INFO:tasks.rados.rados.0.smithi071.stdout:1914: oid 47 version 834 is already newer than 833 2024-09-11T01:42:17.087 INFO:tasks.rados.rados.0.smithi071.stdout:1914: oid 47 updating version 834 to 835 2024-09-11T01:42:17.087 INFO:tasks.rados.rados.0.smithi071.stdout:1914: oid 47 updating version 835 to 836 2024-09-11T01:42:17.087 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 836 (ObjNum 704 snap 213 seq_num 704) dirty exists 2024-09-11T01:42:17.087 INFO:tasks.rados.rados.0.smithi071.stdout:1914: left oid 47 (ObjNum 704 snap 213 seq_num 704) 2024-09-11T01:42:17.993 INFO:tasks.rados.rados.0.smithi071.stdout:1912: done (3 left) 2024-09-11T01:42:17.993 INFO:tasks.rados.rados.0.smithi071.stdout:1913: done (2 left) 2024-09-11T01:42:17.993 INFO:tasks.rados.rados.0.smithi071.stdout:1914: done (1 left) 2024-09-11T01:42:17.993 INFO:tasks.rados.rados.0.smithi071.stdout:1916: done (0 left) 2024-09-11T01:42:17.993 INFO:tasks.rados.rados.0.smithi071.stdout:1917: write (excl) oid 46 current snap is 214 2024-09-11T01:42:17.993 INFO:tasks.rados.rados.0.smithi071.stdout:1917: seq_num 705 ranges {648935=555021,1805217=460909,2694151=465122,3817442=31493} 2024-09-11T01:42:17.999 INFO:tasks.rados.rados.0.smithi071.stdout:1917: writing smithi07134777-46 from 648935 to 1203956 tid 1 2024-09-11T01:42:18.005 INFO:tasks.rados.rados.0.smithi071.stdout:1917: writing smithi07134777-46 from 1805217 to 2266126 tid 2 2024-09-11T01:42:18.010 INFO:tasks.rados.rados.0.smithi071.stdout:1917: writing smithi07134777-46 from 2694151 to 3159273 tid 3 2024-09-11T01:42:18.015 INFO:tasks.rados.rados.0.smithi071.stdout:1917: writing smithi07134777-46 from 3817442 to 3848935 tid 4 2024-09-11T01:42:18.015 INFO:tasks.rados.rados.0.smithi071.stdout:1918: rollback oid 25 current snap is 214 2024-09-11T01:42:18.015 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 25 to 196 2024-09-11T01:42:18.015 INFO:tasks.rados.rados.0.smithi071.stdout:1917: finishing write tid 1 to smithi07134777-46 2024-09-11T01:42:18.015 INFO:tasks.rados.rados.0.smithi071.stdout:1917: finishing write tid 2 to smithi07134777-46 2024-09-11T01:42:18.015 INFO:tasks.rados.rados.0.smithi071.stdout:1917: finishing write tid 3 to smithi07134777-46 2024-09-11T01:42:18.015 INFO:tasks.rados.rados.0.smithi071.stdout:1919: rollback oid 41 current snap is 214 2024-09-11T01:42:18.015 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 41 to 210 2024-09-11T01:42:18.015 INFO:tasks.rados.rados.0.smithi071.stdout:1920: copy_from oid 19 from oid 24 current snap is 214 2024-09-11T01:42:18.015 INFO:tasks.rados.rados.0.smithi071.stdout:1921: snap_remove snap 207 2024-09-11T01:42:18.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.426+0000 7f72c5ffb640 10 monclient: tick 2024-09-11T01:42:18.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.426+0000 7f72c5ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:42:18.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.426+0000 7f72c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:18.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.426+0000 7f72c5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:18.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.435+0000 7f72ce059640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 0x7f72c81cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:18.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.435+0000 7f72ce059640 10 monclient: get_auth_request con 0x7f72c81cd050 auth_method 0 2024-09-11T01:42:18.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.435+0000 7f72ce059640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:18.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.435+0000 7f72ce059640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:18.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.435+0000 7f72ce059640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:42:18.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.435+0000 7f72ce059640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:18.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.435+0000 7f72ce059640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:18.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.435+0000 7f72ce059640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:18.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.435+0000 7f72ce059640 10 monclient(hunting): handle_auth_done global_id 144101 payload 824 2024-09-11T01:42:18.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.435+0000 7f72ce059640 1 -- 172.21.15.12:0/980620166 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f72c813abc0 con 0x7f72c81cd050 2024-09-11T01:42:18.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.437+0000 7f72ce059640 1 -- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 msgr2=0x7f72c81cd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:18.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.437+0000 7f72ce059640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 0x7f72c81cd430 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f72b8034000 tx=0x7f72b80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:18.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.438+0000 7f72c6ffd640 1 -- 172.21.15.12:0/980620166 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b8037040 con 0x7f72c81cd050 2024-09-11T01:42:18.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.438+0000 7f72c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:18.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.438+0000 7f72c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:42:18.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.438+0000 7f72c6ffd640 10 monclient: dump: 2024-09-11T01:42:18.441 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:18.441 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:18.441 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:18.441 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:18.441 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:18.441 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:18.441 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:18.441 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:18.441 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:18.441 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:18.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.438+0000 7f72c6ffd640 1 -- 172.21.15.12:0/980620166 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f72b8009490 con 0x7f72c81cd050 2024-09-11T01:42:18.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.438+0000 7f72c6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:18.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.438+0000 7f72c6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:42:18.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.438+0000 7f72c6ffd640 1 -- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 msgr2=0x7f72c81cd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:18.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.438+0000 7f72c6ffd640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 0x7f72c81cd430 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f72b8034000 tx=0x7f72b80046c0 comp rx=0 tx=0).stop 2024-09-11T01:42:18.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.438+0000 7f72c6ffd640 1 -- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 msgr2=0x7f72c81cd430 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:42:18.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.438+0000 7f72c6ffd640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd050 0x7f72c81cd430 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f72b8034000 tx=0x7f72b80046c0 comp rx=0 tx=0).stop 2024-09-11T01:42:18.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.445+0000 7f72d02bd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:42:18.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.445+0000 7f72d02bd640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd970 0x7f72c8136300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:18.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.445+0000 7f72d02bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:18.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.445+0000 7f72d02bd640 1 -- 172.21.15.12:0/980620166 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f72c806f8a0 con 0x7f72c81cd970 2024-09-11T01:42:18.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.445+0000 7f72ce059640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd970 0x7f72c8136300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:18.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.445+0000 7f72ce059640 10 monclient: get_auth_request con 0x7f72c81cd970 auth_method 0 2024-09-11T01:42:18.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.445+0000 7f72ce059640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:18.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.445+0000 7f72ce059640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:18.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.445+0000 7f72ce059640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:18.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.445+0000 7f72ce059640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:18.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.445+0000 7f72ce059640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:18.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.446+0000 7f72ce059640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:18.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.446+0000 7f72ce059640 10 monclient(hunting): handle_auth_done global_id 144104 payload 824 2024-09-11T01:42:18.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.446+0000 7f72ce059640 1 -- 172.21.15.12:0/980620166 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f72c813eeb0 con 0x7f72c81cd970 2024-09-11T01:42:18.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.448+0000 7f72ce059640 1 -- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd970 msgr2=0x7f72c8136300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:18.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.448+0000 7f72ce059640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd970 0x7f72c8136300 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f72b8009870 tx=0x7f72b8009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:18.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.448+0000 7f72c6ffd640 1 -- 172.21.15.12:0/980620166 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b804c070 con 0x7f72c81cd970 2024-09-11T01:42:18.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.448+0000 7f72c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:18.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.448+0000 7f72c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:42:18.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.448+0000 7f72c6ffd640 10 monclient: dump: 2024-09-11T01:42:18.452 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:18.452 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:18.452 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:18.452 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:18.452 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:18.452 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:18.452 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:18.452 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:18.452 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:18.452 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:18.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.449+0000 7f72c6ffd640 1 -- 172.21.15.12:0/980620166 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7f72b8004440 con 0x7f72c81cd970 2024-09-11T01:42:18.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.449+0000 7f72c6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:42:18.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.449+0000 7f72c6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:42:18.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.449+0000 7f72c6ffd640 1 -- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd970 msgr2=0x7f72c8136300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:18.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.449+0000 7f72c6ffd640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd970 0x7f72c8136300 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f72b8009870 tx=0x7f72b8009240 comp rx=0 tx=0).stop 2024-09-11T01:42:18.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.449+0000 7f72c6ffd640 1 -- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd970 msgr2=0x7f72c8136300 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:42:18.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.449+0000 7f72c6ffd640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72c81cd970 0x7f72c8136300 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f72b8009870 tx=0x7f72b8009240 comp rx=0 tx=0).stop 2024-09-11T01:42:18.457 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:42:18.458 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:42:18.458 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:42:18.458 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:42:18.458 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 122, 2024-09-11T01:42:18.458 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 10390, 2024-09-11T01:42:18.458 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:42:18.458 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:42:18.458 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:42:18.458 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:42:18.458 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:18.459 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 4, 2024-09-11T01:42:18.459 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:42:18.459 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:42:18.459 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:42:18.459 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:42:18.459 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:42:18.459 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:42:18.460 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:42:18.460 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:42:18.460 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:42:18.460 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:42:18.460 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:42:18.460 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:42:18.460 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:42:18.460 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:42:18.460 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:18.461 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:42:18.461 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:42:18.461 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:42:18.461 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:42:18.461 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:42:18.461 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:42:18.461 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:42:18.461 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:42:18.461 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:42:18.461 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:42:18.461 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:42:18.461 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:42:18.461 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:42:18.462 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:18.462 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:18.462 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:42:18.462 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:42:18.462 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:42:18.462 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:42:18.462 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:42:18.462 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:42:18.462 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:42:18.462 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:42:18.462 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:42:18.462 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:42:18.463 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:42:18.463 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:42:18.463 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:42:18.463 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:42:18.463 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:42:18.463 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:42:18.463 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:42:18.463 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:42:18.463 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:42:18.463 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:42:18.463 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:42:18.463 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:42:18.463 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:42:18.464 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:42:18.464 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:42:18.464 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:42:18.464 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:42:18.464 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:42:18.464 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:18.464 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:42:18.464 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:18.464 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:42:18.464 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:18.464 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:42:18.464 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:42:18.465 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:42:18.465 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:42:18.465 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:18.465 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:42:18.465 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:42:18.465 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:18.465 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:18.465 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:18.465 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:42:18.465 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:42:18.465 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:18.466 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:18.466 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:18.466 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:18.466 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:42:18.466 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:42:18.466 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:42:18.466 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:42:18.466 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:42:18.466 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:18.466 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:18.466 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:42:18.466 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:42:18.466 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:42:18.466 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:42:18.467 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:18.467 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:42:18.467 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:42:18.467 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:18.467 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:18.467 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:18.467 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:42:18.467 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:42:18.467 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:18.467 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:18.467 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:18.467 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:18.467 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:42:18.468 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:42:18.468 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:42:18.468 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:42:18.468 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:42:18.468 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:18.468 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:18.468 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:42:18.468 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:42:18.468 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:42:18.468 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:42:18.468 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:18.468 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:42:18.468 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:42:18.468 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:18.469 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:18.469 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:18.469 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:42:18.469 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:42:18.469 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:18.469 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:18.469 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:18.469 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:18.469 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:42:18.469 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:42:18.469 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:42:18.469 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:42:18.469 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:42:18.470 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:18.470 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:18.470 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:18.470 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:42:18.470 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:42:18.470 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:18.470 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:42:18.470 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:42:18.470 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:42:18.470 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:18.470 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:18.470 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:42:18.470 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:18.470 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:42:18.471 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:42:18.471 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:42:18.471 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:18.471 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:18.471 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:18.471 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:42:18.471 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:42:18.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.449+0000 7f72d02bd640 1 -- 172.21.15.12:0/980620166 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f72b403e210 msgr2=0x7f72b40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.449+0000 7f72d02bd640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f72b403e210 0x7f72b40406d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f72c8074880 tx=0x7f72bc009040 comp rx=0 tx=0).stop 2024-09-11T01:42:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.449+0000 7f72d02bd640 10 monclient: shutdown 2024-09-11T01:42:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.449+0000 7f72d02bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.449+0000 7f72d02bd640 1 -- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72c80a38a0 msgr2=0x7f72c809dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.449+0000 7f72d02bd640 1 --2- 172.21.15.12:0/980620166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72c80a38a0 0x7f72c809dc70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f72c0007e30 tx=0x7f72c000e4b0 comp rx=0 tx=0).stop 2024-09-11T01:42:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.449+0000 7f72ce85a640 1 -- 172.21.15.12:0/980620166 reap_dead start 2024-09-11T01:42:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.449+0000 7f72ce85a640 1 -- 172.21.15.12:0/980620166 reap_dead start 2024-09-11T01:42:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.449+0000 7f72d02bd640 1 -- 172.21.15.12:0/980620166 shutdown_connections 2024-09-11T01:42:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.449+0000 7f72d02bd640 1 -- 172.21.15.12:0/980620166 >> 172.21.15.12:0/980620166 conn(0x7f72c80a83b0 msgr2=0x7f72c80a1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.449+0000 7f72d02bd640 1 -- 172.21.15.12:0/980620166 shutdown_connections 2024-09-11T01:42:18.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.449+0000 7f72d02bd640 1 -- 172.21.15.12:0/980620166 wait complete. 2024-09-11T01:42:18.474 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:42:18.474 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:42:18.474 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:42:18.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.582+0000 7f2654ccd640 10 monclient: get_monmap_and_config 2024-09-11T01:42:18.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.582+0000 7f2654ccd640 10 monclient: build_initial_monmap 2024-09-11T01:42:18.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.582+0000 7f2654ccd640 10 monclient: monmap: 2024-09-11T01:42:18.585 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:18.585 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:18.586 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:42:18.584664+0000 2024-09-11T01:42:18.586 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:42:18.584664+0000 2024-09-11T01:42:18.586 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:18.586 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:18.586 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:42:18.586 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:42:18.586 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:42:18.586 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:42:18.586 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:42:18.586 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:18.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.583+0000 7f2654ccd640 1 Processor -- start 2024-09-11T01:42:18.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.583+0000 7f2654ccd640 1 -- start start 2024-09-11T01:42:18.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.583+0000 7f2654ccd640 10 monclient: init 2024-09-11T01:42:18.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f2654ccd640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:18.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f2654ccd640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-11T01:42:18.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f2654ccd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2650073f50 0x7f2650074330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:18.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f2654ccd640 10 monclient(hunting): picked mon.noname-b con 0x7f2650073f50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:18.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f2654ccd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2650074870 addr v1:172.21.15.71:6789/0 2024-09-11T01:42:18.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f2654ccd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f26500a0810 0x7f26500a2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f2654ccd640 10 monclient(hunting): picked mon.noname-c con 0x7f26500a0810 addr v2:172.21.15.71:3300/0 2024-09-11T01:42:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f2654ccd640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2650070160 con 0x7f2650074870 2024-09-11T01:42:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f2654ccd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f2654ccd640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f26500a0440 con 0x7f26500a0810 2024-09-11T01:42:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f2654ccd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f2654ccd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2650070440 con 0x7f2650073f50 2024-09-11T01:42:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f2654ccd640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f2654ccd640 10 monclient(hunting): authenticate will time out at 1545.027347s 2024-09-11T01:42:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f264e59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2650073f50 0x7f2650074330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f264e59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2650073f50 0x7f2650074330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:40824/0 (socket says 172.21.15.12:40824) 2024-09-11T01:42:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f264e59c640 1 -- 172.21.15.12:0/3784815264 learned_addr learned my addr 172.21.15.12:0/3784815264 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f264e59c640 10 monclient(hunting): get_auth_request con 0x7f2650073f50 auth_method 0 2024-09-11T01:42:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f264e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f264e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f264e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f264e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f264e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f264e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f264ed9d640 1 --2- 172.21.15.12:0/3784815264 >> v2:172.21.15.71:3300/0 conn(0x7f26500a0810 0x7f26500a2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f264e59c640 10 monclient(hunting): handle_auth_done global_id 144107 payload 274 2024-09-11T01:42:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f264e59c640 1 -- 172.21.15.12:0/3784815264 >> v2:172.21.15.71:3300/0 conn(0x7f26500a0810 msgr2=0x7f26500a2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f264e59c640 1 --2- 172.21.15.12:0/3784815264 >> v2:172.21.15.71:3300/0 conn(0x7f26500a0810 0x7f26500a2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f264e59c640 1 -- 172.21.15.12:0/3784815264 >> v1:172.21.15.71:6789/0 conn(0x7f2650074870 legacy=0x7f26500a25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:18.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f264e59c640 10 monclient: _finish_hunting 0 2024-09-11T01:42:18.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f264e59c640 1 monclient: found mon.noname-b 2024-09-11T01:42:18.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f264e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:18.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f264e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:42:18.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f264e59c640 1 -- 172.21.15.12:0/3784815264 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2650070720 con 0x7f2650073f50 2024-09-11T01:42:18.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f264ed9d640 10 monclient: get_auth_request con 0x7f26500a0810 auth_method 0 2024-09-11T01:42:18.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.584+0000 7f264ed9d640 1 --2- 172.21.15.12:0/3784815264 >> v2:172.21.15.71:3300/0 conn(0x7f26500a0810 0x7f26500a2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:42:18.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.587+0000 7f264e59c640 1 -- 172.21.15.12:0/3784815264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2650073f50 msgr2=0x7f2650074330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:18.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.587+0000 7f264e59c640 1 --2- 172.21.15.12:0/3784815264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2650073f50 0x7f2650074330 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2640009870 tx=0x7f264002a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca702a7b6a2cda57 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:18.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.587+0000 7f264d59a640 1 -- 172.21.15.12:0/3784815264 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2640037070 con 0x7f2650073f50 2024-09-11T01:42:18.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.587+0000 7f264d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:18.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.587+0000 7f264d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:42:18.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.587+0000 7f264d59a640 10 monclient: dump: 2024-09-11T01:42:18.599 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:18.599 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:18.599 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:18.599 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:18.599 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:18.600 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:18.600 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:18.600 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:18.600 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:18.600 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:18.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.587+0000 7f264d59a640 10 monclient: _finish_auth 0 2024-09-11T01:42:18.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.588+0000 7f264d59a640 10 monclient: _check_auth_tickets 2024-09-11T01:42:18.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.588+0000 7f264d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:18.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.588+0000 7f264d59a640 1 -- 172.21.15.12:0/3784815264 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2640032640 con 0x7f2650073f50 2024-09-11T01:42:18.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.588+0000 7f2654ccd640 5 monclient: authenticate success, global_id 144107 2024-09-11T01:42:18.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.588+0000 7f2654ccd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:42:18.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.588+0000 7f264d59a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:18.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.588+0000 7f2654ccd640 10 monclient: get_monmap_and_config success 2024-09-11T01:42:18.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.588+0000 7f264d59a640 1 -- 172.21.15.12:0/3784815264 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2640032a10 con 0x7f2650073f50 2024-09-11T01:42:18.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.588+0000 7f264d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:18.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.588+0000 7f2654ccd640 4 set_mon_vals no callback set 2024-09-11T01:42:18.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.588+0000 7f264d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:42:18.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.588+0000 7f264d59a640 10 monclient: dump: 2024-09-11T01:42:18.601 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:18.601 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:18.601 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:18.601 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:18.601 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:18.602 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:18.602 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:18.602 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:18.602 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:18.602 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:18.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.588+0000 7f2654ccd640 10 monclient: shutdown 2024-09-11T01:42:18.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.588+0000 7f2654ccd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:18.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.588+0000 7f2654ccd640 1 -- 172.21.15.12:0/3784815264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2650073f50 msgr2=0x7f2650074330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:18.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.588+0000 7f2654ccd640 1 --2- 172.21.15.12:0/3784815264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2650073f50 0x7f2650074330 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2640009870 tx=0x7f264002a1c0 comp rx=0 tx=0).stop 2024-09-11T01:42:18.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.588+0000 7f2654ccd640 1 -- 172.21.15.12:0/3784815264 shutdown_connections 2024-09-11T01:42:18.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.588+0000 7f2654ccd640 1 --2- 172.21.15.12:0/3784815264 >> v2:172.21.15.71:3300/0 conn(0x7f26500a0810 0x7f26500a2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:18.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.588+0000 7f2654ccd640 1 --2- 172.21.15.12:0/3784815264 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2650073f50 0x7f2650074330 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:18.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.588+0000 7f2654ccd640 1 -- 172.21.15.12:0/3784815264 >> 172.21.15.12:0/3784815264 conn(0x7f26500a36c0 msgr2=0x7f26500a3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:18.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.588+0000 7f2654ccd640 1 -- 172.21.15.12:0/3784815264 shutdown_connections 2024-09-11T01:42:18.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.588+0000 7f2654ccd640 1 -- 172.21.15.12:0/3784815264 wait complete. 2024-09-11T01:42:18.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.589+0000 7f2654ccd640 10 monclient: build_initial_monmap 2024-09-11T01:42:18.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.589+0000 7f2654ccd640 10 monclient: monmap: 2024-09-11T01:42:18.603 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:18.603 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:42:18.603 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:42:18.603 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:42:18.603 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:18.603 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:18.603 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:42:18.604 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:42:18.604 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:42:18.604 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:18.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.589+0000 7f2654ccd640 1 Processor -- start 2024-09-11T01:42:18.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.589+0000 7f2654ccd640 1 -- start start 2024-09-11T01:42:18.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.589+0000 7f2654ccd640 10 monclient: init 2024-09-11T01:42:18.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f2654ccd640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:18.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f2654ccd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:42:18.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f2654ccd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2650073f50 0x7f26501ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:18.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f2654ccd640 10 monclient(hunting): picked mon.noname-b con 0x7f2650073f50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:42:18.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f2654ccd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2650074870 0x7f265012f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f2654ccd640 10 monclient(hunting): picked mon.noname-c con 0x7f2650074870 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f2654ccd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26500a0810 0x7f26501bd890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f2654ccd640 10 monclient(hunting): picked mon.noname-a con 0x7f26500a0810 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:42:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f2654ccd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f2654ccd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f26500a33a0 con 0x7f26500a0810 2024-09-11T01:42:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f2654ccd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f2654ccd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f26500a3220 con 0x7f2650073f50 2024-09-11T01:42:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f2654ccd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f2654ccd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2650134bc0 con 0x7f2650074870 2024-09-11T01:42:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f2654ccd640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:18.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f2654ccd640 10 monclient(hunting): authenticate will time out at 1545.033273s 2024-09-11T01:42:18.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264dd9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2650074870 0x7f265012f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:18.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264ed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26500a0810 0x7f26501bd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:18.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264dd9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2650074870 0x7f265012f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:40834/0 (socket says 172.21.15.12:40834) 2024-09-11T01:42:18.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264ed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26500a0810 0x7f26501bd890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52294/0 (socket says 172.21.15.12:52294) 2024-09-11T01:42:18.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264dd9b640 1 -- 172.21.15.12:0/3681135915 learned_addr learned my addr 172.21.15.12:0/3681135915 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:18.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264dd9b640 10 monclient(hunting): get_auth_request con 0x7f2650074870 auth_method 0 2024-09-11T01:42:18.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:18.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:18.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:18.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264ed9d640 10 monclient(hunting): get_auth_request con 0x7f26500a0810 auth_method 0 2024-09-11T01:42:18.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:18.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:18.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:18.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:18.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:18.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:18.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:18.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264e59c640 1 --2- 172.21.15.12:0/3681135915 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2650073f50 0x7f26501ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264e59c640 10 monclient(hunting): get_auth_request con 0x7f2650073f50 auth_method 0 2024-09-11T01:42:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264dd9b640 10 monclient(hunting): handle_auth_done global_id 144110 payload 824 2024-09-11T01:42:18.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264dd9b640 1 -- 172.21.15.12:0/3681135915 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2650073f50 msgr2=0x7f26501ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:18.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264dd9b640 1 --2- 172.21.15.12:0/3681135915 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2650073f50 0x7f26501ccea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:18.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264dd9b640 1 -- 172.21.15.12:0/3681135915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26500a0810 msgr2=0x7f26501bd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:18.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264dd9b640 1 --2- 172.21.15.12:0/3681135915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26500a0810 0x7f26501bd890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:18.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264dd9b640 10 monclient: _finish_hunting 0 2024-09-11T01:42:18.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264dd9b640 1 monclient: found mon.noname-c 2024-09-11T01:42:18.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:18.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:42:18.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264dd9b640 1 -- 172.21.15.12:0/3681135915 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26501cddc0 con 0x7f2650074870 2024-09-11T01:42:18.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.590+0000 7f264ed9d640 1 --2- 172.21.15.12:0/3681135915 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26500a0810 0x7f26501bd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:42:18.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.593+0000 7f264dd9b640 1 -- 172.21.15.12:0/3681135915 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2650074870 msgr2=0x7f265012f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:18.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.593+0000 7f264dd9b640 1 --2- 172.21.15.12:0/3681135915 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2650074870 0x7f265012f4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f264400c400 tx=0x7f264400c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:18.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.593+0000 7f263effd640 1 -- 172.21.15.12:0/3681135915 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2644017070 con 0x7f2650074870 2024-09-11T01:42:18.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.593+0000 7f263effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:18.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.593+0000 7f263effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:42:18.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.593+0000 7f263effd640 10 monclient: dump: 2024-09-11T01:42:18.610 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:18.610 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:18.610 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:18.610 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:18.610 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:18.610 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:18.610 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:18.611 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:18.611 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:18.611 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:18.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.593+0000 7f263effd640 10 monclient: _finish_auth 0 2024-09-11T01:42:18.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.593+0000 7f263effd640 10 monclient: _check_auth_tickets 2024-09-11T01:42:18.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.593+0000 7f263effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:18.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.593+0000 7f263effd640 1 -- 172.21.15.12:0/3681135915 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26440037f0 con 0x7f2650074870 2024-09-11T01:42:18.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.593+0000 7f263effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:18.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.593+0000 7f263effd640 1 -- 172.21.15.12:0/3681135915 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2644003be0 con 0x7f2650074870 2024-09-11T01:42:18.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.593+0000 7f263effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:18.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.593+0000 7f263effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:42:18.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.593+0000 7f263effd640 10 monclient: dump: 2024-09-11T01:42:18.612 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:18.612 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:18.612 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:18.612 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:18.612 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:18.612 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:18.612 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:18.612 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:18.612 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:18.612 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:18.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.593+0000 7f2654ccd640 5 monclient: authenticate success, global_id 144110 2024-09-11T01:42:18.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.593+0000 7f2654ccd640 10 monclient: _renew_subs 2024-09-11T01:42:18.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.593+0000 7f2654ccd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:42:18.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.593+0000 7f264cd99640 4 set_mon_vals no callback set 2024-09-11T01:42:18.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.593+0000 7f2654ccd640 1 -- 172.21.15.12:0/3681135915 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26501c1380 con 0x7f2650074870 2024-09-11T01:42:18.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.593+0000 7f2654ccd640 10 monclient: _renew_subs 2024-09-11T01:42:18.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.593+0000 7f2654ccd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:42:18.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.593+0000 7f2654ccd640 1 -- 172.21.15.12:0/3681135915 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f26501c1660 con 0x7f2650074870 2024-09-11T01:42:18.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.594+0000 7f2654ccd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:18.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.594+0000 7f2654ccd640 1 --2- 172.21.15.12:0/3681135915 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f265012fa20 0x7f265002d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:18.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.594+0000 7f2654ccd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:18.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.594+0000 7f2654ccd640 1 -- 172.21.15.12:0/3681135915 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f2650074330 con 0x7f265012fa20 2024-09-11T01:42:18.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.594+0000 7f263effd640 1 -- 172.21.15.12:0/3681135915 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f2644007020 con 0x7f2650074870 2024-09-11T01:42:18.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.594+0000 7f264e59c640 1 --2- 172.21.15.12:0/3681135915 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f265012fa20 0x7f265002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:18.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.594+0000 7f264e59c640 10 monclient: get_auth_request con 0x7f265012fa20 auth_method 0 2024-09-11T01:42:18.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.594+0000 7f264e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:18.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.594+0000 7f264e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:18.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.594+0000 7f264e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:18.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.594+0000 7f263effd640 1 --2- 172.21.15.12:0/3681135915 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f263003e240 0x7f2630040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:18.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.594+0000 7f264e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:18.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.594+0000 7f264e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:18.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.594+0000 7f264e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:18.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.594+0000 7f263effd640 1 -- 172.21.15.12:0/3681135915 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(440..440 src has 285..440) ==== 4787+0+0 (secure 0 0 0) 0x7f26440520d0 con 0x7f2650074870 2024-09-11T01:42:18.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.594+0000 7f264ed9d640 1 --2- 172.21.15.12:0/3681135915 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f263003e240 0x7f2630040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:18.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.594+0000 7f264ed9d640 10 monclient: get_auth_request con 0x7f263003e240 auth_method 0 2024-09-11T01:42:18.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.594+0000 7f264e59c640 10 monclient(hunting): handle_auth_done global_id 134203 payload 824 2024-09-11T01:42:18.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.594+0000 7f264e59c640 1 -- 172.21.15.12:0/3681135915 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2650072970 con 0x7f265012fa20 2024-09-11T01:42:18.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.597+0000 7f264e59c640 1 -- 172.21.15.12:0/3681135915 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f265012fa20 msgr2=0x7f265002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:18.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.597+0000 7f264e59c640 1 --2- 172.21.15.12:0/3681135915 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f265012fa20 0x7f265002d440 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2640034000 tx=0x7f26400046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:18.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.597+0000 7f264ed9d640 1 --2- 172.21.15.12:0/3681135915 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f263003e240 0x7f2630040700 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f26501342a0 tx=0x7f2648007780 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:18.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.597+0000 7f263effd640 1 -- 172.21.15.12:0/3681135915 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2640037040 con 0x7f265012fa20 2024-09-11T01:42:18.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.597+0000 7f263effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:18.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.597+0000 7f263effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:42:18.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.597+0000 7f263effd640 10 monclient: dump: 2024-09-11T01:42:18.618 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:18.618 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:18.618 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:18.619 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:18.619 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:18.619 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:18.619 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:18.619 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:18.619 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:18.619 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:18.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.597+0000 7f263effd640 1 -- 172.21.15.12:0/3681135915 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f264003c820 con 0x7f265012fa20 2024-09-11T01:42:18.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.597+0000 7f263effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:18.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.597+0000 7f263effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:42:18.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.597+0000 7f263effd640 1 -- 172.21.15.12:0/3681135915 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f265012fa20 msgr2=0x7f265002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:18.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.597+0000 7f263effd640 1 --2- 172.21.15.12:0/3681135915 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f265012fa20 0x7f265002d440 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2640034000 tx=0x7f26400046c0 comp rx=0 tx=0).stop 2024-09-11T01:42:18.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.597+0000 7f263effd640 1 -- 172.21.15.12:0/3681135915 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f265012fa20 msgr2=0x7f265002d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:42:18.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.597+0000 7f263effd640 1 --2- 172.21.15.12:0/3681135915 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f265012fa20 0x7f265002d440 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2640034000 tx=0x7f26400046c0 comp rx=0 tx=0).stop 2024-09-11T01:42:18.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.603+0000 7f2654ccd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:42:18.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.603+0000 7f2654ccd640 1 --2- 172.21.15.12:0/3681135915 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f265002d980 0x7f265002dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:18.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.603+0000 7f2654ccd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:18.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.603+0000 7f2654ccd640 1 -- 172.21.15.12:0/3681135915 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f26501c0dc0 con 0x7f265002d980 2024-09-11T01:42:18.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.604+0000 7f264e59c640 1 --2- 172.21.15.12:0/3681135915 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f265002d980 0x7f265002dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:18.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.604+0000 7f264e59c640 10 monclient: get_auth_request con 0x7f265002d980 auth_method 0 2024-09-11T01:42:18.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.604+0000 7f264e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:18.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.604+0000 7f264e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:18.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.604+0000 7f264e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:18.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.604+0000 7f264e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:18.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.604+0000 7f264e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:18.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.604+0000 7f264e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:18.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.604+0000 7f264e59c640 10 monclient(hunting): handle_auth_done global_id 134206 payload 824 2024-09-11T01:42:18.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.604+0000 7f264e59c640 1 -- 172.21.15.12:0/3681135915 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f265012fe50 con 0x7f265002d980 2024-09-11T01:42:18.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.607+0000 7f264e59c640 1 -- 172.21.15.12:0/3681135915 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f265002d980 msgr2=0x7f265002dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:18.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.607+0000 7f264e59c640 1 --2- 172.21.15.12:0/3681135915 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f265002d980 0x7f265002dd60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f26400090b0 tx=0x7f2640008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:18.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.607+0000 7f263effd640 1 -- 172.21.15.12:0/3681135915 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2640052030 con 0x7f265002d980 2024-09-11T01:42:18.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.607+0000 7f263effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:18.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.607+0000 7f263effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:42:18.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.607+0000 7f263effd640 10 monclient: dump: 2024-09-11T01:42:18.622 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:18.622 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:18.622 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:18.622 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:18.622 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:18.622 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:18.622 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:18.622 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:18.622 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:18.623 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:18.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.607+0000 7f263effd640 1 -- 172.21.15.12:0/3681135915 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7f264004c820 con 0x7f265002d980 2024-09-11T01:42:18.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.607+0000 7f263effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:42:18.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.607+0000 7f263effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:42:18.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.607+0000 7f263effd640 1 -- 172.21.15.12:0/3681135915 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f265002d980 msgr2=0x7f265002dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:18.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.607+0000 7f263effd640 1 --2- 172.21.15.12:0/3681135915 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f265002d980 0x7f265002dd60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f26400090b0 tx=0x7f2640008030 comp rx=0 tx=0).stop 2024-09-11T01:42:18.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.607+0000 7f263effd640 1 -- 172.21.15.12:0/3681135915 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f265002d980 msgr2=0x7f265002dd60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:42:18.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.607+0000 7f263effd640 1 --2- 172.21.15.12:0/3681135915 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f265002d980 0x7f265002dd60 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f26400090b0 tx=0x7f2640008030 comp rx=0 tx=0).stop 2024-09-11T01:42:18.623 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:42:18.624 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:42:18.624 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:42:18.624 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:42:18.624 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 122, 2024-09-11T01:42:18.624 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 71569, 2024-09-11T01:42:18.624 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:42:18.624 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:42:18.624 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:42:18.624 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:42:18.624 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:18.624 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 4, 2024-09-11T01:42:18.625 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:42:18.625 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:42:18.625 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:42:18.625 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:42:18.625 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:42:18.625 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:42:18.625 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:42:18.625 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:42:18.625 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:42:18.625 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:42:18.625 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:42:18.625 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:42:18.626 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:42:18.626 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:42:18.626 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:18.626 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:42:18.626 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:42:18.626 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:42:18.626 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:42:18.626 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:42:18.626 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:42:18.626 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:42:18.626 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:42:18.627 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:42:18.627 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:42:18.627 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:42:18.627 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:42:18.627 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:42:18.627 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:18.627 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:18.627 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:42:18.627 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:42:18.627 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:42:18.627 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:42:18.627 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:42:18.628 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:42:18.628 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:42:18.628 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:42:18.628 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:42:18.628 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:42:18.628 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:42:18.628 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:42:18.628 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:42:18.628 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:42:18.628 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:42:18.628 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:42:18.629 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:42:18.629 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:42:18.629 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:42:18.629 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:42:18.629 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:42:18.629 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:42:18.629 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:42:18.629 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:42:18.629 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:42:18.629 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:42:18.629 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:42:18.629 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:42:18.630 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:18.630 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:42:18.630 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:18.630 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:42:18.630 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:18.630 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:42:18.630 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:42:18.630 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:42:18.630 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:42:18.630 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:18.630 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:42:18.630 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:42:18.631 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:18.631 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:18.631 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:18.631 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:42:18.631 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:42:18.631 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:18.631 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:18.631 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:18.631 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:18.631 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:42:18.631 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:42:18.632 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:42:18.632 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:42:18.632 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:42:18.632 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:18.632 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:18.632 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:42:18.632 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:42:18.632 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:42:18.632 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:42:18.632 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:18.632 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:42:18.632 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:42:18.633 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:18.633 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:18.633 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:18.633 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:42:18.633 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:42:18.633 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:18.633 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:18.633 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:18.633 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:18.633 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:42:18.633 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:42:18.633 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:42:18.634 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:42:18.634 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:42:18.634 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:18.634 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:18.634 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:42:18.634 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:42:18.634 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:42:18.634 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:42:18.634 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:18.634 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:42:18.634 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:42:18.635 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:18.635 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:18.635 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:18.635 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:42:18.635 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:42:18.635 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:18.635 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:18.635 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:18.635 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:18.635 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:42:18.636 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:42:18.636 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:42:18.636 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:42:18.636 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:42:18.636 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:18.636 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:18.636 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:18.636 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:42:18.636 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:42:18.636 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:18.636 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:42:18.637 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:42:18.637 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:42:18.637 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:18.637 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:18.637 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:42:18.637 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:18.637 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:42:18.637 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:42:18.637 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:42:18.637 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:18.638 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:18.638 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:42:18.638 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:18.638 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:42:18.638 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:42:18.638 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:42:18.638 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:18.638 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:18.638 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:18.638 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:42:18.638 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:42:18.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.607+0000 7f2654ccd640 1 -- 172.21.15.12:0/3681135915 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f263003e240 msgr2=0x7f2630040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:18.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.607+0000 7f2654ccd640 1 --2- 172.21.15.12:0/3681135915 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f263003e240 0x7f2630040700 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f26501342a0 tx=0x7f2648007780 comp rx=0 tx=0).stop 2024-09-11T01:42:18.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.607+0000 7f2654ccd640 10 monclient: shutdown 2024-09-11T01:42:18.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.607+0000 7f2654ccd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:18.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.607+0000 7f2654ccd640 1 -- 172.21.15.12:0/3681135915 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2650074870 msgr2=0x7f265012f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:18.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.607+0000 7f264ed9d640 1 -- 172.21.15.12:0/3681135915 reap_dead start 2024-09-11T01:42:18.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.607+0000 7f2654ccd640 1 --2- 172.21.15.12:0/3681135915 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2650074870 0x7f265012f4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f264400c400 tx=0x7f264400c8c0 comp rx=0 tx=0).stop 2024-09-11T01:42:18.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.607+0000 7f2654ccd640 1 -- 172.21.15.12:0/3681135915 shutdown_connections 2024-09-11T01:42:18.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.607+0000 7f2654ccd640 1 --2- 172.21.15.12:0/3681135915 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2650074870 0x7f265012f4e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:18.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.607+0000 7f2654ccd640 1 -- 172.21.15.12:0/3681135915 >> 172.21.15.12:0/3681135915 conn(0x7f26500a36c0 msgr2=0x7f26500a4fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:18.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.608+0000 7f2654ccd640 1 -- 172.21.15.12:0/3681135915 shutdown_connections 2024-09-11T01:42:18.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.608+0000 7f2654ccd640 1 -- 172.21.15.12:0/3681135915 wait complete. 2024-09-11T01:42:18.641 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:42:18.641 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:42:18.641 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:42:18.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.739+0000 7fdf2e88e640 10 monclient: get_monmap_and_config 2024-09-11T01:42:18.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.739+0000 7fdf2e88e640 10 monclient: build_initial_monmap 2024-09-11T01:42:18.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.739+0000 7fdf2e88e640 10 monclient: monmap: 2024-09-11T01:42:18.742 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:18.742 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:18.742 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:42:18.741546+0000 2024-09-11T01:42:18.742 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:42:18.741546+0000 2024-09-11T01:42:18.742 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:18.743 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:18.743 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:42:18.743 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:42:18.743 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:42:18.743 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:42:18.743 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:42:18.743 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:18.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.740+0000 7fdf2e88e640 1 Processor -- start 2024-09-11T01:42:18.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.740+0000 7fdf2e88e640 1 -- start start 2024-09-11T01:42:18.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.740+0000 7fdf2e88e640 10 monclient: init 2024-09-11T01:42:18.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2e88e640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:18.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2e88e640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-11T01:42:18.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2e88e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdf280a03a0 addr v1:172.21.15.71:6789/0 2024-09-11T01:42:18.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2e88e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fdf280a2530 0x7fdf2809f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:18.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2e88e640 10 monclient(hunting): picked mon.noname-c con 0x7fdf280a2530 addr v2:172.21.15.71:3300/0 2024-09-11T01:42:18.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2e88e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdf280a13a0 0x7fdf2813baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:18.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2e88e640 10 monclient(hunting): picked mon.noname-b con 0x7fdf280a13a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:18.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2e88e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf2809e510 con 0x7fdf280a03a0 2024-09-11T01:42:18.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2e88e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:18.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2e88e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fdf280a29b0 con 0x7fdf280a2530 2024-09-11T01:42:18.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2e88e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:18.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2e88e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fdf280a0780 con 0x7fdf280a13a0 2024-09-11T01:42:18.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2e88e640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:18.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2e88e640 10 monclient(hunting): authenticate will time out at 1545.184229s 2024-09-11T01:42:18.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2ce2b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdf280a13a0 0x7fdf2813baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:18.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2ce2b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdf280a13a0 0x7fdf2813baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:40838/0 (socket says 172.21.15.12:40838) 2024-09-11T01:42:18.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2ce2b640 1 -- 172.21.15.12:0/2504976043 learned_addr learned my addr 172.21.15.12:0/2504976043 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:18.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2ce2b640 10 monclient(hunting): get_auth_request con 0x7fdf280a13a0 auth_method 0 2024-09-11T01:42:18.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2ce2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:18.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2ce2b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:18.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2ce2b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:18.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2ce2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:18.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2ce2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:18.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2ce2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:18.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf277fe640 1 --2- 172.21.15.12:0/2504976043 >> v2:172.21.15.71:3300/0 conn(0x7fdf280a2530 0x7fdf2809f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:18.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2ce2b640 10 monclient(hunting): handle_auth_done global_id 144113 payload 274 2024-09-11T01:42:18.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2ce2b640 1 -- 172.21.15.12:0/2504976043 >> v2:172.21.15.71:3300/0 conn(0x7fdf280a2530 msgr2=0x7fdf2809f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:18.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2ce2b640 1 --2- 172.21.15.12:0/2504976043 >> v2:172.21.15.71:3300/0 conn(0x7fdf280a2530 0x7fdf2809f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:18.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2ce2b640 1 -- 172.21.15.12:0/2504976043 >> v1:172.21.15.71:6789/0 conn(0x7fdf280a03a0 legacy=0x7fdf280a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:18.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2ce2b640 10 monclient: _finish_hunting 0 2024-09-11T01:42:18.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2ce2b640 1 monclient: found mon.noname-b 2024-09-11T01:42:18.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2ce2b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:18.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2ce2b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:42:18.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf2ce2b640 1 -- 172.21.15.12:0/2504976043 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf2809edb0 con 0x7fdf280a13a0 2024-09-11T01:42:18.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf277fe640 10 monclient: get_auth_request con 0x7fdf280a2530 auth_method 0 2024-09-11T01:42:18.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.741+0000 7fdf277fe640 1 --2- 172.21.15.12:0/2504976043 >> v2:172.21.15.71:3300/0 conn(0x7fdf280a2530 0x7fdf2809f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:42:18.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.744+0000 7fdf2ce2b640 1 -- 172.21.15.12:0/2504976043 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdf280a13a0 msgr2=0x7fdf2813baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:18.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.744+0000 7fdf2ce2b640 1 --2- 172.21.15.12:0/2504976043 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdf280a13a0 0x7fdf2813baa0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdf20009e50 tx=0x7fdf2002a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df48816e25346af7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:18.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.744+0000 7fdf26ffd640 1 -- 172.21.15.12:0/2504976043 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf20036070 con 0x7fdf280a13a0 2024-09-11T01:42:18.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.744+0000 7fdf26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:18.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.744+0000 7fdf26ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:42:18.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.744+0000 7fdf26ffd640 10 monclient: dump: 2024-09-11T01:42:18.754 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:18.754 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:18.754 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:18.754 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:18.755 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:18.755 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:18.755 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:18.755 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:18.755 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:18.755 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:18.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.744+0000 7fdf26ffd640 10 monclient: _finish_auth 0 2024-09-11T01:42:18.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.744+0000 7fdf26ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:42:18.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.744+0000 7fdf26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:18.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.744+0000 7fdf2e88e640 5 monclient: authenticate success, global_id 144113 2024-09-11T01:42:18.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.744+0000 7fdf2e88e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:42:18.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.744+0000 7fdf26ffd640 1 -- 172.21.15.12:0/2504976043 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf20007800 con 0x7fdf280a13a0 2024-09-11T01:42:18.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.744+0000 7fdf26ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:18.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.744+0000 7fdf2e88e640 10 monclient: get_monmap_and_config success 2024-09-11T01:42:18.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.744+0000 7fdf2e88e640 4 set_mon_vals no callback set 2024-09-11T01:42:18.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.744+0000 7fdf26ffd640 1 -- 172.21.15.12:0/2504976043 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf20007bd0 con 0x7fdf280a13a0 2024-09-11T01:42:18.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.744+0000 7fdf26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:18.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.744+0000 7fdf26ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:42:18.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.744+0000 7fdf26ffd640 10 monclient: dump: 2024-09-11T01:42:18.756 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:18.756 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:18.756 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:18.756 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:18.756 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:18.756 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:18.756 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:18.757 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:18.757 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:18.757 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:18.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.745+0000 7fdf2e88e640 10 monclient: shutdown 2024-09-11T01:42:18.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.745+0000 7fdf2e88e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:18.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.745+0000 7fdf2e88e640 1 -- 172.21.15.12:0/2504976043 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdf280a13a0 msgr2=0x7fdf2813baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:18.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.745+0000 7fdf2e88e640 1 --2- 172.21.15.12:0/2504976043 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdf280a13a0 0x7fdf2813baa0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdf20009e50 tx=0x7fdf2002a560 comp rx=0 tx=0).stop 2024-09-11T01:42:18.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.745+0000 7fdf2e88e640 1 -- 172.21.15.12:0/2504976043 shutdown_connections 2024-09-11T01:42:18.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.745+0000 7fdf2e88e640 1 --2- 172.21.15.12:0/2504976043 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdf280a13a0 0x7fdf2813baa0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:18.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.745+0000 7fdf2e88e640 1 --2- 172.21.15.12:0/2504976043 >> v2:172.21.15.71:3300/0 conn(0x7fdf280a2530 0x7fdf2809f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:18.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.745+0000 7fdf2e88e640 1 -- 172.21.15.12:0/2504976043 >> 172.21.15.12:0/2504976043 conn(0x7fdf280a2f70 msgr2=0x7fdf280a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:18.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.745+0000 7fdf2e88e640 1 -- 172.21.15.12:0/2504976043 shutdown_connections 2024-09-11T01:42:18.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.745+0000 7fdf2e88e640 1 -- 172.21.15.12:0/2504976043 wait complete. 2024-09-11T01:42:18.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.745+0000 7fdf2e88e640 10 monclient: build_initial_monmap 2024-09-11T01:42:18.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.745+0000 7fdf2e88e640 10 monclient: monmap: 2024-09-11T01:42:18.758 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:18.758 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:42:18.758 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:42:18.758 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:42:18.758 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:18.758 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:18.758 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:42:18.758 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:42:18.758 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:42:18.758 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:18.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.746+0000 7fdf2e88e640 1 Processor -- start 2024-09-11T01:42:18.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.746+0000 7fdf2e88e640 1 -- start start 2024-09-11T01:42:18.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.746+0000 7fdf2e88e640 10 monclient: init 2024-09-11T01:42:18.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.746+0000 7fdf2e88e640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:18.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.746+0000 7fdf2e88e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:42:18.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.746+0000 7fdf2e88e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdf280a03a0 0x7fdf28125c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:18.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.746+0000 7fdf2e88e640 10 monclient(hunting): picked mon.noname-c con 0x7fdf280a03a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:18.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.746+0000 7fdf2e88e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf280a13a0 0x7fdf281cb500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:18.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.746+0000 7fdf2e88e640 10 monclient(hunting): picked mon.noname-b con 0x7fdf280a13a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:42:18.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.746+0000 7fdf2e88e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf280a2530 0x7fdf28123660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:18.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.746+0000 7fdf2e88e640 10 monclient(hunting): picked mon.noname-a con 0x7fdf280a2530 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:42:18.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.746+0000 7fdf2e88e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:18.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.746+0000 7fdf2e88e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdf2809ca30 con 0x7fdf280a2530 2024-09-11T01:42:18.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.746+0000 7fdf2e88e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:18.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.746+0000 7fdf2e88e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fdf280a1a30 con 0x7fdf280a13a0 2024-09-11T01:42:18.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.746+0000 7fdf2e88e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:18.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.746+0000 7fdf2e88e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fdf2809cbb0 con 0x7fdf280a03a0 2024-09-11T01:42:18.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.746+0000 7fdf2e88e640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:18.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.746+0000 7fdf2e88e640 10 monclient(hunting): authenticate will time out at 1545.189925s 2024-09-11T01:42:18.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.746+0000 7fdf27fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdf280a03a0 0x7fdf28125c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:18.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf27fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdf280a03a0 0x7fdf28125c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:40842/0 (socket says 172.21.15.12:40842) 2024-09-11T01:42:18.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf27fff640 1 -- 172.21.15.12:0/3710367656 learned_addr learned my addr 172.21.15.12:0/3710367656 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:18.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf27fff640 10 monclient(hunting): get_auth_request con 0x7fdf280a03a0 auth_method 0 2024-09-11T01:42:18.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:18.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf27fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:18.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf27fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:18.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf2ce2b640 1 --2- 172.21.15.12:0/3710367656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf280a2530 0x7fdf28123660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:18.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf2ce2b640 10 monclient(hunting): get_auth_request con 0x7fdf280a2530 auth_method 0 2024-09-11T01:42:18.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf2ce2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:18.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf2ce2b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:18.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf2ce2b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:18.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:18.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:18.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:18.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf2ce2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:18.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf2ce2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:18.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf2ce2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:18.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf277fe640 1 --2- 172.21.15.12:0/3710367656 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf280a13a0 0x7fdf281cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:18.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf27fff640 10 monclient(hunting): handle_auth_done global_id 144116 payload 824 2024-09-11T01:42:18.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf27fff640 1 -- 172.21.15.12:0/3710367656 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf280a13a0 msgr2=0x7fdf281cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:18.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf27fff640 1 --2- 172.21.15.12:0/3710367656 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf280a13a0 0x7fdf281cb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:18.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf27fff640 1 -- 172.21.15.12:0/3710367656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf280a2530 msgr2=0x7fdf28123660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:18.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf27fff640 1 --2- 172.21.15.12:0/3710367656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf280a2530 0x7fdf28123660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:18.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf27fff640 10 monclient: _finish_hunting 0 2024-09-11T01:42:18.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf27fff640 1 monclient: found mon.noname-c 2024-09-11T01:42:18.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:18.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf27fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:42:18.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf27fff640 1 -- 172.21.15.12:0/3710367656 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf2809d480 con 0x7fdf280a03a0 2024-09-11T01:42:18.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf277fe640 10 monclient: get_auth_request con 0x7fdf280a13a0 auth_method 0 2024-09-11T01:42:18.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.747+0000 7fdf277fe640 1 --2- 172.21.15.12:0/3710367656 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf280a13a0 0x7fdf281cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:42:18.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.749+0000 7fdf27fff640 1 -- 172.21.15.12:0/3710367656 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdf280a03a0 msgr2=0x7fdf28125c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:18.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.749+0000 7fdf27fff640 1 --2- 172.21.15.12:0/3710367656 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdf280a03a0 0x7fdf28125c40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdf1800c430 tx=0x7fdf1800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:18.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.749+0000 7fdf24ff9640 1 -- 172.21.15.12:0/3710367656 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf180119a0 con 0x7fdf280a03a0 2024-09-11T01:42:18.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.749+0000 7fdf24ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:18.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.749+0000 7fdf24ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:42:18.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.749+0000 7fdf24ff9640 10 monclient: dump: 2024-09-11T01:42:18.763 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:18.763 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:18.763 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:18.763 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:18.763 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:18.763 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:18.763 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:18.763 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:18.763 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:18.763 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:18.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.749+0000 7fdf24ff9640 10 monclient: _finish_auth 0 2024-09-11T01:42:18.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.749+0000 7fdf24ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:42:18.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.749+0000 7fdf24ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:18.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.749+0000 7fdf24ff9640 1 -- 172.21.15.12:0/3710367656 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf18017070 con 0x7fdf280a03a0 2024-09-11T01:42:18.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.749+0000 7fdf2e88e640 5 monclient: authenticate success, global_id 144116 2024-09-11T01:42:18.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.749+0000 7fdf2e88e640 10 monclient: _renew_subs 2024-09-11T01:42:18.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.749+0000 7fdf2e88e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:42:18.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.749+0000 7fdf2e88e640 1 -- 172.21.15.12:0/3710367656 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf28127160 con 0x7fdf280a03a0 2024-09-11T01:42:18.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.749+0000 7fdf24ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:18.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.749+0000 7fdf24ff9640 1 -- 172.21.15.12:0/3710367656 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf18011d80 con 0x7fdf280a03a0 2024-09-11T01:42:18.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.749+0000 7fdf24ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:18.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.749+0000 7fdf24ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:42:18.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.749+0000 7fdf24ff9640 10 monclient: dump: 2024-09-11T01:42:18.765 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:18.765 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:18.765 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:18.765 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:18.765 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:18.765 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:18.765 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:18.765 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:18.765 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:18.765 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:18.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.749+0000 7fdf2e88e640 10 monclient: _renew_subs 2024-09-11T01:42:18.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.749+0000 7fdf2e88e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:42:18.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.749+0000 7fdf2e88e640 1 -- 172.21.15.12:0/3710367656 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf281265d0 con 0x7fdf280a03a0 2024-09-11T01:42:18.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.749+0000 7fdf267fc640 4 set_mon_vals no callback set 2024-09-11T01:42:18.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.750+0000 7fdf2e88e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:18.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.750+0000 7fdf2e88e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:18.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.750+0000 7fdf2e88e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:42:18.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.750+0000 7fdf24ff9640 1 -- 172.21.15.12:0/3710367656 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fdf1801c460 con 0x7fdf280a03a0 2024-09-11T01:42:18.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.750+0000 7fdf2e88e640 1 -- 172.21.15.12:0/3710367656 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf280a37c0 con 0x7fdf280a03a0 2024-09-11T01:42:18.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.751+0000 7fdf24ff9640 1 --2- 172.21.15.12:0/3710367656 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdf1403e210 0x7fdf140406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:18.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.751+0000 7fdf24ff9640 1 -- 172.21.15.12:0/3710367656 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(440..440 src has 285..440) ==== 4787+0+0 (secure 0 0 0) 0x7fdf18052be0 con 0x7fdf280a03a0 2024-09-11T01:42:18.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.751+0000 7fdf277fe640 1 --2- 172.21.15.12:0/3710367656 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdf1403e210 0x7fdf140406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:18.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.751+0000 7fdf277fe640 10 monclient: get_auth_request con 0x7fdf1403e210 auth_method 0 2024-09-11T01:42:18.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.754+0000 7fdf277fe640 1 --2- 172.21.15.12:0/3710367656 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdf1403e210 0x7fdf140406d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c006fd0 tx=0x7fdf1c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:18.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.756+0000 7fdf24ff9640 1 -- 172.21.15.12:0/3710367656 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fdf280a37c0 con 0x7fdf280a03a0 2024-09-11T01:42:18.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.756+0000 7fdf24ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:18.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:18.756+0000 7fdf24ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:42:18.967 INFO:tasks.rados.rados.0.smithi071.stdout:1921: done (4 left) 2024-09-11T01:42:18.967 INFO:tasks.rados.rados.0.smithi071.stdout:1922: delete oid 36 current snap is 214 2024-09-11T01:42:18.967 INFO:tasks.rados.rados.0.smithi071.stdout:1917: finishing write tid 4 to smithi07134777-46 2024-09-11T01:42:18.967 INFO:tasks.rados.rados.0.smithi071.stdout:1917: finishing write tid 5 to smithi07134777-46 2024-09-11T01:42:18.967 INFO:tasks.rados.rados.0.smithi071.stdout:1917: finishing write tid 6 to smithi07134777-46 2024-09-11T01:42:18.967 INFO:tasks.rados.rados.0.smithi071.stdout:1917: oid 46 updating version 0 to 1038 2024-09-11T01:42:18.967 INFO:tasks.rados.rados.0.smithi071.stdout:1917: oid 46 updating version 1038 to 1039 2024-09-11T01:42:18.968 INFO:tasks.rados.rados.0.smithi071.stdout:1917: oid 46 updating version 1039 to 1040 2024-09-11T01:42:18.968 INFO:tasks.rados.rados.0.smithi071.stdout:1917: oid 46 version 1040 is already newer than 1037 2024-09-11T01:42:18.968 INFO:tasks.rados.rados.0.smithi071.stdout:1917: oid 46 updating version 1040 to 1041 2024-09-11T01:42:18.968 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 1041 (ObjNum 705 snap 214 seq_num 705) dirty exists 2024-09-11T01:42:18.968 INFO:tasks.rados.rados.0.smithi071.stdout:1917: left oid 46 (ObjNum 705 snap 214 seq_num 705) 2024-09-11T01:42:18.968 INFO:tasks.rados.rados.0.smithi071.stdout:1919: finishing rollback tid 0 to smithi07134777-41 2024-09-11T01:42:18.968 INFO:tasks.rados.rados.0.smithi071.stdout:1919: finishing rollback tid 1 to smithi07134777-41 2024-09-11T01:42:18.968 INFO:tasks.rados.rados.0.smithi071.stdout:1918: finishing rollback tid 0 to smithi07134777-25 2024-09-11T01:42:18.968 INFO:tasks.rados.rados.0.smithi071.stdout:1918: finishing rollback tid 1 to smithi07134777-25 2024-09-11T01:42:18.968 INFO:tasks.rados.rados.0.smithi071.stdout:1919: finishing rollback tid 2 to smithi07134777-41 2024-09-11T01:42:18.968 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 390 (ObjNum 665 snap 201 seq_num 665) dirty exists 2024-09-11T01:42:18.968 INFO:tasks.rados.rados.0.smithi071.stdout:1918: finishing rollback tid 2 to smithi07134777-25 2024-09-11T01:42:18.968 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 541 (ObjNum 488 snap 132 seq_num 488) dirty exists 2024-09-11T01:42:18.969 INFO:tasks.rados.rados.0.smithi071.stdout:1920: finishing copy_from racing read to smithi07134777-19 2024-09-11T01:42:18.969 INFO:tasks.rados.rados.0.smithi071.stdout:1920: finishing copy_from to smithi07134777-19 2024-09-11T01:42:18.969 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 700 (ObjNum 647 snap 199 seq_num 647) dirty exists 2024-09-11T01:42:18.972 INFO:tasks.rados.rados.0.smithi071.stdout:1917: done (4 left) 2024-09-11T01:42:18.972 INFO:tasks.rados.rados.0.smithi071.stdout:1918: done (3 left) 2024-09-11T01:42:18.972 INFO:tasks.rados.rados.0.smithi071.stdout:1919: done (2 left) 2024-09-11T01:42:18.972 INFO:tasks.rados.rados.0.smithi071.stdout:1920: done (1 left) 2024-09-11T01:42:18.972 INFO:tasks.rados.rados.0.smithi071.stdout:1922: done (0 left) 2024-09-11T01:42:18.972 INFO:tasks.rados.rados.0.smithi071.stdout:1923: copy_from oid 43 from oid 35 current snap is 214 2024-09-11T01:42:18.972 INFO:tasks.rados.rados.0.smithi071.stdout:1924: read oid 20 snap 208 2024-09-11T01:42:18.972 INFO:tasks.rados.rados.0.smithi071.stdout:1924: expect (ObjNum 681 snap 204 seq_num 681) 2024-09-11T01:42:18.973 INFO:tasks.rados.rados.0.smithi071.stdout:1925: snap_create 2024-09-11T01:42:18.985 INFO:tasks.rados.rados.0.smithi071.stdout:1924: expect (ObjNum 681 snap 204 seq_num 681) 2024-09-11T01:42:19.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:19.024+0000 7fdf2e88e640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:42:19.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:19.024+0000 7fdf2e88e640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:42:19.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:19.024+0000 7fdf2e88e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:42:19.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:19.024+0000 7fdf2e88e640 1 -- 172.21.15.12:0/3710367656 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fdf2809d010 con 0x7fdf280a03a0 2024-09-11T01:42:19.028 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.026+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.029 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.026+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=11,kv=12,logm=54} crc {auth=503027249,config=2586702516,health=1105059601,kv=3263697420,logm=2064044687}) 2024-09-11T01:42:19.029 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.026+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=23,config=2,health=12,kv=10,logm=53} crc {auth=1094351371,config=2586702516,health=4163661969,kv=8604169,logm=1717542391}) 2024-09-11T01:42:19.029 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.026+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.029 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.026+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=11,kv=12,logm=54} crc {auth=503027249,config=2586702516,health=1105059601,kv=3263697420,logm=2064044687}) 2024-09-11T01:42:19.029 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.026+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=22,config=2,health=10,kv=12,logm=54} crc {auth=3928792789,config=2586702516,health=2727378433,kv=3263697420,logm=2676802650}) 2024-09-11T01:42:19.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:19.026+0000 7fdf24ff9640 1 -- 172.21.15.12:0/3710367656 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fdf2809d480 con 0x7fdf280a03a0 2024-09-11T01:42:19.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:19.026+0000 7fdf24ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:42:19.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:19.026+0000 7fdf24ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:42:19.030 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.027+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.030 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.027+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3837275502}) 2024-09-11T01:42:19.030 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.027+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2247958815}) 2024-09-11T01:42:19.030 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.027+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.030 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.027+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3837275502}) 2024-09-11T01:42:19.030 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.027+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=739766171}) 2024-09-11T01:42:19.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.028+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.028+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1202717814}) 2024-09-11T01:42:19.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.028+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3629729110}) 2024-09-11T01:42:19.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.028+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.029+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1202717814}) 2024-09-11T01:42:19.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.029+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1280520373}) 2024-09-11T01:42:19.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:19.029+0000 7fdf2e88e640 1 -- 172.21.15.12:0/3710367656 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdf1403e210 msgr2=0x7fdf140406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:19.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:19.029+0000 7fdf2e88e640 1 --2- 172.21.15.12:0/3710367656 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdf1403e210 0x7fdf140406d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c006fd0 tx=0x7fdf1c008040 comp rx=0 tx=0).stop 2024-09-11T01:42:19.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:19.029+0000 7fdf2e88e640 10 monclient: shutdown 2024-09-11T01:42:19.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:19.029+0000 7fdf2e88e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:19.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:19.029+0000 7fdf2e88e640 1 -- 172.21.15.12:0/3710367656 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdf280a03a0 msgr2=0x7fdf28125c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:19.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:19.029+0000 7fdf2e88e640 1 --2- 172.21.15.12:0/3710367656 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdf280a03a0 0x7fdf28125c40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdf1800c430 tx=0x7fdf1800c8f0 comp rx=0 tx=0).stop 2024-09-11T01:42:19.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:19.029+0000 7fdf2e88e640 1 -- 172.21.15.12:0/3710367656 shutdown_connections 2024-09-11T01:42:19.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:19.029+0000 7fdf2e88e640 1 --2- 172.21.15.12:0/3710367656 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdf1403e210 0x7fdf140406d0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:19.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:19.029+0000 7fdf2e88e640 1 --2- 172.21.15.12:0/3710367656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf280a2530 0x7fdf28123660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:19.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:19.029+0000 7fdf2e88e640 1 --2- 172.21.15.12:0/3710367656 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf280a13a0 0x7fdf281cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:19.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:19.029+0000 7fdf2e88e640 1 --2- 172.21.15.12:0/3710367656 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdf280a03a0 0x7fdf28125c40 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:19.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:19.029+0000 7fdf2e88e640 1 -- 172.21.15.12:0/3710367656 >> 172.21.15.12:0/3710367656 conn(0x7fdf280a2f70 msgr2=0x7fdf280a58b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:19.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:19.029+0000 7fdf2e88e640 1 -- 172.21.15.12:0/3710367656 shutdown_connections 2024-09-11T01:42:19.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:19.029+0000 7fdf2e88e640 1 -- 172.21.15.12:0/3710367656 wait complete. 2024-09-11T01:42:19.035 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.030+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.035 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.030+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2835937180}) 2024-09-11T01:42:19.035 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.030+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3140879016}) 2024-09-11T01:42:19.035 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.030+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.035 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.030+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2835937180}) 2024-09-11T01:42:19.035 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.030+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3308016431}) 2024-09-11T01:42:19.035 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.030+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.035 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.030+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3129092968}) 2024-09-11T01:42:19.035 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.031+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=30565493}) 2024-09-11T01:42:19.036 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.031+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.036 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.031+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3129092968}) 2024-09-11T01:42:19.036 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.031+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1916702361}) 2024-09-11T01:42:19.036 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.031+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.036 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.031+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1691142439}) 2024-09-11T01:42:19.036 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.031+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3955630137}) 2024-09-11T01:42:19.036 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.031+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.036 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.031+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1691142439}) 2024-09-11T01:42:19.036 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.031+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2839365528}) 2024-09-11T01:42:19.036 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.032+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.036 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.032+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2727970855}) 2024-09-11T01:42:19.036 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.032+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=914327630}) 2024-09-11T01:42:19.036 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.032+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.037 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.032+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2727970855}) 2024-09-11T01:42:19.037 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.032+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1163930352}) 2024-09-11T01:42:19.038 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.033+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.039 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.033+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=28364951}) 2024-09-11T01:42:19.039 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.033+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3525955149}) 2024-09-11T01:42:19.039 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.033+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.039 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.033+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=28364951}) 2024-09-11T01:42:19.039 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.033+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=374204803}) 2024-09-11T01:42:19.039 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.034+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.039 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.034+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2172479115}) 2024-09-11T01:42:19.039 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.034+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=689692831}) 2024-09-11T01:42:19.039 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.034+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.039 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.034+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2172479115}) 2024-09-11T01:42:19.039 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.034+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3188260864}) 2024-09-11T01:42:19.039 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.035+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.039 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.035+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2993452509}) 2024-09-11T01:42:19.039 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.035+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3777144888}) 2024-09-11T01:42:19.040 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.035+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.040 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.035+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2993452509}) 2024-09-11T01:42:19.040 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.035+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1201057460}) 2024-09-11T01:42:19.040 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.036+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.040 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.036+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2738810292}) 2024-09-11T01:42:19.040 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.036+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2885533976}) 2024-09-11T01:42:19.040 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.036+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.040 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.036+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2738810292}) 2024-09-11T01:42:19.040 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.036+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=174301742}) 2024-09-11T01:42:19.041 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.037+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.041 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.037+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2472998671}) 2024-09-11T01:42:19.041 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.037+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2413118733}) 2024-09-11T01:42:19.041 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.037+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.041 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.037+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2472998671}) 2024-09-11T01:42:19.041 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.037+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3202230970}) 2024-09-11T01:42:19.044 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:42:19.083 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.080+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.083 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.080+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3539981385}) 2024-09-11T01:42:19.083 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.080+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1202480989}) 2024-09-11T01:42:19.083 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.080+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.083 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.080+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3539981385}) 2024-09-11T01:42:19.083 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.080+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1536861520}) 2024-09-11T01:42:19.083 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:19.134 INFO:tasks.rados.rados.0.smithi071.stdout:1923: finishing copy_from racing read to smithi07134777-43 2024-09-11T01:42:19.134 INFO:tasks.rados.rados.0.smithi071.stdout:1923: finishing copy_from to smithi07134777-43 2024-09-11T01:42:19.134 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 964 (ObjNum 689 snap 204 seq_num 689) dirty exists 2024-09-11T01:42:19.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.785+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.785+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2618153555}) 2024-09-11T01:42:19.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.785+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3427161243}) 2024-09-11T01:42:19.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.785+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.785+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2618153555}) 2024-09-11T01:42:19.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.785+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2019291753}) 2024-09-11T01:42:19.802 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.800+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.802 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.800+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=981929871}) 2024-09-11T01:42:19.802 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.800+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2443428177}) 2024-09-11T01:42:19.802 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.800+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.802 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.800+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=981929871}) 2024-09-11T01:42:19.802 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.800+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3787949021}) 2024-09-11T01:42:19.803 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.800+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.803 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.800+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3589542447}) 2024-09-11T01:42:19.803 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.800+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3432958381}) 2024-09-11T01:42:19.803 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.800+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.803 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.800+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3589542447}) 2024-09-11T01:42:19.803 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.800+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4292809896}) 2024-09-11T01:42:19.805 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.801+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.805 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.801+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3411890260}) 2024-09-11T01:42:19.805 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.801+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3288790261}) 2024-09-11T01:42:19.805 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.801+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.805 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.801+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3411890260}) 2024-09-11T01:42:19.805 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.801+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2080310003}) 2024-09-11T01:42:19.805 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.802+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.805 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.802+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642359671}) 2024-09-11T01:42:19.805 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.802+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3940609655}) 2024-09-11T01:42:19.805 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.802+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.805 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.802+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642359671}) 2024-09-11T01:42:19.806 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.802+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=704517701}) 2024-09-11T01:42:19.807 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.803+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.807 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.803+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2439224175}) 2024-09-11T01:42:19.807 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.803+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3272150557}) 2024-09-11T01:42:19.807 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.803+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.807 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.803+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2439224175}) 2024-09-11T01:42:19.807 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.803+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2472943737}) 2024-09-11T01:42:19.807 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.804+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.807 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.804+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3427811118}) 2024-09-11T01:42:19.807 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.804+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3087315613}) 2024-09-11T01:42:19.807 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.804+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.807 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.804+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3427811118}) 2024-09-11T01:42:19.807 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.804+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1759188851}) 2024-09-11T01:42:19.809 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.805+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.809 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.805+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3501241713}) 2024-09-11T01:42:19.809 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.805+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=666332472}) 2024-09-11T01:42:19.809 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.805+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.809 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.805+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3501241713}) 2024-09-11T01:42:19.809 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.805+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3734952608}) 2024-09-11T01:42:19.809 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.806+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.809 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.806+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=394062052}) 2024-09-11T01:42:19.809 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.806+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=369172476}) 2024-09-11T01:42:19.809 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.806+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.809 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.806+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=394062052}) 2024-09-11T01:42:19.810 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.806+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2339316116}) 2024-09-11T01:42:19.810 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.807+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.811 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.807+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2382913619}) 2024-09-11T01:42:19.811 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.807+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4065647659}) 2024-09-11T01:42:19.811 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.807+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.811 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.807+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2382913619}) 2024-09-11T01:42:19.811 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.807+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2022135604}) 2024-09-11T01:42:19.811 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.808+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.811 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.808+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=91847167}) 2024-09-11T01:42:19.811 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.808+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=281102201}) 2024-09-11T01:42:19.811 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.808+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.811 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.808+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=91847167}) 2024-09-11T01:42:19.811 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.808+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1524992654}) 2024-09-11T01:42:19.812 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.809+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.813 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.809+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2205778085}) 2024-09-11T01:42:19.813 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.809+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2292675951}) 2024-09-11T01:42:19.813 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.809+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.813 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.809+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2205778085}) 2024-09-11T01:42:19.813 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.809+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1286482800}) 2024-09-11T01:42:19.813 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.810+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.813 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.810+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3327641191}) 2024-09-11T01:42:19.813 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.810+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3372104430}) 2024-09-11T01:42:19.813 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.810+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.813 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.810+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3327641191}) 2024-09-11T01:42:19.813 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.810+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1657166587}) 2024-09-11T01:42:19.814 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.811+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.814 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.811+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2217068758}) 2024-09-11T01:42:19.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.811+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2674904129}) 2024-09-11T01:42:19.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.811+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.811+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2217068758}) 2024-09-11T01:42:19.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.811+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1591839676}) 2024-09-11T01:42:19.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.812+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.812+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1101140853}) 2024-09-11T01:42:19.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.812+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1515625728}) 2024-09-11T01:42:19.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.812+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.812+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1101140853}) 2024-09-11T01:42:19.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.812+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1523558278}) 2024-09-11T01:42:19.816 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.813+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.816 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.813+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3593168091}) 2024-09-11T01:42:19.816 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.813+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=833133862}) 2024-09-11T01:42:19.816 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.813+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.817 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.813+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3593168091}) 2024-09-11T01:42:19.817 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.813+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1834785468}) 2024-09-11T01:42:19.817 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.813+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.817 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.813+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1672221320}) 2024-09-11T01:42:19.817 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.814+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1874279933}) 2024-09-11T01:42:19.817 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.814+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.817 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.814+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1672221320}) 2024-09-11T01:42:19.817 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.814+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=579740202}) 2024-09-11T01:42:19.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.814+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.814+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=437371780}) 2024-09-11T01:42:19.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.814+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1160329864}) 2024-09-11T01:42:19.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.814+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.814+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=437371780}) 2024-09-11T01:42:19.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.814+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4272195481}) 2024-09-11T01:42:19.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.815+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.815+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3248316187}) 2024-09-11T01:42:19.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.815+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2742460273}) 2024-09-11T01:42:19.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.815+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.815+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3248316187}) 2024-09-11T01:42:19.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.815+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3764734188}) 2024-09-11T01:42:19.820 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.816+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.820 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.816+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3709994199}) 2024-09-11T01:42:19.820 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.816+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2469210970}) 2024-09-11T01:42:19.820 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.816+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.820 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.816+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3709994199}) 2024-09-11T01:42:19.820 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.816+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3727987922}) 2024-09-11T01:42:19.821 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.817+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.821 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.817+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3033107476}) 2024-09-11T01:42:19.821 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.817+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1611707031}) 2024-09-11T01:42:19.821 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.817+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.821 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.817+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3033107476}) 2024-09-11T01:42:19.821 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.817+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2674483549}) 2024-09-11T01:42:19.822 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.818+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.822 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.818+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3761037937}) 2024-09-11T01:42:19.822 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.818+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4067621333}) 2024-09-11T01:42:19.822 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.818+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.822 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.818+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3761037937}) 2024-09-11T01:42:19.822 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.818+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=112246627}) 2024-09-11T01:42:19.822 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.819+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.823 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.819+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=62725647}) 2024-09-11T01:42:19.823 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.819+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3190586275}) 2024-09-11T01:42:19.823 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.819+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.823 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.819+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=62725647}) 2024-09-11T01:42:19.823 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.819+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1348959414}) 2024-09-11T01:42:19.824 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.820+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.824 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.820+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=897929415}) 2024-09-11T01:42:19.824 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.820+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=65435010}) 2024-09-11T01:42:19.824 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.820+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.824 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.820+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=897929415}) 2024-09-11T01:42:19.824 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.820+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3937937404}) 2024-09-11T01:42:19.824 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.821+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.825 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.821+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1396272971}) 2024-09-11T01:42:19.825 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.821+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3657950310}) 2024-09-11T01:42:19.825 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.821+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.825 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.821+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1396272971}) 2024-09-11T01:42:19.825 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.821+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=776824324}) 2024-09-11T01:42:19.826 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.822+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.826 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.822+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=11,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=34} crc {logm=2401520679,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2528701524,mon_config_key=3330371543,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=2907016926,osd_snap=1656788815}) 2024-09-11T01:42:19.826 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.822+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=12,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=36} crc {logm=3504065225,mdsmap=2837440133,mgr=2672916156,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1747568224,mon_config_key=89421614,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=2907016926,osd_snap=1487264951}) 2024-09-11T01:42:19.826 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.822+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.827 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.822+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=11,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=34} crc {logm=2401520679,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2528701524,mon_config_key=3330371543,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=2907016926,osd_snap=1656788815}) 2024-09-11T01:42:19.827 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.822+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=12,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35} crc {logm=2582099950,mdsmap=3760429986,mgr=4124675564,mgr_metadata=3338432716,mgrstat=2528701524,mon_config_key=3330371543,monmap=3140514009,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=2907016926,osd_snap=1205413009}) 2024-09-11T01:42:19.827 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.823+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.827 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.823+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=57,osdmap=43} crc {osd_snap=2934323674,osdmap=3800469093}) 2024-09-11T01:42:19.827 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.823+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=51,osdmap=49} crc {osd_snap=882367406,osdmap=4242827543}) 2024-09-11T01:42:19.827 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.823+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.827 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.823+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=57,osdmap=43} crc {osd_snap=2934323674,osdmap=3800469093}) 2024-09-11T01:42:19.827 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.824+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=51,osdmap=49} crc {osd_snap=1951494163,osdmap=608963896}) 2024-09-11T01:42:19.828 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.824+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.828 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.825+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1739536095}) 2024-09-11T01:42:19.828 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.825+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=991784440}) 2024-09-11T01:42:19.828 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.825+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.828 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.825+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1739536095}) 2024-09-11T01:42:19.828 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.825+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=971460080}) 2024-09-11T01:42:19.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.963+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.963+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=93} crc {osdmap=641369560}) 2024-09-11T01:42:19.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.963+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=94} crc {osdmap=4294070751}) 2024-09-11T01:42:19.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.963+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:42:19.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.963+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=93} crc {osdmap=641369560}) 2024-09-11T01:42:19.966 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:42:19.963+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=96} crc {osdmap=3584446769}) 2024-09-11T01:42:20.045 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:42:20.046 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:42:20.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.155+0000 7f608d8c6640 10 monclient: get_monmap_and_config 2024-09-11T01:42:20.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.155+0000 7f608d8c6640 10 monclient: build_initial_monmap 2024-09-11T01:42:20.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.155+0000 7f608d8c6640 10 monclient: monmap: 2024-09-11T01:42:20.158 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:20.158 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:20.158 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:42:20.157535+0000 2024-09-11T01:42:20.158 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:42:20.157535+0000 2024-09-11T01:42:20.158 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:20.159 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.159 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:42:20.159 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:42:20.159 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:42:20.159 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:42:20.159 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:42:20.159 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.156+0000 7f608d8c6640 1 Processor -- start 2024-09-11T01:42:20.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.156+0000 7f608d8c6640 1 -- start start 2024-09-11T01:42:20.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.156+0000 7f608d8c6640 10 monclient: init 2024-09-11T01:42:20.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.156+0000 7f608d8c6640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:20.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.156+0000 7f608d8c6640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-11T01:42:20.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.156+0000 7f608d8c6640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f60880a2db0 0x7f608806cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.156+0000 7f608d8c6640 10 monclient(hunting): picked mon.noname-a con 0x7f60880a2db0 addr v2:172.21.15.12:3300/0 2024-09-11T01:42:20.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.156+0000 7f608d8c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f608809fbd0 0x7f608806d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.156+0000 7f608d8c6640 10 monclient(hunting): picked mon.noname-b con 0x7f608809fbd0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:20.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.156+0000 7f608d8c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f608806d5d0 addr v1:172.21.15.12:6789/0 2024-09-11T01:42:20.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.156+0000 7f608d8c6640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f608809eea0 con 0x7f608806d5d0 2024-09-11T01:42:20.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f608d8c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:20.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f608d8c6640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f60880a3850 con 0x7f60880a2db0 2024-09-11T01:42:20.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f608d8c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:20.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f608d8c6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f60880a3190 con 0x7f608809fbd0 2024-09-11T01:42:20.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f608d8c6640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:20.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f608d8c6640 10 monclient(hunting): authenticate will time out at 1546.600018s 2024-09-11T01:42:20.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f60867fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f608809fbd0 0x7f608806d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f6086ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f60880a2db0 0x7f608806cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f60877fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f608806d5d0 0x7f608813e550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:48954/0 (socket says 172.21.15.12:48954) 2024-09-11T01:42:20.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f60877fe640 1 -- 172.21.15.12:0/25853349 learned_addr learned my addr 172.21.15.12:0/25853349 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:42:20.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f60867fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f608809fbd0 0x7f608806d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:40858/0 (socket says 172.21.15.12:40858) 2024-09-11T01:42:20.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f60867fc640 10 monclient(hunting): get_auth_request con 0x7f608809fbd0 auth_method 0 2024-09-11T01:42:20.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f60867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:20.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f60867fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:20.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f60867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:20.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f6086ffd640 10 monclient(hunting): get_auth_request con 0x7f60880a2db0 auth_method 0 2024-09-11T01:42:20.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f6086ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:20.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f6086ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:20.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f6086ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:20.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f60867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:20.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f60867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:20.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f60867fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:20.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f6086ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:20.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f6086ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:20.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f6086ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:20.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f6086ffd640 10 monclient(hunting): handle_auth_done global_id 94578 payload 274 2024-09-11T01:42:20.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f6086ffd640 1 -- 172.21.15.12:0/25853349 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f608809fbd0 msgr2=0x7f608806d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:20.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f6086ffd640 1 --2- 172.21.15.12:0/25853349 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f608809fbd0 0x7f608806d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f6086ffd640 1 -- 172.21.15.12:0/25853349 >> v1:172.21.15.12:6789/0 conn(0x7f608806d5d0 legacy=0x7f608813e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:20.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f6086ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:42:20.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f6086ffd640 1 monclient: found mon.noname-a 2024-09-11T01:42:20.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f6086ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:20.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f6086ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:42:20.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f6086ffd640 1 -- 172.21.15.12:0/25853349 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f608809f180 con 0x7f60880a2db0 2024-09-11T01:42:20.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.157+0000 7f60867fc640 1 --2- 172.21.15.12:0/25853349 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f608809fbd0 0x7f608806d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:42:20.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.160+0000 7f6086ffd640 1 -- 172.21.15.12:0/25853349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60880a2db0 msgr2=0x7f608806cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:20.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.160+0000 7f6086ffd640 1 --2- 172.21.15.12:0/25853349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60880a2db0 0x7f608806cb50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f60780097d0 tx=0x7f607802a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4cc4639caffc801 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:20.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.160+0000 7f6085ffb640 1 -- 172.21.15.12:0/25853349 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6078037070 con 0x7f60880a2db0 2024-09-11T01:42:20.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.160+0000 7f6085ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:20.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.160+0000 7f6085ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:42:20.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.160+0000 7f6085ffb640 10 monclient: dump: 2024-09-11T01:42:20.170 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:20.170 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:20.170 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.170 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:20.171 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.160+0000 7f6085ffb640 10 monclient: _finish_auth 0 2024-09-11T01:42:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.160+0000 7f6085ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:42:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.160+0000 7f6085ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.160+0000 7f608d8c6640 5 monclient: authenticate success, global_id 94578 2024-09-11T01:42:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.160+0000 7f6085ffb640 1 -- 172.21.15.12:0/25853349 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6078032440 con 0x7f60880a2db0 2024-09-11T01:42:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.160+0000 7f608d8c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:42:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.160+0000 7f6085ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.160+0000 7f608d8c6640 10 monclient: get_monmap_and_config success 2024-09-11T01:42:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.160+0000 7f608d8c6640 4 set_mon_vals no callback set 2024-09-11T01:42:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.160+0000 7f6085ffb640 1 -- 172.21.15.12:0/25853349 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6078032810 con 0x7f60880a2db0 2024-09-11T01:42:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.160+0000 7f6085ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.160+0000 7f6085ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.160+0000 7f6085ffb640 10 monclient: dump: 2024-09-11T01:42:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:20.172 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:20.173 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:20.173 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.161+0000 7f608d8c6640 10 monclient: shutdown 2024-09-11T01:42:20.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.161+0000 7f608d8c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:20.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.161+0000 7f608d8c6640 1 -- 172.21.15.12:0/25853349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60880a2db0 msgr2=0x7f608806cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:20.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.161+0000 7f608d8c6640 1 --2- 172.21.15.12:0/25853349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60880a2db0 0x7f608806cb50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f60780097d0 tx=0x7f607802a000 comp rx=0 tx=0).stop 2024-09-11T01:42:20.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.161+0000 7f608d8c6640 1 -- 172.21.15.12:0/25853349 shutdown_connections 2024-09-11T01:42:20.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.161+0000 7f608d8c6640 1 --2- 172.21.15.12:0/25853349 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f608809fbd0 0x7f608806d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.161+0000 7f608d8c6640 1 --2- 172.21.15.12:0/25853349 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f60880a2db0 0x7f608806cb50 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.161+0000 7f608d8c6640 1 -- 172.21.15.12:0/25853349 >> 172.21.15.12:0/25853349 conn(0x7f60880a3ea0 msgr2=0x7f60880a42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:20.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.161+0000 7f608d8c6640 1 -- 172.21.15.12:0/25853349 shutdown_connections 2024-09-11T01:42:20.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.161+0000 7f608d8c6640 1 -- 172.21.15.12:0/25853349 wait complete. 2024-09-11T01:42:20.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.161+0000 7f608d8c6640 10 monclient: build_initial_monmap 2024-09-11T01:42:20.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.161+0000 7f608d8c6640 10 monclient: monmap: 2024-09-11T01:42:20.174 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:20.174 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:42:20.174 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:42:20.174 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:42:20.174 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:20.174 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.174 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:42:20.174 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:42:20.174 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:42:20.174 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f608d8c6640 1 Processor -- start 2024-09-11T01:42:20.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f608d8c6640 1 -- start start 2024-09-11T01:42:20.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f608d8c6640 10 monclient: init 2024-09-11T01:42:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f608d8c6640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f608d8c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:42:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f608d8c6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f608806d5d0 0x7f608813f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f608d8c6640 10 monclient(hunting): picked mon.noname-b con 0x7f608806d5d0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:42:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f608d8c6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f608809fbd0 0x7f60881c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f608d8c6640 10 monclient(hunting): picked mon.noname-a con 0x7f608809fbd0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:42:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f608d8c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f60880a2db0 0x7f60881ca290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f608d8c6640 10 monclient(hunting): picked mon.noname-c con 0x7f60880a2db0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f608d8c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f608d8c6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f608809ea20 con 0x7f608809fbd0 2024-09-11T01:42:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f608d8c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f608d8c6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f608809e8a0 con 0x7f608806d5d0 2024-09-11T01:42:20.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f608d8c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:20.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f608d8c6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f608813fc70 con 0x7f60880a2db0 2024-09-11T01:42:20.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f608d8c6640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:20.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f608d8c6640 10 monclient(hunting): authenticate will time out at 1546.605796s 2024-09-11T01:42:20.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f60877fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f60880a2db0 0x7f60881ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f60867fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f608809fbd0 0x7f60881c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f60867fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f608809fbd0 0x7f60881c9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52330/0 (socket says 172.21.15.12:52330) 2024-09-11T01:42:20.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f60877fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f60880a2db0 0x7f60881ca290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:40868/0 (socket says 172.21.15.12:40868) 2024-09-11T01:42:20.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f60867fc640 1 -- 172.21.15.12:0/1266489210 learned_addr learned my addr 172.21.15.12:0/1266489210 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:20.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f60867fc640 10 monclient(hunting): get_auth_request con 0x7f608809fbd0 auth_method 0 2024-09-11T01:42:20.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f60867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:20.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f60867fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:20.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f60867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:20.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f60877fe640 10 monclient(hunting): get_auth_request con 0x7f60880a2db0 auth_method 0 2024-09-11T01:42:20.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f60877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:20.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f60877fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:20.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.162+0000 7f60877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:20.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.163+0000 7f60877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:20.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.163+0000 7f60877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:20.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.163+0000 7f60877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:20.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.163+0000 7f60867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:20.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.163+0000 7f60867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:20.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.163+0000 7f60867fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:20.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.163+0000 7f6086ffd640 1 --2- 172.21.15.12:0/1266489210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f608806d5d0 0x7f608813f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.163+0000 7f60877fe640 10 monclient(hunting): handle_auth_done global_id 144122 payload 824 2024-09-11T01:42:20.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.163+0000 7f60877fe640 1 -- 172.21.15.12:0/1266489210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f608806d5d0 msgr2=0x7f608813f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.163+0000 7f60877fe640 1 --2- 172.21.15.12:0/1266489210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f608806d5d0 0x7f608813f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.163+0000 7f60877fe640 1 -- 172.21.15.12:0/1266489210 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f608809fbd0 msgr2=0x7f60881c9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.163+0000 7f60877fe640 1 --2- 172.21.15.12:0/1266489210 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f608809fbd0 0x7f60881c9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.163+0000 7f60877fe640 10 monclient: _finish_hunting 0 2024-09-11T01:42:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.163+0000 7f60877fe640 1 monclient: found mon.noname-c 2024-09-11T01:42:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.163+0000 7f60877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.163+0000 7f60877fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:42:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.163+0000 7f60877fe640 1 -- 172.21.15.12:0/1266489210 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f608813db20 con 0x7f60880a2db0 2024-09-11T01:42:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.163+0000 7f6086ffd640 10 monclient: get_auth_request con 0x7f608806d5d0 auth_method 0 2024-09-11T01:42:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.163+0000 7f6086ffd640 1 --2- 172.21.15.12:0/1266489210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f608806d5d0 0x7f608813f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:42:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.163+0000 7f60867fc640 1 --2- 172.21.15.12:0/1266489210 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f608809fbd0 0x7f60881c9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:42:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.165+0000 7f60877fe640 1 -- 172.21.15.12:0/1266489210 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f60880a2db0 msgr2=0x7f60881ca290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:20.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.165+0000 7f60877fe640 1 --2- 172.21.15.12:0/1266489210 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f60880a2db0 0x7f60881ca290 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f608000cb20 tx=0x7f608000cf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:20.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.165+0000 7f607ffff640 1 -- 172.21.15.12:0/1266489210 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6080017070 con 0x7f60880a2db0 2024-09-11T01:42:20.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.165+0000 7f607ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:20.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.165+0000 7f607ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:42:20.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.165+0000 7f607ffff640 10 monclient: dump: 2024-09-11T01:42:20.179 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:20.179 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:20.179 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.179 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.179 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:20.179 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.179 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:20.179 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:20.179 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:20.180 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.165+0000 7f607ffff640 10 monclient: _finish_auth 0 2024-09-11T01:42:20.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.165+0000 7f607ffff640 10 monclient: _check_auth_tickets 2024-09-11T01:42:20.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.165+0000 7f607ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:20.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.165+0000 7f607ffff640 1 -- 172.21.15.12:0/1266489210 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f608000ade0 con 0x7f60880a2db0 2024-09-11T01:42:20.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.165+0000 7f608d8c6640 5 monclient: authenticate success, global_id 144122 2024-09-11T01:42:20.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.165+0000 7f608d8c6640 10 monclient: _renew_subs 2024-09-11T01:42:20.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.165+0000 7f608d8c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:42:20.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.165+0000 7f608d8c6640 1 -- 172.21.15.12:0/1266489210 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60880a6a10 con 0x7f60880a2db0 2024-09-11T01:42:20.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.165+0000 7f607ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:20.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.165+0000 7f607ffff640 1 -- 172.21.15.12:0/1266489210 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6080012640 con 0x7f60880a2db0 2024-09-11T01:42:20.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.165+0000 7f607ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:20.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.165+0000 7f607ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:42:20.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.165+0000 7f607ffff640 10 monclient: dump: 2024-09-11T01:42:20.180 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:20.181 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:20.181 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.181 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.181 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:20.181 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.181 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:20.181 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:20.181 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:20.181 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.165+0000 7f60857fa640 4 set_mon_vals no callback set 2024-09-11T01:42:20.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.165+0000 7f608d8c6640 10 monclient: _renew_subs 2024-09-11T01:42:20.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.165+0000 7f608d8c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:42:20.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.165+0000 7f608d8c6640 1 -- 172.21.15.12:0/1266489210 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f608809e590 con 0x7f60880a2db0 2024-09-11T01:42:20.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.166+0000 7f607ffff640 1 -- 172.21.15.12:0/1266489210 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f6080007070 con 0x7f60880a2db0 2024-09-11T01:42:20.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.166+0000 7f608d8c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:20.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.166+0000 7f608d8c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:20.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.166+0000 7f608d8c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:42:20.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.166+0000 7f608d8c6640 1 -- 172.21.15.12:0/1266489210 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f608806cb50 con 0x7f60880a2db0 2024-09-11T01:42:20.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.167+0000 7f607ffff640 1 --2- 172.21.15.12:0/1266489210 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f606803e240 0x7f6068040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.167+0000 7f607ffff640 1 -- 172.21.15.12:0/1266489210 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(441..441 src has 285..441) ==== 4859+0+0 (secure 0 0 0) 0x7f6080052e00 con 0x7f60880a2db0 2024-09-11T01:42:20.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.167+0000 7f6086ffd640 1 --2- 172.21.15.12:0/1266489210 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f606803e240 0x7f6068040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.167+0000 7f6086ffd640 10 monclient: get_auth_request con 0x7f606803e240 auth_method 0 2024-09-11T01:42:20.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.170+0000 7f6086ffd640 1 --2- 172.21.15.12:0/1266489210 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f606803e240 0x7f6068040700 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6078034000 tx=0x7f607802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:20.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.172+0000 7f607ffff640 1 -- 172.21.15.12:0/1266489210 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f608001ac60 con 0x7f60880a2db0 2024-09-11T01:42:20.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.172+0000 7f607ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:20.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.172+0000 7f607ffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:42:20.200 INFO:tasks.rados.rados.0.smithi071.stdout:1923: done (2 left) 2024-09-11T01:42:20.200 INFO:tasks.rados.rados.0.smithi071.stdout:1924: done (1 left) 2024-09-11T01:42:20.201 INFO:tasks.rados.rados.0.smithi071.stdout:1925: done (0 left) 2024-09-11T01:42:20.201 INFO:tasks.rados.rados.0.smithi071.stdout:1926: write (excl) oid 32 current snap is 215 2024-09-11T01:42:20.201 INFO:tasks.rados.rados.0.smithi071.stdout:1926: seq_num 707 ranges {682549=705519,1922181=464578,3148997=487666} 2024-09-11T01:42:20.208 INFO:tasks.rados.rados.0.smithi071.stdout:1926: writing smithi07134777-32 from 682549 to 1388068 tid 1 2024-09-11T01:42:20.213 INFO:tasks.rados.rados.0.smithi071.stdout:1926: writing smithi07134777-32 from 1922181 to 2386759 tid 2 2024-09-11T01:42:20.220 INFO:tasks.rados.rados.0.smithi071.stdout:1926: writing smithi07134777-32 from 3148997 to 3636663 tid 3 2024-09-11T01:42:20.221 INFO:tasks.rados.rados.0.smithi071.stdout:1927: read oid 28 snap 212 2024-09-11T01:42:20.221 INFO:tasks.rados.rados.0.smithi071.stdout:1927: expect (ObjNum 636 snap 193 seq_num 636) 2024-09-11T01:42:20.221 INFO:tasks.rados.rados.0.smithi071.stdout:1928: read oid 4 snap -1 2024-09-11T01:42:20.221 INFO:tasks.rados.rados.0.smithi071.stdout:1928: expect (ObjNum 644 snap 196 seq_num 644) 2024-09-11T01:42:20.221 INFO:tasks.rados.rados.0.smithi071.stdout:1929: write oid 9 current snap is 215 2024-09-11T01:42:20.221 INFO:tasks.rados.rados.0.smithi071.stdout:1926: finishing write tid 1 to smithi07134777-32 2024-09-11T01:42:20.221 INFO:tasks.rados.rados.0.smithi071.stdout:1926: finishing write tid 2 to smithi07134777-32 2024-09-11T01:42:20.221 INFO:tasks.rados.rados.0.smithi071.stdout:1929: seq_num 708 ranges {699356=497121,1980663=608236,3176420=415291} 2024-09-11T01:42:20.226 INFO:tasks.rados.rados.0.smithi071.stdout:1929: writing smithi07134777-9 from 699356 to 1196477 tid 1 2024-09-11T01:42:20.234 INFO:tasks.rados.rados.0.smithi071.stdout:1929: writing smithi07134777-9 from 1980663 to 2588899 tid 2 2024-09-11T01:42:20.240 INFO:tasks.rados.rados.0.smithi071.stdout:1929: writing smithi07134777-9 from 3176420 to 3591711 tid 3 2024-09-11T01:42:20.240 INFO:tasks.rados.rados.0.smithi071.stdout:1930: snap_create 2024-09-11T01:42:20.240 INFO:tasks.rados.rados.0.smithi071.stdout:1926: finishing write tid 3 to smithi07134777-32 2024-09-11T01:42:20.240 INFO:tasks.rados.rados.0.smithi071.stdout:1926: finishing write tid 4 to smithi07134777-32 2024-09-11T01:42:20.240 INFO:tasks.rados.rados.0.smithi071.stdout:1926: finishing write tid 5 to smithi07134777-32 2024-09-11T01:42:20.240 INFO:tasks.rados.rados.0.smithi071.stdout:1926: oid 32 updating version 0 to 972 2024-09-11T01:42:20.241 INFO:tasks.rados.rados.0.smithi071.stdout:1926: oid 32 updating version 972 to 973 2024-09-11T01:42:20.241 INFO:tasks.rados.rados.0.smithi071.stdout:1926: oid 32 version 973 is already newer than 971 2024-09-11T01:42:20.241 INFO:tasks.rados.rados.0.smithi071.stdout:1926: oid 32 version 973 is already newer than 970 2024-09-11T01:42:20.241 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 973 (ObjNum 707 snap 215 seq_num 707) dirty exists 2024-09-11T01:42:20.241 INFO:tasks.rados.rados.0.smithi071.stdout:1926: left oid 32 (ObjNum 707 snap 215 seq_num 707) 2024-09-11T01:42:20.241 INFO:tasks.rados.rados.0.smithi071.stdout:1927: expect (ObjNum 636 snap 193 seq_num 636) 2024-09-11T01:42:20.269 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:20.316 INFO:tasks.rados.rados.0.smithi071.stdout:1928: expect (ObjNum 644 snap 196 seq_num 644) 2024-09-11T01:42:20.435 INFO:tasks.rados.rados.0.smithi071.stdout:1929: finishing write tid 1 to smithi07134777-9 2024-09-11T01:42:20.436 INFO:tasks.rados.rados.0.smithi071.stdout:1929: finishing write tid 2 to smithi07134777-9 2024-09-11T01:42:20.436 INFO:tasks.rados.rados.0.smithi071.stdout:1929: finishing write tid 3 to smithi07134777-9 2024-09-11T01:42:20.436 INFO:tasks.rados.rados.0.smithi071.stdout:1929: finishing write tid 4 to smithi07134777-9 2024-09-11T01:42:20.436 INFO:tasks.rados.rados.0.smithi071.stdout:1929: finishing write tid 5 to smithi07134777-9 2024-09-11T01:42:20.436 INFO:tasks.rados.rados.0.smithi071.stdout:1929: oid 9 updating version 0 to 542 2024-09-11T01:42:20.436 INFO:tasks.rados.rados.0.smithi071.stdout:1929: oid 9 updating version 542 to 543 2024-09-11T01:42:20.436 INFO:tasks.rados.rados.0.smithi071.stdout:1929: oid 9 updating version 543 to 544 2024-09-11T01:42:20.436 INFO:tasks.rados.rados.0.smithi071.stdout:1929: oid 9 updating version 544 to 545 2024-09-11T01:42:20.436 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 545 (ObjNum 708 snap 215 seq_num 708) dirty exists 2024-09-11T01:42:20.436 INFO:tasks.rados.rados.0.smithi071.stdout:1929: left oid 9 (ObjNum 708 snap 215 seq_num 708) 2024-09-11T01:42:20.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.436+0000 7f608d8c6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:42:20.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.436+0000 7f608d8c6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:42:20.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.436+0000 7f608d8c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:42:20.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.436+0000 7f608d8c6640 1 -- 172.21.15.12:0/1266489210 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f608813a550 con 0x7f60880a2db0 2024-09-11T01:42:20.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.436+0000 7f607ffff640 1 -- 172.21.15.12:0/1266489210 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f60800517b0 con 0x7f60880a2db0 2024-09-11T01:42:20.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.436+0000 7f607ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:42:20.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.436+0000 7f607ffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:42:20.439 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":122,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:42:20.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.438+0000 7f608d8c6640 1 -- 172.21.15.12:0/1266489210 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f606803e240 msgr2=0x7f6068040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:20.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.438+0000 7f608d8c6640 1 --2- 172.21.15.12:0/1266489210 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f606803e240 0x7f6068040700 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6078034000 tx=0x7f607802e000 comp rx=0 tx=0).stop 2024-09-11T01:42:20.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.438+0000 7f608d8c6640 10 monclient: shutdown 2024-09-11T01:42:20.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.439+0000 7f608d8c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:20.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.439+0000 7f608d8c6640 1 -- 172.21.15.12:0/1266489210 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f60880a2db0 msgr2=0x7f60881ca290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:20.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.439+0000 7f608d8c6640 1 --2- 172.21.15.12:0/1266489210 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f60880a2db0 0x7f60881ca290 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f608000cb20 tx=0x7f608000cf20 comp rx=0 tx=0).stop 2024-09-11T01:42:20.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.439+0000 7f608d8c6640 1 -- 172.21.15.12:0/1266489210 shutdown_connections 2024-09-11T01:42:20.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.439+0000 7f608d8c6640 1 --2- 172.21.15.12:0/1266489210 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f60880a2db0 0x7f60881ca290 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.439+0000 7f608d8c6640 1 --2- 172.21.15.12:0/1266489210 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f606803e240 0x7f6068040700 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.439+0000 7f608d8c6640 1 --2- 172.21.15.12:0/1266489210 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f608809fbd0 0x7f60881c9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.439+0000 7f608d8c6640 1 --2- 172.21.15.12:0/1266489210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f608806d5d0 0x7f608813f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.439+0000 7f608d8c6640 1 -- 172.21.15.12:0/1266489210 >> 172.21.15.12:0/1266489210 conn(0x7f60880a3ea0 msgr2=0x7f6088137720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:20.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.439+0000 7f608d8c6640 1 -- 172.21.15.12:0/1266489210 shutdown_connections 2024-09-11T01:42:20.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.439+0000 7f608d8c6640 1 -- 172.21.15.12:0/1266489210 wait complete. 2024-09-11T01:42:20.452 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:42:20.452 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:42:20.452 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:42:20.452 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:42:20.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.552+0000 7ff89d984640 10 monclient: get_monmap_and_config 2024-09-11T01:42:20.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.552+0000 7ff89d984640 10 monclient: build_initial_monmap 2024-09-11T01:42:20.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.552+0000 7ff89d984640 10 monclient: monmap: 2024-09-11T01:42:20.555 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:20.555 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:20.555 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:42:20.553826+0000 2024-09-11T01:42:20.555 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:42:20.553826+0000 2024-09-11T01:42:20.555 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:20.555 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.555 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:42:20.555 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:42:20.555 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:42:20.555 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:42:20.555 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:42:20.556 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.552+0000 7ff89d984640 1 Processor -- start 2024-09-11T01:42:20.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.552+0000 7ff89d984640 1 -- start start 2024-09-11T01:42:20.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.552+0000 7ff89d984640 10 monclient: init 2024-09-11T01:42:20.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff89d984640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:20.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff89d984640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-11T01:42:20.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff89d984640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff898137d50 0x7ff89809d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff89d984640 10 monclient(hunting): picked mon.noname-b con 0x7ff898137d50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:20.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff89d984640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff898138130 addr v1:172.21.15.12:6789/0 2024-09-11T01:42:20.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff89d984640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff89813d280 addr v1:172.21.15.71:6789/0 2024-09-11T01:42:20.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff89d984640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff89809ba20 con 0x7ff898138130 2024-09-11T01:42:20.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff89d984640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff89809c370 con 0x7ff89813d280 2024-09-11T01:42:20.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff89d984640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:20.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff89d984640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff89809c650 con 0x7ff898137d50 2024-09-11T01:42:20.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff89d984640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff89d984640 10 monclient(hunting): authenticate will time out at 1546.996264s 2024-09-11T01:42:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff896ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff898137d50 0x7ff89809d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff8967fc640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff898138130 0x7ff898138550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:48966/0 (socket says 172.21.15.12:48966) 2024-09-11T01:42:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff8967fc640 1 -- 172.21.15.12:0/1099627002 learned_addr learned my addr 172.21.15.12:0/1099627002 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:42:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff896ffd640 10 monclient(hunting): get_auth_request con 0x7ff898137d50 auth_method 0 2024-09-11T01:42:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff896ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff896ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff896ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff896ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff896ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff896ffd640 10 monclient(hunting): handle_auth_done global_id 144125 payload 274 2024-09-11T01:42:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff896ffd640 1 -- 172.21.15.12:0/1099627002 >> v1:172.21.15.71:6789/0 conn(0x7ff89813d280 legacy=0x7ff89813f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff896ffd640 1 -- 172.21.15.12:0/1099627002 >> v1:172.21.15.12:6789/0 conn(0x7ff898138130 legacy=0x7ff898138550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:20.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff896ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:42:20.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff896ffd640 1 monclient: found mon.noname-b 2024-09-11T01:42:20.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff896ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:20.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff896ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:42:20.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.553+0000 7ff896ffd640 1 -- 172.21.15.12:0/1099627002 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8980a6ce0 con 0x7ff898137d50 2024-09-11T01:42:20.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.556+0000 7ff896ffd640 1 -- 172.21.15.12:0/1099627002 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff898137d50 msgr2=0x7ff89809d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:20.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.556+0000 7ff896ffd640 1 --2- 172.21.15.12:0/1099627002 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff898137d50 0x7ff89809d110 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff8880097d0 tx=0x7ff88802a030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=433f263640cff92f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:20.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.556+0000 7ff895ffb640 1 -- 172.21.15.12:0/1099627002 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff888037070 con 0x7ff898137d50 2024-09-11T01:42:20.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.556+0000 7ff895ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:20.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.556+0000 7ff895ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:42:20.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.556+0000 7ff895ffb640 10 monclient: dump: 2024-09-11T01:42:20.567 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:20.567 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:20.567 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.567 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.567 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:20.567 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.567 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:20.567 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:20.567 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:20.567 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.556+0000 7ff895ffb640 10 monclient: _finish_auth 0 2024-09-11T01:42:20.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.556+0000 7ff895ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:42:20.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.556+0000 7ff895ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:20.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.556+0000 7ff895ffb640 1 -- 172.21.15.12:0/1099627002 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff888032640 con 0x7ff898137d50 2024-09-11T01:42:20.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.556+0000 7ff895ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:20.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.556+0000 7ff895ffb640 1 -- 172.21.15.12:0/1099627002 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff888032a10 con 0x7ff898137d50 2024-09-11T01:42:20.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.556+0000 7ff89d984640 5 monclient: authenticate success, global_id 144125 2024-09-11T01:42:20.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.556+0000 7ff89d984640 10 monclient: get_monmap_and_config success 2024-09-11T01:42:20.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.556+0000 7ff89d984640 4 set_mon_vals no callback set 2024-09-11T01:42:20.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.556+0000 7ff895ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:20.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.556+0000 7ff895ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:42:20.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.556+0000 7ff895ffb640 10 monclient: dump: 2024-09-11T01:42:20.569 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:20.569 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:20.569 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.569 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.569 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:20.569 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.569 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:20.569 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:20.569 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:20.569 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.557+0000 7ff89d984640 10 monclient: shutdown 2024-09-11T01:42:20.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.557+0000 7ff89d984640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:20.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.557+0000 7ff89d984640 1 -- 172.21.15.12:0/1099627002 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff898137d50 msgr2=0x7ff89809d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:20.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.557+0000 7ff89d984640 1 --2- 172.21.15.12:0/1099627002 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff898137d50 0x7ff89809d110 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff8880097d0 tx=0x7ff88802a030 comp rx=0 tx=0).stop 2024-09-11T01:42:20.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.557+0000 7ff89d984640 1 -- 172.21.15.12:0/1099627002 shutdown_connections 2024-09-11T01:42:20.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.557+0000 7ff89d984640 1 --2- 172.21.15.12:0/1099627002 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff898137d50 0x7ff89809d110 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.557+0000 7ff89d984640 1 -- 172.21.15.12:0/1099627002 >> 172.21.15.12:0/1099627002 conn(0x7ff8980a7020 msgr2=0x7ff8980a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:20.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.557+0000 7ff89d984640 1 -- 172.21.15.12:0/1099627002 shutdown_connections 2024-09-11T01:42:20.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.557+0000 7ff89d984640 1 -- 172.21.15.12:0/1099627002 wait complete. 2024-09-11T01:42:20.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.557+0000 7ff89d984640 10 monclient: build_initial_monmap 2024-09-11T01:42:20.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.557+0000 7ff89d984640 10 monclient: monmap: 2024-09-11T01:42:20.570 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:20.570 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:42:20.570 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:42:20.570 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:42:20.571 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:20.571 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.571 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:42:20.571 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:42:20.571 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:42:20.571 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.558+0000 7ff89d984640 1 Processor -- start 2024-09-11T01:42:20.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.558+0000 7ff89d984640 1 -- start start 2024-09-11T01:42:20.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.558+0000 7ff89d984640 10 monclient: init 2024-09-11T01:42:20.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.558+0000 7ff89d984640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:20.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.558+0000 7ff89d984640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:42:20.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.558+0000 7ff89d984640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff898137d50 0x7ff89813b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.558+0000 7ff89d984640 10 monclient(hunting): picked mon.noname-b con 0x7ff898137d50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:42:20.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.558+0000 7ff89d984640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff898138130 0x7ff898136dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.558+0000 7ff89d984640 10 monclient(hunting): picked mon.noname-c con 0x7ff898138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:20.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.558+0000 7ff89d984640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff89813d280 0x7ff8981ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.558+0000 7ff89d984640 10 monclient(hunting): picked mon.noname-a con 0x7ff89813d280 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:42:20.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.558+0000 7ff89d984640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:20.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.558+0000 7ff89d984640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff89809d7d0 con 0x7ff89813d280 2024-09-11T01:42:20.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.558+0000 7ff89d984640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:20.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.558+0000 7ff89d984640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff89809d650 con 0x7ff898137d50 2024-09-11T01:42:20.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.558+0000 7ff89d984640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:20.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.558+0000 7ff89d984640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff898140d90 con 0x7ff898138130 2024-09-11T01:42:20.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.558+0000 7ff89d984640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:20.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.558+0000 7ff89d984640 10 monclient(hunting): authenticate will time out at 1547.001890s 2024-09-11T01:42:20.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.558+0000 7ff8967fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff898138130 0x7ff898136dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.558+0000 7ff8977fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff89813d280 0x7ff8981ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.558+0000 7ff8967fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff898138130 0x7ff898136dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:40878/0 (socket says 172.21.15.12:40878) 2024-09-11T01:42:20.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff8967fc640 1 -- 172.21.15.12:0/3402720549 learned_addr learned my addr 172.21.15.12:0/3402720549 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:20.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff8967fc640 10 monclient(hunting): get_auth_request con 0x7ff898138130 auth_method 0 2024-09-11T01:42:20.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff8967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:20.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff8967fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:20.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff8967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:20.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff8977fe640 10 monclient(hunting): get_auth_request con 0x7ff89813d280 auth_method 0 2024-09-11T01:42:20.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff8977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:20.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff8977fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:20.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff8977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:20.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff8967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:20.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff8967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:20.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff896ffd640 1 --2- 172.21.15.12:0/3402720549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff898137d50 0x7ff89813b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff8967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:20.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff8977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:20.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff8977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:20.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff8977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:20.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff896ffd640 10 monclient(hunting): get_auth_request con 0x7ff898137d50 auth_method 0 2024-09-11T01:42:20.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff896ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:20.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff896ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:20.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:20.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff896ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:20.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff896ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:20.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff896ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:20.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff8967fc640 10 monclient(hunting): handle_auth_done global_id 144128 payload 824 2024-09-11T01:42:20.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff8967fc640 1 -- 172.21.15.12:0/3402720549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff898137d50 msgr2=0x7ff89813b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:20.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff8967fc640 1 --2- 172.21.15.12:0/3402720549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff898137d50 0x7ff89813b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff8967fc640 1 -- 172.21.15.12:0/3402720549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff89813d280 msgr2=0x7ff8981ccf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:20.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff8967fc640 1 --2- 172.21.15.12:0/3402720549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff89813d280 0x7ff8981ccf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff8967fc640 10 monclient: _finish_hunting 0 2024-09-11T01:42:20.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff8967fc640 1 monclient: found mon.noname-c 2024-09-11T01:42:20.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff8967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:20.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff8967fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:42:20.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff8967fc640 1 -- 172.21.15.12:0/3402720549 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff89813ce60 con 0x7ff898138130 2024-09-11T01:42:20.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.559+0000 7ff8977fe640 1 --2- 172.21.15.12:0/3402720549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff89813d280 0x7ff8981ccf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:42:20.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.561+0000 7ff8967fc640 1 -- 172.21.15.12:0/3402720549 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff898138130 msgr2=0x7ff898136dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:20.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.561+0000 7ff8967fc640 1 --2- 172.21.15.12:0/3402720549 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff898138130 0x7ff898136dd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff88c00c460 tx=0x7ff88c00c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:20.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.561+0000 7ff87ffff640 1 -- 172.21.15.12:0/3402720549 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88c007070 con 0x7ff898138130 2024-09-11T01:42:20.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.561+0000 7ff87ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:20.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.561+0000 7ff87ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:42:20.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.561+0000 7ff87ffff640 10 monclient: dump: 2024-09-11T01:42:20.575 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:20.575 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:20.576 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.576 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.576 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:20.576 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.576 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:20.576 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:20.576 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:20.576 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.561+0000 7ff87ffff640 10 monclient: _finish_auth 0 2024-09-11T01:42:20.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.561+0000 7ff87ffff640 10 monclient: _check_auth_tickets 2024-09-11T01:42:20.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.561+0000 7ff87ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:20.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.561+0000 7ff87ffff640 1 -- 172.21.15.12:0/3402720549 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff88c007230 con 0x7ff898138130 2024-09-11T01:42:20.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.561+0000 7ff89d984640 5 monclient: authenticate success, global_id 144128 2024-09-11T01:42:20.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.561+0000 7ff87ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:20.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.561+0000 7ff89d984640 10 monclient: _renew_subs 2024-09-11T01:42:20.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.561+0000 7ff87ffff640 1 -- 172.21.15.12:0/3402720549 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88c007620 con 0x7ff898138130 2024-09-11T01:42:20.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.561+0000 7ff89d984640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:42:20.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.561+0000 7ff89d984640 1 -- 172.21.15.12:0/3402720549 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff898140840 con 0x7ff898138130 2024-09-11T01:42:20.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.561+0000 7ff87ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:20.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.561+0000 7ff87ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:42:20.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.561+0000 7ff87ffff640 10 monclient: dump: 2024-09-11T01:42:20.577 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:20.577 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:20.577 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.577 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.577 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:20.578 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.578 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:20.578 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:20.578 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:20.578 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.561+0000 7ff8957fa640 4 set_mon_vals no callback set 2024-09-11T01:42:20.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.562+0000 7ff89d984640 10 monclient: _renew_subs 2024-09-11T01:42:20.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.562+0000 7ff89d984640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:42:20.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.562+0000 7ff89d984640 1 -- 172.21.15.12:0/3402720549 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff89813a5b0 con 0x7ff898138130 2024-09-11T01:42:20.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.562+0000 7ff89d984640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:20.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.562+0000 7ff87ffff640 1 -- 172.21.15.12:0/3402720549 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff88c01b400 con 0x7ff898138130 2024-09-11T01:42:20.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.562+0000 7ff89d984640 1 --2- 172.21.15.12:0/3402720549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8981396b0 0x7ff8981cd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.562+0000 7ff89d984640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:20.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.562+0000 7ff89d984640 1 -- 172.21.15.12:0/3402720549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff89806a250 con 0x7ff8981396b0 2024-09-11T01:42:20.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.562+0000 7ff896ffd640 1 --2- 172.21.15.12:0/3402720549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8981396b0 0x7ff8981cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.562+0000 7ff896ffd640 10 monclient: get_auth_request con 0x7ff8981396b0 auth_method 0 2024-09-11T01:42:20.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.562+0000 7ff896ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:20.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.562+0000 7ff896ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:20.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.562+0000 7ff896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:20.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.562+0000 7ff896ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:20.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.562+0000 7ff896ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:20.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.562+0000 7ff896ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:20.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.563+0000 7ff87ffff640 1 --2- 172.21.15.12:0/3402720549 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff88403e290 0x7ff884040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.563+0000 7ff87ffff640 1 -- 172.21.15.12:0/3402720549 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(442..442 src has 285..442) ==== 4787+0+0 (secure 0 0 0) 0x7ff88c0528d0 con 0x7ff898138130 2024-09-11T01:42:20.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.563+0000 7ff896ffd640 10 monclient(hunting): handle_auth_done global_id 94587 payload 824 2024-09-11T01:42:20.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.563+0000 7ff896ffd640 1 -- 172.21.15.12:0/3402720549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff89809d190 con 0x7ff8981396b0 2024-09-11T01:42:20.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.563+0000 7ff8977fe640 1 --2- 172.21.15.12:0/3402720549 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff88403e290 0x7ff884040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.563+0000 7ff8977fe640 10 monclient: get_auth_request con 0x7ff88403e290 auth_method 0 2024-09-11T01:42:20.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.565+0000 7ff896ffd640 1 -- 172.21.15.12:0/3402720549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8981396b0 msgr2=0x7ff8981cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:20.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.565+0000 7ff896ffd640 1 --2- 172.21.15.12:0/3402720549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8981396b0 0x7ff8981cd480 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff8980a6050 tx=0x7ff8880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:20.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.565+0000 7ff87ffff640 1 -- 172.21.15.12:0/3402720549 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff888037040 con 0x7ff8981396b0 2024-09-11T01:42:20.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.565+0000 7ff87ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:20.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.565+0000 7ff87ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:20.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.565+0000 7ff87ffff640 10 monclient: dump: 2024-09-11T01:42:20.582 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:20.582 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:20.582 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.582 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.583 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:20.583 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.583 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:20.583 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:20.583 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:20.583 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.565+0000 7ff87ffff640 1 -- 172.21.15.12:0/3402720549 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff888032e40 con 0x7ff8981396b0 2024-09-11T01:42:20.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.565+0000 7ff87ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:20.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.565+0000 7ff87ffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:42:20.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.565+0000 7ff87ffff640 1 -- 172.21.15.12:0/3402720549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8981396b0 msgr2=0x7ff8981cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:20.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.565+0000 7ff87ffff640 1 --2- 172.21.15.12:0/3402720549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8981396b0 0x7ff8981cd480 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff8980a6050 tx=0x7ff8880046c0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.565+0000 7ff87ffff640 1 -- 172.21.15.12:0/3402720549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8981396b0 msgr2=0x7ff8981cd480 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:42:20.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.565+0000 7ff87ffff640 1 --2- 172.21.15.12:0/3402720549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8981396b0 0x7ff8981cd480 secure :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff8980a6050 tx=0x7ff8880046c0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.565+0000 7ff8977fe640 1 --2- 172.21.15.12:0/3402720549 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff88403e290 0x7ff884040750 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff89806fd10 tx=0x7ff890009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:20.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.571+0000 7ff89d984640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:42:20.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.571+0000 7ff89d984640 1 --2- 172.21.15.12:0/3402720549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8981cd9c0 0x7ff8980a2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.571+0000 7ff89d984640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:20.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.571+0000 7ff89d984640 1 -- 172.21.15.12:0/3402720549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff8981cdda0 con 0x7ff8981cd9c0 2024-09-11T01:42:20.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.572+0000 7ff896ffd640 1 --2- 172.21.15.12:0/3402720549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8981cd9c0 0x7ff8980a2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.572+0000 7ff896ffd640 10 monclient: get_auth_request con 0x7ff8981cd9c0 auth_method 0 2024-09-11T01:42:20.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.572+0000 7ff896ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:20.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.572+0000 7ff896ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:20.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.572+0000 7ff896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:20.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.572+0000 7ff896ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:20.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.572+0000 7ff896ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:20.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.572+0000 7ff896ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:20.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.572+0000 7ff896ffd640 10 monclient(hunting): handle_auth_done global_id 94590 payload 824 2024-09-11T01:42:20.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.572+0000 7ff896ffd640 1 -- 172.21.15.12:0/3402720549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff898074a40 con 0x7ff8981cd9c0 2024-09-11T01:42:20.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.574+0000 7ff896ffd640 1 -- 172.21.15.12:0/3402720549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8981cd9c0 msgr2=0x7ff8980a2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:20.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.574+0000 7ff896ffd640 1 --2- 172.21.15.12:0/3402720549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8981cd9c0 0x7ff8980a2f10 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff88802a070 tx=0x7ff888008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:20.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.574+0000 7ff87ffff640 1 -- 172.21.15.12:0/3402720549 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff888037040 con 0x7ff8981cd9c0 2024-09-11T01:42:20.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.574+0000 7ff87ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:20.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.574+0000 7ff87ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:20.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.574+0000 7ff87ffff640 10 monclient: dump: 2024-09-11T01:42:20.585 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:20.585 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:20.585 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.585 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.585 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:20.586 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.586 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:20.586 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:20.586 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:20.586 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.574+0000 7ff87ffff640 1 -- 172.21.15.12:0/3402720549 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7ff88804e070 con 0x7ff8981cd9c0 2024-09-11T01:42:20.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.574+0000 7ff87ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:42:20.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.574+0000 7ff87ffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:42:20.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.574+0000 7ff87ffff640 1 -- 172.21.15.12:0/3402720549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8981cd9c0 msgr2=0x7ff8980a2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:20.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.574+0000 7ff87ffff640 1 --2- 172.21.15.12:0/3402720549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8981cd9c0 0x7ff8980a2f10 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff88802a070 tx=0x7ff888008030 comp rx=0 tx=0).stop 2024-09-11T01:42:20.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.574+0000 7ff87ffff640 1 -- 172.21.15.12:0/3402720549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8981cd9c0 msgr2=0x7ff8980a2f10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:42:20.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.574+0000 7ff87ffff640 1 --2- 172.21.15.12:0/3402720549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8981cd9c0 0x7ff8980a2f10 secure :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff88802a070 tx=0x7ff888008030 comp rx=0 tx=0).stop 2024-09-11T01:42:20.587 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:42:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:42:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:42:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:42:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 122, 2024-09-11T01:42:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 286670, 2024-09-11T01:42:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:42:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:42:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:42:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:42:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:20.587 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 6, 2024-09-11T01:42:20.588 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:42:20.588 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:42:20.588 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:42:20.588 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:42:20.588 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:42:20.588 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:42:20.588 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:42:20.588 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:42:20.588 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:42:20.588 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:42:20.588 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:42:20.588 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:42:20.588 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:42:20.589 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:42:20.589 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:20.589 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:42:20.589 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:42:20.589 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:42:20.589 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:42:20.589 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:42:20.589 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:42:20.589 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:42:20.589 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:42:20.589 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:42:20.589 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:42:20.589 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:42:20.589 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:42:20.590 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:42:20.590 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:20.590 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.590 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:42:20.590 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:42:20.590 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:42:20.590 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:42:20.590 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:42:20.590 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:42:20.590 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:42:20.590 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:42:20.590 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:42:20.590 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:42:20.591 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:42:20.591 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:42:20.591 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:42:20.591 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:42:20.591 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:42:20.591 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:42:20.591 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:42:20.591 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:42:20.591 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:42:20.591 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:42:20.591 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:42:20.591 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:42:20.591 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:42:20.591 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:42:20.592 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:42:20.592 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:42:20.592 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:42:20.592 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:42:20.592 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:20.592 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:42:20.592 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.592 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:42:20.592 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.592 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:42:20.592 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:42:20.592 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:42:20.592 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:42:20.593 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.593 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:42:20.593 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:42:20.593 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:20.593 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.593 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.593 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:42:20.593 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:42:20.593 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:20.593 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:20.593 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:20.593 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.593 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:42:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:42:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:42:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:42:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:42:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:42:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:42:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:42:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:42:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:42:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:42:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:20.594 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.595 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.595 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:42:20.595 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:42:20.595 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:20.595 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:20.595 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:20.595 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.595 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:42:20.595 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:42:20.595 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:42:20.595 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:42:20.595 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:42:20.595 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.596 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.596 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:42:20.596 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:42:20.596 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:42:20.596 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:42:20.596 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.596 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:42:20.596 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:42:20.596 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:20.596 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.596 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.596 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:42:20.596 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:42:20.596 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:42:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:42:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:42:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:42:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:42:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:42:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:42:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:42:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:42:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:42:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:42:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:42:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:42:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:42:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:42:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:42:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:42:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:42:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:42:20.599 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:42:20.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.575+0000 7ff89d984640 1 -- 172.21.15.12:0/3402720549 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff88403e290 msgr2=0x7ff884040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:20.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.575+0000 7ff89d984640 1 --2- 172.21.15.12:0/3402720549 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff88403e290 0x7ff884040750 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff89806fd10 tx=0x7ff890009040 comp rx=0 tx=0).stop 2024-09-11T01:42:20.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.575+0000 7ff89d984640 10 monclient: shutdown 2024-09-11T01:42:20.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.575+0000 7ff89d984640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:20.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.575+0000 7ff89d984640 1 -- 172.21.15.12:0/3402720549 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff898138130 msgr2=0x7ff898136dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:20.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.575+0000 7ff89d984640 1 --2- 172.21.15.12:0/3402720549 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff898138130 0x7ff898136dd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff88c00c460 tx=0x7ff88c00c920 comp rx=0 tx=0).stop 2024-09-11T01:42:20.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.575+0000 7ff8977fe640 1 -- 172.21.15.12:0/3402720549 reap_dead start 2024-09-11T01:42:20.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.575+0000 7ff8977fe640 1 -- 172.21.15.12:0/3402720549 reap_dead start 2024-09-11T01:42:20.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.575+0000 7ff89d984640 1 -- 172.21.15.12:0/3402720549 shutdown_connections 2024-09-11T01:42:20.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.575+0000 7ff89d984640 1 -- 172.21.15.12:0/3402720549 >> 172.21.15.12:0/3402720549 conn(0x7ff8980a7020 msgr2=0x7ff8980a2550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:20.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.575+0000 7ff89d984640 1 -- 172.21.15.12:0/3402720549 shutdown_connections 2024-09-11T01:42:20.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.575+0000 7ff89d984640 1 -- 172.21.15.12:0/3402720549 wait complete. 2024-09-11T01:42:20.602 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:42:20.602 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:42:20.602 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:42:20.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.701+0000 7f6ac12e6640 10 monclient: get_monmap_and_config 2024-09-11T01:42:20.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.701+0000 7f6ac12e6640 10 monclient: build_initial_monmap 2024-09-11T01:42:20.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.701+0000 7f6ac12e6640 10 monclient: monmap: 2024-09-11T01:42:20.704 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:20.704 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:20.704 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:42:20.702920+0000 2024-09-11T01:42:20.704 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:42:20.702920+0000 2024-09-11T01:42:20.704 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:20.704 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.704 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:42:20.704 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:42:20.704 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:42:20.704 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:42:20.705 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:42:20.705 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.701+0000 7f6ac12e6640 1 Processor -- start 2024-09-11T01:42:20.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.701+0000 7f6ac12e6640 1 -- start start 2024-09-11T01:42:20.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.701+0000 7f6ac12e6640 10 monclient: init 2024-09-11T01:42:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6ac12e6640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6ac12e6640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-11T01:42:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6ac12e6640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6abc137d50 0x7f6abc09d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6ac12e6640 10 monclient(hunting): picked mon.noname-a con 0x7f6abc137d50 addr v2:172.21.15.12:3300/0 2024-09-11T01:42:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6ac12e6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc138130 0x7f6abc138550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6ac12e6640 10 monclient(hunting): picked mon.noname-b con 0x7f6abc138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6ac12e6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6abc136cd0 addr v1:172.21.15.71:6789/0 2024-09-11T01:42:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6ac12e6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6abc09ba20 con 0x7f6abc136cd0 2024-09-11T01:42:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6ac12e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6ac12e6640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f6abc09c650 con 0x7f6abc137d50 2024-09-11T01:42:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6ac12e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6ac12e6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6abc09c370 con 0x7f6abc138130 2024-09-11T01:42:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6ac12e6640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6ac12e6640 10 monclient(hunting): authenticate will time out at 1547.145355s 2024-09-11T01:42:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6abaffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6abc137d50 0x7f6abc09d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6aba7fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc138130 0x7f6abc138550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6abaffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6abc137d50 0x7f6abc09d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52358/0 (socket says 172.21.15.12:52358) 2024-09-11T01:42:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6abaffd640 1 -- 172.21.15.12:0/1245833756 learned_addr learned my addr 172.21.15.12:0/1245833756 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6aba7fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc138130 0x7f6abc138550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:40894/0 (socket says 172.21.15.12:40894) 2024-09-11T01:42:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6abaffd640 10 monclient(hunting): get_auth_request con 0x7f6abc137d50 auth_method 0 2024-09-11T01:42:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6abaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6abaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6abaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6aba7fc640 10 monclient(hunting): get_auth_request con 0x7f6abc138130 auth_method 0 2024-09-11T01:42:20.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6aba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:20.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6aba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:20.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6aba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:20.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6abaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:20.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6abaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:20.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6abaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:20.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6aba7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:20.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6aba7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:20.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6aba7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:20.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6abaffd640 10 monclient(hunting): handle_auth_done global_id 94593 payload 274 2024-09-11T01:42:20.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6abaffd640 1 -- 172.21.15.12:0/1245833756 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc138130 msgr2=0x7f6abc138550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:20.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6abaffd640 1 --2- 172.21.15.12:0/1245833756 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc138130 0x7f6abc138550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6abaffd640 1 -- 172.21.15.12:0/1245833756 >> v1:172.21.15.71:6789/0 conn(0x7f6abc136cd0 legacy=0x7f6abc137120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:20.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6abaffd640 10 monclient: _finish_hunting 0 2024-09-11T01:42:20.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6abaffd640 1 monclient: found mon.noname-a 2024-09-11T01:42:20.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6abaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:20.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6abaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:42:20.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6abaffd640 1 -- 172.21.15.12:0/1245833756 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6abc0a6ce0 con 0x7f6abc137d50 2024-09-11T01:42:20.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.702+0000 7f6aba7fc640 1 --2- 172.21.15.12:0/1245833756 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc138130 0x7f6abc138550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:42:20.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.705+0000 7f6abaffd640 1 -- 172.21.15.12:0/1245833756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6abc137d50 msgr2=0x7f6abc09d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:20.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.705+0000 7f6abaffd640 1 --2- 172.21.15.12:0/1245833756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6abc137d50 0x7f6abc09d110 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6abc09c7d0 tx=0x7f6aac02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16bed4f8ba2553f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:20.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.705+0000 7f6ab9ffb640 1 -- 172.21.15.12:0/1245833756 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aac036070 con 0x7f6abc137d50 2024-09-11T01:42:20.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.705+0000 7f6ab9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:20.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.705+0000 7f6ab9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:42:20.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.705+0000 7f6ab9ffb640 10 monclient: dump: 2024-09-11T01:42:20.716 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:20.716 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:20.716 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.716 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.716 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:20.717 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.717 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:20.717 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:20.717 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:20.717 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.705+0000 7f6ab9ffb640 10 monclient: _finish_auth 0 2024-09-11T01:42:20.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.705+0000 7f6ab9ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:42:20.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.705+0000 7f6ab9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:20.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.705+0000 7f6ac12e6640 5 monclient: authenticate success, global_id 94593 2024-09-11T01:42:20.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.705+0000 7f6ab9ffb640 1 -- 172.21.15.12:0/1245833756 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6aac031600 con 0x7f6abc137d50 2024-09-11T01:42:20.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.705+0000 7f6ac12e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:42:20.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.705+0000 7f6ab9ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:20.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.705+0000 7f6ac12e6640 10 monclient: get_monmap_and_config success 2024-09-11T01:42:20.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.705+0000 7f6ac12e6640 4 set_mon_vals no callback set 2024-09-11T01:42:20.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.705+0000 7f6ab9ffb640 1 -- 172.21.15.12:0/1245833756 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aac0319d0 con 0x7f6abc137d50 2024-09-11T01:42:20.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.705+0000 7f6ab9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:20.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.705+0000 7f6ab9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:20.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.705+0000 7f6ab9ffb640 10 monclient: dump: 2024-09-11T01:42:20.718 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:20.718 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:20.718 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.718 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.718 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:20.718 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.718 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:20.718 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:20.718 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:20.719 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.706+0000 7f6ac12e6640 10 monclient: shutdown 2024-09-11T01:42:20.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.706+0000 7f6ac12e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:20.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.706+0000 7f6ac12e6640 1 -- 172.21.15.12:0/1245833756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6abc137d50 msgr2=0x7f6abc09d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:20.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.706+0000 7f6ac12e6640 1 --2- 172.21.15.12:0/1245833756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6abc137d50 0x7f6abc09d110 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6abc09c7d0 tx=0x7f6aac02a5d0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.706+0000 7f6ac12e6640 1 -- 172.21.15.12:0/1245833756 shutdown_connections 2024-09-11T01:42:20.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.706+0000 7f6ac12e6640 1 --2- 172.21.15.12:0/1245833756 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc138130 0x7f6abc138550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.706+0000 7f6ac12e6640 1 --2- 172.21.15.12:0/1245833756 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6abc137d50 0x7f6abc09d110 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.706+0000 7f6ac12e6640 1 -- 172.21.15.12:0/1245833756 >> 172.21.15.12:0/1245833756 conn(0x7f6abc0a7020 msgr2=0x7f6abc0a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:20.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.706+0000 7f6ac12e6640 1 -- 172.21.15.12:0/1245833756 shutdown_connections 2024-09-11T01:42:20.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.706+0000 7f6ac12e6640 1 -- 172.21.15.12:0/1245833756 wait complete. 2024-09-11T01:42:20.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.707+0000 7f6ac12e6640 10 monclient: build_initial_monmap 2024-09-11T01:42:20.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.707+0000 7f6ac12e6640 10 monclient: monmap: 2024-09-11T01:42:20.720 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:20.720 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:42:20.720 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:42:20.720 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:42:20.720 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:20.720 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.720 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:42:20.720 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:42:20.720 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:42:20.720 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.707+0000 7f6ac12e6640 1 Processor -- start 2024-09-11T01:42:20.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.707+0000 7f6ac12e6640 1 -- start start 2024-09-11T01:42:20.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.707+0000 7f6ac12e6640 10 monclient: init 2024-09-11T01:42:20.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6ac12e6640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:20.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6ac12e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:42:20.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6ac12e6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6abc136cd0 0x7f6abc1cd620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6ac12e6640 10 monclient(hunting): picked mon.noname-a con 0x7f6abc136cd0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:42:20.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6ac12e6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc137d50 0x7f6abc123590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6ac12e6640 10 monclient(hunting): picked mon.noname-c con 0x7f6abc137d50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:20.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6ac12e6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6abc138130 0x7f6abc123ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6ac12e6640 10 monclient(hunting): picked mon.noname-b con 0x7f6abc138130 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:42:20.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6ac12e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:20.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6ac12e6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6abc09d650 con 0x7f6abc136cd0 2024-09-11T01:42:20.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6ac12e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:20.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6ac12e6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6abc06f710 con 0x7f6abc138130 2024-09-11T01:42:20.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6ac12e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:20.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6ac12e6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6abc09d7d0 con 0x7f6abc137d50 2024-09-11T01:42:20.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6ac12e6640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:20.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6ac12e6640 10 monclient(hunting): authenticate will time out at 1547.151209s 2024-09-11T01:42:20.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6abaffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6abc136cd0 0x7f6abc1cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6aba7fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc137d50 0x7f6abc123590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6aba7fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc137d50 0x7f6abc123590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:40900/0 (socket says 172.21.15.12:40900) 2024-09-11T01:42:20.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6aba7fc640 1 -- 172.21.15.12:0/198895146 learned_addr learned my addr 172.21.15.12:0/198895146 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:20.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6aba7fc640 10 monclient(hunting): get_auth_request con 0x7f6abc137d50 auth_method 0 2024-09-11T01:42:20.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6aba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:20.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6aba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:20.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6aba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:20.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6abaffd640 10 monclient(hunting): get_auth_request con 0x7f6abc136cd0 auth_method 0 2024-09-11T01:42:20.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6abaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:20.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6abaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:20.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6abaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:20.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6aba7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:20.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6aba7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:20.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6aba7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:20.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6abaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:20.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6abaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:20.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6abaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:20.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6abb7fe640 1 --2- 172.21.15.12:0/198895146 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6abc138130 0x7f6abc123ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6abb7fe640 10 monclient(hunting): get_auth_request con 0x7f6abc138130 auth_method 0 2024-09-11T01:42:20.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6abb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:20.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6abb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:20.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6abb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:20.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6abb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:20.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6abb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:20.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6abb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:20.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6aba7fc640 10 monclient(hunting): handle_auth_done global_id 144134 payload 824 2024-09-11T01:42:20.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6aba7fc640 1 -- 172.21.15.12:0/198895146 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6abc138130 msgr2=0x7f6abc123ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:20.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6aba7fc640 1 --2- 172.21.15.12:0/198895146 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6abc138130 0x7f6abc123ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6aba7fc640 1 -- 172.21.15.12:0/198895146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6abc136cd0 msgr2=0x7f6abc1cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:20.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6aba7fc640 1 --2- 172.21.15.12:0/198895146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6abc136cd0 0x7f6abc1cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6aba7fc640 10 monclient: _finish_hunting 0 2024-09-11T01:42:20.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6aba7fc640 1 monclient: found mon.noname-c 2024-09-11T01:42:20.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6aba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:20.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6aba7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:42:20.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.708+0000 7f6aba7fc640 1 -- 172.21.15.12:0/198895146 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6abc136970 con 0x7f6abc137d50 2024-09-11T01:42:20.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.710+0000 7f6aba7fc640 1 -- 172.21.15.12:0/198895146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc137d50 msgr2=0x7f6abc123590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:20.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.710+0000 7f6aba7fc640 1 --2- 172.21.15.12:0/198895146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc137d50 0x7f6abc123590 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400ebe0 tx=0x7f6aa400c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:20.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.710+0000 7f6ab37fe640 1 -- 172.21.15.12:0/198895146 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4017070 con 0x7f6abc137d50 2024-09-11T01:42:20.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.711+0000 7f6ab37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:20.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.711+0000 7f6ab37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:42:20.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.711+0000 7f6ab37fe640 10 monclient: dump: 2024-09-11T01:42:20.725 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:20.725 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:20.725 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.725 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.725 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:20.725 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.725 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:20.725 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:20.725 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:20.725 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.711+0000 7f6ab37fe640 10 monclient: _finish_auth 0 2024-09-11T01:42:20.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.711+0000 7f6ab37fe640 10 monclient: _check_auth_tickets 2024-09-11T01:42:20.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.711+0000 7f6ab37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:20.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.711+0000 7f6ac12e6640 5 monclient: authenticate success, global_id 144134 2024-09-11T01:42:20.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.711+0000 7f6ab37fe640 1 -- 172.21.15.12:0/198895146 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6aa400cde0 con 0x7f6abc137d50 2024-09-11T01:42:20.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.711+0000 7f6ab37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:20.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.711+0000 7f6ab37fe640 1 -- 172.21.15.12:0/198895146 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4013690 con 0x7f6abc137d50 2024-09-11T01:42:20.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.711+0000 7f6ab37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:20.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.711+0000 7f6ab37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:42:20.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.711+0000 7f6ab37fe640 10 monclient: dump: 2024-09-11T01:42:20.726 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:20.726 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:20.726 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.726 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.727 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:20.727 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.727 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:20.727 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:20.727 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:20.727 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.711+0000 7f6ab97fa640 4 set_mon_vals no callback set 2024-09-11T01:42:20.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.711+0000 7f6ac12e6640 10 monclient: _renew_subs 2024-09-11T01:42:20.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.711+0000 7f6ac12e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:42:20.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.711+0000 7f6ac12e6640 1 -- 172.21.15.12:0/198895146 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6abc0a1710 con 0x7f6abc137d50 2024-09-11T01:42:20.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.711+0000 7f6ac12e6640 10 monclient: _renew_subs 2024-09-11T01:42:20.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.711+0000 7f6ac12e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:42:20.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.711+0000 7f6ac12e6640 1 -- 172.21.15.12:0/198895146 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6abc1273b0 con 0x7f6abc137d50 2024-09-11T01:42:20.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.711+0000 7f6ac12e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:20.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.711+0000 7f6ac12e6640 1 --2- 172.21.15.12:0/198895146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc1cdb60 0x7f6abc124010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.711+0000 7f6ac12e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:20.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.711+0000 7f6ac12e6640 1 -- 172.21.15.12:0/198895146 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6abc1379d0 con 0x7f6abc1cdb60 2024-09-11T01:42:20.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.712+0000 7f6abaffd640 1 --2- 172.21.15.12:0/198895146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc1cdb60 0x7f6abc124010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.712+0000 7f6abaffd640 10 monclient: get_auth_request con 0x7f6abc1cdb60 auth_method 0 2024-09-11T01:42:20.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.712+0000 7f6abaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:20.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.712+0000 7f6abaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:20.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.712+0000 7f6abaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:20.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.712+0000 7f6ab37fe640 1 -- 172.21.15.12:0/198895146 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f6aa4024060 con 0x7f6abc137d50 2024-09-11T01:42:20.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.712+0000 7f6abaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:20.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.712+0000 7f6abaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:20.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.712+0000 7f6abaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:20.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.712+0000 7f6abaffd640 10 monclient(hunting): handle_auth_done global_id 144137 payload 824 2024-09-11T01:42:20.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.712+0000 7f6abaffd640 1 -- 172.21.15.12:0/198895146 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6abc126bd0 con 0x7f6abc1cdb60 2024-09-11T01:42:20.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.712+0000 7f6ab37fe640 1 --2- 172.21.15.12:0/198895146 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6a9c03e210 0x7f6a9c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.712+0000 7f6ab37fe640 1 -- 172.21.15.12:0/198895146 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(442..442 src has 285..442) ==== 4787+0+0 (secure 0 0 0) 0x7f6aa4052d10 con 0x7f6abc137d50 2024-09-11T01:42:20.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.712+0000 7f6abb7fe640 1 --2- 172.21.15.12:0/198895146 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6a9c03e210 0x7f6a9c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.712+0000 7f6abb7fe640 10 monclient: get_auth_request con 0x7f6a9c03e210 auth_method 0 2024-09-11T01:42:20.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.714+0000 7f6abaffd640 1 -- 172.21.15.12:0/198895146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc1cdb60 msgr2=0x7f6abc124010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:20.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.714+0000 7f6abaffd640 1 --2- 172.21.15.12:0/198895146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc1cdb60 0x7f6abc124010 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6aac033000 tx=0x7f6aac004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:20.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.714+0000 7f6ab37fe640 1 -- 172.21.15.12:0/198895146 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aac042070 con 0x7f6abc1cdb60 2024-09-11T01:42:20.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.714+0000 7f6ab37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:20.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.714+0000 7f6ab37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:42:20.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.714+0000 7f6ab37fe640 10 monclient: dump: 2024-09-11T01:42:20.732 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:20.732 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:20.732 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.732 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.732 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:20.732 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.732 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:20.732 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:20.733 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:20.733 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.714+0000 7f6abb7fe640 1 --2- 172.21.15.12:0/198895146 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6a9c03e210 0x7f6a9c0406d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6abc125a20 tx=0x7f6ab4007db0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:20.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.714+0000 7f6ab37fe640 1 -- 172.21.15.12:0/198895146 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6aac037040 con 0x7f6abc1cdb60 2024-09-11T01:42:20.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.715+0000 7f6ab37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:20.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.715+0000 7f6ab37fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:42:20.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.715+0000 7f6ab37fe640 1 -- 172.21.15.12:0/198895146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc1cdb60 msgr2=0x7f6abc124010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:20.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.715+0000 7f6ab37fe640 1 --2- 172.21.15.12:0/198895146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc1cdb60 0x7f6abc124010 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6aac033000 tx=0x7f6aac004a40 comp rx=0 tx=0).stop 2024-09-11T01:42:20.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.715+0000 7f6ab37fe640 1 -- 172.21.15.12:0/198895146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc1cdb60 msgr2=0x7f6abc124010 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:42:20.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.715+0000 7f6ab37fe640 1 --2- 172.21.15.12:0/198895146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc1cdb60 0x7f6abc124010 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6aac033000 tx=0x7f6aac004a40 comp rx=0 tx=0).stop 2024-09-11T01:42:20.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.721+0000 7f6ac12e6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:42:20.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.721+0000 7f6ac12e6640 1 --2- 172.21.15.12:0/198895146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc124550 0x7f6abc0a1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.721+0000 7f6ac12e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:20.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.721+0000 7f6ac12e6640 1 -- 172.21.15.12:0/198895146 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6abc124960 con 0x7f6abc124550 2024-09-11T01:42:20.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.721+0000 7f6abaffd640 1 --2- 172.21.15.12:0/198895146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc124550 0x7f6abc0a1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.721+0000 7f6abaffd640 10 monclient: get_auth_request con 0x7f6abc124550 auth_method 0 2024-09-11T01:42:20.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.721+0000 7f6abaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:20.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.721+0000 7f6abaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:20.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.721+0000 7f6abaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:20.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.721+0000 7f6abaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:20.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.721+0000 7f6abaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:20.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.721+0000 7f6abaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:20.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.721+0000 7f6abaffd640 10 monclient(hunting): handle_auth_done global_id 144140 payload 824 2024-09-11T01:42:20.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.721+0000 7f6abaffd640 1 -- 172.21.15.12:0/198895146 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6abc1267e0 con 0x7f6abc124550 2024-09-11T01:42:20.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.723+0000 7f6abaffd640 1 -- 172.21.15.12:0/198895146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc124550 msgr2=0x7f6abc0a1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:20.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.723+0000 7f6abaffd640 1 --2- 172.21.15.12:0/198895146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc124550 0x7f6abc0a1cd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6aac000f80 tx=0x7f6aac02f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:20.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.724+0000 7f6ab37fe640 1 -- 172.21.15.12:0/198895146 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aac036040 con 0x7f6abc124550 2024-09-11T01:42:20.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.724+0000 7f6ab37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:20.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.724+0000 7f6ab37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:42:20.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.724+0000 7f6ab37fe640 10 monclient: dump: 2024-09-11T01:42:20.735 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:20.735 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:20.735 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.735 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.735 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:20.735 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.735 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:20.735 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:20.735 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:20.735 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.724+0000 7f6ab37fe640 1 -- 172.21.15.12:0/198895146 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7f6aac004240 con 0x7f6abc124550 2024-09-11T01:42:20.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.724+0000 7f6ab37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:42:20.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.724+0000 7f6ab37fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:42:20.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.724+0000 7f6ab37fe640 1 -- 172.21.15.12:0/198895146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc124550 msgr2=0x7f6abc0a1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:20.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.724+0000 7f6ab37fe640 1 --2- 172.21.15.12:0/198895146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc124550 0x7f6abc0a1cd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6aac000f80 tx=0x7f6aac02f000 comp rx=0 tx=0).stop 2024-09-11T01:42:20.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.724+0000 7f6ab37fe640 1 -- 172.21.15.12:0/198895146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc124550 msgr2=0x7f6abc0a1cd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:42:20.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.724+0000 7f6ab37fe640 1 --2- 172.21.15.12:0/198895146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc124550 0x7f6abc0a1cd0 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6aac000f80 tx=0x7f6aac02f000 comp rx=0 tx=0).stop 2024-09-11T01:42:20.736 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:42:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:42:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:42:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:42:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 122, 2024-09-11T01:42:20.736 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 12666, 2024-09-11T01:42:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:42:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:42:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:42:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:42:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 6, 2024-09-11T01:42:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:42:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:42:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:42:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:42:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:42:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:42:20.737 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:42:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:42:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:42:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:42:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:42:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:42:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:42:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:42:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:42:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:42:20.738 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:42:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:42:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:42:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:42:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:42:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:42:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:42:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:42:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:42:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:42:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:42:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.739 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:42:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:42:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:42:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:42:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:42:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:42:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:42:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:42:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:42:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:42:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:42:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:42:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:42:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:42:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:42:20.740 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:42:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:42:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:42:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:42:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:42:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:42:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:42:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:42:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:42:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:42:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:42:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:42:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:42:20.741 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:42:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:42:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:42:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:42:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:42:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:42:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:42:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:42:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.742 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:42:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:42:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:42:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:42:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:42:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:42:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:42:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.743 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:42:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:42:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:42:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:42:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:42:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:42:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:42:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:42:20.744 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:42:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:42:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:42:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:42:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:42:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:42:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:42:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:42:20.745 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:42:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:42:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:42:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:42:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:42:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:42:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:42:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:42:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:42:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:42:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:42:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:42:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.747 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:42:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:42:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:42:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:42:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:42:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:42:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:42:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.748 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:42:20.748 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:42:20.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.724+0000 7f6ac12e6640 1 -- 172.21.15.12:0/198895146 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6a9c03e210 msgr2=0x7f6a9c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:20.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.724+0000 7f6ac12e6640 1 --2- 172.21.15.12:0/198895146 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6a9c03e210 0x7f6a9c0406d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6abc125a20 tx=0x7f6ab4007db0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.724+0000 7f6ac12e6640 10 monclient: shutdown 2024-09-11T01:42:20.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.724+0000 7f6ac12e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:20.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.724+0000 7f6ac12e6640 1 -- 172.21.15.12:0/198895146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc137d50 msgr2=0x7f6abc123590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:20.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.724+0000 7f6ac12e6640 1 --2- 172.21.15.12:0/198895146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6abc137d50 0x7f6abc123590 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400ebe0 tx=0x7f6aa400c520 comp rx=0 tx=0).stop 2024-09-11T01:42:20.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.724+0000 7f6abb7fe640 1 -- 172.21.15.12:0/198895146 reap_dead start 2024-09-11T01:42:20.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.724+0000 7f6abb7fe640 1 -- 172.21.15.12:0/198895146 reap_dead start 2024-09-11T01:42:20.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.724+0000 7f6ac12e6640 1 -- 172.21.15.12:0/198895146 shutdown_connections 2024-09-11T01:42:20.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.724+0000 7f6ac12e6640 1 -- 172.21.15.12:0/198895146 >> 172.21.15.12:0/198895146 conn(0x7f6abc0a7020 msgr2=0x7f6abc0a0050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:20.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.724+0000 7f6ac12e6640 1 -- 172.21.15.12:0/198895146 shutdown_connections 2024-09-11T01:42:20.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.724+0000 7f6ac12e6640 1 -- 172.21.15.12:0/198895146 wait complete. 2024-09-11T01:42:20.751 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:42:20.751 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:42:20.751 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:42:20.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.850+0000 7f609ffb5640 10 monclient: get_monmap_and_config 2024-09-11T01:42:20.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.850+0000 7f609ffb5640 10 monclient: build_initial_monmap 2024-09-11T01:42:20.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.850+0000 7f609ffb5640 10 monclient: monmap: 2024-09-11T01:42:20.853 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:20.853 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:20.853 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:42:20.852599+0000 2024-09-11T01:42:20.853 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:42:20.852599+0000 2024-09-11T01:42:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:42:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:42:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:42:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:42:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:42:20.854 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.851+0000 7f609ffb5640 1 Processor -- start 2024-09-11T01:42:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.851+0000 7f609ffb5640 1 -- start start 2024-09-11T01:42:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.851+0000 7f609ffb5640 10 monclient: init 2024-09-11T01:42:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609ffb5640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609ffb5640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-11T01:42:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609ffb5640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6098074180 0x7f6098074560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609ffb5640 10 monclient(hunting): picked mon.noname-a con 0x7f6098074180 addr v2:172.21.15.12:3300/0 2024-09-11T01:42:20.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609ffb5640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f60980a06d0 0x7f60980a25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609ffb5640 10 monclient(hunting): picked mon.noname-c con 0x7f60980a06d0 addr v2:172.21.15.71:3300/0 2024-09-11T01:42:20.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609ffb5640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6098134ac0 0x7f60980a2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609ffb5640 10 monclient(hunting): picked mon.noname-b con 0x7f6098134ac0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:20.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609ffb5640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:20.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609ffb5640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f6098135560 con 0x7f6098074180 2024-09-11T01:42:20.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609ffb5640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:20.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609ffb5640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f60980a0490 con 0x7f60980a06d0 2024-09-11T01:42:20.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609ffb5640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:20.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609ffb5640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6098070740 con 0x7f6098134ac0 2024-09-11T01:42:20.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609ffb5640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:20.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609ffb5640 10 monclient(hunting): authenticate will time out at 1547.295190s 2024-09-11T01:42:20.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609dd51640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6098074180 0x7f6098074560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609e552640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6098134ac0 0x7f60980a2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609e552640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6098134ac0 0x7f60980a2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:40934/0 (socket says 172.21.15.12:40934) 2024-09-11T01:42:20.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609e552640 1 -- 172.21.15.12:0/312872491 learned_addr learned my addr 172.21.15.12:0/312872491 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:20.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609dd51640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6098074180 0x7f6098074560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52376/0 (socket says 172.21.15.12:52376) 2024-09-11T01:42:20.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609e552640 10 monclient(hunting): get_auth_request con 0x7f6098134ac0 auth_method 0 2024-09-11T01:42:20.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609e552640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:20.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609e552640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:20.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609e552640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:20.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609dd51640 10 monclient(hunting): get_auth_request con 0x7f6098074180 auth_method 0 2024-09-11T01:42:20.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609dd51640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:20.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609dd51640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:20.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609dd51640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:20.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609e552640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:20.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609e552640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:20.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609d550640 1 --2- 172.21.15.12:0/312872491 >> v2:172.21.15.71:3300/0 conn(0x7f60980a06d0 0x7f60980a25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609e552640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:20.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609dd51640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:20.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609dd51640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:20.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609dd51640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:20.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609d550640 10 monclient(hunting): get_auth_request con 0x7f60980a06d0 auth_method 0 2024-09-11T01:42:20.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609d550640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:20.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609d550640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:20.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609d550640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:20.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609d550640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:20.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609d550640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:20.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609d550640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:20.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609e552640 10 monclient(hunting): handle_auth_done global_id 144143 payload 274 2024-09-11T01:42:20.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609e552640 1 -- 172.21.15.12:0/312872491 >> v2:172.21.15.71:3300/0 conn(0x7f60980a06d0 msgr2=0x7f60980a25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:20.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609e552640 1 --2- 172.21.15.12:0/312872491 >> v2:172.21.15.71:3300/0 conn(0x7f60980a06d0 0x7f60980a25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609e552640 1 -- 172.21.15.12:0/312872491 >> v2:172.21.15.12:3300/0 conn(0x7f6098074180 msgr2=0x7f6098074560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:20.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609e552640 1 --2- 172.21.15.12:0/312872491 >> v2:172.21.15.12:3300/0 conn(0x7f6098074180 0x7f6098074560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609e552640 10 monclient: _finish_hunting 0 2024-09-11T01:42:20.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609e552640 1 monclient: found mon.noname-b 2024-09-11T01:42:20.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609e552640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:20.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609e552640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:42:20.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609e552640 1 -- 172.21.15.12:0/312872491 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6098074aa0 con 0x7f6098134ac0 2024-09-11T01:42:20.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.852+0000 7f609dd51640 1 --2- 172.21.15.12:0/312872491 >> v2:172.21.15.12:3300/0 conn(0x7f6098074180 0x7f6098074560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:42:20.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.855+0000 7f609e552640 1 -- 172.21.15.12:0/312872491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6098134ac0 msgr2=0x7f60980a2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:20.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.855+0000 7f609e552640 1 --2- 172.21.15.12:0/312872491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6098134ac0 0x7f60980a2b10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6098070bd0 tx=0x7f609002a570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e33f52a017997d7c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:20.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.855+0000 7f609cd4f640 1 -- 172.21.15.12:0/312872491 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6090036070 con 0x7f6098134ac0 2024-09-11T01:42:20.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.855+0000 7f609cd4f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:20.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.855+0000 7f609cd4f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:42:20.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.855+0000 7f609cd4f640 10 monclient: dump: 2024-09-11T01:42:20.869 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:20.869 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:20.869 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.869 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.869 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:20.869 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.869 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:20.869 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:20.869 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:20.869 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.855+0000 7f609cd4f640 10 monclient: _finish_auth 0 2024-09-11T01:42:20.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.855+0000 7f609cd4f640 10 monclient: _check_auth_tickets 2024-09-11T01:42:20.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.855+0000 7f609cd4f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:20.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.855+0000 7f609cd4f640 1 -- 172.21.15.12:0/312872491 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6090007800 con 0x7f6098134ac0 2024-09-11T01:42:20.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.855+0000 7f609ffb5640 5 monclient: authenticate success, global_id 144143 2024-09-11T01:42:20.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.855+0000 7f609ffb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:42:20.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.855+0000 7f609cd4f640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:20.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.855+0000 7f609ffb5640 10 monclient: get_monmap_and_config success 2024-09-11T01:42:20.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.855+0000 7f609cd4f640 1 -- 172.21.15.12:0/312872491 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6090007bd0 con 0x7f6098134ac0 2024-09-11T01:42:20.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.855+0000 7f609cd4f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:20.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.855+0000 7f609ffb5640 4 set_mon_vals no callback set 2024-09-11T01:42:20.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.855+0000 7f609cd4f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:42:20.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.855+0000 7f609cd4f640 10 monclient: dump: 2024-09-11T01:42:20.870 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:20.870 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:20.871 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.871 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.871 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:20.871 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.871 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:20.871 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:20.871 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:20.871 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.856+0000 7f609ffb5640 10 monclient: shutdown 2024-09-11T01:42:20.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.856+0000 7f609ffb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:20.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.856+0000 7f609ffb5640 1 -- 172.21.15.12:0/312872491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6098134ac0 msgr2=0x7f60980a2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:20.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.856+0000 7f609ffb5640 1 --2- 172.21.15.12:0/312872491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6098134ac0 0x7f60980a2b10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6098070bd0 tx=0x7f609002a570 comp rx=0 tx=0).stop 2024-09-11T01:42:20.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.856+0000 7f609ffb5640 1 -- 172.21.15.12:0/312872491 shutdown_connections 2024-09-11T01:42:20.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.856+0000 7f609ffb5640 1 --2- 172.21.15.12:0/312872491 >> v2:172.21.15.71:3300/0 conn(0x7f60980a06d0 0x7f60980a25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.856+0000 7f609ffb5640 1 --2- 172.21.15.12:0/312872491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6098134ac0 0x7f60980a2b10 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.856+0000 7f609ffb5640 1 --2- 172.21.15.12:0/312872491 >> v2:172.21.15.12:3300/0 conn(0x7f6098074180 0x7f6098074560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.856+0000 7f609ffb5640 1 -- 172.21.15.12:0/312872491 >> 172.21.15.12:0/312872491 conn(0x7f60980a36c0 msgr2=0x7f60980a3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:20.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.856+0000 7f609ffb5640 1 -- 172.21.15.12:0/312872491 shutdown_connections 2024-09-11T01:42:20.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.856+0000 7f609ffb5640 1 -- 172.21.15.12:0/312872491 wait complete. 2024-09-11T01:42:20.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.856+0000 7f609ffb5640 10 monclient: build_initial_monmap 2024-09-11T01:42:20.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.856+0000 7f609ffb5640 10 monclient: monmap: 2024-09-11T01:42:20.872 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:20.872 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:42:20.872 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:42:20.872 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:42:20.872 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:20.872 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.873 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:42:20.873 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:42:20.873 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:42:20.873 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609ffb5640 1 Processor -- start 2024-09-11T01:42:20.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609ffb5640 1 -- start start 2024-09-11T01:42:20.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609ffb5640 10 monclient: init 2024-09-11T01:42:20.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609ffb5640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:20.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609ffb5640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:42:20.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609ffb5640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6098074180 0x7f60981cce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609ffb5640 10 monclient(hunting): picked mon.noname-c con 0x7f6098074180 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:20.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609ffb5640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60980a06d0 0x7f6098131590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609ffb5640 10 monclient(hunting): picked mon.noname-b con 0x7f60980a06d0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:42:20.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609ffb5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6098134ac0 0x7f6098131ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609ffb5640 10 monclient(hunting): picked mon.noname-a con 0x7f6098134ac0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:42:20.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609ffb5640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:20.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609ffb5640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6098068d40 con 0x7f6098134ac0 2024-09-11T01:42:20.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609ffb5640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:20.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609ffb5640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6098068bc0 con 0x7f60980a06d0 2024-09-11T01:42:20.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609ffb5640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:20.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609ffb5640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f60980a3050 con 0x7f6098074180 2024-09-11T01:42:20.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609ffb5640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:20.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609ffb5640 10 monclient(hunting): authenticate will time out at 1547.300733s 2024-09-11T01:42:20.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609dd51640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6098074180 0x7f60981cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609dd51640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6098074180 0x7f60981cce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:40936/0 (socket says 172.21.15.12:40936) 2024-09-11T01:42:20.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609dd51640 1 -- 172.21.15.12:0/201932336 learned_addr learned my addr 172.21.15.12:0/201932336 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:20.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609dd51640 10 monclient(hunting): get_auth_request con 0x7f6098074180 auth_method 0 2024-09-11T01:42:20.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609dd51640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:20.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609dd51640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:20.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609dd51640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:20.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609e552640 1 --2- 172.21.15.12:0/201932336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6098134ac0 0x7f6098131ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609e552640 10 monclient(hunting): get_auth_request con 0x7f6098134ac0 auth_method 0 2024-09-11T01:42:20.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609e552640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:20.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609e552640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:20.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609e552640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:20.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609dd51640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:20.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609dd51640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:20.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609dd51640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:20.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609d550640 1 --2- 172.21.15.12:0/201932336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60980a06d0 0x7f6098131590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609d550640 10 monclient(hunting): get_auth_request con 0x7f60980a06d0 auth_method 0 2024-09-11T01:42:20.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609d550640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:20.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609d550640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:20.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609d550640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:20.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609e552640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:20.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609e552640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:20.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.857+0000 7f609e552640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:20.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.858+0000 7f609d550640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:20.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.858+0000 7f609d550640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:20.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.858+0000 7f609d550640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:20.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.858+0000 7f609dd51640 10 monclient(hunting): handle_auth_done global_id 144146 payload 824 2024-09-11T01:42:20.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.858+0000 7f609dd51640 1 -- 172.21.15.12:0/201932336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60980a06d0 msgr2=0x7f6098131590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:20.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.858+0000 7f609dd51640 1 --2- 172.21.15.12:0/201932336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60980a06d0 0x7f6098131590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.858+0000 7f609dd51640 1 -- 172.21.15.12:0/201932336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6098134ac0 msgr2=0x7f6098131ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:20.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.858+0000 7f609dd51640 1 --2- 172.21.15.12:0/201932336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6098134ac0 0x7f6098131ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.858+0000 7f609dd51640 10 monclient: _finish_hunting 0 2024-09-11T01:42:20.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.858+0000 7f609dd51640 1 monclient: found mon.noname-c 2024-09-11T01:42:20.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.858+0000 7f609dd51640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:20.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.858+0000 7f609dd51640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:42:20.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.858+0000 7f609dd51640 1 -- 172.21.15.12:0/201932336 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6098120c30 con 0x7f6098074180 2024-09-11T01:42:20.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.860+0000 7f609dd51640 1 -- 172.21.15.12:0/201932336 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6098074180 msgr2=0x7f60981cce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:20.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.860+0000 7f609dd51640 1 --2- 172.21.15.12:0/201932336 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6098074180 0x7f60981cce30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f609400ebe0 tx=0x7f609400c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:20.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.860+0000 7f608e7fc640 1 -- 172.21.15.12:0/201932336 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6094017070 con 0x7f6098074180 2024-09-11T01:42:20.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.860+0000 7f608e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:20.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.860+0000 7f608e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:42:20.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.860+0000 7f608e7fc640 10 monclient: dump: 2024-09-11T01:42:20.877 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:20.877 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:20.878 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.878 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.878 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:20.878 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.878 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:20.878 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:20.878 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:20.878 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.860+0000 7f608e7fc640 10 monclient: _finish_auth 0 2024-09-11T01:42:20.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.860+0000 7f608e7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:42:20.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.860+0000 7f608e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:20.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.860+0000 7f608e7fc640 1 -- 172.21.15.12:0/201932336 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f609400ce40 con 0x7f6098074180 2024-09-11T01:42:20.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.860+0000 7f609ffb5640 5 monclient: authenticate success, global_id 144146 2024-09-11T01:42:20.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.860+0000 7f609ffb5640 10 monclient: _renew_subs 2024-09-11T01:42:20.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.860+0000 7f609ffb5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:42:20.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.860+0000 7f609ffb5640 1 -- 172.21.15.12:0/201932336 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6098134070 con 0x7f6098074180 2024-09-11T01:42:20.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.860+0000 7f608e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:20.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.860+0000 7f608e7fc640 1 -- 172.21.15.12:0/201932336 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6094013690 con 0x7f6098074180 2024-09-11T01:42:20.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.860+0000 7f608e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:20.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.860+0000 7f608e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:42:20.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.860+0000 7f608e7fc640 10 monclient: dump: 2024-09-11T01:42:20.879 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:20.879 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:20.879 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.879 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.879 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:20.880 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.880 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:20.880 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:20.880 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:20.880 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.860+0000 7f609ffb5640 10 monclient: _renew_subs 2024-09-11T01:42:20.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.860+0000 7f609ffb5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:42:20.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.860+0000 7f609ffb5640 1 -- 172.21.15.12:0/201932336 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6098122f90 con 0x7f6098074180 2024-09-11T01:42:20.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.860+0000 7f608ffff640 4 set_mon_vals no callback set 2024-09-11T01:42:20.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.861+0000 7f609ffb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:20.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.861+0000 7f609ffb5640 1 --2- 172.21.15.12:0/201932336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f609811f1c0 0x7f609811f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.861+0000 7f609ffb5640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:20.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.861+0000 7f609ffb5640 1 -- 172.21.15.12:0/201932336 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6098074560 con 0x7f609811f1c0 2024-09-11T01:42:20.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.861+0000 7f609d550640 1 --2- 172.21.15.12:0/201932336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f609811f1c0 0x7f609811f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.861+0000 7f608e7fc640 1 -- 172.21.15.12:0/201932336 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f6094024060 con 0x7f6098074180 2024-09-11T01:42:20.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.861+0000 7f609d550640 10 monclient: get_auth_request con 0x7f609811f1c0 auth_method 0 2024-09-11T01:42:20.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.861+0000 7f609d550640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:20.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.861+0000 7f609d550640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:20.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.861+0000 7f609d550640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:20.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.861+0000 7f609d550640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:20.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.861+0000 7f609d550640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:20.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.861+0000 7f609d550640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:20.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.861+0000 7f609d550640 10 monclient(hunting): handle_auth_done global_id 134221 payload 824 2024-09-11T01:42:20.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.861+0000 7f608e7fc640 1 --2- 172.21.15.12:0/201932336 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f607c03e210 0x7f607c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.861+0000 7f609d550640 1 -- 172.21.15.12:0/201932336 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6098121790 con 0x7f609811f1c0 2024-09-11T01:42:20.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.861+0000 7f608e7fc640 1 -- 172.21.15.12:0/201932336 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(442..442 src has 285..442) ==== 4787+0+0 (secure 0 0 0) 0x7f6094052cf0 con 0x7f6098074180 2024-09-11T01:42:20.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.862+0000 7f609e552640 1 --2- 172.21.15.12:0/201932336 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f607c03e210 0x7f607c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.862+0000 7f609e552640 10 monclient: get_auth_request con 0x7f607c03e210 auth_method 0 2024-09-11T01:42:20.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.864+0000 7f609d550640 1 -- 172.21.15.12:0/201932336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f609811f1c0 msgr2=0x7f609811f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:20.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.864+0000 7f609d550640 1 --2- 172.21.15.12:0/201932336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f609811f1c0 0x7f609811f5c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6098121a10 tx=0x7f6088011cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:20.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.864+0000 7f608e7fc640 1 -- 172.21.15.12:0/201932336 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6088019070 con 0x7f609811f1c0 2024-09-11T01:42:20.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.864+0000 7f608e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:20.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.864+0000 7f608e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:42:20.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.864+0000 7f608e7fc640 10 monclient: dump: 2024-09-11T01:42:20.882 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:20.882 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:20.882 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.882 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.882 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:20.883 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.883 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:20.883 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:20.883 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:20.883 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.864+0000 7f609e552640 1 --2- 172.21.15.12:0/201932336 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f607c03e210 0x7f607c0406d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f609806ef60 tx=0x7f6090009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:20.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.864+0000 7f608e7fc640 1 -- 172.21.15.12:0/201932336 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f608800f860 con 0x7f609811f1c0 2024-09-11T01:42:20.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.864+0000 7f608e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:20.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.864+0000 7f608e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:42:20.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.864+0000 7f608e7fc640 1 -- 172.21.15.12:0/201932336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f609811f1c0 msgr2=0x7f609811f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:20.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.864+0000 7f608e7fc640 1 --2- 172.21.15.12:0/201932336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f609811f1c0 0x7f609811f5c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6098121a10 tx=0x7f6088011cf0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.864+0000 7f608e7fc640 1 -- 172.21.15.12:0/201932336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f609811f1c0 msgr2=0x7f609811f5c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:42:20.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.864+0000 7f608e7fc640 1 --2- 172.21.15.12:0/201932336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f609811f1c0 0x7f609811f5c0 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6098121a10 tx=0x7f6088011cf0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.870+0000 7f609ffb5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:42:20.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.870+0000 7f609ffb5640 1 --2- 172.21.15.12:0/201932336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f609811fb00 0x7f6098071670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:20.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.870+0000 7f609ffb5640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:20.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.870+0000 7f609ffb5640 1 -- 172.21.15.12:0/201932336 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6098122b60 con 0x7f609811fb00 2024-09-11T01:42:20.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.870+0000 7f609d550640 1 --2- 172.21.15.12:0/201932336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f609811fb00 0x7f6098071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:20.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.870+0000 7f609d550640 10 monclient: get_auth_request con 0x7f609811fb00 auth_method 0 2024-09-11T01:42:20.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.870+0000 7f609d550640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:20.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.870+0000 7f609d550640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:20.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.870+0000 7f609d550640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:20.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.871+0000 7f609d550640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:20.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.871+0000 7f609d550640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:20.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.871+0000 7f609d550640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:20.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.871+0000 7f609d550640 10 monclient(hunting): handle_auth_done global_id 134224 payload 824 2024-09-11T01:42:20.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.871+0000 7f609d550640 1 -- 172.21.15.12:0/201932336 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f609811ff30 con 0x7f609811fb00 2024-09-11T01:42:20.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.873+0000 7f609d550640 1 -- 172.21.15.12:0/201932336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f609811fb00 msgr2=0x7f6098071670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:20.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.873+0000 7f609d550640 1 --2- 172.21.15.12:0/201932336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f609811fb00 0x7f6098071670 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6098069d70 tx=0x7f60880046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:20.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.873+0000 7f608e7fc640 1 -- 172.21.15.12:0/201932336 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6088019040 con 0x7f609811fb00 2024-09-11T01:42:20.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.873+0000 7f608e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:20.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.873+0000 7f608e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:42:20.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.873+0000 7f608e7fc640 10 monclient: dump: 2024-09-11T01:42:20.887 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:20.887 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:20.887 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.887 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:20.887 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:20.887 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:20.887 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:20.887 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:20.887 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:20.887 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:20.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.873+0000 7f608e7fc640 1 -- 172.21.15.12:0/201932336 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7f6088003e50 con 0x7f609811fb00 2024-09-11T01:42:20.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.873+0000 7f608e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:42:20.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.873+0000 7f608e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:42:20.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.873+0000 7f608e7fc640 1 -- 172.21.15.12:0/201932336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f609811fb00 msgr2=0x7f6098071670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:20.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.873+0000 7f608e7fc640 1 --2- 172.21.15.12:0/201932336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f609811fb00 0x7f6098071670 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6098069d70 tx=0x7f60880046c0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.873+0000 7f608e7fc640 1 -- 172.21.15.12:0/201932336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f609811fb00 msgr2=0x7f6098071670 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:42:20.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.873+0000 7f608e7fc640 1 --2- 172.21.15.12:0/201932336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f609811fb00 0x7f6098071670 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6098069d70 tx=0x7f60880046c0 comp rx=0 tx=0).stop 2024-09-11T01:42:20.888 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:42:20.888 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:42:20.888 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:42:20.888 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:42:20.888 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 122, 2024-09-11T01:42:20.888 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 73836, 2024-09-11T01:42:20.888 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:42:20.889 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:42:20.889 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:42:20.889 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:42:20.889 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:20.889 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 7, 2024-09-11T01:42:20.889 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:42:20.889 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:42:20.889 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:42:20.889 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:42:20.889 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:42:20.889 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:42:20.889 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:42:20.889 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:42:20.890 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:42:20.890 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:42:20.890 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:42:20.890 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:42:20.890 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:42:20.890 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:42:20.890 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:20.890 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:42:20.890 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:42:20.890 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:42:20.890 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:42:20.890 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:42:20.890 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:42:20.890 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:42:20.891 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:42:20.891 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:42:20.891 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:42:20.891 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:42:20.891 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:42:20.891 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:42:20.891 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:20.891 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.891 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:42:20.891 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:42:20.891 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:42:20.891 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:42:20.891 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:42:20.891 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:42:20.891 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:42:20.892 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:42:20.892 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:42:20.892 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:42:20.892 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:42:20.892 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:42:20.892 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:42:20.892 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:42:20.892 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:42:20.892 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:42:20.892 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:42:20.892 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:42:20.892 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:42:20.892 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:42:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:42:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:42:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:42:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:42:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:42:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:42:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:42:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:42:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:42:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:42:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:42:20.893 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:42:20.894 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:42:20.894 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:42:20.894 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.894 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:42:20.894 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:42:20.894 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:20.894 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.894 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.894 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:42:20.894 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:42:20.894 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:20.894 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:20.894 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:20.895 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.895 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:42:20.895 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:42:20.895 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:42:20.895 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:42:20.895 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:42:20.895 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.895 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.895 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:42:20.895 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:42:20.895 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:42:20.895 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:42:20.895 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.895 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:42:20.896 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:42:20.896 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:20.896 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.896 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.896 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:42:20.896 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:42:20.896 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:20.896 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:20.896 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:20.896 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.896 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:42:20.896 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:42:20.896 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:42:20.896 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:42:20.897 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:42:20.897 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.897 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.897 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:42:20.897 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:42:20.897 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:42:20.897 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:42:20.897 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.897 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:42:20.897 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:42:20.897 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:20.897 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.897 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.897 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:42:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:42:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:42:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:42:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:42:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:42:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:42:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:42:20.898 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:42:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:42:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:42:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:42:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:42:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:42:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:42:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:42:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:42:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:20.899 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:42:20.900 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:42:20.900 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:42:20.900 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:20.900 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:20.900 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:20.900 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:42:20.900 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:42:20.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.874+0000 7f609ffb5640 1 -- 172.21.15.12:0/201932336 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f607c03e210 msgr2=0x7f607c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:20.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.874+0000 7f609ffb5640 1 --2- 172.21.15.12:0/201932336 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f607c03e210 0x7f607c0406d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f609806ef60 tx=0x7f6090009000 comp rx=0 tx=0).stop 2024-09-11T01:42:20.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.874+0000 7f609ffb5640 10 monclient: shutdown 2024-09-11T01:42:20.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.874+0000 7f609ffb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:20.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.874+0000 7f609ffb5640 1 -- 172.21.15.12:0/201932336 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6098074180 msgr2=0x7f60981cce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:20.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.874+0000 7f609ffb5640 1 --2- 172.21.15.12:0/201932336 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6098074180 0x7f60981cce30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f609400ebe0 tx=0x7f609400c520 comp rx=0 tx=0).stop 2024-09-11T01:42:20.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.874+0000 7f609e552640 1 -- 172.21.15.12:0/201932336 reap_dead start 2024-09-11T01:42:20.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.874+0000 7f609e552640 1 -- 172.21.15.12:0/201932336 reap_dead start 2024-09-11T01:42:20.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.874+0000 7f609ffb5640 1 -- 172.21.15.12:0/201932336 shutdown_connections 2024-09-11T01:42:20.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.874+0000 7f609ffb5640 1 -- 172.21.15.12:0/201932336 >> 172.21.15.12:0/201932336 conn(0x7f60980a36c0 msgr2=0x7f6098073170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:20.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.874+0000 7f609ffb5640 1 -- 172.21.15.12:0/201932336 shutdown_connections 2024-09-11T01:42:20.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:20.874+0000 7f609ffb5640 1 -- 172.21.15.12:0/201932336 wait complete. 2024-09-11T01:42:20.902 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:42:20.903 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-11T01:42:20.903 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T01:42:20.903 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-11T01:42:20.903 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-11T01:42:20.903 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-11T01:42:20.903 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:42:20.913 INFO:tasks.ceph.mon.c:Stopped 2024-09-11T01:42:20.913 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:42:20.914 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:42:21.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.050+0000 7f6c7ff2b640 10 monclient: get_monmap_and_config 2024-09-11T01:42:21.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.050+0000 7f6c7ff2b640 10 monclient: build_initial_monmap 2024-09-11T01:42:21.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.050+0000 7f6c7ff2b640 10 monclient: monmap: 2024-09-11T01:42:21.053 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:21.053 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:21.053 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:42:21.052686+0000 2024-09-11T01:42:21.053 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:42:21.052686+0000 2024-09-11T01:42:21.054 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:21.054 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:21.054 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:42:21.054 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:42:21.054 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:42:21.054 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:42:21.054 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:42:21.054 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:21.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.051+0000 7f6c7ff2b640 1 Processor -- start 2024-09-11T01:42:21.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.051+0000 7f6c7ff2b640 1 -- start start 2024-09-11T01:42:21.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.051+0000 7f6c7ff2b640 10 monclient: init 2024-09-11T01:42:21.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.052+0000 7f6c7ff2b640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:21.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.052+0000 7f6c7ff2b640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-11T01:42:21.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.052+0000 7f6c7ff2b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6c780a2e90 0x7f6c7806dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:21.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.052+0000 7f6c7ff2b640 10 monclient(hunting): picked mon.noname-b con 0x7f6c780a2e90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:21.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.052+0000 7f6c7ff2b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6c780a3270 addr v1:172.21.15.71:6789/0 2024-09-11T01:42:21.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.052+0000 7f6c7ff2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6c7813d490 addr v1:172.21.15.12:6789/0 2024-09-11T01:42:21.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.052+0000 7f6c7ff2b640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c780a46c0 con 0x7f6c7813d490 2024-09-11T01:42:21.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.052+0000 7f6c7ff2b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c7806ee50 con 0x7f6c780a3270 2024-09-11T01:42:21.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.052+0000 7f6c7ff2b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:21.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.052+0000 7f6c7ff2b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6c780a4900 con 0x7f6c780a2e90 2024-09-11T01:42:21.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.052+0000 7f6c7ff2b640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:21.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.052+0000 7f6c7ff2b640 10 monclient(hunting): authenticate will time out at 1547.495297s 2024-09-11T01:42:21.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.052+0000 7f6c7dcc7640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6c780a2e90 msgr2=0x7f6c7806dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:42:21.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.052+0000 7f6c7dcc7640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6c780a2e90 0x7f6c7806dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:21.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.052+0000 7f6c7e4c8640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6c7813d490 0x7f6c7813f880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:48980/0 (socket says 172.21.15.12:48980) 2024-09-11T01:42:21.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.052+0000 7f6c7e4c8640 1 -- 172.21.15.12:0/1349598636 learned_addr learned my addr 172.21.15.12:0/1349598636 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:42:21.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.054+0000 7f6c7e4c8640 1 -- 172.21.15.12:0/1349598636 >> v1:172.21.15.12:6789/0 conn(0x7f6c7813d490 legacy=0x7f6c7813f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:21.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.054+0000 7f6c7d4c6640 1 -- 172.21.15.12:0/1349598636 >> v1:172.21.15.71:6789/0 conn(0x7f6c780a3270 legacy=0x7f6c780a3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:21.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.054+0000 7f6c7ccc5640 1 -- 172.21.15.12:0/1349598636 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3008637158 0 0) 0x7f6c780a46c0 con 0x7f6c7813d490 2024-09-11T01:42:21.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 10 monclient(hunting): _negotiate 2024-09-11T01:42:21.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:21.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:21.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 10 monclient(hunting): my global_id is 94605 2024-09-11T01:42:21.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 1 -- 172.21.15.12:0/1349598636 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c64003520 con 0x7f6c7813d490 2024-09-11T01:42:21.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 1 -- 172.21.15.12:0/1349598636 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 56661544 0 0) 0x7f6c7806ee50 con 0x7f6c780a3270 2024-09-11T01:42:21.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 10 monclient(hunting): _negotiate 2024-09-11T01:42:21.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:21.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:21.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 10 monclient(hunting): my global_id is 134227 2024-09-11T01:42:21.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 1 -- 172.21.15.12:0/1349598636 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c780a46c0 con 0x7f6c780a3270 2024-09-11T01:42:21.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 1 -- 172.21.15.12:0/1349598636 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1435528845 0 0) 0x7f6c64003520 con 0x7f6c7813d490 2024-09-11T01:42:21.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 1 -- 172.21.15.12:0/1349598636 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6c780a2e90 msgr2=0x7f6c7806dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:42:21.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 1 --2- 172.21.15.12:0/1349598636 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6c780a2e90 0x7f6c7806dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:21.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 1 -- 172.21.15.12:0/1349598636 >> v1:172.21.15.71:6789/0 conn(0x7f6c780a3270 legacy=0x7f6c780a3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:21.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 10 monclient: _finish_hunting 0 2024-09-11T01:42:21.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:42:21.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:21.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:42:21.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 1 -- 172.21.15.12:0/1349598636 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c7806eb70 con 0x7f6c7813d490 2024-09-11T01:42:21.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 10 monclient: _finish_auth 0 2024-09-11T01:42:21.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 10 monclient: _check_auth_tickets 2024-09-11T01:42:21.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:21.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ff2b640 5 monclient: authenticate success, global_id 94605 2024-09-11T01:42:21.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 1 -- 172.21.15.12:0/1349598636 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f6c700019a0 con 0x7f6c7813d490 2024-09-11T01:42:21.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ff2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:42:21.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:21.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:42:21.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 10 monclient: dump: 2024-09-11T01:42:21.064 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:21.064 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:21.064 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:21.065 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:42:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 1 -- 172.21.15.12:0/1349598636 >> v1:172.21.15.12:6789/0 conn(0x7f6c7813d490 legacy=0x7f6c7813f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:42:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 1 --2- 172.21.15.12:0/1349598636 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6c780a2e90 0x7f6c64003bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 10 monclient(hunting): picked mon.c con 0x7f6c780a2e90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:21.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 1 --2- 172.21.15.12:0/1349598636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c640041a0 0x7f6c64004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:21.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 10 monclient(hunting): picked mon.a con 0x7f6c640041a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:42:21.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 1 --2- 172.21.15.12:0/1349598636 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c64004f40 0x7f6c6400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:21.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7dcc7640 1 -- 172.21.15.12:0/1349598636 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6c780a2e90 msgr2=0x7f6c64003bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:42:21.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 10 monclient(hunting): picked mon.b con 0x7f6c64004f40 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:42:21.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:21.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 1 -- 172.21.15.12:0/1349598636 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6c780a4900 con 0x7f6c640041a0 2024-09-11T01:42:21.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:21.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 1 -- 172.21.15.12:0/1349598636 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6c6400bf00 con 0x7f6c64004f40 2024-09-11T01:42:21.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:21.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7dcc7640 1 --2- 172.21.15.12:0/1349598636 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6c780a2e90 0x7f6c64003bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:21.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 1 -- 172.21.15.12:0/1349598636 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6c6400c080 con 0x7f6c780a2e90 2024-09-11T01:42:21.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:21.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ccc5640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:42:21.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7ff2b640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:42:21.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7d4c6640 1 --2- 172.21.15.12:0/1349598636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c640041a0 0x7f6c64004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:21.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7d4c6640 10 monclient(hunting): get_auth_request con 0x7f6c640041a0 auth_method 0 2024-09-11T01:42:21.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7d4c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:21.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7d4c6640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:21.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7d4c6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:42:21.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7e4c8640 1 --2- 172.21.15.12:0/1349598636 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c64004f40 0x7f6c6400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:21.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7d4c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:21.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7d4c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:21.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7d4c6640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:42:21.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7e4c8640 10 monclient(hunting): get_auth_request con 0x7f6c64004f40 auth_method 0 2024-09-11T01:42:21.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7e4c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:21.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7e4c8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:21.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7e4c8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:42:21.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7e4c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:21.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7e4c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:21.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7e4c8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:42:21.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7d4c6640 10 monclient(hunting): handle_auth_done global_id 94605 payload 293 2024-09-11T01:42:21.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7d4c6640 1 -- 172.21.15.12:0/1349598636 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6c780a2e90 msgr2=0x7f6c64003bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:42:21.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7d4c6640 1 --2- 172.21.15.12:0/1349598636 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6c780a2e90 0x7f6c64003bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:21.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7d4c6640 1 -- 172.21.15.12:0/1349598636 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c64004f40 msgr2=0x7f6c6400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:21.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7d4c6640 1 --2- 172.21.15.12:0/1349598636 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c64004f40 0x7f6c6400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:21.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7d4c6640 10 monclient: _finish_hunting 0 2024-09-11T01:42:21.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7d4c6640 1 monclient: found mon.a 2024-09-11T01:42:21.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7d4c6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:21.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7d4c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:21.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7d4c6640 1 -- 172.21.15.12:0/1349598636 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c64003520 con 0x7f6c640041a0 2024-09-11T01:42:21.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7d4c6640 10 monclient: _finish_auth 0 2024-09-11T01:42:21.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7d4c6640 10 monclient: _check_auth_tickets 2024-09-11T01:42:21.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.055+0000 7f6c7d4c6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:21.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.058+0000 7f6c7d4c6640 1 -- 172.21.15.12:0/1349598636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c640041a0 msgr2=0x7f6c64004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:21.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.058+0000 7f6c7d4c6640 1 --2- 172.21.15.12:0/1349598636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c640041a0 0x7f6c64004800 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6c68001bf0 tx=0x7f6c64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=206ade680eb4269d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:21.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.058+0000 7f6c7ccc5640 1 -- 172.21.15.12:0/1349598636 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c68007270 con 0x7f6c640041a0 2024-09-11T01:42:21.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.058+0000 7f6c7ccc5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:21.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.058+0000 7f6c7ccc5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:21.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.058+0000 7f6c7ccc5640 10 monclient: dump: 2024-09-11T01:42:21.069 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:21.069 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:21.069 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:21.070 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:21.070 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:21.070 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:21.070 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:21.070 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:21.070 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:21.070 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:21.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.058+0000 7f6c7ccc5640 1 -- 172.21.15.12:0/1349598636 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c68007e50 con 0x7f6c640041a0 2024-09-11T01:42:21.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.058+0000 7f6c7ff2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:42:21.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.059+0000 7f6c7ccc5640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:21.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.059+0000 7f6c7ff2b640 10 monclient: get_monmap_and_config success 2024-09-11T01:42:21.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.059+0000 7f6c7ccc5640 1 -- 172.21.15.12:0/1349598636 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c68007670 con 0x7f6c640041a0 2024-09-11T01:42:21.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.059+0000 7f6c7ccc5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:21.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.059+0000 7f6c7ff2b640 4 set_mon_vals no callback set 2024-09-11T01:42:21.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.059+0000 7f6c7ccc5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:21.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.059+0000 7f6c7ccc5640 10 monclient: dump: 2024-09-11T01:42:21.071 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:21.071 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:21.071 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:21.071 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:21.071 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:21.071 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:21.071 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:21.071 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:21.071 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:21.071 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:21.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.059+0000 7f6c7ff2b640 10 monclient: shutdown 2024-09-11T01:42:21.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.059+0000 7f6c7ff2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:21.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.059+0000 7f6c7ff2b640 1 -- 172.21.15.12:0/1349598636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c640041a0 msgr2=0x7f6c64004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:21.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.059+0000 7f6c7ff2b640 1 --2- 172.21.15.12:0/1349598636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c640041a0 0x7f6c64004800 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6c68001bf0 tx=0x7f6c64005830 comp rx=0 tx=0).stop 2024-09-11T01:42:21.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.059+0000 7f6c7e4c8640 1 -- 172.21.15.12:0/1349598636 reap_dead start 2024-09-11T01:42:21.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.059+0000 7f6c7ff2b640 1 -- 172.21.15.12:0/1349598636 shutdown_connections 2024-09-11T01:42:21.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.059+0000 7f6c7ff2b640 1 -- 172.21.15.12:0/1349598636 >> 172.21.15.12:0/1349598636 conn(0x7f6c780a8490 msgr2=0x7f6c780a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:21.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.059+0000 7f6c7ff2b640 1 -- 172.21.15.12:0/1349598636 shutdown_connections 2024-09-11T01:42:21.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.059+0000 7f6c7ff2b640 1 -- 172.21.15.12:0/1349598636 wait complete. 2024-09-11T01:42:21.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.060+0000 7f6c7ff2b640 10 monclient: build_initial_monmap 2024-09-11T01:42:21.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.060+0000 7f6c7ff2b640 10 monclient: monmap: 2024-09-11T01:42:21.072 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:21.072 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:42:21.072 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:42:21.072 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:42:21.072 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:21.073 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:21.073 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:42:21.073 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:42:21.073 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:42:21.073 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:21.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.060+0000 7f6c7ff2b640 1 Processor -- start 2024-09-11T01:42:21.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.060+0000 7f6c7ff2b640 1 -- start start 2024-09-11T01:42:21.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.060+0000 7f6c7ff2b640 10 monclient: init 2024-09-11T01:42:21.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7ff2b640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:21.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7ff2b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:42:21.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7ff2b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c640041a0 0x7f6c781c9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:21.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7ff2b640 10 monclient(hunting): picked mon.noname-a con 0x7f6c640041a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:42:21.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7ff2b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6c78138560 0x7f6c781389c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:21.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7ff2b640 10 monclient(hunting): picked mon.noname-c con 0x7f6c78138560 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:21.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7ff2b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c781ccbc0 0x7f6c781cd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:21.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7ff2b640 10 monclient(hunting): picked mon.noname-b con 0x7f6c781ccbc0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:42:21.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7ff2b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:21.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7ff2b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7806dde0 con 0x7f6c640041a0 2024-09-11T01:42:21.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7ff2b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:21.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7ff2b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7806dc60 con 0x7f6c781ccbc0 2024-09-11T01:42:21.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7ff2b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:21.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7ff2b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6c7806df60 con 0x7f6c78138560 2024-09-11T01:42:21.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7ff2b640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:21.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7ff2b640 10 monclient(hunting): authenticate will time out at 1547.504190s 2024-09-11T01:42:21.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7d4c6640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6c78138560 msgr2=0x7f6c781389c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:42:21.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7d4c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6c78138560 0x7f6c781389c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:21.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7dcc7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c640041a0 0x7f6c781c9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:21.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7dcc7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c640041a0 0x7f6c781c9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52414/0 (socket says 172.21.15.12:52414) 2024-09-11T01:42:21.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7dcc7640 1 -- 172.21.15.12:0/930186940 learned_addr learned my addr 172.21.15.12:0/930186940 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:21.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7dcc7640 10 monclient(hunting): get_auth_request con 0x7f6c640041a0 auth_method 0 2024-09-11T01:42:21.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7dcc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:21.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7dcc7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:21.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7dcc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:21.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7dcc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:21.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7dcc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:21.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7dcc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:21.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7e4c8640 1 --2- 172.21.15.12:0/930186940 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c781ccbc0 0x7f6c781cd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:21.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7e4c8640 10 monclient(hunting): get_auth_request con 0x7f6c781ccbc0 auth_method 0 2024-09-11T01:42:21.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7e4c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:21.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7e4c8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:21.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7e4c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:21.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7e4c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:21.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7e4c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:21.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7e4c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:21.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7dcc7640 10 monclient(hunting): handle_auth_done global_id 94608 payload 824 2024-09-11T01:42:21.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7dcc7640 1 -- 172.21.15.12:0/930186940 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6c78138560 msgr2=0x7f6c781389c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:42:21.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7dcc7640 1 --2- 172.21.15.12:0/930186940 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6c78138560 0x7f6c781389c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:21.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7dcc7640 1 -- 172.21.15.12:0/930186940 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c781ccbc0 msgr2=0x7f6c781cd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:21.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7dcc7640 1 --2- 172.21.15.12:0/930186940 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c781ccbc0 0x7f6c781cd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:21.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7dcc7640 10 monclient: _finish_hunting 0 2024-09-11T01:42:21.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7dcc7640 1 monclient: found mon.noname-a 2024-09-11T01:42:21.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7dcc7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:21.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7dcc7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:42:21.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.061+0000 7f6c7dcc7640 1 -- 172.21.15.12:0/930186940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c78124030 con 0x7f6c640041a0 2024-09-11T01:42:21.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.063+0000 7f6c7dcc7640 1 -- 172.21.15.12:0/930186940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c640041a0 msgr2=0x7f6c781c9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:21.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.063+0000 7f6c7dcc7640 1 --2- 172.21.15.12:0/930186940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c640041a0 0x7f6c781c9660 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6c7400ba10 tx=0x7f6c7400be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:21.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.063+0000 7f6c6e7fc640 1 -- 172.21.15.12:0/930186940 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c7400cd30 con 0x7f6c640041a0 2024-09-11T01:42:21.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.063+0000 7f6c6e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:21.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.063+0000 7f6c6e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:42:21.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.063+0000 7f6c6e7fc640 10 monclient: dump: 2024-09-11T01:42:21.081 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:21.082 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:21.082 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:21.082 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:21.082 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:21.082 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:21.082 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:21.082 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:21.082 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:21.082 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:21.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.063+0000 7f6c6e7fc640 10 monclient: _finish_auth 0 2024-09-11T01:42:21.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.063+0000 7f6c6e7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:42:21.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.063+0000 7f6c6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:21.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.063+0000 7f6c6e7fc640 1 -- 172.21.15.12:0/930186940 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c74010070 con 0x7f6c640041a0 2024-09-11T01:42:21.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.063+0000 7f6c6e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:21.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.063+0000 7f6c6e7fc640 1 -- 172.21.15.12:0/930186940 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c740156b0 con 0x7f6c640041a0 2024-09-11T01:42:21.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.063+0000 7f6c7ff2b640 5 monclient: authenticate success, global_id 94608 2024-09-11T01:42:21.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.063+0000 7f6c7ff2b640 10 monclient: _renew_subs 2024-09-11T01:42:21.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.063+0000 7f6c7ff2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:21.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.063+0000 7f6c7ff2b640 1 -- 172.21.15.12:0/930186940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c78124770 con 0x7f6c640041a0 2024-09-11T01:42:21.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.063+0000 7f6c6e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:21.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.063+0000 7f6c6e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:21.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.063+0000 7f6c6ffff640 4 set_mon_vals no callback set 2024-09-11T01:42:21.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.063+0000 7f6c6e7fc640 10 monclient: dump: 2024-09-11T01:42:21.083 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:21.083 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:21.083 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:21.083 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:21.083 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:21.084 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:21.084 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:21.084 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:21.084 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:21.084 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:21.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.064+0000 7f6c7ff2b640 10 monclient: _renew_subs 2024-09-11T01:42:21.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.064+0000 7f6c7ff2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:21.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.064+0000 7f6c7ff2b640 1 -- 172.21.15.12:0/930186940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c7809f360 con 0x7f6c640041a0 2024-09-11T01:42:21.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.064+0000 7f6c7ff2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:21.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.064+0000 7f6c7ff2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:21.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.064+0000 7f6c7ff2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:21.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.064+0000 7f6c6e7fc640 1 -- 172.21.15.12:0/930186940 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f6c740050f0 con 0x7f6c640041a0 2024-09-11T01:42:21.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.064+0000 7f6c7ff2b640 1 -- 172.21.15.12:0/930186940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c780a3650 con 0x7f6c640041a0 2024-09-11T01:42:21.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.065+0000 7f6c6e7fc640 1 --2- 172.21.15.12:0/930186940 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6c5c03e240 0x7f6c5c040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:21.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.065+0000 7f6c6e7fc640 1 -- 172.21.15.12:0/930186940 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(442..442 src has 285..442) ==== 4787+0+0 (secure 0 0 0) 0x7f6c74053160 con 0x7f6c640041a0 2024-09-11T01:42:21.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.065+0000 7f6c7d4c6640 1 --2- 172.21.15.12:0/930186940 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6c5c03e240 0x7f6c5c040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:21.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.065+0000 7f6c7d4c6640 10 monclient: get_auth_request con 0x7f6c5c03e240 auth_method 0 2024-09-11T01:42:21.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.067+0000 7f6c7d4c6640 1 --2- 172.21.15.12:0/930186940 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6c5c03e240 0x7f6c5c040700 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6c68032000 tx=0x7f6c6802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:21.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.069+0000 7f6c6e7fc640 1 -- 172.21.15.12:0/930186940 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6c7401a060 con 0x7f6c640041a0 2024-09-11T01:42:21.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.069+0000 7f6c6e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:21.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.069+0000 7f6c6e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:42:21.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.327+0000 7f6c7ff2b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:42:21.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.327+0000 7f6c7ff2b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:42:21.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.327+0000 7f6c7ff2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:21.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.327+0000 7f6c7ff2b640 1 -- 172.21.15.12:0/930186940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6c780698e0 con 0x7f6c640041a0 2024-09-11T01:42:21.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.327+0000 7f6c6e7fc640 1 -- 172.21.15.12:0/930186940 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f6c74051b50 con 0x7f6c640041a0 2024-09-11T01:42:21.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.327+0000 7f6c6e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:42:21.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.327+0000 7f6c6e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:42:21.331 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":122,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:42:21.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.330+0000 7f6c7ff2b640 1 -- 172.21.15.12:0/930186940 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6c5c03e240 msgr2=0x7f6c5c040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:21.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.330+0000 7f6c7ff2b640 1 --2- 172.21.15.12:0/930186940 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6c5c03e240 0x7f6c5c040700 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6c68032000 tx=0x7f6c6802e000 comp rx=0 tx=0).stop 2024-09-11T01:42:21.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.330+0000 7f6c7ff2b640 10 monclient: shutdown 2024-09-11T01:42:21.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.330+0000 7f6c7ff2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:21.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.330+0000 7f6c7ff2b640 1 -- 172.21.15.12:0/930186940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c640041a0 msgr2=0x7f6c781c9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:21.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.330+0000 7f6c7ff2b640 1 --2- 172.21.15.12:0/930186940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c640041a0 0x7f6c781c9660 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6c7400ba10 tx=0x7f6c7400be10 comp rx=0 tx=0).stop 2024-09-11T01:42:21.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.330+0000 7f6c7ff2b640 1 -- 172.21.15.12:0/930186940 shutdown_connections 2024-09-11T01:42:21.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.330+0000 7f6c7ff2b640 1 --2- 172.21.15.12:0/930186940 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c781ccbc0 0x7f6c781cd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:21.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.330+0000 7f6c7ff2b640 1 --2- 172.21.15.12:0/930186940 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6c78138560 0x7f6c781389c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:21.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.330+0000 7f6c7ff2b640 1 --2- 172.21.15.12:0/930186940 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6c5c03e240 0x7f6c5c040700 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:21.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.330+0000 7f6c7ff2b640 1 --2- 172.21.15.12:0/930186940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6c640041a0 0x7f6c781c9660 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:21.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.330+0000 7f6c7ff2b640 1 -- 172.21.15.12:0/930186940 >> 172.21.15.12:0/930186940 conn(0x7f6c780a8490 msgr2=0x7f6c7813ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:21.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.330+0000 7f6c7ff2b640 1 -- 172.21.15.12:0/930186940 shutdown_connections 2024-09-11T01:42:21.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:21.330+0000 7f6c7ff2b640 1 -- 172.21.15.12:0/930186940 wait complete. 2024-09-11T01:42:24.344 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:42:24.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.455+0000 7f7c2c9f3640 10 monclient: get_monmap_and_config 2024-09-11T01:42:24.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.455+0000 7f7c2c9f3640 10 monclient: build_initial_monmap 2024-09-11T01:42:24.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.455+0000 7f7c2c9f3640 10 monclient: monmap: 2024-09-11T01:42:24.458 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:24.458 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:24.459 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:42:24.457678+0000 2024-09-11T01:42:24.459 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:42:24.457678+0000 2024-09-11T01:42:24.459 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:24.459 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:24.459 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:42:24.459 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:42:24.459 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:42:24.459 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:42:24.459 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:42:24.459 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:24.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.456+0000 7f7c2c9f3640 1 Processor -- start 2024-09-11T01:42:24.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.456+0000 7f7c2c9f3640 1 -- start start 2024-09-11T01:42:24.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.456+0000 7f7c2c9f3640 10 monclient: init 2024-09-11T01:42:24.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c2c9f3640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:24.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c2c9f3640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-11T01:42:24.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c2c9f3640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7c28073d90 0x7f7c28074170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:24.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c2c9f3640 10 monclient(hunting): picked mon.noname-c con 0x7f7c28073d90 addr v2:172.21.15.71:3300/0 2024-09-11T01:42:24.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c2c9f3640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7c280a0660 0x7f7c280a2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:24.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c2c9f3640 10 monclient(hunting): picked mon.noname-a con 0x7f7c280a0660 addr v2:172.21.15.12:3300/0 2024-09-11T01:42:24.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c2c9f3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c281380d0 addr v1:172.21.15.12:6789/0 2024-09-11T01:42:24.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c2c9f3640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c28070560 con 0x7f7c281380d0 2024-09-11T01:42:24.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c2c9f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:24.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c2c9f3640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f7c28070340 con 0x7f7c280a0660 2024-09-11T01:42:24.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c2c9f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:24.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c2c9f3640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f7c28070840 con 0x7f7c28073d90 2024-09-11T01:42:24.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c2c9f3640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:24.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c2c9f3640 10 monclient(hunting): authenticate will time out at 1550.900395s 2024-09-11T01:42:24.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c25d9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7c280a0660 0x7f7c280a2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:24.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c26d9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7c281380d0 0x7f7c280a2aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:48982/0 (socket says 172.21.15.12:48982) 2024-09-11T01:42:24.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c26d9d640 1 -- 172.21.15.12:0/115363554 learned_addr learned my addr 172.21.15.12:0/115363554 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:42:24.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c25d9b640 10 monclient(hunting): get_auth_request con 0x7f7c280a0660 auth_method 0 2024-09-11T01:42:24.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c25d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:24.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c25d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:24.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c25d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:24.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c25d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:24.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c25d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:24.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c25d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:24.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c2659c640 1 --2- 172.21.15.12:0/115363554 >> v2:172.21.15.71:3300/0 conn(0x7f7c28073d90 0x7f7c28074170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:24.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c25d9b640 10 monclient(hunting): handle_auth_done global_id 94611 payload 274 2024-09-11T01:42:24.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c25d9b640 1 -- 172.21.15.12:0/115363554 >> v2:172.21.15.71:3300/0 conn(0x7f7c28073d90 msgr2=0x7f7c28074170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:24.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c25d9b640 1 --2- 172.21.15.12:0/115363554 >> v2:172.21.15.71:3300/0 conn(0x7f7c28073d90 0x7f7c28074170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:24.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c25d9b640 1 -- 172.21.15.12:0/115363554 >> v1:172.21.15.12:6789/0 conn(0x7f7c281380d0 legacy=0x7f7c280a2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:24.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c25d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:42:24.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c25d9b640 1 monclient: found mon.noname-a 2024-09-11T01:42:24.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c25d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:24.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c25d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:42:24.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c25d9b640 1 -- 172.21.15.12:0/115363554 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c2809fff0 con 0x7f7c280a0660 2024-09-11T01:42:24.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c2659c640 10 monclient: get_auth_request con 0x7f7c28073d90 auth_method 0 2024-09-11T01:42:24.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.457+0000 7f7c2659c640 1 --2- 172.21.15.12:0/115363554 >> v2:172.21.15.71:3300/0 conn(0x7f7c28073d90 0x7f7c28074170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:42:24.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.460+0000 7f7c25d9b640 1 -- 172.21.15.12:0/115363554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c280a0660 msgr2=0x7f7c280a2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:24.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.460+0000 7f7c25d9b640 1 --2- 172.21.15.12:0/115363554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c280a0660 0x7f7c280a2560 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7c280a7c10 tx=0x7f7c1c02a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=474f4abc302b1e16 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:24.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2559a640 1 -- 172.21.15.12:0/115363554 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c037070 con 0x7f7c280a0660 2024-09-11T01:42:24.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:24.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:42:24.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2559a640 10 monclient: dump: 2024-09-11T01:42:24.469 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:24.470 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:24.470 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:24.470 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:24.470 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:24.470 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:24.470 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:24.470 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:24.470 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:24.470 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:24.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2559a640 10 monclient: _finish_auth 0 2024-09-11T01:42:24.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2559a640 10 monclient: _check_auth_tickets 2024-09-11T01:42:24.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:24.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2559a640 1 -- 172.21.15.12:0/115363554 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c1c032640 con 0x7f7c280a0660 2024-09-11T01:42:24.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2c9f3640 5 monclient: authenticate success, global_id 94611 2024-09-11T01:42:24.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2c9f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:42:24.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2559a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:24.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2c9f3640 10 monclient: get_monmap_and_config success 2024-09-11T01:42:24.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2559a640 1 -- 172.21.15.12:0/115363554 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c032a10 con 0x7f7c280a0660 2024-09-11T01:42:24.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:24.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2c9f3640 4 set_mon_vals no callback set 2024-09-11T01:42:24.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:24.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2559a640 10 monclient: dump: 2024-09-11T01:42:24.471 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:24.471 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:24.471 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:24.471 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:24.472 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:24.472 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:24.472 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:24.472 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:24.472 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:24.472 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:24.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2c9f3640 10 monclient: shutdown 2024-09-11T01:42:24.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2c9f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:24.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2c9f3640 1 -- 172.21.15.12:0/115363554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c280a0660 msgr2=0x7f7c280a2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:24.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2c9f3640 1 --2- 172.21.15.12:0/115363554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c280a0660 0x7f7c280a2560 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7c280a7c10 tx=0x7f7c1c02a000 comp rx=0 tx=0).stop 2024-09-11T01:42:24.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2c9f3640 1 -- 172.21.15.12:0/115363554 shutdown_connections 2024-09-11T01:42:24.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2c9f3640 1 --2- 172.21.15.12:0/115363554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c280a0660 0x7f7c280a2560 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:24.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2c9f3640 1 --2- 172.21.15.12:0/115363554 >> v2:172.21.15.71:3300/0 conn(0x7f7c28073d90 0x7f7c28074170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:24.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2c9f3640 1 -- 172.21.15.12:0/115363554 >> 172.21.15.12:0/115363554 conn(0x7f7c280a3650 msgr2=0x7f7c280a3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:24.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2c9f3640 1 -- 172.21.15.12:0/115363554 shutdown_connections 2024-09-11T01:42:24.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.461+0000 7f7c2c9f3640 1 -- 172.21.15.12:0/115363554 wait complete. 2024-09-11T01:42:24.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.462+0000 7f7c2c9f3640 10 monclient: build_initial_monmap 2024-09-11T01:42:24.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.462+0000 7f7c2c9f3640 10 monclient: monmap: 2024-09-11T01:42:24.473 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:24.473 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:42:24.473 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:42:24.473 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:42:24.473 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:24.473 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:24.473 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:42:24.473 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:42:24.474 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:42:24.474 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:24.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.462+0000 7f7c2c9f3640 1 Processor -- start 2024-09-11T01:42:24.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.462+0000 7f7c2c9f3640 1 -- start start 2024-09-11T01:42:24.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.462+0000 7f7c2c9f3640 10 monclient: init 2024-09-11T01:42:24.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2c9f3640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:24.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2c9f3640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:42:24.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2c9f3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c28073d90 0x7f7c28131240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:24.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2c9f3640 10 monclient(hunting): picked mon.noname-a con 0x7f7c28073d90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:42:24.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2c9f3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c280a0660 0x7f7c28131780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:24.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2c9f3640 10 monclient(hunting): picked mon.noname-b con 0x7f7c280a0660 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:42:24.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2c9f3640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c281380d0 0x7f7c281bf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:24.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2c9f3640 10 monclient(hunting): picked mon.noname-c con 0x7f7c281380d0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:24.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2c9f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:24.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2c9f3640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7c280a44d0 con 0x7f7c28073d90 2024-09-11T01:42:24.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2c9f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:24.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2c9f3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f7c280a4350 con 0x7f7c280a0660 2024-09-11T01:42:24.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2c9f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:24.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2c9f3640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7c280a31b0 con 0x7f7c281380d0 2024-09-11T01:42:24.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2c9f3640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:24.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2c9f3640 10 monclient(hunting): authenticate will time out at 1550.906389s 2024-09-11T01:42:24.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c26d9d640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c281380d0 msgr2=0x7f7c281bf6d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:42:24.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c26d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c281380d0 0x7f7c281bf6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:24.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c28073d90 0x7f7c28131240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:24.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c28073d90 0x7f7c28131240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52422/0 (socket says 172.21.15.12:52422) 2024-09-11T01:42:24.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2659c640 1 -- 172.21.15.12:0/3144870660 learned_addr learned my addr 172.21.15.12:0/3144870660 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:24.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2659c640 10 monclient(hunting): get_auth_request con 0x7f7c28073d90 auth_method 0 2024-09-11T01:42:24.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:24.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:24.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:24.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c25d9b640 1 --2- 172.21.15.12:0/3144870660 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c280a0660 0x7f7c28131780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:24.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:24.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:24.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:24.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c25d9b640 10 monclient(hunting): get_auth_request con 0x7f7c280a0660 auth_method 0 2024-09-11T01:42:24.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c25d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:24.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c25d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:24.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c25d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:24.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c25d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:24.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c25d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:24.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c25d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:24.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2659c640 10 monclient(hunting): handle_auth_done global_id 94614 payload 824 2024-09-11T01:42:24.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2659c640 1 -- 172.21.15.12:0/3144870660 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c281380d0 msgr2=0x7f7c281bf6d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:42:24.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2659c640 1 --2- 172.21.15.12:0/3144870660 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c281380d0 0x7f7c281bf6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:24.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2659c640 1 -- 172.21.15.12:0/3144870660 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c280a0660 msgr2=0x7f7c28131780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:24.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2659c640 1 --2- 172.21.15.12:0/3144870660 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c280a0660 0x7f7c28131780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:24.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2659c640 10 monclient: _finish_hunting 0 2024-09-11T01:42:24.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2659c640 1 monclient: found mon.noname-a 2024-09-11T01:42:24.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:24.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:42:24.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.463+0000 7f7c2659c640 1 -- 172.21.15.12:0/3144870660 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c281c1190 con 0x7f7c28073d90 2024-09-11T01:42:24.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.466+0000 7f7c2659c640 1 -- 172.21.15.12:0/3144870660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c28073d90 msgr2=0x7f7c28131240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:24.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.466+0000 7f7c2659c640 1 --2- 172.21.15.12:0/3144870660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c28073d90 0x7f7c28131240 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7c1800c7f0 tx=0x7f7c1800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:24.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.466+0000 7f7c16ffd640 1 -- 172.21.15.12:0/3144870660 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c18015070 con 0x7f7c28073d90 2024-09-11T01:42:24.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.466+0000 7f7c16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:24.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.466+0000 7f7c16ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:42:24.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.466+0000 7f7c16ffd640 10 monclient: dump: 2024-09-11T01:42:24.478 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:24.478 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:24.478 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:24.478 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:24.478 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:24.478 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:24.478 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:24.478 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:24.479 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:24.479 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:24.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.466+0000 7f7c16ffd640 10 monclient: _finish_auth 0 2024-09-11T01:42:24.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.466+0000 7f7c16ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:42:24.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.466+0000 7f7c16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:24.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.466+0000 7f7c16ffd640 1 -- 172.21.15.12:0/3144870660 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c18007be0 con 0x7f7c28073d90 2024-09-11T01:42:24.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.466+0000 7f7c16ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:24.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.466+0000 7f7c16ffd640 1 -- 172.21.15.12:0/3144870660 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c1800ce40 con 0x7f7c28073d90 2024-09-11T01:42:24.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.466+0000 7f7c16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:24.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.466+0000 7f7c16ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:24.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.466+0000 7f7c16ffd640 10 monclient: dump: 2024-09-11T01:42:24.479 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:24.479 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:24.480 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:24.480 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:24.480 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:24.480 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:24.480 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:24.480 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:24.480 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:24.480 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:24.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.466+0000 7f7c2c9f3640 5 monclient: authenticate success, global_id 94614 2024-09-11T01:42:24.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.466+0000 7f7c2c9f3640 10 monclient: _renew_subs 2024-09-11T01:42:24.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.466+0000 7f7c2c9f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:24.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.466+0000 7f7c2c9f3640 1 -- 172.21.15.12:0/3144870660 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c281c2160 con 0x7f7c28073d90 2024-09-11T01:42:24.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.466+0000 7f7c24d99640 4 set_mon_vals no callback set 2024-09-11T01:42:24.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.466+0000 7f7c2c9f3640 10 monclient: _renew_subs 2024-09-11T01:42:24.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.466+0000 7f7c2c9f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:24.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.466+0000 7f7c2c9f3640 1 -- 172.21.15.12:0/3144870660 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c2811e640 con 0x7f7c28073d90 2024-09-11T01:42:24.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.467+0000 7f7c16ffd640 1 -- 172.21.15.12:0/3144870660 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f7c180113f0 con 0x7f7c28073d90 2024-09-11T01:42:24.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.467+0000 7f7c2c9f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:24.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.467+0000 7f7c2c9f3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:24.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.467+0000 7f7c2c9f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:24.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.467+0000 7f7c2c9f3640 1 -- 172.21.15.12:0/3144870660 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c28074170 con 0x7f7c28073d90 2024-09-11T01:42:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.467+0000 7f7c16ffd640 1 --2- 172.21.15.12:0/3144870660 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7c0803e1c0 0x7f7c08040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.467+0000 7f7c16ffd640 1 -- 172.21.15.12:0/3144870660 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(442..442 src has 285..442) ==== 4787+0+0 (secure 0 0 0) 0x7f7c18053150 con 0x7f7c28073d90 2024-09-11T01:42:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.468+0000 7f7c25d9b640 1 --2- 172.21.15.12:0/3144870660 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7c0803e1c0 0x7f7c08040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.468+0000 7f7c25d9b640 10 monclient: get_auth_request con 0x7f7c0803e1c0 auth_method 0 2024-09-11T01:42:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.470+0000 7f7c25d9b640 1 --2- 172.21.15.12:0/3144870660 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7c0803e1c0 0x7f7c08040680 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7c28070280 tx=0x7f7c1c02e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.472+0000 7f7c16ffd640 1 -- 172.21.15.12:0/3144870660 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7c281c2160 con 0x7f7c28073d90 2024-09-11T01:42:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.472+0000 7f7c16ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:24.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.472+0000 7f7c16ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:42:24.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.727+0000 7f7c2c9f3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:42:24.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.727+0000 7f7c2c9f3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:42:24.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.727+0000 7f7c2c9f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:24.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.727+0000 7f7c2c9f3640 1 -- 172.21.15.12:0/3144870660 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7c2806dca0 con 0x7f7c28073d90 2024-09-11T01:42:24.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.727+0000 7f7c16ffd640 1 -- 172.21.15.12:0/3144870660 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7f7c18051b40 con 0x7f7c28073d90 2024-09-11T01:42:24.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.727+0000 7f7c16ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:42:24.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.727+0000 7f7c16ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:42:24.731 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":122,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:42:24.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.729+0000 7f7c2c9f3640 1 -- 172.21.15.12:0/3144870660 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7c0803e1c0 msgr2=0x7f7c08040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:24.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.729+0000 7f7c2c9f3640 1 --2- 172.21.15.12:0/3144870660 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7c0803e1c0 0x7f7c08040680 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7c28070280 tx=0x7f7c1c02e000 comp rx=0 tx=0).stop 2024-09-11T01:42:24.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.729+0000 7f7c2c9f3640 10 monclient: shutdown 2024-09-11T01:42:24.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.729+0000 7f7c2c9f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:24.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.729+0000 7f7c2c9f3640 1 -- 172.21.15.12:0/3144870660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c28073d90 msgr2=0x7f7c28131240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:24.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.729+0000 7f7c2c9f3640 1 --2- 172.21.15.12:0/3144870660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c28073d90 0x7f7c28131240 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7c1800c7f0 tx=0x7f7c1800cbf0 comp rx=0 tx=0).stop 2024-09-11T01:42:24.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.730+0000 7f7c2c9f3640 1 -- 172.21.15.12:0/3144870660 shutdown_connections 2024-09-11T01:42:24.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.730+0000 7f7c2c9f3640 1 --2- 172.21.15.12:0/3144870660 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c281380d0 0x7f7c281bf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:24.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.730+0000 7f7c2c9f3640 1 --2- 172.21.15.12:0/3144870660 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c280a0660 0x7f7c28131780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:24.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.730+0000 7f7c2c9f3640 1 --2- 172.21.15.12:0/3144870660 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7c0803e1c0 0x7f7c08040680 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:24.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.730+0000 7f7c2c9f3640 1 --2- 172.21.15.12:0/3144870660 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c28073d90 0x7f7c28131240 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:24.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.730+0000 7f7c2c9f3640 1 -- 172.21.15.12:0/3144870660 >> 172.21.15.12:0/3144870660 conn(0x7f7c280a3650 msgr2=0x7f7c28073480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:24.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.730+0000 7f7c2c9f3640 1 -- 172.21.15.12:0/3144870660 shutdown_connections 2024-09-11T01:42:24.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:24.730+0000 7f7c2c9f3640 1 -- 172.21.15.12:0/3144870660 wait complete. 2024-09-11T01:42:27.744 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:42:27.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.855+0000 7fad33775640 10 monclient: get_monmap_and_config 2024-09-11T01:42:27.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.855+0000 7fad33775640 10 monclient: build_initial_monmap 2024-09-11T01:42:27.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.855+0000 7fad33775640 10 monclient: monmap: 2024-09-11T01:42:27.857 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:27.857 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:27.857 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:42:27.856365+0000 2024-09-11T01:42:27.857 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:42:27.856365+0000 2024-09-11T01:42:27.857 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:27.857 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:27.857 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:42:27.858 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:42:27.858 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:42:27.858 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:42:27.858 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:42:27.858 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:27.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.856+0000 7fad33775640 1 Processor -- start 2024-09-11T01:42:27.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.856+0000 7fad33775640 1 -- start start 2024-09-11T01:42:27.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.856+0000 7fad33775640 10 monclient: init 2024-09-11T01:42:27.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad33775640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:27.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad33775640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-11T01:42:27.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad33775640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fad2c0a2e90 addr v1:172.21.15.12:6789/0 2024-09-11T01:42:27.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad33775640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fad2c0a3270 0x7fad2c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:27.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad33775640 10 monclient(hunting): picked mon.noname-a con 0x7fad2c0a3270 addr v2:172.21.15.12:3300/0 2024-09-11T01:42:27.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad33775640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fad2c13d490 0x7fad2c13f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:27.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad33775640 10 monclient(hunting): picked mon.noname-b con 0x7fad2c13d490 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:27.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad33775640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad2c0a46c0 con 0x7fad2c0a2e90 2024-09-11T01:42:27.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad33775640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:27.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad33775640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fad2c0a4900 con 0x7fad2c0a3270 2024-09-11T01:42:27.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad31d12640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fad2c13d490 msgr2=0x7fad2c13f880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:42:27.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad33775640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:27.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad33775640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fad2c06ee50 con 0x7fad2c13d490 2024-09-11T01:42:27.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad31d12640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fad2c13d490 0x7fad2c13f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:27.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad33775640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:27.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad33775640 10 monclient(hunting): authenticate will time out at 1554.299266s 2024-09-11T01:42:27.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad31511640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fad2c0a2e90 0x7fad2c06dc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57562/0 (socket says 172.21.15.12:57562) 2024-09-11T01:42:27.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad31511640 1 -- 172.21.15.12:0/2408943014 learned_addr learned my addr 172.21.15.12:0/2408943014 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:42:27.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad30d10640 1 --2- 172.21.15.12:0/2408943014 >> v2:172.21.15.12:3300/0 conn(0x7fad2c0a3270 0x7fad2c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:27.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad30d10640 10 monclient(hunting): get_auth_request con 0x7fad2c0a3270 auth_method 0 2024-09-11T01:42:27.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad30d10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:27.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad30d10640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:27.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad30d10640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:27.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad30d10640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:27.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad30d10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:27.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad30d10640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:27.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad30d10640 10 monclient(hunting): handle_auth_done global_id 94617 payload 274 2024-09-11T01:42:27.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad30d10640 1 -- 172.21.15.12:0/2408943014 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fad2c13d490 msgr2=0x7fad2c13f880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:42:27.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad30d10640 1 --2- 172.21.15.12:0/2408943014 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fad2c13d490 0x7fad2c13f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:27.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad30d10640 1 -- 172.21.15.12:0/2408943014 >> v1:172.21.15.12:6789/0 conn(0x7fad2c0a2e90 legacy=0x7fad2c06dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:27.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad30d10640 10 monclient: _finish_hunting 0 2024-09-11T01:42:27.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad30d10640 1 monclient: found mon.noname-a 2024-09-11T01:42:27.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad30d10640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:27.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad30d10640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:42:27.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.857+0000 7fad30d10640 1 -- 172.21.15.12:0/2408943014 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad2c06eb70 con 0x7fad2c0a3270 2024-09-11T01:42:27.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.859+0000 7fad30d10640 1 -- 172.21.15.12:0/2408943014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad2c0a3270 msgr2=0x7fad2c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:27.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.859+0000 7fad30d10640 1 --2- 172.21.15.12:0/2408943014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad2c0a3270 0x7fad2c0a3680 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fad2c071820 tx=0x7fad1c029fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=697a0b9f713baf0b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:27.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.859+0000 7fad23fff640 1 -- 172.21.15.12:0/2408943014 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c037070 con 0x7fad2c0a3270 2024-09-11T01:42:27.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.859+0000 7fad23fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:27.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.859+0000 7fad23fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:42:27.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.859+0000 7fad23fff640 10 monclient: dump: 2024-09-11T01:42:27.876 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:27.876 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:27.876 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:27.876 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:27.877 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:27.877 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:27.877 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:27.877 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:27.877 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:27.877 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:27.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.859+0000 7fad23fff640 10 monclient: _finish_auth 0 2024-09-11T01:42:27.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.859+0000 7fad23fff640 10 monclient: _check_auth_tickets 2024-09-11T01:42:27.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.859+0000 7fad23fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:27.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.859+0000 7fad33775640 5 monclient: authenticate success, global_id 94617 2024-09-11T01:42:27.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.859+0000 7fad33775640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:42:27.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.859+0000 7fad23fff640 1 -- 172.21.15.12:0/2408943014 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad1c032440 con 0x7fad2c0a3270 2024-09-11T01:42:27.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.859+0000 7fad23fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:27.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.859+0000 7fad23fff640 1 -- 172.21.15.12:0/2408943014 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c032810 con 0x7fad2c0a3270 2024-09-11T01:42:27.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.859+0000 7fad33775640 10 monclient: get_monmap_and_config success 2024-09-11T01:42:27.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.859+0000 7fad33775640 4 set_mon_vals no callback set 2024-09-11T01:42:27.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.859+0000 7fad23fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:27.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.859+0000 7fad23fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:27.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.859+0000 7fad23fff640 10 monclient: dump: 2024-09-11T01:42:27.880 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:27.880 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:27.880 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:27.880 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:27.881 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:27.881 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:27.881 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:27.881 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:27.881 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:27.881 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:27.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.860+0000 7fad33775640 10 monclient: shutdown 2024-09-11T01:42:27.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.860+0000 7fad33775640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:27.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.860+0000 7fad33775640 1 -- 172.21.15.12:0/2408943014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad2c0a3270 msgr2=0x7fad2c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:27.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.860+0000 7fad33775640 1 --2- 172.21.15.12:0/2408943014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad2c0a3270 0x7fad2c0a3680 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fad2c071820 tx=0x7fad1c029fb0 comp rx=0 tx=0).stop 2024-09-11T01:42:27.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.860+0000 7fad33775640 1 -- 172.21.15.12:0/2408943014 shutdown_connections 2024-09-11T01:42:27.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.860+0000 7fad33775640 1 --2- 172.21.15.12:0/2408943014 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fad2c13d490 0x7fad2c13f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:27.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.860+0000 7fad33775640 1 --2- 172.21.15.12:0/2408943014 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad2c0a3270 0x7fad2c0a3680 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:27.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.860+0000 7fad33775640 1 -- 172.21.15.12:0/2408943014 >> 172.21.15.12:0/2408943014 conn(0x7fad2c0a8490 msgr2=0x7fad2c0a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:27.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.860+0000 7fad33775640 1 -- 172.21.15.12:0/2408943014 shutdown_connections 2024-09-11T01:42:27.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.860+0000 7fad33775640 1 -- 172.21.15.12:0/2408943014 wait complete. 2024-09-11T01:42:27.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.860+0000 7fad33775640 10 monclient: build_initial_monmap 2024-09-11T01:42:27.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.860+0000 7fad33775640 10 monclient: monmap: 2024-09-11T01:42:27.884 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:27.884 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:42:27.884 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:42:27.884 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:42:27.884 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:27.885 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:27.885 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:42:27.885 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:42:27.885 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:42:27.885 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:27.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad33775640 1 Processor -- start 2024-09-11T01:42:27.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad33775640 1 -- start start 2024-09-11T01:42:27.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad33775640 10 monclient: init 2024-09-11T01:42:27.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad33775640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:27.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad33775640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:42:27.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad33775640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fad2c0a2e90 0x7fad2c1cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:27.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad33775640 10 monclient(hunting): picked mon.noname-c con 0x7fad2c0a2e90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:27.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad33775640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad2c0a3270 0x7fad2c1cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:27.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad33775640 10 monclient(hunting): picked mon.noname-a con 0x7fad2c0a3270 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:42:27.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad33775640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad2c13d490 0x7fad2c1cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:27.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad33775640 10 monclient(hunting): picked mon.noname-b con 0x7fad2c13d490 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:42:27.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad33775640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:27.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad33775640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c06fd20 con 0x7fad2c0a3270 2024-09-11T01:42:27.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad33775640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:27.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad33775640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c06fba0 con 0x7fad2c13d490 2024-09-11T01:42:27.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad33775640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:27.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad33775640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fad2c13fdc0 con 0x7fad2c0a2e90 2024-09-11T01:42:27.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad33775640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:27.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad31511640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fad2c0a2e90 msgr2=0x7fad2c1cd1e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:42:27.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad33775640 10 monclient(hunting): authenticate will time out at 1554.304911s 2024-09-11T01:42:27.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad31511640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fad2c0a2e90 0x7fad2c1cd1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:27.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad30d10640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad2c0a3270 0x7fad2c1cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:27.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad30d10640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad2c0a3270 0x7fad2c1cd720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35730/0 (socket says 172.21.15.12:35730) 2024-09-11T01:42:27.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad30d10640 1 -- 172.21.15.12:0/3622571309 learned_addr learned my addr 172.21.15.12:0/3622571309 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:27.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad30d10640 10 monclient(hunting): get_auth_request con 0x7fad2c0a3270 auth_method 0 2024-09-11T01:42:27.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad30d10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:27.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad30d10640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:27.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad30d10640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:27.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad30d10640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:27.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad30d10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:27.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad30d10640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:27.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad31d12640 1 --2- 172.21.15.12:0/3622571309 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad2c13d490 0x7fad2c1cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:27.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad31d12640 10 monclient(hunting): get_auth_request con 0x7fad2c13d490 auth_method 0 2024-09-11T01:42:27.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad31d12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:27.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad31d12640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:27.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad31d12640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:27.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad30d10640 10 monclient(hunting): handle_auth_done global_id 94620 payload 824 2024-09-11T01:42:27.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad30d10640 1 -- 172.21.15.12:0/3622571309 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fad2c0a2e90 msgr2=0x7fad2c1cd1e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:42:27.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad30d10640 1 --2- 172.21.15.12:0/3622571309 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fad2c0a2e90 0x7fad2c1cd1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:27.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad30d10640 1 -- 172.21.15.12:0/3622571309 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad2c13d490 msgr2=0x7fad2c1cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:27.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad30d10640 1 --2- 172.21.15.12:0/3622571309 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad2c13d490 0x7fad2c1cdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:27.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad30d10640 10 monclient: _finish_hunting 0 2024-09-11T01:42:27.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad30d10640 1 monclient: found mon.noname-a 2024-09-11T01:42:27.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad30d10640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:27.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad30d10640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:42:27.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.862+0000 7fad30d10640 1 -- 172.21.15.12:0/3622571309 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad2c0a68a0 con 0x7fad2c0a3270 2024-09-11T01:42:27.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.864+0000 7fad30d10640 1 -- 172.21.15.12:0/3622571309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad2c0a3270 msgr2=0x7fad2c1cd720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:27.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.864+0000 7fad30d10640 1 --2- 172.21.15.12:0/3622571309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad2c0a3270 0x7fad2c1cd720 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fad1c000c00 tx=0x7fad1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:27.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.864+0000 7fad21ffb640 1 -- 172.21.15.12:0/3622571309 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c041070 con 0x7fad2c0a3270 2024-09-11T01:42:27.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.864+0000 7fad21ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:27.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.864+0000 7fad21ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:42:27.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.864+0000 7fad21ffb640 10 monclient: dump: 2024-09-11T01:42:27.895 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:27.895 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:27.895 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:27.895 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:27.895 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:27.895 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:27.896 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:27.896 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:27.896 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:27.896 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:27.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.864+0000 7fad21ffb640 10 monclient: _finish_auth 0 2024-09-11T01:42:27.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.864+0000 7fad21ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:42:27.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.864+0000 7fad21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:27.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.864+0000 7fad21ffb640 1 -- 172.21.15.12:0/3622571309 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad1c03c890 con 0x7fad2c0a3270 2024-09-11T01:42:27.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.864+0000 7fad33775640 5 monclient: authenticate success, global_id 94620 2024-09-11T01:42:27.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.864+0000 7fad21ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:27.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.864+0000 7fad21ffb640 1 -- 172.21.15.12:0/3622571309 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c037040 con 0x7fad2c0a3270 2024-09-11T01:42:27.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.864+0000 7fad33775640 10 monclient: _renew_subs 2024-09-11T01:42:27.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.864+0000 7fad33775640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:27.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.864+0000 7fad33775640 1 -- 172.21.15.12:0/3622571309 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad2c09e670 con 0x7fad2c0a3270 2024-09-11T01:42:27.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.864+0000 7fad21ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:27.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.864+0000 7fad21ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:27.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.864+0000 7fad21ffb640 10 monclient: dump: 2024-09-11T01:42:27.899 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:27.899 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:27.899 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:27.899 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:27.899 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:27.899 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:27.900 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:27.900 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:27.900 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:27.900 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:27.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.864+0000 7fad22ffd640 4 set_mon_vals no callback set 2024-09-11T01:42:27.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.865+0000 7fad33775640 10 monclient: _renew_subs 2024-09-11T01:42:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.865+0000 7fad33775640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.865+0000 7fad33775640 1 -- 172.21.15.12:0/3622571309 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad2c09f680 con 0x7fad2c0a3270 2024-09-11T01:42:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.866+0000 7fad21ffb640 1 -- 172.21.15.12:0/3622571309 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fad1c03ccc0 con 0x7fad2c0a3270 2024-09-11T01:42:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.866+0000 7fad33775640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.866+0000 7fad33775640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.866+0000 7fad33775640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:27.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.866+0000 7fad33775640 1 -- 172.21.15.12:0/3622571309 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad2c13fad0 con 0x7fad2c0a3270 2024-09-11T01:42:27.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.866+0000 7fad21ffb640 1 --2- 172.21.15.12:0/3622571309 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fad1003e1f0 0x7fad100406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:27.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.866+0000 7fad21ffb640 1 -- 172.21.15.12:0/3622571309 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(442..442 src has 285..442) ==== 4787+0+0 (secure 0 0 0) 0x7fad1c073790 con 0x7fad2c0a3270 2024-09-11T01:42:27.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.866+0000 7fad31511640 1 --2- 172.21.15.12:0/3622571309 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fad1003e1f0 0x7fad100406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:27.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.866+0000 7fad31511640 10 monclient: get_auth_request con 0x7fad1003e1f0 auth_method 0 2024-09-11T01:42:27.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.869+0000 7fad31511640 1 --2- 172.21.15.12:0/3622571309 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fad1003e1f0 0x7fad100406b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fad28007fd0 tx=0x7fad28007b20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:27.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.871+0000 7fad21ffb640 1 -- 172.21.15.12:0/3622571309 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fad2c13fad0 con 0x7fad2c0a3270 2024-09-11T01:42:27.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.871+0000 7fad21ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:27.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:27.871+0000 7fad21ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:42:28.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:28.127+0000 7fad33775640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:42:28.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:28.127+0000 7fad33775640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:42:28.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:28.127+0000 7fad33775640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:28.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:28.127+0000 7fad33775640 1 -- 172.21.15.12:0/3622571309 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fad2c0a3680 con 0x7fad2c0a3270 2024-09-11T01:42:28.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:28.128+0000 7fad21ffb640 1 -- 172.21.15.12:0/3622571309 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7fad2c0a68a0 con 0x7fad2c0a3270 2024-09-11T01:42:28.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:28.128+0000 7fad21ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:42:28.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:28.128+0000 7fad21ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:42:28.131 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":122,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:42:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:28.130+0000 7fad33775640 1 -- 172.21.15.12:0/3622571309 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fad1003e1f0 msgr2=0x7fad100406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:28.130+0000 7fad33775640 1 --2- 172.21.15.12:0/3622571309 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fad1003e1f0 0x7fad100406b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fad28007fd0 tx=0x7fad28007b20 comp rx=0 tx=0).stop 2024-09-11T01:42:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:28.130+0000 7fad33775640 10 monclient: shutdown 2024-09-11T01:42:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:28.130+0000 7fad33775640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:28.130+0000 7fad33775640 1 -- 172.21.15.12:0/3622571309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad2c0a3270 msgr2=0x7fad2c1cd720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:28.130+0000 7fad33775640 1 --2- 172.21.15.12:0/3622571309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad2c0a3270 0x7fad2c1cd720 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fad1c000c00 tx=0x7fad1c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:42:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:28.130+0000 7fad33775640 1 -- 172.21.15.12:0/3622571309 shutdown_connections 2024-09-11T01:42:28.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:28.130+0000 7fad33775640 1 --2- 172.21.15.12:0/3622571309 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad2c13d490 0x7fad2c1cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:28.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:28.130+0000 7fad33775640 1 --2- 172.21.15.12:0/3622571309 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fad1003e1f0 0x7fad100406b0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:28.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:28.130+0000 7fad33775640 1 --2- 172.21.15.12:0/3622571309 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad2c0a3270 0x7fad2c1cd720 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:28.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:28.130+0000 7fad33775640 1 --2- 172.21.15.12:0/3622571309 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fad2c0a2e90 0x7fad2c1cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:28.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:28.130+0000 7fad33775640 1 -- 172.21.15.12:0/3622571309 >> 172.21.15.12:0/3622571309 conn(0x7fad2c0a8490 msgr2=0x7fad2c13cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:28.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:28.130+0000 7fad33775640 1 -- 172.21.15.12:0/3622571309 shutdown_connections 2024-09-11T01:42:28.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:28.130+0000 7fad33775640 1 -- 172.21.15.12:0/3622571309 wait complete. 2024-09-11T01:42:31.144 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:42:31.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.256+0000 7f645bb0e640 10 monclient: get_monmap_and_config 2024-09-11T01:42:31.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.256+0000 7f645bb0e640 10 monclient: build_initial_monmap 2024-09-11T01:42:31.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.256+0000 7f645bb0e640 10 monclient: monmap: 2024-09-11T01:42:31.258 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:31.258 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:31.258 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:42:31.256967+0000 2024-09-11T01:42:31.258 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:42:31.256967+0000 2024-09-11T01:42:31.258 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:31.258 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:31.258 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:42:31.258 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:42:31.258 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:42:31.258 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:42:31.258 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:42:31.258 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:31.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.256+0000 7f645bb0e640 1 Processor -- start 2024-09-11T01:42:31.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.257+0000 7f645bb0e640 1 -- start start 2024-09-11T01:42:31.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.257+0000 7f645bb0e640 10 monclient: init 2024-09-11T01:42:31.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.257+0000 7f645bb0e640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:31.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.257+0000 7f645bb0e640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-11T01:42:31.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.257+0000 7f645bb0e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6454138230 0x7f6454138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:31.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.257+0000 7f645bb0e640 10 monclient(hunting): picked mon.noname-b con 0x7f6454138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:31.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.257+0000 7f645bb0e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f645409d030 0x7f64540a5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:31.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.257+0000 7f645bb0e640 10 monclient(hunting): picked mon.noname-a con 0x7f645409d030 addr v2:172.21.15.12:3300/0 2024-09-11T01:42:31.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.257+0000 7f645bb0e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f645409d430 addr v1:172.21.15.71:6789/0 2024-09-11T01:42:31.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.257+0000 7f645bb0e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64540a6e30 con 0x7f645409d430 2024-09-11T01:42:31.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.257+0000 7f645bb0e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:31.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.257+0000 7f645bb0e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f645409c9c0 con 0x7f645409d030 2024-09-11T01:42:31.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.257+0000 7f645bb0e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:31.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.257+0000 7f645bb0e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f645409ba20 con 0x7f6454138230 2024-09-11T01:42:31.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.257+0000 7f645bb0e640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:31.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.257+0000 7f645bb0e640 10 monclient(hunting): authenticate will time out at 1557.699762s 2024-09-11T01:42:31.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.257+0000 7f64598aa640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6454138230 msgr2=0x7f6454138610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:42:31.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.257+0000 7f64598aa640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6454138230 0x7f6454138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:31.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.257+0000 7f64590a9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f645409d030 0x7f64540a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:31.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.257+0000 7f64590a9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f645409d030 0x7f64540a5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35734/0 (socket says 172.21.15.12:35734) 2024-09-11T01:42:31.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.257+0000 7f64590a9640 1 -- 172.21.15.12:0/4141293577 learned_addr learned my addr 172.21.15.12:0/4141293577 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:31.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.257+0000 7f64590a9640 10 monclient(hunting): get_auth_request con 0x7f645409d030 auth_method 0 2024-09-11T01:42:31.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.257+0000 7f64590a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:31.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.257+0000 7f64590a9640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:31.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.257+0000 7f64590a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:31.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.258+0000 7f64590a9640 1 -- 172.21.15.12:0/4141293577 >> v2:172.21.15.12:3300/0 conn(0x7f645409d030 msgr2=0x7f64540a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:42:31.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.258+0000 7f64590a9640 1 -- 172.21.15.12:0/4141293577 >> v2:172.21.15.12:3300/0 conn(0x7f645409d030 msgr2=0x7f64540a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:42:31.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.258+0000 7f64590a9640 1 --2- 172.21.15.12:0/4141293577 >> v2:172.21.15.12:3300/0 conn(0x7f645409d030 0x7f64540a5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:42:31.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.258+0000 7f64590a9640 1 --2- 172.21.15.12:0/4141293577 >> v2:172.21.15.12:3300/0 conn(0x7f645409d030 0x7f64540a5020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:31.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.260+0000 7f645a0ab640 1 -- 172.21.15.12:0/4141293577 >> v1:172.21.15.71:6789/0 conn(0x7f645409d430 legacy=0x7f64540a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:31.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.458+0000 7f64598aa640 1 -- 172.21.15.12:0/4141293577 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6454138230 msgr2=0x7f6454138610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:42:31.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.458+0000 7f64598aa640 1 --2- 172.21.15.12:0/4141293577 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6454138230 0x7f6454138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:42:31.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.459+0000 7f64590a9640 1 --2- 172.21.15.12:0/4141293577 >> v2:172.21.15.12:3300/0 conn(0x7f645409d030 0x7f64540a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:31.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.459+0000 7f64590a9640 10 monclient(hunting): get_auth_request con 0x7f645409d030 auth_method 0 2024-09-11T01:42:31.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.459+0000 7f64590a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:31.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.459+0000 7f64590a9640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:31.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.459+0000 7f64590a9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:42:31.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.459+0000 7f64590a9640 1 -- 172.21.15.12:0/4141293577 >> v2:172.21.15.12:3300/0 conn(0x7f645409d030 msgr2=0x7f64540a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:42:31.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.459+0000 7f64590a9640 1 -- 172.21.15.12:0/4141293577 >> v2:172.21.15.12:3300/0 conn(0x7f645409d030 msgr2=0x7f64540a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:42:31.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.459+0000 7f64590a9640 1 --2- 172.21.15.12:0/4141293577 >> v2:172.21.15.12:3300/0 conn(0x7f645409d030 0x7f64540a5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:42:31.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.459+0000 7f64590a9640 1 --2- 172.21.15.12:0/4141293577 >> v2:172.21.15.12:3300/0 conn(0x7f645409d030 0x7f64540a5020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:42:31.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.859+0000 7f64598aa640 1 -- 172.21.15.12:0/4141293577 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6454138230 msgr2=0x7f6454138610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:42:31.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.859+0000 7f64598aa640 1 --2- 172.21.15.12:0/4141293577 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6454138230 0x7f6454138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:42:31.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.860+0000 7f64590a9640 1 --2- 172.21.15.12:0/4141293577 >> v2:172.21.15.12:3300/0 conn(0x7f645409d030 0x7f64540a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:31.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.860+0000 7f64590a9640 10 monclient(hunting): get_auth_request con 0x7f645409d030 auth_method 0 2024-09-11T01:42:31.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.860+0000 7f64590a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:31.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.860+0000 7f64590a9640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:31.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.860+0000 7f64590a9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:42:31.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.860+0000 7f64590a9640 1 -- 172.21.15.12:0/4141293577 >> v2:172.21.15.12:3300/0 conn(0x7f645409d030 msgr2=0x7f64540a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:42:31.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.860+0000 7f64590a9640 1 -- 172.21.15.12:0/4141293577 >> v2:172.21.15.12:3300/0 conn(0x7f645409d030 msgr2=0x7f64540a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:42:31.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.860+0000 7f64590a9640 1 --2- 172.21.15.12:0/4141293577 >> v2:172.21.15.12:3300/0 conn(0x7f645409d030 0x7f64540a5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:42:31.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:31.860+0000 7f64590a9640 1 --2- 172.21.15.12:0/4141293577 >> v2:172.21.15.12:3300/0 conn(0x7f645409d030 0x7f64540a5020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:42:32.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:32.660+0000 7f64598aa640 1 -- 172.21.15.12:0/4141293577 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6454138230 msgr2=0x7f6454138610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:42:32.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:32.660+0000 7f64598aa640 1 --2- 172.21.15.12:0/4141293577 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6454138230 0x7f6454138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:42:32.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:32.661+0000 7f64590a9640 1 --2- 172.21.15.12:0/4141293577 >> v2:172.21.15.12:3300/0 conn(0x7f645409d030 0x7f64540a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:32.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:32.661+0000 7f64590a9640 10 monclient(hunting): get_auth_request con 0x7f645409d030 auth_method 0 2024-09-11T01:42:32.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:32.661+0000 7f64590a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:32.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:32.661+0000 7f64590a9640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:32.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:32.661+0000 7f64590a9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:42:32.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:32.661+0000 7f64590a9640 1 -- 172.21.15.12:0/4141293577 >> v2:172.21.15.12:3300/0 conn(0x7f645409d030 msgr2=0x7f64540a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:42:32.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:32.661+0000 7f64590a9640 1 -- 172.21.15.12:0/4141293577 >> v2:172.21.15.12:3300/0 conn(0x7f645409d030 msgr2=0x7f64540a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:42:32.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:32.661+0000 7f64590a9640 1 --2- 172.21.15.12:0/4141293577 >> v2:172.21.15.12:3300/0 conn(0x7f645409d030 0x7f64540a5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:42:32.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:32.661+0000 7f64590a9640 1 --2- 172.21.15.12:0/4141293577 >> v2:172.21.15.12:3300/0 conn(0x7f645409d030 0x7f64540a5020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:42:34.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.257+0000 7f644b7fe640 10 monclient(hunting): tick 2024-09-11T01:42:34.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.257+0000 7f644b7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:42:34.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.257+0000 7f644b7fe640 1 monclient(hunting): continuing hunt 2024-09-11T01:42:34.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.257+0000 7f644b7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:42:34.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.257+0000 7f644b7fe640 1 -- 172.21.15.12:0/4141293577 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6454138230 msgr2=0x7f6454138610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:42:34.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.257+0000 7f644b7fe640 1 --2- 172.21.15.12:0/4141293577 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6454138230 0x7f6454138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:34.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.257+0000 7f644b7fe640 1 -- 172.21.15.12:0/4141293577 >> v2:172.21.15.12:3300/0 conn(0x7f645409d030 msgr2=0x7f64540a5020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:42:34.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.257+0000 7f644b7fe640 1 --2- 172.21.15.12:0/4141293577 >> v2:172.21.15.12:3300/0 conn(0x7f645409d030 0x7f64540a5020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:34.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.257+0000 7f644b7fe640 1 -- 172.21.15.12:0/4141293577 >> v1:172.21.15.71:6789/0 conn(0x7f645409d430 legacy=0x7f64540a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:34.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.257+0000 7f644b7fe640 10 monclient: _add_conns ranks=[4,0] 2024-09-11T01:42:34.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.257+0000 7f644b7fe640 1 --2- 172.21.15.12:0/4141293577 >> v2:172.21.15.71:3300/0 conn(0x7f6438003760 0x7f6438005be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:34.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.257+0000 7f644b7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f6438003760 addr v2:172.21.15.71:3300/0 2024-09-11T01:42:34.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.257+0000 7f644b7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f64380062d0 addr v1:172.21.15.12:6789/0 2024-09-11T01:42:34.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.257+0000 7f644b7fe640 1 -- 172.21.15.12:0/4141293577 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64540a66a0 con 0x7f64380062d0 2024-09-11T01:42:34.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.257+0000 7f644b7fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:34.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.257+0000 7f644b7fe640 1 -- 172.21.15.12:0/4141293577 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f645409c9c0 con 0x7f6438003760 2024-09-11T01:42:34.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.257+0000 7f644b7fe640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:34.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.258+0000 7f64598aa640 1 --2- 172.21.15.12:0/4141293577 >> v2:172.21.15.71:3300/0 conn(0x7f6438003760 0x7f6438005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:34.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.258+0000 7f64598aa640 10 monclient(hunting): get_auth_request con 0x7f6438003760 auth_method 0 2024-09-11T01:42:34.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.258+0000 7f64598aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:34.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.258+0000 7f64598aa640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:34.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.258+0000 7f64598aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:34.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.258+0000 7f64598aa640 1 -- 172.21.15.12:0/4141293577 >> v2:172.21.15.71:3300/0 conn(0x7f6438003760 msgr2=0x7f6438005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:42:34.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.258+0000 7f64598aa640 1 -- 172.21.15.12:0/4141293577 >> v2:172.21.15.71:3300/0 conn(0x7f6438003760 msgr2=0x7f6438005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:42:34.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.258+0000 7f64598aa640 1 --2- 172.21.15.12:0/4141293577 >> v2:172.21.15.71:3300/0 conn(0x7f6438003760 0x7f6438005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:42:34.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.258+0000 7f64598aa640 1 --2- 172.21.15.12:0/4141293577 >> v2:172.21.15.71:3300/0 conn(0x7f6438003760 0x7f6438005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:34.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.260+0000 7f64590a9640 1 -- 172.21.15.12:0/4141293577 >> v1:172.21.15.12:6789/0 conn(0x7f64380062d0 legacy=0x7f6438008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:34.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.460+0000 7f64598aa640 1 --2- 172.21.15.12:0/4141293577 >> v2:172.21.15.71:3300/0 conn(0x7f6438003760 0x7f6438005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:34.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.460+0000 7f64598aa640 10 monclient(hunting): get_auth_request con 0x7f6438003760 auth_method 0 2024-09-11T01:42:34.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.460+0000 7f64598aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:34.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.460+0000 7f64598aa640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:34.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.460+0000 7f64598aa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:42:34.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.460+0000 7f64598aa640 1 -- 172.21.15.12:0/4141293577 >> v2:172.21.15.71:3300/0 conn(0x7f6438003760 msgr2=0x7f6438005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:42:34.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.460+0000 7f64598aa640 1 -- 172.21.15.12:0/4141293577 >> v2:172.21.15.71:3300/0 conn(0x7f6438003760 msgr2=0x7f6438005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:42:34.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.460+0000 7f64598aa640 1 --2- 172.21.15.12:0/4141293577 >> v2:172.21.15.71:3300/0 conn(0x7f6438003760 0x7f6438005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:42:34.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.460+0000 7f64598aa640 1 --2- 172.21.15.12:0/4141293577 >> v2:172.21.15.71:3300/0 conn(0x7f6438003760 0x7f6438005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:42:34.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.862+0000 7f64598aa640 1 --2- 172.21.15.12:0/4141293577 >> v2:172.21.15.71:3300/0 conn(0x7f6438003760 0x7f6438005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:34.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.862+0000 7f64598aa640 10 monclient(hunting): get_auth_request con 0x7f6438003760 auth_method 0 2024-09-11T01:42:34.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.862+0000 7f64598aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:34.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.862+0000 7f64598aa640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:34.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.862+0000 7f64598aa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:42:34.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.862+0000 7f64598aa640 1 -- 172.21.15.12:0/4141293577 >> v2:172.21.15.71:3300/0 conn(0x7f6438003760 msgr2=0x7f6438005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:42:34.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.862+0000 7f64598aa640 1 -- 172.21.15.12:0/4141293577 >> v2:172.21.15.71:3300/0 conn(0x7f6438003760 msgr2=0x7f6438005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:42:34.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.862+0000 7f64598aa640 1 --2- 172.21.15.12:0/4141293577 >> v2:172.21.15.71:3300/0 conn(0x7f6438003760 0x7f6438005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:42:34.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:34.862+0000 7f64598aa640 1 --2- 172.21.15.12:0/4141293577 >> v2:172.21.15.71:3300/0 conn(0x7f6438003760 0x7f6438005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:42:35.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.215+0000 7f64588a8640 1 -- 172.21.15.12:0/4141293577 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 841719691 0 0) 0x7f64540a66a0 con 0x7f64380062d0 2024-09-11T01:42:35.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.215+0000 7f64588a8640 10 monclient(hunting): _negotiate 2024-09-11T01:42:35.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.215+0000 7f64588a8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:35.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.215+0000 7f64588a8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:35.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.215+0000 7f64588a8640 10 monclient(hunting): my global_id is 94623 2024-09-11T01:42:35.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.215+0000 7f64588a8640 1 -- 172.21.15.12:0/4141293577 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6440003520 con 0x7f64380062d0 2024-09-11T01:42:35.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 1 -- 172.21.15.12:0/4141293577 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3088574626 0 0) 0x7f6440003520 con 0x7f64380062d0 2024-09-11T01:42:35.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 1 -- 172.21.15.12:0/4141293577 >> v2:172.21.15.71:3300/0 conn(0x7f6438003760 msgr2=0x7f6438005be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:42:35.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 1 --2- 172.21.15.12:0/4141293577 >> v2:172.21.15.71:3300/0 conn(0x7f6438003760 0x7f6438005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 10 monclient: _finish_hunting 0 2024-09-11T01:42:35.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:42:35.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:35.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:42:35.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 1 -- 172.21.15.12:0/4141293577 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f643800b020 con 0x7f64380062d0 2024-09-11T01:42:35.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 10 monclient: _finish_auth 0 2024-09-11T01:42:35.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 10 monclient: _check_auth_tickets 2024-09-11T01:42:35.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:35.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 1 -- 172.21.15.12:0/4141293577 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f644400f040 con 0x7f64380062d0 2024-09-11T01:42:35.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:35.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:42:35.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 10 monclient: dump: 2024-09-11T01:42:35.224 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:35.224 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:35.224 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.224 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.225 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:35.225 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:35.225 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:35.225 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:35.225 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:35.225 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:35.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:42:35.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:35.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 1 -- 172.21.15.12:0/4141293577 >> v1:172.21.15.12:6789/0 conn(0x7f64380062d0 legacy=0x7f6438008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:35.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f645a0ab640 1 -- 172.21.15.12:0/4141293577 reap_dead start 2024-09-11T01:42:35.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:42:35.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 1 --2- 172.21.15.12:0/4141293577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64400039b0 0x7f6440003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:35.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 10 monclient(hunting): picked mon.b con 0x7f64400039b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:42:35.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 1 --2- 172.21.15.12:0/4141293577 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6440004800 0x7f6440004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:35.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 10 monclient(hunting): picked mon.c con 0x7f6440004800 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:35.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 1 --2- 172.21.15.12:0/4141293577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6440005290 0x7f644000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:35.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 10 monclient(hunting): picked mon.a con 0x7f6440005290 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:42:35.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:35.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 1 -- 172.21.15.12:0/4141293577 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f645409c9c0 con 0x7f6440005290 2024-09-11T01:42:35.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:35.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 1 -- 172.21.15.12:0/4141293577 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f644000c1e0 con 0x7f64400039b0 2024-09-11T01:42:35.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:35.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 1 -- 172.21.15.12:0/4141293577 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f644000c3e0 con 0x7f6440004800 2024-09-11T01:42:35.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:35.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64588a8640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:42:35.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.219+0000 7f64590a9640 1 -- 172.21.15.12:0/4141293577 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6440004800 msgr2=0x7f6440004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:42:35.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f64590a9640 1 --2- 172.21.15.12:0/4141293577 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6440004800 0x7f6440004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:35.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f645a0ab640 1 --2- 172.21.15.12:0/4141293577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6440005290 0x7f644000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:35.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f64598aa640 1 --2- 172.21.15.12:0/4141293577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64400039b0 0x7f6440003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:35.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f64598aa640 10 monclient(hunting): get_auth_request con 0x7f64400039b0 auth_method 0 2024-09-11T01:42:35.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f64598aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:35.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f64598aa640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:35.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f64598aa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:42:35.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f645a0ab640 10 monclient(hunting): get_auth_request con 0x7f6440005290 auth_method 0 2024-09-11T01:42:35.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f645a0ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:35.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f645a0ab640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:35.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f645a0ab640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:42:35.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f64598aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:35.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f64598aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:35.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f64598aa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:42:35.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f645a0ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:35.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f645a0ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:35.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f645a0ab640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:42:35.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f64598aa640 10 monclient(hunting): handle_auth_done global_id 94623 payload 293 2024-09-11T01:42:35.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f64598aa640 1 -- 172.21.15.12:0/4141293577 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6440004800 msgr2=0x7f6440004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:42:35.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f64598aa640 1 --2- 172.21.15.12:0/4141293577 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6440004800 0x7f6440004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f64598aa640 1 -- 172.21.15.12:0/4141293577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6440005290 msgr2=0x7f644000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:35.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f64598aa640 1 --2- 172.21.15.12:0/4141293577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6440005290 0x7f644000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f64598aa640 10 monclient: _finish_hunting 0 2024-09-11T01:42:35.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f64598aa640 1 monclient: found mon.b 2024-09-11T01:42:35.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f64598aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:35.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f64598aa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:42:35.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f64598aa640 1 -- 172.21.15.12:0/4141293577 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6440003520 con 0x7f64400039b0 2024-09-11T01:42:35.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f64598aa640 10 monclient: _finish_auth 0 2024-09-11T01:42:35.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f64598aa640 10 monclient: _check_auth_tickets 2024-09-11T01:42:35.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f64598aa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:35.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f645bb0e640 5 monclient: authenticate success, global_id 94623 2024-09-11T01:42:35.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f645bb0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:42:35.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.220+0000 7f645a0ab640 1 --2- 172.21.15.12:0/4141293577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6440005290 0x7f644000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:42:35.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.223+0000 7f64598aa640 1 -- 172.21.15.12:0/4141293577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64400039b0 msgr2=0x7f6440003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:35.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.223+0000 7f64598aa640 1 --2- 172.21.15.12:0/4141293577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64400039b0 0x7f6440003db0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6450009ee0 tx=0x7f6440005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddf27c88bdd58ec1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:35.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.223+0000 7f64588a8640 1 -- 172.21.15.12:0/4141293577 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6450031070 con 0x7f64400039b0 2024-09-11T01:42:35.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.223+0000 7f64588a8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.224+0000 7f64588a8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:42:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.224+0000 7f64588a8640 10 monclient: dump: 2024-09-11T01:42:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:35.237 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:35.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.224+0000 7f64588a8640 1 -- 172.21.15.12:0/4141293577 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f645002be10 con 0x7f64400039b0 2024-09-11T01:42:35.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.224+0000 7f645bb0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:42:35.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.224+0000 7f64588a8640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:35.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.224+0000 7f645bb0e640 10 monclient: get_monmap_and_config success 2024-09-11T01:42:35.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.224+0000 7f64588a8640 1 -- 172.21.15.12:0/4141293577 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6450009ae0 con 0x7f64400039b0 2024-09-11T01:42:35.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.224+0000 7f64588a8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:35.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.224+0000 7f645bb0e640 4 set_mon_vals no callback set 2024-09-11T01:42:35.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.224+0000 7f64588a8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:42:35.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.224+0000 7f64588a8640 10 monclient: dump: 2024-09-11T01:42:35.238 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:35.238 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:35.238 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.238 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.239 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:35.239 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:35.239 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:35.239 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:35.239 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:35.239 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:35.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.224+0000 7f645bb0e640 10 monclient: shutdown 2024-09-11T01:42:35.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.224+0000 7f645bb0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:35.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.224+0000 7f645bb0e640 1 -- 172.21.15.12:0/4141293577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64400039b0 msgr2=0x7f6440003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:35.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.224+0000 7f645bb0e640 1 --2- 172.21.15.12:0/4141293577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64400039b0 0x7f6440003db0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6450009ee0 tx=0x7f6440005830 comp rx=0 tx=0).stop 2024-09-11T01:42:35.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.224+0000 7f645bb0e640 1 -- 172.21.15.12:0/4141293577 shutdown_connections 2024-09-11T01:42:35.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.224+0000 7f645bb0e640 1 --2- 172.21.15.12:0/4141293577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6440005290 0x7f644000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.224+0000 7f645bb0e640 1 --2- 172.21.15.12:0/4141293577 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6440004800 0x7f6440004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.224+0000 7f645bb0e640 1 --2- 172.21.15.12:0/4141293577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64400039b0 0x7f6440003db0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.224+0000 7f645bb0e640 1 -- 172.21.15.12:0/4141293577 >> 172.21.15.12:0/4141293577 conn(0x7f64540a7120 msgr2=0x7f64540a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:35.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.225+0000 7f645bb0e640 1 -- 172.21.15.12:0/4141293577 shutdown_connections 2024-09-11T01:42:35.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.225+0000 7f645bb0e640 1 -- 172.21.15.12:0/4141293577 wait complete. 2024-09-11T01:42:35.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.225+0000 7f645bb0e640 10 monclient: build_initial_monmap 2024-09-11T01:42:35.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.225+0000 7f645bb0e640 10 monclient: monmap: 2024-09-11T01:42:35.240 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:35.240 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:42:35.240 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:42:35.240 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:42:35.240 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:35.240 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:35.240 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:42:35.240 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:42:35.240 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:42:35.241 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.226+0000 7f645bb0e640 1 Processor -- start 2024-09-11T01:42:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.226+0000 7f645bb0e640 1 -- start start 2024-09-11T01:42:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.226+0000 7f645bb0e640 10 monclient: init 2024-09-11T01:42:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.226+0000 7f645bb0e640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.226+0000 7f645bb0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:42:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.226+0000 7f645bb0e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64400039b0 0x7f64541ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.226+0000 7f645bb0e640 10 monclient(hunting): picked mon.noname-b con 0x7f64400039b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:42:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.226+0000 7f645bb0e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6440004800 0x7f64541cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.226+0000 7f645bb0e640 10 monclient(hunting): picked mon.noname-c con 0x7f6440004800 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.226+0000 7f645bb0e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6440005290 0x7f6454124770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.226+0000 7f645bb0e640 10 monclient(hunting): picked mon.noname-a con 0x7f6440005290 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:42:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.226+0000 7f645bb0e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.226+0000 7f645bb0e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6454138790 con 0x7f6440005290 2024-09-11T01:42:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.226+0000 7f645bb0e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.226+0000 7f645bb0e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6454138610 con 0x7f64400039b0 2024-09-11T01:42:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.226+0000 7f645bb0e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.226+0000 7f645bb0e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6454138910 con 0x7f6440004800 2024-09-11T01:42:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.226+0000 7f645bb0e640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.226+0000 7f645bb0e640 10 monclient(hunting): authenticate will time out at 1561.668855s 2024-09-11T01:42:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.226+0000 7f64590a9640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6440004800 msgr2=0x7f64541cd2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:42:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.226+0000 7f64590a9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6440004800 0x7f64541cd2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.226+0000 7f645a0ab640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6440005290 0x7f6454124770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f645a0ab640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6440005290 0x7f6454124770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35782/0 (socket says 172.21.15.12:35782) 2024-09-11T01:42:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f645a0ab640 1 -- 172.21.15.12:0/567084702 learned_addr learned my addr 172.21.15.12:0/567084702 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:35.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f645a0ab640 10 monclient(hunting): get_auth_request con 0x7f6440005290 auth_method 0 2024-09-11T01:42:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f645a0ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f645a0ab640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f645a0ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f64598aa640 1 --2- 172.21.15.12:0/567084702 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64400039b0 0x7f64541ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f64598aa640 10 monclient(hunting): get_auth_request con 0x7f64400039b0 auth_method 0 2024-09-11T01:42:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f64598aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f64598aa640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f64598aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f645a0ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f645a0ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f645a0ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f64598aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f64598aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f64598aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f645a0ab640 10 monclient(hunting): handle_auth_done global_id 94626 payload 824 2024-09-11T01:42:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f645a0ab640 1 -- 172.21.15.12:0/567084702 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6440004800 msgr2=0x7f64541cd2f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:42:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f645a0ab640 1 --2- 172.21.15.12:0/567084702 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6440004800 0x7f64541cd2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f645a0ab640 1 -- 172.21.15.12:0/567084702 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64400039b0 msgr2=0x7f64541ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f645a0ab640 1 --2- 172.21.15.12:0/567084702 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64400039b0 0x7f64541ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f645a0ab640 10 monclient: _finish_hunting 0 2024-09-11T01:42:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f645a0ab640 1 monclient: found mon.noname-a 2024-09-11T01:42:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f645a0ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f645a0ab640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:42:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.227+0000 7f645a0ab640 1 -- 172.21.15.12:0/567084702 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6454126230 con 0x7f6440005290 2024-09-11T01:42:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.229+0000 7f645a0ab640 1 -- 172.21.15.12:0/567084702 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6440005290 msgr2=0x7f6454124770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.229+0000 7f645a0ab640 1 --2- 172.21.15.12:0/567084702 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6440005290 0x7f6454124770 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6438006240 tx=0x7f644c00fe10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.229+0000 7f644a7fc640 1 -- 172.21.15.12:0/567084702 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644c00d700 con 0x7f6440005290 2024-09-11T01:42:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.229+0000 7f644a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.229+0000 7f644a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:42:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.229+0000 7f644a7fc640 10 monclient: dump: 2024-09-11T01:42:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:35.245 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.229+0000 7f644a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:42:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.229+0000 7f644a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:42:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.229+0000 7f644a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.229+0000 7f645bb0e640 5 monclient: authenticate success, global_id 94626 2024-09-11T01:42:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.229+0000 7f644a7fc640 1 -- 172.21.15.12:0/567084702 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f644c018070 con 0x7f6440005290 2024-09-11T01:42:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.229+0000 7f644a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.229+0000 7f644a7fc640 1 -- 172.21.15.12:0/567084702 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644c00daa0 con 0x7f6440005290 2024-09-11T01:42:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.229+0000 7f645bb0e640 10 monclient: _renew_subs 2024-09-11T01:42:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.229+0000 7f645bb0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.229+0000 7f645bb0e640 1 -- 172.21.15.12:0/567084702 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6454128590 con 0x7f6440005290 2024-09-11T01:42:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.229+0000 7f644a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.229+0000 7f644a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.229+0000 7f644a7fc640 10 monclient: dump: 2024-09-11T01:42:35.247 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:35.247 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:35.247 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.247 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.247 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:35.247 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:35.247 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:35.247 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:35.247 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:35.247 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:35.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.229+0000 7f644bfff640 4 set_mon_vals no callback set 2024-09-11T01:42:35.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.230+0000 7f645bb0e640 10 monclient: _renew_subs 2024-09-11T01:42:35.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.230+0000 7f645bb0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:35.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.230+0000 7f645bb0e640 1 -- 172.21.15.12:0/567084702 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6454137b50 con 0x7f6440005290 2024-09-11T01:42:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.231+0000 7f644a7fc640 1 -- 172.21.15.12:0/567084702 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f644c00dc40 con 0x7f6440005290 2024-09-11T01:42:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.231+0000 7f645bb0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.231+0000 7f645bb0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.231+0000 7f645bb0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.231+0000 7f645bb0e640 1 -- 172.21.15.12:0/567084702 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64540a5020 con 0x7f6440005290 2024-09-11T01:42:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.231+0000 7f644a7fc640 1 --2- 172.21.15.12:0/567084702 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f643803ebf0 0x7f64380410b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.231+0000 7f644a7fc640 1 -- 172.21.15.12:0/567084702 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(442..442 src has 285..442) ==== 4787+0+0 (secure 0 0 0) 0x7f644c052a80 con 0x7f6440005290 2024-09-11T01:42:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.231+0000 7f64598aa640 1 --2- 172.21.15.12:0/567084702 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f643803ebf0 0x7f64380410b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.231+0000 7f64598aa640 10 monclient: get_auth_request con 0x7f643803ebf0 auth_method 0 2024-09-11T01:42:35.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.234+0000 7f64598aa640 1 --2- 172.21.15.12:0/567084702 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f643803ebf0 0x7f64380410b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6450033000 tx=0x7f6450031000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:35.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.236+0000 7f644a7fc640 1 -- 172.21.15.12:0/567084702 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f644c051470 con 0x7f6440005290 2024-09-11T01:42:35.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.236+0000 7f644a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:35.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.236+0000 7f644a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:42:35.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.495+0000 7f645bb0e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:42:35.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.495+0000 7f645bb0e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:42:35.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.495+0000 7f645bb0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:35.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.495+0000 7f645bb0e640 1 -- 172.21.15.12:0/567084702 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f64540698e0 con 0x7f6440005290 2024-09-11T01:42:35.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.495+0000 7f644a7fc640 1 -- 172.21.15.12:0/567084702 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f644c052e20 con 0x7f6440005290 2024-09-11T01:42:35.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.496+0000 7f644a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:42:35.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.496+0000 7f644a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:42:35.498 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":126,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:42:35.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.498+0000 7f645bb0e640 1 -- 172.21.15.12:0/567084702 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f643803ebf0 msgr2=0x7f64380410b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:35.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.498+0000 7f645bb0e640 1 --2- 172.21.15.12:0/567084702 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f643803ebf0 0x7f64380410b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6450033000 tx=0x7f6450031000 comp rx=0 tx=0).stop 2024-09-11T01:42:35.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.498+0000 7f645bb0e640 10 monclient: shutdown 2024-09-11T01:42:35.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.498+0000 7f645bb0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:35.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.498+0000 7f645bb0e640 1 -- 172.21.15.12:0/567084702 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6440005290 msgr2=0x7f6454124770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:35.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.498+0000 7f645bb0e640 1 --2- 172.21.15.12:0/567084702 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6440005290 0x7f6454124770 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6438006240 tx=0x7f644c00fe10 comp rx=0 tx=0).stop 2024-09-11T01:42:35.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.498+0000 7f645bb0e640 1 -- 172.21.15.12:0/567084702 shutdown_connections 2024-09-11T01:42:35.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.498+0000 7f645bb0e640 1 --2- 172.21.15.12:0/567084702 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f643803ebf0 0x7f64380410b0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.498+0000 7f645bb0e640 1 --2- 172.21.15.12:0/567084702 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6440005290 0x7f6454124770 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.498+0000 7f645bb0e640 1 --2- 172.21.15.12:0/567084702 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6440004800 0x7f64541cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.498+0000 7f645bb0e640 1 --2- 172.21.15.12:0/567084702 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64400039b0 0x7f64541ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.498+0000 7f645bb0e640 1 -- 172.21.15.12:0/567084702 >> 172.21.15.12:0/567084702 conn(0x7f64540a7120 msgr2=0x7f64540a8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:35.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.498+0000 7f645bb0e640 1 -- 172.21.15.12:0/567084702 shutdown_connections 2024-09-11T01:42:35.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.498+0000 7f645bb0e640 1 -- 172.21.15.12:0/567084702 wait complete. 2024-09-11T01:42:35.510 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:42:35.510 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-11T01:42:35.510 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:42:35.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.610+0000 7ff17cfec640 10 monclient: get_monmap_and_config 2024-09-11T01:42:35.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.610+0000 7ff17cfec640 10 monclient: build_initial_monmap 2024-09-11T01:42:35.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.610+0000 7ff17cfec640 10 monclient: monmap: 2024-09-11T01:42:35.612 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:35.612 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:35.612 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:42:35.611613+0000 2024-09-11T01:42:35.613 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:42:35.611613+0000 2024-09-11T01:42:35.613 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:35.613 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:35.613 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:42:35.613 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:42:35.613 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:42:35.613 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:42:35.613 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:42:35.613 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:35.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.611+0000 7ff17cfec640 1 Processor -- start 2024-09-11T01:42:35.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.611+0000 7ff17cfec640 1 -- start start 2024-09-11T01:42:35.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.611+0000 7ff17cfec640 10 monclient: init 2024-09-11T01:42:35.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.612+0000 7ff17cfec640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:35.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.612+0000 7ff17cfec640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-11T01:42:35.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.612+0000 7ff17cfec640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff1780a2e20 0x7ff17806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:35.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.612+0000 7ff17cfec640 10 monclient(hunting): picked mon.noname-b con 0x7ff1780a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:35.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.612+0000 7ff17cfec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff1780a4970 addr v1:172.21.15.71:6789/0 2024-09-11T01:42:35.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.612+0000 7ff17cfec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff17806f910 addr v1:172.21.15.12:6789/0 2024-09-11T01:42:35.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.612+0000 7ff17cfec640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff17806f4f0 con 0x7ff17806f910 2024-09-11T01:42:35.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.612+0000 7ff17cfec640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1780a3910 con 0x7ff1780a4970 2024-09-11T01:42:35.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.612+0000 7ff17cfec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:35.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.612+0000 7ff17cfec640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff17806ef30 con 0x7ff1780a2e20 2024-09-11T01:42:35.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.612+0000 7ff17cfec640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:35.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.612+0000 7ff17cfec640 10 monclient(hunting): authenticate will time out at 1562.054208s 2024-09-11T01:42:35.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.612+0000 7ff17659c640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff1780a2e20 msgr2=0x7ff17806ded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:42:35.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.612+0000 7ff17659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff1780a2e20 0x7ff17806ded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:35.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.612+0000 7ff176d9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff17806f910 0x7ff17813f430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57582/0 (socket says 172.21.15.12:57582) 2024-09-11T01:42:35.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.612+0000 7ff176d9d640 1 -- 172.21.15.12:0/403587155 learned_addr learned my addr 172.21.15.12:0/403587155 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:42:35.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff176d9d640 1 -- 172.21.15.12:0/403587155 >> v1:172.21.15.12:6789/0 conn(0x7ff17806f910 legacy=0x7ff17813f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:35.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17559a640 1 -- 172.21.15.12:0/403587155 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3618268146 0 0) 0x7ff17806f4f0 con 0x7ff17806f910 2024-09-11T01:42:35.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17559a640 10 monclient(hunting): _negotiate 2024-09-11T01:42:35.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17559a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:35.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:35.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17559a640 10 monclient(hunting): my global_id is 94629 2024-09-11T01:42:35.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17559a640 1 -- 172.21.15.12:0/403587155 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff160003520 con 0x7ff17806f910 2024-09-11T01:42:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff175d9b640 1 -- 172.21.15.12:0/403587155 >> v1:172.21.15.71:6789/0 conn(0x7ff1780a4970 legacy=0x7ff1780a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17559a640 1 -- 172.21.15.12:0/403587155 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3366207468 0 0) 0x7ff160003520 con 0x7ff17806f910 2024-09-11T01:42:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17559a640 1 -- 172.21.15.12:0/403587155 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff1780a2e20 msgr2=0x7ff17806ded0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:42:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17559a640 1 --2- 172.21.15.12:0/403587155 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff1780a2e20 0x7ff17806ded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17559a640 1 -- 172.21.15.12:0/403587155 >> v1:172.21.15.71:6789/0 conn(0x7ff1780a4970 legacy=0x7ff1780a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17559a640 10 monclient: _finish_hunting 0 2024-09-11T01:42:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17559a640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:42:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17559a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:42:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17559a640 1 -- 172.21.15.12:0/403587155 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1780a3ae0 con 0x7ff17806f910 2024-09-11T01:42:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17559a640 10 monclient: _finish_auth 0 2024-09-11T01:42:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17559a640 10 monclient: _check_auth_tickets 2024-09-11T01:42:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17cfec640 5 monclient: authenticate success, global_id 94629 2024-09-11T01:42:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17559a640 1 -- 172.21.15.12:0/403587155 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7ff170001d20 con 0x7ff17806f910 2024-09-11T01:42:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17cfec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:42:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17559a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:42:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17559a640 10 monclient: dump: 2024-09-11T01:42:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:35.628 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:35.628 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:35.628 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:35.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17559a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:42:35.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17559a640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:35.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17559a640 1 -- 172.21.15.12:0/403587155 >> v1:172.21.15.12:6789/0 conn(0x7ff17806f910 legacy=0x7ff17813f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:35.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.614+0000 7ff17559a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:42:35.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17559a640 1 --2- 172.21.15.12:0/403587155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff160003a30 0x7ff160003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:35.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17559a640 10 monclient(hunting): picked mon.a con 0x7ff160003a30 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:42:35.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17559a640 1 --2- 172.21.15.12:0/403587155 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff160004800 0x7ff160004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:35.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17559a640 10 monclient(hunting): picked mon.b con 0x7ff160004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:42:35.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17559a640 1 --2- 172.21.15.12:0/403587155 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff1780a2e20 0x7ff16000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:35.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17559a640 10 monclient(hunting): picked mon.c con 0x7ff1780a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:35.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17559a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:35.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17559a640 1 -- 172.21.15.12:0/403587155 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff17806ef30 con 0x7ff160003a30 2024-09-11T01:42:35.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17559a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:35.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17559a640 1 -- 172.21.15.12:0/403587155 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff16000bf00 con 0x7ff160004800 2024-09-11T01:42:35.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17559a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:35.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17559a640 1 -- 172.21.15.12:0/403587155 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff16000c080 con 0x7ff1780a2e20 2024-09-11T01:42:35.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17559a640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:35.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17559a640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:42:35.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17cfec640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:42:35.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff176d9d640 1 -- 172.21.15.12:0/403587155 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff1780a2e20 msgr2=0x7ff16000b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:42:35.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17659c640 1 --2- 172.21.15.12:0/403587155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff160003a30 0x7ff160003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:35.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff176d9d640 1 --2- 172.21.15.12:0/403587155 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff1780a2e20 0x7ff16000b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:35.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17659c640 10 monclient(hunting): get_auth_request con 0x7ff160003a30 auth_method 0 2024-09-11T01:42:35.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:35.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:35.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:42:35.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:35.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:35.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff175d9b640 1 --2- 172.21.15.12:0/403587155 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff160004800 0x7ff160004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:35.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:42:35.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff175d9b640 10 monclient(hunting): get_auth_request con 0x7ff160004800 auth_method 0 2024-09-11T01:42:35.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff175d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:35.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff175d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:35.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff175d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:42:35.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff175d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:35.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff175d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:35.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff175d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:42:35.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17659c640 10 monclient(hunting): handle_auth_done global_id 94629 payload 293 2024-09-11T01:42:35.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17659c640 1 -- 172.21.15.12:0/403587155 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff1780a2e20 msgr2=0x7ff16000b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:42:35.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17659c640 1 --2- 172.21.15.12:0/403587155 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff1780a2e20 0x7ff16000b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17659c640 1 -- 172.21.15.12:0/403587155 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff160004800 msgr2=0x7ff160004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:35.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17659c640 1 --2- 172.21.15.12:0/403587155 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff160004800 0x7ff160004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17659c640 10 monclient: _finish_hunting 0 2024-09-11T01:42:35.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17659c640 1 monclient: found mon.a 2024-09-11T01:42:35.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:35.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17659c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:35.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17659c640 1 -- 172.21.15.12:0/403587155 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff160003520 con 0x7ff160003a30 2024-09-11T01:42:35.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17659c640 10 monclient: _finish_auth 0 2024-09-11T01:42:35.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17659c640 10 monclient: _check_auth_tickets 2024-09-11T01:42:35.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.615+0000 7ff17659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:35.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.618+0000 7ff17659c640 1 -- 172.21.15.12:0/403587155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff160003a30 msgr2=0x7ff160003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:35.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.618+0000 7ff17659c640 1 --2- 172.21.15.12:0/403587155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff160003a30 0x7ff160003e50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff168009900 tx=0x7ff160005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbffb0ec2f74f7e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:35.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.618+0000 7ff17559a640 1 -- 172.21.15.12:0/403587155 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff168009500 con 0x7ff160003a30 2024-09-11T01:42:35.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.618+0000 7ff17559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:35.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.618+0000 7ff17559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:35.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.618+0000 7ff17559a640 10 monclient: dump: 2024-09-11T01:42:35.632 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:35.632 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:35.632 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.633 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.633 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:35.633 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:35.633 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:35.633 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:35.633 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:35.633 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:35.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.618+0000 7ff17559a640 1 -- 172.21.15.12:0/403587155 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff16802e070 con 0x7ff160003a30 2024-09-11T01:42:35.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.618+0000 7ff17cfec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:42:35.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.618+0000 7ff17559a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:35.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.618+0000 7ff17cfec640 10 monclient: get_monmap_and_config success 2024-09-11T01:42:35.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.618+0000 7ff17559a640 1 -- 172.21.15.12:0/403587155 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16802a910 con 0x7ff160003a30 2024-09-11T01:42:35.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.618+0000 7ff17cfec640 4 set_mon_vals no callback set 2024-09-11T01:42:35.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.618+0000 7ff17559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.618+0000 7ff17559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.618+0000 7ff17559a640 10 monclient: dump: 2024-09-11T01:42:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:35.634 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:35.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.618+0000 7ff17cfec640 10 monclient: shutdown 2024-09-11T01:42:35.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.618+0000 7ff17cfec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:35.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.618+0000 7ff17cfec640 1 -- 172.21.15.12:0/403587155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff160003a30 msgr2=0x7ff160003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:35.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.618+0000 7ff17cfec640 1 --2- 172.21.15.12:0/403587155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff160003a30 0x7ff160003e50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff168009900 tx=0x7ff160005830 comp rx=0 tx=0).stop 2024-09-11T01:42:35.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.618+0000 7ff176d9d640 1 -- 172.21.15.12:0/403587155 reap_dead start 2024-09-11T01:42:35.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.618+0000 7ff17cfec640 1 -- 172.21.15.12:0/403587155 shutdown_connections 2024-09-11T01:42:35.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.618+0000 7ff17cfec640 1 -- 172.21.15.12:0/403587155 >> 172.21.15.12:0/403587155 conn(0x7ff1780a8600 msgr2=0x7ff1780a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:35.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.618+0000 7ff17cfec640 1 -- 172.21.15.12:0/403587155 shutdown_connections 2024-09-11T01:42:35.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.618+0000 7ff17cfec640 1 -- 172.21.15.12:0/403587155 wait complete. 2024-09-11T01:42:35.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.619+0000 7ff17cfec640 10 monclient: build_initial_monmap 2024-09-11T01:42:35.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.619+0000 7ff17cfec640 10 monclient: monmap: 2024-09-11T01:42:35.635 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:35.635 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:42:35.636 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:42:35.636 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:42:35.636 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:35.636 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:35.636 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:42:35.636 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:42:35.636 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:42:35.636 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:35.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.619+0000 7ff17cfec640 1 Processor -- start 2024-09-11T01:42:35.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17cfec640 1 -- start start 2024-09-11T01:42:35.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17cfec640 10 monclient: init 2024-09-11T01:42:35.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17cfec640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:35.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17cfec640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:42:35.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17cfec640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff160003a30 0x7ff178125880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:35.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17cfec640 10 monclient(hunting): picked mon.noname-a con 0x7ff160003a30 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:42:35.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17cfec640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff178125dc0 0x7ff178126200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:35.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17cfec640 10 monclient(hunting): picked mon.noname-c con 0x7ff178125dc0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:35.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17cfec640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1781cd4b0 0x7ff1781cd940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:35.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17cfec640 10 monclient(hunting): picked mon.noname-b con 0x7ff1781cd4b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:42:35.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17cfec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:35.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17cfec640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff17806e260 con 0x7ff160003a30 2024-09-11T01:42:35.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17cfec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:35.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17cfec640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff17806e0e0 con 0x7ff1781cd4b0 2024-09-11T01:42:35.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17cfec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:35.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17cfec640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff17806ec50 con 0x7ff178125dc0 2024-09-11T01:42:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17cfec640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17cfec640 10 monclient(hunting): authenticate will time out at 1562.062546s 2024-09-11T01:42:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff175d9b640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff178125dc0 msgr2=0x7ff178126200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:42:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff175d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff178125dc0 0x7ff178126200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff160003a30 0x7ff178125880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff160003a30 0x7ff178125880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35794/0 (socket says 172.21.15.12:35794) 2024-09-11T01:42:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17659c640 1 -- 172.21.15.12:0/4195049152 learned_addr learned my addr 172.21.15.12:0/4195049152 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17659c640 10 monclient(hunting): get_auth_request con 0x7ff160003a30 auth_method 0 2024-09-11T01:42:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff176d9d640 1 --2- 172.21.15.12:0/4195049152 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1781cd4b0 0x7ff1781cd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff176d9d640 10 monclient(hunting): get_auth_request con 0x7ff1781cd4b0 auth_method 0 2024-09-11T01:42:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff176d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff176d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff176d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff176d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff176d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff176d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17659c640 10 monclient(hunting): handle_auth_done global_id 94632 payload 824 2024-09-11T01:42:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17659c640 1 -- 172.21.15.12:0/4195049152 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff178125dc0 msgr2=0x7ff178126200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:42:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17659c640 1 --2- 172.21.15.12:0/4195049152 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff178125dc0 0x7ff178126200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17659c640 1 -- 172.21.15.12:0/4195049152 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1781cd4b0 msgr2=0x7ff1781cd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17659c640 1 --2- 172.21.15.12:0/4195049152 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1781cd4b0 0x7ff1781cd940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17659c640 10 monclient: _finish_hunting 0 2024-09-11T01:42:35.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17659c640 1 monclient: found mon.noname-a 2024-09-11T01:42:35.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:35.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:42:35.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.620+0000 7ff17659c640 1 -- 172.21.15.12:0/4195049152 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1781299a0 con 0x7ff160003a30 2024-09-11T01:42:35.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.623+0000 7ff17659c640 1 -- 172.21.15.12:0/4195049152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff160003a30 msgr2=0x7ff178125880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:35.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.623+0000 7ff17659c640 1 --2- 172.21.15.12:0/4195049152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff160003a30 0x7ff178125880 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff168000c00 tx=0x7ff168005a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:35.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.623+0000 7ff166ffd640 1 -- 172.21.15.12:0/4195049152 DispatchQueue::entry inject delay of 0.002000 2024-09-11T01:42:35.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.625+0000 7ff166ffd640 1 -- 172.21.15.12:0/4195049152 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16802ae50 con 0x7ff160003a30 2024-09-11T01:42:35.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.625+0000 7ff166ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:35.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.625+0000 7ff166ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:42:35.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.625+0000 7ff166ffd640 10 monclient: dump: 2024-09-11T01:42:35.644 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:35.645 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.625+0000 7ff166ffd640 10 monclient: _finish_auth 0 2024-09-11T01:42:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.625+0000 7ff166ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:42:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.625+0000 7ff166ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.625+0000 7ff166ffd640 1 -- 172.21.15.12:0/4195049152 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff168043070 con 0x7ff160003a30 2024-09-11T01:42:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.625+0000 7ff166ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.625+0000 7ff166ffd640 1 -- 172.21.15.12:0/4195049152 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1680054c0 con 0x7ff160003a30 2024-09-11T01:42:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.625+0000 7ff166ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.625+0000 7ff166ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.625+0000 7ff166ffd640 10 monclient: dump: 2024-09-11T01:42:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:35.646 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:35.647 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:35.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.625+0000 7ff17cfec640 5 monclient: authenticate success, global_id 94632 2024-09-11T01:42:35.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.625+0000 7ff17cfec640 10 monclient: _renew_subs 2024-09-11T01:42:35.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.625+0000 7ff17cfec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:35.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.625+0000 7ff17cfec640 1 -- 172.21.15.12:0/4195049152 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff178129580 con 0x7ff160003a30 2024-09-11T01:42:35.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.625+0000 7ff167fff640 4 set_mon_vals no callback set 2024-09-11T01:42:35.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.625+0000 7ff17cfec640 10 monclient: _renew_subs 2024-09-11T01:42:35.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.625+0000 7ff17cfec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:35.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.625+0000 7ff17cfec640 1 -- 172.21.15.12:0/4195049152 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff178128710 con 0x7ff160003a30 2024-09-11T01:42:35.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.626+0000 7ff17cfec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:35.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.626+0000 7ff17cfec640 1 --2- 172.21.15.12:0/4195049152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1781cde80 0x7ff17809f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:35.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.626+0000 7ff17cfec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:35.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.626+0000 7ff17cfec640 1 -- 172.21.15.12:0/4195049152 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff17806a060 con 0x7ff1781cde80 2024-09-11T01:42:35.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.626+0000 7ff175d9b640 1 --2- 172.21.15.12:0/4195049152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1781cde80 0x7ff17809f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.626+0000 7ff166ffd640 1 -- 172.21.15.12:0/4195049152 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff168033470 con 0x7ff160003a30 2024-09-11T01:42:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.626+0000 7ff175d9b640 10 monclient: get_auth_request con 0x7ff1781cde80 auth_method 0 2024-09-11T01:42:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.626+0000 7ff175d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.626+0000 7ff175d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.626+0000 7ff175d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.626+0000 7ff175d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.626+0000 7ff175d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.626+0000 7ff175d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.626+0000 7ff166ffd640 1 --2- 172.21.15.12:0/4195049152 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff15803e3b0 0x7ff158040870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.626+0000 7ff175d9b640 10 monclient(hunting): handle_auth_done global_id 94635 payload 824 2024-09-11T01:42:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.626+0000 7ff175d9b640 1 -- 172.21.15.12:0/4195049152 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff1780a3200 con 0x7ff1781cde80 2024-09-11T01:42:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.626+0000 7ff166ffd640 1 -- 172.21.15.12:0/4195049152 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(442..442 src has 285..442) ==== 4787+0+0 (secure 0 0 0) 0x7ff168072f00 con 0x7ff160003a30 2024-09-11T01:42:35.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.626+0000 7ff176d9d640 1 --2- 172.21.15.12:0/4195049152 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff15803e3b0 0x7ff158040870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.626+0000 7ff176d9d640 10 monclient: get_auth_request con 0x7ff15803e3b0 auth_method 0 2024-09-11T01:42:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.628+0000 7ff175d9b640 1 -- 172.21.15.12:0/4195049152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1781cde80 msgr2=0x7ff17809f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.628+0000 7ff175d9b640 1 --2- 172.21.15.12:0/4195049152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1781cde80 0x7ff17809f9d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff16c001c60 tx=0x7ff16c006ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.629+0000 7ff166ffd640 1 -- 172.21.15.12:0/4195049152 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16c017020 con 0x7ff1781cde80 2024-09-11T01:42:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.629+0000 7ff166ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.629+0000 7ff166ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.629+0000 7ff166ffd640 10 monclient: dump: 2024-09-11T01:42:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:35.649 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:35.650 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:35.650 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:35.650 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:35.650 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:35.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.629+0000 7ff176d9d640 1 --2- 172.21.15.12:0/4195049152 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff15803e3b0 0x7ff158040870 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff1600039b0 tx=0x7ff1700017e0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:35.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.629+0000 7ff166ffd640 1 -- 172.21.15.12:0/4195049152 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff16c006ee0 con 0x7ff1781cde80 2024-09-11T01:42:35.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.629+0000 7ff166ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:35.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.629+0000 7ff166ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:42:35.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.629+0000 7ff166ffd640 1 -- 172.21.15.12:0/4195049152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1781cde80 msgr2=0x7ff17809f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:35.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.629+0000 7ff166ffd640 1 --2- 172.21.15.12:0/4195049152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1781cde80 0x7ff17809f9d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff16c001c60 tx=0x7ff16c006ba0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.629+0000 7ff166ffd640 1 -- 172.21.15.12:0/4195049152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1781cde80 msgr2=0x7ff17809f9d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:42:35.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.629+0000 7ff166ffd640 1 --2- 172.21.15.12:0/4195049152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff1781cde80 0x7ff17809f9d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.635+0000 7ff17cfec640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:42:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.635+0000 7ff17cfec640 1 --2- 172.21.15.12:0/4195049152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff17809ff10 0x7ff1780a02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.635+0000 7ff17cfec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.635+0000 7ff17cfec640 1 -- 172.21.15.12:0/4195049152 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff178127220 con 0x7ff17809ff10 2024-09-11T01:42:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.635+0000 7ff175d9b640 1 --2- 172.21.15.12:0/4195049152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff17809ff10 0x7ff1780a02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.635+0000 7ff175d9b640 10 monclient: get_auth_request con 0x7ff17809ff10 auth_method 0 2024-09-11T01:42:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.635+0000 7ff175d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.635+0000 7ff175d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.635+0000 7ff175d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.635+0000 7ff175d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.635+0000 7ff175d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.635+0000 7ff175d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.635+0000 7ff175d9b640 10 monclient(hunting): handle_auth_done global_id 94638 payload 824 2024-09-11T01:42:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.635+0000 7ff175d9b640 1 -- 172.21.15.12:0/4195049152 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff1781ce260 con 0x7ff17809ff10 2024-09-11T01:42:35.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.638+0000 7ff175d9b640 1 -- 172.21.15.12:0/4195049152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff17809ff10 msgr2=0x7ff1780a02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:35.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.638+0000 7ff175d9b640 1 --2- 172.21.15.12:0/4195049152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff17809ff10 0x7ff1780a02f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff16c001c60 tx=0x7ff16c001520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:35.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.638+0000 7ff166ffd640 1 -- 172.21.15.12:0/4195049152 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16c017040 con 0x7ff17809ff10 2024-09-11T01:42:35.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.638+0000 7ff166ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:35.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.638+0000 7ff166ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:35.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.638+0000 7ff166ffd640 10 monclient: dump: 2024-09-11T01:42:35.652 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:35.652 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:35.652 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.652 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.652 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:35.652 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:35.652 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:35.653 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.638+0000 7ff166ffd640 1 -- 172.21.15.12:0/4195049152 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4700 (secure 0 0 0) 0x7ff16c01c830 con 0x7ff17809ff10 2024-09-11T01:42:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.638+0000 7ff166ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:42:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.638+0000 7ff166ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:42:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.638+0000 7ff166ffd640 1 -- 172.21.15.12:0/4195049152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff17809ff10 msgr2=0x7ff1780a02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.638+0000 7ff166ffd640 1 --2- 172.21.15.12:0/4195049152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff17809ff10 0x7ff1780a02f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff16c001c60 tx=0x7ff16c001520 comp rx=0 tx=0).stop 2024-09-11T01:42:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.638+0000 7ff166ffd640 1 -- 172.21.15.12:0/4195049152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff17809ff10 msgr2=0x7ff1780a02f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:42:35.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.638+0000 7ff166ffd640 1 --2- 172.21.15.12:0/4195049152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff17809ff10 0x7ff1780a02f0 secure :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff16c001c60 tx=0x7ff16c001520 comp rx=0 tx=0).stop 2024-09-11T01:42:35.653 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:42:35.654 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:42:35.654 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:42:35.654 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:42:35.654 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 126, 2024-09-11T01:42:35.654 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 301734, 2024-09-11T01:42:35.654 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:42:35.654 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:42:35.654 INFO:teuthology.orchestra.run.smithi012.stdout: 1 2024-09-11T01:42:35.654 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:35.654 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:42:35.654 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:42:35.654 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:42:35.654 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:42:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:42:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:42:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:42:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:42:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:42:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:42:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:42:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:42:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:42:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:42:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:42:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:35.655 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:42:35.656 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:42:35.656 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:42:35.656 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:42:35.656 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:42:35.656 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:42:35.656 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:42:35.656 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:42:35.656 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:42:35.656 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:42:35.656 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:42:35.656 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:42:35.656 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:42:35.656 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:42:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:42:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:42:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:42:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:42:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:42:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:42:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:42:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:42:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:42:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:42:35.657 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:42:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:42:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:42:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:42:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:42:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:42:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:42:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:42:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:42:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:42:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:42:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:42:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:42:35.658 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:42:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:42:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:42:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:42:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:42:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:42:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:42:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:42:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:42:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:42:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:35.659 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:42:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:42:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:42:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:42:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:42:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:42:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:42:35.660 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:42:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:42:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:42:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:42:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:42:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:42:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:42:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:42:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:35.661 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:42:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:42:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:42:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:42:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:42:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:42:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:42:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:42:35.662 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:42:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:42:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:42:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:42:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:42:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:42:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:42:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:35.663 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:42:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:42:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:42:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:42:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:42:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:42:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:42:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:42:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:42:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:42:35.664 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:42:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:42:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:42:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:42:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:42:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:42:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:42:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:42:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:35.665 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:35.666 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:35.666 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:42:35.666 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:42:35.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.638+0000 7ff17cfec640 1 -- 172.21.15.12:0/4195049152 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff15803e3b0 msgr2=0x7ff158040870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:35.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.638+0000 7ff17cfec640 1 --2- 172.21.15.12:0/4195049152 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff15803e3b0 0x7ff158040870 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff1600039b0 tx=0x7ff1700017e0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.638+0000 7ff17cfec640 10 monclient: shutdown 2024-09-11T01:42:35.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.638+0000 7ff17cfec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:35.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.638+0000 7ff17cfec640 1 -- 172.21.15.12:0/4195049152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff160003a30 msgr2=0x7ff178125880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:35.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.638+0000 7ff17cfec640 1 --2- 172.21.15.12:0/4195049152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff160003a30 0x7ff178125880 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff168000c00 tx=0x7ff168005a10 comp rx=0 tx=0).stop 2024-09-11T01:42:35.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.638+0000 7ff176d9d640 1 -- 172.21.15.12:0/4195049152 reap_dead start 2024-09-11T01:42:35.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.638+0000 7ff176d9d640 1 -- 172.21.15.12:0/4195049152 reap_dead start 2024-09-11T01:42:35.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.638+0000 7ff17cfec640 1 -- 172.21.15.12:0/4195049152 shutdown_connections 2024-09-11T01:42:35.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.638+0000 7ff17cfec640 1 -- 172.21.15.12:0/4195049152 >> 172.21.15.12:0/4195049152 conn(0x7ff1780a8600 msgr2=0x7ff1780a1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:35.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.638+0000 7ff17cfec640 1 -- 172.21.15.12:0/4195049152 shutdown_connections 2024-09-11T01:42:35.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.638+0000 7ff17cfec640 1 -- 172.21.15.12:0/4195049152 wait complete. 2024-09-11T01:42:35.668 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:42:35.669 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T01:42:35.669 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:42:35.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.769+0000 7ff001e60640 10 monclient: get_monmap_and_config 2024-09-11T01:42:35.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.769+0000 7ff001e60640 10 monclient: build_initial_monmap 2024-09-11T01:42:35.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.769+0000 7ff001e60640 10 monclient: monmap: 2024-09-11T01:42:35.772 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:35.772 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:35.772 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:42:35.770322+0000 2024-09-11T01:42:35.772 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:42:35.770322+0000 2024-09-11T01:42:35.772 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:35.772 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:35.772 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:42:35.772 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:42:35.772 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:42:35.772 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:42:35.772 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:42:35.772 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:35.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.770+0000 7ff001e60640 1 Processor -- start 2024-09-11T01:42:35.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.770+0000 7ff001e60640 1 -- start start 2024-09-11T01:42:35.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.770+0000 7ff001e60640 10 monclient: init 2024-09-11T01:42:35.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.770+0000 7ff001e60640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:35.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.770+0000 7ff001e60640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-11T01:42:35.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.770+0000 7ff001e60640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feffc0a03a0 addr v1:172.21.15.12:6789/0 2024-09-11T01:42:35.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.770+0000 7ff001e60640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7feffc0a2530 0x7feffc09f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:35.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.770+0000 7ff001e60640 10 monclient(hunting): picked mon.noname-c con 0x7feffc0a2530 addr v2:172.21.15.71:3300/0 2024-09-11T01:42:35.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.770+0000 7ff001e60640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feffc0a13a0 0x7feffc13baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:35.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.770+0000 7ff001e60640 10 monclient(hunting): picked mon.noname-b con 0x7feffc0a13a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:35.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.770+0000 7ff001e60640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feffc09e510 con 0x7feffc0a03a0 2024-09-11T01:42:35.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.770+0000 7ff001e60640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:35.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.770+0000 7ff001e60640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7feffc0a29b0 con 0x7feffc0a2530 2024-09-11T01:42:35.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.770+0000 7ff001e60640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:35.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.770+0000 7ff001e60640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7feffc0a0780 con 0x7feffc0a13a0 2024-09-11T01:42:35.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.770+0000 7ff001e60640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:35.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.770+0000 7ff001e60640 10 monclient(hunting): authenticate will time out at 1562.212946s 2024-09-11T01:42:35.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.770+0000 7feffbfff640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feffc0a13a0 msgr2=0x7feffc13baa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:42:35.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.771+0000 7feffbfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feffc0a13a0 0x7feffc13baa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:35.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.771+0000 7feffb7fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7feffc0a03a0 0x7feffc0a0c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57594/0 (socket says 172.21.15.12:57594) 2024-09-11T01:42:35.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.771+0000 7feffb7fe640 1 -- 172.21.15.12:0/2329978620 learned_addr learned my addr 172.21.15.12:0/2329978620 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:42:35.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.771+0000 7feffaffd640 1 --2- 172.21.15.12:0/2329978620 >> v2:172.21.15.71:3300/0 conn(0x7feffc0a2530 0x7feffc09f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:35.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.771+0000 7feffaffd640 10 monclient(hunting): get_auth_request con 0x7feffc0a2530 auth_method 0 2024-09-11T01:42:35.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.771+0000 7feffaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:35.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.771+0000 7feffaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:35.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.771+0000 7feffaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:35.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.771+0000 7feffaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:35.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.771+0000 7feffaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:35.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.771+0000 7feffaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:35.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.771+0000 7feffaffd640 10 monclient(hunting): handle_auth_done global_id 134248 payload 274 2024-09-11T01:42:35.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.771+0000 7feffaffd640 1 -- 172.21.15.12:0/2329978620 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feffc0a13a0 msgr2=0x7feffc13baa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:42:35.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.771+0000 7feffaffd640 1 --2- 172.21.15.12:0/2329978620 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feffc0a13a0 0x7feffc13baa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.771+0000 7feffaffd640 1 -- 172.21.15.12:0/2329978620 >> v1:172.21.15.12:6789/0 conn(0x7feffc0a03a0 legacy=0x7feffc0a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:35.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.771+0000 7feffaffd640 10 monclient: _finish_hunting 0 2024-09-11T01:42:35.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.771+0000 7feffaffd640 1 monclient: found mon.noname-c 2024-09-11T01:42:35.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.771+0000 7feffaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:35.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.771+0000 7feffaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:42:35.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.771+0000 7feffaffd640 1 -- 172.21.15.12:0/2329978620 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feffc09edb0 con 0x7feffc0a2530 2024-09-11T01:42:35.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.774+0000 7feffaffd640 1 -- 172.21.15.12:0/2329978620 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feffc0a2530 msgr2=0x7feffc09f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:35.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.774+0000 7feffaffd640 1 --2- 172.21.15.12:0/2329978620 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feffc0a2530 0x7feffc09f1d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7feff00097a0 tx=0x7feff0029fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ac231a47b8be0ed server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:35.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.774+0000 7feffa7fc640 1 -- 172.21.15.12:0/2329978620 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff0037070 con 0x7feffc0a2530 2024-09-11T01:42:35.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.774+0000 7feffa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:35.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.774+0000 7feffa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:42:35.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.774+0000 7feffa7fc640 10 monclient: dump: 2024-09-11T01:42:35.780 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:35.781 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.774+0000 7feffa7fc640 10 monclient: _finish_auth 0 2024-09-11T01:42:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.774+0000 7feffa7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:42:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.774+0000 7feffa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:35.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.774+0000 7feffa7fc640 1 -- 172.21.15.12:0/2329978620 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feff0032640 con 0x7feffc0a2530 2024-09-11T01:42:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.774+0000 7ff001e60640 5 monclient: authenticate success, global_id 134248 2024-09-11T01:42:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.774+0000 7ff001e60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:42:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.774+0000 7feffa7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.774+0000 7ff001e60640 10 monclient: get_monmap_and_config success 2024-09-11T01:42:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.774+0000 7feffa7fc640 1 -- 172.21.15.12:0/2329978620 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff0032a10 con 0x7feffc0a2530 2024-09-11T01:42:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.774+0000 7ff001e60640 4 set_mon_vals no callback set 2024-09-11T01:42:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.774+0000 7feffa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.774+0000 7feffa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:42:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.774+0000 7feffa7fc640 10 monclient: dump: 2024-09-11T01:42:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.782 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:35.783 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.775+0000 7ff001e60640 10 monclient: shutdown 2024-09-11T01:42:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.775+0000 7ff001e60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.775+0000 7ff001e60640 1 -- 172.21.15.12:0/2329978620 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feffc0a2530 msgr2=0x7feffc09f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.775+0000 7ff001e60640 1 --2- 172.21.15.12:0/2329978620 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feffc0a2530 0x7feffc09f1d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7feff00097a0 tx=0x7feff0029fe0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.775+0000 7ff001e60640 1 -- 172.21.15.12:0/2329978620 shutdown_connections 2024-09-11T01:42:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.775+0000 7ff001e60640 1 --2- 172.21.15.12:0/2329978620 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feffc0a13a0 0x7feffc13baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.775+0000 7ff001e60640 1 --2- 172.21.15.12:0/2329978620 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feffc0a2530 0x7feffc09f1d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.775+0000 7ff001e60640 1 -- 172.21.15.12:0/2329978620 >> 172.21.15.12:0/2329978620 conn(0x7feffc0a2f70 msgr2=0x7feffc0a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:35.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.775+0000 7ff001e60640 1 -- 172.21.15.12:0/2329978620 shutdown_connections 2024-09-11T01:42:35.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.775+0000 7ff001e60640 1 -- 172.21.15.12:0/2329978620 wait complete. 2024-09-11T01:42:35.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.775+0000 7ff001e60640 10 monclient: build_initial_monmap 2024-09-11T01:42:35.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.775+0000 7ff001e60640 10 monclient: monmap: 2024-09-11T01:42:35.784 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:35.784 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:42:35.784 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:42:35.784 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:42:35.784 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:35.784 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:35.784 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:42:35.784 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:42:35.784 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:42:35.784 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:35.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7ff001e60640 1 Processor -- start 2024-09-11T01:42:35.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7ff001e60640 1 -- start start 2024-09-11T01:42:35.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7ff001e60640 10 monclient: init 2024-09-11T01:42:35.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7ff001e60640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:35.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7ff001e60640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:42:35.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7ff001e60640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feffc0a03a0 0x7feffc13ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:35.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7ff001e60640 10 monclient(hunting): picked mon.noname-a con 0x7feffc0a03a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:42:35.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7ff001e60640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feffc0a13a0 0x7feffc13bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:35.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7ff001e60640 10 monclient(hunting): picked mon.noname-c con 0x7feffc0a13a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:35.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7ff001e60640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feffc0a2530 0x7feffc123990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:35.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7ff001e60640 10 monclient(hunting): picked mon.noname-b con 0x7feffc0a2530 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:42:35.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7ff001e60640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:35.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7ff001e60640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7feffc09ca30 con 0x7feffc0a03a0 2024-09-11T01:42:35.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7ff001e60640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:35.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7ff001e60640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7feffc0a19f0 con 0x7feffc0a2530 2024-09-11T01:42:35.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7ff001e60640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:35.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7ff001e60640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7feffc09cbb0 con 0x7feffc0a13a0 2024-09-11T01:42:35.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7ff001e60640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:35.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7ff001e60640 10 monclient(hunting): authenticate will time out at 1562.218760s 2024-09-11T01:42:35.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7feffaffd640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feffc0a13a0 msgr2=0x7feffc13bfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:42:35.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7feffaffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feffc0a13a0 0x7feffc13bfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:35.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7feffb7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feffc0a03a0 0x7feffc13ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:35.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7feffb7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feffc0a03a0 0x7feffc13ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35820/0 (socket says 172.21.15.12:35820) 2024-09-11T01:42:35.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7feffb7fe640 1 -- 172.21.15.12:0/679862549 learned_addr learned my addr 172.21.15.12:0/679862549 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:35.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7feffb7fe640 10 monclient(hunting): get_auth_request con 0x7feffc0a03a0 auth_method 0 2024-09-11T01:42:35.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7feffb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:35.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7feffb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:35.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7feffb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:35.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7feffb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:35.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7feffb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:35.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.776+0000 7feffb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:35.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.777+0000 7feffbfff640 1 --2- 172.21.15.12:0/679862549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feffc0a2530 0x7feffc123990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:35.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.777+0000 7feffbfff640 10 monclient(hunting): get_auth_request con 0x7feffc0a2530 auth_method 0 2024-09-11T01:42:35.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.777+0000 7feffbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:35.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.777+0000 7feffbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:35.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.777+0000 7feffbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:35.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.777+0000 7feffb7fe640 10 monclient(hunting): handle_auth_done global_id 94641 payload 824 2024-09-11T01:42:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.777+0000 7feffb7fe640 1 -- 172.21.15.12:0/679862549 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feffc0a13a0 msgr2=0x7feffc13bfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:42:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.777+0000 7feffb7fe640 1 --2- 172.21.15.12:0/679862549 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feffc0a13a0 0x7feffc13bfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.777+0000 7feffb7fe640 1 -- 172.21.15.12:0/679862549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feffc0a2530 msgr2=0x7feffc123990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.777+0000 7feffb7fe640 1 --2- 172.21.15.12:0/679862549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feffc0a2530 0x7feffc123990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.777+0000 7feffb7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:42:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.777+0000 7feffb7fe640 1 monclient: found mon.noname-a 2024-09-11T01:42:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.777+0000 7feffb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.777+0000 7feffb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:42:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.777+0000 7feffb7fe640 1 -- 172.21.15.12:0/679862549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feffc09d480 con 0x7feffc0a03a0 2024-09-11T01:42:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.777+0000 7feffbfff640 1 --2- 172.21.15.12:0/679862549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feffc0a2530 0x7feffc123990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:42:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.779+0000 7feffb7fe640 1 -- 172.21.15.12:0/679862549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feffc0a03a0 msgr2=0x7feffc13ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.779+0000 7feffb7fe640 1 --2- 172.21.15.12:0/679862549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feffc0a03a0 0x7feffc13ba60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fefec00c460 tx=0x7fefec00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.779+0000 7fefe3fff640 1 -- 172.21.15.12:0/679862549 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec007070 con 0x7feffc0a03a0 2024-09-11T01:42:35.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.779+0000 7fefe3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.779+0000 7fefe3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:42:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.779+0000 7fefe3fff640 10 monclient: dump: 2024-09-11T01:42:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:35.794 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.779+0000 7fefe3fff640 10 monclient: _finish_auth 0 2024-09-11T01:42:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.779+0000 7fefe3fff640 10 monclient: _check_auth_tickets 2024-09-11T01:42:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.779+0000 7fefe3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.779+0000 7fefe3fff640 1 -- 172.21.15.12:0/679862549 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefec007230 con 0x7feffc0a03a0 2024-09-11T01:42:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.779+0000 7ff001e60640 5 monclient: authenticate success, global_id 94641 2024-09-11T01:42:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.779+0000 7ff001e60640 10 monclient: _renew_subs 2024-09-11T01:42:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.779+0000 7ff001e60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.779+0000 7ff001e60640 1 -- 172.21.15.12:0/679862549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feffc126f10 con 0x7feffc0a03a0 2024-09-11T01:42:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.779+0000 7fefe3fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.779+0000 7fefe3fff640 1 -- 172.21.15.12:0/679862549 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec007620 con 0x7feffc0a03a0 2024-09-11T01:42:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.779+0000 7fefe3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.779+0000 7fefe3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.779+0000 7fefe3fff640 10 monclient: dump: 2024-09-11T01:42:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:35.795 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:35.796 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.779+0000 7ff001e60640 10 monclient: _renew_subs 2024-09-11T01:42:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.779+0000 7ff001e60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.779+0000 7ff001e60640 1 -- 172.21.15.12:0/679862549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feffc13cbb0 con 0x7feffc0a03a0 2024-09-11T01:42:35.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.779+0000 7feff9ffb640 4 set_mon_vals no callback set 2024-09-11T01:42:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.780+0000 7ff001e60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.780+0000 7ff001e60640 1 --2- 172.21.15.12:0/679862549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feffc125fb0 0x7feffc124690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.780+0000 7ff001e60640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.780+0000 7ff001e60640 1 -- 172.21.15.12:0/679862549 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7feffc127430 con 0x7feffc125fb0 2024-09-11T01:42:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.780+0000 7fefe3fff640 1 -- 172.21.15.12:0/679862549 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fefec01b400 con 0x7feffc0a03a0 2024-09-11T01:42:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.780+0000 7feffaffd640 1 --2- 172.21.15.12:0/679862549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feffc125fb0 0x7feffc124690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.780+0000 7feffaffd640 10 monclient: get_auth_request con 0x7feffc125fb0 auth_method 0 2024-09-11T01:42:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.780+0000 7feffaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.780+0000 7feffaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.780+0000 7feffaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.780+0000 7feffaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.780+0000 7feffaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.780+0000 7feffaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.780+0000 7fefe3fff640 1 --2- 172.21.15.12:0/679862549 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fefe803e1f0 0x7fefe80406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.780+0000 7fefe3fff640 1 -- 172.21.15.12:0/679862549 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(442..442 src has 285..442) ==== 4787+0+0 (secure 0 0 0) 0x7fefec052a80 con 0x7feffc0a03a0 2024-09-11T01:42:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.781+0000 7feffaffd640 10 monclient(hunting): handle_auth_done global_id 134254 payload 824 2024-09-11T01:42:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.781+0000 7feffaffd640 1 -- 172.21.15.12:0/679862549 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feffc09cfd0 con 0x7feffc125fb0 2024-09-11T01:42:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.781+0000 7feffbfff640 1 --2- 172.21.15.12:0/679862549 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fefe803e1f0 0x7fefe80406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.781+0000 7feffbfff640 10 monclient: get_auth_request con 0x7fefe803e1f0 auth_method 0 2024-09-11T01:42:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.783+0000 7feffaffd640 1 -- 172.21.15.12:0/679862549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feffc125fb0 msgr2=0x7feffc124690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.783+0000 7feffaffd640 1 --2- 172.21.15.12:0/679862549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feffc125fb0 0x7feffc124690 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7feff0034000 tx=0x7feff00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.783+0000 7fefe3fff640 1 -- 172.21.15.12:0/679862549 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff0042070 con 0x7feffc125fb0 2024-09-11T01:42:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.783+0000 7fefe3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.783+0000 7fefe3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:42:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.783+0000 7fefe3fff640 10 monclient: dump: 2024-09-11T01:42:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.798 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.799 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:35.799 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:35.799 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:35.799 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:35.799 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:35.799 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:35.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.783+0000 7feffbfff640 1 --2- 172.21.15.12:0/679862549 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fefe803e1f0 0x7fefe80406b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7feffc127ca0 tx=0x7feff40094f0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:35.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.783+0000 7fefe3fff640 1 -- 172.21.15.12:0/679862549 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7feff0032a00 con 0x7feffc125fb0 2024-09-11T01:42:35.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.783+0000 7fefe3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:35.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.783+0000 7fefe3fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:42:35.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.783+0000 7fefe3fff640 1 -- 172.21.15.12:0/679862549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feffc125fb0 msgr2=0x7feffc124690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:35.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.783+0000 7fefe3fff640 1 --2- 172.21.15.12:0/679862549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feffc125fb0 0x7feffc124690 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7feff0034000 tx=0x7feff00046c0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.783+0000 7fefe3fff640 1 -- 172.21.15.12:0/679862549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feffc125fb0 msgr2=0x7feffc124690 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:42:35.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.783+0000 7fefe3fff640 1 --2- 172.21.15.12:0/679862549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feffc125fb0 0x7feffc124690 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7feff0034000 tx=0x7feff00046c0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.790+0000 7ff001e60640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:42:35.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.790+0000 7ff001e60640 1 --2- 172.21.15.12:0/679862549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feffc124bd0 0x7feffc02d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:35.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.790+0000 7ff001e60640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:35.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.790+0000 7ff001e60640 1 -- 172.21.15.12:0/679862549 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7feffc120000 con 0x7feffc124bd0 2024-09-11T01:42:35.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.790+0000 7feffaffd640 1 --2- 172.21.15.12:0/679862549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feffc124bd0 0x7feffc02d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:35.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.790+0000 7feffaffd640 10 monclient: get_auth_request con 0x7feffc124bd0 auth_method 0 2024-09-11T01:42:35.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.790+0000 7feffaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:35.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.790+0000 7feffaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:35.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.790+0000 7feffaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:35.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.790+0000 7feffaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:35.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.790+0000 7feffaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:35.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.790+0000 7feffaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:35.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.790+0000 7feffaffd640 10 monclient(hunting): handle_auth_done global_id 134257 payload 824 2024-09-11T01:42:35.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.790+0000 7feffaffd640 1 -- 172.21.15.12:0/679862549 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7feffc06d480 con 0x7feffc124bd0 2024-09-11T01:42:35.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.793+0000 7feffaffd640 1 -- 172.21.15.12:0/679862549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feffc124bd0 msgr2=0x7feffc02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:35.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.793+0000 7feffaffd640 1 --2- 172.21.15.12:0/679862549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feffc124bd0 0x7feffc02d440 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7feff0000c00 tx=0x7feff0009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:35.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.793+0000 7fefe3fff640 1 -- 172.21.15.12:0/679862549 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff0037040 con 0x7feffc124bd0 2024-09-11T01:42:35.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.793+0000 7fefe3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:35.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.793+0000 7fefe3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:42:35.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.793+0000 7fefe3fff640 10 monclient: dump: 2024-09-11T01:42:35.803 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:35.803 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:35.803 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.803 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:35.803 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:35.803 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:35.803 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:35.803 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:35.803 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:35.804 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:35.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.793+0000 7fefe3fff640 1 -- 172.21.15.12:0/679862549 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4697 (secure 0 0 0) 0x7feff003de50 con 0x7feffc124bd0 2024-09-11T01:42:35.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.793+0000 7fefe3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:42:35.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.793+0000 7fefe3fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:42:35.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.793+0000 7fefe3fff640 1 -- 172.21.15.12:0/679862549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feffc124bd0 msgr2=0x7feffc02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:35.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.793+0000 7fefe3fff640 1 --2- 172.21.15.12:0/679862549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feffc124bd0 0x7feffc02d440 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7feff0000c00 tx=0x7feff0009190 comp rx=0 tx=0).stop 2024-09-11T01:42:35.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.793+0000 7fefe3fff640 1 -- 172.21.15.12:0/679862549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feffc124bd0 msgr2=0x7feffc02d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:42:35.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.793+0000 7fefe3fff640 1 --2- 172.21.15.12:0/679862549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feffc124bd0 0x7feffc02d440 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7feff0000c00 tx=0x7feff0009190 comp rx=0 tx=0).stop 2024-09-11T01:42:35.804 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:42:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:42:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:42:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:42:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 126, 2024-09-11T01:42:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 88755, 2024-09-11T01:42:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:42:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:42:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: 1 2024-09-11T01:42:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:42:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:42:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:42:35.805 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:42:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:42:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:42:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:42:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:42:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:42:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:42:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:42:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:42:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:42:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:42:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:42:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:42:35.806 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:42:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:42:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:42:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:42:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:42:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:42:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:42:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:42:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:42:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:42:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:42:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:42:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:35.807 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:42:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:42:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:42:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:42:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:42:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:42:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:42:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:42:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:42:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:42:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:42:35.808 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:42:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:42:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:42:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:42:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:42:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:42:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:42:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:42:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:42:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:42:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:42:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:42:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:42:35.809 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:42:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:42:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:42:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:42:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:42:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:42:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:42:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:42:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:42:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:42:35.810 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:42:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:42:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:42:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:42:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:42:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:42:35.811 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:42:35.812 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:42:35.812 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:42:35.812 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:35.812 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:35.812 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:42:35.812 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:42:35.812 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:42:35.812 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:42:35.812 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:35.812 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:42:35.812 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:42:35.812 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:35.812 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:42:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:42:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:42:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:42:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:42:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:42:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:42:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:35.814 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:42:35.814 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:42:35.814 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:42:35.814 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:42:35.814 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:35.814 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:42:35.814 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:42:35.814 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:35.814 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:35.814 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:35.814 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:42:35.814 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:42:35.814 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:42:35.815 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:35.815 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:35.815 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:35.815 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:42:35.815 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:42:35.815 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:42:35.815 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:42:35.815 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:42:35.815 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:35.815 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:35.815 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:35.815 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:42:35.815 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:42:35.815 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:35.816 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:42:35.816 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:42:35.816 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:42:35.816 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:35.816 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:35.816 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:42:35.816 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:35.816 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:42:35.816 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:42:35.816 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:42:35.816 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:35.816 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:42:35.816 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:42:35.817 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:42:35.817 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:42:35.817 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:42:35.817 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:42:35.817 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:42:35.817 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:42:35.817 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:42:35.817 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:42:35.817 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:42:35.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.793+0000 7ff001e60640 1 -- 172.21.15.12:0/679862549 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fefe803e1f0 msgr2=0x7fefe80406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:35.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.793+0000 7ff001e60640 1 --2- 172.21.15.12:0/679862549 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fefe803e1f0 0x7fefe80406b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7feffc127ca0 tx=0x7feff40094f0 comp rx=0 tx=0).stop 2024-09-11T01:42:35.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.793+0000 7ff001e60640 10 monclient: shutdown 2024-09-11T01:42:35.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.793+0000 7ff001e60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:35.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.793+0000 7ff001e60640 1 -- 172.21.15.12:0/679862549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feffc0a03a0 msgr2=0x7feffc13ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:35.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.793+0000 7ff001e60640 1 --2- 172.21.15.12:0/679862549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feffc0a03a0 0x7feffc13ba60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fefec00c460 tx=0x7fefec00c920 comp rx=0 tx=0).stop 2024-09-11T01:42:35.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.793+0000 7feffbfff640 1 -- 172.21.15.12:0/679862549 reap_dead start 2024-09-11T01:42:35.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.793+0000 7feffbfff640 1 -- 172.21.15.12:0/679862549 reap_dead start 2024-09-11T01:42:35.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.793+0000 7ff001e60640 1 -- 172.21.15.12:0/679862549 shutdown_connections 2024-09-11T01:42:35.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.793+0000 7ff001e60640 1 -- 172.21.15.12:0/679862549 >> 172.21.15.12:0/679862549 conn(0x7feffc0a2f70 msgr2=0x7feffc0732d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:35.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.794+0000 7ff001e60640 1 -- 172.21.15.12:0/679862549 shutdown_connections 2024-09-11T01:42:35.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:35.794+0000 7ff001e60640 1 -- 172.21.15.12:0/679862549 wait complete. 2024-09-11T01:42:35.820 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:42:35.820 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:42:36.221 INFO:tasks.rados.rados.0.smithi071.stdout:1926: done (4 left) 2024-09-11T01:42:36.222 INFO:tasks.rados.rados.0.smithi071.stdout:1927: done (3 left) 2024-09-11T01:42:36.222 INFO:tasks.rados.rados.0.smithi071.stdout:1928: done (2 left) 2024-09-11T01:42:36.222 INFO:tasks.rados.rados.0.smithi071.stdout:1929: done (1 left) 2024-09-11T01:42:36.222 INFO:tasks.rados.rados.0.smithi071.stdout:1930: done (0 left) 2024-09-11T01:42:36.222 INFO:tasks.rados.rados.0.smithi071.stdout:1931: delete oid 11 current snap is 216 2024-09-11T01:42:36.226 INFO:tasks.rados.rados.0.smithi071.stdout:1931: done (0 left) 2024-09-11T01:42:36.226 INFO:tasks.rados.rados.0.smithi071.stdout:1932: snap_remove snap 200 2024-09-11T01:42:37.226 INFO:tasks.rados.rados.0.smithi071.stdout:1932: done (0 left) 2024-09-11T01:42:37.226 INFO:tasks.rados.rados.0.smithi071.stdout:1933: copy_from oid 22 from oid 40 current snap is 216 2024-09-11T01:42:37.226 INFO:tasks.rados.rados.0.smithi071.stdout:1934: copy_from oid 20 from oid 39 current snap is 216 2024-09-11T01:42:37.226 INFO:tasks.rados.rados.0.smithi071.stdout:1935: write (excl) oid 30 current snap is 216 2024-09-11T01:42:37.226 INFO:tasks.rados.rados.0.smithi071.stdout:1935: seq_num 710 ranges {732970=647619,2097627=611905,3231266=437835} 2024-09-11T01:42:37.234 INFO:tasks.rados.rados.0.smithi071.stdout:1935: writing smithi07134777-30 from 732970 to 1380589 tid 1 2024-09-11T01:42:37.242 INFO:tasks.rados.rados.0.smithi071.stdout:1935: writing smithi07134777-30 from 2097627 to 2709532 tid 2 2024-09-11T01:42:37.248 INFO:tasks.rados.rados.0.smithi071.stdout:1935: writing smithi07134777-30 from 3231266 to 3669101 tid 3 2024-09-11T01:42:37.248 INFO:tasks.rados.rados.0.smithi071.stdout:1936: copy_from oid 15 from oid 8 current snap is 216 2024-09-11T01:42:37.248 INFO:tasks.rados.rados.0.smithi071.stdout:1935: finishing write tid 1 to smithi07134777-30 2024-09-11T01:42:37.249 INFO:tasks.rados.rados.0.smithi071.stdout:1937: snap_create 2024-09-11T01:42:37.249 INFO:tasks.rados.rados.0.smithi071.stdout:1933: finishing copy_from racing read to smithi07134777-22 2024-09-11T01:42:37.250 INFO:tasks.rados.rados.0.smithi071.stdout:1936: finishing copy_from to smithi07134777-15 2024-09-11T01:42:37.251 INFO:tasks.rados.rados.0.smithi071.stdout:1936: got expected ENOENT (src dne) 2024-09-11T01:42:37.251 INFO:tasks.rados.rados.0.smithi071.stdout:1936: finishing copy_from racing read to smithi07134777-15 2024-09-11T01:42:37.251 INFO:tasks.rados.rados.0.smithi071.stdout:1935: finishing write tid 2 to smithi07134777-30 2024-09-11T01:42:37.253 INFO:tasks.rados.rados.0.smithi071.stdout:1935: finishing write tid 3 to smithi07134777-30 2024-09-11T01:42:37.253 INFO:tasks.rados.rados.0.smithi071.stdout:1935: finishing write tid 4 to smithi07134777-30 2024-09-11T01:42:37.253 INFO:tasks.rados.rados.0.smithi071.stdout:1935: finishing write tid 5 to smithi07134777-30 2024-09-11T01:42:37.253 INFO:tasks.rados.rados.0.smithi071.stdout:1935: oid 30 updating version 0 to 913 2024-09-11T01:42:37.253 INFO:tasks.rados.rados.0.smithi071.stdout:1935: oid 30 updating version 913 to 915 2024-09-11T01:42:37.253 INFO:tasks.rados.rados.0.smithi071.stdout:1935: oid 30 updating version 915 to 916 2024-09-11T01:42:37.253 INFO:tasks.rados.rados.0.smithi071.stdout:1935: oid 30 version 916 is already newer than 914 2024-09-11T01:42:37.254 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 916 (ObjNum 710 snap 216 seq_num 710) dirty exists 2024-09-11T01:42:37.254 INFO:tasks.rados.rados.0.smithi071.stdout:1935: left oid 30 (ObjNum 710 snap 216 seq_num 710) 2024-09-11T01:42:37.256 INFO:tasks.rados.rados.0.smithi071.stdout:1934: finishing copy_from racing read to smithi07134777-20 2024-09-11T01:42:37.257 INFO:tasks.rados.rados.0.smithi071.stdout:1933: finishing copy_from to smithi07134777-22 2024-09-11T01:42:37.257 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 700 (ObjNum 647 snap 199 seq_num 647) dirty exists 2024-09-11T01:42:37.262 INFO:tasks.rados.rados.0.smithi071.stdout:1934: finishing copy_from to smithi07134777-20 2024-09-11T01:42:37.262 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 969 (ObjNum 701 snap 211 seq_num 701) dirty exists 2024-09-11T01:42:38.229 INFO:tasks.rados.rados.0.smithi071.stdout:1933: done (4 left) 2024-09-11T01:42:38.229 INFO:tasks.rados.rados.0.smithi071.stdout:1934: done (3 left) 2024-09-11T01:42:38.229 INFO:tasks.rados.rados.0.smithi071.stdout:1935: done (2 left) 2024-09-11T01:42:38.229 INFO:tasks.rados.rados.0.smithi071.stdout:1936: done (1 left) 2024-09-11T01:42:38.229 INFO:tasks.rados.rados.0.smithi071.stdout:1937: done (0 left) 2024-09-11T01:42:38.229 INFO:tasks.rados.rados.0.smithi071.stdout:1938: snap_create 2024-09-11T01:42:38.965 INFO:tasks.rados.rados.0.smithi071.stdout:1938: done (0 left) 2024-09-11T01:42:38.965 INFO:tasks.rados.rados.0.smithi071.stdout:1939: delete oid 38 current snap is 218 2024-09-11T01:42:38.976 INFO:tasks.rados.rados.0.smithi071.stdout:1939: done (0 left) 2024-09-11T01:42:38.976 INFO:tasks.rados.rados.0.smithi071.stdout:1940: rollback oid 30 current snap is 218 2024-09-11T01:42:38.976 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 30 to 208 2024-09-11T01:42:38.976 INFO:tasks.rados.rados.0.smithi071.stdout:1941: read oid 21 snap -1 2024-09-11T01:42:38.976 INFO:tasks.rados.rados.0.smithi071.stdout:1941: expect (ObjNum 679 snap 204 seq_num 679) 2024-09-11T01:42:38.976 INFO:tasks.rados.rados.0.smithi071.stdout:1942: rollback oid 5 current snap is 218 2024-09-11T01:42:38.976 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 5 to 212 2024-09-11T01:42:38.976 INFO:tasks.rados.rados.0.smithi071.stdout:1943: rollback oid 38 current snap is 218 2024-09-11T01:42:38.976 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 38 to 185 2024-09-11T01:42:38.976 INFO:tasks.rados.rados.0.smithi071.stdout:1944: rollback oid 8 current snap is 218 2024-09-11T01:42:38.976 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 8 to 202 2024-09-11T01:42:38.976 INFO:tasks.rados.rados.0.smithi071.stdout:1945: read oid 45 snap -1 2024-09-11T01:42:38.977 INFO:tasks.rados.rados.0.smithi071.stdout:1945: expect (ObjNum 612 snap 181 seq_num 612) 2024-09-11T01:42:38.977 INFO:tasks.rados.rados.0.smithi071.stdout:1946: delete oid 40 current snap is 218 2024-09-11T01:42:38.977 INFO:tasks.rados.rados.0.smithi071.stdout:1942: finishing rollback tid 1 to smithi07134777-5 2024-09-11T01:42:38.977 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 505 (ObjNum 129 snap 0 seq_num 2371436848) dirty dne 2024-09-11T01:42:38.979 INFO:tasks.rados.rados.0.smithi071.stdout:1943: finishing rollback tid 1 to smithi07134777-38 2024-09-11T01:42:38.986 INFO:tasks.rados.rados.0.smithi071.stdout:1943: finishing rollback tid 2 to smithi07134777-38 2024-09-11T01:42:38.986 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 1131 (ObjNum 577 snap 165 seq_num 577) dirty exists 2024-09-11T01:42:38.987 INFO:tasks.rados.rados.0.smithi071.stdout:1944: finishing rollback tid 1 to smithi07134777-8 2024-09-11T01:42:38.987 INFO:tasks.rados.rados.0.smithi071.stdout:1940: finishing rollback tid 0 to smithi07134777-30 2024-09-11T01:42:38.987 INFO:tasks.rados.rados.0.smithi071.stdout:1944: finishing rollback tid 2 to smithi07134777-8 2024-09-11T01:42:38.987 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 1010 (ObjNum 586 snap 168 seq_num 586) dirty exists 2024-09-11T01:42:38.987 INFO:tasks.rados.rados.0.smithi071.stdout:1942: done (6 left) 2024-09-11T01:42:38.987 INFO:tasks.rados.rados.0.smithi071.stdout:1943: done (5 left) 2024-09-11T01:42:38.987 INFO:tasks.rados.rados.0.smithi071.stdout:1944: done (4 left) 2024-09-11T01:42:38.987 INFO:tasks.rados.rados.0.smithi071.stdout:1946: done (3 left) 2024-09-11T01:42:38.987 INFO:tasks.rados.rados.0.smithi071.stdout:1947: delete oid 27 current snap is 218 2024-09-11T01:42:38.988 INFO:tasks.rados.rados.0.smithi071.stdout:1940: finishing rollback tid 1 to smithi07134777-30 2024-09-11T01:42:38.988 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 918 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-11T01:42:38.993 INFO:tasks.rados.rados.0.smithi071.stdout:1941: expect (ObjNum 679 snap 204 seq_num 679) 2024-09-11T01:42:39.271 INFO:tasks.rados.rados.0.smithi071.stdout:1945: expect (ObjNum 612 snap 181 seq_num 612) 2024-09-11T01:42:39.276 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1018714 2024-09-11T01:42:39.321 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2239179 2024-09-11T01:42:39.321 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:39.356 INFO:tasks.rados.rados.0.smithi071.stdout:1940: done (3 left) 2024-09-11T01:42:39.356 INFO:tasks.rados.rados.0.smithi071.stdout:1941: done (2 left) 2024-09-11T01:42:39.357 INFO:tasks.rados.rados.0.smithi071.stdout:1945: done (1 left) 2024-09-11T01:42:39.357 INFO:tasks.rados.rados.0.smithi071.stdout:1947: done (0 left) 2024-09-11T01:42:39.357 INFO:tasks.rados.rados.0.smithi071.stdout:1948: read oid 12 snap -1 2024-09-11T01:42:39.357 INFO:tasks.rados.rados.0.smithi071.stdout:1948: expect (ObjNum 696 snap 208 seq_num 696) 2024-09-11T01:42:39.357 INFO:tasks.rados.rados.0.smithi071.stdout:1949: copy_from oid 40 from oid 2 current snap is 218 2024-09-11T01:42:39.357 INFO:tasks.rados.rados.0.smithi071.stdout:1950: snap_create 2024-09-11T01:42:39.363 INFO:tasks.rados.rados.0.smithi071.stdout:1949: finishing copy_from racing read to smithi07134777-40 2024-09-11T01:42:39.369 INFO:tasks.rados.rados.0.smithi071.stdout:1949: finishing copy_from to smithi07134777-40 2024-09-11T01:42:39.369 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 832 (ObjNum 686 snap 204 seq_num 686) dirty exists 2024-09-11T01:42:39.369 INFO:tasks.rados.rados.0.smithi071.stdout:1948: expect (ObjNum 696 snap 208 seq_num 696) 2024-09-11T01:42:39.469 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3094613 2024-09-11T01:42:39.480 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 262566 2024-09-11T01:42:40.239 INFO:tasks.rados.rados.0.smithi071.stdout:1948: done (2 left) 2024-09-11T01:42:40.239 INFO:tasks.rados.rados.0.smithi071.stdout:1949: done (1 left) 2024-09-11T01:42:40.239 INFO:tasks.rados.rados.0.smithi071.stdout:1950: done (0 left) 2024-09-11T01:42:40.239 INFO:tasks.rados.rados.0.smithi071.stdout:1951: snap_create 2024-09-11T01:42:41.242 INFO:tasks.rados.rados.0.smithi071.stdout:1951: done (0 left) 2024-09-11T01:42:41.242 INFO:tasks.rados.rados.0.smithi071.stdout:1952: read oid 25 snap 214 2024-09-11T01:42:41.242 INFO:tasks.rados.rados.0.smithi071.stdout:1952: expect (ObjNum 488 snap 132 seq_num 488) 2024-09-11T01:42:41.242 INFO:tasks.rados.rados.0.smithi071.stdout:1953: snap_create 2024-09-11T01:42:41.255 INFO:tasks.rados.rados.0.smithi071.stdout:1952: expect (ObjNum 488 snap 132 seq_num 488) 2024-09-11T01:42:41.256 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 48840 2024-09-11T01:42:42.245 INFO:tasks.rados.rados.0.smithi071.stdout:1952: done (1 left) 2024-09-11T01:42:42.245 INFO:tasks.rados.rados.0.smithi071.stdout:1953: done (0 left) 2024-09-11T01:42:42.245 INFO:tasks.rados.rados.0.smithi071.stdout:1954: write oid 12 current snap is 221 2024-09-11T01:42:42.245 INFO:tasks.rados.rados.0.smithi071.stdout:1954: seq_num 714 ranges {400198=548615,1415202=584996} 2024-09-11T01:42:42.250 INFO:tasks.rados.rados.0.smithi071.stdout:1954: writing smithi07134777-12 from 400198 to 948813 tid 1 2024-09-11T01:42:42.256 INFO:tasks.rados.rados.0.smithi071.stdout:1954: writing smithi07134777-12 from 1415202 to 2000198 tid 2 2024-09-11T01:42:42.256 INFO:tasks.rados.rados.0.smithi071.stdout:1955: write (excl) oid 47 current snap is 221 2024-09-11T01:42:42.256 INFO:tasks.rados.rados.0.smithi071.stdout:1955: seq_num 715 ranges {417005=740217,1590037=426968} 2024-09-11T01:42:42.262 INFO:tasks.rados.rados.0.smithi071.stdout:1955: writing smithi07134777-47 from 417005 to 1157222 tid 1 2024-09-11T01:42:42.268 INFO:tasks.rados.rados.0.smithi071.stdout:1955: writing smithi07134777-47 from 1590037 to 2017005 tid 2 2024-09-11T01:42:42.268 INFO:tasks.rados.rados.0.smithi071.stdout:1956: snap_remove snap 215 2024-09-11T01:42:43.259 INFO:tasks.rados.rados.0.smithi071.stdout:1956: done (2 left) 2024-09-11T01:42:43.259 INFO:tasks.rados.rados.0.smithi071.stdout:1957: write (excl) oid 45 current snap is 221 2024-09-11T01:42:43.259 INFO:tasks.rados.rados.0.smithi071.stdout:1954: finishing write tid 1 to smithi07134777-12 2024-09-11T01:42:43.259 INFO:tasks.rados.rados.0.smithi071.stdout:1957: seq_num 716 ranges {433812=415466,1648519=385293} 2024-09-11T01:42:43.263 INFO:tasks.rados.rados.0.smithi071.stdout:1957: writing smithi07134777-45 from 433812 to 849278 tid 1 2024-09-11T01:42:43.270 INFO:tasks.rados.rados.0.smithi071.stdout:1957: writing smithi07134777-45 from 1648519 to 2033812 tid 2 2024-09-11T01:42:43.271 INFO:tasks.rados.rados.0.smithi071.stdout:1958: copy_from oid 33 from oid 43 current snap is 221 2024-09-11T01:42:43.271 INFO:tasks.rados.rados.0.smithi071.stdout:1954: finishing write tid 2 to smithi07134777-12 2024-09-11T01:42:43.271 INFO:tasks.rados.rados.0.smithi071.stdout:1954: finishing write tid 3 to smithi07134777-12 2024-09-11T01:42:43.271 INFO:tasks.rados.rados.0.smithi071.stdout:1954: finishing write tid 4 to smithi07134777-12 2024-09-11T01:42:43.271 INFO:tasks.rados.rados.0.smithi071.stdout:1954: oid 12 updating version 0 to 1021 2024-09-11T01:42:43.271 INFO:tasks.rados.rados.0.smithi071.stdout:1954: oid 12 version 1021 is already newer than 1020 2024-09-11T01:42:43.271 INFO:tasks.rados.rados.0.smithi071.stdout:1954: oid 12 version 1021 is already newer than 1019 2024-09-11T01:42:43.271 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 1021 (ObjNum 714 snap 221 seq_num 714) dirty exists 2024-09-11T01:42:43.271 INFO:tasks.rados.rados.0.smithi071.stdout:1954: left oid 12 (ObjNum 714 snap 221 seq_num 714) 2024-09-11T01:42:43.272 INFO:tasks.rados.rados.0.smithi071.stdout:1955: finishing write tid 1 to smithi07134777-47 2024-09-11T01:42:43.272 INFO:tasks.rados.rados.0.smithi071.stdout:1955: finishing write tid 2 to smithi07134777-47 2024-09-11T01:42:43.272 INFO:tasks.rados.rados.0.smithi071.stdout:1955: finishing write tid 3 to smithi07134777-47 2024-09-11T01:42:43.272 INFO:tasks.rados.rados.0.smithi071.stdout:1955: finishing write tid 4 to smithi07134777-47 2024-09-11T01:42:43.272 INFO:tasks.rados.rados.0.smithi071.stdout:1955: oid 47 updating version 0 to 837 2024-09-11T01:42:43.272 INFO:tasks.rados.rados.0.smithi071.stdout:1955: oid 47 updating version 837 to 839 2024-09-11T01:42:43.272 INFO:tasks.rados.rados.0.smithi071.stdout:1955: oid 47 version 839 is already newer than 838 2024-09-11T01:42:43.272 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 839 (ObjNum 715 snap 221 seq_num 715) dirty exists 2024-09-11T01:42:43.272 INFO:tasks.rados.rados.0.smithi071.stdout:1955: left oid 47 (ObjNum 715 snap 221 seq_num 715) 2024-09-11T01:42:43.272 INFO:tasks.rados.rados.0.smithi071.stdout:1954: done (3 left) 2024-09-11T01:42:43.272 INFO:tasks.rados.rados.0.smithi071.stdout:1955: done (2 left) 2024-09-11T01:42:43.272 INFO:tasks.rados.rados.0.smithi071.stdout:1959: snap_create 2024-09-11T01:42:43.277 INFO:tasks.rados.rados.0.smithi071.stdout:1957: finishing write tid 1 to smithi07134777-45 2024-09-11T01:42:43.278 INFO:tasks.rados.rados.0.smithi071.stdout:1957: finishing write tid 2 to smithi07134777-45 2024-09-11T01:42:43.279 INFO:tasks.rados.rados.0.smithi071.stdout:1957: finishing write tid 3 to smithi07134777-45 2024-09-11T01:42:43.279 INFO:tasks.rados.rados.0.smithi071.stdout:1957: finishing write tid 4 to smithi07134777-45 2024-09-11T01:42:43.280 INFO:tasks.rados.rados.0.smithi071.stdout:1957: oid 45 updating version 0 to 933 2024-09-11T01:42:43.280 INFO:tasks.rados.rados.0.smithi071.stdout:1957: oid 45 updating version 933 to 934 2024-09-11T01:42:43.280 INFO:tasks.rados.rados.0.smithi071.stdout:1957: oid 45 version 934 is already newer than 932 2024-09-11T01:42:43.280 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 934 (ObjNum 716 snap 221 seq_num 716) dirty exists 2024-09-11T01:42:43.280 INFO:tasks.rados.rados.0.smithi071.stdout:1957: left oid 45 (ObjNum 716 snap 221 seq_num 716) 2024-09-11T01:42:43.286 INFO:tasks.rados.rados.0.smithi071.stdout:1958: finishing copy_from racing read to smithi07134777-33 2024-09-11T01:42:43.290 INFO:tasks.rados.rados.0.smithi071.stdout:1958: finishing copy_from to smithi07134777-33 2024-09-11T01:42:43.290 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 964 (ObjNum 689 snap 204 seq_num 689) dirty exists 2024-09-11T01:42:43.967 INFO:tasks.rados.rados.0.smithi071.stdout:1957: done (2 left) 2024-09-11T01:42:43.967 INFO:tasks.rados.rados.0.smithi071.stdout:1958: done (1 left) 2024-09-11T01:42:43.967 INFO:tasks.rados.rados.0.smithi071.stdout:1959: done (0 left) 2024-09-11T01:42:43.967 INFO:tasks.rados.rados.0.smithi071.stdout:1960: write (excl) oid 5 current snap is 222 2024-09-11T01:42:43.967 INFO:tasks.rados.rados.0.smithi071.stdout:1960: seq_num 717 ranges {450619=490715,1590648=459971} 2024-09-11T01:42:43.972 INFO:tasks.rados.rados.0.smithi071.stdout:1960: writing smithi07134777-5 from 450619 to 941334 tid 1 2024-09-11T01:42:43.978 INFO:tasks.rados.rados.0.smithi071.stdout:1960: writing smithi07134777-5 from 1590648 to 2050619 tid 2 2024-09-11T01:42:43.979 INFO:tasks.rados.rados.0.smithi071.stdout:1961: write oid 8 current snap is 222 2024-09-11T01:42:43.979 INFO:tasks.rados.rados.0.smithi071.stdout:1961: seq_num 718 ranges {467426=565964,1649130=418296} 2024-09-11T01:42:43.984 INFO:tasks.rados.rados.0.smithi071.stdout:1961: writing smithi07134777-8 from 467426 to 1033390 tid 1 2024-09-11T01:42:43.990 INFO:tasks.rados.rados.0.smithi071.stdout:1961: writing smithi07134777-8 from 1649130 to 2067426 tid 2 2024-09-11T01:42:43.990 INFO:tasks.rados.rados.0.smithi071.stdout:1962: snap_create 2024-09-11T01:42:43.990 INFO:tasks.rados.rados.0.smithi071.stdout:1960: finishing write tid 1 to smithi07134777-5 2024-09-11T01:42:43.990 INFO:tasks.rados.rados.0.smithi071.stdout:1960: finishing write tid 2 to smithi07134777-5 2024-09-11T01:42:43.990 INFO:tasks.rados.rados.0.smithi071.stdout:1960: finishing write tid 3 to smithi07134777-5 2024-09-11T01:42:43.990 INFO:tasks.rados.rados.0.smithi071.stdout:1960: finishing write tid 4 to smithi07134777-5 2024-09-11T01:42:43.990 INFO:tasks.rados.rados.0.smithi071.stdout:1960: oid 5 updating version 0 to 508 2024-09-11T01:42:43.990 INFO:tasks.rados.rados.0.smithi071.stdout:1960: oid 5 version 508 is already newer than 506 2024-09-11T01:42:43.990 INFO:tasks.rados.rados.0.smithi071.stdout:1960: oid 5 version 508 is already newer than 507 2024-09-11T01:42:43.990 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 508 (ObjNum 717 snap 222 seq_num 717) dirty exists 2024-09-11T01:42:43.990 INFO:tasks.rados.rados.0.smithi071.stdout:1960: left oid 5 (ObjNum 717 snap 222 seq_num 717) 2024-09-11T01:42:43.994 INFO:tasks.rados.rados.0.smithi071.stdout:1961: finishing write tid 1 to smithi07134777-8 2024-09-11T01:42:43.996 INFO:tasks.rados.rados.0.smithi071.stdout:1961: finishing write tid 2 to smithi07134777-8 2024-09-11T01:42:43.999 INFO:tasks.rados.rados.0.smithi071.stdout:1961: finishing write tid 3 to smithi07134777-8 2024-09-11T01:42:43.999 INFO:tasks.rados.rados.0.smithi071.stdout:1961: finishing write tid 4 to smithi07134777-8 2024-09-11T01:42:43.999 INFO:tasks.rados.rados.0.smithi071.stdout:1961: oid 8 updating version 0 to 1012 2024-09-11T01:42:43.999 INFO:tasks.rados.rados.0.smithi071.stdout:1961: oid 8 updating version 1012 to 1013 2024-09-11T01:42:43.999 INFO:tasks.rados.rados.0.smithi071.stdout:1961: oid 8 version 1013 is already newer than 1011 2024-09-11T01:42:43.999 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 1013 (ObjNum 718 snap 222 seq_num 718) dirty exists 2024-09-11T01:42:43.999 INFO:tasks.rados.rados.0.smithi071.stdout:1961: left oid 8 (ObjNum 718 snap 222 seq_num 718) 2024-09-11T01:42:44.969 INFO:tasks.rados.rados.0.smithi071.stdout:1960: done (2 left) 2024-09-11T01:42:44.969 INFO:tasks.rados.rados.0.smithi071.stdout:1961: done (1 left) 2024-09-11T01:42:44.969 INFO:tasks.rados.rados.0.smithi071.stdout:1962: done (0 left) 2024-09-11T01:42:44.970 INFO:tasks.rados.rados.0.smithi071.stdout:1963: copy_from oid 26 from oid 15 current snap is 223 2024-09-11T01:42:44.970 INFO:tasks.rados.rados.0.smithi071.stdout:1964: read oid 25 snap -1 2024-09-11T01:42:44.970 INFO:tasks.rados.rados.0.smithi071.stdout:1964: expect (ObjNum 488 snap 132 seq_num 488) 2024-09-11T01:42:44.970 INFO:tasks.rados.rados.0.smithi071.stdout:1965: rollback oid 5 current snap is 223 2024-09-11T01:42:44.970 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 5 to 214 2024-09-11T01:42:44.970 INFO:tasks.rados.rados.0.smithi071.stdout:1966: snap_remove snap 210 2024-09-11T01:42:45.973 INFO:tasks.rados.rados.0.smithi071.stdout:1966: done (3 left) 2024-09-11T01:42:45.973 INFO:tasks.rados.rados.0.smithi071.stdout:1967: write oid 28 current snap is 223 2024-09-11T01:42:45.973 INFO:tasks.rados.rados.0.smithi071.stdout:1967: seq_num 719 ranges {484233=641213,1707612=376621} 2024-09-11T01:42:45.980 INFO:tasks.rados.rados.0.smithi071.stdout:1967: writing smithi07134777-28 from 484233 to 1125446 tid 1 2024-09-11T01:42:45.986 INFO:tasks.rados.rados.0.smithi071.stdout:1967: writing smithi07134777-28 from 1707612 to 2084233 tid 2 2024-09-11T01:42:45.986 INFO:tasks.rados.rados.0.smithi071.stdout:1968: rollback oid 23 current snap is 223 2024-09-11T01:42:45.986 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 23 to 203 2024-09-11T01:42:45.986 INFO:tasks.rados.rados.0.smithi071.stdout:1965: finishing rollback tid 0 to smithi07134777-5 2024-09-11T01:42:45.986 INFO:tasks.rados.rados.0.smithi071.stdout:1965: finishing rollback tid 1 to smithi07134777-5 2024-09-11T01:42:45.986 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 510 (ObjNum 177 snap 0 seq_num 229) dirty dne 2024-09-11T01:42:45.986 INFO:tasks.rados.rados.0.smithi071.stdout:1964: expect (ObjNum 488 snap 132 seq_num 488) 2024-09-11T01:42:46.041 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 958517 2024-09-11T01:42:46.078 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 922390 2024-09-11T01:42:46.106 INFO:tasks.rados.rados.0.smithi071.stdout:1963: finishing copy_from racing read to smithi07134777-26 2024-09-11T01:42:46.106 INFO:tasks.rados.rados.0.smithi071.stdout:1963: finishing copy_from to smithi07134777-26 2024-09-11T01:42:46.107 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 922 (ObjNum 565 snap 160 seq_num 565) dirty exists 2024-09-11T01:42:46.107 INFO:tasks.rados.rados.0.smithi071.stdout:1967: finishing write tid 1 to smithi07134777-28 2024-09-11T01:42:46.107 INFO:tasks.rados.rados.0.smithi071.stdout:1967: finishing write tid 2 to smithi07134777-28 2024-09-11T01:42:46.107 INFO:tasks.rados.rados.0.smithi071.stdout:1967: finishing write tid 3 to smithi07134777-28 2024-09-11T01:42:46.107 INFO:tasks.rados.rados.0.smithi071.stdout:1967: finishing write tid 4 to smithi07134777-28 2024-09-11T01:42:46.107 INFO:tasks.rados.rados.0.smithi071.stdout:1967: oid 28 updating version 0 to 937 2024-09-11T01:42:46.107 INFO:tasks.rados.rados.0.smithi071.stdout:1967: oid 28 version 937 is already newer than 935 2024-09-11T01:42:46.107 INFO:tasks.rados.rados.0.smithi071.stdout:1967: oid 28 version 937 is already newer than 936 2024-09-11T01:42:46.107 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 937 (ObjNum 719 snap 223 seq_num 719) dirty exists 2024-09-11T01:42:46.107 INFO:tasks.rados.rados.0.smithi071.stdout:1967: left oid 28 (ObjNum 719 snap 223 seq_num 719) 2024-09-11T01:42:46.107 INFO:tasks.rados.rados.0.smithi071.stdout:1963: done (4 left) 2024-09-11T01:42:46.107 INFO:tasks.rados.rados.0.smithi071.stdout:1964: done (3 left) 2024-09-11T01:42:46.107 INFO:tasks.rados.rados.0.smithi071.stdout:1965: done (2 left) 2024-09-11T01:42:46.107 INFO:tasks.rados.rados.0.smithi071.stdout:1967: done (1 left) 2024-09-11T01:42:46.108 INFO:tasks.rados.rados.0.smithi071.stdout:1969: rollback oid 32 current snap is 223 2024-09-11T01:42:46.108 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 32 to 218 2024-09-11T01:42:46.108 INFO:tasks.rados.rados.0.smithi071.stdout:1968: finishing rollback tid 0 to smithi07134777-23 2024-09-11T01:42:46.108 INFO:tasks.rados.rados.0.smithi071.stdout:1968: finishing rollback tid 1 to smithi07134777-23 2024-09-11T01:42:46.108 INFO:tasks.rados.rados.0.smithi071.stdout:1968: finishing rollback tid 2 to smithi07134777-23 2024-09-11T01:42:46.108 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 972 (ObjNum 652 snap 199 seq_num 652) dirty exists 2024-09-11T01:42:46.108 INFO:tasks.rados.rados.0.smithi071.stdout:1968: done (1 left) 2024-09-11T01:42:46.108 INFO:tasks.rados.rados.0.smithi071.stdout:1970: read oid 40 snap 208 2024-09-11T01:42:46.108 INFO:tasks.rados.rados.0.smithi071.stdout:1970: expect (ObjNum 647 snap 199 seq_num 647) 2024-09-11T01:42:46.108 INFO:tasks.rados.rados.0.smithi071.stdout:1971: read oid 4 snap -1 2024-09-11T01:42:46.108 INFO:tasks.rados.rados.0.smithi071.stdout:1971: expect (ObjNum 644 snap 196 seq_num 644) 2024-09-11T01:42:46.108 INFO:tasks.rados.rados.0.smithi071.stdout:1972: write (excl) oid 10 current snap is 223 2024-09-11T01:42:46.108 INFO:tasks.rados.rados.0.smithi071.stdout:1972: seq_num 720 ranges {501040=716462,1766094=334946} 2024-09-11T01:42:46.113 INFO:tasks.rados.rados.0.smithi071.stdout:1972: writing smithi07134777-10 from 501040 to 1217502 tid 1 2024-09-11T01:42:46.119 INFO:tasks.rados.rados.0.smithi071.stdout:1972: writing smithi07134777-10 from 1766094 to 2101040 tid 2 2024-09-11T01:42:46.119 INFO:tasks.rados.rados.0.smithi071.stdout:1973: read oid 31 snap -1 2024-09-11T01:42:46.119 INFO:tasks.rados.rados.0.smithi071.stdout:1973: expect (ObjNum 619 snap 183 seq_num 619) 2024-09-11T01:42:46.119 INFO:tasks.rados.rados.0.smithi071.stdout:1974: delete oid 5 current snap is 223 2024-09-11T01:42:46.119 INFO:tasks.rados.rados.0.smithi071.stdout:1971: expect (ObjNum 644 snap 196 seq_num 644) 2024-09-11T01:42:46.239 INFO:tasks.rados.rados.0.smithi071.stdout:1969: finishing rollback tid 0 to smithi07134777-32 2024-09-11T01:42:46.240 INFO:tasks.rados.rados.0.smithi071.stdout:1970: expect (ObjNum 647 snap 199 seq_num 647) 2024-09-11T01:42:46.240 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:46.271 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:46.300 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:46.332 INFO:tasks.rados.rados.0.smithi071.stdout:1969: finishing rollback tid 1 to smithi07134777-32 2024-09-11T01:42:46.332 INFO:tasks.rados.rados.0.smithi071.stdout:1969: finishing rollback tid 2 to smithi07134777-32 2024-09-11T01:42:46.332 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 976 (ObjNum 707 snap 215 seq_num 707) dirty exists 2024-09-11T01:42:46.332 INFO:tasks.rados.rados.0.smithi071.stdout:1972: finishing write tid 1 to smithi07134777-10 2024-09-11T01:42:46.332 INFO:tasks.rados.rados.0.smithi071.stdout:1972: finishing write tid 2 to smithi07134777-10 2024-09-11T01:42:46.332 INFO:tasks.rados.rados.0.smithi071.stdout:1973: expect (ObjNum 619 snap 183 seq_num 619) 2024-09-11T01:42:46.332 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:46.383 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 493219 2024-09-11T01:42:46.427 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:46.476 INFO:tasks.rados.rados.0.smithi071.stdout:1972: finishing write tid 3 to smithi07134777-10 2024-09-11T01:42:46.476 INFO:tasks.rados.rados.0.smithi071.stdout:1972: finishing write tid 4 to smithi07134777-10 2024-09-11T01:42:46.476 INFO:tasks.rados.rados.0.smithi071.stdout:1972: oid 10 updating version 0 to 392 2024-09-11T01:42:46.477 INFO:tasks.rados.rados.0.smithi071.stdout:1972: oid 10 updating version 392 to 393 2024-09-11T01:42:46.477 INFO:tasks.rados.rados.0.smithi071.stdout:1972: oid 10 version 393 is already newer than 391 2024-09-11T01:42:46.477 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 393 (ObjNum 720 snap 223 seq_num 720) dirty exists 2024-09-11T01:42:46.477 INFO:tasks.rados.rados.0.smithi071.stdout:1972: left oid 10 (ObjNum 720 snap 223 seq_num 720) 2024-09-11T01:42:46.477 INFO:tasks.rados.rados.0.smithi071.stdout:1969: done (5 left) 2024-09-11T01:42:46.477 INFO:tasks.rados.rados.0.smithi071.stdout:1970: done (4 left) 2024-09-11T01:42:46.477 INFO:tasks.rados.rados.0.smithi071.stdout:1971: done (3 left) 2024-09-11T01:42:46.477 INFO:tasks.rados.rados.0.smithi071.stdout:1972: done (2 left) 2024-09-11T01:42:46.477 INFO:tasks.rados.rados.0.smithi071.stdout:1973: done (1 left) 2024-09-11T01:42:46.477 INFO:tasks.rados.rados.0.smithi071.stdout:1974: done (0 left) 2024-09-11T01:42:46.477 INFO:tasks.rados.rados.0.smithi071.stdout:1975: snap_create 2024-09-11T01:42:46.974 INFO:tasks.rados.rados.0.smithi071.stdout:1975: done (0 left) 2024-09-11T01:42:46.974 INFO:tasks.rados.rados.0.smithi071.stdout:1976: copy_from oid 26 from oid 18 current snap is 224 2024-09-11T01:42:46.974 INFO:tasks.rados.rados.0.smithi071.stdout:1977: write oid 28 current snap is 224 2024-09-11T01:42:46.974 INFO:tasks.rados.rados.0.smithi071.stdout:1977: seq_num 722 ranges {534654=466960,1766705=367949} 2024-09-11T01:42:46.979 INFO:tasks.rados.rados.0.smithi071.stdout:1977: writing smithi07134777-28 from 534654 to 1001614 tid 1 2024-09-11T01:42:46.986 INFO:tasks.rados.rados.0.smithi071.stdout:1977: writing smithi07134777-28 from 1766705 to 2134654 tid 2 2024-09-11T01:42:46.986 INFO:tasks.rados.rados.0.smithi071.stdout:1978: read oid 8 snap -1 2024-09-11T01:42:46.986 INFO:tasks.rados.rados.0.smithi071.stdout:1978: expect (ObjNum 718 snap 222 seq_num 718) 2024-09-11T01:42:46.986 INFO:tasks.rados.rados.0.smithi071.stdout:1979: rollback oid 38 current snap is 224 2024-09-11T01:42:46.986 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 38 to 217 2024-09-11T01:42:46.987 INFO:tasks.rados.rados.0.smithi071.stdout:1977: finishing write tid 1 to smithi07134777-28 2024-09-11T01:42:46.987 INFO:tasks.rados.rados.0.smithi071.stdout:1980: copy_from oid 23 from oid 35 current snap is 224 2024-09-11T01:42:46.987 INFO:tasks.rados.rados.0.smithi071.stdout:1981: write (excl) oid 37 current snap is 224 2024-09-11T01:42:46.987 INFO:tasks.rados.rados.0.smithi071.stdout:1981: seq_num 723 ranges {551461=658562,1941540=209921} 2024-09-11T01:42:46.993 INFO:tasks.rados.rados.0.smithi071.stdout:1981: writing smithi07134777-37 from 551461 to 1210023 tid 1 2024-09-11T01:42:46.999 INFO:tasks.rados.rados.0.smithi071.stdout:1981: writing smithi07134777-37 from 1941540 to 2151461 tid 2 2024-09-11T01:42:46.999 INFO:tasks.rados.rados.0.smithi071.stdout:1982: copy_from oid 46 from oid 20 current snap is 224 2024-09-11T01:42:46.999 INFO:tasks.rados.rados.0.smithi071.stdout:1979: finishing rollback tid 0 to smithi07134777-38 2024-09-11T01:42:46.999 INFO:tasks.rados.rados.0.smithi071.stdout:1979: finishing rollback tid 1 to smithi07134777-38 2024-09-11T01:42:46.999 INFO:tasks.rados.rados.0.smithi071.stdout:1977: finishing write tid 2 to smithi07134777-28 2024-09-11T01:42:46.999 INFO:tasks.rados.rados.0.smithi071.stdout:1979: finishing rollback tid 2 to smithi07134777-38 2024-09-11T01:42:46.999 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 1141 (ObjNum 542 snap 154 seq_num 542) dirty exists 2024-09-11T01:42:47.000 INFO:tasks.rados.rados.0.smithi071.stdout:1976: finishing copy_from racing read to smithi07134777-26 2024-09-11T01:42:47.000 INFO:tasks.rados.rados.0.smithi071.stdout:1977: finishing write tid 3 to smithi07134777-28 2024-09-11T01:42:47.000 INFO:tasks.rados.rados.0.smithi071.stdout:1977: finishing write tid 4 to smithi07134777-28 2024-09-11T01:42:47.000 INFO:tasks.rados.rados.0.smithi071.stdout:1977: oid 28 updating version 0 to 940 2024-09-11T01:42:47.000 INFO:tasks.rados.rados.0.smithi071.stdout:1977: oid 28 version 940 is already newer than 939 2024-09-11T01:42:47.000 INFO:tasks.rados.rados.0.smithi071.stdout:1977: oid 28 version 940 is already newer than 938 2024-09-11T01:42:47.000 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 940 (ObjNum 722 snap 224 seq_num 722) dirty exists 2024-09-11T01:42:47.000 INFO:tasks.rados.rados.0.smithi071.stdout:1977: left oid 28 (ObjNum 722 snap 224 seq_num 722) 2024-09-11T01:42:47.000 INFO:tasks.rados.rados.0.smithi071.stdout:1976: finishing copy_from to smithi07134777-26 2024-09-11T01:42:47.000 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 649 (ObjNum 618 snap 183 seq_num 618) dirty exists 2024-09-11T01:42:47.000 INFO:tasks.rados.rados.0.smithi071.stdout:1978: expect (ObjNum 718 snap 222 seq_num 718) 2024-09-11T01:42:47.003 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 717339 2024-09-11T01:42:47.036 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 658932 2024-09-11T01:42:47.071 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1137748 2024-09-11T01:42:47.092 INFO:tasks.rados.rados.0.smithi071.stdout:1981: finishing write tid 1 to smithi07134777-37 2024-09-11T01:42:47.092 INFO:tasks.rados.rados.0.smithi071.stdout:1981: finishing write tid 2 to smithi07134777-37 2024-09-11T01:42:47.092 INFO:tasks.rados.rados.0.smithi071.stdout:1981: finishing write tid 3 to smithi07134777-37 2024-09-11T01:42:47.092 INFO:tasks.rados.rados.0.smithi071.stdout:1981: finishing write tid 4 to smithi07134777-37 2024-09-11T01:42:47.092 INFO:tasks.rados.rados.0.smithi071.stdout:1981: oid 37 updating version 0 to 977 2024-09-11T01:42:47.092 INFO:tasks.rados.rados.0.smithi071.stdout:1981: oid 37 updating version 977 to 978 2024-09-11T01:42:47.093 INFO:tasks.rados.rados.0.smithi071.stdout:1981: oid 37 updating version 978 to 979 2024-09-11T01:42:47.093 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 979 (ObjNum 723 snap 224 seq_num 723) dirty exists 2024-09-11T01:42:47.093 INFO:tasks.rados.rados.0.smithi071.stdout:1981: left oid 37 (ObjNum 723 snap 224 seq_num 723) 2024-09-11T01:42:47.093 INFO:tasks.rados.rados.0.smithi071.stdout:1980: finishing copy_from racing read to smithi07134777-23 2024-09-11T01:42:47.093 INFO:tasks.rados.rados.0.smithi071.stdout:1980: finishing copy_from to smithi07134777-23 2024-09-11T01:42:47.093 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 964 (ObjNum 689 snap 204 seq_num 689) dirty exists 2024-09-11T01:42:47.093 INFO:tasks.rados.rados.0.smithi071.stdout:1982: finishing copy_from racing read to smithi07134777-46 2024-09-11T01:42:47.093 INFO:tasks.rados.rados.0.smithi071.stdout:1976: done (6 left) 2024-09-11T01:42:47.093 INFO:tasks.rados.rados.0.smithi071.stdout:1977: done (5 left) 2024-09-11T01:42:47.093 INFO:tasks.rados.rados.0.smithi071.stdout:1978: done (4 left) 2024-09-11T01:42:47.093 INFO:tasks.rados.rados.0.smithi071.stdout:1979: done (3 left) 2024-09-11T01:42:47.093 INFO:tasks.rados.rados.0.smithi071.stdout:1980: done (2 left) 2024-09-11T01:42:47.093 INFO:tasks.rados.rados.0.smithi071.stdout:1981: done (1 left) 2024-09-11T01:42:47.093 INFO:tasks.rados.rados.0.smithi071.stdout:1983: read oid 47 snap 198 2024-09-11T01:42:47.093 INFO:tasks.rados.rados.0.smithi071.stdout:1983: expect (ObjNum 599 snap 171 seq_num 599) 2024-09-11T01:42:47.094 INFO:tasks.rados.rados.0.smithi071.stdout:1984: snap_remove snap 217 2024-09-11T01:42:47.986 INFO:tasks.rados.rados.0.smithi071.stdout:1984: done (2 left) 2024-09-11T01:42:47.986 INFO:tasks.rados.rados.0.smithi071.stdout:1985: read oid 9 snap -1 2024-09-11T01:42:47.986 INFO:tasks.rados.rados.0.smithi071.stdout:1985: expect (ObjNum 708 snap 215 seq_num 708) 2024-09-11T01:42:47.986 INFO:tasks.rados.rados.0.smithi071.stdout:1986: copy_from oid 19 from oid 42 current snap is 224 2024-09-11T01:42:47.986 INFO:tasks.rados.rados.0.smithi071.stdout:1982: finishing copy_from to smithi07134777-46 2024-09-11T01:42:47.986 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 969 (ObjNum 701 snap 211 seq_num 701) dirty exists 2024-09-11T01:42:47.987 INFO:tasks.rados.rados.0.smithi071.stdout:1983: expect (ObjNum 599 snap 171 seq_num 599) 2024-09-11T01:42:47.996 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1873026 2024-09-11T01:42:47.998 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 115205 2024-09-11T01:42:48.024 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 675207 2024-09-11T01:42:48.040 INFO:tasks.rados.rados.0.smithi071.stdout:1986: finishing copy_from racing read to smithi07134777-19 2024-09-11T01:42:48.040 INFO:tasks.rados.rados.0.smithi071.stdout:1986: finishing copy_from to smithi07134777-19 2024-09-11T01:42:48.040 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 820 (ObjNum 648 snap 199 seq_num 648) dirty exists 2024-09-11T01:42:48.040 INFO:tasks.rados.rados.0.smithi071.stdout:1985: expect (ObjNum 708 snap 215 seq_num 708) 2024-09-11T01:42:48.100 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:48.221 INFO:tasks.rados.rados.0.smithi071.stdout:1982: done (3 left) 2024-09-11T01:42:48.221 INFO:tasks.rados.rados.0.smithi071.stdout:1983: done (2 left) 2024-09-11T01:42:48.221 INFO:tasks.rados.rados.0.smithi071.stdout:1985: done (1 left) 2024-09-11T01:42:48.221 INFO:tasks.rados.rados.0.smithi071.stdout:1986: done (0 left) 2024-09-11T01:42:48.221 INFO:tasks.rados.rados.0.smithi071.stdout:1987: write oid 28 current snap is 224 2024-09-11T01:42:48.221 INFO:tasks.rados.rados.0.smithi071.stdout:1987: seq_num 724 ranges {568268=733811,2000022=168246} 2024-09-11T01:42:48.227 INFO:tasks.rados.rados.0.smithi071.stdout:1987: writing smithi07134777-28 from 568268 to 1302079 tid 1 2024-09-11T01:42:48.233 INFO:tasks.rados.rados.0.smithi071.stdout:1987: writing smithi07134777-28 from 2000022 to 2168268 tid 2 2024-09-11T01:42:48.233 INFO:tasks.rados.rados.0.smithi071.stdout:1988: delete oid 42 current snap is 224 2024-09-11T01:42:48.233 INFO:tasks.rados.rados.0.smithi071.stdout:1987: finishing write tid 1 to smithi07134777-28 2024-09-11T01:42:48.235 INFO:tasks.rados.rados.0.smithi071.stdout:1988: done (1 left) 2024-09-11T01:42:48.236 INFO:tasks.rados.rados.0.smithi071.stdout:1989: snap_create 2024-09-11T01:42:48.236 INFO:tasks.rados.rados.0.smithi071.stdout:1987: finishing write tid 2 to smithi07134777-28 2024-09-11T01:42:48.238 INFO:tasks.rados.rados.0.smithi071.stdout:1987: finishing write tid 3 to smithi07134777-28 2024-09-11T01:42:48.238 INFO:tasks.rados.rados.0.smithi071.stdout:1987: finishing write tid 4 to smithi07134777-28 2024-09-11T01:42:48.238 INFO:tasks.rados.rados.0.smithi071.stdout:1987: oid 28 updating version 0 to 942 2024-09-11T01:42:48.238 INFO:tasks.rados.rados.0.smithi071.stdout:1987: oid 28 version 942 is already newer than 941 2024-09-11T01:42:48.239 INFO:tasks.rados.rados.0.smithi071.stdout:1987: oid 28 updating version 942 to 943 2024-09-11T01:42:48.239 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 943 (ObjNum 724 snap 224 seq_num 724) dirty exists 2024-09-11T01:42:48.239 INFO:tasks.rados.rados.0.smithi071.stdout:1987: left oid 28 (ObjNum 724 snap 224 seq_num 724) 2024-09-11T01:42:48.970 INFO:tasks.rados.rados.0.smithi071.stdout:1987: done (1 left) 2024-09-11T01:42:48.970 INFO:tasks.rados.rados.0.smithi071.stdout:1989: done (0 left) 2024-09-11T01:42:48.970 INFO:tasks.rados.rados.0.smithi071.stdout:1990: read oid 34 snap -1 2024-09-11T01:42:48.970 INFO:tasks.rados.rados.0.smithi071.stdout:1990: expect (ObjNum 666 snap 202 seq_num 666) 2024-09-11T01:42:48.970 INFO:tasks.rados.rados.0.smithi071.stdout:1991: read oid 19 snap 212 2024-09-11T01:42:48.970 INFO:tasks.rados.rados.0.smithi071.stdout:1991: expect (ObjNum 640 snap 195 seq_num 640) 2024-09-11T01:42:48.970 INFO:tasks.rados.rados.0.smithi071.stdout:1992: rollback oid 29 current snap is 225 2024-09-11T01:42:48.970 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 29 to 203 2024-09-11T01:42:48.970 INFO:tasks.rados.rados.0.smithi071.stdout:1993: copy_from oid 12 from oid 13 current snap is 225 2024-09-11T01:42:48.970 INFO:tasks.rados.rados.0.smithi071.stdout:1994: write (excl) oid 21 current snap is 225 2024-09-11T01:42:48.971 INFO:tasks.rados.rados.0.smithi071.stdout:1994: seq_num 726 ranges {601882=484309,1600633=524904} 2024-09-11T01:42:48.976 INFO:tasks.rados.rados.0.smithi071.stdout:1994: writing smithi07134777-21 from 601882 to 1086191 tid 1 2024-09-11T01:42:48.986 INFO:tasks.rados.rados.0.smithi071.stdout:1994: writing smithi07134777-21 from 1600633 to 2125537 tid 2 2024-09-11T01:42:48.986 INFO:tasks.rados.rados.0.smithi071.stdout:1995: snap_remove snap 224 2024-09-11T01:42:49.995 INFO:tasks.rados.rados.0.smithi071.stdout:1995: done (5 left) 2024-09-11T01:42:49.995 INFO:tasks.rados.rados.0.smithi071.stdout:1996: write oid 41 current snap is 225 2024-09-11T01:42:49.995 INFO:tasks.rados.rados.0.smithi071.stdout:1996: seq_num 727 ranges {618689=559558,1659115=559574} 2024-09-11T01:42:50.001 INFO:tasks.rados.rados.0.smithi071.stdout:1996: writing smithi07134777-41 from 618689 to 1178247 tid 1 2024-09-11T01:42:50.007 INFO:tasks.rados.rados.0.smithi071.stdout:1996: writing smithi07134777-41 from 1659115 to 2218689 tid 2 2024-09-11T01:42:50.007 INFO:tasks.rados.rados.0.smithi071.stdout:1997: delete oid 36 current snap is 225 2024-09-11T01:42:50.007 INFO:tasks.rados.rados.0.smithi071.stdout:1992: finishing rollback tid 0 to smithi07134777-29 2024-09-11T01:42:50.007 INFO:tasks.rados.rados.0.smithi071.stdout:1992: finishing rollback tid 1 to smithi07134777-29 2024-09-11T01:42:50.007 INFO:tasks.rados.rados.0.smithi071.stdout:1991: expect (ObjNum 640 snap 195 seq_num 640) 2024-09-11T01:42:50.021 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1832028 2024-09-11T01:42:50.141 INFO:tasks.rados.rados.0.smithi071.stdout:1992: finishing rollback tid 2 to smithi07134777-29 2024-09-11T01:42:50.141 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 1147 (ObjNum 661 snap 201 seq_num 661) dirty exists 2024-09-11T01:42:50.141 INFO:tasks.rados.rados.0.smithi071.stdout:1990: expect (ObjNum 666 snap 202 seq_num 666) 2024-09-11T01:42:50.170 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1875133 2024-09-11T01:42:50.317 INFO:tasks.rados.rados.0.smithi071.stdout:1994: finishing write tid 1 to smithi07134777-21 2024-09-11T01:42:50.317 INFO:tasks.rados.rados.0.smithi071.stdout:1994: finishing write tid 2 to smithi07134777-21 2024-09-11T01:42:50.318 INFO:tasks.rados.rados.0.smithi071.stdout:1994: finishing write tid 3 to smithi07134777-21 2024-09-11T01:42:50.318 INFO:tasks.rados.rados.0.smithi071.stdout:1994: finishing write tid 4 to smithi07134777-21 2024-09-11T01:42:50.318 INFO:tasks.rados.rados.0.smithi071.stdout:1994: oid 21 updating version 0 to 945 2024-09-11T01:42:50.318 INFO:tasks.rados.rados.0.smithi071.stdout:1994: oid 21 version 945 is already newer than 944 2024-09-11T01:42:50.318 INFO:tasks.rados.rados.0.smithi071.stdout:1994: oid 21 updating version 945 to 946 2024-09-11T01:42:50.318 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 946 (ObjNum 726 snap 225 seq_num 726) dirty exists 2024-09-11T01:42:50.318 INFO:tasks.rados.rados.0.smithi071.stdout:1994: left oid 21 (ObjNum 726 snap 225 seq_num 726) 2024-09-11T01:42:50.318 INFO:tasks.rados.rados.0.smithi071.stdout:1993: finishing copy_from racing read to smithi07134777-12 2024-09-11T01:42:50.318 INFO:tasks.rados.rados.0.smithi071.stdout:1993: finishing copy_from to smithi07134777-12 2024-09-11T01:42:50.318 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 831 (ObjNum 700 snap 211 seq_num 700) dirty exists 2024-09-11T01:42:50.318 INFO:tasks.rados.rados.0.smithi071.stdout:1990: done (6 left) 2024-09-11T01:42:50.318 INFO:tasks.rados.rados.0.smithi071.stdout:1991: done (5 left) 2024-09-11T01:42:50.319 INFO:tasks.rados.rados.0.smithi071.stdout:1992: done (4 left) 2024-09-11T01:42:50.319 INFO:tasks.rados.rados.0.smithi071.stdout:1993: done (3 left) 2024-09-11T01:42:50.319 INFO:tasks.rados.rados.0.smithi071.stdout:1994: done (2 left) 2024-09-11T01:42:50.319 INFO:tasks.rados.rados.0.smithi071.stdout:1997: done (1 left) 2024-09-11T01:42:50.319 INFO:tasks.rados.rados.0.smithi071.stdout:1998: delete oid 26 current snap is 225 2024-09-11T01:42:50.319 INFO:tasks.rados.rados.0.smithi071.stdout:1996: finishing write tid 1 to smithi07134777-41 2024-09-11T01:42:50.319 INFO:tasks.rados.rados.0.smithi071.stdout:1996: finishing write tid 2 to smithi07134777-41 2024-09-11T01:42:50.319 INFO:tasks.rados.rados.0.smithi071.stdout:1996: finishing write tid 3 to smithi07134777-41 2024-09-11T01:42:50.319 INFO:tasks.rados.rados.0.smithi071.stdout:1996: finishing write tid 4 to smithi07134777-41 2024-09-11T01:42:50.319 INFO:tasks.rados.rados.0.smithi071.stdout:1996: oid 41 updating version 0 to 396 2024-09-11T01:42:50.319 INFO:tasks.rados.rados.0.smithi071.stdout:1996: oid 41 version 396 is already newer than 395 2024-09-11T01:42:50.319 INFO:tasks.rados.rados.0.smithi071.stdout:1996: oid 41 version 396 is already newer than 394 2024-09-11T01:42:50.319 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 396 (ObjNum 727 snap 225 seq_num 727) dirty exists 2024-09-11T01:42:50.319 INFO:tasks.rados.rados.0.smithi071.stdout:1996: left oid 41 (ObjNum 727 snap 225 seq_num 727) 2024-09-11T01:42:50.320 INFO:tasks.rados.rados.0.smithi071.stdout:1996: done (1 left) 2024-09-11T01:42:50.320 INFO:tasks.rados.rados.0.smithi071.stdout:1998: done (0 left) 2024-09-11T01:42:50.321 INFO:tasks.rados.rados.0.smithi071.stdout:1999: read oid 2 snap -1 2024-09-11T01:42:50.321 INFO:tasks.rados.rados.0.smithi071.stdout:1999: expect (ObjNum 686 snap 204 seq_num 686) 2024-09-11T01:42:50.321 INFO:tasks.rados.rados.0.smithi071.stdout:2000: rollback oid 18 current snap is 225 2024-09-11T01:42:50.321 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 18 to 185 2024-09-11T01:42:50.321 INFO:tasks.rados.rados.0.smithi071.stdout:2001: rollback oid 29 current snap is 225 2024-09-11T01:42:50.321 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 29 to 206 2024-09-11T01:42:50.321 INFO:tasks.rados.rados.0.smithi071.stdout:2002: snap_remove snap 223 2024-09-11T01:42:50.995 INFO:tasks.rados.rados.0.smithi071.stdout:2002: done (3 left) 2024-09-11T01:42:50.995 INFO:tasks.rados.rados.0.smithi071.stdout:2003: snap_remove snap 191 2024-09-11T01:42:52.000 INFO:tasks.rados.rados.0.smithi071.stdout:2003: done (3 left) 2024-09-11T01:42:52.000 INFO:tasks.rados.rados.0.smithi071.stdout:2004: write (excl) oid 33 current snap is 225 2024-09-11T01:42:52.000 INFO:tasks.rados.rados.0.smithi071.stdout:2004: seq_num 730 ranges {669110=501658,1834561=434549} 2024-09-11T01:42:52.007 INFO:tasks.rados.rados.0.smithi071.stdout:2004: writing smithi07134777-33 from 669110 to 1170768 tid 1 2024-09-11T01:42:52.013 INFO:tasks.rados.rados.0.smithi071.stdout:2004: writing smithi07134777-33 from 1834561 to 2269110 tid 2 2024-09-11T01:42:52.013 INFO:tasks.rados.rados.0.smithi071.stdout:2005: delete oid 10 current snap is 225 2024-09-11T01:42:52.013 INFO:tasks.rados.rados.0.smithi071.stdout:1999: expect (ObjNum 686 snap 204 seq_num 686) 2024-09-11T01:42:52.089 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2446116 2024-09-11T01:42:52.111 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 428896 2024-09-11T01:42:52.163 INFO:tasks.rados.rados.0.smithi071.stdout:2001: finishing rollback tid 0 to smithi07134777-29 2024-09-11T01:42:52.163 INFO:tasks.rados.rados.0.smithi071.stdout:2001: finishing rollback tid 1 to smithi07134777-29 2024-09-11T01:42:52.163 INFO:tasks.rados.rados.0.smithi071.stdout:2000: finishing rollback tid 0 to smithi07134777-18 2024-09-11T01:42:52.163 INFO:tasks.rados.rados.0.smithi071.stdout:2000: finishing rollback tid 1 to smithi07134777-18 2024-09-11T01:42:52.163 INFO:tasks.rados.rados.0.smithi071.stdout:2001: finishing rollback tid 2 to smithi07134777-29 2024-09-11T01:42:52.163 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 1152 (ObjNum 673 snap 204 seq_num 673) dirty exists 2024-09-11T01:42:52.163 INFO:tasks.rados.rados.0.smithi071.stdout:2000: finishing rollback tid 2 to smithi07134777-18 2024-09-11T01:42:52.163 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 921 (ObjNum 618 snap 183 seq_num 618) dirty exists 2024-09-11T01:42:52.163 INFO:tasks.rados.rados.0.smithi071.stdout:1999: done (4 left) 2024-09-11T01:42:52.164 INFO:tasks.rados.rados.0.smithi071.stdout:2000: done (3 left) 2024-09-11T01:42:52.164 INFO:tasks.rados.rados.0.smithi071.stdout:2001: done (2 left) 2024-09-11T01:42:52.164 INFO:tasks.rados.rados.0.smithi071.stdout:2005: done (1 left) 2024-09-11T01:42:52.164 INFO:tasks.rados.rados.0.smithi071.stdout:2006: snap_remove snap 222 2024-09-11T01:42:53.005 INFO:tasks.rados.rados.0.smithi071.stdout:2006: done (1 left) 2024-09-11T01:42:53.005 INFO:tasks.rados.rados.0.smithi071.stdout:2007: copy_from oid 43 from oid 38 current snap is 225 2024-09-11T01:42:53.005 INFO:tasks.rados.rados.0.smithi071.stdout:2004: finishing write tid 1 to smithi07134777-33 2024-09-11T01:42:53.005 INFO:tasks.rados.rados.0.smithi071.stdout:2004: finishing write tid 2 to smithi07134777-33 2024-09-11T01:42:53.006 INFO:tasks.rados.rados.0.smithi071.stdout:2004: finishing write tid 3 to smithi07134777-33 2024-09-11T01:42:53.006 INFO:tasks.rados.rados.0.smithi071.stdout:2004: finishing write tid 4 to smithi07134777-33 2024-09-11T01:42:53.006 INFO:tasks.rados.rados.0.smithi071.stdout:2004: oid 33 updating version 0 to 1023 2024-09-11T01:42:53.006 INFO:tasks.rados.rados.0.smithi071.stdout:2004: oid 33 updating version 1023 to 1025 2024-09-11T01:42:53.006 INFO:tasks.rados.rados.0.smithi071.stdout:2004: oid 33 version 1025 is already newer than 1024 2024-09-11T01:42:53.006 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 1025 (ObjNum 730 snap 225 seq_num 730) dirty exists 2024-09-11T01:42:53.006 INFO:tasks.rados.rados.0.smithi071.stdout:2004: left oid 33 (ObjNum 730 snap 225 seq_num 730) 2024-09-11T01:42:53.006 INFO:tasks.rados.rados.0.smithi071.stdout:2004: done (1 left) 2024-09-11T01:42:53.006 INFO:tasks.rados.rados.0.smithi071.stdout:2008: read oid 42 snap -1 2024-09-11T01:42:53.006 INFO:tasks.rados.rados.0.smithi071.stdout:2008: expect deleted 2024-09-11T01:42:53.006 INFO:tasks.rados.rados.0.smithi071.stdout:2009: read oid 14 snap 196 2024-09-11T01:42:53.006 INFO:tasks.rados.rados.0.smithi071.stdout:2009: expect (ObjNum 542 snap 154 seq_num 542) 2024-09-11T01:42:53.006 INFO:tasks.rados.rados.0.smithi071.stdout:2010: read oid 27 snap -1 2024-09-11T01:42:53.007 INFO:tasks.rados.rados.0.smithi071.stdout:2010: expect deleted 2024-09-11T01:42:53.007 INFO:tasks.rados.rados.0.smithi071.stdout:2011: read oid 6 snap -1 2024-09-11T01:42:53.007 INFO:tasks.rados.rados.0.smithi071.stdout:2011: expect (ObjNum 617 snap 183 seq_num 617) 2024-09-11T01:42:53.007 INFO:tasks.rados.rados.0.smithi071.stdout:2012: copy_from oid 26 from oid 29 current snap is 225 2024-09-11T01:42:53.007 INFO:tasks.rados.rados.0.smithi071.stdout:2013: write oid 3 current snap is 225 2024-09-11T01:42:53.007 INFO:tasks.rados.rados.0.smithi071.stdout:2013: seq_num 732 ranges {702724=652156,1951525=351199} 2024-09-11T01:42:53.012 INFO:tasks.rados.rados.0.smithi071.stdout:2013: writing smithi07134777-3 from 702724 to 1354880 tid 1 2024-09-11T01:42:53.019 INFO:tasks.rados.rados.0.smithi071.stdout:2013: writing smithi07134777-3 from 1951525 to 2302724 tid 2 2024-09-11T01:42:53.019 INFO:tasks.rados.rados.0.smithi071.stdout:2014: snap_create 2024-09-11T01:42:53.019 INFO:tasks.rados.rados.0.smithi071.stdout:2009: expect (ObjNum 542 snap 154 seq_num 542) 2024-09-11T01:42:53.129 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 270667 2024-09-11T01:42:53.182 INFO:tasks.rados.rados.0.smithi071.stdout:2011: expect (ObjNum 617 snap 183 seq_num 617) 2024-09-11T01:42:53.238 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:53.335 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2086565 2024-09-11T01:42:53.337 INFO:tasks.rados.rados.0.smithi071.stdout:2013: finishing write tid 1 to smithi07134777-3 2024-09-11T01:42:53.337 INFO:tasks.rados.rados.0.smithi071.stdout:2013: finishing write tid 2 to smithi07134777-3 2024-09-11T01:42:53.337 INFO:tasks.rados.rados.0.smithi071.stdout:2013: finishing write tid 3 to smithi07134777-3 2024-09-11T01:42:53.337 INFO:tasks.rados.rados.0.smithi071.stdout:2013: finishing write tid 4 to smithi07134777-3 2024-09-11T01:42:53.338 INFO:tasks.rados.rados.0.smithi071.stdout:2013: oid 3 updating version 0 to 703 2024-09-11T01:42:53.338 INFO:tasks.rados.rados.0.smithi071.stdout:2013: oid 3 version 703 is already newer than 701 2024-09-11T01:42:53.338 INFO:tasks.rados.rados.0.smithi071.stdout:2013: oid 3 version 703 is already newer than 702 2024-09-11T01:42:53.338 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 703 (ObjNum 732 snap 225 seq_num 732) dirty exists 2024-09-11T01:42:53.338 INFO:tasks.rados.rados.0.smithi071.stdout:2013: left oid 3 (ObjNum 732 snap 225 seq_num 732) 2024-09-11T01:42:53.338 INFO:tasks.rados.rados.0.smithi071.stdout:2012: finishing copy_from racing read to smithi07134777-26 2024-09-11T01:42:53.338 INFO:tasks.rados.rados.0.smithi071.stdout:2007: finishing copy_from to smithi07134777-43 2024-09-11T01:42:53.338 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 1141 (ObjNum 542 snap 154 seq_num 542) dirty exists 2024-09-11T01:42:53.338 INFO:tasks.rados.rados.0.smithi071.stdout:2007: finishing copy_from racing read to smithi07134777-43 2024-09-11T01:42:53.338 INFO:tasks.rados.rados.0.smithi071.stdout:2012: finishing copy_from to smithi07134777-26 2024-09-11T01:42:53.338 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 1152 (ObjNum 673 snap 204 seq_num 673) dirty exists 2024-09-11T01:42:53.973 INFO:tasks.rados.rados.0.smithi071.stdout:2007: done (7 left) 2024-09-11T01:42:53.973 INFO:tasks.rados.rados.0.smithi071.stdout:2008: done (6 left) 2024-09-11T01:42:53.973 INFO:tasks.rados.rados.0.smithi071.stdout:2009: done (5 left) 2024-09-11T01:42:53.973 INFO:tasks.rados.rados.0.smithi071.stdout:2010: done (4 left) 2024-09-11T01:42:53.973 INFO:tasks.rados.rados.0.smithi071.stdout:2011: done (3 left) 2024-09-11T01:42:53.973 INFO:tasks.rados.rados.0.smithi071.stdout:2012: done (2 left) 2024-09-11T01:42:53.973 INFO:tasks.rados.rados.0.smithi071.stdout:2013: done (1 left) 2024-09-11T01:42:53.973 INFO:tasks.rados.rados.0.smithi071.stdout:2014: done (0 left) 2024-09-11T01:42:53.973 INFO:tasks.rados.rados.0.smithi071.stdout:2015: snap_remove snap 198 2024-09-11T01:42:54.977 INFO:tasks.rados.rados.0.smithi071.stdout:2015: done (0 left) 2024-09-11T01:42:54.977 INFO:tasks.rados.rados.0.smithi071.stdout:2016: write oid 35 current snap is 226 2024-09-11T01:42:54.977 INFO:tasks.rados.rados.0.smithi071.stdout:2016: seq_num 733 ranges {719531=727405,1893654=425877} 2024-09-11T01:42:54.984 INFO:tasks.rados.rados.0.smithi071.stdout:2016: writing smithi07134777-35 from 719531 to 1446936 tid 1 2024-09-11T01:42:54.990 INFO:tasks.rados.rados.0.smithi071.stdout:2016: writing smithi07134777-35 from 1893654 to 2319531 tid 2 2024-09-11T01:42:54.990 INFO:tasks.rados.rados.0.smithi071.stdout:2017: read oid 15 snap -1 2024-09-11T01:42:54.990 INFO:tasks.rados.rados.0.smithi071.stdout:2017: expect (ObjNum 565 snap 160 seq_num 565) 2024-09-11T01:42:54.990 INFO:tasks.rados.rados.0.smithi071.stdout:2018: write oid 34 current snap is 226 2024-09-11T01:42:54.990 INFO:tasks.rados.rados.0.smithi071.stdout:2018: seq_num 734 ranges {736338=402654,1552136=539580} 2024-09-11T01:42:54.997 INFO:tasks.rados.rados.0.smithi071.stdout:2018: writing smithi07134777-34 from 736338 to 1138992 tid 1 2024-09-11T01:42:55.003 INFO:tasks.rados.rados.0.smithi071.stdout:2018: writing smithi07134777-34 from 1552136 to 2091716 tid 2 2024-09-11T01:42:55.003 INFO:tasks.rados.rados.0.smithi071.stdout:2019: delete oid 12 current snap is 226 2024-09-11T01:42:55.003 INFO:tasks.rados.rados.0.smithi071.stdout:2016: finishing write tid 1 to smithi07134777-35 2024-09-11T01:42:55.003 INFO:tasks.rados.rados.0.smithi071.stdout:2016: finishing write tid 2 to smithi07134777-35 2024-09-11T01:42:55.003 INFO:tasks.rados.rados.0.smithi071.stdout:2016: finishing write tid 3 to smithi07134777-35 2024-09-11T01:42:55.003 INFO:tasks.rados.rados.0.smithi071.stdout:2016: finishing write tid 4 to smithi07134777-35 2024-09-11T01:42:55.003 INFO:tasks.rados.rados.0.smithi071.stdout:2016: oid 35 updating version 0 to 982 2024-09-11T01:42:55.003 INFO:tasks.rados.rados.0.smithi071.stdout:2016: oid 35 version 982 is already newer than 981 2024-09-11T01:42:55.003 INFO:tasks.rados.rados.0.smithi071.stdout:2016: oid 35 version 982 is already newer than 980 2024-09-11T01:42:55.004 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 982 (ObjNum 733 snap 226 seq_num 733) dirty exists 2024-09-11T01:42:55.004 INFO:tasks.rados.rados.0.smithi071.stdout:2016: left oid 35 (ObjNum 733 snap 226 seq_num 733) 2024-09-11T01:42:55.004 INFO:tasks.rados.rados.0.smithi071.stdout:2018: finishing write tid 1 to smithi07134777-34 2024-09-11T01:42:55.004 INFO:tasks.rados.rados.0.smithi071.stdout:2017: expect (ObjNum 565 snap 160 seq_num 565) 2024-09-11T01:42:55.186 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2297352 2024-09-11T01:42:55.229 INFO:tasks.rados.rados.0.smithi071.stdout:2016: done (3 left) 2024-09-11T01:42:55.230 INFO:tasks.rados.rados.0.smithi071.stdout:2017: done (2 left) 2024-09-11T01:42:55.230 INFO:tasks.rados.rados.0.smithi071.stdout:2019: done (1 left) 2024-09-11T01:42:55.230 INFO:tasks.rados.rados.0.smithi071.stdout:2020: copy_from oid 42 from oid 3 current snap is 226 2024-09-11T01:42:55.230 INFO:tasks.rados.rados.0.smithi071.stdout:2018: finishing write tid 2 to smithi07134777-34 2024-09-11T01:42:55.230 INFO:tasks.rados.rados.0.smithi071.stdout:2018: finishing write tid 3 to smithi07134777-34 2024-09-11T01:42:55.230 INFO:tasks.rados.rados.0.smithi071.stdout:2018: finishing write tid 4 to smithi07134777-34 2024-09-11T01:42:55.230 INFO:tasks.rados.rados.0.smithi071.stdout:2018: oid 34 updating version 0 to 1153 2024-09-11T01:42:55.230 INFO:tasks.rados.rados.0.smithi071.stdout:2018: oid 34 updating version 1153 to 1154 2024-09-11T01:42:55.230 INFO:tasks.rados.rados.0.smithi071.stdout:2018: oid 34 updating version 1154 to 1155 2024-09-11T01:42:55.230 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 1155 (ObjNum 734 snap 226 seq_num 734) dirty exists 2024-09-11T01:42:55.230 INFO:tasks.rados.rados.0.smithi071.stdout:2018: left oid 34 (ObjNum 734 snap 226 seq_num 734) 2024-09-11T01:42:55.230 INFO:tasks.rados.rados.0.smithi071.stdout:2018: done (1 left) 2024-09-11T01:42:55.230 INFO:tasks.rados.rados.0.smithi071.stdout:2021: write (excl) oid 8 current snap is 226 2024-09-11T01:42:55.230 INFO:tasks.rados.rados.0.smithi071.stdout:2021: seq_num 736 ranges {769952=553152,2069100=300852} 2024-09-11T01:42:55.236 INFO:tasks.rados.rados.0.smithi071.stdout:2021: writing smithi07134777-8 from 769952 to 1323104 tid 1 2024-09-11T01:42:55.243 INFO:tasks.rados.rados.0.smithi071.stdout:2021: writing smithi07134777-8 from 2069100 to 2369952 tid 2 2024-09-11T01:42:55.244 INFO:tasks.rados.rados.0.smithi071.stdout:2022: read oid 46 snap 213 2024-09-11T01:42:55.244 INFO:tasks.rados.rados.0.smithi071.stdout:2022: expect (ObjNum 691 snap 204 seq_num 691) 2024-09-11T01:42:55.244 INFO:tasks.rados.rados.0.smithi071.stdout:2023: read oid 36 snap 216 2024-09-11T01:42:55.244 INFO:tasks.rados.rados.0.smithi071.stdout:2023: expect deleted 2024-09-11T01:42:55.244 INFO:tasks.rados.rados.0.smithi071.stdout:2024: rollback oid 13 current snap is 226 2024-09-11T01:42:55.244 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 13 to 214 2024-09-11T01:42:55.244 INFO:tasks.rados.rados.0.smithi071.stdout:2020: finishing copy_from racing read to smithi07134777-42 2024-09-11T01:42:55.244 INFO:tasks.rados.rados.0.smithi071.stdout:2020: finishing copy_from to smithi07134777-42 2024-09-11T01:42:55.244 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 703 (ObjNum 732 snap 225 seq_num 732) dirty exists 2024-09-11T01:42:55.244 INFO:tasks.rados.rados.0.smithi071.stdout:2021: finishing write tid 1 to smithi07134777-8 2024-09-11T01:42:55.244 INFO:tasks.rados.rados.0.smithi071.stdout:2020: done (4 left) 2024-09-11T01:42:55.244 INFO:tasks.rados.rados.0.smithi071.stdout:2025: read oid 41 snap -1 2024-09-11T01:42:55.245 INFO:tasks.rados.rados.0.smithi071.stdout:2025: expect (ObjNum 727 snap 225 seq_num 727) 2024-09-11T01:42:55.245 INFO:tasks.rados.rados.0.smithi071.stdout:2026: rollback oid 18 current snap is 226 2024-09-11T01:42:55.245 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 18 to 220 2024-09-11T01:42:55.245 INFO:tasks.rados.rados.0.smithi071.stdout:2027: read oid 22 snap -1 2024-09-11T01:42:55.245 INFO:tasks.rados.rados.0.smithi071.stdout:2027: expect (ObjNum 647 snap 199 seq_num 647) 2024-09-11T01:42:55.245 INFO:tasks.rados.rados.0.smithi071.stdout:2028: rollback oid 33 current snap is 226 2024-09-11T01:42:55.245 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 33 to 180 2024-09-11T01:42:55.245 INFO:tasks.rados.rados.0.smithi071.stdout:2029: write oid 31 current snap is 226 2024-09-11T01:42:55.245 INFO:tasks.rados.rados.0.smithi071.stdout:2029: seq_num 737 ranges {786759=628401,2011229=375530} 2024-09-11T01:42:55.251 INFO:tasks.rados.rados.0.smithi071.stdout:2029: writing smithi07134777-31 from 786759 to 1415160 tid 1 2024-09-11T01:42:55.257 INFO:tasks.rados.rados.0.smithi071.stdout:2029: writing smithi07134777-31 from 2011229 to 2386759 tid 2 2024-09-11T01:42:55.257 INFO:tasks.rados.rados.0.smithi071.stdout:2030: snap_remove snap 221 2024-09-11T01:42:55.820 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-11T01:42:55.821 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-11T01:42:55.821 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-11T01:42:55.824 INFO:tasks.ceph.mon.c:Started 2024-09-11T01:42:55.824 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:42:55.824 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:42:55.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.935+0000 7f5e90945640 10 monclient: get_monmap_and_config 2024-09-11T01:42:55.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.935+0000 7f5e90945640 10 monclient: build_initial_monmap 2024-09-11T01:42:55.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.935+0000 7f5e90945640 10 monclient: monmap: 2024-09-11T01:42:55.937 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:55.937 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:55.937 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:42:55.936424+0000 2024-09-11T01:42:55.937 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:42:55.936424+0000 2024-09-11T01:42:55.937 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:55.937 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:55.938 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:42:55.938 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:42:55.938 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:42:55.938 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:42:55.938 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:42:55.938 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:55.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.936+0000 7f5e90945640 1 Processor -- start 2024-09-11T01:42:55.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.936+0000 7f5e90945640 1 -- start start 2024-09-11T01:42:55.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.936+0000 7f5e90945640 10 monclient: init 2024-09-11T01:42:55.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.936+0000 7f5e90945640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:55.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.936+0000 7f5e90945640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-11T01:42:55.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.936+0000 7f5e90945640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5e8c138230 0x7f5e8c138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:55.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.936+0000 7f5e90945640 10 monclient(hunting): picked mon.noname-a con 0x7f5e8c138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:42:55.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.936+0000 7f5e90945640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e8c09d030 addr v1:172.21.15.71:6789/0 2024-09-11T01:42:55.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.936+0000 7f5e90945640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5e8c09d460 0x7f5e8c0a5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:55.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.936+0000 7f5e90945640 10 monclient(hunting): picked mon.noname-c con 0x7f5e8c09d460 addr v2:172.21.15.71:3300/0 2024-09-11T01:42:55.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.936+0000 7f5e90945640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e8c0a6e30 con 0x7f5e8c09d030 2024-09-11T01:42:55.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.936+0000 7f5e90945640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:55.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.936+0000 7f5e90945640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f5e8c09c9c0 con 0x7f5e8c138230 2024-09-11T01:42:55.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.936+0000 7f5e90945640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:55.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.936+0000 7f5e90945640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f5e8c09ba20 con 0x7f5e8c09d460 2024-09-11T01:42:55.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.936+0000 7f5e90945640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:55.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.936+0000 7f5e90945640 10 monclient(hunting): authenticate will time out at 1582.378897s 2024-09-11T01:42:55.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8a59c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5e8c138230 0x7f5e8c138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:55.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8a59c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5e8c138230 0x7f5e8c138610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48412/0 (socket says 172.21.15.12:48412) 2024-09-11T01:42:55.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8a59c640 1 -- 172.21.15.12:0/1474645799 learned_addr learned my addr 172.21.15.12:0/1474645799 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:55.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8a59c640 10 monclient(hunting): get_auth_request con 0x7f5e8c138230 auth_method 0 2024-09-11T01:42:55.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:55.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:55.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:55.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8ad9d640 1 --2- 172.21.15.12:0/1474645799 >> v2:172.21.15.71:3300/0 conn(0x7f5e8c09d460 0x7f5e8c0a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:55.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8ad9d640 10 monclient(hunting): get_auth_request con 0x7f5e8c09d460 auth_method 0 2024-09-11T01:42:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8a59c640 10 monclient(hunting): handle_auth_done global_id 94644 payload 274 2024-09-11T01:42:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8a59c640 1 -- 172.21.15.12:0/1474645799 >> v2:172.21.15.71:3300/0 conn(0x7f5e8c09d460 msgr2=0x7f5e8c0a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:55.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8a59c640 1 --2- 172.21.15.12:0/1474645799 >> v2:172.21.15.71:3300/0 conn(0x7f5e8c09d460 0x7f5e8c0a5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:55.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8a59c640 1 -- 172.21.15.12:0/1474645799 >> v1:172.21.15.71:6789/0 conn(0x7f5e8c09d030 legacy=0x7f5e8c0a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:55.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8a59c640 10 monclient: _finish_hunting 0 2024-09-11T01:42:55.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8a59c640 1 monclient: found mon.noname-a 2024-09-11T01:42:55.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:55.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:42:55.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.937+0000 7f5e8a59c640 1 -- 172.21.15.12:0/1474645799 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e8c0a66a0 con 0x7f5e8c138230 2024-09-11T01:42:55.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.940+0000 7f5e8a59c640 1 -- 172.21.15.12:0/1474645799 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e8c138230 msgr2=0x7f5e8c138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:55.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.940+0000 7f5e8a59c640 1 --2- 172.21.15.12:0/1474645799 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e8c138230 0x7f5e8c138610 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c09bff0 tx=0x7f5e7c02a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6386a1812991609f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:55.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.940+0000 7f5e89d9b640 1 -- 172.21.15.12:0/1474645799 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c037070 con 0x7f5e8c138230 2024-09-11T01:42:55.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.940+0000 7f5e89d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:55.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.940+0000 7f5e89d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:42:55.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.940+0000 7f5e89d9b640 10 monclient: dump: 2024-09-11T01:42:55.950 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:55.950 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:55.950 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:55.950 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:55.950 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:55.950 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:55.950 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:55.950 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:55.951 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:55.951 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:55.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.940+0000 7f5e89d9b640 10 monclient: _finish_auth 0 2024-09-11T01:42:55.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.940+0000 7f5e89d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:42:55.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.940+0000 7f5e89d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:55.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.940+0000 7f5e89d9b640 1 -- 172.21.15.12:0/1474645799 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e7c032640 con 0x7f5e8c138230 2024-09-11T01:42:55.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.940+0000 7f5e89d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:55.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.940+0000 7f5e89d9b640 1 -- 172.21.15.12:0/1474645799 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c032a10 con 0x7f5e8c138230 2024-09-11T01:42:55.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.940+0000 7f5e90945640 5 monclient: authenticate success, global_id 94644 2024-09-11T01:42:55.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.940+0000 7f5e90945640 10 monclient: get_monmap_and_config success 2024-09-11T01:42:55.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.940+0000 7f5e90945640 4 set_mon_vals no callback set 2024-09-11T01:42:55.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.940+0000 7f5e89d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:55.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.940+0000 7f5e89d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:55.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.940+0000 7f5e89d9b640 10 monclient: dump: 2024-09-11T01:42:55.952 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:55.952 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:55.952 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:55.952 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:55.952 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:55.952 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:55.952 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:55.952 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:55.952 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:55.952 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:55.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.940+0000 7f5e90945640 10 monclient: shutdown 2024-09-11T01:42:55.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.940+0000 7f5e90945640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:55.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.940+0000 7f5e90945640 1 -- 172.21.15.12:0/1474645799 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e8c138230 msgr2=0x7f5e8c138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:55.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.940+0000 7f5e90945640 1 --2- 172.21.15.12:0/1474645799 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e8c138230 0x7f5e8c138610 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c09bff0 tx=0x7f5e7c02a0b0 comp rx=0 tx=0).stop 2024-09-11T01:42:55.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.940+0000 7f5e90945640 1 -- 172.21.15.12:0/1474645799 shutdown_connections 2024-09-11T01:42:55.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.941+0000 7f5e90945640 1 --2- 172.21.15.12:0/1474645799 >> v2:172.21.15.71:3300/0 conn(0x7f5e8c09d460 0x7f5e8c0a5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:55.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.941+0000 7f5e90945640 1 --2- 172.21.15.12:0/1474645799 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e8c138230 0x7f5e8c138610 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:55.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.941+0000 7f5e90945640 1 -- 172.21.15.12:0/1474645799 >> 172.21.15.12:0/1474645799 conn(0x7f5e8c0a7120 msgr2=0x7f5e8c0a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:55.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.941+0000 7f5e90945640 1 -- 172.21.15.12:0/1474645799 shutdown_connections 2024-09-11T01:42:55.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.941+0000 7f5e90945640 1 -- 172.21.15.12:0/1474645799 wait complete. 2024-09-11T01:42:55.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.941+0000 7f5e90945640 10 monclient: build_initial_monmap 2024-09-11T01:42:55.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.941+0000 7f5e90945640 10 monclient: monmap: 2024-09-11T01:42:55.953 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:55.954 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:42:55.954 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:42:55.954 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:42:55.954 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:55.954 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:55.954 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:42:55.954 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:42:55.954 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:42:55.954 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:55.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.942+0000 7f5e90945640 1 Processor -- start 2024-09-11T01:42:55.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.942+0000 7f5e90945640 1 -- start start 2024-09-11T01:42:55.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.942+0000 7f5e90945640 10 monclient: init 2024-09-11T01:42:55.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.942+0000 7f5e90945640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.942+0000 7f5e90945640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:42:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.942+0000 7f5e90945640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e8c09d030 0x7f5e8c137030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.942+0000 7f5e90945640 10 monclient(hunting): picked mon.noname-c con 0x7f5e8c09d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.942+0000 7f5e90945640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e8c09d460 0x7f5e8c137570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.942+0000 7f5e90945640 10 monclient(hunting): picked mon.noname-a con 0x7f5e8c09d460 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:42:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.942+0000 7f5e90945640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e8c138230 0x7f5e8c124c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.942+0000 7f5e90945640 10 monclient(hunting): picked mon.noname-b con 0x7f5e8c138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:42:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.942+0000 7f5e90945640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.942+0000 7f5e90945640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5e8c06f970 con 0x7f5e8c09d460 2024-09-11T01:42:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.942+0000 7f5e90945640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.942+0000 7f5e90945640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5e8c06f7f0 con 0x7f5e8c138230 2024-09-11T01:42:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.942+0000 7f5e90945640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.942+0000 7f5e90945640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5e8c0a5c70 con 0x7f5e8c09d030 2024-09-11T01:42:55.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e90945640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:55.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e90945640 10 monclient(hunting): authenticate will time out at 1582.385012s 2024-09-11T01:42:55.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e8a59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e8c09d030 0x7f5e8c137030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:55.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e81d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e8c09d460 0x7f5e8c137570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:55.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e8a59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e8c09d030 0x7f5e8c137030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:56572/0 (socket says 172.21.15.12:56572) 2024-09-11T01:42:55.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e8a59c640 1 -- 172.21.15.12:0/2506153362 learned_addr learned my addr 172.21.15.12:0/2506153362 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:55.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e8a59c640 10 monclient(hunting): get_auth_request con 0x7f5e8c09d030 auth_method 0 2024-09-11T01:42:55.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e8a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:55.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e8a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:55.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e8a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:55.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e81d9b640 10 monclient(hunting): get_auth_request con 0x7f5e8c09d460 auth_method 0 2024-09-11T01:42:55.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e81d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:55.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e81d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:55.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e81d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:55.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e8a59c640 1 -- 172.21.15.12:0/2506153362 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e8c09d030 msgr2=0x7f5e8c137030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:42:55.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e8a59c640 1 -- 172.21.15.12:0/2506153362 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e8c09d030 msgr2=0x7f5e8c137030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:42:55.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e81d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:55.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e81d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:55.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e8a59c640 1 --2- 172.21.15.12:0/2506153362 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e8c09d030 0x7f5e8c137030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:42:55.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e81d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:55.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e8a59c640 1 --2- 172.21.15.12:0/2506153362 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e8c09d030 0x7f5e8c137030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:55.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e8ad9d640 1 --2- 172.21.15.12:0/2506153362 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e8c138230 0x7f5e8c124c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:55.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e8ad9d640 10 monclient(hunting): get_auth_request con 0x7f5e8c138230 auth_method 0 2024-09-11T01:42:55.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e8ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:55.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e8ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:55.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e8ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:55.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e81d9b640 10 monclient(hunting): handle_auth_done global_id 94647 payload 824 2024-09-11T01:42:55.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e81d9b640 1 -- 172.21.15.12:0/2506153362 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e8c09d030 msgr2=0x7f5e8c137030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:42:55.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e81d9b640 1 --2- 172.21.15.12:0/2506153362 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e8c09d030 0x7f5e8c137030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:55.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e81d9b640 1 -- 172.21.15.12:0/2506153362 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e8c138230 msgr2=0x7f5e8c124c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:55.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e81d9b640 1 --2- 172.21.15.12:0/2506153362 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e8c138230 0x7f5e8c124c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:55.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e81d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:42:55.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e81d9b640 1 monclient: found mon.noname-a 2024-09-11T01:42:55.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e81d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:55.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e81d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:42:55.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e81d9b640 1 -- 172.21.15.12:0/2506153362 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e8c137ab0 con 0x7f5e8c09d460 2024-09-11T01:42:55.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.943+0000 7f5e8ad9d640 1 --2- 172.21.15.12:0/2506153362 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e8c138230 0x7f5e8c124c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:42:55.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.945+0000 7f5e81d9b640 1 -- 172.21.15.12:0/2506153362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e8c09d460 msgr2=0x7f5e8c137570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:55.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.945+0000 7f5e81d9b640 1 --2- 172.21.15.12:0/2506153362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e8c09d460 0x7f5e8c137570 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5e7400c410 tx=0x7f5e7400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:55.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.945+0000 7f5e837fe640 1 -- 172.21.15.12:0/2506153362 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e74017070 con 0x7f5e8c09d460 2024-09-11T01:42:55.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.945+0000 7f5e837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:55.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.945+0000 7f5e837fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:42:55.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.945+0000 7f5e837fe640 10 monclient: dump: 2024-09-11T01:42:55.959 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:55.959 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:55.959 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:55.959 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:55.960 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:55.960 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:55.960 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:55.960 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:55.960 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:55.960 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:55.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.945+0000 7f5e837fe640 10 monclient: _finish_auth 0 2024-09-11T01:42:55.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.945+0000 7f5e837fe640 10 monclient: _check_auth_tickets 2024-09-11T01:42:55.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.945+0000 7f5e837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:55.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.945+0000 7f5e837fe640 1 -- 172.21.15.12:0/2506153362 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e74005230 con 0x7f5e8c09d460 2024-09-11T01:42:55.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.945+0000 7f5e90945640 5 monclient: authenticate success, global_id 94647 2024-09-11T01:42:55.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.945+0000 7f5e837fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:55.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.945+0000 7f5e90945640 10 monclient: _renew_subs 2024-09-11T01:42:55.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.945+0000 7f5e837fe640 1 -- 172.21.15.12:0/2506153362 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e740117a0 con 0x7f5e8c09d460 2024-09-11T01:42:55.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.945+0000 7f5e90945640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:55.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.945+0000 7f5e90945640 1 -- 172.21.15.12:0/2506153362 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e8c1266d0 con 0x7f5e8c09d460 2024-09-11T01:42:55.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.945+0000 7f5e837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:55.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.945+0000 7f5e837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:55.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.945+0000 7f5e837fe640 10 monclient: dump: 2024-09-11T01:42:55.961 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:55.961 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:55.961 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:55.961 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:55.962 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:55.962 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:55.962 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:55.962 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:55.962 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:55.962 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:55.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.945+0000 7f5e8959a640 4 set_mon_vals no callback set 2024-09-11T01:42:55.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.945+0000 7f5e90945640 10 monclient: _renew_subs 2024-09-11T01:42:55.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.945+0000 7f5e90945640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:55.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.945+0000 7f5e90945640 1 -- 172.21.15.12:0/2506153362 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e8c127290 con 0x7f5e8c09d460 2024-09-11T01:42:55.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.946+0000 7f5e90945640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:55.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.946+0000 7f5e90945640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:55.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.946+0000 7f5e90945640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:55.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.946+0000 7f5e90945640 1 -- 172.21.15.12:0/2506153362 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e8c1288a0 con 0x7f5e8c09d460 2024-09-11T01:42:55.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.946+0000 7f5e837fe640 1 -- 172.21.15.12:0/2506153362 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f5e740119c0 con 0x7f5e8c09d460 2024-09-11T01:42:55.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.947+0000 7f5e837fe640 1 --2- 172.21.15.12:0/2506153362 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5e6c03e210 0x7f5e6c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:55.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.947+0000 7f5e837fe640 1 -- 172.21.15.12:0/2506153362 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(462..462 src has 285..462) ==== 4815+0+0 (secure 0 0 0) 0x7f5e740522f0 con 0x7f5e8c09d460 2024-09-11T01:42:55.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.947+0000 7f5e8a59c640 1 --2- 172.21.15.12:0/2506153362 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5e6c03e210 0x7f5e6c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:55.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.947+0000 7f5e8a59c640 10 monclient: get_auth_request con 0x7f5e6c03e210 auth_method 0 2024-09-11T01:42:55.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.949+0000 7f5e8a59c640 1 --2- 172.21.15.12:0/2506153362 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5e6c03e210 0x7f5e6c0406d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c0090b0 tx=0x7f5e7c02e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:55.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.951+0000 7f5e837fe640 1 -- 172.21.15.12:0/2506153362 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5e740192c0 con 0x7f5e8c09d460 2024-09-11T01:42:55.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.951+0000 7f5e837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:55.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:55.951+0000 7f5e837fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:42:55.978 INFO:tasks.rados.rados.0.smithi071.stdout:2030: done (9 left) 2024-09-11T01:42:55.978 INFO:tasks.rados.rados.0.smithi071.stdout:2031: write (excl) oid 21 current snap is 226 2024-09-11T01:42:55.978 INFO:tasks.rados.rados.0.smithi071.stdout:2031: seq_num 738 ranges {403566=420003,1386064=546918} 2024-09-11T01:42:55.982 INFO:tasks.rados.rados.0.smithi071.stdout:2031: writing smithi07134777-21 from 403566 to 823569 tid 1 2024-09-11T01:42:55.989 INFO:tasks.rados.rados.0.smithi071.stdout:2031: writing smithi07134777-21 from 1386064 to 1932982 tid 2 2024-09-11T01:42:55.989 INFO:tasks.rados.rados.0.smithi071.stdout:2032: write (excl) oid 45 current snap is 226 2024-09-11T01:42:55.989 INFO:tasks.rados.rados.0.smithi071.stdout:2032: seq_num 739 ranges {420373=495252,1444546=690576} 2024-09-11T01:42:55.994 INFO:tasks.rados.rados.0.smithi071.stdout:2032: writing smithi07134777-45 from 420373 to 915625 tid 1 2024-09-11T01:42:56.001 INFO:tasks.rados.rados.0.smithi071.stdout:2032: writing smithi07134777-45 from 1444546 to 2135122 tid 2 2024-09-11T01:42:56.001 INFO:tasks.rados.rados.0.smithi071.stdout:2033: delete oid 49 current snap is 226 2024-09-11T01:42:56.001 INFO:tasks.rados.rados.0.smithi071.stdout:2025: expect (ObjNum 727 snap 225 seq_num 727) 2024-09-11T01:42:56.078 INFO:tasks.rados.rados.0.smithi071.stdout:2026: finishing rollback tid 0 to smithi07134777-18 2024-09-11T01:42:56.078 INFO:tasks.rados.rados.0.smithi071.stdout:2026: finishing rollback tid 1 to smithi07134777-18 2024-09-11T01:42:56.079 INFO:tasks.rados.rados.0.smithi071.stdout:2024: finishing rollback tid 0 to smithi07134777-13 2024-09-11T01:42:56.079 INFO:tasks.rados.rados.0.smithi071.stdout:2028: finishing rollback tid 0 to smithi07134777-33 2024-09-11T01:42:56.079 INFO:tasks.rados.rados.0.smithi071.stdout:2021: finishing write tid 2 to smithi07134777-8 2024-09-11T01:42:56.079 INFO:tasks.rados.rados.0.smithi071.stdout:2027: expect (ObjNum 647 snap 199 seq_num 647) 2024-09-11T01:42:56.113 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:56.144 INFO:tasks.rados.rados.0.smithi071.stdout:2028: finishing rollback tid 1 to smithi07134777-33 2024-09-11T01:42:56.144 INFO:tasks.rados.rados.0.smithi071.stdout:2021: finishing write tid 3 to smithi07134777-8 2024-09-11T01:42:56.144 INFO:tasks.rados.rados.0.smithi071.stdout:2021: finishing write tid 4 to smithi07134777-8 2024-09-11T01:42:56.144 INFO:tasks.rados.rados.0.smithi071.stdout:2021: oid 8 updating version 0 to 1015 2024-09-11T01:42:56.144 INFO:tasks.rados.rados.0.smithi071.stdout:2021: oid 8 version 1015 is already newer than 1014 2024-09-11T01:42:56.144 INFO:tasks.rados.rados.0.smithi071.stdout:2021: oid 8 updating version 1015 to 1016 2024-09-11T01:42:56.144 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 1016 (ObjNum 736 snap 226 seq_num 736) dirty exists 2024-09-11T01:42:56.144 INFO:tasks.rados.rados.0.smithi071.stdout:2021: left oid 8 (ObjNum 736 snap 226 seq_num 736) 2024-09-11T01:42:56.144 INFO:tasks.rados.rados.0.smithi071.stdout:2026: finishing rollback tid 2 to smithi07134777-18 2024-09-11T01:42:56.145 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 924 (ObjNum 618 snap 183 seq_num 618) dirty exists 2024-09-11T01:42:56.145 INFO:tasks.rados.rados.0.smithi071.stdout:2028: finishing rollback tid 2 to smithi07134777-33 2024-09-11T01:42:56.145 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 1159 (ObjNum 250 snap 63 seq_num 250) dirty exists 2024-09-11T01:42:56.145 INFO:tasks.rados.rados.0.smithi071.stdout:2024: finishing rollback tid 1 to smithi07134777-13 2024-09-11T01:42:56.145 INFO:tasks.rados.rados.0.smithi071.stdout:2024: finishing rollback tid 2 to smithi07134777-13 2024-09-11T01:42:56.145 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 842 (ObjNum 700 snap 211 seq_num 700) dirty exists 2024-09-11T01:42:56.145 INFO:tasks.rados.rados.0.smithi071.stdout:2022: expect (ObjNum 691 snap 204 seq_num 691) 2024-09-11T01:42:56.145 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:56.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:56.217+0000 7f5e90945640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:42:56.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:56.217+0000 7f5e90945640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:42:56.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:56.217+0000 7f5e90945640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:56.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:56.217+0000 7f5e90945640 1 -- 172.21.15.12:0/2506153362 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5e8c138610 con 0x7f5e8c09d460 2024-09-11T01:42:56.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:56.218+0000 7f5e837fe640 1 -- 172.21.15.12:0/2506153362 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f5e74052080 con 0x7f5e8c09d460 2024-09-11T01:42:56.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:56.218+0000 7f5e837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:42:56.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:56.218+0000 7f5e837fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:42:56.220 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":126,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:42:56.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:56.220+0000 7f5e90945640 1 -- 172.21.15.12:0/2506153362 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5e6c03e210 msgr2=0x7f5e6c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:56.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:56.220+0000 7f5e90945640 1 --2- 172.21.15.12:0/2506153362 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5e6c03e210 0x7f5e6c0406d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c0090b0 tx=0x7f5e7c02e000 comp rx=0 tx=0).stop 2024-09-11T01:42:56.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:56.220+0000 7f5e90945640 10 monclient: shutdown 2024-09-11T01:42:56.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:56.220+0000 7f5e90945640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:56.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:56.220+0000 7f5e90945640 1 -- 172.21.15.12:0/2506153362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e8c09d460 msgr2=0x7f5e8c137570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:56.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:56.220+0000 7f5e90945640 1 --2- 172.21.15.12:0/2506153362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e8c09d460 0x7f5e8c137570 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5e7400c410 tx=0x7f5e7400c8d0 comp rx=0 tx=0).stop 2024-09-11T01:42:56.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:56.220+0000 7f5e90945640 1 -- 172.21.15.12:0/2506153362 shutdown_connections 2024-09-11T01:42:56.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:56.220+0000 7f5e90945640 1 --2- 172.21.15.12:0/2506153362 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e8c138230 0x7f5e8c124c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:56.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:56.220+0000 7f5e90945640 1 --2- 172.21.15.12:0/2506153362 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5e6c03e210 0x7f5e6c0406d0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:56.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:56.220+0000 7f5e90945640 1 --2- 172.21.15.12:0/2506153362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5e8c09d460 0x7f5e8c137570 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:56.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:56.220+0000 7f5e90945640 1 --2- 172.21.15.12:0/2506153362 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5e8c09d030 0x7f5e8c137030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:56.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:56.220+0000 7f5e90945640 1 -- 172.21.15.12:0/2506153362 >> 172.21.15.12:0/2506153362 conn(0x7f5e8c0a7120 msgr2=0x7f5e8c0a2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:56.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:56.220+0000 7f5e90945640 1 -- 172.21.15.12:0/2506153362 shutdown_connections 2024-09-11T01:42:56.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:56.220+0000 7f5e90945640 1 -- 172.21.15.12:0/2506153362 wait complete. 2024-09-11T01:42:56.301 INFO:tasks.rados.rados.0.smithi071.stdout:2029: finishing write tid 1 to smithi07134777-31 2024-09-11T01:42:56.302 INFO:tasks.rados.rados.0.smithi071.stdout:2029: finishing write tid 2 to smithi07134777-31 2024-09-11T01:42:56.302 INFO:tasks.rados.rados.0.smithi071.stdout:2029: finishing write tid 3 to smithi07134777-31 2024-09-11T01:42:56.302 INFO:tasks.rados.rados.0.smithi071.stdout:2029: finishing write tid 4 to smithi07134777-31 2024-09-11T01:42:56.302 INFO:tasks.rados.rados.0.smithi071.stdout:2029: oid 31 updating version 0 to 844 2024-09-11T01:42:56.302 INFO:tasks.rados.rados.0.smithi071.stdout:2029: oid 31 updating version 844 to 845 2024-09-11T01:42:56.302 INFO:tasks.rados.rados.0.smithi071.stdout:2029: oid 31 version 845 is already newer than 843 2024-09-11T01:42:56.302 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 845 (ObjNum 737 snap 226 seq_num 737) dirty exists 2024-09-11T01:42:56.302 INFO:tasks.rados.rados.0.smithi071.stdout:2029: left oid 31 (ObjNum 737 snap 226 seq_num 737) 2024-09-11T01:42:56.302 INFO:tasks.rados.rados.0.smithi071.stdout:2021: done (11 left) 2024-09-11T01:42:56.302 INFO:tasks.rados.rados.0.smithi071.stdout:2022: done (10 left) 2024-09-11T01:42:56.302 INFO:tasks.rados.rados.0.smithi071.stdout:2023: done (9 left) 2024-09-11T01:42:56.302 INFO:tasks.rados.rados.0.smithi071.stdout:2024: done (8 left) 2024-09-11T01:42:56.303 INFO:tasks.rados.rados.0.smithi071.stdout:2025: done (7 left) 2024-09-11T01:42:56.303 INFO:tasks.rados.rados.0.smithi071.stdout:2026: done (6 left) 2024-09-11T01:42:56.303 INFO:tasks.rados.rados.0.smithi071.stdout:2027: done (5 left) 2024-09-11T01:42:56.303 INFO:tasks.rados.rados.0.smithi071.stdout:2028: done (4 left) 2024-09-11T01:42:56.303 INFO:tasks.rados.rados.0.smithi071.stdout:2029: done (3 left) 2024-09-11T01:42:56.303 INFO:tasks.rados.rados.0.smithi071.stdout:2033: done (2 left) 2024-09-11T01:42:56.303 INFO:tasks.rados.rados.0.smithi071.stdout:2034: write (excl) oid 34 current snap is 226 2024-09-11T01:42:56.303 INFO:tasks.rados.rados.0.smithi071.stdout:2031: finishing write tid 1 to smithi07134777-21 2024-09-11T01:42:56.303 INFO:tasks.rados.rados.0.smithi071.stdout:2031: finishing write tid 2 to smithi07134777-21 2024-09-11T01:42:56.303 INFO:tasks.rados.rados.0.smithi071.stdout:2031: finishing write tid 3 to smithi07134777-21 2024-09-11T01:42:56.303 INFO:tasks.rados.rados.0.smithi071.stdout:2031: finishing write tid 4 to smithi07134777-21 2024-09-11T01:42:56.303 INFO:tasks.rados.rados.0.smithi071.stdout:2031: oid 21 updating version 0 to 949 2024-09-11T01:42:56.303 INFO:tasks.rados.rados.0.smithi071.stdout:2031: oid 21 version 949 is already newer than 947 2024-09-11T01:42:56.303 INFO:tasks.rados.rados.0.smithi071.stdout:2031: oid 21 version 949 is already newer than 948 2024-09-11T01:42:56.303 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 949 (ObjNum 738 snap 226 seq_num 738) dirty exists 2024-09-11T01:42:56.304 INFO:tasks.rados.rados.0.smithi071.stdout:2031: left oid 21 (ObjNum 738 snap 226 seq_num 738) 2024-09-11T01:42:56.304 INFO:tasks.rados.rados.0.smithi071.stdout:2032: finishing write tid 1 to smithi07134777-45 2024-09-11T01:42:56.304 INFO:tasks.rados.rados.0.smithi071.stdout:2032: finishing write tid 2 to smithi07134777-45 2024-09-11T01:42:56.304 INFO:tasks.rados.rados.0.smithi071.stdout:2034: seq_num 741 ranges {453987=645750,1845157=608830} 2024-09-11T01:42:56.307 INFO:tasks.rados.rados.0.smithi071.stdout:2034: writing smithi07134777-34 from 453987 to 1099737 tid 1 2024-09-11T01:42:56.315 INFO:tasks.rados.rados.0.smithi071.stdout:2034: writing smithi07134777-34 from 1845157 to 2453987 tid 2 2024-09-11T01:42:56.315 INFO:tasks.rados.rados.0.smithi071.stdout:2031: done (2 left) 2024-09-11T01:42:56.315 INFO:tasks.rados.rados.0.smithi071.stdout:2035: rollback oid 27 current snap is 226 2024-09-11T01:42:56.315 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 27 to 212 2024-09-11T01:42:56.315 INFO:tasks.rados.rados.0.smithi071.stdout:2032: finishing write tid 3 to smithi07134777-45 2024-09-11T01:42:56.315 INFO:tasks.rados.rados.0.smithi071.stdout:2032: finishing write tid 4 to smithi07134777-45 2024-09-11T01:42:56.315 INFO:tasks.rados.rados.0.smithi071.stdout:2032: oid 45 updating version 0 to 952 2024-09-11T01:42:56.315 INFO:tasks.rados.rados.0.smithi071.stdout:2032: oid 45 version 952 is already newer than 951 2024-09-11T01:42:56.315 INFO:tasks.rados.rados.0.smithi071.stdout:2032: oid 45 version 952 is already newer than 950 2024-09-11T01:42:56.315 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 952 (ObjNum 739 snap 226 seq_num 739) dirty exists 2024-09-11T01:42:56.315 INFO:tasks.rados.rados.0.smithi071.stdout:2032: left oid 45 (ObjNum 739 snap 226 seq_num 739) 2024-09-11T01:42:56.315 INFO:tasks.rados.rados.0.smithi071.stdout:2034: finishing write tid 1 to smithi07134777-34 2024-09-11T01:42:56.315 INFO:tasks.rados.rados.0.smithi071.stdout:2032: done (2 left) 2024-09-11T01:42:56.316 INFO:tasks.rados.rados.0.smithi071.stdout:2036: copy_from oid 14 from oid 21 current snap is 226 2024-09-11T01:42:56.316 INFO:tasks.rados.rados.0.smithi071.stdout:2037: copy_from oid 1 from oid 3 current snap is 226 2024-09-11T01:42:56.316 INFO:tasks.rados.rados.0.smithi071.stdout:2038: read oid 4 snap -1 2024-09-11T01:42:56.316 INFO:tasks.rados.rados.0.smithi071.stdout:2038: expect (ObjNum 644 snap 196 seq_num 644) 2024-09-11T01:42:56.316 INFO:tasks.rados.rados.0.smithi071.stdout:2039: write oid 44 current snap is 226 2024-09-11T01:42:56.316 INFO:tasks.rados.rados.0.smithi071.stdout:2039: seq_num 742 ranges {470794=720999,1903639=554256} 2024-09-11T01:42:56.321 INFO:tasks.rados.rados.0.smithi071.stdout:2039: writing smithi07134777-44 from 470794 to 1191793 tid 1 2024-09-11T01:42:56.329 INFO:tasks.rados.rados.0.smithi071.stdout:2039: writing smithi07134777-44 from 1903639 to 2457895 tid 2 2024-09-11T01:42:56.329 INFO:tasks.rados.rados.0.smithi071.stdout:2040: copy_from oid 13 from oid 5 current snap is 226 2024-09-11T01:42:56.329 INFO:tasks.rados.rados.0.smithi071.stdout:2034: finishing write tid 2 to smithi07134777-34 2024-09-11T01:42:56.329 INFO:tasks.rados.rados.0.smithi071.stdout:2034: finishing write tid 3 to smithi07134777-34 2024-09-11T01:42:56.329 INFO:tasks.rados.rados.0.smithi071.stdout:2034: finishing write tid 4 to smithi07134777-34 2024-09-11T01:42:56.329 INFO:tasks.rados.rados.0.smithi071.stdout:2034: oid 34 updating version 0 to 1161 2024-09-11T01:42:56.329 INFO:tasks.rados.rados.0.smithi071.stdout:2034: oid 34 updating version 1161 to 1162 2024-09-11T01:42:56.329 INFO:tasks.rados.rados.0.smithi071.stdout:2034: oid 34 version 1162 is already newer than 1160 2024-09-11T01:42:56.329 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 1162 (ObjNum 741 snap 226 seq_num 741) dirty exists 2024-09-11T01:42:56.330 INFO:tasks.rados.rados.0.smithi071.stdout:2034: left oid 34 (ObjNum 741 snap 226 seq_num 741) 2024-09-11T01:42:56.330 INFO:tasks.rados.rados.0.smithi071.stdout:2037: finishing copy_from racing read to smithi07134777-1 2024-09-11T01:42:56.330 INFO:tasks.rados.rados.0.smithi071.stdout:2035: finishing rollback tid 1 to smithi07134777-27 2024-09-11T01:42:56.330 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 1003 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-11T01:42:56.330 INFO:tasks.rados.rados.0.smithi071.stdout:2038: expect (ObjNum 644 snap 196 seq_num 644) 2024-09-11T01:42:56.413 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:56.454 INFO:tasks.rados.rados.0.smithi071.stdout:2037: finishing copy_from to smithi07134777-1 2024-09-11T01:42:56.454 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 703 (ObjNum 732 snap 225 seq_num 732) dirty exists 2024-09-11T01:42:56.454 INFO:tasks.rados.rados.0.smithi071.stdout:2036: finishing copy_from racing read to smithi07134777-14 2024-09-11T01:42:56.455 INFO:tasks.rados.rados.0.smithi071.stdout:2040: finishing copy_from to smithi07134777-13 2024-09-11T01:42:56.455 INFO:tasks.rados.rados.0.smithi071.stdout:2040: got expected ENOENT (src dne) 2024-09-11T01:42:56.455 INFO:tasks.rados.rados.0.smithi071.stdout:2040: finishing copy_from racing read to smithi07134777-13 2024-09-11T01:42:56.455 INFO:tasks.rados.rados.0.smithi071.stdout:2039: finishing write tid 1 to smithi07134777-44 2024-09-11T01:42:56.455 INFO:tasks.rados.rados.0.smithi071.stdout:2039: finishing write tid 2 to smithi07134777-44 2024-09-11T01:42:56.455 INFO:tasks.rados.rados.0.smithi071.stdout:2039: finishing write tid 3 to smithi07134777-44 2024-09-11T01:42:56.455 INFO:tasks.rados.rados.0.smithi071.stdout:2036: finishing copy_from to smithi07134777-14 2024-09-11T01:42:56.455 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 949 (ObjNum 738 snap 226 seq_num 738) dirty exists 2024-09-11T01:42:56.455 INFO:tasks.rados.rados.0.smithi071.stdout:2039: finishing write tid 4 to smithi07134777-44 2024-09-11T01:42:56.455 INFO:tasks.rados.rados.0.smithi071.stdout:2039: oid 44 updating version 0 to 975 2024-09-11T01:42:56.455 INFO:tasks.rados.rados.0.smithi071.stdout:2039: oid 44 version 975 is already newer than 973 2024-09-11T01:42:56.455 INFO:tasks.rados.rados.0.smithi071.stdout:2039: oid 44 version 975 is already newer than 974 2024-09-11T01:42:56.455 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 975 (ObjNum 742 snap 226 seq_num 742) dirty exists 2024-09-11T01:42:56.455 INFO:tasks.rados.rados.0.smithi071.stdout:2039: left oid 44 (ObjNum 742 snap 226 seq_num 742) 2024-09-11T01:42:56.456 INFO:tasks.rados.rados.0.smithi071.stdout:2034: done (6 left) 2024-09-11T01:42:56.456 INFO:tasks.rados.rados.0.smithi071.stdout:2035: done (5 left) 2024-09-11T01:42:56.456 INFO:tasks.rados.rados.0.smithi071.stdout:2036: done (4 left) 2024-09-11T01:42:56.456 INFO:tasks.rados.rados.0.smithi071.stdout:2037: done (3 left) 2024-09-11T01:42:56.456 INFO:tasks.rados.rados.0.smithi071.stdout:2038: done (2 left) 2024-09-11T01:42:56.456 INFO:tasks.rados.rados.0.smithi071.stdout:2039: done (1 left) 2024-09-11T01:42:56.456 INFO:tasks.rados.rados.0.smithi071.stdout:2040: done (0 left) 2024-09-11T01:42:56.456 INFO:tasks.rados.rados.0.smithi071.stdout:2041: write oid 42 current snap is 226 2024-09-11T01:42:56.456 INFO:tasks.rados.rados.0.smithi071.stdout:2041: seq_num 743 ranges {487601=796248,1962121=525480} 2024-09-11T01:42:56.461 INFO:tasks.rados.rados.0.smithi071.stdout:2041: writing smithi07134777-42 from 487601 to 1283849 tid 1 2024-09-11T01:42:56.468 INFO:tasks.rados.rados.0.smithi071.stdout:2041: writing smithi07134777-42 from 1962121 to 2487601 tid 2 2024-09-11T01:42:56.468 INFO:tasks.rados.rados.0.smithi071.stdout:2042: read oid 2 snap -1 2024-09-11T01:42:56.468 INFO:tasks.rados.rados.0.smithi071.stdout:2042: expect (ObjNum 686 snap 204 seq_num 686) 2024-09-11T01:42:56.468 INFO:tasks.rados.rados.0.smithi071.stdout:2043: snap_create 2024-09-11T01:42:56.468 INFO:tasks.rados.rados.0.smithi071.stdout:2041: finishing write tid 1 to smithi07134777-42 2024-09-11T01:42:56.472 INFO:tasks.rados.rados.0.smithi071.stdout:2041: finishing write tid 2 to smithi07134777-42 2024-09-11T01:42:56.472 INFO:tasks.rados.rados.0.smithi071.stdout:2042: expect (ObjNum 686 snap 204 seq_num 686) 2024-09-11T01:42:56.472 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:56.634 INFO:tasks.rados.rados.0.smithi071.stdout:2041: finishing write tid 3 to smithi07134777-42 2024-09-11T01:42:56.635 INFO:tasks.rados.rados.0.smithi071.stdout:2041: finishing write tid 4 to smithi07134777-42 2024-09-11T01:42:56.635 INFO:tasks.rados.rados.0.smithi071.stdout:2041: oid 42 updating version 0 to 835 2024-09-11T01:42:56.635 INFO:tasks.rados.rados.0.smithi071.stdout:2041: oid 42 version 835 is already newer than 834 2024-09-11T01:42:56.635 INFO:tasks.rados.rados.0.smithi071.stdout:2041: oid 42 updating version 835 to 836 2024-09-11T01:42:56.635 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 836 (ObjNum 743 snap 226 seq_num 743) dirty exists 2024-09-11T01:42:56.635 INFO:tasks.rados.rados.0.smithi071.stdout:2041: left oid 42 (ObjNum 743 snap 226 seq_num 743) 2024-09-11T01:42:56.979 INFO:tasks.rados.rados.0.smithi071.stdout:2041: done (2 left) 2024-09-11T01:42:56.979 INFO:tasks.rados.rados.0.smithi071.stdout:2042: done (1 left) 2024-09-11T01:42:56.979 INFO:tasks.rados.rados.0.smithi071.stdout:2043: done (0 left) 2024-09-11T01:42:56.979 INFO:tasks.rados.rados.0.smithi071.stdout:2044: write (excl) oid 24 current snap is 227 2024-09-11T01:42:56.979 INFO:tasks.rados.rados.0.smithi071.stdout:2044: seq_num 744 ranges {504408=471497,1620603=557925} 2024-09-11T01:42:56.984 INFO:tasks.rados.rados.0.smithi071.stdout:2044: writing smithi07134777-24 from 504408 to 975905 tid 1 2024-09-11T01:42:56.991 INFO:tasks.rados.rados.0.smithi071.stdout:2044: writing smithi07134777-24 from 1620603 to 2178528 tid 2 2024-09-11T01:42:56.991 INFO:tasks.rados.rados.0.smithi071.stdout:2045: write oid 32 current snap is 227 2024-09-11T01:42:56.991 INFO:tasks.rados.rados.0.smithi071.stdout:2045: seq_num 745 ranges {521215=546746,1562732=701583} 2024-09-11T01:42:56.996 INFO:tasks.rados.rados.0.smithi071.stdout:2045: writing smithi07134777-32 from 521215 to 1067961 tid 1 2024-09-11T01:42:57.004 INFO:tasks.rados.rados.0.smithi071.stdout:2045: writing smithi07134777-32 from 1562732 to 2264315 tid 2 2024-09-11T01:42:57.004 INFO:tasks.rados.rados.0.smithi071.stdout:2046: rollback oid 13 current snap is 227 2024-09-11T01:42:57.004 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 13 to 185 2024-09-11T01:42:57.004 INFO:tasks.rados.rados.0.smithi071.stdout:2044: finishing write tid 1 to smithi07134777-24 2024-09-11T01:42:57.004 INFO:tasks.rados.rados.0.smithi071.stdout:2044: finishing write tid 2 to smithi07134777-24 2024-09-11T01:42:57.005 INFO:tasks.rados.rados.0.smithi071.stdout:2044: finishing write tid 3 to smithi07134777-24 2024-09-11T01:42:57.005 INFO:tasks.rados.rados.0.smithi071.stdout:2044: finishing write tid 4 to smithi07134777-24 2024-09-11T01:42:57.005 INFO:tasks.rados.rados.0.smithi071.stdout:2044: oid 24 updating version 0 to 955 2024-09-11T01:42:57.005 INFO:tasks.rados.rados.0.smithi071.stdout:2044: oid 24 version 955 is already newer than 954 2024-09-11T01:42:57.005 INFO:tasks.rados.rados.0.smithi071.stdout:2044: oid 24 version 955 is already newer than 953 2024-09-11T01:42:57.005 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 955 (ObjNum 744 snap 227 seq_num 744) dirty exists 2024-09-11T01:42:57.005 INFO:tasks.rados.rados.0.smithi071.stdout:2044: left oid 24 (ObjNum 744 snap 227 seq_num 744) 2024-09-11T01:42:57.005 INFO:tasks.rados.rados.0.smithi071.stdout:2044: done (2 left) 2024-09-11T01:42:57.005 INFO:tasks.rados.rados.0.smithi071.stdout:2047: rollback oid 36 current snap is 227 2024-09-11T01:42:57.005 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 36 to 226 2024-09-11T01:42:57.005 INFO:tasks.rados.rados.0.smithi071.stdout:2048: read oid 1 snap -1 2024-09-11T01:42:57.005 INFO:tasks.rados.rados.0.smithi071.stdout:2048: expect (ObjNum 732 snap 225 seq_num 732) 2024-09-11T01:42:57.005 INFO:tasks.rados.rados.0.smithi071.stdout:2049: read oid 25 snap 209 2024-09-11T01:42:57.005 INFO:tasks.rados.rados.0.smithi071.stdout:2049: expect (ObjNum 651 snap 199 seq_num 651) 2024-09-11T01:42:57.006 INFO:tasks.rados.rados.0.smithi071.stdout:2050: copy_from oid 18 from oid 28 current snap is 227 2024-09-11T01:42:57.006 INFO:tasks.rados.rados.0.smithi071.stdout:2051: rollback oid 16 current snap is 227 2024-09-11T01:42:57.006 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 16 to 185 2024-09-11T01:42:57.006 INFO:tasks.rados.rados.0.smithi071.stdout:2052: read oid 7 snap -1 2024-09-11T01:42:57.006 INFO:tasks.rados.rados.0.smithi071.stdout:2052: expect (ObjNum 581 snap 166 seq_num 581) 2024-09-11T01:42:57.006 INFO:tasks.rados.rados.0.smithi071.stdout:2053: read oid 21 snap 218 2024-09-11T01:42:57.006 INFO:tasks.rados.rados.0.smithi071.stdout:2053: expect (ObjNum 679 snap 204 seq_num 679) 2024-09-11T01:42:57.006 INFO:tasks.rados.rados.0.smithi071.stdout:2054: write (excl) oid 2 current snap is 227 2024-09-11T01:42:57.006 INFO:tasks.rados.rados.0.smithi071.stdout:2045: finishing write tid 1 to smithi07134777-32 2024-09-11T01:42:57.006 INFO:tasks.rados.rados.0.smithi071.stdout:2054: seq_num 746 ranges {538022=738348,1737567=561594} 2024-09-11T01:42:57.011 INFO:tasks.rados.rados.0.smithi071.stdout:2054: writing smithi07134777-2 from 538022 to 1276370 tid 1 2024-09-11T01:42:57.018 INFO:tasks.rados.rados.0.smithi071.stdout:2054: writing smithi07134777-2 from 1737567 to 2299161 tid 2 2024-09-11T01:42:57.018 INFO:tasks.rados.rados.0.smithi071.stdout:2055: rollback oid 22 current snap is 227 2024-09-11T01:42:57.018 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 22 to 213 2024-09-11T01:42:57.018 INFO:tasks.rados.rados.0.smithi071.stdout:2046: finishing rollback tid 0 to smithi07134777-13 2024-09-11T01:42:57.018 INFO:tasks.rados.rados.0.smithi071.stdout:2045: finishing write tid 2 to smithi07134777-32 2024-09-11T01:42:57.018 INFO:tasks.rados.rados.0.smithi071.stdout:2047: finishing rollback tid 1 to smithi07134777-36 2024-09-11T01:42:57.018 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 1008 (ObjNum 33 snap 0 seq_num 2371357536) dirty dne 2024-09-11T01:42:57.018 INFO:tasks.rados.rados.0.smithi071.stdout:2045: finishing write tid 3 to smithi07134777-32 2024-09-11T01:42:57.018 INFO:tasks.rados.rados.0.smithi071.stdout:2052: expect (ObjNum 581 snap 166 seq_num 581) 2024-09-11T01:42:57.061 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:57.102 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:57.142 INFO:tasks.rados.rados.0.smithi071.stdout:2046: finishing rollback tid 1 to smithi07134777-13 2024-09-11T01:42:57.142 INFO:tasks.rados.rados.0.smithi071.stdout:2046: finishing rollback tid 2 to smithi07134777-13 2024-09-11T01:42:57.142 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 848 (ObjNum 551 snap 157 seq_num 551) dirty exists 2024-09-11T01:42:57.142 INFO:tasks.rados.rados.0.smithi071.stdout:2048: expect (ObjNum 732 snap 225 seq_num 732) 2024-09-11T01:42:57.149 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 760088 2024-09-11T01:42:57.220 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1140054 2024-09-11T01:42:57.241 INFO:tasks.rados.rados.0.smithi071.stdout:2045: finishing write tid 4 to smithi07134777-32 2024-09-11T01:42:57.242 INFO:tasks.rados.rados.0.smithi071.stdout:2045: oid 32 updating version 0 to 984 2024-09-11T01:42:57.242 INFO:tasks.rados.rados.0.smithi071.stdout:2045: oid 32 updating version 984 to 985 2024-09-11T01:42:57.242 INFO:tasks.rados.rados.0.smithi071.stdout:2045: oid 32 version 985 is already newer than 983 2024-09-11T01:42:57.242 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 985 (ObjNum 745 snap 227 seq_num 745) dirty exists 2024-09-11T01:42:57.242 INFO:tasks.rados.rados.0.smithi071.stdout:2045: left oid 32 (ObjNum 745 snap 227 seq_num 745) 2024-09-11T01:42:57.242 INFO:tasks.rados.rados.0.smithi071.stdout:2049: expect (ObjNum 651 snap 199 seq_num 651) 2024-09-11T01:42:57.294 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:57.402 INFO:tasks.rados.rados.0.smithi071.stdout:2054: finishing write tid 1 to smithi07134777-2 2024-09-11T01:42:57.402 INFO:tasks.rados.rados.0.smithi071.stdout:2051: finishing rollback tid 1 to smithi07134777-16 2024-09-11T01:42:57.402 INFO:tasks.rados.rados.0.smithi071.stdout:2051: finishing rollback tid 2 to smithi07134777-16 2024-09-11T01:42:57.402 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 957 (ObjNum 411 snap 115 seq_num 411) dirty exists 2024-09-11T01:42:57.402 INFO:tasks.rados.rados.0.smithi071.stdout:2054: finishing write tid 2 to smithi07134777-2 2024-09-11T01:42:57.402 INFO:tasks.rados.rados.0.smithi071.stdout:2054: finishing write tid 3 to smithi07134777-2 2024-09-11T01:42:57.403 INFO:tasks.rados.rados.0.smithi071.stdout:2054: finishing write tid 4 to smithi07134777-2 2024-09-11T01:42:57.403 INFO:tasks.rados.rados.0.smithi071.stdout:2054: oid 2 updating version 0 to 839 2024-09-11T01:42:57.403 INFO:tasks.rados.rados.0.smithi071.stdout:2054: oid 2 version 839 is already newer than 837 2024-09-11T01:42:57.403 INFO:tasks.rados.rados.0.smithi071.stdout:2054: oid 2 version 839 is already newer than 838 2024-09-11T01:42:57.403 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 839 (ObjNum 746 snap 227 seq_num 746) dirty exists 2024-09-11T01:42:57.403 INFO:tasks.rados.rados.0.smithi071.stdout:2054: left oid 2 (ObjNum 746 snap 227 seq_num 746) 2024-09-11T01:42:57.403 INFO:tasks.rados.rados.0.smithi071.stdout:2053: expect (ObjNum 679 snap 204 seq_num 679) 2024-09-11T01:42:57.496 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:57.680 INFO:tasks.rados.rados.0.smithi071.stdout:2055: finishing rollback tid 0 to smithi07134777-22 2024-09-11T01:42:57.680 INFO:tasks.rados.rados.0.smithi071.stdout:2050: finishing copy_from racing read to smithi07134777-18 2024-09-11T01:42:57.680 INFO:tasks.rados.rados.0.smithi071.stdout:2055: finishing rollback tid 1 to smithi07134777-22 2024-09-11T01:42:57.681 INFO:tasks.rados.rados.0.smithi071.stdout:2050: finishing copy_from to smithi07134777-18 2024-09-11T01:42:57.681 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 943 (ObjNum 724 snap 224 seq_num 724) dirty exists 2024-09-11T01:42:57.681 INFO:tasks.rados.rados.0.smithi071.stdout:2055: finishing rollback tid 2 to smithi07134777-22 2024-09-11T01:42:57.681 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 706 (ObjNum 658 snap 201 seq_num 658) dirty exists 2024-09-11T01:42:57.681 INFO:tasks.rados.rados.0.smithi071.stdout:2045: done (10 left) 2024-09-11T01:42:57.681 INFO:tasks.rados.rados.0.smithi071.stdout:2046: done (9 left) 2024-09-11T01:42:57.681 INFO:tasks.rados.rados.0.smithi071.stdout:2047: done (8 left) 2024-09-11T01:42:57.681 INFO:tasks.rados.rados.0.smithi071.stdout:2048: done (7 left) 2024-09-11T01:42:57.681 INFO:tasks.rados.rados.0.smithi071.stdout:2049: done (6 left) 2024-09-11T01:42:57.681 INFO:tasks.rados.rados.0.smithi071.stdout:2050: done (5 left) 2024-09-11T01:42:57.681 INFO:tasks.rados.rados.0.smithi071.stdout:2051: done (4 left) 2024-09-11T01:42:57.681 INFO:tasks.rados.rados.0.smithi071.stdout:2052: done (3 left) 2024-09-11T01:42:57.681 INFO:tasks.rados.rados.0.smithi071.stdout:2053: done (2 left) 2024-09-11T01:42:57.681 INFO:tasks.rados.rados.0.smithi071.stdout:2054: done (1 left) 2024-09-11T01:42:57.682 INFO:tasks.rados.rados.0.smithi071.stdout:2055: done (0 left) 2024-09-11T01:42:57.682 INFO:tasks.rados.rados.0.smithi071.stdout:2056: snap_create 2024-09-11T01:42:57.993 INFO:tasks.rados.rados.0.smithi071.stdout:2056: done (0 left) 2024-09-11T01:42:57.993 INFO:tasks.rados.rados.0.smithi071.stdout:2057: read oid 26 snap 220 2024-09-11T01:42:57.993 INFO:tasks.rados.rados.0.smithi071.stdout:2057: expect (ObjNum 577 snap 165 seq_num 577) 2024-09-11T01:42:57.994 INFO:tasks.rados.rados.0.smithi071.stdout:2058: read oid 43 snap -1 2024-09-11T01:42:57.994 INFO:tasks.rados.rados.0.smithi071.stdout:2058: expect (ObjNum 542 snap 154 seq_num 542) 2024-09-11T01:42:57.994 INFO:tasks.rados.rados.0.smithi071.stdout:2059: delete oid 30 current snap is 228 2024-09-11T01:42:57.998 INFO:tasks.rados.rados.0.smithi071.stdout:2059: done (2 left) 2024-09-11T01:42:57.998 INFO:tasks.rados.rados.0.smithi071.stdout:2060: read oid 36 snap -1 2024-09-11T01:42:57.998 INFO:tasks.rados.rados.0.smithi071.stdout:2060: expect deleted 2024-09-11T01:42:57.998 INFO:tasks.rados.rados.0.smithi071.stdout:2061: delete oid 19 current snap is 228 2024-09-11T01:42:58.008 INFO:tasks.rados.rados.0.smithi071.stdout:2057: expect (ObjNum 577 snap 165 seq_num 577) 2024-09-11T01:42:58.079 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:58.119 INFO:tasks.rados.rados.0.smithi071.stdout:2058: expect (ObjNum 542 snap 154 seq_num 542) 2024-09-11T01:42:58.229 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:58.286 INFO:tasks.rados.rados.0.smithi071.stdout:2057: done (3 left) 2024-09-11T01:42:58.286 INFO:tasks.rados.rados.0.smithi071.stdout:2058: done (2 left) 2024-09-11T01:42:58.286 INFO:tasks.rados.rados.0.smithi071.stdout:2060: done (1 left) 2024-09-11T01:42:58.286 INFO:tasks.rados.rados.0.smithi071.stdout:2061: done (0 left) 2024-09-11T01:42:58.286 INFO:tasks.rados.rados.0.smithi071.stdout:2062: read oid 28 snap -1 2024-09-11T01:42:58.286 INFO:tasks.rados.rados.0.smithi071.stdout:2062: expect (ObjNum 724 snap 224 seq_num 724) 2024-09-11T01:42:58.286 INFO:tasks.rados.rados.0.smithi071.stdout:2063: rollback oid 1 current snap is 228 2024-09-11T01:42:58.286 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 1 to 206 2024-09-11T01:42:58.286 INFO:tasks.rados.rados.0.smithi071.stdout:2064: copy_from oid 38 from oid 19 current snap is 228 2024-09-11T01:42:58.286 INFO:tasks.rados.rados.0.smithi071.stdout:2065: copy_from oid 36 from oid 31 current snap is 228 2024-09-11T01:42:58.286 INFO:tasks.rados.rados.0.smithi071.stdout:2066: delete oid 34 current snap is 228 2024-09-11T01:42:58.287 INFO:tasks.rados.rados.0.smithi071.stdout:2064: finishing copy_from to smithi07134777-38 2024-09-11T01:42:58.287 INFO:tasks.rados.rados.0.smithi071.stdout:2064: got expected ENOENT (src dne) 2024-09-11T01:42:58.287 INFO:tasks.rados.rados.0.smithi071.stdout:2064: finishing copy_from racing read to smithi07134777-38 2024-09-11T01:42:58.292 INFO:tasks.rados.rados.0.smithi071.stdout:2062: expect (ObjNum 724 snap 224 seq_num 724) 2024-09-11T01:42:58.389 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:58.438 INFO:tasks.rados.rados.0.smithi071.stdout:2063: finishing rollback tid 0 to smithi07134777-1 2024-09-11T01:42:58.438 INFO:tasks.rados.rados.0.smithi071.stdout:2063: finishing rollback tid 1 to smithi07134777-1 2024-09-11T01:42:58.438 INFO:tasks.rados.rados.0.smithi071.stdout:2063: finishing rollback tid 2 to smithi07134777-1 2024-09-11T01:42:58.438 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 851 (ObjNum 688 snap 204 seq_num 688) dirty exists 2024-09-11T01:42:58.438 INFO:tasks.rados.rados.0.smithi071.stdout:2065: finishing copy_from racing read to smithi07134777-36 2024-09-11T01:42:58.438 INFO:tasks.rados.rados.0.smithi071.stdout:2065: finishing copy_from to smithi07134777-36 2024-09-11T01:42:58.439 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 845 (ObjNum 737 snap 226 seq_num 737) dirty exists 2024-09-11T01:42:58.439 INFO:tasks.rados.rados.0.smithi071.stdout:2062: done (4 left) 2024-09-11T01:42:58.439 INFO:tasks.rados.rados.0.smithi071.stdout:2063: done (3 left) 2024-09-11T01:42:58.439 INFO:tasks.rados.rados.0.smithi071.stdout:2064: done (2 left) 2024-09-11T01:42:58.439 INFO:tasks.rados.rados.0.smithi071.stdout:2065: done (1 left) 2024-09-11T01:42:58.439 INFO:tasks.rados.rados.0.smithi071.stdout:2066: done (0 left) 2024-09-11T01:42:58.439 INFO:tasks.rados.rados.0.smithi071.stdout:2067: read oid 28 snap -1 2024-09-11T01:42:58.439 INFO:tasks.rados.rados.0.smithi071.stdout:2067: expect (ObjNum 724 snap 224 seq_num 724) 2024-09-11T01:42:58.439 INFO:tasks.rados.rados.0.smithi071.stdout:2068: copy_from oid 25 from oid 37 current snap is 228 2024-09-11T01:42:58.439 INFO:tasks.rados.rados.0.smithi071.stdout:2069: write oid 13 current snap is 228 2024-09-11T01:42:58.439 INFO:tasks.rados.rados.0.smithi071.stdout:2069: seq_num 750 ranges {605250=639344,1855142=452579} 2024-09-11T01:42:58.444 INFO:tasks.rados.rados.0.smithi071.stdout:2069: writing smithi07134777-13 from 605250 to 1244594 tid 1 2024-09-11T01:42:58.450 INFO:tasks.rados.rados.0.smithi071.stdout:2069: writing smithi07134777-13 from 1855142 to 2307721 tid 2 2024-09-11T01:42:58.450 INFO:tasks.rados.rados.0.smithi071.stdout:2070: write oid 24 current snap is 228 2024-09-11T01:42:58.450 INFO:tasks.rados.rados.0.smithi071.stdout:2070: seq_num 751 ranges {622057=714593,1913624=708433} 2024-09-11T01:42:58.457 INFO:tasks.rados.rados.0.smithi071.stdout:2070: writing smithi07134777-24 from 622057 to 1336650 tid 1 2024-09-11T01:42:58.465 INFO:tasks.rados.rados.0.smithi071.stdout:2070: writing smithi07134777-24 from 1913624 to 2622057 tid 2 2024-09-11T01:42:58.465 INFO:tasks.rados.rados.0.smithi071.stdout:2071: read oid 48 snap -1 2024-09-11T01:42:58.465 INFO:tasks.rados.rados.0.smithi071.stdout:2071: expect (ObjNum 565 snap 160 seq_num 565) 2024-09-11T01:42:58.465 INFO:tasks.rados.rados.0.smithi071.stdout:2072: snap_remove snap 196 2024-09-11T01:42:58.971 INFO:tasks.rados.rados.0.smithi071.stdout:2072: done (5 left) 2024-09-11T01:42:58.971 INFO:tasks.rados.rados.0.smithi071.stdout:2073: write oid 39 current snap is 228 2024-09-11T01:42:58.971 INFO:tasks.rados.rados.0.smithi071.stdout:2073: seq_num 752 ranges {638864=789842,1972106=456248} 2024-09-11T01:42:58.980 INFO:tasks.rados.rados.0.smithi071.stdout:2073: writing smithi07134777-39 from 638864 to 1428706 tid 1 2024-09-11T01:42:58.986 INFO:tasks.rados.rados.0.smithi071.stdout:2073: writing smithi07134777-39 from 1972106 to 2428354 tid 2 2024-09-11T01:42:58.986 INFO:tasks.rados.rados.0.smithi071.stdout:2074: rollback oid 9 current snap is 228 2024-09-11T01:42:58.986 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 9 to 208 2024-09-11T01:42:58.986 INFO:tasks.rados.rados.0.smithi071.stdout:2067: expect (ObjNum 724 snap 224 seq_num 724) 2024-09-11T01:42:59.082 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:42:59.131 INFO:tasks.rados.rados.0.smithi071.stdout:2068: finishing copy_from racing read to smithi07134777-25 2024-09-11T01:42:59.131 INFO:tasks.rados.rados.0.smithi071.stdout:2068: finishing copy_from to smithi07134777-25 2024-09-11T01:42:59.131 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 979 (ObjNum 723 snap 224 seq_num 723) dirty exists 2024-09-11T01:42:59.131 INFO:tasks.rados.rados.0.smithi071.stdout:2067: done (6 left) 2024-09-11T01:42:59.131 INFO:tasks.rados.rados.0.smithi071.stdout:2068: done (5 left) 2024-09-11T01:42:59.131 INFO:tasks.rados.rados.0.smithi071.stdout:2075: snap_create 2024-09-11T01:42:59.131 INFO:tasks.rados.rados.0.smithi071.stdout:2069: finishing write tid 1 to smithi07134777-13 2024-09-11T01:42:59.131 INFO:tasks.rados.rados.0.smithi071.stdout:2069: finishing write tid 2 to smithi07134777-13 2024-09-11T01:42:59.132 INFO:tasks.rados.rados.0.smithi071.stdout:2069: finishing write tid 3 to smithi07134777-13 2024-09-11T01:42:59.132 INFO:tasks.rados.rados.0.smithi071.stdout:2069: finishing write tid 4 to smithi07134777-13 2024-09-11T01:42:59.132 INFO:tasks.rados.rados.0.smithi071.stdout:2069: oid 13 updating version 0 to 854 2024-09-11T01:42:59.132 INFO:tasks.rados.rados.0.smithi071.stdout:2069: oid 13 version 854 is already newer than 852 2024-09-11T01:42:59.132 INFO:tasks.rados.rados.0.smithi071.stdout:2069: oid 13 version 854 is already newer than 853 2024-09-11T01:42:59.132 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 854 (ObjNum 750 snap 228 seq_num 750) dirty exists 2024-09-11T01:42:59.132 INFO:tasks.rados.rados.0.smithi071.stdout:2069: left oid 13 (ObjNum 750 snap 228 seq_num 750) 2024-09-11T01:42:59.132 INFO:tasks.rados.rados.0.smithi071.stdout:2070: finishing write tid 1 to smithi07134777-24 2024-09-11T01:42:59.132 INFO:tasks.rados.rados.0.smithi071.stdout:2071: expect (ObjNum 565 snap 160 seq_num 565) 2024-09-11T01:42:59.233 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:42:59.341 INFO:tasks.rados.rados.0.smithi071.stdout:2070: finishing write tid 2 to smithi07134777-24 2024-09-11T01:42:59.341 INFO:tasks.rados.rados.0.smithi071.stdout:2070: finishing write tid 3 to smithi07134777-24 2024-09-11T01:42:59.341 INFO:tasks.rados.rados.0.smithi071.stdout:2070: finishing write tid 4 to smithi07134777-24 2024-09-11T01:42:59.341 INFO:tasks.rados.rados.0.smithi071.stdout:2070: oid 24 updating version 0 to 959 2024-09-11T01:42:59.341 INFO:tasks.rados.rados.0.smithi071.stdout:2070: oid 24 updating version 959 to 960 2024-09-11T01:42:59.341 INFO:tasks.rados.rados.0.smithi071.stdout:2070: oid 24 version 960 is already newer than 958 2024-09-11T01:42:59.341 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 960 (ObjNum 751 snap 228 seq_num 751) dirty exists 2024-09-11T01:42:59.341 INFO:tasks.rados.rados.0.smithi071.stdout:2070: left oid 24 (ObjNum 751 snap 228 seq_num 751) 2024-09-11T01:42:59.341 INFO:tasks.rados.rados.0.smithi071.stdout:2074: finishing rollback tid 0 to smithi07134777-9 2024-09-11T01:42:59.341 INFO:tasks.rados.rados.0.smithi071.stdout:2074: finishing rollback tid 1 to smithi07134777-9 2024-09-11T01:42:59.342 INFO:tasks.rados.rados.0.smithi071.stdout:2073: finishing write tid 1 to smithi07134777-39 2024-09-11T01:42:59.342 INFO:tasks.rados.rados.0.smithi071.stdout:2074: finishing rollback tid 2 to smithi07134777-9 2024-09-11T01:42:59.342 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 982 (ObjNum 635 snap 192 seq_num 635) dirty exists 2024-09-11T01:42:59.342 INFO:tasks.rados.rados.0.smithi071.stdout:2073: finishing write tid 2 to smithi07134777-39 2024-09-11T01:42:59.342 INFO:tasks.rados.rados.0.smithi071.stdout:2073: finishing write tid 3 to smithi07134777-39 2024-09-11T01:42:59.342 INFO:tasks.rados.rados.0.smithi071.stdout:2073: finishing write tid 4 to smithi07134777-39 2024-09-11T01:42:59.342 INFO:tasks.rados.rados.0.smithi071.stdout:2073: oid 39 updating version 0 to 986 2024-09-11T01:42:59.342 INFO:tasks.rados.rados.0.smithi071.stdout:2073: oid 39 updating version 986 to 987 2024-09-11T01:42:59.342 INFO:tasks.rados.rados.0.smithi071.stdout:2073: oid 39 updating version 987 to 988 2024-09-11T01:42:59.342 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 988 (ObjNum 752 snap 228 seq_num 752) dirty exists 2024-09-11T01:42:59.342 INFO:tasks.rados.rados.0.smithi071.stdout:2073: left oid 39 (ObjNum 752 snap 228 seq_num 752) 2024-09-11T01:42:59.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.344+0000 7ff3d7ecd640 10 monclient: get_monmap_and_config 2024-09-11T01:42:59.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.344+0000 7ff3d7ecd640 10 monclient: build_initial_monmap 2024-09-11T01:42:59.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.344+0000 7ff3d7ecd640 10 monclient: monmap: 2024-09-11T01:42:59.346 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:59.346 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:59.346 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:42:59.345489+0000 2024-09-11T01:42:59.346 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:42:59.345489+0000 2024-09-11T01:42:59.346 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:59.346 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:59.346 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:42:59.346 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:42:59.347 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:42:59.347 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:42:59.347 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:42:59.347 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:59.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.345+0000 7ff3d7ecd640 1 Processor -- start 2024-09-11T01:42:59.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.345+0000 7ff3d7ecd640 1 -- start start 2024-09-11T01:42:59.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.345+0000 7ff3d7ecd640 10 monclient: init 2024-09-11T01:42:59.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d7ecd640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:59.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d7ecd640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-11T01:42:59.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d7ecd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff3d0138230 0x7ff3d0138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:59.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d7ecd640 10 monclient(hunting): picked mon.noname-c con 0x7ff3d0138230 addr v2:172.21.15.71:3300/0 2024-09-11T01:42:59.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d7ecd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff3d009d030 0x7ff3d00a5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:59.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d7ecd640 10 monclient(hunting): picked mon.noname-a con 0x7ff3d009d030 addr v2:172.21.15.12:3300/0 2024-09-11T01:42:59.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d7ecd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff3d009d460 addr v1:172.21.15.71:6789/0 2024-09-11T01:42:59.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d7ecd640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3d00a6e30 con 0x7ff3d009d460 2024-09-11T01:42:59.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d7ecd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:59.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d7ecd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff3d009c9c0 con 0x7ff3d009d030 2024-09-11T01:42:59.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d7ecd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:59.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d7ecd640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7ff3d009ba20 con 0x7ff3d0138230 2024-09-11T01:42:59.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d7ecd640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:59.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d7ecd640 10 monclient(hunting): authenticate will time out at 1585.788264s 2024-09-11T01:42:59.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5468640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff3d009d030 0x7ff3d00a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5468640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff3d009d030 0x7ff3d00a5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44986/0 (socket says 172.21.15.12:44986) 2024-09-11T01:42:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5468640 1 -- 172.21.15.12:0/1076052408 learned_addr learned my addr 172.21.15.12:0/1076052408 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5468640 10 monclient(hunting): get_auth_request con 0x7ff3d009d030 auth_method 0 2024-09-11T01:42:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5468640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5468640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5468640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5c69640 1 --2- 172.21.15.12:0/1076052408 >> v2:172.21.15.71:3300/0 conn(0x7ff3d0138230 0x7ff3d0138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5c69640 10 monclient(hunting): get_auth_request con 0x7ff3d0138230 auth_method 0 2024-09-11T01:42:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5c69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5c69640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5c69640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5468640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5468640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:59.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5468640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:59.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5c69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:59.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5c69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:59.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5c69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:59.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5468640 10 monclient(hunting): handle_auth_done global_id 94650 payload 274 2024-09-11T01:42:59.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5468640 1 -- 172.21.15.12:0/1076052408 >> v2:172.21.15.71:3300/0 conn(0x7ff3d0138230 msgr2=0x7ff3d0138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:59.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5468640 1 --2- 172.21.15.12:0/1076052408 >> v2:172.21.15.71:3300/0 conn(0x7ff3d0138230 0x7ff3d0138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:59.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5468640 1 -- 172.21.15.12:0/1076052408 >> v1:172.21.15.71:6789/0 conn(0x7ff3d009d460 legacy=0x7ff3d00a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:42:59.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5468640 10 monclient: _finish_hunting 0 2024-09-11T01:42:59.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5468640 1 monclient: found mon.noname-a 2024-09-11T01:42:59.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5468640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:59.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5468640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:42:59.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.346+0000 7ff3d5468640 1 -- 172.21.15.12:0/1076052408 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3d00a66a0 con 0x7ff3d009d030 2024-09-11T01:42:59.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.349+0000 7ff3d5468640 1 -- 172.21.15.12:0/1076052408 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3d009d030 msgr2=0x7ff3d00a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:59.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.349+0000 7ff3d5468640 1 --2- 172.21.15.12:0/1076052408 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3d009d030 0x7ff3d00a5020 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff3d00a17e0 tx=0x7ff3c402a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebfbec4a20bfef96 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:59.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.349+0000 7ff3d4c67640 1 -- 172.21.15.12:0/1076052408 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c4037070 con 0x7ff3d009d030 2024-09-11T01:42:59.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.349+0000 7ff3d4c67640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:59.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.349+0000 7ff3d4c67640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:42:59.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.349+0000 7ff3d4c67640 10 monclient: dump: 2024-09-11T01:42:59.361 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:59.361 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:59.361 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:59.361 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:59.361 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:59.361 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:59.361 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:59.361 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:59.362 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:59.362 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:59.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.349+0000 7ff3d4c67640 10 monclient: _finish_auth 0 2024-09-11T01:42:59.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.349+0000 7ff3d4c67640 10 monclient: _check_auth_tickets 2024-09-11T01:42:59.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.349+0000 7ff3d4c67640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:59.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.349+0000 7ff3d4c67640 1 -- 172.21.15.12:0/1076052408 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3c4032640 con 0x7ff3d009d030 2024-09-11T01:42:59.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.349+0000 7ff3d7ecd640 5 monclient: authenticate success, global_id 94650 2024-09-11T01:42:59.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.349+0000 7ff3d7ecd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:42:59.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.349+0000 7ff3d4c67640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:59.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.349+0000 7ff3d7ecd640 10 monclient: get_monmap_and_config success 2024-09-11T01:42:59.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.349+0000 7ff3d4c67640 1 -- 172.21.15.12:0/1076052408 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c4032a10 con 0x7ff3d009d030 2024-09-11T01:42:59.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.349+0000 7ff3d4c67640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:59.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.349+0000 7ff3d7ecd640 4 set_mon_vals no callback set 2024-09-11T01:42:59.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.349+0000 7ff3d4c67640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:59.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.349+0000 7ff3d4c67640 10 monclient: dump: 2024-09-11T01:42:59.363 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:59.363 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:59.363 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:59.363 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:59.363 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:59.363 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:59.363 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:59.363 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:59.363 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:59.363 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:59.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.350+0000 7ff3d7ecd640 10 monclient: shutdown 2024-09-11T01:42:59.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.350+0000 7ff3d7ecd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:59.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.350+0000 7ff3d7ecd640 1 -- 172.21.15.12:0/1076052408 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3d009d030 msgr2=0x7ff3d00a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:59.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.350+0000 7ff3d7ecd640 1 --2- 172.21.15.12:0/1076052408 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3d009d030 0x7ff3d00a5020 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff3d00a17e0 tx=0x7ff3c402a0b0 comp rx=0 tx=0).stop 2024-09-11T01:42:59.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.350+0000 7ff3d7ecd640 1 -- 172.21.15.12:0/1076052408 shutdown_connections 2024-09-11T01:42:59.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.350+0000 7ff3d7ecd640 1 --2- 172.21.15.12:0/1076052408 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3d009d030 0x7ff3d00a5020 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:59.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.350+0000 7ff3d7ecd640 1 --2- 172.21.15.12:0/1076052408 >> v2:172.21.15.71:3300/0 conn(0x7ff3d0138230 0x7ff3d0138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:59.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.350+0000 7ff3d7ecd640 1 -- 172.21.15.12:0/1076052408 >> 172.21.15.12:0/1076052408 conn(0x7ff3d00a7120 msgr2=0x7ff3d00a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:59.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.350+0000 7ff3d7ecd640 1 -- 172.21.15.12:0/1076052408 shutdown_connections 2024-09-11T01:42:59.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.350+0000 7ff3d7ecd640 1 -- 172.21.15.12:0/1076052408 wait complete. 2024-09-11T01:42:59.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.351+0000 7ff3d7ecd640 10 monclient: build_initial_monmap 2024-09-11T01:42:59.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.351+0000 7ff3d7ecd640 10 monclient: monmap: 2024-09-11T01:42:59.364 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:42:59.364 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:42:59.364 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:42:59.364 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:42:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:42:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:42:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:42:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:42:59.365 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.351+0000 7ff3d7ecd640 1 Processor -- start 2024-09-11T01:42:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.351+0000 7ff3d7ecd640 1 -- start start 2024-09-11T01:42:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.351+0000 7ff3d7ecd640 10 monclient: init 2024-09-11T01:42:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.351+0000 7ff3d7ecd640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.351+0000 7ff3d7ecd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:42:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.351+0000 7ff3d7ecd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3d009d030 0x7ff3d01ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.351+0000 7ff3d7ecd640 10 monclient(hunting): picked mon.noname-c con 0x7ff3d009d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.351+0000 7ff3d7ecd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3d009d460 0x7ff3d0136eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.351+0000 7ff3d7ecd640 10 monclient(hunting): picked mon.noname-a con 0x7ff3d009d460 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:42:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d7ecd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3d0138230 0x7ff3d01373f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d7ecd640 10 monclient(hunting): picked mon.noname-b con 0x7ff3d0138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:42:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d7ecd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d7ecd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d006f970 con 0x7ff3d009d460 2024-09-11T01:42:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d7ecd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d7ecd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d006f7f0 con 0x7ff3d0138230 2024-09-11T01:42:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d7ecd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d7ecd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff3d00a5c70 con 0x7ff3d009d030 2024-09-11T01:42:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d7ecd640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d7ecd640 10 monclient(hunting): authenticate will time out at 1585.794044s 2024-09-11T01:42:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5c69640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3d009d030 0x7ff3d01ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5468640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3d009d460 0x7ff3d0136eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5c69640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3d009d030 0x7ff3d01ccee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:60594/0 (socket says 172.21.15.12:60594) 2024-09-11T01:42:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5c69640 1 -- 172.21.15.12:0/4105880110 learned_addr learned my addr 172.21.15.12:0/4105880110 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:42:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5c69640 10 monclient(hunting): get_auth_request con 0x7ff3d009d030 auth_method 0 2024-09-11T01:42:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5c69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5c69640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5c69640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5468640 10 monclient(hunting): get_auth_request con 0x7ff3d009d460 auth_method 0 2024-09-11T01:42:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5468640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5468640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5468640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5c69640 1 -- 172.21.15.12:0/4105880110 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3d009d030 msgr2=0x7ff3d01ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:42:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5468640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5468640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5c69640 1 -- 172.21.15.12:0/4105880110 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3d009d030 msgr2=0x7ff3d01ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:42:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5c69640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3d009d030 0x7ff3d01ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:42:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5468640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5c69640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3d009d030 0x7ff3d01ccee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:42:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d646a640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3d0138230 0x7ff3d01373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d646a640 10 monclient(hunting): get_auth_request con 0x7ff3d0138230 auth_method 0 2024-09-11T01:42:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d646a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d646a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d646a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:42:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d646a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d646a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d646a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:42:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5468640 10 monclient(hunting): handle_auth_done global_id 94653 payload 824 2024-09-11T01:42:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5468640 1 -- 172.21.15.12:0/4105880110 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3d009d030 msgr2=0x7ff3d01ccee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:42:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5468640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3d009d030 0x7ff3d01ccee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5468640 1 -- 172.21.15.12:0/4105880110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3d0138230 msgr2=0x7ff3d01373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5468640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3d0138230 0x7ff3d01373f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5468640 10 monclient: _finish_hunting 0 2024-09-11T01:42:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5468640 1 monclient: found mon.noname-a 2024-09-11T01:42:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5468640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5468640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:42:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.352+0000 7ff3d5468640 1 -- 172.21.15.12:0/4105880110 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3d00a1480 con 0x7ff3d009d460 2024-09-11T01:42:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.354+0000 7ff3d5468640 1 -- 172.21.15.12:0/4105880110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3d009d460 msgr2=0x7ff3d0136eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.354+0000 7ff3d5468640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3d009d460 0x7ff3d0136eb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4031000 tx=0x7ff3c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.354+0000 7ff3ce7fc640 1 -- 172.21.15.12:0/4105880110 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c4037040 con 0x7ff3d009d460 2024-09-11T01:42:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.354+0000 7ff3ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.354+0000 7ff3ce7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:42:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.354+0000 7ff3ce7fc640 10 monclient: dump: 2024-09-11T01:42:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:59.370 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.354+0000 7ff3ce7fc640 10 monclient: _finish_auth 0 2024-09-11T01:42:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.354+0000 7ff3ce7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:42:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.354+0000 7ff3ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.354+0000 7ff3d7ecd640 5 monclient: authenticate success, global_id 94653 2024-09-11T01:42:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.354+0000 7ff3ce7fc640 1 -- 172.21.15.12:0/4105880110 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3c4042070 con 0x7ff3d009d460 2024-09-11T01:42:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.354+0000 7ff3ce7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d7ecd640 10 monclient: _renew_subs 2024-09-11T01:42:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.354+0000 7ff3ce7fc640 1 -- 172.21.15.12:0/4105880110 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c403ba80 con 0x7ff3d009d460 2024-09-11T01:42:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d7ecd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d7ecd640 1 -- 172.21.15.12:0/4105880110 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3d00a1b90 con 0x7ff3d009d460 2024-09-11T01:42:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3ce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3ce7fc640 10 monclient: dump: 2024-09-11T01:42:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:59.372 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3cffff640 4 set_mon_vals no callback set 2024-09-11T01:42:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d7ecd640 10 monclient: _renew_subs 2024-09-11T01:42:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d7ecd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d7ecd640 1 -- 172.21.15.12:0/4105880110 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3d0123fd0 con 0x7ff3d009d460 2024-09-11T01:42:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5468640 1 -- 172.21.15.12:0/4105880110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3d009d460 msgr2=0x7ff3d0136eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:42:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5468640 1 -- 172.21.15.12:0/4105880110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3d009d460 msgr2=0x7ff3d0136eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:42:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5468640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3d009d460 0x7ff3d0136eb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4031000 tx=0x7ff3c40046c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:42:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5468640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3d009d460 0x7ff3d0136eb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4031000 tx=0x7ff3c40046c0 comp rx=0 tx=0).stop 2024-09-11T01:42:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3ce7fc640 10 monclient: ms_handle_reset current mon [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:42:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3ce7fc640 10 monclient: _reopen_session rank -1 2024-09-11T01:42:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3ce7fc640 1 -- 172.21.15.12:0/4105880110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3d009d460 msgr2=0x7ff3d0136eb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:42:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3ce7fc640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3d009d460 0x7ff3d0136eb0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3ce7fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:42:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3ce7fc640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3bc003860 0x7ff3bc005c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3ce7fc640 10 monclient(hunting): picked mon.a con 0x7ff3bc003860 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:42:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3ce7fc640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3d009d030 0x7ff3bc008750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3ce7fc640 10 monclient(hunting): picked mon.c con 0x7ff3d009d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:42:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3ce7fc640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3d0138230 0x7ff3bc00b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3ce7fc640 10 monclient(hunting): picked mon.b con 0x7ff3d0138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:42:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3ce7fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3ce7fc640 1 -- 172.21.15.12:0/4105880110 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff3bc00bbe0 con 0x7ff3bc003860 2024-09-11T01:42:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3ce7fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3ce7fc640 1 -- 172.21.15.12:0/4105880110 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff3bc00bd60 con 0x7ff3d0138230 2024-09-11T01:42:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3ce7fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:42:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3ce7fc640 1 -- 172.21.15.12:0/4105880110 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff3bc00bf30 con 0x7ff3d009d030 2024-09-11T01:42:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3ce7fc640 10 monclient(hunting): _renew_subs 2024-09-11T01:42:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5468640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3d009d030 0x7ff3bc008750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5c69640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3bc003860 0x7ff3bc005c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5c69640 10 monclient(hunting): get_auth_request con 0x7ff3bc003860 auth_method 0 2024-09-11T01:42:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5c69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5c69640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5c69640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:42:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5468640 10 monclient(hunting): get_auth_request con 0x7ff3d009d030 auth_method 0 2024-09-11T01:42:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5468640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:42:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5468640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:42:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5468640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:42:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5c69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5c69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5c69640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:42:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5468640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:42:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5468640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:42:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5468640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:42:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d646a640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3d0138230 0x7ff3bc00b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5c69640 10 monclient(hunting): handle_auth_done global_id 94653 payload 293 2024-09-11T01:42:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5c69640 1 -- 172.21.15.12:0/4105880110 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3d009d030 msgr2=0x7ff3bc008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5c69640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3d009d030 0x7ff3bc008750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5c69640 1 -- 172.21.15.12:0/4105880110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3d0138230 msgr2=0x7ff3bc00b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5c69640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3d0138230 0x7ff3bc00b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:59.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5c69640 10 monclient: _finish_hunting 0 2024-09-11T01:42:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5c69640 1 monclient: found mon.a 2024-09-11T01:42:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.355+0000 7ff3d5c69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:42:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.356+0000 7ff3d5c69640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.356+0000 7ff3d5c69640 1 -- 172.21.15.12:0/4105880110 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7ff3bc00c1f0 con 0x7ff3bc003860 2024-09-11T01:42:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.356+0000 7ff3d5c69640 10 monclient: _finish_auth 0 2024-09-11T01:42:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.356+0000 7ff3d5c69640 10 monclient: _check_auth_tickets 2024-09-11T01:42:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.356+0000 7ff3d5c69640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:42:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.356+0000 7ff3d646a640 10 monclient: get_auth_request con 0x7ff3d0138230 auth_method 0 2024-09-11T01:42:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.356+0000 7ff3d646a640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3d0138230 0x7ff3bc00b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:42:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.356+0000 7ff3d7ecd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.356+0000 7ff3d7ecd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.356+0000 7ff3d7ecd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:59.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.356+0000 7ff3d7ecd640 1 -- 172.21.15.12:0/4105880110 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3d0138610 con 0x7ff3bc003860 2024-09-11T01:42:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.358+0000 7ff3d5c69640 1 -- 172.21.15.12:0/4105880110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3bc003860 msgr2=0x7ff3bc005c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:42:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.358+0000 7ff3d5c69640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3bc003860 0x7ff3bc005c50 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8009c20 tx=0x7ff3c402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.358+0000 7ff3ce7fc640 1 -- 172.21.15.12:0/4105880110 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8019030 con 0x7ff3bc003860 2024-09-11T01:42:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.358+0000 7ff3ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:42:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.358+0000 7ff3ce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:42:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.358+0000 7ff3ce7fc640 10 monclient: dump: 2024-09-11T01:42:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:42:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:42:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:42:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:42:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:42:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:42:59.377 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:42:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:42:59.378 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:42:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.358+0000 7ff3ce7fc640 1 -- 172.21.15.12:0/4105880110 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3c8009e30 con 0x7ff3bc003860 2024-09-11T01:42:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.358+0000 7ff3ce7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:42:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.358+0000 7ff3cf7fe640 4 set_mon_vals no callback set 2024-09-11T01:42:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.359+0000 7ff3ce7fc640 1 -- 172.21.15.12:0/4105880110 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff3c8011cd0 con 0x7ff3bc003860 2024-09-11T01:42:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.359+0000 7ff3ce7fc640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff3d009d460 0x7ff3bc048b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:42:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.359+0000 7ff3ce7fc640 1 -- 172.21.15.12:0/4105880110 <== mon.0 v2:172.21.15.12:3300/0 4 ==== osd_map(466..466 src has 285..466) ==== 4859+0+0 (secure 0 0 0) 0x7ff3c8051060 con 0x7ff3bc003860 2024-09-11T01:42:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.359+0000 7ff3d5468640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff3d009d460 0x7ff3bc048b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:42:59.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.359+0000 7ff3d5468640 10 monclient: get_auth_request con 0x7ff3d009d460 auth_method 0 2024-09-11T01:42:59.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.362+0000 7ff3d5468640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff3d009d460 0x7ff3bc048b30 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff3c40090b0 tx=0x7ff3c403a000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:42:59.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.363+0000 7ff3ce7fc640 1 -- 172.21.15.12:0/4105880110 <== mon.0 v2:172.21.15.12:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff3c8005dc0 con 0x7ff3bc003860 2024-09-11T01:42:59.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.363+0000 7ff3ce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:42:59.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.363+0000 7ff3ce7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:42:59.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.617+0000 7ff3d7ecd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:42:59.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.617+0000 7ff3d7ecd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:42:59.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.617+0000 7ff3d7ecd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:42:59.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.617+0000 7ff3d7ecd640 1 -- 172.21.15.12:0/4105880110 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff3d00698e0 con 0x7ff3bc003860 2024-09-11T01:42:59.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.618+0000 7ff3ce7fc640 1 -- 172.21.15.12:0/4105880110 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7ff3c8013d10 con 0x7ff3bc003860 2024-09-11T01:42:59.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.618+0000 7ff3ce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:42:59.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.618+0000 7ff3ce7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:42:59.620 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":126,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:42:59.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.620+0000 7ff3d7ecd640 1 -- 172.21.15.12:0/4105880110 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff3d009d460 msgr2=0x7ff3bc048b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:59.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.620+0000 7ff3d7ecd640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff3d009d460 0x7ff3bc048b30 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff3c40090b0 tx=0x7ff3c403a000 comp rx=0 tx=0).stop 2024-09-11T01:42:59.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.620+0000 7ff3d7ecd640 10 monclient: shutdown 2024-09-11T01:42:59.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.620+0000 7ff3d7ecd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:42:59.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.620+0000 7ff3d7ecd640 1 -- 172.21.15.12:0/4105880110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3bc003860 msgr2=0x7ff3bc005c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:42:59.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.620+0000 7ff3d7ecd640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3bc003860 0x7ff3bc005c50 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8009c20 tx=0x7ff3c402e000 comp rx=0 tx=0).stop 2024-09-11T01:42:59.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.620+0000 7ff3d7ecd640 1 -- 172.21.15.12:0/4105880110 shutdown_connections 2024-09-11T01:42:59.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.620+0000 7ff3d7ecd640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3d0138230 0x7ff3bc00b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:59.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.620+0000 7ff3d7ecd640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff3d009d030 0x7ff3bc008750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:59.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.620+0000 7ff3d7ecd640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff3d009d460 0x7ff3bc048b30 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:59.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.620+0000 7ff3d7ecd640 1 --2- 172.21.15.12:0/4105880110 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3bc003860 0x7ff3bc005c50 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:42:59.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.620+0000 7ff3d7ecd640 1 -- 172.21.15.12:0/4105880110 >> 172.21.15.12:0/4105880110 conn(0x7ff3d00a7120 msgr2=0x7ff3d00a3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:42:59.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.620+0000 7ff3d7ecd640 1 -- 172.21.15.12:0/4105880110 shutdown_connections 2024-09-11T01:42:59.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:42:59.620+0000 7ff3d7ecd640 1 -- 172.21.15.12:0/4105880110 wait complete. 2024-09-11T01:43:00.011 INFO:tasks.rados.rados.0.smithi071.stdout:2069: done (5 left) 2024-09-11T01:43:00.011 INFO:tasks.rados.rados.0.smithi071.stdout:2070: done (4 left) 2024-09-11T01:43:00.012 INFO:tasks.rados.rados.0.smithi071.stdout:2071: done (3 left) 2024-09-11T01:43:00.012 INFO:tasks.rados.rados.0.smithi071.stdout:2073: done (2 left) 2024-09-11T01:43:00.012 INFO:tasks.rados.rados.0.smithi071.stdout:2074: done (1 left) 2024-09-11T01:43:00.012 INFO:tasks.rados.rados.0.smithi071.stdout:2075: done (0 left) 2024-09-11T01:43:00.012 INFO:tasks.rados.rados.0.smithi071.stdout:2076: copy_from oid 8 from oid 3 current snap is 229 2024-09-11T01:43:00.012 INFO:tasks.rados.rados.0.smithi071.stdout:2077: rollback oid 36 current snap is 229 2024-09-11T01:43:00.012 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 36 to 227 2024-09-11T01:43:00.012 INFO:tasks.rados.rados.0.smithi071.stdout:2078: read oid 49 snap -1 2024-09-11T01:43:00.012 INFO:tasks.rados.rados.0.smithi071.stdout:2078: expect deleted 2024-09-11T01:43:00.012 INFO:tasks.rados.rados.0.smithi071.stdout:2079: read oid 22 snap -1 2024-09-11T01:43:00.012 INFO:tasks.rados.rados.0.smithi071.stdout:2079: expect (ObjNum 658 snap 201 seq_num 658) 2024-09-11T01:43:00.013 INFO:tasks.rados.rados.0.smithi071.stdout:2080: snap_remove snap 226 2024-09-11T01:43:01.013 INFO:tasks.rados.rados.0.smithi071.stdout:2080: done (4 left) 2024-09-11T01:43:01.013 INFO:tasks.rados.rados.0.smithi071.stdout:2081: read oid 5 snap -1 2024-09-11T01:43:01.014 INFO:tasks.rados.rados.0.smithi071.stdout:2081: expect deleted 2024-09-11T01:43:01.014 INFO:tasks.rados.rados.0.smithi071.stdout:2082: snap_remove snap 203 2024-09-11T01:43:02.021 INFO:tasks.rados.rados.0.smithi071.stdout:2082: done (5 left) 2024-09-11T01:43:02.021 INFO:tasks.rados.rados.0.smithi071.stdout:2083: read oid 50 snap -1 2024-09-11T01:43:02.021 INFO:tasks.rados.rados.0.smithi071.stdout:2083: expect (ObjNum 631 snap 190 seq_num 631) 2024-09-11T01:43:02.021 INFO:tasks.rados.rados.0.smithi071.stdout:2084: read oid 1 snap -1 2024-09-11T01:43:02.021 INFO:tasks.rados.rados.0.smithi071.stdout:2084: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:43:02.021 INFO:tasks.rados.rados.0.smithi071.stdout:2085: write (excl) oid 41 current snap is 229 2024-09-11T01:43:02.022 INFO:tasks.rados.rados.0.smithi071.stdout:2085: seq_num 753 ranges {655671=581444,2030588=625083} 2024-09-11T01:43:02.028 INFO:tasks.rados.rados.0.smithi071.stdout:2085: writing smithi07134777-41 from 655671 to 1237115 tid 1 2024-09-11T01:43:02.037 INFO:tasks.rados.rados.0.smithi071.stdout:2085: writing smithi07134777-41 from 2030588 to 2655671 tid 2 2024-09-11T01:43:02.037 INFO:tasks.rados.rados.0.smithi071.stdout:2086: write oid 46 current snap is 229 2024-09-11T01:43:02.038 INFO:tasks.rados.rados.0.smithi071.stdout:2086: seq_num 754 ranges {672478=656693,2089070=459917} 2024-09-11T01:43:02.045 INFO:tasks.rados.rados.0.smithi071.stdout:2086: writing smithi07134777-46 from 672478 to 1329171 tid 1 2024-09-11T01:43:02.052 INFO:tasks.rados.rados.0.smithi071.stdout:2086: writing smithi07134777-46 from 2089070 to 2548987 tid 2 2024-09-11T01:43:02.053 INFO:tasks.rados.rados.0.smithi071.stdout:2087: read oid 43 snap -1 2024-09-11T01:43:02.053 INFO:tasks.rados.rados.0.smithi071.stdout:2087: expect (ObjNum 542 snap 154 seq_num 542) 2024-09-11T01:43:02.053 INFO:tasks.rados.rados.0.smithi071.stdout:2088: rollback oid 34 current snap is 229 2024-09-11T01:43:02.053 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 34 to 212 2024-09-11T01:43:02.053 INFO:tasks.rados.rados.0.smithi071.stdout:2079: expect (ObjNum 658 snap 201 seq_num 658) 2024-09-11T01:43:02.100 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:02.194 INFO:tasks.rados.rados.0.smithi071.stdout:2077: finishing rollback tid 0 to smithi07134777-36 2024-09-11T01:43:02.194 INFO:tasks.rados.rados.0.smithi071.stdout:2077: finishing rollback tid 1 to smithi07134777-36 2024-09-11T01:43:02.194 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 1019 (ObjNum 129 snap 0 seq_num 2370914968) dirty dne 2024-09-11T01:43:02.195 INFO:tasks.rados.rados.0.smithi071.stdout:2076: finishing copy_from racing read to smithi07134777-8 2024-09-11T01:43:02.195 INFO:tasks.rados.rados.0.smithi071.stdout:2076: finishing copy_from to smithi07134777-8 2024-09-11T01:43:02.195 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 703 (ObjNum 732 snap 225 seq_num 732) dirty exists 2024-09-11T01:43:02.195 INFO:tasks.rados.rados.0.smithi071.stdout:2076: done (10 left) 2024-09-11T01:43:02.195 INFO:tasks.rados.rados.0.smithi071.stdout:2077: done (9 left) 2024-09-11T01:43:02.195 INFO:tasks.rados.rados.0.smithi071.stdout:2078: done (8 left) 2024-09-11T01:43:02.195 INFO:tasks.rados.rados.0.smithi071.stdout:2079: done (7 left) 2024-09-11T01:43:02.195 INFO:tasks.rados.rados.0.smithi071.stdout:2081: done (6 left) 2024-09-11T01:43:02.195 INFO:tasks.rados.rados.0.smithi071.stdout:2089: rollback oid 2 current snap is 229 2024-09-11T01:43:02.195 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 2 to 220 2024-09-11T01:43:02.195 INFO:tasks.rados.rados.0.smithi071.stdout:2083: expect (ObjNum 631 snap 190 seq_num 631) 2024-09-11T01:43:02.212 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:02.245 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1052010 2024-09-11T01:43:02.261 INFO:tasks.rados.rados.0.smithi071.stdout:2084: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:43:02.318 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3541681 2024-09-11T01:43:02.318 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:02.356 INFO:tasks.rados.rados.0.smithi071.stdout:2085: finishing write tid 1 to smithi07134777-41 2024-09-11T01:43:02.356 INFO:tasks.rados.rados.0.smithi071.stdout:2085: finishing write tid 2 to smithi07134777-41 2024-09-11T01:43:02.357 INFO:tasks.rados.rados.0.smithi071.stdout:2085: finishing write tid 3 to smithi07134777-41 2024-09-11T01:43:02.357 INFO:tasks.rados.rados.0.smithi071.stdout:2085: finishing write tid 4 to smithi07134777-41 2024-09-11T01:43:02.357 INFO:tasks.rados.rados.0.smithi071.stdout:2085: oid 41 updating version 0 to 399 2024-09-11T01:43:02.357 INFO:tasks.rados.rados.0.smithi071.stdout:2085: oid 41 version 399 is already newer than 398 2024-09-11T01:43:02.357 INFO:tasks.rados.rados.0.smithi071.stdout:2085: oid 41 updating version 399 to 400 2024-09-11T01:43:02.357 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 400 (ObjNum 753 snap 229 seq_num 753) dirty exists 2024-09-11T01:43:02.357 INFO:tasks.rados.rados.0.smithi071.stdout:2085: left oid 41 (ObjNum 753 snap 229 seq_num 753) 2024-09-11T01:43:02.357 INFO:tasks.rados.rados.0.smithi071.stdout:2086: finishing write tid 1 to smithi07134777-46 2024-09-11T01:43:02.357 INFO:tasks.rados.rados.0.smithi071.stdout:2086: finishing write tid 2 to smithi07134777-46 2024-09-11T01:43:02.357 INFO:tasks.rados.rados.0.smithi071.stdout:2083: done (6 left) 2024-09-11T01:43:02.357 INFO:tasks.rados.rados.0.smithi071.stdout:2084: done (5 left) 2024-09-11T01:43:02.357 INFO:tasks.rados.rados.0.smithi071.stdout:2085: done (4 left) 2024-09-11T01:43:02.357 INFO:tasks.rados.rados.0.smithi071.stdout:2090: read oid 50 snap -1 2024-09-11T01:43:02.357 INFO:tasks.rados.rados.0.smithi071.stdout:2090: expect (ObjNum 631 snap 190 seq_num 631) 2024-09-11T01:43:02.358 INFO:tasks.rados.rados.0.smithi071.stdout:2091: delete oid 1 current snap is 229 2024-09-11T01:43:02.358 INFO:tasks.rados.rados.0.smithi071.stdout:2086: finishing write tid 3 to smithi07134777-46 2024-09-11T01:43:02.358 INFO:tasks.rados.rados.0.smithi071.stdout:2086: finishing write tid 4 to smithi07134777-46 2024-09-11T01:43:02.358 INFO:tasks.rados.rados.0.smithi071.stdout:2086: oid 46 updating version 0 to 1044 2024-09-11T01:43:02.358 INFO:tasks.rados.rados.0.smithi071.stdout:2086: oid 46 version 1044 is already newer than 1043 2024-09-11T01:43:02.358 INFO:tasks.rados.rados.0.smithi071.stdout:2086: oid 46 updating version 1044 to 1045 2024-09-11T01:43:02.358 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 1045 (ObjNum 754 snap 229 seq_num 754) dirty exists 2024-09-11T01:43:02.358 INFO:tasks.rados.rados.0.smithi071.stdout:2086: left oid 46 (ObjNum 754 snap 229 seq_num 754) 2024-09-11T01:43:02.358 INFO:tasks.rados.rados.0.smithi071.stdout:2087: expect (ObjNum 542 snap 154 seq_num 542) 2024-09-11T01:43:02.358 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:02.422 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1647198 2024-09-11T01:43:02.490 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2654934 2024-09-11T01:43:02.500 INFO:tasks.rados.rados.0.smithi071.stdout:2088: finishing rollback tid 1 to smithi07134777-34 2024-09-11T01:43:02.500 INFO:tasks.rados.rados.0.smithi071.stdout:2088: finishing rollback tid 2 to smithi07134777-34 2024-09-11T01:43:02.501 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 1165 (ObjNum 666 snap 202 seq_num 666) dirty exists 2024-09-11T01:43:02.501 INFO:tasks.rados.rados.0.smithi071.stdout:2089: finishing rollback tid 0 to smithi07134777-2 2024-09-11T01:43:02.501 INFO:tasks.rados.rados.0.smithi071.stdout:2089: finishing rollback tid 1 to smithi07134777-2 2024-09-11T01:43:02.501 INFO:tasks.rados.rados.0.smithi071.stdout:2089: finishing rollback tid 2 to smithi07134777-2 2024-09-11T01:43:02.501 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 842 (ObjNum 686 snap 204 seq_num 686) dirty exists 2024-09-11T01:43:02.501 INFO:tasks.rados.rados.0.smithi071.stdout:2086: done (5 left) 2024-09-11T01:43:02.501 INFO:tasks.rados.rados.0.smithi071.stdout:2087: done (4 left) 2024-09-11T01:43:02.501 INFO:tasks.rados.rados.0.smithi071.stdout:2088: done (3 left) 2024-09-11T01:43:02.501 INFO:tasks.rados.rados.0.smithi071.stdout:2089: done (2 left) 2024-09-11T01:43:02.502 INFO:tasks.rados.rados.0.smithi071.stdout:2091: done (1 left) 2024-09-11T01:43:02.502 INFO:tasks.rados.rados.0.smithi071.stdout:2092: delete oid 14 current snap is 229 2024-09-11T01:43:02.502 INFO:tasks.rados.rados.0.smithi071.stdout:2090: expect (ObjNum 631 snap 190 seq_num 631) 2024-09-11T01:43:02.535 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1259606 2024-09-11T01:43:02.553 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 619250 2024-09-11T01:43:02.574 INFO:tasks.rados.rados.0.smithi071.stdout:2090: done (1 left) 2024-09-11T01:43:02.574 INFO:tasks.rados.rados.0.smithi071.stdout:2092: done (0 left) 2024-09-11T01:43:02.574 INFO:tasks.rados.rados.0.smithi071.stdout:2093: write oid 35 current snap is 229 2024-09-11T01:43:02.574 INFO:tasks.rados.rados.0.smithi071.stdout:2093: seq_num 757 ranges {722899=482440,1748163=723597} 2024-09-11T01:43:02.580 INFO:tasks.rados.rados.0.smithi071.stdout:2093: writing smithi07134777-35 from 722899 to 1205339 tid 1 2024-09-11T01:43:02.587 INFO:tasks.rados.rados.0.smithi071.stdout:2093: writing smithi07134777-35 from 1748163 to 2471760 tid 2 2024-09-11T01:43:02.587 INFO:tasks.rados.rados.0.smithi071.stdout:2094: read oid 30 snap 185 2024-09-11T01:43:02.587 INFO:tasks.rados.rados.0.smithi071.stdout:2094: expect (ObjNum 588 snap 169 seq_num 588) 2024-09-11T01:43:02.588 INFO:tasks.rados.rados.0.smithi071.stdout:2095: read oid 18 snap -1 2024-09-11T01:43:02.588 INFO:tasks.rados.rados.0.smithi071.stdout:2095: expect (ObjNum 724 snap 224 seq_num 724) 2024-09-11T01:43:02.588 INFO:tasks.rados.rados.0.smithi071.stdout:2096: snap_create 2024-09-11T01:43:02.588 INFO:tasks.rados.rados.0.smithi071.stdout:2093: finishing write tid 1 to smithi07134777-35 2024-09-11T01:43:02.594 INFO:tasks.rados.rados.0.smithi071.stdout:2094: expect (ObjNum 588 snap 169 seq_num 588) 2024-09-11T01:43:02.633 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:43:02.709 INFO:tasks.rados.rados.0.smithi071.stdout:2093: finishing write tid 2 to smithi07134777-35 2024-09-11T01:43:02.709 INFO:tasks.rados.rados.0.smithi071.stdout:2093: finishing write tid 3 to smithi07134777-35 2024-09-11T01:43:02.709 INFO:tasks.rados.rados.0.smithi071.stdout:2093: finishing write tid 4 to smithi07134777-35 2024-09-11T01:43:02.709 INFO:tasks.rados.rados.0.smithi071.stdout:2093: oid 35 updating version 0 to 991 2024-09-11T01:43:02.709 INFO:tasks.rados.rados.0.smithi071.stdout:2093: oid 35 version 991 is already newer than 990 2024-09-11T01:43:02.710 INFO:tasks.rados.rados.0.smithi071.stdout:2093: oid 35 version 991 is already newer than 989 2024-09-11T01:43:02.710 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 991 (ObjNum 757 snap 229 seq_num 757) dirty exists 2024-09-11T01:43:02.710 INFO:tasks.rados.rados.0.smithi071.stdout:2093: left oid 35 (ObjNum 757 snap 229 seq_num 757) 2024-09-11T01:43:02.710 INFO:tasks.rados.rados.0.smithi071.stdout:2095: expect (ObjNum 724 snap 224 seq_num 724) 2024-09-11T01:43:02.723 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1473398 2024-09-11T01:43:02.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.742+0000 7f8ac44f6640 10 monclient: get_monmap_and_config 2024-09-11T01:43:02.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.742+0000 7f8ac44f6640 10 monclient: build_initial_monmap 2024-09-11T01:43:02.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.742+0000 7f8ac44f6640 10 monclient: monmap: 2024-09-11T01:43:02.744 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:02.744 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:02.744 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:43:02.743488+0000 2024-09-11T01:43:02.744 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:43:02.743488+0000 2024-09-11T01:43:02.744 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:02.744 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:02.744 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:43:02.744 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:43:02.745 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:43:02.745 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:43:02.745 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:43:02.745 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:02.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.743+0000 7f8ac44f6640 1 Processor -- start 2024-09-11T01:43:02.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.743+0000 7f8ac44f6640 1 -- start start 2024-09-11T01:43:02.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.743+0000 7f8ac44f6640 10 monclient: init 2024-09-11T01:43:02.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac44f6640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:02.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac44f6640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-11T01:43:02.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac44f6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8abc0a2e90 addr v1:172.21.15.71:6789/0 2024-09-11T01:43:02.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac44f6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8abc0a3270 addr v1:172.21.15.12:6789/0 2024-09-11T01:43:02.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac44f6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f8abc13e4a0 0x7f8abc140890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:02.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac44f6640 10 monclient(hunting): picked mon.noname-c con 0x7f8abc13e4a0 addr v2:172.21.15.71:3300/0 2024-09-11T01:43:02.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac44f6640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8abc0a46c0 con 0x7f8abc0a3270 2024-09-11T01:43:02.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac44f6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8abc06ee50 con 0x7f8abc0a2e90 2024-09-11T01:43:02.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac44f6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:02.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac44f6640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f8abc0a4900 con 0x7f8abc13e4a0 2024-09-11T01:43:02.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac44f6640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:02.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac44f6640 10 monclient(hunting): authenticate will time out at 1589.186199s 2024-09-11T01:43:02.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac1a91640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f8abc0a3270 0x7f8abc0a3680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59290/0 (socket says 172.21.15.12:59290) 2024-09-11T01:43:02.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac1a91640 1 -- 172.21.15.12:0/2370393302 learned_addr learned my addr 172.21.15.12:0/2370393302 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:43:02.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac2a93640 1 --2- 172.21.15.12:0/2370393302 >> v2:172.21.15.71:3300/0 conn(0x7f8abc13e4a0 0x7f8abc140890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:02.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac2a93640 10 monclient(hunting): get_auth_request con 0x7f8abc13e4a0 auth_method 0 2024-09-11T01:43:02.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac2a93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:02.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac2a93640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:02.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac2a93640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:02.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac2a93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:02.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac2a93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:02.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac2a93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:02.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac2a93640 10 monclient(hunting): handle_auth_done global_id 134272 payload 274 2024-09-11T01:43:02.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac2a93640 1 -- 172.21.15.12:0/2370393302 >> v1:172.21.15.71:6789/0 conn(0x7f8abc0a2e90 legacy=0x7f8abc06dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:02.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac2a93640 1 -- 172.21.15.12:0/2370393302 >> v1:172.21.15.12:6789/0 conn(0x7f8abc0a3270 legacy=0x7f8abc0a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:02.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac2a93640 10 monclient: _finish_hunting 0 2024-09-11T01:43:02.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac2a93640 1 monclient: found mon.noname-c 2024-09-11T01:43:02.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac2a93640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:02.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac2a93640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:43:02.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.744+0000 7f8ac2a93640 1 -- 172.21.15.12:0/2370393302 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8abc06eb70 con 0x7f8abc13e4a0 2024-09-11T01:43:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.747+0000 7f8ac2a93640 1 -- 172.21.15.12:0/2370393302 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8abc13e4a0 msgr2=0x7f8abc140890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.747+0000 7f8ac2a93640 1 --2- 172.21.15.12:0/2370393302 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8abc13e4a0 0x7f8abc140890 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8abc071140 tx=0x7f8ab402a460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d339b5a816a6e3b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.747+0000 7f8ac1290640 1 -- 172.21.15.12:0/2370393302 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab4007a30 con 0x7f8abc13e4a0 2024-09-11T01:43:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.747+0000 7f8ac1290640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.747+0000 7f8ac1290640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:43:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.747+0000 7f8ac1290640 10 monclient: dump: 2024-09-11T01:43:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:02.757 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.747+0000 7f8ac1290640 10 monclient: _finish_auth 0 2024-09-11T01:43:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.747+0000 7f8ac1290640 10 monclient: _check_auth_tickets 2024-09-11T01:43:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.747+0000 7f8ac1290640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.747+0000 7f8ac1290640 1 -- 172.21.15.12:0/2370393302 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ab4007bd0 con 0x7f8abc13e4a0 2024-09-11T01:43:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.747+0000 7f8ac1290640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:02.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.747+0000 7f8ac1290640 1 -- 172.21.15.12:0/2370393302 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab4031400 con 0x7f8abc13e4a0 2024-09-11T01:43:02.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.747+0000 7f8ac44f6640 5 monclient: authenticate success, global_id 134272 2024-09-11T01:43:02.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.747+0000 7f8ac44f6640 10 monclient: get_monmap_and_config success 2024-09-11T01:43:02.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.747+0000 7f8ac44f6640 4 set_mon_vals no callback set 2024-09-11T01:43:02.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.747+0000 7f8ac1290640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:02.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.747+0000 7f8ac1290640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:43:02.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.747+0000 7f8ac1290640 10 monclient: dump: 2024-09-11T01:43:02.758 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:02.758 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:02.758 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:02.758 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:02.758 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:02.758 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:02.758 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:02.758 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:02.759 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:02.759 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:02.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.748+0000 7f8ac44f6640 10 monclient: shutdown 2024-09-11T01:43:02.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.748+0000 7f8ac44f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:02.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.748+0000 7f8ac44f6640 1 -- 172.21.15.12:0/2370393302 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8abc13e4a0 msgr2=0x7f8abc140890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:02.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.748+0000 7f8ac44f6640 1 --2- 172.21.15.12:0/2370393302 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8abc13e4a0 0x7f8abc140890 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8abc071140 tx=0x7f8ab402a460 comp rx=0 tx=0).stop 2024-09-11T01:43:02.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.748+0000 7f8ac44f6640 1 -- 172.21.15.12:0/2370393302 shutdown_connections 2024-09-11T01:43:02.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.748+0000 7f8ac44f6640 1 --2- 172.21.15.12:0/2370393302 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8abc13e4a0 0x7f8abc140890 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:02.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.748+0000 7f8ac44f6640 1 -- 172.21.15.12:0/2370393302 >> 172.21.15.12:0/2370393302 conn(0x7f8abc0a8490 msgr2=0x7f8abc0a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:02.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.748+0000 7f8ac44f6640 1 -- 172.21.15.12:0/2370393302 shutdown_connections 2024-09-11T01:43:02.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.748+0000 7f8ac44f6640 1 -- 172.21.15.12:0/2370393302 wait complete. 2024-09-11T01:43:02.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.749+0000 7f8ac44f6640 10 monclient: build_initial_monmap 2024-09-11T01:43:02.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.749+0000 7f8ac44f6640 10 monclient: monmap: 2024-09-11T01:43:02.760 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:02.760 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:43:02.760 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:43:02.760 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:43:02.760 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:02.760 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:02.760 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:43:02.760 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:43:02.760 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:43:02.760 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:02.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.749+0000 7f8ac44f6640 1 Processor -- start 2024-09-11T01:43:02.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.749+0000 7f8ac44f6640 1 -- start start 2024-09-11T01:43:02.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.749+0000 7f8ac44f6640 10 monclient: init 2024-09-11T01:43:02.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.749+0000 7f8ac44f6640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:02.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac44f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:43:02.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac44f6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8abc0a2e90 0x7f8abc13d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:02.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac44f6640 10 monclient(hunting): picked mon.noname-b con 0x7f8abc0a2e90 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:02.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac44f6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8abc0a3270 0x7f8abc138500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:02.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac44f6640 10 monclient(hunting): picked mon.noname-c con 0x7f8abc0a3270 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:02.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac44f6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8abc13e4a0 0x7f8abc138a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:02.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac44f6640 10 monclient(hunting): picked mon.noname-a con 0x7f8abc13e4a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:02.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac44f6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:02.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac44f6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8abc140f50 con 0x7f8abc13e4a0 2024-09-11T01:43:02.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac44f6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:02.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac44f6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8abc140dd0 con 0x7f8abc0a2e90 2024-09-11T01:43:02.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac44f6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:02.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac44f6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8abc1410d0 con 0x7f8abc0a3270 2024-09-11T01:43:02.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac44f6640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:02.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac44f6640 10 monclient(hunting): authenticate will time out at 1589.192084s 2024-09-11T01:43:02.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2a93640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8abc13e4a0 0x7f8abc138a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:02.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2292640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8abc0a2e90 0x7f8abc13d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:02.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac1a91640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8abc0a3270 0x7f8abc138500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:02.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2a93640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8abc13e4a0 0x7f8abc138a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45012/0 (socket says 172.21.15.12:45012) 2024-09-11T01:43:02.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2292640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8abc0a2e90 0x7f8abc13d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:60354/0 (socket says 172.21.15.12:60354) 2024-09-11T01:43:02.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2a93640 1 -- 172.21.15.12:0/2524255479 learned_addr learned my addr 172.21.15.12:0/2524255479 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:02.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac1a91640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8abc0a3270 0x7f8abc138500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:60620/0 (socket says 172.21.15.12:60620) 2024-09-11T01:43:02.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac1a91640 10 monclient(hunting): get_auth_request con 0x7f8abc0a3270 auth_method 0 2024-09-11T01:43:02.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac1a91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:02.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac1a91640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:02.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac1a91640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:02.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2292640 10 monclient(hunting): get_auth_request con 0x7f8abc0a2e90 auth_method 0 2024-09-11T01:43:02.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2292640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:02.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2292640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:02.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2292640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:02.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2a93640 10 monclient(hunting): get_auth_request con 0x7f8abc13e4a0 auth_method 0 2024-09-11T01:43:02.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2a93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:02.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2a93640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:02.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2a93640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:02.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2292640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:02.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2292640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:02.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2292640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:02.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2a93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:02.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2a93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:02.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2a93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:02.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac1a91640 1 -- 172.21.15.12:0/2524255479 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8abc0a3270 msgr2=0x7f8abc138500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:43:02.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac1a91640 1 -- 172.21.15.12:0/2524255479 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8abc0a3270 msgr2=0x7f8abc138500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:43:02.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac1a91640 1 --2- 172.21.15.12:0/2524255479 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8abc0a3270 0x7f8abc138500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:02.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac1a91640 1 --2- 172.21.15.12:0/2524255479 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8abc0a3270 0x7f8abc138500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:02.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2292640 10 monclient(hunting): handle_auth_done global_id 134275 payload 824 2024-09-11T01:43:02.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2292640 1 -- 172.21.15.12:0/2524255479 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8abc0a3270 msgr2=0x7f8abc138500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:43:02.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2292640 1 --2- 172.21.15.12:0/2524255479 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8abc0a3270 0x7f8abc138500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:02.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2292640 1 -- 172.21.15.12:0/2524255479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8abc13e4a0 msgr2=0x7f8abc138a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:02.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2292640 1 --2- 172.21.15.12:0/2524255479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8abc13e4a0 0x7f8abc138a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:02.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2292640 10 monclient: _finish_hunting 0 2024-09-11T01:43:02.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2292640 1 monclient: found mon.noname-b 2024-09-11T01:43:02.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2292640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:02.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2292640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-09-11T01:43:02.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.750+0000 7f8ac2292640 1 -- 172.21.15.12:0/2524255479 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8abc13adb0 con 0x7f8abc0a2e90 2024-09-11T01:43:02.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.752+0000 7f8ac2292640 1 -- 172.21.15.12:0/2524255479 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8abc0a2e90 msgr2=0x7f8abc13d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:02.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.752+0000 7f8ac2292640 1 --2- 172.21.15.12:0/2524255479 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8abc0a2e90 0x7f8abc13d8a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8ab000c410 tx=0x7f8ab000c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:02.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.752+0000 7f8aaeffd640 1 -- 172.21.15.12:0/2524255479 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab0017070 con 0x7f8abc0a2e90 2024-09-11T01:43:02.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.752+0000 7f8aaeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:02.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.753+0000 7f8aaeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:43:02.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.753+0000 7f8aaeffd640 10 monclient: dump: 2024-09-11T01:43:02.765 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:02.765 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:02.765 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:02.765 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:02.765 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:02.765 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:02.766 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:02.766 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:02.766 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:02.766 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:02.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.753+0000 7f8aaeffd640 10 monclient: _finish_auth 0 2024-09-11T01:43:02.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.753+0000 7f8aaeffd640 10 monclient: _check_auth_tickets 2024-09-11T01:43:02.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.753+0000 7f8aaeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:02.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.753+0000 7f8aaeffd640 1 -- 172.21.15.12:0/2524255479 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ab0005230 con 0x7f8abc0a2e90 2024-09-11T01:43:02.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.753+0000 7f8ac44f6640 5 monclient: authenticate success, global_id 134275 2024-09-11T01:43:02.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.753+0000 7f8ac44f6640 10 monclient: _renew_subs 2024-09-11T01:43:02.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.753+0000 7f8ac44f6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:43:02.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.753+0000 7f8ac44f6640 1 -- 172.21.15.12:0/2524255479 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8abc13d110 con 0x7f8abc0a2e90 2024-09-11T01:43:02.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.753+0000 7f8aaeffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:02.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.753+0000 7f8aaeffd640 1 -- 172.21.15.12:0/2524255479 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab00117a0 con 0x7f8abc0a2e90 2024-09-11T01:43:02.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.753+0000 7f8aaeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:02.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.753+0000 7f8aaeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:43:02.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.753+0000 7f8aaeffd640 10 monclient: dump: 2024-09-11T01:43:02.767 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:02.767 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:02.767 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:02.767 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:02.767 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:02.767 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:02.767 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:02.768 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:02.768 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:02.768 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:02.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.753+0000 7f8ac44f6640 10 monclient: _renew_subs 2024-09-11T01:43:02.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.753+0000 7f8ac44f6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:43:02.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.753+0000 7f8ac0a8f640 4 set_mon_vals no callback set 2024-09-11T01:43:02.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.753+0000 7f8ac44f6640 1 -- 172.21.15.12:0/2524255479 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8abc138f80 con 0x7f8abc0a2e90 2024-09-11T01:43:02.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.753+0000 7f8ac44f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:02.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.753+0000 7f8ac44f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:02.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.753+0000 7f8ac44f6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:43:02.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.753+0000 7f8aaeffd640 1 -- 172.21.15.12:0/2524255479 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f8ab0011940 con 0x7f8abc0a2e90 2024-09-11T01:43:02.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.753+0000 7f8ac44f6640 1 -- 172.21.15.12:0/2524255479 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8abc06dc60 con 0x7f8abc0a2e90 2024-09-11T01:43:02.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.754+0000 7f8aaeffd640 1 --2- 172.21.15.12:0/2524255479 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8aa003e210 0x7f8aa00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:02.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.754+0000 7f8aaeffd640 1 -- 172.21.15.12:0/2524255479 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(469..469 src has 285..469) ==== 4831+0+0 (secure 0 0 0) 0x7f8ab00523f0 con 0x7f8abc0a2e90 2024-09-11T01:43:02.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.754+0000 7f8ac1a91640 1 --2- 172.21.15.12:0/2524255479 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8aa003e210 0x7f8aa00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:02.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.754+0000 7f8ac1a91640 10 monclient: get_auth_request con 0x7f8aa003e210 auth_method 0 2024-09-11T01:43:02.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.754+0000 7f8ac1a91640 1 -- 172.21.15.12:0/2524255479 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8aa003e210 msgr2=0x7f8aa00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:43:02.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.754+0000 7f8ac1a91640 1 -- 172.21.15.12:0/2524255479 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8aa003e210 msgr2=0x7f8aa00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:43:02.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.754+0000 7f8ac1a91640 1 --2- 172.21.15.12:0/2524255479 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8aa003e210 0x7f8aa00406d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f8ab8007fd0 tx=0x7f8ab8009040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:02.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.754+0000 7f8ac1a91640 1 --2- 172.21.15.12:0/2524255479 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8aa003e210 0x7f8aa00406d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:02.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.758+0000 7f8aaeffd640 1 -- 172.21.15.12:0/2524255479 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8ab0050dc0 con 0x7f8abc0a2e90 2024-09-11T01:43:02.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.758+0000 7f8aaeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:02.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.758+0000 7f8aaeffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:43:02.785 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2087578 2024-09-11T01:43:02.786 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:02.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.955+0000 7f8ac1a91640 1 --2- 172.21.15.12:0/2524255479 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8aa003e210 0x7f8aa00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:02.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.955+0000 7f8ac1a91640 10 monclient: get_auth_request con 0x7f8aa003e210 auth_method 0 2024-09-11T01:43:02.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:02.957+0000 7f8ac1a91640 1 --2- 172.21.15.12:0/2524255479 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8aa003e210 0x7f8aa00406d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8ab8005000 tx=0x7f8ab8006000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:03.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.014+0000 7f8ac44f6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:43:03.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.014+0000 7f8ac44f6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:43:03.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.014+0000 7f8ac44f6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:43:03.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.014+0000 7f8ac44f6640 1 -- 172.21.15.12:0/2524255479 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8abc074ac0 con 0x7f8abc0a2e90 2024-09-11T01:43:03.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.014+0000 7f8aaeffd640 1 -- 172.21.15.12:0/2524255479 <== mon.1 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f8ab00520b0 con 0x7f8abc0a2e90 2024-09-11T01:43:03.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.014+0000 7f8aaeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:43:03.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.014+0000 7f8aaeffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:43:03.016 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":128,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:43:03.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.017+0000 7f8ac44f6640 1 -- 172.21.15.12:0/2524255479 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8aa003e210 msgr2=0x7f8aa00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:03.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.017+0000 7f8ac44f6640 1 --2- 172.21.15.12:0/2524255479 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8aa003e210 0x7f8aa00406d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8ab8005000 tx=0x7f8ab8006000 comp rx=0 tx=0).stop 2024-09-11T01:43:03.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.017+0000 7f8ac44f6640 10 monclient: shutdown 2024-09-11T01:43:03.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.017+0000 7f8ac44f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:03.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.017+0000 7f8ac44f6640 1 -- 172.21.15.12:0/2524255479 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8abc0a2e90 msgr2=0x7f8abc13d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:03.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.017+0000 7f8ac44f6640 1 --2- 172.21.15.12:0/2524255479 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8abc0a2e90 0x7f8abc13d8a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8ab000c410 tx=0x7f8ab000c8d0 comp rx=0 tx=0).stop 2024-09-11T01:43:03.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.017+0000 7f8ac44f6640 1 -- 172.21.15.12:0/2524255479 shutdown_connections 2024-09-11T01:43:03.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.017+0000 7f8ac44f6640 1 --2- 172.21.15.12:0/2524255479 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8aa003e210 0x7f8aa00406d0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:03.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.017+0000 7f8ac44f6640 1 --2- 172.21.15.12:0/2524255479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8abc13e4a0 0x7f8abc138a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:03.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.017+0000 7f8ac44f6640 1 --2- 172.21.15.12:0/2524255479 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8abc0a3270 0x7f8abc138500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:03.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.017+0000 7f8ac44f6640 1 --2- 172.21.15.12:0/2524255479 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8abc0a2e90 0x7f8abc13d8a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:03.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.017+0000 7f8ac44f6640 1 -- 172.21.15.12:0/2524255479 >> 172.21.15.12:0/2524255479 conn(0x7f8abc0a8490 msgr2=0x7f8abc0a66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:03.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.017+0000 7f8ac44f6640 1 -- 172.21.15.12:0/2524255479 shutdown_connections 2024-09-11T01:43:03.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.017+0000 7f8ac44f6640 1 -- 172.21.15.12:0/2524255479 wait complete. 2024-09-11T01:43:03.029 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:43:03.029 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:43:03.029 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:43:03.031 INFO:tasks.rados.rados.0.smithi071.stdout:2093: done (3 left) 2024-09-11T01:43:03.031 INFO:tasks.rados.rados.0.smithi071.stdout:2094: done (2 left) 2024-09-11T01:43:03.031 INFO:tasks.rados.rados.0.smithi071.stdout:2095: done (1 left) 2024-09-11T01:43:03.031 INFO:tasks.rados.rados.0.smithi071.stdout:2096: done (0 left) 2024-09-11T01:43:03.031 INFO:tasks.rados.rados.0.smithi071.stdout:2097: write oid 14 current snap is 230 2024-09-11T01:43:03.031 INFO:tasks.rados.rados.0.smithi071.stdout:2097: seq_num 758 ranges {739706=557689,1806645=467255} 2024-09-11T01:43:03.038 INFO:tasks.rados.rados.0.smithi071.stdout:2097: writing smithi07134777-14 from 739706 to 1297395 tid 1 2024-09-11T01:43:03.044 INFO:tasks.rados.rados.0.smithi071.stdout:2097: writing smithi07134777-14 from 1806645 to 2273900 tid 2 2024-09-11T01:43:03.044 INFO:tasks.rados.rados.0.smithi071.stdout:2098: rollback oid 13 current snap is 230 2024-09-11T01:43:03.045 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 13 to 185 2024-09-11T01:43:03.045 INFO:tasks.rados.rados.0.smithi071.stdout:2097: finishing write tid 1 to smithi07134777-14 2024-09-11T01:43:03.045 INFO:tasks.rados.rados.0.smithi071.stdout:2099: read oid 21 snap 227 2024-09-11T01:43:03.045 INFO:tasks.rados.rados.0.smithi071.stdout:2099: expect (ObjNum 738 snap 226 seq_num 738) 2024-09-11T01:43:03.045 INFO:tasks.rados.rados.0.smithi071.stdout:2100: read oid 34 snap -1 2024-09-11T01:43:03.045 INFO:tasks.rados.rados.0.smithi071.stdout:2100: expect (ObjNum 666 snap 202 seq_num 666) 2024-09-11T01:43:03.045 INFO:tasks.rados.rados.0.smithi071.stdout:2101: delete oid 1 current snap is 230 2024-09-11T01:43:03.049 INFO:tasks.rados.rados.0.smithi071.stdout:2097: finishing write tid 2 to smithi07134777-14 2024-09-11T01:43:03.049 INFO:tasks.rados.rados.0.smithi071.stdout:2097: finishing write tid 3 to smithi07134777-14 2024-09-11T01:43:03.049 INFO:tasks.rados.rados.0.smithi071.stdout:2098: finishing rollback tid 0 to smithi07134777-13 2024-09-11T01:43:03.050 INFO:tasks.rados.rados.0.smithi071.stdout:2097: finishing write tid 4 to smithi07134777-14 2024-09-11T01:43:03.050 INFO:tasks.rados.rados.0.smithi071.stdout:2097: oid 14 updating version 0 to 1210 2024-09-11T01:43:03.050 INFO:tasks.rados.rados.0.smithi071.stdout:2097: oid 14 version 1210 is already newer than 1208 2024-09-11T01:43:03.050 INFO:tasks.rados.rados.0.smithi071.stdout:2097: oid 14 version 1210 is already newer than 1209 2024-09-11T01:43:03.050 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 1210 (ObjNum 758 snap 230 seq_num 758) dirty exists 2024-09-11T01:43:03.050 INFO:tasks.rados.rados.0.smithi071.stdout:2097: left oid 14 (ObjNum 758 snap 230 seq_num 758) 2024-09-11T01:43:03.050 INFO:tasks.rados.rados.0.smithi071.stdout:2098: finishing rollback tid 1 to smithi07134777-13 2024-09-11T01:43:03.050 INFO:tasks.rados.rados.0.smithi071.stdout:2098: finishing rollback tid 2 to smithi07134777-13 2024-09-11T01:43:03.050 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 858 (ObjNum 551 snap 157 seq_num 551) dirty exists 2024-09-11T01:43:03.051 INFO:tasks.rados.rados.0.smithi071.stdout:2097: done (4 left) 2024-09-11T01:43:03.051 INFO:tasks.rados.rados.0.smithi071.stdout:2098: done (3 left) 2024-09-11T01:43:03.051 INFO:tasks.rados.rados.0.smithi071.stdout:2101: done (2 left) 2024-09-11T01:43:03.051 INFO:tasks.rados.rados.0.smithi071.stdout:2102: rollback oid 15 current snap is 230 2024-09-11T01:43:03.051 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 15 to 225 2024-09-11T01:43:03.051 INFO:tasks.rados.rados.0.smithi071.stdout:2103: copy_from oid 31 from oid 5 current snap is 230 2024-09-11T01:43:03.052 INFO:tasks.rados.rados.0.smithi071.stdout:2104: snap_create 2024-09-11T01:43:03.052 INFO:tasks.rados.rados.0.smithi071.stdout:2103: finishing copy_from to smithi07134777-31 2024-09-11T01:43:03.052 INFO:tasks.rados.rados.0.smithi071.stdout:2103: got expected ENOENT (src dne) 2024-09-11T01:43:03.052 INFO:tasks.rados.rados.0.smithi071.stdout:2103: finishing copy_from racing read to smithi07134777-31 2024-09-11T01:43:03.052 INFO:tasks.rados.rados.0.smithi071.stdout:2099: expect (ObjNum 738 snap 226 seq_num 738) 2024-09-11T01:43:03.052 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:03.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.133+0000 7f771948c640 10 monclient: get_monmap_and_config 2024-09-11T01:43:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.133+0000 7f771948c640 10 monclient: build_initial_monmap 2024-09-11T01:43:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.133+0000 7f771948c640 10 monclient: monmap: 2024-09-11T01:43:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:43:03.133842+0000 2024-09-11T01:43:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:43:03.133842+0000 2024-09-11T01:43:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:43:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:43:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:43:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:43:03.135 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:43:03.135 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:03.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.133+0000 7f771948c640 1 Processor -- start 2024-09-11T01:43:03.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.133+0000 7f771948c640 1 -- start start 2024-09-11T01:43:03.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.133+0000 7f771948c640 10 monclient: init 2024-09-11T01:43:03.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f771948c640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:03.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f771948c640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-11T01:43:03.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f771948c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7714137d50 0x7f771409d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:03.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f771948c640 10 monclient(hunting): picked mon.noname-c con 0x7f7714137d50 addr v2:172.21.15.71:3300/0 2024-09-11T01:43:03.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f771948c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7714138130 addr v1:172.21.15.71:6789/0 2024-09-11T01:43:03.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f771948c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f771413d280 0x7f771413f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:03.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f771948c640 10 monclient(hunting): picked mon.noname-b con 0x7f771413d280 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:03.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f771948c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f771409ba20 con 0x7f7714138130 2024-09-11T01:43:03.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f771948c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:03.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f771948c640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f771409c650 con 0x7f7714137d50 2024-09-11T01:43:03.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f771948c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:03.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f771948c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f771409c370 con 0x7f771413d280 2024-09-11T01:43:03.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f771948c640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:03.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f771948c640 10 monclient(hunting): authenticate will time out at 1589.576263s 2024-09-11T01:43:03.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f77137fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f771413d280 0x7f771413f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:03.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f77137fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f771413d280 0x7f771413f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:60622/0 (socket says 172.21.15.12:60622) 2024-09-11T01:43:03.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f77137fe640 1 -- 172.21.15.12:0/271475268 learned_addr learned my addr 172.21.15.12:0/271475268 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:03.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f77137fe640 10 monclient(hunting): get_auth_request con 0x7f771413d280 auth_method 0 2024-09-11T01:43:03.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f77137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:03.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f77137fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:03.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f77137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:03.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f7712ffd640 1 --2- 172.21.15.12:0/271475268 >> v2:172.21.15.71:3300/0 conn(0x7f7714137d50 0x7f771409d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:03.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f7712ffd640 10 monclient(hunting): get_auth_request con 0x7f7714137d50 auth_method 0 2024-09-11T01:43:03.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f7712ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:03.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f7712ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:03.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f7712ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:03.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f77137fe640 1 -- 172.21.15.12:0/271475268 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f771413d280 msgr2=0x7f771413f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:43:03.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f77137fe640 1 -- 172.21.15.12:0/271475268 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f771413d280 msgr2=0x7f771413f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:43:03.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f77137fe640 1 --2- 172.21.15.12:0/271475268 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f771413d280 0x7f771413f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:03.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f7712ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:03.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f7712ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:03.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f77137fe640 1 --2- 172.21.15.12:0/271475268 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f771413d280 0x7f771413f670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:03.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f7712ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:03.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f7712ffd640 10 monclient(hunting): handle_auth_done global_id 134278 payload 274 2024-09-11T01:43:03.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f7712ffd640 1 -- 172.21.15.12:0/271475268 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f771413d280 msgr2=0x7f771413f670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:43:03.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f7712ffd640 1 --2- 172.21.15.12:0/271475268 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f771413d280 0x7f771413f670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:03.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f7712ffd640 1 -- 172.21.15.12:0/271475268 >> v1:172.21.15.71:6789/0 conn(0x7f7714138130 legacy=0x7f7714138580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:03.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f7712ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:43:03.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f7712ffd640 1 monclient: found mon.noname-c 2024-09-11T01:43:03.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f7712ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:03.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f7712ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:43:03.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.134+0000 7f7712ffd640 1 -- 172.21.15.12:0/271475268 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77140a6ce0 con 0x7f7714137d50 2024-09-11T01:43:03.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.137+0000 7f7712ffd640 1 -- 172.21.15.12:0/271475268 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7714137d50 msgr2=0x7f771409d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:03.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.137+0000 7f7712ffd640 1 --2- 172.21.15.12:0/271475268 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7714137d50 0x7f771409d110 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f770c0097a0 tx=0x7f770c02a050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98ce613d476cfc90 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:03.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.137+0000 7f7711ffb640 1 -- 172.21.15.12:0/271475268 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c037070 con 0x7f7714137d50 2024-09-11T01:43:03.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.137+0000 7f7711ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:03.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.137+0000 7f7711ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:43:03.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.137+0000 7f7711ffb640 10 monclient: dump: 2024-09-11T01:43:03.147 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:03.147 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:03.147 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:03.147 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:03.148 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:03.148 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:03.148 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:03.148 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:03.148 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:03.148 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:03.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.137+0000 7f7711ffb640 10 monclient: _finish_auth 0 2024-09-11T01:43:03.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.137+0000 7f7711ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:43:03.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.137+0000 7f7711ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:03.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.137+0000 7f771948c640 5 monclient: authenticate success, global_id 134278 2024-09-11T01:43:03.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.137+0000 7f7711ffb640 1 -- 172.21.15.12:0/271475268 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f770c032440 con 0x7f7714137d50 2024-09-11T01:43:03.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.137+0000 7f7711ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:03.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.137+0000 7f7711ffb640 1 -- 172.21.15.12:0/271475268 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c032810 con 0x7f7714137d50 2024-09-11T01:43:03.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.137+0000 7f7711ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:03.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.137+0000 7f7711ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:43:03.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.137+0000 7f7711ffb640 10 monclient: dump: 2024-09-11T01:43:03.149 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:03.149 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:03.149 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:03.149 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:03.149 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:03.149 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:03.149 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:03.149 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:03.150 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:03.150 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:03.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.137+0000 7f771948c640 10 monclient: get_monmap_and_config success 2024-09-11T01:43:03.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.137+0000 7f771948c640 4 set_mon_vals no callback set 2024-09-11T01:43:03.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.138+0000 7f771948c640 10 monclient: shutdown 2024-09-11T01:43:03.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.138+0000 7f771948c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:03.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.138+0000 7f771948c640 1 -- 172.21.15.12:0/271475268 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7714137d50 msgr2=0x7f771409d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:03.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.138+0000 7f771948c640 1 --2- 172.21.15.12:0/271475268 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7714137d50 0x7f771409d110 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f770c0097a0 tx=0x7f770c02a050 comp rx=0 tx=0).stop 2024-09-11T01:43:03.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.138+0000 7f771948c640 1 -- 172.21.15.12:0/271475268 shutdown_connections 2024-09-11T01:43:03.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.138+0000 7f771948c640 1 --2- 172.21.15.12:0/271475268 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f771413d280 0x7f771413f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:03.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.138+0000 7f771948c640 1 --2- 172.21.15.12:0/271475268 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7714137d50 0x7f771409d110 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:03.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.138+0000 7f771948c640 1 -- 172.21.15.12:0/271475268 >> 172.21.15.12:0/271475268 conn(0x7f77140a7020 msgr2=0x7f77140a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:03.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.138+0000 7f771948c640 1 -- 172.21.15.12:0/271475268 shutdown_connections 2024-09-11T01:43:03.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.138+0000 7f771948c640 1 -- 172.21.15.12:0/271475268 wait complete. 2024-09-11T01:43:03.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.138+0000 7f771948c640 10 monclient: build_initial_monmap 2024-09-11T01:43:03.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.138+0000 7f771948c640 10 monclient: monmap: 2024-09-11T01:43:03.151 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:03.151 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:43:03.151 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:43:03.151 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:43:03.151 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:03.151 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:03.151 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:43:03.151 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:43:03.151 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:43:03.151 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:03.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.139+0000 7f771948c640 1 Processor -- start 2024-09-11T01:43:03.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.139+0000 7f771948c640 1 -- start start 2024-09-11T01:43:03.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.139+0000 7f771948c640 10 monclient: init 2024-09-11T01:43:03.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.139+0000 7f771948c640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:03.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.139+0000 7f771948c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:43:03.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.139+0000 7f771948c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7714137d50 0x7f77141cce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:03.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.139+0000 7f771948c640 10 monclient(hunting): picked mon.noname-a con 0x7f7714137d50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:03.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.139+0000 7f771948c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7714138130 0x7f7714136ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:03.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.139+0000 7f771948c640 10 monclient(hunting): picked mon.noname-c con 0x7f7714138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:03.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.139+0000 7f771948c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f771413d280 0x7f771413a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:03.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.139+0000 7f771948c640 10 monclient(hunting): picked mon.noname-b con 0x7f771413d280 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:03.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.139+0000 7f771948c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:03.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.139+0000 7f771948c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f771409d7d0 con 0x7f7714137d50 2024-09-11T01:43:03.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.139+0000 7f771948c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:03.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.139+0000 7f771948c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f771409d650 con 0x7f771413d280 2024-09-11T01:43:03.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.139+0000 7f771948c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:03.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.139+0000 7f771948c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f771413fbb0 con 0x7f7714138130 2024-09-11T01:43:03.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.139+0000 7f771948c640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:03.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.139+0000 7f771948c640 10 monclient(hunting): authenticate will time out at 1589.581975s 2024-09-11T01:43:03.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f7712ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7714137d50 0x7f77141cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:03.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f77127fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7714138130 0x7f7714136ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:03.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f7712ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7714137d50 0x7f77141cce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45022/0 (socket says 172.21.15.12:45022) 2024-09-11T01:43:03.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f77127fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7714138130 0x7f7714136ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:60636/0 (socket says 172.21.15.12:60636) 2024-09-11T01:43:03.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f7712ffd640 1 -- 172.21.15.12:0/1790813135 learned_addr learned my addr 172.21.15.12:0/1790813135 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:03.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f7712ffd640 10 monclient(hunting): get_auth_request con 0x7f7714137d50 auth_method 0 2024-09-11T01:43:03.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f7712ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:03.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f7712ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:03.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f7712ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:03.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f77127fc640 10 monclient(hunting): get_auth_request con 0x7f7714138130 auth_method 0 2024-09-11T01:43:03.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f77127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:03.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f77127fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:03.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f77127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:03.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f77137fe640 1 --2- 172.21.15.12:0/1790813135 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f771413d280 0x7f771413a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:03.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f77137fe640 10 monclient(hunting): get_auth_request con 0x7f771413d280 auth_method 0 2024-09-11T01:43:03.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f77137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:03.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f77137fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:03.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f77137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:03.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f7712ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:03.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f7712ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:03.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f7712ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:03.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f77127fc640 1 -- 172.21.15.12:0/1790813135 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7714138130 msgr2=0x7f7714136ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:43:03.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f77127fc640 1 -- 172.21.15.12:0/1790813135 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7714138130 msgr2=0x7f7714136ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:43:03.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f77127fc640 1 --2- 172.21.15.12:0/1790813135 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7714138130 0x7f7714136ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:03.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f77137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:03.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f77137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:03.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f77137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:03.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f77127fc640 1 --2- 172.21.15.12:0/1790813135 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7714138130 0x7f7714136ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:03.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f7712ffd640 10 monclient(hunting): handle_auth_done global_id 94659 payload 824 2024-09-11T01:43:03.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f7712ffd640 1 -- 172.21.15.12:0/1790813135 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7714138130 msgr2=0x7f7714136ed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:43:03.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f7712ffd640 1 --2- 172.21.15.12:0/1790813135 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7714138130 0x7f7714136ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:03.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f7712ffd640 1 -- 172.21.15.12:0/1790813135 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f771413d280 msgr2=0x7f771413a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:03.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f7712ffd640 1 --2- 172.21.15.12:0/1790813135 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f771413d280 0x7f771413a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:03.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f7712ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:43:03.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f7712ffd640 1 monclient: found mon.noname-a 2024-09-11T01:43:03.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f7712ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:03.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f7712ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:03.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.140+0000 7f7712ffd640 1 -- 172.21.15.12:0/1790813135 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77140a1710 con 0x7f7714137d50 2024-09-11T01:43:03.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.142+0000 7f7712ffd640 1 -- 172.21.15.12:0/1790813135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7714137d50 msgr2=0x7f77141cce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:03.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.142+0000 7f7712ffd640 1 --2- 172.21.15.12:0/1790813135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7714137d50 0x7f77141cce90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f770c000c00 tx=0x7f770c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:03.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.142+0000 7f770b7fe640 1 -- 172.21.15.12:0/1790813135 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c041020 con 0x7f7714137d50 2024-09-11T01:43:03.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.142+0000 7f770b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:03.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.142+0000 7f770b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:03.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.142+0000 7f770b7fe640 10 monclient: dump: 2024-09-11T01:43:03.156 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:03.156 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:03.156 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:03.156 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:03.156 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:03.156 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:03.156 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:03.157 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:03.157 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:03.157 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:03.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.142+0000 7f770b7fe640 10 monclient: _finish_auth 0 2024-09-11T01:43:03.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.142+0000 7f770b7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:43:03.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.142+0000 7f770b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:03.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.142+0000 7f770b7fe640 1 -- 172.21.15.12:0/1790813135 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f770c03d8f0 con 0x7f7714137d50 2024-09-11T01:43:03.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.142+0000 7f771948c640 5 monclient: authenticate success, global_id 94659 2024-09-11T01:43:03.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.142+0000 7f771948c640 10 monclient: _renew_subs 2024-09-11T01:43:03.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.142+0000 7f771948c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:03.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.142+0000 7f771948c640 1 -- 172.21.15.12:0/1790813135 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f771413bbe0 con 0x7f7714137d50 2024-09-11T01:43:03.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.142+0000 7f770b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:03.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.142+0000 7f770b7fe640 1 -- 172.21.15.12:0/1790813135 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c037040 con 0x7f7714137d50 2024-09-11T01:43:03.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.142+0000 7f770b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:03.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.142+0000 7f770b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:03.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.142+0000 7f770b7fe640 10 monclient: dump: 2024-09-11T01:43:03.158 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:03.158 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:03.158 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:03.158 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:03.158 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:03.158 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:03.158 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:03.158 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:03.158 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:03.158 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:03.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.142+0000 7f77117fa640 4 set_mon_vals no callback set 2024-09-11T01:43:03.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.142+0000 7f771948c640 10 monclient: _renew_subs 2024-09-11T01:43:03.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.142+0000 7f771948c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:03.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.142+0000 7f771948c640 1 -- 172.21.15.12:0/1790813135 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f771413c740 con 0x7f7714137d50 2024-09-11T01:43:03.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.143+0000 7f770b7fe640 1 -- 172.21.15.12:0/1790813135 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f770c03a030 con 0x7f7714137d50 2024-09-11T01:43:03.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.143+0000 7f771948c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:03.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.143+0000 7f771948c640 1 --2- 172.21.15.12:0/1790813135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7714137410 0x7f771413a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:03.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.143+0000 7f771948c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:03.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.143+0000 7f771948c640 1 -- 172.21.15.12:0/1790813135 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f771409d110 con 0x7f7714137410 2024-09-11T01:43:03.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.144+0000 7f77127fc640 1 --2- 172.21.15.12:0/1790813135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7714137410 0x7f771413a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:03.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.144+0000 7f77127fc640 10 monclient: get_auth_request con 0x7f7714137410 auth_method 0 2024-09-11T01:43:03.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.144+0000 7f77127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:03.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.144+0000 7f77127fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:03.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.144+0000 7f77127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:03.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.144+0000 7f770b7fe640 1 --2- 172.21.15.12:0/1790813135 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f76f403e240 0x7f76f4040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:03.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.144+0000 7f77127fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:03.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.144+0000 7f77127fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:03.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.144+0000 7f77127fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:03.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.144+0000 7f770b7fe640 1 -- 172.21.15.12:0/1790813135 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(470..470 src has 285..470) ==== 4803+0+0 (secure 0 0 0) 0x7f770c073330 con 0x7f7714137d50 2024-09-11T01:43:03.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.144+0000 7f77137fe640 1 --2- 172.21.15.12:0/1790813135 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f76f403e240 0x7f76f4040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:03.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.144+0000 7f77127fc640 10 monclient(hunting): handle_auth_done global_id 94662 payload 824 2024-09-11T01:43:03.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.144+0000 7f77127fc640 1 -- 172.21.15.12:0/1790813135 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7714121070 con 0x7f7714137410 2024-09-11T01:43:03.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.144+0000 7f77137fe640 10 monclient: get_auth_request con 0x7f76f403e240 auth_method 0 2024-09-11T01:43:03.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.146+0000 7f77127fc640 1 -- 172.21.15.12:0/1790813135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7714137410 msgr2=0x7f771413a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:03.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.146+0000 7f77127fc640 1 --2- 172.21.15.12:0/1790813135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7714137410 0x7f771413a660 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f76fc007c50 tx=0x7f76fc00e8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:03.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.146+0000 7f770b7fe640 1 -- 172.21.15.12:0/1790813135 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76fc007450 con 0x7f7714137410 2024-09-11T01:43:03.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.146+0000 7f770b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:03.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.146+0000 7f770b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:03.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.146+0000 7f770b7fe640 10 monclient: dump: 2024-09-11T01:43:03.163 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:03.163 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:03.163 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:03.163 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:03.163 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:03.163 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:03.163 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:03.163 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:03.163 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:03.164 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:03.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.146+0000 7f77137fe640 1 --2- 172.21.15.12:0/1790813135 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f76f403e240 0x7f76f4040700 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f771406fbb0 tx=0x7f7704009130 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:03.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.146+0000 7f770b7fe640 1 -- 172.21.15.12:0/1790813135 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f76fc007450 con 0x7f7714137410 2024-09-11T01:43:03.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.146+0000 7f770b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:03.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.146+0000 7f770b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:43:03.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.146+0000 7f770b7fe640 1 -- 172.21.15.12:0/1790813135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7714137410 msgr2=0x7f771413a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:03.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.146+0000 7f770b7fe640 1 --2- 172.21.15.12:0/1790813135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7714137410 0x7f771413a660 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f76fc007c50 tx=0x7f76fc00e8b0 comp rx=0 tx=0).stop 2024-09-11T01:43:03.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.146+0000 7f770b7fe640 1 -- 172.21.15.12:0/1790813135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7714137410 msgr2=0x7f771413a660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:43:03.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.146+0000 7f770b7fe640 1 --2- 172.21.15.12:0/1790813135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7714137410 0x7f771413a660 secure :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f76fc007c50 tx=0x7f76fc00e8b0 comp rx=0 tx=0).stop 2024-09-11T01:43:03.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.153+0000 7f771948c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:43:03.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.153+0000 7f771948c640 1 --2- 172.21.15.12:0/1790813135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f771413aba0 0x7f77140a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:03.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.153+0000 7f771948c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:03.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.153+0000 7f771948c640 1 -- 172.21.15.12:0/1790813135 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f771413af80 con 0x7f771413aba0 2024-09-11T01:43:03.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.153+0000 7f77127fc640 1 --2- 172.21.15.12:0/1790813135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f771413aba0 0x7f77140a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:03.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.153+0000 7f77127fc640 10 monclient: get_auth_request con 0x7f771413aba0 auth_method 0 2024-09-11T01:43:03.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.153+0000 7f77127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:03.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.153+0000 7f77127fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:03.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.153+0000 7f77127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:03.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.153+0000 7f77127fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:03.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.153+0000 7f77127fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:03.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.153+0000 7f77127fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:03.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.153+0000 7f77127fc640 10 monclient(hunting): handle_auth_done global_id 94665 payload 824 2024-09-11T01:43:03.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.153+0000 7f77127fc640 1 -- 172.21.15.12:0/1790813135 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f77141214e0 con 0x7f771413aba0 2024-09-11T01:43:03.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.155+0000 7f77127fc640 1 -- 172.21.15.12:0/1790813135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f771413aba0 msgr2=0x7f77140a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:03.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.155+0000 7f77127fc640 1 --2- 172.21.15.12:0/1790813135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f771413aba0 0x7f77140a1920 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f76fc018000 tx=0x7f76fc003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:03.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.155+0000 7f770b7fe640 1 -- 172.21.15.12:0/1790813135 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76fc01eb70 con 0x7f771413aba0 2024-09-11T01:43:03.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.155+0000 7f770b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:03.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.155+0000 7f770b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:03.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.155+0000 7f770b7fe640 10 monclient: dump: 2024-09-11T01:43:03.166 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:03.166 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:03.166 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:03.166 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:03.166 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:03.166 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:03.166 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:03.166 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:03.166 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:03.166 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:03.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.155+0000 7f770b7fe640 1 -- 172.21.15.12:0/1790813135 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f76fc01eb70 con 0x7f771413aba0 2024-09-11T01:43:03.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.155+0000 7f770b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:43:03.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.155+0000 7f770b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:43:03.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.155+0000 7f770b7fe640 1 -- 172.21.15.12:0/1790813135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f771413aba0 msgr2=0x7f77140a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:03.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.155+0000 7f770b7fe640 1 --2- 172.21.15.12:0/1790813135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f771413aba0 0x7f77140a1920 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f76fc018000 tx=0x7f76fc003760 comp rx=0 tx=0).stop 2024-09-11T01:43:03.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.156+0000 7f770b7fe640 1 -- 172.21.15.12:0/1790813135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f771413aba0 msgr2=0x7f77140a1920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:43:03.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.156+0000 7f770b7fe640 1 --2- 172.21.15.12:0/1790813135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f771413aba0 0x7f77140a1920 secure :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f76fc018000 tx=0x7f76fc003760 comp rx=0 tx=0).stop 2024-09-11T01:43:03.167 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:43:03.167 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:43:03.167 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:43:03.168 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:43:03.168 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 128, 2024-09-11T01:43:03.168 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 329251, 2024-09-11T01:43:03.168 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:43:03.168 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:43:03.168 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:43:03.168 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:43:03.168 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:03.168 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:43:03.168 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:43:03.168 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:43:03.168 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:43:03.168 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:03.169 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:03.169 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:03.169 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:03.169 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:03.169 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:03.169 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:03.169 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:03.169 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:03.169 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:03.169 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:03.169 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:03.169 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:43:03.169 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:43:03.169 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:03.170 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:03.170 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:03.170 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:03.170 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:03.170 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:03.170 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:03.170 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:03.170 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:03.170 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:03.170 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:03.170 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:03.170 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:03.170 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:43:03.171 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:43:03.171 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:43:03.171 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:43:03.171 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:43:03.171 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:43:03.171 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:43:03.171 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:43:03.171 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:43:03.171 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:43:03.171 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:43:03.171 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:43:03.171 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:43:03.171 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:43:03.172 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:43:03.172 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:43:03.172 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:43:03.172 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:03.172 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:03.172 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:03.172 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:03.172 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:03.172 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:03.172 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:03.172 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:03.172 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:03.172 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:03.172 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:03.173 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:03.173 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:43:03.173 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:03.173 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:43:03.173 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:03.173 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:43:03.173 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:43:03.173 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:03.173 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:03.173 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:03.173 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:03.173 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:43:03.174 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:03.174 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:03.174 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:03.174 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:03.174 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:43:03.174 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:03.174 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:03.174 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:03.174 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:03.174 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:43:03.174 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:43:03.174 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:03.174 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:03.175 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:03.175 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:03.175 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:03.175 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:43:03.175 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:43:03.175 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:03.175 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:03.175 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:03.175 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:03.175 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:43:03.175 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:03.175 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:03.175 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:03.175 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:03.175 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:43:03.176 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:03.176 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:03.176 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:03.176 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:03.176 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:43:03.176 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:43:03.176 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:03.176 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:03.176 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:03.176 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:03.176 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:03.176 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:43:03.176 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:43:03.177 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:03.177 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:03.177 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:03.177 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:03.177 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:43:03.177 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:03.177 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:03.177 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:03.177 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:03.177 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:43:03.177 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:03.177 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:03.177 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:03.177 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:03.178 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:43:03.178 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:43:03.178 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:03.178 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:03.178 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:03.178 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:03.178 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:03.178 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:03.178 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:43:03.178 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:43:03.178 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:03.178 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:03.178 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:03.178 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:43:03.179 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:03.179 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:03.179 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:43:03.179 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:03.179 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:03.179 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:03.179 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:43:03.179 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:03.179 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:03.179 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:43:03.179 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:03.179 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:03.179 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:03.179 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:43:03.179 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:03.180 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:03.180 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:03.180 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:43:03.180 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:43:03.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.156+0000 7f771948c640 1 -- 172.21.15.12:0/1790813135 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f76f403e240 msgr2=0x7f76f4040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:03.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.156+0000 7f771948c640 1 --2- 172.21.15.12:0/1790813135 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f76f403e240 0x7f76f4040700 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f771406fbb0 tx=0x7f7704009130 comp rx=0 tx=0).stop 2024-09-11T01:43:03.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.156+0000 7f771948c640 10 monclient: shutdown 2024-09-11T01:43:03.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.156+0000 7f771948c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:03.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.156+0000 7f771948c640 1 -- 172.21.15.12:0/1790813135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7714137d50 msgr2=0x7f77141cce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:03.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.156+0000 7f771948c640 1 --2- 172.21.15.12:0/1790813135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7714137d50 0x7f77141cce90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f770c000c00 tx=0x7f770c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:43:03.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.156+0000 7f77137fe640 1 -- 172.21.15.12:0/1790813135 reap_dead start 2024-09-11T01:43:03.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.156+0000 7f77137fe640 1 -- 172.21.15.12:0/1790813135 reap_dead start 2024-09-11T01:43:03.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.156+0000 7f771948c640 1 -- 172.21.15.12:0/1790813135 shutdown_connections 2024-09-11T01:43:03.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.156+0000 7f771948c640 1 -- 172.21.15.12:0/1790813135 >> 172.21.15.12:0/1790813135 conn(0x7f77140a7020 msgr2=0x7f77140a0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:03.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.156+0000 7f771948c640 1 -- 172.21.15.12:0/1790813135 shutdown_connections 2024-09-11T01:43:03.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.156+0000 7f771948c640 1 -- 172.21.15.12:0/1790813135 wait complete. 2024-09-11T01:43:03.182 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:43:03.182 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:43:03.182 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:43:03.213 INFO:tasks.rados.rados.0.smithi071.stdout:2102: finishing rollback tid 0 to smithi07134777-15 2024-09-11T01:43:03.213 INFO:tasks.rados.rados.0.smithi071.stdout:2102: finishing rollback tid 1 to smithi07134777-15 2024-09-11T01:43:03.213 INFO:tasks.rados.rados.0.smithi071.stdout:2102: finishing rollback tid 2 to smithi07134777-15 2024-09-11T01:43:03.213 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 1214 (ObjNum 565 snap 160 seq_num 565) dirty exists 2024-09-11T01:43:03.213 INFO:tasks.rados.rados.0.smithi071.stdout:2100: expect (ObjNum 666 snap 202 seq_num 666) 2024-09-11T01:43:03.274 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:03.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.283+0000 7f1755ced640 10 monclient: get_monmap_and_config 2024-09-11T01:43:03.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.283+0000 7f1755ced640 10 monclient: build_initial_monmap 2024-09-11T01:43:03.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.283+0000 7f1755ced640 10 monclient: monmap: 2024-09-11T01:43:03.285 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:03.285 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:03.285 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:43:03.284068+0000 2024-09-11T01:43:03.285 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:43:03.284068+0000 2024-09-11T01:43:03.285 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:03.285 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:03.285 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:43:03.285 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:43:03.285 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:43:03.285 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:43:03.285 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:43:03.285 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:03.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.284+0000 7f1755ced640 1 Processor -- start 2024-09-11T01:43:03.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.284+0000 7f1755ced640 1 -- start start 2024-09-11T01:43:03.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.284+0000 7f1755ced640 10 monclient: init 2024-09-11T01:43:03.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.284+0000 7f1755ced640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:03.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.284+0000 7f1755ced640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-11T01:43:03.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.284+0000 7f1755ced640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f17500a1380 0x7f17500a1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:03.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.284+0000 7f1755ced640 10 monclient(hunting): picked mon.noname-a con 0x7f17500a1380 addr v2:172.21.15.12:3300/0 2024-09-11T01:43:03.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.284+0000 7f1755ced640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f17500a0370 addr v1:172.21.15.71:6789/0 2024-09-11T01:43:03.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.284+0000 7f1755ced640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f17500a0c90 0x7f17501361f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:03.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.284+0000 7f1755ced640 10 monclient(hunting): picked mon.noname-b con 0x7f17500a0c90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:03.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.284+0000 7f1755ced640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17500a1ca0 con 0x7f17500a0370 2024-09-11T01:43:03.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.284+0000 7f1755ced640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:03.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.284+0000 7f1755ced640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f17500a21c0 con 0x7f17500a1380 2024-09-11T01:43:03.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.284+0000 7f1755ced640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:03.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.284+0000 7f1755ced640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f17500a24a0 con 0x7f17500a0c90 2024-09-11T01:43:03.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.284+0000 7f1755ced640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:03.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.284+0000 7f1755ced640 10 monclient(hunting): authenticate will time out at 1589.726893s 2024-09-11T01:43:03.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.284+0000 7f174f7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f17500a1380 0x7f17500a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:03.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f17500a0c90 0x7f17501361f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:03.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174f7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f17500a1380 0x7f17500a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45048/0 (socket says 172.21.15.12:45048) 2024-09-11T01:43:03.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174f7fe640 1 -- 172.21.15.12:0/3225992605 learned_addr learned my addr 172.21.15.12:0/3225992605 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:03.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f17500a0c90 0x7f17501361f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:60650/0 (socket says 172.21.15.12:60650) 2024-09-11T01:43:03.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174f7fe640 10 monclient(hunting): get_auth_request con 0x7f17500a1380 auth_method 0 2024-09-11T01:43:03.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:03.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:03.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:03.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174ffff640 10 monclient(hunting): get_auth_request con 0x7f17500a0c90 auth_method 0 2024-09-11T01:43:03.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:03.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:03.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:03.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:03.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:03.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:03.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174ffff640 1 -- 172.21.15.12:0/3225992605 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f17500a0c90 msgr2=0x7f17501361f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:43:03.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174ffff640 1 -- 172.21.15.12:0/3225992605 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f17500a0c90 msgr2=0x7f17501361f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:43:03.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174ffff640 1 --2- 172.21.15.12:0/3225992605 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f17500a0c90 0x7f17501361f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:03.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174ffff640 1 --2- 172.21.15.12:0/3225992605 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f17500a0c90 0x7f17501361f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:03.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174f7fe640 10 monclient(hunting): handle_auth_done global_id 94668 payload 274 2024-09-11T01:43:03.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174f7fe640 1 -- 172.21.15.12:0/3225992605 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f17500a0c90 msgr2=0x7f17501361f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:43:03.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174f7fe640 1 --2- 172.21.15.12:0/3225992605 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f17500a0c90 0x7f17501361f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:03.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174f7fe640 1 -- 172.21.15.12:0/3225992605 >> v1:172.21.15.71:6789/0 conn(0x7f17500a0370 legacy=0x7f1750135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:03.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174f7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:43:03.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174f7fe640 1 monclient: found mon.noname-a 2024-09-11T01:43:03.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:03.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:03.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.285+0000 7f174f7fe640 1 -- 172.21.15.12:0/3225992605 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17500a2780 con 0x7f17500a1380 2024-09-11T01:43:03.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f174f7fe640 1 -- 172.21.15.12:0/3225992605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17500a1380 msgr2=0x7f17500a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:03.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f174f7fe640 1 --2- 172.21.15.12:0/3225992605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17500a1380 0x7f17500a1760 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1740009870 tx=0x7f174002a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ad030fac94103e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:03.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f174effd640 1 -- 172.21.15.12:0/3225992605 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1740037070 con 0x7f17500a1380 2024-09-11T01:43:03.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f174effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:03.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f174effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:03.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f174effd640 10 monclient: dump: 2024-09-11T01:43:03.298 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:03.298 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:03.298 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:03.298 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:03.298 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:03.298 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:03.298 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:03.298 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:03.298 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:03.298 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:03.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f174effd640 10 monclient: _finish_auth 0 2024-09-11T01:43:03.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f174effd640 10 monclient: _check_auth_tickets 2024-09-11T01:43:03.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f174effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:03.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f174effd640 1 -- 172.21.15.12:0/3225992605 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1740032440 con 0x7f17500a1380 2024-09-11T01:43:03.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f1755ced640 5 monclient: authenticate success, global_id 94668 2024-09-11T01:43:03.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f1755ced640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:43:03.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f174effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:03.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f174effd640 1 -- 172.21.15.12:0/3225992605 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1740032810 con 0x7f17500a1380 2024-09-11T01:43:03.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f174effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:03.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f174effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:03.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f174effd640 10 monclient: dump: 2024-09-11T01:43:03.299 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:03.299 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:03.299 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:03.299 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:03.299 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:03.299 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:03.300 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:03.300 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:03.300 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:03.300 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:03.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f1755ced640 10 monclient: get_monmap_and_config success 2024-09-11T01:43:03.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f1755ced640 4 set_mon_vals no callback set 2024-09-11T01:43:03.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f1755ced640 10 monclient: shutdown 2024-09-11T01:43:03.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f1755ced640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:03.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f1755ced640 1 -- 172.21.15.12:0/3225992605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17500a1380 msgr2=0x7f17500a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:03.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f1755ced640 1 --2- 172.21.15.12:0/3225992605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17500a1380 0x7f17500a1760 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1740009870 tx=0x7f174002a0a0 comp rx=0 tx=0).stop 2024-09-11T01:43:03.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f1755ced640 1 -- 172.21.15.12:0/3225992605 shutdown_connections 2024-09-11T01:43:03.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f1755ced640 1 --2- 172.21.15.12:0/3225992605 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f17500a0c90 0x7f17501361f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:03.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f1755ced640 1 --2- 172.21.15.12:0/3225992605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17500a1380 0x7f17500a1760 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.288+0000 7f1755ced640 1 -- 172.21.15.12:0/3225992605 >> 172.21.15.12:0/3225992605 conn(0x7f17500a2f70 msgr2=0x7f17500a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.289+0000 7f1755ced640 1 -- 172.21.15.12:0/3225992605 shutdown_connections 2024-09-11T01:43:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.289+0000 7f1755ced640 1 -- 172.21.15.12:0/3225992605 wait complete. 2024-09-11T01:43:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.289+0000 7f1755ced640 10 monclient: build_initial_monmap 2024-09-11T01:43:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.289+0000 7f1755ced640 10 monclient: monmap: 2024-09-11T01:43:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:43:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:43:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:43:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:43:03.301 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:43:03.302 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:43:03.302 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:03.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.289+0000 7f1755ced640 1 Processor -- start 2024-09-11T01:43:03.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.289+0000 7f1755ced640 1 -- start start 2024-09-11T01:43:03.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.289+0000 7f1755ced640 10 monclient: init 2024-09-11T01:43:03.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f1755ced640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:03.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f1755ced640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:43:03.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f1755ced640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f17500a0370 0x7f17501ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:03.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f1755ced640 10 monclient(hunting): picked mon.noname-c con 0x7f17500a0370 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:03.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f1755ced640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17500a0c90 0x7f1750135bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:03.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f1755ced640 10 monclient(hunting): picked mon.noname-b con 0x7f17500a0c90 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:03.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f1755ced640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17500a1380 0x7f1750136110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:03.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f1755ced640 10 monclient(hunting): picked mon.noname-a con 0x7f17500a1380 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:03.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f1755ced640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:03.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f1755ced640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f175009ce20 con 0x7f17500a1380 2024-09-11T01:43:03.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f1755ced640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:03.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f1755ced640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f175009cca0 con 0x7f17500a0c90 2024-09-11T01:43:03.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f1755ced640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:03.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f1755ced640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1750136730 con 0x7f17500a0370 2024-09-11T01:43:03.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f1755ced640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:03.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f1755ced640 10 monclient(hunting): authenticate will time out at 1589.732429s 2024-09-11T01:43:03.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174f7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f17500a0370 0x7f17501ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:03.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17500a1380 0x7f1750136110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:03.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174f7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f17500a0370 0x7f17501ccd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:60654/0 (socket says 172.21.15.12:60654) 2024-09-11T01:43:03.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174f7fe640 1 -- 172.21.15.12:0/2932903739 learned_addr learned my addr 172.21.15.12:0/2932903739 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:03.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174f7fe640 10 monclient(hunting): get_auth_request con 0x7f17500a0370 auth_method 0 2024-09-11T01:43:03.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:03.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:03.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:03.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174ffff640 10 monclient(hunting): get_auth_request con 0x7f17500a1380 auth_method 0 2024-09-11T01:43:03.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:03.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:03.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:03.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f1747fff640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17500a0c90 0x7f1750135bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:03.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174f7fe640 1 -- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f17500a0370 msgr2=0x7f17501ccd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:43:03.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174f7fe640 1 -- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f17500a0370 msgr2=0x7f17501ccd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:43:03.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174f7fe640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f17500a0370 0x7f17501ccd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:03.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:03.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:03.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:03.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f1747fff640 10 monclient(hunting): get_auth_request con 0x7f17500a0c90 auth_method 0 2024-09-11T01:43:03.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f1747fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:03.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f1747fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:03.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f1747fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:03.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174f7fe640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f17500a0370 0x7f17501ccd70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:03.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f1747fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:03.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f1747fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:03.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f1747fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:03.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174ffff640 10 monclient(hunting): handle_auth_done global_id 94671 payload 824 2024-09-11T01:43:03.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174ffff640 1 -- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f17500a0370 msgr2=0x7f17501ccd70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:43:03.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174ffff640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f17500a0370 0x7f17501ccd70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:03.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174ffff640 1 -- 172.21.15.12:0/2932903739 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17500a0c90 msgr2=0x7f1750135bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:03.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174ffff640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17500a0c90 0x7f1750135bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:03.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:43:03.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174ffff640 1 monclient: found mon.noname-a 2024-09-11T01:43:03.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:03.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:03.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.290+0000 7f174ffff640 1 -- 172.21.15.12:0/2932903739 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1750127df0 con 0x7f17500a1380 2024-09-11T01:43:03.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.293+0000 7f174ffff640 1 -- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17500a1380 msgr2=0x7f1750136110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:03.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.293+0000 7f174ffff640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17500a1380 0x7f1750136110 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1748009610 tx=0x7f174800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:03.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.293+0000 7f174cff9640 1 -- 172.21.15.12:0/2932903739 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1748003990 con 0x7f17500a1380 2024-09-11T01:43:03.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.293+0000 7f174cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:03.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.293+0000 7f174cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:03.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.293+0000 7f174cff9640 10 monclient: dump: 2024-09-11T01:43:03.306 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:03.306 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:03.306 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:03.306 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:03.306 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:03.307 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:03.307 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:03.307 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:03.307 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:03.307 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:03.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.293+0000 7f174cff9640 10 monclient: _finish_auth 0 2024-09-11T01:43:03.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.293+0000 7f174cff9640 10 monclient: _check_auth_tickets 2024-09-11T01:43:03.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.293+0000 7f174cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:03.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.293+0000 7f1755ced640 5 monclient: authenticate success, global_id 94671 2024-09-11T01:43:03.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.293+0000 7f174cff9640 1 -- 172.21.15.12:0/2932903739 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1748003b30 con 0x7f17500a1380 2024-09-11T01:43:03.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.293+0000 7f174cff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:03.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.293+0000 7f174cff9640 1 -- 172.21.15.12:0/2932903739 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1748004020 con 0x7f17500a1380 2024-09-11T01:43:03.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.293+0000 7f174cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:03.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.293+0000 7f174cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:03.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.293+0000 7f174cff9640 10 monclient: dump: 2024-09-11T01:43:03.307 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:03.308 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:03.308 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:03.308 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:03.308 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:03.308 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:03.308 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:03.308 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:03.308 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:03.308 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:03.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.293+0000 7f174e7fc640 4 set_mon_vals no callback set 2024-09-11T01:43:03.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.293+0000 7f1755ced640 10 monclient: _renew_subs 2024-09-11T01:43:03.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.293+0000 7f1755ced640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:03.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.293+0000 7f1755ced640 1 -- 172.21.15.12:0/2932903739 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1750129310 con 0x7f17500a1380 2024-09-11T01:43:03.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.293+0000 7f1755ced640 10 monclient: _renew_subs 2024-09-11T01:43:03.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.293+0000 7f1755ced640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:03.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.293+0000 7f1755ced640 1 -- 172.21.15.12:0/2932903739 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1750127290 con 0x7f17500a1380 2024-09-11T01:43:03.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.294+0000 7f1755ced640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:03.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.294+0000 7f1755ced640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 0x7f175002d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:03.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.294+0000 7f1755ced640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:03.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.294+0000 7f1755ced640 1 -- 172.21.15.12:0/2932903739 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f17500a1760 con 0x7f1750126050 2024-09-11T01:43:03.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.294+0000 7f174f7fe640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 0x7f175002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:03.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.294+0000 7f174cff9640 1 -- 172.21.15.12:0/2932903739 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f1748004280 con 0x7f17500a1380 2024-09-11T01:43:03.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.294+0000 7f174f7fe640 10 monclient: get_auth_request con 0x7f1750126050 auth_method 0 2024-09-11T01:43:03.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.294+0000 7f174f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:03.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.294+0000 7f174f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:03.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.294+0000 7f174f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:03.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.294+0000 7f174f7fe640 1 -- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 msgr2=0x7f175002d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:43:03.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.294+0000 7f174f7fe640 1 -- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 msgr2=0x7f175002d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:43:03.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.294+0000 7f174f7fe640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 0x7f175002d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:03.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.294+0000 7f174f7fe640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 0x7f175002d440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:03.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.294+0000 7f174cff9640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f173003e210 0x7f17300406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:03.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.294+0000 7f174cff9640 1 -- 172.21.15.12:0/2932903739 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(470..470 src has 285..470) ==== 4803+0+0 (secure 0 0 0) 0x7f1748053130 con 0x7f17500a1380 2024-09-11T01:43:03.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.294+0000 7f1747fff640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f173003e210 0x7f17300406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:03.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.294+0000 7f1747fff640 10 monclient: get_auth_request con 0x7f173003e210 auth_method 0 2024-09-11T01:43:03.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.297+0000 7f1747fff640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f173003e210 0x7f17300406d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f175006ecf0 tx=0x7f1738007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:03.344 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 798158 2024-09-11T01:43:03.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.495+0000 7f174f7fe640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 0x7f175002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:03.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.495+0000 7f174f7fe640 10 monclient: get_auth_request con 0x7f1750126050 auth_method 0 2024-09-11T01:43:03.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.495+0000 7f174f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:03.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.495+0000 7f174f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:03.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.495+0000 7f174f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:03.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.495+0000 7f174f7fe640 1 -- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 msgr2=0x7f175002d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:43:03.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.495+0000 7f174f7fe640 1 -- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 msgr2=0x7f175002d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:43:03.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.495+0000 7f174f7fe640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 0x7f175002d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:03.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.495+0000 7f174f7fe640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 0x7f175002d440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:43:03.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.897+0000 7f174f7fe640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 0x7f175002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:03.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.897+0000 7f174f7fe640 10 monclient: get_auth_request con 0x7f1750126050 auth_method 0 2024-09-11T01:43:03.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.897+0000 7f174f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:03.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.897+0000 7f174f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:03.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.897+0000 7f174f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:03.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.897+0000 7f174f7fe640 1 -- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 msgr2=0x7f175002d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:43:03.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.897+0000 7f174f7fe640 1 -- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 msgr2=0x7f175002d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:43:03.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.897+0000 7f174f7fe640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 0x7f175002d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:03.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:03.897+0000 7f174f7fe640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 0x7f175002d440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:43:03.972 INFO:tasks.rados.rados.0.smithi071.stdout:2099: done (4 left) 2024-09-11T01:43:03.972 INFO:tasks.rados.rados.0.smithi071.stdout:2100: done (3 left) 2024-09-11T01:43:03.972 INFO:tasks.rados.rados.0.smithi071.stdout:2102: done (2 left) 2024-09-11T01:43:03.972 INFO:tasks.rados.rados.0.smithi071.stdout:2103: done (1 left) 2024-09-11T01:43:03.973 INFO:tasks.rados.rados.0.smithi071.stdout:2104: done (0 left) 2024-09-11T01:43:03.973 INFO:tasks.rados.rados.0.smithi071.stdout:2105: read oid 44 snap -1 2024-09-11T01:43:03.973 INFO:tasks.rados.rados.0.smithi071.stdout:2105: expect (ObjNum 742 snap 226 seq_num 742) 2024-09-11T01:43:03.973 INFO:tasks.rados.rados.0.smithi071.stdout:2106: delete oid 47 current snap is 231 2024-09-11T01:43:03.974 INFO:tasks.rados.rados.0.smithi071.stdout:2106: done (1 left) 2024-09-11T01:43:03.974 INFO:tasks.rados.rados.0.smithi071.stdout:2107: delete oid 12 current snap is 231 2024-09-11T01:43:03.977 INFO:tasks.rados.rados.0.smithi071.stdout:2105: expect (ObjNum 742 snap 226 seq_num 742) 2024-09-11T01:43:04.032 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1475576 2024-09-11T01:43:04.106 INFO:tasks.rados.rados.0.smithi071.stdout:2105: done (1 left) 2024-09-11T01:43:04.106 INFO:tasks.rados.rados.0.smithi071.stdout:2107: done (0 left) 2024-09-11T01:43:04.106 INFO:tasks.rados.rados.0.smithi071.stdout:2108: write (excl) oid 47 current snap is 231 2024-09-11T01:43:04.106 INFO:tasks.rados.rados.0.smithi071.stdout:2108: seq_num 762 ranges {406934=575038,1640573=474593,2642791=164143} 2024-09-11T01:43:04.111 INFO:tasks.rados.rados.0.smithi071.stdout:2108: writing smithi07134777-47 from 406934 to 981972 tid 1 2024-09-11T01:43:04.117 INFO:tasks.rados.rados.0.smithi071.stdout:2108: writing smithi07134777-47 from 1640573 to 2115166 tid 2 2024-09-11T01:43:04.121 INFO:tasks.rados.rados.0.smithi071.stdout:2108: writing smithi07134777-47 from 2642791 to 2806934 tid 3 2024-09-11T01:43:04.121 INFO:tasks.rados.rados.0.smithi071.stdout:2109: rollback oid 17 current snap is 231 2024-09-11T01:43:04.122 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 17 to 185 2024-09-11T01:43:04.122 INFO:tasks.rados.rados.0.smithi071.stdout:2108: finishing write tid 1 to smithi07134777-47 2024-09-11T01:43:04.122 INFO:tasks.rados.rados.0.smithi071.stdout:2108: finishing write tid 2 to smithi07134777-47 2024-09-11T01:43:04.122 INFO:tasks.rados.rados.0.smithi071.stdout:2110: write (excl) oid 14 current snap is 231 2024-09-11T01:43:04.122 INFO:tasks.rados.rados.0.smithi071.stdout:2110: seq_num 763 ranges {423741=650287,1699055=618251} 2024-09-11T01:43:04.127 INFO:tasks.rados.rados.0.smithi071.stdout:2110: writing smithi07134777-14 from 423741 to 1074028 tid 1 2024-09-11T01:43:04.134 INFO:tasks.rados.rados.0.smithi071.stdout:2110: writing smithi07134777-14 from 1699055 to 2317306 tid 2 2024-09-11T01:43:04.134 INFO:tasks.rados.rados.0.smithi071.stdout:2111: delete oid 36 current snap is 231 2024-09-11T01:43:04.134 INFO:tasks.rados.rados.0.smithi071.stdout:2108: finishing write tid 3 to smithi07134777-47 2024-09-11T01:43:04.134 INFO:tasks.rados.rados.0.smithi071.stdout:2108: finishing write tid 4 to smithi07134777-47 2024-09-11T01:43:04.135 INFO:tasks.rados.rados.0.smithi071.stdout:2108: finishing write tid 5 to smithi07134777-47 2024-09-11T01:43:04.135 INFO:tasks.rados.rados.0.smithi071.stdout:2108: oid 47 updating version 0 to 863 2024-09-11T01:43:04.135 INFO:tasks.rados.rados.0.smithi071.stdout:2108: oid 47 updating version 863 to 865 2024-09-11T01:43:04.135 INFO:tasks.rados.rados.0.smithi071.stdout:2108: oid 47 version 865 is already newer than 864 2024-09-11T01:43:04.135 INFO:tasks.rados.rados.0.smithi071.stdout:2108: oid 47 version 865 is already newer than 862 2024-09-11T01:43:04.135 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 865 (ObjNum 762 snap 231 seq_num 762) dirty exists 2024-09-11T01:43:04.135 INFO:tasks.rados.rados.0.smithi071.stdout:2108: left oid 47 (ObjNum 762 snap 231 seq_num 762) 2024-09-11T01:43:04.135 INFO:tasks.rados.rados.0.smithi071.stdout:2109: finishing rollback tid 1 to smithi07134777-17 2024-09-11T01:43:04.135 INFO:tasks.rados.rados.0.smithi071.stdout:2109: finishing rollback tid 2 to smithi07134777-17 2024-09-11T01:43:04.135 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 867 (ObjNum 551 snap 157 seq_num 551) dirty exists 2024-09-11T01:43:04.135 INFO:tasks.rados.rados.0.smithi071.stdout:2110: finishing write tid 1 to smithi07134777-14 2024-09-11T01:43:04.136 INFO:tasks.rados.rados.0.smithi071.stdout:2108: done (3 left) 2024-09-11T01:43:04.136 INFO:tasks.rados.rados.0.smithi071.stdout:2109: done (2 left) 2024-09-11T01:43:04.136 INFO:tasks.rados.rados.0.smithi071.stdout:2111: done (1 left) 2024-09-11T01:43:04.136 INFO:tasks.rados.rados.0.smithi071.stdout:2112: snap_remove snap 180 2024-09-11T01:43:04.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:04.698+0000 7f174f7fe640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 0x7f175002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:04.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:04.698+0000 7f174f7fe640 10 monclient: get_auth_request con 0x7f1750126050 auth_method 0 2024-09-11T01:43:04.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:04.698+0000 7f174f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:04.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:04.698+0000 7f174f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:04.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:04.698+0000 7f174f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:04.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:04.698+0000 7f174f7fe640 1 -- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 msgr2=0x7f175002d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:43:04.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:04.698+0000 7f174f7fe640 1 -- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 msgr2=0x7f175002d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:43:04.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:04.698+0000 7f174f7fe640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 0x7f175002d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:04.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:04.698+0000 7f174f7fe640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 0x7f175002d440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:43:05.044 INFO:tasks.rados.rados.0.smithi071.stdout:2112: done (1 left) 2024-09-11T01:43:05.044 INFO:tasks.rados.rados.0.smithi071.stdout:2113: write oid 30 current snap is 231 2024-09-11T01:43:05.044 INFO:tasks.rados.rados.0.smithi071.stdout:2113: seq_num 765 ranges {457355=400785,1416019=621920,2725060=132295} 2024-09-11T01:43:05.049 INFO:tasks.rados.rados.0.smithi071.stdout:2113: writing smithi07134777-30 from 457355 to 858140 tid 1 2024-09-11T01:43:05.055 INFO:tasks.rados.rados.0.smithi071.stdout:2113: writing smithi07134777-30 from 1416019 to 2037939 tid 2 2024-09-11T01:43:05.060 INFO:tasks.rados.rados.0.smithi071.stdout:2113: writing smithi07134777-30 from 2725060 to 2857355 tid 3 2024-09-11T01:43:05.060 INFO:tasks.rados.rados.0.smithi071.stdout:2114: snap_create 2024-09-11T01:43:05.061 INFO:tasks.rados.rados.0.smithi071.stdout:2110: finishing write tid 2 to smithi07134777-14 2024-09-11T01:43:05.061 INFO:tasks.rados.rados.0.smithi071.stdout:2110: finishing write tid 3 to smithi07134777-14 2024-09-11T01:43:05.061 INFO:tasks.rados.rados.0.smithi071.stdout:2110: finishing write tid 4 to smithi07134777-14 2024-09-11T01:43:05.061 INFO:tasks.rados.rados.0.smithi071.stdout:2110: oid 14 updating version 0 to 1217 2024-09-11T01:43:05.061 INFO:tasks.rados.rados.0.smithi071.stdout:2110: oid 14 version 1217 is already newer than 1216 2024-09-11T01:43:05.061 INFO:tasks.rados.rados.0.smithi071.stdout:2110: oid 14 updating version 1217 to 1218 2024-09-11T01:43:05.061 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 1218 (ObjNum 763 snap 231 seq_num 763) dirty exists 2024-09-11T01:43:05.061 INFO:tasks.rados.rados.0.smithi071.stdout:2110: left oid 14 (ObjNum 763 snap 231 seq_num 763) 2024-09-11T01:43:05.061 INFO:tasks.rados.rados.0.smithi071.stdout:2113: finishing write tid 1 to smithi07134777-30 2024-09-11T01:43:05.066 INFO:tasks.rados.rados.0.smithi071.stdout:2113: finishing write tid 2 to smithi07134777-30 2024-09-11T01:43:05.069 INFO:tasks.rados.rados.0.smithi071.stdout:2113: finishing write tid 3 to smithi07134777-30 2024-09-11T01:43:05.069 INFO:tasks.rados.rados.0.smithi071.stdout:2113: finishing write tid 4 to smithi07134777-30 2024-09-11T01:43:05.069 INFO:tasks.rados.rados.0.smithi071.stdout:2113: finishing write tid 5 to smithi07134777-30 2024-09-11T01:43:05.069 INFO:tasks.rados.rados.0.smithi071.stdout:2113: oid 30 updating version 0 to 944 2024-09-11T01:43:05.069 INFO:tasks.rados.rados.0.smithi071.stdout:2113: oid 30 updating version 944 to 947 2024-09-11T01:43:05.069 INFO:tasks.rados.rados.0.smithi071.stdout:2113: oid 30 version 947 is already newer than 945 2024-09-11T01:43:05.070 INFO:tasks.rados.rados.0.smithi071.stdout:2113: oid 30 version 947 is already newer than 946 2024-09-11T01:43:05.070 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 947 (ObjNum 765 snap 231 seq_num 765) dirty exists 2024-09-11T01:43:05.070 INFO:tasks.rados.rados.0.smithi071.stdout:2113: left oid 30 (ObjNum 765 snap 231 seq_num 765) 2024-09-11T01:43:06.054 INFO:tasks.rados.rados.0.smithi071.stdout:2110: done (2 left) 2024-09-11T01:43:06.054 INFO:tasks.rados.rados.0.smithi071.stdout:2113: done (1 left) 2024-09-11T01:43:06.054 INFO:tasks.rados.rados.0.smithi071.stdout:2114: done (0 left) 2024-09-11T01:43:06.054 INFO:tasks.rados.rados.0.smithi071.stdout:2115: write (excl) oid 24 current snap is 232 2024-09-11T01:43:06.054 INFO:tasks.rados.rados.0.smithi071.stdout:2115: seq_num 766 ranges {474162=476034,1358148=481931,2352483=465303} 2024-09-11T01:43:06.059 INFO:tasks.rados.rados.0.smithi071.stdout:2115: writing smithi07134777-24 from 474162 to 950196 tid 1 2024-09-11T01:43:06.064 INFO:tasks.rados.rados.0.smithi071.stdout:2115: writing smithi07134777-24 from 1358148 to 1840079 tid 2 2024-09-11T01:43:06.069 INFO:tasks.rados.rados.0.smithi071.stdout:2115: writing smithi07134777-24 from 2352483 to 2817786 tid 3 2024-09-11T01:43:06.070 INFO:tasks.rados.rados.0.smithi071.stdout:2116: write (excl) oid 4 current snap is 232 2024-09-11T01:43:06.070 INFO:tasks.rados.rados.0.smithi071.stdout:2115: finishing write tid 1 to smithi07134777-24 2024-09-11T01:43:06.070 INFO:tasks.rados.rados.0.smithi071.stdout:2115: finishing write tid 2 to smithi07134777-24 2024-09-11T01:43:06.070 INFO:tasks.rados.rados.0.smithi071.stdout:2116: seq_num 767 ranges {490969=551283,1816630=625589} 2024-09-11T01:43:06.075 INFO:tasks.rados.rados.0.smithi071.stdout:2116: writing smithi07134777-4 from 490969 to 1042252 tid 1 2024-09-11T01:43:06.083 INFO:tasks.rados.rados.0.smithi071.stdout:2116: writing smithi07134777-4 from 1816630 to 2442219 tid 2 2024-09-11T01:43:06.083 INFO:tasks.rados.rados.0.smithi071.stdout:2117: copy_from oid 18 from oid 23 current snap is 232 2024-09-11T01:43:06.083 INFO:tasks.rados.rados.0.smithi071.stdout:2115: finishing write tid 3 to smithi07134777-24 2024-09-11T01:43:06.083 INFO:tasks.rados.rados.0.smithi071.stdout:2115: finishing write tid 4 to smithi07134777-24 2024-09-11T01:43:06.083 INFO:tasks.rados.rados.0.smithi071.stdout:2115: finishing write tid 5 to smithi07134777-24 2024-09-11T01:43:06.083 INFO:tasks.rados.rados.0.smithi071.stdout:2115: oid 24 updating version 0 to 961 2024-09-11T01:43:06.084 INFO:tasks.rados.rados.0.smithi071.stdout:2115: oid 24 updating version 961 to 964 2024-09-11T01:43:06.084 INFO:tasks.rados.rados.0.smithi071.stdout:2115: oid 24 version 964 is already newer than 963 2024-09-11T01:43:06.084 INFO:tasks.rados.rados.0.smithi071.stdout:2115: oid 24 version 964 is already newer than 962 2024-09-11T01:43:06.084 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 964 (ObjNum 766 snap 232 seq_num 766) dirty exists 2024-09-11T01:43:06.084 INFO:tasks.rados.rados.0.smithi071.stdout:2115: left oid 24 (ObjNum 766 snap 232 seq_num 766) 2024-09-11T01:43:06.084 INFO:tasks.rados.rados.0.smithi071.stdout:2116: finishing write tid 1 to smithi07134777-4 2024-09-11T01:43:06.084 INFO:tasks.rados.rados.0.smithi071.stdout:2115: done (2 left) 2024-09-11T01:43:06.084 INFO:tasks.rados.rados.0.smithi071.stdout:2118: snap_create 2024-09-11T01:43:06.088 INFO:tasks.rados.rados.0.smithi071.stdout:2116: finishing write tid 2 to smithi07134777-4 2024-09-11T01:43:06.088 INFO:tasks.rados.rados.0.smithi071.stdout:2116: finishing write tid 3 to smithi07134777-4 2024-09-11T01:43:06.088 INFO:tasks.rados.rados.0.smithi071.stdout:2116: finishing write tid 4 to smithi07134777-4 2024-09-11T01:43:06.088 INFO:tasks.rados.rados.0.smithi071.stdout:2116: oid 4 updating version 0 to 950 2024-09-11T01:43:06.088 INFO:tasks.rados.rados.0.smithi071.stdout:2116: oid 4 version 950 is already newer than 949 2024-09-11T01:43:06.088 INFO:tasks.rados.rados.0.smithi071.stdout:2116: oid 4 version 950 is already newer than 948 2024-09-11T01:43:06.089 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 950 (ObjNum 767 snap 232 seq_num 767) dirty exists 2024-09-11T01:43:06.089 INFO:tasks.rados.rados.0.smithi071.stdout:2116: left oid 4 (ObjNum 767 snap 232 seq_num 767) 2024-09-11T01:43:06.102 INFO:tasks.rados.rados.0.smithi071.stdout:2117: finishing copy_from racing read to smithi07134777-18 2024-09-11T01:43:06.105 INFO:tasks.rados.rados.0.smithi071.stdout:2117: finishing copy_from to smithi07134777-18 2024-09-11T01:43:06.105 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 964 (ObjNum 689 snap 204 seq_num 689) dirty exists 2024-09-11T01:43:06.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.290+0000 7f1746ffd640 10 monclient: tick 2024-09-11T01:43:06.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.290+0000 7f1746ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:43:06.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.290+0000 7f1746ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:06.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.290+0000 7f1746ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:06.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.299+0000 7f174f7fe640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 0x7f175002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:06.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.299+0000 7f174f7fe640 10 monclient: get_auth_request con 0x7f1750126050 auth_method 0 2024-09-11T01:43:06.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.299+0000 7f174f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:06.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.299+0000 7f174f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:06.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.299+0000 7f174f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:06.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.299+0000 7f174f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:06.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.299+0000 7f174f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:06.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.299+0000 7f174f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:06.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.299+0000 7f174f7fe640 10 monclient(hunting): handle_auth_done global_id 154100 payload 824 2024-09-11T01:43:06.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.299+0000 7f174f7fe640 1 -- 172.21.15.12:0/2932903739 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1750122400 con 0x7f1750126050 2024-09-11T01:43:06.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.302+0000 7f174f7fe640 1 -- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 msgr2=0x7f175002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:06.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.302+0000 7f174f7fe640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 0x7f175002d440 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f174002a0e0 tx=0x7f17400046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:06.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.302+0000 7f174cff9640 1 -- 172.21.15.12:0/2932903739 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1740037040 con 0x7f1750126050 2024-09-11T01:43:06.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.302+0000 7f174cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:06.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.302+0000 7f174cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:43:06.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.302+0000 7f174cff9640 10 monclient: dump: 2024-09-11T01:43:06.304 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:06.304 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:06.304 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:06.304 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:06.304 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:06.304 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:06.304 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:06.304 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:06.304 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:06.304 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:06.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.302+0000 7f174cff9640 1 -- 172.21.15.12:0/2932903739 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1740009490 con 0x7f1750126050 2024-09-11T01:43:06.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.302+0000 7f174cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:06.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.302+0000 7f174cff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:43:06.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.302+0000 7f174cff9640 1 -- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 msgr2=0x7f175002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:06.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.302+0000 7f174cff9640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 0x7f175002d440 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f174002a0e0 tx=0x7f17400046c0 comp rx=0 tx=0).stop 2024-09-11T01:43:06.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.302+0000 7f174cff9640 1 -- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 msgr2=0x7f175002d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:43:06.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.302+0000 7f174cff9640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126050 0x7f175002d440 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f174002a0e0 tx=0x7f17400046c0 comp rx=0 tx=0).stop 2024-09-11T01:43:06.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.309+0000 7f1755ced640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:43:06.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.309+0000 7f1755ced640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126430 0x7f175002d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:06.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.309+0000 7f1755ced640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:06.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.309+0000 7f1755ced640 1 -- 172.21.15.12:0/2932903739 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1750128db0 con 0x7f1750126430 2024-09-11T01:43:06.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.310+0000 7f174f7fe640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126430 0x7f175002d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:06.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.310+0000 7f174f7fe640 10 monclient: get_auth_request con 0x7f1750126430 auth_method 0 2024-09-11T01:43:06.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.310+0000 7f174f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:06.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.310+0000 7f174f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:06.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.310+0000 7f174f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:06.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.310+0000 7f174f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:06.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.310+0000 7f174f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:06.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.310+0000 7f174f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:06.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.310+0000 7f174f7fe640 10 monclient(hunting): handle_auth_done global_id 154103 payload 824 2024-09-11T01:43:06.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.310+0000 7f174f7fe640 1 -- 172.21.15.12:0/2932903739 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1750129550 con 0x7f1750126430 2024-09-11T01:43:06.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.312+0000 7f174f7fe640 1 -- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126430 msgr2=0x7f175002d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:06.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.312+0000 7f174f7fe640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126430 0x7f175002d980 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f17500a18e0 tx=0x7f1740009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:06.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.312+0000 7f174cff9640 1 -- 172.21.15.12:0/2932903739 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1740037070 con 0x7f1750126430 2024-09-11T01:43:06.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.312+0000 7f174cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:06.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.312+0000 7f174cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:43:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.312+0000 7f174cff9640 10 monclient: dump: 2024-09-11T01:43:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:06.315 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.313+0000 7f174cff9640 1 -- 172.21.15.12:0/2932903739 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7f17400044a0 con 0x7f1750126430 2024-09-11T01:43:06.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.313+0000 7f174cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:43:06.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.313+0000 7f174cff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:43:06.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.313+0000 7f174cff9640 1 -- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126430 msgr2=0x7f175002d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:06.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.313+0000 7f174cff9640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126430 0x7f175002d980 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f17500a18e0 tx=0x7f1740009240 comp rx=0 tx=0).stop 2024-09-11T01:43:06.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.313+0000 7f174cff9640 1 -- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126430 msgr2=0x7f175002d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:43:06.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.313+0000 7f174cff9640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1750126430 0x7f175002d980 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f17500a18e0 tx=0x7f1740009240 comp rx=0 tx=0).stop 2024-09-11T01:43:06.316 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:43:06.316 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:43:06.316 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:43:06.316 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:43:06.316 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 128, 2024-09-11T01:43:06.317 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 10394, 2024-09-11T01:43:06.317 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:43:06.317 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:43:06.317 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:43:06.317 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:43:06.317 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:06.317 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 4, 2024-09-11T01:43:06.317 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:43:06.317 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:43:06.317 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:43:06.317 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:06.317 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:06.317 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:06.318 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:06.318 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:06.318 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:06.318 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:06.318 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:06.318 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:06.318 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:06.318 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:06.318 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:06.318 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:43:06.318 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:43:06.318 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:06.319 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:06.319 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:06.319 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:06.319 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:06.319 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:06.319 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:06.319 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:06.319 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:06.319 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:06.319 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:06.319 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:06.319 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:06.319 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:43:06.320 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:43:06.320 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:43:06.320 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:43:06.320 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:43:06.320 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:43:06.320 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:43:06.320 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:43:06.320 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:43:06.320 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:43:06.320 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:43:06.320 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:43:06.320 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:43:06.320 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:43:06.320 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:43:06.321 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:43:06.321 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:43:06.321 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:06.321 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:06.321 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:06.321 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:06.321 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:06.321 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:06.321 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:06.321 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:06.321 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:06.321 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:06.321 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:06.321 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:06.322 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:43:06.322 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:06.322 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:43:06.322 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:06.322 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:43:06.322 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:43:06.322 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:06.322 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:06.322 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:06.323 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:06.323 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:43:06.323 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:06.323 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:06.323 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:06.323 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:06.323 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:43:06.323 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:06.323 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:06.323 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:06.323 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:06.323 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:43:06.323 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:43:06.324 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:06.324 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:06.324 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:06.324 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:06.324 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:06.324 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:43:06.324 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:43:06.324 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:06.324 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:06.324 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:06.324 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:06.324 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:43:06.324 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:06.325 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:06.325 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:06.325 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:06.325 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:43:06.325 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:06.325 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:06.325 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:06.325 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:06.325 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:43:06.325 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:43:06.325 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:06.325 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:06.325 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:06.326 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:06.326 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:06.326 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:43:06.326 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:43:06.326 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:06.326 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:06.326 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:06.326 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:06.326 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:43:06.326 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:06.326 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:06.326 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:06.326 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:06.326 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:43:06.327 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:06.327 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:06.327 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:06.327 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:06.327 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:43:06.327 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:43:06.327 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:06.327 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:06.327 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:06.327 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:06.327 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:06.327 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:06.327 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:43:06.328 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:43:06.328 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:06.328 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:06.328 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:06.328 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:43:06.328 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:06.328 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:06.328 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:43:06.328 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:06.328 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:06.328 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:06.328 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:43:06.328 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:06.328 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:06.329 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:06.329 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:43:06.329 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:43:06.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.313+0000 7f1755ced640 1 -- 172.21.15.12:0/2932903739 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f173003e210 msgr2=0x7f17300406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:06.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.313+0000 7f1755ced640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f173003e210 0x7f17300406d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f175006ecf0 tx=0x7f1738007e30 comp rx=0 tx=0).stop 2024-09-11T01:43:06.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.313+0000 7f1755ced640 10 monclient: shutdown 2024-09-11T01:43:06.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.313+0000 7f1755ced640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:06.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.313+0000 7f1755ced640 1 -- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17500a1380 msgr2=0x7f1750136110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:06.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.313+0000 7f1755ced640 1 --2- 172.21.15.12:0/2932903739 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17500a1380 0x7f1750136110 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1748009610 tx=0x7f174800e4b0 comp rx=0 tx=0).stop 2024-09-11T01:43:06.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.313+0000 7f174ffff640 1 -- 172.21.15.12:0/2932903739 reap_dead start 2024-09-11T01:43:06.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.313+0000 7f174ffff640 1 -- 172.21.15.12:0/2932903739 reap_dead start 2024-09-11T01:43:06.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.313+0000 7f1755ced640 1 -- 172.21.15.12:0/2932903739 shutdown_connections 2024-09-11T01:43:06.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.313+0000 7f1755ced640 1 -- 172.21.15.12:0/2932903739 >> 172.21.15.12:0/2932903739 conn(0x7f17500a2f70 msgr2=0x7f1750070820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:06.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.313+0000 7f1755ced640 1 -- 172.21.15.12:0/2932903739 shutdown_connections 2024-09-11T01:43:06.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.313+0000 7f1755ced640 1 -- 172.21.15.12:0/2932903739 wait complete. 2024-09-11T01:43:06.331 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:43:06.331 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:43:06.332 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:43:06.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.441+0000 7f113dfec640 10 monclient: get_monmap_and_config 2024-09-11T01:43:06.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.441+0000 7f113dfec640 10 monclient: build_initial_monmap 2024-09-11T01:43:06.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.441+0000 7f113dfec640 10 monclient: monmap: 2024-09-11T01:43:06.443 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:06.443 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:06.443 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:43:06.441905+0000 2024-09-11T01:43:06.443 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:43:06.441905+0000 2024-09-11T01:43:06.443 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:06.443 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:06.443 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:43:06.443 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:43:06.443 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:43:06.443 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:43:06.443 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:43:06.443 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:06.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.441+0000 7f113dfec640 1 Processor -- start 2024-09-11T01:43:06.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f113dfec640 1 -- start start 2024-09-11T01:43:06.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f113dfec640 10 monclient: init 2024-09-11T01:43:06.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f113dfec640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:06.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f113dfec640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-11T01:43:06.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f113dfec640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11380a2e20 0x7f113806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:06.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f113dfec640 10 monclient(hunting): picked mon.noname-b con 0x7f11380a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:06.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f113dfec640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f11380a4970 0x7f11380a3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:06.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f113dfec640 10 monclient(hunting): picked mon.noname-a con 0x7f11380a4970 addr v2:172.21.15.12:3300/0 2024-09-11T01:43:06.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f113dfec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f113806f910 addr v1:172.21.15.12:6789/0 2024-09-11T01:43:06.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f113dfec640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f113806f4f0 con 0x7f113806f910 2024-09-11T01:43:06.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f113dfec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:06.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f113dfec640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f113806ef30 con 0x7f11380a4970 2024-09-11T01:43:06.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f113dfec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:06.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f113dfec640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f113806ec50 con 0x7f11380a2e20 2024-09-11T01:43:06.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f113dfec640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:06.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f113dfec640 10 monclient(hunting): authenticate will time out at 1592.884650s 2024-09-11T01:43:06.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f11377fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11380a2e20 0x7f113806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:06.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f1136ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f11380a4970 0x7f11380a3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:06.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f11377fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11380a2e20 0x7f113806ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41218/0 (socket says 172.21.15.12:41218) 2024-09-11T01:43:06.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f1137fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f113806f910 0x7f1138137fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37656/0 (socket says 172.21.15.12:37656) 2024-09-11T01:43:06.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f11377fe640 1 -- 172.21.15.12:0/531133640 learned_addr learned my addr 172.21.15.12:0/531133640 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:06.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f11377fe640 10 monclient(hunting): get_auth_request con 0x7f11380a2e20 auth_method 0 2024-09-11T01:43:06.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f11377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:06.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f11377fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:06.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f11377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:06.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f1136ffd640 10 monclient(hunting): get_auth_request con 0x7f11380a4970 auth_method 0 2024-09-11T01:43:06.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f1136ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:06.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f1136ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:06.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f1136ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:06.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f1136ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:06.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f1136ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:06.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f1136ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:06.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f11377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:06.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f11377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:06.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.442+0000 7f11377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:06.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.443+0000 7f11377fe640 10 monclient(hunting): handle_auth_done global_id 154106 payload 274 2024-09-11T01:43:06.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.443+0000 7f11377fe640 1 -- 172.21.15.12:0/531133640 >> v2:172.21.15.12:3300/0 conn(0x7f11380a4970 msgr2=0x7f11380a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:06.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.443+0000 7f11377fe640 1 --2- 172.21.15.12:0/531133640 >> v2:172.21.15.12:3300/0 conn(0x7f11380a4970 0x7f11380a3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:06.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.443+0000 7f11377fe640 1 -- 172.21.15.12:0/531133640 >> v1:172.21.15.12:6789/0 conn(0x7f113806f910 legacy=0x7f1138137fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:06.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.443+0000 7f11377fe640 10 monclient: _finish_hunting 0 2024-09-11T01:43:06.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.443+0000 7f11377fe640 1 monclient: found mon.noname-b 2024-09-11T01:43:06.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.443+0000 7f11377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:06.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.443+0000 7f11377fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:43:06.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.443+0000 7f11377fe640 1 -- 172.21.15.12:0/531133640 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11380a3740 con 0x7f11380a2e20 2024-09-11T01:43:06.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.443+0000 7f1136ffd640 1 --2- 172.21.15.12:0/531133640 >> v2:172.21.15.12:3300/0 conn(0x7f11380a4970 0x7f11380a3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:43:06.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.445+0000 7f11377fe640 1 -- 172.21.15.12:0/531133640 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11380a2e20 msgr2=0x7f113806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:06.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.445+0000 7f11377fe640 1 --2- 172.21.15.12:0/531133640 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11380a2e20 0x7f113806ded0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1128009870 tx=0x7f112802a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed818e4d734890a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:06.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.446+0000 7f11367fc640 1 -- 172.21.15.12:0/531133640 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1128037070 con 0x7f11380a2e20 2024-09-11T01:43:06.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.446+0000 7f11367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:06.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.446+0000 7f11367fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:43:06.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.446+0000 7f11367fc640 10 monclient: dump: 2024-09-11T01:43:06.455 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:06.455 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:06.455 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:06.455 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:06.456 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:06.456 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:06.456 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:06.456 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:06.456 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:06.456 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:06.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.446+0000 7f11367fc640 10 monclient: _finish_auth 0 2024-09-11T01:43:06.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.446+0000 7f11367fc640 10 monclient: _check_auth_tickets 2024-09-11T01:43:06.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.446+0000 7f11367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:06.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.446+0000 7f11367fc640 1 -- 172.21.15.12:0/531133640 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1128032440 con 0x7f11380a2e20 2024-09-11T01:43:06.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.446+0000 7f11367fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:06.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.446+0000 7f11367fc640 1 -- 172.21.15.12:0/531133640 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1128032810 con 0x7f11380a2e20 2024-09-11T01:43:06.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.446+0000 7f113dfec640 5 monclient: authenticate success, global_id 154106 2024-09-11T01:43:06.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.446+0000 7f113dfec640 10 monclient: get_monmap_and_config success 2024-09-11T01:43:06.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.446+0000 7f113dfec640 4 set_mon_vals no callback set 2024-09-11T01:43:06.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.446+0000 7f11367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:06.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.446+0000 7f11367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:43:06.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.446+0000 7f11367fc640 10 monclient: dump: 2024-09-11T01:43:06.457 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:06.457 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:06.457 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:06.457 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:06.457 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:06.457 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:06.457 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:06.457 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:06.457 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:06.457 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:06.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.446+0000 7f113dfec640 10 monclient: shutdown 2024-09-11T01:43:06.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.446+0000 7f113dfec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:06.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.446+0000 7f113dfec640 1 -- 172.21.15.12:0/531133640 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11380a2e20 msgr2=0x7f113806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:06.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.446+0000 7f113dfec640 1 --2- 172.21.15.12:0/531133640 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11380a2e20 0x7f113806ded0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1128009870 tx=0x7f112802a0a0 comp rx=0 tx=0).stop 2024-09-11T01:43:06.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.446+0000 7f113dfec640 1 -- 172.21.15.12:0/531133640 shutdown_connections 2024-09-11T01:43:06.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.446+0000 7f113dfec640 1 --2- 172.21.15.12:0/531133640 >> v2:172.21.15.12:3300/0 conn(0x7f11380a4970 0x7f11380a3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:06.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.446+0000 7f113dfec640 1 --2- 172.21.15.12:0/531133640 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11380a2e20 0x7f113806ded0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:06.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.446+0000 7f113dfec640 1 -- 172.21.15.12:0/531133640 >> 172.21.15.12:0/531133640 conn(0x7f11380a8600 msgr2=0x7f11380a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:06.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.446+0000 7f113dfec640 1 -- 172.21.15.12:0/531133640 shutdown_connections 2024-09-11T01:43:06.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.446+0000 7f113dfec640 1 -- 172.21.15.12:0/531133640 wait complete. 2024-09-11T01:43:06.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.447+0000 7f113dfec640 10 monclient: build_initial_monmap 2024-09-11T01:43:06.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.447+0000 7f113dfec640 10 monclient: monmap: 2024-09-11T01:43:06.459 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:06.459 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:43:06.459 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:43:06.459 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:43:06.459 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:06.459 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:06.459 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:43:06.459 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:43:06.459 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:43:06.459 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:06.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.447+0000 7f113dfec640 1 Processor -- start 2024-09-11T01:43:06.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.447+0000 7f113dfec640 1 -- start start 2024-09-11T01:43:06.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.447+0000 7f113dfec640 10 monclient: init 2024-09-11T01:43:06.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f113dfec640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:06.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f113dfec640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:43:06.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f113dfec640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f113806f910 0x7f11381cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:06.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f113dfec640 10 monclient(hunting): picked mon.noname-c con 0x7f113806f910 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:06.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f113dfec640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f11380a2e20 0x7f11381cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:06.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f113dfec640 10 monclient(hunting): picked mon.noname-b con 0x7f11380a2e20 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:06.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f113dfec640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11380a4970 0x7f1138124f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:06.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f113dfec640 10 monclient(hunting): picked mon.noname-a con 0x7f11380a4970 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:06.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f113dfec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:06.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f113dfec640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1138138860 con 0x7f11380a4970 2024-09-11T01:43:06.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f113dfec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:06.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f113dfec640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f11381386e0 con 0x7f11380a2e20 2024-09-11T01:43:06.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f113dfec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:06.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f113dfec640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f11381389e0 con 0x7f113806f910 2024-09-11T01:43:06.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f113dfec640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:06.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f113dfec640 10 monclient(hunting): authenticate will time out at 1592.890362s 2024-09-11T01:43:06.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f11377fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f113806f910 0x7f11381cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:06.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f1137fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11380a4970 0x7f1138124f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:06.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f11377fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f113806f910 0x7f11381cd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41228/0 (socket says 172.21.15.12:41228) 2024-09-11T01:43:06.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f11377fe640 1 -- 172.21.15.12:0/768695216 learned_addr learned my addr 172.21.15.12:0/768695216 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:06.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f11377fe640 10 monclient(hunting): get_auth_request con 0x7f113806f910 auth_method 0 2024-09-11T01:43:06.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f11377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:06.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f11377fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:06.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f11377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:06.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f1137fff640 10 monclient(hunting): get_auth_request con 0x7f11380a4970 auth_method 0 2024-09-11T01:43:06.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f1137fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:06.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f1137fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f1137fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f11377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f11377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f11377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f1137fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f1137fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f1137fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f11377fe640 10 monclient(hunting): handle_auth_done global_id 154109 payload 824 2024-09-11T01:43:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f11377fe640 1 -- 172.21.15.12:0/768695216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f11380a2e20 msgr2=0x7f11381cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f11377fe640 1 --2- 172.21.15.12:0/768695216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f11380a2e20 0x7f11381cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f11377fe640 1 -- 172.21.15.12:0/768695216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11380a4970 msgr2=0x7f1138124f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f11377fe640 1 --2- 172.21.15.12:0/768695216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11380a4970 0x7f1138124f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:06.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f11377fe640 10 monclient: _finish_hunting 0 2024-09-11T01:43:06.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f11377fe640 1 monclient: found mon.noname-c 2024-09-11T01:43:06.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f11377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:06.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f11377fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:43:06.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f11377fe640 1 -- 172.21.15.12:0/768695216 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1138137d70 con 0x7f113806f910 2024-09-11T01:43:06.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.448+0000 7f1137fff640 1 --2- 172.21.15.12:0/768695216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11380a4970 0x7f1138124f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:43:06.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.451+0000 7f11377fe640 1 -- 172.21.15.12:0/768695216 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f113806f910 msgr2=0x7f11381cd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:06.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.451+0000 7f11377fe640 1 --2- 172.21.15.12:0/768695216 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f113806f910 0x7f11381cd470 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f11280090b0 tx=0x7f11280046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:06.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.451+0000 7f111ffff640 1 -- 172.21.15.12:0/768695216 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1128041020 con 0x7f113806f910 2024-09-11T01:43:06.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.451+0000 7f111ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:06.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.451+0000 7f111ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:43:06.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.451+0000 7f111ffff640 10 monclient: dump: 2024-09-11T01:43:06.463 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:06.463 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:06.464 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:06.464 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:06.464 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:06.464 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:06.464 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:06.464 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:06.464 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:06.464 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:06.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.451+0000 7f111ffff640 10 monclient: _finish_auth 0 2024-09-11T01:43:06.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.451+0000 7f111ffff640 10 monclient: _check_auth_tickets 2024-09-11T01:43:06.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.451+0000 7f111ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:06.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.451+0000 7f111ffff640 1 -- 172.21.15.12:0/768695216 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f112803d630 con 0x7f113806f910 2024-09-11T01:43:06.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.451+0000 7f113dfec640 5 monclient: authenticate success, global_id 154109 2024-09-11T01:43:06.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.451+0000 7f113dfec640 10 monclient: _renew_subs 2024-09-11T01:43:06.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.451+0000 7f113dfec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:43:06.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.451+0000 7f113dfec640 1 -- 172.21.15.12:0/768695216 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1138126970 con 0x7f113806f910 2024-09-11T01:43:06.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.451+0000 7f111ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:06.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.451+0000 7f111ffff640 1 -- 172.21.15.12:0/768695216 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1128037040 con 0x7f113806f910 2024-09-11T01:43:06.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.451+0000 7f111ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:06.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.451+0000 7f111ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:43:06.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.451+0000 7f111ffff640 10 monclient: dump: 2024-09-11T01:43:06.465 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:06.465 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:06.465 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:06.465 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:06.465 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:06.465 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:06.465 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:06.465 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:06.466 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:06.466 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:06.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.451+0000 7f113dfec640 10 monclient: _renew_subs 2024-09-11T01:43:06.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.451+0000 7f113dfec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:43:06.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.451+0000 7f113dfec640 1 -- 172.21.15.12:0/768695216 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1138128710 con 0x7f113806f910 2024-09-11T01:43:06.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.451+0000 7f1135ffb640 4 set_mon_vals no callback set 2024-09-11T01:43:06.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.452+0000 7f113dfec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:06.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.452+0000 7f113dfec640 1 --2- 172.21.15.12:0/768695216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1138125440 0x7f1138125820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:06.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.452+0000 7f113dfec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:06.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.452+0000 7f113dfec640 1 -- 172.21.15.12:0/768695216 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1138127590 con 0x7f1138125440 2024-09-11T01:43:06.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.452+0000 7f111ffff640 1 -- 172.21.15.12:0/768695216 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f112803a030 con 0x7f113806f910 2024-09-11T01:43:06.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.452+0000 7f1136ffd640 1 --2- 172.21.15.12:0/768695216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1138125440 0x7f1138125820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:06.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.452+0000 7f111ffff640 1 --2- 172.21.15.12:0/768695216 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f112403e210 0x7f11240406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:06.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.452+0000 7f1136ffd640 10 monclient: get_auth_request con 0x7f1138125440 auth_method 0 2024-09-11T01:43:06.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.452+0000 7f1136ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:06.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.452+0000 7f1136ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:06.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.452+0000 7f1136ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:06.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.452+0000 7f111ffff640 1 -- 172.21.15.12:0/768695216 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(473..473 src has 285..473) ==== 4771+0+0 (secure 0 0 0) 0x7f1128073080 con 0x7f113806f910 2024-09-11T01:43:06.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.452+0000 7f1136ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:06.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.452+0000 7f1136ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:06.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.452+0000 7f1136ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:06.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.452+0000 7f1137fff640 1 --2- 172.21.15.12:0/768695216 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f112403e210 0x7f11240406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:06.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.452+0000 7f1137fff640 10 monclient: get_auth_request con 0x7f112403e210 auth_method 0 2024-09-11T01:43:06.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.452+0000 7f1136ffd640 10 monclient(hunting): handle_auth_done global_id 134287 payload 824 2024-09-11T01:43:06.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.452+0000 7f1136ffd640 1 -- 172.21.15.12:0/768695216 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f11380a65f0 con 0x7f1138125440 2024-09-11T01:43:06.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.455+0000 7f1136ffd640 1 -- 172.21.15.12:0/768695216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1138125440 msgr2=0x7f1138125820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:06.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.455+0000 7f1136ffd640 1 --2- 172.21.15.12:0/768695216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1138125440 0x7f1138125820 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1138127b80 tx=0x7f112c00ec00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:06.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.455+0000 7f1137fff640 1 --2- 172.21.15.12:0/768695216 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f112403e210 0x7f11240406d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f113806fd60 tx=0x7f1130004ee0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:06.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.455+0000 7f111ffff640 1 -- 172.21.15.12:0/768695216 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c017070 con 0x7f1138125440 2024-09-11T01:43:06.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.455+0000 7f111ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:06.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.455+0000 7f111ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:43:06.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.455+0000 7f111ffff640 10 monclient: dump: 2024-09-11T01:43:06.470 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:06.470 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:06.470 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:06.470 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:06.470 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:06.470 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:06.471 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:06.471 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:06.471 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:06.471 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:06.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.455+0000 7f111ffff640 1 -- 172.21.15.12:0/768695216 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f112c00cac0 con 0x7f1138125440 2024-09-11T01:43:06.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.455+0000 7f111ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:06.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.455+0000 7f111ffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:43:06.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.455+0000 7f111ffff640 1 -- 172.21.15.12:0/768695216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1138125440 msgr2=0x7f1138125820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:06.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.455+0000 7f111ffff640 1 --2- 172.21.15.12:0/768695216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1138125440 0x7f1138125820 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1138127b80 tx=0x7f112c00ec00 comp rx=0 tx=0).stop 2024-09-11T01:43:06.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.455+0000 7f111ffff640 1 -- 172.21.15.12:0/768695216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1138125440 msgr2=0x7f1138125820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:43:06.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.455+0000 7f111ffff640 1 --2- 172.21.15.12:0/768695216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1138125440 0x7f1138125820 secure :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1138127b80 tx=0x7f112c00ec00 comp rx=0 tx=0).stop 2024-09-11T01:43:06.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.462+0000 7f113dfec640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:43:06.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.462+0000 7f113dfec640 1 --2- 172.21.15.12:0/768695216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1138125d60 0x7f11380a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:06.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.462+0000 7f113dfec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:06.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.462+0000 7f113dfec640 1 -- 172.21.15.12:0/768695216 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f113806a060 con 0x7f1138125d60 2024-09-11T01:43:06.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.462+0000 7f1136ffd640 1 --2- 172.21.15.12:0/768695216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1138125d60 0x7f11380a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:06.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.462+0000 7f1136ffd640 10 monclient: get_auth_request con 0x7f1138125d60 auth_method 0 2024-09-11T01:43:06.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.462+0000 7f1136ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:06.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.462+0000 7f1136ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:06.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.462+0000 7f1136ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:06.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.462+0000 7f1136ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:06.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.462+0000 7f1136ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:06.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.462+0000 7f1136ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:06.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.462+0000 7f1136ffd640 10 monclient(hunting): handle_auth_done global_id 134290 payload 824 2024-09-11T01:43:06.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.462+0000 7f1136ffd640 1 -- 172.21.15.12:0/768695216 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1138128df0 con 0x7f1138125d60 2024-09-11T01:43:06.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.465+0000 7f1136ffd640 1 -- 172.21.15.12:0/768695216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1138125d60 msgr2=0x7f11380a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:06.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.465+0000 7f1136ffd640 1 --2- 172.21.15.12:0/768695216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1138125d60 0x7f11380a0830 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f112c019000 tx=0x7f112c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:06.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.465+0000 7f111ffff640 1 -- 172.21.15.12:0/768695216 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c017070 con 0x7f1138125d60 2024-09-11T01:43:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.465+0000 7f111ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.465+0000 7f111ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:43:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.465+0000 7f111ffff640 10 monclient: dump: 2024-09-11T01:43:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:06.473 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.465+0000 7f111ffff640 1 -- 172.21.15.12:0/768695216 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f112c006600 con 0x7f1138125d60 2024-09-11T01:43:06.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.465+0000 7f111ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:43:06.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.465+0000 7f111ffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:43:06.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.465+0000 7f111ffff640 1 -- 172.21.15.12:0/768695216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1138125d60 msgr2=0x7f11380a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:06.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.465+0000 7f111ffff640 1 --2- 172.21.15.12:0/768695216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1138125d60 0x7f11380a0830 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f112c019000 tx=0x7f112c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:43:06.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.465+0000 7f111ffff640 1 -- 172.21.15.12:0/768695216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1138125d60 msgr2=0x7f11380a0830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:43:06.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.465+0000 7f111ffff640 1 --2- 172.21.15.12:0/768695216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1138125d60 0x7f11380a0830 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f112c019000 tx=0x7f112c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:43:06.474 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:43:06.474 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:43:06.474 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:43:06.474 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:43:06.474 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 128, 2024-09-11T01:43:06.475 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 119427, 2024-09-11T01:43:06.475 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:43:06.475 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:43:06.475 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:43:06.475 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:43:06.475 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:06.475 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 4, 2024-09-11T01:43:06.475 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:43:06.475 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:43:06.475 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:43:06.475 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:06.475 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:06.475 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:06.475 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:06.475 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:06.476 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:06.476 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:06.476 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:06.476 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:06.476 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:06.476 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:06.476 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:06.476 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:43:06.476 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:43:06.476 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:06.476 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:06.476 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:06.476 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:06.476 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:06.476 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:06.477 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:06.477 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:06.477 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:06.477 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:06.477 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:06.477 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:06.477 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:06.477 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:43:06.477 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:43:06.477 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:43:06.477 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:43:06.477 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:43:06.477 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:43:06.477 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:43:06.478 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:43:06.478 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:43:06.478 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:43:06.478 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:43:06.478 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:43:06.478 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:43:06.478 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:43:06.478 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:43:06.478 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:43:06.478 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:43:06.478 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:06.478 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:06.478 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:06.478 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:06.479 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:06.479 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:06.479 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:06.479 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:06.479 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:06.479 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:06.479 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:06.479 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:06.479 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:43:06.479 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:06.479 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:43:06.479 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:06.479 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:43:06.480 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:43:06.480 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:06.480 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:06.480 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:06.480 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:06.480 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:43:06.480 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:06.480 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:06.480 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:06.480 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:06.480 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:43:06.480 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:06.480 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:06.480 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:06.480 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:06.481 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:43:06.481 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:43:06.481 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:06.481 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:06.481 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:06.481 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:06.481 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:06.481 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:43:06.481 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:43:06.481 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:06.481 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:06.481 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:06.481 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:06.482 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:43:06.482 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:06.482 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:06.482 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:06.482 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:06.482 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:43:06.482 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:06.482 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:06.482 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:06.482 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:06.482 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:43:06.482 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:43:06.482 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:06.482 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:06.483 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:06.483 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:06.483 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:06.483 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:43:06.483 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:43:06.483 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:06.483 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:06.483 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:06.483 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:06.483 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:43:06.483 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:06.483 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:06.483 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:06.483 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:06.484 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:43:06.484 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:06.484 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:06.484 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:06.484 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:06.484 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:43:06.484 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:43:06.484 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:06.484 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:06.484 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:06.484 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:06.484 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:06.484 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:06.485 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:43:06.485 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:43:06.485 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:06.485 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:06.485 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:06.485 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:43:06.485 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:06.485 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:06.485 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:43:06.485 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:06.485 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:06.485 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:06.485 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:43:06.486 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:06.486 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:06.486 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:43:06.486 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:06.486 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:06.486 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:06.486 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:43:06.486 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:06.486 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:06.486 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:43:06.486 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:06.486 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:06.486 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:06.487 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:43:06.487 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:06.487 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:06.487 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:06.487 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:43:06.487 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:43:06.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.466+0000 7f113dfec640 1 -- 172.21.15.12:0/768695216 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f112403e210 msgr2=0x7f11240406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:06.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.466+0000 7f113dfec640 1 --2- 172.21.15.12:0/768695216 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f112403e210 0x7f11240406d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f113806fd60 tx=0x7f1130004ee0 comp rx=0 tx=0).stop 2024-09-11T01:43:06.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.466+0000 7f113dfec640 10 monclient: shutdown 2024-09-11T01:43:06.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.466+0000 7f113dfec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:06.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.466+0000 7f113dfec640 1 -- 172.21.15.12:0/768695216 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f113806f910 msgr2=0x7f11381cd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:06.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.466+0000 7f113dfec640 1 --2- 172.21.15.12:0/768695216 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f113806f910 0x7f11381cd470 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f11280090b0 tx=0x7f11280046c0 comp rx=0 tx=0).stop 2024-09-11T01:43:06.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.466+0000 7f1137fff640 1 -- 172.21.15.12:0/768695216 reap_dead start 2024-09-11T01:43:06.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.466+0000 7f1137fff640 1 -- 172.21.15.12:0/768695216 reap_dead start 2024-09-11T01:43:06.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.466+0000 7f113dfec640 1 -- 172.21.15.12:0/768695216 shutdown_connections 2024-09-11T01:43:06.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.466+0000 7f113dfec640 1 -- 172.21.15.12:0/768695216 >> 172.21.15.12:0/768695216 conn(0x7f11380a8600 msgr2=0x7f1138072e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:06.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.466+0000 7f113dfec640 1 -- 172.21.15.12:0/768695216 shutdown_connections 2024-09-11T01:43:06.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.466+0000 7f113dfec640 1 -- 172.21.15.12:0/768695216 wait complete. 2024-09-11T01:43:06.489 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:43:06.489 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:43:06.490 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:43:06.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.589+0000 7f9608ba0640 10 monclient: get_monmap_and_config 2024-09-11T01:43:06.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.589+0000 7f9608ba0640 10 monclient: build_initial_monmap 2024-09-11T01:43:06.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.589+0000 7f9608ba0640 10 monclient: monmap: 2024-09-11T01:43:06.591 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:06.591 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:06.591 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:43:06.589957+0000 2024-09-11T01:43:06.591 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:43:06.589957+0000 2024-09-11T01:43:06.591 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:06.591 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:06.591 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:43:06.591 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:43:06.591 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:43:06.591 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:43:06.591 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:43:06.591 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.589+0000 7f9608ba0640 1 Processor -- start 2024-09-11T01:43:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.589+0000 7f9608ba0640 1 -- start start 2024-09-11T01:43:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.589+0000 7f9608ba0640 10 monclient: init 2024-09-11T01:43:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.590+0000 7f9608ba0640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.590+0000 7f9608ba0640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-11T01:43:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.590+0000 7f9608ba0640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9604137e50 0x7f960409d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.590+0000 7f9608ba0640 10 monclient(hunting): picked mon.noname-c con 0x7f9604137e50 addr v2:172.21.15.71:3300/0 2024-09-11T01:43:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.590+0000 7f9608ba0640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9604138230 0x7f9604138640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.590+0000 7f9608ba0640 10 monclient(hunting): picked mon.noname-a con 0x7f9604138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:43:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.590+0000 7f9608ba0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9604134cb0 addr v1:172.21.15.71:6789/0 2024-09-11T01:43:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.590+0000 7f9608ba0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96040a62d0 con 0x7f9604134cb0 2024-09-11T01:43:06.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.590+0000 7f9608ba0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.590+0000 7f9608ba0640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f960409bb40 con 0x7f9604138230 2024-09-11T01:43:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.590+0000 7f9608ba0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.590+0000 7f9608ba0640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f960409c4a0 con 0x7f9604137e50 2024-09-11T01:43:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.590+0000 7f9608ba0640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.590+0000 7f9608ba0640 10 monclient(hunting): authenticate will time out at 1593.032632s 2024-09-11T01:43:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.590+0000 7f9601d9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9604138230 0x7f9604138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.590+0000 7f9601d9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9604138230 0x7f9604138640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34850/0 (socket says 172.21.15.12:34850) 2024-09-11T01:43:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.590+0000 7f9601d9b640 1 -- 172.21.15.12:0/4231121603 learned_addr learned my addr 172.21.15.12:0/4231121603 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.590+0000 7f9601d9b640 10 monclient(hunting): get_auth_request con 0x7f9604138230 auth_method 0 2024-09-11T01:43:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.590+0000 7f9601d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.590+0000 7f9601d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.590+0000 7f9601d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:06.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.590+0000 7f9601d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:06.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.590+0000 7f9601d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:06.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.590+0000 7f9601d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:06.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.590+0000 7f960259c640 1 --2- 172.21.15.12:0/4231121603 >> v2:172.21.15.71:3300/0 conn(0x7f9604137e50 0x7f960409d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:06.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.591+0000 7f9601d9b640 10 monclient(hunting): handle_auth_done global_id 94680 payload 274 2024-09-11T01:43:06.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.591+0000 7f9601d9b640 1 -- 172.21.15.12:0/4231121603 >> v2:172.21.15.71:3300/0 conn(0x7f9604137e50 msgr2=0x7f960409d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:06.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.591+0000 7f9601d9b640 1 --2- 172.21.15.12:0/4231121603 >> v2:172.21.15.71:3300/0 conn(0x7f9604137e50 0x7f960409d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:06.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.591+0000 7f9601d9b640 1 -- 172.21.15.12:0/4231121603 >> v1:172.21.15.71:6789/0 conn(0x7f9604134cb0 legacy=0x7f9604135160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:06.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.591+0000 7f9601d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:43:06.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.591+0000 7f9601d9b640 1 monclient: found mon.noname-a 2024-09-11T01:43:06.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.591+0000 7f9601d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:06.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.591+0000 7f9601d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:06.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.591+0000 7f9601d9b640 1 -- 172.21.15.12:0/4231121603 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f960409c780 con 0x7f9604138230 2024-09-11T01:43:06.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.591+0000 7f960259c640 10 monclient: get_auth_request con 0x7f9604137e50 auth_method 0 2024-09-11T01:43:06.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.591+0000 7f960259c640 1 --2- 172.21.15.12:0/4231121603 >> v2:172.21.15.71:3300/0 conn(0x7f9604137e50 0x7f960409d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:43:06.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.593+0000 7f9601d9b640 1 -- 172.21.15.12:0/4231121603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9604138230 msgr2=0x7f9604138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:06.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.593+0000 7f9601d9b640 1 --2- 172.21.15.12:0/4231121603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9604138230 0x7f9604138640 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f960409cf30 tx=0x7f95f802a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c767ec0651d4a8c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:06.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.593+0000 7f960159a640 1 -- 172.21.15.12:0/4231121603 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f8037070 con 0x7f9604138230 2024-09-11T01:43:06.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.594+0000 7f960159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:06.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.594+0000 7f960159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:06.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.594+0000 7f960159a640 10 monclient: dump: 2024-09-11T01:43:06.603 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:06.603 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:06.603 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:06.603 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:06.603 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:06.603 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:06.604 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:06.604 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:06.604 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:06.604 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:06.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.594+0000 7f960159a640 10 monclient: _finish_auth 0 2024-09-11T01:43:06.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.594+0000 7f960159a640 10 monclient: _check_auth_tickets 2024-09-11T01:43:06.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.594+0000 7f960159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:06.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.594+0000 7f960159a640 1 -- 172.21.15.12:0/4231121603 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95f8032640 con 0x7f9604138230 2024-09-11T01:43:06.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.594+0000 7f9608ba0640 5 monclient: authenticate success, global_id 94680 2024-09-11T01:43:06.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.594+0000 7f960159a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:06.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.594+0000 7f960159a640 1 -- 172.21.15.12:0/4231121603 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f8032a10 con 0x7f9604138230 2024-09-11T01:43:06.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.594+0000 7f960159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:06.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.594+0000 7f960159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:06.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.594+0000 7f960159a640 10 monclient: dump: 2024-09-11T01:43:06.605 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:06.605 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:06.605 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:06.605 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:06.605 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:06.605 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:06.605 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:06.605 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:06.605 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:06.605 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:06.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.594+0000 7f9608ba0640 10 monclient: get_monmap_and_config success 2024-09-11T01:43:06.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.594+0000 7f9608ba0640 4 set_mon_vals no callback set 2024-09-11T01:43:06.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.594+0000 7f9608ba0640 10 monclient: shutdown 2024-09-11T01:43:06.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.594+0000 7f9608ba0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:06.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.594+0000 7f9608ba0640 1 -- 172.21.15.12:0/4231121603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9604138230 msgr2=0x7f9604138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:06.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.594+0000 7f9608ba0640 1 --2- 172.21.15.12:0/4231121603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9604138230 0x7f9604138640 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f960409cf30 tx=0x7f95f802a080 comp rx=0 tx=0).stop 2024-09-11T01:43:06.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.594+0000 7f9608ba0640 1 -- 172.21.15.12:0/4231121603 shutdown_connections 2024-09-11T01:43:06.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.594+0000 7f9608ba0640 1 --2- 172.21.15.12:0/4231121603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9604138230 0x7f9604138640 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:06.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.594+0000 7f9608ba0640 1 --2- 172.21.15.12:0/4231121603 >> v2:172.21.15.71:3300/0 conn(0x7f9604137e50 0x7f960409d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:06.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.594+0000 7f9608ba0640 1 -- 172.21.15.12:0/4231121603 >> 172.21.15.12:0/4231121603 conn(0x7f96040a7020 msgr2=0x7f96040a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:06.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.594+0000 7f9608ba0640 1 -- 172.21.15.12:0/4231121603 shutdown_connections 2024-09-11T01:43:06.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.594+0000 7f9608ba0640 1 -- 172.21.15.12:0/4231121603 wait complete. 2024-09-11T01:43:06.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.595+0000 7f9608ba0640 10 monclient: build_initial_monmap 2024-09-11T01:43:06.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.595+0000 7f9608ba0640 10 monclient: monmap: 2024-09-11T01:43:06.606 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:06.607 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:43:06.607 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:43:06.607 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:43:06.607 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:06.607 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:06.607 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:43:06.607 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:43:06.607 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:43:06.607 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:06.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.595+0000 7f9608ba0640 1 Processor -- start 2024-09-11T01:43:06.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.595+0000 7f9608ba0640 1 -- start start 2024-09-11T01:43:06.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.595+0000 7f9608ba0640 10 monclient: init 2024-09-11T01:43:06.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9608ba0640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:06.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9608ba0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:43:06.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9608ba0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9604134cb0 0x7f96041cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:06.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9608ba0640 10 monclient(hunting): picked mon.noname-a con 0x7f9604134cb0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:06.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9608ba0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9604137e50 0x7f96041215c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:06.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9608ba0640 10 monclient(hunting): picked mon.noname-b con 0x7f9604137e50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:06.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9608ba0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9604138230 0x7f9604121b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:06.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9608ba0640 10 monclient(hunting): picked mon.noname-c con 0x7f9604138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:06.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9608ba0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:06.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9608ba0640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f960406f710 con 0x7f9604134cb0 2024-09-11T01:43:06.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9608ba0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:06.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9608ba0640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f9604138b80 con 0x7f9604137e50 2024-09-11T01:43:06.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9608ba0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:06.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9608ba0640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f960406f890 con 0x7f9604138230 2024-09-11T01:43:06.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9608ba0640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:06.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9608ba0640 10 monclient(hunting): authenticate will time out at 1593.038205s 2024-09-11T01:43:06.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f960259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9604134cb0 0x7f96041cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:06.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9602d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9604138230 0x7f9604121b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:06.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f960259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9604134cb0 0x7f96041cd720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34862/0 (socket says 172.21.15.12:34862) 2024-09-11T01:43:06.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f960259c640 1 -- 172.21.15.12:0/2278444086 learned_addr learned my addr 172.21.15.12:0/2278444086 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:06.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f960259c640 10 monclient(hunting): get_auth_request con 0x7f9604134cb0 auth_method 0 2024-09-11T01:43:06.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f960259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:06.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f960259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:06.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f960259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:06.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9602d9d640 10 monclient(hunting): get_auth_request con 0x7f9604138230 auth_method 0 2024-09-11T01:43:06.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9602d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:06.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9602d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:06.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9602d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:06.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f960259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:06.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f960259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:06.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f960259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:06.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9602d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:06.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9602d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:06.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9602d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:06.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9601d9b640 1 --2- 172.21.15.12:0/2278444086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9604137e50 0x7f96041215c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:06.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9601d9b640 10 monclient(hunting): get_auth_request con 0x7f9604137e50 auth_method 0 2024-09-11T01:43:06.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9601d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:06.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9601d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:06.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9601d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:06.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9601d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:06.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9601d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:06.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9601d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:06.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f960259c640 10 monclient(hunting): handle_auth_done global_id 94683 payload 824 2024-09-11T01:43:06.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f960259c640 1 -- 172.21.15.12:0/2278444086 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9604138230 msgr2=0x7f9604121b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:06.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f960259c640 1 --2- 172.21.15.12:0/2278444086 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9604138230 0x7f9604121b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:06.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f960259c640 1 -- 172.21.15.12:0/2278444086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9604137e50 msgr2=0x7f96041215c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:06.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f960259c640 1 --2- 172.21.15.12:0/2278444086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9604137e50 0x7f96041215c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:06.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f960259c640 10 monclient: _finish_hunting 0 2024-09-11T01:43:06.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f960259c640 1 monclient: found mon.noname-a 2024-09-11T01:43:06.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f960259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:06.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f960259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:06.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f960259c640 1 -- 172.21.15.12:0/2278444086 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9604134950 con 0x7f9604134cb0 2024-09-11T01:43:06.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.596+0000 7f9602d9d640 1 --2- 172.21.15.12:0/2278444086 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9604138230 0x7f9604121b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:43:06.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.598+0000 7f960259c640 1 -- 172.21.15.12:0/2278444086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9604134cb0 msgr2=0x7f96041cd720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:06.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.598+0000 7f960259c640 1 --2- 172.21.15.12:0/2278444086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9604134cb0 0x7f96041cd720 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f95f400c7f0 tx=0x7f95f400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:06.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.599+0000 7f95f2ffd640 1 -- 172.21.15.12:0/2278444086 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f4015070 con 0x7f9604134cb0 2024-09-11T01:43:06.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.599+0000 7f95f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:06.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.599+0000 7f95f2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:06.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.599+0000 7f95f2ffd640 10 monclient: dump: 2024-09-11T01:43:06.612 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:06.612 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:06.612 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:06.612 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:06.612 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:06.613 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:06.613 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:06.613 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:06.613 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:06.613 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:06.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.599+0000 7f95f2ffd640 10 monclient: _finish_auth 0 2024-09-11T01:43:06.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.599+0000 7f95f2ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:43:06.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.599+0000 7f95f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:06.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.599+0000 7f9608ba0640 5 monclient: authenticate success, global_id 94683 2024-09-11T01:43:06.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.599+0000 7f95f2ffd640 1 -- 172.21.15.12:0/2278444086 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95f4007be0 con 0x7f9604134cb0 2024-09-11T01:43:06.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.599+0000 7f95f2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:06.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.599+0000 7f95f2ffd640 1 -- 172.21.15.12:0/2278444086 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f400ce40 con 0x7f9604134cb0 2024-09-11T01:43:06.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.599+0000 7f95f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:06.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.599+0000 7f95f2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:06.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.599+0000 7f95f2ffd640 10 monclient: dump: 2024-09-11T01:43:06.614 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:06.614 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:06.614 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:06.614 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:06.614 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:06.614 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:06.614 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:06.614 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:06.614 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:06.614 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.599+0000 7f9600d99640 4 set_mon_vals no callback set 2024-09-11T01:43:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.599+0000 7f9608ba0640 10 monclient: _renew_subs 2024-09-11T01:43:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.599+0000 7f9608ba0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.599+0000 7f9608ba0640 1 -- 172.21.15.12:0/2278444086 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96040a1710 con 0x7f9604134cb0 2024-09-11T01:43:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.599+0000 7f9608ba0640 10 monclient: _renew_subs 2024-09-11T01:43:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.599+0000 7f9608ba0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.599+0000 7f9608ba0640 1 -- 172.21.15.12:0/2278444086 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9604125390 con 0x7f9604134cb0 2024-09-11T01:43:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.600+0000 7f9608ba0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.600+0000 7f9608ba0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.600+0000 7f9608ba0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.600+0000 7f95f2ffd640 1 -- 172.21.15.12:0/2278444086 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f95f40113f0 con 0x7f9604134cb0 2024-09-11T01:43:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.600+0000 7f9608ba0640 1 -- 172.21.15.12:0/2278444086 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9604135900 con 0x7f9604134cb0 2024-09-11T01:43:06.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.600+0000 7f95f2ffd640 1 --2- 172.21.15.12:0/2278444086 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f95e403e210 0x7f95e40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:06.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.600+0000 7f95f2ffd640 1 -- 172.21.15.12:0/2278444086 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(473..473 src has 285..473) ==== 4771+0+0 (secure 0 0 0) 0x7f95f4053170 con 0x7f9604134cb0 2024-09-11T01:43:06.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.600+0000 7f9601d9b640 1 --2- 172.21.15.12:0/2278444086 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f95e403e210 0x7f95e40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:06.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.600+0000 7f9601d9b640 10 monclient: get_auth_request con 0x7f95e403e210 auth_method 0 2024-09-11T01:43:06.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.603+0000 7f9601d9b640 1 --2- 172.21.15.12:0/2278444086 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f95e403e210 0x7f95e40406d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f960409ca90 tx=0x7f95f802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:06.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.604+0000 7f95f2ffd640 1 -- 172.21.15.12:0/2278444086 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f95f4051b70 con 0x7f9604134cb0 2024-09-11T01:43:06.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.604+0000 7f95f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:06.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.604+0000 7f95f2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:43:06.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.861+0000 7f9608ba0640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:43:06.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.861+0000 7f9608ba0640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:43:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.861+0000 7f9608ba0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.861+0000 7f9608ba0640 1 -- 172.21.15.12:0/2278444086 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f9604124bb0 con 0x7f9604134cb0 2024-09-11T01:43:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.862+0000 7f95f2ffd640 1 -- 172.21.15.12:0/2278444086 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f95f4021cf0 con 0x7f9604134cb0 2024-09-11T01:43:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.862+0000 7f95f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:43:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.862+0000 7f95f2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:43:06.865 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.862+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.865 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.862+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=12,kv=11,logm=55} crc {auth=1822356793,config=2586702516,health=2431681264,kv=58696370,logm=125663135}) 2024-09-11T01:43:06.865 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.862+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=24,config=2,health=11,kv=12,logm=51} crc {auth=2358804308,config=2586702516,health=1382759333,kv=3263697420,logm=4132139213}) 2024-09-11T01:43:06.865 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.862+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.865 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.862+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=12,kv=11,logm=55} crc {auth=1822356793,config=2586702516,health=2431681264,kv=58696370,logm=125663135}) 2024-09-11T01:43:06.865 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.862+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=23,config=2,health=12,kv=12,logm=51} crc {auth=1585866975,config=2586702516,health=3548324560,kv=3263697420,logm=1837169709}) 2024-09-11T01:43:06.865 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.863+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.865 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.863+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1310339949}) 2024-09-11T01:43:06.865 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.863+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3345012529}) 2024-09-11T01:43:06.865 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.863+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.865 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.863+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1310339949}) 2024-09-11T01:43:06.866 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.863+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4233049549}) 2024-09-11T01:43:06.866 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.865+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.865+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1934224826}) 2024-09-11T01:43:06.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.865+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1829150890}) 2024-09-11T01:43:06.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.865+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.865+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1934224826}) 2024-09-11T01:43:06.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.865+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3629900560}) 2024-09-11T01:43:06.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.865+0000 7f9608ba0640 1 -- 172.21.15.12:0/2278444086 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f95e403e210 msgr2=0x7f95e40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:06.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.865+0000 7f9608ba0640 1 --2- 172.21.15.12:0/2278444086 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f95e403e210 0x7f95e40406d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f960409ca90 tx=0x7f95f802e000 comp rx=0 tx=0).stop 2024-09-11T01:43:06.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.865+0000 7f9608ba0640 10 monclient: shutdown 2024-09-11T01:43:06.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.865+0000 7f9608ba0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:06.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.865+0000 7f9608ba0640 1 -- 172.21.15.12:0/2278444086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9604134cb0 msgr2=0x7f96041cd720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:06.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.865+0000 7f9608ba0640 1 --2- 172.21.15.12:0/2278444086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9604134cb0 0x7f96041cd720 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f95f400c7f0 tx=0x7f95f400cbf0 comp rx=0 tx=0).stop 2024-09-11T01:43:06.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.865+0000 7f9608ba0640 1 -- 172.21.15.12:0/2278444086 shutdown_connections 2024-09-11T01:43:06.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.865+0000 7f9608ba0640 1 --2- 172.21.15.12:0/2278444086 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9604138230 0x7f9604121b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:06.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.865+0000 7f9608ba0640 1 --2- 172.21.15.12:0/2278444086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9604137e50 0x7f96041215c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:06.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.865+0000 7f9608ba0640 1 --2- 172.21.15.12:0/2278444086 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f95e403e210 0x7f95e40406d0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:06.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.865+0000 7f9608ba0640 1 --2- 172.21.15.12:0/2278444086 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9604134cb0 0x7f96041cd720 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:06.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.865+0000 7f9608ba0640 1 -- 172.21.15.12:0/2278444086 >> 172.21.15.12:0/2278444086 conn(0x7f96040a7020 msgr2=0x7f9604137620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:06.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.865+0000 7f9608ba0640 1 -- 172.21.15.12:0/2278444086 shutdown_connections 2024-09-11T01:43:06.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:06.865+0000 7f9608ba0640 1 -- 172.21.15.12:0/2278444086 wait complete. 2024-09-11T01:43:06.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.866+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.866+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1677676165}) 2024-09-11T01:43:06.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.866+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1507403265}) 2024-09-11T01:43:06.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.866+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.866+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1677676165}) 2024-09-11T01:43:06.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.866+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1917121682}) 2024-09-11T01:43:06.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.867+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.867+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399844486}) 2024-09-11T01:43:06.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.867+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=759046729}) 2024-09-11T01:43:06.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.867+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.867+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399844486}) 2024-09-11T01:43:06.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.867+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=659740751}) 2024-09-11T01:43:06.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.868+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.868+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=557253174}) 2024-09-11T01:43:06.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.868+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4040153451}) 2024-09-11T01:43:06.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.868+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.868+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=557253174}) 2024-09-11T01:43:06.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.868+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=503296478}) 2024-09-11T01:43:06.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.869+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.869+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4249792849}) 2024-09-11T01:43:06.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.869+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=223112550}) 2024-09-11T01:43:06.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.869+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.869+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4249792849}) 2024-09-11T01:43:06.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.869+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=346843413}) 2024-09-11T01:43:06.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.870+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.870+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=144002435}) 2024-09-11T01:43:06.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.870+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3868178649}) 2024-09-11T01:43:06.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.870+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.870+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=144002435}) 2024-09-11T01:43:06.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.870+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2058346895}) 2024-09-11T01:43:06.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.871+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.871+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3201547026}) 2024-09-11T01:43:06.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.871+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2563013685}) 2024-09-11T01:43:06.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.871+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.871+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3201547026}) 2024-09-11T01:43:06.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.871+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3953682051}) 2024-09-11T01:43:06.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.872+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.872+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1922187598}) 2024-09-11T01:43:06.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.872+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2287951940}) 2024-09-11T01:43:06.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.872+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.872+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1922187598}) 2024-09-11T01:43:06.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.872+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=416811687}) 2024-09-11T01:43:06.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.873+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.873+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=347522991}) 2024-09-11T01:43:06.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.873+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2090423214}) 2024-09-11T01:43:06.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.873+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.873+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=347522991}) 2024-09-11T01:43:06.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.873+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3752790293}) 2024-09-11T01:43:06.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.874+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.874+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2605428762}) 2024-09-11T01:43:06.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.874+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=173580386}) 2024-09-11T01:43:06.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.875+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.875+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2605428762}) 2024-09-11T01:43:06.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.875+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2376383349}) 2024-09-11T01:43:06.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.875+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.876+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=504179111}) 2024-09-11T01:43:06.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.876+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2676771933}) 2024-09-11T01:43:06.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.876+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.876+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=504179111}) 2024-09-11T01:43:06.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.876+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3046248114}) 2024-09-11T01:43:06.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.876+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.876+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3371294702}) 2024-09-11T01:43:06.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.876+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2974141225}) 2024-09-11T01:43:06.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.876+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.876+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3371294702}) 2024-09-11T01:43:06.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.876+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=71864136}) 2024-09-11T01:43:06.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.878+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.878+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2336663486}) 2024-09-11T01:43:06.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.878+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3162642556}) 2024-09-11T01:43:06.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.878+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.878+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2336663486}) 2024-09-11T01:43:06.882 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.878+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=861052543}) 2024-09-11T01:43:06.882 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.879+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.882 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.879+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3062401114}) 2024-09-11T01:43:06.882 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.879+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2843646818}) 2024-09-11T01:43:06.882 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.879+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.882 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.879+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3062401114}) 2024-09-11T01:43:06.882 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.879+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3194461656}) 2024-09-11T01:43:06.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.880+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.880+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2339065863}) 2024-09-11T01:43:06.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.880+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1165640422}) 2024-09-11T01:43:06.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.880+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.880+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2339065863}) 2024-09-11T01:43:06.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.880+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3752271047}) 2024-09-11T01:43:06.884 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:43:06.888 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.880+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.888 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.880+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2734534214}) 2024-09-11T01:43:06.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.880+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2676104603}) 2024-09-11T01:43:06.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.880+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.880+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2734534214}) 2024-09-11T01:43:06.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.880+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3413586230}) 2024-09-11T01:43:06.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.881+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.881+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=882709280}) 2024-09-11T01:43:06.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.881+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1656371668}) 2024-09-11T01:43:06.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.881+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.881+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=882709280}) 2024-09-11T01:43:06.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.881+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=604718052}) 2024-09-11T01:43:06.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.882+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.882+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1700172464}) 2024-09-11T01:43:06.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.882+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2199694771}) 2024-09-11T01:43:06.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.882+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.890 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.882+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1700172464}) 2024-09-11T01:43:06.890 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.882+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3811969658}) 2024-09-11T01:43:06.890 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.883+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.890 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.883+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2547772815}) 2024-09-11T01:43:06.890 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.883+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=799257887}) 2024-09-11T01:43:06.890 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.883+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.890 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.883+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2547772815}) 2024-09-11T01:43:06.890 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.883+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=659140880}) 2024-09-11T01:43:06.890 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.884+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.890 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.884+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2288581344}) 2024-09-11T01:43:06.890 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.884+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2923304413}) 2024-09-11T01:43:06.890 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.884+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.890 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.884+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2288581344}) 2024-09-11T01:43:06.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.884+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2417286983}) 2024-09-11T01:43:06.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.884+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.884+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3677580209}) 2024-09-11T01:43:06.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.884+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2194266069}) 2024-09-11T01:43:06.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.884+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.884+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3677580209}) 2024-09-11T01:43:06.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.884+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1929837216}) 2024-09-11T01:43:06.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.885+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.885+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921268143}) 2024-09-11T01:43:06.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.885+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1387830310}) 2024-09-11T01:43:06.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.885+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.885+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921268143}) 2024-09-11T01:43:06.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.885+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3632934715}) 2024-09-11T01:43:06.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.886+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.886+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4192331274}) 2024-09-11T01:43:06.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.886+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=882339665}) 2024-09-11T01:43:06.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.886+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.886+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4192331274}) 2024-09-11T01:43:06.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.886+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1031346390}) 2024-09-11T01:43:06.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.887+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.887+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3884163732}) 2024-09-11T01:43:06.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.887+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4235505817}) 2024-09-11T01:43:06.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.887+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.887+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3884163732}) 2024-09-11T01:43:06.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.887+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=903236741}) 2024-09-11T01:43:06.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.888+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.888+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1580671624}) 2024-09-11T01:43:06.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.888+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=818038692}) 2024-09-11T01:43:06.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.888+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.888+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1580671624}) 2024-09-11T01:43:06.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.888+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=668968943}) 2024-09-11T01:43:06.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.888+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.888+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4261970216}) 2024-09-11T01:43:06.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.888+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2586438678}) 2024-09-11T01:43:06.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.888+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.888+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4261970216}) 2024-09-11T01:43:06.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.888+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2184144307}) 2024-09-11T01:43:06.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.889+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.889+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2393834272}) 2024-09-11T01:43:06.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.889+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2697972958}) 2024-09-11T01:43:06.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.889+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.889+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2393834272}) 2024-09-11T01:43:06.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.889+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=304845164}) 2024-09-11T01:43:06.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.890+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.890+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2152486068}) 2024-09-11T01:43:06.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.890+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1032159799}) 2024-09-11T01:43:06.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.890+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.890+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2152486068}) 2024-09-11T01:43:06.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.890+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2331437396}) 2024-09-11T01:43:06.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.891+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.891+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3823443067}) 2024-09-11T01:43:06.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.891+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3364209252}) 2024-09-11T01:43:06.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.891+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.891+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3823443067}) 2024-09-11T01:43:06.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.891+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2843156012}) 2024-09-11T01:43:06.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.891+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.891+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2692656937}) 2024-09-11T01:43:06.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.891+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=852136488}) 2024-09-11T01:43:06.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.891+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.891+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2692656937}) 2024-09-11T01:43:06.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.891+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3153021785}) 2024-09-11T01:43:06.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.892+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.892+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2505682210}) 2024-09-11T01:43:06.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.892+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3110611506}) 2024-09-11T01:43:06.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.892+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.892+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2505682210}) 2024-09-11T01:43:06.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.892+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=269486403}) 2024-09-11T01:43:06.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.893+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.893+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1384727368}) 2024-09-11T01:43:06.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.893+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1840751094}) 2024-09-11T01:43:06.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.893+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.893+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1384727368}) 2024-09-11T01:43:06.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.893+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=542446856}) 2024-09-11T01:43:06.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.894+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.894+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3836372962}) 2024-09-11T01:43:06.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.894+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4244537962}) 2024-09-11T01:43:06.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.894+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:06.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.894+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3836372962}) 2024-09-11T01:43:06.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:06.894+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1788057548}) 2024-09-11T01:43:07.052 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.050+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.052 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.050+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3447745369}) 2024-09-11T01:43:07.052 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.050+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=741674731}) 2024-09-11T01:43:07.052 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.050+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.052 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.050+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3447745369}) 2024-09-11T01:43:07.052 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.050+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1498218063}) 2024-09-11T01:43:07.057 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.055+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.057 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.055+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=690528897}) 2024-09-11T01:43:07.057 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.055+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=540050409}) 2024-09-11T01:43:07.057 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.055+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.057 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.055+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=690528897}) 2024-09-11T01:43:07.057 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.055+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2428877002}) 2024-09-11T01:43:07.066 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.065+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.066 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.065+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1561219237}) 2024-09-11T01:43:07.066 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.065+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2340043151}) 2024-09-11T01:43:07.066 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.065+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.066 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.065+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1561219237}) 2024-09-11T01:43:07.067 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.065+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4181636389}) 2024-09-11T01:43:07.069 INFO:tasks.rados.rados.0.smithi071.stdout:2116: done (2 left) 2024-09-11T01:43:07.069 INFO:tasks.rados.rados.0.smithi071.stdout:2117: done (1 left) 2024-09-11T01:43:07.069 INFO:tasks.rados.rados.0.smithi071.stdout:2118: done (0 left) 2024-09-11T01:43:07.069 INFO:tasks.rados.rados.0.smithi071.stdout:2119: snap_remove snap 214 2024-09-11T01:43:07.096 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.095+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.096 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.095+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2884178211}) 2024-09-11T01:43:07.096 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.095+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2284489693}) 2024-09-11T01:43:07.096 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.095+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.097 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.095+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2884178211}) 2024-09-11T01:43:07.097 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.095+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=828318444}) 2024-09-11T01:43:07.456 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.454+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.456 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.454+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2025125071}) 2024-09-11T01:43:07.456 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.454+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3884586666}) 2024-09-11T01:43:07.457 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.454+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.457 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.454+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2025125071}) 2024-09-11T01:43:07.457 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.455+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=568327929}) 2024-09-11T01:43:07.457 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.456+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.458 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.456+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1331947144}) 2024-09-11T01:43:07.458 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.456+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2646687516}) 2024-09-11T01:43:07.458 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.456+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.458 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.456+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1331947144}) 2024-09-11T01:43:07.458 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.456+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=566252179}) 2024-09-11T01:43:07.459 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.456+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.459 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.456+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1193532531}) 2024-09-11T01:43:07.459 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.457+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=619476588}) 2024-09-11T01:43:07.459 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.457+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.459 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.457+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1193532531}) 2024-09-11T01:43:07.460 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.457+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1145265365}) 2024-09-11T01:43:07.460 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.457+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.460 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.457+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2387211481}) 2024-09-11T01:43:07.460 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.457+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2651965589}) 2024-09-11T01:43:07.460 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.457+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.460 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.457+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2387211481}) 2024-09-11T01:43:07.460 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.457+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1737069467}) 2024-09-11T01:43:07.461 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.458+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.461 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.458+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=77,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11} crc {logm=146198634,mdsmap=3140514009,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2452581135}) 2024-09-11T01:43:07.461 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.458+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=73,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=2} crc {logm=3681787246,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2933287437,mon_config_key=2416264360}) 2024-09-11T01:43:07.462 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.458+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.462 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.458+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=77,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11} crc {logm=146198634,mdsmap=3140514009,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2452581135}) 2024-09-11T01:43:07.462 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.458+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=75,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=1} crc {logm=3855561451,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2619316617,mon_config_key=1679578512}) 2024-09-11T01:43:07.462 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.459+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.462 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.459+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=16,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=72} crc {mon_config_key=1446728439,monmap=2933275274,nvmeofgw=1596038039,osd_metadata=2942852742,osd_pg_creating=1038009913,osd_snap=2073911045}) 2024-09-11T01:43:07.462 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.459+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=15,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=73} crc {mon_config_key=3643325166,monmap=1324671969,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=1038009913,osd_snap=3738802084}) 2024-09-11T01:43:07.462 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.459+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.462 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.459+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=16,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=72} crc {mon_config_key=1446728439,monmap=2933275274,nvmeofgw=1596038039,osd_metadata=2942852742,osd_pg_creating=1038009913,osd_snap=2073911045}) 2024-09-11T01:43:07.462 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.459+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=16,monmap=3,nvmeofgw=2,osd_metadata=5,osd_pg_creating=1,osd_snap=73} crc {mon_config_key=1809882168,monmap=2933275274,nvmeofgw=1596038039,osd_metadata=118787097,osd_pg_creating=1038009913,osd_snap=2757877084}) 2024-09-11T01:43:07.463 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.460+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.463 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.460+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=23,osdmap=77} crc {osd_snap=2701503198,osdmap=1375308973}) 2024-09-11T01:43:07.464 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.460+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=23,osdmap=77} crc {osd_snap=1857286730,osdmap=2494920791}) 2024-09-11T01:43:07.464 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.461+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.464 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.461+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=23,osdmap=77} crc {osd_snap=2701503198,osdmap=1375308973}) 2024-09-11T01:43:07.464 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.461+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=23,osdmap=77} crc {osd_snap=2701503198,osdmap=2062379878}) 2024-09-11T01:43:07.464 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.462+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.464 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.462+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3619169781}) 2024-09-11T01:43:07.464 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.462+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3131773334}) 2024-09-11T01:43:07.464 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.462+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.464 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.462+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3619169781}) 2024-09-11T01:43:07.464 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.462+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2135542599}) 2024-09-11T01:43:07.465 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.463+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.465 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.463+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3582966387}) 2024-09-11T01:43:07.465 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.464+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3290388242}) 2024-09-11T01:43:07.465 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.464+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.466 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.464+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3582966387}) 2024-09-11T01:43:07.466 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.464+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3035951187}) 2024-09-11T01:43:07.469 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.468+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.469 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.468+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=8} crc {osdmap=3153394267}) 2024-09-11T01:43:07.469 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.468+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=7} crc {osdmap=2529517682}) 2024-09-11T01:43:07.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.468+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:43:07.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.468+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=8} crc {osdmap=3153394267}) 2024-09-11T01:43:07.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:43:07.468+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=7} crc {osdmap=3678218686}) 2024-09-11T01:43:07.885 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:43:07.886 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:43:07.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.997+0000 7fef37dcd640 10 monclient: get_monmap_and_config 2024-09-11T01:43:07.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.997+0000 7fef37dcd640 10 monclient: build_initial_monmap 2024-09-11T01:43:07.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.997+0000 7fef37dcd640 10 monclient: monmap: 2024-09-11T01:43:07.999 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:07.999 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:07.999 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:43:07.998512+0000 2024-09-11T01:43:07.999 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:43:07.998512+0000 2024-09-11T01:43:07.999 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:08.000 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.000 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:43:08.000 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:43:08.000 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:43:08.000 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:43:08.000 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:43:08.000 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.998+0000 7fef37dcd640 1 Processor -- start 2024-09-11T01:43:08.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.998+0000 7fef37dcd640 1 -- start start 2024-09-11T01:43:08.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.998+0000 7fef37dcd640 10 monclient: init 2024-09-11T01:43:08.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef37dcd640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:08.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef37dcd640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-11T01:43:08.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef37dcd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fef300a2e90 0x7fef30073d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef37dcd640 10 monclient(hunting): picked mon.noname-c con 0x7fef300a2e90 addr v2:172.21.15.71:3300/0 2024-09-11T01:43:08.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef37dcd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fef300a3270 0x7fef300a36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef37dcd640 10 monclient(hunting): picked mon.noname-a con 0x7fef300a3270 addr v2:172.21.15.12:3300/0 2024-09-11T01:43:08.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef37dcd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fef30135c50 addr v1:172.21.15.71:6789/0 2024-09-11T01:43:08.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef37dcd640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef300a46c0 con 0x7fef30135c50 2024-09-11T01:43:08.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef37dcd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef37dcd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fef30072730 con 0x7fef300a3270 2024-09-11T01:43:08.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef37dcd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef37dcd640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fef30072fa0 con 0x7fef300a2e90 2024-09-11T01:43:08.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef37dcd640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:08.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef37dcd640 10 monclient(hunting): authenticate will time out at 1594.441273s 2024-09-11T01:43:08.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef35368640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fef300a3270 0x7fef300a36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef35368640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fef300a3270 0x7fef300a36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34868/0 (socket says 172.21.15.12:34868) 2024-09-11T01:43:08.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef35368640 1 -- 172.21.15.12:0/691545802 learned_addr learned my addr 172.21.15.12:0/691545802 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:08.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef35368640 10 monclient(hunting): get_auth_request con 0x7fef300a3270 auth_method 0 2024-09-11T01:43:08.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef35368640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef35368640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef35368640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef35b69640 1 --2- 172.21.15.12:0/691545802 >> v2:172.21.15.71:3300/0 conn(0x7fef300a2e90 0x7fef30073d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef35368640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:08.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef35368640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:08.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef35368640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:08.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef35b69640 10 monclient(hunting): get_auth_request con 0x7fef300a2e90 auth_method 0 2024-09-11T01:43:08.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef35b69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef35b69640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef35b69640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef35368640 10 monclient(hunting): handle_auth_done global_id 94686 payload 274 2024-09-11T01:43:08.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef35368640 1 -- 172.21.15.12:0/691545802 >> v2:172.21.15.71:3300/0 conn(0x7fef300a2e90 msgr2=0x7fef30073d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:08.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef35368640 1 --2- 172.21.15.12:0/691545802 >> v2:172.21.15.71:3300/0 conn(0x7fef300a2e90 0x7fef30073d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef35368640 1 -- 172.21.15.12:0/691545802 >> v1:172.21.15.71:6789/0 conn(0x7fef30135c50 legacy=0x7fef301360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:08.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef35368640 10 monclient: _finish_hunting 0 2024-09-11T01:43:08.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef35368640 1 monclient: found mon.noname-a 2024-09-11T01:43:08.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef35368640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:08.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef35368640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:08.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef35368640 1 -- 172.21.15.12:0/691545802 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef30073280 con 0x7fef300a3270 2024-09-11T01:43:08.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:07.999+0000 7fef35b69640 1 --2- 172.21.15.12:0/691545802 >> v2:172.21.15.71:3300/0 conn(0x7fef300a2e90 0x7fef30073d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:43:08.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.002+0000 7fef35368640 1 -- 172.21.15.12:0/691545802 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef300a3270 msgr2=0x7fef300a36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:08.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.002+0000 7fef35368640 1 --2- 172.21.15.12:0/691545802 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef300a3270 0x7fef300a36a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fef300a4a20 tx=0x7fef2c02a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b82ebc6627edb33f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:08.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.002+0000 7fef34b67640 1 -- 172.21.15.12:0/691545802 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2c037070 con 0x7fef300a3270 2024-09-11T01:43:08.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.002+0000 7fef34b67640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:08.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.002+0000 7fef34b67640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:08.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.002+0000 7fef34b67640 10 monclient: dump: 2024-09-11T01:43:08.014 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:08.014 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.014 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.014 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.014 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:08.014 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.014 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:08.014 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:08.014 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:08.014 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.002+0000 7fef34b67640 10 monclient: _finish_auth 0 2024-09-11T01:43:08.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.002+0000 7fef34b67640 10 monclient: _check_auth_tickets 2024-09-11T01:43:08.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.002+0000 7fef34b67640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:08.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.002+0000 7fef34b67640 1 -- 172.21.15.12:0/691545802 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef2c032440 con 0x7fef300a3270 2024-09-11T01:43:08.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.002+0000 7fef37dcd640 5 monclient: authenticate success, global_id 94686 2024-09-11T01:43:08.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.002+0000 7fef37dcd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:43:08.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.002+0000 7fef34b67640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:08.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.002+0000 7fef34b67640 1 -- 172.21.15.12:0/691545802 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2c032810 con 0x7fef300a3270 2024-09-11T01:43:08.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.002+0000 7fef37dcd640 10 monclient: get_monmap_and_config success 2024-09-11T01:43:08.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.002+0000 7fef37dcd640 4 set_mon_vals no callback set 2024-09-11T01:43:08.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.002+0000 7fef34b67640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:08.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.002+0000 7fef34b67640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:08.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.002+0000 7fef34b67640 10 monclient: dump: 2024-09-11T01:43:08.015 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:08.015 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.016 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.016 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.016 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:08.016 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.016 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:08.016 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:08.016 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:08.016 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.003+0000 7fef37dcd640 10 monclient: shutdown 2024-09-11T01:43:08.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.003+0000 7fef37dcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:08.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.003+0000 7fef37dcd640 1 -- 172.21.15.12:0/691545802 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef300a3270 msgr2=0x7fef300a36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.003+0000 7fef37dcd640 1 --2- 172.21.15.12:0/691545802 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef300a3270 0x7fef300a36a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fef300a4a20 tx=0x7fef2c02a0d0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.003+0000 7fef37dcd640 1 -- 172.21.15.12:0/691545802 shutdown_connections 2024-09-11T01:43:08.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.003+0000 7fef37dcd640 1 --2- 172.21.15.12:0/691545802 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef300a3270 0x7fef300a36a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.003+0000 7fef37dcd640 1 --2- 172.21.15.12:0/691545802 >> v2:172.21.15.71:3300/0 conn(0x7fef300a2e90 0x7fef30073d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.003+0000 7fef37dcd640 1 -- 172.21.15.12:0/691545802 >> 172.21.15.12:0/691545802 conn(0x7fef300a8490 msgr2=0x7fef300a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:08.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.003+0000 7fef37dcd640 1 -- 172.21.15.12:0/691545802 shutdown_connections 2024-09-11T01:43:08.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.003+0000 7fef37dcd640 1 -- 172.21.15.12:0/691545802 wait complete. 2024-09-11T01:43:08.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.004+0000 7fef37dcd640 10 monclient: build_initial_monmap 2024-09-11T01:43:08.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.004+0000 7fef37dcd640 10 monclient: monmap: 2024-09-11T01:43:08.017 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:08.017 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:43:08.017 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:43:08.017 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:43:08.017 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:08.017 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.017 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:43:08.018 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:43:08.018 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:43:08.018 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.004+0000 7fef37dcd640 1 Processor -- start 2024-09-11T01:43:08.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.004+0000 7fef37dcd640 1 -- start start 2024-09-11T01:43:08.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.004+0000 7fef37dcd640 10 monclient: init 2024-09-11T01:43:08.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.004+0000 7fef37dcd640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:08.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.004+0000 7fef37dcd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:43:08.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.004+0000 7fef37dcd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef300a2e90 0x7fef301cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.004+0000 7fef37dcd640 10 monclient(hunting): picked mon.noname-a con 0x7fef300a2e90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:08.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.004+0000 7fef37dcd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fef300a3270 0x7fef301cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.004+0000 7fef37dcd640 10 monclient(hunting): picked mon.noname-c con 0x7fef300a3270 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:08.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.004+0000 7fef37dcd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef30135c50 0x7fef3006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.004+0000 7fef37dcd640 10 monclient(hunting): picked mon.noname-b con 0x7fef30135c50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:08.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.004+0000 7fef37dcd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.004+0000 7fef37dcd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fef30072300 con 0x7fef300a2e90 2024-09-11T01:43:08.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.004+0000 7fef37dcd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.004+0000 7fef37dcd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fef30074310 con 0x7fef30135c50 2024-09-11T01:43:08.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.004+0000 7fef37dcd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.004+0000 7fef37dcd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fef30072480 con 0x7fef300a3270 2024-09-11T01:43:08.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.004+0000 7fef37dcd640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:08.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.004+0000 7fef37dcd640 10 monclient(hunting): authenticate will time out at 1594.446996s 2024-09-11T01:43:08.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35b69640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef300a2e90 0x7fef301cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35b69640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef300a2e90 0x7fef301cd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34882/0 (socket says 172.21.15.12:34882) 2024-09-11T01:43:08.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35368640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fef300a3270 0x7fef301cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35b69640 1 -- 172.21.15.12:0/3672118589 learned_addr learned my addr 172.21.15.12:0/3672118589 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:08.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35b69640 10 monclient(hunting): get_auth_request con 0x7fef300a2e90 auth_method 0 2024-09-11T01:43:08.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35b69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35b69640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35b69640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35368640 10 monclient(hunting): get_auth_request con 0x7fef300a3270 auth_method 0 2024-09-11T01:43:08.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35368640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35368640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35368640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35b69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:08.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35b69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:08.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35b69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:08.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35368640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:08.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35368640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:08.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35368640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:08.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef3636a640 1 --2- 172.21.15.12:0/3672118589 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef30135c50 0x7fef3006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef3636a640 10 monclient(hunting): get_auth_request con 0x7fef30135c50 auth_method 0 2024-09-11T01:43:08.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef3636a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef3636a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef3636a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef3636a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:08.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef3636a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:08.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef3636a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:08.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35b69640 10 monclient(hunting): handle_auth_done global_id 94689 payload 824 2024-09-11T01:43:08.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35b69640 1 -- 172.21.15.12:0/3672118589 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fef300a3270 msgr2=0x7fef301cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35b69640 1 --2- 172.21.15.12:0/3672118589 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fef300a3270 0x7fef301cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35b69640 1 -- 172.21.15.12:0/3672118589 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef30135c50 msgr2=0x7fef3006dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35b69640 1 --2- 172.21.15.12:0/3672118589 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef30135c50 0x7fef3006dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35b69640 10 monclient: _finish_hunting 0 2024-09-11T01:43:08.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35b69640 1 monclient: found mon.noname-a 2024-09-11T01:43:08.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35b69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:08.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35b69640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:08.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35b69640 1 -- 172.21.15.12:0/3672118589 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef300a7d40 con 0x7fef300a2e90 2024-09-11T01:43:08.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.005+0000 7fef35368640 1 --2- 172.21.15.12:0/3672118589 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fef300a3270 0x7fef301cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:43:08.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.007+0000 7fef35b69640 1 -- 172.21.15.12:0/3672118589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef300a2e90 msgr2=0x7fef301cd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:08.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.007+0000 7fef35b69640 1 --2- 172.21.15.12:0/3672118589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef300a2e90 0x7fef301cd1a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fef2400e730 tx=0x7fef2400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:08.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.007+0000 7fef227fc640 1 -- 172.21.15.12:0/3672118589 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2400f040 con 0x7fef300a2e90 2024-09-11T01:43:08.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.007+0000 7fef227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:08.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.007+0000 7fef227fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:08.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.007+0000 7fef227fc640 10 monclient: dump: 2024-09-11T01:43:08.023 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:08.023 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.023 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.023 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.023 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:08.023 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.023 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:08.023 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:08.023 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:08.023 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.007+0000 7fef227fc640 10 monclient: _finish_auth 0 2024-09-11T01:43:08.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.007+0000 7fef227fc640 10 monclient: _check_auth_tickets 2024-09-11T01:43:08.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.007+0000 7fef227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:08.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.007+0000 7fef227fc640 1 -- 172.21.15.12:0/3672118589 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef24004230 con 0x7fef300a2e90 2024-09-11T01:43:08.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.007+0000 7fef37dcd640 5 monclient: authenticate success, global_id 94689 2024-09-11T01:43:08.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.007+0000 7fef37dcd640 10 monclient: _renew_subs 2024-09-11T01:43:08.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.007+0000 7fef37dcd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:08.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.007+0000 7fef37dcd640 1 -- 172.21.15.12:0/3672118589 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef3009f680 con 0x7fef300a2e90 2024-09-11T01:43:08.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.007+0000 7fef227fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:08.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.007+0000 7fef227fc640 1 -- 172.21.15.12:0/3672118589 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2400ee40 con 0x7fef300a2e90 2024-09-11T01:43:08.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.008+0000 7fef227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:08.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.008+0000 7fef227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:08.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.008+0000 7fef227fc640 10 monclient: dump: 2024-09-11T01:43:08.024 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:08.024 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.025 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.025 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.025 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:08.025 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.025 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:08.025 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:08.025 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:08.025 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.007+0000 7fef237fe640 4 set_mon_vals no callback set 2024-09-11T01:43:08.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.008+0000 7fef37dcd640 10 monclient: _renew_subs 2024-09-11T01:43:08.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.008+0000 7fef37dcd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:08.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.008+0000 7fef37dcd640 1 -- 172.21.15.12:0/3672118589 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef301cdc20 con 0x7fef300a2e90 2024-09-11T01:43:08.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.008+0000 7fef37dcd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:08.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.008+0000 7fef37dcd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:08.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.008+0000 7fef37dcd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:08.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.008+0000 7fef37dcd640 1 -- 172.21.15.12:0/3672118589 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef30136980 con 0x7fef300a2e90 2024-09-11T01:43:08.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.009+0000 7fef227fc640 1 -- 172.21.15.12:0/3672118589 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fef24012050 con 0x7fef300a2e90 2024-09-11T01:43:08.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.009+0000 7fef227fc640 1 --2- 172.21.15.12:0/3672118589 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fef1403e210 0x7fef140406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.009+0000 7fef227fc640 1 -- 172.21.15.12:0/3672118589 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(474..474 src has 285..474) ==== 4771+0+0 (secure 0 0 0) 0x7fef24052910 con 0x7fef300a2e90 2024-09-11T01:43:08.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.009+0000 7fef35368640 1 --2- 172.21.15.12:0/3672118589 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fef1403e210 0x7fef140406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.009+0000 7fef35368640 10 monclient: get_auth_request con 0x7fef1403e210 auth_method 0 2024-09-11T01:43:08.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.010+0000 7fef35368640 1 -- 172.21.15.12:0/3672118589 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fef1403e210 msgr2=0x7fef140406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:43:08.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.010+0000 7fef35368640 1 -- 172.21.15.12:0/3672118589 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fef1403e210 msgr2=0x7fef140406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:43:08.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.010+0000 7fef35368640 1 --2- 172.21.15.12:0/3672118589 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fef1403e210 0x7fef140406d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7fef30073560 tx=0x7fef2c02e000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:08.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.010+0000 7fef35368640 1 --2- 172.21.15.12:0/3672118589 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fef1403e210 0x7fef140406d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:08.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.013+0000 7fef227fc640 1 -- 172.21.15.12:0/3672118589 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fef24017020 con 0x7fef300a2e90 2024-09-11T01:43:08.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.014+0000 7fef227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:08.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.014+0000 7fef227fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:43:08.079 INFO:tasks.rados.rados.0.smithi071.stdout:2119: done (0 left) 2024-09-11T01:43:08.079 INFO:tasks.rados.rados.0.smithi071.stdout:2120: rollback oid 45 current snap is 233 2024-09-11T01:43:08.079 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 45 to 206 2024-09-11T01:43:08.079 INFO:tasks.rados.rados.0.smithi071.stdout:2121: read oid 31 snap -1 2024-09-11T01:43:08.079 INFO:tasks.rados.rados.0.smithi071.stdout:2121: expect (ObjNum 737 snap 226 seq_num 737) 2024-09-11T01:43:08.079 INFO:tasks.rados.rados.0.smithi071.stdout:2122: read oid 39 snap -1 2024-09-11T01:43:08.079 INFO:tasks.rados.rados.0.smithi071.stdout:2122: expect (ObjNum 752 snap 228 seq_num 752) 2024-09-11T01:43:08.079 INFO:tasks.rados.rados.0.smithi071.stdout:2123: delete oid 34 current snap is 233 2024-09-11T01:43:08.083 INFO:tasks.rados.rados.0.smithi071.stdout:2121: expect (ObjNum 737 snap 226 seq_num 737) 2024-09-11T01:43:08.098 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1441005 2024-09-11T01:43:08.139 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1555162 2024-09-11T01:43:08.193 INFO:tasks.rados.rados.0.smithi071.stdout:2122: expect (ObjNum 752 snap 228 seq_num 752) 2024-09-11T01:43:08.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.211+0000 7fef35368640 1 --2- 172.21.15.12:0/3672118589 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fef1403e210 0x7fef140406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.211+0000 7fef35368640 10 monclient: get_auth_request con 0x7fef1403e210 auth_method 0 2024-09-11T01:43:08.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.214+0000 7fef35368640 1 --2- 172.21.15.12:0/3672118589 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fef1403e210 0x7fef140406d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fef2c03d000 tx=0x7fef2c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:08.249 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1570812 2024-09-11T01:43:08.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.284+0000 7fef37dcd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:43:08.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.284+0000 7fef37dcd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:43:08.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.284+0000 7fef37dcd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:08.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.284+0000 7fef37dcd640 1 -- 172.21.15.12:0/3672118589 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fef300a1eb0 con 0x7fef300a2e90 2024-09-11T01:43:08.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.284+0000 7fef227fc640 1 -- 172.21.15.12:0/3672118589 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fef2400c8e0 con 0x7fef300a2e90 2024-09-11T01:43:08.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.284+0000 7fef227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:43:08.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.284+0000 7fef227fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:43:08.286 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":128,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:43:08.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.286+0000 7fef37dcd640 1 -- 172.21.15.12:0/3672118589 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fef1403e210 msgr2=0x7fef140406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.286+0000 7fef37dcd640 1 --2- 172.21.15.12:0/3672118589 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fef1403e210 0x7fef140406d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fef2c03d000 tx=0x7fef2c008030 comp rx=0 tx=0).stop 2024-09-11T01:43:08.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.286+0000 7fef37dcd640 10 monclient: shutdown 2024-09-11T01:43:08.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.286+0000 7fef37dcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:08.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.286+0000 7fef37dcd640 1 -- 172.21.15.12:0/3672118589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef300a2e90 msgr2=0x7fef301cd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.286+0000 7fef37dcd640 1 --2- 172.21.15.12:0/3672118589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef300a2e90 0x7fef301cd1a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fef2400e730 tx=0x7fef2400ebf0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.286+0000 7fef37dcd640 1 -- 172.21.15.12:0/3672118589 shutdown_connections 2024-09-11T01:43:08.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.286+0000 7fef37dcd640 1 --2- 172.21.15.12:0/3672118589 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef30135c50 0x7fef3006dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.286+0000 7fef37dcd640 1 --2- 172.21.15.12:0/3672118589 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fef300a3270 0x7fef301cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.286+0000 7fef37dcd640 1 --2- 172.21.15.12:0/3672118589 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fef1403e210 0x7fef140406d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.286+0000 7fef37dcd640 1 --2- 172.21.15.12:0/3672118589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fef300a2e90 0x7fef301cd1a0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.286+0000 7fef37dcd640 1 -- 172.21.15.12:0/3672118589 >> 172.21.15.12:0/3672118589 conn(0x7fef300a8490 msgr2=0x7fef300a6bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:08.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.286+0000 7fef37dcd640 1 -- 172.21.15.12:0/3672118589 shutdown_connections 2024-09-11T01:43:08.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.286+0000 7fef37dcd640 1 -- 172.21.15.12:0/3672118589 wait complete. 2024-09-11T01:43:08.298 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:43:08.298 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:43:08.298 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:43:08.299 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:43:08.332 INFO:tasks.rados.rados.0.smithi071.stdout:2120: finishing rollback tid 0 to smithi07134777-45 2024-09-11T01:43:08.333 INFO:tasks.rados.rados.0.smithi071.stdout:2120: finishing rollback tid 1 to smithi07134777-45 2024-09-11T01:43:08.333 INFO:tasks.rados.rados.0.smithi071.stdout:2120: finishing rollback tid 2 to smithi07134777-45 2024-09-11T01:43:08.333 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 967 (ObjNum 612 snap 181 seq_num 612) dirty exists 2024-09-11T01:43:08.333 INFO:tasks.rados.rados.0.smithi071.stdout:2120: done (3 left) 2024-09-11T01:43:08.333 INFO:tasks.rados.rados.0.smithi071.stdout:2121: done (2 left) 2024-09-11T01:43:08.333 INFO:tasks.rados.rados.0.smithi071.stdout:2122: done (1 left) 2024-09-11T01:43:08.333 INFO:tasks.rados.rados.0.smithi071.stdout:2123: done (0 left) 2024-09-11T01:43:08.333 INFO:tasks.rados.rados.0.smithi071.stdout:2124: rollback oid 34 current snap is 233 2024-09-11T01:43:08.333 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 34 to 174 2024-09-11T01:43:08.333 INFO:tasks.rados.rados.0.smithi071.stdout:2125: write oid 7 current snap is 233 2024-09-11T01:43:08.333 INFO:tasks.rados.rados.0.smithi071.stdout:2125: seq_num 769 ranges {524583=418134,1649947=629258,2834752=89831} 2024-09-11T01:43:08.337 INFO:tasks.rados.rados.0.smithi071.stdout:2125: writing smithi07134777-7 from 524583 to 942717 tid 1 2024-09-11T01:43:08.345 INFO:tasks.rados.rados.0.smithi071.stdout:2125: writing smithi07134777-7 from 1649947 to 2279205 tid 2 2024-09-11T01:43:08.348 INFO:tasks.rados.rados.0.smithi071.stdout:2125: writing smithi07134777-7 from 2834752 to 2924583 tid 3 2024-09-11T01:43:08.348 INFO:tasks.rados.rados.0.smithi071.stdout:2126: snap_remove snap 218 2024-09-11T01:43:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.400+0000 7f6ed5662640 10 monclient: get_monmap_and_config 2024-09-11T01:43:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.400+0000 7f6ed5662640 10 monclient: build_initial_monmap 2024-09-11T01:43:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.400+0000 7f6ed5662640 10 monclient: monmap: 2024-09-11T01:43:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:43:08.401407+0000 2024-09-11T01:43:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:43:08.401407+0000 2024-09-11T01:43:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:43:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:43:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:43:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:43:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:43:08.403 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.401+0000 7f6ed5662640 1 Processor -- start 2024-09-11T01:43:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.401+0000 7f6ed5662640 1 -- start start 2024-09-11T01:43:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.401+0000 7f6ed5662640 10 monclient: init 2024-09-11T01:43:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.401+0000 7f6ed5662640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.401+0000 7f6ed5662640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-11T01:43:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.401+0000 7f6ed5662640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6ed00a2e20 0x7f6ed006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.401+0000 7f6ed5662640 10 monclient(hunting): picked mon.noname-b con 0x7f6ed00a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6ed5662640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6ed00a4970 addr v1:172.21.15.71:6789/0 2024-09-11T01:43:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6ed5662640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6ed006f910 0x7f6ed013f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6ed5662640 10 monclient(hunting): picked mon.noname-a con 0x7f6ed006f910 addr v2:172.21.15.12:3300/0 2024-09-11T01:43:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6ed5662640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ed006f4f0 con 0x7f6ed00a4970 2024-09-11T01:43:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6ed5662640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6ed5662640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f6ed006ef30 con 0x7f6ed006f910 2024-09-11T01:43:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6ed5662640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6ed5662640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6ed006ec50 con 0x7f6ed00a2e20 2024-09-11T01:43:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6ed5662640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6ed5662640 10 monclient(hunting): authenticate will time out at 1594.844057s 2024-09-11T01:43:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6ecf7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6ed006f910 0x7f6ed013f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6eceffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6ed00a2e20 0x7f6ed006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6ecf7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6ed006f910 0x7f6ed013f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34892/0 (socket says 172.21.15.12:34892) 2024-09-11T01:43:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6ecf7fe640 1 -- 172.21.15.12:0/2439919900 learned_addr learned my addr 172.21.15.12:0/2439919900 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6eceffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6ed00a2e20 0x7f6ed006ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41252/0 (socket says 172.21.15.12:41252) 2024-09-11T01:43:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6ecf7fe640 10 monclient(hunting): get_auth_request con 0x7f6ed006f910 auth_method 0 2024-09-11T01:43:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6ecf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6ecf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6ecf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6eceffd640 10 monclient(hunting): get_auth_request con 0x7f6ed00a2e20 auth_method 0 2024-09-11T01:43:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6eceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6eceffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6eceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6ecf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6ecf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6ecf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6eceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6eceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6eceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6eceffd640 10 monclient(hunting): handle_auth_done global_id 154118 payload 274 2024-09-11T01:43:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6eceffd640 1 -- 172.21.15.12:0/2439919900 >> v2:172.21.15.12:3300/0 conn(0x7f6ed006f910 msgr2=0x7f6ed013f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6eceffd640 1 --2- 172.21.15.12:0/2439919900 >> v2:172.21.15.12:3300/0 conn(0x7f6ed006f910 0x7f6ed013f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6eceffd640 1 -- 172.21.15.12:0/2439919900 >> v1:172.21.15.71:6789/0 conn(0x7f6ed00a4970 legacy=0x7f6ed00a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6eceffd640 10 monclient: _finish_hunting 0 2024-09-11T01:43:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6eceffd640 1 monclient: found mon.noname-b 2024-09-11T01:43:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6eceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6eceffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:43:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6eceffd640 1 -- 172.21.15.12:0/2439919900 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ed00a3910 con 0x7f6ed00a2e20 2024-09-11T01:43:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.402+0000 7f6ecf7fe640 1 --2- 172.21.15.12:0/2439919900 >> v2:172.21.15.12:3300/0 conn(0x7f6ed006f910 0x7f6ed013f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:43:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.405+0000 7f6eceffd640 1 -- 172.21.15.12:0/2439919900 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6ed00a2e20 msgr2=0x7f6ed006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.405+0000 7f6eceffd640 1 --2- 172.21.15.12:0/2439919900 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6ed00a2e20 0x7f6ed006ded0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6ec00097a0 tx=0x7f6ec0029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59be361646aa471 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.405+0000 7f6ece7fc640 1 -- 172.21.15.12:0/2439919900 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec0037070 con 0x7f6ed00a2e20 2024-09-11T01:43:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.405+0000 7f6ece7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.405+0000 7f6ece7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:43:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.405+0000 7f6ece7fc640 10 monclient: dump: 2024-09-11T01:43:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:08.412 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.405+0000 7f6ece7fc640 10 monclient: _finish_auth 0 2024-09-11T01:43:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.405+0000 7f6ece7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:43:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.405+0000 7f6ece7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.405+0000 7f6ece7fc640 1 -- 172.21.15.12:0/2439919900 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ec0032440 con 0x7f6ed00a2e20 2024-09-11T01:43:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.405+0000 7f6ed5662640 5 monclient: authenticate success, global_id 154118 2024-09-11T01:43:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.405+0000 7f6ed5662640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:43:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.405+0000 7f6ece7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.405+0000 7f6ed5662640 10 monclient: get_monmap_and_config success 2024-09-11T01:43:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.405+0000 7f6ece7fc640 1 -- 172.21.15.12:0/2439919900 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec0032810 con 0x7f6ed00a2e20 2024-09-11T01:43:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.405+0000 7f6ece7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.405+0000 7f6ed5662640 4 set_mon_vals no callback set 2024-09-11T01:43:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.405+0000 7f6ece7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:43:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.405+0000 7f6ece7fc640 10 monclient: dump: 2024-09-11T01:43:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:08.414 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.406+0000 7f6ed5662640 10 monclient: shutdown 2024-09-11T01:43:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.406+0000 7f6ed5662640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.406+0000 7f6ed5662640 1 -- 172.21.15.12:0/2439919900 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6ed00a2e20 msgr2=0x7f6ed006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.406+0000 7f6ed5662640 1 --2- 172.21.15.12:0/2439919900 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6ed00a2e20 0x7f6ed006ded0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6ec00097a0 tx=0x7f6ec0029fd0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.406+0000 7f6ed5662640 1 -- 172.21.15.12:0/2439919900 shutdown_connections 2024-09-11T01:43:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.406+0000 7f6ed5662640 1 --2- 172.21.15.12:0/2439919900 >> v2:172.21.15.12:3300/0 conn(0x7f6ed006f910 0x7f6ed013f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.406+0000 7f6ed5662640 1 --2- 172.21.15.12:0/2439919900 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6ed00a2e20 0x7f6ed006ded0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.406+0000 7f6ed5662640 1 -- 172.21.15.12:0/2439919900 >> 172.21.15.12:0/2439919900 conn(0x7f6ed00a8600 msgr2=0x7f6ed00a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.406+0000 7f6ed5662640 1 -- 172.21.15.12:0/2439919900 shutdown_connections 2024-09-11T01:43:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.406+0000 7f6ed5662640 1 -- 172.21.15.12:0/2439919900 wait complete. 2024-09-11T01:43:08.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.406+0000 7f6ed5662640 10 monclient: build_initial_monmap 2024-09-11T01:43:08.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.406+0000 7f6ed5662640 10 monclient: monmap: 2024-09-11T01:43:08.420 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:08.420 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:43:08.420 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:43:08.420 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:43:08.420 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:08.420 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.420 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:43:08.420 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:43:08.421 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:43:08.421 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ed5662640 1 Processor -- start 2024-09-11T01:43:08.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ed5662640 1 -- start start 2024-09-11T01:43:08.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ed5662640 10 monclient: init 2024-09-11T01:43:08.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ed5662640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:08.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ed5662640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:43:08.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ed5662640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ed006f910 0x7f6ed01398c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ed5662640 10 monclient(hunting): picked mon.noname-a con 0x7f6ed006f910 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:08.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ed5662640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6ed00a2e20 0x7f6ed0139e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ed5662640 10 monclient(hunting): picked mon.noname-b con 0x7f6ed00a2e20 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:08.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ed5662640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6ed00a4970 0x7f6ed01ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ed5662640 10 monclient(hunting): picked mon.noname-c con 0x7f6ed00a4970 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:08.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ed5662640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ed5662640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed013fd00 con 0x7f6ed006f910 2024-09-11T01:43:08.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ed5662640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ed5662640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed013fb80 con 0x7f6ed00a2e20 2024-09-11T01:43:08.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ed5662640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ed5662640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6ed013fe80 con 0x7f6ed00a4970 2024-09-11T01:43:08.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ed5662640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:08.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ed5662640 10 monclient(hunting): authenticate will time out at 1594.849746s 2024-09-11T01:43:08.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6eceffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ed006f910 0x7f6ed01398c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6eceffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ed006f910 0x7f6ed01398c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34900/0 (socket says 172.21.15.12:34900) 2024-09-11T01:43:08.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6eceffd640 1 -- 172.21.15.12:0/3518582428 learned_addr learned my addr 172.21.15.12:0/3518582428 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:08.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6eceffd640 10 monclient(hunting): get_auth_request con 0x7f6ed006f910 auth_method 0 2024-09-11T01:43:08.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6eceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6eceffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6eceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ecf7fe640 1 --2- 172.21.15.12:0/3518582428 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6ed00a4970 0x7f6ed01ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ecf7fe640 10 monclient(hunting): get_auth_request con 0x7f6ed00a4970 auth_method 0 2024-09-11T01:43:08.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ecf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ecf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ecf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6eceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:08.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6eceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:08.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6eceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:08.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ecf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:08.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ecf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:08.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.407+0000 7f6ecf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:08.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.408+0000 7f6ec7fff640 1 --2- 172.21.15.12:0/3518582428 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6ed00a2e20 0x7f6ed0139e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.408+0000 7f6eceffd640 10 monclient(hunting): handle_auth_done global_id 94695 payload 824 2024-09-11T01:43:08.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.408+0000 7f6eceffd640 1 -- 172.21.15.12:0/3518582428 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6ed00a4970 msgr2=0x7f6ed01ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.408+0000 7f6eceffd640 1 --2- 172.21.15.12:0/3518582428 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6ed00a4970 0x7f6ed01ccfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.408+0000 7f6eceffd640 1 -- 172.21.15.12:0/3518582428 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6ed00a2e20 msgr2=0x7f6ed0139e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.408+0000 7f6eceffd640 1 --2- 172.21.15.12:0/3518582428 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6ed00a2e20 0x7f6ed0139e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.408+0000 7f6eceffd640 10 monclient: _finish_hunting 0 2024-09-11T01:43:08.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.408+0000 7f6eceffd640 1 monclient: found mon.noname-a 2024-09-11T01:43:08.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.408+0000 7f6eceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:08.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.408+0000 7f6eceffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:08.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.408+0000 7f6eceffd640 1 -- 172.21.15.12:0/3518582428 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ed013f760 con 0x7f6ed006f910 2024-09-11T01:43:08.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.408+0000 7f6ec7fff640 10 monclient: get_auth_request con 0x7f6ed00a2e20 auth_method 0 2024-09-11T01:43:08.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.408+0000 7f6ec7fff640 1 --2- 172.21.15.12:0/3518582428 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6ed00a2e20 0x7f6ed0139e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:43:08.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.410+0000 7f6eceffd640 1 -- 172.21.15.12:0/3518582428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ed006f910 msgr2=0x7f6ed01398c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:08.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.410+0000 7f6eceffd640 1 --2- 172.21.15.12:0/3518582428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ed006f910 0x7f6ed01398c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6ec00098c0 tx=0x7f6ec00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:08.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.410+0000 7f6ec77fe640 1 -- 172.21.15.12:0/3518582428 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec0037040 con 0x7f6ed006f910 2024-09-11T01:43:08.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.410+0000 7f6ec77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:08.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.410+0000 7f6ec77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:08.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.410+0000 7f6ec77fe640 10 monclient: dump: 2024-09-11T01:43:08.425 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:08.425 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.425 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.425 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.425 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:08.426 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.426 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:08.426 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:08.426 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:08.426 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.410+0000 7f6ec77fe640 10 monclient: _finish_auth 0 2024-09-11T01:43:08.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.410+0000 7f6ec77fe640 10 monclient: _check_auth_tickets 2024-09-11T01:43:08.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.410+0000 7f6ec77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:08.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.410+0000 7f6ec77fe640 1 -- 172.21.15.12:0/3518582428 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ec003b6c0 con 0x7f6ed006f910 2024-09-11T01:43:08.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.410+0000 7f6ed5662640 5 monclient: authenticate success, global_id 94695 2024-09-11T01:43:08.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.410+0000 7f6ed5662640 10 monclient: _renew_subs 2024-09-11T01:43:08.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.410+0000 7f6ed5662640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:08.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.410+0000 7f6ed5662640 1 -- 172.21.15.12:0/3518582428 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ed009f610 con 0x7f6ed006f910 2024-09-11T01:43:08.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.410+0000 7f6ec77fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:08.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.410+0000 7f6ec77fe640 1 -- 172.21.15.12:0/3518582428 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec003ba90 con 0x7f6ed006f910 2024-09-11T01:43:08.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.410+0000 7f6ec77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:08.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.410+0000 7f6ec77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:08.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.410+0000 7f6ec77fe640 10 monclient: dump: 2024-09-11T01:43:08.427 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:08.427 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.427 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.427 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.427 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:08.427 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.427 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:08.427 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:08.428 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:08.428 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.410+0000 7f6ed5662640 10 monclient: _renew_subs 2024-09-11T01:43:08.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.410+0000 7f6ecdffb640 4 set_mon_vals no callback set 2024-09-11T01:43:08.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.410+0000 7f6ed5662640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:08.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.410+0000 7f6ed5662640 1 -- 172.21.15.12:0/3518582428 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ed013c030 con 0x7f6ed006f910 2024-09-11T01:43:08.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.411+0000 7f6ed5662640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:08.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.411+0000 7f6ed5662640 1 --2- 172.21.15.12:0/3518582428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ed01cd520 0x7f6ed01cd900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.411+0000 7f6ed5662640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.411+0000 7f6ed5662640 1 -- 172.21.15.12:0/3518582428 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed006a060 con 0x7f6ed01cd520 2024-09-11T01:43:08.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.411+0000 7f6ec77fe640 1 -- 172.21.15.12:0/3518582428 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f6ec003a030 con 0x7f6ed006f910 2024-09-11T01:43:08.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.411+0000 7f6ec7fff640 1 --2- 172.21.15.12:0/3518582428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ed01cd520 0x7f6ed01cd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.411+0000 7f6ec7fff640 10 monclient: get_auth_request con 0x7f6ed01cd520 auth_method 0 2024-09-11T01:43:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.411+0000 7f6ec7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.411+0000 7f6ec7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.411+0000 7f6ec7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.411+0000 7f6ec7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.411+0000 7f6ec7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.411+0000 7f6ec7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.411+0000 7f6ec7fff640 10 monclient(hunting): handle_auth_done global_id 94698 payload 824 2024-09-11T01:43:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.411+0000 7f6ec7fff640 1 -- 172.21.15.12:0/3518582428 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6ed01cde40 con 0x7f6ed01cd520 2024-09-11T01:43:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.411+0000 7f6ec77fe640 1 --2- 172.21.15.12:0/3518582428 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6eb003e260 0x7f6eb0040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.411+0000 7f6ec77fe640 1 -- 172.21.15.12:0/3518582428 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(475..475 src has 285..475) ==== 4815+0+0 (secure 0 0 0) 0x7f6ec0072e80 con 0x7f6ed006f910 2024-09-11T01:43:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.412+0000 7f6ecf7fe640 1 --2- 172.21.15.12:0/3518582428 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6eb003e260 0x7f6eb0040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.412+0000 7f6ecf7fe640 10 monclient: get_auth_request con 0x7f6eb003e260 auth_method 0 2024-09-11T01:43:08.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.414+0000 7f6ec7fff640 1 -- 172.21.15.12:0/3518582428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ed01cd520 msgr2=0x7f6ed01cd900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.414+0000 7f6ec7fff640 1 --2- 172.21.15.12:0/3518582428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ed01cd520 0x7f6ed01cd900 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6ed013f4e0 tx=0x7f6eb800d950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.414+0000 7f6ec77fe640 1 -- 172.21.15.12:0/3518582428 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8012a10 con 0x7f6ed01cd520 2024-09-11T01:43:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.414+0000 7f6ec77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.414+0000 7f6ec77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.414+0000 7f6ec77fe640 10 monclient: dump: 2024-09-11T01:43:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:08.430 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:08.431 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.414+0000 7f6ec77fe640 1 -- 172.21.15.12:0/3518582428 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6eb8012a10 con 0x7f6ed01cd520 2024-09-11T01:43:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.414+0000 7f6ec77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.414+0000 7f6ec77fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:43:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.414+0000 7f6ec77fe640 1 -- 172.21.15.12:0/3518582428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ed01cd520 msgr2=0x7f6ed01cd900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.414+0000 7f6ec77fe640 1 --2- 172.21.15.12:0/3518582428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ed01cd520 0x7f6ed01cd900 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6ed013f4e0 tx=0x7f6eb800d950 comp rx=0 tx=0).stop 2024-09-11T01:43:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.414+0000 7f6ec77fe640 1 -- 172.21.15.12:0/3518582428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ed01cd520 msgr2=0x7f6ed01cd900 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:43:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.414+0000 7f6ec77fe640 1 --2- 172.21.15.12:0/3518582428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ed01cd520 0x7f6ed01cd900 secure :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6ed013f4e0 tx=0x7f6eb800d950 comp rx=0 tx=0).stop 2024-09-11T01:43:08.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.414+0000 7f6ecf7fe640 1 --2- 172.21.15.12:0/3518582428 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6eb003e260 0x7f6eb0040720 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8009c90 tx=0x7f6ec800b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:08.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.420+0000 7f6ed5662640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:43:08.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.420+0000 7f6ed5662640 1 --2- 172.21.15.12:0/3518582428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ed009d330 0x7f6ed009d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.420+0000 7f6ed5662640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.420+0000 7f6ed5662640 1 -- 172.21.15.12:0/3518582428 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed006ded0 con 0x7f6ed009d330 2024-09-11T01:43:08.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.420+0000 7f6ec7fff640 1 --2- 172.21.15.12:0/3518582428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ed009d330 0x7f6ed009d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.420+0000 7f6ec7fff640 10 monclient: get_auth_request con 0x7f6ed009d330 auth_method 0 2024-09-11T01:43:08.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.420+0000 7f6ec7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.420+0000 7f6ec7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.420+0000 7f6ec7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.420+0000 7f6ec7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:08.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.420+0000 7f6ec7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:08.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.420+0000 7f6ec7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:08.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.420+0000 7f6ec7fff640 10 monclient(hunting): handle_auth_done global_id 94701 payload 824 2024-09-11T01:43:08.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.420+0000 7f6ec7fff640 1 -- 172.21.15.12:0/3518582428 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6ed013ef00 con 0x7f6ed009d330 2024-09-11T01:43:08.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.423+0000 7f6ec7fff640 1 -- 172.21.15.12:0/3518582428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ed009d330 msgr2=0x7f6ed009d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:08.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.423+0000 7f6ec7fff640 1 --2- 172.21.15.12:0/3518582428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ed009d330 0x7f6ed009d740 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f6ed013f4e0 tx=0x7f6eb8003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:08.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.423+0000 7f6ec77fe640 1 -- 172.21.15.12:0/3518582428 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8028020 con 0x7f6ed009d330 2024-09-11T01:43:08.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.423+0000 7f6ec77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:08.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.423+0000 7f6ec77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:08.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.423+0000 7f6ec77fe640 10 monclient: dump: 2024-09-11T01:43:08.434 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:08.435 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.435 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.435 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.435 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:08.435 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.435 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:08.435 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:08.435 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:08.435 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.423+0000 7f6ec77fe640 1 -- 172.21.15.12:0/3518582428 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f6eb8006df0 con 0x7f6ed009d330 2024-09-11T01:43:08.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.423+0000 7f6ec77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:43:08.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.423+0000 7f6ec77fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:43:08.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.423+0000 7f6ec77fe640 1 -- 172.21.15.12:0/3518582428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ed009d330 msgr2=0x7f6ed009d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.423+0000 7f6ec77fe640 1 --2- 172.21.15.12:0/3518582428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ed009d330 0x7f6ed009d740 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f6ed013f4e0 tx=0x7f6eb8003760 comp rx=0 tx=0).stop 2024-09-11T01:43:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.423+0000 7f6ec77fe640 1 -- 172.21.15.12:0/3518582428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ed009d330 msgr2=0x7f6ed009d740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:43:08.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.423+0000 7f6ec77fe640 1 --2- 172.21.15.12:0/3518582428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ed009d330 0x7f6ed009d740 secure :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f6ed013f4e0 tx=0x7f6eb8003760 comp rx=0 tx=0).stop 2024-09-11T01:43:08.436 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:43:08.436 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:43:08.436 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:43:08.436 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:43:08.436 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 128, 2024-09-11T01:43:08.436 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 334519, 2024-09-11T01:43:08.436 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:43:08.437 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:43:08.437 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:43:08.437 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:43:08.437 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:08.437 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 6, 2024-09-11T01:43:08.437 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:43:08.437 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:43:08.437 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:43:08.437 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:08.437 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:08.437 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:08.437 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:08.437 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:08.438 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:08.438 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:08.438 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:08.438 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:08.438 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:08.438 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:08.438 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:08.438 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:43:08.438 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:43:08.438 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:08.438 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:08.438 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:08.439 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:08.439 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:08.439 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:08.439 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:08.439 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:08.439 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:08.439 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:08.439 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:08.439 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:08.439 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.439 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:43:08.439 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:43:08.439 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:43:08.439 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:43:08.440 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:43:08.440 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:43:08.440 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:43:08.440 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:43:08.440 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:43:08.440 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:43:08.440 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:43:08.440 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:43:08.440 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:43:08.440 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:43:08.440 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:43:08.440 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:43:08.440 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:43:08.441 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:08.441 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:08.441 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:08.441 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:08.441 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:08.441 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:08.441 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:08.441 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:08.441 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:08.441 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:08.441 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:08.441 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:08.441 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:43:08.442 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.442 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:43:08.442 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.442 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:43:08.442 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:43:08.442 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:08.442 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:08.442 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.442 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:08.442 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:43:08.442 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:08.442 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.442 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.442 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:08.443 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:43:08.443 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:08.443 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:08.443 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:08.443 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.443 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:43:08.443 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:43:08.443 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:08.443 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:08.443 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:08.443 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.443 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.443 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:43:08.443 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:43:08.444 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:08.444 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:08.444 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.444 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:08.444 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:43:08.444 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:08.444 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.444 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.444 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:08.444 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:43:08.444 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:08.444 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:08.444 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:08.445 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.445 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:43:08.445 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:43:08.445 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:08.445 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:08.445 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:08.445 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.445 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.445 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:43:08.445 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:43:08.445 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:08.445 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:08.445 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.445 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:08.446 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:43:08.446 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:08.446 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.446 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.446 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:08.446 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:43:08.446 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:08.446 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:08.446 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:08.446 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.446 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:43:08.446 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:43:08.446 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:08.446 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:08.447 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:08.447 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:08.447 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:08.447 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.447 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:43:08.447 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:43:08.447 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.447 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:08.447 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:08.447 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:43:08.447 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:08.447 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:08.447 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:43:08.448 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.448 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:08.448 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:08.448 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:43:08.448 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:08.448 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:08.448 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:43:08.448 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.448 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:08.448 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:08.448 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:43:08.448 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:08.448 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:08.449 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.449 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:43:08.449 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:43:08.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.423+0000 7f6ed5662640 1 -- 172.21.15.12:0/3518582428 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6eb003e260 msgr2=0x7f6eb0040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.423+0000 7f6ed5662640 1 --2- 172.21.15.12:0/3518582428 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6eb003e260 0x7f6eb0040720 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8009c90 tx=0x7f6ec800b040 comp rx=0 tx=0).stop 2024-09-11T01:43:08.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.423+0000 7f6ed5662640 10 monclient: shutdown 2024-09-11T01:43:08.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.423+0000 7f6ed5662640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:08.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.423+0000 7f6ed5662640 1 -- 172.21.15.12:0/3518582428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ed006f910 msgr2=0x7f6ed01398c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.423+0000 7f6ed5662640 1 --2- 172.21.15.12:0/3518582428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ed006f910 0x7f6ed01398c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6ec00098c0 tx=0x7f6ec00046c0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.423+0000 7f6ecf7fe640 1 -- 172.21.15.12:0/3518582428 reap_dead start 2024-09-11T01:43:08.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.423+0000 7f6ed5662640 1 -- 172.21.15.12:0/3518582428 shutdown_connections 2024-09-11T01:43:08.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.423+0000 7f6ed5662640 1 --2- 172.21.15.12:0/3518582428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6ed006f910 0x7f6ed01398c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.423+0000 7f6ed5662640 1 -- 172.21.15.12:0/3518582428 >> 172.21.15.12:0/3518582428 conn(0x7f6ed00a8600 msgr2=0x7f6ed00a6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:08.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.423+0000 7f6ed5662640 1 -- 172.21.15.12:0/3518582428 shutdown_connections 2024-09-11T01:43:08.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.423+0000 7f6ed5662640 1 -- 172.21.15.12:0/3518582428 wait complete. 2024-09-11T01:43:08.451 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:43:08.451 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:43:08.452 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:43:08.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.551+0000 7fe47ff94640 10 monclient: get_monmap_and_config 2024-09-11T01:43:08.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.551+0000 7fe47ff94640 10 monclient: build_initial_monmap 2024-09-11T01:43:08.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.551+0000 7fe47ff94640 10 monclient: monmap: 2024-09-11T01:43:08.553 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:08.553 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.553 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:43:08.552600+0000 2024-09-11T01:43:08.553 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:43:08.552600+0000 2024-09-11T01:43:08.553 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:08.554 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.554 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:43:08.554 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:43:08.554 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:43:08.554 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:43:08.554 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:43:08.554 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.552+0000 7fe47ff94640 1 Processor -- start 2024-09-11T01:43:08.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.552+0000 7fe47ff94640 1 -- start start 2024-09-11T01:43:08.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.552+0000 7fe47ff94640 10 monclient: init 2024-09-11T01:43:08.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47ff94640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:08.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47ff94640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-11T01:43:08.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47ff94640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe4780a1380 0x7fe4780a1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47ff94640 10 monclient(hunting): picked mon.noname-a con 0x7fe4780a1380 addr v2:172.21.15.12:3300/0 2024-09-11T01:43:08.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47ff94640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4780a0370 0x7fe478135ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47ff94640 10 monclient(hunting): picked mon.noname-b con 0x7fe4780a0370 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:08.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47ff94640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe4780a0c90 addr v1:172.21.15.12:6789/0 2024-09-11T01:43:08.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47ff94640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4780a1ca0 con 0x7fe4780a0c90 2024-09-11T01:43:08.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47ff94640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47ff94640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fe4780a21c0 con 0x7fe4780a1380 2024-09-11T01:43:08.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47ff94640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47ff94640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe4780a24a0 con 0x7fe4780a0370 2024-09-11T01:43:08.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47ff94640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:08.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47ff94640 10 monclient(hunting): authenticate will time out at 1594.995222s 2024-09-11T01:43:08.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47dd30640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe4780a1380 0x7fe4780a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47d52f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4780a0370 0x7fe478135ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47e531640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe4780a0c90 0x7fe478136020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37662/0 (socket says 172.21.15.12:37662) 2024-09-11T01:43:08.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47e531640 1 -- 172.21.15.12:0/2801798442 learned_addr learned my addr 172.21.15.12:0/2801798442 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:43:08.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47dd30640 10 monclient(hunting): get_auth_request con 0x7fe4780a1380 auth_method 0 2024-09-11T01:43:08.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47dd30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47dd30640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47dd30640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47d52f640 10 monclient(hunting): get_auth_request con 0x7fe4780a0370 auth_method 0 2024-09-11T01:43:08.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47d52f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47d52f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47d52f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47dd30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:08.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47dd30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:08.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47dd30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:08.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47d52f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:08.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47d52f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:08.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47d52f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:08.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47dd30640 10 monclient(hunting): handle_auth_done global_id 94704 payload 274 2024-09-11T01:43:08.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47dd30640 1 -- 172.21.15.12:0/2801798442 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4780a0370 msgr2=0x7fe478135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:08.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47dd30640 1 --2- 172.21.15.12:0/2801798442 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4780a0370 0x7fe478135ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47dd30640 1 -- 172.21.15.12:0/2801798442 >> v1:172.21.15.12:6789/0 conn(0x7fe4780a0c90 legacy=0x7fe478136020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:08.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47dd30640 10 monclient: _finish_hunting 0 2024-09-11T01:43:08.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47dd30640 1 monclient: found mon.noname-a 2024-09-11T01:43:08.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47dd30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:08.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47dd30640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:08.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47dd30640 1 -- 172.21.15.12:0/2801798442 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4780a2780 con 0x7fe4780a1380 2024-09-11T01:43:08.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.553+0000 7fe47d52f640 1 --2- 172.21.15.12:0/2801798442 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4780a0370 0x7fe478135ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:43:08.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.556+0000 7fe47dd30640 1 -- 172.21.15.12:0/2801798442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4780a1380 msgr2=0x7fe4780a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.556+0000 7fe47dd30640 1 --2- 172.21.15.12:0/2801798442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4780a1380 0x7fe4780a1760 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe4740097a0 tx=0x7fe474029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ba71eef0325a94a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.556+0000 7fe47cd2e640 1 -- 172.21.15.12:0/2801798442 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe474037070 con 0x7fe4780a1380 2024-09-11T01:43:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.556+0000 7fe47cd2e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.556+0000 7fe47cd2e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.556+0000 7fe47cd2e640 10 monclient: dump: 2024-09-11T01:43:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:08.567 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:08.568 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:08.568 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.556+0000 7fe47cd2e640 10 monclient: _finish_auth 0 2024-09-11T01:43:08.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.556+0000 7fe47cd2e640 10 monclient: _check_auth_tickets 2024-09-11T01:43:08.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.556+0000 7fe47cd2e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:08.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.556+0000 7fe47cd2e640 1 -- 172.21.15.12:0/2801798442 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe474032440 con 0x7fe4780a1380 2024-09-11T01:43:08.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.556+0000 7fe47ff94640 5 monclient: authenticate success, global_id 94704 2024-09-11T01:43:08.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.556+0000 7fe47ff94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:43:08.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.556+0000 7fe47cd2e640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:08.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.556+0000 7fe47cd2e640 1 -- 172.21.15.12:0/2801798442 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe474032810 con 0x7fe4780a1380 2024-09-11T01:43:08.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.556+0000 7fe47ff94640 10 monclient: get_monmap_and_config success 2024-09-11T01:43:08.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.556+0000 7fe47ff94640 4 set_mon_vals no callback set 2024-09-11T01:43:08.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.556+0000 7fe47cd2e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:08.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.556+0000 7fe47cd2e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:08.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.556+0000 7fe47cd2e640 10 monclient: dump: 2024-09-11T01:43:08.569 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:08.569 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.569 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.569 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.569 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:08.569 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.569 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:08.569 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:08.569 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:08.569 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.557+0000 7fe47ff94640 10 monclient: shutdown 2024-09-11T01:43:08.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.557+0000 7fe47ff94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:08.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.557+0000 7fe47ff94640 1 -- 172.21.15.12:0/2801798442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4780a1380 msgr2=0x7fe4780a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.557+0000 7fe47ff94640 1 --2- 172.21.15.12:0/2801798442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4780a1380 0x7fe4780a1760 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe4740097a0 tx=0x7fe474029fd0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.557+0000 7fe47ff94640 1 -- 172.21.15.12:0/2801798442 shutdown_connections 2024-09-11T01:43:08.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.557+0000 7fe47ff94640 1 --2- 172.21.15.12:0/2801798442 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4780a0370 0x7fe478135ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.557+0000 7fe47ff94640 1 --2- 172.21.15.12:0/2801798442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4780a1380 0x7fe4780a1760 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.557+0000 7fe47ff94640 1 -- 172.21.15.12:0/2801798442 >> 172.21.15.12:0/2801798442 conn(0x7fe4780a2f70 msgr2=0x7fe4780a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:08.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.557+0000 7fe47ff94640 1 -- 172.21.15.12:0/2801798442 shutdown_connections 2024-09-11T01:43:08.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.557+0000 7fe47ff94640 1 -- 172.21.15.12:0/2801798442 wait complete. 2024-09-11T01:43:08.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.557+0000 7fe47ff94640 10 monclient: build_initial_monmap 2024-09-11T01:43:08.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.557+0000 7fe47ff94640 10 monclient: monmap: 2024-09-11T01:43:08.570 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:08.570 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:43:08.571 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:43:08.571 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:43:08.571 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:08.571 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.571 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:43:08.571 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:43:08.571 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:43:08.571 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47ff94640 1 Processor -- start 2024-09-11T01:43:08.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47ff94640 1 -- start start 2024-09-11T01:43:08.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47ff94640 10 monclient: init 2024-09-11T01:43:08.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47ff94640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:08.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47ff94640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:43:08.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47ff94640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4780a0370 0x7fe478127730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47ff94640 10 monclient(hunting): picked mon.noname-c con 0x7fe4780a0370 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:08.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47ff94640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe4780a0c90 0x7fe4781c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47ff94640 10 monclient(hunting): picked mon.noname-b con 0x7fe4780a0c90 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:08.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47ff94640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4780a1380 0x7fe4781354f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47ff94640 10 monclient(hunting): picked mon.noname-a con 0x7fe4780a1380 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:08.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47ff94640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47ff94640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe47809ce20 con 0x7fe4780a1380 2024-09-11T01:43:08.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47ff94640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47ff94640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe47809cca0 con 0x7fe4780a0c90 2024-09-11T01:43:08.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47ff94640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47ff94640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe478136730 con 0x7fe4780a0370 2024-09-11T01:43:08.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47ff94640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:08.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47ff94640 10 monclient(hunting): authenticate will time out at 1595.000663s 2024-09-11T01:43:08.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47e531640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4780a1380 0x7fe4781354f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47dd30640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4780a0370 0x7fe478127730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47e531640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4780a1380 0x7fe4781354f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34940/0 (socket says 172.21.15.12:34940) 2024-09-11T01:43:08.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47e531640 1 -- 172.21.15.12:0/2536808634 learned_addr learned my addr 172.21.15.12:0/2536808634 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:08.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47e531640 10 monclient(hunting): get_auth_request con 0x7fe4780a1380 auth_method 0 2024-09-11T01:43:08.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47e531640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47e531640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47e531640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47dd30640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4780a0370 0x7fe478127730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41278/0 (socket says 172.21.15.12:41278) 2024-09-11T01:43:08.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47dd30640 10 monclient(hunting): get_auth_request con 0x7fe4780a0370 auth_method 0 2024-09-11T01:43:08.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47dd30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47dd30640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47dd30640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47e531640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:08.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47e531640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:08.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47e531640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:08.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47dd30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:08.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47dd30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:08.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.558+0000 7fe47dd30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:08.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.559+0000 7fe47d52f640 1 --2- 172.21.15.12:0/2536808634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe4780a0c90 0x7fe4781c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.559+0000 7fe47e531640 10 monclient(hunting): handle_auth_done global_id 94707 payload 824 2024-09-11T01:43:08.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.559+0000 7fe47e531640 1 -- 172.21.15.12:0/2536808634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4780a0370 msgr2=0x7fe478127730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.559+0000 7fe47e531640 1 --2- 172.21.15.12:0/2536808634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4780a0370 0x7fe478127730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.559+0000 7fe47e531640 1 -- 172.21.15.12:0/2536808634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe4780a0c90 msgr2=0x7fe4781c9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.559+0000 7fe47e531640 1 --2- 172.21.15.12:0/2536808634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe4780a0c90 0x7fe4781c9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.559+0000 7fe47e531640 10 monclient: _finish_hunting 0 2024-09-11T01:43:08.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.559+0000 7fe47e531640 1 monclient: found mon.noname-a 2024-09-11T01:43:08.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.559+0000 7fe47e531640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:08.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.559+0000 7fe47e531640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:08.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.559+0000 7fe47e531640 1 -- 172.21.15.12:0/2536808634 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4781280c0 con 0x7fe4780a1380 2024-09-11T01:43:08.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.559+0000 7fe47dd30640 1 --2- 172.21.15.12:0/2536808634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4780a0370 0x7fe478127730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:43:08.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.559+0000 7fe47d52f640 10 monclient: get_auth_request con 0x7fe4780a0c90 auth_method 0 2024-09-11T01:43:08.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.559+0000 7fe47d52f640 1 --2- 172.21.15.12:0/2536808634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe4780a0c90 0x7fe4781c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:43:08.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.561+0000 7fe47e531640 1 -- 172.21.15.12:0/2536808634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4780a1380 msgr2=0x7fe4781354f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:08.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.561+0000 7fe47e531640 1 --2- 172.21.15.12:0/2536808634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4780a1380 0x7fe4781354f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe47000caf0 tx=0x7fe47000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:08.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.561+0000 7fe46a7fc640 1 -- 172.21.15.12:0/2536808634 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47000ae50 con 0x7fe4780a1380 2024-09-11T01:43:08.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.561+0000 7fe46a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:08.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.561+0000 7fe46a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:08.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.561+0000 7fe46a7fc640 10 monclient: dump: 2024-09-11T01:43:08.576 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:08.576 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.576 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.576 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.576 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:08.576 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.576 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:08.576 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:08.576 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:08.576 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.561+0000 7fe46a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:43:08.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.561+0000 7fe46a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:43:08.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.561+0000 7fe46a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:08.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.561+0000 7fe46a7fc640 1 -- 172.21.15.12:0/2536808634 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe470012400 con 0x7fe4780a1380 2024-09-11T01:43:08.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.561+0000 7fe47ff94640 5 monclient: authenticate success, global_id 94707 2024-09-11T01:43:08.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.561+0000 7fe47ff94640 10 monclient: _renew_subs 2024-09-11T01:43:08.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.561+0000 7fe47ff94640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:08.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.561+0000 7fe47ff94640 1 -- 172.21.15.12:0/2536808634 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe47809d480 con 0x7fe4780a1380 2024-09-11T01:43:08.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.561+0000 7fe46a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:08.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.561+0000 7fe46a7fc640 1 -- 172.21.15.12:0/2536808634 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4700127d0 con 0x7fe4780a1380 2024-09-11T01:43:08.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.561+0000 7fe46a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:08.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.561+0000 7fe46a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:08.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.561+0000 7fe46a7fc640 10 monclient: dump: 2024-09-11T01:43:08.577 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:08.577 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.578 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.578 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.578 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:08.578 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.578 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:08.578 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:08.578 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:08.578 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.561+0000 7fe46bfff640 4 set_mon_vals no callback set 2024-09-11T01:43:08.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.561+0000 7fe47ff94640 10 monclient: _renew_subs 2024-09-11T01:43:08.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.561+0000 7fe47ff94640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:08.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.561+0000 7fe47ff94640 1 -- 172.21.15.12:0/2536808634 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe478128f30 con 0x7fe4780a1380 2024-09-11T01:43:08.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.562+0000 7fe47ff94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:08.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.562+0000 7fe47ff94640 1 --2- 172.21.15.12:0/2536808634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4781ccfa0 0x7fe4781cd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.562+0000 7fe47ff94640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.562+0000 7fe47ff94640 1 -- 172.21.15.12:0/2536808634 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe478128bb0 con 0x7fe4781ccfa0 2024-09-11T01:43:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.562+0000 7fe47dd30640 1 --2- 172.21.15.12:0/2536808634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4781ccfa0 0x7fe4781cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.562+0000 7fe47dd30640 10 monclient: get_auth_request con 0x7fe4781ccfa0 auth_method 0 2024-09-11T01:43:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.562+0000 7fe47dd30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.562+0000 7fe47dd30640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.562+0000 7fe47dd30640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.562+0000 7fe47dd30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.562+0000 7fe47dd30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.562+0000 7fe47dd30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.562+0000 7fe46a7fc640 1 -- 172.21.15.12:0/2536808634 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fe470012970 con 0x7fe4780a1380 2024-09-11T01:43:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.562+0000 7fe47dd30640 10 monclient(hunting): handle_auth_done global_id 154130 payload 824 2024-09-11T01:43:08.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.562+0000 7fe47dd30640 1 -- 172.21.15.12:0/2536808634 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe4780a1760 con 0x7fe4781ccfa0 2024-09-11T01:43:08.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.562+0000 7fe46a7fc640 1 --2- 172.21.15.12:0/2536808634 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe45c03e3b0 0x7fe45c040870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.562+0000 7fe46a7fc640 1 -- 172.21.15.12:0/2536808634 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(475..475 src has 285..475) ==== 4815+0+0 (secure 0 0 0) 0x7fe4700533d0 con 0x7fe4780a1380 2024-09-11T01:43:08.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.563+0000 7fe47d52f640 1 --2- 172.21.15.12:0/2536808634 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe45c03e3b0 0x7fe45c040870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.563+0000 7fe47d52f640 10 monclient: get_auth_request con 0x7fe45c03e3b0 auth_method 0 2024-09-11T01:43:08.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.564+0000 7fe47dd30640 1 -- 172.21.15.12:0/2536808634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4781ccfa0 msgr2=0x7fe4781cd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:08.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.564+0000 7fe47dd30640 1 --2- 172.21.15.12:0/2536808634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4781ccfa0 0x7fe4781cd380 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe474034000 tx=0x7fe4740046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:08.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.565+0000 7fe46a7fc640 1 -- 172.21.15.12:0/2536808634 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe474043070 con 0x7fe4781ccfa0 2024-09-11T01:43:08.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.565+0000 7fe46a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:08.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.565+0000 7fe46a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:43:08.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.565+0000 7fe46a7fc640 10 monclient: dump: 2024-09-11T01:43:08.580 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:08.580 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.580 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.580 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.581 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:08.581 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.581 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:08.581 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:08.581 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:08.581 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.565+0000 7fe46a7fc640 1 -- 172.21.15.12:0/2536808634 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe47403ca60 con 0x7fe4781ccfa0 2024-09-11T01:43:08.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.565+0000 7fe46a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:08.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.565+0000 7fe46a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:43:08.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.565+0000 7fe46a7fc640 1 -- 172.21.15.12:0/2536808634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4781ccfa0 msgr2=0x7fe4781cd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.565+0000 7fe46a7fc640 1 --2- 172.21.15.12:0/2536808634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4781ccfa0 0x7fe4781cd380 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe474034000 tx=0x7fe4740046c0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.565+0000 7fe46a7fc640 1 -- 172.21.15.12:0/2536808634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4781ccfa0 msgr2=0x7fe4781cd380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:43:08.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.565+0000 7fe46a7fc640 1 --2- 172.21.15.12:0/2536808634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4781ccfa0 0x7fe4781cd380 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe474034000 tx=0x7fe4740046c0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.565+0000 7fe47d52f640 1 --2- 172.21.15.12:0/2536808634 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe45c03e3b0 0x7fe45c040870 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe478128e60 tx=0x7fe46c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:08.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.571+0000 7fe47ff94640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:43:08.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.571+0000 7fe47ff94640 1 --2- 172.21.15.12:0/2536808634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4781cd8c0 0x7fe478120cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.571+0000 7fe47ff94640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.571+0000 7fe47ff94640 1 -- 172.21.15.12:0/2536808634 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe47806d480 con 0x7fe4781cd8c0 2024-09-11T01:43:08.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.571+0000 7fe47dd30640 1 --2- 172.21.15.12:0/2536808634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4781cd8c0 0x7fe478120cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.571+0000 7fe47dd30640 10 monclient: get_auth_request con 0x7fe4781cd8c0 auth_method 0 2024-09-11T01:43:08.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.571+0000 7fe47dd30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.571+0000 7fe47dd30640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.571+0000 7fe47dd30640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.571+0000 7fe47dd30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:08.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.571+0000 7fe47dd30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:08.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.571+0000 7fe47dd30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:08.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.571+0000 7fe47dd30640 10 monclient(hunting): handle_auth_done global_id 154133 payload 824 2024-09-11T01:43:08.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.571+0000 7fe47dd30640 1 -- 172.21.15.12:0/2536808634 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe4781cdcd0 con 0x7fe4781cd8c0 2024-09-11T01:43:08.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.574+0000 7fe47dd30640 1 -- 172.21.15.12:0/2536808634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4781cd8c0 msgr2=0x7fe478120cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:08.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.574+0000 7fe47dd30640 1 --2- 172.21.15.12:0/2536808634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4781cd8c0 0x7fe478120cf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe47402a010 tx=0x7fe474009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:08.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.574+0000 7fe46a7fc640 1 -- 172.21.15.12:0/2536808634 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe474037040 con 0x7fe4781cd8c0 2024-09-11T01:43:08.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.574+0000 7fe46a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:08.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.574+0000 7fe46a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:43:08.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.574+0000 7fe46a7fc640 10 monclient: dump: 2024-09-11T01:43:08.585 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:08.585 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.585 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.585 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.585 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:08.585 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.585 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:08.585 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:08.585 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:08.585 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.574+0000 7fe46a7fc640 1 -- 172.21.15.12:0/2536808634 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7fe47403ce00 con 0x7fe4781cd8c0 2024-09-11T01:43:08.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.574+0000 7fe46a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:43:08.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.574+0000 7fe46a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:43:08.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.574+0000 7fe46a7fc640 1 -- 172.21.15.12:0/2536808634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4781cd8c0 msgr2=0x7fe478120cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.574+0000 7fe46a7fc640 1 --2- 172.21.15.12:0/2536808634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4781cd8c0 0x7fe478120cf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe47402a010 tx=0x7fe474009190 comp rx=0 tx=0).stop 2024-09-11T01:43:08.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.574+0000 7fe46a7fc640 1 -- 172.21.15.12:0/2536808634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4781cd8c0 msgr2=0x7fe478120cf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:43:08.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.574+0000 7fe46a7fc640 1 --2- 172.21.15.12:0/2536808634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4781cd8c0 0x7fe478120cf0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe47402a010 tx=0x7fe474009190 comp rx=0 tx=0).stop 2024-09-11T01:43:08.586 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:43:08.586 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:43:08.586 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:43:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:43:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 128, 2024-09-11T01:43:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 12656, 2024-09-11T01:43:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:43:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:43:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:43:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:43:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 6, 2024-09-11T01:43:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:43:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:43:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:43:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:43:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:43:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:08.589 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:08.589 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:08.589 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:08.589 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:08.589 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:08.589 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:08.589 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:08.589 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:08.589 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:08.589 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:08.589 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.589 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:43:08.589 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:43:08.590 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:43:08.590 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:43:08.590 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:43:08.590 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:43:08.590 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:43:08.590 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:43:08.590 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:43:08.590 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:43:08.590 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:43:08.590 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:43:08.590 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:43:08.590 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:43:08.590 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:43:08.590 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:43:08.591 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:43:08.591 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:08.591 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:08.591 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:08.591 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:08.591 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:08.591 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:08.591 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:08.591 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:08.591 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:08.591 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:08.591 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:08.591 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:08.592 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:43:08.592 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.592 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:43:08.592 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.592 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:43:08.592 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:43:08.592 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:08.592 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:08.592 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.592 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:08.592 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:43:08.592 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:08.592 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.593 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.593 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:08.593 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:43:08.593 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:08.593 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:08.593 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:08.593 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.593 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:43:08.593 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:43:08.593 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:08.593 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:08.593 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:08.593 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.594 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.594 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:43:08.594 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:43:08.594 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:08.594 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:08.594 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.594 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:08.594 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:43:08.594 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:08.594 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.594 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.594 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:08.594 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:43:08.594 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:08.595 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:08.595 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:08.595 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.595 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:43:08.595 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:43:08.595 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:08.595 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:08.595 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:08.595 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.595 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.595 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:43:08.595 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:43:08.595 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:43:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:43:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:43:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:43:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:43:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:43:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:08.598 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:43:08.598 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:08.598 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:08.598 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:43:08.598 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.598 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:08.598 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:08.598 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:43:08.598 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:08.598 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:08.598 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.598 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:43:08.598 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:43:08.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.574+0000 7fe47ff94640 1 -- 172.21.15.12:0/2536808634 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe45c03e3b0 msgr2=0x7fe45c040870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.574+0000 7fe47ff94640 1 --2- 172.21.15.12:0/2536808634 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe45c03e3b0 0x7fe45c040870 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe478128e60 tx=0x7fe46c008040 comp rx=0 tx=0).stop 2024-09-11T01:43:08.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.574+0000 7fe47ff94640 10 monclient: shutdown 2024-09-11T01:43:08.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.574+0000 7fe47ff94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:08.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.574+0000 7fe47ff94640 1 -- 172.21.15.12:0/2536808634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4780a1380 msgr2=0x7fe4781354f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.574+0000 7fe47ff94640 1 --2- 172.21.15.12:0/2536808634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4780a1380 0x7fe4781354f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe47000caf0 tx=0x7fe47000a060 comp rx=0 tx=0).stop 2024-09-11T01:43:08.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.574+0000 7fe47e531640 1 -- 172.21.15.12:0/2536808634 reap_dead start 2024-09-11T01:43:08.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.574+0000 7fe47e531640 1 -- 172.21.15.12:0/2536808634 reap_dead start 2024-09-11T01:43:08.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.574+0000 7fe47ff94640 1 -- 172.21.15.12:0/2536808634 shutdown_connections 2024-09-11T01:43:08.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.574+0000 7fe47ff94640 1 -- 172.21.15.12:0/2536808634 >> 172.21.15.12:0/2536808634 conn(0x7fe4780a2f70 msgr2=0x7fe4780a5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:08.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.574+0000 7fe47ff94640 1 -- 172.21.15.12:0/2536808634 shutdown_connections 2024-09-11T01:43:08.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.574+0000 7fe47ff94640 1 -- 172.21.15.12:0/2536808634 wait complete. 2024-09-11T01:43:08.601 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:43:08.601 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:43:08.601 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:43:08.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.701+0000 7fb7c86f7640 10 monclient: get_monmap_and_config 2024-09-11T01:43:08.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.701+0000 7fb7c86f7640 10 monclient: build_initial_monmap 2024-09-11T01:43:08.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.701+0000 7fb7c86f7640 10 monclient: monmap: 2024-09-11T01:43:08.703 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:08.703 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.703 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:43:08.702204+0000 2024-09-11T01:43:08.703 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:43:08.702204+0000 2024-09-11T01:43:08.703 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:08.703 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.703 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:43:08.703 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:43:08.703 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:43:08.704 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:43:08.704 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:43:08.704 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.702+0000 7fb7c86f7640 1 Processor -- start 2024-09-11T01:43:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.702+0000 7fb7c86f7640 1 -- start start 2024-09-11T01:43:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.702+0000 7fb7c86f7640 10 monclient: init 2024-09-11T01:43:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.702+0000 7fb7c86f7640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.702+0000 7fb7c86f7640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-11T01:43:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.702+0000 7fb7c86f7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb7c01381c0 addr v1:172.21.15.71:6789/0 2024-09-11T01:43:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.702+0000 7fb7c86f7640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb7c00a6aa0 0x7fb7c009cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.702+0000 7fb7c86f7640 10 monclient(hunting): picked mon.noname-c con 0x7fb7c00a6aa0 addr v2:172.21.15.71:3300/0 2024-09-11T01:43:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.702+0000 7fb7c86f7640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb7c013d000 0x7fb7c013f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.702+0000 7fb7c86f7640 10 monclient(hunting): picked mon.noname-b con 0x7fb7c013d000 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.702+0000 7fb7c86f7640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7c009c310 con 0x7fb7c01381c0 2024-09-11T01:43:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.702+0000 7fb7c86f7640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.702+0000 7fb7c86f7640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fb7c006fc90 con 0x7fb7c00a6aa0 2024-09-11T01:43:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.702+0000 7fb7c86f7640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.702+0000 7fb7c86f7640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb7c009c5f0 con 0x7fb7c013d000 2024-09-11T01:43:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.702+0000 7fb7c86f7640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.702+0000 7fb7c86f7640 10 monclient(hunting): authenticate will time out at 1595.144931s 2024-09-11T01:43:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c6c94640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb7c013d000 0x7fb7c013f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c5c92640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb7c00a6aa0 0x7fb7c009cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c6c94640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb7c013d000 0x7fb7c013f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41300/0 (socket says 172.21.15.12:41300) 2024-09-11T01:43:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c6493640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fb7c01381c0 0x7fb7c01385a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.12:57188/0 (socket says 172.21.15.12:57188) 2024-09-11T01:43:08.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c6c94640 1 -- 172.21.15.12:0/2107053416 learned_addr learned my addr 172.21.15.12:0/2107053416 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:08.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c5c92640 10 monclient(hunting): get_auth_request con 0x7fb7c00a6aa0 auth_method 0 2024-09-11T01:43:08.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c5c92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c5c92640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c5c92640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c6c94640 10 monclient(hunting): get_auth_request con 0x7fb7c013d000 auth_method 0 2024-09-11T01:43:08.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c6c94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c6c94640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c6c94640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c6c94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:08.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c6c94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:08.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c6c94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:08.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c5c92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:08.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c5c92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:08.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c5c92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:08.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c6c94640 10 monclient(hunting): handle_auth_done global_id 154136 payload 274 2024-09-11T01:43:08.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c6c94640 1 -- 172.21.15.12:0/2107053416 >> v2:172.21.15.71:3300/0 conn(0x7fb7c00a6aa0 msgr2=0x7fb7c009cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:08.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c6c94640 1 --2- 172.21.15.12:0/2107053416 >> v2:172.21.15.71:3300/0 conn(0x7fb7c00a6aa0 0x7fb7c009cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c6c94640 1 -- 172.21.15.12:0/2107053416 >> v1:172.21.15.71:6789/0 conn(0x7fb7c01381c0 legacy=0x7fb7c01385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:08.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c6c94640 10 monclient: _finish_hunting 0 2024-09-11T01:43:08.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c6c94640 1 monclient: found mon.noname-b 2024-09-11T01:43:08.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c6c94640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:08.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c6c94640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:43:08.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.703+0000 7fb7c6c94640 1 -- 172.21.15.12:0/2107053416 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7c009c8d0 con 0x7fb7c013d000 2024-09-11T01:43:08.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c6c94640 1 -- 172.21.15.12:0/2107053416 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb7c013d000 msgr2=0x7fb7c013f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:08.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c6c94640 1 --2- 172.21.15.12:0/2107053416 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb7c013d000 0x7fb7c013f3f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009e50 tx=0x7fb7b802a550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd168f6db3ed8ef2 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:08.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c5491640 1 -- 172.21.15.12:0/2107053416 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8036070 con 0x7fb7c013d000 2024-09-11T01:43:08.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c5491640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c5491640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:43:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c5491640 10 monclient: dump: 2024-09-11T01:43:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:08.716 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c5491640 10 monclient: _finish_auth 0 2024-09-11T01:43:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c5491640 10 monclient: _check_auth_tickets 2024-09-11T01:43:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c5491640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c5491640 1 -- 172.21.15.12:0/2107053416 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7b8007800 con 0x7fb7c013d000 2024-09-11T01:43:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c86f7640 5 monclient: authenticate success, global_id 154136 2024-09-11T01:43:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c86f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:43:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c5491640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c86f7640 10 monclient: get_monmap_and_config success 2024-09-11T01:43:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c5491640 1 -- 172.21.15.12:0/2107053416 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8007bd0 con 0x7fb7c013d000 2024-09-11T01:43:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c86f7640 4 set_mon_vals no callback set 2024-09-11T01:43:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c5491640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c5491640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:43:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c5491640 10 monclient: dump: 2024-09-11T01:43:08.717 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:08.718 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c86f7640 10 monclient: shutdown 2024-09-11T01:43:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c86f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:08.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c86f7640 1 -- 172.21.15.12:0/2107053416 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb7c013d000 msgr2=0x7fb7c013f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c86f7640 1 --2- 172.21.15.12:0/2107053416 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb7c013d000 0x7fb7c013f3f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009e50 tx=0x7fb7b802a550 comp rx=0 tx=0).stop 2024-09-11T01:43:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c86f7640 1 -- 172.21.15.12:0/2107053416 shutdown_connections 2024-09-11T01:43:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c86f7640 1 --2- 172.21.15.12:0/2107053416 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb7c013d000 0x7fb7c013f3f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c86f7640 1 --2- 172.21.15.12:0/2107053416 >> v2:172.21.15.71:3300/0 conn(0x7fb7c00a6aa0 0x7fb7c009cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c86f7640 1 -- 172.21.15.12:0/2107053416 >> 172.21.15.12:0/2107053416 conn(0x7fb7c00a7440 msgr2=0x7fb7c00a7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c86f7640 1 -- 172.21.15.12:0/2107053416 shutdown_connections 2024-09-11T01:43:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.706+0000 7fb7c86f7640 1 -- 172.21.15.12:0/2107053416 wait complete. 2024-09-11T01:43:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.707+0000 7fb7c86f7640 10 monclient: build_initial_monmap 2024-09-11T01:43:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.707+0000 7fb7c86f7640 10 monclient: monmap: 2024-09-11T01:43:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:43:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:43:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:43:08.719 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:43:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:43:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:43:08.720 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.707+0000 7fb7c86f7640 1 Processor -- start 2024-09-11T01:43:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.707+0000 7fb7c86f7640 1 -- start start 2024-09-11T01:43:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.707+0000 7fb7c86f7640 10 monclient: init 2024-09-11T01:43:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c86f7640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c86f7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:43:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c86f7640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb7c00a6aa0 0x7fb7c01cd250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c86f7640 10 monclient(hunting): picked mon.noname-c con 0x7fb7c00a6aa0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:08.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c86f7640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7c01381c0 0x7fb7c013c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c86f7640 10 monclient(hunting): picked mon.noname-b con 0x7fb7c01381c0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:08.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c86f7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7c013d000 0x7fb7c013c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c86f7640 10 monclient(hunting): picked mon.noname-a con 0x7fb7c013d000 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:08.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c86f7640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c86f7640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c013fab0 con 0x7fb7c013d000 2024-09-11T01:43:08.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c86f7640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c86f7640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c013f930 con 0x7fb7c01381c0 2024-09-11T01:43:08.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c86f7640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c86f7640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb7c013fc30 con 0x7fb7c00a6aa0 2024-09-11T01:43:08.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c86f7640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:08.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c86f7640 10 monclient(hunting): authenticate will time out at 1595.150239s 2024-09-11T01:43:08.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6493640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb7c00a6aa0 0x7fb7c01cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6c94640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7c013d000 0x7fb7c013c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6493640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb7c00a6aa0 0x7fb7c01cd250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41308/0 (socket says 172.21.15.12:41308) 2024-09-11T01:43:08.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6493640 1 -- 172.21.15.12:0/136263925 learned_addr learned my addr 172.21.15.12:0/136263925 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:08.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6493640 10 monclient(hunting): get_auth_request con 0x7fb7c00a6aa0 auth_method 0 2024-09-11T01:43:08.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6493640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6493640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6493640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6c94640 10 monclient(hunting): get_auth_request con 0x7fb7c013d000 auth_method 0 2024-09-11T01:43:08.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6c94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6c94640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6c94640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c5c92640 1 --2- 172.21.15.12:0/136263925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7c01381c0 0x7fb7c013c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6493640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6493640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6493640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c5c92640 10 monclient(hunting): get_auth_request con 0x7fb7c01381c0 auth_method 0 2024-09-11T01:43:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c5c92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c5c92640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c5c92640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6c94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6c94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6c94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:08.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c5c92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c5c92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c5c92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6493640 10 monclient(hunting): handle_auth_done global_id 154139 payload 824 2024-09-11T01:43:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6493640 1 -- 172.21.15.12:0/136263925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7c01381c0 msgr2=0x7fb7c013c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6493640 1 --2- 172.21.15.12:0/136263925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7c01381c0 0x7fb7c013c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6493640 1 -- 172.21.15.12:0/136263925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7c013d000 msgr2=0x7fb7c013c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6493640 1 --2- 172.21.15.12:0/136263925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7c013d000 0x7fb7c013c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6493640 10 monclient: _finish_hunting 0 2024-09-11T01:43:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6493640 1 monclient: found mon.noname-c 2024-09-11T01:43:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6493640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6493640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:43:08.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6493640 1 -- 172.21.15.12:0/136263925 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7c00a3600 con 0x7fb7c00a6aa0 2024-09-11T01:43:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.708+0000 7fb7c6c94640 1 --2- 172.21.15.12:0/136263925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7c013d000 0x7fb7c013c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:43:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.710+0000 7fb7c6493640 1 -- 172.21.15.12:0/136263925 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb7c00a6aa0 msgr2=0x7fb7c01cd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.710+0000 7fb7c6493640 1 --2- 172.21.15.12:0/136263925 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb7c00a6aa0 0x7fb7c01cd250 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00c450 tx=0x7fb7bc00c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.711+0000 7fb7b6ffd640 1 -- 172.21.15.12:0/136263925 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc017070 con 0x7fb7c00a6aa0 2024-09-11T01:43:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.711+0000 7fb7b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.711+0000 7fb7b6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:43:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.711+0000 7fb7b6ffd640 10 monclient: dump: 2024-09-11T01:43:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.725 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:08.726 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.711+0000 7fb7b6ffd640 10 monclient: _finish_auth 0 2024-09-11T01:43:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.711+0000 7fb7b6ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:43:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.711+0000 7fb7b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.711+0000 7fb7b6ffd640 1 -- 172.21.15.12:0/136263925 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7bc005810 con 0x7fb7c00a6aa0 2024-09-11T01:43:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.711+0000 7fb7c86f7640 5 monclient: authenticate success, global_id 154139 2024-09-11T01:43:08.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.711+0000 7fb7c86f7640 10 monclient: _renew_subs 2024-09-11T01:43:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.711+0000 7fb7c86f7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:43:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.711+0000 7fb7c86f7640 1 -- 172.21.15.12:0/136263925 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7c00a1710 con 0x7fb7c00a6aa0 2024-09-11T01:43:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.711+0000 7fb7b6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.711+0000 7fb7b6ffd640 1 -- 172.21.15.12:0/136263925 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc005c00 con 0x7fb7c00a6aa0 2024-09-11T01:43:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.711+0000 7fb7b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.711+0000 7fb7b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:43:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.711+0000 7fb7b6ffd640 10 monclient: dump: 2024-09-11T01:43:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.727 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:08.728 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.711+0000 7fb7c4c90640 4 set_mon_vals no callback set 2024-09-11T01:43:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.711+0000 7fb7c86f7640 10 monclient: _renew_subs 2024-09-11T01:43:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.711+0000 7fb7c86f7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:43:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.711+0000 7fb7c86f7640 1 -- 172.21.15.12:0/136263925 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7c01390c0 con 0x7fb7c00a6aa0 2024-09-11T01:43:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.712+0000 7fb7c86f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.712+0000 7fb7c86f7640 1 --2- 172.21.15.12:0/136263925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7c00a1bd0 0x7fb7c00a1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.712+0000 7fb7c86f7640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.712+0000 7fb7c86f7640 1 -- 172.21.15.12:0/136263925 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c00a4210 con 0x7fb7c00a1bd0 2024-09-11T01:43:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.712+0000 7fb7c6493640 1 --2- 172.21.15.12:0/136263925 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb7c00a6aa0 0x7fb7c01cd250 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00c450 tx=0x7fb7bc00c910 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7fb7bc052560 osd_map(475..475 src has 285..475) 2024-09-11T01:43:08.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.712+0000 7fb7b6ffd640 1 -- 172.21.15.12:0/136263925 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fb7bc01b3b0 con 0x7fb7c00a6aa0 2024-09-11T01:43:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.712+0000 7fb7c5c92640 1 --2- 172.21.15.12:0/136263925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7c00a1bd0 0x7fb7c00a1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.712+0000 7fb7c5c92640 10 monclient: get_auth_request con 0x7fb7c00a1bd0 auth_method 0 2024-09-11T01:43:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.712+0000 7fb7c5c92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.712+0000 7fb7c5c92640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.712+0000 7fb7c5c92640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.712+0000 7fb7b6ffd640 1 --2- 172.21.15.12:0/136263925 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb7a403e260 0x7fb7a4040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.712+0000 7fb7c5c92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.712+0000 7fb7c5c92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.712+0000 7fb7c5c92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.712+0000 7fb7c6c94640 1 --2- 172.21.15.12:0/136263925 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb7a403e260 0x7fb7a4040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.712+0000 7fb7c6c94640 10 monclient: get_auth_request con 0x7fb7a403e260 auth_method 0 2024-09-11T01:43:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.712+0000 7fb7c5c92640 10 monclient(hunting): handle_auth_done global_id 134308 payload 824 2024-09-11T01:43:08.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.712+0000 7fb7c5c92640 1 -- 172.21.15.12:0/136263925 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb7c009cfd0 con 0x7fb7c00a1bd0 2024-09-11T01:43:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.715+0000 7fb7c5c92640 1 -- 172.21.15.12:0/136263925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7c00a1bd0 msgr2=0x7fb7c00a1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.715+0000 7fb7c5c92640 1 --2- 172.21.15.12:0/136263925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7c00a1bd0 0x7fb7c00a1fb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00a3980 tx=0x7fb7b0011d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.715+0000 7fb7c6c94640 1 --2- 172.21.15.12:0/136263925 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb7a403e260 0x7fb7a4040720 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009e50 tx=0x7fb7b8009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:08.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.715+0000 7fb7b6ffd640 1 -- 172.21.15.12:0/136263925 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0019070 con 0x7fb7c00a1bd0 2024-09-11T01:43:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.715+0000 7fb7b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.715+0000 7fb7b6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:43:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.715+0000 7fb7b6ffd640 10 monclient: dump: 2024-09-11T01:43:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:08.733 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.715+0000 7fb7b6ffd640 1 -- 172.21.15.12:0/136263925 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb7b000f830 con 0x7fb7c00a1bd0 2024-09-11T01:43:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.715+0000 7fb7b6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.715+0000 7fb7b6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:43:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.715+0000 7fb7b6ffd640 1 -- 172.21.15.12:0/136263925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7c00a1bd0 msgr2=0x7fb7c00a1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.715+0000 7fb7b6ffd640 1 --2- 172.21.15.12:0/136263925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7c00a1bd0 0x7fb7c00a1fb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00a3980 tx=0x7fb7b0011d60 comp rx=0 tx=0).stop 2024-09-11T01:43:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.715+0000 7fb7b6ffd640 1 -- 172.21.15.12:0/136263925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7c00a1bd0 msgr2=0x7fb7c00a1fb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:43:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.715+0000 7fb7b6ffd640 1 --2- 172.21.15.12:0/136263925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7c00a1bd0 0x7fb7c00a1fb0 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00a3980 tx=0x7fb7b0011d60 comp rx=0 tx=0).stop 2024-09-11T01:43:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.722+0000 7fb7c86f7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:43:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.722+0000 7fb7c86f7640 1 --2- 172.21.15.12:0/136263925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7c00a24f0 0x7fb7c009efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.722+0000 7fb7c86f7640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.722+0000 7fb7c86f7640 1 -- 172.21.15.12:0/136263925 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c0139a50 con 0x7fb7c00a24f0 2024-09-11T01:43:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.722+0000 7fb7c5c92640 1 --2- 172.21.15.12:0/136263925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7c00a24f0 0x7fb7c009efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.722+0000 7fb7c5c92640 10 monclient: get_auth_request con 0x7fb7c00a24f0 auth_method 0 2024-09-11T01:43:08.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.722+0000 7fb7c5c92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.722+0000 7fb7c5c92640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.722+0000 7fb7c5c92640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.722+0000 7fb7c5c92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.722+0000 7fb7c5c92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.722+0000 7fb7c5c92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.722+0000 7fb7c5c92640 10 monclient(hunting): handle_auth_done global_id 134311 payload 824 2024-09-11T01:43:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.722+0000 7fb7c5c92640 1 -- 172.21.15.12:0/136263925 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb7c0069840 con 0x7fb7c00a24f0 2024-09-11T01:43:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.724+0000 7fb7c5c92640 1 -- 172.21.15.12:0/136263925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7c00a24f0 msgr2=0x7fb7c009efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.724+0000 7fb7c5c92640 1 --2- 172.21.15.12:0/136263925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7c00a24f0 0x7fb7c009efa0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb7b001b000 tx=0x7fb7b00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.725+0000 7fb7b6ffd640 1 -- 172.21.15.12:0/136263925 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0019040 con 0x7fb7c00a24f0 2024-09-11T01:43:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.725+0000 7fb7b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.725+0000 7fb7b6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:43:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.725+0000 7fb7b6ffd640 10 monclient: dump: 2024-09-11T01:43:08.735 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:08.736 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.725+0000 7fb7b6ffd640 1 -- 172.21.15.12:0/136263925 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7fb7b0003e00 con 0x7fb7c00a24f0 2024-09-11T01:43:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.725+0000 7fb7b6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:43:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.725+0000 7fb7b6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:43:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.725+0000 7fb7b6ffd640 1 -- 172.21.15.12:0/136263925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7c00a24f0 msgr2=0x7fb7c009efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.725+0000 7fb7b6ffd640 1 --2- 172.21.15.12:0/136263925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7c00a24f0 0x7fb7c009efa0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb7b001b000 tx=0x7fb7b00046c0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.725+0000 7fb7b6ffd640 1 -- 172.21.15.12:0/136263925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7c00a24f0 msgr2=0x7fb7c009efa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:43:08.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.725+0000 7fb7b6ffd640 1 --2- 172.21.15.12:0/136263925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7c00a24f0 0x7fb7c009efa0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.737 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:43:08.737 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:43:08.737 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:43:08.737 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:43:08.737 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 128, 2024-09-11T01:43:08.737 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 121686, 2024-09-11T01:43:08.737 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:43:08.737 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:43:08.738 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:43:08.738 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:43:08.738 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:08.738 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 7, 2024-09-11T01:43:08.738 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:43:08.738 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:43:08.738 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:43:08.738 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:08.738 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:08.738 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:08.738 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:08.738 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:08.738 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:08.739 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:08.739 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:08.739 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:08.739 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:08.739 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:08.739 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:08.739 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:43:08.739 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:43:08.739 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:08.739 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:08.739 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:08.739 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:08.739 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:08.739 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:08.740 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:08.740 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:08.740 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:08.740 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:08.740 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:08.740 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:08.740 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.740 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:43:08.740 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:43:08.740 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:43:08.740 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:43:08.740 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:43:08.740 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:43:08.741 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:43:08.741 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:43:08.741 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:43:08.741 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:43:08.741 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:43:08.741 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:43:08.741 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:43:08.741 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:43:08.741 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:43:08.741 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:43:08.741 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:43:08.741 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:08.741 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:08.742 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:08.742 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:08.742 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:08.742 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:08.742 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:08.742 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:08.742 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:08.742 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:08.742 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:08.742 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:08.742 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:43:08.742 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.742 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:43:08.742 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.743 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:43:08.743 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:43:08.743 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:08.743 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:08.743 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.743 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:08.743 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:43:08.743 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:08.743 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.743 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.743 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:08.743 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:43:08.743 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:08.744 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:08.744 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:08.744 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.744 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:43:08.744 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:43:08.744 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:08.744 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:08.744 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:08.744 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.744 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.744 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:43:08.744 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:43:08.744 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:08.744 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:08.745 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.745 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:08.745 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:43:08.745 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:08.745 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.745 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.745 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:08.745 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:43:08.745 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:08.745 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:08.745 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:08.745 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.745 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:43:08.746 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:43:08.746 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:08.746 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:08.746 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:08.746 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.746 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.746 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:43:08.746 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:43:08.746 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:08.746 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:08.746 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.746 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:08.746 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:43:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:43:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:43:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:43:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:08.747 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:43:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:43:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:43:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:08.748 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:43:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:43:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:43:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:43:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:08.749 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:43:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:43:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:08.750 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:43:08.750 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:43:08.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.725+0000 7fb7c86f7640 1 -- 172.21.15.12:0/136263925 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb7a403e260 msgr2=0x7fb7a4040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.725+0000 7fb7c86f7640 1 --2- 172.21.15.12:0/136263925 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb7a403e260 0x7fb7a4040720 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009e50 tx=0x7fb7b8009000 comp rx=0 tx=0).stop 2024-09-11T01:43:08.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.725+0000 7fb7c86f7640 10 monclient: shutdown 2024-09-11T01:43:08.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.725+0000 7fb7c86f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:08.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.725+0000 7fb7c86f7640 1 -- 172.21.15.12:0/136263925 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb7c00a6aa0 msgr2=0x7fb7c01cd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.725+0000 7fb7c86f7640 1 --2- 172.21.15.12:0/136263925 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb7c00a6aa0 0x7fb7c01cd250 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00c450 tx=0x7fb7bc00c910 comp rx=0 tx=0).stop 2024-09-11T01:43:08.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.725+0000 7fb7c6c94640 1 -- 172.21.15.12:0/136263925 reap_dead start 2024-09-11T01:43:08.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.725+0000 7fb7b6ffd640 1 -- 172.21.15.12:0/136263925 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(475..475 src has 285..475) ==== 4815+0+0 (unknown 0 0 0) 0x7fb7bc052560 con 0x7fb7c00a6aa0 2024-09-11T01:43:08.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.725+0000 7fb7c86f7640 1 -- 172.21.15.12:0/136263925 shutdown_connections 2024-09-11T01:43:08.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.725+0000 7fb7c86f7640 1 --2- 172.21.15.12:0/136263925 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb7c00a6aa0 0x7fb7c01cd250 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.725+0000 7fb7c86f7640 1 -- 172.21.15.12:0/136263925 >> 172.21.15.12:0/136263925 conn(0x7fb7c00a7440 msgr2=0x7fb7c0137440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:08.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.725+0000 7fb7c86f7640 1 -- 172.21.15.12:0/136263925 shutdown_connections 2024-09-11T01:43:08.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.725+0000 7fb7c86f7640 1 -- 172.21.15.12:0/136263925 wait complete. 2024-09-11T01:43:08.753 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:43:08.753 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-11T01:43:08.753 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T01:43:08.753 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-11T01:43:08.753 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-11T01:43:08.753 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-11T01:43:08.753 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:43:08.763 INFO:tasks.ceph.mon.c:Stopped 2024-09-11T01:43:08.763 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:43:08.764 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:43:08.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.902+0000 7fab9c517640 10 monclient: get_monmap_and_config 2024-09-11T01:43:08.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.902+0000 7fab9c517640 10 monclient: build_initial_monmap 2024-09-11T01:43:08.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.902+0000 7fab9c517640 10 monclient: monmap: 2024-09-11T01:43:08.903 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:08.904 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.904 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:43:08.902821+0000 2024-09-11T01:43:08.904 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:43:08.902821+0000 2024-09-11T01:43:08.904 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:08.904 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.904 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:43:08.904 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:43:08.904 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:43:08.904 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:43:08.904 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:43:08.904 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.902+0000 7fab9c517640 1 Processor -- start 2024-09-11T01:43:08.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.902+0000 7fab9c517640 1 -- start start 2024-09-11T01:43:08.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.902+0000 7fab9c517640 10 monclient: init 2024-09-11T01:43:08.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9c517640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:08.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9c517640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-11T01:43:08.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9c517640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fab94138210 addr v1:172.21.15.71:6789/0 2024-09-11T01:43:08.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9c517640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fab940a6810 0x7fab9413d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9c517640 10 monclient(hunting): picked mon.noname-b con 0x7fab940a6810 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:08.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9c517640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fab9413d640 0x7fab9413fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9c517640 10 monclient(hunting): picked mon.noname-a con 0x7fab9413d640 addr v2:172.21.15.12:3300/0 2024-09-11T01:43:08.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9c517640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab94138d00 con 0x7fab94138210 2024-09-11T01:43:08.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9c517640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9c517640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fab9409c7a0 con 0x7fab9413d640 2024-09-11T01:43:08.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9c517640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9c517640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fab9409c4c0 con 0x7fab940a6810 2024-09-11T01:43:08.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9c517640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:08.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9c517640 10 monclient(hunting): authenticate will time out at 1595.345433s 2024-09-11T01:43:08.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab99ab2640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fab940a6810 msgr2=0x7fab9413d100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:43:08.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab99ab2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fab940a6810 0x7fab9413d100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:08.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9aab4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fab9413d640 0x7fab9413fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9aab4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fab9413d640 0x7fab9413fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34962/0 (socket says 172.21.15.12:34962) 2024-09-11T01:43:08.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9aab4640 1 -- 172.21.15.12:0/859664662 learned_addr learned my addr 172.21.15.12:0/859664662 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:08.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9aab4640 10 monclient(hunting): get_auth_request con 0x7fab9413d640 auth_method 0 2024-09-11T01:43:08.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9aab4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9aab4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9aab4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9aab4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:08.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9aab4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:08.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9aab4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:08.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9aab4640 10 monclient(hunting): handle_auth_done global_id 94713 payload 274 2024-09-11T01:43:08.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9aab4640 1 -- 172.21.15.12:0/859664662 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fab940a6810 msgr2=0x7fab9413d100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:43:08.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9aab4640 1 --2- 172.21.15.12:0/859664662 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fab940a6810 0x7fab9413d100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9aab4640 1 -- 172.21.15.12:0/859664662 >> v1:172.21.15.71:6789/0 conn(0x7fab94138210 legacy=0x7fab941385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:08.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9aab4640 10 monclient: _finish_hunting 0 2024-09-11T01:43:08.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9aab4640 1 monclient: found mon.noname-a 2024-09-11T01:43:08.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9aab4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:08.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9aab4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:08.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.903+0000 7fab9aab4640 1 -- 172.21.15.12:0/859664662 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab9409ba20 con 0x7fab9413d640 2024-09-11T01:43:08.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.906+0000 7fab9aab4640 1 -- 172.21.15.12:0/859664662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab9413d640 msgr2=0x7fab9413fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:08.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.906+0000 7fab9aab4640 1 --2- 172.21.15.12:0/859664662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab9413d640 0x7fab9413fa30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fab8c009e50 tx=0x7fab8c02a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9916b85360e66b36 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:08.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.906+0000 7fab992b1640 1 -- 172.21.15.12:0/859664662 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab8c036070 con 0x7fab9413d640 2024-09-11T01:43:08.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.906+0000 7fab992b1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:08.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.906+0000 7fab992b1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:08.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.906+0000 7fab992b1640 10 monclient: dump: 2024-09-11T01:43:08.915 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:08.915 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.915 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.915 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.915 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:08.915 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.915 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:08.916 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:08.916 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:08.916 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.906+0000 7fab992b1640 10 monclient: _finish_auth 0 2024-09-11T01:43:08.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.906+0000 7fab992b1640 10 monclient: _check_auth_tickets 2024-09-11T01:43:08.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.906+0000 7fab992b1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:08.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.906+0000 7fab992b1640 1 -- 172.21.15.12:0/859664662 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab8c007800 con 0x7fab9413d640 2024-09-11T01:43:08.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.906+0000 7fab9c517640 5 monclient: authenticate success, global_id 94713 2024-09-11T01:43:08.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.906+0000 7fab9c517640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:43:08.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.906+0000 7fab992b1640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:08.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.906+0000 7fab9c517640 10 monclient: get_monmap_and_config success 2024-09-11T01:43:08.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.906+0000 7fab9c517640 4 set_mon_vals no callback set 2024-09-11T01:43:08.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.906+0000 7fab992b1640 1 -- 172.21.15.12:0/859664662 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab8c007bd0 con 0x7fab9413d640 2024-09-11T01:43:08.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.906+0000 7fab992b1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:08.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.906+0000 7fab992b1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:08.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.906+0000 7fab992b1640 10 monclient: dump: 2024-09-11T01:43:08.917 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:08.917 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.917 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.917 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.917 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:08.917 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.917 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:08.917 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:08.917 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:08.917 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.907+0000 7fab9c517640 10 monclient: shutdown 2024-09-11T01:43:08.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.907+0000 7fab9c517640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:08.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.907+0000 7fab9c517640 1 -- 172.21.15.12:0/859664662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab9413d640 msgr2=0x7fab9413fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.907+0000 7fab9c517640 1 --2- 172.21.15.12:0/859664662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab9413d640 0x7fab9413fa30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fab8c009e50 tx=0x7fab8c02a570 comp rx=0 tx=0).stop 2024-09-11T01:43:08.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.907+0000 7fab9c517640 1 -- 172.21.15.12:0/859664662 shutdown_connections 2024-09-11T01:43:08.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.907+0000 7fab9c517640 1 --2- 172.21.15.12:0/859664662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab9413d640 0x7fab9413fa30 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.907+0000 7fab9c517640 1 --2- 172.21.15.12:0/859664662 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fab940a6810 0x7fab9413d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.907+0000 7fab9c517640 1 -- 172.21.15.12:0/859664662 >> 172.21.15.12:0/859664662 conn(0x7fab940aaa80 msgr2=0x7fab940aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:08.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.907+0000 7fab9c517640 1 -- 172.21.15.12:0/859664662 shutdown_connections 2024-09-11T01:43:08.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.907+0000 7fab9c517640 1 -- 172.21.15.12:0/859664662 wait complete. 2024-09-11T01:43:08.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.908+0000 7fab9c517640 10 monclient: build_initial_monmap 2024-09-11T01:43:08.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.908+0000 7fab9c517640 10 monclient: monmap: 2024-09-11T01:43:08.918 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:08.918 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:43:08.919 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:43:08.919 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:43:08.919 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:08.919 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.919 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:43:08.919 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:43:08.919 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:43:08.919 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.908+0000 7fab9c517640 1 Processor -- start 2024-09-11T01:43:08.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.908+0000 7fab9c517640 1 -- start start 2024-09-11T01:43:08.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.908+0000 7fab9c517640 10 monclient: init 2024-09-11T01:43:08.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.908+0000 7fab9c517640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:08.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.908+0000 7fab9c517640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:43:08.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.908+0000 7fab9c517640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab940a6810 0x7fab9409f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.908+0000 7fab9c517640 10 monclient(hunting): picked mon.noname-a con 0x7fab940a6810 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:08.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.908+0000 7fab9c517640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fab94138210 0x7fab9409fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.908+0000 7fab9c517640 10 monclient(hunting): picked mon.noname-c con 0x7fab94138210 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:08.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.908+0000 7fab9c517640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab9413d640 0x7fab940a0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.908+0000 7fab9c517640 10 monclient(hunting): picked mon.noname-b con 0x7fab9413d640 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:08.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.908+0000 7fab9c517640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.908+0000 7fab9c517640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fab9406f4f0 con 0x7fab940a6810 2024-09-11T01:43:08.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.908+0000 7fab9c517640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.908+0000 7fab9c517640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fab9409d620 con 0x7fab9413d640 2024-09-11T01:43:08.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.908+0000 7fab9c517640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:08.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.908+0000 7fab9c517640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fab9406f670 con 0x7fab94138210 2024-09-11T01:43:08.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.908+0000 7fab9c517640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:08.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.908+0000 7fab9c517640 10 monclient(hunting): authenticate will time out at 1595.350881s 2024-09-11T01:43:08.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.908+0000 7fab99ab2640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fab94138210 msgr2=0x7fab9409fb30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:43:08.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.908+0000 7fab99ab2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fab94138210 0x7fab9409fb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:08.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.908+0000 7fab9a2b3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab940a6810 0x7fab9409f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.909+0000 7fab9a2b3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab940a6810 0x7fab9409f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34978/0 (socket says 172.21.15.12:34978) 2024-09-11T01:43:08.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.909+0000 7fab9a2b3640 1 -- 172.21.15.12:0/712362905 learned_addr learned my addr 172.21.15.12:0/712362905 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:08.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.909+0000 7fab9a2b3640 10 monclient(hunting): get_auth_request con 0x7fab940a6810 auth_method 0 2024-09-11T01:43:08.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.909+0000 7fab9a2b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.909+0000 7fab9a2b3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.909+0000 7fab9a2b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.909+0000 7fab9a2b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:08.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.909+0000 7fab9a2b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:08.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.909+0000 7fab9a2b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:08.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.909+0000 7fab9aab4640 1 --2- 172.21.15.12:0/712362905 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab9413d640 0x7fab940a0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.909+0000 7fab9aab4640 10 monclient(hunting): get_auth_request con 0x7fab9413d640 auth_method 0 2024-09-11T01:43:08.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.909+0000 7fab9aab4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:08.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.909+0000 7fab9aab4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:08.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.909+0000 7fab9aab4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:08.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.909+0000 7fab9a2b3640 10 monclient(hunting): handle_auth_done global_id 94716 payload 824 2024-09-11T01:43:08.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.909+0000 7fab9a2b3640 1 -- 172.21.15.12:0/712362905 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fab94138210 msgr2=0x7fab9409fb30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:43:08.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.909+0000 7fab9a2b3640 1 --2- 172.21.15.12:0/712362905 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fab94138210 0x7fab9409fb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.909+0000 7fab9a2b3640 1 -- 172.21.15.12:0/712362905 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab9413d640 msgr2=0x7fab940a0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:08.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.909+0000 7fab9a2b3640 1 --2- 172.21.15.12:0/712362905 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab9413d640 0x7fab940a0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:08.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.909+0000 7fab9a2b3640 10 monclient: _finish_hunting 0 2024-09-11T01:43:08.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.909+0000 7fab9a2b3640 1 monclient: found mon.noname-a 2024-09-11T01:43:08.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.909+0000 7fab9a2b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:08.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.909+0000 7fab9a2b3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.909+0000 7fab9a2b3640 1 -- 172.21.15.12:0/712362905 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab9413a870 con 0x7fab940a6810 2024-09-11T01:43:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.909+0000 7fab9aab4640 1 --2- 172.21.15.12:0/712362905 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab9413d640 0x7fab940a0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:43:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.911+0000 7fab9a2b3640 1 -- 172.21.15.12:0/712362905 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab940a6810 msgr2=0x7fab9409f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.911+0000 7fab9a2b3640 1 --2- 172.21.15.12:0/712362905 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab940a6810 0x7fab9409f5f0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fab9000c430 tx=0x7fab9000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.911+0000 7fab8affd640 1 -- 172.21.15.12:0/712362905 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab900119a0 con 0x7fab940a6810 2024-09-11T01:43:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.911+0000 7fab8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.911+0000 7fab8affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.911+0000 7fab8affd640 10 monclient: dump: 2024-09-11T01:43:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.923 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:08.924 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.924 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:08.924 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:08.924 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:08.924 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.911+0000 7fab8affd640 10 monclient: _finish_auth 0 2024-09-11T01:43:08.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.911+0000 7fab8affd640 10 monclient: _check_auth_tickets 2024-09-11T01:43:08.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.911+0000 7fab8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:08.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.911+0000 7fab8affd640 1 -- 172.21.15.12:0/712362905 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab90017070 con 0x7fab940a6810 2024-09-11T01:43:08.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.911+0000 7fab8affd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:08.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.911+0000 7fab8affd640 1 -- 172.21.15.12:0/712362905 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab90011d80 con 0x7fab940a6810 2024-09-11T01:43:08.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.911+0000 7fab9c517640 5 monclient: authenticate success, global_id 94716 2024-09-11T01:43:08.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.911+0000 7fab9c517640 10 monclient: _renew_subs 2024-09-11T01:43:08.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.911+0000 7fab9c517640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:08.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.911+0000 7fab9c517640 1 -- 172.21.15.12:0/712362905 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab9413ab50 con 0x7fab940a6810 2024-09-11T01:43:08.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.911+0000 7fab8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:08.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.911+0000 7fab8affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:08.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.911+0000 7fab8affd640 10 monclient: dump: 2024-09-11T01:43:08.925 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:08.925 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:08.925 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.925 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:08.925 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:08.925 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:08.925 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:08.925 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:08.926 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:08.926 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:08.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.911+0000 7fab98ab0640 4 set_mon_vals no callback set 2024-09-11T01:43:08.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.911+0000 7fab9c517640 10 monclient: _renew_subs 2024-09-11T01:43:08.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.911+0000 7fab9c517640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:08.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.911+0000 7fab9c517640 1 -- 172.21.15.12:0/712362905 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab940a1880 con 0x7fab940a6810 2024-09-11T01:43:08.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.912+0000 7fab9c517640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:08.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.912+0000 7fab9c517640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:08.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.912+0000 7fab9c517640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:08.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.912+0000 7fab8affd640 1 -- 172.21.15.12:0/712362905 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fab9001c460 con 0x7fab940a6810 2024-09-11T01:43:08.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.912+0000 7fab9c517640 1 -- 172.21.15.12:0/712362905 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab941385f0 con 0x7fab940a6810 2024-09-11T01:43:08.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.913+0000 7fab8affd640 1 --2- 172.21.15.12:0/712362905 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fab7803e210 0x7fab780406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:08.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.913+0000 7fab8affd640 1 -- 172.21.15.12:0/712362905 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(475..475 src has 285..475) ==== 4815+0+0 (secure 0 0 0) 0x7fab90052bc0 con 0x7fab940a6810 2024-09-11T01:43:08.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.913+0000 7fab99ab2640 1 --2- 172.21.15.12:0/712362905 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fab7803e210 0x7fab780406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:08.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.913+0000 7fab99ab2640 10 monclient: get_auth_request con 0x7fab7803e210 auth_method 0 2024-09-11T01:43:08.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.915+0000 7fab99ab2640 1 --2- 172.21.15.12:0/712362905 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fab7803e210 0x7fab780406d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fab94139de0 tx=0x7fab84006d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:08.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.917+0000 7fab8affd640 1 -- 172.21.15.12:0/712362905 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fab900515a0 con 0x7fab940a6810 2024-09-11T01:43:08.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.917+0000 7fab8affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:08.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:08.917+0000 7fab8affd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:43:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:09.177+0000 7fab9c517640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:43:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:09.177+0000 7fab9c517640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:43:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:09.177+0000 7fab9c517640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:09.177+0000 7fab9c517640 1 -- 172.21.15.12:0/712362905 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fab9413ca80 con 0x7fab940a6810 2024-09-11T01:43:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:09.177+0000 7fab8affd640 1 -- 172.21.15.12:0/712362905 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fab9001c960 con 0x7fab940a6810 2024-09-11T01:43:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:09.177+0000 7fab8affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:43:09.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:09.177+0000 7fab8affd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:43:09.180 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":128,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:43:09.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:09.180+0000 7fab9c517640 1 -- 172.21.15.12:0/712362905 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fab7803e210 msgr2=0x7fab780406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:09.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:09.180+0000 7fab9c517640 1 --2- 172.21.15.12:0/712362905 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fab7803e210 0x7fab780406d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fab94139de0 tx=0x7fab84006d30 comp rx=0 tx=0).stop 2024-09-11T01:43:09.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:09.180+0000 7fab9c517640 10 monclient: shutdown 2024-09-11T01:43:09.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:09.180+0000 7fab9c517640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:09.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:09.180+0000 7fab9c517640 1 -- 172.21.15.12:0/712362905 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab940a6810 msgr2=0x7fab9409f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:09.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:09.180+0000 7fab9c517640 1 --2- 172.21.15.12:0/712362905 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab940a6810 0x7fab9409f5f0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fab9000c430 tx=0x7fab9000c8f0 comp rx=0 tx=0).stop 2024-09-11T01:43:09.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:09.180+0000 7fab9c517640 1 -- 172.21.15.12:0/712362905 shutdown_connections 2024-09-11T01:43:09.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:09.180+0000 7fab9c517640 1 --2- 172.21.15.12:0/712362905 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab9413d640 0x7fab940a0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:09.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:09.180+0000 7fab9c517640 1 --2- 172.21.15.12:0/712362905 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fab94138210 0x7fab9409fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:09.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:09.180+0000 7fab9c517640 1 --2- 172.21.15.12:0/712362905 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fab7803e210 0x7fab780406d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:09.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:09.180+0000 7fab9c517640 1 --2- 172.21.15.12:0/712362905 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fab940a6810 0x7fab9409f5f0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:09.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:09.180+0000 7fab9c517640 1 -- 172.21.15.12:0/712362905 >> 172.21.15.12:0/712362905 conn(0x7fab940aaa80 msgr2=0x7fab94137040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:09.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:09.180+0000 7fab9c517640 1 -- 172.21.15.12:0/712362905 shutdown_connections 2024-09-11T01:43:09.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:09.180+0000 7fab9c517640 1 -- 172.21.15.12:0/712362905 wait complete. 2024-09-11T01:43:12.193 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:43:12.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.305+0000 7f7d1d15d640 10 monclient: get_monmap_and_config 2024-09-11T01:43:12.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.305+0000 7f7d1d15d640 10 monclient: build_initial_monmap 2024-09-11T01:43:12.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.305+0000 7f7d1d15d640 10 monclient: monmap: 2024-09-11T01:43:12.307 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:12.307 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:12.307 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:43:12.306502+0000 2024-09-11T01:43:12.307 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:43:12.306502+0000 2024-09-11T01:43:12.307 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:12.308 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:12.308 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:43:12.308 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:43:12.308 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:43:12.308 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:43:12.308 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:43:12.308 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:12.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.306+0000 7f7d1d15d640 1 Processor -- start 2024-09-11T01:43:12.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.306+0000 7f7d1d15d640 1 -- start start 2024-09-11T01:43:12.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.306+0000 7f7d1d15d640 10 monclient: init 2024-09-11T01:43:12.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d1d15d640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:12.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d1d15d640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-11T01:43:12.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d1d15d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7d1809da80 addr v1:172.21.15.71:6789/0 2024-09-11T01:43:12.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d1d15d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7d18136010 0x7f7d1813ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:12.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d1d15d640 10 monclient(hunting): picked mon.noname-c con 0x7f7d18136010 addr v2:172.21.15.71:3300/0 2024-09-11T01:43:12.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d1d15d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7d1813b390 addr v1:172.21.15.12:6789/0 2024-09-11T01:43:12.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d1d15d640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d1806d5b0 con 0x7f7d1813b390 2024-09-11T01:43:12.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d1d15d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d1806d890 con 0x7f7d1809da80 2024-09-11T01:43:12.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d1d15d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:12.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d1d15d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f7d180a1590 con 0x7f7d18136010 2024-09-11T01:43:12.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d1d15d640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:12.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d1d15d640 10 monclient(hunting): authenticate will time out at 1598.749184s 2024-09-11T01:43:12.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d1759e640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7d1813b390 0x7f7d1813d780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37674/0 (socket says 172.21.15.12:37674) 2024-09-11T01:43:12.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d1759e640 1 -- 172.21.15.12:0/1767353224 learned_addr learned my addr 172.21.15.12:0/1767353224 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:43:12.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d0ffff640 1 --2- 172.21.15.12:0/1767353224 >> v2:172.21.15.71:3300/0 conn(0x7f7d18136010 0x7f7d1813ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:12.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d0ffff640 10 monclient(hunting): get_auth_request con 0x7f7d18136010 auth_method 0 2024-09-11T01:43:12.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:12.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:12.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:12.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:12.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:12.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:12.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d0ffff640 10 monclient(hunting): handle_auth_done global_id 134317 payload 274 2024-09-11T01:43:12.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d0ffff640 1 -- 172.21.15.12:0/1767353224 >> v1:172.21.15.71:6789/0 conn(0x7f7d1809da80 legacy=0x7f7d1809de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:12.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d0ffff640 1 -- 172.21.15.12:0/1767353224 >> v1:172.21.15.12:6789/0 conn(0x7f7d1813b390 legacy=0x7f7d1813d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:12.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d0ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:43:12.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d0ffff640 1 monclient: found mon.noname-c 2024-09-11T01:43:12.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:12.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d0ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:43:12.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.307+0000 7f7d0ffff640 1 -- 172.21.15.12:0/1767353224 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d1806d2d0 con 0x7f7d18136010 2024-09-11T01:43:12.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.311+0000 7f7d0ffff640 1 -- 172.21.15.12:0/1767353224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d18136010 msgr2=0x7f7d1813ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:12.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.311+0000 7f7d0ffff640 1 --2- 172.21.15.12:0/1767353224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d18136010 0x7f7d1813ae50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7d000097a0 tx=0x7f7d00029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eaaf1a0af0f4eb0b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:12.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.311+0000 7f7d1659c640 1 -- 172.21.15.12:0/1767353224 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d00037070 con 0x7f7d18136010 2024-09-11T01:43:12.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.311+0000 7f7d1659c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:12.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.311+0000 7f7d1659c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:43:12.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.311+0000 7f7d1659c640 10 monclient: dump: 2024-09-11T01:43:12.317 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:12.317 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:12.317 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:12.317 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:12.317 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:12.317 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:12.317 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:12.317 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:12.317 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:12.317 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:12.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.311+0000 7f7d1659c640 10 monclient: _finish_auth 0 2024-09-11T01:43:12.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.311+0000 7f7d1659c640 10 monclient: _check_auth_tickets 2024-09-11T01:43:12.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.311+0000 7f7d1659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:12.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.311+0000 7f7d1659c640 1 -- 172.21.15.12:0/1767353224 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d00032640 con 0x7f7d18136010 2024-09-11T01:43:12.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.311+0000 7f7d1659c640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:12.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.311+0000 7f7d1659c640 1 -- 172.21.15.12:0/1767353224 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d00032a10 con 0x7f7d18136010 2024-09-11T01:43:12.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.311+0000 7f7d1659c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:12.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.311+0000 7f7d1659c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:43:12.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.311+0000 7f7d1659c640 10 monclient: dump: 2024-09-11T01:43:12.318 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:12.318 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:12.318 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:12.318 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:12.318 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:12.318 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:12.319 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:12.319 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:12.319 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:12.319 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:12.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.311+0000 7f7d1d15d640 5 monclient: authenticate success, global_id 134317 2024-09-11T01:43:12.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.311+0000 7f7d1d15d640 10 monclient: get_monmap_and_config success 2024-09-11T01:43:12.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.311+0000 7f7d1d15d640 4 set_mon_vals no callback set 2024-09-11T01:43:12.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.311+0000 7f7d1d15d640 10 monclient: shutdown 2024-09-11T01:43:12.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.311+0000 7f7d1d15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:12.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.311+0000 7f7d1d15d640 1 -- 172.21.15.12:0/1767353224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d18136010 msgr2=0x7f7d1813ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:12.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.311+0000 7f7d1d15d640 1 --2- 172.21.15.12:0/1767353224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d18136010 0x7f7d1813ae50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7d000097a0 tx=0x7f7d00029fd0 comp rx=0 tx=0).stop 2024-09-11T01:43:12.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.312+0000 7f7d1d15d640 1 -- 172.21.15.12:0/1767353224 shutdown_connections 2024-09-11T01:43:12.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.312+0000 7f7d1d15d640 1 --2- 172.21.15.12:0/1767353224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d18136010 0x7f7d1813ae50 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:12.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.312+0000 7f7d1d15d640 1 -- 172.21.15.12:0/1767353224 >> 172.21.15.12:0/1767353224 conn(0x7f7d180a3ea0 msgr2=0x7f7d180a42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:12.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.312+0000 7f7d1d15d640 1 -- 172.21.15.12:0/1767353224 shutdown_connections 2024-09-11T01:43:12.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.312+0000 7f7d1d15d640 1 -- 172.21.15.12:0/1767353224 wait complete. 2024-09-11T01:43:12.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.312+0000 7f7d1d15d640 10 monclient: build_initial_monmap 2024-09-11T01:43:12.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.312+0000 7f7d1d15d640 10 monclient: monmap: 2024-09-11T01:43:12.320 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:12.320 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:43:12.320 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:43:12.320 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:43:12.320 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:12.320 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:12.320 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:43:12.320 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:43:12.321 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:43:12.321 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:12.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d1d15d640 1 Processor -- start 2024-09-11T01:43:12.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d1d15d640 1 -- start start 2024-09-11T01:43:12.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d1d15d640 10 monclient: init 2024-09-11T01:43:12.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d1d15d640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:12.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d1d15d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:43:12.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d1d15d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d1809da80 0x7f7d1813a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:12.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d1d15d640 10 monclient(hunting): picked mon.noname-a con 0x7f7d1809da80 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:12.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d1d15d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d18136010 0x7f7d181cafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:12.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d1d15d640 10 monclient(hunting): picked mon.noname-b con 0x7f7d18136010 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:12.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d1d15d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7d1813b390 0x7f7d181cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:12.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d1d15d640 10 monclient(hunting): picked mon.noname-c con 0x7f7d1813b390 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:12.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d1d15d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:12.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d1d15d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7d180a0ea0 con 0x7f7d1809da80 2024-09-11T01:43:12.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d1d15d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:12.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d1d15d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f7d180a0d20 con 0x7f7d18136010 2024-09-11T01:43:12.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d1d15d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:12.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d1d15d640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7d1813eea0 con 0x7f7d1813b390 2024-09-11T01:43:12.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d1d15d640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:12.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d1d15d640 10 monclient(hunting): authenticate will time out at 1598.755736s 2024-09-11T01:43:12.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d1759e640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7d1813b390 msgr2=0x7f7d181cb4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:43:12.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d1759e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7d1813b390 0x7f7d181cb4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:12.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d16d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d1809da80 0x7f7d1813a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:12.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d16d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d1809da80 0x7f7d1813a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34990/0 (socket says 172.21.15.12:34990) 2024-09-11T01:43:12.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d16d9d640 1 -- 172.21.15.12:0/1544725365 learned_addr learned my addr 172.21.15.12:0/1544725365 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:12.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d16d9d640 10 monclient(hunting): get_auth_request con 0x7f7d1809da80 auth_method 0 2024-09-11T01:43:12.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d16d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:12.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d16d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:12.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d16d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:12.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d16d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:12.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d16d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:12.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.313+0000 7f7d16d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:12.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.314+0000 7f7d0ffff640 1 --2- 172.21.15.12:0/1544725365 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d18136010 0x7f7d181cafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:12.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.314+0000 7f7d16d9d640 10 monclient(hunting): handle_auth_done global_id 94719 payload 824 2024-09-11T01:43:12.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.314+0000 7f7d16d9d640 1 -- 172.21.15.12:0/1544725365 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7d1813b390 msgr2=0x7f7d181cb4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:43:12.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.314+0000 7f7d16d9d640 1 --2- 172.21.15.12:0/1544725365 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7d1813b390 0x7f7d181cb4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:12.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.314+0000 7f7d16d9d640 1 -- 172.21.15.12:0/1544725365 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d18136010 msgr2=0x7f7d181cafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:12.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.314+0000 7f7d16d9d640 1 --2- 172.21.15.12:0/1544725365 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d18136010 0x7f7d181cafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:12.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.314+0000 7f7d16d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:43:12.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.314+0000 7f7d16d9d640 1 monclient: found mon.noname-a 2024-09-11T01:43:12.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.314+0000 7f7d16d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:12.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.314+0000 7f7d16d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:12.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.314+0000 7f7d16d9d640 1 -- 172.21.15.12:0/1544725365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d1813e2b0 con 0x7f7d1809da80 2024-09-11T01:43:12.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.314+0000 7f7d0ffff640 10 monclient: get_auth_request con 0x7f7d18136010 auth_method 0 2024-09-11T01:43:12.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.314+0000 7f7d0ffff640 1 --2- 172.21.15.12:0/1544725365 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d18136010 0x7f7d181cafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:43:12.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.316+0000 7f7d16d9d640 1 -- 172.21.15.12:0/1544725365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d1809da80 msgr2=0x7f7d1813a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:12.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.316+0000 7f7d16d9d640 1 --2- 172.21.15.12:0/1544725365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d1809da80 0x7f7d1813a2c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7d0800c430 tx=0x7f7d0800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:12.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.316+0000 7f7d0f7fe640 1 -- 172.21.15.12:0/1544725365 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d080119a0 con 0x7f7d1809da80 2024-09-11T01:43:12.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.316+0000 7f7d0f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:12.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.316+0000 7f7d0f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:12.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.316+0000 7f7d0f7fe640 10 monclient: dump: 2024-09-11T01:43:12.327 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:12.327 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:12.327 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:12.327 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:12.327 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:12.327 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:12.327 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:12.327 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:12.327 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:12.327 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:12.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.316+0000 7f7d0f7fe640 10 monclient: _finish_auth 0 2024-09-11T01:43:12.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.316+0000 7f7d0f7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:43:12.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.316+0000 7f7d0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:12.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.316+0000 7f7d0f7fe640 1 -- 172.21.15.12:0/1544725365 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d08017070 con 0x7f7d1809da80 2024-09-11T01:43:12.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.316+0000 7f7d0f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:12.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.316+0000 7f7d0f7fe640 1 -- 172.21.15.12:0/1544725365 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d08011d80 con 0x7f7d1809da80 2024-09-11T01:43:12.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.316+0000 7f7d1d15d640 5 monclient: authenticate success, global_id 94719 2024-09-11T01:43:12.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.316+0000 7f7d1d15d640 10 monclient: _renew_subs 2024-09-11T01:43:12.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.316+0000 7f7d1d15d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:12.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.316+0000 7f7d1d15d640 1 -- 172.21.15.12:0/1544725365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d1813abb0 con 0x7f7d1809da80 2024-09-11T01:43:12.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.316+0000 7f7d0f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:12.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.316+0000 7f7d0f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:12.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.316+0000 7f7d1559a640 4 set_mon_vals no callback set 2024-09-11T01:43:12.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.316+0000 7f7d0f7fe640 10 monclient: dump: 2024-09-11T01:43:12.328 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:12.328 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:12.328 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:12.329 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:12.329 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:12.329 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:12.329 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:12.329 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:12.329 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:12.329 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:12.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.316+0000 7f7d1d15d640 10 monclient: _renew_subs 2024-09-11T01:43:12.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.316+0000 7f7d1d15d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:12.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.316+0000 7f7d1d15d640 1 -- 172.21.15.12:0/1544725365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d1813da70 con 0x7f7d1809da80 2024-09-11T01:43:12.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.317+0000 7f7d1d15d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:12.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.317+0000 7f7d1d15d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:12.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.317+0000 7f7d1d15d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:12.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.317+0000 7f7d0f7fe640 1 -- 172.21.15.12:0/1544725365 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f7d0801c460 con 0x7f7d1809da80 2024-09-11T01:43:12.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.317+0000 7f7d1d15d640 1 -- 172.21.15.12:0/1544725365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d18139200 con 0x7f7d1809da80 2024-09-11T01:43:12.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.317+0000 7f7d0f7fe640 1 --2- 172.21.15.12:0/1544725365 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7cf803e1c0 0x7f7cf8040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:12.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.317+0000 7f7d0f7fe640 1 -- 172.21.15.12:0/1544725365 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(475..475 src has 285..475) ==== 4815+0+0 (secure 0 0 0) 0x7f7d08052d80 con 0x7f7d1809da80 2024-09-11T01:43:12.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.318+0000 7f7d0ffff640 1 --2- 172.21.15.12:0/1544725365 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7cf803e1c0 0x7f7cf8040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:12.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.318+0000 7f7d0ffff640 10 monclient: get_auth_request con 0x7f7cf803e1c0 auth_method 0 2024-09-11T01:43:12.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.320+0000 7f7d0ffff640 1 --2- 172.21.15.12:0/1544725365 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7cf803e1c0 0x7f7cf8040680 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7d180a1a40 tx=0x7f7d0002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:12.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.322+0000 7f7d0f7fe640 1 -- 172.21.15.12:0/1544725365 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7d08051760 con 0x7f7d1809da80 2024-09-11T01:43:12.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.322+0000 7f7d0f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:12.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.322+0000 7f7d0f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:43:12.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.578+0000 7f7d1d15d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:43:12.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.578+0000 7f7d1d15d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:43:12.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.578+0000 7f7d1d15d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:12.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.578+0000 7f7d1d15d640 1 -- 172.21.15.12:0/1544725365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7d1809dee0 con 0x7f7d1809da80 2024-09-11T01:43:12.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.578+0000 7f7d0f7fe640 1 -- 172.21.15.12:0/1544725365 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7f7d0801c960 con 0x7f7d1809da80 2024-09-11T01:43:12.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.578+0000 7f7d0f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:43:12.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.578+0000 7f7d0f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:43:12.580 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":128,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:43:12.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.580+0000 7f7d1d15d640 1 -- 172.21.15.12:0/1544725365 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7cf803e1c0 msgr2=0x7f7cf8040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:12.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.580+0000 7f7d1d15d640 1 --2- 172.21.15.12:0/1544725365 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7cf803e1c0 0x7f7cf8040680 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7d180a1a40 tx=0x7f7d0002e000 comp rx=0 tx=0).stop 2024-09-11T01:43:12.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.580+0000 7f7d1d15d640 10 monclient: shutdown 2024-09-11T01:43:12.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.580+0000 7f7d1d15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:12.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.580+0000 7f7d1d15d640 1 -- 172.21.15.12:0/1544725365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d1809da80 msgr2=0x7f7d1813a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:12.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.580+0000 7f7d1d15d640 1 --2- 172.21.15.12:0/1544725365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d1809da80 0x7f7d1813a2c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7d0800c430 tx=0x7f7d0800c8f0 comp rx=0 tx=0).stop 2024-09-11T01:43:12.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.580+0000 7f7d1d15d640 1 -- 172.21.15.12:0/1544725365 shutdown_connections 2024-09-11T01:43:12.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.580+0000 7f7d1d15d640 1 --2- 172.21.15.12:0/1544725365 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7d1813b390 0x7f7d181cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:12.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.580+0000 7f7d1d15d640 1 --2- 172.21.15.12:0/1544725365 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d18136010 0x7f7d181cafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:12.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.580+0000 7f7d1d15d640 1 --2- 172.21.15.12:0/1544725365 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7cf803e1c0 0x7f7cf8040680 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:12.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.580+0000 7f7d1d15d640 1 --2- 172.21.15.12:0/1544725365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7d1809da80 0x7f7d1813a2c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:12.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.580+0000 7f7d1d15d640 1 -- 172.21.15.12:0/1544725365 >> 172.21.15.12:0/1544725365 conn(0x7f7d180a3ea0 msgr2=0x7f7d180a3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:12.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.580+0000 7f7d1d15d640 1 -- 172.21.15.12:0/1544725365 shutdown_connections 2024-09-11T01:43:12.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:12.580+0000 7f7d1d15d640 1 -- 172.21.15.12:0/1544725365 wait complete. 2024-09-11T01:43:15.593 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:43:15.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.705+0000 7ff49e591640 10 monclient: get_monmap_and_config 2024-09-11T01:43:15.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.705+0000 7ff49e591640 10 monclient: build_initial_monmap 2024-09-11T01:43:15.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.705+0000 7ff49e591640 10 monclient: monmap: 2024-09-11T01:43:15.707 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:15.707 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:15.707 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:43:15.705808+0000 2024-09-11T01:43:15.707 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:43:15.705808+0000 2024-09-11T01:43:15.707 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:15.707 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:15.707 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:43:15.707 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:43:15.707 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:43:15.707 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:43:15.707 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:43:15.707 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:15.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.705+0000 7ff49e591640 1 Processor -- start 2024-09-11T01:43:15.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.705+0000 7ff49e591640 1 -- start start 2024-09-11T01:43:15.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.705+0000 7ff49e591640 10 monclient: init 2024-09-11T01:43:15.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff49e591640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:15.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff49e591640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-11T01:43:15.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff49e591640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff498138210 0x7ff4981385f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:15.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff49e591640 10 monclient(hunting): picked mon.noname-a con 0x7ff498138210 addr v2:172.21.15.12:3300/0 2024-09-11T01:43:15.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff49e591640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff4980a6810 0x7ff4980a5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:15.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff49e591640 10 monclient(hunting): picked mon.noname-c con 0x7ff4980a6810 addr v2:172.21.15.71:3300/0 2024-09-11T01:43:15.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff49e591640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff4980a55d0 addr v1:172.21.15.71:6789/0 2024-09-11T01:43:15.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff49e591640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff49809ba20 con 0x7ff4980a55d0 2024-09-11T01:43:15.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff49e591640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:15.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff49e591640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff49809c4c0 con 0x7ff498138210 2024-09-11T01:43:15.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff49e591640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:15.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff49e591640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7ff4980a6590 con 0x7ff4980a6810 2024-09-11T01:43:15.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff49e591640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:15.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff49e591640 10 monclient(hunting): authenticate will time out at 1602.148469s 2024-09-11T01:43:15.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff497fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff498138210 0x7ff4981385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:15.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff497fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff498138210 0x7ff4981385f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35004/0 (socket says 172.21.15.12:35004) 2024-09-11T01:43:15.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff497fff640 1 -- 172.21.15.12:0/2510176055 learned_addr learned my addr 172.21.15.12:0/2510176055 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:15.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff497fff640 10 monclient(hunting): get_auth_request con 0x7ff498138210 auth_method 0 2024-09-11T01:43:15.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff497fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:15.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff497fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:15.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff497fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:15.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff497fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:15.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff497fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:15.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff497fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:15.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff4977fe640 1 --2- 172.21.15.12:0/2510176055 >> v2:172.21.15.71:3300/0 conn(0x7ff4980a6810 0x7ff4980a5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:15.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff4977fe640 10 monclient(hunting): get_auth_request con 0x7ff4980a6810 auth_method 0 2024-09-11T01:43:15.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff4977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:15.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff4977fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:15.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff4977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:15.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff497fff640 10 monclient(hunting): handle_auth_done global_id 94722 payload 274 2024-09-11T01:43:15.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff497fff640 1 -- 172.21.15.12:0/2510176055 >> v2:172.21.15.71:3300/0 conn(0x7ff4980a6810 msgr2=0x7ff4980a5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:15.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff497fff640 1 --2- 172.21.15.12:0/2510176055 >> v2:172.21.15.71:3300/0 conn(0x7ff4980a6810 0x7ff4980a5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:15.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff497fff640 1 -- 172.21.15.12:0/2510176055 >> v1:172.21.15.71:6789/0 conn(0x7ff4980a55d0 legacy=0x7ff49813f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:15.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff497fff640 10 monclient: _finish_hunting 0 2024-09-11T01:43:15.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff497fff640 1 monclient: found mon.noname-a 2024-09-11T01:43:15.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff497fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:15.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff497fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:15.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.706+0000 7ff497fff640 1 -- 172.21.15.12:0/2510176055 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff49809c1e0 con 0x7ff498138210 2024-09-11T01:43:15.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.709+0000 7ff497fff640 1 -- 172.21.15.12:0/2510176055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff498138210 msgr2=0x7ff4981385f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:15.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.709+0000 7ff497fff640 1 --2- 172.21.15.12:0/2510176055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff498138210 0x7ff4981385f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff498074ee0 tx=0x7ff48802a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa704419b74f77e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:15.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff496ffd640 1 -- 172.21.15.12:0/2510176055 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff488037070 con 0x7ff498138210 2024-09-11T01:43:15.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff496ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:15.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff496ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:15.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff496ffd640 10 monclient: dump: 2024-09-11T01:43:15.725 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:15.725 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:15.725 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:15.725 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:15.725 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:15.725 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:15.725 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:15.725 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:15.725 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:15.726 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:15.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff496ffd640 10 monclient: _finish_auth 0 2024-09-11T01:43:15.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff496ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:43:15.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff496ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:15.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff496ffd640 1 -- 172.21.15.12:0/2510176055 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff488032640 con 0x7ff498138210 2024-09-11T01:43:15.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff49e591640 5 monclient: authenticate success, global_id 94722 2024-09-11T01:43:15.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff49e591640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:43:15.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff496ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:15.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff496ffd640 1 -- 172.21.15.12:0/2510176055 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff488032a10 con 0x7ff498138210 2024-09-11T01:43:15.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff49e591640 10 monclient: get_monmap_and_config success 2024-09-11T01:43:15.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff49e591640 4 set_mon_vals no callback set 2024-09-11T01:43:15.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff496ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:15.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff496ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:15.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff496ffd640 10 monclient: dump: 2024-09-11T01:43:15.727 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:15.727 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:15.727 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:15.727 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:15.727 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:15.727 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:15.727 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:15.727 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:15.727 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:15.727 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:15.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff49e591640 10 monclient: shutdown 2024-09-11T01:43:15.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff49e591640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:15.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff49e591640 1 -- 172.21.15.12:0/2510176055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff498138210 msgr2=0x7ff4981385f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:15.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff49e591640 1 --2- 172.21.15.12:0/2510176055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff498138210 0x7ff4981385f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff498074ee0 tx=0x7ff48802a0b0 comp rx=0 tx=0).stop 2024-09-11T01:43:15.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff49e591640 1 -- 172.21.15.12:0/2510176055 shutdown_connections 2024-09-11T01:43:15.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff49e591640 1 --2- 172.21.15.12:0/2510176055 >> v2:172.21.15.71:3300/0 conn(0x7ff4980a6810 0x7ff4980a5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:15.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff49e591640 1 --2- 172.21.15.12:0/2510176055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff498138210 0x7ff4981385f0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:15.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff49e591640 1 -- 172.21.15.12:0/2510176055 >> 172.21.15.12:0/2510176055 conn(0x7ff4980aaa80 msgr2=0x7ff4980aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:15.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff49e591640 1 -- 172.21.15.12:0/2510176055 shutdown_connections 2024-09-11T01:43:15.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.710+0000 7ff49e591640 1 -- 172.21.15.12:0/2510176055 wait complete. 2024-09-11T01:43:15.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.711+0000 7ff49e591640 10 monclient: build_initial_monmap 2024-09-11T01:43:15.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.711+0000 7ff49e591640 10 monclient: monmap: 2024-09-11T01:43:15.728 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:15.729 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:43:15.729 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:43:15.729 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:43:15.729 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:15.729 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:15.729 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:43:15.729 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:43:15.729 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:43:15.729 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:15.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.711+0000 7ff49e591640 1 Processor -- start 2024-09-11T01:43:15.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.711+0000 7ff49e591640 1 -- start start 2024-09-11T01:43:15.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.711+0000 7ff49e591640 10 monclient: init 2024-09-11T01:43:15.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff49e591640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:15.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff49e591640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:43:15.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff49e591640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4980a55d0 0x7ff4981cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:15.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff49e591640 10 monclient(hunting): picked mon.noname-b con 0x7ff4980a55d0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:15.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff49e591640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4980a6810 0x7ff498136ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:15.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff49e591640 10 monclient(hunting): picked mon.noname-a con 0x7ff4980a6810 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:15.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff49e591640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff498138210 0x7ff498137430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:15.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff49e591640 10 monclient(hunting): picked mon.noname-c con 0x7ff498138210 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:15.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff49e591640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:15.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff49e591640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff498138b30 con 0x7ff4980a6810 2024-09-11T01:43:15.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff49e591640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:15.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff49e591640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff49809d6e0 con 0x7ff4980a55d0 2024-09-11T01:43:15.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff49e591640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:15.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff49e591640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff498138cb0 con 0x7ff498138210 2024-09-11T01:43:15.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff49e591640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:15.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff49e591640 10 monclient(hunting): authenticate will time out at 1602.154506s 2024-09-11T01:43:15.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff49cb2e640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff498138210 msgr2=0x7ff498137430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:43:15.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff49cb2e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff498138210 0x7ff498137430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:15.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff4977fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4980a6810 0x7ff498136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:15.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff4977fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4980a6810 0x7ff498136ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35016/0 (socket says 172.21.15.12:35016) 2024-09-11T01:43:15.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff4977fe640 1 -- 172.21.15.12:0/1154737354 learned_addr learned my addr 172.21.15.12:0/1154737354 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:15.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff4977fe640 10 monclient(hunting): get_auth_request con 0x7ff4980a6810 auth_method 0 2024-09-11T01:43:15.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff4977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:15.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff4977fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:15.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff4977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:15.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff497fff640 1 --2- 172.21.15.12:0/1154737354 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4980a55d0 0x7ff4981cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:15.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff497fff640 10 monclient(hunting): get_auth_request con 0x7ff4980a55d0 auth_method 0 2024-09-11T01:43:15.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff497fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:15.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff497fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:15.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff497fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:15.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff4977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:15.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff4977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:15.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff4977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:15.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff497fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:15.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff497fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:15.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff497fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:15.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff4977fe640 10 monclient(hunting): handle_auth_done global_id 94725 payload 824 2024-09-11T01:43:15.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff4977fe640 1 -- 172.21.15.12:0/1154737354 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff498138210 msgr2=0x7ff498137430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:43:15.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff4977fe640 1 --2- 172.21.15.12:0/1154737354 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff498138210 0x7ff498137430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:15.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff4977fe640 1 -- 172.21.15.12:0/1154737354 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4980a55d0 msgr2=0x7ff4981cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:15.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff4977fe640 1 --2- 172.21.15.12:0/1154737354 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4980a55d0 0x7ff4981cd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:15.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff4977fe640 10 monclient: _finish_hunting 0 2024-09-11T01:43:15.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff4977fe640 1 monclient: found mon.noname-a 2024-09-11T01:43:15.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff4977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:15.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff4977fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:15.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.712+0000 7ff4977fe640 1 -- 172.21.15.12:0/1154737354 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff498124010 con 0x7ff4980a6810 2024-09-11T01:43:15.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.715+0000 7ff4977fe640 1 -- 172.21.15.12:0/1154737354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4980a6810 msgr2=0x7ff498136ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:15.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.715+0000 7ff4977fe640 1 --2- 172.21.15.12:0/1154737354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4980a6810 0x7ff498136ef0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff48c00d800 tx=0x7ff48c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:15.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.715+0000 7ff494ff9640 1 -- 172.21.15.12:0/1154737354 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48c00b870 con 0x7ff4980a6810 2024-09-11T01:43:15.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.715+0000 7ff494ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:15.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.715+0000 7ff494ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:15.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.715+0000 7ff494ff9640 10 monclient: dump: 2024-09-11T01:43:15.734 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:15.734 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:15.734 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:15.734 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:15.734 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:15.734 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:15.734 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:15.734 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:15.734 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:15.734 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:15.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.715+0000 7ff494ff9640 10 monclient: _finish_auth 0 2024-09-11T01:43:15.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.715+0000 7ff494ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:43:15.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.715+0000 7ff494ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:15.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.715+0000 7ff49e591640 5 monclient: authenticate success, global_id 94725 2024-09-11T01:43:15.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.715+0000 7ff494ff9640 1 -- 172.21.15.12:0/1154737354 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff48c019070 con 0x7ff4980a6810 2024-09-11T01:43:15.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.715+0000 7ff494ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:15.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.715+0000 7ff494ff9640 1 -- 172.21.15.12:0/1154737354 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48c00bc30 con 0x7ff4980a6810 2024-09-11T01:43:15.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.715+0000 7ff49e591640 10 monclient: _renew_subs 2024-09-11T01:43:15.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.715+0000 7ff49e591640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:15.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.715+0000 7ff49e591640 1 -- 172.21.15.12:0/1154737354 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4981242f0 con 0x7ff4980a6810 2024-09-11T01:43:15.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.715+0000 7ff494ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:15.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.715+0000 7ff494ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:15.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.715+0000 7ff494ff9640 10 monclient: dump: 2024-09-11T01:43:15.735 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:15.735 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:15.735 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:15.735 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:15.736 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:15.736 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:15.736 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:15.736 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:15.736 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:15.736 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:15.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.715+0000 7ff4967fc640 4 set_mon_vals no callback set 2024-09-11T01:43:15.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.715+0000 7ff49e591640 10 monclient: _renew_subs 2024-09-11T01:43:15.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.715+0000 7ff49e591640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:15.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.715+0000 7ff49e591640 1 -- 172.21.15.12:0/1154737354 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff498120ec0 con 0x7ff4980a6810 2024-09-11T01:43:15.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.716+0000 7ff494ff9640 1 -- 172.21.15.12:0/1154737354 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff48c013070 con 0x7ff4980a6810 2024-09-11T01:43:15.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.716+0000 7ff49e591640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:15.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.716+0000 7ff49e591640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:15.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.716+0000 7ff49e591640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:15.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.716+0000 7ff49e591640 1 -- 172.21.15.12:0/1154737354 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4980a5090 con 0x7ff4980a6810 2024-09-11T01:43:15.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.716+0000 7ff494ff9640 1 --2- 172.21.15.12:0/1154737354 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff48403e1c0 0x7ff484040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:15.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.716+0000 7ff494ff9640 1 -- 172.21.15.12:0/1154737354 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(475..475 src has 285..475) ==== 4815+0+0 (secure 0 0 0) 0x7ff48c052e80 con 0x7ff4980a6810 2024-09-11T01:43:15.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.716+0000 7ff497fff640 1 --2- 172.21.15.12:0/1154737354 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff48403e1c0 0x7ff484040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:15.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.717+0000 7ff497fff640 10 monclient: get_auth_request con 0x7ff48403e1c0 auth_method 0 2024-09-11T01:43:15.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.719+0000 7ff497fff640 1 --2- 172.21.15.12:0/1154737354 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff48403e1c0 0x7ff484040680 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff49809cf00 tx=0x7ff48802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:15.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.721+0000 7ff494ff9640 1 -- 172.21.15.12:0/1154737354 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff48c051860 con 0x7ff4980a6810 2024-09-11T01:43:15.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.721+0000 7ff494ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:15.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.721+0000 7ff494ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:43:15.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.978+0000 7ff49e591640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:43:15.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.978+0000 7ff49e591640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:43:15.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.978+0000 7ff49e591640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:15.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.978+0000 7ff49e591640 1 -- 172.21.15.12:0/1154737354 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff498074b50 con 0x7ff4980a6810 2024-09-11T01:43:15.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.979+0000 7ff494ff9640 1 -- 172.21.15.12:0/1154737354 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7ff48c00bdd0 con 0x7ff4980a6810 2024-09-11T01:43:15.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.979+0000 7ff494ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:43:15.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.979+0000 7ff494ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:43:15.982 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":128,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:43:15.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.981+0000 7ff49e591640 1 -- 172.21.15.12:0/1154737354 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff48403e1c0 msgr2=0x7ff484040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:15.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.981+0000 7ff49e591640 1 --2- 172.21.15.12:0/1154737354 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff48403e1c0 0x7ff484040680 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff49809cf00 tx=0x7ff48802e000 comp rx=0 tx=0).stop 2024-09-11T01:43:15.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.981+0000 7ff49e591640 10 monclient: shutdown 2024-09-11T01:43:15.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.981+0000 7ff49e591640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:15.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.981+0000 7ff49e591640 1 -- 172.21.15.12:0/1154737354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4980a6810 msgr2=0x7ff498136ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:15.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.981+0000 7ff49e591640 1 --2- 172.21.15.12:0/1154737354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4980a6810 0x7ff498136ef0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff48c00d800 tx=0x7ff48c00dcc0 comp rx=0 tx=0).stop 2024-09-11T01:43:15.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.981+0000 7ff49e591640 1 -- 172.21.15.12:0/1154737354 shutdown_connections 2024-09-11T01:43:15.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.981+0000 7ff49e591640 1 --2- 172.21.15.12:0/1154737354 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff498138210 0x7ff498137430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:15.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.981+0000 7ff49e591640 1 --2- 172.21.15.12:0/1154737354 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff48403e1c0 0x7ff484040680 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:15.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.981+0000 7ff49e591640 1 --2- 172.21.15.12:0/1154737354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4980a6810 0x7ff498136ef0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:15.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.981+0000 7ff49e591640 1 --2- 172.21.15.12:0/1154737354 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4980a55d0 0x7ff4981cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:15.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.981+0000 7ff49e591640 1 -- 172.21.15.12:0/1154737354 >> 172.21.15.12:0/1154737354 conn(0x7ff4980aaa80 msgr2=0x7ff4980a8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:15.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.981+0000 7ff49e591640 1 -- 172.21.15.12:0/1154737354 shutdown_connections 2024-09-11T01:43:15.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:15.981+0000 7ff49e591640 1 -- 172.21.15.12:0/1154737354 wait complete. 2024-09-11T01:43:18.995 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:43:19.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.106+0000 7f2c7e8a9640 10 monclient: get_monmap_and_config 2024-09-11T01:43:19.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.106+0000 7f2c7e8a9640 10 monclient: build_initial_monmap 2024-09-11T01:43:19.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.106+0000 7f2c7e8a9640 10 monclient: monmap: 2024-09-11T01:43:19.108 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:19.108 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:19.108 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:43:19.107408+0000 2024-09-11T01:43:19.108 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:43:19.107408+0000 2024-09-11T01:43:19.108 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:19.108 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:19.108 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:43:19.109 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:43:19.109 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:43:19.109 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:43:19.109 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:43:19.109 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.107+0000 7f2c7e8a9640 1 Processor -- start 2024-09-11T01:43:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.107+0000 7f2c7e8a9640 1 -- start start 2024-09-11T01:43:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.107+0000 7f2c7e8a9640 10 monclient: init 2024-09-11T01:43:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.107+0000 7f2c7e8a9640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:19.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.108+0000 7f2c7e8a9640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-11T01:43:19.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.108+0000 7f2c7e8a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2c780a2e90 addr v1:172.21.15.71:6789/0 2024-09-11T01:43:19.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.108+0000 7f2c7e8a9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2c780a3270 0x7f2c780a3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:19.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.108+0000 7f2c7e8a9640 10 monclient(hunting): picked mon.noname-a con 0x7f2c780a3270 addr v2:172.21.15.12:3300/0 2024-09-11T01:43:19.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.108+0000 7f2c7e8a9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c7813d490 0x7f2c7813f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:19.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.108+0000 7f2c7e8a9640 10 monclient(hunting): picked mon.noname-b con 0x7f2c7813d490 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:19.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.108+0000 7f2c7e8a9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c780a46c0 con 0x7f2c780a2e90 2024-09-11T01:43:19.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.108+0000 7f2c7e8a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:19.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.108+0000 7f2c7e8a9640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f2c780a4900 con 0x7f2c780a3270 2024-09-11T01:43:19.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.108+0000 7f2c7e8a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:19.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.108+0000 7f2c7e8a9640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2c7806ee50 con 0x7f2c7813d490 2024-09-11T01:43:19.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.108+0000 7f2c7e8a9640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:19.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.108+0000 7f2c7e8a9640 10 monclient(hunting): authenticate will time out at 1605.550115s 2024-09-11T01:43:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.108+0000 7f2c7ce46640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c7813d490 msgr2=0x7f2c7813f880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:43:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.108+0000 7f2c7ce46640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c7813d490 0x7f2c7813f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.108+0000 7f2c777fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2c780a3270 0x7f2c780a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.108+0000 7f2c777fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2c780a3270 0x7f2c780a3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39078/0 (socket says 172.21.15.12:39078) 2024-09-11T01:43:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.108+0000 7f2c777fe640 1 -- 172.21.15.12:0/3739851872 learned_addr learned my addr 172.21.15.12:0/3739851872 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.108+0000 7f2c777fe640 10 monclient(hunting): get_auth_request con 0x7f2c780a3270 auth_method 0 2024-09-11T01:43:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.108+0000 7f2c777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.108+0000 7f2c777fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.108+0000 7f2c777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.108+0000 7f2c777fe640 1 -- 172.21.15.12:0/3739851872 >> v2:172.21.15.12:3300/0 conn(0x7f2c780a3270 msgr2=0x7f2c780a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:43:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.108+0000 7f2c777fe640 1 -- 172.21.15.12:0/3739851872 >> v2:172.21.15.12:3300/0 conn(0x7f2c780a3270 msgr2=0x7f2c780a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:43:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.108+0000 7f2c777fe640 1 --2- 172.21.15.12:0/3739851872 >> v2:172.21.15.12:3300/0 conn(0x7f2c780a3270 0x7f2c780a3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:19.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.108+0000 7f2c777fe640 1 --2- 172.21.15.12:0/3739851872 >> v2:172.21.15.12:3300/0 conn(0x7f2c780a3270 0x7f2c780a3680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:19.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.110+0000 7f2c77fff640 1 -- 172.21.15.12:0/3739851872 >> v1:172.21.15.71:6789/0 conn(0x7f2c780a2e90 legacy=0x7f2c7806dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:19.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.309+0000 7f2c7ce46640 1 -- 172.21.15.12:0/3739851872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c7813d490 msgr2=0x7f2c7813f880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:43:19.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.309+0000 7f2c7ce46640 1 --2- 172.21.15.12:0/3739851872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c7813d490 0x7f2c7813f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:43:19.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.309+0000 7f2c777fe640 1 --2- 172.21.15.12:0/3739851872 >> v2:172.21.15.12:3300/0 conn(0x7f2c780a3270 0x7f2c780a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:19.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.310+0000 7f2c777fe640 10 monclient(hunting): get_auth_request con 0x7f2c780a3270 auth_method 0 2024-09-11T01:43:19.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.310+0000 7f2c777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:19.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.310+0000 7f2c777fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:19.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.310+0000 7f2c777fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:19.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.310+0000 7f2c777fe640 1 -- 172.21.15.12:0/3739851872 >> v2:172.21.15.12:3300/0 conn(0x7f2c780a3270 msgr2=0x7f2c780a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:43:19.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.310+0000 7f2c777fe640 1 -- 172.21.15.12:0/3739851872 >> v2:172.21.15.12:3300/0 conn(0x7f2c780a3270 msgr2=0x7f2c780a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:43:19.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.310+0000 7f2c777fe640 1 --2- 172.21.15.12:0/3739851872 >> v2:172.21.15.12:3300/0 conn(0x7f2c780a3270 0x7f2c780a3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:19.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.310+0000 7f2c777fe640 1 --2- 172.21.15.12:0/3739851872 >> v2:172.21.15.12:3300/0 conn(0x7f2c780a3270 0x7f2c780a3680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:43:19.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.710+0000 7f2c7ce46640 1 -- 172.21.15.12:0/3739851872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c7813d490 msgr2=0x7f2c7813f880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:43:19.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.710+0000 7f2c7ce46640 1 --2- 172.21.15.12:0/3739851872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c7813d490 0x7f2c7813f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:43:19.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.712+0000 7f2c777fe640 1 --2- 172.21.15.12:0/3739851872 >> v2:172.21.15.12:3300/0 conn(0x7f2c780a3270 0x7f2c780a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:19.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.712+0000 7f2c777fe640 10 monclient(hunting): get_auth_request con 0x7f2c780a3270 auth_method 0 2024-09-11T01:43:19.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.712+0000 7f2c777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:19.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.712+0000 7f2c777fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:19.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.712+0000 7f2c777fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:19.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.712+0000 7f2c777fe640 1 -- 172.21.15.12:0/3739851872 >> v2:172.21.15.12:3300/0 conn(0x7f2c780a3270 msgr2=0x7f2c780a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:43:19.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.712+0000 7f2c777fe640 1 -- 172.21.15.12:0/3739851872 >> v2:172.21.15.12:3300/0 conn(0x7f2c780a3270 msgr2=0x7f2c780a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:43:19.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.712+0000 7f2c777fe640 1 --2- 172.21.15.12:0/3739851872 >> v2:172.21.15.12:3300/0 conn(0x7f2c780a3270 0x7f2c780a3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:19.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:19.712+0000 7f2c777fe640 1 --2- 172.21.15.12:0/3739851872 >> v2:172.21.15.12:3300/0 conn(0x7f2c780a3270 0x7f2c780a3680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:43:20.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:20.511+0000 7f2c7ce46640 1 -- 172.21.15.12:0/3739851872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c7813d490 msgr2=0x7f2c7813f880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:43:20.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:20.511+0000 7f2c7ce46640 1 --2- 172.21.15.12:0/3739851872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c7813d490 0x7f2c7813f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:43:20.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:20.513+0000 7f2c777fe640 1 --2- 172.21.15.12:0/3739851872 >> v2:172.21.15.12:3300/0 conn(0x7f2c780a3270 0x7f2c780a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:20.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:20.513+0000 7f2c777fe640 10 monclient(hunting): get_auth_request con 0x7f2c780a3270 auth_method 0 2024-09-11T01:43:20.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:20.513+0000 7f2c777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:20.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:20.513+0000 7f2c777fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:20.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:20.513+0000 7f2c777fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:20.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:20.513+0000 7f2c777fe640 1 -- 172.21.15.12:0/3739851872 >> v2:172.21.15.12:3300/0 conn(0x7f2c780a3270 msgr2=0x7f2c780a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:43:20.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:20.513+0000 7f2c777fe640 1 -- 172.21.15.12:0/3739851872 >> v2:172.21.15.12:3300/0 conn(0x7f2c780a3270 msgr2=0x7f2c780a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:43:20.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:20.513+0000 7f2c777fe640 1 --2- 172.21.15.12:0/3739851872 >> v2:172.21.15.12:3300/0 conn(0x7f2c780a3270 0x7f2c780a3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:20.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:20.513+0000 7f2c777fe640 1 --2- 172.21.15.12:0/3739851872 >> v2:172.21.15.12:3300/0 conn(0x7f2c780a3270 0x7f2c780a3680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:43:22.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.108+0000 7f2c75ffb640 10 monclient(hunting): tick 2024-09-11T01:43:22.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.108+0000 7f2c75ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:43:22.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.108+0000 7f2c75ffb640 1 monclient(hunting): continuing hunt 2024-09-11T01:43:22.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.108+0000 7f2c75ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:43:22.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.108+0000 7f2c75ffb640 1 -- 172.21.15.12:0/3739851872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c7813d490 msgr2=0x7f2c7813f880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:43:22.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.108+0000 7f2c75ffb640 1 --2- 172.21.15.12:0/3739851872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c7813d490 0x7f2c7813f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:22.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.108+0000 7f2c75ffb640 1 -- 172.21.15.12:0/3739851872 >> v2:172.21.15.12:3300/0 conn(0x7f2c780a3270 msgr2=0x7f2c780a3680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:43:22.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.108+0000 7f2c75ffb640 1 --2- 172.21.15.12:0/3739851872 >> v2:172.21.15.12:3300/0 conn(0x7f2c780a3270 0x7f2c780a3680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:22.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.108+0000 7f2c75ffb640 1 -- 172.21.15.12:0/3739851872 >> v1:172.21.15.71:6789/0 conn(0x7f2c780a2e90 legacy=0x7f2c7806dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:22.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.108+0000 7f2c75ffb640 10 monclient: _add_conns ranks=[0,4] 2024-09-11T01:43:22.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.108+0000 7f2c75ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2c64003760 addr v1:172.21.15.12:6789/0 2024-09-11T01:43:22.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.108+0000 7f2c75ffb640 1 --2- 172.21.15.12:0/3739851872 >> v2:172.21.15.71:3300/0 conn(0x7f2c640074b0 0x7f2c64009970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:22.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.108+0000 7f2c75ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f2c640074b0 addr v2:172.21.15.71:3300/0 2024-09-11T01:43:22.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.108+0000 7f2c75ffb640 1 -- 172.21.15.12:0/3739851872 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c7806eb70 con 0x7f2c64003760 2024-09-11T01:43:22.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.108+0000 7f2c75ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:22.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.108+0000 7f2c75ffb640 1 -- 172.21.15.12:0/3739851872 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f2c780a4900 con 0x7f2c640074b0 2024-09-11T01:43:22.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.108+0000 7f2c75ffb640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:22.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.108+0000 7f2c777fe640 1 --2- 172.21.15.12:0/3739851872 >> v2:172.21.15.71:3300/0 conn(0x7f2c640074b0 0x7f2c64009970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:22.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.108+0000 7f2c777fe640 10 monclient(hunting): get_auth_request con 0x7f2c640074b0 auth_method 0 2024-09-11T01:43:22.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.108+0000 7f2c777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:22.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.108+0000 7f2c777fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:22.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.108+0000 7f2c777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:22.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.108+0000 7f2c777fe640 1 -- 172.21.15.12:0/3739851872 >> v2:172.21.15.71:3300/0 conn(0x7f2c640074b0 msgr2=0x7f2c64009970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:43:22.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.108+0000 7f2c777fe640 1 -- 172.21.15.12:0/3739851872 >> v2:172.21.15.71:3300/0 conn(0x7f2c640074b0 msgr2=0x7f2c64009970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:43:22.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.108+0000 7f2c777fe640 1 --2- 172.21.15.12:0/3739851872 >> v2:172.21.15.71:3300/0 conn(0x7f2c640074b0 0x7f2c64009970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:22.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.108+0000 7f2c777fe640 1 --2- 172.21.15.12:0/3739851872 >> v2:172.21.15.71:3300/0 conn(0x7f2c640074b0 0x7f2c64009970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:22.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.110+0000 7f2c77fff640 1 -- 172.21.15.12:0/3739851872 >> v1:172.21.15.12:6789/0 conn(0x7f2c64003760 legacy=0x7f2c64005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.309+0000 7f2c777fe640 1 --2- 172.21.15.12:0/3739851872 >> v2:172.21.15.71:3300/0 conn(0x7f2c640074b0 0x7f2c64009970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.309+0000 7f2c777fe640 10 monclient(hunting): get_auth_request con 0x7f2c640074b0 auth_method 0 2024-09-11T01:43:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.309+0000 7f2c777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.309+0000 7f2c777fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.309+0000 7f2c777fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.309+0000 7f2c777fe640 1 -- 172.21.15.12:0/3739851872 >> v2:172.21.15.71:3300/0 conn(0x7f2c640074b0 msgr2=0x7f2c64009970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:43:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.309+0000 7f2c777fe640 1 -- 172.21.15.12:0/3739851872 >> v2:172.21.15.71:3300/0 conn(0x7f2c640074b0 msgr2=0x7f2c64009970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:43:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.309+0000 7f2c777fe640 1 --2- 172.21.15.12:0/3739851872 >> v2:172.21.15.71:3300/0 conn(0x7f2c640074b0 0x7f2c64009970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.309+0000 7f2c777fe640 1 --2- 172.21.15.12:0/3739851872 >> v2:172.21.15.71:3300/0 conn(0x7f2c640074b0 0x7f2c64009970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:43:22.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.710+0000 7f2c777fe640 1 --2- 172.21.15.12:0/3739851872 >> v2:172.21.15.71:3300/0 conn(0x7f2c640074b0 0x7f2c64009970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:22.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.710+0000 7f2c777fe640 10 monclient(hunting): get_auth_request con 0x7f2c640074b0 auth_method 0 2024-09-11T01:43:22.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.710+0000 7f2c777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:22.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.710+0000 7f2c777fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:22.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.710+0000 7f2c777fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:22.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.710+0000 7f2c777fe640 1 -- 172.21.15.12:0/3739851872 >> v2:172.21.15.71:3300/0 conn(0x7f2c640074b0 msgr2=0x7f2c64009970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:43:22.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.710+0000 7f2c777fe640 1 -- 172.21.15.12:0/3739851872 >> v2:172.21.15.71:3300/0 conn(0x7f2c640074b0 msgr2=0x7f2c64009970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:43:22.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.710+0000 7f2c777fe640 1 --2- 172.21.15.12:0/3739851872 >> v2:172.21.15.71:3300/0 conn(0x7f2c640074b0 0x7f2c64009970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:22.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:22.710+0000 7f2c777fe640 1 --2- 172.21.15.12:0/3739851872 >> v2:172.21.15.71:3300/0 conn(0x7f2c640074b0 0x7f2c64009970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:43:23.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.091+0000 7f2c76ffd640 1 -- 172.21.15.12:0/3739851872 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3403375920 0 0) 0x7f2c7806eb70 con 0x7f2c64003760 2024-09-11T01:43:23.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.091+0000 7f2c76ffd640 10 monclient(hunting): _negotiate 2024-09-11T01:43:23.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.091+0000 7f2c76ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:23.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.091+0000 7f2c76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:23.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.091+0000 7f2c76ffd640 10 monclient(hunting): my global_id is 94728 2024-09-11T01:43:23.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.091+0000 7f2c76ffd640 1 -- 172.21.15.12:0/3739851872 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c60003520 con 0x7f2c64003760 2024-09-11T01:43:23.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.092+0000 7f2c76ffd640 1 -- 172.21.15.12:0/3739851872 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 441131882 0 0) 0x7f2c60003520 con 0x7f2c64003760 2024-09-11T01:43:23.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.092+0000 7f2c76ffd640 1 -- 172.21.15.12:0/3739851872 >> v2:172.21.15.71:3300/0 conn(0x7f2c640074b0 msgr2=0x7f2c64009970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:43:23.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.092+0000 7f2c76ffd640 1 --2- 172.21.15.12:0/3739851872 >> v2:172.21.15.71:3300/0 conn(0x7f2c640074b0 0x7f2c64009970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.092+0000 7f2c76ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:43:23.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.092+0000 7f2c76ffd640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:43:23.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.092+0000 7f2c76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:23.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.092+0000 7f2c76ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:43:23.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.092+0000 7f2c76ffd640 1 -- 172.21.15.12:0/3739851872 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6400b0a0 con 0x7f2c64003760 2024-09-11T01:43:23.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.092+0000 7f2c76ffd640 10 monclient: _finish_auth 0 2024-09-11T01:43:23.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.092+0000 7f2c76ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:43:23.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.092+0000 7f2c76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.092+0000 7f2c76ffd640 1 -- 172.21.15.12:0/3739851872 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f2c68001a00 con 0x7f2c64003760 2024-09-11T01:43:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.092+0000 7f2c76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.092+0000 7f2c76ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:43:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.092+0000 7f2c76ffd640 10 monclient: dump: 2024-09-11T01:43:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:23.097 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:23.098 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:23.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.092+0000 7f2c76ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:43:23.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.092+0000 7f2c76ffd640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:23.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.092+0000 7f2c76ffd640 1 -- 172.21.15.12:0/3739851872 >> v1:172.21.15.12:6789/0 conn(0x7f2c64003760 legacy=0x7f2c64005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:23.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.093+0000 7f2c7ce46640 1 -- 172.21.15.12:0/3739851872 reap_dead start 2024-09-11T01:43:23.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.093+0000 7f2c76ffd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:43:23.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.093+0000 7f2c76ffd640 1 --2- 172.21.15.12:0/3739851872 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c600039b0 0x7f2c60003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:23.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.093+0000 7f2c76ffd640 10 monclient(hunting): picked mon.a con 0x7f2c600039b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:23.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.093+0000 7f2c76ffd640 1 --2- 172.21.15.12:0/3739851872 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c60004800 0x7f2c60004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:23.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.093+0000 7f2c76ffd640 10 monclient(hunting): picked mon.b con 0x7f2c60004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:23.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.093+0000 7f2c76ffd640 1 --2- 172.21.15.12:0/3739851872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c60005290 0x7f2c6000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:23.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.093+0000 7f2c7ce46640 1 -- 172.21.15.12:0/3739851872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c60005290 msgr2=0x7f2c6000b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:43:23.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.093+0000 7f2c76ffd640 10 monclient(hunting): picked mon.c con 0x7f2c60005290 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:23.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.093+0000 7f2c76ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:23.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.093+0000 7f2c76ffd640 1 -- 172.21.15.12:0/3739851872 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2c780a4900 con 0x7f2c600039b0 2024-09-11T01:43:23.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.093+0000 7f2c76ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:23.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.093+0000 7f2c76ffd640 1 -- 172.21.15.12:0/3739851872 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6000c1e0 con 0x7f2c60004800 2024-09-11T01:43:23.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.093+0000 7f2c76ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:23.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.093+0000 7f2c76ffd640 1 -- 172.21.15.12:0/3739851872 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2c6000c3e0 con 0x7f2c60005290 2024-09-11T01:43:23.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.093+0000 7f2c76ffd640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:23.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.093+0000 7f2c76ffd640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:43:23.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.093+0000 7f2c7ce46640 1 --2- 172.21.15.12:0/3739851872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c60005290 0x7f2c6000b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:23.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.094+0000 7f2c777fe640 1 --2- 172.21.15.12:0/3739851872 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c60004800 0x7f2c60004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:23.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.094+0000 7f2c777fe640 10 monclient(hunting): get_auth_request con 0x7f2c60004800 auth_method 0 2024-09-11T01:43:23.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.094+0000 7f2c777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:23.100 INFO:tasks.rados.rados.0.smithi071.stdout:2124: finishing rollback tid 1 to smithi07134777-34 2024-09-11T01:43:23.100 INFO:tasks.rados.rados.0.smithi071.stdout:2124: finishing rollback tid 2 to smithi07134777-34 2024-09-11T01:43:23.100 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 1168 (ObjNum 566 snap 160 seq_num 566) dirty exists 2024-09-11T01:43:23.100 INFO:tasks.rados.rados.0.smithi071.stdout:2125: finishing write tid 1 to smithi07134777-7 2024-09-11T01:43:23.100 INFO:tasks.rados.rados.0.smithi071.stdout:2125: finishing write tid 2 to smithi07134777-7 2024-09-11T01:43:23.100 INFO:tasks.rados.rados.0.smithi071.stdout:2125: finishing write tid 3 to smithi07134777-7 2024-09-11T01:43:23.100 INFO:tasks.rados.rados.0.smithi071.stdout:2125: finishing write tid 4 to smithi07134777-7 2024-09-11T01:43:23.101 INFO:tasks.rados.rados.0.smithi071.stdout:2125: finishing write tid 5 to smithi07134777-7 2024-09-11T01:43:23.101 INFO:tasks.rados.rados.0.smithi071.stdout:2125: oid 7 updating version 0 to 1240 2024-09-11T01:43:23.101 INFO:tasks.rados.rados.0.smithi071.stdout:2125: oid 7 updating version 1240 to 1242 2024-09-11T01:43:23.101 INFO:tasks.rados.rados.0.smithi071.stdout:2125: oid 7 version 1242 is already newer than 1241 2024-09-11T01:43:23.101 INFO:tasks.rados.rados.0.smithi071.stdout:2125: oid 7 updating version 1242 to 1243 2024-09-11T01:43:23.101 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 1243 (ObjNum 769 snap 233 seq_num 769) dirty exists 2024-09-11T01:43:23.101 INFO:tasks.rados.rados.0.smithi071.stdout:2125: left oid 7 (ObjNum 769 snap 233 seq_num 769) 2024-09-11T01:43:23.101 INFO:tasks.rados.rados.0.smithi071.stdout:2124: done (2 left) 2024-09-11T01:43:23.101 INFO:tasks.rados.rados.0.smithi071.stdout:2125: done (1 left) 2024-09-11T01:43:23.101 INFO:tasks.rados.rados.0.smithi071.stdout:2126: done (0 left) 2024-09-11T01:43:23.101 INFO:tasks.rados.rados.0.smithi071.stdout:2127: snap_remove snap 220 2024-09-11T01:43:23.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.094+0000 7f2c777fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:23.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.094+0000 7f2c777fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:23.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.094+0000 7f2c777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:23.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.094+0000 7f2c777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:23.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.094+0000 7f2c777fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:43:23.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.095+0000 7f2c777fe640 10 monclient(hunting): handle_auth_done global_id 94728 payload 293 2024-09-11T01:43:23.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.095+0000 7f2c777fe640 1 -- 172.21.15.12:0/3739851872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c60005290 msgr2=0x7f2c6000b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:43:23.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.095+0000 7f2c777fe640 1 --2- 172.21.15.12:0/3739851872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c60005290 0x7f2c6000b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.096+0000 7f2c777fe640 1 -- 172.21.15.12:0/3739851872 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c600039b0 msgr2=0x7f2c60003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:23.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.096+0000 7f2c777fe640 1 --2- 172.21.15.12:0/3739851872 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c600039b0 0x7f2c60003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.096+0000 7f2c777fe640 10 monclient: _finish_hunting 0 2024-09-11T01:43:23.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.096+0000 7f2c777fe640 1 monclient: found mon.b 2024-09-11T01:43:23.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.096+0000 7f2c777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:23.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.096+0000 7f2c777fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:43:23.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.096+0000 7f2c777fe640 1 -- 172.21.15.12:0/3739851872 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c60003520 con 0x7f2c60004800 2024-09-11T01:43:23.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.096+0000 7f2c777fe640 10 monclient: _finish_auth 0 2024-09-11T01:43:23.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.096+0000 7f2c777fe640 10 monclient: _check_auth_tickets 2024-09-11T01:43:23.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.096+0000 7f2c777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:23.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.096+0000 7f2c7e8a9640 5 monclient: authenticate success, global_id 94728 2024-09-11T01:43:23.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.096+0000 7f2c7e8a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:43:23.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.099+0000 7f2c777fe640 1 -- 172.21.15.12:0/3739851872 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c60004800 msgr2=0x7f2c60004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:23.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.099+0000 7f2c777fe640 1 --2- 172.21.15.12:0/3739851872 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c60004800 0x7f2c60004c80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c018ac0 tx=0x7f2c60005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1cbe0a990e5bfc02 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:23.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.100+0000 7f2c76ffd640 1 -- 172.21.15.12:0/3739851872 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c00f040 con 0x7f2c60004800 2024-09-11T01:43:23.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.100+0000 7f2c76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:23.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.100+0000 7f2c76ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:43:23.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.100+0000 7f2c76ffd640 10 monclient: dump: 2024-09-11T01:43:23.115 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:23.115 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:23.115 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.115 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.115 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:23.115 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:23.115 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:23.115 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:23.115 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:23.115 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:23.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.100+0000 7f2c76ffd640 1 -- 172.21.15.12:0/3739851872 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c6c02d6c0 con 0x7f2c60004800 2024-09-11T01:43:23.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.100+0000 7f2c7e8a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:43:23.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.100+0000 7f2c76ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:23.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.100+0000 7f2c76ffd640 1 -- 172.21.15.12:0/3739851872 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c02dab0 con 0x7f2c60004800 2024-09-11T01:43:23.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.100+0000 7f2c76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:23.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.100+0000 7f2c76ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:43:23.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.100+0000 7f2c76ffd640 10 monclient: dump: 2024-09-11T01:43:23.116 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:23.116 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:23.116 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.116 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.116 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:23.116 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:23.116 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:23.117 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:23.117 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:23.117 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:23.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.100+0000 7f2c7e8a9640 10 monclient: get_monmap_and_config success 2024-09-11T01:43:23.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.100+0000 7f2c7e8a9640 4 set_mon_vals no callback set 2024-09-11T01:43:23.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.101+0000 7f2c7e8a9640 10 monclient: shutdown 2024-09-11T01:43:23.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.101+0000 7f2c7e8a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:23.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.101+0000 7f2c7e8a9640 1 -- 172.21.15.12:0/3739851872 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c60004800 msgr2=0x7f2c60004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:23.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.101+0000 7f2c7e8a9640 1 --2- 172.21.15.12:0/3739851872 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c60004800 0x7f2c60004c80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c018ac0 tx=0x7f2c60005830 comp rx=0 tx=0).stop 2024-09-11T01:43:23.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.101+0000 7f2c7e8a9640 1 -- 172.21.15.12:0/3739851872 shutdown_connections 2024-09-11T01:43:23.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.102+0000 7f2c7e8a9640 1 --2- 172.21.15.12:0/3739851872 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c60005290 0x7f2c6000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.102+0000 7f2c7e8a9640 1 --2- 172.21.15.12:0/3739851872 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c60004800 0x7f2c60004c80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.102+0000 7f2c7e8a9640 1 --2- 172.21.15.12:0/3739851872 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c600039b0 0x7f2c60003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.102+0000 7f2c7e8a9640 1 -- 172.21.15.12:0/3739851872 >> 172.21.15.12:0/3739851872 conn(0x7f2c780a8490 msgr2=0x7f2c780a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:23.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.102+0000 7f2c7e8a9640 1 -- 172.21.15.12:0/3739851872 shutdown_connections 2024-09-11T01:43:23.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.102+0000 7f2c7e8a9640 1 -- 172.21.15.12:0/3739851872 wait complete. 2024-09-11T01:43:23.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.103+0000 7f2c7e8a9640 10 monclient: build_initial_monmap 2024-09-11T01:43:23.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.103+0000 7f2c7e8a9640 10 monclient: monmap: 2024-09-11T01:43:23.118 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:23.118 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:43:23.118 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:43:23.118 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:43:23.118 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:23.118 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:23.118 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:43:23.118 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:43:23.119 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:43:23.119 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:23.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.103+0000 7f2c7e8a9640 1 Processor -- start 2024-09-11T01:43:23.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.104+0000 7f2c7e8a9640 1 -- start start 2024-09-11T01:43:23.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.104+0000 7f2c7e8a9640 10 monclient: init 2024-09-11T01:43:23.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c7e8a9640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:23.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c7e8a9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:43:23.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c7e8a9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c600039b0 0x7f2c781cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:23.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c7e8a9640 10 monclient(hunting): picked mon.noname-b con 0x7f2c600039b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:23.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c7e8a9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c60004800 0x7f2c781373b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:23.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c7e8a9640 10 monclient(hunting): picked mon.noname-a con 0x7f2c60004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:23.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c7e8a9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c60005290 0x7f2c781378f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:23.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c7e8a9640 10 monclient(hunting): picked mon.noname-c con 0x7f2c60005290 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:23.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c7e8a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:23.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c7e8a9640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2c7806dde0 con 0x7f2c60004800 2024-09-11T01:43:23.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c7e8a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:23.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c7e8a9640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f2c7806dc60 con 0x7f2c600039b0 2024-09-11T01:43:23.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c7e8a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:23.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c7e8a9640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2c7806df60 con 0x7f2c60005290 2024-09-11T01:43:23.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c7ce46640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c60005290 msgr2=0x7f2c781378f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:43:23.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c7ce46640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c60005290 0x7f2c781378f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:23.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c777fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c60004800 0x7f2c781373b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:23.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c777fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c60004800 0x7f2c781373b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39124/0 (socket says 172.21.15.12:39124) 2024-09-11T01:43:23.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c777fe640 1 -- 172.21.15.12:0/891250624 learned_addr learned my addr 172.21.15.12:0/891250624 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:23.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c77fff640 1 --2- 172.21.15.12:0/891250624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c600039b0 0x7f2c781cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:23.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c7e8a9640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:23.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c7e8a9640 10 monclient(hunting): authenticate will time out at 1609.547577s 2024-09-11T01:43:23.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c777fe640 10 monclient(hunting): get_auth_request con 0x7f2c60004800 auth_method 0 2024-09-11T01:43:23.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:23.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c777fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:23.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:23.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c77fff640 10 monclient(hunting): get_auth_request con 0x7f2c600039b0 auth_method 0 2024-09-11T01:43:23.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:23.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c77fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:23.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:23.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:23.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:23.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:23.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:23.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:23.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.105+0000 7f2c77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:23.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.106+0000 7f2c77fff640 10 monclient(hunting): handle_auth_done global_id 134326 payload 824 2024-09-11T01:43:23.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.106+0000 7f2c77fff640 1 -- 172.21.15.12:0/891250624 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c60005290 msgr2=0x7f2c781378f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:43:23.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.106+0000 7f2c77fff640 1 --2- 172.21.15.12:0/891250624 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c60005290 0x7f2c781378f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.106+0000 7f2c77fff640 1 -- 172.21.15.12:0/891250624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c60004800 msgr2=0x7f2c781373b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:23.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.106+0000 7f2c77fff640 1 --2- 172.21.15.12:0/891250624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c60004800 0x7f2c781373b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.106+0000 7f2c77fff640 10 monclient: _finish_hunting 0 2024-09-11T01:43:23.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.106+0000 7f2c77fff640 1 monclient: found mon.noname-b 2024-09-11T01:43:23.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.106+0000 7f2c77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:23.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.106+0000 7f2c77fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-09-11T01:43:23.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.106+0000 7f2c77fff640 1 -- 172.21.15.12:0/891250624 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c78123b80 con 0x7f2c600039b0 2024-09-11T01:43:23.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.106+0000 7f2c777fe640 1 --2- 172.21.15.12:0/891250624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c60004800 0x7f2c781373b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:43:23.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.108+0000 7f2c77fff640 1 -- 172.21.15.12:0/891250624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c600039b0 msgr2=0x7f2c781cc9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:23.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.108+0000 7f2c77fff640 1 --2- 172.21.15.12:0/891250624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c600039b0 0x7f2c781cc9a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2c780a4890 tx=0x7f2c6800cb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:23.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.108+0000 7f2c74ff9640 1 -- 172.21.15.12:0/891250624 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c68017070 con 0x7f2c600039b0 2024-09-11T01:43:23.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.108+0000 7f2c74ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:23.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.108+0000 7f2c74ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:43:23.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.108+0000 7f2c74ff9640 10 monclient: dump: 2024-09-11T01:43:23.123 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:23.123 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:23.123 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.123 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.123 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:23.123 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:23.123 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:23.123 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:23.123 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:23.124 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:23.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.108+0000 7f2c74ff9640 10 monclient: _finish_auth 0 2024-09-11T01:43:23.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.108+0000 7f2c74ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:43:23.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.108+0000 7f2c74ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:23.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.108+0000 7f2c74ff9640 1 -- 172.21.15.12:0/891250624 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c68007480 con 0x7f2c600039b0 2024-09-11T01:43:23.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.108+0000 7f2c7e8a9640 5 monclient: authenticate success, global_id 134326 2024-09-11T01:43:23.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.108+0000 7f2c7e8a9640 10 monclient: _renew_subs 2024-09-11T01:43:23.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.108+0000 7f2c7e8a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:43:23.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.108+0000 7f2c7e8a9640 1 -- 172.21.15.12:0/891250624 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c78123e60 con 0x7f2c600039b0 2024-09-11T01:43:23.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.108+0000 7f2c74ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:23.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.108+0000 7f2c74ff9640 1 -- 172.21.15.12:0/891250624 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c680059e0 con 0x7f2c600039b0 2024-09-11T01:43:23.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.108+0000 7f2c74ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:23.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.108+0000 7f2c74ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:43:23.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.108+0000 7f2c74ff9640 10 monclient: dump: 2024-09-11T01:43:23.125 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:23.125 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:23.125 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.125 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.125 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:23.125 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:23.125 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:23.125 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:23.125 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:23.125 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:23.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.108+0000 7f2c7e8a9640 10 monclient: _renew_subs 2024-09-11T01:43:23.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.108+0000 7f2c7e8a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:43:23.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.108+0000 7f2c7e8a9640 1 -- 172.21.15.12:0/891250624 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c7813be20 con 0x7f2c600039b0 2024-09-11T01:43:23.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.108+0000 7f2c767fc640 4 set_mon_vals no callback set 2024-09-11T01:43:23.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.110+0000 7f2c7e8a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:23.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.110+0000 7f2c7e8a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:23.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.110+0000 7f2c7e8a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:43:23.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.110+0000 7f2c74ff9640 1 -- 172.21.15.12:0/891250624 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f2c68005c40 con 0x7f2c600039b0 2024-09-11T01:43:23.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.110+0000 7f2c7e8a9640 1 -- 172.21.15.12:0/891250624 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c780a3680 con 0x7f2c600039b0 2024-09-11T01:43:23.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.110+0000 7f2c74ff9640 1 --2- 172.21.15.12:0/891250624 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2c6403ebd0 0x7f2c64041090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:23.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.110+0000 7f2c74ff9640 1 -- 172.21.15.12:0/891250624 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(476..476 src has 285..476) ==== 4843+0+0 (secure 0 0 0) 0x7f2c68052360 con 0x7f2c600039b0 2024-09-11T01:43:23.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.111+0000 7f2c777fe640 1 --2- 172.21.15.12:0/891250624 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2c6403ebd0 0x7f2c64041090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:23.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.111+0000 7f2c777fe640 10 monclient: get_auth_request con 0x7f2c6403ebd0 auth_method 0 2024-09-11T01:43:23.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.113+0000 7f2c777fe640 1 --2- 172.21.15.12:0/891250624 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2c6403ebd0 0x7f2c64041090 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2c7813b900 tx=0x7f2c6c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:23.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.116+0000 7f2c74ff9640 1 -- 172.21.15.12:0/891250624 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2c68050d20 con 0x7f2c600039b0 2024-09-11T01:43:23.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.116+0000 7f2c74ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:23.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.116+0000 7f2c74ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:43:23.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.376+0000 7f2c7e8a9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:43:23.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.376+0000 7f2c7e8a9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:43:23.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.376+0000 7f2c7e8a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:43:23.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.376+0000 7f2c7e8a9640 1 -- 172.21.15.12:0/891250624 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2c7813bb60 con 0x7f2c600039b0 2024-09-11T01:43:23.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.377+0000 7f2c74ff9640 1 -- 172.21.15.12:0/891250624 <== mon.1 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f2c68052080 con 0x7f2c600039b0 2024-09-11T01:43:23.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.377+0000 7f2c74ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:43:23.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.377+0000 7f2c74ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:43:23.379 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":132,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:43:23.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.379+0000 7f2c7e8a9640 1 -- 172.21.15.12:0/891250624 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2c6403ebd0 msgr2=0x7f2c64041090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:23.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.379+0000 7f2c7e8a9640 1 --2- 172.21.15.12:0/891250624 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2c6403ebd0 0x7f2c64041090 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2c7813b900 tx=0x7f2c6c030000 comp rx=0 tx=0).stop 2024-09-11T01:43:23.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.379+0000 7f2c7e8a9640 10 monclient: shutdown 2024-09-11T01:43:23.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.379+0000 7f2c7e8a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:23.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.379+0000 7f2c7e8a9640 1 -- 172.21.15.12:0/891250624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c600039b0 msgr2=0x7f2c781cc9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:23.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.379+0000 7f2c7e8a9640 1 --2- 172.21.15.12:0/891250624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c600039b0 0x7f2c781cc9a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2c780a4890 tx=0x7f2c6800cb90 comp rx=0 tx=0).stop 2024-09-11T01:43:23.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.379+0000 7f2c7e8a9640 1 -- 172.21.15.12:0/891250624 shutdown_connections 2024-09-11T01:43:23.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.379+0000 7f2c7e8a9640 1 --2- 172.21.15.12:0/891250624 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c60005290 0x7f2c781378f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.379+0000 7f2c7e8a9640 1 --2- 172.21.15.12:0/891250624 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2c6403ebd0 0x7f2c64041090 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.379+0000 7f2c7e8a9640 1 --2- 172.21.15.12:0/891250624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c60004800 0x7f2c781373b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.379+0000 7f2c7e8a9640 1 --2- 172.21.15.12:0/891250624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c600039b0 0x7f2c781cc9a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.379+0000 7f2c7e8a9640 1 -- 172.21.15.12:0/891250624 >> 172.21.15.12:0/891250624 conn(0x7f2c780a8490 msgr2=0x7f2c7813c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:23.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.379+0000 7f2c7e8a9640 1 -- 172.21.15.12:0/891250624 shutdown_connections 2024-09-11T01:43:23.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.379+0000 7f2c7e8a9640 1 -- 172.21.15.12:0/891250624 wait complete. 2024-09-11T01:43:23.391 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:43:23.392 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-11T01:43:23.392 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:43:23.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.492+0000 7fcf55f52640 10 monclient: get_monmap_and_config 2024-09-11T01:43:23.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.492+0000 7fcf55f52640 10 monclient: build_initial_monmap 2024-09-11T01:43:23.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.492+0000 7fcf55f52640 10 monclient: monmap: 2024-09-11T01:43:23.494 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:23.494 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:23.494 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:43:23.493303+0000 2024-09-11T01:43:23.494 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:43:23.493303+0000 2024-09-11T01:43:23.494 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:23.494 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:23.494 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:43:23.494 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:43:23.495 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:43:23.495 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:43:23.495 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:43:23.495 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:23.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.493+0000 7fcf55f52640 1 Processor -- start 2024-09-11T01:43:23.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.493+0000 7fcf55f52640 1 -- start start 2024-09-11T01:43:23.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.493+0000 7fcf55f52640 10 monclient: init 2024-09-11T01:43:23.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.493+0000 7fcf55f52640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:23.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.493+0000 7fcf55f52640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-11T01:43:23.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.493+0000 7fcf55f52640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcf500a1380 addr v1:172.21.15.71:6789/0 2024-09-11T01:43:23.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.493+0000 7fcf55f52640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcf500a0370 0x7fcf5013ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:23.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.493+0000 7fcf55f52640 10 monclient(hunting): picked mon.noname-c con 0x7fcf500a0370 addr v2:172.21.15.71:3300/0 2024-09-11T01:43:23.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.493+0000 7fcf55f52640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcf500a0c90 0x7fcf5013d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:23.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.493+0000 7fcf55f52640 10 monclient(hunting): picked mon.noname-b con 0x7fcf500a0c90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:23.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf55f52640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf500a21c0 con 0x7fcf500a1380 2024-09-11T01:43:23.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf55f52640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:23.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf55f52640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fcf500a24a0 con 0x7fcf500a0370 2024-09-11T01:43:23.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf55f52640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:23.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf55f52640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcf500a2780 con 0x7fcf500a0c90 2024-09-11T01:43:23.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf55f52640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:23.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf55f52640 10 monclient(hunting): authenticate will time out at 1609.936038s 2024-09-11T01:43:23.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf4ffff640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcf500a0c90 msgr2=0x7fcf5013d350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:43:23.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf4ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcf500a0c90 0x7fcf5013d350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:23.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf4effd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcf500a0370 0x7fcf5013ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:23.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf4effd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcf500a0370 0x7fcf5013ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:42350/0 (socket says 172.21.15.12:42350) 2024-09-11T01:43:23.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf4effd640 1 -- 172.21.15.12:0/1271524440 learned_addr learned my addr 172.21.15.12:0/1271524440 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:23.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf4f7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fcf500a1380 0x7fcf500a1760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.12:40652/0 (socket says 172.21.15.12:40652) 2024-09-11T01:43:23.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf4effd640 10 monclient(hunting): get_auth_request con 0x7fcf500a0370 auth_method 0 2024-09-11T01:43:23.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf4effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:23.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf4effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:23.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:23.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf4effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:23.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf4effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:23.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf4effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:23.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf4effd640 10 monclient(hunting): handle_auth_done global_id 134329 payload 274 2024-09-11T01:43:23.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf4effd640 1 -- 172.21.15.12:0/1271524440 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcf500a0c90 msgr2=0x7fcf5013d350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:43:23.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf4effd640 1 --2- 172.21.15.12:0/1271524440 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcf500a0c90 0x7fcf5013d350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf4effd640 1 -- 172.21.15.12:0/1271524440 >> v1:172.21.15.71:6789/0 conn(0x7fcf500a1380 legacy=0x7fcf500a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:23.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf4effd640 10 monclient: _finish_hunting 0 2024-09-11T01:43:23.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf4effd640 1 monclient: found mon.noname-c 2024-09-11T01:43:23.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:23.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf4effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:43:23.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.494+0000 7fcf4effd640 1 -- 172.21.15.12:0/1271524440 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf5009fb60 con 0x7fcf500a0370 2024-09-11T01:43:23.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.497+0000 7fcf4effd640 1 -- 172.21.15.12:0/1271524440 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf500a0370 msgr2=0x7fcf5013ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:23.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.497+0000 7fcf4effd640 1 --2- 172.21.15.12:0/1271524440 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf500a0370 0x7fcf5013ae00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcf44009870 tx=0x7fcf4402a0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab1e336deb0ee483 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:23.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.497+0000 7fcf4e7fc640 1 -- 172.21.15.12:0/1271524440 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf44037070 con 0x7fcf500a0370 2024-09-11T01:43:23.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.497+0000 7fcf4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:23.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.497+0000 7fcf4e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:43:23.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.497+0000 7fcf4e7fc640 10 monclient: dump: 2024-09-11T01:43:23.504 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:23.504 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:23.504 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.505 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.505 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:23.505 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:23.505 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:23.505 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:23.505 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:23.505 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:23.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.498+0000 7fcf4e7fc640 10 monclient: _finish_auth 0 2024-09-11T01:43:23.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.498+0000 7fcf4e7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:43:23.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.498+0000 7fcf4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:23.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.498+0000 7fcf55f52640 5 monclient: authenticate success, global_id 134329 2024-09-11T01:43:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.498+0000 7fcf55f52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:43:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.498+0000 7fcf4e7fc640 1 -- 172.21.15.12:0/1271524440 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf44032640 con 0x7fcf500a0370 2024-09-11T01:43:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.498+0000 7fcf4e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.498+0000 7fcf55f52640 10 monclient: get_monmap_and_config success 2024-09-11T01:43:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.498+0000 7fcf55f52640 4 set_mon_vals no callback set 2024-09-11T01:43:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.498+0000 7fcf4e7fc640 1 -- 172.21.15.12:0/1271524440 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf44032a10 con 0x7fcf500a0370 2024-09-11T01:43:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.498+0000 7fcf4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.498+0000 7fcf4e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:43:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.498+0000 7fcf4e7fc640 10 monclient: dump: 2024-09-11T01:43:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:23.507 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.498+0000 7fcf55f52640 10 monclient: shutdown 2024-09-11T01:43:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.498+0000 7fcf55f52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.498+0000 7fcf55f52640 1 -- 172.21.15.12:0/1271524440 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf500a0370 msgr2=0x7fcf5013ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.498+0000 7fcf55f52640 1 --2- 172.21.15.12:0/1271524440 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf500a0370 0x7fcf5013ae00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcf44009870 tx=0x7fcf4402a0d0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.498+0000 7fcf55f52640 1 -- 172.21.15.12:0/1271524440 shutdown_connections 2024-09-11T01:43:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.498+0000 7fcf55f52640 1 --2- 172.21.15.12:0/1271524440 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcf500a0c90 0x7fcf5013d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.498+0000 7fcf55f52640 1 --2- 172.21.15.12:0/1271524440 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf500a0370 0x7fcf5013ae00 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.498+0000 7fcf55f52640 1 -- 172.21.15.12:0/1271524440 >> 172.21.15.12:0/1271524440 conn(0x7fcf500a2f70 msgr2=0x7fcf500a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.498+0000 7fcf55f52640 1 -- 172.21.15.12:0/1271524440 shutdown_connections 2024-09-11T01:43:23.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.498+0000 7fcf55f52640 1 -- 172.21.15.12:0/1271524440 wait complete. 2024-09-11T01:43:23.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.499+0000 7fcf55f52640 10 monclient: build_initial_monmap 2024-09-11T01:43:23.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.499+0000 7fcf55f52640 10 monclient: monmap: 2024-09-11T01:43:23.508 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:23.508 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:43:23.508 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:43:23.508 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:43:23.508 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:23.508 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:23.508 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:43:23.508 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:43:23.508 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:43:23.508 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:23.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.499+0000 7fcf55f52640 1 Processor -- start 2024-09-11T01:43:23.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.499+0000 7fcf55f52640 1 -- start start 2024-09-11T01:43:23.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.499+0000 7fcf55f52640 10 monclient: init 2024-09-11T01:43:23.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf55f52640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:23.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf55f52640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:43:23.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf55f52640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcf500a0370 0x7fcf501cd060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:23.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf55f52640 10 monclient(hunting): picked mon.noname-c con 0x7fcf500a0370 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:23.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf55f52640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf500a0c90 0x7fcf5013cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:23.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf55f52640 10 monclient(hunting): picked mon.noname-b con 0x7fcf500a0c90 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:23.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf55f52640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf500a1380 0x7fcf5013d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:23.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf55f52640 10 monclient(hunting): picked mon.noname-a con 0x7fcf500a1380 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:23.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf55f52640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:23.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf55f52640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5013da10 con 0x7fcf500a1380 2024-09-11T01:43:23.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf55f52640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:23.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf55f52640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5013d890 con 0x7fcf500a0c90 2024-09-11T01:43:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf55f52640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf55f52640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcf5013db90 con 0x7fcf500a0370 2024-09-11T01:43:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf55f52640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf55f52640 10 monclient(hunting): authenticate will time out at 1609.942080s 2024-09-11T01:43:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4f7fe640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcf500a0370 msgr2=0x7fcf501cd060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:43:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4f7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcf500a0370 0x7fcf501cd060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf500a1380 0x7fcf5013d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf500a1380 0x7fcf5013d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39130/0 (socket says 172.21.15.12:39130) 2024-09-11T01:43:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4ffff640 1 -- 172.21.15.12:0/2882331096 learned_addr learned my addr 172.21.15.12:0/2882331096 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4ffff640 10 monclient(hunting): get_auth_request con 0x7fcf500a1380 auth_method 0 2024-09-11T01:43:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:23.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:23.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:23.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4effd640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf500a0c90 0x7fcf5013cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:23.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4effd640 10 monclient(hunting): get_auth_request con 0x7fcf500a0c90 auth_method 0 2024-09-11T01:43:23.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:23.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:23.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:23.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4ffff640 10 monclient(hunting): handle_auth_done global_id 94734 payload 824 2024-09-11T01:43:23.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4ffff640 1 -- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcf500a0370 msgr2=0x7fcf501cd060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:43:23.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4ffff640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcf500a0370 0x7fcf501cd060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4ffff640 1 -- 172.21.15.12:0/2882331096 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf500a0c90 msgr2=0x7fcf5013cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:23.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4ffff640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf500a0c90 0x7fcf5013cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:43:23.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4ffff640 1 monclient: found mon.noname-a 2024-09-11T01:43:23.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:23.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:23.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4ffff640 1 -- 172.21.15.12:0/2882331096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf50073570 con 0x7fcf500a1380 2024-09-11T01:43:23.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4effd640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf500a0c90 0x7fcf5013cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:43:23.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4ffff640 1 -- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf500a1380 msgr2=0x7fcf5013d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:43:23.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4ffff640 1 -- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf500a1380 msgr2=0x7fcf5013d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:43:23.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4ffff640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf500a1380 0x7fcf5013d240 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7fcf4800be30 tx=0x7fcf4800bf20 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:23.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.500+0000 7fcf4ffff640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf500a1380 0x7fcf5013d240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:23.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4ffff640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf500a1380 0x7fcf5013d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:23.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4ffff640 10 monclient: get_auth_request con 0x7fcf500a1380 auth_method 0 2024-09-11T01:43:23.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4ffff640 0 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf500a1380 0x7fcf5013d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-11T01:43:23.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4ffff640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf500a1380 0x7fcf5013d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf37fff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:23.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf37fff640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:23.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf37fff640 1 -- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf500a1380 msgr2=0x7fcf5013d240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:43:23.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf37fff640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf500a1380 0x7fcf5013d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf37fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:43:23.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf37fff640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf3c0034d0 0x7fcf3c005920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:23.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf37fff640 10 monclient(hunting): picked mon.noname-a con 0x7fcf3c0034d0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:23.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf37fff640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcf500a0370 0x7fcf3c0084b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:23.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf37fff640 10 monclient(hunting): picked mon.noname-c con 0x7fcf500a0370 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:23.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf37fff640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf500a0c90 0x7fcf3c00af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:23.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf37fff640 10 monclient(hunting): picked mon.noname-b con 0x7fcf500a0c90 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:23.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf37fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:23.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf37fff640 1 -- 172.21.15.12:0/2882331096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcf3c00b9a0 con 0x7fcf3c0034d0 2024-09-11T01:43:23.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf37fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:23.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf37fff640 1 -- 172.21.15.12:0/2882331096 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fcf3c00bb20 con 0x7fcf500a0c90 2024-09-11T01:43:23.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf37fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:23.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf37fff640 1 -- 172.21.15.12:0/2882331096 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcf3c00bd20 con 0x7fcf500a0370 2024-09-11T01:43:23.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf37fff640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:23.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4effd640 1 -- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcf500a0370 msgr2=0x7fcf3c0084b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:43:23.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4effd640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcf500a0370 0x7fcf3c0084b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:23.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4f7fe640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf3c0034d0 0x7fcf3c005920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:23.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4f7fe640 10 monclient(hunting): get_auth_request con 0x7fcf3c0034d0 auth_method 0 2024-09-11T01:43:23.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:23.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:23.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:23.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:23.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:23.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:43:23.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4ffff640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf500a0c90 0x7fcf3c00af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:23.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4ffff640 10 monclient(hunting): get_auth_request con 0x7fcf500a0c90 auth_method 0 2024-09-11T01:43:23.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:23.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:23.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:23.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4f7fe640 10 monclient(hunting): handle_auth_done global_id 94734 payload 293 2024-09-11T01:43:23.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4f7fe640 1 -- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcf500a0370 msgr2=0x7fcf3c0084b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:43:23.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4f7fe640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcf500a0370 0x7fcf3c0084b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4f7fe640 1 -- 172.21.15.12:0/2882331096 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf500a0c90 msgr2=0x7fcf3c00af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:23.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4f7fe640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf500a0c90 0x7fcf3c00af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4f7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:43:23.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4f7fe640 1 monclient: found mon.noname-a 2024-09-11T01:43:23.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:23.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:23.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.701+0000 7fcf4f7fe640 1 -- 172.21.15.12:0/2882331096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf3c00bf80 con 0x7fcf3c0034d0 2024-09-11T01:43:23.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.703+0000 7fcf4f7fe640 1 -- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf3c0034d0 msgr2=0x7fcf3c005920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:23.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.703+0000 7fcf4f7fe640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf3c0034d0 0x7fcf3c005920 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fcf40005020 tx=0x7fcf48009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:23.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.704+0000 7fcf37fff640 1 -- 172.21.15.12:0/2882331096 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf40004c20 con 0x7fcf3c0034d0 2024-09-11T01:43:23.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.704+0000 7fcf37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:23.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.704+0000 7fcf37fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:23.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.704+0000 7fcf37fff640 10 monclient: dump: 2024-09-11T01:43:23.712 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:23.712 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:23.712 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.712 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.712 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:23.712 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:23.712 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:23.713 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:23.713 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:23.713 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:23.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.704+0000 7fcf37fff640 10 monclient: _finish_auth 0 2024-09-11T01:43:23.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.704+0000 7fcf37fff640 10 monclient: _check_auth_tickets 2024-09-11T01:43:23.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.704+0000 7fcf37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:23.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.704+0000 7fcf55f52640 5 monclient: authenticate success, global_id 94734 2024-09-11T01:43:23.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.704+0000 7fcf37fff640 1 -- 172.21.15.12:0/2882331096 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf400074e0 con 0x7fcf3c0034d0 2024-09-11T01:43:23.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.704+0000 7fcf37fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:23.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.704+0000 7fcf37fff640 1 -- 172.21.15.12:0/2882331096 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf40007690 con 0x7fcf3c0034d0 2024-09-11T01:43:23.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.704+0000 7fcf55f52640 10 monclient: _renew_subs 2024-09-11T01:43:23.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.704+0000 7fcf55f52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:23.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.704+0000 7fcf55f52640 1 -- 172.21.15.12:0/2882331096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf5009f280 con 0x7fcf3c0034d0 2024-09-11T01:43:23.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.704+0000 7fcf37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:23.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.704+0000 7fcf37fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:23.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.704+0000 7fcf37fff640 10 monclient: dump: 2024-09-11T01:43:23.714 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:23.715 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.704+0000 7fcf4dffb640 4 set_mon_vals no callback set 2024-09-11T01:43:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.704+0000 7fcf55f52640 10 monclient: _renew_subs 2024-09-11T01:43:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.704+0000 7fcf55f52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.704+0000 7fcf55f52640 1 -- 172.21.15.12:0/2882331096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf50128e30 con 0x7fcf3c0034d0 2024-09-11T01:43:23.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.705+0000 7fcf37fff640 1 -- 172.21.15.12:0/2882331096 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fcf40005900 con 0x7fcf3c0034d0 2024-09-11T01:43:23.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.705+0000 7fcf55f52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:23.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.705+0000 7fcf55f52640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf501cd060 0x7fcf5009d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:23.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.705+0000 7fcf55f52640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:23.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.705+0000 7fcf55f52640 1 -- 172.21.15.12:0/2882331096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcf500a1760 con 0x7fcf501cd060 2024-09-11T01:43:23.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.705+0000 7fcf4effd640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf501cd060 0x7fcf5009d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:23.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.705+0000 7fcf4effd640 10 monclient: get_auth_request con 0x7fcf501cd060 auth_method 0 2024-09-11T01:43:23.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.705+0000 7fcf4effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:23.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.705+0000 7fcf4effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:23.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.705+0000 7fcf4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:23.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.705+0000 7fcf4effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:23.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.705+0000 7fcf4effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:23.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.705+0000 7fcf4effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:23.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.705+0000 7fcf37fff640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcf500a1380 0x7fcf3c048a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:23.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.705+0000 7fcf37fff640 1 -- 172.21.15.12:0/2882331096 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(476..476 src has 285..476) ==== 4843+0+0 (secure 0 0 0) 0x7fcf40051080 con 0x7fcf3c0034d0 2024-09-11T01:43:23.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.705+0000 7fcf4effd640 10 monclient(hunting): handle_auth_done global_id 94737 payload 824 2024-09-11T01:43:23.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.705+0000 7fcf4effd640 1 -- 172.21.15.12:0/2882331096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcf500008d0 con 0x7fcf501cd060 2024-09-11T01:43:23.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.705+0000 7fcf4ffff640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcf500a1380 0x7fcf3c048a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:23.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.705+0000 7fcf4ffff640 10 monclient: get_auth_request con 0x7fcf500a1380 auth_method 0 2024-09-11T01:43:23.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.707+0000 7fcf4effd640 1 -- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf501cd060 msgr2=0x7fcf5009d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:23.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.707+0000 7fcf4effd640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf501cd060 0x7fcf5009d7c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fcf44034000 tx=0x7fcf440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:23.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.707+0000 7fcf37fff640 1 -- 172.21.15.12:0/2882331096 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf44037040 con 0x7fcf501cd060 2024-09-11T01:43:23.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.707+0000 7fcf37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:23.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.708+0000 7fcf37fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:23.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.708+0000 7fcf37fff640 10 monclient: dump: 2024-09-11T01:43:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:23.721 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.708+0000 7fcf4ffff640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcf500a1380 0x7fcf3c048a90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcf48009580 tx=0x7fcf48005040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.708+0000 7fcf37fff640 1 -- 172.21.15.12:0/2882331096 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcf4403c640 con 0x7fcf501cd060 2024-09-11T01:43:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.708+0000 7fcf37fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:23.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.708+0000 7fcf37fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:43:23.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.708+0000 7fcf37fff640 1 -- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf501cd060 msgr2=0x7fcf5009d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:23.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.708+0000 7fcf37fff640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf501cd060 0x7fcf5009d7c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fcf44034000 tx=0x7fcf440046c0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.708+0000 7fcf37fff640 1 -- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf501cd060 msgr2=0x7fcf5009d7c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:43:23.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.708+0000 7fcf37fff640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf501cd060 0x7fcf5009d7c0 secure :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fcf44034000 tx=0x7fcf440046c0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.714+0000 7fcf55f52640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:43:23.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.714+0000 7fcf55f52640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf5009dd00 0x7fcf5009e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:23.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.714+0000 7fcf55f52640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:23.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.714+0000 7fcf55f52640 1 -- 172.21.15.12:0/2882331096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5009e650 con 0x7fcf5009dd00 2024-09-11T01:43:23.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.714+0000 7fcf4effd640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf5009dd00 0x7fcf5009e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:23.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.714+0000 7fcf4effd640 10 monclient: get_auth_request con 0x7fcf5009dd00 auth_method 0 2024-09-11T01:43:23.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.714+0000 7fcf4effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:23.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.714+0000 7fcf4effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:23.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.714+0000 7fcf4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:23.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.714+0000 7fcf4effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:23.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.714+0000 7fcf4effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:23.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.714+0000 7fcf4effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:23.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.714+0000 7fcf4effd640 10 monclient(hunting): handle_auth_done global_id 94740 payload 824 2024-09-11T01:43:23.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.714+0000 7fcf4effd640 1 -- 172.21.15.12:0/2882331096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcf5013d4c0 con 0x7fcf5009dd00 2024-09-11T01:43:23.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.717+0000 7fcf4effd640 1 -- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf5009dd00 msgr2=0x7fcf5009e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:23.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.717+0000 7fcf4effd640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf5009dd00 0x7fcf5009e110 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fcf44009870 tx=0x7fcf44008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:23.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.717+0000 7fcf37fff640 1 -- 172.21.15.12:0/2882331096 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4403ce60 con 0x7fcf5009dd00 2024-09-11T01:43:23.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.717+0000 7fcf37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:23.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.717+0000 7fcf37fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:23.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.717+0000 7fcf37fff640 10 monclient: dump: 2024-09-11T01:43:23.725 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:23.725 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:23.725 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.725 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.725 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:23.726 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:23.726 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:23.726 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:23.726 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:23.726 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:23.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.717+0000 7fcf37fff640 1 -- 172.21.15.12:0/2882331096 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4700 (secure 0 0 0) 0x7fcf44003a20 con 0x7fcf5009dd00 2024-09-11T01:43:23.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.717+0000 7fcf37fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:43:23.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.717+0000 7fcf37fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:43:23.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.717+0000 7fcf37fff640 1 -- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf5009dd00 msgr2=0x7fcf5009e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:23.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.717+0000 7fcf37fff640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf5009dd00 0x7fcf5009e110 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fcf44009870 tx=0x7fcf44008030 comp rx=0 tx=0).stop 2024-09-11T01:43:23.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.717+0000 7fcf37fff640 1 -- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf5009dd00 msgr2=0x7fcf5009e110 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:43:23.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.717+0000 7fcf37fff640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf5009dd00 0x7fcf5009e110 secure :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fcf44009870 tx=0x7fcf44008030 comp rx=0 tx=0).stop 2024-09-11T01:43:23.727 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:43:23.727 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:43:23.727 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:43:23.727 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:43:23.727 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 132, 2024-09-11T01:43:23.727 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 349813, 2024-09-11T01:43:23.727 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:43:23.727 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:43:23.727 INFO:teuthology.orchestra.run.smithi012.stdout: 1 2024-09-11T01:43:23.727 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:23.727 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:43:23.727 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:43:23.728 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:43:23.728 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:43:23.728 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:23.728 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:23.728 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:23.728 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:23.728 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:23.728 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:23.728 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:23.728 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:23.728 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:23.728 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:23.728 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:23.729 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:23.729 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:43:23.729 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:43:23.729 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:23.729 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:23.729 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:23.729 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:23.729 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:23.729 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:23.729 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:23.729 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:23.729 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:23.729 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:23.729 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:23.730 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:23.730 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:23.730 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:43:23.730 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:43:23.730 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:43:23.730 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:43:23.730 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:43:23.730 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:43:23.730 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:43:23.730 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:43:23.731 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:43:23.731 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:43:23.731 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:43:23.731 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:43:23.731 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:43:23.731 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:43:23.731 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:43:23.731 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:43:23.731 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:43:23.731 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:23.731 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:23.731 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:23.731 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:23.731 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:23.732 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:23.732 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:23.732 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:23.732 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:23.732 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:23.732 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:23.732 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:23.732 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:43:23.732 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:23.732 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:43:23.732 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:23.732 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:43:23.732 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:43:23.733 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:23.733 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:23.733 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:23.733 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:23.733 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:43:23.733 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:23.733 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:23.733 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:23.733 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:23.734 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:43:23.734 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:23.734 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:23.734 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:23.734 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:23.734 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:43:23.734 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:43:23.734 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:23.734 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:23.734 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:23.734 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:23.734 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:23.734 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:43:23.734 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:43:23.735 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:23.735 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:23.735 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:23.735 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:23.735 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:43:23.735 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:23.735 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:23.735 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:23.735 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:23.735 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:43:23.735 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:23.735 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:23.735 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:23.735 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:23.736 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:43:23.736 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:43:23.736 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:23.736 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:23.736 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:23.736 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:23.736 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:23.736 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:43:23.736 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:43:23.736 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:23.736 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:23.736 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:23.736 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:23.737 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:43:23.737 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:23.737 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:23.737 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:23.737 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:23.737 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:43:23.737 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:23.737 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:23.737 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:23.737 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:23.737 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:43:23.737 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:43:23.737 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:23.737 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:23.738 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:23.738 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:23.738 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:23.738 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:23.738 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:43:23.738 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:43:23.738 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:23.738 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:23.738 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:23.738 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:43:23.738 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:23.738 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:23.738 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:43:23.738 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:23.739 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:23.739 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:23.739 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:43:23.739 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:23.739 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:23.739 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:43:23.739 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:23.739 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:23.739 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:23.739 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:43:23.739 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:23.739 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:23.739 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:23.739 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:43:23.739 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:43:23.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.717+0000 7fcf55f52640 1 -- 172.21.15.12:0/2882331096 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcf500a1380 msgr2=0x7fcf3c048a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:23.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.717+0000 7fcf55f52640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcf500a1380 0x7fcf3c048a90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcf48009580 tx=0x7fcf48005040 comp rx=0 tx=0).stop 2024-09-11T01:43:23.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.717+0000 7fcf55f52640 10 monclient: shutdown 2024-09-11T01:43:23.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.717+0000 7fcf55f52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:23.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.717+0000 7fcf55f52640 1 -- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf3c0034d0 msgr2=0x7fcf3c005920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:23.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.717+0000 7fcf55f52640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf3c0034d0 0x7fcf3c005920 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fcf40005020 tx=0x7fcf48009660 comp rx=0 tx=0).stop 2024-09-11T01:43:23.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.717+0000 7fcf4ffff640 1 -- 172.21.15.12:0/2882331096 reap_dead start 2024-09-11T01:43:23.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.717+0000 7fcf55f52640 1 -- 172.21.15.12:0/2882331096 shutdown_connections 2024-09-11T01:43:23.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.717+0000 7fcf55f52640 1 --2- 172.21.15.12:0/2882331096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcf3c0034d0 0x7fcf3c005920 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.717+0000 7fcf55f52640 1 -- 172.21.15.12:0/2882331096 >> 172.21.15.12:0/2882331096 conn(0x7fcf500a2f70 msgr2=0x7fcf50071d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:23.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.718+0000 7fcf55f52640 1 -- 172.21.15.12:0/2882331096 shutdown_connections 2024-09-11T01:43:23.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.718+0000 7fcf55f52640 1 -- 172.21.15.12:0/2882331096 wait complete. 2024-09-11T01:43:23.742 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:43:23.742 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T01:43:23.743 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:43:23.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.843+0000 7fda21aa3640 10 monclient: get_monmap_and_config 2024-09-11T01:43:23.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.843+0000 7fda21aa3640 10 monclient: build_initial_monmap 2024-09-11T01:43:23.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.843+0000 7fda21aa3640 10 monclient: monmap: 2024-09-11T01:43:23.845 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:23.845 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:23.845 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:43:23.844256+0000 2024-09-11T01:43:23.845 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:43:23.844256+0000 2024-09-11T01:43:23.845 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:23.845 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:23.845 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:43:23.846 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:43:23.846 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:43:23.846 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:43:23.846 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:43:23.846 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:23.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.844+0000 7fda21aa3640 1 Processor -- start 2024-09-11T01:43:23.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.844+0000 7fda21aa3640 1 -- start start 2024-09-11T01:43:23.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.844+0000 7fda21aa3640 10 monclient: init 2024-09-11T01:43:23.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.844+0000 7fda21aa3640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:23.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.844+0000 7fda21aa3640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-11T01:43:23.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.844+0000 7fda21aa3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fda1c074160 addr v1:172.21.15.71:6789/0 2024-09-11T01:43:23.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.844+0000 7fda21aa3640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fda1c0a0540 0x7fda1c13cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:23.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.844+0000 7fda21aa3640 10 monclient(hunting): picked mon.noname-c con 0x7fda1c0a0540 addr v2:172.21.15.71:3300/0 2024-09-11T01:43:23.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.844+0000 7fda21aa3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fda1c0a2ab0 addr v1:172.21.15.12:6789/0 2024-09-11T01:43:23.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.844+0000 7fda21aa3640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda1c0a0230 con 0x7fda1c0a2ab0 2024-09-11T01:43:23.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.844+0000 7fda21aa3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda1c070160 con 0x7fda1c074160 2024-09-11T01:43:23.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.844+0000 7fda21aa3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:23.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.844+0000 7fda21aa3640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fda1c070440 con 0x7fda1c0a0540 2024-09-11T01:43:23.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.844+0000 7fda21aa3640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:23.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.844+0000 7fda21aa3640 10 monclient(hunting): authenticate will time out at 1610.286829s 2024-09-11T01:43:23.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.844+0000 7fda1bfff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fda1c0a2ab0 0x7fda1c13f4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53224/0 (socket says 172.21.15.12:53224) 2024-09-11T01:43:23.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.844+0000 7fda1bfff640 1 -- 172.21.15.12:0/1820546396 learned_addr learned my addr 172.21.15.12:0/1820546396 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:43:23.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.845+0000 7fda1affd640 1 --2- 172.21.15.12:0/1820546396 >> v2:172.21.15.71:3300/0 conn(0x7fda1c0a0540 0x7fda1c13cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:23.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.845+0000 7fda1affd640 10 monclient(hunting): get_auth_request con 0x7fda1c0a0540 auth_method 0 2024-09-11T01:43:23.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.845+0000 7fda1affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:23.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.845+0000 7fda1affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:23.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.845+0000 7fda1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:23.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.845+0000 7fda1affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:23.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.845+0000 7fda1affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:23.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.845+0000 7fda1affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:23.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.845+0000 7fda1affd640 10 monclient(hunting): handle_auth_done global_id 134335 payload 274 2024-09-11T01:43:23.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.845+0000 7fda1affd640 1 -- 172.21.15.12:0/1820546396 >> v1:172.21.15.71:6789/0 conn(0x7fda1c074160 legacy=0x7fda1c074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:23.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.845+0000 7fda1affd640 1 -- 172.21.15.12:0/1820546396 >> v1:172.21.15.12:6789/0 conn(0x7fda1c0a2ab0 legacy=0x7fda1c13f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:23.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.845+0000 7fda1affd640 10 monclient: _finish_hunting 0 2024-09-11T01:43:23.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.845+0000 7fda1affd640 1 monclient: found mon.noname-c 2024-09-11T01:43:23.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.845+0000 7fda1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:23.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.845+0000 7fda1affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:43:23.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.845+0000 7fda1affd640 1 -- 172.21.15.12:0/1820546396 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda1c070720 con 0x7fda1c0a0540 2024-09-11T01:43:23.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.848+0000 7fda1affd640 1 -- 172.21.15.12:0/1820546396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda1c0a0540 msgr2=0x7fda1c13cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:23.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.848+0000 7fda1affd640 1 --2- 172.21.15.12:0/1820546396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda1c0a0540 0x7fda1c13cfa0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fda100097a0 tx=0x7fda10029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=403e6b65030102b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:23.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.848+0000 7fda1a7fc640 1 -- 172.21.15.12:0/1820546396 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda10037070 con 0x7fda1c0a0540 2024-09-11T01:43:23.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.848+0000 7fda1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:23.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.848+0000 7fda1a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:43:23.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.848+0000 7fda1a7fc640 10 monclient: dump: 2024-09-11T01:43:23.855 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:23.855 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:23.855 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.855 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.855 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:23.855 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:23.855 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:23.855 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:23.855 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:23.855 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:23.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.848+0000 7fda1a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:43:23.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.848+0000 7fda1a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:43:23.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.848+0000 7fda1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:23.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.848+0000 7fda1a7fc640 1 -- 172.21.15.12:0/1820546396 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda10032640 con 0x7fda1c0a0540 2024-09-11T01:43:23.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.848+0000 7fda1a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:23.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.848+0000 7fda1a7fc640 1 -- 172.21.15.12:0/1820546396 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda10032a10 con 0x7fda1c0a0540 2024-09-11T01:43:23.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.848+0000 7fda21aa3640 5 monclient: authenticate success, global_id 134335 2024-09-11T01:43:23.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.848+0000 7fda21aa3640 10 monclient: get_monmap_and_config success 2024-09-11T01:43:23.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.848+0000 7fda21aa3640 4 set_mon_vals no callback set 2024-09-11T01:43:23.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.848+0000 7fda1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:23.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.848+0000 7fda1a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:43:23.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.848+0000 7fda1a7fc640 10 monclient: dump: 2024-09-11T01:43:23.856 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:23.856 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:23.856 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.856 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.856 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:23.857 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:23.857 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:23.857 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:23.857 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:23.857 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:23.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.849+0000 7fda21aa3640 10 monclient: shutdown 2024-09-11T01:43:23.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.849+0000 7fda21aa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:23.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.849+0000 7fda21aa3640 1 -- 172.21.15.12:0/1820546396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda1c0a0540 msgr2=0x7fda1c13cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:23.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.849+0000 7fda21aa3640 1 --2- 172.21.15.12:0/1820546396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda1c0a0540 0x7fda1c13cfa0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fda100097a0 tx=0x7fda10029fd0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.849+0000 7fda21aa3640 1 -- 172.21.15.12:0/1820546396 shutdown_connections 2024-09-11T01:43:23.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.849+0000 7fda21aa3640 1 --2- 172.21.15.12:0/1820546396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda1c0a0540 0x7fda1c13cfa0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.849+0000 7fda21aa3640 1 -- 172.21.15.12:0/1820546396 >> 172.21.15.12:0/1820546396 conn(0x7fda1c0a36c0 msgr2=0x7fda1c0a3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:23.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.849+0000 7fda21aa3640 1 -- 172.21.15.12:0/1820546396 shutdown_connections 2024-09-11T01:43:23.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.849+0000 7fda21aa3640 1 -- 172.21.15.12:0/1820546396 wait complete. 2024-09-11T01:43:23.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.849+0000 7fda21aa3640 10 monclient: build_initial_monmap 2024-09-11T01:43:23.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.849+0000 7fda21aa3640 10 monclient: monmap: 2024-09-11T01:43:23.858 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:23.858 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:43:23.858 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:43:23.858 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:43:23.858 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:23.858 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:23.858 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:43:23.858 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:43:23.858 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:43:23.858 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:23.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda21aa3640 1 Processor -- start 2024-09-11T01:43:23.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda21aa3640 1 -- start start 2024-09-11T01:43:23.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda21aa3640 10 monclient: init 2024-09-11T01:43:23.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda21aa3640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:23.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda21aa3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:43:23.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda21aa3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda1c074160 0x7fda1c1316c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:23.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda21aa3640 10 monclient(hunting): picked mon.noname-a con 0x7fda1c074160 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:23.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda21aa3640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda1c0a0540 0x7fda1c131c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:23.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda21aa3640 10 monclient(hunting): picked mon.noname-c con 0x7fda1c0a0540 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:23.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda21aa3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda1c0a2ab0 0x7fda1c139f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:23.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda21aa3640 10 monclient(hunting): picked mon.noname-b con 0x7fda1c0a2ab0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:23.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda21aa3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:23.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda21aa3640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c140d90 con 0x7fda1c074160 2024-09-11T01:43:23.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda21aa3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:23.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda21aa3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c140c10 con 0x7fda1c0a2ab0 2024-09-11T01:43:23.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda21aa3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:23.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda21aa3640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fda1c140f10 con 0x7fda1c0a0540 2024-09-11T01:43:23.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda21aa3640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:23.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda21aa3640 10 monclient(hunting): authenticate will time out at 1610.292710s 2024-09-11T01:43:23.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda1affd640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda1c0a0540 msgr2=0x7fda1c131c00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:43:23.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda1affd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda1c0a0540 0x7fda1c131c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:23.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda1b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda1c074160 0x7fda1c1316c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:23.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda1b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda1c074160 0x7fda1c1316c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39190/0 (socket says 172.21.15.12:39190) 2024-09-11T01:43:23.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda1b7fe640 1 -- 172.21.15.12:0/1114242767 learned_addr learned my addr 172.21.15.12:0/1114242767 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:23.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda1b7fe640 10 monclient(hunting): get_auth_request con 0x7fda1c074160 auth_method 0 2024-09-11T01:43:23.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:23.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda1b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:23.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:23.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:23.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:23.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.850+0000 7fda1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:23.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.851+0000 7fda1bfff640 1 --2- 172.21.15.12:0/1114242767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda1c0a2ab0 0x7fda1c139f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:23.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.851+0000 7fda1bfff640 10 monclient(hunting): get_auth_request con 0x7fda1c0a2ab0 auth_method 0 2024-09-11T01:43:23.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.851+0000 7fda1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:23.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.851+0000 7fda1bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:23.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.851+0000 7fda1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:23.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.851+0000 7fda1b7fe640 10 monclient(hunting): handle_auth_done global_id 94743 payload 824 2024-09-11T01:43:23.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.851+0000 7fda1b7fe640 1 -- 172.21.15.12:0/1114242767 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda1c0a0540 msgr2=0x7fda1c131c00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:43:23.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.851+0000 7fda1b7fe640 1 --2- 172.21.15.12:0/1114242767 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda1c0a0540 0x7fda1c131c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.851+0000 7fda1b7fe640 1 -- 172.21.15.12:0/1114242767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda1c0a2ab0 msgr2=0x7fda1c139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:23.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.851+0000 7fda1b7fe640 1 --2- 172.21.15.12:0/1114242767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda1c0a2ab0 0x7fda1c139f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.851+0000 7fda1b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:43:23.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.851+0000 7fda1b7fe640 1 monclient: found mon.noname-a 2024-09-11T01:43:23.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.851+0000 7fda1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:23.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.851+0000 7fda1b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:23.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.851+0000 7fda1b7fe640 1 -- 172.21.15.12:0/1114242767 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda1c0713e0 con 0x7fda1c074160 2024-09-11T01:43:23.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.851+0000 7fda1bfff640 1 --2- 172.21.15.12:0/1114242767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda1c0a2ab0 0x7fda1c139f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:43:23.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.853+0000 7fda1b7fe640 1 -- 172.21.15.12:0/1114242767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda1c074160 msgr2=0x7fda1c1316c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:23.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.853+0000 7fda1b7fe640 1 --2- 172.21.15.12:0/1114242767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda1c074160 0x7fda1c1316c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fda0c00c430 tx=0x7fda0c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:23.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.853+0000 7fda03fff640 1 -- 172.21.15.12:0/1114242767 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0c0119a0 con 0x7fda1c074160 2024-09-11T01:43:23.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.853+0000 7fda03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:23.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.853+0000 7fda03fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:23.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.853+0000 7fda03fff640 10 monclient: dump: 2024-09-11T01:43:23.865 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:23.865 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:23.865 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.865 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.865 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:23.865 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:23.866 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:23.866 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:23.866 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:23.866 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:23.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.853+0000 7fda03fff640 10 monclient: _finish_auth 0 2024-09-11T01:43:23.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.853+0000 7fda03fff640 10 monclient: _check_auth_tickets 2024-09-11T01:43:23.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.853+0000 7fda03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:23.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.853+0000 7fda03fff640 1 -- 172.21.15.12:0/1114242767 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda0c017070 con 0x7fda1c074160 2024-09-11T01:43:23.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.853+0000 7fda03fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:23.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.853+0000 7fda03fff640 1 -- 172.21.15.12:0/1114242767 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0c011d80 con 0x7fda1c074160 2024-09-11T01:43:23.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.853+0000 7fda21aa3640 5 monclient: authenticate success, global_id 94743 2024-09-11T01:43:23.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.853+0000 7fda21aa3640 10 monclient: _renew_subs 2024-09-11T01:43:23.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.853+0000 7fda21aa3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:23.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.853+0000 7fda21aa3640 1 -- 172.21.15.12:0/1114242767 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda1c075000 con 0x7fda1c074160 2024-09-11T01:43:23.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.853+0000 7fda03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:23.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.853+0000 7fda03fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:23.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.853+0000 7fda03fff640 10 monclient: dump: 2024-09-11T01:43:23.867 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:23.867 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:23.867 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.867 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.867 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:23.867 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:23.867 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:23.867 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:23.867 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:23.867 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:23.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.853+0000 7fda19ffb640 4 set_mon_vals no callback set 2024-09-11T01:43:23.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.854+0000 7fda21aa3640 10 monclient: _renew_subs 2024-09-11T01:43:23.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.854+0000 7fda21aa3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:23.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.854+0000 7fda21aa3640 1 -- 172.21.15.12:0/1114242767 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda1c13c590 con 0x7fda1c074160 2024-09-11T01:43:23.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.854+0000 7fda21aa3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:23.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.854+0000 7fda21aa3640 1 --2- 172.21.15.12:0/1114242767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda1c13ac70 0x7fda1c02d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:23.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.854+0000 7fda21aa3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:23.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.854+0000 7fda21aa3640 1 -- 172.21.15.12:0/1114242767 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c074540 con 0x7fda1c13ac70 2024-09-11T01:43:23.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.854+0000 7fda03fff640 1 -- 172.21.15.12:0/1114242767 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fda0c01c460 con 0x7fda1c074160 2024-09-11T01:43:23.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.855+0000 7fda1affd640 1 --2- 172.21.15.12:0/1114242767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda1c13ac70 0x7fda1c02d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:23.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.855+0000 7fda1affd640 10 monclient: get_auth_request con 0x7fda1c13ac70 auth_method 0 2024-09-11T01:43:23.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.855+0000 7fda1affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:23.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.855+0000 7fda1affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:23.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.855+0000 7fda1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:23.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.855+0000 7fda03fff640 1 --2- 172.21.15.12:0/1114242767 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda0803e240 0x7fda08040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.855+0000 7fda1affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.855+0000 7fda1affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.855+0000 7fda1affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.855+0000 7fda03fff640 1 -- 172.21.15.12:0/1114242767 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(476..476 src has 285..476) ==== 4843+0+0 (secure 0 0 0) 0x7fda0c052c30 con 0x7fda1c074160 2024-09-11T01:43:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.855+0000 7fda1affd640 10 monclient(hunting): handle_auth_done global_id 134341 payload 824 2024-09-11T01:43:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.855+0000 7fda1affd640 1 -- 172.21.15.12:0/1114242767 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fda1c135e70 con 0x7fda1c13ac70 2024-09-11T01:43:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.855+0000 7fda1bfff640 1 --2- 172.21.15.12:0/1114242767 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda0803e240 0x7fda08040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.855+0000 7fda1bfff640 10 monclient: get_auth_request con 0x7fda0803e240 auth_method 0 2024-09-11T01:43:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.857+0000 7fda1affd640 1 -- 172.21.15.12:0/1114242767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda1c13ac70 msgr2=0x7fda1c02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.857+0000 7fda1affd640 1 --2- 172.21.15.12:0/1114242767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda1c13ac70 0x7fda1c02d440 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fda10034000 tx=0x7fda100046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.858+0000 7fda1bfff640 1 --2- 172.21.15.12:0/1114242767 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda0803e240 0x7fda08040700 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fda1c13d5a0 tx=0x7fda14009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.858+0000 7fda03fff640 1 -- 172.21.15.12:0/1114242767 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda10042070 con 0x7fda1c13ac70 2024-09-11T01:43:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.858+0000 7fda03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:23.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.858+0000 7fda03fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:43:23.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.858+0000 7fda03fff640 10 monclient: dump: 2024-09-11T01:43:23.870 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:23.870 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:23.870 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.870 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.870 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:23.870 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:23.870 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:23.870 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:23.870 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:23.870 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:23.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.858+0000 7fda03fff640 1 -- 172.21.15.12:0/1114242767 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fda10032a00 con 0x7fda1c13ac70 2024-09-11T01:43:23.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.858+0000 7fda03fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:23.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.858+0000 7fda03fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:43:23.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.858+0000 7fda03fff640 1 -- 172.21.15.12:0/1114242767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda1c13ac70 msgr2=0x7fda1c02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:23.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.858+0000 7fda03fff640 1 --2- 172.21.15.12:0/1114242767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda1c13ac70 0x7fda1c02d440 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fda10034000 tx=0x7fda100046c0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.858+0000 7fda03fff640 1 -- 172.21.15.12:0/1114242767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda1c13ac70 msgr2=0x7fda1c02d440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:43:23.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.858+0000 7fda03fff640 1 --2- 172.21.15.12:0/1114242767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda1c13ac70 0x7fda1c02d440 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.864+0000 7fda21aa3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:43:23.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.864+0000 7fda21aa3640 1 --2- 172.21.15.12:0/1114242767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda1c13b050 0x7fda1c02d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:23.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.864+0000 7fda21aa3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:23.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.864+0000 7fda21aa3640 1 -- 172.21.15.12:0/1114242767 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c06dc80 con 0x7fda1c13b050 2024-09-11T01:43:23.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.864+0000 7fda1affd640 1 --2- 172.21.15.12:0/1114242767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda1c13b050 0x7fda1c02d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:23.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.864+0000 7fda1affd640 10 monclient: get_auth_request con 0x7fda1c13b050 auth_method 0 2024-09-11T01:43:23.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.864+0000 7fda1affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:23.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.864+0000 7fda1affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:23.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.864+0000 7fda1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:23.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.865+0000 7fda1affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:23.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.865+0000 7fda1affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:23.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.865+0000 7fda1affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:23.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.865+0000 7fda1affd640 10 monclient(hunting): handle_auth_done global_id 134344 payload 824 2024-09-11T01:43:23.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.865+0000 7fda1affd640 1 -- 172.21.15.12:0/1114242767 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fda1c132140 con 0x7fda1c13b050 2024-09-11T01:43:23.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.867+0000 7fda1affd640 1 -- 172.21.15.12:0/1114242767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda1c13b050 msgr2=0x7fda1c02d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:23.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.867+0000 7fda1affd640 1 --2- 172.21.15.12:0/1114242767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda1c13b050 0x7fda1c02d980 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fda10000c00 tx=0x7fda10009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:23.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.867+0000 7fda03fff640 1 -- 172.21.15.12:0/1114242767 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda10037040 con 0x7fda1c13b050 2024-09-11T01:43:23.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.867+0000 7fda03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:23.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.867+0000 7fda03fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:43:23.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.867+0000 7fda03fff640 10 monclient: dump: 2024-09-11T01:43:23.874 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:23.874 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:23.874 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.874 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:23.874 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:23.874 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:23.874 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:23.874 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:23.875 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:23.875 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:23.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.867+0000 7fda03fff640 1 -- 172.21.15.12:0/1114242767 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4865 (secure 0 0 0) 0x7fda10003820 con 0x7fda1c13b050 2024-09-11T01:43:23.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.867+0000 7fda03fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:43:23.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.867+0000 7fda03fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:43:23.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.867+0000 7fda03fff640 1 -- 172.21.15.12:0/1114242767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda1c13b050 msgr2=0x7fda1c02d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:23.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.867+0000 7fda03fff640 1 --2- 172.21.15.12:0/1114242767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda1c13b050 0x7fda1c02d980 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fda10000c00 tx=0x7fda10009190 comp rx=0 tx=0).stop 2024-09-11T01:43:23.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.867+0000 7fda03fff640 1 -- 172.21.15.12:0/1114242767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda1c13b050 msgr2=0x7fda1c02d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:43:23.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.867+0000 7fda03fff640 1 --2- 172.21.15.12:0/1114242767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda1c13b050 0x7fda1c02d980 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.875 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:43:23.876 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:43:23.876 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:43:23.876 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:43:23.876 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 132, 2024-09-11T01:43:23.876 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 136830, 2024-09-11T01:43:23.876 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:43:23.876 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:43:23.876 INFO:teuthology.orchestra.run.smithi012.stdout: 1 2024-09-11T01:43:23.876 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:23.876 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:43:23.876 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:43:23.876 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:43:23.876 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:43:23.877 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:23.877 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:23.877 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:23.877 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:23.877 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:23.877 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:23.877 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:23.877 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:23.877 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:23.877 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:23.877 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:23.877 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:23.877 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:43:23.877 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:43:23.878 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:23.878 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:23.878 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:23.878 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:23.878 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:23.878 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:23.878 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:23.878 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:23.878 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:23.878 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:23.878 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:23.878 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:23.878 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:23.878 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:43:23.879 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:43:23.879 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:43:23.879 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:43:23.879 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:43:23.879 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:43:23.879 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:43:23.879 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:43:23.879 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:43:23.879 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:43:23.879 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:43:23.879 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:43:23.879 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:43:23.879 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:43:23.879 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:43:23.880 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:43:23.880 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:43:23.880 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:23.880 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:23.880 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:23.880 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:23.880 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:23.880 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:23.880 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:23.880 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:23.880 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:23.880 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:23.880 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:23.880 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:23.881 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:43:23.881 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:23.881 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:43:23.881 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:23.881 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:43:23.881 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:43:23.881 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:23.881 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:23.881 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:23.881 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:23.881 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:43:23.881 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:23.881 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:23.881 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:23.882 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:23.882 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:43:23.882 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:23.882 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:23.882 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:23.882 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:23.882 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:43:23.882 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:43:23.882 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:23.882 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:23.882 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:23.882 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:23.882 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:23.882 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:43:23.883 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:43:23.883 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:23.883 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:23.883 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:23.883 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:23.883 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:43:23.883 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:23.883 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:23.883 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:23.883 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:23.883 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:43:23.883 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:23.883 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:23.883 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:23.884 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:23.884 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:43:23.884 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:43:23.884 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:23.884 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:23.884 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:23.884 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:23.884 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:23.884 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:43:23.884 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:43:23.884 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:23.884 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:23.884 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:23.885 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:23.885 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:43:23.885 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:23.885 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:23.885 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:23.885 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:23.885 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:43:23.885 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:23.885 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:23.885 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:23.885 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:23.885 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:43:23.885 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:43:23.885 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:23.886 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:23.886 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:23.886 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:23.886 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:23.886 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:23.886 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:43:23.886 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:43:23.886 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:23.886 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:23.886 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:23.886 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:43:23.886 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:23.886 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:23.886 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:43:23.886 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:23.887 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:23.887 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:23.887 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:43:23.887 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:23.887 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:23.887 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:43:23.887 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:23.887 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:23.887 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:23.887 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:43:23.887 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:23.887 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:23.887 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:43:23.887 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:23.888 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:23.888 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:23.888 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:43:23.888 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:23.888 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:23.888 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:23.888 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:43:23.888 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:43:23.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.868+0000 7fda21aa3640 1 -- 172.21.15.12:0/1114242767 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda0803e240 msgr2=0x7fda08040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:23.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.868+0000 7fda21aa3640 1 --2- 172.21.15.12:0/1114242767 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda0803e240 0x7fda08040700 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fda1c13d5a0 tx=0x7fda14009040 comp rx=0 tx=0).stop 2024-09-11T01:43:23.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.868+0000 7fda21aa3640 10 monclient: shutdown 2024-09-11T01:43:23.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.868+0000 7fda21aa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:23.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.868+0000 7fda21aa3640 1 -- 172.21.15.12:0/1114242767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda1c074160 msgr2=0x7fda1c1316c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:23.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.868+0000 7fda21aa3640 1 --2- 172.21.15.12:0/1114242767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda1c074160 0x7fda1c1316c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fda0c00c430 tx=0x7fda0c00c8f0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.868+0000 7fda1bfff640 1 -- 172.21.15.12:0/1114242767 reap_dead start 2024-09-11T01:43:23.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.868+0000 7fda21aa3640 1 -- 172.21.15.12:0/1114242767 shutdown_connections 2024-09-11T01:43:23.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.868+0000 7fda21aa3640 1 --2- 172.21.15.12:0/1114242767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda1c074160 0x7fda1c1316c0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:23.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.868+0000 7fda21aa3640 1 -- 172.21.15.12:0/1114242767 >> 172.21.15.12:0/1114242767 conn(0x7fda1c0a36c0 msgr2=0x7fda1c071be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:23.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.868+0000 7fda21aa3640 1 -- 172.21.15.12:0/1114242767 shutdown_connections 2024-09-11T01:43:23.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:23.868+0000 7fda21aa3640 1 -- 172.21.15.12:0/1114242767 wait complete. 2024-09-11T01:43:23.891 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:43:23.891 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:43:24.770 INFO:tasks.rados.rados.0.smithi071.stdout:2127: done (0 left) 2024-09-11T01:43:24.770 INFO:tasks.rados.rados.0.smithi071.stdout:2128: read oid 28 snap -1 2024-09-11T01:43:24.770 INFO:tasks.rados.rados.0.smithi071.stdout:2128: expect (ObjNum 724 snap 224 seq_num 724) 2024-09-11T01:43:24.771 INFO:tasks.rados.rados.0.smithi071.stdout:2129: write (excl) oid 3 current snap is 233 2024-09-11T01:43:24.771 INFO:tasks.rados.rados.0.smithi071.stdout:2129: seq_num 770 ranges {541390=493383,1592076=489269,2862175=79215} 2024-09-11T01:43:24.775 INFO:tasks.rados.rados.0.smithi071.stdout:2129: writing smithi07134777-3 from 541390 to 1034773 tid 1 2024-09-11T01:43:24.782 INFO:tasks.rados.rados.0.smithi071.stdout:2129: writing smithi07134777-3 from 1592076 to 2081345 tid 2 2024-09-11T01:43:24.787 INFO:tasks.rados.rados.0.smithi071.stdout:2129: writing smithi07134777-3 from 2862175 to 2941390 tid 3 2024-09-11T01:43:24.788 INFO:tasks.rados.rados.0.smithi071.stdout:2130: read oid 25 snap 231 2024-09-11T01:43:24.788 INFO:tasks.rados.rados.0.smithi071.stdout:2130: expect (ObjNum 723 snap 224 seq_num 723) 2024-09-11T01:43:24.788 INFO:tasks.rados.rados.0.smithi071.stdout:2131: snap_create 2024-09-11T01:43:24.788 INFO:tasks.rados.rados.0.smithi071.stdout:2128: expect (ObjNum 724 snap 224 seq_num 724) 2024-09-11T01:43:24.788 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:24.875 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2056502 2024-09-11T01:43:24.925 INFO:tasks.rados.rados.0.smithi071.stdout:2129: finishing write tid 1 to smithi07134777-3 2024-09-11T01:43:24.925 INFO:tasks.rados.rados.0.smithi071.stdout:2129: finishing write tid 2 to smithi07134777-3 2024-09-11T01:43:24.925 INFO:tasks.rados.rados.0.smithi071.stdout:2129: finishing write tid 3 to smithi07134777-3 2024-09-11T01:43:24.925 INFO:tasks.rados.rados.0.smithi071.stdout:2129: finishing write tid 4 to smithi07134777-3 2024-09-11T01:43:24.925 INFO:tasks.rados.rados.0.smithi071.stdout:2129: finishing write tid 5 to smithi07134777-3 2024-09-11T01:43:24.926 INFO:tasks.rados.rados.0.smithi071.stdout:2129: oid 3 updating version 0 to 707 2024-09-11T01:43:24.926 INFO:tasks.rados.rados.0.smithi071.stdout:2129: oid 3 updating version 707 to 710 2024-09-11T01:43:24.926 INFO:tasks.rados.rados.0.smithi071.stdout:2129: oid 3 version 710 is already newer than 709 2024-09-11T01:43:24.926 INFO:tasks.rados.rados.0.smithi071.stdout:2129: oid 3 version 710 is already newer than 708 2024-09-11T01:43:24.926 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 710 (ObjNum 770 snap 233 seq_num 770) dirty exists 2024-09-11T01:43:24.926 INFO:tasks.rados.rados.0.smithi071.stdout:2129: left oid 3 (ObjNum 770 snap 233 seq_num 770) 2024-09-11T01:43:24.926 INFO:tasks.rados.rados.0.smithi071.stdout:2130: expect (ObjNum 723 snap 224 seq_num 723) 2024-09-11T01:43:24.926 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:24.949 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:24.976 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 772341 2024-09-11T01:43:25.772 INFO:tasks.rados.rados.0.smithi071.stdout:2128: done (3 left) 2024-09-11T01:43:25.772 INFO:tasks.rados.rados.0.smithi071.stdout:2129: done (2 left) 2024-09-11T01:43:25.772 INFO:tasks.rados.rados.0.smithi071.stdout:2130: done (1 left) 2024-09-11T01:43:25.772 INFO:tasks.rados.rados.0.smithi071.stdout:2131: done (0 left) 2024-09-11T01:43:25.772 INFO:tasks.rados.rados.0.smithi071.stdout:2132: delete oid 2 current snap is 234 2024-09-11T01:43:25.776 INFO:tasks.rados.rados.0.smithi071.stdout:2132: done (0 left) 2024-09-11T01:43:25.776 INFO:tasks.rados.rados.0.smithi071.stdout:2133: write (excl) oid 4 current snap is 234 2024-09-11T01:43:25.776 INFO:tasks.rados.rados.0.smithi071.stdout:2133: seq_num 772 ranges {575004=643881,1709040=492938,2917021=57983} 2024-09-11T01:43:25.782 INFO:tasks.rados.rados.0.smithi071.stdout:2133: writing smithi07134777-4 from 575004 to 1218885 tid 1 2024-09-11T01:43:25.787 INFO:tasks.rados.rados.0.smithi071.stdout:2133: writing smithi07134777-4 from 1709040 to 2201978 tid 2 2024-09-11T01:43:25.792 INFO:tasks.rados.rados.0.smithi071.stdout:2133: writing smithi07134777-4 from 2917021 to 2975004 tid 3 2024-09-11T01:43:25.792 INFO:tasks.rados.rados.0.smithi071.stdout:2134: snap_create 2024-09-11T01:43:25.792 INFO:tasks.rados.rados.0.smithi071.stdout:2133: finishing write tid 1 to smithi07134777-4 2024-09-11T01:43:25.792 INFO:tasks.rados.rados.0.smithi071.stdout:2133: finishing write tid 2 to smithi07134777-4 2024-09-11T01:43:25.794 INFO:tasks.rados.rados.0.smithi071.stdout:2133: finishing write tid 3 to smithi07134777-4 2024-09-11T01:43:25.796 INFO:tasks.rados.rados.0.smithi071.stdout:2133: finishing write tid 4 to smithi07134777-4 2024-09-11T01:43:25.796 INFO:tasks.rados.rados.0.smithi071.stdout:2133: finishing write tid 5 to smithi07134777-4 2024-09-11T01:43:25.796 INFO:tasks.rados.rados.0.smithi071.stdout:2133: oid 4 updating version 0 to 965 2024-09-11T01:43:25.796 INFO:tasks.rados.rados.0.smithi071.stdout:2133: oid 4 updating version 965 to 967 2024-09-11T01:43:25.797 INFO:tasks.rados.rados.0.smithi071.stdout:2133: oid 4 updating version 967 to 968 2024-09-11T01:43:25.797 INFO:tasks.rados.rados.0.smithi071.stdout:2133: oid 4 version 968 is already newer than 966 2024-09-11T01:43:25.797 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 968 (ObjNum 772 snap 234 seq_num 772) dirty exists 2024-09-11T01:43:25.797 INFO:tasks.rados.rados.0.smithi071.stdout:2133: left oid 4 (ObjNum 772 snap 234 seq_num 772) 2024-09-11T01:43:26.774 INFO:tasks.rados.rados.0.smithi071.stdout:2133: done (1 left) 2024-09-11T01:43:26.774 INFO:tasks.rados.rados.0.smithi071.stdout:2134: done (0 left) 2024-09-11T01:43:26.774 INFO:tasks.rados.rados.0.smithi071.stdout:2135: snap_create 2024-09-11T01:43:27.782 INFO:tasks.rados.rados.0.smithi071.stdout:2135: done (0 left) 2024-09-11T01:43:27.782 INFO:tasks.rados.rados.0.smithi071.stdout:2136: read oid 11 snap 227 2024-09-11T01:43:27.782 INFO:tasks.rados.rados.0.smithi071.stdout:2136: expect deleted 2024-09-11T01:43:27.782 INFO:tasks.rados.rados.0.smithi071.stdout:2137: write (excl) oid 20 current snap is 236 2024-09-11T01:43:27.782 INFO:tasks.rados.rados.0.smithi071.stdout:2137: seq_num 773 ranges {591811=719130,1767522=636596,2828091=163720} 2024-09-11T01:43:27.788 INFO:tasks.rados.rados.0.smithi071.stdout:2137: writing smithi07134777-20 from 591811 to 1310941 tid 1 2024-09-11T01:43:27.795 INFO:tasks.rados.rados.0.smithi071.stdout:2137: writing smithi07134777-20 from 1767522 to 2404118 tid 2 2024-09-11T01:43:27.799 INFO:tasks.rados.rados.0.smithi071.stdout:2137: writing smithi07134777-20 from 2828091 to 2991811 tid 3 2024-09-11T01:43:27.799 INFO:tasks.rados.rados.0.smithi071.stdout:2138: copy_from oid 15 from oid 49 current snap is 236 2024-09-11T01:43:27.799 INFO:tasks.rados.rados.0.smithi071.stdout:2137: finishing write tid 1 to smithi07134777-20 2024-09-11T01:43:27.799 INFO:tasks.rados.rados.0.smithi071.stdout:2137: finishing write tid 2 to smithi07134777-20 2024-09-11T01:43:27.799 INFO:tasks.rados.rados.0.smithi071.stdout:2136: done (2 left) 2024-09-11T01:43:27.799 INFO:tasks.rados.rados.0.smithi071.stdout:2139: read oid 7 snap -1 2024-09-11T01:43:27.799 INFO:tasks.rados.rados.0.smithi071.stdout:2139: expect (ObjNum 769 snap 233 seq_num 769) 2024-09-11T01:43:27.799 INFO:tasks.rados.rados.0.smithi071.stdout:2140: write oid 36 current snap is 236 2024-09-11T01:43:27.799 INFO:tasks.rados.rados.0.smithi071.stdout:2140: seq_num 774 ranges {608618=794379,2109651=780254} 2024-09-11T01:43:27.807 INFO:tasks.rados.rados.0.smithi071.stdout:2140: writing smithi07134777-36 from 608618 to 1402997 tid 1 2024-09-11T01:43:27.815 INFO:tasks.rados.rados.0.smithi071.stdout:2140: writing smithi07134777-36 from 2109651 to 2889905 tid 2 2024-09-11T01:43:27.816 INFO:tasks.rados.rados.0.smithi071.stdout:2141: read oid 5 snap -1 2024-09-11T01:43:27.816 INFO:tasks.rados.rados.0.smithi071.stdout:2141: expect deleted 2024-09-11T01:43:27.816 INFO:tasks.rados.rados.0.smithi071.stdout:2142: read oid 35 snap 227 2024-09-11T01:43:27.816 INFO:tasks.rados.rados.0.smithi071.stdout:2142: expect (ObjNum 733 snap 226 seq_num 733) 2024-09-11T01:43:27.816 INFO:tasks.rados.rados.0.smithi071.stdout:2143: copy_from oid 4 from oid 50 current snap is 236 2024-09-11T01:43:27.816 INFO:tasks.rados.rados.0.smithi071.stdout:2137: finishing write tid 3 to smithi07134777-20 2024-09-11T01:43:27.816 INFO:tasks.rados.rados.0.smithi071.stdout:2137: finishing write tid 4 to smithi07134777-20 2024-09-11T01:43:27.816 INFO:tasks.rados.rados.0.smithi071.stdout:2137: finishing write tid 5 to smithi07134777-20 2024-09-11T01:43:27.816 INFO:tasks.rados.rados.0.smithi071.stdout:2137: oid 20 updating version 0 to 976 2024-09-11T01:43:27.816 INFO:tasks.rados.rados.0.smithi071.stdout:2137: oid 20 updating version 976 to 978 2024-09-11T01:43:27.816 INFO:tasks.rados.rados.0.smithi071.stdout:2137: oid 20 updating version 978 to 979 2024-09-11T01:43:27.816 INFO:tasks.rados.rados.0.smithi071.stdout:2137: oid 20 version 979 is already newer than 977 2024-09-11T01:43:27.816 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 979 (ObjNum 773 snap 236 seq_num 773) dirty exists 2024-09-11T01:43:27.817 INFO:tasks.rados.rados.0.smithi071.stdout:2137: left oid 20 (ObjNum 773 snap 236 seq_num 773) 2024-09-11T01:43:27.817 INFO:tasks.rados.rados.0.smithi071.stdout:2139: expect (ObjNum 769 snap 233 seq_num 769) 2024-09-11T01:43:27.894 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:27.940 INFO:tasks.rados.rados.0.smithi071.stdout:2138: finishing copy_from to smithi07134777-15 2024-09-11T01:43:27.940 INFO:tasks.rados.rados.0.smithi071.stdout:2138: got expected ENOENT (src dne) 2024-09-11T01:43:27.940 INFO:tasks.rados.rados.0.smithi071.stdout:2138: finishing copy_from racing read to smithi07134777-15 2024-09-11T01:43:27.940 INFO:tasks.rados.rados.0.smithi071.stdout:2140: finishing write tid 1 to smithi07134777-36 2024-09-11T01:43:27.940 INFO:tasks.rados.rados.0.smithi071.stdout:2140: finishing write tid 2 to smithi07134777-36 2024-09-11T01:43:27.940 INFO:tasks.rados.rados.0.smithi071.stdout:2140: finishing write tid 3 to smithi07134777-36 2024-09-11T01:43:27.940 INFO:tasks.rados.rados.0.smithi071.stdout:2140: finishing write tid 4 to smithi07134777-36 2024-09-11T01:43:27.940 INFO:tasks.rados.rados.0.smithi071.stdout:2140: oid 36 updating version 0 to 1020 2024-09-11T01:43:27.940 INFO:tasks.rados.rados.0.smithi071.stdout:2140: oid 36 updating version 1020 to 1022 2024-09-11T01:43:27.940 INFO:tasks.rados.rados.0.smithi071.stdout:2140: oid 36 version 1022 is already newer than 1021 2024-09-11T01:43:27.940 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 1022 (ObjNum 774 snap 236 seq_num 774) dirty exists 2024-09-11T01:43:27.941 INFO:tasks.rados.rados.0.smithi071.stdout:2140: left oid 36 (ObjNum 774 snap 236 seq_num 774) 2024-09-11T01:43:27.941 INFO:tasks.rados.rados.0.smithi071.stdout:2143: finishing copy_from racing read to smithi07134777-4 2024-09-11T01:43:27.941 INFO:tasks.rados.rados.0.smithi071.stdout:2142: expect (ObjNum 733 snap 226 seq_num 733) 2024-09-11T01:43:28.022 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 185103 2024-09-11T01:43:28.063 INFO:tasks.rados.rados.0.smithi071.stdout:2143: finishing copy_from to smithi07134777-4 2024-09-11T01:43:28.063 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 366 (ObjNum 631 snap 190 seq_num 631) dirty exists 2024-09-11T01:43:28.063 INFO:tasks.rados.rados.0.smithi071.stdout:2137: done (6 left) 2024-09-11T01:43:28.063 INFO:tasks.rados.rados.0.smithi071.stdout:2138: done (5 left) 2024-09-11T01:43:28.063 INFO:tasks.rados.rados.0.smithi071.stdout:2139: done (4 left) 2024-09-11T01:43:28.063 INFO:tasks.rados.rados.0.smithi071.stdout:2140: done (3 left) 2024-09-11T01:43:28.063 INFO:tasks.rados.rados.0.smithi071.stdout:2141: done (2 left) 2024-09-11T01:43:28.063 INFO:tasks.rados.rados.0.smithi071.stdout:2142: done (1 left) 2024-09-11T01:43:28.063 INFO:tasks.rados.rados.0.smithi071.stdout:2143: done (0 left) 2024-09-11T01:43:28.064 INFO:tasks.rados.rados.0.smithi071.stdout:2144: write (excl) oid 14 current snap is 236 2024-09-11T01:43:28.064 INFO:tasks.rados.rados.0.smithi071.stdout:2144: seq_num 775 ranges {625425=469628,1768133=640265} 2024-09-11T01:43:28.068 INFO:tasks.rados.rados.0.smithi071.stdout:2144: writing smithi07134777-14 from 625425 to 1095053 tid 1 2024-09-11T01:43:28.075 INFO:tasks.rados.rados.0.smithi071.stdout:2144: writing smithi07134777-14 from 1768133 to 2408398 tid 2 2024-09-11T01:43:28.076 INFO:tasks.rados.rados.0.smithi071.stdout:2145: write oid 20 current snap is 236 2024-09-11T01:43:28.076 INFO:tasks.rados.rados.0.smithi071.stdout:2144: finishing write tid 1 to smithi07134777-14 2024-09-11T01:43:28.076 INFO:tasks.rados.rados.0.smithi071.stdout:2145: seq_num 776 ranges {642232=661230,1942968=783923} 2024-09-11T01:43:28.083 INFO:tasks.rados.rados.0.smithi071.stdout:2145: writing smithi07134777-20 from 642232 to 1303462 tid 1 2024-09-11T01:43:28.091 INFO:tasks.rados.rados.0.smithi071.stdout:2145: writing smithi07134777-20 from 1942968 to 2726891 tid 2 2024-09-11T01:43:28.091 INFO:tasks.rados.rados.0.smithi071.stdout:2146: delete oid 4 current snap is 236 2024-09-11T01:43:28.091 INFO:tasks.rados.rados.0.smithi071.stdout:2144: finishing write tid 2 to smithi07134777-14 2024-09-11T01:43:28.091 INFO:tasks.rados.rados.0.smithi071.stdout:2144: finishing write tid 3 to smithi07134777-14 2024-09-11T01:43:28.091 INFO:tasks.rados.rados.0.smithi071.stdout:2144: finishing write tid 4 to smithi07134777-14 2024-09-11T01:43:28.091 INFO:tasks.rados.rados.0.smithi071.stdout:2144: oid 14 updating version 0 to 1262 2024-09-11T01:43:28.092 INFO:tasks.rados.rados.0.smithi071.stdout:2144: oid 14 version 1262 is already newer than 1261 2024-09-11T01:43:28.092 INFO:tasks.rados.rados.0.smithi071.stdout:2144: oid 14 updating version 1262 to 1263 2024-09-11T01:43:28.092 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 1263 (ObjNum 775 snap 236 seq_num 775) dirty exists 2024-09-11T01:43:28.092 INFO:tasks.rados.rados.0.smithi071.stdout:2144: left oid 14 (ObjNum 775 snap 236 seq_num 775) 2024-09-11T01:43:28.092 INFO:tasks.rados.rados.0.smithi071.stdout:2145: finishing write tid 1 to smithi07134777-20 2024-09-11T01:43:28.093 INFO:tasks.rados.rados.0.smithi071.stdout:2144: done (2 left) 2024-09-11T01:43:28.093 INFO:tasks.rados.rados.0.smithi071.stdout:2146: done (1 left) 2024-09-11T01:43:28.093 INFO:tasks.rados.rados.0.smithi071.stdout:2147: copy_from oid 29 from oid 27 current snap is 236 2024-09-11T01:43:28.093 INFO:tasks.rados.rados.0.smithi071.stdout:2148: write oid 28 current snap is 236 2024-09-11T01:43:28.094 INFO:tasks.rados.rados.0.smithi071.stdout:2148: seq_num 778 ranges {675846=411728,1543579=787592,2848853=226993} 2024-09-11T01:43:28.099 INFO:tasks.rados.rados.0.smithi071.stdout:2148: writing smithi07134777-28 from 675846 to 1087574 tid 1 2024-09-11T01:43:28.106 INFO:tasks.rados.rados.0.smithi071.stdout:2148: writing smithi07134777-28 from 1543579 to 2331171 tid 2 2024-09-11T01:43:28.110 INFO:tasks.rados.rados.0.smithi071.stdout:2148: writing smithi07134777-28 from 2848853 to 3075846 tid 3 2024-09-11T01:43:28.110 INFO:tasks.rados.rados.0.smithi071.stdout:2149: write oid 44 current snap is 236 2024-09-11T01:43:28.110 INFO:tasks.rados.rados.0.smithi071.stdout:2147: finishing copy_from to smithi07134777-29 2024-09-11T01:43:28.110 INFO:tasks.rados.rados.0.smithi071.stdout:2147: got expected ENOENT (src dne) 2024-09-11T01:43:28.111 INFO:tasks.rados.rados.0.smithi071.stdout:2149: seq_num 779 ranges {692653=486977,1602061=647603,2992629=100024} 2024-09-11T01:43:28.116 INFO:tasks.rados.rados.0.smithi071.stdout:2149: writing smithi07134777-44 from 692653 to 1179630 tid 1 2024-09-11T01:43:28.123 INFO:tasks.rados.rados.0.smithi071.stdout:2149: writing smithi07134777-44 from 1602061 to 2249664 tid 2 2024-09-11T01:43:28.128 INFO:tasks.rados.rados.0.smithi071.stdout:2149: writing smithi07134777-44 from 2992629 to 3092653 tid 3 2024-09-11T01:43:28.129 INFO:tasks.rados.rados.0.smithi071.stdout:2150: rollback oid 49 current snap is 236 2024-09-11T01:43:28.129 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 49 to 185 2024-09-11T01:43:28.129 INFO:tasks.rados.rados.0.smithi071.stdout:2147: finishing copy_from racing read to smithi07134777-29 2024-09-11T01:43:28.129 INFO:tasks.rados.rados.0.smithi071.stdout:2145: finishing write tid 2 to smithi07134777-20 2024-09-11T01:43:28.129 INFO:tasks.rados.rados.0.smithi071.stdout:2145: finishing write tid 3 to smithi07134777-20 2024-09-11T01:43:28.129 INFO:tasks.rados.rados.0.smithi071.stdout:2145: finishing write tid 4 to smithi07134777-20 2024-09-11T01:43:28.129 INFO:tasks.rados.rados.0.smithi071.stdout:2145: oid 20 updating version 0 to 981 2024-09-11T01:43:28.129 INFO:tasks.rados.rados.0.smithi071.stdout:2145: oid 20 updating version 981 to 982 2024-09-11T01:43:28.129 INFO:tasks.rados.rados.0.smithi071.stdout:2145: oid 20 version 982 is already newer than 980 2024-09-11T01:43:28.129 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 982 (ObjNum 776 snap 236 seq_num 776) dirty exists 2024-09-11T01:43:28.129 INFO:tasks.rados.rados.0.smithi071.stdout:2145: left oid 20 (ObjNum 776 snap 236 seq_num 776) 2024-09-11T01:43:28.129 INFO:tasks.rados.rados.0.smithi071.stdout:2148: finishing write tid 1 to smithi07134777-28 2024-09-11T01:43:28.129 INFO:tasks.rados.rados.0.smithi071.stdout:2148: finishing write tid 2 to smithi07134777-28 2024-09-11T01:43:28.130 INFO:tasks.rados.rados.0.smithi071.stdout:2145: done (4 left) 2024-09-11T01:43:28.130 INFO:tasks.rados.rados.0.smithi071.stdout:2147: done (3 left) 2024-09-11T01:43:28.130 INFO:tasks.rados.rados.0.smithi071.stdout:2151: copy_from oid 43 from oid 8 current snap is 236 2024-09-11T01:43:28.130 INFO:tasks.rados.rados.0.smithi071.stdout:2148: finishing write tid 3 to smithi07134777-28 2024-09-11T01:43:28.130 INFO:tasks.rados.rados.0.smithi071.stdout:2148: finishing write tid 4 to smithi07134777-28 2024-09-11T01:43:28.130 INFO:tasks.rados.rados.0.smithi071.stdout:2148: finishing write tid 5 to smithi07134777-28 2024-09-11T01:43:28.130 INFO:tasks.rados.rados.0.smithi071.stdout:2148: oid 28 updating version 0 to 968 2024-09-11T01:43:28.130 INFO:tasks.rados.rados.0.smithi071.stdout:2148: oid 28 updating version 968 to 970 2024-09-11T01:43:28.130 INFO:tasks.rados.rados.0.smithi071.stdout:2148: oid 28 version 970 is already newer than 969 2024-09-11T01:43:28.130 INFO:tasks.rados.rados.0.smithi071.stdout:2148: oid 28 updating version 970 to 971 2024-09-11T01:43:28.130 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 971 (ObjNum 778 snap 236 seq_num 778) dirty exists 2024-09-11T01:43:28.130 INFO:tasks.rados.rados.0.smithi071.stdout:2148: left oid 28 (ObjNum 778 snap 236 seq_num 778) 2024-09-11T01:43:28.130 INFO:tasks.rados.rados.0.smithi071.stdout:2149: finishing write tid 1 to smithi07134777-44 2024-09-11T01:43:28.130 INFO:tasks.rados.rados.0.smithi071.stdout:2149: finishing write tid 2 to smithi07134777-44 2024-09-11T01:43:28.131 INFO:tasks.rados.rados.0.smithi071.stdout:2148: done (3 left) 2024-09-11T01:43:28.131 INFO:tasks.rados.rados.0.smithi071.stdout:2152: rollback oid 46 current snap is 236 2024-09-11T01:43:28.131 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 46 to 208 2024-09-11T01:43:28.131 INFO:tasks.rados.rados.0.smithi071.stdout:2153: read oid 30 snap -1 2024-09-11T01:43:28.131 INFO:tasks.rados.rados.0.smithi071.stdout:2153: expect (ObjNum 765 snap 231 seq_num 765) 2024-09-11T01:43:28.131 INFO:tasks.rados.rados.0.smithi071.stdout:2154: snap_create 2024-09-11T01:43:28.132 INFO:tasks.rados.rados.0.smithi071.stdout:2150: finishing rollback tid 1 to smithi07134777-49 2024-09-11T01:43:28.132 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 367 (ObjNum 1201 snap 0 seq_num 2370546016) dirty dne 2024-09-11T01:43:28.132 INFO:tasks.rados.rados.0.smithi071.stdout:2149: finishing write tid 3 to smithi07134777-44 2024-09-11T01:43:28.132 INFO:tasks.rados.rados.0.smithi071.stdout:2149: finishing write tid 4 to smithi07134777-44 2024-09-11T01:43:28.132 INFO:tasks.rados.rados.0.smithi071.stdout:2149: finishing write tid 5 to smithi07134777-44 2024-09-11T01:43:28.132 INFO:tasks.rados.rados.0.smithi071.stdout:2149: oid 44 updating version 0 to 986 2024-09-11T01:43:28.132 INFO:tasks.rados.rados.0.smithi071.stdout:2149: oid 44 version 986 is already newer than 984 2024-09-11T01:43:28.132 INFO:tasks.rados.rados.0.smithi071.stdout:2149: oid 44 version 986 is already newer than 983 2024-09-11T01:43:28.132 INFO:tasks.rados.rados.0.smithi071.stdout:2149: oid 44 version 986 is already newer than 985 2024-09-11T01:43:28.132 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 986 (ObjNum 779 snap 236 seq_num 779) dirty exists 2024-09-11T01:43:28.132 INFO:tasks.rados.rados.0.smithi071.stdout:2149: left oid 44 (ObjNum 779 snap 236 seq_num 779) 2024-09-11T01:43:28.133 INFO:tasks.rados.rados.0.smithi071.stdout:2152: finishing rollback tid 0 to smithi07134777-46 2024-09-11T01:43:28.133 INFO:tasks.rados.rados.0.smithi071.stdout:2152: finishing rollback tid 1 to smithi07134777-46 2024-09-11T01:43:28.133 INFO:tasks.rados.rados.0.smithi071.stdout:2153: expect (ObjNum 765 snap 231 seq_num 765) 2024-09-11T01:43:28.142 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1597460 2024-09-11T01:43:28.205 INFO:tasks.rados.rados.0.smithi071.stdout:2152: finishing rollback tid 2 to smithi07134777-46 2024-09-11T01:43:28.205 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 1048 (ObjNum 691 snap 204 seq_num 691) dirty exists 2024-09-11T01:43:28.206 INFO:tasks.rados.rados.0.smithi071.stdout:2151: finishing copy_from racing read to smithi07134777-43 2024-09-11T01:43:28.206 INFO:tasks.rados.rados.0.smithi071.stdout:2151: finishing copy_from to smithi07134777-43 2024-09-11T01:43:28.206 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 703 (ObjNum 732 snap 225 seq_num 732) dirty exists 2024-09-11T01:43:28.786 INFO:tasks.rados.rados.0.smithi071.stdout:2149: done (5 left) 2024-09-11T01:43:28.786 INFO:tasks.rados.rados.0.smithi071.stdout:2150: done (4 left) 2024-09-11T01:43:28.786 INFO:tasks.rados.rados.0.smithi071.stdout:2151: done (3 left) 2024-09-11T01:43:28.786 INFO:tasks.rados.rados.0.smithi071.stdout:2152: done (2 left) 2024-09-11T01:43:28.786 INFO:tasks.rados.rados.0.smithi071.stdout:2153: done (1 left) 2024-09-11T01:43:28.786 INFO:tasks.rados.rados.0.smithi071.stdout:2154: done (0 left) 2024-09-11T01:43:28.787 INFO:tasks.rados.rados.0.smithi071.stdout:2155: read oid 32 snap 225 2024-09-11T01:43:28.787 INFO:tasks.rados.rados.0.smithi071.stdout:2155: expect (ObjNum 707 snap 215 seq_num 707) 2024-09-11T01:43:28.787 INFO:tasks.rados.rados.0.smithi071.stdout:2156: write (excl) oid 13 current snap is 237 2024-09-11T01:43:28.787 INFO:tasks.rados.rados.0.smithi071.stdout:2156: seq_num 780 ranges {709460=562226,2060543=791261} 2024-09-11T01:43:28.792 INFO:tasks.rados.rados.0.smithi071.stdout:2156: writing smithi07134777-13 from 709460 to 1271686 tid 1 2024-09-11T01:43:28.802 INFO:tasks.rados.rados.0.smithi071.stdout:2156: writing smithi07134777-13 from 2060543 to 2851804 tid 2 2024-09-11T01:43:28.802 INFO:tasks.rados.rados.0.smithi071.stdout:2157: delete oid 1 current snap is 237 2024-09-11T01:43:28.802 INFO:tasks.rados.rados.0.smithi071.stdout:2156: finishing write tid 1 to smithi07134777-13 2024-09-11T01:43:28.802 INFO:tasks.rados.rados.0.smithi071.stdout:2155: expect (ObjNum 707 snap 215 seq_num 707) 2024-09-11T01:43:28.988 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:29.080 INFO:tasks.rados.rados.0.smithi071.stdout:2156: finishing write tid 2 to smithi07134777-13 2024-09-11T01:43:29.080 INFO:tasks.rados.rados.0.smithi071.stdout:2156: finishing write tid 3 to smithi07134777-13 2024-09-11T01:43:29.080 INFO:tasks.rados.rados.0.smithi071.stdout:2156: finishing write tid 4 to smithi07134777-13 2024-09-11T01:43:29.080 INFO:tasks.rados.rados.0.smithi071.stdout:2156: oid 13 updating version 0 to 904 2024-09-11T01:43:29.081 INFO:tasks.rados.rados.0.smithi071.stdout:2156: oid 13 version 904 is already newer than 903 2024-09-11T01:43:29.081 INFO:tasks.rados.rados.0.smithi071.stdout:2156: oid 13 updating version 904 to 905 2024-09-11T01:43:29.081 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 905 (ObjNum 780 snap 237 seq_num 780) dirty exists 2024-09-11T01:43:29.081 INFO:tasks.rados.rados.0.smithi071.stdout:2156: left oid 13 (ObjNum 780 snap 237 seq_num 780) 2024-09-11T01:43:29.081 INFO:tasks.rados.rados.0.smithi071.stdout:2155: done (2 left) 2024-09-11T01:43:29.081 INFO:tasks.rados.rados.0.smithi071.stdout:2156: done (1 left) 2024-09-11T01:43:29.081 INFO:tasks.rados.rados.0.smithi071.stdout:2157: done (0 left) 2024-09-11T01:43:29.081 INFO:tasks.rados.rados.0.smithi071.stdout:2158: copy_from oid 40 from oid 1 current snap is 237 2024-09-11T01:43:29.081 INFO:tasks.rados.rados.0.smithi071.stdout:2159: delete oid 30 current snap is 237 2024-09-11T01:43:29.083 INFO:tasks.rados.rados.0.smithi071.stdout:2158: finishing copy_from to smithi07134777-40 2024-09-11T01:43:29.083 INFO:tasks.rados.rados.0.smithi071.stdout:2158: got expected ENOENT (src dne) 2024-09-11T01:43:29.083 INFO:tasks.rados.rados.0.smithi071.stdout:2158: finishing copy_from racing read to smithi07134777-40 2024-09-11T01:43:29.086 INFO:tasks.rados.rados.0.smithi071.stdout:2158: done (1 left) 2024-09-11T01:43:29.086 INFO:tasks.rados.rados.0.smithi071.stdout:2159: done (0 left) 2024-09-11T01:43:29.086 INFO:tasks.rados.rados.0.smithi071.stdout:2160: write (excl) oid 18 current snap is 237 2024-09-11T01:43:29.086 INFO:tasks.rados.rados.0.smithi071.stdout:2160: seq_num 783 ranges {759881=787973,2119636=654941} 2024-09-11T01:43:29.094 INFO:tasks.rados.rados.0.smithi071.stdout:2160: writing smithi07134777-18 from 759881 to 1547854 tid 1 2024-09-11T01:43:29.101 INFO:tasks.rados.rados.0.smithi071.stdout:2160: writing smithi07134777-18 from 2119636 to 2774577 tid 2 2024-09-11T01:43:29.101 INFO:tasks.rados.rados.0.smithi071.stdout:2161: delete oid 13 current snap is 237 2024-09-11T01:43:29.103 INFO:tasks.rados.rados.0.smithi071.stdout:2161: done (1 left) 2024-09-11T01:43:29.103 INFO:tasks.rados.rados.0.smithi071.stdout:2162: rollback oid 5 current snap is 237 2024-09-11T01:43:29.104 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 5 to 229 2024-09-11T01:43:29.104 INFO:tasks.rados.rados.0.smithi071.stdout:2163: delete oid 38 current snap is 237 2024-09-11T01:43:29.105 INFO:tasks.rados.rados.0.smithi071.stdout:2162: finishing rollback tid 1 to smithi07134777-5 2024-09-11T01:43:29.105 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 510 (ObjNum 273 snap 0 seq_num 1687737548) dirty dne 2024-09-11T01:43:29.106 INFO:tasks.rados.rados.0.smithi071.stdout:2160: finishing write tid 1 to smithi07134777-18 2024-09-11T01:43:29.106 INFO:tasks.rados.rados.0.smithi071.stdout:2162: done (2 left) 2024-09-11T01:43:29.106 INFO:tasks.rados.rados.0.smithi071.stdout:2163: done (1 left) 2024-09-11T01:43:29.106 INFO:tasks.rados.rados.0.smithi071.stdout:2164: read oid 1 snap 229 2024-09-11T01:43:29.106 INFO:tasks.rados.rados.0.smithi071.stdout:2164: expect deleted 2024-09-11T01:43:29.107 INFO:tasks.rados.rados.0.smithi071.stdout:2165: snap_create 2024-09-11T01:43:29.107 INFO:tasks.rados.rados.0.smithi071.stdout:2160: finishing write tid 2 to smithi07134777-18 2024-09-11T01:43:29.110 INFO:tasks.rados.rados.0.smithi071.stdout:2160: finishing write tid 3 to smithi07134777-18 2024-09-11T01:43:29.110 INFO:tasks.rados.rados.0.smithi071.stdout:2160: finishing write tid 4 to smithi07134777-18 2024-09-11T01:43:29.111 INFO:tasks.rados.rados.0.smithi071.stdout:2160: oid 18 updating version 0 to 972 2024-09-11T01:43:29.111 INFO:tasks.rados.rados.0.smithi071.stdout:2160: oid 18 version 972 is already newer than 971 2024-09-11T01:43:29.111 INFO:tasks.rados.rados.0.smithi071.stdout:2160: oid 18 updating version 972 to 973 2024-09-11T01:43:29.111 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 973 (ObjNum 783 snap 237 seq_num 783) dirty exists 2024-09-11T01:43:29.111 INFO:tasks.rados.rados.0.smithi071.stdout:2160: left oid 18 (ObjNum 783 snap 237 seq_num 783) 2024-09-11T01:43:29.990 INFO:tasks.rados.rados.0.smithi071.stdout:2160: done (2 left) 2024-09-11T01:43:29.990 INFO:tasks.rados.rados.0.smithi071.stdout:2164: done (1 left) 2024-09-11T01:43:29.990 INFO:tasks.rados.rados.0.smithi071.stdout:2165: done (0 left) 2024-09-11T01:43:29.990 INFO:tasks.rados.rados.0.smithi071.stdout:2166: delete oid 10 current snap is 238 2024-09-11T01:43:29.991 INFO:tasks.rados.rados.0.smithi071.stdout:2166: done (0 left) 2024-09-11T01:43:29.991 INFO:tasks.rados.rados.0.smithi071.stdout:2167: write oid 34 current snap is 238 2024-09-11T01:43:29.991 INFO:tasks.rados.rados.0.smithi071.stdout:2167: seq_num 787 ranges {427109=405322,1553564=545926,2579307=647802} 2024-09-11T01:43:29.995 INFO:tasks.rados.rados.0.smithi071.stdout:2167: writing smithi07134777-34 from 427109 to 832431 tid 1 2024-09-11T01:43:30.002 INFO:tasks.rados.rados.0.smithi071.stdout:2167: writing smithi07134777-34 from 1553564 to 2099490 tid 2 2024-09-11T01:43:30.008 INFO:tasks.rados.rados.0.smithi071.stdout:2167: writing smithi07134777-34 from 2579307 to 3227109 tid 3 2024-09-11T01:43:30.009 INFO:tasks.rados.rados.0.smithi071.stdout:2168: snap_create 2024-09-11T01:43:30.009 INFO:tasks.rados.rados.0.smithi071.stdout:2167: finishing write tid 1 to smithi07134777-34 2024-09-11T01:43:30.015 INFO:tasks.rados.rados.0.smithi071.stdout:2167: finishing write tid 2 to smithi07134777-34 2024-09-11T01:43:30.016 INFO:tasks.rados.rados.0.smithi071.stdout:2167: finishing write tid 3 to smithi07134777-34 2024-09-11T01:43:30.016 INFO:tasks.rados.rados.0.smithi071.stdout:2167: finishing write tid 4 to smithi07134777-34 2024-09-11T01:43:30.016 INFO:tasks.rados.rados.0.smithi071.stdout:2167: finishing write tid 5 to smithi07134777-34 2024-09-11T01:43:30.016 INFO:tasks.rados.rados.0.smithi071.stdout:2167: oid 34 updating version 0 to 1172 2024-09-11T01:43:30.017 INFO:tasks.rados.rados.0.smithi071.stdout:2167: oid 34 version 1172 is already newer than 1170 2024-09-11T01:43:30.017 INFO:tasks.rados.rados.0.smithi071.stdout:2167: oid 34 version 1172 is already newer than 1171 2024-09-11T01:43:30.017 INFO:tasks.rados.rados.0.smithi071.stdout:2167: oid 34 version 1172 is already newer than 1169 2024-09-11T01:43:30.017 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 1172 (ObjNum 787 snap 238 seq_num 787) dirty exists 2024-09-11T01:43:30.017 INFO:tasks.rados.rados.0.smithi071.stdout:2167: left oid 34 (ObjNum 787 snap 238 seq_num 787) 2024-09-11T01:43:30.997 INFO:tasks.rados.rados.0.smithi071.stdout:2167: done (1 left) 2024-09-11T01:43:30.998 INFO:tasks.rados.rados.0.smithi071.stdout:2168: done (0 left) 2024-09-11T01:43:30.998 INFO:tasks.rados.rados.0.smithi071.stdout:2169: snap_remove snap 208 2024-09-11T01:43:32.005 INFO:tasks.rados.rados.0.smithi071.stdout:2169: done (0 left) 2024-09-11T01:43:32.005 INFO:tasks.rados.rados.0.smithi071.stdout:2170: read oid 41 snap -1 2024-09-11T01:43:32.005 INFO:tasks.rados.rados.0.smithi071.stdout:2170: expect (ObjNum 753 snap 229 seq_num 753) 2024-09-11T01:43:32.005 INFO:tasks.rados.rados.0.smithi071.stdout:2171: read oid 23 snap -1 2024-09-11T01:43:32.005 INFO:tasks.rados.rados.0.smithi071.stdout:2171: expect (ObjNum 689 snap 204 seq_num 689) 2024-09-11T01:43:32.005 INFO:tasks.rados.rados.0.smithi071.stdout:2172: read oid 15 snap 213 2024-09-11T01:43:32.005 INFO:tasks.rados.rados.0.smithi071.stdout:2172: expect (ObjNum 565 snap 160 seq_num 565) 2024-09-11T01:43:32.005 INFO:tasks.rados.rados.0.smithi071.stdout:2173: snap_create 2024-09-11T01:43:32.015 INFO:tasks.rados.rados.0.smithi071.stdout:2171: expect (ObjNum 689 snap 204 seq_num 689) 2024-09-11T01:43:32.048 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3156907 2024-09-11T01:43:32.051 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 438531 2024-09-11T01:43:32.101 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:32.153 INFO:tasks.rados.rados.0.smithi071.stdout:2170: expect (ObjNum 753 snap 229 seq_num 753) 2024-09-11T01:43:32.198 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 190476 2024-09-11T01:43:32.241 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:32.284 INFO:tasks.rados.rados.0.smithi071.stdout:2172: expect (ObjNum 565 snap 160 seq_num 565) 2024-09-11T01:43:32.343 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3475688 2024-09-11T01:43:32.346 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:32.443 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1445344 2024-09-11T01:43:33.001 INFO:tasks.rados.rados.0.smithi071.stdout:2170: done (3 left) 2024-09-11T01:43:33.001 INFO:tasks.rados.rados.0.smithi071.stdout:2171: done (2 left) 2024-09-11T01:43:33.001 INFO:tasks.rados.rados.0.smithi071.stdout:2172: done (1 left) 2024-09-11T01:43:33.001 INFO:tasks.rados.rados.0.smithi071.stdout:2173: done (0 left) 2024-09-11T01:43:33.001 INFO:tasks.rados.rados.0.smithi071.stdout:2174: write (excl) oid 19 current snap is 240 2024-09-11T01:43:33.001 INFO:tasks.rados.rados.0.smithi071.stdout:2174: seq_num 788 ranges {443916=480571,1612046=405937,2606730=495052} 2024-09-11T01:43:33.005 INFO:tasks.rados.rados.0.smithi071.stdout:2174: writing smithi07134777-19 from 443916 to 924487 tid 1 2024-09-11T01:43:33.011 INFO:tasks.rados.rados.0.smithi071.stdout:2174: writing smithi07134777-19 from 1612046 to 2017983 tid 2 2024-09-11T01:43:33.017 INFO:tasks.rados.rados.0.smithi071.stdout:2174: writing smithi07134777-19 from 2606730 to 3101782 tid 3 2024-09-11T01:43:33.017 INFO:tasks.rados.rados.0.smithi071.stdout:2175: snap_create 2024-09-11T01:43:33.017 INFO:tasks.rados.rados.0.smithi071.stdout:2174: finishing write tid 1 to smithi07134777-19 2024-09-11T01:43:33.017 INFO:tasks.rados.rados.0.smithi071.stdout:2174: finishing write tid 2 to smithi07134777-19 2024-09-11T01:43:33.020 INFO:tasks.rados.rados.0.smithi071.stdout:2174: finishing write tid 3 to smithi07134777-19 2024-09-11T01:43:33.023 INFO:tasks.rados.rados.0.smithi071.stdout:2174: finishing write tid 4 to smithi07134777-19 2024-09-11T01:43:33.023 INFO:tasks.rados.rados.0.smithi071.stdout:2174: finishing write tid 5 to smithi07134777-19 2024-09-11T01:43:33.023 INFO:tasks.rados.rados.0.smithi071.stdout:2174: oid 19 updating version 0 to 1025 2024-09-11T01:43:33.023 INFO:tasks.rados.rados.0.smithi071.stdout:2174: oid 19 version 1025 is already newer than 1024 2024-09-11T01:43:33.023 INFO:tasks.rados.rados.0.smithi071.stdout:2174: oid 19 updating version 1025 to 1026 2024-09-11T01:43:33.023 INFO:tasks.rados.rados.0.smithi071.stdout:2174: oid 19 version 1026 is already newer than 1023 2024-09-11T01:43:33.023 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 1026 (ObjNum 788 snap 240 seq_num 788) dirty exists 2024-09-11T01:43:33.023 INFO:tasks.rados.rados.0.smithi071.stdout:2174: left oid 19 (ObjNum 788 snap 240 seq_num 788) 2024-09-11T01:43:33.978 INFO:tasks.rados.rados.0.smithi071.stdout:2174: done (1 left) 2024-09-11T01:43:33.978 INFO:tasks.rados.rados.0.smithi071.stdout:2175: done (0 left) 2024-09-11T01:43:33.978 INFO:tasks.rados.rados.0.smithi071.stdout:2176: snap_remove snap 234 2024-09-11T01:43:34.979 INFO:tasks.rados.rados.0.smithi071.stdout:2176: done (0 left) 2024-09-11T01:43:34.979 INFO:tasks.rados.rados.0.smithi071.stdout:2177: snap_remove snap 232 2024-09-11T01:43:35.986 INFO:tasks.rados.rados.0.smithi071.stdout:2177: done (0 left) 2024-09-11T01:43:35.987 INFO:tasks.rados.rados.0.smithi071.stdout:2178: delete oid 9 current snap is 241 2024-09-11T01:43:35.993 INFO:tasks.rados.rados.0.smithi071.stdout:2178: done (0 left) 2024-09-11T01:43:35.993 INFO:tasks.rados.rados.0.smithi071.stdout:2179: copy_from oid 5 from oid 29 current snap is 241 2024-09-11T01:43:35.993 INFO:tasks.rados.rados.0.smithi071.stdout:2180: delete oid 23 current snap is 241 2024-09-11T01:43:36.004 INFO:tasks.rados.rados.0.smithi071.stdout:2180: done (1 left) 2024-09-11T01:43:36.004 INFO:tasks.rados.rados.0.smithi071.stdout:2181: write oid 6 current snap is 241 2024-09-11T01:43:36.004 INFO:tasks.rados.rados.0.smithi071.stdout:2181: seq_num 791 ranges {494337=422671,1387492=553264,2688999=445221} 2024-09-11T01:43:36.009 INFO:tasks.rados.rados.0.smithi071.stdout:2181: writing smithi07134777-6 from 494337 to 917008 tid 1 2024-09-11T01:43:36.015 INFO:tasks.rados.rados.0.smithi071.stdout:2181: writing smithi07134777-6 from 1387492 to 1940756 tid 2 2024-09-11T01:43:36.021 INFO:tasks.rados.rados.0.smithi071.stdout:2181: writing smithi07134777-6 from 2688999 to 3134220 tid 3 2024-09-11T01:43:36.021 INFO:tasks.rados.rados.0.smithi071.stdout:2182: write (excl) oid 19 current snap is 241 2024-09-11T01:43:36.022 INFO:tasks.rados.rados.0.smithi071.stdout:2179: finishing copy_from racing read to smithi07134777-5 2024-09-11T01:43:36.022 INFO:tasks.rados.rados.0.smithi071.stdout:2182: seq_num 792 ranges {511144=497920,1445974=413275,2316422=656493} 2024-09-11T01:43:36.026 INFO:tasks.rados.rados.0.smithi071.stdout:2182: writing smithi07134777-19 from 511144 to 1009064 tid 1 2024-09-11T01:43:36.031 INFO:tasks.rados.rados.0.smithi071.stdout:2182: writing smithi07134777-19 from 1445974 to 1859249 tid 2 2024-09-11T01:43:36.037 INFO:tasks.rados.rados.0.smithi071.stdout:2182: writing smithi07134777-19 from 2316422 to 2972915 tid 3 2024-09-11T01:43:36.037 INFO:tasks.rados.rados.0.smithi071.stdout:2183: snap_remove snap 202 2024-09-11T01:43:36.993 INFO:tasks.rados.rados.0.smithi071.stdout:2183: done (3 left) 2024-09-11T01:43:36.993 INFO:tasks.rados.rados.0.smithi071.stdout:2184: rollback oid 3 current snap is 241 2024-09-11T01:43:36.996 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 3 to 229 2024-09-11T01:43:36.996 INFO:tasks.rados.rados.0.smithi071.stdout:2179: finishing copy_from to smithi07134777-5 2024-09-11T01:43:36.996 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 1152 (ObjNum 673 snap 204 seq_num 673) dirty exists 2024-09-11T01:43:36.996 INFO:tasks.rados.rados.0.smithi071.stdout:2181: finishing write tid 1 to smithi07134777-6 2024-09-11T01:43:36.996 INFO:tasks.rados.rados.0.smithi071.stdout:2181: finishing write tid 2 to smithi07134777-6 2024-09-11T01:43:36.996 INFO:tasks.rados.rados.0.smithi071.stdout:2181: finishing write tid 3 to smithi07134777-6 2024-09-11T01:43:36.996 INFO:tasks.rados.rados.0.smithi071.stdout:2181: finishing write tid 4 to smithi07134777-6 2024-09-11T01:43:36.996 INFO:tasks.rados.rados.0.smithi071.stdout:2181: finishing write tid 5 to smithi07134777-6 2024-09-11T01:43:36.996 INFO:tasks.rados.rados.0.smithi071.stdout:2181: oid 6 updating version 0 to 1175 2024-09-11T01:43:36.996 INFO:tasks.rados.rados.0.smithi071.stdout:2181: oid 6 version 1175 is already newer than 1174 2024-09-11T01:43:36.997 INFO:tasks.rados.rados.0.smithi071.stdout:2181: oid 6 version 1175 is already newer than 1173 2024-09-11T01:43:36.997 INFO:tasks.rados.rados.0.smithi071.stdout:2181: oid 6 updating version 1175 to 1176 2024-09-11T01:43:36.997 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 1176 (ObjNum 791 snap 241 seq_num 791) dirty exists 2024-09-11T01:43:36.997 INFO:tasks.rados.rados.0.smithi071.stdout:2181: left oid 6 (ObjNum 791 snap 241 seq_num 791) 2024-09-11T01:43:36.997 INFO:tasks.rados.rados.0.smithi071.stdout:2182: finishing write tid 1 to smithi07134777-19 2024-09-11T01:43:36.997 INFO:tasks.rados.rados.0.smithi071.stdout:2182: finishing write tid 2 to smithi07134777-19 2024-09-11T01:43:36.997 INFO:tasks.rados.rados.0.smithi071.stdout:2182: finishing write tid 3 to smithi07134777-19 2024-09-11T01:43:36.997 INFO:tasks.rados.rados.0.smithi071.stdout:2182: finishing write tid 4 to smithi07134777-19 2024-09-11T01:43:36.997 INFO:tasks.rados.rados.0.smithi071.stdout:2182: finishing write tid 5 to smithi07134777-19 2024-09-11T01:43:36.997 INFO:tasks.rados.rados.0.smithi071.stdout:2182: oid 19 updating version 0 to 1028 2024-09-11T01:43:36.997 INFO:tasks.rados.rados.0.smithi071.stdout:2182: oid 19 updating version 1028 to 1030 2024-09-11T01:43:36.997 INFO:tasks.rados.rados.0.smithi071.stdout:2182: oid 19 version 1030 is already newer than 1027 2024-09-11T01:43:36.998 INFO:tasks.rados.rados.0.smithi071.stdout:2182: oid 19 version 1030 is already newer than 1029 2024-09-11T01:43:36.998 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 1030 (ObjNum 792 snap 241 seq_num 792) dirty exists 2024-09-11T01:43:36.998 INFO:tasks.rados.rados.0.smithi071.stdout:2182: left oid 19 (ObjNum 792 snap 241 seq_num 792) 2024-09-11T01:43:36.998 INFO:tasks.rados.rados.0.smithi071.stdout:2179: done (3 left) 2024-09-11T01:43:36.998 INFO:tasks.rados.rados.0.smithi071.stdout:2181: done (2 left) 2024-09-11T01:43:36.998 INFO:tasks.rados.rados.0.smithi071.stdout:2182: done (1 left) 2024-09-11T01:43:36.998 INFO:tasks.rados.rados.0.smithi071.stdout:2185: snap_create 2024-09-11T01:43:36.999 INFO:tasks.rados.rados.0.smithi071.stdout:2184: finishing rollback tid 0 to smithi07134777-3 2024-09-11T01:43:37.001 INFO:tasks.rados.rados.0.smithi071.stdout:2184: finishing rollback tid 1 to smithi07134777-3 2024-09-11T01:43:37.001 INFO:tasks.rados.rados.0.smithi071.stdout:2184: finishing rollback tid 2 to smithi07134777-3 2024-09-11T01:43:37.001 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 713 (ObjNum 732 snap 225 seq_num 732) dirty exists 2024-09-11T01:43:38.204 INFO:tasks.rados.rados.0.smithi071.stdout:2184: done (1 left) 2024-09-11T01:43:38.205 INFO:tasks.rados.rados.0.smithi071.stdout:2185: done (0 left) 2024-09-11T01:43:38.205 INFO:tasks.rados.rados.0.smithi071.stdout:2186: snap_create 2024-09-11T01:43:38.979 INFO:tasks.rados.rados.0.smithi071.stdout:2186: done (0 left) 2024-09-11T01:43:38.979 INFO:tasks.rados.rados.0.smithi071.stdout:2187: snap_remove snap 185 2024-09-11T01:43:39.990 INFO:tasks.rados.rados.0.smithi071.stdout:2187: done (0 left) 2024-09-11T01:43:39.990 INFO:tasks.rados.rados.0.smithi071.stdout:2188: rollback oid 18 current snap is 243 2024-09-11T01:43:39.990 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 18 to 230 2024-09-11T01:43:39.990 INFO:tasks.rados.rados.0.smithi071.stdout:2189: read oid 49 snap -1 2024-09-11T01:43:39.990 INFO:tasks.rados.rados.0.smithi071.stdout:2189: expect deleted 2024-09-11T01:43:39.990 INFO:tasks.rados.rados.0.smithi071.stdout:2190: read oid 21 snap -1 2024-09-11T01:43:39.990 INFO:tasks.rados.rados.0.smithi071.stdout:2190: expect (ObjNum 738 snap 226 seq_num 738) 2024-09-11T01:43:39.991 INFO:tasks.rados.rados.0.smithi071.stdout:2191: write oid 45 current snap is 243 2024-09-11T01:43:39.991 INFO:tasks.rados.rados.0.smithi071.stdout:2191: seq_num 793 ranges {527951=573169,1504456=556933,2743845=467765} 2024-09-11T01:43:39.997 INFO:tasks.rados.rados.0.smithi071.stdout:2191: writing smithi07134777-45 from 527951 to 1101120 tid 1 2024-09-11T01:43:40.003 INFO:tasks.rados.rados.0.smithi071.stdout:2191: writing smithi07134777-45 from 1504456 to 2061389 tid 2 2024-09-11T01:43:40.010 INFO:tasks.rados.rados.0.smithi071.stdout:2191: writing smithi07134777-45 from 2743845 to 3211610 tid 3 2024-09-11T01:43:40.010 INFO:tasks.rados.rados.0.smithi071.stdout:2192: rollback oid 7 current snap is 243 2024-09-11T01:43:40.010 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 7 to 239 2024-09-11T01:43:40.010 INFO:tasks.rados.rados.0.smithi071.stdout:2190: expect (ObjNum 738 snap 226 seq_num 738) 2024-09-11T01:43:40.169 INFO:tasks.rados.rados.0.smithi071.stdout:2188: finishing rollback tid 0 to smithi07134777-18 2024-09-11T01:43:40.169 INFO:tasks.rados.rados.0.smithi071.stdout:2188: finishing rollback tid 1 to smithi07134777-18 2024-09-11T01:43:40.169 INFO:tasks.rados.rados.0.smithi071.stdout:2188: finishing rollback tid 2 to smithi07134777-18 2024-09-11T01:43:40.169 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 976 (ObjNum 724 snap 224 seq_num 724) dirty exists 2024-09-11T01:43:40.169 INFO:tasks.rados.rados.0.smithi071.stdout:2191: finishing write tid 1 to smithi07134777-45 2024-09-11T01:43:40.169 INFO:tasks.rados.rados.0.smithi071.stdout:2191: finishing write tid 2 to smithi07134777-45 2024-09-11T01:43:40.169 INFO:tasks.rados.rados.0.smithi071.stdout:2191: finishing write tid 3 to smithi07134777-45 2024-09-11T01:43:40.170 INFO:tasks.rados.rados.0.smithi071.stdout:2188: done (4 left) 2024-09-11T01:43:40.170 INFO:tasks.rados.rados.0.smithi071.stdout:2189: done (3 left) 2024-09-11T01:43:40.170 INFO:tasks.rados.rados.0.smithi071.stdout:2190: done (2 left) 2024-09-11T01:43:40.170 INFO:tasks.rados.rados.0.smithi071.stdout:2193: write (excl) oid 35 current snap is 243 2024-09-11T01:43:40.170 INFO:tasks.rados.rados.0.smithi071.stdout:2191: finishing write tid 4 to smithi07134777-45 2024-09-11T01:43:40.170 INFO:tasks.rados.rados.0.smithi071.stdout:2191: finishing write tid 5 to smithi07134777-45 2024-09-11T01:43:40.170 INFO:tasks.rados.rados.0.smithi071.stdout:2191: oid 45 updating version 0 to 975 2024-09-11T01:43:40.170 INFO:tasks.rados.rados.0.smithi071.stdout:2191: oid 45 version 975 is already newer than 974 2024-09-11T01:43:40.170 INFO:tasks.rados.rados.0.smithi071.stdout:2191: oid 45 version 975 is already newer than 972 2024-09-11T01:43:40.170 INFO:tasks.rados.rados.0.smithi071.stdout:2191: oid 45 version 975 is already newer than 973 2024-09-11T01:43:40.170 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 975 (ObjNum 793 snap 243 seq_num 793) dirty exists 2024-09-11T01:43:40.170 INFO:tasks.rados.rados.0.smithi071.stdout:2191: left oid 45 (ObjNum 793 snap 243 seq_num 793) 2024-09-11T01:43:40.170 INFO:tasks.rados.rados.0.smithi071.stdout:2192: finishing rollback tid 0 to smithi07134777-7 2024-09-11T01:43:40.171 INFO:tasks.rados.rados.0.smithi071.stdout:2192: finishing rollback tid 1 to smithi07134777-7 2024-09-11T01:43:40.171 INFO:tasks.rados.rados.0.smithi071.stdout:2193: seq_num 794 ranges {544758=648418,1846585=416944,2771268=573490} 2024-09-11T01:43:40.175 INFO:tasks.rados.rados.0.smithi071.stdout:2193: writing smithi07134777-35 from 544758 to 1193176 tid 1 2024-09-11T01:43:40.181 INFO:tasks.rados.rados.0.smithi071.stdout:2193: writing smithi07134777-35 from 1846585 to 2263529 tid 2 2024-09-11T01:43:40.187 INFO:tasks.rados.rados.0.smithi071.stdout:2193: writing smithi07134777-35 from 2771268 to 3344758 tid 3 2024-09-11T01:43:40.187 INFO:tasks.rados.rados.0.smithi071.stdout:2191: done (2 left) 2024-09-11T01:43:40.187 INFO:tasks.rados.rados.0.smithi071.stdout:2194: delete oid 8 current snap is 243 2024-09-11T01:43:40.187 INFO:tasks.rados.rados.0.smithi071.stdout:2192: finishing rollback tid 2 to smithi07134777-7 2024-09-11T01:43:40.187 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 1315 (ObjNum 769 snap 233 seq_num 769) dirty exists 2024-09-11T01:43:40.187 INFO:tasks.rados.rados.0.smithi071.stdout:2193: finishing write tid 1 to smithi07134777-35 2024-09-11T01:43:40.187 INFO:tasks.rados.rados.0.smithi071.stdout:2193: finishing write tid 2 to smithi07134777-35 2024-09-11T01:43:40.190 INFO:tasks.rados.rados.0.smithi071.stdout:2192: done (2 left) 2024-09-11T01:43:40.190 INFO:tasks.rados.rados.0.smithi071.stdout:2194: done (1 left) 2024-09-11T01:43:40.190 INFO:tasks.rados.rados.0.smithi071.stdout:2195: write oid 33 current snap is 243 2024-09-11T01:43:40.191 INFO:tasks.rados.rados.0.smithi071.stdout:2195: seq_num 796 ranges {578372=798916,1963549=420613,2826114=417934} 2024-09-11T01:43:40.198 INFO:tasks.rados.rados.0.smithi071.stdout:2195: writing smithi07134777-33 from 578372 to 1377288 tid 1 2024-09-11T01:43:40.203 INFO:tasks.rados.rados.0.smithi071.stdout:2195: writing smithi07134777-33 from 1963549 to 2384162 tid 2 2024-09-11T01:43:40.209 INFO:tasks.rados.rados.0.smithi071.stdout:2195: writing smithi07134777-33 from 2826114 to 3244048 tid 3 2024-09-11T01:43:40.209 INFO:tasks.rados.rados.0.smithi071.stdout:2196: rollback oid 8 current snap is 243 2024-09-11T01:43:40.209 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 8 to 236 2024-09-11T01:43:40.209 INFO:tasks.rados.rados.0.smithi071.stdout:2193: finishing write tid 3 to smithi07134777-35 2024-09-11T01:43:40.209 INFO:tasks.rados.rados.0.smithi071.stdout:2193: finishing write tid 4 to smithi07134777-35 2024-09-11T01:43:40.209 INFO:tasks.rados.rados.0.smithi071.stdout:2193: finishing write tid 5 to smithi07134777-35 2024-09-11T01:43:40.209 INFO:tasks.rados.rados.0.smithi071.stdout:2193: oid 35 updating version 0 to 994 2024-09-11T01:43:40.209 INFO:tasks.rados.rados.0.smithi071.stdout:2193: oid 35 version 994 is already newer than 992 2024-09-11T01:43:40.209 INFO:tasks.rados.rados.0.smithi071.stdout:2193: oid 35 updating version 994 to 995 2024-09-11T01:43:40.210 INFO:tasks.rados.rados.0.smithi071.stdout:2193: oid 35 version 995 is already newer than 993 2024-09-11T01:43:40.210 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 995 (ObjNum 794 snap 243 seq_num 794) dirty exists 2024-09-11T01:43:40.210 INFO:tasks.rados.rados.0.smithi071.stdout:2193: left oid 35 (ObjNum 794 snap 243 seq_num 794) 2024-09-11T01:43:40.210 INFO:tasks.rados.rados.0.smithi071.stdout:2195: finishing write tid 1 to smithi07134777-33 2024-09-11T01:43:40.210 INFO:tasks.rados.rados.0.smithi071.stdout:2195: finishing write tid 2 to smithi07134777-33 2024-09-11T01:43:40.210 INFO:tasks.rados.rados.0.smithi071.stdout:2193: done (2 left) 2024-09-11T01:43:40.210 INFO:tasks.rados.rados.0.smithi071.stdout:2197: write oid 22 current snap is 243 2024-09-11T01:43:40.210 INFO:tasks.rados.rados.0.smithi071.stdout:2197: seq_num 797 ranges {595179=474165,1622031=564271,2737184=629206} 2024-09-11T01:43:40.214 INFO:tasks.rados.rados.0.smithi071.stdout:2197: writing smithi07134777-22 from 595179 to 1069344 tid 1 2024-09-11T01:43:40.220 INFO:tasks.rados.rados.0.smithi071.stdout:2197: writing smithi07134777-22 from 1622031 to 2186302 tid 2 2024-09-11T01:43:40.227 INFO:tasks.rados.rados.0.smithi071.stdout:2197: writing smithi07134777-22 from 2737184 to 3366390 tid 3 2024-09-11T01:43:40.227 INFO:tasks.rados.rados.0.smithi071.stdout:2198: write oid 36 current snap is 243 2024-09-11T01:43:40.227 INFO:tasks.rados.rados.0.smithi071.stdout:2196: finishing rollback tid 1 to smithi07134777-8 2024-09-11T01:43:40.227 INFO:tasks.rados.rados.0.smithi071.stdout:2196: finishing rollback tid 2 to smithi07134777-8 2024-09-11T01:43:40.227 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 1033 (ObjNum 732 snap 225 seq_num 732) dirty exists 2024-09-11T01:43:40.228 INFO:tasks.rados.rados.0.smithi071.stdout:2195: finishing write tid 3 to smithi07134777-33 2024-09-11T01:43:40.228 INFO:tasks.rados.rados.0.smithi071.stdout:2195: finishing write tid 4 to smithi07134777-33 2024-09-11T01:43:40.228 INFO:tasks.rados.rados.0.smithi071.stdout:2195: finishing write tid 5 to smithi07134777-33 2024-09-11T01:43:40.228 INFO:tasks.rados.rados.0.smithi071.stdout:2195: oid 33 updating version 0 to 1178 2024-09-11T01:43:40.228 INFO:tasks.rados.rados.0.smithi071.stdout:2195: oid 33 updating version 1178 to 1179 2024-09-11T01:43:40.228 INFO:tasks.rados.rados.0.smithi071.stdout:2195: oid 33 updating version 1179 to 1180 2024-09-11T01:43:40.228 INFO:tasks.rados.rados.0.smithi071.stdout:2195: oid 33 version 1180 is already newer than 1177 2024-09-11T01:43:40.228 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 1180 (ObjNum 796 snap 243 seq_num 796) dirty exists 2024-09-11T01:43:40.228 INFO:tasks.rados.rados.0.smithi071.stdout:2195: left oid 33 (ObjNum 796 snap 243 seq_num 796) 2024-09-11T01:43:40.228 INFO:tasks.rados.rados.0.smithi071.stdout:2197: finishing write tid 1 to smithi07134777-22 2024-09-11T01:43:40.228 INFO:tasks.rados.rados.0.smithi071.stdout:2197: finishing write tid 2 to smithi07134777-22 2024-09-11T01:43:40.228 INFO:tasks.rados.rados.0.smithi071.stdout:2198: seq_num 798 ranges {611986=549414,1564160=424282,2764607=440478} 2024-09-11T01:43:40.233 INFO:tasks.rados.rados.0.smithi071.stdout:2198: writing smithi07134777-36 from 611986 to 1161400 tid 1 2024-09-11T01:43:40.237 INFO:tasks.rados.rados.0.smithi071.stdout:2198: writing smithi07134777-36 from 1564160 to 1988442 tid 2 2024-09-11T01:43:40.244 INFO:tasks.rados.rados.0.smithi071.stdout:2198: writing smithi07134777-36 from 2764607 to 3205085 tid 3 2024-09-11T01:43:40.244 INFO:tasks.rados.rados.0.smithi071.stdout:2195: done (3 left) 2024-09-11T01:43:40.244 INFO:tasks.rados.rados.0.smithi071.stdout:2196: done (2 left) 2024-09-11T01:43:40.244 INFO:tasks.rados.rados.0.smithi071.stdout:2199: snap_remove snap 233 2024-09-11T01:43:41.213 INFO:tasks.rados.rados.0.smithi071.stdout:2199: done (2 left) 2024-09-11T01:43:41.214 INFO:tasks.rados.rados.0.smithi071.stdout:2200: delete oid 28 current snap is 243 2024-09-11T01:43:41.214 INFO:tasks.rados.rados.0.smithi071.stdout:2197: finishing write tid 3 to smithi07134777-22 2024-09-11T01:43:41.214 INFO:tasks.rados.rados.0.smithi071.stdout:2197: finishing write tid 4 to smithi07134777-22 2024-09-11T01:43:41.214 INFO:tasks.rados.rados.0.smithi071.stdout:2197: finishing write tid 5 to smithi07134777-22 2024-09-11T01:43:41.214 INFO:tasks.rados.rados.0.smithi071.stdout:2197: oid 22 updating version 0 to 715 2024-09-11T01:43:41.214 INFO:tasks.rados.rados.0.smithi071.stdout:2197: oid 22 updating version 715 to 717 2024-09-11T01:43:41.214 INFO:tasks.rados.rados.0.smithi071.stdout:2197: oid 22 version 717 is already newer than 714 2024-09-11T01:43:41.214 INFO:tasks.rados.rados.0.smithi071.stdout:2197: oid 22 version 717 is already newer than 716 2024-09-11T01:43:41.215 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 717 (ObjNum 797 snap 243 seq_num 797) dirty exists 2024-09-11T01:43:41.215 INFO:tasks.rados.rados.0.smithi071.stdout:2197: left oid 22 (ObjNum 797 snap 243 seq_num 797) 2024-09-11T01:43:41.215 INFO:tasks.rados.rados.0.smithi071.stdout:2198: finishing write tid 1 to smithi07134777-36 2024-09-11T01:43:41.215 INFO:tasks.rados.rados.0.smithi071.stdout:2198: finishing write tid 2 to smithi07134777-36 2024-09-11T01:43:41.215 INFO:tasks.rados.rados.0.smithi071.stdout:2198: finishing write tid 3 to smithi07134777-36 2024-09-11T01:43:41.215 INFO:tasks.rados.rados.0.smithi071.stdout:2198: finishing write tid 4 to smithi07134777-36 2024-09-11T01:43:41.216 INFO:tasks.rados.rados.0.smithi071.stdout:2198: finishing write tid 5 to smithi07134777-36 2024-09-11T01:43:41.216 INFO:tasks.rados.rados.0.smithi071.stdout:2198: oid 36 updating version 0 to 1036 2024-09-11T01:43:41.216 INFO:tasks.rados.rados.0.smithi071.stdout:2198: oid 36 version 1036 is already newer than 1034 2024-09-11T01:43:41.216 INFO:tasks.rados.rados.0.smithi071.stdout:2198: oid 36 updating version 1036 to 1037 2024-09-11T01:43:41.216 INFO:tasks.rados.rados.0.smithi071.stdout:2198: oid 36 version 1037 is already newer than 1035 2024-09-11T01:43:41.217 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 1037 (ObjNum 798 snap 243 seq_num 798) dirty exists 2024-09-11T01:43:41.217 INFO:tasks.rados.rados.0.smithi071.stdout:2198: left oid 36 (ObjNum 798 snap 243 seq_num 798) 2024-09-11T01:43:41.227 INFO:tasks.rados.rados.0.smithi071.stdout:2197: done (2 left) 2024-09-11T01:43:41.227 INFO:tasks.rados.rados.0.smithi071.stdout:2198: done (1 left) 2024-09-11T01:43:41.227 INFO:tasks.rados.rados.0.smithi071.stdout:2200: done (0 left) 2024-09-11T01:43:41.227 INFO:tasks.rados.rados.0.smithi071.stdout:2201: write (excl) oid 32 current snap is 243 2024-09-11T01:43:41.227 INFO:tasks.rados.rados.0.smithi071.stdout:2201: seq_num 800 ranges {645600=416265,1797477=711598,3219453=226147} 2024-09-11T01:43:41.232 INFO:tasks.rados.rados.0.smithi071.stdout:2201: writing smithi07134777-32 from 645600 to 1061865 tid 1 2024-09-11T01:43:41.240 INFO:tasks.rados.rados.0.smithi071.stdout:2201: writing smithi07134777-32 from 1797477 to 2509075 tid 2 2024-09-11T01:43:41.245 INFO:tasks.rados.rados.0.smithi071.stdout:2201: writing smithi07134777-32 from 3219453 to 3445600 tid 3 2024-09-11T01:43:41.245 INFO:tasks.rados.rados.0.smithi071.stdout:2202: read oid 24 snap -1 2024-09-11T01:43:41.246 INFO:tasks.rados.rados.0.smithi071.stdout:2202: expect (ObjNum 766 snap 232 seq_num 766) 2024-09-11T01:43:41.246 INFO:tasks.rados.rados.0.smithi071.stdout:2203: write oid 29 current snap is 243 2024-09-11T01:43:41.246 INFO:tasks.rados.rados.0.smithi071.stdout:2201: finishing write tid 1 to smithi07134777-32 2024-09-11T01:43:41.246 INFO:tasks.rados.rados.0.smithi071.stdout:2203: seq_num 801 ranges {662407=491514,1855959=571609,2846876=615531} 2024-09-11T01:43:41.252 INFO:tasks.rados.rados.0.smithi071.stdout:2203: writing smithi07134777-29 from 662407 to 1153921 tid 1 2024-09-11T01:43:41.259 INFO:tasks.rados.rados.0.smithi071.stdout:2203: writing smithi07134777-29 from 1855959 to 2427568 tid 2 2024-09-11T01:43:41.265 INFO:tasks.rados.rados.0.smithi071.stdout:2203: writing smithi07134777-29 from 2846876 to 3462407 tid 3 2024-09-11T01:43:41.265 INFO:tasks.rados.rados.0.smithi071.stdout:2204: write (excl) oid 14 current snap is 243 2024-09-11T01:43:41.265 INFO:tasks.rados.rados.0.smithi071.stdout:2201: finishing write tid 2 to smithi07134777-32 2024-09-11T01:43:41.265 INFO:tasks.rados.rados.0.smithi071.stdout:2201: finishing write tid 3 to smithi07134777-32 2024-09-11T01:43:41.265 INFO:tasks.rados.rados.0.smithi071.stdout:2201: finishing write tid 4 to smithi07134777-32 2024-09-11T01:43:41.265 INFO:tasks.rados.rados.0.smithi071.stdout:2204: seq_num 802 ranges {679214=566763,1798088=715267,3157946=321268} 2024-09-11T01:43:41.272 INFO:tasks.rados.rados.0.smithi071.stdout:2204: writing smithi07134777-14 from 679214 to 1245977 tid 1 2024-09-11T01:43:41.279 INFO:tasks.rados.rados.0.smithi071.stdout:2204: writing smithi07134777-14 from 1798088 to 2513355 tid 2 2024-09-11T01:43:41.285 INFO:tasks.rados.rados.0.smithi071.stdout:2204: writing smithi07134777-14 from 3157946 to 3479214 tid 3 2024-09-11T01:43:41.285 INFO:tasks.rados.rados.0.smithi071.stdout:2205: copy_from oid 30 from oid 18 current snap is 243 2024-09-11T01:43:41.285 INFO:tasks.rados.rados.0.smithi071.stdout:2201: finishing write tid 5 to smithi07134777-32 2024-09-11T01:43:41.285 INFO:tasks.rados.rados.0.smithi071.stdout:2201: oid 32 updating version 0 to 999 2024-09-11T01:43:41.285 INFO:tasks.rados.rados.0.smithi071.stdout:2201: oid 32 version 999 is already newer than 996 2024-09-11T01:43:41.285 INFO:tasks.rados.rados.0.smithi071.stdout:2201: oid 32 version 999 is already newer than 997 2024-09-11T01:43:41.285 INFO:tasks.rados.rados.0.smithi071.stdout:2201: oid 32 version 999 is already newer than 998 2024-09-11T01:43:41.285 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 999 (ObjNum 800 snap 243 seq_num 800) dirty exists 2024-09-11T01:43:41.285 INFO:tasks.rados.rados.0.smithi071.stdout:2201: left oid 32 (ObjNum 800 snap 243 seq_num 800) 2024-09-11T01:43:41.286 INFO:tasks.rados.rados.0.smithi071.stdout:2202: expect (ObjNum 766 snap 232 seq_num 766) 2024-09-11T01:43:41.291 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 770005 2024-09-11T01:43:41.377 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:41.431 INFO:tasks.rados.rados.0.smithi071.stdout:2203: finishing write tid 1 to smithi07134777-29 2024-09-11T01:43:41.431 INFO:tasks.rados.rados.0.smithi071.stdout:2203: finishing write tid 2 to smithi07134777-29 2024-09-11T01:43:41.431 INFO:tasks.rados.rados.0.smithi071.stdout:2203: finishing write tid 3 to smithi07134777-29 2024-09-11T01:43:41.432 INFO:tasks.rados.rados.0.smithi071.stdout:2203: finishing write tid 4 to smithi07134777-29 2024-09-11T01:43:41.432 INFO:tasks.rados.rados.0.smithi071.stdout:2203: finishing write tid 5 to smithi07134777-29 2024-09-11T01:43:41.432 INFO:tasks.rados.rados.0.smithi071.stdout:2203: oid 29 updating version 0 to 1325 2024-09-11T01:43:41.432 INFO:tasks.rados.rados.0.smithi071.stdout:2203: oid 29 updating version 1325 to 1326 2024-09-11T01:43:41.432 INFO:tasks.rados.rados.0.smithi071.stdout:2203: oid 29 updating version 1326 to 1327 2024-09-11T01:43:41.432 INFO:tasks.rados.rados.0.smithi071.stdout:2203: oid 29 updating version 1327 to 1328 2024-09-11T01:43:41.432 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 1328 (ObjNum 801 snap 243 seq_num 801) dirty exists 2024-09-11T01:43:41.433 INFO:tasks.rados.rados.0.smithi071.stdout:2203: left oid 29 (ObjNum 801 snap 243 seq_num 801) 2024-09-11T01:43:41.433 INFO:tasks.rados.rados.0.smithi071.stdout:2204: finishing write tid 1 to smithi07134777-14 2024-09-11T01:43:41.433 INFO:tasks.rados.rados.0.smithi071.stdout:2204: finishing write tid 2 to smithi07134777-14 2024-09-11T01:43:41.433 INFO:tasks.rados.rados.0.smithi071.stdout:2204: finishing write tid 3 to smithi07134777-14 2024-09-11T01:43:41.433 INFO:tasks.rados.rados.0.smithi071.stdout:2204: finishing write tid 4 to smithi07134777-14 2024-09-11T01:43:41.433 INFO:tasks.rados.rados.0.smithi071.stdout:2201: done (4 left) 2024-09-11T01:43:41.433 INFO:tasks.rados.rados.0.smithi071.stdout:2202: done (3 left) 2024-09-11T01:43:41.433 INFO:tasks.rados.rados.0.smithi071.stdout:2203: done (2 left) 2024-09-11T01:43:41.433 INFO:tasks.rados.rados.0.smithi071.stdout:2206: delete oid 40 current snap is 243 2024-09-11T01:43:41.434 INFO:tasks.rados.rados.0.smithi071.stdout:2204: finishing write tid 5 to smithi07134777-14 2024-09-11T01:43:41.434 INFO:tasks.rados.rados.0.smithi071.stdout:2204: oid 14 updating version 0 to 1332 2024-09-11T01:43:41.434 INFO:tasks.rados.rados.0.smithi071.stdout:2204: oid 14 version 1332 is already newer than 1330 2024-09-11T01:43:41.434 INFO:tasks.rados.rados.0.smithi071.stdout:2204: oid 14 version 1332 is already newer than 1331 2024-09-11T01:43:41.434 INFO:tasks.rados.rados.0.smithi071.stdout:2204: oid 14 updating version 1332 to 1333 2024-09-11T01:43:41.434 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 1333 (ObjNum 802 snap 243 seq_num 802) dirty exists 2024-09-11T01:43:41.434 INFO:tasks.rados.rados.0.smithi071.stdout:2204: left oid 14 (ObjNum 802 snap 243 seq_num 802) 2024-09-11T01:43:41.434 INFO:tasks.rados.rados.0.smithi071.stdout:2205: finishing copy_from racing read to smithi07134777-30 2024-09-11T01:43:41.434 INFO:tasks.rados.rados.0.smithi071.stdout:2205: finishing copy_from to smithi07134777-30 2024-09-11T01:43:41.434 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 976 (ObjNum 724 snap 224 seq_num 724) dirty exists 2024-09-11T01:43:41.435 INFO:tasks.rados.rados.0.smithi071.stdout:2204: done (2 left) 2024-09-11T01:43:41.435 INFO:tasks.rados.rados.0.smithi071.stdout:2205: done (1 left) 2024-09-11T01:43:41.435 INFO:tasks.rados.rados.0.smithi071.stdout:2206: done (0 left) 2024-09-11T01:43:41.435 INFO:tasks.rados.rados.0.smithi071.stdout:2207: read oid 12 snap 237 2024-09-11T01:43:41.435 INFO:tasks.rados.rados.0.smithi071.stdout:2207: expect deleted 2024-09-11T01:43:41.435 INFO:tasks.rados.rados.0.smithi071.stdout:2208: write (excl) oid 28 current snap is 243 2024-09-11T01:43:41.435 INFO:tasks.rados.rados.0.smithi071.stdout:2208: seq_num 804 ranges {712828=717261,1915052=718936,3212792=300036} 2024-09-11T01:43:41.441 INFO:tasks.rados.rados.0.smithi071.stdout:2208: writing smithi07134777-28 from 712828 to 1430089 tid 1 2024-09-11T01:43:41.447 INFO:tasks.rados.rados.0.smithi071.stdout:2208: writing smithi07134777-28 from 1915052 to 2633988 tid 2 2024-09-11T01:43:41.453 INFO:tasks.rados.rados.0.smithi071.stdout:2208: writing smithi07134777-28 from 3212792 to 3512828 tid 3 2024-09-11T01:43:41.453 INFO:tasks.rados.rados.0.smithi071.stdout:2209: write (excl) oid 13 current snap is 243 2024-09-11T01:43:41.453 INFO:tasks.rados.rados.0.smithi071.stdout:2209: seq_num 805 ranges {729635=792510,1973534=578947,3240215=289420} 2024-09-11T01:43:41.461 INFO:tasks.rados.rados.0.smithi071.stdout:2209: writing smithi07134777-13 from 729635 to 1522145 tid 1 2024-09-11T01:43:41.466 INFO:tasks.rados.rados.0.smithi071.stdout:2209: writing smithi07134777-13 from 1973534 to 2552481 tid 2 2024-09-11T01:43:41.472 INFO:tasks.rados.rados.0.smithi071.stdout:2209: writing smithi07134777-13 from 3240215 to 3529635 tid 3 2024-09-11T01:43:41.472 INFO:tasks.rados.rados.0.smithi071.stdout:2210: read oid 31 snap -1 2024-09-11T01:43:41.473 INFO:tasks.rados.rados.0.smithi071.stdout:2210: expect (ObjNum 737 snap 226 seq_num 737) 2024-09-11T01:43:41.473 INFO:tasks.rados.rados.0.smithi071.stdout:2211: snap_create 2024-09-11T01:43:41.473 INFO:tasks.rados.rados.0.smithi071.stdout:2208: finishing write tid 1 to smithi07134777-28 2024-09-11T01:43:41.473 INFO:tasks.rados.rados.0.smithi071.stdout:2208: finishing write tid 2 to smithi07134777-28 2024-09-11T01:43:41.473 INFO:tasks.rados.rados.0.smithi071.stdout:2208: finishing write tid 3 to smithi07134777-28 2024-09-11T01:43:41.473 INFO:tasks.rados.rados.0.smithi071.stdout:2208: finishing write tid 4 to smithi07134777-28 2024-09-11T01:43:41.473 INFO:tasks.rados.rados.0.smithi071.stdout:2208: finishing write tid 5 to smithi07134777-28 2024-09-11T01:43:41.473 INFO:tasks.rados.rados.0.smithi071.stdout:2208: oid 28 updating version 0 to 979 2024-09-11T01:43:41.473 INFO:tasks.rados.rados.0.smithi071.stdout:2208: oid 28 updating version 979 to 980 2024-09-11T01:43:41.473 INFO:tasks.rados.rados.0.smithi071.stdout:2208: oid 28 version 980 is already newer than 978 2024-09-11T01:43:41.473 INFO:tasks.rados.rados.0.smithi071.stdout:2208: oid 28 version 980 is already newer than 977 2024-09-11T01:43:41.473 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 980 (ObjNum 804 snap 243 seq_num 804) dirty exists 2024-09-11T01:43:41.474 INFO:tasks.rados.rados.0.smithi071.stdout:2208: left oid 28 (ObjNum 804 snap 243 seq_num 804) 2024-09-11T01:43:41.474 INFO:tasks.rados.rados.0.smithi071.stdout:2209: finishing write tid 1 to smithi07134777-13 2024-09-11T01:43:41.474 INFO:tasks.rados.rados.0.smithi071.stdout:2209: finishing write tid 2 to smithi07134777-13 2024-09-11T01:43:41.477 INFO:tasks.rados.rados.0.smithi071.stdout:2210: expect (ObjNum 737 snap 226 seq_num 737) 2024-09-11T01:43:41.477 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 40903 2024-09-11T01:43:41.520 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 567046 2024-09-11T01:43:41.562 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1262062 2024-09-11T01:43:41.588 INFO:tasks.rados.rados.0.smithi071.stdout:2209: finishing write tid 3 to smithi07134777-13 2024-09-11T01:43:41.588 INFO:tasks.rados.rados.0.smithi071.stdout:2209: finishing write tid 4 to smithi07134777-13 2024-09-11T01:43:41.588 INFO:tasks.rados.rados.0.smithi071.stdout:2209: finishing write tid 5 to smithi07134777-13 2024-09-11T01:43:41.588 INFO:tasks.rados.rados.0.smithi071.stdout:2209: oid 13 updating version 0 to 943 2024-09-11T01:43:41.588 INFO:tasks.rados.rados.0.smithi071.stdout:2209: oid 13 updating version 943 to 945 2024-09-11T01:43:41.589 INFO:tasks.rados.rados.0.smithi071.stdout:2209: oid 13 version 945 is already newer than 944 2024-09-11T01:43:41.589 INFO:tasks.rados.rados.0.smithi071.stdout:2209: oid 13 version 945 is already newer than 942 2024-09-11T01:43:41.589 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 945 (ObjNum 805 snap 243 seq_num 805) dirty exists 2024-09-11T01:43:41.589 INFO:tasks.rados.rados.0.smithi071.stdout:2209: left oid 13 (ObjNum 805 snap 243 seq_num 805) 2024-09-11T01:43:42.214 INFO:tasks.rados.rados.0.smithi071.stdout:2207: done (4 left) 2024-09-11T01:43:42.214 INFO:tasks.rados.rados.0.smithi071.stdout:2208: done (3 left) 2024-09-11T01:43:42.214 INFO:tasks.rados.rados.0.smithi071.stdout:2209: done (2 left) 2024-09-11T01:43:42.214 INFO:tasks.rados.rados.0.smithi071.stdout:2210: done (1 left) 2024-09-11T01:43:42.214 INFO:tasks.rados.rados.0.smithi071.stdout:2211: done (0 left) 2024-09-11T01:43:42.214 INFO:tasks.rados.rados.0.smithi071.stdout:2212: copy_from oid 14 from oid 22 current snap is 244 2024-09-11T01:43:42.214 INFO:tasks.rados.rados.0.smithi071.stdout:2213: write (excl) oid 42 current snap is 244 2024-09-11T01:43:42.214 INFO:tasks.rados.rados.0.smithi071.stdout:2213: seq_num 806 ranges {746442=584112,1748369=722605,2983991=562451} 2024-09-11T01:43:42.223 INFO:tasks.rados.rados.0.smithi071.stdout:2213: writing smithi07134777-42 from 746442 to 1330554 tid 1 2024-09-11T01:43:42.243 INFO:tasks.rados.rados.0.smithi071.stdout:2213: writing smithi07134777-42 from 1748369 to 2470974 tid 2 2024-09-11T01:43:42.246 INFO:tasks.rados.rados.0.smithi071.stdout:2213: writing smithi07134777-42 from 2983991 to 3546442 tid 3 2024-09-11T01:43:42.246 INFO:tasks.rados.rados.0.smithi071.stdout:2214: copy_from oid 2 from oid 50 current snap is 244 2024-09-11T01:43:42.247 INFO:tasks.rados.rados.0.smithi071.stdout:2213: finishing write tid 1 to smithi07134777-42 2024-09-11T01:43:42.247 INFO:tasks.rados.rados.0.smithi071.stdout:2215: delete oid 11 current snap is 244 2024-09-11T01:43:42.247 INFO:tasks.rados.rados.0.smithi071.stdout:2215: done (3 left) 2024-09-11T01:43:42.247 INFO:tasks.rados.rados.0.smithi071.stdout:2216: snap_create 2024-09-11T01:43:42.247 INFO:tasks.rados.rados.0.smithi071.stdout:2212: finishing copy_from racing read to smithi07134777-14 2024-09-11T01:43:42.247 INFO:tasks.rados.rados.0.smithi071.stdout:2212: finishing copy_from to smithi07134777-14 2024-09-11T01:43:42.247 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 717 (ObjNum 797 snap 243 seq_num 797) dirty exists 2024-09-11T01:43:42.247 INFO:tasks.rados.rados.0.smithi071.stdout:2213: finishing write tid 2 to smithi07134777-42 2024-09-11T01:43:42.247 INFO:tasks.rados.rados.0.smithi071.stdout:2213: finishing write tid 3 to smithi07134777-42 2024-09-11T01:43:42.247 INFO:tasks.rados.rados.0.smithi071.stdout:2213: finishing write tid 4 to smithi07134777-42 2024-09-11T01:43:42.248 INFO:tasks.rados.rados.0.smithi071.stdout:2213: finishing write tid 5 to smithi07134777-42 2024-09-11T01:43:42.248 INFO:tasks.rados.rados.0.smithi071.stdout:2213: oid 42 updating version 0 to 847 2024-09-11T01:43:42.248 INFO:tasks.rados.rados.0.smithi071.stdout:2213: oid 42 version 847 is already newer than 845 2024-09-11T01:43:42.248 INFO:tasks.rados.rados.0.smithi071.stdout:2213: oid 42 version 847 is already newer than 844 2024-09-11T01:43:42.248 INFO:tasks.rados.rados.0.smithi071.stdout:2213: oid 42 version 847 is already newer than 846 2024-09-11T01:43:42.248 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 847 (ObjNum 806 snap 244 seq_num 806) dirty exists 2024-09-11T01:43:42.248 INFO:tasks.rados.rados.0.smithi071.stdout:2213: left oid 42 (ObjNum 806 snap 244 seq_num 806) 2024-09-11T01:43:42.249 INFO:tasks.rados.rados.0.smithi071.stdout:2214: finishing copy_from racing read to smithi07134777-2 2024-09-11T01:43:42.249 INFO:tasks.rados.rados.0.smithi071.stdout:2214: finishing copy_from to smithi07134777-2 2024-09-11T01:43:42.249 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 366 (ObjNum 631 snap 190 seq_num 631) dirty exists 2024-09-11T01:43:43.226 INFO:tasks.rados.rados.0.smithi071.stdout:2212: done (3 left) 2024-09-11T01:43:43.226 INFO:tasks.rados.rados.0.smithi071.stdout:2213: done (2 left) 2024-09-11T01:43:43.226 INFO:tasks.rados.rados.0.smithi071.stdout:2214: done (1 left) 2024-09-11T01:43:43.227 INFO:tasks.rados.rados.0.smithi071.stdout:2216: done (0 left) 2024-09-11T01:43:43.227 INFO:tasks.rados.rados.0.smithi071.stdout:2217: snap_create 2024-09-11T01:43:43.891 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-11T01:43:43.891 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-11T01:43:43.892 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-11T01:43:43.894 INFO:tasks.ceph.mon.c:Started 2024-09-11T01:43:43.894 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:43:43.895 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:43:43.980 INFO:tasks.rados.rados.0.smithi071.stdout:2217: done (0 left) 2024-09-11T01:43:43.981 INFO:tasks.rados.rados.0.smithi071.stdout:2218: read oid 14 snap -1 2024-09-11T01:43:43.981 INFO:tasks.rados.rados.0.smithi071.stdout:2218: expect (ObjNum 797 snap 243 seq_num 797) 2024-09-11T01:43:43.981 INFO:tasks.rados.rados.0.smithi071.stdout:2219: write oid 13 current snap is 246 2024-09-11T01:43:43.981 INFO:tasks.rados.rados.0.smithi071.stdout:2219: seq_num 808 ranges {780056=734610,2148980=726274,3322484=257572} 2024-09-11T01:43:43.988 INFO:tasks.rados.rados.0.smithi071.stdout:2219: writing smithi07134777-13 from 780056 to 1514666 tid 1 2024-09-11T01:43:43.996 INFO:tasks.rados.rados.0.smithi071.stdout:2219: writing smithi07134777-13 from 2148980 to 2875254 tid 2 2024-09-11T01:43:44.001 INFO:tasks.rados.rados.0.smithi071.stdout:2219: writing smithi07134777-13 from 3322484 to 3580056 tid 3 2024-09-11T01:43:44.001 INFO:tasks.rados.rados.0.smithi071.stdout:2220: rollback oid 23 current snap is 246 2024-09-11T01:43:44.001 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 23 to 206 2024-09-11T01:43:44.001 INFO:tasks.rados.rados.0.smithi071.stdout:2218: expect (ObjNum 797 snap 243 seq_num 797) 2024-09-11T01:43:44.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.007+0000 7f188d447640 10 monclient: get_monmap_and_config 2024-09-11T01:43:44.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.007+0000 7f188d447640 10 monclient: build_initial_monmap 2024-09-11T01:43:44.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.007+0000 7f188d447640 10 monclient: monmap: 2024-09-11T01:43:44.009 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:44.009 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:44.009 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:43:44.008201+0000 2024-09-11T01:43:44.009 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:43:44.008201+0000 2024-09-11T01:43:44.009 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:44.009 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:44.009 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:43:44.009 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:43:44.009 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:43:44.009 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:43:44.009 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:43:44.009 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:44.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f188d447640 1 Processor -- start 2024-09-11T01:43:44.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f188d447640 1 -- start start 2024-09-11T01:43:44.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f188d447640 10 monclient: init 2024-09-11T01:43:44.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f188d447640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:44.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f188d447640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-11T01:43:44.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f188d447640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1888138230 0x7f1888138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:44.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f188d447640 10 monclient(hunting): picked mon.noname-b con 0x7f1888138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:44.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f188d447640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f188809d030 addr v1:172.21.15.12:6789/0 2024-09-11T01:43:44.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f188d447640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f188809d430 0x7f18880a5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:44.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f188d447640 10 monclient(hunting): picked mon.noname-a con 0x7f188809d430 addr v2:172.21.15.12:3300/0 2024-09-11T01:43:44.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f188d447640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18880a6e30 con 0x7f188809d030 2024-09-11T01:43:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f188d447640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f188d447640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f188809c9c0 con 0x7f188809d430 2024-09-11T01:43:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f188d447640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f188d447640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f188809ba20 con 0x7f1888138230 2024-09-11T01:43:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f188d447640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f188d447640 10 monclient(hunting): authenticate will time out at 1630.450678s 2024-09-11T01:43:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f1886ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1888138230 0x7f1888138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f18877fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f188809d430 0x7f18880a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f18867fc640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f188809d030 0x7f18880a5020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59476/0 (socket says 172.21.15.12:59476) 2024-09-11T01:43:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f18867fc640 1 -- 172.21.15.12:0/3558121362 learned_addr learned my addr 172.21.15.12:0/3558121362 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:43:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f18877fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f188809d430 0x7f18880a5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53184/0 (socket says 172.21.15.12:53184) 2024-09-11T01:43:44.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f1886ffd640 10 monclient(hunting): get_auth_request con 0x7f1888138230 auth_method 0 2024-09-11T01:43:44.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f1886ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:44.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f1886ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:44.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f1886ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:44.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f18877fe640 10 monclient(hunting): get_auth_request con 0x7f188809d430 auth_method 0 2024-09-11T01:43:44.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f18877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:44.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f18877fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:44.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f18877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:44.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f18877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:44.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f18877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:44.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f1886ffd640 1 -- 172.21.15.12:0/3558121362 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1888138230 msgr2=0x7f1888138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:43:44.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f18877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:44.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f1886ffd640 1 -- 172.21.15.12:0/3558121362 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1888138230 msgr2=0x7f1888138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:43:44.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f1886ffd640 1 --2- 172.21.15.12:0/3558121362 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1888138230 0x7f1888138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:44.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.008+0000 7f1886ffd640 1 --2- 172.21.15.12:0/3558121362 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1888138230 0x7f1888138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:44.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.009+0000 7f18877fe640 10 monclient(hunting): handle_auth_done global_id 94746 payload 274 2024-09-11T01:43:44.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.009+0000 7f18877fe640 1 -- 172.21.15.12:0/3558121362 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1888138230 msgr2=0x7f1888138610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:43:44.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.009+0000 7f18877fe640 1 --2- 172.21.15.12:0/3558121362 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1888138230 0x7f1888138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:44.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.009+0000 7f18877fe640 1 -- 172.21.15.12:0/3558121362 >> v1:172.21.15.12:6789/0 conn(0x7f188809d030 legacy=0x7f18880a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:44.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.009+0000 7f18877fe640 10 monclient: _finish_hunting 0 2024-09-11T01:43:44.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.009+0000 7f18877fe640 1 monclient: found mon.noname-a 2024-09-11T01:43:44.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.009+0000 7f18877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:44.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.009+0000 7f18877fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:44.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.009+0000 7f18877fe640 1 -- 172.21.15.12:0/3558121362 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18880a66a0 con 0x7f188809d430 2024-09-11T01:43:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.012+0000 7f18877fe640 1 -- 172.21.15.12:0/3558121362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f188809d430 msgr2=0x7f18880a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.012+0000 7f18877fe640 1 --2- 172.21.15.12:0/3558121362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f188809d430 0x7f18880a5730 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f188809cf80 tx=0x7f188002a460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45c180c62ed5d97 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.012+0000 7f1885ffb640 1 -- 172.21.15.12:0/3558121362 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1880007950 con 0x7f188809d430 2024-09-11T01:43:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.012+0000 7f1885ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.012+0000 7f1885ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.012+0000 7f1885ffb640 10 monclient: dump: 2024-09-11T01:43:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:44.021 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:44.022 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.012+0000 7f1885ffb640 10 monclient: _finish_auth 0 2024-09-11T01:43:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.012+0000 7f1885ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:43:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.012+0000 7f1885ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.012+0000 7f188d447640 5 monclient: authenticate success, global_id 94746 2024-09-11T01:43:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.012+0000 7f1885ffb640 1 -- 172.21.15.12:0/3558121362 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1880007af0 con 0x7f188809d430 2024-09-11T01:43:44.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.012+0000 7f188d447640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:43:44.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.012+0000 7f1885ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:44.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.012+0000 7f188d447640 10 monclient: get_monmap_and_config success 2024-09-11T01:43:44.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.012+0000 7f188d447640 4 set_mon_vals no callback set 2024-09-11T01:43:44.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.012+0000 7f1885ffb640 1 -- 172.21.15.12:0/3558121362 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1880031400 con 0x7f188809d430 2024-09-11T01:43:44.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.012+0000 7f1885ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:44.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.012+0000 7f1885ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:44.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.012+0000 7f1885ffb640 10 monclient: dump: 2024-09-11T01:43:44.023 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:44.023 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:44.023 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:44.023 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:44.023 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:44.023 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:44.023 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:44.024 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:44.024 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:44.024 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:44.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.012+0000 7f188d447640 10 monclient: shutdown 2024-09-11T01:43:44.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.012+0000 7f188d447640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:44.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.012+0000 7f188d447640 1 -- 172.21.15.12:0/3558121362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f188809d430 msgr2=0x7f18880a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:44.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.012+0000 7f188d447640 1 --2- 172.21.15.12:0/3558121362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f188809d430 0x7f18880a5730 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f188809cf80 tx=0x7f188002a460 comp rx=0 tx=0).stop 2024-09-11T01:43:44.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.013+0000 7f188d447640 1 -- 172.21.15.12:0/3558121362 shutdown_connections 2024-09-11T01:43:44.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.013+0000 7f188d447640 1 --2- 172.21.15.12:0/3558121362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f188809d430 0x7f18880a5730 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:44.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.013+0000 7f188d447640 1 --2- 172.21.15.12:0/3558121362 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1888138230 0x7f1888138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:44.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.013+0000 7f188d447640 1 -- 172.21.15.12:0/3558121362 >> 172.21.15.12:0/3558121362 conn(0x7f18880a7120 msgr2=0x7f18880a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:44.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.013+0000 7f188d447640 1 -- 172.21.15.12:0/3558121362 shutdown_connections 2024-09-11T01:43:44.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.013+0000 7f188d447640 1 -- 172.21.15.12:0/3558121362 wait complete. 2024-09-11T01:43:44.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.013+0000 7f188d447640 10 monclient: build_initial_monmap 2024-09-11T01:43:44.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.013+0000 7f188d447640 10 monclient: monmap: 2024-09-11T01:43:44.025 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:44.025 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:43:44.025 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:43:44.025 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:43:44.025 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:44.025 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:44.025 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:43:44.025 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:43:44.025 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:43:44.025 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:44.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f188d447640 1 Processor -- start 2024-09-11T01:43:44.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f188d447640 1 -- start start 2024-09-11T01:43:44.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f188d447640 10 monclient: init 2024-09-11T01:43:44.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f188d447640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:44.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f188d447640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:43:44.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f188d447640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f188809d030 0x7f18881368f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:44.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f188d447640 10 monclient(hunting): picked mon.noname-b con 0x7f188809d030 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:44.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f188d447640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f188809d430 0x7f1888136e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:44.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f188d447640 10 monclient(hunting): picked mon.noname-c con 0x7f188809d430 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:44.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f188d447640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1888138230 0x7f1888120c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:44.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f188d447640 10 monclient(hunting): picked mon.noname-a con 0x7f1888138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:44.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f188d447640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:44.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f188d447640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f188806f970 con 0x7f1888138230 2024-09-11T01:43:44.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f188d447640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:44.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f188d447640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f188806f7f0 con 0x7f188809d030 2024-09-11T01:43:44.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f188d447640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:44.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f188d447640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f18880a5c70 con 0x7f188809d430 2024-09-11T01:43:44.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f188d447640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:44.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f188d447640 10 monclient(hunting): authenticate will time out at 1630.456720s 2024-09-11T01:43:44.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f18877fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1888138230 0x7f1888120c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:44.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f18867fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f188809d430 0x7f1888136e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:44.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f18877fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1888138230 0x7f1888120c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53192/0 (socket says 172.21.15.12:53192) 2024-09-11T01:43:44.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f18877fe640 1 -- 172.21.15.12:0/458190295 learned_addr learned my addr 172.21.15.12:0/458190295 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:44.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f18877fe640 10 monclient(hunting): get_auth_request con 0x7f1888138230 auth_method 0 2024-09-11T01:43:44.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f18877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:44.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f18877fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:44.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f18877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:44.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f18867fc640 10 monclient(hunting): get_auth_request con 0x7f188809d430 auth_method 0 2024-09-11T01:43:44.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f18867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:44.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f18867fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:44.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f18867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:44.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f1886ffd640 1 --2- 172.21.15.12:0/458190295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f188809d030 0x7f18881368f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:44.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f18877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:44.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f18877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:44.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.014+0000 7f18867fc640 1 -- 172.21.15.12:0/458190295 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f188809d430 msgr2=0x7f1888136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:43:44.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.015+0000 7f18867fc640 1 -- 172.21.15.12:0/458190295 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f188809d430 msgr2=0x7f1888136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:43:44.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.015+0000 7f18867fc640 1 --2- 172.21.15.12:0/458190295 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f188809d430 0x7f1888136e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:44.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.015+0000 7f18877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:44.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.015+0000 7f1886ffd640 10 monclient(hunting): get_auth_request con 0x7f188809d030 auth_method 0 2024-09-11T01:43:44.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.015+0000 7f1886ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:44.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.015+0000 7f1886ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:44.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.015+0000 7f1886ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:44.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.015+0000 7f18867fc640 1 --2- 172.21.15.12:0/458190295 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f188809d430 0x7f1888136e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:44.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.015+0000 7f1886ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:44.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.015+0000 7f1886ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:44.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.015+0000 7f1886ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:44.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.015+0000 7f18877fe640 10 monclient(hunting): handle_auth_done global_id 94749 payload 824 2024-09-11T01:43:44.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.015+0000 7f18877fe640 1 -- 172.21.15.12:0/458190295 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f188809d430 msgr2=0x7f1888136e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:43:44.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.015+0000 7f18877fe640 1 --2- 172.21.15.12:0/458190295 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f188809d430 0x7f1888136e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:44.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.015+0000 7f18877fe640 1 -- 172.21.15.12:0/458190295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f188809d030 msgr2=0x7f18881368f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:44.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.015+0000 7f18877fe640 1 --2- 172.21.15.12:0/458190295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f188809d030 0x7f18881368f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:44.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.015+0000 7f18877fe640 10 monclient: _finish_hunting 0 2024-09-11T01:43:44.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.015+0000 7f18877fe640 1 monclient: found mon.noname-a 2024-09-11T01:43:44.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.015+0000 7f18877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:44.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.015+0000 7f18877fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:44.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.015+0000 7f18877fe640 1 -- 172.21.15.12:0/458190295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18881223d0 con 0x7f1888138230 2024-09-11T01:43:44.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.017+0000 7f18877fe640 1 -- 172.21.15.12:0/458190295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1888138230 msgr2=0x7f1888120c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:44.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.017+0000 7f18877fe640 1 --2- 172.21.15.12:0/458190295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1888138230 0x7f1888120c70 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f188002aa50 tx=0x7f1880004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:44.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.017+0000 7f186f7fe640 1 -- 172.21.15.12:0/458190295 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1880040070 con 0x7f1888138230 2024-09-11T01:43:44.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.017+0000 7f186f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:44.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.017+0000 7f186f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:44.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.017+0000 7f186f7fe640 10 monclient: dump: 2024-09-11T01:43:44.031 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:44.031 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:44.031 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:44.031 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:44.031 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:44.031 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:44.031 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:44.031 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:44.031 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:44.031 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:44.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.017+0000 7f186f7fe640 10 monclient: _finish_auth 0 2024-09-11T01:43:44.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.017+0000 7f186f7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:43:44.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.017+0000 7f186f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:44.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.017+0000 7f186f7fe640 1 -- 172.21.15.12:0/458190295 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1880045660 con 0x7f1888138230 2024-09-11T01:43:44.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.017+0000 7f186f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:44.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.017+0000 7f186f7fe640 1 -- 172.21.15.12:0/458190295 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18800314c0 con 0x7f1888138230 2024-09-11T01:43:44.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.017+0000 7f186f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:44.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.017+0000 7f186f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:44.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.017+0000 7f186f7fe640 10 monclient: dump: 2024-09-11T01:43:44.032 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:44.033 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:44.033 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:44.033 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:44.033 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:44.033 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:44.033 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:44.033 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:44.034 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:44.034 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:44.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.017+0000 7f188d447640 5 monclient: authenticate success, global_id 94749 2024-09-11T01:43:44.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.017+0000 7f18857fa640 4 set_mon_vals no callback set 2024-09-11T01:43:44.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.017+0000 7f188d447640 10 monclient: _renew_subs 2024-09-11T01:43:44.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.017+0000 7f188d447640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:44.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.017+0000 7f188d447640 1 -- 172.21.15.12:0/458190295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1888123e90 con 0x7f1888138230 2024-09-11T01:43:44.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.018+0000 7f188d447640 10 monclient: _renew_subs 2024-09-11T01:43:44.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.018+0000 7f188d447640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:44.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.018+0000 7f188d447640 1 -- 172.21.15.12:0/458190295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1888137370 con 0x7f1888138230 2024-09-11T01:43:44.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.019+0000 7f188d447640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:44.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.019+0000 7f188d447640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:44.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.019+0000 7f188d447640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:44.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.019+0000 7f188d447640 1 -- 172.21.15.12:0/458190295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1888138610 con 0x7f1888138230 2024-09-11T01:43:44.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.019+0000 7f186f7fe640 1 -- 172.21.15.12:0/458190295 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f1880007960 con 0x7f1888138230 2024-09-11T01:43:44.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.019+0000 7f186f7fe640 1 --2- 172.21.15.12:0/458190295 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f187403e210 0x7f18740406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:44.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.019+0000 7f186f7fe640 1 -- 172.21.15.12:0/458190295 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(497..497 src has 317..497) ==== 4815+0+0 (secure 0 0 0) 0x7f1880074ae0 con 0x7f1888138230 2024-09-11T01:43:44.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.019+0000 7f1886ffd640 1 --2- 172.21.15.12:0/458190295 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f187403e210 0x7f18740406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:44.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.019+0000 7f1886ffd640 10 monclient: get_auth_request con 0x7f187403e210 auth_method 0 2024-09-11T01:43:44.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.022+0000 7f1886ffd640 1 --2- 172.21.15.12:0/458190295 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f187403e210 0x7f18740406d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1878006fd0 tx=0x7f1878008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:44.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.024+0000 7f186f7fe640 1 -- 172.21.15.12:0/458190295 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f188003b3b0 con 0x7f1888138230 2024-09-11T01:43:44.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.024+0000 7f186f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:44.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.024+0000 7f186f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:43:44.044 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:44.097 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:44.149 INFO:tasks.rados.rados.0.smithi071.stdout:2219: finishing write tid 1 to smithi07134777-13 2024-09-11T01:43:44.150 INFO:tasks.rados.rados.0.smithi071.stdout:2219: finishing write tid 2 to smithi07134777-13 2024-09-11T01:43:44.150 INFO:tasks.rados.rados.0.smithi071.stdout:2219: finishing write tid 3 to smithi07134777-13 2024-09-11T01:43:44.150 INFO:tasks.rados.rados.0.smithi071.stdout:2219: finishing write tid 4 to smithi07134777-13 2024-09-11T01:43:44.150 INFO:tasks.rados.rados.0.smithi071.stdout:2219: finishing write tid 5 to smithi07134777-13 2024-09-11T01:43:44.150 INFO:tasks.rados.rados.0.smithi071.stdout:2219: oid 13 updating version 0 to 949 2024-09-11T01:43:44.150 INFO:tasks.rados.rados.0.smithi071.stdout:2219: oid 13 updating version 949 to 950 2024-09-11T01:43:44.150 INFO:tasks.rados.rados.0.smithi071.stdout:2219: oid 13 version 950 is already newer than 948 2024-09-11T01:43:44.150 INFO:tasks.rados.rados.0.smithi071.stdout:2219: oid 13 version 950 is already newer than 947 2024-09-11T01:43:44.150 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 950 (ObjNum 808 snap 246 seq_num 808) dirty exists 2024-09-11T01:43:44.150 INFO:tasks.rados.rados.0.smithi071.stdout:2219: left oid 13 (ObjNum 808 snap 246 seq_num 808) 2024-09-11T01:43:44.150 INFO:tasks.rados.rados.0.smithi071.stdout:2220: finishing rollback tid 1 to smithi07134777-23 2024-09-11T01:43:44.150 INFO:tasks.rados.rados.0.smithi071.stdout:2220: finishing rollback tid 2 to smithi07134777-23 2024-09-11T01:43:44.150 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 989 (ObjNum 652 snap 199 seq_num 652) dirty exists 2024-09-11T01:43:44.150 INFO:tasks.rados.rados.0.smithi071.stdout:2218: done (2 left) 2024-09-11T01:43:44.151 INFO:tasks.rados.rados.0.smithi071.stdout:2219: done (1 left) 2024-09-11T01:43:44.151 INFO:tasks.rados.rados.0.smithi071.stdout:2220: done (0 left) 2024-09-11T01:43:44.151 INFO:tasks.rados.rados.0.smithi071.stdout:2221: read oid 16 snap 174 2024-09-11T01:43:44.151 INFO:tasks.rados.rados.0.smithi071.stdout:2221: expect (ObjNum 411 snap 115 seq_num 411) 2024-09-11T01:43:44.151 INFO:tasks.rados.rados.0.smithi071.stdout:2222: delete oid 35 current snap is 246 2024-09-11T01:43:44.154 INFO:tasks.rados.rados.0.smithi071.stdout:2222: done (1 left) 2024-09-11T01:43:44.155 INFO:tasks.rados.rados.0.smithi071.stdout:2223: read oid 2 snap -1 2024-09-11T01:43:44.155 INFO:tasks.rados.rados.0.smithi071.stdout:2223: expect (ObjNum 631 snap 190 seq_num 631) 2024-09-11T01:43:44.155 INFO:tasks.rados.rados.0.smithi071.stdout:2224: rollback oid 8 current snap is 246 2024-09-11T01:43:44.155 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 8 to 228 2024-09-11T01:43:44.155 INFO:tasks.rados.rados.0.smithi071.stdout:2225: copy_from oid 6 from oid 23 current snap is 246 2024-09-11T01:43:44.155 INFO:tasks.rados.rados.0.smithi071.stdout:2226: copy_from oid 17 from oid 25 current snap is 246 2024-09-11T01:43:44.155 INFO:tasks.rados.rados.0.smithi071.stdout:2227: read oid 48 snap -1 2024-09-11T01:43:44.155 INFO:tasks.rados.rados.0.smithi071.stdout:2227: expect (ObjNum 565 snap 160 seq_num 565) 2024-09-11T01:43:44.155 INFO:tasks.rados.rados.0.smithi071.stdout:2228: copy_from oid 39 from oid 28 current snap is 246 2024-09-11T01:43:44.155 INFO:tasks.rados.rados.0.smithi071.stdout:2229: read oid 42 snap -1 2024-09-11T01:43:44.156 INFO:tasks.rados.rados.0.smithi071.stdout:2229: expect (ObjNum 806 snap 244 seq_num 806) 2024-09-11T01:43:44.156 INFO:tasks.rados.rados.0.smithi071.stdout:2230: write oid 3 current snap is 246 2024-09-11T01:43:44.156 INFO:tasks.rados.rados.0.smithi071.stdout:2230: seq_num 810 ranges {413670=485108,1465944=729943,2977330=524801} 2024-09-11T01:43:44.160 INFO:tasks.rados.rados.0.smithi071.stdout:2230: writing smithi07134777-3 from 413670 to 898778 tid 1 2024-09-11T01:43:44.168 INFO:tasks.rados.rados.0.smithi071.stdout:2230: writing smithi07134777-3 from 1465944 to 2195887 tid 2 2024-09-11T01:43:44.176 INFO:tasks.rados.rados.0.smithi071.stdout:2230: writing smithi07134777-3 from 2977330 to 3502131 tid 3 2024-09-11T01:43:44.176 INFO:tasks.rados.rados.0.smithi071.stdout:2231: write oid 34 current snap is 246 2024-09-11T01:43:44.176 INFO:tasks.rados.rados.0.smithi071.stdout:2231: seq_num 811 ranges {430477=560357,1408073=473601,2488400=736073} 2024-09-11T01:43:44.181 INFO:tasks.rados.rados.0.smithi071.stdout:2231: writing smithi07134777-34 from 430477 to 990834 tid 1 2024-09-11T01:43:44.186 INFO:tasks.rados.rados.0.smithi071.stdout:2231: writing smithi07134777-34 from 1408073 to 1881674 tid 2 2024-09-11T01:43:44.194 INFO:tasks.rados.rados.0.smithi071.stdout:2231: writing smithi07134777-34 from 2488400 to 3224473 tid 3 2024-09-11T01:43:44.194 INFO:tasks.rados.rados.0.smithi071.stdout:2232: write oid 30 current snap is 246 2024-09-11T01:43:44.194 INFO:tasks.rados.rados.0.smithi071.stdout:2232: seq_num 812 ranges {447284=635606,1866555=733612,3315823=331461} 2024-09-11T01:43:44.199 INFO:tasks.rados.rados.0.smithi071.stdout:2232: writing smithi07134777-30 from 447284 to 1082890 tid 1 2024-09-11T01:43:44.207 INFO:tasks.rados.rados.0.smithi071.stdout:2232: writing smithi07134777-30 from 1866555 to 2600167 tid 2 2024-09-11T01:43:44.213 INFO:tasks.rados.rados.0.smithi071.stdout:2232: writing smithi07134777-30 from 3315823 to 3647284 tid 3 2024-09-11T01:43:44.213 INFO:tasks.rados.rados.0.smithi071.stdout:2233: copy_from oid 40 from oid 14 current snap is 246 2024-09-11T01:43:44.213 INFO:tasks.rados.rados.0.smithi071.stdout:2223: expect (ObjNum 631 snap 190 seq_num 631) 2024-09-11T01:43:44.262 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1197531 2024-09-11T01:43:44.277 INFO:tasks.rados.rados.0.smithi071.stdout:2227: expect (ObjNum 565 snap 160 seq_num 565) 2024-09-11T01:43:44.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.298+0000 7f188d447640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:43:44.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.298+0000 7f188d447640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:43:44.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.298+0000 7f188d447640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:44.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.298+0000 7f188d447640 1 -- 172.21.15.12:0/458190295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f18880698e0 con 0x7f1888138230 2024-09-11T01:43:44.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.298+0000 7f186f7fe640 1 -- 172.21.15.12:0/458190295 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f188004fe10 con 0x7f1888138230 2024-09-11T01:43:44.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.298+0000 7f186f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:43:44.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.298+0000 7f186f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:43:44.300 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":132,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:43:44.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.300+0000 7f188d447640 1 -- 172.21.15.12:0/458190295 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f187403e210 msgr2=0x7f18740406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:44.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.300+0000 7f188d447640 1 --2- 172.21.15.12:0/458190295 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f187403e210 0x7f18740406d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1878006fd0 tx=0x7f1878008040 comp rx=0 tx=0).stop 2024-09-11T01:43:44.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.300+0000 7f188d447640 10 monclient: shutdown 2024-09-11T01:43:44.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.300+0000 7f188d447640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:44.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.300+0000 7f188d447640 1 -- 172.21.15.12:0/458190295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1888138230 msgr2=0x7f1888120c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:44.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.300+0000 7f188d447640 1 --2- 172.21.15.12:0/458190295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1888138230 0x7f1888120c70 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f188002aa50 tx=0x7f1880004a40 comp rx=0 tx=0).stop 2024-09-11T01:43:44.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.300+0000 7f188d447640 1 -- 172.21.15.12:0/458190295 shutdown_connections 2024-09-11T01:43:44.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.300+0000 7f188d447640 1 --2- 172.21.15.12:0/458190295 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f187403e210 0x7f18740406d0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:44.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.300+0000 7f188d447640 1 --2- 172.21.15.12:0/458190295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1888138230 0x7f1888120c70 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:44.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.300+0000 7f188d447640 1 --2- 172.21.15.12:0/458190295 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f188809d430 0x7f1888136e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:44.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.300+0000 7f188d447640 1 --2- 172.21.15.12:0/458190295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f188809d030 0x7f18881368f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:44.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.300+0000 7f188d447640 1 -- 172.21.15.12:0/458190295 >> 172.21.15.12:0/458190295 conn(0x7f18880a7120 msgr2=0x7f18880a2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:44.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.300+0000 7f188d447640 1 -- 172.21.15.12:0/458190295 shutdown_connections 2024-09-11T01:43:44.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:44.300+0000 7f188d447640 1 -- 172.21.15.12:0/458190295 wait complete. 2024-09-11T01:43:44.421 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 570745 2024-09-11T01:43:44.493 INFO:tasks.rados.rados.0.smithi071.stdout:2224: finishing rollback tid 0 to smithi07134777-8 2024-09-11T01:43:44.494 INFO:tasks.rados.rados.0.smithi071.stdout:2221: expect (ObjNum 411 snap 115 seq_num 411) 2024-09-11T01:43:44.494 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 418668 2024-09-11T01:43:44.613 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2500004 2024-09-11T01:43:44.617 INFO:tasks.rados.rados.0.smithi071.stdout:2224: finishing rollback tid 1 to smithi07134777-8 2024-09-11T01:43:44.617 INFO:tasks.rados.rados.0.smithi071.stdout:2226: finishing copy_from racing read to smithi07134777-17 2024-09-11T01:43:44.617 INFO:tasks.rados.rados.0.smithi071.stdout:2229: expect (ObjNum 806 snap 244 seq_num 806) 2024-09-11T01:43:44.617 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:44.779 INFO:tasks.rados.rados.0.smithi071.stdout:2226: finishing copy_from to smithi07134777-17 2024-09-11T01:43:44.780 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 979 (ObjNum 723 snap 224 seq_num 723) dirty exists 2024-09-11T01:43:44.780 INFO:tasks.rados.rados.0.smithi071.stdout:2224: finishing rollback tid 2 to smithi07134777-8 2024-09-11T01:43:44.780 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 1040 (ObjNum 736 snap 226 seq_num 736) dirty exists 2024-09-11T01:43:44.780 INFO:tasks.rados.rados.0.smithi071.stdout:2230: finishing write tid 1 to smithi07134777-3 2024-09-11T01:43:44.780 INFO:tasks.rados.rados.0.smithi071.stdout:2225: finishing copy_from racing read to smithi07134777-6 2024-09-11T01:43:44.780 INFO:tasks.rados.rados.0.smithi071.stdout:2228: finishing copy_from racing read to smithi07134777-39 2024-09-11T01:43:44.780 INFO:tasks.rados.rados.0.smithi071.stdout:2230: finishing write tid 2 to smithi07134777-3 2024-09-11T01:43:44.780 INFO:tasks.rados.rados.0.smithi071.stdout:2225: finishing copy_from to smithi07134777-6 2024-09-11T01:43:44.780 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 989 (ObjNum 652 snap 199 seq_num 652) dirty exists 2024-09-11T01:43:44.780 INFO:tasks.rados.rados.0.smithi071.stdout:2230: finishing write tid 3 to smithi07134777-3 2024-09-11T01:43:44.780 INFO:tasks.rados.rados.0.smithi071.stdout:2230: finishing write tid 4 to smithi07134777-3 2024-09-11T01:43:44.780 INFO:tasks.rados.rados.0.smithi071.stdout:2230: finishing write tid 5 to smithi07134777-3 2024-09-11T01:43:44.780 INFO:tasks.rados.rados.0.smithi071.stdout:2230: oid 3 updating version 0 to 719 2024-09-11T01:43:44.781 INFO:tasks.rados.rados.0.smithi071.stdout:2230: oid 3 updating version 719 to 721 2024-09-11T01:43:44.781 INFO:tasks.rados.rados.0.smithi071.stdout:2230: oid 3 version 721 is already newer than 720 2024-09-11T01:43:44.781 INFO:tasks.rados.rados.0.smithi071.stdout:2230: oid 3 version 721 is already newer than 718 2024-09-11T01:43:44.781 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 721 (ObjNum 810 snap 246 seq_num 810) dirty exists 2024-09-11T01:43:44.781 INFO:tasks.rados.rados.0.smithi071.stdout:2230: left oid 3 (ObjNum 810 snap 246 seq_num 810) 2024-09-11T01:43:44.781 INFO:tasks.rados.rados.0.smithi071.stdout:2228: finishing copy_from to smithi07134777-39 2024-09-11T01:43:44.781 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 980 (ObjNum 804 snap 243 seq_num 804) dirty exists 2024-09-11T01:43:44.781 INFO:tasks.rados.rados.0.smithi071.stdout:2231: finishing write tid 1 to smithi07134777-34 2024-09-11T01:43:44.781 INFO:tasks.rados.rados.0.smithi071.stdout:2221: done (11 left) 2024-09-11T01:43:44.781 INFO:tasks.rados.rados.0.smithi071.stdout:2223: done (10 left) 2024-09-11T01:43:44.781 INFO:tasks.rados.rados.0.smithi071.stdout:2224: done (9 left) 2024-09-11T01:43:44.781 INFO:tasks.rados.rados.0.smithi071.stdout:2225: done (8 left) 2024-09-11T01:43:44.781 INFO:tasks.rados.rados.0.smithi071.stdout:2226: done (7 left) 2024-09-11T01:43:44.781 INFO:tasks.rados.rados.0.smithi071.stdout:2227: done (6 left) 2024-09-11T01:43:44.782 INFO:tasks.rados.rados.0.smithi071.stdout:2228: done (5 left) 2024-09-11T01:43:44.782 INFO:tasks.rados.rados.0.smithi071.stdout:2229: done (4 left) 2024-09-11T01:43:44.782 INFO:tasks.rados.rados.0.smithi071.stdout:2230: done (3 left) 2024-09-11T01:43:44.782 INFO:tasks.rados.rados.0.smithi071.stdout:2231: finishing write tid 2 to smithi07134777-34 2024-09-11T01:43:44.782 INFO:tasks.rados.rados.0.smithi071.stdout:2231: finishing write tid 3 to smithi07134777-34 2024-09-11T01:43:44.782 INFO:tasks.rados.rados.0.smithi071.stdout:2231: finishing write tid 4 to smithi07134777-34 2024-09-11T01:43:44.782 INFO:tasks.rados.rados.0.smithi071.stdout:2231: finishing write tid 5 to smithi07134777-34 2024-09-11T01:43:44.782 INFO:tasks.rados.rados.0.smithi071.stdout:2231: oid 34 updating version 0 to 1183 2024-09-11T01:43:44.782 INFO:tasks.rados.rados.0.smithi071.stdout:2231: oid 34 version 1183 is already newer than 1182 2024-09-11T01:43:44.782 INFO:tasks.rados.rados.0.smithi071.stdout:2231: oid 34 version 1183 is already newer than 1181 2024-09-11T01:43:44.782 INFO:tasks.rados.rados.0.smithi071.stdout:2231: oid 34 updating version 1183 to 1184 2024-09-11T01:43:44.782 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 1184 (ObjNum 811 snap 246 seq_num 811) dirty exists 2024-09-11T01:43:44.782 INFO:tasks.rados.rados.0.smithi071.stdout:2231: left oid 34 (ObjNum 811 snap 246 seq_num 811) 2024-09-11T01:43:44.782 INFO:tasks.rados.rados.0.smithi071.stdout:2232: finishing write tid 1 to smithi07134777-30 2024-09-11T01:43:44.783 INFO:tasks.rados.rados.0.smithi071.stdout:2234: read oid 42 snap -1 2024-09-11T01:43:44.783 INFO:tasks.rados.rados.0.smithi071.stdout:2234: expect (ObjNum 806 snap 244 seq_num 806) 2024-09-11T01:43:44.783 INFO:tasks.rados.rados.0.smithi071.stdout:2231: done (3 left) 2024-09-11T01:43:44.783 INFO:tasks.rados.rados.0.smithi071.stdout:2235: delete oid 4 current snap is 246 2024-09-11T01:43:44.783 INFO:tasks.rados.rados.0.smithi071.stdout:2232: finishing write tid 2 to smithi07134777-30 2024-09-11T01:43:44.783 INFO:tasks.rados.rados.0.smithi071.stdout:2232: finishing write tid 3 to smithi07134777-30 2024-09-11T01:43:44.783 INFO:tasks.rados.rados.0.smithi071.stdout:2232: finishing write tid 4 to smithi07134777-30 2024-09-11T01:43:44.783 INFO:tasks.rados.rados.0.smithi071.stdout:2232: finishing write tid 5 to smithi07134777-30 2024-09-11T01:43:44.783 INFO:tasks.rados.rados.0.smithi071.stdout:2232: oid 30 updating version 0 to 978 2024-09-11T01:43:44.783 INFO:tasks.rados.rados.0.smithi071.stdout:2232: oid 30 updating version 978 to 980 2024-09-11T01:43:44.783 INFO:tasks.rados.rados.0.smithi071.stdout:2232: oid 30 version 980 is already newer than 977 2024-09-11T01:43:44.783 INFO:tasks.rados.rados.0.smithi071.stdout:2232: oid 30 version 980 is already newer than 979 2024-09-11T01:43:44.783 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 980 (ObjNum 812 snap 246 seq_num 812) dirty exists 2024-09-11T01:43:44.783 INFO:tasks.rados.rados.0.smithi071.stdout:2232: left oid 30 (ObjNum 812 snap 246 seq_num 812) 2024-09-11T01:43:44.784 INFO:tasks.rados.rados.0.smithi071.stdout:2233: finishing copy_from racing read to smithi07134777-40 2024-09-11T01:43:44.784 INFO:tasks.rados.rados.0.smithi071.stdout:2233: finishing copy_from to smithi07134777-40 2024-09-11T01:43:44.784 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 717 (ObjNum 797 snap 243 seq_num 797) dirty exists 2024-09-11T01:43:44.784 INFO:tasks.rados.rados.0.smithi071.stdout:2234: expect (ObjNum 806 snap 244 seq_num 806) 2024-09-11T01:43:44.834 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1283055 2024-09-11T01:43:44.926 INFO:tasks.rados.rados.0.smithi071.stdout:2232: done (3 left) 2024-09-11T01:43:44.926 INFO:tasks.rados.rados.0.smithi071.stdout:2233: done (2 left) 2024-09-11T01:43:44.926 INFO:tasks.rados.rados.0.smithi071.stdout:2234: done (1 left) 2024-09-11T01:43:44.926 INFO:tasks.rados.rados.0.smithi071.stdout:2235: done (0 left) 2024-09-11T01:43:44.926 INFO:tasks.rados.rados.0.smithi071.stdout:2236: write oid 39 current snap is 246 2024-09-11T01:43:44.926 INFO:tasks.rados.rados.0.smithi071.stdout:2236: seq_num 814 ranges {480898=502457,1699872=737281,3087022=593876} 2024-09-11T01:43:44.931 INFO:tasks.rados.rados.0.smithi071.stdout:2236: writing smithi07134777-39 from 480898 to 983355 tid 1 2024-09-11T01:43:44.938 INFO:tasks.rados.rados.0.smithi071.stdout:2236: writing smithi07134777-39 from 1699872 to 2437153 tid 2 2024-09-11T01:43:44.945 INFO:tasks.rados.rados.0.smithi071.stdout:2236: writing smithi07134777-39 from 3087022 to 3680898 tid 3 2024-09-11T01:43:44.945 INFO:tasks.rados.rados.0.smithi071.stdout:2237: copy_from oid 9 from oid 23 current snap is 246 2024-09-11T01:43:44.945 INFO:tasks.rados.rados.0.smithi071.stdout:2236: finishing write tid 1 to smithi07134777-39 2024-09-11T01:43:44.946 INFO:tasks.rados.rados.0.smithi071.stdout:2236: finishing write tid 2 to smithi07134777-39 2024-09-11T01:43:44.946 INFO:tasks.rados.rados.0.smithi071.stdout:2238: read oid 46 snap -1 2024-09-11T01:43:44.946 INFO:tasks.rados.rados.0.smithi071.stdout:2238: expect (ObjNum 691 snap 204 seq_num 691) 2024-09-11T01:43:44.946 INFO:tasks.rados.rados.0.smithi071.stdout:2239: snap_create 2024-09-11T01:43:44.950 INFO:tasks.rados.rados.0.smithi071.stdout:2236: finishing write tid 3 to smithi07134777-39 2024-09-11T01:43:44.950 INFO:tasks.rados.rados.0.smithi071.stdout:2236: finishing write tid 4 to smithi07134777-39 2024-09-11T01:43:44.950 INFO:tasks.rados.rados.0.smithi071.stdout:2236: finishing write tid 5 to smithi07134777-39 2024-09-11T01:43:44.950 INFO:tasks.rados.rados.0.smithi071.stdout:2236: oid 39 updating version 0 to 1004 2024-09-11T01:43:44.950 INFO:tasks.rados.rados.0.smithi071.stdout:2236: oid 39 version 1004 is already newer than 1001 2024-09-11T01:43:44.951 INFO:tasks.rados.rados.0.smithi071.stdout:2236: oid 39 version 1004 is already newer than 1003 2024-09-11T01:43:44.951 INFO:tasks.rados.rados.0.smithi071.stdout:2236: oid 39 version 1004 is already newer than 1002 2024-09-11T01:43:44.951 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 1004 (ObjNum 814 snap 246 seq_num 814) dirty exists 2024-09-11T01:43:44.951 INFO:tasks.rados.rados.0.smithi071.stdout:2236: left oid 39 (ObjNum 814 snap 246 seq_num 814) 2024-09-11T01:43:44.951 INFO:tasks.rados.rados.0.smithi071.stdout:2238: expect (ObjNum 691 snap 204 seq_num 691) 2024-09-11T01:43:44.970 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2435136 2024-09-11T01:43:45.012 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2468304 2024-09-11T01:43:45.034 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:45.087 INFO:tasks.rados.rados.0.smithi071.stdout:2237: finishing copy_from racing read to smithi07134777-9 2024-09-11T01:43:45.087 INFO:tasks.rados.rados.0.smithi071.stdout:2237: finishing copy_from to smithi07134777-9 2024-09-11T01:43:45.087 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 989 (ObjNum 652 snap 199 seq_num 652) dirty exists 2024-09-11T01:43:45.236 INFO:tasks.rados.rados.0.smithi071.stdout:2236: done (3 left) 2024-09-11T01:43:45.236 INFO:tasks.rados.rados.0.smithi071.stdout:2237: done (2 left) 2024-09-11T01:43:45.236 INFO:tasks.rados.rados.0.smithi071.stdout:2238: done (1 left) 2024-09-11T01:43:45.236 INFO:tasks.rados.rados.0.smithi071.stdout:2239: done (0 left) 2024-09-11T01:43:45.236 INFO:tasks.rados.rados.0.smithi071.stdout:2240: read oid 44 snap -1 2024-09-11T01:43:45.236 INFO:tasks.rados.rados.0.smithi071.stdout:2240: expect (ObjNum 779 snap 236 seq_num 779) 2024-09-11T01:43:45.236 INFO:tasks.rados.rados.0.smithi071.stdout:2241: read oid 21 snap -1 2024-09-11T01:43:45.236 INFO:tasks.rados.rados.0.smithi071.stdout:2241: expect (ObjNum 738 snap 226 seq_num 738) 2024-09-11T01:43:45.236 INFO:tasks.rados.rados.0.smithi071.stdout:2242: snap_create 2024-09-11T01:43:45.240 INFO:tasks.rados.rados.0.smithi071.stdout:2240: expect (ObjNum 779 snap 236 seq_num 779) 2024-09-11T01:43:45.408 INFO:tasks.rados.rados.0.smithi071.stdout:2241: expect (ObjNum 738 snap 226 seq_num 738) 2024-09-11T01:43:45.408 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:45.513 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:46.237 INFO:tasks.rados.rados.0.smithi071.stdout:2240: done (2 left) 2024-09-11T01:43:46.238 INFO:tasks.rados.rados.0.smithi071.stdout:2241: done (1 left) 2024-09-11T01:43:46.238 INFO:tasks.rados.rados.0.smithi071.stdout:2242: done (0 left) 2024-09-11T01:43:46.238 INFO:tasks.rados.rados.0.smithi071.stdout:2243: rollback oid 1 current snap is 248 2024-09-11T01:43:46.238 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 1 to 209 2024-09-11T01:43:46.238 INFO:tasks.rados.rados.0.smithi071.stdout:2244: copy_from oid 21 from oid 14 current snap is 248 2024-09-11T01:43:46.238 INFO:tasks.rados.rados.0.smithi071.stdout:2245: rollback oid 5 current snap is 248 2024-09-11T01:43:46.238 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 5 to 209 2024-09-11T01:43:46.238 INFO:tasks.rados.rados.0.smithi071.stdout:2246: snap_create 2024-09-11T01:43:46.242 INFO:tasks.rados.rados.0.smithi071.stdout:2243: finishing rollback tid 1 to smithi07134777-1 2024-09-11T01:43:46.242 INFO:tasks.rados.rados.0.smithi071.stdout:2243: finishing rollback tid 2 to smithi07134777-1 2024-09-11T01:43:46.242 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 981 (ObjNum 688 snap 204 seq_num 688) dirty exists 2024-09-11T01:43:46.243 INFO:tasks.rados.rados.0.smithi071.stdout:2245: finishing rollback tid 0 to smithi07134777-5 2024-09-11T01:43:46.243 INFO:tasks.rados.rados.0.smithi071.stdout:2245: finishing rollback tid 1 to smithi07134777-5 2024-09-11T01:43:46.243 INFO:tasks.rados.rados.0.smithi071.stdout:2245: finishing rollback tid 2 to smithi07134777-5 2024-09-11T01:43:46.244 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 1155 (ObjNum 681 snap 204 seq_num 681) dirty exists 2024-09-11T01:43:46.254 INFO:tasks.rados.rados.0.smithi071.stdout:2244: finishing copy_from racing read to smithi07134777-21 2024-09-11T01:43:46.256 INFO:tasks.rados.rados.0.smithi071.stdout:2244: finishing copy_from to smithi07134777-21 2024-09-11T01:43:46.256 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 717 (ObjNum 797 snap 243 seq_num 797) dirty exists 2024-09-11T01:43:47.248 INFO:tasks.rados.rados.0.smithi071.stdout:2243: done (3 left) 2024-09-11T01:43:47.248 INFO:tasks.rados.rados.0.smithi071.stdout:2244: done (2 left) 2024-09-11T01:43:47.248 INFO:tasks.rados.rados.0.smithi071.stdout:2245: done (1 left) 2024-09-11T01:43:47.248 INFO:tasks.rados.rados.0.smithi071.stdout:2246: done (0 left) 2024-09-11T01:43:47.249 INFO:tasks.rados.rados.0.smithi071.stdout:2247: rollback oid 32 current snap is 249 2024-09-11T01:43:47.249 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 32 to 235 2024-09-11T01:43:47.249 INFO:tasks.rados.rados.0.smithi071.stdout:2248: read oid 35 snap -1 2024-09-11T01:43:47.249 INFO:tasks.rados.rados.0.smithi071.stdout:2248: expect deleted 2024-09-11T01:43:47.249 INFO:tasks.rados.rados.0.smithi071.stdout:2249: snap_create 2024-09-11T01:43:47.253 INFO:tasks.rados.rados.0.smithi071.stdout:2247: finishing rollback tid 0 to smithi07134777-32 2024-09-11T01:43:47.254 INFO:tasks.rados.rados.0.smithi071.stdout:2247: finishing rollback tid 1 to smithi07134777-32 2024-09-11T01:43:47.257 INFO:tasks.rados.rados.0.smithi071.stdout:2247: finishing rollback tid 2 to smithi07134777-32 2024-09-11T01:43:47.257 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 1007 (ObjNum 745 snap 227 seq_num 745) dirty exists 2024-09-11T01:43:47.313 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:43:47.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.426+0000 7fc8d7ced640 10 monclient: get_monmap_and_config 2024-09-11T01:43:47.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.426+0000 7fc8d7ced640 10 monclient: build_initial_monmap 2024-09-11T01:43:47.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.426+0000 7fc8d7ced640 10 monclient: monmap: 2024-09-11T01:43:47.428 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:47.428 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:47.428 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:43:47.426935+0000 2024-09-11T01:43:47.428 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:43:47.426935+0000 2024-09-11T01:43:47.428 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:47.428 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:47.428 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:43:47.428 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:43:47.428 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:43:47.428 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:43:47.428 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:43:47.428 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:47.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.426+0000 7fc8d7ced640 1 Processor -- start 2024-09-11T01:43:47.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.427+0000 7fc8d7ced640 1 -- start start 2024-09-11T01:43:47.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.427+0000 7fc8d7ced640 10 monclient: init 2024-09-11T01:43:47.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.427+0000 7fc8d7ced640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:47.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.427+0000 7fc8d7ced640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-11T01:43:47.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.427+0000 7fc8d7ced640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8d0138230 0x7fc8d0138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:47.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.427+0000 7fc8d7ced640 10 monclient(hunting): picked mon.noname-b con 0x7fc8d0138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:47.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.427+0000 7fc8d7ced640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc8d009d030 addr v1:172.21.15.71:6789/0 2024-09-11T01:43:47.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.427+0000 7fc8d7ced640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc8d009d430 addr v1:172.21.15.12:6789/0 2024-09-11T01:43:47.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.427+0000 7fc8d7ced640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8d00a5e40 con 0x7fc8d009d430 2024-09-11T01:43:47.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.427+0000 7fc8d7ced640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8d00a6e30 con 0x7fc8d009d030 2024-09-11T01:43:47.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.427+0000 7fc8d7ced640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:47.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.427+0000 7fc8d7ced640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc8d009c9c0 con 0x7fc8d0138230 2024-09-11T01:43:47.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.427+0000 7fc8d7ced640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:47.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.427+0000 7fc8d7ced640 10 monclient(hunting): authenticate will time out at 1633.869641s 2024-09-11T01:43:47.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.427+0000 7fc8d5a89640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8d0138230 0x7fc8d0138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:47.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.427+0000 7fc8d5a89640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8d0138230 0x7fc8d0138610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:53150/0 (socket says 172.21.15.12:53150) 2024-09-11T01:43:47.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.427+0000 7fc8d628a640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc8d009d430 0x7fc8d00a5730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37620/0 (socket says 172.21.15.12:37620) 2024-09-11T01:43:47.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.427+0000 7fc8d5a89640 1 -- 172.21.15.12:0/3999595400 learned_addr learned my addr 172.21.15.12:0/3999595400 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:47.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.427+0000 7fc8d5a89640 10 monclient(hunting): get_auth_request con 0x7fc8d0138230 auth_method 0 2024-09-11T01:43:47.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.427+0000 7fc8d5a89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:47.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.427+0000 7fc8d5a89640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:47.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.427+0000 7fc8d5a89640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:47.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.427+0000 7fc8d5a89640 1 -- 172.21.15.12:0/3999595400 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8d0138230 msgr2=0x7fc8d0138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:43:47.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.427+0000 7fc8d5a89640 1 -- 172.21.15.12:0/3999595400 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8d0138230 msgr2=0x7fc8d0138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:43:47.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.427+0000 7fc8d5a89640 1 --2- 172.21.15.12:0/3999595400 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8d0138230 0x7fc8d0138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:47.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.427+0000 7fc8d5a89640 1 --2- 172.21.15.12:0/3999595400 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8d0138230 0x7fc8d0138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:47.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.429+0000 7fc8d628a640 1 -- 172.21.15.12:0/3999595400 >> v1:172.21.15.12:6789/0 conn(0x7fc8d009d430 legacy=0x7fc8d00a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:47.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 1 -- 172.21.15.12:0/3999595400 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1060474275 0 0) 0x7fc8d00a5e40 con 0x7fc8d009d430 2024-09-11T01:43:47.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 10 monclient(hunting): _negotiate 2024-09-11T01:43:47.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:47.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:47.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 10 monclient(hunting): my global_id is 94752 2024-09-11T01:43:47.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 1 -- 172.21.15.12:0/3999595400 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8bc003520 con 0x7fc8d009d430 2024-09-11T01:43:47.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d5288640 1 -- 172.21.15.12:0/3999595400 >> v1:172.21.15.71:6789/0 conn(0x7fc8d009d030 legacy=0x7fc8d00a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:47.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 1 -- 172.21.15.12:0/3999595400 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3757599227 0 0) 0x7fc8bc003520 con 0x7fc8d009d430 2024-09-11T01:43:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 1 -- 172.21.15.12:0/3999595400 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8d0138230 msgr2=0x7fc8d0138610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:43:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 1 --2- 172.21.15.12:0/3999595400 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8d0138230 0x7fc8d0138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 1 -- 172.21.15.12:0/3999595400 >> v1:172.21.15.71:6789/0 conn(0x7fc8d009d030 legacy=0x7fc8d00a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 10 monclient: _finish_hunting 0 2024-09-11T01:43:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:43:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:43:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 1 -- 172.21.15.12:0/3999595400 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d009ba20 con 0x7fc8d009d430 2024-09-11T01:43:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 10 monclient: _finish_auth 0 2024-09-11T01:43:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 10 monclient: _check_auth_tickets 2024-09-11T01:43:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d7ced640 5 monclient: authenticate success, global_id 94752 2024-09-11T01:43:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d7ced640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:43:47.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 1 -- 172.21.15.12:0/3999595400 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7fc8c8001d20 con 0x7fc8d009d430 2024-09-11T01:43:47.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:47.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:43:47.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 10 monclient: dump: 2024-09-11T01:43:47.441 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:47.441 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:47.441 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:47.441 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:47.441 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:47.441 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:47.441 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:47.441 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:47.441 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:47.441 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:43:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 1 -- 172.21.15.12:0/3999595400 >> v1:172.21.15.12:6789/0 conn(0x7fc8d009d430 legacy=0x7fc8d00a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:43:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 1 --2- 172.21.15.12:0/3999595400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc8bc003b00 0x7fc8bc003f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 10 monclient(hunting): picked mon.b con 0x7fc8bc003b00 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 1 --2- 172.21.15.12:0/3999595400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8bc004800 0x7fc8bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 10 monclient(hunting): picked mon.a con 0x7fc8bc004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 1 --2- 172.21.15.12:0/3999595400 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8d0138230 0x7fc8bc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 10 monclient(hunting): picked mon.c con 0x7fc8d0138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 1 -- 172.21.15.12:0/3999595400 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d009c9c0 con 0x7fc8bc004800 2024-09-11T01:43:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 1 -- 172.21.15.12:0/3999595400 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc8bc00bf00 con 0x7fc8bc003b00 2024-09-11T01:43:47.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:47.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 1 -- 172.21.15.12:0/3999595400 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc8bc00c080 con 0x7fc8d0138230 2024-09-11T01:43:47.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:47.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d4a87640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:43:47.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d7ced640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:43:47.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d5288640 1 --2- 172.21.15.12:0/3999595400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8bc004800 0x7fc8bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:47.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d628a640 1 --2- 172.21.15.12:0/3999595400 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8d0138230 0x7fc8bc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:47.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d5288640 10 monclient(hunting): get_auth_request con 0x7fc8bc004800 auth_method 0 2024-09-11T01:43:47.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d5288640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:47.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d5288640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:47.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d5288640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:47.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d628a640 10 monclient(hunting): get_auth_request con 0x7fc8d0138230 auth_method 0 2024-09-11T01:43:47.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d628a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:47.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d628a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:47.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d628a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:47.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d5a89640 1 --2- 172.21.15.12:0/3999595400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc8bc003b00 0x7fc8bc003f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:47.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d5a89640 10 monclient(hunting): get_auth_request con 0x7fc8bc003b00 auth_method 0 2024-09-11T01:43:47.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d5a89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:47.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d5a89640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:47.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d5a89640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:47.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d5288640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:47.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d5288640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:47.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d5288640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:43:47.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d628a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:47.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d628a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:47.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d628a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:43:47.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d5a89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:47.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d5a89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:47.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d5a89640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:43:47.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d5288640 10 monclient(hunting): handle_auth_done global_id 94752 payload 293 2024-09-11T01:43:47.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.430+0000 7fc8d5288640 1 -- 172.21.15.12:0/3999595400 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8d0138230 msgr2=0x7fc8bc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:47.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.431+0000 7fc8d5288640 1 --2- 172.21.15.12:0/3999595400 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8d0138230 0x7fc8bc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:47.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.431+0000 7fc8d5288640 1 -- 172.21.15.12:0/3999595400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc8bc003b00 msgr2=0x7fc8bc003f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:47.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.431+0000 7fc8d5288640 1 --2- 172.21.15.12:0/3999595400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc8bc003b00 0x7fc8bc003f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:47.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.431+0000 7fc8d5288640 10 monclient: _finish_hunting 0 2024-09-11T01:43:47.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.431+0000 7fc8d5288640 1 monclient: found mon.a 2024-09-11T01:43:47.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.431+0000 7fc8d5288640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:47.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.431+0000 7fc8d5288640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:47.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.431+0000 7fc8d5288640 1 -- 172.21.15.12:0/3999595400 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8bc003520 con 0x7fc8bc004800 2024-09-11T01:43:47.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.431+0000 7fc8d5288640 10 monclient: _finish_auth 0 2024-09-11T01:43:47.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.431+0000 7fc8d5288640 10 monclient: _check_auth_tickets 2024-09-11T01:43:47.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.431+0000 7fc8d5288640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:47.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.433+0000 7fc8d5288640 1 -- 172.21.15.12:0/3999595400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8bc004800 msgr2=0x7fc8bc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:47.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.433+0000 7fc8d5288640 1 --2- 172.21.15.12:0/3999595400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8bc004800 0x7fc8bc004c80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0001990 tx=0x7fc8bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5434e7dde9f2fd25 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:47.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.433+0000 7fc8d4a87640 1 -- 172.21.15.12:0/3999595400 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c0001690 con 0x7fc8bc004800 2024-09-11T01:43:47.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.434+0000 7fc8d4a87640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:47.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.434+0000 7fc8d4a87640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:47.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.434+0000 7fc8d4a87640 10 monclient: dump: 2024-09-11T01:43:47.446 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:47.446 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:47.446 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:47.446 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:47.446 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:47.446 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:47.446 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:47.447 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:47.447 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:47.447 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:47.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.434+0000 7fc8d7ced640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:43:47.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.434+0000 7fc8d4a87640 1 -- 172.21.15.12:0/3999595400 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8c002e070 con 0x7fc8bc004800 2024-09-11T01:43:47.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.434+0000 7fc8d4a87640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:47.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.434+0000 7fc8d7ced640 10 monclient: get_monmap_and_config success 2024-09-11T01:43:47.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.434+0000 7fc8d4a87640 1 -- 172.21.15.12:0/3999595400 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c0007070 con 0x7fc8bc004800 2024-09-11T01:43:47.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.434+0000 7fc8d7ced640 4 set_mon_vals no callback set 2024-09-11T01:43:47.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.434+0000 7fc8d4a87640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:47.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.434+0000 7fc8d4a87640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:47.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.434+0000 7fc8d4a87640 10 monclient: dump: 2024-09-11T01:43:47.447 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:47.447 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:47.448 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:47.448 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:47.448 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:47.448 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:47.448 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:47.448 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:47.448 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:47.448 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:47.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.434+0000 7fc8d7ced640 10 monclient: shutdown 2024-09-11T01:43:47.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.434+0000 7fc8d7ced640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:47.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.434+0000 7fc8d7ced640 1 -- 172.21.15.12:0/3999595400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8bc004800 msgr2=0x7fc8bc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:47.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.434+0000 7fc8d7ced640 1 --2- 172.21.15.12:0/3999595400 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8bc004800 0x7fc8bc004c80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0001990 tx=0x7fc8bc005830 comp rx=0 tx=0).stop 2024-09-11T01:43:47.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.434+0000 7fc8d628a640 1 -- 172.21.15.12:0/3999595400 reap_dead start 2024-09-11T01:43:47.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.434+0000 7fc8d7ced640 1 -- 172.21.15.12:0/3999595400 shutdown_connections 2024-09-11T01:43:47.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.434+0000 7fc8d7ced640 1 -- 172.21.15.12:0/3999595400 >> 172.21.15.12:0/3999595400 conn(0x7fc8d00a7120 msgr2=0x7fc8d00a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:47.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.434+0000 7fc8d7ced640 1 -- 172.21.15.12:0/3999595400 shutdown_connections 2024-09-11T01:43:47.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.434+0000 7fc8d7ced640 1 -- 172.21.15.12:0/3999595400 wait complete. 2024-09-11T01:43:47.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.435+0000 7fc8d7ced640 10 monclient: build_initial_monmap 2024-09-11T01:43:47.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.435+0000 7fc8d7ced640 10 monclient: monmap: 2024-09-11T01:43:47.449 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:47.449 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:43:47.449 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:43:47.449 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:43:47.449 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:47.449 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:47.449 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:43:47.449 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:43:47.449 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:43:47.450 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:47.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.435+0000 7fc8d7ced640 1 Processor -- start 2024-09-11T01:43:47.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.435+0000 7fc8d7ced640 1 -- start start 2024-09-11T01:43:47.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.435+0000 7fc8d7ced640 10 monclient: init 2024-09-11T01:43:47.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d7ced640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:47.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d7ced640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:43:47.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d7ced640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8d0137130 0x7fc8d0137510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:47.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d7ced640 10 monclient(hunting): picked mon.noname-a con 0x7fc8d0137130 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:47.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d7ced640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc8d01cd160 0x7fc8d0124d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:47.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d7ced640 10 monclient(hunting): picked mon.noname-b con 0x7fc8d01cd160 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:47.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d7ced640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8d0125250 0x7fc8d01256e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:47.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d7ced640 10 monclient(hunting): picked mon.noname-c con 0x7fc8d0125250 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:47.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d7ced640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:47.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d7ced640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d0138610 con 0x7fc8d0137130 2024-09-11T01:43:47.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d7ced640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:47.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d7ced640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d00a6730 con 0x7fc8d01cd160 2024-09-11T01:43:47.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d7ced640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:47.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d7ced640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc8d0138790 con 0x7fc8d0125250 2024-09-11T01:43:47.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d7ced640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:47.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d7ced640 10 monclient(hunting): authenticate will time out at 1633.878220s 2024-09-11T01:43:47.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d5a89640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8d0137130 0x7fc8d0137510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:47.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d628a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8d0125250 0x7fc8d01256e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:47.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d5a89640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8d0137130 0x7fc8d0137510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35676/0 (socket says 172.21.15.12:35676) 2024-09-11T01:43:47.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d5a89640 1 -- 172.21.15.12:0/3387731785 learned_addr learned my addr 172.21.15.12:0/3387731785 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:47.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d5a89640 10 monclient(hunting): get_auth_request con 0x7fc8d0137130 auth_method 0 2024-09-11T01:43:47.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d5a89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:47.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d5a89640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:47.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d5a89640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:47.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d628a640 10 monclient(hunting): get_auth_request con 0x7fc8d0125250 auth_method 0 2024-09-11T01:43:47.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d628a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:47.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d628a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:47.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d628a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:47.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d628a640 1 -- 172.21.15.12:0/3387731785 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8d0125250 msgr2=0x7fc8d01256e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:43:47.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d628a640 1 -- 172.21.15.12:0/3387731785 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8d0125250 msgr2=0x7fc8d01256e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:43:47.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d5a89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:47.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d5a89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:47.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d628a640 1 --2- 172.21.15.12:0/3387731785 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8d0125250 0x7fc8d01256e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:47.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d5a89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:47.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d628a640 1 --2- 172.21.15.12:0/3387731785 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8d0125250 0x7fc8d01256e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:47.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d5288640 1 --2- 172.21.15.12:0/3387731785 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc8d01cd160 0x7fc8d0124d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:47.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d5288640 10 monclient(hunting): get_auth_request con 0x7fc8d01cd160 auth_method 0 2024-09-11T01:43:47.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d5288640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:47.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d5288640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:47.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d5288640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:47.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d5a89640 10 monclient(hunting): handle_auth_done global_id 94755 payload 824 2024-09-11T01:43:47.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d5a89640 1 -- 172.21.15.12:0/3387731785 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8d0125250 msgr2=0x7fc8d01256e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:43:47.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d5a89640 1 --2- 172.21.15.12:0/3387731785 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8d0125250 0x7fc8d01256e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:47.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d5a89640 1 -- 172.21.15.12:0/3387731785 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc8d01cd160 msgr2=0x7fc8d0124d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:47.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d5a89640 1 --2- 172.21.15.12:0/3387731785 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc8d01cd160 0x7fc8d0124d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:47.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d5a89640 10 monclient: _finish_hunting 0 2024-09-11T01:43:47.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d5a89640 1 monclient: found mon.noname-a 2024-09-11T01:43:47.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d5a89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:47.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d5a89640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:47.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.436+0000 7fc8d5a89640 1 -- 172.21.15.12:0/3387731785 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d0128b30 con 0x7fc8d0137130 2024-09-11T01:43:47.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.438+0000 7fc8d5a89640 1 -- 172.21.15.12:0/3387731785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8d0137130 msgr2=0x7fc8d0137510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:47.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.438+0000 7fc8d5a89640 1 --2- 172.21.15.12:0/3387731785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8d0137130 0x7fc8d0137510 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc011680 tx=0x7fc8cc011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:47.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.439+0000 7fc8c67fc640 1 -- 172.21.15.12:0/3387731785 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc019070 con 0x7fc8d0137130 2024-09-11T01:43:47.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.439+0000 7fc8c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:47.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.439+0000 7fc8c67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:47.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.439+0000 7fc8c67fc640 10 monclient: dump: 2024-09-11T01:43:47.456 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:47.456 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:47.456 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:47.456 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:47.456 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:47.456 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:47.457 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:47.457 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:47.457 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:47.457 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:47.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.439+0000 7fc8c67fc640 10 monclient: _finish_auth 0 2024-09-11T01:43:47.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.439+0000 7fc8c67fc640 10 monclient: _check_auth_tickets 2024-09-11T01:43:47.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.439+0000 7fc8c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:47.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.439+0000 7fc8d7ced640 5 monclient: authenticate success, global_id 94755 2024-09-11T01:43:47.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.439+0000 7fc8d7ced640 10 monclient: _renew_subs 2024-09-11T01:43:47.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.439+0000 7fc8c67fc640 1 -- 172.21.15.12:0/3387731785 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8cc00fbd0 con 0x7fc8d0137130 2024-09-11T01:43:47.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.439+0000 7fc8d7ced640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:47.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.439+0000 7fc8d7ced640 1 -- 172.21.15.12:0/3387731785 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8d00a1810 con 0x7fc8d0137130 2024-09-11T01:43:47.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.439+0000 7fc8c67fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:47.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.439+0000 7fc8c67fc640 1 -- 172.21.15.12:0/3387731785 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc00f040 con 0x7fc8d0137130 2024-09-11T01:43:47.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.439+0000 7fc8c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:47.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.439+0000 7fc8c67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:47.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.439+0000 7fc8c67fc640 10 monclient: dump: 2024-09-11T01:43:47.458 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:47.458 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:47.458 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:47.458 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:47.458 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:47.458 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:47.458 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:47.458 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:47.458 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:47.458 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:47.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.439+0000 7fc8c7fff640 4 set_mon_vals no callback set 2024-09-11T01:43:47.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.439+0000 7fc8d7ced640 10 monclient: _renew_subs 2024-09-11T01:43:47.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.439+0000 7fc8d7ced640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:47.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.439+0000 7fc8d7ced640 1 -- 172.21.15.12:0/3387731785 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8d01267d0 con 0x7fc8d0137130 2024-09-11T01:43:47.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.440+0000 7fc8d7ced640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:47.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.440+0000 7fc8d7ced640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:47.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.440+0000 7fc8d7ced640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:47.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.440+0000 7fc8d7ced640 1 -- 172.21.15.12:0/3387731785 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8d0138b60 con 0x7fc8d0137130 2024-09-11T01:43:47.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.440+0000 7fc8c67fc640 1 -- 172.21.15.12:0/3387731785 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fc8cc015460 con 0x7fc8d0137130 2024-09-11T01:43:47.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.440+0000 7fc8c67fc640 1 --2- 172.21.15.12:0/3387731785 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc8b403e210 0x7fc8b40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:47.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.440+0000 7fc8c67fc640 1 -- 172.21.15.12:0/3387731785 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(500..500 src has 317..500) ==== 4755+0+0 (secure 0 0 0) 0x7fc8cc053610 con 0x7fc8d0137130 2024-09-11T01:43:47.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.441+0000 7fc8d5288640 1 --2- 172.21.15.12:0/3387731785 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc8b403e210 0x7fc8b40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:47.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.441+0000 7fc8d5288640 10 monclient: get_auth_request con 0x7fc8b403e210 auth_method 0 2024-09-11T01:43:47.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.443+0000 7fc8d5288640 1 --2- 172.21.15.12:0/3387731785 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc8b403e210 0x7fc8b40406d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc8c002f000 tx=0x7fc8c0007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:47.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.445+0000 7fc8c67fc640 1 -- 172.21.15.12:0/3387731785 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc8cc013d50 con 0x7fc8d0137130 2024-09-11T01:43:47.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.445+0000 7fc8c67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:47.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.445+0000 7fc8c67fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:43:47.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.705+0000 7fc8d7ced640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:43:47.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.705+0000 7fc8d7ced640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:43:47.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.705+0000 7fc8d7ced640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:47.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.705+0000 7fc8d7ced640 1 -- 172.21.15.12:0/3387731785 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc8d0128290 con 0x7fc8d0137130 2024-09-11T01:43:47.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.705+0000 7fc8c67fc640 1 -- 172.21.15.12:0/3387731785 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7fc8d0128290 con 0x7fc8d0137130 2024-09-11T01:43:47.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.705+0000 7fc8c67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:43:47.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.705+0000 7fc8c67fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:43:47.707 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":132,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:43:47.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.707+0000 7fc8d7ced640 1 -- 172.21.15.12:0/3387731785 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc8b403e210 msgr2=0x7fc8b40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:47.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.707+0000 7fc8d7ced640 1 --2- 172.21.15.12:0/3387731785 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc8b403e210 0x7fc8b40406d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc8c002f000 tx=0x7fc8c0007000 comp rx=0 tx=0).stop 2024-09-11T01:43:47.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.707+0000 7fc8d7ced640 10 monclient: shutdown 2024-09-11T01:43:47.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.707+0000 7fc8d7ced640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:47.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.707+0000 7fc8d7ced640 1 -- 172.21.15.12:0/3387731785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8d0137130 msgr2=0x7fc8d0137510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:47.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.707+0000 7fc8d7ced640 1 --2- 172.21.15.12:0/3387731785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8d0137130 0x7fc8d0137510 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc011680 tx=0x7fc8cc011b40 comp rx=0 tx=0).stop 2024-09-11T01:43:47.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.707+0000 7fc8d7ced640 1 -- 172.21.15.12:0/3387731785 shutdown_connections 2024-09-11T01:43:47.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.707+0000 7fc8d7ced640 1 --2- 172.21.15.12:0/3387731785 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8d0125250 0x7fc8d01256e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:47.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.707+0000 7fc8d7ced640 1 --2- 172.21.15.12:0/3387731785 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc8d01cd160 0x7fc8d0124d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:47.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.707+0000 7fc8d7ced640 1 --2- 172.21.15.12:0/3387731785 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc8b403e210 0x7fc8b40406d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:47.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.707+0000 7fc8d7ced640 1 --2- 172.21.15.12:0/3387731785 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8d0137130 0x7fc8d0137510 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:47.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.707+0000 7fc8d7ced640 1 -- 172.21.15.12:0/3387731785 >> 172.21.15.12:0/3387731785 conn(0x7fc8d00a7120 msgr2=0x7fc8d00a0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:47.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.707+0000 7fc8d7ced640 1 -- 172.21.15.12:0/3387731785 shutdown_connections 2024-09-11T01:43:47.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:47.707+0000 7fc8d7ced640 1 -- 172.21.15.12:0/3387731785 wait complete. 2024-09-11T01:43:48.250 INFO:tasks.rados.rados.0.smithi071.stdout:2247: done (2 left) 2024-09-11T01:43:48.250 INFO:tasks.rados.rados.0.smithi071.stdout:2248: done (1 left) 2024-09-11T01:43:48.251 INFO:tasks.rados.rados.0.smithi071.stdout:2249: done (0 left) 2024-09-11T01:43:48.251 INFO:tasks.rados.rados.0.smithi071.stdout:2250: read oid 16 snap -1 2024-09-11T01:43:48.251 INFO:tasks.rados.rados.0.smithi071.stdout:2250: expect (ObjNum 411 snap 115 seq_num 411) 2024-09-11T01:43:48.251 INFO:tasks.rados.rados.0.smithi071.stdout:2251: write oid 18 current snap is 250 2024-09-11T01:43:48.251 INFO:tasks.rados.rados.0.smithi071.stdout:2251: seq_num 815 ranges {497705=577706,1642001=480939,2598092=497514} 2024-09-11T01:43:48.256 INFO:tasks.rados.rados.0.smithi071.stdout:2251: writing smithi07134777-18 from 497705 to 1075411 tid 1 2024-09-11T01:43:48.262 INFO:tasks.rados.rados.0.smithi071.stdout:2251: writing smithi07134777-18 from 1642001 to 2122940 tid 2 2024-09-11T01:43:48.269 INFO:tasks.rados.rados.0.smithi071.stdout:2251: writing smithi07134777-18 from 2598092 to 3095606 tid 3 2024-09-11T01:43:48.269 INFO:tasks.rados.rados.0.smithi071.stdout:2252: rollback oid 15 current snap is 250 2024-09-11T01:43:48.269 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 15 to 248 2024-09-11T01:43:48.269 INFO:tasks.rados.rados.0.smithi071.stdout:2250: expect (ObjNum 411 snap 115 seq_num 411) 2024-09-11T01:43:48.269 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:48.322 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:48.423 INFO:tasks.rados.rados.0.smithi071.stdout:2251: finishing write tid 1 to smithi07134777-18 2024-09-11T01:43:48.423 INFO:tasks.rados.rados.0.smithi071.stdout:2251: finishing write tid 2 to smithi07134777-18 2024-09-11T01:43:48.423 INFO:tasks.rados.rados.0.smithi071.stdout:2251: finishing write tid 3 to smithi07134777-18 2024-09-11T01:43:48.423 INFO:tasks.rados.rados.0.smithi071.stdout:2251: finishing write tid 4 to smithi07134777-18 2024-09-11T01:43:48.423 INFO:tasks.rados.rados.0.smithi071.stdout:2251: finishing write tid 5 to smithi07134777-18 2024-09-11T01:43:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:2251: oid 18 updating version 0 to 981 2024-09-11T01:43:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:2251: oid 18 updating version 981 to 984 2024-09-11T01:43:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:2251: oid 18 version 984 is already newer than 982 2024-09-11T01:43:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:2251: oid 18 version 984 is already newer than 983 2024-09-11T01:43:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 984 (ObjNum 815 snap 250 seq_num 815) dirty exists 2024-09-11T01:43:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:2251: left oid 18 (ObjNum 815 snap 250 seq_num 815) 2024-09-11T01:43:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:2252: finishing rollback tid 0 to smithi07134777-15 2024-09-11T01:43:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:2252: finishing rollback tid 1 to smithi07134777-15 2024-09-11T01:43:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:2250: done (2 left) 2024-09-11T01:43:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:2251: done (1 left) 2024-09-11T01:43:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:2253: snap_remove snap 240 2024-09-11T01:43:48.983 INFO:tasks.rados.rados.0.smithi071.stdout:2253: done (1 left) 2024-09-11T01:43:48.983 INFO:tasks.rados.rados.0.smithi071.stdout:2254: read oid 37 snap -1 2024-09-11T01:43:48.983 INFO:tasks.rados.rados.0.smithi071.stdout:2254: expect (ObjNum 723 snap 224 seq_num 723) 2024-09-11T01:43:48.983 INFO:tasks.rados.rados.0.smithi071.stdout:2255: snap_remove snap 247 2024-09-11T01:43:50.720 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:43:50.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.834+0000 7f28891f4640 10 monclient: get_monmap_and_config 2024-09-11T01:43:50.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.834+0000 7f28891f4640 10 monclient: build_initial_monmap 2024-09-11T01:43:50.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.834+0000 7f28891f4640 10 monclient: monmap: 2024-09-11T01:43:50.836 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:50.836 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:50.836 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:43:50.835646+0000 2024-09-11T01:43:50.837 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:43:50.835646+0000 2024-09-11T01:43:50.837 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:50.837 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:50.837 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:43:50.837 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:43:50.837 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:43:50.837 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:43:50.837 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:43:50.837 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:50.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.835+0000 7f28891f4640 1 Processor -- start 2024-09-11T01:43:50.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.835+0000 7f28891f4640 1 -- start start 2024-09-11T01:43:50.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.835+0000 7f28891f4640 10 monclient: init 2024-09-11T01:43:50.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f28891f4640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f28891f4640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-11T01:43:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f28891f4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28840a2e90 0x7f288406dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f28891f4640 10 monclient(hunting): picked mon.noname-b con 0x7f28840a2e90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f28891f4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f28840a3270 addr v1:172.21.15.12:6789/0 2024-09-11T01:43:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f28891f4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f288413d490 0x7f288413f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f28891f4640 10 monclient(hunting): picked mon.noname-a con 0x7f288413d490 addr v2:172.21.15.12:3300/0 2024-09-11T01:43:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f28891f4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28840a46c0 con 0x7f28840a3270 2024-09-11T01:43:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f28891f4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f28891f4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f28840a4900 con 0x7f288413d490 2024-09-11T01:43:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f28891f4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f28891f4640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f288406ee50 con 0x7f28840a2e90 2024-09-11T01:43:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f28891f4640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f28891f4640 10 monclient(hunting): authenticate will time out at 1637.278140s 2024-09-11T01:43:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f2882d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28840a2e90 0x7f288406dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f288359e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f288413d490 0x7f288413f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f288359e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f288413d490 0x7f288413f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35690/0 (socket says 172.21.15.12:35690) 2024-09-11T01:43:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f288359e640 1 -- 172.21.15.12:0/3469203288 learned_addr learned my addr 172.21.15.12:0/3469203288 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f2882d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28840a2e90 0x7f288406dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:53190/0 (socket says 172.21.15.12:53190) 2024-09-11T01:43:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f288359e640 10 monclient(hunting): get_auth_request con 0x7f288413d490 auth_method 0 2024-09-11T01:43:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f288359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:50.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f288359e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:50.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f288359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:50.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f2882d9d640 10 monclient(hunting): get_auth_request con 0x7f28840a2e90 auth_method 0 2024-09-11T01:43:50.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f2882d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:50.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f2882d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:50.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f2882d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:50.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f288359e640 1 -- 172.21.15.12:0/3469203288 >> v2:172.21.15.12:3300/0 conn(0x7f288413d490 msgr2=0x7f288413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:43:50.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f288359e640 1 -- 172.21.15.12:0/3469203288 >> v2:172.21.15.12:3300/0 conn(0x7f288413d490 msgr2=0x7f288413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:43:50.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f288359e640 1 --2- 172.21.15.12:0/3469203288 >> v2:172.21.15.12:3300/0 conn(0x7f288413d490 0x7f288413f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:50.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f2882d9d640 1 -- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28840a2e90 msgr2=0x7f288406dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:43:50.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f2882d9d640 1 -- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28840a2e90 msgr2=0x7f288406dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:43:50.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f2882d9d640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28840a2e90 0x7f288406dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:50.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f288359e640 1 --2- 172.21.15.12:0/3469203288 >> v2:172.21.15.12:3300/0 conn(0x7f288413d490 0x7f288413f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:50.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.836+0000 7f2882d9d640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28840a2e90 0x7f288406dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:50.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:50.838+0000 7f288259c640 1 -- 172.21.15.12:0/3469203288 >> v1:172.21.15.12:6789/0 conn(0x7f28840a3270 legacy=0x7f28840a3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:51.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.037+0000 7f2882d9d640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28840a2e90 0x7f288406dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.037+0000 7f288359e640 1 --2- 172.21.15.12:0/3469203288 >> v2:172.21.15.12:3300/0 conn(0x7f288413d490 0x7f288413f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.037+0000 7f2882d9d640 10 monclient(hunting): get_auth_request con 0x7f28840a2e90 auth_method 0 2024-09-11T01:43:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.037+0000 7f2882d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.037+0000 7f2882d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.037+0000 7f2882d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.037+0000 7f288359e640 10 monclient(hunting): get_auth_request con 0x7f288413d490 auth_method 0 2024-09-11T01:43:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.037+0000 7f288359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.037+0000 7f288359e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.037+0000 7f288359e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.037+0000 7f288359e640 1 -- 172.21.15.12:0/3469203288 >> v2:172.21.15.12:3300/0 conn(0x7f288413d490 msgr2=0x7f288413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:43:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.037+0000 7f288359e640 1 -- 172.21.15.12:0/3469203288 >> v2:172.21.15.12:3300/0 conn(0x7f288413d490 msgr2=0x7f288413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:43:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.037+0000 7f288359e640 1 --2- 172.21.15.12:0/3469203288 >> v2:172.21.15.12:3300/0 conn(0x7f288413d490 0x7f288413f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:51.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.037+0000 7f2882d9d640 1 -- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28840a2e90 msgr2=0x7f288406dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:43:51.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.037+0000 7f2882d9d640 1 -- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28840a2e90 msgr2=0x7f288406dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:43:51.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.037+0000 7f2882d9d640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28840a2e90 0x7f288406dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:51.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.037+0000 7f288359e640 1 --2- 172.21.15.12:0/3469203288 >> v2:172.21.15.12:3300/0 conn(0x7f288413d490 0x7f288413f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:43:51.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.037+0000 7f2882d9d640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28840a2e90 0x7f288406dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:43:51.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.438+0000 7f288359e640 1 --2- 172.21.15.12:0/3469203288 >> v2:172.21.15.12:3300/0 conn(0x7f288413d490 0x7f288413f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:51.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.438+0000 7f2882d9d640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28840a2e90 0x7f288406dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:51.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.438+0000 7f2882d9d640 10 monclient(hunting): get_auth_request con 0x7f28840a2e90 auth_method 0 2024-09-11T01:43:51.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.438+0000 7f2882d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:51.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.438+0000 7f2882d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:51.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.438+0000 7f2882d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:51.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.439+0000 7f288359e640 10 monclient(hunting): get_auth_request con 0x7f288413d490 auth_method 0 2024-09-11T01:43:51.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.439+0000 7f288359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:51.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.439+0000 7f288359e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:51.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.439+0000 7f288359e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:51.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.439+0000 7f288359e640 1 -- 172.21.15.12:0/3469203288 >> v2:172.21.15.12:3300/0 conn(0x7f288413d490 msgr2=0x7f288413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:43:51.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.439+0000 7f288359e640 1 -- 172.21.15.12:0/3469203288 >> v2:172.21.15.12:3300/0 conn(0x7f288413d490 msgr2=0x7f288413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:43:51.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.439+0000 7f288359e640 1 --2- 172.21.15.12:0/3469203288 >> v2:172.21.15.12:3300/0 conn(0x7f288413d490 0x7f288413f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:51.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.439+0000 7f2882d9d640 1 -- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28840a2e90 msgr2=0x7f288406dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:43:51.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.439+0000 7f2882d9d640 1 -- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28840a2e90 msgr2=0x7f288406dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:43:51.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.439+0000 7f2882d9d640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28840a2e90 0x7f288406dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:51.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.439+0000 7f2882d9d640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28840a2e90 0x7f288406dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:43:51.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:51.439+0000 7f288359e640 1 --2- 172.21.15.12:0/3469203288 >> v2:172.21.15.12:3300/0 conn(0x7f288413d490 0x7f288413f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:43:52.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:52.240+0000 7f288359e640 1 --2- 172.21.15.12:0/3469203288 >> v2:172.21.15.12:3300/0 conn(0x7f288413d490 0x7f288413f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:52.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:52.240+0000 7f2882d9d640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28840a2e90 0x7f288406dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:52.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:52.240+0000 7f2882d9d640 10 monclient(hunting): get_auth_request con 0x7f28840a2e90 auth_method 0 2024-09-11T01:43:52.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:52.240+0000 7f2882d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:52.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:52.240+0000 7f2882d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:52.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:52.240+0000 7f2882d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:52.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:52.240+0000 7f288359e640 10 monclient(hunting): get_auth_request con 0x7f288413d490 auth_method 0 2024-09-11T01:43:52.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:52.240+0000 7f288359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:52.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:52.240+0000 7f288359e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:52.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:52.240+0000 7f288359e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:52.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:52.240+0000 7f2882d9d640 1 -- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28840a2e90 msgr2=0x7f288406dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:43:52.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:52.240+0000 7f2882d9d640 1 -- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28840a2e90 msgr2=0x7f288406dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:43:52.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:52.240+0000 7f2882d9d640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28840a2e90 0x7f288406dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:52.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:52.240+0000 7f288359e640 1 -- 172.21.15.12:0/3469203288 >> v2:172.21.15.12:3300/0 conn(0x7f288413d490 msgr2=0x7f288413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:43:52.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:52.240+0000 7f288359e640 1 -- 172.21.15.12:0/3469203288 >> v2:172.21.15.12:3300/0 conn(0x7f288413d490 msgr2=0x7f288413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:43:52.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:52.240+0000 7f288359e640 1 --2- 172.21.15.12:0/3469203288 >> v2:172.21.15.12:3300/0 conn(0x7f288413d490 0x7f288413f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:52.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:52.240+0000 7f2882d9d640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28840a2e90 0x7f288406dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:43:52.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:52.240+0000 7f288359e640 1 --2- 172.21.15.12:0/3469203288 >> v2:172.21.15.12:3300/0 conn(0x7f288413d490 0x7f288413f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:43:53.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.836+0000 7f2880d99640 10 monclient(hunting): tick 2024-09-11T01:43:53.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.836+0000 7f2880d99640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:43:53.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.836+0000 7f2880d99640 1 monclient(hunting): continuing hunt 2024-09-11T01:43:53.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.836+0000 7f2880d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:43:53.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.836+0000 7f2880d99640 1 -- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28840a2e90 msgr2=0x7f288406dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:43:53.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.836+0000 7f2880d99640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28840a2e90 0x7f288406dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:53.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.836+0000 7f2880d99640 1 -- 172.21.15.12:0/3469203288 >> v2:172.21.15.12:3300/0 conn(0x7f288413d490 msgr2=0x7f288413f880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:43:53.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.836+0000 7f2880d99640 1 --2- 172.21.15.12:0/3469203288 >> v2:172.21.15.12:3300/0 conn(0x7f288413d490 0x7f288413f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:53.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.836+0000 7f2880d99640 1 -- 172.21.15.12:0/3469203288 >> v1:172.21.15.12:6789/0 conn(0x7f28840a3270 legacy=0x7f28840a3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:53.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.836+0000 7f2880d99640 10 monclient: _add_conns ranks=[4,1] 2024-09-11T01:43:53.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.836+0000 7f2880d99640 1 --2- 172.21.15.12:0/3469203288 >> v2:172.21.15.71:3300/0 conn(0x7f2870003740 0x7f2870005ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:53.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.836+0000 7f2880d99640 10 monclient(hunting): picked mon.noname-c con 0x7f2870003740 addr v2:172.21.15.71:3300/0 2024-09-11T01:43:53.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.836+0000 7f2880d99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2870006290 addr v1:172.21.15.71:6789/0 2024-09-11T01:43:53.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.836+0000 7f2880d99640 1 -- 172.21.15.12:0/3469203288 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f288406eb70 con 0x7f2870006290 2024-09-11T01:43:53.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.836+0000 7f2880d99640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:53.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.836+0000 7f2880d99640 1 -- 172.21.15.12:0/3469203288 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f28840a4900 con 0x7f2870003740 2024-09-11T01:43:53.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.836+0000 7f2880d99640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:53.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.836+0000 7f2882d9d640 1 --2- 172.21.15.12:0/3469203288 >> v2:172.21.15.71:3300/0 conn(0x7f2870003740 0x7f2870005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:53.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.836+0000 7f2882d9d640 10 monclient(hunting): get_auth_request con 0x7f2870003740 auth_method 0 2024-09-11T01:43:53.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.836+0000 7f2882d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:53.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.836+0000 7f2882d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:53.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.836+0000 7f2882d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:53.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.836+0000 7f2882d9d640 1 -- 172.21.15.12:0/3469203288 >> v2:172.21.15.71:3300/0 conn(0x7f2870003740 msgr2=0x7f2870005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:43:53.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.836+0000 7f2882d9d640 1 -- 172.21.15.12:0/3469203288 >> v2:172.21.15.71:3300/0 conn(0x7f2870003740 msgr2=0x7f2870005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:43:53.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.836+0000 7f2882d9d640 1 --2- 172.21.15.12:0/3469203288 >> v2:172.21.15.71:3300/0 conn(0x7f2870003740 0x7f2870005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:53.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.837+0000 7f2882d9d640 1 --2- 172.21.15.12:0/3469203288 >> v2:172.21.15.71:3300/0 conn(0x7f2870003740 0x7f2870005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:53.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:53.839+0000 7f288259c640 1 -- 172.21.15.12:0/3469203288 >> v1:172.21.15.71:6789/0 conn(0x7f2870006290 legacy=0x7f2870008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:54.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.038+0000 7f2882d9d640 1 --2- 172.21.15.12:0/3469203288 >> v2:172.21.15.71:3300/0 conn(0x7f2870003740 0x7f2870005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:54.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.038+0000 7f2882d9d640 10 monclient(hunting): get_auth_request con 0x7f2870003740 auth_method 0 2024-09-11T01:43:54.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.038+0000 7f2882d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:54.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.038+0000 7f2882d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:54.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.038+0000 7f2882d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:54.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.038+0000 7f2882d9d640 1 -- 172.21.15.12:0/3469203288 >> v2:172.21.15.71:3300/0 conn(0x7f2870003740 msgr2=0x7f2870005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:43:54.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.038+0000 7f2882d9d640 1 -- 172.21.15.12:0/3469203288 >> v2:172.21.15.71:3300/0 conn(0x7f2870003740 msgr2=0x7f2870005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:43:54.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.038+0000 7f2882d9d640 1 --2- 172.21.15.12:0/3469203288 >> v2:172.21.15.71:3300/0 conn(0x7f2870003740 0x7f2870005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:54.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.038+0000 7f2882d9d640 1 --2- 172.21.15.12:0/3469203288 >> v2:172.21.15.71:3300/0 conn(0x7f2870003740 0x7f2870005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:43:54.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 1 -- 172.21.15.12:0/3469203288 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3237898722 0 0) 0x7f288406eb70 con 0x7f2870006290 2024-09-11T01:43:54.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 10 monclient(hunting): _negotiate 2024-09-11T01:43:54.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:54.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:54.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 10 monclient(hunting): my global_id is 134356 2024-09-11T01:43:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 1 -- 172.21.15.12:0/3469203288 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f286c003520 con 0x7f2870006290 2024-09-11T01:43:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 1 -- 172.21.15.12:0/3469203288 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2156547948 0 0) 0x7f286c003520 con 0x7f2870006290 2024-09-11T01:43:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 1 -- 172.21.15.12:0/3469203288 >> v2:172.21.15.71:3300/0 conn(0x7f2870003740 msgr2=0x7f2870005ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:43:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 1 --2- 172.21.15.12:0/3469203288 >> v2:172.21.15.71:3300/0 conn(0x7f2870003740 0x7f2870005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:43:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 1 monclient: found mon.noname-c-legacy 2024-09-11T01:43:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-09-11T01:43:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 1 -- 172.21.15.12:0/3469203288 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f287000b030 con 0x7f2870006290 2024-09-11T01:43:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 10 monclient: _finish_auth 0 2024-09-11T01:43:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:43:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 1 -- 172.21.15.12:0/3469203288 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f2878001a30 con 0x7f2870006290 2024-09-11T01:43:54.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:54.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-11T01:43:54.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 10 monclient: dump: 2024-09-11T01:43:54.255 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:54.255 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:54.255 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:54.255 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:54.255 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:54.255 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:54.255 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:54.255 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:54.255 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:54.255 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-09-11T01:43:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 1 -- 172.21.15.12:0/3469203288 >> v1:172.21.15.71:6789/0 conn(0x7f2870006290 legacy=0x7f2870008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:43:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f288359e640 1 -- 172.21.15.12:0/3469203288 reap_dead start 2024-09-11T01:43:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f286c003aa0 0x7f286c003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 10 monclient(hunting): picked mon.c con 0x7f286c003aa0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f286c004800 0x7f286c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 10 monclient(hunting): picked mon.a con 0x7f286c004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f286c005410 0x7f286c00ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 10 monclient(hunting): picked mon.b con 0x7f286c005410 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 1 -- 172.21.15.12:0/3469203288 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f28840a4900 con 0x7f286c004800 2024-09-11T01:43:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 1 -- 172.21.15.12:0/3469203288 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f286c00c320 con 0x7f286c005410 2024-09-11T01:43:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 1 -- 172.21.15.12:0/3469203288 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f286c00c520 con 0x7f286c003aa0 2024-09-11T01:43:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2881d9b640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:43:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2882d9d640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f286c003aa0 0x7f286c003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2882d9d640 10 monclient(hunting): get_auth_request con 0x7f286c003aa0 auth_method 0 2024-09-11T01:43:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2882d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2882d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2882d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f288359e640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f286c005410 0x7f286c00ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f288359e640 10 monclient(hunting): get_auth_request con 0x7f286c005410 auth_method 0 2024-09-11T01:43:54.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f288359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:54.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f288359e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:54.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f288359e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:54.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2882d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:54.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.249+0000 7f2882d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:54.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.250+0000 7f2882d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:43:54.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.250+0000 7f288359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:54.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.250+0000 7f288359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:54.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.250+0000 7f288359e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:43:54.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.250+0000 7f288259c640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f286c004800 0x7f286c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:54.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.250+0000 7f288259c640 10 monclient(hunting): get_auth_request con 0x7f286c004800 auth_method 0 2024-09-11T01:43:54.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.250+0000 7f288259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:54.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.250+0000 7f288259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:54.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.250+0000 7f288259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:54.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.250+0000 7f2882d9d640 10 monclient(hunting): handle_auth_done global_id 134356 payload 293 2024-09-11T01:43:54.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.250+0000 7f2882d9d640 1 -- 172.21.15.12:0/3469203288 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f286c005410 msgr2=0x7f286c00ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:54.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.250+0000 7f2882d9d640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f286c005410 0x7f286c00ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:54.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.250+0000 7f2882d9d640 1 -- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f286c004800 msgr2=0x7f286c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:54.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.250+0000 7f2882d9d640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f286c004800 0x7f286c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:54.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.250+0000 7f2882d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:43:54.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.250+0000 7f2882d9d640 1 monclient: found mon.c 2024-09-11T01:43:54.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.250+0000 7f2882d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:54.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.250+0000 7f2882d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:43:54.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.250+0000 7f2882d9d640 1 -- 172.21.15.12:0/3469203288 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f286c003520 con 0x7f286c003aa0 2024-09-11T01:43:54.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.250+0000 7f2882d9d640 10 monclient: _finish_auth 0 2024-09-11T01:43:54.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.250+0000 7f2882d9d640 10 monclient: _check_auth_tickets 2024-09-11T01:43:54.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.250+0000 7f2882d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:54.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.250+0000 7f288359e640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f286c005410 0x7f286c00ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:43:54.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.250+0000 7f28891f4640 5 monclient: authenticate success, global_id 134356 2024-09-11T01:43:54.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.250+0000 7f28891f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:43:54.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.250+0000 7f288259c640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f286c004800 0x7f286c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:43:54.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.253+0000 7f2882d9d640 1 -- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f286c003aa0 msgr2=0x7f286c003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:54.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.253+0000 7f2882d9d640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f286c003aa0 0x7f286c003ea0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2874015c50 tx=0x7f286c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4749d09ae116abe1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:54.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.253+0000 7f2881d9b640 1 -- 172.21.15.12:0/3469203288 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2874032070 con 0x7f286c003aa0 2024-09-11T01:43:54.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.253+0000 7f2881d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:54.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.253+0000 7f2881d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:43:54.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.253+0000 7f2881d9b640 10 monclient: dump: 2024-09-11T01:43:54.266 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:54.266 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:54.266 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:54.266 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:54.266 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:54.266 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:54.267 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:54.267 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:54.267 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:54.267 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:54.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.253+0000 7f28891f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:43:54.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.253+0000 7f2881d9b640 1 -- 172.21.15.12:0/3469203288 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f287402c5c0 con 0x7f286c003aa0 2024-09-11T01:43:54.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.253+0000 7f2881d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:54.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.253+0000 7f28891f4640 10 monclient: get_monmap_and_config success 2024-09-11T01:43:54.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.253+0000 7f28891f4640 4 set_mon_vals no callback set 2024-09-11T01:43:54.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.253+0000 7f2881d9b640 1 -- 172.21.15.12:0/3469203288 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2874015850 con 0x7f286c003aa0 2024-09-11T01:43:54.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.253+0000 7f2881d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:54.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.253+0000 7f2881d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:43:54.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.253+0000 7f2881d9b640 10 monclient: dump: 2024-09-11T01:43:54.268 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:54.268 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:54.268 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:54.268 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:54.268 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:54.268 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:54.268 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:54.268 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:54.268 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:54.268 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:54.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.254+0000 7f28891f4640 10 monclient: shutdown 2024-09-11T01:43:54.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.254+0000 7f28891f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:54.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.254+0000 7f28891f4640 1 -- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f286c003aa0 msgr2=0x7f286c003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:54.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.254+0000 7f28891f4640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f286c003aa0 0x7f286c003ea0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2874015c50 tx=0x7f286c005830 comp rx=0 tx=0).stop 2024-09-11T01:43:54.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.254+0000 7f28891f4640 1 -- 172.21.15.12:0/3469203288 shutdown_connections 2024-09-11T01:43:54.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.254+0000 7f28891f4640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f286c005410 0x7f286c00ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:54.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.254+0000 7f28891f4640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f286c004800 0x7f286c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:54.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.254+0000 7f28891f4640 1 --2- 172.21.15.12:0/3469203288 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f286c003aa0 0x7f286c003ea0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:54.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.254+0000 7f28891f4640 1 -- 172.21.15.12:0/3469203288 >> 172.21.15.12:0/3469203288 conn(0x7f28840a8490 msgr2=0x7f28840a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:54.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.254+0000 7f28891f4640 1 -- 172.21.15.12:0/3469203288 shutdown_connections 2024-09-11T01:43:54.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.255+0000 7f28891f4640 1 -- 172.21.15.12:0/3469203288 wait complete. 2024-09-11T01:43:54.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.256+0000 7f28891f4640 10 monclient: build_initial_monmap 2024-09-11T01:43:54.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.256+0000 7f28891f4640 10 monclient: monmap: 2024-09-11T01:43:54.269 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:54.269 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:43:54.270 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:43:54.270 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:43:54.270 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:54.270 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:54.270 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:43:54.270 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:43:54.270 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:43:54.270 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:54.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.256+0000 7f28891f4640 1 Processor -- start 2024-09-11T01:43:54.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f28891f4640 1 -- start start 2024-09-11T01:43:54.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f28891f4640 10 monclient: init 2024-09-11T01:43:54.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f28891f4640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:54.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f28891f4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:43:54.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f28891f4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f286c003aa0 0x7f28841cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f28891f4640 10 monclient(hunting): picked mon.noname-b con 0x7f286c003aa0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f28891f4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f286c004800 0x7f2884137260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f28891f4640 10 monclient(hunting): picked mon.noname-a con 0x7f286c004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f28891f4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f286c005410 0x7f2884139e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f28891f4640 10 monclient(hunting): picked mon.noname-c con 0x7f286c005410 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f28891f4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f28891f4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f288406dde0 con 0x7f286c004800 2024-09-11T01:43:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f28891f4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f28891f4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f288406dc60 con 0x7f286c003aa0 2024-09-11T01:43:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f28891f4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f28891f4640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f288406df60 con 0x7f286c005410 2024-09-11T01:43:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f28891f4640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f28891f4640 10 monclient(hunting): authenticate will time out at 1640.699635s 2024-09-11T01:43:54.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f288259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f286c004800 0x7f2884137260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f288359e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f286c005410 0x7f2884139e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f288259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f286c004800 0x7f2884137260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35756/0 (socket says 172.21.15.12:35756) 2024-09-11T01:43:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f288259c640 1 -- 172.21.15.12:0/1932888606 learned_addr learned my addr 172.21.15.12:0/1932888606 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f288259c640 10 monclient(hunting): get_auth_request con 0x7f286c004800 auth_method 0 2024-09-11T01:43:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f288259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f288259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f288259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f288359e640 10 monclient(hunting): get_auth_request con 0x7f286c005410 auth_method 0 2024-09-11T01:43:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f288359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f288359e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f288359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f2882d9d640 1 --2- 172.21.15.12:0/1932888606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f286c003aa0 0x7f28841cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:54.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f2882d9d640 10 monclient(hunting): get_auth_request con 0x7f286c003aa0 auth_method 0 2024-09-11T01:43:54.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f2882d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:54.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f2882d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:54.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f2882d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:54.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f288259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:54.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f288259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:54.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f288359e640 1 -- 172.21.15.12:0/1932888606 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f286c005410 msgr2=0x7f2884139e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:43:54.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f288359e640 1 -- 172.21.15.12:0/1932888606 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f286c005410 msgr2=0x7f2884139e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:43:54.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f288259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:54.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.257+0000 7f288359e640 1 --2- 172.21.15.12:0/1932888606 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f286c005410 0x7f2884139e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:54.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.258+0000 7f2882d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:54.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.258+0000 7f2882d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:54.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.258+0000 7f2882d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:54.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.258+0000 7f288359e640 1 --2- 172.21.15.12:0/1932888606 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f286c005410 0x7f2884139e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.258+0000 7f288259c640 10 monclient(hunting): handle_auth_done global_id 94758 payload 824 2024-09-11T01:43:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.258+0000 7f288259c640 1 -- 172.21.15.12:0/1932888606 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f286c005410 msgr2=0x7f2884139e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:43:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.258+0000 7f288259c640 1 --2- 172.21.15.12:0/1932888606 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f286c005410 0x7f2884139e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.258+0000 7f288259c640 1 -- 172.21.15.12:0/1932888606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f286c003aa0 msgr2=0x7f28841cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.258+0000 7f288259c640 1 --2- 172.21.15.12:0/1932888606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f286c003aa0 0x7f28841cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.258+0000 7f288259c640 10 monclient: _finish_hunting 0 2024-09-11T01:43:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.258+0000 7f288259c640 1 monclient: found mon.noname-a 2024-09-11T01:43:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.258+0000 7f288259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.258+0000 7f288259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.258+0000 7f288259c640 1 -- 172.21.15.12:0/1932888606 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f288409f680 con 0x7f286c004800 2024-09-11T01:43:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.258+0000 7f2882d9d640 1 --2- 172.21.15.12:0/1932888606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f286c003aa0 0x7f28841cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:43:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.260+0000 7f288259c640 1 -- 172.21.15.12:0/1932888606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f286c004800 msgr2=0x7f2884137260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:54.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.260+0000 7f288259c640 1 --2- 172.21.15.12:0/1932888606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f286c004800 0x7f2884137260 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f287800e7a0 tx=0x7f287800ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.260+0000 7f286b7fe640 1 -- 172.21.15.12:0/1932888606 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f287800ee50 con 0x7f286c004800 2024-09-11T01:43:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.260+0000 7f286b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.260+0000 7f286b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.260+0000 7f286b7fe640 10 monclient: dump: 2024-09-11T01:43:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:54.275 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:54.275 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.260+0000 7f286b7fe640 10 monclient: _finish_auth 0 2024-09-11T01:43:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.260+0000 7f286b7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:43:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.260+0000 7f286b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.260+0000 7f286b7fe640 1 -- 172.21.15.12:0/1932888606 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2878018070 con 0x7f286c004800 2024-09-11T01:43:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.260+0000 7f286b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.260+0000 7f286b7fe640 1 -- 172.21.15.12:0/1932888606 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28780057c0 con 0x7f286c004800 2024-09-11T01:43:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.260+0000 7f28891f4640 5 monclient: authenticate success, global_id 94758 2024-09-11T01:43:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.260+0000 7f28891f4640 10 monclient: _renew_subs 2024-09-11T01:43:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.260+0000 7f28891f4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.260+0000 7f28891f4640 1 -- 172.21.15.12:0/1932888606 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f288413b940 con 0x7f286c004800 2024-09-11T01:43:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.260+0000 7f286b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.260+0000 7f286b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:54.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.260+0000 7f286b7fe640 10 monclient: dump: 2024-09-11T01:43:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:54.277 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.260+0000 7f288159a640 4 set_mon_vals no callback set 2024-09-11T01:43:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.261+0000 7f28891f4640 10 monclient: _renew_subs 2024-09-11T01:43:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.261+0000 7f28891f4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:54.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.261+0000 7f28891f4640 1 -- 172.21.15.12:0/1932888606 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2884123370 con 0x7f286c004800 2024-09-11T01:43:54.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.262+0000 7f286b7fe640 1 -- 172.21.15.12:0/1932888606 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f2878005960 con 0x7f286c004800 2024-09-11T01:43:54.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.262+0000 7f28891f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:54.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.262+0000 7f28891f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:54.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.262+0000 7f28891f4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:54.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.262+0000 7f28891f4640 1 -- 172.21.15.12:0/1932888606 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2884122ef0 con 0x7f286c004800 2024-09-11T01:43:54.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.262+0000 7f286b7fe640 1 --2- 172.21.15.12:0/1932888606 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f287003ed30 0x7f28700411f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:54.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.262+0000 7f286b7fe640 1 -- 172.21.15.12:0/1932888606 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(502..502 src has 317..502) ==== 4811+0+0 (secure 0 0 0) 0x7f2878052bb0 con 0x7f286c004800 2024-09-11T01:43:54.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.262+0000 7f2882d9d640 1 --2- 172.21.15.12:0/1932888606 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f287003ed30 0x7f28700411f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:54.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.262+0000 7f2882d9d640 10 monclient: get_auth_request con 0x7f287003ed30 auth_method 0 2024-09-11T01:43:54.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.265+0000 7f2882d9d640 1 --2- 172.21.15.12:0/1932888606 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f287003ed30 0x7f28700411f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2874034000 tx=0x7f2874032000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:54.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.267+0000 7f286b7fe640 1 -- 172.21.15.12:0/1932888606 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2878051590 con 0x7f286c004800 2024-09-11T01:43:54.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.267+0000 7f286b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:54.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.267+0000 7f286b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:43:54.319 INFO:tasks.rados.rados.0.smithi071.stdout:2255: done (2 left) 2024-09-11T01:43:54.319 INFO:tasks.rados.rados.0.smithi071.stdout:2256: rollback oid 47 current snap is 250 2024-09-11T01:43:54.319 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 47 to 219 2024-09-11T01:43:54.320 INFO:tasks.rados.rados.0.smithi071.stdout:2252: finishing rollback tid 2 to smithi07134777-15 2024-09-11T01:43:54.320 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 1339 (ObjNum 565 snap 160 seq_num 565) dirty exists 2024-09-11T01:43:54.320 INFO:tasks.rados.rados.0.smithi071.stdout:2254: expect (ObjNum 723 snap 224 seq_num 723) 2024-09-11T01:43:54.345 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 10979 2024-09-11T01:43:54.393 INFO:tasks.rados.rados.0.smithi071.stdout:2256: finishing rollback tid 0 to smithi07134777-47 2024-09-11T01:43:54.393 INFO:tasks.rados.rados.0.smithi071.stdout:2256: finishing rollback tid 1 to smithi07134777-47 2024-09-11T01:43:54.393 INFO:tasks.rados.rados.0.smithi071.stdout:2256: finishing rollback tid 2 to smithi07134777-47 2024-09-11T01:43:54.393 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 984 (ObjNum 704 snap 213 seq_num 704) dirty exists 2024-09-11T01:43:54.393 INFO:tasks.rados.rados.0.smithi071.stdout:2252: done (2 left) 2024-09-11T01:43:54.393 INFO:tasks.rados.rados.0.smithi071.stdout:2254: done (1 left) 2024-09-11T01:43:54.393 INFO:tasks.rados.rados.0.smithi071.stdout:2256: done (0 left) 2024-09-11T01:43:54.393 INFO:tasks.rados.rados.0.smithi071.stdout:2257: read oid 2 snap -1 2024-09-11T01:43:54.393 INFO:tasks.rados.rados.0.smithi071.stdout:2257: expect (ObjNum 631 snap 190 seq_num 631) 2024-09-11T01:43:54.393 INFO:tasks.rados.rados.0.smithi071.stdout:2258: write (excl) oid 33 current snap is 250 2024-09-11T01:43:54.393 INFO:tasks.rados.rados.0.smithi071.stdout:2258: seq_num 816 ranges {514512=652955,1700483=740950,3025515=688997} 2024-09-11T01:43:54.399 INFO:tasks.rados.rados.0.smithi071.stdout:2258: writing smithi07134777-33 from 514512 to 1167467 tid 1 2024-09-11T01:43:54.406 INFO:tasks.rados.rados.0.smithi071.stdout:2258: writing smithi07134777-33 from 1700483 to 2441433 tid 2 2024-09-11T01:43:54.413 INFO:tasks.rados.rados.0.smithi071.stdout:2258: writing smithi07134777-33 from 3025515 to 3714512 tid 3 2024-09-11T01:43:54.413 INFO:tasks.rados.rados.0.smithi071.stdout:2259: copy_from oid 26 from oid 14 current snap is 250 2024-09-11T01:43:54.413 INFO:tasks.rados.rados.0.smithi071.stdout:2257: expect (ObjNum 631 snap 190 seq_num 631) 2024-09-11T01:43:54.413 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:54.441 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 370905 2024-09-11T01:43:54.494 INFO:tasks.rados.rados.0.smithi071.stdout:2258: finishing write tid 1 to smithi07134777-33 2024-09-11T01:43:54.495 INFO:tasks.rados.rados.0.smithi071.stdout:2258: finishing write tid 2 to smithi07134777-33 2024-09-11T01:43:54.495 INFO:tasks.rados.rados.0.smithi071.stdout:2258: finishing write tid 3 to smithi07134777-33 2024-09-11T01:43:54.495 INFO:tasks.rados.rados.0.smithi071.stdout:2258: finishing write tid 4 to smithi07134777-33 2024-09-11T01:43:54.495 INFO:tasks.rados.rados.0.smithi071.stdout:2258: finishing write tid 5 to smithi07134777-33 2024-09-11T01:43:54.495 INFO:tasks.rados.rados.0.smithi071.stdout:2258: oid 33 updating version 0 to 1195 2024-09-11T01:43:54.495 INFO:tasks.rados.rados.0.smithi071.stdout:2258: oid 33 version 1195 is already newer than 1193 2024-09-11T01:43:54.495 INFO:tasks.rados.rados.0.smithi071.stdout:2258: oid 33 version 1195 is already newer than 1192 2024-09-11T01:43:54.495 INFO:tasks.rados.rados.0.smithi071.stdout:2258: oid 33 version 1195 is already newer than 1194 2024-09-11T01:43:54.495 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 1195 (ObjNum 816 snap 250 seq_num 816) dirty exists 2024-09-11T01:43:54.495 INFO:tasks.rados.rados.0.smithi071.stdout:2258: left oid 33 (ObjNum 816 snap 250 seq_num 816) 2024-09-11T01:43:54.495 INFO:tasks.rados.rados.0.smithi071.stdout:2259: finishing copy_from racing read to smithi07134777-26 2024-09-11T01:43:54.495 INFO:tasks.rados.rados.0.smithi071.stdout:2259: finishing copy_from to smithi07134777-26 2024-09-11T01:43:54.495 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 717 (ObjNum 797 snap 243 seq_num 797) dirty exists 2024-09-11T01:43:54.496 INFO:tasks.rados.rados.0.smithi071.stdout:2257: done (2 left) 2024-09-11T01:43:54.496 INFO:tasks.rados.rados.0.smithi071.stdout:2258: done (1 left) 2024-09-11T01:43:54.496 INFO:tasks.rados.rados.0.smithi071.stdout:2259: done (0 left) 2024-09-11T01:43:54.496 INFO:tasks.rados.rados.0.smithi071.stdout:2260: read oid 16 snap -1 2024-09-11T01:43:54.496 INFO:tasks.rados.rados.0.smithi071.stdout:2260: expect (ObjNum 411 snap 115 seq_num 411) 2024-09-11T01:43:54.496 INFO:tasks.rados.rados.0.smithi071.stdout:2261: copy_from oid 39 from oid 49 current snap is 250 2024-09-11T01:43:54.496 INFO:tasks.rados.rados.0.smithi071.stdout:2262: snap_create 2024-09-11T01:43:54.496 INFO:tasks.rados.rados.0.smithi071.stdout:2261: finishing copy_from to smithi07134777-39 2024-09-11T01:43:54.497 INFO:tasks.rados.rados.0.smithi071.stdout:2261: got expected ENOENT (src dne) 2024-09-11T01:43:54.497 INFO:tasks.rados.rados.0.smithi071.stdout:2261: finishing copy_from racing read to smithi07134777-39 2024-09-11T01:43:54.501 INFO:tasks.rados.rados.0.smithi071.stdout:2260: expect (ObjNum 411 snap 115 seq_num 411) 2024-09-11T01:43:54.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.540+0000 7f28891f4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:43:54.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.540+0000 7f28891f4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:43:54.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.540+0000 7f28891f4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:54.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.540+0000 7f28891f4640 1 -- 172.21.15.12:0/1932888606 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f28840a3650 con 0x7f286c004800 2024-09-11T01:43:54.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.540+0000 7f286b7fe640 1 -- 172.21.15.12:0/1932888606 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f287800cbe0 con 0x7f286c004800 2024-09-11T01:43:54.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.540+0000 7f286b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:43:54.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.540+0000 7f286b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:43:54.542 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":136,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:43:54.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.543+0000 7f28891f4640 1 -- 172.21.15.12:0/1932888606 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f287003ed30 msgr2=0x7f28700411f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:54.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.543+0000 7f28891f4640 1 --2- 172.21.15.12:0/1932888606 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f287003ed30 0x7f28700411f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2874034000 tx=0x7f2874032000 comp rx=0 tx=0).stop 2024-09-11T01:43:54.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.543+0000 7f28891f4640 10 monclient: shutdown 2024-09-11T01:43:54.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.543+0000 7f28891f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:54.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.543+0000 7f28891f4640 1 -- 172.21.15.12:0/1932888606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f286c004800 msgr2=0x7f2884137260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:54.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.543+0000 7f28891f4640 1 --2- 172.21.15.12:0/1932888606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f286c004800 0x7f2884137260 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f287800e7a0 tx=0x7f287800ec60 comp rx=0 tx=0).stop 2024-09-11T01:43:54.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.543+0000 7f28891f4640 1 -- 172.21.15.12:0/1932888606 shutdown_connections 2024-09-11T01:43:54.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.543+0000 7f28891f4640 1 --2- 172.21.15.12:0/1932888606 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f286c005410 0x7f2884139e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:54.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.543+0000 7f28891f4640 1 --2- 172.21.15.12:0/1932888606 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f287003ed30 0x7f28700411f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:54.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.543+0000 7f28891f4640 1 --2- 172.21.15.12:0/1932888606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f286c004800 0x7f2884137260 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:54.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.543+0000 7f28891f4640 1 --2- 172.21.15.12:0/1932888606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f286c003aa0 0x7f28841cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:54.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.543+0000 7f28891f4640 1 -- 172.21.15.12:0/1932888606 >> 172.21.15.12:0/1932888606 conn(0x7f28840a8490 msgr2=0x7f2884138250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:54.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.543+0000 7f28891f4640 1 -- 172.21.15.12:0/1932888606 shutdown_connections 2024-09-11T01:43:54.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:54.543+0000 7f28891f4640 1 -- 172.21.15.12:0/1932888606 wait complete. 2024-09-11T01:43:54.552 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:55.714 INFO:tasks.rados.rados.0.smithi071.stdout:2260: done (2 left) 2024-09-11T01:43:55.715 INFO:tasks.rados.rados.0.smithi071.stdout:2261: done (1 left) 2024-09-11T01:43:55.715 INFO:tasks.rados.rados.0.smithi071.stdout:2262: done (0 left) 2024-09-11T01:43:55.715 INFO:tasks.rados.rados.0.smithi071.stdout:2263: copy_from oid 36 from oid 4 current snap is 251 2024-09-11T01:43:55.715 INFO:tasks.rados.rados.0.smithi071.stdout:2264: write (excl) oid 26 current snap is 251 2024-09-11T01:43:55.715 INFO:tasks.rados.rados.0.smithi071.stdout:2264: seq_num 817 ranges {531319=728204,1758965=484608,2652938=520058} 2024-09-11T01:43:55.721 INFO:tasks.rados.rados.0.smithi071.stdout:2264: writing smithi07134777-26 from 531319 to 1259523 tid 1 2024-09-11T01:43:55.726 INFO:tasks.rados.rados.0.smithi071.stdout:2264: writing smithi07134777-26 from 1758965 to 2243573 tid 2 2024-09-11T01:43:55.731 INFO:tasks.rados.rados.0.smithi071.stdout:2264: writing smithi07134777-26 from 2652938 to 3172996 tid 3 2024-09-11T01:43:55.732 INFO:tasks.rados.rados.0.smithi071.stdout:2265: write (excl) oid 44 current snap is 251 2024-09-11T01:43:55.732 INFO:tasks.rados.rados.0.smithi071.stdout:2265: seq_num 818 ranges {548126=403453,1417447=744619,2680361=447683,3680450=67676} 2024-09-11T01:43:55.736 INFO:tasks.rados.rados.0.smithi071.stdout:2265: writing smithi07134777-44 from 548126 to 951579 tid 1 2024-09-11T01:43:55.743 INFO:tasks.rados.rados.0.smithi071.stdout:2265: writing smithi07134777-44 from 1417447 to 2162066 tid 2 2024-09-11T01:43:55.749 INFO:tasks.rados.rados.0.smithi071.stdout:2265: writing smithi07134777-44 from 2680361 to 3128044 tid 3 2024-09-11T01:43:55.753 INFO:tasks.rados.rados.0.smithi071.stdout:2265: writing smithi07134777-44 from 3680450 to 3748126 tid 4 2024-09-11T01:43:55.753 INFO:tasks.rados.rados.0.smithi071.stdout:2266: delete oid 25 current snap is 251 2024-09-11T01:43:55.753 INFO:tasks.rados.rados.0.smithi071.stdout:2263: finishing copy_from to smithi07134777-36 2024-09-11T01:43:55.753 INFO:tasks.rados.rados.0.smithi071.stdout:2263: got expected ENOENT (src dne) 2024-09-11T01:43:55.753 INFO:tasks.rados.rados.0.smithi071.stdout:2263: finishing copy_from racing read to smithi07134777-36 2024-09-11T01:43:55.754 INFO:tasks.rados.rados.0.smithi071.stdout:2264: finishing write tid 1 to smithi07134777-26 2024-09-11T01:43:55.754 INFO:tasks.rados.rados.0.smithi071.stdout:2264: finishing write tid 2 to smithi07134777-26 2024-09-11T01:43:55.754 INFO:tasks.rados.rados.0.smithi071.stdout:2264: finishing write tid 3 to smithi07134777-26 2024-09-11T01:43:55.754 INFO:tasks.rados.rados.0.smithi071.stdout:2264: finishing write tid 4 to smithi07134777-26 2024-09-11T01:43:55.754 INFO:tasks.rados.rados.0.smithi071.stdout:2264: finishing write tid 5 to smithi07134777-26 2024-09-11T01:43:55.754 INFO:tasks.rados.rados.0.smithi071.stdout:2264: oid 26 updating version 0 to 1199 2024-09-11T01:43:55.754 INFO:tasks.rados.rados.0.smithi071.stdout:2264: oid 26 updating version 1199 to 1200 2024-09-11T01:43:55.754 INFO:tasks.rados.rados.0.smithi071.stdout:2264: oid 26 updating version 1200 to 1201 2024-09-11T01:43:55.754 INFO:tasks.rados.rados.0.smithi071.stdout:2264: oid 26 updating version 1201 to 1202 2024-09-11T01:43:55.754 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 1202 (ObjNum 817 snap 251 seq_num 817) dirty exists 2024-09-11T01:43:55.754 INFO:tasks.rados.rados.0.smithi071.stdout:2264: left oid 26 (ObjNum 817 snap 251 seq_num 817) 2024-09-11T01:43:55.754 INFO:tasks.rados.rados.0.smithi071.stdout:2265: finishing write tid 1 to smithi07134777-44 2024-09-11T01:43:55.754 INFO:tasks.rados.rados.0.smithi071.stdout:2265: finishing write tid 2 to smithi07134777-44 2024-09-11T01:43:55.754 INFO:tasks.rados.rados.0.smithi071.stdout:2265: finishing write tid 3 to smithi07134777-44 2024-09-11T01:43:55.755 INFO:tasks.rados.rados.0.smithi071.stdout:2265: finishing write tid 4 to smithi07134777-44 2024-09-11T01:43:55.755 INFO:tasks.rados.rados.0.smithi071.stdout:2265: finishing write tid 5 to smithi07134777-44 2024-09-11T01:43:55.755 INFO:tasks.rados.rados.0.smithi071.stdout:2265: finishing write tid 6 to smithi07134777-44 2024-09-11T01:43:55.755 INFO:tasks.rados.rados.0.smithi071.stdout:2265: oid 44 updating version 0 to 991 2024-09-11T01:43:55.756 INFO:tasks.rados.rados.0.smithi071.stdout:2265: oid 44 version 991 is already newer than 990 2024-09-11T01:43:55.756 INFO:tasks.rados.rados.0.smithi071.stdout:2265: oid 44 updating version 991 to 992 2024-09-11T01:43:55.756 INFO:tasks.rados.rados.0.smithi071.stdout:2265: oid 44 updating version 992 to 994 2024-09-11T01:43:55.756 INFO:tasks.rados.rados.0.smithi071.stdout:2265: oid 44 version 994 is already newer than 993 2024-09-11T01:43:55.756 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 994 (ObjNum 818 snap 251 seq_num 818) dirty exists 2024-09-11T01:43:55.756 INFO:tasks.rados.rados.0.smithi071.stdout:2265: left oid 44 (ObjNum 818 snap 251 seq_num 818) 2024-09-11T01:43:55.756 INFO:tasks.rados.rados.0.smithi071.stdout:2263: done (3 left) 2024-09-11T01:43:55.756 INFO:tasks.rados.rados.0.smithi071.stdout:2264: done (2 left) 2024-09-11T01:43:55.756 INFO:tasks.rados.rados.0.smithi071.stdout:2265: done (1 left) 2024-09-11T01:43:55.756 INFO:tasks.rados.rados.0.smithi071.stdout:2266: done (0 left) 2024-09-11T01:43:55.756 INFO:tasks.rados.rados.0.smithi071.stdout:2267: snap_remove snap 206 2024-09-11T01:43:56.715 INFO:tasks.rados.rados.0.smithi071.stdout:2267: done (0 left) 2024-09-11T01:43:56.715 INFO:tasks.rados.rados.0.smithi071.stdout:2268: delete oid 12 current snap is 251 2024-09-11T01:43:56.727 INFO:tasks.rados.rados.0.smithi071.stdout:2268: done (0 left) 2024-09-11T01:43:56.727 INFO:tasks.rados.rados.0.smithi071.stdout:2269: write oid 19 current snap is 251 2024-09-11T01:43:56.727 INFO:tasks.rados.rados.0.smithi071.stdout:2269: seq_num 821 ranges {598547=629200,1876540=491946,3046277=681499} 2024-09-11T01:43:56.733 INFO:tasks.rados.rados.0.smithi071.stdout:2269: writing smithi07134777-19 from 598547 to 1227747 tid 1 2024-09-11T01:43:56.740 INFO:tasks.rados.rados.0.smithi071.stdout:2269: writing smithi07134777-19 from 1876540 to 2368486 tid 2 2024-09-11T01:43:56.748 INFO:tasks.rados.rados.0.smithi071.stdout:2269: writing smithi07134777-19 from 3046277 to 3727776 tid 3 2024-09-11T01:43:56.748 INFO:tasks.rados.rados.0.smithi071.stdout:2270: delete oid 40 current snap is 251 2024-09-11T01:43:56.748 INFO:tasks.rados.rados.0.smithi071.stdout:2269: finishing write tid 1 to smithi07134777-19 2024-09-11T01:43:56.748 INFO:tasks.rados.rados.0.smithi071.stdout:2269: finishing write tid 2 to smithi07134777-19 2024-09-11T01:43:56.753 INFO:tasks.rados.rados.0.smithi071.stdout:2270: done (1 left) 2024-09-11T01:43:56.753 INFO:tasks.rados.rados.0.smithi071.stdout:2271: read oid 35 snap 243 2024-09-11T01:43:56.753 INFO:tasks.rados.rados.0.smithi071.stdout:2271: expect (ObjNum 794 snap 243 seq_num 794) 2024-09-11T01:43:56.753 INFO:tasks.rados.rados.0.smithi071.stdout:2272: delete oid 6 current snap is 251 2024-09-11T01:43:56.757 INFO:tasks.rados.rados.0.smithi071.stdout:2269: finishing write tid 3 to smithi07134777-19 2024-09-11T01:43:56.757 INFO:tasks.rados.rados.0.smithi071.stdout:2269: finishing write tid 4 to smithi07134777-19 2024-09-11T01:43:56.757 INFO:tasks.rados.rados.0.smithi071.stdout:2269: finishing write tid 5 to smithi07134777-19 2024-09-11T01:43:56.757 INFO:tasks.rados.rados.0.smithi071.stdout:2269: oid 19 updating version 0 to 1043 2024-09-11T01:43:56.757 INFO:tasks.rados.rados.0.smithi071.stdout:2269: oid 19 version 1043 is already newer than 1042 2024-09-11T01:43:56.758 INFO:tasks.rados.rados.0.smithi071.stdout:2269: oid 19 updating version 1043 to 1044 2024-09-11T01:43:56.758 INFO:tasks.rados.rados.0.smithi071.stdout:2269: oid 19 version 1044 is already newer than 1041 2024-09-11T01:43:56.758 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 1044 (ObjNum 821 snap 251 seq_num 821) dirty exists 2024-09-11T01:43:56.758 INFO:tasks.rados.rados.0.smithi071.stdout:2269: left oid 19 (ObjNum 821 snap 251 seq_num 821) 2024-09-11T01:43:56.760 INFO:tasks.rados.rados.0.smithi071.stdout:2269: done (2 left) 2024-09-11T01:43:56.760 INFO:tasks.rados.rados.0.smithi071.stdout:2272: done (1 left) 2024-09-11T01:43:56.760 INFO:tasks.rados.rados.0.smithi071.stdout:2273: delete oid 3 current snap is 251 2024-09-11T01:43:56.764 INFO:tasks.rados.rados.0.smithi071.stdout:2271: expect (ObjNum 794 snap 243 seq_num 794) 2024-09-11T01:43:56.886 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:43:56.953 INFO:tasks.rados.rados.0.smithi071.stdout:2271: done (1 left) 2024-09-11T01:43:56.954 INFO:tasks.rados.rados.0.smithi071.stdout:2273: done (0 left) 2024-09-11T01:43:56.954 INFO:tasks.rados.rados.0.smithi071.stdout:2274: snap_remove snap 242 2024-09-11T01:43:57.556 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:43:57.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.668+0000 7f4106acd640 10 monclient: get_monmap_and_config 2024-09-11T01:43:57.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.668+0000 7f4106acd640 10 monclient: build_initial_monmap 2024-09-11T01:43:57.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.668+0000 7f4106acd640 10 monclient: monmap: 2024-09-11T01:43:57.670 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:57.670 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:57.670 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:43:57.669399+0000 2024-09-11T01:43:57.670 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:43:57.669399+0000 2024-09-11T01:43:57.671 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:57.671 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:57.671 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:43:57.671 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:43:57.671 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:43:57.671 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:43:57.671 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:43:57.671 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:57.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.669+0000 7f4106acd640 1 Processor -- start 2024-09-11T01:43:57.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.669+0000 7f4106acd640 1 -- start start 2024-09-11T01:43:57.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.669+0000 7f4106acd640 10 monclient: init 2024-09-11T01:43:57.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.669+0000 7f4106acd640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:57.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.669+0000 7f4106acd640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-11T01:43:57.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.669+0000 7f4106acd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f41000a1380 0x7f41000a1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:57.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.669+0000 7f4106acd640 10 monclient(hunting): picked mon.noname-c con 0x7f41000a1380 addr v2:172.21.15.71:3300/0 2024-09-11T01:43:57.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.669+0000 7f4106acd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f41000a0c90 0x7f4100135ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:57.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f4106acd640 10 monclient(hunting): picked mon.noname-b con 0x7f41000a0c90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:57.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f4106acd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4100136020 addr v1:172.21.15.12:6789/0 2024-09-11T01:43:57.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f4106acd640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41000a2b40 con 0x7f4100136020 2024-09-11T01:43:57.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f4106acd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:57.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f4106acd640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f410009f9a0 con 0x7f41000a1380 2024-09-11T01:43:57.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f4106acd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:57.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f4106acd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f41000a2230 con 0x7f41000a0c90 2024-09-11T01:43:57.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f4106acd640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:57.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f4106acd640 10 monclient(hunting): authenticate will time out at 1644.112075s 2024-09-11T01:43:57.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f40fffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f41000a0c90 0x7f4100135ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:57.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f410506a640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4100136020 0x7f410013ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53754/0 (socket says 172.21.15.12:53754) 2024-09-11T01:43:57.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f410506a640 1 -- 172.21.15.12:0/3431352311 learned_addr learned my addr 172.21.15.12:0/3431352311 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:43:57.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f40fffff640 10 monclient(hunting): get_auth_request con 0x7f41000a0c90 auth_method 0 2024-09-11T01:43:57.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f40fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:57.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f40fffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:57.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f40fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:57.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f40fffff640 1 -- 172.21.15.12:0/3431352311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f41000a0c90 msgr2=0x7f4100135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:43:57.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f40fffff640 1 -- 172.21.15.12:0/3431352311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f41000a0c90 msgr2=0x7f4100135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:43:57.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f40fffff640 1 --2- 172.21.15.12:0/3431352311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f41000a0c90 0x7f4100135ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:57.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f40fffff640 1 --2- 172.21.15.12:0/3431352311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f41000a0c90 0x7f4100135ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:57.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f4104869640 1 --2- 172.21.15.12:0/3431352311 >> v2:172.21.15.71:3300/0 conn(0x7f41000a1380 0x7f41000a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:57.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f4104869640 10 monclient(hunting): get_auth_request con 0x7f41000a1380 auth_method 0 2024-09-11T01:43:57.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f4104869640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:57.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f4104869640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:57.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f4104869640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:57.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f4104869640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:57.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f4104869640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:57.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f4104869640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:57.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f4104869640 10 monclient(hunting): handle_auth_done global_id 134362 payload 274 2024-09-11T01:43:57.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f4104869640 1 -- 172.21.15.12:0/3431352311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f41000a0c90 msgr2=0x7f4100135ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:43:57.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f4104869640 1 --2- 172.21.15.12:0/3431352311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f41000a0c90 0x7f4100135ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:57.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f4104869640 1 -- 172.21.15.12:0/3431352311 >> v1:172.21.15.12:6789/0 conn(0x7f4100136020 legacy=0x7f410013ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:57.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f4104869640 10 monclient: _finish_hunting 0 2024-09-11T01:43:57.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f4104869640 1 monclient: found mon.noname-c 2024-09-11T01:43:57.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f4104869640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:57.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f4104869640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:43:57.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.670+0000 7f4104869640 1 -- 172.21.15.12:0/3431352311 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41000a2510 con 0x7f41000a1380 2024-09-11T01:43:57.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.673+0000 7f4104869640 1 -- 172.21.15.12:0/3431352311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41000a1380 msgr2=0x7f41000a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:57.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.673+0000 7f4104869640 1 --2- 172.21.15.12:0/3431352311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41000a1380 0x7f41000a1760 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f40f40097a0 tx=0x7f40f4029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7eef9c36ce351759 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:57.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.674+0000 7f40ff7fe640 1 -- 172.21.15.12:0/3431352311 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f4037070 con 0x7f41000a1380 2024-09-11T01:43:57.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.674+0000 7f40ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:57.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.674+0000 7f40ff7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:43:57.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.674+0000 7f40ff7fe640 10 monclient: dump: 2024-09-11T01:43:57.685 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:57.685 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:57.685 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:57.685 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:57.685 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:57.685 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:57.685 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:57.685 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:57.685 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:57.686 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:57.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.674+0000 7f40ff7fe640 10 monclient: _finish_auth 0 2024-09-11T01:43:57.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.674+0000 7f40ff7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:43:57.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.674+0000 7f40ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:57.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.674+0000 7f40ff7fe640 1 -- 172.21.15.12:0/3431352311 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40f4032440 con 0x7f41000a1380 2024-09-11T01:43:57.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.674+0000 7f40ff7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:57.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.674+0000 7f40ff7fe640 1 -- 172.21.15.12:0/3431352311 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f4032810 con 0x7f41000a1380 2024-09-11T01:43:57.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.674+0000 7f4106acd640 5 monclient: authenticate success, global_id 134362 2024-09-11T01:43:57.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.674+0000 7f4106acd640 10 monclient: get_monmap_and_config success 2024-09-11T01:43:57.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.674+0000 7f4106acd640 4 set_mon_vals no callback set 2024-09-11T01:43:57.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.674+0000 7f40ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:57.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.674+0000 7f40ff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:43:57.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.674+0000 7f40ff7fe640 10 monclient: dump: 2024-09-11T01:43:57.687 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:57.687 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:57.687 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:57.687 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:57.687 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:57.687 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:57.687 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:57.687 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:57.687 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:57.688 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:57.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.674+0000 7f4106acd640 10 monclient: shutdown 2024-09-11T01:43:57.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.674+0000 7f4106acd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:57.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.674+0000 7f4106acd640 1 -- 172.21.15.12:0/3431352311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41000a1380 msgr2=0x7f41000a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:57.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.674+0000 7f4106acd640 1 --2- 172.21.15.12:0/3431352311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41000a1380 0x7f41000a1760 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f40f40097a0 tx=0x7f40f4029fd0 comp rx=0 tx=0).stop 2024-09-11T01:43:57.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.674+0000 7f4106acd640 1 -- 172.21.15.12:0/3431352311 shutdown_connections 2024-09-11T01:43:57.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.674+0000 7f4106acd640 1 --2- 172.21.15.12:0/3431352311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f41000a0c90 0x7f4100135ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:57.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.674+0000 7f4106acd640 1 --2- 172.21.15.12:0/3431352311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41000a1380 0x7f41000a1760 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:57.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.674+0000 7f4106acd640 1 -- 172.21.15.12:0/3431352311 >> 172.21.15.12:0/3431352311 conn(0x7f41000a2d90 msgr2=0x7f41000a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:57.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.674+0000 7f4106acd640 1 -- 172.21.15.12:0/3431352311 shutdown_connections 2024-09-11T01:43:57.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.674+0000 7f4106acd640 1 -- 172.21.15.12:0/3431352311 wait complete. 2024-09-11T01:43:57.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.675+0000 7f4106acd640 10 monclient: build_initial_monmap 2024-09-11T01:43:57.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.675+0000 7f4106acd640 10 monclient: monmap: 2024-09-11T01:43:57.689 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:57.689 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:43:57.689 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:43:57.689 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:43:57.689 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:57.689 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:57.689 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:43:57.689 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:43:57.689 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:43:57.689 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:57.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.675+0000 7f4106acd640 1 Processor -- start 2024-09-11T01:43:57.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.675+0000 7f4106acd640 1 -- start start 2024-09-11T01:43:57.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.675+0000 7f4106acd640 10 monclient: init 2024-09-11T01:43:57.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f4106acd640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:57.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f4106acd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:43:57.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f4106acd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41000a0c90 0x7f410009da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:57.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f4106acd640 10 monclient(hunting): picked mon.noname-b con 0x7f41000a0c90 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:57.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f4106acd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41000a1380 0x7f410009df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:57.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f4106acd640 10 monclient(hunting): picked mon.noname-a con 0x7f41000a1380 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:57.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f4106acd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4100136020 0x7f410009e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:57.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f4106acd640 10 monclient(hunting): picked mon.noname-c con 0x7f4100136020 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:57.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f4106acd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:57.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f4106acd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f410009ce20 con 0x7f41000a1380 2024-09-11T01:43:57.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f4106acd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:57.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f4106acd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f410009cca0 con 0x7f41000a0c90 2024-09-11T01:43:57.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f4106acd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:57.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f4106acd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f41000a0390 con 0x7f4100136020 2024-09-11T01:43:57.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f4106acd640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:57.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f4106acd640 10 monclient(hunting): authenticate will time out at 1644.118640s 2024-09-11T01:43:57.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f40fffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41000a1380 0x7f410009df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:57.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f410506a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4100136020 0x7f410009e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:57.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f40fffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41000a1380 0x7f410009df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35248/0 (socket says 172.21.15.12:35248) 2024-09-11T01:43:57.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f40fffff640 1 -- 172.21.15.12:0/2542077382 learned_addr learned my addr 172.21.15.12:0/2542077382 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:57.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f410506a640 10 monclient(hunting): get_auth_request con 0x7f4100136020 auth_method 0 2024-09-11T01:43:57.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f410506a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:57.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f410506a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:57.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f410506a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:57.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f40fffff640 10 monclient(hunting): get_auth_request con 0x7f41000a1380 auth_method 0 2024-09-11T01:43:57.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f40fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:57.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f40fffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:57.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f40fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:57.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f4104869640 1 --2- 172.21.15.12:0/2542077382 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41000a0c90 0x7f410009da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:57.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f40fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:57.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f40fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:57.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f410506a640 1 -- 172.21.15.12:0/2542077382 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4100136020 msgr2=0x7f410009e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:43:57.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f410506a640 1 -- 172.21.15.12:0/2542077382 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4100136020 msgr2=0x7f410009e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:43:57.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f40fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:57.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f410506a640 1 --2- 172.21.15.12:0/2542077382 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4100136020 0x7f410009e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:57.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f4104869640 10 monclient(hunting): get_auth_request con 0x7f41000a0c90 auth_method 0 2024-09-11T01:43:57.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f4104869640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:57.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f4104869640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:57.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f4104869640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:57.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f410506a640 1 --2- 172.21.15.12:0/2542077382 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4100136020 0x7f410009e4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:57.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f4104869640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:57.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f4104869640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:57.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.676+0000 7f4104869640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:57.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.677+0000 7f40fffff640 10 monclient(hunting): handle_auth_done global_id 94761 payload 824 2024-09-11T01:43:57.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.677+0000 7f40fffff640 1 -- 172.21.15.12:0/2542077382 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4100136020 msgr2=0x7f410009e4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:43:57.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.677+0000 7f40fffff640 1 --2- 172.21.15.12:0/2542077382 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4100136020 0x7f410009e4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:57.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.677+0000 7f40fffff640 1 -- 172.21.15.12:0/2542077382 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41000a0c90 msgr2=0x7f410009da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:57.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.677+0000 7f40fffff640 1 --2- 172.21.15.12:0/2542077382 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41000a0c90 0x7f410009da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:57.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.677+0000 7f40fffff640 10 monclient: _finish_hunting 0 2024-09-11T01:43:57.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.677+0000 7f40fffff640 1 monclient: found mon.noname-a 2024-09-11T01:43:57.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.677+0000 7f40fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:57.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.677+0000 7f40fffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:57.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.677+0000 7f40fffff640 1 -- 172.21.15.12:0/2542077382 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4100128d00 con 0x7f41000a1380 2024-09-11T01:43:57.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.679+0000 7f40fffff640 1 -- 172.21.15.12:0/2542077382 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41000a1380 msgr2=0x7f410009df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:57.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.679+0000 7f40fffff640 1 --2- 172.21.15.12:0/2542077382 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41000a1380 0x7f410009df70 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f40ec00d7d0 tx=0x7f40ec00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:57.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.679+0000 7f40fd7fa640 1 -- 172.21.15.12:0/2542077382 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40ec00b8d0 con 0x7f41000a1380 2024-09-11T01:43:57.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.679+0000 7f40fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:57.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.679+0000 7f40fd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:57.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.679+0000 7f40fd7fa640 10 monclient: dump: 2024-09-11T01:43:57.697 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:57.697 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:57.697 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:57.697 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:57.697 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:57.697 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:57.697 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:57.697 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:57.697 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:57.697 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:57.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.679+0000 7f40fd7fa640 10 monclient: _finish_auth 0 2024-09-11T01:43:57.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.679+0000 7f40fd7fa640 10 monclient: _check_auth_tickets 2024-09-11T01:43:57.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.679+0000 7f40fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:57.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.679+0000 7f40fd7fa640 1 -- 172.21.15.12:0/2542077382 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40ec019070 con 0x7f41000a1380 2024-09-11T01:43:57.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.679+0000 7f40fd7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:57.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.679+0000 7f40fd7fa640 1 -- 172.21.15.12:0/2542077382 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40ec00bcd0 con 0x7f41000a1380 2024-09-11T01:43:57.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.679+0000 7f40fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:57.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.679+0000 7f40fd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:57.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.679+0000 7f40fd7fa640 10 monclient: dump: 2024-09-11T01:43:57.698 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:57.698 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:57.698 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:57.698 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:57.699 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:57.699 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:57.699 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:57.699 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:57.699 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:57.699 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:57.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.679+0000 7f4106acd640 5 monclient: authenticate success, global_id 94761 2024-09-11T01:43:57.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.679+0000 7f4106acd640 10 monclient: _renew_subs 2024-09-11T01:43:57.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.679+0000 7f4106acd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:57.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.679+0000 7f4106acd640 1 -- 172.21.15.12:0/2542077382 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f410009f280 con 0x7f41000a1380 2024-09-11T01:43:57.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.679+0000 7f40feffd640 4 set_mon_vals no callback set 2024-09-11T01:43:57.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.679+0000 7f4106acd640 10 monclient: _renew_subs 2024-09-11T01:43:57.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.679+0000 7f4106acd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:57.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.679+0000 7f4106acd640 1 -- 172.21.15.12:0/2542077382 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4100127940 con 0x7f41000a1380 2024-09-11T01:43:57.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.680+0000 7f40fd7fa640 1 -- 172.21.15.12:0/2542077382 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f40ec013070 con 0x7f41000a1380 2024-09-11T01:43:57.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.680+0000 7f4106acd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:57.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.680+0000 7f4106acd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:57.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.680+0000 7f4106acd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:57.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.680+0000 7f4106acd640 1 -- 172.21.15.12:0/2542077382 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4100135ae0 con 0x7f41000a1380 2024-09-11T01:43:57.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.680+0000 7f40fd7fa640 1 --2- 172.21.15.12:0/2542077382 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f40e403e240 0x7f40e4040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:57.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.680+0000 7f40fd7fa640 1 -- 172.21.15.12:0/2542077382 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(505..505 src has 317..505) ==== 4799+0+0 (secure 0 0 0) 0x7f40ec052f00 con 0x7f41000a1380 2024-09-11T01:43:57.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.681+0000 7f4104869640 1 --2- 172.21.15.12:0/2542077382 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f40e403e240 0x7f40e4040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:57.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.681+0000 7f4104869640 10 monclient: get_auth_request con 0x7f40e403e240 auth_method 0 2024-09-11T01:43:57.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.683+0000 7f4104869640 1 --2- 172.21.15.12:0/2542077382 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f40e403e240 0x7f40e4040700 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f40f4031000 tx=0x7f40f402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:57.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.685+0000 7f40fd7fa640 1 -- 172.21.15.12:0/2542077382 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f40ec0518f0 con 0x7f41000a1380 2024-09-11T01:43:57.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.685+0000 7f40fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:57.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.685+0000 7f40fd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:43:57.732 INFO:tasks.rados.rados.0.smithi071.stdout:2274: done (0 left) 2024-09-11T01:43:57.732 INFO:tasks.rados.rados.0.smithi071.stdout:2275: snap_remove snap 230 2024-09-11T01:43:57.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.946+0000 7f4106acd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:43:57.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.946+0000 7f4106acd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:43:57.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.946+0000 7f4106acd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:57.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.946+0000 7f4106acd640 1 -- 172.21.15.12:0/2542077382 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f410011f000 con 0x7f41000a1380 2024-09-11T01:43:57.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.947+0000 7f40fd7fa640 1 -- 172.21.15.12:0/2542077382 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f40ec01cd70 con 0x7f41000a1380 2024-09-11T01:43:57.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.947+0000 7f40fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:43:57.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.947+0000 7f40fd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:43:57.949 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:43:57.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.949+0000 7f4106acd640 1 -- 172.21.15.12:0/2542077382 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f40e403e240 msgr2=0x7f40e4040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:57.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.949+0000 7f4106acd640 1 --2- 172.21.15.12:0/2542077382 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f40e403e240 0x7f40e4040700 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f40f4031000 tx=0x7f40f402e000 comp rx=0 tx=0).stop 2024-09-11T01:43:57.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.949+0000 7f4106acd640 10 monclient: shutdown 2024-09-11T01:43:57.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.949+0000 7f4106acd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:57.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.949+0000 7f4106acd640 1 -- 172.21.15.12:0/2542077382 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41000a1380 msgr2=0x7f410009df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:57.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.949+0000 7f4106acd640 1 --2- 172.21.15.12:0/2542077382 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41000a1380 0x7f410009df70 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f40ec00d7d0 tx=0x7f40ec00dc90 comp rx=0 tx=0).stop 2024-09-11T01:43:57.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.949+0000 7f4106acd640 1 -- 172.21.15.12:0/2542077382 shutdown_connections 2024-09-11T01:43:57.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.949+0000 7f4106acd640 1 --2- 172.21.15.12:0/2542077382 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4100136020 0x7f410009e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:57.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.949+0000 7f4106acd640 1 --2- 172.21.15.12:0/2542077382 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f40e403e240 0x7f40e4040700 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:57.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.949+0000 7f4106acd640 1 --2- 172.21.15.12:0/2542077382 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f41000a1380 0x7f410009df70 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:57.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.949+0000 7f4106acd640 1 --2- 172.21.15.12:0/2542077382 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41000a0c90 0x7f410009da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:57.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.949+0000 7f4106acd640 1 -- 172.21.15.12:0/2542077382 >> 172.21.15.12:0/2542077382 conn(0x7f41000a2d90 msgr2=0x7f4100070170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:57.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.949+0000 7f4106acd640 1 -- 172.21.15.12:0/2542077382 shutdown_connections 2024-09-11T01:43:57.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:57.949+0000 7f4106acd640 1 -- 172.21.15.12:0/2542077382 wait complete. 2024-09-11T01:43:57.961 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:43:57.961 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:43:57.962 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:43:58.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.071+0000 7fb7162e6640 10 monclient: get_monmap_and_config 2024-09-11T01:43:58.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.071+0000 7fb7162e6640 10 monclient: build_initial_monmap 2024-09-11T01:43:58.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.071+0000 7fb7162e6640 10 monclient: monmap: 2024-09-11T01:43:58.073 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:58.073 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:58.073 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:43:58.071864+0000 2024-09-11T01:43:58.073 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:43:58.071864+0000 2024-09-11T01:43:58.073 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:58.073 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:58.073 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:43:58.073 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:43:58.073 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:43:58.073 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:43:58.074 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:43:58.074 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:58.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.071+0000 7fb7162e6640 1 Processor -- start 2024-09-11T01:43:58.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb7162e6640 1 -- start start 2024-09-11T01:43:58.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb7162e6640 10 monclient: init 2024-09-11T01:43:58.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb7162e6640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:58.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb7162e6640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-11T01:43:58.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb7162e6640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb710137e50 0x7fb71009d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:58.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb7162e6640 10 monclient(hunting): picked mon.noname-a con 0x7fb710137e50 addr v2:172.21.15.12:3300/0 2024-09-11T01:43:58.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb7162e6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb710138230 0x7fb710138640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:58.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb7162e6640 10 monclient(hunting): picked mon.noname-c con 0x7fb710138230 addr v2:172.21.15.71:3300/0 2024-09-11T01:43:58.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb7162e6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb710134cb0 addr v1:172.21.15.12:6789/0 2024-09-11T01:43:58.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb7162e6640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7100a62d0 con 0x7fb710134cb0 2024-09-11T01:43:58.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb7162e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:58.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb7162e6640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fb71009bb40 con 0x7fb710137e50 2024-09-11T01:43:58.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb7162e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:58.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb7162e6640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fb71009c4a0 con 0x7fb710138230 2024-09-11T01:43:58.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb7162e6640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:58.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb7162e6640 10 monclient(hunting): authenticate will time out at 1644.514668s 2024-09-11T01:43:58.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb70ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb710137e50 0x7fb71009d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:58.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb714883640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb710134cb0 0x7fb710135160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53758/0 (socket says 172.21.15.12:53758) 2024-09-11T01:43:58.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb714883640 1 -- 172.21.15.12:0/1563980366 learned_addr learned my addr 172.21.15.12:0/1563980366 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:43:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb70ffff640 10 monclient(hunting): get_auth_request con 0x7fb710137e50 auth_method 0 2024-09-11T01:43:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb70ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb70ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb70ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb70ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb70ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb70ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb707fff640 1 --2- 172.21.15.12:0/1563980366 >> v2:172.21.15.71:3300/0 conn(0x7fb710138230 0x7fb710138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:58.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb707fff640 10 monclient(hunting): get_auth_request con 0x7fb710138230 auth_method 0 2024-09-11T01:43:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb707fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb707fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.072+0000 7fb707fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.073+0000 7fb70ffff640 10 monclient(hunting): handle_auth_done global_id 94764 payload 274 2024-09-11T01:43:58.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.073+0000 7fb70ffff640 1 -- 172.21.15.12:0/1563980366 >> v2:172.21.15.71:3300/0 conn(0x7fb710138230 msgr2=0x7fb710138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:58.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.073+0000 7fb70ffff640 1 --2- 172.21.15.12:0/1563980366 >> v2:172.21.15.71:3300/0 conn(0x7fb710138230 0x7fb710138640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:58.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.073+0000 7fb70ffff640 1 -- 172.21.15.12:0/1563980366 >> v1:172.21.15.12:6789/0 conn(0x7fb710134cb0 legacy=0x7fb710135160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:58.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.073+0000 7fb70ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:43:58.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.073+0000 7fb70ffff640 1 monclient: found mon.noname-a 2024-09-11T01:43:58.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.073+0000 7fb70ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:58.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.073+0000 7fb70ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:58.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.073+0000 7fb70ffff640 1 -- 172.21.15.12:0/1563980366 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb71009c780 con 0x7fb710137e50 2024-09-11T01:43:58.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.073+0000 7fb707fff640 1 --2- 172.21.15.12:0/1563980366 >> v2:172.21.15.71:3300/0 conn(0x7fb710138230 0x7fb710138640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:43:58.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.075+0000 7fb70ffff640 1 -- 172.21.15.12:0/1563980366 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb710137e50 msgr2=0x7fb71009d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:58.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.075+0000 7fb70ffff640 1 --2- 172.21.15.12:0/1563980366 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb710137e50 0x7fb71009d160 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb7100ab580 tx=0x7fb700029fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c535c5758969d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:58.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.076+0000 7fb70f7fe640 1 -- 172.21.15.12:0/1563980366 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb700037070 con 0x7fb710137e50 2024-09-11T01:43:58.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.076+0000 7fb70f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:58.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.076+0000 7fb70f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:58.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.076+0000 7fb70f7fe640 10 monclient: dump: 2024-09-11T01:43:58.086 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:58.086 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:58.086 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:58.086 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:58.086 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:58.087 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:58.087 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:58.087 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:58.087 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:58.087 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:58.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.076+0000 7fb70f7fe640 10 monclient: _finish_auth 0 2024-09-11T01:43:58.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.076+0000 7fb70f7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:43:58.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.076+0000 7fb70f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:58.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.076+0000 7fb7162e6640 5 monclient: authenticate success, global_id 94764 2024-09-11T01:43:58.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.076+0000 7fb70f7fe640 1 -- 172.21.15.12:0/1563980366 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb700032440 con 0x7fb710137e50 2024-09-11T01:43:58.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.076+0000 7fb70f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:58.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.076+0000 7fb7162e6640 10 monclient: get_monmap_and_config success 2024-09-11T01:43:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.076+0000 7fb7162e6640 4 set_mon_vals no callback set 2024-09-11T01:43:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.076+0000 7fb70f7fe640 1 -- 172.21.15.12:0/1563980366 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb700032810 con 0x7fb710137e50 2024-09-11T01:43:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.076+0000 7fb70f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.076+0000 7fb70f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.076+0000 7fb70f7fe640 10 monclient: dump: 2024-09-11T01:43:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:58.089 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.076+0000 7fb7162e6640 10 monclient: shutdown 2024-09-11T01:43:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.076+0000 7fb7162e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.076+0000 7fb7162e6640 1 -- 172.21.15.12:0/1563980366 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb710137e50 msgr2=0x7fb71009d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.076+0000 7fb7162e6640 1 --2- 172.21.15.12:0/1563980366 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb710137e50 0x7fb71009d160 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb7100ab580 tx=0x7fb700029fb0 comp rx=0 tx=0).stop 2024-09-11T01:43:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.076+0000 7fb7162e6640 1 -- 172.21.15.12:0/1563980366 shutdown_connections 2024-09-11T01:43:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.076+0000 7fb7162e6640 1 --2- 172.21.15.12:0/1563980366 >> v2:172.21.15.71:3300/0 conn(0x7fb710138230 0x7fb710138640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.076+0000 7fb7162e6640 1 --2- 172.21.15.12:0/1563980366 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb710137e50 0x7fb71009d160 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.076+0000 7fb7162e6640 1 -- 172.21.15.12:0/1563980366 >> 172.21.15.12:0/1563980366 conn(0x7fb7100a7020 msgr2=0x7fb7100a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.076+0000 7fb7162e6640 1 -- 172.21.15.12:0/1563980366 shutdown_connections 2024-09-11T01:43:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.076+0000 7fb7162e6640 1 -- 172.21.15.12:0/1563980366 wait complete. 2024-09-11T01:43:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.077+0000 7fb7162e6640 10 monclient: build_initial_monmap 2024-09-11T01:43:58.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.077+0000 7fb7162e6640 10 monclient: monmap: 2024-09-11T01:43:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:43:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:43:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:43:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:43:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:43:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:43:58.090 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.077+0000 7fb7162e6640 1 Processor -- start 2024-09-11T01:43:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.077+0000 7fb7162e6640 1 -- start start 2024-09-11T01:43:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.077+0000 7fb7162e6640 10 monclient: init 2024-09-11T01:43:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb7162e6640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb7162e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:43:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb7162e6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb710134cb0 0x7fb710133940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb7162e6640 10 monclient(hunting): picked mon.noname-a con 0x7fb710134cb0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb7162e6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb710137e50 0x7fb710133e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb7162e6640 10 monclient(hunting): picked mon.noname-c con 0x7fb710137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb7162e6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb710138230 0x7fb7101bee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb7162e6640 10 monclient(hunting): picked mon.noname-b con 0x7fb710138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb7162e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb7162e6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb71006f710 con 0x7fb710134cb0 2024-09-11T01:43:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb7162e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb7162e6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb710138b80 con 0x7fb710138230 2024-09-11T01:43:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb7162e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb7162e6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb71006f890 con 0x7fb710137e50 2024-09-11T01:43:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb7162e6640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb7162e6640 10 monclient(hunting): authenticate will time out at 1644.520121s 2024-09-11T01:43:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb70ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb710134cb0 0x7fb710133940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb707fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb710137e50 0x7fb710133e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb70ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb710134cb0 0x7fb710133940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35272/0 (socket says 172.21.15.12:35272) 2024-09-11T01:43:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb70ffff640 1 -- 172.21.15.12:0/51660218 learned_addr learned my addr 172.21.15.12:0/51660218 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb70ffff640 10 monclient(hunting): get_auth_request con 0x7fb710134cb0 auth_method 0 2024-09-11T01:43:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb70ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb70ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb70ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb707fff640 10 monclient(hunting): get_auth_request con 0x7fb710137e50 auth_method 0 2024-09-11T01:43:58.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb707fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb707fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb707fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb70ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb70ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb70ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb707fff640 1 -- 172.21.15.12:0/51660218 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb710137e50 msgr2=0x7fb710133e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:43:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb707fff640 1 -- 172.21.15.12:0/51660218 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb710137e50 msgr2=0x7fb710133e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:43:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb707fff640 1 --2- 172.21.15.12:0/51660218 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb710137e50 0x7fb710133e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb707fff640 1 --2- 172.21.15.12:0/51660218 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb710137e50 0x7fb710133e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb70ffff640 10 monclient(hunting): handle_auth_done global_id 94767 payload 824 2024-09-11T01:43:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb714883640 1 --2- 172.21.15.12:0/51660218 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb710138230 0x7fb7101bee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb70ffff640 1 -- 172.21.15.12:0/51660218 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb710137e50 msgr2=0x7fb710133e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:43:58.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb70ffff640 1 --2- 172.21.15.12:0/51660218 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb710137e50 0x7fb710133e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:58.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb70ffff640 1 -- 172.21.15.12:0/51660218 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb710138230 msgr2=0x7fb7101bee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:58.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb70ffff640 1 --2- 172.21.15.12:0/51660218 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb710138230 0x7fb7101bee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:58.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb70ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:43:58.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb70ffff640 1 monclient: found mon.noname-a 2024-09-11T01:43:58.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb70ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:58.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb70ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:58.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb70ffff640 1 -- 172.21.15.12:0/51660218 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb710121060 con 0x7fb710134cb0 2024-09-11T01:43:58.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb714883640 10 monclient: get_auth_request con 0x7fb710138230 auth_method 0 2024-09-11T01:43:58.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.078+0000 7fb714883640 1 --2- 172.21.15.12:0/51660218 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb710138230 0x7fb7101bee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:43:58.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.080+0000 7fb70ffff640 1 -- 172.21.15.12:0/51660218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb710134cb0 msgr2=0x7fb710133940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:58.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.080+0000 7fb70ffff640 1 --2- 172.21.15.12:0/51660218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb710134cb0 0x7fb710133940 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb700009860 tx=0x7fb7000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:58.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.080+0000 7fb70d7fa640 1 -- 172.21.15.12:0/51660218 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb700041020 con 0x7fb710134cb0 2024-09-11T01:43:58.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.081+0000 7fb70d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:58.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.081+0000 7fb70d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:58.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.081+0000 7fb70d7fa640 10 monclient: dump: 2024-09-11T01:43:58.095 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:58.095 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:58.095 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:58.095 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:58.095 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:58.095 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:58.095 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:58.095 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:58.095 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:58.095 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:58.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.081+0000 7fb70d7fa640 10 monclient: _finish_auth 0 2024-09-11T01:43:58.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.081+0000 7fb70d7fa640 10 monclient: _check_auth_tickets 2024-09-11T01:43:58.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.081+0000 7fb70d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:58.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.081+0000 7fb70d7fa640 1 -- 172.21.15.12:0/51660218 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb70003d910 con 0x7fb710134cb0 2024-09-11T01:43:58.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.081+0000 7fb7162e6640 5 monclient: authenticate success, global_id 94767 2024-09-11T01:43:58.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.081+0000 7fb7162e6640 10 monclient: _renew_subs 2024-09-11T01:43:58.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.081+0000 7fb7162e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:58.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.081+0000 7fb7162e6640 1 -- 172.21.15.12:0/51660218 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb710137280 con 0x7fb710134cb0 2024-09-11T01:43:58.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.081+0000 7fb70d7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:58.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.081+0000 7fb70d7fa640 1 -- 172.21.15.12:0/51660218 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb700037040 con 0x7fb710134cb0 2024-09-11T01:43:58.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.081+0000 7fb70d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:58.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.081+0000 7fb70d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:58.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.081+0000 7fb70d7fa640 10 monclient: dump: 2024-09-11T01:43:58.096 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:58.096 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:58.096 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:58.096 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:58.097 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.081+0000 7fb70effd640 4 set_mon_vals no callback set 2024-09-11T01:43:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.081+0000 7fb7162e6640 10 monclient: _renew_subs 2024-09-11T01:43:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.081+0000 7fb7162e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.081+0000 7fb7162e6640 1 -- 172.21.15.12:0/51660218 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7101343c0 con 0x7fb710134cb0 2024-09-11T01:43:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.081+0000 7fb7162e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.081+0000 7fb7162e6640 1 --2- 172.21.15.12:0/51660218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7101bf370 0x7fb7101bf750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.081+0000 7fb7162e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.081+0000 7fb7162e6640 1 -- 172.21.15.12:0/51660218 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb71006a060 con 0x7fb7101bf370 2024-09-11T01:43:58.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.081+0000 7fb707fff640 1 --2- 172.21.15.12:0/51660218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7101bf370 0x7fb7101bf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.082+0000 7fb707fff640 10 monclient: get_auth_request con 0x7fb7101bf370 auth_method 0 2024-09-11T01:43:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.082+0000 7fb707fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.082+0000 7fb707fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.082+0000 7fb707fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.082+0000 7fb707fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.082+0000 7fb707fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.082+0000 7fb707fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.082+0000 7fb70d7fa640 1 -- 172.21.15.12:0/51660218 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fb70003a030 con 0x7fb710134cb0 2024-09-11T01:43:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.082+0000 7fb707fff640 10 monclient(hunting): handle_auth_done global_id 94770 payload 824 2024-09-11T01:43:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.082+0000 7fb707fff640 1 -- 172.21.15.12:0/51660218 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb7101214d0 con 0x7fb7101bf370 2024-09-11T01:43:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.082+0000 7fb70d7fa640 1 --2- 172.21.15.12:0/51660218 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb6f003e210 0x7fb6f00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.082+0000 7fb70d7fa640 1 -- 172.21.15.12:0/51660218 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(506..506 src has 317..506) ==== 4815+0+0 (secure 0 0 0) 0x7fb7000734a0 con 0x7fb710134cb0 2024-09-11T01:43:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.082+0000 7fb714883640 1 --2- 172.21.15.12:0/51660218 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb6f003e210 0x7fb6f00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:58.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.082+0000 7fb714883640 10 monclient: get_auth_request con 0x7fb6f003e210 auth_method 0 2024-09-11T01:43:58.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.084+0000 7fb707fff640 1 -- 172.21.15.12:0/51660218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7101bf370 msgr2=0x7fb7101bf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:58.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.084+0000 7fb707fff640 1 --2- 172.21.15.12:0/51660218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7101bf370 0x7fb7101bf750 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8010790 tx=0x7fb6f8010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:58.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.084+0000 7fb70d7fa640 1 -- 172.21.15.12:0/51660218 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f801a020 con 0x7fb7101bf370 2024-09-11T01:43:58.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.084+0000 7fb70d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:58.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.084+0000 7fb70d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:58.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.084+0000 7fb70d7fa640 10 monclient: dump: 2024-09-11T01:43:58.101 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:58.101 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:58.101 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:58.101 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:58.101 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:58.101 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:58.102 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.085+0000 7fb714883640 1 --2- 172.21.15.12:0/51660218 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb6f003e210 0x7fb6f00406d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb7100a1850 tx=0x7fb708009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.085+0000 7fb70d7fa640 1 -- 172.21.15.12:0/51660218 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb6f800ede0 con 0x7fb7101bf370 2024-09-11T01:43:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.085+0000 7fb70d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.085+0000 7fb70d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:43:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.085+0000 7fb70d7fa640 1 -- 172.21.15.12:0/51660218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7101bf370 msgr2=0x7fb7101bf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.085+0000 7fb70d7fa640 1 --2- 172.21.15.12:0/51660218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7101bf370 0x7fb7101bf750 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8010790 tx=0x7fb6f8010c50 comp rx=0 tx=0).stop 2024-09-11T01:43:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.085+0000 7fb70d7fa640 1 -- 172.21.15.12:0/51660218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7101bf370 msgr2=0x7fb7101bf750 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:43:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.085+0000 7fb70d7fa640 1 --2- 172.21.15.12:0/51660218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7101bf370 0x7fb7101bf750 secure :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8010790 tx=0x7fb6f8010c50 comp rx=0 tx=0).stop 2024-09-11T01:43:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.091+0000 7fb7162e6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:43:58.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.091+0000 7fb7162e6640 1 --2- 172.21.15.12:0/51660218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7101bfc90 0x7fb7100a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:58.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.091+0000 7fb7162e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:58.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.091+0000 7fb7162e6640 1 -- 172.21.15.12:0/51660218 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb710135900 con 0x7fb7101bfc90 2024-09-11T01:43:58.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.091+0000 7fb707fff640 1 --2- 172.21.15.12:0/51660218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7101bfc90 0x7fb7100a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:58.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.091+0000 7fb707fff640 10 monclient: get_auth_request con 0x7fb7101bfc90 auth_method 0 2024-09-11T01:43:58.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.091+0000 7fb707fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:58.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.091+0000 7fb707fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:58.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.091+0000 7fb707fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:58.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.091+0000 7fb707fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:58.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.091+0000 7fb707fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:58.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.091+0000 7fb707fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:58.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.091+0000 7fb707fff640 10 monclient(hunting): handle_auth_done global_id 94773 payload 824 2024-09-11T01:43:58.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.091+0000 7fb707fff640 1 -- 172.21.15.12:0/51660218 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb710069950 con 0x7fb7101bfc90 2024-09-11T01:43:58.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.094+0000 7fb707fff640 1 -- 172.21.15.12:0/51660218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7101bfc90 msgr2=0x7fb7100a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:58.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.094+0000 7fb707fff640 1 --2- 172.21.15.12:0/51660218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7101bfc90 0x7fb7100a1920 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb6f801b000 tx=0x7fb6f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:58.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.094+0000 7fb70d7fa640 1 -- 172.21.15.12:0/51660218 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f801a040 con 0x7fb7101bfc90 2024-09-11T01:43:58.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.094+0000 7fb70d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:58.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.094+0000 7fb70d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:58.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.094+0000 7fb70d7fa640 10 monclient: dump: 2024-09-11T01:43:58.104 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:58.104 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:58.104 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:58.104 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:58.104 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:58.104 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:58.104 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:58.104 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:58.104 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:58.104 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:58.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.094+0000 7fb70d7fa640 1 -- 172.21.15.12:0/51660218 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7fb6f8015670 con 0x7fb7101bfc90 2024-09-11T01:43:58.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.094+0000 7fb70d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:43:58.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.094+0000 7fb70d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:43:58.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.094+0000 7fb70d7fa640 1 -- 172.21.15.12:0/51660218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7101bfc90 msgr2=0x7fb7100a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:58.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.094+0000 7fb70d7fa640 1 --2- 172.21.15.12:0/51660218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7101bfc90 0x7fb7100a1920 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb6f801b000 tx=0x7fb6f80046c0 comp rx=0 tx=0).stop 2024-09-11T01:43:58.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.094+0000 7fb70d7fa640 1 -- 172.21.15.12:0/51660218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7101bfc90 msgr2=0x7fb7100a1920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:43:58.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.094+0000 7fb70d7fa640 1 --2- 172.21.15.12:0/51660218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7101bfc90 0x7fb7100a1920 secure :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb6f801b000 tx=0x7fb6f80046c0 comp rx=0 tx=0).stop 2024-09-11T01:43:58.105 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:43:58.105 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:43:58.105 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:43:58.106 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:43:58.106 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 138, 2024-09-11T01:43:58.106 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 384190, 2024-09-11T01:43:58.106 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:43:58.106 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:43:58.106 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:43:58.106 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:43:58.106 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:58.106 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:43:58.106 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:43:58.106 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:43:58.106 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:43:58.107 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:58.107 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:58.107 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:58.107 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:58.107 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:58.107 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:58.107 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:58.107 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:58.107 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:58.107 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:58.107 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:58.107 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:58.107 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:43:58.107 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:43:58.108 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:58.108 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:58.108 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:58.108 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:58.108 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:58.108 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:58.108 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:58.108 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:58.108 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:58.108 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:58.108 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:58.108 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:58.108 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:58.109 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:43:58.109 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:43:58.109 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:43:58.109 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:43:58.109 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:43:58.109 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:43:58.109 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:43:58.109 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:43:58.109 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:43:58.109 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:43:58.109 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:43:58.109 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:43:58.109 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:43:58.109 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:43:58.110 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:43:58.110 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:43:58.110 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:43:58.110 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:58.110 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:58.110 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:58.110 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:58.110 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:58.110 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:58.110 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:58.110 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:58.110 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:58.110 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:58.111 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:58.111 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:58.111 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:43:58.111 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:58.111 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:43:58.111 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:58.111 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:43:58.111 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:43:58.111 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:58.111 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:58.111 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:58.111 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:58.111 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:43:58.112 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:58.112 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:58.112 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:58.112 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:58.112 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:43:58.112 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:58.112 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:58.112 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:58.112 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:58.112 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:43:58.112 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:43:58.112 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:58.112 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:58.113 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:58.113 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:58.113 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:58.113 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:43:58.113 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:43:58.113 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:58.113 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:58.113 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:58.113 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:58.113 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:43:58.113 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:58.113 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:58.113 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:58.113 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:58.114 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:43:58.114 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:58.114 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:58.114 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:58.114 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:58.114 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:43:58.114 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:43:58.114 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:58.114 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:58.115 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:58.115 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:58.115 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:58.115 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:43:58.115 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:43:58.115 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:58.115 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:58.115 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:58.115 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:58.115 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:43:58.115 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:58.115 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:58.115 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:58.116 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:58.116 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:43:58.116 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:58.116 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:58.116 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:58.116 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:58.116 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:43:58.116 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:43:58.116 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:58.116 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:58.116 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:58.116 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:58.116 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:58.116 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:58.117 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:43:58.117 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:43:58.117 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:58.117 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:58.117 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:58.117 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:43:58.117 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:58.117 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:58.117 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:43:58.117 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:58.117 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:58.117 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:58.117 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:43:58.118 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:58.118 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:58.118 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:43:58.118 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:58.118 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:58.118 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:58.118 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:43:58.118 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:58.118 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:58.118 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:58.118 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:43:58.118 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:43:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.094+0000 7fb7162e6640 1 -- 172.21.15.12:0/51660218 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb6f003e210 msgr2=0x7fb6f00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.094+0000 7fb7162e6640 1 --2- 172.21.15.12:0/51660218 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb6f003e210 0x7fb6f00406d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb7100a1850 tx=0x7fb708009040 comp rx=0 tx=0).stop 2024-09-11T01:43:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.094+0000 7fb7162e6640 10 monclient: shutdown 2024-09-11T01:43:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.094+0000 7fb7162e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.094+0000 7fb7162e6640 1 -- 172.21.15.12:0/51660218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb710134cb0 msgr2=0x7fb710133940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.094+0000 7fb7162e6640 1 --2- 172.21.15.12:0/51660218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb710134cb0 0x7fb710133940 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb700009860 tx=0x7fb7000046c0 comp rx=0 tx=0).stop 2024-09-11T01:43:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.094+0000 7fb714883640 1 -- 172.21.15.12:0/51660218 reap_dead start 2024-09-11T01:43:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.094+0000 7fb714883640 1 -- 172.21.15.12:0/51660218 reap_dead start 2024-09-11T01:43:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.094+0000 7fb7162e6640 1 -- 172.21.15.12:0/51660218 shutdown_connections 2024-09-11T01:43:58.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.094+0000 7fb7162e6640 1 -- 172.21.15.12:0/51660218 >> 172.21.15.12:0/51660218 conn(0x7fb7100a7020 msgr2=0x7fb7100a0430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:58.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.094+0000 7fb7162e6640 1 -- 172.21.15.12:0/51660218 shutdown_connections 2024-09-11T01:43:58.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.094+0000 7fb7162e6640 1 -- 172.21.15.12:0/51660218 wait complete. 2024-09-11T01:43:58.121 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:43:58.121 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:43:58.121 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:43:58.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.222+0000 7fc03b252640 10 monclient: get_monmap_and_config 2024-09-11T01:43:58.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.222+0000 7fc03b252640 10 monclient: build_initial_monmap 2024-09-11T01:43:58.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.222+0000 7fc03b252640 10 monclient: monmap: 2024-09-11T01:43:58.223 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:58.224 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:58.224 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:43:58.222824+0000 2024-09-11T01:43:58.224 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:43:58.222824+0000 2024-09-11T01:43:58.224 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:58.224 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:58.224 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:43:58.224 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:43:58.224 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:43:58.224 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:43:58.224 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:43:58.224 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:58.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.222+0000 7fc03b252640 1 Processor -- start 2024-09-11T01:43:58.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.222+0000 7fc03b252640 1 -- start start 2024-09-11T01:43:58.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.222+0000 7fc03b252640 10 monclient: init 2024-09-11T01:43:58.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc03b252640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:58.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc03b252640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-11T01:43:58.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc03b252640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034137e50 0x7fc03409d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:58.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc03b252640 10 monclient(hunting): picked mon.noname-b con 0x7fc034137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:58.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc03b252640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc034138230 0x7fc034138630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:58.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc03b252640 10 monclient(hunting): picked mon.noname-a con 0x7fc034138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:43:58.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc03b252640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc034134cb0 addr v1:172.21.15.12:6789/0 2024-09-11T01:43:58.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc03b252640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0340a62d0 con 0x7fc034134cb0 2024-09-11T01:43:58.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc03b252640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:58.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc03b252640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc0340a65b0 con 0x7fc034138230 2024-09-11T01:43:58.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc03b252640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:58.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc03b252640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc03409bb90 con 0x7fc034137e50 2024-09-11T01:43:58.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc03b252640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:58.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc03b252640 10 monclient(hunting): authenticate will time out at 1644.665490s 2024-09-11T01:43:58.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc038fee640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034137e50 0x7fc03409d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:58.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc033fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc034138230 0x7fc034138630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:58.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc038fee640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034137e50 0x7fc03409d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52608/0 (socket says 172.21.15.12:52608) 2024-09-11T01:43:58.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc038fee640 1 -- 172.21.15.12:0/817424239 learned_addr learned my addr 172.21.15.12:0/817424239 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:58.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc038fee640 10 monclient(hunting): get_auth_request con 0x7fc034137e50 auth_method 0 2024-09-11T01:43:58.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc038fee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:58.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc038fee640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:58.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc038fee640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:58.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc033fff640 10 monclient(hunting): get_auth_request con 0x7fc034138230 auth_method 0 2024-09-11T01:43:58.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc033fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:58.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc033fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:58.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc033fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:58.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc038fee640 1 -- 172.21.15.12:0/817424239 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034137e50 msgr2=0x7fc03409d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:43:58.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc038fee640 1 -- 172.21.15.12:0/817424239 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034137e50 msgr2=0x7fc03409d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:43:58.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc033fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:58.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc033fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:58.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc038fee640 1 --2- 172.21.15.12:0/817424239 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034137e50 0x7fc03409d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:58.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc033fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:58.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc038fee640 1 --2- 172.21.15.12:0/817424239 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034137e50 0x7fc03409d160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:58.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc033fff640 10 monclient(hunting): handle_auth_done global_id 94776 payload 274 2024-09-11T01:43:58.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc033fff640 1 -- 172.21.15.12:0/817424239 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034137e50 msgr2=0x7fc03409d160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:43:58.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc033fff640 1 --2- 172.21.15.12:0/817424239 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034137e50 0x7fc03409d160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:58.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc033fff640 1 -- 172.21.15.12:0/817424239 >> v1:172.21.15.12:6789/0 conn(0x7fc034134cb0 legacy=0x7fc034135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:58.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc033fff640 10 monclient: _finish_hunting 0 2024-09-11T01:43:58.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc033fff640 1 monclient: found mon.noname-a 2024-09-11T01:43:58.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc033fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:58.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc033fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:58.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.223+0000 7fc033fff640 1 -- 172.21.15.12:0/817424239 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc03409c4f0 con 0x7fc034138230 2024-09-11T01:43:58.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.226+0000 7fc033fff640 1 -- 172.21.15.12:0/817424239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc034138230 msgr2=0x7fc034138630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:58.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.226+0000 7fc033fff640 1 --2- 172.21.15.12:0/817424239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc034138230 0x7fc034138630 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc0280097a0 tx=0x7fc028029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c058da20915104af server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:58.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.226+0000 7fc0337fe640 1 -- 172.21.15.12:0/817424239 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc028037070 con 0x7fc034138230 2024-09-11T01:43:58.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.226+0000 7fc0337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:58.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.226+0000 7fc0337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:58.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.226+0000 7fc0337fe640 10 monclient: dump: 2024-09-11T01:43:58.237 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:58.237 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:58.237 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:58.237 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:58.237 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:58.237 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:58.237 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:58.237 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:58.237 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:58.238 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:58.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.226+0000 7fc0337fe640 10 monclient: _finish_auth 0 2024-09-11T01:43:58.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.226+0000 7fc0337fe640 10 monclient: _check_auth_tickets 2024-09-11T01:43:58.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.226+0000 7fc0337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:58.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.226+0000 7fc0337fe640 1 -- 172.21.15.12:0/817424239 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc028032440 con 0x7fc034138230 2024-09-11T01:43:58.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.226+0000 7fc03b252640 5 monclient: authenticate success, global_id 94776 2024-09-11T01:43:58.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.226+0000 7fc03b252640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:43:58.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.226+0000 7fc0337fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:58.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.226+0000 7fc03b252640 10 monclient: get_monmap_and_config success 2024-09-11T01:43:58.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.226+0000 7fc0337fe640 1 -- 172.21.15.12:0/817424239 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc028032810 con 0x7fc034138230 2024-09-11T01:43:58.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.226+0000 7fc0337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:58.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.226+0000 7fc03b252640 4 set_mon_vals no callback set 2024-09-11T01:43:58.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.226+0000 7fc0337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:58.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.226+0000 7fc0337fe640 10 monclient: dump: 2024-09-11T01:43:58.239 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:58.239 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:58.239 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:58.239 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:58.239 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:58.239 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:58.239 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:58.239 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:58.239 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:58.239 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:58.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.227+0000 7fc03b252640 10 monclient: shutdown 2024-09-11T01:43:58.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.227+0000 7fc03b252640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:58.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.227+0000 7fc03b252640 1 -- 172.21.15.12:0/817424239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc034138230 msgr2=0x7fc034138630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:58.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.227+0000 7fc03b252640 1 --2- 172.21.15.12:0/817424239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc034138230 0x7fc034138630 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc0280097a0 tx=0x7fc028029fd0 comp rx=0 tx=0).stop 2024-09-11T01:43:58.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.227+0000 7fc03b252640 1 -- 172.21.15.12:0/817424239 shutdown_connections 2024-09-11T01:43:58.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.227+0000 7fc03b252640 1 --2- 172.21.15.12:0/817424239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc034138230 0x7fc034138630 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:58.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.227+0000 7fc03b252640 1 --2- 172.21.15.12:0/817424239 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034137e50 0x7fc03409d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:58.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.227+0000 7fc03b252640 1 -- 172.21.15.12:0/817424239 >> 172.21.15.12:0/817424239 conn(0x7fc0340a7020 msgr2=0x7fc0340a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:58.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.227+0000 7fc03b252640 1 -- 172.21.15.12:0/817424239 shutdown_connections 2024-09-11T01:43:58.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.227+0000 7fc03b252640 1 -- 172.21.15.12:0/817424239 wait complete. 2024-09-11T01:43:58.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.228+0000 7fc03b252640 10 monclient: build_initial_monmap 2024-09-11T01:43:58.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.228+0000 7fc03b252640 10 monclient: monmap: 2024-09-11T01:43:58.240 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:58.240 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:43:58.240 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:43:58.240 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:43:58.240 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:58.241 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:58.241 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:43:58.241 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:43:58.241 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:43:58.241 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:58.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.228+0000 7fc03b252640 1 Processor -- start 2024-09-11T01:43:58.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.228+0000 7fc03b252640 1 -- start start 2024-09-11T01:43:58.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.228+0000 7fc03b252640 10 monclient: init 2024-09-11T01:43:58.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc03b252640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:58.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc03b252640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:43:58.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc03b252640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc034134cb0 0x7fc0341cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:58.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc03b252640 10 monclient(hunting): picked mon.noname-a con 0x7fc034134cb0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:58.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc03b252640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc034137e50 0x7fc0341cd910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:58.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc03b252640 10 monclient(hunting): picked mon.noname-b con 0x7fc034137e50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:58.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc03b252640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034138230 0x7fc034121780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:58.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc03b252640 10 monclient(hunting): picked mon.noname-c con 0x7fc034138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:58.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc03b252640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:58.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc03b252640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc03406f710 con 0x7fc034134cb0 2024-09-11T01:43:58.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc03b252640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:58.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc03b252640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc034138b70 con 0x7fc034137e50 2024-09-11T01:43:58.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc03b252640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:58.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc03b252640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc03406f890 con 0x7fc034138230 2024-09-11T01:43:58.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc03b252640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:58.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc03b252640 10 monclient(hunting): authenticate will time out at 1644.671160s 2024-09-11T01:43:58.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc038fee640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc034134cb0 0x7fc0341cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:58.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc0397ef640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034138230 0x7fc034121780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:58.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc038fee640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc034134cb0 0x7fc0341cd3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35314/0 (socket says 172.21.15.12:35314) 2024-09-11T01:43:58.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc0397ef640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034138230 0x7fc034121780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52616/0 (socket says 172.21.15.12:52616) 2024-09-11T01:43:58.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc038fee640 1 -- 172.21.15.12:0/4178987960 learned_addr learned my addr 172.21.15.12:0/4178987960 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:58.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc038fee640 10 monclient(hunting): get_auth_request con 0x7fc034134cb0 auth_method 0 2024-09-11T01:43:58.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc038fee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:58.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc038fee640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:58.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc038fee640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:58.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc0397ef640 10 monclient(hunting): get_auth_request con 0x7fc034138230 auth_method 0 2024-09-11T01:43:58.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc0397ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:58.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc0397ef640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:58.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc0397ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:58.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc038fee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:58.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc038fee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:58.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc038fee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:58.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc033fff640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc034137e50 0x7fc0341cd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:58.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc0397ef640 1 -- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034138230 msgr2=0x7fc034121780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:43:58.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc0397ef640 1 -- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034138230 msgr2=0x7fc034121780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:43:58.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc0397ef640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034138230 0x7fc034121780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:58.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc033fff640 10 monclient(hunting): get_auth_request con 0x7fc034137e50 auth_method 0 2024-09-11T01:43:58.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc033fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:58.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc033fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:58.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc033fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:58.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc0397ef640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034138230 0x7fc034121780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:58.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc033fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:58.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc033fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:58.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc033fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:58.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc038fee640 10 monclient(hunting): handle_auth_done global_id 94779 payload 824 2024-09-11T01:43:58.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc038fee640 1 -- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034138230 msgr2=0x7fc034121780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:43:58.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc038fee640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034138230 0x7fc034121780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:58.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc038fee640 1 -- 172.21.15.12:0/4178987960 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc034137e50 msgr2=0x7fc0341cd910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:58.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc038fee640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc034137e50 0x7fc0341cd910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:58.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc038fee640 10 monclient: _finish_hunting 0 2024-09-11T01:43:58.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc038fee640 1 monclient: found mon.noname-a 2024-09-11T01:43:58.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc038fee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:58.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc038fee640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:58.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.229+0000 7fc038fee640 1 -- 172.21.15.12:0/4178987960 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0341377f0 con 0x7fc034134cb0 2024-09-11T01:43:58.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.231+0000 7fc038fee640 1 -- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc034134cb0 msgr2=0x7fc0341cd3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:58.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.231+0000 7fc038fee640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc034134cb0 0x7fc0341cd3d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc02400e810 tx=0x7fc02400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:58.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.232+0000 7fc0317fa640 1 -- 172.21.15.12:0/4178987960 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc024017070 con 0x7fc034134cb0 2024-09-11T01:43:58.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.232+0000 7fc0317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:58.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.232+0000 7fc0317fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:58.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.232+0000 7fc0317fa640 10 monclient: dump: 2024-09-11T01:43:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:58.246 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.232+0000 7fc0317fa640 10 monclient: _finish_auth 0 2024-09-11T01:43:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.232+0000 7fc0317fa640 10 monclient: _check_auth_tickets 2024-09-11T01:43:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.232+0000 7fc0317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.232+0000 7fc03b252640 5 monclient: authenticate success, global_id 94779 2024-09-11T01:43:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.232+0000 7fc0317fa640 1 -- 172.21.15.12:0/4178987960 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc024004020 con 0x7fc034134cb0 2024-09-11T01:43:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.232+0000 7fc0317fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.232+0000 7fc0317fa640 1 -- 172.21.15.12:0/4178987960 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc02400f040 con 0x7fc034134cb0 2024-09-11T01:43:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.232+0000 7fc0317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.232+0000 7fc0317fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.232+0000 7fc0317fa640 10 monclient: dump: 2024-09-11T01:43:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:58.248 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:58.248 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:58.248 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:58.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.232+0000 7fc03b252640 10 monclient: _renew_subs 2024-09-11T01:43:58.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.232+0000 7fc03b252640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:58.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.232+0000 7fc03b252640 1 -- 172.21.15.12:0/4178987960 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc034123240 con 0x7fc034134cb0 2024-09-11T01:43:58.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.232+0000 7fc032ffd640 4 set_mon_vals no callback set 2024-09-11T01:43:58.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.232+0000 7fc03b252640 10 monclient: _renew_subs 2024-09-11T01:43:58.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.232+0000 7fc03b252640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:58.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.232+0000 7fc03b252640 1 -- 172.21.15.12:0/4178987960 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc034124d00 con 0x7fc034134cb0 2024-09-11T01:43:58.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.233+0000 7fc0317fa640 1 -- 172.21.15.12:0/4178987960 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fc024012050 con 0x7fc034134cb0 2024-09-11T01:43:58.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.233+0000 7fc03b252640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:58.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.233+0000 7fc03b252640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034121cc0 0x7fc0341220a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:58.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.233+0000 7fc03b252640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.233+0000 7fc03b252640 1 -- 172.21.15.12:0/4178987960 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc034135950 con 0x7fc034121cc0 2024-09-11T01:43:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.233+0000 7fc033fff640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034121cc0 0x7fc0341220a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.233+0000 7fc033fff640 10 monclient: get_auth_request con 0x7fc034121cc0 auth_method 0 2024-09-11T01:43:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.233+0000 7fc033fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.233+0000 7fc033fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.233+0000 7fc033fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.233+0000 7fc033fff640 1 -- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034121cc0 msgr2=0x7fc0341220a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:43:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.233+0000 7fc0317fa640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc02003e210 0x7fc0200406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.233+0000 7fc033fff640 1 -- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034121cc0 msgr2=0x7fc0341220a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:43:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.233+0000 7fc033fff640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034121cc0 0x7fc0341220a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.233+0000 7fc033fff640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034121cc0 0x7fc0341220a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:43:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.233+0000 7fc0317fa640 1 -- 172.21.15.12:0/4178987960 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(506..506 src has 317..506) ==== 4815+0+0 (secure 0 0 0) 0x7fc024052760 con 0x7fc034134cb0 2024-09-11T01:43:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.233+0000 7fc0397ef640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc02003e210 0x7fc0200406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.233+0000 7fc0397ef640 10 monclient: get_auth_request con 0x7fc02003e210 auth_method 0 2024-09-11T01:43:58.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.236+0000 7fc0397ef640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc02003e210 0x7fc0200406d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc02c007f50 tx=0x7fc02c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:58.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.434+0000 7fc033fff640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034121cc0 0x7fc0341220a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:58.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.434+0000 7fc033fff640 10 monclient: get_auth_request con 0x7fc034121cc0 auth_method 0 2024-09-11T01:43:58.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.434+0000 7fc033fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:58.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.434+0000 7fc033fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:58.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.434+0000 7fc033fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:58.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.434+0000 7fc033fff640 1 -- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034121cc0 msgr2=0x7fc0341220a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:43:58.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.434+0000 7fc033fff640 1 -- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034121cc0 msgr2=0x7fc0341220a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:43:58.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.434+0000 7fc033fff640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034121cc0 0x7fc0341220a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:58.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.434+0000 7fc033fff640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034121cc0 0x7fc0341220a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:43:58.736 INFO:tasks.rados.rados.0.smithi071.stdout:2275: done (0 left) 2024-09-11T01:43:58.736 INFO:tasks.rados.rados.0.smithi071.stdout:2276: read oid 6 snap -1 2024-09-11T01:43:58.736 INFO:tasks.rados.rados.0.smithi071.stdout:2276: expect deleted 2024-09-11T01:43:58.737 INFO:tasks.rados.rados.0.smithi071.stdout:2277: write (excl) oid 15 current snap is 251 2024-09-11T01:43:58.737 INFO:tasks.rados.rados.0.smithi071.stdout:2277: seq_num 825 ranges {665775=646549,2110468=499284,3155969=442940} 2024-09-11T01:43:58.744 INFO:tasks.rados.rados.0.smithi071.stdout:2277: writing smithi07134777-15 from 665775 to 1312324 tid 1 2024-09-11T01:43:58.751 INFO:tasks.rados.rados.0.smithi071.stdout:2277: writing smithi07134777-15 from 2110468 to 2609752 tid 2 2024-09-11T01:43:58.757 INFO:tasks.rados.rados.0.smithi071.stdout:2277: writing smithi07134777-15 from 3155969 to 3598909 tid 3 2024-09-11T01:43:58.757 INFO:tasks.rados.rados.0.smithi071.stdout:2278: snap_remove snap 249 2024-09-11T01:43:58.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.835+0000 7fc033fff640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034121cc0 0x7fc0341220a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:58.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.835+0000 7fc033fff640 10 monclient: get_auth_request con 0x7fc034121cc0 auth_method 0 2024-09-11T01:43:58.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.835+0000 7fc033fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:58.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.835+0000 7fc033fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:58.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.835+0000 7fc033fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:58.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.835+0000 7fc033fff640 1 -- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034121cc0 msgr2=0x7fc0341220a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:43:58.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.835+0000 7fc033fff640 1 -- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034121cc0 msgr2=0x7fc0341220a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:43:58.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.835+0000 7fc033fff640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034121cc0 0x7fc0341220a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:43:58.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:58.835+0000 7fc033fff640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034121cc0 0x7fc0341220a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:43:59.404 INFO:tasks.rados.rados.0.smithi071.stdout:2278: done (2 left) 2024-09-11T01:43:59.404 INFO:tasks.rados.rados.0.smithi071.stdout:2279: snap_remove snap 213 2024-09-11T01:43:59.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.636+0000 7fc033fff640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034121cc0 0x7fc0341220a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:59.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.636+0000 7fc033fff640 10 monclient: get_auth_request con 0x7fc034121cc0 auth_method 0 2024-09-11T01:43:59.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.636+0000 7fc033fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:59.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.636+0000 7fc033fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:59.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.636+0000 7fc033fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:43:59.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.636+0000 7fc033fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:59.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.636+0000 7fc033fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:59.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.636+0000 7fc033fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:59.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.636+0000 7fc033fff640 10 monclient(hunting): handle_auth_done global_id 164099 payload 824 2024-09-11T01:43:59.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.636+0000 7fc033fff640 1 -- 172.21.15.12:0/4178987960 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc0341250c0 con 0x7fc034121cc0 2024-09-11T01:43:59.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.638+0000 7fc033fff640 1 -- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034121cc0 msgr2=0x7fc0341220a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:59.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.639+0000 7fc033fff640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034121cc0 0x7fc0341220a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc028034000 tx=0x7fc028004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:59.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.639+0000 7fc0317fa640 1 -- 172.21.15.12:0/4178987960 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc02804b070 con 0x7fc034121cc0 2024-09-11T01:43:59.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.639+0000 7fc0317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:59.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.639+0000 7fc0317fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:43:59.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.639+0000 7fc0317fa640 10 monclient: dump: 2024-09-11T01:43:59.641 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:59.641 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:59.641 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:59.641 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:59.641 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:59.641 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:59.641 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:59.641 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:59.641 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:59.641 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:59.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.639+0000 7fc0317fa640 1 -- 172.21.15.12:0/4178987960 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc02803c610 con 0x7fc034121cc0 2024-09-11T01:43:59.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.639+0000 7fc0317fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:59.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.639+0000 7fc0317fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:43:59.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.639+0000 7fc0317fa640 1 -- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034121cc0 msgr2=0x7fc0341220a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:59.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.639+0000 7fc0317fa640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034121cc0 0x7fc0341220a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc028034000 tx=0x7fc028004730 comp rx=0 tx=0).stop 2024-09-11T01:43:59.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.639+0000 7fc0317fa640 1 -- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034121cc0 msgr2=0x7fc0341220a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:43:59.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.639+0000 7fc0317fa640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc034121cc0 0x7fc0341220a0 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc028034000 tx=0x7fc028004730 comp rx=0 tx=0).stop 2024-09-11T01:43:59.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.646+0000 7fc03b252640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:43:59.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.646+0000 7fc03b252640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0341225e0 0x7fc03409e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:59.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.646+0000 7fc03b252640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:59.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.646+0000 7fc03b252640 1 -- 172.21.15.12:0/4178987960 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc034069950 con 0x7fc0341225e0 2024-09-11T01:43:59.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.646+0000 7fc033fff640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0341225e0 0x7fc03409e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:59.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.647+0000 7fc033fff640 10 monclient: get_auth_request con 0x7fc0341225e0 auth_method 0 2024-09-11T01:43:59.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.647+0000 7fc033fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:59.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.647+0000 7fc033fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:59.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.647+0000 7fc033fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:59.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.647+0000 7fc033fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:59.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.647+0000 7fc033fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:59.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.647+0000 7fc033fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:59.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.647+0000 7fc033fff640 10 monclient(hunting): handle_auth_done global_id 164102 payload 824 2024-09-11T01:43:59.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.647+0000 7fc033fff640 1 -- 172.21.15.12:0/4178987960 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc0341cdeb0 con 0x7fc0341225e0 2024-09-11T01:43:59.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.649+0000 7fc033fff640 1 -- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0341225e0 msgr2=0x7fc03409e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:59.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.649+0000 7fc033fff640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0341225e0 0x7fc03409e420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc02802a010 tx=0x7fc0280046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:59.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.649+0000 7fc0317fa640 1 -- 172.21.15.12:0/4178987960 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc02804b040 con 0x7fc0341225e0 2024-09-11T01:43:59.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.649+0000 7fc0317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:59.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.649+0000 7fc0317fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:43:59.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.649+0000 7fc0317fa640 10 monclient: dump: 2024-09-11T01:43:59.652 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:59.652 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:59.652 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:59.652 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:59.652 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:59.652 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:59.652 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:59.652 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:59.652 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:59.652 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:59.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.650+0000 7fc0317fa640 1 -- 172.21.15.12:0/4178987960 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7fc02804d820 con 0x7fc0341225e0 2024-09-11T01:43:59.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.650+0000 7fc0317fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:43:59.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.650+0000 7fc0317fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:43:59.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.650+0000 7fc0317fa640 1 -- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0341225e0 msgr2=0x7fc03409e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:59.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.650+0000 7fc0317fa640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0341225e0 0x7fc03409e420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc02802a010 tx=0x7fc0280046c0 comp rx=0 tx=0).stop 2024-09-11T01:43:59.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.650+0000 7fc0317fa640 1 -- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0341225e0 msgr2=0x7fc03409e420 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:43:59.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.650+0000 7fc0317fa640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0341225e0 0x7fc03409e420 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc02802a010 tx=0x7fc0280046c0 comp rx=0 tx=0).stop 2024-09-11T01:43:59.654 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:43:59.654 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:43:59.654 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:43:59.654 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:43:59.654 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 138, 2024-09-11T01:43:59.654 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 15662, 2024-09-11T01:43:59.654 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:43:59.654 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:43:59.654 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:43:59.654 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:43:59.655 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:59.655 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 4, 2024-09-11T01:43:59.655 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:43:59.655 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:43:59.655 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:43:59.655 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:59.655 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:59.655 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:59.655 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:59.655 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:59.655 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:59.656 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:59.656 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:59.656 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:59.656 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:59.656 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:59.656 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:59.656 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:43:59.656 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:43:59.656 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:59.656 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:59.656 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:59.656 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:59.656 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:59.657 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:59.657 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:59.657 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:59.657 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:59.657 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:59.657 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:59.657 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:59.657 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:59.657 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:43:59.657 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:43:59.657 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:43:59.658 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:43:59.658 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:43:59.658 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:43:59.658 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:43:59.658 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:43:59.659 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:43:59.659 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:43:59.659 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:43:59.659 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:43:59.659 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:43:59.659 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:43:59.659 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:43:59.660 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:43:59.660 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:43:59.660 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:59.660 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:59.660 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:59.660 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:59.660 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:59.660 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:59.660 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:59.660 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:59.660 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:59.660 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:59.660 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:59.661 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:59.661 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:43:59.661 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:59.661 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:43:59.661 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:59.661 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:43:59.661 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:43:59.661 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:59.661 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:59.661 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:59.661 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:59.662 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:43:59.662 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:59.662 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:59.662 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:59.662 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:59.662 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:43:59.662 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:59.662 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:59.662 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:59.662 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:59.662 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:43:59.662 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:43:59.663 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:59.663 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:59.663 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:59.663 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:59.663 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:59.663 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:43:59.663 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:43:59.663 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:59.663 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:59.663 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:59.664 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:59.664 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:43:59.664 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:59.664 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:59.664 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:59.664 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:59.664 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:43:59.664 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:59.664 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:59.664 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:59.665 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:59.665 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:43:59.665 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:43:59.665 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:59.665 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:59.665 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:59.665 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:59.665 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:59.665 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:43:59.665 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:43:59.665 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:59.665 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:59.665 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:59.666 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:59.666 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:43:59.666 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:59.666 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:59.666 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:59.666 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:59.666 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:43:59.666 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:59.666 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:59.666 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:59.666 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:59.666 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:43:59.667 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:43:59.667 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:59.667 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:59.667 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:59.667 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:59.667 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:59.667 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:59.667 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:43:59.667 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:43:59.667 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:59.668 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:59.668 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:59.668 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:43:59.668 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:59.668 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:59.668 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:43:59.668 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:59.668 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:59.668 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:59.668 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:43:59.668 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:59.669 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:59.669 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:59.669 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:43:59.669 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:43:59.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.650+0000 7fc03b252640 1 -- 172.21.15.12:0/4178987960 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc02003e210 msgr2=0x7fc0200406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.650+0000 7fc03b252640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc02003e210 0x7fc0200406d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc02c007f50 tx=0x7fc02c009040 comp rx=0 tx=0).stop 2024-09-11T01:43:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.650+0000 7fc03b252640 10 monclient: shutdown 2024-09-11T01:43:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.650+0000 7fc03b252640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.650+0000 7fc03b252640 1 -- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc034134cb0 msgr2=0x7fc0341cd3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.650+0000 7fc03b252640 1 --2- 172.21.15.12:0/4178987960 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc034134cb0 0x7fc0341cd3d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc02400e810 tx=0x7fc02400ecd0 comp rx=0 tx=0).stop 2024-09-11T01:43:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.650+0000 7fc0397ef640 1 -- 172.21.15.12:0/4178987960 reap_dead start 2024-09-11T01:43:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.650+0000 7fc0397ef640 1 -- 172.21.15.12:0/4178987960 reap_dead start 2024-09-11T01:43:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.650+0000 7fc03b252640 1 -- 172.21.15.12:0/4178987960 shutdown_connections 2024-09-11T01:43:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.650+0000 7fc03b252640 1 -- 172.21.15.12:0/4178987960 >> 172.21.15.12:0/4178987960 conn(0x7fc0340a7020 msgr2=0x7fc034136100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.650+0000 7fc03b252640 1 -- 172.21.15.12:0/4178987960 shutdown_connections 2024-09-11T01:43:59.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.651+0000 7fc03b252640 1 -- 172.21.15.12:0/4178987960 wait complete. 2024-09-11T01:43:59.672 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:43:59.672 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:43:59.672 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:43:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.782+0000 7f1711c18640 10 monclient: get_monmap_and_config 2024-09-11T01:43:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.782+0000 7f1711c18640 10 monclient: build_initial_monmap 2024-09-11T01:43:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.782+0000 7f1711c18640 10 monclient: monmap: 2024-09-11T01:43:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:43:59.784302+0000 2024-09-11T01:43:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:43:59.784302+0000 2024-09-11T01:43:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:43:59.785 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:43:59.786 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:43:59.786 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:43:59.786 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:43:59.786 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:59.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.783+0000 7f1711c18640 1 Processor -- start 2024-09-11T01:43:59.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.783+0000 7f1711c18640 1 -- start start 2024-09-11T01:43:59.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.783+0000 7f1711c18640 10 monclient: init 2024-09-11T01:43:59.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.783+0000 7f1711c18640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:59.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.783+0000 7f1711c18640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-11T01:43:59.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.783+0000 7f1711c18640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f170c0a2e20 addr v1:172.21.15.12:6789/0 2024-09-11T01:43:59.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.783+0000 7f1711c18640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f170c0a4970 addr v1:172.21.15.71:6789/0 2024-09-11T01:43:59.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.783+0000 7f1711c18640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f170c06f910 0x7f170c06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:59.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.783+0000 7f1711c18640 10 monclient(hunting): picked mon.noname-a con 0x7f170c06f910 addr v2:172.21.15.12:3300/0 2024-09-11T01:43:59.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.783+0000 7f1711c18640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f170c06f4f0 con 0x7f170c0a2e20 2024-09-11T01:43:59.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.783+0000 7f1711c18640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f170c0a3910 con 0x7f170c0a4970 2024-09-11T01:43:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.783+0000 7f1711c18640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.783+0000 7f1711c18640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f170c06ef30 con 0x7f170c06f910 2024-09-11T01:43:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.783+0000 7f1711c18640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.783+0000 7f1711c18640 10 monclient(hunting): authenticate will time out at 1646.226983s 2024-09-11T01:43:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.784+0000 7f170b7fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f170c0a2e20 0x7f170c0a3200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53782/0 (socket says 172.21.15.12:53782) 2024-09-11T01:43:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.784+0000 7f170bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f170c06f910 0x7f170c06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.784+0000 7f170b7fe640 1 -- 172.21.15.12:0/1810504008 learned_addr learned my addr 172.21.15.12:0/1810504008 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:43:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.784+0000 7f170bfff640 10 monclient(hunting): get_auth_request con 0x7f170c06f910 auth_method 0 2024-09-11T01:43:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.784+0000 7f170bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.784+0000 7f170bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.784+0000 7f170bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.784+0000 7f170bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.784+0000 7f170bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:59.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.784+0000 7f170bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:59.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.784+0000 7f170bfff640 10 monclient(hunting): handle_auth_done global_id 94782 payload 274 2024-09-11T01:43:59.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.784+0000 7f170bfff640 1 -- 172.21.15.12:0/1810504008 >> v1:172.21.15.71:6789/0 conn(0x7f170c0a4970 legacy=0x7f170c13d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:59.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.784+0000 7f170bfff640 1 -- 172.21.15.12:0/1810504008 >> v1:172.21.15.12:6789/0 conn(0x7f170c0a2e20 legacy=0x7f170c0a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:59.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.784+0000 7f170bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:43:59.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.784+0000 7f170bfff640 1 monclient: found mon.noname-a 2024-09-11T01:43:59.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.784+0000 7f170bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:59.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.784+0000 7f170bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:59.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.784+0000 7f170bfff640 1 -- 172.21.15.12:0/1810504008 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f170c0a3ae0 con 0x7f170c06f910 2024-09-11T01:43:59.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f170bfff640 1 -- 172.21.15.12:0/1810504008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f170c06f910 msgr2=0x7f170c06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:59.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f170bfff640 1 --2- 172.21.15.12:0/1810504008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f170c06f910 0x7f170c06ded0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f1704009d90 tx=0x7f170402a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de6d6c6cca209327 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:59.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f170a7fc640 1 -- 172.21.15.12:0/1810504008 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1704007a30 con 0x7f170c06f910 2024-09-11T01:43:59.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f170a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:59.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f170a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:59.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f170a7fc640 10 monclient: dump: 2024-09-11T01:43:59.797 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:59.797 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:59.797 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:59.797 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:59.797 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:59.798 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f170a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:43:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f170a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:43:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f170a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f170a7fc640 1 -- 172.21.15.12:0/1810504008 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1704007bd0 con 0x7f170c06f910 2024-09-11T01:43:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f1711c18640 5 monclient: authenticate success, global_id 94782 2024-09-11T01:43:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f1711c18640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:43:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f170a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:59.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f1711c18640 10 monclient: get_monmap_and_config success 2024-09-11T01:43:59.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f170a7fc640 1 -- 172.21.15.12:0/1810504008 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1704031400 con 0x7f170c06f910 2024-09-11T01:43:59.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f170a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:59.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f1711c18640 4 set_mon_vals no callback set 2024-09-11T01:43:59.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f170a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:59.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f170a7fc640 10 monclient: dump: 2024-09-11T01:43:59.799 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:59.799 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:59.799 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:59.799 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:59.799 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:59.799 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:59.800 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:59.800 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:59.800 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:59.800 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:59.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f1711c18640 10 monclient: shutdown 2024-09-11T01:43:59.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f1711c18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:59.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f1711c18640 1 -- 172.21.15.12:0/1810504008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f170c06f910 msgr2=0x7f170c06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:59.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f1711c18640 1 --2- 172.21.15.12:0/1810504008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f170c06f910 0x7f170c06ded0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f1704009d90 tx=0x7f170402a490 comp rx=0 tx=0).stop 2024-09-11T01:43:59.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f1711c18640 1 -- 172.21.15.12:0/1810504008 shutdown_connections 2024-09-11T01:43:59.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f1711c18640 1 --2- 172.21.15.12:0/1810504008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f170c06f910 0x7f170c06ded0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:59.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f1711c18640 1 -- 172.21.15.12:0/1810504008 >> 172.21.15.12:0/1810504008 conn(0x7f170c0a8600 msgr2=0x7f170c0a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:59.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f1711c18640 1 -- 172.21.15.12:0/1810504008 shutdown_connections 2024-09-11T01:43:59.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.787+0000 7f1711c18640 1 -- 172.21.15.12:0/1810504008 wait complete. 2024-09-11T01:43:59.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.788+0000 7f1711c18640 10 monclient: build_initial_monmap 2024-09-11T01:43:59.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.788+0000 7f1711c18640 10 monclient: monmap: 2024-09-11T01:43:59.801 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:59.801 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:43:59.801 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:43:59.801 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:43:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:43:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:43:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:43:59.802 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.788+0000 7f1711c18640 1 Processor -- start 2024-09-11T01:43:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f1711c18640 1 -- start start 2024-09-11T01:43:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f1711c18640 10 monclient: init 2024-09-11T01:43:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f1711c18640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f1711c18640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:43:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f1711c18640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f170c06f910 0x7f170c13f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:59.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f1711c18640 10 monclient(hunting): picked mon.noname-b con 0x7f170c06f910 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f1711c18640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f170c0a2e20 0x7f170c13fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f1711c18640 10 monclient(hunting): picked mon.noname-c con 0x7f170c0a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f1711c18640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f170c0a4970 0x7f170c1371b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f1711c18640 10 monclient(hunting): picked mon.noname-a con 0x7f170c0a4970 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f1711c18640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f1711c18640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f170c140d60 con 0x7f170c0a4970 2024-09-11T01:43:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f1711c18640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f1711c18640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f170c06ec50 con 0x7f170c06f910 2024-09-11T01:43:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f1711c18640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f1711c18640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f170c140ee0 con 0x7f170c0a2e20 2024-09-11T01:43:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f1711c18640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f1711c18640 10 monclient(hunting): authenticate will time out at 1646.232549s 2024-09-11T01:43:59.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f170c0a4970 0x7f170c1371b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170affd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f170c0a2e20 0x7f170c13fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f170c0a4970 0x7f170c1371b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35328/0 (socket says 172.21.15.12:35328) 2024-09-11T01:43:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170bfff640 1 -- 172.21.15.12:0/4222294704 learned_addr learned my addr 172.21.15.12:0/4222294704 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170bfff640 10 monclient(hunting): get_auth_request con 0x7f170c0a4970 auth_method 0 2024-09-11T01:43:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170affd640 10 monclient(hunting): get_auth_request con 0x7f170c0a2e20 auth_method 0 2024-09-11T01:43:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:59.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:59.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170b7fe640 1 --2- 172.21.15.12:0/4222294704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f170c06f910 0x7f170c13f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:59.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:59.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:59.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:59.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:59.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:59.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:59.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170b7fe640 10 monclient(hunting): get_auth_request con 0x7f170c06f910 auth_method 0 2024-09-11T01:43:59.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:59.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:59.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:59.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:59.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:59.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:59.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.789+0000 7f170bfff640 10 monclient(hunting): handle_auth_done global_id 94785 payload 824 2024-09-11T01:43:59.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.790+0000 7f170bfff640 1 -- 172.21.15.12:0/4222294704 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f170c0a2e20 msgr2=0x7f170c13fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:59.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.790+0000 7f170bfff640 1 --2- 172.21.15.12:0/4222294704 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f170c0a2e20 0x7f170c13fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:59.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.790+0000 7f170bfff640 1 -- 172.21.15.12:0/4222294704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f170c06f910 msgr2=0x7f170c13f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:59.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.790+0000 7f170bfff640 1 --2- 172.21.15.12:0/4222294704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f170c06f910 0x7f170c13f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:59.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.790+0000 7f170bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:43:59.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.790+0000 7f170bfff640 1 monclient: found mon.noname-a 2024-09-11T01:43:59.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.790+0000 7f170bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:59.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.790+0000 7f170bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:59.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.790+0000 7f170bfff640 1 -- 172.21.15.12:0/4222294704 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f170c09f610 con 0x7f170c0a4970 2024-09-11T01:43:59.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.790+0000 7f170affd640 1 --2- 172.21.15.12:0/4222294704 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f170c0a2e20 0x7f170c13fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:43:59.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.792+0000 7f170bfff640 1 -- 172.21.15.12:0/4222294704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f170c0a4970 msgr2=0x7f170c1371b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:59.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.792+0000 7f170bfff640 1 --2- 172.21.15.12:0/4222294704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f170c0a4970 0x7f170c1371b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1704009eb0 tx=0x7f1704004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:59.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.792+0000 7f1703fff640 1 -- 172.21.15.12:0/4222294704 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1704040070 con 0x7f170c0a4970 2024-09-11T01:43:59.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.792+0000 7f1703fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:59.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.792+0000 7f1703fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:59.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.792+0000 7f1703fff640 10 monclient: dump: 2024-09-11T01:43:59.807 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:59.807 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:59.807 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:59.807 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:59.807 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:59.807 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:59.807 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:59.808 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.792+0000 7f1703fff640 10 monclient: _finish_auth 0 2024-09-11T01:43:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.792+0000 7f1703fff640 10 monclient: _check_auth_tickets 2024-09-11T01:43:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.792+0000 7f1703fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.792+0000 7f1711c18640 5 monclient: authenticate success, global_id 94785 2024-09-11T01:43:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.792+0000 7f1703fff640 1 -- 172.21.15.12:0/4222294704 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1704045660 con 0x7f170c0a4970 2024-09-11T01:43:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.792+0000 7f1703fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.792+0000 7f1703fff640 1 -- 172.21.15.12:0/4222294704 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17040314c0 con 0x7f170c0a4970 2024-09-11T01:43:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.792+0000 7f1711c18640 10 monclient: _renew_subs 2024-09-11T01:43:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.792+0000 7f1711c18640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.792+0000 7f1711c18640 1 -- 172.21.15.12:0/4222294704 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f170c140060 con 0x7f170c0a4970 2024-09-11T01:43:59.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.792+0000 7f1709ffb640 4 set_mon_vals no callback set 2024-09-11T01:43:59.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.792+0000 7f1703fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:59.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.792+0000 7f1703fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:59.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.792+0000 7f1703fff640 10 monclient: dump: 2024-09-11T01:43:59.809 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:59.809 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:59.809 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:59.809 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:59.809 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:59.809 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:59.809 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:59.809 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:59.809 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:59.809 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:59.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.792+0000 7f1711c18640 10 monclient: _renew_subs 2024-09-11T01:43:59.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.792+0000 7f1711c18640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:59.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.792+0000 7f1711c18640 1 -- 172.21.15.12:0/4222294704 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f170c13aae0 con 0x7f170c0a4970 2024-09-11T01:43:59.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.793+0000 7f1711c18640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:59.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.793+0000 7f1711c18640 1 --2- 172.21.15.12:0/4222294704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f170c138b70 0x7f170c137eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:59.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.793+0000 7f1711c18640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:59.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.793+0000 7f1711c18640 1 -- 172.21.15.12:0/4222294704 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f170c06a060 con 0x7f170c138b70 2024-09-11T01:43:59.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.793+0000 7f1703fff640 1 -- 172.21.15.12:0/4222294704 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f1704007a40 con 0x7f170c0a4970 2024-09-11T01:43:59.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.793+0000 7f170b7fe640 1 --2- 172.21.15.12:0/4222294704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f170c138b70 0x7f170c137eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:59.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.793+0000 7f170b7fe640 10 monclient: get_auth_request con 0x7f170c138b70 auth_method 0 2024-09-11T01:43:59.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.793+0000 7f170b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:59.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.793+0000 7f170b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:59.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.793+0000 7f170b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:59.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.793+0000 7f170b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:59.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.793+0000 7f170b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:59.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.793+0000 7f170b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:59.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.793+0000 7f1703fff640 1 --2- 172.21.15.12:0/4222294704 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f16ec03e210 0x7f16ec0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:59.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.793+0000 7f1703fff640 1 -- 172.21.15.12:0/4222294704 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(508..508 src has 317..508) ==== 4859+0+0 (secure 0 0 0) 0x7f1704074c10 con 0x7f170c0a4970 2024-09-11T01:43:59.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.793+0000 7f170b7fe640 10 monclient(hunting): handle_auth_done global_id 134377 payload 824 2024-09-11T01:43:59.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.793+0000 7f170b7fe640 1 -- 172.21.15.12:0/4222294704 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f170c13b000 con 0x7f170c138b70 2024-09-11T01:43:59.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.794+0000 7f170affd640 1 --2- 172.21.15.12:0/4222294704 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f16ec03e210 0x7f16ec0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:59.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.794+0000 7f170affd640 10 monclient: get_auth_request con 0x7f16ec03e210 auth_method 0 2024-09-11T01:43:59.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.796+0000 7f170b7fe640 1 -- 172.21.15.12:0/4222294704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f170c138b70 msgr2=0x7f170c137eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:59.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.796+0000 7f170b7fe640 1 --2- 172.21.15.12:0/4222294704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f170c138b70 0x7f170c137eb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f16f400ea70 tx=0x7f16f400ef30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:59.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.796+0000 7f1703fff640 1 -- 172.21.15.12:0/4222294704 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f401a070 con 0x7f170c138b70 2024-09-11T01:43:59.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.796+0000 7f1703fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:59.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.796+0000 7f1703fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:43:59.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.796+0000 7f1703fff640 10 monclient: dump: 2024-09-11T01:43:59.816 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:59.816 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:59.816 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:59.816 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:59.816 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:59.816 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:59.816 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:59.816 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:59.816 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:59.816 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:59.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.796+0000 7f170affd640 1 --2- 172.21.15.12:0/4222294704 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f16ec03e210 0x7f16ec0406d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f16fc004320 tx=0x7f16fc007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:59.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.796+0000 7f1703fff640 1 -- 172.21.15.12:0/4222294704 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f16f400cc50 con 0x7f170c138b70 2024-09-11T01:43:59.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.796+0000 7f1703fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:59.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.796+0000 7f1703fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:43:59.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.796+0000 7f1703fff640 1 -- 172.21.15.12:0/4222294704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f170c138b70 msgr2=0x7f170c137eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:59.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.796+0000 7f1703fff640 1 --2- 172.21.15.12:0/4222294704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f170c138b70 0x7f170c137eb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f16f400ea70 tx=0x7f16f400ef30 comp rx=0 tx=0).stop 2024-09-11T01:43:59.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.796+0000 7f1703fff640 1 -- 172.21.15.12:0/4222294704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f170c138b70 msgr2=0x7f170c137eb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:43:59.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.796+0000 7f1703fff640 1 --2- 172.21.15.12:0/4222294704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f170c138b70 0x7f170c137eb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:59.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.803+0000 7f1711c18640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:43:59.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.803+0000 7f1711c18640 1 --2- 172.21.15.12:0/4222294704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f170c1383f0 0x7f170c02d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:59.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.803+0000 7f1711c18640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:59.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.803+0000 7f1711c18640 1 -- 172.21.15.12:0/4222294704 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f170c06ded0 con 0x7f170c1383f0 2024-09-11T01:43:59.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.803+0000 7f170b7fe640 1 --2- 172.21.15.12:0/4222294704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f170c1383f0 0x7f170c02d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:59.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.803+0000 7f170b7fe640 10 monclient: get_auth_request con 0x7f170c1383f0 auth_method 0 2024-09-11T01:43:59.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.803+0000 7f170b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:59.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.803+0000 7f170b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:59.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.803+0000 7f170b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:59.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.803+0000 7f170b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:59.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.803+0000 7f170b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:59.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.803+0000 7f170b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:59.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.803+0000 7f170b7fe640 10 monclient(hunting): handle_auth_done global_id 134380 payload 824 2024-09-11T01:43:59.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.803+0000 7f170b7fe640 1 -- 172.21.15.12:0/4222294704 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f170c0a0f30 con 0x7f170c1383f0 2024-09-11T01:43:59.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.806+0000 7f170b7fe640 1 -- 172.21.15.12:0/4222294704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f170c1383f0 msgr2=0x7f170c02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:59.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.806+0000 7f170b7fe640 1 --2- 172.21.15.12:0/4222294704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f170c1383f0 0x7f170c02d440 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f16f4017000 tx=0x7f16f4004940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:59.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.806+0000 7f1703fff640 1 -- 172.21.15.12:0/4222294704 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f401a050 con 0x7f170c1383f0 2024-09-11T01:43:59.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.806+0000 7f1703fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:59.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.806+0000 7f1703fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:43:59.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.806+0000 7f1703fff640 10 monclient: dump: 2024-09-11T01:43:59.818 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:59.819 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.806+0000 7f1703fff640 1 -- 172.21.15.12:0/4222294704 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f16f401b040 con 0x7f170c1383f0 2024-09-11T01:43:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.806+0000 7f1703fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:43:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.806+0000 7f1703fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:43:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.806+0000 7f1703fff640 1 -- 172.21.15.12:0/4222294704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f170c1383f0 msgr2=0x7f170c02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.806+0000 7f1703fff640 1 --2- 172.21.15.12:0/4222294704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f170c1383f0 0x7f170c02d440 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f16f4017000 tx=0x7f16f4004940 comp rx=0 tx=0).stop 2024-09-11T01:43:59.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.806+0000 7f1703fff640 1 -- 172.21.15.12:0/4222294704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f170c1383f0 msgr2=0x7f170c02d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:43:59.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.806+0000 7f1703fff640 1 --2- 172.21.15.12:0/4222294704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f170c1383f0 0x7f170c02d440 secure :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f16f4017000 tx=0x7f16f4004940 comp rx=0 tx=0).stop 2024-09-11T01:43:59.820 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:43:59.820 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:43:59.820 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:43:59.820 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:43:59.820 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 138, 2024-09-11T01:43:59.820 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 172769, 2024-09-11T01:43:59.820 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:43:59.820 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:43:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:43:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:43:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 4, 2024-09-11T01:43:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:43:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:43:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:43:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:59.821 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:59.822 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:59.822 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:59.822 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:59.822 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:59.822 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:59.822 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:43:59.822 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:43:59.822 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:59.822 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:59.822 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:59.822 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:59.822 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:59.822 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:43:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:43:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:43:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:43:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:43:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:43:59.823 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:43:59.824 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:43:59.824 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:43:59.824 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:43:59.824 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:43:59.824 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:43:59.824 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:43:59.824 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:43:59.824 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:43:59.824 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:43:59.824 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:43:59.824 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:43:59.824 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:43:59.824 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:43:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:43:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:43:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:43:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:43:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:43:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:43:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:43:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:43:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:43:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:43:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:59.825 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:43:59.826 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:43:59.826 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:59.826 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:59.826 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:59.826 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:59.826 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:43:59.826 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:59.826 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:59.826 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:59.826 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:59.826 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:43:59.826 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:59.826 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:59.826 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:59.827 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:59.827 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:43:59.827 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:43:59.827 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:59.827 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:59.827 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:59.827 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:59.827 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:59.827 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:43:59.827 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:43:59.827 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:59.827 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:59.827 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:43:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:43:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:43:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:43:59.828 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:59.829 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:59.829 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:59.829 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:59.829 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:59.829 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:43:59.829 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:43:59.829 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:43:59.829 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:43:59.829 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:59.829 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:43:59.829 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:43:59.829 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:59.829 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:43:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:43:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:43:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:43:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:43:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:43:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:43:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:43:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:59.830 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:43:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:43:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:43:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:43:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:59.831 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:43:59.832 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:59.832 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:59.832 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:43:59.832 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:59.832 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:59.832 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:59.832 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:43:59.832 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:59.832 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:43:59.832 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:43:59.832 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:43:59.833 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:43:59.833 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:43:59.833 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:43:59.833 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:43:59.833 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:43:59.833 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:43:59.833 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:43:59.833 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:43:59.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.807+0000 7f1711c18640 1 -- 172.21.15.12:0/4222294704 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f16ec03e210 msgr2=0x7f16ec0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:59.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.807+0000 7f1711c18640 1 --2- 172.21.15.12:0/4222294704 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f16ec03e210 0x7f16ec0406d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f16fc004320 tx=0x7f16fc007e30 comp rx=0 tx=0).stop 2024-09-11T01:43:59.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.807+0000 7f1711c18640 10 monclient: shutdown 2024-09-11T01:43:59.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.807+0000 7f1711c18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:59.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.807+0000 7f1711c18640 1 -- 172.21.15.12:0/4222294704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f170c0a4970 msgr2=0x7f170c1371b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:59.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.807+0000 7f1711c18640 1 --2- 172.21.15.12:0/4222294704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f170c0a4970 0x7f170c1371b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1704009eb0 tx=0x7f1704004a40 comp rx=0 tx=0).stop 2024-09-11T01:43:59.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.807+0000 7f170bfff640 1 -- 172.21.15.12:0/4222294704 reap_dead start 2024-09-11T01:43:59.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.807+0000 7f170bfff640 1 -- 172.21.15.12:0/4222294704 reap_dead start 2024-09-11T01:43:59.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.807+0000 7f1711c18640 1 -- 172.21.15.12:0/4222294704 shutdown_connections 2024-09-11T01:43:59.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.807+0000 7f1711c18640 1 -- 172.21.15.12:0/4222294704 >> 172.21.15.12:0/4222294704 conn(0x7f170c0a8600 msgr2=0x7f170c09e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:59.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.807+0000 7f1711c18640 1 -- 172.21.15.12:0/4222294704 shutdown_connections 2024-09-11T01:43:59.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.807+0000 7f1711c18640 1 -- 172.21.15.12:0/4222294704 wait complete. 2024-09-11T01:43:59.836 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:43:59.836 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:43:59.836 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:43:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.935+0000 7fe97ac18640 10 monclient: get_monmap_and_config 2024-09-11T01:43:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.935+0000 7fe97ac18640 10 monclient: build_initial_monmap 2024-09-11T01:43:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.935+0000 7fe97ac18640 10 monclient: monmap: 2024-09-11T01:43:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:59.937 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:43:59.936823+0000 2024-09-11T01:43:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:43:59.936823+0000 2024-09-11T01:43:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:43:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:43:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:43:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:43:59.938 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:43:59.938 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:59.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.935+0000 7fe97ac18640 1 Processor -- start 2024-09-11T01:43:59.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.935+0000 7fe97ac18640 1 -- start start 2024-09-11T01:43:59.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.935+0000 7fe97ac18640 10 monclient: init 2024-09-11T01:43:59.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe97ac18640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:59.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe97ac18640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-11T01:43:59.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe97ac18640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe974137e50 addr v1:172.21.15.12:6789/0 2024-09-11T01:43:59.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe97ac18640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe974138230 0x7fe974138640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:59.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe97ac18640 10 monclient(hunting): picked mon.noname-a con 0x7fe974138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:43:59.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe97ac18640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe97413d380 addr v1:172.21.15.71:6789/0 2024-09-11T01:43:59.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe97ac18640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9740a62d0 con 0x7fe974137e50 2024-09-11T01:43:59.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe97ac18640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe97409bb40 con 0x7fe97413d380 2024-09-11T01:43:59.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe97ac18640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:59.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe97ac18640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fe97409c4a0 con 0x7fe974138230 2024-09-11T01:43:59.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe97ac18640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:59.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe97ac18640 10 monclient(hunting): authenticate will time out at 1646.379466s 2024-09-11T01:43:59.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe973fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe974138230 0x7fe974138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:59.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe973fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe974138230 0x7fe974138640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35332/0 (socket says 172.21.15.12:35332) 2024-09-11T01:43:59.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe973fff640 1 -- 172.21.15.12:0/614836807 learned_addr learned my addr 172.21.15.12:0/614836807 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:59.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe9789b4640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe974137e50 0x7fe97409d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53798/0 (socket says 172.21.15.12:53798) 2024-09-11T01:43:59.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe973fff640 10 monclient(hunting): get_auth_request con 0x7fe974138230 auth_method 0 2024-09-11T01:43:59.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe973fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:59.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe973fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:59.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe973fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:59.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe973fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:59.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe973fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:59.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe973fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:59.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe973fff640 10 monclient(hunting): handle_auth_done global_id 94788 payload 274 2024-09-11T01:43:59.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe973fff640 1 -- 172.21.15.12:0/614836807 >> v1:172.21.15.71:6789/0 conn(0x7fe97413d380 legacy=0x7fe97413f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:59.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe973fff640 1 -- 172.21.15.12:0/614836807 >> v1:172.21.15.12:6789/0 conn(0x7fe974137e50 legacy=0x7fe97409d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:43:59.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe973fff640 10 monclient: _finish_hunting 0 2024-09-11T01:43:59.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe973fff640 1 monclient: found mon.noname-a 2024-09-11T01:43:59.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe973fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:59.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe973fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:59.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.936+0000 7fe973fff640 1 -- 172.21.15.12:0/614836807 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe97409c780 con 0x7fe974138230 2024-09-11T01:43:59.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.939+0000 7fe973fff640 1 -- 172.21.15.12:0/614836807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe974138230 msgr2=0x7fe974138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:59.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.939+0000 7fe973fff640 1 --2- 172.21.15.12:0/614836807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe974138230 0x7fe974138640 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe9740a6f80 tx=0x7fe95c02a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e87bb36c80962378 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:59.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.940+0000 7fe9737fe640 1 -- 172.21.15.12:0/614836807 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe95c037070 con 0x7fe974138230 2024-09-11T01:43:59.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.940+0000 7fe9737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:59.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.940+0000 7fe9737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:59.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.940+0000 7fe9737fe640 10 monclient: dump: 2024-09-11T01:43:59.947 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:59.947 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:59.947 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:59.947 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:59.947 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:59.947 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:59.947 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:59.947 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:59.947 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:59.947 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.940+0000 7fe9737fe640 10 monclient: _finish_auth 0 2024-09-11T01:43:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.940+0000 7fe9737fe640 10 monclient: _check_auth_tickets 2024-09-11T01:43:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.940+0000 7fe9737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.940+0000 7fe9737fe640 1 -- 172.21.15.12:0/614836807 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe95c032640 con 0x7fe974138230 2024-09-11T01:43:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.940+0000 7fe9737fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.940+0000 7fe9737fe640 1 -- 172.21.15.12:0/614836807 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe95c032a10 con 0x7fe974138230 2024-09-11T01:43:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.940+0000 7fe97ac18640 5 monclient: authenticate success, global_id 94788 2024-09-11T01:43:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.940+0000 7fe97ac18640 10 monclient: get_monmap_and_config success 2024-09-11T01:43:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.940+0000 7fe97ac18640 4 set_mon_vals no callback set 2024-09-11T01:43:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.940+0000 7fe9737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.940+0000 7fe9737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.940+0000 7fe9737fe640 10 monclient: dump: 2024-09-11T01:43:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:59.949 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:59.949 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:59.949 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:59.949 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:59.949 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:59.949 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:59.949 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:59.949 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:59.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.940+0000 7fe97ac18640 10 monclient: shutdown 2024-09-11T01:43:59.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.940+0000 7fe97ac18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:43:59.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.940+0000 7fe97ac18640 1 -- 172.21.15.12:0/614836807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe974138230 msgr2=0x7fe974138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:59.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.940+0000 7fe97ac18640 1 --2- 172.21.15.12:0/614836807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe974138230 0x7fe974138640 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe9740a6f80 tx=0x7fe95c02a060 comp rx=0 tx=0).stop 2024-09-11T01:43:59.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.940+0000 7fe97ac18640 1 -- 172.21.15.12:0/614836807 shutdown_connections 2024-09-11T01:43:59.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.940+0000 7fe97ac18640 1 --2- 172.21.15.12:0/614836807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe974138230 0x7fe974138640 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:59.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.940+0000 7fe97ac18640 1 -- 172.21.15.12:0/614836807 >> 172.21.15.12:0/614836807 conn(0x7fe9740a7020 msgr2=0x7fe9740a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:43:59.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.940+0000 7fe97ac18640 1 -- 172.21.15.12:0/614836807 shutdown_connections 2024-09-11T01:43:59.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.940+0000 7fe97ac18640 1 -- 172.21.15.12:0/614836807 wait complete. 2024-09-11T01:43:59.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.941+0000 7fe97ac18640 10 monclient: build_initial_monmap 2024-09-11T01:43:59.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.941+0000 7fe97ac18640 10 monclient: monmap: 2024-09-11T01:43:59.950 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:43:59.950 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:43:59.950 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:43:59.950 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:43:59.950 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:43:59.950 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:59.950 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:43:59.950 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:43:59.950 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:43:59.951 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:59.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.941+0000 7fe97ac18640 1 Processor -- start 2024-09-11T01:43:59.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.941+0000 7fe97ac18640 1 -- start start 2024-09-11T01:43:59.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.941+0000 7fe97ac18640 10 monclient: init 2024-09-11T01:43:59.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe97ac18640 10 monclient: _reopen_session rank -1 2024-09-11T01:43:59.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe97ac18640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:43:59.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe97ac18640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe974137e50 0x7fe97413fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:59.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe97ac18640 10 monclient(hunting): picked mon.noname-a con 0x7fe974137e50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:43:59.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe97ac18640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe974138230 0x7fe9740a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:59.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe97ac18640 10 monclient(hunting): picked mon.noname-b con 0x7fe974138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:43:59.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe97ac18640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe97413d380 0x7fe9740a1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:59.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe97ac18640 10 monclient(hunting): picked mon.noname-c con 0x7fe97413d380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:43:59.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe97ac18640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:59.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe97ac18640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe97406f890 con 0x7fe974137e50 2024-09-11T01:43:59.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe97ac18640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:59.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe97ac18640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe97406f710 con 0x7fe974138230 2024-09-11T01:43:59.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe97ac18640 10 monclient(hunting): start opening mon connection 2024-09-11T01:43:59.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe97ac18640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe974140e90 con 0x7fe97413d380 2024-09-11T01:43:59.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe97ac18640 10 monclient(hunting): _renew_subs 2024-09-11T01:43:59.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe97ac18640 10 monclient(hunting): authenticate will time out at 1646.385360s 2024-09-11T01:43:59.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9789b4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe974137e50 0x7fe97413fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:59.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9789b4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe974137e50 0x7fe97413fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35336/0 (socket says 172.21.15.12:35336) 2024-09-11T01:43:59.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9791b5640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe97413d380 0x7fe9740a1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:59.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9789b4640 1 -- 172.21.15.12:0/1873838087 learned_addr learned my addr 172.21.15.12:0/1873838087 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:43:59.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9789b4640 10 monclient(hunting): get_auth_request con 0x7fe974137e50 auth_method 0 2024-09-11T01:43:59.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9789b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:59.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9789b4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:59.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9789b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:59.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9791b5640 10 monclient(hunting): get_auth_request con 0x7fe97413d380 auth_method 0 2024-09-11T01:43:59.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9791b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:59.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9791b5640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:59.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9791b5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:59.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9789b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:59.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9789b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:59.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9789b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:59.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9791b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:59.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9791b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:59.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe973fff640 1 --2- 172.21.15.12:0/1873838087 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe974138230 0x7fe9740a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:59.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9791b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe973fff640 10 monclient(hunting): get_auth_request con 0x7fe974138230 auth_method 0 2024-09-11T01:43:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe973fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:43:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe973fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:43:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe973fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:43:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe973fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:43:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe973fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:43:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe973fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:43:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9789b4640 10 monclient(hunting): handle_auth_done global_id 94791 payload 824 2024-09-11T01:43:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9789b4640 1 -- 172.21.15.12:0/1873838087 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe97413d380 msgr2=0x7fe9740a1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9789b4640 1 --2- 172.21.15.12:0/1873838087 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe97413d380 0x7fe9740a1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9789b4640 1 -- 172.21.15.12:0/1873838087 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe974138230 msgr2=0x7fe9740a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:43:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9789b4640 1 --2- 172.21.15.12:0/1873838087 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe974138230 0x7fe9740a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:43:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9789b4640 10 monclient: _finish_hunting 0 2024-09-11T01:43:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9789b4640 1 monclient: found mon.noname-a 2024-09-11T01:43:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9789b4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:43:59.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9789b4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:43:59.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9789b4640 1 -- 172.21.15.12:0/1873838087 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe974140af0 con 0x7fe974137e50 2024-09-11T01:43:59.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.942+0000 7fe9791b5640 1 --2- 172.21.15.12:0/1873838087 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe97413d380 0x7fe9740a1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:43:59.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.945+0000 7fe9789b4640 1 -- 172.21.15.12:0/1873838087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe974137e50 msgr2=0x7fe97413fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:43:59.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.945+0000 7fe9789b4640 1 --2- 172.21.15.12:0/1873838087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe974137e50 0x7fe97413fae0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe96400c450 tx=0x7fe96400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:59.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.945+0000 7fe9717fa640 1 -- 172.21.15.12:0/1873838087 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe964017070 con 0x7fe974137e50 2024-09-11T01:43:59.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.945+0000 7fe9717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:59.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.945+0000 7fe9717fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:43:59.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.945+0000 7fe9717fa640 10 monclient: dump: 2024-09-11T01:43:59.957 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:59.957 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:59.957 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:59.957 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:59.957 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:59.957 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:59.958 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:59.958 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:59.958 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:59.958 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:59.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.945+0000 7fe9717fa640 10 monclient: _finish_auth 0 2024-09-11T01:43:59.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.945+0000 7fe9717fa640 10 monclient: _check_auth_tickets 2024-09-11T01:43:59.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.945+0000 7fe9717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:43:59.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.945+0000 7fe9717fa640 1 -- 172.21.15.12:0/1873838087 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe964005810 con 0x7fe974137e50 2024-09-11T01:43:59.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.945+0000 7fe97ac18640 5 monclient: authenticate success, global_id 94791 2024-09-11T01:43:59.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.945+0000 7fe97ac18640 10 monclient: _renew_subs 2024-09-11T01:43:59.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.945+0000 7fe97ac18640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:59.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.945+0000 7fe97ac18640 1 -- 172.21.15.12:0/1873838087 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9741403e0 con 0x7fe974137e50 2024-09-11T01:43:59.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.945+0000 7fe9717fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:43:59.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.945+0000 7fe9717fa640 1 -- 172.21.15.12:0/1873838087 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe964007690 con 0x7fe974137e50 2024-09-11T01:43:59.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.945+0000 7fe9717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:43:59.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.945+0000 7fe9717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:43:59.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.945+0000 7fe9717fa640 10 monclient: dump: 2024-09-11T01:43:59.959 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:43:59.959 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:43:59.959 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:59.959 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:43:59.959 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:43:59.959 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:43:59.959 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:43:59.959 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:43:59.959 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:43:59.959 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:43:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.945+0000 7fe972ffd640 4 set_mon_vals no callback set 2024-09-11T01:43:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.945+0000 7fe97ac18640 10 monclient: _renew_subs 2024-09-11T01:43:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.945+0000 7fe97ac18640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.945+0000 7fe97ac18640 1 -- 172.21.15.12:0/1873838087 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe974137670 con 0x7fe974137e50 2024-09-11T01:43:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.946+0000 7fe97ac18640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.946+0000 7fe97ac18640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.946+0000 7fe97ac18640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:43:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.946+0000 7fe9717fa640 1 -- 172.21.15.12:0/1873838087 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fe964005c00 con 0x7fe974137e50 2024-09-11T01:43:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.946+0000 7fe97ac18640 1 -- 172.21.15.12:0/1873838087 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe974138640 con 0x7fe974137e50 2024-09-11T01:43:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.946+0000 7fe9717fa640 1 --2- 172.21.15.12:0/1873838087 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe95403e210 0x7fe9540406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:43:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.946+0000 7fe9717fa640 1 -- 172.21.15.12:0/1873838087 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(508..508 src has 317..508) ==== 4859+0+0 (secure 0 0 0) 0x7fe9640524a0 con 0x7fe974137e50 2024-09-11T01:43:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.946+0000 7fe973fff640 1 --2- 172.21.15.12:0/1873838087 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe95403e210 0x7fe9540406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:43:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.946+0000 7fe973fff640 10 monclient: get_auth_request con 0x7fe95403e210 auth_method 0 2024-09-11T01:43:59.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.949+0000 7fe973fff640 1 --2- 172.21.15.12:0/1873838087 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe95403e210 0x7fe9540406d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe9740ab580 tx=0x7fe95c02e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:43:59.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.950+0000 7fe9717fa640 1 -- 172.21.15.12:0/1873838087 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe964050e50 con 0x7fe974137e50 2024-09-11T01:43:59.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.950+0000 7fe9717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:43:59.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:43:59.950+0000 7fe9717fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:44:00.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:00.207+0000 7fe97ac18640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:44:00.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:00.207+0000 7fe97ac18640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:44:00.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:00.207+0000 7fe97ac18640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:44:00.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:00.207+0000 7fe97ac18640 1 -- 172.21.15.12:0/1873838087 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fe9740a43e0 con 0x7fe974137e50 2024-09-11T01:44:00.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:00.208+0000 7fe9717fa640 1 -- 172.21.15.12:0/1873838087 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fe96401bde0 con 0x7fe974137e50 2024-09-11T01:44:00.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:00.208+0000 7fe9717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:44:00.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:00.208+0000 7fe9717fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:44:00.211 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.208+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.211 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.208+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=12,kv=11,logm=51} crc {auth=2291129928,config=2586702516,health=1273036416,kv=4243670985,logm=2399855622}) 2024-09-11T01:44:00.211 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.208+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=22,config=2,health=11,kv=11,logm=54} crc {auth=3905885279,config=2586702516,health=4184654746,kv=2563302918,logm=611832084}) 2024-09-11T01:44:00.211 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.208+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.211 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.208+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=12,kv=11,logm=51} crc {auth=2291129928,config=2586702516,health=1273036416,kv=4243670985,logm=2399855622}) 2024-09-11T01:44:00.211 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.208+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=11,kv=12,logm=51} crc {auth=513449937,config=2586702516,health=406487437,kv=3263697420,logm=438343888}) 2024-09-11T01:44:00.212 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.209+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.212 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.209+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=746139862}) 2024-09-11T01:44:00.212 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.209+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2745324463}) 2024-09-11T01:44:00.212 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.209+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.213 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.209+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=746139862}) 2024-09-11T01:44:00.213 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.209+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1972153355}) 2024-09-11T01:44:00.213 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.211+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.214 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.211+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=327914108}) 2024-09-11T01:44:00.214 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.211+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=545356728}) 2024-09-11T01:44:00.214 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.211+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.214 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.211+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=327914108}) 2024-09-11T01:44:00.214 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.211+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1948254661}) 2024-09-11T01:44:00.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:00.211+0000 7fe97ac18640 1 -- 172.21.15.12:0/1873838087 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe95403e210 msgr2=0x7fe9540406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:00.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:00.211+0000 7fe97ac18640 1 --2- 172.21.15.12:0/1873838087 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe95403e210 0x7fe9540406d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe9740ab580 tx=0x7fe95c02e000 comp rx=0 tx=0).stop 2024-09-11T01:44:00.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:00.211+0000 7fe97ac18640 10 monclient: shutdown 2024-09-11T01:44:00.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:00.211+0000 7fe97ac18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:44:00.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:00.211+0000 7fe97ac18640 1 -- 172.21.15.12:0/1873838087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe974137e50 msgr2=0x7fe97413fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:00.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:00.211+0000 7fe97ac18640 1 --2- 172.21.15.12:0/1873838087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe974137e50 0x7fe97413fae0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe96400c450 tx=0x7fe96400c910 comp rx=0 tx=0).stop 2024-09-11T01:44:00.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:00.212+0000 7fe97ac18640 1 -- 172.21.15.12:0/1873838087 shutdown_connections 2024-09-11T01:44:00.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:00.212+0000 7fe97ac18640 1 --2- 172.21.15.12:0/1873838087 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe97413d380 0x7fe9740a1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:00.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:00.212+0000 7fe97ac18640 1 --2- 172.21.15.12:0/1873838087 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe974138230 0x7fe9740a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:00.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:00.212+0000 7fe97ac18640 1 --2- 172.21.15.12:0/1873838087 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe95403e210 0x7fe9540406d0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:00.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:00.212+0000 7fe97ac18640 1 --2- 172.21.15.12:0/1873838087 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe974137e50 0x7fe97413fae0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:00.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:00.212+0000 7fe97ac18640 1 -- 172.21.15.12:0/1873838087 >> 172.21.15.12:0/1873838087 conn(0x7fe9740a7020 msgr2=0x7fe97409eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:44:00.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:00.212+0000 7fe97ac18640 1 -- 172.21.15.12:0/1873838087 shutdown_connections 2024-09-11T01:44:00.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:00.212+0000 7fe97ac18640 1 -- 172.21.15.12:0/1873838087 wait complete. 2024-09-11T01:44:00.218 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.212+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.218 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.212+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1244671300}) 2024-09-11T01:44:00.218 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.212+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1287052802}) 2024-09-11T01:44:00.218 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.212+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.218 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.212+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1244671300}) 2024-09-11T01:44:00.218 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.212+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3529765236}) 2024-09-11T01:44:00.218 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.213+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.219 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.213+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4033062548}) 2024-09-11T01:44:00.219 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.213+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3630547230}) 2024-09-11T01:44:00.219 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.213+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.219 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.213+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4033062548}) 2024-09-11T01:44:00.219 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.213+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3272369518}) 2024-09-11T01:44:00.219 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.213+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.219 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.213+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3524642574}) 2024-09-11T01:44:00.219 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.213+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1097895065}) 2024-09-11T01:44:00.219 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.213+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.219 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.214+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3524642574}) 2024-09-11T01:44:00.219 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.214+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3544864850}) 2024-09-11T01:44:00.219 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.214+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.219 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.214+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1707513018}) 2024-09-11T01:44:00.219 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.214+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2241214440}) 2024-09-11T01:44:00.220 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.214+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.220 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.214+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1707513018}) 2024-09-11T01:44:00.220 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.214+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1319051484}) 2024-09-11T01:44:00.220 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.215+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.220 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.215+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2278957530}) 2024-09-11T01:44:00.220 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.215+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2945313846}) 2024-09-11T01:44:00.220 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.215+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.220 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.215+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2278957530}) 2024-09-11T01:44:00.220 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.215+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=314869856}) 2024-09-11T01:44:00.223 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.216+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.223 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.216+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1815670565}) 2024-09-11T01:44:00.223 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.216+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2243022287}) 2024-09-11T01:44:00.223 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.216+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.223 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.216+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1815670565}) 2024-09-11T01:44:00.223 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.216+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2772199108}) 2024-09-11T01:44:00.223 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.217+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.223 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.217+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876618308}) 2024-09-11T01:44:00.223 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.217+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=957972625}) 2024-09-11T01:44:00.223 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.217+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.223 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.217+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876618308}) 2024-09-11T01:44:00.224 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.217+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4182137990}) 2024-09-11T01:44:00.224 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.218+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.224 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.218+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2381004544}) 2024-09-11T01:44:00.224 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.218+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3463415929}) 2024-09-11T01:44:00.224 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.218+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.224 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.218+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2381004544}) 2024-09-11T01:44:00.224 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.218+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2399371667}) 2024-09-11T01:44:00.224 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.219+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.224 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.219+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1146813944}) 2024-09-11T01:44:00.224 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.219+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=147787622}) 2024-09-11T01:44:00.224 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.219+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.224 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.219+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1146813944}) 2024-09-11T01:44:00.224 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.219+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2885765960}) 2024-09-11T01:44:00.225 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.220+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.225 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.220+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2952517985}) 2024-09-11T01:44:00.225 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.220+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4249411060}) 2024-09-11T01:44:00.225 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.220+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.225 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.220+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2952517985}) 2024-09-11T01:44:00.225 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.220+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2906734710}) 2024-09-11T01:44:00.228 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.221+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.228 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.221+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=158119834}) 2024-09-11T01:44:00.228 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.221+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1308660863}) 2024-09-11T01:44:00.228 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.221+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.228 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.221+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=158119834}) 2024-09-11T01:44:00.228 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.221+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=106232602}) 2024-09-11T01:44:00.228 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.222+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.229 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.222+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2714036752}) 2024-09-11T01:44:00.229 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.222+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3919410197}) 2024-09-11T01:44:00.229 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.222+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.229 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.222+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2714036752}) 2024-09-11T01:44:00.229 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.222+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3418308846}) 2024-09-11T01:44:00.229 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.223+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.229 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.223+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=36130926}) 2024-09-11T01:44:00.229 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.223+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1920883063}) 2024-09-11T01:44:00.229 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.223+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.229 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.223+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=36130926}) 2024-09-11T01:44:00.229 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.223+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3084719427}) 2024-09-11T01:44:00.229 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.224+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.229 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.224+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4169925105}) 2024-09-11T01:44:00.230 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.224+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=72174819}) 2024-09-11T01:44:00.230 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.224+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.230 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.224+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4169925105}) 2024-09-11T01:44:00.230 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.224+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4146583539}) 2024-09-11T01:44:00.230 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.224+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.230 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.224+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2861573700}) 2024-09-11T01:44:00.230 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.224+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1238698470}) 2024-09-11T01:44:00.230 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.224+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.230 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.224+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2861573700}) 2024-09-11T01:44:00.230 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.224+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3605753615}) 2024-09-11T01:44:00.230 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.225+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.230 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.225+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=971650441}) 2024-09-11T01:44:00.230 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.225+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1727483914}) 2024-09-11T01:44:00.230 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.225+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.231 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.225+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=971650441}) 2024-09-11T01:44:00.231 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.225+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3048396265}) 2024-09-11T01:44:00.231 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.226+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.231 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.226+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2340480134}) 2024-09-11T01:44:00.231 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.226+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1049528965}) 2024-09-11T01:44:00.231 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.226+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.231 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.226+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2340480134}) 2024-09-11T01:44:00.232 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.226+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1531797341}) 2024-09-11T01:44:00.232 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:44:00.399 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.397+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.400 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.397+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2406393536}) 2024-09-11T01:44:00.400 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.397+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3230541904}) 2024-09-11T01:44:00.400 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.397+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.400 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.397+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2406393536}) 2024-09-11T01:44:00.400 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.397+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1005034661}) 2024-09-11T01:44:00.400 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.398+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.401 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.398+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3211302959}) 2024-09-11T01:44:00.401 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.398+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3048018326}) 2024-09-11T01:44:00.401 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.398+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.401 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.398+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3211302959}) 2024-09-11T01:44:00.401 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.398+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=110757787}) 2024-09-11T01:44:00.406 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.403+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.406 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.403+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3920370922}) 2024-09-11T01:44:00.406 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.403+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3365632544}) 2024-09-11T01:44:00.406 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.403+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.406 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.403+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3920370922}) 2024-09-11T01:44:00.406 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.403+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4288961679}) 2024-09-11T01:44:00.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.574+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.574+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1479499411}) 2024-09-11T01:44:00.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.574+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2195780110}) 2024-09-11T01:44:00.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.574+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.574+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1479499411}) 2024-09-11T01:44:00.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.574+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1261042926}) 2024-09-11T01:44:00.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.581+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.581+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=396737442}) 2024-09-11T01:44:00.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.581+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3869612573}) 2024-09-11T01:44:00.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.581+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.581+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=396737442}) 2024-09-11T01:44:00.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.581+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2442586311}) 2024-09-11T01:44:00.585 INFO:tasks.rados.rados.0.smithi071.stdout:2279: done (2 left) 2024-09-11T01:44:00.585 INFO:tasks.rados.rados.0.smithi071.stdout:2280: delete oid 48 current snap is 251 2024-09-11T01:44:00.585 INFO:tasks.rados.rados.0.smithi071.stdout:2277: finishing write tid 1 to smithi07134777-15 2024-09-11T01:44:00.585 INFO:tasks.rados.rados.0.smithi071.stdout:2277: finishing write tid 2 to smithi07134777-15 2024-09-11T01:44:00.585 INFO:tasks.rados.rados.0.smithi071.stdout:2277: finishing write tid 3 to smithi07134777-15 2024-09-11T01:44:00.585 INFO:tasks.rados.rados.0.smithi071.stdout:2277: finishing write tid 4 to smithi07134777-15 2024-09-11T01:44:00.585 INFO:tasks.rados.rados.0.smithi071.stdout:2277: finishing write tid 5 to smithi07134777-15 2024-09-11T01:44:00.585 INFO:tasks.rados.rados.0.smithi071.stdout:2277: oid 15 updating version 0 to 1382 2024-09-11T01:44:00.585 INFO:tasks.rados.rados.0.smithi071.stdout:2277: oid 15 version 1382 is already newer than 1380 2024-09-11T01:44:00.585 INFO:tasks.rados.rados.0.smithi071.stdout:2277: oid 15 version 1382 is already newer than 1381 2024-09-11T01:44:00.586 INFO:tasks.rados.rados.0.smithi071.stdout:2277: oid 15 version 1382 is already newer than 1377 2024-09-11T01:44:00.586 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 1382 (ObjNum 825 snap 251 seq_num 825) dirty exists 2024-09-11T01:44:00.586 INFO:tasks.rados.rados.0.smithi071.stdout:2277: left oid 15 (ObjNum 825 snap 251 seq_num 825) 2024-09-11T01:44:00.600 INFO:tasks.rados.rados.0.smithi071.stdout:2276: done (2 left) 2024-09-11T01:44:00.600 INFO:tasks.rados.rados.0.smithi071.stdout:2277: done (1 left) 2024-09-11T01:44:00.600 INFO:tasks.rados.rados.0.smithi071.stdout:2280: done (0 left) 2024-09-11T01:44:00.600 INFO:tasks.rados.rados.0.smithi071.stdout:2281: snap_create 2024-09-11T01:44:00.601 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.598+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.601 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.598+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589263604}) 2024-09-11T01:44:00.601 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.599+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1816745630}) 2024-09-11T01:44:00.601 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.599+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.602 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.599+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589263604}) 2024-09-11T01:44:00.602 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.599+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3787696018}) 2024-09-11T01:44:00.617 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.615+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.617 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.615+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4146754001}) 2024-09-11T01:44:00.618 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.615+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2946139833}) 2024-09-11T01:44:00.618 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.615+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.618 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.615+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4146754001}) 2024-09-11T01:44:00.618 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.615+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=532179508}) 2024-09-11T01:44:00.619 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.616+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.619 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.616+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=898993460}) 2024-09-11T01:44:00.619 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.616+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2237625279}) 2024-09-11T01:44:00.619 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.616+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.619 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.616+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=898993460}) 2024-09-11T01:44:00.619 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.616+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=795650392}) 2024-09-11T01:44:00.620 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.617+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.620 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.617+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1742583551}) 2024-09-11T01:44:00.620 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.617+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=562083747}) 2024-09-11T01:44:00.620 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.617+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.620 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.617+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1742583551}) 2024-09-11T01:44:00.620 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.617+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=133915163}) 2024-09-11T01:44:00.623 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.617+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.623 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.617+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4129502626}) 2024-09-11T01:44:00.623 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.617+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=756568185}) 2024-09-11T01:44:00.624 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.617+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.624 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.617+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4129502626}) 2024-09-11T01:44:00.624 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.617+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2873592307}) 2024-09-11T01:44:00.624 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.618+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.624 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.618+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=110478075}) 2024-09-11T01:44:00.624 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.618+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3584946588}) 2024-09-11T01:44:00.624 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.618+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.624 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.618+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=110478075}) 2024-09-11T01:44:00.624 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.618+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3381919322}) 2024-09-11T01:44:00.624 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.619+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.624 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.619+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4229236793}) 2024-09-11T01:44:00.624 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.619+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2308209592}) 2024-09-11T01:44:00.625 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.619+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.625 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.619+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4229236793}) 2024-09-11T01:44:00.625 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.619+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2542344842}) 2024-09-11T01:44:00.625 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.619+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.625 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.619+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3702170036}) 2024-09-11T01:44:00.625 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.619+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=895563679}) 2024-09-11T01:44:00.625 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.619+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.625 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.619+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3702170036}) 2024-09-11T01:44:00.625 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.619+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1038587048}) 2024-09-11T01:44:00.625 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.620+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.625 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.620+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=431183790}) 2024-09-11T01:44:00.625 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.620+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=125318289}) 2024-09-11T01:44:00.625 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.620+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.625 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.620+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=431183790}) 2024-09-11T01:44:00.626 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.620+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3304095334}) 2024-09-11T01:44:00.626 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.621+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.626 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.621+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=453446669}) 2024-09-11T01:44:00.626 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.621+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3436931188}) 2024-09-11T01:44:00.626 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.621+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.626 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.621+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=453446669}) 2024-09-11T01:44:00.626 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.621+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=216061637}) 2024-09-11T01:44:00.732 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.729+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.732 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.729+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2662750912}) 2024-09-11T01:44:00.732 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.730+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1982334643}) 2024-09-11T01:44:00.733 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.730+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.733 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.730+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2662750912}) 2024-09-11T01:44:00.733 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.730+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1416791873}) 2024-09-11T01:44:00.741 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.738+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.741 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.738+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3515308577}) 2024-09-11T01:44:00.741 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.738+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=678001595}) 2024-09-11T01:44:00.741 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.738+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:00.741 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.738+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3515308577}) 2024-09-11T01:44:00.741 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:00.738+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3744933679}) 2024-09-11T01:44:01.067 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.064+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.067 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.064+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4101374946}) 2024-09-11T01:44:01.067 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.064+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1800895407}) 2024-09-11T01:44:01.067 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.064+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.067 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.064+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4101374946}) 2024-09-11T01:44:01.067 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.064+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1851803597}) 2024-09-11T01:44:01.068 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.065+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.068 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.065+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4029468844}) 2024-09-11T01:44:01.068 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.065+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2046995651}) 2024-09-11T01:44:01.068 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.065+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.069 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.065+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4029468844}) 2024-09-11T01:44:01.069 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.065+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2485823467}) 2024-09-11T01:44:01.070 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.066+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.070 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.066+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=463914781}) 2024-09-11T01:44:01.070 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.066+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=34360582}) 2024-09-11T01:44:01.070 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.066+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.070 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.066+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=463914781}) 2024-09-11T01:44:01.070 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.066+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3332588225}) 2024-09-11T01:44:01.070 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.067+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.070 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.067+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3697787576}) 2024-09-11T01:44:01.071 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.067+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2928857991}) 2024-09-11T01:44:01.071 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.067+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.071 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.067+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3697787576}) 2024-09-11T01:44:01.071 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.067+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=368635480}) 2024-09-11T01:44:01.073 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.068+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.073 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.068+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2379771298}) 2024-09-11T01:44:01.073 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.068+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3229211804}) 2024-09-11T01:44:01.073 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.068+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.073 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.068+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2379771298}) 2024-09-11T01:44:01.073 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.068+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1822400362}) 2024-09-11T01:44:01.073 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.069+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.074 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.069+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2762412777}) 2024-09-11T01:44:01.074 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.069+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1044696999}) 2024-09-11T01:44:01.074 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.069+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.074 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.069+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2762412777}) 2024-09-11T01:44:01.074 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.069+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4171539492}) 2024-09-11T01:44:01.074 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.070+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.074 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.070+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1315315054}) 2024-09-11T01:44:01.074 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.070+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=881187658}) 2024-09-11T01:44:01.074 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.070+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.074 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.070+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1315315054}) 2024-09-11T01:44:01.074 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.070+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2708895472}) 2024-09-11T01:44:01.074 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.070+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.074 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.070+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3311888757}) 2024-09-11T01:44:01.075 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.070+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2635976454}) 2024-09-11T01:44:01.075 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.070+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.075 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.070+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3311888757}) 2024-09-11T01:44:01.075 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.071+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2425890866}) 2024-09-11T01:44:01.078 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.071+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.078 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.071+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1767743609}) 2024-09-11T01:44:01.078 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.071+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3648126533}) 2024-09-11T01:44:01.078 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.071+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.078 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.071+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1767743609}) 2024-09-11T01:44:01.079 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.071+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1081217075}) 2024-09-11T01:44:01.079 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.072+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.079 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.072+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1004056310}) 2024-09-11T01:44:01.079 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.072+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3464323090}) 2024-09-11T01:44:01.079 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.072+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.079 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.072+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1004056310}) 2024-09-11T01:44:01.079 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.072+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1358693903}) 2024-09-11T01:44:01.079 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.072+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.079 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.072+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=633960581}) 2024-09-11T01:44:01.079 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.072+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2184902722}) 2024-09-11T01:44:01.079 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.072+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.079 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.072+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=633960581}) 2024-09-11T01:44:01.079 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.072+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=523081674}) 2024-09-11T01:44:01.080 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.073+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.080 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.073+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=15,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=34} crc {logm=4187841937,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1441348706,mon_config_key=1184109388,monmap=2933275274,nvmeofgw=598337686,osd_metadata=1404192910,osd_pg_creating=1589180998,osd_snap=1555953249}) 2024-09-11T01:44:01.080 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.074+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=14,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=34} crc {logm=2684765005,mdsmap=3760429986,mgr=4080174641,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1441348706,mon_config_key=3330371543,monmap=1324671969,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=1589180998,osd_snap=1426234889}) 2024-09-11T01:44:01.080 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.074+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.080 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.074+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=15,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=34} crc {logm=4187841937,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1441348706,mon_config_key=1184109388,monmap=2933275274,nvmeofgw=598337686,osd_metadata=1404192910,osd_pg_creating=1589180998,osd_snap=1555953249}) 2024-09-11T01:44:01.080 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.074+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=15,mdsmap=3,mgr=6,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=36} crc {logm=3729862506,mdsmap=3760429986,mgr=1816177914,mgr_metadata=3338432716,mgrstat=4258056223,mon_config_key=3330371543,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=1589180998,osd_snap=1396929418}) 2024-09-11T01:44:01.080 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.074+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.080 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.074+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=72,osdmap=28} crc {osd_snap=2132676897,osdmap=1021384583}) 2024-09-11T01:44:01.080 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.074+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=70,osdmap=30} crc {osd_snap=482683509,osdmap=383266953}) 2024-09-11T01:44:01.080 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.074+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.080 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.074+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=72,osdmap=28} crc {osd_snap=2132676897,osdmap=1021384583}) 2024-09-11T01:44:01.080 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.074+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=72,osdmap=28} crc {osd_snap=536131318,osdmap=1752626263}) 2024-09-11T01:44:01.080 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.075+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.080 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.075+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2441261997}) 2024-09-11T01:44:01.081 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.075+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=526389212}) 2024-09-11T01:44:01.081 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.075+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.081 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.075+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2441261997}) 2024-09-11T01:44:01.081 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.075+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2047955764}) 2024-09-11T01:44:01.082 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.077+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.082 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.077+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3515071276}) 2024-09-11T01:44:01.082 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.077+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=360100300}) 2024-09-11T01:44:01.082 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.077+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.082 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.077+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3515071276}) 2024-09-11T01:44:01.082 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.077+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3570803951}) 2024-09-11T01:44:01.085 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.082+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.085 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.082+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=70} crc {osdmap=3096671289}) 2024-09-11T01:44:01.086 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.082+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=72} crc {osdmap=3540647971}) 2024-09-11T01:44:01.086 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.082+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:44:01.086 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.082+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=70} crc {osdmap=3096671289}) 2024-09-11T01:44:01.086 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:44:01.082+0000 7efd15fc6640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=70} crc {osdmap=2528869997}) 2024-09-11T01:44:01.233 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:44:01.233 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:44:01.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.344+0000 7f246aa60640 10 monclient: get_monmap_and_config 2024-09-11T01:44:01.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.344+0000 7f246aa60640 10 monclient: build_initial_monmap 2024-09-11T01:44:01.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.344+0000 7f246aa60640 10 monclient: monmap: 2024-09-11T01:44:01.347 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:44:01.347 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:01.347 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:44:01.346041+0000 2024-09-11T01:44:01.347 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:44:01.346041+0000 2024-09-11T01:44:01.347 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:44:01.347 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:01.347 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:44:01.347 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:44:01.347 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:44:01.348 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:44:01.348 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:44:01.348 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:01.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.345+0000 7f246aa60640 1 Processor -- start 2024-09-11T01:44:01.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.345+0000 7f246aa60640 1 -- start start 2024-09-11T01:44:01.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.345+0000 7f246aa60640 10 monclient: init 2024-09-11T01:44:01.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.345+0000 7f246aa60640 10 monclient: _reopen_session rank -1 2024-09-11T01:44:01.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.345+0000 7f246aa60640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-11T01:44:01.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.345+0000 7f246aa60640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2464138230 addr v1:172.21.15.12:6789/0 2024-09-11T01:44:01.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.345+0000 7f246aa60640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f246409d030 0x7f246413d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:01.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.345+0000 7f246aa60640 10 monclient(hunting): picked mon.noname-a con 0x7f246409d030 addr v2:172.21.15.12:3300/0 2024-09-11T01:44:01.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.345+0000 7f246aa60640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f246409d460 0x7f246413f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:01.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.345+0000 7f246aa60640 10 monclient(hunting): picked mon.noname-c con 0x7f246409d460 addr v2:172.21.15.71:3300/0 2024-09-11T01:44:01.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.345+0000 7f246aa60640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24640a6e30 con 0x7f2464138230 2024-09-11T01:44:01.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.345+0000 7f246aa60640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:01.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.345+0000 7f246aa60640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f246409c9c0 con 0x7f246409d030 2024-09-11T01:44:01.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.345+0000 7f246aa60640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:01.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.345+0000 7f246aa60640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f246409ba20 con 0x7f246409d460 2024-09-11T01:44:01.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.345+0000 7f246aa60640 10 monclient(hunting): _renew_subs 2024-09-11T01:44:01.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.345+0000 7f246aa60640 10 monclient(hunting): authenticate will time out at 1647.788787s 2024-09-11T01:44:01.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.345+0000 7f24637fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f246409d030 0x7f246413d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:01.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.345+0000 7f2463fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2464138230 0x7f2464138610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53810/0 (socket says 172.21.15.12:53810) 2024-09-11T01:44:01.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.345+0000 7f24637fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f246409d030 0x7f246413d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35348/0 (socket says 172.21.15.12:35348) 2024-09-11T01:44:01.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.345+0000 7f2463fff640 1 -- 172.21.15.12:0/2183018807 learned_addr learned my addr 172.21.15.12:0/2183018807 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:44:01.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.345+0000 7f24637fe640 10 monclient(hunting): get_auth_request con 0x7f246409d030 auth_method 0 2024-09-11T01:44:01.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.345+0000 7f24637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:01.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.345+0000 7f24637fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:01.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.345+0000 7f24637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:01.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.346+0000 7f24637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:01.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.346+0000 7f24637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:01.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.346+0000 7f24637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:01.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.346+0000 7f2468ffd640 1 --2- 172.21.15.12:0/2183018807 >> v2:172.21.15.71:3300/0 conn(0x7f246409d460 0x7f246413f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:01.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.346+0000 7f2468ffd640 10 monclient(hunting): get_auth_request con 0x7f246409d460 auth_method 0 2024-09-11T01:44:01.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.346+0000 7f2468ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:01.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.346+0000 7f2468ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:01.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.346+0000 7f2468ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:01.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.346+0000 7f24637fe640 10 monclient(hunting): handle_auth_done global_id 94794 payload 274 2024-09-11T01:44:01.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.346+0000 7f24637fe640 1 -- 172.21.15.12:0/2183018807 >> v2:172.21.15.71:3300/0 conn(0x7f246409d460 msgr2=0x7f246413f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:44:01.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.346+0000 7f24637fe640 1 --2- 172.21.15.12:0/2183018807 >> v2:172.21.15.71:3300/0 conn(0x7f246409d460 0x7f246413f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.346+0000 7f24637fe640 1 -- 172.21.15.12:0/2183018807 >> v1:172.21.15.12:6789/0 conn(0x7f2464138230 legacy=0x7f2464138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:44:01.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.346+0000 7f24637fe640 10 monclient: _finish_hunting 0 2024-09-11T01:44:01.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.346+0000 7f24637fe640 1 monclient: found mon.noname-a 2024-09-11T01:44:01.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.346+0000 7f24637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:44:01.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.346+0000 7f24637fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:44:01.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.346+0000 7f24637fe640 1 -- 172.21.15.12:0/2183018807 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24640a66a0 con 0x7f246409d030 2024-09-11T01:44:01.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f24637fe640 1 -- 172.21.15.12:0/2183018807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f246409d030 msgr2=0x7f246413d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:01.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f24637fe640 1 --2- 172.21.15.12:0/2183018807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f246409d030 0x7f246413d090 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f246409bff0 tx=0x7f245802a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c878920fa59d50b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:01.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f2462ffd640 1 -- 172.21.15.12:0/2183018807 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2458037070 con 0x7f246409d030 2024-09-11T01:44:01.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f2462ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:01.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f2462ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:44:01.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f2462ffd640 10 monclient: dump: 2024-09-11T01:44:01.359 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:01.359 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:01.359 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.359 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.359 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:01.359 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:01.359 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:01.359 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:01.359 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:01.359 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:01.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f2462ffd640 10 monclient: _finish_auth 0 2024-09-11T01:44:01.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f2462ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:44:01.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f2462ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:44:01.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f2462ffd640 1 -- 172.21.15.12:0/2183018807 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2458032440 con 0x7f246409d030 2024-09-11T01:44:01.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f246aa60640 5 monclient: authenticate success, global_id 94794 2024-09-11T01:44:01.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f246aa60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:44:01.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f2462ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:44:01.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f246aa60640 10 monclient: get_monmap_and_config success 2024-09-11T01:44:01.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f2462ffd640 1 -- 172.21.15.12:0/2183018807 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2458032810 con 0x7f246409d030 2024-09-11T01:44:01.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f2462ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:01.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f246aa60640 4 set_mon_vals no callback set 2024-09-11T01:44:01.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f2462ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:44:01.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f2462ffd640 10 monclient: dump: 2024-09-11T01:44:01.361 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:01.361 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:01.361 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.361 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.361 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:01.361 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:01.361 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:01.361 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:01.361 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:01.361 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:01.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f246aa60640 10 monclient: shutdown 2024-09-11T01:44:01.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f246aa60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:44:01.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f246aa60640 1 -- 172.21.15.12:0/2183018807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f246409d030 msgr2=0x7f246413d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:01.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f246aa60640 1 --2- 172.21.15.12:0/2183018807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f246409d030 0x7f246413d090 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f246409bff0 tx=0x7f245802a000 comp rx=0 tx=0).stop 2024-09-11T01:44:01.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f246aa60640 1 -- 172.21.15.12:0/2183018807 shutdown_connections 2024-09-11T01:44:01.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f246aa60640 1 --2- 172.21.15.12:0/2183018807 >> v2:172.21.15.71:3300/0 conn(0x7f246409d460 0x7f246413f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f246aa60640 1 --2- 172.21.15.12:0/2183018807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f246409d030 0x7f246413d090 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f246aa60640 1 -- 172.21.15.12:0/2183018807 >> 172.21.15.12:0/2183018807 conn(0x7f24640a7120 msgr2=0x7f24640a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:44:01.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f246aa60640 1 -- 172.21.15.12:0/2183018807 shutdown_connections 2024-09-11T01:44:01.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.349+0000 7f246aa60640 1 -- 172.21.15.12:0/2183018807 wait complete. 2024-09-11T01:44:01.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.350+0000 7f246aa60640 10 monclient: build_initial_monmap 2024-09-11T01:44:01.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.350+0000 7f246aa60640 10 monclient: monmap: 2024-09-11T01:44:01.362 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:44:01.362 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:44:01.362 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:44:01.362 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:44:01.363 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:44:01.363 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:01.363 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:44:01.363 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:44:01.363 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:44:01.363 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:01.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.350+0000 7f246aa60640 1 Processor -- start 2024-09-11T01:44:01.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f246aa60640 1 -- start start 2024-09-11T01:44:01.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f246aa60640 10 monclient: init 2024-09-11T01:44:01.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f246aa60640 10 monclient: _reopen_session rank -1 2024-09-11T01:44:01.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f246aa60640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:44:01.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f246aa60640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f246409d030 0x7f24641ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:01.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f246aa60640 10 monclient(hunting): picked mon.noname-b con 0x7f246409d030 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:44:01.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f246aa60640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f246409d460 0x7f24641cd770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:01.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f246aa60640 10 monclient(hunting): picked mon.noname-a con 0x7f246409d460 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:44:01.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f246aa60640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2464138230 0x7f246413cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:01.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f246aa60640 10 monclient(hunting): picked mon.noname-c con 0x7f2464138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:44:01.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f246aa60640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:01.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f246aa60640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f246413fca0 con 0x7f246409d460 2024-09-11T01:44:01.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f246aa60640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:01.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f246aa60640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f246413fb20 con 0x7f246409d030 2024-09-11T01:44:01.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f246aa60640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:01.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f246aa60640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f246413fe20 con 0x7f2464138230 2024-09-11T01:44:01.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f246aa60640 10 monclient(hunting): _renew_subs 2024-09-11T01:44:01.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f246aa60640 10 monclient(hunting): authenticate will time out at 1647.794549s 2024-09-11T01:44:01.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f24637fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f246409d460 0x7f24641cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:01.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f2468ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2464138230 0x7f246413cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:01.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f24637fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f246409d460 0x7f24641cd770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35354/0 (socket says 172.21.15.12:35354) 2024-09-11T01:44:01.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f24637fe640 1 -- 172.21.15.12:0/3719043974 learned_addr learned my addr 172.21.15.12:0/3719043974 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:44:01.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f2468ffd640 10 monclient(hunting): get_auth_request con 0x7f2464138230 auth_method 0 2024-09-11T01:44:01.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f2468ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:01.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f2468ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:01.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f2468ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:01.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f24637fe640 10 monclient(hunting): get_auth_request con 0x7f246409d460 auth_method 0 2024-09-11T01:44:01.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f24637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:01.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f24637fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:01.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f24637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:01.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f2463fff640 1 --2- 172.21.15.12:0/3719043974 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f246409d030 0x7f24641ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:01.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f24637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:01.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f24637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:01.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f24637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f2468ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f2468ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f2468ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f2463fff640 10 monclient(hunting): get_auth_request con 0x7f246409d030 auth_method 0 2024-09-11T01:44:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f2463fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f2463fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f2463fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f2463fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f2463fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.351+0000 7f2463fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.352+0000 7f24637fe640 10 monclient(hunting): handle_auth_done global_id 94797 payload 824 2024-09-11T01:44:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.352+0000 7f24637fe640 1 -- 172.21.15.12:0/3719043974 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2464138230 msgr2=0x7f246413cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.352+0000 7f24637fe640 1 --2- 172.21.15.12:0/3719043974 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2464138230 0x7f246413cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.352+0000 7f24637fe640 1 -- 172.21.15.12:0/3719043974 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f246409d030 msgr2=0x7f24641ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.352+0000 7f24637fe640 1 --2- 172.21.15.12:0/3719043974 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f246409d030 0x7f24641ccfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.352+0000 7f24637fe640 10 monclient: _finish_hunting 0 2024-09-11T01:44:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.352+0000 7f24637fe640 1 monclient: found mon.noname-a 2024-09-11T01:44:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.352+0000 7f24637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:44:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.352+0000 7f24637fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:44:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.352+0000 7f24637fe640 1 -- 172.21.15.12:0/3719043974 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2464121460 con 0x7f246409d460 2024-09-11T01:44:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.352+0000 7f2468ffd640 1 --2- 172.21.15.12:0/3719043974 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2464138230 0x7f246413cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:44:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.354+0000 7f24637fe640 1 -- 172.21.15.12:0/3719043974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f246409d460 msgr2=0x7f24641cd770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.354+0000 7f24637fe640 1 --2- 172.21.15.12:0/3719043974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f246409d460 0x7f24641cd770 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2458009890 tx=0x7f24580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.354+0000 7f2460ff9640 1 -- 172.21.15.12:0/3719043974 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2458042020 con 0x7f246409d460 2024-09-11T01:44:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.354+0000 7f2460ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.354+0000 7f2460ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:44:01.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.354+0000 7f2460ff9640 10 monclient: dump: 2024-09-11T01:44:01.368 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:01.368 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:01.368 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.368 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.368 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:01.368 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:01.368 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:01.368 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:01.368 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:01.368 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:01.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.354+0000 7f2460ff9640 10 monclient: _finish_auth 0 2024-09-11T01:44:01.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.354+0000 7f2460ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:44:01.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.354+0000 7f2460ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:44:01.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.354+0000 7f2460ff9640 1 -- 172.21.15.12:0/3719043974 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f245803d680 con 0x7f246409d460 2024-09-11T01:44:01.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.354+0000 7f246aa60640 5 monclient: authenticate success, global_id 94797 2024-09-11T01:44:01.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.354+0000 7f246aa60640 10 monclient: _renew_subs 2024-09-11T01:44:01.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.354+0000 7f246aa60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:44:01.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.354+0000 7f246aa60640 1 -- 172.21.15.12:0/3719043974 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2464121740 con 0x7f246409d460 2024-09-11T01:44:01.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.354+0000 7f2460ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:44:01.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.354+0000 7f2460ff9640 1 -- 172.21.15.12:0/3719043974 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2458037040 con 0x7f246409d460 2024-09-11T01:44:01.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.354+0000 7f2460ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:01.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.354+0000 7f2460ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:44:01.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.354+0000 7f2460ff9640 10 monclient: dump: 2024-09-11T01:44:01.369 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:01.369 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:01.369 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.369 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.370 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:01.370 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:01.370 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:01.370 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:01.370 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:01.370 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:01.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.354+0000 7f246aa60640 10 monclient: _renew_subs 2024-09-11T01:44:01.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.354+0000 7f24627fc640 4 set_mon_vals no callback set 2024-09-11T01:44:01.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.354+0000 7f246aa60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:44:01.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.354+0000 7f246aa60640 1 -- 172.21.15.12:0/3719043974 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2464121180 con 0x7f246409d460 2024-09-11T01:44:01.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.355+0000 7f2460ff9640 1 -- 172.21.15.12:0/3719043974 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f245803da70 con 0x7f246409d460 2024-09-11T01:44:01.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.355+0000 7f246aa60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:44:01.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.355+0000 7f246aa60640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:44:01.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.355+0000 7f246aa60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:44:01.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.355+0000 7f246aa60640 1 -- 172.21.15.12:0/3719043974 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2464138610 con 0x7f246409d460 2024-09-11T01:44:01.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.355+0000 7f2460ff9640 1 --2- 172.21.15.12:0/3719043974 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f245003e210 0x7f24500406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:01.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.355+0000 7f2460ff9640 1 -- 172.21.15.12:0/3719043974 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(509..509 src has 317..509) ==== 4847+0+0 (secure 0 0 0) 0x7f2458073860 con 0x7f246409d460 2024-09-11T01:44:01.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.356+0000 7f2463fff640 1 --2- 172.21.15.12:0/3719043974 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f245003e210 0x7f24500406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:01.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.356+0000 7f2463fff640 10 monclient: get_auth_request con 0x7f245003e210 auth_method 0 2024-09-11T01:44:01.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.358+0000 7f2463fff640 1 --2- 172.21.15.12:0/3719043974 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f245003e210 0x7f24500406d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2454007c90 tx=0x7f2454007be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:01.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.360+0000 7f2460ff9640 1 -- 172.21.15.12:0/3719043974 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f245804c030 con 0x7f246409d460 2024-09-11T01:44:01.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.360+0000 7f2460ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:44:01.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.360+0000 7f2460ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:44:01.586 INFO:tasks.rados.rados.0.smithi071.stdout:2281: done (0 left) 2024-09-11T01:44:01.587 INFO:tasks.rados.rados.0.smithi071.stdout:2282: read oid 25 snap -1 2024-09-11T01:44:01.587 INFO:tasks.rados.rados.0.smithi071.stdout:2282: expect deleted 2024-09-11T01:44:01.587 INFO:tasks.rados.rados.0.smithi071.stdout:2283: read oid 41 snap -1 2024-09-11T01:44:01.587 INFO:tasks.rados.rados.0.smithi071.stdout:2283: expect (ObjNum 753 snap 229 seq_num 753) 2024-09-11T01:44:01.587 INFO:tasks.rados.rados.0.smithi071.stdout:2284: delete oid 7 current snap is 252 2024-09-11T01:44:01.592 INFO:tasks.rados.rados.0.smithi071.stdout:2283: expect (ObjNum 753 snap 229 seq_num 753) 2024-09-11T01:44:01.620 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 80335 2024-09-11T01:44:01.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.618+0000 7f246aa60640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:44:01.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.618+0000 7f246aa60640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:44:01.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.618+0000 7f246aa60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:44:01.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.618+0000 7f246aa60640 1 -- 172.21.15.12:0/3719043974 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f246413ec50 con 0x7f246409d460 2024-09-11T01:44:01.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.618+0000 7f2460ff9640 1 -- 172.21.15.12:0/3719043974 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f245803b370 con 0x7f246409d460 2024-09-11T01:44:01.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.618+0000 7f2460ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:44:01.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.618+0000 7f2460ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:44:01.621 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:44:01.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.620+0000 7f246aa60640 1 -- 172.21.15.12:0/3719043974 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f245003e210 msgr2=0x7f24500406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:01.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.620+0000 7f246aa60640 1 --2- 172.21.15.12:0/3719043974 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f245003e210 0x7f24500406d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2454007c90 tx=0x7f2454007be0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.620+0000 7f246aa60640 10 monclient: shutdown 2024-09-11T01:44:01.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.620+0000 7f246aa60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:44:01.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.620+0000 7f246aa60640 1 -- 172.21.15.12:0/3719043974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f246409d460 msgr2=0x7f24641cd770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:01.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.620+0000 7f246aa60640 1 --2- 172.21.15.12:0/3719043974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f246409d460 0x7f24641cd770 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2458009890 tx=0x7f24580046c0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.620+0000 7f246aa60640 1 -- 172.21.15.12:0/3719043974 shutdown_connections 2024-09-11T01:44:01.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.620+0000 7f246aa60640 1 --2- 172.21.15.12:0/3719043974 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2464138230 0x7f246413cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.620+0000 7f246aa60640 1 --2- 172.21.15.12:0/3719043974 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f245003e210 0x7f24500406d0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.620+0000 7f246aa60640 1 --2- 172.21.15.12:0/3719043974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f246409d460 0x7f24641cd770 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.620+0000 7f246aa60640 1 --2- 172.21.15.12:0/3719043974 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f246409d030 0x7f24641ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.620+0000 7f246aa60640 1 -- 172.21.15.12:0/3719043974 >> 172.21.15.12:0/3719043974 conn(0x7f24640a7120 msgr2=0x7f24640a8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:44:01.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.621+0000 7f246aa60640 1 -- 172.21.15.12:0/3719043974 shutdown_connections 2024-09-11T01:44:01.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.621+0000 7f246aa60640 1 -- 172.21.15.12:0/3719043974 wait complete. 2024-09-11T01:44:01.633 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:44:01.633 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:44:01.634 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:44:01.634 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:44:01.692 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2651637 2024-09-11T01:44:01.692 INFO:tasks.rados.rados.0.smithi071.stdout:2282: done (2 left) 2024-09-11T01:44:01.692 INFO:tasks.rados.rados.0.smithi071.stdout:2283: done (1 left) 2024-09-11T01:44:01.693 INFO:tasks.rados.rados.0.smithi071.stdout:2284: done (0 left) 2024-09-11T01:44:01.693 INFO:tasks.rados.rados.0.smithi071.stdout:2285: copy_from oid 35 from oid 11 current snap is 252 2024-09-11T01:44:01.693 INFO:tasks.rados.rados.0.smithi071.stdout:2286: rollback oid 39 current snap is 252 2024-09-11T01:44:01.693 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 39 to 227 2024-09-11T01:44:01.693 INFO:tasks.rados.rados.0.smithi071.stdout:2287: write (excl) oid 49 current snap is 252 2024-09-11T01:44:01.693 INFO:tasks.rados.rados.0.smithi071.stdout:2287: seq_num 828 ranges {716196=472296,1769561=646611,3121885=793109} 2024-09-11T01:44:01.698 INFO:tasks.rados.rados.0.smithi071.stdout:2287: writing smithi07134777-49 from 716196 to 1188492 tid 1 2024-09-11T01:44:01.704 INFO:tasks.rados.rados.0.smithi071.stdout:2287: writing smithi07134777-49 from 1769561 to 2416172 tid 2 2024-09-11T01:44:01.712 INFO:tasks.rados.rados.0.smithi071.stdout:2287: writing smithi07134777-49 from 3121885 to 3914994 tid 3 2024-09-11T01:44:01.713 INFO:tasks.rados.rados.0.smithi071.stdout:2288: read oid 42 snap -1 2024-09-11T01:44:01.713 INFO:tasks.rados.rados.0.smithi071.stdout:2288: expect (ObjNum 806 snap 244 seq_num 806) 2024-09-11T01:44:01.713 INFO:tasks.rados.rados.0.smithi071.stdout:2289: write (excl) oid 36 current snap is 252 2024-09-11T01:44:01.713 INFO:tasks.rados.rados.0.smithi071.stdout:2289: seq_num 829 ranges {733003=663898,1944396=506622,2865661=604381} 2024-09-11T01:44:01.720 INFO:tasks.rados.rados.0.smithi071.stdout:2289: writing smithi07134777-36 from 733003 to 1396901 tid 1 2024-09-11T01:44:01.726 INFO:tasks.rados.rados.0.smithi071.stdout:2289: writing smithi07134777-36 from 1944396 to 2451018 tid 2 2024-09-11T01:44:01.732 INFO:tasks.rados.rados.0.smithi071.stdout:2289: writing smithi07134777-36 from 2865661 to 3470042 tid 3 2024-09-11T01:44:01.732 INFO:tasks.rados.rados.0.smithi071.stdout:2285: finishing copy_from to smithi07134777-35 2024-09-11T01:44:01.732 INFO:tasks.rados.rados.0.smithi071.stdout:2285: got expected ENOENT (src dne) 2024-09-11T01:44:01.732 INFO:tasks.rados.rados.0.smithi071.stdout:2290: read oid 33 snap -1 2024-09-11T01:44:01.732 INFO:tasks.rados.rados.0.smithi071.stdout:2290: expect (ObjNum 816 snap 250 seq_num 816) 2024-09-11T01:44:01.732 INFO:tasks.rados.rados.0.smithi071.stdout:2291: read oid 15 snap -1 2024-09-11T01:44:01.732 INFO:tasks.rados.rados.0.smithi071.stdout:2291: expect (ObjNum 825 snap 251 seq_num 825) 2024-09-11T01:44:01.733 INFO:tasks.rados.rados.0.smithi071.stdout:2292: read oid 10 snap -1 2024-09-11T01:44:01.733 INFO:tasks.rados.rados.0.smithi071.stdout:2292: expect deleted 2024-09-11T01:44:01.733 INFO:tasks.rados.rados.0.smithi071.stdout:2293: delete oid 40 current snap is 252 2024-09-11T01:44:01.733 INFO:tasks.rados.rados.0.smithi071.stdout:2285: finishing copy_from racing read to smithi07134777-35 2024-09-11T01:44:01.733 INFO:tasks.rados.rados.0.smithi071.stdout:2286: finishing rollback tid 0 to smithi07134777-39 2024-09-11T01:44:01.733 INFO:tasks.rados.rados.0.smithi071.stdout:2286: finishing rollback tid 1 to smithi07134777-39 2024-09-11T01:44:01.734 INFO:tasks.rados.rados.0.smithi071.stdout:2286: finishing rollback tid 2 to smithi07134777-39 2024-09-11T01:44:01.734 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 1010 (ObjNum 701 snap 211 seq_num 701) dirty exists 2024-09-11T01:44:01.734 INFO:tasks.rados.rados.0.smithi071.stdout:2287: finishing write tid 1 to smithi07134777-49 2024-09-11T01:44:01.734 INFO:tasks.rados.rados.0.smithi071.stdout:2287: finishing write tid 2 to smithi07134777-49 2024-09-11T01:44:01.734 INFO:tasks.rados.rados.0.smithi071.stdout:2288: expect (ObjNum 806 snap 244 seq_num 806) 2024-09-11T01:44:01.734 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:44:01.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.746+0000 7f4653c08640 10 monclient: get_monmap_and_config 2024-09-11T01:44:01.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.746+0000 7f4653c08640 10 monclient: build_initial_monmap 2024-09-11T01:44:01.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.747+0000 7f4653c08640 10 monclient: monmap: 2024-09-11T01:44:01.749 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:44:01.749 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:01.749 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:44:01.748725+0000 2024-09-11T01:44:01.749 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:44:01.748725+0000 2024-09-11T01:44:01.750 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:44:01.750 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:01.750 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:44:01.750 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:44:01.750 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:44:01.750 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:44:01.750 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:44:01.750 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:01.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.747+0000 7f4653c08640 1 Processor -- start 2024-09-11T01:44:01.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.747+0000 7f4653c08640 1 -- start start 2024-09-11T01:44:01.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.747+0000 7f4653c08640 10 monclient: init 2024-09-11T01:44:01.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f4653c08640 10 monclient: _reopen_session rank -1 2024-09-11T01:44:01.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f4653c08640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-11T01:44:01.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f4653c08640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f464c06f540 0x7f464c0a5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:01.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f4653c08640 10 monclient(hunting): picked mon.noname-a con 0x7f464c06f540 addr v2:172.21.15.12:3300/0 2024-09-11T01:44:01.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f4653c08640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f464c09cff0 0x7f464c0a5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:01.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f4653c08640 10 monclient(hunting): picked mon.noname-c con 0x7f464c09cff0 addr v2:172.21.15.71:3300/0 2024-09-11T01:44:01.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f4653c08640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f464c0a5aa0 0x7f464c137de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:01.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f4653c08640 10 monclient(hunting): picked mon.noname-b con 0x7f464c0a5aa0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:44:01.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f4653c08640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:01.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f4653c08640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f464c0a6230 con 0x7f464c06f540 2024-09-11T01:44:01.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f4653c08640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:01.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f4653c08640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f464c06f920 con 0x7f464c09cff0 2024-09-11T01:44:01.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f4653c08640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:01.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f4653c08640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f464c0a6ea0 con 0x7f464c0a5aa0 2024-09-11T01:44:01.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f4653c08640 10 monclient(hunting): _renew_subs 2024-09-11T01:44:01.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f4653c08640 10 monclient(hunting): authenticate will time out at 1648.191440s 2024-09-11T01:44:01.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46519a4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f464c06f540 0x7f464c0a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:01.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46521a5640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f464c0a5aa0 0x7f464c137de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:01.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46519a4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f464c06f540 0x7f464c0a5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35370/0 (socket says 172.21.15.12:35370) 2024-09-11T01:44:01.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46519a4640 1 -- 172.21.15.12:0/4098335779 learned_addr learned my addr 172.21.15.12:0/4098335779 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:44:01.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46519a4640 10 monclient(hunting): get_auth_request con 0x7f464c06f540 auth_method 0 2024-09-11T01:44:01.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46519a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:01.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46519a4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:01.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46519a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:01.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46521a5640 10 monclient(hunting): get_auth_request con 0x7f464c0a5aa0 auth_method 0 2024-09-11T01:44:01.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46521a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:01.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46521a5640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:01.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46521a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:01.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46511a3640 1 --2- 172.21.15.12:0/4098335779 >> v2:172.21.15.71:3300/0 conn(0x7f464c09cff0 0x7f464c0a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:01.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46519a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:01.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46519a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:01.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46519a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:01.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46521a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:01.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46521a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:01.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46521a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:01.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46511a3640 10 monclient(hunting): get_auth_request con 0x7f464c09cff0 auth_method 0 2024-09-11T01:44:01.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46511a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:01.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46511a3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:01.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46511a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:01.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46511a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:01.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46511a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:01.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46511a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:01.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46519a4640 10 monclient(hunting): handle_auth_done global_id 94800 payload 274 2024-09-11T01:44:01.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46519a4640 1 -- 172.21.15.12:0/4098335779 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f464c0a5aa0 msgr2=0x7f464c137de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:44:01.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46519a4640 1 --2- 172.21.15.12:0/4098335779 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f464c0a5aa0 0x7f464c137de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46519a4640 1 -- 172.21.15.12:0/4098335779 >> v2:172.21.15.71:3300/0 conn(0x7f464c09cff0 msgr2=0x7f464c0a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:44:01.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46519a4640 1 --2- 172.21.15.12:0/4098335779 >> v2:172.21.15.71:3300/0 conn(0x7f464c09cff0 0x7f464c0a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46519a4640 10 monclient: _finish_hunting 0 2024-09-11T01:44:01.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46519a4640 1 monclient: found mon.noname-a 2024-09-11T01:44:01.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46519a4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:44:01.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46519a4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:44:01.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46519a4640 1 -- 172.21.15.12:0/4098335779 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f464c09d4d0 con 0x7f464c06f540 2024-09-11T01:44:01.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.748+0000 7f46521a5640 1 --2- 172.21.15.12:0/4098335779 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f464c0a5aa0 0x7f464c137de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:44:01.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.751+0000 7f46519a4640 1 -- 172.21.15.12:0/4098335779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c06f540 msgr2=0x7f464c0a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:01.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.751+0000 7f46519a4640 1 --2- 172.21.15.12:0/4098335779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c06f540 0x7f464c0a5020 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f464c09ca80 tx=0x7f464802a0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a681e6169a7169c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:01.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.751+0000 7f46509a2640 1 -- 172.21.15.12:0/4098335779 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4648037070 con 0x7f464c06f540 2024-09-11T01:44:01.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.751+0000 7f46509a2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:01.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.751+0000 7f46509a2640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:44:01.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.751+0000 7f46509a2640 10 monclient: dump: 2024-09-11T01:44:01.760 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:01.760 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:01.760 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.760 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.760 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:01.761 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:01.761 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:01.761 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:01.761 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:01.761 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:01.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.751+0000 7f46509a2640 10 monclient: _finish_auth 0 2024-09-11T01:44:01.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.751+0000 7f46509a2640 10 monclient: _check_auth_tickets 2024-09-11T01:44:01.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.751+0000 7f46509a2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:44:01.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.751+0000 7f46509a2640 1 -- 172.21.15.12:0/4098335779 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4648032640 con 0x7f464c06f540 2024-09-11T01:44:01.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.751+0000 7f4653c08640 5 monclient: authenticate success, global_id 94800 2024-09-11T01:44:01.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.751+0000 7f4653c08640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:44:01.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.751+0000 7f46509a2640 10 monclient: handle_config config(0 keys) 2024-09-11T01:44:01.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.751+0000 7f4653c08640 10 monclient: get_monmap_and_config success 2024-09-11T01:44:01.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.751+0000 7f4653c08640 4 set_mon_vals no callback set 2024-09-11T01:44:01.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.751+0000 7f46509a2640 1 -- 172.21.15.12:0/4098335779 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4648032a10 con 0x7f464c06f540 2024-09-11T01:44:01.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.751+0000 7f46509a2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:01.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.751+0000 7f46509a2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:44:01.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.751+0000 7f46509a2640 10 monclient: dump: 2024-09-11T01:44:01.762 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:01.762 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:01.762 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.762 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.762 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:01.762 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:01.762 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:01.762 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:01.762 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:01.763 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:01.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.752+0000 7f4653c08640 10 monclient: shutdown 2024-09-11T01:44:01.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.752+0000 7f4653c08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:44:01.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.752+0000 7f4653c08640 1 -- 172.21.15.12:0/4098335779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c06f540 msgr2=0x7f464c0a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:01.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.752+0000 7f4653c08640 1 --2- 172.21.15.12:0/4098335779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c06f540 0x7f464c0a5020 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f464c09ca80 tx=0x7f464802a0f0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.752+0000 7f4653c08640 1 -- 172.21.15.12:0/4098335779 shutdown_connections 2024-09-11T01:44:01.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.752+0000 7f4653c08640 1 --2- 172.21.15.12:0/4098335779 >> v2:172.21.15.71:3300/0 conn(0x7f464c09cff0 0x7f464c0a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.752+0000 7f4653c08640 1 --2- 172.21.15.12:0/4098335779 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f464c0a5aa0 0x7f464c137de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.752+0000 7f4653c08640 1 --2- 172.21.15.12:0/4098335779 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c06f540 0x7f464c0a5020 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.752+0000 7f4653c08640 1 -- 172.21.15.12:0/4098335779 >> 172.21.15.12:0/4098335779 conn(0x7f464c0a7300 msgr2=0x7f464c0a7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:44:01.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.752+0000 7f4653c08640 1 -- 172.21.15.12:0/4098335779 shutdown_connections 2024-09-11T01:44:01.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.752+0000 7f4653c08640 1 -- 172.21.15.12:0/4098335779 wait complete. 2024-09-11T01:44:01.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.753+0000 7f4653c08640 10 monclient: build_initial_monmap 2024-09-11T01:44:01.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.753+0000 7f4653c08640 10 monclient: monmap: 2024-09-11T01:44:01.764 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:44:01.764 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:44:01.764 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:44:01.764 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:44:01.764 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:44:01.764 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:01.764 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:44:01.764 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:44:01.764 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:44:01.764 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:01.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.753+0000 7f4653c08640 1 Processor -- start 2024-09-11T01:44:01.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.753+0000 7f4653c08640 1 -- start start 2024-09-11T01:44:01.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.753+0000 7f4653c08640 10 monclient: init 2024-09-11T01:44:01.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.753+0000 7f4653c08640 10 monclient: _reopen_session rank -1 2024-09-11T01:44:01.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.753+0000 7f4653c08640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:44:01.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.753+0000 7f4653c08640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c06f540 0x7f464c1cd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:01.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.753+0000 7f4653c08640 10 monclient(hunting): picked mon.noname-a con 0x7f464c06f540 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:44:01.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.753+0000 7f4653c08640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f464c09cff0 0x7f464c1cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:01.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f4653c08640 10 monclient(hunting): picked mon.noname-b con 0x7f464c09cff0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:44:01.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f4653c08640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f464c0a5aa0 0x7f464c1249d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:01.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f4653c08640 10 monclient(hunting): picked mon.noname-c con 0x7f464c0a5aa0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:44:01.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f4653c08640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:01.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f4653c08640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f464c138320 con 0x7f464c06f540 2024-09-11T01:44:01.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f4653c08640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:01.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f4653c08640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f464c1384a0 con 0x7f464c09cff0 2024-09-11T01:44:01.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f4653c08640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:01.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f4653c08640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f464c138620 con 0x7f464c0a5aa0 2024-09-11T01:44:01.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f4653c08640 10 monclient(hunting): _renew_subs 2024-09-11T01:44:01.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f4653c08640 10 monclient(hunting): authenticate will time out at 1648.197053s 2024-09-11T01:44:01.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46519a4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c06f540 0x7f464c1cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:01.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46521a5640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f464c0a5aa0 0x7f464c1249d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:01.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46519a4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c06f540 0x7f464c1cd280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35382/0 (socket says 172.21.15.12:35382) 2024-09-11T01:44:01.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46519a4640 1 -- 172.21.15.12:0/1572615596 learned_addr learned my addr 172.21.15.12:0/1572615596 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:44:01.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46519a4640 10 monclient(hunting): get_auth_request con 0x7f464c06f540 auth_method 0 2024-09-11T01:44:01.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46519a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:01.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46519a4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:01.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46519a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:01.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46521a5640 10 monclient(hunting): get_auth_request con 0x7f464c0a5aa0 auth_method 0 2024-09-11T01:44:01.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46521a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:01.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46521a5640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:01.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46521a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:01.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46511a3640 1 --2- 172.21.15.12:0/1572615596 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f464c09cff0 0x7f464c1cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:01.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46519a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:01.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46519a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:01.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46519a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:01.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46511a3640 10 monclient(hunting): get_auth_request con 0x7f464c09cff0 auth_method 0 2024-09-11T01:44:01.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46511a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:01.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46511a3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:01.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46511a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:01.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46521a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:01.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46521a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:01.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46521a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:01.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46511a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:01.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46511a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:01.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46511a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:01.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46519a4640 10 monclient(hunting): handle_auth_done global_id 94803 payload 824 2024-09-11T01:44:01.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46519a4640 1 -- 172.21.15.12:0/1572615596 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f464c0a5aa0 msgr2=0x7f464c1249d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:01.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46519a4640 1 --2- 172.21.15.12:0/1572615596 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f464c0a5aa0 0x7f464c1249d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46519a4640 1 -- 172.21.15.12:0/1572615596 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f464c09cff0 msgr2=0x7f464c1cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:01.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46519a4640 1 --2- 172.21.15.12:0/1572615596 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f464c09cff0 0x7f464c1cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46519a4640 10 monclient: _finish_hunting 0 2024-09-11T01:44:01.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46519a4640 1 monclient: found mon.noname-a 2024-09-11T01:44:01.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46519a4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:44:01.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46519a4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:44:01.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.754+0000 7f46519a4640 1 -- 172.21.15.12:0/1572615596 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f464c0a1810 con 0x7f464c06f540 2024-09-11T01:44:01.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.756+0000 7f46519a4640 1 -- 172.21.15.12:0/1572615596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c06f540 msgr2=0x7f464c1cd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:01.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.756+0000 7f46519a4640 1 --2- 172.21.15.12:0/1572615596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c06f540 0x7f464c1cd280 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4648000c00 tx=0x7f46480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:01.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.756+0000 7f46427fc640 1 -- 172.21.15.12:0/1572615596 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4648041020 con 0x7f464c06f540 2024-09-11T01:44:01.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.756+0000 7f46427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:01.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.756+0000 7f46427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:44:01.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.756+0000 7f46427fc640 10 monclient: dump: 2024-09-11T01:44:01.775 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:01.775 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:01.775 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.775 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.775 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:01.775 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:01.775 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:01.775 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:01.775 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:01.775 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:01.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.756+0000 7f46427fc640 10 monclient: _finish_auth 0 2024-09-11T01:44:01.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.756+0000 7f46427fc640 10 monclient: _check_auth_tickets 2024-09-11T01:44:01.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.756+0000 7f46427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:44:01.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.756+0000 7f46427fc640 1 -- 172.21.15.12:0/1572615596 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f464803d630 con 0x7f464c06f540 2024-09-11T01:44:01.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.756+0000 7f4653c08640 5 monclient: authenticate success, global_id 94803 2024-09-11T01:44:01.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.756+0000 7f4653c08640 10 monclient: _renew_subs 2024-09-11T01:44:01.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.756+0000 7f4653c08640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:44:01.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.756+0000 7f4653c08640 1 -- 172.21.15.12:0/1572615596 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f464c126450 con 0x7f464c06f540 2024-09-11T01:44:01.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.756+0000 7f46427fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:44:01.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.756+0000 7f46427fc640 1 -- 172.21.15.12:0/1572615596 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4648037040 con 0x7f464c06f540 2024-09-11T01:44:01.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.756+0000 7f46427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:01.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.756+0000 7f46427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:44:01.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.756+0000 7f46427fc640 10 monclient: dump: 2024-09-11T01:44:01.776 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:01.777 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:01.777 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.777 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.777 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:01.777 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:01.777 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:01.777 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:01.777 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:01.777 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:01.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.756+0000 7f4653c08640 10 monclient: _renew_subs 2024-09-11T01:44:01.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.756+0000 7f4643fff640 4 set_mon_vals no callback set 2024-09-11T01:44:01.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.756+0000 7f4653c08640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:44:01.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.756+0000 7f4653c08640 1 -- 172.21.15.12:0/1572615596 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f464c137dd0 con 0x7f464c06f540 2024-09-11T01:44:01.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.757+0000 7f4653c08640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:44:01.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.757+0000 7f4653c08640 1 --2- 172.21.15.12:0/1572615596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c126fb0 0x7f464c124f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:01.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.757+0000 7f4653c08640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:01.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.757+0000 7f4653c08640 1 -- 172.21.15.12:0/1572615596 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f464c09c3e0 con 0x7f464c126fb0 2024-09-11T01:44:01.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.757+0000 7f46427fc640 1 -- 172.21.15.12:0/1572615596 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f464803a030 con 0x7f464c06f540 2024-09-11T01:44:01.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.757+0000 7f46511a3640 1 --2- 172.21.15.12:0/1572615596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c126fb0 0x7f464c124f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:01.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.757+0000 7f46511a3640 10 monclient: get_auth_request con 0x7f464c126fb0 auth_method 0 2024-09-11T01:44:01.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.757+0000 7f46511a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:01.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.757+0000 7f46511a3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:01.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.757+0000 7f46511a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:01.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.757+0000 7f46511a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:01.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.757+0000 7f46511a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:01.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.757+0000 7f46511a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:01.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.758+0000 7f46511a3640 10 monclient(hunting): handle_auth_done global_id 94806 payload 824 2024-09-11T01:44:01.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.758+0000 7f46511a3640 1 -- 172.21.15.12:0/1572615596 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f464c1285f0 con 0x7f464c126fb0 2024-09-11T01:44:01.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.758+0000 7f46427fc640 1 --2- 172.21.15.12:0/1572615596 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f463003e260 0x7f4630040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:01.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.758+0000 7f46427fc640 1 -- 172.21.15.12:0/1572615596 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(510..510 src has 317..510) ==== 4819+0+0 (secure 0 0 0) 0x7f46480730e0 con 0x7f464c06f540 2024-09-11T01:44:01.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.758+0000 7f46521a5640 1 --2- 172.21.15.12:0/1572615596 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f463003e260 0x7f4630040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:01.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.758+0000 7f46521a5640 10 monclient: get_auth_request con 0x7f463003e260 auth_method 0 2024-09-11T01:44:01.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.760+0000 7f46511a3640 1 -- 172.21.15.12:0/1572615596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c126fb0 msgr2=0x7f464c124f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:01.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.760+0000 7f46511a3640 1 --2- 172.21.15.12:0/1572615596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c126fb0 0x7f464c124f10 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f463c0118e0 tx=0x7f463c011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:01.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.760+0000 7f46427fc640 1 -- 172.21.15.12:0/1572615596 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463c019070 con 0x7f464c126fb0 2024-09-11T01:44:01.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.760+0000 7f46427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:01.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.760+0000 7f46427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:44:01.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.760+0000 7f46427fc640 10 monclient: dump: 2024-09-11T01:44:01.779 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:01.779 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:01.779 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:01.780 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.760+0000 7f46427fc640 1 -- 172.21.15.12:0/1572615596 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f463c00f950 con 0x7f464c126fb0 2024-09-11T01:44:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.760+0000 7f46427fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:44:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.760+0000 7f46427fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:44:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.760+0000 7f46427fc640 1 -- 172.21.15.12:0/1572615596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c126fb0 msgr2=0x7f464c124f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.760+0000 7f46427fc640 1 --2- 172.21.15.12:0/1572615596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c126fb0 0x7f464c124f10 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f463c0118e0 tx=0x7f463c011da0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.760+0000 7f46427fc640 1 -- 172.21.15.12:0/1572615596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c126fb0 msgr2=0x7f464c124f10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:44:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.760+0000 7f46427fc640 1 --2- 172.21.15.12:0/1572615596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c126fb0 0x7f464c124f10 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.760+0000 7f46521a5640 1 --2- 172.21.15.12:0/1572615596 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f463003e260 0x7f4630040720 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f464c129290 tx=0x7f464400b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.766+0000 7f4653c08640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:44:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.766+0000 7f4653c08640 1 --2- 172.21.15.12:0/1572615596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c125450 0x7f464c125860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.766+0000 7f4653c08640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.766+0000 7f4653c08640 1 -- 172.21.15.12:0/1572615596 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f464c125da0 con 0x7f464c125450 2024-09-11T01:44:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.767+0000 7f46511a3640 1 --2- 172.21.15.12:0/1572615596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c125450 0x7f464c125860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.767+0000 7f46511a3640 10 monclient: get_auth_request con 0x7f464c125450 auth_method 0 2024-09-11T01:44:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.767+0000 7f46511a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.767+0000 7f46511a3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.767+0000 7f46511a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.767+0000 7f46511a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.767+0000 7f46511a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:01.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.767+0000 7f46511a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:01.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.767+0000 7f46511a3640 10 monclient(hunting): handle_auth_done global_id 94809 payload 824 2024-09-11T01:44:01.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.767+0000 7f46511a3640 1 -- 172.21.15.12:0/1572615596 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f464c138a30 con 0x7f464c125450 2024-09-11T01:44:01.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.769+0000 7f46511a3640 1 -- 172.21.15.12:0/1572615596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c125450 msgr2=0x7f464c125860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:01.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.769+0000 7f46511a3640 1 --2- 172.21.15.12:0/1572615596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c125450 0x7f464c125860 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f463c01b000 tx=0x7f463c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:01.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.769+0000 7f46427fc640 1 -- 172.21.15.12:0/1572615596 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463c019040 con 0x7f464c125450 2024-09-11T01:44:01.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.769+0000 7f46427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:01.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.769+0000 7f46427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:44:01.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.769+0000 7f46427fc640 10 monclient: dump: 2024-09-11T01:44:01.783 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:01.783 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:01.783 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.783 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.784 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:01.784 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:01.784 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:01.784 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:01.784 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:01.784 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:01.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.769+0000 7f46427fc640 1 -- 172.21.15.12:0/1572615596 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f463c01e660 con 0x7f464c125450 2024-09-11T01:44:01.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.769+0000 7f46427fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:44:01.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.769+0000 7f46427fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:44:01.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.769+0000 7f46427fc640 1 -- 172.21.15.12:0/1572615596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c125450 msgr2=0x7f464c125860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:01.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.769+0000 7f46427fc640 1 --2- 172.21.15.12:0/1572615596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c125450 0x7f464c125860 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f463c01b000 tx=0x7f463c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.769+0000 7f46427fc640 1 -- 172.21.15.12:0/1572615596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c125450 msgr2=0x7f464c125860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:44:01.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.769+0000 7f46427fc640 1 --2- 172.21.15.12:0/1572615596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c125450 0x7f464c125860 secure :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f463c01b000 tx=0x7f463c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.785 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:44:01.785 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:44:01.785 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:44:01.785 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:44:01.785 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 138, 2024-09-11T01:44:01.785 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 387865, 2024-09-11T01:44:01.785 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:44:01.786 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:44:01.786 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:44:01.786 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:44:01.786 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:01.786 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 6, 2024-09-11T01:44:01.786 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:44:01.786 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:44:01.786 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:44:01.786 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:44:01.786 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:44:01.786 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:44:01.786 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:44:01.786 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:44:01.787 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:44:01.787 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:44:01.787 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:44:01.787 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:44:01.787 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:44:01.787 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:44:01.787 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:01.787 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:44:01.787 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:44:01.787 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:44:01.787 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:44:01.787 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:44:01.787 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:44:01.787 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:44:01.788 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:44:01.788 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:44:01.788 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:44:01.788 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:44:01.788 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:44:01.788 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:44:01.788 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:01.788 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:01.788 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:44:01.788 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:44:01.788 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:44:01.788 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:44:01.788 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:44:01.788 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:44:01.789 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:44:01.789 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:44:01.789 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:44:01.789 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:44:01.789 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:44:01.789 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:44:01.789 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:44:01.789 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:44:01.789 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:44:01.789 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:44:01.789 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:44:01.789 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:44:01.789 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:44:01.790 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:44:01.790 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:44:01.790 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:44:01.790 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:44:01.790 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:44:01.790 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:44:01.790 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:44:01.790 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:44:01.790 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:44:01.790 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:01.790 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:44:01.790 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:01.790 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:44:01.790 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:01.791 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:44:01.791 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:44:01.791 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:44:01.791 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:44:01.791 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:01.791 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:44:01.791 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:44:01.791 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:01.791 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:01.791 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:01.791 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:44:01.791 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:44:01.791 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:01.791 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:01.792 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:01.792 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:01.792 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:44:01.792 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:44:01.792 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:44:01.792 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:44:01.792 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:44:01.792 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:01.792 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:01.792 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:44:01.792 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:44:01.792 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:44:01.792 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:44:01.792 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:01.793 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:44:01.793 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:44:01.793 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:01.793 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:01.796 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:01.796 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:44:01.796 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:44:01.796 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:01.796 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:01.796 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:01.796 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:01.796 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:44:01.796 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:44:01.797 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:44:01.797 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:44:01.797 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:44:01.797 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:01.797 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:01.797 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:44:01.797 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:44:01.797 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:44:01.797 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:44:01.797 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:01.797 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:44:01.797 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:44:01.797 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:01.798 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:01.798 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:01.798 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:44:01.798 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:44:01.798 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:01.798 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:01.798 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:01.798 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:01.798 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:44:01.798 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:44:01.798 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:44:01.798 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:44:01.798 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:44:01.798 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:01.799 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:01.799 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:01.799 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:44:01.799 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:44:01.799 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:01.799 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:44:01.799 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:44:01.799 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:44:01.799 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:01.799 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:01.799 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:44:01.799 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:01.799 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:44:01.799 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:44:01.800 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:44:01.800 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:01.800 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:01.800 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:44:01.800 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:01.800 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:44:01.800 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:44:01.800 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:44:01.800 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:01.800 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:01.800 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:01.800 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:44:01.800 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:44:01.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.770+0000 7f4653c08640 1 -- 172.21.15.12:0/1572615596 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f463003e260 msgr2=0x7f4630040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:01.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.770+0000 7f4653c08640 1 --2- 172.21.15.12:0/1572615596 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f463003e260 0x7f4630040720 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f464c129290 tx=0x7f464400b040 comp rx=0 tx=0).stop 2024-09-11T01:44:01.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.770+0000 7f4653c08640 10 monclient: shutdown 2024-09-11T01:44:01.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.770+0000 7f4653c08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:44:01.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.770+0000 7f4653c08640 1 -- 172.21.15.12:0/1572615596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c06f540 msgr2=0x7f464c1cd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:01.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.770+0000 7f4653c08640 1 --2- 172.21.15.12:0/1572615596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f464c06f540 0x7f464c1cd280 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4648000c00 tx=0x7f46480046c0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.770+0000 7f46521a5640 1 -- 172.21.15.12:0/1572615596 reap_dead start 2024-09-11T01:44:01.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.770+0000 7f46521a5640 1 -- 172.21.15.12:0/1572615596 reap_dead start 2024-09-11T01:44:01.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.770+0000 7f4653c08640 1 -- 172.21.15.12:0/1572615596 shutdown_connections 2024-09-11T01:44:01.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.770+0000 7f4653c08640 1 -- 172.21.15.12:0/1572615596 >> 172.21.15.12:0/1572615596 conn(0x7f464c0a7300 msgr2=0x7f464c0a39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:44:01.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.770+0000 7f4653c08640 1 -- 172.21.15.12:0/1572615596 shutdown_connections 2024-09-11T01:44:01.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.770+0000 7f4653c08640 1 -- 172.21.15.12:0/1572615596 wait complete. 2024-09-11T01:44:01.804 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:44:01.804 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:44:01.804 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:44:01.811 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2040458 2024-09-11T01:44:01.845 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1092525 2024-09-11T01:44:01.887 INFO:tasks.rados.rados.0.smithi071.stdout:2287: finishing write tid 3 to smithi07134777-49 2024-09-11T01:44:01.887 INFO:tasks.rados.rados.0.smithi071.stdout:2287: finishing write tid 4 to smithi07134777-49 2024-09-11T01:44:01.887 INFO:tasks.rados.rados.0.smithi071.stdout:2287: finishing write tid 5 to smithi07134777-49 2024-09-11T01:44:01.887 INFO:tasks.rados.rados.0.smithi071.stdout:2287: oid 49 updating version 0 to 368 2024-09-11T01:44:01.887 INFO:tasks.rados.rados.0.smithi071.stdout:2287: oid 49 updating version 368 to 369 2024-09-11T01:44:01.887 INFO:tasks.rados.rados.0.smithi071.stdout:2287: oid 49 updating version 369 to 371 2024-09-11T01:44:01.888 INFO:tasks.rados.rados.0.smithi071.stdout:2287: oid 49 version 371 is already newer than 370 2024-09-11T01:44:01.888 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 371 (ObjNum 828 snap 252 seq_num 828) dirty exists 2024-09-11T01:44:01.888 INFO:tasks.rados.rados.0.smithi071.stdout:2287: left oid 49 (ObjNum 828 snap 252 seq_num 828) 2024-09-11T01:44:01.888 INFO:tasks.rados.rados.0.smithi071.stdout:2289: finishing write tid 1 to smithi07134777-36 2024-09-11T01:44:01.888 INFO:tasks.rados.rados.0.smithi071.stdout:2289: finishing write tid 2 to smithi07134777-36 2024-09-11T01:44:01.888 INFO:tasks.rados.rados.0.smithi071.stdout:2285: done (8 left) 2024-09-11T01:44:01.888 INFO:tasks.rados.rados.0.smithi071.stdout:2286: done (7 left) 2024-09-11T01:44:01.888 INFO:tasks.rados.rados.0.smithi071.stdout:2287: done (6 left) 2024-09-11T01:44:01.888 INFO:tasks.rados.rados.0.smithi071.stdout:2288: done (5 left) 2024-09-11T01:44:01.888 INFO:tasks.rados.rados.0.smithi071.stdout:2292: done (4 left) 2024-09-11T01:44:01.888 INFO:tasks.rados.rados.0.smithi071.stdout:2293: done (3 left) 2024-09-11T01:44:01.888 INFO:tasks.rados.rados.0.smithi071.stdout:2294: snap_create 2024-09-11T01:44:01.889 INFO:tasks.rados.rados.0.smithi071.stdout:2291: expect (ObjNum 825 snap 251 seq_num 825) 2024-09-11T01:44:01.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.903+0000 7fd6c16b0640 10 monclient: get_monmap_and_config 2024-09-11T01:44:01.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.903+0000 7fd6c16b0640 10 monclient: build_initial_monmap 2024-09-11T01:44:01.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.903+0000 7fd6c16b0640 10 monclient: monmap: 2024-09-11T01:44:01.906 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:44:01.906 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:01.907 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:44:01.905504+0000 2024-09-11T01:44:01.907 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:44:01.905504+0000 2024-09-11T01:44:01.907 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:44:01.907 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:01.907 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:44:01.907 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:44:01.907 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:44:01.907 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:44:01.907 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:44:01.907 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:01.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.904+0000 7fd6c16b0640 1 Processor -- start 2024-09-11T01:44:01.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.904+0000 7fd6c16b0640 1 -- start start 2024-09-11T01:44:01.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.904+0000 7fd6c16b0640 10 monclient: init 2024-09-11T01:44:01.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.904+0000 7fd6c16b0640 10 monclient: _reopen_session rank -1 2024-09-11T01:44:01.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.904+0000 7fd6c16b0640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-11T01:44:01.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.904+0000 7fd6c16b0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6bc137e50 addr v1:172.21.15.12:6789/0 2024-09-11T01:44:01.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.904+0000 7fd6c16b0640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd6bc138230 0x7fd6bc138640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:01.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.904+0000 7fd6c16b0640 10 monclient(hunting): picked mon.noname-c con 0x7fd6bc138230 addr v2:172.21.15.71:3300/0 2024-09-11T01:44:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.904+0000 7fd6c16b0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc13d380 0x7fd6bc13f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.904+0000 7fd6c16b0640 10 monclient(hunting): picked mon.noname-b con 0x7fd6bc13d380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:44:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.904+0000 7fd6c16b0640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6bc0a6580 con 0x7fd6bc137e50 2024-09-11T01:44:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.904+0000 7fd6c16b0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.904+0000 7fd6c16b0640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fd6bc0a6310 con 0x7fd6bc138230 2024-09-11T01:44:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.904+0000 7fd6c16b0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.904+0000 7fd6c16b0640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd6bc09bb90 con 0x7fd6bc13d380 2024-09-11T01:44:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.904+0000 7fd6c16b0640 10 monclient(hunting): _renew_subs 2024-09-11T01:44:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.904+0000 7fd6c16b0640 10 monclient(hunting): authenticate will time out at 1648.347977s 2024-09-11T01:44:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6baffd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd6bc137e50 0x7fd6bc09d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:53816/0 (socket says 172.21.15.12:53816) 2024-09-11T01:44:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6baffd640 1 -- 172.21.15.12:0/3877815057 learned_addr learned my addr 172.21.15.12:0/3877815057 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:44:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6bb7fe640 1 --2- 172.21.15.12:0/3877815057 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc13d380 0x7fd6bc13f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6bb7fe640 10 monclient(hunting): get_auth_request con 0x7fd6bc13d380 auth_method 0 2024-09-11T01:44:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:01.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:01.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:01.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6ba7fc640 1 --2- 172.21.15.12:0/3877815057 >> v2:172.21.15.71:3300/0 conn(0x7fd6bc138230 0x7fd6bc138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:01.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6ba7fc640 10 monclient(hunting): get_auth_request con 0x7fd6bc138230 auth_method 0 2024-09-11T01:44:01.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6ba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:01.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6ba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:01.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:01.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:01.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:01.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:01.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6ba7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:01.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6ba7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:01.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6ba7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:01.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6bb7fe640 10 monclient(hunting): handle_auth_done global_id 164120 payload 274 2024-09-11T01:44:01.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6bb7fe640 1 -- 172.21.15.12:0/3877815057 >> v2:172.21.15.71:3300/0 conn(0x7fd6bc138230 msgr2=0x7fd6bc138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:44:01.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6bb7fe640 1 --2- 172.21.15.12:0/3877815057 >> v2:172.21.15.71:3300/0 conn(0x7fd6bc138230 0x7fd6bc138640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6bb7fe640 1 -- 172.21.15.12:0/3877815057 >> v1:172.21.15.12:6789/0 conn(0x7fd6bc137e50 legacy=0x7fd6bc09d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:44:01.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6bb7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:44:01.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6bb7fe640 1 monclient: found mon.noname-b 2024-09-11T01:44:01.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:44:01.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6bb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:44:01.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.905+0000 7fd6bb7fe640 1 -- 172.21.15.12:0/3877815057 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6bc09c780 con 0x7fd6bc13d380 2024-09-11T01:44:01.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6bb7fe640 1 -- 172.21.15.12:0/3877815057 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc13d380 msgr2=0x7fd6bc13f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:01.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6bb7fe640 1 --2- 172.21.15.12:0/3877815057 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc13d380 0x7fd6bc13f770 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4009d90 tx=0x7fd6b402a490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d42fd0124d4f06fd server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:01.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6b9ffb640 1 -- 172.21.15.12:0/3877815057 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b4007950 con 0x7fd6bc13d380 2024-09-11T01:44:01.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:01.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6b9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:44:01.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6b9ffb640 10 monclient: dump: 2024-09-11T01:44:01.916 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:01.916 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:01.916 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.916 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.916 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:01.916 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:01.916 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:01.917 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:01.917 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:01.917 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:01.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6b9ffb640 10 monclient: _finish_auth 0 2024-09-11T01:44:01.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6b9ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:44:01.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:44:01.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6b9ffb640 1 -- 172.21.15.12:0/3877815057 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6b4007af0 con 0x7fd6bc13d380 2024-09-11T01:44:01.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6c16b0640 5 monclient: authenticate success, global_id 164120 2024-09-11T01:44:01.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6c16b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:44:01.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6b9ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:44:01.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6c16b0640 10 monclient: get_monmap_and_config success 2024-09-11T01:44:01.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6b9ffb640 1 -- 172.21.15.12:0/3877815057 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b4031400 con 0x7fd6bc13d380 2024-09-11T01:44:01.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:01.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6c16b0640 4 set_mon_vals no callback set 2024-09-11T01:44:01.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6b9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:44:01.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6b9ffb640 10 monclient: dump: 2024-09-11T01:44:01.918 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:01.918 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:01.918 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.918 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.918 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:01.918 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:01.918 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:01.918 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:01.918 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:01.918 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:01.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6c16b0640 10 monclient: shutdown 2024-09-11T01:44:01.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6c16b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:44:01.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6c16b0640 1 -- 172.21.15.12:0/3877815057 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc13d380 msgr2=0x7fd6bc13f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:01.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6c16b0640 1 --2- 172.21.15.12:0/3877815057 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc13d380 0x7fd6bc13f770 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4009d90 tx=0x7fd6b402a490 comp rx=0 tx=0).stop 2024-09-11T01:44:01.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6c16b0640 1 -- 172.21.15.12:0/3877815057 shutdown_connections 2024-09-11T01:44:01.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6c16b0640 1 --2- 172.21.15.12:0/3877815057 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc13d380 0x7fd6bc13f770 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6c16b0640 1 --2- 172.21.15.12:0/3877815057 >> v2:172.21.15.71:3300/0 conn(0x7fd6bc138230 0x7fd6bc138640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6c16b0640 1 -- 172.21.15.12:0/3877815057 >> 172.21.15.12:0/3877815057 conn(0x7fd6bc0a7020 msgr2=0x7fd6bc0a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:44:01.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6c16b0640 1 -- 172.21.15.12:0/3877815057 shutdown_connections 2024-09-11T01:44:01.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.908+0000 7fd6c16b0640 1 -- 172.21.15.12:0/3877815057 wait complete. 2024-09-11T01:44:01.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.909+0000 7fd6c16b0640 10 monclient: build_initial_monmap 2024-09-11T01:44:01.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.909+0000 7fd6c16b0640 10 monclient: monmap: 2024-09-11T01:44:01.919 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:44:01.919 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:44:01.919 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:44:01.919 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:44:01.920 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:44:01.920 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:01.920 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:44:01.920 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:44:01.920 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:44:01.920 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:01.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.909+0000 7fd6c16b0640 1 Processor -- start 2024-09-11T01:44:01.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.909+0000 7fd6c16b0640 1 -- start start 2024-09-11T01:44:01.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.909+0000 7fd6c16b0640 10 monclient: init 2024-09-11T01:44:01.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6c16b0640 10 monclient: _reopen_session rank -1 2024-09-11T01:44:01.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6c16b0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:44:01.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6c16b0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc137e50 0x7fd6bc1cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:01.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6c16b0640 10 monclient(hunting): picked mon.noname-c con 0x7fd6bc137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:44:01.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6c16b0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6bc138230 0x7fd6bc137080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:01.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6c16b0640 10 monclient(hunting): picked mon.noname-a con 0x7fd6bc138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:44:01.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6c16b0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6bc13d380 0x7fd6bc13a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:01.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6c16b0640 10 monclient(hunting): picked mon.noname-b con 0x7fd6bc13d380 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:44:01.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6c16b0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:01.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6c16b0640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd6bc06f890 con 0x7fd6bc138230 2024-09-11T01:44:01.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6c16b0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:01.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6c16b0640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd6bc06f710 con 0x7fd6bc13d380 2024-09-11T01:44:01.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6c16b0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:01.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6c16b0640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd6bc13fcb0 con 0x7fd6bc137e50 2024-09-11T01:44:01.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6c16b0640 10 monclient(hunting): _renew_subs 2024-09-11T01:44:01.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6c16b0640 10 monclient(hunting): authenticate will time out at 1648.353463s 2024-09-11T01:44:01.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6ba7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6bc138230 0x7fd6bc137080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:01.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6baffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc137e50 0x7fd6bc1cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:01.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6baffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc137e50 0x7fd6bc1cd040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52730/0 (socket says 172.21.15.12:52730) 2024-09-11T01:44:01.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6ba7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6bc138230 0x7fd6bc137080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35402/0 (socket says 172.21.15.12:35402) 2024-09-11T01:44:01.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6baffd640 1 -- 172.21.15.12:0/2109117929 learned_addr learned my addr 172.21.15.12:0/2109117929 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:44:01.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6baffd640 10 monclient(hunting): get_auth_request con 0x7fd6bc137e50 auth_method 0 2024-09-11T01:44:01.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:01.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6baffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:01.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:01.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6ba7fc640 10 monclient(hunting): get_auth_request con 0x7fd6bc138230 auth_method 0 2024-09-11T01:44:01.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6ba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:01.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6ba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:01.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:01.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6bb7fe640 1 --2- 172.21.15.12:0/2109117929 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6bc13d380 0x7fd6bc13a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:01.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6bb7fe640 10 monclient(hunting): get_auth_request con 0x7fd6bc13d380 auth_method 0 2024-09-11T01:44:01.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:01.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:01.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:01.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6baffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:01.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6baffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:01.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6baffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:01.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6ba7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:01.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6ba7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:01.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6ba7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:01.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:01.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:01.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:01.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6ba7fc640 10 monclient(hunting): handle_auth_done global_id 94812 payload 824 2024-09-11T01:44:01.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6ba7fc640 1 -- 172.21.15.12:0/2109117929 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc137e50 msgr2=0x7fd6bc1cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:01.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6ba7fc640 1 --2- 172.21.15.12:0/2109117929 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc137e50 0x7fd6bc1cd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6ba7fc640 1 -- 172.21.15.12:0/2109117929 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6bc13d380 msgr2=0x7fd6bc13a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:01.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6ba7fc640 1 --2- 172.21.15.12:0/2109117929 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6bc13d380 0x7fd6bc13a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6ba7fc640 10 monclient: _finish_hunting 0 2024-09-11T01:44:01.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6ba7fc640 1 monclient: found mon.noname-a 2024-09-11T01:44:01.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:44:01.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6ba7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:44:01.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6ba7fc640 1 -- 172.21.15.12:0/2109117929 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6bc0a1710 con 0x7fd6bc138230 2024-09-11T01:44:01.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.910+0000 7fd6baffd640 1 --2- 172.21.15.12:0/2109117929 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc137e50 0x7fd6bc1cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:44:01.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.913+0000 7fd6ba7fc640 1 -- 172.21.15.12:0/2109117929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6bc138230 msgr2=0x7fd6bc137080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:01.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.913+0000 7fd6ba7fc640 1 --2- 172.21.15.12:0/2109117929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6bc138230 0x7fd6bc137080 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd6a400e850 tx=0x7fd6a400ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:01.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.913+0000 7fd6b37fe640 1 -- 172.21.15.12:0/2109117929 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4017070 con 0x7fd6bc138230 2024-09-11T01:44:01.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.913+0000 7fd6b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:01.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.913+0000 7fd6b37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:44:01.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.913+0000 7fd6b37fe640 10 monclient: dump: 2024-09-11T01:44:01.931 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:01.931 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:01.931 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.931 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.931 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:01.931 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:01.932 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:01.932 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:01.932 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:01.932 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:01.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.913+0000 7fd6b37fe640 10 monclient: _finish_auth 0 2024-09-11T01:44:01.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.913+0000 7fd6b37fe640 10 monclient: _check_auth_tickets 2024-09-11T01:44:01.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.913+0000 7fd6b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:44:01.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.913+0000 7fd6c16b0640 5 monclient: authenticate success, global_id 94812 2024-09-11T01:44:01.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.913+0000 7fd6b37fe640 1 -- 172.21.15.12:0/2109117929 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6a400cb50 con 0x7fd6bc138230 2024-09-11T01:44:01.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.913+0000 7fd6c16b0640 10 monclient: _renew_subs 2024-09-11T01:44:01.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.913+0000 7fd6c16b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:44:01.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.913+0000 7fd6c16b0640 1 -- 172.21.15.12:0/2109117929 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6bc1cdf40 con 0x7fd6bc138230 2024-09-11T01:44:01.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.913+0000 7fd6b37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:44:01.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.913+0000 7fd6b37fe640 1 -- 172.21.15.12:0/2109117929 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4013470 con 0x7fd6bc138230 2024-09-11T01:44:01.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.913+0000 7fd6b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:01.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.913+0000 7fd6b37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:44:01.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.913+0000 7fd6b37fe640 10 monclient: dump: 2024-09-11T01:44:01.933 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:01.933 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:01.933 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.933 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.933 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:01.933 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:01.933 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:01.933 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:01.933 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:01.933 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:01.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.913+0000 7fd6c16b0640 10 monclient: _renew_subs 2024-09-11T01:44:01.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.913+0000 7fd6b97fa640 4 set_mon_vals no callback set 2024-09-11T01:44:01.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.913+0000 7fd6c16b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:44:01.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.913+0000 7fd6c16b0640 1 -- 172.21.15.12:0/2109117929 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6bc13bec0 con 0x7fd6bc138230 2024-09-11T01:44:01.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.914+0000 7fd6c16b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:44:01.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.914+0000 7fd6c16b0640 1 --2- 172.21.15.12:0/2109117929 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc13a940 0x7fd6bc13ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:01.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.914+0000 7fd6c16b0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:01.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.914+0000 7fd6c16b0640 1 -- 172.21.15.12:0/2109117929 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd6bc06a060 con 0x7fd6bc13a940 2024-09-11T01:44:01.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.914+0000 7fd6ba7fc640 1 --2- 172.21.15.12:0/2109117929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6bc138230 0x7fd6bc137080 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd6a400e850 tx=0x7fd6a400ed10 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7fd6a4052d20 osd_map(510..510 src has 317..510) 2024-09-11T01:44:01.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.914+0000 7fd6b37fe640 1 -- 172.21.15.12:0/2109117929 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fd6a4024060 con 0x7fd6bc138230 2024-09-11T01:44:01.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.914+0000 7fd6baffd640 1 --2- 172.21.15.12:0/2109117929 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc13a940 0x7fd6bc13ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:01.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.914+0000 7fd6baffd640 10 monclient: get_auth_request con 0x7fd6bc13a940 auth_method 0 2024-09-11T01:44:01.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.914+0000 7fd6baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:01.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.914+0000 7fd6baffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:01.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.914+0000 7fd6baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:01.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.914+0000 7fd6baffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:01.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.914+0000 7fd6baffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:01.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.914+0000 7fd6baffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:01.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.914+0000 7fd6b37fe640 1 --2- 172.21.15.12:0/2109117929 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd69c03e240 0x7fd69c040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:01.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.914+0000 7fd6baffd640 10 monclient(hunting): handle_auth_done global_id 164126 payload 824 2024-09-11T01:44:01.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.914+0000 7fd6baffd640 1 -- 172.21.15.12:0/2109117929 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd6bc13b260 con 0x7fd6bc13a940 2024-09-11T01:44:01.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.914+0000 7fd6bb7fe640 1 --2- 172.21.15.12:0/2109117929 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd69c03e240 0x7fd69c040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:01.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.914+0000 7fd6bb7fe640 10 monclient: get_auth_request con 0x7fd69c03e240 auth_method 0 2024-09-11T01:44:01.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.916+0000 7fd6baffd640 1 -- 172.21.15.12:0/2109117929 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc13a940 msgr2=0x7fd6bc13ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:01.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.916+0000 7fd6baffd640 1 --2- 172.21.15.12:0/2109117929 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc13a940 0x7fd6bc13ad20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac0072e0 tx=0x7fd6ac0077a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:01.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.917+0000 7fd6b37fe640 1 -- 172.21.15.12:0/2109117929 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac00fd40 con 0x7fd6bc13a940 2024-09-11T01:44:01.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.917+0000 7fd6b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:01.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.917+0000 7fd6b37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:44:01.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.917+0000 7fd6b37fe640 10 monclient: dump: 2024-09-11T01:44:01.936 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:01.936 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:01.936 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.936 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.937 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:01.937 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:01.937 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:01.937 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:01.937 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:01.937 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:01.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.917+0000 7fd6b37fe640 1 -- 172.21.15.12:0/2109117929 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd6ac00fd40 con 0x7fd6bc13a940 2024-09-11T01:44:01.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.917+0000 7fd6b37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:44:01.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.917+0000 7fd6b37fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:44:01.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.917+0000 7fd6b37fe640 1 -- 172.21.15.12:0/2109117929 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc13a940 msgr2=0x7fd6bc13ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:01.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.917+0000 7fd6b37fe640 1 --2- 172.21.15.12:0/2109117929 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc13a940 0x7fd6bc13ad20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac0072e0 tx=0x7fd6ac0077a0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.917+0000 7fd6bb7fe640 1 --2- 172.21.15.12:0/2109117929 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd69c03e240 0x7fd69c040700 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc0ab580 tx=0x7fd6b402f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:01.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.917+0000 7fd6b37fe640 1 -- 172.21.15.12:0/2109117929 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc13a940 msgr2=0x7fd6bc13ad20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:44:01.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.917+0000 7fd6b37fe640 1 --2- 172.21.15.12:0/2109117929 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc13a940 0x7fd6bc13ad20 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac0072e0 tx=0x7fd6ac0077a0 comp rx=0 tx=0).stop 2024-09-11T01:44:01.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.923+0000 7fd6c16b0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:44:01.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.923+0000 7fd6c16b0640 1 --2- 172.21.15.12:0/2109117929 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc0a1920 0x7fd6bc0a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:01.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.923+0000 7fd6c16b0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:01.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.923+0000 7fd6c16b0640 1 -- 172.21.15.12:0/2109117929 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd6bc138640 con 0x7fd6bc0a1920 2024-09-11T01:44:01.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.923+0000 7fd6baffd640 1 --2- 172.21.15.12:0/2109117929 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc0a1920 0x7fd6bc0a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:01.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.923+0000 7fd6baffd640 10 monclient: get_auth_request con 0x7fd6bc0a1920 auth_method 0 2024-09-11T01:44:01.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.923+0000 7fd6baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:01.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.923+0000 7fd6baffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:01.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.923+0000 7fd6baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:01.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.923+0000 7fd6baffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:01.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.923+0000 7fd6baffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:01.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.923+0000 7fd6baffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:01.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.924+0000 7fd6baffd640 10 monclient(hunting): handle_auth_done global_id 164129 payload 824 2024-09-11T01:44:01.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.924+0000 7fd6baffd640 1 -- 172.21.15.12:0/2109117929 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd6bc069950 con 0x7fd6bc0a1920 2024-09-11T01:44:01.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.926+0000 7fd6baffd640 1 -- 172.21.15.12:0/2109117929 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc0a1920 msgr2=0x7fd6bc0a1d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:01.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.926+0000 7fd6baffd640 1 --2- 172.21.15.12:0/2109117929 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc0a1920 0x7fd6bc0a1d30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac017000 tx=0x7fd6ac004940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:01.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.926+0000 7fd6b37fe640 1 -- 172.21.15.12:0/2109117929 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac006030 con 0x7fd6bc0a1920 2024-09-11T01:44:01.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.926+0000 7fd6b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:01.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.926+0000 7fd6b37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:44:01.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.926+0000 7fd6b37fe640 10 monclient: dump: 2024-09-11T01:44:01.941 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:01.941 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:01.941 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.941 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:01.941 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:01.941 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:01.941 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:01.941 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:01.941 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:01.941 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:01.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.926+0000 7fd6b37fe640 1 -- 172.21.15.12:0/2109117929 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7fd6ac006030 con 0x7fd6bc0a1920 2024-09-11T01:44:01.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.926+0000 7fd6b37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:44:01.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.926+0000 7fd6b37fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:44:01.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.926+0000 7fd6b37fe640 1 -- 172.21.15.12:0/2109117929 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc0a1920 msgr2=0x7fd6bc0a1d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:01.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.926+0000 7fd6b37fe640 1 --2- 172.21.15.12:0/2109117929 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc0a1920 0x7fd6bc0a1d30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac017000 tx=0x7fd6ac004940 comp rx=0 tx=0).stop 2024-09-11T01:44:01.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.926+0000 7fd6b37fe640 1 -- 172.21.15.12:0/2109117929 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc0a1920 msgr2=0x7fd6bc0a1d30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:44:01.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.926+0000 7fd6b37fe640 1 --2- 172.21.15.12:0/2109117929 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6bc0a1920 0x7fd6bc0a1d30 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac017000 tx=0x7fd6ac004940 comp rx=0 tx=0).stop 2024-09-11T01:44:01.942 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:44:01.942 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:44:01.942 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:44:01.943 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:44:01.943 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 138, 2024-09-11T01:44:01.943 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 17940, 2024-09-11T01:44:01.943 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:44:01.943 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:44:01.943 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:44:01.943 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:44:01.943 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:01.943 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 6, 2024-09-11T01:44:01.943 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:44:01.943 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:44:01.943 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:44:01.943 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:44:01.943 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:44:01.944 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:44:01.944 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:44:01.944 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:44:01.944 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:44:01.944 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:44:01.944 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:44:01.944 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:44:01.944 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:44:01.944 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:44:01.944 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:01.944 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:44:01.944 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:44:01.944 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:44:01.944 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:44:01.945 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:44:01.945 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:44:01.945 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:44:01.945 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:44:01.945 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:44:01.945 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:44:01.945 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:44:01.945 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:44:01.945 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:44:01.945 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:01.945 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:01.945 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:44:01.945 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:44:01.945 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:44:01.946 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:44:01.946 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:44:01.946 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:44:01.946 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:44:01.946 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:44:01.946 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:44:01.946 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:44:01.946 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:44:01.946 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:44:01.946 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:44:01.946 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:44:01.946 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:44:01.946 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:44:01.946 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:44:01.947 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:44:01.947 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:44:01.947 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:44:01.947 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:44:01.947 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:44:01.947 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:44:01.947 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:44:01.947 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:44:01.947 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:44:01.947 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:44:01.947 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:44:01.947 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:01.947 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:44:01.947 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:01.947 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:44:01.948 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:01.948 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:44:01.948 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:44:01.948 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:44:01.948 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:44:01.948 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:01.948 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:44:01.948 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:44:01.948 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:01.948 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:01.948 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:01.948 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:44:01.948 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:44:01.948 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:01.949 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:01.949 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:01.949 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:01.949 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:44:01.949 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:44:01.949 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:44:01.949 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:44:01.949 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:44:01.949 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:01.949 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:01.949 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:44:01.949 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:44:01.950 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:44:01.950 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:44:01.950 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:01.950 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:44:01.950 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:44:01.950 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:01.950 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:01.950 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:01.950 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:44:01.950 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:44:01.950 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:01.950 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:01.950 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:01.950 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:01.951 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:44:01.951 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:44:01.951 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:44:01.951 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:44:01.951 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:44:01.951 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:01.951 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:01.951 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:44:01.951 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:44:01.951 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:44:01.951 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:44:01.951 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:01.951 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:44:01.951 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:44:01.952 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:01.952 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:01.952 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:01.952 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:44:01.952 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:44:01.952 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:01.952 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:01.952 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:01.952 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:01.952 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:44:01.952 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:44:01.952 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:44:01.952 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:44:01.952 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:44:01.953 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:01.953 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:01.953 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:01.953 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:44:01.953 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:44:01.953 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:01.953 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:44:01.953 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:44:01.953 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:44:01.953 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:01.953 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:01.953 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:44:01.954 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:01.954 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:44:01.954 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:44:01.954 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:44:01.954 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:01.954 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:01.954 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:01.954 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:44:01.954 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:44:01.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.927+0000 7fd6c16b0640 1 -- 172.21.15.12:0/2109117929 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd69c03e240 msgr2=0x7fd69c040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.927+0000 7fd6c16b0640 1 --2- 172.21.15.12:0/2109117929 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd69c03e240 0x7fd69c040700 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc0ab580 tx=0x7fd6b402f000 comp rx=0 tx=0).stop 2024-09-11T01:44:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.927+0000 7fd6c16b0640 10 monclient: shutdown 2024-09-11T01:44:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.927+0000 7fd6c16b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:44:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.927+0000 7fd6c16b0640 1 -- 172.21.15.12:0/2109117929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6bc138230 msgr2=0x7fd6bc137080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.927+0000 7fd6c16b0640 1 --2- 172.21.15.12:0/2109117929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6bc138230 0x7fd6bc137080 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd6a400e850 tx=0x7fd6a400ed10 comp rx=0 tx=0).stop 2024-09-11T01:44:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.927+0000 7fd6bb7fe640 1 -- 172.21.15.12:0/2109117929 reap_dead start 2024-09-11T01:44:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.927+0000 7fd6b37fe640 1 -- 172.21.15.12:0/2109117929 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(510..510 src has 317..510) ==== 4819+0+0 (unknown 0 0 0) 0x7fd6a4052d20 con 0x7fd6bc138230 2024-09-11T01:44:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.927+0000 7fd6bb7fe640 1 -- 172.21.15.12:0/2109117929 reap_dead start 2024-09-11T01:44:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.927+0000 7fd6c16b0640 1 -- 172.21.15.12:0/2109117929 shutdown_connections 2024-09-11T01:44:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.927+0000 7fd6c16b0640 1 -- 172.21.15.12:0/2109117929 >> 172.21.15.12:0/2109117929 conn(0x7fd6bc0a7020 msgr2=0x7fd6bc13e840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:44:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.927+0000 7fd6c16b0640 1 -- 172.21.15.12:0/2109117929 shutdown_connections 2024-09-11T01:44:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:01.927+0000 7fd6c16b0640 1 -- 172.21.15.12:0/2109117929 wait complete. 2024-09-11T01:44:01.957 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:44:01.957 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:44:01.957 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:44:01.987 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:44:02.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.056+0000 7fb48e12b640 10 monclient: get_monmap_and_config 2024-09-11T01:44:02.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.056+0000 7fb48e12b640 10 monclient: build_initial_monmap 2024-09-11T01:44:02.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.056+0000 7fb48e12b640 10 monclient: monmap: 2024-09-11T01:44:02.059 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:44:02.059 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:02.059 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:44:02.058509+0000 2024-09-11T01:44:02.060 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:44:02.058509+0000 2024-09-11T01:44:02.060 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:44:02.060 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:02.060 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:44:02.060 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:44:02.060 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:44:02.060 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:44:02.060 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:44:02.060 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:02.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.057+0000 7fb48e12b640 1 Processor -- start 2024-09-11T01:44:02.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.057+0000 7fb48e12b640 1 -- start start 2024-09-11T01:44:02.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.057+0000 7fb48e12b640 10 monclient: init 2024-09-11T01:44:02.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb48e12b640 10 monclient: _reopen_session rank -1 2024-09-11T01:44:02.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb48e12b640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-11T01:44:02.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb48e12b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4881381c0 0x7fb4881385a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:02.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb48e12b640 10 monclient(hunting): picked mon.noname-b con 0x7fb4881381c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:44:02.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb48e12b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb4880a6aa0 addr v1:172.21.15.71:6789/0 2024-09-11T01:44:02.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb48e12b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb48813d000 addr v1:172.21.15.12:6789/0 2024-09-11T01:44:02.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb48e12b640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb48809c310 con 0x7fb48813d000 2024-09-11T01:44:02.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb48e12b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb48809c8d0 con 0x7fb4880a6aa0 2024-09-11T01:44:02.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb48e12b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:02.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb48e12b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb48806fc90 con 0x7fb4881381c0 2024-09-11T01:44:02.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb48e12b640 10 monclient(hunting): _renew_subs 2024-09-11T01:44:02.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb48e12b640 10 monclient(hunting): authenticate will time out at 1648.501273s 2024-09-11T01:44:02.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb4877fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4881381c0 0x7fb4881385a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:02.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb4877fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4881381c0 0x7fb4881385a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52752/0 (socket says 172.21.15.12:52752) 2024-09-11T01:44:02.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb4877fe640 1 -- 172.21.15.12:0/36558251 learned_addr learned my addr 172.21.15.12:0/36558251 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:44:02.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb4877fe640 10 monclient(hunting): get_auth_request con 0x7fb4881381c0 auth_method 0 2024-09-11T01:44:02.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb4877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:02.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb4877fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:02.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb4877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:02.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb4877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:02.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb4877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:02.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb4877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:02.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb4877fe640 10 monclient(hunting): handle_auth_done global_id 164132 payload 274 2024-09-11T01:44:02.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb4877fe640 1 -- 172.21.15.12:0/36558251 >> v1:172.21.15.71:6789/0 conn(0x7fb4880a6aa0 legacy=0x7fb48809cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:44:02.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb4877fe640 1 -- 172.21.15.12:0/36558251 >> v1:172.21.15.12:6789/0 conn(0x7fb48813d000 legacy=0x7fb48813f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:44:02.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb4877fe640 10 monclient: _finish_hunting 0 2024-09-11T01:44:02.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb4877fe640 1 monclient: found mon.noname-b 2024-09-11T01:44:02.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb4877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:44:02.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb4877fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:44:02.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.058+0000 7fb4877fe640 1 -- 172.21.15.12:0/36558251 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb488138ae0 con 0x7fb4881381c0 2024-09-11T01:44:02.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.061+0000 7fb4877fe640 1 -- 172.21.15.12:0/36558251 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4881381c0 msgr2=0x7fb4881385a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:02.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.061+0000 7fb4877fe640 1 --2- 172.21.15.12:0/36558251 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4881381c0 0x7fb4881385a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb4780098a0 tx=0x7fb47802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e921413ce802d3e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:02.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.061+0000 7fb4867fc640 1 -- 172.21.15.12:0/36558251 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb478037070 con 0x7fb4881381c0 2024-09-11T01:44:02.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.061+0000 7fb4867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:02.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.061+0000 7fb4867fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:44:02.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.061+0000 7fb4867fc640 10 monclient: dump: 2024-09-11T01:44:02.072 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:02.072 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:02.072 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:02.072 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:02.072 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:02.072 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:02.072 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:02.072 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:02.073 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:02.073 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:02.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.061+0000 7fb4867fc640 10 monclient: _finish_auth 0 2024-09-11T01:44:02.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.061+0000 7fb4867fc640 10 monclient: _check_auth_tickets 2024-09-11T01:44:02.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.061+0000 7fb4867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:44:02.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.061+0000 7fb4867fc640 1 -- 172.21.15.12:0/36558251 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb478032640 con 0x7fb4881381c0 2024-09-11T01:44:02.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.061+0000 7fb48e12b640 5 monclient: authenticate success, global_id 164132 2024-09-11T01:44:02.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.061+0000 7fb48e12b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:44:02.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.061+0000 7fb4867fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:44:02.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.061+0000 7fb48e12b640 10 monclient: get_monmap_and_config success 2024-09-11T01:44:02.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.061+0000 7fb4867fc640 1 -- 172.21.15.12:0/36558251 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb478032a10 con 0x7fb4881381c0 2024-09-11T01:44:02.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.061+0000 7fb4867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:02.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.061+0000 7fb48e12b640 4 set_mon_vals no callback set 2024-09-11T01:44:02.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.061+0000 7fb4867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:44:02.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.061+0000 7fb4867fc640 10 monclient: dump: 2024-09-11T01:44:02.074 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:02.074 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:02.074 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:02.074 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:02.074 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:02.074 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:02.074 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:02.074 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:02.074 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:02.074 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:02.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.062+0000 7fb48e12b640 10 monclient: shutdown 2024-09-11T01:44:02.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.062+0000 7fb48e12b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:44:02.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.062+0000 7fb48e12b640 1 -- 172.21.15.12:0/36558251 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4881381c0 msgr2=0x7fb4881385a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:02.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.062+0000 7fb48e12b640 1 --2- 172.21.15.12:0/36558251 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4881381c0 0x7fb4881385a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb4780098a0 tx=0x7fb47802a100 comp rx=0 tx=0).stop 2024-09-11T01:44:02.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.062+0000 7fb48e12b640 1 -- 172.21.15.12:0/36558251 shutdown_connections 2024-09-11T01:44:02.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.062+0000 7fb48e12b640 1 --2- 172.21.15.12:0/36558251 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4881381c0 0x7fb4881385a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:02.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.062+0000 7fb48e12b640 1 -- 172.21.15.12:0/36558251 >> 172.21.15.12:0/36558251 conn(0x7fb4880a7440 msgr2=0x7fb4880a7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:44:02.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.062+0000 7fb48e12b640 1 -- 172.21.15.12:0/36558251 shutdown_connections 2024-09-11T01:44:02.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.062+0000 7fb48e12b640 1 -- 172.21.15.12:0/36558251 wait complete. 2024-09-11T01:44:02.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.062+0000 7fb48e12b640 10 monclient: build_initial_monmap 2024-09-11T01:44:02.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.062+0000 7fb48e12b640 10 monclient: monmap: 2024-09-11T01:44:02.075 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:44:02.075 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:44:02.075 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:44:02.075 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:44:02.075 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:44:02.075 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:02.076 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:44:02.076 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:44:02.076 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:44:02.076 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:02.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb48e12b640 1 Processor -- start 2024-09-11T01:44:02.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb48e12b640 1 -- start start 2024-09-11T01:44:02.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb48e12b640 10 monclient: init 2024-09-11T01:44:02.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb48e12b640 10 monclient: _reopen_session rank -1 2024-09-11T01:44:02.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb48e12b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:44:02.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb48e12b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4880a6aa0 0x7fb488131fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:02.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb48e12b640 10 monclient(hunting): picked mon.noname-a con 0x7fb4880a6aa0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:44:02.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb48e12b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4881381c0 0x7fb488132510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:02.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb48e12b640 10 monclient(hunting): picked mon.noname-c con 0x7fb4881381c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:44:02.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb48e12b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb48813d000 0x7fb488132ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:02.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb48e12b640 10 monclient(hunting): picked mon.noname-b con 0x7fb48813d000 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:44:02.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb48e12b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:02.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb48e12b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb488140d20 con 0x7fb4880a6aa0 2024-09-11T01:44:02.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb48e12b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:02.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb48e12b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb48809c5f0 con 0x7fb48813d000 2024-09-11T01:44:02.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb48e12b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:02.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb48e12b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb488140ea0 con 0x7fb4881381c0 2024-09-11T01:44:02.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb48e12b640 10 monclient(hunting): _renew_subs 2024-09-11T01:44:02.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb48e12b640 10 monclient(hunting): authenticate will time out at 1648.506843s 2024-09-11T01:44:02.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb486ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4881381c0 0x7fb488132510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:02.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb4877fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4880a6aa0 0x7fb488131fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:02.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb4877fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4880a6aa0 0x7fb488131fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35408/0 (socket says 172.21.15.12:35408) 2024-09-11T01:44:02.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb4877fe640 1 -- 172.21.15.12:0/3107983053 learned_addr learned my addr 172.21.15.12:0/3107983053 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:44:02.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb4877fe640 10 monclient(hunting): get_auth_request con 0x7fb4880a6aa0 auth_method 0 2024-09-11T01:44:02.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb4877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:02.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb4877fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:02.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb4877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:02.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb486ffd640 10 monclient(hunting): get_auth_request con 0x7fb4881381c0 auth_method 0 2024-09-11T01:44:02.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb486ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:02.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb486ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:02.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.063+0000 7fb486ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:02.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.064+0000 7fb487fff640 1 --2- 172.21.15.12:0/3107983053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb48813d000 0x7fb488132ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:02.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.064+0000 7fb487fff640 10 monclient(hunting): get_auth_request con 0x7fb48813d000 auth_method 0 2024-09-11T01:44:02.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.064+0000 7fb487fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:02.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.064+0000 7fb487fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:02.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.064+0000 7fb487fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:02.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.064+0000 7fb4877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.064+0000 7fb4877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.064+0000 7fb4877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.064+0000 7fb486ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.064+0000 7fb486ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.064+0000 7fb486ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.064+0000 7fb487fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.064+0000 7fb487fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.064+0000 7fb487fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.064+0000 7fb4877fe640 10 monclient(hunting): handle_auth_done global_id 94815 payload 824 2024-09-11T01:44:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.064+0000 7fb4877fe640 1 -- 172.21.15.12:0/3107983053 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4881381c0 msgr2=0x7fb488132510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.064+0000 7fb4877fe640 1 --2- 172.21.15.12:0/3107983053 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4881381c0 0x7fb488132510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.064+0000 7fb4877fe640 1 -- 172.21.15.12:0/3107983053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb48813d000 msgr2=0x7fb488132ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.064+0000 7fb4877fe640 1 --2- 172.21.15.12:0/3107983053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb48813d000 0x7fb488132ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.064+0000 7fb4877fe640 10 monclient: _finish_hunting 0 2024-09-11T01:44:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.064+0000 7fb4877fe640 1 monclient: found mon.noname-a 2024-09-11T01:44:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.064+0000 7fb4877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:44:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.064+0000 7fb4877fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:44:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.064+0000 7fb4877fe640 1 -- 172.21.15.12:0/3107983053 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb488135b30 con 0x7fb4880a6aa0 2024-09-11T01:44:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.066+0000 7fb4877fe640 1 -- 172.21.15.12:0/3107983053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4880a6aa0 msgr2=0x7fb488131fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.066+0000 7fb4877fe640 1 --2- 172.21.15.12:0/3107983053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4880a6aa0 0x7fb488131fd0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb4780097f0 tx=0x7fb4780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.066+0000 7fb46ffff640 1 -- 172.21.15.12:0/3107983053 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb478041070 con 0x7fb4880a6aa0 2024-09-11T01:44:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.066+0000 7fb46ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.066+0000 7fb46ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:44:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.066+0000 7fb46ffff640 10 monclient: dump: 2024-09-11T01:44:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:02.081 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.066+0000 7fb46ffff640 10 monclient: _finish_auth 0 2024-09-11T01:44:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.066+0000 7fb46ffff640 10 monclient: _check_auth_tickets 2024-09-11T01:44:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.066+0000 7fb46ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:44:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.066+0000 7fb46ffff640 1 -- 172.21.15.12:0/3107983053 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb47803c880 con 0x7fb4880a6aa0 2024-09-11T01:44:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.066+0000 7fb48e12b640 5 monclient: authenticate success, global_id 94815 2024-09-11T01:44:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.066+0000 7fb48e12b640 10 monclient: _renew_subs 2024-09-11T01:44:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.066+0000 7fb48e12b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:44:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.066+0000 7fb48e12b640 1 -- 172.21.15.12:0/3107983053 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb488140410 con 0x7fb4880a6aa0 2024-09-11T01:44:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.066+0000 7fb46ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:44:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.066+0000 7fb46ffff640 1 -- 172.21.15.12:0/3107983053 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb478037040 con 0x7fb4880a6aa0 2024-09-11T01:44:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.066+0000 7fb46ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.066+0000 7fb46ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:44:02.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.066+0000 7fb46ffff640 10 monclient: dump: 2024-09-11T01:44:02.083 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:02.083 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:02.083 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:02.083 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:02.083 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:02.083 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:02.083 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:02.083 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:02.083 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:02.083 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:02.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.066+0000 7fb48e12b640 10 monclient: _renew_subs 2024-09-11T01:44:02.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.066+0000 7fb48e12b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:44:02.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.066+0000 7fb48e12b640 1 -- 172.21.15.12:0/3107983053 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4880a09e0 con 0x7fb4880a6aa0 2024-09-11T01:44:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.066+0000 7fb485ffb640 4 set_mon_vals no callback set 2024-09-11T01:44:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.067+0000 7fb48e12b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:44:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.067+0000 7fb48e12b640 1 --2- 172.21.15.12:0/3107983053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4880a01f0 0x7fb4881354d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.067+0000 7fb48e12b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.067+0000 7fb48e12b640 1 -- 172.21.15.12:0/3107983053 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb488136560 con 0x7fb4880a01f0 2024-09-11T01:44:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.067+0000 7fb46ffff640 1 -- 172.21.15.12:0/3107983053 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fb47803cc70 con 0x7fb4880a6aa0 2024-09-11T01:44:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.067+0000 7fb486ffd640 1 --2- 172.21.15.12:0/3107983053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4880a01f0 0x7fb4881354d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.067+0000 7fb486ffd640 10 monclient: get_auth_request con 0x7fb4880a01f0 auth_method 0 2024-09-11T01:44:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.067+0000 7fb486ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.067+0000 7fb486ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.067+0000 7fb486ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.067+0000 7fb486ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.067+0000 7fb486ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.067+0000 7fb486ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.068+0000 7fb46ffff640 1 --2- 172.21.15.12:0/3107983053 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb47403e260 0x7fb474040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.068+0000 7fb486ffd640 10 monclient(hunting): handle_auth_done global_id 134407 payload 824 2024-09-11T01:44:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.068+0000 7fb486ffd640 1 -- 172.21.15.12:0/3107983053 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb4881385a0 con 0x7fb4880a01f0 2024-09-11T01:44:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.068+0000 7fb46ffff640 1 -- 172.21.15.12:0/3107983053 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(510..510 src has 317..510) ==== 4819+0+0 (secure 0 0 0) 0x7fb478073350 con 0x7fb4880a6aa0 2024-09-11T01:44:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.068+0000 7fb487fff640 1 --2- 172.21.15.12:0/3107983053 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb47403e260 0x7fb474040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.068+0000 7fb487fff640 10 monclient: get_auth_request con 0x7fb47403e260 auth_method 0 2024-09-11T01:44:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.070+0000 7fb486ffd640 1 -- 172.21.15.12:0/3107983053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4880a01f0 msgr2=0x7fb4881354d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.070+0000 7fb486ffd640 1 --2- 172.21.15.12:0/3107983053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4880a01f0 0x7fb4881354d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb4880a4be0 tx=0x7fb47c00fc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.070+0000 7fb46ffff640 1 -- 172.21.15.12:0/3107983053 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb47c018030 con 0x7fb4880a01f0 2024-09-11T01:44:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.070+0000 7fb46ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.070+0000 7fb46ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:44:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.070+0000 7fb46ffff640 10 monclient: dump: 2024-09-11T01:44:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:02.088 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.070+0000 7fb487fff640 1 --2- 172.21.15.12:0/3107983053 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb47403e260 0x7fb474040720 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb4800055d0 tx=0x7fb480003c60 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.070+0000 7fb46ffff640 1 -- 172.21.15.12:0/3107983053 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb47c00d9b0 con 0x7fb4880a01f0 2024-09-11T01:44:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.070+0000 7fb46ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:44:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.070+0000 7fb46ffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:44:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.070+0000 7fb46ffff640 1 -- 172.21.15.12:0/3107983053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4880a01f0 msgr2=0x7fb4881354d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.070+0000 7fb46ffff640 1 --2- 172.21.15.12:0/3107983053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4880a01f0 0x7fb4881354d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb4880a4be0 tx=0x7fb47c00fc00 comp rx=0 tx=0).stop 2024-09-11T01:44:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.070+0000 7fb46ffff640 1 -- 172.21.15.12:0/3107983053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4880a01f0 msgr2=0x7fb4881354d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:44:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.070+0000 7fb46ffff640 1 --2- 172.21.15.12:0/3107983053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4880a01f0 0x7fb4881354d0 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb4880a4be0 tx=0x7fb47c00fc00 comp rx=0 tx=0).stop 2024-09-11T01:44:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.076+0000 7fb48e12b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:44:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.077+0000 7fb48e12b640 1 --2- 172.21.15.12:0/3107983053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb48802d440 0x7fb48802d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.077+0000 7fb48e12b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.077+0000 7fb48e12b640 1 -- 172.21.15.12:0/3107983053 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb488132a50 con 0x7fb48802d440 2024-09-11T01:44:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.077+0000 7fb486ffd640 1 --2- 172.21.15.12:0/3107983053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb48802d440 0x7fb48802d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.077+0000 7fb486ffd640 10 monclient: get_auth_request con 0x7fb48802d440 auth_method 0 2024-09-11T01:44:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.077+0000 7fb486ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.077+0000 7fb486ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.077+0000 7fb486ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.077+0000 7fb486ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.077+0000 7fb486ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.077+0000 7fb486ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.077+0000 7fb486ffd640 10 monclient(hunting): handle_auth_done global_id 134410 payload 824 2024-09-11T01:44:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.077+0000 7fb486ffd640 1 -- 172.21.15.12:0/3107983053 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb488069840 con 0x7fb48802d440 2024-09-11T01:44:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.079+0000 7fb486ffd640 1 -- 172.21.15.12:0/3107983053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb48802d440 msgr2=0x7fb48802d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.079+0000 7fb486ffd640 1 --2- 172.21.15.12:0/3107983053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb48802d440 0x7fb48802d850 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb4880a4be0 tx=0x7fb47c003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.079+0000 7fb46ffff640 1 -- 172.21.15.12:0/3107983053 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb47c01dbe0 con 0x7fb48802d440 2024-09-11T01:44:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.079+0000 7fb46ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.079+0000 7fb46ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:44:02.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.079+0000 7fb46ffff640 10 monclient: dump: 2024-09-11T01:44:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:02.091 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.080+0000 7fb46ffff640 1 -- 172.21.15.12:0/3107983053 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7fb47c01dbe0 con 0x7fb48802d440 2024-09-11T01:44:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.080+0000 7fb46ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:44:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.080+0000 7fb46ffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:44:02.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.080+0000 7fb46ffff640 1 -- 172.21.15.12:0/3107983053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb48802d440 msgr2=0x7fb48802d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:02.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.080+0000 7fb46ffff640 1 --2- 172.21.15.12:0/3107983053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb48802d440 0x7fb48802d850 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb4880a4be0 tx=0x7fb47c003760 comp rx=0 tx=0).stop 2024-09-11T01:44:02.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.080+0000 7fb46ffff640 1 -- 172.21.15.12:0/3107983053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb48802d440 msgr2=0x7fb48802d850 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:44:02.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.080+0000 7fb46ffff640 1 --2- 172.21.15.12:0/3107983053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb48802d440 0x7fb48802d850 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb4880a4be0 tx=0x7fb47c003760 comp rx=0 tx=0).stop 2024-09-11T01:44:02.092 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:44:02.092 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:44:02.092 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:44:02.092 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:44:02.092 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 138, 2024-09-11T01:44:02.092 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 175042, 2024-09-11T01:44:02.092 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:44:02.093 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:44:02.093 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:44:02.093 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:44:02.093 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:02.093 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 6, 2024-09-11T01:44:02.093 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:44:02.093 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:44:02.093 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:44:02.093 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:44:02.093 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:44:02.093 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:44:02.094 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:44:02.094 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:44:02.094 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:44:02.094 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:44:02.094 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:44:02.094 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:44:02.094 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:44:02.094 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:44:02.094 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:02.094 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:44:02.094 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:44:02.094 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:44:02.094 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:44:02.095 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:44:02.095 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:44:02.095 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:44:02.095 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:44:02.095 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:44:02.095 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:44:02.095 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:44:02.095 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:44:02.095 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:44:02.095 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:02.095 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:02.095 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:44:02.095 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:44:02.096 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:44:02.096 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:44:02.096 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:44:02.096 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:44:02.096 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:44:02.096 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:44:02.096 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:44:02.096 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:44:02.096 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:44:02.096 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:44:02.096 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:44:02.096 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:44:02.096 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:44:02.096 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:44:02.097 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:44:02.097 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:44:02.097 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:44:02.097 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:44:02.097 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:44:02.097 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:44:02.097 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:44:02.097 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:44:02.097 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:44:02.097 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:44:02.097 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:44:02.097 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:44:02.097 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:02.098 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:44:02.098 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:02.098 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:44:02.098 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:02.098 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:44:02.098 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:44:02.098 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:44:02.098 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:44:02.098 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:02.098 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:44:02.098 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:44:02.098 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:02.098 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:02.098 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:02.098 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:44:02.099 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:44:02.099 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:02.099 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:02.099 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:02.099 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:02.099 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:44:02.099 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:44:02.099 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:44:02.099 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:44:02.099 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:44:02.099 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:02.099 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:02.099 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:44:02.099 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:44:02.100 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:44:02.100 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:44:02.100 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:02.100 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:44:02.100 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:44:02.100 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:02.100 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:02.100 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:02.100 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:44:02.100 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:44:02.100 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:02.100 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:02.100 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:02.100 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:02.100 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:44:02.101 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:44:02.101 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:44:02.101 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:44:02.101 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:44:02.101 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:02.101 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:02.101 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:44:02.101 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:44:02.101 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:44:02.101 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:44:02.101 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:02.101 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:44:02.101 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:44:02.101 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:02.102 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:02.102 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:02.102 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:44:02.102 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:44:02.102 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:02.102 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:02.102 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:02.102 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:02.102 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:44:02.102 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:44:02.102 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:44:02.102 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:44:02.102 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:44:02.102 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:02.102 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:02.103 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:02.103 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:44:02.103 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:44:02.103 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:02.103 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:44:02.103 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:44:02.103 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:44:02.103 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:02.103 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:02.103 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:44:02.103 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:02.103 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:44:02.103 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:44:02.103 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:44:02.104 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:02.104 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:02.104 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:44:02.104 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:02.104 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:44:02.104 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:44:02.104 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:44:02.104 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:02.104 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:02.104 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:44:02.104 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:02.104 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:44:02.104 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:44:02.105 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:44:02.105 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:02.105 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:02.105 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:02.105 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:44:02.105 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:44:02.106 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 731607 2024-09-11T01:44:02.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.080+0000 7fb48e12b640 1 -- 172.21.15.12:0/3107983053 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb47403e260 msgr2=0x7fb474040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:02.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.080+0000 7fb48e12b640 1 --2- 172.21.15.12:0/3107983053 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb47403e260 0x7fb474040720 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb4800055d0 tx=0x7fb480003c60 comp rx=0 tx=0).stop 2024-09-11T01:44:02.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.080+0000 7fb48e12b640 10 monclient: shutdown 2024-09-11T01:44:02.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.080+0000 7fb48e12b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:44:02.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.080+0000 7fb48e12b640 1 -- 172.21.15.12:0/3107983053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4880a6aa0 msgr2=0x7fb488131fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:02.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.080+0000 7fb48e12b640 1 --2- 172.21.15.12:0/3107983053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4880a6aa0 0x7fb488131fd0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb4780097f0 tx=0x7fb4780046c0 comp rx=0 tx=0).stop 2024-09-11T01:44:02.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.080+0000 7fb487fff640 1 -- 172.21.15.12:0/3107983053 reap_dead start 2024-09-11T01:44:02.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.080+0000 7fb48e12b640 1 -- 172.21.15.12:0/3107983053 shutdown_connections 2024-09-11T01:44:02.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.080+0000 7fb48e12b640 1 --2- 172.21.15.12:0/3107983053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4880a6aa0 0x7fb488131fd0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:02.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.080+0000 7fb48e12b640 1 -- 172.21.15.12:0/3107983053 >> 172.21.15.12:0/3107983053 conn(0x7fb4880a7440 msgr2=0x7fb48813f970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:44:02.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.080+0000 7fb48e12b640 1 -- 172.21.15.12:0/3107983053 shutdown_connections 2024-09-11T01:44:02.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:02.080+0000 7fb48e12b640 1 -- 172.21.15.12:0/3107983053 wait complete. 2024-09-11T01:44:02.108 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:44:02.108 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-11T01:44:02.108 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-11T01:44:02.108 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-11T01:44:02.108 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-11T01:44:02.109 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T01:44:02.109 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:44:02.124 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T01:44:02.124 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-11T01:44:02.124 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-11T01:44:02.124 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T01:44:02.185 INFO:tasks.rados.rados.0.smithi071.stdout:2289: finishing write tid 3 to smithi07134777-36 2024-09-11T01:44:02.185 INFO:tasks.rados.rados.0.smithi071.stdout:2289: finishing write tid 4 to smithi07134777-36 2024-09-11T01:44:02.185 INFO:tasks.rados.rados.0.smithi071.stdout:2289: finishing write tid 5 to smithi07134777-36 2024-09-11T01:44:02.186 INFO:tasks.rados.rados.0.smithi071.stdout:2289: oid 36 updating version 0 to 1046 2024-09-11T01:44:02.186 INFO:tasks.rados.rados.0.smithi071.stdout:2289: oid 36 updating version 1046 to 1048 2024-09-11T01:44:02.186 INFO:tasks.rados.rados.0.smithi071.stdout:2289: oid 36 version 1048 is already newer than 1047 2024-09-11T01:44:02.186 INFO:tasks.rados.rados.0.smithi071.stdout:2289: oid 36 version 1048 is already newer than 1045 2024-09-11T01:44:02.186 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 1048 (ObjNum 829 snap 252 seq_num 829) dirty exists 2024-09-11T01:44:02.186 INFO:tasks.rados.rados.0.smithi071.stdout:2289: left oid 36 (ObjNum 829 snap 252 seq_num 829) 2024-09-11T01:44:02.186 INFO:tasks.rados.rados.0.smithi071.stdout:2290: expect (ObjNum 816 snap 250 seq_num 816) 2024-09-11T01:44:02.186 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:44:02.260 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:44:02.388 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3275546 2024-09-11T01:44:17.125 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-11T01:44:17.126 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-11T01:44:17.126 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:44:17.126 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:44:17.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.236+0000 7fa86f859640 10 monclient: get_monmap_and_config 2024-09-11T01:44:17.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.236+0000 7fa86f859640 10 monclient: build_initial_monmap 2024-09-11T01:44:17.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.236+0000 7fa86f859640 10 monclient: monmap: 2024-09-11T01:44:17.239 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:44:17.240 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:17.240 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:44:17.238389+0000 2024-09-11T01:44:17.240 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:44:17.238389+0000 2024-09-11T01:44:17.240 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:44:17.240 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:17.240 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:44:17.240 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:44:17.240 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:44:17.240 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:44:17.240 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:44:17.240 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:17.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.237+0000 7fa86f859640 1 Processor -- start 2024-09-11T01:44:17.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.237+0000 7fa86f859640 1 -- start start 2024-09-11T01:44:17.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.237+0000 7fa86f859640 10 monclient: init 2024-09-11T01:44:17.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.238+0000 7fa86f859640 10 monclient: _reopen_session rank -1 2024-09-11T01:44:17.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.238+0000 7fa86f859640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-11T01:44:17.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.238+0000 7fa86f859640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8680a1380 0x7fa8680a1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:17.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.238+0000 7fa86f859640 10 monclient(hunting): picked mon.noname-b con 0x7fa8680a1380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:44:17.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.238+0000 7fa86f859640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa8680a0c90 addr v1:172.21.15.12:6789/0 2024-09-11T01:44:17.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.238+0000 7fa86f859640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa8681361f0 addr v1:172.21.15.71:6789/0 2024-09-11T01:44:17.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.238+0000 7fa86f859640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8680a2b40 con 0x7fa8680a0c90 2024-09-11T01:44:17.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.238+0000 7fa86f859640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa86809f9a0 con 0x7fa8681361f0 2024-09-11T01:44:17.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.238+0000 7fa86f859640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:17.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.238+0000 7fa86f859640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa8680a2230 con 0x7fa8680a1380 2024-09-11T01:44:17.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.238+0000 7fa86f859640 10 monclient(hunting): _renew_subs 2024-09-11T01:44:17.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.238+0000 7fa86f859640 10 monclient(hunting): authenticate will time out at 1663.681219s 2024-09-11T01:44:17.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.238+0000 7fa86cdf4640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7fa8680a0c90 legacy=0x7fa868135ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:44:17.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.238+0000 7fa86d5f5640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8680a1380 0x7fa8680a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:17.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.238+0000 7fa86d5f5640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8680a1380 0x7fa8680a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45164/0 (socket says 172.21.15.12:45164) 2024-09-11T01:44:17.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.238+0000 7fa86d5f5640 1 -- 172.21.15.12:0/1523612842 learned_addr learned my addr 172.21.15.12:0/1523612842 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:44:17.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.238+0000 7fa86d5f5640 10 monclient(hunting): get_auth_request con 0x7fa8680a1380 auth_method 0 2024-09-11T01:44:17.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.238+0000 7fa86d5f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:17.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.238+0000 7fa86d5f5640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:17.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.238+0000 7fa86d5f5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:17.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.238+0000 7fa86d5f5640 1 -- 172.21.15.12:0/1523612842 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8680a1380 msgr2=0x7fa8680a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:44:17.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.238+0000 7fa86d5f5640 1 -- 172.21.15.12:0/1523612842 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8680a1380 msgr2=0x7fa8680a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:44:17.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.238+0000 7fa86d5f5640 1 --2- 172.21.15.12:0/1523612842 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8680a1380 0x7fa8680a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:44:17.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.238+0000 7fa86d5f5640 1 --2- 172.21.15.12:0/1523612842 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8680a1380 0x7fa8680a1760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:44:17.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.240+0000 7fa86ddf6640 1 -- 172.21.15.12:0/1523612842 >> v1:172.21.15.71:6789/0 conn(0x7fa8681361f0 legacy=0x7fa86813de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:17.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.439+0000 7fa86cdf4640 1 -- 172.21.15.12:0/1523612842 >> v1:172.21.15.12:6789/0 conn(0x7fa8680a0c90 legacy=0x7fa868135ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:44:17.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.439+0000 7fa86d5f5640 1 --2- 172.21.15.12:0/1523612842 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8680a1380 0x7fa8680a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:17.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.439+0000 7fa86d5f5640 10 monclient(hunting): get_auth_request con 0x7fa8680a1380 auth_method 0 2024-09-11T01:44:17.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.439+0000 7fa86d5f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:17.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.439+0000 7fa86d5f5640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:17.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.439+0000 7fa86d5f5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:44:17.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.439+0000 7fa86d5f5640 1 -- 172.21.15.12:0/1523612842 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8680a1380 msgr2=0x7fa8680a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:44:17.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.439+0000 7fa86d5f5640 1 -- 172.21.15.12:0/1523612842 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8680a1380 msgr2=0x7fa8680a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:44:17.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.439+0000 7fa86d5f5640 1 --2- 172.21.15.12:0/1523612842 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8680a1380 0x7fa8680a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:44:17.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.439+0000 7fa86d5f5640 1 --2- 172.21.15.12:0/1523612842 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8680a1380 0x7fa8680a1760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:44:17.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.840+0000 7fa86cdf4640 1 -- 172.21.15.12:0/1523612842 >> v1:172.21.15.12:6789/0 conn(0x7fa8680a0c90 legacy=0x7fa868135ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:44:17.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.840+0000 7fa86d5f5640 1 --2- 172.21.15.12:0/1523612842 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8680a1380 0x7fa8680a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:17.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.841+0000 7fa86d5f5640 10 monclient(hunting): get_auth_request con 0x7fa8680a1380 auth_method 0 2024-09-11T01:44:17.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.841+0000 7fa86d5f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:17.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.841+0000 7fa86d5f5640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:17.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.841+0000 7fa86d5f5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:44:17.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.841+0000 7fa86d5f5640 1 -- 172.21.15.12:0/1523612842 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8680a1380 msgr2=0x7fa8680a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:44:17.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.841+0000 7fa86d5f5640 1 -- 172.21.15.12:0/1523612842 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8680a1380 msgr2=0x7fa8680a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:44:17.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.841+0000 7fa86d5f5640 1 --2- 172.21.15.12:0/1523612842 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8680a1380 0x7fa8680a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:44:17.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:17.841+0000 7fa86d5f5640 1 --2- 172.21.15.12:0/1523612842 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8680a1380 0x7fa8680a1760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:44:18.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:18.642+0000 7fa86cdf4640 1 -- 172.21.15.12:0/1523612842 >> v1:172.21.15.12:6789/0 conn(0x7fa8680a0c90 legacy=0x7fa868135ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:44:18.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:18.642+0000 7fa86d5f5640 1 --2- 172.21.15.12:0/1523612842 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8680a1380 0x7fa8680a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:18.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:18.642+0000 7fa86d5f5640 10 monclient(hunting): get_auth_request con 0x7fa8680a1380 auth_method 0 2024-09-11T01:44:18.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:18.642+0000 7fa86d5f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:18.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:18.642+0000 7fa86d5f5640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:18.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:18.642+0000 7fa86d5f5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:44:18.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:18.642+0000 7fa86d5f5640 1 -- 172.21.15.12:0/1523612842 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8680a1380 msgr2=0x7fa8680a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:44:18.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:18.642+0000 7fa86d5f5640 1 -- 172.21.15.12:0/1523612842 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8680a1380 msgr2=0x7fa8680a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:44:18.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:18.642+0000 7fa86d5f5640 1 --2- 172.21.15.12:0/1523612842 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8680a1380 0x7fa8680a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:44:18.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:18.642+0000 7fa86d5f5640 1 --2- 172.21.15.12:0/1523612842 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8680a1380 0x7fa8680a1760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:44:20.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa85effd640 10 monclient(hunting): tick 2024-09-11T01:44:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa85effd640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:44:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa85effd640 1 monclient(hunting): continuing hunt 2024-09-11T01:44:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa85effd640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:44:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa85effd640 1 -- 172.21.15.12:0/1523612842 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8680a1380 msgr2=0x7fa8680a1760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:44:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa85effd640 1 --2- 172.21.15.12:0/1523612842 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8680a1380 0x7fa8680a1760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa85effd640 1 -- 172.21.15.12:0/1523612842 >> v1:172.21.15.71:6789/0 conn(0x7fa8681361f0 legacy=0x7fa86813de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa85effd640 1 -- 172.21.15.12:0/1523612842 >> v1:172.21.15.12:6789/0 conn(0x7fa8680a0c90 legacy=0x7fa868135ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:44:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa85effd640 10 monclient: _add_conns ranks=[3,4] 2024-09-11T01:44:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa85effd640 1 --2- 172.21.15.12:0/1523612842 >> v2:172.21.15.12:3300/0 conn(0x7fa84c003770 0x7fa84c005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa85effd640 10 monclient(hunting): picked mon.noname-a con 0x7fa84c003770 addr v2:172.21.15.12:3300/0 2024-09-11T01:44:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa85effd640 1 --2- 172.21.15.12:0/1523612842 >> v2:172.21.15.71:3300/0 conn(0x7fa84c0062c0 0x7fa84c008780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa85effd640 10 monclient(hunting): picked mon.noname-c con 0x7fa84c0062c0 addr v2:172.21.15.71:3300/0 2024-09-11T01:44:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa85effd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:20.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa85effd640 1 -- 172.21.15.12:0/1523612842 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fa8680a2230 con 0x7fa84c003770 2024-09-11T01:44:20.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa85effd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:20.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa85effd640 1 -- 172.21.15.12:0/1523612842 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fa84c008e40 con 0x7fa84c0062c0 2024-09-11T01:44:20.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa85effd640 10 monclient(hunting): _renew_subs 2024-09-11T01:44:20.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa86d5f5640 1 -- 172.21.15.12:0/1523612842 >> v2:172.21.15.12:3300/0 conn(0x7fa84c003770 msgr2=0x7fa84c005bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:44:20.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa86d5f5640 1 --2- 172.21.15.12:0/1523612842 >> v2:172.21.15.12:3300/0 conn(0x7fa84c003770 0x7fa84c005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:44:20.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa86cdf4640 1 --2- 172.21.15.12:0/1523612842 >> v2:172.21.15.71:3300/0 conn(0x7fa84c0062c0 0x7fa84c008780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:20.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa86cdf4640 10 monclient(hunting): get_auth_request con 0x7fa84c0062c0 auth_method 0 2024-09-11T01:44:20.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa86cdf4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:20.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa86cdf4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:20.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa86cdf4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:20.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa86cdf4640 1 -- 172.21.15.12:0/1523612842 >> v2:172.21.15.71:3300/0 conn(0x7fa84c0062c0 msgr2=0x7fa84c008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:44:20.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa86cdf4640 1 -- 172.21.15.12:0/1523612842 >> v2:172.21.15.71:3300/0 conn(0x7fa84c0062c0 msgr2=0x7fa84c008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:44:20.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa86cdf4640 1 --2- 172.21.15.12:0/1523612842 >> v2:172.21.15.71:3300/0 conn(0x7fa84c0062c0 0x7fa84c008780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:44:20.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.239+0000 7fa86cdf4640 1 --2- 172.21.15.12:0/1523612842 >> v2:172.21.15.71:3300/0 conn(0x7fa84c0062c0 0x7fa84c008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:44:20.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.440+0000 7fa86cdf4640 1 --2- 172.21.15.12:0/1523612842 >> v2:172.21.15.71:3300/0 conn(0x7fa84c0062c0 0x7fa84c008780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:20.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.440+0000 7fa86cdf4640 10 monclient(hunting): get_auth_request con 0x7fa84c0062c0 auth_method 0 2024-09-11T01:44:20.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.440+0000 7fa86cdf4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:20.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.440+0000 7fa86cdf4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:20.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.440+0000 7fa86cdf4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:44:20.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.440+0000 7fa86cdf4640 1 -- 172.21.15.12:0/1523612842 >> v2:172.21.15.71:3300/0 conn(0x7fa84c0062c0 msgr2=0x7fa84c008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:44:20.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.440+0000 7fa86cdf4640 1 -- 172.21.15.12:0/1523612842 >> v2:172.21.15.71:3300/0 conn(0x7fa84c0062c0 msgr2=0x7fa84c008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:44:20.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.440+0000 7fa86cdf4640 1 --2- 172.21.15.12:0/1523612842 >> v2:172.21.15.71:3300/0 conn(0x7fa84c0062c0 0x7fa84c008780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:44:20.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.440+0000 7fa86cdf4640 1 --2- 172.21.15.12:0/1523612842 >> v2:172.21.15.71:3300/0 conn(0x7fa84c0062c0 0x7fa84c008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:44:20.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.440+0000 7fa86d5f5640 1 -- 172.21.15.12:0/1523612842 >> v2:172.21.15.12:3300/0 conn(0x7fa84c003770 msgr2=0x7fa84c005bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:44:20.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.440+0000 7fa86d5f5640 1 --2- 172.21.15.12:0/1523612842 >> v2:172.21.15.12:3300/0 conn(0x7fa84c003770 0x7fa84c005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:44:20.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.842+0000 7fa86cdf4640 1 --2- 172.21.15.12:0/1523612842 >> v2:172.21.15.71:3300/0 conn(0x7fa84c0062c0 0x7fa84c008780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:20.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.842+0000 7fa86d5f5640 1 -- 172.21.15.12:0/1523612842 >> v2:172.21.15.12:3300/0 conn(0x7fa84c003770 msgr2=0x7fa84c005bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:44:20.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.842+0000 7fa86cdf4640 10 monclient(hunting): get_auth_request con 0x7fa84c0062c0 auth_method 0 2024-09-11T01:44:20.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.842+0000 7fa86cdf4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:20.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.842+0000 7fa86cdf4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:20.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.842+0000 7fa86cdf4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:44:20.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.842+0000 7fa86d5f5640 1 --2- 172.21.15.12:0/1523612842 >> v2:172.21.15.12:3300/0 conn(0x7fa84c003770 0x7fa84c005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:44:20.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.842+0000 7fa86cdf4640 1 -- 172.21.15.12:0/1523612842 >> v2:172.21.15.71:3300/0 conn(0x7fa84c0062c0 msgr2=0x7fa84c008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:44:20.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.842+0000 7fa86cdf4640 1 -- 172.21.15.12:0/1523612842 >> v2:172.21.15.71:3300/0 conn(0x7fa84c0062c0 msgr2=0x7fa84c008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:44:20.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.842+0000 7fa86cdf4640 1 --2- 172.21.15.12:0/1523612842 >> v2:172.21.15.71:3300/0 conn(0x7fa84c0062c0 0x7fa84c008780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:44:20.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:20.842+0000 7fa86cdf4640 1 --2- 172.21.15.12:0/1523612842 >> v2:172.21.15.71:3300/0 conn(0x7fa84c0062c0 0x7fa84c008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:44:21.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:21.643+0000 7fa86d5f5640 1 -- 172.21.15.12:0/1523612842 >> v2:172.21.15.12:3300/0 conn(0x7fa84c003770 msgr2=0x7fa84c005bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:44:21.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:21.643+0000 7fa86d5f5640 1 --2- 172.21.15.12:0/1523612842 >> v2:172.21.15.12:3300/0 conn(0x7fa84c003770 0x7fa84c005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:44:21.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:21.643+0000 7fa86cdf4640 1 --2- 172.21.15.12:0/1523612842 >> v2:172.21.15.71:3300/0 conn(0x7fa84c0062c0 0x7fa84c008780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:21.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:21.643+0000 7fa86cdf4640 10 monclient(hunting): get_auth_request con 0x7fa84c0062c0 auth_method 0 2024-09-11T01:44:21.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:21.643+0000 7fa86cdf4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:21.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:21.643+0000 7fa86cdf4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:21.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:21.643+0000 7fa86cdf4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:44:21.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:21.643+0000 7fa86cdf4640 1 -- 172.21.15.12:0/1523612842 >> v2:172.21.15.71:3300/0 conn(0x7fa84c0062c0 msgr2=0x7fa84c008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:44:21.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:21.643+0000 7fa86cdf4640 1 -- 172.21.15.12:0/1523612842 >> v2:172.21.15.71:3300/0 conn(0x7fa84c0062c0 msgr2=0x7fa84c008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:44:21.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:21.643+0000 7fa86cdf4640 1 --2- 172.21.15.12:0/1523612842 >> v2:172.21.15.71:3300/0 conn(0x7fa84c0062c0 0x7fa84c008780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:44:21.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:21.643+0000 7fa86cdf4640 1 --2- 172.21.15.12:0/1523612842 >> v2:172.21.15.71:3300/0 conn(0x7fa84c0062c0 0x7fa84c008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:44:23.140 INFO:tasks.rados.rados.0.smithi071.stdout:2289: done (3 left) 2024-09-11T01:44:23.140 INFO:tasks.rados.rados.0.smithi071.stdout:2290: done (2 left) 2024-09-11T01:44:23.140 INFO:tasks.rados.rados.0.smithi071.stdout:2291: done (1 left) 2024-09-11T01:44:23.140 INFO:tasks.rados.rados.0.smithi071.stdout:2294: done (0 left) 2024-09-11T01:44:23.140 INFO:tasks.rados.rados.0.smithi071.stdout:2295: read oid 24 snap -1 2024-09-11T01:44:23.140 INFO:tasks.rados.rados.0.smithi071.stdout:2295: expect (ObjNum 766 snap 232 seq_num 766) 2024-09-11T01:44:23.141 INFO:tasks.rados.rados.0.smithi071.stdout:2296: rollback oid 25 current snap is 253 2024-09-11T01:44:23.141 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 25 to 212 2024-09-11T01:44:23.141 INFO:tasks.rados.rados.0.smithi071.stdout:2297: delete oid 15 current snap is 253 2024-09-11T01:44:23.146 INFO:tasks.rados.rados.0.smithi071.stdout:2297: done (2 left) 2024-09-11T01:44:23.146 INFO:tasks.rados.rados.0.smithi071.stdout:2298: read oid 14 snap -1 2024-09-11T01:44:23.146 INFO:tasks.rados.rados.0.smithi071.stdout:2298: expect (ObjNum 797 snap 243 seq_num 797) 2024-09-11T01:44:23.146 INFO:tasks.rados.rados.0.smithi071.stdout:2299: snap_create 2024-09-11T01:44:23.147 INFO:tasks.rados.rados.0.smithi071.stdout:2296: finishing rollback tid 1 to smithi07134777-25 2024-09-11T01:44:23.147 INFO:tasks.rados.rados.0.smithi071.stdout:2296: finishing rollback tid 2 to smithi07134777-25 2024-09-11T01:44:23.147 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 992 (ObjNum 651 snap 199 seq_num 651) dirty exists 2024-09-11T01:44:23.151 INFO:tasks.rados.rados.0.smithi071.stdout:2298: expect (ObjNum 797 snap 243 seq_num 797) 2024-09-11T01:44:23.151 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:44:23.217 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1610319 2024-09-11T01:44:23.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.238+0000 7fa85effd640 10 monclient(hunting): tick 2024-09-11T01:44:23.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.238+0000 7fa85effd640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:44:23.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.238+0000 7fa85effd640 1 monclient(hunting): continuing hunt 2024-09-11T01:44:23.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.238+0000 7fa85effd640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:44:23.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.238+0000 7fa85effd640 1 -- 172.21.15.12:0/1523612842 >> v2:172.21.15.71:3300/0 conn(0x7fa84c0062c0 msgr2=0x7fa84c008780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:44:23.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.238+0000 7fa85effd640 1 --2- 172.21.15.12:0/1523612842 >> v2:172.21.15.71:3300/0 conn(0x7fa84c0062c0 0x7fa84c008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:23.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.238+0000 7fa85effd640 1 -- 172.21.15.12:0/1523612842 >> v2:172.21.15.12:3300/0 conn(0x7fa84c003770 msgr2=0x7fa84c005bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:44:23.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.238+0000 7fa85effd640 1 --2- 172.21.15.12:0/1523612842 >> v2:172.21.15.12:3300/0 conn(0x7fa84c003770 0x7fa84c005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:23.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.238+0000 7fa85effd640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-11T01:44:23.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.238+0000 7fa85effd640 1 --2- 172.21.15.12:0/1523612842 >> v2:172.21.15.71:3300/0 conn(0x7fa84c009310 0x7fa84c0096f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:23.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.238+0000 7fa85effd640 10 monclient(hunting): picked mon.noname-c con 0x7fa84c009310 addr v2:172.21.15.71:3300/0 2024-09-11T01:44:23.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.238+0000 7fa86ddf6640 1 -- 172.21.15.12:0/1523612842 reap_dead start 2024-09-11T01:44:23.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.238+0000 7fa85effd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa8681361f0 addr v1:172.21.15.71:6789/0 2024-09-11T01:44:23.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.238+0000 7fa85effd640 1 --2- 172.21.15.12:0/1523612842 >> v2:172.21.15.12:3300/0 conn(0x7fa84c003770 0x7fa84c005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:23.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.238+0000 7fa85effd640 10 monclient(hunting): picked mon.noname-a con 0x7fa84c003770 addr v2:172.21.15.12:3300/0 2024-09-11T01:44:23.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.238+0000 7fa85effd640 1 -- 172.21.15.12:0/1523612842 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8680a2510 con 0x7fa8681361f0 2024-09-11T01:44:23.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.238+0000 7fa85effd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:23.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.238+0000 7fa85effd640 1 -- 172.21.15.12:0/1523612842 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fa8680a2230 con 0x7fa84c003770 2024-09-11T01:44:23.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.238+0000 7fa85effd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:23.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.238+0000 7fa85effd640 1 -- 172.21.15.12:0/1523612842 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fa84c008e40 con 0x7fa84c009310 2024-09-11T01:44:23.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.238+0000 7fa85effd640 10 monclient(hunting): _renew_subs 2024-09-11T01:44:23.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.238+0000 7fa86ddf6640 1 -- 172.21.15.12:0/1523612842 >> v2:172.21.15.12:3300/0 conn(0x7fa84c003770 msgr2=0x7fa84c005bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:44:23.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.238+0000 7fa86ddf6640 1 --2- 172.21.15.12:0/1523612842 >> v2:172.21.15.12:3300/0 conn(0x7fa84c003770 0x7fa84c005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:44:23.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.238+0000 7fa86d5f5640 1 --2- 172.21.15.12:0/1523612842 >> v2:172.21.15.71:3300/0 conn(0x7fa84c009310 0x7fa84c0096f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:23.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.239+0000 7fa86d5f5640 10 monclient(hunting): get_auth_request con 0x7fa84c009310 auth_method 0 2024-09-11T01:44:23.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.239+0000 7fa86d5f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:23.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.239+0000 7fa86d5f5640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:23.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.239+0000 7fa86d5f5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:23.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.239+0000 7fa86d5f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:23.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.239+0000 7fa86d5f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:23.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.239+0000 7fa86d5f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:23.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.241+0000 7fa86cdf4640 1 -- 172.21.15.12:0/1523612842 >> v1:172.21.15.71:6789/0 conn(0x7fa8681361f0 legacy=0x7fa84c00bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:23.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.241+0000 7fa86d5f5640 10 monclient(hunting): handle_auth_done global_id 134413 payload 274 2024-09-11T01:44:23.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.241+0000 7fa86d5f5640 1 -- 172.21.15.12:0/1523612842 >> v2:172.21.15.12:3300/0 conn(0x7fa84c003770 msgr2=0x7fa84c005bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:44:23.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.241+0000 7fa86d5f5640 1 --2- 172.21.15.12:0/1523612842 >> v2:172.21.15.12:3300/0 conn(0x7fa84c003770 0x7fa84c005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:23.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.241+0000 7fa86d5f5640 1 -- 172.21.15.12:0/1523612842 >> v1:172.21.15.71:6789/0 conn(0x7fa8681361f0 legacy=0x7fa84c00bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:23.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.241+0000 7fa86d5f5640 10 monclient: _finish_hunting 0 2024-09-11T01:44:23.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.241+0000 7fa86d5f5640 1 monclient: found mon.noname-c 2024-09-11T01:44:23.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.241+0000 7fa86d5f5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:44:23.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.241+0000 7fa86d5f5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:44:23.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.241+0000 7fa86d5f5640 1 -- 172.21.15.12:0/1523612842 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8680a2b40 con 0x7fa84c009310 2024-09-11T01:44:23.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.244+0000 7fa86d5f5640 1 -- 172.21.15.12:0/1523612842 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa84c009310 msgr2=0x7fa84c0096f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:23.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.244+0000 7fa86d5f5640 1 --2- 172.21.15.12:0/1523612842 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa84c009310 0x7fa84c0096f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa86806ece0 tx=0x7fa86402af70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=834809afcfc9ed6f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:23.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.244+0000 7fa85ffff640 1 -- 172.21.15.12:0/1523612842 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa864031070 con 0x7fa84c009310 2024-09-11T01:44:23.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.244+0000 7fa85ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:23.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.244+0000 7fa85ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:44:23.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.244+0000 7fa85ffff640 10 monclient: dump: 2024-09-11T01:44:23.248 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:23.248 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:23.248 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.248 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.248 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:23.248 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:23.248 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:23.248 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:23.248 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:23.248 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:23.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.244+0000 7fa85ffff640 10 monclient: _finish_auth 0 2024-09-11T01:44:23.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.244+0000 7fa85ffff640 10 monclient: _check_auth_tickets 2024-09-11T01:44:23.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.244+0000 7fa85ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:44:23.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.244+0000 7fa85ffff640 1 -- 172.21.15.12:0/1523612842 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa864035400 con 0x7fa84c009310 2024-09-11T01:44:23.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.244+0000 7fa85ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:44:23.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.244+0000 7fa85ffff640 1 -- 172.21.15.12:0/1523612842 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86400f040 con 0x7fa84c009310 2024-09-11T01:44:23.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.244+0000 7fa86f859640 5 monclient: authenticate success, global_id 134413 2024-09-11T01:44:23.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.244+0000 7fa86f859640 10 monclient: get_monmap_and_config success 2024-09-11T01:44:23.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.244+0000 7fa86f859640 4 set_mon_vals no callback set 2024-09-11T01:44:23.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.244+0000 7fa85ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:23.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.244+0000 7fa85ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:44:23.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.244+0000 7fa85ffff640 10 monclient: dump: 2024-09-11T01:44:23.249 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:23.249 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:23.249 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.250 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.250 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:23.250 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:23.250 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:23.250 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:23.250 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:23.250 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:23.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.245+0000 7fa86f859640 10 monclient: shutdown 2024-09-11T01:44:23.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.245+0000 7fa86f859640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:44:23.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.245+0000 7fa86f859640 1 -- 172.21.15.12:0/1523612842 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa84c009310 msgr2=0x7fa84c0096f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:23.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.245+0000 7fa86f859640 1 --2- 172.21.15.12:0/1523612842 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa84c009310 0x7fa84c0096f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa86806ece0 tx=0x7fa86402af70 comp rx=0 tx=0).stop 2024-09-11T01:44:23.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.245+0000 7fa86f859640 1 -- 172.21.15.12:0/1523612842 shutdown_connections 2024-09-11T01:44:23.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.245+0000 7fa86f859640 1 --2- 172.21.15.12:0/1523612842 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa84c009310 0x7fa84c0096f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:23.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.245+0000 7fa86f859640 1 --2- 172.21.15.12:0/1523612842 >> v2:172.21.15.12:3300/0 conn(0x7fa84c003770 0x7fa84c005bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:23.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.245+0000 7fa86f859640 1 -- 172.21.15.12:0/1523612842 >> 172.21.15.12:0/1523612842 conn(0x7fa8680a2d90 msgr2=0x7fa8680a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:44:23.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.245+0000 7fa86f859640 1 -- 172.21.15.12:0/1523612842 shutdown_connections 2024-09-11T01:44:23.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.245+0000 7fa86f859640 1 -- 172.21.15.12:0/1523612842 wait complete. 2024-09-11T01:44:23.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.246+0000 7fa86f859640 10 monclient: build_initial_monmap 2024-09-11T01:44:23.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.246+0000 7fa86f859640 10 monclient: monmap: 2024-09-11T01:44:23.257 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:44:23.257 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:44:23.257 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:44:23.257 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:44:23.257 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:44:23.257 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:23.257 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:44:23.257 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:44:23.257 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:44:23.257 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:23.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.246+0000 7fa86f859640 1 Processor -- start 2024-09-11T01:44:23.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86f859640 1 -- start start 2024-09-11T01:44:23.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86f859640 10 monclient: init 2024-09-11T01:44:23.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86f859640 10 monclient: _reopen_session rank -1 2024-09-11T01:44:23.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86f859640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:44:23.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86f859640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa84c003770 0x7fa8681cad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:23.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86f859640 10 monclient(hunting): picked mon.noname-a con 0x7fa84c003770 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:44:23.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86f859640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa84c009310 0x7fa868134e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:23.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86f859640 10 monclient(hunting): picked mon.noname-b con 0x7fa84c009310 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:44:23.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86f859640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8681361f0 0x7fa868122840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:23.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86f859640 10 monclient(hunting): picked mon.noname-c con 0x7fa8681361f0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:44:23.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86f859640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:23.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86f859640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa8680a0510 con 0x7fa84c003770 2024-09-11T01:44:23.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86f859640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:23.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86f859640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa8680a0390 con 0x7fa84c009310 2024-09-11T01:44:23.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86f859640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:23.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86f859640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa868136640 con 0x7fa8681361f0 2024-09-11T01:44:23.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86f859640 10 monclient(hunting): _renew_subs 2024-09-11T01:44:23.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86d5f5640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa84c003770 msgr2=0x7fa8681cad20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:44:23.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86f859640 10 monclient(hunting): authenticate will time out at 1669.690687s 2024-09-11T01:44:23.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86d5f5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa84c003770 0x7fa8681cad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:44:23.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86cdf4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa84c009310 0x7fa868134e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:23.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86ddf6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8681361f0 0x7fa868122840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:23.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86cdf4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa84c009310 0x7fa868134e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:52296/0 (socket says 172.21.15.12:52296) 2024-09-11T01:44:23.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86cdf4640 1 -- 172.21.15.12:0/1721048684 learned_addr learned my addr 172.21.15.12:0/1721048684 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:44:23.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86cdf4640 10 monclient(hunting): get_auth_request con 0x7fa84c009310 auth_method 0 2024-09-11T01:44:23.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86cdf4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:23.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86cdf4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:23.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86cdf4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:23.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86ddf6640 10 monclient(hunting): get_auth_request con 0x7fa8681361f0 auth_method 0 2024-09-11T01:44:23.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86ddf6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:23.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86ddf6640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:23.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86ddf6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:23.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86cdf4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:23.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.247+0000 7fa86cdf4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:23.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.248+0000 7fa86cdf4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:23.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.248+0000 7fa86ddf6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:23.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.248+0000 7fa86ddf6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:23.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.248+0000 7fa86ddf6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:23.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.248+0000 7fa86cdf4640 10 monclient(hunting): handle_auth_done global_id 134419 payload 824 2024-09-11T01:44:23.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.248+0000 7fa86cdf4640 1 -- 172.21.15.12:0/1721048684 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8681361f0 msgr2=0x7fa868122840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:23.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.248+0000 7fa86cdf4640 1 --2- 172.21.15.12:0/1721048684 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8681361f0 0x7fa868122840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:23.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.248+0000 7fa86cdf4640 1 -- 172.21.15.12:0/1721048684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa84c003770 msgr2=0x7fa8681cad20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:44:23.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.248+0000 7fa86cdf4640 1 --2- 172.21.15.12:0/1721048684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa84c003770 0x7fa8681cad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:23.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.248+0000 7fa86cdf4640 10 monclient: _finish_hunting 0 2024-09-11T01:44:23.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.248+0000 7fa86cdf4640 1 monclient: found mon.noname-b 2024-09-11T01:44:23.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.248+0000 7fa86cdf4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:44:23.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.248+0000 7fa86cdf4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-09-11T01:44:23.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.248+0000 7fa86cdf4640 1 -- 172.21.15.12:0/1721048684 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa868124300 con 0x7fa84c009310 2024-09-11T01:44:23.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.248+0000 7fa86ddf6640 1 --2- 172.21.15.12:0/1721048684 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8681361f0 0x7fa868122840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:44:23.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.250+0000 7fa86cdf4640 1 -- 172.21.15.12:0/1721048684 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa84c009310 msgr2=0x7fa868134e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:23.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.250+0000 7fa86cdf4640 1 --2- 172.21.15.12:0/1721048684 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa84c009310 0x7fa868134e90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa85800ba70 tx=0x7fa85800bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:23.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.250+0000 7fa85dffb640 1 -- 172.21.15.12:0/1721048684 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa858017680 con 0x7fa84c009310 2024-09-11T01:44:23.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.250+0000 7fa85dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:23.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.250+0000 7fa85dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:44:23.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.250+0000 7fa85dffb640 10 monclient: dump: 2024-09-11T01:44:23.262 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:23.262 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:23.262 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.262 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.262 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:23.262 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:23.262 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:23.262 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:23.263 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:23.263 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:23.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.250+0000 7fa85dffb640 10 monclient: _finish_auth 0 2024-09-11T01:44:23.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.250+0000 7fa85dffb640 10 monclient: _check_auth_tickets 2024-09-11T01:44:23.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.250+0000 7fa85dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:44:23.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.250+0000 7fa85dffb640 1 -- 172.21.15.12:0/1721048684 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa85801c070 con 0x7fa84c009310 2024-09-11T01:44:23.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.250+0000 7fa85dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:44:23.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.250+0000 7fa85dffb640 1 -- 172.21.15.12:0/1721048684 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa858017a60 con 0x7fa84c009310 2024-09-11T01:44:23.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.250+0000 7fa85dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:23.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.250+0000 7fa85dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:44:23.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.250+0000 7fa85dffb640 10 monclient: dump: 2024-09-11T01:44:23.264 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:23.264 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:23.264 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.264 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.264 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:23.264 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:23.264 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:23.264 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:23.264 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:23.264 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:23.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.250+0000 7fa86f859640 5 monclient: authenticate success, global_id 134419 2024-09-11T01:44:23.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.250+0000 7fa86f859640 10 monclient: _renew_subs 2024-09-11T01:44:23.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.250+0000 7fa86f859640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:44:23.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.250+0000 7fa86f859640 1 -- 172.21.15.12:0/1721048684 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa86809d480 con 0x7fa84c009310 2024-09-11T01:44:23.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.250+0000 7fa85f7fe640 4 set_mon_vals no callback set 2024-09-11T01:44:23.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.251+0000 7fa86f859640 10 monclient: _renew_subs 2024-09-11T01:44:23.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.251+0000 7fa86f859640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:44:23.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.251+0000 7fa86f859640 1 -- 172.21.15.12:0/1721048684 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa868125dc0 con 0x7fa84c009310 2024-09-11T01:44:23.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.252+0000 7fa85dffb640 1 -- 172.21.15.12:0/1721048684 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fa858003ce0 con 0x7fa84c009310 2024-09-11T01:44:23.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.252+0000 7fa86f859640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:44:23.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.252+0000 7fa86f859640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:44:23.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.252+0000 7fa86f859640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:44:23.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.252+0000 7fa86f859640 1 -- 172.21.15.12:0/1721048684 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa868135ae0 con 0x7fa84c009310 2024-09-11T01:44:23.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.252+0000 7fa85dffb640 1 --2- 172.21.15.12:0/1721048684 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa84c03f6c0 0x7fa84c041b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:23.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.252+0000 7fa85dffb640 1 -- 172.21.15.12:0/1721048684 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(511..511 src has 317..511) ==== 4819+0+0 (secure 0 0 0) 0x7fa858053850 con 0x7fa84c009310 2024-09-11T01:44:23.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.252+0000 7fa86d5f5640 1 --2- 172.21.15.12:0/1721048684 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa84c03f6c0 0x7fa84c041b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:23.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.252+0000 7fa86d5f5640 10 monclient: get_auth_request con 0x7fa84c03f6c0 auth_method 0 2024-09-11T01:44:23.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.255+0000 7fa86d5f5640 1 --2- 172.21.15.12:0/1721048684 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa84c03f6c0 0x7fa84c041b80 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa864033000 tx=0x7fa864010040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:23.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.257+0000 7fa85dffb640 1 -- 172.21.15.12:0/1721048684 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa858012050 con 0x7fa84c009310 2024-09-11T01:44:23.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.257+0000 7fa85dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:44:23.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.257+0000 7fa85dffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:44:23.302 INFO:tasks.rados.rados.0.smithi071.stdout:2295: expect (ObjNum 766 snap 232 seq_num 766) 2024-09-11T01:44:23.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.518+0000 7fa86f859640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:44:23.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.518+0000 7fa86f859640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:44:23.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.518+0000 7fa86f859640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:44:23.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.518+0000 7fa86f859640 1 -- 172.21.15.12:0/1721048684 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa868124e60 con 0x7fa84c009310 2024-09-11T01:44:23.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.519+0000 7fa85dffb640 1 -- 172.21.15.12:0/1721048684 <== mon.1 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fa858005e20 con 0x7fa84c009310 2024-09-11T01:44:23.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.519+0000 7fa85dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:44:23.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.519+0000 7fa85dffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:44:23.522 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":140,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:44:23.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.521+0000 7fa86f859640 1 -- 172.21.15.12:0/1721048684 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa84c03f6c0 msgr2=0x7fa84c041b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:23.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.521+0000 7fa86f859640 1 --2- 172.21.15.12:0/1721048684 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa84c03f6c0 0x7fa84c041b80 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa864033000 tx=0x7fa864010040 comp rx=0 tx=0).stop 2024-09-11T01:44:23.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.521+0000 7fa86f859640 10 monclient: shutdown 2024-09-11T01:44:23.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.521+0000 7fa86f859640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:44:23.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.521+0000 7fa86f859640 1 -- 172.21.15.12:0/1721048684 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa84c009310 msgr2=0x7fa868134e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:23.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.521+0000 7fa86f859640 1 --2- 172.21.15.12:0/1721048684 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa84c009310 0x7fa868134e90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa85800ba70 tx=0x7fa85800bf30 comp rx=0 tx=0).stop 2024-09-11T01:44:23.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.521+0000 7fa86f859640 1 -- 172.21.15.12:0/1721048684 shutdown_connections 2024-09-11T01:44:23.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.521+0000 7fa86f859640 1 --2- 172.21.15.12:0/1721048684 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8681361f0 0x7fa868122840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:23.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.521+0000 7fa86f859640 1 --2- 172.21.15.12:0/1721048684 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa84c009310 0x7fa868134e90 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:23.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.521+0000 7fa86f859640 1 --2- 172.21.15.12:0/1721048684 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa84c03f6c0 0x7fa84c041b80 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:23.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.521+0000 7fa86f859640 1 --2- 172.21.15.12:0/1721048684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa84c003770 0x7fa8681cad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:23.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.521+0000 7fa86f859640 1 -- 172.21.15.12:0/1721048684 >> 172.21.15.12:0/1721048684 conn(0x7fa8680a2d90 msgr2=0x7fa86811c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:44:23.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.521+0000 7fa86f859640 1 -- 172.21.15.12:0/1721048684 shutdown_connections 2024-09-11T01:44:23.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.521+0000 7fa86f859640 1 -- 172.21.15.12:0/1721048684 wait complete. 2024-09-11T01:44:23.534 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:44:23.534 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T01:44:23.534 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:44:23.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.633+0000 7fddfafcd640 10 monclient: get_monmap_and_config 2024-09-11T01:44:23.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.633+0000 7fddfafcd640 10 monclient: build_initial_monmap 2024-09-11T01:44:23.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.633+0000 7fddfafcd640 10 monclient: monmap: 2024-09-11T01:44:23.636 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:44:23.636 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:23.636 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:44:23.635402+0000 2024-09-11T01:44:23.636 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:44:23.635402+0000 2024-09-11T01:44:23.636 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:44:23.636 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:23.637 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:44:23.637 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:44:23.637 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:44:23.637 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:44:23.637 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:44:23.637 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:23.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.634+0000 7fddfafcd640 1 Processor -- start 2024-09-11T01:44:23.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.634+0000 7fddfafcd640 1 -- start start 2024-09-11T01:44:23.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.634+0000 7fddfafcd640 10 monclient: init 2024-09-11T01:44:23.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.634+0000 7fddfafcd640 10 monclient: _reopen_session rank -1 2024-09-11T01:44:23.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.634+0000 7fddfafcd640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-11T01:44:23.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.634+0000 7fddfafcd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fddf40a2e20 0x7fddf406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:23.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.634+0000 7fddfafcd640 10 monclient(hunting): picked mon.noname-c con 0x7fddf40a2e20 addr v2:172.21.15.71:3300/0 2024-09-11T01:44:23.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.634+0000 7fddfafcd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fddf40a4970 addr v1:172.21.15.12:6789/0 2024-09-11T01:44:23.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.634+0000 7fddfafcd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf406f910 0x7fddf413f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:23.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.634+0000 7fddfafcd640 10 monclient(hunting): picked mon.noname-b con 0x7fddf406f910 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:44:23.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.634+0000 7fddfafcd640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddf406f4f0 con 0x7fddf40a4970 2024-09-11T01:44:23.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.634+0000 7fddfafcd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:23.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.634+0000 7fddfafcd640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fddf406ef30 con 0x7fddf40a2e20 2024-09-11T01:44:23.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.634+0000 7fddfafcd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:23.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.634+0000 7fddfafcd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fddf406ec50 con 0x7fddf406f910 2024-09-11T01:44:23.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.634+0000 7fddfafcd640 10 monclient(hunting): _renew_subs 2024-09-11T01:44:23.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.634+0000 7fddfafcd640 10 monclient(hunting): authenticate will time out at 1670.077969s 2024-09-11T01:44:23.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.634+0000 7fddf3fff640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7fddf40a4970 legacy=0x7fddf40a3200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:44:23.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf956a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf406f910 0x7fddf413f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:23.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf956a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf406f910 0x7fddf413f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45210/0 (socket says 172.21.15.12:45210) 2024-09-11T01:44:23.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf956a640 1 -- 172.21.15.12:0/4099326292 learned_addr learned my addr 172.21.15.12:0/4099326292 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:44:23.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf956a640 10 monclient(hunting): get_auth_request con 0x7fddf406f910 auth_method 0 2024-09-11T01:44:23.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf956a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:23.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf956a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:23.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf956a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:23.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf8d69640 1 --2- 172.21.15.12:0/4099326292 >> v2:172.21.15.71:3300/0 conn(0x7fddf40a2e20 0x7fddf406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:23.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf956a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:23.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf956a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:23.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf956a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:23.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf8d69640 10 monclient(hunting): get_auth_request con 0x7fddf40a2e20 auth_method 0 2024-09-11T01:44:23.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf8d69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:23.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf8d69640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:23.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf8d69640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:23.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf8d69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:23.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf8d69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:23.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf8d69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:23.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf956a640 10 monclient(hunting): handle_auth_done global_id 164141 payload 274 2024-09-11T01:44:23.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf956a640 1 -- 172.21.15.12:0/4099326292 >> v2:172.21.15.71:3300/0 conn(0x7fddf40a2e20 msgr2=0x7fddf406ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:44:23.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf956a640 1 --2- 172.21.15.12:0/4099326292 >> v2:172.21.15.71:3300/0 conn(0x7fddf40a2e20 0x7fddf406ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:23.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf956a640 1 -- 172.21.15.12:0/4099326292 >> v1:172.21.15.12:6789/0 conn(0x7fddf40a4970 legacy=0x7fddf40a3200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:44:23.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf956a640 10 monclient: _finish_hunting 0 2024-09-11T01:44:23.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf956a640 1 monclient: found mon.noname-b 2024-09-11T01:44:23.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf956a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:44:23.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf956a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:44:23.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.635+0000 7fddf956a640 1 -- 172.21.15.12:0/4099326292 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddf40a3910 con 0x7fddf406f910 2024-09-11T01:44:23.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddf956a640 1 -- 172.21.15.12:0/4099326292 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf406f910 msgr2=0x7fddf413f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:23.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddf956a640 1 --2- 172.21.15.12:0/4099326292 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf406f910 0x7fddf413f430 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fddec009e50 tx=0x7fddec02a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55ef5d5253f39018 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:23.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddf37fe640 1 -- 172.21.15.12:0/4099326292 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddec036070 con 0x7fddf406f910 2024-09-11T01:44:23.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddf37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:23.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddf37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:44:23.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddf37fe640 10 monclient: dump: 2024-09-11T01:44:23.648 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:23.648 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:23.649 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.649 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.649 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:23.649 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:23.649 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:23.649 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:23.649 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:23.649 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:23.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddf37fe640 10 monclient: _finish_auth 0 2024-09-11T01:44:23.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddf37fe640 10 monclient: _check_auth_tickets 2024-09-11T01:44:23.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddf37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:44:23.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddf37fe640 1 -- 172.21.15.12:0/4099326292 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddec007800 con 0x7fddf406f910 2024-09-11T01:44:23.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddfafcd640 5 monclient: authenticate success, global_id 164141 2024-09-11T01:44:23.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddfafcd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:44:23.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddf37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:44:23.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddfafcd640 10 monclient: get_monmap_and_config success 2024-09-11T01:44:23.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddf37fe640 1 -- 172.21.15.12:0/4099326292 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddec007bd0 con 0x7fddf406f910 2024-09-11T01:44:23.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddfafcd640 4 set_mon_vals no callback set 2024-09-11T01:44:23.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddf37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:23.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddf37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:44:23.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddf37fe640 10 monclient: dump: 2024-09-11T01:44:23.650 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:23.650 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:23.650 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.650 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.650 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:23.650 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:23.650 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:23.651 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddfafcd640 10 monclient: shutdown 2024-09-11T01:44:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddfafcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:44:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddfafcd640 1 -- 172.21.15.12:0/4099326292 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf406f910 msgr2=0x7fddf413f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddfafcd640 1 --2- 172.21.15.12:0/4099326292 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf406f910 0x7fddf413f430 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fddec009e50 tx=0x7fddec02a560 comp rx=0 tx=0).stop 2024-09-11T01:44:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddfafcd640 1 -- 172.21.15.12:0/4099326292 shutdown_connections 2024-09-11T01:44:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddfafcd640 1 --2- 172.21.15.12:0/4099326292 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf406f910 0x7fddf413f430 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddfafcd640 1 --2- 172.21.15.12:0/4099326292 >> v2:172.21.15.71:3300/0 conn(0x7fddf40a2e20 0x7fddf406ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddfafcd640 1 -- 172.21.15.12:0/4099326292 >> 172.21.15.12:0/4099326292 conn(0x7fddf40a8600 msgr2=0x7fddf40a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:44:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddfafcd640 1 -- 172.21.15.12:0/4099326292 shutdown_connections 2024-09-11T01:44:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.638+0000 7fddfafcd640 1 -- 172.21.15.12:0/4099326292 wait complete. 2024-09-11T01:44:23.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.639+0000 7fddfafcd640 10 monclient: build_initial_monmap 2024-09-11T01:44:23.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.639+0000 7fddfafcd640 10 monclient: monmap: 2024-09-11T01:44:23.652 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:44:23.652 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:44:23.652 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:44:23.652 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:44:23.652 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:44:23.652 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:23.652 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:44:23.652 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:44:23.652 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:44:23.652 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:23.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.639+0000 7fddfafcd640 1 Processor -- start 2024-09-11T01:44:23.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.639+0000 7fddfafcd640 1 -- start start 2024-09-11T01:44:23.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.639+0000 7fddfafcd640 10 monclient: init 2024-09-11T01:44:23.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddfafcd640 10 monclient: _reopen_session rank -1 2024-09-11T01:44:23.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddfafcd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:44:23.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddfafcd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf406f910 0x7fddf4139400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:23.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddfafcd640 10 monclient(hunting): picked mon.noname-c con 0x7fddf406f910 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:44:23.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddfafcd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fddf40a2e20 0x7fddf4139940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:23.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddfafcd640 10 monclient(hunting): picked mon.noname-b con 0x7fddf40a2e20 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:44:23.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddfafcd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fddf40a4970 0x7fddf4139e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:23.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddfafcd640 10 monclient(hunting): picked mon.noname-a con 0x7fddf40a4970 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:44:23.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddfafcd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddfafcd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fddf413fd00 con 0x7fddf40a4970 2024-09-11T01:44:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddfafcd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddfafcd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fddf413fb80 con 0x7fddf40a2e20 2024-09-11T01:44:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddfafcd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddfafcd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fddf413fe80 con 0x7fddf406f910 2024-09-11T01:44:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddfafcd640 10 monclient(hunting): _renew_subs 2024-09-11T01:44:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddfafcd640 10 monclient(hunting): authenticate will time out at 1670.083435s 2024-09-11T01:44:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf956a640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fddf40a4970 msgr2=0x7fddf4139e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:44:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf956a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fddf40a4970 0x7fddf4139e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:44:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf8d69640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf406f910 0x7fddf4139400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf8d69640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf406f910 0x7fddf4139400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45222/0 (socket says 172.21.15.12:45222) 2024-09-11T01:44:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf8d69640 1 -- 172.21.15.12:0/3493546962 learned_addr learned my addr 172.21.15.12:0/3493546962 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:44:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf8d69640 10 monclient(hunting): get_auth_request con 0x7fddf406f910 auth_method 0 2024-09-11T01:44:23.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf8d69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:23.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf8d69640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:23.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf8d69640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:23.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf8d69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:23.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf8d69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:23.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf3fff640 1 --2- 172.21.15.12:0/3493546962 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fddf40a2e20 0x7fddf4139940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:23.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf8d69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:23.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf3fff640 10 monclient(hunting): get_auth_request con 0x7fddf40a2e20 auth_method 0 2024-09-11T01:44:23.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:23.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:23.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:23.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:23.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf8d69640 10 monclient(hunting): handle_auth_done global_id 164144 payload 824 2024-09-11T01:44:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf8d69640 1 -- 172.21.15.12:0/3493546962 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fddf40a2e20 msgr2=0x7fddf4139940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf8d69640 1 --2- 172.21.15.12:0/3493546962 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fddf40a2e20 0x7fddf4139940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf8d69640 1 -- 172.21.15.12:0/3493546962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fddf40a4970 msgr2=0x7fddf4139e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:44:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf8d69640 1 --2- 172.21.15.12:0/3493546962 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fddf40a4970 0x7fddf4139e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf8d69640 10 monclient: _finish_hunting 0 2024-09-11T01:44:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf8d69640 1 monclient: found mon.noname-c 2024-09-11T01:44:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf8d69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:44:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf8d69640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:44:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.640+0000 7fddf8d69640 1 -- 172.21.15.12:0/3493546962 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddf413bd70 con 0x7fddf406f910 2024-09-11T01:44:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddf8d69640 1 -- 172.21.15.12:0/3493546962 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf406f910 msgr2=0x7fddf4139400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:23.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddf8d69640 1 --2- 172.21.15.12:0/3493546962 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf406f910 0x7fddf4139400 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdde800e9b0 tx=0x7fdde800edb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:23.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddf17fa640 1 -- 172.21.15.12:0/3493546962 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde8017070 con 0x7fddf406f910 2024-09-11T01:44:23.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddf17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:23.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddf17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:44:23.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddf17fa640 10 monclient: dump: 2024-09-11T01:44:23.657 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:23.657 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:23.657 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.657 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.657 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:23.657 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:23.657 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:23.657 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:23.657 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:23.657 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:23.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddf17fa640 10 monclient: _finish_auth 0 2024-09-11T01:44:23.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddf17fa640 10 monclient: _check_auth_tickets 2024-09-11T01:44:23.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddf17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:44:23.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddf17fa640 1 -- 172.21.15.12:0/3493546962 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdde800cb50 con 0x7fddf406f910 2024-09-11T01:44:23.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddfafcd640 5 monclient: authenticate success, global_id 164144 2024-09-11T01:44:23.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddfafcd640 10 monclient: _renew_subs 2024-09-11T01:44:23.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddfafcd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:44:23.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddfafcd640 1 -- 172.21.15.12:0/3493546962 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddf40a6520 con 0x7fddf406f910 2024-09-11T01:44:23.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddf17fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:44:23.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddf17fa640 1 -- 172.21.15.12:0/3493546962 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde8013470 con 0x7fddf406f910 2024-09-11T01:44:23.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddf17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:23.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddf17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:44:23.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddf17fa640 10 monclient: dump: 2024-09-11T01:44:23.658 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:23.658 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:23.659 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.659 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.659 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:23.659 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:23.659 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:23.659 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:23.659 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:23.659 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:23.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddf2ffd640 4 set_mon_vals no callback set 2024-09-11T01:44:23.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddfafcd640 10 monclient: _renew_subs 2024-09-11T01:44:23.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddfafcd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:44:23.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddfafcd640 1 -- 172.21.15.12:0/3493546962 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fddf409f610 con 0x7fddf406f910 2024-09-11T01:44:23.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddfafcd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:44:23.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddfafcd640 1 --2- 172.21.15.12:0/3493546962 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf4135760 0x7fddf402d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:23.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddfafcd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:23.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.643+0000 7fddfafcd640 1 -- 172.21.15.12:0/3493546962 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fddf406a060 con 0x7fddf4135760 2024-09-11T01:44:23.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.644+0000 7fddf3fff640 1 --2- 172.21.15.12:0/3493546962 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf4135760 0x7fddf402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:23.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.644+0000 7fddf3fff640 10 monclient: get_auth_request con 0x7fddf4135760 auth_method 0 2024-09-11T01:44:23.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.644+0000 7fddf3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:23.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.644+0000 7fddf3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:23.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.644+0000 7fddf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:23.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.644+0000 7fddf17fa640 1 -- 172.21.15.12:0/3493546962 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fdde8024060 con 0x7fddf406f910 2024-09-11T01:44:23.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.644+0000 7fddf3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:23.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.644+0000 7fddf3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:23.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.644+0000 7fddf3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:23.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.644+0000 7fddf3fff640 10 monclient(hunting): handle_auth_done global_id 164147 payload 824 2024-09-11T01:44:23.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.644+0000 7fddf3fff640 1 -- 172.21.15.12:0/3493546962 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fddf406ded0 con 0x7fddf4135760 2024-09-11T01:44:23.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.644+0000 7fddf17fa640 1 --2- 172.21.15.12:0/3493546962 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fddd803e210 0x7fddd80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:23.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.644+0000 7fddf17fa640 1 -- 172.21.15.12:0/3493546962 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(511..511 src has 317..511) ==== 4819+0+0 (secure 0 0 0) 0x7fdde8052cd0 con 0x7fddf406f910 2024-09-11T01:44:23.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.644+0000 7fddf956a640 1 --2- 172.21.15.12:0/3493546962 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fddd803e210 0x7fddd80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:23.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.644+0000 7fddf956a640 10 monclient: get_auth_request con 0x7fddd803e210 auth_method 0 2024-09-11T01:44:23.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.646+0000 7fddf3fff640 1 -- 172.21.15.12:0/3493546962 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf4135760 msgr2=0x7fddf402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:23.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.646+0000 7fddf3fff640 1 --2- 172.21.15.12:0/3493546962 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf4135760 0x7fddf402d440 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdde0003980 tx=0x7fdde0009aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:23.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.646+0000 7fddf17fa640 1 -- 172.21.15.12:0/3493546962 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde0016070 con 0x7fddf4135760 2024-09-11T01:44:23.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.646+0000 7fddf17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:23.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.646+0000 7fddf17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:44:23.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.646+0000 7fddf17fa640 10 monclient: dump: 2024-09-11T01:44:23.664 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:23.664 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:23.664 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.664 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.664 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:23.664 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:23.664 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:23.664 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:23.664 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:23.664 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:23.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.647+0000 7fddf17fa640 1 -- 172.21.15.12:0/3493546962 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdde0005a40 con 0x7fddf4135760 2024-09-11T01:44:23.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.647+0000 7fddf17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:44:23.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.647+0000 7fddf17fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:44:23.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.647+0000 7fddf956a640 1 --2- 172.21.15.12:0/3493546962 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fddd803e210 0x7fddd80406d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fddf4074f40 tx=0x7fddec009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:23.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.647+0000 7fddf17fa640 1 -- 172.21.15.12:0/3493546962 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf4135760 msgr2=0x7fddf402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:23.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.647+0000 7fddf17fa640 1 --2- 172.21.15.12:0/3493546962 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf4135760 0x7fddf402d440 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdde0003980 tx=0x7fdde0009aa0 comp rx=0 tx=0).stop 2024-09-11T01:44:23.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.647+0000 7fddf17fa640 1 -- 172.21.15.12:0/3493546962 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf4135760 msgr2=0x7fddf402d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:44:23.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.647+0000 7fddf17fa640 1 --2- 172.21.15.12:0/3493546962 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf4135760 0x7fddf402d440 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdde0003980 tx=0x7fdde0009aa0 comp rx=0 tx=0).stop 2024-09-11T01:44:23.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.653+0000 7fddfafcd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:44:23.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.653+0000 7fddfafcd640 1 --2- 172.21.15.12:0/3493546962 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf4135b40 0x7fddf402d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:23.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.653+0000 7fddfafcd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:23.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.653+0000 7fddfafcd640 1 -- 172.21.15.12:0/3493546962 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fddf41378d0 con 0x7fddf4135b40 2024-09-11T01:44:23.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.653+0000 7fddf3fff640 1 --2- 172.21.15.12:0/3493546962 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf4135b40 0x7fddf402d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:23.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.653+0000 7fddf3fff640 10 monclient: get_auth_request con 0x7fddf4135b40 auth_method 0 2024-09-11T01:44:23.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.653+0000 7fddf3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:23.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.653+0000 7fddf3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:23.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.653+0000 7fddf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:23.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.653+0000 7fddf3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:23.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.653+0000 7fddf3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:23.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.653+0000 7fddf3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:23.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.653+0000 7fddf3fff640 10 monclient(hunting): handle_auth_done global_id 164150 payload 824 2024-09-11T01:44:23.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.653+0000 7fddf3fff640 1 -- 172.21.15.12:0/3493546962 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fddf4069950 con 0x7fddf4135b40 2024-09-11T01:44:23.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.655+0000 7fddf3fff640 1 -- 172.21.15.12:0/3493546962 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf4135b40 msgr2=0x7fddf402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:23.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.655+0000 7fddf3fff640 1 --2- 172.21.15.12:0/3493546962 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf4135b40 0x7fddf402d980 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdde0017000 tx=0x7fdde00059d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:23.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.656+0000 7fddf17fa640 1 -- 172.21.15.12:0/3493546962 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde0016050 con 0x7fddf4135b40 2024-09-11T01:44:23.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.656+0000 7fddf17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:23.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.656+0000 7fddf17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:44:23.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.656+0000 7fddf17fa640 10 monclient: dump: 2024-09-11T01:44:23.666 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:23.666 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:23.666 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.666 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.667 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:23.667 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:23.667 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:23.667 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:23.667 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:23.667 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:23.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.656+0000 7fddf17fa640 1 -- 172.21.15.12:0/3493546962 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4864 (secure 0 0 0) 0x7fdde00067a0 con 0x7fddf4135b40 2024-09-11T01:44:23.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.656+0000 7fddf17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:44:23.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.656+0000 7fddf17fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:44:23.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.656+0000 7fddf17fa640 1 -- 172.21.15.12:0/3493546962 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf4135b40 msgr2=0x7fddf402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:23.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.656+0000 7fddf17fa640 1 --2- 172.21.15.12:0/3493546962 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf4135b40 0x7fddf402d980 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdde0017000 tx=0x7fdde00059d0 comp rx=0 tx=0).stop 2024-09-11T01:44:23.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.656+0000 7fddf17fa640 1 -- 172.21.15.12:0/3493546962 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf4135b40 msgr2=0x7fddf402d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:44:23.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.656+0000 7fddf17fa640 1 --2- 172.21.15.12:0/3493546962 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf4135b40 0x7fddf402d980 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdde0017000 tx=0x7fdde00059d0 comp rx=0 tx=0).stop 2024-09-11T01:44:23.668 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:44:23.668 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:44:23.668 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:44:23.668 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:44:23.668 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 140, 2024-09-11T01:44:23.668 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 39670, 2024-09-11T01:44:23.668 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:44:23.668 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:44:23.668 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:44:23.668 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:23.668 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:44:23.669 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:44:23.669 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:44:23.669 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:44:23.669 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:44:23.669 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:44:23.669 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:44:23.669 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:44:23.669 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:44:23.669 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:44:23.669 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:44:23.669 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:44:23.669 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:44:23.669 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:44:23.669 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:44:23.670 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:23.670 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:44:23.670 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:44:23.670 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:44:23.670 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:44:23.670 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:44:23.670 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:44:23.670 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:44:23.670 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:44:23.670 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:44:23.670 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:44:23.670 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:44:23.670 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:44:23.670 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:44:23.671 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:23.671 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:23.671 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:44:23.671 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:44:23.671 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:44:23.671 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:44:23.671 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:44:23.671 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:44:23.671 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:44:23.671 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:44:23.671 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:44:23.671 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:44:23.671 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:44:23.672 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:44:23.672 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:44:23.672 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:44:23.672 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:44:23.672 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:44:23.672 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:44:23.672 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:44:23.672 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:44:23.672 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:44:23.672 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:44:23.672 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:44:23.672 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:44:23.672 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:44:23.672 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:44:23.673 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:44:23.673 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:44:23.673 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:44:23.673 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:23.673 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:44:23.673 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:23.673 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:44:23.673 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:23.673 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:44:23.673 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:44:23.673 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:44:23.673 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:44:23.673 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:23.673 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:44:23.674 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:44:23.674 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:23.674 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:23.674 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:23.674 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:44:23.674 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:44:23.674 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:23.674 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:23.674 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:23.674 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:23.674 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:44:23.674 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:44:23.674 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:44:23.674 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:44:23.674 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:44:23.675 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:23.675 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:23.675 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:44:23.675 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:44:23.675 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:44:23.675 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:44:23.675 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:23.675 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:44:23.675 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:44:23.675 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:23.675 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:23.675 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:23.675 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:44:23.675 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:44:23.676 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:23.676 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:23.676 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:23.676 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:23.676 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:44:23.676 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:44:23.676 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:44:23.676 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:44:23.676 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:44:23.676 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:23.676 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:23.676 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:44:23.676 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:44:23.676 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:44:23.677 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:44:23.677 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:23.677 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:44:23.677 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:44:23.677 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:23.677 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:23.677 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:23.677 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:44:23.677 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:44:23.677 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:23.677 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:23.677 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:23.677 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:23.677 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:44:23.678 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:44:23.678 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:44:23.678 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:44:23.678 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:44:23.678 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:23.678 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:23.678 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:23.678 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:44:23.678 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:44:23.678 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:23.678 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:44:23.678 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:44:23.678 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:44:23.678 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:23.678 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:23.678 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:44:23.679 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:23.679 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:44:23.679 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:44:23.679 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:44:23.679 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:23.679 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:23.679 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:44:23.679 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:23.679 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:44:23.679 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:44:23.679 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 7 2024-09-11T01:44:23.679 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:23.679 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:23.679 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:44:23.679 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:23.680 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:44:23.680 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:44:23.680 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:44:23.680 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:23.680 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:23.680 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:23.680 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:44:23.680 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:44:23.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.656+0000 7fddfafcd640 1 -- 172.21.15.12:0/3493546962 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fddd803e210 msgr2=0x7fddd80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:23.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.656+0000 7fddfafcd640 1 --2- 172.21.15.12:0/3493546962 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fddd803e210 0x7fddd80406d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fddf4074f40 tx=0x7fddec009000 comp rx=0 tx=0).stop 2024-09-11T01:44:23.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.656+0000 7fddfafcd640 10 monclient: shutdown 2024-09-11T01:44:23.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.656+0000 7fddfafcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:44:23.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.656+0000 7fddfafcd640 1 -- 172.21.15.12:0/3493546962 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf406f910 msgr2=0x7fddf4139400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:23.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.656+0000 7fddfafcd640 1 --2- 172.21.15.12:0/3493546962 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fddf406f910 0x7fddf4139400 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdde800e9b0 tx=0x7fdde800edb0 comp rx=0 tx=0).stop 2024-09-11T01:44:23.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.656+0000 7fddf956a640 1 -- 172.21.15.12:0/3493546962 reap_dead start 2024-09-11T01:44:23.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.656+0000 7fddf956a640 1 -- 172.21.15.12:0/3493546962 reap_dead start 2024-09-11T01:44:23.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.656+0000 7fddfafcd640 1 -- 172.21.15.12:0/3493546962 shutdown_connections 2024-09-11T01:44:23.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.656+0000 7fddfafcd640 1 -- 172.21.15.12:0/3493546962 >> 172.21.15.12:0/3493546962 conn(0x7fddf40a8600 msgr2=0x7fddf40a1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:44:23.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.656+0000 7fddfafcd640 1 -- 172.21.15.12:0/3493546962 shutdown_connections 2024-09-11T01:44:23.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.656+0000 7fddfafcd640 1 -- 172.21.15.12:0/3493546962 wait complete. 2024-09-11T01:44:23.683 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:44:23.683 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T01:44:23.683 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:44:23.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.782+0000 7f14c2cb0640 10 monclient: get_monmap_and_config 2024-09-11T01:44:23.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.782+0000 7f14c2cb0640 10 monclient: build_initial_monmap 2024-09-11T01:44:23.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.782+0000 7f14c2cb0640 10 monclient: monmap: 2024-09-11T01:44:23.785 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:44:23.785 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:23.785 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:44:23.784363+0000 2024-09-11T01:44:23.785 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:44:23.784363+0000 2024-09-11T01:44:23.786 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:44:23.786 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:23.786 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:44:23.786 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:44:23.786 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:44:23.786 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:44:23.786 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:44:23.786 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:23.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.783+0000 7f14c2cb0640 1 Processor -- start 2024-09-11T01:44:23.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.783+0000 7f14c2cb0640 1 -- start start 2024-09-11T01:44:23.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.783+0000 7f14c2cb0640 10 monclient: init 2024-09-11T01:44:23.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.783+0000 7f14c2cb0640 10 monclient: _reopen_session rank -1 2024-09-11T01:44:23.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.783+0000 7f14c2cb0640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-11T01:44:23.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.783+0000 7f14c2cb0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f14bc0a1380 addr v1:172.21.15.12:6789/0 2024-09-11T01:44:23.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.783+0000 7f14c2cb0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f14bc0a0370 addr v1:172.21.15.71:6789/0 2024-09-11T01:44:23.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.783+0000 7f14c2cb0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14bc0a0c90 0x7f14bc13e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:23.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.783+0000 7f14c2cb0640 10 monclient(hunting): picked mon.noname-b con 0x7f14bc0a0c90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:44:23.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.783+0000 7f14c0a4c640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7f14bc0a1380 legacy=0x7f14bc0a1760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:44:23.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.783+0000 7f14c2cb0640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14bc0a21c0 con 0x7f14bc0a1380 2024-09-11T01:44:23.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.783+0000 7f14c2cb0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14bc0a24a0 con 0x7f14bc0a0370 2024-09-11T01:44:23.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.783+0000 7f14c2cb0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:23.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.783+0000 7f14c2cb0640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f14bc0a2780 con 0x7f14bc0a0c90 2024-09-11T01:44:23.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.783+0000 7f14c2cb0640 10 monclient(hunting): _renew_subs 2024-09-11T01:44:23.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.784+0000 7f14c2cb0640 10 monclient(hunting): authenticate will time out at 1670.227010s 2024-09-11T01:44:23.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.784+0000 7f14c124d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14bc0a0c90 0x7f14bc13e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:23.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.784+0000 7f14c124d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14bc0a0c90 0x7f14bc13e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45232/0 (socket says 172.21.15.12:45232) 2024-09-11T01:44:23.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.784+0000 7f14c124d640 1 -- 172.21.15.12:0/889047966 learned_addr learned my addr 172.21.15.12:0/889047966 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:44:23.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.784+0000 7f14c124d640 10 monclient(hunting): get_auth_request con 0x7f14bc0a0c90 auth_method 0 2024-09-11T01:44:23.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.784+0000 7f14c124d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:23.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.784+0000 7f14c124d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:23.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.784+0000 7f14c124d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:23.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.784+0000 7f14c124d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:23.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.784+0000 7f14c124d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:23.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.784+0000 7f14c124d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:23.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.784+0000 7f14c124d640 10 monclient(hunting): handle_auth_done global_id 164153 payload 274 2024-09-11T01:44:23.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.784+0000 7f14c124d640 1 -- 172.21.15.12:0/889047966 >> v1:172.21.15.71:6789/0 conn(0x7f14bc0a0370 legacy=0x7f14bc13ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:44:23.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.784+0000 7f14c124d640 1 -- 172.21.15.12:0/889047966 >> v1:172.21.15.12:6789/0 conn(0x7f14bc0a1380 legacy=0x7f14bc0a1760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:44:23.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.784+0000 7f14c124d640 10 monclient: _finish_hunting 0 2024-09-11T01:44:23.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.784+0000 7f14c124d640 1 monclient: found mon.noname-b 2024-09-11T01:44:23.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.784+0000 7f14c124d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:44:23.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.784+0000 7f14c124d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:44:23.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.784+0000 7f14c124d640 1 -- 172.21.15.12:0/889047966 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14bc09fb60 con 0x7f14bc0a0c90 2024-09-11T01:44:23.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14c124d640 1 -- 172.21.15.12:0/889047966 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14bc0a0c90 msgr2=0x7f14bc13e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:23.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14c124d640 1 --2- 172.21.15.12:0/889047966 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14bc0a0c90 0x7f14bc13e530 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f14b4009e80 tx=0x7f14b402a530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b6dba391406b2c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:23.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14bb7fe640 1 -- 172.21.15.12:0/889047966 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4036070 con 0x7f14bc0a0c90 2024-09-11T01:44:23.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:23.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14bb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:44:23.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14bb7fe640 10 monclient: dump: 2024-09-11T01:44:23.800 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:23.800 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:23.800 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.800 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.801 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:23.801 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:23.801 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:23.801 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:23.801 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:23.801 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:23.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14bb7fe640 10 monclient: _finish_auth 0 2024-09-11T01:44:23.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14bb7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:44:23.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:44:23.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14bb7fe640 1 -- 172.21.15.12:0/889047966 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14b4007800 con 0x7f14bc0a0c90 2024-09-11T01:44:23.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14c2cb0640 5 monclient: authenticate success, global_id 164153 2024-09-11T01:44:23.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14c2cb0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:44:23.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14bb7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:44:23.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14c2cb0640 10 monclient: get_monmap_and_config success 2024-09-11T01:44:23.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14c2cb0640 4 set_mon_vals no callback set 2024-09-11T01:44:23.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14bb7fe640 1 -- 172.21.15.12:0/889047966 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4007bd0 con 0x7f14bc0a0c90 2024-09-11T01:44:23.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:23.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14bb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:44:23.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14bb7fe640 10 monclient: dump: 2024-09-11T01:44:23.802 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:23.802 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:23.802 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.803 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.803 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:23.803 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:23.803 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:23.803 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:23.803 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:23.803 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:23.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14c2cb0640 10 monclient: shutdown 2024-09-11T01:44:23.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14c2cb0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:44:23.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14c2cb0640 1 -- 172.21.15.12:0/889047966 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14bc0a0c90 msgr2=0x7f14bc13e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:23.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14c2cb0640 1 --2- 172.21.15.12:0/889047966 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14bc0a0c90 0x7f14bc13e530 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f14b4009e80 tx=0x7f14b402a530 comp rx=0 tx=0).stop 2024-09-11T01:44:23.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14c2cb0640 1 -- 172.21.15.12:0/889047966 shutdown_connections 2024-09-11T01:44:23.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14c2cb0640 1 --2- 172.21.15.12:0/889047966 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14bc0a0c90 0x7f14bc13e530 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:23.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14c2cb0640 1 -- 172.21.15.12:0/889047966 >> 172.21.15.12:0/889047966 conn(0x7f14bc0a2f70 msgr2=0x7f14bc0a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:44:23.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14c2cb0640 1 -- 172.21.15.12:0/889047966 shutdown_connections 2024-09-11T01:44:23.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.787+0000 7f14c2cb0640 1 -- 172.21.15.12:0/889047966 wait complete. 2024-09-11T01:44:23.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.788+0000 7f14c2cb0640 10 monclient: build_initial_monmap 2024-09-11T01:44:23.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.788+0000 7f14c2cb0640 10 monclient: monmap: 2024-09-11T01:44:23.804 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:44:23.804 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:44:23.804 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:44:23.804 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:44:23.804 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:44:23.804 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:23.804 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:44:23.804 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:44:23.805 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:44:23.805 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:23.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.788+0000 7f14c2cb0640 1 Processor -- start 2024-09-11T01:44:23.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.788+0000 7f14c2cb0640 1 -- start start 2024-09-11T01:44:23.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.788+0000 7f14c2cb0640 10 monclient: init 2024-09-11T01:44:23.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c2cb0640 10 monclient: _reopen_session rank -1 2024-09-11T01:44:23.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c2cb0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:44:23.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c2cb0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14bc0a0370 0x7f14bc13db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:23.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c2cb0640 10 monclient(hunting): picked mon.noname-c con 0x7f14bc0a0370 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:44:23.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c2cb0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14bc0a0c90 0x7f14bc09d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:23.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c2cb0640 10 monclient(hunting): picked mon.noname-b con 0x7f14bc0a0c90 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:44:23.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c2cb0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14bc0a1380 0x7f14bc09dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:23.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c2cb0640 10 monclient(hunting): picked mon.noname-a con 0x7f14bc0a1380 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:44:23.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c2cb0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:23.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c2cb0640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc13ebf0 con 0x7f14bc0a1380 2024-09-11T01:44:23.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c2cb0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:23.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c2cb0640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc13ea70 con 0x7f14bc0a0c90 2024-09-11T01:44:23.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c2cb0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:23.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c2cb0640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f14bc13ed70 con 0x7f14bc0a0370 2024-09-11T01:44:23.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c2cb0640 10 monclient(hunting): _renew_subs 2024-09-11T01:44:23.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c2cb0640 10 monclient(hunting): authenticate will time out at 1670.232493s 2024-09-11T01:44:23.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c124d640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14bc0a1380 msgr2=0x7f14bc09dd00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:44:23.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c124d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14bc0a1380 0x7f14bc09dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:44:23.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c0a4c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14bc0a0370 0x7f14bc13db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:23.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c0a4c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14bc0a0370 0x7f14bc13db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45246/0 (socket says 172.21.15.12:45246) 2024-09-11T01:44:23.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c0a4c640 1 -- 172.21.15.12:0/3103233657 learned_addr learned my addr 172.21.15.12:0/3103233657 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:44:23.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c0a4c640 10 monclient(hunting): get_auth_request con 0x7f14bc0a0370 auth_method 0 2024-09-11T01:44:23.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c0a4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:23.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c0a4c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:23.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c0a4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:23.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14bbfff640 1 --2- 172.21.15.12:0/3103233657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14bc0a0c90 0x7f14bc09d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:23.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14bbfff640 10 monclient(hunting): get_auth_request con 0x7f14bc0a0c90 auth_method 0 2024-09-11T01:44:23.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:23.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:23.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:23.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c0a4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:23.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c0a4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:23.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c0a4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:23.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:23.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:23.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:23.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c0a4c640 10 monclient(hunting): handle_auth_done global_id 164156 payload 824 2024-09-11T01:44:23.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c0a4c640 1 -- 172.21.15.12:0/3103233657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14bc0a0c90 msgr2=0x7f14bc09d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:23.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c0a4c640 1 --2- 172.21.15.12:0/3103233657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14bc0a0c90 0x7f14bc09d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:23.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c0a4c640 1 -- 172.21.15.12:0/3103233657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14bc0a1380 msgr2=0x7f14bc09dd00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:44:23.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c0a4c640 1 --2- 172.21.15.12:0/3103233657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14bc0a1380 0x7f14bc09dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:23.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c0a4c640 10 monclient: _finish_hunting 0 2024-09-11T01:44:23.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c0a4c640 1 monclient: found mon.noname-c 2024-09-11T01:44:23.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c0a4c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:44:23.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c0a4c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:44:23.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.789+0000 7f14c0a4c640 1 -- 172.21.15.12:0/3103233657 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14bc1294d0 con 0x7f14bc0a0370 2024-09-11T01:44:23.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.792+0000 7f14c0a4c640 1 -- 172.21.15.12:0/3103233657 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14bc0a0370 msgr2=0x7f14bc13db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:23.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.792+0000 7f14c0a4c640 1 --2- 172.21.15.12:0/3103233657 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14bc0a0370 0x7f14bc13db30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f14b000b550 tx=0x7f14b000ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:23.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.792+0000 7f14b97fa640 1 -- 172.21.15.12:0/3103233657 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b0005070 con 0x7f14bc0a0370 2024-09-11T01:44:23.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.792+0000 7f14b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:23.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.792+0000 7f14b97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:44:23.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.792+0000 7f14b97fa640 10 monclient: dump: 2024-09-11T01:44:23.810 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:23.810 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:23.810 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.810 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.810 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:23.810 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:23.810 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:23.810 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:23.810 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:23.810 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:23.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.792+0000 7f14b97fa640 10 monclient: _finish_auth 0 2024-09-11T01:44:23.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.792+0000 7f14b97fa640 10 monclient: _check_auth_tickets 2024-09-11T01:44:23.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.792+0000 7f14b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:44:23.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.792+0000 7f14b97fa640 1 -- 172.21.15.12:0/3103233657 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14b0015070 con 0x7f14bc0a0370 2024-09-11T01:44:23.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.792+0000 7f14c2cb0640 5 monclient: authenticate success, global_id 164156 2024-09-11T01:44:23.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.792+0000 7f14c2cb0640 10 monclient: _renew_subs 2024-09-11T01:44:23.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.792+0000 7f14c2cb0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:44:23.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.792+0000 7f14c2cb0640 1 -- 172.21.15.12:0/3103233657 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14bc070190 con 0x7f14bc0a0370 2024-09-11T01:44:23.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.792+0000 7f14b97fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:44:23.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.792+0000 7f14b97fa640 1 -- 172.21.15.12:0/3103233657 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b0005450 con 0x7f14bc0a0370 2024-09-11T01:44:23.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.792+0000 7f14b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:23.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.792+0000 7f14b97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:44:23.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.792+0000 7f14b97fa640 10 monclient: dump: 2024-09-11T01:44:23.812 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:23.812 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:23.812 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.812 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.812 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:23.812 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:23.812 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:23.812 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:23.812 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:23.812 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:23.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.792+0000 7f14baffd640 4 set_mon_vals no callback set 2024-09-11T01:44:23.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.792+0000 7f14c2cb0640 10 monclient: _renew_subs 2024-09-11T01:44:23.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.792+0000 7f14c2cb0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:44:23.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.792+0000 7f14c2cb0640 1 -- 172.21.15.12:0/3103233657 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f14bc1291f0 con 0x7f14bc0a0370 2024-09-11T01:44:23.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.793+0000 7f14c2cb0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:44:23.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.793+0000 7f14c2cb0640 1 --2- 172.21.15.12:0/3103233657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14bc13e070 0x7f14bc02e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:23.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.793+0000 7f14c2cb0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:23.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.793+0000 7f14c2cb0640 1 -- 172.21.15.12:0/3103233657 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc1cde20 con 0x7f14bc13e070 2024-09-11T01:44:23.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.793+0000 7f14b97fa640 1 -- 172.21.15.12:0/3103233657 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f14b0005630 con 0x7f14bc0a0370 2024-09-11T01:44:23.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.793+0000 7f14bbfff640 1 --2- 172.21.15.12:0/3103233657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14bc13e070 0x7f14bc02e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:23.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.793+0000 7f14bbfff640 10 monclient: get_auth_request con 0x7f14bc13e070 auth_method 0 2024-09-11T01:44:23.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.793+0000 7f14bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:23.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.793+0000 7f14bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:23.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.793+0000 7f14bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:23.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.793+0000 7f14b97fa640 1 --2- 172.21.15.12:0/3103233657 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f14a003e1c0 0x7f14a0040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:23.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.793+0000 7f14bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:23.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.793+0000 7f14bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:23.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.793+0000 7f14bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:23.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.793+0000 7f14b97fa640 1 -- 172.21.15.12:0/3103233657 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(511..511 src has 317..511) ==== 4819+0+0 (secure 0 0 0) 0x7f14b0052b40 con 0x7f14bc0a0370 2024-09-11T01:44:23.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.793+0000 7f14bbfff640 10 monclient(hunting): handle_auth_done global_id 134431 payload 824 2024-09-11T01:44:23.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.793+0000 7f14c124d640 1 --2- 172.21.15.12:0/3103233657 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f14a003e1c0 0x7f14a0040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:23.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.793+0000 7f14bbfff640 1 -- 172.21.15.12:0/3103233657 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f14bc128da0 con 0x7f14bc13e070 2024-09-11T01:44:23.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.793+0000 7f14c124d640 10 monclient: get_auth_request con 0x7f14a003e1c0 auth_method 0 2024-09-11T01:44:23.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.796+0000 7f14bbfff640 1 -- 172.21.15.12:0/3103233657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14bc13e070 msgr2=0x7f14bc02e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:23.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.796+0000 7f14bbfff640 1 --2- 172.21.15.12:0/3103233657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14bc13e070 0x7f14bc02e5c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f14a800f770 tx=0x7f14a800fc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:23.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.796+0000 7f14b97fa640 1 -- 172.21.15.12:0/3103233657 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a8018030 con 0x7f14bc13e070 2024-09-11T01:44:23.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.796+0000 7f14b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:23.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.796+0000 7f14b97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:44:23.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.796+0000 7f14b97fa640 10 monclient: dump: 2024-09-11T01:44:23.814 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:23.815 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:23.815 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.815 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.815 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:23.815 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:23.815 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:23.815 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:23.815 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:23.815 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:23.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.796+0000 7f14c124d640 1 --2- 172.21.15.12:0/3103233657 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f14a003e1c0 0x7f14a0040680 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f14bc0a72b0 tx=0x7f14b4009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:23.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.796+0000 7f14b97fa640 1 -- 172.21.15.12:0/3103233657 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f14a800d9b0 con 0x7f14bc13e070 2024-09-11T01:44:23.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.796+0000 7f14b97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:44:23.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.796+0000 7f14b97fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:44:23.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.796+0000 7f14b97fa640 1 -- 172.21.15.12:0/3103233657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14bc13e070 msgr2=0x7f14bc02e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:23.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.796+0000 7f14b97fa640 1 --2- 172.21.15.12:0/3103233657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14bc13e070 0x7f14bc02e5c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f14a800f770 tx=0x7f14a800fc30 comp rx=0 tx=0).stop 2024-09-11T01:44:23.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.796+0000 7f14b97fa640 1 -- 172.21.15.12:0/3103233657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14bc13e070 msgr2=0x7f14bc02e5c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:44:23.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.796+0000 7f14b97fa640 1 --2- 172.21.15.12:0/3103233657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14bc13e070 0x7f14bc02e5c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:23.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.802+0000 7f14c2cb0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:44:23.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.802+0000 7f14c2cb0640 1 --2- 172.21.15.12:0/3103233657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14bc09e240 0x7f14bc063d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:23.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.802+0000 7f14c2cb0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:23.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.802+0000 7f14c2cb0640 1 -- 172.21.15.12:0/3103233657 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc072d20 con 0x7f14bc09e240 2024-09-11T01:44:23.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.802+0000 7f14bbfff640 1 --2- 172.21.15.12:0/3103233657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14bc09e240 0x7f14bc063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:23.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.802+0000 7f14bbfff640 10 monclient: get_auth_request con 0x7f14bc09e240 auth_method 0 2024-09-11T01:44:23.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.802+0000 7f14bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:23.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.802+0000 7f14bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:23.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.802+0000 7f14bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:23.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.803+0000 7f14bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.803+0000 7f14bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.803+0000 7f14bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.803+0000 7f14bbfff640 10 monclient(hunting): handle_auth_done global_id 134434 payload 824 2024-09-11T01:44:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.803+0000 7f14bbfff640 1 -- 172.21.15.12:0/3103233657 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f14bc09e620 con 0x7f14bc09e240 2024-09-11T01:44:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.805+0000 7f14bbfff640 1 -- 172.21.15.12:0/3103233657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14bc09e240 msgr2=0x7f14bc063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.805+0000 7f14bbfff640 1 --2- 172.21.15.12:0/3103233657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14bc09e240 0x7f14bc063d50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f14a800f770 tx=0x7f14a8003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.805+0000 7f14b97fa640 1 -- 172.21.15.12:0/3103233657 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a801de60 con 0x7f14bc09e240 2024-09-11T01:44:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.805+0000 7f14b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.805+0000 7f14b97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:44:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.805+0000 7f14b97fa640 10 monclient: dump: 2024-09-11T01:44:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:23.819 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:23.820 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:23.820 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:23.820 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:23.820 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:23.820 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:23.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.805+0000 7f14b97fa640 1 -- 172.21.15.12:0/3103233657 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4700 (secure 0 0 0) 0x7f14a8003e00 con 0x7f14bc09e240 2024-09-11T01:44:23.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.805+0000 7f14b97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:44:23.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.805+0000 7f14b97fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:44:23.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.805+0000 7f14b97fa640 1 -- 172.21.15.12:0/3103233657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14bc09e240 msgr2=0x7f14bc063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:23.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.805+0000 7f14b97fa640 1 --2- 172.21.15.12:0/3103233657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14bc09e240 0x7f14bc063d50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f14a800f770 tx=0x7f14a8003760 comp rx=0 tx=0).stop 2024-09-11T01:44:23.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.805+0000 7f14b97fa640 1 -- 172.21.15.12:0/3103233657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14bc09e240 msgr2=0x7f14bc063d50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:44:23.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.805+0000 7f14b97fa640 1 --2- 172.21.15.12:0/3103233657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14bc09e240 0x7f14bc063d50 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:23.821 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:44:23.821 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:44:23.821 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:44:23.821 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:44:23.821 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 140, 2024-09-11T01:44:23.821 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 196769, 2024-09-11T01:44:23.821 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:44:23.821 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:44:23.821 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:44:23.821 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:23.821 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:44:23.821 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:44:23.821 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:44:23.822 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:44:23.822 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:44:23.822 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:44:23.822 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:44:23.822 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:44:23.822 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:44:23.822 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:44:23.822 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:44:23.822 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:44:23.822 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:44:23.822 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:44:23.822 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:44:23.822 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:23.822 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:44:23.823 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:44:23.823 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:44:23.823 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:44:23.823 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:44:23.823 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:44:23.823 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:44:23.823 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:44:23.823 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:44:23.823 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:44:23.823 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:44:23.823 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:44:23.823 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:44:23.823 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:23.824 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:23.824 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:44:23.824 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:44:23.824 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:44:23.824 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:44:23.824 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:44:23.824 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:44:23.824 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:44:23.824 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:44:23.824 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:44:23.824 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:44:23.824 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:44:23.824 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:44:23.824 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:44:23.825 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:44:23.825 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:44:23.825 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:44:23.825 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:44:23.825 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:44:23.825 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:44:23.825 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:44:23.825 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:44:23.825 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:44:23.825 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:44:23.825 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:44:23.825 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:44:23.825 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:44:23.825 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:44:23.826 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:44:23.826 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:23.826 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:44:23.826 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:23.826 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:44:23.826 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:23.826 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:44:23.826 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:44:23.826 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:44:23.826 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:44:23.826 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:23.826 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:44:23.826 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:44:23.827 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:23.827 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:23.827 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:23.827 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:44:23.827 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:44:23.827 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:23.827 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:23.827 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:23.827 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:23.827 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:44:23.827 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:44:23.827 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:44:23.827 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:44:23.828 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:44:23.828 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:23.828 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:23.828 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:44:23.828 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:44:23.828 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:44:23.828 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:44:23.828 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:23.828 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:44:23.828 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:44:23.828 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:23.828 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:23.828 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:23.828 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:44:23.829 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:44:23.829 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:23.829 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:23.829 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:23.829 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:23.829 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:44:23.829 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:44:23.829 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:44:23.829 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:44:23.829 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:44:23.829 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:23.829 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:23.829 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:44:23.830 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:44:23.830 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:44:23.830 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:44:23.830 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:23.830 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:44:23.830 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:44:23.830 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:23.830 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:23.830 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:23.830 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:44:23.830 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:44:23.830 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:23.830 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:23.830 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:23.831 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:23.831 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:44:23.831 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:44:23.831 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:44:23.831 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:44:23.831 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:44:23.831 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:23.831 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:23.831 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:23.831 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:44:23.831 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:44:23.831 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:23.831 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:44:23.831 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:44:23.832 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:44:23.832 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:23.832 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:23.832 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:44:23.832 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:23.832 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:44:23.832 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:44:23.832 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:44:23.832 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:23.832 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:23.832 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:44:23.832 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:23.832 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:44:23.832 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:44:23.833 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:44:23.833 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:23.833 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:23.833 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:23.833 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:44:23.833 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:44:23.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.806+0000 7f14c2cb0640 1 -- 172.21.15.12:0/3103233657 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f14a003e1c0 msgr2=0x7f14a0040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:23.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.806+0000 7f14c2cb0640 1 --2- 172.21.15.12:0/3103233657 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f14a003e1c0 0x7f14a0040680 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f14bc0a72b0 tx=0x7f14b4009000 comp rx=0 tx=0).stop 2024-09-11T01:44:23.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.806+0000 7f14c2cb0640 10 monclient: shutdown 2024-09-11T01:44:23.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.806+0000 7f14c2cb0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:44:23.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.806+0000 7f14c2cb0640 1 -- 172.21.15.12:0/3103233657 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14bc0a0370 msgr2=0x7f14bc13db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:23.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.806+0000 7f14c2cb0640 1 --2- 172.21.15.12:0/3103233657 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f14bc0a0370 0x7f14bc13db30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f14b000b550 tx=0x7f14b000ba10 comp rx=0 tx=0).stop 2024-09-11T01:44:23.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.806+0000 7f14c124d640 1 -- 172.21.15.12:0/3103233657 reap_dead start 2024-09-11T01:44:23.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.806+0000 7f14c124d640 1 -- 172.21.15.12:0/3103233657 reap_dead start 2024-09-11T01:44:23.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.806+0000 7f14c2cb0640 1 -- 172.21.15.12:0/3103233657 shutdown_connections 2024-09-11T01:44:23.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.806+0000 7f14c2cb0640 1 -- 172.21.15.12:0/3103233657 >> 172.21.15.12:0/3103233657 conn(0x7f14bc0a2f70 msgr2=0x7f14bc0713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:44:23.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.806+0000 7f14c2cb0640 1 -- 172.21.15.12:0/3103233657 shutdown_connections 2024-09-11T01:44:23.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:23.806+0000 7f14c2cb0640 1 -- 172.21.15.12:0/3103233657 wait complete. 2024-09-11T01:44:23.835 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:44:23.835 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:44:24.142 INFO:tasks.rados.rados.0.smithi071.stdout:2295: done (3 left) 2024-09-11T01:44:24.142 INFO:tasks.rados.rados.0.smithi071.stdout:2296: done (2 left) 2024-09-11T01:44:24.142 INFO:tasks.rados.rados.0.smithi071.stdout:2298: done (1 left) 2024-09-11T01:44:24.142 INFO:tasks.rados.rados.0.smithi071.stdout:2299: done (0 left) 2024-09-11T01:44:24.143 INFO:tasks.rados.rados.0.smithi071.stdout:2300: write (excl) oid 41 current snap is 254 2024-09-11T01:44:24.143 INFO:tasks.rados.rados.0.smithi071.stdout:2300: seq_num 832 ranges {783424=489645,2003489=653949,3231577=554550} 2024-09-11T01:44:24.149 INFO:tasks.rados.rados.0.smithi071.stdout:2300: writing smithi07134777-41 from 783424 to 1273069 tid 1 2024-09-11T01:44:24.157 INFO:tasks.rados.rados.0.smithi071.stdout:2300: writing smithi07134777-41 from 2003489 to 2657438 tid 2 2024-09-11T01:44:24.164 INFO:tasks.rados.rados.0.smithi071.stdout:2300: writing smithi07134777-41 from 3231577 to 3786127 tid 3 2024-09-11T01:44:24.164 INFO:tasks.rados.rados.0.smithi071.stdout:2301: copy_from oid 15 from oid 13 current snap is 254 2024-09-11T01:44:24.164 INFO:tasks.rados.rados.0.smithi071.stdout:2300: finishing write tid 1 to smithi07134777-41 2024-09-11T01:44:24.164 INFO:tasks.rados.rados.0.smithi071.stdout:2300: finishing write tid 2 to smithi07134777-41 2024-09-11T01:44:24.164 INFO:tasks.rados.rados.0.smithi071.stdout:2302: snap_remove snap 227 2024-09-11T01:44:25.160 INFO:tasks.rados.rados.0.smithi071.stdout:2302: done (2 left) 2024-09-11T01:44:25.160 INFO:tasks.rados.rados.0.smithi071.stdout:2303: snap_remove snap 241 2024-09-11T01:44:26.165 INFO:tasks.rados.rados.0.smithi071.stdout:2303: done (2 left) 2024-09-11T01:44:26.165 INFO:tasks.rados.rados.0.smithi071.stdout:2304: write (excl) oid 16 current snap is 254 2024-09-11T01:44:26.165 INFO:tasks.rados.rados.0.smithi071.stdout:2304: seq_num 833 ranges {400231=564894,1661971=338260} 2024-09-11T01:44:26.169 INFO:tasks.rados.rados.0.smithi071.stdout:2304: writing smithi07134777-16 from 400231 to 965125 tid 1 2024-09-11T01:44:26.175 INFO:tasks.rados.rados.0.smithi071.stdout:2304: writing smithi07134777-16 from 1661971 to 2000231 tid 2 2024-09-11T01:44:26.175 INFO:tasks.rados.rados.0.smithi071.stdout:2305: write oid 22 current snap is 254 2024-09-11T01:44:26.175 INFO:tasks.rados.rados.0.smithi071.stdout:2305: seq_num 834 ranges {417038=640143,1720453=296585} 2024-09-11T01:44:26.182 INFO:tasks.rados.rados.0.smithi071.stdout:2305: writing smithi07134777-22 from 417038 to 1057181 tid 1 2024-09-11T01:44:26.188 INFO:tasks.rados.rados.0.smithi071.stdout:2305: writing smithi07134777-22 from 1720453 to 2017038 tid 2 2024-09-11T01:44:26.188 INFO:tasks.rados.rados.0.smithi071.stdout:2306: rollback oid 35 current snap is 254 2024-09-11T01:44:26.188 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 35 to 248 2024-09-11T01:44:26.188 INFO:tasks.rados.rados.0.smithi071.stdout:2300: finishing write tid 3 to smithi07134777-41 2024-09-11T01:44:26.189 INFO:tasks.rados.rados.0.smithi071.stdout:2300: finishing write tid 4 to smithi07134777-41 2024-09-11T01:44:26.189 INFO:tasks.rados.rados.0.smithi071.stdout:2300: finishing write tid 5 to smithi07134777-41 2024-09-11T01:44:26.189 INFO:tasks.rados.rados.0.smithi071.stdout:2300: oid 41 updating version 0 to 401 2024-09-11T01:44:26.189 INFO:tasks.rados.rados.0.smithi071.stdout:2300: oid 41 updating version 401 to 404 2024-09-11T01:44:26.189 INFO:tasks.rados.rados.0.smithi071.stdout:2300: oid 41 version 404 is already newer than 403 2024-09-11T01:44:26.189 INFO:tasks.rados.rados.0.smithi071.stdout:2300: oid 41 version 404 is already newer than 402 2024-09-11T01:44:26.189 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 404 (ObjNum 832 snap 254 seq_num 832) dirty exists 2024-09-11T01:44:26.189 INFO:tasks.rados.rados.0.smithi071.stdout:2300: left oid 41 (ObjNum 832 snap 254 seq_num 832) 2024-09-11T01:44:26.189 INFO:tasks.rados.rados.0.smithi071.stdout:2300: done (4 left) 2024-09-11T01:44:26.189 INFO:tasks.rados.rados.0.smithi071.stdout:2307: rollback oid 2 current snap is 254 2024-09-11T01:44:26.189 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 2 to 245 2024-09-11T01:44:26.189 INFO:tasks.rados.rados.0.smithi071.stdout:2301: finishing copy_from racing read to smithi07134777-15 2024-09-11T01:44:26.189 INFO:tasks.rados.rados.0.smithi071.stdout:2301: finishing copy_from to smithi07134777-15 2024-09-11T01:44:26.189 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 950 (ObjNum 808 snap 246 seq_num 808) dirty exists 2024-09-11T01:44:26.190 INFO:tasks.rados.rados.0.smithi071.stdout:2304: finishing write tid 1 to smithi07134777-16 2024-09-11T01:44:26.190 INFO:tasks.rados.rados.0.smithi071.stdout:2304: finishing write tid 2 to smithi07134777-16 2024-09-11T01:44:26.190 INFO:tasks.rados.rados.0.smithi071.stdout:2304: finishing write tid 3 to smithi07134777-16 2024-09-11T01:44:26.190 INFO:tasks.rados.rados.0.smithi071.stdout:2304: finishing write tid 4 to smithi07134777-16 2024-09-11T01:44:26.190 INFO:tasks.rados.rados.0.smithi071.stdout:2304: oid 16 updating version 0 to 1014 2024-09-11T01:44:26.190 INFO:tasks.rados.rados.0.smithi071.stdout:2304: oid 16 version 1014 is already newer than 1012 2024-09-11T01:44:26.190 INFO:tasks.rados.rados.0.smithi071.stdout:2304: oid 16 version 1014 is already newer than 1013 2024-09-11T01:44:26.190 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 1014 (ObjNum 833 snap 254 seq_num 833) dirty exists 2024-09-11T01:44:26.190 INFO:tasks.rados.rados.0.smithi071.stdout:2304: left oid 16 (ObjNum 833 snap 254 seq_num 833) 2024-09-11T01:44:26.190 INFO:tasks.rados.rados.0.smithi071.stdout:2305: finishing write tid 1 to smithi07134777-22 2024-09-11T01:44:26.190 INFO:tasks.rados.rados.0.smithi071.stdout:2301: done (4 left) 2024-09-11T01:44:26.190 INFO:tasks.rados.rados.0.smithi071.stdout:2304: done (3 left) 2024-09-11T01:44:26.190 INFO:tasks.rados.rados.0.smithi071.stdout:2308: write oid 26 current snap is 254 2024-09-11T01:44:26.190 INFO:tasks.rados.rados.0.smithi071.stdout:2308: seq_num 835 ranges {433845=715392,1662582=371263} 2024-09-11T01:44:26.194 INFO:tasks.rados.rados.0.smithi071.stdout:2308: writing smithi07134777-26 from 433845 to 1149237 tid 1 2024-09-11T01:44:26.199 INFO:tasks.rados.rados.0.smithi071.stdout:2308: writing smithi07134777-26 from 1662582 to 2033845 tid 2 2024-09-11T01:44:26.199 INFO:tasks.rados.rados.0.smithi071.stdout:2309: snap_remove snap 216 2024-09-11T01:44:27.069 INFO:tasks.rados.rados.0.smithi071.stdout:2309: done (4 left) 2024-09-11T01:44:27.069 INFO:tasks.rados.rados.0.smithi071.stdout:2310: read oid 7 snap -1 2024-09-11T01:44:27.069 INFO:tasks.rados.rados.0.smithi071.stdout:2310: expect deleted 2024-09-11T01:44:27.069 INFO:tasks.rados.rados.0.smithi071.stdout:2311: snap_create 2024-09-11T01:44:27.069 INFO:tasks.rados.rados.0.smithi071.stdout:2306: finishing rollback tid 1 to smithi07134777-35 2024-09-11T01:44:27.069 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 1000 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-11T01:44:27.069 INFO:tasks.rados.rados.0.smithi071.stdout:2305: finishing write tid 2 to smithi07134777-22 2024-09-11T01:44:27.070 INFO:tasks.rados.rados.0.smithi071.stdout:2307: finishing rollback tid 0 to smithi07134777-2 2024-09-11T01:44:27.070 INFO:tasks.rados.rados.0.smithi071.stdout:2305: finishing write tid 3 to smithi07134777-22 2024-09-11T01:44:27.070 INFO:tasks.rados.rados.0.smithi071.stdout:2307: finishing rollback tid 1 to smithi07134777-2 2024-09-11T01:44:27.070 INFO:tasks.rados.rados.0.smithi071.stdout:2305: finishing write tid 4 to smithi07134777-22 2024-09-11T01:44:27.070 INFO:tasks.rados.rados.0.smithi071.stdout:2305: oid 22 updating version 0 to 723 2024-09-11T01:44:27.070 INFO:tasks.rados.rados.0.smithi071.stdout:2305: oid 22 updating version 723 to 725 2024-09-11T01:44:27.070 INFO:tasks.rados.rados.0.smithi071.stdout:2305: oid 22 version 725 is already newer than 724 2024-09-11T01:44:27.070 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 725 (ObjNum 834 snap 254 seq_num 834) dirty exists 2024-09-11T01:44:27.070 INFO:tasks.rados.rados.0.smithi071.stdout:2305: left oid 22 (ObjNum 834 snap 254 seq_num 834) 2024-09-11T01:44:27.070 INFO:tasks.rados.rados.0.smithi071.stdout:2307: finishing rollback tid 2 to smithi07134777-2 2024-09-11T01:44:27.070 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 850 (ObjNum 631 snap 190 seq_num 631) dirty exists 2024-09-11T01:44:27.070 INFO:tasks.rados.rados.0.smithi071.stdout:2308: finishing write tid 1 to smithi07134777-26 2024-09-11T01:44:27.071 INFO:tasks.rados.rados.0.smithi071.stdout:2308: finishing write tid 2 to smithi07134777-26 2024-09-11T01:44:27.071 INFO:tasks.rados.rados.0.smithi071.stdout:2308: finishing write tid 3 to smithi07134777-26 2024-09-11T01:44:27.071 INFO:tasks.rados.rados.0.smithi071.stdout:2308: finishing write tid 4 to smithi07134777-26 2024-09-11T01:44:27.071 INFO:tasks.rados.rados.0.smithi071.stdout:2308: oid 26 updating version 0 to 1271 2024-09-11T01:44:27.071 INFO:tasks.rados.rados.0.smithi071.stdout:2308: oid 26 version 1271 is already newer than 1270 2024-09-11T01:44:27.071 INFO:tasks.rados.rados.0.smithi071.stdout:2308: oid 26 version 1271 is already newer than 1269 2024-09-11T01:44:27.071 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 1271 (ObjNum 835 snap 254 seq_num 835) dirty exists 2024-09-11T01:44:27.071 INFO:tasks.rados.rados.0.smithi071.stdout:2308: left oid 26 (ObjNum 835 snap 254 seq_num 835) 2024-09-11T01:44:28.074 INFO:tasks.rados.rados.0.smithi071.stdout:2305: done (5 left) 2024-09-11T01:44:28.074 INFO:tasks.rados.rados.0.smithi071.stdout:2306: done (4 left) 2024-09-11T01:44:28.075 INFO:tasks.rados.rados.0.smithi071.stdout:2307: done (3 left) 2024-09-11T01:44:28.075 INFO:tasks.rados.rados.0.smithi071.stdout:2308: done (2 left) 2024-09-11T01:44:28.075 INFO:tasks.rados.rados.0.smithi071.stdout:2310: done (1 left) 2024-09-11T01:44:28.075 INFO:tasks.rados.rados.0.smithi071.stdout:2311: done (0 left) 2024-09-11T01:44:28.075 INFO:tasks.rados.rados.0.smithi071.stdout:2312: read oid 34 snap 219 2024-09-11T01:44:28.075 INFO:tasks.rados.rados.0.smithi071.stdout:2312: expect (ObjNum 666 snap 202 seq_num 666) 2024-09-11T01:44:28.075 INFO:tasks.rados.rados.0.smithi071.stdout:2313: write oid 35 current snap is 255 2024-09-11T01:44:28.075 INFO:tasks.rados.rados.0.smithi071.stdout:2313: seq_num 836 ranges {450652=790641,1721064=329588} 2024-09-11T01:44:28.081 INFO:tasks.rados.rados.0.smithi071.stdout:2313: writing smithi07134777-35 from 450652 to 1241293 tid 1 2024-09-11T01:44:28.085 INFO:tasks.rados.rados.0.smithi071.stdout:2313: writing smithi07134777-35 from 1721064 to 2050652 tid 2 2024-09-11T01:44:28.085 INFO:tasks.rados.rados.0.smithi071.stdout:2314: rollback oid 40 current snap is 255 2024-09-11T01:44:28.086 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 40 to 209 2024-09-11T01:44:28.086 INFO:tasks.rados.rados.0.smithi071.stdout:2315: rollback oid 38 current snap is 255 2024-09-11T01:44:28.086 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 38 to 237 2024-09-11T01:44:28.086 INFO:tasks.rados.rados.0.smithi071.stdout:2316: copy_from oid 14 from oid 29 current snap is 255 2024-09-11T01:44:28.086 INFO:tasks.rados.rados.0.smithi071.stdout:2317: delete oid 3 current snap is 255 2024-09-11T01:44:28.086 INFO:tasks.rados.rados.0.smithi071.stdout:2312: expect (ObjNum 666 snap 202 seq_num 666) 2024-09-11T01:44:28.142 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:44:28.258 INFO:tasks.rados.rados.0.smithi071.stdout:2314: finishing rollback tid 1 to smithi07134777-40 2024-09-11T01:44:28.258 INFO:tasks.rados.rados.0.smithi071.stdout:2314: finishing rollback tid 2 to smithi07134777-40 2024-09-11T01:44:28.258 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 1052 (ObjNum 647 snap 199 seq_num 647) dirty exists 2024-09-11T01:44:28.258 INFO:tasks.rados.rados.0.smithi071.stdout:2313: finishing write tid 1 to smithi07134777-35 2024-09-11T01:44:28.258 INFO:tasks.rados.rados.0.smithi071.stdout:2313: finishing write tid 2 to smithi07134777-35 2024-09-11T01:44:28.258 INFO:tasks.rados.rados.0.smithi071.stdout:2313: finishing write tid 3 to smithi07134777-35 2024-09-11T01:44:28.258 INFO:tasks.rados.rados.0.smithi071.stdout:2313: finishing write tid 4 to smithi07134777-35 2024-09-11T01:44:28.258 INFO:tasks.rados.rados.0.smithi071.stdout:2313: oid 35 updating version 0 to 1013 2024-09-11T01:44:28.259 INFO:tasks.rados.rados.0.smithi071.stdout:2313: oid 35 version 1013 is already newer than 1012 2024-09-11T01:44:28.259 INFO:tasks.rados.rados.0.smithi071.stdout:2313: oid 35 version 1013 is already newer than 1011 2024-09-11T01:44:28.259 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 1013 (ObjNum 836 snap 255 seq_num 836) dirty exists 2024-09-11T01:44:28.259 INFO:tasks.rados.rados.0.smithi071.stdout:2313: left oid 35 (ObjNum 836 snap 255 seq_num 836) 2024-09-11T01:44:28.259 INFO:tasks.rados.rados.0.smithi071.stdout:2315: finishing rollback tid 1 to smithi07134777-38 2024-09-11T01:44:28.259 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 1265 (ObjNum 129 snap 0 seq_num 2371074160) dirty dne 2024-09-11T01:44:28.259 INFO:tasks.rados.rados.0.smithi071.stdout:2316: finishing copy_from racing read to smithi07134777-14 2024-09-11T01:44:28.259 INFO:tasks.rados.rados.0.smithi071.stdout:2312: done (5 left) 2024-09-11T01:44:28.259 INFO:tasks.rados.rados.0.smithi071.stdout:2313: done (4 left) 2024-09-11T01:44:28.259 INFO:tasks.rados.rados.0.smithi071.stdout:2314: done (3 left) 2024-09-11T01:44:28.259 INFO:tasks.rados.rados.0.smithi071.stdout:2315: done (2 left) 2024-09-11T01:44:28.260 INFO:tasks.rados.rados.0.smithi071.stdout:2317: done (1 left) 2024-09-11T01:44:28.260 INFO:tasks.rados.rados.0.smithi071.stdout:2318: read oid 23 snap -1 2024-09-11T01:44:28.260 INFO:tasks.rados.rados.0.smithi071.stdout:2318: expect (ObjNum 652 snap 199 seq_num 652) 2024-09-11T01:44:28.260 INFO:tasks.rados.rados.0.smithi071.stdout:2319: write (excl) oid 5 current snap is 255 2024-09-11T01:44:28.260 INFO:tasks.rados.rados.0.smithi071.stdout:2316: finishing copy_from to smithi07134777-14 2024-09-11T01:44:28.260 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 1328 (ObjNum 801 snap 243 seq_num 801) dirty exists 2024-09-11T01:44:28.260 INFO:tasks.rados.rados.0.smithi071.stdout:2319: seq_num 838 ranges {484266=657492,1554381=529885} 2024-09-11T01:44:28.263 INFO:tasks.rados.rados.0.smithi071.stdout:2319: writing smithi07134777-5 from 484266 to 1141758 tid 1 2024-09-11T01:44:28.269 INFO:tasks.rados.rados.0.smithi071.stdout:2319: writing smithi07134777-5 from 1554381 to 2084266 tid 2 2024-09-11T01:44:28.269 INFO:tasks.rados.rados.0.smithi071.stdout:2316: done (2 left) 2024-09-11T01:44:28.269 INFO:tasks.rados.rados.0.smithi071.stdout:2320: write (excl) oid 35 current snap is 255 2024-09-11T01:44:28.269 INFO:tasks.rados.rados.0.smithi071.stdout:2320: seq_num 839 ranges {501073=732741,1896510=204563} 2024-09-11T01:44:28.276 INFO:tasks.rados.rados.0.smithi071.stdout:2320: writing smithi07134777-35 from 501073 to 1233814 tid 1 2024-09-11T01:44:28.281 INFO:tasks.rados.rados.0.smithi071.stdout:2320: writing smithi07134777-35 from 1896510 to 2101073 tid 2 2024-09-11T01:44:28.281 INFO:tasks.rados.rados.0.smithi071.stdout:2321: rollback oid 8 current snap is 255 2024-09-11T01:44:28.281 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 8 to 246 2024-09-11T01:44:28.281 INFO:tasks.rados.rados.0.smithi071.stdout:2318: expect (ObjNum 652 snap 199 seq_num 652) 2024-09-11T01:44:28.312 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2103322 2024-09-11T01:44:28.442 INFO:tasks.rados.rados.0.smithi071.stdout:2319: finishing write tid 1 to smithi07134777-5 2024-09-11T01:44:28.443 INFO:tasks.rados.rados.0.smithi071.stdout:2319: finishing write tid 2 to smithi07134777-5 2024-09-11T01:44:28.443 INFO:tasks.rados.rados.0.smithi071.stdout:2319: finishing write tid 3 to smithi07134777-5 2024-09-11T01:44:28.443 INFO:tasks.rados.rados.0.smithi071.stdout:2319: finishing write tid 4 to smithi07134777-5 2024-09-11T01:44:28.443 INFO:tasks.rados.rados.0.smithi071.stdout:2319: oid 5 updating version 0 to 1156 2024-09-11T01:44:28.443 INFO:tasks.rados.rados.0.smithi071.stdout:2319: oid 5 updating version 1156 to 1157 2024-09-11T01:44:28.443 INFO:tasks.rados.rados.0.smithi071.stdout:2319: oid 5 updating version 1157 to 1158 2024-09-11T01:44:28.443 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 1158 (ObjNum 838 snap 255 seq_num 838) dirty exists 2024-09-11T01:44:28.443 INFO:tasks.rados.rados.0.smithi071.stdout:2319: left oid 5 (ObjNum 838 snap 255 seq_num 838) 2024-09-11T01:44:28.443 INFO:tasks.rados.rados.0.smithi071.stdout:2320: finishing write tid 1 to smithi07134777-35 2024-09-11T01:44:28.443 INFO:tasks.rados.rados.0.smithi071.stdout:2321: finishing rollback tid 0 to smithi07134777-8 2024-09-11T01:44:28.443 INFO:tasks.rados.rados.0.smithi071.stdout:2320: finishing write tid 2 to smithi07134777-35 2024-09-11T01:44:28.444 INFO:tasks.rados.rados.0.smithi071.stdout:2321: finishing rollback tid 1 to smithi07134777-8 2024-09-11T01:44:28.444 INFO:tasks.rados.rados.0.smithi071.stdout:2320: finishing write tid 3 to smithi07134777-35 2024-09-11T01:44:28.444 INFO:tasks.rados.rados.0.smithi071.stdout:2318: done (3 left) 2024-09-11T01:44:28.444 INFO:tasks.rados.rados.0.smithi071.stdout:2319: done (2 left) 2024-09-11T01:44:28.444 INFO:tasks.rados.rados.0.smithi071.stdout:2322: read oid 20 snap -1 2024-09-11T01:44:28.444 INFO:tasks.rados.rados.0.smithi071.stdout:2322: expect (ObjNum 776 snap 236 seq_num 776) 2024-09-11T01:44:28.444 INFO:tasks.rados.rados.0.smithi071.stdout:2323: write (excl) oid 17 current snap is 255 2024-09-11T01:44:28.444 INFO:tasks.rados.rados.0.smithi071.stdout:2320: finishing write tid 4 to smithi07134777-35 2024-09-11T01:44:28.444 INFO:tasks.rados.rados.0.smithi071.stdout:2320: oid 35 updating version 0 to 1015 2024-09-11T01:44:28.444 INFO:tasks.rados.rados.0.smithi071.stdout:2320: oid 35 version 1015 is already newer than 1014 2024-09-11T01:44:28.444 INFO:tasks.rados.rados.0.smithi071.stdout:2320: oid 35 updating version 1015 to 1016 2024-09-11T01:44:28.444 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 1016 (ObjNum 839 snap 255 seq_num 839) dirty exists 2024-09-11T01:44:28.444 INFO:tasks.rados.rados.0.smithi071.stdout:2320: left oid 35 (ObjNum 839 snap 255 seq_num 839) 2024-09-11T01:44:28.445 INFO:tasks.rados.rados.0.smithi071.stdout:2321: finishing rollback tid 2 to smithi07134777-8 2024-09-11T01:44:28.445 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 1051 (ObjNum 736 snap 226 seq_num 736) dirty exists 2024-09-11T01:44:28.445 INFO:tasks.rados.rados.0.smithi071.stdout:2323: seq_num 840 ranges {517880=407990,1554992=562888} 2024-09-11T01:44:28.448 INFO:tasks.rados.rados.0.smithi071.stdout:2323: writing smithi07134777-17 from 517880 to 925870 tid 1 2024-09-11T01:44:28.454 INFO:tasks.rados.rados.0.smithi071.stdout:2323: writing smithi07134777-17 from 1554992 to 2117880 tid 2 2024-09-11T01:44:28.454 INFO:tasks.rados.rados.0.smithi071.stdout:2320: done (3 left) 2024-09-11T01:44:28.454 INFO:tasks.rados.rados.0.smithi071.stdout:2321: done (2 left) 2024-09-11T01:44:28.455 INFO:tasks.rados.rados.0.smithi071.stdout:2324: write oid 46 current snap is 255 2024-09-11T01:44:28.455 INFO:tasks.rados.rados.0.smithi071.stdout:2324: seq_num 841 ranges {534687=483239,1613474=412283} 2024-09-11T01:44:28.460 INFO:tasks.rados.rados.0.smithi071.stdout:2324: writing smithi07134777-46 from 534687 to 1017926 tid 1 2024-09-11T01:44:28.466 INFO:tasks.rados.rados.0.smithi071.stdout:2324: writing smithi07134777-46 from 1613474 to 2025757 tid 2 2024-09-11T01:44:28.466 INFO:tasks.rados.rados.0.smithi071.stdout:2325: write oid 14 current snap is 255 2024-09-11T01:44:28.466 INFO:tasks.rados.rados.0.smithi071.stdout:2325: seq_num 842 ranges {551494=558488,1671956=479538} 2024-09-11T01:44:28.472 INFO:tasks.rados.rados.0.smithi071.stdout:2325: writing smithi07134777-14 from 551494 to 1109982 tid 1 2024-09-11T01:44:28.477 INFO:tasks.rados.rados.0.smithi071.stdout:2325: writing smithi07134777-14 from 1671956 to 2151494 tid 2 2024-09-11T01:44:28.478 INFO:tasks.rados.rados.0.smithi071.stdout:2326: snap_remove snap 237 2024-09-11T01:44:29.176 INFO:tasks.rados.rados.0.smithi071.stdout:2326: done (4 left) 2024-09-11T01:44:29.176 INFO:tasks.rados.rados.0.smithi071.stdout:2327: write oid 21 current snap is 255 2024-09-11T01:44:29.176 INFO:tasks.rados.rados.0.smithi071.stdout:2327: seq_num 843 ranges {568301=633737,1614085=415952} 2024-09-11T01:44:29.183 INFO:tasks.rados.rados.0.smithi071.stdout:2327: writing smithi07134777-21 from 568301 to 1202038 tid 1 2024-09-11T01:44:29.188 INFO:tasks.rados.rados.0.smithi071.stdout:2327: writing smithi07134777-21 from 1614085 to 2030037 tid 2 2024-09-11T01:44:29.188 INFO:tasks.rados.rados.0.smithi071.stdout:2328: write (excl) oid 40 current snap is 255 2024-09-11T01:44:29.188 INFO:tasks.rados.rados.0.smithi071.stdout:2328: seq_num 844 ranges {585108=425339,1788920=396188} 2024-09-11T01:44:29.194 INFO:tasks.rados.rados.0.smithi071.stdout:2328: writing smithi07134777-40 from 585108 to 1010447 tid 1 2024-09-11T01:44:29.201 INFO:tasks.rados.rados.0.smithi071.stdout:2328: writing smithi07134777-40 from 1788920 to 2185108 tid 2 2024-09-11T01:44:29.201 INFO:tasks.rados.rados.0.smithi071.stdout:2329: snap_remove snap 209 2024-09-11T01:44:30.179 INFO:tasks.rados.rados.0.smithi071.stdout:2329: done (6 left) 2024-09-11T01:44:30.179 INFO:tasks.rados.rados.0.smithi071.stdout:2330: rollback oid 38 current snap is 255 2024-09-11T01:44:30.179 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 38 to 235 2024-09-11T01:44:30.179 INFO:tasks.rados.rados.0.smithi071.stdout:2322: expect (ObjNum 776 snap 236 seq_num 776) 2024-09-11T01:44:30.343 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2272798 2024-09-11T01:44:30.362 INFO:tasks.rados.rados.0.smithi071.stdout:2323: finishing write tid 1 to smithi07134777-17 2024-09-11T01:44:30.362 INFO:tasks.rados.rados.0.smithi071.stdout:2323: finishing write tid 2 to smithi07134777-17 2024-09-11T01:44:30.363 INFO:tasks.rados.rados.0.smithi071.stdout:2323: finishing write tid 3 to smithi07134777-17 2024-09-11T01:44:30.363 INFO:tasks.rados.rados.0.smithi071.stdout:2323: finishing write tid 4 to smithi07134777-17 2024-09-11T01:44:30.363 INFO:tasks.rados.rados.0.smithi071.stdout:2323: oid 17 updating version 0 to 1008 2024-09-11T01:44:30.363 INFO:tasks.rados.rados.0.smithi071.stdout:2323: oid 17 updating version 1008 to 1010 2024-09-11T01:44:30.363 INFO:tasks.rados.rados.0.smithi071.stdout:2323: oid 17 version 1010 is already newer than 1009 2024-09-11T01:44:30.363 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 1010 (ObjNum 840 snap 255 seq_num 840) dirty exists 2024-09-11T01:44:30.363 INFO:tasks.rados.rados.0.smithi071.stdout:2323: left oid 17 (ObjNum 840 snap 255 seq_num 840) 2024-09-11T01:44:30.363 INFO:tasks.rados.rados.0.smithi071.stdout:2322: done (6 left) 2024-09-11T01:44:30.363 INFO:tasks.rados.rados.0.smithi071.stdout:2323: done (5 left) 2024-09-11T01:44:30.363 INFO:tasks.rados.rados.0.smithi071.stdout:2331: write (excl) oid 30 current snap is 255 2024-09-11T01:44:30.363 INFO:tasks.rados.rados.0.smithi071.stdout:2331: seq_num 845 ranges {601915=500588,1847402=354513} 2024-09-11T01:44:30.368 INFO:tasks.rados.rados.0.smithi071.stdout:2331: writing smithi07134777-30 from 601915 to 1102503 tid 1 2024-09-11T01:44:30.375 INFO:tasks.rados.rados.0.smithi071.stdout:2331: writing smithi07134777-30 from 1847402 to 2201915 tid 2 2024-09-11T01:44:30.375 INFO:tasks.rados.rados.0.smithi071.stdout:2332: delete oid 11 current snap is 255 2024-09-11T01:44:30.375 INFO:tasks.rados.rados.0.smithi071.stdout:2324: finishing write tid 1 to smithi07134777-46 2024-09-11T01:44:30.375 INFO:tasks.rados.rados.0.smithi071.stdout:2324: finishing write tid 2 to smithi07134777-46 2024-09-11T01:44:30.375 INFO:tasks.rados.rados.0.smithi071.stdout:2324: finishing write tid 3 to smithi07134777-46 2024-09-11T01:44:30.375 INFO:tasks.rados.rados.0.smithi071.stdout:2324: finishing write tid 4 to smithi07134777-46 2024-09-11T01:44:30.376 INFO:tasks.rados.rados.0.smithi071.stdout:2324: oid 46 updating version 0 to 1053 2024-09-11T01:44:30.376 INFO:tasks.rados.rados.0.smithi071.stdout:2324: oid 46 updating version 1053 to 1055 2024-09-11T01:44:30.376 INFO:tasks.rados.rados.0.smithi071.stdout:2324: oid 46 version 1055 is already newer than 1054 2024-09-11T01:44:30.376 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 1055 (ObjNum 841 snap 255 seq_num 841) dirty exists 2024-09-11T01:44:30.376 INFO:tasks.rados.rados.0.smithi071.stdout:2324: left oid 46 (ObjNum 841 snap 255 seq_num 841) 2024-09-11T01:44:30.376 INFO:tasks.rados.rados.0.smithi071.stdout:2325: finishing write tid 1 to smithi07134777-14 2024-09-11T01:44:30.376 INFO:tasks.rados.rados.0.smithi071.stdout:2325: finishing write tid 2 to smithi07134777-14 2024-09-11T01:44:30.376 INFO:tasks.rados.rados.0.smithi071.stdout:2325: finishing write tid 3 to smithi07134777-14 2024-09-11T01:44:30.376 INFO:tasks.rados.rados.0.smithi071.stdout:2325: finishing write tid 4 to smithi07134777-14 2024-09-11T01:44:30.376 INFO:tasks.rados.rados.0.smithi071.stdout:2325: oid 14 updating version 0 to 1437 2024-09-11T01:44:30.376 INFO:tasks.rados.rados.0.smithi071.stdout:2325: oid 14 version 1437 is already newer than 1436 2024-09-11T01:44:30.376 INFO:tasks.rados.rados.0.smithi071.stdout:2325: oid 14 version 1437 is already newer than 1435 2024-09-11T01:44:30.376 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 1437 (ObjNum 842 snap 255 seq_num 842) dirty exists 2024-09-11T01:44:30.377 INFO:tasks.rados.rados.0.smithi071.stdout:2325: left oid 14 (ObjNum 842 snap 255 seq_num 842) 2024-09-11T01:44:30.377 INFO:tasks.rados.rados.0.smithi071.stdout:2327: finishing write tid 1 to smithi07134777-21 2024-09-11T01:44:30.377 INFO:tasks.rados.rados.0.smithi071.stdout:2328: finishing write tid 1 to smithi07134777-40 2024-09-11T01:44:30.377 INFO:tasks.rados.rados.0.smithi071.stdout:2327: finishing write tid 2 to smithi07134777-21 2024-09-11T01:44:30.377 INFO:tasks.rados.rados.0.smithi071.stdout:2327: finishing write tid 3 to smithi07134777-21 2024-09-11T01:44:30.377 INFO:tasks.rados.rados.0.smithi071.stdout:2327: finishing write tid 4 to smithi07134777-21 2024-09-11T01:44:30.377 INFO:tasks.rados.rados.0.smithi071.stdout:2327: oid 21 updating version 0 to 1031 2024-09-11T01:44:30.377 INFO:tasks.rados.rados.0.smithi071.stdout:2327: oid 21 version 1031 is already newer than 1030 2024-09-11T01:44:30.377 INFO:tasks.rados.rados.0.smithi071.stdout:2327: oid 21 updating version 1031 to 1032 2024-09-11T01:44:30.377 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 1032 (ObjNum 843 snap 255 seq_num 843) dirty exists 2024-09-11T01:44:30.377 INFO:tasks.rados.rados.0.smithi071.stdout:2327: left oid 21 (ObjNum 843 snap 255 seq_num 843) 2024-09-11T01:44:30.377 INFO:tasks.rados.rados.0.smithi071.stdout:2328: finishing write tid 2 to smithi07134777-40 2024-09-11T01:44:30.377 INFO:tasks.rados.rados.0.smithi071.stdout:2328: finishing write tid 3 to smithi07134777-40 2024-09-11T01:44:30.377 INFO:tasks.rados.rados.0.smithi071.stdout:2328: finishing write tid 4 to smithi07134777-40 2024-09-11T01:44:30.378 INFO:tasks.rados.rados.0.smithi071.stdout:2328: oid 40 updating version 0 to 1058 2024-09-11T01:44:30.378 INFO:tasks.rados.rados.0.smithi071.stdout:2328: oid 40 version 1058 is already newer than 1057 2024-09-11T01:44:30.378 INFO:tasks.rados.rados.0.smithi071.stdout:2328: oid 40 version 1058 is already newer than 1056 2024-09-11T01:44:30.378 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 1058 (ObjNum 844 snap 255 seq_num 844) dirty exists 2024-09-11T01:44:30.378 INFO:tasks.rados.rados.0.smithi071.stdout:2328: left oid 40 (ObjNum 844 snap 255 seq_num 844) 2024-09-11T01:44:30.378 INFO:tasks.rados.rados.0.smithi071.stdout:2330: finishing rollback tid 1 to smithi07134777-38 2024-09-11T01:44:30.378 INFO:tasks.rados.rados.0.smithi071.stdout:2330: finishing rollback tid 2 to smithi07134777-38 2024-09-11T01:44:30.378 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 1449 (ObjNum 542 snap 154 seq_num 542) dirty exists 2024-09-11T01:44:30.378 INFO:tasks.rados.rados.0.smithi071.stdout:2331: finishing write tid 1 to smithi07134777-30 2024-09-11T01:44:30.379 INFO:tasks.rados.rados.0.smithi071.stdout:2331: finishing write tid 2 to smithi07134777-30 2024-09-11T01:44:30.380 INFO:tasks.rados.rados.0.smithi071.stdout:2331: finishing write tid 3 to smithi07134777-30 2024-09-11T01:44:30.380 INFO:tasks.rados.rados.0.smithi071.stdout:2331: finishing write tid 4 to smithi07134777-30 2024-09-11T01:44:30.380 INFO:tasks.rados.rados.0.smithi071.stdout:2331: oid 30 updating version 0 to 985 2024-09-11T01:44:30.380 INFO:tasks.rados.rados.0.smithi071.stdout:2331: oid 30 updating version 985 to 987 2024-09-11T01:44:30.380 INFO:tasks.rados.rados.0.smithi071.stdout:2331: oid 30 version 987 is already newer than 986 2024-09-11T01:44:30.380 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 987 (ObjNum 845 snap 255 seq_num 845) dirty exists 2024-09-11T01:44:30.381 INFO:tasks.rados.rados.0.smithi071.stdout:2331: left oid 30 (ObjNum 845 snap 255 seq_num 845) 2024-09-11T01:44:30.381 INFO:tasks.rados.rados.0.smithi071.stdout:2324: done (6 left) 2024-09-11T01:44:30.381 INFO:tasks.rados.rados.0.smithi071.stdout:2325: done (5 left) 2024-09-11T01:44:30.381 INFO:tasks.rados.rados.0.smithi071.stdout:2327: done (4 left) 2024-09-11T01:44:30.381 INFO:tasks.rados.rados.0.smithi071.stdout:2328: done (3 left) 2024-09-11T01:44:30.381 INFO:tasks.rados.rados.0.smithi071.stdout:2330: done (2 left) 2024-09-11T01:44:30.381 INFO:tasks.rados.rados.0.smithi071.stdout:2331: done (1 left) 2024-09-11T01:44:30.381 INFO:tasks.rados.rados.0.smithi071.stdout:2332: done (0 left) 2024-09-11T01:44:30.381 INFO:tasks.rados.rados.0.smithi071.stdout:2333: write (excl) oid 11 current snap is 255 2024-09-11T01:44:30.381 INFO:tasks.rados.rados.0.smithi071.stdout:2333: seq_num 847 ranges {635529=651086,1848013=387516} 2024-09-11T01:44:30.386 INFO:tasks.rados.rados.0.smithi071.stdout:2333: writing smithi07134777-11 from 635529 to 1286615 tid 1 2024-09-11T01:44:30.392 INFO:tasks.rados.rados.0.smithi071.stdout:2333: writing smithi07134777-11 from 1848013 to 2235529 tid 2 2024-09-11T01:44:30.392 INFO:tasks.rados.rados.0.smithi071.stdout:2334: read oid 44 snap -1 2024-09-11T01:44:30.392 INFO:tasks.rados.rados.0.smithi071.stdout:2334: expect (ObjNum 818 snap 251 seq_num 818) 2024-09-11T01:44:30.392 INFO:tasks.rados.rados.0.smithi071.stdout:2335: write (excl) oid 43 current snap is 255 2024-09-11T01:44:30.392 INFO:tasks.rados.rados.0.smithi071.stdout:2335: seq_num 848 ranges {652336=726335,1906495=345841} 2024-09-11T01:44:30.400 INFO:tasks.rados.rados.0.smithi071.stdout:2335: writing smithi07134777-43 from 652336 to 1378671 tid 1 2024-09-11T01:44:30.405 INFO:tasks.rados.rados.0.smithi071.stdout:2335: writing smithi07134777-43 from 1906495 to 2252336 tid 2 2024-09-11T01:44:30.405 INFO:tasks.rados.rados.0.smithi071.stdout:2336: snap_create 2024-09-11T01:44:30.405 INFO:tasks.rados.rados.0.smithi071.stdout:2333: finishing write tid 1 to smithi07134777-11 2024-09-11T01:44:30.405 INFO:tasks.rados.rados.0.smithi071.stdout:2334: expect (ObjNum 818 snap 251 seq_num 818) 2024-09-11T01:44:30.486 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:44:30.568 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:44:30.649 INFO:tasks.rados.rados.0.smithi071.stdout:2333: finishing write tid 2 to smithi07134777-11 2024-09-11T01:44:30.649 INFO:tasks.rados.rados.0.smithi071.stdout:2333: finishing write tid 3 to smithi07134777-11 2024-09-11T01:44:30.649 INFO:tasks.rados.rados.0.smithi071.stdout:2333: finishing write tid 4 to smithi07134777-11 2024-09-11T01:44:30.649 INFO:tasks.rados.rados.0.smithi071.stdout:2333: oid 11 updating version 0 to 990 2024-09-11T01:44:30.649 INFO:tasks.rados.rados.0.smithi071.stdout:2333: oid 11 version 990 is already newer than 989 2024-09-11T01:44:30.649 INFO:tasks.rados.rados.0.smithi071.stdout:2333: oid 11 version 990 is already newer than 988 2024-09-11T01:44:30.650 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 990 (ObjNum 847 snap 255 seq_num 847) dirty exists 2024-09-11T01:44:30.650 INFO:tasks.rados.rados.0.smithi071.stdout:2333: left oid 11 (ObjNum 847 snap 255 seq_num 847) 2024-09-11T01:44:30.650 INFO:tasks.rados.rados.0.smithi071.stdout:2335: finishing write tid 1 to smithi07134777-43 2024-09-11T01:44:30.650 INFO:tasks.rados.rados.0.smithi071.stdout:2335: finishing write tid 2 to smithi07134777-43 2024-09-11T01:44:30.650 INFO:tasks.rados.rados.0.smithi071.stdout:2335: finishing write tid 3 to smithi07134777-43 2024-09-11T01:44:30.650 INFO:tasks.rados.rados.0.smithi071.stdout:2335: finishing write tid 4 to smithi07134777-43 2024-09-11T01:44:30.650 INFO:tasks.rados.rados.0.smithi071.stdout:2335: oid 43 updating version 0 to 1301 2024-09-11T01:44:30.650 INFO:tasks.rados.rados.0.smithi071.stdout:2335: oid 43 updating version 1301 to 1302 2024-09-11T01:44:30.650 INFO:tasks.rados.rados.0.smithi071.stdout:2335: oid 43 updating version 1302 to 1303 2024-09-11T01:44:30.650 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 1303 (ObjNum 848 snap 255 seq_num 848) dirty exists 2024-09-11T01:44:30.650 INFO:tasks.rados.rados.0.smithi071.stdout:2335: left oid 43 (ObjNum 848 snap 255 seq_num 848) 2024-09-11T01:44:31.183 INFO:tasks.rados.rados.0.smithi071.stdout:2333: done (3 left) 2024-09-11T01:44:31.183 INFO:tasks.rados.rados.0.smithi071.stdout:2334: done (2 left) 2024-09-11T01:44:31.183 INFO:tasks.rados.rados.0.smithi071.stdout:2335: done (1 left) 2024-09-11T01:44:31.183 INFO:tasks.rados.rados.0.smithi071.stdout:2336: done (0 left) 2024-09-11T01:44:31.183 INFO:tasks.rados.rados.0.smithi071.stdout:2337: delete oid 14 current snap is 256 2024-09-11T01:44:31.188 INFO:tasks.rados.rados.0.smithi071.stdout:2337: done (0 left) 2024-09-11T01:44:31.188 INFO:tasks.rados.rados.0.smithi071.stdout:2338: write (excl) oid 38 current snap is 256 2024-09-11T01:44:31.188 INFO:tasks.rados.rados.0.smithi071.stdout:2338: seq_num 850 ranges {685950=476833,1623459=570617} 2024-09-11T01:44:31.194 INFO:tasks.rados.rados.0.smithi071.stdout:2338: writing smithi07134777-38 from 685950 to 1162783 tid 1 2024-09-11T01:44:31.200 INFO:tasks.rados.rados.0.smithi071.stdout:2338: writing smithi07134777-38 from 1623459 to 2194076 tid 2 2024-09-11T01:44:31.200 INFO:tasks.rados.rados.0.smithi071.stdout:2339: read oid 34 snap 212 2024-09-11T01:44:31.200 INFO:tasks.rados.rados.0.smithi071.stdout:2339: expect (ObjNum 666 snap 202 seq_num 666) 2024-09-11T01:44:31.200 INFO:tasks.rados.rados.0.smithi071.stdout:2340: write oid 10 current snap is 256 2024-09-11T01:44:31.201 INFO:tasks.rados.rados.0.smithi071.stdout:2338: finishing write tid 1 to smithi07134777-38 2024-09-11T01:44:31.201 INFO:tasks.rados.rados.0.smithi071.stdout:2340: seq_num 851 ranges {702757=552082,1681941=430628} 2024-09-11T01:44:31.207 INFO:tasks.rados.rados.0.smithi071.stdout:2340: writing smithi07134777-10 from 702757 to 1254839 tid 1 2024-09-11T01:44:31.212 INFO:tasks.rados.rados.0.smithi071.stdout:2340: writing smithi07134777-10 from 1681941 to 2112569 tid 2 2024-09-11T01:44:31.213 INFO:tasks.rados.rados.0.smithi071.stdout:2341: copy_from oid 45 from oid 15 current snap is 256 2024-09-11T01:44:31.213 INFO:tasks.rados.rados.0.smithi071.stdout:2338: finishing write tid 2 to smithi07134777-38 2024-09-11T01:44:31.213 INFO:tasks.rados.rados.0.smithi071.stdout:2338: finishing write tid 3 to smithi07134777-38 2024-09-11T01:44:31.213 INFO:tasks.rados.rados.0.smithi071.stdout:2338: finishing write tid 4 to smithi07134777-38 2024-09-11T01:44:31.213 INFO:tasks.rados.rados.0.smithi071.stdout:2338: oid 38 updating version 0 to 1465 2024-09-11T01:44:31.213 INFO:tasks.rados.rados.0.smithi071.stdout:2338: oid 38 updating version 1465 to 1466 2024-09-11T01:44:31.213 INFO:tasks.rados.rados.0.smithi071.stdout:2338: oid 38 updating version 1466 to 1467 2024-09-11T01:44:31.213 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 1467 (ObjNum 850 snap 256 seq_num 850) dirty exists 2024-09-11T01:44:31.213 INFO:tasks.rados.rados.0.smithi071.stdout:2338: left oid 38 (ObjNum 850 snap 256 seq_num 850) 2024-09-11T01:44:31.213 INFO:tasks.rados.rados.0.smithi071.stdout:2339: expect (ObjNum 666 snap 202 seq_num 666) 2024-09-11T01:44:31.256 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3115237 2024-09-11T01:44:31.333 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1022401 2024-09-11T01:44:31.379 INFO:tasks.rados.rados.0.smithi071.stdout:2340: finishing write tid 1 to smithi07134777-10 2024-09-11T01:44:31.379 INFO:tasks.rados.rados.0.smithi071.stdout:2340: finishing write tid 2 to smithi07134777-10 2024-09-11T01:44:31.379 INFO:tasks.rados.rados.0.smithi071.stdout:2340: finishing write tid 3 to smithi07134777-10 2024-09-11T01:44:31.379 INFO:tasks.rados.rados.0.smithi071.stdout:2340: finishing write tid 4 to smithi07134777-10 2024-09-11T01:44:31.379 INFO:tasks.rados.rados.0.smithi071.stdout:2340: oid 10 updating version 0 to 407 2024-09-11T01:44:31.380 INFO:tasks.rados.rados.0.smithi071.stdout:2340: oid 10 version 407 is already newer than 406 2024-09-11T01:44:31.380 INFO:tasks.rados.rados.0.smithi071.stdout:2340: oid 10 version 407 is already newer than 405 2024-09-11T01:44:31.380 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 407 (ObjNum 851 snap 256 seq_num 851) dirty exists 2024-09-11T01:44:31.380 INFO:tasks.rados.rados.0.smithi071.stdout:2340: left oid 10 (ObjNum 851 snap 256 seq_num 851) 2024-09-11T01:44:31.380 INFO:tasks.rados.rados.0.smithi071.stdout:2341: finishing copy_from racing read to smithi07134777-45 2024-09-11T01:44:31.380 INFO:tasks.rados.rados.0.smithi071.stdout:2341: finishing copy_from to smithi07134777-45 2024-09-11T01:44:31.380 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 950 (ObjNum 808 snap 246 seq_num 808) dirty exists 2024-09-11T01:44:31.380 INFO:tasks.rados.rados.0.smithi071.stdout:2338: done (3 left) 2024-09-11T01:44:31.380 INFO:tasks.rados.rados.0.smithi071.stdout:2339: done (2 left) 2024-09-11T01:44:31.380 INFO:tasks.rados.rados.0.smithi071.stdout:2340: done (1 left) 2024-09-11T01:44:31.381 INFO:tasks.rados.rados.0.smithi071.stdout:2341: done (0 left) 2024-09-11T01:44:31.381 INFO:tasks.rados.rados.0.smithi071.stdout:2342: read oid 34 snap 251 2024-09-11T01:44:31.381 INFO:tasks.rados.rados.0.smithi071.stdout:2342: expect (ObjNum 811 snap 246 seq_num 811) 2024-09-11T01:44:31.381 INFO:tasks.rados.rados.0.smithi071.stdout:2343: delete oid 39 current snap is 256 2024-09-11T01:44:31.386 INFO:tasks.rados.rados.0.smithi071.stdout:2343: done (1 left) 2024-09-11T01:44:31.386 INFO:tasks.rados.rados.0.smithi071.stdout:2344: read oid 15 snap 239 2024-09-11T01:44:31.386 INFO:tasks.rados.rados.0.smithi071.stdout:2344: expect (ObjNum 565 snap 160 seq_num 565) 2024-09-11T01:44:31.386 INFO:tasks.rados.rados.0.smithi071.stdout:2345: snap_create 2024-09-11T01:44:31.390 INFO:tasks.rados.rados.0.smithi071.stdout:2342: expect (ObjNum 811 snap 246 seq_num 811) 2024-09-11T01:44:31.391 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:44:31.515 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3158841 2024-09-11T01:44:31.556 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2069853 2024-09-11T01:44:31.593 INFO:tasks.rados.rados.0.smithi071.stdout:2344: expect (ObjNum 565 snap 160 seq_num 565) 2024-09-11T01:44:32.067 INFO:tasks.rados.rados.0.smithi071.stdout:2342: done (2 left) 2024-09-11T01:44:32.067 INFO:tasks.rados.rados.0.smithi071.stdout:2344: done (1 left) 2024-09-11T01:44:32.068 INFO:tasks.rados.rados.0.smithi071.stdout:2345: done (0 left) 2024-09-11T01:44:32.068 INFO:tasks.rados.rados.0.smithi071.stdout:2346: delete oid 50 current snap is 257 2024-09-11T01:44:32.073 INFO:tasks.rados.rados.0.smithi071.stdout:2346: done (0 left) 2024-09-11T01:44:32.073 INFO:tasks.rados.rados.0.smithi071.stdout:2347: snap_remove snap 255 2024-09-11T01:44:33.070 INFO:tasks.rados.rados.0.smithi071.stdout:2347: done (0 left) 2024-09-11T01:44:33.071 INFO:tasks.rados.rados.0.smithi071.stdout:2348: read oid 34 snap 174 2024-09-11T01:44:33.071 INFO:tasks.rados.rados.0.smithi071.stdout:2348: expect (ObjNum 566 snap 160 seq_num 566) 2024-09-11T01:44:33.071 INFO:tasks.rados.rados.0.smithi071.stdout:2349: rollback oid 31 current snap is 257 2024-09-11T01:44:33.071 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 31 to 252 2024-09-11T01:44:33.071 INFO:tasks.rados.rados.0.smithi071.stdout:2350: copy_from oid 21 from oid 13 current snap is 257 2024-09-11T01:44:33.071 INFO:tasks.rados.rados.0.smithi071.stdout:2351: write (excl) oid 45 current snap is 257 2024-09-11T01:44:33.071 INFO:tasks.rados.rados.0.smithi071.stdout:2351: seq_num 854 ranges {753178=494182,1857387=495791} 2024-09-11T01:44:33.078 INFO:tasks.rados.rados.0.smithi071.stdout:2351: writing smithi07134777-45 from 753178 to 1247360 tid 1 2024-09-11T01:44:33.085 INFO:tasks.rados.rados.0.smithi071.stdout:2351: writing smithi07134777-45 from 1857387 to 2353178 tid 2 2024-09-11T01:44:33.085 INFO:tasks.rados.rados.0.smithi071.stdout:2352: snap_create 2024-09-11T01:44:33.085 INFO:tasks.rados.rados.0.smithi071.stdout:2349: finishing rollback tid 0 to smithi07134777-31 2024-09-11T01:44:33.085 INFO:tasks.rados.rados.0.smithi071.stdout:2349: finishing rollback tid 1 to smithi07134777-31 2024-09-11T01:44:33.085 INFO:tasks.rados.rados.0.smithi071.stdout:2349: finishing rollback tid 2 to smithi07134777-31 2024-09-11T01:44:33.086 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 1028 (ObjNum 737 snap 226 seq_num 737) dirty exists 2024-09-11T01:44:33.086 INFO:tasks.rados.rados.0.smithi071.stdout:2348: expect (ObjNum 566 snap 160 seq_num 566) 2024-09-11T01:44:33.144 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 925711 2024-09-11T01:44:33.217 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1754241 2024-09-11T01:44:33.248 INFO:tasks.rados.rados.0.smithi071.stdout:2350: finishing copy_from racing read to smithi07134777-21 2024-09-11T01:44:33.248 INFO:tasks.rados.rados.0.smithi071.stdout:2351: finishing write tid 1 to smithi07134777-45 2024-09-11T01:44:33.248 INFO:tasks.rados.rados.0.smithi071.stdout:2350: finishing copy_from to smithi07134777-21 2024-09-11T01:44:33.248 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 950 (ObjNum 808 snap 246 seq_num 808) dirty exists 2024-09-11T01:44:33.248 INFO:tasks.rados.rados.0.smithi071.stdout:2351: finishing write tid 2 to smithi07134777-45 2024-09-11T01:44:33.248 INFO:tasks.rados.rados.0.smithi071.stdout:2351: finishing write tid 3 to smithi07134777-45 2024-09-11T01:44:33.248 INFO:tasks.rados.rados.0.smithi071.stdout:2351: finishing write tid 4 to smithi07134777-45 2024-09-11T01:44:33.248 INFO:tasks.rados.rados.0.smithi071.stdout:2351: oid 45 updating version 0 to 1052 2024-09-11T01:44:33.249 INFO:tasks.rados.rados.0.smithi071.stdout:2351: oid 45 updating version 1052 to 1055 2024-09-11T01:44:33.249 INFO:tasks.rados.rados.0.smithi071.stdout:2351: oid 45 updating version 1055 to 1056 2024-09-11T01:44:33.249 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 1056 (ObjNum 854 snap 257 seq_num 854) dirty exists 2024-09-11T01:44:33.249 INFO:tasks.rados.rados.0.smithi071.stdout:2351: left oid 45 (ObjNum 854 snap 257 seq_num 854) 2024-09-11T01:44:34.070 INFO:tasks.rados.rados.0.smithi071.stdout:2348: done (4 left) 2024-09-11T01:44:34.070 INFO:tasks.rados.rados.0.smithi071.stdout:2349: done (3 left) 2024-09-11T01:44:34.070 INFO:tasks.rados.rados.0.smithi071.stdout:2350: done (2 left) 2024-09-11T01:44:34.070 INFO:tasks.rados.rados.0.smithi071.stdout:2351: done (1 left) 2024-09-11T01:44:34.070 INFO:tasks.rados.rados.0.smithi071.stdout:2352: done (0 left) 2024-09-11T01:44:34.070 INFO:tasks.rados.rados.0.smithi071.stdout:2353: delete oid 30 current snap is 258 2024-09-11T01:44:34.078 INFO:tasks.rados.rados.0.smithi071.stdout:2353: done (0 left) 2024-09-11T01:44:34.078 INFO:tasks.rados.rados.0.smithi071.stdout:2354: read oid 12 snap 253 2024-09-11T01:44:34.078 INFO:tasks.rados.rados.0.smithi071.stdout:2354: expect deleted 2024-09-11T01:44:34.078 INFO:tasks.rados.rados.0.smithi071.stdout:2355: copy_from oid 11 from oid 33 current snap is 258 2024-09-11T01:44:34.078 INFO:tasks.rados.rados.0.smithi071.stdout:2356: snap_create 2024-09-11T01:44:34.094 INFO:tasks.rados.rados.0.smithi071.stdout:2355: finishing copy_from racing read to smithi07134777-11 2024-09-11T01:44:34.097 INFO:tasks.rados.rados.0.smithi071.stdout:2355: finishing copy_from to smithi07134777-11 2024-09-11T01:44:34.097 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 1195 (ObjNum 816 snap 250 seq_num 816) dirty exists 2024-09-11T01:44:35.080 INFO:tasks.rados.rados.0.smithi071.stdout:2354: done (2 left) 2024-09-11T01:44:35.081 INFO:tasks.rados.rados.0.smithi071.stdout:2355: done (1 left) 2024-09-11T01:44:35.081 INFO:tasks.rados.rados.0.smithi071.stdout:2356: done (0 left) 2024-09-11T01:44:35.081 INFO:tasks.rados.rados.0.smithi071.stdout:2357: snap_create 2024-09-11T01:44:36.082 INFO:tasks.rados.rados.0.smithi071.stdout:2357: done (0 left) 2024-09-11T01:44:36.083 INFO:tasks.rados.rados.0.smithi071.stdout:2358: snap_remove snap 231 2024-09-11T01:44:37.067 INFO:tasks.rados.rados.0.smithi071.stdout:2358: done (0 left) 2024-09-11T01:44:37.067 INFO:tasks.rados.rados.0.smithi071.stdout:2359: snap_remove snap 246 2024-09-11T01:44:38.075 INFO:tasks.rados.rados.0.smithi071.stdout:2359: done (0 left) 2024-09-11T01:44:38.075 INFO:tasks.rados.rados.0.smithi071.stdout:2360: rollback oid 8 current snap is 260 2024-09-11T01:44:38.075 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 8 to 257 2024-09-11T01:44:38.075 INFO:tasks.rados.rados.0.smithi071.stdout:2361: snap_create 2024-09-11T01:44:38.084 INFO:tasks.rados.rados.0.smithi071.stdout:2360: finishing rollback tid 0 to smithi07134777-8 2024-09-11T01:44:38.084 INFO:tasks.rados.rados.0.smithi071.stdout:2360: finishing rollback tid 1 to smithi07134777-8 2024-09-11T01:44:38.085 INFO:tasks.rados.rados.0.smithi071.stdout:2360: finishing rollback tid 2 to smithi07134777-8 2024-09-11T01:44:38.085 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 1054 (ObjNum 736 snap 226 seq_num 736) dirty exists 2024-09-11T01:44:39.076 INFO:tasks.rados.rados.0.smithi071.stdout:2360: done (1 left) 2024-09-11T01:44:39.077 INFO:tasks.rados.rados.0.smithi071.stdout:2361: done (0 left) 2024-09-11T01:44:39.077 INFO:tasks.rados.rados.0.smithi071.stdout:2362: snap_remove snap 256 2024-09-11T01:44:40.081 INFO:tasks.rados.rados.0.smithi071.stdout:2362: done (0 left) 2024-09-11T01:44:40.081 INFO:tasks.rados.rados.0.smithi071.stdout:2363: read oid 31 snap -1 2024-09-11T01:44:40.081 INFO:tasks.rados.rados.0.smithi071.stdout:2363: expect (ObjNum 737 snap 226 seq_num 737) 2024-09-11T01:44:40.081 INFO:tasks.rados.rados.0.smithi071.stdout:2364: snap_create 2024-09-11T01:44:40.089 INFO:tasks.rados.rados.0.smithi071.stdout:2363: expect (ObjNum 737 snap 226 seq_num 737) 2024-09-11T01:44:40.131 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:44:40.191 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1674744 2024-09-11T01:44:41.086 INFO:tasks.rados.rados.0.smithi071.stdout:2363: done (1 left) 2024-09-11T01:44:41.087 INFO:tasks.rados.rados.0.smithi071.stdout:2364: done (0 left) 2024-09-11T01:44:41.087 INFO:tasks.rados.rados.0.smithi071.stdout:2365: read oid 21 snap -1 2024-09-11T01:44:41.087 INFO:tasks.rados.rados.0.smithi071.stdout:2365: expect (ObjNum 808 snap 246 seq_num 808) 2024-09-11T01:44:41.087 INFO:tasks.rados.rados.0.smithi071.stdout:2366: read oid 41 snap 248 2024-09-11T01:44:41.087 INFO:tasks.rados.rados.0.smithi071.stdout:2366: expect (ObjNum 753 snap 229 seq_num 753) 2024-09-11T01:44:41.087 INFO:tasks.rados.rados.0.smithi071.stdout:2367: read oid 5 snap 252 2024-09-11T01:44:41.087 INFO:tasks.rados.rados.0.smithi071.stdout:2367: expect (ObjNum 681 snap 204 seq_num 681) 2024-09-11T01:44:41.087 INFO:tasks.rados.rados.0.smithi071.stdout:2368: rollback oid 25 current snap is 262 2024-09-11T01:44:41.087 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 25 to 245 2024-09-11T01:44:41.087 INFO:tasks.rados.rados.0.smithi071.stdout:2369: delete oid 8 current snap is 262 2024-09-11T01:44:41.093 INFO:tasks.rados.rados.0.smithi071.stdout:2369: done (4 left) 2024-09-11T01:44:41.093 INFO:tasks.rados.rados.0.smithi071.stdout:2370: snap_create 2024-09-11T01:44:41.094 INFO:tasks.rados.rados.0.smithi071.stdout:2367: expect (ObjNum 681 snap 204 seq_num 681) 2024-09-11T01:44:41.120 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2995727 2024-09-11T01:44:41.163 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3315097 2024-09-11T01:44:41.170 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1822736 2024-09-11T01:44:41.205 INFO:tasks.rados.rados.0.smithi071.stdout:2368: finishing rollback tid 0 to smithi07134777-25 2024-09-11T01:44:41.205 INFO:tasks.rados.rados.0.smithi071.stdout:2368: finishing rollback tid 1 to smithi07134777-25 2024-09-11T01:44:41.205 INFO:tasks.rados.rados.0.smithi071.stdout:2368: finishing rollback tid 2 to smithi07134777-25 2024-09-11T01:44:41.205 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 995 (ObjNum 723 snap 224 seq_num 723) dirty exists 2024-09-11T01:44:41.205 INFO:tasks.rados.rados.0.smithi071.stdout:2366: expect (ObjNum 753 snap 229 seq_num 753) 2024-09-11T01:44:41.205 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:44:41.325 INFO:tasks.rados.rados.0.smithi071.stdout:2365: expect (ObjNum 808 snap 246 seq_num 808) 2024-09-11T01:44:41.336 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1464023 2024-09-11T01:44:41.371 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:44:41.424 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:44:42.068 INFO:tasks.rados.rados.0.smithi071.stdout:2365: done (4 left) 2024-09-11T01:44:42.068 INFO:tasks.rados.rados.0.smithi071.stdout:2366: done (3 left) 2024-09-11T01:44:42.068 INFO:tasks.rados.rados.0.smithi071.stdout:2367: done (2 left) 2024-09-11T01:44:42.068 INFO:tasks.rados.rados.0.smithi071.stdout:2368: done (1 left) 2024-09-11T01:44:42.068 INFO:tasks.rados.rados.0.smithi071.stdout:2370: done (0 left) 2024-09-11T01:44:42.069 INFO:tasks.rados.rados.0.smithi071.stdout:2371: snap_remove snap 257 2024-09-11T01:44:43.069 INFO:tasks.rados.rados.0.smithi071.stdout:2371: done (0 left) 2024-09-11T01:44:43.069 INFO:tasks.rados.rados.0.smithi071.stdout:2372: write oid 34 current snap is 263 2024-09-11T01:44:43.070 INFO:tasks.rados.rados.0.smithi071.stdout:2372: seq_num 857 ranges {403599=719929,1916480=441635} 2024-09-11T01:44:43.075 INFO:tasks.rados.rados.0.smithi071.stdout:2372: writing smithi07134777-34 from 403599 to 1123528 tid 1 2024-09-11T01:44:43.082 INFO:tasks.rados.rados.0.smithi071.stdout:2372: writing smithi07134777-34 from 1916480 to 2358115 tid 2 2024-09-11T01:44:43.082 INFO:tasks.rados.rados.0.smithi071.stdout:2373: snap_create 2024-09-11T01:44:43.090 INFO:tasks.rados.rados.0.smithi071.stdout:2372: finishing write tid 1 to smithi07134777-34 2024-09-11T01:44:43.092 INFO:tasks.rados.rados.0.smithi071.stdout:2372: finishing write tid 2 to smithi07134777-34 2024-09-11T01:44:43.092 INFO:tasks.rados.rados.0.smithi071.stdout:2372: finishing write tid 3 to smithi07134777-34 2024-09-11T01:44:43.092 INFO:tasks.rados.rados.0.smithi071.stdout:2372: finishing write tid 4 to smithi07134777-34 2024-09-11T01:44:43.092 INFO:tasks.rados.rados.0.smithi071.stdout:2372: oid 34 updating version 0 to 1325 2024-09-11T01:44:43.092 INFO:tasks.rados.rados.0.smithi071.stdout:2372: oid 34 version 1325 is already newer than 1324 2024-09-11T01:44:43.092 INFO:tasks.rados.rados.0.smithi071.stdout:2372: oid 34 version 1325 is already newer than 1323 2024-09-11T01:44:43.093 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 1325 (ObjNum 857 snap 263 seq_num 857) dirty exists 2024-09-11T01:44:43.093 INFO:tasks.rados.rados.0.smithi071.stdout:2372: left oid 34 (ObjNum 857 snap 263 seq_num 857) 2024-09-11T01:44:43.836 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-11T01:44:43.836 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T01:44:43.836 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T01:44:43.839 INFO:tasks.ceph.mon.a:Started 2024-09-11T01:44:43.839 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-11T01:44:43.840 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-11T01:44:43.840 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T01:44:58.840 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-11T01:44:58.841 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-11T01:44:58.841 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:44:58.841 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:44:58.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.952+0000 7fc26acbe640 10 monclient: get_monmap_and_config 2024-09-11T01:44:58.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.952+0000 7fc26acbe640 10 monclient: build_initial_monmap 2024-09-11T01:44:58.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.952+0000 7fc26acbe640 10 monclient: monmap: 2024-09-11T01:44:58.954 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:44:58.954 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:58.954 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:44:58.953497+0000 2024-09-11T01:44:58.954 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:44:58.953497+0000 2024-09-11T01:44:58.955 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:44:58.955 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:58.955 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:44:58.955 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:44:58.955 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:44:58.955 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:44:58.955 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:44:58.955 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:58.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.953+0000 7fc26acbe640 1 Processor -- start 2024-09-11T01:44:58.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.953+0000 7fc26acbe640 1 -- start start 2024-09-11T01:44:58.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.953+0000 7fc26acbe640 10 monclient: init 2024-09-11T01:44:58.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.954+0000 7fc26acbe640 10 monclient: _reopen_session rank -1 2024-09-11T01:44:58.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.954+0000 7fc26acbe640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-11T01:44:58.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.954+0000 7fc26acbe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc264138230 addr v1:172.21.15.71:6789/0 2024-09-11T01:44:58.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.954+0000 7fc26acbe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc26409d030 0x7fc26413d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:58.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.954+0000 7fc26acbe640 10 monclient(hunting): picked mon.noname-b con 0x7fc26409d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:44:58.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.954+0000 7fc26acbe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc26409d460 addr v1:172.21.15.12:6789/0 2024-09-11T01:44:58.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.954+0000 7fc26acbe640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2640a6e30 con 0x7fc26409d460 2024-09-11T01:44:58.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.954+0000 7fc26acbe640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc26409ba20 con 0x7fc264138230 2024-09-11T01:44:58.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.954+0000 7fc26acbe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:58.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.954+0000 7fc26acbe640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc26409c9c0 con 0x7fc26409d030 2024-09-11T01:44:58.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.954+0000 7fc26acbe640 10 monclient(hunting): _renew_subs 2024-09-11T01:44:58.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.954+0000 7fc26acbe640 10 monclient(hunting): authenticate will time out at 1705.396223s 2024-09-11T01:44:58.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.954+0000 7fc26925b640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc26409d460 0x7fc26413f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37714/0 (socket says 172.21.15.12:37714) 2024-09-11T01:44:58.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.954+0000 7fc26925b640 1 -- 172.21.15.12:0/1290337875 learned_addr learned my addr 172.21.15.12:0/1290337875 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:44:58.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.954+0000 7fc263fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc26409d030 0x7fc26413d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:58.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.954+0000 7fc263fff640 10 monclient(hunting): get_auth_request con 0x7fc26409d030 auth_method 0 2024-09-11T01:44:58.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.954+0000 7fc263fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:58.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.954+0000 7fc263fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:58.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.954+0000 7fc263fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:58.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.954+0000 7fc263fff640 1 -- 172.21.15.12:0/1290337875 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc26409d030 msgr2=0x7fc26413d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:44:58.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.954+0000 7fc263fff640 1 -- 172.21.15.12:0/1290337875 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc26409d030 msgr2=0x7fc26413d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:44:58.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.954+0000 7fc263fff640 1 --2- 172.21.15.12:0/1290337875 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc26409d030 0x7fc26413d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:44:58.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.954+0000 7fc263fff640 1 --2- 172.21.15.12:0/1290337875 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc26409d030 0x7fc26413d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:44:58.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.956+0000 7fc26925b640 1 -- 172.21.15.12:0/1290337875 >> v1:172.21.15.12:6789/0 conn(0x7fc26409d460 legacy=0x7fc26413f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:58.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:58.956+0000 7fc268a5a640 1 -- 172.21.15.12:0/1290337875 >> v1:172.21.15.71:6789/0 conn(0x7fc264138230 legacy=0x7fc264138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:59.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.156+0000 7fc263fff640 1 --2- 172.21.15.12:0/1290337875 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc26409d030 0x7fc26413d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.156+0000 7fc263fff640 10 monclient(hunting): get_auth_request con 0x7fc26409d030 auth_method 0 2024-09-11T01:44:59.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.156+0000 7fc263fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:59.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.156+0000 7fc263fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:59.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.156+0000 7fc263fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:44:59.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.156+0000 7fc263fff640 1 -- 172.21.15.12:0/1290337875 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc26409d030 msgr2=0x7fc26413d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:44:59.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.156+0000 7fc263fff640 1 -- 172.21.15.12:0/1290337875 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc26409d030 msgr2=0x7fc26413d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:44:59.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.156+0000 7fc263fff640 1 --2- 172.21.15.12:0/1290337875 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc26409d030 0x7fc26413d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:44:59.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.156+0000 7fc263fff640 1 --2- 172.21.15.12:0/1290337875 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc26409d030 0x7fc26413d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:44:59.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 1 -- 172.21.15.12:0/1290337875 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3335307768 0 0) 0x7fc26409ba20 con 0x7fc264138230 2024-09-11T01:44:59.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 10 monclient(hunting): _negotiate 2024-09-11T01:44:59.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:59.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:59.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 10 monclient(hunting): my global_id is 134437 2024-09-11T01:44:59.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 1 -- 172.21.15.12:0/1290337875 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc24c003520 con 0x7fc264138230 2024-09-11T01:44:59.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 1 -- 172.21.15.12:0/1290337875 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3978445494 0 0) 0x7fc24c003520 con 0x7fc264138230 2024-09-11T01:44:59.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 1 -- 172.21.15.12:0/1290337875 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc26409d030 msgr2=0x7fc26413d090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:44:59.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 1 --2- 172.21.15.12:0/1290337875 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc26409d030 0x7fc26413d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 1 -- 172.21.15.12:0/1290337875 >> v1:172.21.15.12:6789/0 conn(0x7fc26409d460 legacy=0x7fc26413f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:59.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 10 monclient: _finish_hunting 0 2024-09-11T01:44:59.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 1 monclient: found mon.noname-c-legacy 2024-09-11T01:44:59.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:44:59.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-09-11T01:44:59.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 1 -- 172.21.15.12:0/1290337875 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2640a66a0 con 0x7fc264138230 2024-09-11T01:44:59.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 10 monclient: _finish_auth 0 2024-09-11T01:44:59.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 10 monclient: _check_auth_tickets 2024-09-11T01:44:59.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:44:59.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 1 -- 172.21.15.12:0/1290337875 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7fc2540019f0 con 0x7fc264138230 2024-09-11T01:44:59.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:59.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-11T01:44:59.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 10 monclient: dump: 2024-09-11T01:44:59.266 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:59.266 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:59.266 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.266 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.266 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:59.266 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:59.266 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:59.266 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:59.267 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:59.267 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:59.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-09-11T01:44:59.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 10 monclient: _reopen_session rank -1 2024-09-11T01:44:59.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 1 -- 172.21.15.12:0/1290337875 >> v1:172.21.15.71:6789/0 conn(0x7fc264138230 legacy=0x7fc264138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:59.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:44:59.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 1 --2- 172.21.15.12:0/1290337875 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc24c003ad0 0x7fc24c003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:59.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 10 monclient(hunting): picked mon.b con 0x7fc24c003ad0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:44:59.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 1 --2- 172.21.15.12:0/1290337875 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc26409d030 0x7fc24c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:59.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 10 monclient(hunting): picked mon.c con 0x7fc26409d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:44:59.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.260+0000 7fc2637fe640 1 --2- 172.21.15.12:0/1290337875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc24c004f40 0x7fc24c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:59.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc2637fe640 10 monclient(hunting): picked mon.a con 0x7fc24c004f40 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:44:59.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc2637fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:59.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc2637fe640 1 -- 172.21.15.12:0/1290337875 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc26409c9c0 con 0x7fc24c004f40 2024-09-11T01:44:59.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc2637fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:59.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc2637fe640 1 -- 172.21.15.12:0/1290337875 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c00bf00 con 0x7fc24c003ad0 2024-09-11T01:44:59.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc2637fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:59.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc2637fe640 1 -- 172.21.15.12:0/1290337875 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc24c00c080 con 0x7fc26409d030 2024-09-11T01:44:59.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc2637fe640 10 monclient(hunting): _renew_subs 2024-09-11T01:44:59.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc2637fe640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:44:59.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc268a5a640 1 --2- 172.21.15.12:0/1290337875 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc24c003ad0 0x7fc24c003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc268a5a640 10 monclient(hunting): get_auth_request con 0x7fc24c003ad0 auth_method 0 2024-09-11T01:44:59.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc268a5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:59.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc268a5a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:59.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc268a5a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:44:59.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc263fff640 1 --2- 172.21.15.12:0/1290337875 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc26409d030 0x7fc24c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc26925b640 1 --2- 172.21.15.12:0/1290337875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc24c004f40 0x7fc24c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc263fff640 10 monclient(hunting): get_auth_request con 0x7fc26409d030 auth_method 0 2024-09-11T01:44:59.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc263fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:59.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc263fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:59.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc263fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:44:59.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc26925b640 10 monclient(hunting): get_auth_request con 0x7fc24c004f40 auth_method 0 2024-09-11T01:44:59.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc26925b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:59.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc26925b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:59.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc26925b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:44:59.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc268a5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:59.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc268a5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:59.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc268a5a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:44:59.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc26925b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:59.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc26925b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:59.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc26925b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:44:59.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc263fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:59.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc263fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:59.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc263fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:44:59.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc268a5a640 10 monclient(hunting): handle_auth_done global_id 134437 payload 293 2024-09-11T01:44:59.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc268a5a640 1 -- 172.21.15.12:0/1290337875 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc26409d030 msgr2=0x7fc24c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:44:59.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc268a5a640 1 --2- 172.21.15.12:0/1290337875 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc26409d030 0x7fc24c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc268a5a640 1 -- 172.21.15.12:0/1290337875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc24c004f40 msgr2=0x7fc24c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:44:59.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc268a5a640 1 --2- 172.21.15.12:0/1290337875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc24c004f40 0x7fc24c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc268a5a640 10 monclient: _finish_hunting 0 2024-09-11T01:44:59.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc268a5a640 1 monclient: found mon.b 2024-09-11T01:44:59.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc268a5a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:44:59.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc268a5a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:44:59.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc268a5a640 1 -- 172.21.15.12:0/1290337875 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc24c003520 con 0x7fc24c003ad0 2024-09-11T01:44:59.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc268a5a640 10 monclient: _finish_auth 0 2024-09-11T01:44:59.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc268a5a640 10 monclient: _check_auth_tickets 2024-09-11T01:44:59.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc268a5a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:44:59.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc26acbe640 5 monclient: authenticate success, global_id 134437 2024-09-11T01:44:59.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.261+0000 7fc26acbe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:44:59.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.265+0000 7fc268a5a640 1 -- 172.21.15.12:0/1290337875 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc24c003ad0 msgr2=0x7fc24c003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:59.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.265+0000 7fc268a5a640 1 --2- 172.21.15.12:0/1290337875 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc24c003ad0 0x7fc24c003ef0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc254001c10 tx=0x7fc24c005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd446b89f8d22758 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:59.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.265+0000 7fc2637fe640 1 -- 172.21.15.12:0/1290337875 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25400c0e0 con 0x7fc24c003ad0 2024-09-11T01:44:59.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.265+0000 7fc2637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:59.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.265+0000 7fc2637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:44:59.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.265+0000 7fc2637fe640 10 monclient: dump: 2024-09-11T01:44:59.277 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:59.277 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:59.277 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.277 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.277 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:59.277 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:59.277 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:59.277 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:59.277 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:59.277 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:59.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.265+0000 7fc2637fe640 1 -- 172.21.15.12:0/1290337875 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc254007070 con 0x7fc24c003ad0 2024-09-11T01:44:59.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.265+0000 7fc26acbe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:44:59.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.265+0000 7fc2637fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:44:59.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.265+0000 7fc26acbe640 10 monclient: get_monmap_and_config success 2024-09-11T01:44:59.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.265+0000 7fc26acbe640 4 set_mon_vals no callback set 2024-09-11T01:44:59.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.265+0000 7fc2637fe640 1 -- 172.21.15.12:0/1290337875 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc254007460 con 0x7fc24c003ad0 2024-09-11T01:44:59.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.265+0000 7fc2637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:59.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.265+0000 7fc2637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:44:59.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.265+0000 7fc2637fe640 10 monclient: dump: 2024-09-11T01:44:59.278 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:59.278 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:59.278 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.278 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.278 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:59.278 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:59.278 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:59.279 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.266+0000 7fc26acbe640 10 monclient: shutdown 2024-09-11T01:44:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.266+0000 7fc26acbe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:44:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.266+0000 7fc26acbe640 1 -- 172.21.15.12:0/1290337875 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc24c003ad0 msgr2=0x7fc24c003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.266+0000 7fc26acbe640 1 --2- 172.21.15.12:0/1290337875 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc24c003ad0 0x7fc24c003ef0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc254001c10 tx=0x7fc24c005830 comp rx=0 tx=0).stop 2024-09-11T01:44:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.266+0000 7fc26925b640 1 -- 172.21.15.12:0/1290337875 reap_dead start 2024-09-11T01:44:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.266+0000 7fc26acbe640 1 -- 172.21.15.12:0/1290337875 shutdown_connections 2024-09-11T01:44:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.266+0000 7fc26acbe640 1 -- 172.21.15.12:0/1290337875 >> 172.21.15.12:0/1290337875 conn(0x7fc2640a7120 msgr2=0x7fc2640a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:44:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.266+0000 7fc26acbe640 1 -- 172.21.15.12:0/1290337875 shutdown_connections 2024-09-11T01:44:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.266+0000 7fc26acbe640 1 -- 172.21.15.12:0/1290337875 wait complete. 2024-09-11T01:44:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.267+0000 7fc26acbe640 10 monclient: build_initial_monmap 2024-09-11T01:44:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.267+0000 7fc26acbe640 10 monclient: monmap: 2024-09-11T01:44:59.280 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:44:59.280 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:44:59.280 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:44:59.280 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:44:59.280 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:44:59.280 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:59.280 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:44:59.280 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:44:59.280 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:44:59.280 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:59.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.267+0000 7fc26acbe640 1 Processor -- start 2024-09-11T01:44:59.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.267+0000 7fc26acbe640 1 -- start start 2024-09-11T01:44:59.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.267+0000 7fc26acbe640 10 monclient: init 2024-09-11T01:44:59.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.268+0000 7fc26acbe640 10 monclient: _reopen_session rank -1 2024-09-11T01:44:59.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.268+0000 7fc26acbe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:44:59.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc26acbe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc24c003ad0 0x7fc264122450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:59.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc26acbe640 10 monclient(hunting): picked mon.noname-a con 0x7fc24c003ad0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:44:59.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc26acbe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc264122990 0x7fc264122e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:59.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc26acbe640 10 monclient(hunting): picked mon.noname-c con 0x7fc264122990 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:44:59.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc26acbe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2641d1560 0x7fc2641d3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:59.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc263fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc264122990 0x7fc264122e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc268a5a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc24c003ad0 0x7fc264122450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc263fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc264122990 0x7fc264122e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:54028/0 (socket says 172.21.15.12:54028) 2024-09-11T01:44:59.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc268a5a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc24c003ad0 0x7fc264122450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44668/0 (socket says 172.21.15.12:44668) 2024-09-11T01:44:59.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc26acbe640 10 monclient(hunting): picked mon.noname-b con 0x7fc2641d1560 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:44:59.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc263fff640 1 -- 172.21.15.12:0/480555922 learned_addr learned my addr 172.21.15.12:0/480555922 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:44:59.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc26acbe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:59.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc26925b640 1 --2- 172.21.15.12:0/480555922 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2641d1560 0x7fc2641d3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc26acbe640 1 -- 172.21.15.12:0/480555922 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc26413b200 con 0x7fc24c003ad0 2024-09-11T01:44:59.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc26acbe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:59.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc26acbe640 1 -- 172.21.15.12:0/480555922 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc26413b080 con 0x7fc2641d1560 2024-09-11T01:44:59.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc26acbe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:59.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc26acbe640 1 -- 172.21.15.12:0/480555922 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc26413b380 con 0x7fc264122990 2024-09-11T01:44:59.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc26acbe640 10 monclient(hunting): _renew_subs 2024-09-11T01:44:59.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc26acbe640 10 monclient(hunting): authenticate will time out at 1705.711567s 2024-09-11T01:44:59.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc263fff640 10 monclient(hunting): get_auth_request con 0x7fc264122990 auth_method 0 2024-09-11T01:44:59.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc263fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:59.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc263fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:59.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc263fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:59.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc268a5a640 10 monclient(hunting): get_auth_request con 0x7fc24c003ad0 auth_method 0 2024-09-11T01:44:59.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc268a5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:59.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc268a5a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:59.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc268a5a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:59.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc26925b640 10 monclient(hunting): get_auth_request con 0x7fc2641d1560 auth_method 0 2024-09-11T01:44:59.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc26925b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:59.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc26925b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:59.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc26925b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:59.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc263fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:59.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc263fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:59.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc263fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:59.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc26925b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:59.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc26925b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:59.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc26925b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:59.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc268a5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:59.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc268a5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:59.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc268a5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:59.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc263fff640 10 monclient(hunting): handle_auth_done global_id 164159 payload 824 2024-09-11T01:44:59.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.269+0000 7fc263fff640 1 -- 172.21.15.12:0/480555922 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2641d1560 msgr2=0x7fc2641d3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:59.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.270+0000 7fc263fff640 1 --2- 172.21.15.12:0/480555922 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2641d1560 0x7fc2641d3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.270+0000 7fc263fff640 1 -- 172.21.15.12:0/480555922 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc24c003ad0 msgr2=0x7fc264122450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:59.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.270+0000 7fc263fff640 1 --2- 172.21.15.12:0/480555922 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc24c003ad0 0x7fc264122450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.270+0000 7fc263fff640 10 monclient: _finish_hunting 0 2024-09-11T01:44:59.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.270+0000 7fc263fff640 1 monclient: found mon.noname-c 2024-09-11T01:44:59.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.270+0000 7fc263fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:44:59.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.270+0000 7fc263fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:44:59.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.270+0000 7fc263fff640 1 -- 172.21.15.12:0/480555922 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc264126270 con 0x7fc264122990 2024-09-11T01:44:59.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.270+0000 7fc26925b640 1 --2- 172.21.15.12:0/480555922 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2641d1560 0x7fc2641d3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:44:59.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.272+0000 7fc263fff640 1 -- 172.21.15.12:0/480555922 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc264122990 msgr2=0x7fc264122e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:59.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.272+0000 7fc263fff640 1 --2- 172.21.15.12:0/480555922 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc264122990 0x7fc264122e10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc258003fd0 tx=0x7fc25800d9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:59.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.272+0000 7fc2617fa640 1 -- 172.21.15.12:0/480555922 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc258019030 con 0x7fc264122990 2024-09-11T01:44:59.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.272+0000 7fc2617fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:59.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.272+0000 7fc2617fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:44:59.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.272+0000 7fc2617fa640 10 monclient: dump: 2024-09-11T01:44:59.285 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:59.285 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:59.285 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.285 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.285 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:59.285 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:59.286 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:59.286 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:59.286 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:59.286 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:59.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.272+0000 7fc2617fa640 10 monclient: _finish_auth 0 2024-09-11T01:44:59.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.272+0000 7fc2617fa640 10 monclient: _check_auth_tickets 2024-09-11T01:44:59.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.272+0000 7fc2617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:44:59.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.272+0000 7fc2617fa640 1 -- 172.21.15.12:0/480555922 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc258012910 con 0x7fc264122990 2024-09-11T01:44:59.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.272+0000 7fc26acbe640 5 monclient: authenticate success, global_id 164159 2024-09-11T01:44:59.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.272+0000 7fc2617fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:44:59.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.272+0000 7fc26acbe640 10 monclient: _renew_subs 2024-09-11T01:44:59.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.272+0000 7fc2617fa640 1 -- 172.21.15.12:0/480555922 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc258012ac0 con 0x7fc264122990 2024-09-11T01:44:59.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.272+0000 7fc26acbe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:44:59.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.272+0000 7fc26acbe640 1 -- 172.21.15.12:0/480555922 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2640a1810 con 0x7fc264122990 2024-09-11T01:44:59.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.272+0000 7fc2617fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:59.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.272+0000 7fc2617fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:44:59.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.272+0000 7fc2617fa640 10 monclient: dump: 2024-09-11T01:44:59.287 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:59.287 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:59.287 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.287 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.287 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:59.287 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:59.287 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:59.287 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:59.287 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:59.288 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:59.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.272+0000 7fc262ffd640 4 set_mon_vals no callback set 2024-09-11T01:44:59.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.272+0000 7fc26acbe640 10 monclient: _renew_subs 2024-09-11T01:44:59.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.272+0000 7fc26acbe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:44:59.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.272+0000 7fc26acbe640 1 -- 172.21.15.12:0/480555922 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc264123f10 con 0x7fc264122990 2024-09-11T01:44:59.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.273+0000 7fc2617fa640 1 -- 172.21.15.12:0/480555922 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fc258012c60 con 0x7fc264122990 2024-09-11T01:44:59.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.274+0000 7fc2617fa640 1 --2- 172.21.15.12:0/480555922 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc25003e210 0x7fc2500406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:59.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.274+0000 7fc26acbe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:44:59.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.274+0000 7fc26acbe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:44:59.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.274+0000 7fc26acbe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:44:59.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.274+0000 7fc26acbe640 1 -- 172.21.15.12:0/480555922 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2640a9530 con 0x7fc264122990 2024-09-11T01:44:59.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.274+0000 7fc2617fa640 1 -- 172.21.15.12:0/480555922 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(532..532 src has 317..532) ==== 4755+0+0 (secure 0 0 0) 0x7fc258052fc0 con 0x7fc264122990 2024-09-11T01:44:59.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.274+0000 7fc268a5a640 1 --2- 172.21.15.12:0/480555922 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc25003e210 0x7fc2500406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.274+0000 7fc268a5a640 10 monclient: get_auth_request con 0x7fc25003e210 auth_method 0 2024-09-11T01:44:59.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.277+0000 7fc268a5a640 1 --2- 172.21.15.12:0/480555922 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc25003e210 0x7fc2500406d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc254030000 tx=0x7fc254009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:59.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.279+0000 7fc2617fa640 1 -- 172.21.15.12:0/480555922 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc2640a9530 con 0x7fc264122990 2024-09-11T01:44:59.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.279+0000 7fc2617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:44:59.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.279+0000 7fc2617fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:44:59.325 INFO:tasks.rados.rados.0.smithi071.stdout:2372: done (1 left) 2024-09-11T01:44:59.325 INFO:tasks.rados.rados.0.smithi071.stdout:2373: done (0 left) 2024-09-11T01:44:59.325 INFO:tasks.rados.rados.0.smithi071.stdout:2374: write (excl) oid 19 current snap is 264 2024-09-11T01:44:59.325 INFO:tasks.rados.rados.0.smithi071.stdout:2374: seq_num 858 ranges {420406=795178,1974962=445444} 2024-09-11T01:44:59.331 INFO:tasks.rados.rados.0.smithi071.stdout:2374: writing smithi07134777-19 from 420406 to 1215584 tid 1 2024-09-11T01:44:59.338 INFO:tasks.rados.rados.0.smithi071.stdout:2374: writing smithi07134777-19 from 1974962 to 2420406 tid 2 2024-09-11T01:44:59.338 INFO:tasks.rados.rados.0.smithi071.stdout:2375: delete oid 14 current snap is 264 2024-09-11T01:44:59.343 INFO:tasks.rados.rados.0.smithi071.stdout:2374: finishing write tid 1 to smithi07134777-19 2024-09-11T01:44:59.343 INFO:tasks.rados.rados.0.smithi071.stdout:2374: finishing write tid 2 to smithi07134777-19 2024-09-11T01:44:59.343 INFO:tasks.rados.rados.0.smithi071.stdout:2374: finishing write tid 3 to smithi07134777-19 2024-09-11T01:44:59.343 INFO:tasks.rados.rados.0.smithi071.stdout:2374: finishing write tid 4 to smithi07134777-19 2024-09-11T01:44:59.343 INFO:tasks.rados.rados.0.smithi071.stdout:2374: oid 19 updating version 0 to 1056 2024-09-11T01:44:59.344 INFO:tasks.rados.rados.0.smithi071.stdout:2374: oid 19 updating version 1056 to 1058 2024-09-11T01:44:59.344 INFO:tasks.rados.rados.0.smithi071.stdout:2374: oid 19 version 1058 is already newer than 1057 2024-09-11T01:44:59.344 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 1058 (ObjNum 858 snap 264 seq_num 858) dirty exists 2024-09-11T01:44:59.344 INFO:tasks.rados.rados.0.smithi071.stdout:2374: left oid 19 (ObjNum 858 snap 264 seq_num 858) 2024-09-11T01:44:59.344 INFO:tasks.rados.rados.0.smithi071.stdout:2374: done (1 left) 2024-09-11T01:44:59.344 INFO:tasks.rados.rados.0.smithi071.stdout:2375: done (0 left) 2024-09-11T01:44:59.344 INFO:tasks.rados.rados.0.smithi071.stdout:2376: rollback oid 15 current snap is 264 2024-09-11T01:44:59.344 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 15 to 238 2024-09-11T01:44:59.344 INFO:tasks.rados.rados.0.smithi071.stdout:2377: copy_from oid 33 from oid 1 current snap is 264 2024-09-11T01:44:59.344 INFO:tasks.rados.rados.0.smithi071.stdout:2378: rollback oid 44 current snap is 264 2024-09-11T01:44:59.344 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 44 to 235 2024-09-11T01:44:59.344 INFO:tasks.rados.rados.0.smithi071.stdout:2379: delete oid 20 current snap is 264 2024-09-11T01:44:59.350 INFO:tasks.rados.rados.0.smithi071.stdout:2376: finishing rollback tid 0 to smithi07134777-15 2024-09-11T01:44:59.352 INFO:tasks.rados.rados.0.smithi071.stdout:2376: finishing rollback tid 1 to smithi07134777-15 2024-09-11T01:44:59.352 INFO:tasks.rados.rados.0.smithi071.stdout:2378: finishing rollback tid 0 to smithi07134777-44 2024-09-11T01:44:59.353 INFO:tasks.rados.rados.0.smithi071.stdout:2378: finishing rollback tid 1 to smithi07134777-44 2024-09-11T01:44:59.353 INFO:tasks.rados.rados.0.smithi071.stdout:2376: finishing rollback tid 2 to smithi07134777-15 2024-09-11T01:44:59.353 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 1506 (ObjNum 565 snap 160 seq_num 565) dirty exists 2024-09-11T01:44:59.353 INFO:tasks.rados.rados.0.smithi071.stdout:2378: finishing rollback tid 2 to smithi07134777-44 2024-09-11T01:44:59.353 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 997 (ObjNum 742 snap 226 seq_num 742) dirty exists 2024-09-11T01:44:59.354 INFO:tasks.rados.rados.0.smithi071.stdout:2376: done (3 left) 2024-09-11T01:44:59.355 INFO:tasks.rados.rados.0.smithi071.stdout:2378: done (2 left) 2024-09-11T01:44:59.355 INFO:tasks.rados.rados.0.smithi071.stdout:2379: done (1 left) 2024-09-11T01:44:59.355 INFO:tasks.rados.rados.0.smithi071.stdout:2380: delete oid 7 current snap is 264 2024-09-11T01:44:59.356 INFO:tasks.rados.rados.0.smithi071.stdout:2380: done (1 left) 2024-09-11T01:44:59.357 INFO:tasks.rados.rados.0.smithi071.stdout:2381: rollback oid 17 current snap is 264 2024-09-11T01:44:59.357 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 17 to 219 2024-09-11T01:44:59.357 INFO:tasks.rados.rados.0.smithi071.stdout:2382: read oid 11 snap -1 2024-09-11T01:44:59.357 INFO:tasks.rados.rados.0.smithi071.stdout:2382: expect (ObjNum 816 snap 250 seq_num 816) 2024-09-11T01:44:59.357 INFO:tasks.rados.rados.0.smithi071.stdout:2383: snap_remove snap 228 2024-09-11T01:44:59.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.556+0000 7fc26acbe640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:44:59.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.556+0000 7fc26acbe640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:44:59.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.556+0000 7fc26acbe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:44:59.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.556+0000 7fc26acbe640 1 -- 172.21.15.12:0/480555922 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc264138610 con 0x7fc264122990 2024-09-11T01:44:59.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.556+0000 7fc2617fa640 1 -- 172.21.15.12:0/480555922 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fc258015ca0 con 0x7fc264122990 2024-09-11T01:44:59.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.556+0000 7fc2617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:44:59.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.556+0000 7fc2617fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:44:59.558 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":148,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:44:59.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.559+0000 7fc26acbe640 1 -- 172.21.15.12:0/480555922 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc25003e210 msgr2=0x7fc2500406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:59.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.559+0000 7fc26acbe640 1 --2- 172.21.15.12:0/480555922 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc25003e210 0x7fc2500406d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc254030000 tx=0x7fc254009030 comp rx=0 tx=0).stop 2024-09-11T01:44:59.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.559+0000 7fc26acbe640 10 monclient: shutdown 2024-09-11T01:44:59.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.559+0000 7fc26acbe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:44:59.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.559+0000 7fc26acbe640 1 -- 172.21.15.12:0/480555922 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc264122990 msgr2=0x7fc264122e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:59.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.559+0000 7fc26acbe640 1 --2- 172.21.15.12:0/480555922 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc264122990 0x7fc264122e10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc258003fd0 tx=0x7fc25800d9e0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.559+0000 7fc26acbe640 1 -- 172.21.15.12:0/480555922 shutdown_connections 2024-09-11T01:44:59.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.559+0000 7fc26acbe640 1 --2- 172.21.15.12:0/480555922 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2641d1560 0x7fc2641d3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.559+0000 7fc26acbe640 1 --2- 172.21.15.12:0/480555922 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc264122990 0x7fc264122e10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.559+0000 7fc26acbe640 1 --2- 172.21.15.12:0/480555922 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc25003e210 0x7fc2500406d0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.559+0000 7fc26acbe640 1 --2- 172.21.15.12:0/480555922 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc24c003ad0 0x7fc264122450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.559+0000 7fc26acbe640 1 -- 172.21.15.12:0/480555922 >> 172.21.15.12:0/480555922 conn(0x7fc2640a7120 msgr2=0x7fc2640a4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:44:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.559+0000 7fc26acbe640 1 -- 172.21.15.12:0/480555922 shutdown_connections 2024-09-11T01:44:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.559+0000 7fc26acbe640 1 -- 172.21.15.12:0/480555922 wait complete. 2024-09-11T01:44:59.571 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:44:59.571 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:44:59.571 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:44:59.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.671+0000 7f1b2fa60640 10 monclient: get_monmap_and_config 2024-09-11T01:44:59.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.671+0000 7f1b2fa60640 10 monclient: build_initial_monmap 2024-09-11T01:44:59.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.671+0000 7f1b2fa60640 10 monclient: monmap: 2024-09-11T01:44:59.673 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:44:59.673 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:59.673 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:44:59.671786+0000 2024-09-11T01:44:59.673 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:44:59.671786+0000 2024-09-11T01:44:59.673 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:44:59.673 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:44:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:44:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:44:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:44:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:44:59.674 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.671+0000 7f1b2fa60640 1 Processor -- start 2024-09-11T01:44:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.671+0000 7f1b2fa60640 1 -- start start 2024-09-11T01:44:59.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.671+0000 7f1b2fa60640 10 monclient: init 2024-09-11T01:44:59.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2fa60640 10 monclient: _reopen_session rank -1 2024-09-11T01:44:59.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2fa60640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-11T01:44:59.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2fa60640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1b28137e50 addr v1:172.21.15.12:6789/0 2024-09-11T01:44:59.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2fa60640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1b28138230 addr v1:172.21.15.71:6789/0 2024-09-11T01:44:59.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2fa60640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1b2813e390 0x7f1b28140780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:59.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2fa60640 10 monclient(hunting): picked mon.noname-a con 0x7f1b2813e390 addr v2:172.21.15.12:3300/0 2024-09-11T01:44:59.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2fa60640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b280a6580 con 0x7f1b28137e50 2024-09-11T01:44:59.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2fa60640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b280a62d0 con 0x7f1b28138230 2024-09-11T01:44:59.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2fa60640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:59.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2fa60640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f1b2809bb40 con 0x7f1b2813e390 2024-09-11T01:44:59.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2fa60640 10 monclient(hunting): _renew_subs 2024-09-11T01:44:59.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2fa60640 10 monclient(hunting): authenticate will time out at 1706.114366s 2024-09-11T01:44:59.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2dffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1b2813e390 0x7f1b28140780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2d7fc640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1b28137e50 0x7f1b2809d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37730/0 (socket says 172.21.15.12:37730) 2024-09-11T01:44:59.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2d7fc640 1 -- 172.21.15.12:0/4215610354 learned_addr learned my addr 172.21.15.12:0/4215610354 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:44:59.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2dffd640 10 monclient(hunting): get_auth_request con 0x7f1b2813e390 auth_method 0 2024-09-11T01:44:59.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2dffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:59.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2dffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:59.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2dffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:59.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2dffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:59.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2dffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:59.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2dffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:59.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2dffd640 10 monclient(hunting): handle_auth_done global_id 174102 payload 274 2024-09-11T01:44:59.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2dffd640 1 -- 172.21.15.12:0/4215610354 >> v1:172.21.15.71:6789/0 conn(0x7f1b28138230 legacy=0x7f1b28138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:44:59.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2dffd640 1 -- 172.21.15.12:0/4215610354 >> v1:172.21.15.12:6789/0 conn(0x7f1b28137e50 legacy=0x7f1b2809d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:44:59.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2dffd640 10 monclient: _finish_hunting 0 2024-09-11T01:44:59.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2dffd640 1 monclient: found mon.noname-a 2024-09-11T01:44:59.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2dffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:44:59.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2dffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:44:59.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.672+0000 7f1b2dffd640 1 -- 172.21.15.12:0/4215610354 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b2809c4a0 con 0x7f1b2813e390 2024-09-11T01:44:59.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.675+0000 7f1b2dffd640 1 -- 172.21.15.12:0/4215610354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b2813e390 msgr2=0x7f1b28140780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:59.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.675+0000 7f1b2dffd640 1 --2- 172.21.15.12:0/4215610354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b2813e390 0x7f1b28140780 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1b2809bd20 tx=0x7f1b18029fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17fe9ce2b9e1dce2 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:59.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.675+0000 7f1b27fff640 1 -- 172.21.15.12:0/4215610354 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b18037070 con 0x7f1b2813e390 2024-09-11T01:44:59.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.675+0000 7f1b27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:59.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.675+0000 7f1b27fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:44:59.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.675+0000 7f1b27fff640 10 monclient: dump: 2024-09-11T01:44:59.687 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:59.687 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:59.687 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.687 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.687 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:59.687 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:59.687 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:59.687 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:59.687 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:59.687 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:59.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.675+0000 7f1b27fff640 10 monclient: _finish_auth 0 2024-09-11T01:44:59.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.675+0000 7f1b27fff640 10 monclient: _check_auth_tickets 2024-09-11T01:44:59.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.675+0000 7f1b27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:44:59.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.675+0000 7f1b27fff640 1 -- 172.21.15.12:0/4215610354 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b18032440 con 0x7f1b2813e390 2024-09-11T01:44:59.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.675+0000 7f1b2fa60640 5 monclient: authenticate success, global_id 174102 2024-09-11T01:44:59.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.675+0000 7f1b2fa60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:44:59.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.675+0000 7f1b27fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:44:59.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.675+0000 7f1b27fff640 1 -- 172.21.15.12:0/4215610354 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b18032810 con 0x7f1b2813e390 2024-09-11T01:44:59.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.675+0000 7f1b27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:59.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.675+0000 7f1b27fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:44:59.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.675+0000 7f1b27fff640 10 monclient: dump: 2024-09-11T01:44:59.688 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:59.688 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:59.688 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.688 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:59.689 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.675+0000 7f1b2fa60640 10 monclient: get_monmap_and_config success 2024-09-11T01:44:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.675+0000 7f1b2fa60640 4 set_mon_vals no callback set 2024-09-11T01:44:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.676+0000 7f1b2fa60640 10 monclient: shutdown 2024-09-11T01:44:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.676+0000 7f1b2fa60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:44:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.676+0000 7f1b2fa60640 1 -- 172.21.15.12:0/4215610354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b2813e390 msgr2=0x7f1b28140780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.676+0000 7f1b2fa60640 1 --2- 172.21.15.12:0/4215610354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b2813e390 0x7f1b28140780 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1b2809bd20 tx=0x7f1b18029fb0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.676+0000 7f1b2fa60640 1 -- 172.21.15.12:0/4215610354 shutdown_connections 2024-09-11T01:44:59.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.676+0000 7f1b2fa60640 1 --2- 172.21.15.12:0/4215610354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b2813e390 0x7f1b28140780 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.676+0000 7f1b2fa60640 1 -- 172.21.15.12:0/4215610354 >> 172.21.15.12:0/4215610354 conn(0x7f1b280a7020 msgr2=0x7f1b280a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:44:59.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.676+0000 7f1b2fa60640 1 -- 172.21.15.12:0/4215610354 shutdown_connections 2024-09-11T01:44:59.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.676+0000 7f1b2fa60640 1 -- 172.21.15.12:0/4215610354 wait complete. 2024-09-11T01:44:59.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.677+0000 7f1b2fa60640 10 monclient: build_initial_monmap 2024-09-11T01:44:59.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.677+0000 7f1b2fa60640 10 monclient: monmap: 2024-09-11T01:44:59.690 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:44:59.690 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:44:59.690 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:44:59.690 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:44:59.690 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:44:59.690 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:59.690 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:44:59.691 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:44:59.691 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:44:59.691 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:59.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.677+0000 7f1b2fa60640 1 Processor -- start 2024-09-11T01:44:59.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.677+0000 7f1b2fa60640 1 -- start start 2024-09-11T01:44:59.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.677+0000 7f1b2fa60640 10 monclient: init 2024-09-11T01:44:59.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2fa60640 10 monclient: _reopen_session rank -1 2024-09-11T01:44:59.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2fa60640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:44:59.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2fa60640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b28137e50 0x7f1b281cd150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:59.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2fa60640 10 monclient(hunting): picked mon.noname-b con 0x7f1b28137e50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:44:59.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2fa60640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b28138230 0x7f1b28136fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:59.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2fa60640 10 monclient(hunting): picked mon.noname-c con 0x7f1b28138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:44:59.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2fa60640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b2813e390 0x7f1b28139360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2fa60640 10 monclient(hunting): picked mon.noname-a con 0x7f1b2813e390 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:44:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2fa60640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2fa60640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1b28140e40 con 0x7f1b2813e390 2024-09-11T01:44:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2fa60640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2fa60640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1b28140cc0 con 0x7f1b28137e50 2024-09-11T01:44:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2fa60640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2fa60640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1b28140fc0 con 0x7f1b28138230 2024-09-11T01:44:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2fa60640 10 monclient(hunting): _renew_subs 2024-09-11T01:44:59.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2fa60640 10 monclient(hunting): authenticate will time out at 1706.120163s 2024-09-11T01:44:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2dffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b2813e390 0x7f1b28139360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2cffb640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b28138230 0x7f1b28136fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2dffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b2813e390 0x7f1b28139360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44690/0 (socket says 172.21.15.12:44690) 2024-09-11T01:44:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2dffd640 1 -- 172.21.15.12:0/963808386 learned_addr learned my addr 172.21.15.12:0/963808386 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:44:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2dffd640 10 monclient(hunting): get_auth_request con 0x7f1b2813e390 auth_method 0 2024-09-11T01:44:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2dffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2dffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2dffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2cffb640 10 monclient(hunting): get_auth_request con 0x7f1b28138230 auth_method 0 2024-09-11T01:44:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2cffb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:59.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2cffb640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:59.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2cffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:59.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2dffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:59.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2dffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:59.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2dffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:59.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2cffb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:59.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2cffb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:59.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2cffb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:59.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2d7fc640 1 --2- 172.21.15.12:0/963808386 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b28137e50 0x7f1b281cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2d7fc640 10 monclient(hunting): get_auth_request con 0x7f1b28137e50 auth_method 0 2024-09-11T01:44:59.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2d7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:59.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2d7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:59.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2d7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:59.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2dffd640 10 monclient(hunting): handle_auth_done global_id 174105 payload 824 2024-09-11T01:44:59.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2dffd640 1 -- 172.21.15.12:0/963808386 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b28138230 msgr2=0x7f1b28136fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:59.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2dffd640 1 --2- 172.21.15.12:0/963808386 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b28138230 0x7f1b28136fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2dffd640 1 -- 172.21.15.12:0/963808386 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b28137e50 msgr2=0x7f1b281cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:59.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2dffd640 1 --2- 172.21.15.12:0/963808386 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b28137e50 0x7f1b281cd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2dffd640 10 monclient: _finish_hunting 0 2024-09-11T01:44:59.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2dffd640 1 monclient: found mon.noname-a 2024-09-11T01:44:59.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2dffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:44:59.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2dffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:44:59.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2dffd640 1 -- 172.21.15.12:0/963808386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b2813d130 con 0x7f1b2813e390 2024-09-11T01:44:59.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2cffb640 1 --2- 172.21.15.12:0/963808386 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b28138230 0x7f1b28136fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:44:59.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.678+0000 7f1b2d7fc640 1 --2- 172.21.15.12:0/963808386 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b28137e50 0x7f1b281cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:44:59.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.680+0000 7f1b2dffd640 1 -- 172.21.15.12:0/963808386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b2813e390 msgr2=0x7f1b28139360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:59.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.680+0000 7f1b2dffd640 1 --2- 172.21.15.12:0/963808386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b2813e390 0x7f1b28139360 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1b18009890 tx=0x7f1b180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:59.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.681+0000 7f1b25ffb640 1 -- 172.21.15.12:0/963808386 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b18041070 con 0x7f1b2813e390 2024-09-11T01:44:59.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.681+0000 7f1b25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:59.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.681+0000 7f1b25ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:44:59.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.681+0000 7f1b25ffb640 10 monclient: dump: 2024-09-11T01:44:59.696 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:59.696 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:59.696 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.696 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.696 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:59.696 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:59.696 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:59.697 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:59.697 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:59.697 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:59.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.681+0000 7f1b25ffb640 10 monclient: _finish_auth 0 2024-09-11T01:44:59.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.681+0000 7f1b25ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:44:59.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.681+0000 7f1b25ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:44:59.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.681+0000 7f1b25ffb640 1 -- 172.21.15.12:0/963808386 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b1803c660 con 0x7f1b2813e390 2024-09-11T01:44:59.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.681+0000 7f1b25ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:44:59.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.681+0000 7f1b25ffb640 1 -- 172.21.15.12:0/963808386 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b18037040 con 0x7f1b2813e390 2024-09-11T01:44:59.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.681+0000 7f1b25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:59.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.681+0000 7f1b25ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:44:59.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.681+0000 7f1b25ffb640 10 monclient: dump: 2024-09-11T01:44:59.697 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:59.698 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:59.698 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.698 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.698 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:59.698 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:59.698 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:59.698 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:59.698 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:59.698 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:59.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.681+0000 7f1b2fa60640 5 monclient: authenticate success, global_id 174105 2024-09-11T01:44:59.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.681+0000 7f1b2fa60640 10 monclient: _renew_subs 2024-09-11T01:44:59.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.681+0000 7f1b2fa60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:44:59.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.681+0000 7f1b2fa60640 1 -- 172.21.15.12:0/963808386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b28140910 con 0x7f1b2813e390 2024-09-11T01:44:59.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.681+0000 7f1b277fe640 4 set_mon_vals no callback set 2024-09-11T01:44:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.681+0000 7f1b2fa60640 10 monclient: _renew_subs 2024-09-11T01:44:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.681+0000 7f1b2fa60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:44:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.681+0000 7f1b2fa60640 1 -- 172.21.15.12:0/963808386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b2813ce50 con 0x7f1b2813e390 2024-09-11T01:44:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.682+0000 7f1b2fa60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:44:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.682+0000 7f1b25ffb640 1 -- 172.21.15.12:0/963808386 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f1b1803ca90 con 0x7f1b2813e390 2024-09-11T01:44:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.682+0000 7f1b2fa60640 1 --2- 172.21.15.12:0/963808386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b281398a0 0x7f1b28139c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.682+0000 7f1b2fa60640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.682+0000 7f1b2fa60640 1 -- 172.21.15.12:0/963808386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1b2813a1c0 con 0x7f1b281398a0 2024-09-11T01:44:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.682+0000 7f1b2d7fc640 1 --2- 172.21.15.12:0/963808386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b281398a0 0x7f1b28139c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.682+0000 7f1b2d7fc640 10 monclient: get_auth_request con 0x7f1b281398a0 auth_method 0 2024-09-11T01:44:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.682+0000 7f1b2d7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.682+0000 7f1b2d7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:59.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.682+0000 7f1b2d7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.682+0000 7f1b2d7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.682+0000 7f1b2d7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.682+0000 7f1b25ffb640 1 --2- 172.21.15.12:0/963808386 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1b1403e260 0x7f1b14040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.682+0000 7f1b2d7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.682+0000 7f1b25ffb640 1 -- 172.21.15.12:0/963808386 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(533..533 src has 317..533) ==== 4755+0+0 (secure 0 0 0) 0x7f1b18073720 con 0x7f1b2813e390 2024-09-11T01:44:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.682+0000 7f1b2d7fc640 10 monclient(hunting): handle_auth_done global_id 174108 payload 824 2024-09-11T01:44:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.682+0000 7f1b2d7fc640 1 -- 172.21.15.12:0/963808386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1b2809d160 con 0x7f1b281398a0 2024-09-11T01:44:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.682+0000 7f1b2cffb640 1 --2- 172.21.15.12:0/963808386 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1b1403e260 0x7f1b14040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.682+0000 7f1b2cffb640 10 monclient: get_auth_request con 0x7f1b1403e260 auth_method 0 2024-09-11T01:44:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.684+0000 7f1b2d7fc640 1 -- 172.21.15.12:0/963808386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b281398a0 msgr2=0x7f1b28139c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.684+0000 7f1b2d7fc640 1 --2- 172.21.15.12:0/963808386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b281398a0 0x7f1b28139c80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1b281401a0 tx=0x7f1b2000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.684+0000 7f1b25ffb640 1 -- 172.21.15.12:0/963808386 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b20005ae0 con 0x7f1b281398a0 2024-09-11T01:44:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.684+0000 7f1b25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:59.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.684+0000 7f1b25ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:44:59.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.684+0000 7f1b25ffb640 10 monclient: dump: 2024-09-11T01:44:59.701 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:59.701 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:59.701 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.701 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.701 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:59.701 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:59.701 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:59.701 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:59.701 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:59.701 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:59.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.685+0000 7f1b2cffb640 1 --2- 172.21.15.12:0/963808386 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1b1403e260 0x7f1b14040720 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1b281406c0 tx=0x7f1b1c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:59.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.685+0000 7f1b25ffb640 1 -- 172.21.15.12:0/963808386 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1b20018070 con 0x7f1b281398a0 2024-09-11T01:44:59.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.685+0000 7f1b25ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:44:59.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.685+0000 7f1b25ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:44:59.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.685+0000 7f1b25ffb640 1 -- 172.21.15.12:0/963808386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b281398a0 msgr2=0x7f1b28139c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:59.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.685+0000 7f1b25ffb640 1 --2- 172.21.15.12:0/963808386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b281398a0 0x7f1b28139c80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1b281401a0 tx=0x7f1b2000eb90 comp rx=0 tx=0).stop 2024-09-11T01:44:59.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.685+0000 7f1b25ffb640 1 -- 172.21.15.12:0/963808386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b281398a0 msgr2=0x7f1b28139c80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:44:59.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.685+0000 7f1b25ffb640 1 --2- 172.21.15.12:0/963808386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b281398a0 0x7f1b28139c80 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1b281401a0 tx=0x7f1b2000eb90 comp rx=0 tx=0).stop 2024-09-11T01:44:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.691+0000 7f1b2fa60640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:44:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.691+0000 7f1b2fa60640 1 --2- 172.21.15.12:0/963808386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b280a0430 0x7f1b280a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.691+0000 7f1b2fa60640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.691+0000 7f1b2fa60640 1 -- 172.21.15.12:0/963808386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1b2813c500 con 0x7f1b280a0430 2024-09-11T01:44:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.691+0000 7f1b2d7fc640 1 --2- 172.21.15.12:0/963808386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b280a0430 0x7f1b280a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.691+0000 7f1b2d7fc640 10 monclient: get_auth_request con 0x7f1b280a0430 auth_method 0 2024-09-11T01:44:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.691+0000 7f1b2d7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.691+0000 7f1b2d7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.691+0000 7f1b2d7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:59.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.691+0000 7f1b2d7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.691+0000 7f1b2d7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.691+0000 7f1b2d7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.691+0000 7f1b2d7fc640 10 monclient(hunting): handle_auth_done global_id 174111 payload 824 2024-09-11T01:44:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.691+0000 7f1b2d7fc640 1 -- 172.21.15.12:0/963808386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1b2813add0 con 0x7f1b280a0430 2024-09-11T01:44:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.693+0000 7f1b2d7fc640 1 -- 172.21.15.12:0/963808386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b280a0430 msgr2=0x7f1b280a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.693+0000 7f1b2d7fc640 1 --2- 172.21.15.12:0/963808386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b280a0430 0x7f1b280a0810 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1b20017000 tx=0x7f1b20004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.694+0000 7f1b25ffb640 1 -- 172.21.15.12:0/963808386 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b200070d0 con 0x7f1b280a0430 2024-09-11T01:44:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.694+0000 7f1b25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.694+0000 7f1b25ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:44:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.694+0000 7f1b25ffb640 10 monclient: dump: 2024-09-11T01:44:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:59.706 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:59.707 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.694+0000 7f1b25ffb640 1 -- 172.21.15.12:0/963808386 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7f1b200070d0 con 0x7f1b280a0430 2024-09-11T01:44:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.694+0000 7f1b25ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:44:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.694+0000 7f1b25ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:44:59.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.694+0000 7f1b25ffb640 1 -- 172.21.15.12:0/963808386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b280a0430 msgr2=0x7f1b280a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:59.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.694+0000 7f1b25ffb640 1 --2- 172.21.15.12:0/963808386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b280a0430 0x7f1b280a0810 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1b20017000 tx=0x7f1b20004940 comp rx=0 tx=0).stop 2024-09-11T01:44:59.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.694+0000 7f1b25ffb640 1 -- 172.21.15.12:0/963808386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b280a0430 msgr2=0x7f1b280a0810 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:44:59.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.694+0000 7f1b25ffb640 1 --2- 172.21.15.12:0/963808386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b280a0430 0x7f1b280a0810 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1b20017000 tx=0x7f1b20004940 comp rx=0 tx=0).stop 2024-09-11T01:44:59.708 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:44:59.708 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:44:59.708 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:44:59.708 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:44:59.708 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 148, 2024-09-11T01:44:59.708 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 15764, 2024-09-11T01:44:59.709 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:44:59.709 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:44:59.709 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:44:59.709 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:44:59.709 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:59.709 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:44:59.709 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:44:59.709 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:44:59.709 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:44:59.709 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:44:59.709 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:44:59.709 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:44:59.709 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:44:59.709 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:44:59.710 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:44:59.710 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:44:59.710 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:44:59.710 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:44:59.710 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:44:59.710 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:44:59.710 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:59.710 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:44:59.710 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:44:59.710 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:44:59.710 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:44:59.710 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:44:59.710 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:44:59.710 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:44:59.711 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:44:59.711 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:44:59.711 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:44:59.711 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:44:59.711 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:44:59.711 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:44:59.711 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:59.711 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:59.711 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:44:59.711 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:44:59.711 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:44:59.711 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:44:59.711 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:44:59.712 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:44:59.712 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:44:59.712 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:44:59.712 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:44:59.712 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:44:59.712 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:44:59.712 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:44:59.712 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:44:59.712 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:44:59.712 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:44:59.712 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:44:59.712 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:44:59.712 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:44:59.713 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:44:59.713 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:44:59.713 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:44:59.713 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:44:59.713 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:44:59.713 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:44:59.713 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:44:59.713 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:44:59.713 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:44:59.713 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:44:59.713 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:59.713 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:44:59.713 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:59.713 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:44:59.714 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:59.714 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:44:59.714 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:44:59.714 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:44:59.714 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:44:59.714 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:59.714 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:44:59.714 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:44:59.714 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:59.714 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:59.714 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:59.714 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:44:59.714 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:44:59.714 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:59.715 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:59.715 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:59.715 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:59.715 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:44:59.715 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:44:59.715 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:44:59.715 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:44:59.715 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:44:59.715 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:59.715 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:59.715 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:44:59.715 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:44:59.715 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:44:59.715 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:44:59.716 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:59.716 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:44:59.716 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:44:59.716 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:59.716 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:59.716 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:59.716 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:44:59.716 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:44:59.716 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:59.716 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:59.716 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:59.716 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:59.716 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:44:59.717 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:44:59.717 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:44:59.717 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:44:59.717 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:44:59.717 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:59.717 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:59.717 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:44:59.717 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:44:59.717 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:44:59.717 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:44:59.717 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:59.717 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:44:59.717 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:44:59.717 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:59.718 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:59.718 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:59.718 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:44:59.718 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:44:59.718 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:59.718 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:59.718 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:59.718 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:59.718 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:44:59.719 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:44:59.719 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:44:59.719 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:44:59.719 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:44:59.719 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:59.719 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:59.719 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:59.719 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:44:59.719 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:44:59.719 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:59.719 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:44:59.719 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:44:59.719 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:44:59.719 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:59.719 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:59.720 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:44:59.720 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:59.720 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:44:59.720 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:44:59.720 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:44:59.720 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:59.720 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:59.720 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:59.720 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:44:59.720 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:44:59.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.694+0000 7f1b2fa60640 1 -- 172.21.15.12:0/963808386 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1b1403e260 msgr2=0x7f1b14040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:59.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.694+0000 7f1b2fa60640 1 --2- 172.21.15.12:0/963808386 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1b1403e260 0x7f1b14040720 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1b281406c0 tx=0x7f1b1c009040 comp rx=0 tx=0).stop 2024-09-11T01:44:59.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.694+0000 7f1b2fa60640 10 monclient: shutdown 2024-09-11T01:44:59.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.694+0000 7f1b2fa60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:44:59.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.694+0000 7f1b2fa60640 1 -- 172.21.15.12:0/963808386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b2813e390 msgr2=0x7f1b28139360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:59.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.694+0000 7f1b2fa60640 1 --2- 172.21.15.12:0/963808386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b2813e390 0x7f1b28139360 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1b18009890 tx=0x7f1b180046c0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.694+0000 7f1b2dffd640 1 -- 172.21.15.12:0/963808386 reap_dead start 2024-09-11T01:44:59.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.694+0000 7f1b2dffd640 1 -- 172.21.15.12:0/963808386 reap_dead start 2024-09-11T01:44:59.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.694+0000 7f1b2fa60640 1 -- 172.21.15.12:0/963808386 shutdown_connections 2024-09-11T01:44:59.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.694+0000 7f1b2fa60640 1 -- 172.21.15.12:0/963808386 >> 172.21.15.12:0/963808386 conn(0x7f1b280a7020 msgr2=0x7f1b2813eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:44:59.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.694+0000 7f1b2fa60640 1 -- 172.21.15.12:0/963808386 shutdown_connections 2024-09-11T01:44:59.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.694+0000 7f1b2fa60640 1 -- 172.21.15.12:0/963808386 wait complete. 2024-09-11T01:44:59.723 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:44:59.723 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:44:59.723 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:44:59.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.823+0000 7feeffa9b640 10 monclient: get_monmap_and_config 2024-09-11T01:44:59.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.823+0000 7feeffa9b640 10 monclient: build_initial_monmap 2024-09-11T01:44:59.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.823+0000 7feeffa9b640 10 monclient: monmap: 2024-09-11T01:44:59.824 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:44:59.825 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:59.825 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:44:59.823923+0000 2024-09-11T01:44:59.825 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:44:59.823923+0000 2024-09-11T01:44:59.825 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:44:59.825 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:59.825 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:44:59.825 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:44:59.825 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:44:59.825 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:44:59.825 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:44:59.825 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:59.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.823+0000 7feeffa9b640 1 Processor -- start 2024-09-11T01:44:59.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feeffa9b640 1 -- start start 2024-09-11T01:44:59.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feeffa9b640 10 monclient: init 2024-09-11T01:44:59.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feeffa9b640 10 monclient: _reopen_session rank -1 2024-09-11T01:44:59.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feeffa9b640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-11T01:44:59.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feeffa9b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feef8137e50 addr v1:172.21.15.71:6789/0 2024-09-11T01:44:59.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feeffa9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef8138230 0x7feef8138660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:59.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feeffa9b640 10 monclient(hunting): picked mon.noname-b con 0x7feef8138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:44:59.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feeffa9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feef813d380 0x7feef813f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:59.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feeffa9b640 10 monclient(hunting): picked mon.noname-a con 0x7feef813d380 addr v2:172.21.15.12:3300/0 2024-09-11T01:44:59.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feeffa9b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feef80a6580 con 0x7feef8137e50 2024-09-11T01:44:59.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feeffa9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:59.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feeffa9b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7feef80a62d0 con 0x7feef813d380 2024-09-11T01:44:59.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feeffa9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:59.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feeffa9b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7feef809bb40 con 0x7feef8138230 2024-09-11T01:44:59.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feeffa9b640 10 monclient(hunting): _renew_subs 2024-09-11T01:44:59.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feeffa9b640 10 monclient(hunting): authenticate will time out at 1706.266564s 2024-09-11T01:44:59.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feefe038640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feef813d380 0x7feef813f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feefd036640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef8138230 0x7feef8138660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feefe038640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feef813d380 0x7feef813f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44724/0 (socket says 172.21.15.12:44724) 2024-09-11T01:44:59.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feefd036640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef8138230 0x7feef8138660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:54076/0 (socket says 172.21.15.12:54076) 2024-09-11T01:44:59.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feefe038640 1 -- 172.21.15.12:0/1295583987 learned_addr learned my addr 172.21.15.12:0/1295583987 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:44:59.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feefd036640 10 monclient(hunting): get_auth_request con 0x7feef8138230 auth_method 0 2024-09-11T01:44:59.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feefd036640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:59.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feefd036640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:59.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feefd036640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:59.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feefe038640 10 monclient(hunting): get_auth_request con 0x7feef813d380 auth_method 0 2024-09-11T01:44:59.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feefe038640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:59.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feefe038640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:59.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feefe038640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:59.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feefd036640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:59.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feefd036640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:59.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feefd036640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:59.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feefe038640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:59.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feefe038640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:59.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feefe038640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:59.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feefd036640 10 monclient(hunting): handle_auth_done global_id 164165 payload 274 2024-09-11T01:44:59.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feefd036640 1 -- 172.21.15.12:0/1295583987 >> v2:172.21.15.12:3300/0 conn(0x7feef813d380 msgr2=0x7feef813f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:44:59.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feefd036640 1 --2- 172.21.15.12:0/1295583987 >> v2:172.21.15.12:3300/0 conn(0x7feef813d380 0x7feef813f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.824+0000 7feefd036640 1 -- 172.21.15.12:0/1295583987 >> v1:172.21.15.71:6789/0 conn(0x7feef8137e50 legacy=0x7feef809d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:44:59.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.825+0000 7feefd036640 10 monclient: _finish_hunting 0 2024-09-11T01:44:59.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.825+0000 7feefd036640 1 monclient: found mon.noname-b 2024-09-11T01:44:59.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.825+0000 7feefd036640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:44:59.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.825+0000 7feefd036640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:44:59.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.825+0000 7feefd036640 1 -- 172.21.15.12:0/1295583987 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feef809c4a0 con 0x7feef8138230 2024-09-11T01:44:59.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.825+0000 7feefe038640 1 --2- 172.21.15.12:0/1295583987 >> v2:172.21.15.12:3300/0 conn(0x7feef813d380 0x7feef813f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:44:59.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.827+0000 7feefd036640 1 -- 172.21.15.12:0/1295583987 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef8138230 msgr2=0x7feef8138660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:59.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.827+0000 7feefd036640 1 --2- 172.21.15.12:0/1295583987 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef8138230 0x7feef8138660 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7feee80097a0 tx=0x7feee8029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69fd4cf8419cdca6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:59.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feefc835640 1 -- 172.21.15.12:0/1295583987 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee8037070 con 0x7feef8138230 2024-09-11T01:44:59.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feefc835640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:59.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feefc835640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:44:59.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feefc835640 10 monclient: dump: 2024-09-11T01:44:59.837 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:59.837 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:59.837 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.837 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.837 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:59.837 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:59.837 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:59.837 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:59.837 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:59.837 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:59.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feefc835640 10 monclient: _finish_auth 0 2024-09-11T01:44:59.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feefc835640 10 monclient: _check_auth_tickets 2024-09-11T01:44:59.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feefc835640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:44:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feeffa9b640 5 monclient: authenticate success, global_id 164165 2024-09-11T01:44:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feeffa9b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:44:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feefc835640 1 -- 172.21.15.12:0/1295583987 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feee8032440 con 0x7feef8138230 2024-09-11T01:44:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feefc835640 10 monclient: handle_config config(0 keys) 2024-09-11T01:44:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feeffa9b640 10 monclient: get_monmap_and_config success 2024-09-11T01:44:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feeffa9b640 4 set_mon_vals no callback set 2024-09-11T01:44:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feefc835640 1 -- 172.21.15.12:0/1295583987 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee8032810 con 0x7feef8138230 2024-09-11T01:44:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feefc835640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feefc835640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:44:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feefc835640 10 monclient: dump: 2024-09-11T01:44:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.838 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:59.839 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:59.839 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:59.839 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:59.839 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:59.839 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:59.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feeffa9b640 10 monclient: shutdown 2024-09-11T01:44:59.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feeffa9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:44:59.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feeffa9b640 1 -- 172.21.15.12:0/1295583987 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef8138230 msgr2=0x7feef8138660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:59.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feeffa9b640 1 --2- 172.21.15.12:0/1295583987 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef8138230 0x7feef8138660 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7feee80097a0 tx=0x7feee8029fd0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feeffa9b640 1 -- 172.21.15.12:0/1295583987 shutdown_connections 2024-09-11T01:44:59.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feeffa9b640 1 --2- 172.21.15.12:0/1295583987 >> v2:172.21.15.12:3300/0 conn(0x7feef813d380 0x7feef813f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feeffa9b640 1 --2- 172.21.15.12:0/1295583987 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef8138230 0x7feef8138660 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feeffa9b640 1 -- 172.21.15.12:0/1295583987 >> 172.21.15.12:0/1295583987 conn(0x7feef80a7020 msgr2=0x7feef80a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:44:59.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feeffa9b640 1 -- 172.21.15.12:0/1295583987 shutdown_connections 2024-09-11T01:44:59.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.828+0000 7feeffa9b640 1 -- 172.21.15.12:0/1295583987 wait complete. 2024-09-11T01:44:59.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.829+0000 7feeffa9b640 10 monclient: build_initial_monmap 2024-09-11T01:44:59.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.829+0000 7feeffa9b640 10 monclient: monmap: 2024-09-11T01:44:59.840 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:44:59.840 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:44:59.840 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:44:59.840 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:44:59.840 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:44:59.841 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:59.841 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:44:59.841 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:44:59.841 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:44:59.841 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:59.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.829+0000 7feeffa9b640 1 Processor -- start 2024-09-11T01:44:59.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.829+0000 7feeffa9b640 1 -- start start 2024-09-11T01:44:59.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.829+0000 7feeffa9b640 10 monclient: init 2024-09-11T01:44:59.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feeffa9b640 10 monclient: _reopen_session rank -1 2024-09-11T01:44:59.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feeffa9b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:44:59.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feeffa9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feef8137e50 0x7feef81ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feeffa9b640 10 monclient(hunting): picked mon.noname-b con 0x7feef8137e50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:44:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feeffa9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef8138230 0x7feef8136f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feeffa9b640 10 monclient(hunting): picked mon.noname-c con 0x7feef8138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:44:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feeffa9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feef813d380 0x7feef8139650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feeffa9b640 10 monclient(hunting): picked mon.noname-a con 0x7feef813d380 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:44:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feeffa9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feeffa9b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7feef806f890 con 0x7feef813d380 2024-09-11T01:44:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feeffa9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feeffa9b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7feef806f710 con 0x7feef8137e50 2024-09-11T01:44:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feeffa9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feeffa9b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7feef813fcb0 con 0x7feef8138230 2024-09-11T01:44:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feeffa9b640 10 monclient(hunting): _renew_subs 2024-09-11T01:44:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feeffa9b640 10 monclient(hunting): authenticate will time out at 1706.272345s 2024-09-11T01:44:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefd036640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef8138230 0x7feef8136f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefe038640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feef813d380 0x7feef8139650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefd036640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef8138230 0x7feef8136f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:54080/0 (socket says 172.21.15.12:54080) 2024-09-11T01:44:59.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefe038640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feef813d380 0x7feef8139650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44728/0 (socket says 172.21.15.12:44728) 2024-09-11T01:44:59.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefd036640 1 -- 172.21.15.12:0/750301215 learned_addr learned my addr 172.21.15.12:0/750301215 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:44:59.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefd036640 10 monclient(hunting): get_auth_request con 0x7feef8138230 auth_method 0 2024-09-11T01:44:59.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefd036640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:59.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefd036640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:59.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefd036640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:59.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefe038640 10 monclient(hunting): get_auth_request con 0x7feef813d380 auth_method 0 2024-09-11T01:44:59.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefe038640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:59.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefe038640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:59.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefe038640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:59.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefe038640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:59.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefe038640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefe038640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefd036640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefd036640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefd036640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefd837640 1 --2- 172.21.15.12:0/750301215 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feef8137e50 0x7feef81ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefd837640 10 monclient(hunting): get_auth_request con 0x7feef8137e50 auth_method 0 2024-09-11T01:44:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefd837640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefd837640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefd837640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefd837640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefd837640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefd837640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefe038640 10 monclient(hunting): handle_auth_done global_id 174117 payload 824 2024-09-11T01:44:59.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefe038640 1 -- 172.21.15.12:0/750301215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef8138230 msgr2=0x7feef8136f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefe038640 1 --2- 172.21.15.12:0/750301215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef8138230 0x7feef8136f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefe038640 1 -- 172.21.15.12:0/750301215 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feef8137e50 msgr2=0x7feef81ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefe038640 1 --2- 172.21.15.12:0/750301215 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feef8137e50 0x7feef81ccff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefe038640 10 monclient: _finish_hunting 0 2024-09-11T01:44:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefe038640 1 monclient: found mon.noname-a 2024-09-11T01:44:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefe038640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:44:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefe038640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:44:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefe038640 1 -- 172.21.15.12:0/750301215 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feef813afd0 con 0x7feef813d380 2024-09-11T01:44:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.830+0000 7feefd036640 1 --2- 172.21.15.12:0/750301215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef8138230 0x7feef8136f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:44:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.832+0000 7feefe038640 1 -- 172.21.15.12:0/750301215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feef813d380 msgr2=0x7feef8139650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.833+0000 7feefe038640 1 --2- 172.21.15.12:0/750301215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feef813d380 0x7feef8139650 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7feef0009e50 tx=0x7feef000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.833+0000 7feeee7fc640 1 -- 172.21.15.12:0/750301215 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef000cd90 con 0x7feef813d380 2024-09-11T01:44:59.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.833+0000 7feeee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:59.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.833+0000 7feeee7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:44:59.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.833+0000 7feeee7fc640 10 monclient: dump: 2024-09-11T01:44:59.846 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:59.846 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:59.846 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.846 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.846 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:59.846 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:59.846 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:59.846 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:59.846 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:59.846 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:59.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.833+0000 7feeee7fc640 10 monclient: _finish_auth 0 2024-09-11T01:44:59.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.833+0000 7feeee7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:44:59.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.833+0000 7feeee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:44:59.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.833+0000 7feeee7fc640 1 -- 172.21.15.12:0/750301215 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feef0003990 con 0x7feef813d380 2024-09-11T01:44:59.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.833+0000 7feeffa9b640 5 monclient: authenticate success, global_id 174117 2024-09-11T01:44:59.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.833+0000 7feeee7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:44:59.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.833+0000 7feeee7fc640 1 -- 172.21.15.12:0/750301215 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef0003d80 con 0x7feef813d380 2024-09-11T01:44:59.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.833+0000 7feeee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:59.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.833+0000 7feeee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:44:59.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.833+0000 7feeee7fc640 10 monclient: dump: 2024-09-11T01:44:59.847 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:59.847 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:59.848 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.848 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.848 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:59.848 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:59.848 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:59.848 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:59.848 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:59.848 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:59.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.833+0000 7feeffa9b640 10 monclient: _renew_subs 2024-09-11T01:44:59.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.833+0000 7feeffa9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:44:59.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.833+0000 7feeef7fe640 4 set_mon_vals no callback set 2024-09-11T01:44:59.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.833+0000 7feeffa9b640 1 -- 172.21.15.12:0/750301215 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feef80a1710 con 0x7feef813d380 2024-09-11T01:44:59.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.833+0000 7feeffa9b640 10 monclient: _renew_subs 2024-09-11T01:44:59.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.833+0000 7feeffa9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:44:59.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.833+0000 7feeffa9b640 1 -- 172.21.15.12:0/750301215 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feef8121060 con 0x7feef813d380 2024-09-11T01:44:59.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.833+0000 7feeffa9b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:44:59.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.833+0000 7feeffa9b640 1 --2- 172.21.15.12:0/750301215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef81374b0 0x7feef8139b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:59.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.834+0000 7feeffa9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:59.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.834+0000 7feeffa9b640 1 -- 172.21.15.12:0/750301215 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7feef806a060 con 0x7feef81374b0 2024-09-11T01:44:59.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.834+0000 7feefd837640 1 --2- 172.21.15.12:0/750301215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef81374b0 0x7feef8139b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.834+0000 7feefd837640 10 monclient: get_auth_request con 0x7feef81374b0 auth_method 0 2024-09-11T01:44:59.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.834+0000 7feefd837640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:59.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.834+0000 7feefd837640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:59.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.834+0000 7feefd837640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:59.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.834+0000 7feeee7fc640 1 -- 172.21.15.12:0/750301215 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7feef0003f20 con 0x7feef813d380 2024-09-11T01:44:59.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.834+0000 7feefd837640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:59.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.834+0000 7feefd837640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:59.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.834+0000 7feefd837640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:59.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.834+0000 7feefd837640 10 monclient(hunting): handle_auth_done global_id 164171 payload 824 2024-09-11T01:44:59.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.834+0000 7feefd837640 1 -- 172.21.15.12:0/750301215 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feef8138660 con 0x7feef81374b0 2024-09-11T01:44:59.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.834+0000 7feeee7fc640 1 --2- 172.21.15.12:0/750301215 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7feedc03e260 0x7feedc040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:59.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.834+0000 7feeee7fc640 1 -- 172.21.15.12:0/750301215 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(533..533 src has 317..533) ==== 4755+0+0 (secure 0 0 0) 0x7feef0053440 con 0x7feef813d380 2024-09-11T01:44:59.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.834+0000 7feefd036640 1 --2- 172.21.15.12:0/750301215 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7feedc03e260 0x7feedc040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.834+0000 7feefd036640 10 monclient: get_auth_request con 0x7feedc03e260 auth_method 0 2024-09-11T01:44:59.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.836+0000 7feefd837640 1 -- 172.21.15.12:0/750301215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef81374b0 msgr2=0x7feef8139b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:59.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.836+0000 7feefd837640 1 --2- 172.21.15.12:0/750301215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef81374b0 0x7feef8139b90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7feef4006fd0 tx=0x7feef400ec00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:59.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.836+0000 7feeee7fc640 1 -- 172.21.15.12:0/750301215 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef4017070 con 0x7feef81374b0 2024-09-11T01:44:59.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.836+0000 7feeee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:59.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.836+0000 7feeee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:44:59.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.836+0000 7feeee7fc640 10 monclient: dump: 2024-09-11T01:44:59.853 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:59.853 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:59.853 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.853 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.853 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:59.854 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.837+0000 7feeee7fc640 1 -- 172.21.15.12:0/750301215 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7feef400c9b0 con 0x7feef81374b0 2024-09-11T01:44:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.837+0000 7feeee7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:44:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.837+0000 7feeee7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:44:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.837+0000 7feeee7fc640 1 -- 172.21.15.12:0/750301215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef81374b0 msgr2=0x7feef8139b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.837+0000 7feeee7fc640 1 --2- 172.21.15.12:0/750301215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef81374b0 0x7feef8139b90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7feef4006fd0 tx=0x7feef400ec00 comp rx=0 tx=0).stop 2024-09-11T01:44:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.837+0000 7feeee7fc640 1 -- 172.21.15.12:0/750301215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef81374b0 msgr2=0x7feef8139b90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:44:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.837+0000 7feeee7fc640 1 --2- 172.21.15.12:0/750301215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef81374b0 0x7feef8139b90 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7feef4006fd0 tx=0x7feef400ec00 comp rx=0 tx=0).stop 2024-09-11T01:44:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.837+0000 7feefd036640 1 --2- 172.21.15.12:0/750301215 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7feedc03e260 0x7feedc040720 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7feef813bac0 tx=0x7feee802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:59.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.843+0000 7feeffa9b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:44:59.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.843+0000 7feeffa9b640 1 --2- 172.21.15.12:0/750301215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef813a0d0 0x7feef813ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:59.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.843+0000 7feeffa9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:59.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.843+0000 7feeffa9b640 1 -- 172.21.15.12:0/750301215 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7feef813a4e0 con 0x7feef813a0d0 2024-09-11T01:44:59.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.843+0000 7feefd837640 1 --2- 172.21.15.12:0/750301215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef813a0d0 0x7feef813ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.843+0000 7feefd837640 10 monclient: get_auth_request con 0x7feef813a0d0 auth_method 0 2024-09-11T01:44:59.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.843+0000 7feefd837640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:59.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.843+0000 7feefd837640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:59.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.843+0000 7feefd837640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:59.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.843+0000 7feefd837640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:59.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.843+0000 7feefd837640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:59.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.843+0000 7feefd837640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:59.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.843+0000 7feefd837640 10 monclient(hunting): handle_auth_done global_id 164174 payload 824 2024-09-11T01:44:59.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.843+0000 7feefd837640 1 -- 172.21.15.12:0/750301215 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7feef813bb30 con 0x7feef813a0d0 2024-09-11T01:44:59.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.845+0000 7feefd837640 1 -- 172.21.15.12:0/750301215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef813a0d0 msgr2=0x7feef813ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:59.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.845+0000 7feefd837640 1 --2- 172.21.15.12:0/750301215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef813a0d0 0x7feef813ddb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7feef4006fd0 tx=0x7feef40046f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:59.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.845+0000 7feeee7fc640 1 -- 172.21.15.12:0/750301215 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef4017030 con 0x7feef813a0d0 2024-09-11T01:44:59.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.845+0000 7feeee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:59.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.845+0000 7feeee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:44:59.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.845+0000 7feeee7fc640 10 monclient: dump: 2024-09-11T01:44:59.856 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:59.856 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:59.856 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.856 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.856 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:59.856 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:59.856 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:59.856 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:59.857 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:59.857 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:59.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.846+0000 7feeee7fc640 1 -- 172.21.15.12:0/750301215 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7feef4004470 con 0x7feef813a0d0 2024-09-11T01:44:59.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.846+0000 7feeee7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:44:59.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.846+0000 7feeee7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:44:59.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.846+0000 7feeee7fc640 1 -- 172.21.15.12:0/750301215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef813a0d0 msgr2=0x7feef813ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:59.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.846+0000 7feeee7fc640 1 --2- 172.21.15.12:0/750301215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef813a0d0 0x7feef813ddb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7feef4006fd0 tx=0x7feef40046f0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.846+0000 7feeee7fc640 1 -- 172.21.15.12:0/750301215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef813a0d0 msgr2=0x7feef813ddb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:44:59.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.846+0000 7feeee7fc640 1 --2- 172.21.15.12:0/750301215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feef813a0d0 0x7feef813ddb0 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7feef4006fd0 tx=0x7feef40046f0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.858 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:44:59.858 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:44:59.858 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:44:59.858 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:44:59.858 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 148, 2024-09-11T01:44:59.858 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 75859, 2024-09-11T01:44:59.858 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:44:59.858 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:44:59.859 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:44:59.859 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:44:59.859 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:59.859 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:44:59.859 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:44:59.859 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:44:59.859 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:44:59.859 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:44:59.859 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:44:59.859 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:44:59.859 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:44:59.859 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:44:59.859 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:44:59.859 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:44:59.860 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:44:59.860 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:44:59.860 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:44:59.860 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:44:59.860 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:59.860 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:44:59.860 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:44:59.860 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:44:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:44:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:44:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:44:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:44:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:44:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:44:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:44:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:44:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:44:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:44:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:44:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:44:59.861 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:44:59.862 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:44:59.862 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:44:59.862 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:44:59.862 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:44:59.862 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:44:59.862 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:44:59.862 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:44:59.862 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:44:59.862 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:44:59.862 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:44:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:44:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:44:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:44:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:44:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:44:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:44:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:44:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:44:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:44:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:44:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:44:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:44:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:44:59.863 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:44:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:44:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:44:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:44:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:44:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:44:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:44:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:44:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:44:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:44:59.864 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:59.865 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:59.865 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:59.865 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:44:59.865 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:44:59.865 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:59.865 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:59.865 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:59.865 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:59.865 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:44:59.865 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:44:59.865 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:44:59.866 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:44:59.866 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:44:59.866 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:59.866 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:59.866 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:44:59.866 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:44:59.866 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:44:59.866 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:44:59.866 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:59.866 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:44:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:44:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:44:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:44:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:44:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:44:59.867 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:44:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:44:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:44:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:44:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:44:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:44:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:44:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:44:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:44:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:59.868 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:44:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:44:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:44:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:44:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:44:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:44:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:44:59.869 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:44:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:44:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:44:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:44:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:44:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:44:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:44:59.870 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:44:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:44:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:44:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:44:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:44:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:44:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:44:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:44:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:44:59.871 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:44:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:44:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:44:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:44:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:44:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:44:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:44:59.872 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:44:59.872 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:44:59.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.846+0000 7feeffa9b640 1 -- 172.21.15.12:0/750301215 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7feedc03e260 msgr2=0x7feedc040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:59.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.846+0000 7feeffa9b640 1 --2- 172.21.15.12:0/750301215 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7feedc03e260 0x7feedc040720 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7feef813bac0 tx=0x7feee802e000 comp rx=0 tx=0).stop 2024-09-11T01:44:59.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.846+0000 7feeffa9b640 10 monclient: shutdown 2024-09-11T01:44:59.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.846+0000 7feeffa9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:44:59.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.846+0000 7feeffa9b640 1 -- 172.21.15.12:0/750301215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feef813d380 msgr2=0x7feef8139650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:59.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.846+0000 7feeffa9b640 1 --2- 172.21.15.12:0/750301215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feef813d380 0x7feef8139650 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7feef0009e50 tx=0x7feef000c490 comp rx=0 tx=0).stop 2024-09-11T01:44:59.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.846+0000 7feefe038640 1 -- 172.21.15.12:0/750301215 reap_dead start 2024-09-11T01:44:59.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.846+0000 7feefe038640 1 -- 172.21.15.12:0/750301215 reap_dead start 2024-09-11T01:44:59.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.846+0000 7feeffa9b640 1 -- 172.21.15.12:0/750301215 shutdown_connections 2024-09-11T01:44:59.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.846+0000 7feeffa9b640 1 -- 172.21.15.12:0/750301215 >> 172.21.15.12:0/750301215 conn(0x7feef80a7020 msgr2=0x7feef80a8420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:44:59.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.846+0000 7feeffa9b640 1 -- 172.21.15.12:0/750301215 shutdown_connections 2024-09-11T01:44:59.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.846+0000 7feeffa9b640 1 -- 172.21.15.12:0/750301215 wait complete. 2024-09-11T01:44:59.875 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:44:59.875 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:44:59.875 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:44:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.975+0000 7f98e4f89640 10 monclient: get_monmap_and_config 2024-09-11T01:44:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.975+0000 7f98e4f89640 10 monclient: build_initial_monmap 2024-09-11T01:44:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.975+0000 7f98e4f89640 10 monclient: monmap: 2024-09-11T01:44:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:44:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:44:59.976374+0000 2024-09-11T01:44:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:44:59.976374+0000 2024-09-11T01:44:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:44:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:44:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:44:59.978 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:44:59.978 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:44:59.978 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:44:59.978 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.976+0000 7f98e4f89640 1 Processor -- start 2024-09-11T01:44:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.976+0000 7f98e4f89640 1 -- start start 2024-09-11T01:44:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.976+0000 7f98e4f89640 10 monclient: init 2024-09-11T01:44:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.976+0000 7f98e4f89640 10 monclient: _reopen_session rank -1 2024-09-11T01:44:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.976+0000 7f98e4f89640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-11T01:44:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.976+0000 7f98e4f89640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f98e0135fc0 0x7f98e01363a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.976+0000 7f98e4f89640 10 monclient(hunting): picked mon.noname-c con 0x7f98e0135fc0 addr v2:172.21.15.71:3300/0 2024-09-11T01:44:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.976+0000 7f98e4f89640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f98e0074000 addr v1:172.21.15.12:6789/0 2024-09-11T01:44:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.976+0000 7f98e4f89640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f98e0074410 addr v1:172.21.15.71:6789/0 2024-09-11T01:44:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.976+0000 7f98e4f89640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98e00732d0 con 0x7f98e0074000 2024-09-11T01:44:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.977+0000 7f98e4f89640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98e0073890 con 0x7f98e0074410 2024-09-11T01:44:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.977+0000 7f98e4f89640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.977+0000 7f98e4f89640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f98e0074820 con 0x7f98e0135fc0 2024-09-11T01:44:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.977+0000 7f98e4f89640 10 monclient(hunting): _renew_subs 2024-09-11T01:44:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.977+0000 7f98e4f89640 10 monclient(hunting): authenticate will time out at 1706.419023s 2024-09-11T01:44:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.977+0000 7f98d7fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f98e0074000 0x7f98e00a2e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37738/0 (socket says 172.21.15.12:37738) 2024-09-11T01:44:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.977+0000 7f98d7fff640 1 -- 172.21.15.12:0/3577818163 learned_addr learned my addr 172.21.15.12:0/3577818163 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:44:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.977+0000 7f98de59c640 1 --2- 172.21.15.12:0/3577818163 >> v2:172.21.15.71:3300/0 conn(0x7f98e0135fc0 0x7f98e01363a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.977+0000 7f98de59c640 10 monclient(hunting): get_auth_request con 0x7f98e0135fc0 auth_method 0 2024-09-11T01:44:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.977+0000 7f98de59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.977+0000 7f98de59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.977+0000 7f98de59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.977+0000 7f98de59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.977+0000 7f98de59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.977+0000 7f98de59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.977+0000 7f98de59c640 10 monclient(hunting): handle_auth_done global_id 134449 payload 274 2024-09-11T01:44:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.977+0000 7f98de59c640 1 -- 172.21.15.12:0/3577818163 >> v1:172.21.15.71:6789/0 conn(0x7f98e0074410 legacy=0x7f98e00a3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:44:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.977+0000 7f98de59c640 1 -- 172.21.15.12:0/3577818163 >> v1:172.21.15.12:6789/0 conn(0x7f98e0074000 legacy=0x7f98e00a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:44:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.977+0000 7f98de59c640 10 monclient: _finish_hunting 0 2024-09-11T01:44:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.977+0000 7f98de59c640 1 monclient: found mon.noname-c 2024-09-11T01:44:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.977+0000 7f98de59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:44:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.977+0000 7f98de59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:44:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.977+0000 7f98de59c640 1 -- 172.21.15.12:0/3577818163 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e00a3c40 con 0x7f98e0135fc0 2024-09-11T01:44:59.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.980+0000 7f98de59c640 1 -- 172.21.15.12:0/3577818163 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98e0135fc0 msgr2=0x7f98e01363a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:59.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.980+0000 7f98de59c640 1 --2- 172.21.15.12:0/3577818163 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98e0135fc0 0x7f98e01363a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f98d00097a0 tx=0x7f98d0029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b975058c7dad4b89 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:59.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.981+0000 7f98ddd9b640 1 -- 172.21.15.12:0/3577818163 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d0037070 con 0x7f98e0135fc0 2024-09-11T01:44:59.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.981+0000 7f98ddd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.981+0000 7f98ddd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:44:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.981+0000 7f98ddd9b640 10 monclient: dump: 2024-09-11T01:44:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:59.988 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:59.988 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:59.988 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:59.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.981+0000 7f98ddd9b640 10 monclient: _finish_auth 0 2024-09-11T01:44:59.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.981+0000 7f98ddd9b640 10 monclient: _check_auth_tickets 2024-09-11T01:44:59.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.981+0000 7f98ddd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:44:59.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.981+0000 7f98ddd9b640 1 -- 172.21.15.12:0/3577818163 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98d0032640 con 0x7f98e0135fc0 2024-09-11T01:44:59.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.981+0000 7f98e4f89640 5 monclient: authenticate success, global_id 134449 2024-09-11T01:44:59.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.981+0000 7f98e4f89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:44:59.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.981+0000 7f98ddd9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:44:59.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.981+0000 7f98e4f89640 10 monclient: get_monmap_and_config success 2024-09-11T01:44:59.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.981+0000 7f98ddd9b640 1 -- 172.21.15.12:0/3577818163 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d0032a10 con 0x7f98e0135fc0 2024-09-11T01:44:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.981+0000 7f98ddd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.981+0000 7f98e4f89640 4 set_mon_vals no callback set 2024-09-11T01:44:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.981+0000 7f98ddd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:44:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.981+0000 7f98ddd9b640 10 monclient: dump: 2024-09-11T01:44:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:44:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:44:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:44:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:44:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:44:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:44:59.989 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:44:59.989 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.981+0000 7f98e4f89640 10 monclient: shutdown 2024-09-11T01:44:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.981+0000 7f98e4f89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:44:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.981+0000 7f98e4f89640 1 -- 172.21.15.12:0/3577818163 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98e0135fc0 msgr2=0x7f98e01363a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.981+0000 7f98e4f89640 1 --2- 172.21.15.12:0/3577818163 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98e0135fc0 0x7f98e01363a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f98d00097a0 tx=0x7f98d0029fd0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.981+0000 7f98e4f89640 1 -- 172.21.15.12:0/3577818163 shutdown_connections 2024-09-11T01:44:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.981+0000 7f98e4f89640 1 --2- 172.21.15.12:0/3577818163 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98e0135fc0 0x7f98e01363a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.981+0000 7f98e4f89640 1 -- 172.21.15.12:0/3577818163 >> 172.21.15.12:0/3577818163 conn(0x7f98e00a8600 msgr2=0x7f98e00a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:44:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.981+0000 7f98e4f89640 1 -- 172.21.15.12:0/3577818163 shutdown_connections 2024-09-11T01:44:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.981+0000 7f98e4f89640 1 -- 172.21.15.12:0/3577818163 wait complete. 2024-09-11T01:44:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.982+0000 7f98e4f89640 10 monclient: build_initial_monmap 2024-09-11T01:44:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.982+0000 7f98e4f89640 10 monclient: monmap: 2024-09-11T01:44:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:44:59.990 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:44:59.991 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:44:59.991 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:44:59.991 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:44:59.991 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:44:59.991 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:44:59.991 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:44:59.991 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:44:59.991 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:44:59.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.982+0000 7f98e4f89640 1 Processor -- start 2024-09-11T01:44:59.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.982+0000 7f98e4f89640 1 -- start start 2024-09-11T01:44:59.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.982+0000 7f98e4f89640 10 monclient: init 2024-09-11T01:44:59.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98e4f89640 10 monclient: _reopen_session rank -1 2024-09-11T01:44:59.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98e4f89640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:44:59.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98e4f89640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98e0074000 0x7f98e013dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:59.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98e4f89640 10 monclient(hunting): picked mon.noname-a con 0x7f98e0074000 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:44:59.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98e4f89640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98e0074410 0x7f98e006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:59.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98e4f89640 10 monclient(hunting): picked mon.noname-b con 0x7f98e0074410 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:44:59.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98e4f89640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f98e0135fc0 0x7f98e006e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:44:59.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98e4f89640 10 monclient(hunting): picked mon.noname-c con 0x7f98e0135fc0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:44:59.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98e4f89640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:59.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98e4f89640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f98e01368e0 con 0x7f98e0074000 2024-09-11T01:44:59.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98e4f89640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:59.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98e4f89640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f98e0072ff0 con 0x7f98e0074410 2024-09-11T01:44:59.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98e4f89640 10 monclient(hunting): start opening mon connection 2024-09-11T01:44:59.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98e4f89640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f98e00a7ec0 con 0x7f98e0135fc0 2024-09-11T01:44:59.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98e4f89640 10 monclient(hunting): _renew_subs 2024-09-11T01:44:59.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98e4f89640 10 monclient(hunting): authenticate will time out at 1706.425179s 2024-09-11T01:44:59.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98de59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98e0074000 0x7f98e013dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98de59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98e0074000 0x7f98e013dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44732/0 (socket says 172.21.15.12:44732) 2024-09-11T01:44:59.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98de59c640 1 -- 172.21.15.12:0/1726904443 learned_addr learned my addr 172.21.15.12:0/1726904443 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:44:59.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98de59c640 10 monclient(hunting): get_auth_request con 0x7f98e0074000 auth_method 0 2024-09-11T01:44:59.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98ded9d640 1 --2- 172.21.15.12:0/1726904443 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f98e0135fc0 0x7f98e006e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98de59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:59.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98de59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:59.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98de59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:59.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98ded9d640 10 monclient(hunting): get_auth_request con 0x7f98e0135fc0 auth_method 0 2024-09-11T01:44:59.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98ded9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:59.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98ded9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:59.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98ded9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:59.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98de59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:59.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98de59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:59.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98de59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:59.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98ded9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:44:59.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98ded9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:44:59.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98ded9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:44:59.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98d7fff640 1 --2- 172.21.15.12:0/1726904443 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98e0074410 0x7f98e006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:44:59.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98d7fff640 10 monclient(hunting): get_auth_request con 0x7f98e0074410 auth_method 0 2024-09-11T01:44:59.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:44:59.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:44:59.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:44:59.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98de59c640 10 monclient(hunting): handle_auth_done global_id 174120 payload 824 2024-09-11T01:44:59.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98de59c640 1 -- 172.21.15.12:0/1726904443 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f98e0135fc0 msgr2=0x7f98e006e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:59.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98de59c640 1 --2- 172.21.15.12:0/1726904443 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f98e0135fc0 0x7f98e006e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98de59c640 1 -- 172.21.15.12:0/1726904443 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98e0074410 msgr2=0x7f98e006dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:44:59.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98de59c640 1 --2- 172.21.15.12:0/1726904443 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98e0074410 0x7f98e006dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:44:59.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98de59c640 10 monclient: _finish_hunting 0 2024-09-11T01:44:59.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98de59c640 1 monclient: found mon.noname-a 2024-09-11T01:44:59.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98de59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:44:59.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98de59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:44:59.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98de59c640 1 -- 172.21.15.12:0/1726904443 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e013c1c0 con 0x7f98e0074000 2024-09-11T01:44:59.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.983+0000 7f98ded9d640 1 --2- 172.21.15.12:0/1726904443 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f98e0135fc0 0x7f98e006e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:44:59.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.985+0000 7f98de59c640 1 -- 172.21.15.12:0/1726904443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98e0074000 msgr2=0x7f98e013dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:44:59.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.985+0000 7f98de59c640 1 --2- 172.21.15.12:0/1726904443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98e0074000 0x7f98e013dcb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f98d00098c0 tx=0x7f98d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:44:59.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.986+0000 7f98d6ffd640 1 -- 172.21.15.12:0/1726904443 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d0041070 con 0x7f98e0074000 2024-09-11T01:44:59.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.986+0000 7f98d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:44:59.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.986+0000 7f98d6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:44:59.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.986+0000 7f98d6ffd640 10 monclient: dump: 2024-09-11T01:45:00.000 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:00.000 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:00.000 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:00.000 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:00.000 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:00.000 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:00.000 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:00.000 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:00.000 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:00.000 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:00.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.986+0000 7f98d6ffd640 10 monclient: _finish_auth 0 2024-09-11T01:45:00.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.986+0000 7f98d6ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:45:00.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.986+0000 7f98d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:00.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.986+0000 7f98d6ffd640 1 -- 172.21.15.12:0/1726904443 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98d003c880 con 0x7f98e0074000 2024-09-11T01:45:00.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.986+0000 7f98d6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:00.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.986+0000 7f98d6ffd640 1 -- 172.21.15.12:0/1726904443 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d0037040 con 0x7f98e0074000 2024-09-11T01:45:00.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.986+0000 7f98e4f89640 5 monclient: authenticate success, global_id 174120 2024-09-11T01:45:00.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.986+0000 7f98d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:00.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.986+0000 7f98d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:00.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.986+0000 7f98d6ffd640 10 monclient: dump: 2024-09-11T01:45:00.001 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:00.001 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:00.001 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:00.001 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:00.001 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:00.001 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:00.002 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:00.002 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:00.002 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:00.002 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:00.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.986+0000 7f98e4f89640 10 monclient: _renew_subs 2024-09-11T01:45:00.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.986+0000 7f98e4f89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:00.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.986+0000 7f98dd59a640 4 set_mon_vals no callback set 2024-09-11T01:45:00.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.986+0000 7f98e4f89640 1 -- 172.21.15.12:0/1726904443 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98e00a7900 con 0x7f98e0074000 2024-09-11T01:45:00.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.986+0000 7f98e4f89640 10 monclient: _renew_subs 2024-09-11T01:45:00.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.986+0000 7f98e4f89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:00.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.986+0000 7f98e4f89640 1 -- 172.21.15.12:0/1726904443 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98e013c780 con 0x7f98e0074000 2024-09-11T01:45:00.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.986+0000 7f98e4f89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:00.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.986+0000 7f98e4f89640 1 --2- 172.21.15.12:0/1726904443 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98e013e1f0 0x7f98e006e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:00.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.987+0000 7f98e4f89640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:00.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.987+0000 7f98e4f89640 1 -- 172.21.15.12:0/1726904443 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f98e006a060 con 0x7f98e013e1f0 2024-09-11T01:45:00.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.987+0000 7f98d6ffd640 1 -- 172.21.15.12:0/1726904443 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f98d003cc70 con 0x7f98e0074000 2024-09-11T01:45:00.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.987+0000 7f98d6ffd640 1 --2- 172.21.15.12:0/1726904443 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f98c003e210 0x7f98c00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:00.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.987+0000 7f98d6ffd640 1 -- 172.21.15.12:0/1726904443 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(533..533 src has 317..533) ==== 4755+0+0 (secure 0 0 0) 0x7f98d0073a60 con 0x7f98e0074000 2024-09-11T01:45:00.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.987+0000 7f98d7fff640 1 --2- 172.21.15.12:0/1726904443 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98e013e1f0 0x7f98e006e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:00.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.987+0000 7f98d7fff640 10 monclient: get_auth_request con 0x7f98e013e1f0 auth_method 0 2024-09-11T01:45:00.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.987+0000 7f98d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:00.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.987+0000 7f98d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:00.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.987+0000 7f98d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:00.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.987+0000 7f98ded9d640 1 --2- 172.21.15.12:0/1726904443 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f98c003e210 0x7f98c00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:00.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.987+0000 7f98d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:00.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.987+0000 7f98d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:00.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.987+0000 7f98d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:00.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.987+0000 7f98ded9d640 10 monclient: get_auth_request con 0x7f98c003e210 auth_method 0 2024-09-11T01:45:00.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.987+0000 7f98d7fff640 10 monclient(hunting): handle_auth_done global_id 134455 payload 824 2024-09-11T01:45:00.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.987+0000 7f98d7fff640 1 -- 172.21.15.12:0/1726904443 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f98e01363a0 con 0x7f98e013e1f0 2024-09-11T01:45:00.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.990+0000 7f98d7fff640 1 -- 172.21.15.12:0/1726904443 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98e013e1f0 msgr2=0x7f98e006e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:00.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.990+0000 7f98d7fff640 1 --2- 172.21.15.12:0/1726904443 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98e013e1f0 0x7f98e006e710 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f98c800e4d0 tx=0x7f98c800e990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:00.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.990+0000 7f98ded9d640 1 --2- 172.21.15.12:0/1726904443 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f98c003e210 0x7f98c00406d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f98d800f1d0 tx=0x7f98d8003d10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:00.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.990+0000 7f98d6ffd640 1 -- 172.21.15.12:0/1726904443 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98c8004e30 con 0x7f98e013e1f0 2024-09-11T01:45:00.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.990+0000 7f98d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:00.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.990+0000 7f98d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:45:00.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.990+0000 7f98d6ffd640 10 monclient: dump: 2024-09-11T01:45:00.004 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:00.004 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:00.005 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:00.005 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:00.005 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:00.005 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:00.005 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:00.005 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:00.005 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:00.005 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:00.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.990+0000 7f98d6ffd640 1 -- 172.21.15.12:0/1726904443 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f98c8018070 con 0x7f98e013e1f0 2024-09-11T01:45:00.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.990+0000 7f98d6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:00.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.990+0000 7f98d6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:45:00.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.990+0000 7f98d6ffd640 1 -- 172.21.15.12:0/1726904443 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98e013e1f0 msgr2=0x7f98e006e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:00.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.990+0000 7f98d6ffd640 1 --2- 172.21.15.12:0/1726904443 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98e013e1f0 0x7f98e006e710 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f98c800e4d0 tx=0x7f98c800e990 comp rx=0 tx=0).stop 2024-09-11T01:45:00.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.990+0000 7f98d6ffd640 1 -- 172.21.15.12:0/1726904443 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98e013e1f0 msgr2=0x7f98e006e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:00.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.990+0000 7f98d6ffd640 1 --2- 172.21.15.12:0/1726904443 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98e013e1f0 0x7f98e006e710 secure :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f98c800e4d0 tx=0x7f98c800e990 comp rx=0 tx=0).stop 2024-09-11T01:45:00.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.996+0000 7f98e4f89640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:45:00.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.996+0000 7f98e4f89640 1 --2- 172.21.15.12:0/1726904443 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98e0136a60 0x7f98e0136e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:00.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.996+0000 7f98e4f89640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:00.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.996+0000 7f98e4f89640 1 -- 172.21.15.12:0/1726904443 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f98e0069950 con 0x7f98e0136a60 2024-09-11T01:45:00.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.997+0000 7f98d7fff640 1 --2- 172.21.15.12:0/1726904443 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98e0136a60 0x7f98e0136e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:00.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.997+0000 7f98d7fff640 10 monclient: get_auth_request con 0x7f98e0136a60 auth_method 0 2024-09-11T01:45:00.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.997+0000 7f98d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:00.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.997+0000 7f98d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:00.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.997+0000 7f98d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:00.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.997+0000 7f98d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:00.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.997+0000 7f98d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:00.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.997+0000 7f98d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:00.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.997+0000 7f98d7fff640 10 monclient(hunting): handle_auth_done global_id 134458 payload 824 2024-09-11T01:45:00.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:44:59.997+0000 7f98d7fff640 1 -- 172.21.15.12:0/1726904443 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f98e0135390 con 0x7f98e0136a60 2024-09-11T01:45:00.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.000+0000 7f98d7fff640 1 -- 172.21.15.12:0/1726904443 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98e0136a60 msgr2=0x7f98e0136e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:00.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.000+0000 7f98d7fff640 1 --2- 172.21.15.12:0/1726904443 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98e0136a60 0x7f98e0136e70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f98c8017000 tx=0x7f98c8004940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:00.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.000+0000 7f98d6ffd640 1 -- 172.21.15.12:0/1726904443 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98c80064e0 con 0x7f98e0136a60 2024-09-11T01:45:00.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.000+0000 7f98d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:00.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.000+0000 7f98d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:45:00.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.000+0000 7f98d6ffd640 10 monclient: dump: 2024-09-11T01:45:00.008 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:00.008 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:00.008 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:00.008 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:00.008 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:00.009 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:00.009 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:00.009 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:00.009 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:00.009 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:00.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.000+0000 7f98d6ffd640 1 -- 172.21.15.12:0/1726904443 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7f98c80064e0 con 0x7f98e0136a60 2024-09-11T01:45:00.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.000+0000 7f98d6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:45:00.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.000+0000 7f98d6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:45:00.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.000+0000 7f98d6ffd640 1 -- 172.21.15.12:0/1726904443 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98e0136a60 msgr2=0x7f98e0136e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:00.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.000+0000 7f98d6ffd640 1 --2- 172.21.15.12:0/1726904443 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98e0136a60 0x7f98e0136e70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f98c8017000 tx=0x7f98c8004940 comp rx=0 tx=0).stop 2024-09-11T01:45:00.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.000+0000 7f98d6ffd640 1 -- 172.21.15.12:0/1726904443 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98e0136a60 msgr2=0x7f98e0136e70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:00.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.000+0000 7f98d6ffd640 1 --2- 172.21.15.12:0/1726904443 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98e0136a60 0x7f98e0136e70 secure :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f98c8017000 tx=0x7f98c8004940 comp rx=0 tx=0).stop 2024-09-11T01:45:00.010 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:45:00.010 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:45:00.010 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:45:00.010 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:45:00.010 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 148, 2024-09-11T01:45:00.010 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 232962, 2024-09-11T01:45:00.010 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:45:00.010 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:45:00.010 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:45:00.010 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:45:00.010 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:00.011 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:45:00.011 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:45:00.011 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:45:00.011 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:45:00.011 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:00.011 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:00.011 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:00.011 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:00.011 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:00.011 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:00.011 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:00.011 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:00.011 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:00.012 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:00.012 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:00.012 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:00.012 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:45:00.012 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:45:00.012 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:00.012 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:00.012 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:00.012 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:00.012 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:00.012 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:00.012 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:00.012 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:00.013 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:00.013 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:00.013 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:00.013 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:00.013 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:00.013 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:45:00.013 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:45:00.013 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:45:00.013 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:45:00.013 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:45:00.013 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:45:00.013 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:45:00.013 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:45:00.013 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:45:00.014 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:45:00.014 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:45:00.014 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:45:00.014 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:45:00.014 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:45:00.014 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:45:00.014 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:45:00.014 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:45:00.014 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:00.014 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:00.014 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:00.014 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:00.014 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:00.019 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:00.019 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:00.019 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:00.019 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:00.019 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:00.019 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:00.020 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:00.020 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:45:00.020 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:00.020 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:45:00.020 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:00.020 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:45:00.020 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:45:00.021 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:00.021 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:00.021 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:00.021 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:00.021 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:45:00.021 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:00.021 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:00.022 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:00.022 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:00.022 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:45:00.022 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:00.022 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:00.022 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:00.022 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:00.022 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:45:00.023 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:45:00.023 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:00.023 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:00.023 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:00.023 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:00.023 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:00.023 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:45:00.023 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:45:00.023 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:00.023 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:00.023 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:00.023 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:00.023 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:45:00.023 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:00.024 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:00.024 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:00.024 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:00.024 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:45:00.024 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:00.024 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:00.024 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:00.024 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:00.024 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:45:00.024 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:45:00.024 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:00.024 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:00.024 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:00.025 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:00.025 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:00.025 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:45:00.025 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:45:00.025 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:00.025 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:00.025 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:00.025 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:00.025 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:45:00.025 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:00.025 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:00.025 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:00.025 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:00.026 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:45:00.026 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:00.026 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:00.026 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:00.026 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:00.026 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:45:00.026 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:45:00.026 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:00.026 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:00.026 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:00.026 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:00.026 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:00.026 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:00.026 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:45:00.027 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:45:00.027 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:00.027 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:00.027 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:00.027 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:45:00.027 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:00.027 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:00.027 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:45:00.027 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:00.027 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:00.028 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:00.028 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:45:00.028 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:00.028 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:00.028 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:45:00.028 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:00.028 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:00.028 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:00.028 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:45:00.028 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:00.029 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:00.029 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:00.029 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:45:00.029 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:45:00.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.001+0000 7f98e4f89640 1 -- 172.21.15.12:0/1726904443 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f98c003e210 msgr2=0x7f98c00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:00.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.001+0000 7f98e4f89640 1 --2- 172.21.15.12:0/1726904443 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f98c003e210 0x7f98c00406d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f98d800f1d0 tx=0x7f98d8003d10 comp rx=0 tx=0).stop 2024-09-11T01:45:00.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.001+0000 7f98e4f89640 10 monclient: shutdown 2024-09-11T01:45:00.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.001+0000 7f98e4f89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:00.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.001+0000 7f98e4f89640 1 -- 172.21.15.12:0/1726904443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98e0074000 msgr2=0x7f98e013dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:00.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.001+0000 7f98e4f89640 1 --2- 172.21.15.12:0/1726904443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98e0074000 0x7f98e013dcb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f98d00098c0 tx=0x7f98d00046c0 comp rx=0 tx=0).stop 2024-09-11T01:45:00.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.001+0000 7f98ded9d640 1 -- 172.21.15.12:0/1726904443 reap_dead start 2024-09-11T01:45:00.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.001+0000 7f98ded9d640 1 -- 172.21.15.12:0/1726904443 reap_dead start 2024-09-11T01:45:00.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.001+0000 7f98e4f89640 1 -- 172.21.15.12:0/1726904443 shutdown_connections 2024-09-11T01:45:00.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.001+0000 7f98e4f89640 1 -- 172.21.15.12:0/1726904443 >> 172.21.15.12:0/1726904443 conn(0x7f98e00a8600 msgr2=0x7f98e00a6c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:00.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.001+0000 7f98e4f89640 1 -- 172.21.15.12:0/1726904443 shutdown_connections 2024-09-11T01:45:00.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.001+0000 7f98e4f89640 1 -- 172.21.15.12:0/1726904443 wait complete. 2024-09-11T01:45:00.031 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:45:00.031 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:45:00.032 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:45:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.130+0000 7f70e673a640 10 monclient: get_monmap_and_config 2024-09-11T01:45:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.130+0000 7f70e673a640 10 monclient: build_initial_monmap 2024-09-11T01:45:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.130+0000 7f70e673a640 10 monclient: monmap: 2024-09-11T01:45:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:45:00.131346+0000 2024-09-11T01:45:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:45:00.131346+0000 2024-09-11T01:45:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:45:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:45:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:45:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:45:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:45:00.133 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:00.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.131+0000 7f70e673a640 1 Processor -- start 2024-09-11T01:45:00.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.131+0000 7f70e673a640 1 -- start start 2024-09-11T01:45:00.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.131+0000 7f70e673a640 10 monclient: init 2024-09-11T01:45:00.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.131+0000 7f70e673a640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:00.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.131+0000 7f70e673a640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-11T01:45:00.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.131+0000 7f70e673a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70e0134ad0 0x7f70e0134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:00.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70e673a640 10 monclient(hunting): picked mon.noname-b con 0x7f70e0134ad0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:00.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70e673a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f70e00a2200 addr v1:172.21.15.71:6789/0 2024-09-11T01:45:00.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70e673a640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f70e01353f0 0x7f70e0134160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:00.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70e673a640 10 monclient(hunting): picked mon.noname-c con 0x7f70e01353f0 addr v2:172.21.15.71:3300/0 2024-09-11T01:45:00.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70e673a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70e009ed30 con 0x7f70e00a2200 2024-09-11T01:45:00.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70e673a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:00.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70e673a640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f70e009f5d0 con 0x7f70e01353f0 2024-09-11T01:45:00.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70e673a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:00.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70e673a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f70e009f010 con 0x7f70e0134ad0 2024-09-11T01:45:00.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70e673a640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:00.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70e673a640 10 monclient(hunting): authenticate will time out at 1706.574110s 2024-09-11T01:45:00.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70dffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70e0134ad0 0x7f70e0134eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:00.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70dffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70e0134ad0 0x7f70e0134eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:54114/0 (socket says 172.21.15.12:54114) 2024-09-11T01:45:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70dffff640 1 -- 172.21.15.12:0/2845446220 learned_addr learned my addr 172.21.15.12:0/2845446220 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70dffff640 10 monclient(hunting): get_auth_request con 0x7f70e0134ad0 auth_method 0 2024-09-11T01:45:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70dffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70dffff640 10 monclient(hunting): handle_auth_done global_id 164180 payload 274 2024-09-11T01:45:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70e4cd7640 1 --2- 172.21.15.12:0/2845446220 >> v2:172.21.15.71:3300/0 conn(0x7f70e01353f0 0x7f70e0134160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70dffff640 1 -- 172.21.15.12:0/2845446220 >> v2:172.21.15.71:3300/0 conn(0x7f70e01353f0 msgr2=0x7f70e0134160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70dffff640 1 --2- 172.21.15.12:0/2845446220 >> v2:172.21.15.71:3300/0 conn(0x7f70e01353f0 0x7f70e0134160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70dffff640 1 -- 172.21.15.12:0/2845446220 >> v1:172.21.15.71:6789/0 conn(0x7f70e00a2200 legacy=0x7f70e0133a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70dffff640 10 monclient: _finish_hunting 0 2024-09-11T01:45:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70dffff640 1 monclient: found mon.noname-b 2024-09-11T01:45:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70dffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:45:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70dffff640 1 -- 172.21.15.12:0/2845446220 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70e009f2f0 con 0x7f70e0134ad0 2024-09-11T01:45:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70e4cd7640 10 monclient: get_auth_request con 0x7f70e01353f0 auth_method 0 2024-09-11T01:45:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.132+0000 7f70e4cd7640 1 --2- 172.21.15.12:0/2845446220 >> v2:172.21.15.71:3300/0 conn(0x7f70e01353f0 0x7f70e0134160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:45:00.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.135+0000 7f70dffff640 1 -- 172.21.15.12:0/2845446220 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70e0134ad0 msgr2=0x7f70e0134eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:00.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.135+0000 7f70dffff640 1 --2- 172.21.15.12:0/2845446220 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70e0134ad0 0x7f70e0134eb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f70d0009870 tx=0x7f70d002a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7a9527bdc2a677d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:00.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.135+0000 7f70deffd640 1 -- 172.21.15.12:0/2845446220 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d0037070 con 0x7f70e0134ad0 2024-09-11T01:45:00.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.135+0000 7f70deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:00.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.135+0000 7f70deffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:45:00.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.135+0000 7f70deffd640 10 monclient: dump: 2024-09-11T01:45:00.144 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:00.144 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:00.144 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:00.144 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:00.144 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:00.144 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:00.144 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:00.144 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:00.144 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:00.145 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:00.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.135+0000 7f70deffd640 10 monclient: _finish_auth 0 2024-09-11T01:45:00.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.135+0000 7f70deffd640 10 monclient: _check_auth_tickets 2024-09-11T01:45:00.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.135+0000 7f70deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:00.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.135+0000 7f70e673a640 5 monclient: authenticate success, global_id 164180 2024-09-11T01:45:00.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.135+0000 7f70e673a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:45:00.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.135+0000 7f70deffd640 1 -- 172.21.15.12:0/2845446220 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70d0032640 con 0x7f70e0134ad0 2024-09-11T01:45:00.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.135+0000 7f70deffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:00.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.135+0000 7f70deffd640 1 -- 172.21.15.12:0/2845446220 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d0032a10 con 0x7f70e0134ad0 2024-09-11T01:45:00.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.135+0000 7f70e673a640 10 monclient: get_monmap_and_config success 2024-09-11T01:45:00.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.135+0000 7f70e673a640 4 set_mon_vals no callback set 2024-09-11T01:45:00.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.135+0000 7f70deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:00.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.135+0000 7f70deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:45:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.135+0000 7f70deffd640 10 monclient: dump: 2024-09-11T01:45:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:00.146 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.135+0000 7f70e673a640 10 monclient: shutdown 2024-09-11T01:45:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.135+0000 7f70e673a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:00.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.135+0000 7f70e673a640 1 -- 172.21.15.12:0/2845446220 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70e0134ad0 msgr2=0x7f70e0134eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:00.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.135+0000 7f70e673a640 1 --2- 172.21.15.12:0/2845446220 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70e0134ad0 0x7f70e0134eb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f70d0009870 tx=0x7f70d002a0a0 comp rx=0 tx=0).stop 2024-09-11T01:45:00.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.136+0000 7f70e673a640 1 -- 172.21.15.12:0/2845446220 shutdown_connections 2024-09-11T01:45:00.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.136+0000 7f70e673a640 1 --2- 172.21.15.12:0/2845446220 >> v2:172.21.15.71:3300/0 conn(0x7f70e01353f0 0x7f70e0134160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:00.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.136+0000 7f70e673a640 1 --2- 172.21.15.12:0/2845446220 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70e0134ad0 0x7f70e0134eb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:00.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.136+0000 7f70e673a640 1 -- 172.21.15.12:0/2845446220 >> 172.21.15.12:0/2845446220 conn(0x7f70e00a2f70 msgr2=0x7f70e00a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:00.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.136+0000 7f70e673a640 1 -- 172.21.15.12:0/2845446220 shutdown_connections 2024-09-11T01:45:00.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.136+0000 7f70e673a640 1 -- 172.21.15.12:0/2845446220 wait complete. 2024-09-11T01:45:00.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.136+0000 7f70e673a640 10 monclient: build_initial_monmap 2024-09-11T01:45:00.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.136+0000 7f70e673a640 10 monclient: monmap: 2024-09-11T01:45:00.147 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:00.147 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:45:00.147 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:45:00.147 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:45:00.148 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:00.148 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:00.148 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:45:00.148 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:45:00.148 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:45:00.148 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:00.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.136+0000 7f70e673a640 1 Processor -- start 2024-09-11T01:45:00.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e673a640 1 -- start start 2024-09-11T01:45:00.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e673a640 10 monclient: init 2024-09-11T01:45:00.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e673a640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:00.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e673a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:45:00.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e673a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70e00a2200 0x7f70e006f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:00.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e673a640 10 monclient(hunting): picked mon.noname-b con 0x7f70e00a2200 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:45:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e673a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70e0134ad0 0x7f70e006fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e673a640 10 monclient(hunting): picked mon.noname-c con 0x7f70e0134ad0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e673a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70e01353f0 0x7f70e0070110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e673a640 10 monclient(hunting): picked mon.noname-a con 0x7f70e01353f0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:45:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e673a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e673a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f70e0134820 con 0x7f70e01353f0 2024-09-11T01:45:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e673a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e673a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f70e01346a0 con 0x7f70e00a2200 2024-09-11T01:45:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e673a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e673a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f70e009ffd0 con 0x7f70e0134ad0 2024-09-11T01:45:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e673a640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e673a640 10 monclient(hunting): authenticate will time out at 1706.579739s 2024-09-11T01:45:00.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70df7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70e0134ad0 0x7f70e006fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:00.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e4cd7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70e01353f0 0x7f70e0070110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:00.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70df7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70e0134ad0 0x7f70e006fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:54128/0 (socket says 172.21.15.12:54128) 2024-09-11T01:45:00.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e4cd7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70e01353f0 0x7f70e0070110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44744/0 (socket says 172.21.15.12:44744) 2024-09-11T01:45:00.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70df7fe640 1 -- 172.21.15.12:0/1636499029 learned_addr learned my addr 172.21.15.12:0/1636499029 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:00.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e4cd7640 10 monclient(hunting): get_auth_request con 0x7f70e01353f0 auth_method 0 2024-09-11T01:45:00.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e4cd7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:00.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e4cd7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:00.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e4cd7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:00.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70df7fe640 10 monclient(hunting): get_auth_request con 0x7f70e0134ad0 auth_method 0 2024-09-11T01:45:00.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:00.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:00.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:00.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e4cd7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:00.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e4cd7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:00.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70e4cd7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:00.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:00.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.137+0000 7f70df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:00.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.138+0000 7f70df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:00.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.138+0000 7f70dffff640 1 --2- 172.21.15.12:0/1636499029 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70e00a2200 0x7f70e006f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:00.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.138+0000 7f70dffff640 10 monclient(hunting): get_auth_request con 0x7f70e00a2200 auth_method 0 2024-09-11T01:45:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.138+0000 7f70dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.138+0000 7f70dffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.138+0000 7f70dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.138+0000 7f70e4cd7640 10 monclient(hunting): handle_auth_done global_id 174123 payload 824 2024-09-11T01:45:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.138+0000 7f70e4cd7640 1 -- 172.21.15.12:0/1636499029 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70e0134ad0 msgr2=0x7f70e006fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.138+0000 7f70e4cd7640 1 --2- 172.21.15.12:0/1636499029 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70e0134ad0 0x7f70e006fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.138+0000 7f70e4cd7640 1 -- 172.21.15.12:0/1636499029 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70e00a2200 msgr2=0x7f70e006f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.138+0000 7f70e4cd7640 1 --2- 172.21.15.12:0/1636499029 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70e00a2200 0x7f70e006f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.138+0000 7f70e4cd7640 10 monclient: _finish_hunting 0 2024-09-11T01:45:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.138+0000 7f70e4cd7640 1 monclient: found mon.noname-a 2024-09-11T01:45:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.138+0000 7f70e4cd7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.138+0000 7f70e4cd7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:45:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.138+0000 7f70e4cd7640 1 -- 172.21.15.12:0/1636499029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70e0072e40 con 0x7f70e01353f0 2024-09-11T01:45:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.138+0000 7f70df7fe640 1 --2- 172.21.15.12:0/1636499029 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70e0134ad0 0x7f70e006fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:45:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.140+0000 7f70e4cd7640 1 -- 172.21.15.12:0/1636499029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70e01353f0 msgr2=0x7f70e0070110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.140+0000 7f70e4cd7640 1 --2- 172.21.15.12:0/1636499029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70e01353f0 0x7f70e0070110 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f70d800ce20 tx=0x7f70d800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.140+0000 7f70dcff9640 1 -- 172.21.15.12:0/1636499029 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d801a070 con 0x7f70e01353f0 2024-09-11T01:45:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.140+0000 7f70dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.140+0000 7f70dcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:45:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.140+0000 7f70dcff9640 10 monclient: dump: 2024-09-11T01:45:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:00.156 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.140+0000 7f70dcff9640 10 monclient: _finish_auth 0 2024-09-11T01:45:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.140+0000 7f70dcff9640 10 monclient: _check_auth_tickets 2024-09-11T01:45:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.140+0000 7f70dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.140+0000 7f70e673a640 5 monclient: authenticate success, global_id 174123 2024-09-11T01:45:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.140+0000 7f70dcff9640 1 -- 172.21.15.12:0/1636499029 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70d800da50 con 0x7f70e01353f0 2024-09-11T01:45:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.140+0000 7f70e673a640 10 monclient: _renew_subs 2024-09-11T01:45:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.140+0000 7f70e673a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.140+0000 7f70e673a640 1 -- 172.21.15.12:0/1636499029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70e0073120 con 0x7f70e01353f0 2024-09-11T01:45:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.140+0000 7f70dcff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.140+0000 7f70dcff9640 1 -- 172.21.15.12:0/1636499029 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d800de40 con 0x7f70e01353f0 2024-09-11T01:45:00.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.140+0000 7f70dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:00.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.140+0000 7f70dcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:00.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.140+0000 7f70dcff9640 10 monclient: dump: 2024-09-11T01:45:00.157 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:00.157 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:00.157 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:00.157 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:00.157 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:00.157 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:00.157 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:00.157 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:00.158 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:00.158 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:00.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.140+0000 7f70e673a640 10 monclient: _renew_subs 2024-09-11T01:45:00.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.140+0000 7f70e673a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:00.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.140+0000 7f70e673a640 1 -- 172.21.15.12:0/1636499029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70e0071670 con 0x7f70e01353f0 2024-09-11T01:45:00.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.140+0000 7f70de7fc640 4 set_mon_vals no callback set 2024-09-11T01:45:00.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.141+0000 7f70dcff9640 1 -- 172.21.15.12:0/1636499029 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f70d801e3b0 con 0x7f70e01353f0 2024-09-11T01:45:00.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.141+0000 7f70e673a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:00.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.141+0000 7f70e673a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:00.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.141+0000 7f70e673a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:00.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.141+0000 7f70e673a640 1 -- 172.21.15.12:0/1636499029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70e0134eb0 con 0x7f70e01353f0 2024-09-11T01:45:00.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.141+0000 7f70dcff9640 1 --2- 172.21.15.12:0/1636499029 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f70cc03e210 0x7f70cc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:00.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.141+0000 7f70dcff9640 1 -- 172.21.15.12:0/1636499029 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(533..533 src has 317..533) ==== 4755+0+0 (secure 0 0 0) 0x7f70d80538a0 con 0x7f70e01353f0 2024-09-11T01:45:00.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.141+0000 7f70dffff640 1 --2- 172.21.15.12:0/1636499029 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f70cc03e210 0x7f70cc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:00.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.141+0000 7f70dffff640 10 monclient: get_auth_request con 0x7f70cc03e210 auth_method 0 2024-09-11T01:45:00.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.144+0000 7f70dffff640 1 --2- 172.21.15.12:0/1636499029 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f70cc03e210 0x7f70cc0406d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f70e006ce00 tx=0x7f70d002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:00.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.146+0000 7f70dcff9640 1 -- 172.21.15.12:0/1636499029 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f70d80522b0 con 0x7f70e01353f0 2024-09-11T01:45:00.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.146+0000 7f70dcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:00.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.146+0000 7f70dcff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:45:00.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.407+0000 7f70e673a640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:45:00.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.407+0000 7f70e673a640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:45:00.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.407+0000 7f70e673a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:00.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.407+0000 7f70e673a640 1 -- 172.21.15.12:0/1636499029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f70e0072ac0 con 0x7f70e01353f0 2024-09-11T01:45:00.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.408+0000 7f70dcff9640 1 -- 172.21.15.12:0/1636499029 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f70d801ccd0 con 0x7f70e01353f0 2024-09-11T01:45:00.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.408+0000 7f70dcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:45:00.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.408+0000 7f70dcff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:45:00.412 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.410+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.412 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.410+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=12,kv=11,logm=50} crc {auth=2885979850,config=2586702516,health=327601140,kv=1703855226,logm=2602320018}) 2024-09-11T01:45:00.413 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.410+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=25,config=2,health=12,kv=12,logm=49} crc {auth=2792245813,config=2586702516,health=327601140,kv=3263697420,logm=3492151329}) 2024-09-11T01:45:00.413 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.410+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.413 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.410+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=12,kv=11,logm=50} crc {auth=2885979850,config=2586702516,health=327601140,kv=1703855226,logm=2602320018}) 2024-09-11T01:45:00.413 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.410+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=11,kv=12,logm=49} crc {auth=2261948905,config=2586702516,health=2473239523,kv=3263697420,logm=3697066476}) 2024-09-11T01:45:00.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.411+0000 7f70e673a640 1 -- 172.21.15.12:0/1636499029 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f70cc03e210 msgr2=0x7f70cc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:00.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.411+0000 7f70e673a640 1 --2- 172.21.15.12:0/1636499029 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f70cc03e210 0x7f70cc0406d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f70e006ce00 tx=0x7f70d002e000 comp rx=0 tx=0).stop 2024-09-11T01:45:00.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.411+0000 7f70e673a640 10 monclient: shutdown 2024-09-11T01:45:00.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.411+0000 7f70e673a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:00.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.411+0000 7f70e673a640 1 -- 172.21.15.12:0/1636499029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70e01353f0 msgr2=0x7f70e0070110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:00.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.411+0000 7f70e673a640 1 --2- 172.21.15.12:0/1636499029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70e01353f0 0x7f70e0070110 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f70d800ce20 tx=0x7f70d800a710 comp rx=0 tx=0).stop 2024-09-11T01:45:00.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.411+0000 7f70e673a640 1 -- 172.21.15.12:0/1636499029 shutdown_connections 2024-09-11T01:45:00.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.411+0000 7f70e673a640 1 --2- 172.21.15.12:0/1636499029 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f70cc03e210 0x7f70cc0406d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:00.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.411+0000 7f70e673a640 1 --2- 172.21.15.12:0/1636499029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70e01353f0 0x7f70e0070110 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:00.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.411+0000 7f70e673a640 1 --2- 172.21.15.12:0/1636499029 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70e0134ad0 0x7f70e006fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:00.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.411+0000 7f70e673a640 1 --2- 172.21.15.12:0/1636499029 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70e00a2200 0x7f70e006f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:00.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.411+0000 7f70e673a640 1 -- 172.21.15.12:0/1636499029 >> 172.21.15.12:0/1636499029 conn(0x7f70e00a2f70 msgr2=0x7f70e013c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:00.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.411+0000 7f70e673a640 1 -- 172.21.15.12:0/1636499029 shutdown_connections 2024-09-11T01:45:00.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:00.411+0000 7f70e673a640 1 -- 172.21.15.12:0/1636499029 wait complete. 2024-09-11T01:45:00.415 INFO:tasks.rados.rados.0.smithi071.stdout:2383: done (3 left) 2024-09-11T01:45:00.416 INFO:tasks.rados.rados.0.smithi071.stdout:2384: delete oid 19 current snap is 264 2024-09-11T01:45:00.416 INFO:tasks.rados.rados.0.smithi071.stdout:2381: finishing rollback tid 0 to smithi07134777-17 2024-09-11T01:45:00.416 INFO:tasks.rados.rados.0.smithi071.stdout:2377: finishing copy_from racing read to smithi07134777-33 2024-09-11T01:45:00.416 INFO:tasks.rados.rados.0.smithi071.stdout:2381: finishing rollback tid 1 to smithi07134777-17 2024-09-11T01:45:00.416 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 1047 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-11T01:45:00.416 INFO:tasks.rados.rados.0.smithi071.stdout:2377: finishing copy_from to smithi07134777-33 2024-09-11T01:45:00.416 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 981 (ObjNum 688 snap 204 seq_num 688) dirty exists 2024-09-11T01:45:00.416 INFO:tasks.rados.rados.0.smithi071.stdout:2382: expect (ObjNum 816 snap 250 seq_num 816) 2024-09-11T01:45:00.429 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:45:00.442 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.440+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.442 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.440+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=674037017}) 2024-09-11T01:45:00.442 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.440+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2634087522}) 2024-09-11T01:45:00.442 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.440+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.442 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.440+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=674037017}) 2024-09-11T01:45:00.442 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.440+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1766190672}) 2024-09-11T01:45:00.443 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.442+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.443 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.442+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938081920}) 2024-09-11T01:45:00.443 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.442+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2239922262}) 2024-09-11T01:45:00.443 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.442+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.443 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.442+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938081920}) 2024-09-11T01:45:00.443 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.442+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2891015650}) 2024-09-11T01:45:00.444 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.443+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.444 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.443+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3422001039}) 2024-09-11T01:45:00.445 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.443+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3534659998}) 2024-09-11T01:45:00.445 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.443+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.445 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.443+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3422001039}) 2024-09-11T01:45:00.445 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.443+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3951653541}) 2024-09-11T01:45:00.446 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.444+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.446 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.444+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1140715020}) 2024-09-11T01:45:00.446 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.444+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4217061433}) 2024-09-11T01:45:00.446 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.444+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.446 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.444+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1140715020}) 2024-09-11T01:45:00.446 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.444+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1101240626}) 2024-09-11T01:45:00.447 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.445+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.447 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.445+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3110072748}) 2024-09-11T01:45:00.447 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.445+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1414411748}) 2024-09-11T01:45:00.447 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.445+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.447 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.445+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3110072748}) 2024-09-11T01:45:00.447 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.445+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2794198194}) 2024-09-11T01:45:00.448 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.446+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.448 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.446+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3418908115}) 2024-09-11T01:45:00.448 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.446+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3172135973}) 2024-09-11T01:45:00.448 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.446+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.448 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.446+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3418908115}) 2024-09-11T01:45:00.448 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.446+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3283505796}) 2024-09-11T01:45:00.449 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.447+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.449 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.447+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3916210644}) 2024-09-11T01:45:00.449 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.447+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2948303172}) 2024-09-11T01:45:00.449 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.447+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.449 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.447+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3916210644}) 2024-09-11T01:45:00.450 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.447+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3089039161}) 2024-09-11T01:45:00.450 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.448+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.450 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.448+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2130027323}) 2024-09-11T01:45:00.450 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.448+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=835187179}) 2024-09-11T01:45:00.450 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.448+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.450 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.448+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2130027323}) 2024-09-11T01:45:00.451 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.448+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=359029376}) 2024-09-11T01:45:00.451 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.449+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.451 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.449+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3651539425}) 2024-09-11T01:45:00.451 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.449+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3612237061}) 2024-09-11T01:45:00.451 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.449+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.451 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.449+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3651539425}) 2024-09-11T01:45:00.451 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.449+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1489082161}) 2024-09-11T01:45:00.452 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2088447 2024-09-11T01:45:00.453 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.450+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.453 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.450+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735849279}) 2024-09-11T01:45:00.453 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.450+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2458500201}) 2024-09-11T01:45:00.453 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.450+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.453 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.450+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735849279}) 2024-09-11T01:45:00.453 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.450+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=608191694}) 2024-09-11T01:45:00.453 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.451+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.453 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.451+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3768506832}) 2024-09-11T01:45:00.453 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.451+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2211646601}) 2024-09-11T01:45:00.454 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.451+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.454 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.451+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3768506832}) 2024-09-11T01:45:00.454 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.451+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=361697800}) 2024-09-11T01:45:00.454 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.452+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.454 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.452+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3414435035}) 2024-09-11T01:45:00.454 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.452+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3086283284}) 2024-09-11T01:45:00.454 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.452+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.455 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.452+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3414435035}) 2024-09-11T01:45:00.455 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.452+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=41696767}) 2024-09-11T01:45:00.455 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.453+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.455 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.453+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2932689710}) 2024-09-11T01:45:00.455 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.453+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=744600928}) 2024-09-11T01:45:00.455 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.453+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.455 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.453+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2932689710}) 2024-09-11T01:45:00.456 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.453+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4223054217}) 2024-09-11T01:45:00.456 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.454+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.456 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.454+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=809251382}) 2024-09-11T01:45:00.456 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.454+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2127050570}) 2024-09-11T01:45:00.456 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.454+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:00.456 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.454+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=809251382}) 2024-09-11T01:45:00.457 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:00.454+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2346741170}) 2024-09-11T01:45:00.485 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:45:00.630 INFO:tasks.rados.rados.0.smithi071.stdout:2377: done (3 left) 2024-09-11T01:45:00.630 INFO:tasks.rados.rados.0.smithi071.stdout:2381: done (2 left) 2024-09-11T01:45:00.630 INFO:tasks.rados.rados.0.smithi071.stdout:2382: done (1 left) 2024-09-11T01:45:00.630 INFO:tasks.rados.rados.0.smithi071.stdout:2384: done (0 left) 2024-09-11T01:45:00.630 INFO:tasks.rados.rados.0.smithi071.stdout:2385: snap_create 2024-09-11T01:45:01.172 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.171+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:01.173 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.171+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3501244198}) 2024-09-11T01:45:01.173 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.171+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3667323154}) 2024-09-11T01:45:01.173 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.171+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:01.173 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.171+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3501244198}) 2024-09-11T01:45:01.173 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.171+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=948940142}) 2024-09-11T01:45:01.182 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.180+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:01.182 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.180+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3936821556}) 2024-09-11T01:45:01.182 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.180+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=801958933}) 2024-09-11T01:45:01.182 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.180+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:01.183 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.180+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3936821556}) 2024-09-11T01:45:01.183 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.180+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=323631684}) 2024-09-11T01:45:01.184 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.181+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:01.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.181+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3507246776}) 2024-09-11T01:45:01.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.181+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3024336201}) 2024-09-11T01:45:01.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.181+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:01.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.181+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3507246776}) 2024-09-11T01:45:01.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.181+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2333060922}) 2024-09-11T01:45:01.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.183+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:01.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.183+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=815598972}) 2024-09-11T01:45:01.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.183+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1437544224}) 2024-09-11T01:45:01.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.183+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:01.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.183+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=815598972}) 2024-09-11T01:45:01.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.183+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4209123464}) 2024-09-11T01:45:01.189 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.184+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:01.189 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.184+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2331595952}) 2024-09-11T01:45:01.189 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.184+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2007562100}) 2024-09-11T01:45:01.189 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.184+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:01.189 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.184+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2331595952}) 2024-09-11T01:45:01.189 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.184+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1067084877}) 2024-09-11T01:45:01.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.185+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:01.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.185+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3949151960}) 2024-09-11T01:45:01.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.185+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=83799199}) 2024-09-11T01:45:01.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.185+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:01.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.185+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3949151960}) 2024-09-11T01:45:01.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.185+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1746225260}) 2024-09-11T01:45:01.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.186+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:01.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.186+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3913977528}) 2024-09-11T01:45:01.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.186+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=183265540}) 2024-09-11T01:45:01.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.186+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:01.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.186+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3913977528}) 2024-09-11T01:45:01.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.186+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2329684391}) 2024-09-11T01:45:01.254 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.252+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:01.254 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.252+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2479671227}) 2024-09-11T01:45:01.254 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.252+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1527706606}) 2024-09-11T01:45:01.254 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.252+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:01.254 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.252+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2479671227}) 2024-09-11T01:45:01.254 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.252+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=128515198}) 2024-09-11T01:45:01.325 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.323+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:01.325 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.323+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3384943678}) 2024-09-11T01:45:01.325 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.323+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1913548592}) 2024-09-11T01:45:01.326 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.323+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:01.326 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.323+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3384943678}) 2024-09-11T01:45:01.326 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.323+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1079447510}) 2024-09-11T01:45:01.332 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.330+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:01.332 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.330+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1515822232}) 2024-09-11T01:45:01.332 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.330+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1373629950}) 2024-09-11T01:45:01.332 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.330+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:01.332 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.330+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1515822232}) 2024-09-11T01:45:01.332 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.330+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2797268429}) 2024-09-11T01:45:01.412 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.410+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:01.412 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.410+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=590059433}) 2024-09-11T01:45:01.412 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.410+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2525587005}) 2024-09-11T01:45:01.412 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.410+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:01.412 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.410+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=590059433}) 2024-09-11T01:45:01.412 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.410+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2972328022}) 2024-09-11T01:45:01.415 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.413+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:01.415 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.413+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3281316823}) 2024-09-11T01:45:01.415 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.413+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3418087291}) 2024-09-11T01:45:01.415 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.413+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:01.415 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.413+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3281316823}) 2024-09-11T01:45:01.415 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.413+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2506583023}) 2024-09-11T01:45:01.419 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.418+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:01.419 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.418+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1231060418}) 2024-09-11T01:45:01.419 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.418+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1543296318}) 2024-09-11T01:45:01.420 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.418+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:01.420 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.418+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1231060418}) 2024-09-11T01:45:01.420 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:01.418+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3276225184}) 2024-09-11T01:45:01.423 INFO:tasks.rados.rados.0.smithi071.stdout:2385: done (0 left) 2024-09-11T01:45:01.423 INFO:tasks.rados.rados.0.smithi071.stdout:2386: write (excl) oid 28 current snap is 265 2024-09-11T01:45:01.423 INFO:tasks.rados.rados.0.smithi071.stdout:2386: seq_num 863 ranges {504441=487776,1467372=736289} 2024-09-11T01:45:01.428 INFO:tasks.rados.rados.0.smithi071.stdout:2386: writing smithi07134777-28 from 504441 to 992217 tid 1 2024-09-11T01:45:01.430 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:45:01.431 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:45:01.436 INFO:tasks.rados.rados.0.smithi071.stdout:2386: writing smithi07134777-28 from 1467372 to 2203661 tid 2 2024-09-11T01:45:01.436 INFO:tasks.rados.rados.0.smithi071.stdout:2387: rollback oid 43 current snap is 265 2024-09-11T01:45:01.436 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 43 to 252 2024-09-11T01:45:01.436 INFO:tasks.rados.rados.0.smithi071.stdout:2388: copy_from oid 20 from oid 41 current snap is 265 2024-09-11T01:45:01.436 INFO:tasks.rados.rados.0.smithi071.stdout:2389: snap_create 2024-09-11T01:45:01.443 INFO:tasks.rados.rados.0.smithi071.stdout:2386: finishing write tid 1 to smithi07134777-28 2024-09-11T01:45:01.445 INFO:tasks.rados.rados.0.smithi071.stdout:2387: finishing rollback tid 0 to smithi07134777-43 2024-09-11T01:45:01.445 INFO:tasks.rados.rados.0.smithi071.stdout:2386: finishing write tid 2 to smithi07134777-28 2024-09-11T01:45:01.445 INFO:tasks.rados.rados.0.smithi071.stdout:2387: finishing rollback tid 1 to smithi07134777-43 2024-09-11T01:45:01.445 INFO:tasks.rados.rados.0.smithi071.stdout:2386: finishing write tid 3 to smithi07134777-28 2024-09-11T01:45:01.445 INFO:tasks.rados.rados.0.smithi071.stdout:2387: finishing rollback tid 2 to smithi07134777-43 2024-09-11T01:45:01.445 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 1339 (ObjNum 732 snap 225 seq_num 732) dirty exists 2024-09-11T01:45:01.446 INFO:tasks.rados.rados.0.smithi071.stdout:2386: finishing write tid 4 to smithi07134777-28 2024-09-11T01:45:01.446 INFO:tasks.rados.rados.0.smithi071.stdout:2386: oid 28 updating version 0 to 1088 2024-09-11T01:45:01.446 INFO:tasks.rados.rados.0.smithi071.stdout:2386: oid 28 updating version 1088 to 1089 2024-09-11T01:45:01.446 INFO:tasks.rados.rados.0.smithi071.stdout:2386: oid 28 updating version 1089 to 1090 2024-09-11T01:45:01.446 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 1090 (ObjNum 863 snap 265 seq_num 863) dirty exists 2024-09-11T01:45:01.446 INFO:tasks.rados.rados.0.smithi071.stdout:2386: left oid 28 (ObjNum 863 snap 265 seq_num 863) 2024-09-11T01:45:01.453 INFO:tasks.rados.rados.0.smithi071.stdout:2388: finishing copy_from racing read to smithi07134777-20 2024-09-11T01:45:01.460 INFO:tasks.rados.rados.0.smithi071.stdout:2388: finishing copy_from to smithi07134777-20 2024-09-11T01:45:01.460 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 404 (ObjNum 832 snap 254 seq_num 832) dirty exists 2024-09-11T01:45:01.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.539+0000 7fb40eae2640 10 monclient: get_monmap_and_config 2024-09-11T01:45:01.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.539+0000 7fb40eae2640 10 monclient: build_initial_monmap 2024-09-11T01:45:01.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.539+0000 7fb40eae2640 10 monclient: monmap: 2024-09-11T01:45:01.541 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:01.541 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:01.541 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:45:01.540373+0000 2024-09-11T01:45:01.541 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:45:01.540373+0000 2024-09-11T01:45:01.541 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:01.542 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:01.542 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:45:01.542 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:45:01.542 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:45:01.542 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:45:01.542 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:45:01.542 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:01.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.540+0000 7fb40eae2640 1 Processor -- start 2024-09-11T01:45:01.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.540+0000 7fb40eae2640 1 -- start start 2024-09-11T01:45:01.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.540+0000 7fb40eae2640 10 monclient: init 2024-09-11T01:45:01.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.540+0000 7fb40eae2640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:01.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.540+0000 7fb40eae2640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-11T01:45:01.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.540+0000 7fb40eae2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb40809da80 addr v1:172.21.15.12:6789/0 2024-09-11T01:45:01.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.540+0000 7fb40eae2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb408136010 0x7fb40813ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:01.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.540+0000 7fb40eae2640 10 monclient(hunting): picked mon.noname-b con 0x7fb408136010 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:01.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.540+0000 7fb40eae2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb40813b390 addr v1:172.21.15.71:6789/0 2024-09-11T01:45:01.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.540+0000 7fb40eae2640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb40806d5b0 con 0x7fb40809da80 2024-09-11T01:45:01.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.540+0000 7fb40eae2640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb40806d890 con 0x7fb40813b390 2024-09-11T01:45:01.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.540+0000 7fb40eae2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:01.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.540+0000 7fb40eae2640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb4080a1590 con 0x7fb408136010 2024-09-11T01:45:01.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.540+0000 7fb40eae2640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:01.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.540+0000 7fb40eae2640 10 monclient(hunting): authenticate will time out at 1707.982961s 2024-09-11T01:45:01.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.541+0000 7fb40c87e640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb40809da80 0x7fb40809de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37750/0 (socket says 172.21.15.12:37750) 2024-09-11T01:45:01.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.541+0000 7fb407fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb408136010 0x7fb40813ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:01.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.541+0000 7fb40c87e640 1 -- 172.21.15.12:0/2252116635 learned_addr learned my addr 172.21.15.12:0/2252116635 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:45:01.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.541+0000 7fb407fff640 10 monclient(hunting): get_auth_request con 0x7fb408136010 auth_method 0 2024-09-11T01:45:01.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.541+0000 7fb407fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:01.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.541+0000 7fb407fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:01.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.541+0000 7fb407fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:01.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.541+0000 7fb407fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:01.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.541+0000 7fb407fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:01.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.541+0000 7fb407fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:01.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.541+0000 7fb407fff640 10 monclient(hunting): handle_auth_done global_id 164186 payload 274 2024-09-11T01:45:01.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.541+0000 7fb407fff640 1 -- 172.21.15.12:0/2252116635 >> v1:172.21.15.71:6789/0 conn(0x7fb40813b390 legacy=0x7fb40813d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:01.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.541+0000 7fb407fff640 1 -- 172.21.15.12:0/2252116635 >> v1:172.21.15.12:6789/0 conn(0x7fb40809da80 legacy=0x7fb40809de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:01.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.541+0000 7fb407fff640 10 monclient: _finish_hunting 0 2024-09-11T01:45:01.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.541+0000 7fb407fff640 1 monclient: found mon.noname-b 2024-09-11T01:45:01.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.541+0000 7fb407fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:01.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.541+0000 7fb407fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:45:01.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.541+0000 7fb407fff640 1 -- 172.21.15.12:0/2252116635 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb40806d2d0 con 0x7fb408136010 2024-09-11T01:45:01.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.544+0000 7fb407fff640 1 -- 172.21.15.12:0/2252116635 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb408136010 msgr2=0x7fb40813ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:01.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.544+0000 7fb407fff640 1 --2- 172.21.15.12:0/2252116635 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb408136010 0x7fb40813ae50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb3f40097d0 tx=0x7fb3f4029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9732ce1234256f7c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:01.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.544+0000 7fb4077fe640 1 -- 172.21.15.12:0/2252116635 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4037070 con 0x7fb408136010 2024-09-11T01:45:01.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.544+0000 7fb4077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:01.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.544+0000 7fb4077fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:45:01.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.544+0000 7fb4077fe640 10 monclient: dump: 2024-09-11T01:45:01.553 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:01.553 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:01.553 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:01.553 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:01.553 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:01.553 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:01.553 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:01.554 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:01.554 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:01.554 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:01.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.544+0000 7fb4077fe640 10 monclient: _finish_auth 0 2024-09-11T01:45:01.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.544+0000 7fb4077fe640 10 monclient: _check_auth_tickets 2024-09-11T01:45:01.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.544+0000 7fb4077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:01.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.544+0000 7fb40eae2640 5 monclient: authenticate success, global_id 164186 2024-09-11T01:45:01.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.544+0000 7fb4077fe640 1 -- 172.21.15.12:0/2252116635 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3f4032440 con 0x7fb408136010 2024-09-11T01:45:01.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.544+0000 7fb4077fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:01.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.544+0000 7fb4077fe640 1 -- 172.21.15.12:0/2252116635 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4032810 con 0x7fb408136010 2024-09-11T01:45:01.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.544+0000 7fb40eae2640 10 monclient: get_monmap_and_config success 2024-09-11T01:45:01.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.544+0000 7fb40eae2640 4 set_mon_vals no callback set 2024-09-11T01:45:01.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.544+0000 7fb4077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:01.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.544+0000 7fb4077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:45:01.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.544+0000 7fb4077fe640 10 monclient: dump: 2024-09-11T01:45:01.555 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:01.555 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:01.555 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:01.555 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:01.555 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:01.555 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:01.555 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:01.555 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:01.555 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:01.555 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:01.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.545+0000 7fb40eae2640 10 monclient: shutdown 2024-09-11T01:45:01.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.545+0000 7fb40eae2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:01.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.545+0000 7fb40eae2640 1 -- 172.21.15.12:0/2252116635 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb408136010 msgr2=0x7fb40813ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:01.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.545+0000 7fb40eae2640 1 --2- 172.21.15.12:0/2252116635 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb408136010 0x7fb40813ae50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb3f40097d0 tx=0x7fb3f4029fb0 comp rx=0 tx=0).stop 2024-09-11T01:45:01.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.545+0000 7fb40eae2640 1 -- 172.21.15.12:0/2252116635 shutdown_connections 2024-09-11T01:45:01.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.545+0000 7fb40eae2640 1 --2- 172.21.15.12:0/2252116635 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb408136010 0x7fb40813ae50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:01.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.545+0000 7fb40eae2640 1 -- 172.21.15.12:0/2252116635 >> 172.21.15.12:0/2252116635 conn(0x7fb4080a3ea0 msgr2=0x7fb4080a42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:01.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.545+0000 7fb40eae2640 1 -- 172.21.15.12:0/2252116635 shutdown_connections 2024-09-11T01:45:01.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.545+0000 7fb40eae2640 1 -- 172.21.15.12:0/2252116635 wait complete. 2024-09-11T01:45:01.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.545+0000 7fb40eae2640 10 monclient: build_initial_monmap 2024-09-11T01:45:01.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.545+0000 7fb40eae2640 10 monclient: monmap: 2024-09-11T01:45:01.556 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:01.556 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:45:01.556 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:45:01.556 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:45:01.557 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:01.557 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:01.557 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:45:01.557 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:45:01.557 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:45:01.557 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:01.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40eae2640 1 Processor -- start 2024-09-11T01:45:01.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40eae2640 1 -- start start 2024-09-11T01:45:01.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40eae2640 10 monclient: init 2024-09-11T01:45:01.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40eae2640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:01.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40eae2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:45:01.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40eae2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb40809da80 0x7fb408136c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:01.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40eae2640 10 monclient(hunting): picked mon.noname-c con 0x7fb40809da80 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40eae2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb408136010 0x7fb408137160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40eae2640 10 monclient(hunting): picked mon.noname-b con 0x7fb408136010 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:45:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40eae2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb40813b390 0x7fb4081376a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40eae2640 10 monclient(hunting): picked mon.noname-a con 0x7fb40813b390 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:45:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40eae2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40eae2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb4080a0ea0 con 0x7fb40813b390 2024-09-11T01:45:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40eae2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40eae2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb4080a0d20 con 0x7fb408136010 2024-09-11T01:45:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40eae2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40eae2640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb40813eea0 con 0x7fb40809da80 2024-09-11T01:45:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40eae2640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40eae2640 10 monclient(hunting): authenticate will time out at 1707.988722s 2024-09-11T01:45:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40c87e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb40809da80 0x7fb408136c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40c87e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb40809da80 0x7fb408136c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:54146/0 (socket says 172.21.15.12:54146) 2024-09-11T01:45:01.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40c87e640 1 -- 172.21.15.12:0/1438267941 learned_addr learned my addr 172.21.15.12:0/1438267941 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:01.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40c87e640 10 monclient(hunting): get_auth_request con 0x7fb40809da80 auth_method 0 2024-09-11T01:45:01.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40c87e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:01.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40d07f640 1 --2- 172.21.15.12:0/1438267941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb40813b390 0x7fb4081376a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:01.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40c87e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:01.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40c87e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:01.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40d07f640 10 monclient(hunting): get_auth_request con 0x7fb40813b390 auth_method 0 2024-09-11T01:45:01.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40d07f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:01.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40d07f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:01.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40d07f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:01.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40c87e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:01.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40c87e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:01.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40c87e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:01.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40d07f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:01.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40d07f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:01.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.546+0000 7fb40d07f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:01.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.547+0000 7fb407fff640 1 --2- 172.21.15.12:0/1438267941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb408136010 0x7fb408137160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:01.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.547+0000 7fb407fff640 10 monclient(hunting): get_auth_request con 0x7fb408136010 auth_method 0 2024-09-11T01:45:01.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.547+0000 7fb407fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:01.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.547+0000 7fb407fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:01.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.547+0000 7fb407fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:01.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.547+0000 7fb40c87e640 10 monclient(hunting): handle_auth_done global_id 164189 payload 824 2024-09-11T01:45:01.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.547+0000 7fb40c87e640 1 -- 172.21.15.12:0/1438267941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb408136010 msgr2=0x7fb408137160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:01.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.547+0000 7fb40c87e640 1 --2- 172.21.15.12:0/1438267941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb408136010 0x7fb408137160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:01.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.547+0000 7fb40c87e640 1 -- 172.21.15.12:0/1438267941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb40813b390 msgr2=0x7fb4081376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:01.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.547+0000 7fb40c87e640 1 --2- 172.21.15.12:0/1438267941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb40813b390 0x7fb4081376a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:01.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.547+0000 7fb40c87e640 10 monclient: _finish_hunting 0 2024-09-11T01:45:01.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.547+0000 7fb40c87e640 1 monclient: found mon.noname-c 2024-09-11T01:45:01.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.547+0000 7fb40c87e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:01.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.547+0000 7fb40c87e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:45:01.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.547+0000 7fb40c87e640 1 -- 172.21.15.12:0/1438267941 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb40813a250 con 0x7fb40809da80 2024-09-11T01:45:01.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.547+0000 7fb40d07f640 1 --2- 172.21.15.12:0/1438267941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb40813b390 0x7fb4081376a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:45:01.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.549+0000 7fb40c87e640 1 -- 172.21.15.12:0/1438267941 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb40809da80 msgr2=0x7fb408136c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:01.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.549+0000 7fb40c87e640 1 --2- 172.21.15.12:0/1438267941 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb40809da80 0x7fb408136c20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00c490 tx=0x7fb3fc00c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:01.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.549+0000 7fb4057fa640 1 -- 172.21.15.12:0/1438267941 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc007070 con 0x7fb40809da80 2024-09-11T01:45:01.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.549+0000 7fb4057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:01.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.549+0000 7fb4057fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:45:01.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.549+0000 7fb4057fa640 10 monclient: dump: 2024-09-11T01:45:01.561 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:01.561 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:01.562 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:01.562 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:01.562 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:01.562 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:01.562 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:01.562 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:01.562 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:01.562 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:01.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.549+0000 7fb4057fa640 10 monclient: _finish_auth 0 2024-09-11T01:45:01.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.549+0000 7fb4057fa640 10 monclient: _check_auth_tickets 2024-09-11T01:45:01.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.549+0000 7fb4057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:01.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.549+0000 7fb40eae2640 5 monclient: authenticate success, global_id 164189 2024-09-11T01:45:01.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.549+0000 7fb4057fa640 1 -- 172.21.15.12:0/1438267941 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3fc007230 con 0x7fb40809da80 2024-09-11T01:45:01.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.549+0000 7fb40eae2640 10 monclient: _renew_subs 2024-09-11T01:45:01.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.549+0000 7fb40eae2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:45:01.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.549+0000 7fb40eae2640 1 -- 172.21.15.12:0/1438267941 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4081395b0 con 0x7fb40809da80 2024-09-11T01:45:01.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.549+0000 7fb4057fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:01.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.549+0000 7fb4057fa640 1 -- 172.21.15.12:0/1438267941 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc007620 con 0x7fb40809da80 2024-09-11T01:45:01.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.549+0000 7fb4057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:01.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.549+0000 7fb4057fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:45:01.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.549+0000 7fb4057fa640 10 monclient: dump: 2024-09-11T01:45:01.563 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:01.563 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:01.563 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:01.563 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:01.563 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:01.564 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:01.564 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:01.564 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:01.564 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:01.564 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:01.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.549+0000 7fb406ffd640 4 set_mon_vals no callback set 2024-09-11T01:45:01.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.549+0000 7fb40eae2640 10 monclient: _renew_subs 2024-09-11T01:45:01.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.549+0000 7fb40eae2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:45:01.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.549+0000 7fb40eae2640 1 -- 172.21.15.12:0/1438267941 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb40813e910 con 0x7fb40809da80 2024-09-11T01:45:01.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.550+0000 7fb40eae2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:01.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.550+0000 7fb40eae2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:01.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.550+0000 7fb40eae2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:45:01.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.550+0000 7fb40eae2640 1 -- 172.21.15.12:0/1438267941 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb40809de60 con 0x7fb40809da80 2024-09-11T01:45:01.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.550+0000 7fb4057fa640 1 -- 172.21.15.12:0/1438267941 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fb3fc01b400 con 0x7fb40809da80 2024-09-11T01:45:01.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.551+0000 7fb4057fa640 1 --2- 172.21.15.12:0/1438267941 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb3ec03e240 0x7fb3ec040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:01.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.551+0000 7fb4057fa640 1 -- 172.21.15.12:0/1438267941 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(535..535 src has 317..535) ==== 4771+0+0 (secure 0 0 0) 0x7fb3fc0529e0 con 0x7fb40809da80 2024-09-11T01:45:01.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.551+0000 7fb407fff640 1 --2- 172.21.15.12:0/1438267941 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb3ec03e240 0x7fb3ec040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:01.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.551+0000 7fb407fff640 10 monclient: get_auth_request con 0x7fb3ec03e240 auth_method 0 2024-09-11T01:45:01.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.553+0000 7fb407fff640 1 --2- 172.21.15.12:0/1438267941 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb3ec03e240 0x7fb3ec040700 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4034000 tx=0x7fb3f402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:01.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.555+0000 7fb4057fa640 1 -- 172.21.15.12:0/1438267941 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb40813a250 con 0x7fb40809da80 2024-09-11T01:45:01.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.555+0000 7fb4057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:01.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.555+0000 7fb4057fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:45:01.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.814+0000 7fb40eae2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:45:01.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.814+0000 7fb40eae2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:45:01.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.814+0000 7fb40eae2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:45:01.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.814+0000 7fb40eae2640 1 -- 172.21.15.12:0/1438267941 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb408139230 con 0x7fb40809da80 2024-09-11T01:45:01.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.815+0000 7fb4057fa640 1 -- 172.21.15.12:0/1438267941 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fb3fc012400 con 0x7fb40809da80 2024-09-11T01:45:01.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.815+0000 7fb4057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:45:01.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.815+0000 7fb4057fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:45:01.817 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":148,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:45:01.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.817+0000 7fb40eae2640 1 -- 172.21.15.12:0/1438267941 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb3ec03e240 msgr2=0x7fb3ec040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:01.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.817+0000 7fb40eae2640 1 --2- 172.21.15.12:0/1438267941 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb3ec03e240 0x7fb3ec040700 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4034000 tx=0x7fb3f402e000 comp rx=0 tx=0).stop 2024-09-11T01:45:01.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.817+0000 7fb40eae2640 10 monclient: shutdown 2024-09-11T01:45:01.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.817+0000 7fb40eae2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:01.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.817+0000 7fb40eae2640 1 -- 172.21.15.12:0/1438267941 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb40809da80 msgr2=0x7fb408136c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:01.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.817+0000 7fb40eae2640 1 --2- 172.21.15.12:0/1438267941 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb40809da80 0x7fb408136c20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00c490 tx=0x7fb3fc00c950 comp rx=0 tx=0).stop 2024-09-11T01:45:01.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.817+0000 7fb40eae2640 1 -- 172.21.15.12:0/1438267941 shutdown_connections 2024-09-11T01:45:01.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.817+0000 7fb40eae2640 1 --2- 172.21.15.12:0/1438267941 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb3ec03e240 0x7fb3ec040700 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:01.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.817+0000 7fb40eae2640 1 --2- 172.21.15.12:0/1438267941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb40813b390 0x7fb4081376a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:01.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.817+0000 7fb40eae2640 1 --2- 172.21.15.12:0/1438267941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb408136010 0x7fb408137160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:01.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.817+0000 7fb40eae2640 1 --2- 172.21.15.12:0/1438267941 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb40809da80 0x7fb408136c20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:01.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.817+0000 7fb40eae2640 1 -- 172.21.15.12:0/1438267941 >> 172.21.15.12:0/1438267941 conn(0x7fb4080a3ea0 msgr2=0x7fb4080a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:01.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.817+0000 7fb40eae2640 1 -- 172.21.15.12:0/1438267941 shutdown_connections 2024-09-11T01:45:01.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.817+0000 7fb40eae2640 1 -- 172.21.15.12:0/1438267941 wait complete. 2024-09-11T01:45:01.829 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:45:01.829 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:45:01.829 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:45:01.829 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:45:01.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.929+0000 7fe5d3a7f640 10 monclient: get_monmap_and_config 2024-09-11T01:45:01.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.929+0000 7fe5d3a7f640 10 monclient: build_initial_monmap 2024-09-11T01:45:01.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.929+0000 7fe5d3a7f640 10 monclient: monmap: 2024-09-11T01:45:01.931 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:01.931 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:01.931 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:45:01.930336+0000 2024-09-11T01:45:01.931 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:45:01.930336+0000 2024-09-11T01:45:01.931 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:01.931 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:01.931 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:45:01.932 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:45:01.932 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:45:01.932 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:45:01.932 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:45:01.932 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:01.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.930+0000 7fe5d3a7f640 1 Processor -- start 2024-09-11T01:45:01.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.930+0000 7fe5d3a7f640 1 -- start start 2024-09-11T01:45:01.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.930+0000 7fe5d3a7f640 10 monclient: init 2024-09-11T01:45:01.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.930+0000 7fe5d3a7f640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:01.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.930+0000 7fe5d3a7f640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-11T01:45:01.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.930+0000 7fe5d3a7f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe5cc131710 addr v1:172.21.15.12:6789/0 2024-09-11T01:45:01.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.930+0000 7fe5d3a7f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe5cc0a0d70 0x7fe5cc13adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:01.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.930+0000 7fe5d3a7f640 10 monclient(hunting): picked mon.noname-a con 0x7fe5cc0a0d70 addr v2:172.21.15.12:3300/0 2024-09-11T01:45:01.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.930+0000 7fe5d3a7f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe5cc13b2f0 addr v1:172.21.15.71:6789/0 2024-09-11T01:45:01.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.930+0000 7fe5d3a7f640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5cc074a70 con 0x7fe5cc131710 2024-09-11T01:45:01.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.930+0000 7fe5d3a7f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5cc073af0 con 0x7fe5cc13b2f0 2024-09-11T01:45:01.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.930+0000 7fe5d3a7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:01.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.930+0000 7fe5d3a7f640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fe5cc073d90 con 0x7fe5cc0a0d70 2024-09-11T01:45:01.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.930+0000 7fe5d3a7f640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:01.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.930+0000 7fe5d3a7f640 10 monclient(hunting): authenticate will time out at 1708.372914s 2024-09-11T01:45:01.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.931+0000 7fe5d101a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe5cc0a0d70 0x7fe5cc13adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:01.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.931+0000 7fe5d181b640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe5cc131710 0x7fe5cc131af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37756/0 (socket says 172.21.15.12:37756) 2024-09-11T01:45:01.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.931+0000 7fe5d101a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe5cc0a0d70 0x7fe5cc13adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44774/0 (socket says 172.21.15.12:44774) 2024-09-11T01:45:01.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.931+0000 7fe5d181b640 1 -- 172.21.15.12:0/1245104113 learned_addr learned my addr 172.21.15.12:0/1245104113 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:45:01.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.931+0000 7fe5d101a640 10 monclient(hunting): get_auth_request con 0x7fe5cc0a0d70 auth_method 0 2024-09-11T01:45:01.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.931+0000 7fe5d101a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:01.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.931+0000 7fe5d101a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:01.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.931+0000 7fe5d101a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:01.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.931+0000 7fe5d101a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:01.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.931+0000 7fe5d101a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:01.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.931+0000 7fe5d101a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:01.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.931+0000 7fe5d101a640 10 monclient(hunting): handle_auth_done global_id 174129 payload 274 2024-09-11T01:45:01.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.931+0000 7fe5d101a640 1 -- 172.21.15.12:0/1245104113 >> v1:172.21.15.71:6789/0 conn(0x7fe5cc13b2f0 legacy=0x7fe5cc13d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:01.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.931+0000 7fe5d101a640 1 -- 172.21.15.12:0/1245104113 >> v1:172.21.15.12:6789/0 conn(0x7fe5cc131710 legacy=0x7fe5cc131af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:01.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.931+0000 7fe5d101a640 10 monclient: _finish_hunting 0 2024-09-11T01:45:01.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.931+0000 7fe5d101a640 1 monclient: found mon.noname-a 2024-09-11T01:45:01.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.931+0000 7fe5d101a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:01.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.931+0000 7fe5d101a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:45:01.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.931+0000 7fe5d101a640 1 -- 172.21.15.12:0/1245104113 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5cc074070 con 0x7fe5cc0a0d70 2024-09-11T01:45:01.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d101a640 1 -- 172.21.15.12:0/1245104113 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc0a0d70 msgr2=0x7fe5cc13adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:01.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d101a640 1 --2- 172.21.15.12:0/1245104113 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc0a0d70 0x7fe5cc13adb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc0097a0 tx=0x7fe5bc02a0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b6b207ade970464 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:01.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d0819640 1 -- 172.21.15.12:0/1245104113 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc037070 con 0x7fe5cc0a0d70 2024-09-11T01:45:01.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d0819640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:01.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d0819640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:45:01.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d0819640 10 monclient: dump: 2024-09-11T01:45:01.942 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:01.943 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:01.943 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:01.943 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:01.943 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:01.943 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:01.943 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:01.943 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:01.943 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:01.943 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:01.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d0819640 10 monclient: _finish_auth 0 2024-09-11T01:45:01.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d0819640 10 monclient: _check_auth_tickets 2024-09-11T01:45:01.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d0819640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:01.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d3a7f640 5 monclient: authenticate success, global_id 174129 2024-09-11T01:45:01.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d3a7f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:45:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d0819640 1 -- 172.21.15.12:0/1245104113 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5bc032640 con 0x7fe5cc0a0d70 2024-09-11T01:45:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d0819640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d0819640 1 -- 172.21.15.12:0/1245104113 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc032a10 con 0x7fe5cc0a0d70 2024-09-11T01:45:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d0819640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d0819640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d0819640 10 monclient: dump: 2024-09-11T01:45:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:01.944 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:01.945 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:01.945 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:01.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d3a7f640 10 monclient: get_monmap_and_config success 2024-09-11T01:45:01.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d3a7f640 4 set_mon_vals no callback set 2024-09-11T01:45:01.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d3a7f640 10 monclient: shutdown 2024-09-11T01:45:01.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d3a7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:01.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d3a7f640 1 -- 172.21.15.12:0/1245104113 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc0a0d70 msgr2=0x7fe5cc13adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:01.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d3a7f640 1 --2- 172.21.15.12:0/1245104113 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc0a0d70 0x7fe5cc13adb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc0097a0 tx=0x7fe5bc02a0e0 comp rx=0 tx=0).stop 2024-09-11T01:45:01.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d3a7f640 1 -- 172.21.15.12:0/1245104113 shutdown_connections 2024-09-11T01:45:01.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d3a7f640 1 --2- 172.21.15.12:0/1245104113 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc0a0d70 0x7fe5cc13adb0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:01.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d3a7f640 1 -- 172.21.15.12:0/1245104113 >> 172.21.15.12:0/1245104113 conn(0x7fe5cc0a36c0 msgr2=0x7fe5cc0a3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:01.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d3a7f640 1 -- 172.21.15.12:0/1245104113 shutdown_connections 2024-09-11T01:45:01.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.934+0000 7fe5d3a7f640 1 -- 172.21.15.12:0/1245104113 wait complete. 2024-09-11T01:45:01.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.935+0000 7fe5d3a7f640 10 monclient: build_initial_monmap 2024-09-11T01:45:01.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.935+0000 7fe5d3a7f640 10 monclient: monmap: 2024-09-11T01:45:01.946 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:01.946 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:45:01.946 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:45:01.946 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:45:01.946 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:01.946 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:01.946 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:45:01.946 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:45:01.947 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:45:01.947 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:01.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.935+0000 7fe5d3a7f640 1 Processor -- start 2024-09-11T01:45:01.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.935+0000 7fe5d3a7f640 1 -- start start 2024-09-11T01:45:01.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.935+0000 7fe5d3a7f640 10 monclient: init 2024-09-11T01:45:01.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d3a7f640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:01.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d3a7f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:45:01.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d3a7f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5cc0a0d70 0x7fe5cc139c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:01.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d3a7f640 10 monclient(hunting): picked mon.noname-c con 0x7fe5cc0a0d70 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:01.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d3a7f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe5cc131710 0x7fe5cc130700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:01.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d3a7f640 10 monclient(hunting): picked mon.noname-b con 0x7fe5cc131710 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:45:01.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d3a7f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc13b2f0 0x7fe5cc130c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:01.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d3a7f640 10 monclient(hunting): picked mon.noname-a con 0x7fe5cc13b2f0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:45:01.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d3a7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:01.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d3a7f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe5cc073350 con 0x7fe5cc13b2f0 2024-09-11T01:45:01.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d3a7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:01.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d3a7f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe5cc0731d0 con 0x7fe5cc131710 2024-09-11T01:45:01.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d3a7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:01.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d3a7f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe5cc13ee00 con 0x7fe5cc0a0d70 2024-09-11T01:45:01.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d3a7f640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:01.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d3a7f640 10 monclient(hunting): authenticate will time out at 1708.378253s 2024-09-11T01:45:01.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d181b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5cc0a0d70 0x7fe5cc139c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:01.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d201c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc13b2f0 0x7fe5cc130c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:01.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d201c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc13b2f0 0x7fe5cc130c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44780/0 (socket says 172.21.15.12:44780) 2024-09-11T01:45:01.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d181b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5cc0a0d70 0x7fe5cc139c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:54156/0 (socket says 172.21.15.12:54156) 2024-09-11T01:45:01.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d201c640 1 -- 172.21.15.12:0/1437263909 learned_addr learned my addr 172.21.15.12:0/1437263909 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:01.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d201c640 10 monclient(hunting): get_auth_request con 0x7fe5cc13b2f0 auth_method 0 2024-09-11T01:45:01.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d201c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:01.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d201c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:01.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d201c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:01.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d181b640 10 monclient(hunting): get_auth_request con 0x7fe5cc0a0d70 auth_method 0 2024-09-11T01:45:01.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d181b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:01.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d181b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:01.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d181b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:01.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d101a640 1 --2- 172.21.15.12:0/1437263909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe5cc131710 0x7fe5cc130700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:01.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d201c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:01.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d201c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:01.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d201c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:01.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d181b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:01.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d181b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:01.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d181b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:01.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d101a640 10 monclient(hunting): get_auth_request con 0x7fe5cc131710 auth_method 0 2024-09-11T01:45:01.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d101a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:01.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d101a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:01.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d101a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:01.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d101a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:01.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d101a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:01.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d101a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:01.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d201c640 10 monclient(hunting): handle_auth_done global_id 174132 payload 824 2024-09-11T01:45:01.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d201c640 1 -- 172.21.15.12:0/1437263909 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5cc0a0d70 msgr2=0x7fe5cc139c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:01.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d201c640 1 --2- 172.21.15.12:0/1437263909 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5cc0a0d70 0x7fe5cc139c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:01.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d201c640 1 -- 172.21.15.12:0/1437263909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe5cc131710 msgr2=0x7fe5cc130700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:01.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d201c640 1 --2- 172.21.15.12:0/1437263909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe5cc131710 0x7fe5cc130700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:01.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d201c640 10 monclient: _finish_hunting 0 2024-09-11T01:45:01.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d201c640 1 monclient: found mon.noname-a 2024-09-11T01:45:01.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d201c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:01.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d201c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:45:01.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.936+0000 7fe5d201c640 1 -- 172.21.15.12:0/1437263909 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5cc139500 con 0x7fe5cc13b2f0 2024-09-11T01:45:01.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.938+0000 7fe5d201c640 1 -- 172.21.15.12:0/1437263909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc13b2f0 msgr2=0x7fe5cc130c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:01.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.938+0000 7fe5d201c640 1 --2- 172.21.15.12:0/1437263909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc13b2f0 0x7fe5cc130c40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe5c400c780 tx=0x7fe5c400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:01.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.939+0000 7fe5c27fc640 1 -- 172.21.15.12:0/1437263909 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c400ac60 con 0x7fe5cc13b2f0 2024-09-11T01:45:01.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.939+0000 7fe5c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:01.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.939+0000 7fe5c27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:45:01.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.939+0000 7fe5c27fc640 10 monclient: dump: 2024-09-11T01:45:01.952 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:01.952 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:01.952 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:01.952 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:01.953 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:01.953 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:01.953 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:01.953 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:01.953 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:01.953 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:01.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.939+0000 7fe5c27fc640 10 monclient: _finish_auth 0 2024-09-11T01:45:01.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.939+0000 7fe5c27fc640 10 monclient: _check_auth_tickets 2024-09-11T01:45:01.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.939+0000 7fe5c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:01.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.939+0000 7fe5d3a7f640 5 monclient: authenticate success, global_id 174132 2024-09-11T01:45:01.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.939+0000 7fe5c27fc640 1 -- 172.21.15.12:0/1437263909 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5c400ae00 con 0x7fe5cc13b2f0 2024-09-11T01:45:01.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.939+0000 7fe5d3a7f640 10 monclient: _renew_subs 2024-09-11T01:45:01.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.939+0000 7fe5d3a7f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:01.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.939+0000 7fe5d3a7f640 1 -- 172.21.15.12:0/1437263909 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5cc13ad00 con 0x7fe5cc13b2f0 2024-09-11T01:45:01.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.939+0000 7fe5c27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:01.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.939+0000 7fe5c27fc640 1 -- 172.21.15.12:0/1437263909 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c4004ba0 con 0x7fe5cc13b2f0 2024-09-11T01:45:01.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.939+0000 7fe5c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:01.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.939+0000 7fe5c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:01.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.939+0000 7fe5c27fc640 10 monclient: dump: 2024-09-11T01:45:01.954 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:01.954 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:01.954 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:01.954 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:01.954 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:01.955 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:01.955 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:01.955 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:01.955 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:01.955 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:01.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.939+0000 7fe5d3a7f640 10 monclient: _renew_subs 2024-09-11T01:45:01.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.939+0000 7fe5d3a7f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:01.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.939+0000 7fe5c37fe640 4 set_mon_vals no callback set 2024-09-11T01:45:01.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.939+0000 7fe5d3a7f640 1 -- 172.21.15.12:0/1437263909 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5cc131180 con 0x7fe5cc13b2f0 2024-09-11T01:45:01.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.939+0000 7fe5d3a7f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.939+0000 7fe5d3a7f640 1 --2- 172.21.15.12:0/1437263909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc1cd670 0x7fe5cc02d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.939+0000 7fe5d3a7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.939+0000 7fe5d3a7f640 1 -- 172.21.15.12:0/1437263909 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe5cc075000 con 0x7fe5cc1cd670 2024-09-11T01:45:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.940+0000 7fe5d181b640 1 --2- 172.21.15.12:0/1437263909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc1cd670 0x7fe5cc02d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.940+0000 7fe5d181b640 10 monclient: get_auth_request con 0x7fe5cc1cd670 auth_method 0 2024-09-11T01:45:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.940+0000 7fe5d181b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.940+0000 7fe5d181b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.940+0000 7fe5d181b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.940+0000 7fe5c27fc640 1 -- 172.21.15.12:0/1437263909 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fe5c4005030 con 0x7fe5cc13b2f0 2024-09-11T01:45:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.940+0000 7fe5d181b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:01.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.940+0000 7fe5d181b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:01.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.940+0000 7fe5d181b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:01.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.940+0000 7fe5d181b640 10 monclient(hunting): handle_auth_done global_id 174135 payload 824 2024-09-11T01:45:01.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.940+0000 7fe5d181b640 1 -- 172.21.15.12:0/1437263909 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe5cc0a1a10 con 0x7fe5cc1cd670 2024-09-11T01:45:01.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.940+0000 7fe5c27fc640 1 --2- 172.21.15.12:0/1437263909 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe5b003e260 0x7fe5b0040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:01.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.940+0000 7fe5c27fc640 1 -- 172.21.15.12:0/1437263909 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(535..535 src has 317..535) ==== 4771+0+0 (secure 0 0 0) 0x7fe5c4052f50 con 0x7fe5cc13b2f0 2024-09-11T01:45:01.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.940+0000 7fe5d101a640 1 --2- 172.21.15.12:0/1437263909 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe5b003e260 0x7fe5b0040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:01.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.940+0000 7fe5d101a640 10 monclient: get_auth_request con 0x7fe5b003e260 auth_method 0 2024-09-11T01:45:01.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.942+0000 7fe5d181b640 1 -- 172.21.15.12:0/1437263909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc1cd670 msgr2=0x7fe5cc02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:01.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.942+0000 7fe5d181b640 1 --2- 172.21.15.12:0/1437263909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc1cd670 0x7fe5cc02d440 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe5c800ea70 tx=0x7fe5c800ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:01.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.942+0000 7fe5c27fc640 1 -- 172.21.15.12:0/1437263909 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c801a070 con 0x7fe5cc1cd670 2024-09-11T01:45:01.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.942+0000 7fe5c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:01.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.942+0000 7fe5c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:01.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.942+0000 7fe5c27fc640 10 monclient: dump: 2024-09-11T01:45:01.960 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:01.960 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:01.960 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:01.960 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:01.961 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:01.961 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:01.961 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:01.961 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:01.961 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:01.961 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:01.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.942+0000 7fe5c27fc640 1 -- 172.21.15.12:0/1437263909 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe5c800cc50 con 0x7fe5cc1cd670 2024-09-11T01:45:01.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.942+0000 7fe5c27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:01.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.942+0000 7fe5c27fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:45:01.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.942+0000 7fe5c27fc640 1 -- 172.21.15.12:0/1437263909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc1cd670 msgr2=0x7fe5cc02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:01.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.942+0000 7fe5c27fc640 1 --2- 172.21.15.12:0/1437263909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc1cd670 0x7fe5cc02d440 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe5c800ea70 tx=0x7fe5c800ef30 comp rx=0 tx=0).stop 2024-09-11T01:45:01.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.942+0000 7fe5c27fc640 1 -- 172.21.15.12:0/1437263909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc1cd670 msgr2=0x7fe5cc02d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:01.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.942+0000 7fe5c27fc640 1 --2- 172.21.15.12:0/1437263909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc1cd670 0x7fe5cc02d440 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe5c800ea70 tx=0x7fe5c800ef30 comp rx=0 tx=0).stop 2024-09-11T01:45:01.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.943+0000 7fe5d101a640 1 --2- 172.21.15.12:0/1437263909 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe5b003e260 0x7fe5b0040720 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc06ef60 tx=0x7fe5bc02e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:01.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.949+0000 7fe5d3a7f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:45:01.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.949+0000 7fe5d3a7f640 1 --2- 172.21.15.12:0/1437263909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc1cda50 0x7fe5cc02d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:01.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.949+0000 7fe5d3a7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:01.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.949+0000 7fe5d3a7f640 1 -- 172.21.15.12:0/1437263909 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe5cc131af0 con 0x7fe5cc1cda50 2024-09-11T01:45:01.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.949+0000 7fe5d181b640 1 --2- 172.21.15.12:0/1437263909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc1cda50 0x7fe5cc02d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:01.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.949+0000 7fe5d181b640 10 monclient: get_auth_request con 0x7fe5cc1cda50 auth_method 0 2024-09-11T01:45:01.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.949+0000 7fe5d181b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:01.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.949+0000 7fe5d181b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:01.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.949+0000 7fe5d181b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:01.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.949+0000 7fe5d181b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:01.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.949+0000 7fe5d181b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:01.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.949+0000 7fe5d181b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:01.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.949+0000 7fe5d181b640 10 monclient(hunting): handle_auth_done global_id 174138 payload 824 2024-09-11T01:45:01.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.949+0000 7fe5d181b640 1 -- 172.21.15.12:0/1437263909 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe5cc09d130 con 0x7fe5cc1cda50 2024-09-11T01:45:01.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.951+0000 7fe5d181b640 1 -- 172.21.15.12:0/1437263909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc1cda50 msgr2=0x7fe5cc02d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:01.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.951+0000 7fe5d181b640 1 --2- 172.21.15.12:0/1437263909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc1cda50 0x7fe5cc02d980 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8017000 tx=0x7fe5c8004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:01.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.951+0000 7fe5c27fc640 1 -- 172.21.15.12:0/1437263909 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c801a070 con 0x7fe5cc1cda50 2024-09-11T01:45:01.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.951+0000 7fe5c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:01.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.951+0000 7fe5c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:01.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.951+0000 7fe5c27fc640 10 monclient: dump: 2024-09-11T01:45:01.963 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:01.963 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:01.963 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:01.963 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:01.963 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:01.963 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:01.963 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:01.963 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:01.964 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:01.964 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:01.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.952+0000 7fe5c27fc640 1 -- 172.21.15.12:0/1437263909 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7fe5c801b040 con 0x7fe5cc1cda50 2024-09-11T01:45:01.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.952+0000 7fe5c27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:45:01.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.952+0000 7fe5c27fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:45:01.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.952+0000 7fe5c27fc640 1 -- 172.21.15.12:0/1437263909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc1cda50 msgr2=0x7fe5cc02d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:01.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.952+0000 7fe5c27fc640 1 --2- 172.21.15.12:0/1437263909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc1cda50 0x7fe5cc02d980 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8017000 tx=0x7fe5c8004940 comp rx=0 tx=0).stop 2024-09-11T01:45:01.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.952+0000 7fe5c27fc640 1 -- 172.21.15.12:0/1437263909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc1cda50 msgr2=0x7fe5cc02d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:01.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.952+0000 7fe5c27fc640 1 --2- 172.21.15.12:0/1437263909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc1cda50 0x7fe5cc02d980 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8017000 tx=0x7fe5c8004940 comp rx=0 tx=0).stop 2024-09-11T01:45:01.964 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:45:01.965 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:45:01.965 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:45:01.965 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:45:01.965 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 148, 2024-09-11T01:45:01.965 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 18021, 2024-09-11T01:45:01.965 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:45:01.965 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:45:01.965 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:45:01.965 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:45:01.965 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:01.965 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:45:01.965 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:45:01.965 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:45:01.966 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:45:01.966 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:01.966 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:01.966 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:01.966 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:01.966 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:01.966 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:01.966 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:01.966 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:01.966 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:01.966 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:01.966 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:01.966 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:01.966 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:45:01.967 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:45:01.967 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:01.967 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:01.967 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:01.967 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:01.967 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:01.967 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:01.967 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:01.967 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:01.967 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:01.967 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:01.967 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:01.967 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:01.968 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:01.968 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:45:01.968 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:45:01.968 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:45:01.968 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:45:01.968 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:45:01.968 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:45:01.968 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:45:01.968 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:45:01.968 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:45:01.968 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:45:01.968 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:45:01.968 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:45:01.968 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:45:01.969 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:45:01.969 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:45:01.969 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:45:01.969 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:45:01.969 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:01.969 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:01.969 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:01.969 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:01.969 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:01.969 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:01.969 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:01.969 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:01.969 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:01.970 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:01.970 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:01.970 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:01.970 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:45:01.970 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:01.970 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:45:01.970 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:01.970 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:45:01.970 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:45:01.970 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:01.970 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:01.970 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:01.971 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:01.971 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:45:01.971 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:01.971 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:01.971 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:01.971 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:01.971 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:45:01.971 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:01.971 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:01.971 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:01.971 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:01.971 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:45:01.971 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:45:01.971 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:01.972 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:01.972 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:01.972 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:01.972 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:01.972 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:45:01.972 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:45:01.972 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:01.972 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:01.972 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:01.972 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:01.972 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:45:01.972 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:01.972 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:01.972 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:01.973 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:01.973 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:45:01.973 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:01.973 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:01.973 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:01.973 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:01.973 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:45:01.973 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:45:01.973 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:01.973 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:01.973 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:01.973 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:01.973 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:01.974 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:45:01.974 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:45:01.974 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:01.974 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:01.974 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:01.974 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:01.974 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:45:01.974 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:01.974 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:01.974 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:01.974 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:01.974 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:45:01.974 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:01.974 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:01.974 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:01.975 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:01.975 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:45:01.975 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:45:01.975 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:01.975 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:01.975 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:01.975 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:01.975 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:01.975 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:01.975 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:45:01.975 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:45:01.975 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:01.975 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:01.975 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:01.976 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:45:01.976 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:01.976 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:01.976 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:45:01.976 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:01.976 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:01.976 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:01.976 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:45:01.976 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:01.976 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:01.976 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:01.976 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:45:01.976 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:45:01.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.952+0000 7fe5d3a7f640 1 -- 172.21.15.12:0/1437263909 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe5b003e260 msgr2=0x7fe5b0040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:01.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.952+0000 7fe5d3a7f640 1 --2- 172.21.15.12:0/1437263909 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe5b003e260 0x7fe5b0040720 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc06ef60 tx=0x7fe5bc02e000 comp rx=0 tx=0).stop 2024-09-11T01:45:01.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.952+0000 7fe5d3a7f640 10 monclient: shutdown 2024-09-11T01:45:01.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.952+0000 7fe5d3a7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:01.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.952+0000 7fe5d3a7f640 1 -- 172.21.15.12:0/1437263909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc13b2f0 msgr2=0x7fe5cc130c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:01.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.952+0000 7fe5d3a7f640 1 --2- 172.21.15.12:0/1437263909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5cc13b2f0 0x7fe5cc130c40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe5c400c780 tx=0x7fe5c400cc40 comp rx=0 tx=0).stop 2024-09-11T01:45:01.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.952+0000 7fe5d201c640 1 -- 172.21.15.12:0/1437263909 reap_dead start 2024-09-11T01:45:01.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.952+0000 7fe5d201c640 1 -- 172.21.15.12:0/1437263909 reap_dead start 2024-09-11T01:45:01.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.952+0000 7fe5d3a7f640 1 -- 172.21.15.12:0/1437263909 shutdown_connections 2024-09-11T01:45:01.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.952+0000 7fe5d3a7f640 1 -- 172.21.15.12:0/1437263909 >> 172.21.15.12:0/1437263909 conn(0x7fe5cc0a36c0 msgr2=0x7fe5cc13de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:01.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.952+0000 7fe5d3a7f640 1 -- 172.21.15.12:0/1437263909 shutdown_connections 2024-09-11T01:45:01.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:01.952+0000 7fe5d3a7f640 1 -- 172.21.15.12:0/1437263909 wait complete. 2024-09-11T01:45:01.979 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:45:01.979 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:45:01.979 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:45:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.079+0000 7f86a4c2a640 10 monclient: get_monmap_and_config 2024-09-11T01:45:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.079+0000 7f86a4c2a640 10 monclient: build_initial_monmap 2024-09-11T01:45:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.079+0000 7f86a4c2a640 10 monclient: monmap: 2024-09-11T01:45:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:45:02.080265+0000 2024-09-11T01:45:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:45:02.080265+0000 2024-09-11T01:45:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:45:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:45:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:45:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:45:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:45:02.082 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.080+0000 7f86a4c2a640 1 Processor -- start 2024-09-11T01:45:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.080+0000 7f86a4c2a640 1 -- start start 2024-09-11T01:45:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.080+0000 7f86a4c2a640 10 monclient: init 2024-09-11T01:45:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.080+0000 7f86a4c2a640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.080+0000 7f86a4c2a640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-11T01:45:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.080+0000 7f86a4c2a640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f86a00a2e20 0x7f86a006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.080+0000 7f86a4c2a640 10 monclient(hunting): picked mon.noname-c con 0x7f86a00a2e20 addr v2:172.21.15.71:3300/0 2024-09-11T01:45:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.080+0000 7f86a4c2a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f86a00a4970 addr v1:172.21.15.12:6789/0 2024-09-11T01:45:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.080+0000 7f86a4c2a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a006f910 0x7f86a013f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.080+0000 7f86a4c2a640 10 monclient(hunting): picked mon.noname-b con 0x7f86a006f910 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.080+0000 7f86a4c2a640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86a006f4f0 con 0x7f86a00a4970 2024-09-11T01:45:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.080+0000 7f86a4c2a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.080+0000 7f86a4c2a640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f86a006ef30 con 0x7f86a00a2e20 2024-09-11T01:45:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.080+0000 7f86a4c2a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.080+0000 7f86a4c2a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f86a006ec50 con 0x7f86a006f910 2024-09-11T01:45:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.080+0000 7f86a4c2a640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.080+0000 7f86a4c2a640 10 monclient(hunting): authenticate will time out at 1708.522817s 2024-09-11T01:45:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.080+0000 7f869dd9b640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f86a00a4970 0x7f86a00a3200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37760/0 (socket says 172.21.15.12:37760) 2024-09-11T01:45:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.080+0000 7f869ed9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a006f910 0x7f86a013f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.080+0000 7f869dd9b640 1 -- 172.21.15.12:0/3450823125 learned_addr learned my addr 172.21.15.12:0/3450823125 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:45:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.081+0000 7f869ed9d640 10 monclient(hunting): get_auth_request con 0x7f86a006f910 auth_method 0 2024-09-11T01:45:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.081+0000 7f869ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.081+0000 7f869ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.081+0000 7f869ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.081+0000 7f869ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.081+0000 7f869ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.081+0000 7f869e59c640 1 --2- 172.21.15.12:0/3450823125 >> v2:172.21.15.71:3300/0 conn(0x7f86a00a2e20 0x7f86a006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.081+0000 7f869ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.081+0000 7f869e59c640 10 monclient(hunting): get_auth_request con 0x7f86a00a2e20 auth_method 0 2024-09-11T01:45:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.081+0000 7f869e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.081+0000 7f869e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.081+0000 7f869e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.081+0000 7f869e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.081+0000 7f869e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.081+0000 7f869e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.081+0000 7f869ed9d640 10 monclient(hunting): handle_auth_done global_id 164195 payload 274 2024-09-11T01:45:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.081+0000 7f869ed9d640 1 -- 172.21.15.12:0/3450823125 >> v2:172.21.15.71:3300/0 conn(0x7f86a00a2e20 msgr2=0x7f86a006ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.081+0000 7f869ed9d640 1 --2- 172.21.15.12:0/3450823125 >> v2:172.21.15.71:3300/0 conn(0x7f86a00a2e20 0x7f86a006ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.081+0000 7f869ed9d640 1 -- 172.21.15.12:0/3450823125 >> v1:172.21.15.12:6789/0 conn(0x7f86a00a4970 legacy=0x7f86a00a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.081+0000 7f869ed9d640 10 monclient: _finish_hunting 0 2024-09-11T01:45:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.081+0000 7f869ed9d640 1 monclient: found mon.noname-b 2024-09-11T01:45:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.081+0000 7f869ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.081+0000 7f869ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:45:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.081+0000 7f869ed9d640 1 -- 172.21.15.12:0/3450823125 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a00a3910 con 0x7f86a006f910 2024-09-11T01:45:02.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f869ed9d640 1 -- 172.21.15.12:0/3450823125 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a006f910 msgr2=0x7f86a013f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:02.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f869ed9d640 1 --2- 172.21.15.12:0/3450823125 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a006f910 0x7f86a013f430 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f86980097a0 tx=0x7f8698029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57c5c1d5115b6733 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:02.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f869d59a640 1 -- 172.21.15.12:0/3450823125 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8698037070 con 0x7f86a006f910 2024-09-11T01:45:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f869d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f869d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:45:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f869d59a640 10 monclient: dump: 2024-09-11T01:45:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:02.094 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:02.094 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f869d59a640 10 monclient: _finish_auth 0 2024-09-11T01:45:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f869d59a640 10 monclient: _check_auth_tickets 2024-09-11T01:45:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f869d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f869d59a640 1 -- 172.21.15.12:0/3450823125 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8698032440 con 0x7f86a006f910 2024-09-11T01:45:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f869d59a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f869d59a640 1 -- 172.21.15.12:0/3450823125 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8698032810 con 0x7f86a006f910 2024-09-11T01:45:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f86a4c2a640 5 monclient: authenticate success, global_id 164195 2024-09-11T01:45:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f86a4c2a640 10 monclient: get_monmap_and_config success 2024-09-11T01:45:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f86a4c2a640 4 set_mon_vals no callback set 2024-09-11T01:45:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f869d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f869d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:45:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f869d59a640 10 monclient: dump: 2024-09-11T01:45:02.095 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:02.096 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:02.096 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.096 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.096 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:02.096 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:02.096 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:02.096 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:02.096 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:02.096 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:02.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f86a4c2a640 10 monclient: shutdown 2024-09-11T01:45:02.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f86a4c2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:02.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f86a4c2a640 1 -- 172.21.15.12:0/3450823125 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a006f910 msgr2=0x7f86a013f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:02.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f86a4c2a640 1 --2- 172.21.15.12:0/3450823125 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a006f910 0x7f86a013f430 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f86980097a0 tx=0x7f8698029fd0 comp rx=0 tx=0).stop 2024-09-11T01:45:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f86a4c2a640 1 -- 172.21.15.12:0/3450823125 shutdown_connections 2024-09-11T01:45:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f86a4c2a640 1 --2- 172.21.15.12:0/3450823125 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a006f910 0x7f86a013f430 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f86a4c2a640 1 --2- 172.21.15.12:0/3450823125 >> v2:172.21.15.71:3300/0 conn(0x7f86a00a2e20 0x7f86a006ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f86a4c2a640 1 -- 172.21.15.12:0/3450823125 >> 172.21.15.12:0/3450823125 conn(0x7f86a00a8600 msgr2=0x7f86a00a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f86a4c2a640 1 -- 172.21.15.12:0/3450823125 shutdown_connections 2024-09-11T01:45:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.084+0000 7f86a4c2a640 1 -- 172.21.15.12:0/3450823125 wait complete. 2024-09-11T01:45:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.085+0000 7f86a4c2a640 10 monclient: build_initial_monmap 2024-09-11T01:45:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.085+0000 7f86a4c2a640 10 monclient: monmap: 2024-09-11T01:45:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:45:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:45:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:45:02.097 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:45:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:45:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:45:02.098 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f86a4c2a640 1 Processor -- start 2024-09-11T01:45:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f86a4c2a640 1 -- start start 2024-09-11T01:45:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f86a4c2a640 10 monclient: init 2024-09-11T01:45:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f86a4c2a640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f86a4c2a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:45:02.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f86a4c2a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86a006f910 0x7f86a013e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f86a4c2a640 10 monclient(hunting): picked mon.noname-b con 0x7f86a006f910 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:45:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f86a4c2a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a00a2e20 0x7f86a013ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f86a4c2a640 10 monclient(hunting): picked mon.noname-c con 0x7f86a00a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f86a4c2a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86a00a4970 0x7f86a01cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f86a4c2a640 10 monclient(hunting): picked mon.noname-a con 0x7f86a00a4970 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:45:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f86a4c2a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f86a4c2a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f86a013fd00 con 0x7f86a00a4970 2024-09-11T01:45:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f86a4c2a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f86a4c2a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f86a013fb80 con 0x7f86a006f910 2024-09-11T01:45:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f86a4c2a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f86a4c2a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f86a013fe80 con 0x7f86a00a2e20 2024-09-11T01:45:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f86a4c2a640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f86a4c2a640 10 monclient(hunting): authenticate will time out at 1708.528701s 2024-09-11T01:45:02.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f869dd9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a00a2e20 0x7f86a013ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f869ed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86a00a4970 0x7f86a01cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f869dd9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a00a2e20 0x7f86a013ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:54182/0 (socket says 172.21.15.12:54182) 2024-09-11T01:45:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f869dd9b640 1 -- 172.21.15.12:0/2488619684 learned_addr learned my addr 172.21.15.12:0/2488619684 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f869dd9b640 10 monclient(hunting): get_auth_request con 0x7f86a00a2e20 auth_method 0 2024-09-11T01:45:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f869dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f869dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f869dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f869ed9d640 10 monclient(hunting): get_auth_request con 0x7f86a00a4970 auth_method 0 2024-09-11T01:45:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f869ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f869ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f869ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:02.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f869dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:02.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f869dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:02.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f869dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:02.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f869ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:02.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f869ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:02.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f869e59c640 1 --2- 172.21.15.12:0/2488619684 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86a006f910 0x7f86a013e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:02.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f869ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:02.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f869e59c640 10 monclient(hunting): get_auth_request con 0x7f86a006f910 auth_method 0 2024-09-11T01:45:02.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f869e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:02.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f869e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:02.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.086+0000 7f869e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:02.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.087+0000 7f869dd9b640 10 monclient(hunting): handle_auth_done global_id 164198 payload 824 2024-09-11T01:45:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.087+0000 7f869dd9b640 1 -- 172.21.15.12:0/2488619684 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86a006f910 msgr2=0x7f86a013e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.087+0000 7f869dd9b640 1 --2- 172.21.15.12:0/2488619684 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86a006f910 0x7f86a013e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.087+0000 7f869dd9b640 1 -- 172.21.15.12:0/2488619684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86a00a4970 msgr2=0x7f86a01cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.087+0000 7f869dd9b640 1 --2- 172.21.15.12:0/2488619684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86a00a4970 0x7f86a01cd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.087+0000 7f869dd9b640 10 monclient: _finish_hunting 0 2024-09-11T01:45:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.087+0000 7f869dd9b640 1 monclient: found mon.noname-c 2024-09-11T01:45:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.087+0000 7f869dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.087+0000 7f869dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:45:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.087+0000 7f869dd9b640 1 -- 172.21.15.12:0/2488619684 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a009f0b0 con 0x7f86a00a2e20 2024-09-11T01:45:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.087+0000 7f869e59c640 1 --2- 172.21.15.12:0/2488619684 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86a006f910 0x7f86a013e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:45:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.087+0000 7f869ed9d640 1 --2- 172.21.15.12:0/2488619684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86a00a4970 0x7f86a01cd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:45:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.089+0000 7f869dd9b640 1 -- 172.21.15.12:0/2488619684 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a00a2e20 msgr2=0x7f86a013ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:02.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.089+0000 7f869dd9b640 1 --2- 172.21.15.12:0/2488619684 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a00a2e20 0x7f86a013ed50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f868800c460 tx=0x7f868800c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:02.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.089+0000 7f8696ffd640 1 -- 172.21.15.12:0/2488619684 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8688007070 con 0x7f86a00a2e20 2024-09-11T01:45:02.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.089+0000 7f8696ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:02.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.089+0000 7f8696ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:45:02.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.089+0000 7f8696ffd640 10 monclient: dump: 2024-09-11T01:45:02.103 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:02.103 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:02.103 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.103 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.104 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:02.104 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:02.104 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:02.104 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:02.104 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:02.104 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:02.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.089+0000 7f8696ffd640 10 monclient: _finish_auth 0 2024-09-11T01:45:02.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.089+0000 7f8696ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:45:02.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.089+0000 7f8696ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:02.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.089+0000 7f8696ffd640 1 -- 172.21.15.12:0/2488619684 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8688007230 con 0x7f86a00a2e20 2024-09-11T01:45:02.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.089+0000 7f8696ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:02.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.089+0000 7f8696ffd640 1 -- 172.21.15.12:0/2488619684 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8688007620 con 0x7f86a00a2e20 2024-09-11T01:45:02.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.089+0000 7f86a4c2a640 5 monclient: authenticate success, global_id 164198 2024-09-11T01:45:02.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.089+0000 7f86a4c2a640 10 monclient: _renew_subs 2024-09-11T01:45:02.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.089+0000 7f86a4c2a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:45:02.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.089+0000 7f86a4c2a640 1 -- 172.21.15.12:0/2488619684 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86a013ba30 con 0x7f86a00a2e20 2024-09-11T01:45:02.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.089+0000 7f8696ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:02.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.089+0000 7f8696ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:45:02.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.089+0000 7f869cd99640 4 set_mon_vals no callback set 2024-09-11T01:45:02.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.089+0000 7f8696ffd640 10 monclient: dump: 2024-09-11T01:45:02.105 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:02.105 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:02.105 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.105 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.105 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:02.105 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:02.105 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:02.105 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:02.106 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:02.106 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:02.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.089+0000 7f86a4c2a640 10 monclient: _renew_subs 2024-09-11T01:45:02.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.089+0000 7f86a4c2a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:45:02.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.089+0000 7f86a4c2a640 1 -- 172.21.15.12:0/2488619684 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f86a009e870 con 0x7f86a00a2e20 2024-09-11T01:45:02.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.090+0000 7f86a4c2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:02.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.090+0000 7f86a4c2a640 1 --2- 172.21.15.12:0/2488619684 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a013bc00 0x7f86a002d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:02.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.090+0000 7f86a4c2a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:02.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.090+0000 7f86a4c2a640 1 -- 172.21.15.12:0/2488619684 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f86a006a060 con 0x7f86a013bc00 2024-09-11T01:45:02.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.090+0000 7f869e59c640 1 --2- 172.21.15.12:0/2488619684 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a013bc00 0x7f86a002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:02.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.090+0000 7f869e59c640 10 monclient: get_auth_request con 0x7f86a013bc00 auth_method 0 2024-09-11T01:45:02.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.090+0000 7f869e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:02.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.090+0000 7f869e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:02.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.090+0000 7f869e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:02.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.090+0000 7f8696ffd640 1 -- 172.21.15.12:0/2488619684 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f868801b400 con 0x7f86a00a2e20 2024-09-11T01:45:02.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.090+0000 7f869e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:02.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.090+0000 7f869e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:02.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.090+0000 7f869e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:02.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.090+0000 7f869e59c640 10 monclient(hunting): handle_auth_done global_id 164201 payload 824 2024-09-11T01:45:02.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.090+0000 7f869e59c640 1 -- 172.21.15.12:0/2488619684 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f86a006e4d0 con 0x7f86a013bc00 2024-09-11T01:45:02.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.090+0000 7f8696ffd640 1 --2- 172.21.15.12:0/2488619684 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f868003e210 0x7f86800406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:02.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.090+0000 7f8696ffd640 1 -- 172.21.15.12:0/2488619684 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(535..535 src has 317..535) ==== 4771+0+0 (secure 0 0 0) 0x7f8688052a00 con 0x7f86a00a2e20 2024-09-11T01:45:02.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.091+0000 7f869ed9d640 1 --2- 172.21.15.12:0/2488619684 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f868003e210 0x7f86800406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:02.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.091+0000 7f869ed9d640 10 monclient: get_auth_request con 0x7f868003e210 auth_method 0 2024-09-11T01:45:02.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.092+0000 7f869e59c640 1 -- 172.21.15.12:0/2488619684 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a013bc00 msgr2=0x7f86a002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:02.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.092+0000 7f869e59c640 1 --2- 172.21.15.12:0/2488619684 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a013bc00 0x7f86a002d440 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8690007e90 tx=0x7f8690011520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:02.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.093+0000 7f8696ffd640 1 -- 172.21.15.12:0/2488619684 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f869001a030 con 0x7f86a013bc00 2024-09-11T01:45:02.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.093+0000 7f8696ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:02.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.093+0000 7f8696ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:45:02.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.093+0000 7f8696ffd640 10 monclient: dump: 2024-09-11T01:45:02.110 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:02.110 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:02.110 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:02.111 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.093+0000 7f8696ffd640 1 -- 172.21.15.12:0/2488619684 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8690003990 con 0x7f86a013bc00 2024-09-11T01:45:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.093+0000 7f8696ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.093+0000 7f8696ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:45:02.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.093+0000 7f8696ffd640 1 -- 172.21.15.12:0/2488619684 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a013bc00 msgr2=0x7f86a002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:02.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.093+0000 7f8696ffd640 1 --2- 172.21.15.12:0/2488619684 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a013bc00 0x7f86a002d440 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8690007e90 tx=0x7f8690011520 comp rx=0 tx=0).stop 2024-09-11T01:45:02.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.093+0000 7f8696ffd640 1 -- 172.21.15.12:0/2488619684 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a013bc00 msgr2=0x7f86a002d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:02.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.093+0000 7f8696ffd640 1 --2- 172.21.15.12:0/2488619684 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a013bc00 0x7f86a002d440 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8690007e90 tx=0x7f8690011520 comp rx=0 tx=0).stop 2024-09-11T01:45:02.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.093+0000 7f869ed9d640 1 --2- 172.21.15.12:0/2488619684 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f868003e210 0x7f86800406d0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f86a0074f40 tx=0x7f869802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:02.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.099+0000 7f86a4c2a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:45:02.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.099+0000 7f86a4c2a640 1 --2- 172.21.15.12:0/2488619684 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a01398c0 0x7f86a002d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:02.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.099+0000 7f86a4c2a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:02.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.099+0000 7f86a4c2a640 1 -- 172.21.15.12:0/2488619684 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f86a006ded0 con 0x7f86a01398c0 2024-09-11T01:45:02.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.099+0000 7f869e59c640 1 --2- 172.21.15.12:0/2488619684 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a01398c0 0x7f86a002d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:02.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.099+0000 7f869e59c640 10 monclient: get_auth_request con 0x7f86a01398c0 auth_method 0 2024-09-11T01:45:02.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.099+0000 7f869e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:02.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.099+0000 7f869e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:02.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.099+0000 7f869e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:02.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.099+0000 7f869e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:02.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.099+0000 7f869e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:02.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.099+0000 7f869e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:02.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.099+0000 7f869e59c640 10 monclient(hunting): handle_auth_done global_id 164204 payload 824 2024-09-11T01:45:02.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.099+0000 7f869e59c640 1 -- 172.21.15.12:0/2488619684 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f86a00a0f30 con 0x7f86a01398c0 2024-09-11T01:45:02.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.102+0000 7f869e59c640 1 -- 172.21.15.12:0/2488619684 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a01398c0 msgr2=0x7f86a002d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:02.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.102+0000 7f869e59c640 1 --2- 172.21.15.12:0/2488619684 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a01398c0 0x7f86a002d980 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8690007e90 tx=0x7f8690004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:02.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.102+0000 7f8696ffd640 1 -- 172.21.15.12:0/2488619684 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8690016600 con 0x7f86a01398c0 2024-09-11T01:45:02.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.102+0000 7f8696ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:02.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.102+0000 7f8696ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:45:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.102+0000 7f8696ffd640 10 monclient: dump: 2024-09-11T01:45:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:02.115 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:02.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.102+0000 7f8696ffd640 1 -- 172.21.15.12:0/2488619684 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7f8690016600 con 0x7f86a01398c0 2024-09-11T01:45:02.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.102+0000 7f8696ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:45:02.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.102+0000 7f8696ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:45:02.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.102+0000 7f8696ffd640 1 -- 172.21.15.12:0/2488619684 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a01398c0 msgr2=0x7f86a002d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:02.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.102+0000 7f8696ffd640 1 --2- 172.21.15.12:0/2488619684 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a01398c0 0x7f86a002d980 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8690007e90 tx=0x7f8690004a40 comp rx=0 tx=0).stop 2024-09-11T01:45:02.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.102+0000 7f8696ffd640 1 -- 172.21.15.12:0/2488619684 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a01398c0 msgr2=0x7f86a002d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:02.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.102+0000 7f8696ffd640 1 --2- 172.21.15.12:0/2488619684 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a01398c0 0x7f86a002d980 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8690007e90 tx=0x7f8690004a40 comp rx=0 tx=0).stop 2024-09-11T01:45:02.115 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:45:02.116 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:45:02.116 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:45:02.116 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:45:02.116 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 148, 2024-09-11T01:45:02.116 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 78116, 2024-09-11T01:45:02.116 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:45:02.116 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:45:02.116 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:45:02.116 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:45:02.117 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:02.117 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:45:02.117 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:45:02.117 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:45:02.117 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:45:02.117 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:02.117 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:02.117 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:02.117 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:02.117 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:02.117 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:02.118 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:02.118 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:02.118 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:02.118 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:02.118 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:02.118 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:02.118 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:45:02.118 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:45:02.118 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:02.118 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:02.119 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:02.119 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:02.119 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:02.119 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:02.119 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:02.119 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:02.119 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:02.119 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:02.119 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:02.119 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:02.119 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:02.119 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:45:02.119 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:45:02.119 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:45:02.119 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:45:02.120 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:45:02.120 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:45:02.120 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:45:02.120 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:45:02.120 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:45:02.120 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:45:02.120 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:45:02.120 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:45:02.120 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:45:02.120 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:45:02.120 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:45:02.120 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:45:02.120 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:45:02.120 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:02.121 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:02.121 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:02.121 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:02.121 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:02.121 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:02.121 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:02.121 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:02.121 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:02.121 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:02.121 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:02.121 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:02.121 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:45:02.121 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:02.121 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:45:02.122 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:02.122 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:45:02.122 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:45:02.122 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:02.122 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:02.122 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:02.122 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:02.122 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:45:02.122 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:02.122 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:02.122 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:02.122 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:02.122 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:45:02.122 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:02.123 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:02.123 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:02.123 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:02.123 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:45:02.123 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:45:02.123 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:02.123 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:02.123 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:02.123 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:02.123 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:02.123 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:45:02.123 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:45:02.123 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:02.123 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:02.124 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:02.124 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:02.124 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:45:02.124 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:02.124 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:02.124 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:02.124 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:02.124 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:45:02.124 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:02.124 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:02.124 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:02.124 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:02.124 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:45:02.125 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:45:02.125 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:02.125 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:02.125 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:02.125 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:02.125 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:02.125 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:45:02.125 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:45:02.125 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:02.125 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:02.125 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:02.125 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:02.125 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:45:02.126 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:02.126 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:02.126 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:02.126 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:02.126 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:45:02.126 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:02.126 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:02.126 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:02.126 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:02.126 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:45:02.126 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:45:02.126 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:02.126 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:02.126 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:02.127 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:02.127 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:02.127 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:02.127 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:45:02.127 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:45:02.127 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:02.127 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:02.127 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:02.127 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:45:02.127 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:02.127 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:02.127 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:45:02.127 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:02.127 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:02.128 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:02.128 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:45:02.128 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:02.128 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:02.128 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:45:02.128 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:02.128 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:02.128 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:02.128 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:45:02.128 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:02.128 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:02.128 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:45:02.128 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:02.129 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:02.129 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:02.129 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:45:02.129 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:02.129 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:02.129 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:02.129 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:45:02.129 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:45:02.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.102+0000 7f86a4c2a640 1 -- 172.21.15.12:0/2488619684 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f868003e210 msgr2=0x7f86800406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:02.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.102+0000 7f86a4c2a640 1 --2- 172.21.15.12:0/2488619684 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f868003e210 0x7f86800406d0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f86a0074f40 tx=0x7f869802e000 comp rx=0 tx=0).stop 2024-09-11T01:45:02.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.102+0000 7f86a4c2a640 10 monclient: shutdown 2024-09-11T01:45:02.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.102+0000 7f86a4c2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:02.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.102+0000 7f86a4c2a640 1 -- 172.21.15.12:0/2488619684 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a00a2e20 msgr2=0x7f86a013ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:02.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.102+0000 7f86a4c2a640 1 --2- 172.21.15.12:0/2488619684 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86a00a2e20 0x7f86a013ed50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f868800c460 tx=0x7f868800c920 comp rx=0 tx=0).stop 2024-09-11T01:45:02.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.102+0000 7f869ed9d640 1 -- 172.21.15.12:0/2488619684 reap_dead start 2024-09-11T01:45:02.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.102+0000 7f869ed9d640 1 -- 172.21.15.12:0/2488619684 reap_dead start 2024-09-11T01:45:02.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.102+0000 7f86a4c2a640 1 -- 172.21.15.12:0/2488619684 shutdown_connections 2024-09-11T01:45:02.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.102+0000 7f86a4c2a640 1 -- 172.21.15.12:0/2488619684 >> 172.21.15.12:0/2488619684 conn(0x7f86a00a8600 msgr2=0x7f86a00736b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:02.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.102+0000 7f86a4c2a640 1 -- 172.21.15.12:0/2488619684 shutdown_connections 2024-09-11T01:45:02.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.102+0000 7f86a4c2a640 1 -- 172.21.15.12:0/2488619684 wait complete. 2024-09-11T01:45:02.132 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:45:02.132 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:45:02.132 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:45:02.172 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.171+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.172 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.171+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3502719442}) 2024-09-11T01:45:02.172 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.171+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=26385530}) 2024-09-11T01:45:02.172 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.171+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.172 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.171+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3502719442}) 2024-09-11T01:45:02.172 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.171+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2265188028}) 2024-09-11T01:45:02.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.191+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.191+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4080247344}) 2024-09-11T01:45:02.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.191+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=551960545}) 2024-09-11T01:45:02.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.191+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.191+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4080247344}) 2024-09-11T01:45:02.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.191+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=199083530}) 2024-09-11T01:45:02.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.233+0000 7fe609a1f640 10 monclient: get_monmap_and_config 2024-09-11T01:45:02.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.233+0000 7fe609a1f640 10 monclient: build_initial_monmap 2024-09-11T01:45:02.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.233+0000 7fe609a1f640 10 monclient: monmap: 2024-09-11T01:45:02.235 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:02.235 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:02.235 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:45:02.234533+0000 2024-09-11T01:45:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:45:02.234533+0000 2024-09-11T01:45:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:45:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:45:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:45:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:45:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:45:02.236 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.234+0000 7fe609a1f640 1 Processor -- start 2024-09-11T01:45:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.234+0000 7fe609a1f640 1 -- start start 2024-09-11T01:45:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.234+0000 7fe609a1f640 10 monclient: init 2024-09-11T01:45:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.234+0000 7fe609a1f640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.234+0000 7fe609a1f640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-11T01:45:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.234+0000 7fe609a1f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe6040a03a0 addr v1:172.21.15.71:6789/0 2024-09-11T01:45:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.234+0000 7fe609a1f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe6040a2530 0x7fe60409f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.234+0000 7fe609a1f640 10 monclient(hunting): picked mon.noname-c con 0x7fe6040a2530 addr v2:172.21.15.71:3300/0 2024-09-11T01:45:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.234+0000 7fe609a1f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6040a13a0 0x7fe60413baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.234+0000 7fe609a1f640 10 monclient(hunting): picked mon.noname-b con 0x7fe6040a13a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.234+0000 7fe609a1f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe60409e510 con 0x7fe6040a03a0 2024-09-11T01:45:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.234+0000 7fe609a1f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.234+0000 7fe609a1f640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fe6040a29b0 con 0x7fe6040a2530 2024-09-11T01:45:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.234+0000 7fe609a1f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.234+0000 7fe609a1f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe6040a0780 con 0x7fe6040a13a0 2024-09-11T01:45:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.234+0000 7fe609a1f640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.234+0000 7fe609a1f640 10 monclient(hunting): authenticate will time out at 1708.676981s 2024-09-11T01:45:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.235+0000 7fe6037fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6040a13a0 0x7fe60413baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.235+0000 7fe6037fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6040a13a0 0x7fe60413baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:54196/0 (socket says 172.21.15.12:54196) 2024-09-11T01:45:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.235+0000 7fe6037fe640 1 -- 172.21.15.12:0/497429708 learned_addr learned my addr 172.21.15.12:0/497429708 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.235+0000 7fe6037fe640 10 monclient(hunting): get_auth_request con 0x7fe6040a13a0 auth_method 0 2024-09-11T01:45:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.235+0000 7fe6037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.235+0000 7fe6037fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.235+0000 7fe6037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.235+0000 7fe6037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.235+0000 7fe6037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.235+0000 7fe6037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.235+0000 7fe6037fe640 10 monclient(hunting): handle_auth_done global_id 164207 payload 274 2024-09-11T01:45:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.235+0000 7fe6027fc640 1 --2- 172.21.15.12:0/497429708 >> v2:172.21.15.71:3300/0 conn(0x7fe6040a2530 0x7fe60409f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.235+0000 7fe6037fe640 1 -- 172.21.15.12:0/497429708 >> v2:172.21.15.71:3300/0 conn(0x7fe6040a2530 msgr2=0x7fe60409f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.235+0000 7fe6037fe640 1 --2- 172.21.15.12:0/497429708 >> v2:172.21.15.71:3300/0 conn(0x7fe6040a2530 0x7fe60409f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.235+0000 7fe6037fe640 1 -- 172.21.15.12:0/497429708 >> v1:172.21.15.71:6789/0 conn(0x7fe6040a03a0 legacy=0x7fe6040a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.235+0000 7fe6037fe640 10 monclient: _finish_hunting 0 2024-09-11T01:45:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.235+0000 7fe6037fe640 1 monclient: found mon.noname-b 2024-09-11T01:45:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.235+0000 7fe6037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.235+0000 7fe6037fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:45:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.235+0000 7fe6037fe640 1 -- 172.21.15.12:0/497429708 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe60409edb0 con 0x7fe6040a13a0 2024-09-11T01:45:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe6037fe640 1 -- 172.21.15.12:0/497429708 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6040a13a0 msgr2=0x7fe60413baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe6037fe640 1 --2- 172.21.15.12:0/497429708 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6040a13a0 0x7fe60413baa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc009e50 tx=0x7fe5fc02a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=421997b5b409055d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:02.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe601ffb640 1 -- 172.21.15.12:0/497429708 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc036070 con 0x7fe6040a13a0 2024-09-11T01:45:02.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe601ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:02.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe601ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:45:02.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe601ffb640 10 monclient: dump: 2024-09-11T01:45:02.249 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:02.249 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:02.249 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.250 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.250 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:02.250 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:02.250 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:02.250 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:02.250 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:02.250 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:02.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe601ffb640 10 monclient: _finish_auth 0 2024-09-11T01:45:02.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe601ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:45:02.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe601ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:02.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe601ffb640 1 -- 172.21.15.12:0/497429708 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5fc007800 con 0x7fe6040a13a0 2024-09-11T01:45:02.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe601ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:02.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe601ffb640 1 -- 172.21.15.12:0/497429708 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc007bd0 con 0x7fe6040a13a0 2024-09-11T01:45:02.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe609a1f640 5 monclient: authenticate success, global_id 164207 2024-09-11T01:45:02.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe609a1f640 10 monclient: get_monmap_and_config success 2024-09-11T01:45:02.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe601ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:02.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe609a1f640 4 set_mon_vals no callback set 2024-09-11T01:45:02.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe601ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:45:02.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe601ffb640 10 monclient: dump: 2024-09-11T01:45:02.251 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:02.251 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:02.251 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.252 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.252 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:02.252 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:02.252 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:02.252 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:02.252 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:02.252 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:02.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe609a1f640 10 monclient: shutdown 2024-09-11T01:45:02.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe609a1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:02.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe609a1f640 1 -- 172.21.15.12:0/497429708 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6040a13a0 msgr2=0x7fe60413baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:02.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe609a1f640 1 --2- 172.21.15.12:0/497429708 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6040a13a0 0x7fe60413baa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc009e50 tx=0x7fe5fc02a560 comp rx=0 tx=0).stop 2024-09-11T01:45:02.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe609a1f640 1 -- 172.21.15.12:0/497429708 shutdown_connections 2024-09-11T01:45:02.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe609a1f640 1 --2- 172.21.15.12:0/497429708 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6040a13a0 0x7fe60413baa0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:02.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe609a1f640 1 --2- 172.21.15.12:0/497429708 >> v2:172.21.15.71:3300/0 conn(0x7fe6040a2530 0x7fe60409f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:02.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe609a1f640 1 -- 172.21.15.12:0/497429708 >> 172.21.15.12:0/497429708 conn(0x7fe6040a2f70 msgr2=0x7fe6040a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:02.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe609a1f640 1 -- 172.21.15.12:0/497429708 shutdown_connections 2024-09-11T01:45:02.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.238+0000 7fe609a1f640 1 -- 172.21.15.12:0/497429708 wait complete. 2024-09-11T01:45:02.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.239+0000 7fe609a1f640 10 monclient: build_initial_monmap 2024-09-11T01:45:02.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.239+0000 7fe609a1f640 10 monclient: monmap: 2024-09-11T01:45:02.253 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:02.253 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:45:02.253 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:45:02.254 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:45:02.254 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:02.254 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:02.254 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:45:02.254 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:45:02.254 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:45:02.254 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:02.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.239+0000 7fe609a1f640 1 Processor -- start 2024-09-11T01:45:02.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe609a1f640 1 -- start start 2024-09-11T01:45:02.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe609a1f640 10 monclient: init 2024-09-11T01:45:02.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe609a1f640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:02.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe609a1f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:45:02.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe609a1f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6040a03a0 0x7fe60411fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:02.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe609a1f640 10 monclient(hunting): picked mon.noname-a con 0x7fe6040a03a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:45:02.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe609a1f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6040a13a0 0x7fe6041ca560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:02.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe609a1f640 10 monclient(hunting): picked mon.noname-b con 0x7fe6040a13a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:45:02.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe609a1f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6040a2530 0x7fe60411d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:02.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe609a1f640 10 monclient(hunting): picked mon.noname-c con 0x7fe6040a2530 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:02.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe609a1f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:02.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe609a1f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe60409ca30 con 0x7fe6040a03a0 2024-09-11T01:45:02.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe609a1f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:02.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe609a1f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe6040a1a30 con 0x7fe6040a13a0 2024-09-11T01:45:02.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe609a1f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:02.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe609a1f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe60409cbb0 con 0x7fe6040a2530 2024-09-11T01:45:02.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe609a1f640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:02.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe609a1f640 10 monclient(hunting): authenticate will time out at 1708.682591s 2024-09-11T01:45:02.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe6037fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6040a2530 0x7fe60411d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:02.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe602ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6040a03a0 0x7fe60411fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:02.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe6037fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6040a2530 0x7fe60411d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:54206/0 (socket says 172.21.15.12:54206) 2024-09-11T01:45:02.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe6037fe640 1 -- 172.21.15.12:0/2634249643 learned_addr learned my addr 172.21.15.12:0/2634249643 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:02.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe6037fe640 10 monclient(hunting): get_auth_request con 0x7fe6040a2530 auth_method 0 2024-09-11T01:45:02.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe6037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:02.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe6037fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:02.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe6037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:02.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe602ffd640 10 monclient(hunting): get_auth_request con 0x7fe6040a03a0 auth_method 0 2024-09-11T01:45:02.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe602ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:02.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe602ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:02.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe602ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:02.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe6037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:02.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe6037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:02.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe6037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:02.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe602ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:02.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe602ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:02.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe602ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:02.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe6027fc640 1 --2- 172.21.15.12:0/2634249643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6040a13a0 0x7fe6041ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:02.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe6027fc640 10 monclient(hunting): get_auth_request con 0x7fe6040a13a0 auth_method 0 2024-09-11T01:45:02.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe6027fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:02.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe6027fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:02.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe6027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:02.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe6027fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:02.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe6027fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:02.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe6027fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:02.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.240+0000 7fe602ffd640 10 monclient(hunting): handle_auth_done global_id 174144 payload 824 2024-09-11T01:45:02.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.241+0000 7fe602ffd640 1 -- 172.21.15.12:0/2634249643 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6040a2530 msgr2=0x7fe60411d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:02.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.241+0000 7fe602ffd640 1 --2- 172.21.15.12:0/2634249643 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6040a2530 0x7fe60411d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:02.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.241+0000 7fe602ffd640 1 -- 172.21.15.12:0/2634249643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6040a13a0 msgr2=0x7fe6041ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:02.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.241+0000 7fe602ffd640 1 --2- 172.21.15.12:0/2634249643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6040a13a0 0x7fe6041ca560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:02.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.241+0000 7fe602ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:45:02.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.241+0000 7fe602ffd640 1 monclient: found mon.noname-a 2024-09-11T01:45:02.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.241+0000 7fe602ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:02.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.241+0000 7fe602ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:45:02.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.241+0000 7fe602ffd640 1 -- 172.21.15.12:0/2634249643 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe60409d480 con 0x7fe6040a03a0 2024-09-11T01:45:02.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.241+0000 7fe6037fe640 1 --2- 172.21.15.12:0/2634249643 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6040a2530 0x7fe60411d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:45:02.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.243+0000 7fe602ffd640 1 -- 172.21.15.12:0/2634249643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6040a03a0 msgr2=0x7fe60411fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:02.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.243+0000 7fe602ffd640 1 --2- 172.21.15.12:0/2634249643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6040a03a0 0x7fe60411fa20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe5f400c650 tx=0x7fe5f400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:02.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.243+0000 7fe5ebfff640 1 -- 172.21.15.12:0/2634249643 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f40052e0 con 0x7fe6040a03a0 2024-09-11T01:45:02.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.243+0000 7fe5ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:02.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.243+0000 7fe5ebfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:45:02.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.243+0000 7fe5ebfff640 10 monclient: dump: 2024-09-11T01:45:02.260 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:02.260 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:02.260 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.260 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.260 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:02.260 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:02.260 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:02.260 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:02.261 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:02.261 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:02.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.243+0000 7fe5ebfff640 10 monclient: _finish_auth 0 2024-09-11T01:45:02.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.243+0000 7fe5ebfff640 10 monclient: _check_auth_tickets 2024-09-11T01:45:02.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.243+0000 7fe5ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:02.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.243+0000 7fe5ebfff640 1 -- 172.21.15.12:0/2634249643 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5f4016020 con 0x7fe6040a03a0 2024-09-11T01:45:02.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.243+0000 7fe609a1f640 5 monclient: authenticate success, global_id 174144 2024-09-11T01:45:02.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.243+0000 7fe609a1f640 10 monclient: _renew_subs 2024-09-11T01:45:02.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.243+0000 7fe609a1f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:02.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.243+0000 7fe609a1f640 1 -- 172.21.15.12:0/2634249643 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe604120c60 con 0x7fe6040a03a0 2024-09-11T01:45:02.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.243+0000 7fe5ebfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:02.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.243+0000 7fe5ebfff640 1 -- 172.21.15.12:0/2634249643 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f4001940 con 0x7fe6040a03a0 2024-09-11T01:45:02.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.243+0000 7fe5ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:02.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.243+0000 7fe5ebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:02.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.243+0000 7fe5ebfff640 10 monclient: dump: 2024-09-11T01:45:02.262 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:02.262 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:02.262 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.262 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.262 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:02.262 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:02.262 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:02.263 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:02.263 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:02.263 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:02.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.243+0000 7fe6017fa640 4 set_mon_vals no callback set 2024-09-11T01:45:02.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.243+0000 7fe609a1f640 10 monclient: _renew_subs 2024-09-11T01:45:02.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.243+0000 7fe609a1f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:02.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.243+0000 7fe609a1f640 1 -- 172.21.15.12:0/2634249643 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6041c2e10 con 0x7fe6040a03a0 2024-09-11T01:45:02.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.244+0000 7fe5ebfff640 1 -- 172.21.15.12:0/2634249643 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fe5f400f040 con 0x7fe6040a03a0 2024-09-11T01:45:02.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.244+0000 7fe609a1f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:02.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.244+0000 7fe609a1f640 1 --2- 172.21.15.12:0/2634249643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6041caaa0 0x7fe60413cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:02.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.244+0000 7fe609a1f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:02.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.244+0000 7fe609a1f640 1 -- 172.21.15.12:0/2634249643 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe60409c4d0 con 0x7fe6041caaa0 2024-09-11T01:45:02.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.244+0000 7fe5ebfff640 1 --2- 172.21.15.12:0/2634249643 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe5f003e210 0x7fe5f00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:02.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.244+0000 7fe6027fc640 1 --2- 172.21.15.12:0/2634249643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6041caaa0 0x7fe60413cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:02.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.244+0000 7fe6027fc640 10 monclient: get_auth_request con 0x7fe6041caaa0 auth_method 0 2024-09-11T01:45:02.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.244+0000 7fe6027fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:02.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.244+0000 7fe6027fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:02.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.244+0000 7fe6027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:02.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.244+0000 7fe5ebfff640 1 -- 172.21.15.12:0/2634249643 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(535..535 src has 317..535) ==== 4771+0+0 (secure 0 0 0) 0x7fe5f40552d0 con 0x7fe6040a03a0 2024-09-11T01:45:02.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.244+0000 7fe6027fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:02.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.244+0000 7fe6027fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:02.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.244+0000 7fe6027fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:02.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.244+0000 7fe6037fe640 1 --2- 172.21.15.12:0/2634249643 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe5f003e210 0x7fe5f00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:02.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.244+0000 7fe6037fe640 10 monclient: get_auth_request con 0x7fe5f003e210 auth_method 0 2024-09-11T01:45:02.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.244+0000 7fe6027fc640 10 monclient(hunting): handle_auth_done global_id 134479 payload 824 2024-09-11T01:45:02.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.244+0000 7fe6027fc640 1 -- 172.21.15.12:0/2634249643 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe60409d010 con 0x7fe6041caaa0 2024-09-11T01:45:02.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.247+0000 7fe6027fc640 1 -- 172.21.15.12:0/2634249643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6041caaa0 msgr2=0x7fe60413cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:02.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.247+0000 7fe6027fc640 1 --2- 172.21.15.12:0/2634249643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6041caaa0 0x7fe60413cff0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe604121150 tx=0x7fe5f800fc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:02.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.247+0000 7fe6037fe640 1 --2- 172.21.15.12:0/2634249643 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe5f003e210 0x7fe5f00406d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc02a5a0 tx=0x7fe5fc009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:02.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.247+0000 7fe5ebfff640 1 -- 172.21.15.12:0/2634249643 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f800de00 con 0x7fe6041caaa0 2024-09-11T01:45:02.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.247+0000 7fe5ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:02.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.247+0000 7fe5ebfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:45:02.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.247+0000 7fe5ebfff640 10 monclient: dump: 2024-09-11T01:45:02.265 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:02.266 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:02.266 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.266 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.266 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:02.266 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:02.266 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:02.266 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:02.266 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:02.266 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:02.267 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.248+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.267 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.248+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1858462115}) 2024-09-11T01:45:02.267 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.248+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=28725618}) 2024-09-11T01:45:02.267 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.248+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.267 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.248+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1858462115}) 2024-09-11T01:45:02.267 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.248+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3414679477}) 2024-09-11T01:45:02.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.248+0000 7fe5ebfff640 1 -- 172.21.15.12:0/2634249643 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe5f800de00 con 0x7fe6041caaa0 2024-09-11T01:45:02.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.248+0000 7fe5ebfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:02.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.248+0000 7fe5ebfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:45:02.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.248+0000 7fe5ebfff640 1 -- 172.21.15.12:0/2634249643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6041caaa0 msgr2=0x7fe60413cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:02.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.248+0000 7fe5ebfff640 1 --2- 172.21.15.12:0/2634249643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6041caaa0 0x7fe60413cff0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe604121150 tx=0x7fe5f800fc70 comp rx=0 tx=0).stop 2024-09-11T01:45:02.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.248+0000 7fe5ebfff640 1 -- 172.21.15.12:0/2634249643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6041caaa0 msgr2=0x7fe60413cff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:02.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.248+0000 7fe5ebfff640 1 --2- 172.21.15.12:0/2634249643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6041caaa0 0x7fe60413cff0 secure :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe604121150 tx=0x7fe5f800fc70 comp rx=0 tx=0).stop 2024-09-11T01:45:02.275 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.248+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.275 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.248+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2175898207}) 2024-09-11T01:45:02.275 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.248+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3477461716}) 2024-09-11T01:45:02.275 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.248+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.276 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.248+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2175898207}) 2024-09-11T01:45:02.276 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.249+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1886334228}) 2024-09-11T01:45:02.276 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.249+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.276 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.249+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364830730}) 2024-09-11T01:45:02.276 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.249+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=783027187}) 2024-09-11T01:45:02.276 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.249+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.276 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.249+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364830730}) 2024-09-11T01:45:02.276 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.249+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1993920972}) 2024-09-11T01:45:02.276 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.250+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.276 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.250+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1415246352}) 2024-09-11T01:45:02.276 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.250+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1231110529}) 2024-09-11T01:45:02.277 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.250+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.277 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.250+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1415246352}) 2024-09-11T01:45:02.277 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.250+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1728079833}) 2024-09-11T01:45:02.277 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.251+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.277 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.251+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=646083090}) 2024-09-11T01:45:02.277 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.251+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3420603854}) 2024-09-11T01:45:02.277 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.251+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.277 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.251+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=646083090}) 2024-09-11T01:45:02.277 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.251+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4079044567}) 2024-09-11T01:45:02.277 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.252+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.277 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.252+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2703627488}) 2024-09-11T01:45:02.277 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.252+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1297567006}) 2024-09-11T01:45:02.278 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.252+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.278 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.252+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2703627488}) 2024-09-11T01:45:02.278 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.252+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=733806646}) 2024-09-11T01:45:02.278 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.253+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.278 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.253+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=961715905}) 2024-09-11T01:45:02.278 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.253+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=122129082}) 2024-09-11T01:45:02.278 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.253+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.278 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.253+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=961715905}) 2024-09-11T01:45:02.278 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.253+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1007557536}) 2024-09-11T01:45:02.278 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.254+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.278 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.254+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3075805728}) 2024-09-11T01:45:02.278 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.254+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2396381723}) 2024-09-11T01:45:02.279 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.254+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.279 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.254+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3075805728}) 2024-09-11T01:45:02.279 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.254+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3333952249}) 2024-09-11T01:45:02.279 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.255+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.279 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.255+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3577269369}) 2024-09-11T01:45:02.279 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.255+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=697249606}) 2024-09-11T01:45:02.279 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.255+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.279 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.255+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3577269369}) 2024-09-11T01:45:02.279 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.255+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3294172762}) 2024-09-11T01:45:02.279 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.256+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.279 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.256+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=352357113}) 2024-09-11T01:45:02.279 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.256+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=57347948}) 2024-09-11T01:45:02.280 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.256+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.280 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.256+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=352357113}) 2024-09-11T01:45:02.280 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.256+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3229719036}) 2024-09-11T01:45:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.260+0000 7fe609a1f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:45:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.260+0000 7fe609a1f640 1 --2- 172.21.15.12:0/2634249643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe60413d530 0x7fe60413d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.260+0000 7fe609a1f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.260+0000 7fe609a1f640 1 -- 172.21.15.12:0/2634249643 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe60411eec0 con 0x7fe60413d530 2024-09-11T01:45:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.261+0000 7fe6027fc640 1 --2- 172.21.15.12:0/2634249643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe60413d530 0x7fe60413d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.261+0000 7fe6027fc640 10 monclient: get_auth_request con 0x7fe60413d530 auth_method 0 2024-09-11T01:45:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.261+0000 7fe6027fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.261+0000 7fe6027fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.261+0000 7fe6027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.261+0000 7fe6027fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.261+0000 7fe6027fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:02.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.261+0000 7fe6027fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:02.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.261+0000 7fe6027fc640 10 monclient(hunting): handle_auth_done global_id 134482 payload 824 2024-09-11T01:45:02.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.262+0000 7fe6027fc640 1 -- 172.21.15.12:0/2634249643 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe604121280 con 0x7fe60413d530 2024-09-11T01:45:02.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.264+0000 7fe6027fc640 1 -- 172.21.15.12:0/2634249643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe60413d530 msgr2=0x7fe60413d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:02.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.264+0000 7fe6027fc640 1 --2- 172.21.15.12:0/2634249643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe60413d530 0x7fe60413d940 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe604121150 tx=0x7fe5f80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:02.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.264+0000 7fe5ebfff640 1 -- 172.21.15.12:0/2634249643 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f8006510 con 0x7fe60413d530 2024-09-11T01:45:02.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.264+0000 7fe5ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:02.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.264+0000 7fe5ebfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:45:02.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.264+0000 7fe5ebfff640 10 monclient: dump: 2024-09-11T01:45:02.283 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:02.283 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:02.284 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.284 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.284 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:02.284 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:02.284 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:02.284 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:02.284 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:02.284 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:02.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.265+0000 7fe5ebfff640 1 -- 172.21.15.12:0/2634249643 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7fe5f8006510 con 0x7fe60413d530 2024-09-11T01:45:02.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.265+0000 7fe5ebfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:45:02.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.265+0000 7fe5ebfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:45:02.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.265+0000 7fe5ebfff640 1 -- 172.21.15.12:0/2634249643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe60413d530 msgr2=0x7fe60413d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:02.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.265+0000 7fe5ebfff640 1 --2- 172.21.15.12:0/2634249643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe60413d530 0x7fe60413d940 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe604121150 tx=0x7fe5f80046c0 comp rx=0 tx=0).stop 2024-09-11T01:45:02.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.265+0000 7fe5ebfff640 1 -- 172.21.15.12:0/2634249643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe60413d530 msgr2=0x7fe60413d940 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:02.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.265+0000 7fe5ebfff640 1 --2- 172.21.15.12:0/2634249643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe60413d530 0x7fe60413d940 secure :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe604121150 tx=0x7fe5f80046c0 comp rx=0 tx=0).stop 2024-09-11T01:45:02.285 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:45:02.285 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:45:02.286 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:45:02.286 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:45:02.286 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 148, 2024-09-11T01:45:02.286 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 235226, 2024-09-11T01:45:02.286 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:45:02.286 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:45:02.286 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:45:02.286 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:45:02.286 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:02.286 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:45:02.286 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:45:02.287 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:45:02.287 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:45:02.287 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:02.287 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:02.287 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:02.287 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:02.287 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:02.287 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:02.287 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:02.287 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:02.287 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:02.288 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:02.288 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:02.288 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:02.288 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:45:02.288 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:45:02.288 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:02.288 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:02.288 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:02.288 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:02.288 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:02.288 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:02.289 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:02.289 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:02.289 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:02.289 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:02.289 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:02.289 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:02.289 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:02.289 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:45:02.289 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:45:02.289 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:45:02.289 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:45:02.290 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:45:02.290 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:45:02.290 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:45:02.290 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:45:02.290 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:45:02.290 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:45:02.290 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:45:02.290 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:45:02.290 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:45:02.290 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:45:02.291 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:45:02.291 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:45:02.291 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:45:02.291 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:02.291 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:02.291 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:02.291 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:02.291 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:02.291 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:02.291 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:02.291 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:02.291 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:02.292 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:02.292 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:02.292 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:02.292 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:45:02.292 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:02.292 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:45:02.292 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:02.292 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:45:02.292 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:45:02.292 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:02.292 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:02.292 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:02.293 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:02.293 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:45:02.293 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:02.293 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:02.293 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:02.293 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:02.293 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:45:02.293 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:02.293 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:02.293 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:02.293 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:02.293 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:45:02.294 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:45:02.294 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:02.294 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:02.294 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:02.294 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:02.294 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:02.294 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:45:02.294 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:45:02.294 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:02.294 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:02.294 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:02.295 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:02.295 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:45:02.295 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:02.295 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:02.295 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:02.295 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:02.295 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:45:02.295 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:02.295 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:02.295 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:02.295 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:02.296 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:45:02.296 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:45:02.296 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:02.296 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:02.296 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:02.296 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:02.296 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:02.296 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:45:02.297 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:45:02.297 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:02.297 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:02.297 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:02.297 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:02.297 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:45:02.297 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:02.297 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:02.297 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:02.297 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:02.298 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:45:02.298 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:02.298 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:02.298 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:02.298 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:02.298 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:45:02.298 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:45:02.298 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:02.298 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:02.298 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:02.298 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:02.299 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:02.299 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:02.299 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:45:02.299 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:45:02.299 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:02.299 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:02.299 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:02.300 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:45:02.300 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:02.300 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:02.300 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:45:02.300 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:02.300 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:02.300 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:02.300 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:45:02.300 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:02.300 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:02.300 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:45:02.301 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:02.301 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:02.301 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:02.301 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:45:02.301 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:02.301 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:02.301 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:02.301 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:45:02.301 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:45:02.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.265+0000 7fe609a1f640 1 -- 172.21.15.12:0/2634249643 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe5f003e210 msgr2=0x7fe5f00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:02.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.265+0000 7fe609a1f640 1 --2- 172.21.15.12:0/2634249643 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe5f003e210 0x7fe5f00406d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc02a5a0 tx=0x7fe5fc009000 comp rx=0 tx=0).stop 2024-09-11T01:45:02.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.265+0000 7fe609a1f640 10 monclient: shutdown 2024-09-11T01:45:02.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.265+0000 7fe609a1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:02.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.265+0000 7fe609a1f640 1 -- 172.21.15.12:0/2634249643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6040a03a0 msgr2=0x7fe60411fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:02.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.265+0000 7fe609a1f640 1 --2- 172.21.15.12:0/2634249643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6040a03a0 0x7fe60411fa20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe5f400c650 tx=0x7fe5f400cb10 comp rx=0 tx=0).stop 2024-09-11T01:45:02.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.265+0000 7fe6037fe640 1 -- 172.21.15.12:0/2634249643 reap_dead start 2024-09-11T01:45:02.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.265+0000 7fe6037fe640 1 -- 172.21.15.12:0/2634249643 reap_dead start 2024-09-11T01:45:02.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.265+0000 7fe609a1f640 1 -- 172.21.15.12:0/2634249643 shutdown_connections 2024-09-11T01:45:02.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.265+0000 7fe609a1f640 1 -- 172.21.15.12:0/2634249643 >> 172.21.15.12:0/2634249643 conn(0x7fe6040a2f70 msgr2=0x7fe604072e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:02.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.265+0000 7fe609a1f640 1 -- 172.21.15.12:0/2634249643 shutdown_connections 2024-09-11T01:45:02.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.265+0000 7fe609a1f640 1 -- 172.21.15.12:0/2634249643 wait complete. 2024-09-11T01:45:02.310 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.268+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.310 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.268+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=711348160}) 2024-09-11T01:45:02.310 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.268+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2855390671}) 2024-09-11T01:45:02.310 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.268+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.310 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.268+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=711348160}) 2024-09-11T01:45:02.311 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.268+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4227823155}) 2024-09-11T01:45:02.311 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.269+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.311 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.269+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2693218468}) 2024-09-11T01:45:02.311 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.269+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=226819758}) 2024-09-11T01:45:02.311 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.269+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.311 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.269+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2693218468}) 2024-09-11T01:45:02.311 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.269+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3919687422}) 2024-09-11T01:45:02.311 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.270+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.312 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.270+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1192875517}) 2024-09-11T01:45:02.312 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.270+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1864885286}) 2024-09-11T01:45:02.312 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.270+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.312 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.270+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1192875517}) 2024-09-11T01:45:02.312 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.270+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1210056129}) 2024-09-11T01:45:02.312 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.271+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.312 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.271+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1786148020}) 2024-09-11T01:45:02.312 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.271+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3705430252}) 2024-09-11T01:45:02.312 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.271+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.312 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.271+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1786148020}) 2024-09-11T01:45:02.312 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.271+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=595475154}) 2024-09-11T01:45:02.313 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.271+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.313 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.271+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4139953387}) 2024-09-11T01:45:02.313 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.271+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2734924784}) 2024-09-11T01:45:02.313 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.271+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.313 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.271+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4139953387}) 2024-09-11T01:45:02.313 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.271+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4076972690}) 2024-09-11T01:45:02.313 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.272+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.313 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.272+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3753198210}) 2024-09-11T01:45:02.313 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.272+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2871437098}) 2024-09-11T01:45:02.313 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.272+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.313 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.272+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3753198210}) 2024-09-11T01:45:02.313 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.272+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3405868539}) 2024-09-11T01:45:02.314 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.273+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.314 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.273+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1476989125}) 2024-09-11T01:45:02.314 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.273+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1148948815}) 2024-09-11T01:45:02.314 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.273+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.314 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.273+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1476989125}) 2024-09-11T01:45:02.314 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.273+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=72135594}) 2024-09-11T01:45:02.314 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.273+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.314 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.273+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1304839499}) 2024-09-11T01:45:02.314 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.273+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1537384447}) 2024-09-11T01:45:02.314 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.274+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.314 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.274+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1304839499}) 2024-09-11T01:45:02.314 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.274+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3400536889}) 2024-09-11T01:45:02.315 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.274+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.315 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.274+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3488706952}) 2024-09-11T01:45:02.315 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.274+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3057359347}) 2024-09-11T01:45:02.315 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.274+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.315 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.274+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3488706952}) 2024-09-11T01:45:02.315 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.274+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3823590586}) 2024-09-11T01:45:02.315 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.275+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.315 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.275+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1846714304}) 2024-09-11T01:45:02.315 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.275+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1792548301}) 2024-09-11T01:45:02.315 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.275+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.315 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.275+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1846714304}) 2024-09-11T01:45:02.315 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.275+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1763722166}) 2024-09-11T01:45:02.316 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.276+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.316 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.276+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=506828942}) 2024-09-11T01:45:02.316 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.276+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2651810402}) 2024-09-11T01:45:02.316 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.276+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.316 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.276+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=506828942}) 2024-09-11T01:45:02.317 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:45:02.317 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-11T01:45:02.317 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T01:45:02.317 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-11T01:45:02.317 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-11T01:45:02.318 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-11T01:45:02.318 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:45:02.319 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.276+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=134776981}) 2024-09-11T01:45:02.319 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.277+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.319 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.277+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=2827306145,mdsmap=3760429986,mgr=1944970001}) 2024-09-11T01:45:02.319 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.277+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=2085156806,mdsmap=3760429986,mgr=4133961934}) 2024-09-11T01:45:02.320 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.277+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.320 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.277+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=2827306145,mdsmap=3760429986,mgr=1944970001}) 2024-09-11T01:45:02.320 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.277+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=92,mdsmap=3,mgr=5} crc {logm=2591367348,mdsmap=3760429986,mgr=3525911568}) 2024-09-11T01:45:02.320 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.278+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.320 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.278+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=16,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=52} crc {mgr=1417461417,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=900638839,mon_config_key=272090419,monmap=2933275274,nvmeofgw=1596038039,osd_metadata=2942852742,osd_pg_creating=781936494,osd_snap=963953528}) 2024-09-11T01:45:02.320 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.278+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=53} crc {mgr=1417461417,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=3623370575,mon_config_key=2592700263,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=781936494,osd_snap=1055825974}) 2024-09-11T01:45:02.320 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.278+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.320 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.278+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=16,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=52} crc {mgr=1417461417,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=900638839,mon_config_key=272090419,monmap=2933275274,nvmeofgw=1596038039,osd_metadata=2942852742,osd_pg_creating=781936494,osd_snap=963953528}) 2024-09-11T01:45:02.320 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.278+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=2,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=55} crc {mgr=543305786,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=928370379,mon_config_key=3330371543,monmap=1324671969,nvmeofgw=1596038039,osd_metadata=2942852742,osd_pg_creating=781936494,osd_snap=3065908126}) 2024-09-11T01:45:02.328 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.326+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.328 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.326+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=56,osdmap=44} crc {osd_snap=3019302138,osdmap=3376513161}) 2024-09-11T01:45:02.328 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.326+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=53,osdmap=47} crc {osd_snap=1228798292,osdmap=3709602732}) 2024-09-11T01:45:02.328 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.326+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:02.328 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.326+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=56,osdmap=44} crc {osd_snap=3019302138,osdmap=3376513161}) 2024-09-11T01:45:02.328 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:02.326+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=52,osdmap=48} crc {osd_snap=2902264808,osdmap=884672845}) 2024-09-11T01:45:02.338 INFO:tasks.ceph.mon.c:Stopped 2024-09-11T01:45:02.338 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:45:02.339 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:45:02.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.477+0000 7ffabb665640 10 monclient: get_monmap_and_config 2024-09-11T01:45:02.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.477+0000 7ffabb665640 10 monclient: build_initial_monmap 2024-09-11T01:45:02.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.477+0000 7ffabb665640 10 monclient: monmap: 2024-09-11T01:45:02.479 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:02.479 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:02.479 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:45:02.478299+0000 2024-09-11T01:45:02.479 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:45:02.478299+0000 2024-09-11T01:45:02.479 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:02.479 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:02.480 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:45:02.480 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:45:02.480 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:45:02.480 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:45:02.480 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:45:02.480 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:02.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.478+0000 7ffabb665640 1 Processor -- start 2024-09-11T01:45:02.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.478+0000 7ffabb665640 1 -- start start 2024-09-11T01:45:02.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.478+0000 7ffabb665640 10 monclient: init 2024-09-11T01:45:02.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.479+0000 7ffabb665640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:02.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.479+0000 7ffabb665640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-11T01:45:02.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.479+0000 7ffabb665640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffab40a03a0 addr v1:172.21.15.12:6789/0 2024-09-11T01:45:02.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.479+0000 7ffabb665640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffab40a13a0 addr v1:172.21.15.71:6789/0 2024-09-11T01:45:02.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.479+0000 7ffabb665640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ffab40a17d0 0x7ffab413d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:02.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.479+0000 7ffabb665640 10 monclient(hunting): picked mon.noname-c con 0x7ffab40a17d0 addr v2:172.21.15.71:3300/0 2024-09-11T01:45:02.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.479+0000 7ffabb665640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffab409fa00 con 0x7ffab40a03a0 2024-09-11T01:45:02.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.479+0000 7ffab9401640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ffab40a03a0 0x7ffab40a0c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37762/0 (socket says 172.21.15.12:37762) 2024-09-11T01:45:02.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.479+0000 7ffab9401640 1 -- 172.21.15.12:0/1995264730 learned_addr learned my addr 172.21.15.12:0/1995264730 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:45:02.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.479+0000 7ffabb665640 1 -- 172.21.15.12:0/1995264730 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffab40a29b0 con 0x7ffab40a13a0 2024-09-11T01:45:02.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.479+0000 7ffab9c02640 1 --2- 172.21.15.12:0/1995264730 >> v2:172.21.15.71:3300/0 conn(0x7ffab40a17d0 0x7ffab413d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:02.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.479+0000 7ffabb665640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:02.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.479+0000 7ffabb665640 1 -- 172.21.15.12:0/1995264730 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7ffab40a0780 con 0x7ffab40a17d0 2024-09-11T01:45:02.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.479+0000 7ffabb665640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:02.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.479+0000 7ffabb665640 10 monclient(hunting): authenticate will time out at 1708.921503s 2024-09-11T01:45:02.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.479+0000 7ffab9c02640 10 monclient(hunting): get_auth_request con 0x7ffab40a17d0 auth_method 0 2024-09-11T01:45:02.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.479+0000 7ffab9c02640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:02.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.479+0000 7ffab9c02640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:02.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.479+0000 7ffab9c02640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:02.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.481+0000 7ffab9c02640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:02.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.481+0000 7ffab9c02640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:02.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.481+0000 7ffab8c00640 1 -- 172.21.15.12:0/1995264730 >> v1:172.21.15.71:6789/0 conn(0x7ffab40a13a0 legacy=0x7ffab4139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:02.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.481+0000 7ffab9c02640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:02.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.481+0000 7ffab9401640 1 -- 172.21.15.12:0/1995264730 >> v1:172.21.15.12:6789/0 conn(0x7ffab40a03a0 legacy=0x7ffab40a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:02.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.481+0000 7ffab9c02640 10 monclient(hunting): handle_auth_done global_id 134485 payload 274 2024-09-11T01:45:02.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.481+0000 7ffab9c02640 1 -- 172.21.15.12:0/1995264730 >> v1:172.21.15.71:6789/0 conn(0x7ffab40a13a0 legacy=0x7ffab4139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:02.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.482+0000 7ffab9c02640 1 -- 172.21.15.12:0/1995264730 >> v1:172.21.15.12:6789/0 conn(0x7ffab40a03a0 legacy=0x7ffab40a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:02.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.482+0000 7ffab9c02640 10 monclient: _finish_hunting 0 2024-09-11T01:45:02.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.482+0000 7ffab9c02640 1 monclient: found mon.noname-c 2024-09-11T01:45:02.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.482+0000 7ffab9c02640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:02.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.482+0000 7ffab9c02640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:45:02.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.482+0000 7ffab9c02640 1 -- 172.21.15.12:0/1995264730 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab409e510 con 0x7ffab40a17d0 2024-09-11T01:45:02.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.484+0000 7ffab9c02640 1 -- 172.21.15.12:0/1995264730 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffab40a17d0 msgr2=0x7ffab413d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:02.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.484+0000 7ffab9c02640 1 --2- 172.21.15.12:0/1995264730 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffab40a17d0 0x7ffab413d1c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009e90 tx=0x7ffaac02a6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f11c50c92f89fe8e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:02.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.485+0000 7ffaabfff640 1 -- 172.21.15.12:0/1995264730 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaac007a30 con 0x7ffab40a17d0 2024-09-11T01:45:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.485+0000 7ffaabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.485+0000 7ffaabfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:45:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.485+0000 7ffaabfff640 10 monclient: dump: 2024-09-11T01:45:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:02.487 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.485+0000 7ffaabfff640 10 monclient: _finish_auth 0 2024-09-11T01:45:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.485+0000 7ffaabfff640 10 monclient: _check_auth_tickets 2024-09-11T01:45:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.485+0000 7ffaabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.485+0000 7ffaabfff640 1 -- 172.21.15.12:0/1995264730 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaac007bd0 con 0x7ffab40a17d0 2024-09-11T01:45:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.485+0000 7ffaabfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.485+0000 7ffabb665640 5 monclient: authenticate success, global_id 134485 2024-09-11T01:45:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.485+0000 7ffaabfff640 1 -- 172.21.15.12:0/1995264730 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaac031400 con 0x7ffab40a17d0 2024-09-11T01:45:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.485+0000 7ffaabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.485+0000 7ffaabfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:45:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.485+0000 7ffaabfff640 10 monclient: dump: 2024-09-11T01:45:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:02.489 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.485+0000 7ffabb665640 10 monclient: get_monmap_and_config success 2024-09-11T01:45:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.485+0000 7ffabb665640 4 set_mon_vals no callback set 2024-09-11T01:45:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.485+0000 7ffabb665640 10 monclient: shutdown 2024-09-11T01:45:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.485+0000 7ffabb665640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.485+0000 7ffabb665640 1 -- 172.21.15.12:0/1995264730 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffab40a17d0 msgr2=0x7ffab413d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.485+0000 7ffabb665640 1 --2- 172.21.15.12:0/1995264730 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffab40a17d0 0x7ffab413d1c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009e90 tx=0x7ffaac02a6d0 comp rx=0 tx=0).stop 2024-09-11T01:45:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.485+0000 7ffabb665640 1 -- 172.21.15.12:0/1995264730 shutdown_connections 2024-09-11T01:45:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.485+0000 7ffabb665640 1 --2- 172.21.15.12:0/1995264730 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffab40a17d0 0x7ffab413d1c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.485+0000 7ffabb665640 1 -- 172.21.15.12:0/1995264730 >> 172.21.15.12:0/1995264730 conn(0x7ffab40a2d90 msgr2=0x7ffab40a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.486+0000 7ffabb665640 1 -- 172.21.15.12:0/1995264730 shutdown_connections 2024-09-11T01:45:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.486+0000 7ffabb665640 1 -- 172.21.15.12:0/1995264730 wait complete. 2024-09-11T01:45:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.486+0000 7ffabb665640 10 monclient: build_initial_monmap 2024-09-11T01:45:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.486+0000 7ffabb665640 10 monclient: monmap: 2024-09-11T01:45:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:45:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:45:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:45:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:45:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:45:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:45:02.496 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.486+0000 7ffabb665640 1 Processor -- start 2024-09-11T01:45:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffabb665640 1 -- start start 2024-09-11T01:45:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffabb665640 10 monclient: init 2024-09-11T01:45:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffabb665640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffabb665640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:45:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffabb665640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffab40a03a0 0x7ffab4121b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffabb665640 10 monclient(hunting): picked mon.noname-c con 0x7ffab40a03a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffabb665640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffab40a13a0 0x7ffab4122080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffabb665640 10 monclient(hunting): picked mon.noname-b con 0x7ffab40a13a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:45:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffabb665640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffab40a17d0 0x7ffab41225c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffabb665640 10 monclient(hunting): picked mon.noname-a con 0x7ffab40a17d0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:45:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffabb665640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffab9401640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffab40a03a0 msgr2=0x7ffab4121b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:45:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffabb665640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ffab409f1d0 con 0x7ffab40a17d0 2024-09-11T01:45:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffabb665640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffabb665640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ffab409f350 con 0x7ffab40a13a0 2024-09-11T01:45:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffabb665640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffabb665640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ffab4074ba0 con 0x7ffab40a03a0 2024-09-11T01:45:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffab9401640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffab40a03a0 0x7ffab4121b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffabb665640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffabb665640 10 monclient(hunting): authenticate will time out at 1708.929567s 2024-09-11T01:45:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffab9c02640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffab40a17d0 0x7ffab41225c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffab9c02640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffab40a17d0 0x7ffab41225c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44860/0 (socket says 172.21.15.12:44860) 2024-09-11T01:45:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffab9c02640 1 -- 172.21.15.12:0/1427701545 learned_addr learned my addr 172.21.15.12:0/1427701545 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffab9c02640 10 monclient(hunting): get_auth_request con 0x7ffab40a17d0 auth_method 0 2024-09-11T01:45:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffab9c02640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffab9c02640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffab9c02640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffab8c00640 1 --2- 172.21.15.12:0/1427701545 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffab40a13a0 0x7ffab4122080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffab9c02640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffab9c02640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffab9c02640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffab8c00640 10 monclient(hunting): get_auth_request con 0x7ffab40a13a0 auth_method 0 2024-09-11T01:45:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffab8c00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffab8c00640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffab8c00640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffab8c00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffab8c00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffab8c00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.487+0000 7ffab9c02640 10 monclient(hunting): handle_auth_done global_id 174150 payload 824 2024-09-11T01:45:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.488+0000 7ffab9c02640 1 -- 172.21.15.12:0/1427701545 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffab40a03a0 msgr2=0x7ffab4121b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:45:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.488+0000 7ffab9c02640 1 --2- 172.21.15.12:0/1427701545 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffab40a03a0 0x7ffab4121b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.488+0000 7ffab9c02640 1 -- 172.21.15.12:0/1427701545 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffab40a13a0 msgr2=0x7ffab4122080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.488+0000 7ffab9c02640 1 --2- 172.21.15.12:0/1427701545 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffab40a13a0 0x7ffab4122080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.488+0000 7ffab9c02640 10 monclient: _finish_hunting 0 2024-09-11T01:45:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.488+0000 7ffab9c02640 1 monclient: found mon.noname-a 2024-09-11T01:45:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.488+0000 7ffab9c02640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.488+0000 7ffab9c02640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:45:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.488+0000 7ffab9c02640 1 -- 172.21.15.12:0/1427701545 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab40a5dc0 con 0x7ffab40a17d0 2024-09-11T01:45:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.490+0000 7ffab9c02640 1 -- 172.21.15.12:0/1427701545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffab40a17d0 msgr2=0x7ffab41225c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.490+0000 7ffab9c02640 1 --2- 172.21.15.12:0/1427701545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffab40a17d0 0x7ffab41225c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009fb0 tx=0x7ffaac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.490+0000 7ffaa9ffb640 1 -- 172.21.15.12:0/1427701545 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaac040070 con 0x7ffab40a17d0 2024-09-11T01:45:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.490+0000 7ffaa9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.490+0000 7ffaa9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:45:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.490+0000 7ffaa9ffb640 10 monclient: dump: 2024-09-11T01:45:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:02.501 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:02.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.490+0000 7ffaa9ffb640 10 monclient: _finish_auth 0 2024-09-11T01:45:02.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.490+0000 7ffaa9ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:45:02.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.490+0000 7ffaa9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:02.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.490+0000 7ffabb665640 5 monclient: authenticate success, global_id 174150 2024-09-11T01:45:02.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.490+0000 7ffaa9ffb640 1 -- 172.21.15.12:0/1427701545 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaac045660 con 0x7ffab40a17d0 2024-09-11T01:45:02.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.490+0000 7ffabb665640 10 monclient: _renew_subs 2024-09-11T01:45:02.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.490+0000 7ffabb665640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:02.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.490+0000 7ffabb665640 1 -- 172.21.15.12:0/1427701545 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffab409d480 con 0x7ffab40a17d0 2024-09-11T01:45:02.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.490+0000 7ffaa9ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:02.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.490+0000 7ffaa9ffb640 1 -- 172.21.15.12:0/1427701545 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaac045a30 con 0x7ffab40a17d0 2024-09-11T01:45:02.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.490+0000 7ffaa9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:02.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.490+0000 7ffaa9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:02.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.490+0000 7ffaa9ffb640 10 monclient: dump: 2024-09-11T01:45:02.502 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:02.502 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:02.502 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.502 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:02.502 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:02.502 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:02.502 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:02.502 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:02.502 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:02.503 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:02.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.490+0000 7ffabb665640 10 monclient: _renew_subs 2024-09-11T01:45:02.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.490+0000 7ffaab7fe640 4 set_mon_vals no callback set 2024-09-11T01:45:02.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.490+0000 7ffabb665640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:02.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.490+0000 7ffabb665640 1 -- 172.21.15.12:0/1427701545 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffab4125680 con 0x7ffab40a17d0 2024-09-11T01:45:02.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.491+0000 7ffaa9ffb640 1 -- 172.21.15.12:0/1427701545 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ffaac007a40 con 0x7ffab40a17d0 2024-09-11T01:45:02.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.491+0000 7ffabb665640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:02.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.491+0000 7ffabb665640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:02.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.491+0000 7ffabb665640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:02.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.491+0000 7ffabb665640 1 -- 172.21.15.12:0/1427701545 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffab409d820 con 0x7ffab40a17d0 2024-09-11T01:45:02.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.491+0000 7ffaa9ffb640 1 --2- 172.21.15.12:0/1427701545 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffa9803e1c0 0x7ffa98040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:02.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.492+0000 7ffaa9ffb640 1 -- 172.21.15.12:0/1427701545 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(535..535 src has 317..535) ==== 4771+0+0 (secure 0 0 0) 0x7ffaac074920 con 0x7ffab40a17d0 2024-09-11T01:45:02.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.492+0000 7ffab9401640 1 --2- 172.21.15.12:0/1427701545 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffa9803e1c0 0x7ffa98040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:02.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.492+0000 7ffab9401640 10 monclient: get_auth_request con 0x7ffa9803e1c0 auth_method 0 2024-09-11T01:45:02.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.494+0000 7ffab9401640 1 --2- 172.21.15.12:0/1427701545 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffa9803e1c0 0x7ffa98040680 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ffab41240f0 tx=0x7ffab0007a60 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:02.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.496+0000 7ffaa9ffb640 1 -- 172.21.15.12:0/1427701545 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ffab409d820 con 0x7ffab40a17d0 2024-09-11T01:45:02.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.496+0000 7ffaa9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:02.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.496+0000 7ffaa9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:45:02.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.755+0000 7ffabb665640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:45:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.755+0000 7ffabb665640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:45:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.755+0000 7ffabb665640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.755+0000 7ffabb665640 1 -- 172.21.15.12:0/1427701545 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ffab40a0c90 con 0x7ffab40a17d0 2024-09-11T01:45:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.755+0000 7ffaa9ffb640 1 -- 172.21.15.12:0/1427701545 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7ffaac031400 con 0x7ffab40a17d0 2024-09-11T01:45:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.755+0000 7ffaa9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:45:02.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.755+0000 7ffaa9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:45:02.757 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":148,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:45:02.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.757+0000 7ffabb665640 1 -- 172.21.15.12:0/1427701545 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffa9803e1c0 msgr2=0x7ffa98040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:02.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.757+0000 7ffabb665640 1 --2- 172.21.15.12:0/1427701545 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffa9803e1c0 0x7ffa98040680 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ffab41240f0 tx=0x7ffab0007a60 comp rx=0 tx=0).stop 2024-09-11T01:45:02.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.758+0000 7ffabb665640 10 monclient: shutdown 2024-09-11T01:45:02.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.758+0000 7ffabb665640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:02.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.758+0000 7ffabb665640 1 -- 172.21.15.12:0/1427701545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffab40a17d0 msgr2=0x7ffab41225c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:02.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.758+0000 7ffabb665640 1 --2- 172.21.15.12:0/1427701545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffab40a17d0 0x7ffab41225c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009fb0 tx=0x7ffaac004a40 comp rx=0 tx=0).stop 2024-09-11T01:45:02.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.758+0000 7ffabb665640 1 -- 172.21.15.12:0/1427701545 shutdown_connections 2024-09-11T01:45:02.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.758+0000 7ffabb665640 1 --2- 172.21.15.12:0/1427701545 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffa9803e1c0 0x7ffa98040680 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:02.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.758+0000 7ffabb665640 1 --2- 172.21.15.12:0/1427701545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffab40a17d0 0x7ffab41225c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:02.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.758+0000 7ffabb665640 1 --2- 172.21.15.12:0/1427701545 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffab40a13a0 0x7ffab4122080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:02.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.758+0000 7ffabb665640 1 --2- 172.21.15.12:0/1427701545 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffab40a03a0 0x7ffab4121b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:02.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.758+0000 7ffabb665640 1 -- 172.21.15.12:0/1427701545 >> 172.21.15.12:0/1427701545 conn(0x7ffab40a2d90 msgr2=0x7ffab40a49b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:02.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.758+0000 7ffabb665640 1 -- 172.21.15.12:0/1427701545 shutdown_connections 2024-09-11T01:45:02.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:02.758+0000 7ffabb665640 1 -- 172.21.15.12:0/1427701545 wait complete. 2024-09-11T01:45:05.771 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:45:05.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.886+0000 7fceb559b640 10 monclient: get_monmap_and_config 2024-09-11T01:45:05.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.886+0000 7fceb559b640 10 monclient: build_initial_monmap 2024-09-11T01:45:05.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.886+0000 7fceb559b640 10 monclient: monmap: 2024-09-11T01:45:05.888 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:05.888 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:05.888 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:45:05.887007+0000 2024-09-11T01:45:05.889 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:45:05.887007+0000 2024-09-11T01:45:05.889 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:05.889 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:05.889 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:45:05.889 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:45:05.889 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:45:05.889 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:45:05.889 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:45:05.889 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:05.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.886+0000 7fceb559b640 1 Processor -- start 2024-09-11T01:45:05.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.886+0000 7fceb559b640 1 -- start start 2024-09-11T01:45:05.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.886+0000 7fceb559b640 10 monclient: init 2024-09-11T01:45:05.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceb559b640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:05.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceb559b640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-11T01:45:05.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceb559b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb0138210 0x7fceb01385f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:05.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceb559b640 10 monclient(hunting): picked mon.noname-b con 0x7fceb0138210 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:05.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceb559b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fceb00a6810 addr v1:172.21.15.71:6789/0 2024-09-11T01:45:05.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceb559b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fceb00a57a0 0x7fceb0140120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:05.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceb559b640 10 monclient(hunting): picked mon.noname-a con 0x7fceb00a57a0 addr v2:172.21.15.12:3300/0 2024-09-11T01:45:05.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceb559b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fceb009ba20 con 0x7fceb00a6810 2024-09-11T01:45:05.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceb559b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:05.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceb559b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fceb009c7a0 con 0x7fceb00a57a0 2024-09-11T01:45:05.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceb559b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:05.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceb559b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fceb009c4c0 con 0x7fceb0138210 2024-09-11T01:45:05.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceb559b640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:05.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceb559b640 10 monclient(hunting): authenticate will time out at 1712.329534s 2024-09-11T01:45:05.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceaeffd640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb0138210 msgr2=0x7fceb01385f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:45:05.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceaeffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb0138210 0x7fceb01385f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:05.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceaf7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fceb00a57a0 0x7fceb0140120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:05.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceaf7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fceb00a57a0 0x7fceb0140120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44872/0 (socket says 172.21.15.12:44872) 2024-09-11T01:45:05.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceaf7fe640 1 -- 172.21.15.12:0/2914460569 learned_addr learned my addr 172.21.15.12:0/2914460569 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:05.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceaf7fe640 10 monclient(hunting): get_auth_request con 0x7fceb00a57a0 auth_method 0 2024-09-11T01:45:05.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceaf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:05.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceaf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:05.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:05.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceaf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:05.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceaf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:05.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceaf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:05.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceaf7fe640 10 monclient(hunting): handle_auth_done global_id 174153 payload 274 2024-09-11T01:45:05.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceaf7fe640 1 -- 172.21.15.12:0/2914460569 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb0138210 msgr2=0x7fceb01385f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:45:05.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceaf7fe640 1 --2- 172.21.15.12:0/2914460569 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb0138210 0x7fceb01385f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:05.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceaf7fe640 1 -- 172.21.15.12:0/2914460569 >> v1:172.21.15.71:6789/0 conn(0x7fceb00a6810 legacy=0x7fceb00a5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:05.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceaf7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:45:05.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.887+0000 7fceaf7fe640 1 monclient: found mon.noname-a 2024-09-11T01:45:05.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.888+0000 7fceaf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:05.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.888+0000 7fceaf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:45:05.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.888+0000 7fceaf7fe640 1 -- 172.21.15.12:0/2914460569 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceb00a6590 con 0x7fceb00a57a0 2024-09-11T01:45:05.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.890+0000 7fceaf7fe640 1 -- 172.21.15.12:0/2914460569 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceb00a57a0 msgr2=0x7fceb0140120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:05.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.890+0000 7fceaf7fe640 1 --2- 172.21.15.12:0/2914460569 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceb00a57a0 0x7fceb0140120 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcea8009870 tx=0x7fcea802a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d34640e685881f55 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:05.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.890+0000 7fceadffb640 1 -- 172.21.15.12:0/2914460569 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea8037070 con 0x7fceb00a57a0 2024-09-11T01:45:05.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.891+0000 7fceadffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:05.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.891+0000 7fceadffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:45:05.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.891+0000 7fceadffb640 10 monclient: dump: 2024-09-11T01:45:05.902 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:05.902 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:05.902 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:05.903 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:05.903 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:05.903 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:05.903 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:05.903 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:05.903 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:05.903 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:05.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.891+0000 7fceadffb640 10 monclient: _finish_auth 0 2024-09-11T01:45:05.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.891+0000 7fceadffb640 10 monclient: _check_auth_tickets 2024-09-11T01:45:05.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.891+0000 7fceadffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:05.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.891+0000 7fceadffb640 1 -- 172.21.15.12:0/2914460569 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcea8032640 con 0x7fceb00a57a0 2024-09-11T01:45:05.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.891+0000 7fceb559b640 5 monclient: authenticate success, global_id 174153 2024-09-11T01:45:05.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.891+0000 7fceb559b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:45:05.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.891+0000 7fceadffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:05.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.891+0000 7fceadffb640 1 -- 172.21.15.12:0/2914460569 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea8032a10 con 0x7fceb00a57a0 2024-09-11T01:45:05.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.891+0000 7fceb559b640 10 monclient: get_monmap_and_config success 2024-09-11T01:45:05.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.891+0000 7fceb559b640 4 set_mon_vals no callback set 2024-09-11T01:45:05.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.891+0000 7fceadffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:05.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.891+0000 7fceadffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:05.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.891+0000 7fceadffb640 10 monclient: dump: 2024-09-11T01:45:05.904 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:05.905 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:05.905 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:05.905 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:05.905 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:05.905 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:05.905 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:05.905 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:05.905 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:05.905 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:05.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.891+0000 7fceb559b640 10 monclient: shutdown 2024-09-11T01:45:05.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.891+0000 7fceb559b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:05.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.891+0000 7fceb559b640 1 -- 172.21.15.12:0/2914460569 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceb00a57a0 msgr2=0x7fceb0140120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:05.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.891+0000 7fceb559b640 1 --2- 172.21.15.12:0/2914460569 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceb00a57a0 0x7fceb0140120 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcea8009870 tx=0x7fcea802a0d0 comp rx=0 tx=0).stop 2024-09-11T01:45:05.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.891+0000 7fceb559b640 1 -- 172.21.15.12:0/2914460569 shutdown_connections 2024-09-11T01:45:05.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.891+0000 7fceb559b640 1 --2- 172.21.15.12:0/2914460569 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceb00a57a0 0x7fceb0140120 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:05.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.891+0000 7fceb559b640 1 --2- 172.21.15.12:0/2914460569 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb0138210 0x7fceb01385f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:05.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.891+0000 7fceb559b640 1 -- 172.21.15.12:0/2914460569 >> 172.21.15.12:0/2914460569 conn(0x7fceb00aaa80 msgr2=0x7fceb00aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:05.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.891+0000 7fceb559b640 1 -- 172.21.15.12:0/2914460569 shutdown_connections 2024-09-11T01:45:05.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.891+0000 7fceb559b640 1 -- 172.21.15.12:0/2914460569 wait complete. 2024-09-11T01:45:05.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.892+0000 7fceb559b640 10 monclient: build_initial_monmap 2024-09-11T01:45:05.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.892+0000 7fceb559b640 10 monclient: monmap: 2024-09-11T01:45:05.906 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:05.907 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:45:05.907 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:45:05.907 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:45:05.907 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:05.907 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:05.907 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:45:05.907 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:45:05.907 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:45:05.907 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:05.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.892+0000 7fceb559b640 1 Processor -- start 2024-09-11T01:45:05.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.892+0000 7fceb559b640 1 -- start start 2024-09-11T01:45:05.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.892+0000 7fceb559b640 10 monclient: init 2024-09-11T01:45:05.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceb559b640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:05.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceb559b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:45:05.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceb559b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb00a57a0 0x7fceb01254b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:05.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceb559b640 10 monclient(hunting): picked mon.noname-c con 0x7fceb00a57a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:05.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceb559b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fceb00a6810 0x7fceb01259f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:05.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceb559b640 10 monclient(hunting): picked mon.noname-b con 0x7fceb00a6810 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:45:05.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceb559b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceb0138210 0x7fceb0125f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:05.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceb559b640 10 monclient(hunting): picked mon.noname-a con 0x7fceb0138210 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:45:05.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceb559b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:05.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceb559b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fceb006f670 con 0x7fceb0138210 2024-09-11T01:45:05.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceb559b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:05.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceb559b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fceb006f4f0 con 0x7fceb00a6810 2024-09-11T01:45:05.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceb559b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:05.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceb559b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fceb0138b30 con 0x7fceb00a57a0 2024-09-11T01:45:05.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceb559b640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:05.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceb559b640 10 monclient(hunting): authenticate will time out at 1712.335196s 2024-09-11T01:45:05.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceaeffd640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb00a57a0 msgr2=0x7fceb01254b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:45:05.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceaeffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb00a57a0 0x7fceb01254b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:05.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceaf7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceb0138210 0x7fceb0125f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:05.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceaf7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceb0138210 0x7fceb0125f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44884/0 (socket says 172.21.15.12:44884) 2024-09-11T01:45:05.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceaf7fe640 1 -- 172.21.15.12:0/3764011045 learned_addr learned my addr 172.21.15.12:0/3764011045 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:05.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceaf7fe640 10 monclient(hunting): get_auth_request con 0x7fceb0138210 auth_method 0 2024-09-11T01:45:05.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceaf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:05.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceaf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:05.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:05.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceaf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:05.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceaf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:05.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceaf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:05.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceae7fc640 1 --2- 172.21.15.12:0/3764011045 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fceb00a6810 0x7fceb01259f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:05.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceae7fc640 10 monclient(hunting): get_auth_request con 0x7fceb00a6810 auth_method 0 2024-09-11T01:45:05.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:05.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:05.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:05.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:05.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:05.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:05.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceaf7fe640 10 monclient(hunting): handle_auth_done global_id 174156 payload 824 2024-09-11T01:45:05.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceaf7fe640 1 -- 172.21.15.12:0/3764011045 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb00a57a0 msgr2=0x7fceb01254b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:45:05.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceaf7fe640 1 --2- 172.21.15.12:0/3764011045 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb00a57a0 0x7fceb01254b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:05.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceaf7fe640 1 -- 172.21.15.12:0/3764011045 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fceb00a6810 msgr2=0x7fceb01259f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:05.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceaf7fe640 1 --2- 172.21.15.12:0/3764011045 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fceb00a6810 0x7fceb01259f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:05.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceaf7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:45:05.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceaf7fe640 1 monclient: found mon.noname-a 2024-09-11T01:45:05.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceaf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:05.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceaf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:45:05.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.893+0000 7fceaf7fe640 1 -- 172.21.15.12:0/3764011045 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceb0127c30 con 0x7fceb0138210 2024-09-11T01:45:05.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.895+0000 7fceaf7fe640 1 -- 172.21.15.12:0/3764011045 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceb0138210 msgr2=0x7fceb0125f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:05.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.895+0000 7fceaf7fe640 1 --2- 172.21.15.12:0/3764011045 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceb0138210 0x7fceb0125f30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcea8035040 tx=0x7fcea80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:05.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.896+0000 7fce977fe640 1 -- 172.21.15.12:0/3764011045 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea8041070 con 0x7fceb0138210 2024-09-11T01:45:05.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.896+0000 7fce977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:05.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.896+0000 7fce977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:45:05.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.896+0000 7fce977fe640 10 monclient: dump: 2024-09-11T01:45:05.912 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:05.912 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:05.912 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:05.912 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:05.912 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:05.912 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:05.912 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:05.913 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:05.913 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:05.913 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:05.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.896+0000 7fce977fe640 10 monclient: _finish_auth 0 2024-09-11T01:45:05.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.896+0000 7fce977fe640 10 monclient: _check_auth_tickets 2024-09-11T01:45:05.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.896+0000 7fce977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:05.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.896+0000 7fce977fe640 1 -- 172.21.15.12:0/3764011045 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcea803c880 con 0x7fceb0138210 2024-09-11T01:45:05.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.896+0000 7fceb559b640 5 monclient: authenticate success, global_id 174156 2024-09-11T01:45:05.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.896+0000 7fceb559b640 10 monclient: _renew_subs 2024-09-11T01:45:05.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.896+0000 7fceb559b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:05.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.896+0000 7fceb559b640 1 -- 172.21.15.12:0/3764011045 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fceb0128d80 con 0x7fceb0138210 2024-09-11T01:45:05.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.896+0000 7fce977fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:05.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.896+0000 7fce977fe640 1 -- 172.21.15.12:0/3764011045 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea8037040 con 0x7fceb0138210 2024-09-11T01:45:05.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.896+0000 7fce977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:05.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.896+0000 7fce977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:05.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.896+0000 7fce977fe640 10 monclient: dump: 2024-09-11T01:45:05.914 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:05.914 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:05.914 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:05.914 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:05.914 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:05.914 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:05.914 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:05.914 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:05.915 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:05.915 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:05.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.896+0000 7fcead7fa640 4 set_mon_vals no callback set 2024-09-11T01:45:05.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.896+0000 7fceb559b640 10 monclient: _renew_subs 2024-09-11T01:45:05.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.896+0000 7fceb559b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:05.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.896+0000 7fceb559b640 1 -- 172.21.15.12:0/3764011045 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fceb00a7ae0 con 0x7fceb0138210 2024-09-11T01:45:05.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.897+0000 7fceb559b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:05.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.897+0000 7fceb559b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:05.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.897+0000 7fceb559b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:05.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.897+0000 7fceb559b640 1 -- 172.21.15.12:0/3764011045 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fceb0126d00 con 0x7fceb0138210 2024-09-11T01:45:05.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.897+0000 7fce977fe640 1 -- 172.21.15.12:0/3764011045 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fcea803cc50 con 0x7fceb0138210 2024-09-11T01:45:05.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.897+0000 7fce977fe640 1 --2- 172.21.15.12:0/3764011045 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fce9c03e1c0 0x7fce9c040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:05.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.897+0000 7fce977fe640 1 -- 172.21.15.12:0/3764011045 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(535..535 src has 317..535) ==== 4771+0+0 (secure 0 0 0) 0x7fcea8073460 con 0x7fceb0138210 2024-09-11T01:45:05.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.898+0000 7fceaeffd640 1 --2- 172.21.15.12:0/3764011045 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fce9c03e1c0 0x7fce9c040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:05.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.898+0000 7fceaeffd640 10 monclient: get_auth_request con 0x7fce9c03e1c0 auth_method 0 2024-09-11T01:45:05.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.900+0000 7fceaeffd640 1 --2- 172.21.15.12:0/3764011045 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fce9c03e1c0 0x7fce9c040680 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fcea0003d00 tx=0x7fcea0009270 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:05.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.902+0000 7fce977fe640 1 -- 172.21.15.12:0/3764011045 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fcea804b070 con 0x7fceb0138210 2024-09-11T01:45:05.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.902+0000 7fce977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:05.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:05.902+0000 7fce977fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:45:06.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:06.161+0000 7fceb559b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:45:06.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:06.161+0000 7fceb559b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:45:06.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:06.161+0000 7fceb559b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:06.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:06.161+0000 7fceb559b640 1 -- 172.21.15.12:0/3764011045 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fceb00a10a0 con 0x7fceb0138210 2024-09-11T01:45:06.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:06.161+0000 7fce977fe640 1 -- 172.21.15.12:0/3764011045 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fceb009ba20 con 0x7fceb0138210 2024-09-11T01:45:06.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:06.161+0000 7fce977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:45:06.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:06.161+0000 7fce977fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:45:06.163 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":148,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:45:06.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:06.163+0000 7fceb559b640 1 -- 172.21.15.12:0/3764011045 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fce9c03e1c0 msgr2=0x7fce9c040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:06.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:06.163+0000 7fceb559b640 1 --2- 172.21.15.12:0/3764011045 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fce9c03e1c0 0x7fce9c040680 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fcea0003d00 tx=0x7fcea0009270 comp rx=0 tx=0).stop 2024-09-11T01:45:06.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:06.163+0000 7fceb559b640 10 monclient: shutdown 2024-09-11T01:45:06.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:06.163+0000 7fceb559b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:06.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:06.163+0000 7fceb559b640 1 -- 172.21.15.12:0/3764011045 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceb0138210 msgr2=0x7fceb0125f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:06.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:06.163+0000 7fceb559b640 1 --2- 172.21.15.12:0/3764011045 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceb0138210 0x7fceb0125f30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcea8035040 tx=0x7fcea80046c0 comp rx=0 tx=0).stop 2024-09-11T01:45:06.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:06.164+0000 7fceb559b640 1 -- 172.21.15.12:0/3764011045 shutdown_connections 2024-09-11T01:45:06.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:06.164+0000 7fceb559b640 1 --2- 172.21.15.12:0/3764011045 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fce9c03e1c0 0x7fce9c040680 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:06.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:06.164+0000 7fceb559b640 1 --2- 172.21.15.12:0/3764011045 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fceb0138210 0x7fceb0125f30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:06.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:06.164+0000 7fceb559b640 1 --2- 172.21.15.12:0/3764011045 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fceb00a6810 0x7fceb01259f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:06.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:06.164+0000 7fceb559b640 1 --2- 172.21.15.12:0/3764011045 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fceb00a57a0 0x7fceb01254b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:06.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:06.164+0000 7fceb559b640 1 -- 172.21.15.12:0/3764011045 >> 172.21.15.12:0/3764011045 conn(0x7fceb00aaa80 msgr2=0x7fceb00a9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:06.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:06.164+0000 7fceb559b640 1 -- 172.21.15.12:0/3764011045 shutdown_connections 2024-09-11T01:45:06.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:06.164+0000 7fceb559b640 1 -- 172.21.15.12:0/3764011045 wait complete. 2024-09-11T01:45:09.177 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:45:09.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.288+0000 7f409683c640 10 monclient: get_monmap_and_config 2024-09-11T01:45:09.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.288+0000 7f409683c640 10 monclient: build_initial_monmap 2024-09-11T01:45:09.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.288+0000 7f409683c640 10 monclient: monmap: 2024-09-11T01:45:09.289 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:09.290 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:09.290 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:45:09.288913+0000 2024-09-11T01:45:09.290 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:45:09.288913+0000 2024-09-11T01:45:09.290 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:09.290 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:09.290 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:45:09.290 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:45:09.290 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:45:09.290 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:45:09.290 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:45:09.290 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:09.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.288+0000 7f409683c640 1 Processor -- start 2024-09-11T01:45:09.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f409683c640 1 -- start start 2024-09-11T01:45:09.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f409683c640 10 monclient: init 2024-09-11T01:45:09.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f409683c640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:09.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f409683c640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-11T01:45:09.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f409683c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f40900a2e90 0x7f409006dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:09.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f409683c640 10 monclient(hunting): picked mon.noname-a con 0x7f40900a2e90 addr v2:172.21.15.12:3300/0 2024-09-11T01:45:09.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f409683c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f40900a3270 0x7f40900a3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:09.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f409683c640 10 monclient(hunting): picked mon.noname-c con 0x7f40900a3270 addr v2:172.21.15.71:3300/0 2024-09-11T01:45:09.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f409683c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4090138040 addr v1:172.21.15.71:6789/0 2024-09-11T01:45:09.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f409683c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40900a46c0 con 0x7f4090138040 2024-09-11T01:45:09.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f409683c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:09.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f409683c640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f40900a4900 con 0x7f40900a2e90 2024-09-11T01:45:09.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f409683c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:09.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f409683c640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f409006ee50 con 0x7f40900a3270 2024-09-11T01:45:09.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f409683c640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:09.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f409683c640 10 monclient(hunting): authenticate will time out at 1715.731612s 2024-09-11T01:45:09.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f408ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f40900a2e90 0x7f409006dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:09.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f408ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f40900a2e90 0x7f409006dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34464/0 (socket says 172.21.15.12:34464) 2024-09-11T01:45:09.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f408ffff640 1 -- 172.21.15.12:0/4185177131 learned_addr learned my addr 172.21.15.12:0/4185177131 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:09.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f408ffff640 10 monclient(hunting): get_auth_request con 0x7f40900a2e90 auth_method 0 2024-09-11T01:45:09.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f408ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:09.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f408ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:09.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f408ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:09.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f408ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:09.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f408ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:09.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f408ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:09.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f408f7fe640 1 --2- 172.21.15.12:0/4185177131 >> v2:172.21.15.71:3300/0 conn(0x7f40900a3270 0x7f40900a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:09.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f408f7fe640 10 monclient(hunting): get_auth_request con 0x7f40900a3270 auth_method 0 2024-09-11T01:45:09.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f408f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:09.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f408f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:09.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.289+0000 7f408f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:09.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.290+0000 7f408ffff640 10 monclient(hunting): handle_auth_done global_id 174159 payload 274 2024-09-11T01:45:09.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.290+0000 7f408ffff640 1 -- 172.21.15.12:0/4185177131 >> v2:172.21.15.71:3300/0 conn(0x7f40900a3270 msgr2=0x7f40900a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:09.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.290+0000 7f408ffff640 1 --2- 172.21.15.12:0/4185177131 >> v2:172.21.15.71:3300/0 conn(0x7f40900a3270 0x7f40900a3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:09.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.290+0000 7f408ffff640 1 -- 172.21.15.12:0/4185177131 >> v1:172.21.15.71:6789/0 conn(0x7f4090138040 legacy=0x7f40901384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:09.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.290+0000 7f408ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:45:09.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.290+0000 7f408ffff640 1 monclient: found mon.noname-a 2024-09-11T01:45:09.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.290+0000 7f408ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:09.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.290+0000 7f408ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:45:09.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.290+0000 7f408ffff640 1 -- 172.21.15.12:0/4185177131 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409006eb70 con 0x7f40900a2e90 2024-09-11T01:45:09.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.290+0000 7f408f7fe640 1 --2- 172.21.15.12:0/4185177131 >> v2:172.21.15.71:3300/0 conn(0x7f40900a3270 0x7f40900a3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:45:09.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.292+0000 7f408ffff640 1 -- 172.21.15.12:0/4185177131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40900a2e90 msgr2=0x7f409006dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:09.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.292+0000 7f408ffff640 1 --2- 172.21.15.12:0/4185177131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40900a2e90 0x7f409006dc60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4090071820 tx=0x7f408002a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94a317f16c67e451 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:09.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f408effd640 1 -- 172.21.15.12:0/4185177131 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4080037070 con 0x7f40900a2e90 2024-09-11T01:45:09.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f408effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:09.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f408effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:45:09.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f408effd640 10 monclient: dump: 2024-09-11T01:45:09.302 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:09.302 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:09.302 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:09.302 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:09.302 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:09.302 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:09.302 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:09.302 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:09.302 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:09.302 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:09.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f408effd640 10 monclient: _finish_auth 0 2024-09-11T01:45:09.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f408effd640 10 monclient: _check_auth_tickets 2024-09-11T01:45:09.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f408effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:09.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f409683c640 5 monclient: authenticate success, global_id 174159 2024-09-11T01:45:09.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f409683c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:45:09.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f408effd640 1 -- 172.21.15.12:0/4185177131 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4080032640 con 0x7f40900a2e90 2024-09-11T01:45:09.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f408effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:09.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f409683c640 10 monclient: get_monmap_and_config success 2024-09-11T01:45:09.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f409683c640 4 set_mon_vals no callback set 2024-09-11T01:45:09.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f408effd640 1 -- 172.21.15.12:0/4185177131 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4080032a10 con 0x7f40900a2e90 2024-09-11T01:45:09.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f408effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:09.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f408effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:09.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f408effd640 10 monclient: dump: 2024-09-11T01:45:09.303 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:09.304 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:09.304 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:09.304 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:09.304 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:09.304 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:09.304 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:09.304 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:09.304 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:09.304 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:09.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f409683c640 10 monclient: shutdown 2024-09-11T01:45:09.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f409683c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:09.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f409683c640 1 -- 172.21.15.12:0/4185177131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40900a2e90 msgr2=0x7f409006dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:09.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f409683c640 1 --2- 172.21.15.12:0/4185177131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40900a2e90 0x7f409006dc60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4090071820 tx=0x7f408002a0b0 comp rx=0 tx=0).stop 2024-09-11T01:45:09.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f409683c640 1 -- 172.21.15.12:0/4185177131 shutdown_connections 2024-09-11T01:45:09.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f409683c640 1 --2- 172.21.15.12:0/4185177131 >> v2:172.21.15.71:3300/0 conn(0x7f40900a3270 0x7f40900a3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:09.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f409683c640 1 --2- 172.21.15.12:0/4185177131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f40900a2e90 0x7f409006dc60 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:09.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f409683c640 1 -- 172.21.15.12:0/4185177131 >> 172.21.15.12:0/4185177131 conn(0x7f40900a8490 msgr2=0x7f40900a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:09.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f409683c640 1 -- 172.21.15.12:0/4185177131 shutdown_connections 2024-09-11T01:45:09.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.293+0000 7f409683c640 1 -- 172.21.15.12:0/4185177131 wait complete. 2024-09-11T01:45:09.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.294+0000 7f409683c640 10 monclient: build_initial_monmap 2024-09-11T01:45:09.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.294+0000 7f409683c640 10 monclient: monmap: 2024-09-11T01:45:09.305 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:09.305 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:45:09.305 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:45:09.305 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:45:09.305 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:09.306 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:09.306 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:45:09.306 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:45:09.306 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:45:09.306 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:09.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.294+0000 7f409683c640 1 Processor -- start 2024-09-11T01:45:09.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.294+0000 7f409683c640 1 -- start start 2024-09-11T01:45:09.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.294+0000 7f409683c640 10 monclient: init 2024-09-11T01:45:09.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f409683c640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:09.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f409683c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:45:09.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f409683c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f40900a2e90 0x7f40901247f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:09.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f409683c640 10 monclient(hunting): picked mon.noname-c con 0x7f40900a2e90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:09.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f409683c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f40900a3270 0x7f4090124d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:09.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f409683c640 10 monclient(hunting): picked mon.noname-b con 0x7f40900a3270 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:45:09.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f409683c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4090138040 0x7f4090125270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:09.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f409683c640 10 monclient(hunting): picked mon.noname-a con 0x7f4090138040 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:45:09.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f409683c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:09.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f409683c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f409006fba0 con 0x7f4090138040 2024-09-11T01:45:09.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f409683c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:09.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f409683c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f40900a3bc0 con 0x7f40900a3270 2024-09-11T01:45:09.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f409683c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:09.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f409683c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f409006fd20 con 0x7f40900a2e90 2024-09-11T01:45:09.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f409683c640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:09.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f409683c640 10 monclient(hunting): authenticate will time out at 1715.737426s 2024-09-11T01:45:09.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f408ffff640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f40900a2e90 msgr2=0x7f40901247f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:45:09.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f408ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f40900a2e90 0x7f40901247f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:09.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f4094dd9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4090138040 0x7f4090125270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:09.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f4094dd9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4090138040 0x7f4090125270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34480/0 (socket says 172.21.15.12:34480) 2024-09-11T01:45:09.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f4094dd9640 1 -- 172.21.15.12:0/479538539 learned_addr learned my addr 172.21.15.12:0/479538539 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:09.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f4094dd9640 10 monclient(hunting): get_auth_request con 0x7f4090138040 auth_method 0 2024-09-11T01:45:09.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f4094dd9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:09.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f4094dd9640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:09.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f4094dd9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:09.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f408f7fe640 1 --2- 172.21.15.12:0/479538539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f40900a3270 0x7f4090124d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:09.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f4094dd9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:09.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f4094dd9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:09.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f4094dd9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:09.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f408f7fe640 10 monclient(hunting): get_auth_request con 0x7f40900a3270 auth_method 0 2024-09-11T01:45:09.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f408f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:09.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f408f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:09.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f408f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:09.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f408f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:09.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f408f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:09.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f408f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:09.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f4094dd9640 10 monclient(hunting): handle_auth_done global_id 174162 payload 824 2024-09-11T01:45:09.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f4094dd9640 1 -- 172.21.15.12:0/479538539 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f40900a2e90 msgr2=0x7f40901247f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:45:09.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f4094dd9640 1 --2- 172.21.15.12:0/479538539 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f40900a2e90 0x7f40901247f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:09.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f4094dd9640 1 -- 172.21.15.12:0/479538539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f40900a3270 msgr2=0x7f4090124d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:09.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f4094dd9640 1 --2- 172.21.15.12:0/479538539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f40900a3270 0x7f4090124d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:09.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f4094dd9640 10 monclient: _finish_hunting 0 2024-09-11T01:45:09.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f4094dd9640 1 monclient: found mon.noname-a 2024-09-11T01:45:09.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f4094dd9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:09.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f4094dd9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:45:09.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.295+0000 7f4094dd9640 1 -- 172.21.15.12:0/479538539 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40901282e0 con 0x7f4090138040 2024-09-11T01:45:09.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.298+0000 7f4094dd9640 1 -- 172.21.15.12:0/479538539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4090138040 msgr2=0x7f4090125270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:09.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.298+0000 7f4094dd9640 1 --2- 172.21.15.12:0/479538539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4090138040 0x7f4090125270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f408800c780 tx=0x7f408800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:09.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.298+0000 7f408cff9640 1 -- 172.21.15.12:0/479538539 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408800ac60 con 0x7f4090138040 2024-09-11T01:45:09.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.298+0000 7f408cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:09.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.298+0000 7f408cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:45:09.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.298+0000 7f408cff9640 10 monclient: dump: 2024-09-11T01:45:09.310 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:09.310 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:09.310 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:09.310 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:09.310 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:09.311 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:09.311 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:09.311 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:09.311 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:09.311 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:09.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.298+0000 7f408cff9640 10 monclient: _finish_auth 0 2024-09-11T01:45:09.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.298+0000 7f408cff9640 10 monclient: _check_auth_tickets 2024-09-11T01:45:09.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.298+0000 7f408cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:09.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.298+0000 7f408cff9640 1 -- 172.21.15.12:0/479538539 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f408800ae00 con 0x7f4090138040 2024-09-11T01:45:09.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.298+0000 7f409683c640 5 monclient: authenticate success, global_id 174162 2024-09-11T01:45:09.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.298+0000 7f409683c640 10 monclient: _renew_subs 2024-09-11T01:45:09.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.298+0000 7f409683c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:09.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.298+0000 7f409683c640 1 -- 172.21.15.12:0/479538539 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4090128a70 con 0x7f4090138040 2024-09-11T01:45:09.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.298+0000 7f408cff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:09.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.298+0000 7f408cff9640 1 -- 172.21.15.12:0/479538539 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4088004ba0 con 0x7f4090138040 2024-09-11T01:45:09.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.298+0000 7f408cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:09.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.298+0000 7f408cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:09.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.298+0000 7f408cff9640 10 monclient: dump: 2024-09-11T01:45:09.312 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:09.312 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:09.312 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:09.312 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:09.312 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:09.312 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:09.312 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:09.312 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:09.313 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:09.313 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:09.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.298+0000 7f409683c640 10 monclient: _renew_subs 2024-09-11T01:45:09.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.298+0000 7f409683c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:09.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.298+0000 7f408e7fc640 4 set_mon_vals no callback set 2024-09-11T01:45:09.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.298+0000 7f409683c640 1 -- 172.21.15.12:0/479538539 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f409009f680 con 0x7f4090138040 2024-09-11T01:45:09.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.299+0000 7f408cff9640 1 -- 172.21.15.12:0/479538539 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f4088023060 con 0x7f4090138040 2024-09-11T01:45:09.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.299+0000 7f409683c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:09.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.299+0000 7f409683c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:09.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.299+0000 7f409683c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:09.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.299+0000 7f409683c640 1 -- 172.21.15.12:0/479538539 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f409006dc60 con 0x7f4090138040 2024-09-11T01:45:09.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.299+0000 7f408cff9640 1 --2- 172.21.15.12:0/479538539 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f407c03e210 0x7f407c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:09.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.299+0000 7f408cff9640 1 -- 172.21.15.12:0/479538539 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(535..535 src has 317..535) ==== 4771+0+0 (secure 0 0 0) 0x7f4088052ef0 con 0x7f4090138040 2024-09-11T01:45:09.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.299+0000 7f408ffff640 1 --2- 172.21.15.12:0/479538539 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f407c03e210 0x7f407c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:09.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.300+0000 7f408ffff640 10 monclient: get_auth_request con 0x7f407c03e210 auth_method 0 2024-09-11T01:45:09.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.302+0000 7f408ffff640 1 --2- 172.21.15.12:0/479538539 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f407c03e210 0x7f407c0406d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f40900a94a0 tx=0x7f408002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:09.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.304+0000 7f408cff9640 1 -- 172.21.15.12:0/479538539 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f40880518f0 con 0x7f4090138040 2024-09-11T01:45:09.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.304+0000 7f408cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:09.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.304+0000 7f408cff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:45:09.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.558+0000 7f409683c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:45:09.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.558+0000 7f409683c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:45:09.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.558+0000 7f409683c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:09.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.558+0000 7f409683c640 1 -- 172.21.15.12:0/479538539 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4090127750 con 0x7f4090138040 2024-09-11T01:45:09.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.558+0000 7f408cff9640 1 -- 172.21.15.12:0/479538539 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7f4088012420 con 0x7f4090138040 2024-09-11T01:45:09.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.559+0000 7f408cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:45:09.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.559+0000 7f408cff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:45:09.561 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":148,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:45:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.561+0000 7f409683c640 1 -- 172.21.15.12:0/479538539 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f407c03e210 msgr2=0x7f407c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.561+0000 7f409683c640 1 --2- 172.21.15.12:0/479538539 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f407c03e210 0x7f407c0406d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f40900a94a0 tx=0x7f408002e000 comp rx=0 tx=0).stop 2024-09-11T01:45:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.561+0000 7f409683c640 10 monclient: shutdown 2024-09-11T01:45:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.561+0000 7f409683c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.561+0000 7f409683c640 1 -- 172.21.15.12:0/479538539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4090138040 msgr2=0x7f4090125270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.561+0000 7f409683c640 1 --2- 172.21.15.12:0/479538539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4090138040 0x7f4090125270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f408800c780 tx=0x7f408800cc40 comp rx=0 tx=0).stop 2024-09-11T01:45:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.561+0000 7f409683c640 1 -- 172.21.15.12:0/479538539 shutdown_connections 2024-09-11T01:45:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.561+0000 7f409683c640 1 --2- 172.21.15.12:0/479538539 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f407c03e210 0x7f407c0406d0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.561+0000 7f409683c640 1 --2- 172.21.15.12:0/479538539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4090138040 0x7f4090125270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:09.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.561+0000 7f409683c640 1 --2- 172.21.15.12:0/479538539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f40900a3270 0x7f4090124d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:09.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.561+0000 7f409683c640 1 --2- 172.21.15.12:0/479538539 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f40900a2e90 0x7f40901247f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:09.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.561+0000 7f409683c640 1 -- 172.21.15.12:0/479538539 >> 172.21.15.12:0/479538539 conn(0x7f40900a8490 msgr2=0x7f40900a2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:09.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.561+0000 7f409683c640 1 -- 172.21.15.12:0/479538539 shutdown_connections 2024-09-11T01:45:09.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:09.561+0000 7f409683c640 1 -- 172.21.15.12:0/479538539 wait complete. 2024-09-11T01:45:12.574 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:45:12.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.685+0000 7fbee1e1f640 10 monclient: get_monmap_and_config 2024-09-11T01:45:12.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.685+0000 7fbee1e1f640 10 monclient: build_initial_monmap 2024-09-11T01:45:12.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.685+0000 7fbee1e1f640 10 monclient: monmap: 2024-09-11T01:45:12.687 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:12.687 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:12.687 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:45:12.686503+0000 2024-09-11T01:45:12.687 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:45:12.686503+0000 2024-09-11T01:45:12.687 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:12.688 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:12.688 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:45:12.688 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:45:12.688 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:45:12.688 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:45:12.688 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:45:12.688 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:12.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.686+0000 7fbee1e1f640 1 Processor -- start 2024-09-11T01:45:12.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.686+0000 7fbee1e1f640 1 -- start start 2024-09-11T01:45:12.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.686+0000 7fbee1e1f640 10 monclient: init 2024-09-11T01:45:12.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.687+0000 7fbee1e1f640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:12.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.687+0000 7fbee1e1f640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-11T01:45:12.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.687+0000 7fbee1e1f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbedc0a4170 addr v1:172.21.15.71:6789/0 2024-09-11T01:45:12.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.687+0000 7fbee1e1f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbedc13af80 addr v1:172.21.15.12:6789/0 2024-09-11T01:45:12.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.687+0000 7fbee1e1f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbedc13ea90 0x7fbedc140e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:12.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.687+0000 7fbee1e1f640 10 monclient(hunting): picked mon.noname-a con 0x7fbedc13ea90 addr v2:172.21.15.12:3300/0 2024-09-11T01:45:12.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.687+0000 7fbee1e1f640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbedc1380d0 con 0x7fbedc13af80 2024-09-11T01:45:12.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.687+0000 7fbee1e1f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbedc09fff0 con 0x7fbedc0a4170 2024-09-11T01:45:12.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.687+0000 7fbee1e1f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:12.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.687+0000 7fbee1e1f640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fbedc138350 con 0x7fbedc13ea90 2024-09-11T01:45:12.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.687+0000 7fbee1e1f640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:12.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.687+0000 7fbee1e1f640 10 monclient(hunting): authenticate will time out at 1719.129231s 2024-09-11T01:45:12.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.687+0000 7fbedbfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbedc13ea90 0x7fbedc140e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:12.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.687+0000 7fbedaffd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbedc13af80 0x7fbedc13d370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34778/0 (socket says 172.21.15.12:34778) 2024-09-11T01:45:12.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.687+0000 7fbedaffd640 1 -- 172.21.15.12:0/2668618547 learned_addr learned my addr 172.21.15.12:0/2668618547 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:45:12.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.687+0000 7fbedbfff640 10 monclient(hunting): get_auth_request con 0x7fbedc13ea90 auth_method 0 2024-09-11T01:45:12.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.687+0000 7fbedbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:12.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.687+0000 7fbedbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:12.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.687+0000 7fbedbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:12.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.687+0000 7fbedbfff640 1 -- 172.21.15.12:0/2668618547 >> v2:172.21.15.12:3300/0 conn(0x7fbedc13ea90 msgr2=0x7fbedc140e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:45:12.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.687+0000 7fbedbfff640 1 -- 172.21.15.12:0/2668618547 >> v2:172.21.15.12:3300/0 conn(0x7fbedc13ea90 msgr2=0x7fbedc140e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:45:12.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.687+0000 7fbedbfff640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.12:3300/0 conn(0x7fbedc13ea90 0x7fbedc140e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:45:12.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.687+0000 7fbedbfff640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.12:3300/0 conn(0x7fbedc13ea90 0x7fbedc140e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:12.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.689+0000 7fbedaffd640 1 -- 172.21.15.12:0/2668618547 >> v1:172.21.15.12:6789/0 conn(0x7fbedc13af80 legacy=0x7fbedc13d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:12.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.689+0000 7fbedb7fe640 1 -- 172.21.15.12:0/2668618547 >> v1:172.21.15.71:6789/0 conn(0x7fbedc0a4170 legacy=0x7fbedc073fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:12.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.889+0000 7fbedbfff640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.12:3300/0 conn(0x7fbedc13ea90 0x7fbedc140e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:12.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.889+0000 7fbedbfff640 10 monclient(hunting): get_auth_request con 0x7fbedc13ea90 auth_method 0 2024-09-11T01:45:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.889+0000 7fbedbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.889+0000 7fbedbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.889+0000 7fbedbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:45:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.889+0000 7fbedbfff640 1 -- 172.21.15.12:0/2668618547 >> v2:172.21.15.12:3300/0 conn(0x7fbedc13ea90 msgr2=0x7fbedc140e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:45:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.889+0000 7fbedbfff640 1 -- 172.21.15.12:0/2668618547 >> v2:172.21.15.12:3300/0 conn(0x7fbedc13ea90 msgr2=0x7fbedc140e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:45:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.889+0000 7fbedbfff640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.12:3300/0 conn(0x7fbedc13ea90 0x7fbedc140e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:45:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:12.889+0000 7fbedbfff640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.12:3300/0 conn(0x7fbedc13ea90 0x7fbedc140e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:45:13.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:13.291+0000 7fbedbfff640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.12:3300/0 conn(0x7fbedc13ea90 0x7fbedc140e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:13.291+0000 7fbedbfff640 10 monclient(hunting): get_auth_request con 0x7fbedc13ea90 auth_method 0 2024-09-11T01:45:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:13.291+0000 7fbedbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:13.291+0000 7fbedbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:13.291+0000 7fbedbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:45:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:13.291+0000 7fbedbfff640 1 -- 172.21.15.12:0/2668618547 >> v2:172.21.15.12:3300/0 conn(0x7fbedc13ea90 msgr2=0x7fbedc140e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:45:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:13.291+0000 7fbedbfff640 1 -- 172.21.15.12:0/2668618547 >> v2:172.21.15.12:3300/0 conn(0x7fbedc13ea90 msgr2=0x7fbedc140e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:45:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:13.291+0000 7fbedbfff640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.12:3300/0 conn(0x7fbedc13ea90 0x7fbedc140e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:45:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:13.291+0000 7fbedbfff640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.12:3300/0 conn(0x7fbedc13ea90 0x7fbedc140e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:45:14.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:14.092+0000 7fbedbfff640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.12:3300/0 conn(0x7fbedc13ea90 0x7fbedc140e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:14.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:14.092+0000 7fbedbfff640 10 monclient(hunting): get_auth_request con 0x7fbedc13ea90 auth_method 0 2024-09-11T01:45:14.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:14.092+0000 7fbedbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:14.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:14.092+0000 7fbedbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:14.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:14.092+0000 7fbedbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:45:14.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:14.092+0000 7fbedbfff640 1 -- 172.21.15.12:0/2668618547 >> v2:172.21.15.12:3300/0 conn(0x7fbedc13ea90 msgr2=0x7fbedc140e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:45:14.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:14.092+0000 7fbedbfff640 1 -- 172.21.15.12:0/2668618547 >> v2:172.21.15.12:3300/0 conn(0x7fbedc13ea90 msgr2=0x7fbedc140e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:45:14.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:14.092+0000 7fbedbfff640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.12:3300/0 conn(0x7fbedc13ea90 0x7fbedc140e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:45:14.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:14.092+0000 7fbedbfff640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.12:3300/0 conn(0x7fbedc13ea90 0x7fbedc140e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:45:15.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbed97fa640 10 monclient(hunting): tick 2024-09-11T01:45:15.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbed97fa640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:45:15.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbed97fa640 1 monclient(hunting): continuing hunt 2024-09-11T01:45:15.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbed97fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:45:15.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbed97fa640 1 -- 172.21.15.12:0/2668618547 >> v2:172.21.15.12:3300/0 conn(0x7fbedc13ea90 msgr2=0x7fbedc140e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:45:15.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbed97fa640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.12:3300/0 conn(0x7fbedc13ea90 0x7fbedc140e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:15.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbed97fa640 1 -- 172.21.15.12:0/2668618547 >> v1:172.21.15.71:6789/0 conn(0x7fbedc0a4170 legacy=0x7fbedc073fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:15.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbed97fa640 1 -- 172.21.15.12:0/2668618547 >> v1:172.21.15.12:6789/0 conn(0x7fbedc13af80 legacy=0x7fbedc13d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:15.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbed97fa640 10 monclient: _add_conns ranks=[2,4] 2024-09-11T01:45:15.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbed97fa640 1 --2- 172.21.15.12:0/2668618547 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbec8003830 0x7fbec8005c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:15.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbed97fa640 10 monclient(hunting): picked mon.noname-b con 0x7fbec8003830 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:15.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbed97fa640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 0x7fbec8008830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:15.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbed97fa640 10 monclient(hunting): picked mon.noname-c con 0x7fbec8006390 addr v2:172.21.15.71:3300/0 2024-09-11T01:45:15.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbed97fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:15.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbed97fa640 1 -- 172.21.15.12:0/2668618547 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fbedc138350 con 0x7fbec8006390 2024-09-11T01:45:15.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbed97fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:15.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbed97fa640 1 -- 172.21.15.12:0/2668618547 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fbec8008ef0 con 0x7fbec8003830 2024-09-11T01:45:15.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbed97fa640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:15.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbedb7fe640 1 -- 172.21.15.12:0/2668618547 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbec8003830 msgr2=0x7fbec8005c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:45:15.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbedb7fe640 1 --2- 172.21.15.12:0/2668618547 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbec8003830 0x7fbec8005c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:15.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbedaffd640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 0x7fbec8008830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:15.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbedaffd640 10 monclient(hunting): get_auth_request con 0x7fbec8006390 auth_method 0 2024-09-11T01:45:15.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbedaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:15.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbedaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:15.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbedaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:15.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbedaffd640 1 -- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 msgr2=0x7fbec8008830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:45:15.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbedaffd640 1 -- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 msgr2=0x7fbec8008830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:45:15.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbedaffd640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 0x7fbec8008830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:45:15.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.687+0000 7fbedaffd640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 0x7fbec8008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:15.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.888+0000 7fbedb7fe640 1 -- 172.21.15.12:0/2668618547 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbec8003830 msgr2=0x7fbec8005c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:45:15.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.888+0000 7fbedb7fe640 1 --2- 172.21.15.12:0/2668618547 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbec8003830 0x7fbec8005c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:45:15.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.888+0000 7fbedaffd640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 0x7fbec8008830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:15.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.888+0000 7fbedaffd640 10 monclient(hunting): get_auth_request con 0x7fbec8006390 auth_method 0 2024-09-11T01:45:15.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.888+0000 7fbedaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:15.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.888+0000 7fbedaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:15.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.888+0000 7fbedaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:45:15.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.888+0000 7fbedaffd640 1 -- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 msgr2=0x7fbec8008830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:45:15.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.888+0000 7fbedaffd640 1 -- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 msgr2=0x7fbec8008830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:45:15.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.888+0000 7fbedaffd640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 0x7fbec8008830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:45:15.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:15.888+0000 7fbedaffd640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 0x7fbec8008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:45:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:16.289+0000 7fbedaffd640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 0x7fbec8008830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:16.289+0000 7fbedaffd640 10 monclient(hunting): get_auth_request con 0x7fbec8006390 auth_method 0 2024-09-11T01:45:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:16.289+0000 7fbedaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:16.289+0000 7fbedaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:16.289+0000 7fbedaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:45:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:16.289+0000 7fbedb7fe640 1 -- 172.21.15.12:0/2668618547 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbec8003830 msgr2=0x7fbec8005c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:45:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:16.289+0000 7fbedb7fe640 1 --2- 172.21.15.12:0/2668618547 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbec8003830 0x7fbec8005c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:45:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:16.289+0000 7fbedaffd640 1 -- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 msgr2=0x7fbec8008830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:45:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:16.289+0000 7fbedaffd640 1 -- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 msgr2=0x7fbec8008830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:45:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:16.289+0000 7fbedaffd640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 0x7fbec8008830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:45:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:16.289+0000 7fbedaffd640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 0x7fbec8008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:45:17.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:17.090+0000 7fbedb7fe640 1 -- 172.21.15.12:0/2668618547 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbec8003830 msgr2=0x7fbec8005c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:45:17.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:17.090+0000 7fbedb7fe640 1 --2- 172.21.15.12:0/2668618547 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbec8003830 0x7fbec8005c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:45:17.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:17.090+0000 7fbedaffd640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 0x7fbec8008830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:17.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:17.090+0000 7fbedaffd640 10 monclient(hunting): get_auth_request con 0x7fbec8006390 auth_method 0 2024-09-11T01:45:17.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:17.090+0000 7fbedaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:17.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:17.090+0000 7fbedaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:17.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:17.090+0000 7fbedaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:45:17.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:17.091+0000 7fbedaffd640 1 -- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 msgr2=0x7fbec8008830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:45:17.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:17.091+0000 7fbedaffd640 1 -- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 msgr2=0x7fbec8008830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:45:17.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:17.091+0000 7fbedaffd640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 0x7fbec8008830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:45:17.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:17.091+0000 7fbedaffd640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 0x7fbec8008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:45:18.447 INFO:tasks.rados.rados.0.smithi071.stdout:2386: done (3 left) 2024-09-11T01:45:18.448 INFO:tasks.rados.rados.0.smithi071.stdout:2387: done (2 left) 2024-09-11T01:45:18.448 INFO:tasks.rados.rados.0.smithi071.stdout:2388: done (1 left) 2024-09-11T01:45:18.448 INFO:tasks.rados.rados.0.smithi071.stdout:2389: done (0 left) 2024-09-11T01:45:18.448 INFO:tasks.rados.rados.0.smithi071.stdout:2390: read oid 32 snap -1 2024-09-11T01:45:18.448 INFO:tasks.rados.rados.0.smithi071.stdout:2390: expect (ObjNum 745 snap 227 seq_num 745) 2024-09-11T01:45:18.448 INFO:tasks.rados.rados.0.smithi071.stdout:2391: rollback oid 29 current snap is 266 2024-09-11T01:45:18.448 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 29 to 174 2024-09-11T01:45:18.448 INFO:tasks.rados.rados.0.smithi071.stdout:2392: read oid 23 snap -1 2024-09-11T01:45:18.448 INFO:tasks.rados.rados.0.smithi071.stdout:2392: expect (ObjNum 652 snap 199 seq_num 652) 2024-09-11T01:45:18.448 INFO:tasks.rados.rados.0.smithi071.stdout:2393: delete oid 15 current snap is 266 2024-09-11T01:45:18.455 INFO:tasks.rados.rados.0.smithi071.stdout:2392: expect (ObjNum 652 snap 199 seq_num 652) 2024-09-11T01:45:18.634 INFO:tasks.rados.rados.0.smithi071.stdout:2390: expect (ObjNum 745 snap 227 seq_num 745) 2024-09-11T01:45:18.673 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2217245 2024-09-11T01:45:18.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.687+0000 7fbed97fa640 10 monclient(hunting): tick 2024-09-11T01:45:18.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.687+0000 7fbed97fa640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:45:18.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.687+0000 7fbed97fa640 1 monclient(hunting): continuing hunt 2024-09-11T01:45:18.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.687+0000 7fbed97fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:45:18.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.687+0000 7fbed97fa640 1 -- 172.21.15.12:0/2668618547 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbec8003830 msgr2=0x7fbec8005c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:45:18.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.687+0000 7fbed97fa640 1 --2- 172.21.15.12:0/2668618547 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbec8003830 0x7fbec8005c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:18.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.687+0000 7fbed97fa640 1 -- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 msgr2=0x7fbec8008830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:45:18.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.687+0000 7fbed97fa640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 0x7fbec8008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:18.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.687+0000 7fbed97fa640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-11T01:45:18.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.687+0000 7fbedbfff640 1 -- 172.21.15.12:0/2668618547 reap_dead start 2024-09-11T01:45:18.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.687+0000 7fbed97fa640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 0x7fbec8008830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:18.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.687+0000 7fbed97fa640 10 monclient(hunting): picked mon.noname-c con 0x7fbec8006390 addr v2:172.21.15.71:3300/0 2024-09-11T01:45:18.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.687+0000 7fbed97fa640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.12:3300/0 conn(0x7fbec8003c50 0x7fbec8009390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:18.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.687+0000 7fbed97fa640 10 monclient(hunting): picked mon.noname-a con 0x7fbec8003c50 addr v2:172.21.15.12:3300/0 2024-09-11T01:45:18.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.687+0000 7fbed97fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbec80098d0 addr v1:172.21.15.71:6789/0 2024-09-11T01:45:18.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.687+0000 7fbed97fa640 1 -- 172.21.15.12:0/2668618547 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbedc070280 con 0x7fbec80098d0 2024-09-11T01:45:18.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.687+0000 7fbed97fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:18.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.687+0000 7fbed97fa640 1 -- 172.21.15.12:0/2668618547 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fbedc138350 con 0x7fbec8003c50 2024-09-11T01:45:18.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.687+0000 7fbed97fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:18.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.687+0000 7fbed97fa640 1 -- 172.21.15.12:0/2668618547 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fbec8008ef0 con 0x7fbec8006390 2024-09-11T01:45:18.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.687+0000 7fbed97fa640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:18.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.688+0000 7fbedaffd640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.12:3300/0 conn(0x7fbec8003c50 0x7fbec8009390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:18.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.688+0000 7fbedb7fe640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 0x7fbec8008830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:18.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.688+0000 7fbedaffd640 10 monclient(hunting): get_auth_request con 0x7fbec8003c50 auth_method 0 2024-09-11T01:45:18.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.688+0000 7fbedaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:18.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.688+0000 7fbedaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:18.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.688+0000 7fbedaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:18.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.688+0000 7fbedb7fe640 10 monclient(hunting): get_auth_request con 0x7fbec8006390 auth_method 0 2024-09-11T01:45:18.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.688+0000 7fbedb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:18.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.688+0000 7fbedb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:18.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.688+0000 7fbedb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:18.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.688+0000 7fbedaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:18.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.688+0000 7fbedaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:18.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.688+0000 7fbedaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:18.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.688+0000 7fbedaffd640 10 monclient(hunting): handle_auth_done global_id 174165 payload 274 2024-09-11T01:45:18.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.688+0000 7fbedaffd640 1 -- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 msgr2=0x7fbec8008830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:18.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.688+0000 7fbedaffd640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 0x7fbec8008830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:18.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.688+0000 7fbedaffd640 1 -- 172.21.15.12:0/2668618547 >> v1:172.21.15.71:6789/0 conn(0x7fbec80098d0 legacy=0x7fbec800bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:18.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.688+0000 7fbedaffd640 10 monclient: _finish_hunting 0 2024-09-11T01:45:18.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.688+0000 7fbedaffd640 1 monclient: found mon.noname-a 2024-09-11T01:45:18.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.688+0000 7fbedaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:18.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.688+0000 7fbedaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:45:18.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.688+0000 7fbedaffd640 1 -- 172.21.15.12:0/2668618547 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbec800e640 con 0x7fbec8003c50 2024-09-11T01:45:18.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.691+0000 7fbedaffd640 1 -- 172.21.15.12:0/2668618547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbec8003c50 msgr2=0x7fbec8009390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:18.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.691+0000 7fbedaffd640 1 --2- 172.21.15.12:0/2668618547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbec8003c50 0x7fbec8009390 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbedc09ffc0 tx=0x7fbed002adf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3e5b44a4baba6e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:18.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.691+0000 7fbeda7fc640 1 -- 172.21.15.12:0/2668618547 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed0031070 con 0x7fbec8003c50 2024-09-11T01:45:18.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.691+0000 7fbeda7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:18.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.692+0000 7fbeda7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:45:18.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.692+0000 7fbeda7fc640 10 monclient: dump: 2024-09-11T01:45:18.696 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:18.696 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:18.696 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:18.696 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:18.696 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:18.697 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:18.697 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:18.697 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:18.697 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:18.697 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:18.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.692+0000 7fbeda7fc640 10 monclient: _finish_auth 0 2024-09-11T01:45:18.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.692+0000 7fbeda7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:45:18.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.692+0000 7fbeda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:18.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.692+0000 7fbeda7fc640 1 -- 172.21.15.12:0/2668618547 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbed002b910 con 0x7fbec8003c50 2024-09-11T01:45:18.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.692+0000 7fbeda7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:18.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.692+0000 7fbeda7fc640 1 -- 172.21.15.12:0/2668618547 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed002bd00 con 0x7fbec8003c50 2024-09-11T01:45:18.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.692+0000 7fbee1e1f640 5 monclient: authenticate success, global_id 174165 2024-09-11T01:45:18.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.692+0000 7fbee1e1f640 10 monclient: get_monmap_and_config success 2024-09-11T01:45:18.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.692+0000 7fbee1e1f640 4 set_mon_vals no callback set 2024-09-11T01:45:18.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.692+0000 7fbeda7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:18.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.692+0000 7fbeda7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:18.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.692+0000 7fbeda7fc640 10 monclient: dump: 2024-09-11T01:45:18.698 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:18.698 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:18.698 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:18.698 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:18.698 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:18.698 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:18.698 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:18.698 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:18.699 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:18.699 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:18.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.692+0000 7fbee1e1f640 10 monclient: shutdown 2024-09-11T01:45:18.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.692+0000 7fbee1e1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:18.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.692+0000 7fbee1e1f640 1 -- 172.21.15.12:0/2668618547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbec8003c50 msgr2=0x7fbec8009390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:18.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.692+0000 7fbee1e1f640 1 --2- 172.21.15.12:0/2668618547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbec8003c50 0x7fbec8009390 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbedc09ffc0 tx=0x7fbed002adf0 comp rx=0 tx=0).stop 2024-09-11T01:45:18.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.692+0000 7fbee1e1f640 1 -- 172.21.15.12:0/2668618547 shutdown_connections 2024-09-11T01:45:18.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.692+0000 7fbee1e1f640 1 --2- 172.21.15.12:0/2668618547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbec8003c50 0x7fbec8009390 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:18.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.693+0000 7fbee1e1f640 1 --2- 172.21.15.12:0/2668618547 >> v2:172.21.15.71:3300/0 conn(0x7fbec8006390 0x7fbec8008830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:18.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.693+0000 7fbee1e1f640 1 -- 172.21.15.12:0/2668618547 >> 172.21.15.12:0/2668618547 conn(0x7fbedc0a3650 msgr2=0x7fbedc0a3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:18.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.693+0000 7fbee1e1f640 1 -- 172.21.15.12:0/2668618547 shutdown_connections 2024-09-11T01:45:18.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.693+0000 7fbee1e1f640 1 -- 172.21.15.12:0/2668618547 wait complete. 2024-09-11T01:45:18.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.693+0000 7fbee1e1f640 10 monclient: build_initial_monmap 2024-09-11T01:45:18.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.693+0000 7fbee1e1f640 10 monclient: monmap: 2024-09-11T01:45:18.700 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:18.700 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:45:18.700 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:45:18.700 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:45:18.700 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:18.700 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:18.700 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:45:18.700 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:45:18.700 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:45:18.700 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:18.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.694+0000 7fbee1e1f640 1 Processor -- start 2024-09-11T01:45:18.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.694+0000 7fbee1e1f640 1 -- start start 2024-09-11T01:45:18.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.694+0000 7fbee1e1f640 10 monclient: init 2024-09-11T01:45:18.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbee1e1f640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:18.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbee1e1f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:45:18.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbee1e1f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbec8003c50 0x7fbedc1cce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:18.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbee1e1f640 10 monclient(hunting): picked mon.noname-c con 0x7fbec8003c50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:18.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbee1e1f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbec8006390 0x7fbedc131770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:18.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbee1e1f640 10 monclient(hunting): picked mon.noname-b con 0x7fbec8006390 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:45:18.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbee1e1f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbec80098d0 0x7fbedc131cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:18.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbee1e1f640 10 monclient(hunting): picked mon.noname-a con 0x7fbec80098d0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:45:18.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbee1e1f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:18.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbee1e1f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbedc0a0660 con 0x7fbec80098d0 2024-09-11T01:45:18.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbee1e1f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:18.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbee1e1f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbedc0a07e0 con 0x7fbec8006390 2024-09-11T01:45:18.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbee1e1f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:18.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbee1e1f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fbedc073fc0 con 0x7fbec8003c50 2024-09-11T01:45:18.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbee1e1f640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:18.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbee1e1f640 10 monclient(hunting): authenticate will time out at 1725.137207s 2024-09-11T01:45:18.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedb7fe640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbec8003c50 msgr2=0x7fbedc1cce90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:45:18.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedb7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbec8003c50 0x7fbedc1cce90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:18.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbec80098d0 0x7fbedc131cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:18.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbec80098d0 0x7fbedc131cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56760/0 (socket says 172.21.15.12:56760) 2024-09-11T01:45:18.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedbfff640 1 -- 172.21.15.12:0/1572686385 learned_addr learned my addr 172.21.15.12:0/1572686385 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:18.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedbfff640 10 monclient(hunting): get_auth_request con 0x7fbec80098d0 auth_method 0 2024-09-11T01:45:18.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:18.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:18.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:18.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedaffd640 1 --2- 172.21.15.12:0/1572686385 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbec8006390 0x7fbedc131770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:18.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedaffd640 10 monclient(hunting): get_auth_request con 0x7fbec8006390 auth_method 0 2024-09-11T01:45:18.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:18.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:18.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:18.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:18.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:18.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:18.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:18.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:18.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:18.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedbfff640 10 monclient(hunting): handle_auth_done global_id 174168 payload 824 2024-09-11T01:45:18.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedbfff640 1 -- 172.21.15.12:0/1572686385 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbec8003c50 msgr2=0x7fbedc1cce90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:45:18.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedbfff640 1 --2- 172.21.15.12:0/1572686385 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbec8003c50 0x7fbedc1cce90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:18.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedbfff640 1 -- 172.21.15.12:0/1572686385 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbec8006390 msgr2=0x7fbedc131770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:18.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedbfff640 1 --2- 172.21.15.12:0/1572686385 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbec8006390 0x7fbedc131770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:18.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedbfff640 10 monclient: _finish_hunting 0 2024-09-11T01:45:18.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedbfff640 1 monclient: found mon.noname-a 2024-09-11T01:45:18.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:18.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:45:18.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedbfff640 1 -- 172.21.15.12:0/1572686385 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbedc133c40 con 0x7fbec80098d0 2024-09-11T01:45:18.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.695+0000 7fbedaffd640 1 --2- 172.21.15.12:0/1572686385 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbec8006390 0x7fbedc131770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:45:18.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.698+0000 7fbedbfff640 1 -- 172.21.15.12:0/1572686385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbec80098d0 msgr2=0x7fbedc131cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:18.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.698+0000 7fbedbfff640 1 --2- 172.21.15.12:0/1572686385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbec80098d0 0x7fbedc131cb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbedc09ff90 tx=0x7fbed4009d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:18.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.698+0000 7fbec3fff640 1 -- 172.21.15.12:0/1572686385 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed401b070 con 0x7fbec80098d0 2024-09-11T01:45:18.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.698+0000 7fbec3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:18.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.698+0000 7fbec3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:45:18.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.698+0000 7fbec3fff640 10 monclient: dump: 2024-09-11T01:45:18.708 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:18.708 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:18.708 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:18.708 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:18.709 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:18.709 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:18.709 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:18.709 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:18.709 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:18.709 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:18.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.698+0000 7fbec3fff640 10 monclient: _finish_auth 0 2024-09-11T01:45:18.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.698+0000 7fbec3fff640 10 monclient: _check_auth_tickets 2024-09-11T01:45:18.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.698+0000 7fbec3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:18.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.698+0000 7fbec3fff640 1 -- 172.21.15.12:0/1572686385 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbed4013c20 con 0x7fbec80098d0 2024-09-11T01:45:18.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.698+0000 7fbee1e1f640 5 monclient: authenticate success, global_id 174168 2024-09-11T01:45:18.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.698+0000 7fbee1e1f640 10 monclient: _renew_subs 2024-09-11T01:45:18.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.698+0000 7fbee1e1f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:18.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.698+0000 7fbee1e1f640 1 -- 172.21.15.12:0/1572686385 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbedc0a5ee0 con 0x7fbec80098d0 2024-09-11T01:45:18.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.698+0000 7fbec3fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:18.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.698+0000 7fbec3fff640 1 -- 172.21.15.12:0/1572686385 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed4017430 con 0x7fbec80098d0 2024-09-11T01:45:18.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.698+0000 7fbec3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:18.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.698+0000 7fbec3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:18.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.698+0000 7fbec3fff640 10 monclient: dump: 2024-09-11T01:45:18.710 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:18.710 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:18.710 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:18.710 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:18.710 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:18.710 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:18.710 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:18.710 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:18.711 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:18.711 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:18.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.698+0000 7fbee1e1f640 10 monclient: _renew_subs 2024-09-11T01:45:18.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.698+0000 7fbee1e1f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:18.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.698+0000 7fbee1e1f640 1 -- 172.21.15.12:0/1572686385 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbedc1347a0 con 0x7fbec80098d0 2024-09-11T01:45:18.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.698+0000 7fbed9ffb640 4 set_mon_vals no callback set 2024-09-11T01:45:18.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.699+0000 7fbee1e1f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:18.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.699+0000 7fbee1e1f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:18.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.699+0000 7fbee1e1f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:18.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.699+0000 7fbec3fff640 1 -- 172.21.15.12:0/1572686385 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fbed40175d0 con 0x7fbec80098d0 2024-09-11T01:45:18.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.699+0000 7fbee1e1f640 1 -- 172.21.15.12:0/1572686385 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbedc0743f0 con 0x7fbec80098d0 2024-09-11T01:45:18.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.699+0000 7fbec3fff640 1 --2- 172.21.15.12:0/1572686385 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbec803fb90 0x7fbec8042050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:18.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.699+0000 7fbec3fff640 1 -- 172.21.15.12:0/1572686385 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(537..537 src has 317..537) ==== 4771+0+0 (secure 0 0 0) 0x7fbed40539c0 con 0x7fbec80098d0 2024-09-11T01:45:18.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.700+0000 7fbedb7fe640 1 --2- 172.21.15.12:0/1572686385 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbec803fb90 0x7fbec8042050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:18.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.700+0000 7fbedb7fe640 10 monclient: get_auth_request con 0x7fbec803fb90 auth_method 0 2024-09-11T01:45:18.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.702+0000 7fbedb7fe640 1 --2- 172.21.15.12:0/1572686385 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbec803fb90 0x7fbec8042050 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fbecc0014f0 tx=0x7fbecc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:18.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.704+0000 7fbec3fff640 1 -- 172.21.15.12:0/1572686385 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbed40523c0 con 0x7fbec80098d0 2024-09-11T01:45:18.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.704+0000 7fbec3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:18.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.704+0000 7fbec3fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:45:18.800 INFO:tasks.rados.rados.0.smithi071.stdout:2391: finishing rollback tid 0 to smithi07134777-29 2024-09-11T01:45:18.800 INFO:tasks.rados.rados.0.smithi071.stdout:2391: finishing rollback tid 1 to smithi07134777-29 2024-09-11T01:45:18.800 INFO:tasks.rados.rados.0.smithi071.stdout:2391: finishing rollback tid 2 to smithi07134777-29 2024-09-11T01:45:18.800 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 1525 (ObjNum 570 snap 161 seq_num 570) dirty exists 2024-09-11T01:45:18.800 INFO:tasks.rados.rados.0.smithi071.stdout:2390: done (3 left) 2024-09-11T01:45:18.801 INFO:tasks.rados.rados.0.smithi071.stdout:2391: done (2 left) 2024-09-11T01:45:18.801 INFO:tasks.rados.rados.0.smithi071.stdout:2392: done (1 left) 2024-09-11T01:45:18.801 INFO:tasks.rados.rados.0.smithi071.stdout:2393: done (0 left) 2024-09-11T01:45:18.801 INFO:tasks.rados.rados.0.smithi071.stdout:2394: write oid 14 current snap is 266 2024-09-11T01:45:18.801 INFO:tasks.rados.rados.0.smithi071.stdout:2394: seq_num 865 ranges {538055=638274,1867983=670072} 2024-09-11T01:45:18.806 INFO:tasks.rados.rados.0.smithi071.stdout:2394: writing smithi07134777-14 from 538055 to 1176329 tid 1 2024-09-11T01:45:18.814 INFO:tasks.rados.rados.0.smithi071.stdout:2394: writing smithi07134777-14 from 1867983 to 2538055 tid 2 2024-09-11T01:45:18.814 INFO:tasks.rados.rados.0.smithi071.stdout:2395: read oid 31 snap 174 2024-09-11T01:45:18.814 INFO:tasks.rados.rados.0.smithi071.stdout:2395: expect (ObjNum 509 snap 139 seq_num 509) 2024-09-11T01:45:18.814 INFO:tasks.rados.rados.0.smithi071.stdout:2396: delete oid 10 current snap is 266 2024-09-11T01:45:18.814 INFO:tasks.rados.rados.0.smithi071.stdout:2394: finishing write tid 1 to smithi07134777-14 2024-09-11T01:45:18.817 INFO:tasks.rados.rados.0.smithi071.stdout:2396: done (2 left) 2024-09-11T01:45:18.817 INFO:tasks.rados.rados.0.smithi071.stdout:2397: read oid 38 snap 235 2024-09-11T01:45:18.817 INFO:tasks.rados.rados.0.smithi071.stdout:2397: expect (ObjNum 542 snap 154 seq_num 542) 2024-09-11T01:45:18.817 INFO:tasks.rados.rados.0.smithi071.stdout:2398: write (excl) oid 21 current snap is 266 2024-09-11T01:45:18.817 INFO:tasks.rados.rados.0.smithi071.stdout:2398: seq_num 867 ranges {571669=505125,1701300=743627} 2024-09-11T01:45:18.823 INFO:tasks.rados.rados.0.smithi071.stdout:2398: writing smithi07134777-21 from 571669 to 1076794 tid 1 2024-09-11T01:45:18.831 INFO:tasks.rados.rados.0.smithi071.stdout:2398: writing smithi07134777-21 from 1701300 to 2444927 tid 2 2024-09-11T01:45:18.831 INFO:tasks.rados.rados.0.smithi071.stdout:2399: read oid 36 snap -1 2024-09-11T01:45:18.831 INFO:tasks.rados.rados.0.smithi071.stdout:2399: expect (ObjNum 829 snap 252 seq_num 829) 2024-09-11T01:45:18.831 INFO:tasks.rados.rados.0.smithi071.stdout:2400: write (excl) oid 4 current snap is 266 2024-09-11T01:45:18.831 INFO:tasks.rados.rados.0.smithi071.stdout:2400: seq_num 868 ranges {588476=580374,1643429=603638} 2024-09-11T01:45:18.837 INFO:tasks.rados.rados.0.smithi071.stdout:2400: writing smithi07134777-4 from 588476 to 1168850 tid 1 2024-09-11T01:45:18.844 INFO:tasks.rados.rados.0.smithi071.stdout:2400: writing smithi07134777-4 from 1643429 to 2247067 tid 2 2024-09-11T01:45:18.844 INFO:tasks.rados.rados.0.smithi071.stdout:2401: snap_remove snap 245 2024-09-11T01:45:18.944 INFO:tasks.rados.rados.0.smithi071.stdout:2395: expect (ObjNum 509 snap 139 seq_num 509) 2024-09-11T01:45:18.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.978+0000 7fbee1e1f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:45:18.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.978+0000 7fbee1e1f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:45:18.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.978+0000 7fbee1e1f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:18.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.978+0000 7fbee1e1f640 1 -- 172.21.15.12:0/1572686385 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbedc06dc80 con 0x7fbec80098d0 2024-09-11T01:45:18.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.978+0000 7fbec3fff640 1 -- 172.21.15.12:0/1572686385 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fbed4015d40 con 0x7fbec80098d0 2024-09-11T01:45:18.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.978+0000 7fbec3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:45:18.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.978+0000 7fbec3fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:45:18.981 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":152,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:45:18.982 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:45:18.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.980+0000 7fbee1e1f640 1 -- 172.21.15.12:0/1572686385 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbec803fb90 msgr2=0x7fbec8042050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:18.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.980+0000 7fbee1e1f640 1 --2- 172.21.15.12:0/1572686385 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbec803fb90 0x7fbec8042050 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fbecc0014f0 tx=0x7fbecc009040 comp rx=0 tx=0).stop 2024-09-11T01:45:18.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.980+0000 7fbee1e1f640 10 monclient: shutdown 2024-09-11T01:45:18.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.980+0000 7fbee1e1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:18.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.980+0000 7fbee1e1f640 1 -- 172.21.15.12:0/1572686385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbec80098d0 msgr2=0x7fbedc131cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:18.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.980+0000 7fbee1e1f640 1 --2- 172.21.15.12:0/1572686385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbec80098d0 0x7fbedc131cb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbedc09ff90 tx=0x7fbed4009d00 comp rx=0 tx=0).stop 2024-09-11T01:45:18.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.980+0000 7fbee1e1f640 1 -- 172.21.15.12:0/1572686385 shutdown_connections 2024-09-11T01:45:18.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.980+0000 7fbee1e1f640 1 --2- 172.21.15.12:0/1572686385 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbec803fb90 0x7fbec8042050 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:18.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.980+0000 7fbee1e1f640 1 --2- 172.21.15.12:0/1572686385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbec80098d0 0x7fbedc131cb0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:18.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.980+0000 7fbee1e1f640 1 --2- 172.21.15.12:0/1572686385 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbec8006390 0x7fbedc131770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:18.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.980+0000 7fbee1e1f640 1 --2- 172.21.15.12:0/1572686385 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbec8003c50 0x7fbedc1cce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:18.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.981+0000 7fbee1e1f640 1 -- 172.21.15.12:0/1572686385 >> 172.21.15.12:0/1572686385 conn(0x7fbedc0a3650 msgr2=0x7fbedc1365c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:18.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.981+0000 7fbee1e1f640 1 -- 172.21.15.12:0/1572686385 shutdown_connections 2024-09-11T01:45:18.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:18.981+0000 7fbee1e1f640 1 -- 172.21.15.12:0/1572686385 wait complete. 2024-09-11T01:45:18.993 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:45:18.994 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-11T01:45:18.994 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:45:19.069 INFO:tasks.rados.rados.0.smithi071.stdout:2394: finishing write tid 2 to smithi07134777-14 2024-09-11T01:45:19.069 INFO:tasks.rados.rados.0.smithi071.stdout:2394: finishing write tid 3 to smithi07134777-14 2024-09-11T01:45:19.069 INFO:tasks.rados.rados.0.smithi071.stdout:2394: finishing write tid 4 to smithi07134777-14 2024-09-11T01:45:19.069 INFO:tasks.rados.rados.0.smithi071.stdout:2394: oid 14 updating version 0 to 1530 2024-09-11T01:45:19.069 INFO:tasks.rados.rados.0.smithi071.stdout:2394: oid 14 version 1530 is already newer than 1529 2024-09-11T01:45:19.069 INFO:tasks.rados.rados.0.smithi071.stdout:2394: oid 14 version 1530 is already newer than 1528 2024-09-11T01:45:19.069 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 1530 (ObjNum 865 snap 266 seq_num 865) dirty exists 2024-09-11T01:45:19.069 INFO:tasks.rados.rados.0.smithi071.stdout:2394: left oid 14 (ObjNum 865 snap 266 seq_num 865) 2024-09-11T01:45:19.069 INFO:tasks.rados.rados.0.smithi071.stdout:2397: expect (ObjNum 542 snap 154 seq_num 542) 2024-09-11T01:45:19.069 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:45:19.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.094+0000 7efc9187f640 10 monclient: get_monmap_and_config 2024-09-11T01:45:19.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.094+0000 7efc9187f640 10 monclient: build_initial_monmap 2024-09-11T01:45:19.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.094+0000 7efc9187f640 10 monclient: monmap: 2024-09-11T01:45:19.096 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:19.096 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:19.096 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:45:19.095447+0000 2024-09-11T01:45:19.096 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:45:19.095447+0000 2024-09-11T01:45:19.096 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:19.096 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:19.097 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:45:19.097 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:45:19.097 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:45:19.097 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:45:19.097 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:45:19.097 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:19.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.095+0000 7efc9187f640 1 Processor -- start 2024-09-11T01:45:19.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.095+0000 7efc9187f640 1 -- start start 2024-09-11T01:45:19.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.095+0000 7efc9187f640 10 monclient: init 2024-09-11T01:45:19.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.095+0000 7efc9187f640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:19.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.095+0000 7efc9187f640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-11T01:45:19.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.095+0000 7efc9187f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efc8c131850 addr v1:172.21.15.12:6789/0 2024-09-11T01:45:19.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.095+0000 7efc9187f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efc8c0a2fb0 addr v1:172.21.15.71:6789/0 2024-09-11T01:45:19.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.095+0000 7efc9187f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7efc8c09df40 0x7efc8c13d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:19.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.095+0000 7efc9187f640 10 monclient(hunting): picked mon.noname-c con 0x7efc8c09df40 addr v2:172.21.15.71:3300/0 2024-09-11T01:45:19.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.095+0000 7efc9187f640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc8c09cbe0 con 0x7efc8c131850 2024-09-11T01:45:19.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.095+0000 7efc9187f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc8c09cec0 con 0x7efc8c0a2fb0 2024-09-11T01:45:19.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.095+0000 7efc9187f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:19.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.095+0000 7efc9187f640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7efc8c132340 con 0x7efc8c09df40 2024-09-11T01:45:19.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.095+0000 7efc9187f640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:19.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.095+0000 7efc9187f640 10 monclient(hunting): authenticate will time out at 1725.537875s 2024-09-11T01:45:19.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.095+0000 7efc8affd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7efc8c131850 0x7efc8c131c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35282/0 (socket says 172.21.15.12:35282) 2024-09-11T01:45:19.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.095+0000 7efc8affd640 1 -- 172.21.15.12:0/3772059687 learned_addr learned my addr 172.21.15.12:0/3772059687 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:45:19.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.096+0000 7efc8b7fe640 1 --2- 172.21.15.12:0/3772059687 >> v2:172.21.15.71:3300/0 conn(0x7efc8c09df40 0x7efc8c13d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:19.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.096+0000 7efc8b7fe640 10 monclient(hunting): get_auth_request con 0x7efc8c09df40 auth_method 0 2024-09-11T01:45:19.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.096+0000 7efc8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:19.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.096+0000 7efc8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:19.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.096+0000 7efc8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:19.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.096+0000 7efc8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:19.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.096+0000 7efc8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:19.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.096+0000 7efc8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:19.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.098+0000 7efc8affd640 1 -- 172.21.15.12:0/3772059687 >> v1:172.21.15.12:6789/0 conn(0x7efc8c131850 legacy=0x7efc8c131c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:19.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.098+0000 7efc8a7fc640 1 -- 172.21.15.12:0/3772059687 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2494477261 0 0) 0x7efc8c09cbe0 con 0x7efc8c131850 2024-09-11T01:45:19.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.098+0000 7efc8a7fc640 10 monclient(hunting): _negotiate 2024-09-11T01:45:19.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.098+0000 7efc8a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:19.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.098+0000 7efc8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:19.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.098+0000 7efc8a7fc640 10 monclient(hunting): my global_id is 174171 2024-09-11T01:45:19.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.098+0000 7efc8a7fc640 1 -- 172.21.15.12:0/3772059687 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc78003520 con 0x7efc8c131850 2024-09-11T01:45:19.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.098+0000 7efc827fc640 1 -- 172.21.15.12:0/3772059687 >> v1:172.21.15.71:6789/0 conn(0x7efc8c0a2fb0 legacy=0x7efc8c139d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:19.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.098+0000 7efc8b7fe640 10 monclient(hunting): handle_auth_done global_id 134506 payload 274 2024-09-11T01:45:19.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.098+0000 7efc8b7fe640 1 -- 172.21.15.12:0/3772059687 >> v1:172.21.15.71:6789/0 conn(0x7efc8c0a2fb0 legacy=0x7efc8c139d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:19.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.098+0000 7efc8b7fe640 1 -- 172.21.15.12:0/3772059687 >> v1:172.21.15.12:6789/0 conn(0x7efc8c131850 legacy=0x7efc8c131c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:19.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.098+0000 7efc8b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:45:19.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.098+0000 7efc8b7fe640 1 monclient: found mon.noname-c 2024-09-11T01:45:19.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.098+0000 7efc8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:19.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.098+0000 7efc8b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:45:19.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.098+0000 7efc8b7fe640 1 -- 172.21.15.12:0/3772059687 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc8c131520 con 0x7efc8c09df40 2024-09-11T01:45:19.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.101+0000 7efc8b7fe640 1 -- 172.21.15.12:0/3772059687 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc8c09df40 msgr2=0x7efc8c13d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:19.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.101+0000 7efc8b7fe640 1 --2- 172.21.15.12:0/3772059687 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc8c09df40 0x7efc8c13d470 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7efc84009e90 tx=0x7efc8402a6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18c734449954845e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:19.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.101+0000 7efc8a7fc640 1 -- 172.21.15.12:0/3772059687 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc84007950 con 0x7efc8c09df40 2024-09-11T01:45:19.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.101+0000 7efc8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:19.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.101+0000 7efc8a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:45:19.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.101+0000 7efc8a7fc640 10 monclient: dump: 2024-09-11T01:45:19.107 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:19.107 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:19.108 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:19.108 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:19.108 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:19.108 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:19.108 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:19.108 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:19.108 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:19.108 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:19.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.101+0000 7efc8a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:45:19.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.101+0000 7efc8a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:45:19.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.101+0000 7efc8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:19.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.101+0000 7efc9187f640 5 monclient: authenticate success, global_id 134506 2024-09-11T01:45:19.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.101+0000 7efc8a7fc640 1 -- 172.21.15.12:0/3772059687 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc84007af0 con 0x7efc8c09df40 2024-09-11T01:45:19.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.101+0000 7efc9187f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:45:19.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.101+0000 7efc8a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:19.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.101+0000 7efc9187f640 10 monclient: get_monmap_and_config success 2024-09-11T01:45:19.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.101+0000 7efc8a7fc640 1 -- 172.21.15.12:0/3772059687 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc84031400 con 0x7efc8c09df40 2024-09-11T01:45:19.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.101+0000 7efc8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:19.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.101+0000 7efc9187f640 4 set_mon_vals no callback set 2024-09-11T01:45:19.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.101+0000 7efc8a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:45:19.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.101+0000 7efc8a7fc640 10 monclient: dump: 2024-09-11T01:45:19.109 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:19.109 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:19.109 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:19.109 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:19.109 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:19.110 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:19.110 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:19.110 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:19.110 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:19.110 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:19.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.102+0000 7efc9187f640 10 monclient: shutdown 2024-09-11T01:45:19.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.102+0000 7efc9187f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:19.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.102+0000 7efc9187f640 1 -- 172.21.15.12:0/3772059687 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc8c09df40 msgr2=0x7efc8c13d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:19.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.102+0000 7efc9187f640 1 --2- 172.21.15.12:0/3772059687 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc8c09df40 0x7efc8c13d470 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7efc84009e90 tx=0x7efc8402a6d0 comp rx=0 tx=0).stop 2024-09-11T01:45:19.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.102+0000 7efc9187f640 1 -- 172.21.15.12:0/3772059687 shutdown_connections 2024-09-11T01:45:19.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.102+0000 7efc9187f640 1 --2- 172.21.15.12:0/3772059687 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc8c09df40 0x7efc8c13d470 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:19.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.102+0000 7efc9187f640 1 -- 172.21.15.12:0/3772059687 >> 172.21.15.12:0/3772059687 conn(0x7efc8c0a3e30 msgr2=0x7efc8c0a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:19.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.102+0000 7efc9187f640 1 -- 172.21.15.12:0/3772059687 shutdown_connections 2024-09-11T01:45:19.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.102+0000 7efc9187f640 1 -- 172.21.15.12:0/3772059687 wait complete. 2024-09-11T01:45:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.102+0000 7efc9187f640 10 monclient: build_initial_monmap 2024-09-11T01:45:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.102+0000 7efc9187f640 10 monclient: monmap: 2024-09-11T01:45:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:45:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:45:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:45:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:45:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:45:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:45:19.111 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc9187f640 1 Processor -- start 2024-09-11T01:45:19.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc9187f640 1 -- start start 2024-09-11T01:45:19.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc9187f640 10 monclient: init 2024-09-11T01:45:19.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc9187f640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:19.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc9187f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:45:19.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc9187f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc8c09df40 0x7efc8c13c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:19.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc9187f640 10 monclient(hunting): picked mon.noname-b con 0x7efc8c09df40 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:45:19.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc9187f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc8c0a2fb0 0x7efc8c13cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:19.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc9187f640 10 monclient(hunting): picked mon.noname-c con 0x7efc8c0a2fb0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc9187f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc8c131850 0x7efc8c1caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc9187f640 10 monclient(hunting): picked mon.noname-a con 0x7efc8c131850 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:45:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc9187f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc9187f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7efc8c13dd40 con 0x7efc8c131850 2024-09-11T01:45:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc9187f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc9187f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7efc8c13dbc0 con 0x7efc8c09df40 2024-09-11T01:45:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc9187f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc9187f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7efc8c13dec0 con 0x7efc8c0a2fb0 2024-09-11T01:45:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc9187f640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc9187f640 10 monclient(hunting): authenticate will time out at 1725.545837s 2024-09-11T01:45:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc827fc640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc8c0a2fb0 msgr2=0x7efc8c13cee0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:45:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc827fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc8c0a2fb0 0x7efc8c13cee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc8b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc8c131850 0x7efc8c1caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:19.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc8b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc8c131850 0x7efc8c1caeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56776/0 (socket says 172.21.15.12:56776) 2024-09-11T01:45:19.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc8b7fe640 1 -- 172.21.15.12:0/3914589204 learned_addr learned my addr 172.21.15.12:0/3914589204 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:19.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc8b7fe640 10 monclient(hunting): get_auth_request con 0x7efc8c131850 auth_method 0 2024-09-11T01:45:19.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:19.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:19.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:19.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.103+0000 7efc8affd640 1 --2- 172.21.15.12:0/3914589204 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc8c09df40 0x7efc8c13c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:19.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.104+0000 7efc8affd640 10 monclient(hunting): get_auth_request con 0x7efc8c09df40 auth_method 0 2024-09-11T01:45:19.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.104+0000 7efc8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:19.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.104+0000 7efc8affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:19.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.104+0000 7efc8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:19.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.104+0000 7efc8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:19.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.104+0000 7efc8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:19.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.104+0000 7efc8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:19.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.104+0000 7efc8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:19.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.104+0000 7efc8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:19.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.104+0000 7efc8affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:19.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.104+0000 7efc8b7fe640 10 monclient(hunting): handle_auth_done global_id 174174 payload 824 2024-09-11T01:45:19.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.104+0000 7efc8b7fe640 1 -- 172.21.15.12:0/3914589204 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc8c0a2fb0 msgr2=0x7efc8c13cee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:45:19.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.104+0000 7efc8b7fe640 1 --2- 172.21.15.12:0/3914589204 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efc8c0a2fb0 0x7efc8c13cee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:19.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.104+0000 7efc8b7fe640 1 -- 172.21.15.12:0/3914589204 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc8c09df40 msgr2=0x7efc8c13c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:19.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.104+0000 7efc8b7fe640 1 --2- 172.21.15.12:0/3914589204 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc8c09df40 0x7efc8c13c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:19.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.104+0000 7efc8b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:45:19.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.104+0000 7efc8b7fe640 1 monclient: found mon.noname-a 2024-09-11T01:45:19.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.104+0000 7efc8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:19.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.104+0000 7efc8b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:45:19.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.104+0000 7efc8b7fe640 1 -- 172.21.15.12:0/3914589204 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc8c130f80 con 0x7efc8c131850 2024-09-11T01:45:19.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.106+0000 7efc8b7fe640 1 -- 172.21.15.12:0/3914589204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc8c131850 msgr2=0x7efc8c1caeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:19.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.106+0000 7efc8b7fe640 1 --2- 172.21.15.12:0/3914589204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc8c131850 0x7efc8c1caeb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efc84009fb0 tx=0x7efc84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:19.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.106+0000 7efc88ff9640 1 -- 172.21.15.12:0/3914589204 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc84040070 con 0x7efc8c131850 2024-09-11T01:45:19.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.106+0000 7efc88ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:19.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.106+0000 7efc88ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:45:19.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.106+0000 7efc88ff9640 10 monclient: dump: 2024-09-11T01:45:19.122 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:19.122 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:19.123 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:19.123 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:19.123 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:19.123 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:19.123 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:19.123 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:19.123 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:19.123 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:19.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.106+0000 7efc88ff9640 10 monclient: _finish_auth 0 2024-09-11T01:45:19.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.106+0000 7efc88ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:45:19.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.106+0000 7efc88ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:19.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.106+0000 7efc9187f640 5 monclient: authenticate success, global_id 174174 2024-09-11T01:45:19.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.106+0000 7efc88ff9640 1 -- 172.21.15.12:0/3914589204 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc84045660 con 0x7efc8c131850 2024-09-11T01:45:19.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.106+0000 7efc9187f640 10 monclient: _renew_subs 2024-09-11T01:45:19.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.106+0000 7efc9187f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:19.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.106+0000 7efc9187f640 1 -- 172.21.15.12:0/3914589204 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc8c09e520 con 0x7efc8c131850 2024-09-11T01:45:19.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.106+0000 7efc88ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:19.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.106+0000 7efc88ff9640 1 -- 172.21.15.12:0/3914589204 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc84045a30 con 0x7efc8c131850 2024-09-11T01:45:19.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.106+0000 7efc88ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:19.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.106+0000 7efc88ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:19.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.106+0000 7efc88ff9640 10 monclient: dump: 2024-09-11T01:45:19.124 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:19.124 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:19.124 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:19.125 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:19.125 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:19.125 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:19.125 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:19.125 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:19.125 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:19.125 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:19.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.106+0000 7efc9187f640 10 monclient: _renew_subs 2024-09-11T01:45:19.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.106+0000 7efc89ffb640 4 set_mon_vals no callback set 2024-09-11T01:45:19.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.106+0000 7efc9187f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:19.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.106+0000 7efc9187f640 1 -- 172.21.15.12:0/3914589204 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc8c135410 con 0x7efc8c131850 2024-09-11T01:45:19.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.107+0000 7efc9187f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:19.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.107+0000 7efc9187f640 1 --2- 172.21.15.12:0/3914589204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc8c1cb3f0 0x7efc8c1cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:19.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.107+0000 7efc9187f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:19.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.107+0000 7efc9187f640 1 -- 172.21.15.12:0/3914589204 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7efc8c1cbd10 con 0x7efc8c1cb3f0 2024-09-11T01:45:19.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.107+0000 7efc88ff9640 1 -- 172.21.15.12:0/3914589204 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7efc84007960 con 0x7efc8c131850 2024-09-11T01:45:19.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.107+0000 7efc8affd640 1 --2- 172.21.15.12:0/3914589204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc8c1cb3f0 0x7efc8c1cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:19.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.107+0000 7efc8affd640 10 monclient: get_auth_request con 0x7efc8c1cb3f0 auth_method 0 2024-09-11T01:45:19.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.107+0000 7efc8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:19.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.107+0000 7efc8affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:19.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.107+0000 7efc8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:19.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.107+0000 7efc8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:19.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.107+0000 7efc8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:19.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.107+0000 7efc8affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:19.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.107+0000 7efc88ff9640 1 --2- 172.21.15.12:0/3914589204 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efc6c03e1c0 0x7efc6c040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:19.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.107+0000 7efc8affd640 10 monclient(hunting): handle_auth_done global_id 174177 payload 824 2024-09-11T01:45:19.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.107+0000 7efc8affd640 1 -- 172.21.15.12:0/3914589204 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efc8c077070 con 0x7efc8c1cb3f0 2024-09-11T01:45:19.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.107+0000 7efc88ff9640 1 -- 172.21.15.12:0/3914589204 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(538..538 src has 317..538) ==== 4827+0+0 (secure 0 0 0) 0x7efc840747f0 con 0x7efc8c131850 2024-09-11T01:45:19.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.108+0000 7efc827fc640 1 --2- 172.21.15.12:0/3914589204 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efc6c03e1c0 0x7efc6c040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:19.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.108+0000 7efc827fc640 10 monclient: get_auth_request con 0x7efc6c03e1c0 auth_method 0 2024-09-11T01:45:19.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.110+0000 7efc8affd640 1 -- 172.21.15.12:0/3914589204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc8c1cb3f0 msgr2=0x7efc8c1cb7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:19.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.110+0000 7efc8affd640 1 --2- 172.21.15.12:0/3914589204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc8c1cb3f0 0x7efc8c1cb7d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efc7c0014d0 tx=0x7efc7c007410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:19.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.110+0000 7efc88ff9640 1 -- 172.21.15.12:0/3914589204 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7c011870 con 0x7efc8c1cb3f0 2024-09-11T01:45:19.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.110+0000 7efc88ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:19.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.110+0000 7efc88ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:19.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.110+0000 7efc88ff9640 10 monclient: dump: 2024-09-11T01:45:19.127 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:19.128 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:19.128 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:19.128 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:19.128 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:19.128 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:19.128 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:19.128 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:19.128 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:19.128 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:19.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.110+0000 7efc827fc640 1 --2- 172.21.15.12:0/3914589204 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efc6c03e1c0 0x7efc6c040680 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7efc8c133e90 tx=0x7efc74007990 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:19.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.110+0000 7efc88ff9640 1 -- 172.21.15.12:0/3914589204 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7efc7c018070 con 0x7efc8c1cb3f0 2024-09-11T01:45:19.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.110+0000 7efc88ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:19.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.110+0000 7efc88ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:45:19.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.110+0000 7efc88ff9640 1 -- 172.21.15.12:0/3914589204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc8c1cb3f0 msgr2=0x7efc8c1cb7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:19.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.110+0000 7efc88ff9640 1 --2- 172.21.15.12:0/3914589204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc8c1cb3f0 0x7efc8c1cb7d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efc7c0014d0 tx=0x7efc7c007410 comp rx=0 tx=0).stop 2024-09-11T01:45:19.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.110+0000 7efc88ff9640 1 -- 172.21.15.12:0/3914589204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc8c1cb3f0 msgr2=0x7efc8c1cb7d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:19.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.110+0000 7efc88ff9640 1 --2- 172.21.15.12:0/3914589204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc8c1cb3f0 0x7efc8c1cb7d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:19.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.116+0000 7efc9187f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:45:19.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.116+0000 7efc9187f640 1 --2- 172.21.15.12:0/3914589204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc8c12c3f0 0x7efc8c12c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:19.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.116+0000 7efc9187f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:19.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.116+0000 7efc9187f640 1 -- 172.21.15.12:0/3914589204 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7efc8c131c30 con 0x7efc8c12c3f0 2024-09-11T01:45:19.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.116+0000 7efc8affd640 1 --2- 172.21.15.12:0/3914589204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc8c12c3f0 0x7efc8c12c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:19.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.116+0000 7efc8affd640 10 monclient: get_auth_request con 0x7efc8c12c3f0 auth_method 0 2024-09-11T01:45:19.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.116+0000 7efc8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:19.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.116+0000 7efc8affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:19.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.116+0000 7efc8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:19.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.116+0000 7efc8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:19.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.116+0000 7efc8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:19.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.116+0000 7efc8affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:19.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.117+0000 7efc8affd640 10 monclient(hunting): handle_auth_done global_id 174180 payload 824 2024-09-11T01:45:19.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.117+0000 7efc8affd640 1 -- 172.21.15.12:0/3914589204 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efc8c133c10 con 0x7efc8c12c3f0 2024-09-11T01:45:19.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.119+0000 7efc8affd640 1 -- 172.21.15.12:0/3914589204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc8c12c3f0 msgr2=0x7efc8c12c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:19.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.119+0000 7efc8affd640 1 --2- 172.21.15.12:0/3914589204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc8c12c3f0 0x7efc8c12c800 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7efc7c0014d0 tx=0x7efc7c001500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:19.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.119+0000 7efc88ff9640 1 -- 172.21.15.12:0/3914589204 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7c01de20 con 0x7efc8c12c3f0 2024-09-11T01:45:19.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.119+0000 7efc88ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:19.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.119+0000 7efc88ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:19.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.119+0000 7efc88ff9640 10 monclient: dump: 2024-09-11T01:45:19.131 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:19.131 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:19.131 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:19.131 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:19.131 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:19.131 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:19.131 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:19.131 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:19.131 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:19.131 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:19.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.119+0000 7efc88ff9640 1 -- 172.21.15.12:0/3914589204 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4699 (secure 0 0 0) 0x7efc7c013810 con 0x7efc8c12c3f0 2024-09-11T01:45:19.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.119+0000 7efc88ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:45:19.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.119+0000 7efc88ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:45:19.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.119+0000 7efc88ff9640 1 -- 172.21.15.12:0/3914589204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc8c12c3f0 msgr2=0x7efc8c12c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:19.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.119+0000 7efc88ff9640 1 --2- 172.21.15.12:0/3914589204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc8c12c3f0 0x7efc8c12c800 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7efc7c0014d0 tx=0x7efc7c001500 comp rx=0 tx=0).stop 2024-09-11T01:45:19.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.119+0000 7efc88ff9640 1 -- 172.21.15.12:0/3914589204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc8c12c3f0 msgr2=0x7efc8c12c800 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:19.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.119+0000 7efc88ff9640 1 --2- 172.21.15.12:0/3914589204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc8c12c3f0 0x7efc8c12c800 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7efc7c0014d0 tx=0x7efc7c001500 comp rx=0 tx=0).stop 2024-09-11T01:45:19.132 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:45:19.132 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:45:19.133 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:45:19.133 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:45:19.133 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 152, 2024-09-11T01:45:19.133 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 35189, 2024-09-11T01:45:19.133 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:45:19.133 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:45:19.133 INFO:teuthology.orchestra.run.smithi012.stdout: 1 2024-09-11T01:45:19.133 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:19.133 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:45:19.134 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:45:19.134 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:45:19.134 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:45:19.134 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:19.134 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:19.134 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:19.134 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:19.134 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:19.134 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:19.134 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:19.135 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:19.135 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:19.135 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:19.135 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:19.135 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:19.135 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:45:19.135 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:45:19.135 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:19.135 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:19.136 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:19.136 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:19.136 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:19.136 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:19.136 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:19.136 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:19.136 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:19.136 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:19.136 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:19.137 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:19.137 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:19.137 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:45:19.137 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:45:19.137 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:45:19.137 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:45:19.137 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:45:19.137 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:45:19.138 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:45:19.138 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:45:19.138 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:45:19.138 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:45:19.138 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:45:19.138 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:45:19.138 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:45:19.138 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:45:19.138 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:45:19.138 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:45:19.138 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:45:19.138 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:19.138 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:19.139 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:19.139 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:19.139 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:19.139 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:19.139 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:19.139 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:19.139 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:19.139 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:19.139 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:19.139 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:19.139 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:45:19.139 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:19.139 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:45:19.139 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:19.140 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:45:19.140 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:45:19.140 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:19.140 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:19.140 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:19.140 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:19.140 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:45:19.140 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:19.140 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:19.140 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:19.140 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:19.140 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:45:19.140 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:19.140 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:19.141 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:19.141 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:19.141 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:45:19.141 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:45:19.141 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:19.141 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:19.141 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:19.141 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:19.141 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:19.141 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:45:19.141 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:45:19.141 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:19.142 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:19.142 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:19.142 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:19.142 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:45:19.142 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:19.142 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:19.142 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:19.142 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:19.142 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:45:19.142 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:19.142 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:19.142 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:19.142 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:19.142 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:45:19.143 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:45:19.143 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:19.143 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:19.143 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:19.143 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:19.143 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:19.143 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:45:19.143 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:45:19.143 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:19.143 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:19.143 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:19.143 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:19.144 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:45:19.144 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:19.144 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:19.144 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:19.144 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:19.144 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:45:19.144 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:19.144 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:19.145 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:19.145 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:19.145 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:45:19.145 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:45:19.145 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:19.145 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:19.145 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:19.145 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:19.145 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:19.145 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:19.146 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:45:19.146 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:45:19.146 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:19.146 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:19.146 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:19.146 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:45:19.146 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:19.146 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:19.146 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:45:19.146 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:19.146 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:19.146 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:19.146 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:45:19.146 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:19.147 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:19.147 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:45:19.147 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:19.147 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:19.147 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:19.147 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:45:19.147 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:19.147 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:19.147 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:19.147 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:45:19.147 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:45:19.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.120+0000 7efc9187f640 1 -- 172.21.15.12:0/3914589204 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efc6c03e1c0 msgr2=0x7efc6c040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:19.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.120+0000 7efc9187f640 1 --2- 172.21.15.12:0/3914589204 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efc6c03e1c0 0x7efc6c040680 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7efc8c133e90 tx=0x7efc74007990 comp rx=0 tx=0).stop 2024-09-11T01:45:19.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.120+0000 7efc9187f640 10 monclient: shutdown 2024-09-11T01:45:19.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.120+0000 7efc9187f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:19.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.120+0000 7efc9187f640 1 -- 172.21.15.12:0/3914589204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc8c131850 msgr2=0x7efc8c1caeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:19.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.120+0000 7efc9187f640 1 --2- 172.21.15.12:0/3914589204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc8c131850 0x7efc8c1caeb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efc84009fb0 tx=0x7efc84004a40 comp rx=0 tx=0).stop 2024-09-11T01:45:19.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.120+0000 7efc8b7fe640 1 -- 172.21.15.12:0/3914589204 reap_dead start 2024-09-11T01:45:19.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.120+0000 7efc8b7fe640 1 -- 172.21.15.12:0/3914589204 reap_dead start 2024-09-11T01:45:19.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.120+0000 7efc9187f640 1 -- 172.21.15.12:0/3914589204 shutdown_connections 2024-09-11T01:45:19.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.120+0000 7efc9187f640 1 -- 172.21.15.12:0/3914589204 >> 172.21.15.12:0/3914589204 conn(0x7efc8c0a3e30 msgr2=0x7efc8c06cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:19.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.120+0000 7efc9187f640 1 -- 172.21.15.12:0/3914589204 shutdown_connections 2024-09-11T01:45:19.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.120+0000 7efc9187f640 1 -- 172.21.15.12:0/3914589204 wait complete. 2024-09-11T01:45:19.150 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:45:19.150 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T01:45:19.150 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:45:19.236 INFO:tasks.rados.rados.0.smithi071.stdout:2398: finishing write tid 1 to smithi07134777-21 2024-09-11T01:45:19.236 INFO:tasks.rados.rados.0.smithi071.stdout:2394: done (6 left) 2024-09-11T01:45:19.237 INFO:tasks.rados.rados.0.smithi071.stdout:2395: done (5 left) 2024-09-11T01:45:19.237 INFO:tasks.rados.rados.0.smithi071.stdout:2397: done (4 left) 2024-09-11T01:45:19.237 INFO:tasks.rados.rados.0.smithi071.stdout:2401: done (3 left) 2024-09-11T01:45:19.237 INFO:tasks.rados.rados.0.smithi071.stdout:2402: copy_from oid 25 from oid 30 current snap is 266 2024-09-11T01:45:19.237 INFO:tasks.rados.rados.0.smithi071.stdout:2398: finishing write tid 2 to smithi07134777-21 2024-09-11T01:45:19.237 INFO:tasks.rados.rados.0.smithi071.stdout:2398: finishing write tid 3 to smithi07134777-21 2024-09-11T01:45:19.237 INFO:tasks.rados.rados.0.smithi071.stdout:2398: finishing write tid 4 to smithi07134777-21 2024-09-11T01:45:19.237 INFO:tasks.rados.rados.0.smithi071.stdout:2398: oid 21 updating version 0 to 1092 2024-09-11T01:45:19.237 INFO:tasks.rados.rados.0.smithi071.stdout:2398: oid 21 updating version 1092 to 1093 2024-09-11T01:45:19.237 INFO:tasks.rados.rados.0.smithi071.stdout:2398: oid 21 updating version 1093 to 1094 2024-09-11T01:45:19.237 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 1094 (ObjNum 867 snap 266 seq_num 867) dirty exists 2024-09-11T01:45:19.238 INFO:tasks.rados.rados.0.smithi071.stdout:2398: left oid 21 (ObjNum 867 snap 266 seq_num 867) 2024-09-11T01:45:19.238 INFO:tasks.rados.rados.0.smithi071.stdout:2399: expect (ObjNum 829 snap 252 seq_num 829) 2024-09-11T01:45:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.250+0000 7f64f6975640 10 monclient: get_monmap_and_config 2024-09-11T01:45:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.250+0000 7f64f6975640 10 monclient: build_initial_monmap 2024-09-11T01:45:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.250+0000 7f64f6975640 10 monclient: monmap: 2024-09-11T01:45:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:45:19.251384+0000 2024-09-11T01:45:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:45:19.251384+0000 2024-09-11T01:45:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:45:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:45:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:45:19.253 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:45:19.253 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:45:19.253 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:19.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.251+0000 7f64f6975640 1 Processor -- start 2024-09-11T01:45:19.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.251+0000 7f64f6975640 1 -- start start 2024-09-11T01:45:19.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.251+0000 7f64f6975640 10 monclient: init 2024-09-11T01:45:19.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.251+0000 7f64f6975640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:19.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.251+0000 7f64f6975640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-11T01:45:19.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.251+0000 7f64f6975640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f64f00a03a0 addr v1:172.21.15.71:6789/0 2024-09-11T01:45:19.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.251+0000 7f64f6975640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f64f00a2530 0x7f64f009f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:19.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.251+0000 7f64f6975640 10 monclient(hunting): picked mon.noname-b con 0x7f64f00a2530 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:19.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.251+0000 7f64f6975640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f64f009ca30 addr v1:172.21.15.12:6789/0 2024-09-11T01:45:19.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.251+0000 7f64f6975640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64f00a1ab0 con 0x7f64f009ca30 2024-09-11T01:45:19.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.251+0000 7f64f6975640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64f009e510 con 0x7f64f00a03a0 2024-09-11T01:45:19.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.251+0000 7f64f6975640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:19.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.251+0000 7f64f6975640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f64f00a29b0 con 0x7f64f00a2530 2024-09-11T01:45:19.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.251+0000 7f64f6975640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:19.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.251+0000 7f64f6975640 10 monclient(hunting): authenticate will time out at 1725.693969s 2024-09-11T01:45:19.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.251+0000 7f64ef7fe640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f64f00a2530 msgr2=0x7f64f009f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:45:19.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.252+0000 7f64ef7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f64f00a2530 0x7f64f009f1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:19.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.252+0000 7f64f4f12640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f64f009ca30 0x7f64f00a13a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35298/0 (socket says 172.21.15.12:35298) 2024-09-11T01:45:19.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.252+0000 7f64f4f12640 1 -- 172.21.15.12:0/2106289755 learned_addr learned my addr 172.21.15.12:0/2106289755 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:45:19.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.252+0000 7f64effff640 1 --1- 172.21.15.12:0/2106289755 >> v1:172.21.15.71:6789/0 conn(0x7f64f00a03a0 0x7f64f00a0c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.12:55394/0 (socket says 172.21.15.12:55394) 2024-09-11T01:45:19.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64f4f12640 1 -- 172.21.15.12:0/2106289755 >> v1:172.21.15.12:6789/0 conn(0x7f64f009ca30 legacy=0x7f64f00a13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:19.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64effff640 1 -- 172.21.15.12:0/2106289755 >> v1:172.21.15.71:6789/0 conn(0x7f64f00a03a0 legacy=0x7f64f00a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:19.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 1 -- 172.21.15.12:0/2106289755 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1794304843 0 0) 0x7f64f00a1ab0 con 0x7f64f009ca30 2024-09-11T01:45:19.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 10 monclient(hunting): _negotiate 2024-09-11T01:45:19.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:19.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:19.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 10 monclient(hunting): my global_id is 174183 2024-09-11T01:45:19.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 1 -- 172.21.15.12:0/2106289755 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64d8003520 con 0x7f64f009ca30 2024-09-11T01:45:19.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 1 -- 172.21.15.12:0/2106289755 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 369442150 0 0) 0x7f64f009e510 con 0x7f64f00a03a0 2024-09-11T01:45:19.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 10 monclient(hunting): _negotiate 2024-09-11T01:45:19.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:19.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:19.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 10 monclient(hunting): my global_id is 134515 2024-09-11T01:45:19.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 1 -- 172.21.15.12:0/2106289755 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64f00a1ab0 con 0x7f64f00a03a0 2024-09-11T01:45:19.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 1 -- 172.21.15.12:0/2106289755 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 370322963 0 0) 0x7f64f00a1ab0 con 0x7f64f00a03a0 2024-09-11T01:45:19.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 1 -- 172.21.15.12:0/2106289755 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f64f00a2530 msgr2=0x7f64f009f1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:45:19.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 1 --2- 172.21.15.12:0/2106289755 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f64f00a2530 0x7f64f009f1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:19.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 1 -- 172.21.15.12:0/2106289755 >> v1:172.21.15.12:6789/0 conn(0x7f64f009ca30 legacy=0x7f64f00a13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:19.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 10 monclient: _finish_hunting 0 2024-09-11T01:45:19.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 1 monclient: found mon.noname-c-legacy 2024-09-11T01:45:19.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:19.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-09-11T01:45:19.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 1 -- 172.21.15.12:0/2106289755 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64f009edb0 con 0x7f64f00a03a0 2024-09-11T01:45:19.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 10 monclient: _finish_auth 0 2024-09-11T01:45:19.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 10 monclient: _check_auth_tickets 2024-09-11T01:45:19.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:19.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 1 -- 172.21.15.12:0/2106289755 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f64e0001a40 con 0x7f64f00a03a0 2024-09-11T01:45:19.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:19.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-11T01:45:19.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 10 monclient: dump: 2024-09-11T01:45:19.262 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:19.262 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:19.263 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:19.263 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:19.263 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:19.263 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:19.263 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:19.263 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:19.263 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:19.263 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:19.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-09-11T01:45:19.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:19.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 1 -- 172.21.15.12:0/2106289755 >> v1:172.21.15.71:6789/0 conn(0x7f64f00a03a0 legacy=0x7f64f00a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:19.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:45:19.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 1 --2- 172.21.15.12:0/2106289755 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f64f00a2530 0x7f64d8004bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:19.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 10 monclient(hunting): picked mon.c con 0x7f64f00a2530 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:19.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 1 --2- 172.21.15.12:0/2106289755 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64d8005180 0x7f64d80037a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:19.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 10 monclient(hunting): picked mon.a con 0x7f64d8005180 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:45:19.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 1 --2- 172.21.15.12:0/2106289755 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64d8003e30 0x7f64d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:19.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 10 monclient(hunting): picked mon.b con 0x7f64d8003e30 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:45:19.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:19.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 1 -- 172.21.15.12:0/2106289755 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f64d8004620 con 0x7f64d8005180 2024-09-11T01:45:19.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:19.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 1 -- 172.21.15.12:0/2106289755 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f64d800bf30 con 0x7f64d8003e30 2024-09-11T01:45:19.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:19.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 1 -- 172.21.15.12:0/2106289755 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f64d800c130 con 0x7f64f00a2530 2024-09-11T01:45:19.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:19.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64effff640 1 -- 172.21.15.12:0/2106289755 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f64f00a2530 msgr2=0x7f64d8004bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:45:19.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64eeffd640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:45:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64effff640 1 --2- 172.21.15.12:0/2106289755 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f64f00a2530 0x7f64d8004bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64ef7fe640 1 --2- 172.21.15.12:0/2106289755 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64d8005180 0x7f64d80037a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64ef7fe640 10 monclient(hunting): get_auth_request con 0x7f64d8005180 auth_method 0 2024-09-11T01:45:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.254+0000 7f64ef7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:45:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.255+0000 7f64f4f12640 1 --2- 172.21.15.12:0/2106289755 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64d8003e30 0x7f64d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.255+0000 7f64ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.255+0000 7f64ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.255+0000 7f64ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:45:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.255+0000 7f64f4f12640 10 monclient(hunting): get_auth_request con 0x7f64d8003e30 auth_method 0 2024-09-11T01:45:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.255+0000 7f64f4f12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.255+0000 7f64f4f12640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:19.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.255+0000 7f64f4f12640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:45:19.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.255+0000 7f64ef7fe640 10 monclient(hunting): handle_auth_done global_id 134515 payload 293 2024-09-11T01:45:19.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.255+0000 7f64ef7fe640 1 -- 172.21.15.12:0/2106289755 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f64f00a2530 msgr2=0x7f64d8004bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:45:19.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.255+0000 7f64ef7fe640 1 --2- 172.21.15.12:0/2106289755 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f64f00a2530 0x7f64d8004bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:19.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.255+0000 7f64ef7fe640 1 -- 172.21.15.12:0/2106289755 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64d8003e30 msgr2=0x7f64d800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:19.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.255+0000 7f64ef7fe640 1 --2- 172.21.15.12:0/2106289755 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64d8003e30 0x7f64d800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:19.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.255+0000 7f64ef7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:45:19.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.255+0000 7f64ef7fe640 1 monclient: found mon.a 2024-09-11T01:45:19.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.255+0000 7f64ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:19.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.255+0000 7f64ef7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:19.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.255+0000 7f64ef7fe640 1 -- 172.21.15.12:0/2106289755 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64f00a1ab0 con 0x7f64d8005180 2024-09-11T01:45:19.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.255+0000 7f64ef7fe640 10 monclient: _finish_auth 0 2024-09-11T01:45:19.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.255+0000 7f64ef7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:45:19.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.255+0000 7f64ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:19.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.255+0000 7f64f6975640 5 monclient: authenticate success, global_id 134515 2024-09-11T01:45:19.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.255+0000 7f64f6975640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:45:19.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.255+0000 7f64f4f12640 1 --2- 172.21.15.12:0/2106289755 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64d8003e30 0x7f64d800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:45:19.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.258+0000 7f64ef7fe640 1 -- 172.21.15.12:0/2106289755 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64d8005180 msgr2=0x7f64d80037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:19.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.258+0000 7f64ef7fe640 1 --2- 172.21.15.12:0/2106289755 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64d8005180 0x7f64d80037a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f64f00a2910 tx=0x7f64d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47054881bba28961 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:19.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.258+0000 7f64eeffd640 1 -- 172.21.15.12:0/2106289755 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e4009520 con 0x7f64d8005180 2024-09-11T01:45:19.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.258+0000 7f64eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:19.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.258+0000 7f64eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:19.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.258+0000 7f64eeffd640 10 monclient: dump: 2024-09-11T01:45:19.269 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:19.269 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:19.269 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:19.270 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:19.270 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:19.270 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:19.270 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:19.270 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:19.270 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:19.270 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:19.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.258+0000 7f64eeffd640 1 -- 172.21.15.12:0/2106289755 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64e402e070 con 0x7f64d8005180 2024-09-11T01:45:19.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.258+0000 7f64f6975640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:45:19.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.258+0000 7f64eeffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:19.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.258+0000 7f64f6975640 10 monclient: get_monmap_and_config success 2024-09-11T01:45:19.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.258+0000 7f64eeffd640 1 -- 172.21.15.12:0/2106289755 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e402a8e0 con 0x7f64d8005180 2024-09-11T01:45:19.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.258+0000 7f64eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:19.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.258+0000 7f64f6975640 4 set_mon_vals no callback set 2024-09-11T01:45:19.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.258+0000 7f64eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:19.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.258+0000 7f64eeffd640 10 monclient: dump: 2024-09-11T01:45:19.272 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:19.272 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:19.272 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:19.272 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:19.272 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:19.273 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:19.273 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:19.273 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:19.273 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:19.273 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:19.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.258+0000 7f64f6975640 10 monclient: shutdown 2024-09-11T01:45:19.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.258+0000 7f64f6975640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:19.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.258+0000 7f64f6975640 1 -- 172.21.15.12:0/2106289755 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64d8005180 msgr2=0x7f64d80037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:19.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.258+0000 7f64f6975640 1 --2- 172.21.15.12:0/2106289755 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64d8005180 0x7f64d80037a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f64f00a2910 tx=0x7f64d8005830 comp rx=0 tx=0).stop 2024-09-11T01:45:19.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.258+0000 7f64f4f12640 1 -- 172.21.15.12:0/2106289755 reap_dead start 2024-09-11T01:45:19.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.258+0000 7f64f6975640 1 -- 172.21.15.12:0/2106289755 shutdown_connections 2024-09-11T01:45:19.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.258+0000 7f64f6975640 1 -- 172.21.15.12:0/2106289755 >> 172.21.15.12:0/2106289755 conn(0x7f64f00a2f70 msgr2=0x7f64f00a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:19.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.258+0000 7f64f6975640 1 -- 172.21.15.12:0/2106289755 shutdown_connections 2024-09-11T01:45:19.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.258+0000 7f64f6975640 1 -- 172.21.15.12:0/2106289755 wait complete. 2024-09-11T01:45:19.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.259+0000 7f64f6975640 10 monclient: build_initial_monmap 2024-09-11T01:45:19.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.259+0000 7f64f6975640 10 monclient: monmap: 2024-09-11T01:45:19.275 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:19.275 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:45:19.275 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:45:19.275 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:45:19.275 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:19.275 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:19.275 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:45:19.275 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:45:19.276 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:45:19.276 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:19.276 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2737859 2024-09-11T01:45:19.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.259+0000 7f64f6975640 1 Processor -- start 2024-09-11T01:45:19.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.259+0000 7f64f6975640 1 -- start start 2024-09-11T01:45:19.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.259+0000 7f64f6975640 10 monclient: init 2024-09-11T01:45:19.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64f6975640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:19.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64f6975640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:45:19.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64f6975640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64d8005180 0x7f64f0128210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:19.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64f6975640 10 monclient(hunting): picked mon.noname-a con 0x7f64d8005180 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:45:19.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64f6975640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f64f0126150 0x7f64f013f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:19.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64f6975640 10 monclient(hunting): picked mon.noname-c con 0x7f64f0126150 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:19.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64f6975640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64f013f5c0 0x7f64f01cce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:19.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64f6975640 10 monclient(hunting): picked mon.noname-b con 0x7f64f013f5c0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:45:19.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64f6975640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:19.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64f6975640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f64f00a0780 con 0x7f64d8005180 2024-09-11T01:45:19.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64f6975640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:19.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64f6975640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f64f009d070 con 0x7f64f013f5c0 2024-09-11T01:45:19.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64f6975640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:19.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64f6975640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f64f009f1d0 con 0x7f64f0126150 2024-09-11T01:45:19.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64f6975640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:19.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64f6975640 10 monclient(hunting): authenticate will time out at 1725.702367s 2024-09-11T01:45:19.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64ef7fe640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f64f0126150 msgr2=0x7f64f013f080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:45:19.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64ef7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f64f0126150 0x7f64f013f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:19.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64effff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64d8005180 0x7f64f0128210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:19.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64effff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64d8005180 0x7f64f0128210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56806/0 (socket says 172.21.15.12:56806) 2024-09-11T01:45:19.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64effff640 1 -- 172.21.15.12:0/1494770241 learned_addr learned my addr 172.21.15.12:0/1494770241 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:19.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64effff640 10 monclient(hunting): get_auth_request con 0x7f64d8005180 auth_method 0 2024-09-11T01:45:19.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:19.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64effff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:19.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:19.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64f4f12640 1 --2- 172.21.15.12:0/1494770241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64f013f5c0 0x7f64f01cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:19.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:19.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:19.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:19.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64f4f12640 10 monclient(hunting): get_auth_request con 0x7f64f013f5c0 auth_method 0 2024-09-11T01:45:19.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64f4f12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:19.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64f4f12640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:19.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64f4f12640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:19.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64f4f12640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:19.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64f4f12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:19.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64f4f12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:19.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64effff640 10 monclient(hunting): handle_auth_done global_id 174186 payload 824 2024-09-11T01:45:19.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64effff640 1 -- 172.21.15.12:0/1494770241 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f64f0126150 msgr2=0x7f64f013f080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:45:19.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64effff640 1 --2- 172.21.15.12:0/1494770241 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f64f0126150 0x7f64f013f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:19.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64effff640 1 -- 172.21.15.12:0/1494770241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64f013f5c0 msgr2=0x7f64f01cce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:19.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64effff640 1 --2- 172.21.15.12:0/1494770241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64f013f5c0 0x7f64f01cce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:19.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64effff640 10 monclient: _finish_hunting 0 2024-09-11T01:45:19.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64effff640 1 monclient: found mon.noname-a 2024-09-11T01:45:19.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:19.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64effff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:45:19.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.260+0000 7f64effff640 1 -- 172.21.15.12:0/1494770241 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64f01276b0 con 0x7f64d8005180 2024-09-11T01:45:19.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.263+0000 7f64effff640 1 -- 172.21.15.12:0/1494770241 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64d8005180 msgr2=0x7f64f0128210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:19.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.263+0000 7f64effff640 1 --2- 172.21.15.12:0/1494770241 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64d8005180 0x7f64f0128210 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f64e000c730 tx=0x7f64e000cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:19.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.263+0000 7f64ecff9640 1 -- 172.21.15.12:0/1494770241 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e00074e0 con 0x7f64d8005180 2024-09-11T01:45:19.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.263+0000 7f64ecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:19.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.263+0000 7f64ecff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:45:19.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.263+0000 7f64ecff9640 10 monclient: dump: 2024-09-11T01:45:19.287 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:19.287 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:19.287 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:19.287 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:19.287 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:19.287 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:19.288 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:19.288 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:19.288 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:19.288 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:19.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.263+0000 7f64ecff9640 10 monclient: _finish_auth 0 2024-09-11T01:45:19.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.263+0000 7f64ecff9640 10 monclient: _check_auth_tickets 2024-09-11T01:45:19.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.263+0000 7f64ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:19.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.263+0000 7f64ecff9640 1 -- 172.21.15.12:0/1494770241 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64e00047e0 con 0x7f64d8005180 2024-09-11T01:45:19.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.263+0000 7f64f6975640 5 monclient: authenticate success, global_id 174186 2024-09-11T01:45:19.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.263+0000 7f64f6975640 10 monclient: _renew_subs 2024-09-11T01:45:19.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.263+0000 7f64f6975640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:19.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.263+0000 7f64f6975640 1 -- 172.21.15.12:0/1494770241 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64f0127120 con 0x7f64d8005180 2024-09-11T01:45:19.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.263+0000 7f64ecff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:19.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.263+0000 7f64ecff9640 1 -- 172.21.15.12:0/1494770241 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e0004bd0 con 0x7f64d8005180 2024-09-11T01:45:19.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.263+0000 7f64ecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:19.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.263+0000 7f64ecff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:19.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.263+0000 7f64ecff9640 10 monclient: dump: 2024-09-11T01:45:19.289 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:19.289 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:19.289 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:19.289 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:19.289 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:19.290 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:19.290 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:19.290 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:19.290 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:19.290 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:19.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.263+0000 7f64f6975640 10 monclient: _renew_subs 2024-09-11T01:45:19.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.263+0000 7f64f6975640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:19.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.263+0000 7f64f6975640 1 -- 172.21.15.12:0/1494770241 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64f0129170 con 0x7f64d8005180 2024-09-11T01:45:19.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.263+0000 7f64ee7fc640 4 set_mon_vals no callback set 2024-09-11T01:45:19.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.263+0000 7f64f6975640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:19.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.264+0000 7f64f6975640 1 --2- 172.21.15.12:0/1494770241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64f002d440 0x7f64f002d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:19.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.264+0000 7f64f6975640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:19.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.264+0000 7f64f6975640 1 -- 172.21.15.12:0/1494770241 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f64f009f600 con 0x7f64f002d440 2024-09-11T01:45:19.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.264+0000 7f64ef7fe640 1 --2- 172.21.15.12:0/1494770241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64f002d440 0x7f64f002d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:19.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.264+0000 7f64ef7fe640 10 monclient: get_auth_request con 0x7f64f002d440 auth_method 0 2024-09-11T01:45:19.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.264+0000 7f64ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:19.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.264+0000 7f64ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:19.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.264+0000 7f64ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:19.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.264+0000 7f64ecff9640 1 -- 172.21.15.12:0/1494770241 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f64e000a640 con 0x7f64d8005180 2024-09-11T01:45:19.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.264+0000 7f64ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:19.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.264+0000 7f64ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:19.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.264+0000 7f64ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:19.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.264+0000 7f64ecff9640 1 --2- 172.21.15.12:0/1494770241 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f64dc03e240 0x7f64dc040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:19.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.264+0000 7f64ecff9640 1 -- 172.21.15.12:0/1494770241 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(538..538 src has 317..538) ==== 4827+0+0 (secure 0 0 0) 0x7f64e0052bf0 con 0x7f64d8005180 2024-09-11T01:45:19.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.264+0000 7f64ef7fe640 10 monclient(hunting): handle_auth_done global_id 134521 payload 824 2024-09-11T01:45:19.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.264+0000 7f64ef7fe640 1 -- 172.21.15.12:0/1494770241 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f64f009d400 con 0x7f64f002d440 2024-09-11T01:45:19.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.264+0000 7f64f4f12640 1 --2- 172.21.15.12:0/1494770241 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f64dc03e240 0x7f64dc040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:19.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.264+0000 7f64f4f12640 10 monclient: get_auth_request con 0x7f64dc03e240 auth_method 0 2024-09-11T01:45:19.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.266+0000 7f64ef7fe640 1 -- 172.21.15.12:0/1494770241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64f002d440 msgr2=0x7f64f002d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:19.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.266+0000 7f64ef7fe640 1 --2- 172.21.15.12:0/1494770241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64f002d440 0x7f64f002d820 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f64e4032000 tx=0x7f64e4005a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:19.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.267+0000 7f64ecff9640 1 -- 172.21.15.12:0/1494770241 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e4044070 con 0x7f64f002d440 2024-09-11T01:45:19.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.267+0000 7f64ecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:19.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.267+0000 7f64ecff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:45:19.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.267+0000 7f64ecff9640 10 monclient: dump: 2024-09-11T01:45:19.292 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:19.292 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:19.292 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:19.292 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:19.293 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:19.293 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:19.293 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:19.293 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:19.293 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:19.293 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:19.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.267+0000 7f64f4f12640 1 --2- 172.21.15.12:0/1494770241 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f64dc03e240 0x7f64dc040700 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f64d800c300 tx=0x7f64e8001990 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:19.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.267+0000 7f64ecff9640 1 -- 172.21.15.12:0/1494770241 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f64e402e040 con 0x7f64f002d440 2024-09-11T01:45:19.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.267+0000 7f64ecff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:19.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.267+0000 7f64ecff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:45:19.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.267+0000 7f64ecff9640 1 -- 172.21.15.12:0/1494770241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64f002d440 msgr2=0x7f64f002d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:19.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.267+0000 7f64ecff9640 1 --2- 172.21.15.12:0/1494770241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64f002d440 0x7f64f002d820 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f64e4032000 tx=0x7f64e4005a00 comp rx=0 tx=0).stop 2024-09-11T01:45:19.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.267+0000 7f64ecff9640 1 -- 172.21.15.12:0/1494770241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64f002d440 msgr2=0x7f64f002d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:19.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.267+0000 7f64ecff9640 1 --2- 172.21.15.12:0/1494770241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64f002d440 0x7f64f002d820 secure :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f64e4032000 tx=0x7f64e4005a00 comp rx=0 tx=0).stop 2024-09-11T01:45:19.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.273+0000 7f64f6975640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:45:19.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.273+0000 7f64f6975640 1 --2- 172.21.15.12:0/1494770241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64f002dd60 0x7f64f002e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:19.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.273+0000 7f64f6975640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:19.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.273+0000 7f64f6975640 1 -- 172.21.15.12:0/1494770241 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f64f011dff0 con 0x7f64f002dd60 2024-09-11T01:45:19.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.273+0000 7f64ef7fe640 1 --2- 172.21.15.12:0/1494770241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64f002dd60 0x7f64f002e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:19.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.273+0000 7f64ef7fe640 10 monclient: get_auth_request con 0x7f64f002dd60 auth_method 0 2024-09-11T01:45:19.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.273+0000 7f64ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:19.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.273+0000 7f64ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:19.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.273+0000 7f64ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:19.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.273+0000 7f64ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:19.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.273+0000 7f64ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:19.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.273+0000 7f64ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:19.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.273+0000 7f64ef7fe640 10 monclient(hunting): handle_auth_done global_id 134524 payload 824 2024-09-11T01:45:19.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.273+0000 7f64ef7fe640 1 -- 172.21.15.12:0/1494770241 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f64f0129530 con 0x7f64f002dd60 2024-09-11T01:45:19.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.276+0000 7f64ef7fe640 1 -- 172.21.15.12:0/1494770241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64f002dd60 msgr2=0x7f64f002e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:19.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.276+0000 7f64ef7fe640 1 --2- 172.21.15.12:0/1494770241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64f002dd60 0x7f64f002e140 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f64e4000c00 tx=0x7f64e40090e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:19.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.276+0000 7f64ecff9640 1 -- 172.21.15.12:0/1494770241 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e4044050 con 0x7f64f002dd60 2024-09-11T01:45:19.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.276+0000 7f64ecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:19.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.276+0000 7f64ecff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:45:19.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.276+0000 7f64ecff9640 10 monclient: dump: 2024-09-11T01:45:19.295 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:19.295 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:19.295 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:19.295 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:19.295 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:19.296 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:19.296 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:19.296 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:19.296 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:19.296 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:19.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.276+0000 7f64ecff9640 1 -- 172.21.15.12:0/1494770241 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4865 (secure 0 0 0) 0x7f64e4045040 con 0x7f64f002dd60 2024-09-11T01:45:19.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.276+0000 7f64ecff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:45:19.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.276+0000 7f64ecff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:45:19.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.276+0000 7f64ecff9640 1 -- 172.21.15.12:0/1494770241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64f002dd60 msgr2=0x7f64f002e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:19.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.276+0000 7f64ecff9640 1 --2- 172.21.15.12:0/1494770241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64f002dd60 0x7f64f002e140 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f64e4000c00 tx=0x7f64e40090e0 comp rx=0 tx=0).stop 2024-09-11T01:45:19.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.276+0000 7f64ecff9640 1 -- 172.21.15.12:0/1494770241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64f002dd60 msgr2=0x7f64f002e140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:19.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.276+0000 7f64ecff9640 1 --2- 172.21.15.12:0/1494770241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64f002dd60 0x7f64f002e140 secure :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f64e4000c00 tx=0x7f64e40090e0 comp rx=0 tx=0).stop 2024-09-11T01:45:19.297 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:45:19.297 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:45:19.297 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:45:19.297 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:45:19.297 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 152, 2024-09-11T01:45:19.297 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 252239, 2024-09-11T01:45:19.297 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:45:19.297 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:45:19.297 INFO:teuthology.orchestra.run.smithi012.stdout: 1 2024-09-11T01:45:19.297 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:19.297 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:45:19.297 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:45:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:45:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:45:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:19.298 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:19.299 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:45:19.299 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:45:19.299 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:19.299 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:19.299 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:19.299 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:19.299 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:19.299 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:19.299 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:19.299 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:19.299 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:19.299 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:19.299 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:19.299 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:19.300 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:19.300 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:45:19.300 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:45:19.300 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:45:19.300 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:45:19.300 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:45:19.300 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:45:19.300 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:45:19.300 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:45:19.300 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:45:19.300 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:45:19.300 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:45:19.300 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:45:19.300 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:45:19.301 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:45:19.301 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:45:19.301 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:45:19.301 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:45:19.301 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:19.301 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:19.301 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:19.301 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:19.301 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:19.301 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:19.301 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:19.301 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:19.301 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:19.302 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:19.302 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:19.302 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:19.302 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:45:19.302 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:19.302 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:45:19.302 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:19.302 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:45:19.302 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:45:19.302 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:19.302 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:19.302 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:19.302 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:19.302 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:45:19.302 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:19.303 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:19.303 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:19.303 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:19.303 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:45:19.303 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:19.303 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:19.303 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:19.303 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:19.303 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:45:19.303 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:45:19.303 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:19.303 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:19.303 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:19.304 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:19.304 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:19.304 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:45:19.304 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:45:19.304 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:19.304 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:19.304 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:19.304 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:19.304 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:45:19.304 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:19.304 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:19.304 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:19.304 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:19.304 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:45:19.305 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:19.305 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:19.305 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:19.305 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:19.305 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:45:19.305 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:45:19.305 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:19.305 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:19.305 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:19.305 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:19.305 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:19.305 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:45:19.305 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:45:19.305 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:19.306 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:19.306 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:19.306 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:19.306 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:45:19.306 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:19.306 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:19.306 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:19.306 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:19.306 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:45:19.306 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:19.306 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:19.306 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:19.306 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:19.306 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:45:19.306 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:45:19.307 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:19.307 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:19.307 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:19.307 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:19.307 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:19.307 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:19.307 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:45:19.307 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:45:19.307 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:19.307 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:19.307 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:19.307 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:45:19.307 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:19.307 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:19.308 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:45:19.308 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:19.308 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:19.308 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:19.308 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:45:19.308 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:19.308 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:19.308 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:45:19.308 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:19.308 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:19.308 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:19.308 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:45:19.308 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:19.308 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:19.309 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:45:19.309 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:19.309 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:19.309 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:19.309 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:45:19.309 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:19.309 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:19.309 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:19.309 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:45:19.309 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:45:19.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.277+0000 7f64f6975640 1 -- 172.21.15.12:0/1494770241 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f64dc03e240 msgr2=0x7f64dc040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:19.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.277+0000 7f64f6975640 1 --2- 172.21.15.12:0/1494770241 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f64dc03e240 0x7f64dc040700 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f64d800c300 tx=0x7f64e8001990 comp rx=0 tx=0).stop 2024-09-11T01:45:19.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.277+0000 7f64f6975640 10 monclient: shutdown 2024-09-11T01:45:19.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.277+0000 7f64f6975640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:19.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.277+0000 7f64f6975640 1 -- 172.21.15.12:0/1494770241 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64d8005180 msgr2=0x7f64f0128210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:19.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.277+0000 7f64f6975640 1 --2- 172.21.15.12:0/1494770241 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f64d8005180 0x7f64f0128210 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f64e000c730 tx=0x7f64e000cb30 comp rx=0 tx=0).stop 2024-09-11T01:45:19.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.277+0000 7f64f4f12640 1 -- 172.21.15.12:0/1494770241 reap_dead start 2024-09-11T01:45:19.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.277+0000 7f64f4f12640 1 -- 172.21.15.12:0/1494770241 reap_dead start 2024-09-11T01:45:19.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.277+0000 7f64f6975640 1 -- 172.21.15.12:0/1494770241 shutdown_connections 2024-09-11T01:45:19.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.277+0000 7f64f6975640 1 -- 172.21.15.12:0/1494770241 >> 172.21.15.12:0/1494770241 conn(0x7f64f00a2f70 msgr2=0x7f64f00a49b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:19.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.277+0000 7f64f6975640 1 -- 172.21.15.12:0/1494770241 shutdown_connections 2024-09-11T01:45:19.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:19.277+0000 7f64f6975640 1 -- 172.21.15.12:0/1494770241 wait complete. 2024-09-11T01:45:19.312 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:45:19.312 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:45:19.404 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2826456 2024-09-11T01:45:19.425 INFO:tasks.rados.rados.0.smithi071.stdout:2400: finishing write tid 1 to smithi07134777-4 2024-09-11T01:45:19.425 INFO:tasks.rados.rados.0.smithi071.stdout:2400: finishing write tid 2 to smithi07134777-4 2024-09-11T01:45:19.425 INFO:tasks.rados.rados.0.smithi071.stdout:2400: finishing write tid 3 to smithi07134777-4 2024-09-11T01:45:19.425 INFO:tasks.rados.rados.0.smithi071.stdout:2400: finishing write tid 4 to smithi07134777-4 2024-09-11T01:45:19.425 INFO:tasks.rados.rados.0.smithi071.stdout:2400: oid 4 updating version 0 to 1198 2024-09-11T01:45:19.426 INFO:tasks.rados.rados.0.smithi071.stdout:2400: oid 4 version 1198 is already newer than 1197 2024-09-11T01:45:19.426 INFO:tasks.rados.rados.0.smithi071.stdout:2400: oid 4 version 1198 is already newer than 1196 2024-09-11T01:45:19.426 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 1198 (ObjNum 868 snap 266 seq_num 868) dirty exists 2024-09-11T01:45:19.426 INFO:tasks.rados.rados.0.smithi071.stdout:2400: left oid 4 (ObjNum 868 snap 266 seq_num 868) 2024-09-11T01:45:19.426 INFO:tasks.rados.rados.0.smithi071.stdout:2398: done (3 left) 2024-09-11T01:45:19.426 INFO:tasks.rados.rados.0.smithi071.stdout:2399: done (2 left) 2024-09-11T01:45:19.426 INFO:tasks.rados.rados.0.smithi071.stdout:2400: done (1 left) 2024-09-11T01:45:19.426 INFO:tasks.rados.rados.0.smithi071.stdout:2403: read oid 49 snap 243 2024-09-11T01:45:19.426 INFO:tasks.rados.rados.0.smithi071.stdout:2403: expect deleted 2024-09-11T01:45:19.426 INFO:tasks.rados.rados.0.smithi071.stdout:2404: copy_from oid 7 from oid 26 current snap is 266 2024-09-11T01:45:19.426 INFO:tasks.rados.rados.0.smithi071.stdout:2402: finishing copy_from to smithi07134777-25 2024-09-11T01:45:19.426 INFO:tasks.rados.rados.0.smithi071.stdout:2402: got expected ENOENT (src dne) 2024-09-11T01:45:19.426 INFO:tasks.rados.rados.0.smithi071.stdout:2402: finishing copy_from racing read to smithi07134777-25 2024-09-11T01:45:19.426 INFO:tasks.rados.rados.0.smithi071.stdout:2402: done (2 left) 2024-09-11T01:45:19.426 INFO:tasks.rados.rados.0.smithi071.stdout:2405: copy_from oid 5 from oid 44 current snap is 266 2024-09-11T01:45:19.427 INFO:tasks.rados.rados.0.smithi071.stdout:2406: snap_create 2024-09-11T01:45:19.432 INFO:tasks.rados.rados.0.smithi071.stdout:2405: finishing copy_from racing read to smithi07134777-5 2024-09-11T01:45:19.435 INFO:tasks.rados.rados.0.smithi071.stdout:2404: finishing copy_from racing read to smithi07134777-7 2024-09-11T01:45:19.435 INFO:tasks.rados.rados.0.smithi071.stdout:2405: finishing copy_from to smithi07134777-5 2024-09-11T01:45:19.435 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 997 (ObjNum 742 snap 226 seq_num 742) dirty exists 2024-09-11T01:45:19.437 INFO:tasks.rados.rados.0.smithi071.stdout:2404: finishing copy_from to smithi07134777-7 2024-09-11T01:45:19.438 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 1271 (ObjNum 835 snap 254 seq_num 835) dirty exists 2024-09-11T01:45:19.956 INFO:tasks.rados.rados.0.smithi071.stdout:2403: done (3 left) 2024-09-11T01:45:19.957 INFO:tasks.rados.rados.0.smithi071.stdout:2404: done (2 left) 2024-09-11T01:45:19.957 INFO:tasks.rados.rados.0.smithi071.stdout:2405: done (1 left) 2024-09-11T01:45:19.957 INFO:tasks.rados.rados.0.smithi071.stdout:2406: done (0 left) 2024-09-11T01:45:19.957 INFO:tasks.rados.rados.0.smithi071.stdout:2407: read oid 9 snap -1 2024-09-11T01:45:19.957 INFO:tasks.rados.rados.0.smithi071.stdout:2407: expect (ObjNum 652 snap 199 seq_num 652) 2024-09-11T01:45:19.957 INFO:tasks.rados.rados.0.smithi071.stdout:2408: copy_from oid 22 from oid 50 current snap is 267 2024-09-11T01:45:19.957 INFO:tasks.rados.rados.0.smithi071.stdout:2409: read oid 15 snap -1 2024-09-11T01:45:19.957 INFO:tasks.rados.rados.0.smithi071.stdout:2409: expect deleted 2024-09-11T01:45:19.957 INFO:tasks.rados.rados.0.smithi071.stdout:2410: write oid 16 current snap is 267 2024-09-11T01:45:19.957 INFO:tasks.rados.rados.0.smithi071.stdout:2410: seq_num 869 ranges {605283=655623,1701911=747296} 2024-09-11T01:45:19.963 INFO:tasks.rados.rados.0.smithi071.stdout:2410: writing smithi07134777-16 from 605283 to 1260906 tid 1 2024-09-11T01:45:19.970 INFO:tasks.rados.rados.0.smithi071.stdout:2410: writing smithi07134777-16 from 1701911 to 2449207 tid 2 2024-09-11T01:45:19.970 INFO:tasks.rados.rados.0.smithi071.stdout:2411: rollback oid 11 current snap is 267 2024-09-11T01:45:19.970 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 11 to 225 2024-09-11T01:45:19.970 INFO:tasks.rados.rados.0.smithi071.stdout:2408: finishing copy_from to smithi07134777-22 2024-09-11T01:45:19.970 INFO:tasks.rados.rados.0.smithi071.stdout:2408: got expected ENOENT (src dne) 2024-09-11T01:45:19.970 INFO:tasks.rados.rados.0.smithi071.stdout:2408: finishing copy_from racing read to smithi07134777-22 2024-09-11T01:45:19.970 INFO:tasks.rados.rados.0.smithi071.stdout:2407: expect (ObjNum 652 snap 199 seq_num 652) 2024-09-11T01:45:20.006 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2387302 2024-09-11T01:45:20.128 INFO:tasks.rados.rados.0.smithi071.stdout:2410: finishing write tid 1 to smithi07134777-16 2024-09-11T01:45:20.128 INFO:tasks.rados.rados.0.smithi071.stdout:2410: finishing write tid 2 to smithi07134777-16 2024-09-11T01:45:20.128 INFO:tasks.rados.rados.0.smithi071.stdout:2410: finishing write tid 3 to smithi07134777-16 2024-09-11T01:45:20.128 INFO:tasks.rados.rados.0.smithi071.stdout:2410: finishing write tid 4 to smithi07134777-16 2024-09-11T01:45:20.128 INFO:tasks.rados.rados.0.smithi071.stdout:2410: oid 16 updating version 0 to 1105 2024-09-11T01:45:20.128 INFO:tasks.rados.rados.0.smithi071.stdout:2410: oid 16 version 1105 is already newer than 1104 2024-09-11T01:45:20.129 INFO:tasks.rados.rados.0.smithi071.stdout:2410: oid 16 updating version 1105 to 1106 2024-09-11T01:45:20.129 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 1106 (ObjNum 869 snap 267 seq_num 869) dirty exists 2024-09-11T01:45:20.129 INFO:tasks.rados.rados.0.smithi071.stdout:2410: left oid 16 (ObjNum 869 snap 267 seq_num 869) 2024-09-11T01:45:20.129 INFO:tasks.rados.rados.0.smithi071.stdout:2411: finishing rollback tid 0 to smithi07134777-11 2024-09-11T01:45:20.129 INFO:tasks.rados.rados.0.smithi071.stdout:2407: done (4 left) 2024-09-11T01:45:20.129 INFO:tasks.rados.rados.0.smithi071.stdout:2408: done (3 left) 2024-09-11T01:45:20.129 INFO:tasks.rados.rados.0.smithi071.stdout:2409: done (2 left) 2024-09-11T01:45:20.129 INFO:tasks.rados.rados.0.smithi071.stdout:2410: done (1 left) 2024-09-11T01:45:20.129 INFO:tasks.rados.rados.0.smithi071.stdout:2412: rollback oid 1 current snap is 267 2024-09-11T01:45:20.129 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 1 to 262 2024-09-11T01:45:20.129 INFO:tasks.rados.rados.0.smithi071.stdout:2411: finishing rollback tid 1 to smithi07134777-11 2024-09-11T01:45:20.129 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 1200 (ObjNum 177 snap 0 seq_num 2371367104) dirty dne 2024-09-11T01:45:20.129 INFO:tasks.rados.rados.0.smithi071.stdout:2411: done (1 left) 2024-09-11T01:45:20.129 INFO:tasks.rados.rados.0.smithi071.stdout:2413: rollback oid 15 current snap is 267 2024-09-11T01:45:20.130 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 15 to 261 2024-09-11T01:45:20.130 INFO:tasks.rados.rados.0.smithi071.stdout:2414: write oid 35 current snap is 267 2024-09-11T01:45:20.130 INFO:tasks.rados.rados.0.smithi071.stdout:2414: seq_num 870 ranges {622090=730872,1760393=607307} 2024-09-11T01:45:20.135 INFO:tasks.rados.rados.0.smithi071.stdout:2414: writing smithi07134777-35 from 622090 to 1352962 tid 1 2024-09-11T01:45:20.142 INFO:tasks.rados.rados.0.smithi071.stdout:2414: writing smithi07134777-35 from 1760393 to 2367700 tid 2 2024-09-11T01:45:20.142 INFO:tasks.rados.rados.0.smithi071.stdout:2415: copy_from oid 26 from oid 28 current snap is 267 2024-09-11T01:45:20.142 INFO:tasks.rados.rados.0.smithi071.stdout:2413: finishing rollback tid 1 to smithi07134777-15 2024-09-11T01:45:20.142 INFO:tasks.rados.rados.0.smithi071.stdout:2413: finishing rollback tid 2 to smithi07134777-15 2024-09-11T01:45:20.142 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 1545 (ObjNum 808 snap 246 seq_num 808) dirty exists 2024-09-11T01:45:20.142 INFO:tasks.rados.rados.0.smithi071.stdout:2412: finishing rollback tid 0 to smithi07134777-1 2024-09-11T01:45:20.142 INFO:tasks.rados.rados.0.smithi071.stdout:2412: finishing rollback tid 1 to smithi07134777-1 2024-09-11T01:45:20.142 INFO:tasks.rados.rados.0.smithi071.stdout:2412: finishing rollback tid 2 to smithi07134777-1 2024-09-11T01:45:20.142 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 1067 (ObjNum 688 snap 204 seq_num 688) dirty exists 2024-09-11T01:45:20.142 INFO:tasks.rados.rados.0.smithi071.stdout:2412: done (3 left) 2024-09-11T01:45:20.142 INFO:tasks.rados.rados.0.smithi071.stdout:2413: done (2 left) 2024-09-11T01:45:20.143 INFO:tasks.rados.rados.0.smithi071.stdout:2416: write (excl) oid 30 current snap is 267 2024-09-11T01:45:20.143 INFO:tasks.rados.rados.0.smithi071.stdout:2416: seq_num 871 ranges {638897=406121,1818875=750965} 2024-09-11T01:45:20.147 INFO:tasks.rados.rados.0.smithi071.stdout:2416: writing smithi07134777-30 from 638897 to 1045018 tid 1 2024-09-11T01:45:20.155 INFO:tasks.rados.rados.0.smithi071.stdout:2416: writing smithi07134777-30 from 1818875 to 2569840 tid 2 2024-09-11T01:45:20.156 INFO:tasks.rados.rados.0.smithi071.stdout:2417: write (excl) oid 18 current snap is 267 2024-09-11T01:45:20.156 INFO:tasks.rados.rados.0.smithi071.stdout:2417: seq_num 872 ranges {655704=481370,1761004=494623} 2024-09-11T01:45:20.161 INFO:tasks.rados.rados.0.smithi071.stdout:2417: writing smithi07134777-18 from 655704 to 1137074 tid 1 2024-09-11T01:45:20.169 INFO:tasks.rados.rados.0.smithi071.stdout:2417: writing smithi07134777-18 from 1761004 to 2255627 tid 2 2024-09-11T01:45:20.169 INFO:tasks.rados.rados.0.smithi071.stdout:2418: write oid 15 current snap is 267 2024-09-11T01:45:20.169 INFO:tasks.rados.rados.0.smithi071.stdout:2418: seq_num 873 ranges {672511=556619,1819486=754634} 2024-09-11T01:45:20.174 INFO:tasks.rados.rados.0.smithi071.stdout:2418: writing smithi07134777-15 from 672511 to 1229130 tid 1 2024-09-11T01:45:20.182 INFO:tasks.rados.rados.0.smithi071.stdout:2418: writing smithi07134777-15 from 1819486 to 2574120 tid 2 2024-09-11T01:45:20.182 INFO:tasks.rados.rados.0.smithi071.stdout:2419: copy_from oid 4 from oid 49 current snap is 267 2024-09-11T01:45:20.182 INFO:tasks.rados.rados.0.smithi071.stdout:2414: finishing write tid 1 to smithi07134777-35 2024-09-11T01:45:20.182 INFO:tasks.rados.rados.0.smithi071.stdout:2414: finishing write tid 2 to smithi07134777-35 2024-09-11T01:45:20.182 INFO:tasks.rados.rados.0.smithi071.stdout:2414: finishing write tid 3 to smithi07134777-35 2024-09-11T01:45:20.182 INFO:tasks.rados.rados.0.smithi071.stdout:2414: finishing write tid 4 to smithi07134777-35 2024-09-11T01:45:20.183 INFO:tasks.rados.rados.0.smithi071.stdout:2414: oid 35 updating version 0 to 1020 2024-09-11T01:45:20.183 INFO:tasks.rados.rados.0.smithi071.stdout:2414: oid 35 version 1020 is already newer than 1018 2024-09-11T01:45:20.183 INFO:tasks.rados.rados.0.smithi071.stdout:2414: oid 35 version 1020 is already newer than 1019 2024-09-11T01:45:20.183 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 1020 (ObjNum 870 snap 267 seq_num 870) dirty exists 2024-09-11T01:45:20.183 INFO:tasks.rados.rados.0.smithi071.stdout:2414: left oid 35 (ObjNum 870 snap 267 seq_num 870) 2024-09-11T01:45:20.183 INFO:tasks.rados.rados.0.smithi071.stdout:2416: finishing write tid 1 to smithi07134777-30 2024-09-11T01:45:20.183 INFO:tasks.rados.rados.0.smithi071.stdout:2415: finishing copy_from racing read to smithi07134777-26 2024-09-11T01:45:20.183 INFO:tasks.rados.rados.0.smithi071.stdout:2415: finishing copy_from to smithi07134777-26 2024-09-11T01:45:20.183 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 1090 (ObjNum 863 snap 265 seq_num 863) dirty exists 2024-09-11T01:45:20.183 INFO:tasks.rados.rados.0.smithi071.stdout:2416: finishing write tid 2 to smithi07134777-30 2024-09-11T01:45:20.183 INFO:tasks.rados.rados.0.smithi071.stdout:2416: finishing write tid 3 to smithi07134777-30 2024-09-11T01:45:20.183 INFO:tasks.rados.rados.0.smithi071.stdout:2416: finishing write tid 4 to smithi07134777-30 2024-09-11T01:45:20.183 INFO:tasks.rados.rados.0.smithi071.stdout:2416: oid 30 updating version 0 to 1202 2024-09-11T01:45:20.184 INFO:tasks.rados.rados.0.smithi071.stdout:2416: oid 30 version 1202 is already newer than 1201 2024-09-11T01:45:20.184 INFO:tasks.rados.rados.0.smithi071.stdout:2416: oid 30 updating version 1202 to 1203 2024-09-11T01:45:20.184 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 1203 (ObjNum 871 snap 267 seq_num 871) dirty exists 2024-09-11T01:45:20.184 INFO:tasks.rados.rados.0.smithi071.stdout:2416: left oid 30 (ObjNum 871 snap 267 seq_num 871) 2024-09-11T01:45:20.184 INFO:tasks.rados.rados.0.smithi071.stdout:2417: finishing write tid 1 to smithi07134777-18 2024-09-11T01:45:20.184 INFO:tasks.rados.rados.0.smithi071.stdout:2414: done (5 left) 2024-09-11T01:45:20.184 INFO:tasks.rados.rados.0.smithi071.stdout:2415: done (4 left) 2024-09-11T01:45:20.184 INFO:tasks.rados.rados.0.smithi071.stdout:2416: done (3 left) 2024-09-11T01:45:20.184 INFO:tasks.rados.rados.0.smithi071.stdout:2420: delete oid 2 current snap is 267 2024-09-11T01:45:20.184 INFO:tasks.rados.rados.0.smithi071.stdout:2417: finishing write tid 2 to smithi07134777-18 2024-09-11T01:45:20.184 INFO:tasks.rados.rados.0.smithi071.stdout:2417: finishing write tid 3 to smithi07134777-18 2024-09-11T01:45:20.184 INFO:tasks.rados.rados.0.smithi071.stdout:2417: finishing write tid 4 to smithi07134777-18 2024-09-11T01:45:20.184 INFO:tasks.rados.rados.0.smithi071.stdout:2417: oid 18 updating version 0 to 1206 2024-09-11T01:45:20.185 INFO:tasks.rados.rados.0.smithi071.stdout:2417: oid 18 version 1206 is already newer than 1205 2024-09-11T01:45:20.185 INFO:tasks.rados.rados.0.smithi071.stdout:2417: oid 18 version 1206 is already newer than 1204 2024-09-11T01:45:20.185 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 1206 (ObjNum 872 snap 267 seq_num 872) dirty exists 2024-09-11T01:45:20.185 INFO:tasks.rados.rados.0.smithi071.stdout:2417: left oid 18 (ObjNum 872 snap 267 seq_num 872) 2024-09-11T01:45:20.185 INFO:tasks.rados.rados.0.smithi071.stdout:2418: finishing write tid 1 to smithi07134777-15 2024-09-11T01:45:20.185 INFO:tasks.rados.rados.0.smithi071.stdout:2417: done (3 left) 2024-09-11T01:45:20.185 INFO:tasks.rados.rados.0.smithi071.stdout:2420: done (2 left) 2024-09-11T01:45:20.185 INFO:tasks.rados.rados.0.smithi071.stdout:2421: write (excl) oid 2 current snap is 267 2024-09-11T01:45:20.186 INFO:tasks.rados.rados.0.smithi071.stdout:2421: seq_num 875 ranges {706125=423470,1652803=758303} 2024-09-11T01:45:20.190 INFO:tasks.rados.rados.0.smithi071.stdout:2421: writing smithi07134777-2 from 706125 to 1129595 tid 1 2024-09-11T01:45:20.198 INFO:tasks.rados.rados.0.smithi071.stdout:2421: writing smithi07134777-2 from 1652803 to 2411106 tid 2 2024-09-11T01:45:20.198 INFO:tasks.rados.rados.0.smithi071.stdout:2422: read oid 36 snap -1 2024-09-11T01:45:20.198 INFO:tasks.rados.rados.0.smithi071.stdout:2422: expect (ObjNum 829 snap 252 seq_num 829) 2024-09-11T01:45:20.198 INFO:tasks.rados.rados.0.smithi071.stdout:2423: snap_create 2024-09-11T01:45:20.198 INFO:tasks.rados.rados.0.smithi071.stdout:2418: finishing write tid 2 to smithi07134777-15 2024-09-11T01:45:20.199 INFO:tasks.rados.rados.0.smithi071.stdout:2418: finishing write tid 3 to smithi07134777-15 2024-09-11T01:45:20.199 INFO:tasks.rados.rados.0.smithi071.stdout:2418: finishing write tid 4 to smithi07134777-15 2024-09-11T01:45:20.199 INFO:tasks.rados.rados.0.smithi071.stdout:2418: oid 15 updating version 0 to 1546 2024-09-11T01:45:20.199 INFO:tasks.rados.rados.0.smithi071.stdout:2418: oid 15 updating version 1546 to 1547 2024-09-11T01:45:20.199 INFO:tasks.rados.rados.0.smithi071.stdout:2418: oid 15 updating version 1547 to 1548 2024-09-11T01:45:20.199 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 1548 (ObjNum 873 snap 267 seq_num 873) dirty exists 2024-09-11T01:45:20.199 INFO:tasks.rados.rados.0.smithi071.stdout:2418: left oid 15 (ObjNum 873 snap 267 seq_num 873) 2024-09-11T01:45:20.199 INFO:tasks.rados.rados.0.smithi071.stdout:2419: finishing copy_from racing read to smithi07134777-4 2024-09-11T01:45:20.199 INFO:tasks.rados.rados.0.smithi071.stdout:2419: finishing copy_from to smithi07134777-4 2024-09-11T01:45:20.199 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 371 (ObjNum 828 snap 252 seq_num 828) dirty exists 2024-09-11T01:45:20.199 INFO:tasks.rados.rados.0.smithi071.stdout:2421: finishing write tid 1 to smithi07134777-2 2024-09-11T01:45:20.202 INFO:tasks.rados.rados.0.smithi071.stdout:2421: finishing write tid 2 to smithi07134777-2 2024-09-11T01:45:20.202 INFO:tasks.rados.rados.0.smithi071.stdout:2421: finishing write tid 3 to smithi07134777-2 2024-09-11T01:45:20.202 INFO:tasks.rados.rados.0.smithi071.stdout:2421: finishing write tid 4 to smithi07134777-2 2024-09-11T01:45:20.202 INFO:tasks.rados.rados.0.smithi071.stdout:2421: oid 2 updating version 0 to 853 2024-09-11T01:45:20.202 INFO:tasks.rados.rados.0.smithi071.stdout:2421: oid 2 updating version 853 to 854 2024-09-11T01:45:20.202 INFO:tasks.rados.rados.0.smithi071.stdout:2421: oid 2 version 854 is already newer than 852 2024-09-11T01:45:20.202 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 854 (ObjNum 875 snap 267 seq_num 875) dirty exists 2024-09-11T01:45:20.202 INFO:tasks.rados.rados.0.smithi071.stdout:2421: left oid 2 (ObjNum 875 snap 267 seq_num 875) 2024-09-11T01:45:20.209 INFO:tasks.rados.rados.0.smithi071.stdout:2422: expect (ObjNum 829 snap 252 seq_num 829) 2024-09-11T01:45:20.209 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:45:20.959 INFO:tasks.rados.rados.0.smithi071.stdout:2418: done (4 left) 2024-09-11T01:45:20.959 INFO:tasks.rados.rados.0.smithi071.stdout:2419: done (3 left) 2024-09-11T01:45:20.959 INFO:tasks.rados.rados.0.smithi071.stdout:2421: done (2 left) 2024-09-11T01:45:20.959 INFO:tasks.rados.rados.0.smithi071.stdout:2422: done (1 left) 2024-09-11T01:45:20.959 INFO:tasks.rados.rados.0.smithi071.stdout:2423: done (0 left) 2024-09-11T01:45:20.959 INFO:tasks.rados.rados.0.smithi071.stdout:2424: read oid 48 snap 252 2024-09-11T01:45:20.959 INFO:tasks.rados.rados.0.smithi071.stdout:2424: expect deleted 2024-09-11T01:45:20.959 INFO:tasks.rados.rados.0.smithi071.stdout:2425: read oid 7 snap 259 2024-09-11T01:45:20.959 INFO:tasks.rados.rados.0.smithi071.stdout:2425: expect deleted 2024-09-11T01:45:20.959 INFO:tasks.rados.rados.0.smithi071.stdout:2426: read oid 11 snap 174 2024-09-11T01:45:20.959 INFO:tasks.rados.rados.0.smithi071.stdout:2426: expect (ObjNum 597 snap 170 seq_num 597) 2024-09-11T01:45:20.960 INFO:tasks.rados.rados.0.smithi071.stdout:2427: read oid 19 snap -1 2024-09-11T01:45:20.960 INFO:tasks.rados.rados.0.smithi071.stdout:2427: expect deleted 2024-09-11T01:45:20.960 INFO:tasks.rados.rados.0.smithi071.stdout:2428: copy_from oid 28 from oid 26 current snap is 268 2024-09-11T01:45:20.960 INFO:tasks.rados.rados.0.smithi071.stdout:2429: rollback oid 23 current snap is 268 2024-09-11T01:45:20.960 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 23 to 250 2024-09-11T01:45:20.960 INFO:tasks.rados.rados.0.smithi071.stdout:2430: write oid 6 current snap is 268 2024-09-11T01:45:20.960 INFO:tasks.rados.rados.0.smithi071.stdout:2430: seq_num 876 ranges {722932=498719,1994932=501961} 2024-09-11T01:45:20.965 INFO:tasks.rados.rados.0.smithi071.stdout:2430: writing smithi07134777-6 from 722932 to 1221651 tid 1 2024-09-11T01:45:20.973 INFO:tasks.rados.rados.0.smithi071.stdout:2430: writing smithi07134777-6 from 1994932 to 2496893 tid 2 2024-09-11T01:45:20.973 INFO:tasks.rados.rados.0.smithi071.stdout:2431: read oid 13 snap -1 2024-09-11T01:45:20.973 INFO:tasks.rados.rados.0.smithi071.stdout:2431: expect (ObjNum 808 snap 246 seq_num 808) 2024-09-11T01:45:20.973 INFO:tasks.rados.rados.0.smithi071.stdout:2432: snap_create 2024-09-11T01:45:20.973 INFO:tasks.rados.rados.0.smithi071.stdout:2429: finishing rollback tid 0 to smithi07134777-23 2024-09-11T01:45:20.973 INFO:tasks.rados.rados.0.smithi071.stdout:2429: finishing rollback tid 1 to smithi07134777-23 2024-09-11T01:45:20.973 INFO:tasks.rados.rados.0.smithi071.stdout:2426: expect (ObjNum 597 snap 170 seq_num 597) 2024-09-11T01:45:20.991 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1859310 2024-09-11T01:45:21.023 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:45:21.053 INFO:tasks.rados.rados.0.smithi071.stdout:2429: finishing rollback tid 2 to smithi07134777-23 2024-09-11T01:45:21.053 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 1001 (ObjNum 652 snap 199 seq_num 652) dirty exists 2024-09-11T01:45:21.053 INFO:tasks.rados.rados.0.smithi071.stdout:2430: finishing write tid 1 to smithi07134777-6 2024-09-11T01:45:21.053 INFO:tasks.rados.rados.0.smithi071.stdout:2428: finishing copy_from racing read to smithi07134777-28 2024-09-11T01:45:21.053 INFO:tasks.rados.rados.0.smithi071.stdout:2428: finishing copy_from to smithi07134777-28 2024-09-11T01:45:21.053 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 1090 (ObjNum 863 snap 265 seq_num 863) dirty exists 2024-09-11T01:45:21.053 INFO:tasks.rados.rados.0.smithi071.stdout:2430: finishing write tid 2 to smithi07134777-6 2024-09-11T01:45:21.053 INFO:tasks.rados.rados.0.smithi071.stdout:2430: finishing write tid 3 to smithi07134777-6 2024-09-11T01:45:21.053 INFO:tasks.rados.rados.0.smithi071.stdout:2430: finishing write tid 4 to smithi07134777-6 2024-09-11T01:45:21.053 INFO:tasks.rados.rados.0.smithi071.stdout:2430: oid 6 updating version 0 to 1352 2024-09-11T01:45:21.054 INFO:tasks.rados.rados.0.smithi071.stdout:2430: oid 6 updating version 1352 to 1354 2024-09-11T01:45:21.054 INFO:tasks.rados.rados.0.smithi071.stdout:2430: oid 6 version 1354 is already newer than 1353 2024-09-11T01:45:21.054 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 1354 (ObjNum 876 snap 268 seq_num 876) dirty exists 2024-09-11T01:45:21.054 INFO:tasks.rados.rados.0.smithi071.stdout:2430: left oid 6 (ObjNum 876 snap 268 seq_num 876) 2024-09-11T01:45:21.054 INFO:tasks.rados.rados.0.smithi071.stdout:2431: expect (ObjNum 808 snap 246 seq_num 808) 2024-09-11T01:45:21.142 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3530040 2024-09-11T01:45:21.142 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 340612 2024-09-11T01:45:21.963 INFO:tasks.rados.rados.0.smithi071.stdout:2424: done (8 left) 2024-09-11T01:45:21.964 INFO:tasks.rados.rados.0.smithi071.stdout:2425: done (7 left) 2024-09-11T01:45:21.964 INFO:tasks.rados.rados.0.smithi071.stdout:2426: done (6 left) 2024-09-11T01:45:21.964 INFO:tasks.rados.rados.0.smithi071.stdout:2427: done (5 left) 2024-09-11T01:45:21.964 INFO:tasks.rados.rados.0.smithi071.stdout:2428: done (4 left) 2024-09-11T01:45:21.964 INFO:tasks.rados.rados.0.smithi071.stdout:2429: done (3 left) 2024-09-11T01:45:21.964 INFO:tasks.rados.rados.0.smithi071.stdout:2430: done (2 left) 2024-09-11T01:45:21.964 INFO:tasks.rados.rados.0.smithi071.stdout:2431: done (1 left) 2024-09-11T01:45:21.964 INFO:tasks.rados.rados.0.smithi071.stdout:2432: done (0 left) 2024-09-11T01:45:21.964 INFO:tasks.rados.rados.0.smithi071.stdout:2433: write oid 10 current snap is 269 2024-09-11T01:45:21.964 INFO:tasks.rados.rados.0.smithi071.stdout:2433: seq_num 877 ranges {739739=573968,2053414=686325} 2024-09-11T01:45:21.970 INFO:tasks.rados.rados.0.smithi071.stdout:2433: writing smithi07134777-10 from 739739 to 1313707 tid 1 2024-09-11T01:45:21.978 INFO:tasks.rados.rados.0.smithi071.stdout:2433: writing smithi07134777-10 from 2053414 to 2739739 tid 2 2024-09-11T01:45:21.979 INFO:tasks.rados.rados.0.smithi071.stdout:2434: read oid 39 snap -1 2024-09-11T01:45:21.979 INFO:tasks.rados.rados.0.smithi071.stdout:2434: expect deleted 2024-09-11T01:45:21.979 INFO:tasks.rados.rados.0.smithi071.stdout:2435: write (excl) oid 6 current snap is 269 2024-09-11T01:45:21.979 INFO:tasks.rados.rados.0.smithi071.stdout:2433: finishing write tid 1 to smithi07134777-10 2024-09-11T01:45:21.979 INFO:tasks.rados.rados.0.smithi071.stdout:2435: seq_num 878 ranges {756546=649217,2111896=505630} 2024-09-11T01:45:21.986 INFO:tasks.rados.rados.0.smithi071.stdout:2435: writing smithi07134777-6 from 756546 to 1405763 tid 1 2024-09-11T01:45:21.993 INFO:tasks.rados.rados.0.smithi071.stdout:2435: writing smithi07134777-6 from 2111896 to 2617526 tid 2 2024-09-11T01:45:21.993 INFO:tasks.rados.rados.0.smithi071.stdout:2436: copy_from oid 20 from oid 17 current snap is 269 2024-09-11T01:45:21.993 INFO:tasks.rados.rados.0.smithi071.stdout:2433: finishing write tid 2 to smithi07134777-10 2024-09-11T01:45:21.993 INFO:tasks.rados.rados.0.smithi071.stdout:2433: finishing write tid 3 to smithi07134777-10 2024-09-11T01:45:21.993 INFO:tasks.rados.rados.0.smithi071.stdout:2433: finishing write tid 4 to smithi07134777-10 2024-09-11T01:45:21.994 INFO:tasks.rados.rados.0.smithi071.stdout:2433: oid 10 updating version 0 to 411 2024-09-11T01:45:21.994 INFO:tasks.rados.rados.0.smithi071.stdout:2433: oid 10 version 411 is already newer than 410 2024-09-11T01:45:21.994 INFO:tasks.rados.rados.0.smithi071.stdout:2433: oid 10 version 411 is already newer than 409 2024-09-11T01:45:21.994 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 411 (ObjNum 877 snap 269 seq_num 877) dirty exists 2024-09-11T01:45:21.994 INFO:tasks.rados.rados.0.smithi071.stdout:2433: left oid 10 (ObjNum 877 snap 269 seq_num 877) 2024-09-11T01:45:21.994 INFO:tasks.rados.rados.0.smithi071.stdout:2435: finishing write tid 1 to smithi07134777-6 2024-09-11T01:45:21.994 INFO:tasks.rados.rados.0.smithi071.stdout:2433: done (3 left) 2024-09-11T01:45:21.994 INFO:tasks.rados.rados.0.smithi071.stdout:2434: done (2 left) 2024-09-11T01:45:21.994 INFO:tasks.rados.rados.0.smithi071.stdout:2437: write (excl) oid 44 current snap is 269 2024-09-11T01:45:21.994 INFO:tasks.rados.rados.0.smithi071.stdout:2437: seq_num 879 ranges {773353=724466,2170378=602975} 2024-09-11T01:45:22.001 INFO:tasks.rados.rados.0.smithi071.stdout:2437: writing smithi07134777-44 from 773353 to 1497819 tid 1 2024-09-11T01:45:22.009 INFO:tasks.rados.rados.0.smithi071.stdout:2437: writing smithi07134777-44 from 2170378 to 2773353 tid 2 2024-09-11T01:45:22.009 INFO:tasks.rados.rados.0.smithi071.stdout:2436: finishing copy_from to smithi07134777-20 2024-09-11T01:45:22.009 INFO:tasks.rados.rados.0.smithi071.stdout:2436: got expected ENOENT (src dne) 2024-09-11T01:45:22.009 INFO:tasks.rados.rados.0.smithi071.stdout:2438: read oid 34 snap 264 2024-09-11T01:45:22.009 INFO:tasks.rados.rados.0.smithi071.stdout:2438: expect (ObjNum 857 snap 263 seq_num 857) 2024-09-11T01:45:22.009 INFO:tasks.rados.rados.0.smithi071.stdout:2439: snap_create 2024-09-11T01:45:22.009 INFO:tasks.rados.rados.0.smithi071.stdout:2436: finishing copy_from racing read to smithi07134777-20 2024-09-11T01:45:22.009 INFO:tasks.rados.rados.0.smithi071.stdout:2435: finishing write tid 2 to smithi07134777-6 2024-09-11T01:45:22.009 INFO:tasks.rados.rados.0.smithi071.stdout:2435: finishing write tid 3 to smithi07134777-6 2024-09-11T01:45:22.009 INFO:tasks.rados.rados.0.smithi071.stdout:2435: finishing write tid 4 to smithi07134777-6 2024-09-11T01:45:22.010 INFO:tasks.rados.rados.0.smithi071.stdout:2435: oid 6 updating version 0 to 1357 2024-09-11T01:45:22.010 INFO:tasks.rados.rados.0.smithi071.stdout:2435: oid 6 version 1357 is already newer than 1356 2024-09-11T01:45:22.010 INFO:tasks.rados.rados.0.smithi071.stdout:2435: oid 6 updating version 1357 to 1358 2024-09-11T01:45:22.010 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 1358 (ObjNum 878 snap 269 seq_num 878) dirty exists 2024-09-11T01:45:22.010 INFO:tasks.rados.rados.0.smithi071.stdout:2435: left oid 6 (ObjNum 878 snap 269 seq_num 878) 2024-09-11T01:45:22.014 INFO:tasks.rados.rados.0.smithi071.stdout:2437: finishing write tid 1 to smithi07134777-44 2024-09-11T01:45:22.014 INFO:tasks.rados.rados.0.smithi071.stdout:2437: finishing write tid 2 to smithi07134777-44 2024-09-11T01:45:22.014 INFO:tasks.rados.rados.0.smithi071.stdout:2437: finishing write tid 3 to smithi07134777-44 2024-09-11T01:45:22.014 INFO:tasks.rados.rados.0.smithi071.stdout:2437: finishing write tid 4 to smithi07134777-44 2024-09-11T01:45:22.015 INFO:tasks.rados.rados.0.smithi071.stdout:2437: oid 44 updating version 0 to 1002 2024-09-11T01:45:22.015 INFO:tasks.rados.rados.0.smithi071.stdout:2437: oid 44 updating version 1002 to 1004 2024-09-11T01:45:22.015 INFO:tasks.rados.rados.0.smithi071.stdout:2437: oid 44 version 1004 is already newer than 1003 2024-09-11T01:45:22.015 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 1004 (ObjNum 879 snap 269 seq_num 879) dirty exists 2024-09-11T01:45:22.015 INFO:tasks.rados.rados.0.smithi071.stdout:2437: left oid 44 (ObjNum 879 snap 269 seq_num 879) 2024-09-11T01:45:22.015 INFO:tasks.rados.rados.0.smithi071.stdout:2438: expect (ObjNum 857 snap 263 seq_num 857) 2024-09-11T01:45:22.070 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:45:22.966 INFO:tasks.rados.rados.0.smithi071.stdout:2435: done (4 left) 2024-09-11T01:45:22.966 INFO:tasks.rados.rados.0.smithi071.stdout:2436: done (3 left) 2024-09-11T01:45:22.966 INFO:tasks.rados.rados.0.smithi071.stdout:2437: done (2 left) 2024-09-11T01:45:22.966 INFO:tasks.rados.rados.0.smithi071.stdout:2438: done (1 left) 2024-09-11T01:45:22.966 INFO:tasks.rados.rados.0.smithi071.stdout:2439: done (0 left) 2024-09-11T01:45:22.966 INFO:tasks.rados.rados.0.smithi071.stdout:2440: rollback oid 31 current snap is 270 2024-09-11T01:45:22.966 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 31 to 239 2024-09-11T01:45:22.966 INFO:tasks.rados.rados.0.smithi071.stdout:2441: rollback oid 18 current snap is 270 2024-09-11T01:45:22.966 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 18 to 225 2024-09-11T01:45:22.966 INFO:tasks.rados.rados.0.smithi071.stdout:2442: write oid 24 current snap is 270 2024-09-11T01:45:22.967 INFO:tasks.rados.rados.0.smithi071.stdout:2442: seq_num 880 ranges {790160=799715,2112507=509299} 2024-09-11T01:45:22.974 INFO:tasks.rados.rados.0.smithi071.stdout:2442: writing smithi07134777-24 from 790160 to 1589875 tid 1 2024-09-11T01:45:22.981 INFO:tasks.rados.rados.0.smithi071.stdout:2442: writing smithi07134777-24 from 2112507 to 2621806 tid 2 2024-09-11T01:45:22.981 INFO:tasks.rados.rados.0.smithi071.stdout:2443: write (excl) oid 45 current snap is 270 2024-09-11T01:45:22.981 INFO:tasks.rados.rados.0.smithi071.stdout:2441: finishing rollback tid 0 to smithi07134777-18 2024-09-11T01:45:22.981 INFO:tasks.rados.rados.0.smithi071.stdout:2441: finishing rollback tid 1 to smithi07134777-18 2024-09-11T01:45:22.981 INFO:tasks.rados.rados.0.smithi071.stdout:2443: seq_num 881 ranges {406967=474964,1370989=769310,2677186=129781} 2024-09-11T01:45:22.985 INFO:tasks.rados.rados.0.smithi071.stdout:2443: writing smithi07134777-45 from 406967 to 881931 tid 1 2024-09-11T01:45:22.993 INFO:tasks.rados.rados.0.smithi071.stdout:2443: writing smithi07134777-45 from 1370989 to 2140299 tid 2 2024-09-11T01:45:22.997 INFO:tasks.rados.rados.0.smithi071.stdout:2443: writing smithi07134777-45 from 2677186 to 2806967 tid 3 2024-09-11T01:45:22.997 INFO:tasks.rados.rados.0.smithi071.stdout:2444: snap_create 2024-09-11T01:45:22.997 INFO:tasks.rados.rados.0.smithi071.stdout:2441: finishing rollback tid 2 to smithi07134777-18 2024-09-11T01:45:22.997 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 1209 (ObjNum 618 snap 183 seq_num 618) dirty exists 2024-09-11T01:45:22.997 INFO:tasks.rados.rados.0.smithi071.stdout:2440: finishing rollback tid 0 to smithi07134777-31 2024-09-11T01:45:22.997 INFO:tasks.rados.rados.0.smithi071.stdout:2440: finishing rollback tid 1 to smithi07134777-31 2024-09-11T01:45:22.997 INFO:tasks.rados.rados.0.smithi071.stdout:2440: finishing rollback tid 2 to smithi07134777-31 2024-09-11T01:45:22.997 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 1071 (ObjNum 737 snap 226 seq_num 737) dirty exists 2024-09-11T01:45:22.997 INFO:tasks.rados.rados.0.smithi071.stdout:2442: finishing write tid 1 to smithi07134777-24 2024-09-11T01:45:22.998 INFO:tasks.rados.rados.0.smithi071.stdout:2442: finishing write tid 2 to smithi07134777-24 2024-09-11T01:45:22.998 INFO:tasks.rados.rados.0.smithi071.stdout:2442: finishing write tid 3 to smithi07134777-24 2024-09-11T01:45:22.998 INFO:tasks.rados.rados.0.smithi071.stdout:2442: finishing write tid 4 to smithi07134777-24 2024-09-11T01:45:22.998 INFO:tasks.rados.rados.0.smithi071.stdout:2442: oid 24 updating version 0 to 1110 2024-09-11T01:45:22.998 INFO:tasks.rados.rados.0.smithi071.stdout:2442: oid 24 updating version 1110 to 1112 2024-09-11T01:45:22.998 INFO:tasks.rados.rados.0.smithi071.stdout:2442: oid 24 version 1112 is already newer than 1111 2024-09-11T01:45:22.998 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 1112 (ObjNum 880 snap 270 seq_num 880) dirty exists 2024-09-11T01:45:22.998 INFO:tasks.rados.rados.0.smithi071.stdout:2442: left oid 24 (ObjNum 880 snap 270 seq_num 880) 2024-09-11T01:45:22.998 INFO:tasks.rados.rados.0.smithi071.stdout:2443: finishing write tid 1 to smithi07134777-45 2024-09-11T01:45:22.999 INFO:tasks.rados.rados.0.smithi071.stdout:2443: finishing write tid 2 to smithi07134777-45 2024-09-11T01:45:22.999 INFO:tasks.rados.rados.0.smithi071.stdout:2443: finishing write tid 3 to smithi07134777-45 2024-09-11T01:45:22.999 INFO:tasks.rados.rados.0.smithi071.stdout:2443: finishing write tid 4 to smithi07134777-45 2024-09-11T01:45:23.000 INFO:tasks.rados.rados.0.smithi071.stdout:2443: finishing write tid 5 to smithi07134777-45 2024-09-11T01:45:23.000 INFO:tasks.rados.rados.0.smithi071.stdout:2443: oid 45 updating version 0 to 1117 2024-09-11T01:45:23.000 INFO:tasks.rados.rados.0.smithi071.stdout:2443: oid 45 version 1117 is already newer than 1114 2024-09-11T01:45:23.000 INFO:tasks.rados.rados.0.smithi071.stdout:2443: oid 45 version 1117 is already newer than 1116 2024-09-11T01:45:23.000 INFO:tasks.rados.rados.0.smithi071.stdout:2443: oid 45 version 1117 is already newer than 1115 2024-09-11T01:45:23.000 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 1117 (ObjNum 881 snap 270 seq_num 881) dirty exists 2024-09-11T01:45:23.000 INFO:tasks.rados.rados.0.smithi071.stdout:2443: left oid 45 (ObjNum 881 snap 270 seq_num 881) 2024-09-11T01:45:23.974 INFO:tasks.rados.rados.0.smithi071.stdout:2440: done (4 left) 2024-09-11T01:45:23.974 INFO:tasks.rados.rados.0.smithi071.stdout:2441: done (3 left) 2024-09-11T01:45:23.974 INFO:tasks.rados.rados.0.smithi071.stdout:2442: done (2 left) 2024-09-11T01:45:23.975 INFO:tasks.rados.rados.0.smithi071.stdout:2443: done (1 left) 2024-09-11T01:45:23.975 INFO:tasks.rados.rados.0.smithi071.stdout:2444: done (0 left) 2024-09-11T01:45:23.975 INFO:tasks.rados.rados.0.smithi071.stdout:2445: write (excl) oid 47 current snap is 271 2024-09-11T01:45:23.975 INFO:tasks.rados.rados.0.smithi071.stdout:2445: seq_num 882 ranges {423774=666566,1545824=512968,2704609=119165} 2024-09-11T01:45:23.980 INFO:tasks.rados.rados.0.smithi071.stdout:2445: writing smithi07134777-47 from 423774 to 1090340 tid 1 2024-09-11T01:45:23.986 INFO:tasks.rados.rados.0.smithi071.stdout:2445: writing smithi07134777-47 from 1545824 to 2058792 tid 2 2024-09-11T01:45:23.991 INFO:tasks.rados.rados.0.smithi071.stdout:2445: writing smithi07134777-47 from 2704609 to 2823774 tid 3 2024-09-11T01:45:23.991 INFO:tasks.rados.rados.0.smithi071.stdout:2446: snap_create 2024-09-11T01:45:23.991 INFO:tasks.rados.rados.0.smithi071.stdout:2445: finishing write tid 1 to smithi07134777-47 2024-09-11T01:45:23.991 INFO:tasks.rados.rados.0.smithi071.stdout:2445: finishing write tid 2 to smithi07134777-47 2024-09-11T01:45:23.994 INFO:tasks.rados.rados.0.smithi071.stdout:2445: finishing write tid 3 to smithi07134777-47 2024-09-11T01:45:23.994 INFO:tasks.rados.rados.0.smithi071.stdout:2445: finishing write tid 4 to smithi07134777-47 2024-09-11T01:45:23.994 INFO:tasks.rados.rados.0.smithi071.stdout:2445: finishing write tid 5 to smithi07134777-47 2024-09-11T01:45:23.994 INFO:tasks.rados.rados.0.smithi071.stdout:2445: oid 47 updating version 0 to 1076 2024-09-11T01:45:23.994 INFO:tasks.rados.rados.0.smithi071.stdout:2445: oid 47 version 1076 is already newer than 1073 2024-09-11T01:45:23.994 INFO:tasks.rados.rados.0.smithi071.stdout:2445: oid 47 version 1076 is already newer than 1075 2024-09-11T01:45:23.995 INFO:tasks.rados.rados.0.smithi071.stdout:2445: oid 47 version 1076 is already newer than 1074 2024-09-11T01:45:23.995 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 1076 (ObjNum 882 snap 271 seq_num 882) dirty exists 2024-09-11T01:45:23.995 INFO:tasks.rados.rados.0.smithi071.stdout:2445: left oid 47 (ObjNum 882 snap 271 seq_num 882) 2024-09-11T01:45:24.976 INFO:tasks.rados.rados.0.smithi071.stdout:2445: done (1 left) 2024-09-11T01:45:24.976 INFO:tasks.rados.rados.0.smithi071.stdout:2446: done (0 left) 2024-09-11T01:45:24.976 INFO:tasks.rados.rados.0.smithi071.stdout:2447: snap_remove snap 258 2024-09-11T01:45:25.992 INFO:tasks.rados.rados.0.smithi071.stdout:2447: done (0 left) 2024-09-11T01:45:25.992 INFO:tasks.rados.rados.0.smithi071.stdout:2448: write (excl) oid 30 current snap is 272 2024-09-11T01:45:25.992 INFO:tasks.rados.rados.0.smithi071.stdout:2448: seq_num 883 ranges {440581=741815,1604306=656626,2732032=108549} 2024-09-11T01:45:25.999 INFO:tasks.rados.rados.0.smithi071.stdout:2448: writing smithi07134777-30 from 440581 to 1182396 tid 1 2024-09-11T01:45:26.005 INFO:tasks.rados.rados.0.smithi071.stdout:2448: writing smithi07134777-30 from 1604306 to 2260932 tid 2 2024-09-11T01:45:26.008 INFO:tasks.rados.rados.0.smithi071.stdout:2448: writing smithi07134777-30 from 2732032 to 2840581 tid 3 2024-09-11T01:45:26.008 INFO:tasks.rados.rados.0.smithi071.stdout:2449: delete oid 48 current snap is 272 2024-09-11T01:45:26.009 INFO:tasks.rados.rados.0.smithi071.stdout:2448: finishing write tid 1 to smithi07134777-30 2024-09-11T01:45:26.009 INFO:tasks.rados.rados.0.smithi071.stdout:2448: finishing write tid 2 to smithi07134777-30 2024-09-11T01:45:26.011 INFO:tasks.rados.rados.0.smithi071.stdout:2449: done (1 left) 2024-09-11T01:45:26.011 INFO:tasks.rados.rados.0.smithi071.stdout:2450: write (excl) oid 19 current snap is 272 2024-09-11T01:45:26.011 INFO:tasks.rados.rados.0.smithi071.stdout:2450: seq_num 885 ranges {474195=492313,1604917=660295,2670525=203670} 2024-09-11T01:45:26.016 INFO:tasks.rados.rados.0.smithi071.stdout:2450: writing smithi07134777-19 from 474195 to 966508 tid 1 2024-09-11T01:45:26.023 INFO:tasks.rados.rados.0.smithi071.stdout:2450: writing smithi07134777-19 from 1604917 to 2265212 tid 2 2024-09-11T01:45:26.027 INFO:tasks.rados.rados.0.smithi071.stdout:2450: writing smithi07134777-19 from 2670525 to 2874195 tid 3 2024-09-11T01:45:26.027 INFO:tasks.rados.rados.0.smithi071.stdout:2451: rollback oid 12 current snap is 272 2024-09-11T01:45:26.027 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 12 to 263 2024-09-11T01:45:26.027 INFO:tasks.rados.rados.0.smithi071.stdout:2448: finishing write tid 3 to smithi07134777-30 2024-09-11T01:45:26.027 INFO:tasks.rados.rados.0.smithi071.stdout:2448: finishing write tid 4 to smithi07134777-30 2024-09-11T01:45:26.027 INFO:tasks.rados.rados.0.smithi071.stdout:2448: finishing write tid 5 to smithi07134777-30 2024-09-11T01:45:26.027 INFO:tasks.rados.rados.0.smithi071.stdout:2448: oid 30 updating version 0 to 1212 2024-09-11T01:45:26.028 INFO:tasks.rados.rados.0.smithi071.stdout:2448: oid 30 updating version 1212 to 1213 2024-09-11T01:45:26.028 INFO:tasks.rados.rados.0.smithi071.stdout:2448: oid 30 version 1213 is already newer than 1211 2024-09-11T01:45:26.028 INFO:tasks.rados.rados.0.smithi071.stdout:2448: oid 30 version 1213 is already newer than 1210 2024-09-11T01:45:26.028 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 1213 (ObjNum 883 snap 272 seq_num 883) dirty exists 2024-09-11T01:45:26.028 INFO:tasks.rados.rados.0.smithi071.stdout:2448: left oid 30 (ObjNum 883 snap 272 seq_num 883) 2024-09-11T01:45:26.028 INFO:tasks.rados.rados.0.smithi071.stdout:2450: finishing write tid 1 to smithi07134777-19 2024-09-11T01:45:26.028 INFO:tasks.rados.rados.0.smithi071.stdout:2448: done (2 left) 2024-09-11T01:45:26.028 INFO:tasks.rados.rados.0.smithi071.stdout:2452: write (excl) oid 5 current snap is 272 2024-09-11T01:45:26.028 INFO:tasks.rados.rados.0.smithi071.stdout:2452: seq_num 886 ranges {491002=567562,1663399=520306,2814301=76701} 2024-09-11T01:45:26.032 INFO:tasks.rados.rados.0.smithi071.stdout:2452: writing smithi07134777-5 from 491002 to 1058564 tid 1 2024-09-11T01:45:26.039 INFO:tasks.rados.rados.0.smithi071.stdout:2452: writing smithi07134777-5 from 1663399 to 2183705 tid 2 2024-09-11T01:45:26.043 INFO:tasks.rados.rados.0.smithi071.stdout:2452: writing smithi07134777-5 from 2814301 to 2891002 tid 3 2024-09-11T01:45:26.043 INFO:tasks.rados.rados.0.smithi071.stdout:2453: read oid 15 snap -1 2024-09-11T01:45:26.043 INFO:tasks.rados.rados.0.smithi071.stdout:2453: expect (ObjNum 873 snap 267 seq_num 873) 2024-09-11T01:45:26.044 INFO:tasks.rados.rados.0.smithi071.stdout:2454: delete oid 3 current snap is 272 2024-09-11T01:45:26.044 INFO:tasks.rados.rados.0.smithi071.stdout:2451: finishing rollback tid 1 to smithi07134777-12 2024-09-11T01:45:26.044 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 1156 (ObjNum 225 snap 0 seq_num 3922450795) dirty dne 2024-09-11T01:45:26.044 INFO:tasks.rados.rados.0.smithi071.stdout:2450: finishing write tid 2 to smithi07134777-19 2024-09-11T01:45:26.044 INFO:tasks.rados.rados.0.smithi071.stdout:2450: finishing write tid 3 to smithi07134777-19 2024-09-11T01:45:26.044 INFO:tasks.rados.rados.0.smithi071.stdout:2450: finishing write tid 4 to smithi07134777-19 2024-09-11T01:45:26.044 INFO:tasks.rados.rados.0.smithi071.stdout:2450: finishing write tid 5 to smithi07134777-19 2024-09-11T01:45:26.044 INFO:tasks.rados.rados.0.smithi071.stdout:2450: oid 19 updating version 0 to 1062 2024-09-11T01:45:26.044 INFO:tasks.rados.rados.0.smithi071.stdout:2450: oid 19 updating version 1062 to 1063 2024-09-11T01:45:26.044 INFO:tasks.rados.rados.0.smithi071.stdout:2450: oid 19 version 1063 is already newer than 1061 2024-09-11T01:45:26.044 INFO:tasks.rados.rados.0.smithi071.stdout:2450: oid 19 version 1063 is already newer than 1060 2024-09-11T01:45:26.044 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 1063 (ObjNum 885 snap 272 seq_num 885) dirty exists 2024-09-11T01:45:26.044 INFO:tasks.rados.rados.0.smithi071.stdout:2450: left oid 19 (ObjNum 885 snap 272 seq_num 885) 2024-09-11T01:45:26.045 INFO:tasks.rados.rados.0.smithi071.stdout:2452: finishing write tid 1 to smithi07134777-5 2024-09-11T01:45:26.045 INFO:tasks.rados.rados.0.smithi071.stdout:2452: finishing write tid 2 to smithi07134777-5 2024-09-11T01:45:26.046 INFO:tasks.rados.rados.0.smithi071.stdout:2450: done (4 left) 2024-09-11T01:45:26.046 INFO:tasks.rados.rados.0.smithi071.stdout:2451: done (3 left) 2024-09-11T01:45:26.046 INFO:tasks.rados.rados.0.smithi071.stdout:2454: done (2 left) 2024-09-11T01:45:26.046 INFO:tasks.rados.rados.0.smithi071.stdout:2455: snap_create 2024-09-11T01:45:26.046 INFO:tasks.rados.rados.0.smithi071.stdout:2452: finishing write tid 3 to smithi07134777-5 2024-09-11T01:45:26.046 INFO:tasks.rados.rados.0.smithi071.stdout:2452: finishing write tid 4 to smithi07134777-5 2024-09-11T01:45:26.046 INFO:tasks.rados.rados.0.smithi071.stdout:2452: finishing write tid 5 to smithi07134777-5 2024-09-11T01:45:26.046 INFO:tasks.rados.rados.0.smithi071.stdout:2452: oid 5 updating version 0 to 1161 2024-09-11T01:45:26.046 INFO:tasks.rados.rados.0.smithi071.stdout:2452: oid 5 updating version 1161 to 1162 2024-09-11T01:45:26.046 INFO:tasks.rados.rados.0.smithi071.stdout:2452: oid 5 version 1162 is already newer than 1160 2024-09-11T01:45:26.046 INFO:tasks.rados.rados.0.smithi071.stdout:2452: oid 5 version 1162 is already newer than 1159 2024-09-11T01:45:26.047 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 1162 (ObjNum 886 snap 272 seq_num 886) dirty exists 2024-09-11T01:45:26.047 INFO:tasks.rados.rados.0.smithi071.stdout:2452: left oid 5 (ObjNum 886 snap 272 seq_num 886) 2024-09-11T01:45:26.048 INFO:tasks.rados.rados.0.smithi071.stdout:2453: expect (ObjNum 873 snap 267 seq_num 873) 2024-09-11T01:45:26.076 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2496188 2024-09-11T01:45:26.986 INFO:tasks.rados.rados.0.smithi071.stdout:2452: done (2 left) 2024-09-11T01:45:26.986 INFO:tasks.rados.rados.0.smithi071.stdout:2453: done (1 left) 2024-09-11T01:45:26.986 INFO:tasks.rados.rados.0.smithi071.stdout:2455: done (0 left) 2024-09-11T01:45:26.986 INFO:tasks.rados.rados.0.smithi071.stdout:2456: snap_create 2024-09-11T01:45:28.197 INFO:tasks.rados.rados.0.smithi071.stdout:2456: done (0 left) 2024-09-11T01:45:28.197 INFO:tasks.rados.rados.0.smithi071.stdout:2457: copy_from oid 40 from oid 14 current snap is 274 2024-09-11T01:45:28.197 INFO:tasks.rados.rados.0.smithi071.stdout:2458: snap_remove snap 253 2024-09-11T01:45:28.949 INFO:tasks.rados.rados.0.smithi071.stdout:2458: done (1 left) 2024-09-11T01:45:28.949 INFO:tasks.rados.rados.0.smithi071.stdout:2459: snap_remove snap 263 2024-09-11T01:45:29.965 INFO:tasks.rados.rados.0.smithi071.stdout:2459: done (1 left) 2024-09-11T01:45:29.965 INFO:tasks.rados.rados.0.smithi071.stdout:2460: delete oid 49 current snap is 274 2024-09-11T01:45:29.965 INFO:tasks.rados.rados.0.smithi071.stdout:2457: finishing copy_from racing read to smithi07134777-40 2024-09-11T01:45:29.966 INFO:tasks.rados.rados.0.smithi071.stdout:2457: finishing copy_from to smithi07134777-40 2024-09-11T01:45:29.966 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 1530 (ObjNum 865 snap 266 seq_num 865) dirty exists 2024-09-11T01:45:29.970 INFO:tasks.rados.rados.0.smithi071.stdout:2457: done (1 left) 2024-09-11T01:45:29.970 INFO:tasks.rados.rados.0.smithi071.stdout:2460: done (0 left) 2024-09-11T01:45:29.970 INFO:tasks.rados.rados.0.smithi071.stdout:2461: read oid 19 snap -1 2024-09-11T01:45:29.970 INFO:tasks.rados.rados.0.smithi071.stdout:2461: expect (ObjNum 885 snap 272 seq_num 885) 2024-09-11T01:45:29.970 INFO:tasks.rados.rados.0.smithi071.stdout:2462: rollback oid 31 current snap is 274 2024-09-11T01:45:29.971 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 31 to 236 2024-09-11T01:45:29.971 INFO:tasks.rados.rados.0.smithi071.stdout:2463: read oid 45 snap -1 2024-09-11T01:45:29.971 INFO:tasks.rados.rados.0.smithi071.stdout:2463: expect (ObjNum 881 snap 270 seq_num 881) 2024-09-11T01:45:29.971 INFO:tasks.rados.rados.0.smithi071.stdout:2464: write (excl) oid 43 current snap is 274 2024-09-11T01:45:29.971 INFO:tasks.rados.rados.0.smithi071.stdout:2464: seq_num 889 ranges {541423=793309,2122492=667633} 2024-09-11T01:45:29.978 INFO:tasks.rados.rados.0.smithi071.stdout:2464: writing smithi07134777-43 from 541423 to 1334732 tid 1 2024-09-11T01:45:29.990 INFO:tasks.rados.rados.0.smithi071.stdout:2464: writing smithi07134777-43 from 2122492 to 2790125 tid 2 2024-09-11T01:45:29.990 INFO:tasks.rados.rados.0.smithi071.stdout:2465: delete oid 48 current snap is 274 2024-09-11T01:45:29.990 INFO:tasks.rados.rados.0.smithi071.stdout:2462: finishing rollback tid 0 to smithi07134777-31 2024-09-11T01:45:29.990 INFO:tasks.rados.rados.0.smithi071.stdout:2462: finishing rollback tid 1 to smithi07134777-31 2024-09-11T01:45:29.990 INFO:tasks.rados.rados.0.smithi071.stdout:2462: finishing rollback tid 2 to smithi07134777-31 2024-09-11T01:45:29.990 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 1100 (ObjNum 737 snap 226 seq_num 737) dirty exists 2024-09-11T01:45:29.990 INFO:tasks.rados.rados.0.smithi071.stdout:2463: expect (ObjNum 881 snap 270 seq_num 881) 2024-09-11T01:45:30.055 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2329105 2024-09-11T01:45:30.112 INFO:tasks.rados.rados.0.smithi071.stdout:2461: expect (ObjNum 885 snap 272 seq_num 885) 2024-09-11T01:45:30.112 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:45:30.142 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:45:30.177 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 673374 2024-09-11T01:45:30.201 INFO:tasks.rados.rados.0.smithi071.stdout:2464: finishing write tid 1 to smithi07134777-43 2024-09-11T01:45:30.201 INFO:tasks.rados.rados.0.smithi071.stdout:2464: finishing write tid 2 to smithi07134777-43 2024-09-11T01:45:30.201 INFO:tasks.rados.rados.0.smithi071.stdout:2464: finishing write tid 3 to smithi07134777-43 2024-09-11T01:45:30.201 INFO:tasks.rados.rados.0.smithi071.stdout:2464: finishing write tid 4 to smithi07134777-43 2024-09-11T01:45:30.202 INFO:tasks.rados.rados.0.smithi071.stdout:2464: oid 43 updating version 0 to 1383 2024-09-11T01:45:30.202 INFO:tasks.rados.rados.0.smithi071.stdout:2464: oid 43 updating version 1383 to 1384 2024-09-11T01:45:30.202 INFO:tasks.rados.rados.0.smithi071.stdout:2464: oid 43 updating version 1384 to 1385 2024-09-11T01:45:30.202 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 1385 (ObjNum 889 snap 274 seq_num 889) dirty exists 2024-09-11T01:45:30.202 INFO:tasks.rados.rados.0.smithi071.stdout:2464: left oid 43 (ObjNum 889 snap 274 seq_num 889) 2024-09-11T01:45:30.202 INFO:tasks.rados.rados.0.smithi071.stdout:2461: done (4 left) 2024-09-11T01:45:30.202 INFO:tasks.rados.rados.0.smithi071.stdout:2462: done (3 left) 2024-09-11T01:45:30.202 INFO:tasks.rados.rados.0.smithi071.stdout:2463: done (2 left) 2024-09-11T01:45:30.202 INFO:tasks.rados.rados.0.smithi071.stdout:2464: done (1 left) 2024-09-11T01:45:30.202 INFO:tasks.rados.rados.0.smithi071.stdout:2465: done (0 left) 2024-09-11T01:45:30.202 INFO:tasks.rados.rados.0.smithi071.stdout:2466: copy_from oid 4 from oid 8 current snap is 274 2024-09-11T01:45:30.202 INFO:tasks.rados.rados.0.smithi071.stdout:2467: copy_from oid 43 from oid 19 current snap is 274 2024-09-11T01:45:30.202 INFO:tasks.rados.rados.0.smithi071.stdout:2468: write oid 6 current snap is 274 2024-09-11T01:45:30.203 INFO:tasks.rados.rados.0.smithi071.stdout:2468: seq_num 891 ranges {575037=660160,1955809=671302} 2024-09-11T01:45:30.207 INFO:tasks.rados.rados.0.smithi071.stdout:2468: writing smithi07134777-6 from 575037 to 1235197 tid 1 2024-09-11T01:45:30.215 INFO:tasks.rados.rados.0.smithi071.stdout:2468: writing smithi07134777-6 from 1955809 to 2627111 tid 2 2024-09-11T01:45:30.215 INFO:tasks.rados.rados.0.smithi071.stdout:2469: snap_remove snap 239 2024-09-11T01:45:31.190 INFO:tasks.rados.rados.0.smithi071.stdout:2469: done (3 left) 2024-09-11T01:45:31.190 INFO:tasks.rados.rados.0.smithi071.stdout:2470: rollback oid 45 current snap is 274 2024-09-11T01:45:31.190 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 45 to 266 2024-09-11T01:45:31.191 INFO:tasks.rados.rados.0.smithi071.stdout:2466: finishing copy_from to smithi07134777-4 2024-09-11T01:45:31.191 INFO:tasks.rados.rados.0.smithi071.stdout:2466: got expected ENOENT (src dne) 2024-09-11T01:45:31.191 INFO:tasks.rados.rados.0.smithi071.stdout:2466: finishing copy_from racing read to smithi07134777-4 2024-09-11T01:45:31.191 INFO:tasks.rados.rados.0.smithi071.stdout:2467: finishing copy_from racing read to smithi07134777-43 2024-09-11T01:45:31.191 INFO:tasks.rados.rados.0.smithi071.stdout:2467: finishing copy_from to smithi07134777-43 2024-09-11T01:45:31.191 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 1063 (ObjNum 885 snap 272 seq_num 885) dirty exists 2024-09-11T01:45:31.191 INFO:tasks.rados.rados.0.smithi071.stdout:2468: finishing write tid 1 to smithi07134777-6 2024-09-11T01:45:31.191 INFO:tasks.rados.rados.0.smithi071.stdout:2468: finishing write tid 2 to smithi07134777-6 2024-09-11T01:45:31.191 INFO:tasks.rados.rados.0.smithi071.stdout:2468: finishing write tid 3 to smithi07134777-6 2024-09-11T01:45:31.191 INFO:tasks.rados.rados.0.smithi071.stdout:2468: finishing write tid 4 to smithi07134777-6 2024-09-11T01:45:31.191 INFO:tasks.rados.rados.0.smithi071.stdout:2468: oid 6 updating version 0 to 1390 2024-09-11T01:45:31.191 INFO:tasks.rados.rados.0.smithi071.stdout:2468: oid 6 version 1390 is already newer than 1389 2024-09-11T01:45:31.191 INFO:tasks.rados.rados.0.smithi071.stdout:2468: oid 6 version 1390 is already newer than 1388 2024-09-11T01:45:31.192 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 1390 (ObjNum 891 snap 274 seq_num 891) dirty exists 2024-09-11T01:45:31.192 INFO:tasks.rados.rados.0.smithi071.stdout:2468: left oid 6 (ObjNum 891 snap 274 seq_num 891) 2024-09-11T01:45:31.192 INFO:tasks.rados.rados.0.smithi071.stdout:2466: done (3 left) 2024-09-11T01:45:31.192 INFO:tasks.rados.rados.0.smithi071.stdout:2467: done (2 left) 2024-09-11T01:45:31.192 INFO:tasks.rados.rados.0.smithi071.stdout:2468: done (1 left) 2024-09-11T01:45:31.192 INFO:tasks.rados.rados.0.smithi071.stdout:2471: snap_remove snap 229 2024-09-11T01:45:32.198 INFO:tasks.rados.rados.0.smithi071.stdout:2471: done (1 left) 2024-09-11T01:45:32.198 INFO:tasks.rados.rados.0.smithi071.stdout:2472: write (excl) oid 10 current snap is 274 2024-09-11T01:45:32.198 INFO:tasks.rados.rados.0.smithi071.stdout:2470: finishing rollback tid 0 to smithi07134777-45 2024-09-11T01:45:32.198 INFO:tasks.rados.rados.0.smithi071.stdout:2472: seq_num 892 ranges {591844=735409,1897938=531313,2862486=129358} 2024-09-11T01:45:32.206 INFO:tasks.rados.rados.0.smithi071.stdout:2472: writing smithi07134777-10 from 591844 to 1327253 tid 1 2024-09-11T01:45:32.213 INFO:tasks.rados.rados.0.smithi071.stdout:2472: writing smithi07134777-10 from 1897938 to 2429251 tid 2 2024-09-11T01:45:32.217 INFO:tasks.rados.rados.0.smithi071.stdout:2472: writing smithi07134777-10 from 2862486 to 2991844 tid 3 2024-09-11T01:45:32.217 INFO:tasks.rados.rados.0.smithi071.stdout:2473: rollback oid 47 current snap is 274 2024-09-11T01:45:32.218 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 47 to 260 2024-09-11T01:45:32.218 INFO:tasks.rados.rados.0.smithi071.stdout:2470: finishing rollback tid 1 to smithi07134777-45 2024-09-11T01:45:32.218 INFO:tasks.rados.rados.0.smithi071.stdout:2470: finishing rollback tid 2 to smithi07134777-45 2024-09-11T01:45:32.218 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 1151 (ObjNum 854 snap 257 seq_num 854) dirty exists 2024-09-11T01:45:32.218 INFO:tasks.rados.rados.0.smithi071.stdout:2472: finishing write tid 1 to smithi07134777-10 2024-09-11T01:45:32.219 INFO:tasks.rados.rados.0.smithi071.stdout:2470: done (2 left) 2024-09-11T01:45:32.219 INFO:tasks.rados.rados.0.smithi071.stdout:2474: write oid 9 current snap is 274 2024-09-11T01:45:32.219 INFO:tasks.rados.rados.0.smithi071.stdout:2474: seq_num 893 ranges {608651=410658,1556420=674971,2773556=235095} 2024-09-11T01:45:32.223 INFO:tasks.rados.rados.0.smithi071.stdout:2474: writing smithi07134777-9 from 608651 to 1019309 tid 1 2024-09-11T01:45:32.230 INFO:tasks.rados.rados.0.smithi071.stdout:2474: writing smithi07134777-9 from 1556420 to 2231391 tid 2 2024-09-11T01:45:32.235 INFO:tasks.rados.rados.0.smithi071.stdout:2474: writing smithi07134777-9 from 2773556 to 3008651 tid 3 2024-09-11T01:45:32.235 INFO:tasks.rados.rados.0.smithi071.stdout:2475: read oid 39 snap 212 2024-09-11T01:45:32.235 INFO:tasks.rados.rados.0.smithi071.stdout:2475: expect (ObjNum 701 snap 211 seq_num 701) 2024-09-11T01:45:32.235 INFO:tasks.rados.rados.0.smithi071.stdout:2476: read oid 28 snap -1 2024-09-11T01:45:32.235 INFO:tasks.rados.rados.0.smithi071.stdout:2476: expect (ObjNum 863 snap 265 seq_num 863) 2024-09-11T01:45:32.236 INFO:tasks.rados.rados.0.smithi071.stdout:2477: snap_remove snap 243 2024-09-11T01:45:33.206 INFO:tasks.rados.rados.0.smithi071.stdout:2477: done (5 left) 2024-09-11T01:45:33.206 INFO:tasks.rados.rados.0.smithi071.stdout:2478: copy_from oid 12 from oid 34 current snap is 274 2024-09-11T01:45:33.206 INFO:tasks.rados.rados.0.smithi071.stdout:2472: finishing write tid 2 to smithi07134777-10 2024-09-11T01:45:33.206 INFO:tasks.rados.rados.0.smithi071.stdout:2472: finishing write tid 3 to smithi07134777-10 2024-09-11T01:45:33.206 INFO:tasks.rados.rados.0.smithi071.stdout:2472: finishing write tid 4 to smithi07134777-10 2024-09-11T01:45:33.206 INFO:tasks.rados.rados.0.smithi071.stdout:2472: finishing write tid 5 to smithi07134777-10 2024-09-11T01:45:33.206 INFO:tasks.rados.rados.0.smithi071.stdout:2472: oid 10 updating version 0 to 413 2024-09-11T01:45:33.206 INFO:tasks.rados.rados.0.smithi071.stdout:2472: oid 10 version 413 is already newer than 412 2024-09-11T01:45:33.206 INFO:tasks.rados.rados.0.smithi071.stdout:2472: oid 10 updating version 413 to 415 2024-09-11T01:45:33.207 INFO:tasks.rados.rados.0.smithi071.stdout:2472: oid 10 version 415 is already newer than 414 2024-09-11T01:45:33.207 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 415 (ObjNum 892 snap 274 seq_num 892) dirty exists 2024-09-11T01:45:33.207 INFO:tasks.rados.rados.0.smithi071.stdout:2472: left oid 10 (ObjNum 892 snap 274 seq_num 892) 2024-09-11T01:45:33.207 INFO:tasks.rados.rados.0.smithi071.stdout:2473: finishing rollback tid 0 to smithi07134777-47 2024-09-11T01:45:33.207 INFO:tasks.rados.rados.0.smithi071.stdout:2473: finishing rollback tid 1 to smithi07134777-47 2024-09-11T01:45:33.207 INFO:tasks.rados.rados.0.smithi071.stdout:2473: finishing rollback tid 2 to smithi07134777-47 2024-09-11T01:45:33.207 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 1118 (ObjNum 704 snap 213 seq_num 704) dirty exists 2024-09-11T01:45:33.207 INFO:tasks.rados.rados.0.smithi071.stdout:2474: finishing write tid 1 to smithi07134777-9 2024-09-11T01:45:33.207 INFO:tasks.rados.rados.0.smithi071.stdout:2474: finishing write tid 2 to smithi07134777-9 2024-09-11T01:45:33.207 INFO:tasks.rados.rados.0.smithi071.stdout:2474: finishing write tid 3 to smithi07134777-9 2024-09-11T01:45:33.207 INFO:tasks.rados.rados.0.smithi071.stdout:2474: finishing write tid 4 to smithi07134777-9 2024-09-11T01:45:33.207 INFO:tasks.rados.rados.0.smithi071.stdout:2474: finishing write tid 5 to smithi07134777-9 2024-09-11T01:45:33.207 INFO:tasks.rados.rados.0.smithi071.stdout:2474: oid 9 updating version 0 to 998 2024-09-11T01:45:33.207 INFO:tasks.rados.rados.0.smithi071.stdout:2474: oid 9 version 998 is already newer than 997 2024-09-11T01:45:33.208 INFO:tasks.rados.rados.0.smithi071.stdout:2474: oid 9 version 998 is already newer than 996 2024-09-11T01:45:33.208 INFO:tasks.rados.rados.0.smithi071.stdout:2474: oid 9 updating version 998 to 999 2024-09-11T01:45:33.208 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 999 (ObjNum 893 snap 274 seq_num 893) dirty exists 2024-09-11T01:45:33.208 INFO:tasks.rados.rados.0.smithi071.stdout:2474: left oid 9 (ObjNum 893 snap 274 seq_num 893) 2024-09-11T01:45:33.208 INFO:tasks.rados.rados.0.smithi071.stdout:2476: expect (ObjNum 863 snap 265 seq_num 863) 2024-09-11T01:45:33.313 INFO:tasks.rados.rados.0.smithi071.stdout:2475: expect (ObjNum 701 snap 211 seq_num 701) 2024-09-11T01:45:33.373 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2251566 2024-09-11T01:45:33.396 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:45:33.453 INFO:tasks.rados.rados.0.smithi071.stdout:2478: finishing copy_from racing read to smithi07134777-12 2024-09-11T01:45:33.453 INFO:tasks.rados.rados.0.smithi071.stdout:2478: finishing copy_from to smithi07134777-12 2024-09-11T01:45:33.453 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 1325 (ObjNum 857 snap 263 seq_num 857) dirty exists 2024-09-11T01:45:33.453 INFO:tasks.rados.rados.0.smithi071.stdout:2472: done (5 left) 2024-09-11T01:45:33.453 INFO:tasks.rados.rados.0.smithi071.stdout:2473: done (4 left) 2024-09-11T01:45:33.453 INFO:tasks.rados.rados.0.smithi071.stdout:2474: done (3 left) 2024-09-11T01:45:33.454 INFO:tasks.rados.rados.0.smithi071.stdout:2475: done (2 left) 2024-09-11T01:45:33.454 INFO:tasks.rados.rados.0.smithi071.stdout:2476: done (1 left) 2024-09-11T01:45:33.454 INFO:tasks.rados.rados.0.smithi071.stdout:2478: done (0 left) 2024-09-11T01:45:33.454 INFO:tasks.rados.rados.0.smithi071.stdout:2479: delete oid 6 current snap is 274 2024-09-11T01:45:33.456 INFO:tasks.rados.rados.0.smithi071.stdout:2479: done (0 left) 2024-09-11T01:45:33.456 INFO:tasks.rados.rados.0.smithi071.stdout:2480: read oid 33 snap -1 2024-09-11T01:45:33.456 INFO:tasks.rados.rados.0.smithi071.stdout:2480: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:45:33.456 INFO:tasks.rados.rados.0.smithi071.stdout:2481: read oid 1 snap -1 2024-09-11T01:45:33.456 INFO:tasks.rados.rados.0.smithi071.stdout:2481: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:45:33.456 INFO:tasks.rados.rados.0.smithi071.stdout:2482: snap_create 2024-09-11T01:45:33.459 INFO:tasks.rados.rados.0.smithi071.stdout:2481: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:45:33.548 INFO:tasks.rados.rados.0.smithi071.stdout:2480: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:45:33.624 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 353274 2024-09-11T01:45:33.947 INFO:tasks.rados.rados.0.smithi071.stdout:2480: done (2 left) 2024-09-11T01:45:33.947 INFO:tasks.rados.rados.0.smithi071.stdout:2481: done (1 left) 2024-09-11T01:45:33.947 INFO:tasks.rados.rados.0.smithi071.stdout:2482: done (0 left) 2024-09-11T01:45:33.948 INFO:tasks.rados.rados.0.smithi071.stdout:2483: rollback oid 44 current snap is 275 2024-09-11T01:45:33.948 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 44 to 248 2024-09-11T01:45:33.948 INFO:tasks.rados.rados.0.smithi071.stdout:2484: delete oid 9 current snap is 275 2024-09-11T01:45:33.951 INFO:tasks.rados.rados.0.smithi071.stdout:2484: done (1 left) 2024-09-11T01:45:33.951 INFO:tasks.rados.rados.0.smithi071.stdout:2485: snap_create 2024-09-11T01:45:33.951 INFO:tasks.rados.rados.0.smithi071.stdout:2483: finishing rollback tid 0 to smithi07134777-44 2024-09-11T01:45:33.952 INFO:tasks.rados.rados.0.smithi071.stdout:2483: finishing rollback tid 1 to smithi07134777-44 2024-09-11T01:45:33.952 INFO:tasks.rados.rados.0.smithi071.stdout:2483: finishing rollback tid 2 to smithi07134777-44 2024-09-11T01:45:33.952 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 1007 (ObjNum 779 snap 236 seq_num 779) dirty exists 2024-09-11T01:45:34.949 INFO:tasks.rados.rados.0.smithi071.stdout:2483: done (1 left) 2024-09-11T01:45:34.949 INFO:tasks.rados.rados.0.smithi071.stdout:2485: done (0 left) 2024-09-11T01:45:34.949 INFO:tasks.rados.rados.0.smithi071.stdout:2486: write oid 40 current snap is 276 2024-09-11T01:45:34.949 INFO:tasks.rados.rados.0.smithi071.stdout:2486: seq_num 896 ranges {659072=636405,1731866=422298,2855825=203247} 2024-09-11T01:45:34.956 INFO:tasks.rados.rados.0.smithi071.stdout:2486: writing smithi07134777-40 from 659072 to 1295477 tid 1 2024-09-11T01:45:34.960 INFO:tasks.rados.rados.0.smithi071.stdout:2486: writing smithi07134777-40 from 1731866 to 2154164 tid 2 2024-09-11T01:45:34.966 INFO:tasks.rados.rados.0.smithi071.stdout:2486: writing smithi07134777-40 from 2855825 to 3059072 tid 3 2024-09-11T01:45:34.966 INFO:tasks.rados.rados.0.smithi071.stdout:2487: copy_from oid 24 from oid 10 current snap is 276 2024-09-11T01:45:34.966 INFO:tasks.rados.rados.0.smithi071.stdout:2486: finishing write tid 1 to smithi07134777-40 2024-09-11T01:45:34.966 INFO:tasks.rados.rados.0.smithi071.stdout:2486: finishing write tid 2 to smithi07134777-40 2024-09-11T01:45:34.966 INFO:tasks.rados.rados.0.smithi071.stdout:2488: delete oid 26 current snap is 276 2024-09-11T01:45:34.968 INFO:tasks.rados.rados.0.smithi071.stdout:2486: finishing write tid 3 to smithi07134777-40 2024-09-11T01:45:34.968 INFO:tasks.rados.rados.0.smithi071.stdout:2486: finishing write tid 4 to smithi07134777-40 2024-09-11T01:45:34.969 INFO:tasks.rados.rados.0.smithi071.stdout:2486: finishing write tid 5 to smithi07134777-40 2024-09-11T01:45:34.969 INFO:tasks.rados.rados.0.smithi071.stdout:2486: oid 40 updating version 0 to 1531 2024-09-11T01:45:34.969 INFO:tasks.rados.rados.0.smithi071.stdout:2486: oid 40 updating version 1531 to 1534 2024-09-11T01:45:34.969 INFO:tasks.rados.rados.0.smithi071.stdout:2486: oid 40 version 1534 is already newer than 1533 2024-09-11T01:45:34.969 INFO:tasks.rados.rados.0.smithi071.stdout:2486: oid 40 version 1534 is already newer than 1532 2024-09-11T01:45:34.969 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 1534 (ObjNum 896 snap 276 seq_num 896) dirty exists 2024-09-11T01:45:34.969 INFO:tasks.rados.rados.0.smithi071.stdout:2486: left oid 40 (ObjNum 896 snap 276 seq_num 896) 2024-09-11T01:45:34.970 INFO:tasks.rados.rados.0.smithi071.stdout:2486: done (2 left) 2024-09-11T01:45:34.970 INFO:tasks.rados.rados.0.smithi071.stdout:2488: done (1 left) 2024-09-11T01:45:34.970 INFO:tasks.rados.rados.0.smithi071.stdout:2489: copy_from oid 22 from oid 32 current snap is 276 2024-09-11T01:45:34.971 INFO:tasks.rados.rados.0.smithi071.stdout:2490: rollback oid 39 current snap is 276 2024-09-11T01:45:34.971 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 39 to 271 2024-09-11T01:45:34.971 INFO:tasks.rados.rados.0.smithi071.stdout:2491: read oid 4 snap -1 2024-09-11T01:45:34.971 INFO:tasks.rados.rados.0.smithi071.stdout:2491: expect (ObjNum 828 snap 252 seq_num 828) 2024-09-11T01:45:34.971 INFO:tasks.rados.rados.0.smithi071.stdout:2492: rollback oid 37 current snap is 276 2024-09-11T01:45:34.971 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 37 to 254 2024-09-11T01:45:34.971 INFO:tasks.rados.rados.0.smithi071.stdout:2493: snap_create 2024-09-11T01:45:34.976 INFO:tasks.rados.rados.0.smithi071.stdout:2490: finishing rollback tid 1 to smithi07134777-39 2024-09-11T01:45:34.976 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 1017 (ObjNum 129 snap 0 seq_num 2370740352) dirty dne 2024-09-11T01:45:34.981 INFO:tasks.rados.rados.0.smithi071.stdout:2492: finishing rollback tid 0 to smithi07134777-37 2024-09-11T01:45:34.982 INFO:tasks.rados.rados.0.smithi071.stdout:2492: finishing rollback tid 1 to smithi07134777-37 2024-09-11T01:45:34.982 INFO:tasks.rados.rados.0.smithi071.stdout:2492: finishing rollback tid 2 to smithi07134777-37 2024-09-11T01:45:34.983 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 1023 (ObjNum 723 snap 224 seq_num 723) dirty exists 2024-09-11T01:45:34.985 INFO:tasks.rados.rados.0.smithi071.stdout:2487: finishing copy_from racing read to smithi07134777-24 2024-09-11T01:45:34.988 INFO:tasks.rados.rados.0.smithi071.stdout:2491: expect (ObjNum 828 snap 252 seq_num 828) 2024-09-11T01:45:35.031 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:45:35.072 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:45:35.114 INFO:tasks.rados.rados.0.smithi071.stdout:2487: finishing copy_from to smithi07134777-24 2024-09-11T01:45:35.114 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 415 (ObjNum 892 snap 274 seq_num 892) dirty exists 2024-09-11T01:45:35.114 INFO:tasks.rados.rados.0.smithi071.stdout:2489: finishing copy_from racing read to smithi07134777-22 2024-09-11T01:45:35.114 INFO:tasks.rados.rados.0.smithi071.stdout:2489: finishing copy_from to smithi07134777-22 2024-09-11T01:45:35.114 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 1007 (ObjNum 745 snap 227 seq_num 745) dirty exists 2024-09-11T01:45:35.953 INFO:tasks.rados.rados.0.smithi071.stdout:2487: done (5 left) 2024-09-11T01:45:35.954 INFO:tasks.rados.rados.0.smithi071.stdout:2489: done (4 left) 2024-09-11T01:45:35.954 INFO:tasks.rados.rados.0.smithi071.stdout:2490: done (3 left) 2024-09-11T01:45:35.954 INFO:tasks.rados.rados.0.smithi071.stdout:2491: done (2 left) 2024-09-11T01:45:35.954 INFO:tasks.rados.rados.0.smithi071.stdout:2492: done (1 left) 2024-09-11T01:45:35.954 INFO:tasks.rados.rados.0.smithi071.stdout:2493: done (0 left) 2024-09-11T01:45:35.954 INFO:tasks.rados.rados.0.smithi071.stdout:2494: delete oid 25 current snap is 277 2024-09-11T01:45:35.959 INFO:tasks.rados.rados.0.smithi071.stdout:2494: done (0 left) 2024-09-11T01:45:35.959 INFO:tasks.rados.rados.0.smithi071.stdout:2495: read oid 14 snap -1 2024-09-11T01:45:35.959 INFO:tasks.rados.rados.0.smithi071.stdout:2495: expect (ObjNum 865 snap 266 seq_num 865) 2024-09-11T01:45:35.959 INFO:tasks.rados.rados.0.smithi071.stdout:2496: rollback oid 8 current snap is 277 2024-09-11T01:45:35.959 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 8 to 272 2024-09-11T01:45:35.959 INFO:tasks.rados.rados.0.smithi071.stdout:2497: read oid 44 snap -1 2024-09-11T01:45:35.959 INFO:tasks.rados.rados.0.smithi071.stdout:2497: expect (ObjNum 779 snap 236 seq_num 779) 2024-09-11T01:45:35.959 INFO:tasks.rados.rados.0.smithi071.stdout:2498: write oid 20 current snap is 277 2024-09-11T01:45:35.960 INFO:tasks.rados.rados.0.smithi071.stdout:2498: seq_num 899 ranges {709493=578505,1907312=685978} 2024-09-11T01:45:35.966 INFO:tasks.rados.rados.0.smithi071.stdout:2498: writing smithi07134777-20 from 709493 to 1287998 tid 1 2024-09-11T01:45:35.974 INFO:tasks.rados.rados.0.smithi071.stdout:2498: writing smithi07134777-20 from 1907312 to 2593290 tid 2 2024-09-11T01:45:35.974 INFO:tasks.rados.rados.0.smithi071.stdout:2499: delete oid 22 current snap is 277 2024-09-11T01:45:35.974 INFO:tasks.rados.rados.0.smithi071.stdout:2496: finishing rollback tid 1 to smithi07134777-8 2024-09-11T01:45:35.974 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 1055 (ObjNum 129 snap 0 seq_num 2370486560) dirty dne 2024-09-11T01:45:35.974 INFO:tasks.rados.rados.0.smithi071.stdout:2495: expect (ObjNum 865 snap 266 seq_num 865) 2024-09-11T01:45:36.005 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1776599 2024-09-11T01:45:36.022 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 636789 2024-09-11T01:45:36.040 INFO:tasks.rados.rados.0.smithi071.stdout:2497: expect (ObjNum 779 snap 236 seq_num 779) 2024-09-11T01:45:36.093 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:45:36.213 INFO:tasks.rados.rados.0.smithi071.stdout:2498: finishing write tid 1 to smithi07134777-20 2024-09-11T01:45:36.213 INFO:tasks.rados.rados.0.smithi071.stdout:2498: finishing write tid 2 to smithi07134777-20 2024-09-11T01:45:36.213 INFO:tasks.rados.rados.0.smithi071.stdout:2498: finishing write tid 3 to smithi07134777-20 2024-09-11T01:45:36.214 INFO:tasks.rados.rados.0.smithi071.stdout:2498: finishing write tid 4 to smithi07134777-20 2024-09-11T01:45:36.214 INFO:tasks.rados.rados.0.smithi071.stdout:2498: oid 20 updating version 0 to 1009 2024-09-11T01:45:36.214 INFO:tasks.rados.rados.0.smithi071.stdout:2498: oid 20 version 1009 is already newer than 1008 2024-09-11T01:45:36.214 INFO:tasks.rados.rados.0.smithi071.stdout:2498: oid 20 updating version 1009 to 1010 2024-09-11T01:45:36.214 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 1010 (ObjNum 899 snap 277 seq_num 899) dirty exists 2024-09-11T01:45:36.214 INFO:tasks.rados.rados.0.smithi071.stdout:2498: left oid 20 (ObjNum 899 snap 277 seq_num 899) 2024-09-11T01:45:36.214 INFO:tasks.rados.rados.0.smithi071.stdout:2495: done (4 left) 2024-09-11T01:45:36.214 INFO:tasks.rados.rados.0.smithi071.stdout:2496: done (3 left) 2024-09-11T01:45:36.214 INFO:tasks.rados.rados.0.smithi071.stdout:2497: done (2 left) 2024-09-11T01:45:36.214 INFO:tasks.rados.rados.0.smithi071.stdout:2498: done (1 left) 2024-09-11T01:45:36.214 INFO:tasks.rados.rados.0.smithi071.stdout:2499: done (0 left) 2024-09-11T01:45:36.214 INFO:tasks.rados.rados.0.smithi071.stdout:2500: read oid 2 snap -1 2024-09-11T01:45:36.214 INFO:tasks.rados.rados.0.smithi071.stdout:2500: expect (ObjNum 875 snap 267 seq_num 875) 2024-09-11T01:45:36.214 INFO:tasks.rados.rados.0.smithi071.stdout:2501: snap_create 2024-09-11T01:45:36.216 INFO:tasks.rados.rados.0.smithi071.stdout:2500: expect (ObjNum 875 snap 267 seq_num 875) 2024-09-11T01:45:36.275 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:45:36.959 INFO:tasks.rados.rados.0.smithi071.stdout:2500: done (1 left) 2024-09-11T01:45:36.959 INFO:tasks.rados.rados.0.smithi071.stdout:2501: done (0 left) 2024-09-11T01:45:36.960 INFO:tasks.rados.rados.0.smithi071.stdout:2502: copy_from oid 26 from oid 1 current snap is 278 2024-09-11T01:45:36.960 INFO:tasks.rados.rados.0.smithi071.stdout:2503: rollback oid 15 current snap is 278 2024-09-11T01:45:36.960 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 15 to 264 2024-09-11T01:45:36.960 INFO:tasks.rados.rados.0.smithi071.stdout:2504: read oid 30 snap 262 2024-09-11T01:45:36.960 INFO:tasks.rados.rados.0.smithi071.stdout:2504: expect deleted 2024-09-11T01:45:36.960 INFO:tasks.rados.rados.0.smithi071.stdout:2505: snap_create 2024-09-11T01:45:36.967 INFO:tasks.rados.rados.0.smithi071.stdout:2503: finishing rollback tid 0 to smithi07134777-15 2024-09-11T01:45:36.967 INFO:tasks.rados.rados.0.smithi071.stdout:2503: finishing rollback tid 1 to smithi07134777-15 2024-09-11T01:45:36.967 INFO:tasks.rados.rados.0.smithi071.stdout:2503: finishing rollback tid 2 to smithi07134777-15 2024-09-11T01:45:36.967 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 1604 (ObjNum 565 snap 160 seq_num 565) dirty exists 2024-09-11T01:45:36.972 INFO:tasks.rados.rados.0.smithi071.stdout:2502: finishing copy_from racing read to smithi07134777-26 2024-09-11T01:45:36.973 INFO:tasks.rados.rados.0.smithi071.stdout:2502: finishing copy_from to smithi07134777-26 2024-09-11T01:45:36.973 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 1067 (ObjNum 688 snap 204 seq_num 688) dirty exists 2024-09-11T01:45:37.964 INFO:tasks.rados.rados.0.smithi071.stdout:2502: done (3 left) 2024-09-11T01:45:37.964 INFO:tasks.rados.rados.0.smithi071.stdout:2503: done (2 left) 2024-09-11T01:45:37.964 INFO:tasks.rados.rados.0.smithi071.stdout:2504: done (1 left) 2024-09-11T01:45:37.964 INFO:tasks.rados.rados.0.smithi071.stdout:2505: done (0 left) 2024-09-11T01:45:37.964 INFO:tasks.rados.rados.0.smithi071.stdout:2506: copy_from oid 20 from oid 37 current snap is 279 2024-09-11T01:45:37.964 INFO:tasks.rados.rados.0.smithi071.stdout:2507: rollback oid 40 current snap is 279 2024-09-11T01:45:37.964 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 40 to 238 2024-09-11T01:45:37.965 INFO:tasks.rados.rados.0.smithi071.stdout:2508: snap_remove snap 275 2024-09-11T01:45:38.953 INFO:tasks.rados.rados.0.smithi071.stdout:2508: done (2 left) 2024-09-11T01:45:38.953 INFO:tasks.rados.rados.0.smithi071.stdout:2509: snap_create 2024-09-11T01:45:38.953 INFO:tasks.rados.rados.0.smithi071.stdout:2507: finishing rollback tid 0 to smithi07134777-40 2024-09-11T01:45:38.953 INFO:tasks.rados.rados.0.smithi071.stdout:2507: finishing rollback tid 1 to smithi07134777-40 2024-09-11T01:45:38.953 INFO:tasks.rados.rados.0.smithi071.stdout:2507: finishing rollback tid 2 to smithi07134777-40 2024-09-11T01:45:38.953 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 1537 (ObjNum 686 snap 204 seq_num 686) dirty exists 2024-09-11T01:45:38.953 INFO:tasks.rados.rados.0.smithi071.stdout:2506: finishing copy_from racing read to smithi07134777-20 2024-09-11T01:45:38.953 INFO:tasks.rados.rados.0.smithi071.stdout:2506: finishing copy_from to smithi07134777-20 2024-09-11T01:45:38.953 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 1023 (ObjNum 723 snap 224 seq_num 723) dirty exists 2024-09-11T01:45:39.312 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-11T01:45:39.313 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-11T01:45:39.313 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-11T01:45:39.316 INFO:tasks.ceph.mon.c:Started 2024-09-11T01:45:39.316 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:45:39.316 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:45:39.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.427+0000 7f887e7a1640 10 monclient: get_monmap_and_config 2024-09-11T01:45:39.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.427+0000 7f887e7a1640 10 monclient: build_initial_monmap 2024-09-11T01:45:39.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.427+0000 7f887e7a1640 10 monclient: monmap: 2024-09-11T01:45:39.429 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:39.429 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:39.429 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:45:39.428409+0000 2024-09-11T01:45:39.429 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:45:39.428409+0000 2024-09-11T01:45:39.429 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:39.429 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:39.430 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:45:39.430 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:45:39.430 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:45:39.430 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:45:39.430 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:45:39.430 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:39.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.428+0000 7f887e7a1640 1 Processor -- start 2024-09-11T01:45:39.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.428+0000 7f887e7a1640 1 -- start start 2024-09-11T01:45:39.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.428+0000 7f887e7a1640 10 monclient: init 2024-09-11T01:45:39.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.428+0000 7f887e7a1640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:39.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.428+0000 7f887e7a1640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-11T01:45:39.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887e7a1640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8878138130 0x7f8878138510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:39.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887e7a1640 10 monclient(hunting): picked mon.noname-a con 0x7f8878138130 addr v2:172.21.15.12:3300/0 2024-09-11T01:45:39.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887e7a1640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f88780a4f20 0x7f88780a5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:39.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887e7a1640 10 monclient(hunting): picked mon.noname-c con 0x7f88780a4f20 addr v2:172.21.15.71:3300/0 2024-09-11T01:45:39.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887e7a1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f88780a5890 addr v1:172.21.15.12:6789/0 2024-09-11T01:45:39.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887e7a1640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88780a6050 con 0x7f88780a5890 2024-09-11T01:45:39.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887e7a1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:39.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887e7a1640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f88780a6710 con 0x7f8878138130 2024-09-11T01:45:39.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887e7a1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:39.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887e7a1640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f887809ba20 con 0x7f88780a4f20 2024-09-11T01:45:39.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887e7a1640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:39.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887e7a1640 10 monclient(hunting): authenticate will time out at 1745.871080s 2024-09-11T01:45:39.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887d79f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8878138130 0x7f8878138510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:39.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887d79f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8878138130 0x7f8878138510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59776/0 (socket says 172.21.15.12:59776) 2024-09-11T01:45:39.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887d79f640 1 -- 172.21.15.12:0/631367680 learned_addr learned my addr 172.21.15.12:0/631367680 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:39.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887d79f640 10 monclient(hunting): get_auth_request con 0x7f8878138130 auth_method 0 2024-09-11T01:45:39.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887d79f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:39.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887d79f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:39.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887d79f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:39.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887cf9e640 1 --2- 172.21.15.12:0/631367680 >> v2:172.21.15.71:3300/0 conn(0x7f88780a4f20 0x7f88780a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:39.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887cf9e640 10 monclient(hunting): get_auth_request con 0x7f88780a4f20 auth_method 0 2024-09-11T01:45:39.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887cf9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:39.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887cf9e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:39.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887cf9e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:39.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887cf9e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:39.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887cf9e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:39.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887cf9e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:39.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887cf9e640 10 monclient(hunting): handle_auth_done global_id 134527 payload 274 2024-09-11T01:45:39.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887cf9e640 1 -- 172.21.15.12:0/631367680 >> v2:172.21.15.12:3300/0 conn(0x7f8878138130 msgr2=0x7f8878138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:39.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887cf9e640 1 --2- 172.21.15.12:0/631367680 >> v2:172.21.15.12:3300/0 conn(0x7f8878138130 0x7f8878138510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:39.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887cf9e640 1 -- 172.21.15.12:0/631367680 >> v1:172.21.15.12:6789/0 conn(0x7f88780a5890 legacy=0x7f887813ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:39.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887cf9e640 10 monclient: _finish_hunting 0 2024-09-11T01:45:39.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887cf9e640 1 monclient: found mon.noname-c 2024-09-11T01:45:39.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887cf9e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:39.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887cf9e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:45:39.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.429+0000 7f887cf9e640 1 -- 172.21.15.12:0/631367680 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88780a6e10 con 0x7f88780a4f20 2024-09-11T01:45:39.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.432+0000 7f887cf9e640 1 -- 172.21.15.12:0/631367680 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88780a4f20 msgr2=0x7f88780a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:39.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.432+0000 7f887cf9e640 1 --2- 172.21.15.12:0/631367680 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88780a4f20 0x7f88780a5350 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f88780a6a70 tx=0x7f886802a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80595cf154667fdc server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:39.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.433+0000 7f886ffff640 1 -- 172.21.15.12:0/631367680 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8868037070 con 0x7f88780a4f20 2024-09-11T01:45:39.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.433+0000 7f886ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:39.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.433+0000 7f886ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:45:39.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.433+0000 7f886ffff640 10 monclient: dump: 2024-09-11T01:45:39.440 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:39.440 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:39.440 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:39.440 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:39.440 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:39.440 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:39.441 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.433+0000 7f886ffff640 10 monclient: _finish_auth 0 2024-09-11T01:45:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.433+0000 7f886ffff640 10 monclient: _check_auth_tickets 2024-09-11T01:45:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.433+0000 7f886ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.433+0000 7f886ffff640 1 -- 172.21.15.12:0/631367680 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8868032440 con 0x7f88780a4f20 2024-09-11T01:45:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.433+0000 7f886ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.433+0000 7f886ffff640 1 -- 172.21.15.12:0/631367680 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8868032810 con 0x7f88780a4f20 2024-09-11T01:45:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.433+0000 7f886ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.433+0000 7f886ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:45:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.433+0000 7f886ffff640 10 monclient: dump: 2024-09-11T01:45:39.442 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:39.442 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:39.442 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:39.442 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:39.442 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:39.442 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:39.442 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:39.442 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:39.442 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:39.442 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:39.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.433+0000 7f887e7a1640 5 monclient: authenticate success, global_id 134527 2024-09-11T01:45:39.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.433+0000 7f887e7a1640 10 monclient: get_monmap_and_config success 2024-09-11T01:45:39.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.433+0000 7f887e7a1640 4 set_mon_vals no callback set 2024-09-11T01:45:39.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.433+0000 7f887e7a1640 10 monclient: shutdown 2024-09-11T01:45:39.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.433+0000 7f887e7a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:39.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.433+0000 7f887e7a1640 1 -- 172.21.15.12:0/631367680 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88780a4f20 msgr2=0x7f88780a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:39.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.433+0000 7f887e7a1640 1 --2- 172.21.15.12:0/631367680 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88780a4f20 0x7f88780a5350 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f88780a6a70 tx=0x7f886802a000 comp rx=0 tx=0).stop 2024-09-11T01:45:39.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.433+0000 7f887e7a1640 1 -- 172.21.15.12:0/631367680 shutdown_connections 2024-09-11T01:45:39.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.433+0000 7f887e7a1640 1 --2- 172.21.15.12:0/631367680 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88780a4f20 0x7f88780a5350 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:39.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.433+0000 7f887e7a1640 1 --2- 172.21.15.12:0/631367680 >> v2:172.21.15.12:3300/0 conn(0x7f8878138130 0x7f8878138510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:39.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.433+0000 7f887e7a1640 1 -- 172.21.15.12:0/631367680 >> 172.21.15.12:0/631367680 conn(0x7f8878068b30 msgr2=0x7f8878068f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:39.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.433+0000 7f887e7a1640 1 -- 172.21.15.12:0/631367680 shutdown_connections 2024-09-11T01:45:39.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.434+0000 7f887e7a1640 1 -- 172.21.15.12:0/631367680 wait complete. 2024-09-11T01:45:39.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.434+0000 7f887e7a1640 10 monclient: build_initial_monmap 2024-09-11T01:45:39.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.434+0000 7f887e7a1640 10 monclient: monmap: 2024-09-11T01:45:39.443 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:39.443 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:45:39.444 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:45:39.444 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:45:39.444 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:39.444 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:39.444 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:45:39.444 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:45:39.444 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:45:39.444 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:39.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.434+0000 7f887e7a1640 1 Processor -- start 2024-09-11T01:45:39.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.434+0000 7f887e7a1640 1 -- start start 2024-09-11T01:45:39.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.434+0000 7f887e7a1640 10 monclient: init 2024-09-11T01:45:39.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887e7a1640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:39.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887e7a1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:45:39.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887e7a1640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88780a4f20 0x7f88781ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:39.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887e7a1640 10 monclient(hunting): picked mon.noname-c con 0x7f88780a4f20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:39.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887e7a1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88780a5890 0x7f8878136d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:39.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887e7a1640 10 monclient(hunting): picked mon.noname-b con 0x7f88780a5890 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:45:39.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887e7a1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8878138130 0x7f8878137250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:39.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887e7a1640 10 monclient(hunting): picked mon.noname-a con 0x7f8878138130 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:45:39.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887e7a1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:39.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887e7a1640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f88780a5cc0 con 0x7f8878138130 2024-09-11T01:45:39.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887e7a1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:39.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887e7a1640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f887809caf0 con 0x7f88780a5890 2024-09-11T01:45:39.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887e7a1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:39.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887e7a1640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8878138a50 con 0x7f88780a4f20 2024-09-11T01:45:39.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887e7a1640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:39.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887e7a1640 10 monclient(hunting): authenticate will time out at 1745.877474s 2024-09-11T01:45:39.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887d79f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88780a4f20 0x7f88781ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:39.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887dfa0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8878138130 0x7f8878137250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:39.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887d79f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88780a4f20 0x7f88781ccdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42666/0 (socket says 172.21.15.12:42666) 2024-09-11T01:45:39.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887d79f640 1 -- 172.21.15.12:0/3632374380 learned_addr learned my addr 172.21.15.12:0/3632374380 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:39.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887d79f640 10 monclient(hunting): get_auth_request con 0x7f88780a4f20 auth_method 0 2024-09-11T01:45:39.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887d79f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:39.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887d79f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:39.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887d79f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:39.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887dfa0640 10 monclient(hunting): get_auth_request con 0x7f8878138130 auth_method 0 2024-09-11T01:45:39.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887dfa0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:39.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887dfa0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:39.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887dfa0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:39.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887d79f640 1 -- 172.21.15.12:0/3632374380 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88780a4f20 msgr2=0x7f88781ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:45:39.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887d79f640 1 -- 172.21.15.12:0/3632374380 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88780a4f20 msgr2=0x7f88781ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:45:39.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887d79f640 1 --2- 172.21.15.12:0/3632374380 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88780a4f20 0x7f88781ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:45:39.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887cf9e640 1 --2- 172.21.15.12:0/3632374380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88780a5890 0x7f8878136d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:39.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887dfa0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:39.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887dfa0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:39.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887d79f640 1 --2- 172.21.15.12:0/3632374380 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88780a4f20 0x7f88781ccdb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:39.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887dfa0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:39.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887cf9e640 10 monclient(hunting): get_auth_request con 0x7f88780a5890 auth_method 0 2024-09-11T01:45:39.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887cf9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:39.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887cf9e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:39.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887cf9e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:39.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887cf9e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:39.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887cf9e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:39.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887cf9e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:39.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887dfa0640 10 monclient(hunting): handle_auth_done global_id 174192 payload 824 2024-09-11T01:45:39.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887dfa0640 1 -- 172.21.15.12:0/3632374380 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88780a4f20 msgr2=0x7f88781ccdb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:45:39.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887dfa0640 1 --2- 172.21.15.12:0/3632374380 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88780a4f20 0x7f88781ccdb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:39.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887dfa0640 1 -- 172.21.15.12:0/3632374380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88780a5890 msgr2=0x7f8878136d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:39.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887dfa0640 1 --2- 172.21.15.12:0/3632374380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88780a5890 0x7f8878136d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:39.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887dfa0640 10 monclient: _finish_hunting 0 2024-09-11T01:45:39.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887dfa0640 1 monclient: found mon.noname-a 2024-09-11T01:45:39.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.435+0000 7f887dfa0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:39.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.436+0000 7f887dfa0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:45:39.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.436+0000 7f887dfa0640 1 -- 172.21.15.12:0/3632374380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88781cd2f0 con 0x7f8878138130 2024-09-11T01:45:39.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.438+0000 7f887dfa0640 1 -- 172.21.15.12:0/3632374380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8878138130 msgr2=0x7f8878137250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:39.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.438+0000 7f887dfa0640 1 --2- 172.21.15.12:0/3632374380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8878138130 0x7f8878137250 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f887000c780 tx=0x7f887000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:39.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.438+0000 7f886dffb640 1 -- 172.21.15.12:0/3632374380 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8870016020 con 0x7f8878138130 2024-09-11T01:45:39.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.438+0000 7f886dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:39.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.438+0000 7f886dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:45:39.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.438+0000 7f886dffb640 10 monclient: dump: 2024-09-11T01:45:39.449 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:39.449 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:39.449 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:39.449 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:39.449 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:39.449 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:39.449 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:39.449 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:39.449 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:39.449 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:39.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.438+0000 7f886dffb640 10 monclient: _finish_auth 0 2024-09-11T01:45:39.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.438+0000 7f886dffb640 10 monclient: _check_auth_tickets 2024-09-11T01:45:39.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.438+0000 7f886dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:39.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.438+0000 7f887e7a1640 5 monclient: authenticate success, global_id 174192 2024-09-11T01:45:39.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.438+0000 7f886dffb640 1 -- 172.21.15.12:0/3632374380 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f887000ac60 con 0x7f8878138130 2024-09-11T01:45:39.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.438+0000 7f886dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:39.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.438+0000 7f887e7a1640 10 monclient: _renew_subs 2024-09-11T01:45:39.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.438+0000 7f886dffb640 1 -- 172.21.15.12:0/3632374380 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8870012660 con 0x7f8878138130 2024-09-11T01:45:39.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.438+0000 7f887e7a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:39.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.438+0000 7f887e7a1640 1 -- 172.21.15.12:0/3632374380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8878123e30 con 0x7f8878138130 2024-09-11T01:45:39.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.438+0000 7f886dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:39.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.438+0000 7f886dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:39.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.438+0000 7f886f7fe640 4 set_mon_vals no callback set 2024-09-11T01:45:39.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.438+0000 7f886dffb640 10 monclient: dump: 2024-09-11T01:45:39.451 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:39.451 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:39.451 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:39.451 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:39.451 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:39.451 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:39.451 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:39.451 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:39.451 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:39.451 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:39.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.438+0000 7f887e7a1640 10 monclient: _renew_subs 2024-09-11T01:45:39.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.438+0000 7f887e7a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:39.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.438+0000 7f887e7a1640 1 -- 172.21.15.12:0/3632374380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88780a1880 con 0x7f8878138130 2024-09-11T01:45:39.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.439+0000 7f886dffb640 1 -- 172.21.15.12:0/3632374380 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f88700128c0 con 0x7f8878138130 2024-09-11T01:45:39.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.439+0000 7f887e7a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:39.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.439+0000 7f887e7a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:39.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.439+0000 7f887e7a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:39.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.439+0000 7f886dffb640 1 --2- 172.21.15.12:0/3632374380 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f885c03e210 0x7f885c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:39.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.439+0000 7f887e7a1640 1 -- 172.21.15.12:0/3632374380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f887806f710 con 0x7f8878138130 2024-09-11T01:45:39.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.439+0000 7f886dffb640 1 -- 172.21.15.12:0/3632374380 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(558..558 src has 317..558) ==== 4843+0+0 (secure 0 0 0) 0x7f8870052f60 con 0x7f8878138130 2024-09-11T01:45:39.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.440+0000 7f887d79f640 1 --2- 172.21.15.12:0/3632374380 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f885c03e210 0x7f885c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:39.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.440+0000 7f887d79f640 10 monclient: get_auth_request con 0x7f885c03e210 auth_method 0 2024-09-11T01:45:39.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.442+0000 7f887d79f640 1 --2- 172.21.15.12:0/3632374380 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f885c03e210 0x7f885c0406d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8874006a90 tx=0x7f8874008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:39.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.444+0000 7f886dffb640 1 -- 172.21.15.12:0/3632374380 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8870020030 con 0x7f8878138130 2024-09-11T01:45:39.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.444+0000 7f886dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:39.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.444+0000 7f886dffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:45:39.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.703+0000 7f887e7a1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:45:39.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.703+0000 7f887e7a1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:45:39.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.703+0000 7f887e7a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:39.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.703+0000 7f887e7a1640 1 -- 172.21.15.12:0/3632374380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8878074a60 con 0x7f8878138130 2024-09-11T01:45:39.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.703+0000 7f886dffb640 1 -- 172.21.15.12:0/3632374380 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f887000ae10 con 0x7f8878138130 2024-09-11T01:45:39.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.703+0000 7f886dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:45:39.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.703+0000 7f886dffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:45:39.705 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":152,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:45:39.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.706+0000 7f887e7a1640 1 -- 172.21.15.12:0/3632374380 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f885c03e210 msgr2=0x7f885c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:39.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.706+0000 7f887e7a1640 1 --2- 172.21.15.12:0/3632374380 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f885c03e210 0x7f885c0406d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8874006a90 tx=0x7f8874008040 comp rx=0 tx=0).stop 2024-09-11T01:45:39.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.706+0000 7f887e7a1640 10 monclient: shutdown 2024-09-11T01:45:39.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.706+0000 7f887e7a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:39.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.706+0000 7f887e7a1640 1 -- 172.21.15.12:0/3632374380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8878138130 msgr2=0x7f8878137250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:39.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.706+0000 7f887e7a1640 1 --2- 172.21.15.12:0/3632374380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8878138130 0x7f8878137250 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f887000c780 tx=0x7f887000cc40 comp rx=0 tx=0).stop 2024-09-11T01:45:39.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.706+0000 7f887e7a1640 1 -- 172.21.15.12:0/3632374380 shutdown_connections 2024-09-11T01:45:39.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.706+0000 7f887e7a1640 1 --2- 172.21.15.12:0/3632374380 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f885c03e210 0x7f885c0406d0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:39.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.706+0000 7f887e7a1640 1 --2- 172.21.15.12:0/3632374380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8878138130 0x7f8878137250 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:39.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.706+0000 7f887e7a1640 1 --2- 172.21.15.12:0/3632374380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88780a5890 0x7f8878136d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:39.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.706+0000 7f887e7a1640 1 --2- 172.21.15.12:0/3632374380 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f88780a4f20 0x7f88781ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:39.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.706+0000 7f887e7a1640 1 -- 172.21.15.12:0/3632374380 >> 172.21.15.12:0/3632374380 conn(0x7f8878068b30 msgr2=0x7f88780a02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:39.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.706+0000 7f887e7a1640 1 -- 172.21.15.12:0/3632374380 shutdown_connections 2024-09-11T01:45:39.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:39.706+0000 7f887e7a1640 1 -- 172.21.15.12:0/3632374380 wait complete. 2024-09-11T01:45:39.958 INFO:tasks.rados.rados.0.smithi071.stdout:2506: done (2 left) 2024-09-11T01:45:39.958 INFO:tasks.rados.rados.0.smithi071.stdout:2507: done (1 left) 2024-09-11T01:45:39.958 INFO:tasks.rados.rados.0.smithi071.stdout:2509: done (0 left) 2024-09-11T01:45:39.958 INFO:tasks.rados.rados.0.smithi071.stdout:2510: rollback oid 24 current snap is 280 2024-09-11T01:45:39.958 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 24 to 266 2024-09-11T01:45:39.958 INFO:tasks.rados.rados.0.smithi071.stdout:2511: snap_create 2024-09-11T01:45:39.962 INFO:tasks.rados.rados.0.smithi071.stdout:2510: finishing rollback tid 0 to smithi07134777-24 2024-09-11T01:45:39.962 INFO:tasks.rados.rados.0.smithi071.stdout:2510: finishing rollback tid 1 to smithi07134777-24 2024-09-11T01:45:39.962 INFO:tasks.rados.rados.0.smithi071.stdout:2510: finishing rollback tid 2 to smithi07134777-24 2024-09-11T01:45:39.962 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 1189 (ObjNum 766 snap 232 seq_num 766) dirty exists 2024-09-11T01:45:40.959 INFO:tasks.rados.rados.0.smithi071.stdout:2510: done (1 left) 2024-09-11T01:45:40.959 INFO:tasks.rados.rados.0.smithi071.stdout:2511: done (0 left) 2024-09-11T01:45:40.959 INFO:tasks.rados.rados.0.smithi071.stdout:2512: snap_remove snap 238 2024-09-11T01:45:42.580 INFO:tasks.rados.rados.0.smithi071.stdout:2512: done (0 left) 2024-09-11T01:45:42.580 INFO:tasks.rados.rados.0.smithi071.stdout:2513: snap_remove snap 260 2024-09-11T01:45:42.719 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:45:42.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.835+0000 7f5d384ef640 10 monclient: get_monmap_and_config 2024-09-11T01:45:42.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.835+0000 7f5d384ef640 10 monclient: build_initial_monmap 2024-09-11T01:45:42.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.835+0000 7f5d384ef640 10 monclient: monmap: 2024-09-11T01:45:42.836 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:42.837 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:42.837 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:45:42.835848+0000 2024-09-11T01:45:42.837 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:45:42.835848+0000 2024-09-11T01:45:42.837 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:42.837 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:42.837 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:45:42.837 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:45:42.837 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:45:42.837 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:45:42.837 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:45:42.837 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:42.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.835+0000 7f5d384ef640 1 Processor -- start 2024-09-11T01:45:42.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.835+0000 7f5d384ef640 1 -- start start 2024-09-11T01:45:42.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.835+0000 7f5d384ef640 10 monclient: init 2024-09-11T01:45:42.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d384ef640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:42.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d384ef640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-11T01:45:42.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d384ef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5d30138230 addr v1:172.21.15.71:6789/0 2024-09-11T01:45:42.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d384ef640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d3009d030 0x7f5d3013d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:42.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d384ef640 10 monclient(hunting): picked mon.noname-b con 0x7f5d3009d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:42.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d384ef640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5d3009d460 0x7f5d3013f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:42.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d384ef640 10 monclient(hunting): picked mon.noname-c con 0x7f5d3009d460 addr v2:172.21.15.71:3300/0 2024-09-11T01:45:42.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d384ef640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d300a6e30 con 0x7f5d30138230 2024-09-11T01:45:42.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d384ef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:42.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d384ef640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f5d3009c9c0 con 0x7f5d3009d460 2024-09-11T01:45:42.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d384ef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:42.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d384ef640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5d3009ba20 con 0x7f5d3009d030 2024-09-11T01:45:42.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d384ef640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:42.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d384ef640 10 monclient(hunting): authenticate will time out at 1749.278499s 2024-09-11T01:45:42.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d35a8a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d3009d030 0x7f5d3013d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:42.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d35a8a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d3009d030 0x7f5d3013d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42694/0 (socket says 172.21.15.12:42694) 2024-09-11T01:45:42.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d35a8a640 1 -- 172.21.15.12:0/2665149779 learned_addr learned my addr 172.21.15.12:0/2665149779 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:42.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d35a8a640 10 monclient(hunting): get_auth_request con 0x7f5d3009d030 auth_method 0 2024-09-11T01:45:42.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d35a8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:42.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d35a8a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:42.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d35a8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:42.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d36a8c640 1 --2- 172.21.15.12:0/2665149779 >> v2:172.21.15.71:3300/0 conn(0x7f5d3009d460 0x7f5d3013f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:42.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d36a8c640 10 monclient(hunting): get_auth_request con 0x7f5d3009d460 auth_method 0 2024-09-11T01:45:42.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d36a8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:42.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d35a8a640 1 -- 172.21.15.12:0/2665149779 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d3009d030 msgr2=0x7f5d3013d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:45:42.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d36a8c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:42.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d36a8c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:42.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d35a8a640 1 -- 172.21.15.12:0/2665149779 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d3009d030 msgr2=0x7f5d3013d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:45:42.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d35a8a640 1 --2- 172.21.15.12:0/2665149779 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d3009d030 0x7f5d3013d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:45:42.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.836+0000 7f5d35a8a640 1 --2- 172.21.15.12:0/2665149779 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d3009d030 0x7f5d3013d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:42.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.838+0000 7f5d3628b640 1 -- 172.21.15.12:0/2665149779 >> v1:172.21.15.71:6789/0 conn(0x7f5d30138230 legacy=0x7f5d30138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:42.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.838+0000 7f5d36a8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:42.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.838+0000 7f5d36a8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:42.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.839+0000 7f5d36a8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:42.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.839+0000 7f5d36a8c640 10 monclient(hunting): handle_auth_done global_id 134533 payload 274 2024-09-11T01:45:42.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.839+0000 7f5d36a8c640 1 -- 172.21.15.12:0/2665149779 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d3009d030 msgr2=0x7f5d3013d090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:45:42.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.839+0000 7f5d36a8c640 1 --2- 172.21.15.12:0/2665149779 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d3009d030 0x7f5d3013d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:42.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.839+0000 7f5d36a8c640 1 -- 172.21.15.12:0/2665149779 >> v1:172.21.15.71:6789/0 conn(0x7f5d30138230 legacy=0x7f5d30138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:42.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.839+0000 7f5d36a8c640 10 monclient: _finish_hunting 0 2024-09-11T01:45:42.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.839+0000 7f5d36a8c640 1 monclient: found mon.noname-c 2024-09-11T01:45:42.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.839+0000 7f5d36a8c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:42.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.839+0000 7f5d36a8c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:45:42.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.839+0000 7f5d36a8c640 1 -- 172.21.15.12:0/2665149779 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d300a66a0 con 0x7f5d3009d460 2024-09-11T01:45:42.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.842+0000 7f5d36a8c640 1 -- 172.21.15.12:0/2665149779 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d3009d460 msgr2=0x7f5d3013f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:42.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.842+0000 7f5d36a8c640 1 --2- 172.21.15.12:0/2665149779 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d3009d460 0x7f5d3013f5e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5d20009e10 tx=0x7f5d2002a5b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bce84a13793668c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:42.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.842+0000 7f5d35289640 1 -- 172.21.15.12:0/2665149779 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d20007950 con 0x7f5d3009d460 2024-09-11T01:45:42.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.842+0000 7f5d35289640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:42.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.842+0000 7f5d35289640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:45:42.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.842+0000 7f5d35289640 10 monclient: dump: 2024-09-11T01:45:42.851 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:42.851 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:42.851 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:42.851 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:42.851 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:42.851 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:42.851 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:42.851 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:42.851 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:42.851 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:42.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.842+0000 7f5d35289640 10 monclient: _finish_auth 0 2024-09-11T01:45:42.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.842+0000 7f5d35289640 10 monclient: _check_auth_tickets 2024-09-11T01:45:42.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.842+0000 7f5d35289640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:42.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.842+0000 7f5d35289640 1 -- 172.21.15.12:0/2665149779 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d20007af0 con 0x7f5d3009d460 2024-09-11T01:45:42.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.842+0000 7f5d35289640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:42.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.842+0000 7f5d35289640 1 -- 172.21.15.12:0/2665149779 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d20031400 con 0x7f5d3009d460 2024-09-11T01:45:42.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.842+0000 7f5d35289640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:42.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.842+0000 7f5d35289640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:45:42.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.842+0000 7f5d35289640 10 monclient: dump: 2024-09-11T01:45:42.852 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:42.852 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:42.852 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:42.852 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:42.853 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:42.853 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:42.853 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:42.853 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:42.853 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:42.853 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:42.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.842+0000 7f5d384ef640 5 monclient: authenticate success, global_id 134533 2024-09-11T01:45:42.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.842+0000 7f5d384ef640 10 monclient: get_monmap_and_config success 2024-09-11T01:45:42.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.842+0000 7f5d384ef640 4 set_mon_vals no callback set 2024-09-11T01:45:42.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.843+0000 7f5d384ef640 10 monclient: shutdown 2024-09-11T01:45:42.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.843+0000 7f5d384ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:42.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.843+0000 7f5d384ef640 1 -- 172.21.15.12:0/2665149779 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d3009d460 msgr2=0x7f5d3013f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:42.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.843+0000 7f5d384ef640 1 --2- 172.21.15.12:0/2665149779 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d3009d460 0x7f5d3013f5e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5d20009e10 tx=0x7f5d2002a5b0 comp rx=0 tx=0).stop 2024-09-11T01:45:42.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.843+0000 7f5d384ef640 1 -- 172.21.15.12:0/2665149779 shutdown_connections 2024-09-11T01:45:42.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.843+0000 7f5d384ef640 1 --2- 172.21.15.12:0/2665149779 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d3009d460 0x7f5d3013f5e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:42.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.843+0000 7f5d384ef640 1 --2- 172.21.15.12:0/2665149779 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d3009d030 0x7f5d3013d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:42.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.843+0000 7f5d384ef640 1 -- 172.21.15.12:0/2665149779 >> 172.21.15.12:0/2665149779 conn(0x7f5d300a7120 msgr2=0x7f5d300a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:42.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.843+0000 7f5d384ef640 1 -- 172.21.15.12:0/2665149779 shutdown_connections 2024-09-11T01:45:42.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.843+0000 7f5d384ef640 1 -- 172.21.15.12:0/2665149779 wait complete. 2024-09-11T01:45:42.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.843+0000 7f5d384ef640 10 monclient: build_initial_monmap 2024-09-11T01:45:42.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.843+0000 7f5d384ef640 10 monclient: monmap: 2024-09-11T01:45:42.854 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:42.854 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:45:42.854 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:45:42.854 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:45:42.854 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:42.855 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:42.855 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:45:42.855 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:45:42.855 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:45:42.855 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:42.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d384ef640 1 Processor -- start 2024-09-11T01:45:42.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d384ef640 1 -- start start 2024-09-11T01:45:42.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d384ef640 10 monclient: init 2024-09-11T01:45:42.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d384ef640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:42.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d384ef640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:45:42.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d384ef640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d3009d030 0x7f5d3013ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:42.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d384ef640 10 monclient(hunting): picked mon.noname-a con 0x7f5d3009d030 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:45:42.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d384ef640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d3009d460 0x7f5d301cd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:42.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d384ef640 10 monclient(hunting): picked mon.noname-b con 0x7f5d3009d460 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:45:42.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d384ef640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d30138230 0x7f5d301cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:42.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d384ef640 10 monclient(hunting): picked mon.noname-c con 0x7f5d30138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:42.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d384ef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:42.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d384ef640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3013fca0 con 0x7f5d3009d030 2024-09-11T01:45:42.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d384ef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:42.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d384ef640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3013fb20 con 0x7f5d3009d460 2024-09-11T01:45:42.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d384ef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:42.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d384ef640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5d3013fe20 con 0x7f5d30138230 2024-09-11T01:45:42.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d384ef640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:42.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d384ef640 10 monclient(hunting): authenticate will time out at 1749.286671s 2024-09-11T01:45:42.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d3628b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d3009d030 0x7f5d3013ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:42.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d36a8c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d30138230 0x7f5d301cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:42.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d3628b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d3009d030 0x7f5d3013ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59834/0 (socket says 172.21.15.12:59834) 2024-09-11T01:45:42.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d3628b640 1 -- 172.21.15.12:0/413391965 learned_addr learned my addr 172.21.15.12:0/413391965 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:42.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d3628b640 10 monclient(hunting): get_auth_request con 0x7f5d3009d030 auth_method 0 2024-09-11T01:45:42.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d3628b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:42.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d3628b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:42.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d3628b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:42.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d36a8c640 10 monclient(hunting): get_auth_request con 0x7f5d30138230 auth_method 0 2024-09-11T01:45:42.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d36a8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:42.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d36a8c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:42.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d36a8c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:42.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d35a8a640 1 --2- 172.21.15.12:0/413391965 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d3009d460 0x7f5d301cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:42.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d35a8a640 10 monclient(hunting): get_auth_request con 0x7f5d3009d460 auth_method 0 2024-09-11T01:45:42.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d35a8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:42.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d35a8a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:42.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d35a8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:42.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d3628b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:42.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d3628b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:42.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d3628b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:42.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d36a8c640 1 -- 172.21.15.12:0/413391965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d30138230 msgr2=0x7f5d301cd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:45:42.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d36a8c640 1 -- 172.21.15.12:0/413391965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d30138230 msgr2=0x7f5d301cd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:45:42.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d36a8c640 1 --2- 172.21.15.12:0/413391965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d30138230 0x7f5d301cd8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:45:42.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d35a8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d35a8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.845+0000 7f5d35a8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.844+0000 7f5d36a8c640 1 --2- 172.21.15.12:0/413391965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d30138230 0x7f5d301cd8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.845+0000 7f5d3628b640 10 monclient(hunting): handle_auth_done global_id 174195 payload 824 2024-09-11T01:45:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.845+0000 7f5d3628b640 1 -- 172.21.15.12:0/413391965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d30138230 msgr2=0x7f5d301cd8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:45:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.845+0000 7f5d3628b640 1 --2- 172.21.15.12:0/413391965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d30138230 0x7f5d301cd8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.845+0000 7f5d3628b640 1 -- 172.21.15.12:0/413391965 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d3009d460 msgr2=0x7f5d301cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.845+0000 7f5d3628b640 1 --2- 172.21.15.12:0/413391965 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d3009d460 0x7f5d301cd380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.845+0000 7f5d3628b640 10 monclient: _finish_hunting 0 2024-09-11T01:45:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.845+0000 7f5d3628b640 1 monclient: found mon.noname-a 2024-09-11T01:45:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.845+0000 7f5d3628b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:42.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.845+0000 7f5d3628b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:45:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.845+0000 7f5d3628b640 1 -- 172.21.15.12:0/413391965 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d30121290 con 0x7f5d3009d030 2024-09-11T01:45:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.845+0000 7f5d35a8a640 1 --2- 172.21.15.12:0/413391965 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d3009d460 0x7f5d301cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:45:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.847+0000 7f5d3628b640 1 -- 172.21.15.12:0/413391965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d3009d030 msgr2=0x7f5d3013ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.847+0000 7f5d3628b640 1 --2- 172.21.15.12:0/413391965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d3009d030 0x7f5d3013ec20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5d2800c490 tx=0x7f5d2800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.847+0000 7f5d2effd640 1 -- 172.21.15.12:0/413391965 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d28017070 con 0x7f5d3009d030 2024-09-11T01:45:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.847+0000 7f5d2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.847+0000 7f5d2effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:45:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.847+0000 7f5d2effd640 10 monclient: dump: 2024-09-11T01:45:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:42.860 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:42.861 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.847+0000 7f5d2effd640 10 monclient: _finish_auth 0 2024-09-11T01:45:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.847+0000 7f5d2effd640 10 monclient: _check_auth_tickets 2024-09-11T01:45:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.847+0000 7f5d2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.847+0000 7f5d384ef640 5 monclient: authenticate success, global_id 174195 2024-09-11T01:45:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.847+0000 7f5d2effd640 1 -- 172.21.15.12:0/413391965 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d280057c0 con 0x7f5d3009d030 2024-09-11T01:45:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.847+0000 7f5d2effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:42.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.847+0000 7f5d384ef640 10 monclient: _renew_subs 2024-09-11T01:45:42.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.847+0000 7f5d2effd640 1 -- 172.21.15.12:0/413391965 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d28005bb0 con 0x7f5d3009d030 2024-09-11T01:45:42.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.847+0000 7f5d384ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:42.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.847+0000 7f5d384ef640 1 -- 172.21.15.12:0/413391965 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d30121570 con 0x7f5d3009d030 2024-09-11T01:45:42.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.847+0000 7f5d2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:42.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.847+0000 7f5d2effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:42.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.847+0000 7f5d2effd640 10 monclient: dump: 2024-09-11T01:45:42.862 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:42.862 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:42.862 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:42.864 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:42.864 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:42.864 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:42.865 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:42.865 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:42.865 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:42.865 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:42.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.847+0000 7f5d34a88640 4 set_mon_vals no callback set 2024-09-11T01:45:42.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.847+0000 7f5d384ef640 10 monclient: _renew_subs 2024-09-11T01:45:42.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.847+0000 7f5d384ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:42.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.847+0000 7f5d384ef640 1 -- 172.21.15.12:0/413391965 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d30120fb0 con 0x7f5d3009d030 2024-09-11T01:45:42.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.848+0000 7f5d384ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:42.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.848+0000 7f5d384ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:42.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.848+0000 7f5d2effd640 1 -- 172.21.15.12:0/413391965 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f5d2801b3b0 con 0x7f5d3009d030 2024-09-11T01:45:42.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.848+0000 7f5d384ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:42.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.848+0000 7f5d384ef640 1 -- 172.21.15.12:0/413391965 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d30138610 con 0x7f5d3009d030 2024-09-11T01:45:42.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.848+0000 7f5d2effd640 1 --2- 172.21.15.12:0/413391965 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5d1c03e210 0x7f5d1c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:42.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.848+0000 7f5d2effd640 1 -- 172.21.15.12:0/413391965 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(561..561 src has 317..561) ==== 4799+0+0 (secure 0 0 0) 0x7f5d280522d0 con 0x7f5d3009d030 2024-09-11T01:45:42.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.849+0000 7f5d35a8a640 1 --2- 172.21.15.12:0/413391965 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5d1c03e210 0x7f5d1c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:42.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.849+0000 7f5d35a8a640 10 monclient: get_auth_request con 0x7f5d1c03e210 auth_method 0 2024-09-11T01:45:42.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.851+0000 7f5d35a8a640 1 --2- 172.21.15.12:0/413391965 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5d1c03e210 0x7f5d1c0406d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5d3013ebb0 tx=0x7f5d24008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:42.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.853+0000 7f5d2effd640 1 -- 172.21.15.12:0/413391965 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5d28050cc0 con 0x7f5d3009d030 2024-09-11T01:45:42.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.853+0000 7f5d2effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:42.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:42.853+0000 7f5d2effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:45:43.065 INFO:tasks.rados.rados.0.smithi071.stdout:2513: done (0 left) 2024-09-11T01:45:43.065 INFO:tasks.rados.rados.0.smithi071.stdout:2514: snap_create 2024-09-11T01:45:43.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.117+0000 7f5d384ef640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:45:43.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.117+0000 7f5d384ef640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:45:43.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.117+0000 7f5d384ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:43.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.117+0000 7f5d384ef640 1 -- 172.21.15.12:0/413391965 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5d300698e0 con 0x7f5d3009d030 2024-09-11T01:45:43.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.118+0000 7f5d2effd640 1 -- 172.21.15.12:0/413391965 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f5d28052080 con 0x7f5d3009d030 2024-09-11T01:45:43.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.118+0000 7f5d2effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:45:43.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.118+0000 7f5d2effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:45:43.120 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":154,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:45:43.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.120+0000 7f5d384ef640 1 -- 172.21.15.12:0/413391965 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5d1c03e210 msgr2=0x7f5d1c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:43.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.120+0000 7f5d384ef640 1 --2- 172.21.15.12:0/413391965 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5d1c03e210 0x7f5d1c0406d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5d3013ebb0 tx=0x7f5d24008040 comp rx=0 tx=0).stop 2024-09-11T01:45:43.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.120+0000 7f5d384ef640 10 monclient: shutdown 2024-09-11T01:45:43.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.120+0000 7f5d384ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:43.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.120+0000 7f5d384ef640 1 -- 172.21.15.12:0/413391965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d3009d030 msgr2=0x7f5d3013ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:43.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.120+0000 7f5d384ef640 1 --2- 172.21.15.12:0/413391965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d3009d030 0x7f5d3013ec20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5d2800c490 tx=0x7f5d2800c950 comp rx=0 tx=0).stop 2024-09-11T01:45:43.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.120+0000 7f5d384ef640 1 -- 172.21.15.12:0/413391965 shutdown_connections 2024-09-11T01:45:43.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.120+0000 7f5d384ef640 1 --2- 172.21.15.12:0/413391965 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5d30138230 0x7f5d301cd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:43.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.120+0000 7f5d384ef640 1 --2- 172.21.15.12:0/413391965 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d3009d460 0x7f5d301cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:43.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.120+0000 7f5d384ef640 1 --2- 172.21.15.12:0/413391965 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5d1c03e210 0x7f5d1c0406d0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:43.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.120+0000 7f5d384ef640 1 --2- 172.21.15.12:0/413391965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d3009d030 0x7f5d3013ec20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:43.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.120+0000 7f5d384ef640 1 -- 172.21.15.12:0/413391965 >> 172.21.15.12:0/413391965 conn(0x7f5d300a7120 msgr2=0x7f5d300a2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:43.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.120+0000 7f5d384ef640 1 -- 172.21.15.12:0/413391965 shutdown_connections 2024-09-11T01:45:43.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.120+0000 7f5d384ef640 1 -- 172.21.15.12:0/413391965 wait complete. 2024-09-11T01:45:43.132 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:45:43.132 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:45:43.133 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:45:43.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.235+0000 7f7780cc6640 10 monclient: get_monmap_and_config 2024-09-11T01:45:43.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.235+0000 7f7780cc6640 10 monclient: build_initial_monmap 2024-09-11T01:45:43.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.235+0000 7f7780cc6640 10 monclient: monmap: 2024-09-11T01:45:43.237 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:43.237 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:43.237 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:45:43.236052+0000 2024-09-11T01:45:43.237 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:45:43.236052+0000 2024-09-11T01:45:43.237 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:43.237 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:43.237 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:45:43.237 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:45:43.237 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:45:43.238 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:45:43.238 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:45:43.238 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:43.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.235+0000 7f7780cc6640 1 Processor -- start 2024-09-11T01:45:43.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.235+0000 7f7780cc6640 1 -- start start 2024-09-11T01:45:43.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.235+0000 7f7780cc6640 10 monclient: init 2024-09-11T01:45:43.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f7780cc6640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:43.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f7780cc6640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-11T01:45:43.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f7780cc6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f777c134ad0 0x7f777c134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:43.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f7780cc6640 10 monclient(hunting): picked mon.noname-b con 0x7f777c134ad0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:43.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f7780cc6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f777c0a2200 0x7f777c1353f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:43.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f7780cc6640 10 monclient(hunting): picked mon.noname-c con 0x7f777c0a2200 addr v2:172.21.15.71:3300/0 2024-09-11T01:45:43.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f7780cc6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f777c133a50 addr v1:172.21.15.71:6789/0 2024-09-11T01:45:43.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f7780cc6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f777c09ed30 con 0x7f777c133a50 2024-09-11T01:45:43.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f7780cc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:43.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f7780cc6640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f777c09f5d0 con 0x7f777c0a2200 2024-09-11T01:45:43.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f7780cc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:43.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f7780cc6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f777c135930 con 0x7f777c134ad0 2024-09-11T01:45:43.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f7780cc6640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:43.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f7780cc6640 10 monclient(hunting): authenticate will time out at 1749.678588s 2024-09-11T01:45:43.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f777a59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f777c134ad0 0x7f777c134eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:43.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f777a59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f777c134ad0 0x7f777c134eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42716/0 (socket says 172.21.15.12:42716) 2024-09-11T01:45:43.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f777a59c640 1 -- 172.21.15.12:0/4225381505 learned_addr learned my addr 172.21.15.12:0/4225381505 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:43.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f777a59c640 10 monclient(hunting): get_auth_request con 0x7f777c134ad0 auth_method 0 2024-09-11T01:45:43.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f777a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:43.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f777a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:43.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f777a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:43.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f777a59c640 1 -- 172.21.15.12:0/4225381505 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f777c134ad0 msgr2=0x7f777c134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:45:43.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f777a59c640 1 -- 172.21.15.12:0/4225381505 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f777c134ad0 msgr2=0x7f777c134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:45:43.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f777a59c640 1 --2- 172.21.15.12:0/4225381505 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f777c134ad0 0x7f777c134eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:45:43.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f7779d9b640 1 --2- 172.21.15.12:0/4225381505 >> v2:172.21.15.71:3300/0 conn(0x7f777c0a2200 0x7f777c1353f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:43.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f777a59c640 1 --2- 172.21.15.12:0/4225381505 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f777c134ad0 0x7f777c134eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:43.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f7779d9b640 10 monclient(hunting): get_auth_request con 0x7f777c0a2200 auth_method 0 2024-09-11T01:45:43.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f7779d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:43.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f7779d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:43.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f7779d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:43.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f7779d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:43.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f7779d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:43.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.236+0000 7f7779d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:43.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.239+0000 7f777ad9d640 1 -- 172.21.15.12:0/4225381505 >> v1:172.21.15.71:6789/0 conn(0x7f777c133a50 legacy=0x7f777c133f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:43.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.239+0000 7f7779d9b640 10 monclient(hunting): handle_auth_done global_id 134539 payload 274 2024-09-11T01:45:43.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.239+0000 7f7779d9b640 1 -- 172.21.15.12:0/4225381505 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f777c134ad0 msgr2=0x7f777c134eb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:45:43.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.239+0000 7f7779d9b640 1 --2- 172.21.15.12:0/4225381505 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f777c134ad0 0x7f777c134eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:43.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.239+0000 7f7779d9b640 1 -- 172.21.15.12:0/4225381505 >> v1:172.21.15.71:6789/0 conn(0x7f777c133a50 legacy=0x7f777c133f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:43.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.239+0000 7f7779d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:45:43.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.239+0000 7f7779d9b640 1 monclient: found mon.noname-c 2024-09-11T01:45:43.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.239+0000 7f7779d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:43.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.239+0000 7f7779d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:45:43.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.239+0000 7f7779d9b640 1 -- 172.21.15.12:0/4225381505 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f777c09f010 con 0x7f777c0a2200 2024-09-11T01:45:43.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.242+0000 7f7779d9b640 1 -- 172.21.15.12:0/4225381505 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f777c0a2200 msgr2=0x7f777c1353f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:43.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.242+0000 7f7779d9b640 1 --2- 172.21.15.12:0/4225381505 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f777c0a2200 0x7f777c1353f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7764009820 tx=0x7f776402a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3900f462f7560b63 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:43.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.242+0000 7f777959a640 1 -- 172.21.15.12:0/4225381505 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7764037070 con 0x7f777c0a2200 2024-09-11T01:45:43.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.242+0000 7f777959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:43.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.242+0000 7f777959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:45:43.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.242+0000 7f777959a640 10 monclient: dump: 2024-09-11T01:45:43.255 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:43.255 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:43.256 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.242+0000 7f777959a640 10 monclient: _finish_auth 0 2024-09-11T01:45:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.242+0000 7f777959a640 10 monclient: _check_auth_tickets 2024-09-11T01:45:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.242+0000 7f777959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.242+0000 7f777959a640 1 -- 172.21.15.12:0/4225381505 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7764032640 con 0x7f777c0a2200 2024-09-11T01:45:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.242+0000 7f777959a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:43.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.242+0000 7f777959a640 1 -- 172.21.15.12:0/4225381505 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776402ae60 con 0x7f777c0a2200 2024-09-11T01:45:43.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.242+0000 7f7780cc6640 5 monclient: authenticate success, global_id 134539 2024-09-11T01:45:43.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.242+0000 7f7780cc6640 10 monclient: get_monmap_and_config success 2024-09-11T01:45:43.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.242+0000 7f7780cc6640 4 set_mon_vals no callback set 2024-09-11T01:45:43.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.242+0000 7f777959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:43.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.242+0000 7f777959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:45:43.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.242+0000 7f777959a640 10 monclient: dump: 2024-09-11T01:45:43.257 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:43.257 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:43.257 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:43.257 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:43.257 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:43.257 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:43.257 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:43.258 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:43.258 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:43.258 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:43.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.243+0000 7f7780cc6640 10 monclient: shutdown 2024-09-11T01:45:43.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.243+0000 7f7780cc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:43.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.243+0000 7f7780cc6640 1 -- 172.21.15.12:0/4225381505 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f777c0a2200 msgr2=0x7f777c1353f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:43.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.243+0000 7f7780cc6640 1 --2- 172.21.15.12:0/4225381505 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f777c0a2200 0x7f777c1353f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7764009820 tx=0x7f776402a100 comp rx=0 tx=0).stop 2024-09-11T01:45:43.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.243+0000 7f7780cc6640 1 -- 172.21.15.12:0/4225381505 shutdown_connections 2024-09-11T01:45:43.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.243+0000 7f7780cc6640 1 --2- 172.21.15.12:0/4225381505 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f777c0a2200 0x7f777c1353f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:43.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.243+0000 7f7780cc6640 1 --2- 172.21.15.12:0/4225381505 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f777c134ad0 0x7f777c134eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:43.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.243+0000 7f7780cc6640 1 -- 172.21.15.12:0/4225381505 >> 172.21.15.12:0/4225381505 conn(0x7f777c0a2f70 msgr2=0x7f777c0a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.243+0000 7f7780cc6640 1 -- 172.21.15.12:0/4225381505 shutdown_connections 2024-09-11T01:45:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.243+0000 7f7780cc6640 1 -- 172.21.15.12:0/4225381505 wait complete. 2024-09-11T01:45:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.244+0000 7f7780cc6640 10 monclient: build_initial_monmap 2024-09-11T01:45:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.244+0000 7f7780cc6640 10 monclient: monmap: 2024-09-11T01:45:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:45:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:45:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:45:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:45:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:45:43.259 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:45:43.259 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:43.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.244+0000 7f7780cc6640 1 Processor -- start 2024-09-11T01:45:43.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.244+0000 7f7780cc6640 1 -- start start 2024-09-11T01:45:43.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.244+0000 7f7780cc6640 10 monclient: init 2024-09-11T01:45:43.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.244+0000 7f7780cc6640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:43.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.244+0000 7f7780cc6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:45:43.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.244+0000 7f7780cc6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777c0a2200 0x7f777c1cd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:43.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.244+0000 7f7780cc6640 10 monclient(hunting): picked mon.noname-a con 0x7f777c0a2200 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:45:43.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.244+0000 7f7780cc6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f777c133a50 0x7f777c1cd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:43.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.244+0000 7f7780cc6640 10 monclient(hunting): picked mon.noname-c con 0x7f777c133a50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:43.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.244+0000 7f7780cc6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f777c134ad0 0x7f777c1235c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:43.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.244+0000 7f7780cc6640 10 monclient(hunting): picked mon.noname-b con 0x7f777c134ad0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:45:43.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.244+0000 7f7780cc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:43.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.244+0000 7f7780cc6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f777c0a0150 con 0x7f777c0a2200 2024-09-11T01:45:43.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.244+0000 7f7780cc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:43.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.244+0000 7f7780cc6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f777c09ffd0 con 0x7f777c134ad0 2024-09-11T01:45:43.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.244+0000 7f7780cc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:43.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.244+0000 7f7780cc6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f777c134610 con 0x7f777c133a50 2024-09-11T01:45:43.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.244+0000 7f7780cc6640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:43.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.244+0000 7f7780cc6640 10 monclient(hunting): authenticate will time out at 1749.686906s 2024-09-11T01:45:43.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.244+0000 7f777a59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777c0a2200 0x7f777c1cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:43.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.244+0000 7f7779d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f777c133a50 0x7f777c1cd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:43.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f777a59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777c0a2200 0x7f777c1cd080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59840/0 (socket says 172.21.15.12:59840) 2024-09-11T01:45:43.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f777a59c640 1 -- 172.21.15.12:0/891625632 learned_addr learned my addr 172.21.15.12:0/891625632 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:43.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f777a59c640 10 monclient(hunting): get_auth_request con 0x7f777c0a2200 auth_method 0 2024-09-11T01:45:43.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f777a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:43.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f777a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:43.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f777a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:43.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f7779d9b640 10 monclient(hunting): get_auth_request con 0x7f777c133a50 auth_method 0 2024-09-11T01:45:43.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f7779d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:43.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f7779d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:43.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f7779d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:43.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f777a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:43.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f777a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:43.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f7779d9b640 1 -- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f777c133a50 msgr2=0x7f777c1cd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:45:43.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f7779d9b640 1 -- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f777c133a50 msgr2=0x7f777c1cd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:45:43.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f7779d9b640 1 --2- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f777c133a50 0x7f777c1cd5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:45:43.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f777a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:43.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f7779d9b640 1 --2- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f777c133a50 0x7f777c1cd5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:43.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f777a59c640 10 monclient(hunting): handle_auth_done global_id 174198 payload 824 2024-09-11T01:45:43.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f777a59c640 1 -- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f777c133a50 msgr2=0x7f777c1cd5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:45:43.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f777a59c640 1 --2- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f777c133a50 0x7f777c1cd5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:43.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f777a59c640 1 -- 172.21.15.12:0/891625632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f777c134ad0 msgr2=0x7f777c1235c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:43.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f777a59c640 1 --2- 172.21.15.12:0/891625632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f777c134ad0 0x7f777c1235c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:43.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f777a59c640 10 monclient: _finish_hunting 0 2024-09-11T01:45:43.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f777a59c640 1 monclient: found mon.noname-a 2024-09-11T01:45:43.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f777a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:43.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f777a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:45:43.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.245+0000 7f777a59c640 1 -- 172.21.15.12:0/891625632 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f777c09d480 con 0x7f777c0a2200 2024-09-11T01:45:43.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.247+0000 7f777a59c640 1 -- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777c0a2200 msgr2=0x7f777c1cd080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:43.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.247+0000 7f777a59c640 1 --2- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777c0a2200 0x7f777c1cd080 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f776c00e810 tx=0x7f776c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:43.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.247+0000 7f7772ffd640 1 -- 172.21.15.12:0/891625632 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c017070 con 0x7f777c0a2200 2024-09-11T01:45:43.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.247+0000 7f7772ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:43.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.247+0000 7f7772ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:45:43.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.247+0000 7f7772ffd640 10 monclient: dump: 2024-09-11T01:45:43.265 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:43.265 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:43.265 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:43.265 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:43.265 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:43.265 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:43.265 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:43.265 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:43.265 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:43.265 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:43.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.247+0000 7f7772ffd640 10 monclient: _finish_auth 0 2024-09-11T01:45:43.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.247+0000 7f7772ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:45:43.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.247+0000 7f7772ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:43.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.247+0000 7f7772ffd640 1 -- 172.21.15.12:0/891625632 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f776c004020 con 0x7f777c0a2200 2024-09-11T01:45:43.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.247+0000 7f7780cc6640 5 monclient: authenticate success, global_id 174198 2024-09-11T01:45:43.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.247+0000 7f7772ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:43.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.247+0000 7f7772ffd640 1 -- 172.21.15.12:0/891625632 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c00f040 con 0x7f777c0a2200 2024-09-11T01:45:43.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.247+0000 7f7772ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:43.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.247+0000 7f7772ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:43.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.247+0000 7f7772ffd640 10 monclient: dump: 2024-09-11T01:45:43.266 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:43.266 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:43.266 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:43.266 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:43.266 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:43.267 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:43.267 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:43.267 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:43.267 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:43.267 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:43.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.247+0000 7f7780cc6640 10 monclient: _renew_subs 2024-09-11T01:45:43.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.247+0000 7f7780cc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:43.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.247+0000 7f7778d99640 4 set_mon_vals no callback set 2024-09-11T01:45:43.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.247+0000 7f7780cc6640 1 -- 172.21.15.12:0/891625632 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f777c125080 con 0x7f777c0a2200 2024-09-11T01:45:43.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.247+0000 7f7780cc6640 10 monclient: _renew_subs 2024-09-11T01:45:43.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.247+0000 7f7780cc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:43.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.247+0000 7f7780cc6640 1 -- 172.21.15.12:0/891625632 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f777c127dc0 con 0x7f777c0a2200 2024-09-11T01:45:43.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.248+0000 7f7780cc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:43.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.248+0000 7f7780cc6640 1 --2- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777c125be0 0x7f777c123b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:43.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.248+0000 7f7780cc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:43.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.248+0000 7f7780cc6640 1 -- 172.21.15.12:0/891625632 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f777c133f00 con 0x7f777c125be0 2024-09-11T01:45:43.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.248+0000 7f7779d9b640 1 --2- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777c125be0 0x7f777c123b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:43.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.248+0000 7f7779d9b640 10 monclient: get_auth_request con 0x7f777c125be0 auth_method 0 2024-09-11T01:45:43.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.248+0000 7f7779d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:43.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.248+0000 7f7779d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:43.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.248+0000 7f7779d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:43.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.248+0000 7f7779d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:43.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.248+0000 7f7779d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:43.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.248+0000 7f7772ffd640 1 -- 172.21.15.12:0/891625632 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f776c012050 con 0x7f777c0a2200 2024-09-11T01:45:43.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.248+0000 7f7779d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:43.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.248+0000 7f7779d9b640 10 monclient(hunting): handle_auth_done global_id 174201 payload 824 2024-09-11T01:45:43.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.248+0000 7f7779d9b640 1 -- 172.21.15.12:0/891625632 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f777c0008d0 con 0x7f777c125be0 2024-09-11T01:45:43.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.249+0000 7f7772ffd640 1 --2- 172.21.15.12:0/891625632 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f775c03e210 0x7f775c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:43.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.249+0000 7f7772ffd640 1 -- 172.21.15.12:0/891625632 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(562..562 src has 317..562) ==== 4815+0+0 (secure 0 0 0) 0x7f776c052740 con 0x7f777c0a2200 2024-09-11T01:45:43.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.249+0000 7f777ad9d640 1 --2- 172.21.15.12:0/891625632 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f775c03e210 0x7f775c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:43.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.249+0000 7f777ad9d640 10 monclient: get_auth_request con 0x7f775c03e210 auth_method 0 2024-09-11T01:45:43.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.251+0000 7f7779d9b640 1 -- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777c125be0 msgr2=0x7f777c123b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:43.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.251+0000 7f7779d9b640 1 --2- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777c125be0 0x7f777c123b00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7764034000 tx=0x7f77640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:43.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.251+0000 7f7772ffd640 1 -- 172.21.15.12:0/891625632 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7764037040 con 0x7f777c125be0 2024-09-11T01:45:43.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.251+0000 7f7772ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:43.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.251+0000 7f7772ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:43.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.251+0000 7f7772ffd640 10 monclient: dump: 2024-09-11T01:45:43.270 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:43.270 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:43.270 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:43.270 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:43.270 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:43.270 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:43.270 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:43.272 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:43.273 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:43.273 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:43.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.251+0000 7f7772ffd640 1 -- 172.21.15.12:0/891625632 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f776403c870 con 0x7f777c125be0 2024-09-11T01:45:43.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.251+0000 7f7772ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:43.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.251+0000 7f7772ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:45:43.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.251+0000 7f7772ffd640 1 -- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777c125be0 msgr2=0x7f777c123b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:43.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.251+0000 7f7772ffd640 1 --2- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777c125be0 0x7f777c123b00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7764034000 tx=0x7f77640046c0 comp rx=0 tx=0).stop 2024-09-11T01:45:43.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.251+0000 7f7772ffd640 1 -- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777c125be0 msgr2=0x7f777c123b00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:43.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.251+0000 7f7772ffd640 1 --2- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777c125be0 0x7f777c123b00 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7764034000 tx=0x7f77640046c0 comp rx=0 tx=0).stop 2024-09-11T01:45:43.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.252+0000 7f777ad9d640 1 --2- 172.21.15.12:0/891625632 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f775c03e210 0x7f775c0406d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f777c125560 tx=0x7f7774007cf0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:43.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.257+0000 7f7780cc6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:45:43.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.257+0000 7f7780cc6640 1 --2- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777c124040 0x7f777c124450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.257+0000 7f7780cc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.257+0000 7f7780cc6640 1 -- 172.21.15.12:0/891625632 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f777c124990 con 0x7f777c124040 2024-09-11T01:45:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.257+0000 7f7779d9b640 1 --2- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777c124040 0x7f777c124450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.258+0000 7f7779d9b640 10 monclient: get_auth_request con 0x7f777c124040 auth_method 0 2024-09-11T01:45:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.258+0000 7f7779d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.258+0000 7f7779d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.258+0000 7f7779d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.258+0000 7f7779d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.258+0000 7f7779d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.258+0000 7f7779d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.258+0000 7f7779d9b640 10 monclient(hunting): handle_auth_done global_id 174204 payload 824 2024-09-11T01:45:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.258+0000 7f7779d9b640 1 -- 172.21.15.12:0/891625632 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f777c1cdb00 con 0x7f777c124040 2024-09-11T01:45:43.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.260+0000 7f7779d9b640 1 -- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777c124040 msgr2=0x7f777c124450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:43.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.260+0000 7f7779d9b640 1 --2- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777c124040 0x7f777c124450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7764009820 tx=0x7f7764008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:43.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.260+0000 7f7772ffd640 1 -- 172.21.15.12:0/891625632 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7764037050 con 0x7f777c124040 2024-09-11T01:45:43.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.260+0000 7f7772ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:43.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.260+0000 7f7772ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:43.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.260+0000 7f7772ffd640 10 monclient: dump: 2024-09-11T01:45:43.277 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:43.277 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:43.277 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:43.277 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:43.277 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:43.277 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:43.277 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:43.277 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:43.278 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:43.278 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:43.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.260+0000 7f7772ffd640 1 -- 172.21.15.12:0/891625632 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4710 (secure 0 0 0) 0x7f7764047c60 con 0x7f777c124040 2024-09-11T01:45:43.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.260+0000 7f7772ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:45:43.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.260+0000 7f7772ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:45:43.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.260+0000 7f7772ffd640 1 -- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777c124040 msgr2=0x7f777c124450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:43.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.260+0000 7f7772ffd640 1 --2- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777c124040 0x7f777c124450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7764009820 tx=0x7f7764008030 comp rx=0 tx=0).stop 2024-09-11T01:45:43.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.260+0000 7f7772ffd640 1 -- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777c124040 msgr2=0x7f777c124450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:43.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.260+0000 7f7772ffd640 1 --2- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777c124040 0x7f777c124450 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7764009820 tx=0x7f7764008030 comp rx=0 tx=0).stop 2024-09-11T01:45:43.278 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:45:43.279 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:45:43.279 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:45:43.279 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:45:43.279 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 154, 2024-09-11T01:45:43.279 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 59330, 2024-09-11T01:45:43.279 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:45:43.279 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:45:43.279 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:45:43.279 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:45:43.279 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:43.279 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:45:43.279 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:45:43.279 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:45:43.280 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:45:43.280 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:43.280 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:43.280 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:43.280 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:43.280 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:43.280 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:43.280 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:43.280 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:43.280 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:43.280 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:43.280 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:43.280 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:43.281 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:45:43.281 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:45:43.281 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:43.281 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:43.281 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:43.281 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:43.281 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:43.281 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:43.281 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:43.281 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:43.281 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:43.281 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:43.282 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:43.282 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:43.282 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:43.282 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:45:43.282 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:45:43.282 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:45:43.282 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:45:43.282 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:45:43.282 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:45:43.282 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:45:43.282 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:45:43.282 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:45:43.282 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:45:43.282 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:45:43.283 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:45:43.283 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:45:43.283 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:45:43.283 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:45:43.283 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:45:43.283 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:45:43.283 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:43.283 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:43.283 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:43.283 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:43.283 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:43.283 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:43.283 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:43.283 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:43.284 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:43.284 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:43.284 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:43.284 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:43.284 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:45:43.284 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:43.285 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:45:43.285 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:43.285 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:45:43.285 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:45:43.285 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:43.285 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:43.286 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:43.286 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:43.286 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:45:43.286 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:43.286 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:43.286 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:43.287 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:43.287 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:45:43.287 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:43.287 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:43.287 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:43.287 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:43.288 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:45:43.288 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:45:43.288 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:43.288 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:43.288 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:43.288 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:43.288 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:43.288 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:45:43.288 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:45:43.289 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:43.289 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:43.289 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:43.289 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:43.289 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:45:43.289 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:43.289 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:43.289 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:43.289 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:43.289 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:45:43.289 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:43.289 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:43.289 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:43.290 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:43.290 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:45:43.290 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:45:43.290 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:43.290 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:43.290 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:43.290 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:43.290 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:43.290 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:45:43.290 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:45:43.290 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:43.290 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:43.291 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:43.291 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:43.291 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:45:43.291 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:43.291 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:43.291 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:43.291 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:43.291 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:45:43.291 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:43.291 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:43.291 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:43.291 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:43.291 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:45:43.291 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:45:43.292 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:43.292 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:43.292 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:43.292 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:43.292 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:43.292 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:43.292 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:45:43.292 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:45:43.292 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:43.292 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:43.292 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:43.292 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:45:43.292 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:43.292 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:43.293 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:45:43.293 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:43.293 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:43.293 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:43.293 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:45:43.293 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:43.293 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:43.293 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:45:43.293 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:43.293 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:43.293 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:43.293 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:45:43.293 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:43.293 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:43.294 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:43.294 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:45:43.294 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:45:43.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.261+0000 7f7780cc6640 1 -- 172.21.15.12:0/891625632 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f775c03e210 msgr2=0x7f775c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:43.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.261+0000 7f7780cc6640 1 --2- 172.21.15.12:0/891625632 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f775c03e210 0x7f775c0406d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f777c125560 tx=0x7f7774007cf0 comp rx=0 tx=0).stop 2024-09-11T01:45:43.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.261+0000 7f7780cc6640 10 monclient: shutdown 2024-09-11T01:45:43.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.261+0000 7f7780cc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:43.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.261+0000 7f7780cc6640 1 -- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777c0a2200 msgr2=0x7f777c1cd080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:43.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.261+0000 7f7780cc6640 1 --2- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777c0a2200 0x7f777c1cd080 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f776c00e810 tx=0x7f776c00ecd0 comp rx=0 tx=0).stop 2024-09-11T01:45:43.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.261+0000 7f777ad9d640 1 -- 172.21.15.12:0/891625632 reap_dead start 2024-09-11T01:45:43.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.261+0000 7f7780cc6640 1 -- 172.21.15.12:0/891625632 shutdown_connections 2024-09-11T01:45:43.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.261+0000 7f7780cc6640 1 --2- 172.21.15.12:0/891625632 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f777c0a2200 0x7f777c1cd080 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:43.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.261+0000 7f7780cc6640 1 -- 172.21.15.12:0/891625632 >> 172.21.15.12:0/891625632 conn(0x7f777c0a2f70 msgr2=0x7f777c0a4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:43.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.261+0000 7f7780cc6640 1 -- 172.21.15.12:0/891625632 shutdown_connections 2024-09-11T01:45:43.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.261+0000 7f7780cc6640 1 -- 172.21.15.12:0/891625632 wait complete. 2024-09-11T01:45:43.296 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:45:43.296 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:45:43.297 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:45:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.397+0000 7f2be60dd640 10 monclient: get_monmap_and_config 2024-09-11T01:45:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.397+0000 7f2be60dd640 10 monclient: build_initial_monmap 2024-09-11T01:45:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.397+0000 7f2be60dd640 10 monclient: monmap: 2024-09-11T01:45:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:45:43.398047+0000 2024-09-11T01:45:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:45:43.398047+0000 2024-09-11T01:45:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:45:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:45:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:45:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:45:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:45:43.401 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2be60dd640 1 Processor -- start 2024-09-11T01:45:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2be60dd640 1 -- start start 2024-09-11T01:45:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2be60dd640 10 monclient: init 2024-09-11T01:45:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2be60dd640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2be60dd640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-11T01:45:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2be60dd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2be00a2db0 0x7f2be006f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2be60dd640 10 monclient(hunting): picked mon.noname-c con 0x7f2be00a2db0 addr v2:172.21.15.71:3300/0 2024-09-11T01:45:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2be60dd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2be006ded0 0x7f2be00a3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2be60dd640 10 monclient(hunting): picked mon.noname-a con 0x7f2be006ded0 addr v2:172.21.15.12:3300/0 2024-09-11T01:45:43.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2be60dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2be00a36d0 addr v1:172.21.15.71:6789/0 2024-09-11T01:45:43.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2be60dd640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be00a4650 con 0x7f2be00a36d0 2024-09-11T01:45:43.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2be60dd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:43.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2be60dd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f2be006f1a0 con 0x7f2be006ded0 2024-09-11T01:45:43.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2be60dd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:43.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2be60dd640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f2be006eec0 con 0x7f2be00a2db0 2024-09-11T01:45:43.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2be60dd640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:43.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2be60dd640 10 monclient(hunting): authenticate will time out at 1749.840682s 2024-09-11T01:45:43.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2bdeffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2be006ded0 0x7f2be00a3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:43.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2bdeffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2be006ded0 0x7f2be00a3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59866/0 (socket says 172.21.15.12:59866) 2024-09-11T01:45:43.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2bdeffd640 1 -- 172.21.15.12:0/1495485054 learned_addr learned my addr 172.21.15.12:0/1495485054 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:43.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2bdeffd640 10 monclient(hunting): get_auth_request con 0x7f2be006ded0 auth_method 0 2024-09-11T01:45:43.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2bdeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:43.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2bdeffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:43.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2bdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2bdeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2bdeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.398+0000 7f2bdeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.399+0000 7f2bdf7fe640 1 --2- 172.21.15.12:0/1495485054 >> v2:172.21.15.71:3300/0 conn(0x7f2be00a2db0 0x7f2be006f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.399+0000 7f2bdeffd640 10 monclient(hunting): handle_auth_done global_id 174207 payload 274 2024-09-11T01:45:43.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.399+0000 7f2bdeffd640 1 -- 172.21.15.12:0/1495485054 >> v2:172.21.15.71:3300/0 conn(0x7f2be00a2db0 msgr2=0x7f2be006f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:43.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.399+0000 7f2bdeffd640 1 --2- 172.21.15.12:0/1495485054 >> v2:172.21.15.71:3300/0 conn(0x7f2be00a2db0 0x7f2be006f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:43.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.399+0000 7f2bdeffd640 1 -- 172.21.15.12:0/1495485054 >> v1:172.21.15.71:6789/0 conn(0x7f2be00a36d0 legacy=0x7f2be0137f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:43.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.399+0000 7f2bdeffd640 10 monclient: _finish_hunting 0 2024-09-11T01:45:43.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.399+0000 7f2bdeffd640 1 monclient: found mon.noname-a 2024-09-11T01:45:43.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.399+0000 7f2bdeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.399+0000 7f2bdeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:45:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.399+0000 7f2bdeffd640 1 -- 172.21.15.12:0/1495485054 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be006e2e0 con 0x7f2be006ded0 2024-09-11T01:45:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.399+0000 7f2bdf7fe640 10 monclient: get_auth_request con 0x7f2be00a2db0 auth_method 0 2024-09-11T01:45:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.399+0000 7f2bdf7fe640 1 --2- 172.21.15.12:0/1495485054 >> v2:172.21.15.71:3300/0 conn(0x7f2be00a2db0 0x7f2be006f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:45:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2bdeffd640 1 -- 172.21.15.12:0/1495485054 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2be006ded0 msgr2=0x7f2be00a3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:43.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2bdeffd640 1 --2- 172.21.15.12:0/1495485054 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2be006ded0 0x7f2be00a3190 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2be0070b40 tx=0x7f2bd402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dae19da6b8f0b3a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:43.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2bde7fc640 1 -- 172.21.15.12:0/1495485054 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4037070 con 0x7f2be006ded0 2024-09-11T01:45:43.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2bde7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:43.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2bde7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:45:43.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2bde7fc640 10 monclient: dump: 2024-09-11T01:45:43.426 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:43.427 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2bde7fc640 10 monclient: _finish_auth 0 2024-09-11T01:45:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2bde7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:45:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2bde7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2bde7fc640 1 -- 172.21.15.12:0/1495485054 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bd4032640 con 0x7f2be006ded0 2024-09-11T01:45:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2be60dd640 5 monclient: authenticate success, global_id 174207 2024-09-11T01:45:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2be60dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:45:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2bde7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2be60dd640 10 monclient: get_monmap_and_config success 2024-09-11T01:45:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2bde7fc640 1 -- 172.21.15.12:0/1495485054 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4032a10 con 0x7f2be006ded0 2024-09-11T01:45:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2be60dd640 4 set_mon_vals no callback set 2024-09-11T01:45:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2bde7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2bde7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2bde7fc640 10 monclient: dump: 2024-09-11T01:45:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:43.429 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2be60dd640 10 monclient: shutdown 2024-09-11T01:45:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2be60dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2be60dd640 1 -- 172.21.15.12:0/1495485054 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2be006ded0 msgr2=0x7f2be00a3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2be60dd640 1 --2- 172.21.15.12:0/1495485054 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2be006ded0 0x7f2be00a3190 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2be0070b40 tx=0x7f2bd402a140 comp rx=0 tx=0).stop 2024-09-11T01:45:43.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2be60dd640 1 -- 172.21.15.12:0/1495485054 shutdown_connections 2024-09-11T01:45:43.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2be60dd640 1 --2- 172.21.15.12:0/1495485054 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2be006ded0 0x7f2be00a3190 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:43.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2be60dd640 1 --2- 172.21.15.12:0/1495485054 >> v2:172.21.15.71:3300/0 conn(0x7f2be00a2db0 0x7f2be006f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:43.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2be60dd640 1 -- 172.21.15.12:0/1495485054 >> 172.21.15.12:0/1495485054 conn(0x7f2be00a83b0 msgr2=0x7f2be00a87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:43.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2be60dd640 1 -- 172.21.15.12:0/1495485054 shutdown_connections 2024-09-11T01:45:43.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.402+0000 7f2be60dd640 1 -- 172.21.15.12:0/1495485054 wait complete. 2024-09-11T01:45:43.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.403+0000 7f2be60dd640 10 monclient: build_initial_monmap 2024-09-11T01:45:43.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.403+0000 7f2be60dd640 10 monclient: monmap: 2024-09-11T01:45:43.430 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:43.430 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:45:43.430 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:45:43.430 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:45:43.430 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:43.430 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:45:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:45:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:45:43.431 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.403+0000 7f2be60dd640 1 Processor -- start 2024-09-11T01:45:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.403+0000 7f2be60dd640 1 -- start start 2024-09-11T01:45:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.403+0000 7f2be60dd640 10 monclient: init 2024-09-11T01:45:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2be60dd640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2be60dd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:45:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2be60dd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2be006ded0 0x7f2be01cd400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2be60dd640 10 monclient(hunting): picked mon.noname-b con 0x7f2be006ded0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:45:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2be60dd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be00a2db0 0x7f2be01cd940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2be60dd640 10 monclient(hunting): picked mon.noname-c con 0x7f2be00a2db0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2be60dd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2be00a36d0 0x7f2be01247d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2be60dd640 10 monclient(hunting): picked mon.noname-a con 0x7f2be00a36d0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:45:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2be60dd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2be60dd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2be01387f0 con 0x7f2be00a36d0 2024-09-11T01:45:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2be60dd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2be60dd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f2be0138670 con 0x7f2be006ded0 2024-09-11T01:45:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2be60dd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2be60dd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2be0138970 con 0x7f2be00a2db0 2024-09-11T01:45:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2be60dd640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2be60dd640 10 monclient(hunting): authenticate will time out at 1749.846412s 2024-09-11T01:45:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2be00a36d0 0x7f2be01247d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdeffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be00a2db0 0x7f2be01cd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2be00a36d0 0x7f2be01247d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59870/0 (socket says 172.21.15.12:59870) 2024-09-11T01:45:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdeffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be00a2db0 0x7f2be01cd940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42746/0 (socket says 172.21.15.12:42746) 2024-09-11T01:45:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdffff640 1 -- 172.21.15.12:0/1045936544 learned_addr learned my addr 172.21.15.12:0/1045936544 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdffff640 10 monclient(hunting): get_auth_request con 0x7f2be00a36d0 auth_method 0 2024-09-11T01:45:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdeffd640 10 monclient(hunting): get_auth_request con 0x7f2be00a2db0 auth_method 0 2024-09-11T01:45:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdeffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdf7fe640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2be006ded0 0x7f2be01cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdeffd640 1 -- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be00a2db0 msgr2=0x7f2be01cd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:45:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdeffd640 1 -- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be00a2db0 msgr2=0x7f2be01cd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:45:43.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdeffd640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be00a2db0 0x7f2be01cd940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:45:43.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:43.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:43.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:43.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdf7fe640 10 monclient(hunting): get_auth_request con 0x7f2be006ded0 auth_method 0 2024-09-11T01:45:43.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:43.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:43.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:43.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdeffd640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be00a2db0 0x7f2be01cd940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:43.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:43.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:43.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:43.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdffff640 10 monclient(hunting): handle_auth_done global_id 174210 payload 824 2024-09-11T01:45:43.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdffff640 1 -- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be00a2db0 msgr2=0x7f2be01cd940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:45:43.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdffff640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be00a2db0 0x7f2be01cd940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:43.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdffff640 1 -- 172.21.15.12:0/1045936544 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2be006ded0 msgr2=0x7f2be01cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:43.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdffff640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2be006ded0 0x7f2be01cd400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:43.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdffff640 10 monclient: _finish_hunting 0 2024-09-11T01:45:43.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdffff640 1 monclient: found mon.noname-a 2024-09-11T01:45:43.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:43.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:45:43.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.404+0000 7f2bdffff640 1 -- 172.21.15.12:0/1045936544 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be0137bb0 con 0x7f2be00a36d0 2024-09-11T01:45:43.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.407+0000 7f2bdffff640 1 -- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2be00a36d0 msgr2=0x7f2be01247d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:43.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.407+0000 7f2bdffff640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2be00a36d0 0x7f2be01247d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2bd800c780 tx=0x7f2bd800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:43.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.407+0000 7f2bc7fff640 1 -- 172.21.15.12:0/1045936544 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd800ac60 con 0x7f2be00a36d0 2024-09-11T01:45:43.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.407+0000 7f2bc7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:43.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.407+0000 7f2bc7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:45:43.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.407+0000 7f2bc7fff640 10 monclient: dump: 2024-09-11T01:45:43.436 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:43.436 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:43.436 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:43.436 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:43.436 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:43.436 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:43.436 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:43.436 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:43.436 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:43.436 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:43.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.407+0000 7f2bc7fff640 10 monclient: _finish_auth 0 2024-09-11T01:45:43.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.407+0000 7f2bc7fff640 10 monclient: _check_auth_tickets 2024-09-11T01:45:43.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.407+0000 7f2bc7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:43.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.407+0000 7f2be60dd640 5 monclient: authenticate success, global_id 174210 2024-09-11T01:45:43.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.407+0000 7f2be60dd640 10 monclient: _renew_subs 2024-09-11T01:45:43.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.407+0000 7f2bc7fff640 1 -- 172.21.15.12:0/1045936544 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bd800ae00 con 0x7f2be00a36d0 2024-09-11T01:45:43.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.407+0000 7f2be60dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:43.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.407+0000 7f2be60dd640 1 -- 172.21.15.12:0/1045936544 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2be01285f0 con 0x7f2be00a36d0 2024-09-11T01:45:43.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.407+0000 7f2bc7fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:43.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.407+0000 7f2bc7fff640 1 -- 172.21.15.12:0/1045936544 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd8004ba0 con 0x7f2be00a36d0 2024-09-11T01:45:43.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.407+0000 7f2bc7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:43.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.407+0000 7f2bc7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:43.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.407+0000 7f2bc7fff640 10 monclient: dump: 2024-09-11T01:45:43.437 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:43.437 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:43.438 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:43.438 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:43.438 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:43.438 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:43.438 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:43.438 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:43.438 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:43.438 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:43.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.407+0000 7f2be60dd640 10 monclient: _renew_subs 2024-09-11T01:45:43.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.407+0000 7f2be60dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:43.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.407+0000 7f2be60dd640 1 -- 172.21.15.12:0/1045936544 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2be00a5620 con 0x7f2be00a36d0 2024-09-11T01:45:43.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.407+0000 7f2bddffb640 4 set_mon_vals no callback set 2024-09-11T01:45:43.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.408+0000 7f2be60dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:43.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.408+0000 7f2be60dd640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 0x7f2be002d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:43.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.408+0000 7f2be60dd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:43.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.408+0000 7f2be60dd640 1 -- 172.21.15.12:0/1045936544 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2be006a060 con 0x7f2be01254d0 2024-09-11T01:45:43.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.408+0000 7f2bdf7fe640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 0x7f2be002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:43.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.408+0000 7f2bc7fff640 1 -- 172.21.15.12:0/1045936544 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f2bd8005030 con 0x7f2be00a36d0 2024-09-11T01:45:43.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.408+0000 7f2bdf7fe640 10 monclient: get_auth_request con 0x7f2be01254d0 auth_method 0 2024-09-11T01:45:43.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.408+0000 7f2bdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:43.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.408+0000 7f2bdf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:43.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.408+0000 7f2bdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:43.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.408+0000 7f2bdf7fe640 1 -- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 msgr2=0x7f2be002d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:45:43.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.408+0000 7f2bdf7fe640 1 -- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 msgr2=0x7f2be002d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:45:43.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.408+0000 7f2bdf7fe640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 0x7f2be002d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:45:43.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.408+0000 7f2bdf7fe640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 0x7f2be002d440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:43.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.408+0000 7f2bc7fff640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2bcc03e210 0x7f2bcc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:43.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.408+0000 7f2bc7fff640 1 -- 172.21.15.12:0/1045936544 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(562..562 src has 317..562) ==== 4815+0+0 (secure 0 0 0) 0x7f2bd8052fb0 con 0x7f2be00a36d0 2024-09-11T01:45:43.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.409+0000 7f2bdeffd640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2bcc03e210 0x7f2bcc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:43.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.409+0000 7f2bdeffd640 10 monclient: get_auth_request con 0x7f2bcc03e210 auth_method 0 2024-09-11T01:45:43.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.411+0000 7f2bdeffd640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2bcc03e210 0x7f2bcc0406d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2be006fde0 tx=0x7f2bd402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:43.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.609+0000 7f2bdf7fe640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 0x7f2be002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.609+0000 7f2bdf7fe640 10 monclient: get_auth_request con 0x7f2be01254d0 auth_method 0 2024-09-11T01:45:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.609+0000 7f2bdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.609+0000 7f2bdf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.609+0000 7f2bdf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:45:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.609+0000 7f2bdf7fe640 1 -- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 msgr2=0x7f2be002d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:45:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.609+0000 7f2bdf7fe640 1 -- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 msgr2=0x7f2be002d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:45:43.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.609+0000 7f2bdf7fe640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 0x7f2be002d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:45:43.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:43.609+0000 7f2bdf7fe640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 0x7f2be002d440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:45:43.953 INFO:tasks.rados.rados.0.smithi071.stdout:2514: done (0 left) 2024-09-11T01:45:43.953 INFO:tasks.rados.rados.0.smithi071.stdout:2515: read oid 48 snap -1 2024-09-11T01:45:43.953 INFO:tasks.rados.rados.0.smithi071.stdout:2515: expect deleted 2024-09-11T01:45:43.953 INFO:tasks.rados.rados.0.smithi071.stdout:2516: snap_remove snap 266 2024-09-11T01:45:44.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.010+0000 7f2bdf7fe640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 0x7f2be002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:44.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.010+0000 7f2bdf7fe640 10 monclient: get_auth_request con 0x7f2be01254d0 auth_method 0 2024-09-11T01:45:44.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.010+0000 7f2bdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:44.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.010+0000 7f2bdf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:44.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.010+0000 7f2bdf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:45:44.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.010+0000 7f2bdf7fe640 1 -- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 msgr2=0x7f2be002d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:45:44.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.010+0000 7f2bdf7fe640 1 -- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 msgr2=0x7f2be002d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:45:44.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.010+0000 7f2bdf7fe640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 0x7f2be002d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:45:44.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.010+0000 7f2bdf7fe640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 0x7f2be002d440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:45:44.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.811+0000 7f2bdf7fe640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 0x7f2be002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:44.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.811+0000 7f2bdf7fe640 10 monclient: get_auth_request con 0x7f2be01254d0 auth_method 0 2024-09-11T01:45:44.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.811+0000 7f2bdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:44.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.811+0000 7f2bdf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:44.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.811+0000 7f2bdf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:45:44.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.811+0000 7f2bdf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:44.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.811+0000 7f2bdf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:44.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.811+0000 7f2bdf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:44.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.811+0000 7f2bdf7fe640 10 monclient(hunting): handle_auth_done global_id 184100 payload 824 2024-09-11T01:45:44.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.811+0000 7f2bdf7fe640 1 -- 172.21.15.12:0/1045936544 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2be0138cb0 con 0x7f2be01254d0 2024-09-11T01:45:44.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.814+0000 7f2bdf7fe640 1 -- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 msgr2=0x7f2be002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:44.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.814+0000 7f2bdf7fe640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 0x7f2be002d440 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0016ab0 tx=0x7f2bd000af80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:44.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.814+0000 7f2bdf7fe640 1 -- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 msgr2=0x7f2be002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:45:44.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.814+0000 7f2bdf7fe640 1 -- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 msgr2=0x7f2be002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:45:44.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.814+0000 7f2bdf7fe640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 0x7f2be002d440 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0016ab0 tx=0x7f2bd000af80 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:45:44.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.814+0000 7f2bdf7fe640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 0x7f2be002d440 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0016ab0 tx=0x7f2bd000af80 comp rx=0 tx=0).stop 2024-09-11T01:45:44.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.814+0000 7f2bc7fff640 1 -- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 msgr2=0x7f2be002d440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:44.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.814+0000 7f2bc7fff640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 0x7f2be002d440 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:44.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.814+0000 7f2bc7fff640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2bcc044290 0x7f2bcc046680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:44.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.814+0000 7f2bc7fff640 1 -- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 msgr2=0x7f2be002d440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:44.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.814+0000 7f2bc7fff640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01254d0 0x7f2be002d440 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:44.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.814+0000 7f2bc7fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:44.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.814+0000 7f2bc7fff640 1 -- 172.21.15.12:0/1045936544 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2bcc046cc0 con 0x7f2bcc044290 2024-09-11T01:45:44.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.814+0000 7f2bdf7fe640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2bcc044290 0x7f2bcc046680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:44.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.814+0000 7f2bdf7fe640 10 monclient: get_auth_request con 0x7f2bcc044290 auth_method 0 2024-09-11T01:45:44.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.814+0000 7f2bdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:44.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.814+0000 7f2bdf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:44.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.814+0000 7f2bdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:44.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.814+0000 7f2bdf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:44.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.814+0000 7f2bdf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:44.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.814+0000 7f2bdf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:44.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.814+0000 7f2bdf7fe640 10 monclient(hunting): handle_auth_done global_id 184103 payload 824 2024-09-11T01:45:44.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.814+0000 7f2bdf7fe640 1 -- 172.21.15.12:0/1045936544 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2bcc046e70 con 0x7f2bcc044290 2024-09-11T01:45:44.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.817+0000 7f2bdf7fe640 1 -- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2bcc044290 msgr2=0x7f2bcc046680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.817+0000 7f2bdf7fe640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2bcc044290 0x7f2bcc046680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0020000 tx=0x7f2bd0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.817+0000 7f2bc7fff640 1 -- 172.21.15.12:0/1045936544 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd002f020 con 0x7f2bcc044290 2024-09-11T01:45:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.817+0000 7f2bc7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.817+0000 7f2bc7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:45:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.817+0000 7f2bc7fff640 10 monclient: dump: 2024-09-11T01:45:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:44.819 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:44.820 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:44.820 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:44.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.817+0000 7f2bc7fff640 1 -- 172.21.15.12:0/1045936544 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2bd0017440 con 0x7f2bcc044290 2024-09-11T01:45:44.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.817+0000 7f2bc7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:44.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.817+0000 7f2bc7fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:45:44.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.817+0000 7f2bc7fff640 1 -- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2bcc044290 msgr2=0x7f2bcc046680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:44.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.817+0000 7f2bc7fff640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2bcc044290 0x7f2bcc046680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0020000 tx=0x7f2bd0004730 comp rx=0 tx=0).stop 2024-09-11T01:45:44.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.817+0000 7f2bc7fff640 1 -- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2bcc044290 msgr2=0x7f2bcc046680 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:44.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.817+0000 7f2bc7fff640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2bcc044290 0x7f2bcc046680 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0020000 tx=0x7f2bd0004730 comp rx=0 tx=0).stop 2024-09-11T01:45:44.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.824+0000 7f2be60dd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:45:44.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.824+0000 7f2be60dd640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01258b0 0x7f2be002d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:44.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.824+0000 7f2be60dd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:44.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.824+0000 7f2be60dd640 1 -- 172.21.15.12:0/1045936544 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2be009d9c0 con 0x7f2be01258b0 2024-09-11T01:45:44.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.824+0000 7f2bdf7fe640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01258b0 0x7f2be002d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:44.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.824+0000 7f2bdf7fe640 10 monclient: get_auth_request con 0x7f2be01258b0 auth_method 0 2024-09-11T01:45:44.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.824+0000 7f2bdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:44.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.824+0000 7f2bdf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:44.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.824+0000 7f2bdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:44.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.824+0000 7f2bdf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:44.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.824+0000 7f2bdf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:44.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.824+0000 7f2bdf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:44.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.824+0000 7f2bdf7fe640 10 monclient(hunting): handle_auth_done global_id 184106 payload 824 2024-09-11T01:45:44.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.825+0000 7f2bdf7fe640 1 -- 172.21.15.12:0/1045936544 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2be0126df0 con 0x7f2be01258b0 2024-09-11T01:45:44.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.827+0000 7f2bdf7fe640 1 -- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01258b0 msgr2=0x7f2be002d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:44.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.827+0000 7f2bdf7fe640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01258b0 0x7f2be002d980 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc046e40 tx=0x7f2bd0021000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:44.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.827+0000 7f2bc7fff640 1 -- 172.21.15.12:0/1045936544 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd002f040 con 0x7f2be01258b0 2024-09-11T01:45:44.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.827+0000 7f2bc7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:44.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.827+0000 7f2bc7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:45:44.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.827+0000 7f2bc7fff640 10 monclient: dump: 2024-09-11T01:45:44.829 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:44.829 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:44.829 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:44.830 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:44.830 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:44.830 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:44.830 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:44.830 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:44.830 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:44.830 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:44.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.827+0000 7f2bc7fff640 1 -- 172.21.15.12:0/1045936544 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4540 (secure 0 0 0) 0x7f2bd0004100 con 0x7f2be01258b0 2024-09-11T01:45:44.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.827+0000 7f2bc7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:45:44.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.827+0000 7f2bc7fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:45:44.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.827+0000 7f2bc7fff640 1 -- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01258b0 msgr2=0x7f2be002d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:44.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.827+0000 7f2bc7fff640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01258b0 0x7f2be002d980 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc046e40 tx=0x7f2bd0021000 comp rx=0 tx=0).stop 2024-09-11T01:45:44.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.827+0000 7f2bc7fff640 1 -- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01258b0 msgr2=0x7f2be002d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:44.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.827+0000 7f2bc7fff640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2be01258b0 0x7f2be002d980 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc046e40 tx=0x7f2bd0021000 comp rx=0 tx=0).stop 2024-09-11T01:45:44.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.827+0000 7f2bdffff640 1 -- 172.21.15.12:0/1045936544 reap_dead start 2024-09-11T01:45:44.831 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:45:44.831 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:45:44.831 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:45:44.831 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:45:44.831 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 154, 2024-09-11T01:45:44.831 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 5420, 2024-09-11T01:45:44.831 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:45:44.831 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:45:44.831 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:45:44.832 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:45:44.832 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:44.832 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:45:44.832 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:45:44.832 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:45:44.832 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:45:44.832 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:44.832 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:44.832 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:44.832 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:44.832 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:44.832 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:44.832 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:44.832 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:44.833 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:44.833 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:44.833 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:44.833 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:44.833 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:45:44.833 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:45:44.833 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:44.833 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:44.833 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:44.833 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:44.833 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:44.833 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:44.833 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:44.834 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:44.834 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:44.834 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:44.834 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:44.834 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:44.834 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:44.834 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:45:44.834 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:45:44.834 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:45:44.834 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:45:44.834 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:45:44.834 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:45:44.834 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:45:44.834 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:45:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:45:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:45:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:45:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:45:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:45:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:45:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:45:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:45:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:45:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:44.835 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:45:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:45:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:45:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:45:44.836 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:45:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:45:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:44.837 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:45:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:45:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:45:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:45:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:44.838 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:45:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:45:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:45:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:45:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:44.839 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:45:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:45:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:45:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:44.840 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:45:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:45:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:45:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:45:44.841 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:45:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:45:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:45:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:45:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:44.842 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:44.843 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:45:44.843 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:45:44.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.828+0000 7f2be60dd640 1 -- 172.21.15.12:0/1045936544 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2bcc03e210 msgr2=0x7f2bcc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:44.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.828+0000 7f2be60dd640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2bcc03e210 0x7f2bcc0406d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2be006fde0 tx=0x7f2bd402e000 comp rx=0 tx=0).stop 2024-09-11T01:45:44.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.828+0000 7f2be60dd640 10 monclient: shutdown 2024-09-11T01:45:44.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.828+0000 7f2be60dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:44.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.828+0000 7f2be60dd640 1 -- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2be00a36d0 msgr2=0x7f2be01247d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:44.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.828+0000 7f2be60dd640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2be00a36d0 0x7f2be01247d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2bd800c780 tx=0x7f2bd800cc40 comp rx=0 tx=0).stop 2024-09-11T01:45:44.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.828+0000 7f2be60dd640 1 -- 172.21.15.12:0/1045936544 shutdown_connections 2024-09-11T01:45:44.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.828+0000 7f2be60dd640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2bcc03e210 0x7f2bcc0406d0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:44.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.828+0000 7f2be60dd640 1 --2- 172.21.15.12:0/1045936544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2be00a36d0 0x7f2be01247d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:44.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.828+0000 7f2be60dd640 1 -- 172.21.15.12:0/1045936544 >> 172.21.15.12:0/1045936544 conn(0x7f2be00a83b0 msgr2=0x7f2be00a22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:44.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.828+0000 7f2be60dd640 1 -- 172.21.15.12:0/1045936544 shutdown_connections 2024-09-11T01:45:44.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.828+0000 7f2be60dd640 1 -- 172.21.15.12:0/1045936544 wait complete. 2024-09-11T01:45:44.845 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:45:44.845 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:45:44.845 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:45:44.954 INFO:tasks.rados.rados.0.smithi071.stdout:2516: done (1 left) 2024-09-11T01:45:44.955 INFO:tasks.rados.rados.0.smithi071.stdout:2517: rollback oid 14 current snap is 282 2024-09-11T01:45:44.955 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 14 to 274 2024-09-11T01:45:44.955 INFO:tasks.rados.rados.0.smithi071.stdout:2515: done (1 left) 2024-09-11T01:45:44.955 INFO:tasks.rados.rados.0.smithi071.stdout:2518: snap_remove snap 280 2024-09-11T01:45:44.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.959+0000 7f4ce6794640 10 monclient: get_monmap_and_config 2024-09-11T01:45:44.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.959+0000 7f4ce6794640 10 monclient: build_initial_monmap 2024-09-11T01:45:44.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.959+0000 7f4ce6794640 10 monclient: monmap: 2024-09-11T01:45:44.961 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:44.961 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:44.961 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:45:44.959852+0000 2024-09-11T01:45:44.961 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:45:44.959852+0000 2024-09-11T01:45:44.961 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:44.961 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:44.961 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:45:44.961 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:45:44.961 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:45:44.961 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:45:44.962 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:45:44.962 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:44.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.960+0000 7f4ce6794640 1 Processor -- start 2024-09-11T01:45:44.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.960+0000 7f4ce6794640 1 -- start start 2024-09-11T01:45:44.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.960+0000 7f4ce6794640 10 monclient: init 2024-09-11T01:45:44.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.961+0000 7f4ce6794640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:44.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.961+0000 7f4ce6794640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-11T01:45:44.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.961+0000 7f4ce6794640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4ce0137e50 0x7f4ce009d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:44.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.961+0000 7f4ce6794640 10 monclient(hunting): picked mon.noname-a con 0x7f4ce0137e50 addr v2:172.21.15.12:3300/0 2024-09-11T01:45:44.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.961+0000 7f4ce6794640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4ce0138230 0x7f4ce0138640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:44.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.961+0000 7f4ce6794640 10 monclient(hunting): picked mon.noname-c con 0x7f4ce0138230 addr v2:172.21.15.71:3300/0 2024-09-11T01:45:44.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.961+0000 7f4ce6794640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4ce0134cb0 addr v1:172.21.15.12:6789/0 2024-09-11T01:45:44.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.961+0000 7f4ce6794640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ce00a62d0 con 0x7f4ce0134cb0 2024-09-11T01:45:44.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.961+0000 7f4ce6794640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:44.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.961+0000 7f4ce6794640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f4ce009bb40 con 0x7f4ce0137e50 2024-09-11T01:45:44.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.961+0000 7f4ce6794640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:44.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.961+0000 7f4ce6794640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f4ce009c4a0 con 0x7f4ce0138230 2024-09-11T01:45:44.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.961+0000 7f4ce6794640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:44.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.961+0000 7f4ce6794640 10 monclient(hunting): authenticate will time out at 1751.403748s 2024-09-11T01:45:44.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.961+0000 7f4cdffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4ce0137e50 0x7f4ce009d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:44.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.961+0000 7f4cdffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4ce0137e50 0x7f4ce009d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59890/0 (socket says 172.21.15.12:59890) 2024-09-11T01:45:44.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.961+0000 7f4cdffff640 1 -- 172.21.15.12:0/1653787430 learned_addr learned my addr 172.21.15.12:0/1653787430 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:44.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.961+0000 7f4cdffff640 10 monclient(hunting): get_auth_request con 0x7f4ce0137e50 auth_method 0 2024-09-11T01:45:44.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.961+0000 7f4cdffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:44.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.961+0000 7f4cdffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.961+0000 7f4cdffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.961+0000 7f4cdf7fe640 1 --2- 172.21.15.12:0/1653787430 >> v2:172.21.15.71:3300/0 conn(0x7f4ce0138230 0x7f4ce0138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.962+0000 7f4cdf7fe640 10 monclient(hunting): get_auth_request con 0x7f4ce0138230 auth_method 0 2024-09-11T01:45:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.962+0000 7f4cdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.962+0000 7f4cdf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.962+0000 7f4cdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.962+0000 7f4cdffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.962+0000 7f4cdffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.962+0000 7f4cdffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.962+0000 7f4cdf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.962+0000 7f4cdf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:44.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.962+0000 7f4cdf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:44.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.962+0000 7f4cdffff640 10 monclient(hunting): handle_auth_done global_id 174213 payload 274 2024-09-11T01:45:44.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.962+0000 7f4cdffff640 1 -- 172.21.15.12:0/1653787430 >> v2:172.21.15.71:3300/0 conn(0x7f4ce0138230 msgr2=0x7f4ce0138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:44.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.962+0000 7f4cdffff640 1 --2- 172.21.15.12:0/1653787430 >> v2:172.21.15.71:3300/0 conn(0x7f4ce0138230 0x7f4ce0138640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:44.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.962+0000 7f4cdffff640 1 -- 172.21.15.12:0/1653787430 >> v1:172.21.15.12:6789/0 conn(0x7f4ce0134cb0 legacy=0x7f4ce0135160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:44.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.962+0000 7f4cdffff640 10 monclient: _finish_hunting 0 2024-09-11T01:45:44.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.962+0000 7f4cdffff640 1 monclient: found mon.noname-a 2024-09-11T01:45:44.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.962+0000 7f4cdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:44.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.962+0000 7f4cdffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:45:44.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.962+0000 7f4cdffff640 1 -- 172.21.15.12:0/1653787430 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ce009c780 con 0x7f4ce0137e50 2024-09-11T01:45:44.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.965+0000 7f4cdffff640 1 -- 172.21.15.12:0/1653787430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce0137e50 msgr2=0x7f4ce009d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:44.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.965+0000 7f4cdffff640 1 --2- 172.21.15.12:0/1653787430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce0137e50 0x7f4ce009d160 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4ce00ab580 tx=0x7f4cd402a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9040105f7ae7f99d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:44.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.965+0000 7f4cdeffd640 1 -- 172.21.15.12:0/1653787430 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd4037070 con 0x7f4ce0137e50 2024-09-11T01:45:44.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.965+0000 7f4cdeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:44.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.965+0000 7f4cdeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:45:44.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.965+0000 7f4cdeffd640 10 monclient: dump: 2024-09-11T01:45:44.981 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:44.981 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:44.981 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:44.981 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:44.981 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:44.981 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:44.981 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:44.981 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:44.981 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:44.981 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:44.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.965+0000 7f4cdeffd640 10 monclient: _finish_auth 0 2024-09-11T01:45:44.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.965+0000 7f4cdeffd640 10 monclient: _check_auth_tickets 2024-09-11T01:45:44.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.965+0000 7f4cdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:44.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.965+0000 7f4cdeffd640 1 -- 172.21.15.12:0/1653787430 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cd4032440 con 0x7f4ce0137e50 2024-09-11T01:45:44.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.965+0000 7f4ce6794640 5 monclient: authenticate success, global_id 174213 2024-09-11T01:45:44.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.965+0000 7f4ce6794640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:45:44.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.965+0000 7f4cdeffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:44.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.965+0000 7f4ce6794640 10 monclient: get_monmap_and_config success 2024-09-11T01:45:44.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.965+0000 7f4cdeffd640 1 -- 172.21.15.12:0/1653787430 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd40327f0 con 0x7f4ce0137e50 2024-09-11T01:45:44.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.965+0000 7f4ce6794640 4 set_mon_vals no callback set 2024-09-11T01:45:44.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.965+0000 7f4cdeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:44.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.965+0000 7f4cdeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:44.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.965+0000 7f4cdeffd640 10 monclient: dump: 2024-09-11T01:45:44.982 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:44.983 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:44.983 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:44.983 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:44.983 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:44.983 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:44.983 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:44.983 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:44.983 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:44.983 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:44.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.966+0000 7f4ce6794640 10 monclient: shutdown 2024-09-11T01:45:44.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.966+0000 7f4ce6794640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:44.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.966+0000 7f4ce6794640 1 -- 172.21.15.12:0/1653787430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce0137e50 msgr2=0x7f4ce009d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:44.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.966+0000 7f4ce6794640 1 --2- 172.21.15.12:0/1653787430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce0137e50 0x7f4ce009d160 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4ce00ab580 tx=0x7f4cd402a080 comp rx=0 tx=0).stop 2024-09-11T01:45:44.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.966+0000 7f4ce6794640 1 -- 172.21.15.12:0/1653787430 shutdown_connections 2024-09-11T01:45:44.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.966+0000 7f4ce6794640 1 --2- 172.21.15.12:0/1653787430 >> v2:172.21.15.71:3300/0 conn(0x7f4ce0138230 0x7f4ce0138640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:44.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.966+0000 7f4ce6794640 1 --2- 172.21.15.12:0/1653787430 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce0137e50 0x7f4ce009d160 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:44.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.966+0000 7f4ce6794640 1 -- 172.21.15.12:0/1653787430 >> 172.21.15.12:0/1653787430 conn(0x7f4ce00a7020 msgr2=0x7f4ce00a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:44.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.966+0000 7f4ce6794640 1 -- 172.21.15.12:0/1653787430 shutdown_connections 2024-09-11T01:45:44.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.966+0000 7f4ce6794640 1 -- 172.21.15.12:0/1653787430 wait complete. 2024-09-11T01:45:44.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.966+0000 7f4ce6794640 10 monclient: build_initial_monmap 2024-09-11T01:45:44.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.966+0000 7f4ce6794640 10 monclient: monmap: 2024-09-11T01:45:44.984 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:44.984 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:45:44.984 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:45:44.984 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:45:44.985 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:44.985 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:44.985 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:45:44.985 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:45:44.985 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:45:44.985 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:44.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.967+0000 7f4ce6794640 1 Processor -- start 2024-09-11T01:45:44.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.967+0000 7f4ce6794640 1 -- start start 2024-09-11T01:45:44.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.967+0000 7f4ce6794640 10 monclient: init 2024-09-11T01:45:44.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.967+0000 7f4ce6794640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:44.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4ce6794640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:45:44.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4ce6794640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce0134cb0 0x7f4ce0133920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:44.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4ce6794640 10 monclient(hunting): picked mon.noname-a con 0x7f4ce0134cb0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:45:44.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4ce6794640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ce0137e50 0x7f4ce0133e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:44.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4ce6794640 10 monclient(hunting): picked mon.noname-c con 0x7f4ce0137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:44.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4ce6794640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce0138230 0x7f4ce01beef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:44.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4ce6794640 10 monclient(hunting): picked mon.noname-b con 0x7f4ce0138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:45:44.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4ce6794640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:44.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4ce6794640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce006f710 con 0x7f4ce0134cb0 2024-09-11T01:45:44.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4ce6794640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:44.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4ce6794640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce0138b80 con 0x7f4ce0138230 2024-09-11T01:45:44.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4ce6794640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:44.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4ce6794640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4ce006f890 con 0x7f4ce0137e50 2024-09-11T01:45:44.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4ce6794640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:44.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4ce6794640 10 monclient(hunting): authenticate will time out at 1751.410119s 2024-09-11T01:45:44.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce0134cb0 0x7f4ce0133920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:44.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdf7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ce0137e50 0x7f4ce0133e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:44.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce0134cb0 0x7f4ce0133920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59894/0 (socket says 172.21.15.12:59894) 2024-09-11T01:45:44.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdffff640 1 -- 172.21.15.12:0/1284213928 learned_addr learned my addr 172.21.15.12:0/1284213928 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:44.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdffff640 10 monclient(hunting): get_auth_request con 0x7f4ce0134cb0 auth_method 0 2024-09-11T01:45:44.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:44.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:44.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:44.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdf7fe640 10 monclient(hunting): get_auth_request con 0x7f4ce0137e50 auth_method 0 2024-09-11T01:45:44.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:44.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:44.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:44.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4ce4d31640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce0138230 0x7f4ce01beef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:44.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4ce4d31640 10 monclient(hunting): get_auth_request con 0x7f4ce0138230 auth_method 0 2024-09-11T01:45:44.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4ce4d31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:44.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4ce4d31640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:44.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4ce4d31640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:44.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:44.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:44.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:44.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:44.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:44.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:44.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4ce4d31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:44.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4ce4d31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:44.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4ce4d31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:44.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdf7fe640 10 monclient(hunting): handle_auth_done global_id 184109 payload 824 2024-09-11T01:45:44.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdf7fe640 1 -- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce0138230 msgr2=0x7f4ce01beef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:44.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdf7fe640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce0138230 0x7f4ce01beef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:44.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdf7fe640 1 -- 172.21.15.12:0/1284213928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce0134cb0 msgr2=0x7f4ce0133920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:44.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdf7fe640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce0134cb0 0x7f4ce0133920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:44.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdf7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:45:44.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdf7fe640 1 monclient: found mon.noname-c 2024-09-11T01:45:44.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:44.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:45:44.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdf7fe640 1 -- 172.21.15.12:0/1284213928 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ce01210d0 con 0x7f4ce0137e50 2024-09-11T01:45:44.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.968+0000 7f4cdffff640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce0134cb0 0x7f4ce0133920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:45:44.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.970+0000 7f4cdf7fe640 1 -- 172.21.15.12:0/1284213928 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ce0137e50 msgr2=0x7f4ce0133e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:44.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.970+0000 7f4cdf7fe640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ce0137e50 0x7f4ce0133e60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4cd000e850 tx=0x7f4cd000ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:44.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.970+0000 7f4cdf7fe640 1 -- 172.21.15.12:0/1284213928 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ce0137e50 msgr2=0x7f4ce0133e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:45:44.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.970+0000 7f4cdf7fe640 1 -- 172.21.15.12:0/1284213928 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ce0137e50 msgr2=0x7f4ce0133e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:45:44.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.970+0000 7f4cdf7fe640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ce0137e50 0x7f4ce0133e60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4cd000e850 tx=0x7f4cd000ed10 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:45:44.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.970+0000 7f4cdf7fe640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ce0137e50 0x7f4ce0133e60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4cd000e850 tx=0x7f4cd000ed10 comp rx=0 tx=0).stop 2024-09-11T01:45:44.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.970+0000 7f4cdcff9640 10 monclient: ms_handle_reset current mon [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:44.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.970+0000 7f4cdcff9640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:44.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.970+0000 7f4cdcff9640 1 -- 172.21.15.12:0/1284213928 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ce0137e50 msgr2=0x7f4ce0133e60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:44.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.970+0000 7f4cdcff9640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ce0137e50 0x7f4ce0133e60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:44.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.970+0000 7f4cdcff9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:45:44.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.970+0000 7f4cdcff9640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ccc0034a0 0x7f4ccc0058f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:44.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdcff9640 10 monclient(hunting): picked mon.noname-c con 0x7f4ccc0034a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:44.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdcff9640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce0138230 0x7f4ccc008460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:44.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdcff9640 10 monclient(hunting): picked mon.noname-b con 0x7f4ce0138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:45:44.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdcff9640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce0134cb0 0x7f4ccc00aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:44.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdcff9640 10 monclient(hunting): picked mon.noname-a con 0x7f4ce0134cb0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:45:44.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdcff9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:44.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdcff9640 1 -- 172.21.15.12:0/1284213928 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4ccc00b900 con 0x7f4ce0134cb0 2024-09-11T01:45:44.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdcff9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:44.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdcff9640 1 -- 172.21.15.12:0/1284213928 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4ccc00ba80 con 0x7f4ce0138230 2024-09-11T01:45:44.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdcff9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:44.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdcff9640 1 -- 172.21.15.12:0/1284213928 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4ccc00bc80 con 0x7f4ccc0034a0 2024-09-11T01:45:44.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdcff9640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:44.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdffff640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ccc0034a0 0x7f4ccc0058f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:44.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdffff640 10 monclient(hunting): get_auth_request con 0x7f4ccc0034a0 auth_method 0 2024-09-11T01:45:44.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:44.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:44.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:45:44.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4ce4d31640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce0134cb0 0x7f4ccc00aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:44.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4ce4d31640 10 monclient(hunting): get_auth_request con 0x7f4ce0134cb0 auth_method 0 2024-09-11T01:45:44.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4ce4d31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:44.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4ce4d31640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:44.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4ce4d31640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:45:44.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdf7fe640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce0138230 0x7f4ccc008460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:44.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdf7fe640 10 monclient(hunting): get_auth_request con 0x7f4ce0138230 auth_method 0 2024-09-11T01:45:44.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:44.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:44.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:45:44.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:44.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:44.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:45:44.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4ce4d31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:44.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4ce4d31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:44.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4ce4d31640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:45:44.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:44.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:44.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:45:44.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdffff640 10 monclient(hunting): handle_auth_done global_id 184109 payload 293 2024-09-11T01:45:44.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdffff640 1 -- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce0138230 msgr2=0x7f4ccc008460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:44.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdffff640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce0138230 0x7f4ccc008460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:44.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdffff640 1 -- 172.21.15.12:0/1284213928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce0134cb0 msgr2=0x7f4ccc00aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:44.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdffff640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce0134cb0 0x7f4ccc00aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:44.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdffff640 10 monclient: _finish_hunting 0 2024-09-11T01:45:44.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdffff640 1 monclient: found mon.noname-c 2024-09-11T01:45:44.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:44.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:45:44.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4cdffff640 1 -- 172.21.15.12:0/1284213928 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ccc00bf40 con 0x7f4ccc0034a0 2024-09-11T01:45:44.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.971+0000 7f4ce4d31640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce0134cb0 0x7f4ccc00aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:45:44.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.973+0000 7f4cdffff640 1 -- 172.21.15.12:0/1284213928 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ccc0034a0 msgr2=0x7f4ccc0058f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:44.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.973+0000 7f4cdffff640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ccc0034a0 0x7f4ccc0058f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4cd40090b0 tx=0x7f4cd0009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:44.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.973+0000 7f4cdcff9640 1 -- 172.21.15.12:0/1284213928 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd4037040 con 0x7f4ccc0034a0 2024-09-11T01:45:44.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.973+0000 7f4cdcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:44.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.973+0000 7f4cdcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:45:44.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.973+0000 7f4cdcff9640 10 monclient: dump: 2024-09-11T01:45:44.996 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:44.996 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:44.996 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:44.996 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:44.996 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:44.997 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:44.997 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:44.997 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:44.997 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:44.997 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:44.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.973+0000 7f4cdcff9640 10 monclient: _finish_auth 0 2024-09-11T01:45:44.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.973+0000 7f4cdcff9640 10 monclient: _check_auth_tickets 2024-09-11T01:45:44.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.973+0000 7f4cdcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:44.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.973+0000 7f4cdcff9640 1 -- 172.21.15.12:0/1284213928 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cd403c640 con 0x7f4ccc0034a0 2024-09-11T01:45:44.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.973+0000 7f4cdcff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:44.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.973+0000 7f4cdcff9640 1 -- 172.21.15.12:0/1284213928 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd403ca10 con 0x7f4ccc0034a0 2024-09-11T01:45:44.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.973+0000 7f4ce6794640 5 monclient: authenticate success, global_id 184109 2024-09-11T01:45:44.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.973+0000 7f4ce6794640 10 monclient: _renew_subs 2024-09-11T01:45:44.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.973+0000 7f4ce6794640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:45:44.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.973+0000 7f4ce6794640 1 -- 172.21.15.12:0/1284213928 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ce01213b0 con 0x7f4ccc0034a0 2024-09-11T01:45:44.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.973+0000 7f4cdcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:44.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.973+0000 7f4cde7fc640 4 set_mon_vals no callback set 2024-09-11T01:45:44.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.973+0000 7f4cdcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:45:44.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.973+0000 7f4cdcff9640 10 monclient: dump: 2024-09-11T01:45:44.998 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:44.998 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:44.998 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:44.999 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:44.999 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:44.999 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:44.999 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:44.999 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:44.999 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:44.999 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:44.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.973+0000 7f4ce6794640 10 monclient: _renew_subs 2024-09-11T01:45:44.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.974+0000 7f4ce6794640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:45:44.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.974+0000 7f4ce6794640 1 -- 172.21.15.12:0/1284213928 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ce01c0f50 con 0x7f4ccc0034a0 2024-09-11T01:45:45.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.974+0000 7f4ce6794640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:45.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.974+0000 7f4ce6794640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce0133920 0x7f4ce0133d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:45.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.974+0000 7f4ce6794640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:45.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.974+0000 7f4ce6794640 1 -- 172.21.15.12:0/1284213928 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce00a1600 con 0x7f4ce0133920 2024-09-11T01:45:45.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.974+0000 7f4cdcff9640 1 -- 172.21.15.12:0/1284213928 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f4cd403cbb0 con 0x7f4ccc0034a0 2024-09-11T01:45:45.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.975+0000 7f4cdf7fe640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce0133920 0x7f4ce0133d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:45.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.975+0000 7f4cdf7fe640 10 monclient: get_auth_request con 0x7f4ce0133920 auth_method 0 2024-09-11T01:45:45.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.975+0000 7f4cdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:45.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.975+0000 7f4cdf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:45.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.975+0000 7f4cdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:45.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.975+0000 7f4cdf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:45.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.975+0000 7f4cdf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:45.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.975+0000 7f4cdcff9640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4ce0137e50 0x7f4ccc048860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:45.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.975+0000 7f4cdf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:45.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.975+0000 7f4cdcff9640 1 -- 172.21.15.12:0/1284213928 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(564..564 src has 317..564) ==== 4799+0+0 (secure 0 0 0) 0x7f4cd4072ea0 con 0x7f4ccc0034a0 2024-09-11T01:45:45.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.975+0000 7f4cdf7fe640 10 monclient(hunting): handle_auth_done global_id 134551 payload 824 2024-09-11T01:45:45.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.975+0000 7f4cdf7fe640 1 -- 172.21.15.12:0/1284213928 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4ce01372d0 con 0x7f4ce0133920 2024-09-11T01:45:45.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.975+0000 7f4ce4d31640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4ce0137e50 0x7f4ccc048860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:45.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.975+0000 7f4ce4d31640 10 monclient: get_auth_request con 0x7f4ce0137e50 auth_method 0 2024-09-11T01:45:45.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.977+0000 7f4cdf7fe640 1 -- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce0133920 msgr2=0x7f4ce0133d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:45.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.977+0000 7f4cdf7fe640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce0133920 0x7f4ce0133d00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4ce01c0ee0 tx=0x7f4cd00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:45.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.977+0000 7f4cdcff9640 1 -- 172.21.15.12:0/1284213928 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd002d070 con 0x7f4ce0133920 2024-09-11T01:45:45.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.977+0000 7f4cdcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:45.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.977+0000 7f4cdcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:45:45.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.977+0000 7f4cdcff9640 10 monclient: dump: 2024-09-11T01:45:45.002 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:45.002 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:45.002 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:45.002 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:45.002 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:45.003 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:45.003 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:45.003 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:45.003 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:45.003 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:45.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.977+0000 7f4ce4d31640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4ce0137e50 0x7f4ccc048860 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4ce01c15a0 tx=0x7f4cd8003d10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:45.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.978+0000 7f4cdcff9640 1 -- 172.21.15.12:0/1284213928 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4cd0003a30 con 0x7f4ce0133920 2024-09-11T01:45:45.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.978+0000 7f4cdcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:45.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.978+0000 7f4cdcff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:45:45.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.978+0000 7f4cdcff9640 1 -- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce0133920 msgr2=0x7f4ce0133d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:45.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.978+0000 7f4cdcff9640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce0133920 0x7f4ce0133d00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4ce01c0ee0 tx=0x7f4cd00046c0 comp rx=0 tx=0).stop 2024-09-11T01:45:45.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.978+0000 7f4cdcff9640 1 -- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce0133920 msgr2=0x7f4ce0133d00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:45.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.978+0000 7f4cdcff9640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce0133920 0x7f4ce0133d00 secure :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4ce01c0ee0 tx=0x7f4cd00046c0 comp rx=0 tx=0).stop 2024-09-11T01:45:45.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.984+0000 7f4ce6794640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:45:45.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.984+0000 7f4ce6794640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce0134240 0x7f4ce00a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:45.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.984+0000 7f4ce6794640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:45.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.984+0000 7f4ce6794640 1 -- 172.21.15.12:0/1284213928 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce006a060 con 0x7f4ce0134240 2024-09-11T01:45:45.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.985+0000 7f4cdf7fe640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce0134240 0x7f4ce00a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:45.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.985+0000 7f4cdf7fe640 10 monclient: get_auth_request con 0x7f4ce0134240 auth_method 0 2024-09-11T01:45:45.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.985+0000 7f4cdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:45.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.985+0000 7f4cdf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:45.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.985+0000 7f4cdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:45.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.985+0000 7f4cdf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:45.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.985+0000 7f4cdf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:45.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.985+0000 7f4cdf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:45.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.985+0000 7f4cdf7fe640 10 monclient(hunting): handle_auth_done global_id 134554 payload 824 2024-09-11T01:45:45.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.985+0000 7f4cdf7fe640 1 -- 172.21.15.12:0/1284213928 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4ce01c1180 con 0x7f4ce0134240 2024-09-11T01:45:45.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.987+0000 7f4cdf7fe640 1 -- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce0134240 msgr2=0x7f4ce00a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:45.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.987+0000 7f4cdf7fe640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce0134240 0x7f4ce00a1920 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4ce009d670 tx=0x7f4cd0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:45.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.988+0000 7f4cdcff9640 1 -- 172.21.15.12:0/1284213928 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd002d050 con 0x7f4ce0134240 2024-09-11T01:45:45.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.988+0000 7f4cdcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:45.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.988+0000 7f4cdcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:45:45.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.988+0000 7f4cdcff9640 10 monclient: dump: 2024-09-11T01:45:45.007 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:45.007 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:45.007 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:45.007 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:45.008 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:45.008 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:45.008 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:45.008 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:45.008 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:45.008 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:45.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.988+0000 7f4cdcff9640 1 -- 172.21.15.12:0/1284213928 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f4cd002e040 con 0x7f4ce0134240 2024-09-11T01:45:45.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.988+0000 7f4cdcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:45:45.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.988+0000 7f4cdcff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:45:45.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.988+0000 7f4cdcff9640 1 -- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce0134240 msgr2=0x7f4ce00a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:45.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.988+0000 7f4cdcff9640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce0134240 0x7f4ce00a1920 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4ce009d670 tx=0x7f4cd0004730 comp rx=0 tx=0).stop 2024-09-11T01:45:45.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.988+0000 7f4cdcff9640 1 -- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce0134240 msgr2=0x7f4ce00a1920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:45.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.988+0000 7f4cdcff9640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce0134240 0x7f4ce00a1920 secure :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4ce009d670 tx=0x7f4cd0004730 comp rx=0 tx=0).stop 2024-09-11T01:45:45.009 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:45:45.009 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:45:45.009 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:45:45.009 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:45:45.009 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 154, 2024-09-11T01:45:45.009 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 277949, 2024-09-11T01:45:45.009 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:45:45.010 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:45:45.010 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:45:45.010 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:45:45.010 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:45.010 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:45:45.010 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:45:45.010 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:45:45.010 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:45:45.010 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:45.010 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:45.010 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:45.010 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:45.011 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:45.011 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:45.011 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:45.011 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:45.011 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:45.011 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:45.011 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:45.011 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:45.011 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:45:45.012 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:45:45.012 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:45.012 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:45.012 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:45.012 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:45.012 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:45.012 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:45.012 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:45.012 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:45.012 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:45.012 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:45.013 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:45.013 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:45.013 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:45.013 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:45:45.013 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:45:45.013 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:45:45.013 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:45:45.013 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:45:45.013 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:45:45.013 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:45:45.013 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:45:45.013 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:45:45.013 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:45:45.014 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:45:45.014 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:45:45.014 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:45:45.014 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:45:45.014 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:45:45.014 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:45:45.014 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:45:45.014 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:45.014 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:45.014 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:45.014 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:45.014 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:45.014 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:45.015 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:45.015 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:45.015 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:45.015 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:45.015 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:45.015 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:45.015 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:45:45.015 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:45.015 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:45:45.015 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:45.015 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:45:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:45:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:45:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:45:45.016 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:45:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:45:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:45:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:45:45.017 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:45:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:45:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:45.018 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:45:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:45:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:45:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:45:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:45.019 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:45:45.020 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:45.020 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:45.020 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:45.020 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:45.020 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:45:45.020 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:45.020 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:45.020 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:45.020 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:45.020 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:45:45.020 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:45:45.020 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:45.020 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:45.021 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:45.021 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:45.021 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:45.021 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:45.021 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:45:45.021 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:45:45.021 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:45.021 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:45.021 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:45.021 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:45:45.021 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:45.022 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:45.022 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:45:45.022 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:45.022 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:45.022 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:45.022 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:45:45.022 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:45.022 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:45.022 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:45:45.022 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:45.022 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:45.022 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:45.022 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:45:45.023 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:45.023 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:45.023 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:45:45.023 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:45.023 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:45.023 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:45.023 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:45:45.023 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:45.023 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:45.023 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:45.023 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:45:45.023 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:45:45.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.988+0000 7f4ce6794640 1 -- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4ce0137e50 msgr2=0x7f4ccc048860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:45.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.988+0000 7f4ce6794640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4ce0137e50 0x7f4ccc048860 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4ce01c15a0 tx=0x7f4cd8003d10 comp rx=0 tx=0).stop 2024-09-11T01:45:45.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.988+0000 7f4ce6794640 10 monclient: shutdown 2024-09-11T01:45:45.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.988+0000 7f4ce6794640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:45.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.988+0000 7f4ce6794640 1 -- 172.21.15.12:0/1284213928 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ccc0034a0 msgr2=0x7f4ccc0058f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:45.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.988+0000 7f4ce6794640 1 --2- 172.21.15.12:0/1284213928 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ccc0034a0 0x7f4ccc0058f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4cd40090b0 tx=0x7f4cd0009190 comp rx=0 tx=0).stop 2024-09-11T01:45:45.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.988+0000 7f4ce4d31640 1 -- 172.21.15.12:0/1284213928 reap_dead start 2024-09-11T01:45:45.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.988+0000 7f4ce4d31640 1 -- 172.21.15.12:0/1284213928 reap_dead start 2024-09-11T01:45:45.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.988+0000 7f4ce6794640 1 -- 172.21.15.12:0/1284213928 shutdown_connections 2024-09-11T01:45:45.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.988+0000 7f4ce6794640 1 -- 172.21.15.12:0/1284213928 >> 172.21.15.12:0/1284213928 conn(0x7f4ce00a7020 msgr2=0x7f4ce00a0430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:45.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.989+0000 7f4ce6794640 1 -- 172.21.15.12:0/1284213928 shutdown_connections 2024-09-11T01:45:45.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:44.989+0000 7f4ce6794640 1 -- 172.21.15.12:0/1284213928 wait complete. 2024-09-11T01:45:45.026 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:45:45.026 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:45:45.026 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:45:45.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.126+0000 7f2695693640 10 monclient: get_monmap_and_config 2024-09-11T01:45:45.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.126+0000 7f2695693640 10 monclient: build_initial_monmap 2024-09-11T01:45:45.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.126+0000 7f2695693640 10 monclient: monmap: 2024-09-11T01:45:45.128 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:45.128 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:45.128 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:45:45.127093+0000 2024-09-11T01:45:45.128 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:45:45.127093+0000 2024-09-11T01:45:45.128 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:45.128 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:45.128 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:45:45.128 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:45:45.128 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:45:45.129 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:45:45.129 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:45:45.129 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f2695693640 1 Processor -- start 2024-09-11T01:45:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f2695693640 1 -- start start 2024-09-11T01:45:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f2695693640 10 monclient: init 2024-09-11T01:45:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f2695693640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f2695693640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-11T01:45:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f2695693640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f26900a2e20 0x7f269006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f2695693640 10 monclient(hunting): picked mon.noname-a con 0x7f26900a2e20 addr v2:172.21.15.12:3300/0 2024-09-11T01:45:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f2695693640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f26900a4970 addr v1:172.21.15.71:6789/0 2024-09-11T01:45:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f2695693640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f269006f910 0x7f269013f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f2695693640 10 monclient(hunting): picked mon.noname-b con 0x7f269006f910 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f2695693640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f269006f4f0 con 0x7f26900a4970 2024-09-11T01:45:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f2695693640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f2695693640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f269006ef30 con 0x7f26900a2e20 2024-09-11T01:45:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f2695693640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f2695693640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f269006ec50 con 0x7f269006f910 2024-09-11T01:45:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f2695693640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f2695693640 10 monclient(hunting): authenticate will time out at 1751.569660s 2024-09-11T01:45:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f268effd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f26900a2e20 0x7f269006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f268f7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f269006f910 0x7f269013f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f268effd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f26900a2e20 0x7f269006ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59898/0 (socket says 172.21.15.12:59898) 2024-09-11T01:45:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f268f7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f269006f910 0x7f269013f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42816/0 (socket says 172.21.15.12:42816) 2024-09-11T01:45:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f268effd640 1 -- 172.21.15.12:0/3106573971 learned_addr learned my addr 172.21.15.12:0/3106573971 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f268effd640 10 monclient(hunting): get_auth_request con 0x7f26900a2e20 auth_method 0 2024-09-11T01:45:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f268effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f268effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f268effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f268f7fe640 10 monclient(hunting): get_auth_request con 0x7f269006f910 auth_method 0 2024-09-11T01:45:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f268f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f268f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f268f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f268effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f268effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f268effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f268f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f268f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.127+0000 7f268f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.128+0000 7f268effd640 10 monclient(hunting): handle_auth_done global_id 174219 payload 274 2024-09-11T01:45:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.128+0000 7f268effd640 1 -- 172.21.15.12:0/3106573971 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f269006f910 msgr2=0x7f269013f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.128+0000 7f268effd640 1 --2- 172.21.15.12:0/3106573971 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f269006f910 0x7f269013f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.128+0000 7f268effd640 1 -- 172.21.15.12:0/3106573971 >> v1:172.21.15.71:6789/0 conn(0x7f26900a4970 legacy=0x7f26900a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.128+0000 7f268effd640 10 monclient: _finish_hunting 0 2024-09-11T01:45:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.128+0000 7f268effd640 1 monclient: found mon.noname-a 2024-09-11T01:45:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.128+0000 7f268effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:45.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.128+0000 7f268effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:45:45.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.128+0000 7f268effd640 1 -- 172.21.15.12:0/3106573971 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26900a3910 con 0x7f26900a2e20 2024-09-11T01:45:45.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.128+0000 7f268f7fe640 1 --2- 172.21.15.12:0/3106573971 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f269006f910 0x7f269013f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:45:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.130+0000 7f268effd640 1 -- 172.21.15.12:0/3106573971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26900a2e20 msgr2=0x7f269006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.130+0000 7f268effd640 1 --2- 172.21.15.12:0/3106573971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26900a2e20 0x7f269006ded0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2680009870 tx=0x7f268002a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c027324c1a14c38f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f268e7fc640 1 -- 172.21.15.12:0/3106573971 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2680037070 con 0x7f26900a2e20 2024-09-11T01:45:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f268e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f268e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:45:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f268e7fc640 10 monclient: dump: 2024-09-11T01:45:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:45.142 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f268e7fc640 10 monclient: _finish_auth 0 2024-09-11T01:45:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f268e7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:45:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f268e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f268e7fc640 1 -- 172.21.15.12:0/3106573971 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2680032440 con 0x7f26900a2e20 2024-09-11T01:45:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f2695693640 5 monclient: authenticate success, global_id 174219 2024-09-11T01:45:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f2695693640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:45:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f268e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f2695693640 10 monclient: get_monmap_and_config success 2024-09-11T01:45:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f268e7fc640 1 -- 172.21.15.12:0/3106573971 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2680032810 con 0x7f26900a2e20 2024-09-11T01:45:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f2695693640 4 set_mon_vals no callback set 2024-09-11T01:45:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f268e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f268e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f268e7fc640 10 monclient: dump: 2024-09-11T01:45:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:45.143 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f2695693640 10 monclient: shutdown 2024-09-11T01:45:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f2695693640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f2695693640 1 -- 172.21.15.12:0/3106573971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26900a2e20 msgr2=0x7f269006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f2695693640 1 --2- 172.21.15.12:0/3106573971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26900a2e20 0x7f269006ded0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2680009870 tx=0x7f268002a0a0 comp rx=0 tx=0).stop 2024-09-11T01:45:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f2695693640 1 -- 172.21.15.12:0/3106573971 shutdown_connections 2024-09-11T01:45:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f2695693640 1 --2- 172.21.15.12:0/3106573971 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f269006f910 0x7f269013f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f2695693640 1 --2- 172.21.15.12:0/3106573971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26900a2e20 0x7f269006ded0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f2695693640 1 -- 172.21.15.12:0/3106573971 >> 172.21.15.12:0/3106573971 conn(0x7f26900a8600 msgr2=0x7f26900a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f2695693640 1 -- 172.21.15.12:0/3106573971 shutdown_connections 2024-09-11T01:45:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.131+0000 7f2695693640 1 -- 172.21.15.12:0/3106573971 wait complete. 2024-09-11T01:45:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.132+0000 7f2695693640 10 monclient: build_initial_monmap 2024-09-11T01:45:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.132+0000 7f2695693640 10 monclient: monmap: 2024-09-11T01:45:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:45:45.145 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:45:45.145 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:45:45.145 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:45.145 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:45.145 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:45:45.145 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:45:45.145 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:45:45.145 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:45.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.132+0000 7f2695693640 1 Processor -- start 2024-09-11T01:45:45.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.132+0000 7f2695693640 1 -- start start 2024-09-11T01:45:45.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.132+0000 7f2695693640 10 monclient: init 2024-09-11T01:45:45.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f2695693640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:45.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f2695693640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:45:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f2695693640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f269006f910 0x7f26901398e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f2695693640 10 monclient(hunting): picked mon.noname-b con 0x7f269006f910 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:45:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f2695693640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f26900a2e20 0x7f2690139e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f2695693640 10 monclient(hunting): picked mon.noname-c con 0x7f26900a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f2695693640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26900a4970 0x7f26901cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f2695693640 10 monclient(hunting): picked mon.noname-a con 0x7f26900a4970 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:45:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f2695693640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f2695693640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f269013fd00 con 0x7f26900a4970 2024-09-11T01:45:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f2695693640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f2695693640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f269013fb80 con 0x7f269006f910 2024-09-11T01:45:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f2695693640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f2695693640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f269013fe80 con 0x7f26900a2e20 2024-09-11T01:45:45.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f2695693640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:45.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f2695693640 10 monclient(hunting): authenticate will time out at 1751.575242s 2024-09-11T01:45:45.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f2687fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f26900a2e20 0x7f2690139e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:45.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f268f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26900a4970 0x7f26901cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:45.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f2687fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f26900a2e20 0x7f2690139e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42830/0 (socket says 172.21.15.12:42830) 2024-09-11T01:45:45.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f268f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26900a4970 0x7f26901cd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59910/0 (socket says 172.21.15.12:59910) 2024-09-11T01:45:45.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f2687fff640 1 -- 172.21.15.12:0/2652821420 learned_addr learned my addr 172.21.15.12:0/2652821420 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:45.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f2687fff640 10 monclient(hunting): get_auth_request con 0x7f26900a2e20 auth_method 0 2024-09-11T01:45:45.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f2687fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:45.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f2687fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:45.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f2687fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:45.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f268f7fe640 10 monclient(hunting): get_auth_request con 0x7f26900a4970 auth_method 0 2024-09-11T01:45:45.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f268f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:45.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f268f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f268f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f268f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f268f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f268f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f2687fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f2687fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f2687fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f268effd640 1 --2- 172.21.15.12:0/2652821420 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f269006f910 0x7f26901398e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f268effd640 10 monclient(hunting): get_auth_request con 0x7f269006f910 auth_method 0 2024-09-11T01:45:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f268effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f268effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f268effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f268f7fe640 10 monclient(hunting): handle_auth_done global_id 174222 payload 824 2024-09-11T01:45:45.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f268f7fe640 1 -- 172.21.15.12:0/2652821420 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f26900a2e20 msgr2=0x7f2690139e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:45.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f268f7fe640 1 --2- 172.21.15.12:0/2652821420 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f26900a2e20 0x7f2690139e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:45.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f268f7fe640 1 -- 172.21.15.12:0/2652821420 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f269006f910 msgr2=0x7f26901398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:45.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f268f7fe640 1 --2- 172.21.15.12:0/2652821420 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f269006f910 0x7f26901398e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:45.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f268f7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:45:45.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f268f7fe640 1 monclient: found mon.noname-a 2024-09-11T01:45:45.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f268f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:45.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f268f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:45:45.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f268f7fe640 1 -- 172.21.15.12:0/2652821420 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f269013c050 con 0x7f26900a4970 2024-09-11T01:45:45.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.133+0000 7f268effd640 1 --2- 172.21.15.12:0/2652821420 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f269006f910 0x7f26901398e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:45:45.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.135+0000 7f268f7fe640 1 -- 172.21.15.12:0/2652821420 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26900a4970 msgr2=0x7f26901cd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:45.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.135+0000 7f268f7fe640 1 --2- 172.21.15.12:0/2652821420 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26900a4970 0x7f26901cd030 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f268800ef90 tx=0x7f268800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:45.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.136+0000 7f26877fe640 1 -- 172.21.15.12:0/2652821420 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2688018020 con 0x7f26900a4970 2024-09-11T01:45:45.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.136+0000 7f26877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:45.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.136+0000 7f26877fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:45:45.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.136+0000 7f26877fe640 10 monclient: dump: 2024-09-11T01:45:45.150 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:45.150 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:45.150 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:45.150 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:45.150 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:45.150 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:45.150 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:45.150 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:45.150 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:45.150 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:45.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.136+0000 7f26877fe640 10 monclient: _finish_auth 0 2024-09-11T01:45:45.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.136+0000 7f26877fe640 10 monclient: _check_auth_tickets 2024-09-11T01:45:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.136+0000 7f26877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.136+0000 7f2695693640 5 monclient: authenticate success, global_id 174222 2024-09-11T01:45:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.136+0000 7f26877fe640 1 -- 172.21.15.12:0/2652821420 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26880146a0 con 0x7f26900a4970 2024-09-11T01:45:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.136+0000 7f2695693640 10 monclient: _renew_subs 2024-09-11T01:45:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.136+0000 7f2695693640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.136+0000 7f2695693640 1 -- 172.21.15.12:0/2652821420 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2690072070 con 0x7f26900a4970 2024-09-11T01:45:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.136+0000 7f26877fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.136+0000 7f26877fe640 1 -- 172.21.15.12:0/2652821420 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2688014a90 con 0x7f26900a4970 2024-09-11T01:45:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.136+0000 7f26877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.136+0000 7f26877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.136+0000 7f26877fe640 10 monclient: dump: 2024-09-11T01:45:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:45.151 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:45.152 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:45.152 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:45.152 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:45.152 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:45.152 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:45.152 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:45.152 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:45.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.136+0000 7f268dffb640 4 set_mon_vals no callback set 2024-09-11T01:45:45.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.136+0000 7f2695693640 10 monclient: _renew_subs 2024-09-11T01:45:45.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.136+0000 7f2695693640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:45.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.136+0000 7f2695693640 1 -- 172.21.15.12:0/2652821420 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f269013bd70 con 0x7f26900a4970 2024-09-11T01:45:45.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.137+0000 7f26877fe640 1 -- 172.21.15.12:0/2652821420 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f2688014c30 con 0x7f26900a4970 2024-09-11T01:45:45.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.137+0000 7f2695693640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:45.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.137+0000 7f2695693640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:45.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.137+0000 7f2695693640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:45.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.137+0000 7f2695693640 1 -- 172.21.15.12:0/2652821420 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f269013efd0 con 0x7f26900a4970 2024-09-11T01:45:45.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.137+0000 7f26877fe640 1 --2- 172.21.15.12:0/2652821420 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f267003e260 0x7f2670040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:45.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.137+0000 7f26877fe640 1 -- 172.21.15.12:0/2652821420 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(564..564 src has 317..564) ==== 4799+0+0 (secure 0 0 0) 0x7f2688053f10 con 0x7f26900a4970 2024-09-11T01:45:45.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.137+0000 7f268effd640 1 --2- 172.21.15.12:0/2652821420 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f267003e260 0x7f2670040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:45.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.137+0000 7f268effd640 10 monclient: get_auth_request con 0x7f267003e260 auth_method 0 2024-09-11T01:45:45.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.140+0000 7f268effd640 1 --2- 172.21.15.12:0/2652821420 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f267003e260 0x7f2670040720 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f2680034000 tx=0x7f268002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:45.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.141+0000 7f26877fe640 1 -- 172.21.15.12:0/2652821420 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f269013efd0 con 0x7f26900a4970 2024-09-11T01:45:45.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.141+0000 7f26877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:45.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.141+0000 7f26877fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:45:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.398+0000 7f2695693640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:45:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.398+0000 7f2695693640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:45:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.398+0000 7f2695693640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.398+0000 7f2695693640 1 -- 172.21.15.12:0/2652821420 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f269006ded0 con 0x7f26900a4970 2024-09-11T01:45:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.399+0000 7f26877fe640 1 -- 172.21.15.12:0/2652821420 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f2688022030 con 0x7f26900a4970 2024-09-11T01:45:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.399+0000 7f26877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:45:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.399+0000 7f26877fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:45:45.401 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.399+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.401 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.399+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=12,logm=46} crc {auth=1849695780,config=2586702516,health=327601140,kv=3263697420,logm=2918558878}) 2024-09-11T01:45:45.401 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.399+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=12,kv=11,logm=47} crc {auth=2679820648,config=2586702516,health=327601140,kv=3848642485,logm=507836376}) 2024-09-11T01:45:45.401 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.399+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.401 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.399+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=12,logm=46} crc {auth=1849695780,config=2586702516,health=327601140,kv=3263697420,logm=2918558878}) 2024-09-11T01:45:45.401 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.399+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=11,kv=12,logm=50} crc {auth=193628420,config=2586702516,health=2473239523,kv=3263697420,logm=2610457389}) 2024-09-11T01:45:45.402 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.400+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.402 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.400+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3995348075}) 2024-09-11T01:45:45.402 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.400+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=104052156}) 2024-09-11T01:45:45.402 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.400+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.402 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.400+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3995348075}) 2024-09-11T01:45:45.402 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.400+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=86651402}) 2024-09-11T01:45:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.401+0000 7f2695693640 1 -- 172.21.15.12:0/2652821420 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f267003e260 msgr2=0x7f2670040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.401+0000 7f2695693640 1 --2- 172.21.15.12:0/2652821420 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f267003e260 0x7f2670040720 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f2680034000 tx=0x7f268002e000 comp rx=0 tx=0).stop 2024-09-11T01:45:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.401+0000 7f2695693640 10 monclient: shutdown 2024-09-11T01:45:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.401+0000 7f2695693640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.401+0000 7f2695693640 1 -- 172.21.15.12:0/2652821420 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26900a4970 msgr2=0x7f26901cd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.401+0000 7f2695693640 1 --2- 172.21.15.12:0/2652821420 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26900a4970 0x7f26901cd030 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f268800ef90 tx=0x7f268800c490 comp rx=0 tx=0).stop 2024-09-11T01:45:45.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.401+0000 7f2695693640 1 -- 172.21.15.12:0/2652821420 shutdown_connections 2024-09-11T01:45:45.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.401+0000 7f2695693640 1 --2- 172.21.15.12:0/2652821420 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f267003e260 0x7f2670040720 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:45.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.401+0000 7f2695693640 1 --2- 172.21.15.12:0/2652821420 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f26900a4970 0x7f26901cd030 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:45.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.401+0000 7f2695693640 1 --2- 172.21.15.12:0/2652821420 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f26900a2e20 0x7f2690139e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:45.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.401+0000 7f2695693640 1 --2- 172.21.15.12:0/2652821420 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f269006f910 0x7f26901398e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:45.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.401+0000 7f2695693640 1 -- 172.21.15.12:0/2652821420 >> 172.21.15.12:0/2652821420 conn(0x7f26900a8600 msgr2=0x7f26901381d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:45.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.401+0000 7f2695693640 1 -- 172.21.15.12:0/2652821420 shutdown_connections 2024-09-11T01:45:45.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:45.401+0000 7f2695693640 1 -- 172.21.15.12:0/2652821420 wait complete. 2024-09-11T01:45:45.405 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.401+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.405 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.401+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1666219252}) 2024-09-11T01:45:45.406 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.401+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4245677266}) 2024-09-11T01:45:45.406 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.401+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.406 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.401+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1666219252}) 2024-09-11T01:45:45.406 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.401+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1588622555}) 2024-09-11T01:45:45.406 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.402+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.406 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.402+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=565479067}) 2024-09-11T01:45:45.406 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.402+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=786149820}) 2024-09-11T01:45:45.406 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.402+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.406 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.402+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=565479067}) 2024-09-11T01:45:45.406 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.402+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1470737710}) 2024-09-11T01:45:45.406 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.404+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.406 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.404+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2826500386}) 2024-09-11T01:45:45.406 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.404+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2541239082}) 2024-09-11T01:45:45.407 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.404+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.407 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.404+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2826500386}) 2024-09-11T01:45:45.407 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.404+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3160544410}) 2024-09-11T01:45:45.408 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.405+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.408 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.405+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3310666076}) 2024-09-11T01:45:45.408 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.405+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4189176245}) 2024-09-11T01:45:45.408 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.405+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.408 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.405+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3310666076}) 2024-09-11T01:45:45.408 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.405+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2490674981}) 2024-09-11T01:45:45.409 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.405+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.409 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.405+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4218450957}) 2024-09-11T01:45:45.409 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.405+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2485316353}) 2024-09-11T01:45:45.409 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.405+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.409 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.405+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4218450957}) 2024-09-11T01:45:45.409 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.405+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3523753297}) 2024-09-11T01:45:45.409 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.406+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.409 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.406+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1363650974}) 2024-09-11T01:45:45.409 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.406+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1484536636}) 2024-09-11T01:45:45.409 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.406+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.409 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.406+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1363650974}) 2024-09-11T01:45:45.409 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.406+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3445368534}) 2024-09-11T01:45:45.411 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.407+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.411 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.407+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=347710786}) 2024-09-11T01:45:45.411 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.407+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1661450029}) 2024-09-11T01:45:45.411 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.407+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.411 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.407+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=347710786}) 2024-09-11T01:45:45.411 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.407+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=315868254}) 2024-09-11T01:45:45.411 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.408+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.411 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.408+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=542543944}) 2024-09-11T01:45:45.411 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.408+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4236651113}) 2024-09-11T01:45:45.411 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.408+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.412 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.408+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=542543944}) 2024-09-11T01:45:45.412 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.408+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1025141791}) 2024-09-11T01:45:45.412 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.409+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.412 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.409+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1719622403}) 2024-09-11T01:45:45.412 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.409+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3509264389}) 2024-09-11T01:45:45.412 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.409+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.412 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.409+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1719622403}) 2024-09-11T01:45:45.412 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.409+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4166292279}) 2024-09-11T01:45:45.412 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.410+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.412 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.410+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=763074200}) 2024-09-11T01:45:45.412 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.410+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=648573846}) 2024-09-11T01:45:45.412 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.410+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.412 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.410+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=763074200}) 2024-09-11T01:45:45.413 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.410+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3242505720}) 2024-09-11T01:45:45.414 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.410+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.414 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.410+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1073711091}) 2024-09-11T01:45:45.414 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.410+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=189505607}) 2024-09-11T01:45:45.414 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.410+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.414 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.410+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1073711091}) 2024-09-11T01:45:45.414 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.410+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=459624049}) 2024-09-11T01:45:45.414 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.411+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.415 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.411+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3383138553}) 2024-09-11T01:45:45.415 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.411+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4217796629}) 2024-09-11T01:45:45.415 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.411+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.415 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.411+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3383138553}) 2024-09-11T01:45:45.415 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.411+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3020217430}) 2024-09-11T01:45:45.415 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.412+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.415 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.412+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1641475286}) 2024-09-11T01:45:45.415 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.412+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2145275761}) 2024-09-11T01:45:45.415 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.412+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.415 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.412+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1641475286}) 2024-09-11T01:45:45.415 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.412+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2848341355}) 2024-09-11T01:45:45.416 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:45:45.418 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.413+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.418 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.413+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4172428999}) 2024-09-11T01:45:45.418 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.413+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2008485613}) 2024-09-11T01:45:45.418 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.413+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.418 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.413+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4172428999}) 2024-09-11T01:45:45.419 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.413+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2524962128}) 2024-09-11T01:45:45.419 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.413+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.419 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.413+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=79796422}) 2024-09-11T01:45:45.419 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.413+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2363884296}) 2024-09-11T01:45:45.419 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.413+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.419 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.413+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=79796422}) 2024-09-11T01:45:45.419 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.413+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3451478515}) 2024-09-11T01:45:45.419 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.414+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.419 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.414+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1971327284}) 2024-09-11T01:45:45.419 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.414+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2578907044}) 2024-09-11T01:45:45.419 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.414+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.419 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.414+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1971327284}) 2024-09-11T01:45:45.419 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.414+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2014124199}) 2024-09-11T01:45:45.420 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.415+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.420 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.415+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2317400973}) 2024-09-11T01:45:45.420 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.415+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3811050427}) 2024-09-11T01:45:45.420 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.415+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.420 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.415+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2317400973}) 2024-09-11T01:45:45.420 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.415+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1206744431}) 2024-09-11T01:45:45.420 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.416+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.420 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.416+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=325338200}) 2024-09-11T01:45:45.420 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.416+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3467884980}) 2024-09-11T01:45:45.420 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.416+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.420 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.416+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=325338200}) 2024-09-11T01:45:45.420 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.416+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1116442820}) 2024-09-11T01:45:45.426 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.424+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.426 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.424+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1512310935}) 2024-09-11T01:45:45.426 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.424+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3831117788}) 2024-09-11T01:45:45.426 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.424+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.426 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.424+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1512310935}) 2024-09-11T01:45:45.426 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.424+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1275303743}) 2024-09-11T01:45:45.756 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.754+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.756 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.754+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3773641377}) 2024-09-11T01:45:45.756 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.754+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3555900490}) 2024-09-11T01:45:45.756 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.754+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.756 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.754+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3773641377}) 2024-09-11T01:45:45.756 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.754+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1099983934}) 2024-09-11T01:45:45.924 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.923+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.924 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.923+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2115142282}) 2024-09-11T01:45:45.925 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.923+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4033062498}) 2024-09-11T01:45:45.925 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.923+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.925 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.923+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2115142282}) 2024-09-11T01:45:45.925 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.923+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2395571179}) 2024-09-11T01:45:45.926 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.924+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.926 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.924+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3951845096}) 2024-09-11T01:45:45.926 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.924+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=144185273}) 2024-09-11T01:45:45.926 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.924+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.926 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.924+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3951845096}) 2024-09-11T01:45:45.926 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.924+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=612119558}) 2024-09-11T01:45:45.927 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.925+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.927 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.925+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=331281974}) 2024-09-11T01:45:45.927 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.925+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3014287206}) 2024-09-11T01:45:45.927 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.925+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.928 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.925+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=331281974}) 2024-09-11T01:45:45.928 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.925+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=707719767}) 2024-09-11T01:45:45.928 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.926+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.928 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.926+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3521208223}) 2024-09-11T01:45:45.928 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.926+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1986741711}) 2024-09-11T01:45:45.928 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.926+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.928 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.926+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3521208223}) 2024-09-11T01:45:45.928 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.926+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=304187173}) 2024-09-11T01:45:45.930 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.926+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.930 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.926+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1175514201}) 2024-09-11T01:45:45.930 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.926+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=276940536}) 2024-09-11T01:45:45.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.926+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.926+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1175514201}) 2024-09-11T01:45:45.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.926+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1086364870}) 2024-09-11T01:45:45.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.927+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.927+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1920938775}) 2024-09-11T01:45:45.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.927+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1627920375}) 2024-09-11T01:45:45.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.927+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.927+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1920938775}) 2024-09-11T01:45:45.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.927+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=516536191}) 2024-09-11T01:45:45.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.928+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.928+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2467723435}) 2024-09-11T01:45:45.932 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.928+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3634090209}) 2024-09-11T01:45:45.932 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.928+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.932 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.928+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2467723435}) 2024-09-11T01:45:45.932 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.928+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3945705551}) 2024-09-11T01:45:45.932 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.929+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.932 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.929+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2160595522}) 2024-09-11T01:45:45.932 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.929+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=536669969}) 2024-09-11T01:45:45.932 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.929+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.932 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.929+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2160595522}) 2024-09-11T01:45:45.933 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.929+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1342242912}) 2024-09-11T01:45:45.936 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.929+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.937 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.929+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1617810641}) 2024-09-11T01:45:45.937 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.929+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2527456672}) 2024-09-11T01:45:45.937 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.929+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.937 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.929+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1617810641}) 2024-09-11T01:45:45.937 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.929+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3227760501}) 2024-09-11T01:45:45.937 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.930+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.937 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.930+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3055700420}) 2024-09-11T01:45:45.937 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.930+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1148321147}) 2024-09-11T01:45:45.937 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.930+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.937 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.930+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3055700420}) 2024-09-11T01:45:45.937 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.930+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4120527905}) 2024-09-11T01:45:45.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.931+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.931+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4065838453}) 2024-09-11T01:45:45.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.931+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1628495543}) 2024-09-11T01:45:45.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.931+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.931+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4065838453}) 2024-09-11T01:45:45.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.931+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1190929089}) 2024-09-11T01:45:45.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.932+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.932+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3370185936}) 2024-09-11T01:45:45.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.932+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1173096167}) 2024-09-11T01:45:45.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.932+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.932+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3370185936}) 2024-09-11T01:45:45.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.932+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=68938087}) 2024-09-11T01:45:45.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.933+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.933+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2711923582}) 2024-09-11T01:45:45.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.933+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2120493471}) 2024-09-11T01:45:45.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.933+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.933+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2711923582}) 2024-09-11T01:45:45.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.933+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1627697089}) 2024-09-11T01:45:45.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.933+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.933+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3261898571}) 2024-09-11T01:45:45.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.933+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=206130389}) 2024-09-11T01:45:45.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.933+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.933+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3261898571}) 2024-09-11T01:45:45.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.933+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1846788390}) 2024-09-11T01:45:45.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.934+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.940 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.934+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1493217888}) 2024-09-11T01:45:45.940 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.934+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=988003666}) 2024-09-11T01:45:45.940 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.934+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.940 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.934+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1493217888}) 2024-09-11T01:45:45.940 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.934+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1165316523}) 2024-09-11T01:45:45.940 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.935+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.940 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.935+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2057177708}) 2024-09-11T01:45:45.940 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.935+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1070856761}) 2024-09-11T01:45:45.940 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.935+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.940 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.935+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2057177708}) 2024-09-11T01:45:45.940 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.935+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2424343716}) 2024-09-11T01:45:45.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.936+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.936+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3531871088}) 2024-09-11T01:45:45.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.936+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1403830589}) 2024-09-11T01:45:45.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.936+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.936+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3531871088}) 2024-09-11T01:45:45.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.936+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2946249859}) 2024-09-11T01:45:45.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.936+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.936+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3719644504}) 2024-09-11T01:45:45.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.936+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4198095288}) 2024-09-11T01:45:45.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.936+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.936+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3719644504}) 2024-09-11T01:45:45.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.936+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1150238938}) 2024-09-11T01:45:45.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.937+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.937+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761272574}) 2024-09-11T01:45:45.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.937+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1441986063}) 2024-09-11T01:45:45.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.937+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.937+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761272574}) 2024-09-11T01:45:45.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.937+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=415285963}) 2024-09-11T01:45:45.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.938+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.938+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450289090}) 2024-09-11T01:45:45.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.938+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=622449603}) 2024-09-11T01:45:45.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.938+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.938+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450289090}) 2024-09-11T01:45:45.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.938+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4051426373}) 2024-09-11T01:45:45.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.939+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.939+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2114295885}) 2024-09-11T01:45:45.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.939+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3638309363}) 2024-09-11T01:45:45.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.939+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.939+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2114295885}) 2024-09-11T01:45:45.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.939+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3046475121}) 2024-09-11T01:45:45.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.940+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.940+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2213702882}) 2024-09-11T01:45:45.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.940+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1825053280}) 2024-09-11T01:45:45.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.940+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.940+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2213702882}) 2024-09-11T01:45:45.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.940+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1093326978}) 2024-09-11T01:45:45.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.941+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.941+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1097190951}) 2024-09-11T01:45:45.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.941+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3787424266}) 2024-09-11T01:45:45.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.941+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.941+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1097190951}) 2024-09-11T01:45:45.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.941+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=628567343}) 2024-09-11T01:45:45.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.942+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.942+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3859562322}) 2024-09-11T01:45:45.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.942+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1766328557}) 2024-09-11T01:45:45.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.942+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.942+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3859562322}) 2024-09-11T01:45:45.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.942+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1782231751}) 2024-09-11T01:45:45.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.942+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.942+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=235489069}) 2024-09-11T01:45:45.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.942+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3084153134}) 2024-09-11T01:45:45.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.942+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.942+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=235489069}) 2024-09-11T01:45:45.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.942+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3559837435}) 2024-09-11T01:45:45.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.943+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.943+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2404411440}) 2024-09-11T01:45:45.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.943+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=578321363}) 2024-09-11T01:45:45.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.943+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.943+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2404411440}) 2024-09-11T01:45:45.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.943+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3950200709}) 2024-09-11T01:45:45.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.944+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.944+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1996980143}) 2024-09-11T01:45:45.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.944+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2287539144}) 2024-09-11T01:45:45.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.944+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.944+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1996980143}) 2024-09-11T01:45:45.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.944+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2027098681}) 2024-09-11T01:45:45.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.945+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.945+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3373347102}) 2024-09-11T01:45:45.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.945+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3841708208}) 2024-09-11T01:45:45.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.945+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.945+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3373347102}) 2024-09-11T01:45:45.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.945+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1992766863}) 2024-09-11T01:45:45.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.946+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.946+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1146416689}) 2024-09-11T01:45:45.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.946+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2227285578}) 2024-09-11T01:45:45.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.946+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.946+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1146416689}) 2024-09-11T01:45:45.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.946+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=942839580}) 2024-09-11T01:45:45.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.947+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.947+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2737195266}) 2024-09-11T01:45:45.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.947+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1547374242}) 2024-09-11T01:45:45.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.947+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.947+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2737195266}) 2024-09-11T01:45:45.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.947+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=500772190}) 2024-09-11T01:45:45.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.950+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.950+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=597073020}) 2024-09-11T01:45:45.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.950+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1143903473}) 2024-09-11T01:45:45.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.950+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.950+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=597073020}) 2024-09-11T01:45:45.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.950+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=231552791}) 2024-09-11T01:45:45.956 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.953+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.956 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.953+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1870182961}) 2024-09-11T01:45:45.956 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.953+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2922024941}) 2024-09-11T01:45:45.956 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.953+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.956 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.953+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1870182961}) 2024-09-11T01:45:45.956 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.953+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3878845363}) 2024-09-11T01:45:45.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.958+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.958+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1914287388}) 2024-09-11T01:45:45.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.958+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=706465475}) 2024-09-11T01:45:45.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.958+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.958+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1914287388}) 2024-09-11T01:45:45.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.958+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3465254021}) 2024-09-11T01:45:45.960 INFO:tasks.rados.rados.0.smithi071.stdout:2518: done (1 left) 2024-09-11T01:45:45.961 INFO:tasks.rados.rados.0.smithi071.stdout:2519: snap_create 2024-09-11T01:45:45.961 INFO:tasks.rados.rados.0.smithi071.stdout:2517: finishing rollback tid 0 to smithi07134777-14 2024-09-11T01:45:45.961 INFO:tasks.rados.rados.0.smithi071.stdout:2517: finishing rollback tid 1 to smithi07134777-14 2024-09-11T01:45:45.961 INFO:tasks.rados.rados.0.smithi071.stdout:2517: finishing rollback tid 2 to smithi07134777-14 2024-09-11T01:45:45.961 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 1626 (ObjNum 865 snap 266 seq_num 865) dirty exists 2024-09-11T01:45:45.962 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.959+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.962 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.959+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=85,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=2} crc {logm=3686714747,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=315698294}) 2024-09-11T01:45:45.962 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.959+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=82,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=4250584674,mdsmap=3140514009,mgr=1335250786,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=3222362401}) 2024-09-11T01:45:45.962 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.959+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.962 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.959+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=85,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=2} crc {logm=3686714747,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=315698294}) 2024-09-11T01:45:45.962 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.959+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=79,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8} crc {logm=2725915999,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2403429723}) 2024-09-11T01:45:45.963 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.960+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.963 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.960+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=10,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=61} crc {mgrstat=2685926183,mon_config_key=3330371543,monmap=2933275274,nvmeofgw=598337686,osd_metadata=1404192910,osd_pg_creating=1466293538,osd_snap=3882465300}) 2024-09-11T01:45:45.963 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.960+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=9,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=61} crc {mgrstat=208565962,mon_config_key=3330371543,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=1466293538,osd_snap=2275324772}) 2024-09-11T01:45:45.963 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.960+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.963 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.960+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=10,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=61} crc {mgrstat=2685926183,mon_config_key=3330371543,monmap=2933275274,nvmeofgw=598337686,osd_metadata=1404192910,osd_pg_creating=1466293538,osd_snap=3882465300}) 2024-09-11T01:45:45.963 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.960+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=9,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=62} crc {mgrstat=1151288872,mon_config_key=1702164636,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=1466293538,osd_snap=2178579446}) 2024-09-11T01:45:45.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.961+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.961+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=55,osdmap=45} crc {osd_snap=3617754431,osdmap=4175106480}) 2024-09-11T01:45:45.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.961+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=52,osdmap=48} crc {osd_snap=491548663,osdmap=529858079}) 2024-09-11T01:45:45.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.961+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.961+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=55,osdmap=45} crc {osd_snap=3617754431,osdmap=4175106480}) 2024-09-11T01:45:45.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.961+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=56,osdmap=44} crc {osd_snap=1055310878,osdmap=3035352928}) 2024-09-11T01:45:45.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.962+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.962+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3771180530}) 2024-09-11T01:45:45.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.962+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2009030828}) 2024-09-11T01:45:45.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.962+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.962+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3771180530}) 2024-09-11T01:45:45.966 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.962+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2003013767}) 2024-09-11T01:45:45.966 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.963+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.966 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.963+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3108151924}) 2024-09-11T01:45:45.966 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.963+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3625185928}) 2024-09-11T01:45:45.966 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.963+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.966 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.963+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3108151924}) 2024-09-11T01:45:45.966 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.963+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2272312060}) 2024-09-11T01:45:45.967 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.965+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.967 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.965+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2897171050}) 2024-09-11T01:45:45.967 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.965+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3891835027}) 2024-09-11T01:45:45.967 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.965+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.967 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.965+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2897171050}) 2024-09-11T01:45:45.968 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.965+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2212291349}) 2024-09-11T01:45:45.970 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.969+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.970 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.969+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=35} crc {osdmap=103312633}) 2024-09-11T01:45:45.971 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.969+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=33} crc {osdmap=39277097}) 2024-09-11T01:45:45.971 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.969+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:45:45.971 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.969+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=35} crc {osdmap=103312633}) 2024-09-11T01:45:45.971 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:45:45.969+0000 7fb62d99b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=32} crc {osdmap=3511934032}) 2024-09-11T01:45:46.417 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:45:46.417 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:45:46.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.527+0000 7fa81b49b640 10 monclient: get_monmap_and_config 2024-09-11T01:45:46.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.527+0000 7fa81b49b640 10 monclient: build_initial_monmap 2024-09-11T01:45:46.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.527+0000 7fa81b49b640 10 monclient: monmap: 2024-09-11T01:45:46.529 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:46.529 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:46.529 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:45:46.528155+0000 2024-09-11T01:45:46.529 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:45:46.528155+0000 2024-09-11T01:45:46.529 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:46.529 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:46.529 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:45:46.529 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:45:46.529 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:45:46.529 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:45:46.530 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:45:46.530 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:46.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.528+0000 7fa81b49b640 1 Processor -- start 2024-09-11T01:45:46.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.528+0000 7fa81b49b640 1 -- start start 2024-09-11T01:45:46.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.528+0000 7fa81b49b640 10 monclient: init 2024-09-11T01:45:46.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.528+0000 7fa81b49b640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:46.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.528+0000 7fa81b49b640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-11T01:45:46.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.528+0000 7fa81b49b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8140a2e90 0x7fa81406dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:46.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.528+0000 7fa81b49b640 10 monclient(hunting): picked mon.noname-b con 0x7fa8140a2e90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:46.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.528+0000 7fa81b49b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa8140a3270 addr v1:172.21.15.71:6789/0 2024-09-11T01:45:46.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.528+0000 7fa81b49b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa81413d490 0x7fa81413f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:46.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.528+0000 7fa81b49b640 10 monclient(hunting): picked mon.noname-a con 0x7fa81413d490 addr v2:172.21.15.12:3300/0 2024-09-11T01:45:46.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.528+0000 7fa81b49b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8140a46c0 con 0x7fa8140a3270 2024-09-11T01:45:46.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.528+0000 7fa81b49b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:46.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.528+0000 7fa81b49b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fa8140a4900 con 0x7fa81413d490 2024-09-11T01:45:46.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.528+0000 7fa81b49b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:46.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.528+0000 7fa81b49b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa81406ee50 con 0x7fa8140a2e90 2024-09-11T01:45:46.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.528+0000 7fa81b49b640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:46.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.528+0000 7fa81b49b640 10 monclient(hunting): authenticate will time out at 1752.970963s 2024-09-11T01:45:46.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.529+0000 7fa819237640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8140a2e90 0x7fa81406dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:46.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.529+0000 7fa819a38640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa81413d490 0x7fa81413f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:46.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.529+0000 7fa819237640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8140a2e90 0x7fa81406dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51966/0 (socket says 172.21.15.12:51966) 2024-09-11T01:45:46.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.529+0000 7fa819a38640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7fa81413d490 msgr2=0x7fa81413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:45:46.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.529+0000 7fa819237640 1 -- 172.21.15.12:0/4114725858 learned_addr learned my addr 172.21.15.12:0/4114725858 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:46.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.529+0000 7fa819a38640 1 -- 172.21.15.12:0/4114725858 >> v2:172.21.15.12:3300/0 conn(0x7fa81413d490 msgr2=0x7fa81413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:45:46.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.529+0000 7fa819237640 10 monclient(hunting): get_auth_request con 0x7fa8140a2e90 auth_method 0 2024-09-11T01:45:46.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.529+0000 7fa819237640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:46.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.529+0000 7fa819237640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:46.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.529+0000 7fa819a38640 1 --2- 172.21.15.12:0/4114725858 >> v2:172.21.15.12:3300/0 conn(0x7fa81413d490 0x7fa81413f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:45:46.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.529+0000 7fa819237640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:46.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.529+0000 7fa819a38640 1 --2- 172.21.15.12:0/4114725858 >> v2:172.21.15.12:3300/0 conn(0x7fa81413d490 0x7fa81413f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:46.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.529+0000 7fa819237640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:46.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.529+0000 7fa819237640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:46.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.529+0000 7fa819237640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:46.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.529+0000 7fa819237640 10 monclient(hunting): handle_auth_done global_id 184118 payload 274 2024-09-11T01:45:46.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.529+0000 7fa819237640 1 -- 172.21.15.12:0/4114725858 >> v2:172.21.15.12:3300/0 conn(0x7fa81413d490 msgr2=0x7fa81413f880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:45:46.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.529+0000 7fa819237640 1 --2- 172.21.15.12:0/4114725858 >> v2:172.21.15.12:3300/0 conn(0x7fa81413d490 0x7fa81413f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:46.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.529+0000 7fa819237640 1 -- 172.21.15.12:0/4114725858 >> v1:172.21.15.71:6789/0 conn(0x7fa8140a3270 legacy=0x7fa8140a3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:46.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.529+0000 7fa819237640 10 monclient: _finish_hunting 0 2024-09-11T01:45:46.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.529+0000 7fa819237640 1 monclient: found mon.noname-b 2024-09-11T01:45:46.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.529+0000 7fa819237640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:46.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.529+0000 7fa819237640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:45:46.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.529+0000 7fa819237640 1 -- 172.21.15.12:0/4114725858 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa81406eb70 con 0x7fa8140a2e90 2024-09-11T01:45:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.532+0000 7fa819237640 1 -- 172.21.15.12:0/4114725858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8140a2e90 msgr2=0x7fa81406dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.532+0000 7fa819237640 1 --2- 172.21.15.12:0/4114725858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8140a2e90 0x7fa81406dc60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa8140a4a80 tx=0x7fa81002a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9068859eba991afc server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.532+0000 7fa80bfff640 1 -- 172.21.15.12:0/4114725858 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa810037070 con 0x7fa8140a2e90 2024-09-11T01:45:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.532+0000 7fa80bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.532+0000 7fa80bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:45:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.532+0000 7fa80bfff640 10 monclient: dump: 2024-09-11T01:45:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:46.542 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.532+0000 7fa80bfff640 10 monclient: _finish_auth 0 2024-09-11T01:45:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.532+0000 7fa80bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:45:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.532+0000 7fa80bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.532+0000 7fa81b49b640 5 monclient: authenticate success, global_id 184118 2024-09-11T01:45:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.532+0000 7fa81b49b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:45:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.532+0000 7fa80bfff640 1 -- 172.21.15.12:0/4114725858 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa810032640 con 0x7fa8140a2e90 2024-09-11T01:45:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.532+0000 7fa80bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.532+0000 7fa81b49b640 10 monclient: get_monmap_and_config success 2024-09-11T01:45:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.532+0000 7fa81b49b640 4 set_mon_vals no callback set 2024-09-11T01:45:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.532+0000 7fa80bfff640 1 -- 172.21.15.12:0/4114725858 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa810032a10 con 0x7fa8140a2e90 2024-09-11T01:45:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.532+0000 7fa80bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:46.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.532+0000 7fa80bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:45:46.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.532+0000 7fa80bfff640 10 monclient: dump: 2024-09-11T01:45:46.543 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:46.543 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:46.543 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:46.543 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:46.543 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:46.543 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:46.543 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:46.543 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:46.543 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:46.543 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:46.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.532+0000 7fa81b49b640 10 monclient: shutdown 2024-09-11T01:45:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.532+0000 7fa81b49b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.532+0000 7fa81b49b640 1 -- 172.21.15.12:0/4114725858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8140a2e90 msgr2=0x7fa81406dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.532+0000 7fa81b49b640 1 --2- 172.21.15.12:0/4114725858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8140a2e90 0x7fa81406dc60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa8140a4a80 tx=0x7fa81002a080 comp rx=0 tx=0).stop 2024-09-11T01:45:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.533+0000 7fa81b49b640 1 -- 172.21.15.12:0/4114725858 shutdown_connections 2024-09-11T01:45:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.533+0000 7fa81b49b640 1 --2- 172.21.15.12:0/4114725858 >> v2:172.21.15.12:3300/0 conn(0x7fa81413d490 0x7fa81413f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.533+0000 7fa81b49b640 1 --2- 172.21.15.12:0/4114725858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8140a2e90 0x7fa81406dc60 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.533+0000 7fa81b49b640 1 -- 172.21.15.12:0/4114725858 >> 172.21.15.12:0/4114725858 conn(0x7fa8140a8490 msgr2=0x7fa8140a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.533+0000 7fa81b49b640 1 -- 172.21.15.12:0/4114725858 shutdown_connections 2024-09-11T01:45:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.533+0000 7fa81b49b640 1 -- 172.21.15.12:0/4114725858 wait complete. 2024-09-11T01:45:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.533+0000 7fa81b49b640 10 monclient: build_initial_monmap 2024-09-11T01:45:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.533+0000 7fa81b49b640 10 monclient: monmap: 2024-09-11T01:45:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:45:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:45:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:45:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:45:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:45:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:45:46.545 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa81b49b640 1 Processor -- start 2024-09-11T01:45:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa81b49b640 1 -- start start 2024-09-11T01:45:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa81b49b640 10 monclient: init 2024-09-11T01:45:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa81b49b640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa81b49b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:45:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa81b49b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8140a2e90 0x7fa81409e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:46.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa81b49b640 10 monclient(hunting): picked mon.noname-b con 0x7fa8140a2e90 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:45:46.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa81b49b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8140a3270 0x7fa81409ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:46.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa81b49b640 10 monclient(hunting): picked mon.noname-c con 0x7fa8140a3270 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:46.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa81b49b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa81413d490 0x7fa8140a0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:46.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa81b49b640 10 monclient(hunting): picked mon.noname-a con 0x7fa81413d490 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:45:46.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa81b49b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:46.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa81b49b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa81406fd20 con 0x7fa81413d490 2024-09-11T01:45:46.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa81b49b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:46.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa81b49b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa81406fba0 con 0x7fa8140a2e90 2024-09-11T01:45:46.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa81b49b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:46.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa81b49b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa81413fdc0 con 0x7fa8140a3270 2024-09-11T01:45:46.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa81b49b640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:46.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa81b49b640 10 monclient(hunting): authenticate will time out at 1752.976664s 2024-09-11T01:45:46.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa818a36640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8140a3270 0x7fa81409ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:46.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa819a38640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa81413d490 0x7fa8140a0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:46.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa818a36640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8140a3270 0x7fa81409ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51982/0 (socket says 172.21.15.12:51982) 2024-09-11T01:45:46.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa819a38640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa81413d490 0x7fa8140a0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47342/0 (socket says 172.21.15.12:47342) 2024-09-11T01:45:46.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa818a36640 1 -- 172.21.15.12:0/3459726460 learned_addr learned my addr 172.21.15.12:0/3459726460 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:46.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa818a36640 10 monclient(hunting): get_auth_request con 0x7fa8140a3270 auth_method 0 2024-09-11T01:45:46.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa818a36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:46.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa818a36640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:46.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa818a36640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:46.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa819a38640 10 monclient(hunting): get_auth_request con 0x7fa81413d490 auth_method 0 2024-09-11T01:45:46.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa819a38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:46.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa819a38640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:46.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa819a38640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:46.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa818a36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:46.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa818a36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:46.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa818a36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:46.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa819a38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:46.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa819a38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:46.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa819a38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:46.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.534+0000 7fa819237640 1 --2- 172.21.15.12:0/3459726460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8140a2e90 0x7fa81409e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:46.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.535+0000 7fa818a36640 10 monclient(hunting): handle_auth_done global_id 184121 payload 824 2024-09-11T01:45:46.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.535+0000 7fa818a36640 1 -- 172.21.15.12:0/3459726460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8140a2e90 msgr2=0x7fa81409e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:46.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.535+0000 7fa818a36640 1 --2- 172.21.15.12:0/3459726460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8140a2e90 0x7fa81409e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:46.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.535+0000 7fa818a36640 1 -- 172.21.15.12:0/3459726460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa81413d490 msgr2=0x7fa8140a0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:46.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.535+0000 7fa818a36640 1 --2- 172.21.15.12:0/3459726460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa81413d490 0x7fa8140a0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:46.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.535+0000 7fa818a36640 10 monclient: _finish_hunting 0 2024-09-11T01:45:46.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.535+0000 7fa818a36640 1 monclient: found mon.noname-c 2024-09-11T01:45:46.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.535+0000 7fa818a36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:46.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.535+0000 7fa818a36640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:45:46.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.535+0000 7fa818a36640 1 -- 172.21.15.12:0/3459726460 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa81409ef70 con 0x7fa8140a3270 2024-09-11T01:45:46.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.535+0000 7fa819237640 10 monclient: get_auth_request con 0x7fa8140a2e90 auth_method 0 2024-09-11T01:45:46.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.535+0000 7fa819237640 1 --2- 172.21.15.12:0/3459726460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8140a2e90 0x7fa81409e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:45:46.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.537+0000 7fa818a36640 1 -- 172.21.15.12:0/3459726460 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8140a3270 msgr2=0x7fa81409ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:46.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.537+0000 7fa818a36640 1 --2- 172.21.15.12:0/3459726460 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8140a3270 0x7fa81409ff50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa80400c430 tx=0x7fa80400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:46.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.537+0000 7fa809ffb640 1 -- 172.21.15.12:0/3459726460 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8040119a0 con 0x7fa8140a3270 2024-09-11T01:45:46.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.537+0000 7fa809ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:46.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.537+0000 7fa809ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:45:46.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.537+0000 7fa809ffb640 10 monclient: dump: 2024-09-11T01:45:46.550 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:46.550 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:46.550 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:46.550 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:46.550 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:46.550 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:46.551 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:46.551 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:46.551 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:46.551 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:46.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.537+0000 7fa809ffb640 10 monclient: _finish_auth 0 2024-09-11T01:45:46.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.537+0000 7fa809ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:45:46.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.537+0000 7fa809ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:46.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.537+0000 7fa809ffb640 1 -- 172.21.15.12:0/3459726460 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa804017070 con 0x7fa8140a3270 2024-09-11T01:45:46.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.537+0000 7fa809ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:46.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.537+0000 7fa809ffb640 1 -- 172.21.15.12:0/3459726460 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa804011d80 con 0x7fa8140a3270 2024-09-11T01:45:46.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.537+0000 7fa81b49b640 5 monclient: authenticate success, global_id 184121 2024-09-11T01:45:46.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.537+0000 7fa81b49b640 10 monclient: _renew_subs 2024-09-11T01:45:46.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.537+0000 7fa81b49b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:45:46.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.537+0000 7fa81b49b640 1 -- 172.21.15.12:0/3459726460 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8140a2720 con 0x7fa8140a3270 2024-09-11T01:45:46.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.537+0000 7fa80b7fe640 4 set_mon_vals no callback set 2024-09-11T01:45:46.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.537+0000 7fa809ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:46.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.537+0000 7fa809ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:45:46.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.537+0000 7fa809ffb640 10 monclient: dump: 2024-09-11T01:45:46.552 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:46.552 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:46.552 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:46.552 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:46.552 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:46.552 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:46.552 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:46.552 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:46.552 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:46.553 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:46.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.537+0000 7fa81b49b640 10 monclient: _renew_subs 2024-09-11T01:45:46.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.537+0000 7fa81b49b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:45:46.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.537+0000 7fa81b49b640 1 -- 172.21.15.12:0/3459726460 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8140a1e80 con 0x7fa8140a3270 2024-09-11T01:45:46.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.538+0000 7fa809ffb640 1 -- 172.21.15.12:0/3459726460 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fa80401c460 con 0x7fa8140a3270 2024-09-11T01:45:46.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.538+0000 7fa81b49b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:46.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.538+0000 7fa81b49b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:46.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.538+0000 7fa81b49b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:45:46.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.538+0000 7fa81b49b640 1 -- 172.21.15.12:0/3459726460 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa81406dc60 con 0x7fa8140a3270 2024-09-11T01:45:46.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.539+0000 7fa809ffb640 1 --2- 172.21.15.12:0/3459726460 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa7f803e210 0x7fa7f80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:46.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.539+0000 7fa809ffb640 1 -- 172.21.15.12:0/3459726460 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(565..565 src has 317..565) ==== 4815+0+0 (secure 0 0 0) 0x7fa804052c10 con 0x7fa8140a3270 2024-09-11T01:45:46.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.539+0000 7fa819237640 1 --2- 172.21.15.12:0/3459726460 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa7f803e210 0x7fa7f80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:46.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.539+0000 7fa819237640 10 monclient: get_auth_request con 0x7fa7f803e210 auth_method 0 2024-09-11T01:45:46.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.542+0000 7fa819237640 1 --2- 172.21.15.12:0/3459726460 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa7f803e210 0x7fa7f80406d0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa8140a92c0 tx=0x7fa81002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:46.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.544+0000 7fa809ffb640 1 -- 172.21.15.12:0/3459726460 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa8040515f0 con 0x7fa8140a3270 2024-09-11T01:45:46.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.544+0000 7fa809ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:46.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.544+0000 7fa809ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:45:46.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.799+0000 7fa81b49b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:45:46.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.799+0000 7fa81b49b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:45:46.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.799+0000 7fa81b49b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:45:46.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.799+0000 7fa81b49b640 1 -- 172.21.15.12:0/3459726460 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa8140698e0 con 0x7fa8140a3270 2024-09-11T01:45:46.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.800+0000 7fa809ffb640 1 -- 172.21.15.12:0/3459726460 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fa804012470 con 0x7fa8140a3270 2024-09-11T01:45:46.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.800+0000 7fa809ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:45:46.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.800+0000 7fa809ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:45:46.802 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":154,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:45:46.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.802+0000 7fa81b49b640 1 -- 172.21.15.12:0/3459726460 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa7f803e210 msgr2=0x7fa7f80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:46.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.802+0000 7fa81b49b640 1 --2- 172.21.15.12:0/3459726460 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa7f803e210 0x7fa7f80406d0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa8140a92c0 tx=0x7fa81002e000 comp rx=0 tx=0).stop 2024-09-11T01:45:46.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.802+0000 7fa81b49b640 10 monclient: shutdown 2024-09-11T01:45:46.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.802+0000 7fa81b49b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:46.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.802+0000 7fa81b49b640 1 -- 172.21.15.12:0/3459726460 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8140a3270 msgr2=0x7fa81409ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:46.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.802+0000 7fa81b49b640 1 --2- 172.21.15.12:0/3459726460 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8140a3270 0x7fa81409ff50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa80400c430 tx=0x7fa80400c8f0 comp rx=0 tx=0).stop 2024-09-11T01:45:46.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.802+0000 7fa81b49b640 1 -- 172.21.15.12:0/3459726460 shutdown_connections 2024-09-11T01:45:46.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.802+0000 7fa81b49b640 1 --2- 172.21.15.12:0/3459726460 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa7f803e210 0x7fa7f80406d0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:46.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.802+0000 7fa81b49b640 1 --2- 172.21.15.12:0/3459726460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa81413d490 0x7fa8140a0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:46.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.802+0000 7fa81b49b640 1 --2- 172.21.15.12:0/3459726460 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa8140a3270 0x7fa81409ff50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:46.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.802+0000 7fa81b49b640 1 --2- 172.21.15.12:0/3459726460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8140a2e90 0x7fa81409e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:46.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.802+0000 7fa81b49b640 1 -- 172.21.15.12:0/3459726460 >> 172.21.15.12:0/3459726460 conn(0x7fa8140a8490 msgr2=0x7fa81413c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:46.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.802+0000 7fa81b49b640 1 -- 172.21.15.12:0/3459726460 shutdown_connections 2024-09-11T01:45:46.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.802+0000 7fa81b49b640 1 -- 172.21.15.12:0/3459726460 wait complete. 2024-09-11T01:45:46.814 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:45:46.814 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:45:46.814 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:45:46.814 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:45:46.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.914+0000 7fb657552640 10 monclient: get_monmap_and_config 2024-09-11T01:45:46.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.914+0000 7fb657552640 10 monclient: build_initial_monmap 2024-09-11T01:45:46.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.914+0000 7fb657552640 10 monclient: monmap: 2024-09-11T01:45:46.916 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:45:46.915415+0000 2024-09-11T01:45:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:45:46.915415+0000 2024-09-11T01:45:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:45:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:45:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:45:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:45:46.917 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:45:46.917 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:46.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.915+0000 7fb657552640 1 Processor -- start 2024-09-11T01:45:46.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.915+0000 7fb657552640 1 -- start start 2024-09-11T01:45:46.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.915+0000 7fb657552640 10 monclient: init 2024-09-11T01:45:46.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.915+0000 7fb657552640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:46.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.915+0000 7fb657552640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-11T01:45:46.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.915+0000 7fb657552640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb650074180 addr v1:172.21.15.12:6789/0 2024-09-11T01:45:46.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.915+0000 7fb657552640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb6500a06d0 0x7fb65013cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:46.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.915+0000 7fb657552640 10 monclient(hunting): picked mon.noname-c con 0x7fb6500a06d0 addr v2:172.21.15.71:3300/0 2024-09-11T01:45:46.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.915+0000 7fb657552640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb650134ac0 0x7fb65013f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:46.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb657552640 10 monclient(hunting): picked mon.noname-a con 0x7fb650134ac0 addr v2:172.21.15.12:3300/0 2024-09-11T01:45:46.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb657552640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb650070160 con 0x7fb650074180 2024-09-11T01:45:46.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb657552640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:46.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb657552640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fb650135560 con 0x7fb650134ac0 2024-09-11T01:45:46.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb657552640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:46.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb657552640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fb6500a0490 con 0x7fb6500a06d0 2024-09-11T01:45:46.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb657552640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:46.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb657552640 10 monclient(hunting): authenticate will time out at 1753.358056s 2024-09-11T01:45:46.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb655aef640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb650134ac0 0x7fb65013f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:46.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb6552ee640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb650074180 0x7fb650074560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42920/0 (socket says 172.21.15.12:42920) 2024-09-11T01:45:46.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb6552ee640 1 -- 172.21.15.12:0/1407948284 learned_addr learned my addr 172.21.15.12:0/1407948284 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:45:46.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb655aef640 10 monclient(hunting): get_auth_request con 0x7fb650134ac0 auth_method 0 2024-09-11T01:45:46.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb655aef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:46.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb655aef640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:46.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb655aef640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:46.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb655aef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:46.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb655aef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:46.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb655aef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:46.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb654aed640 1 --2- 172.21.15.12:0/1407948284 >> v2:172.21.15.71:3300/0 conn(0x7fb6500a06d0 0x7fb65013cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:46.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb654aed640 10 monclient(hunting): get_auth_request con 0x7fb6500a06d0 auth_method 0 2024-09-11T01:45:46.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb654aed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:46.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb654aed640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:46.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb654aed640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:46.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb654aed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb654aed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb654aed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb655aef640 10 monclient(hunting): handle_auth_done global_id 174228 payload 274 2024-09-11T01:45:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb655aef640 1 -- 172.21.15.12:0/1407948284 >> v2:172.21.15.71:3300/0 conn(0x7fb6500a06d0 msgr2=0x7fb65013cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb655aef640 1 --2- 172.21.15.12:0/1407948284 >> v2:172.21.15.71:3300/0 conn(0x7fb6500a06d0 0x7fb65013cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb655aef640 1 -- 172.21.15.12:0/1407948284 >> v1:172.21.15.12:6789/0 conn(0x7fb650074180 legacy=0x7fb650074560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb655aef640 10 monclient: _finish_hunting 0 2024-09-11T01:45:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb655aef640 1 monclient: found mon.noname-a 2024-09-11T01:45:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb655aef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb655aef640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:45:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.916+0000 7fb655aef640 1 -- 172.21.15.12:0/1407948284 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb650070440 con 0x7fb650134ac0 2024-09-11T01:45:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.919+0000 7fb655aef640 1 -- 172.21.15.12:0/1407948284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb650134ac0 msgr2=0x7fb65013f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.919+0000 7fb655aef640 1 --2- 172.21.15.12:0/1407948284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb650134ac0 0x7fb65013f410 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb650070bd0 tx=0x7fb64802a4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=449dc9bb13d84e09 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:46.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.919+0000 7fb647fff640 1 -- 172.21.15.12:0/1407948284 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb648007950 con 0x7fb650134ac0 2024-09-11T01:45:46.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.919+0000 7fb647fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:46.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.919+0000 7fb647fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:45:46.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.919+0000 7fb647fff640 10 monclient: dump: 2024-09-11T01:45:46.930 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:46.930 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:46.930 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:46.931 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:46.931 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:46.931 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:46.931 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:46.931 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:46.931 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:46.931 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:46.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.919+0000 7fb647fff640 10 monclient: _finish_auth 0 2024-09-11T01:45:46.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.919+0000 7fb647fff640 10 monclient: _check_auth_tickets 2024-09-11T01:45:46.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.919+0000 7fb647fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:46.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.919+0000 7fb647fff640 1 -- 172.21.15.12:0/1407948284 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb648007af0 con 0x7fb650134ac0 2024-09-11T01:45:46.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.919+0000 7fb647fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:46.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.919+0000 7fb657552640 5 monclient: authenticate success, global_id 174228 2024-09-11T01:45:46.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.919+0000 7fb647fff640 1 -- 172.21.15.12:0/1407948284 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb648031400 con 0x7fb650134ac0 2024-09-11T01:45:46.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.919+0000 7fb657552640 10 monclient: get_monmap_and_config success 2024-09-11T01:45:46.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.919+0000 7fb647fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:46.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.919+0000 7fb657552640 4 set_mon_vals no callback set 2024-09-11T01:45:46.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.919+0000 7fb647fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:46.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.919+0000 7fb647fff640 10 monclient: dump: 2024-09-11T01:45:46.932 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:46.932 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:46.932 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:46.932 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:46.932 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:46.932 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:46.932 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:46.932 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:46.933 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:46.933 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:46.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.920+0000 7fb657552640 10 monclient: shutdown 2024-09-11T01:45:46.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.920+0000 7fb657552640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:46.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.920+0000 7fb657552640 1 -- 172.21.15.12:0/1407948284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb650134ac0 msgr2=0x7fb65013f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:46.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.920+0000 7fb657552640 1 --2- 172.21.15.12:0/1407948284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb650134ac0 0x7fb65013f410 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb650070bd0 tx=0x7fb64802a4b0 comp rx=0 tx=0).stop 2024-09-11T01:45:46.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.920+0000 7fb657552640 1 -- 172.21.15.12:0/1407948284 shutdown_connections 2024-09-11T01:45:46.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.920+0000 7fb657552640 1 --2- 172.21.15.12:0/1407948284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb650134ac0 0x7fb65013f410 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:46.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.920+0000 7fb657552640 1 --2- 172.21.15.12:0/1407948284 >> v2:172.21.15.71:3300/0 conn(0x7fb6500a06d0 0x7fb65013cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:46.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.920+0000 7fb657552640 1 -- 172.21.15.12:0/1407948284 >> 172.21.15.12:0/1407948284 conn(0x7fb6500a36c0 msgr2=0x7fb6500a3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:46.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.920+0000 7fb657552640 1 -- 172.21.15.12:0/1407948284 shutdown_connections 2024-09-11T01:45:46.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.920+0000 7fb657552640 1 -- 172.21.15.12:0/1407948284 wait complete. 2024-09-11T01:45:46.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.920+0000 7fb657552640 10 monclient: build_initial_monmap 2024-09-11T01:45:46.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.920+0000 7fb657552640 10 monclient: monmap: 2024-09-11T01:45:46.934 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:46.934 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:45:46.934 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:45:46.934 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:45:46.934 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:46.934 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:46.934 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:45:46.934 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:45:46.934 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:45:46.934 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:46.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb657552640 1 Processor -- start 2024-09-11T01:45:46.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb657552640 1 -- start start 2024-09-11T01:45:46.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb657552640 10 monclient: init 2024-09-11T01:45:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb657552640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb657552640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:45:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb657552640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb650074180 0x7fb65013a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb657552640 10 monclient(hunting): picked mon.noname-b con 0x7fb650074180 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:45:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb657552640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb6500a06d0 0x7fb65013a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb657552640 10 monclient(hunting): picked mon.noname-c con 0x7fb6500a06d0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb657552640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb650134ac0 0x7fb65013ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb657552640 10 monclient(hunting): picked mon.noname-a con 0x7fb650134ac0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:45:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb657552640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb657552640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb65013fad0 con 0x7fb650134ac0 2024-09-11T01:45:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb657552640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:46.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb657552640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb65013f950 con 0x7fb650074180 2024-09-11T01:45:46.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb657552640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:46.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb657552640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb65013fc50 con 0x7fb6500a06d0 2024-09-11T01:45:46.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb657552640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:46.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb657552640 10 monclient(hunting): authenticate will time out at 1753.363763s 2024-09-11T01:45:46.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb655aef640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb650134ac0 0x7fb65013ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:46.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb654aed640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb6500a06d0 0x7fb65013a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:46.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb655aef640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb650134ac0 0x7fb65013ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47368/0 (socket says 172.21.15.12:47368) 2024-09-11T01:45:46.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb655aef640 1 -- 172.21.15.12:0/1071483068 learned_addr learned my addr 172.21.15.12:0/1071483068 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:46.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb655aef640 10 monclient(hunting): get_auth_request con 0x7fb650134ac0 auth_method 0 2024-09-11T01:45:46.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb655aef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:46.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb655aef640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb655aef640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb654aed640 10 monclient(hunting): get_auth_request con 0x7fb6500a06d0 auth_method 0 2024-09-11T01:45:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb654aed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb654aed640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.921+0000 7fb654aed640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.922+0000 7fb654aed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.922+0000 7fb654aed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.922+0000 7fb654aed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.922+0000 7fb655aef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.922+0000 7fb655aef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.922+0000 7fb655aef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.922+0000 7fb6552ee640 1 --2- 172.21.15.12:0/1071483068 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb650074180 0x7fb65013a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.922+0000 7fb6552ee640 10 monclient(hunting): get_auth_request con 0x7fb650074180 auth_method 0 2024-09-11T01:45:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.922+0000 7fb6552ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.922+0000 7fb6552ee640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.922+0000 7fb6552ee640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.922+0000 7fb6552ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.922+0000 7fb6552ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.922+0000 7fb6552ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.922+0000 7fb654aed640 10 monclient(hunting): handle_auth_done global_id 184124 payload 824 2024-09-11T01:45:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.922+0000 7fb654aed640 1 -- 172.21.15.12:0/1071483068 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb650074180 msgr2=0x7fb65013a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.922+0000 7fb654aed640 1 --2- 172.21.15.12:0/1071483068 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb650074180 0x7fb65013a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.922+0000 7fb654aed640 1 -- 172.21.15.12:0/1071483068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb650134ac0 msgr2=0x7fb65013ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.922+0000 7fb654aed640 1 --2- 172.21.15.12:0/1071483068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb650134ac0 0x7fb65013ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.922+0000 7fb654aed640 10 monclient: _finish_hunting 0 2024-09-11T01:45:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.922+0000 7fb654aed640 1 monclient: found mon.noname-c 2024-09-11T01:45:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.922+0000 7fb654aed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.922+0000 7fb654aed640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:45:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.922+0000 7fb654aed640 1 -- 172.21.15.12:0/1071483068 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb65013bec0 con 0x7fb6500a06d0 2024-09-11T01:45:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.922+0000 7fb655aef640 1 --2- 172.21.15.12:0/1071483068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb650134ac0 0x7fb65013ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:45:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.924+0000 7fb654aed640 1 -- 172.21.15.12:0/1071483068 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb6500a06d0 msgr2=0x7fb65013a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.924+0000 7fb654aed640 1 --2- 172.21.15.12:0/1071483068 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb6500a06d0 0x7fb65013a940 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb64000e850 tx=0x7fb64000ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.924+0000 7fb645ffb640 1 -- 172.21.15.12:0/1071483068 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb640017070 con 0x7fb6500a06d0 2024-09-11T01:45:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.924+0000 7fb645ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.924+0000 7fb645ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:45:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.924+0000 7fb645ffb640 10 monclient: dump: 2024-09-11T01:45:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:46.940 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.924+0000 7fb645ffb640 10 monclient: _finish_auth 0 2024-09-11T01:45:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.924+0000 7fb645ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:45:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.924+0000 7fb645ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.924+0000 7fb645ffb640 1 -- 172.21.15.12:0/1071483068 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb64000cb50 con 0x7fb6500a06d0 2024-09-11T01:45:46.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.924+0000 7fb657552640 5 monclient: authenticate success, global_id 184124 2024-09-11T01:45:46.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.924+0000 7fb657552640 10 monclient: _renew_subs 2024-09-11T01:45:46.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.924+0000 7fb657552640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:45:46.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.924+0000 7fb657552640 1 -- 172.21.15.12:0/1071483068 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb65013e220 con 0x7fb6500a06d0 2024-09-11T01:45:46.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.924+0000 7fb645ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:46.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.924+0000 7fb645ffb640 1 -- 172.21.15.12:0/1071483068 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb640013470 con 0x7fb6500a06d0 2024-09-11T01:45:46.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.924+0000 7fb645ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:46.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.924+0000 7fb645ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:45:46.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.924+0000 7fb645ffb640 10 monclient: dump: 2024-09-11T01:45:46.941 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:46.941 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:46.941 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:46.941 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:46.942 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:46.942 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:46.942 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:46.942 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:46.942 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:46.942 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:46.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.924+0000 7fb6477fe640 4 set_mon_vals no callback set 2024-09-11T01:45:46.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.924+0000 7fb657552640 10 monclient: _renew_subs 2024-09-11T01:45:46.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.924+0000 7fb657552640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:45:46.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.924+0000 7fb657552640 1 -- 172.21.15.12:0/1071483068 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb65013cb30 con 0x7fb6500a06d0 2024-09-11T01:45:46.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.925+0000 7fb657552640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:46.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.925+0000 7fb657552640 1 --2- 172.21.15.12:0/1071483068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6501ccd40 0x7fb6501cd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:46.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.925+0000 7fb657552640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:46.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.925+0000 7fb657552640 1 -- 172.21.15.12:0/1071483068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb650134420 con 0x7fb6501ccd40 2024-09-11T01:45:46.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.925+0000 7fb6552ee640 1 --2- 172.21.15.12:0/1071483068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6501ccd40 0x7fb6501cd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:46.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.925+0000 7fb6552ee640 10 monclient: get_auth_request con 0x7fb6501ccd40 auth_method 0 2024-09-11T01:45:46.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.925+0000 7fb6552ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:46.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.925+0000 7fb6552ee640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:46.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.925+0000 7fb6552ee640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:46.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.925+0000 7fb6552ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:46.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.925+0000 7fb6552ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:46.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.925+0000 7fb645ffb640 1 -- 172.21.15.12:0/1071483068 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fb640024060 con 0x7fb6500a06d0 2024-09-11T01:45:46.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.925+0000 7fb6552ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:46.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.926+0000 7fb6552ee640 10 monclient(hunting): handle_auth_done global_id 174234 payload 824 2024-09-11T01:45:46.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.926+0000 7fb6552ee640 1 -- 172.21.15.12:0/1071483068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb650074560 con 0x7fb6501ccd40 2024-09-11T01:45:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.926+0000 7fb645ffb640 1 --2- 172.21.15.12:0/1071483068 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb63403e240 0x7fb634040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.926+0000 7fb645ffb640 1 -- 172.21.15.12:0/1071483068 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(565..565 src has 317..565) ==== 4815+0+0 (secure 0 0 0) 0x7fb640052d30 con 0x7fb6500a06d0 2024-09-11T01:45:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.926+0000 7fb655aef640 1 --2- 172.21.15.12:0/1071483068 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb63403e240 0x7fb634040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.926+0000 7fb655aef640 10 monclient: get_auth_request con 0x7fb63403e240 auth_method 0 2024-09-11T01:45:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.928+0000 7fb6552ee640 1 -- 172.21.15.12:0/1071483068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6501ccd40 msgr2=0x7fb6501cd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.928+0000 7fb6552ee640 1 --2- 172.21.15.12:0/1071483068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6501ccd40 0x7fb6501cd120 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb64c00ea70 tx=0x7fb64c00ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.928+0000 7fb645ffb640 1 -- 172.21.15.12:0/1071483068 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb64c01a070 con 0x7fb6501ccd40 2024-09-11T01:45:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.928+0000 7fb645ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.928+0000 7fb645ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.928+0000 7fb645ffb640 10 monclient: dump: 2024-09-11T01:45:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:46.945 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.928+0000 7fb645ffb640 1 -- 172.21.15.12:0/1071483068 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb64c00cc50 con 0x7fb6501ccd40 2024-09-11T01:45:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.928+0000 7fb645ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.928+0000 7fb645ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:45:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.928+0000 7fb645ffb640 1 -- 172.21.15.12:0/1071483068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6501ccd40 msgr2=0x7fb6501cd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.928+0000 7fb645ffb640 1 --2- 172.21.15.12:0/1071483068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6501ccd40 0x7fb6501cd120 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb64c00ea70 tx=0x7fb64c00ef30 comp rx=0 tx=0).stop 2024-09-11T01:45:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.928+0000 7fb645ffb640 1 -- 172.21.15.12:0/1071483068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6501ccd40 msgr2=0x7fb6501cd120 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:46.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.928+0000 7fb645ffb640 1 --2- 172.21.15.12:0/1071483068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6501ccd40 0x7fb6501cd120 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb64c00ea70 tx=0x7fb64c00ef30 comp rx=0 tx=0).stop 2024-09-11T01:45:46.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.928+0000 7fb655aef640 1 --2- 172.21.15.12:0/1071483068 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb63403e240 0x7fb634040700 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb648009bd0 tx=0x7fb64802f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:46.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.934+0000 7fb657552640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:45:46.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.934+0000 7fb657552640 1 --2- 172.21.15.12:0/1071483068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6501cd660 0x7fb650135710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:46.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.934+0000 7fb657552640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:46.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.934+0000 7fb657552640 1 -- 172.21.15.12:0/1071483068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb650074fe0 con 0x7fb6501cd660 2024-09-11T01:45:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.934+0000 7fb6552ee640 1 --2- 172.21.15.12:0/1071483068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6501cd660 0x7fb650135710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.935+0000 7fb6552ee640 10 monclient: get_auth_request con 0x7fb6501cd660 auth_method 0 2024-09-11T01:45:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.935+0000 7fb6552ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.935+0000 7fb6552ee640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.935+0000 7fb6552ee640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.935+0000 7fb6552ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.935+0000 7fb6552ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.935+0000 7fb6552ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.935+0000 7fb6552ee640 10 monclient(hunting): handle_auth_done global_id 174237 payload 824 2024-09-11T01:45:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.935+0000 7fb6552ee640 1 -- 172.21.15.12:0/1071483068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb65006dc80 con 0x7fb6501cd660 2024-09-11T01:45:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.937+0000 7fb6552ee640 1 -- 172.21.15.12:0/1071483068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6501cd660 msgr2=0x7fb650135710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.937+0000 7fb6552ee640 1 --2- 172.21.15.12:0/1071483068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6501cd660 0x7fb650135710 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb64c017000 tx=0x7fb64c004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.937+0000 7fb645ffb640 1 -- 172.21.15.12:0/1071483068 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb64c01a070 con 0x7fb6501cd660 2024-09-11T01:45:46.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.937+0000 7fb645ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:46.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.937+0000 7fb645ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:46.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.937+0000 7fb645ffb640 10 monclient: dump: 2024-09-11T01:45:46.949 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:46.949 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:46.949 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:46.949 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:46.949 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:46.949 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:46.949 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:46.949 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:46.949 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:46.949 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:46.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.937+0000 7fb645ffb640 1 -- 172.21.15.12:0/1071483068 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4710 (secure 0 0 0) 0x7fb64c01b040 con 0x7fb6501cd660 2024-09-11T01:45:46.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.937+0000 7fb645ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:45:46.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.937+0000 7fb645ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:45:46.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.937+0000 7fb645ffb640 1 -- 172.21.15.12:0/1071483068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6501cd660 msgr2=0x7fb650135710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:46.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.937+0000 7fb645ffb640 1 --2- 172.21.15.12:0/1071483068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6501cd660 0x7fb650135710 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb64c017000 tx=0x7fb64c004940 comp rx=0 tx=0).stop 2024-09-11T01:45:46.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.937+0000 7fb645ffb640 1 -- 172.21.15.12:0/1071483068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6501cd660 msgr2=0x7fb650135710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:46.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.937+0000 7fb645ffb640 1 --2- 172.21.15.12:0/1071483068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb6501cd660 0x7fb650135710 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb64c017000 tx=0x7fb64c004940 comp rx=0 tx=0).stop 2024-09-11T01:45:46.950 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:45:46.950 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:45:46.950 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:45:46.951 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:45:46.951 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 154, 2024-09-11T01:45:46.951 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 63007, 2024-09-11T01:45:46.951 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:45:46.951 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:45:46.951 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:45:46.951 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:45:46.951 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:46.951 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 4, 2024-09-11T01:45:46.951 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:45:46.951 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:45:46.951 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:45:46.951 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:46.952 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:46.952 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:46.952 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:46.952 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:46.952 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:46.952 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:46.952 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:46.952 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:46.952 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:46.952 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:46.952 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:46.952 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:45:46.952 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:45:46.953 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:46.953 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:46.953 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:46.953 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:46.953 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:46.953 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:46.953 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:46.953 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:46.953 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:46.953 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:46.953 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:46.953 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:46.953 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:46.954 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:45:46.954 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:45:46.954 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:45:46.954 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:45:46.954 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:45:46.954 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:45:46.954 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:45:46.954 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:45:46.954 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:45:46.954 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:45:46.954 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:45:46.954 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:45:46.954 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:45:46.955 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:45:46.955 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:45:46.955 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:45:46.955 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:45:46.955 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:46.955 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:46.955 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:46.955 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:46.955 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:46.955 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:46.955 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:46.955 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:46.956 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:46.956 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:46.956 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:46.956 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:46.956 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:45:46.956 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:46.956 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:45:46.956 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:46.956 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:45:46.956 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:45:46.956 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:46.956 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:46.956 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:46.956 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:46.957 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:45:46.957 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:46.957 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:46.957 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:46.957 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:46.957 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:45:46.957 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:46.957 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:46.957 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:46.957 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:46.957 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:45:46.957 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:45:46.957 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:46.958 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:46.958 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:46.958 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:46.958 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:46.958 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:45:46.958 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:45:46.958 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:46.958 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:46.958 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:46.958 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:46.958 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:45:46.958 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:46.958 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:45:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:45:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:45:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:46.960 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:45:46.960 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:45:46.960 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:46.960 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:46.960 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:46.960 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:46.960 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:45:46.960 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:46.960 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:46.960 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:46.960 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:46.960 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:45:46.960 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:46.961 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:46.961 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:46.961 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:46.961 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:45:46.961 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:45:46.961 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:46.961 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:46.961 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:46.961 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:46.961 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:46.961 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:46.961 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:45:46.961 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:45:46.962 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:46.962 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:46.962 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:46.962 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:45:46.962 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:46.962 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:46.962 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:45:46.962 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:46.962 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:46.962 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:46.962 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:45:46.962 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:46.962 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:46.963 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:45:46.963 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:46.963 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:46.963 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:46.963 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:45:46.963 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:46.963 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:46.963 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:46.963 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:45:46.963 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:45:46.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.938+0000 7fb657552640 1 -- 172.21.15.12:0/1071483068 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb63403e240 msgr2=0x7fb634040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:46.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.938+0000 7fb657552640 1 --2- 172.21.15.12:0/1071483068 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb63403e240 0x7fb634040700 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb648009bd0 tx=0x7fb64802f000 comp rx=0 tx=0).stop 2024-09-11T01:45:46.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.938+0000 7fb657552640 10 monclient: shutdown 2024-09-11T01:45:46.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.938+0000 7fb657552640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:46.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.938+0000 7fb657552640 1 -- 172.21.15.12:0/1071483068 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb6500a06d0 msgr2=0x7fb65013a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:46.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.938+0000 7fb657552640 1 --2- 172.21.15.12:0/1071483068 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb6500a06d0 0x7fb65013a940 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb64000e850 tx=0x7fb64000ed10 comp rx=0 tx=0).stop 2024-09-11T01:45:46.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.938+0000 7fb655aef640 1 -- 172.21.15.12:0/1071483068 reap_dead start 2024-09-11T01:45:46.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.938+0000 7fb655aef640 1 -- 172.21.15.12:0/1071483068 reap_dead start 2024-09-11T01:45:46.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.938+0000 7fb657552640 1 -- 172.21.15.12:0/1071483068 shutdown_connections 2024-09-11T01:45:46.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.938+0000 7fb657552640 1 -- 172.21.15.12:0/1071483068 >> 172.21.15.12:0/1071483068 conn(0x7fb6500a36c0 msgr2=0x7fb650073670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:46.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.938+0000 7fb657552640 1 -- 172.21.15.12:0/1071483068 shutdown_connections 2024-09-11T01:45:46.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:46.938+0000 7fb657552640 1 -- 172.21.15.12:0/1071483068 wait complete. 2024-09-11T01:45:46.966 INFO:tasks.rados.rados.0.smithi071.stdout:2517: done (1 left) 2024-09-11T01:45:46.966 INFO:tasks.rados.rados.0.smithi071.stdout:2519: done (0 left) 2024-09-11T01:45:46.966 INFO:tasks.rados.rados.0.smithi071.stdout:2520: read oid 20 snap -1 2024-09-11T01:45:46.966 INFO:tasks.rados.rados.0.smithi071.stdout:2520: expect (ObjNum 723 snap 224 seq_num 723) 2024-09-11T01:45:46.966 INFO:tasks.rados.rados.0.smithi071.stdout:2521: write oid 27 current snap is 283 2024-09-11T01:45:46.966 INFO:tasks.rados.rados.0.smithi071.stdout:2521: seq_num 901 ranges {743107=729003,1907923=689647} 2024-09-11T01:45:46.966 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:45:46.966 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:45:46.967 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:45:46.967 INFO:tasks.rados.rados.0.smithi071.stdout:2521: writing smithi07134777-27 from 743107 to 1472110 tid 1 2024-09-11T01:45:46.972 INFO:tasks.rados.rados.0.smithi071.stdout:2521: writing smithi07134777-27 from 1907923 to 2597570 tid 2 2024-09-11T01:45:46.973 INFO:tasks.rados.rados.0.smithi071.stdout:2522: read oid 15 snap -1 2024-09-11T01:45:46.973 INFO:tasks.rados.rados.0.smithi071.stdout:2522: expect (ObjNum 565 snap 160 seq_num 565) 2024-09-11T01:45:46.973 INFO:tasks.rados.rados.0.smithi071.stdout:2523: delete oid 39 current snap is 283 2024-09-11T01:45:46.973 INFO:tasks.rados.rados.0.smithi071.stdout:2520: expect (ObjNum 723 snap 224 seq_num 723) 2024-09-11T01:45:46.983 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1970670 2024-09-11T01:45:47.025 INFO:tasks.rados.rados.0.smithi071.stdout:2521: finishing write tid 1 to smithi07134777-27 2024-09-11T01:45:47.025 INFO:tasks.rados.rados.0.smithi071.stdout:2522: expect (ObjNum 565 snap 160 seq_num 565) 2024-09-11T01:45:47.025 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 403129 2024-09-11T01:45:47.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.068+0000 7fa00dbcd640 10 monclient: get_monmap_and_config 2024-09-11T01:45:47.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.068+0000 7fa00dbcd640 10 monclient: build_initial_monmap 2024-09-11T01:45:47.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.068+0000 7fa00dbcd640 10 monclient: monmap: 2024-09-11T01:45:47.070 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:47.070 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:47.070 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:45:47.069514+0000 2024-09-11T01:45:47.070 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:45:47.069514+0000 2024-09-11T01:45:47.070 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:47.070 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:47.070 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:45:47.071 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:45:47.071 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:45:47.071 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:45:47.071 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:45:47.071 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:47.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.069+0000 7fa00dbcd640 1 Processor -- start 2024-09-11T01:45:47.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.069+0000 7fa00dbcd640 1 -- start start 2024-09-11T01:45:47.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.069+0000 7fa00dbcd640 10 monclient: init 2024-09-11T01:45:47.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa00dbcd640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:47.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa00dbcd640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-11T01:45:47.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa00dbcd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0080a2e20 0x7fa00806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:47.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa00dbcd640 10 monclient(hunting): picked mon.noname-b con 0x7fa0080a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:47.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa00dbcd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa0080a4970 0x7fa0080a3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:47.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa00dbcd640 10 monclient(hunting): picked mon.noname-a con 0x7fa0080a4970 addr v2:172.21.15.12:3300/0 2024-09-11T01:45:47.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa00dbcd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa00806f910 addr v1:172.21.15.71:6789/0 2024-09-11T01:45:47.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa00dbcd640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa00806f4f0 con 0x7fa00806f910 2024-09-11T01:45:47.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa00dbcd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:47.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa00dbcd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fa00806ef30 con 0x7fa0080a4970 2024-09-11T01:45:47.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa00dbcd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:47.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa00dbcd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa00806ec50 con 0x7fa0080a2e20 2024-09-11T01:45:47.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa00dbcd640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:47.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa00dbcd640 10 monclient(hunting): authenticate will time out at 1753.512123s 2024-09-11T01:45:47.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa0077fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0080a2e20 0x7fa00806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:47.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa006ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa0080a4970 0x7fa0080a3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:47.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa006ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa0080a4970 0x7fa0080a3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47394/0 (socket says 172.21.15.12:47394) 2024-09-11T01:45:47.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa0077fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0080a2e20 0x7fa00806ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52004/0 (socket says 172.21.15.12:52004) 2024-09-11T01:45:47.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa006ffd640 1 -- 172.21.15.12:0/4267725002 learned_addr learned my addr 172.21.15.12:0/4267725002 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:47.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa006ffd640 10 monclient(hunting): get_auth_request con 0x7fa0080a4970 auth_method 0 2024-09-11T01:45:47.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa006ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:47.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa006ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:47.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa006ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:47.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa0077fe640 10 monclient(hunting): get_auth_request con 0x7fa0080a2e20 auth_method 0 2024-09-11T01:45:47.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa0077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:47.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa0077fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:47.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa0077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:47.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa006ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:47.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa006ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:47.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa006ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:47.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa0077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:47.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa0077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:47.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa0077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:47.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa006ffd640 10 monclient(hunting): handle_auth_done global_id 174240 payload 274 2024-09-11T01:45:47.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa006ffd640 1 -- 172.21.15.12:0/4267725002 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0080a2e20 msgr2=0x7fa00806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:47.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa006ffd640 1 --2- 172.21.15.12:0/4267725002 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0080a2e20 0x7fa00806ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:47.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa006ffd640 1 -- 172.21.15.12:0/4267725002 >> v1:172.21.15.71:6789/0 conn(0x7fa00806f910 legacy=0x7fa008137fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:47.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa006ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:45:47.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa006ffd640 1 monclient: found mon.noname-a 2024-09-11T01:45:47.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa006ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:47.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa006ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:45:47.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.070+0000 7fa006ffd640 1 -- 172.21.15.12:0/4267725002 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0080a3740 con 0x7fa0080a4970 2024-09-11T01:45:47.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.073+0000 7fa006ffd640 1 -- 172.21.15.12:0/4267725002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0080a4970 msgr2=0x7fa0080a3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:47.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.073+0000 7fa006ffd640 1 --2- 172.21.15.12:0/4267725002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0080a4970 0x7fa0080a3200 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc009870 tx=0x7f9ffc02a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bac205d14512afd1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:47.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.073+0000 7fa0067fc640 1 -- 172.21.15.12:0/4267725002 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc037070 con 0x7fa0080a4970 2024-09-11T01:45:47.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.073+0000 7fa0067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:47.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.073+0000 7fa0067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:45:47.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.073+0000 7fa0067fc640 10 monclient: dump: 2024-09-11T01:45:47.083 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:47.083 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:47.083 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.083 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.083 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:47.083 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:47.083 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:47.083 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:47.083 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:47.083 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:47.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.073+0000 7fa0067fc640 10 monclient: _finish_auth 0 2024-09-11T01:45:47.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.073+0000 7fa0067fc640 10 monclient: _check_auth_tickets 2024-09-11T01:45:47.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.073+0000 7fa0067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:47.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.073+0000 7fa0067fc640 1 -- 172.21.15.12:0/4267725002 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ffc032440 con 0x7fa0080a4970 2024-09-11T01:45:47.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.073+0000 7fa00dbcd640 5 monclient: authenticate success, global_id 174240 2024-09-11T01:45:47.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.073+0000 7fa00dbcd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:45:47.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.073+0000 7fa0067fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:47.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.073+0000 7fa0067fc640 1 -- 172.21.15.12:0/4267725002 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc032810 con 0x7fa0080a4970 2024-09-11T01:45:47.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.073+0000 7fa0067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:47.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.073+0000 7fa0067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:47.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.073+0000 7fa0067fc640 10 monclient: dump: 2024-09-11T01:45:47.084 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:47.084 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:47.084 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.084 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.085 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:47.085 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:47.085 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:47.085 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:47.085 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:47.085 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:47.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.073+0000 7fa00dbcd640 10 monclient: get_monmap_and_config success 2024-09-11T01:45:47.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.073+0000 7fa00dbcd640 4 set_mon_vals no callback set 2024-09-11T01:45:47.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.074+0000 7fa00dbcd640 10 monclient: shutdown 2024-09-11T01:45:47.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.074+0000 7fa00dbcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:47.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.074+0000 7fa00dbcd640 1 -- 172.21.15.12:0/4267725002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0080a4970 msgr2=0x7fa0080a3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:47.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.074+0000 7fa00dbcd640 1 --2- 172.21.15.12:0/4267725002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0080a4970 0x7fa0080a3200 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc009870 tx=0x7f9ffc02a0a0 comp rx=0 tx=0).stop 2024-09-11T01:45:47.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.074+0000 7fa00dbcd640 1 -- 172.21.15.12:0/4267725002 shutdown_connections 2024-09-11T01:45:47.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.074+0000 7fa00dbcd640 1 --2- 172.21.15.12:0/4267725002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0080a4970 0x7fa0080a3200 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:47.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.074+0000 7fa00dbcd640 1 --2- 172.21.15.12:0/4267725002 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0080a2e20 0x7fa00806ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:47.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.074+0000 7fa00dbcd640 1 -- 172.21.15.12:0/4267725002 >> 172.21.15.12:0/4267725002 conn(0x7fa0080a8600 msgr2=0x7fa0080a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:47.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.074+0000 7fa00dbcd640 1 -- 172.21.15.12:0/4267725002 shutdown_connections 2024-09-11T01:45:47.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.074+0000 7fa00dbcd640 1 -- 172.21.15.12:0/4267725002 wait complete. 2024-09-11T01:45:47.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.074+0000 7fa00dbcd640 10 monclient: build_initial_monmap 2024-09-11T01:45:47.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.074+0000 7fa00dbcd640 10 monclient: monmap: 2024-09-11T01:45:47.086 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:47.086 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:45:47.086 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:45:47.086 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:45:47.086 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:47.086 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:47.087 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:45:47.087 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:45:47.087 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:45:47.087 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:47.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa00dbcd640 1 Processor -- start 2024-09-11T01:45:47.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa00dbcd640 1 -- start start 2024-09-11T01:45:47.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa00dbcd640 10 monclient: init 2024-09-11T01:45:47.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa00dbcd640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:47.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa00dbcd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:45:47.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa00dbcd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa00806f910 0x7fa0081cd840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:47.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa00dbcd640 10 monclient(hunting): picked mon.noname-b con 0x7fa00806f910 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:45:47.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa00dbcd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0080a2e20 0x7fa0081377c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:47.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa00dbcd640 10 monclient(hunting): picked mon.noname-a con 0x7fa0080a2e20 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:45:47.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa00dbcd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0080a4970 0x7fa008137d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:47.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa00dbcd640 10 monclient(hunting): picked mon.noname-c con 0x7fa0080a4970 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:47.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa00dbcd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:47.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa00dbcd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa008138860 con 0x7fa0080a2e20 2024-09-11T01:45:47.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa00dbcd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:47.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa00dbcd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa0081386e0 con 0x7fa00806f910 2024-09-11T01:45:47.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa00dbcd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:47.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa00dbcd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa0081389e0 con 0x7fa0080a4970 2024-09-11T01:45:47.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa00dbcd640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:47.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa00dbcd640 10 monclient(hunting): authenticate will time out at 1753.517730s 2024-09-11T01:45:47.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa006ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0080a2e20 0x7fa0081377c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:47.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa007fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0080a4970 0x7fa008137d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:47.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa006ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0080a2e20 0x7fa0081377c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47408/0 (socket says 172.21.15.12:47408) 2024-09-11T01:45:47.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa006ffd640 1 -- 172.21.15.12:0/3762568853 learned_addr learned my addr 172.21.15.12:0/3762568853 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:47.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa006ffd640 10 monclient(hunting): get_auth_request con 0x7fa0080a2e20 auth_method 0 2024-09-11T01:45:47.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa006ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:47.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa006ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:47.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa006ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:47.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa007fff640 10 monclient(hunting): get_auth_request con 0x7fa0080a4970 auth_method 0 2024-09-11T01:45:47.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa007fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:47.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa007fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:47.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa007fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:47.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa0077fe640 1 --2- 172.21.15.12:0/3762568853 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa00806f910 0x7fa0081cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:47.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa006ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa006ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa006ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa0077fe640 10 monclient(hunting): get_auth_request con 0x7fa00806f910 auth_method 0 2024-09-11T01:45:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa0077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa0077fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa0077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa007fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa007fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.075+0000 7fa007fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.076+0000 7fa0077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.076+0000 7fa0077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.076+0000 7fa0077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.076+0000 7fa006ffd640 10 monclient(hunting): handle_auth_done global_id 174243 payload 824 2024-09-11T01:45:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.076+0000 7fa006ffd640 1 -- 172.21.15.12:0/3762568853 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0080a4970 msgr2=0x7fa008137d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.076+0000 7fa006ffd640 1 --2- 172.21.15.12:0/3762568853 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0080a4970 0x7fa008137d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.076+0000 7fa006ffd640 1 -- 172.21.15.12:0/3762568853 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa00806f910 msgr2=0x7fa0081cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.076+0000 7fa006ffd640 1 --2- 172.21.15.12:0/3762568853 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa00806f910 0x7fa0081cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.076+0000 7fa006ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:45:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.076+0000 7fa006ffd640 1 monclient: found mon.noname-a 2024-09-11T01:45:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.076+0000 7fa006ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.076+0000 7fa006ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:45:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.076+0000 7fa006ffd640 1 -- 172.21.15.12:0/3762568853 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0081cdd80 con 0x7fa0080a2e20 2024-09-11T01:45:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.076+0000 7fa007fff640 1 --2- 172.21.15.12:0/3762568853 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0080a4970 0x7fa008137d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:45:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.078+0000 7fa006ffd640 1 -- 172.21.15.12:0/3762568853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0080a2e20 msgr2=0x7fa0081377c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.078+0000 7fa006ffd640 1 --2- 172.21.15.12:0/3762568853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0080a2e20 0x7fa0081377c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc009870 tx=0x7f9ffc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.078+0000 7f9feffff640 1 -- 172.21.15.12:0/3762568853 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc041020 con 0x7fa0080a2e20 2024-09-11T01:45:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.078+0000 7f9feffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.078+0000 7f9feffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:45:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.078+0000 7f9feffff640 10 monclient: dump: 2024-09-11T01:45:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:47.092 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.078+0000 7f9feffff640 10 monclient: _finish_auth 0 2024-09-11T01:45:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.078+0000 7f9feffff640 10 monclient: _check_auth_tickets 2024-09-11T01:45:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.078+0000 7f9feffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.078+0000 7f9feffff640 1 -- 172.21.15.12:0/3762568853 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ffc03d630 con 0x7fa0080a2e20 2024-09-11T01:45:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.078+0000 7fa00dbcd640 5 monclient: authenticate success, global_id 174243 2024-09-11T01:45:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.078+0000 7fa00dbcd640 10 monclient: _renew_subs 2024-09-11T01:45:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.078+0000 7fa00dbcd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.078+0000 7fa00dbcd640 1 -- 172.21.15.12:0/3762568853 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa00809f350 con 0x7fa0080a2e20 2024-09-11T01:45:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.078+0000 7f9feffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.078+0000 7f9feffff640 1 -- 172.21.15.12:0/3762568853 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc037040 con 0x7fa0080a2e20 2024-09-11T01:45:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.078+0000 7f9feffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.078+0000 7f9feffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.078+0000 7f9feffff640 10 monclient: dump: 2024-09-11T01:45:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:47.094 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.078+0000 7fa00dbcd640 10 monclient: _renew_subs 2024-09-11T01:45:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.078+0000 7fa00dbcd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.078+0000 7fa00dbcd640 1 -- 172.21.15.12:0/3762568853 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0081248e0 con 0x7fa0080a2e20 2024-09-11T01:45:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.078+0000 7fa005ffb640 4 set_mon_vals no callback set 2024-09-11T01:45:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.079+0000 7fa00dbcd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.079+0000 7fa00dbcd640 1 --2- 172.21.15.12:0/3762568853 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0081c0df0 0x7fa00802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.079+0000 7fa00dbcd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.079+0000 7fa00dbcd640 1 -- 172.21.15.12:0/3762568853 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa00806a060 con 0x7fa0081c0df0 2024-09-11T01:45:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.079+0000 7fa006ffd640 1 --2- 172.21.15.12:0/3762568853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0080a2e20 0x7fa0081377c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc009870 tx=0x7f9ffc0046c0 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7f9ffc073110 osd_map(566..566 src has 317..566) 2024-09-11T01:45:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.079+0000 7f9feffff640 1 -- 172.21.15.12:0/3762568853 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f9ffc03a030 con 0x7fa0080a2e20 2024-09-11T01:45:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.079+0000 7fa0077fe640 1 --2- 172.21.15.12:0/3762568853 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0081c0df0 0x7fa00802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.079+0000 7fa0077fe640 10 monclient: get_auth_request con 0x7fa0081c0df0 auth_method 0 2024-09-11T01:45:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.079+0000 7fa0077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.079+0000 7fa0077fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.079+0000 7fa0077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.079+0000 7fa0077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.079+0000 7fa0077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.079+0000 7fa0077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.079+0000 7fa0077fe640 10 monclient(hunting): handle_auth_done global_id 184133 payload 824 2024-09-11T01:45:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.079+0000 7f9feffff640 1 --2- 172.21.15.12:0/3762568853 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9ff403e210 0x7f9ff40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.079+0000 7fa0077fe640 1 -- 172.21.15.12:0/3762568853 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa00809ead0 con 0x7fa0081c0df0 2024-09-11T01:45:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.080+0000 7fa007fff640 1 --2- 172.21.15.12:0/3762568853 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9ff403e210 0x7f9ff40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.080+0000 7fa007fff640 10 monclient: get_auth_request con 0x7f9ff403e210 auth_method 0 2024-09-11T01:45:47.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.082+0000 7fa0077fe640 1 -- 172.21.15.12:0/3762568853 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0081c0df0 msgr2=0x7fa00802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:47.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.082+0000 7fa0077fe640 1 --2- 172.21.15.12:0/3762568853 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0081c0df0 0x7fa00802d440 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9ff80118b0 tx=0x7f9ff8011d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:47.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.082+0000 7f9feffff640 1 -- 172.21.15.12:0/3762568853 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff8019070 con 0x7fa0081c0df0 2024-09-11T01:45:47.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.082+0000 7f9feffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:47.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.082+0000 7f9feffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:45:47.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.082+0000 7f9feffff640 10 monclient: dump: 2024-09-11T01:45:47.099 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:47.099 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:47.099 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.099 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.099 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:47.099 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:47.099 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:47.099 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:47.099 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:47.099 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:47.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.082+0000 7fa007fff640 1 --2- 172.21.15.12:0/3762568853 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9ff403e210 0x7f9ff40406d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa00806fd60 tx=0x7fa000003d10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:47.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.082+0000 7f9feffff640 1 -- 172.21.15.12:0/3762568853 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9ff800f890 con 0x7fa0081c0df0 2024-09-11T01:45:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.082+0000 7f9feffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.082+0000 7f9feffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:45:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.082+0000 7f9feffff640 1 -- 172.21.15.12:0/3762568853 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0081c0df0 msgr2=0x7fa00802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.082+0000 7f9feffff640 1 --2- 172.21.15.12:0/3762568853 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0081c0df0 0x7fa00802d440 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9ff80118b0 tx=0x7f9ff8011d70 comp rx=0 tx=0).stop 2024-09-11T01:45:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.082+0000 7f9feffff640 1 -- 172.21.15.12:0/3762568853 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0081c0df0 msgr2=0x7fa00802d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.082+0000 7f9feffff640 1 --2- 172.21.15.12:0/3762568853 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0081c0df0 0x7fa00802d440 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9ff80118b0 tx=0x7f9ff8011d70 comp rx=0 tx=0).stop 2024-09-11T01:45:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.088+0000 7fa00dbcd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:45:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.088+0000 7fa00dbcd640 1 --2- 172.21.15.12:0/3762568853 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0081c11d0 0x7fa00802d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.088+0000 7fa00dbcd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.088+0000 7fa00dbcd640 1 -- 172.21.15.12:0/3762568853 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa008074530 con 0x7fa0081c11d0 2024-09-11T01:45:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.088+0000 7fa0077fe640 1 --2- 172.21.15.12:0/3762568853 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0081c11d0 0x7fa00802d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.088+0000 7fa0077fe640 10 monclient: get_auth_request con 0x7fa0081c11d0 auth_method 0 2024-09-11T01:45:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.088+0000 7fa0077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.088+0000 7fa0077fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.088+0000 7fa0077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.089+0000 7fa0077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.089+0000 7fa0077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.089+0000 7fa0077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.089+0000 7fa0077fe640 10 monclient(hunting): handle_auth_done global_id 184136 payload 824 2024-09-11T01:45:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.089+0000 7fa0077fe640 1 -- 172.21.15.12:0/3762568853 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa008074a80 con 0x7fa0081c11d0 2024-09-11T01:45:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.091+0000 7fa0077fe640 1 -- 172.21.15.12:0/3762568853 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0081c11d0 msgr2=0x7fa00802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.091+0000 7fa0077fe640 1 --2- 172.21.15.12:0/3762568853 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0081c11d0 0x7fa00802d980 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9ff801b000 tx=0x7f9ff80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.091+0000 7f9feffff640 1 -- 172.21.15.12:0/3762568853 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff8019040 con 0x7fa0081c11d0 2024-09-11T01:45:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.091+0000 7f9feffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.091+0000 7f9feffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:45:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.091+0000 7f9feffff640 10 monclient: dump: 2024-09-11T01:45:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:47.102 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.092+0000 7f9feffff640 1 -- 172.21.15.12:0/3762568853 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4540 (secure 0 0 0) 0x7f9ff8003e40 con 0x7fa0081c11d0 2024-09-11T01:45:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.092+0000 7f9feffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:45:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.092+0000 7f9feffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:45:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.092+0000 7f9feffff640 1 -- 172.21.15.12:0/3762568853 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0081c11d0 msgr2=0x7fa00802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.092+0000 7f9feffff640 1 --2- 172.21.15.12:0/3762568853 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0081c11d0 0x7fa00802d980 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9ff801b000 tx=0x7f9ff80046c0 comp rx=0 tx=0).stop 2024-09-11T01:45:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.092+0000 7f9feffff640 1 -- 172.21.15.12:0/3762568853 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0081c11d0 msgr2=0x7fa00802d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:47.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.092+0000 7f9feffff640 1 --2- 172.21.15.12:0/3762568853 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa0081c11d0 0x7fa00802d980 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9ff801b000 tx=0x7f9ff80046c0 comp rx=0 tx=0).stop 2024-09-11T01:45:47.103 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:45:47.103 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:45:47.103 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:45:47.103 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:45:47.103 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 154, 2024-09-11T01:45:47.103 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 7684, 2024-09-11T01:45:47.103 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:45:47.103 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:45:47.104 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:45:47.104 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:45:47.104 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:47.104 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 4, 2024-09-11T01:45:47.104 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:45:47.104 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:45:47.104 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:45:47.104 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:47.104 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:47.104 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:47.104 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:47.104 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:47.104 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:47.104 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:47.105 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:47.105 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:47.105 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:47.105 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:47.105 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:47.105 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:45:47.105 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:45:47.105 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:47.105 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:47.105 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:47.105 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:47.105 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:47.105 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:47.106 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:47.106 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:47.106 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:47.106 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:47.106 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:47.106 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:47.106 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:47.106 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:45:47.106 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:45:47.106 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:45:47.106 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:45:47.106 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:45:47.106 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:45:47.107 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:45:47.107 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:45:47.107 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:45:47.107 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:45:47.107 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:45:47.107 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:45:47.107 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:45:47.107 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:45:47.107 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:45:47.107 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:45:47.107 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:45:47.107 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:47.107 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:47.108 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:47.108 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:47.108 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:47.108 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:47.108 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:47.108 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:47.108 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:47.108 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:47.108 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:47.108 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:47.108 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:45:47.108 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:47.108 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:45:47.108 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:47.109 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:45:47.109 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:45:47.109 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:47.109 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:47.109 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:47.109 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:47.109 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:45:47.109 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:47.109 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:47.109 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:47.109 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:47.109 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:45:47.109 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:47.109 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:47.110 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:47.110 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:47.110 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:45:47.110 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:45:47.110 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:47.110 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:47.110 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:47.110 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:47.110 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:47.110 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:45:47.110 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:45:47.110 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:47.110 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:47.111 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:47.111 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:47.111 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:45:47.111 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:47.111 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:47.111 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:47.111 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:47.111 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:45:47.111 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:47.111 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:47.111 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:47.111 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:47.111 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:45:47.111 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:45:47.112 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:47.112 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:47.112 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:47.112 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:47.112 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:47.112 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:45:47.112 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:45:47.112 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:47.112 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:47.112 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:47.112 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:47.112 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:45:47.112 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:47.113 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:47.113 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:47.113 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:47.113 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:45:47.113 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:47.113 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:47.113 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:47.113 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:47.113 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:45:47.113 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:45:47.113 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:47.113 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:47.113 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:47.114 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:47.114 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:47.114 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:47.114 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:45:47.114 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:45:47.114 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:47.114 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:47.114 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:47.114 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:45:47.114 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:47.114 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:47.114 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:45:47.114 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:47.114 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:47.115 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:47.115 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:45:47.115 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:47.115 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:47.115 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:47.115 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:45:47.115 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:45:47.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.092+0000 7fa00dbcd640 1 -- 172.21.15.12:0/3762568853 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9ff403e210 msgr2=0x7f9ff40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:47.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.092+0000 7fa00dbcd640 1 --2- 172.21.15.12:0/3762568853 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9ff403e210 0x7f9ff40406d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa00806fd60 tx=0x7fa000003d10 comp rx=0 tx=0).stop 2024-09-11T01:45:47.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.092+0000 7fa00dbcd640 10 monclient: shutdown 2024-09-11T01:45:47.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.092+0000 7fa00dbcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:47.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.092+0000 7fa00dbcd640 1 -- 172.21.15.12:0/3762568853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0080a2e20 msgr2=0x7fa0081377c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:47.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.092+0000 7fa00dbcd640 1 --2- 172.21.15.12:0/3762568853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0080a2e20 0x7fa0081377c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc009870 tx=0x7f9ffc0046c0 comp rx=0 tx=0).stop 2024-09-11T01:45:47.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.092+0000 7fa007fff640 1 -- 172.21.15.12:0/3762568853 reap_dead start 2024-09-11T01:45:47.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.092+0000 7f9feffff640 1 -- 172.21.15.12:0/3762568853 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(566..566 src has 317..566) ==== 4787+0+0 (unknown 0 0 0) 0x7f9ffc073110 con 0x7fa0080a2e20 2024-09-11T01:45:47.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.092+0000 7fa007fff640 1 -- 172.21.15.12:0/3762568853 reap_dead start 2024-09-11T01:45:47.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.092+0000 7fa00dbcd640 1 -- 172.21.15.12:0/3762568853 shutdown_connections 2024-09-11T01:45:47.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.092+0000 7fa00dbcd640 1 -- 172.21.15.12:0/3762568853 >> 172.21.15.12:0/3762568853 conn(0x7fa0080a8600 msgr2=0x7fa00809e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:47.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.092+0000 7fa00dbcd640 1 -- 172.21.15.12:0/3762568853 shutdown_connections 2024-09-11T01:45:47.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.092+0000 7fa00dbcd640 1 -- 172.21.15.12:0/3762568853 wait complete. 2024-09-11T01:45:47.117 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:45:47.118 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:45:47.118 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:45:47.141 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2555244 2024-09-11T01:45:47.172 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 433781 2024-09-11T01:45:47.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.218+0000 7f2995b8a640 10 monclient: get_monmap_and_config 2024-09-11T01:45:47.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.218+0000 7f2995b8a640 10 monclient: build_initial_monmap 2024-09-11T01:45:47.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.218+0000 7f2995b8a640 10 monclient: monmap: 2024-09-11T01:45:47.220 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:47.220 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:47.220 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:45:47.219314+0000 2024-09-11T01:45:47.220 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:45:47.219314+0000 2024-09-11T01:45:47.220 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:47.220 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:47.220 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:45:47.220 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:45:47.221 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:45:47.221 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:45:47.221 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:45:47.221 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:47.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.219+0000 7f2995b8a640 1 Processor -- start 2024-09-11T01:45:47.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.219+0000 7f2995b8a640 1 -- start start 2024-09-11T01:45:47.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.219+0000 7f2995b8a640 10 monclient: init 2024-09-11T01:45:47.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.219+0000 7f2995b8a640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:47.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.219+0000 7f2995b8a640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-11T01:45:47.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.219+0000 7f2995b8a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f29900a08c0 0x7f2990137be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:47.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.219+0000 7f2995b8a640 10 monclient(hunting): picked mon.noname-a con 0x7f29900a08c0 addr v2:172.21.15.12:3300/0 2024-09-11T01:45:47.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.219+0000 7f2995b8a640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2990073ef0 0x7f29900a5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:47.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.219+0000 7f2995b8a640 10 monclient(hunting): picked mon.noname-c con 0x7f2990073ef0 addr v2:172.21.15.71:3300/0 2024-09-11T01:45:47.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.219+0000 7f2995b8a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2990074360 0x7f29900a5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:47.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.219+0000 7f2995b8a640 10 monclient(hunting): picked mon.noname-b con 0x7f2990074360 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:47.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.219+0000 7f2995b8a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:47.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.219+0000 7f2995b8a640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f29900a06f0 con 0x7f29900a08c0 2024-09-11T01:45:47.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.219+0000 7f2995b8a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:47.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.219+0000 7f2995b8a640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f29900a0ca0 con 0x7f2990073ef0 2024-09-11T01:45:47.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.219+0000 7f2995b8a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:47.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.219+0000 7f2995b8a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2990138a80 con 0x7f2990074360 2024-09-11T01:45:47.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.219+0000 7f2995b8a640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:47.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.219+0000 7f2995b8a640 10 monclient(hunting): authenticate will time out at 1753.661875s 2024-09-11T01:45:47.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.219+0000 7f298f7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f29900a08c0 0x7f2990137be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:47.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2990074360 0x7f29900a5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:47.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298f7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f29900a08c0 0x7f2990137be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47412/0 (socket says 172.21.15.12:47412) 2024-09-11T01:45:47.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298f7fe640 1 -- 172.21.15.12:0/3942854315 learned_addr learned my addr 172.21.15.12:0/3942854315 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:47.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298f7fe640 10 monclient(hunting): get_auth_request con 0x7f29900a08c0 auth_method 0 2024-09-11T01:45:47.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:47.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:47.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:47.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298ffff640 10 monclient(hunting): get_auth_request con 0x7f2990074360 auth_method 0 2024-09-11T01:45:47.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:47.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:47.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:47.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298effd640 1 --2- 172.21.15.12:0/3942854315 >> v2:172.21.15.71:3300/0 conn(0x7f2990073ef0 0x7f29900a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:47.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298effd640 10 monclient(hunting): get_auth_request con 0x7f2990073ef0 auth_method 0 2024-09-11T01:45:47.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:47.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:47.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:47.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:47.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:47.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:47.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:47.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:47.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:47.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:47.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:47.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:47.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298f7fe640 10 monclient(hunting): handle_auth_done global_id 174246 payload 274 2024-09-11T01:45:47.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298f7fe640 1 -- 172.21.15.12:0/3942854315 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2990074360 msgr2=0x7f29900a5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:47.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298f7fe640 1 --2- 172.21.15.12:0/3942854315 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2990074360 0x7f29900a5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:47.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298f7fe640 1 -- 172.21.15.12:0/3942854315 >> v2:172.21.15.71:3300/0 conn(0x7f2990073ef0 msgr2=0x7f29900a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:47.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298f7fe640 1 --2- 172.21.15.12:0/3942854315 >> v2:172.21.15.71:3300/0 conn(0x7f2990073ef0 0x7f29900a5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:47.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298f7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:45:47.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298f7fe640 1 monclient: found mon.noname-a 2024-09-11T01:45:47.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:47.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:45:47.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298f7fe640 1 -- 172.21.15.12:0/3942854315 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2990070100 con 0x7f29900a08c0 2024-09-11T01:45:47.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.220+0000 7f298ffff640 1 --2- 172.21.15.12:0/3942854315 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2990074360 0x7f29900a5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:45:47.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f298f7fe640 1 -- 172.21.15.12:0/3942854315 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29900a08c0 msgr2=0x7f2990137be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:47.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f298f7fe640 1 --2- 172.21.15.12:0/3942854315 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29900a08c0 0x7f2990137be0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f29900a00e0 tx=0x7f298002a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48ae36a8a855575b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:47.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f298e7fc640 1 -- 172.21.15.12:0/3942854315 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2980037070 con 0x7f29900a08c0 2024-09-11T01:45:47.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f298e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:47.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f298e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:45:47.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f298e7fc640 10 monclient: dump: 2024-09-11T01:45:47.234 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:47.234 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:47.234 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.234 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.234 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:47.234 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:47.234 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:47.234 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:47.235 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:47.235 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:47.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f298e7fc640 10 monclient: _finish_auth 0 2024-09-11T01:45:47.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f298e7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:45:47.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f298e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:47.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f298e7fc640 1 -- 172.21.15.12:0/3942854315 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2980032640 con 0x7f29900a08c0 2024-09-11T01:45:47.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f2995b8a640 5 monclient: authenticate success, global_id 174246 2024-09-11T01:45:47.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f298e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:47.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f298e7fc640 1 -- 172.21.15.12:0/3942854315 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2980032a10 con 0x7f29900a08c0 2024-09-11T01:45:47.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f298e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:47.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f298e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:47.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f298e7fc640 10 monclient: dump: 2024-09-11T01:45:47.235 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:47.235 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:47.235 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.236 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.236 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:47.236 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:47.236 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:47.236 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:47.236 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:47.236 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:47.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f2995b8a640 10 monclient: get_monmap_and_config success 2024-09-11T01:45:47.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f2995b8a640 4 set_mon_vals no callback set 2024-09-11T01:45:47.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f2995b8a640 10 monclient: shutdown 2024-09-11T01:45:47.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f2995b8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:47.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f2995b8a640 1 -- 172.21.15.12:0/3942854315 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29900a08c0 msgr2=0x7f2990137be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:47.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f2995b8a640 1 --2- 172.21.15.12:0/3942854315 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29900a08c0 0x7f2990137be0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f29900a00e0 tx=0x7f298002a0b0 comp rx=0 tx=0).stop 2024-09-11T01:45:47.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f2995b8a640 1 -- 172.21.15.12:0/3942854315 shutdown_connections 2024-09-11T01:45:47.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f2995b8a640 1 --2- 172.21.15.12:0/3942854315 >> v2:172.21.15.71:3300/0 conn(0x7f2990073ef0 0x7f29900a5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:47.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f2995b8a640 1 --2- 172.21.15.12:0/3942854315 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2990074360 0x7f29900a5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:47.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f2995b8a640 1 --2- 172.21.15.12:0/3942854315 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29900a08c0 0x7f2990137be0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:47.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f2995b8a640 1 -- 172.21.15.12:0/3942854315 >> 172.21.15.12:0/3942854315 conn(0x7f2990068950 msgr2=0x7f2990068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:47.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f2995b8a640 1 -- 172.21.15.12:0/3942854315 shutdown_connections 2024-09-11T01:45:47.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.223+0000 7f2995b8a640 1 -- 172.21.15.12:0/3942854315 wait complete. 2024-09-11T01:45:47.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.224+0000 7f2995b8a640 10 monclient: build_initial_monmap 2024-09-11T01:45:47.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.224+0000 7f2995b8a640 10 monclient: monmap: 2024-09-11T01:45:47.237 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:47.238 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:45:47.238 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:45:47.238 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:45:47.238 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:47.238 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:47.238 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:45:47.238 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:45:47.238 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:45:47.238 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:47.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.224+0000 7f2995b8a640 1 Processor -- start 2024-09-11T01:45:47.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f2995b8a640 1 -- start start 2024-09-11T01:45:47.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f2995b8a640 10 monclient: init 2024-09-11T01:45:47.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f2995b8a640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:47.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f2995b8a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:45:47.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f2995b8a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2990073ef0 0x7f299012e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:47.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f2995b8a640 10 monclient(hunting): picked mon.noname-b con 0x7f2990073ef0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:45:47.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f2995b8a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2990074360 0x7f299012e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:47.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f2995b8a640 10 monclient(hunting): picked mon.noname-c con 0x7f2990074360 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:47.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f2995b8a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29900a08c0 0x7f29901bd960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:47.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f2995b8a640 10 monclient(hunting): picked mon.noname-a con 0x7f29900a08c0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:45:47.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f2995b8a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:47.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f2995b8a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f29901382a0 con 0x7f29900a08c0 2024-09-11T01:45:47.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f2995b8a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:47.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f2995b8a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f2990138120 con 0x7f2990073ef0 2024-09-11T01:45:47.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f2995b8a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:47.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f2995b8a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f29900a6110 con 0x7f2990074360 2024-09-11T01:45:47.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f2995b8a640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:47.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f2995b8a640 10 monclient(hunting): authenticate will time out at 1753.667580s 2024-09-11T01:45:47.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29900a08c0 0x7f29901bd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:47.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298effd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2990074360 0x7f299012e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:47.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298effd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2990074360 0x7f299012e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52050/0 (socket says 172.21.15.12:52050) 2024-09-11T01:45:47.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29900a08c0 0x7f29901bd960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47416/0 (socket says 172.21.15.12:47416) 2024-09-11T01:45:47.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298effd640 1 -- 172.21.15.12:0/2166401808 learned_addr learned my addr 172.21.15.12:0/2166401808 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:47.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298effd640 10 monclient(hunting): get_auth_request con 0x7f2990074360 auth_method 0 2024-09-11T01:45:47.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:47.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:47.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:47.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298ffff640 10 monclient(hunting): get_auth_request con 0x7f29900a08c0 auth_method 0 2024-09-11T01:45:47.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:47.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:47.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:47.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298f7fe640 1 --2- 172.21.15.12:0/2166401808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2990073ef0 0x7f299012e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:47.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298f7fe640 10 monclient(hunting): get_auth_request con 0x7f2990073ef0 auth_method 0 2024-09-11T01:45:47.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:47.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:47.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:47.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:47.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:47.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:47.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:47.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:47.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:47.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:47.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:47.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:47.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298ffff640 10 monclient(hunting): handle_auth_done global_id 174249 payload 824 2024-09-11T01:45:47.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298ffff640 1 -- 172.21.15.12:0/2166401808 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2990074360 msgr2=0x7f299012e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:47.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298ffff640 1 --2- 172.21.15.12:0/2166401808 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2990074360 0x7f299012e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:47.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298ffff640 1 -- 172.21.15.12:0/2166401808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2990073ef0 msgr2=0x7f299012e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:47.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298ffff640 1 --2- 172.21.15.12:0/2166401808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2990073ef0 0x7f299012e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:47.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:45:47.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298ffff640 1 monclient: found mon.noname-a 2024-09-11T01:45:47.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:47.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:45:47.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.225+0000 7f298ffff640 1 -- 172.21.15.12:0/2166401808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29901c04d0 con 0x7f29900a08c0 2024-09-11T01:45:47.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.226+0000 7f298effd640 1 --2- 172.21.15.12:0/2166401808 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2990074360 0x7f299012e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:45:47.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.226+0000 7f298f7fe640 1 --2- 172.21.15.12:0/2166401808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2990073ef0 0x7f299012e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:45:47.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.228+0000 7f298ffff640 1 -- 172.21.15.12:0/2166401808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29900a08c0 msgr2=0x7f29901bd960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:47.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.228+0000 7f298ffff640 1 --2- 172.21.15.12:0/2166401808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29900a08c0 0x7f29901bd960 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f298800efc0 tx=0x7f298800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:47.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.228+0000 7f2977fff640 1 -- 172.21.15.12:0/2166401808 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2988018030 con 0x7f29900a08c0 2024-09-11T01:45:47.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.228+0000 7f2977fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:47.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.228+0000 7f2977fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:45:47.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.228+0000 7f2977fff640 10 monclient: dump: 2024-09-11T01:45:47.243 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:47.243 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:47.243 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.243 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.243 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:47.243 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:47.243 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:47.243 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:47.243 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:47.243 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:47.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.228+0000 7f2977fff640 10 monclient: _finish_auth 0 2024-09-11T01:45:47.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.228+0000 7f2977fff640 10 monclient: _check_auth_tickets 2024-09-11T01:45:47.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.228+0000 7f2977fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:47.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.228+0000 7f2977fff640 1 -- 172.21.15.12:0/2166401808 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2988003990 con 0x7f29900a08c0 2024-09-11T01:45:47.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.228+0000 7f2995b8a640 5 monclient: authenticate success, global_id 174249 2024-09-11T01:45:47.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.228+0000 7f2995b8a640 10 monclient: _renew_subs 2024-09-11T01:45:47.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.228+0000 7f2995b8a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:47.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.228+0000 7f2995b8a640 1 -- 172.21.15.12:0/2166401808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29900a2690 con 0x7f29900a08c0 2024-09-11T01:45:47.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.228+0000 7f2977fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:47.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.228+0000 7f2977fff640 1 -- 172.21.15.12:0/2166401808 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2988003d60 con 0x7f29900a08c0 2024-09-11T01:45:47.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.228+0000 7f2977fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:47.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.228+0000 7f2977fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:45:47.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.228+0000 7f2977fff640 10 monclient: dump: 2024-09-11T01:45:47.244 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:47.245 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:47.245 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.245 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.245 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:47.245 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:47.245 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:47.245 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:47.245 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:47.245 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:47.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.228+0000 7f298dffb640 4 set_mon_vals no callback set 2024-09-11T01:45:47.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.228+0000 7f2995b8a640 10 monclient: _renew_subs 2024-09-11T01:45:47.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.228+0000 7f2995b8a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:45:47.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.228+0000 7f2995b8a640 1 -- 172.21.15.12:0/2166401808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2990134ea0 con 0x7f29900a08c0 2024-09-11T01:45:47.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.229+0000 7f2977fff640 1 -- 172.21.15.12:0/2166401808 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f2988003f40 con 0x7f29900a08c0 2024-09-11T01:45:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.229+0000 7f2995b8a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.229+0000 7f2995b8a640 1 --2- 172.21.15.12:0/2166401808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f299002d440 0x7f299002d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.229+0000 7f2995b8a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.229+0000 7f2995b8a640 1 -- 172.21.15.12:0/2166401808 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f29901bfd30 con 0x7f299002d440 2024-09-11T01:45:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.229+0000 7f2977fff640 1 --2- 172.21.15.12:0/2166401808 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f297c03e240 0x7f297c040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.229+0000 7f298f7fe640 1 --2- 172.21.15.12:0/2166401808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f299002d440 0x7f299002d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.229+0000 7f298f7fe640 10 monclient: get_auth_request con 0x7f299002d440 auth_method 0 2024-09-11T01:45:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.229+0000 7f298f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.229+0000 7f298f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.229+0000 7f298f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.229+0000 7f2977fff640 1 -- 172.21.15.12:0/2166401808 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(566..566 src has 317..566) ==== 4787+0+0 (secure 0 0 0) 0x7f2988053a20 con 0x7f29900a08c0 2024-09-11T01:45:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.229+0000 7f298f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.229+0000 7f298f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.229+0000 7f298f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:47.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.229+0000 7f298effd640 1 --2- 172.21.15.12:0/2166401808 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f297c03e240 0x7f297c040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.229+0000 7f298effd640 10 monclient: get_auth_request con 0x7f297c03e240 auth_method 0 2024-09-11T01:45:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.229+0000 7f298f7fe640 10 monclient(hunting): handle_auth_done global_id 134575 payload 824 2024-09-11T01:45:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.229+0000 7f298f7fe640 1 -- 172.21.15.12:0/2166401808 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f29901306e0 con 0x7f299002d440 2024-09-11T01:45:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.232+0000 7f298f7fe640 1 -- 172.21.15.12:0/2166401808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f299002d440 msgr2=0x7f299002d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.232+0000 7f298f7fe640 1 --2- 172.21.15.12:0/2166401808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f299002d440 0x7f299002d820 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2980034000 tx=0x7f29800046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.232+0000 7f298effd640 1 --2- 172.21.15.12:0/2166401808 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f297c03e240 0x7f297c040700 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f29901324c0 tx=0x7f2984009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.232+0000 7f2977fff640 1 -- 172.21.15.12:0/2166401808 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2980037040 con 0x7f299002d440 2024-09-11T01:45:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.232+0000 7f2977fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.232+0000 7f2977fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:45:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.232+0000 7f2977fff640 10 monclient: dump: 2024-09-11T01:45:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.247 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:47.248 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.232+0000 7f2977fff640 1 -- 172.21.15.12:0/2166401808 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f298003c850 con 0x7f299002d440 2024-09-11T01:45:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.232+0000 7f2977fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.232+0000 7f2977fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:45:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.232+0000 7f2977fff640 1 -- 172.21.15.12:0/2166401808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f299002d440 msgr2=0x7f299002d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.232+0000 7f2977fff640 1 --2- 172.21.15.12:0/2166401808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f299002d440 0x7f299002d820 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2980034000 tx=0x7f29800046c0 comp rx=0 tx=0).stop 2024-09-11T01:45:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.232+0000 7f2977fff640 1 -- 172.21.15.12:0/2166401808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f299002d440 msgr2=0x7f299002d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:47.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.232+0000 7f2977fff640 1 --2- 172.21.15.12:0/2166401808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f299002d440 0x7f299002d820 secure :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2980034000 tx=0x7f29800046c0 comp rx=0 tx=0).stop 2024-09-11T01:45:47.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.238+0000 7f2995b8a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:45:47.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.238+0000 7f2995b8a640 1 --2- 172.21.15.12:0/2166401808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f299002dd60 0x7f299002e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:47.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.238+0000 7f2995b8a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:47.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.238+0000 7f2995b8a640 1 -- 172.21.15.12:0/2166401808 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f29901c0250 con 0x7f299002dd60 2024-09-11T01:45:47.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.239+0000 7f298f7fe640 1 --2- 172.21.15.12:0/2166401808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f299002dd60 0x7f299002e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:47.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.239+0000 7f298f7fe640 10 monclient: get_auth_request con 0x7f299002dd60 auth_method 0 2024-09-11T01:45:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.239+0000 7f298f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.239+0000 7f298f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.239+0000 7f298f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.239+0000 7f298f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.239+0000 7f298f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.239+0000 7f298f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.239+0000 7f298f7fe640 10 monclient(hunting): handle_auth_done global_id 134578 payload 824 2024-09-11T01:45:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.239+0000 7f298f7fe640 1 -- 172.21.15.12:0/2166401808 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f299006d8c0 con 0x7f299002dd60 2024-09-11T01:45:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.242+0000 7f298f7fe640 1 -- 172.21.15.12:0/2166401808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f299002dd60 msgr2=0x7f299002e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.242+0000 7f298f7fe640 1 --2- 172.21.15.12:0/2166401808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f299002dd60 0x7f299002e140 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f298002a0f0 tx=0x7f2980009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.242+0000 7f2977fff640 1 -- 172.21.15.12:0/2166401808 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2980050070 con 0x7f299002dd60 2024-09-11T01:45:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.242+0000 7f2977fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.242+0000 7f2977fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:45:47.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.242+0000 7f2977fff640 10 monclient: dump: 2024-09-11T01:45:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:47.252 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.242+0000 7f2977fff640 1 -- 172.21.15.12:0/2166401808 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f2980049c70 con 0x7f299002dd60 2024-09-11T01:45:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.242+0000 7f2977fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:45:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.242+0000 7f2977fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:45:47.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.242+0000 7f2977fff640 1 -- 172.21.15.12:0/2166401808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f299002dd60 msgr2=0x7f299002e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.242+0000 7f2977fff640 1 --2- 172.21.15.12:0/2166401808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f299002dd60 0x7f299002e140 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f298002a0f0 tx=0x7f2980009240 comp rx=0 tx=0).stop 2024-09-11T01:45:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.242+0000 7f2977fff640 1 -- 172.21.15.12:0/2166401808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f299002dd60 msgr2=0x7f299002e140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:45:47.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.242+0000 7f2977fff640 1 --2- 172.21.15.12:0/2166401808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f299002dd60 0x7f299002e140 secure :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f298002a0f0 tx=0x7f2980009240 comp rx=0 tx=0).stop 2024-09-11T01:45:47.253 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:45:47.253 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:45:47.253 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:45:47.253 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:45:47.253 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 154, 2024-09-11T01:45:47.253 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 280203, 2024-09-11T01:45:47.254 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:45:47.254 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:45:47.254 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:45:47.254 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:45:47.254 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:47.254 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 4, 2024-09-11T01:45:47.254 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:45:47.254 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:45:47.254 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:45:47.254 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:47.254 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:47.254 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:47.254 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:47.254 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:47.255 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:47.255 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:47.255 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:47.255 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:47.255 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:47.255 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:47.255 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:47.255 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:45:47.255 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:45:47.255 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:47.255 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:47.255 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:47.255 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:47.255 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:47.256 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:47.256 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:47.256 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:47.256 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:47.256 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:47.256 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:47.256 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:47.256 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:47.256 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:45:47.256 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:45:47.256 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:45:47.256 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:45:47.256 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:45:47.257 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:45:47.257 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:45:47.257 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:45:47.257 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:45:47.257 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:45:47.257 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:45:47.257 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:45:47.257 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:45:47.257 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:45:47.257 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:45:47.257 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:45:47.257 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:45:47.257 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:45:47.257 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:45:47.258 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:45:47.258 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:45:47.258 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:45:47.258 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:45:47.258 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:45:47.258 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:45:47.258 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:45:47.258 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:45:47.258 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:45:47.258 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:47.258 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:45:47.258 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:47.258 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:45:47.259 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:47.259 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:45:47.259 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:45:47.259 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:47.259 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:47.259 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:47.259 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:47.259 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:45:47.259 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:47.259 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:47.259 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:47.259 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:47.259 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:45:47.259 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:47.260 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:47.260 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:47.260 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:47.260 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:45:47.260 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:45:47.260 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:47.260 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:47.260 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:47.260 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:47.260 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:47.260 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:45:47.260 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:45:47.260 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:47.261 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:47.261 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:47.261 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:47.261 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:45:47.261 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:47.261 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:47.261 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:47.261 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:47.261 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:45:47.261 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:47.261 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:47.261 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:47.261 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:47.262 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:45:47.262 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:45:47.262 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:47.262 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:47.262 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:47.262 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:47.262 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:47.262 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:45:47.262 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:45:47.262 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:45:47.262 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:45:47.262 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:47.262 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:45:47.262 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:45:47.263 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:47.263 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:47.263 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:47.263 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:45:47.263 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:45:47.263 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:45:47.263 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:47.263 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:47.263 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:47.263 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:45:47.263 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:45:47.263 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:45:47.263 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:45:47.263 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:45:47.264 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:47.264 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:47.264 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:47.264 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:45:47.264 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:45:47.264 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:47.264 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:47.264 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:47.264 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:45:47.264 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:47.264 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:47.264 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:45:47.264 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:47.265 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:47.265 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:47.265 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:45:47.265 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:47.265 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:47.265 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:45:47.265 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:47.265 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:47.265 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:47.265 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:45:47.265 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:47.265 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:45:47.265 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:45:47.265 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:45:47.266 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:45:47.266 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:45:47.266 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:45:47.266 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:45:47.266 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:45:47.266 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:45:47.266 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:45:47.266 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:45:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.242+0000 7f2995b8a640 1 -- 172.21.15.12:0/2166401808 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f297c03e240 msgr2=0x7f297c040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.242+0000 7f2995b8a640 1 --2- 172.21.15.12:0/2166401808 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f297c03e240 0x7f297c040700 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f29901324c0 tx=0x7f2984009190 comp rx=0 tx=0).stop 2024-09-11T01:45:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.242+0000 7f2995b8a640 10 monclient: shutdown 2024-09-11T01:45:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.242+0000 7f2995b8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.242+0000 7f2995b8a640 1 -- 172.21.15.12:0/2166401808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29900a08c0 msgr2=0x7f29901bd960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.242+0000 7f2995b8a640 1 --2- 172.21.15.12:0/2166401808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f29900a08c0 0x7f29901bd960 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f298800efc0 tx=0x7f298800c490 comp rx=0 tx=0).stop 2024-09-11T01:45:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.242+0000 7f298ffff640 1 -- 172.21.15.12:0/2166401808 reap_dead start 2024-09-11T01:45:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.242+0000 7f298ffff640 1 -- 172.21.15.12:0/2166401808 reap_dead start 2024-09-11T01:45:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.242+0000 7f2995b8a640 1 -- 172.21.15.12:0/2166401808 shutdown_connections 2024-09-11T01:45:47.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.242+0000 7f2995b8a640 1 -- 172.21.15.12:0/2166401808 >> 172.21.15.12:0/2166401808 conn(0x7f2990068950 msgr2=0x7f29900730a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:47.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.242+0000 7f2995b8a640 1 -- 172.21.15.12:0/2166401808 shutdown_connections 2024-09-11T01:45:47.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.242+0000 7f2995b8a640 1 -- 172.21.15.12:0/2166401808 wait complete. 2024-09-11T01:45:47.269 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:45:47.269 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-11T01:45:47.269 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T01:45:47.269 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-11T01:45:47.270 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-11T01:45:47.270 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T01:45:47.270 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:45:47.271 INFO:tasks.rados.rados.0.smithi071.stdout:2521: finishing write tid 2 to smithi07134777-27 2024-09-11T01:45:47.271 INFO:tasks.rados.rados.0.smithi071.stdout:2521: finishing write tid 3 to smithi07134777-27 2024-09-11T01:45:47.271 INFO:tasks.rados.rados.0.smithi071.stdout:2521: finishing write tid 4 to smithi07134777-27 2024-09-11T01:45:47.271 INFO:tasks.rados.rados.0.smithi071.stdout:2521: oid 27 updating version 0 to 1452 2024-09-11T01:45:47.271 INFO:tasks.rados.rados.0.smithi071.stdout:2521: oid 27 version 1452 is already newer than 1450 2024-09-11T01:45:47.271 INFO:tasks.rados.rados.0.smithi071.stdout:2521: oid 27 version 1452 is already newer than 1451 2024-09-11T01:45:47.271 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 1452 (ObjNum 901 snap 283 seq_num 901) dirty exists 2024-09-11T01:45:47.271 INFO:tasks.rados.rados.0.smithi071.stdout:2521: left oid 27 (ObjNum 901 snap 283 seq_num 901) 2024-09-11T01:45:47.271 INFO:tasks.rados.rados.0.smithi071.stdout:2520: done (3 left) 2024-09-11T01:45:47.271 INFO:tasks.rados.rados.0.smithi071.stdout:2521: done (2 left) 2024-09-11T01:45:47.271 INFO:tasks.rados.rados.0.smithi071.stdout:2522: done (1 left) 2024-09-11T01:45:47.272 INFO:tasks.rados.rados.0.smithi071.stdout:2523: done (0 left) 2024-09-11T01:45:47.272 INFO:tasks.rados.rados.0.smithi071.stdout:2524: rollback oid 40 current snap is 283 2024-09-11T01:45:47.272 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 40 to 274 2024-09-11T01:45:47.272 INFO:tasks.rados.rados.0.smithi071.stdout:2525: delete oid 21 current snap is 283 2024-09-11T01:45:47.272 INFO:tasks.rados.rados.0.smithi071.stdout:2524: finishing rollback tid 0 to smithi07134777-40 2024-09-11T01:45:47.272 INFO:tasks.rados.rados.0.smithi071.stdout:2524: finishing rollback tid 1 to smithi07134777-40 2024-09-11T01:45:47.272 INFO:tasks.rados.rados.0.smithi071.stdout:2525: done (1 left) 2024-09-11T01:45:47.272 INFO:tasks.rados.rados.0.smithi071.stdout:2526: snap_create 2024-09-11T01:45:47.272 INFO:tasks.rados.rados.0.smithi071.stdout:2524: finishing rollback tid 2 to smithi07134777-40 2024-09-11T01:45:47.272 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 1540 (ObjNum 865 snap 266 seq_num 865) dirty exists 2024-09-11T01:45:47.281 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T01:45:47.281 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:45:47.281 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:45:47.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.419+0000 7f261bf9b640 10 monclient: get_monmap_and_config 2024-09-11T01:45:47.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.419+0000 7f261bf9b640 10 monclient: build_initial_monmap 2024-09-11T01:45:47.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.419+0000 7f261bf9b640 10 monclient: monmap: 2024-09-11T01:45:47.421 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:47.421 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:47.421 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:45:47.420610+0000 2024-09-11T01:45:47.421 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:45:47.420610+0000 2024-09-11T01:45:47.422 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:47.422 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:47.422 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:45:47.422 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:45:47.422 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:45:47.422 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:45:47.422 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:45:47.422 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:47.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.420+0000 7f261bf9b640 1 Processor -- start 2024-09-11T01:45:47.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.420+0000 7f261bf9b640 1 -- start start 2024-09-11T01:45:47.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.420+0000 7f261bf9b640 10 monclient: init 2024-09-11T01:45:47.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f261bf9b640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:47.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f261bf9b640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-11T01:45:47.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f261bf9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f26140a2e90 0x7f261406dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:47.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f261bf9b640 10 monclient(hunting): picked mon.noname-b con 0x7f26140a2e90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:47.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f261bf9b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f26140a3270 addr v1:172.21.15.12:6789/0 2024-09-11T01:45:47.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f261bf9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f261413d490 0x7f261413f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:47.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f261bf9b640 10 monclient(hunting): picked mon.noname-a con 0x7f261413d490 addr v2:172.21.15.12:3300/0 2024-09-11T01:45:47.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f261bf9b640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26140a46c0 con 0x7f26140a3270 2024-09-11T01:45:47.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f261bf9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:47.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f261bf9b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f26140a4900 con 0x7f261413d490 2024-09-11T01:45:47.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f261bf9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:47.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f261bf9b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f261406ee50 con 0x7f26140a2e90 2024-09-11T01:45:47.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f261bf9b640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:47.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f261bf9b640 10 monclient(hunting): authenticate will time out at 1753.863229s 2024-09-11T01:45:47.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f2619536640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7f26140a3270 legacy=0x7f26140a3650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:45:47.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f261a538640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7f261413d490 msgr2=0x7f261413f880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:45:47.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f261a538640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f261413d490 0x7f261413f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:47.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f2619d37640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f26140a2e90 0x7f261406dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:47.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f2619d37640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f26140a2e90 0x7f261406dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52070/0 (socket says 172.21.15.12:52070) 2024-09-11T01:45:47.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f2619d37640 1 -- 172.21.15.12:0/3324431638 learned_addr learned my addr 172.21.15.12:0/3324431638 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:47.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f2619d37640 10 monclient(hunting): get_auth_request con 0x7f26140a2e90 auth_method 0 2024-09-11T01:45:47.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f2619d37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:47.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f2619d37640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:47.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f2619d37640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:47.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f2619d37640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:47.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f2619d37640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:47.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f2619d37640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:47.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f2619d37640 10 monclient(hunting): handle_auth_done global_id 184145 payload 274 2024-09-11T01:45:47.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f2619d37640 1 -- 172.21.15.12:0/3324431638 >> v2:172.21.15.12:3300/0 conn(0x7f261413d490 msgr2=0x7f261413f880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:45:47.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f2619d37640 1 --2- 172.21.15.12:0/3324431638 >> v2:172.21.15.12:3300/0 conn(0x7f261413d490 0x7f261413f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:47.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f2619d37640 1 -- 172.21.15.12:0/3324431638 >> v1:172.21.15.12:6789/0 conn(0x7f26140a3270 legacy=0x7f26140a3650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:45:47.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f2619d37640 10 monclient: _finish_hunting 0 2024-09-11T01:45:47.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f2619d37640 1 monclient: found mon.noname-b 2024-09-11T01:45:47.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f2619d37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:47.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f2619d37640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:45:47.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.421+0000 7f2619d37640 1 -- 172.21.15.12:0/3324431638 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f261406eb70 con 0x7f26140a2e90 2024-09-11T01:45:47.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.424+0000 7f2619d37640 1 -- 172.21.15.12:0/3324431638 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f26140a2e90 msgr2=0x7f261406dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:47.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.424+0000 7f2619d37640 1 --2- 172.21.15.12:0/3324431638 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f26140a2e90 0x7f261406dc60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f26140a4a80 tx=0x7f261002a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3f91b8ddf1f1b61 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:47.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.424+0000 7f2618d35640 1 -- 172.21.15.12:0/3324431638 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2610037070 con 0x7f26140a2e90 2024-09-11T01:45:47.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.424+0000 7f2618d35640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:47.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.424+0000 7f2618d35640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:45:47.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.424+0000 7f2618d35640 10 monclient: dump: 2024-09-11T01:45:47.433 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:47.433 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:47.433 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.433 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.433 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:47.433 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:47.433 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:47.433 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:47.433 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:47.433 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:47.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.424+0000 7f2618d35640 10 monclient: _finish_auth 0 2024-09-11T01:45:47.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.424+0000 7f2618d35640 10 monclient: _check_auth_tickets 2024-09-11T01:45:47.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.424+0000 7f2618d35640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:47.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.424+0000 7f2618d35640 1 -- 172.21.15.12:0/3324431638 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2610032640 con 0x7f26140a2e90 2024-09-11T01:45:47.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.424+0000 7f261bf9b640 5 monclient: authenticate success, global_id 184145 2024-09-11T01:45:47.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.424+0000 7f2618d35640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:47.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.424+0000 7f2618d35640 1 -- 172.21.15.12:0/3324431638 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2610032a10 con 0x7f26140a2e90 2024-09-11T01:45:47.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.424+0000 7f2618d35640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:47.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.424+0000 7f2618d35640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:45:47.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.424+0000 7f2618d35640 10 monclient: dump: 2024-09-11T01:45:47.434 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:47.434 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:47.434 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.434 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.434 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:47.435 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:47.435 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:47.435 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:47.435 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:47.435 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:47.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.424+0000 7f261bf9b640 10 monclient: get_monmap_and_config success 2024-09-11T01:45:47.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.424+0000 7f261bf9b640 4 set_mon_vals no callback set 2024-09-11T01:45:47.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.425+0000 7f261bf9b640 10 monclient: shutdown 2024-09-11T01:45:47.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.425+0000 7f261bf9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:47.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.425+0000 7f261bf9b640 1 -- 172.21.15.12:0/3324431638 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f26140a2e90 msgr2=0x7f261406dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:47.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.425+0000 7f261bf9b640 1 --2- 172.21.15.12:0/3324431638 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f26140a2e90 0x7f261406dc60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f26140a4a80 tx=0x7f261002a0a0 comp rx=0 tx=0).stop 2024-09-11T01:45:47.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.425+0000 7f261bf9b640 1 -- 172.21.15.12:0/3324431638 shutdown_connections 2024-09-11T01:45:47.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.425+0000 7f261bf9b640 1 --2- 172.21.15.12:0/3324431638 >> v2:172.21.15.12:3300/0 conn(0x7f261413d490 0x7f261413f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:47.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.425+0000 7f261bf9b640 1 --2- 172.21.15.12:0/3324431638 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f26140a2e90 0x7f261406dc60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:47.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.425+0000 7f261bf9b640 1 -- 172.21.15.12:0/3324431638 >> 172.21.15.12:0/3324431638 conn(0x7f26140a8490 msgr2=0x7f26140a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:47.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.425+0000 7f261bf9b640 1 -- 172.21.15.12:0/3324431638 shutdown_connections 2024-09-11T01:45:47.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.425+0000 7f261bf9b640 1 -- 172.21.15.12:0/3324431638 wait complete. 2024-09-11T01:45:47.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.425+0000 7f261bf9b640 10 monclient: build_initial_monmap 2024-09-11T01:45:47.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.425+0000 7f261bf9b640 10 monclient: monmap: 2024-09-11T01:45:47.436 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:47.436 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:45:47.436 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:45:47.436 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:45:47.436 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:47.436 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:47.436 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:45:47.436 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:45:47.436 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:45:47.436 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:47.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f261bf9b640 1 Processor -- start 2024-09-11T01:45:47.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f261bf9b640 1 -- start start 2024-09-11T01:45:47.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f261bf9b640 10 monclient: init 2024-09-11T01:45:47.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f261bf9b640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:47.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f261bf9b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:45:47.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f261bf9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f26140a2e90 0x7f261409c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:47.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f261bf9b640 10 monclient(hunting): picked mon.noname-b con 0x7f26140a2e90 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:45:47.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f261bf9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f26140a3270 0x7f261409c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:47.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f261bf9b640 10 monclient(hunting): picked mon.noname-c con 0x7f26140a3270 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:47.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f261bf9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f261413d490 0x7f261409ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:47.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f261bf9b640 10 monclient(hunting): picked mon.noname-a con 0x7f261413d490 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:45:47.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f261bf9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:47.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f261bf9b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f261406fd20 con 0x7f261413d490 2024-09-11T01:45:47.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f261bf9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:47.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f261bf9b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f261406fba0 con 0x7f26140a2e90 2024-09-11T01:45:47.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f261bf9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:47.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f261bf9b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f261413fdc0 con 0x7f26140a3270 2024-09-11T01:45:47.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f261bf9b640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:47.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f261bf9b640 10 monclient(hunting): authenticate will time out at 1753.868746s 2024-09-11T01:45:47.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f261a538640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f261413d490 msgr2=0x7f261409ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:45:47.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f261a538640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f261413d490 0x7f261409ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:47.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f2619536640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f26140a3270 0x7f261409c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:47.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f2619536640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f26140a3270 0x7f261409c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52080/0 (socket says 172.21.15.12:52080) 2024-09-11T01:45:47.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f2619536640 1 -- 172.21.15.12:0/1935976889 learned_addr learned my addr 172.21.15.12:0/1935976889 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:47.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f2619536640 10 monclient(hunting): get_auth_request con 0x7f26140a3270 auth_method 0 2024-09-11T01:45:47.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f2619536640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:47.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f2619536640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:47.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f2619536640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:47.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f2619536640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:47.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f2619536640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:47.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.426+0000 7f2619536640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:47.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.427+0000 7f2619d37640 1 --2- 172.21.15.12:0/1935976889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f26140a2e90 0x7f261409c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:47.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.427+0000 7f2619536640 10 monclient(hunting): handle_auth_done global_id 184148 payload 824 2024-09-11T01:45:47.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.427+0000 7f2619536640 1 -- 172.21.15.12:0/1935976889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f26140a2e90 msgr2=0x7f261409c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:47.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.427+0000 7f2619536640 1 --2- 172.21.15.12:0/1935976889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f26140a2e90 0x7f261409c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:47.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.427+0000 7f2619536640 1 -- 172.21.15.12:0/1935976889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f261413d490 msgr2=0x7f261409ce10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:45:47.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.427+0000 7f2619536640 1 --2- 172.21.15.12:0/1935976889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f261413d490 0x7f261409ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:47.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.427+0000 7f2619536640 10 monclient: _finish_hunting 0 2024-09-11T01:45:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.427+0000 7f2619536640 1 monclient: found mon.noname-c 2024-09-11T01:45:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.427+0000 7f2619536640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.427+0000 7f2619536640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:45:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.427+0000 7f2619536640 1 -- 172.21.15.12:0/1935976889 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f261409f1d0 con 0x7f26140a3270 2024-09-11T01:45:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.429+0000 7f2619536640 1 -- 172.21.15.12:0/1935976889 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f26140a3270 msgr2=0x7f261409c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.429+0000 7f2619536640 1 --2- 172.21.15.12:0/1935976889 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f26140a3270 0x7f261409c8d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f260400b550 tx=0x7f260400ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.429+0000 7f260a7fc640 1 -- 172.21.15.12:0/1935976889 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2604005070 con 0x7f26140a3270 2024-09-11T01:45:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.429+0000 7f260a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.429+0000 7f260a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:45:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.429+0000 7f260a7fc640 10 monclient: dump: 2024-09-11T01:45:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.440 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:47.441 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:47.441 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:47.441 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:47.441 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:47.441 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:47.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.429+0000 7f260a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:45:47.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.429+0000 7f260a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:45:47.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.429+0000 7f260a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:47.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.429+0000 7f261bf9b640 5 monclient: authenticate success, global_id 184148 2024-09-11T01:45:47.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.429+0000 7f260a7fc640 1 -- 172.21.15.12:0/1935976889 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2604015070 con 0x7f26140a3270 2024-09-11T01:45:47.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.429+0000 7f261bf9b640 10 monclient: _renew_subs 2024-09-11T01:45:47.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.429+0000 7f261bf9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:45:47.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.429+0000 7f261bf9b640 1 -- 172.21.15.12:0/1935976889 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26141396c0 con 0x7f26140a3270 2024-09-11T01:45:47.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.429+0000 7f260a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.429+0000 7f260a7fc640 1 -- 172.21.15.12:0/1935976889 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2604005450 con 0x7f26140a3270 2024-09-11T01:45:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.429+0000 7f260a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.429+0000 7f260a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:45:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.429+0000 7f260a7fc640 10 monclient: dump: 2024-09-11T01:45:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:47.442 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:47.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.429+0000 7f261bf9b640 10 monclient: _renew_subs 2024-09-11T01:45:47.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.429+0000 7f261bf9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:45:47.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.429+0000 7f260bfff640 4 set_mon_vals no callback set 2024-09-11T01:45:47.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.429+0000 7f261bf9b640 1 -- 172.21.15.12:0/1935976889 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f261409eef0 con 0x7f26140a3270 2024-09-11T01:45:47.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.430+0000 7f260a7fc640 1 -- 172.21.15.12:0/1935976889 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f26040055f0 con 0x7f26140a3270 2024-09-11T01:45:47.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.430+0000 7f261bf9b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:47.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.430+0000 7f261bf9b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:47.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.430+0000 7f261bf9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:45:47.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.430+0000 7f261bf9b640 1 -- 172.21.15.12:0/1935976889 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f261406dc60 con 0x7f26140a3270 2024-09-11T01:45:47.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.430+0000 7f260a7fc640 1 --2- 172.21.15.12:0/1935976889 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f25f803e1c0 0x7f25f8040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:47.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.430+0000 7f260a7fc640 1 -- 172.21.15.12:0/1935976889 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(566..566 src has 317..566) ==== 4787+0+0 (secure 0 0 0) 0x7f2604052d00 con 0x7f26140a3270 2024-09-11T01:45:47.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.431+0000 7f2619d37640 1 --2- 172.21.15.12:0/1935976889 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f25f803e1c0 0x7f25f8040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:47.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.431+0000 7f2619d37640 10 monclient: get_auth_request con 0x7f25f803e1c0 auth_method 0 2024-09-11T01:45:47.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.433+0000 7f2619d37640 1 --2- 172.21.15.12:0/1935976889 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f25f803e1c0 0x7f25f8040680 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f26140a92c0 tx=0x7f261002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:47.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.435+0000 7f260a7fc640 1 -- 172.21.15.12:0/1935976889 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f26040516f0 con 0x7f26140a3270 2024-09-11T01:45:47.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.435+0000 7f260a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:47.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.435+0000 7f260a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:45:47.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.692+0000 7f261bf9b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:45:47.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.692+0000 7f261bf9b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:45:47.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.692+0000 7f261bf9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:45:47.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.692+0000 7f261bf9b640 1 -- 172.21.15.12:0/1935976889 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f26140698e0 con 0x7f26140a3270 2024-09-11T01:45:47.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:47.692+0000 7f2619536640 1 --2- 172.21.15.12:0/1935976889 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f26140a3270 0x7f261409c8d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f260400b550 tx=0x7f260400ba10 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7f260401a460 mon_command_ack([{"prefix": "quorum_status"}]=0 v0) 2024-09-11T01:45:48.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:48.521+0000 7f260a7fc640 1 -- 172.21.15.12:0/1935976889 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f260401a460 con 0x7f26140a3270 2024-09-11T01:45:48.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:48.521+0000 7f260a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:45:48.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:48.521+0000 7f260a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:45:48.522 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":154,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:45:48.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:48.524+0000 7f261bf9b640 1 -- 172.21.15.12:0/1935976889 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f25f803e1c0 msgr2=0x7f25f8040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:48.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:48.524+0000 7f261bf9b640 1 --2- 172.21.15.12:0/1935976889 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f25f803e1c0 0x7f25f8040680 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f26140a92c0 tx=0x7f261002e000 comp rx=0 tx=0).stop 2024-09-11T01:45:48.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:48.524+0000 7f261bf9b640 10 monclient: shutdown 2024-09-11T01:45:48.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:48.524+0000 7f261bf9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:48.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:48.524+0000 7f261bf9b640 1 -- 172.21.15.12:0/1935976889 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f26140a3270 msgr2=0x7f261409c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:48.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:48.524+0000 7f261bf9b640 1 --2- 172.21.15.12:0/1935976889 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f26140a3270 0x7f261409c8d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f260400b550 tx=0x7f260400ba10 comp rx=0 tx=0).stop 2024-09-11T01:45:48.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:48.524+0000 7f261bf9b640 1 -- 172.21.15.12:0/1935976889 shutdown_connections 2024-09-11T01:45:48.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:48.524+0000 7f261bf9b640 1 --2- 172.21.15.12:0/1935976889 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f25f803e1c0 0x7f25f8040680 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:48.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:48.524+0000 7f261bf9b640 1 --2- 172.21.15.12:0/1935976889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f261413d490 0x7f261409ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:48.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:48.524+0000 7f261bf9b640 1 --2- 172.21.15.12:0/1935976889 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f26140a3270 0x7f261409c8d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:48.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:48.524+0000 7f261bf9b640 1 --2- 172.21.15.12:0/1935976889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f26140a2e90 0x7f261409c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:48.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:48.524+0000 7f261bf9b640 1 -- 172.21.15.12:0/1935976889 >> 172.21.15.12:0/1935976889 conn(0x7f26140a8490 msgr2=0x7f261413cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:48.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:48.524+0000 7f261bf9b640 1 -- 172.21.15.12:0/1935976889 shutdown_connections 2024-09-11T01:45:48.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:48.524+0000 7f261bf9b640 1 -- 172.21.15.12:0/1935976889 wait complete. 2024-09-11T01:45:51.541 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:45:51.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.652+0000 7f673cf16640 10 monclient: get_monmap_and_config 2024-09-11T01:45:51.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.652+0000 7f673cf16640 10 monclient: build_initial_monmap 2024-09-11T01:45:51.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.652+0000 7f673cf16640 10 monclient: monmap: 2024-09-11T01:45:51.654 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:51.654 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:51.654 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:45:51.653405+0000 2024-09-11T01:45:51.654 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:45:51.653405+0000 2024-09-11T01:45:51.654 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:51.654 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:51.654 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:45:51.655 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:45:51.655 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:45:51.655 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:45:51.655 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:45:51.655 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:51.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.653+0000 7f673cf16640 1 Processor -- start 2024-09-11T01:45:51.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.653+0000 7f673cf16640 1 -- start start 2024-09-11T01:45:51.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.653+0000 7f673cf16640 10 monclient: init 2024-09-11T01:45:51.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.653+0000 7f673cf16640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:51.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f673cf16640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-11T01:45:51.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f673cf16640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f67380a2e90 0x7f673806dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:51.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f673cf16640 10 monclient(hunting): picked mon.noname-a con 0x7f67380a2e90 addr v2:172.21.15.12:3300/0 2024-09-11T01:45:51.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f673cf16640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f67380a3270 0x7f67380a3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:51.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f673cf16640 10 monclient(hunting): picked mon.noname-c con 0x7f67380a3270 addr v2:172.21.15.71:3300/0 2024-09-11T01:45:51.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f673cf16640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6738138040 addr v1:172.21.15.12:6789/0 2024-09-11T01:45:51.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f673cf16640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67380a46c0 con 0x7f6738138040 2024-09-11T01:45:51.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f673cf16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:51.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f673cf16640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f67380a4900 con 0x7f67380a2e90 2024-09-11T01:45:51.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f673cf16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:51.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f673cf16640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f673806ee50 con 0x7f67380a3270 2024-09-11T01:45:51.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f673cf16640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:51.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f673cf16640 10 monclient(hunting): authenticate will time out at 1758.096121s 2024-09-11T01:45:51.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f6736d9d640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7f6738138040 legacy=0x7f67381384d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:45:51.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f673659c640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7f67380a2e90 msgr2=0x7f673806dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:45:51.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f673659c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f67380a2e90 0x7f673806dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:51.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f6735d9b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f67380a3270 0x7f67380a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:51.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f6735d9b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f67380a3270 0x7f67380a3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:56152/0 (socket says 172.21.15.12:56152) 2024-09-11T01:45:51.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f6735d9b640 1 -- 172.21.15.12:0/2481085423 learned_addr learned my addr 172.21.15.12:0/2481085423 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:51.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f6735d9b640 10 monclient(hunting): get_auth_request con 0x7f67380a3270 auth_method 0 2024-09-11T01:45:51.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f6735d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:51.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f6735d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:51.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f6735d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:51.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f6735d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:51.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f6735d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:51.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f6735d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:51.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f6735d9b640 10 monclient(hunting): handle_auth_done global_id 134581 payload 274 2024-09-11T01:45:51.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f6735d9b640 1 -- 172.21.15.12:0/2481085423 >> v2:172.21.15.12:3300/0 conn(0x7f67380a2e90 msgr2=0x7f673806dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:45:51.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f6735d9b640 1 --2- 172.21.15.12:0/2481085423 >> v2:172.21.15.12:3300/0 conn(0x7f67380a2e90 0x7f673806dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:51.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f6735d9b640 1 -- 172.21.15.12:0/2481085423 >> v1:172.21.15.12:6789/0 conn(0x7f6738138040 legacy=0x7f67381384d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:45:51.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f6735d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:45:51.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f6735d9b640 1 monclient: found mon.noname-c 2024-09-11T01:45:51.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f6735d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:51.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f6735d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:45:51.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.654+0000 7f6735d9b640 1 -- 172.21.15.12:0/2481085423 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f673806eb70 con 0x7f67380a3270 2024-09-11T01:45:51.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.657+0000 7f6735d9b640 1 -- 172.21.15.12:0/2481085423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f67380a3270 msgr2=0x7f67380a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:51.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.657+0000 7f6735d9b640 1 --2- 172.21.15.12:0/2481085423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f67380a3270 0x7f67380a3680 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6738071820 tx=0x7f672002a0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e26f54926e3d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:51.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.658+0000 7f673559a640 1 -- 172.21.15.12:0/2481085423 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6720037070 con 0x7f67380a3270 2024-09-11T01:45:51.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.658+0000 7f673559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:51.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.658+0000 7f673559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:45:51.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.658+0000 7f673559a640 10 monclient: dump: 2024-09-11T01:45:51.665 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:51.665 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:51.665 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:51.665 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:51.665 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:51.665 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:51.665 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:51.665 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:51.665 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:51.666 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:51.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.658+0000 7f673559a640 10 monclient: _finish_auth 0 2024-09-11T01:45:51.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.658+0000 7f673559a640 10 monclient: _check_auth_tickets 2024-09-11T01:45:51.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.658+0000 7f673559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:51.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.658+0000 7f673cf16640 5 monclient: authenticate success, global_id 134581 2024-09-11T01:45:51.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.658+0000 7f673559a640 1 -- 172.21.15.12:0/2481085423 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6720032640 con 0x7f67380a3270 2024-09-11T01:45:51.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.658+0000 7f673559a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:51.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.658+0000 7f673cf16640 10 monclient: get_monmap_and_config success 2024-09-11T01:45:51.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.658+0000 7f673559a640 1 -- 172.21.15.12:0/2481085423 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6720032a10 con 0x7f67380a3270 2024-09-11T01:45:51.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.658+0000 7f673559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:51.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.658+0000 7f673cf16640 4 set_mon_vals no callback set 2024-09-11T01:45:51.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.658+0000 7f673559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:45:51.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.658+0000 7f673559a640 10 monclient: dump: 2024-09-11T01:45:51.667 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:51.667 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:51.667 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:51.667 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:51.667 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:51.667 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:51.667 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:51.667 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:51.667 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:51.667 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:51.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.659+0000 7f673cf16640 10 monclient: shutdown 2024-09-11T01:45:51.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.659+0000 7f673cf16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:51.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.659+0000 7f673cf16640 1 -- 172.21.15.12:0/2481085423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f67380a3270 msgr2=0x7f67380a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:51.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.659+0000 7f673cf16640 1 --2- 172.21.15.12:0/2481085423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f67380a3270 0x7f67380a3680 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6738071820 tx=0x7f672002a0d0 comp rx=0 tx=0).stop 2024-09-11T01:45:51.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.659+0000 7f673cf16640 1 -- 172.21.15.12:0/2481085423 shutdown_connections 2024-09-11T01:45:51.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.659+0000 7f673cf16640 1 --2- 172.21.15.12:0/2481085423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f67380a3270 0x7f67380a3680 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:51.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.659+0000 7f673cf16640 1 --2- 172.21.15.12:0/2481085423 >> v2:172.21.15.12:3300/0 conn(0x7f67380a2e90 0x7f673806dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:51.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.659+0000 7f673cf16640 1 -- 172.21.15.12:0/2481085423 >> 172.21.15.12:0/2481085423 conn(0x7f67380a8490 msgr2=0x7f67380a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:51.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.659+0000 7f673cf16640 1 -- 172.21.15.12:0/2481085423 shutdown_connections 2024-09-11T01:45:51.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.659+0000 7f673cf16640 1 -- 172.21.15.12:0/2481085423 wait complete. 2024-09-11T01:45:51.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.660+0000 7f673cf16640 10 monclient: build_initial_monmap 2024-09-11T01:45:51.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.660+0000 7f673cf16640 10 monclient: monmap: 2024-09-11T01:45:51.668 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:51.668 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:45:51.668 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:45:51.668 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:45:51.668 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:51.669 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:51.669 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:45:51.669 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:45:51.669 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:45:51.669 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:51.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.660+0000 7f673cf16640 1 Processor -- start 2024-09-11T01:45:51.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.660+0000 7f673cf16640 1 -- start start 2024-09-11T01:45:51.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.660+0000 7f673cf16640 10 monclient: init 2024-09-11T01:45:51.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.660+0000 7f673cf16640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:51.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.660+0000 7f673cf16640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:45:51.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.660+0000 7f673cf16640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67380a2e90 0x7f67381cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:51.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.660+0000 7f673cf16640 10 monclient(hunting): picked mon.noname-a con 0x7f67380a2e90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:45:51.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.660+0000 7f673cf16640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f67380a3270 0x7f67381cda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:51.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f673cf16640 10 monclient(hunting): picked mon.noname-c con 0x7f67380a3270 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:51.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f673cf16640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6738138040 0x7f6738124910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:51.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f673cf16640 10 monclient(hunting): picked mon.noname-b con 0x7f6738138040 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:45:51.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f673cf16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:51.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f673cf16640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f673806fba0 con 0x7f67380a2e90 2024-09-11T01:45:51.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f673cf16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:51.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f673cf16640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f67380a3bc0 con 0x7f6738138040 2024-09-11T01:45:51.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f673cf16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:51.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f673cf16640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f673806fd20 con 0x7f67380a3270 2024-09-11T01:45:51.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f673cf16640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:51.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f673659c640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67380a2e90 msgr2=0x7f67381cd4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:45:51.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f673cf16640 10 monclient(hunting): authenticate will time out at 1758.103051s 2024-09-11T01:45:51.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f673659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67380a2e90 0x7f67381cd4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f6735d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f67380a3270 0x7f67381cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f6735d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f67380a3270 0x7f67381cda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52086/0 (socket says 172.21.15.12:52086) 2024-09-11T01:45:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f6735d9b640 1 -- 172.21.15.12:0/2373475112 learned_addr learned my addr 172.21.15.12:0/2373475112 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f6735d9b640 10 monclient(hunting): get_auth_request con 0x7f67380a3270 auth_method 0 2024-09-11T01:45:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f6735d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f6735d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f6735d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f6735d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f6735d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f6735d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f6736d9d640 1 --2- 172.21.15.12:0/2373475112 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6738138040 0x7f6738124910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f6736d9d640 10 monclient(hunting): get_auth_request con 0x7f6738138040 auth_method 0 2024-09-11T01:45:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f6736d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:51.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f6736d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:51.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f6736d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:51.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f6735d9b640 10 monclient(hunting): handle_auth_done global_id 184151 payload 824 2024-09-11T01:45:51.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f6735d9b640 1 -- 172.21.15.12:0/2373475112 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6738138040 msgr2=0x7f6738124910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:51.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f6735d9b640 1 --2- 172.21.15.12:0/2373475112 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6738138040 0x7f6738124910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:51.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f6735d9b640 1 -- 172.21.15.12:0/2373475112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67380a2e90 msgr2=0x7f67381cd4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:45:51.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f6735d9b640 1 --2- 172.21.15.12:0/2373475112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67380a2e90 0x7f67381cd4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:51.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f6735d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:45:51.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f6735d9b640 1 monclient: found mon.noname-c 2024-09-11T01:45:51.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f6735d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:51.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f6735d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:45:51.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.661+0000 7f6735d9b640 1 -- 172.21.15.12:0/2373475112 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6738128730 con 0x7f67380a3270 2024-09-11T01:45:51.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.663+0000 7f6735d9b640 1 -- 172.21.15.12:0/2373475112 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f67380a3270 msgr2=0x7f67381cda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:51.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.663+0000 7f6735d9b640 1 --2- 172.21.15.12:0/2373475112 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f67380a3270 0x7f67381cda20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6720009960 tx=0x7f67200046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.663+0000 7f672effd640 1 -- 172.21.15.12:0/2373475112 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6720041070 con 0x7f67380a3270 2024-09-11T01:45:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.663+0000 7f672effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.663+0000 7f672effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:45:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.663+0000 7f672effd640 10 monclient: dump: 2024-09-11T01:45:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:51.675 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:51.676 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:51.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.663+0000 7f672effd640 10 monclient: _finish_auth 0 2024-09-11T01:45:51.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.663+0000 7f672effd640 10 monclient: _check_auth_tickets 2024-09-11T01:45:51.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.663+0000 7f672effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:51.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.663+0000 7f673cf16640 5 monclient: authenticate success, global_id 184151 2024-09-11T01:45:51.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.663+0000 7f673cf16640 10 monclient: _renew_subs 2024-09-11T01:45:51.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.663+0000 7f673cf16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:45:51.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.663+0000 7f673cf16640 1 -- 172.21.15.12:0/2373475112 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f673809f680 con 0x7f67380a3270 2024-09-11T01:45:51.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.664+0000 7f673cf16640 10 monclient: _renew_subs 2024-09-11T01:45:51.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.664+0000 7f673cf16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:45:51.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.664+0000 7f673cf16640 1 -- 172.21.15.12:0/2373475112 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6738127e90 con 0x7f67380a3270 2024-09-11T01:45:51.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.664+0000 7f672effd640 1 -- 172.21.15.12:0/2373475112 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f672003c880 con 0x7f67380a3270 2024-09-11T01:45:51.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.664+0000 7f672effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:51.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.664+0000 7f672effd640 1 -- 172.21.15.12:0/2373475112 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6720037040 con 0x7f67380a3270 2024-09-11T01:45:51.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.664+0000 7f672effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:51.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.664+0000 7f672effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:45:51.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.664+0000 7f672effd640 10 monclient: dump: 2024-09-11T01:45:51.677 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:51.677 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:51.677 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:51.677 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:51.677 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:51.677 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:51.677 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:51.677 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:51.677 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:51.677 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:51.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.664+0000 7f6734d99640 4 set_mon_vals no callback set 2024-09-11T01:45:51.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.665+0000 7f672effd640 1 -- 172.21.15.12:0/2373475112 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f672004e060 con 0x7f67380a3270 2024-09-11T01:45:51.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.665+0000 7f673cf16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:51.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.665+0000 7f673cf16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:51.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.665+0000 7f673cf16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:45:51.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.665+0000 7f673cf16640 1 -- 172.21.15.12:0/2373475112 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6738127bc0 con 0x7f67380a3270 2024-09-11T01:45:51.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.665+0000 7f672effd640 1 --2- 172.21.15.12:0/2373475112 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f671803e1c0 0x7f6718040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:51.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.665+0000 7f672effd640 1 -- 172.21.15.12:0/2373475112 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(566..566 src has 317..566) ==== 4787+0+0 (secure 0 0 0) 0x7f6720073210 con 0x7f67380a3270 2024-09-11T01:45:51.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.665+0000 7f673659c640 1 --2- 172.21.15.12:0/2373475112 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f671803e1c0 0x7f6718040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:51.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.665+0000 7f673659c640 10 monclient: get_auth_request con 0x7f671803e1c0 auth_method 0 2024-09-11T01:45:51.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.668+0000 7f673659c640 1 --2- 172.21.15.12:0/2373475112 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f671803e1c0 0x7f6718040680 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f67280038b0 tx=0x7f6728006d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:51.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.670+0000 7f672effd640 1 -- 172.21.15.12:0/2373475112 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f672003ac60 con 0x7f67380a3270 2024-09-11T01:45:51.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.670+0000 7f672effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:51.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.670+0000 7f672effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:45:51.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.928+0000 7f673cf16640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:45:51.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.928+0000 7f673cf16640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:45:51.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.928+0000 7f673cf16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:45:51.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.928+0000 7f673cf16640 1 -- 172.21.15.12:0/2373475112 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6738138c70 con 0x7f67380a3270 2024-09-11T01:45:51.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.929+0000 7f672effd640 1 -- 172.21.15.12:0/2373475112 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f6720071c00 con 0x7f67380a3270 2024-09-11T01:45:51.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.929+0000 7f672effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:45:51.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.929+0000 7f672effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:45:51.931 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":154,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:45:51.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.931+0000 7f673cf16640 1 -- 172.21.15.12:0/2373475112 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f671803e1c0 msgr2=0x7f6718040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:51.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.931+0000 7f673cf16640 1 --2- 172.21.15.12:0/2373475112 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f671803e1c0 0x7f6718040680 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f67280038b0 tx=0x7f6728006d30 comp rx=0 tx=0).stop 2024-09-11T01:45:51.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.931+0000 7f673cf16640 10 monclient: shutdown 2024-09-11T01:45:51.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.931+0000 7f673cf16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:51.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.931+0000 7f673cf16640 1 -- 172.21.15.12:0/2373475112 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f67380a3270 msgr2=0x7f67381cda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:51.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.931+0000 7f673cf16640 1 --2- 172.21.15.12:0/2373475112 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f67380a3270 0x7f67381cda20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6720009960 tx=0x7f67200046c0 comp rx=0 tx=0).stop 2024-09-11T01:45:51.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.931+0000 7f673cf16640 1 -- 172.21.15.12:0/2373475112 shutdown_connections 2024-09-11T01:45:51.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.931+0000 7f673cf16640 1 --2- 172.21.15.12:0/2373475112 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6738138040 0x7f6738124910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:51.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.931+0000 7f673cf16640 1 --2- 172.21.15.12:0/2373475112 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f67380a3270 0x7f67381cda20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:51.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.931+0000 7f673cf16640 1 --2- 172.21.15.12:0/2373475112 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f671803e1c0 0x7f6718040680 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:51.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.931+0000 7f673cf16640 1 --2- 172.21.15.12:0/2373475112 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f67380a2e90 0x7f67381cd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:51.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.931+0000 7f673cf16640 1 -- 172.21.15.12:0/2373475112 >> 172.21.15.12:0/2373475112 conn(0x7f67380a8490 msgr2=0x7f6738072f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:51.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.931+0000 7f673cf16640 1 -- 172.21.15.12:0/2373475112 shutdown_connections 2024-09-11T01:45:51.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:51.931+0000 7f673cf16640 1 -- 172.21.15.12:0/2373475112 wait complete. 2024-09-11T01:45:54.944 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:45:55.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.055+0000 7f4cee3da640 10 monclient: get_monmap_and_config 2024-09-11T01:45:55.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.055+0000 7f4cee3da640 10 monclient: build_initial_monmap 2024-09-11T01:45:55.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.055+0000 7f4cee3da640 10 monclient: monmap: 2024-09-11T01:45:55.057 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:55.057 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:55.058 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:45:55.056655+0000 2024-09-11T01:45:55.058 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:45:55.056655+0000 2024-09-11T01:45:55.058 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:55.058 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:55.058 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:45:55.058 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:45:55.058 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:45:55.058 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:45:55.058 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:45:55.058 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:55.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.056+0000 7f4cee3da640 1 Processor -- start 2024-09-11T01:45:55.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.056+0000 7f4cee3da640 1 -- start start 2024-09-11T01:45:55.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.056+0000 7f4cee3da640 10 monclient: init 2024-09-11T01:45:55.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4cee3da640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:55.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4cee3da640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-11T01:45:55.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4cee3da640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4ce8138230 addr v1:172.21.15.71:6789/0 2024-09-11T01:45:55.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4cee3da640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4ce809d030 0x7f4ce813d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:55.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4cee3da640 10 monclient(hunting): picked mon.noname-c con 0x7f4ce809d030 addr v2:172.21.15.71:3300/0 2024-09-11T01:45:55.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4cee3da640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4ce809d460 0x7f4ce813f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:55.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4cee3da640 10 monclient(hunting): picked mon.noname-a con 0x7f4ce809d460 addr v2:172.21.15.12:3300/0 2024-09-11T01:45:55.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4cee3da640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ce80a6e30 con 0x7f4ce8138230 2024-09-11T01:45:55.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4cee3da640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:55.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4cee3da640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f4ce809c9c0 con 0x7f4ce809d460 2024-09-11T01:45:55.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4cee3da640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:55.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4cee3da640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f4ce809ba20 con 0x7f4ce809d030 2024-09-11T01:45:55.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4cee3da640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:55.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4cee3da640 10 monclient(hunting): authenticate will time out at 1761.499514s 2024-09-11T01:45:55.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4cec977640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7f4ce809d460 msgr2=0x7f4ce813f5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:45:55.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4cec977640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4ce809d460 0x7f4ce813f5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:55.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4ce77fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4ce809d030 0x7f4ce813d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:55.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4ce7fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f4ce8138230 0x7f4ce8138610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.12:38890/0 (socket says 172.21.15.12:38890) 2024-09-11T01:45:55.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4ce77fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4ce809d030 0x7f4ce813d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:56166/0 (socket says 172.21.15.12:56166) 2024-09-11T01:45:55.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4ce7fff640 1 -- 172.21.15.12:0/1091818797 learned_addr learned my addr 172.21.15.12:0/1091818797 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:45:55.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4ce77fe640 10 monclient(hunting): get_auth_request con 0x7f4ce809d030 auth_method 0 2024-09-11T01:45:55.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4ce77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:55.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4ce77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:55.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4ce77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:55.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4ce77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:55.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4ce77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:55.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.057+0000 7f4ce77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:55.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.058+0000 7f4ce77fe640 10 monclient(hunting): handle_auth_done global_id 134587 payload 274 2024-09-11T01:45:55.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.058+0000 7f4ce77fe640 1 -- 172.21.15.12:0/1091818797 >> v2:172.21.15.12:3300/0 conn(0x7f4ce809d460 msgr2=0x7f4ce813f5e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:45:55.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.058+0000 7f4ce77fe640 1 --2- 172.21.15.12:0/1091818797 >> v2:172.21.15.12:3300/0 conn(0x7f4ce809d460 0x7f4ce813f5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:55.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.058+0000 7f4ce77fe640 1 -- 172.21.15.12:0/1091818797 >> v1:172.21.15.71:6789/0 conn(0x7f4ce8138230 legacy=0x7f4ce8138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:45:55.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.058+0000 7f4ce77fe640 10 monclient: _finish_hunting 0 2024-09-11T01:45:55.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.058+0000 7f4ce77fe640 1 monclient: found mon.noname-c 2024-09-11T01:45:55.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.058+0000 7f4ce77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:55.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.058+0000 7f4ce77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:45:55.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.058+0000 7f4ce77fe640 1 -- 172.21.15.12:0/1091818797 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ce80a66a0 con 0x7f4ce809d030 2024-09-11T01:45:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.061+0000 7f4ce77fe640 1 -- 172.21.15.12:0/1091818797 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce809d030 msgr2=0x7f4ce813d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.061+0000 7f4ce77fe640 1 --2- 172.21.15.12:0/1091818797 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce809d030 0x7f4ce813d090 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4ce809dae0 tx=0x7f4cdc02a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25d60df2f9040b5a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.061+0000 7f4ce6ffd640 1 -- 172.21.15.12:0/1091818797 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc037070 con 0x7f4ce809d030 2024-09-11T01:45:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.061+0000 7f4ce6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.061+0000 7f4ce6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:45:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.061+0000 7f4ce6ffd640 10 monclient: dump: 2024-09-11T01:45:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:55.070 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.061+0000 7f4ce6ffd640 10 monclient: _finish_auth 0 2024-09-11T01:45:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.061+0000 7f4ce6ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:45:55.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.061+0000 7f4ce6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:55.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.061+0000 7f4cee3da640 5 monclient: authenticate success, global_id 134587 2024-09-11T01:45:55.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.061+0000 7f4ce6ffd640 1 -- 172.21.15.12:0/1091818797 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cdc032640 con 0x7f4ce809d030 2024-09-11T01:45:55.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.061+0000 7f4cee3da640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:45:55.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.061+0000 7f4ce6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:55.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.061+0000 7f4cee3da640 10 monclient: get_monmap_and_config success 2024-09-11T01:45:55.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.061+0000 7f4ce6ffd640 1 -- 172.21.15.12:0/1091818797 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc032a10 con 0x7f4ce809d030 2024-09-11T01:45:55.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.061+0000 7f4ce6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:55.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.061+0000 7f4cee3da640 4 set_mon_vals no callback set 2024-09-11T01:45:55.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.061+0000 7f4ce6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:45:55.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.061+0000 7f4ce6ffd640 10 monclient: dump: 2024-09-11T01:45:55.071 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:55.072 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:55.072 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:55.072 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:55.072 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:55.072 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:55.072 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:55.072 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:55.072 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:55.072 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:55.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.061+0000 7f4cee3da640 10 monclient: shutdown 2024-09-11T01:45:55.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.061+0000 7f4cee3da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:55.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.061+0000 7f4cee3da640 1 -- 172.21.15.12:0/1091818797 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce809d030 msgr2=0x7f4ce813d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:55.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.061+0000 7f4cee3da640 1 --2- 172.21.15.12:0/1091818797 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce809d030 0x7f4ce813d090 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4ce809dae0 tx=0x7f4cdc02a000 comp rx=0 tx=0).stop 2024-09-11T01:45:55.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.062+0000 7f4cee3da640 1 -- 172.21.15.12:0/1091818797 shutdown_connections 2024-09-11T01:45:55.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.062+0000 7f4cee3da640 1 --2- 172.21.15.12:0/1091818797 >> v2:172.21.15.12:3300/0 conn(0x7f4ce809d460 0x7f4ce813f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:55.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.062+0000 7f4cee3da640 1 --2- 172.21.15.12:0/1091818797 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce809d030 0x7f4ce813d090 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:55.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.062+0000 7f4cee3da640 1 -- 172.21.15.12:0/1091818797 >> 172.21.15.12:0/1091818797 conn(0x7f4ce80a7120 msgr2=0x7f4ce80a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:55.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.062+0000 7f4cee3da640 1 -- 172.21.15.12:0/1091818797 shutdown_connections 2024-09-11T01:45:55.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.062+0000 7f4cee3da640 1 -- 172.21.15.12:0/1091818797 wait complete. 2024-09-11T01:45:55.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.062+0000 7f4cee3da640 10 monclient: build_initial_monmap 2024-09-11T01:45:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.062+0000 7f4cee3da640 10 monclient: monmap: 2024-09-11T01:45:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:45:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:45:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:45:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:45:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:45:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:45:55.074 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.062+0000 7f4cee3da640 1 Processor -- start 2024-09-11T01:45:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4cee3da640 1 -- start start 2024-09-11T01:45:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4cee3da640 10 monclient: init 2024-09-11T01:45:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4cee3da640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4cee3da640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:45:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4cee3da640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ce809d030 0x7f4ce81cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4cee3da640 10 monclient(hunting): picked mon.noname-c con 0x7f4ce809d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:45:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4cee3da640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce809d460 0x7f4ce8137040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4cee3da640 10 monclient(hunting): picked mon.noname-a con 0x7f4ce809d460 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:45:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4cee3da640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce8138230 0x7f4ce813caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4cee3da640 10 monclient(hunting): picked mon.noname-b con 0x7f4ce8138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:45:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4cee3da640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4cee3da640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce813fca0 con 0x7f4ce809d460 2024-09-11T01:45:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4cee3da640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4cee3da640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce813fb20 con 0x7f4ce8138230 2024-09-11T01:45:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4cee3da640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4cee3da640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4ce813fe20 con 0x7f4ce809d030 2024-09-11T01:45:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4cee3da640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4cee3da640 10 monclient(hunting): authenticate will time out at 1761.505672s 2024-09-11T01:45:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4ce77fe640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce809d460 msgr2=0x7f4ce8137040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:45:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4ce77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce809d460 0x7f4ce8137040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4ce7fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ce809d030 0x7f4ce81cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4ce7fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ce809d030 0x7f4ce81cd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52094/0 (socket says 172.21.15.12:52094) 2024-09-11T01:45:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4ce7fff640 1 -- 172.21.15.12:0/1184764045 learned_addr learned my addr 172.21.15.12:0/1184764045 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:45:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4ce7fff640 10 monclient(hunting): get_auth_request con 0x7f4ce809d030 auth_method 0 2024-09-11T01:45:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4ce7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4ce7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4ce7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4ce7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:45:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4ce7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:45:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.063+0000 7f4ce7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:45:55.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.064+0000 7f4cec977640 1 --2- 172.21.15.12:0/1184764045 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce8138230 0x7f4ce813caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:55.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.064+0000 7f4cec977640 10 monclient(hunting): get_auth_request con 0x7f4ce8138230 auth_method 0 2024-09-11T01:45:55.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.064+0000 7f4cec977640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:45:55.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.064+0000 7f4cec977640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:45:55.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.064+0000 7f4cec977640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:45:55.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.064+0000 7f4ce7fff640 10 monclient(hunting): handle_auth_done global_id 184154 payload 824 2024-09-11T01:45:55.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.064+0000 7f4ce7fff640 1 -- 172.21.15.12:0/1184764045 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce8138230 msgr2=0x7f4ce813caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:55.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.064+0000 7f4ce7fff640 1 --2- 172.21.15.12:0/1184764045 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce8138230 0x7f4ce813caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:55.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.064+0000 7f4ce7fff640 1 -- 172.21.15.12:0/1184764045 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce809d460 msgr2=0x7f4ce8137040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:45:55.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.064+0000 7f4ce7fff640 1 --2- 172.21.15.12:0/1184764045 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce809d460 0x7f4ce8137040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:55.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.064+0000 7f4ce7fff640 10 monclient: _finish_hunting 0 2024-09-11T01:45:55.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.064+0000 7f4ce7fff640 1 monclient: found mon.noname-c 2024-09-11T01:45:55.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.064+0000 7f4ce7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:45:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.064+0000 7f4ce7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:45:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.064+0000 7f4ce7fff640 1 -- 172.21.15.12:0/1184764045 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ce813e510 con 0x7f4ce809d030 2024-09-11T01:45:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.066+0000 7f4ce7fff640 1 -- 172.21.15.12:0/1184764045 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ce809d030 msgr2=0x7f4ce81cd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.066+0000 7f4ce7fff640 1 --2- 172.21.15.12:0/1184764045 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ce809d030 0x7f4ce81cd030 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4cd800c460 tx=0x7f4cd800c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.066+0000 7f4ce4ff9640 1 -- 172.21.15.12:0/1184764045 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8007070 con 0x7f4ce809d030 2024-09-11T01:45:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.066+0000 7f4ce4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.066+0000 7f4ce4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:45:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.066+0000 7f4ce4ff9640 10 monclient: dump: 2024-09-11T01:45:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:55.079 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.066+0000 7f4ce4ff9640 10 monclient: _finish_auth 0 2024-09-11T01:45:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.066+0000 7f4ce4ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:45:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.066+0000 7f4ce4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:45:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.066+0000 7f4ce4ff9640 1 -- 172.21.15.12:0/1184764045 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cd8007230 con 0x7f4ce809d030 2024-09-11T01:45:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.066+0000 7f4cee3da640 5 monclient: authenticate success, global_id 184154 2024-09-11T01:45:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.066+0000 7f4ce4ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:45:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.066+0000 7f4ce4ff9640 1 -- 172.21.15.12:0/1184764045 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8007620 con 0x7f4ce809d030 2024-09-11T01:45:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.066+0000 7f4cee3da640 10 monclient: _renew_subs 2024-09-11T01:45:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.066+0000 7f4cee3da640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:45:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.066+0000 7f4cee3da640 1 -- 172.21.15.12:0/1184764045 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ce81219d0 con 0x7f4ce809d030 2024-09-11T01:45:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.066+0000 7f4ce4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:45:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.066+0000 7f4ce4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:45:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.066+0000 7f4ce4ff9640 10 monclient: dump: 2024-09-11T01:45:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:45:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:45:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:45:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:45:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:45:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:45:55.081 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.066+0000 7f4ce5ffb640 4 set_mon_vals no callback set 2024-09-11T01:45:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.066+0000 7f4cee3da640 10 monclient: _renew_subs 2024-09-11T01:45:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.066+0000 7f4cee3da640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:45:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.066+0000 7f4cee3da640 1 -- 172.21.15.12:0/1184764045 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ce813f070 con 0x7f4ce809d030 2024-09-11T01:45:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.067+0000 7f4ce4ff9640 1 -- 172.21.15.12:0/1184764045 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f4cd801b400 con 0x7f4ce809d030 2024-09-11T01:45:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.067+0000 7f4cee3da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.067+0000 7f4cee3da640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.067+0000 7f4cee3da640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:45:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.067+0000 7f4cee3da640 1 -- 172.21.15.12:0/1184764045 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ce8138610 con 0x7f4ce809d030 2024-09-11T01:45:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.067+0000 7f4ce4ff9640 1 --2- 172.21.15.12:0/1184764045 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4cd403e1f0 0x7f4cd40406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.067+0000 7f4ce4ff9640 1 -- 172.21.15.12:0/1184764045 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(566..566 src has 317..566) ==== 4787+0+0 (secure 0 0 0) 0x7f4cd8052ac0 con 0x7f4ce809d030 2024-09-11T01:45:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.068+0000 7f4ce77fe640 1 --2- 172.21.15.12:0/1184764045 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4cd403e1f0 0x7f4cd40406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:45:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.068+0000 7f4ce77fe640 10 monclient: get_auth_request con 0x7f4cd403e1f0 auth_method 0 2024-09-11T01:45:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.070+0000 7f4ce77fe640 1 --2- 172.21.15.12:0/1184764045 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4cd403e1f0 0x7f4cd40406b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80a17e0 tx=0x7f4cdc02e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:45:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.072+0000 7f4ce4ff9640 1 -- 172.21.15.12:0/1184764045 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4cd80514b0 con 0x7f4ce809d030 2024-09-11T01:45:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.072+0000 7f4ce4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:45:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.072+0000 7f4ce4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:45:55.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.329+0000 7f4cee3da640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:45:55.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.329+0000 7f4cee3da640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:45:55.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.329+0000 7f4cee3da640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:45:55.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.329+0000 7f4cee3da640 1 -- 172.21.15.12:0/1184764045 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4ce81214f0 con 0x7f4ce809d030 2024-09-11T01:45:55.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.329+0000 7f4ce4ff9640 1 -- 172.21.15.12:0/1184764045 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7f4cd8010d10 con 0x7f4ce809d030 2024-09-11T01:45:55.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.329+0000 7f4ce4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:45:55.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.329+0000 7f4ce4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:45:55.331 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":154,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:45:55.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.331+0000 7f4cee3da640 1 -- 172.21.15.12:0/1184764045 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4cd403e1f0 msgr2=0x7f4cd40406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:55.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.331+0000 7f4cee3da640 1 --2- 172.21.15.12:0/1184764045 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4cd403e1f0 0x7f4cd40406b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80a17e0 tx=0x7f4cdc02e000 comp rx=0 tx=0).stop 2024-09-11T01:45:55.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.331+0000 7f4cee3da640 10 monclient: shutdown 2024-09-11T01:45:55.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.331+0000 7f4cee3da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:45:55.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.331+0000 7f4cee3da640 1 -- 172.21.15.12:0/1184764045 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ce809d030 msgr2=0x7f4ce81cd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:45:55.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.331+0000 7f4cee3da640 1 --2- 172.21.15.12:0/1184764045 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ce809d030 0x7f4ce81cd030 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4cd800c460 tx=0x7f4cd800c920 comp rx=0 tx=0).stop 2024-09-11T01:45:55.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.332+0000 7f4cee3da640 1 -- 172.21.15.12:0/1184764045 shutdown_connections 2024-09-11T01:45:55.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.332+0000 7f4cee3da640 1 --2- 172.21.15.12:0/1184764045 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce8138230 0x7f4ce813caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:55.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.332+0000 7f4cee3da640 1 --2- 172.21.15.12:0/1184764045 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4cd403e1f0 0x7f4cd40406b0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:55.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.332+0000 7f4cee3da640 1 --2- 172.21.15.12:0/1184764045 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ce809d460 0x7f4ce8137040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:55.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.332+0000 7f4cee3da640 1 --2- 172.21.15.12:0/1184764045 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ce809d030 0x7f4ce81cd030 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:45:55.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.332+0000 7f4cee3da640 1 -- 172.21.15.12:0/1184764045 >> 172.21.15.12:0/1184764045 conn(0x7f4ce80a7120 msgr2=0x7f4ce80a0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:45:55.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.332+0000 7f4cee3da640 1 -- 172.21.15.12:0/1184764045 shutdown_connections 2024-09-11T01:45:55.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:55.332+0000 7f4cee3da640 1 -- 172.21.15.12:0/1184764045 wait complete. 2024-09-11T01:45:58.345 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:45:58.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.457+0000 7f27aca40640 10 monclient: get_monmap_and_config 2024-09-11T01:45:58.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.457+0000 7f27aca40640 10 monclient: build_initial_monmap 2024-09-11T01:45:58.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.457+0000 7f27aca40640 10 monclient: monmap: 2024-09-11T01:45:58.459 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:45:58.459 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:45:58.459 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:45:58.457894+0000 2024-09-11T01:45:58.459 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:45:58.457894+0000 2024-09-11T01:45:58.459 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:45:58.459 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:45:58.459 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:45:58.459 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:45:58.459 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:45:58.459 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:45:58.459 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:45:58.459 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:45:58.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.457+0000 7f27aca40640 1 Processor -- start 2024-09-11T01:45:58.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.457+0000 7f27aca40640 1 -- start start 2024-09-11T01:45:58.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.457+0000 7f27aca40640 10 monclient: init 2024-09-11T01:45:58.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.458+0000 7f27aca40640 10 monclient: _reopen_session rank -1 2024-09-11T01:45:58.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.458+0000 7f27aca40640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-11T01:45:58.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.458+0000 7f27aca40640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f27a8138230 0x7f27a8138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:45:58.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.458+0000 7f27aca40640 10 monclient(hunting): picked mon.noname-a con 0x7f27a8138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:45:58.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.458+0000 7f27aca40640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f27a809d030 addr v1:172.21.15.71:6789/0 2024-09-11T01:45:58.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.458+0000 7f27aca40640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f27a809d460 addr v1:172.21.15.12:6789/0 2024-09-11T01:45:58.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.458+0000 7f27aca40640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27a80a5e40 con 0x7f27a809d460 2024-09-11T01:45:58.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.458+0000 7f27aca40640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27a80a6e30 con 0x7f27a809d030 2024-09-11T01:45:58.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.458+0000 7f27aca40640 10 monclient(hunting): start opening mon connection 2024-09-11T01:45:58.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.458+0000 7f27aca40640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f27a809c9c0 con 0x7f27a8138230 2024-09-11T01:45:58.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.458+0000 7f27aca40640 10 monclient(hunting): _renew_subs 2024-09-11T01:45:58.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.458+0000 7f27aca40640 10 monclient(hunting): authenticate will time out at 1764.900563s 2024-09-11T01:45:58.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.458+0000 7f27a659c640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7f27a8138230 msgr2=0x7f27a8138610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:45:58.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.458+0000 7f27a659c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f27a8138230 0x7f27a8138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:45:58.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.458+0000 7f27a6d9d640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7f27a809d460 legacy=0x7f27a80a5730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:45:58.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.458+0000 7f27a5d9b640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f27a809d030 0x7f27a80a5020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.12:59324/0 (socket says 172.21.15.12:59324) 2024-09-11T01:45:58.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.459+0000 7f27a5d9b640 1 -- 172.21.15.12:0/3631064770 learned_addr learned my addr 172.21.15.12:0/3631064770 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:45:58.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.461+0000 7f27a5d9b640 1 -- 172.21.15.12:0/3631064770 >> v1:172.21.15.71:6789/0 conn(0x7f27a809d030 legacy=0x7f27a80a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:45:58.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.659+0000 7f27a659c640 1 -- 172.21.15.12:0/3631064770 >> v2:172.21.15.12:3300/0 conn(0x7f27a8138230 msgr2=0x7f27a8138610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:45:58.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.659+0000 7f27a6d9d640 1 -- 172.21.15.12:0/3631064770 >> v1:172.21.15.12:6789/0 conn(0x7f27a809d460 legacy=0x7f27a80a5730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:45:58.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:58.659+0000 7f27a659c640 1 --2- 172.21.15.12:0/3631064770 >> v2:172.21.15.12:3300/0 conn(0x7f27a8138230 0x7f27a8138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:45:59.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:59.060+0000 7f27a6d9d640 1 -- 172.21.15.12:0/3631064770 >> v1:172.21.15.12:6789/0 conn(0x7f27a809d460 legacy=0x7f27a80a5730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:45:59.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:59.060+0000 7f27a659c640 1 -- 172.21.15.12:0/3631064770 >> v2:172.21.15.12:3300/0 conn(0x7f27a8138230 msgr2=0x7f27a8138610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:45:59.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:59.060+0000 7f27a659c640 1 --2- 172.21.15.12:0/3631064770 >> v2:172.21.15.12:3300/0 conn(0x7f27a8138230 0x7f27a8138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:45:59.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:59.861+0000 7f27a6d9d640 1 -- 172.21.15.12:0/3631064770 >> v1:172.21.15.12:6789/0 conn(0x7f27a809d460 legacy=0x7f27a80a5730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:45:59.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:59.861+0000 7f27a659c640 1 -- 172.21.15.12:0/3631064770 >> v2:172.21.15.12:3300/0 conn(0x7f27a8138230 msgr2=0x7f27a8138610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:45:59.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:45:59.861+0000 7f27a659c640 1 --2- 172.21.15.12:0/3631064770 >> v2:172.21.15.12:3300/0 conn(0x7f27a8138230 0x7f27a8138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:46:01.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.458+0000 7f2797fff640 10 monclient(hunting): tick 2024-09-11T01:46:01.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.458+0000 7f2797fff640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:46:01.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.458+0000 7f2797fff640 1 monclient(hunting): continuing hunt 2024-09-11T01:46:01.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.458+0000 7f2797fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:46:01.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.458+0000 7f2797fff640 1 -- 172.21.15.12:0/3631064770 >> v2:172.21.15.12:3300/0 conn(0x7f27a8138230 msgr2=0x7f27a8138610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:46:01.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.458+0000 7f2797fff640 1 --2- 172.21.15.12:0/3631064770 >> v2:172.21.15.12:3300/0 conn(0x7f27a8138230 0x7f27a8138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:01.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.458+0000 7f2797fff640 1 -- 172.21.15.12:0/3631064770 >> v1:172.21.15.71:6789/0 conn(0x7f27a809d030 legacy=0x7f27a80a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:01.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.458+0000 7f2797fff640 1 -- 172.21.15.12:0/3631064770 >> v1:172.21.15.12:6789/0 conn(0x7f27a809d460 legacy=0x7f27a80a5730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:46:01.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.458+0000 7f2797fff640 10 monclient: _add_conns ranks=[4,2] 2024-09-11T01:46:01.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.458+0000 7f2797fff640 1 --2- 172.21.15.12:0/3631064770 >> v2:172.21.15.71:3300/0 conn(0x7f2788003740 0x7f2788005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:01.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.458+0000 7f2797fff640 10 monclient(hunting): picked mon.noname-c con 0x7f2788003740 addr v2:172.21.15.71:3300/0 2024-09-11T01:46:01.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.458+0000 7f2797fff640 1 --2- 172.21.15.12:0/3631064770 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2788006250 0x7f2788008710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:01.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.458+0000 7f2797fff640 10 monclient(hunting): picked mon.noname-b con 0x7f2788006250 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:01.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.458+0000 7f2797fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:01.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.458+0000 7f2797fff640 1 -- 172.21.15.12:0/3631064770 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f27a809c9c0 con 0x7f2788003740 2024-09-11T01:46:01.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.458+0000 7f2797fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:01.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.458+0000 7f2797fff640 1 -- 172.21.15.12:0/3631064770 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2788008df0 con 0x7f2788006250 2024-09-11T01:46:01.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.458+0000 7f2797fff640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:01.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.458+0000 7f27a5d9b640 1 --2- 172.21.15.12:0/3631064770 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2788006250 0x7f2788008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:01.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.458+0000 7f27a5d9b640 10 monclient(hunting): get_auth_request con 0x7f2788006250 auth_method 0 2024-09-11T01:46:01.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.458+0000 7f27a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:01.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.458+0000 7f27a5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:01.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.458+0000 7f27a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:01.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.459+0000 7f27a659c640 1 --2- 172.21.15.12:0/3631064770 >> v2:172.21.15.71:3300/0 conn(0x7f2788003740 0x7f2788005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:01.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.459+0000 7f27a5d9b640 1 -- 172.21.15.12:0/3631064770 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2788006250 msgr2=0x7f2788008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:46:01.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.459+0000 7f27a5d9b640 1 -- 172.21.15.12:0/3631064770 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2788006250 msgr2=0x7f2788008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:46:01.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.459+0000 7f27a5d9b640 1 --2- 172.21.15.12:0/3631064770 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2788006250 0x7f2788008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:46:01.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.459+0000 7f27a659c640 10 monclient(hunting): get_auth_request con 0x7f2788003740 auth_method 0 2024-09-11T01:46:01.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.459+0000 7f27a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:01.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.459+0000 7f27a659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:01.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.459+0000 7f27a659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:01.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.459+0000 7f27a5d9b640 1 --2- 172.21.15.12:0/3631064770 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2788006250 0x7f2788008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:46:01.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.459+0000 7f27a659c640 1 -- 172.21.15.12:0/3631064770 >> v2:172.21.15.71:3300/0 conn(0x7f2788003740 msgr2=0x7f2788005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:46:01.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.459+0000 7f27a659c640 1 -- 172.21.15.12:0/3631064770 >> v2:172.21.15.71:3300/0 conn(0x7f2788003740 msgr2=0x7f2788005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:46:01.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.459+0000 7f27a659c640 1 --2- 172.21.15.12:0/3631064770 >> v2:172.21.15.71:3300/0 conn(0x7f2788003740 0x7f2788005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:46:01.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.459+0000 7f27a659c640 1 --2- 172.21.15.12:0/3631064770 >> v2:172.21.15.71:3300/0 conn(0x7f2788003740 0x7f2788005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:46:01.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.660+0000 7f27a5d9b640 1 --2- 172.21.15.12:0/3631064770 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2788006250 0x7f2788008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:01.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.660+0000 7f27a5d9b640 10 monclient(hunting): get_auth_request con 0x7f2788006250 auth_method 0 2024-09-11T01:46:01.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.660+0000 7f27a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:01.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.660+0000 7f27a5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:01.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.660+0000 7f27a5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:46:01.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.661+0000 7f27a659c640 1 --2- 172.21.15.12:0/3631064770 >> v2:172.21.15.71:3300/0 conn(0x7f2788003740 0x7f2788005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:01.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.661+0000 7f27a5d9b640 1 -- 172.21.15.12:0/3631064770 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2788006250 msgr2=0x7f2788008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:46:01.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.661+0000 7f27a5d9b640 1 -- 172.21.15.12:0/3631064770 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2788006250 msgr2=0x7f2788008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:46:01.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.661+0000 7f27a659c640 10 monclient(hunting): get_auth_request con 0x7f2788003740 auth_method 0 2024-09-11T01:46:01.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.661+0000 7f27a5d9b640 1 --2- 172.21.15.12:0/3631064770 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2788006250 0x7f2788008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:46:01.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.661+0000 7f27a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:01.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.661+0000 7f27a659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:01.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.661+0000 7f27a659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:46:01.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.661+0000 7f27a5d9b640 1 --2- 172.21.15.12:0/3631064770 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2788006250 0x7f2788008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:46:01.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.661+0000 7f27a659c640 1 -- 172.21.15.12:0/3631064770 >> v2:172.21.15.71:3300/0 conn(0x7f2788003740 msgr2=0x7f2788005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:46:01.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.661+0000 7f27a659c640 1 -- 172.21.15.12:0/3631064770 >> v2:172.21.15.71:3300/0 conn(0x7f2788003740 msgr2=0x7f2788005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:46:01.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.661+0000 7f27a659c640 1 --2- 172.21.15.12:0/3631064770 >> v2:172.21.15.71:3300/0 conn(0x7f2788003740 0x7f2788005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:46:01.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:01.661+0000 7f27a659c640 1 --2- 172.21.15.12:0/3631064770 >> v2:172.21.15.71:3300/0 conn(0x7f2788003740 0x7f2788005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:46:02.026 INFO:tasks.rados.rados.0.smithi071.stdout:2524: done (1 left) 2024-09-11T01:46:02.026 INFO:tasks.rados.rados.0.smithi071.stdout:2526: done (0 left) 2024-09-11T01:46:02.026 INFO:tasks.rados.rados.0.smithi071.stdout:2527: read oid 48 snap 259 2024-09-11T01:46:02.026 INFO:tasks.rados.rados.0.smithi071.stdout:2527: expect deleted 2024-09-11T01:46:02.026 INFO:tasks.rados.rados.0.smithi071.stdout:2528: read oid 34 snap 273 2024-09-11T01:46:02.026 INFO:tasks.rados.rados.0.smithi071.stdout:2528: expect (ObjNum 857 snap 263 seq_num 857) 2024-09-11T01:46:02.026 INFO:tasks.rados.rados.0.smithi071.stdout:2529: read oid 5 snap -1 2024-09-11T01:46:02.026 INFO:tasks.rados.rados.0.smithi071.stdout:2529: expect (ObjNum 886 snap 272 seq_num 886) 2024-09-11T01:46:02.027 INFO:tasks.rados.rados.0.smithi071.stdout:2530: snap_create 2024-09-11T01:46:02.034 INFO:tasks.rados.rados.0.smithi071.stdout:2529: expect (ObjNum 886 snap 272 seq_num 886) 2024-09-11T01:46:02.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.062+0000 7f27a5d9b640 1 --2- 172.21.15.12:0/3631064770 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2788006250 0x7f2788008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:02.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.063+0000 7f27a5d9b640 10 monclient(hunting): get_auth_request con 0x7f2788006250 auth_method 0 2024-09-11T01:46:02.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.063+0000 7f27a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:02.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.063+0000 7f27a5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:02.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.063+0000 7f27a5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:46:02.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.063+0000 7f27a659c640 1 --2- 172.21.15.12:0/3631064770 >> v2:172.21.15.71:3300/0 conn(0x7f2788003740 0x7f2788005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:02.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.063+0000 7f27a659c640 10 monclient(hunting): get_auth_request con 0x7f2788003740 auth_method 0 2024-09-11T01:46:02.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.063+0000 7f27a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:02.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.063+0000 7f27a659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:02.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.063+0000 7f27a659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:46:02.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.063+0000 7f27a5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:02.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.063+0000 7f27a5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:02.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.063+0000 7f27a5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:02.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.063+0000 7f27a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:02.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.063+0000 7f27a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:02.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.063+0000 7f27a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:02.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.063+0000 7f27a5d9b640 10 monclient(hunting): handle_auth_done global_id 184157 payload 274 2024-09-11T01:46:02.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.063+0000 7f27a5d9b640 1 -- 172.21.15.12:0/3631064770 >> v2:172.21.15.71:3300/0 conn(0x7f2788003740 msgr2=0x7f2788005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:46:02.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.063+0000 7f27a5d9b640 1 --2- 172.21.15.12:0/3631064770 >> v2:172.21.15.71:3300/0 conn(0x7f2788003740 0x7f2788005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.063+0000 7f27a5d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:46:02.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.063+0000 7f27a5d9b640 1 monclient: found mon.noname-b 2024-09-11T01:46:02.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.063+0000 7f27a5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:02.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.063+0000 7f27a5d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:46:02.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.063+0000 7f27a5d9b640 1 -- 172.21.15.12:0/3631064770 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27a809ba20 con 0x7f2788006250 2024-09-11T01:46:02.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.066+0000 7f27a5d9b640 1 -- 172.21.15.12:0/3631064770 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2788006250 msgr2=0x7f2788008710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:02.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.066+0000 7f27a5d9b640 1 --2- 172.21.15.12:0/3631064770 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2788006250 0x7f2788008710 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f279c007fd0 tx=0x7f279c02a6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3191c1d36a51f7c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:02.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.066+0000 7f27a559a640 1 -- 172.21.15.12:0/3631064770 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279c038070 con 0x7f2788006250 2024-09-11T01:46:02.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.066+0000 7f27a559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:02.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.066+0000 7f27a559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:46:02.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.066+0000 7f27a559a640 10 monclient: dump: 2024-09-11T01:46:02.069 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:02.069 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:02.069 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.070 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.070 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:02.070 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:02.070 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:02.070 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:02.070 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:02.070 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:02.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.066+0000 7f27a559a640 10 monclient: _finish_auth 0 2024-09-11T01:46:02.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.066+0000 7f27a559a640 10 monclient: _check_auth_tickets 2024-09-11T01:46:02.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.066+0000 7f27a559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:02.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.066+0000 7f27a559a640 1 -- 172.21.15.12:0/3631064770 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f279c033410 con 0x7f2788006250 2024-09-11T01:46:02.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.066+0000 7f27aca40640 5 monclient: authenticate success, global_id 184157 2024-09-11T01:46:02.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.066+0000 7f27aca40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:46:02.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.066+0000 7f27a559a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:02.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.066+0000 7f27a559a640 1 -- 172.21.15.12:0/3631064770 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279c0337e0 con 0x7f2788006250 2024-09-11T01:46:02.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.066+0000 7f27a559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:02.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.066+0000 7f27a559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:02.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.066+0000 7f27a559a640 10 monclient: dump: 2024-09-11T01:46:02.073 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:02.073 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:02.073 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.073 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.073 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:02.073 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:02.073 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:02.073 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:02.074 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:02.074 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:02.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.066+0000 7f27aca40640 10 monclient: get_monmap_and_config success 2024-09-11T01:46:02.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.066+0000 7f27aca40640 4 set_mon_vals no callback set 2024-09-11T01:46:02.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.067+0000 7f27aca40640 10 monclient: shutdown 2024-09-11T01:46:02.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.067+0000 7f27aca40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:02.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.067+0000 7f27aca40640 1 -- 172.21.15.12:0/3631064770 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2788006250 msgr2=0x7f2788008710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:02.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.067+0000 7f27aca40640 1 --2- 172.21.15.12:0/3631064770 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2788006250 0x7f2788008710 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f279c007fd0 tx=0x7f279c02a6a0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.067+0000 7f27a6d9d640 1 -- 172.21.15.12:0/3631064770 reap_dead start 2024-09-11T01:46:02.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.067+0000 7f27aca40640 1 -- 172.21.15.12:0/3631064770 shutdown_connections 2024-09-11T01:46:02.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.067+0000 7f27aca40640 1 -- 172.21.15.12:0/3631064770 >> 172.21.15.12:0/3631064770 conn(0x7f27a80a7120 msgr2=0x7f27a80a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:02.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.067+0000 7f27aca40640 1 -- 172.21.15.12:0/3631064770 shutdown_connections 2024-09-11T01:46:02.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.067+0000 7f27aca40640 1 -- 172.21.15.12:0/3631064770 wait complete. 2024-09-11T01:46:02.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.068+0000 7f27aca40640 10 monclient: build_initial_monmap 2024-09-11T01:46:02.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.068+0000 7f27aca40640 10 monclient: monmap: 2024-09-11T01:46:02.077 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:02.077 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:46:02.077 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:46:02.077 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:46:02.078 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:02.078 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:02.078 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:46:02.078 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:46:02.078 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:46:02.078 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:02.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.069+0000 7f27aca40640 1 Processor -- start 2024-09-11T01:46:02.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.069+0000 7f27aca40640 1 -- start start 2024-09-11T01:46:02.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.069+0000 7f27aca40640 10 monclient: init 2024-09-11T01:46:02.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.069+0000 7f27aca40640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:02.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.069+0000 7f27aca40640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:46:02.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.069+0000 7f27aca40640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2788006250 0x7f27a8136c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:02.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.069+0000 7f27aca40640 10 monclient(hunting): picked mon.noname-a con 0x7f2788006250 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:46:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.069+0000 7f27aca40640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f27a813cc10 0x7f27a813d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.069+0000 7f27aca40640 10 monclient(hunting): picked mon.noname-c con 0x7f27a813cc10 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.069+0000 7f27aca40640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f27a813d5d0 0x7f27a813da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.069+0000 7f27aca40640 10 monclient(hunting): picked mon.noname-b con 0x7f27a813d5d0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:46:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.069+0000 7f27aca40640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.069+0000 7f27aca40640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f27a8138b50 con 0x7f2788006250 2024-09-11T01:46:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.069+0000 7f27aca40640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.069+0000 7f27aca40640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f27a806f7f0 con 0x7f27a813d5d0 2024-09-11T01:46:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.069+0000 7f27aca40640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.069+0000 7f27aca40640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f27a813f230 con 0x7f27a813cc10 2024-09-11T01:46:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.069+0000 7f27aca40640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:02.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.069+0000 7f27aca40640 10 monclient(hunting): authenticate will time out at 1768.511855s 2024-09-11T01:46:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.069+0000 7f27a659c640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2788006250 msgr2=0x7f27a8136c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:46:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.069+0000 7f27a659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2788006250 0x7f27a8136c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:46:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.069+0000 7f27a5d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f27a813cc10 0x7f27a813d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a6d9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f27a813d5d0 0x7f27a813da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a5d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f27a813cc10 0x7f27a813d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:38208/0 (socket says 172.21.15.12:38208) 2024-09-11T01:46:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a5d9b640 1 -- 172.21.15.12:0/3820914393 learned_addr learned my addr 172.21.15.12:0/3820914393 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a5d9b640 10 monclient(hunting): get_auth_request con 0x7f27a813cc10 auth_method 0 2024-09-11T01:46:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a6d9d640 10 monclient(hunting): get_auth_request con 0x7f27a813d5d0 auth_method 0 2024-09-11T01:46:02.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a5d9b640 10 monclient(hunting): handle_auth_done global_id 184160 payload 824 2024-09-11T01:46:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a5d9b640 1 -- 172.21.15.12:0/3820914393 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f27a813d5d0 msgr2=0x7f27a813da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a5d9b640 1 --2- 172.21.15.12:0/3820914393 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f27a813d5d0 0x7f27a813da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a5d9b640 1 -- 172.21.15.12:0/3820914393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2788006250 msgr2=0x7f27a8136c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:46:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a5d9b640 1 --2- 172.21.15.12:0/3820914393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2788006250 0x7f27a8136c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a5d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:46:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a5d9b640 1 monclient: found mon.noname-c 2024-09-11T01:46:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:46:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.070+0000 7f27a5d9b640 1 -- 172.21.15.12:0/3820914393 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27a8140750 con 0x7f27a813cc10 2024-09-11T01:46:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.072+0000 7f27a5d9b640 1 -- 172.21.15.12:0/3820914393 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f27a813cc10 msgr2=0x7f27a813d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:02.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.072+0000 7f27a5d9b640 1 --2- 172.21.15.12:0/3820914393 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f27a813cc10 0x7f27a813d090 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f279c036040 tx=0x7f279c001460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.078+0000 7f2796ffd640 1 -- 172.21.15.12:0/3820914393 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279c042030 con 0x7f27a813cc10 2024-09-11T01:46:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.078+0000 7f2796ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.078+0000 7f2796ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:46:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.078+0000 7f2796ffd640 10 monclient: dump: 2024-09-11T01:46:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:02.084 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:02.085 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.078+0000 7f2796ffd640 10 monclient: _finish_auth 0 2024-09-11T01:46:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.078+0000 7f2796ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:46:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.078+0000 7f2796ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.078+0000 7f2796ffd640 1 -- 172.21.15.12:0/3820914393 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f279c005250 con 0x7f27a813cc10 2024-09-11T01:46:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.078+0000 7f2796ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.078+0000 7f2796ffd640 1 -- 172.21.15.12:0/3820914393 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279c038040 con 0x7f27a813cc10 2024-09-11T01:46:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.078+0000 7f2796ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.078+0000 7f2796ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:02.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.078+0000 7f2796ffd640 10 monclient: dump: 2024-09-11T01:46:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:02.086 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.078+0000 7f27aca40640 5 monclient: authenticate success, global_id 184160 2024-09-11T01:46:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.078+0000 7f27aca40640 10 monclient: _renew_subs 2024-09-11T01:46:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.078+0000 7f27aca40640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:02.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.078+0000 7f27aca40640 1 -- 172.21.15.12:0/3820914393 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27a8140a30 con 0x7f27a813cc10 2024-09-11T01:46:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.078+0000 7f27a4d99640 4 set_mon_vals no callback set 2024-09-11T01:46:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.078+0000 7f27aca40640 10 monclient: _renew_subs 2024-09-11T01:46:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.078+0000 7f27aca40640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.078+0000 7f27aca40640 1 -- 172.21.15.12:0/3820914393 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f27a8140470 con 0x7f27a813cc10 2024-09-11T01:46:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.079+0000 7f2796ffd640 1 -- 172.21.15.12:0/3820914393 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f279c04f060 con 0x7f27a813cc10 2024-09-11T01:46:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.079+0000 7f27aca40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.079+0000 7f27aca40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.079+0000 7f27aca40640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.079+0000 7f27aca40640 1 -- 172.21.15.12:0/3820914393 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27a8138610 con 0x7f27a813cc10 2024-09-11T01:46:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.079+0000 7f2796ffd640 1 --2- 172.21.15.12:0/3820914393 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f278803f160 0x7f2788041620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.080+0000 7f27a659c640 1 --2- 172.21.15.12:0/3820914393 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f278803f160 0x7f2788041620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:02.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.080+0000 7f27a659c640 10 monclient: get_auth_request con 0x7f278803f160 auth_method 0 2024-09-11T01:46:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.083+0000 7f27a659c640 1 --2- 172.21.15.12:0/3820914393 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f278803f160 0x7f2788041620 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2798003d00 tx=0x7f2798005040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:02.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.083+0000 7f2796ffd640 1 -- 172.21.15.12:0/3820914393 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(568..568 src has 317..568) ==== 4787+0+0 (secure 0 0 0) 0x7f279c005600 con 0x7f27a813cc10 2024-09-11T01:46:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.088+0000 7f2796ffd640 1 -- 172.21.15.12:0/3820914393 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f279c03ec20 con 0x7f27a813cc10 2024-09-11T01:46:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.088+0000 7f2796ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:02.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.088+0000 7f2796ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:46:02.128 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:46:02.186 INFO:tasks.rados.rados.0.smithi071.stdout:2528: expect (ObjNum 857 snap 263 seq_num 857) 2024-09-11T01:46:02.186 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:46:02.346 INFO:tasks.rados.rados.0.smithi071.stdout:2527: done (3 left) 2024-09-11T01:46:02.346 INFO:tasks.rados.rados.0.smithi071.stdout:2528: done (2 left) 2024-09-11T01:46:02.346 INFO:tasks.rados.rados.0.smithi071.stdout:2529: done (1 left) 2024-09-11T01:46:02.346 INFO:tasks.rados.rados.0.smithi071.stdout:2530: done (0 left) 2024-09-11T01:46:02.346 INFO:tasks.rados.rados.0.smithi071.stdout:2531: rollback oid 1 current snap is 285 2024-09-11T01:46:02.346 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 1 to 270 2024-09-11T01:46:02.346 INFO:tasks.rados.rados.0.smithi071.stdout:2532: snap_create 2024-09-11T01:46:02.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.348+0000 7f27aca40640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:46:02.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.348+0000 7f27aca40640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:46:02.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.348+0000 7f27aca40640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:02.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.348+0000 7f27aca40640 1 -- 172.21.15.12:0/3820914393 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f27a8138610 con 0x7f27a813cc10 2024-09-11T01:46:02.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.348+0000 7f2796ffd640 1 -- 172.21.15.12:0/3820914393 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f279c03d380 con 0x7f27a813cc10 2024-09-11T01:46:02.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.348+0000 7f2796ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:46:02.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.348+0000 7f2796ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:46:02.350 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":158,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:46:02.350 INFO:tasks.rados.rados.0.smithi071.stdout:2531: finishing rollback tid 0 to smithi07134777-1 2024-09-11T01:46:02.351 INFO:tasks.rados.rados.0.smithi071.stdout:2531: finishing rollback tid 1 to smithi07134777-1 2024-09-11T01:46:02.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.350+0000 7f27aca40640 1 -- 172.21.15.12:0/3820914393 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f278803f160 msgr2=0x7f2788041620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:02.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.350+0000 7f27aca40640 1 --2- 172.21.15.12:0/3820914393 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f278803f160 0x7f2788041620 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2798003d00 tx=0x7f2798005040 comp rx=0 tx=0).stop 2024-09-11T01:46:02.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.350+0000 7f27aca40640 10 monclient: shutdown 2024-09-11T01:46:02.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.350+0000 7f27aca40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:02.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.350+0000 7f27aca40640 1 -- 172.21.15.12:0/3820914393 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f27a813cc10 msgr2=0x7f27a813d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:02.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.350+0000 7f27aca40640 1 --2- 172.21.15.12:0/3820914393 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f27a813cc10 0x7f27a813d090 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f279c036040 tx=0x7f279c001460 comp rx=0 tx=0).stop 2024-09-11T01:46:02.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.350+0000 7f27aca40640 1 -- 172.21.15.12:0/3820914393 shutdown_connections 2024-09-11T01:46:02.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.350+0000 7f27aca40640 1 --2- 172.21.15.12:0/3820914393 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f27a813d5d0 0x7f27a813da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.350+0000 7f27aca40640 1 --2- 172.21.15.12:0/3820914393 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f27a813cc10 0x7f27a813d090 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.350+0000 7f27aca40640 1 --2- 172.21.15.12:0/3820914393 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f278803f160 0x7f2788041620 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.350+0000 7f27aca40640 1 --2- 172.21.15.12:0/3820914393 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2788006250 0x7f27a8136c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.350+0000 7f27aca40640 1 -- 172.21.15.12:0/3820914393 >> 172.21.15.12:0/3820914393 conn(0x7f27a80a7120 msgr2=0x7f27a80a1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:02.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.350+0000 7f27aca40640 1 -- 172.21.15.12:0/3820914393 shutdown_connections 2024-09-11T01:46:02.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.350+0000 7f27aca40640 1 -- 172.21.15.12:0/3820914393 wait complete. 2024-09-11T01:46:02.357 INFO:tasks.rados.rados.0.smithi071.stdout:2531: finishing rollback tid 2 to smithi07134777-1 2024-09-11T01:46:02.357 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 1152 (ObjNum 688 snap 204 seq_num 688) dirty exists 2024-09-11T01:46:02.363 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:46:02.363 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T01:46:02.363 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:46:02.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.464+0000 7f2d5cca9640 10 monclient: get_monmap_and_config 2024-09-11T01:46:02.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.464+0000 7f2d5cca9640 10 monclient: build_initial_monmap 2024-09-11T01:46:02.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.464+0000 7f2d5cca9640 10 monclient: monmap: 2024-09-11T01:46:02.466 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:02.466 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:02.466 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:46:02.465119+0000 2024-09-11T01:46:02.466 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:46:02.465119+0000 2024-09-11T01:46:02.466 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:02.466 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:02.466 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:46:02.466 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:46:02.466 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:46:02.466 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:46:02.466 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:46:02.466 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:02.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d5cca9640 1 Processor -- start 2024-09-11T01:46:02.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d5cca9640 1 -- start start 2024-09-11T01:46:02.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d5cca9640 10 monclient: init 2024-09-11T01:46:02.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d5cca9640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:02.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d5cca9640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-11T01:46:02.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d5cca9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2d580a03a0 0x7f2d5809f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:02.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d5cca9640 10 monclient(hunting): picked mon.noname-a con 0x7f2d580a03a0 addr v2:172.21.15.12:3300/0 2024-09-11T01:46:02.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d5cca9640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2d580a2530 0x7f2d580a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:02.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d5cca9640 10 monclient(hunting): picked mon.noname-c con 0x7f2d580a2530 addr v2:172.21.15.71:3300/0 2024-09-11T01:46:02.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d5cca9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d580a11d0 0x7f2d580a1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:02.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d5cca9640 10 monclient(hunting): picked mon.noname-b con 0x7f2d580a11d0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:02.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d5cca9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:02.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d5cca9640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f2d580a29b0 con 0x7f2d580a03a0 2024-09-11T01:46:02.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d5cca9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:02.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d5cca9640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f2d580a0780 con 0x7f2d580a2530 2024-09-11T01:46:02.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d5cca9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:02.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d5cca9640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2d5809e510 con 0x7f2d580a11d0 2024-09-11T01:46:02.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d5cca9640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:02.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d5cca9640 10 monclient(hunting): authenticate will time out at 1768.907655s 2024-09-11T01:46:02.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d5659c640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7f2d580a03a0 msgr2=0x7f2d5809f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:46:02.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d5659c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2d580a03a0 0x7f2d5809f1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:46:02.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d56d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d580a11d0 0x7f2d580a1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:02.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d56d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d580a11d0 0x7f2d580a1610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:38222/0 (socket says 172.21.15.12:38222) 2024-09-11T01:46:02.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d56d9d640 1 -- 172.21.15.12:0/1636657932 learned_addr learned my addr 172.21.15.12:0/1636657932 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:02.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d56d9d640 10 monclient(hunting): get_auth_request con 0x7f2d580a11d0 auth_method 0 2024-09-11T01:46:02.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d56d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:02.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d56d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:02.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d56d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:02.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d4ffff640 1 --2- 172.21.15.12:0/1636657932 >> v2:172.21.15.71:3300/0 conn(0x7f2d580a2530 0x7f2d580a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:02.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d4ffff640 10 monclient(hunting): get_auth_request con 0x7f2d580a2530 auth_method 0 2024-09-11T01:46:02.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:02.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:02.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:02.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d56d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:02.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d56d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:02.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d56d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:02.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:02.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:02.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.465+0000 7f2d4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:02.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.466+0000 7f2d56d9d640 10 monclient(hunting): handle_auth_done global_id 184163 payload 274 2024-09-11T01:46:02.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.466+0000 7f2d56d9d640 1 -- 172.21.15.12:0/1636657932 >> v2:172.21.15.71:3300/0 conn(0x7f2d580a2530 msgr2=0x7f2d580a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:46:02.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.466+0000 7f2d56d9d640 1 --2- 172.21.15.12:0/1636657932 >> v2:172.21.15.71:3300/0 conn(0x7f2d580a2530 0x7f2d580a0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.466+0000 7f2d56d9d640 1 -- 172.21.15.12:0/1636657932 >> v2:172.21.15.12:3300/0 conn(0x7f2d580a03a0 msgr2=0x7f2d5809f1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:46:02.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.466+0000 7f2d56d9d640 1 --2- 172.21.15.12:0/1636657932 >> v2:172.21.15.12:3300/0 conn(0x7f2d580a03a0 0x7f2d5809f1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.466+0000 7f2d56d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:46:02.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.466+0000 7f2d56d9d640 1 monclient: found mon.noname-b 2024-09-11T01:46:02.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.466+0000 7f2d56d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:02.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.466+0000 7f2d56d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:46:02.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.466+0000 7f2d56d9d640 1 -- 172.21.15.12:0/1636657932 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d5809edb0 con 0x7f2d580a11d0 2024-09-11T01:46:02.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.466+0000 7f2d4ffff640 1 --2- 172.21.15.12:0/1636657932 >> v2:172.21.15.71:3300/0 conn(0x7f2d580a2530 0x7f2d580a0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:46:02.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.468+0000 7f2d56d9d640 1 -- 172.21.15.12:0/1636657932 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d580a11d0 msgr2=0x7f2d580a1610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:02.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.468+0000 7f2d56d9d640 1 --2- 172.21.15.12:0/1636657932 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d580a11d0 0x7f2d580a1610 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2d580a0120 tx=0x7f2d5002c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78d4ad7742400f3b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:02.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d55d9b640 1 -- 172.21.15.12:0/1636657932 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d50038070 con 0x7f2d580a11d0 2024-09-11T01:46:02.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d55d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:02.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d55d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:46:02.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d55d9b640 10 monclient: dump: 2024-09-11T01:46:02.480 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:02.480 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:02.480 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.481 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.481 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:02.481 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:02.481 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:02.481 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:02.481 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:02.481 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:02.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d55d9b640 10 monclient: _finish_auth 0 2024-09-11T01:46:02.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d55d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:46:02.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d55d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:02.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d55d9b640 1 -- 172.21.15.12:0/1636657932 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d50009800 con 0x7f2d580a11d0 2024-09-11T01:46:02.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d5cca9640 5 monclient: authenticate success, global_id 184163 2024-09-11T01:46:02.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d5cca9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:46:02.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d55d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:02.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d55d9b640 1 -- 172.21.15.12:0/1636657932 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d50009bd0 con 0x7f2d580a11d0 2024-09-11T01:46:02.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d5cca9640 10 monclient: get_monmap_and_config success 2024-09-11T01:46:02.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d5cca9640 4 set_mon_vals no callback set 2024-09-11T01:46:02.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d55d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:02.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d55d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:02.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d55d9b640 10 monclient: dump: 2024-09-11T01:46:02.482 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:02.482 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:02.483 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.483 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.483 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:02.483 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:02.483 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:02.483 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:02.483 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:02.483 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:02.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d5cca9640 10 monclient: shutdown 2024-09-11T01:46:02.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d5cca9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:02.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d5cca9640 1 -- 172.21.15.12:0/1636657932 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d580a11d0 msgr2=0x7f2d580a1610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:02.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d5cca9640 1 --2- 172.21.15.12:0/1636657932 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d580a11d0 0x7f2d580a1610 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2d580a0120 tx=0x7f2d5002c7e0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d5cca9640 1 -- 172.21.15.12:0/1636657932 shutdown_connections 2024-09-11T01:46:02.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d5cca9640 1 --2- 172.21.15.12:0/1636657932 >> v2:172.21.15.71:3300/0 conn(0x7f2d580a2530 0x7f2d580a0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d5cca9640 1 --2- 172.21.15.12:0/1636657932 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d580a11d0 0x7f2d580a1610 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d5cca9640 1 --2- 172.21.15.12:0/1636657932 >> v2:172.21.15.12:3300/0 conn(0x7f2d580a03a0 0x7f2d5809f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d5cca9640 1 -- 172.21.15.12:0/1636657932 >> 172.21.15.12:0/1636657932 conn(0x7f2d580a2f70 msgr2=0x7f2d580a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:02.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d5cca9640 1 -- 172.21.15.12:0/1636657932 shutdown_connections 2024-09-11T01:46:02.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.469+0000 7f2d5cca9640 1 -- 172.21.15.12:0/1636657932 wait complete. 2024-09-11T01:46:02.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.470+0000 7f2d5cca9640 10 monclient: build_initial_monmap 2024-09-11T01:46:02.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.470+0000 7f2d5cca9640 10 monclient: monmap: 2024-09-11T01:46:02.484 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:02.484 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:46:02.484 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:46:02.484 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:46:02.484 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:02.484 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:02.484 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:46:02.485 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:46:02.485 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:46:02.485 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:02.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.470+0000 7f2d5cca9640 1 Processor -- start 2024-09-11T01:46:02.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.470+0000 7f2d5cca9640 1 -- start start 2024-09-11T01:46:02.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.470+0000 7f2d5cca9640 10 monclient: init 2024-09-11T01:46:02.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5cca9640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:02.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5cca9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:46:02.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5cca9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d580a03a0 0x7f2d581c0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:02.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5cca9640 10 monclient(hunting): picked mon.noname-c con 0x7f2d580a03a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:02.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5cca9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d580a11d0 0x7f2d581c10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:02.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5cca9640 10 monclient(hunting): picked mon.noname-b con 0x7f2d580a11d0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:46:02.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5cca9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d580a2530 0x7f2d581aa720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:02.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5cca9640 10 monclient(hunting): picked mon.noname-a con 0x7f2d580a2530 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:46:02.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5cca9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:02.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5cca9640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5809ce00 con 0x7f2d580a2530 2024-09-11T01:46:02.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5cca9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:02.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5cca9640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5809cc80 con 0x7f2d580a11d0 2024-09-11T01:46:02.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5cca9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:02.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5cca9640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2d5809cf80 con 0x7f2d580a03a0 2024-09-11T01:46:02.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5cca9640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:02.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5cca9640 10 monclient(hunting): authenticate will time out at 1768.913366s 2024-09-11T01:46:02.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d56d9d640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d580a2530 msgr2=0x7f2d581aa720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:46:02.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d56d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d580a2530 0x7f2d581aa720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:46:02.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d580a03a0 0x7f2d581c0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:02.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d580a03a0 0x7f2d581c0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:38232/0 (socket says 172.21.15.12:38232) 2024-09-11T01:46:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5659c640 1 -- 172.21.15.12:0/1021909215 learned_addr learned my addr 172.21.15.12:0/1021909215 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5659c640 10 monclient(hunting): get_auth_request con 0x7f2d580a03a0 auth_method 0 2024-09-11T01:46:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d4ffff640 1 --2- 172.21.15.12:0/1021909215 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d580a11d0 0x7f2d581c10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d4ffff640 10 monclient(hunting): get_auth_request con 0x7f2d580a11d0 auth_method 0 2024-09-11T01:46:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:02.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5659c640 10 monclient(hunting): handle_auth_done global_id 184166 payload 824 2024-09-11T01:46:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5659c640 1 -- 172.21.15.12:0/1021909215 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d580a11d0 msgr2=0x7f2d581c10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5659c640 1 --2- 172.21.15.12:0/1021909215 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d580a11d0 0x7f2d581c10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5659c640 1 -- 172.21.15.12:0/1021909215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d580a2530 msgr2=0x7f2d581aa720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:46:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5659c640 1 --2- 172.21.15.12:0/1021909215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d580a2530 0x7f2d581aa720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5659c640 10 monclient: _finish_hunting 0 2024-09-11T01:46:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5659c640 1 monclient: found mon.noname-c 2024-09-11T01:46:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:46:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.471+0000 7f2d5659c640 1 -- 172.21.15.12:0/1021909215 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d581adca0 con 0x7f2d580a03a0 2024-09-11T01:46:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.473+0000 7f2d5659c640 1 -- 172.21.15.12:0/1021909215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d580a03a0 msgr2=0x7f2d581c0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:02.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.473+0000 7f2d5659c640 1 --2- 172.21.15.12:0/1021909215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d580a03a0 0x7f2d581c0b80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2d4800b580 tx=0x7f2d4800ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.474+0000 7f2d4effd640 1 -- 172.21.15.12:0/1021909215 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d480050f0 con 0x7f2d580a03a0 2024-09-11T01:46:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.474+0000 7f2d4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.474+0000 7f2d4effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:46:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.474+0000 7f2d4effd640 10 monclient: dump: 2024-09-11T01:46:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:02.489 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:02.490 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:02.490 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:02.490 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:02.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.474+0000 7f2d4effd640 10 monclient: _finish_auth 0 2024-09-11T01:46:02.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.474+0000 7f2d4effd640 10 monclient: _check_auth_tickets 2024-09-11T01:46:02.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.474+0000 7f2d4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:02.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.474+0000 7f2d4effd640 1 -- 172.21.15.12:0/1021909215 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d48015070 con 0x7f2d580a03a0 2024-09-11T01:46:02.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.474+0000 7f2d5cca9640 5 monclient: authenticate success, global_id 184166 2024-09-11T01:46:02.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.474+0000 7f2d4effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:02.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.474+0000 7f2d4effd640 1 -- 172.21.15.12:0/1021909215 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d480054d0 con 0x7f2d580a03a0 2024-09-11T01:46:02.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.474+0000 7f2d5cca9640 10 monclient: _renew_subs 2024-09-11T01:46:02.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.474+0000 7f2d5cca9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:02.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.474+0000 7f2d5cca9640 1 -- 172.21.15.12:0/1021909215 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d5809d480 con 0x7f2d580a03a0 2024-09-11T01:46:02.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.474+0000 7f2d4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:02.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.474+0000 7f2d4effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:02.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.474+0000 7f2d5559a640 4 set_mon_vals no callback set 2024-09-11T01:46:02.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.474+0000 7f2d4effd640 10 monclient: dump: 2024-09-11T01:46:02.492 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:02.492 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:02.492 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.492 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.492 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:02.492 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:02.492 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:02.492 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:02.492 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:02.492 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:02.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.474+0000 7f2d5cca9640 10 monclient: _renew_subs 2024-09-11T01:46:02.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.474+0000 7f2d5cca9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:02.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.474+0000 7f2d5cca9640 1 -- 172.21.15.12:0/1021909215 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d581ae540 con 0x7f2d580a03a0 2024-09-11T01:46:02.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.474+0000 7f2d5cca9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:02.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.474+0000 7f2d5cca9640 1 --2- 172.21.15.12:0/1021909215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d581aac60 0x7f2d581ab040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:02.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.474+0000 7f2d5cca9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:02.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.474+0000 7f2d5cca9640 1 -- 172.21.15.12:0/1021909215 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2d581ae3c0 con 0x7f2d581aac60 2024-09-11T01:46:02.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.475+0000 7f2d4effd640 1 -- 172.21.15.12:0/1021909215 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f2d480056b0 con 0x7f2d580a03a0 2024-09-11T01:46:02.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.475+0000 7f2d4ffff640 1 --2- 172.21.15.12:0/1021909215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d581aac60 0x7f2d581ab040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:02.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.475+0000 7f2d4ffff640 10 monclient: get_auth_request con 0x7f2d581aac60 auth_method 0 2024-09-11T01:46:02.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.475+0000 7f2d4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:02.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.475+0000 7f2d4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.475+0000 7f2d4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.475+0000 7f2d4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.475+0000 7f2d4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.475+0000 7f2d4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.475+0000 7f2d4ffff640 10 monclient(hunting): handle_auth_done global_id 184169 payload 824 2024-09-11T01:46:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.475+0000 7f2d4ffff640 1 -- 172.21.15.12:0/1021909215 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2d5809c4d0 con 0x7f2d581aac60 2024-09-11T01:46:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.475+0000 7f2d4effd640 1 --2- 172.21.15.12:0/1021909215 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2d3803e1f0 0x7f2d380406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.475+0000 7f2d4effd640 1 -- 172.21.15.12:0/1021909215 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(568..568 src has 317..568) ==== 4787+0+0 (secure 0 0 0) 0x7f2d48052ba0 con 0x7f2d580a03a0 2024-09-11T01:46:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.475+0000 7f2d56d9d640 1 --2- 172.21.15.12:0/1021909215 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2d3803e1f0 0x7f2d380406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.475+0000 7f2d56d9d640 10 monclient: get_auth_request con 0x7f2d3803e1f0 auth_method 0 2024-09-11T01:46:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.477+0000 7f2d4ffff640 1 -- 172.21.15.12:0/1021909215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d581aac60 msgr2=0x7f2d581ab040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.477+0000 7f2d4ffff640 1 --2- 172.21.15.12:0/1021909215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d581aac60 0x7f2d581ab040 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2d400118b0 tx=0x7f2d40011d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.477+0000 7f2d4effd640 1 -- 172.21.15.12:0/1021909215 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4000fe50 con 0x7f2d581aac60 2024-09-11T01:46:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.477+0000 7f2d4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.477+0000 7f2d4effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.477+0000 7f2d4effd640 10 monclient: dump: 2024-09-11T01:46:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:02.495 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.478+0000 7f2d56d9d640 1 --2- 172.21.15.12:0/1021909215 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2d3803e1f0 0x7f2d380406b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2d50035000 tx=0x7f2d5000b000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.478+0000 7f2d4effd640 1 -- 172.21.15.12:0/1021909215 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2d4000fe50 con 0x7f2d581aac60 2024-09-11T01:46:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.478+0000 7f2d4effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.478+0000 7f2d4effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:46:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.478+0000 7f2d4effd640 1 -- 172.21.15.12:0/1021909215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d581aac60 msgr2=0x7f2d581ab040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.478+0000 7f2d4effd640 1 --2- 172.21.15.12:0/1021909215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d581aac60 0x7f2d581ab040 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2d400118b0 tx=0x7f2d40011d70 comp rx=0 tx=0).stop 2024-09-11T01:46:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.478+0000 7f2d4effd640 1 -- 172.21.15.12:0/1021909215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d581aac60 msgr2=0x7f2d581ab040 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:46:02.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.478+0000 7f2d4effd640 1 --2- 172.21.15.12:0/1021909215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d581aac60 0x7f2d581ab040 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2d400118b0 tx=0x7f2d40011d70 comp rx=0 tx=0).stop 2024-09-11T01:46:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.484+0000 7f2d5cca9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:46:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.484+0000 7f2d5cca9640 1 --2- 172.21.15.12:0/1021909215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d581ab580 0x7f2d581a6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.484+0000 7f2d5cca9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.484+0000 7f2d5cca9640 1 -- 172.21.15.12:0/1021909215 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2d581ae060 con 0x7f2d581ab580 2024-09-11T01:46:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.484+0000 7f2d4ffff640 1 --2- 172.21.15.12:0/1021909215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d581ab580 0x7f2d581a6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.484+0000 7f2d4ffff640 10 monclient: get_auth_request con 0x7f2d581ab580 auth_method 0 2024-09-11T01:46:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.484+0000 7f2d4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.484+0000 7f2d4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.484+0000 7f2d4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.484+0000 7f2d4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.484+0000 7f2d4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.484+0000 7f2d4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.484+0000 7f2d4ffff640 10 monclient(hunting): handle_auth_done global_id 184172 payload 824 2024-09-11T01:46:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.484+0000 7f2d4ffff640 1 -- 172.21.15.12:0/1021909215 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2d5806d480 con 0x7f2d581ab580 2024-09-11T01:46:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.486+0000 7f2d4ffff640 1 -- 172.21.15.12:0/1021909215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d581ab580 msgr2=0x7f2d581a6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.486+0000 7f2d4ffff640 1 --2- 172.21.15.12:0/1021909215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d581ab580 0x7f2d581a6a10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2d400118b0 tx=0x7f2d400046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.486+0000 7f2d4effd640 1 -- 172.21.15.12:0/1021909215 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d40004220 con 0x7f2d581ab580 2024-09-11T01:46:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.486+0000 7f2d4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.486+0000 7f2d4effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.486+0000 7f2d4effd640 10 monclient: dump: 2024-09-11T01:46:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:02.500 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.487+0000 7f2d4effd640 1 -- 172.21.15.12:0/1021909215 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4530 (secure 0 0 0) 0x7f2d40004220 con 0x7f2d581ab580 2024-09-11T01:46:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.487+0000 7f2d4effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:46:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.487+0000 7f2d4effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:46:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.487+0000 7f2d4effd640 1 -- 172.21.15.12:0/1021909215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d581ab580 msgr2=0x7f2d581a6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.487+0000 7f2d4effd640 1 --2- 172.21.15.12:0/1021909215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d581ab580 0x7f2d581a6a10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2d400118b0 tx=0x7f2d400046c0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.487+0000 7f2d4effd640 1 -- 172.21.15.12:0/1021909215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d581ab580 msgr2=0x7f2d581a6a10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:46:02.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.487+0000 7f2d4effd640 1 --2- 172.21.15.12:0/1021909215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d581ab580 0x7f2d581a6a10 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2d400118b0 tx=0x7f2d400046c0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.501 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:46:02.501 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:46:02.501 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:46:02.501 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:46:02.501 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 158, 2024-09-11T01:46:02.501 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 23079, 2024-09-11T01:46:02.501 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:46:02.501 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:46:02.502 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:46:02.502 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:02.502 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:46:02.502 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:46:02.502 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:46:02.502 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:46:02.502 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:02.502 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:02.502 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:02.502 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:02.502 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:02.502 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:02.502 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:02.502 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:02.503 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:02.503 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:02.503 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:02.503 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:02.503 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:46:02.503 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:46:02.503 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:02.503 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:02.503 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:02.503 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:02.503 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:02.503 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:02.503 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:02.504 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:02.504 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:02.504 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:02.504 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:02.504 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:02.504 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:02.504 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:46:02.504 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:46:02.504 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:46:02.504 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:46:02.504 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:46:02.504 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:46:02.504 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:46:02.504 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:46:02.505 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:46:02.505 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:46:02.505 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:46:02.505 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:46:02.505 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:46:02.505 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:46:02.505 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:46:02.505 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:46:02.505 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:46:02.505 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:02.505 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:02.505 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:02.505 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:02.505 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:02.506 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:02.506 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:02.506 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:02.506 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:02.506 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:02.506 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:02.506 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:02.506 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:46:02.506 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:02.506 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:46:02.506 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:02.506 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:46:02.506 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:46:02.507 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:02.507 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:02.507 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:02.507 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:02.507 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:46:02.507 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:02.507 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:02.507 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:02.507 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:02.507 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:46:02.507 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:02.507 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:02.507 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:02.507 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:02.508 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:46:02.508 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:46:02.508 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:02.508 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:02.508 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:02.508 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:02.508 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:02.508 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:46:02.508 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:46:02.508 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:02.508 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:02.508 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:02.508 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:02.509 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:46:02.509 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:02.509 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:02.509 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:02.509 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:02.509 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:46:02.509 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:02.509 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:02.509 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:02.509 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:02.509 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:46:02.509 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:46:02.509 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:02.510 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:02.510 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:02.510 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:02.510 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:02.510 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:46:02.510 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:46:02.510 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:02.510 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:02.510 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:02.510 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:02.510 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:46:02.510 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:02.510 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:02.510 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:02.511 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:02.511 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:46:02.511 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:02.511 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:02.511 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:02.511 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:02.511 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:46:02.511 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:46:02.511 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:02.511 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:02.511 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:02.511 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:02.511 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:02.511 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:02.512 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:46:02.512 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:46:02.512 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:02.512 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:02.512 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:02.512 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:46:02.512 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:02.512 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:02.512 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:46:02.512 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:02.512 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:02.512 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:02.512 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:46:02.513 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:02.513 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:02.513 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:02.513 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:46:02.513 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:46:02.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.487+0000 7f2d5cca9640 1 -- 172.21.15.12:0/1021909215 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2d3803e1f0 msgr2=0x7f2d380406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:02.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.487+0000 7f2d5cca9640 1 --2- 172.21.15.12:0/1021909215 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2d3803e1f0 0x7f2d380406b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2d50035000 tx=0x7f2d5000b000 comp rx=0 tx=0).stop 2024-09-11T01:46:02.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.487+0000 7f2d5cca9640 10 monclient: shutdown 2024-09-11T01:46:02.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.487+0000 7f2d5cca9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:02.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.487+0000 7f2d5cca9640 1 -- 172.21.15.12:0/1021909215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d580a03a0 msgr2=0x7f2d581c0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:02.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.487+0000 7f2d5cca9640 1 --2- 172.21.15.12:0/1021909215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d580a03a0 0x7f2d581c0b80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2d4800b580 tx=0x7f2d4800ba40 comp rx=0 tx=0).stop 2024-09-11T01:46:02.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.487+0000 7f2d56d9d640 1 -- 172.21.15.12:0/1021909215 reap_dead start 2024-09-11T01:46:02.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.487+0000 7f2d56d9d640 1 -- 172.21.15.12:0/1021909215 reap_dead start 2024-09-11T01:46:02.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.487+0000 7f2d5cca9640 1 -- 172.21.15.12:0/1021909215 shutdown_connections 2024-09-11T01:46:02.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.487+0000 7f2d5cca9640 1 -- 172.21.15.12:0/1021909215 >> 172.21.15.12:0/1021909215 conn(0x7f2d580a2f70 msgr2=0x7f2d58072350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:02.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.487+0000 7f2d5cca9640 1 -- 172.21.15.12:0/1021909215 shutdown_connections 2024-09-11T01:46:02.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.487+0000 7f2d5cca9640 1 -- 172.21.15.12:0/1021909215 wait complete. 2024-09-11T01:46:02.515 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:46:02.515 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T01:46:02.516 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:46:02.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.616+0000 7f70ab46e640 10 monclient: get_monmap_and_config 2024-09-11T01:46:02.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.616+0000 7f70ab46e640 10 monclient: build_initial_monmap 2024-09-11T01:46:02.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.617+0000 7f70ab46e640 10 monclient: monmap: 2024-09-11T01:46:02.618 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:02.619 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:02.619 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:46:02.617730+0000 2024-09-11T01:46:02.619 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:46:02.617730+0000 2024-09-11T01:46:02.619 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:02.619 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:02.619 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:46:02.619 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:46:02.619 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:46:02.619 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:46:02.619 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:46:02.619 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:02.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.617+0000 7f70ab46e640 1 Processor -- start 2024-09-11T01:46:02.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.617+0000 7f70ab46e640 1 -- start start 2024-09-11T01:46:02.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.617+0000 7f70ab46e640 10 monclient: init 2024-09-11T01:46:02.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70ab46e640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:02.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70ab46e640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-11T01:46:02.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70ab46e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70a41381c0 0x7f70a41385a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:02.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70ab46e640 10 monclient(hunting): picked mon.noname-b con 0x7f70a41381c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:02.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70ab46e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f70a40a6aa0 addr v1:172.21.15.12:6789/0 2024-09-11T01:46:02.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70ab46e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f70a413d000 0x7f70a413f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:02.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70ab46e640 10 monclient(hunting): picked mon.noname-c con 0x7f70a413d000 addr v2:172.21.15.71:3300/0 2024-09-11T01:46:02.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70ab46e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70a409c310 con 0x7f70a40a6aa0 2024-09-11T01:46:02.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70ab46e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:02.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70ab46e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f70a406fc90 con 0x7f70a413d000 2024-09-11T01:46:02.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70ab46e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:02.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70ab46e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f70a409c5f0 con 0x7f70a41381c0 2024-09-11T01:46:02.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70ab46e640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:02.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70ab46e640 10 monclient(hunting): authenticate will time out at 1769.060382s 2024-09-11T01:46:02.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70a8a09640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7f70a40a6aa0 legacy=0x7f70a409cfd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:46:02.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70a920a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70a41381c0 0x7f70a41385a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:02.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70a920a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70a41381c0 0x7f70a41385a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:38252/0 (socket says 172.21.15.12:38252) 2024-09-11T01:46:02.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70a920a640 1 -- 172.21.15.12:0/2291301803 learned_addr learned my addr 172.21.15.12:0/2291301803 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:02.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70a920a640 10 monclient(hunting): get_auth_request con 0x7f70a41381c0 auth_method 0 2024-09-11T01:46:02.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70a920a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:02.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70a920a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:02.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70a920a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:02.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70a920a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:02.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70a920a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:02.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70a920a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:02.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70a9a0b640 1 --2- 172.21.15.12:0/2291301803 >> v2:172.21.15.71:3300/0 conn(0x7f70a413d000 0x7f70a413f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:02.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70a9a0b640 10 monclient(hunting): get_auth_request con 0x7f70a413d000 auth_method 0 2024-09-11T01:46:02.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70a9a0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:02.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70a9a0b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:02.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70a9a0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:02.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70a920a640 10 monclient(hunting): handle_auth_done global_id 184175 payload 274 2024-09-11T01:46:02.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70a920a640 1 -- 172.21.15.12:0/2291301803 >> v2:172.21.15.71:3300/0 conn(0x7f70a413d000 msgr2=0x7f70a413f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:46:02.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70a920a640 1 --2- 172.21.15.12:0/2291301803 >> v2:172.21.15.71:3300/0 conn(0x7f70a413d000 0x7f70a413f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70a920a640 1 -- 172.21.15.12:0/2291301803 >> v1:172.21.15.12:6789/0 conn(0x7f70a40a6aa0 legacy=0x7f70a409cfd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:46:02.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70a920a640 10 monclient: _finish_hunting 0 2024-09-11T01:46:02.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70a920a640 1 monclient: found mon.noname-b 2024-09-11T01:46:02.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70a920a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:02.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70a920a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:46:02.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70a920a640 1 -- 172.21.15.12:0/2291301803 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a409c8d0 con 0x7f70a41381c0 2024-09-11T01:46:02.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.618+0000 7f70a9a0b640 1 --2- 172.21.15.12:0/2291301803 >> v2:172.21.15.71:3300/0 conn(0x7f70a413d000 0x7f70a413f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:46:02.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.621+0000 7f70a920a640 1 -- 172.21.15.12:0/2291301803 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70a41381c0 msgr2=0x7f70a41385a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:02.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.621+0000 7f70a920a640 1 --2- 172.21.15.12:0/2291301803 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70a41381c0 0x7f70a41385a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f70a0009870 tx=0x7f70a002a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=381e1bdda0f76509 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:02.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.621+0000 7f709bfff640 1 -- 172.21.15.12:0/2291301803 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a0037070 con 0x7f70a41381c0 2024-09-11T01:46:02.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.621+0000 7f709bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:02.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.621+0000 7f709bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:46:02.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.621+0000 7f709bfff640 10 monclient: dump: 2024-09-11T01:46:02.632 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:02.632 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:02.632 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.632 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.632 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:02.632 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:02.632 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:02.632 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:02.632 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:02.633 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:02.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.621+0000 7f709bfff640 10 monclient: _finish_auth 0 2024-09-11T01:46:02.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.621+0000 7f709bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:46:02.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.621+0000 7f709bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:02.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.621+0000 7f70ab46e640 5 monclient: authenticate success, global_id 184175 2024-09-11T01:46:02.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.621+0000 7f70ab46e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:46:02.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.621+0000 7f709bfff640 1 -- 172.21.15.12:0/2291301803 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70a0032640 con 0x7f70a41381c0 2024-09-11T01:46:02.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.621+0000 7f709bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:02.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.621+0000 7f70ab46e640 10 monclient: get_monmap_and_config success 2024-09-11T01:46:02.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.621+0000 7f70ab46e640 4 set_mon_vals no callback set 2024-09-11T01:46:02.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.621+0000 7f709bfff640 1 -- 172.21.15.12:0/2291301803 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a0032a10 con 0x7f70a41381c0 2024-09-11T01:46:02.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.621+0000 7f709bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:02.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.621+0000 7f709bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:02.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.621+0000 7f709bfff640 10 monclient: dump: 2024-09-11T01:46:02.634 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:02.634 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:02.634 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.634 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.634 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:02.634 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:02.634 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:02.634 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:02.634 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:02.634 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:02.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.622+0000 7f70ab46e640 10 monclient: shutdown 2024-09-11T01:46:02.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.622+0000 7f70ab46e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:02.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.622+0000 7f70ab46e640 1 -- 172.21.15.12:0/2291301803 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70a41381c0 msgr2=0x7f70a41385a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:02.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.622+0000 7f70ab46e640 1 --2- 172.21.15.12:0/2291301803 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70a41381c0 0x7f70a41385a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f70a0009870 tx=0x7f70a002a130 comp rx=0 tx=0).stop 2024-09-11T01:46:02.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.622+0000 7f70ab46e640 1 -- 172.21.15.12:0/2291301803 shutdown_connections 2024-09-11T01:46:02.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.622+0000 7f70ab46e640 1 --2- 172.21.15.12:0/2291301803 >> v2:172.21.15.71:3300/0 conn(0x7f70a413d000 0x7f70a413f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.622+0000 7f70ab46e640 1 --2- 172.21.15.12:0/2291301803 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70a41381c0 0x7f70a41385a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.622+0000 7f70ab46e640 1 -- 172.21.15.12:0/2291301803 >> 172.21.15.12:0/2291301803 conn(0x7f70a40a7440 msgr2=0x7f70a40a7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:02.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.622+0000 7f70ab46e640 1 -- 172.21.15.12:0/2291301803 shutdown_connections 2024-09-11T01:46:02.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.622+0000 7f70ab46e640 1 -- 172.21.15.12:0/2291301803 wait complete. 2024-09-11T01:46:02.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.623+0000 7f70ab46e640 10 monclient: build_initial_monmap 2024-09-11T01:46:02.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.623+0000 7f70ab46e640 10 monclient: monmap: 2024-09-11T01:46:02.635 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:02.635 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:46:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:46:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:46:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:46:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:46:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:46:02.636 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.623+0000 7f70ab46e640 1 Processor -- start 2024-09-11T01:46:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.623+0000 7f70ab46e640 1 -- start start 2024-09-11T01:46:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.623+0000 7f70ab46e640 10 monclient: init 2024-09-11T01:46:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.623+0000 7f70ab46e640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:02.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.623+0000 7f70ab46e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:46:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.623+0000 7f70ab46e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70a40a6aa0 0x7f70a4135070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.623+0000 7f70ab46e640 10 monclient(hunting): picked mon.noname-c con 0x7f70a40a6aa0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.623+0000 7f70ab46e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70a41381c0 0x7f70a41355b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.623+0000 7f70ab46e640 10 monclient(hunting): picked mon.noname-a con 0x7f70a41381c0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:46:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.623+0000 7f70ab46e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70a413d000 0x7f70a4138da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.623+0000 7f70ab46e640 10 monclient(hunting): picked mon.noname-b con 0x7f70a413d000 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:46:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.623+0000 7f70ab46e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.623+0000 7f70ab46e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f70a413fab0 con 0x7f70a41381c0 2024-09-11T01:46:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.623+0000 7f70ab46e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.623+0000 7f70ab46e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f70a413f930 con 0x7f70a413d000 2024-09-11T01:46:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.623+0000 7f70ab46e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.623+0000 7f70ab46e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f70a413fc30 con 0x7f70a40a6aa0 2024-09-11T01:46:02.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.623+0000 7f70ab46e640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:02.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.623+0000 7f70ab46e640 10 monclient(hunting): authenticate will time out at 1769.065895s 2024-09-11T01:46:02.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.623+0000 7f70a8a09640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70a41381c0 msgr2=0x7f70a41355b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:46:02.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.623+0000 7f70a8a09640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70a41381c0 0x7f70a41355b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:46:02.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.623+0000 7f70a920a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70a40a6aa0 0x7f70a4135070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:02.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.624+0000 7f70a920a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70a40a6aa0 0x7f70a4135070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:38266/0 (socket says 172.21.15.12:38266) 2024-09-11T01:46:02.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.624+0000 7f70a920a640 1 -- 172.21.15.12:0/717603178 learned_addr learned my addr 172.21.15.12:0/717603178 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:02.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.624+0000 7f70a920a640 10 monclient(hunting): get_auth_request con 0x7f70a40a6aa0 auth_method 0 2024-09-11T01:46:02.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.624+0000 7f70a920a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:02.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.624+0000 7f70a920a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:02.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.624+0000 7f70a920a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:02.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.624+0000 7f70a920a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:02.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.624+0000 7f70a920a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.624+0000 7f70a920a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.624+0000 7f70a9a0b640 1 --2- 172.21.15.12:0/717603178 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70a413d000 0x7f70a4138da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.624+0000 7f70a9a0b640 10 monclient(hunting): get_auth_request con 0x7f70a413d000 auth_method 0 2024-09-11T01:46:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.624+0000 7f70a9a0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.624+0000 7f70a9a0b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.624+0000 7f70a9a0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.624+0000 7f70a920a640 10 monclient(hunting): handle_auth_done global_id 184178 payload 824 2024-09-11T01:46:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.624+0000 7f70a920a640 1 -- 172.21.15.12:0/717603178 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70a413d000 msgr2=0x7f70a4138da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.624+0000 7f70a920a640 1 --2- 172.21.15.12:0/717603178 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70a413d000 0x7f70a4138da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.624+0000 7f70a920a640 1 -- 172.21.15.12:0/717603178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70a41381c0 msgr2=0x7f70a41355b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:46:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.624+0000 7f70a920a640 1 --2- 172.21.15.12:0/717603178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70a41381c0 0x7f70a41355b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.624+0000 7f70a920a640 10 monclient: _finish_hunting 0 2024-09-11T01:46:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.624+0000 7f70a920a640 1 monclient: found mon.noname-c 2024-09-11T01:46:02.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.624+0000 7f70a920a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:02.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.624+0000 7f70a920a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:46:02.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.624+0000 7f70a920a640 1 -- 172.21.15.12:0/717603178 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a40a44e0 con 0x7f70a40a6aa0 2024-09-11T01:46:02.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.624+0000 7f70a9a0b640 1 --2- 172.21.15.12:0/717603178 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70a413d000 0x7f70a4138da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:46:02.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.626+0000 7f70a920a640 1 -- 172.21.15.12:0/717603178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70a40a6aa0 msgr2=0x7f70a4135070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:02.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.626+0000 7f70a920a640 1 --2- 172.21.15.12:0/717603178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70a40a6aa0 0x7f70a4135070 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f70a00090b0 tx=0x7f70a00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:02.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.626+0000 7f7099ffb640 1 -- 172.21.15.12:0/717603178 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a0041020 con 0x7f70a40a6aa0 2024-09-11T01:46:02.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.626+0000 7f7099ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:02.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.626+0000 7f7099ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:46:02.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.626+0000 7f7099ffb640 10 monclient: dump: 2024-09-11T01:46:02.640 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:02.640 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:02.640 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.640 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.641 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:02.641 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:02.641 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:02.641 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:02.641 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:02.641 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:02.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.626+0000 7f7099ffb640 10 monclient: _finish_auth 0 2024-09-11T01:46:02.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.626+0000 7f7099ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:46:02.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.626+0000 7f7099ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:02.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.626+0000 7f70ab46e640 5 monclient: authenticate success, global_id 184178 2024-09-11T01:46:02.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.626+0000 7f7099ffb640 1 -- 172.21.15.12:0/717603178 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70a003d830 con 0x7f70a40a6aa0 2024-09-11T01:46:02.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.626+0000 7f7099ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.626+0000 7f7099ffb640 1 -- 172.21.15.12:0/717603178 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a0037040 con 0x7f70a40a6aa0 2024-09-11T01:46:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.626+0000 7f70ab46e640 10 monclient: _renew_subs 2024-09-11T01:46:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.626+0000 7f70ab46e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.626+0000 7f70ab46e640 1 -- 172.21.15.12:0/717603178 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70a413bba0 con 0x7f70a40a6aa0 2024-09-11T01:46:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.626+0000 7f709b7fe640 4 set_mon_vals no callback set 2024-09-11T01:46:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.626+0000 7f7099ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.626+0000 7f7099ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.626+0000 7f7099ffb640 10 monclient: dump: 2024-09-11T01:46:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:02.642 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:02.643 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:02.643 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:02.643 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:02.643 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:02.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.626+0000 7f70ab46e640 10 monclient: _renew_subs 2024-09-11T01:46:02.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.626+0000 7f70ab46e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:02.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.626+0000 7f70ab46e640 1 -- 172.21.15.12:0/717603178 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f70a413a3a0 con 0x7f70a40a6aa0 2024-09-11T01:46:02.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.627+0000 7f70ab46e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:02.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.627+0000 7f70ab46e640 1 --2- 172.21.15.12:0/717603178 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70a40a3ca0 0x7f70a41392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:02.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.627+0000 7f70ab46e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:02.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.627+0000 7f70ab46e640 1 -- 172.21.15.12:0/717603178 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f70a40a1710 con 0x7f70a40a3ca0 2024-09-11T01:46:02.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.627+0000 7f7099ffb640 1 -- 172.21.15.12:0/717603178 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f70a003a030 con 0x7f70a40a6aa0 2024-09-11T01:46:02.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.628+0000 7f7099ffb640 1 --2- 172.21.15.12:0/717603178 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f708803e210 0x7f70880406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:02.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.628+0000 7f70a8a09640 1 --2- 172.21.15.12:0/717603178 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70a40a3ca0 0x7f70a41392e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:02.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.628+0000 7f7099ffb640 1 -- 172.21.15.12:0/717603178 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(568..568 src has 317..568) ==== 4787+0+0 (secure 0 0 0) 0x7f70a0073320 con 0x7f70a40a6aa0 2024-09-11T01:46:02.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.628+0000 7f70a8a09640 10 monclient: get_auth_request con 0x7f70a40a3ca0 auth_method 0 2024-09-11T01:46:02.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.628+0000 7f70a8a09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:02.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.628+0000 7f70a8a09640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:02.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.628+0000 7f70a8a09640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:02.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.628+0000 7f70a8a09640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:02.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.628+0000 7f70a8a09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:02.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.628+0000 7f70a8a09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:02.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.628+0000 7f70a9a0b640 1 --2- 172.21.15.12:0/717603178 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f708803e210 0x7f70880406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:02.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.628+0000 7f70a9a0b640 10 monclient: get_auth_request con 0x7f708803e210 auth_method 0 2024-09-11T01:46:02.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.628+0000 7f70a8a09640 10 monclient(hunting): handle_auth_done global_id 134611 payload 824 2024-09-11T01:46:02.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.628+0000 7f70a8a09640 1 -- 172.21.15.12:0/717603178 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f70a41385a0 con 0x7f70a40a3ca0 2024-09-11T01:46:02.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.630+0000 7f70a8a09640 1 -- 172.21.15.12:0/717603178 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70a40a3ca0 msgr2=0x7f70a41392e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:02.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.630+0000 7f70a8a09640 1 --2- 172.21.15.12:0/717603178 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70a40a3ca0 0x7f70a41392e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f70a413a620 tx=0x7f709400b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:02.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.630+0000 7f70a9a0b640 1 --2- 172.21.15.12:0/717603178 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f708803e210 0x7f70880406d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f709c003fd0 tx=0x7f709c003990 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:02.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.631+0000 7f7099ffb640 1 -- 172.21.15.12:0/717603178 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7094016070 con 0x7f70a40a3ca0 2024-09-11T01:46:02.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.631+0000 7f7099ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:02.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.631+0000 7f7099ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:46:02.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.631+0000 7f7099ffb640 10 monclient: dump: 2024-09-11T01:46:02.647 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:02.647 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:02.647 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.647 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.648 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:02.648 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:02.648 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:02.648 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:02.648 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:02.648 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:02.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.631+0000 7f7099ffb640 1 -- 172.21.15.12:0/717603178 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7094005270 con 0x7f70a40a3ca0 2024-09-11T01:46:02.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.631+0000 7f7099ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:02.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.631+0000 7f7099ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:46:02.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.631+0000 7f7099ffb640 1 -- 172.21.15.12:0/717603178 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70a40a3ca0 msgr2=0x7f70a41392e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:02.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.631+0000 7f7099ffb640 1 --2- 172.21.15.12:0/717603178 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70a40a3ca0 0x7f70a41392e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f70a413a620 tx=0x7f709400b8e0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.631+0000 7f7099ffb640 1 -- 172.21.15.12:0/717603178 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70a40a3ca0 msgr2=0x7f70a41392e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:46:02.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.631+0000 7f7099ffb640 1 --2- 172.21.15.12:0/717603178 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70a40a3ca0 0x7f70a41392e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.637+0000 7f70ab46e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:46:02.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.637+0000 7f70ab46e640 1 --2- 172.21.15.12:0/717603178 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70a4139820 0x7f70a409efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:02.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.637+0000 7f70ab46e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:02.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.637+0000 7f70ab46e640 1 -- 172.21.15.12:0/717603178 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f70a4139c30 con 0x7f70a4139820 2024-09-11T01:46:02.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.637+0000 7f70a8a09640 1 --2- 172.21.15.12:0/717603178 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70a4139820 0x7f70a409efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:02.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.637+0000 7f70a8a09640 10 monclient: get_auth_request con 0x7f70a4139820 auth_method 0 2024-09-11T01:46:02.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.637+0000 7f70a8a09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:02.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.637+0000 7f70a8a09640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:02.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.637+0000 7f70a8a09640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:02.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.637+0000 7f70a8a09640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:02.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.637+0000 7f70a8a09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:02.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.637+0000 7f70a8a09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:02.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.638+0000 7f70a8a09640 10 monclient(hunting): handle_auth_done global_id 134614 payload 824 2024-09-11T01:46:02.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.638+0000 7f70a8a09640 1 -- 172.21.15.12:0/717603178 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f70a413b6a0 con 0x7f70a4139820 2024-09-11T01:46:02.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.640+0000 7f70a8a09640 1 -- 172.21.15.12:0/717603178 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70a4139820 msgr2=0x7f70a409efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:02.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.640+0000 7f70a8a09640 1 --2- 172.21.15.12:0/717603178 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70a4139820 0x7f70a409efa0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7094013000 tx=0x7f70940059a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:02.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.640+0000 7f7099ffb640 1 -- 172.21.15.12:0/717603178 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7094016040 con 0x7f70a4139820 2024-09-11T01:46:02.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.640+0000 7f7099ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:02.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.640+0000 7f7099ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:46:02.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.640+0000 7f7099ffb640 10 monclient: dump: 2024-09-11T01:46:02.650 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:02.650 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:02.650 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.650 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:02.650 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:02.650 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:02.650 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:02.650 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:02.651 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:02.651 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:02.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.640+0000 7f7099ffb640 1 -- 172.21.15.12:0/717603178 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4867 (secure 0 0 0) 0x7f7094011400 con 0x7f70a4139820 2024-09-11T01:46:02.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.640+0000 7f7099ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:46:02.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.640+0000 7f7099ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:46:02.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.640+0000 7f7099ffb640 1 -- 172.21.15.12:0/717603178 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70a4139820 msgr2=0x7f70a409efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:02.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.640+0000 7f7099ffb640 1 --2- 172.21.15.12:0/717603178 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70a4139820 0x7f70a409efa0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7094013000 tx=0x7f70940059a0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.640+0000 7f7099ffb640 1 -- 172.21.15.12:0/717603178 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70a4139820 msgr2=0x7f70a409efa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:46:02.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.640+0000 7f7099ffb640 1 --2- 172.21.15.12:0/717603178 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70a4139820 0x7f70a409efa0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.651 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:46:02.651 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:46:02.652 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:46:02.652 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:46:02.652 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 158, 2024-09-11T01:46:02.652 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 295603, 2024-09-11T01:46:02.652 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:46:02.652 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:46:02.652 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:46:02.652 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:02.652 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:46:02.652 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:46:02.652 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:46:02.652 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:46:02.653 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:02.653 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:02.653 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:02.653 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:02.653 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:02.653 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:02.653 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:02.653 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:02.653 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:02.653 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:02.653 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:02.653 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:02.653 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:46:02.653 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:46:02.654 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:02.654 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:02.654 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:02.654 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:02.654 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:02.654 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:02.654 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:02.654 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:02.654 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:02.654 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:02.654 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:02.654 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:02.654 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:02.655 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:46:02.655 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:46:02.655 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:46:02.655 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:46:02.655 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:46:02.655 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:46:02.655 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:46:02.655 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:46:02.655 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:46:02.655 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:46:02.655 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:46:02.655 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:46:02.655 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:46:02.656 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:46:02.656 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:46:02.656 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:46:02.656 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:46:02.656 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:02.656 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:02.656 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:02.656 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:02.656 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:02.656 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:02.656 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:02.656 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:02.657 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:02.657 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:02.657 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:02.657 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:02.657 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:46:02.657 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:02.657 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:46:02.657 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:02.657 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:46:02.657 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:46:02.657 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:02.657 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:02.657 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:02.657 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:02.658 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:46:02.658 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:02.658 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:02.658 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:02.658 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:02.658 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:46:02.658 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:02.658 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:02.658 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:02.658 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:02.658 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:46:02.658 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:46:02.658 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:02.658 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:02.658 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:02.659 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:02.659 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:02.659 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:46:02.659 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:46:02.659 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:02.659 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:02.659 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:02.659 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:02.659 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:46:02.659 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:02.659 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:02.659 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:02.659 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:02.659 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:46:02.660 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:02.660 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:02.660 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:02.660 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:02.660 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:46:02.660 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:46:02.660 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:02.660 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:02.660 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:02.660 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:02.660 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:02.660 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:46:02.660 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:46:02.661 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:02.661 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:02.661 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:02.661 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:02.661 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:46:02.661 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:02.661 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:02.661 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:02.661 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:02.661 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:46:02.661 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:02.661 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:02.661 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:02.662 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:02.662 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:46:02.662 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:46:02.662 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:02.662 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:02.662 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:02.662 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:02.662 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:02.662 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:02.662 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:46:02.662 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:46:02.662 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:02.662 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:02.663 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:02.663 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:46:02.663 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:02.663 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:02.663 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:46:02.663 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:02.663 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:02.663 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:02.663 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:46:02.663 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:02.663 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:02.663 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:46:02.663 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:02.663 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:02.664 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:02.664 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:46:02.664 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:02.664 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:02.664 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:46:02.664 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:02.664 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:02.664 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:02.664 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:46:02.664 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:02.664 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:02.664 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:02.664 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:46:02.665 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:46:02.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.641+0000 7f70ab46e640 1 -- 172.21.15.12:0/717603178 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f708803e210 msgr2=0x7f70880406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:02.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.641+0000 7f70ab46e640 1 --2- 172.21.15.12:0/717603178 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f708803e210 0x7f70880406d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f709c003fd0 tx=0x7f709c003990 comp rx=0 tx=0).stop 2024-09-11T01:46:02.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.641+0000 7f70ab46e640 10 monclient: shutdown 2024-09-11T01:46:02.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.641+0000 7f70ab46e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:02.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.641+0000 7f70ab46e640 1 -- 172.21.15.12:0/717603178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70a40a6aa0 msgr2=0x7f70a4135070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:02.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.641+0000 7f70ab46e640 1 --2- 172.21.15.12:0/717603178 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70a40a6aa0 0x7f70a4135070 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f70a00090b0 tx=0x7f70a00046c0 comp rx=0 tx=0).stop 2024-09-11T01:46:02.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.641+0000 7f70a9a0b640 1 -- 172.21.15.12:0/717603178 reap_dead start 2024-09-11T01:46:02.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.641+0000 7f70a9a0b640 1 -- 172.21.15.12:0/717603178 reap_dead start 2024-09-11T01:46:02.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.641+0000 7f70ab46e640 1 -- 172.21.15.12:0/717603178 shutdown_connections 2024-09-11T01:46:02.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.641+0000 7f70ab46e640 1 -- 172.21.15.12:0/717603178 >> 172.21.15.12:0/717603178 conn(0x7f70a40a7440 msgr2=0x7f70a40a8ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:02.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.641+0000 7f70ab46e640 1 -- 172.21.15.12:0/717603178 shutdown_connections 2024-09-11T01:46:02.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:02.641+0000 7f70ab46e640 1 -- 172.21.15.12:0/717603178 wait complete. 2024-09-11T01:46:02.667 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:46:02.667 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:46:03.084 INFO:tasks.rados.rados.0.smithi071.stdout:2531: done (1 left) 2024-09-11T01:46:03.084 INFO:tasks.rados.rados.0.smithi071.stdout:2532: done (0 left) 2024-09-11T01:46:03.084 INFO:tasks.rados.rados.0.smithi071.stdout:2533: copy_from oid 17 from oid 15 current snap is 286 2024-09-11T01:46:03.084 INFO:tasks.rados.rados.0.smithi071.stdout:2534: rollback oid 25 current snap is 286 2024-09-11T01:46:03.084 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 25 to 236 2024-09-11T01:46:03.084 INFO:tasks.rados.rados.0.smithi071.stdout:2535: write oid 23 current snap is 286 2024-09-11T01:46:03.084 INFO:tasks.rados.rados.0.smithi071.stdout:2535: seq_num 904 ranges {793528=554750,2083369=436974,2958856=234672} 2024-09-11T01:46:03.092 INFO:tasks.rados.rados.0.smithi071.stdout:2535: writing smithi07134777-23 from 793528 to 1348278 tid 1 2024-09-11T01:46:03.098 INFO:tasks.rados.rados.0.smithi071.stdout:2535: writing smithi07134777-23 from 2083369 to 2520343 tid 2 2024-09-11T01:46:03.103 INFO:tasks.rados.rados.0.smithi071.stdout:2535: writing smithi07134777-23 from 2958856 to 3193528 tid 3 2024-09-11T01:46:03.103 INFO:tasks.rados.rados.0.smithi071.stdout:2536: copy_from oid 30 from oid 39 current snap is 286 2024-09-11T01:46:03.103 INFO:tasks.rados.rados.0.smithi071.stdout:2534: finishing rollback tid 1 to smithi07134777-25 2024-09-11T01:46:03.103 INFO:tasks.rados.rados.0.smithi071.stdout:2534: finishing rollback tid 2 to smithi07134777-25 2024-09-11T01:46:03.103 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 1003 (ObjNum 723 snap 224 seq_num 723) dirty exists 2024-09-11T01:46:03.103 INFO:tasks.rados.rados.0.smithi071.stdout:2533: finishing copy_from racing read to smithi07134777-17 2024-09-11T01:46:03.104 INFO:tasks.rados.rados.0.smithi071.stdout:2533: finishing copy_from to smithi07134777-17 2024-09-11T01:46:03.104 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 1604 (ObjNum 565 snap 160 seq_num 565) dirty exists 2024-09-11T01:46:03.104 INFO:tasks.rados.rados.0.smithi071.stdout:2535: finishing write tid 1 to smithi07134777-23 2024-09-11T01:46:03.104 INFO:tasks.rados.rados.0.smithi071.stdout:2535: finishing write tid 2 to smithi07134777-23 2024-09-11T01:46:03.104 INFO:tasks.rados.rados.0.smithi071.stdout:2533: done (3 left) 2024-09-11T01:46:03.104 INFO:tasks.rados.rados.0.smithi071.stdout:2534: done (2 left) 2024-09-11T01:46:03.104 INFO:tasks.rados.rados.0.smithi071.stdout:2537: read oid 41 snap 278 2024-09-11T01:46:03.104 INFO:tasks.rados.rados.0.smithi071.stdout:2537: expect (ObjNum 832 snap 254 seq_num 832) 2024-09-11T01:46:03.104 INFO:tasks.rados.rados.0.smithi071.stdout:2538: delete oid 38 current snap is 286 2024-09-11T01:46:03.105 INFO:tasks.rados.rados.0.smithi071.stdout:2536: finishing copy_from to smithi07134777-30 2024-09-11T01:46:03.105 INFO:tasks.rados.rados.0.smithi071.stdout:2536: got expected ENOENT (src dne) 2024-09-11T01:46:03.106 INFO:tasks.rados.rados.0.smithi071.stdout:2536: finishing copy_from racing read to smithi07134777-30 2024-09-11T01:46:03.106 INFO:tasks.rados.rados.0.smithi071.stdout:2537: expect (ObjNum 832 snap 254 seq_num 832) 2024-09-11T01:46:03.343 INFO:tasks.rados.rados.0.smithi071.stdout:2535: finishing write tid 3 to smithi07134777-23 2024-09-11T01:46:03.343 INFO:tasks.rados.rados.0.smithi071.stdout:2535: finishing write tid 4 to smithi07134777-23 2024-09-11T01:46:03.343 INFO:tasks.rados.rados.0.smithi071.stdout:2536: done (3 left) 2024-09-11T01:46:03.343 INFO:tasks.rados.rados.0.smithi071.stdout:2537: done (2 left) 2024-09-11T01:46:03.343 INFO:tasks.rados.rados.0.smithi071.stdout:2538: done (1 left) 2024-09-11T01:46:03.343 INFO:tasks.rados.rados.0.smithi071.stdout:2539: read oid 49 snap -1 2024-09-11T01:46:03.344 INFO:tasks.rados.rados.0.smithi071.stdout:2539: expect deleted 2024-09-11T01:46:03.344 INFO:tasks.rados.rados.0.smithi071.stdout:2540: delete oid 4 current snap is 286 2024-09-11T01:46:03.344 INFO:tasks.rados.rados.0.smithi071.stdout:2535: finishing write tid 5 to smithi07134777-23 2024-09-11T01:46:03.344 INFO:tasks.rados.rados.0.smithi071.stdout:2535: oid 23 updating version 0 to 1024 2024-09-11T01:46:03.344 INFO:tasks.rados.rados.0.smithi071.stdout:2535: oid 23 updating version 1024 to 1025 2024-09-11T01:46:03.344 INFO:tasks.rados.rados.0.smithi071.stdout:2535: oid 23 updating version 1025 to 1026 2024-09-11T01:46:03.344 INFO:tasks.rados.rados.0.smithi071.stdout:2535: oid 23 updating version 1026 to 1027 2024-09-11T01:46:03.344 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 1027 (ObjNum 904 snap 286 seq_num 904) dirty exists 2024-09-11T01:46:03.344 INFO:tasks.rados.rados.0.smithi071.stdout:2535: left oid 23 (ObjNum 904 snap 286 seq_num 904) 2024-09-11T01:46:03.350 INFO:tasks.rados.rados.0.smithi071.stdout:2535: done (2 left) 2024-09-11T01:46:03.350 INFO:tasks.rados.rados.0.smithi071.stdout:2539: done (1 left) 2024-09-11T01:46:03.350 INFO:tasks.rados.rados.0.smithi071.stdout:2540: done (0 left) 2024-09-11T01:46:03.351 INFO:tasks.rados.rados.0.smithi071.stdout:2541: snap_remove snap 268 2024-09-11T01:46:04.087 INFO:tasks.rados.rados.0.smithi071.stdout:2541: done (0 left) 2024-09-11T01:46:04.087 INFO:tasks.rados.rados.0.smithi071.stdout:2542: snap_create 2024-09-11T01:46:05.092 INFO:tasks.rados.rados.0.smithi071.stdout:2542: done (0 left) 2024-09-11T01:46:05.092 INFO:tasks.rados.rados.0.smithi071.stdout:2543: delete oid 18 current snap is 287 2024-09-11T01:46:05.097 INFO:tasks.rados.rados.0.smithi071.stdout:2543: done (0 left) 2024-09-11T01:46:05.097 INFO:tasks.rados.rados.0.smithi071.stdout:2544: read oid 29 snap -1 2024-09-11T01:46:05.098 INFO:tasks.rados.rados.0.smithi071.stdout:2544: expect (ObjNum 570 snap 161 seq_num 570) 2024-09-11T01:46:05.098 INFO:tasks.rados.rados.0.smithi071.stdout:2545: write (excl) oid 37 current snap is 287 2024-09-11T01:46:05.098 INFO:tasks.rados.rados.0.smithi071.stdout:2545: seq_num 908 ranges {460756=572099,1517297=444312,2668548=472870} 2024-09-11T01:46:05.103 INFO:tasks.rados.rados.0.smithi071.stdout:2545: writing smithi07134777-37 from 460756 to 1032855 tid 1 2024-09-11T01:46:05.108 INFO:tasks.rados.rados.0.smithi071.stdout:2545: writing smithi07134777-37 from 1517297 to 1961609 tid 2 2024-09-11T01:46:05.115 INFO:tasks.rados.rados.0.smithi071.stdout:2545: writing smithi07134777-37 from 2668548 to 3141418 tid 3 2024-09-11T01:46:05.115 INFO:tasks.rados.rados.0.smithi071.stdout:2546: read oid 24 snap -1 2024-09-11T01:46:05.115 INFO:tasks.rados.rados.0.smithi071.stdout:2546: expect (ObjNum 766 snap 232 seq_num 766) 2024-09-11T01:46:05.115 INFO:tasks.rados.rados.0.smithi071.stdout:2547: snap_remove snap 265 2024-09-11T01:46:06.098 INFO:tasks.rados.rados.0.smithi071.stdout:2547: done (3 left) 2024-09-11T01:46:06.098 INFO:tasks.rados.rados.0.smithi071.stdout:2548: write oid 44 current snap is 287 2024-09-11T01:46:06.098 INFO:tasks.rados.rados.0.smithi071.stdout:2548: seq_num 909 ranges {477563=647348,1859426=587970,2979618=297945} 2024-09-11T01:46:06.105 INFO:tasks.rados.rados.0.smithi071.stdout:2548: writing smithi07134777-44 from 477563 to 1124911 tid 1 2024-09-11T01:46:06.113 INFO:tasks.rados.rados.0.smithi071.stdout:2548: writing smithi07134777-44 from 1859426 to 2447396 tid 2 2024-09-11T01:46:06.118 INFO:tasks.rados.rados.0.smithi071.stdout:2548: writing smithi07134777-44 from 2979618 to 3277563 tid 3 2024-09-11T01:46:06.118 INFO:tasks.rados.rados.0.smithi071.stdout:2549: write oid 27 current snap is 287 2024-09-11T01:46:06.118 INFO:tasks.rados.rados.0.smithi071.stdout:2549: seq_num 910 ranges {494370=722597,1917908=447981,3007041=287329} 2024-09-11T01:46:06.125 INFO:tasks.rados.rados.0.smithi071.stdout:2549: writing smithi07134777-27 from 494370 to 1216967 tid 1 2024-09-11T01:46:06.131 INFO:tasks.rados.rados.0.smithi071.stdout:2549: writing smithi07134777-27 from 1917908 to 2365889 tid 2 2024-09-11T01:46:06.136 INFO:tasks.rados.rados.0.smithi071.stdout:2549: writing smithi07134777-27 from 3007041 to 3294370 tid 3 2024-09-11T01:46:06.136 INFO:tasks.rados.rados.0.smithi071.stdout:2550: read oid 46 snap -1 2024-09-11T01:46:06.136 INFO:tasks.rados.rados.0.smithi071.stdout:2550: expect (ObjNum 841 snap 255 seq_num 841) 2024-09-11T01:46:06.137 INFO:tasks.rados.rados.0.smithi071.stdout:2551: write oid 45 current snap is 287 2024-09-11T01:46:06.137 INFO:tasks.rados.rados.0.smithi071.stdout:2551: seq_num 911 ranges {511177=797846,1976390=591639,3034464=276713} 2024-09-11T01:46:06.143 INFO:tasks.rados.rados.0.smithi071.stdout:2551: writing smithi07134777-45 from 511177 to 1309023 tid 1 2024-09-11T01:46:06.150 INFO:tasks.rados.rados.0.smithi071.stdout:2551: writing smithi07134777-45 from 1976390 to 2568029 tid 2 2024-09-11T01:46:06.154 INFO:tasks.rados.rados.0.smithi071.stdout:2551: writing smithi07134777-45 from 3034464 to 3311177 tid 3 2024-09-11T01:46:06.154 INFO:tasks.rados.rados.0.smithi071.stdout:2552: snap_create 2024-09-11T01:46:06.155 INFO:tasks.rados.rados.0.smithi071.stdout:2544: expect (ObjNum 570 snap 161 seq_num 570) 2024-09-11T01:46:06.366 INFO:tasks.rados.rados.0.smithi071.stdout:2545: finishing write tid 1 to smithi07134777-37 2024-09-11T01:46:06.366 INFO:tasks.rados.rados.0.smithi071.stdout:2545: finishing write tid 2 to smithi07134777-37 2024-09-11T01:46:06.366 INFO:tasks.rados.rados.0.smithi071.stdout:2545: finishing write tid 3 to smithi07134777-37 2024-09-11T01:46:06.366 INFO:tasks.rados.rados.0.smithi071.stdout:2545: finishing write tid 4 to smithi07134777-37 2024-09-11T01:46:06.366 INFO:tasks.rados.rados.0.smithi071.stdout:2545: finishing write tid 5 to smithi07134777-37 2024-09-11T01:46:06.367 INFO:tasks.rados.rados.0.smithi071.stdout:2545: oid 37 updating version 0 to 1027 2024-09-11T01:46:06.367 INFO:tasks.rados.rados.0.smithi071.stdout:2545: oid 37 version 1027 is already newer than 1026 2024-09-11T01:46:06.367 INFO:tasks.rados.rados.0.smithi071.stdout:2545: oid 37 version 1027 is already newer than 1025 2024-09-11T01:46:06.367 INFO:tasks.rados.rados.0.smithi071.stdout:2545: oid 37 version 1027 is already newer than 1024 2024-09-11T01:46:06.367 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 1027 (ObjNum 908 snap 287 seq_num 908) dirty exists 2024-09-11T01:46:06.367 INFO:tasks.rados.rados.0.smithi071.stdout:2545: left oid 37 (ObjNum 908 snap 287 seq_num 908) 2024-09-11T01:46:06.367 INFO:tasks.rados.rados.0.smithi071.stdout:2546: expect (ObjNum 766 snap 232 seq_num 766) 2024-09-11T01:46:06.443 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2341930 2024-09-11T01:46:06.509 INFO:tasks.rados.rados.0.smithi071.stdout:2548: finishing write tid 1 to smithi07134777-44 2024-09-11T01:46:06.509 INFO:tasks.rados.rados.0.smithi071.stdout:2548: finishing write tid 2 to smithi07134777-44 2024-09-11T01:46:06.509 INFO:tasks.rados.rados.0.smithi071.stdout:2548: finishing write tid 3 to smithi07134777-44 2024-09-11T01:46:06.509 INFO:tasks.rados.rados.0.smithi071.stdout:2548: finishing write tid 4 to smithi07134777-44 2024-09-11T01:46:06.509 INFO:tasks.rados.rados.0.smithi071.stdout:2548: finishing write tid 5 to smithi07134777-44 2024-09-11T01:46:06.509 INFO:tasks.rados.rados.0.smithi071.stdout:2548: oid 44 updating version 0 to 1030 2024-09-11T01:46:06.509 INFO:tasks.rados.rados.0.smithi071.stdout:2548: oid 44 version 1030 is already newer than 1028 2024-09-11T01:46:06.510 INFO:tasks.rados.rados.0.smithi071.stdout:2548: oid 44 version 1030 is already newer than 1029 2024-09-11T01:46:06.510 INFO:tasks.rados.rados.0.smithi071.stdout:2548: oid 44 updating version 1030 to 1031 2024-09-11T01:46:06.510 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 1031 (ObjNum 909 snap 287 seq_num 909) dirty exists 2024-09-11T01:46:06.510 INFO:tasks.rados.rados.0.smithi071.stdout:2548: left oid 44 (ObjNum 909 snap 287 seq_num 909) 2024-09-11T01:46:06.510 INFO:tasks.rados.rados.0.smithi071.stdout:2549: finishing write tid 1 to smithi07134777-27 2024-09-11T01:46:06.510 INFO:tasks.rados.rados.0.smithi071.stdout:2549: finishing write tid 2 to smithi07134777-27 2024-09-11T01:46:06.510 INFO:tasks.rados.rados.0.smithi071.stdout:2549: finishing write tid 3 to smithi07134777-27 2024-09-11T01:46:06.510 INFO:tasks.rados.rados.0.smithi071.stdout:2550: expect (ObjNum 841 snap 255 seq_num 841) 2024-09-11T01:46:06.520 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1213908 2024-09-11T01:46:06.632 INFO:tasks.rados.rados.0.smithi071.stdout:2549: finishing write tid 4 to smithi07134777-27 2024-09-11T01:46:06.633 INFO:tasks.rados.rados.0.smithi071.stdout:2549: finishing write tid 5 to smithi07134777-27 2024-09-11T01:46:06.633 INFO:tasks.rados.rados.0.smithi071.stdout:2549: oid 27 updating version 0 to 1466 2024-09-11T01:46:06.633 INFO:tasks.rados.rados.0.smithi071.stdout:2549: oid 27 version 1466 is already newer than 1464 2024-09-11T01:46:06.633 INFO:tasks.rados.rados.0.smithi071.stdout:2549: oid 27 version 1466 is already newer than 1465 2024-09-11T01:46:06.633 INFO:tasks.rados.rados.0.smithi071.stdout:2549: oid 27 updating version 1466 to 1467 2024-09-11T01:46:06.633 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 1467 (ObjNum 910 snap 287 seq_num 910) dirty exists 2024-09-11T01:46:06.633 INFO:tasks.rados.rados.0.smithi071.stdout:2549: left oid 27 (ObjNum 910 snap 287 seq_num 910) 2024-09-11T01:46:06.633 INFO:tasks.rados.rados.0.smithi071.stdout:2551: finishing write tid 1 to smithi07134777-45 2024-09-11T01:46:06.633 INFO:tasks.rados.rados.0.smithi071.stdout:2551: finishing write tid 2 to smithi07134777-45 2024-09-11T01:46:06.633 INFO:tasks.rados.rados.0.smithi071.stdout:2551: finishing write tid 3 to smithi07134777-45 2024-09-11T01:46:06.633 INFO:tasks.rados.rados.0.smithi071.stdout:2551: finishing write tid 4 to smithi07134777-45 2024-09-11T01:46:06.634 INFO:tasks.rados.rados.0.smithi071.stdout:2551: finishing write tid 5 to smithi07134777-45 2024-09-11T01:46:06.634 INFO:tasks.rados.rados.0.smithi071.stdout:2551: oid 45 updating version 0 to 1238 2024-09-11T01:46:06.634 INFO:tasks.rados.rados.0.smithi071.stdout:2551: oid 45 updating version 1238 to 1239 2024-09-11T01:46:06.634 INFO:tasks.rados.rados.0.smithi071.stdout:2551: oid 45 version 1239 is already newer than 1237 2024-09-11T01:46:06.634 INFO:tasks.rados.rados.0.smithi071.stdout:2551: oid 45 updating version 1239 to 1240 2024-09-11T01:46:06.634 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 1240 (ObjNum 911 snap 287 seq_num 911) dirty exists 2024-09-11T01:46:06.634 INFO:tasks.rados.rados.0.smithi071.stdout:2551: left oid 45 (ObjNum 911 snap 287 seq_num 911) 2024-09-11T01:46:07.102 INFO:tasks.rados.rados.0.smithi071.stdout:2544: done (7 left) 2024-09-11T01:46:07.102 INFO:tasks.rados.rados.0.smithi071.stdout:2545: done (6 left) 2024-09-11T01:46:07.103 INFO:tasks.rados.rados.0.smithi071.stdout:2546: done (5 left) 2024-09-11T01:46:07.103 INFO:tasks.rados.rados.0.smithi071.stdout:2548: done (4 left) 2024-09-11T01:46:07.103 INFO:tasks.rados.rados.0.smithi071.stdout:2549: done (3 left) 2024-09-11T01:46:07.103 INFO:tasks.rados.rados.0.smithi071.stdout:2550: done (2 left) 2024-09-11T01:46:07.103 INFO:tasks.rados.rados.0.smithi071.stdout:2551: done (1 left) 2024-09-11T01:46:07.103 INFO:tasks.rados.rados.0.smithi071.stdout:2552: done (0 left) 2024-09-11T01:46:07.103 INFO:tasks.rados.rados.0.smithi071.stdout:2553: copy_from oid 26 from oid 18 current snap is 288 2024-09-11T01:46:07.103 INFO:tasks.rados.rados.0.smithi071.stdout:2554: snap_create 2024-09-11T01:46:07.110 INFO:tasks.rados.rados.0.smithi071.stdout:2553: finishing copy_from to smithi07134777-26 2024-09-11T01:46:07.110 INFO:tasks.rados.rados.0.smithi071.stdout:2553: got expected ENOENT (src dne) 2024-09-11T01:46:07.111 INFO:tasks.rados.rados.0.smithi071.stdout:2553: finishing copy_from racing read to smithi07134777-26 2024-09-11T01:46:08.106 INFO:tasks.rados.rados.0.smithi071.stdout:2553: done (1 left) 2024-09-11T01:46:08.106 INFO:tasks.rados.rados.0.smithi071.stdout:2554: done (0 left) 2024-09-11T01:46:08.106 INFO:tasks.rados.rados.0.smithi071.stdout:2555: snap_create 2024-09-11T01:46:09.110 INFO:tasks.rados.rados.0.smithi071.stdout:2555: done (0 left) 2024-09-11T01:46:09.110 INFO:tasks.rados.rados.0.smithi071.stdout:2556: copy_from oid 28 from oid 21 current snap is 290 2024-09-11T01:46:09.110 INFO:tasks.rados.rados.0.smithi071.stdout:2557: snap_remove snap 262 2024-09-11T01:46:10.111 INFO:tasks.rados.rados.0.smithi071.stdout:2557: done (1 left) 2024-09-11T01:46:10.111 INFO:tasks.rados.rados.0.smithi071.stdout:2558: rollback oid 29 current snap is 290 2024-09-11T01:46:10.111 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 29 to 261 2024-09-11T01:46:10.111 INFO:tasks.rados.rados.0.smithi071.stdout:2556: finishing copy_from to smithi07134777-28 2024-09-11T01:46:10.112 INFO:tasks.rados.rados.0.smithi071.stdout:2556: got expected ENOENT (src dne) 2024-09-11T01:46:10.112 INFO:tasks.rados.rados.0.smithi071.stdout:2556: finishing copy_from racing read to smithi07134777-28 2024-09-11T01:46:10.112 INFO:tasks.rados.rados.0.smithi071.stdout:2556: done (1 left) 2024-09-11T01:46:10.112 INFO:tasks.rados.rados.0.smithi071.stdout:2559: read oid 49 snap -1 2024-09-11T01:46:10.112 INFO:tasks.rados.rados.0.smithi071.stdout:2559: expect deleted 2024-09-11T01:46:10.112 INFO:tasks.rados.rados.0.smithi071.stdout:2560: delete oid 9 current snap is 290 2024-09-11T01:46:10.122 INFO:tasks.rados.rados.0.smithi071.stdout:2558: finishing rollback tid 0 to smithi07134777-29 2024-09-11T01:46:10.122 INFO:tasks.rados.rados.0.smithi071.stdout:2558: finishing rollback tid 1 to smithi07134777-29 2024-09-11T01:46:10.122 INFO:tasks.rados.rados.0.smithi071.stdout:2558: finishing rollback tid 2 to smithi07134777-29 2024-09-11T01:46:10.123 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 1646 (ObjNum 801 snap 243 seq_num 801) dirty exists 2024-09-11T01:46:10.124 INFO:tasks.rados.rados.0.smithi071.stdout:2558: done (2 left) 2024-09-11T01:46:10.124 INFO:tasks.rados.rados.0.smithi071.stdout:2559: done (1 left) 2024-09-11T01:46:10.124 INFO:tasks.rados.rados.0.smithi071.stdout:2560: done (0 left) 2024-09-11T01:46:10.124 INFO:tasks.rados.rados.0.smithi071.stdout:2561: delete oid 46 current snap is 290 2024-09-11T01:46:10.132 INFO:tasks.rados.rados.0.smithi071.stdout:2561: done (0 left) 2024-09-11T01:46:10.132 INFO:tasks.rados.rados.0.smithi071.stdout:2562: write (excl) oid 49 current snap is 290 2024-09-11T01:46:10.132 INFO:tasks.rados.rados.0.smithi071.stdout:2562: seq_num 914 ranges {561598=739946,1751836=455319,2716733=644865} 2024-09-11T01:46:10.139 INFO:tasks.rados.rados.0.smithi071.stdout:2562: writing smithi07134777-49 from 561598 to 1301544 tid 1 2024-09-11T01:46:10.144 INFO:tasks.rados.rados.0.smithi071.stdout:2562: writing smithi07134777-49 from 1751836 to 2207155 tid 2 2024-09-11T01:46:10.151 INFO:tasks.rados.rados.0.smithi071.stdout:2562: writing smithi07134777-49 from 2716733 to 3361598 tid 3 2024-09-11T01:46:10.151 INFO:tasks.rados.rados.0.smithi071.stdout:2563: write (excl) oid 16 current snap is 290 2024-09-11T01:46:10.151 INFO:tasks.rados.rados.0.smithi071.stdout:2562: finishing write tid 1 to smithi07134777-49 2024-09-11T01:46:10.151 INFO:tasks.rados.rados.0.smithi071.stdout:2563: seq_num 915 ranges {578405=415195,1410318=598977,2744156=584480} 2024-09-11T01:46:10.156 INFO:tasks.rados.rados.0.smithi071.stdout:2563: writing smithi07134777-16 from 578405 to 993600 tid 1 2024-09-11T01:46:10.162 INFO:tasks.rados.rados.0.smithi071.stdout:2563: writing smithi07134777-16 from 1410318 to 2009295 tid 2 2024-09-11T01:46:10.171 INFO:tasks.rados.rados.0.smithi071.stdout:2563: writing smithi07134777-16 from 2744156 to 3328636 tid 3 2024-09-11T01:46:10.171 INFO:tasks.rados.rados.0.smithi071.stdout:2564: copy_from oid 22 from oid 29 current snap is 290 2024-09-11T01:46:10.171 INFO:tasks.rados.rados.0.smithi071.stdout:2562: finishing write tid 2 to smithi07134777-49 2024-09-11T01:46:10.171 INFO:tasks.rados.rados.0.smithi071.stdout:2562: finishing write tid 3 to smithi07134777-49 2024-09-11T01:46:10.171 INFO:tasks.rados.rados.0.smithi071.stdout:2562: finishing write tid 4 to smithi07134777-49 2024-09-11T01:46:10.171 INFO:tasks.rados.rados.0.smithi071.stdout:2562: finishing write tid 5 to smithi07134777-49 2024-09-11T01:46:10.172 INFO:tasks.rados.rados.0.smithi071.stdout:2562: oid 49 updating version 0 to 373 2024-09-11T01:46:10.172 INFO:tasks.rados.rados.0.smithi071.stdout:2562: oid 49 updating version 373 to 375 2024-09-11T01:46:10.172 INFO:tasks.rados.rados.0.smithi071.stdout:2562: oid 49 updating version 375 to 376 2024-09-11T01:46:10.172 INFO:tasks.rados.rados.0.smithi071.stdout:2562: oid 49 version 376 is already newer than 374 2024-09-11T01:46:10.172 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 376 (ObjNum 914 snap 290 seq_num 914) dirty exists 2024-09-11T01:46:10.172 INFO:tasks.rados.rados.0.smithi071.stdout:2562: left oid 49 (ObjNum 914 snap 290 seq_num 914) 2024-09-11T01:46:10.172 INFO:tasks.rados.rados.0.smithi071.stdout:2563: finishing write tid 1 to smithi07134777-16 2024-09-11T01:46:10.172 INFO:tasks.rados.rados.0.smithi071.stdout:2563: finishing write tid 2 to smithi07134777-16 2024-09-11T01:46:10.172 INFO:tasks.rados.rados.0.smithi071.stdout:2562: done (2 left) 2024-09-11T01:46:10.172 INFO:tasks.rados.rados.0.smithi071.stdout:2565: read oid 30 snap -1 2024-09-11T01:46:10.172 INFO:tasks.rados.rados.0.smithi071.stdout:2565: expect (ObjNum 883 snap 272 seq_num 883) 2024-09-11T01:46:10.172 INFO:tasks.rados.rados.0.smithi071.stdout:2566: read oid 25 snap -1 2024-09-11T01:46:10.172 INFO:tasks.rados.rados.0.smithi071.stdout:2566: expect (ObjNum 723 snap 224 seq_num 723) 2024-09-11T01:46:10.173 INFO:tasks.rados.rados.0.smithi071.stdout:2567: rollback oid 36 current snap is 290 2024-09-11T01:46:10.173 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 36 to 264 2024-09-11T01:46:10.173 INFO:tasks.rados.rados.0.smithi071.stdout:2568: copy_from oid 27 from oid 2 current snap is 290 2024-09-11T01:46:10.173 INFO:tasks.rados.rados.0.smithi071.stdout:2569: rollback oid 3 current snap is 290 2024-09-11T01:46:10.173 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 3 to 288 2024-09-11T01:46:10.173 INFO:tasks.rados.rados.0.smithi071.stdout:2570: write (excl) oid 45 current snap is 290 2024-09-11T01:46:10.173 INFO:tasks.rados.rados.0.smithi071.stdout:2570: seq_num 916 ranges {595212=490444,1868800=458988,2887932=507280} 2024-09-11T01:46:10.176 INFO:tasks.rados.rados.0.smithi071.stdout:2570: writing smithi07134777-45 from 595212 to 1085656 tid 1 2024-09-11T01:46:10.184 INFO:tasks.rados.rados.0.smithi071.stdout:2570: writing smithi07134777-45 from 1868800 to 2327788 tid 2 2024-09-11T01:46:10.190 INFO:tasks.rados.rados.0.smithi071.stdout:2570: writing smithi07134777-45 from 2887932 to 3395212 tid 3 2024-09-11T01:46:10.190 INFO:tasks.rados.rados.0.smithi071.stdout:2571: snap_remove snap 269 2024-09-11T01:46:11.132 INFO:tasks.rados.rados.0.smithi071.stdout:2571: done (8 left) 2024-09-11T01:46:11.132 INFO:tasks.rados.rados.0.smithi071.stdout:2572: read oid 6 snap -1 2024-09-11T01:46:11.133 INFO:tasks.rados.rados.0.smithi071.stdout:2572: expect deleted 2024-09-11T01:46:11.133 INFO:tasks.rados.rados.0.smithi071.stdout:2573: write oid 7 current snap is 290 2024-09-11T01:46:11.133 INFO:tasks.rados.rados.0.smithi071.stdout:2569: finishing rollback tid 1 to smithi07134777-3 2024-09-11T01:46:11.133 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 722 (ObjNum 2609 snap 0 seq_num 2065670736) dirty dne 2024-09-11T01:46:11.133 INFO:tasks.rados.rados.0.smithi071.stdout:2573: seq_num 917 ranges {612019=565693,1810929=602646,3082649=329370} 2024-09-11T01:46:11.139 INFO:tasks.rados.rados.0.smithi071.stdout:2573: writing smithi07134777-7 from 612019 to 1177712 tid 1 2024-09-11T01:46:11.147 INFO:tasks.rados.rados.0.smithi071.stdout:2573: writing smithi07134777-7 from 1810929 to 2413575 tid 2 2024-09-11T01:46:11.153 INFO:tasks.rados.rados.0.smithi071.stdout:2573: writing smithi07134777-7 from 3082649 to 3412019 tid 3 2024-09-11T01:46:11.153 INFO:tasks.rados.rados.0.smithi071.stdout:2569: done (9 left) 2024-09-11T01:46:11.153 INFO:tasks.rados.rados.0.smithi071.stdout:2574: read oid 32 snap 254 2024-09-11T01:46:11.153 INFO:tasks.rados.rados.0.smithi071.stdout:2574: expect (ObjNum 745 snap 227 seq_num 745) 2024-09-11T01:46:11.153 INFO:tasks.rados.rados.0.smithi071.stdout:2575: snap_remove snap 235 2024-09-11T01:46:12.081 INFO:tasks.rados.rados.0.smithi071.stdout:2575: done (10 left) 2024-09-11T01:46:12.081 INFO:tasks.rados.rados.0.smithi071.stdout:2576: rollback oid 37 current snap is 290 2024-09-11T01:46:12.081 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 37 to 264 2024-09-11T01:46:12.081 INFO:tasks.rados.rados.0.smithi071.stdout:2563: finishing write tid 3 to smithi07134777-16 2024-09-11T01:46:12.081 INFO:tasks.rados.rados.0.smithi071.stdout:2563: finishing write tid 4 to smithi07134777-16 2024-09-11T01:46:12.081 INFO:tasks.rados.rados.0.smithi071.stdout:2563: finishing write tid 5 to smithi07134777-16 2024-09-11T01:46:12.082 INFO:tasks.rados.rados.0.smithi071.stdout:2563: oid 16 updating version 0 to 1253 2024-09-11T01:46:12.082 INFO:tasks.rados.rados.0.smithi071.stdout:2563: oid 16 updating version 1253 to 1254 2024-09-11T01:46:12.082 INFO:tasks.rados.rados.0.smithi071.stdout:2563: oid 16 updating version 1254 to 1255 2024-09-11T01:46:12.082 INFO:tasks.rados.rados.0.smithi071.stdout:2563: oid 16 version 1255 is already newer than 1252 2024-09-11T01:46:12.082 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 1255 (ObjNum 915 snap 290 seq_num 915) dirty exists 2024-09-11T01:46:12.082 INFO:tasks.rados.rados.0.smithi071.stdout:2563: left oid 16 (ObjNum 915 snap 290 seq_num 915) 2024-09-11T01:46:12.082 INFO:tasks.rados.rados.0.smithi071.stdout:2564: finishing copy_from racing read to smithi07134777-22 2024-09-11T01:46:12.082 INFO:tasks.rados.rados.0.smithi071.stdout:2565: expect (ObjNum 883 snap 272 seq_num 883) 2024-09-11T01:46:12.084 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 536211 2024-09-11T01:46:12.177 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1629662 2024-09-11T01:46:12.202 INFO:tasks.rados.rados.0.smithi071.stdout:2567: finishing rollback tid 0 to smithi07134777-36 2024-09-11T01:46:12.202 INFO:tasks.rados.rados.0.smithi071.stdout:2567: finishing rollback tid 1 to smithi07134777-36 2024-09-11T01:46:12.202 INFO:tasks.rados.rados.0.smithi071.stdout:2566: expect (ObjNum 723 snap 224 seq_num 723) 2024-09-11T01:46:12.203 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:46:12.225 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:46:12.265 INFO:tasks.rados.rados.0.smithi071.stdout:2570: finishing write tid 1 to smithi07134777-45 2024-09-11T01:46:12.265 INFO:tasks.rados.rados.0.smithi071.stdout:2568: finishing copy_from racing read to smithi07134777-27 2024-09-11T01:46:12.265 INFO:tasks.rados.rados.0.smithi071.stdout:2567: finishing rollback tid 2 to smithi07134777-36 2024-09-11T01:46:12.265 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 1066 (ObjNum 829 snap 252 seq_num 829) dirty exists 2024-09-11T01:46:12.265 INFO:tasks.rados.rados.0.smithi071.stdout:2568: finishing copy_from to smithi07134777-27 2024-09-11T01:46:12.266 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 854 (ObjNum 875 snap 267 seq_num 875) dirty exists 2024-09-11T01:46:12.266 INFO:tasks.rados.rados.0.smithi071.stdout:2570: finishing write tid 2 to smithi07134777-45 2024-09-11T01:46:12.266 INFO:tasks.rados.rados.0.smithi071.stdout:2564: finishing copy_from to smithi07134777-22 2024-09-11T01:46:12.266 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 1646 (ObjNum 801 snap 243 seq_num 801) dirty exists 2024-09-11T01:46:12.266 INFO:tasks.rados.rados.0.smithi071.stdout:2570: finishing write tid 3 to smithi07134777-45 2024-09-11T01:46:12.266 INFO:tasks.rados.rados.0.smithi071.stdout:2570: finishing write tid 4 to smithi07134777-45 2024-09-11T01:46:12.266 INFO:tasks.rados.rados.0.smithi071.stdout:2570: finishing write tid 5 to smithi07134777-45 2024-09-11T01:46:12.266 INFO:tasks.rados.rados.0.smithi071.stdout:2570: oid 45 updating version 0 to 1260 2024-09-11T01:46:12.266 INFO:tasks.rados.rados.0.smithi071.stdout:2570: oid 45 version 1260 is already newer than 1257 2024-09-11T01:46:12.266 INFO:tasks.rados.rados.0.smithi071.stdout:2570: oid 45 version 1260 is already newer than 1258 2024-09-11T01:46:12.266 INFO:tasks.rados.rados.0.smithi071.stdout:2570: oid 45 version 1260 is already newer than 1259 2024-09-11T01:46:12.266 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 1260 (ObjNum 916 snap 290 seq_num 916) dirty exists 2024-09-11T01:46:12.266 INFO:tasks.rados.rados.0.smithi071.stdout:2570: left oid 45 (ObjNum 916 snap 290 seq_num 916) 2024-09-11T01:46:12.266 INFO:tasks.rados.rados.0.smithi071.stdout:2563: done (10 left) 2024-09-11T01:46:12.267 INFO:tasks.rados.rados.0.smithi071.stdout:2564: done (9 left) 2024-09-11T01:46:12.267 INFO:tasks.rados.rados.0.smithi071.stdout:2565: done (8 left) 2024-09-11T01:46:12.267 INFO:tasks.rados.rados.0.smithi071.stdout:2566: done (7 left) 2024-09-11T01:46:12.267 INFO:tasks.rados.rados.0.smithi071.stdout:2567: done (6 left) 2024-09-11T01:46:12.267 INFO:tasks.rados.rados.0.smithi071.stdout:2568: done (5 left) 2024-09-11T01:46:12.267 INFO:tasks.rados.rados.0.smithi071.stdout:2570: done (4 left) 2024-09-11T01:46:12.267 INFO:tasks.rados.rados.0.smithi071.stdout:2577: write oid 24 current snap is 290 2024-09-11T01:46:12.267 INFO:tasks.rados.rados.0.smithi071.stdout:2577: seq_num 918 ranges {628826=640942,1869411=462657,2826425=534649} 2024-09-11T01:46:12.270 INFO:tasks.rados.rados.0.smithi071.stdout:2577: writing smithi07134777-24 from 628826 to 1269768 tid 1 2024-09-11T01:46:12.277 INFO:tasks.rados.rados.0.smithi071.stdout:2577: writing smithi07134777-24 from 1869411 to 2332068 tid 2 2024-09-11T01:46:12.283 INFO:tasks.rados.rados.0.smithi071.stdout:2577: writing smithi07134777-24 from 2826425 to 3361074 tid 3 2024-09-11T01:46:12.283 INFO:tasks.rados.rados.0.smithi071.stdout:2578: write oid 44 current snap is 290 2024-09-11T01:46:12.283 INFO:tasks.rados.rados.0.smithi071.stdout:2578: seq_num 919 ranges {645633=716191,1927893=606315,3137495=308138} 2024-09-11T01:46:12.290 INFO:tasks.rados.rados.0.smithi071.stdout:2578: writing smithi07134777-44 from 645633 to 1361824 tid 1 2024-09-11T01:46:12.297 INFO:tasks.rados.rados.0.smithi071.stdout:2578: writing smithi07134777-44 from 1927893 to 2534208 tid 2 2024-09-11T01:46:12.302 INFO:tasks.rados.rados.0.smithi071.stdout:2578: writing smithi07134777-44 from 3137495 to 3445633 tid 3 2024-09-11T01:46:12.302 INFO:tasks.rados.rados.0.smithi071.stdout:2574: expect (ObjNum 745 snap 227 seq_num 745) 2024-09-11T01:46:12.339 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1594650 2024-09-11T01:46:12.483 INFO:tasks.rados.rados.0.smithi071.stdout:2573: finishing write tid 1 to smithi07134777-7 2024-09-11T01:46:12.483 INFO:tasks.rados.rados.0.smithi071.stdout:2573: finishing write tid 2 to smithi07134777-7 2024-09-11T01:46:12.483 INFO:tasks.rados.rados.0.smithi071.stdout:2573: finishing write tid 3 to smithi07134777-7 2024-09-11T01:46:12.483 INFO:tasks.rados.rados.0.smithi071.stdout:2573: finishing write tid 4 to smithi07134777-7 2024-09-11T01:46:12.483 INFO:tasks.rados.rados.0.smithi071.stdout:2579: read oid 16 snap -1 2024-09-11T01:46:12.483 INFO:tasks.rados.rados.0.smithi071.stdout:2579: expect (ObjNum 915 snap 290 seq_num 915) 2024-09-11T01:46:12.483 INFO:tasks.rados.rados.0.smithi071.stdout:2572: done (6 left) 2024-09-11T01:46:12.483 INFO:tasks.rados.rados.0.smithi071.stdout:2574: done (5 left) 2024-09-11T01:46:12.483 INFO:tasks.rados.rados.0.smithi071.stdout:2580: write oid 48 current snap is 290 2024-09-11T01:46:12.483 INFO:tasks.rados.rados.0.smithi071.stdout:2573: finishing write tid 5 to smithi07134777-7 2024-09-11T01:46:12.483 INFO:tasks.rados.rados.0.smithi071.stdout:2573: oid 7 updating version 0 to 1664 2024-09-11T01:46:12.483 INFO:tasks.rados.rados.0.smithi071.stdout:2573: oid 7 version 1664 is already newer than 1663 2024-09-11T01:46:12.484 INFO:tasks.rados.rados.0.smithi071.stdout:2573: oid 7 version 1664 is already newer than 1658 2024-09-11T01:46:12.484 INFO:tasks.rados.rados.0.smithi071.stdout:2573: oid 7 updating version 1664 to 1665 2024-09-11T01:46:12.484 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 1665 (ObjNum 917 snap 290 seq_num 917) dirty exists 2024-09-11T01:46:12.484 INFO:tasks.rados.rados.0.smithi071.stdout:2573: left oid 7 (ObjNum 917 snap 290 seq_num 917) 2024-09-11T01:46:12.484 INFO:tasks.rados.rados.0.smithi071.stdout:2576: finishing rollback tid 0 to smithi07134777-37 2024-09-11T01:46:12.484 INFO:tasks.rados.rados.0.smithi071.stdout:2576: finishing rollback tid 1 to smithi07134777-37 2024-09-11T01:46:12.484 INFO:tasks.rados.rados.0.smithi071.stdout:2576: finishing rollback tid 2 to smithi07134777-37 2024-09-11T01:46:12.484 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 1030 (ObjNum 723 snap 224 seq_num 723) dirty exists 2024-09-11T01:46:12.484 INFO:tasks.rados.rados.0.smithi071.stdout:2580: seq_num 920 ranges {662440=507793,1702728=749973,2881271=557193} 2024-09-11T01:46:12.488 INFO:tasks.rados.rados.0.smithi071.stdout:2580: writing smithi07134777-48 from 662440 to 1170233 tid 1 2024-09-11T01:46:12.496 INFO:tasks.rados.rados.0.smithi071.stdout:2580: writing smithi07134777-48 from 1702728 to 2452701 tid 2 2024-09-11T01:46:12.502 INFO:tasks.rados.rados.0.smithi071.stdout:2580: writing smithi07134777-48 from 2881271 to 3438464 tid 3 2024-09-11T01:46:12.502 INFO:tasks.rados.rados.0.smithi071.stdout:2573: done (5 left) 2024-09-11T01:46:12.502 INFO:tasks.rados.rados.0.smithi071.stdout:2576: done (4 left) 2024-09-11T01:46:12.502 INFO:tasks.rados.rados.0.smithi071.stdout:2581: delete oid 40 current snap is 290 2024-09-11T01:46:12.502 INFO:tasks.rados.rados.0.smithi071.stdout:2577: finishing write tid 1 to smithi07134777-24 2024-09-11T01:46:12.502 INFO:tasks.rados.rados.0.smithi071.stdout:2577: finishing write tid 2 to smithi07134777-24 2024-09-11T01:46:12.502 INFO:tasks.rados.rados.0.smithi071.stdout:2577: finishing write tid 3 to smithi07134777-24 2024-09-11T01:46:12.502 INFO:tasks.rados.rados.0.smithi071.stdout:2577: finishing write tid 4 to smithi07134777-24 2024-09-11T01:46:12.503 INFO:tasks.rados.rados.0.smithi071.stdout:2577: finishing write tid 5 to smithi07134777-24 2024-09-11T01:46:12.503 INFO:tasks.rados.rados.0.smithi071.stdout:2577: oid 24 updating version 0 to 1282 2024-09-11T01:46:12.503 INFO:tasks.rados.rados.0.smithi071.stdout:2577: oid 24 version 1282 is already newer than 1281 2024-09-11T01:46:12.503 INFO:tasks.rados.rados.0.smithi071.stdout:2577: oid 24 updating version 1282 to 1283 2024-09-11T01:46:12.503 INFO:tasks.rados.rados.0.smithi071.stdout:2577: oid 24 version 1283 is already newer than 1280 2024-09-11T01:46:12.503 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 1283 (ObjNum 918 snap 290 seq_num 918) dirty exists 2024-09-11T01:46:12.503 INFO:tasks.rados.rados.0.smithi071.stdout:2577: left oid 24 (ObjNum 918 snap 290 seq_num 918) 2024-09-11T01:46:12.503 INFO:tasks.rados.rados.0.smithi071.stdout:2578: finishing write tid 1 to smithi07134777-44 2024-09-11T01:46:12.503 INFO:tasks.rados.rados.0.smithi071.stdout:2578: finishing write tid 2 to smithi07134777-44 2024-09-11T01:46:12.503 INFO:tasks.rados.rados.0.smithi071.stdout:2578: finishing write tid 3 to smithi07134777-44 2024-09-11T01:46:12.503 INFO:tasks.rados.rados.0.smithi071.stdout:2578: finishing write tid 4 to smithi07134777-44 2024-09-11T01:46:12.503 INFO:tasks.rados.rados.0.smithi071.stdout:2578: finishing write tid 5 to smithi07134777-44 2024-09-11T01:46:12.503 INFO:tasks.rados.rados.0.smithi071.stdout:2578: oid 44 updating version 0 to 1033 2024-09-11T01:46:12.504 INFO:tasks.rados.rados.0.smithi071.stdout:2578: oid 44 updating version 1033 to 1035 2024-09-11T01:46:12.504 INFO:tasks.rados.rados.0.smithi071.stdout:2578: oid 44 version 1035 is already newer than 1032 2024-09-11T01:46:12.504 INFO:tasks.rados.rados.0.smithi071.stdout:2578: oid 44 version 1035 is already newer than 1034 2024-09-11T01:46:12.504 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 1035 (ObjNum 919 snap 290 seq_num 919) dirty exists 2024-09-11T01:46:12.504 INFO:tasks.rados.rados.0.smithi071.stdout:2578: left oid 44 (ObjNum 919 snap 290 seq_num 919) 2024-09-11T01:46:12.504 INFO:tasks.rados.rados.0.smithi071.stdout:2580: finishing write tid 1 to smithi07134777-48 2024-09-11T01:46:12.504 INFO:tasks.rados.rados.0.smithi071.stdout:2579: expect (ObjNum 915 snap 290 seq_num 915) 2024-09-11T01:46:12.504 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:46:12.703 INFO:tasks.rados.rados.0.smithi071.stdout:2580: finishing write tid 2 to smithi07134777-48 2024-09-11T01:46:12.704 INFO:tasks.rados.rados.0.smithi071.stdout:2580: finishing write tid 3 to smithi07134777-48 2024-09-11T01:46:12.704 INFO:tasks.rados.rados.0.smithi071.stdout:2580: finishing write tid 4 to smithi07134777-48 2024-09-11T01:46:12.704 INFO:tasks.rados.rados.0.smithi071.stdout:2580: finishing write tid 5 to smithi07134777-48 2024-09-11T01:46:12.704 INFO:tasks.rados.rados.0.smithi071.stdout:2580: oid 48 updating version 0 to 1680 2024-09-11T01:46:12.704 INFO:tasks.rados.rados.0.smithi071.stdout:2580: oid 48 updating version 1680 to 1683 2024-09-11T01:46:12.704 INFO:tasks.rados.rados.0.smithi071.stdout:2580: oid 48 version 1683 is already newer than 1681 2024-09-11T01:46:12.704 INFO:tasks.rados.rados.0.smithi071.stdout:2580: oid 48 version 1683 is already newer than 1682 2024-09-11T01:46:12.704 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 1683 (ObjNum 920 snap 290 seq_num 920) dirty exists 2024-09-11T01:46:12.704 INFO:tasks.rados.rados.0.smithi071.stdout:2580: left oid 48 (ObjNum 920 snap 290 seq_num 920) 2024-09-11T01:46:12.704 INFO:tasks.rados.rados.0.smithi071.stdout:2577: done (4 left) 2024-09-11T01:46:12.704 INFO:tasks.rados.rados.0.smithi071.stdout:2578: done (3 left) 2024-09-11T01:46:12.704 INFO:tasks.rados.rados.0.smithi071.stdout:2579: done (2 left) 2024-09-11T01:46:12.704 INFO:tasks.rados.rados.0.smithi071.stdout:2580: done (1 left) 2024-09-11T01:46:12.705 INFO:tasks.rados.rados.0.smithi071.stdout:2581: done (0 left) 2024-09-11T01:46:12.705 INFO:tasks.rados.rados.0.smithi071.stdout:2582: write oid 6 current snap is 290 2024-09-11T01:46:12.705 INFO:tasks.rados.rados.0.smithi071.stdout:2582: seq_num 922 ranges {696054=658291,2103339=753642} 2024-09-11T01:46:12.710 INFO:tasks.rados.rados.0.smithi071.stdout:2582: writing smithi07134777-6 from 696054 to 1354345 tid 1 2024-09-11T01:46:12.718 INFO:tasks.rados.rados.0.smithi071.stdout:2582: writing smithi07134777-6 from 2103339 to 2856981 tid 2 2024-09-11T01:46:12.718 INFO:tasks.rados.rados.0.smithi071.stdout:2583: write (excl) oid 10 current snap is 290 2024-09-11T01:46:12.718 INFO:tasks.rados.rados.0.smithi071.stdout:2583: seq_num 923 ranges {712861=733540,2161821=613653,3247187=265674} 2024-09-11T01:46:12.729 INFO:tasks.rados.rados.0.smithi071.stdout:2583: writing smithi07134777-10 from 712861 to 1446401 tid 1 2024-09-11T01:46:12.733 INFO:tasks.rados.rados.0.smithi071.stdout:2583: writing smithi07134777-10 from 2161821 to 2775474 tid 2 2024-09-11T01:46:12.738 INFO:tasks.rados.rados.0.smithi071.stdout:2583: writing smithi07134777-10 from 3247187 to 3512861 tid 3 2024-09-11T01:46:12.738 INFO:tasks.rados.rados.0.smithi071.stdout:2584: copy_from oid 29 from oid 22 current snap is 290 2024-09-11T01:46:12.738 INFO:tasks.rados.rados.0.smithi071.stdout:2582: finishing write tid 1 to smithi07134777-6 2024-09-11T01:46:12.738 INFO:tasks.rados.rados.0.smithi071.stdout:2582: finishing write tid 2 to smithi07134777-6 2024-09-11T01:46:12.738 INFO:tasks.rados.rados.0.smithi071.stdout:2582: finishing write tid 3 to smithi07134777-6 2024-09-11T01:46:12.738 INFO:tasks.rados.rados.0.smithi071.stdout:2582: finishing write tid 4 to smithi07134777-6 2024-09-11T01:46:12.738 INFO:tasks.rados.rados.0.smithi071.stdout:2582: oid 6 updating version 0 to 1496 2024-09-11T01:46:12.738 INFO:tasks.rados.rados.0.smithi071.stdout:2582: oid 6 version 1496 is already newer than 1495 2024-09-11T01:46:12.738 INFO:tasks.rados.rados.0.smithi071.stdout:2582: oid 6 version 1496 is already newer than 1494 2024-09-11T01:46:12.739 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 1496 (ObjNum 922 snap 290 seq_num 922) dirty exists 2024-09-11T01:46:12.739 INFO:tasks.rados.rados.0.smithi071.stdout:2582: left oid 6 (ObjNum 922 snap 290 seq_num 922) 2024-09-11T01:46:12.739 INFO:tasks.rados.rados.0.smithi071.stdout:2583: finishing write tid 1 to smithi07134777-10 2024-09-11T01:46:12.739 INFO:tasks.rados.rados.0.smithi071.stdout:2583: finishing write tid 2 to smithi07134777-10 2024-09-11T01:46:12.739 INFO:tasks.rados.rados.0.smithi071.stdout:2582: done (2 left) 2024-09-11T01:46:12.739 INFO:tasks.rados.rados.0.smithi071.stdout:2585: write oid 7 current snap is 290 2024-09-11T01:46:12.739 INFO:tasks.rados.rados.0.smithi071.stdout:2585: seq_num 924 ranges {729668=408789,1820303=757311,3274610=255058} 2024-09-11T01:46:12.744 INFO:tasks.rados.rados.0.smithi071.stdout:2585: writing smithi07134777-7 from 729668 to 1138457 tid 1 2024-09-11T01:46:12.752 INFO:tasks.rados.rados.0.smithi071.stdout:2585: writing smithi07134777-7 from 1820303 to 2577614 tid 2 2024-09-11T01:46:12.757 INFO:tasks.rados.rados.0.smithi071.stdout:2585: writing smithi07134777-7 from 3274610 to 3529668 tid 3 2024-09-11T01:46:12.758 INFO:tasks.rados.rados.0.smithi071.stdout:2586: snap_create 2024-09-11T01:46:12.758 INFO:tasks.rados.rados.0.smithi071.stdout:2583: finishing write tid 3 to smithi07134777-10 2024-09-11T01:46:12.758 INFO:tasks.rados.rados.0.smithi071.stdout:2583: finishing write tid 4 to smithi07134777-10 2024-09-11T01:46:12.758 INFO:tasks.rados.rados.0.smithi071.stdout:2583: finishing write tid 5 to smithi07134777-10 2024-09-11T01:46:12.758 INFO:tasks.rados.rados.0.smithi071.stdout:2583: oid 10 updating version 0 to 422 2024-09-11T01:46:12.758 INFO:tasks.rados.rados.0.smithi071.stdout:2583: oid 10 version 422 is already newer than 420 2024-09-11T01:46:12.758 INFO:tasks.rados.rados.0.smithi071.stdout:2583: oid 10 version 422 is already newer than 421 2024-09-11T01:46:12.758 INFO:tasks.rados.rados.0.smithi071.stdout:2583: oid 10 version 422 is already newer than 419 2024-09-11T01:46:12.758 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 422 (ObjNum 923 snap 290 seq_num 923) dirty exists 2024-09-11T01:46:12.758 INFO:tasks.rados.rados.0.smithi071.stdout:2583: left oid 10 (ObjNum 923 snap 290 seq_num 923) 2024-09-11T01:46:12.758 INFO:tasks.rados.rados.0.smithi071.stdout:2584: finishing copy_from racing read to smithi07134777-29 2024-09-11T01:46:12.758 INFO:tasks.rados.rados.0.smithi071.stdout:2585: finishing write tid 1 to smithi07134777-7 2024-09-11T01:46:12.759 INFO:tasks.rados.rados.0.smithi071.stdout:2584: finishing copy_from to smithi07134777-29 2024-09-11T01:46:12.759 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 1646 (ObjNum 801 snap 243 seq_num 801) dirty exists 2024-09-11T01:46:12.759 INFO:tasks.rados.rados.0.smithi071.stdout:2585: finishing write tid 2 to smithi07134777-7 2024-09-11T01:46:12.762 INFO:tasks.rados.rados.0.smithi071.stdout:2585: finishing write tid 3 to smithi07134777-7 2024-09-11T01:46:12.764 INFO:tasks.rados.rados.0.smithi071.stdout:2585: finishing write tid 4 to smithi07134777-7 2024-09-11T01:46:12.764 INFO:tasks.rados.rados.0.smithi071.stdout:2585: finishing write tid 5 to smithi07134777-7 2024-09-11T01:46:12.764 INFO:tasks.rados.rados.0.smithi071.stdout:2585: oid 7 updating version 0 to 1684 2024-09-11T01:46:12.764 INFO:tasks.rados.rados.0.smithi071.stdout:2585: oid 7 updating version 1684 to 1687 2024-09-11T01:46:12.764 INFO:tasks.rados.rados.0.smithi071.stdout:2585: oid 7 updating version 1687 to 1688 2024-09-11T01:46:12.764 INFO:tasks.rados.rados.0.smithi071.stdout:2585: oid 7 version 1688 is already newer than 1686 2024-09-11T01:46:12.764 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 1688 (ObjNum 924 snap 290 seq_num 924) dirty exists 2024-09-11T01:46:12.764 INFO:tasks.rados.rados.0.smithi071.stdout:2585: left oid 7 (ObjNum 924 snap 290 seq_num 924) 2024-09-11T01:46:13.140 INFO:tasks.rados.rados.0.smithi071.stdout:2583: done (3 left) 2024-09-11T01:46:13.140 INFO:tasks.rados.rados.0.smithi071.stdout:2584: done (2 left) 2024-09-11T01:46:13.141 INFO:tasks.rados.rados.0.smithi071.stdout:2585: done (1 left) 2024-09-11T01:46:13.141 INFO:tasks.rados.rados.0.smithi071.stdout:2586: done (0 left) 2024-09-11T01:46:13.141 INFO:tasks.rados.rados.0.smithi071.stdout:2587: read oid 1 snap 261 2024-09-11T01:46:13.141 INFO:tasks.rados.rados.0.smithi071.stdout:2587: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:46:13.141 INFO:tasks.rados.rados.0.smithi071.stdout:2588: rollback oid 14 current snap is 291 2024-09-11T01:46:13.141 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 14 to 254 2024-09-11T01:46:13.141 INFO:tasks.rados.rados.0.smithi071.stdout:2589: read oid 47 snap -1 2024-09-11T01:46:13.141 INFO:tasks.rados.rados.0.smithi071.stdout:2589: expect (ObjNum 704 snap 213 seq_num 704) 2024-09-11T01:46:13.141 INFO:tasks.rados.rados.0.smithi071.stdout:2590: snap_create 2024-09-11T01:46:13.143 INFO:tasks.rados.rados.0.smithi071.stdout:2587: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:46:13.153 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3071437 2024-09-11T01:46:13.221 INFO:tasks.rados.rados.0.smithi071.stdout:2588: finishing rollback tid 0 to smithi07134777-14 2024-09-11T01:46:13.222 INFO:tasks.rados.rados.0.smithi071.stdout:2588: finishing rollback tid 1 to smithi07134777-14 2024-09-11T01:46:13.222 INFO:tasks.rados.rados.0.smithi071.stdout:2588: finishing rollback tid 2 to smithi07134777-14 2024-09-11T01:46:13.222 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 1692 (ObjNum 797 snap 243 seq_num 797) dirty exists 2024-09-11T01:46:13.222 INFO:tasks.rados.rados.0.smithi071.stdout:2589: expect (ObjNum 704 snap 213 seq_num 704) 2024-09-11T01:46:13.235 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2074199 2024-09-11T01:46:13.329 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:46:14.139 INFO:tasks.rados.rados.0.smithi071.stdout:2587: done (3 left) 2024-09-11T01:46:14.139 INFO:tasks.rados.rados.0.smithi071.stdout:2588: done (2 left) 2024-09-11T01:46:14.139 INFO:tasks.rados.rados.0.smithi071.stdout:2589: done (1 left) 2024-09-11T01:46:14.139 INFO:tasks.rados.rados.0.smithi071.stdout:2590: done (0 left) 2024-09-11T01:46:14.139 INFO:tasks.rados.rados.0.smithi071.stdout:2591: copy_from oid 1 from oid 34 current snap is 292 2024-09-11T01:46:14.139 INFO:tasks.rados.rados.0.smithi071.stdout:2592: write (excl) oid 41 current snap is 292 2024-09-11T01:46:14.139 INFO:tasks.rados.rados.0.smithi071.stdout:2592: seq_num 925 ranges {746475=484038,1762432=617322,2785680=529906} 2024-09-11T01:46:14.145 INFO:tasks.rados.rados.0.smithi071.stdout:2592: writing smithi07134777-41 from 746475 to 1230513 tid 1 2024-09-11T01:46:14.152 INFO:tasks.rados.rados.0.smithi071.stdout:2592: writing smithi07134777-41 from 1762432 to 2379754 tid 2 2024-09-11T01:46:14.158 INFO:tasks.rados.rados.0.smithi071.stdout:2592: writing smithi07134777-41 from 2785680 to 3315586 tid 3 2024-09-11T01:46:14.158 INFO:tasks.rados.rados.0.smithi071.stdout:2593: write oid 6 current snap is 292 2024-09-11T01:46:14.158 INFO:tasks.rados.rados.0.smithi071.stdout:2592: finishing write tid 1 to smithi07134777-41 2024-09-11T01:46:14.158 INFO:tasks.rados.rados.0.smithi071.stdout:2593: seq_num 926 ranges {763282=559287,1820914=760980,3213103=350179} 2024-09-11T01:46:14.165 INFO:tasks.rados.rados.0.smithi071.stdout:2593: writing smithi07134777-6 from 763282 to 1322569 tid 1 2024-09-11T01:46:14.173 INFO:tasks.rados.rados.0.smithi071.stdout:2593: writing smithi07134777-6 from 1820914 to 2581894 tid 2 2024-09-11T01:46:14.178 INFO:tasks.rados.rados.0.smithi071.stdout:2593: writing smithi07134777-6 from 3213103 to 3563282 tid 3 2024-09-11T01:46:14.178 INFO:tasks.rados.rados.0.smithi071.stdout:2594: copy_from oid 4 from oid 24 current snap is 292 2024-09-11T01:46:14.178 INFO:tasks.rados.rados.0.smithi071.stdout:2592: finishing write tid 2 to smithi07134777-41 2024-09-11T01:46:14.178 INFO:tasks.rados.rados.0.smithi071.stdout:2591: finishing copy_from racing read to smithi07134777-1 2024-09-11T01:46:14.178 INFO:tasks.rados.rados.0.smithi071.stdout:2591: finishing copy_from to smithi07134777-1 2024-09-11T01:46:14.178 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 1325 (ObjNum 857 snap 263 seq_num 857) dirty exists 2024-09-11T01:46:14.178 INFO:tasks.rados.rados.0.smithi071.stdout:2592: finishing write tid 3 to smithi07134777-41 2024-09-11T01:46:14.178 INFO:tasks.rados.rados.0.smithi071.stdout:2592: finishing write tid 4 to smithi07134777-41 2024-09-11T01:46:14.178 INFO:tasks.rados.rados.0.smithi071.stdout:2592: finishing write tid 5 to smithi07134777-41 2024-09-11T01:46:14.178 INFO:tasks.rados.rados.0.smithi071.stdout:2592: oid 41 updating version 0 to 425 2024-09-11T01:46:14.179 INFO:tasks.rados.rados.0.smithi071.stdout:2592: oid 41 updating version 425 to 427 2024-09-11T01:46:14.179 INFO:tasks.rados.rados.0.smithi071.stdout:2592: oid 41 version 427 is already newer than 424 2024-09-11T01:46:14.179 INFO:tasks.rados.rados.0.smithi071.stdout:2592: oid 41 version 427 is already newer than 426 2024-09-11T01:46:14.179 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 427 (ObjNum 925 snap 292 seq_num 925) dirty exists 2024-09-11T01:46:14.179 INFO:tasks.rados.rados.0.smithi071.stdout:2592: left oid 41 (ObjNum 925 snap 292 seq_num 925) 2024-09-11T01:46:14.179 INFO:tasks.rados.rados.0.smithi071.stdout:2593: finishing write tid 1 to smithi07134777-6 2024-09-11T01:46:14.179 INFO:tasks.rados.rados.0.smithi071.stdout:2591: done (3 left) 2024-09-11T01:46:14.179 INFO:tasks.rados.rados.0.smithi071.stdout:2592: done (2 left) 2024-09-11T01:46:14.179 INFO:tasks.rados.rados.0.smithi071.stdout:2595: delete oid 30 current snap is 292 2024-09-11T01:46:14.179 INFO:tasks.rados.rados.0.smithi071.stdout:2593: finishing write tid 2 to smithi07134777-6 2024-09-11T01:46:14.190 INFO:tasks.rados.rados.0.smithi071.stdout:2593: finishing write tid 3 to smithi07134777-6 2024-09-11T01:46:14.191 INFO:tasks.rados.rados.0.smithi071.stdout:2593: finishing write tid 4 to smithi07134777-6 2024-09-11T01:46:14.191 INFO:tasks.rados.rados.0.smithi071.stdout:2594: finishing copy_from racing read to smithi07134777-4 2024-09-11T01:46:14.191 INFO:tasks.rados.rados.0.smithi071.stdout:2593: finishing write tid 5 to smithi07134777-6 2024-09-11T01:46:14.191 INFO:tasks.rados.rados.0.smithi071.stdout:2593: oid 6 updating version 0 to 1498 2024-09-11T01:46:14.191 INFO:tasks.rados.rados.0.smithi071.stdout:2593: oid 6 updating version 1498 to 1500 2024-09-11T01:46:14.191 INFO:tasks.rados.rados.0.smithi071.stdout:2593: oid 6 updating version 1500 to 1501 2024-09-11T01:46:14.191 INFO:tasks.rados.rados.0.smithi071.stdout:2593: oid 6 version 1501 is already newer than 1499 2024-09-11T01:46:14.191 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 1501 (ObjNum 926 snap 292 seq_num 926) dirty exists 2024-09-11T01:46:14.191 INFO:tasks.rados.rados.0.smithi071.stdout:2593: left oid 6 (ObjNum 926 snap 292 seq_num 926) 2024-09-11T01:46:14.192 INFO:tasks.rados.rados.0.smithi071.stdout:2593: done (2 left) 2024-09-11T01:46:14.192 INFO:tasks.rados.rados.0.smithi071.stdout:2595: done (1 left) 2024-09-11T01:46:14.192 INFO:tasks.rados.rados.0.smithi071.stdout:2596: delete oid 43 current snap is 292 2024-09-11T01:46:14.195 INFO:tasks.rados.rados.0.smithi071.stdout:2596: done (1 left) 2024-09-11T01:46:14.195 INFO:tasks.rados.rados.0.smithi071.stdout:2597: read oid 48 snap -1 2024-09-11T01:46:14.195 INFO:tasks.rados.rados.0.smithi071.stdout:2597: expect (ObjNum 920 snap 290 seq_num 920) 2024-09-11T01:46:14.195 INFO:tasks.rados.rados.0.smithi071.stdout:2598: delete oid 46 current snap is 292 2024-09-11T01:46:14.197 INFO:tasks.rados.rados.0.smithi071.stdout:2598: done (2 left) 2024-09-11T01:46:14.197 INFO:tasks.rados.rados.0.smithi071.stdout:2599: snap_remove snap 285 2024-09-11T01:46:15.149 INFO:tasks.rados.rados.0.smithi071.stdout:2599: done (2 left) 2024-09-11T01:46:15.149 INFO:tasks.rados.rados.0.smithi071.stdout:2600: snap_create 2024-09-11T01:46:15.149 INFO:tasks.rados.rados.0.smithi071.stdout:2594: finishing copy_from to smithi07134777-4 2024-09-11T01:46:15.149 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 1283 (ObjNum 918 snap 290 seq_num 918) dirty exists 2024-09-11T01:46:15.149 INFO:tasks.rados.rados.0.smithi071.stdout:2597: expect (ObjNum 920 snap 290 seq_num 920) 2024-09-11T01:46:15.152 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1329547 2024-09-11T01:46:15.180 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:46:15.217 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:46:16.149 INFO:tasks.rados.rados.0.smithi071.stdout:2594: done (2 left) 2024-09-11T01:46:16.149 INFO:tasks.rados.rados.0.smithi071.stdout:2597: done (1 left) 2024-09-11T01:46:16.149 INFO:tasks.rados.rados.0.smithi071.stdout:2600: done (0 left) 2024-09-11T01:46:16.149 INFO:tasks.rados.rados.0.smithi071.stdout:2601: read oid 18 snap -1 2024-09-11T01:46:16.149 INFO:tasks.rados.rados.0.smithi071.stdout:2601: expect deleted 2024-09-11T01:46:16.149 INFO:tasks.rados.rados.0.smithi071.stdout:2602: read oid 15 snap -1 2024-09-11T01:46:16.150 INFO:tasks.rados.rados.0.smithi071.stdout:2602: expect (ObjNum 565 snap 160 seq_num 565) 2024-09-11T01:46:16.150 INFO:tasks.rados.rados.0.smithi071.stdout:2603: write oid 13 current snap is 293 2024-09-11T01:46:16.150 INFO:tasks.rados.rados.0.smithi071.stdout:2603: seq_num 930 ranges {430510=576636,1654842=768318,2922795=502619} 2024-09-11T01:46:16.154 INFO:tasks.rados.rados.0.smithi071.stdout:2603: writing smithi07134777-13 from 430510 to 1007146 tid 1 2024-09-11T01:46:16.162 INFO:tasks.rados.rados.0.smithi071.stdout:2603: writing smithi07134777-13 from 1654842 to 2423160 tid 2 2024-09-11T01:46:16.168 INFO:tasks.rados.rados.0.smithi071.stdout:2603: writing smithi07134777-13 from 2922795 to 3425414 tid 3 2024-09-11T01:46:16.168 INFO:tasks.rados.rados.0.smithi071.stdout:2604: write oid 17 current snap is 293 2024-09-11T01:46:16.168 INFO:tasks.rados.rados.0.smithi071.stdout:2604: seq_num 931 ranges {447317=651885,1713324=628329,3066571=580746} 2024-09-11T01:46:16.174 INFO:tasks.rados.rados.0.smithi071.stdout:2604: writing smithi07134777-17 from 447317 to 1099202 tid 1 2024-09-11T01:46:16.181 INFO:tasks.rados.rados.0.smithi071.stdout:2604: writing smithi07134777-17 from 1713324 to 2341653 tid 2 2024-09-11T01:46:16.189 INFO:tasks.rados.rados.0.smithi071.stdout:2604: writing smithi07134777-17 from 3066571 to 3647317 tid 3 2024-09-11T01:46:16.189 INFO:tasks.rados.rados.0.smithi071.stdout:2605: write (excl) oid 45 current snap is 293 2024-09-11T01:46:16.189 INFO:tasks.rados.rados.0.smithi071.stdout:2602: expect (ObjNum 565 snap 160 seq_num 565) 2024-09-11T01:46:16.189 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:46:16.421 INFO:tasks.rados.rados.0.smithi071.stdout:2603: finishing write tid 1 to smithi07134777-13 2024-09-11T01:46:16.421 INFO:tasks.rados.rados.0.smithi071.stdout:2603: finishing write tid 2 to smithi07134777-13 2024-09-11T01:46:16.421 INFO:tasks.rados.rados.0.smithi071.stdout:2603: finishing write tid 3 to smithi07134777-13 2024-09-11T01:46:16.421 INFO:tasks.rados.rados.0.smithi071.stdout:2603: finishing write tid 4 to smithi07134777-13 2024-09-11T01:46:16.422 INFO:tasks.rados.rados.0.smithi071.stdout:2603: finishing write tid 5 to smithi07134777-13 2024-09-11T01:46:16.422 INFO:tasks.rados.rados.0.smithi071.stdout:2603: oid 13 updating version 0 to 1608 2024-09-11T01:46:16.422 INFO:tasks.rados.rados.0.smithi071.stdout:2603: oid 13 version 1608 is already newer than 1605 2024-09-11T01:46:16.422 INFO:tasks.rados.rados.0.smithi071.stdout:2603: oid 13 version 1608 is already newer than 1606 2024-09-11T01:46:16.422 INFO:tasks.rados.rados.0.smithi071.stdout:2603: oid 13 version 1608 is already newer than 1607 2024-09-11T01:46:16.422 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 1608 (ObjNum 930 snap 293 seq_num 930) dirty exists 2024-09-11T01:46:16.422 INFO:tasks.rados.rados.0.smithi071.stdout:2603: left oid 13 (ObjNum 930 snap 293 seq_num 930) 2024-09-11T01:46:16.422 INFO:tasks.rados.rados.0.smithi071.stdout:2604: finishing write tid 1 to smithi07134777-17 2024-09-11T01:46:16.422 INFO:tasks.rados.rados.0.smithi071.stdout:2604: finishing write tid 2 to smithi07134777-17 2024-09-11T01:46:16.422 INFO:tasks.rados.rados.0.smithi071.stdout:2604: finishing write tid 3 to smithi07134777-17 2024-09-11T01:46:16.422 INFO:tasks.rados.rados.0.smithi071.stdout:2605: seq_num 932 ranges {464124=727134,1771806=771987,2977641=641516} 2024-09-11T01:46:16.425 INFO:tasks.rados.rados.0.smithi071.stdout:2605: writing smithi07134777-45 from 464124 to 1191258 tid 1 2024-09-11T01:46:16.432 INFO:tasks.rados.rados.0.smithi071.stdout:2605: writing smithi07134777-45 from 1771806 to 2543793 tid 2 2024-09-11T01:46:16.440 INFO:tasks.rados.rados.0.smithi071.stdout:2605: writing smithi07134777-45 from 2977641 to 3619157 tid 3 2024-09-11T01:46:16.440 INFO:tasks.rados.rados.0.smithi071.stdout:2601: done (4 left) 2024-09-11T01:46:16.440 INFO:tasks.rados.rados.0.smithi071.stdout:2602: done (3 left) 2024-09-11T01:46:16.440 INFO:tasks.rados.rados.0.smithi071.stdout:2603: done (2 left) 2024-09-11T01:46:16.441 INFO:tasks.rados.rados.0.smithi071.stdout:2606: read oid 33 snap -1 2024-09-11T01:46:16.441 INFO:tasks.rados.rados.0.smithi071.stdout:2606: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:46:16.441 INFO:tasks.rados.rados.0.smithi071.stdout:2607: read oid 29 snap -1 2024-09-11T01:46:16.441 INFO:tasks.rados.rados.0.smithi071.stdout:2607: expect (ObjNum 801 snap 243 seq_num 801) 2024-09-11T01:46:16.441 INFO:tasks.rados.rados.0.smithi071.stdout:2608: read oid 12 snap 270 2024-09-11T01:46:16.441 INFO:tasks.rados.rados.0.smithi071.stdout:2608: expect deleted 2024-09-11T01:46:16.441 INFO:tasks.rados.rados.0.smithi071.stdout:2609: delete oid 42 current snap is 293 2024-09-11T01:46:16.441 INFO:tasks.rados.rados.0.smithi071.stdout:2604: finishing write tid 4 to smithi07134777-17 2024-09-11T01:46:16.441 INFO:tasks.rados.rados.0.smithi071.stdout:2604: finishing write tid 5 to smithi07134777-17 2024-09-11T01:46:16.441 INFO:tasks.rados.rados.0.smithi071.stdout:2604: oid 17 updating version 0 to 1609 2024-09-11T01:46:16.441 INFO:tasks.rados.rados.0.smithi071.stdout:2604: oid 17 updating version 1609 to 1612 2024-09-11T01:46:16.441 INFO:tasks.rados.rados.0.smithi071.stdout:2604: oid 17 version 1612 is already newer than 1610 2024-09-11T01:46:16.441 INFO:tasks.rados.rados.0.smithi071.stdout:2604: oid 17 version 1612 is already newer than 1611 2024-09-11T01:46:16.441 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 1612 (ObjNum 931 snap 293 seq_num 931) dirty exists 2024-09-11T01:46:16.442 INFO:tasks.rados.rados.0.smithi071.stdout:2604: left oid 17 (ObjNum 931 snap 293 seq_num 931) 2024-09-11T01:46:16.442 INFO:tasks.rados.rados.0.smithi071.stdout:2605: finishing write tid 1 to smithi07134777-45 2024-09-11T01:46:16.442 INFO:tasks.rados.rados.0.smithi071.stdout:2605: finishing write tid 2 to smithi07134777-45 2024-09-11T01:46:16.446 INFO:tasks.rados.rados.0.smithi071.stdout:2607: expect (ObjNum 801 snap 243 seq_num 801) 2024-09-11T01:46:16.446 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:46:16.501 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:46:16.612 INFO:tasks.rados.rados.0.smithi071.stdout:2606: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:46:16.719 INFO:tasks.rados.rados.0.smithi071.stdout:2605: finishing write tid 3 to smithi07134777-45 2024-09-11T01:46:16.719 INFO:tasks.rados.rados.0.smithi071.stdout:2605: finishing write tid 4 to smithi07134777-45 2024-09-11T01:46:16.719 INFO:tasks.rados.rados.0.smithi071.stdout:2605: finishing write tid 5 to smithi07134777-45 2024-09-11T01:46:16.719 INFO:tasks.rados.rados.0.smithi071.stdout:2605: oid 45 updating version 0 to 1291 2024-09-11T01:46:16.719 INFO:tasks.rados.rados.0.smithi071.stdout:2605: oid 45 updating version 1291 to 1293 2024-09-11T01:46:16.719 INFO:tasks.rados.rados.0.smithi071.stdout:2605: oid 45 updating version 1293 to 1294 2024-09-11T01:46:16.719 INFO:tasks.rados.rados.0.smithi071.stdout:2605: oid 45 version 1294 is already newer than 1292 2024-09-11T01:46:16.719 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 1294 (ObjNum 932 snap 293 seq_num 932) dirty exists 2024-09-11T01:46:16.719 INFO:tasks.rados.rados.0.smithi071.stdout:2605: left oid 45 (ObjNum 932 snap 293 seq_num 932) 2024-09-11T01:46:16.719 INFO:tasks.rados.rados.0.smithi071.stdout:2604: done (5 left) 2024-09-11T01:46:16.719 INFO:tasks.rados.rados.0.smithi071.stdout:2605: done (4 left) 2024-09-11T01:46:16.719 INFO:tasks.rados.rados.0.smithi071.stdout:2606: done (3 left) 2024-09-11T01:46:16.719 INFO:tasks.rados.rados.0.smithi071.stdout:2607: done (2 left) 2024-09-11T01:46:16.720 INFO:tasks.rados.rados.0.smithi071.stdout:2608: done (1 left) 2024-09-11T01:46:16.720 INFO:tasks.rados.rados.0.smithi071.stdout:2609: done (0 left) 2024-09-11T01:46:16.720 INFO:tasks.rados.rados.0.smithi071.stdout:2610: read oid 22 snap -1 2024-09-11T01:46:16.720 INFO:tasks.rados.rados.0.smithi071.stdout:2610: expect (ObjNum 801 snap 243 seq_num 801) 2024-09-11T01:46:16.720 INFO:tasks.rados.rados.0.smithi071.stdout:2611: write oid 10 current snap is 293 2024-09-11T01:46:16.720 INFO:tasks.rados.rados.0.smithi071.stdout:2611: seq_num 934 ranges {497738=477632,1772417=775656,3316134=381604} 2024-09-11T01:46:16.723 INFO:tasks.rados.rados.0.smithi071.stdout:2611: writing smithi07134777-10 from 497738 to 975370 tid 1 2024-09-11T01:46:16.732 INFO:tasks.rados.rados.0.smithi071.stdout:2611: writing smithi07134777-10 from 1772417 to 2548073 tid 2 2024-09-11T01:46:16.739 INFO:tasks.rados.rados.0.smithi071.stdout:2611: writing smithi07134777-10 from 3316134 to 3697738 tid 3 2024-09-11T01:46:16.739 INFO:tasks.rados.rados.0.smithi071.stdout:2612: copy_from oid 48 from oid 26 current snap is 293 2024-09-11T01:46:16.739 INFO:tasks.rados.rados.0.smithi071.stdout:2610: expect (ObjNum 801 snap 243 seq_num 801) 2024-09-11T01:46:16.893 INFO:tasks.rados.rados.0.smithi071.stdout:2611: finishing write tid 1 to smithi07134777-10 2024-09-11T01:46:16.893 INFO:tasks.rados.rados.0.smithi071.stdout:2611: finishing write tid 2 to smithi07134777-10 2024-09-11T01:46:16.893 INFO:tasks.rados.rados.0.smithi071.stdout:2611: finishing write tid 3 to smithi07134777-10 2024-09-11T01:46:16.894 INFO:tasks.rados.rados.0.smithi071.stdout:2611: finishing write tid 4 to smithi07134777-10 2024-09-11T01:46:16.894 INFO:tasks.rados.rados.0.smithi071.stdout:2611: finishing write tid 5 to smithi07134777-10 2024-09-11T01:46:16.894 INFO:tasks.rados.rados.0.smithi071.stdout:2611: oid 10 updating version 0 to 435 2024-09-11T01:46:16.894 INFO:tasks.rados.rados.0.smithi071.stdout:2611: oid 10 version 435 is already newer than 434 2024-09-11T01:46:16.894 INFO:tasks.rados.rados.0.smithi071.stdout:2611: oid 10 updating version 435 to 436 2024-09-11T01:46:16.894 INFO:tasks.rados.rados.0.smithi071.stdout:2611: oid 10 version 436 is already newer than 433 2024-09-11T01:46:16.894 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 436 (ObjNum 934 snap 293 seq_num 934) dirty exists 2024-09-11T01:46:16.894 INFO:tasks.rados.rados.0.smithi071.stdout:2611: left oid 10 (ObjNum 934 snap 293 seq_num 934) 2024-09-11T01:46:16.894 INFO:tasks.rados.rados.0.smithi071.stdout:2612: finishing copy_from racing read to smithi07134777-48 2024-09-11T01:46:16.894 INFO:tasks.rados.rados.0.smithi071.stdout:2612: finishing copy_from to smithi07134777-48 2024-09-11T01:46:16.894 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 1067 (ObjNum 688 snap 204 seq_num 688) dirty exists 2024-09-11T01:46:16.894 INFO:tasks.rados.rados.0.smithi071.stdout:2610: done (2 left) 2024-09-11T01:46:16.895 INFO:tasks.rados.rados.0.smithi071.stdout:2611: done (1 left) 2024-09-11T01:46:16.895 INFO:tasks.rados.rados.0.smithi071.stdout:2612: done (0 left) 2024-09-11T01:46:16.895 INFO:tasks.rados.rados.0.smithi071.stdout:2613: write oid 49 current snap is 293 2024-09-11T01:46:16.895 INFO:tasks.rados.rados.0.smithi071.stdout:2613: seq_num 935 ranges {514545=552881,1830899=519314,2943557=475332} 2024-09-11T01:46:16.898 INFO:tasks.rados.rados.0.smithi071.stdout:2613: writing smithi07134777-49 from 514545 to 1067426 tid 1 2024-09-11T01:46:16.905 INFO:tasks.rados.rados.0.smithi071.stdout:2613: writing smithi07134777-49 from 1830899 to 2350213 tid 2 2024-09-11T01:46:16.910 INFO:tasks.rados.rados.0.smithi071.stdout:2613: writing smithi07134777-49 from 2943557 to 3418889 tid 3 2024-09-11T01:46:16.911 INFO:tasks.rados.rados.0.smithi071.stdout:2614: snap_remove snap 278 2024-09-11T01:46:17.081 INFO:tasks.rados.rados.0.smithi071.stdout:2614: done (1 left) 2024-09-11T01:46:17.081 INFO:tasks.rados.rados.0.smithi071.stdout:2615: snap_remove snap 219 2024-09-11T01:46:18.081 INFO:tasks.rados.rados.0.smithi071.stdout:2615: done (1 left) 2024-09-11T01:46:18.081 INFO:tasks.rados.rados.0.smithi071.stdout:2616: write oid 13 current snap is 293 2024-09-11T01:46:18.082 INFO:tasks.rados.rados.0.smithi071.stdout:2613: finishing write tid 1 to smithi07134777-49 2024-09-11T01:46:18.082 INFO:tasks.rados.rados.0.smithi071.stdout:2616: seq_num 936 ranges {531352=744483,2005734=779325,3487333=244019} 2024-09-11T01:46:18.088 INFO:tasks.rados.rados.0.smithi071.stdout:2616: writing smithi07134777-13 from 531352 to 1275835 tid 1 2024-09-11T01:46:18.097 INFO:tasks.rados.rados.0.smithi071.stdout:2616: writing smithi07134777-13 from 2005734 to 2785059 tid 2 2024-09-11T01:46:18.103 INFO:tasks.rados.rados.0.smithi071.stdout:2616: writing smithi07134777-13 from 3487333 to 3731352 tid 3 2024-09-11T01:46:18.104 INFO:tasks.rados.rados.0.smithi071.stdout:2617: copy_from oid 22 from oid 32 current snap is 293 2024-09-11T01:46:18.104 INFO:tasks.rados.rados.0.smithi071.stdout:2618: rollback oid 38 current snap is 293 2024-09-11T01:46:18.104 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 38 to 276 2024-09-11T01:46:18.104 INFO:tasks.rados.rados.0.smithi071.stdout:2619: write (excl) oid 8 current snap is 293 2024-09-11T01:46:18.104 INFO:tasks.rados.rados.0.smithi071.stdout:2619: seq_num 937 ranges {548159=419732,1664216=522983,2714756=614229} 2024-09-11T01:46:18.112 INFO:tasks.rados.rados.0.smithi071.stdout:2619: writing smithi07134777-8 from 548159 to 967891 tid 1 2024-09-11T01:46:18.117 INFO:tasks.rados.rados.0.smithi071.stdout:2619: writing smithi07134777-8 from 1664216 to 2187199 tid 2 2024-09-11T01:46:18.124 INFO:tasks.rados.rados.0.smithi071.stdout:2619: writing smithi07134777-8 from 2714756 to 3328985 tid 3 2024-09-11T01:46:18.124 INFO:tasks.rados.rados.0.smithi071.stdout:2620: snap_create 2024-09-11T01:46:18.124 INFO:tasks.rados.rados.0.smithi071.stdout:2613: finishing write tid 2 to smithi07134777-49 2024-09-11T01:46:18.124 INFO:tasks.rados.rados.0.smithi071.stdout:2613: finishing write tid 3 to smithi07134777-49 2024-09-11T01:46:18.124 INFO:tasks.rados.rados.0.smithi071.stdout:2613: finishing write tid 4 to smithi07134777-49 2024-09-11T01:46:18.124 INFO:tasks.rados.rados.0.smithi071.stdout:2613: finishing write tid 5 to smithi07134777-49 2024-09-11T01:46:18.124 INFO:tasks.rados.rados.0.smithi071.stdout:2613: oid 49 updating version 0 to 380 2024-09-11T01:46:18.124 INFO:tasks.rados.rados.0.smithi071.stdout:2613: oid 49 version 380 is already newer than 379 2024-09-11T01:46:18.125 INFO:tasks.rados.rados.0.smithi071.stdout:2613: oid 49 version 380 is already newer than 377 2024-09-11T01:46:18.125 INFO:tasks.rados.rados.0.smithi071.stdout:2613: oid 49 version 380 is already newer than 378 2024-09-11T01:46:18.125 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 380 (ObjNum 935 snap 293 seq_num 935) dirty exists 2024-09-11T01:46:18.125 INFO:tasks.rados.rados.0.smithi071.stdout:2613: left oid 49 (ObjNum 935 snap 293 seq_num 935) 2024-09-11T01:46:18.125 INFO:tasks.rados.rados.0.smithi071.stdout:2616: finishing write tid 1 to smithi07134777-13 2024-09-11T01:46:18.125 INFO:tasks.rados.rados.0.smithi071.stdout:2616: finishing write tid 2 to smithi07134777-13 2024-09-11T01:46:18.125 INFO:tasks.rados.rados.0.smithi071.stdout:2616: finishing write tid 3 to smithi07134777-13 2024-09-11T01:46:18.125 INFO:tasks.rados.rados.0.smithi071.stdout:2616: finishing write tid 4 to smithi07134777-13 2024-09-11T01:46:18.125 INFO:tasks.rados.rados.0.smithi071.stdout:2616: finishing write tid 5 to smithi07134777-13 2024-09-11T01:46:18.125 INFO:tasks.rados.rados.0.smithi071.stdout:2616: oid 13 updating version 0 to 1614 2024-09-11T01:46:18.125 INFO:tasks.rados.rados.0.smithi071.stdout:2616: oid 13 version 1614 is already newer than 1613 2024-09-11T01:46:18.125 INFO:tasks.rados.rados.0.smithi071.stdout:2616: oid 13 updating version 1614 to 1615 2024-09-11T01:46:18.125 INFO:tasks.rados.rados.0.smithi071.stdout:2616: oid 13 updating version 1615 to 1616 2024-09-11T01:46:18.125 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 1616 (ObjNum 936 snap 293 seq_num 936) dirty exists 2024-09-11T01:46:18.126 INFO:tasks.rados.rados.0.smithi071.stdout:2616: left oid 13 (ObjNum 936 snap 293 seq_num 936) 2024-09-11T01:46:18.126 INFO:tasks.rados.rados.0.smithi071.stdout:2619: finishing write tid 1 to smithi07134777-8 2024-09-11T01:46:18.126 INFO:tasks.rados.rados.0.smithi071.stdout:2617: finishing copy_from racing read to smithi07134777-22 2024-09-11T01:46:18.126 INFO:tasks.rados.rados.0.smithi071.stdout:2618: finishing rollback tid 1 to smithi07134777-38 2024-09-11T01:46:18.126 INFO:tasks.rados.rados.0.smithi071.stdout:2618: finishing rollback tid 2 to smithi07134777-38 2024-09-11T01:46:18.126 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 1720 (ObjNum 850 snap 256 seq_num 850) dirty exists 2024-09-11T01:46:18.126 INFO:tasks.rados.rados.0.smithi071.stdout:2617: finishing copy_from to smithi07134777-22 2024-09-11T01:46:18.126 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 1007 (ObjNum 745 snap 227 seq_num 745) dirty exists 2024-09-11T01:46:18.126 INFO:tasks.rados.rados.0.smithi071.stdout:2619: finishing write tid 2 to smithi07134777-8 2024-09-11T01:46:18.127 INFO:tasks.rados.rados.0.smithi071.stdout:2619: finishing write tid 3 to smithi07134777-8 2024-09-11T01:46:18.127 INFO:tasks.rados.rados.0.smithi071.stdout:2619: finishing write tid 4 to smithi07134777-8 2024-09-11T01:46:18.127 INFO:tasks.rados.rados.0.smithi071.stdout:2619: finishing write tid 5 to smithi07134777-8 2024-09-11T01:46:18.127 INFO:tasks.rados.rados.0.smithi071.stdout:2619: oid 8 updating version 0 to 1069 2024-09-11T01:46:18.128 INFO:tasks.rados.rados.0.smithi071.stdout:2619: oid 8 version 1069 is already newer than 1068 2024-09-11T01:46:18.128 INFO:tasks.rados.rados.0.smithi071.stdout:2619: oid 8 version 1069 is already newer than 1067 2024-09-11T01:46:18.128 INFO:tasks.rados.rados.0.smithi071.stdout:2619: oid 8 updating version 1069 to 1070 2024-09-11T01:46:18.128 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 1070 (ObjNum 937 snap 293 seq_num 937) dirty exists 2024-09-11T01:46:18.128 INFO:tasks.rados.rados.0.smithi071.stdout:2619: left oid 8 (ObjNum 937 snap 293 seq_num 937) 2024-09-11T01:46:19.083 INFO:tasks.rados.rados.0.smithi071.stdout:2613: done (5 left) 2024-09-11T01:46:19.083 INFO:tasks.rados.rados.0.smithi071.stdout:2616: done (4 left) 2024-09-11T01:46:19.083 INFO:tasks.rados.rados.0.smithi071.stdout:2617: done (3 left) 2024-09-11T01:46:19.083 INFO:tasks.rados.rados.0.smithi071.stdout:2618: done (2 left) 2024-09-11T01:46:19.083 INFO:tasks.rados.rados.0.smithi071.stdout:2619: done (1 left) 2024-09-11T01:46:19.083 INFO:tasks.rados.rados.0.smithi071.stdout:2620: done (0 left) 2024-09-11T01:46:19.083 INFO:tasks.rados.rados.0.smithi071.stdout:2621: write oid 21 current snap is 294 2024-09-11T01:46:19.083 INFO:tasks.rados.rados.0.smithi071.stdout:2621: seq_num 938 ranges {564966=494981,1606345=782994,3025826=425501} 2024-09-11T01:46:19.088 INFO:tasks.rados.rados.0.smithi071.stdout:2621: writing smithi07134777-21 from 564966 to 1059947 tid 1 2024-09-11T01:46:19.096 INFO:tasks.rados.rados.0.smithi071.stdout:2621: writing smithi07134777-21 from 1606345 to 2389339 tid 2 2024-09-11T01:46:19.103 INFO:tasks.rados.rados.0.smithi071.stdout:2621: writing smithi07134777-21 from 3025826 to 3451327 tid 3 2024-09-11T01:46:19.103 INFO:tasks.rados.rados.0.smithi071.stdout:2622: rollback oid 15 current snap is 294 2024-09-11T01:46:19.103 INFO:tasks.rados.rados.0.smithi071.stdout:2621: finishing write tid 1 to smithi07134777-21 2024-09-11T01:46:19.103 INFO:tasks.rados.rados.0.smithi071.stdout:2621: finishing write tid 2 to smithi07134777-21 2024-09-11T01:46:19.103 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 15 to 274 2024-09-11T01:46:19.103 INFO:tasks.rados.rados.0.smithi071.stdout:2623: snap_remove snap 270 2024-09-11T01:46:20.094 INFO:tasks.rados.rados.0.smithi071.stdout:2623: done (2 left) 2024-09-11T01:46:20.094 INFO:tasks.rados.rados.0.smithi071.stdout:2624: write (excl) oid 25 current snap is 294 2024-09-11T01:46:20.094 INFO:tasks.rados.rados.0.smithi071.stdout:2621: finishing write tid 3 to smithi07134777-21 2024-09-11T01:46:20.094 INFO:tasks.rados.rados.0.smithi071.stdout:2624: seq_num 939 ranges {581773=570230,1664827=526652,2653249=636773} 2024-09-11T01:46:20.100 INFO:tasks.rados.rados.0.smithi071.stdout:2624: writing smithi07134777-25 from 581773 to 1152003 tid 1 2024-09-11T01:46:20.107 INFO:tasks.rados.rados.0.smithi071.stdout:2624: writing smithi07134777-25 from 1664827 to 2191479 tid 2 2024-09-11T01:46:20.114 INFO:tasks.rados.rados.0.smithi071.stdout:2624: writing smithi07134777-25 from 2653249 to 3290022 tid 3 2024-09-11T01:46:20.114 INFO:tasks.rados.rados.0.smithi071.stdout:2625: snap_create 2024-09-11T01:46:20.114 INFO:tasks.rados.rados.0.smithi071.stdout:2621: finishing write tid 4 to smithi07134777-21 2024-09-11T01:46:20.114 INFO:tasks.rados.rados.0.smithi071.stdout:2621: finishing write tid 5 to smithi07134777-21 2024-09-11T01:46:20.114 INFO:tasks.rados.rados.0.smithi071.stdout:2621: oid 21 updating version 0 to 1312 2024-09-11T01:46:20.115 INFO:tasks.rados.rados.0.smithi071.stdout:2621: oid 21 updating version 1312 to 1314 2024-09-11T01:46:20.115 INFO:tasks.rados.rados.0.smithi071.stdout:2621: oid 21 version 1314 is already newer than 1313 2024-09-11T01:46:20.115 INFO:tasks.rados.rados.0.smithi071.stdout:2621: oid 21 version 1314 is already newer than 1311 2024-09-11T01:46:20.115 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 1314 (ObjNum 938 snap 294 seq_num 938) dirty exists 2024-09-11T01:46:20.115 INFO:tasks.rados.rados.0.smithi071.stdout:2621: left oid 21 (ObjNum 938 snap 294 seq_num 938) 2024-09-11T01:46:20.115 INFO:tasks.rados.rados.0.smithi071.stdout:2622: finishing rollback tid 0 to smithi07134777-15 2024-09-11T01:46:20.115 INFO:tasks.rados.rados.0.smithi071.stdout:2622: finishing rollback tid 1 to smithi07134777-15 2024-09-11T01:46:20.115 INFO:tasks.rados.rados.0.smithi071.stdout:2622: finishing rollback tid 2 to smithi07134777-15 2024-09-11T01:46:20.115 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 1728 (ObjNum 873 snap 267 seq_num 873) dirty exists 2024-09-11T01:46:20.115 INFO:tasks.rados.rados.0.smithi071.stdout:2624: finishing write tid 1 to smithi07134777-25 2024-09-11T01:46:20.115 INFO:tasks.rados.rados.0.smithi071.stdout:2624: finishing write tid 2 to smithi07134777-25 2024-09-11T01:46:20.124 INFO:tasks.rados.rados.0.smithi071.stdout:2624: finishing write tid 3 to smithi07134777-25 2024-09-11T01:46:20.124 INFO:tasks.rados.rados.0.smithi071.stdout:2624: finishing write tid 4 to smithi07134777-25 2024-09-11T01:46:20.124 INFO:tasks.rados.rados.0.smithi071.stdout:2624: finishing write tid 5 to smithi07134777-25 2024-09-11T01:46:20.124 INFO:tasks.rados.rados.0.smithi071.stdout:2624: oid 25 updating version 0 to 1005 2024-09-11T01:46:20.124 INFO:tasks.rados.rados.0.smithi071.stdout:2624: oid 25 updating version 1005 to 1006 2024-09-11T01:46:20.124 INFO:tasks.rados.rados.0.smithi071.stdout:2624: oid 25 updating version 1006 to 1007 2024-09-11T01:46:20.124 INFO:tasks.rados.rados.0.smithi071.stdout:2624: oid 25 version 1007 is already newer than 1004 2024-09-11T01:46:20.124 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 1007 (ObjNum 939 snap 294 seq_num 939) dirty exists 2024-09-11T01:46:20.124 INFO:tasks.rados.rados.0.smithi071.stdout:2624: left oid 25 (ObjNum 939 snap 294 seq_num 939) 2024-09-11T01:46:21.096 INFO:tasks.rados.rados.0.smithi071.stdout:2621: done (3 left) 2024-09-11T01:46:21.096 INFO:tasks.rados.rados.0.smithi071.stdout:2622: done (2 left) 2024-09-11T01:46:21.096 INFO:tasks.rados.rados.0.smithi071.stdout:2624: done (1 left) 2024-09-11T01:46:21.096 INFO:tasks.rados.rados.0.smithi071.stdout:2625: done (0 left) 2024-09-11T01:46:21.096 INFO:tasks.rados.rados.0.smithi071.stdout:2626: snap_remove snap 264 2024-09-11T01:46:22.082 INFO:tasks.rados.rados.0.smithi071.stdout:2626: done (0 left) 2024-09-11T01:46:22.083 INFO:tasks.rados.rados.0.smithi071.stdout:2627: snap_remove snap 293 2024-09-11T01:46:22.668 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-11T01:46:22.668 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T01:46:22.668 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T01:46:22.671 INFO:tasks.ceph.mon.a:Started 2024-09-11T01:46:22.671 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:46:22.672 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:46:22.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.783+0000 7f446c391640 10 monclient: get_monmap_and_config 2024-09-11T01:46:22.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.783+0000 7f446c391640 10 monclient: build_initial_monmap 2024-09-11T01:46:22.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.783+0000 7f446c391640 10 monclient: monmap: 2024-09-11T01:46:22.786 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:22.786 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:22.786 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:46:22.785591+0000 2024-09-11T01:46:22.786 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:46:22.785591+0000 2024-09-11T01:46:22.787 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:22.787 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:22.787 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:46:22.787 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:46:22.787 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:46:22.787 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:46:22.787 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:46:22.787 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:22.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.784+0000 7f446c391640 1 Processor -- start 2024-09-11T01:46:22.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.784+0000 7f446c391640 1 -- start start 2024-09-11T01:46:22.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.784+0000 7f446c391640 10 monclient: init 2024-09-11T01:46:22.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446c391640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:22.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446c391640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-11T01:46:22.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446c391640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f44640a2e90 addr v1:172.21.15.71:6789/0 2024-09-11T01:46:22.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446c391640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44640a3270 0x7f44640a3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:22.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446c391640 10 monclient(hunting): picked mon.noname-b con 0x7f44640a3270 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:22.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446c391640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f446413d490 0x7f446413f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:22.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446c391640 10 monclient(hunting): picked mon.noname-c con 0x7f446413d490 addr v2:172.21.15.71:3300/0 2024-09-11T01:46:22.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446c391640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44640a46c0 con 0x7f44640a2e90 2024-09-11T01:46:22.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446c391640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:22.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446c391640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f44640a4900 con 0x7f446413d490 2024-09-11T01:46:22.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446c391640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:22.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446c391640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f446406ee50 con 0x7f44640a3270 2024-09-11T01:46:22.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446c391640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:22.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446c391640 10 monclient(hunting): authenticate will time out at 1789.228280s 2024-09-11T01:46:22.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446992c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44640a3270 0x7f44640a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:22.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446992c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44640a3270 0x7f44640a3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55078/0 (socket says 172.21.15.12:55078) 2024-09-11T01:46:22.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446992c640 1 -- 172.21.15.12:0/970014695 learned_addr learned my addr 172.21.15.12:0/970014695 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:22.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446992c640 10 monclient(hunting): get_auth_request con 0x7f44640a3270 auth_method 0 2024-09-11T01:46:22.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446992c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:22.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446992c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:22.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446992c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:22.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446992c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:22.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446992c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:22.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446a92e640 1 --2- 172.21.15.12:0/970014695 >> v2:172.21.15.71:3300/0 conn(0x7f446413d490 0x7f446413f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:22.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446992c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:22.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446a92e640 10 monclient(hunting): get_auth_request con 0x7f446413d490 auth_method 0 2024-09-11T01:46:22.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446a92e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:22.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446a92e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:22.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446a92e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:22.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446992c640 10 monclient(hunting): handle_auth_done global_id 184181 payload 274 2024-09-11T01:46:22.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446992c640 1 -- 172.21.15.12:0/970014695 >> v2:172.21.15.71:3300/0 conn(0x7f446413d490 msgr2=0x7f446413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:46:22.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446992c640 1 --2- 172.21.15.12:0/970014695 >> v2:172.21.15.71:3300/0 conn(0x7f446413d490 0x7f446413f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:22.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446992c640 1 -- 172.21.15.12:0/970014695 >> v1:172.21.15.71:6789/0 conn(0x7f44640a2e90 legacy=0x7f446406dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:46:22.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446992c640 10 monclient: _finish_hunting 0 2024-09-11T01:46:22.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446992c640 1 monclient: found mon.noname-b 2024-09-11T01:46:22.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446992c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:22.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446992c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:46:22.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446992c640 1 -- 172.21.15.12:0/970014695 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f446406eb70 con 0x7f44640a3270 2024-09-11T01:46:22.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.785+0000 7f446a92e640 1 --2- 172.21.15.12:0/970014695 >> v2:172.21.15.71:3300/0 conn(0x7f446413d490 0x7f446413f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:46:22.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.788+0000 7f446992c640 1 -- 172.21.15.12:0/970014695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44640a3270 msgr2=0x7f44640a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:22.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.788+0000 7f446992c640 1 --2- 172.21.15.12:0/970014695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44640a3270 0x7f44640a3680 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f44640a4a80 tx=0x7f445402a050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee789409229f2c59 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:22.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.788+0000 7f446912b640 1 -- 172.21.15.12:0/970014695 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4454037070 con 0x7f44640a3270 2024-09-11T01:46:22.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.788+0000 7f446912b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:22.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.788+0000 7f446912b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:46:22.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.788+0000 7f446912b640 10 monclient: dump: 2024-09-11T01:46:22.798 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:22.798 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:22.798 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:22.798 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:22.798 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:22.798 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:22.798 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:22.798 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:22.798 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:22.798 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:22.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.788+0000 7f446912b640 10 monclient: _finish_auth 0 2024-09-11T01:46:22.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.788+0000 7f446912b640 10 monclient: _check_auth_tickets 2024-09-11T01:46:22.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.788+0000 7f446912b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:22.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.788+0000 7f446c391640 5 monclient: authenticate success, global_id 184181 2024-09-11T01:46:22.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.788+0000 7f446c391640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:46:22.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.788+0000 7f446912b640 1 -- 172.21.15.12:0/970014695 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4454032440 con 0x7f44640a3270 2024-09-11T01:46:22.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.788+0000 7f446912b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:22.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.788+0000 7f446c391640 10 monclient: get_monmap_and_config success 2024-09-11T01:46:22.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.788+0000 7f446c391640 4 set_mon_vals no callback set 2024-09-11T01:46:22.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.788+0000 7f446912b640 1 -- 172.21.15.12:0/970014695 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4454032810 con 0x7f44640a3270 2024-09-11T01:46:22.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.788+0000 7f446912b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:22.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.788+0000 7f446912b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:22.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.788+0000 7f446912b640 10 monclient: dump: 2024-09-11T01:46:22.800 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:22.800 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:22.800 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:22.800 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:22.800 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:22.800 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:22.800 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:22.800 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:22.800 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:22.800 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:22.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.789+0000 7f446c391640 10 monclient: shutdown 2024-09-11T01:46:22.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.789+0000 7f446c391640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:22.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.789+0000 7f446c391640 1 -- 172.21.15.12:0/970014695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44640a3270 msgr2=0x7f44640a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:22.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.789+0000 7f446c391640 1 --2- 172.21.15.12:0/970014695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44640a3270 0x7f44640a3680 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f44640a4a80 tx=0x7f445402a050 comp rx=0 tx=0).stop 2024-09-11T01:46:22.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.789+0000 7f446c391640 1 -- 172.21.15.12:0/970014695 shutdown_connections 2024-09-11T01:46:22.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.789+0000 7f446c391640 1 --2- 172.21.15.12:0/970014695 >> v2:172.21.15.71:3300/0 conn(0x7f446413d490 0x7f446413f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:22.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.789+0000 7f446c391640 1 --2- 172.21.15.12:0/970014695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44640a3270 0x7f44640a3680 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:22.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.789+0000 7f446c391640 1 -- 172.21.15.12:0/970014695 >> 172.21.15.12:0/970014695 conn(0x7f44640a8490 msgr2=0x7f44640a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:22.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.789+0000 7f446c391640 1 -- 172.21.15.12:0/970014695 shutdown_connections 2024-09-11T01:46:22.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.789+0000 7f446c391640 1 -- 172.21.15.12:0/970014695 wait complete. 2024-09-11T01:46:22.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.790+0000 7f446c391640 10 monclient: build_initial_monmap 2024-09-11T01:46:22.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.790+0000 7f446c391640 10 monclient: monmap: 2024-09-11T01:46:22.801 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:22.801 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:46:22.802 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:46:22.802 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:46:22.802 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:22.802 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:22.802 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:46:22.802 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:46:22.802 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:46:22.802 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:22.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.790+0000 7f446c391640 1 Processor -- start 2024-09-11T01:46:22.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.790+0000 7f446c391640 1 -- start start 2024-09-11T01:46:22.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.790+0000 7f446c391640 10 monclient: init 2024-09-11T01:46:22.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.790+0000 7f446c391640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:22.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.790+0000 7f446c391640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:46:22.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.790+0000 7f446c391640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44640a2e90 0x7f4464137b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:22.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.790+0000 7f446c391640 10 monclient(hunting): picked mon.noname-c con 0x7f44640a2e90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:22.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.790+0000 7f446c391640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44640a3270 0x7f4464138050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:22.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.790+0000 7f446c391640 10 monclient(hunting): picked mon.noname-b con 0x7f44640a3270 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:46:22.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.790+0000 7f446c391640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f446413d490 0x7f4464138590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:22.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.790+0000 7f446c391640 10 monclient(hunting): picked mon.noname-a con 0x7f446413d490 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:46:22.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.790+0000 7f446c391640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:22.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.790+0000 7f446c391640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f446406fd20 con 0x7f446413d490 2024-09-11T01:46:22.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.790+0000 7f446c391640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:22.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.790+0000 7f446c391640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f446406fba0 con 0x7f44640a3270 2024-09-11T01:46:22.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.790+0000 7f446c391640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:22.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.790+0000 7f446c391640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f446413fdc0 con 0x7f44640a2e90 2024-09-11T01:46:22.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.790+0000 7f446c391640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:22.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.790+0000 7f446c391640 10 monclient(hunting): authenticate will time out at 1789.233856s 2024-09-11T01:46:22.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.790+0000 7f446a12d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44640a2e90 0x7f4464137b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:22.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a12d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44640a2e90 0x7f4464137b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55084/0 (socket says 172.21.15.12:55084) 2024-09-11T01:46:22.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a12d640 1 -- 172.21.15.12:0/1226035730 learned_addr learned my addr 172.21.15.12:0/1226035730 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:22.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a12d640 10 monclient(hunting): get_auth_request con 0x7f44640a2e90 auth_method 0 2024-09-11T01:46:22.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a12d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:22.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a12d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:22.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a12d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:22.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a92e640 1 --2- 172.21.15.12:0/1226035730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f446413d490 0x7f4464138590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:22.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a92e640 10 monclient(hunting): get_auth_request con 0x7f446413d490 auth_method 0 2024-09-11T01:46:22.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a92e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:22.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a92e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:22.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a92e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:22.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a12d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:22.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a12d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:22.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a12d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:22.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a92e640 1 -- 172.21.15.12:0/1226035730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f446413d490 msgr2=0x7f4464138590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:46:22.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a92e640 1 -- 172.21.15.12:0/1226035730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f446413d490 msgr2=0x7f4464138590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:46:22.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a92e640 1 --2- 172.21.15.12:0/1226035730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f446413d490 0x7f4464138590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:46:22.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446992c640 1 --2- 172.21.15.12:0/1226035730 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44640a3270 0x7f4464138050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:22.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a92e640 1 --2- 172.21.15.12:0/1226035730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f446413d490 0x7f4464138590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:46:22.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446992c640 10 monclient(hunting): get_auth_request con 0x7f44640a3270 auth_method 0 2024-09-11T01:46:22.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446992c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:22.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446992c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:22.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446992c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:22.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a12d640 10 monclient(hunting): handle_auth_done global_id 184184 payload 824 2024-09-11T01:46:22.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a12d640 1 -- 172.21.15.12:0/1226035730 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44640a3270 msgr2=0x7f4464138050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:22.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a12d640 1 --2- 172.21.15.12:0/1226035730 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44640a3270 0x7f4464138050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:22.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a12d640 1 -- 172.21.15.12:0/1226035730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f446413d490 msgr2=0x7f4464138590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:46:22.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a12d640 1 --2- 172.21.15.12:0/1226035730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f446413d490 0x7f4464138590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:22.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a12d640 10 monclient: _finish_hunting 0 2024-09-11T01:46:22.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a12d640 1 monclient: found mon.noname-c 2024-09-11T01:46:22.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a12d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:22.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a12d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:46:22.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.791+0000 7f446a12d640 1 -- 172.21.15.12:0/1226035730 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f446409efb0 con 0x7f44640a2e90 2024-09-11T01:46:22.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.793+0000 7f446a12d640 1 -- 172.21.15.12:0/1226035730 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44640a2e90 msgr2=0x7f4464137b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:22.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.793+0000 7f446a12d640 1 --2- 172.21.15.12:0/1226035730 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44640a2e90 0x7f4464137b10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f446000c430 tx=0x7f446000c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:22.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.793+0000 7f445affd640 1 -- 172.21.15.12:0/1226035730 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44600119a0 con 0x7f44640a2e90 2024-09-11T01:46:22.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.793+0000 7f445affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:22.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.793+0000 7f445affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:46:22.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.793+0000 7f445affd640 10 monclient: dump: 2024-09-11T01:46:22.807 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:22.807 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:22.807 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:22.807 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:22.807 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:22.807 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:22.807 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:22.807 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:22.807 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:22.807 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:22.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.793+0000 7f445affd640 10 monclient: _finish_auth 0 2024-09-11T01:46:22.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.793+0000 7f445affd640 10 monclient: _check_auth_tickets 2024-09-11T01:46:22.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.793+0000 7f445affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:22.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.793+0000 7f445affd640 1 -- 172.21.15.12:0/1226035730 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4460017070 con 0x7f44640a2e90 2024-09-11T01:46:22.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.793+0000 7f446c391640 5 monclient: authenticate success, global_id 184184 2024-09-11T01:46:22.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.793+0000 7f446c391640 10 monclient: _renew_subs 2024-09-11T01:46:22.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.793+0000 7f446c391640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:22.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.793+0000 7f446c391640 1 -- 172.21.15.12:0/1226035730 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44640a2720 con 0x7f44640a2e90 2024-09-11T01:46:22.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.793+0000 7f445affd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:22.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.793+0000 7f445affd640 1 -- 172.21.15.12:0/1226035730 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4460011d80 con 0x7f44640a2e90 2024-09-11T01:46:22.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.793+0000 7f445affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:22.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.793+0000 7f445affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:22.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.793+0000 7f445affd640 10 monclient: dump: 2024-09-11T01:46:22.808 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:22.808 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:22.808 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:22.809 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:22.809 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:22.809 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:22.809 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:22.809 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:22.809 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:22.809 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:22.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.793+0000 7f445bfff640 4 set_mon_vals no callback set 2024-09-11T01:46:22.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.793+0000 7f446c391640 10 monclient: _renew_subs 2024-09-11T01:46:22.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.793+0000 7f446c391640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:22.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.793+0000 7f446c391640 1 -- 172.21.15.12:0/1226035730 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f446409e7c0 con 0x7f44640a2e90 2024-09-11T01:46:22.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.794+0000 7f445affd640 1 -- 172.21.15.12:0/1226035730 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f446001c460 con 0x7f44640a2e90 2024-09-11T01:46:22.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.794+0000 7f446c391640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:22.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.794+0000 7f446c391640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:22.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.794+0000 7f446c391640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:22.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.794+0000 7f446c391640 1 -- 172.21.15.12:0/1226035730 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44640a3680 con 0x7f44640a2e90 2024-09-11T01:46:22.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.794+0000 7f445affd640 1 --2- 172.21.15.12:0/1226035730 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f444803e260 0x7f4448040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:22.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.795+0000 7f445affd640 1 -- 172.21.15.12:0/1226035730 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(588..588 src has 507..588) ==== 4843+0+0 (secure 0 0 0) 0x7f4460052c80 con 0x7f44640a2e90 2024-09-11T01:46:22.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.795+0000 7f446992c640 1 --2- 172.21.15.12:0/1226035730 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f444803e260 0x7f4448040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:22.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.795+0000 7f446992c640 10 monclient: get_auth_request con 0x7f444803e260 auth_method 0 2024-09-11T01:46:22.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.797+0000 7f446992c640 1 --2- 172.21.15.12:0/1226035730 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f444803e260 0x7f4448040720 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4464071140 tx=0x7f445402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:22.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.799+0000 7f445affd640 1 -- 172.21.15.12:0/1226035730 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f446001ac60 con 0x7f44640a2e90 2024-09-11T01:46:22.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.799+0000 7f445affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:22.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:22.799+0000 7f445affd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:46:23.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:23.054+0000 7f446c391640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:46:23.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:23.054+0000 7f446c391640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:46:23.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:23.054+0000 7f446c391640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:23.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:23.054+0000 7f446c391640 1 -- 172.21.15.12:0/1226035730 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f44640a2260 con 0x7f44640a2e90 2024-09-11T01:46:23.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:23.054+0000 7f445affd640 1 -- 172.21.15.12:0/1226035730 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f4460051640 con 0x7f44640a2e90 2024-09-11T01:46:23.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:23.055+0000 7f445affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:46:23.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:23.055+0000 7f445affd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:46:23.058 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":158,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:46:23.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:23.057+0000 7f446c391640 1 -- 172.21.15.12:0/1226035730 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f444803e260 msgr2=0x7f4448040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:23.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:23.057+0000 7f446c391640 1 --2- 172.21.15.12:0/1226035730 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f444803e260 0x7f4448040720 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4464071140 tx=0x7f445402e000 comp rx=0 tx=0).stop 2024-09-11T01:46:23.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:23.057+0000 7f446c391640 10 monclient: shutdown 2024-09-11T01:46:23.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:23.057+0000 7f446c391640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:23.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:23.057+0000 7f446c391640 1 -- 172.21.15.12:0/1226035730 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44640a2e90 msgr2=0x7f4464137b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:23.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:23.057+0000 7f446c391640 1 --2- 172.21.15.12:0/1226035730 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44640a2e90 0x7f4464137b10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f446000c430 tx=0x7f446000c8f0 comp rx=0 tx=0).stop 2024-09-11T01:46:23.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:23.057+0000 7f446c391640 1 -- 172.21.15.12:0/1226035730 shutdown_connections 2024-09-11T01:46:23.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:23.057+0000 7f446c391640 1 --2- 172.21.15.12:0/1226035730 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f444803e260 0x7f4448040720 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:23.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:23.057+0000 7f446c391640 1 --2- 172.21.15.12:0/1226035730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f446413d490 0x7f4464138590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:23.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:23.057+0000 7f446c391640 1 --2- 172.21.15.12:0/1226035730 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44640a3270 0x7f4464138050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:23.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:23.057+0000 7f446c391640 1 --2- 172.21.15.12:0/1226035730 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44640a2e90 0x7f4464137b10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:23.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:23.057+0000 7f446c391640 1 -- 172.21.15.12:0/1226035730 >> 172.21.15.12:0/1226035730 conn(0x7f44640a8490 msgr2=0x7f446413c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:23.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:23.057+0000 7f446c391640 1 -- 172.21.15.12:0/1226035730 shutdown_connections 2024-09-11T01:46:23.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:23.057+0000 7f446c391640 1 -- 172.21.15.12:0/1226035730 wait complete. 2024-09-11T01:46:23.081 INFO:tasks.rados.rados.0.smithi071.stdout:2627: done (0 left) 2024-09-11T01:46:23.082 INFO:tasks.rados.rados.0.smithi071.stdout:2628: read oid 14 snap 286 2024-09-11T01:46:23.082 INFO:tasks.rados.rados.0.smithi071.stdout:2628: expect (ObjNum 865 snap 266 seq_num 865) 2024-09-11T01:46:23.082 INFO:tasks.rados.rados.0.smithi071.stdout:2629: delete oid 41 current snap is 295 2024-09-11T01:46:23.088 INFO:tasks.rados.rados.0.smithi071.stdout:2629: done (1 left) 2024-09-11T01:46:23.088 INFO:tasks.rados.rados.0.smithi071.stdout:2630: read oid 15 snap -1 2024-09-11T01:46:23.088 INFO:tasks.rados.rados.0.smithi071.stdout:2630: expect (ObjNum 873 snap 267 seq_num 873) 2024-09-11T01:46:23.088 INFO:tasks.rados.rados.0.smithi071.stdout:2631: delete oid 20 current snap is 295 2024-09-11T01:46:23.092 INFO:tasks.rados.rados.0.smithi071.stdout:2628: expect (ObjNum 865 snap 266 seq_num 865) 2024-09-11T01:46:23.095 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1509571 2024-09-11T01:46:23.111 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 171372 2024-09-11T01:46:23.137 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:46:23.164 INFO:tasks.rados.rados.0.smithi071.stdout:2630: expect (ObjNum 873 snap 267 seq_num 873) 2024-09-11T01:46:23.168 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 877021 2024-09-11T01:46:23.277 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2321379 2024-09-11T01:46:23.289 INFO:tasks.rados.rados.0.smithi071.stdout:2628: done (2 left) 2024-09-11T01:46:23.289 INFO:tasks.rados.rados.0.smithi071.stdout:2630: done (1 left) 2024-09-11T01:46:23.289 INFO:tasks.rados.rados.0.smithi071.stdout:2631: done (0 left) 2024-09-11T01:46:23.289 INFO:tasks.rados.rados.0.smithi071.stdout:2632: read oid 18 snap -1 2024-09-11T01:46:23.289 INFO:tasks.rados.rados.0.smithi071.stdout:2632: expect deleted 2024-09-11T01:46:23.289 INFO:tasks.rados.rados.0.smithi071.stdout:2633: read oid 39 snap -1 2024-09-11T01:46:23.289 INFO:tasks.rados.rados.0.smithi071.stdout:2633: expect deleted 2024-09-11T01:46:23.289 INFO:tasks.rados.rados.0.smithi071.stdout:2634: snap_create 2024-09-11T01:46:24.280 INFO:tasks.rados.rados.0.smithi071.stdout:2632: done (2 left) 2024-09-11T01:46:24.280 INFO:tasks.rados.rados.0.smithi071.stdout:2633: done (1 left) 2024-09-11T01:46:24.280 INFO:tasks.rados.rados.0.smithi071.stdout:2634: done (0 left) 2024-09-11T01:46:24.280 INFO:tasks.rados.rados.0.smithi071.stdout:2635: read oid 16 snap -1 2024-09-11T01:46:24.280 INFO:tasks.rados.rados.0.smithi071.stdout:2635: expect (ObjNum 915 snap 290 seq_num 915) 2024-09-11T01:46:24.280 INFO:tasks.rados.rados.0.smithi071.stdout:2636: write oid 14 current snap is 296 2024-09-11T01:46:24.280 INFO:tasks.rados.rados.0.smithi071.stdout:2636: seq_num 942 ranges {632194=795977,2123920=790332,3419165=413029} 2024-09-11T01:46:24.287 INFO:tasks.rados.rados.0.smithi071.stdout:2636: writing smithi07134777-14 from 632194 to 1428171 tid 1 2024-09-11T01:46:24.296 INFO:tasks.rados.rados.0.smithi071.stdout:2636: writing smithi07134777-14 from 2123920 to 2914252 tid 2 2024-09-11T01:46:24.301 INFO:tasks.rados.rados.0.smithi071.stdout:2636: writing smithi07134777-14 from 3419165 to 3832194 tid 3 2024-09-11T01:46:24.302 INFO:tasks.rados.rados.0.smithi071.stdout:2637: delete oid 28 current snap is 296 2024-09-11T01:46:24.302 INFO:tasks.rados.rados.0.smithi071.stdout:2636: finishing write tid 1 to smithi07134777-14 2024-09-11T01:46:24.302 INFO:tasks.rados.rados.0.smithi071.stdout:2635: expect (ObjNum 915 snap 290 seq_num 915) 2024-09-11T01:46:24.302 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:46:24.501 INFO:tasks.rados.rados.0.smithi071.stdout:2636: finishing write tid 2 to smithi07134777-14 2024-09-11T01:46:24.501 INFO:tasks.rados.rados.0.smithi071.stdout:2636: finishing write tid 3 to smithi07134777-14 2024-09-11T01:46:24.501 INFO:tasks.rados.rados.0.smithi071.stdout:2636: finishing write tid 4 to smithi07134777-14 2024-09-11T01:46:24.501 INFO:tasks.rados.rados.0.smithi071.stdout:2636: finishing write tid 5 to smithi07134777-14 2024-09-11T01:46:24.501 INFO:tasks.rados.rados.0.smithi071.stdout:2636: oid 14 updating version 0 to 1750 2024-09-11T01:46:24.502 INFO:tasks.rados.rados.0.smithi071.stdout:2636: oid 14 version 1750 is already newer than 1749 2024-09-11T01:46:24.502 INFO:tasks.rados.rados.0.smithi071.stdout:2636: oid 14 version 1750 is already newer than 1748 2024-09-11T01:46:24.502 INFO:tasks.rados.rados.0.smithi071.stdout:2636: oid 14 updating version 1750 to 1751 2024-09-11T01:46:24.502 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 1751 (ObjNum 942 snap 296 seq_num 942) dirty exists 2024-09-11T01:46:24.502 INFO:tasks.rados.rados.0.smithi071.stdout:2636: left oid 14 (ObjNum 942 snap 296 seq_num 942) 2024-09-11T01:46:24.502 INFO:tasks.rados.rados.0.smithi071.stdout:2635: done (2 left) 2024-09-11T01:46:24.502 INFO:tasks.rados.rados.0.smithi071.stdout:2636: done (1 left) 2024-09-11T01:46:24.502 INFO:tasks.rados.rados.0.smithi071.stdout:2637: done (0 left) 2024-09-11T01:46:24.502 INFO:tasks.rados.rados.0.smithi071.stdout:2638: snap_remove snap 254 2024-09-11T01:46:25.297 INFO:tasks.rados.rados.0.smithi071.stdout:2638: done (0 left) 2024-09-11T01:46:25.297 INFO:tasks.rados.rados.0.smithi071.stdout:2639: read oid 23 snap 279 2024-09-11T01:46:25.298 INFO:tasks.rados.rados.0.smithi071.stdout:2639: expect (ObjNum 652 snap 199 seq_num 652) 2024-09-11T01:46:25.298 INFO:tasks.rados.rados.0.smithi071.stdout:2640: snap_remove snap 295 2024-09-11T01:46:26.070 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:46:26.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.181+0000 7f97c5f84640 10 monclient: get_monmap_and_config 2024-09-11T01:46:26.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.181+0000 7f97c5f84640 10 monclient: build_initial_monmap 2024-09-11T01:46:26.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.181+0000 7f97c5f84640 10 monclient: monmap: 2024-09-11T01:46:26.184 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:26.184 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:26.184 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:46:26.182835+0000 2024-09-11T01:46:26.184 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:46:26.182835+0000 2024-09-11T01:46:26.184 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:26.184 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:26.184 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:46:26.184 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:46:26.184 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:46:26.184 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:46:26.185 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:46:26.185 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:26.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.181+0000 7f97c5f84640 1 Processor -- start 2024-09-11T01:46:26.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.181+0000 7f97c5f84640 1 -- start start 2024-09-11T01:46:26.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.181+0000 7f97c5f84640 10 monclient: init 2024-09-11T01:46:26.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97c5f84640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:26.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97c5f84640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-11T01:46:26.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97c5f84640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f97c0138230 0x7f97c0138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:26.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97c5f84640 10 monclient(hunting): picked mon.noname-a con 0x7f97c0138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:46:26.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97c5f84640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f97c009d030 0x7f97c00a5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:26.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97c5f84640 10 monclient(hunting): picked mon.noname-c con 0x7f97c009d030 addr v2:172.21.15.71:3300/0 2024-09-11T01:46:26.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97c5f84640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97c009d460 0x7f97c00a5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:26.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97c5f84640 10 monclient(hunting): picked mon.noname-b con 0x7f97c009d460 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:26.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97c5f84640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:26.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97c5f84640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f97c009c9c0 con 0x7f97c0138230 2024-09-11T01:46:26.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97c5f84640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:26.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97c5f84640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f97c00a6e30 con 0x7f97c009d030 2024-09-11T01:46:26.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97c5f84640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:26.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97c5f84640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f97c009ba20 con 0x7f97c009d460 2024-09-11T01:46:26.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97c5f84640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:26.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97c5f84640 10 monclient(hunting): authenticate will time out at 1792.625614s 2024-09-11T01:46:26.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97bf7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f97c0138230 0x7f97c0138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:26.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97bffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97c009d460 0x7f97c00a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:26.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97bf7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f97c0138230 0x7f97c0138610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45580/0 (socket says 172.21.15.12:45580) 2024-09-11T01:46:26.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97bf7fe640 1 -- 172.21.15.12:0/1932388902 learned_addr learned my addr 172.21.15.12:0/1932388902 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:26.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97bf7fe640 10 monclient(hunting): get_auth_request con 0x7f97c0138230 auth_method 0 2024-09-11T01:46:26.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:26.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:26.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:26.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97bffff640 10 monclient(hunting): get_auth_request con 0x7f97c009d460 auth_method 0 2024-09-11T01:46:26.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:26.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97bffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:26.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:26.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97bf7fe640 1 -- 172.21.15.12:0/1932388902 >> v2:172.21.15.12:3300/0 conn(0x7f97c0138230 msgr2=0x7f97c0138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:46:26.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97bf7fe640 1 -- 172.21.15.12:0/1932388902 >> v2:172.21.15.12:3300/0 conn(0x7f97c0138230 msgr2=0x7f97c0138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:46:26.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97bf7fe640 1 --2- 172.21.15.12:0/1932388902 >> v2:172.21.15.12:3300/0 conn(0x7f97c0138230 0x7f97c0138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:46:26.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:26.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:26.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:26.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97bf7fe640 1 --2- 172.21.15.12:0/1932388902 >> v2:172.21.15.12:3300/0 conn(0x7f97c0138230 0x7f97c0138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:46:26.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97beffd640 1 --2- 172.21.15.12:0/1932388902 >> v2:172.21.15.71:3300/0 conn(0x7f97c009d030 0x7f97c00a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:26.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97beffd640 10 monclient(hunting): get_auth_request con 0x7f97c009d030 auth_method 0 2024-09-11T01:46:26.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:26.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97beffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:26.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.182+0000 7f97beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:26.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.183+0000 7f97bffff640 10 monclient(hunting): handle_auth_done global_id 184187 payload 274 2024-09-11T01:46:26.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.183+0000 7f97bffff640 1 -- 172.21.15.12:0/1932388902 >> v2:172.21.15.71:3300/0 conn(0x7f97c009d030 msgr2=0x7f97c00a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:46:26.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.183+0000 7f97bffff640 1 --2- 172.21.15.12:0/1932388902 >> v2:172.21.15.71:3300/0 conn(0x7f97c009d030 0x7f97c00a5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:26.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.183+0000 7f97bffff640 1 -- 172.21.15.12:0/1932388902 >> v2:172.21.15.12:3300/0 conn(0x7f97c0138230 msgr2=0x7f97c0138610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:46:26.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.183+0000 7f97bffff640 1 --2- 172.21.15.12:0/1932388902 >> v2:172.21.15.12:3300/0 conn(0x7f97c0138230 0x7f97c0138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:26.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.183+0000 7f97bffff640 10 monclient: _finish_hunting 0 2024-09-11T01:46:26.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.183+0000 7f97bffff640 1 monclient: found mon.noname-b 2024-09-11T01:46:26.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.183+0000 7f97bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:26.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.183+0000 7f97bffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:46:26.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.183+0000 7f97bffff640 1 -- 172.21.15.12:0/1932388902 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c00a66a0 con 0x7f97c009d460 2024-09-11T01:46:26.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.185+0000 7f97bffff640 1 -- 172.21.15.12:0/1932388902 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97c009d460 msgr2=0x7f97c00a5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:26.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.185+0000 7f97bffff640 1 --2- 172.21.15.12:0/1932388902 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97c009d460 0x7f97c00a5560 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f97c009dae0 tx=0x7f97b802a4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f9d28160ef38946 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:26.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97be7fc640 1 -- 172.21.15.12:0/1932388902 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b8007950 con 0x7f97c009d460 2024-09-11T01:46:26.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:26.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97be7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:46:26.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97be7fc640 10 monclient: dump: 2024-09-11T01:46:26.197 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:26.197 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:26.197 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:26.197 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:26.197 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:26.197 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:26.197 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:26.198 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:26.198 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:26.198 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:26.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97be7fc640 10 monclient: _finish_auth 0 2024-09-11T01:46:26.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97be7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:46:26.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:26.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97be7fc640 1 -- 172.21.15.12:0/1932388902 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97b8007af0 con 0x7f97c009d460 2024-09-11T01:46:26.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97c5f84640 5 monclient: authenticate success, global_id 184187 2024-09-11T01:46:26.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97c5f84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:46:26.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97be7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:26.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97c5f84640 10 monclient: get_monmap_and_config success 2024-09-11T01:46:26.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97be7fc640 1 -- 172.21.15.12:0/1932388902 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b8031400 con 0x7f97c009d460 2024-09-11T01:46:26.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:26.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97c5f84640 4 set_mon_vals no callback set 2024-09-11T01:46:26.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:26.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97be7fc640 10 monclient: dump: 2024-09-11T01:46:26.199 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:26.199 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:26.199 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:26.199 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:26.199 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:26.199 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:26.199 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:26.199 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:26.199 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:26.199 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:26.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97c5f84640 10 monclient: shutdown 2024-09-11T01:46:26.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97c5f84640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:26.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97c5f84640 1 -- 172.21.15.12:0/1932388902 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97c009d460 msgr2=0x7f97c00a5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:26.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97c5f84640 1 --2- 172.21.15.12:0/1932388902 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97c009d460 0x7f97c00a5560 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f97c009dae0 tx=0x7f97b802a4b0 comp rx=0 tx=0).stop 2024-09-11T01:46:26.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97c5f84640 1 -- 172.21.15.12:0/1932388902 shutdown_connections 2024-09-11T01:46:26.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97c5f84640 1 --2- 172.21.15.12:0/1932388902 >> v2:172.21.15.71:3300/0 conn(0x7f97c009d030 0x7f97c00a5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:26.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97c5f84640 1 --2- 172.21.15.12:0/1932388902 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97c009d460 0x7f97c00a5560 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:26.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97c5f84640 1 --2- 172.21.15.12:0/1932388902 >> v2:172.21.15.12:3300/0 conn(0x7f97c0138230 0x7f97c0138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:26.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97c5f84640 1 -- 172.21.15.12:0/1932388902 >> 172.21.15.12:0/1932388902 conn(0x7f97c00a7120 msgr2=0x7f97c00a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:26.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97c5f84640 1 -- 172.21.15.12:0/1932388902 shutdown_connections 2024-09-11T01:46:26.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.186+0000 7f97c5f84640 1 -- 172.21.15.12:0/1932388902 wait complete. 2024-09-11T01:46:26.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.187+0000 7f97c5f84640 10 monclient: build_initial_monmap 2024-09-11T01:46:26.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.187+0000 7f97c5f84640 10 monclient: monmap: 2024-09-11T01:46:26.200 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:26.200 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:46:26.201 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:46:26.201 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:46:26.201 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:26.201 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:26.201 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:46:26.201 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:46:26.201 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:46:26.201 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:26.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.187+0000 7f97c5f84640 1 Processor -- start 2024-09-11T01:46:26.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.187+0000 7f97c5f84640 1 -- start start 2024-09-11T01:46:26.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.187+0000 7f97c5f84640 10 monclient: init 2024-09-11T01:46:26.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97c5f84640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:26.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97c5f84640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:46:26.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97c5f84640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97c009d030 0x7f97c0137130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:26.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97c5f84640 10 monclient(hunting): picked mon.noname-b con 0x7f97c009d030 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:46:26.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97c5f84640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97c009d460 0x7f97c0137670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:26.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97c5f84640 10 monclient(hunting): picked mon.noname-c con 0x7f97c009d460 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:26.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97c5f84640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97c0138230 0x7f97c0124d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:26.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97c5f84640 10 monclient(hunting): picked mon.noname-a con 0x7f97c0138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:46:26.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97c5f84640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:26.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97c5f84640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f97c006f970 con 0x7f97c0138230 2024-09-11T01:46:26.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97c5f84640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:26.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97c5f84640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f97c006f7f0 con 0x7f97c009d030 2024-09-11T01:46:26.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97c5f84640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:26.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97c5f84640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f97c00a5aa0 con 0x7f97c009d460 2024-09-11T01:46:26.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97c5f84640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:26.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97c5f84640 10 monclient(hunting): authenticate will time out at 1792.631400s 2024-09-11T01:46:26.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97beffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97c009d460 0x7f97c0137670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:26.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97bffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97c0138230 0x7f97c0124d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:26.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97beffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97c009d460 0x7f97c0137670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55156/0 (socket says 172.21.15.12:55156) 2024-09-11T01:46:26.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97bffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97c0138230 0x7f97c0124d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45588/0 (socket says 172.21.15.12:45588) 2024-09-11T01:46:26.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97beffd640 1 -- 172.21.15.12:0/1281623885 learned_addr learned my addr 172.21.15.12:0/1281623885 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:26.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97beffd640 10 monclient(hunting): get_auth_request con 0x7f97c009d460 auth_method 0 2024-09-11T01:46:26.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:26.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97beffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:26.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:26.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97bffff640 10 monclient(hunting): get_auth_request con 0x7f97c0138230 auth_method 0 2024-09-11T01:46:26.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:26.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97bffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:26.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:26.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:26.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:26.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97bffff640 1 -- 172.21.15.12:0/1281623885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97c0138230 msgr2=0x7f97c0124d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:46:26.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97bffff640 1 -- 172.21.15.12:0/1281623885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97c0138230 msgr2=0x7f97c0124d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:46:26.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:26.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97bffff640 1 --2- 172.21.15.12:0/1281623885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97c0138230 0x7f97c0124d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:46:26.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97bf7fe640 1 --2- 172.21.15.12:0/1281623885 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97c009d030 0x7f97c0137130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:26.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97bffff640 1 --2- 172.21.15.12:0/1281623885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97c0138230 0x7f97c0124d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:46:26.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97bf7fe640 10 monclient(hunting): get_auth_request con 0x7f97c009d030 auth_method 0 2024-09-11T01:46:26.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:26.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:26.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:26.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:26.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:26.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:26.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97beffd640 10 monclient(hunting): handle_auth_done global_id 184190 payload 824 2024-09-11T01:46:26.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97beffd640 1 -- 172.21.15.12:0/1281623885 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97c009d030 msgr2=0x7f97c0137130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:26.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97beffd640 1 --2- 172.21.15.12:0/1281623885 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97c009d030 0x7f97c0137130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:26.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97beffd640 1 -- 172.21.15.12:0/1281623885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97c0138230 msgr2=0x7f97c0124d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:46:26.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97beffd640 1 --2- 172.21.15.12:0/1281623885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97c0138230 0x7f97c0124d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:26.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97beffd640 10 monclient: _finish_hunting 0 2024-09-11T01:46:26.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97beffd640 1 monclient: found mon.noname-c 2024-09-11T01:46:26.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:26.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97beffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:46:26.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.188+0000 7f97beffd640 1 -- 172.21.15.12:0/1281623885 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c0128b30 con 0x7f97c009d460 2024-09-11T01:46:26.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.191+0000 7f97beffd640 1 -- 172.21.15.12:0/1281623885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97c009d460 msgr2=0x7f97c0137670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:26.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.191+0000 7f97beffd640 1 --2- 172.21.15.12:0/1281623885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97c009d460 0x7f97c0137670 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f97b400d800 tx=0x7f97b400dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:26.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.191+0000 7f97a7fff640 1 -- 172.21.15.12:0/1281623885 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b400b8d0 con 0x7f97c009d460 2024-09-11T01:46:26.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.191+0000 7f97a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:26.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.191+0000 7f97a7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:46:26.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.191+0000 7f97a7fff640 10 monclient: dump: 2024-09-11T01:46:26.206 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:26.206 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:26.206 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:26.206 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:26.206 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:26.207 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:26.207 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:26.207 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:26.207 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:26.207 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:26.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.191+0000 7f97a7fff640 10 monclient: _finish_auth 0 2024-09-11T01:46:26.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.191+0000 7f97a7fff640 10 monclient: _check_auth_tickets 2024-09-11T01:46:26.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.191+0000 7f97a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:26.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.191+0000 7f97c5f84640 5 monclient: authenticate success, global_id 184190 2024-09-11T01:46:26.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.191+0000 7f97a7fff640 1 -- 172.21.15.12:0/1281623885 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97b4019070 con 0x7f97c009d460 2024-09-11T01:46:26.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.191+0000 7f97a7fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:26.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.191+0000 7f97a7fff640 1 -- 172.21.15.12:0/1281623885 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b400bc90 con 0x7f97c009d460 2024-09-11T01:46:26.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.191+0000 7f97c5f84640 10 monclient: _renew_subs 2024-09-11T01:46:26.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.191+0000 7f97c5f84640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:26.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.191+0000 7f97c5f84640 1 -- 172.21.15.12:0/1281623885 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97c00a1810 con 0x7f97c009d460 2024-09-11T01:46:26.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.191+0000 7f97a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:26.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.191+0000 7f97a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:26.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.191+0000 7f97a7fff640 10 monclient: dump: 2024-09-11T01:46:26.208 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:26.208 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:26.208 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:26.208 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:26.208 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:26.208 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:26.208 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:26.208 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:26.208 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:26.208 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:26.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.191+0000 7f97bdffb640 4 set_mon_vals no callback set 2024-09-11T01:46:26.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.191+0000 7f97c5f84640 10 monclient: _renew_subs 2024-09-11T01:46:26.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.191+0000 7f97c5f84640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:26.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.191+0000 7f97c5f84640 1 -- 172.21.15.12:0/1281623885 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f97c01267d0 con 0x7f97c009d460 2024-09-11T01:46:26.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.192+0000 7f97a7fff640 1 -- 172.21.15.12:0/1281623885 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f97b4013070 con 0x7f97c009d460 2024-09-11T01:46:26.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.192+0000 7f97c5f84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:26.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.192+0000 7f97c5f84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:26.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.192+0000 7f97c5f84640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:26.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.192+0000 7f97c5f84640 1 -- 172.21.15.12:0/1281623885 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97c00a32b0 con 0x7f97c009d460 2024-09-11T01:46:26.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.192+0000 7f97a7fff640 1 --2- 172.21.15.12:0/1281623885 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f97ac03e210 0x7f97ac0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:26.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.192+0000 7f97a7fff640 1 -- 172.21.15.12:0/1281623885 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(591..591 src has 507..591) ==== 4831+0+0 (secure 0 0 0) 0x7f97b4052f80 con 0x7f97c009d460 2024-09-11T01:46:26.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.192+0000 7f97bf7fe640 1 --2- 172.21.15.12:0/1281623885 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f97ac03e210 0x7f97ac0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:26.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.192+0000 7f97bf7fe640 10 monclient: get_auth_request con 0x7f97ac03e210 auth_method 0 2024-09-11T01:46:26.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.195+0000 7f97bf7fe640 1 --2- 172.21.15.12:0/1281623885 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f97ac03e210 0x7f97ac0406d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f97c0126380 tx=0x7f97b0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:26.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.197+0000 7f97a7fff640 1 -- 172.21.15.12:0/1281623885 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f97c00a32b0 con 0x7f97c009d460 2024-09-11T01:46:26.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.197+0000 7f97a7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:26.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.197+0000 7f97a7fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:46:26.304 INFO:tasks.rados.rados.0.smithi071.stdout:2640: done (1 left) 2024-09-11T01:46:26.304 INFO:tasks.rados.rados.0.smithi071.stdout:2641: read oid 1 snap -1 2024-09-11T01:46:26.304 INFO:tasks.rados.rados.0.smithi071.stdout:2641: expect (ObjNum 857 snap 263 seq_num 857) 2024-09-11T01:46:26.304 INFO:tasks.rados.rados.0.smithi071.stdout:2642: read oid 36 snap -1 2024-09-11T01:46:26.304 INFO:tasks.rados.rados.0.smithi071.stdout:2642: expect (ObjNum 829 snap 252 seq_num 829) 2024-09-11T01:46:26.304 INFO:tasks.rados.rados.0.smithi071.stdout:2643: snap_remove snap 290 2024-09-11T01:46:26.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.455+0000 7f97c5f84640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:46:26.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.455+0000 7f97c5f84640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:46:26.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.455+0000 7f97c5f84640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:26.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.455+0000 7f97c5f84640 1 -- 172.21.15.12:0/1281623885 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f97c00a9630 con 0x7f97c009d460 2024-09-11T01:46:26.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.455+0000 7f97a7fff640 1 -- 172.21.15.12:0/1281623885 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f97c00a9630 con 0x7f97c009d460 2024-09-11T01:46:26.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.455+0000 7f97a7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:46:26.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.455+0000 7f97a7fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:46:26.458 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":158,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:46:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.457+0000 7f97c5f84640 1 -- 172.21.15.12:0/1281623885 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f97ac03e210 msgr2=0x7f97ac0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.457+0000 7f97c5f84640 1 --2- 172.21.15.12:0/1281623885 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f97ac03e210 0x7f97ac0406d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f97c0126380 tx=0x7f97b0008040 comp rx=0 tx=0).stop 2024-09-11T01:46:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.457+0000 7f97c5f84640 10 monclient: shutdown 2024-09-11T01:46:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.457+0000 7f97c5f84640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.457+0000 7f97c5f84640 1 -- 172.21.15.12:0/1281623885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97c009d460 msgr2=0x7f97c0137670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.457+0000 7f97c5f84640 1 --2- 172.21.15.12:0/1281623885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97c009d460 0x7f97c0137670 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f97b400d800 tx=0x7f97b400dcc0 comp rx=0 tx=0).stop 2024-09-11T01:46:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.457+0000 7f97c5f84640 1 -- 172.21.15.12:0/1281623885 shutdown_connections 2024-09-11T01:46:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.457+0000 7f97c5f84640 1 --2- 172.21.15.12:0/1281623885 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f97ac03e210 0x7f97ac0406d0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.457+0000 7f97c5f84640 1 --2- 172.21.15.12:0/1281623885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f97c0138230 0x7f97c0124d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.457+0000 7f97c5f84640 1 --2- 172.21.15.12:0/1281623885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f97c009d460 0x7f97c0137670 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.457+0000 7f97c5f84640 1 --2- 172.21.15.12:0/1281623885 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97c009d030 0x7f97c0137130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.457+0000 7f97c5f84640 1 -- 172.21.15.12:0/1281623885 >> 172.21.15.12:0/1281623885 conn(0x7f97c00a7120 msgr2=0x7f97c00a36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.458+0000 7f97c5f84640 1 -- 172.21.15.12:0/1281623885 shutdown_connections 2024-09-11T01:46:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:26.458+0000 7f97c5f84640 1 -- 172.21.15.12:0/1281623885 wait complete. 2024-09-11T01:46:27.082 INFO:tasks.rados.rados.0.smithi071.stdout:2643: done (3 left) 2024-09-11T01:46:27.082 INFO:tasks.rados.rados.0.smithi071.stdout:2644: copy_from oid 17 from oid 48 current snap is 296 2024-09-11T01:46:27.082 INFO:tasks.rados.rados.0.smithi071.stdout:2639: expect (ObjNum 652 snap 199 seq_num 652) 2024-09-11T01:46:27.202 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:46:27.264 INFO:tasks.rados.rados.0.smithi071.stdout:2641: expect (ObjNum 857 snap 263 seq_num 857) 2024-09-11T01:46:27.320 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:46:27.427 INFO:tasks.rados.rados.0.smithi071.stdout:2642: expect (ObjNum 829 snap 252 seq_num 829) 2024-09-11T01:46:27.552 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:46:27.624 INFO:tasks.rados.rados.0.smithi071.stdout:2644: finishing copy_from racing read to smithi07134777-17 2024-09-11T01:46:27.625 INFO:tasks.rados.rados.0.smithi071.stdout:2644: finishing copy_from to smithi07134777-17 2024-09-11T01:46:27.625 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 1067 (ObjNum 688 snap 204 seq_num 688) dirty exists 2024-09-11T01:46:27.625 INFO:tasks.rados.rados.0.smithi071.stdout:2639: done (3 left) 2024-09-11T01:46:27.625 INFO:tasks.rados.rados.0.smithi071.stdout:2641: done (2 left) 2024-09-11T01:46:27.625 INFO:tasks.rados.rados.0.smithi071.stdout:2642: done (1 left) 2024-09-11T01:46:27.625 INFO:tasks.rados.rados.0.smithi071.stdout:2644: done (0 left) 2024-09-11T01:46:27.625 INFO:tasks.rados.rados.0.smithi071.stdout:2645: snap_remove snap 281 2024-09-11T01:46:28.282 INFO:tasks.rados.rados.0.smithi071.stdout:2645: done (0 left) 2024-09-11T01:46:28.282 INFO:tasks.rados.rados.0.smithi071.stdout:2646: snap_create 2024-09-11T01:46:29.472 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:46:29.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.583+0000 7ff3a16e2640 10 monclient: get_monmap_and_config 2024-09-11T01:46:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.583+0000 7ff3a16e2640 10 monclient: build_initial_monmap 2024-09-11T01:46:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.583+0000 7ff3a16e2640 10 monclient: monmap: 2024-09-11T01:46:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:46:29.585072+0000 2024-09-11T01:46:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:46:29.585072+0000 2024-09-11T01:46:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:46:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:46:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:46:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:46:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:46:29.587 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff3a16e2640 1 Processor -- start 2024-09-11T01:46:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff3a16e2640 1 -- start start 2024-09-11T01:46:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff3a16e2640 10 monclient: init 2024-09-11T01:46:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff3a16e2640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff3a16e2640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-11T01:46:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff3a16e2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff39c0a4170 addr v1:172.21.15.71:6789/0 2024-09-11T01:46:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff3a16e2640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff39c13af80 0x7ff39c13d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff3a16e2640 10 monclient(hunting): picked mon.noname-a con 0x7ff39c13af80 addr v2:172.21.15.12:3300/0 2024-09-11T01:46:29.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff3a16e2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff39c13d8b0 0x7ff39c13fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:29.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff3a16e2640 10 monclient(hunting): picked mon.noname-b con 0x7ff39c13d8b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:29.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff3a16e2640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff39c1380d0 con 0x7ff39c0a4170 2024-09-11T01:46:29.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff3a16e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:29.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff3a16e2640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff39c09fff0 con 0x7ff39c13af80 2024-09-11T01:46:29.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff3a16e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:29.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff3a16e2640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff39c138350 con 0x7ff39c13d8b0 2024-09-11T01:46:29.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff3a16e2640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:29.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff3a16e2640 10 monclient(hunting): authenticate will time out at 1796.027722s 2024-09-11T01:46:29.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff393fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff39c13af80 0x7ff39c13d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:29.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff39b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff39c13d8b0 0x7ff39c13fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:29.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff393fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff39c13af80 0x7ff39c13d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55222/0 (socket says 172.21.15.12:55222) 2024-09-11T01:46:29.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff393fff640 1 -- 172.21.15.12:0/249874719 learned_addr learned my addr 172.21.15.12:0/249874719 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:29.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff393fff640 10 monclient(hunting): get_auth_request con 0x7ff39c13af80 auth_method 0 2024-09-11T01:46:29.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff393fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:29.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff393fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:29.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff393fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:29.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff39b7fe640 10 monclient(hunting): get_auth_request con 0x7ff39c13d8b0 auth_method 0 2024-09-11T01:46:29.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff39b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:29.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff39b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:29.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.584+0000 7ff39b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:29.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.585+0000 7ff39b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:29.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.585+0000 7ff39b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:29.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.585+0000 7ff39b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:29.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.585+0000 7ff393fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:29.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.585+0000 7ff393fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:29.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.585+0000 7ff393fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:29.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.585+0000 7ff39b7fe640 10 monclient(hunting): handle_auth_done global_id 184193 payload 274 2024-09-11T01:46:29.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.585+0000 7ff39b7fe640 1 -- 172.21.15.12:0/249874719 >> v2:172.21.15.12:3300/0 conn(0x7ff39c13af80 msgr2=0x7ff39c13d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:46:29.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.585+0000 7ff39b7fe640 1 --2- 172.21.15.12:0/249874719 >> v2:172.21.15.12:3300/0 conn(0x7ff39c13af80 0x7ff39c13d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:29.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.585+0000 7ff39b7fe640 1 -- 172.21.15.12:0/249874719 >> v1:172.21.15.71:6789/0 conn(0x7ff39c0a4170 legacy=0x7ff39c073fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:46:29.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.585+0000 7ff39b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:46:29.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.585+0000 7ff39b7fe640 1 monclient: found mon.noname-b 2024-09-11T01:46:29.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.585+0000 7ff39b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:29.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.585+0000 7ff39b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:46:29.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.585+0000 7ff39b7fe640 1 -- 172.21.15.12:0/249874719 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff39c070280 con 0x7ff39c13d8b0 2024-09-11T01:46:29.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.588+0000 7ff39b7fe640 1 -- 172.21.15.12:0/249874719 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff39c13d8b0 msgr2=0x7ff39c13fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:29.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.588+0000 7ff39b7fe640 1 --2- 172.21.15.12:0/249874719 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff39c13d8b0 0x7ff39c13fca0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff394009d90 tx=0x7ff39402a490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a38b380683ec3a53 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:29.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.588+0000 7ff39a7fc640 1 -- 172.21.15.12:0/249874719 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff394007950 con 0x7ff39c13d8b0 2024-09-11T01:46:29.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.588+0000 7ff39a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:29.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.588+0000 7ff39a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:46:29.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.588+0000 7ff39a7fc640 10 monclient: dump: 2024-09-11T01:46:29.599 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:29.599 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:29.599 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:29.599 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:29.599 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:29.599 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:29.599 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:29.599 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:29.599 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:29.599 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:29.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.588+0000 7ff39a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:46:29.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.588+0000 7ff39a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:46:29.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.588+0000 7ff39a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:29.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.588+0000 7ff3a16e2640 5 monclient: authenticate success, global_id 184193 2024-09-11T01:46:29.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.588+0000 7ff3a16e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:46:29.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.588+0000 7ff39a7fc640 1 -- 172.21.15.12:0/249874719 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff394007af0 con 0x7ff39c13d8b0 2024-09-11T01:46:29.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.588+0000 7ff39a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:29.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.588+0000 7ff3a16e2640 10 monclient: get_monmap_and_config success 2024-09-11T01:46:29.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.588+0000 7ff3a16e2640 4 set_mon_vals no callback set 2024-09-11T01:46:29.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.588+0000 7ff39a7fc640 1 -- 172.21.15.12:0/249874719 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff394031400 con 0x7ff39c13d8b0 2024-09-11T01:46:29.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.588+0000 7ff39a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:29.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.588+0000 7ff39a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:29.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.588+0000 7ff39a7fc640 10 monclient: dump: 2024-09-11T01:46:29.601 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:29.601 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:29.601 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:29.601 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:29.601 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:29.601 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:29.601 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:29.601 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:29.601 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:29.601 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:29.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.589+0000 7ff3a16e2640 10 monclient: shutdown 2024-09-11T01:46:29.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.589+0000 7ff3a16e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:29.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.589+0000 7ff3a16e2640 1 -- 172.21.15.12:0/249874719 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff39c13d8b0 msgr2=0x7ff39c13fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:29.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.589+0000 7ff3a16e2640 1 --2- 172.21.15.12:0/249874719 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff39c13d8b0 0x7ff39c13fca0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff394009d90 tx=0x7ff39402a490 comp rx=0 tx=0).stop 2024-09-11T01:46:29.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.589+0000 7ff3a16e2640 1 -- 172.21.15.12:0/249874719 shutdown_connections 2024-09-11T01:46:29.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.589+0000 7ff3a16e2640 1 --2- 172.21.15.12:0/249874719 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff39c13d8b0 0x7ff39c13fca0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:29.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.589+0000 7ff3a16e2640 1 --2- 172.21.15.12:0/249874719 >> v2:172.21.15.12:3300/0 conn(0x7ff39c13af80 0x7ff39c13d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:29.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.589+0000 7ff3a16e2640 1 -- 172.21.15.12:0/249874719 >> 172.21.15.12:0/249874719 conn(0x7ff39c0a3650 msgr2=0x7ff39c0a3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:29.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.589+0000 7ff3a16e2640 1 -- 172.21.15.12:0/249874719 shutdown_connections 2024-09-11T01:46:29.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.589+0000 7ff3a16e2640 1 -- 172.21.15.12:0/249874719 wait complete. 2024-09-11T01:46:29.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.589+0000 7ff3a16e2640 10 monclient: build_initial_monmap 2024-09-11T01:46:29.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.589+0000 7ff3a16e2640 10 monclient: monmap: 2024-09-11T01:46:29.602 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:29.602 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:46:29.602 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:46:29.602 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:46:29.602 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:29.602 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:29.603 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:46:29.603 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:46:29.603 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:46:29.603 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:29.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff3a16e2640 1 Processor -- start 2024-09-11T01:46:29.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff3a16e2640 1 -- start start 2024-09-11T01:46:29.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff3a16e2640 10 monclient: init 2024-09-11T01:46:29.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff3a16e2640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:29.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff3a16e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:46:29.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff3a16e2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff39c0a4170 0x7ff39c1cce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:29.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff3a16e2640 10 monclient(hunting): picked mon.noname-c con 0x7ff39c0a4170 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:29.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff3a16e2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff39c13af80 0x7ff39c131780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:29.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff3a16e2640 10 monclient(hunting): picked mon.noname-a con 0x7ff39c13af80 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:46:29.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff3a16e2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff39c13d8b0 0x7ff39c131cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:29.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff3a16e2640 10 monclient(hunting): picked mon.noname-b con 0x7ff39c13d8b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:46:29.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff3a16e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:29.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff3a16e2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff39c074850 con 0x7ff39c13af80 2024-09-11T01:46:29.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff3a16e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:29.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff3a16e2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff39c0746d0 con 0x7ff39c13d8b0 2024-09-11T01:46:29.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff3a16e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:29.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff3a16e2640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff39c0a0660 con 0x7ff39c0a4170 2024-09-11T01:46:29.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff3a16e2640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:29.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff3a16e2640 10 monclient(hunting): authenticate will time out at 1796.033758s 2024-09-11T01:46:29.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff393fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff39c13af80 0x7ff39c131780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:29.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff39affd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff39c0a4170 0x7ff39c1cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:29.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff393fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff39c13af80 0x7ff39c131780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55226/0 (socket says 172.21.15.12:55226) 2024-09-11T01:46:29.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff39affd640 10 monclient(hunting): get_auth_request con 0x7ff39c0a4170 auth_method 0 2024-09-11T01:46:29.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff393fff640 1 -- 172.21.15.12:0/446686994 learned_addr learned my addr 172.21.15.12:0/446686994 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:29.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff39affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:29.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff39affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:29.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff39affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:29.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff393fff640 10 monclient(hunting): get_auth_request con 0x7ff39c13af80 auth_method 0 2024-09-11T01:46:29.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff393fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:29.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff393fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:29.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff393fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:29.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff39affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:29.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff39affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:29.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff39affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:29.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff393fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:29.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff393fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:29.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.590+0000 7ff393fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:29.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.591+0000 7ff39b7fe640 1 --2- 172.21.15.12:0/446686994 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff39c13d8b0 0x7ff39c131cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:29.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.591+0000 7ff39b7fe640 10 monclient(hunting): get_auth_request con 0x7ff39c13d8b0 auth_method 0 2024-09-11T01:46:29.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.591+0000 7ff39b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:29.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.591+0000 7ff39b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:29.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.591+0000 7ff39b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:29.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.591+0000 7ff393fff640 10 monclient(hunting): handle_auth_done global_id 194100 payload 824 2024-09-11T01:46:29.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.591+0000 7ff393fff640 1 -- 172.21.15.12:0/446686994 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff39c0a4170 msgr2=0x7ff39c1cce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:29.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.591+0000 7ff393fff640 1 --2- 172.21.15.12:0/446686994 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff39c0a4170 0x7ff39c1cce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:29.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.591+0000 7ff393fff640 1 -- 172.21.15.12:0/446686994 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff39c13d8b0 msgr2=0x7ff39c131cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:29.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.591+0000 7ff393fff640 1 --2- 172.21.15.12:0/446686994 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff39c13d8b0 0x7ff39c131cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:29.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.591+0000 7ff393fff640 10 monclient: _finish_hunting 0 2024-09-11T01:46:29.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.591+0000 7ff393fff640 1 monclient: found mon.noname-a 2024-09-11T01:46:29.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.591+0000 7ff393fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:29.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.591+0000 7ff393fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:46:29.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.591+0000 7ff393fff640 1 -- 172.21.15.12:0/446686994 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff39c11e130 con 0x7ff39c13af80 2024-09-11T01:46:29.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.591+0000 7ff39affd640 1 --2- 172.21.15.12:0/446686994 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff39c0a4170 0x7ff39c1cce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:46:29.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.593+0000 7ff393fff640 1 -- 172.21.15.12:0/446686994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff39c13af80 msgr2=0x7ff39c131780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:29.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.593+0000 7ff393fff640 1 --2- 172.21.15.12:0/446686994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff39c13af80 0x7ff39c131780 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff384009fd0 tx=0x7ff38400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:29.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.593+0000 7ff3937fe640 1 -- 172.21.15.12:0/446686994 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff384017070 con 0x7ff39c13af80 2024-09-11T01:46:29.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.593+0000 7ff3937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:29.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.593+0000 7ff3937fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:46:29.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.593+0000 7ff3937fe640 10 monclient: dump: 2024-09-11T01:46:29.607 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:29.607 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:29.608 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:29.608 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:29.608 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:29.608 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:29.608 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:29.608 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:29.608 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:29.608 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:29.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.593+0000 7ff3937fe640 10 monclient: _finish_auth 0 2024-09-11T01:46:29.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.593+0000 7ff3937fe640 10 monclient: _check_auth_tickets 2024-09-11T01:46:29.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.593+0000 7ff3937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:29.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.593+0000 7ff3937fe640 1 -- 172.21.15.12:0/446686994 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff38400cb50 con 0x7ff39c13af80 2024-09-11T01:46:29.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.593+0000 7ff3a16e2640 5 monclient: authenticate success, global_id 194100 2024-09-11T01:46:29.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.593+0000 7ff3a16e2640 10 monclient: _renew_subs 2024-09-11T01:46:29.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.593+0000 7ff3a16e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:46:29.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.593+0000 7ff3a16e2640 1 -- 172.21.15.12:0/446686994 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff39c074fd0 con 0x7ff39c13af80 2024-09-11T01:46:29.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.593+0000 7ff3937fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:29.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.593+0000 7ff3937fe640 1 -- 172.21.15.12:0/446686994 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff384013470 con 0x7ff39c13af80 2024-09-11T01:46:29.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.593+0000 7ff3937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:29.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.593+0000 7ff3937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:46:29.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.593+0000 7ff3937fe640 10 monclient: dump: 2024-09-11T01:46:29.609 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:29.609 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:29.609 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:29.609 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:29.609 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:29.609 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:29.609 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:29.610 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:29.610 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:29.610 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:29.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.593+0000 7ff3a16e2640 10 monclient: _renew_subs 2024-09-11T01:46:29.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.593+0000 7ff3a16e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:46:29.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.593+0000 7ff3a16e2640 1 -- 172.21.15.12:0/446686994 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff39c11d890 con 0x7ff39c13af80 2024-09-11T01:46:29.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.593+0000 7ff399ffb640 4 set_mon_vals no callback set 2024-09-11T01:46:29.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.594+0000 7ff3937fe640 1 -- 172.21.15.12:0/446686994 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff384024060 con 0x7ff39c13af80 2024-09-11T01:46:29.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.594+0000 7ff3a16e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:29.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.594+0000 7ff3a16e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:29.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.594+0000 7ff3a16e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:46:29.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.594+0000 7ff3a16e2640 1 -- 172.21.15.12:0/446686994 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff39c073fc0 con 0x7ff39c13af80 2024-09-11T01:46:29.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.594+0000 7ff3937fe640 1 --2- 172.21.15.12:0/446686994 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff37c03e210 0x7ff37c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:29.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.594+0000 7ff3937fe640 1 -- 172.21.15.12:0/446686994 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(594..594 src has 507..594) ==== 4831+0+0 (secure 0 0 0) 0x7ff384052cf0 con 0x7ff39c13af80 2024-09-11T01:46:29.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.595+0000 7ff39affd640 1 --2- 172.21.15.12:0/446686994 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff37c03e210 0x7ff37c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:29.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.595+0000 7ff39affd640 10 monclient: get_auth_request con 0x7ff37c03e210 auth_method 0 2024-09-11T01:46:29.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.597+0000 7ff39affd640 1 --2- 172.21.15.12:0/446686994 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff37c03e210 0x7ff37c0406d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ff38c006fd0 tx=0x7ff38c009060 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:29.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.599+0000 7ff3937fe640 1 -- 172.21.15.12:0/446686994 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff384058020 con 0x7ff39c13af80 2024-09-11T01:46:29.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.599+0000 7ff3937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:29.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.599+0000 7ff3937fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:46:29.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.858+0000 7ff3a16e2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:46:29.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.858+0000 7ff3a16e2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:46:29.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.858+0000 7ff3a16e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:46:29.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.858+0000 7ff3a16e2640 1 -- 172.21.15.12:0/446686994 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff39c11d560 con 0x7ff39c13af80 2024-09-11T01:46:29.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.858+0000 7ff3937fe640 1 -- 172.21.15.12:0/446686994 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7ff3840516c0 con 0x7ff39c13af80 2024-09-11T01:46:29.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.858+0000 7ff3937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:46:29.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.858+0000 7ff3937fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:46:29.862 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":162,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:46:29.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.861+0000 7ff3a16e2640 1 -- 172.21.15.12:0/446686994 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff37c03e210 msgr2=0x7ff37c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:29.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.861+0000 7ff3a16e2640 1 --2- 172.21.15.12:0/446686994 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff37c03e210 0x7ff37c0406d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ff38c006fd0 tx=0x7ff38c009060 comp rx=0 tx=0).stop 2024-09-11T01:46:29.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.861+0000 7ff3a16e2640 10 monclient: shutdown 2024-09-11T01:46:29.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.861+0000 7ff3a16e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:29.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.861+0000 7ff3a16e2640 1 -- 172.21.15.12:0/446686994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff39c13af80 msgr2=0x7ff39c131780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:29.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.861+0000 7ff3a16e2640 1 --2- 172.21.15.12:0/446686994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff39c13af80 0x7ff39c131780 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff384009fd0 tx=0x7ff38400ed40 comp rx=0 tx=0).stop 2024-09-11T01:46:29.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.861+0000 7ff3a16e2640 1 -- 172.21.15.12:0/446686994 shutdown_connections 2024-09-11T01:46:29.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.861+0000 7ff3a16e2640 1 --2- 172.21.15.12:0/446686994 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff39c13d8b0 0x7ff39c131cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:29.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.861+0000 7ff3a16e2640 1 --2- 172.21.15.12:0/446686994 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff37c03e210 0x7ff37c0406d0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:29.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.861+0000 7ff3a16e2640 1 --2- 172.21.15.12:0/446686994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff39c13af80 0x7ff39c131780 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:29.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.861+0000 7ff3a16e2640 1 --2- 172.21.15.12:0/446686994 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff39c0a4170 0x7ff39c1cce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:29.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.861+0000 7ff3a16e2640 1 -- 172.21.15.12:0/446686994 >> 172.21.15.12:0/446686994 conn(0x7ff39c0a3650 msgr2=0x7ff39c1364a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:29.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.861+0000 7ff3a16e2640 1 -- 172.21.15.12:0/446686994 shutdown_connections 2024-09-11T01:46:29.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.861+0000 7ff3a16e2640 1 -- 172.21.15.12:0/446686994 wait complete. 2024-09-11T01:46:29.874 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:46:29.874 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:46:29.874 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:46:29.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.973+0000 7fc40ce8c640 10 monclient: get_monmap_and_config 2024-09-11T01:46:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.973+0000 7fc40ce8c640 10 monclient: build_initial_monmap 2024-09-11T01:46:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.973+0000 7fc40ce8c640 10 monclient: monmap: 2024-09-11T01:46:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:46:29.975650+0000 2024-09-11T01:46:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:46:29.975650+0000 2024-09-11T01:46:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:46:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:46:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:46:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:46:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:46:29.977 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:29.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.974+0000 7fc40ce8c640 1 Processor -- start 2024-09-11T01:46:29.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.974+0000 7fc40ce8c640 1 -- start start 2024-09-11T01:46:29.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.974+0000 7fc40ce8c640 10 monclient: init 2024-09-11T01:46:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40ce8c640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40ce8c640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-11T01:46:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40ce8c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc408134ad0 0x7fc408134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40ce8c640 10 monclient(hunting): picked mon.noname-a con 0x7fc408134ad0 addr v2:172.21.15.12:3300/0 2024-09-11T01:46:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40ce8c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc4080a2200 0x7fc4081353f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40ce8c640 10 monclient(hunting): picked mon.noname-c con 0x7fc4080a2200 addr v2:172.21.15.71:3300/0 2024-09-11T01:46:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40ce8c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc408133a50 addr v1:172.21.15.12:6789/0 2024-09-11T01:46:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40ce8c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc40809ed30 con 0x7fc408133a50 2024-09-11T01:46:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40ce8c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40ce8c640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc40809f5d0 con 0x7fc408134ad0 2024-09-11T01:46:29.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40ce8c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40ce8c640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fc408135930 con 0x7fc4080a2200 2024-09-11T01:46:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40ce8c640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40ce8c640 10 monclient(hunting): authenticate will time out at 1796.418235s 2024-09-11T01:46:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40659c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc408134ad0 0x7fc408134eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc406d9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc408133a50 0x7fc408133ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54216/0 (socket says 172.21.15.12:54216) 2024-09-11T01:46:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc406d9d640 1 -- 172.21.15.12:0/2134595907 learned_addr learned my addr 172.21.15.12:0/2134595907 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:46:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40659c640 10 monclient(hunting): get_auth_request con 0x7fc408134ad0 auth_method 0 2024-09-11T01:46:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:29.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc405d9b640 1 --2- 172.21.15.12:0/2134595907 >> v2:172.21.15.71:3300/0 conn(0x7fc4080a2200 0x7fc4081353f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:29.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40659c640 10 monclient(hunting): handle_auth_done global_id 194103 payload 274 2024-09-11T01:46:29.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40659c640 1 -- 172.21.15.12:0/2134595907 >> v2:172.21.15.71:3300/0 conn(0x7fc4080a2200 msgr2=0x7fc4081353f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:46:29.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40659c640 1 --2- 172.21.15.12:0/2134595907 >> v2:172.21.15.71:3300/0 conn(0x7fc4080a2200 0x7fc4081353f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:29.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40659c640 1 -- 172.21.15.12:0/2134595907 >> v1:172.21.15.12:6789/0 conn(0x7fc408133a50 legacy=0x7fc408133ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:46:29.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40659c640 10 monclient: _finish_hunting 0 2024-09-11T01:46:29.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40659c640 1 monclient: found mon.noname-a 2024-09-11T01:46:29.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:29.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:46:29.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.975+0000 7fc40659c640 1 -- 172.21.15.12:0/2134595907 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc40809f010 con 0x7fc408134ad0 2024-09-11T01:46:29.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.978+0000 7fc40659c640 1 -- 172.21.15.12:0/2134595907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc408134ad0 msgr2=0x7fc408134eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:29.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.978+0000 7fc40659c640 1 --2- 172.21.15.12:0/2134595907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc408134ad0 0x7fc408134eb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc3f80097a0 tx=0x7fc3f802a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a91dbbf919d433b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:29.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.978+0000 7fc40559a640 1 -- 172.21.15.12:0/2134595907 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8037070 con 0x7fc408134ad0 2024-09-11T01:46:29.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.978+0000 7fc40559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:29.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.978+0000 7fc40559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:46:29.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.978+0000 7fc40559a640 10 monclient: dump: 2024-09-11T01:46:29.988 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:29.988 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:29.988 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:29.988 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:29.988 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:29.988 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:29.988 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:29.989 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:29.989 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:29.989 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:29.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.978+0000 7fc40559a640 10 monclient: _finish_auth 0 2024-09-11T01:46:29.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.978+0000 7fc40559a640 10 monclient: _check_auth_tickets 2024-09-11T01:46:29.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.978+0000 7fc40559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:29.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.978+0000 7fc40559a640 1 -- 172.21.15.12:0/2134595907 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3f8032640 con 0x7fc408134ad0 2024-09-11T01:46:29.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.978+0000 7fc40ce8c640 5 monclient: authenticate success, global_id 194103 2024-09-11T01:46:29.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.978+0000 7fc40ce8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:46:29.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.978+0000 7fc40559a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:29.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.978+0000 7fc40ce8c640 10 monclient: get_monmap_and_config success 2024-09-11T01:46:29.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.978+0000 7fc40ce8c640 4 set_mon_vals no callback set 2024-09-11T01:46:29.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.978+0000 7fc40559a640 1 -- 172.21.15.12:0/2134595907 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8032a10 con 0x7fc408134ad0 2024-09-11T01:46:29.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.978+0000 7fc40559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:29.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.978+0000 7fc40559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:46:29.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.978+0000 7fc40559a640 10 monclient: dump: 2024-09-11T01:46:29.990 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:29.990 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:29.990 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:29.990 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:29.990 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:29.990 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:29.990 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:29.990 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:29.990 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:29.990 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:29.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.979+0000 7fc40ce8c640 10 monclient: shutdown 2024-09-11T01:46:29.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.979+0000 7fc40ce8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:29.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.979+0000 7fc40ce8c640 1 -- 172.21.15.12:0/2134595907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc408134ad0 msgr2=0x7fc408134eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:29.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.979+0000 7fc40ce8c640 1 --2- 172.21.15.12:0/2134595907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc408134ad0 0x7fc408134eb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc3f80097a0 tx=0x7fc3f802a0a0 comp rx=0 tx=0).stop 2024-09-11T01:46:29.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.979+0000 7fc40ce8c640 1 -- 172.21.15.12:0/2134595907 shutdown_connections 2024-09-11T01:46:29.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.979+0000 7fc40ce8c640 1 --2- 172.21.15.12:0/2134595907 >> v2:172.21.15.71:3300/0 conn(0x7fc4080a2200 0x7fc4081353f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:29.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.979+0000 7fc40ce8c640 1 --2- 172.21.15.12:0/2134595907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc408134ad0 0x7fc408134eb0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:29.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.979+0000 7fc40ce8c640 1 -- 172.21.15.12:0/2134595907 >> 172.21.15.12:0/2134595907 conn(0x7fc4080a2f70 msgr2=0x7fc4080a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:29.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.979+0000 7fc40ce8c640 1 -- 172.21.15.12:0/2134595907 shutdown_connections 2024-09-11T01:46:29.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.979+0000 7fc40ce8c640 1 -- 172.21.15.12:0/2134595907 wait complete. 2024-09-11T01:46:29.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.980+0000 7fc40ce8c640 10 monclient: build_initial_monmap 2024-09-11T01:46:29.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.980+0000 7fc40ce8c640 10 monclient: monmap: 2024-09-11T01:46:29.991 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:29.991 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:46:29.991 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:46:29.992 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:46:29.992 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:29.992 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:29.992 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:46:29.992 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:46:29.992 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:46:29.992 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:29.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.980+0000 7fc40ce8c640 1 Processor -- start 2024-09-11T01:46:29.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.980+0000 7fc40ce8c640 1 -- start start 2024-09-11T01:46:29.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.980+0000 7fc40ce8c640 10 monclient: init 2024-09-11T01:46:29.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.980+0000 7fc40ce8c640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:29.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.980+0000 7fc40ce8c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:46:29.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.980+0000 7fc40ce8c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc4080a2200 0x7fc4081389d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:29.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.980+0000 7fc40ce8c640 10 monclient(hunting): picked mon.noname-c con 0x7fc4080a2200 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:29.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.980+0000 7fc40ce8c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc408133a50 0x7fc408138f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:29.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.980+0000 7fc40ce8c640 10 monclient(hunting): picked mon.noname-a con 0x7fc408133a50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:46:29.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.980+0000 7fc40ce8c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc408134ad0 0x7fc408139450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:29.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.980+0000 7fc40ce8c640 10 monclient(hunting): picked mon.noname-b con 0x7fc408134ad0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:46:29.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.980+0000 7fc40ce8c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:29.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.980+0000 7fc40ce8c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc4080a0150 con 0x7fc408133a50 2024-09-11T01:46:29.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.980+0000 7fc40ce8c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:29.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.980+0000 7fc40ce8c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc40809ffd0 con 0x7fc408134ad0 2024-09-11T01:46:29.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.980+0000 7fc40ce8c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:29.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.980+0000 7fc40ce8c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc4081345d0 con 0x7fc4080a2200 2024-09-11T01:46:29.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.980+0000 7fc40ce8c640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:29.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.980+0000 7fc40ce8c640 10 monclient(hunting): authenticate will time out at 1796.423994s 2024-09-11T01:46:29.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc40659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc4080a2200 0x7fc4081389d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc405d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc408133a50 0x7fc408138f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc40659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc4080a2200 0x7fc4081389d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:43430/0 (socket says 172.21.15.12:43430) 2024-09-11T01:46:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc40659c640 1 -- 172.21.15.12:0/2506350216 learned_addr learned my addr 172.21.15.12:0/2506350216 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc40659c640 10 monclient(hunting): get_auth_request con 0x7fc4080a2200 auth_method 0 2024-09-11T01:46:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc405d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc408133a50 0x7fc408138f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55248/0 (socket says 172.21.15.12:55248) 2024-09-11T01:46:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc40659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc40659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc40659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc405d9b640 10 monclient(hunting): get_auth_request con 0x7fc408133a50 auth_method 0 2024-09-11T01:46:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc405d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc405d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc405d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc40659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:29.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc40659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc40659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc405d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc405d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc405d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc406d9d640 1 --2- 172.21.15.12:0/2506350216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc408134ad0 0x7fc408139450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc405d9b640 10 monclient(hunting): handle_auth_done global_id 194106 payload 824 2024-09-11T01:46:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc405d9b640 1 -- 172.21.15.12:0/2506350216 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc4080a2200 msgr2=0x7fc4081389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc405d9b640 1 --2- 172.21.15.12:0/2506350216 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc4080a2200 0x7fc4081389d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc405d9b640 1 -- 172.21.15.12:0/2506350216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc408134ad0 msgr2=0x7fc408139450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc405d9b640 1 --2- 172.21.15.12:0/2506350216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc408134ad0 0x7fc408139450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc405d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:46:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc405d9b640 1 monclient: found mon.noname-a 2024-09-11T01:46:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc405d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:29.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc405d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:46:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc405d9b640 1 -- 172.21.15.12:0/2506350216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc40813c7a0 con 0x7fc408133a50 2024-09-11T01:46:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc406d9d640 10 monclient: get_auth_request con 0x7fc408134ad0 auth_method 0 2024-09-11T01:46:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc406d9d640 1 --2- 172.21.15.12:0/2506350216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc408134ad0 0x7fc408139450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:46:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.981+0000 7fc40659c640 1 --2- 172.21.15.12:0/2506350216 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc4080a2200 0x7fc4081389d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:46:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.983+0000 7fc405d9b640 1 -- 172.21.15.12:0/2506350216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc408133a50 msgr2=0x7fc408138f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.983+0000 7fc405d9b640 1 --2- 172.21.15.12:0/2506350216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc408133a50 0x7fc408138f10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc0109b0 tx=0x7fc3fc010e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.983+0000 7fc3f6ffd640 1 -- 172.21.15.12:0/2506350216 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc01c070 con 0x7fc408133a50 2024-09-11T01:46:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.983+0000 7fc3f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.983+0000 7fc3f6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:46:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.983+0000 7fc3f6ffd640 10 monclient: dump: 2024-09-11T01:46:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:29.996 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:29.997 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:29.997 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:29.997 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:29.997 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:29.997 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:29.997 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:29.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.983+0000 7fc3f6ffd640 10 monclient: _finish_auth 0 2024-09-11T01:46:29.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.983+0000 7fc3f6ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:46:29.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.983+0000 7fc3f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:29.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.983+0000 7fc3f6ffd640 1 -- 172.21.15.12:0/2506350216 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3fc00e2e0 con 0x7fc408133a50 2024-09-11T01:46:29.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.983+0000 7fc3f6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:29.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.983+0000 7fc40ce8c640 5 monclient: authenticate success, global_id 194106 2024-09-11T01:46:29.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.983+0000 7fc3f6ffd640 1 -- 172.21.15.12:0/2506350216 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc004270 con 0x7fc408133a50 2024-09-11T01:46:29.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.983+0000 7fc40ce8c640 10 monclient: _renew_subs 2024-09-11T01:46:29.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.983+0000 7fc40ce8c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:46:29.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.983+0000 7fc40ce8c640 1 -- 172.21.15.12:0/2506350216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc40809d480 con 0x7fc408133a50 2024-09-11T01:46:29.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.983+0000 7fc3f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:29.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.983+0000 7fc3f6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:46:29.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.983+0000 7fc3f6ffd640 10 monclient: dump: 2024-09-11T01:46:29.998 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:29.998 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:29.998 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:29.998 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:29.998 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:29.998 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:29.998 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:29.998 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:29.998 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:29.998 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:29.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.983+0000 7fc404d99640 4 set_mon_vals no callback set 2024-09-11T01:46:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.983+0000 7fc40ce8c640 10 monclient: _renew_subs 2024-09-11T01:46:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.983+0000 7fc40ce8c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:46:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.983+0000 7fc40ce8c640 1 -- 172.21.15.12:0/2506350216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc40813c4c0 con 0x7fc408133a50 2024-09-11T01:46:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.984+0000 7fc40ce8c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.984+0000 7fc40ce8c640 1 --2- 172.21.15.12:0/2506350216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc40813cf30 0x7fc4081cafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.984+0000 7fc40ce8c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.984+0000 7fc40ce8c640 1 -- 172.21.15.12:0/2506350216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc408140040 con 0x7fc40813cf30 2024-09-11T01:46:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.984+0000 7fc40659c640 1 --2- 172.21.15.12:0/2506350216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc40813cf30 0x7fc4081cafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.984+0000 7fc40659c640 10 monclient: get_auth_request con 0x7fc40813cf30 auth_method 0 2024-09-11T01:46:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.984+0000 7fc40659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.984+0000 7fc40659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.984+0000 7fc40659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.984+0000 7fc3f6ffd640 1 -- 172.21.15.12:0/2506350216 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fc3fc016070 con 0x7fc408133a50 2024-09-11T01:46:29.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.984+0000 7fc40659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:30.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.984+0000 7fc40659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:30.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.984+0000 7fc40659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:30.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.985+0000 7fc40659c640 10 monclient(hunting): handle_auth_done global_id 194109 payload 824 2024-09-11T01:46:30.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.985+0000 7fc3f6ffd640 1 --2- 172.21.15.12:0/2506350216 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc3e803e290 0x7fc3e8040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:30.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.985+0000 7fc40659c640 1 -- 172.21.15.12:0/2506350216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc408133ec0 con 0x7fc40813cf30 2024-09-11T01:46:30.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.985+0000 7fc3f6ffd640 1 -- 172.21.15.12:0/2506350216 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(594..594 src has 507..594) ==== 4831+0+0 (secure 0 0 0) 0x7fc3fc056340 con 0x7fc408133a50 2024-09-11T01:46:30.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.985+0000 7fc406d9d640 1 --2- 172.21.15.12:0/2506350216 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc3e803e290 0x7fc3e8040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:30.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.985+0000 7fc406d9d640 10 monclient: get_auth_request con 0x7fc3e803e290 auth_method 0 2024-09-11T01:46:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.987+0000 7fc40659c640 1 -- 172.21.15.12:0/2506350216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc40813cf30 msgr2=0x7fc4081cafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.987+0000 7fc40659c640 1 --2- 172.21.15.12:0/2506350216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc40813cf30 0x7fc4081cafb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc3f80097a0 tx=0x7fc3f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.987+0000 7fc3f6ffd640 1 -- 172.21.15.12:0/2506350216 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8043070 con 0x7fc40813cf30 2024-09-11T01:46:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.987+0000 7fc3f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.987+0000 7fc3f6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:46:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.987+0000 7fc3f6ffd640 10 monclient: dump: 2024-09-11T01:46:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:30.003 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:30.004 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:30.004 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:30.004 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:30.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.987+0000 7fc3f6ffd640 1 -- 172.21.15.12:0/2506350216 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc3f8038040 con 0x7fc40813cf30 2024-09-11T01:46:30.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.987+0000 7fc3f6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:30.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.987+0000 7fc3f6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:46:30.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.987+0000 7fc3f6ffd640 1 -- 172.21.15.12:0/2506350216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc40813cf30 msgr2=0x7fc4081cafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:30.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.987+0000 7fc3f6ffd640 1 --2- 172.21.15.12:0/2506350216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc40813cf30 0x7fc4081cafb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc3f80097a0 tx=0x7fc3f80046c0 comp rx=0 tx=0).stop 2024-09-11T01:46:30.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.987+0000 7fc3f6ffd640 1 -- 172.21.15.12:0/2506350216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc40813cf30 msgr2=0x7fc4081cafb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:46:30.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.987+0000 7fc3f6ffd640 1 --2- 172.21.15.12:0/2506350216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc40813cf30 0x7fc4081cafb0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:30.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.988+0000 7fc406d9d640 1 --2- 172.21.15.12:0/2506350216 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc3e803e290 0x7fc3e8040750 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fc400005ac0 tx=0x7fc400009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:30.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.993+0000 7fc40ce8c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:46:30.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.993+0000 7fc40ce8c640 1 --2- 172.21.15.12:0/2506350216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4081cb4f0 0x7fc4081cb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:30.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.993+0000 7fc40ce8c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:30.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.993+0000 7fc40ce8c640 1 -- 172.21.15.12:0/2506350216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc4081cbe10 con 0x7fc4081cb4f0 2024-09-11T01:46:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.993+0000 7fc40659c640 1 --2- 172.21.15.12:0/2506350216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4081cb4f0 0x7fc4081cb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.994+0000 7fc40659c640 10 monclient: get_auth_request con 0x7fc4081cb4f0 auth_method 0 2024-09-11T01:46:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.994+0000 7fc40659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.994+0000 7fc40659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.994+0000 7fc40659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.994+0000 7fc40659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.994+0000 7fc40659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.994+0000 7fc40659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.994+0000 7fc40659c640 10 monclient(hunting): handle_auth_done global_id 194112 payload 824 2024-09-11T01:46:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.994+0000 7fc40659c640 1 -- 172.21.15.12:0/2506350216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc40813afa0 con 0x7fc4081cb4f0 2024-09-11T01:46:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.996+0000 7fc40659c640 1 -- 172.21.15.12:0/2506350216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4081cb4f0 msgr2=0x7fc4081cb8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.996+0000 7fc40659c640 1 --2- 172.21.15.12:0/2506350216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4081cb4f0 0x7fc4081cb8d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8042000 tx=0x7fc3f8008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.996+0000 7fc3f6ffd640 1 -- 172.21.15.12:0/2506350216 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8037040 con 0x7fc4081cb4f0 2024-09-11T01:46:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.996+0000 7fc3f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.996+0000 7fc3f6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:46:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.996+0000 7fc3f6ffd640 10 monclient: dump: 2024-09-11T01:46:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:30.006 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.996+0000 7fc3f6ffd640 1 -- 172.21.15.12:0/2506350216 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7fc3f8003d90 con 0x7fc4081cb4f0 2024-09-11T01:46:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.996+0000 7fc3f6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:46:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.996+0000 7fc3f6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:46:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.996+0000 7fc3f6ffd640 1 -- 172.21.15.12:0/2506350216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4081cb4f0 msgr2=0x7fc4081cb8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.996+0000 7fc3f6ffd640 1 --2- 172.21.15.12:0/2506350216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4081cb4f0 0x7fc4081cb8d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8042000 tx=0x7fc3f8008030 comp rx=0 tx=0).stop 2024-09-11T01:46:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.996+0000 7fc3f6ffd640 1 -- 172.21.15.12:0/2506350216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4081cb4f0 msgr2=0x7fc4081cb8d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:46:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.996+0000 7fc3f6ffd640 1 --2- 172.21.15.12:0/2506350216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4081cb4f0 0x7fc4081cb8d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:30.007 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:46:30.007 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:46:30.007 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:46:30.007 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:46:30.007 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 162, 2024-09-11T01:46:30.007 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 7233, 2024-09-11T01:46:30.007 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:46:30.007 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:46:30.007 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:46:30.008 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:46:30.008 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:30.008 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:46:30.008 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:46:30.008 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:46:30.008 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:46:30.008 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:30.008 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:30.008 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:30.008 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:30.008 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:30.008 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:30.008 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:30.008 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:30.009 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:30.009 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:30.009 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:30.009 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:30.009 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:46:30.009 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:46:30.009 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:30.009 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:30.009 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:30.009 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:30.009 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:30.009 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:30.009 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:30.009 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:30.010 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:30.010 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:30.010 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:30.010 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:30.010 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.010 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:46:30.010 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:46:30.010 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:46:30.010 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:46:30.010 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:46:30.010 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:46:30.010 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:46:30.010 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:46:30.010 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:46:30.010 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:46:30.011 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:46:30.011 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:46:30.011 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:46:30.011 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:46:30.011 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:46:30.011 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:46:30.011 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:46:30.011 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:30.011 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:30.011 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:30.011 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:30.011 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:30.011 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:30.011 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:30.011 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:30.012 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:30.012 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:30.012 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:30.012 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:30.012 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:46:30.012 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.012 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:46:30.012 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.012 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:46:30.012 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:46:30.012 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:30.012 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:30.012 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.012 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:30.013 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:46:30.013 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:30.013 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.013 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.013 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:30.013 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:46:30.013 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:30.013 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:30.013 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:30.013 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.013 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:46:30.013 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:46:30.013 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:30.013 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:30.013 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:30.014 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.014 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.014 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:46:30.014 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:46:30.014 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:30.014 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:30.014 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.014 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:30.014 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:46:30.014 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:30.014 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.014 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.014 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:30.014 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:46:30.014 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:30.015 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:30.015 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:30.015 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.015 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:46:30.015 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:46:30.015 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:30.015 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:30.015 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:30.015 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.015 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.015 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:46:30.015 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:46:30.015 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:30.015 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:30.016 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.016 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:30.016 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:46:30.016 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:30.016 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.016 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.016 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:30.016 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:46:30.016 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:30.016 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:30.016 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:30.016 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.016 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:46:30.016 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:46:30.017 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:30.017 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:30.017 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:30.017 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:30.017 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:30.017 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.017 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:46:30.017 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:46:30.017 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.017 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:30.017 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:30.017 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:46:30.017 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:30.017 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:30.018 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:46:30.018 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.018 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:30.018 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:30.018 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:46:30.018 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:30.018 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:30.018 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:46:30.018 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.018 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:30.018 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:30.018 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:46:30.018 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:30.018 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:30.018 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.019 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:46:30.019 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:46:30.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.997+0000 7fc40ce8c640 1 -- 172.21.15.12:0/2506350216 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc3e803e290 msgr2=0x7fc3e8040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:30.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.997+0000 7fc40ce8c640 1 --2- 172.21.15.12:0/2506350216 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc3e803e290 0x7fc3e8040750 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fc400005ac0 tx=0x7fc400009040 comp rx=0 tx=0).stop 2024-09-11T01:46:30.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.997+0000 7fc40ce8c640 10 monclient: shutdown 2024-09-11T01:46:30.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.997+0000 7fc40ce8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:30.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.997+0000 7fc40ce8c640 1 -- 172.21.15.12:0/2506350216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc408133a50 msgr2=0x7fc408138f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:30.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.997+0000 7fc40ce8c640 1 --2- 172.21.15.12:0/2506350216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc408133a50 0x7fc408138f10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc0109b0 tx=0x7fc3fc010e70 comp rx=0 tx=0).stop 2024-09-11T01:46:30.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.997+0000 7fc406d9d640 1 -- 172.21.15.12:0/2506350216 reap_dead start 2024-09-11T01:46:30.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.997+0000 7fc406d9d640 1 -- 172.21.15.12:0/2506350216 reap_dead start 2024-09-11T01:46:30.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.997+0000 7fc40ce8c640 1 -- 172.21.15.12:0/2506350216 shutdown_connections 2024-09-11T01:46:30.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.997+0000 7fc40ce8c640 1 -- 172.21.15.12:0/2506350216 >> 172.21.15.12:0/2506350216 conn(0x7fc4080a2f70 msgr2=0x7fc40809d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:30.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.997+0000 7fc40ce8c640 1 -- 172.21.15.12:0/2506350216 shutdown_connections 2024-09-11T01:46:30.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:29.997+0000 7fc40ce8c640 1 -- 172.21.15.12:0/2506350216 wait complete. 2024-09-11T01:46:30.021 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:46:30.021 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:46:30.021 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:46:30.035 INFO:tasks.rados.rados.0.smithi071.stdout:2646: done (0 left) 2024-09-11T01:46:30.035 INFO:tasks.rados.rados.0.smithi071.stdout:2647: rollback oid 46 current snap is 297 2024-09-11T01:46:30.035 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 46 to 174 2024-09-11T01:46:30.035 INFO:tasks.rados.rados.0.smithi071.stdout:2648: read oid 41 snap 283 2024-09-11T01:46:30.035 INFO:tasks.rados.rados.0.smithi071.stdout:2648: expect (ObjNum 832 snap 254 seq_num 832) 2024-09-11T01:46:30.035 INFO:tasks.rados.rados.0.smithi071.stdout:2649: snap_remove snap 288 2024-09-11T01:46:30.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.122+0000 7f44f5c5a640 10 monclient: get_monmap_and_config 2024-09-11T01:46:30.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.122+0000 7f44f5c5a640 10 monclient: build_initial_monmap 2024-09-11T01:46:30.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.122+0000 7f44f5c5a640 10 monclient: monmap: 2024-09-11T01:46:30.125 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:30.125 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:30.125 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:46:30.124255+0000 2024-09-11T01:46:30.125 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:46:30.124255+0000 2024-09-11T01:46:30.125 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:30.125 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:30.125 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:46:30.125 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:46:30.125 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:46:30.126 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:46:30.126 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:46:30.126 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:30.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.123+0000 7f44f5c5a640 1 Processor -- start 2024-09-11T01:46:30.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.123+0000 7f44f5c5a640 1 -- start start 2024-09-11T01:46:30.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.123+0000 7f44f5c5a640 10 monclient: init 2024-09-11T01:46:30.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.123+0000 7f44f5c5a640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:30.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.123+0000 7f44f5c5a640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-11T01:46:30.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.123+0000 7f44f5c5a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f44f009dcd0 addr v1:172.21.15.71:6789/0 2024-09-11T01:46:30.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.123+0000 7f44f5c5a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f44f009e0b0 addr v1:172.21.15.12:6789/0 2024-09-11T01:46:30.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.123+0000 7f44f5c5a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f44f013c6c0 0x7f44f013eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:30.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.123+0000 7f44f5c5a640 10 monclient(hunting): picked mon.noname-a con 0x7f44f013c6c0 addr v2:172.21.15.12:3300/0 2024-09-11T01:46:30.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.123+0000 7f44f5c5a640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44f009caa0 con 0x7f44f009e0b0 2024-09-11T01:46:30.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.123+0000 7f44f5c5a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44f00a2b50 con 0x7f44f009dcd0 2024-09-11T01:46:30.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.123+0000 7f44f5c5a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:30.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.123+0000 7f44f5c5a640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f44f00a3b30 con 0x7f44f013c6c0 2024-09-11T01:46:30.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.123+0000 7f44f5c5a640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:30.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.123+0000 7f44f5c5a640 10 monclient(hunting): authenticate will time out at 1796.566812s 2024-09-11T01:46:30.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.123+0000 7f44effff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f44f013c6c0 0x7f44f013eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:30.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.123+0000 7f44effff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f44f013c6c0 0x7f44f013eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55260/0 (socket says 172.21.15.12:55260) 2024-09-11T01:46:30.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.123+0000 7f44eeffd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f44f009e0b0 0x7f44f00a2440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54228/0 (socket says 172.21.15.12:54228) 2024-09-11T01:46:30.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.123+0000 7f44effff640 1 -- 172.21.15.12:0/3371477859 learned_addr learned my addr 172.21.15.12:0/3371477859 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:30.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.123+0000 7f44effff640 10 monclient(hunting): get_auth_request con 0x7f44f013c6c0 auth_method 0 2024-09-11T01:46:30.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.123+0000 7f44effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:30.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.123+0000 7f44effff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:30.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.123+0000 7f44effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:30.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.124+0000 7f44effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:30.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.124+0000 7f44effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:30.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.124+0000 7f44effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:30.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.124+0000 7f44effff640 10 monclient(hunting): handle_auth_done global_id 194115 payload 274 2024-09-11T01:46:30.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.124+0000 7f44effff640 1 -- 172.21.15.12:0/3371477859 >> v1:172.21.15.71:6789/0 conn(0x7f44f009dcd0 legacy=0x7f44f00a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:46:30.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.124+0000 7f44effff640 1 -- 172.21.15.12:0/3371477859 >> v1:172.21.15.12:6789/0 conn(0x7f44f009e0b0 legacy=0x7f44f00a2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:46:30.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.124+0000 7f44effff640 10 monclient: _finish_hunting 0 2024-09-11T01:46:30.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.124+0000 7f44effff640 1 monclient: found mon.noname-a 2024-09-11T01:46:30.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.124+0000 7f44effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:30.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.124+0000 7f44effff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:46:30.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.124+0000 7f44effff640 1 -- 172.21.15.12:0/3371477859 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f00a3470 con 0x7f44f013c6c0 2024-09-11T01:46:30.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.127+0000 7f44effff640 1 -- 172.21.15.12:0/3371477859 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44f013c6c0 msgr2=0x7f44f013eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:30.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.127+0000 7f44effff640 1 --2- 172.21.15.12:0/3371477859 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44f013c6c0 0x7f44f013eab0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f44e8009e50 tx=0x7f44e802a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5364250b59e8f3f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:30.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.127+0000 7f44ee7fc640 1 -- 172.21.15.12:0/3371477859 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e8036070 con 0x7f44f013c6c0 2024-09-11T01:46:30.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.127+0000 7f44ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:30.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.127+0000 7f44ee7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:46:30.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.127+0000 7f44ee7fc640 10 monclient: dump: 2024-09-11T01:46:30.134 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:30.134 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:30.134 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.134 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.134 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:30.134 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:30.135 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:30.135 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:30.135 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:30.135 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:30.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.127+0000 7f44ee7fc640 10 monclient: _finish_auth 0 2024-09-11T01:46:30.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.127+0000 7f44ee7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:46:30.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.127+0000 7f44ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:30.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.127+0000 7f44ee7fc640 1 -- 172.21.15.12:0/3371477859 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44e8007800 con 0x7f44f013c6c0 2024-09-11T01:46:30.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.127+0000 7f44f5c5a640 5 monclient: authenticate success, global_id 194115 2024-09-11T01:46:30.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.127+0000 7f44f5c5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:46:30.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.127+0000 7f44ee7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:30.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.127+0000 7f44f5c5a640 10 monclient: get_monmap_and_config success 2024-09-11T01:46:30.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.127+0000 7f44ee7fc640 1 -- 172.21.15.12:0/3371477859 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e8007bd0 con 0x7f44f013c6c0 2024-09-11T01:46:30.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.127+0000 7f44f5c5a640 4 set_mon_vals no callback set 2024-09-11T01:46:30.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.127+0000 7f44ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:30.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.127+0000 7f44ee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:46:30.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.127+0000 7f44ee7fc640 10 monclient: dump: 2024-09-11T01:46:30.136 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:30.136 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:30.136 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.136 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.136 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:30.136 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:30.136 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:30.136 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:30.136 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:30.136 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:30.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.127+0000 7f44f5c5a640 10 monclient: shutdown 2024-09-11T01:46:30.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.127+0000 7f44f5c5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:30.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.127+0000 7f44f5c5a640 1 -- 172.21.15.12:0/3371477859 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44f013c6c0 msgr2=0x7f44f013eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:30.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.127+0000 7f44f5c5a640 1 --2- 172.21.15.12:0/3371477859 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44f013c6c0 0x7f44f013eab0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f44e8009e50 tx=0x7f44e802a560 comp rx=0 tx=0).stop 2024-09-11T01:46:30.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.128+0000 7f44f5c5a640 1 -- 172.21.15.12:0/3371477859 shutdown_connections 2024-09-11T01:46:30.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.128+0000 7f44f5c5a640 1 --2- 172.21.15.12:0/3371477859 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44f013c6c0 0x7f44f013eab0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:30.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.128+0000 7f44f5c5a640 1 -- 172.21.15.12:0/3371477859 >> 172.21.15.12:0/3371477859 conn(0x7f44f00a7da0 msgr2=0x7f44f00a3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:30.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.128+0000 7f44f5c5a640 1 -- 172.21.15.12:0/3371477859 shutdown_connections 2024-09-11T01:46:30.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.128+0000 7f44f5c5a640 1 -- 172.21.15.12:0/3371477859 wait complete. 2024-09-11T01:46:30.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.128+0000 7f44f5c5a640 10 monclient: build_initial_monmap 2024-09-11T01:46:30.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.128+0000 7f44f5c5a640 10 monclient: monmap: 2024-09-11T01:46:30.137 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:30.137 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:46:30.137 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:46:30.137 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:46:30.137 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:30.138 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:30.138 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:46:30.138 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:46:30.138 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:46:30.138 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:30.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.128+0000 7f44f5c5a640 1 Processor -- start 2024-09-11T01:46:30.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44f5c5a640 1 -- start start 2024-09-11T01:46:30.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44f5c5a640 10 monclient: init 2024-09-11T01:46:30.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44f5c5a640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:30.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44f5c5a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:46:30.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44f5c5a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44f009dcd0 0x7f44f01c4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:30.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44f5c5a640 10 monclient(hunting): picked mon.noname-b con 0x7f44f009dcd0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:46:30.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44f5c5a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44f009e0b0 0x7f44f013bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:30.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44f5c5a640 10 monclient(hunting): picked mon.noname-a con 0x7f44f009e0b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:46:30.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44f5c5a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44f013c6c0 0x7f44f012f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:30.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44f5c5a640 10 monclient(hunting): picked mon.noname-c con 0x7f44f013c6c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:30.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44f5c5a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:30.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44f5c5a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f44f009d790 con 0x7f44f009e0b0 2024-09-11T01:46:30.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44f5c5a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:30.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44f5c5a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f44f009d610 con 0x7f44f009dcd0 2024-09-11T01:46:30.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44f5c5a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:30.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44f5c5a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f44f0074b50 con 0x7f44f013c6c0 2024-09-11T01:46:30.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44f5c5a640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:30.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44f5c5a640 10 monclient(hunting): authenticate will time out at 1796.572546s 2024-09-11T01:46:30.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44eeffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44f009e0b0 0x7f44f013bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:30.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44effff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44f013c6c0 0x7f44f012f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:30.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44eeffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44f009e0b0 0x7f44f013bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55274/0 (socket says 172.21.15.12:55274) 2024-09-11T01:46:30.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44eeffd640 1 -- 172.21.15.12:0/1034756864 learned_addr learned my addr 172.21.15.12:0/1034756864 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:30.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44eeffd640 10 monclient(hunting): get_auth_request con 0x7f44f009e0b0 auth_method 0 2024-09-11T01:46:30.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44effff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44f013c6c0 0x7f44f012f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:43436/0 (socket says 172.21.15.12:43436) 2024-09-11T01:46:30.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:30.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:30.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:30.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44effff640 10 monclient(hunting): get_auth_request con 0x7f44f013c6c0 auth_method 0 2024-09-11T01:46:30.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:30.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44effff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:30.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:30.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:30.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:30.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:30.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44ef7fe640 1 --2- 172.21.15.12:0/1034756864 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44f009dcd0 0x7f44f01c4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:30.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:30.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:30.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:30.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44ef7fe640 10 monclient(hunting): get_auth_request con 0x7f44f009dcd0 auth_method 0 2024-09-11T01:46:30.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:30.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:30.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:30.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:30.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:30.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:30.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44eeffd640 10 monclient(hunting): handle_auth_done global_id 194118 payload 824 2024-09-11T01:46:30.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44eeffd640 1 -- 172.21.15.12:0/1034756864 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44f013c6c0 msgr2=0x7f44f012f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:30.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44eeffd640 1 --2- 172.21.15.12:0/1034756864 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44f013c6c0 0x7f44f012f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:30.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44eeffd640 1 -- 172.21.15.12:0/1034756864 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44f009dcd0 msgr2=0x7f44f01c4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:30.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.129+0000 7f44eeffd640 1 --2- 172.21.15.12:0/1034756864 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44f009dcd0 0x7f44f01c4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:30.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.130+0000 7f44eeffd640 10 monclient: _finish_hunting 0 2024-09-11T01:46:30.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.130+0000 7f44eeffd640 1 monclient: found mon.noname-a 2024-09-11T01:46:30.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.130+0000 7f44eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:30.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.130+0000 7f44eeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:46:30.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.130+0000 7f44eeffd640 1 -- 172.21.15.12:0/1034756864 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f01cbd20 con 0x7f44f009e0b0 2024-09-11T01:46:30.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.132+0000 7f44eeffd640 1 -- 172.21.15.12:0/1034756864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44f009e0b0 msgr2=0x7f44f013bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:30.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.132+0000 7f44eeffd640 1 --2- 172.21.15.12:0/1034756864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44f009e0b0 0x7f44f013bcd0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f44e400c450 tx=0x7f44e400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:30.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.132+0000 7f44d7fff640 1 -- 172.21.15.12:0/1034756864 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e4017070 con 0x7f44f009e0b0 2024-09-11T01:46:30.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.132+0000 7f44d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:30.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.132+0000 7f44d7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:46:30.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.132+0000 7f44d7fff640 10 monclient: dump: 2024-09-11T01:46:30.146 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:30.146 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:30.146 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.146 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.146 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:30.146 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:30.147 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:30.147 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:30.147 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:30.147 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:30.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.132+0000 7f44d7fff640 10 monclient: _finish_auth 0 2024-09-11T01:46:30.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.132+0000 7f44d7fff640 10 monclient: _check_auth_tickets 2024-09-11T01:46:30.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.132+0000 7f44d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:30.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.132+0000 7f44f5c5a640 5 monclient: authenticate success, global_id 194118 2024-09-11T01:46:30.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.132+0000 7f44d7fff640 1 -- 172.21.15.12:0/1034756864 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44e4005810 con 0x7f44f009e0b0 2024-09-11T01:46:30.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.132+0000 7f44d7fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:30.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.132+0000 7f44d7fff640 1 -- 172.21.15.12:0/1034756864 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e4005c00 con 0x7f44f009e0b0 2024-09-11T01:46:30.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.132+0000 7f44d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:30.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.132+0000 7f44d7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:46:30.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.132+0000 7f44d7fff640 10 monclient: dump: 2024-09-11T01:46:30.148 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:30.148 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:30.148 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.148 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.148 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:30.148 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:30.148 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:30.148 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:30.148 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:30.148 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:30.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.132+0000 7f44f5c5a640 10 monclient: _renew_subs 2024-09-11T01:46:30.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.132+0000 7f44f5c5a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:46:30.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.132+0000 7f44f5c5a640 1 -- 172.21.15.12:0/1034756864 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44f0130f30 con 0x7f44f009e0b0 2024-09-11T01:46:30.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.132+0000 7f44edffb640 4 set_mon_vals no callback set 2024-09-11T01:46:30.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.132+0000 7f44f5c5a640 10 monclient: _renew_subs 2024-09-11T01:46:30.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.132+0000 7f44f5c5a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:46:30.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.132+0000 7f44f5c5a640 1 -- 172.21.15.12:0/1034756864 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44f01329f0 con 0x7f44f009e0b0 2024-09-11T01:46:30.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.133+0000 7f44d7fff640 1 -- 172.21.15.12:0/1034756864 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f44e401b3b0 con 0x7f44f009e0b0 2024-09-11T01:46:30.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.133+0000 7f44f5c5a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:30.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.133+0000 7f44f5c5a640 1 --2- 172.21.15.12:0/1034756864 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44f012fb90 0x7f44f012ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:30.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.133+0000 7f44f5c5a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:30.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.133+0000 7f44f5c5a640 1 -- 172.21.15.12:0/1034756864 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f44f0131b80 con 0x7f44f012fb90 2024-09-11T01:46:30.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.133+0000 7f44d7fff640 1 --2- 172.21.15.12:0/1034756864 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f44dc03e210 0x7f44dc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:30.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.133+0000 7f44ef7fe640 1 --2- 172.21.15.12:0/1034756864 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44f012fb90 0x7f44f012ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:30.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.133+0000 7f44ef7fe640 10 monclient: get_auth_request con 0x7f44f012fb90 auth_method 0 2024-09-11T01:46:30.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.133+0000 7f44ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:30.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.133+0000 7f44ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:30.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.133+0000 7f44ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:30.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.133+0000 7f44d7fff640 1 -- 172.21.15.12:0/1034756864 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(595..595 src has 507..595) ==== 4803+0+0 (secure 0 0 0) 0x7f44e4052560 con 0x7f44f009e0b0 2024-09-11T01:46:30.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.133+0000 7f44ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:30.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.133+0000 7f44ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:30.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.133+0000 7f44ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:30.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.133+0000 7f44ef7fe640 10 monclient(hunting): handle_auth_done global_id 184205 payload 824 2024-09-11T01:46:30.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.133+0000 7f44ef7fe640 1 -- 172.21.15.12:0/1034756864 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f44f00a1d30 con 0x7f44f012fb90 2024-09-11T01:46:30.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.133+0000 7f44effff640 1 --2- 172.21.15.12:0/1034756864 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f44dc03e210 0x7f44dc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:30.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.133+0000 7f44effff640 10 monclient: get_auth_request con 0x7f44dc03e210 auth_method 0 2024-09-11T01:46:30.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.136+0000 7f44ef7fe640 1 -- 172.21.15.12:0/1034756864 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44f012fb90 msgr2=0x7f44f012ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:30.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.136+0000 7f44ef7fe640 1 --2- 172.21.15.12:0/1034756864 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44f012fb90 0x7f44f012ff70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f44e000f770 tx=0x7f44e000fc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:30.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.136+0000 7f44d7fff640 1 -- 172.21.15.12:0/1034756864 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e0018030 con 0x7f44f012fb90 2024-09-11T01:46:30.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.136+0000 7f44d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:30.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.136+0000 7f44d7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:30.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.136+0000 7f44d7fff640 10 monclient: dump: 2024-09-11T01:46:30.151 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:30.151 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:30.151 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.151 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.151 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:30.151 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:30.151 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:30.151 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:30.151 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:30.151 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:30.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.136+0000 7f44d7fff640 1 -- 172.21.15.12:0/1034756864 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f44e000fe40 con 0x7f44f012fb90 2024-09-11T01:46:30.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.136+0000 7f44d7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:30.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.136+0000 7f44d7fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:46:30.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.136+0000 7f44d7fff640 1 -- 172.21.15.12:0/1034756864 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44f012fb90 msgr2=0x7f44f012ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:30.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.136+0000 7f44d7fff640 1 --2- 172.21.15.12:0/1034756864 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44f012fb90 0x7f44f012ff70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f44e000f770 tx=0x7f44e000fc30 comp rx=0 tx=0).stop 2024-09-11T01:46:30.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.136+0000 7f44d7fff640 1 -- 172.21.15.12:0/1034756864 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44f012fb90 msgr2=0x7f44f012ff70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:46:30.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.136+0000 7f44d7fff640 1 --2- 172.21.15.12:0/1034756864 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44f012fb90 0x7f44f012ff70 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:30.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.136+0000 7f44effff640 1 --2- 172.21.15.12:0/1034756864 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f44dc03e210 0x7f44dc0406d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f44f006dbf0 tx=0x7f44e8009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:30.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.142+0000 7f44f5c5a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:46:30.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.142+0000 7f44f5c5a640 1 --2- 172.21.15.12:0/1034756864 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44f01304b0 0x7f44f0071740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:30.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.142+0000 7f44f5c5a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:30.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.142+0000 7f44f5c5a640 1 -- 172.21.15.12:0/1034756864 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f44f013c210 con 0x7f44f01304b0 2024-09-11T01:46:30.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.142+0000 7f44ef7fe640 1 --2- 172.21.15.12:0/1034756864 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44f01304b0 0x7f44f0071740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:30.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.142+0000 7f44ef7fe640 10 monclient: get_auth_request con 0x7f44f01304b0 auth_method 0 2024-09-11T01:46:30.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.142+0000 7f44ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:30.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.142+0000 7f44ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:30.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.142+0000 7f44ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:30.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.142+0000 7f44ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:30.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.142+0000 7f44ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:30.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.142+0000 7f44ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:30.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.142+0000 7f44ef7fe640 10 monclient(hunting): handle_auth_done global_id 184208 payload 824 2024-09-11T01:46:30.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.142+0000 7f44ef7fe640 1 -- 172.21.15.12:0/1034756864 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f44f01308c0 con 0x7f44f01304b0 2024-09-11T01:46:30.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.145+0000 7f44ef7fe640 1 -- 172.21.15.12:0/1034756864 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44f01304b0 msgr2=0x7f44f0071740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:30.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.145+0000 7f44ef7fe640 1 --2- 172.21.15.12:0/1034756864 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44f01304b0 0x7f44f0071740 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f44e000f770 tx=0x7f44e0003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:30.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.145+0000 7f44d7fff640 1 -- 172.21.15.12:0/1034756864 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e00075e0 con 0x7f44f01304b0 2024-09-11T01:46:30.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.145+0000 7f44d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:30.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.145+0000 7f44d7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:30.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.145+0000 7f44d7fff640 10 monclient: dump: 2024-09-11T01:46:30.154 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:30.154 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:30.155 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.155 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.155 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:30.155 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:30.155 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:30.155 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:30.155 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:30.155 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:30.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.145+0000 7f44d7fff640 1 -- 172.21.15.12:0/1034756864 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7f44e00075e0 con 0x7f44f01304b0 2024-09-11T01:46:30.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.145+0000 7f44d7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:46:30.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.145+0000 7f44d7fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:46:30.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.145+0000 7f44d7fff640 1 -- 172.21.15.12:0/1034756864 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44f01304b0 msgr2=0x7f44f0071740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:30.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.145+0000 7f44d7fff640 1 --2- 172.21.15.12:0/1034756864 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44f01304b0 0x7f44f0071740 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f44e000f770 tx=0x7f44e0003760 comp rx=0 tx=0).stop 2024-09-11T01:46:30.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.145+0000 7f44d7fff640 1 -- 172.21.15.12:0/1034756864 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44f01304b0 msgr2=0x7f44f0071740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:46:30.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.145+0000 7f44d7fff640 1 --2- 172.21.15.12:0/1034756864 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f44f01304b0 0x7f44f0071740 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f44e000f770 tx=0x7f44e0003760 comp rx=0 tx=0).stop 2024-09-11T01:46:30.156 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:46:30.156 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:46:30.156 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:46:30.156 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:46:30.156 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 162, 2024-09-11T01:46:30.156 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 50739, 2024-09-11T01:46:30.156 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:46:30.156 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:46:30.156 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:46:30.157 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:46:30.157 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:30.157 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:46:30.157 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:46:30.157 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:46:30.157 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:46:30.157 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:30.157 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:30.157 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:30.157 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:30.157 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:30.157 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:30.157 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:30.158 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:30.158 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:30.158 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:30.158 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:30.158 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:30.158 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:46:30.158 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:46:30.158 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:30.158 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:30.158 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:30.158 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:30.158 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:30.158 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:30.159 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:30.159 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:30.159 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:30.159 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:30.159 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:30.159 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:30.159 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.159 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:46:30.159 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:46:30.159 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:46:30.159 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:46:30.159 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:46:30.159 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:46:30.160 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:46:30.160 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:46:30.160 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:46:30.160 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:46:30.160 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:46:30.160 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:46:30.160 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:46:30.160 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:46:30.160 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:46:30.160 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:46:30.160 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:46:30.160 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:30.160 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:30.160 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:30.161 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:30.161 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:30.161 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:30.161 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:30.161 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:30.161 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:30.161 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:30.161 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:30.161 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:30.161 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:46:30.161 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.161 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:46:30.161 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.162 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:46:30.162 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:46:30.162 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:30.162 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:30.162 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.162 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:30.162 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:46:30.162 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:30.162 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.162 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.162 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:30.162 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:46:30.162 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:30.162 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:30.162 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:30.163 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.163 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:46:30.163 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:46:30.163 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:30.163 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:30.163 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:30.163 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.163 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.163 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:46:30.163 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:46:30.163 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:30.163 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:30.163 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.163 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:30.164 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:46:30.164 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:30.164 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.164 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.164 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:30.164 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:46:30.164 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:30.164 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:30.164 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:30.164 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.164 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:46:30.164 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:46:30.164 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:30.164 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:30.165 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:30.165 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.165 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.165 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:46:30.165 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:46:30.165 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:30.165 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:30.165 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.165 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:30.165 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:46:30.165 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:30.165 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.165 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.165 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:30.165 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:46:30.166 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:30.166 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:30.166 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:30.166 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.166 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:46:30.166 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:46:30.166 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:30.166 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:30.166 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:30.166 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:30.166 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:30.166 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.166 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:46:30.166 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:46:30.166 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.167 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:30.167 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:30.167 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:46:30.167 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:30.167 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:30.167 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:46:30.167 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.167 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:30.167 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:30.167 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:46:30.167 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:30.167 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:30.167 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.167 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:46:30.167 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:46:30.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.146+0000 7f44f5c5a640 1 -- 172.21.15.12:0/1034756864 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f44dc03e210 msgr2=0x7f44dc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:30.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.146+0000 7f44f5c5a640 1 --2- 172.21.15.12:0/1034756864 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f44dc03e210 0x7f44dc0406d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f44f006dbf0 tx=0x7f44e8009000 comp rx=0 tx=0).stop 2024-09-11T01:46:30.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.146+0000 7f44f5c5a640 10 monclient: shutdown 2024-09-11T01:46:30.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.146+0000 7f44f5c5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:30.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.146+0000 7f44f5c5a640 1 -- 172.21.15.12:0/1034756864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44f009e0b0 msgr2=0x7f44f013bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:30.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.146+0000 7f44f5c5a640 1 --2- 172.21.15.12:0/1034756864 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44f009e0b0 0x7f44f013bcd0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f44e400c450 tx=0x7f44e400c910 comp rx=0 tx=0).stop 2024-09-11T01:46:30.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.146+0000 7f44effff640 1 -- 172.21.15.12:0/1034756864 reap_dead start 2024-09-11T01:46:30.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.146+0000 7f44effff640 1 -- 172.21.15.12:0/1034756864 reap_dead start 2024-09-11T01:46:30.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.146+0000 7f44f5c5a640 1 -- 172.21.15.12:0/1034756864 shutdown_connections 2024-09-11T01:46:30.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.146+0000 7f44f5c5a640 1 -- 172.21.15.12:0/1034756864 >> 172.21.15.12:0/1034756864 conn(0x7f44f00a7da0 msgr2=0x7f44f00a0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:30.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.146+0000 7f44f5c5a640 1 -- 172.21.15.12:0/1034756864 shutdown_connections 2024-09-11T01:46:30.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.146+0000 7f44f5c5a640 1 -- 172.21.15.12:0/1034756864 wait complete. 2024-09-11T01:46:30.170 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:46:30.170 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:46:30.170 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:46:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.270+0000 7ff559fc6640 10 monclient: get_monmap_and_config 2024-09-11T01:46:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.270+0000 7ff559fc6640 10 monclient: build_initial_monmap 2024-09-11T01:46:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.270+0000 7ff559fc6640 10 monclient: monmap: 2024-09-11T01:46:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:46:30.271871+0000 2024-09-11T01:46:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:46:30.271871+0000 2024-09-11T01:46:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:46:30.273 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:46:30.274 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:46:30.274 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:46:30.274 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:46:30.274 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:30.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.270+0000 7ff559fc6640 1 Processor -- start 2024-09-11T01:46:30.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.270+0000 7ff559fc6640 1 -- start start 2024-09-11T01:46:30.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.270+0000 7ff559fc6640 10 monclient: init 2024-09-11T01:46:30.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff559fc6640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:30.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff559fc6640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-11T01:46:30.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff559fc6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff554137d50 addr v1:172.21.15.71:6789/0 2024-09-11T01:46:30.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff559fc6640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff554138130 0x7ff554138580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:30.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff559fc6640 10 monclient(hunting): picked mon.noname-a con 0x7ff554138130 addr v2:172.21.15.12:3300/0 2024-09-11T01:46:30.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff559fc6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff55413d280 0x7ff55413f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:30.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff559fc6640 10 monclient(hunting): picked mon.noname-b con 0x7ff55413d280 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:30.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff559fc6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff55409d820 con 0x7ff554137d50 2024-09-11T01:46:30.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff559fc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:30.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff559fc6640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff55409c650 con 0x7ff554138130 2024-09-11T01:46:30.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff559fc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:30.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff559fc6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff55409ba20 con 0x7ff55413d280 2024-09-11T01:46:30.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff559fc6640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:30.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff559fc6640 10 monclient(hunting): authenticate will time out at 1796.714530s 2024-09-11T01:46:30.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff552ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff554138130 0x7ff554138580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:30.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff553fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff55413d280 0x7ff55413f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:30.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff552ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff554138130 0x7ff554138580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55282/0 (socket says 172.21.15.12:55282) 2024-09-11T01:46:30.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff552ffd640 1 -- 172.21.15.12:0/161702538 learned_addr learned my addr 172.21.15.12:0/161702538 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:30.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff552ffd640 10 monclient(hunting): get_auth_request con 0x7ff554138130 auth_method 0 2024-09-11T01:46:30.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff553fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff55413d280 0x7ff55413f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:43458/0 (socket says 172.21.15.12:43458) 2024-09-11T01:46:30.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff552ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:30.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff552ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:30.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff552ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:30.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff553fff640 10 monclient(hunting): get_auth_request con 0x7ff55413d280 auth_method 0 2024-09-11T01:46:30.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff553fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:30.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff553fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:30.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff553fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:30.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff552ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:30.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff552ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:30.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff552ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:30.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff553fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:30.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff553fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:30.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff553fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:30.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff552ffd640 10 monclient(hunting): handle_auth_done global_id 194121 payload 274 2024-09-11T01:46:30.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff552ffd640 1 -- 172.21.15.12:0/161702538 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff55413d280 msgr2=0x7ff55413f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:46:30.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff552ffd640 1 --2- 172.21.15.12:0/161702538 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff55413d280 0x7ff55413f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:30.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff552ffd640 1 -- 172.21.15.12:0/161702538 >> v1:172.21.15.71:6789/0 conn(0x7ff554137d50 legacy=0x7ff55409d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:46:30.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff552ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:46:30.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff552ffd640 1 monclient: found mon.noname-a 2024-09-11T01:46:30.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff552ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:30.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff552ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:46:30.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.271+0000 7ff552ffd640 1 -- 172.21.15.12:0/161702538 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff55409c370 con 0x7ff554138130 2024-09-11T01:46:30.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.274+0000 7ff552ffd640 1 -- 172.21.15.12:0/161702538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff554138130 msgr2=0x7ff554138580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:30.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.274+0000 7ff552ffd640 1 --2- 172.21.15.12:0/161702538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff554138130 0x7ff554138580 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff548009e50 tx=0x7ff54802a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=914948c8ff7045ac server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:30.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.274+0000 7ff5527fc640 1 -- 172.21.15.12:0/161702538 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff548036070 con 0x7ff554138130 2024-09-11T01:46:30.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.274+0000 7ff5527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:30.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.274+0000 7ff5527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:46:30.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.274+0000 7ff5527fc640 10 monclient: dump: 2024-09-11T01:46:30.285 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:30.285 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:30.285 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.285 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.285 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:30.285 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:30.285 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:30.285 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:30.286 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:30.286 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:30.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.274+0000 7ff5527fc640 10 monclient: _finish_auth 0 2024-09-11T01:46:30.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.274+0000 7ff5527fc640 10 monclient: _check_auth_tickets 2024-09-11T01:46:30.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.274+0000 7ff5527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:30.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.274+0000 7ff559fc6640 5 monclient: authenticate success, global_id 194121 2024-09-11T01:46:30.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.274+0000 7ff5527fc640 1 -- 172.21.15.12:0/161702538 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff548007800 con 0x7ff554138130 2024-09-11T01:46:30.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.274+0000 7ff559fc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:46:30.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.274+0000 7ff5527fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:30.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.274+0000 7ff559fc6640 10 monclient: get_monmap_and_config success 2024-09-11T01:46:30.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.274+0000 7ff559fc6640 4 set_mon_vals no callback set 2024-09-11T01:46:30.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.274+0000 7ff5527fc640 1 -- 172.21.15.12:0/161702538 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff548007bd0 con 0x7ff554138130 2024-09-11T01:46:30.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.274+0000 7ff5527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:30.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.274+0000 7ff5527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:46:30.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.274+0000 7ff5527fc640 10 monclient: dump: 2024-09-11T01:46:30.287 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:30.287 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:30.287 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.287 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.287 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:30.287 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:30.287 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:30.287 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:30.287 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:30.287 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:30.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.275+0000 7ff559fc6640 10 monclient: shutdown 2024-09-11T01:46:30.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.275+0000 7ff559fc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:30.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.275+0000 7ff559fc6640 1 -- 172.21.15.12:0/161702538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff554138130 msgr2=0x7ff554138580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:30.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.275+0000 7ff559fc6640 1 --2- 172.21.15.12:0/161702538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff554138130 0x7ff554138580 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff548009e50 tx=0x7ff54802a5f0 comp rx=0 tx=0).stop 2024-09-11T01:46:30.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.275+0000 7ff559fc6640 1 -- 172.21.15.12:0/161702538 shutdown_connections 2024-09-11T01:46:30.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.275+0000 7ff559fc6640 1 --2- 172.21.15.12:0/161702538 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff55413d280 0x7ff55413f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:30.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.275+0000 7ff559fc6640 1 --2- 172.21.15.12:0/161702538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff554138130 0x7ff554138580 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:30.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.275+0000 7ff559fc6640 1 -- 172.21.15.12:0/161702538 >> 172.21.15.12:0/161702538 conn(0x7ff5540a7020 msgr2=0x7ff5540a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:30.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.275+0000 7ff559fc6640 1 -- 172.21.15.12:0/161702538 shutdown_connections 2024-09-11T01:46:30.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.275+0000 7ff559fc6640 1 -- 172.21.15.12:0/161702538 wait complete. 2024-09-11T01:46:30.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.276+0000 7ff559fc6640 10 monclient: build_initial_monmap 2024-09-11T01:46:30.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.276+0000 7ff559fc6640 10 monclient: monmap: 2024-09-11T01:46:30.288 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:30.288 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:46:30.288 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:46:30.288 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:46:30.288 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:30.289 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:30.289 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:46:30.289 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:46:30.289 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:46:30.289 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:30.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.276+0000 7ff559fc6640 1 Processor -- start 2024-09-11T01:46:30.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.276+0000 7ff559fc6640 1 -- start start 2024-09-11T01:46:30.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.276+0000 7ff559fc6640 10 monclient: init 2024-09-11T01:46:30.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff559fc6640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:30.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff559fc6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:46:30.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff559fc6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff554137d50 0x7ff5541ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:30.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff559fc6640 10 monclient(hunting): picked mon.noname-b con 0x7ff554137d50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:46:30.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff559fc6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff554138130 0x7ff554136e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:30.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff559fc6640 10 monclient(hunting): picked mon.noname-c con 0x7ff554138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:30.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff559fc6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff55413d280 0x7ff55413a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:30.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff559fc6640 10 monclient(hunting): picked mon.noname-a con 0x7ff55413d280 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:46:30.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff559fc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:30.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff559fc6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff55406f850 con 0x7ff55413d280 2024-09-11T01:46:30.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff559fc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:30.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff559fc6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff55406f6d0 con 0x7ff554137d50 2024-09-11T01:46:30.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff559fc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:30.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff559fc6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff55413fbb0 con 0x7ff554138130 2024-09-11T01:46:30.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff559fc6640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:30.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff559fc6640 10 monclient(hunting): authenticate will time out at 1796.720096s 2024-09-11T01:46:30.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff553fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff55413d280 0x7ff55413a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:30.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff553fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff55413d280 0x7ff55413a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55298/0 (socket says 172.21.15.12:55298) 2024-09-11T01:46:30.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff553fff640 1 -- 172.21.15.12:0/2248855468 learned_addr learned my addr 172.21.15.12:0/2248855468 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:30.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff552ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff554138130 0x7ff554136e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:30.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff553fff640 10 monclient(hunting): get_auth_request con 0x7ff55413d280 auth_method 0 2024-09-11T01:46:30.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff553fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:30.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff553fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:30.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff553fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:30.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff552ffd640 10 monclient(hunting): get_auth_request con 0x7ff554138130 auth_method 0 2024-09-11T01:46:30.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff552ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:30.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff552ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:30.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff552ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:30.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff553fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:30.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff553fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:30.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff553fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:30.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff552ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:30.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff552ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:30.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff552ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:30.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff5537fe640 1 --2- 172.21.15.12:0/2248855468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff554137d50 0x7ff5541ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:30.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff5537fe640 10 monclient(hunting): get_auth_request con 0x7ff554137d50 auth_method 0 2024-09-11T01:46:30.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff5537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:30.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff5537fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:30.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff5537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:30.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff553fff640 10 monclient(hunting): handle_auth_done global_id 194124 payload 824 2024-09-11T01:46:30.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff553fff640 1 -- 172.21.15.12:0/2248855468 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff554138130 msgr2=0x7ff554136e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:30.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff553fff640 1 --2- 172.21.15.12:0/2248855468 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff554138130 0x7ff554136e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:30.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff553fff640 1 -- 172.21.15.12:0/2248855468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff554137d50 msgr2=0x7ff5541ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:30.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff553fff640 1 --2- 172.21.15.12:0/2248855468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff554137d50 0x7ff5541ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:30.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff553fff640 10 monclient: _finish_hunting 0 2024-09-11T01:46:30.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff553fff640 1 monclient: found mon.noname-a 2024-09-11T01:46:30.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff553fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:30.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff553fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:46:30.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff553fff640 1 -- 172.21.15.12:0/2248855468 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5540a1710 con 0x7ff55413d280 2024-09-11T01:46:30.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.277+0000 7ff552ffd640 1 --2- 172.21.15.12:0/2248855468 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff554138130 0x7ff554136e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:46:30.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.279+0000 7ff553fff640 1 -- 172.21.15.12:0/2248855468 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff55413d280 msgr2=0x7ff55413a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:30.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.279+0000 7ff553fff640 1 --2- 172.21.15.12:0/2248855468 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff55413d280 0x7ff55413a240 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff54c00e850 tx=0x7ff54c00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:30.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.279+0000 7ff53bfff640 1 -- 172.21.15.12:0/2248855468 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54c017070 con 0x7ff55413d280 2024-09-11T01:46:30.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.279+0000 7ff53bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:30.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.279+0000 7ff53bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:46:30.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.279+0000 7ff53bfff640 10 monclient: dump: 2024-09-11T01:46:30.296 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:30.297 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:30.297 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.297 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.297 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:30.297 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:30.297 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:30.297 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:30.297 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:30.297 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:30.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.279+0000 7ff53bfff640 10 monclient: _finish_auth 0 2024-09-11T01:46:30.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.279+0000 7ff53bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:46:30.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.279+0000 7ff53bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:30.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.279+0000 7ff53bfff640 1 -- 172.21.15.12:0/2248855468 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff54c00cb50 con 0x7ff55413d280 2024-09-11T01:46:30.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.279+0000 7ff559fc6640 5 monclient: authenticate success, global_id 194124 2024-09-11T01:46:30.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.279+0000 7ff53bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:30.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.279+0000 7ff53bfff640 1 -- 172.21.15.12:0/2248855468 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54c013470 con 0x7ff55413d280 2024-09-11T01:46:30.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.279+0000 7ff559fc6640 10 monclient: _renew_subs 2024-09-11T01:46:30.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.279+0000 7ff559fc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:46:30.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.279+0000 7ff559fc6640 1 -- 172.21.15.12:0/2248855468 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff55413bcb0 con 0x7ff55413d280 2024-09-11T01:46:30.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.279+0000 7ff53bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:30.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.280+0000 7ff53bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:46:30.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.280+0000 7ff53bfff640 10 monclient: dump: 2024-09-11T01:46:30.298 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:30.298 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:30.298 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.298 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.298 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:30.298 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:30.299 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:30.299 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:30.299 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:30.299 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:30.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.279+0000 7ff551ffb640 4 set_mon_vals no callback set 2024-09-11T01:46:30.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.280+0000 7ff559fc6640 10 monclient: _renew_subs 2024-09-11T01:46:30.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.280+0000 7ff559fc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:46:30.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.280+0000 7ff559fc6640 1 -- 172.21.15.12:0/2248855468 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff554121060 con 0x7ff55413d280 2024-09-11T01:46:30.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.281+0000 7ff559fc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:30.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.281+0000 7ff559fc6640 1 --2- 172.21.15.12:0/2248855468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5541373b0 0x7ff55413a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:30.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.281+0000 7ff559fc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:30.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.281+0000 7ff559fc6640 1 -- 172.21.15.12:0/2248855468 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff55413c8d0 con 0x7ff5541373b0 2024-09-11T01:46:30.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.281+0000 7ff53bfff640 1 -- 172.21.15.12:0/2248855468 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff54c024060 con 0x7ff55413d280 2024-09-11T01:46:30.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.281+0000 7ff53bfff640 1 --2- 172.21.15.12:0/2248855468 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff54003e210 0x7ff5400406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:30.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.281+0000 7ff53bfff640 1 -- 172.21.15.12:0/2248855468 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(595..595 src has 507..595) ==== 4803+0+0 (secure 0 0 0) 0x7ff54c052d80 con 0x7ff55413d280 2024-09-11T01:46:30.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.281+0000 7ff5537fe640 1 --2- 172.21.15.12:0/2248855468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5541373b0 0x7ff55413a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:30.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.281+0000 7ff5537fe640 10 monclient: get_auth_request con 0x7ff5541373b0 auth_method 0 2024-09-11T01:46:30.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.281+0000 7ff5537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:30.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.281+0000 7ff5537fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:30.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.281+0000 7ff5537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:30.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.281+0000 7ff5537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:30.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.281+0000 7ff5537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:30.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.281+0000 7ff5537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:30.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.281+0000 7ff552ffd640 1 --2- 172.21.15.12:0/2248855468 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff54003e210 0x7ff5400406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:30.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.281+0000 7ff552ffd640 10 monclient: get_auth_request con 0x7ff54003e210 auth_method 0 2024-09-11T01:46:30.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.281+0000 7ff5537fe640 10 monclient(hunting): handle_auth_done global_id 134638 payload 824 2024-09-11T01:46:30.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.281+0000 7ff5537fe640 1 -- 172.21.15.12:0/2248855468 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff554138580 con 0x7ff5541373b0 2024-09-11T01:46:30.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.284+0000 7ff5537fe640 1 -- 172.21.15.12:0/2248855468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5541373b0 msgr2=0x7ff55413a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:30.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.284+0000 7ff5537fe640 1 --2- 172.21.15.12:0/2248855468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5541373b0 0x7ff55413a780 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff554120e60 tx=0x7ff54400e960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:30.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.284+0000 7ff552ffd640 1 --2- 172.21.15.12:0/2248855468 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff54003e210 0x7ff5400406d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff55406d4f0 tx=0x7ff548009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:30.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.284+0000 7ff53bfff640 1 -- 172.21.15.12:0/2248855468 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff544007450 con 0x7ff5541373b0 2024-09-11T01:46:30.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.284+0000 7ff53bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:30.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.284+0000 7ff53bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:46:30.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.284+0000 7ff53bfff640 10 monclient: dump: 2024-09-11T01:46:30.303 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:30.304 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:30.304 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.304 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.304 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:30.304 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:30.304 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:30.304 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:30.304 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:30.304 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:30.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.284+0000 7ff53bfff640 1 -- 172.21.15.12:0/2248855468 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff544007450 con 0x7ff5541373b0 2024-09-11T01:46:30.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.284+0000 7ff53bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:30.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.284+0000 7ff53bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:46:30.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.284+0000 7ff53bfff640 1 -- 172.21.15.12:0/2248855468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5541373b0 msgr2=0x7ff55413a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:30.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.284+0000 7ff53bfff640 1 --2- 172.21.15.12:0/2248855468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5541373b0 0x7ff55413a780 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff554120e60 tx=0x7ff54400e960 comp rx=0 tx=0).stop 2024-09-11T01:46:30.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.284+0000 7ff53bfff640 1 -- 172.21.15.12:0/2248855468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5541373b0 msgr2=0x7ff55413a780 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:46:30.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.284+0000 7ff53bfff640 1 --2- 172.21.15.12:0/2248855468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5541373b0 0x7ff55413a780 secure :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff554120e60 tx=0x7ff54400e960 comp rx=0 tx=0).stop 2024-09-11T01:46:30.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.290+0000 7ff559fc6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:46:30.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.290+0000 7ff559fc6640 1 --2- 172.21.15.12:0/2248855468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff55413acc0 0x7ff5540a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:30.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.290+0000 7ff559fc6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:30.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.290+0000 7ff559fc6640 1 -- 172.21.15.12:0/2248855468 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff55413b0d0 con 0x7ff55413acc0 2024-09-11T01:46:30.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.291+0000 7ff5537fe640 1 --2- 172.21.15.12:0/2248855468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff55413acc0 0x7ff5540a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:30.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.291+0000 7ff5537fe640 10 monclient: get_auth_request con 0x7ff55413acc0 auth_method 0 2024-09-11T01:46:30.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.291+0000 7ff5537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:30.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.291+0000 7ff5537fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:30.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.291+0000 7ff5537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:30.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.291+0000 7ff5537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:30.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.291+0000 7ff5537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:30.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.291+0000 7ff5537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:30.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.291+0000 7ff5537fe640 10 monclient(hunting): handle_auth_done global_id 134641 payload 824 2024-09-11T01:46:30.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.291+0000 7ff5537fe640 1 -- 172.21.15.12:0/2248855468 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff55413f4f0 con 0x7ff55413acc0 2024-09-11T01:46:30.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.293+0000 7ff5537fe640 1 -- 172.21.15.12:0/2248855468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff55413acc0 msgr2=0x7ff5540a18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:30.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.293+0000 7ff5537fe640 1 --2- 172.21.15.12:0/2248855468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff55413acc0 0x7ff5540a18e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff554120e60 tx=0x7ff544003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:30.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.293+0000 7ff53bfff640 1 -- 172.21.15.12:0/2248855468 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54401eb70 con 0x7ff55413acc0 2024-09-11T01:46:30.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.293+0000 7ff53bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:30.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.293+0000 7ff53bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:46:30.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.294+0000 7ff53bfff640 10 monclient: dump: 2024-09-11T01:46:30.306 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:30.306 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:30.306 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.306 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.306 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:30.307 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:30.307 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:30.307 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:30.307 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:30.307 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:30.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.294+0000 7ff53bfff640 1 -- 172.21.15.12:0/2248855468 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7ff544006e10 con 0x7ff55413acc0 2024-09-11T01:46:30.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.294+0000 7ff53bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:46:30.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.294+0000 7ff53bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:46:30.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.294+0000 7ff53bfff640 1 -- 172.21.15.12:0/2248855468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff55413acc0 msgr2=0x7ff5540a18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:30.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.294+0000 7ff53bfff640 1 --2- 172.21.15.12:0/2248855468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff55413acc0 0x7ff5540a18e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff554120e60 tx=0x7ff544003760 comp rx=0 tx=0).stop 2024-09-11T01:46:30.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.294+0000 7ff53bfff640 1 -- 172.21.15.12:0/2248855468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff55413acc0 msgr2=0x7ff5540a18e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:46:30.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.294+0000 7ff53bfff640 1 --2- 172.21.15.12:0/2248855468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff55413acc0 0x7ff5540a18e0 secure :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff554120e60 tx=0x7ff544003760 comp rx=0 tx=0).stop 2024-09-11T01:46:30.308 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:46:30.308 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:46:30.308 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:46:30.308 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:46:30.308 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 162, 2024-09-11T01:46:30.308 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 323256, 2024-09-11T01:46:30.308 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:46:30.308 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:46:30.308 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:46:30.308 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:46:30.308 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:30.308 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:46:30.309 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:46:30.309 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:46:30.309 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:46:30.309 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:30.309 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:30.309 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:30.309 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:30.309 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:30.309 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:30.309 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:30.309 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:30.309 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:30.309 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:30.309 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:30.310 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:30.310 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:46:30.310 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:46:30.310 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:30.310 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:30.310 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:30.310 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:30.310 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:30.310 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:30.310 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:30.310 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:30.310 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:30.310 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:30.310 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:30.310 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:30.311 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.311 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:46:30.311 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:46:30.311 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:46:30.311 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:46:30.311 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:46:30.311 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:46:30.311 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:46:30.311 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:46:30.311 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:46:30.311 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:46:30.311 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:46:30.311 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:46:30.311 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:46:30.311 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:46:30.312 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:46:30.312 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:46:30.312 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:46:30.312 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:30.312 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:30.312 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:30.312 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:30.312 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:30.312 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:30.312 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:30.312 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:30.312 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:30.312 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:30.313 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:30.313 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:30.313 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:46:30.313 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.313 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:46:30.313 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.313 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:46:30.313 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:46:30.313 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:30.313 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:30.313 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.313 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:30.313 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:46:30.314 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:30.314 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.314 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.314 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:30.314 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:46:30.314 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:30.314 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:30.314 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:30.314 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.314 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:46:30.314 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:46:30.314 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:30.314 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:30.314 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:30.314 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.315 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.315 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:46:30.315 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:46:30.315 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:30.315 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:30.315 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.315 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:30.315 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:46:30.315 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:30.315 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.315 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.315 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:30.315 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:46:30.315 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:30.316 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:30.316 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:30.316 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.316 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:46:30.316 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:46:30.316 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:30.316 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:30.316 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:30.316 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.316 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.316 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:46:30.316 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:46:30.316 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:30.316 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:30.316 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.317 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:30.317 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:46:30.317 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:30.317 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.317 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.317 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:30.317 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:46:30.317 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:30.317 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:30.317 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:30.317 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.317 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:46:30.317 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:46:30.318 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:30.318 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:30.318 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:30.318 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:30.318 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:30.318 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.318 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:46:30.318 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:46:30.318 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.318 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:30.318 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:30.318 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:46:30.318 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:30.318 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:30.318 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:46:30.319 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.319 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:30.319 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:30.319 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:46:30.319 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:30.319 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:30.319 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:46:30.319 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.319 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:30.319 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:30.319 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:46:30.319 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:30.319 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:30.319 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:46:30.319 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:30.319 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:30.320 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:30.320 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:46:30.320 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:30.320 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:30.320 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:30.320 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:46:30.320 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:46:30.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.294+0000 7ff559fc6640 1 -- 172.21.15.12:0/2248855468 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff54003e210 msgr2=0x7ff5400406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:30.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.294+0000 7ff559fc6640 1 --2- 172.21.15.12:0/2248855468 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff54003e210 0x7ff5400406d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff55406d4f0 tx=0x7ff548009000 comp rx=0 tx=0).stop 2024-09-11T01:46:30.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.294+0000 7ff559fc6640 10 monclient: shutdown 2024-09-11T01:46:30.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.294+0000 7ff559fc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:30.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.294+0000 7ff559fc6640 1 -- 172.21.15.12:0/2248855468 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff55413d280 msgr2=0x7ff55413a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:30.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.294+0000 7ff559fc6640 1 --2- 172.21.15.12:0/2248855468 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff55413d280 0x7ff55413a240 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff54c00e850 tx=0x7ff54c00ed10 comp rx=0 tx=0).stop 2024-09-11T01:46:30.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.294+0000 7ff553fff640 1 -- 172.21.15.12:0/2248855468 reap_dead start 2024-09-11T01:46:30.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.294+0000 7ff553fff640 1 -- 172.21.15.12:0/2248855468 reap_dead start 2024-09-11T01:46:30.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.294+0000 7ff559fc6640 1 -- 172.21.15.12:0/2248855468 shutdown_connections 2024-09-11T01:46:30.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.294+0000 7ff559fc6640 1 -- 172.21.15.12:0/2248855468 >> 172.21.15.12:0/2248855468 conn(0x7ff5540a7020 msgr2=0x7ff55413e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:30.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.294+0000 7ff559fc6640 1 -- 172.21.15.12:0/2248855468 shutdown_connections 2024-09-11T01:46:30.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.294+0000 7ff559fc6640 1 -- 172.21.15.12:0/2248855468 wait complete. 2024-09-11T01:46:30.322 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:46:30.322 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:46:30.322 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:46:30.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.421+0000 7f7e9a208640 10 monclient: get_monmap_and_config 2024-09-11T01:46:30.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.421+0000 7f7e9a208640 10 monclient: build_initial_monmap 2024-09-11T01:46:30.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.421+0000 7f7e9a208640 10 monclient: monmap: 2024-09-11T01:46:30.424 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:30.424 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:30.424 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:46:30.423599+0000 2024-09-11T01:46:30.425 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:46:30.423599+0000 2024-09-11T01:46:30.425 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:30.425 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:30.425 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:46:30.425 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:46:30.425 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:46:30.425 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:46:30.425 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:46:30.425 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:30.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.422+0000 7f7e9a208640 1 Processor -- start 2024-09-11T01:46:30.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.422+0000 7f7e9a208640 1 -- start start 2024-09-11T01:46:30.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.422+0000 7f7e9a208640 10 monclient: init 2024-09-11T01:46:30.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e9a208640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:30.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e9a208640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-11T01:46:30.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e9a208640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7e94137e50 0x7f7e9409d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:30.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e9a208640 10 monclient(hunting): picked mon.noname-c con 0x7f7e94137e50 addr v2:172.21.15.71:3300/0 2024-09-11T01:46:30.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e9a208640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7e94138230 addr v1:172.21.15.71:6789/0 2024-09-11T01:46:30.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e9a208640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7e9413d380 0x7f7e9413f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:30.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e9a208640 10 monclient(hunting): picked mon.noname-a con 0x7f7e9413d380 addr v2:172.21.15.12:3300/0 2024-09-11T01:46:30.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e9a208640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e940a62d0 con 0x7f7e94138230 2024-09-11T01:46:30.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e9a208640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:30.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e9a208640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f7e9409bb40 con 0x7f7e9413d380 2024-09-11T01:46:30.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e9a208640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:30.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e9a208640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f7e9409c4a0 con 0x7f7e94137e50 2024-09-11T01:46:30.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e9a208640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:30.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e9a208640 10 monclient(hunting): authenticate will time out at 1796.866278s 2024-09-11T01:46:30.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e93fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7e9413d380 0x7f7e9413f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e93fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7e9413d380 0x7f7e9413f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55312/0 (socket says 172.21.15.12:55312) 2024-09-11T01:46:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e93fff640 1 -- 172.21.15.12:0/535121342 learned_addr learned my addr 172.21.15.12:0/535121342 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e93fff640 10 monclient(hunting): get_auth_request con 0x7f7e9413d380 auth_method 0 2024-09-11T01:46:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e93fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e937fe640 1 --2- 172.21.15.12:0/535121342 >> v2:172.21.15.71:3300/0 conn(0x7f7e94137e50 0x7f7e9409d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e937fe640 10 monclient(hunting): get_auth_request con 0x7f7e94137e50 auth_method 0 2024-09-11T01:46:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e937fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:30.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:30.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:30.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:30.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e93fff640 10 monclient(hunting): handle_auth_done global_id 194127 payload 274 2024-09-11T01:46:30.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e93fff640 1 -- 172.21.15.12:0/535121342 >> v2:172.21.15.71:3300/0 conn(0x7f7e94137e50 msgr2=0x7f7e9409d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:46:30.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e93fff640 1 --2- 172.21.15.12:0/535121342 >> v2:172.21.15.71:3300/0 conn(0x7f7e94137e50 0x7f7e9409d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:30.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e93fff640 1 -- 172.21.15.12:0/535121342 >> v1:172.21.15.71:6789/0 conn(0x7f7e94138230 legacy=0x7f7e94138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:46:30.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e93fff640 10 monclient: _finish_hunting 0 2024-09-11T01:46:30.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e93fff640 1 monclient: found mon.noname-a 2024-09-11T01:46:30.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:30.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e93fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:46:30.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.423+0000 7f7e93fff640 1 -- 172.21.15.12:0/535121342 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e9409c780 con 0x7f7e9413d380 2024-09-11T01:46:30.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.426+0000 7f7e93fff640 1 -- 172.21.15.12:0/535121342 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e9413d380 msgr2=0x7f7e9413f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:30.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.426+0000 7f7e93fff640 1 --2- 172.21.15.12:0/535121342 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e9413d380 0x7f7e9413f770 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7e94138d80 tx=0x7f7e8c02a540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dc2acba18aa7220 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:30.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.426+0000 7f7e927fc640 1 -- 172.21.15.12:0/535121342 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c036070 con 0x7f7e9413d380 2024-09-11T01:46:30.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.426+0000 7f7e927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:30.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.426+0000 7f7e927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:46:30.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.426+0000 7f7e927fc640 10 monclient: dump: 2024-09-11T01:46:30.436 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:30.436 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:30.436 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.436 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.436 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:30.436 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:30.436 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:30.436 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:30.436 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:30.437 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:30.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.426+0000 7f7e927fc640 10 monclient: _finish_auth 0 2024-09-11T01:46:30.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.426+0000 7f7e927fc640 10 monclient: _check_auth_tickets 2024-09-11T01:46:30.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.426+0000 7f7e927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:30.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.426+0000 7f7e927fc640 1 -- 172.21.15.12:0/535121342 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e8c007800 con 0x7f7e9413d380 2024-09-11T01:46:30.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.426+0000 7f7e927fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:30.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.426+0000 7f7e927fc640 1 -- 172.21.15.12:0/535121342 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c007bd0 con 0x7f7e9413d380 2024-09-11T01:46:30.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.426+0000 7f7e9a208640 5 monclient: authenticate success, global_id 194127 2024-09-11T01:46:30.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.426+0000 7f7e9a208640 10 monclient: get_monmap_and_config success 2024-09-11T01:46:30.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.426+0000 7f7e9a208640 4 set_mon_vals no callback set 2024-09-11T01:46:30.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.426+0000 7f7e927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:30.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.426+0000 7f7e927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:46:30.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.426+0000 7f7e927fc640 10 monclient: dump: 2024-09-11T01:46:30.438 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:30.438 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:30.438 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.438 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.438 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:30.438 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:30.438 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:30.438 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:30.438 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:30.438 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:30.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.427+0000 7f7e9a208640 10 monclient: shutdown 2024-09-11T01:46:30.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.427+0000 7f7e9a208640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:30.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.427+0000 7f7e9a208640 1 -- 172.21.15.12:0/535121342 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e9413d380 msgr2=0x7f7e9413f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:30.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.427+0000 7f7e9a208640 1 --2- 172.21.15.12:0/535121342 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e9413d380 0x7f7e9413f770 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7e94138d80 tx=0x7f7e8c02a540 comp rx=0 tx=0).stop 2024-09-11T01:46:30.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.427+0000 7f7e9a208640 1 -- 172.21.15.12:0/535121342 shutdown_connections 2024-09-11T01:46:30.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.427+0000 7f7e9a208640 1 --2- 172.21.15.12:0/535121342 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e9413d380 0x7f7e9413f770 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:30.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.427+0000 7f7e9a208640 1 --2- 172.21.15.12:0/535121342 >> v2:172.21.15.71:3300/0 conn(0x7f7e94137e50 0x7f7e9409d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:30.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.427+0000 7f7e9a208640 1 -- 172.21.15.12:0/535121342 >> 172.21.15.12:0/535121342 conn(0x7f7e940a7020 msgr2=0x7f7e940a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:30.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.427+0000 7f7e9a208640 1 -- 172.21.15.12:0/535121342 shutdown_connections 2024-09-11T01:46:30.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.427+0000 7f7e9a208640 1 -- 172.21.15.12:0/535121342 wait complete. 2024-09-11T01:46:30.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.427+0000 7f7e9a208640 10 monclient: build_initial_monmap 2024-09-11T01:46:30.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.427+0000 7f7e9a208640 10 monclient: monmap: 2024-09-11T01:46:30.439 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:30.439 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:46:30.439 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:46:30.439 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:46:30.439 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:30.439 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:46:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:46:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:46:30.440 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.428+0000 7f7e9a208640 1 Processor -- start 2024-09-11T01:46:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.428+0000 7f7e9a208640 1 -- start start 2024-09-11T01:46:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.428+0000 7f7e9a208640 10 monclient: init 2024-09-11T01:46:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.428+0000 7f7e9a208640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.428+0000 7f7e9a208640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:46:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.428+0000 7f7e9a208640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e94137e50 0x7f7e941cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.428+0000 7f7e9a208640 10 monclient(hunting): picked mon.noname-a con 0x7f7e94137e50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:46:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.428+0000 7f7e9a208640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e94138230 0x7f7e941370c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:30.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.428+0000 7f7e9a208640 10 monclient(hunting): picked mon.noname-c con 0x7f7e94138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.428+0000 7f7e9a208640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e9413d380 0x7f7e9413a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.428+0000 7f7e9a208640 10 monclient(hunting): picked mon.noname-b con 0x7f7e9413d380 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:46:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.428+0000 7f7e9a208640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.428+0000 7f7e9a208640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7e9406f890 con 0x7f7e94137e50 2024-09-11T01:46:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.428+0000 7f7e9a208640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.428+0000 7f7e9a208640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f7e9406f710 con 0x7f7e9413d380 2024-09-11T01:46:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.428+0000 7f7e9a208640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.428+0000 7f7e9a208640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7e9413fcb0 con 0x7f7e94138230 2024-09-11T01:46:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.428+0000 7f7e9a208640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.428+0000 7f7e9a208640 10 monclient(hunting): authenticate will time out at 1796.871918s 2024-09-11T01:46:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.428+0000 7f7e937fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e94137e50 0x7f7e941cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e937fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e94137e50 0x7f7e941cd020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55320/0 (socket says 172.21.15.12:55320) 2024-09-11T01:46:30.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e937fe640 1 -- 172.21.15.12:0/2748227703 learned_addr learned my addr 172.21.15.12:0/2748227703 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:30.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e937fe640 10 monclient(hunting): get_auth_request con 0x7f7e94137e50 auth_method 0 2024-09-11T01:46:30.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:30.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e937fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:30.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:30.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e92ffd640 1 --2- 172.21.15.12:0/2748227703 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e94138230 0x7f7e941370c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:30.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e92ffd640 10 monclient(hunting): get_auth_request con 0x7f7e94138230 auth_method 0 2024-09-11T01:46:30.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e92ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:30.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e92ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:30.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:30.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:30.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:30.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:30.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e92ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:30.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e92ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:30.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e92ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:30.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e93fff640 1 --2- 172.21.15.12:0/2748227703 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e9413d380 0x7f7e9413a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:30.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e937fe640 10 monclient(hunting): handle_auth_done global_id 194130 payload 824 2024-09-11T01:46:30.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e937fe640 1 -- 172.21.15.12:0/2748227703 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e94138230 msgr2=0x7f7e941370c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:30.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e937fe640 1 --2- 172.21.15.12:0/2748227703 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e94138230 0x7f7e941370c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:30.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e937fe640 1 -- 172.21.15.12:0/2748227703 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e9413d380 msgr2=0x7f7e9413a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:30.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e937fe640 1 --2- 172.21.15.12:0/2748227703 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e9413d380 0x7f7e9413a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:30.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e937fe640 10 monclient: _finish_hunting 0 2024-09-11T01:46:30.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e937fe640 1 monclient: found mon.noname-a 2024-09-11T01:46:30.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:30.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e937fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:46:30.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e937fe640 1 -- 172.21.15.12:0/2748227703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e94121770 con 0x7f7e94137e50 2024-09-11T01:46:30.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e93fff640 10 monclient: get_auth_request con 0x7f7e9413d380 auth_method 0 2024-09-11T01:46:30.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e92ffd640 1 --2- 172.21.15.12:0/2748227703 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e94138230 0x7f7e941370c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:46:30.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.429+0000 7f7e93fff640 1 --2- 172.21.15.12:0/2748227703 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e9413d380 0x7f7e9413a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:46:30.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.431+0000 7f7e937fe640 1 -- 172.21.15.12:0/2748227703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e94137e50 msgr2=0x7f7e941cd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:30.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.431+0000 7f7e937fe640 1 --2- 172.21.15.12:0/2748227703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e94137e50 0x7f7e941cd020 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7e8400e9b0 tx=0x7f7e8400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:30.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.431+0000 7f7e7bfff640 1 -- 172.21.15.12:0/2748227703 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e84017070 con 0x7f7e94137e50 2024-09-11T01:46:30.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.431+0000 7f7e7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:30.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.431+0000 7f7e7bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:46:30.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.431+0000 7f7e7bfff640 10 monclient: dump: 2024-09-11T01:46:30.444 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:30.444 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:30.444 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.444 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:30.445 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.431+0000 7f7e7bfff640 10 monclient: _finish_auth 0 2024-09-11T01:46:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.431+0000 7f7e7bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:46:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.431+0000 7f7e7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.431+0000 7f7e9a208640 5 monclient: authenticate success, global_id 194130 2024-09-11T01:46:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.431+0000 7f7e7bfff640 1 -- 172.21.15.12:0/2748227703 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e8400cbb0 con 0x7f7e94137e50 2024-09-11T01:46:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.431+0000 7f7e9a208640 10 monclient: _renew_subs 2024-09-11T01:46:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.431+0000 7f7e9a208640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:46:30.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.431+0000 7f7e9a208640 1 -- 172.21.15.12:0/2748227703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e9413ca50 con 0x7f7e94137e50 2024-09-11T01:46:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.431+0000 7f7e7bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.431+0000 7f7e7bfff640 1 -- 172.21.15.12:0/2748227703 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e84013470 con 0x7f7e94137e50 2024-09-11T01:46:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.431+0000 7f7e7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.431+0000 7f7e7bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:46:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.431+0000 7f7e7bfff640 10 monclient: dump: 2024-09-11T01:46:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:30.447 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.431+0000 7f7e9a208640 10 monclient: _renew_subs 2024-09-11T01:46:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.431+0000 7f7e9a208640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:46:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.431+0000 7f7e9a208640 1 -- 172.21.15.12:0/2748227703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e9413bef0 con 0x7f7e94137e50 2024-09-11T01:46:30.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.431+0000 7f7e91ffb640 4 set_mon_vals no callback set 2024-09-11T01:46:30.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.432+0000 7f7e7bfff640 1 -- 172.21.15.12:0/2748227703 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f7e84024060 con 0x7f7e94137e50 2024-09-11T01:46:30.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.432+0000 7f7e9a208640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:30.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.432+0000 7f7e9a208640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:30.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.432+0000 7f7e9a208640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:46:30.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.432+0000 7f7e9a208640 1 -- 172.21.15.12:0/2748227703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e940a1600 con 0x7f7e94137e50 2024-09-11T01:46:30.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.433+0000 7f7e7bfff640 1 --2- 172.21.15.12:0/2748227703 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7e8003e240 0x7f7e80040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:30.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.433+0000 7f7e7bfff640 1 -- 172.21.15.12:0/2748227703 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(595..595 src has 507..595) ==== 4803+0+0 (secure 0 0 0) 0x7f7e84052dd0 con 0x7f7e94137e50 2024-09-11T01:46:30.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.433+0000 7f7e92ffd640 1 --2- 172.21.15.12:0/2748227703 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7e8003e240 0x7f7e80040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:30.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.433+0000 7f7e92ffd640 10 monclient: get_auth_request con 0x7f7e8003e240 auth_method 0 2024-09-11T01:46:30.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.435+0000 7f7e92ffd640 1 --2- 172.21.15.12:0/2748227703 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7e8003e240 0x7f7e80040700 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7e94120ef0 tx=0x7f7e88007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:30.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.437+0000 7f7e7bfff640 1 -- 172.21.15.12:0/2748227703 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7e840517b0 con 0x7f7e94137e50 2024-09-11T01:46:30.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.437+0000 7f7e7bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:30.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.437+0000 7f7e7bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:46:30.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.695+0000 7f7e9a208640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:46:30.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.695+0000 7f7e9a208640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:46:30.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.695+0000 7f7e9a208640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:46:30.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.695+0000 7f7e9a208640 1 -- 172.21.15.12:0/2748227703 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f7e94137c10 con 0x7f7e94137e50 2024-09-11T01:46:30.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.696+0000 7f7e7bfff640 1 -- 172.21.15.12:0/2748227703 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f7e94137c10 con 0x7f7e94137e50 2024-09-11T01:46:30.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.696+0000 7f7e7bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:46:30.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.696+0000 7f7e7bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:46:30.699 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.696+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:30.699 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.696+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=12,kv=12,logm=44} crc {auth=2303436422,config=2586702516,health=3365320580,kv=3263697420,logm=3197815024}) 2024-09-11T01:46:30.699 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.696+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=12,kv=12,logm=46} crc {auth=1941267841,config=2586702516,health=3365320580,kv=3263697420,logm=1021490198}) 2024-09-11T01:46:30.699 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.696+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:30.699 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.696+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=12,kv=12,logm=44} crc {auth=2303436422,config=2586702516,health=3365320580,kv=3263697420,logm=3197815024}) 2024-09-11T01:46:30.699 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.696+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=12,logm=46} crc {auth=1481733101,config=2586702516,health=3365320580,kv=3263697420,logm=1623510692}) 2024-09-11T01:46:30.700 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.697+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:30.700 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.697+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=716360}) 2024-09-11T01:46:30.700 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.697+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1045043336}) 2024-09-11T01:46:30.700 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.697+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:30.700 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.697+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=716360}) 2024-09-11T01:46:30.700 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.697+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2554541135}) 2024-09-11T01:46:30.701 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.698+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:30.705 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.698+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3591849804}) 2024-09-11T01:46:30.705 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.698+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3026211464}) 2024-09-11T01:46:30.705 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.698+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:30.705 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.698+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3591849804}) 2024-09-11T01:46:30.705 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.698+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1277200774}) 2024-09-11T01:46:30.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.699+0000 7f7e9a208640 1 -- 172.21.15.12:0/2748227703 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7e8003e240 msgr2=0x7f7e80040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:30.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.699+0000 7f7e9a208640 1 --2- 172.21.15.12:0/2748227703 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7e8003e240 0x7f7e80040700 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7e94120ef0 tx=0x7f7e88007e30 comp rx=0 tx=0).stop 2024-09-11T01:46:30.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.699+0000 7f7e9a208640 10 monclient: shutdown 2024-09-11T01:46:30.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.699+0000 7f7e9a208640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:30.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.699+0000 7f7e9a208640 1 -- 172.21.15.12:0/2748227703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e94137e50 msgr2=0x7f7e941cd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:30.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.699+0000 7f7e9a208640 1 --2- 172.21.15.12:0/2748227703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e94137e50 0x7f7e941cd020 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7e8400e9b0 tx=0x7f7e8400ee70 comp rx=0 tx=0).stop 2024-09-11T01:46:30.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.699+0000 7f7e9a208640 1 -- 172.21.15.12:0/2748227703 shutdown_connections 2024-09-11T01:46:30.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.699+0000 7f7e9a208640 1 --2- 172.21.15.12:0/2748227703 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e9413d380 0x7f7e9413a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:30.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.699+0000 7f7e9a208640 1 --2- 172.21.15.12:0/2748227703 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e94138230 0x7f7e941370c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:30.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.699+0000 7f7e9a208640 1 --2- 172.21.15.12:0/2748227703 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7e8003e240 0x7f7e80040700 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:30.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.699+0000 7f7e9a208640 1 --2- 172.21.15.12:0/2748227703 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7e94137e50 0x7f7e941cd020 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:30.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.699+0000 7f7e9a208640 1 -- 172.21.15.12:0/2748227703 >> 172.21.15.12:0/2748227703 conn(0x7f7e940a7020 msgr2=0x7f7e940a0430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:30.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.699+0000 7f7e9a208640 1 -- 172.21.15.12:0/2748227703 shutdown_connections 2024-09-11T01:46:30.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:30.699+0000 7f7e9a208640 1 -- 172.21.15.12:0/2748227703 wait complete. 2024-09-11T01:46:30.708 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.700+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:30.708 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.700+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1456898962}) 2024-09-11T01:46:30.708 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.700+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1931794080}) 2024-09-11T01:46:30.708 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.700+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:30.708 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.700+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1456898962}) 2024-09-11T01:46:30.708 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.700+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1469696901}) 2024-09-11T01:46:30.708 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.701+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:30.708 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.701+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3326042209}) 2024-09-11T01:46:30.709 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.701+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2448129284}) 2024-09-11T01:46:30.709 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.701+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:30.709 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.701+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3326042209}) 2024-09-11T01:46:30.709 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:30.701+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3195089424}) 2024-09-11T01:46:30.712 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:46:31.004 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.001+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.004 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.001+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3988693123}) 2024-09-11T01:46:31.004 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.001+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2199685059}) 2024-09-11T01:46:31.004 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.001+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.004 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.001+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3988693123}) 2024-09-11T01:46:31.004 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.001+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=820301782}) 2024-09-11T01:46:31.030 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.028+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.028+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3662547014}) 2024-09-11T01:46:31.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.028+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3809674894}) 2024-09-11T01:46:31.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.028+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.028+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3662547014}) 2024-09-11T01:46:31.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.028+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4273523377}) 2024-09-11T01:46:31.034 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.031+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.034 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.031+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2517467936}) 2024-09-11T01:46:31.034 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.031+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=414755139}) 2024-09-11T01:46:31.034 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.031+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.034 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.031+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2517467936}) 2024-09-11T01:46:31.034 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.031+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2933565444}) 2024-09-11T01:46:31.037 INFO:tasks.rados.rados.0.smithi071.stdout:2649: done (2 left) 2024-09-11T01:46:31.037 INFO:tasks.rados.rados.0.smithi071.stdout:2650: copy_from oid 28 from oid 39 current snap is 297 2024-09-11T01:46:31.037 INFO:tasks.rados.rados.0.smithi071.stdout:2647: finishing rollback tid 1 to smithi07134777-46 2024-09-11T01:46:31.037 INFO:tasks.rados.rados.0.smithi071.stdout:2647: finishing rollback tid 2 to smithi07134777-46 2024-09-11T01:46:31.037 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 1544 (ObjNum 494 snap 132 seq_num 494) dirty exists 2024-09-11T01:46:31.038 INFO:tasks.rados.rados.0.smithi071.stdout:2648: expect (ObjNum 832 snap 254 seq_num 832) 2024-09-11T01:46:31.038 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 458197 2024-09-11T01:46:31.070 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.067+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.070 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.067+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=660475490}) 2024-09-11T01:46:31.070 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.067+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=517643108}) 2024-09-11T01:46:31.070 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.067+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.070 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.067+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=660475490}) 2024-09-11T01:46:31.070 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.067+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3827030277}) 2024-09-11T01:46:31.128 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.126+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.128 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.126+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=108592144}) 2024-09-11T01:46:31.128 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.126+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3958119202}) 2024-09-11T01:46:31.129 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.126+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.129 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.126+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=108592144}) 2024-09-11T01:46:31.129 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.126+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1342094028}) 2024-09-11T01:46:31.129 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.127+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.129 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.127+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=503499741}) 2024-09-11T01:46:31.130 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.127+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1371774130}) 2024-09-11T01:46:31.130 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.127+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.130 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.127+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=503499741}) 2024-09-11T01:46:31.130 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.127+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1860221785}) 2024-09-11T01:46:31.130 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.128+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.131 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.128+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=547633999}) 2024-09-11T01:46:31.131 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.128+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3057483982}) 2024-09-11T01:46:31.131 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.128+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.131 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.128+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=547633999}) 2024-09-11T01:46:31.131 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.128+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3201142948}) 2024-09-11T01:46:31.132 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.129+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.132 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.129+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2736655974}) 2024-09-11T01:46:31.132 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.129+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=778704117}) 2024-09-11T01:46:31.132 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.129+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.132 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.129+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2736655974}) 2024-09-11T01:46:31.132 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.129+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1430686988}) 2024-09-11T01:46:31.147 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.145+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.147 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.145+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1537056417}) 2024-09-11T01:46:31.147 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.145+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=809025134}) 2024-09-11T01:46:31.147 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.145+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.147 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.145+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1537056417}) 2024-09-11T01:46:31.147 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.145+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3050216444}) 2024-09-11T01:46:31.154 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.152+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.154 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.152+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=895467569}) 2024-09-11T01:46:31.154 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.152+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3028298828}) 2024-09-11T01:46:31.154 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.152+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.155 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.152+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=895467569}) 2024-09-11T01:46:31.155 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.152+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3420293920}) 2024-09-11T01:46:31.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.157+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.157+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1871367602}) 2024-09-11T01:46:31.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.157+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2785112578}) 2024-09-11T01:46:31.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.157+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.157+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1871367602}) 2024-09-11T01:46:31.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.157+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1460106569}) 2024-09-11T01:46:31.182 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.180+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.182 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.180+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1228466625}) 2024-09-11T01:46:31.182 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.180+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2546214301}) 2024-09-11T01:46:31.182 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.180+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.182 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.180+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1228466625}) 2024-09-11T01:46:31.183 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.180+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=811528341}) 2024-09-11T01:46:31.183 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.181+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.184 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.181+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3386227364}) 2024-09-11T01:46:31.184 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.181+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=382448613}) 2024-09-11T01:46:31.184 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.181+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.184 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.181+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3386227364}) 2024-09-11T01:46:31.184 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.181+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=380606341}) 2024-09-11T01:46:31.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.182+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.182+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2491810018}) 2024-09-11T01:46:31.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.182+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=735535163}) 2024-09-11T01:46:31.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.182+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.182+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2491810018}) 2024-09-11T01:46:31.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.182+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4011496053}) 2024-09-11T01:46:31.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.182+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.182+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2936902631}) 2024-09-11T01:46:31.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.182+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=914890351}) 2024-09-11T01:46:31.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.182+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.182+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2936902631}) 2024-09-11T01:46:31.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.182+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1178096119}) 2024-09-11T01:46:31.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.183+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.183+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=180567749}) 2024-09-11T01:46:31.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.183+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=407613080}) 2024-09-11T01:46:31.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.183+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.183+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=180567749}) 2024-09-11T01:46:31.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.183+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2651802851}) 2024-09-11T01:46:31.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.184+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.184+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1485239615}) 2024-09-11T01:46:31.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.184+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=196339230}) 2024-09-11T01:46:31.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.184+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.184+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1485239615}) 2024-09-11T01:46:31.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.184+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=56470948}) 2024-09-11T01:46:31.189 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.185+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.189 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.185+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=9730839}) 2024-09-11T01:46:31.189 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.185+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=597799629}) 2024-09-11T01:46:31.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.185+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.186+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=9730839}) 2024-09-11T01:46:31.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.186+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3846313302}) 2024-09-11T01:46:31.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.187+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.187+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3460118955}) 2024-09-11T01:46:31.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.187+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=844489407}) 2024-09-11T01:46:31.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.187+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.187+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3460118955}) 2024-09-11T01:46:31.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.187+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=847548137}) 2024-09-11T01:46:31.199 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 405817 2024-09-11T01:46:31.275 INFO:tasks.rados.rados.0.smithi071.stdout:2650: finishing copy_from to smithi07134777-28 2024-09-11T01:46:31.275 INFO:tasks.rados.rados.0.smithi071.stdout:2650: got expected ENOENT (src dne) 2024-09-11T01:46:31.275 INFO:tasks.rados.rados.0.smithi071.stdout:2650: finishing copy_from racing read to smithi07134777-28 2024-09-11T01:46:31.275 INFO:tasks.rados.rados.0.smithi071.stdout:2647: done (2 left) 2024-09-11T01:46:31.275 INFO:tasks.rados.rados.0.smithi071.stdout:2648: done (1 left) 2024-09-11T01:46:31.276 INFO:tasks.rados.rados.0.smithi071.stdout:2650: done (0 left) 2024-09-11T01:46:31.276 INFO:tasks.rados.rados.0.smithi071.stdout:2651: read oid 40 snap -1 2024-09-11T01:46:31.276 INFO:tasks.rados.rados.0.smithi071.stdout:2651: expect deleted 2024-09-11T01:46:31.276 INFO:tasks.rados.rados.0.smithi071.stdout:2652: read oid 41 snap -1 2024-09-11T01:46:31.276 INFO:tasks.rados.rados.0.smithi071.stdout:2652: expect deleted 2024-09-11T01:46:31.276 INFO:tasks.rados.rados.0.smithi071.stdout:2653: write oid 13 current snap is 297 2024-09-11T01:46:31.276 INFO:tasks.rados.rados.0.smithi071.stdout:2653: seq_num 944 ranges {665808=662828,1957237=677648,3074011=609486} 2024-09-11T01:46:31.282 INFO:tasks.rados.rados.0.smithi071.stdout:2653: writing smithi07134777-13 from 665808 to 1328636 tid 1 2024-09-11T01:46:31.289 INFO:tasks.rados.rados.0.smithi071.stdout:2653: writing smithi07134777-13 from 1957237 to 2634885 tid 2 2024-09-11T01:46:31.295 INFO:tasks.rados.rados.0.smithi071.stdout:2653: writing smithi07134777-13 from 3074011 to 3683497 tid 3 2024-09-11T01:46:31.295 INFO:tasks.rados.rados.0.smithi071.stdout:2654: delete oid 15 current snap is 297 2024-09-11T01:46:31.295 INFO:tasks.rados.rados.0.smithi071.stdout:2653: finishing write tid 1 to smithi07134777-13 2024-09-11T01:46:31.295 INFO:tasks.rados.rados.0.smithi071.stdout:2653: finishing write tid 2 to smithi07134777-13 2024-09-11T01:46:31.304 INFO:tasks.rados.rados.0.smithi071.stdout:2653: finishing write tid 3 to smithi07134777-13 2024-09-11T01:46:31.304 INFO:tasks.rados.rados.0.smithi071.stdout:2651: done (3 left) 2024-09-11T01:46:31.304 INFO:tasks.rados.rados.0.smithi071.stdout:2652: done (2 left) 2024-09-11T01:46:31.304 INFO:tasks.rados.rados.0.smithi071.stdout:2654: done (1 left) 2024-09-11T01:46:31.304 INFO:tasks.rados.rados.0.smithi071.stdout:2655: write oid 50 current snap is 297 2024-09-11T01:46:31.304 INFO:tasks.rados.rados.0.smithi071.stdout:2655: seq_num 946 ranges {699422=413326,1557848=681317,3012504=748383} 2024-09-11T01:46:31.310 INFO:tasks.rados.rados.0.smithi071.stdout:2655: writing smithi07134777-50 from 699422 to 1112748 tid 1 2024-09-11T01:46:31.316 INFO:tasks.rados.rados.0.smithi071.stdout:2655: writing smithi07134777-50 from 1557848 to 2239165 tid 2 2024-09-11T01:46:31.325 INFO:tasks.rados.rados.0.smithi071.stdout:2655: writing smithi07134777-50 from 3012504 to 3760887 tid 3 2024-09-11T01:46:31.325 INFO:tasks.rados.rados.0.smithi071.stdout:2656: snap_create 2024-09-11T01:46:31.325 INFO:tasks.rados.rados.0.smithi071.stdout:2653: finishing write tid 4 to smithi07134777-13 2024-09-11T01:46:31.325 INFO:tasks.rados.rados.0.smithi071.stdout:2653: finishing write tid 5 to smithi07134777-13 2024-09-11T01:46:31.325 INFO:tasks.rados.rados.0.smithi071.stdout:2653: oid 13 updating version 0 to 1618 2024-09-11T01:46:31.325 INFO:tasks.rados.rados.0.smithi071.stdout:2653: oid 13 updating version 1618 to 1620 2024-09-11T01:46:31.325 INFO:tasks.rados.rados.0.smithi071.stdout:2653: oid 13 version 1620 is already newer than 1617 2024-09-11T01:46:31.325 INFO:tasks.rados.rados.0.smithi071.stdout:2653: oid 13 version 1620 is already newer than 1619 2024-09-11T01:46:31.325 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 1620 (ObjNum 944 snap 297 seq_num 944) dirty exists 2024-09-11T01:46:31.325 INFO:tasks.rados.rados.0.smithi071.stdout:2653: left oid 13 (ObjNum 944 snap 297 seq_num 944) 2024-09-11T01:46:31.325 INFO:tasks.rados.rados.0.smithi071.stdout:2655: finishing write tid 1 to smithi07134777-50 2024-09-11T01:46:31.325 INFO:tasks.rados.rados.0.smithi071.stdout:2655: finishing write tid 2 to smithi07134777-50 2024-09-11T01:46:31.328 INFO:tasks.rados.rados.0.smithi071.stdout:2655: finishing write tid 3 to smithi07134777-50 2024-09-11T01:46:31.328 INFO:tasks.rados.rados.0.smithi071.stdout:2655: finishing write tid 4 to smithi07134777-50 2024-09-11T01:46:31.328 INFO:tasks.rados.rados.0.smithi071.stdout:2655: finishing write tid 5 to smithi07134777-50 2024-09-11T01:46:31.328 INFO:tasks.rados.rados.0.smithi071.stdout:2655: oid 50 updating version 0 to 1790 2024-09-11T01:46:31.328 INFO:tasks.rados.rados.0.smithi071.stdout:2655: oid 50 version 1790 is already newer than 1789 2024-09-11T01:46:31.328 INFO:tasks.rados.rados.0.smithi071.stdout:2655: oid 50 updating version 1790 to 1791 2024-09-11T01:46:31.328 INFO:tasks.rados.rados.0.smithi071.stdout:2655: oid 50 version 1791 is already newer than 1788 2024-09-11T01:46:31.329 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 1791 (ObjNum 946 snap 297 seq_num 946) dirty exists 2024-09-11T01:46:31.329 INFO:tasks.rados.rados.0.smithi071.stdout:2655: left oid 50 (ObjNum 946 snap 297 seq_num 946) 2024-09-11T01:46:31.542 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.539+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.542 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.539+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=328362224}) 2024-09-11T01:46:31.542 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.539+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3593281851}) 2024-09-11T01:46:31.542 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.539+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.542 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.539+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=328362224}) 2024-09-11T01:46:31.542 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.539+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1856641451}) 2024-09-11T01:46:31.543 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.540+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.543 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.540+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3914773206}) 2024-09-11T01:46:31.543 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.540+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1312881911}) 2024-09-11T01:46:31.543 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.540+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.543 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.540+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3914773206}) 2024-09-11T01:46:31.543 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.540+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=473072264}) 2024-09-11T01:46:31.544 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.542+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.544 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.542+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1970478848}) 2024-09-11T01:46:31.544 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.542+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=121618500}) 2024-09-11T01:46:31.544 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.542+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.544 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.542+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1970478848}) 2024-09-11T01:46:31.545 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.542+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1371029418}) 2024-09-11T01:46:31.545 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.543+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.546 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.543+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3954697982}) 2024-09-11T01:46:31.546 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.543+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2918759433}) 2024-09-11T01:46:31.546 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.543+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.546 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.543+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3954697982}) 2024-09-11T01:46:31.546 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.543+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2652326452}) 2024-09-11T01:46:31.547 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.544+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.547 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.544+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2037472587}) 2024-09-11T01:46:31.547 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.544+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3001613719}) 2024-09-11T01:46:31.547 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.544+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.547 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.544+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2037472587}) 2024-09-11T01:46:31.547 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.544+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3212641047}) 2024-09-11T01:46:31.548 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.545+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.548 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.545+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4257704021}) 2024-09-11T01:46:31.548 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.545+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4114954236}) 2024-09-11T01:46:31.548 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.545+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.548 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.545+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4257704021}) 2024-09-11T01:46:31.548 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.545+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3978209693}) 2024-09-11T01:46:31.549 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.547+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.549 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.547+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=297637188}) 2024-09-11T01:46:31.549 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.547+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=61380897}) 2024-09-11T01:46:31.549 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.547+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.549 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.547+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=297637188}) 2024-09-11T01:46:31.550 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.547+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2877944784}) 2024-09-11T01:46:31.550 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.548+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.551 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.548+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2788068172}) 2024-09-11T01:46:31.551 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.548+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2514620527}) 2024-09-11T01:46:31.551 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.548+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.551 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.548+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2788068172}) 2024-09-11T01:46:31.551 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.548+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=751226185}) 2024-09-11T01:46:31.552 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.549+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.552 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.549+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1720137810}) 2024-09-11T01:46:31.552 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.549+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1841611648}) 2024-09-11T01:46:31.552 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.549+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.552 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.549+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1720137810}) 2024-09-11T01:46:31.552 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.549+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2317161818}) 2024-09-11T01:46:31.553 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.551+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.553 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.551+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3976347644}) 2024-09-11T01:46:31.553 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.551+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2928772283}) 2024-09-11T01:46:31.553 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.551+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.554 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.551+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3976347644}) 2024-09-11T01:46:31.554 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.551+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=183898574}) 2024-09-11T01:46:31.554 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.552+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.555 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.552+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=60108116}) 2024-09-11T01:46:31.555 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.552+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1135212656}) 2024-09-11T01:46:31.555 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.552+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.555 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.552+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=60108116}) 2024-09-11T01:46:31.555 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.552+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4253247761}) 2024-09-11T01:46:31.556 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.553+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.556 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.553+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2386410899}) 2024-09-11T01:46:31.556 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.553+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1124320331}) 2024-09-11T01:46:31.556 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.553+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.556 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.553+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2386410899}) 2024-09-11T01:46:31.556 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.553+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2675984172}) 2024-09-11T01:46:31.557 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.554+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.557 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.554+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2623508056}) 2024-09-11T01:46:31.557 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.554+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4068801689}) 2024-09-11T01:46:31.557 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.554+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.557 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.554+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2623508056}) 2024-09-11T01:46:31.557 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.554+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2601362837}) 2024-09-11T01:46:31.558 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.556+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.558 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.556+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=888485759}) 2024-09-11T01:46:31.558 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.556+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=387356999}) 2024-09-11T01:46:31.558 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.556+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.558 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.556+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=888485759}) 2024-09-11T01:46:31.558 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.556+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=197330867}) 2024-09-11T01:46:31.559 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.557+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.559 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.557+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1131827353}) 2024-09-11T01:46:31.560 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.557+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4188188662}) 2024-09-11T01:46:31.560 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.557+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.560 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.557+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1131827353}) 2024-09-11T01:46:31.560 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.557+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2139630767}) 2024-09-11T01:46:31.561 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.558+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.561 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.558+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3756661369}) 2024-09-11T01:46:31.561 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.558+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1814903085}) 2024-09-11T01:46:31.561 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.558+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.561 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.558+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3756661369}) 2024-09-11T01:46:31.561 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.558+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1916746229}) 2024-09-11T01:46:31.562 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.559+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.562 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.559+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399084809}) 2024-09-11T01:46:31.562 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.559+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3503818828}) 2024-09-11T01:46:31.562 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.559+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.562 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.559+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399084809}) 2024-09-11T01:46:31.562 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.559+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=511534083}) 2024-09-11T01:46:31.563 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.561+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.563 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.561+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1885750981}) 2024-09-11T01:46:31.563 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.561+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3635800434}) 2024-09-11T01:46:31.563 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.561+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.563 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.561+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1885750981}) 2024-09-11T01:46:31.563 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.561+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1640594318}) 2024-09-11T01:46:31.564 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.562+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.565 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.562+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2972767450}) 2024-09-11T01:46:31.565 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.562+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1470512801}) 2024-09-11T01:46:31.565 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.562+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.565 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.562+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2972767450}) 2024-09-11T01:46:31.565 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.562+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1491366881}) 2024-09-11T01:46:31.565 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.563+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.566 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.563+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=168332111}) 2024-09-11T01:46:31.566 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.563+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2528911128}) 2024-09-11T01:46:31.566 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.563+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.566 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.563+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=168332111}) 2024-09-11T01:46:31.566 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.563+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=171739958}) 2024-09-11T01:46:31.567 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.564+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.567 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.564+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2792731814}) 2024-09-11T01:46:31.567 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.564+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4032842931}) 2024-09-11T01:46:31.567 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.564+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.567 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.564+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2792731814}) 2024-09-11T01:46:31.567 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.564+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2037423162}) 2024-09-11T01:46:31.568 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.566+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.568 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.566+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2429048315}) 2024-09-11T01:46:31.568 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.566+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1962967133}) 2024-09-11T01:46:31.568 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.566+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.568 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.566+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2429048315}) 2024-09-11T01:46:31.568 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.566+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1893052016}) 2024-09-11T01:46:31.569 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.567+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.569 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.567+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=178299025}) 2024-09-11T01:46:31.569 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.567+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=475307139}) 2024-09-11T01:46:31.569 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.567+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.569 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.567+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=178299025}) 2024-09-11T01:46:31.570 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.567+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=508718643}) 2024-09-11T01:46:31.570 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.568+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.570 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.568+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1752138249}) 2024-09-11T01:46:31.571 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.568+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1533502936}) 2024-09-11T01:46:31.571 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.568+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.571 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.568+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1752138249}) 2024-09-11T01:46:31.571 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.568+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3983028904}) 2024-09-11T01:46:31.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.569+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.569+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3529473475}) 2024-09-11T01:46:31.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.569+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1405785793}) 2024-09-11T01:46:31.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.569+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.569+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3529473475}) 2024-09-11T01:46:31.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.569+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1380575865}) 2024-09-11T01:46:31.573 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.570+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.573 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.570+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3722037318}) 2024-09-11T01:46:31.573 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.570+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3849716904}) 2024-09-11T01:46:31.573 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.570+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.573 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.570+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3722037318}) 2024-09-11T01:46:31.573 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.570+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1743531438}) 2024-09-11T01:46:31.574 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.572+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.574 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.572+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1759029685}) 2024-09-11T01:46:31.574 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.572+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2507221076}) 2024-09-11T01:46:31.574 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.572+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.575 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.572+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1759029685}) 2024-09-11T01:46:31.575 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.572+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=894461858}) 2024-09-11T01:46:31.575 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.573+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.575 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.573+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1717368360}) 2024-09-11T01:46:31.575 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.573+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3044781562}) 2024-09-11T01:46:31.576 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.573+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.576 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.573+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1717368360}) 2024-09-11T01:46:31.576 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.573+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3718637574}) 2024-09-11T01:46:31.576 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.574+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.576 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.574+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=899369921}) 2024-09-11T01:46:31.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.574+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2847952420}) 2024-09-11T01:46:31.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.574+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.574+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=899369921}) 2024-09-11T01:46:31.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.574+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=254773989}) 2024-09-11T01:46:31.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.575+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.575+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3517053772}) 2024-09-11T01:46:31.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.575+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2692129771}) 2024-09-11T01:46:31.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.575+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.575+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3517053772}) 2024-09-11T01:46:31.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.575+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3700082419}) 2024-09-11T01:46:31.579 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.576+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.579 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.576+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1728495205}) 2024-09-11T01:46:31.579 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.576+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1682121065}) 2024-09-11T01:46:31.579 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.576+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.579 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.576+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1728495205}) 2024-09-11T01:46:31.579 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.576+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2290116775}) 2024-09-11T01:46:31.580 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.578+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.580 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.578+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3766360285}) 2024-09-11T01:46:31.580 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.578+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2873456217}) 2024-09-11T01:46:31.580 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.578+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.580 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.578+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3766360285}) 2024-09-11T01:46:31.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.578+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1053789129}) 2024-09-11T01:46:31.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.579+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.579+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3856094998}) 2024-09-11T01:46:31.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.579+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2425480323}) 2024-09-11T01:46:31.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.579+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.582 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.579+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3856094998}) 2024-09-11T01:46:31.582 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.579+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1995698669}) 2024-09-11T01:46:31.582 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.580+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.582 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.580+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3649522029}) 2024-09-11T01:46:31.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.580+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2512282761}) 2024-09-11T01:46:31.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.580+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.580+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3649522029}) 2024-09-11T01:46:31.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.580+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=235209921}) 2024-09-11T01:46:31.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.581+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.581+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2613838532}) 2024-09-11T01:46:31.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.581+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3280181943}) 2024-09-11T01:46:31.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.581+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.581+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2613838532}) 2024-09-11T01:46:31.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.581+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3410746382}) 2024-09-11T01:46:31.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.582+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.582+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1784016816}) 2024-09-11T01:46:31.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.582+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2101351112}) 2024-09-11T01:46:31.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.582+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.582+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1784016816}) 2024-09-11T01:46:31.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.582+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=95545592}) 2024-09-11T01:46:31.586 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.584+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.586 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.584+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=33,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=19} crc {logm=2347880456,mdsmap=3760429986,mgr=4075317205,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1072816039,mon_config_key=3458328233,monmap=2933275274,nvmeofgw=598337686,osd_metadata=118787097,osd_pg_creating=10880733,osd_snap=1371421250}) 2024-09-11T01:46:31.587 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.584+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=33,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=14} crc {logm=3464657738,mdsmap=3760429986,mgr=3526296668,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2603491547,mon_config_key=3330371543,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=10880733,osd_snap=1822652526}) 2024-09-11T01:46:31.587 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.584+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.587 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.584+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=33,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=19} crc {logm=2347880456,mdsmap=3760429986,mgr=4075317205,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1072816039,mon_config_key=3458328233,monmap=2933275274,nvmeofgw=598337686,osd_metadata=118787097,osd_pg_creating=10880733,osd_snap=1371421250}) 2024-09-11T01:46:31.587 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.584+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=33,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=12} crc {logm=3007655108,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1722889106,mon_config_key=4059469443,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=10880733,osd_snap=1684411045}) 2024-09-11T01:46:31.587 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.585+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.588 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.585+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=3917262686}) 2024-09-11T01:46:31.588 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.585+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=100} crc {osd_snap=568174983}) 2024-09-11T01:46:31.588 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.585+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.588 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.585+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=3917262686}) 2024-09-11T01:46:31.588 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.585+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=100} crc {osd_snap=2481587809}) 2024-09-11T01:46:31.589 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.586+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.589 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.586+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=5,osdmap=95} crc {osd_snap=1431092450,osdmap=1939941328}) 2024-09-11T01:46:31.589 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.587+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=5,osdmap=95} crc {osd_snap=1431092450,osdmap=1478888925}) 2024-09-11T01:46:31.589 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.587+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.589 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.587+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=5,osdmap=95} crc {osd_snap=1431092450,osdmap=1939941328}) 2024-09-11T01:46:31.589 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.587+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=5,osdmap=95} crc {osd_snap=1431092450,osdmap=4121128198}) 2024-09-11T01:46:31.597 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.594+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.597 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.595+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=53} crc {osdmap=1414467499}) 2024-09-11T01:46:31.597 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.595+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=52} crc {osdmap=2524052898}) 2024-09-11T01:46:31.597 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.595+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:46:31.597 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.595+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=53} crc {osdmap=1414467499}) 2024-09-11T01:46:31.597 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:46:31.595+0000 7fe0431af640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=57} crc {osdmap=4062520619}) 2024-09-11T01:46:31.713 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:46:31.714 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:46:31.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.820+0000 7f72dd039640 10 monclient: get_monmap_and_config 2024-09-11T01:46:31.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.820+0000 7f72dd039640 10 monclient: build_initial_monmap 2024-09-11T01:46:31.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.820+0000 7f72dd039640 10 monclient: monmap: 2024-09-11T01:46:31.823 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:31.823 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:31.823 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:46:31.822328+0000 2024-09-11T01:46:31.823 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:46:31.822328+0000 2024-09-11T01:46:31.823 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:31.823 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:31.823 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:46:31.824 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:46:31.824 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:46:31.824 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:46:31.824 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:46:31.824 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:31.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.821+0000 7f72dd039640 1 Processor -- start 2024-09-11T01:46:31.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.821+0000 7f72dd039640 1 -- start start 2024-09-11T01:46:31.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.821+0000 7f72dd039640 10 monclient: init 2024-09-11T01:46:31.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.821+0000 7f72dd039640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:31.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.821+0000 7f72dd039640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-11T01:46:31.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.821+0000 7f72dd039640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f72d80a1380 0x7f72d80a1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:31.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.821+0000 7f72dd039640 10 monclient(hunting): picked mon.noname-c con 0x7f72d80a1380 addr v2:172.21.15.71:3300/0 2024-09-11T01:46:31.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.821+0000 7f72dd039640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f72d80a0c90 0x7f72d8135ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:31.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.821+0000 7f72dd039640 10 monclient(hunting): picked mon.noname-a con 0x7f72d80a0c90 addr v2:172.21.15.12:3300/0 2024-09-11T01:46:31.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.821+0000 7f72dd039640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72d8136020 0x7f72d81364d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:31.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.821+0000 7f72dd039640 10 monclient(hunting): picked mon.noname-b con 0x7f72d8136020 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:31.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.821+0000 7f72dd039640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:31.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.821+0000 7f72dd039640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f72d80a2b40 con 0x7f72d80a0c90 2024-09-11T01:46:31.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.821+0000 7f72dd039640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:31.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.821+0000 7f72dd039640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f72d809f9a0 con 0x7f72d80a1380 2024-09-11T01:46:31.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.821+0000 7f72dd039640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:31.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.821+0000 7f72dd039640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f72d80a2230 con 0x7f72d8136020 2024-09-11T01:46:31.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.821+0000 7f72dd039640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:31.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.821+0000 7f72dd039640 10 monclient(hunting): authenticate will time out at 1798.264964s 2024-09-11T01:46:31.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72cffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f72d80a0c90 0x7f72d8135ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:31.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72d759e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72d8136020 0x7f72d81364d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:31.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72cffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f72d80a0c90 0x7f72d8135ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55330/0 (socket says 172.21.15.12:55330) 2024-09-11T01:46:31.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72cffff640 1 -- 172.21.15.12:0/328512893 learned_addr learned my addr 172.21.15.12:0/328512893 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:31.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72cffff640 10 monclient(hunting): get_auth_request con 0x7f72d80a0c90 auth_method 0 2024-09-11T01:46:31.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:31.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72cffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:31.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:31.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72d759e640 10 monclient(hunting): get_auth_request con 0x7f72d8136020 auth_method 0 2024-09-11T01:46:31.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72d759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:31.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72d759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:31.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72d759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:31.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:31.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:31.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:31.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72d759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:31.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72d759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:31.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72d6d9d640 1 --2- 172.21.15.12:0/328512893 >> v2:172.21.15.71:3300/0 conn(0x7f72d80a1380 0x7f72d80a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:31.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72d759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:31.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72d6d9d640 10 monclient(hunting): get_auth_request con 0x7f72d80a1380 auth_method 0 2024-09-11T01:46:31.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72d6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:31.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72d6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:31.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72d6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:31.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72cffff640 10 monclient(hunting): handle_auth_done global_id 194133 payload 274 2024-09-11T01:46:31.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72cffff640 1 -- 172.21.15.12:0/328512893 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72d8136020 msgr2=0x7f72d81364d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:46:31.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72cffff640 1 --2- 172.21.15.12:0/328512893 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72d8136020 0x7f72d81364d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:31.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72cffff640 1 -- 172.21.15.12:0/328512893 >> v2:172.21.15.71:3300/0 conn(0x7f72d80a1380 msgr2=0x7f72d80a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:46:31.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72cffff640 1 --2- 172.21.15.12:0/328512893 >> v2:172.21.15.71:3300/0 conn(0x7f72d80a1380 0x7f72d80a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:31.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72cffff640 10 monclient: _finish_hunting 0 2024-09-11T01:46:31.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72cffff640 1 monclient: found mon.noname-a 2024-09-11T01:46:31.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:31.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72cffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:46:31.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72cffff640 1 -- 172.21.15.12:0/328512893 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d80a2510 con 0x7f72d80a0c90 2024-09-11T01:46:31.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72d6d9d640 1 --2- 172.21.15.12:0/328512893 >> v2:172.21.15.71:3300/0 conn(0x7f72d80a1380 0x7f72d80a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:46:31.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.822+0000 7f72d759e640 1 --2- 172.21.15.12:0/328512893 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72d8136020 0x7f72d81364d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:46:31.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.825+0000 7f72cffff640 1 -- 172.21.15.12:0/328512893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72d80a0c90 msgr2=0x7f72d8135ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:31.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.825+0000 7f72cffff640 1 --2- 172.21.15.12:0/328512893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72d80a0c90 0x7f72d8135ae0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f72d80a3d60 tx=0x7f72c002a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb6e116160b9ae50 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:31.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.825+0000 7f72d659c640 1 -- 172.21.15.12:0/328512893 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c0037070 con 0x7f72d80a0c90 2024-09-11T01:46:31.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.825+0000 7f72d659c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:31.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.825+0000 7f72d659c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:46:31.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.825+0000 7f72d659c640 10 monclient: dump: 2024-09-11T01:46:31.838 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:31.838 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:31.838 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:31.838 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:31.838 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:31.838 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:31.838 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:31.838 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:31.838 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:31.838 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:31.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.825+0000 7f72d659c640 10 monclient: _finish_auth 0 2024-09-11T01:46:31.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.825+0000 7f72d659c640 10 monclient: _check_auth_tickets 2024-09-11T01:46:31.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.825+0000 7f72d659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:31.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.825+0000 7f72dd039640 5 monclient: authenticate success, global_id 194133 2024-09-11T01:46:31.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.825+0000 7f72d659c640 1 -- 172.21.15.12:0/328512893 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72c0032640 con 0x7f72d80a0c90 2024-09-11T01:46:31.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.825+0000 7f72dd039640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:46:31.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.825+0000 7f72d659c640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:31.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.825+0000 7f72dd039640 10 monclient: get_monmap_and_config success 2024-09-11T01:46:31.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.825+0000 7f72dd039640 4 set_mon_vals no callback set 2024-09-11T01:46:31.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.825+0000 7f72d659c640 1 -- 172.21.15.12:0/328512893 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c0032a10 con 0x7f72d80a0c90 2024-09-11T01:46:31.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.825+0000 7f72d659c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:31.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.825+0000 7f72d659c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:46:31.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.825+0000 7f72d659c640 10 monclient: dump: 2024-09-11T01:46:31.839 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:31.839 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:31.839 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:31.839 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:31.840 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:31.840 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:31.840 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:31.840 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:31.840 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:31.840 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:31.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.825+0000 7f72dd039640 10 monclient: shutdown 2024-09-11T01:46:31.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.825+0000 7f72dd039640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:31.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.825+0000 7f72dd039640 1 -- 172.21.15.12:0/328512893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72d80a0c90 msgr2=0x7f72d8135ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:31.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.825+0000 7f72dd039640 1 --2- 172.21.15.12:0/328512893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72d80a0c90 0x7f72d8135ae0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f72d80a3d60 tx=0x7f72c002a0b0 comp rx=0 tx=0).stop 2024-09-11T01:46:31.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.826+0000 7f72dd039640 1 -- 172.21.15.12:0/328512893 shutdown_connections 2024-09-11T01:46:31.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.826+0000 7f72dd039640 1 --2- 172.21.15.12:0/328512893 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72d8136020 0x7f72d81364d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:31.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.826+0000 7f72dd039640 1 --2- 172.21.15.12:0/328512893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72d80a0c90 0x7f72d8135ae0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:31.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.826+0000 7f72dd039640 1 --2- 172.21.15.12:0/328512893 >> v2:172.21.15.71:3300/0 conn(0x7f72d80a1380 0x7f72d80a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:31.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.826+0000 7f72dd039640 1 -- 172.21.15.12:0/328512893 >> 172.21.15.12:0/328512893 conn(0x7f72d80a2d90 msgr2=0x7f72d80a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.826+0000 7f72dd039640 1 -- 172.21.15.12:0/328512893 shutdown_connections 2024-09-11T01:46:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.826+0000 7f72dd039640 1 -- 172.21.15.12:0/328512893 wait complete. 2024-09-11T01:46:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.826+0000 7f72dd039640 10 monclient: build_initial_monmap 2024-09-11T01:46:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.826+0000 7f72dd039640 10 monclient: monmap: 2024-09-11T01:46:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:46:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:46:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:46:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:46:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:46:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:46:31.841 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:31.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72dd039640 1 Processor -- start 2024-09-11T01:46:31.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72dd039640 1 -- start start 2024-09-11T01:46:31.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72dd039640 10 monclient: init 2024-09-11T01:46:31.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72dd039640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:31.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72dd039640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:46:31.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72dd039640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72d80a0c90 0x7f72d81ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:31.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72dd039640 10 monclient(hunting): picked mon.noname-a con 0x7f72d80a0c90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:46:31.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72dd039640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72d80a1380 0x7f72d8134f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:31.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72dd039640 10 monclient(hunting): picked mon.noname-c con 0x7f72d80a1380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:31.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72dd039640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72d8136020 0x7f72d81354d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:31.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72dd039640 10 monclient(hunting): picked mon.noname-b con 0x7f72d8136020 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:46:31.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72dd039640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:31.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72dd039640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f72d809ce20 con 0x7f72d80a0c90 2024-09-11T01:46:31.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72dd039640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:31.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72dd039640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f72d809cca0 con 0x7f72d8136020 2024-09-11T01:46:31.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72dd039640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:31.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72dd039640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f72d80a0390 con 0x7f72d80a1380 2024-09-11T01:46:31.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72dd039640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:31.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72dd039640 10 monclient(hunting): authenticate will time out at 1798.270784s 2024-09-11T01:46:31.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72d6d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72d80a0c90 0x7f72d81ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:31.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72cffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72d80a1380 0x7f72d8134f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:31.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72d6d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72d80a0c90 0x7f72d81ccdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55338/0 (socket says 172.21.15.12:55338) 2024-09-11T01:46:31.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72cffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72d80a1380 0x7f72d8134f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:43490/0 (socket says 172.21.15.12:43490) 2024-09-11T01:46:31.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72d6d9d640 1 -- 172.21.15.12:0/1250825197 learned_addr learned my addr 172.21.15.12:0/1250825197 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:31.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72d6d9d640 10 monclient(hunting): get_auth_request con 0x7f72d80a0c90 auth_method 0 2024-09-11T01:46:31.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72d6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:31.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72d6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:31.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72d6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:31.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72cffff640 10 monclient(hunting): get_auth_request con 0x7f72d80a1380 auth_method 0 2024-09-11T01:46:31.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:31.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72cffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:31.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.827+0000 7f72cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:31.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.828+0000 7f72d6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:31.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.828+0000 7f72d6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:31.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.828+0000 7f72d6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:31.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.828+0000 7f72cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:31.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.828+0000 7f72cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:31.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.828+0000 7f72d759e640 1 --2- 172.21.15.12:0/1250825197 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72d8136020 0x7f72d81354d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:31.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.828+0000 7f72cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:31.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.828+0000 7f72d759e640 10 monclient(hunting): get_auth_request con 0x7f72d8136020 auth_method 0 2024-09-11T01:46:31.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.828+0000 7f72d759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:31.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.828+0000 7f72d759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:31.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.828+0000 7f72d759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:31.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.828+0000 7f72d759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:31.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.828+0000 7f72d759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:31.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.828+0000 7f72d759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:31.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.828+0000 7f72d6d9d640 10 monclient(hunting): handle_auth_done global_id 194136 payload 824 2024-09-11T01:46:31.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.828+0000 7f72d6d9d640 1 -- 172.21.15.12:0/1250825197 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72d80a1380 msgr2=0x7f72d8134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:31.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.828+0000 7f72d6d9d640 1 --2- 172.21.15.12:0/1250825197 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72d80a1380 0x7f72d8134f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:31.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.828+0000 7f72d6d9d640 1 -- 172.21.15.12:0/1250825197 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72d8136020 msgr2=0x7f72d81354d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:31.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.828+0000 7f72d6d9d640 1 --2- 172.21.15.12:0/1250825197 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72d8136020 0x7f72d81354d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:31.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.828+0000 7f72d6d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:46:31.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.828+0000 7f72d6d9d640 1 monclient: found mon.noname-a 2024-09-11T01:46:31.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.828+0000 7f72d6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:31.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.828+0000 7f72d6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:46:31.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.828+0000 7f72d6d9d640 1 -- 172.21.15.12:0/1250825197 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d8127960 con 0x7f72d80a0c90 2024-09-11T01:46:31.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.828+0000 7f72cffff640 1 --2- 172.21.15.12:0/1250825197 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72d80a1380 0x7f72d8134f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:46:31.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.830+0000 7f72d6d9d640 1 -- 172.21.15.12:0/1250825197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72d80a0c90 msgr2=0x7f72d81ccdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:31.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.830+0000 7f72d6d9d640 1 --2- 172.21.15.12:0/1250825197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72d80a0c90 0x7f72d81ccdf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f72c800e730 tx=0x7f72c800eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:31.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.830+0000 7f72cf7fe640 1 -- 172.21.15.12:0/1250825197 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c800f040 con 0x7f72d80a0c90 2024-09-11T01:46:31.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.830+0000 7f72cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:31.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.830+0000 7f72cf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:46:31.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.830+0000 7f72cf7fe640 10 monclient: dump: 2024-09-11T01:46:31.846 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:31.846 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:31.846 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:31.846 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:31.846 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:31.846 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:31.846 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:31.846 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:31.846 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:31.847 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:31.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.830+0000 7f72cf7fe640 10 monclient: _finish_auth 0 2024-09-11T01:46:31.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.830+0000 7f72cf7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:46:31.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.830+0000 7f72cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:31.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.830+0000 7f72dd039640 5 monclient: authenticate success, global_id 194136 2024-09-11T01:46:31.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.830+0000 7f72cf7fe640 1 -- 172.21.15.12:0/1250825197 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72c8004230 con 0x7f72d80a0c90 2024-09-11T01:46:31.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.830+0000 7f72dd039640 10 monclient: _renew_subs 2024-09-11T01:46:31.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.830+0000 7f72dd039640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:46:31.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.830+0000 7f72dd039640 1 -- 172.21.15.12:0/1250825197 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72d806fc20 con 0x7f72d80a0c90 2024-09-11T01:46:31.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.830+0000 7f72cf7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:31.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.830+0000 7f72cf7fe640 1 -- 172.21.15.12:0/1250825197 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c800c6d0 con 0x7f72d80a0c90 2024-09-11T01:46:31.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.830+0000 7f72cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:31.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.830+0000 7f72cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:46:31.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.830+0000 7f72cf7fe640 10 monclient: dump: 2024-09-11T01:46:31.848 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:31.848 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:31.848 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:31.848 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:31.848 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:31.848 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:31.848 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:31.848 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:31.848 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:31.848 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:31.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.830+0000 7f72dd039640 10 monclient: _renew_subs 2024-09-11T01:46:31.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.830+0000 7f72dd039640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:46:31.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.830+0000 7f72d5d9b640 4 set_mon_vals no callback set 2024-09-11T01:46:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.830+0000 7f72dd039640 1 -- 172.21.15.12:0/1250825197 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72d8128410 con 0x7f72d80a0c90 2024-09-11T01:46:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.831+0000 7f72dd039640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.831+0000 7f72dd039640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.831+0000 7f72dd039640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:46:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.831+0000 7f72dd039640 1 -- 172.21.15.12:0/1250825197 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72d80a1760 con 0x7f72d80a0c90 2024-09-11T01:46:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.831+0000 7f72cf7fe640 1 -- 172.21.15.12:0/1250825197 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f72c8012050 con 0x7f72d80a0c90 2024-09-11T01:46:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.832+0000 7f72cf7fe640 1 --2- 172.21.15.12:0/1250825197 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f72b803e210 0x7f72b80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.832+0000 7f72cf7fe640 1 -- 172.21.15.12:0/1250825197 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(596..596 src has 507..596) ==== 4847+0+0 (secure 0 0 0) 0x7f72c8052880 con 0x7f72d80a0c90 2024-09-11T01:46:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.832+0000 7f72cffff640 1 --2- 172.21.15.12:0/1250825197 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f72b803e210 0x7f72b80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.832+0000 7f72cffff640 10 monclient: get_auth_request con 0x7f72b803e210 auth_method 0 2024-09-11T01:46:31.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.835+0000 7f72cffff640 1 --2- 172.21.15.12:0/1250825197 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f72b803e210 0x7f72b80406d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f72d806d7a0 tx=0x7f72c002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:31.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.837+0000 7f72cf7fe640 1 -- 172.21.15.12:0/1250825197 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f72c8017020 con 0x7f72d80a0c90 2024-09-11T01:46:31.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.837+0000 7f72cf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:31.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:31.837+0000 7f72cf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:46:32.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.092+0000 7f72dd039640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:46:32.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.092+0000 7f72dd039640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:46:32.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.092+0000 7f72dd039640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:46:32.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.092+0000 7f72dd039640 1 -- 172.21.15.12:0/1250825197 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f72d806d460 con 0x7f72d80a0c90 2024-09-11T01:46:32.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.093+0000 7f72cf7fe640 1 -- 172.21.15.12:0/1250825197 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f72c800c8b0 con 0x7f72d80a0c90 2024-09-11T01:46:32.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.093+0000 7f72cf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:46:32.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.093+0000 7f72cf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:46:32.096 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":162,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:46:32.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.095+0000 7f72dd039640 1 -- 172.21.15.12:0/1250825197 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f72b803e210 msgr2=0x7f72b80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:32.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.095+0000 7f72dd039640 1 --2- 172.21.15.12:0/1250825197 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f72b803e210 0x7f72b80406d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f72d806d7a0 tx=0x7f72c002e000 comp rx=0 tx=0).stop 2024-09-11T01:46:32.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.095+0000 7f72dd039640 10 monclient: shutdown 2024-09-11T01:46:32.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.095+0000 7f72dd039640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:32.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.095+0000 7f72dd039640 1 -- 172.21.15.12:0/1250825197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72d80a0c90 msgr2=0x7f72d81ccdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:32.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.095+0000 7f72dd039640 1 --2- 172.21.15.12:0/1250825197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72d80a0c90 0x7f72d81ccdf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f72c800e730 tx=0x7f72c800eb30 comp rx=0 tx=0).stop 2024-09-11T01:46:32.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.095+0000 7f72dd039640 1 -- 172.21.15.12:0/1250825197 shutdown_connections 2024-09-11T01:46:32.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.095+0000 7f72dd039640 1 --2- 172.21.15.12:0/1250825197 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72d8136020 0x7f72d81354d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.095+0000 7f72dd039640 1 --2- 172.21.15.12:0/1250825197 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72d80a1380 0x7f72d8134f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.095+0000 7f72dd039640 1 --2- 172.21.15.12:0/1250825197 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f72b803e210 0x7f72b80406d0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.095+0000 7f72dd039640 1 --2- 172.21.15.12:0/1250825197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72d80a0c90 0x7f72d81ccdf0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.095+0000 7f72dd039640 1 -- 172.21.15.12:0/1250825197 >> 172.21.15.12:0/1250825197 conn(0x7f72d80a2d90 msgr2=0x7f72d80a5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:32.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.095+0000 7f72dd039640 1 -- 172.21.15.12:0/1250825197 shutdown_connections 2024-09-11T01:46:32.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.095+0000 7f72dd039640 1 -- 172.21.15.12:0/1250825197 wait complete. 2024-09-11T01:46:32.108 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:46:32.108 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:46:32.108 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:46:32.108 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:46:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.207+0000 7fd7b44a9640 10 monclient: get_monmap_and_config 2024-09-11T01:46:32.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.207+0000 7fd7b44a9640 10 monclient: build_initial_monmap 2024-09-11T01:46:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.207+0000 7fd7b44a9640 10 monclient: monmap: 2024-09-11T01:46:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:46:32.209638+0000 2024-09-11T01:46:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:46:32.209638+0000 2024-09-11T01:46:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:32.211 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:46:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:46:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:46:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:46:32.212 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:46:32.212 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.208+0000 7fd7b44a9640 1 Processor -- start 2024-09-11T01:46:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.208+0000 7fd7b44a9640 1 -- start start 2024-09-11T01:46:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.208+0000 7fd7b44a9640 10 monclient: init 2024-09-11T01:46:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b44a9640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b44a9640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-11T01:46:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b44a9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd7ac1381c0 0x7fd7ac1385a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b44a9640 10 monclient(hunting): picked mon.noname-b con 0x7fd7ac1381c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b44a9640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd7ac0a6aa0 0x7fd7ac09cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b44a9640 10 monclient(hunting): picked mon.noname-c con 0x7fd7ac0a6aa0 addr v2:172.21.15.71:3300/0 2024-09-11T01:46:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b44a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd7ac0a4f20 addr v1:172.21.15.12:6789/0 2024-09-11T01:46:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b44a9640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7ac09c310 con 0x7fd7ac0a4f20 2024-09-11T01:46:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b44a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b44a9640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fd7ac06fc90 con 0x7fd7ac0a6aa0 2024-09-11T01:46:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b44a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b44a9640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd7ac09c5f0 con 0x7fd7ac1381c0 2024-09-11T01:46:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b44a9640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b44a9640 10 monclient(hunting): authenticate will time out at 1798.652295s 2024-09-11T01:46:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b2245640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd7ac1381c0 0x7fd7ac1385a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b2a46640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd7ac0a4f20 0x7fd7ac0a5390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:54232/0 (socket says 172.21.15.12:54232) 2024-09-11T01:46:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b2a46640 1 -- 172.21.15.12:0/2450284997 learned_addr learned my addr 172.21.15.12:0/2450284997 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:46:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b2245640 10 monclient(hunting): get_auth_request con 0x7fd7ac1381c0 auth_method 0 2024-09-11T01:46:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b2245640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b2245640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b2245640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b2245640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b2245640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b2245640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b1a44640 1 --2- 172.21.15.12:0/2450284997 >> v2:172.21.15.71:3300/0 conn(0x7fd7ac0a6aa0 0x7fd7ac09cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b1a44640 10 monclient(hunting): get_auth_request con 0x7fd7ac0a6aa0 auth_method 0 2024-09-11T01:46:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b1a44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b1a44640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b1a44640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b1a44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b1a44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b1a44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b2245640 10 monclient(hunting): handle_auth_done global_id 184226 payload 274 2024-09-11T01:46:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b2245640 1 -- 172.21.15.12:0/2450284997 >> v2:172.21.15.71:3300/0 conn(0x7fd7ac0a6aa0 msgr2=0x7fd7ac09cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:46:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b2245640 1 --2- 172.21.15.12:0/2450284997 >> v2:172.21.15.71:3300/0 conn(0x7fd7ac0a6aa0 0x7fd7ac09cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b2245640 1 -- 172.21.15.12:0/2450284997 >> v1:172.21.15.12:6789/0 conn(0x7fd7ac0a4f20 legacy=0x7fd7ac0a5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:46:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b2245640 10 monclient: _finish_hunting 0 2024-09-11T01:46:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b2245640 1 monclient: found mon.noname-b 2024-09-11T01:46:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b2245640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b2245640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:46:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.209+0000 7fd7b2245640 1 -- 172.21.15.12:0/2450284997 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7ac09c8d0 con 0x7fd7ac1381c0 2024-09-11T01:46:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.212+0000 7fd7b2245640 1 -- 172.21.15.12:0/2450284997 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd7ac1381c0 msgr2=0x7fd7ac1385a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.212+0000 7fd7b2245640 1 --2- 172.21.15.12:0/2450284997 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd7ac1381c0 0x7fd7ac1385a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd7a80097a0 tx=0x7fd7a802a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ac9f207953e85e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.212+0000 7fd7b1243640 1 -- 172.21.15.12:0/2450284997 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a8037070 con 0x7fd7ac1381c0 2024-09-11T01:46:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.212+0000 7fd7b1243640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.212+0000 7fd7b1243640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:46:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.212+0000 7fd7b1243640 10 monclient: dump: 2024-09-11T01:46:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:32.222 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.212+0000 7fd7b1243640 10 monclient: _finish_auth 0 2024-09-11T01:46:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.212+0000 7fd7b1243640 10 monclient: _check_auth_tickets 2024-09-11T01:46:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.212+0000 7fd7b1243640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.212+0000 7fd7b1243640 1 -- 172.21.15.12:0/2450284997 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7a8032440 con 0x7fd7ac1381c0 2024-09-11T01:46:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.212+0000 7fd7b44a9640 5 monclient: authenticate success, global_id 184226 2024-09-11T01:46:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.212+0000 7fd7b44a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:46:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.212+0000 7fd7b1243640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.212+0000 7fd7b44a9640 10 monclient: get_monmap_and_config success 2024-09-11T01:46:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.212+0000 7fd7b1243640 1 -- 172.21.15.12:0/2450284997 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a8032810 con 0x7fd7ac1381c0 2024-09-11T01:46:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.212+0000 7fd7b44a9640 4 set_mon_vals no callback set 2024-09-11T01:46:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.212+0000 7fd7b1243640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.212+0000 7fd7b1243640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.212+0000 7fd7b1243640 10 monclient: dump: 2024-09-11T01:46:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:32.224 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.213+0000 7fd7b44a9640 10 monclient: shutdown 2024-09-11T01:46:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.213+0000 7fd7b44a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:32.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.213+0000 7fd7b44a9640 1 -- 172.21.15.12:0/2450284997 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd7ac1381c0 msgr2=0x7fd7ac1385a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:32.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.213+0000 7fd7b44a9640 1 --2- 172.21.15.12:0/2450284997 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd7ac1381c0 0x7fd7ac1385a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd7a80097a0 tx=0x7fd7a802a040 comp rx=0 tx=0).stop 2024-09-11T01:46:32.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.213+0000 7fd7b44a9640 1 -- 172.21.15.12:0/2450284997 shutdown_connections 2024-09-11T01:46:32.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.213+0000 7fd7b44a9640 1 --2- 172.21.15.12:0/2450284997 >> v2:172.21.15.71:3300/0 conn(0x7fd7ac0a6aa0 0x7fd7ac09cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.213+0000 7fd7b44a9640 1 --2- 172.21.15.12:0/2450284997 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd7ac1381c0 0x7fd7ac1385a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.213+0000 7fd7b44a9640 1 -- 172.21.15.12:0/2450284997 >> 172.21.15.12:0/2450284997 conn(0x7fd7ac0a7440 msgr2=0x7fd7ac0a7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:32.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.213+0000 7fd7b44a9640 1 -- 172.21.15.12:0/2450284997 shutdown_connections 2024-09-11T01:46:32.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.213+0000 7fd7b44a9640 1 -- 172.21.15.12:0/2450284997 wait complete. 2024-09-11T01:46:32.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.213+0000 7fd7b44a9640 10 monclient: build_initial_monmap 2024-09-11T01:46:32.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.213+0000 7fd7b44a9640 10 monclient: monmap: 2024-09-11T01:46:32.225 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:32.225 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:46:32.225 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:46:32.225 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:46:32.226 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:32.226 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:32.226 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:46:32.226 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:46:32.226 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:46:32.226 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:32.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b44a9640 1 Processor -- start 2024-09-11T01:46:32.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b44a9640 1 -- start start 2024-09-11T01:46:32.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b44a9640 10 monclient: init 2024-09-11T01:46:32.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b44a9640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:32.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b44a9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:46:32.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b44a9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7ac0a4f20 0x7fd7ac135db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:32.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b44a9640 10 monclient(hunting): picked mon.noname-a con 0x7fd7ac0a4f20 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:46:32.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b44a9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd7ac0a6aa0 0x7fd7ac1362f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:32.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b44a9640 10 monclient(hunting): picked mon.noname-c con 0x7fd7ac0a6aa0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:32.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b44a9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd7ac1381c0 0x7fd7ac123990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:32.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b44a9640 10 monclient(hunting): picked mon.noname-b con 0x7fd7ac1381c0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:46:32.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b44a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:32.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b44a9640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ac09d690 con 0x7fd7ac0a4f20 2024-09-11T01:46:32.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b44a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:32.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b44a9640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ac09d510 con 0x7fd7ac1381c0 2024-09-11T01:46:32.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b44a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:32.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b44a9640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd7ac0a5aa0 con 0x7fd7ac0a6aa0 2024-09-11T01:46:32.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b44a9640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:32.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b44a9640 10 monclient(hunting): authenticate will time out at 1798.657672s 2024-09-11T01:46:32.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b2245640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7ac0a4f20 0x7fd7ac135db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:32.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b1a44640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd7ac0a6aa0 0x7fd7ac1362f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b2245640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7ac0a4f20 0x7fd7ac135db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55354/0 (socket says 172.21.15.12:55354) 2024-09-11T01:46:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b2245640 1 -- 172.21.15.12:0/106013787 learned_addr learned my addr 172.21.15.12:0/106013787 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b2245640 10 monclient(hunting): get_auth_request con 0x7fd7ac0a4f20 auth_method 0 2024-09-11T01:46:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b2245640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b2245640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b2245640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b1a44640 10 monclient(hunting): get_auth_request con 0x7fd7ac0a6aa0 auth_method 0 2024-09-11T01:46:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b1a44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b1a44640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b1a44640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b1a44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b1a44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b1a44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:32.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b2245640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:32.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b2245640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:32.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b2a46640 1 --2- 172.21.15.12:0/106013787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd7ac1381c0 0x7fd7ac123990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:32.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b2245640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:32.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b2a46640 10 monclient(hunting): get_auth_request con 0x7fd7ac1381c0 auth_method 0 2024-09-11T01:46:32.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b2a46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:32.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b2a46640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:32.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.214+0000 7fd7b2a46640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:32.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.215+0000 7fd7b2a46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:32.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.215+0000 7fd7b2a46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:32.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.215+0000 7fd7b2a46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:32.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.215+0000 7fd7b1a44640 10 monclient(hunting): handle_auth_done global_id 184229 payload 824 2024-09-11T01:46:32.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.215+0000 7fd7b1a44640 1 -- 172.21.15.12:0/106013787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd7ac1381c0 msgr2=0x7fd7ac123990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:32.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.215+0000 7fd7b1a44640 1 --2- 172.21.15.12:0/106013787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd7ac1381c0 0x7fd7ac123990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.215+0000 7fd7b1a44640 1 -- 172.21.15.12:0/106013787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7ac0a4f20 msgr2=0x7fd7ac135db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:32.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.215+0000 7fd7b1a44640 1 --2- 172.21.15.12:0/106013787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7ac0a4f20 0x7fd7ac135db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.215+0000 7fd7b1a44640 10 monclient: _finish_hunting 0 2024-09-11T01:46:32.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.215+0000 7fd7b1a44640 1 monclient: found mon.noname-c 2024-09-11T01:46:32.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.215+0000 7fd7b1a44640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:32.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.215+0000 7fd7b1a44640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:46:32.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.215+0000 7fd7b1a44640 1 -- 172.21.15.12:0/106013787 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7ac1cdbb0 con 0x7fd7ac0a6aa0 2024-09-11T01:46:32.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.215+0000 7fd7b2245640 1 --2- 172.21.15.12:0/106013787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7ac0a4f20 0x7fd7ac135db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:46:32.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.217+0000 7fd7b1a44640 1 -- 172.21.15.12:0/106013787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd7ac0a6aa0 msgr2=0x7fd7ac1362f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:32.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.217+0000 7fd7b1a44640 1 --2- 172.21.15.12:0/106013787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd7ac0a6aa0 0x7fd7ac1362f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd79c00e820 tx=0x7fd79c00ece0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:32.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.217+0000 7fd7a2ffd640 1 -- 172.21.15.12:0/106013787 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c017070 con 0x7fd7ac0a6aa0 2024-09-11T01:46:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.217+0000 7fd7a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.217+0000 7fd7a2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:46:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.217+0000 7fd7a2ffd640 10 monclient: dump: 2024-09-11T01:46:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:32.235 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.217+0000 7fd7a2ffd640 10 monclient: _finish_auth 0 2024-09-11T01:46:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.217+0000 7fd7a2ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:46:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.217+0000 7fd7a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.217+0000 7fd7a2ffd640 1 -- 172.21.15.12:0/106013787 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd79c00cb50 con 0x7fd7ac0a6aa0 2024-09-11T01:46:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.217+0000 7fd7b44a9640 5 monclient: authenticate success, global_id 184229 2024-09-11T01:46:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.217+0000 7fd7b44a9640 10 monclient: _renew_subs 2024-09-11T01:46:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.217+0000 7fd7b44a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.217+0000 7fd7b44a9640 1 -- 172.21.15.12:0/106013787 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7ac1277b0 con 0x7fd7ac0a6aa0 2024-09-11T01:46:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.217+0000 7fd7a2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.217+0000 7fd7a2ffd640 1 -- 172.21.15.12:0/106013787 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c013470 con 0x7fd7ac0a6aa0 2024-09-11T01:46:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.217+0000 7fd7a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.217+0000 7fd7a2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.217+0000 7fd7a2ffd640 10 monclient: dump: 2024-09-11T01:46:32.237 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:32.237 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:32.237 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.237 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.237 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:32.237 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:32.237 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:32.237 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:32.237 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:32.237 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:32.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.217+0000 7fd7b0a42640 4 set_mon_vals no callback set 2024-09-11T01:46:32.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.217+0000 7fd7b44a9640 10 monclient: _renew_subs 2024-09-11T01:46:32.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.217+0000 7fd7b44a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.217+0000 7fd7b44a9640 1 -- 172.21.15.12:0/106013787 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7ac136830 con 0x7fd7ac0a6aa0 2024-09-11T01:46:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.218+0000 7fd7b44a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.218+0000 7fd7b44a9640 1 --2- 172.21.15.12:0/106013787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7ac1ccf00 0x7fd7ac123ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.218+0000 7fd7b44a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.218+0000 7fd7b44a9640 1 -- 172.21.15.12:0/106013787 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ac09cfd0 con 0x7fd7ac1ccf00 2024-09-11T01:46:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.218+0000 7fd7b2245640 1 --2- 172.21.15.12:0/106013787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7ac1ccf00 0x7fd7ac123ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.218+0000 7fd7b1a44640 1 --2- 172.21.15.12:0/106013787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd7ac0a6aa0 0x7fd7ac1362f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd79c00e820 tx=0x7fd79c00ece0 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7fd79c024060 mgrmap(e 5) 2024-09-11T01:46:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.218+0000 7fd7b2245640 10 monclient: get_auth_request con 0x7fd7ac1ccf00 auth_method 0 2024-09-11T01:46:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.218+0000 7fd7b2245640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.218+0000 7fd7b2245640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.218+0000 7fd7b2245640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.218+0000 7fd7a2ffd640 1 -- 172.21.15.12:0/106013787 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fd79c024060 con 0x7fd7ac0a6aa0 2024-09-11T01:46:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.218+0000 7fd7b2245640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.218+0000 7fd7b2245640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.218+0000 7fd7b2245640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.218+0000 7fd7b2245640 10 monclient(hunting): handle_auth_done global_id 194142 payload 824 2024-09-11T01:46:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.218+0000 7fd7b2245640 1 -- 172.21.15.12:0/106013787 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd7ac125fb0 con 0x7fd7ac1ccf00 2024-09-11T01:46:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.218+0000 7fd7a2ffd640 1 --2- 172.21.15.12:0/106013787 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd79003e210 0x7fd7900406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.219+0000 7fd7b2a46640 1 --2- 172.21.15.12:0/106013787 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd79003e210 0x7fd7900406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.219+0000 7fd7b2a46640 10 monclient: get_auth_request con 0x7fd79003e210 auth_method 0 2024-09-11T01:46:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.220+0000 7fd7b2245640 1 -- 172.21.15.12:0/106013787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7ac1ccf00 msgr2=0x7fd7ac123ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.221+0000 7fd7b2245640 1 --2- 172.21.15.12:0/106013787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7ac1ccf00 0x7fd7ac123ed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8035040 tx=0x7fd7a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.221+0000 7fd7a2ffd640 1 -- 172.21.15.12:0/106013787 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a8043070 con 0x7fd7ac1ccf00 2024-09-11T01:46:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.221+0000 7fd7a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.221+0000 7fd7a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:46:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.221+0000 7fd7a2ffd640 10 monclient: dump: 2024-09-11T01:46:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:32.240 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.221+0000 7fd7a2ffd640 1 -- 172.21.15.12:0/106013787 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd7a8038040 con 0x7fd7ac1ccf00 2024-09-11T01:46:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.221+0000 7fd7a2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.221+0000 7fd7a2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:46:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.221+0000 7fd7a2ffd640 1 -- 172.21.15.12:0/106013787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7ac1ccf00 msgr2=0x7fd7ac123ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.221+0000 7fd7a2ffd640 1 --2- 172.21.15.12:0/106013787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7ac1ccf00 0x7fd7ac123ed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8035040 tx=0x7fd7a80046c0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.221+0000 7fd7a2ffd640 1 -- 172.21.15.12:0/106013787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7ac1ccf00 msgr2=0x7fd7ac123ed0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:46:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.221+0000 7fd7a2ffd640 1 --2- 172.21.15.12:0/106013787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7ac1ccf00 0x7fd7ac123ed0 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8035040 tx=0x7fd7a80046c0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.221+0000 7fd7b2a46640 1 --2- 172.21.15.12:0/106013787 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd79003e210 0x7fd7900406d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac125880 tx=0x7fd7a4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.227+0000 7fd7b44a9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:46:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.227+0000 7fd7b44a9640 1 --2- 172.21.15.12:0/106013787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7ac124410 0x7fd7ac0a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.227+0000 7fd7b44a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.227+0000 7fd7b44a9640 1 -- 172.21.15.12:0/106013787 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ac124820 con 0x7fd7ac124410 2024-09-11T01:46:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.227+0000 7fd7b2245640 1 --2- 172.21.15.12:0/106013787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7ac124410 0x7fd7ac0a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.227+0000 7fd7b2245640 10 monclient: get_auth_request con 0x7fd7ac124410 auth_method 0 2024-09-11T01:46:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.227+0000 7fd7b2245640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.227+0000 7fd7b2245640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.227+0000 7fd7b2245640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.227+0000 7fd7b2245640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.227+0000 7fd7b2245640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.227+0000 7fd7b2245640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:32.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.227+0000 7fd7b2245640 10 monclient(hunting): handle_auth_done global_id 194145 payload 824 2024-09-11T01:46:32.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.227+0000 7fd7b2245640 1 -- 172.21.15.12:0/106013787 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd7ac126fd0 con 0x7fd7ac124410 2024-09-11T01:46:32.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.230+0000 7fd7b2245640 1 -- 172.21.15.12:0/106013787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7ac124410 msgr2=0x7fd7ac0a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:32.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.230+0000 7fd7b2245640 1 --2- 172.21.15.12:0/106013787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7ac124410 0x7fd7ac0a18f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8042000 tx=0x7fd7a8009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:32.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.230+0000 7fd7a2ffd640 1 -- 172.21.15.12:0/106013787 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a8037040 con 0x7fd7ac124410 2024-09-11T01:46:32.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.230+0000 7fd7a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:32.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.230+0000 7fd7a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:46:32.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.230+0000 7fd7a2ffd640 10 monclient: dump: 2024-09-11T01:46:32.242 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:32.242 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:32.242 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.242 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.242 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:32.242 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:32.243 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:32.243 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:32.243 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:32.243 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:32.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.230+0000 7fd7a2ffd640 1 -- 172.21.15.12:0/106013787 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7fd7a8004230 con 0x7fd7ac124410 2024-09-11T01:46:32.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.230+0000 7fd7a2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:46:32.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.230+0000 7fd7a2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:46:32.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.230+0000 7fd7a2ffd640 1 -- 172.21.15.12:0/106013787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7ac124410 msgr2=0x7fd7ac0a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:32.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.230+0000 7fd7a2ffd640 1 --2- 172.21.15.12:0/106013787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7ac124410 0x7fd7ac0a18f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8042000 tx=0x7fd7a8009190 comp rx=0 tx=0).stop 2024-09-11T01:46:32.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.230+0000 7fd7a2ffd640 1 -- 172.21.15.12:0/106013787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7ac124410 msgr2=0x7fd7ac0a18f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:46:32.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.230+0000 7fd7a2ffd640 1 --2- 172.21.15.12:0/106013787 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7ac124410 0x7fd7ac0a18f0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8042000 tx=0x7fd7a8009190 comp rx=0 tx=0).stop 2024-09-11T01:46:32.244 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:46:32.244 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:46:32.244 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:46:32.244 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:46:32.244 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 162, 2024-09-11T01:46:32.244 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 9467, 2024-09-11T01:46:32.244 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:46:32.244 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:46:32.244 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:46:32.244 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:46:32.244 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:32.244 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:46:32.244 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:46:32.245 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:46:32.245 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:46:32.245 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:32.245 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:32.245 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:32.245 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:32.245 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:32.245 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:32.245 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:32.245 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:32.245 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:32.245 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:32.245 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:32.246 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:32.246 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:46:32.246 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:46:32.246 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:32.246 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:32.246 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:32.246 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:32.246 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:32.246 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:32.246 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:32.246 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:32.246 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:32.246 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:32.247 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:32.247 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:32.247 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.247 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:46:32.247 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:46:32.247 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:46:32.247 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:46:32.247 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:46:32.247 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:46:32.247 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:46:32.247 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:46:32.247 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:46:32.247 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:46:32.248 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:46:32.248 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:46:32.248 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:46:32.248 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:46:32.248 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:46:32.248 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:46:32.248 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:46:32.248 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:32.248 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:32.248 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:32.248 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:32.248 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:32.248 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:32.248 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:32.249 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:32.249 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:32.249 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:32.249 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:32.249 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:32.249 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:46:32.249 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.249 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:46:32.249 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.249 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:46:32.249 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:46:32.249 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:32.249 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:32.250 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.250 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:32.250 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:46:32.250 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:32.250 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.250 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.250 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:32.250 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:46:32.250 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:32.250 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:32.250 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:32.250 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.250 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:46:32.250 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:46:32.251 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:32.251 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:32.251 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:32.251 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.251 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.251 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:46:32.251 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:46:32.251 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:32.251 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:32.251 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.251 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:32.251 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:46:32.251 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:32.251 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.252 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.252 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:32.252 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:46:32.252 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:32.252 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:32.252 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:32.252 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.252 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:46:32.252 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:46:32.252 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:32.252 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:32.252 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:32.252 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.253 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.253 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:46:32.253 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:46:32.253 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:32.253 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:32.253 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.253 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:32.253 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:46:32.253 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:32.253 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.253 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.253 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:32.253 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:46:32.253 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:32.254 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:32.254 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:32.254 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.254 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:46:32.254 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:46:32.254 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:32.254 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:32.254 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:32.254 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:32.254 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:32.254 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.254 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:46:32.254 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:46:32.254 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.255 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:32.255 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:32.255 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:46:32.255 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:32.255 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:32.255 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:46:32.255 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.255 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:32.255 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:32.255 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:46:32.255 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:32.255 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:32.255 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:46:32.255 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.256 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:32.256 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:32.256 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:46:32.256 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:32.256 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:32.256 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.256 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:46:32.256 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:46:32.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.230+0000 7fd7b44a9640 1 -- 172.21.15.12:0/106013787 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd79003e210 msgr2=0x7fd7900406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:32.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.230+0000 7fd7b44a9640 1 --2- 172.21.15.12:0/106013787 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd79003e210 0x7fd7900406d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac125880 tx=0x7fd7a4009040 comp rx=0 tx=0).stop 2024-09-11T01:46:32.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.230+0000 7fd7b44a9640 10 monclient: shutdown 2024-09-11T01:46:32.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.230+0000 7fd7b44a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:32.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.230+0000 7fd7b44a9640 1 -- 172.21.15.12:0/106013787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd7ac0a6aa0 msgr2=0x7fd7ac1362f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:32.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.230+0000 7fd7b44a9640 1 --2- 172.21.15.12:0/106013787 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd7ac0a6aa0 0x7fd7ac1362f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd79c00e820 tx=0x7fd79c00ece0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.230+0000 7fd7b2a46640 1 -- 172.21.15.12:0/106013787 reap_dead start 2024-09-11T01:46:32.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.230+0000 7fd7a2ffd640 1 -- 172.21.15.12:0/106013787 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(596..596 src has 507..596) ==== 4847+0+0 (unknown 0 0 0) 0x7fd79c052d20 con 0x7fd7ac0a6aa0 2024-09-11T01:46:32.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.230+0000 7fd7b2a46640 1 -- 172.21.15.12:0/106013787 reap_dead start 2024-09-11T01:46:32.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.230+0000 7fd7b44a9640 1 -- 172.21.15.12:0/106013787 shutdown_connections 2024-09-11T01:46:32.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.230+0000 7fd7b44a9640 1 -- 172.21.15.12:0/106013787 >> 172.21.15.12:0/106013787 conn(0x7fd7ac0a7440 msgr2=0x7fd7ac09ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:32.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.231+0000 7fd7b44a9640 1 -- 172.21.15.12:0/106013787 shutdown_connections 2024-09-11T01:46:32.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.231+0000 7fd7b44a9640 1 -- 172.21.15.12:0/106013787 wait complete. 2024-09-11T01:46:32.259 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:46:32.259 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:46:32.259 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:46:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.359+0000 7fc3a4270640 10 monclient: get_monmap_and_config 2024-09-11T01:46:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.359+0000 7fc3a4270640 10 monclient: build_initial_monmap 2024-09-11T01:46:32.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.359+0000 7fc3a4270640 10 monclient: monmap: 2024-09-11T01:46:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:46:32.360893+0000 2024-09-11T01:46:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:46:32.360893+0000 2024-09-11T01:46:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:46:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:46:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:46:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:46:32.362 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:46:32.362 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.359+0000 7fc3a4270640 1 Processor -- start 2024-09-11T01:46:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.359+0000 7fc3a4270640 1 -- start start 2024-09-11T01:46:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.359+0000 7fc3a4270640 10 monclient: init 2024-09-11T01:46:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a4270640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a4270640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-11T01:46:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a4270640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc39c0a2db0 0x7fc39c06f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a4270640 10 monclient(hunting): picked mon.noname-c con 0x7fc39c0a2db0 addr v2:172.21.15.71:3300/0 2024-09-11T01:46:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a4270640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc39c06ded0 addr v1:172.21.15.71:6789/0 2024-09-11T01:46:32.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a4270640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc39c0a38a0 0x7fc39c13f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a4270640 10 monclient(hunting): picked mon.noname-a con 0x7fc39c0a38a0 addr v2:172.21.15.12:3300/0 2024-09-11T01:46:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a4270640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc39c0a4650 con 0x7fc39c06ded0 2024-09-11T01:46:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a4270640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a4270640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc39c06f1a0 con 0x7fc39c0a38a0 2024-09-11T01:46:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a4270640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a4270640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fc39c06eec0 con 0x7fc39c0a2db0 2024-09-11T01:46:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a4270640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a4270640 10 monclient(hunting): authenticate will time out at 1798.803480s 2024-09-11T01:46:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a280d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc39c0a38a0 0x7fc39c13f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a280d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc39c0a38a0 0x7fc39c13f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55386/0 (socket says 172.21.15.12:55386) 2024-09-11T01:46:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a280d640 1 -- 172.21.15.12:0/3098753302 learned_addr learned my addr 172.21.15.12:0/3098753302 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a280d640 10 monclient(hunting): get_auth_request con 0x7fc39c0a38a0 auth_method 0 2024-09-11T01:46:32.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a280d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:32.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a280d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:32.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a280d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:32.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a200c640 1 --2- 172.21.15.12:0/3098753302 >> v2:172.21.15.71:3300/0 conn(0x7fc39c0a2db0 0x7fc39c06f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:32.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a200c640 10 monclient(hunting): get_auth_request con 0x7fc39c0a2db0 auth_method 0 2024-09-11T01:46:32.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a200c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:32.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a200c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:32.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a200c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:32.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a280d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:32.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a280d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:32.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a280d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:32.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a200c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:32.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a200c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:32.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a200c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:32.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a280d640 10 monclient(hunting): handle_auth_done global_id 194148 payload 274 2024-09-11T01:46:32.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a280d640 1 -- 172.21.15.12:0/3098753302 >> v2:172.21.15.71:3300/0 conn(0x7fc39c0a2db0 msgr2=0x7fc39c06f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:46:32.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a280d640 1 --2- 172.21.15.12:0/3098753302 >> v2:172.21.15.71:3300/0 conn(0x7fc39c0a2db0 0x7fc39c06f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a280d640 1 -- 172.21.15.12:0/3098753302 >> v1:172.21.15.71:6789/0 conn(0x7fc39c06ded0 legacy=0x7fc39c0a3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:46:32.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a280d640 10 monclient: _finish_hunting 0 2024-09-11T01:46:32.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a280d640 1 monclient: found mon.noname-a 2024-09-11T01:46:32.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a280d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:32.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a280d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:46:32.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.360+0000 7fc3a280d640 1 -- 172.21.15.12:0/3098753302 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc39c06e2e0 con 0x7fc39c0a38a0 2024-09-11T01:46:32.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.363+0000 7fc3a280d640 1 -- 172.21.15.12:0/3098753302 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc39c0a38a0 msgr2=0x7fc39c13f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:32.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.363+0000 7fc3a280d640 1 --2- 172.21.15.12:0/3098753302 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc39c0a38a0 0x7fc39c13f3c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc39c070b40 tx=0x7fc39402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=412c19f5e8096ae9 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:32.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.363+0000 7fc3a100a640 1 -- 172.21.15.12:0/3098753302 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc394036070 con 0x7fc39c0a38a0 2024-09-11T01:46:32.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.363+0000 7fc3a100a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:32.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.363+0000 7fc3a100a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:46:32.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.363+0000 7fc3a100a640 10 monclient: dump: 2024-09-11T01:46:32.374 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:32.374 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:32.374 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.374 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.374 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:32.374 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:32.374 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:32.374 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:32.375 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:32.375 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:32.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.363+0000 7fc3a100a640 10 monclient: _finish_auth 0 2024-09-11T01:46:32.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.363+0000 7fc3a100a640 10 monclient: _check_auth_tickets 2024-09-11T01:46:32.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.363+0000 7fc3a100a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:32.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.363+0000 7fc3a100a640 1 -- 172.21.15.12:0/3098753302 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc394007800 con 0x7fc39c0a38a0 2024-09-11T01:46:32.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.363+0000 7fc3a100a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:32.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.363+0000 7fc3a100a640 1 -- 172.21.15.12:0/3098753302 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc394007bd0 con 0x7fc39c0a38a0 2024-09-11T01:46:32.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.363+0000 7fc3a4270640 5 monclient: authenticate success, global_id 194148 2024-09-11T01:46:32.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.363+0000 7fc3a4270640 10 monclient: get_monmap_and_config success 2024-09-11T01:46:32.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.363+0000 7fc3a100a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:32.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.363+0000 7fc3a4270640 4 set_mon_vals no callback set 2024-09-11T01:46:32.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.363+0000 7fc3a100a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:46:32.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.363+0000 7fc3a100a640 10 monclient: dump: 2024-09-11T01:46:32.376 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:32.376 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:32.376 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.376 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.376 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:32.376 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:32.376 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:32.376 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:32.376 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:32.376 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:32.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.364+0000 7fc3a4270640 10 monclient: shutdown 2024-09-11T01:46:32.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.364+0000 7fc3a4270640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:32.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.364+0000 7fc3a4270640 1 -- 172.21.15.12:0/3098753302 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc39c0a38a0 msgr2=0x7fc39c13f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:32.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.364+0000 7fc3a4270640 1 --2- 172.21.15.12:0/3098753302 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc39c0a38a0 0x7fc39c13f3c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc39c070b40 tx=0x7fc39402a610 comp rx=0 tx=0).stop 2024-09-11T01:46:32.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.364+0000 7fc3a4270640 1 -- 172.21.15.12:0/3098753302 shutdown_connections 2024-09-11T01:46:32.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.364+0000 7fc3a4270640 1 --2- 172.21.15.12:0/3098753302 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc39c0a38a0 0x7fc39c13f3c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.364+0000 7fc3a4270640 1 --2- 172.21.15.12:0/3098753302 >> v2:172.21.15.71:3300/0 conn(0x7fc39c0a2db0 0x7fc39c06f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.364+0000 7fc3a4270640 1 -- 172.21.15.12:0/3098753302 >> 172.21.15.12:0/3098753302 conn(0x7fc39c0a83b0 msgr2=0x7fc39c0a87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:32.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.364+0000 7fc3a4270640 1 -- 172.21.15.12:0/3098753302 shutdown_connections 2024-09-11T01:46:32.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.364+0000 7fc3a4270640 1 -- 172.21.15.12:0/3098753302 wait complete. 2024-09-11T01:46:32.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.365+0000 7fc3a4270640 10 monclient: build_initial_monmap 2024-09-11T01:46:32.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.365+0000 7fc3a4270640 10 monclient: monmap: 2024-09-11T01:46:32.377 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:32.377 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:46:32.377 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:46:32.377 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:46:32.378 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:32.378 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:32.378 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:46:32.378 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:46:32.378 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:46:32.378 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:32.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.365+0000 7fc3a4270640 1 Processor -- start 2024-09-11T01:46:32.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.365+0000 7fc3a4270640 1 -- start start 2024-09-11T01:46:32.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.365+0000 7fc3a4270640 10 monclient: init 2024-09-11T01:46:32.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.365+0000 7fc3a4270640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:32.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a4270640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:46:32.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a4270640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc39c06ded0 0x7fc39c134f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:32.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a4270640 10 monclient(hunting): picked mon.noname-a con 0x7fc39c06ded0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:46:32.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a4270640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc39c0a2db0 0x7fc39c135470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:32.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a4270640 10 monclient(hunting): picked mon.noname-b con 0x7fc39c0a2db0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:46:32.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a4270640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc39c0a38a0 0x7fc39c1359b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:32.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a4270640 10 monclient(hunting): picked mon.noname-c con 0x7fc39c0a38a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:32.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a4270640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:32.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a4270640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc39c13fc90 con 0x7fc39c06ded0 2024-09-11T01:46:32.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a4270640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:32.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a4270640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc39c13fb10 con 0x7fc39c0a2db0 2024-09-11T01:46:32.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a4270640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:32.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a4270640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc39c13fe10 con 0x7fc39c0a38a0 2024-09-11T01:46:32.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a4270640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:32.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a4270640 10 monclient(hunting): authenticate will time out at 1798.809130s 2024-09-11T01:46:32.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a200c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc39c06ded0 0x7fc39c134f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:32.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a200c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc39c06ded0 0x7fc39c134f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55392/0 (socket says 172.21.15.12:55392) 2024-09-11T01:46:32.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a200c640 1 -- 172.21.15.12:0/4018791991 learned_addr learned my addr 172.21.15.12:0/4018791991 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:32.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a200c640 10 monclient(hunting): get_auth_request con 0x7fc39c06ded0 auth_method 0 2024-09-11T01:46:32.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a200c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:32.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a200c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:32.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a200c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:32.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a280d640 1 --2- 172.21.15.12:0/4018791991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc39c0a38a0 0x7fc39c1359b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:32.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a200c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:32.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a200c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:32.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a200c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:32.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a280d640 10 monclient(hunting): get_auth_request con 0x7fc39c0a38a0 auth_method 0 2024-09-11T01:46:32.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a280d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:32.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a280d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:32.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a280d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:32.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a280d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:32.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a280d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:32.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a280d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:32.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a200c640 10 monclient(hunting): handle_auth_done global_id 194151 payload 824 2024-09-11T01:46:32.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a180b640 1 --2- 172.21.15.12:0/4018791991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc39c0a2db0 0x7fc39c135470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:32.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a200c640 1 -- 172.21.15.12:0/4018791991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc39c0a38a0 msgr2=0x7fc39c1359b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:32.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a200c640 1 --2- 172.21.15.12:0/4018791991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc39c0a38a0 0x7fc39c1359b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a200c640 1 -- 172.21.15.12:0/4018791991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc39c0a2db0 msgr2=0x7fc39c135470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:32.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a200c640 1 --2- 172.21.15.12:0/4018791991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc39c0a2db0 0x7fc39c135470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a200c640 10 monclient: _finish_hunting 0 2024-09-11T01:46:32.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a200c640 1 monclient: found mon.noname-a 2024-09-11T01:46:32.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a200c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:32.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a200c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:46:32.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a200c640 1 -- 172.21.15.12:0/4018791991 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc39c13b5b0 con 0x7fc39c06ded0 2024-09-11T01:46:32.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a180b640 10 monclient: get_auth_request con 0x7fc39c0a2db0 auth_method 0 2024-09-11T01:46:32.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.366+0000 7fc3a180b640 1 --2- 172.21.15.12:0/4018791991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc39c0a2db0 0x7fc39c135470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:46:32.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.368+0000 7fc3a200c640 1 -- 172.21.15.12:0/4018791991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc39c06ded0 msgr2=0x7fc39c134f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:32.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.368+0000 7fc3a200c640 1 --2- 172.21.15.12:0/4018791991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc39c06ded0 0x7fc39c134f30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc39800e9e0 tx=0x7fc39800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:32.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.369+0000 7fc392ffd640 1 -- 172.21.15.12:0/4018791991 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc398017070 con 0x7fc39c06ded0 2024-09-11T01:46:32.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.369+0000 7fc392ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:32.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.369+0000 7fc392ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:46:32.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.369+0000 7fc392ffd640 10 monclient: dump: 2024-09-11T01:46:32.382 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:32.382 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:32.382 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.382 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:32.383 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.369+0000 7fc392ffd640 10 monclient: _finish_auth 0 2024-09-11T01:46:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.369+0000 7fc392ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:46:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.369+0000 7fc392ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.369+0000 7fc392ffd640 1 -- 172.21.15.12:0/4018791991 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc39800cbe0 con 0x7fc39c06ded0 2024-09-11T01:46:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.369+0000 7fc392ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.369+0000 7fc392ffd640 1 -- 172.21.15.12:0/4018791991 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc398013470 con 0x7fc39c06ded0 2024-09-11T01:46:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.369+0000 7fc3a4270640 5 monclient: authenticate success, global_id 194151 2024-09-11T01:46:32.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.369+0000 7fc3a4270640 10 monclient: _renew_subs 2024-09-11T01:46:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.369+0000 7fc3a4270640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:46:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.369+0000 7fc3a4270640 1 -- 172.21.15.12:0/4018791991 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc39c09f5a0 con 0x7fc39c06ded0 2024-09-11T01:46:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.369+0000 7fc392ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.369+0000 7fc392ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:46:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.369+0000 7fc3a0809640 4 set_mon_vals no callback set 2024-09-11T01:46:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.369+0000 7fc392ffd640 10 monclient: dump: 2024-09-11T01:46:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:32.385 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:32.385 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:32.385 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:32.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.369+0000 7fc3a4270640 10 monclient: _renew_subs 2024-09-11T01:46:32.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.369+0000 7fc3a4270640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:46:32.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.369+0000 7fc3a4270640 1 -- 172.21.15.12:0/4018791991 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc39c09e410 con 0x7fc39c06ded0 2024-09-11T01:46:32.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.369+0000 7fc392ffd640 1 -- 172.21.15.12:0/4018791991 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fc398024060 con 0x7fc39c06ded0 2024-09-11T01:46:32.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.369+0000 7fc3a4270640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:32.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.370+0000 7fc3a4270640 1 --2- 172.21.15.12:0/4018791991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc39c063d50 0x7fc39c064130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:32.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.370+0000 7fc3a4270640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:32.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.370+0000 7fc3a4270640 1 -- 172.21.15.12:0/4018791991 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc39c06a060 con 0x7fc39c063d50 2024-09-11T01:46:32.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.370+0000 7fc3a180b640 1 --2- 172.21.15.12:0/4018791991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc39c063d50 0x7fc39c064130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:32.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.370+0000 7fc3a180b640 10 monclient: get_auth_request con 0x7fc39c063d50 auth_method 0 2024-09-11T01:46:32.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.370+0000 7fc3a180b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:32.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.370+0000 7fc3a180b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:32.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.370+0000 7fc3a180b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:32.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.370+0000 7fc392ffd640 1 --2- 172.21.15.12:0/4018791991 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc38003e260 0x7fc380040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:32.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.370+0000 7fc392ffd640 1 -- 172.21.15.12:0/4018791991 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(596..596 src has 507..596) ==== 4847+0+0 (secure 0 0 0) 0x7fc398052d40 con 0x7fc39c06ded0 2024-09-11T01:46:32.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.370+0000 7fc3a180b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:32.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.370+0000 7fc3a180b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:32.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.370+0000 7fc3a180b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:32.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.370+0000 7fc3a180b640 10 monclient(hunting): handle_auth_done global_id 184235 payload 824 2024-09-11T01:46:32.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.370+0000 7fc3a180b640 1 -- 172.21.15.12:0/4018791991 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc39c13ba20 con 0x7fc39c063d50 2024-09-11T01:46:32.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.370+0000 7fc3a280d640 1 --2- 172.21.15.12:0/4018791991 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc38003e260 0x7fc380040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:32.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.370+0000 7fc3a280d640 10 monclient: get_auth_request con 0x7fc38003e260 auth_method 0 2024-09-11T01:46:32.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.372+0000 7fc3a180b640 1 -- 172.21.15.12:0/4018791991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc39c063d50 msgr2=0x7fc39c064130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:32.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.372+0000 7fc3a180b640 1 --2- 172.21.15.12:0/4018791991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc39c063d50 0x7fc39c064130 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc38c00d4f0 tx=0x7fc38c00d9b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:32.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.372+0000 7fc392ffd640 1 -- 172.21.15.12:0/4018791991 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38c012a10 con 0x7fc39c063d50 2024-09-11T01:46:32.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.372+0000 7fc392ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:32.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.372+0000 7fc392ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:32.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.372+0000 7fc392ffd640 10 monclient: dump: 2024-09-11T01:46:32.389 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:32.389 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:32.389 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.389 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.390 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:32.390 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:32.390 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:32.390 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:32.390 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:32.390 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:32.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.372+0000 7fc3a280d640 1 --2- 172.21.15.12:0/4018791991 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc38003e260 0x7fc380040720 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc39c072030 tx=0x7fc394009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:32.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.373+0000 7fc392ffd640 1 -- 172.21.15.12:0/4018791991 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc38c012a10 con 0x7fc39c063d50 2024-09-11T01:46:32.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.373+0000 7fc392ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:32.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.373+0000 7fc392ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:46:32.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.373+0000 7fc392ffd640 1 -- 172.21.15.12:0/4018791991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc39c063d50 msgr2=0x7fc39c064130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:32.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.373+0000 7fc392ffd640 1 --2- 172.21.15.12:0/4018791991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc39c063d50 0x7fc39c064130 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc38c00d4f0 tx=0x7fc38c00d9b0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.373+0000 7fc392ffd640 1 -- 172.21.15.12:0/4018791991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc39c063d50 msgr2=0x7fc39c064130 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:46:32.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.373+0000 7fc392ffd640 1 --2- 172.21.15.12:0/4018791991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc39c063d50 0x7fc39c064130 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc38c00d4f0 tx=0x7fc38c00d9b0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.379+0000 7fc3a4270640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:46:32.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.379+0000 7fc3a4270640 1 --2- 172.21.15.12:0/4018791991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc39c02d440 0x7fc39c02d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:32.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.379+0000 7fc3a4270640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:32.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.379+0000 7fc3a4270640 1 -- 172.21.15.12:0/4018791991 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc39c137860 con 0x7fc39c02d440 2024-09-11T01:46:32.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.379+0000 7fc3a180b640 1 --2- 172.21.15.12:0/4018791991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc39c02d440 0x7fc39c02d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:32.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.379+0000 7fc3a180b640 10 monclient: get_auth_request con 0x7fc39c02d440 auth_method 0 2024-09-11T01:46:32.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.379+0000 7fc3a180b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:32.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.379+0000 7fc3a180b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:32.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.379+0000 7fc3a180b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:32.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.379+0000 7fc3a180b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:32.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.379+0000 7fc3a180b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:32.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.379+0000 7fc3a180b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:32.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.379+0000 7fc3a180b640 10 monclient(hunting): handle_auth_done global_id 184238 payload 824 2024-09-11T01:46:32.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.379+0000 7fc3a180b640 1 -- 172.21.15.12:0/4018791991 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc39c069990 con 0x7fc39c02d440 2024-09-11T01:46:32.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.381+0000 7fc3a180b640 1 -- 172.21.15.12:0/4018791991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc39c02d440 msgr2=0x7fc39c02d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:32.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.381+0000 7fc3a180b640 1 --2- 172.21.15.12:0/4018791991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc39c02d440 0x7fc39c02d850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc38c00d4f0 tx=0x7fc38c003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:32.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.381+0000 7fc392ffd640 1 -- 172.21.15.12:0/4018791991 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38c028020 con 0x7fc39c02d440 2024-09-11T01:46:32.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.381+0000 7fc392ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:32.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.381+0000 7fc392ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:32.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.381+0000 7fc392ffd640 10 monclient: dump: 2024-09-11T01:46:32.392 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:32.392 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:32.392 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.392 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.393 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:32.393 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:32.393 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:32.393 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:32.393 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:32.393 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:32.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.382+0000 7fc392ffd640 1 -- 172.21.15.12:0/4018791991 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7fc38c023350 con 0x7fc39c02d440 2024-09-11T01:46:32.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.382+0000 7fc392ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:46:32.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.382+0000 7fc392ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:46:32.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.382+0000 7fc392ffd640 1 -- 172.21.15.12:0/4018791991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc39c02d440 msgr2=0x7fc39c02d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:32.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.382+0000 7fc392ffd640 1 --2- 172.21.15.12:0/4018791991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc39c02d440 0x7fc39c02d850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc38c00d4f0 tx=0x7fc38c003760 comp rx=0 tx=0).stop 2024-09-11T01:46:32.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.382+0000 7fc392ffd640 1 -- 172.21.15.12:0/4018791991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc39c02d440 msgr2=0x7fc39c02d850 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:46:32.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.382+0000 7fc392ffd640 1 --2- 172.21.15.12:0/4018791991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc39c02d440 0x7fc39c02d850 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc38c00d4f0 tx=0x7fc38c003760 comp rx=0 tx=0).stop 2024-09-11T01:46:32.394 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:46:32.394 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:46:32.394 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:46:32.394 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:46:32.394 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 162, 2024-09-11T01:46:32.394 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 52975, 2024-09-11T01:46:32.394 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:46:32.394 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:46:32.394 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:46:32.394 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:46:32.394 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:32.394 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:46:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:46:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:46:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:46:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:32.395 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:46:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:46:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:32.396 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:46:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:46:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:46:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:46:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:46:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:46:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:46:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:46:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:46:32.397 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:46:32.398 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:46:32.398 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:46:32.398 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:46:32.398 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:46:32.398 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:46:32.398 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:46:32.398 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:46:32.398 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:32.398 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:32.398 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:32.398 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:32.398 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:32.398 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:32.399 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:32.399 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:32.399 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:32.399 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:32.399 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:32.399 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:32.399 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:46:32.399 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.399 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:46:32.399 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.399 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:46:32.399 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:46:32.399 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:32.400 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:32.400 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.400 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:32.400 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:46:32.400 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:32.400 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.400 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.400 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:32.400 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:46:32.400 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:32.400 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:32.400 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:32.400 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.401 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:46:32.401 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:46:32.401 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:32.401 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:32.401 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:32.401 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.401 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.401 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:46:32.401 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:46:32.401 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:32.401 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:32.401 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.401 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:32.401 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:46:32.402 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:32.402 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.402 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.402 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:32.402 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:46:32.402 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:32.402 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:32.402 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:32.402 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.402 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:46:32.402 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:46:32.402 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:32.402 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:32.403 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:32.403 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.403 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.403 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:46:32.403 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:46:32.403 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:32.403 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:32.403 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.403 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:32.403 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:46:32.403 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:32.403 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.403 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.404 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:32.404 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:46:32.404 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:32.404 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:32.404 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:32.404 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.404 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:46:32.404 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:46:32.404 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:32.404 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:32.404 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:32.404 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:32.404 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:32.405 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.405 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:46:32.405 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:46:32.405 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.405 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:32.405 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:32.405 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:46:32.405 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:32.405 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:32.405 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:46:32.405 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.405 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:32.405 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:32.406 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:46:32.406 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:32.406 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:32.406 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.406 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:46:32.406 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:46:32.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.382+0000 7fc3a4270640 1 -- 172.21.15.12:0/4018791991 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc38003e260 msgr2=0x7fc380040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:32.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.382+0000 7fc3a4270640 1 --2- 172.21.15.12:0/4018791991 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc38003e260 0x7fc380040720 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc39c072030 tx=0x7fc394009000 comp rx=0 tx=0).stop 2024-09-11T01:46:32.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.382+0000 7fc3a4270640 10 monclient: shutdown 2024-09-11T01:46:32.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.382+0000 7fc3a4270640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:32.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.382+0000 7fc3a4270640 1 -- 172.21.15.12:0/4018791991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc39c06ded0 msgr2=0x7fc39c134f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:32.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.382+0000 7fc3a4270640 1 --2- 172.21.15.12:0/4018791991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc39c06ded0 0x7fc39c134f30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc39800e9e0 tx=0x7fc39800eea0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.382+0000 7fc3a280d640 1 -- 172.21.15.12:0/4018791991 reap_dead start 2024-09-11T01:46:32.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.382+0000 7fc3a280d640 1 -- 172.21.15.12:0/4018791991 reap_dead start 2024-09-11T01:46:32.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.382+0000 7fc3a4270640 1 -- 172.21.15.12:0/4018791991 shutdown_connections 2024-09-11T01:46:32.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.382+0000 7fc3a4270640 1 -- 172.21.15.12:0/4018791991 >> 172.21.15.12:0/4018791991 conn(0x7fc39c0a83b0 msgr2=0x7fc39c0a1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:32.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.382+0000 7fc3a4270640 1 -- 172.21.15.12:0/4018791991 shutdown_connections 2024-09-11T01:46:32.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.382+0000 7fc3a4270640 1 -- 172.21.15.12:0/4018791991 wait complete. 2024-09-11T01:46:32.408 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:46:32.409 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:46:32.409 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:46:32.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.508+0000 7fc6df3a0640 10 monclient: get_monmap_and_config 2024-09-11T01:46:32.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.508+0000 7fc6df3a0640 10 monclient: build_initial_monmap 2024-09-11T01:46:32.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.508+0000 7fc6df3a0640 10 monclient: monmap: 2024-09-11T01:46:32.511 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:32.511 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:32.511 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:46:32.510466+0000 2024-09-11T01:46:32.511 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:46:32.510466+0000 2024-09-11T01:46:32.511 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:32.512 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:32.512 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:46:32.512 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:46:32.512 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:46:32.512 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:46:32.512 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:46:32.512 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:32.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.509+0000 7fc6df3a0640 1 Processor -- start 2024-09-11T01:46:32.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.509+0000 7fc6df3a0640 1 -- start start 2024-09-11T01:46:32.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.509+0000 7fc6df3a0640 10 monclient: init 2024-09-11T01:46:32.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.509+0000 7fc6df3a0640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:32.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.509+0000 7fc6df3a0640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-11T01:46:32.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6df3a0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc6d8137e50 addr v1:172.21.15.71:6789/0 2024-09-11T01:46:32.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6df3a0640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc6d8138230 0x7fc6d8138640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:32.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6df3a0640 10 monclient(hunting): picked mon.noname-a con 0x7fc6d8138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:46:32.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6df3a0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6d813d380 0x7fc6d813f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:32.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6df3a0640 10 monclient(hunting): picked mon.noname-b con 0x7fc6d813d380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:32.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6df3a0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6d80a6580 con 0x7fc6d8137e50 2024-09-11T01:46:32.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6df3a0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:32.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6df3a0640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc6d80a6310 con 0x7fc6d8138230 2024-09-11T01:46:32.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6df3a0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:32.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6df3a0640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc6d809bb90 con 0x7fc6d813d380 2024-09-11T01:46:32.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6df3a0640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:32.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6df3a0640 10 monclient(hunting): authenticate will time out at 1798.953078s 2024-09-11T01:46:32.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dc93b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc6d8138230 0x7fc6d8138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:32.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dd93d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6d813d380 0x7fc6d813f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:32.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dd93d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6d813d380 0x7fc6d813f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:43540/0 (socket says 172.21.15.12:43540) 2024-09-11T01:46:32.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dc93b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc6d8138230 0x7fc6d8138640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55394/0 (socket says 172.21.15.12:55394) 2024-09-11T01:46:32.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dd93d640 1 -- 172.21.15.12:0/1756405034 learned_addr learned my addr 172.21.15.12:0/1756405034 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:32.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dd93d640 10 monclient(hunting): get_auth_request con 0x7fc6d813d380 auth_method 0 2024-09-11T01:46:32.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dd93d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:32.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dd93d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:32.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dd93d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:32.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dc93b640 10 monclient(hunting): get_auth_request con 0x7fc6d8138230 auth_method 0 2024-09-11T01:46:32.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dc93b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:32.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dc93b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:32.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dc93b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:32.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dd93d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:32.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dd93d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:32.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dd93d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:32.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dc93b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:32.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dc93b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:32.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dc93b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:32.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dd93d640 10 monclient(hunting): handle_auth_done global_id 184241 payload 274 2024-09-11T01:46:32.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dd93d640 1 -- 172.21.15.12:0/1756405034 >> v2:172.21.15.12:3300/0 conn(0x7fc6d8138230 msgr2=0x7fc6d8138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:46:32.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dd93d640 1 --2- 172.21.15.12:0/1756405034 >> v2:172.21.15.12:3300/0 conn(0x7fc6d8138230 0x7fc6d8138640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dd93d640 1 -- 172.21.15.12:0/1756405034 >> v1:172.21.15.71:6789/0 conn(0x7fc6d8137e50 legacy=0x7fc6d809d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:46:32.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dd93d640 10 monclient: _finish_hunting 0 2024-09-11T01:46:32.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dd93d640 1 monclient: found mon.noname-b 2024-09-11T01:46:32.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dd93d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:32.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dd93d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:46:32.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.510+0000 7fc6dd93d640 1 -- 172.21.15.12:0/1756405034 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d809c780 con 0x7fc6d813d380 2024-09-11T01:46:32.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.513+0000 7fc6dd93d640 1 -- 172.21.15.12:0/1756405034 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6d813d380 msgr2=0x7fc6d813f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:32.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.513+0000 7fc6dd93d640 1 --2- 172.21.15.12:0/1756405034 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6d813d380 0x7fc6d813f770 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0009e50 tx=0x7fc6d002a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=766d67c6b7bc963f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:32.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.513+0000 7fc6cbfff640 1 -- 172.21.15.12:0/1756405034 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0036070 con 0x7fc6d813d380 2024-09-11T01:46:32.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.513+0000 7fc6cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:32.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.513+0000 7fc6cbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:46:32.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.513+0000 7fc6cbfff640 10 monclient: dump: 2024-09-11T01:46:32.524 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:32.524 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:32.524 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.524 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.524 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:32.525 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:32.525 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:32.525 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:32.525 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:32.525 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:32.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.513+0000 7fc6cbfff640 10 monclient: _finish_auth 0 2024-09-11T01:46:32.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.513+0000 7fc6cbfff640 10 monclient: _check_auth_tickets 2024-09-11T01:46:32.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.513+0000 7fc6cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:32.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.513+0000 7fc6cbfff640 1 -- 172.21.15.12:0/1756405034 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6d0007800 con 0x7fc6d813d380 2024-09-11T01:46:32.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.513+0000 7fc6df3a0640 5 monclient: authenticate success, global_id 184241 2024-09-11T01:46:32.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.513+0000 7fc6df3a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:46:32.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.513+0000 7fc6cbfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:32.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.513+0000 7fc6df3a0640 10 monclient: get_monmap_and_config success 2024-09-11T01:46:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.513+0000 7fc6cbfff640 1 -- 172.21.15.12:0/1756405034 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0007bd0 con 0x7fc6d813d380 2024-09-11T01:46:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.513+0000 7fc6df3a0640 4 set_mon_vals no callback set 2024-09-11T01:46:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.513+0000 7fc6cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.513+0000 7fc6cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.513+0000 7fc6cbfff640 10 monclient: dump: 2024-09-11T01:46:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:32.527 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.513+0000 7fc6df3a0640 10 monclient: shutdown 2024-09-11T01:46:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.513+0000 7fc6df3a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.513+0000 7fc6df3a0640 1 -- 172.21.15.12:0/1756405034 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6d813d380 msgr2=0x7fc6d813f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.513+0000 7fc6df3a0640 1 --2- 172.21.15.12:0/1756405034 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6d813d380 0x7fc6d813f770 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0009e50 tx=0x7fc6d002a560 comp rx=0 tx=0).stop 2024-09-11T01:46:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.514+0000 7fc6df3a0640 1 -- 172.21.15.12:0/1756405034 shutdown_connections 2024-09-11T01:46:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.514+0000 7fc6df3a0640 1 --2- 172.21.15.12:0/1756405034 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6d813d380 0x7fc6d813f770 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.514+0000 7fc6df3a0640 1 --2- 172.21.15.12:0/1756405034 >> v2:172.21.15.12:3300/0 conn(0x7fc6d8138230 0x7fc6d8138640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.514+0000 7fc6df3a0640 1 -- 172.21.15.12:0/1756405034 >> 172.21.15.12:0/1756405034 conn(0x7fc6d80a7020 msgr2=0x7fc6d80a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.514+0000 7fc6df3a0640 1 -- 172.21.15.12:0/1756405034 shutdown_connections 2024-09-11T01:46:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.514+0000 7fc6df3a0640 1 -- 172.21.15.12:0/1756405034 wait complete. 2024-09-11T01:46:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.514+0000 7fc6df3a0640 10 monclient: build_initial_monmap 2024-09-11T01:46:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.514+0000 7fc6df3a0640 10 monclient: monmap: 2024-09-11T01:46:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:32.528 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:46:32.528 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:46:32.528 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:46:32.528 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:32.528 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:32.528 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:46:32.528 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:46:32.528 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:46:32.528 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:32.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6df3a0640 1 Processor -- start 2024-09-11T01:46:32.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6df3a0640 1 -- start start 2024-09-11T01:46:32.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6df3a0640 10 monclient: init 2024-09-11T01:46:32.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6df3a0640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:32.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6df3a0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:46:32.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6df3a0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6d8137e50 0x7fc6d81ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:32.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6df3a0640 10 monclient(hunting): picked mon.noname-a con 0x7fc6d8137e50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:46:32.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6df3a0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6d8138230 0x7fc6d81cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:32.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6df3a0640 10 monclient(hunting): picked mon.noname-b con 0x7fc6d8138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:46:32.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6df3a0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6d813d380 0x7fc6d80a1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:32.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6df3a0640 10 monclient(hunting): picked mon.noname-c con 0x7fc6d813d380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:32.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6df3a0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:32.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6df3a0640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d806f890 con 0x7fc6d8137e50 2024-09-11T01:46:32.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6df3a0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:32.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6df3a0640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d806f710 con 0x7fc6d8138230 2024-09-11T01:46:32.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6df3a0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:32.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6df3a0640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc6d813fcb0 con 0x7fc6d813d380 2024-09-11T01:46:32.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6df3a0640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:32.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6df3a0640 10 monclient(hunting): authenticate will time out at 1798.958561s 2024-09-11T01:46:32.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd13c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6d8137e50 0x7fc6d81ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:32.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd93d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6d813d380 0x7fc6d80a1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:32.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd13c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6d8137e50 0x7fc6d81ccf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55408/0 (socket says 172.21.15.12:55408) 2024-09-11T01:46:32.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd13c640 1 -- 172.21.15.12:0/2907118042 learned_addr learned my addr 172.21.15.12:0/2907118042 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:32.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd13c640 10 monclient(hunting): get_auth_request con 0x7fc6d8137e50 auth_method 0 2024-09-11T01:46:32.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd13c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:32.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd13c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:32.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd13c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:32.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd93d640 10 monclient(hunting): get_auth_request con 0x7fc6d813d380 auth_method 0 2024-09-11T01:46:32.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd93d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:32.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd93d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:32.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd93d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd93d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd93d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd93d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd13c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd13c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd13c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dc93b640 1 --2- 172.21.15.12:0/2907118042 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6d8138230 0x7fc6d81cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dc93b640 10 monclient(hunting): get_auth_request con 0x7fc6d8138230 auth_method 0 2024-09-11T01:46:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dc93b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dc93b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dc93b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd93d640 10 monclient(hunting): handle_auth_done global_id 184244 payload 824 2024-09-11T01:46:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd93d640 1 -- 172.21.15.12:0/2907118042 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6d8138230 msgr2=0x7fc6d81cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd93d640 1 --2- 172.21.15.12:0/2907118042 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6d8138230 0x7fc6d81cd6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd93d640 1 -- 172.21.15.12:0/2907118042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6d8137e50 msgr2=0x7fc6d81ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd93d640 1 --2- 172.21.15.12:0/2907118042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6d8137e50 0x7fc6d81ccf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd93d640 10 monclient: _finish_hunting 0 2024-09-11T01:46:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd93d640 1 monclient: found mon.noname-c 2024-09-11T01:46:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd93d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd93d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:46:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dd93d640 1 -- 172.21.15.12:0/2907118042 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d80a1710 con 0x7fc6d813d380 2024-09-11T01:46:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.515+0000 7fc6dc93b640 1 --2- 172.21.15.12:0/2907118042 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6d8138230 0x7fc6d81cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:46:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.516+0000 7fc6dd13c640 1 --2- 172.21.15.12:0/2907118042 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc6d8137e50 0x7fc6d81ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:46:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.518+0000 7fc6dd93d640 1 -- 172.21.15.12:0/2907118042 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6d813d380 msgr2=0x7fc6d80a1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.518+0000 7fc6dd93d640 1 --2- 172.21.15.12:0/2907118042 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6d813d380 0x7fc6d80a1b80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0030000 tx=0x7fc6d0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.518+0000 7fc6c9ffb640 1 -- 172.21.15.12:0/2907118042 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0041020 con 0x7fc6d813d380 2024-09-11T01:46:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.518+0000 7fc6c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.518+0000 7fc6c9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:46:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.518+0000 7fc6c9ffb640 10 monclient: dump: 2024-09-11T01:46:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:32.533 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.518+0000 7fc6c9ffb640 10 monclient: _finish_auth 0 2024-09-11T01:46:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.518+0000 7fc6c9ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:46:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.518+0000 7fc6c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.518+0000 7fc6c9ffb640 1 -- 172.21.15.12:0/2907118042 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6d00318c0 con 0x7fc6d813d380 2024-09-11T01:46:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.518+0000 7fc6c9ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.518+0000 7fc6c9ffb640 1 -- 172.21.15.12:0/2907118042 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0036040 con 0x7fc6d813d380 2024-09-11T01:46:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.518+0000 7fc6df3a0640 5 monclient: authenticate success, global_id 184244 2024-09-11T01:46:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.518+0000 7fc6c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.518+0000 7fc6c9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.518+0000 7fc6c9ffb640 10 monclient: dump: 2024-09-11T01:46:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:32.535 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:32.535 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:32.535 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:32.535 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:32.535 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:32.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.518+0000 7fc6df3a0640 10 monclient: _renew_subs 2024-09-11T01:46:32.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.518+0000 7fc6df3a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:32.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.518+0000 7fc6df3a0640 1 -- 172.21.15.12:0/2907118042 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6d8121940 con 0x7fc6d813d380 2024-09-11T01:46:32.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.518+0000 7fc6cb7fe640 4 set_mon_vals no callback set 2024-09-11T01:46:32.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.518+0000 7fc6df3a0640 10 monclient: _renew_subs 2024-09-11T01:46:32.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.518+0000 7fc6df3a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:32.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.518+0000 7fc6df3a0640 1 -- 172.21.15.12:0/2907118042 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6d8121660 con 0x7fc6d813d380 2024-09-11T01:46:32.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.519+0000 7fc6df3a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.519+0000 7fc6df3a0640 1 --2- 172.21.15.12:0/2907118042 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6d81cdbf0 0x7fc6d80a20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.519+0000 7fc6df3a0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.519+0000 7fc6df3a0640 1 -- 172.21.15.12:0/2907118042 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d806a060 con 0x7fc6d81cdbf0 2024-09-11T01:46:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.519+0000 7fc6c9ffb640 1 -- 172.21.15.12:0/2907118042 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fc6d003c470 con 0x7fc6d813d380 2024-09-11T01:46:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.519+0000 7fc6c9ffb640 1 --2- 172.21.15.12:0/2907118042 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc6bc03e210 0x7fc6bc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.519+0000 7fc6dd13c640 1 --2- 172.21.15.12:0/2907118042 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6d81cdbf0 0x7fc6d80a20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.519+0000 7fc6dd13c640 10 monclient: get_auth_request con 0x7fc6d81cdbf0 auth_method 0 2024-09-11T01:46:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.519+0000 7fc6dd13c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.519+0000 7fc6dd13c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.519+0000 7fc6dd13c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.519+0000 7fc6c9ffb640 1 -- 172.21.15.12:0/2907118042 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(596..596 src has 507..596) ==== 4847+0+0 (secure 0 0 0) 0x7fc6d0073d60 con 0x7fc6d813d380 2024-09-11T01:46:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.519+0000 7fc6dd13c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.519+0000 7fc6dd13c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:32.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.519+0000 7fc6dd13c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:32.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.519+0000 7fc6dc93b640 1 --2- 172.21.15.12:0/2907118042 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc6bc03e210 0x7fc6bc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:32.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.520+0000 7fc6dc93b640 10 monclient: get_auth_request con 0x7fc6bc03e210 auth_method 0 2024-09-11T01:46:32.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.520+0000 7fc6dd13c640 10 monclient(hunting): handle_auth_done global_id 134665 payload 824 2024-09-11T01:46:32.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.520+0000 7fc6dd13c640 1 -- 172.21.15.12:0/2907118042 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc6d8138640 con 0x7fc6d81cdbf0 2024-09-11T01:46:32.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.522+0000 7fc6dd13c640 1 -- 172.21.15.12:0/2907118042 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6d81cdbf0 msgr2=0x7fc6d80a20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:32.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.522+0000 7fc6dd13c640 1 --2- 172.21.15.12:0/2907118042 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6d81cdbf0 0x7fc6d80a20c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc00fb10 tx=0x7fc6cc004580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:32.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.522+0000 7fc6dc93b640 1 --2- 172.21.15.12:0/2907118042 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc6bc03e210 0x7fc6bc0406d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc6d80a31f0 tx=0x7fc6d4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:32.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.522+0000 7fc6c9ffb640 1 -- 172.21.15.12:0/2907118042 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc018020 con 0x7fc6d81cdbf0 2024-09-11T01:46:32.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.522+0000 7fc6c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:32.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.522+0000 7fc6c9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:46:32.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.522+0000 7fc6c9ffb640 10 monclient: dump: 2024-09-11T01:46:32.540 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:32.540 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:32.540 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.540 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.540 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:32.540 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:32.540 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:32.540 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:32.541 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:32.541 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:32.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.523+0000 7fc6c9ffb640 1 -- 172.21.15.12:0/2907118042 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc6cc00de50 con 0x7fc6d81cdbf0 2024-09-11T01:46:32.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.523+0000 7fc6c9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:32.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.523+0000 7fc6c9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:46:32.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.523+0000 7fc6c9ffb640 1 -- 172.21.15.12:0/2907118042 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6d81cdbf0 msgr2=0x7fc6d80a20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:32.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.523+0000 7fc6c9ffb640 1 --2- 172.21.15.12:0/2907118042 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6d81cdbf0 0x7fc6d80a20c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc00fb10 tx=0x7fc6cc004580 comp rx=0 tx=0).stop 2024-09-11T01:46:32.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.523+0000 7fc6c9ffb640 1 -- 172.21.15.12:0/2907118042 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6d81cdbf0 msgr2=0x7fc6d80a20c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:46:32.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.523+0000 7fc6c9ffb640 1 --2- 172.21.15.12:0/2907118042 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6d81cdbf0 0x7fc6d80a20c0 secure :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc00fb10 tx=0x7fc6cc004580 comp rx=0 tx=0).stop 2024-09-11T01:46:32.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.529+0000 7fc6df3a0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:46:32.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.529+0000 7fc6df3a0640 1 --2- 172.21.15.12:0/2907118042 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6d80a2600 0x7fc6d809e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:32.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.529+0000 7fc6df3a0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:32.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.529+0000 7fc6df3a0640 1 -- 172.21.15.12:0/2907118042 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d80a2a10 con 0x7fc6d80a2600 2024-09-11T01:46:32.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.529+0000 7fc6dd13c640 1 --2- 172.21.15.12:0/2907118042 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6d80a2600 0x7fc6d809e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:32.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.529+0000 7fc6dd13c640 10 monclient: get_auth_request con 0x7fc6d80a2600 auth_method 0 2024-09-11T01:46:32.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.529+0000 7fc6dd13c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:32.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.529+0000 7fc6dd13c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:32.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.529+0000 7fc6dd13c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:32.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.529+0000 7fc6dd13c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:32.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.529+0000 7fc6dd13c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:32.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.529+0000 7fc6dd13c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:32.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.529+0000 7fc6dd13c640 10 monclient(hunting): handle_auth_done global_id 134668 payload 824 2024-09-11T01:46:32.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.529+0000 7fc6dd13c640 1 -- 172.21.15.12:0/2907118042 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc6d80a4150 con 0x7fc6d80a2600 2024-09-11T01:46:32.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.532+0000 7fc6dd13c640 1 -- 172.21.15.12:0/2907118042 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6d80a2600 msgr2=0x7fc6d809e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:32.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.532+0000 7fc6dd13c640 1 --2- 172.21.15.12:0/2907118042 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6d80a2600 0x7fc6d809e420 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc019000 tx=0x7fc6cc0036f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:32.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.532+0000 7fc6c9ffb640 1 -- 172.21.15.12:0/2907118042 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc018030 con 0x7fc6d80a2600 2024-09-11T01:46:32.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.532+0000 7fc6c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:32.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.532+0000 7fc6c9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:46:32.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.532+0000 7fc6c9ffb640 10 monclient: dump: 2024-09-11T01:46:32.543 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:32.543 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:32.543 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.543 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:32.543 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:32.543 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:32.543 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:32.543 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:32.543 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:32.543 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:32.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.532+0000 7fc6c9ffb640 1 -- 172.21.15.12:0/2907118042 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7fc6cc014b80 con 0x7fc6d80a2600 2024-09-11T01:46:32.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.532+0000 7fc6c9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:46:32.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.532+0000 7fc6c9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:46:32.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.532+0000 7fc6c9ffb640 1 -- 172.21.15.12:0/2907118042 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6d80a2600 msgr2=0x7fc6d809e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:32.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.532+0000 7fc6c9ffb640 1 --2- 172.21.15.12:0/2907118042 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6d80a2600 0x7fc6d809e420 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc019000 tx=0x7fc6cc0036f0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.532+0000 7fc6c9ffb640 1 -- 172.21.15.12:0/2907118042 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6d80a2600 msgr2=0x7fc6d809e420 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:46:32.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.532+0000 7fc6c9ffb640 1 --2- 172.21.15.12:0/2907118042 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6d80a2600 0x7fc6d809e420 secure :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc019000 tx=0x7fc6cc0036f0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.544 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:46:32.544 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:46:32.544 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:46:32.545 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:46:32.545 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 162, 2024-09-11T01:46:32.545 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 325495, 2024-09-11T01:46:32.545 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:46:32.545 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:46:32.545 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:46:32.545 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:46:32.545 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:32.545 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:46:32.545 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:46:32.545 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:46:32.545 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:46:32.545 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:32.546 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:32.546 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:32.546 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:32.546 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:32.546 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:32.546 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:32.546 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:32.546 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:32.546 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:32.546 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:32.546 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:32.546 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:46:32.546 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:46:32.547 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:32.547 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:32.547 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:32.547 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:32.547 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:32.547 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:32.547 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:32.547 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:32.547 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:32.547 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:32.547 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:32.547 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:32.547 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.548 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:46:32.548 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:46:32.548 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:46:32.548 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:46:32.548 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:46:32.548 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:46:32.548 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:46:32.548 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:46:32.548 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:46:32.548 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:46:32.548 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:46:32.548 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:46:32.548 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:46:32.549 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:46:32.549 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:46:32.549 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:46:32.549 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:46:32.549 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:32.549 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:32.549 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:32.549 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:32.549 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:32.549 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:32.549 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:32.549 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:32.549 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:32.550 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:32.550 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:32.550 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:32.550 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:46:32.550 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.550 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:46:32.550 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.550 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:46:32.550 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:46:32.550 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:32.550 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:32.550 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.550 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:32.551 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:46:32.551 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:32.551 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.551 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.551 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:32.551 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:46:32.551 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:32.551 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:32.551 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:32.551 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.551 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:46:32.551 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:46:32.551 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:32.552 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:32.552 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:32.552 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.552 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.552 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:46:32.552 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:46:32.552 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:32.552 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:32.552 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.552 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:32.552 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:46:32.552 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:32.552 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.553 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.553 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:32.553 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:46:32.553 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:32.553 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:32.553 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:32.553 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.553 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:46:32.553 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:46:32.553 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:32.553 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:32.553 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:32.553 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.553 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.554 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:46:32.554 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:46:32.554 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:32.554 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:32.554 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.554 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:32.554 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:46:32.554 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:32.554 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.554 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.554 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:32.554 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:46:32.554 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:32.554 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:32.555 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:32.555 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.555 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:46:32.555 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:46:32.555 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:32.555 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:32.555 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:32.555 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:32.555 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:32.555 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.555 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:46:32.555 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:46:32.555 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.556 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:32.556 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:32.556 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:46:32.556 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:32.556 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:32.556 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:46:32.556 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.556 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:32.556 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:32.556 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:46:32.556 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:32.556 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:32.556 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:46:32.557 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.557 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:32.557 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:32.557 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:46:32.557 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:32.557 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:32.557 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:46:32.557 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:32.557 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:32.557 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:32.557 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:46:32.557 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:32.557 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:32.557 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:32.557 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:46:32.558 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:46:32.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.533+0000 7fc6df3a0640 1 -- 172.21.15.12:0/2907118042 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc6bc03e210 msgr2=0x7fc6bc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:32.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.533+0000 7fc6df3a0640 1 --2- 172.21.15.12:0/2907118042 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc6bc03e210 0x7fc6bc0406d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc6d80a31f0 tx=0x7fc6d4008040 comp rx=0 tx=0).stop 2024-09-11T01:46:32.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.533+0000 7fc6df3a0640 10 monclient: shutdown 2024-09-11T01:46:32.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.533+0000 7fc6df3a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:32.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.533+0000 7fc6df3a0640 1 -- 172.21.15.12:0/2907118042 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6d813d380 msgr2=0x7fc6d80a1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:32.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.533+0000 7fc6df3a0640 1 --2- 172.21.15.12:0/2907118042 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6d813d380 0x7fc6d80a1b80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0030000 tx=0x7fc6d0004a40 comp rx=0 tx=0).stop 2024-09-11T01:46:32.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.533+0000 7fc6dd93d640 1 -- 172.21.15.12:0/2907118042 reap_dead start 2024-09-11T01:46:32.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.533+0000 7fc6df3a0640 1 -- 172.21.15.12:0/2907118042 shutdown_connections 2024-09-11T01:46:32.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.533+0000 7fc6df3a0640 1 --2- 172.21.15.12:0/2907118042 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc6d813d380 0x7fc6d80a1b80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:32.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.533+0000 7fc6df3a0640 1 -- 172.21.15.12:0/2907118042 >> 172.21.15.12:0/2907118042 conn(0x7fc6d80a7020 msgr2=0x7fc6d813e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:32.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.533+0000 7fc6df3a0640 1 -- 172.21.15.12:0/2907118042 shutdown_connections 2024-09-11T01:46:32.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:32.533+0000 7fc6df3a0640 1 -- 172.21.15.12:0/2907118042 wait complete. 2024-09-11T01:46:32.560 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:46:32.560 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-11T01:46:32.560 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-11T01:46:32.560 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-11T01:46:32.561 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-11T01:46:32.561 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T01:46:32.561 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:46:32.571 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T01:46:32.571 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-11T01:46:32.572 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-11T01:46:32.572 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T01:46:47.573 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-11T01:46:47.573 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-11T01:46:47.573 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:46:47.574 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:46:47.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.685+0000 7f1e58687640 10 monclient: get_monmap_and_config 2024-09-11T01:46:47.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.685+0000 7f1e58687640 10 monclient: build_initial_monmap 2024-09-11T01:46:47.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.685+0000 7f1e58687640 10 monclient: monmap: 2024-09-11T01:46:47.687 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:47.687 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:47.687 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:46:47.685917+0000 2024-09-11T01:46:47.687 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:46:47.685917+0000 2024-09-11T01:46:47.687 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:47.687 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:47.688 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:46:47.688 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:46:47.688 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:46:47.688 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:46:47.688 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:46:47.688 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:47.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.685+0000 7f1e58687640 1 Processor -- start 2024-09-11T01:46:47.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.685+0000 7f1e58687640 1 -- start start 2024-09-11T01:46:47.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.685+0000 7f1e58687640 10 monclient: init 2024-09-11T01:46:47.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.686+0000 7f1e58687640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:47.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.686+0000 7f1e58687640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-11T01:46:47.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.686+0000 7f1e58687640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1e50070130 addr v1:172.21.15.12:6789/0 2024-09-11T01:46:47.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.686+0000 7f1e58687640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1e50136630 0x7f1e5013ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:47.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.686+0000 7f1e58687640 10 monclient(hunting): picked mon.noname-a con 0x7f1e50136630 addr v2:172.21.15.12:3300/0 2024-09-11T01:46:47.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.686+0000 7f1e58687640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1e500741b0 addr v1:172.21.15.71:6789/0 2024-09-11T01:46:47.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.686+0000 7f1e58687640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e50135ad0 con 0x7f1e50070130 2024-09-11T01:46:47.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.686+0000 7f1e58687640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e50073440 con 0x7f1e500741b0 2024-09-11T01:46:47.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.686+0000 7f1e58687640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:47.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.686+0000 7f1e58687640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f1e500699b0 con 0x7f1e50136630 2024-09-11T01:46:47.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.686+0000 7f1e58687640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:47.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.686+0000 7f1e56423640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7f1e50070130 legacy=0x7f1e50135f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:46:47.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.686+0000 7f1e58687640 10 monclient(hunting): authenticate will time out at 1814.128682s 2024-09-11T01:46:47.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.686+0000 7f1e55c22640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7f1e50136630 msgr2=0x7f1e5013ad40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:46:47.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.686+0000 7f1e55c22640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1e50136630 0x7f1e5013ad40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:46:47.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.686+0000 7f1e56c24640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f1e500741b0 0x7f1e5013d290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.12:52410/0 (socket says 172.21.15.12:52410) 2024-09-11T01:46:47.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.686+0000 7f1e56c24640 1 -- 172.21.15.12:0/2333472780 learned_addr learned my addr 172.21.15.12:0/2333472780 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:46:47.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.689+0000 7f1e56c24640 1 -- 172.21.15.12:0/2333472780 >> v1:172.21.15.71:6789/0 conn(0x7f1e500741b0 legacy=0x7f1e5013d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:47.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.887+0000 7f1e55c22640 1 -- 172.21.15.12:0/2333472780 >> v2:172.21.15.12:3300/0 conn(0x7f1e50136630 msgr2=0x7f1e5013ad40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:46:47.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.887+0000 7f1e56423640 1 -- 172.21.15.12:0/2333472780 >> v1:172.21.15.12:6789/0 conn(0x7f1e50070130 legacy=0x7f1e50135f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:46:47.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:47.887+0000 7f1e55c22640 1 --2- 172.21.15.12:0/2333472780 >> v2:172.21.15.12:3300/0 conn(0x7f1e50136630 0x7f1e5013ad40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:46:48.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:48.288+0000 7f1e55c22640 1 -- 172.21.15.12:0/2333472780 >> v2:172.21.15.12:3300/0 conn(0x7f1e50136630 msgr2=0x7f1e5013ad40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:46:48.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:48.288+0000 7f1e56423640 1 -- 172.21.15.12:0/2333472780 >> v1:172.21.15.12:6789/0 conn(0x7f1e50070130 legacy=0x7f1e50135f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:46:48.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:48.288+0000 7f1e55c22640 1 --2- 172.21.15.12:0/2333472780 >> v2:172.21.15.12:3300/0 conn(0x7f1e50136630 0x7f1e5013ad40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:46:49.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:49.089+0000 7f1e56423640 1 -- 172.21.15.12:0/2333472780 >> v1:172.21.15.12:6789/0 conn(0x7f1e50070130 legacy=0x7f1e50135f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:46:49.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:49.089+0000 7f1e55c22640 1 -- 172.21.15.12:0/2333472780 >> v2:172.21.15.12:3300/0 conn(0x7f1e50136630 msgr2=0x7f1e5013ad40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:46:49.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:49.089+0000 7f1e55c22640 1 --2- 172.21.15.12:0/2333472780 >> v2:172.21.15.12:3300/0 conn(0x7f1e50136630 0x7f1e5013ad40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:46:50.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.686+0000 7f1e47fff640 10 monclient(hunting): tick 2024-09-11T01:46:50.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.686+0000 7f1e47fff640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:46:50.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.686+0000 7f1e47fff640 1 monclient(hunting): continuing hunt 2024-09-11T01:46:50.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.686+0000 7f1e47fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:46:50.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.686+0000 7f1e47fff640 1 -- 172.21.15.12:0/2333472780 >> v2:172.21.15.12:3300/0 conn(0x7f1e50136630 msgr2=0x7f1e5013ad40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:46:50.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.686+0000 7f1e47fff640 1 --2- 172.21.15.12:0/2333472780 >> v2:172.21.15.12:3300/0 conn(0x7f1e50136630 0x7f1e5013ad40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:50.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.686+0000 7f1e47fff640 1 -- 172.21.15.12:0/2333472780 >> v1:172.21.15.71:6789/0 conn(0x7f1e500741b0 legacy=0x7f1e5013d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:50.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.686+0000 7f1e47fff640 1 -- 172.21.15.12:0/2333472780 >> v1:172.21.15.12:6789/0 conn(0x7f1e50070130 legacy=0x7f1e50135f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:46:50.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.686+0000 7f1e47fff640 10 monclient: _add_conns ranks=[2,4] 2024-09-11T01:46:50.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.686+0000 7f1e47fff640 1 --2- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e34003740 0x7f1e34005b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:50.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.686+0000 7f1e47fff640 10 monclient(hunting): picked mon.noname-b con 0x7f1e34003740 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:50.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.686+0000 7f1e47fff640 1 --2- 172.21.15.12:0/2333472780 >> v2:172.21.15.71:3300/0 conn(0x7f1e34006270 0x7f1e34008730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:50.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.686+0000 7f1e47fff640 10 monclient(hunting): picked mon.noname-c con 0x7f1e34006270 addr v2:172.21.15.71:3300/0 2024-09-11T01:46:50.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.686+0000 7f1e47fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:50.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.686+0000 7f1e47fff640 1 -- 172.21.15.12:0/2333472780 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f1e34008dc0 con 0x7f1e34006270 2024-09-11T01:46:50.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.686+0000 7f1e47fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:50.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.686+0000 7f1e47fff640 1 -- 172.21.15.12:0/2333472780 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1e34008f70 con 0x7f1e34003740 2024-09-11T01:46:50.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.686+0000 7f1e47fff640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:50.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.687+0000 7f1e56423640 1 --2- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e34003740 0x7f1e34005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:50.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.687+0000 7f1e56423640 10 monclient(hunting): get_auth_request con 0x7f1e34003740 auth_method 0 2024-09-11T01:46:50.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.687+0000 7f1e56423640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:50.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.687+0000 7f1e56423640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:50.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.687+0000 7f1e56423640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:50.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.687+0000 7f1e56423640 1 -- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e34003740 msgr2=0x7f1e34005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:46:50.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.687+0000 7f1e56423640 1 -- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e34003740 msgr2=0x7f1e34005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:46:50.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.687+0000 7f1e56423640 1 --2- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e34003740 0x7f1e34005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:46:50.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.687+0000 7f1e55c22640 1 --2- 172.21.15.12:0/2333472780 >> v2:172.21.15.71:3300/0 conn(0x7f1e34006270 0x7f1e34008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:50.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.687+0000 7f1e56423640 1 --2- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e34003740 0x7f1e34005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:46:50.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.687+0000 7f1e55c22640 10 monclient(hunting): get_auth_request con 0x7f1e34006270 auth_method 0 2024-09-11T01:46:50.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.687+0000 7f1e55c22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:50.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.687+0000 7f1e55c22640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:50.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.687+0000 7f1e55c22640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:50.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.687+0000 7f1e55c22640 1 -- 172.21.15.12:0/2333472780 >> v2:172.21.15.71:3300/0 conn(0x7f1e34006270 msgr2=0x7f1e34008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:46:50.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.687+0000 7f1e55c22640 1 -- 172.21.15.12:0/2333472780 >> v2:172.21.15.71:3300/0 conn(0x7f1e34006270 msgr2=0x7f1e34008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:46:50.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.687+0000 7f1e55c22640 1 --2- 172.21.15.12:0/2333472780 >> v2:172.21.15.71:3300/0 conn(0x7f1e34006270 0x7f1e34008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:46:50.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.687+0000 7f1e55c22640 1 --2- 172.21.15.12:0/2333472780 >> v2:172.21.15.71:3300/0 conn(0x7f1e34006270 0x7f1e34008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:46:50.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.888+0000 7f1e56423640 1 --2- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e34003740 0x7f1e34005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:50.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.889+0000 7f1e56423640 10 monclient(hunting): get_auth_request con 0x7f1e34003740 auth_method 0 2024-09-11T01:46:50.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.889+0000 7f1e56423640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:50.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.889+0000 7f1e56423640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:50.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.889+0000 7f1e56423640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:46:50.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.889+0000 7f1e56423640 1 -- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e34003740 msgr2=0x7f1e34005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:46:50.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.889+0000 7f1e56423640 1 -- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e34003740 msgr2=0x7f1e34005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:46:50.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.889+0000 7f1e56423640 1 --2- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e34003740 0x7f1e34005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:46:50.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.889+0000 7f1e55c22640 1 --2- 172.21.15.12:0/2333472780 >> v2:172.21.15.71:3300/0 conn(0x7f1e34006270 0x7f1e34008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:50.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.889+0000 7f1e56423640 1 --2- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e34003740 0x7f1e34005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:46:50.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.889+0000 7f1e55c22640 10 monclient(hunting): get_auth_request con 0x7f1e34006270 auth_method 0 2024-09-11T01:46:50.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.889+0000 7f1e55c22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:50.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.889+0000 7f1e55c22640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:50.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.889+0000 7f1e55c22640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:46:50.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.889+0000 7f1e55c22640 1 -- 172.21.15.12:0/2333472780 >> v2:172.21.15.71:3300/0 conn(0x7f1e34006270 msgr2=0x7f1e34008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:46:50.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.889+0000 7f1e55c22640 1 -- 172.21.15.12:0/2333472780 >> v2:172.21.15.71:3300/0 conn(0x7f1e34006270 msgr2=0x7f1e34008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:46:50.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.889+0000 7f1e55c22640 1 --2- 172.21.15.12:0/2333472780 >> v2:172.21.15.71:3300/0 conn(0x7f1e34006270 0x7f1e34008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:46:50.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:50.889+0000 7f1e55c22640 1 --2- 172.21.15.12:0/2333472780 >> v2:172.21.15.71:3300/0 conn(0x7f1e34006270 0x7f1e34008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:46:51.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:51.290+0000 7f1e56423640 1 --2- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e34003740 0x7f1e34005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:51.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:51.290+0000 7f1e56423640 10 monclient(hunting): get_auth_request con 0x7f1e34003740 auth_method 0 2024-09-11T01:46:51.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:51.290+0000 7f1e56423640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:51.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:51.290+0000 7f1e56423640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:51.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:51.290+0000 7f1e56423640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:46:51.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:51.291+0000 7f1e56423640 1 -- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e34003740 msgr2=0x7f1e34005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:46:51.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:51.291+0000 7f1e56423640 1 -- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e34003740 msgr2=0x7f1e34005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:46:51.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:51.291+0000 7f1e56423640 1 --2- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e34003740 0x7f1e34005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:46:51.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:51.291+0000 7f1e56423640 1 --2- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e34003740 0x7f1e34005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:46:51.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:51.291+0000 7f1e55c22640 1 --2- 172.21.15.12:0/2333472780 >> v2:172.21.15.71:3300/0 conn(0x7f1e34006270 0x7f1e34008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:51.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:51.291+0000 7f1e55c22640 10 monclient(hunting): get_auth_request con 0x7f1e34006270 auth_method 0 2024-09-11T01:46:51.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:51.291+0000 7f1e55c22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:51.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:51.291+0000 7f1e55c22640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:51.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:51.291+0000 7f1e55c22640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:46:51.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:51.291+0000 7f1e55c22640 1 -- 172.21.15.12:0/2333472780 >> v2:172.21.15.71:3300/0 conn(0x7f1e34006270 msgr2=0x7f1e34008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:46:51.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:51.291+0000 7f1e55c22640 1 -- 172.21.15.12:0/2333472780 >> v2:172.21.15.71:3300/0 conn(0x7f1e34006270 msgr2=0x7f1e34008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:46:51.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:51.291+0000 7f1e55c22640 1 --2- 172.21.15.12:0/2333472780 >> v2:172.21.15.71:3300/0 conn(0x7f1e34006270 0x7f1e34008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:46:51.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:51.291+0000 7f1e55c22640 1 --2- 172.21.15.12:0/2333472780 >> v2:172.21.15.71:3300/0 conn(0x7f1e34006270 0x7f1e34008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:46:52.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:52.092+0000 7f1e56423640 1 --2- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e34003740 0x7f1e34005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:52.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:52.092+0000 7f1e56423640 10 monclient(hunting): get_auth_request con 0x7f1e34003740 auth_method 0 2024-09-11T01:46:52.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:52.092+0000 7f1e56423640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:52.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:52.092+0000 7f1e56423640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:52.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:52.092+0000 7f1e56423640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:46:52.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:52.092+0000 7f1e56423640 1 -- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e34003740 msgr2=0x7f1e34005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:46:52.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:52.092+0000 7f1e56423640 1 -- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e34003740 msgr2=0x7f1e34005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:46:52.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:52.092+0000 7f1e56423640 1 --2- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e34003740 0x7f1e34005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:46:52.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:52.092+0000 7f1e56423640 1 --2- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e34003740 0x7f1e34005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:46:52.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:52.092+0000 7f1e55c22640 1 --2- 172.21.15.12:0/2333472780 >> v2:172.21.15.71:3300/0 conn(0x7f1e34006270 0x7f1e34008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:52.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:52.092+0000 7f1e55c22640 10 monclient(hunting): get_auth_request con 0x7f1e34006270 auth_method 0 2024-09-11T01:46:52.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:52.092+0000 7f1e55c22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:52.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:52.092+0000 7f1e55c22640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:52.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:52.092+0000 7f1e55c22640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:46:52.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:52.092+0000 7f1e55c22640 1 -- 172.21.15.12:0/2333472780 >> v2:172.21.15.71:3300/0 conn(0x7f1e34006270 msgr2=0x7f1e34008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:46:52.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:52.092+0000 7f1e55c22640 1 -- 172.21.15.12:0/2333472780 >> v2:172.21.15.71:3300/0 conn(0x7f1e34006270 msgr2=0x7f1e34008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:46:52.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:52.092+0000 7f1e55c22640 1 --2- 172.21.15.12:0/2333472780 >> v2:172.21.15.71:3300/0 conn(0x7f1e34006270 0x7f1e34008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:46:52.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:52.092+0000 7f1e55c22640 1 --2- 172.21.15.12:0/2333472780 >> v2:172.21.15.71:3300/0 conn(0x7f1e34006270 0x7f1e34008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:46:53.605 INFO:tasks.rados.rados.0.smithi071.stdout:2653: done (2 left) 2024-09-11T01:46:53.605 INFO:tasks.rados.rados.0.smithi071.stdout:2655: done (1 left) 2024-09-11T01:46:53.605 INFO:tasks.rados.rados.0.smithi071.stdout:2656: done (0 left) 2024-09-11T01:46:53.605 INFO:tasks.rados.rados.0.smithi071.stdout:2657: read oid 26 snap -1 2024-09-11T01:46:53.605 INFO:tasks.rados.rados.0.smithi071.stdout:2657: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:46:53.606 INFO:tasks.rados.rados.0.smithi071.stdout:2658: rollback oid 10 current snap is 298 2024-09-11T01:46:53.606 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 10 to 236 2024-09-11T01:46:53.606 INFO:tasks.rados.rados.0.smithi071.stdout:2659: snap_remove snap 282 2024-09-11T01:46:53.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e47fff640 10 monclient(hunting): tick 2024-09-11T01:46:53.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e47fff640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:46:53.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e47fff640 1 monclient(hunting): continuing hunt 2024-09-11T01:46:53.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e47fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:46:53.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e47fff640 1 -- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e34003740 msgr2=0x7f1e34005b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:46:53.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e47fff640 1 --2- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e34003740 0x7f1e34005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:53.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e47fff640 1 -- 172.21.15.12:0/2333472780 >> v2:172.21.15.71:3300/0 conn(0x7f1e34006270 msgr2=0x7f1e34008730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:46:53.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e47fff640 1 --2- 172.21.15.12:0/2333472780 >> v2:172.21.15.71:3300/0 conn(0x7f1e34006270 0x7f1e34008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:53.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e47fff640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-11T01:46:53.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e47fff640 1 --2- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e340094a0 0x7f1e34009880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:53.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e47fff640 10 monclient(hunting): picked mon.noname-b con 0x7f1e340094a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:53.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e56c24640 1 -- 172.21.15.12:0/2333472780 reap_dead start 2024-09-11T01:46:53.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e47fff640 1 --2- 172.21.15.12:0/2333472780 >> v2:172.21.15.12:3300/0 conn(0x7f1e50136630 0x7f1e34005b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:53.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e47fff640 10 monclient(hunting): picked mon.noname-a con 0x7f1e50136630 addr v2:172.21.15.12:3300/0 2024-09-11T01:46:53.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e47fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1e50070130 addr v1:172.21.15.12:6789/0 2024-09-11T01:46:53.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e47fff640 1 -- 172.21.15.12:0/2333472780 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e50073720 con 0x7f1e50070130 2024-09-11T01:46:53.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e47fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:53.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e47fff640 1 -- 172.21.15.12:0/2333472780 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f1e34008dc0 con 0x7f1e50136630 2024-09-11T01:46:53.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e47fff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:53.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e47fff640 1 -- 172.21.15.12:0/2333472780 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1e34008f70 con 0x7f1e340094a0 2024-09-11T01:46:53.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e47fff640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:53.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e55c22640 1 -- 172.21.15.12:0/2333472780 >> v2:172.21.15.12:3300/0 conn(0x7f1e50136630 msgr2=0x7f1e34005b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:46:53.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e56c24640 1 -- 172.21.15.12:0/2333472780 >> v1:172.21.15.12:6789/0 conn(0x7f1e50070130 legacy=0x7f1e3400df10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:46:53.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e55c22640 1 --2- 172.21.15.12:0/2333472780 >> v2:172.21.15.12:3300/0 conn(0x7f1e50136630 0x7f1e34005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:46:53.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e56423640 1 --2- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e340094a0 0x7f1e34009880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:53.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e56423640 10 monclient(hunting): get_auth_request con 0x7f1e340094a0 auth_method 0 2024-09-11T01:46:53.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e56423640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:53.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e56423640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:53.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e56423640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:53.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e56423640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:53.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e56423640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:53.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e56423640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:53.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e56423640 10 monclient(hunting): handle_auth_done global_id 184247 payload 274 2024-09-11T01:46:53.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e56423640 1 -- 172.21.15.12:0/2333472780 >> v2:172.21.15.12:3300/0 conn(0x7f1e50136630 msgr2=0x7f1e34005b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:46:53.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e56423640 1 --2- 172.21.15.12:0/2333472780 >> v2:172.21.15.12:3300/0 conn(0x7f1e50136630 0x7f1e34005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:53.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e56423640 1 -- 172.21.15.12:0/2333472780 >> v1:172.21.15.12:6789/0 conn(0x7f1e50070130 legacy=0x7f1e3400df10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:46:53.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e56423640 10 monclient: _finish_hunting 0 2024-09-11T01:46:53.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e56423640 1 monclient: found mon.noname-b 2024-09-11T01:46:53.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e56423640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:53.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e56423640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:46:53.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.687+0000 7f1e56423640 1 -- 172.21.15.12:0/2333472780 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e50135ad0 con 0x7f1e340094a0 2024-09-11T01:46:53.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.690+0000 7f1e56423640 1 -- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e340094a0 msgr2=0x7f1e34009880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:53.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.690+0000 7f1e56423640 1 --2- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e340094a0 0x7f1e34009880 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1e340036d0 tx=0x7f1e4c02b340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31693930a0af0f80 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:53.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.690+0000 7f1e55421640 1 -- 172.21.15.12:0/2333472780 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c036480 con 0x7f1e340094a0 2024-09-11T01:46:53.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.691+0000 7f1e55421640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:53.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.691+0000 7f1e55421640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:46:53.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.691+0000 7f1e55421640 10 monclient: dump: 2024-09-11T01:46:53.699 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:53.699 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:53.699 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:53.699 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:53.699 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:53.699 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:53.699 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:53.699 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:53.699 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:53.699 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:53.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.691+0000 7f1e55421640 10 monclient: _finish_auth 0 2024-09-11T01:46:53.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.691+0000 7f1e55421640 10 monclient: _check_auth_tickets 2024-09-11T01:46:53.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.691+0000 7f1e55421640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:53.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.691+0000 7f1e55421640 1 -- 172.21.15.12:0/2333472780 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e4c036620 con 0x7f1e340094a0 2024-09-11T01:46:53.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.691+0000 7f1e58687640 5 monclient: authenticate success, global_id 184247 2024-09-11T01:46:53.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.691+0000 7f1e58687640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:46:53.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.691+0000 7f1e55421640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:53.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.691+0000 7f1e58687640 10 monclient: get_monmap_and_config success 2024-09-11T01:46:53.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.691+0000 7f1e55421640 1 -- 172.21.15.12:0/2333472780 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c036a10 con 0x7f1e340094a0 2024-09-11T01:46:53.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.691+0000 7f1e58687640 4 set_mon_vals no callback set 2024-09-11T01:46:53.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.691+0000 7f1e55421640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:53.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.691+0000 7f1e55421640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:53.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.691+0000 7f1e55421640 10 monclient: dump: 2024-09-11T01:46:53.700 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:53.700 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:53.700 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:53.700 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:53.701 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:53.701 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:53.701 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:53.701 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:53.701 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:53.701 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:53.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.691+0000 7f1e58687640 10 monclient: shutdown 2024-09-11T01:46:53.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.691+0000 7f1e58687640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:53.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.691+0000 7f1e58687640 1 -- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e340094a0 msgr2=0x7f1e34009880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:53.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.691+0000 7f1e58687640 1 --2- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e340094a0 0x7f1e34009880 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1e340036d0 tx=0x7f1e4c02b340 comp rx=0 tx=0).stop 2024-09-11T01:46:53.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.692+0000 7f1e58687640 1 -- 172.21.15.12:0/2333472780 shutdown_connections 2024-09-11T01:46:53.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.692+0000 7f1e58687640 1 --2- 172.21.15.12:0/2333472780 >> v2:172.21.15.12:3300/0 conn(0x7f1e50136630 0x7f1e34005b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:53.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.692+0000 7f1e58687640 1 --2- 172.21.15.12:0/2333472780 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e340094a0 0x7f1e34009880 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:53.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.692+0000 7f1e58687640 1 -- 172.21.15.12:0/2333472780 >> 172.21.15.12:0/2333472780 conn(0x7f1e500a3de0 msgr2=0x7f1e50068950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:53.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.692+0000 7f1e58687640 1 -- 172.21.15.12:0/2333472780 shutdown_connections 2024-09-11T01:46:53.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.692+0000 7f1e58687640 1 -- 172.21.15.12:0/2333472780 wait complete. 2024-09-11T01:46:53.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.692+0000 7f1e58687640 10 monclient: build_initial_monmap 2024-09-11T01:46:53.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.692+0000 7f1e58687640 10 monclient: monmap: 2024-09-11T01:46:53.702 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:53.702 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:46:53.702 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:46:53.702 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:46:53.702 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:53.702 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:53.702 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:46:53.702 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:46:53.702 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:46:53.703 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:53.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.693+0000 7f1e58687640 1 Processor -- start 2024-09-11T01:46:53.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.693+0000 7f1e58687640 1 -- start start 2024-09-11T01:46:53.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.693+0000 7f1e58687640 10 monclient: init 2024-09-11T01:46:53.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e58687640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:53.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e58687640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:46:53.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e58687640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1e340094a0 0x7f1e5012ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:53.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e58687640 10 monclient(hunting): picked mon.noname-b con 0x7f1e340094a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:46:53.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e58687640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e50070130 0x7f1e5012f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:53.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e58687640 10 monclient(hunting): picked mon.noname-a con 0x7f1e50070130 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:46:53.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e58687640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e50136630 0x7f1e5009c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:53.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e58687640 10 monclient(hunting): picked mon.noname-c con 0x7f1e50136630 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:53.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e58687640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:53.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e58687640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1e50135f20 con 0x7f1e50070130 2024-09-11T01:46:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e58687640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e58687640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1e50070510 con 0x7f1e340094a0 2024-09-11T01:46:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e58687640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e58687640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1e501360a0 con 0x7f1e50136630 2024-09-11T01:46:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e58687640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e55c22640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e50070130 msgr2=0x7f1e5012f0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:46:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e58687640 10 monclient(hunting): authenticate will time out at 1820.136182s 2024-09-11T01:46:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e55c22640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e50070130 0x7f1e5012f0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:46:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56c24640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e50136630 0x7f1e5009c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56c24640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e50136630 0x7f1e5009c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:58868/0 (socket says 172.21.15.12:58868) 2024-09-11T01:46:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56c24640 1 -- 172.21.15.12:0/1022207873 learned_addr learned my addr 172.21.15.12:0/1022207873 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56c24640 10 monclient(hunting): get_auth_request con 0x7f1e50136630 auth_method 0 2024-09-11T01:46:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56c24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:53.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56c24640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:53.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56c24640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:53.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56c24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:53.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56c24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:53.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56c24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:53.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56423640 1 --2- 172.21.15.12:0/1022207873 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1e340094a0 0x7f1e5012ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:53.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56423640 10 monclient(hunting): get_auth_request con 0x7f1e340094a0 auth_method 0 2024-09-11T01:46:53.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56423640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:53.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56423640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:53.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56423640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:53.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56423640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:53.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56423640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:53.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56423640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:53.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56c24640 10 monclient(hunting): handle_auth_done global_id 184250 payload 824 2024-09-11T01:46:53.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56c24640 1 -- 172.21.15.12:0/1022207873 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1e340094a0 msgr2=0x7f1e5012ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:53.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56c24640 1 --2- 172.21.15.12:0/1022207873 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1e340094a0 0x7f1e5012ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:53.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56c24640 1 -- 172.21.15.12:0/1022207873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e50070130 msgr2=0x7f1e5012f0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:46:53.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56c24640 1 --2- 172.21.15.12:0/1022207873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e50070130 0x7f1e5012f0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:53.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56c24640 10 monclient: _finish_hunting 0 2024-09-11T01:46:53.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56c24640 1 monclient: found mon.noname-c 2024-09-11T01:46:53.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56c24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:53.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56c24640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:46:53.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.694+0000 7f1e56c24640 1 -- 172.21.15.12:0/1022207873 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e5009e280 con 0x7f1e50136630 2024-09-11T01:46:53.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.697+0000 7f1e56c24640 1 -- 172.21.15.12:0/1022207873 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e50136630 msgr2=0x7f1e5009c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:53.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.697+0000 7f1e56c24640 1 --2- 172.21.15.12:0/1022207873 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e50136630 0x7f1e5009c8b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1e50073000 tx=0x7f1e4800ce10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:53.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.697+0000 7f1e46ffd640 1 -- 172.21.15.12:0/1022207873 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e48017070 con 0x7f1e50136630 2024-09-11T01:46:53.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.697+0000 7f1e46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:53.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.697+0000 7f1e46ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:46:53.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.697+0000 7f1e46ffd640 10 monclient: dump: 2024-09-11T01:46:53.709 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:53.709 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:53.709 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:53.709 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:53.709 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:53.709 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:53.709 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:53.709 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:53.709 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:53.709 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:53.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.697+0000 7f1e46ffd640 10 monclient: _finish_auth 0 2024-09-11T01:46:53.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.697+0000 7f1e46ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:46:53.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.697+0000 7f1e46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:53.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.697+0000 7f1e46ffd640 1 -- 172.21.15.12:0/1022207873 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e4800abd0 con 0x7f1e50136630 2024-09-11T01:46:53.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.697+0000 7f1e46ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:53.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.697+0000 7f1e46ffd640 1 -- 172.21.15.12:0/1022207873 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e48012400 con 0x7f1e50136630 2024-09-11T01:46:53.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.697+0000 7f1e58687640 5 monclient: authenticate success, global_id 184250 2024-09-11T01:46:53.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.697+0000 7f1e58687640 10 monclient: _renew_subs 2024-09-11T01:46:53.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.697+0000 7f1e58687640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:53.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.697+0000 7f1e58687640 1 -- 172.21.15.12:0/1022207873 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e500a05e0 con 0x7f1e50136630 2024-09-11T01:46:53.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.697+0000 7f1e46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:53.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.697+0000 7f1e46ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:53.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.697+0000 7f1e54c20640 4 set_mon_vals no callback set 2024-09-11T01:46:53.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.697+0000 7f1e46ffd640 10 monclient: dump: 2024-09-11T01:46:53.711 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:53.711 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:53.711 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:53.711 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:53.711 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:53.711 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:53.711 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:53.711 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:53.711 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:53.711 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:53.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.697+0000 7f1e58687640 10 monclient: _renew_subs 2024-09-11T01:46:53.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.697+0000 7f1e58687640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:53.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.697+0000 7f1e58687640 1 -- 172.21.15.12:0/1022207873 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e5009ede0 con 0x7f1e50136630 2024-09-11T01:46:53.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.698+0000 7f1e46ffd640 1 -- 172.21.15.12:0/1022207873 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f1e48007070 con 0x7f1e50136630 2024-09-11T01:46:53.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.698+0000 7f1e58687640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:53.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.698+0000 7f1e58687640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:53.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.698+0000 7f1e58687640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:53.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.698+0000 7f1e58687640 1 -- 172.21.15.12:0/1022207873 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e50132460 con 0x7f1e50136630 2024-09-11T01:46:53.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.698+0000 7f1e46ffd640 1 --2- 172.21.15.12:0/1022207873 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1e3403f780 0x7f1e34041c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:53.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.698+0000 7f1e46ffd640 1 -- 172.21.15.12:0/1022207873 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(597..597 src has 507..597) ==== 4819+0+0 (secure 0 0 0) 0x7f1e48052aa0 con 0x7f1e50136630 2024-09-11T01:46:53.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.699+0000 7f1e56423640 1 --2- 172.21.15.12:0/1022207873 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1e3403f780 0x7f1e34041c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:53.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.699+0000 7f1e56423640 10 monclient: get_auth_request con 0x7f1e3403f780 auth_method 0 2024-09-11T01:46:53.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.702+0000 7f1e56423640 1 --2- 172.21.15.12:0/1022207873 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1e3403f780 0x7f1e34041c40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c035000 tx=0x7f1e4c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:53.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.704+0000 7f1e46ffd640 1 -- 172.21.15.12:0/1022207873 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1e4801ac60 con 0x7f1e50136630 2024-09-11T01:46:53.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.704+0000 7f1e46ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:53.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.704+0000 7f1e46ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:46:53.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.965+0000 7f1e58687640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:46:53.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.965+0000 7f1e58687640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:46:53.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.965+0000 7f1e58687640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:53.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.965+0000 7f1e58687640 1 -- 172.21.15.12:0/1022207873 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1e5012fd60 con 0x7f1e50136630 2024-09-11T01:46:53.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.965+0000 7f1e46ffd640 1 -- 172.21.15.12:0/1022207873 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f1e48051470 con 0x7f1e50136630 2024-09-11T01:46:53.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.965+0000 7f1e46ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:46:53.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.965+0000 7f1e46ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:46:53.968 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":164,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:46:53.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.967+0000 7f1e58687640 1 -- 172.21.15.12:0/1022207873 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1e3403f780 msgr2=0x7f1e34041c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:53.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.967+0000 7f1e58687640 1 --2- 172.21.15.12:0/1022207873 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1e3403f780 0x7f1e34041c40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c035000 tx=0x7f1e4c030000 comp rx=0 tx=0).stop 2024-09-11T01:46:53.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.967+0000 7f1e58687640 10 monclient: shutdown 2024-09-11T01:46:53.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.967+0000 7f1e58687640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:53.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.967+0000 7f1e58687640 1 -- 172.21.15.12:0/1022207873 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e50136630 msgr2=0x7f1e5009c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:53.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.967+0000 7f1e58687640 1 --2- 172.21.15.12:0/1022207873 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e50136630 0x7f1e5009c8b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1e50073000 tx=0x7f1e4800ce10 comp rx=0 tx=0).stop 2024-09-11T01:46:53.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.967+0000 7f1e58687640 1 -- 172.21.15.12:0/1022207873 shutdown_connections 2024-09-11T01:46:53.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.967+0000 7f1e58687640 1 --2- 172.21.15.12:0/1022207873 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e50136630 0x7f1e5009c8b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:53.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.967+0000 7f1e58687640 1 --2- 172.21.15.12:0/1022207873 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1e3403f780 0x7f1e34041c40 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:53.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.967+0000 7f1e58687640 1 --2- 172.21.15.12:0/1022207873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e50070130 0x7f1e5012f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:53.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.967+0000 7f1e58687640 1 --2- 172.21.15.12:0/1022207873 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1e340094a0 0x7f1e5012ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:53.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.967+0000 7f1e58687640 1 -- 172.21.15.12:0/1022207873 >> 172.21.15.12:0/1022207873 conn(0x7f1e500a3de0 msgr2=0x7f1e500a51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:53.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.967+0000 7f1e58687640 1 -- 172.21.15.12:0/1022207873 shutdown_connections 2024-09-11T01:46:53.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:53.967+0000 7f1e58687640 1 -- 172.21.15.12:0/1022207873 wait complete. 2024-09-11T01:46:53.980 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:46:53.980 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T01:46:53.980 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:46:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.080+0000 7f0a58057640 10 monclient: get_monmap_and_config 2024-09-11T01:46:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.080+0000 7f0a58057640 10 monclient: build_initial_monmap 2024-09-11T01:46:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.080+0000 7f0a58057640 10 monclient: monmap: 2024-09-11T01:46:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:46:54.081541+0000 2024-09-11T01:46:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:46:54.081541+0000 2024-09-11T01:46:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:54.083 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:46:54.084 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:46:54.084 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:46:54.084 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:46:54.084 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:46:54.084 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:54.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.081+0000 7f0a58057640 1 Processor -- start 2024-09-11T01:46:54.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.081+0000 7f0a58057640 1 -- start start 2024-09-11T01:46:54.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.081+0000 7f0a58057640 10 monclient: init 2024-09-11T01:46:54.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a58057640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:54.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a58057640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-11T01:46:54.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a58057640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0a50135fc0 0x7f0a501363a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:54.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a58057640 10 monclient(hunting): picked mon.noname-a con 0x7f0a50135fc0 addr v2:172.21.15.12:3300/0 2024-09-11T01:46:54.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a58057640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0a50074000 addr v1:172.21.15.12:6789/0 2024-09-11T01:46:54.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a58057640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0a50074410 0x7f0a500a3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a58057640 10 monclient(hunting): picked mon.noname-c con 0x7f0a50074410 addr v2:172.21.15.71:3300/0 2024-09-11T01:46:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a58057640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a500732d0 con 0x7f0a50074000 2024-09-11T01:46:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a58057640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a58057640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f0a50074820 con 0x7f0a50135fc0 2024-09-11T01:46:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a58057640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a58057640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f0a50073890 con 0x7f0a50074410 2024-09-11T01:46:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a58057640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a58057640 10 monclient(hunting): authenticate will time out at 1820.524145s 2024-09-11T01:46:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a555f2640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7f0a50074000 legacy=0x7f0a500a2e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:46:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a55df3640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7f0a50135fc0 msgr2=0x7f0a501363a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:46:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a55df3640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0a50135fc0 0x7f0a501363a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:46:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a565f4640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0a50074410 0x7f0a500a3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:54.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a565f4640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0a50074410 0x7f0a500a3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:42424/0 (socket says 172.21.15.12:42424) 2024-09-11T01:46:54.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a565f4640 1 -- 172.21.15.12:0/1841668250 learned_addr learned my addr 172.21.15.12:0/1841668250 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:54.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a565f4640 10 monclient(hunting): get_auth_request con 0x7f0a50074410 auth_method 0 2024-09-11T01:46:54.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a565f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:54.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a565f4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:54.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a565f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:54.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a565f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:54.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a565f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:54.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a565f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:54.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a565f4640 10 monclient(hunting): handle_auth_done global_id 134674 payload 274 2024-09-11T01:46:54.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a565f4640 1 -- 172.21.15.12:0/1841668250 >> v2:172.21.15.12:3300/0 conn(0x7f0a50135fc0 msgr2=0x7f0a501363a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:46:54.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a565f4640 1 --2- 172.21.15.12:0/1841668250 >> v2:172.21.15.12:3300/0 conn(0x7f0a50135fc0 0x7f0a501363a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:54.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a565f4640 1 -- 172.21.15.12:0/1841668250 >> v1:172.21.15.12:6789/0 conn(0x7f0a50074000 legacy=0x7f0a500a2e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:46:54.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a565f4640 10 monclient: _finish_hunting 0 2024-09-11T01:46:54.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a565f4640 1 monclient: found mon.noname-c 2024-09-11T01:46:54.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a565f4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:54.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a565f4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:46:54.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.082+0000 7f0a565f4640 1 -- 172.21.15.12:0/1841668250 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a50072000 con 0x7f0a50074410 2024-09-11T01:46:54.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.085+0000 7f0a565f4640 1 -- 172.21.15.12:0/1841668250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a50074410 msgr2=0x7f0a500a3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:54.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.085+0000 7f0a565f4640 1 --2- 172.21.15.12:0/1841668250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a50074410 0x7f0a500a3530 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0a50075070 tx=0x7f0a4802a0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=391fb0a7ba8fc675 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:54.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a54df1640 1 -- 172.21.15.12:0/1841668250 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a48037070 con 0x7f0a50074410 2024-09-11T01:46:54.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a54df1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:54.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a54df1640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:46:54.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a54df1640 10 monclient: dump: 2024-09-11T01:46:54.097 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:54.097 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:54.097 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:54.097 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:54.097 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:54.097 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:54.097 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:54.097 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:54.097 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:54.097 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:54.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a54df1640 10 monclient: _finish_auth 0 2024-09-11T01:46:54.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a54df1640 10 monclient: _check_auth_tickets 2024-09-11T01:46:54.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a54df1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:54.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a54df1640 1 -- 172.21.15.12:0/1841668250 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a48032640 con 0x7f0a50074410 2024-09-11T01:46:54.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a58057640 5 monclient: authenticate success, global_id 134674 2024-09-11T01:46:54.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a58057640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:46:54.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a54df1640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:54.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a54df1640 1 -- 172.21.15.12:0/1841668250 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a48032a10 con 0x7f0a50074410 2024-09-11T01:46:54.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a54df1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:54.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a54df1640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:46:54.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a54df1640 10 monclient: dump: 2024-09-11T01:46:54.098 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:54.099 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:54.099 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:54.099 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:54.099 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:54.099 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:54.099 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:54.099 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:54.099 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:54.099 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:54.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a58057640 10 monclient: get_monmap_and_config success 2024-09-11T01:46:54.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a58057640 4 set_mon_vals no callback set 2024-09-11T01:46:54.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a58057640 10 monclient: shutdown 2024-09-11T01:46:54.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a58057640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:54.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a58057640 1 -- 172.21.15.12:0/1841668250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a50074410 msgr2=0x7f0a500a3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:54.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a58057640 1 --2- 172.21.15.12:0/1841668250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a50074410 0x7f0a500a3530 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0a50075070 tx=0x7f0a4802a0d0 comp rx=0 tx=0).stop 2024-09-11T01:46:54.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a58057640 1 -- 172.21.15.12:0/1841668250 shutdown_connections 2024-09-11T01:46:54.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a58057640 1 --2- 172.21.15.12:0/1841668250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a50074410 0x7f0a500a3530 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:54.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a58057640 1 --2- 172.21.15.12:0/1841668250 >> v2:172.21.15.12:3300/0 conn(0x7f0a50135fc0 0x7f0a501363a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:54.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a58057640 1 -- 172.21.15.12:0/1841668250 >> 172.21.15.12:0/1841668250 conn(0x7f0a500a8600 msgr2=0x7f0a500a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:54.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a58057640 1 -- 172.21.15.12:0/1841668250 shutdown_connections 2024-09-11T01:46:54.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.086+0000 7f0a58057640 1 -- 172.21.15.12:0/1841668250 wait complete. 2024-09-11T01:46:54.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.087+0000 7f0a58057640 10 monclient: build_initial_monmap 2024-09-11T01:46:54.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.087+0000 7f0a58057640 10 monclient: monmap: 2024-09-11T01:46:54.101 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:54.101 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:46:54.101 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:46:54.101 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:46:54.101 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:54.101 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:54.101 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:46:54.101 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:46:54.101 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:46:54.102 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:54.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.087+0000 7f0a58057640 1 Processor -- start 2024-09-11T01:46:54.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.087+0000 7f0a58057640 1 -- start start 2024-09-11T01:46:54.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.087+0000 7f0a58057640 10 monclient: init 2024-09-11T01:46:54.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a58057640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:54.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a58057640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:46:54.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a58057640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a50074000 0x7f0a501cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:54.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a58057640 10 monclient(hunting): picked mon.noname-b con 0x7f0a50074000 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:46:54.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a58057640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a50074410 0x7f0a50134cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:54.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a58057640 10 monclient(hunting): picked mon.noname-c con 0x7f0a50074410 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:54.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a58057640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a50135fc0 0x7f0a5013ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:54.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a58057640 10 monclient(hunting): picked mon.noname-a con 0x7f0a50135fc0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:46:54.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a58057640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:54.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a58057640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0a501368e0 con 0x7f0a50135fc0 2024-09-11T01:46:54.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a58057640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:54.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a58057640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0a500a3a70 con 0x7f0a50074000 2024-09-11T01:46:54.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a58057640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:54.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a58057640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f0a5009f610 con 0x7f0a50074410 2024-09-11T01:46:54.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a58057640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:54.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a58057640 10 monclient(hunting): authenticate will time out at 1820.530162s 2024-09-11T01:46:54.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a565f4640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a50135fc0 msgr2=0x7f0a5013ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:46:54.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a565f4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a50135fc0 0x7f0a5013ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:46:54.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a555f2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a50074410 0x7f0a50134cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:54.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a555f2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a50074410 0x7f0a50134cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:58872/0 (socket says 172.21.15.12:58872) 2024-09-11T01:46:54.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a555f2640 1 -- 172.21.15.12:0/2326869776 learned_addr learned my addr 172.21.15.12:0/2326869776 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:54.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a555f2640 10 monclient(hunting): get_auth_request con 0x7f0a50074410 auth_method 0 2024-09-11T01:46:54.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a555f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:54.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a555f2640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:54.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a555f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:54.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a555f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:54.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a555f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:54.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a555f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:54.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a55df3640 1 --2- 172.21.15.12:0/2326869776 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a50074000 0x7f0a501cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:54.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a55df3640 10 monclient(hunting): get_auth_request con 0x7f0a50074000 auth_method 0 2024-09-11T01:46:54.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a55df3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:54.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a55df3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:54.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a55df3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:54.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a555f2640 10 monclient(hunting): handle_auth_done global_id 184253 payload 824 2024-09-11T01:46:54.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a555f2640 1 -- 172.21.15.12:0/2326869776 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a50074000 msgr2=0x7f0a501cadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:54.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a555f2640 1 --2- 172.21.15.12:0/2326869776 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a50074000 0x7f0a501cadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:54.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a555f2640 1 -- 172.21.15.12:0/2326869776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a50135fc0 msgr2=0x7f0a5013ac10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:46:54.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a555f2640 1 --2- 172.21.15.12:0/2326869776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a50135fc0 0x7f0a5013ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:54.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a555f2640 10 monclient: _finish_hunting 0 2024-09-11T01:46:54.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a555f2640 1 monclient: found mon.noname-c 2024-09-11T01:46:54.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a555f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:54.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a555f2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:46:54.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.088+0000 7f0a555f2640 1 -- 172.21.15.12:0/2326869776 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a501cbd10 con 0x7f0a50074410 2024-09-11T01:46:54.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.090+0000 7f0a555f2640 1 -- 172.21.15.12:0/2326869776 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a50074410 msgr2=0x7f0a50134cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:54.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.090+0000 7f0a555f2640 1 --2- 172.21.15.12:0/2326869776 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a50074410 0x7f0a50134cd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0a4400b550 tx=0x7f0a4400ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:54.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.091+0000 7f0a4e7fc640 1 -- 172.21.15.12:0/2326869776 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a44005070 con 0x7f0a50074410 2024-09-11T01:46:54.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.091+0000 7f0a4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:54.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.091+0000 7f0a4e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:46:54.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.091+0000 7f0a4e7fc640 10 monclient: dump: 2024-09-11T01:46:54.106 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:54.106 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:54.106 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:54.106 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:54.106 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:54.106 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:54.106 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:54.106 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:54.106 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:54.107 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:54.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.091+0000 7f0a4e7fc640 10 monclient: _finish_auth 0 2024-09-11T01:46:54.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.091+0000 7f0a4e7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:46:54.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.091+0000 7f0a4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:54.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.091+0000 7f0a58057640 5 monclient: authenticate success, global_id 184253 2024-09-11T01:46:54.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.091+0000 7f0a58057640 10 monclient: _renew_subs 2024-09-11T01:46:54.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.091+0000 7f0a58057640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:54.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.091+0000 7f0a4e7fc640 1 -- 172.21.15.12:0/2326869776 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a44015070 con 0x7f0a50074410 2024-09-11T01:46:54.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.091+0000 7f0a58057640 1 -- 172.21.15.12:0/2326869776 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a500a7ea0 con 0x7f0a50074410 2024-09-11T01:46:54.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.091+0000 7f0a4e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:54.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.091+0000 7f0a4e7fc640 1 -- 172.21.15.12:0/2326869776 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a44005450 con 0x7f0a50074410 2024-09-11T01:46:54.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.091+0000 7f0a4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:54.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.091+0000 7f0a4e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:54.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.091+0000 7f0a4e7fc640 10 monclient: dump: 2024-09-11T01:46:54.108 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:54.108 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:54.108 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:54.108 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:54.108 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:54.108 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:54.108 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:54.108 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:54.108 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:54.108 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:54.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.091+0000 7f0a4ffff640 4 set_mon_vals no callback set 2024-09-11T01:46:54.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.091+0000 7f0a58057640 10 monclient: _renew_subs 2024-09-11T01:46:54.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.091+0000 7f0a58057640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:54.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.091+0000 7f0a58057640 1 -- 172.21.15.12:0/2326869776 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a5009f990 con 0x7f0a50074410 2024-09-11T01:46:54.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.091+0000 7f0a58057640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:54.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.091+0000 7f0a58057640 1 --2- 172.21.15.12:0/2326869776 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a50135210 0x7f0a5013b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:54.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.091+0000 7f0a58057640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:54.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.091+0000 7f0a58057640 1 -- 172.21.15.12:0/2326869776 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f0a5013c730 con 0x7f0a50135210 2024-09-11T01:46:54.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.092+0000 7f0a55df3640 1 --2- 172.21.15.12:0/2326869776 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a50135210 0x7f0a5013b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:54.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.092+0000 7f0a55df3640 10 monclient: get_auth_request con 0x7f0a50135210 auth_method 0 2024-09-11T01:46:54.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.092+0000 7f0a55df3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:54.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.092+0000 7f0a55df3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:54.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.092+0000 7f0a55df3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:54.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.092+0000 7f0a4e7fc640 1 -- 172.21.15.12:0/2326869776 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f0a44005630 con 0x7f0a50074410 2024-09-11T01:46:54.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.092+0000 7f0a55df3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:54.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.092+0000 7f0a55df3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:54.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.092+0000 7f0a55df3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:54.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.092+0000 7f0a55df3640 10 monclient(hunting): handle_auth_done global_id 184256 payload 824 2024-09-11T01:46:54.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.092+0000 7f0a55df3640 1 -- 172.21.15.12:0/2326869776 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0a5013d290 con 0x7f0a50135210 2024-09-11T01:46:54.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.092+0000 7f0a4e7fc640 1 --2- 172.21.15.12:0/2326869776 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0a3c03e1f0 0x7f0a3c0406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:54.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.092+0000 7f0a4e7fc640 1 -- 172.21.15.12:0/2326869776 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(597..597 src has 507..597) ==== 4819+0+0 (secure 0 0 0) 0x7f0a44052bf0 con 0x7f0a50074410 2024-09-11T01:46:54.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.092+0000 7f0a565f4640 1 --2- 172.21.15.12:0/2326869776 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0a3c03e1f0 0x7f0a3c0406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:54.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.092+0000 7f0a565f4640 10 monclient: get_auth_request con 0x7f0a3c03e1f0 auth_method 0 2024-09-11T01:46:54.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.094+0000 7f0a55df3640 1 -- 172.21.15.12:0/2326869776 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a50135210 msgr2=0x7f0a5013b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:54.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.094+0000 7f0a55df3640 1 --2- 172.21.15.12:0/2326869776 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a50135210 0x7f0a5013b150 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0a40009fd0 tx=0x7f0a40005420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:54.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.094+0000 7f0a4e7fc640 1 -- 172.21.15.12:0/2326869776 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a40019070 con 0x7f0a50135210 2024-09-11T01:46:54.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.094+0000 7f0a4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:54.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.094+0000 7f0a4e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:54.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.094+0000 7f0a4e7fc640 10 monclient: dump: 2024-09-11T01:46:54.111 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:54.111 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:54.111 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:54.111 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:54.111 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:54.111 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:54.111 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:54.111 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:54.111 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:54.111 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:54.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.095+0000 7f0a4e7fc640 1 -- 172.21.15.12:0/2326869776 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0a40012730 con 0x7f0a50135210 2024-09-11T01:46:54.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.095+0000 7f0a4e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:54.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.095+0000 7f0a4e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:46:54.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.095+0000 7f0a4e7fc640 1 -- 172.21.15.12:0/2326869776 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a50135210 msgr2=0x7f0a5013b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:54.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.095+0000 7f0a4e7fc640 1 --2- 172.21.15.12:0/2326869776 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a50135210 0x7f0a5013b150 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0a40009fd0 tx=0x7f0a40005420 comp rx=0 tx=0).stop 2024-09-11T01:46:54.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.095+0000 7f0a4e7fc640 1 -- 172.21.15.12:0/2326869776 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a50135210 msgr2=0x7f0a5013b150 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:46:54.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.095+0000 7f0a4e7fc640 1 --2- 172.21.15.12:0/2326869776 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a50135210 0x7f0a5013b150 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:54.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.095+0000 7f0a565f4640 1 --2- 172.21.15.12:0/2326869776 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0a3c03e1f0 0x7f0a3c0406b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0a50074cd0 tx=0x7f0a4802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:54.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.101+0000 7f0a58057640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:46:54.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.101+0000 7f0a58057640 1 --2- 172.21.15.12:0/2326869776 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a5013b690 0x7f0a50136a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:54.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.101+0000 7f0a58057640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:54.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.101+0000 7f0a58057640 1 -- 172.21.15.12:0/2326869776 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f0a5013baa0 con 0x7f0a5013b690 2024-09-11T01:46:54.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.101+0000 7f0a55df3640 1 --2- 172.21.15.12:0/2326869776 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a5013b690 0x7f0a50136a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:54.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.101+0000 7f0a55df3640 10 monclient: get_auth_request con 0x7f0a5013b690 auth_method 0 2024-09-11T01:46:54.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.101+0000 7f0a55df3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:54.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.101+0000 7f0a55df3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:54.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.101+0000 7f0a55df3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:54.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.101+0000 7f0a55df3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:54.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.101+0000 7f0a55df3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:54.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.101+0000 7f0a55df3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:54.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.101+0000 7f0a55df3640 10 monclient(hunting): handle_auth_done global_id 184259 payload 824 2024-09-11T01:46:54.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.101+0000 7f0a55df3640 1 -- 172.21.15.12:0/2326869776 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0a501363a0 con 0x7f0a5013b690 2024-09-11T01:46:54.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.103+0000 7f0a55df3640 1 -- 172.21.15.12:0/2326869776 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a5013b690 msgr2=0x7f0a50136a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:54.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.103+0000 7f0a55df3640 1 --2- 172.21.15.12:0/2326869776 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a5013b690 0x7f0a50136a60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0a40009fd0 tx=0x7f0a40005cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:54.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.103+0000 7f0a4e7fc640 1 -- 172.21.15.12:0/2326869776 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a40019030 con 0x7f0a5013b690 2024-09-11T01:46:54.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.103+0000 7f0a4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:54.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.103+0000 7f0a4e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:54.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.103+0000 7f0a4e7fc640 10 monclient: dump: 2024-09-11T01:46:54.115 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:54.115 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:54.115 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:54.115 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:54.115 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:54.116 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:54.116 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:54.116 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:54.116 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:54.116 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:54.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.104+0000 7f0a4e7fc640 1 -- 172.21.15.12:0/2326869776 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4697 (secure 0 0 0) 0x7f0a40024410 con 0x7f0a5013b690 2024-09-11T01:46:54.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.104+0000 7f0a4e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:46:54.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.104+0000 7f0a4e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:46:54.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.104+0000 7f0a4e7fc640 1 -- 172.21.15.12:0/2326869776 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a5013b690 msgr2=0x7f0a50136a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:54.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.104+0000 7f0a4e7fc640 1 --2- 172.21.15.12:0/2326869776 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a5013b690 0x7f0a50136a60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0a40009fd0 tx=0x7f0a40005cf0 comp rx=0 tx=0).stop 2024-09-11T01:46:54.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.104+0000 7f0a4e7fc640 1 -- 172.21.15.12:0/2326869776 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a5013b690 msgr2=0x7f0a50136a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:46:54.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.104+0000 7f0a4e7fc640 1 --2- 172.21.15.12:0/2326869776 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a5013b690 0x7f0a50136a60 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0a40009fd0 tx=0x7f0a40005cf0 comp rx=0 tx=0).stop 2024-09-11T01:46:54.117 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:46:54.117 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:46:54.117 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:46:54.117 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:46:54.117 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 164, 2024-09-11T01:46:54.117 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 74697, 2024-09-11T01:46:54.117 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:46:54.117 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:46:54.118 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:46:54.118 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:54.118 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:46:54.118 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:46:54.118 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:46:54.118 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:46:54.118 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:54.118 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:54.118 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:54.118 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:54.118 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:54.118 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:54.118 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:54.118 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:54.119 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:54.119 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:54.119 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:54.119 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:54.119 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:46:54.119 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:46:54.119 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:54.119 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:54.119 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:54.119 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:54.119 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:54.119 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:54.119 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:54.119 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:54.120 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:54.120 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:54.120 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:54.120 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:54.120 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:54.120 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:46:54.120 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:46:54.120 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:46:54.120 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:46:54.120 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:46:54.120 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:46:54.120 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:46:54.120 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:46:54.120 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:46:54.121 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:46:54.121 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:46:54.121 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:46:54.121 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:46:54.121 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:46:54.121 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:46:54.121 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:46:54.121 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:46:54.121 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:54.121 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:54.121 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:54.121 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:54.121 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:54.121 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:54.122 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:54.122 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:54.122 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:54.122 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:54.122 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:54.122 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:54.122 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:46:54.122 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:54.122 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:46:54.122 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:54.122 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:46:54.122 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:46:54.122 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:54.122 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:54.122 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:54.123 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:54.123 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:46:54.123 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:54.123 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:54.123 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:54.123 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:54.123 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:46:54.123 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:54.123 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:54.123 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:54.123 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:54.123 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:46:54.123 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:46:54.124 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:54.124 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:54.124 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:54.124 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:54.124 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:54.124 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:46:54.124 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:46:54.124 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:54.124 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:54.124 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:54.124 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:54.124 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:46:54.124 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:54.124 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:54.124 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:54.125 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:54.125 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:46:54.125 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:54.125 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:54.125 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:54.125 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:54.125 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:46:54.125 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:46:54.125 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:54.125 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:54.125 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:54.125 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:54.125 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:54.126 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:46:54.126 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:46:54.126 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:54.126 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:54.126 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:54.126 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:54.126 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:46:54.126 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:54.126 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:54.126 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:54.126 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:54.126 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:46:54.126 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:54.127 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:54.127 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:54.127 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:54.127 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:46:54.127 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:46:54.127 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:54.127 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:54.127 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:54.127 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:54.127 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:54.127 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:54.127 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:46:54.127 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:46:54.127 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:54.127 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:54.128 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:54.128 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:46:54.128 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:54.128 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:54.128 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:46:54.128 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:54.128 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:54.128 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:54.128 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:46:54.128 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:54.128 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:54.128 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:46:54.128 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:54.129 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:54.129 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:54.129 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:46:54.129 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:54.129 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:54.129 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:54.129 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:46:54.129 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:46:54.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.104+0000 7f0a58057640 1 -- 172.21.15.12:0/2326869776 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0a3c03e1f0 msgr2=0x7f0a3c0406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:54.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.104+0000 7f0a58057640 1 --2- 172.21.15.12:0/2326869776 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0a3c03e1f0 0x7f0a3c0406b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0a50074cd0 tx=0x7f0a4802e000 comp rx=0 tx=0).stop 2024-09-11T01:46:54.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.104+0000 7f0a58057640 10 monclient: shutdown 2024-09-11T01:46:54.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.104+0000 7f0a58057640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:54.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.104+0000 7f0a58057640 1 -- 172.21.15.12:0/2326869776 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a50074410 msgr2=0x7f0a50134cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:54.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.104+0000 7f0a58057640 1 --2- 172.21.15.12:0/2326869776 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a50074410 0x7f0a50134cd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0a4400b550 tx=0x7f0a4400ba10 comp rx=0 tx=0).stop 2024-09-11T01:46:54.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.104+0000 7f0a565f4640 1 -- 172.21.15.12:0/2326869776 reap_dead start 2024-09-11T01:46:54.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.104+0000 7f0a58057640 1 -- 172.21.15.12:0/2326869776 shutdown_connections 2024-09-11T01:46:54.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.104+0000 7f0a58057640 1 --2- 172.21.15.12:0/2326869776 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a50074410 0x7f0a50134cd0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:54.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.104+0000 7f0a58057640 1 -- 172.21.15.12:0/2326869776 >> 172.21.15.12:0/2326869776 conn(0x7f0a500a8600 msgr2=0x7f0a5009e330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:54.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.104+0000 7f0a58057640 1 -- 172.21.15.12:0/2326869776 shutdown_connections 2024-09-11T01:46:54.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.104+0000 7f0a58057640 1 -- 172.21.15.12:0/2326869776 wait complete. 2024-09-11T01:46:54.131 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:46:54.131 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T01:46:54.132 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:46:54.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.231+0000 7f564e52b640 10 monclient: get_monmap_and_config 2024-09-11T01:46:54.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.231+0000 7f564e52b640 10 monclient: build_initial_monmap 2024-09-11T01:46:54.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.231+0000 7f564e52b640 10 monclient: monmap: 2024-09-11T01:46:54.234 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:54.234 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:54.234 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:46:54.232539+0000 2024-09-11T01:46:54.234 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:46:54.232539+0000 2024-09-11T01:46:54.234 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:54.234 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:54.234 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:46:54.234 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:46:54.234 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:46:54.234 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:46:54.234 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:46:54.234 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:54.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.232+0000 7f564e52b640 1 Processor -- start 2024-09-11T01:46:54.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.232+0000 7f564e52b640 1 -- start start 2024-09-11T01:46:54.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.232+0000 7f564e52b640 10 monclient: init 2024-09-11T01:46:54.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f564e52b640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:54.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f564e52b640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-11T01:46:54.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f564e52b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56480a1380 0x7f56480a1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:54.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f564e52b640 10 monclient(hunting): picked mon.noname-b con 0x7f56480a1380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:54.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f564e52b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f56480a0370 0x7f5648135ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:54.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f564e52b640 10 monclient(hunting): picked mon.noname-c con 0x7f56480a0370 addr v2:172.21.15.71:3300/0 2024-09-11T01:46:54.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f564e52b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f56480a0c90 addr v1:172.21.15.71:6789/0 2024-09-11T01:46:54.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f564e52b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56480a1ca0 con 0x7f56480a0c90 2024-09-11T01:46:54.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f564e52b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:54.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f564e52b640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f56480a21c0 con 0x7f56480a0370 2024-09-11T01:46:54.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f564e52b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:54.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f564e52b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f56480a24a0 con 0x7f56480a1380 2024-09-11T01:46:54.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f564e52b640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:54.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f564e52b640 10 monclient(hunting): authenticate will time out at 1820.675117s 2024-09-11T01:46:54.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f5647fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56480a1380 0x7f56480a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:54.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f5647fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56480a1380 0x7f56480a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:58918/0 (socket says 172.21.15.12:58918) 2024-09-11T01:46:54.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f5647fff640 1 -- 172.21.15.12:0/1021041656 learned_addr learned my addr 172.21.15.12:0/1021041656 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:54.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f5647fff640 10 monclient(hunting): get_auth_request con 0x7f56480a1380 auth_method 0 2024-09-11T01:46:54.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f5647fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:54.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f5647fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:54.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f5647fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:54.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f5647fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:54.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f5647fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:54.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f5647fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:54.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f56477fe640 1 --2- 172.21.15.12:0/1021041656 >> v2:172.21.15.71:3300/0 conn(0x7f56480a0370 0x7f5648135ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:54.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f56477fe640 10 monclient(hunting): get_auth_request con 0x7f56480a0370 auth_method 0 2024-09-11T01:46:54.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f56477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:54.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f56477fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:54.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f56477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:54.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f5647fff640 10 monclient(hunting): handle_auth_done global_id 184262 payload 274 2024-09-11T01:46:54.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f5647fff640 1 -- 172.21.15.12:0/1021041656 >> v2:172.21.15.71:3300/0 conn(0x7f56480a0370 msgr2=0x7f5648135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:46:54.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f5647fff640 1 --2- 172.21.15.12:0/1021041656 >> v2:172.21.15.71:3300/0 conn(0x7f56480a0370 0x7f5648135ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:54.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f5647fff640 1 -- 172.21.15.12:0/1021041656 >> v1:172.21.15.71:6789/0 conn(0x7f56480a0c90 legacy=0x7f5648136020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:46:54.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f5647fff640 10 monclient: _finish_hunting 0 2024-09-11T01:46:54.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f5647fff640 1 monclient: found mon.noname-b 2024-09-11T01:46:54.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f5647fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:54.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f5647fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:46:54.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.233+0000 7f5647fff640 1 -- 172.21.15.12:0/1021041656 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56480a2780 con 0x7f56480a1380 2024-09-11T01:46:54.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.236+0000 7f5647fff640 1 -- 172.21.15.12:0/1021041656 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56480a1380 msgr2=0x7f56480a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:54.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.236+0000 7f5647fff640 1 --2- 172.21.15.12:0/1021041656 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56480a1380 0x7f56480a1760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5638009870 tx=0x7f563802a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3976d5d06bf8d00 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:54.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.236+0000 7f5646ffd640 1 -- 172.21.15.12:0/1021041656 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5638037070 con 0x7f56480a1380 2024-09-11T01:46:54.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.236+0000 7f5646ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:54.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.236+0000 7f5646ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:46:54.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.236+0000 7f5646ffd640 10 monclient: dump: 2024-09-11T01:46:54.246 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:54.246 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:54.246 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:54.246 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:54.247 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:54.247 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:54.247 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:54.247 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:54.247 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:54.247 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:54.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.236+0000 7f5646ffd640 10 monclient: _finish_auth 0 2024-09-11T01:46:54.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.236+0000 7f5646ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:46:54.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.236+0000 7f5646ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:54.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.236+0000 7f564e52b640 5 monclient: authenticate success, global_id 184262 2024-09-11T01:46:54.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.236+0000 7f564e52b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:46:54.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.236+0000 7f5646ffd640 1 -- 172.21.15.12:0/1021041656 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5638032440 con 0x7f56480a1380 2024-09-11T01:46:54.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.236+0000 7f5646ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:54.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.236+0000 7f564e52b640 10 monclient: get_monmap_and_config success 2024-09-11T01:46:54.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.236+0000 7f564e52b640 4 set_mon_vals no callback set 2024-09-11T01:46:54.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.236+0000 7f5646ffd640 1 -- 172.21.15.12:0/1021041656 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5638032810 con 0x7f56480a1380 2024-09-11T01:46:54.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.236+0000 7f5646ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:54.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.236+0000 7f5646ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:54.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.236+0000 7f5646ffd640 10 monclient: dump: 2024-09-11T01:46:54.248 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:54.248 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:54.248 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:54.248 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:54.248 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:54.248 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:54.248 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:54.248 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:54.249 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:54.249 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:54.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.237+0000 7f564e52b640 10 monclient: shutdown 2024-09-11T01:46:54.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.237+0000 7f564e52b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:54.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.237+0000 7f564e52b640 1 -- 172.21.15.12:0/1021041656 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56480a1380 msgr2=0x7f56480a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:54.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.237+0000 7f564e52b640 1 --2- 172.21.15.12:0/1021041656 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56480a1380 0x7f56480a1760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5638009870 tx=0x7f563802a0a0 comp rx=0 tx=0).stop 2024-09-11T01:46:54.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.237+0000 7f564e52b640 1 -- 172.21.15.12:0/1021041656 shutdown_connections 2024-09-11T01:46:54.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.237+0000 7f564e52b640 1 --2- 172.21.15.12:0/1021041656 >> v2:172.21.15.71:3300/0 conn(0x7f56480a0370 0x7f5648135ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:54.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.237+0000 7f564e52b640 1 --2- 172.21.15.12:0/1021041656 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56480a1380 0x7f56480a1760 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:54.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.237+0000 7f564e52b640 1 -- 172.21.15.12:0/1021041656 >> 172.21.15.12:0/1021041656 conn(0x7f56480a2f70 msgr2=0x7f56480a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:54.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.237+0000 7f564e52b640 1 -- 172.21.15.12:0/1021041656 shutdown_connections 2024-09-11T01:46:54.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.237+0000 7f564e52b640 1 -- 172.21.15.12:0/1021041656 wait complete. 2024-09-11T01:46:54.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.237+0000 7f564e52b640 10 monclient: build_initial_monmap 2024-09-11T01:46:54.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.237+0000 7f564e52b640 10 monclient: monmap: 2024-09-11T01:46:54.250 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:46:54.250 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:46:54.250 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:46:54.250 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:46:54.250 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:46:54.250 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:54.250 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:46:54.250 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:46:54.250 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:46:54.250 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:54.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f564e52b640 1 Processor -- start 2024-09-11T01:46:54.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f564e52b640 1 -- start start 2024-09-11T01:46:54.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f564e52b640 10 monclient: init 2024-09-11T01:46:54.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f564e52b640 10 monclient: _reopen_session rank -1 2024-09-11T01:46:54.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f564e52b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:46:54.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f564e52b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56480a0370 0x7f5648135c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:54.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f564e52b640 10 monclient(hunting): picked mon.noname-a con 0x7f56480a0370 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:46:54.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f564e52b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56480a0c90 0x7f56481361c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:54.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f564e52b640 10 monclient(hunting): picked mon.noname-c con 0x7f56480a0c90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:46:54.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f564e52b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56480a1380 0x7f56481ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:54.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f564e52b640 10 monclient(hunting): picked mon.noname-b con 0x7f56480a1380 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:46:54.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f564e52b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:54.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f5647fff640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56480a0370 msgr2=0x7f5648135c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:46:54.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f564e52b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f564809ce20 con 0x7f56480a0370 2024-09-11T01:46:54.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f564e52b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:54.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f564e52b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f564809cca0 con 0x7f56480a1380 2024-09-11T01:46:54.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f564e52b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:54.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f564e52b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5648136730 con 0x7f56480a0c90 2024-09-11T01:46:54.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f564e52b640 10 monclient(hunting): _renew_subs 2024-09-11T01:46:54.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f5647fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56480a0370 0x7f5648135c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:46:54.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f564e52b640 10 monclient(hunting): authenticate will time out at 1820.680878s 2024-09-11T01:46:54.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f56477fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56480a0c90 0x7f56481361c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:54.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f56477fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56480a0c90 0x7f56481361c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:58920/0 (socket says 172.21.15.12:58920) 2024-09-11T01:46:54.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f56477fe640 1 -- 172.21.15.12:0/3556195088 learned_addr learned my addr 172.21.15.12:0/3556195088 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:46:54.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f56477fe640 10 monclient(hunting): get_auth_request con 0x7f56480a0c90 auth_method 0 2024-09-11T01:46:54.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f56477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:54.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f56477fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:54.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.238+0000 7f56477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:54.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.239+0000 7f56477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:54.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.239+0000 7f56477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:54.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.239+0000 7f56477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:54.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.239+0000 7f56477fe640 10 monclient(hunting): handle_auth_done global_id 184265 payload 824 2024-09-11T01:46:54.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.239+0000 7f56477fe640 1 -- 172.21.15.12:0/3556195088 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56480a1380 msgr2=0x7f56481ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:54.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.239+0000 7f56477fe640 1 --2- 172.21.15.12:0/3556195088 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56480a1380 0x7f56481ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:54.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.239+0000 7f56477fe640 1 -- 172.21.15.12:0/3556195088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56480a0370 msgr2=0x7f5648135c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:46:54.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.239+0000 7f56477fe640 1 --2- 172.21.15.12:0/3556195088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56480a0370 0x7f5648135c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:46:54.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.239+0000 7f56477fe640 10 monclient: _finish_hunting 0 2024-09-11T01:46:54.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.239+0000 7f56477fe640 1 monclient: found mon.noname-c 2024-09-11T01:46:54.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.239+0000 7f56477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:46:54.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.239+0000 7f56477fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:46:54.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.239+0000 7f56477fe640 1 -- 172.21.15.12:0/3556195088 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f564809f1e0 con 0x7f56480a0c90 2024-09-11T01:46:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.241+0000 7f56477fe640 1 -- 172.21.15.12:0/3556195088 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56480a0c90 msgr2=0x7f56481361c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.241+0000 7f56477fe640 1 --2- 172.21.15.12:0/3556195088 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56480a0c90 0x7f56481361c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f563c00d800 tx=0x7f563c00dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.241+0000 7f5644ff9640 1 -- 172.21.15.12:0/3556195088 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f563c00b8d0 con 0x7f56480a0c90 2024-09-11T01:46:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.241+0000 7f5644ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.241+0000 7f5644ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:46:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.241+0000 7f5644ff9640 10 monclient: dump: 2024-09-11T01:46:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:54.255 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:54.255 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:54.255 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:54.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.241+0000 7f5644ff9640 10 monclient: _finish_auth 0 2024-09-11T01:46:54.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.241+0000 7f5644ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:46:54.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.241+0000 7f5644ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:46:54.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.241+0000 7f5644ff9640 1 -- 172.21.15.12:0/3556195088 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f563c019070 con 0x7f56480a0c90 2024-09-11T01:46:54.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.241+0000 7f564e52b640 5 monclient: authenticate success, global_id 184265 2024-09-11T01:46:54.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.241+0000 7f564e52b640 10 monclient: _renew_subs 2024-09-11T01:46:54.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.241+0000 7f564e52b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:54.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.241+0000 7f564e52b640 1 -- 172.21.15.12:0/3556195088 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56481ca340 con 0x7f56480a0c90 2024-09-11T01:46:54.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.241+0000 7f5644ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:46:54.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.241+0000 7f5644ff9640 1 -- 172.21.15.12:0/3556195088 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f563c00bc90 con 0x7f56480a0c90 2024-09-11T01:46:54.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.241+0000 7f5644ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.241+0000 7f5644ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:46:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.241+0000 7f5644ff9640 10 monclient: dump: 2024-09-11T01:46:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:54.256 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.241+0000 7f564e52b640 10 monclient: _renew_subs 2024-09-11T01:46:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.241+0000 7f56467fc640 4 set_mon_vals no callback set 2024-09-11T01:46:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.241+0000 7f564e52b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:46:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.241+0000 7f564e52b640 1 -- 172.21.15.12:0/3556195088 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f56481293c0 con 0x7f56480a0c90 2024-09-11T01:46:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.242+0000 7f564e52b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.242+0000 7f5644ff9640 1 -- 172.21.15.12:0/3556195088 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f563c013070 con 0x7f56480a0c90 2024-09-11T01:46:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.242+0000 7f564e52b640 1 --2- 172.21.15.12:0/3556195088 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f564802d440 0x7f564802d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.242+0000 7f564e52b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.242+0000 7f564e52b640 1 -- 172.21.15.12:0/3556195088 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f564809ed20 con 0x7f564802d440 2024-09-11T01:46:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.242+0000 7f5644ff9640 1 --2- 172.21.15.12:0/3556195088 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f563403e210 0x7f56340406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.242+0000 7f5644ff9640 1 -- 172.21.15.12:0/3556195088 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(597..597 src has 507..597) ==== 4819+0+0 (secure 0 0 0) 0x7f563c052ee0 con 0x7f56480a0c90 2024-09-11T01:46:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.242+0000 7f5647fff640 1 --2- 172.21.15.12:0/3556195088 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f564802d440 0x7f564802d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.242+0000 7f5647fff640 10 monclient: get_auth_request con 0x7f564802d440 auth_method 0 2024-09-11T01:46:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.242+0000 7f5647fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.242+0000 7f5647fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:54.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.242+0000 7f5647fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:54.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.242+0000 7f5647fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:54.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.242+0000 7f5647fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:54.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.242+0000 7f5647fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:54.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.242+0000 7f564cac8640 1 --2- 172.21.15.12:0/3556195088 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f563403e210 0x7f56340406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:54.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.242+0000 7f564cac8640 10 monclient: get_auth_request con 0x7f563403e210 auth_method 0 2024-09-11T01:46:54.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.243+0000 7f5647fff640 10 monclient(hunting): handle_auth_done global_id 134683 payload 824 2024-09-11T01:46:54.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.243+0000 7f5647fff640 1 -- 172.21.15.12:0/3556195088 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5648128e60 con 0x7f564802d440 2024-09-11T01:46:54.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.245+0000 7f5647fff640 1 -- 172.21.15.12:0/3556195088 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f564802d440 msgr2=0x7f564802d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:54.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.245+0000 7f5647fff640 1 --2- 172.21.15.12:0/3556195088 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f564802d440 0x7f564802d820 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5638034000 tx=0x7f56380046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:54.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.245+0000 7f564cac8640 1 --2- 172.21.15.12:0/3556195088 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f563403e210 0x7f56340406d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f56480a1120 tx=0x7f5640007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:54.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.245+0000 7f5644ff9640 1 -- 172.21.15.12:0/3556195088 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5638037040 con 0x7f564802d440 2024-09-11T01:46:54.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.245+0000 7f5644ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:54.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.245+0000 7f5644ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:46:54.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.245+0000 7f5644ff9640 10 monclient: dump: 2024-09-11T01:46:54.261 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:54.261 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:54.261 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:54.261 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:54.261 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:54.261 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:54.261 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:54.262 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:54.262 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:54.262 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:54.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.245+0000 7f5644ff9640 1 -- 172.21.15.12:0/3556195088 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f563803d470 con 0x7f564802d440 2024-09-11T01:46:54.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.245+0000 7f5644ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:46:54.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.245+0000 7f5644ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:46:54.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.245+0000 7f5644ff9640 1 -- 172.21.15.12:0/3556195088 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f564802d440 msgr2=0x7f564802d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:54.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.245+0000 7f5644ff9640 1 --2- 172.21.15.12:0/3556195088 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f564802d440 0x7f564802d820 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5638034000 tx=0x7f56380046c0 comp rx=0 tx=0).stop 2024-09-11T01:46:54.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.245+0000 7f5644ff9640 1 -- 172.21.15.12:0/3556195088 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f564802d440 msgr2=0x7f564802d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:46:54.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.245+0000 7f5644ff9640 1 --2- 172.21.15.12:0/3556195088 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f564802d440 0x7f564802d820 secure :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5638034000 tx=0x7f56380046c0 comp rx=0 tx=0).stop 2024-09-11T01:46:54.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.251+0000 7f564e52b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:46:54.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.251+0000 7f564e52b640 1 --2- 172.21.15.12:0/3556195088 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f564802dd60 0x7f564802e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:46:54.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.251+0000 7f564e52b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:46:54.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.251+0000 7f564e52b640 1 -- 172.21.15.12:0/3556195088 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f56480a1760 con 0x7f564802dd60 2024-09-11T01:46:54.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.252+0000 7f5647fff640 1 --2- 172.21.15.12:0/3556195088 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f564802dd60 0x7f564802e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:46:54.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.252+0000 7f5647fff640 10 monclient: get_auth_request con 0x7f564802dd60 auth_method 0 2024-09-11T01:46:54.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.252+0000 7f5647fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:46:54.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.252+0000 7f5647fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:46:54.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.252+0000 7f5647fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:46:54.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.252+0000 7f5647fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:46:54.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.252+0000 7f5647fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:46:54.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.252+0000 7f5647fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:46:54.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.252+0000 7f5647fff640 10 monclient(hunting): handle_auth_done global_id 134686 payload 824 2024-09-11T01:46:54.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.252+0000 7f5647fff640 1 -- 172.21.15.12:0/3556195088 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f56481280a0 con 0x7f564802dd60 2024-09-11T01:46:54.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.254+0000 7f5647fff640 1 -- 172.21.15.12:0/3556195088 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f564802dd60 msgr2=0x7f564802e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:46:54.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.254+0000 7f5647fff640 1 --2- 172.21.15.12:0/3556195088 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f564802dd60 0x7f564802e170 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5638000c00 tx=0x7f5638009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:46:54.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.255+0000 7f5644ff9640 1 -- 172.21.15.12:0/3556195088 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5638037040 con 0x7f564802dd60 2024-09-11T01:46:54.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.255+0000 7f5644ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:46:54.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.255+0000 7f5644ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:46:54.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.255+0000 7f5644ff9640 10 monclient: dump: 2024-09-11T01:46:54.264 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:46:54.264 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:46:54.264 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:54.264 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:46:54.264 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:46:54.264 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:46:54.264 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:46:54.264 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:46:54.264 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:46:54.264 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:46:54.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.255+0000 7f5644ff9640 1 -- 172.21.15.12:0/3556195088 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4867 (secure 0 0 0) 0x7f5638048020 con 0x7f564802dd60 2024-09-11T01:46:54.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.255+0000 7f5644ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:46:54.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.255+0000 7f5644ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:46:54.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.255+0000 7f5644ff9640 1 -- 172.21.15.12:0/3556195088 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f564802dd60 msgr2=0x7f564802e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:54.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.255+0000 7f5644ff9640 1 --2- 172.21.15.12:0/3556195088 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f564802dd60 0x7f564802e170 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5638000c00 tx=0x7f5638009240 comp rx=0 tx=0).stop 2024-09-11T01:46:54.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.255+0000 7f5644ff9640 1 -- 172.21.15.12:0/3556195088 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f564802dd60 msgr2=0x7f564802e170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:46:54.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.255+0000 7f5644ff9640 1 --2- 172.21.15.12:0/3556195088 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f564802dd60 0x7f564802e170 secure :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5638000c00 tx=0x7f5638009240 comp rx=0 tx=0).stop 2024-09-11T01:46:54.265 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:46:54.265 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:46:54.266 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:46:54.266 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:46:54.266 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 164, 2024-09-11T01:46:54.266 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 347217, 2024-09-11T01:46:54.266 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:46:54.266 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:46:54.266 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:46:54.266 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:54.266 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:46:54.266 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:46:54.266 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:46:54.266 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:46:54.266 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:54.266 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:54.267 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:54.267 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:54.267 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:54.267 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:54.267 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:54.267 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:54.267 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:54.267 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:54.267 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:54.267 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:54.267 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:46:54.267 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:46:54.267 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:54.268 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:54.268 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:54.268 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:54.268 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:54.268 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:54.268 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:54.268 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:54.268 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:54.268 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:54.268 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:54.268 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:54.268 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:54.268 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:46:54.268 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:46:54.269 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:46:54.269 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:46:54.269 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:46:54.269 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:46:54.269 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:46:54.269 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:46:54.269 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:46:54.269 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:46:54.269 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:46:54.269 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:46:54.269 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:46:54.269 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:46:54.269 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:46:54.270 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:46:54.270 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:46:54.270 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:46:54.270 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:46:54.270 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:46:54.270 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:46:54.270 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:46:54.270 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:46:54.270 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:46:54.270 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:46:54.270 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:46:54.270 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:46:54.270 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:46:54.270 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:54.271 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:46:54.271 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:54.271 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:46:54.271 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:54.271 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:46:54.271 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:46:54.271 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:54.271 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:54.271 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:54.271 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:54.271 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:46:54.271 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:54.271 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:54.271 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:54.272 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:54.272 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:46:54.272 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:54.272 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:54.272 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:54.272 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:54.272 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:46:54.272 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:46:54.272 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:54.272 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:54.272 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:54.272 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:54.272 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:54.273 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:46:54.273 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:46:54.273 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:54.273 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:54.273 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:54.273 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:54.273 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:46:54.273 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:54.273 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:54.273 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:54.273 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:54.273 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:46:54.273 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:54.273 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:54.274 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:54.274 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:54.274 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:46:54.274 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:46:54.274 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:54.274 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:54.274 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:54.274 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:54.274 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:54.274 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:46:54.274 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:46:54.274 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:46:54.274 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:46:54.274 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:54.275 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:46:54.275 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:46:54.275 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:54.275 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:54.275 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:54.275 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:46:54.275 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:46:54.275 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:46:54.275 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:54.275 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:54.275 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:54.275 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:46:54.275 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:46:54.275 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:46:54.276 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:46:54.276 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:46:54.276 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:54.276 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:54.276 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:54.276 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:46:54.276 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:46:54.276 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:54.276 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:54.276 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:54.276 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:46:54.276 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:54.276 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:54.277 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:46:54.277 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:54.277 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:54.277 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:54.277 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:46:54.277 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:54.277 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:54.277 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:46:54.277 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:54.277 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:54.277 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:54.277 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:46:54.277 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:54.278 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:46:54.278 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:46:54.278 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:46:54.278 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:46:54.278 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:46:54.278 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:46:54.278 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:46:54.278 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:46:54.278 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:46:54.278 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:46:54.278 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:46:54.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.255+0000 7f564e52b640 1 -- 172.21.15.12:0/3556195088 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f563403e210 msgr2=0x7f56340406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:54.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.255+0000 7f564e52b640 1 --2- 172.21.15.12:0/3556195088 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f563403e210 0x7f56340406d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f56480a1120 tx=0x7f5640007d30 comp rx=0 tx=0).stop 2024-09-11T01:46:54.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.255+0000 7f564e52b640 10 monclient: shutdown 2024-09-11T01:46:54.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.255+0000 7f564e52b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:46:54.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.255+0000 7f564e52b640 1 -- 172.21.15.12:0/3556195088 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56480a0c90 msgr2=0x7f56481361c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:46:54.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.255+0000 7f564e52b640 1 --2- 172.21.15.12:0/3556195088 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56480a0c90 0x7f56481361c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f563c00d800 tx=0x7f563c00dcc0 comp rx=0 tx=0).stop 2024-09-11T01:46:54.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.255+0000 7f564cac8640 1 -- 172.21.15.12:0/3556195088 reap_dead start 2024-09-11T01:46:54.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.255+0000 7f564cac8640 1 -- 172.21.15.12:0/3556195088 reap_dead start 2024-09-11T01:46:54.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.255+0000 7f564e52b640 1 -- 172.21.15.12:0/3556195088 shutdown_connections 2024-09-11T01:46:54.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.255+0000 7f564e52b640 1 -- 172.21.15.12:0/3556195088 >> 172.21.15.12:0/3556195088 conn(0x7f56480a2f70 msgr2=0x7f5648071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:46:54.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.255+0000 7f564e52b640 1 -- 172.21.15.12:0/3556195088 shutdown_connections 2024-09-11T01:46:54.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:46:54.255+0000 7f564e52b640 1 -- 172.21.15.12:0/3556195088 wait complete. 2024-09-11T01:46:54.281 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:46:54.281 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:46:54.687 INFO:tasks.rados.rados.0.smithi071.stdout:2659: done (2 left) 2024-09-11T01:46:54.687 INFO:tasks.rados.rados.0.smithi071.stdout:2660: rollback oid 47 current snap is 298 2024-09-11T01:46:54.687 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 47 to 259 2024-09-11T01:46:54.687 INFO:tasks.rados.rados.0.smithi071.stdout:2658: finishing rollback tid 0 to smithi07134777-10 2024-09-11T01:46:54.687 INFO:tasks.rados.rados.0.smithi071.stdout:2658: finishing rollback tid 1 to smithi07134777-10 2024-09-11T01:46:54.687 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 469 (ObjNum 129 snap 0 seq_num 2371479872) dirty dne 2024-09-11T01:46:54.688 INFO:tasks.rados.rados.0.smithi071.stdout:2657: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:46:54.785 INFO:tasks.rados.rados.0.smithi071.stdout:2660: finishing rollback tid 0 to smithi07134777-47 2024-09-11T01:46:54.785 INFO:tasks.rados.rados.0.smithi071.stdout:2660: finishing rollback tid 1 to smithi07134777-47 2024-09-11T01:46:54.785 INFO:tasks.rados.rados.0.smithi071.stdout:2660: finishing rollback tid 2 to smithi07134777-47 2024-09-11T01:46:54.785 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 1623 (ObjNum 704 snap 213 seq_num 704) dirty exists 2024-09-11T01:46:54.785 INFO:tasks.rados.rados.0.smithi071.stdout:2657: done (2 left) 2024-09-11T01:46:54.785 INFO:tasks.rados.rados.0.smithi071.stdout:2658: done (1 left) 2024-09-11T01:46:54.785 INFO:tasks.rados.rados.0.smithi071.stdout:2660: done (0 left) 2024-09-11T01:46:54.785 INFO:tasks.rados.rados.0.smithi071.stdout:2661: read oid 10 snap -1 2024-09-11T01:46:54.785 INFO:tasks.rados.rados.0.smithi071.stdout:2661: expect deleted 2024-09-11T01:46:54.785 INFO:tasks.rados.rados.0.smithi071.stdout:2662: rollback oid 46 current snap is 298 2024-09-11T01:46:54.786 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 46 to 289 2024-09-11T01:46:54.786 INFO:tasks.rados.rados.0.smithi071.stdout:2663: rollback oid 25 current snap is 298 2024-09-11T01:46:54.786 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 25 to 250 2024-09-11T01:46:54.786 INFO:tasks.rados.rados.0.smithi071.stdout:2664: write oid 31 current snap is 298 2024-09-11T01:46:54.786 INFO:tasks.rados.rados.0.smithi071.stdout:2664: seq_num 947 ranges {716229=488575,1616330=541328,2756280=559655} 2024-09-11T01:46:54.791 INFO:tasks.rados.rados.0.smithi071.stdout:2664: writing smithi07134777-31 from 716229 to 1204804 tid 1 2024-09-11T01:46:54.796 INFO:tasks.rados.rados.0.smithi071.stdout:2664: writing smithi07134777-31 from 1616330 to 2157658 tid 2 2024-09-11T01:46:54.804 INFO:tasks.rados.rados.0.smithi071.stdout:2664: writing smithi07134777-31 from 2756280 to 3315935 tid 3 2024-09-11T01:46:54.804 INFO:tasks.rados.rados.0.smithi071.stdout:2665: rollback oid 42 current snap is 298 2024-09-11T01:46:54.804 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 42 to 272 2024-09-11T01:46:54.804 INFO:tasks.rados.rados.0.smithi071.stdout:2662: finishing rollback tid 0 to smithi07134777-46 2024-09-11T01:46:54.804 INFO:tasks.rados.rados.0.smithi071.stdout:2662: finishing rollback tid 1 to smithi07134777-46 2024-09-11T01:46:54.804 INFO:tasks.rados.rados.0.smithi071.stdout:2662: finishing rollback tid 2 to smithi07134777-46 2024-09-11T01:46:54.804 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 1547 (ObjNum 841 snap 255 seq_num 841) dirty exists 2024-09-11T01:46:54.804 INFO:tasks.rados.rados.0.smithi071.stdout:2663: finishing rollback tid 0 to smithi07134777-25 2024-09-11T01:46:54.804 INFO:tasks.rados.rados.0.smithi071.stdout:2663: finishing rollback tid 1 to smithi07134777-25 2024-09-11T01:46:54.804 INFO:tasks.rados.rados.0.smithi071.stdout:2663: finishing rollback tid 2 to smithi07134777-25 2024-09-11T01:46:54.804 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 1010 (ObjNum 723 snap 224 seq_num 723) dirty exists 2024-09-11T01:46:54.804 INFO:tasks.rados.rados.0.smithi071.stdout:2664: finishing write tid 1 to smithi07134777-31 2024-09-11T01:46:54.805 INFO:tasks.rados.rados.0.smithi071.stdout:2664: finishing write tid 2 to smithi07134777-31 2024-09-11T01:46:54.805 INFO:tasks.rados.rados.0.smithi071.stdout:2661: done (4 left) 2024-09-11T01:46:54.805 INFO:tasks.rados.rados.0.smithi071.stdout:2662: done (3 left) 2024-09-11T01:46:54.805 INFO:tasks.rados.rados.0.smithi071.stdout:2663: done (2 left) 2024-09-11T01:46:54.805 INFO:tasks.rados.rados.0.smithi071.stdout:2666: read oid 30 snap -1 2024-09-11T01:46:54.805 INFO:tasks.rados.rados.0.smithi071.stdout:2666: expect deleted 2024-09-11T01:46:54.805 INFO:tasks.rados.rados.0.smithi071.stdout:2667: delete oid 5 current snap is 298 2024-09-11T01:46:54.806 INFO:tasks.rados.rados.0.smithi071.stdout:2665: finishing rollback tid 1 to smithi07134777-42 2024-09-11T01:46:54.806 INFO:tasks.rados.rados.0.smithi071.stdout:2665: finishing rollback tid 2 to smithi07134777-42 2024-09-11T01:46:54.806 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 857 (ObjNum 806 snap 244 seq_num 806) dirty exists 2024-09-11T01:46:54.807 INFO:tasks.rados.rados.0.smithi071.stdout:2665: done (3 left) 2024-09-11T01:46:54.807 INFO:tasks.rados.rados.0.smithi071.stdout:2666: done (2 left) 2024-09-11T01:46:54.807 INFO:tasks.rados.rados.0.smithi071.stdout:2667: done (1 left) 2024-09-11T01:46:54.807 INFO:tasks.rados.rados.0.smithi071.stdout:2668: snap_create 2024-09-11T01:46:54.807 INFO:tasks.rados.rados.0.smithi071.stdout:2664: finishing write tid 3 to smithi07134777-31 2024-09-11T01:46:54.808 INFO:tasks.rados.rados.0.smithi071.stdout:2664: finishing write tid 4 to smithi07134777-31 2024-09-11T01:46:54.808 INFO:tasks.rados.rados.0.smithi071.stdout:2664: finishing write tid 5 to smithi07134777-31 2024-09-11T01:46:54.808 INFO:tasks.rados.rados.0.smithi071.stdout:2664: oid 31 updating version 0 to 1625 2024-09-11T01:46:54.808 INFO:tasks.rados.rados.0.smithi071.stdout:2664: oid 31 updating version 1625 to 1627 2024-09-11T01:46:54.808 INFO:tasks.rados.rados.0.smithi071.stdout:2664: oid 31 version 1627 is already newer than 1624 2024-09-11T01:46:54.808 INFO:tasks.rados.rados.0.smithi071.stdout:2664: oid 31 version 1627 is already newer than 1626 2024-09-11T01:46:54.808 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 1627 (ObjNum 947 snap 298 seq_num 947) dirty exists 2024-09-11T01:46:54.808 INFO:tasks.rados.rados.0.smithi071.stdout:2664: left oid 31 (ObjNum 947 snap 298 seq_num 947) 2024-09-11T01:46:55.696 INFO:tasks.rados.rados.0.smithi071.stdout:2664: done (1 left) 2024-09-11T01:46:55.696 INFO:tasks.rados.rados.0.smithi071.stdout:2668: done (0 left) 2024-09-11T01:46:55.696 INFO:tasks.rados.rados.0.smithi071.stdout:2669: rollback oid 19 current snap is 299 2024-09-11T01:46:55.696 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 19 to 174 2024-09-11T01:46:55.696 INFO:tasks.rados.rados.0.smithi071.stdout:2670: read oid 40 snap -1 2024-09-11T01:46:55.696 INFO:tasks.rados.rados.0.smithi071.stdout:2670: expect deleted 2024-09-11T01:46:55.696 INFO:tasks.rados.rados.0.smithi071.stdout:2671: snap_create 2024-09-11T01:46:55.701 INFO:tasks.rados.rados.0.smithi071.stdout:2669: finishing rollback tid 0 to smithi07134777-19 2024-09-11T01:46:55.701 INFO:tasks.rados.rados.0.smithi071.stdout:2669: finishing rollback tid 1 to smithi07134777-19 2024-09-11T01:46:55.701 INFO:tasks.rados.rados.0.smithi071.stdout:2669: finishing rollback tid 2 to smithi07134777-19 2024-09-11T01:46:55.701 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 1073 (ObjNum 598 snap 170 seq_num 598) dirty exists 2024-09-11T01:46:56.698 INFO:tasks.rados.rados.0.smithi071.stdout:2669: done (2 left) 2024-09-11T01:46:56.699 INFO:tasks.rados.rados.0.smithi071.stdout:2670: done (1 left) 2024-09-11T01:46:56.699 INFO:tasks.rados.rados.0.smithi071.stdout:2671: done (0 left) 2024-09-11T01:46:56.699 INFO:tasks.rados.rados.0.smithi071.stdout:2672: read oid 27 snap -1 2024-09-11T01:46:56.699 INFO:tasks.rados.rados.0.smithi071.stdout:2672: expect (ObjNum 875 snap 267 seq_num 875) 2024-09-11T01:46:56.699 INFO:tasks.rados.rados.0.smithi071.stdout:2673: read oid 15 snap -1 2024-09-11T01:46:56.699 INFO:tasks.rados.rados.0.smithi071.stdout:2673: expect deleted 2024-09-11T01:46:56.699 INFO:tasks.rados.rados.0.smithi071.stdout:2674: copy_from oid 28 from oid 47 current snap is 300 2024-09-11T01:46:56.699 INFO:tasks.rados.rados.0.smithi071.stdout:2675: read oid 31 snap -1 2024-09-11T01:46:56.699 INFO:tasks.rados.rados.0.smithi071.stdout:2675: expect (ObjNum 947 snap 298 seq_num 947) 2024-09-11T01:46:56.699 INFO:tasks.rados.rados.0.smithi071.stdout:2676: copy_from oid 34 from oid 14 current snap is 300 2024-09-11T01:46:56.700 INFO:tasks.rados.rados.0.smithi071.stdout:2677: read oid 26 snap -1 2024-09-11T01:46:56.700 INFO:tasks.rados.rados.0.smithi071.stdout:2677: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:46:56.700 INFO:tasks.rados.rados.0.smithi071.stdout:2678: write oid 49 current snap is 300 2024-09-11T01:46:56.700 INFO:tasks.rados.rados.0.smithi071.stdout:2678: seq_num 949 ranges {749843=639073,2133294=544997,3211126=582199} 2024-09-11T01:46:56.707 INFO:tasks.rados.rados.0.smithi071.stdout:2678: writing smithi07134777-49 from 749843 to 1388916 tid 1 2024-09-11T01:46:56.714 INFO:tasks.rados.rados.0.smithi071.stdout:2678: writing smithi07134777-49 from 2133294 to 2678291 tid 2 2024-09-11T01:46:56.721 INFO:tasks.rados.rados.0.smithi071.stdout:2678: writing smithi07134777-49 from 3211126 to 3793325 tid 3 2024-09-11T01:46:56.721 INFO:tasks.rados.rados.0.smithi071.stdout:2679: write oid 5 current snap is 300 2024-09-11T01:46:56.721 INFO:tasks.rados.rados.0.smithi071.stdout:2679: seq_num 950 ranges {766650=714322,2075423=688655,3522196=444454} 2024-09-11T01:46:56.728 INFO:tasks.rados.rados.0.smithi071.stdout:2679: writing smithi07134777-5 from 766650 to 1480972 tid 1 2024-09-11T01:46:56.736 INFO:tasks.rados.rados.0.smithi071.stdout:2679: writing smithi07134777-5 from 2075423 to 2764078 tid 2 2024-09-11T01:46:56.743 INFO:tasks.rados.rados.0.smithi071.stdout:2679: writing smithi07134777-5 from 3522196 to 3966650 tid 3 2024-09-11T01:46:56.743 INFO:tasks.rados.rados.0.smithi071.stdout:2680: read oid 17 snap 277 2024-09-11T01:46:56.743 INFO:tasks.rados.rados.0.smithi071.stdout:2680: expect deleted 2024-09-11T01:46:56.743 INFO:tasks.rados.rados.0.smithi071.stdout:2681: delete oid 11 current snap is 300 2024-09-11T01:46:56.743 INFO:tasks.rados.rados.0.smithi071.stdout:2672: expect (ObjNum 875 snap 267 seq_num 875) 2024-09-11T01:46:56.760 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:46:56.817 INFO:tasks.rados.rados.0.smithi071.stdout:2675: expect (ObjNum 947 snap 298 seq_num 947) 2024-09-11T01:46:56.817 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:46:57.033 INFO:tasks.rados.rados.0.smithi071.stdout:2677: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:46:57.118 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2584041 2024-09-11T01:46:57.134 INFO:tasks.rados.rados.0.smithi071.stdout:2678: finishing write tid 1 to smithi07134777-49 2024-09-11T01:46:57.135 INFO:tasks.rados.rados.0.smithi071.stdout:2678: finishing write tid 2 to smithi07134777-49 2024-09-11T01:46:57.135 INFO:tasks.rados.rados.0.smithi071.stdout:2678: finishing write tid 3 to smithi07134777-49 2024-09-11T01:46:57.135 INFO:tasks.rados.rados.0.smithi071.stdout:2678: finishing write tid 4 to smithi07134777-49 2024-09-11T01:46:57.135 INFO:tasks.rados.rados.0.smithi071.stdout:2678: finishing write tid 5 to smithi07134777-49 2024-09-11T01:46:57.135 INFO:tasks.rados.rados.0.smithi071.stdout:2678: oid 49 updating version 0 to 382 2024-09-11T01:46:57.135 INFO:tasks.rados.rados.0.smithi071.stdout:2678: oid 49 updating version 382 to 384 2024-09-11T01:46:57.135 INFO:tasks.rados.rados.0.smithi071.stdout:2678: oid 49 version 384 is already newer than 383 2024-09-11T01:46:57.135 INFO:tasks.rados.rados.0.smithi071.stdout:2678: oid 49 version 384 is already newer than 381 2024-09-11T01:46:57.135 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 384 (ObjNum 949 snap 300 seq_num 949) dirty exists 2024-09-11T01:46:57.135 INFO:tasks.rados.rados.0.smithi071.stdout:2678: left oid 49 (ObjNum 949 snap 300 seq_num 949) 2024-09-11T01:46:57.135 INFO:tasks.rados.rados.0.smithi071.stdout:2676: finishing copy_from racing read to smithi07134777-34 2024-09-11T01:46:57.135 INFO:tasks.rados.rados.0.smithi071.stdout:2676: finishing copy_from to smithi07134777-34 2024-09-11T01:46:57.136 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 1751 (ObjNum 942 snap 296 seq_num 942) dirty exists 2024-09-11T01:46:57.136 INFO:tasks.rados.rados.0.smithi071.stdout:2674: finishing copy_from racing read to smithi07134777-28 2024-09-11T01:46:57.136 INFO:tasks.rados.rados.0.smithi071.stdout:2679: finishing write tid 1 to smithi07134777-5 2024-09-11T01:46:57.136 INFO:tasks.rados.rados.0.smithi071.stdout:2674: finishing copy_from to smithi07134777-28 2024-09-11T01:46:57.136 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 1623 (ObjNum 704 snap 213 seq_num 704) dirty exists 2024-09-11T01:46:57.136 INFO:tasks.rados.rados.0.smithi071.stdout:2679: finishing write tid 2 to smithi07134777-5 2024-09-11T01:46:57.136 INFO:tasks.rados.rados.0.smithi071.stdout:2672: done (9 left) 2024-09-11T01:46:57.136 INFO:tasks.rados.rados.0.smithi071.stdout:2673: done (8 left) 2024-09-11T01:46:57.136 INFO:tasks.rados.rados.0.smithi071.stdout:2674: done (7 left) 2024-09-11T01:46:57.136 INFO:tasks.rados.rados.0.smithi071.stdout:2675: done (6 left) 2024-09-11T01:46:57.136 INFO:tasks.rados.rados.0.smithi071.stdout:2676: done (5 left) 2024-09-11T01:46:57.136 INFO:tasks.rados.rados.0.smithi071.stdout:2677: done (4 left) 2024-09-11T01:46:57.136 INFO:tasks.rados.rados.0.smithi071.stdout:2678: done (3 left) 2024-09-11T01:46:57.137 INFO:tasks.rados.rados.0.smithi071.stdout:2681: done (2 left) 2024-09-11T01:46:57.137 INFO:tasks.rados.rados.0.smithi071.stdout:2682: snap_create 2024-09-11T01:46:57.137 INFO:tasks.rados.rados.0.smithi071.stdout:2679: finishing write tid 3 to smithi07134777-5 2024-09-11T01:46:57.137 INFO:tasks.rados.rados.0.smithi071.stdout:2679: finishing write tid 4 to smithi07134777-5 2024-09-11T01:46:57.137 INFO:tasks.rados.rados.0.smithi071.stdout:2679: finishing write tid 5 to smithi07134777-5 2024-09-11T01:46:57.137 INFO:tasks.rados.rados.0.smithi071.stdout:2679: oid 5 updating version 0 to 1165 2024-09-11T01:46:57.137 INFO:tasks.rados.rados.0.smithi071.stdout:2679: oid 5 updating version 1165 to 1166 2024-09-11T01:46:57.137 INFO:tasks.rados.rados.0.smithi071.stdout:2679: oid 5 version 1166 is already newer than 1164 2024-09-11T01:46:57.137 INFO:tasks.rados.rados.0.smithi071.stdout:2679: oid 5 updating version 1166 to 1167 2024-09-11T01:46:57.137 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 1167 (ObjNum 950 snap 300 seq_num 950) dirty exists 2024-09-11T01:46:57.137 INFO:tasks.rados.rados.0.smithi071.stdout:2679: left oid 5 (ObjNum 950 snap 300 seq_num 950) 2024-09-11T01:46:57.580 INFO:tasks.rados.rados.0.smithi071.stdout:2679: done (2 left) 2024-09-11T01:46:57.580 INFO:tasks.rados.rados.0.smithi071.stdout:2680: done (1 left) 2024-09-11T01:46:57.580 INFO:tasks.rados.rados.0.smithi071.stdout:2682: done (0 left) 2024-09-11T01:46:57.580 INFO:tasks.rados.rados.0.smithi071.stdout:2683: write (excl) oid 2 current snap is 301 2024-09-11T01:46:57.580 INFO:tasks.rados.rados.0.smithi071.stdout:2683: seq_num 952 ranges {400264=581173,1508740=491524} 2024-09-11T01:46:57.585 INFO:tasks.rados.rados.0.smithi071.stdout:2683: writing smithi07134777-2 from 400264 to 981437 tid 1 2024-09-11T01:46:57.591 INFO:tasks.rados.rados.0.smithi071.stdout:2683: writing smithi07134777-2 from 1508740 to 2000264 tid 2 2024-09-11T01:46:57.591 INFO:tasks.rados.rados.0.smithi071.stdout:2684: rollback oid 5 current snap is 301 2024-09-11T01:46:57.591 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 5 to 272 2024-09-11T01:46:57.591 INFO:tasks.rados.rados.0.smithi071.stdout:2685: copy_from oid 12 from oid 11 current snap is 301 2024-09-11T01:46:57.591 INFO:tasks.rados.rados.0.smithi071.stdout:2686: write oid 15 current snap is 301 2024-09-11T01:46:57.591 INFO:tasks.rados.rados.0.smithi071.stdout:2686: seq_num 953 ranges {417071=656422,1567222=449849} 2024-09-11T01:46:57.597 INFO:tasks.rados.rados.0.smithi071.stdout:2686: writing smithi07134777-15 from 417071 to 1073493 tid 1 2024-09-11T01:46:57.603 INFO:tasks.rados.rados.0.smithi071.stdout:2686: writing smithi07134777-15 from 1567222 to 2017071 tid 2 2024-09-11T01:46:57.603 INFO:tasks.rados.rados.0.smithi071.stdout:2687: snap_remove snap 298 2024-09-11T01:46:58.586 INFO:tasks.rados.rados.0.smithi071.stdout:2687: done (4 left) 2024-09-11T01:46:58.586 INFO:tasks.rados.rados.0.smithi071.stdout:2688: copy_from oid 31 from oid 14 current snap is 301 2024-09-11T01:46:58.586 INFO:tasks.rados.rados.0.smithi071.stdout:2683: finishing write tid 1 to smithi07134777-2 2024-09-11T01:46:58.586 INFO:tasks.rados.rados.0.smithi071.stdout:2685: finishing copy_from to smithi07134777-12 2024-09-11T01:46:58.586 INFO:tasks.rados.rados.0.smithi071.stdout:2685: got expected ENOENT (src dne) 2024-09-11T01:46:58.586 INFO:tasks.rados.rados.0.smithi071.stdout:2685: finishing copy_from racing read to smithi07134777-12 2024-09-11T01:46:58.586 INFO:tasks.rados.rados.0.smithi071.stdout:2684: finishing rollback tid 0 to smithi07134777-5 2024-09-11T01:46:58.586 INFO:tasks.rados.rados.0.smithi071.stdout:2683: finishing write tid 2 to smithi07134777-2 2024-09-11T01:46:58.587 INFO:tasks.rados.rados.0.smithi071.stdout:2683: finishing write tid 3 to smithi07134777-2 2024-09-11T01:46:58.587 INFO:tasks.rados.rados.0.smithi071.stdout:2686: finishing write tid 1 to smithi07134777-15 2024-09-11T01:46:58.587 INFO:tasks.rados.rados.0.smithi071.stdout:2683: finishing write tid 4 to smithi07134777-2 2024-09-11T01:46:58.587 INFO:tasks.rados.rados.0.smithi071.stdout:2683: oid 2 updating version 0 to 860 2024-09-11T01:46:58.587 INFO:tasks.rados.rados.0.smithi071.stdout:2683: oid 2 version 860 is already newer than 858 2024-09-11T01:46:58.587 INFO:tasks.rados.rados.0.smithi071.stdout:2683: oid 2 version 860 is already newer than 859 2024-09-11T01:46:58.587 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 860 (ObjNum 952 snap 301 seq_num 952) dirty exists 2024-09-11T01:46:58.587 INFO:tasks.rados.rados.0.smithi071.stdout:2683: left oid 2 (ObjNum 952 snap 301 seq_num 952) 2024-09-11T01:46:58.587 INFO:tasks.rados.rados.0.smithi071.stdout:2686: finishing write tid 2 to smithi07134777-15 2024-09-11T01:46:58.587 INFO:tasks.rados.rados.0.smithi071.stdout:2686: finishing write tid 3 to smithi07134777-15 2024-09-11T01:46:58.587 INFO:tasks.rados.rados.0.smithi071.stdout:2686: finishing write tid 4 to smithi07134777-15 2024-09-11T01:46:58.587 INFO:tasks.rados.rados.0.smithi071.stdout:2686: oid 15 updating version 0 to 1803 2024-09-11T01:46:58.587 INFO:tasks.rados.rados.0.smithi071.stdout:2686: oid 15 updating version 1803 to 1804 2024-09-11T01:46:58.587 INFO:tasks.rados.rados.0.smithi071.stdout:2686: oid 15 version 1804 is already newer than 1802 2024-09-11T01:46:58.588 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 1804 (ObjNum 953 snap 301 seq_num 953) dirty exists 2024-09-11T01:46:58.588 INFO:tasks.rados.rados.0.smithi071.stdout:2686: left oid 15 (ObjNum 953 snap 301 seq_num 953) 2024-09-11T01:46:58.588 INFO:tasks.rados.rados.0.smithi071.stdout:2684: finishing rollback tid 1 to smithi07134777-5 2024-09-11T01:46:58.588 INFO:tasks.rados.rados.0.smithi071.stdout:2684: finishing rollback tid 2 to smithi07134777-5 2024-09-11T01:46:58.588 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 1170 (ObjNum 886 snap 272 seq_num 886) dirty exists 2024-09-11T01:46:58.588 INFO:tasks.rados.rados.0.smithi071.stdout:2683: done (4 left) 2024-09-11T01:46:58.588 INFO:tasks.rados.rados.0.smithi071.stdout:2684: done (3 left) 2024-09-11T01:46:58.588 INFO:tasks.rados.rados.0.smithi071.stdout:2685: done (2 left) 2024-09-11T01:46:58.588 INFO:tasks.rados.rados.0.smithi071.stdout:2686: done (1 left) 2024-09-11T01:46:58.588 INFO:tasks.rados.rados.0.smithi071.stdout:2689: copy_from oid 38 from oid 19 current snap is 301 2024-09-11T01:46:58.588 INFO:tasks.rados.rados.0.smithi071.stdout:2690: rollback oid 13 current snap is 301 2024-09-11T01:46:58.588 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 13 to 276 2024-09-11T01:46:58.588 INFO:tasks.rados.rados.0.smithi071.stdout:2691: snap_create 2024-09-11T01:46:58.605 INFO:tasks.rados.rados.0.smithi071.stdout:2688: finishing copy_from racing read to smithi07134777-31 2024-09-11T01:46:58.612 INFO:tasks.rados.rados.0.smithi071.stdout:2690: finishing rollback tid 0 to smithi07134777-13 2024-09-11T01:46:58.612 INFO:tasks.rados.rados.0.smithi071.stdout:2690: finishing rollback tid 1 to smithi07134777-13 2024-09-11T01:46:58.612 INFO:tasks.rados.rados.0.smithi071.stdout:2690: finishing rollback tid 2 to smithi07134777-13 2024-09-11T01:46:58.612 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 1630 (ObjNum 808 snap 246 seq_num 808) dirty exists 2024-09-11T01:46:58.613 INFO:tasks.rados.rados.0.smithi071.stdout:2688: finishing copy_from to smithi07134777-31 2024-09-11T01:46:58.613 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 1751 (ObjNum 942 snap 296 seq_num 942) dirty exists 2024-09-11T01:46:58.615 INFO:tasks.rados.rados.0.smithi071.stdout:2689: finishing copy_from racing read to smithi07134777-38 2024-09-11T01:46:58.615 INFO:tasks.rados.rados.0.smithi071.stdout:2689: finishing copy_from to smithi07134777-38 2024-09-11T01:46:58.615 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 1073 (ObjNum 598 snap 170 seq_num 598) dirty exists 2024-09-11T01:46:59.594 INFO:tasks.rados.rados.0.smithi071.stdout:2688: done (3 left) 2024-09-11T01:46:59.594 INFO:tasks.rados.rados.0.smithi071.stdout:2689: done (2 left) 2024-09-11T01:46:59.594 INFO:tasks.rados.rados.0.smithi071.stdout:2690: done (1 left) 2024-09-11T01:46:59.594 INFO:tasks.rados.rados.0.smithi071.stdout:2691: done (0 left) 2024-09-11T01:46:59.594 INFO:tasks.rados.rados.0.smithi071.stdout:2692: rollback oid 10 current snap is 302 2024-09-11T01:46:59.595 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 10 to 174 2024-09-11T01:46:59.595 INFO:tasks.rados.rados.0.smithi071.stdout:2693: read oid 48 snap -1 2024-09-11T01:46:59.595 INFO:tasks.rados.rados.0.smithi071.stdout:2693: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:46:59.595 INFO:tasks.rados.rados.0.smithi071.stdout:2694: write (excl) oid 32 current snap is 302 2024-09-11T01:46:59.595 INFO:tasks.rados.rados.0.smithi071.stdout:2694: seq_num 954 ranges {433878=731671,1909351=124527} 2024-09-11T01:46:59.601 INFO:tasks.rados.rados.0.smithi071.stdout:2694: writing smithi07134777-32 from 433878 to 1165549 tid 1 2024-09-11T01:46:59.606 INFO:tasks.rados.rados.0.smithi071.stdout:2694: writing smithi07134777-32 from 1909351 to 2033878 tid 2 2024-09-11T01:46:59.606 INFO:tasks.rados.rados.0.smithi071.stdout:2695: snap_create 2024-09-11T01:46:59.606 INFO:tasks.rados.rados.0.smithi071.stdout:2692: finishing rollback tid 1 to smithi07134777-10 2024-09-11T01:46:59.606 INFO:tasks.rados.rados.0.smithi071.stdout:2692: finishing rollback tid 2 to smithi07134777-10 2024-09-11T01:46:59.606 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 475 (ObjNum 601 snap 172 seq_num 601) dirty exists 2024-09-11T01:46:59.606 INFO:tasks.rados.rados.0.smithi071.stdout:2693: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:46:59.610 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 796608 2024-09-11T01:46:59.639 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:46:59.677 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:46:59.716 INFO:tasks.rados.rados.0.smithi071.stdout:2694: finishing write tid 1 to smithi07134777-32 2024-09-11T01:46:59.716 INFO:tasks.rados.rados.0.smithi071.stdout:2694: finishing write tid 2 to smithi07134777-32 2024-09-11T01:46:59.717 INFO:tasks.rados.rados.0.smithi071.stdout:2694: finishing write tid 3 to smithi07134777-32 2024-09-11T01:46:59.717 INFO:tasks.rados.rados.0.smithi071.stdout:2694: finishing write tid 4 to smithi07134777-32 2024-09-11T01:46:59.717 INFO:tasks.rados.rados.0.smithi071.stdout:2694: oid 32 updating version 0 to 1032 2024-09-11T01:46:59.717 INFO:tasks.rados.rados.0.smithi071.stdout:2694: oid 32 updating version 1032 to 1033 2024-09-11T01:46:59.717 INFO:tasks.rados.rados.0.smithi071.stdout:2694: oid 32 version 1033 is already newer than 1031 2024-09-11T01:46:59.717 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 1033 (ObjNum 954 snap 302 seq_num 954) dirty exists 2024-09-11T01:46:59.717 INFO:tasks.rados.rados.0.smithi071.stdout:2694: left oid 32 (ObjNum 954 snap 302 seq_num 954) 2024-09-11T01:47:00.601 INFO:tasks.rados.rados.0.smithi071.stdout:2692: done (3 left) 2024-09-11T01:47:00.601 INFO:tasks.rados.rados.0.smithi071.stdout:2693: done (2 left) 2024-09-11T01:47:00.602 INFO:tasks.rados.rados.0.smithi071.stdout:2694: done (1 left) 2024-09-11T01:47:00.602 INFO:tasks.rados.rados.0.smithi071.stdout:2695: done (0 left) 2024-09-11T01:47:00.602 INFO:tasks.rados.rados.0.smithi071.stdout:2696: delete oid 23 current snap is 303 2024-09-11T01:47:00.607 INFO:tasks.rados.rados.0.smithi071.stdout:2696: done (0 left) 2024-09-11T01:47:00.607 INFO:tasks.rados.rados.0.smithi071.stdout:2697: snap_create 2024-09-11T01:47:01.606 INFO:tasks.rados.rados.0.smithi071.stdout:2697: done (0 left) 2024-09-11T01:47:01.607 INFO:tasks.rados.rados.0.smithi071.stdout:2698: delete oid 12 current snap is 304 2024-09-11T01:47:01.613 INFO:tasks.rados.rados.0.smithi071.stdout:2698: done (0 left) 2024-09-11T01:47:01.613 INFO:tasks.rados.rados.0.smithi071.stdout:2699: write oid 3 current snap is 304 2024-09-11T01:47:01.613 INFO:tasks.rados.rados.0.smithi071.stdout:2699: seq_num 957 ranges {484299=557418,1684797=399502} 2024-09-11T01:47:01.618 INFO:tasks.rados.rados.0.smithi071.stdout:2699: writing smithi07134777-3 from 484299 to 1041717 tid 1 2024-09-11T01:47:01.624 INFO:tasks.rados.rados.0.smithi071.stdout:2699: writing smithi07134777-3 from 1684797 to 2084299 tid 2 2024-09-11T01:47:01.624 INFO:tasks.rados.rados.0.smithi071.stdout:2700: snap_remove snap 279 2024-09-11T01:47:02.584 INFO:tasks.rados.rados.0.smithi071.stdout:2700: done (1 left) 2024-09-11T01:47:02.584 INFO:tasks.rados.rados.0.smithi071.stdout:2701: rollback oid 23 current snap is 304 2024-09-11T01:47:02.584 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 23 to 225 2024-09-11T01:47:02.585 INFO:tasks.rados.rados.0.smithi071.stdout:2699: finishing write tid 1 to smithi07134777-3 2024-09-11T01:47:02.585 INFO:tasks.rados.rados.0.smithi071.stdout:2699: finishing write tid 2 to smithi07134777-3 2024-09-11T01:47:02.585 INFO:tasks.rados.rados.0.smithi071.stdout:2699: finishing write tid 3 to smithi07134777-3 2024-09-11T01:47:02.585 INFO:tasks.rados.rados.0.smithi071.stdout:2699: finishing write tid 4 to smithi07134777-3 2024-09-11T01:47:02.585 INFO:tasks.rados.rados.0.smithi071.stdout:2699: oid 3 updating version 0 to 1647 2024-09-11T01:47:02.585 INFO:tasks.rados.rados.0.smithi071.stdout:2699: oid 3 updating version 1647 to 1648 2024-09-11T01:47:02.585 INFO:tasks.rados.rados.0.smithi071.stdout:2699: oid 3 updating version 1648 to 1649 2024-09-11T01:47:02.585 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 1649 (ObjNum 957 snap 304 seq_num 957) dirty exists 2024-09-11T01:47:02.585 INFO:tasks.rados.rados.0.smithi071.stdout:2699: left oid 3 (ObjNum 957 snap 304 seq_num 957) 2024-09-11T01:47:02.585 INFO:tasks.rados.rados.0.smithi071.stdout:2699: done (1 left) 2024-09-11T01:47:02.585 INFO:tasks.rados.rados.0.smithi071.stdout:2702: write oid 20 current snap is 304 2024-09-11T01:47:02.585 INFO:tasks.rados.rados.0.smithi071.stdout:2702: seq_num 958 ranges {501106=749020,1743279=357827} 2024-09-11T01:47:02.592 INFO:tasks.rados.rados.0.smithi071.stdout:2702: writing smithi07134777-20 from 501106 to 1250126 tid 1 2024-09-11T01:47:02.597 INFO:tasks.rados.rados.0.smithi071.stdout:2702: writing smithi07134777-20 from 1743279 to 2101106 tid 2 2024-09-11T01:47:02.597 INFO:tasks.rados.rados.0.smithi071.stdout:2703: snap_remove snap 297 2024-09-11T01:47:03.585 INFO:tasks.rados.rados.0.smithi071.stdout:2703: done (2 left) 2024-09-11T01:47:03.585 INFO:tasks.rados.rados.0.smithi071.stdout:2704: read oid 21 snap -1 2024-09-11T01:47:03.585 INFO:tasks.rados.rados.0.smithi071.stdout:2704: expect (ObjNum 938 snap 294 seq_num 938) 2024-09-11T01:47:03.585 INFO:tasks.rados.rados.0.smithi071.stdout:2705: rollback oid 34 current snap is 304 2024-09-11T01:47:03.585 INFO:tasks.rados.rados.0.smithi071.stdout:2701: finishing rollback tid 1 to smithi07134777-23 2024-09-11T01:47:03.585 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 34 to 248 2024-09-11T01:47:03.585 INFO:tasks.rados.rados.0.smithi071.stdout:2701: finishing rollback tid 2 to smithi07134777-23 2024-09-11T01:47:03.586 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 1039 (ObjNum 689 snap 204 seq_num 689) dirty exists 2024-09-11T01:47:03.586 INFO:tasks.rados.rados.0.smithi071.stdout:2702: finishing write tid 1 to smithi07134777-20 2024-09-11T01:47:03.586 INFO:tasks.rados.rados.0.smithi071.stdout:2702: finishing write tid 2 to smithi07134777-20 2024-09-11T01:47:03.586 INFO:tasks.rados.rados.0.smithi071.stdout:2702: finishing write tid 3 to smithi07134777-20 2024-09-11T01:47:03.586 INFO:tasks.rados.rados.0.smithi071.stdout:2702: finishing write tid 4 to smithi07134777-20 2024-09-11T01:47:03.586 INFO:tasks.rados.rados.0.smithi071.stdout:2702: oid 20 updating version 0 to 1040 2024-09-11T01:47:03.586 INFO:tasks.rados.rados.0.smithi071.stdout:2702: oid 20 updating version 1040 to 1041 2024-09-11T01:47:03.586 INFO:tasks.rados.rados.0.smithi071.stdout:2702: oid 20 updating version 1041 to 1042 2024-09-11T01:47:03.586 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 1042 (ObjNum 958 snap 304 seq_num 958) dirty exists 2024-09-11T01:47:03.586 INFO:tasks.rados.rados.0.smithi071.stdout:2702: left oid 20 (ObjNum 958 snap 304 seq_num 958) 2024-09-11T01:47:03.586 INFO:tasks.rados.rados.0.smithi071.stdout:2701: done (3 left) 2024-09-11T01:47:03.586 INFO:tasks.rados.rados.0.smithi071.stdout:2702: done (2 left) 2024-09-11T01:47:03.586 INFO:tasks.rados.rados.0.smithi071.stdout:2706: read oid 29 snap -1 2024-09-11T01:47:03.586 INFO:tasks.rados.rados.0.smithi071.stdout:2706: expect (ObjNum 801 snap 243 seq_num 801) 2024-09-11T01:47:03.586 INFO:tasks.rados.rados.0.smithi071.stdout:2707: write oid 22 current snap is 304 2024-09-11T01:47:03.587 INFO:tasks.rados.rados.0.smithi071.stdout:2707: seq_num 959 ranges {517913=424269,1401761=446989} 2024-09-11T01:47:03.590 INFO:tasks.rados.rados.0.smithi071.stdout:2707: writing smithi07134777-22 from 517913 to 942182 tid 1 2024-09-11T01:47:03.596 INFO:tasks.rados.rados.0.smithi071.stdout:2707: writing smithi07134777-22 from 1401761 to 1848750 tid 2 2024-09-11T01:47:03.597 INFO:tasks.rados.rados.0.smithi071.stdout:2708: read oid 23 snap -1 2024-09-11T01:47:03.597 INFO:tasks.rados.rados.0.smithi071.stdout:2708: expect (ObjNum 689 snap 204 seq_num 689) 2024-09-11T01:47:03.597 INFO:tasks.rados.rados.0.smithi071.stdout:2709: read oid 25 snap -1 2024-09-11T01:47:03.597 INFO:tasks.rados.rados.0.smithi071.stdout:2709: expect (ObjNum 723 snap 224 seq_num 723) 2024-09-11T01:47:03.597 INFO:tasks.rados.rados.0.smithi071.stdout:2710: write oid 41 current snap is 304 2024-09-11T01:47:03.597 INFO:tasks.rados.rados.0.smithi071.stdout:2710: seq_num 960 ranges {534720=499518,1460243=674477} 2024-09-11T01:47:03.603 INFO:tasks.rados.rados.0.smithi071.stdout:2710: writing smithi07134777-41 from 534720 to 1034238 tid 1 2024-09-11T01:47:03.610 INFO:tasks.rados.rados.0.smithi071.stdout:2710: writing smithi07134777-41 from 1460243 to 2134720 tid 2 2024-09-11T01:47:03.610 INFO:tasks.rados.rados.0.smithi071.stdout:2711: snap_remove snap 292 2024-09-11T01:47:04.614 INFO:tasks.rados.rados.0.smithi071.stdout:2711: done (7 left) 2024-09-11T01:47:04.615 INFO:tasks.rados.rados.0.smithi071.stdout:2712: snap_create 2024-09-11T01:47:04.615 INFO:tasks.rados.rados.0.smithi071.stdout:2706: expect (ObjNum 801 snap 243 seq_num 801) 2024-09-11T01:47:04.615 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:47:04.674 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:47:04.741 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1540984 2024-09-11T01:47:04.775 INFO:tasks.rados.rados.0.smithi071.stdout:2707: finishing write tid 1 to smithi07134777-22 2024-09-11T01:47:04.775 INFO:tasks.rados.rados.0.smithi071.stdout:2707: finishing write tid 2 to smithi07134777-22 2024-09-11T01:47:04.775 INFO:tasks.rados.rados.0.smithi071.stdout:2705: finishing rollback tid 0 to smithi07134777-34 2024-09-11T01:47:04.775 INFO:tasks.rados.rados.0.smithi071.stdout:2705: finishing rollback tid 1 to smithi07134777-34 2024-09-11T01:47:04.775 INFO:tasks.rados.rados.0.smithi071.stdout:2705: finishing rollback tid 2 to smithi07134777-34 2024-09-11T01:47:04.775 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 1755 (ObjNum 811 snap 246 seq_num 811) dirty exists 2024-09-11T01:47:04.776 INFO:tasks.rados.rados.0.smithi071.stdout:2707: finishing write tid 3 to smithi07134777-22 2024-09-11T01:47:04.776 INFO:tasks.rados.rados.0.smithi071.stdout:2707: finishing write tid 4 to smithi07134777-22 2024-09-11T01:47:04.776 INFO:tasks.rados.rados.0.smithi071.stdout:2707: oid 22 updating version 0 to 1650 2024-09-11T01:47:04.776 INFO:tasks.rados.rados.0.smithi071.stdout:2707: oid 22 updating version 1650 to 1651 2024-09-11T01:47:04.776 INFO:tasks.rados.rados.0.smithi071.stdout:2707: oid 22 updating version 1651 to 1652 2024-09-11T01:47:04.776 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 1652 (ObjNum 959 snap 304 seq_num 959) dirty exists 2024-09-11T01:47:04.776 INFO:tasks.rados.rados.0.smithi071.stdout:2707: left oid 22 (ObjNum 959 snap 304 seq_num 959) 2024-09-11T01:47:04.776 INFO:tasks.rados.rados.0.smithi071.stdout:2704: expect (ObjNum 938 snap 294 seq_num 938) 2024-09-11T01:47:04.832 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3128229 2024-09-11T01:47:04.859 INFO:tasks.rados.rados.0.smithi071.stdout:2709: expect (ObjNum 723 snap 224 seq_num 723) 2024-09-11T01:47:04.862 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 604260 2024-09-11T01:47:04.914 INFO:tasks.rados.rados.0.smithi071.stdout:2710: finishing write tid 1 to smithi07134777-41 2024-09-11T01:47:04.914 INFO:tasks.rados.rados.0.smithi071.stdout:2708: expect (ObjNum 689 snap 204 seq_num 689) 2024-09-11T01:47:05.006 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:47:05.057 INFO:tasks.rados.rados.0.smithi071.stdout:2710: finishing write tid 2 to smithi07134777-41 2024-09-11T01:47:05.057 INFO:tasks.rados.rados.0.smithi071.stdout:2710: finishing write tid 3 to smithi07134777-41 2024-09-11T01:47:05.057 INFO:tasks.rados.rados.0.smithi071.stdout:2710: finishing write tid 4 to smithi07134777-41 2024-09-11T01:47:05.057 INFO:tasks.rados.rados.0.smithi071.stdout:2710: oid 41 updating version 0 to 483 2024-09-11T01:47:05.058 INFO:tasks.rados.rados.0.smithi071.stdout:2710: oid 41 updating version 483 to 484 2024-09-11T01:47:05.058 INFO:tasks.rados.rados.0.smithi071.stdout:2710: oid 41 version 484 is already newer than 482 2024-09-11T01:47:05.058 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 484 (ObjNum 960 snap 304 seq_num 960) dirty exists 2024-09-11T01:47:05.058 INFO:tasks.rados.rados.0.smithi071.stdout:2710: left oid 41 (ObjNum 960 snap 304 seq_num 960) 2024-09-11T01:47:05.616 INFO:tasks.rados.rados.0.smithi071.stdout:2704: done (7 left) 2024-09-11T01:47:05.616 INFO:tasks.rados.rados.0.smithi071.stdout:2705: done (6 left) 2024-09-11T01:47:05.616 INFO:tasks.rados.rados.0.smithi071.stdout:2706: done (5 left) 2024-09-11T01:47:05.616 INFO:tasks.rados.rados.0.smithi071.stdout:2707: done (4 left) 2024-09-11T01:47:05.616 INFO:tasks.rados.rados.0.smithi071.stdout:2708: done (3 left) 2024-09-11T01:47:05.616 INFO:tasks.rados.rados.0.smithi071.stdout:2709: done (2 left) 2024-09-11T01:47:05.616 INFO:tasks.rados.rados.0.smithi071.stdout:2710: done (1 left) 2024-09-11T01:47:05.617 INFO:tasks.rados.rados.0.smithi071.stdout:2712: done (0 left) 2024-09-11T01:47:05.617 INFO:tasks.rados.rados.0.smithi071.stdout:2713: snap_remove snap 212 2024-09-11T01:47:06.618 INFO:tasks.rados.rados.0.smithi071.stdout:2713: done (0 left) 2024-09-11T01:47:06.619 INFO:tasks.rados.rados.0.smithi071.stdout:2714: read oid 1 snap -1 2024-09-11T01:47:06.619 INFO:tasks.rados.rados.0.smithi071.stdout:2714: expect (ObjNum 857 snap 263 seq_num 857) 2024-09-11T01:47:06.619 INFO:tasks.rados.rados.0.smithi071.stdout:2715: copy_from oid 50 from oid 7 current snap is 305 2024-09-11T01:47:06.619 INFO:tasks.rados.rados.0.smithi071.stdout:2716: read oid 22 snap -1 2024-09-11T01:47:06.619 INFO:tasks.rados.rados.0.smithi071.stdout:2716: expect (ObjNum 959 snap 304 seq_num 959) 2024-09-11T01:47:06.619 INFO:tasks.rados.rados.0.smithi071.stdout:2717: read oid 14 snap 294 2024-09-11T01:47:06.619 INFO:tasks.rados.rados.0.smithi071.stdout:2717: expect (ObjNum 797 snap 243 seq_num 797) 2024-09-11T01:47:06.619 INFO:tasks.rados.rados.0.smithi071.stdout:2718: snap_remove snap 291 2024-09-11T01:47:07.585 INFO:tasks.rados.rados.0.smithi071.stdout:2718: done (4 left) 2024-09-11T01:47:07.585 INFO:tasks.rados.rados.0.smithi071.stdout:2719: write oid 38 current snap is 305 2024-09-11T01:47:07.585 INFO:tasks.rados.rados.0.smithi071.stdout:2719: seq_num 961 ranges {551527=574767,1918725=232802} 2024-09-11T01:47:07.590 INFO:tasks.rados.rados.0.smithi071.stdout:2719: writing smithi07134777-38 from 551527 to 1126294 tid 1 2024-09-11T01:47:07.597 INFO:tasks.rados.rados.0.smithi071.stdout:2719: writing smithi07134777-38 from 1918725 to 2151527 tid 2 2024-09-11T01:47:07.597 INFO:tasks.rados.rados.0.smithi071.stdout:2720: snap_remove snap 248 2024-09-11T01:47:08.590 INFO:tasks.rados.rados.0.smithi071.stdout:2720: done (5 left) 2024-09-11T01:47:08.590 INFO:tasks.rados.rados.0.smithi071.stdout:2721: delete oid 44 current snap is 305 2024-09-11T01:47:08.590 INFO:tasks.rados.rados.0.smithi071.stdout:2714: expect (ObjNum 857 snap 263 seq_num 857) 2024-09-11T01:47:08.591 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:47:08.759 INFO:tasks.rados.rados.0.smithi071.stdout:2716: expect (ObjNum 959 snap 304 seq_num 959) 2024-09-11T01:47:08.886 INFO:tasks.rados.rados.0.smithi071.stdout:2717: expect (ObjNum 797 snap 243 seq_num 797) 2024-09-11T01:47:08.915 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2783221 2024-09-11T01:47:08.926 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:47:08.978 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:47:09.032 INFO:tasks.rados.rados.0.smithi071.stdout:2715: finishing copy_from racing read to smithi07134777-50 2024-09-11T01:47:09.033 INFO:tasks.rados.rados.0.smithi071.stdout:2715: finishing copy_from to smithi07134777-50 2024-09-11T01:47:09.033 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 1688 (ObjNum 924 snap 290 seq_num 924) dirty exists 2024-09-11T01:47:09.033 INFO:tasks.rados.rados.0.smithi071.stdout:2714: done (5 left) 2024-09-11T01:47:09.033 INFO:tasks.rados.rados.0.smithi071.stdout:2715: done (4 left) 2024-09-11T01:47:09.033 INFO:tasks.rados.rados.0.smithi071.stdout:2716: done (3 left) 2024-09-11T01:47:09.033 INFO:tasks.rados.rados.0.smithi071.stdout:2717: done (2 left) 2024-09-11T01:47:09.033 INFO:tasks.rados.rados.0.smithi071.stdout:2721: done (1 left) 2024-09-11T01:47:09.033 INFO:tasks.rados.rados.0.smithi071.stdout:2722: read oid 1 snap 277 2024-09-11T01:47:09.033 INFO:tasks.rados.rados.0.smithi071.stdout:2722: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:47:09.033 INFO:tasks.rados.rados.0.smithi071.stdout:2723: copy_from oid 27 from oid 9 current snap is 305 2024-09-11T01:47:09.033 INFO:tasks.rados.rados.0.smithi071.stdout:2719: finishing write tid 1 to smithi07134777-38 2024-09-11T01:47:09.033 INFO:tasks.rados.rados.0.smithi071.stdout:2719: finishing write tid 2 to smithi07134777-38 2024-09-11T01:47:09.034 INFO:tasks.rados.rados.0.smithi071.stdout:2719: finishing write tid 3 to smithi07134777-38 2024-09-11T01:47:09.034 INFO:tasks.rados.rados.0.smithi071.stdout:2719: finishing write tid 4 to smithi07134777-38 2024-09-11T01:47:09.034 INFO:tasks.rados.rados.0.smithi071.stdout:2719: oid 38 updating version 0 to 1840 2024-09-11T01:47:09.034 INFO:tasks.rados.rados.0.smithi071.stdout:2719: oid 38 updating version 1840 to 1842 2024-09-11T01:47:09.034 INFO:tasks.rados.rados.0.smithi071.stdout:2719: oid 38 version 1842 is already newer than 1841 2024-09-11T01:47:09.034 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 1842 (ObjNum 961 snap 305 seq_num 961) dirty exists 2024-09-11T01:47:09.034 INFO:tasks.rados.rados.0.smithi071.stdout:2719: left oid 38 (ObjNum 961 snap 305 seq_num 961) 2024-09-11T01:47:09.034 INFO:tasks.rados.rados.0.smithi071.stdout:2719: done (2 left) 2024-09-11T01:47:09.034 INFO:tasks.rados.rados.0.smithi071.stdout:2724: read oid 7 snap -1 2024-09-11T01:47:09.034 INFO:tasks.rados.rados.0.smithi071.stdout:2724: expect (ObjNum 924 snap 290 seq_num 924) 2024-09-11T01:47:09.034 INFO:tasks.rados.rados.0.smithi071.stdout:2725: rollback oid 16 current snap is 305 2024-09-11T01:47:09.034 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 16 to 273 2024-09-11T01:47:09.034 INFO:tasks.rados.rados.0.smithi071.stdout:2726: write (excl) oid 26 current snap is 305 2024-09-11T01:47:09.034 INFO:tasks.rados.rados.0.smithi071.stdout:2726: seq_num 963 ranges {585141=725265,1919336=265805} 2024-09-11T01:47:09.039 INFO:tasks.rados.rados.0.smithi071.stdout:2726: writing smithi07134777-26 from 585141 to 1310406 tid 1 2024-09-11T01:47:09.044 INFO:tasks.rados.rados.0.smithi071.stdout:2726: writing smithi07134777-26 from 1919336 to 2185141 tid 2 2024-09-11T01:47:09.044 INFO:tasks.rados.rados.0.smithi071.stdout:2727: copy_from oid 22 from oid 30 current snap is 305 2024-09-11T01:47:09.044 INFO:tasks.rados.rados.0.smithi071.stdout:2723: finishing copy_from to smithi07134777-27 2024-09-11T01:47:09.044 INFO:tasks.rados.rados.0.smithi071.stdout:2723: got expected ENOENT (src dne) 2024-09-11T01:47:09.044 INFO:tasks.rados.rados.0.smithi071.stdout:2723: finishing copy_from racing read to smithi07134777-27 2024-09-11T01:47:09.045 INFO:tasks.rados.rados.0.smithi071.stdout:2724: expect (ObjNum 924 snap 290 seq_num 924) 2024-09-11T01:47:09.181 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2171512 2024-09-11T01:47:09.294 INFO:tasks.rados.rados.0.smithi071.stdout:2722: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:47:09.376 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 982381 2024-09-11T01:47:09.404 INFO:tasks.rados.rados.0.smithi071.stdout:2725: finishing rollback tid 0 to smithi07134777-16 2024-09-11T01:47:09.404 INFO:tasks.rados.rados.0.smithi071.stdout:2725: finishing rollback tid 1 to smithi07134777-16 2024-09-11T01:47:09.404 INFO:tasks.rados.rados.0.smithi071.stdout:2725: finishing rollback tid 2 to smithi07134777-16 2024-09-11T01:47:09.404 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 1626 (ObjNum 869 snap 267 seq_num 869) dirty exists 2024-09-11T01:47:09.404 INFO:tasks.rados.rados.0.smithi071.stdout:2727: finishing copy_from to smithi07134777-22 2024-09-11T01:47:09.404 INFO:tasks.rados.rados.0.smithi071.stdout:2727: got expected ENOENT (src dne) 2024-09-11T01:47:09.404 INFO:tasks.rados.rados.0.smithi071.stdout:2727: finishing copy_from racing read to smithi07134777-22 2024-09-11T01:47:09.404 INFO:tasks.rados.rados.0.smithi071.stdout:2726: finishing write tid 1 to smithi07134777-26 2024-09-11T01:47:09.404 INFO:tasks.rados.rados.0.smithi071.stdout:2726: finishing write tid 2 to smithi07134777-26 2024-09-11T01:47:09.404 INFO:tasks.rados.rados.0.smithi071.stdout:2726: finishing write tid 3 to smithi07134777-26 2024-09-11T01:47:09.405 INFO:tasks.rados.rados.0.smithi071.stdout:2726: finishing write tid 4 to smithi07134777-26 2024-09-11T01:47:09.405 INFO:tasks.rados.rados.0.smithi071.stdout:2726: oid 26 updating version 0 to 1758 2024-09-11T01:47:09.405 INFO:tasks.rados.rados.0.smithi071.stdout:2726: oid 26 version 1758 is already newer than 1756 2024-09-11T01:47:09.405 INFO:tasks.rados.rados.0.smithi071.stdout:2726: oid 26 version 1758 is already newer than 1757 2024-09-11T01:47:09.405 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 1758 (ObjNum 963 snap 305 seq_num 963) dirty exists 2024-09-11T01:47:09.405 INFO:tasks.rados.rados.0.smithi071.stdout:2726: left oid 26 (ObjNum 963 snap 305 seq_num 963) 2024-09-11T01:47:09.405 INFO:tasks.rados.rados.0.smithi071.stdout:2722: done (5 left) 2024-09-11T01:47:09.405 INFO:tasks.rados.rados.0.smithi071.stdout:2723: done (4 left) 2024-09-11T01:47:09.405 INFO:tasks.rados.rados.0.smithi071.stdout:2724: done (3 left) 2024-09-11T01:47:09.405 INFO:tasks.rados.rados.0.smithi071.stdout:2725: done (2 left) 2024-09-11T01:47:09.405 INFO:tasks.rados.rados.0.smithi071.stdout:2726: done (1 left) 2024-09-11T01:47:09.405 INFO:tasks.rados.rados.0.smithi071.stdout:2727: done (0 left) 2024-09-11T01:47:09.405 INFO:tasks.rados.rados.0.smithi071.stdout:2728: read oid 14 snap 303 2024-09-11T01:47:09.405 INFO:tasks.rados.rados.0.smithi071.stdout:2728: expect (ObjNum 942 snap 296 seq_num 942) 2024-09-11T01:47:09.405 INFO:tasks.rados.rados.0.smithi071.stdout:2729: read oid 8 snap 250 2024-09-11T01:47:09.406 INFO:tasks.rados.rados.0.smithi071.stdout:2729: expect (ObjNum 736 snap 226 seq_num 736) 2024-09-11T01:47:09.406 INFO:tasks.rados.rados.0.smithi071.stdout:2730: delete oid 50 current snap is 305 2024-09-11T01:47:09.411 INFO:tasks.rados.rados.0.smithi071.stdout:2728: expect (ObjNum 942 snap 296 seq_num 942) 2024-09-11T01:47:09.414 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 587319 2024-09-11T01:47:09.545 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:47:09.618 INFO:tasks.rados.rados.0.smithi071.stdout:2729: expect (ObjNum 736 snap 226 seq_num 736) 2024-09-11T01:47:09.656 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 754023 2024-09-11T01:47:09.707 INFO:tasks.rados.rados.0.smithi071.stdout:2728: done (2 left) 2024-09-11T01:47:09.708 INFO:tasks.rados.rados.0.smithi071.stdout:2729: done (1 left) 2024-09-11T01:47:09.708 INFO:tasks.rados.rados.0.smithi071.stdout:2730: done (0 left) 2024-09-11T01:47:09.708 INFO:tasks.rados.rados.0.smithi071.stdout:2731: read oid 29 snap -1 2024-09-11T01:47:09.708 INFO:tasks.rados.rados.0.smithi071.stdout:2731: expect (ObjNum 801 snap 243 seq_num 801) 2024-09-11T01:47:09.708 INFO:tasks.rados.rados.0.smithi071.stdout:2732: write oid 28 current snap is 305 2024-09-11T01:47:09.708 INFO:tasks.rados.rados.0.smithi071.stdout:2732: seq_num 965 ranges {618755=475763,1636300=457996} 2024-09-11T01:47:09.713 INFO:tasks.rados.rados.0.smithi071.stdout:2732: writing smithi07134777-28 from 618755 to 1094518 tid 1 2024-09-11T01:47:09.719 INFO:tasks.rados.rados.0.smithi071.stdout:2732: writing smithi07134777-28 from 1636300 to 2094296 tid 2 2024-09-11T01:47:09.719 INFO:tasks.rados.rados.0.smithi071.stdout:2733: write oid 11 current snap is 305 2024-09-11T01:47:09.719 INFO:tasks.rados.rados.0.smithi071.stdout:2733: seq_num 966 ranges {635562=667365,2094782=140780} 2024-09-11T01:47:09.726 INFO:tasks.rados.rados.0.smithi071.stdout:2733: writing smithi07134777-11 from 635562 to 1302927 tid 1 2024-09-11T01:47:09.731 INFO:tasks.rados.rados.0.smithi071.stdout:2733: writing smithi07134777-11 from 2094782 to 2235562 tid 2 2024-09-11T01:47:09.731 INFO:tasks.rados.rados.0.smithi071.stdout:2734: delete oid 2 current snap is 305 2024-09-11T01:47:09.731 INFO:tasks.rados.rados.0.smithi071.stdout:2731: expect (ObjNum 801 snap 243 seq_num 801) 2024-09-11T01:47:09.740 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1142831 2024-09-11T01:47:09.780 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:47:09.839 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 532219 2024-09-11T01:47:09.888 INFO:tasks.rados.rados.0.smithi071.stdout:2732: finishing write tid 1 to smithi07134777-28 2024-09-11T01:47:09.888 INFO:tasks.rados.rados.0.smithi071.stdout:2732: finishing write tid 2 to smithi07134777-28 2024-09-11T01:47:09.889 INFO:tasks.rados.rados.0.smithi071.stdout:2732: finishing write tid 3 to smithi07134777-28 2024-09-11T01:47:09.889 INFO:tasks.rados.rados.0.smithi071.stdout:2732: finishing write tid 4 to smithi07134777-28 2024-09-11T01:47:09.889 INFO:tasks.rados.rados.0.smithi071.stdout:2732: oid 28 updating version 0 to 1627 2024-09-11T01:47:09.889 INFO:tasks.rados.rados.0.smithi071.stdout:2732: oid 28 updating version 1627 to 1628 2024-09-11T01:47:09.889 INFO:tasks.rados.rados.0.smithi071.stdout:2732: oid 28 updating version 1628 to 1629 2024-09-11T01:47:09.889 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 1629 (ObjNum 965 snap 305 seq_num 965) dirty exists 2024-09-11T01:47:09.889 INFO:tasks.rados.rados.0.smithi071.stdout:2732: left oid 28 (ObjNum 965 snap 305 seq_num 965) 2024-09-11T01:47:09.889 INFO:tasks.rados.rados.0.smithi071.stdout:2733: finishing write tid 1 to smithi07134777-11 2024-09-11T01:47:09.889 INFO:tasks.rados.rados.0.smithi071.stdout:2733: finishing write tid 2 to smithi07134777-11 2024-09-11T01:47:09.889 INFO:tasks.rados.rados.0.smithi071.stdout:2731: done (3 left) 2024-09-11T01:47:09.889 INFO:tasks.rados.rados.0.smithi071.stdout:2732: done (2 left) 2024-09-11T01:47:09.889 INFO:tasks.rados.rados.0.smithi071.stdout:2734: done (1 left) 2024-09-11T01:47:09.889 INFO:tasks.rados.rados.0.smithi071.stdout:2735: write (excl) oid 40 current snap is 305 2024-09-11T01:47:09.890 INFO:tasks.rados.rados.0.smithi071.stdout:2733: finishing write tid 3 to smithi07134777-11 2024-09-11T01:47:09.890 INFO:tasks.rados.rados.0.smithi071.stdout:2735: seq_num 968 ranges {669176=417863,1811746=457430} 2024-09-11T01:47:09.893 INFO:tasks.rados.rados.0.smithi071.stdout:2735: writing smithi07134777-40 from 669176 to 1087039 tid 1 2024-09-11T01:47:09.900 INFO:tasks.rados.rados.0.smithi071.stdout:2735: writing smithi07134777-40 from 1811746 to 2269176 tid 2 2024-09-11T01:47:09.900 INFO:tasks.rados.rados.0.smithi071.stdout:2736: write oid 42 current snap is 305 2024-09-11T01:47:09.900 INFO:tasks.rados.rados.0.smithi071.stdout:2733: finishing write tid 4 to smithi07134777-11 2024-09-11T01:47:09.900 INFO:tasks.rados.rados.0.smithi071.stdout:2733: oid 11 updating version 0 to 1284 2024-09-11T01:47:09.901 INFO:tasks.rados.rados.0.smithi071.stdout:2733: oid 11 updating version 1284 to 1286 2024-09-11T01:47:09.901 INFO:tasks.rados.rados.0.smithi071.stdout:2733: oid 11 version 1286 is already newer than 1285 2024-09-11T01:47:09.901 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 1286 (ObjNum 966 snap 305 seq_num 966) dirty exists 2024-09-11T01:47:09.901 INFO:tasks.rados.rados.0.smithi071.stdout:2733: left oid 11 (ObjNum 966 snap 305 seq_num 966) 2024-09-11T01:47:09.901 INFO:tasks.rados.rados.0.smithi071.stdout:2735: finishing write tid 1 to smithi07134777-40 2024-09-11T01:47:09.901 INFO:tasks.rados.rados.0.smithi071.stdout:2736: seq_num 969 ranges {685983=493112,1870228=415755} 2024-09-11T01:47:09.906 INFO:tasks.rados.rados.0.smithi071.stdout:2736: writing smithi07134777-42 from 685983 to 1179095 tid 1 2024-09-11T01:47:09.913 INFO:tasks.rados.rados.0.smithi071.stdout:2736: writing smithi07134777-42 from 1870228 to 2285983 tid 2 2024-09-11T01:47:09.914 INFO:tasks.rados.rados.0.smithi071.stdout:2733: done (2 left) 2024-09-11T01:47:09.914 INFO:tasks.rados.rados.0.smithi071.stdout:2737: write oid 26 current snap is 305 2024-09-11T01:47:09.914 INFO:tasks.rados.rados.0.smithi071.stdout:2735: finishing write tid 2 to smithi07134777-40 2024-09-11T01:47:09.914 INFO:tasks.rados.rados.0.smithi071.stdout:2735: finishing write tid 3 to smithi07134777-40 2024-09-11T01:47:09.914 INFO:tasks.rados.rados.0.smithi071.stdout:2735: finishing write tid 4 to smithi07134777-40 2024-09-11T01:47:09.914 INFO:tasks.rados.rados.0.smithi071.stdout:2735: oid 40 updating version 0 to 1549 2024-09-11T01:47:09.914 INFO:tasks.rados.rados.0.smithi071.stdout:2735: oid 40 updating version 1549 to 1550 2024-09-11T01:47:09.914 INFO:tasks.rados.rados.0.smithi071.stdout:2735: oid 40 version 1550 is already newer than 1548 2024-09-11T01:47:09.914 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 1550 (ObjNum 968 snap 305 seq_num 968) dirty exists 2024-09-11T01:47:09.914 INFO:tasks.rados.rados.0.smithi071.stdout:2735: left oid 40 (ObjNum 968 snap 305 seq_num 968) 2024-09-11T01:47:09.914 INFO:tasks.rados.rados.0.smithi071.stdout:2736: finishing write tid 1 to smithi07134777-42 2024-09-11T01:47:09.914 INFO:tasks.rados.rados.0.smithi071.stdout:2737: seq_num 970 ranges {702790=568361,1812357=490433} 2024-09-11T01:47:09.920 INFO:tasks.rados.rados.0.smithi071.stdout:2737: writing smithi07134777-26 from 702790 to 1271151 tid 1 2024-09-11T01:47:09.925 INFO:tasks.rados.rados.0.smithi071.stdout:2737: writing smithi07134777-26 from 1812357 to 2302790 tid 2 2024-09-11T01:47:09.926 INFO:tasks.rados.rados.0.smithi071.stdout:2735: done (2 left) 2024-09-11T01:47:09.926 INFO:tasks.rados.rados.0.smithi071.stdout:2738: delete oid 29 current snap is 305 2024-09-11T01:47:09.926 INFO:tasks.rados.rados.0.smithi071.stdout:2736: finishing write tid 2 to smithi07134777-42 2024-09-11T01:47:09.926 INFO:tasks.rados.rados.0.smithi071.stdout:2736: finishing write tid 3 to smithi07134777-42 2024-09-11T01:47:09.926 INFO:tasks.rados.rados.0.smithi071.stdout:2736: finishing write tid 4 to smithi07134777-42 2024-09-11T01:47:09.926 INFO:tasks.rados.rados.0.smithi071.stdout:2736: oid 42 updating version 0 to 864 2024-09-11T01:47:09.926 INFO:tasks.rados.rados.0.smithi071.stdout:2736: oid 42 version 864 is already newer than 863 2024-09-11T01:47:09.926 INFO:tasks.rados.rados.0.smithi071.stdout:2736: oid 42 version 864 is already newer than 862 2024-09-11T01:47:09.926 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 864 (ObjNum 969 snap 305 seq_num 969) dirty exists 2024-09-11T01:47:09.926 INFO:tasks.rados.rados.0.smithi071.stdout:2736: left oid 42 (ObjNum 969 snap 305 seq_num 969) 2024-09-11T01:47:09.926 INFO:tasks.rados.rados.0.smithi071.stdout:2737: finishing write tid 1 to smithi07134777-26 2024-09-11T01:47:09.929 INFO:tasks.rados.rados.0.smithi071.stdout:2736: done (2 left) 2024-09-11T01:47:09.929 INFO:tasks.rados.rados.0.smithi071.stdout:2738: done (1 left) 2024-09-11T01:47:09.929 INFO:tasks.rados.rados.0.smithi071.stdout:2739: write (excl) oid 9 current snap is 305 2024-09-11T01:47:09.929 INFO:tasks.rados.rados.0.smithi071.stdout:2739: seq_num 972 ranges {736404=718859,1929321=407083} 2024-09-11T01:47:09.936 INFO:tasks.rados.rados.0.smithi071.stdout:2739: writing smithi07134777-9 from 736404 to 1455263 tid 1 2024-09-11T01:47:09.942 INFO:tasks.rados.rados.0.smithi071.stdout:2739: writing smithi07134777-9 from 1929321 to 2336404 tid 2 2024-09-11T01:47:09.942 INFO:tasks.rados.rados.0.smithi071.stdout:2740: write oid 18 current snap is 305 2024-09-11T01:47:09.942 INFO:tasks.rados.rados.0.smithi071.stdout:2737: finishing write tid 2 to smithi07134777-26 2024-09-11T01:47:09.942 INFO:tasks.rados.rados.0.smithi071.stdout:2737: finishing write tid 3 to smithi07134777-26 2024-09-11T01:47:09.942 INFO:tasks.rados.rados.0.smithi071.stdout:2737: finishing write tid 4 to smithi07134777-26 2024-09-11T01:47:09.942 INFO:tasks.rados.rados.0.smithi071.stdout:2737: oid 26 updating version 0 to 1760 2024-09-11T01:47:09.943 INFO:tasks.rados.rados.0.smithi071.stdout:2737: oid 26 updating version 1760 to 1761 2024-09-11T01:47:09.943 INFO:tasks.rados.rados.0.smithi071.stdout:2737: oid 26 version 1761 is already newer than 1759 2024-09-11T01:47:09.943 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 1761 (ObjNum 970 snap 305 seq_num 970) dirty exists 2024-09-11T01:47:09.943 INFO:tasks.rados.rados.0.smithi071.stdout:2737: left oid 26 (ObjNum 970 snap 305 seq_num 970) 2024-09-11T01:47:09.943 INFO:tasks.rados.rados.0.smithi071.stdout:2739: finishing write tid 1 to smithi07134777-9 2024-09-11T01:47:09.943 INFO:tasks.rados.rados.0.smithi071.stdout:2740: seq_num 973 ranges {753211=794108,1987803=365408} 2024-09-11T01:47:09.950 INFO:tasks.rados.rados.0.smithi071.stdout:2740: writing smithi07134777-18 from 753211 to 1547319 tid 1 2024-09-11T01:47:09.954 INFO:tasks.rados.rados.0.smithi071.stdout:2740: writing smithi07134777-18 from 1987803 to 2353211 tid 2 2024-09-11T01:47:09.955 INFO:tasks.rados.rados.0.smithi071.stdout:2737: done (2 left) 2024-09-11T01:47:09.955 INFO:tasks.rados.rados.0.smithi071.stdout:2741: write (excl) oid 34 current snap is 305 2024-09-11T01:47:09.955 INFO:tasks.rados.rados.0.smithi071.stdout:2739: finishing write tid 2 to smithi07134777-9 2024-09-11T01:47:09.955 INFO:tasks.rados.rados.0.smithi071.stdout:2739: finishing write tid 3 to smithi07134777-9 2024-09-11T01:47:09.955 INFO:tasks.rados.rados.0.smithi071.stdout:2741: seq_num 974 ranges {770018=585710,2046285=323733} 2024-09-11T01:47:09.962 INFO:tasks.rados.rados.0.smithi071.stdout:2741: writing smithi07134777-34 from 770018 to 1355728 tid 1 2024-09-11T01:47:09.967 INFO:tasks.rados.rados.0.smithi071.stdout:2741: writing smithi07134777-34 from 2046285 to 2370018 tid 2 2024-09-11T01:47:09.967 INFO:tasks.rados.rados.0.smithi071.stdout:2742: write oid 15 current snap is 305 2024-09-11T01:47:09.967 INFO:tasks.rados.rados.0.smithi071.stdout:2742: seq_num 975 ranges {786825=660959,2104767=282058} 2024-09-11T01:47:09.975 INFO:tasks.rados.rados.0.smithi071.stdout:2742: writing smithi07134777-15 from 786825 to 1447784 tid 1 2024-09-11T01:47:09.980 INFO:tasks.rados.rados.0.smithi071.stdout:2742: writing smithi07134777-15 from 2104767 to 2386825 tid 2 2024-09-11T01:47:09.980 INFO:tasks.rados.rados.0.smithi071.stdout:2743: snap_remove snap 272 2024-09-11T01:47:10.635 INFO:tasks.rados.rados.0.smithi071.stdout:2743: done (4 left) 2024-09-11T01:47:10.635 INFO:tasks.rados.rados.0.smithi071.stdout:2744: delete oid 12 current snap is 305 2024-09-11T01:47:10.636 INFO:tasks.rados.rados.0.smithi071.stdout:2739: finishing write tid 4 to smithi07134777-9 2024-09-11T01:47:10.636 INFO:tasks.rados.rados.0.smithi071.stdout:2739: oid 9 updating version 0 to 1011 2024-09-11T01:47:10.636 INFO:tasks.rados.rados.0.smithi071.stdout:2739: oid 9 updating version 1011 to 1012 2024-09-11T01:47:10.636 INFO:tasks.rados.rados.0.smithi071.stdout:2739: oid 9 updating version 1012 to 1013 2024-09-11T01:47:10.636 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 1013 (ObjNum 972 snap 305 seq_num 972) dirty exists 2024-09-11T01:47:10.636 INFO:tasks.rados.rados.0.smithi071.stdout:2739: left oid 9 (ObjNum 972 snap 305 seq_num 972) 2024-09-11T01:47:10.636 INFO:tasks.rados.rados.0.smithi071.stdout:2740: finishing write tid 1 to smithi07134777-18 2024-09-11T01:47:10.636 INFO:tasks.rados.rados.0.smithi071.stdout:2740: finishing write tid 2 to smithi07134777-18 2024-09-11T01:47:10.636 INFO:tasks.rados.rados.0.smithi071.stdout:2740: finishing write tid 3 to smithi07134777-18 2024-09-11T01:47:10.636 INFO:tasks.rados.rados.0.smithi071.stdout:2740: finishing write tid 4 to smithi07134777-18 2024-09-11T01:47:10.636 INFO:tasks.rados.rados.0.smithi071.stdout:2740: oid 18 updating version 0 to 1289 2024-09-11T01:47:10.636 INFO:tasks.rados.rados.0.smithi071.stdout:2740: oid 18 version 1289 is already newer than 1287 2024-09-11T01:47:10.637 INFO:tasks.rados.rados.0.smithi071.stdout:2740: oid 18 version 1289 is already newer than 1288 2024-09-11T01:47:10.637 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 1289 (ObjNum 973 snap 305 seq_num 973) dirty exists 2024-09-11T01:47:10.637 INFO:tasks.rados.rados.0.smithi071.stdout:2740: left oid 18 (ObjNum 973 snap 305 seq_num 973) 2024-09-11T01:47:10.637 INFO:tasks.rados.rados.0.smithi071.stdout:2741: finishing write tid 1 to smithi07134777-34 2024-09-11T01:47:10.637 INFO:tasks.rados.rados.0.smithi071.stdout:2741: finishing write tid 2 to smithi07134777-34 2024-09-11T01:47:10.637 INFO:tasks.rados.rados.0.smithi071.stdout:2741: finishing write tid 3 to smithi07134777-34 2024-09-11T01:47:10.637 INFO:tasks.rados.rados.0.smithi071.stdout:2741: finishing write tid 4 to smithi07134777-34 2024-09-11T01:47:10.637 INFO:tasks.rados.rados.0.smithi071.stdout:2741: oid 34 updating version 0 to 1764 2024-09-11T01:47:10.637 INFO:tasks.rados.rados.0.smithi071.stdout:2741: oid 34 version 1764 is already newer than 1763 2024-09-11T01:47:10.637 INFO:tasks.rados.rados.0.smithi071.stdout:2741: oid 34 version 1764 is already newer than 1762 2024-09-11T01:47:10.637 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 1764 (ObjNum 974 snap 305 seq_num 974) dirty exists 2024-09-11T01:47:10.637 INFO:tasks.rados.rados.0.smithi071.stdout:2741: left oid 34 (ObjNum 974 snap 305 seq_num 974) 2024-09-11T01:47:10.637 INFO:tasks.rados.rados.0.smithi071.stdout:2742: finishing write tid 1 to smithi07134777-15 2024-09-11T01:47:10.638 INFO:tasks.rados.rados.0.smithi071.stdout:2742: finishing write tid 2 to smithi07134777-15 2024-09-11T01:47:10.638 INFO:tasks.rados.rados.0.smithi071.stdout:2742: finishing write tid 3 to smithi07134777-15 2024-09-11T01:47:10.638 INFO:tasks.rados.rados.0.smithi071.stdout:2742: finishing write tid 4 to smithi07134777-15 2024-09-11T01:47:10.638 INFO:tasks.rados.rados.0.smithi071.stdout:2742: oid 15 updating version 0 to 1866 2024-09-11T01:47:10.638 INFO:tasks.rados.rados.0.smithi071.stdout:2742: oid 15 updating version 1866 to 1867 2024-09-11T01:47:10.638 INFO:tasks.rados.rados.0.smithi071.stdout:2742: oid 15 version 1867 is already newer than 1865 2024-09-11T01:47:10.638 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 1867 (ObjNum 975 snap 305 seq_num 975) dirty exists 2024-09-11T01:47:10.638 INFO:tasks.rados.rados.0.smithi071.stdout:2742: left oid 15 (ObjNum 975 snap 305 seq_num 975) 2024-09-11T01:47:10.643 INFO:tasks.rados.rados.0.smithi071.stdout:2739: done (4 left) 2024-09-11T01:47:10.643 INFO:tasks.rados.rados.0.smithi071.stdout:2740: done (3 left) 2024-09-11T01:47:10.643 INFO:tasks.rados.rados.0.smithi071.stdout:2741: done (2 left) 2024-09-11T01:47:10.643 INFO:tasks.rados.rados.0.smithi071.stdout:2742: done (1 left) 2024-09-11T01:47:10.643 INFO:tasks.rados.rados.0.smithi071.stdout:2744: done (0 left) 2024-09-11T01:47:10.643 INFO:tasks.rados.rados.0.smithi071.stdout:2745: delete oid 11 current snap is 305 2024-09-11T01:47:10.647 INFO:tasks.rados.rados.0.smithi071.stdout:2745: done (0 left) 2024-09-11T01:47:10.647 INFO:tasks.rados.rados.0.smithi071.stdout:2746: snap_remove snap 289 2024-09-11T01:47:11.644 INFO:tasks.rados.rados.0.smithi071.stdout:2746: done (0 left) 2024-09-11T01:47:11.644 INFO:tasks.rados.rados.0.smithi071.stdout:2747: rollback oid 47 current snap is 305 2024-09-11T01:47:11.644 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 47 to 225 2024-09-11T01:47:11.645 INFO:tasks.rados.rados.0.smithi071.stdout:2748: snap_create 2024-09-11T01:47:11.655 INFO:tasks.rados.rados.0.smithi071.stdout:2747: finishing rollback tid 0 to smithi07134777-47 2024-09-11T01:47:11.656 INFO:tasks.rados.rados.0.smithi071.stdout:2747: finishing rollback tid 1 to smithi07134777-47 2024-09-11T01:47:11.656 INFO:tasks.rados.rados.0.smithi071.stdout:2747: finishing rollback tid 2 to smithi07134777-47 2024-09-11T01:47:11.656 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 1754 (ObjNum 715 snap 221 seq_num 715) dirty exists 2024-09-11T01:47:12.584 INFO:tasks.rados.rados.0.smithi071.stdout:2747: done (1 left) 2024-09-11T01:47:12.584 INFO:tasks.rados.rados.0.smithi071.stdout:2748: done (0 left) 2024-09-11T01:47:12.584 INFO:tasks.rados.rados.0.smithi071.stdout:2749: write oid 2 current snap is 306 2024-09-11T01:47:12.584 INFO:tasks.rados.rados.0.smithi071.stdout:2749: seq_num 978 ranges {437246=486706,1363860=623668} 2024-09-11T01:47:12.589 INFO:tasks.rados.rados.0.smithi071.stdout:2749: writing smithi07134777-2 from 437246 to 923952 tid 1 2024-09-11T01:47:12.595 INFO:tasks.rados.rados.0.smithi071.stdout:2749: writing smithi07134777-2 from 1363860 to 1987528 tid 2 2024-09-11T01:47:12.595 INFO:tasks.rados.rados.0.smithi071.stdout:2750: snap_remove snap 174 2024-09-11T01:47:13.586 INFO:tasks.rados.rados.0.smithi071.stdout:2750: done (1 left) 2024-09-11T01:47:13.586 INFO:tasks.rados.rados.0.smithi071.stdout:2751: rollback oid 1 current snap is 306 2024-09-11T01:47:13.586 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 1 to 259 2024-09-11T01:47:13.586 INFO:tasks.rados.rados.0.smithi071.stdout:2749: finishing write tid 1 to smithi07134777-2 2024-09-11T01:47:13.586 INFO:tasks.rados.rados.0.smithi071.stdout:2749: finishing write tid 2 to smithi07134777-2 2024-09-11T01:47:13.586 INFO:tasks.rados.rados.0.smithi071.stdout:2749: finishing write tid 3 to smithi07134777-2 2024-09-11T01:47:13.587 INFO:tasks.rados.rados.0.smithi071.stdout:2749: finishing write tid 4 to smithi07134777-2 2024-09-11T01:47:13.587 INFO:tasks.rados.rados.0.smithi071.stdout:2749: oid 2 updating version 0 to 866 2024-09-11T01:47:13.587 INFO:tasks.rados.rados.0.smithi071.stdout:2749: oid 2 version 866 is already newer than 865 2024-09-11T01:47:13.587 INFO:tasks.rados.rados.0.smithi071.stdout:2749: oid 2 updating version 866 to 867 2024-09-11T01:47:13.587 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 867 (ObjNum 978 snap 306 seq_num 978) dirty exists 2024-09-11T01:47:13.587 INFO:tasks.rados.rados.0.smithi071.stdout:2749: left oid 2 (ObjNum 978 snap 306 seq_num 978) 2024-09-11T01:47:13.587 INFO:tasks.rados.rados.0.smithi071.stdout:2749: done (1 left) 2024-09-11T01:47:13.587 INFO:tasks.rados.rados.0.smithi071.stdout:2752: read oid 21 snap -1 2024-09-11T01:47:13.587 INFO:tasks.rados.rados.0.smithi071.stdout:2752: expect (ObjNum 938 snap 294 seq_num 938) 2024-09-11T01:47:13.587 INFO:tasks.rados.rados.0.smithi071.stdout:2753: write (excl) oid 33 current snap is 306 2024-09-11T01:47:13.587 INFO:tasks.rados.rados.0.smithi071.stdout:2753: seq_num 979 ranges {454053=561955,1422342=483679,2368404=85649} 2024-09-11T01:47:13.592 INFO:tasks.rados.rados.0.smithi071.stdout:2753: writing smithi07134777-33 from 454053 to 1016008 tid 1 2024-09-11T01:47:13.597 INFO:tasks.rados.rados.0.smithi071.stdout:2753: writing smithi07134777-33 from 1422342 to 1906021 tid 2 2024-09-11T01:47:13.601 INFO:tasks.rados.rados.0.smithi071.stdout:2753: writing smithi07134777-33 from 2368404 to 2454053 tid 3 2024-09-11T01:47:13.601 INFO:tasks.rados.rados.0.smithi071.stdout:2754: delete oid 22 current snap is 306 2024-09-11T01:47:13.601 INFO:tasks.rados.rados.0.smithi071.stdout:2752: expect (ObjNum 938 snap 294 seq_num 938) 2024-09-11T01:47:13.713 INFO:tasks.rados.rados.0.smithi071.stdout:2751: finishing rollback tid 0 to smithi07134777-1 2024-09-11T01:47:13.713 INFO:tasks.rados.rados.0.smithi071.stdout:2751: finishing rollback tid 1 to smithi07134777-1 2024-09-11T01:47:13.713 INFO:tasks.rados.rados.0.smithi071.stdout:2752: done (3 left) 2024-09-11T01:47:13.713 INFO:tasks.rados.rados.0.smithi071.stdout:2754: done (2 left) 2024-09-11T01:47:13.713 INFO:tasks.rados.rados.0.smithi071.stdout:2755: snap_create 2024-09-11T01:47:13.713 INFO:tasks.rados.rados.0.smithi071.stdout:2751: finishing rollback tid 2 to smithi07134777-1 2024-09-11T01:47:13.714 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 1757 (ObjNum 688 snap 204 seq_num 688) dirty exists 2024-09-11T01:47:13.714 INFO:tasks.rados.rados.0.smithi071.stdout:2753: finishing write tid 1 to smithi07134777-33 2024-09-11T01:47:13.714 INFO:tasks.rados.rados.0.smithi071.stdout:2753: finishing write tid 2 to smithi07134777-33 2024-09-11T01:47:13.714 INFO:tasks.rados.rados.0.smithi071.stdout:2753: finishing write tid 3 to smithi07134777-33 2024-09-11T01:47:13.714 INFO:tasks.rados.rados.0.smithi071.stdout:2753: finishing write tid 4 to smithi07134777-33 2024-09-11T01:47:13.714 INFO:tasks.rados.rados.0.smithi071.stdout:2753: finishing write tid 5 to smithi07134777-33 2024-09-11T01:47:13.714 INFO:tasks.rados.rados.0.smithi071.stdout:2753: oid 33 updating version 0 to 1768 2024-09-11T01:47:13.714 INFO:tasks.rados.rados.0.smithi071.stdout:2753: oid 33 version 1768 is already newer than 1767 2024-09-11T01:47:13.714 INFO:tasks.rados.rados.0.smithi071.stdout:2753: oid 33 version 1768 is already newer than 1766 2024-09-11T01:47:13.714 INFO:tasks.rados.rados.0.smithi071.stdout:2753: oid 33 version 1768 is already newer than 1765 2024-09-11T01:47:13.714 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 1768 (ObjNum 979 snap 306 seq_num 979) dirty exists 2024-09-11T01:47:13.714 INFO:tasks.rados.rados.0.smithi071.stdout:2753: left oid 33 (ObjNum 979 snap 306 seq_num 979) 2024-09-11T01:47:14.281 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-11T01:47:14.282 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T01:47:14.282 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T01:47:14.285 INFO:tasks.ceph.mon.a:Started 2024-09-11T01:47:14.285 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-11T01:47:14.285 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-11T01:47:14.286 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T01:47:29.286 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-11T01:47:29.287 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-11T01:47:29.287 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:47:29.287 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:47:29.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.398+0000 7ff633e65640 10 monclient: get_monmap_and_config 2024-09-11T01:47:29.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.398+0000 7ff633e65640 10 monclient: build_initial_monmap 2024-09-11T01:47:29.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.398+0000 7ff633e65640 10 monclient: monmap: 2024-09-11T01:47:29.400 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:47:29.400 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:29.400 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:47:29.398846+0000 2024-09-11T01:47:29.400 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:47:29.398846+0000 2024-09-11T01:47:29.400 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:47:29.400 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:29.400 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:47:29.400 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:47:29.400 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:47:29.400 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:47:29.400 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:47:29.400 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:29.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.398+0000 7ff633e65640 1 Processor -- start 2024-09-11T01:47:29.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.399+0000 7ff633e65640 1 -- start start 2024-09-11T01:47:29.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.399+0000 7ff633e65640 10 monclient: init 2024-09-11T01:47:29.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.399+0000 7ff633e65640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:29.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.399+0000 7ff633e65640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-11T01:47:29.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.399+0000 7ff633e65640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c138230 0x7ff62c138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:29.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.399+0000 7ff633e65640 10 monclient(hunting): picked mon.noname-b con 0x7ff62c138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:47:29.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.399+0000 7ff633e65640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff62c09d030 0x7ff62c0a5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:29.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.399+0000 7ff633e65640 10 monclient(hunting): picked mon.noname-a con 0x7ff62c09d030 addr v2:172.21.15.12:3300/0 2024-09-11T01:47:29.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.399+0000 7ff633e65640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff62c09d430 0x7ff62c0a5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:29.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.399+0000 7ff633e65640 10 monclient(hunting): picked mon.noname-c con 0x7ff62c09d430 addr v2:172.21.15.71:3300/0 2024-09-11T01:47:29.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.399+0000 7ff633e65640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:29.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.399+0000 7ff633e65640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff62c09c9c0 con 0x7ff62c09d030 2024-09-11T01:47:29.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.399+0000 7ff633e65640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:29.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.399+0000 7ff633e65640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7ff62c0a6e30 con 0x7ff62c09d430 2024-09-11T01:47:29.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.399+0000 7ff633e65640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:29.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.399+0000 7ff633e65640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff62c09ba20 con 0x7ff62c138230 2024-09-11T01:47:29.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff633e65640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:29.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff633e65640 10 monclient(hunting): authenticate will time out at 1855.842011s 2024-09-11T01:47:29.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff631400640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff62c09d030 0x7ff62c0a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:29.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff631400640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff62c09d030 0x7ff62c0a5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38414/0 (socket says 172.21.15.12:38414) 2024-09-11T01:47:29.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff631400640 1 -- 172.21.15.12:0/3424312083 learned_addr learned my addr 172.21.15.12:0/3424312083 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:47:29.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff631400640 10 monclient(hunting): get_auth_request con 0x7ff62c09d030 auth_method 0 2024-09-11T01:47:29.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff631400640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:29.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff631400640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff631400640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff631400640 1 -- 172.21.15.12:0/3424312083 >> v2:172.21.15.12:3300/0 conn(0x7ff62c09d030 msgr2=0x7ff62c0a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:47:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff631c01640 1 --2- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c138230 0x7ff62c138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff631400640 1 -- 172.21.15.12:0/3424312083 >> v2:172.21.15.12:3300/0 conn(0x7ff62c09d030 msgr2=0x7ff62c0a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:47:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff631400640 1 --2- 172.21.15.12:0/3424312083 >> v2:172.21.15.12:3300/0 conn(0x7ff62c09d030 0x7ff62c0a5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:47:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff631c01640 10 monclient(hunting): get_auth_request con 0x7ff62c138230 auth_method 0 2024-09-11T01:47:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff631c01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff631c01640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff631c01640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff631400640 1 --2- 172.21.15.12:0/3424312083 >> v2:172.21.15.12:3300/0 conn(0x7ff62c09d030 0x7ff62c0a5020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:47:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff632402640 1 --2- 172.21.15.12:0/3424312083 >> v2:172.21.15.71:3300/0 conn(0x7ff62c09d430 0x7ff62c0a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff631c01640 1 -- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c138230 msgr2=0x7ff62c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:47:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff631c01640 1 -- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c138230 msgr2=0x7ff62c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:47:29.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff631c01640 1 --2- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c138230 0x7ff62c138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:47:29.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff631c01640 1 --2- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c138230 0x7ff62c138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:47:29.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff632402640 10 monclient(hunting): get_auth_request con 0x7ff62c09d430 auth_method 0 2024-09-11T01:47:29.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff632402640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:29.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff632402640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:29.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff632402640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:29.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff632402640 1 -- 172.21.15.12:0/3424312083 >> v2:172.21.15.71:3300/0 conn(0x7ff62c09d430 msgr2=0x7ff62c0a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:47:29.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff632402640 1 -- 172.21.15.12:0/3424312083 >> v2:172.21.15.71:3300/0 conn(0x7ff62c09d430 msgr2=0x7ff62c0a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:47:29.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff632402640 1 --2- 172.21.15.12:0/3424312083 >> v2:172.21.15.71:3300/0 conn(0x7ff62c09d430 0x7ff62c0a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:47:29.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.400+0000 7ff632402640 1 --2- 172.21.15.12:0/3424312083 >> v2:172.21.15.71:3300/0 conn(0x7ff62c09d430 0x7ff62c0a5560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:47:29.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff631400640 1 --2- 172.21.15.12:0/3424312083 >> v2:172.21.15.12:3300/0 conn(0x7ff62c09d030 0x7ff62c0a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:29.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff631400640 10 monclient(hunting): get_auth_request con 0x7ff62c09d030 auth_method 0 2024-09-11T01:47:29.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff631400640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:29.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff631400640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:29.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff631400640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:47:29.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff631c01640 1 --2- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c138230 0x7ff62c138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:29.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff631c01640 10 monclient(hunting): get_auth_request con 0x7ff62c138230 auth_method 0 2024-09-11T01:47:29.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff631c01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:29.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff631c01640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:29.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff631c01640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:47:29.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff631400640 1 -- 172.21.15.12:0/3424312083 >> v2:172.21.15.12:3300/0 conn(0x7ff62c09d030 msgr2=0x7ff62c0a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:47:29.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff631400640 1 -- 172.21.15.12:0/3424312083 >> v2:172.21.15.12:3300/0 conn(0x7ff62c09d030 msgr2=0x7ff62c0a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:47:29.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff631400640 1 --2- 172.21.15.12:0/3424312083 >> v2:172.21.15.12:3300/0 conn(0x7ff62c09d030 0x7ff62c0a5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:47:29.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff631c01640 1 -- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c138230 msgr2=0x7ff62c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:47:29.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff631c01640 1 -- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c138230 msgr2=0x7ff62c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:47:29.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff631c01640 1 --2- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c138230 0x7ff62c138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:47:29.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff631400640 1 --2- 172.21.15.12:0/3424312083 >> v2:172.21.15.12:3300/0 conn(0x7ff62c09d030 0x7ff62c0a5020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:47:29.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff631c01640 1 --2- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c138230 0x7ff62c138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:47:29.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff632402640 1 --2- 172.21.15.12:0/3424312083 >> v2:172.21.15.71:3300/0 conn(0x7ff62c09d430 0x7ff62c0a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:29.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff632402640 10 monclient(hunting): get_auth_request con 0x7ff62c09d430 auth_method 0 2024-09-11T01:47:29.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff632402640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:29.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff632402640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:29.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff632402640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:47:29.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff632402640 1 -- 172.21.15.12:0/3424312083 >> v2:172.21.15.71:3300/0 conn(0x7ff62c09d430 msgr2=0x7ff62c0a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:47:29.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff632402640 1 -- 172.21.15.12:0/3424312083 >> v2:172.21.15.71:3300/0 conn(0x7ff62c09d430 msgr2=0x7ff62c0a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:47:29.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff632402640 1 --2- 172.21.15.12:0/3424312083 >> v2:172.21.15.71:3300/0 conn(0x7ff62c09d430 0x7ff62c0a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:47:29.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:29.601+0000 7ff632402640 1 --2- 172.21.15.12:0/3424312083 >> v2:172.21.15.71:3300/0 conn(0x7ff62c09d430 0x7ff62c0a5560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:47:30.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631400640 1 --2- 172.21.15.12:0/3424312083 >> v2:172.21.15.12:3300/0 conn(0x7ff62c09d030 0x7ff62c0a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631c01640 1 --2- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c138230 0x7ff62c138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631c01640 10 monclient(hunting): get_auth_request con 0x7ff62c138230 auth_method 0 2024-09-11T01:47:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631c01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631c01640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631c01640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:47:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631400640 10 monclient(hunting): get_auth_request con 0x7ff62c09d030 auth_method 0 2024-09-11T01:47:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631400640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631400640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631400640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:47:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631c01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631c01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631c01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631400640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631400640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631400640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631400640 10 monclient(hunting): handle_auth_done global_id 204099 payload 274 2024-09-11T01:47:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631400640 1 -- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c138230 msgr2=0x7ff62c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:47:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631400640 1 --2- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c138230 0x7ff62c138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631400640 1 -- 172.21.15.12:0/3424312083 >> v2:172.21.15.71:3300/0 conn(0x7ff62c09d430 msgr2=0x7ff62c0a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:47:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff632402640 1 --2- 172.21.15.12:0/3424312083 >> v2:172.21.15.71:3300/0 conn(0x7ff62c09d430 0x7ff62c0a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631400640 1 --2- 172.21.15.12:0/3424312083 >> v2:172.21.15.71:3300/0 conn(0x7ff62c09d430 0x7ff62c0a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631400640 10 monclient: _finish_hunting 0 2024-09-11T01:47:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631400640 1 monclient: found mon.noname-a 2024-09-11T01:47:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631400640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631400640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:47:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631400640 1 -- 172.21.15.12:0/3424312083 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff62c0a66a0 con 0x7ff62c09d030 2024-09-11T01:47:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff631c01640 1 --2- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c138230 0x7ff62c138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:47:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff632402640 10 monclient: get_auth_request con 0x7ff62c09d430 auth_method 0 2024-09-11T01:47:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.002+0000 7ff632402640 1 --2- 172.21.15.12:0/3424312083 >> v2:172.21.15.71:3300/0 conn(0x7ff62c09d430 0x7ff62c0a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:47:30.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff631400640 1 -- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c09d030 msgr2=0x7ff62c0a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:30.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff631400640 1 --2- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c09d030 0x7ff62c0a5020 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff62c09cf80 tx=0x7ff61c02a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67f9d8a1e81537ba server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:30.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff630bff640 1 -- 172.21.15.12:0/3424312083 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff61c038070 con 0x7ff62c09d030 2024-09-11T01:47:30.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff630bff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:30.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff630bff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:47:30.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff631400640 1 -- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c09d030 msgr2=0x7ff62c0a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-11T01:47:30.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff631400640 1 -- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c09d030 msgr2=0x7ff62c0a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:47:30.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff630bff640 10 monclient: dump: 2024-09-11T01:47:30.011 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:30.011 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.011 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.011 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.011 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:30.011 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.012 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:30.012 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:30.012 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:30.012 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff631400640 1 --2- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c09d030 0x7ff62c0a5020 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff62c09cf80 tx=0x7ff61c02a630 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:47:30.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff631400640 1 --2- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c09d030 0x7ff62c0a5020 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff62c09cf80 tx=0x7ff61c02a630 comp rx=0 tx=0).stop 2024-09-11T01:47:30.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff630bff640 10 monclient: _finish_auth 0 2024-09-11T01:47:30.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff630bff640 10 monclient: _check_auth_tickets 2024-09-11T01:47:30.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff630bff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:30.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff630bff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:47:30.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff630bff640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:30.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff630bff640 1 -- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c09d030 msgr2=0x7ff62c0a5020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:47:30.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff630bff640 1 --2- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c09d030 0x7ff62c0a5020 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff630bff640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:47:30.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff630bff640 1 --2- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c138230 0x7ff618005760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff630bff640 10 monclient(hunting): picked mon.c con 0x7ff62c138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:47:30.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff630bff640 1 --2- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff618005ec0 0x7ff618008350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff630bff640 10 monclient(hunting): picked mon.a con 0x7ff618005ec0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:47:30.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff630bff640 1 --2- 172.21.15.12:0/3424312083 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff618008b90 0x7ff61800b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff630bff640 10 monclient(hunting): picked mon.b con 0x7ff618008b90 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:47:30.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff630bff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff630bff640 1 -- 172.21.15.12:0/3424312083 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff61800b960 con 0x7ff618005ec0 2024-09-11T01:47:30.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff630bff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff630bff640 1 -- 172.21.15.12:0/3424312083 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff61800bb10 con 0x7ff618008b90 2024-09-11T01:47:30.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff630bff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff630bff640 1 -- 172.21.15.12:0/3424312083 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff61800bd10 con 0x7ff62c138230 2024-09-11T01:47:30.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff630bff640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:30.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff630bff640 10 monclient(hunting): ms_handle_reset hunted mon [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:47:30.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff631c01640 1 --2- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c138230 0x7ff618005760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff631400640 1 --2- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff618005ec0 0x7ff618008350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff631c01640 10 monclient(hunting): get_auth_request con 0x7ff62c138230 auth_method 0 2024-09-11T01:47:30.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff631c01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff631c01640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff631c01640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:47:30.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.005+0000 7ff631400640 10 monclient(hunting): get_auth_request con 0x7ff618005ec0 auth_method 0 2024-09-11T01:47:30.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff631400640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff631400640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff631400640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:47:30.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff631c01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff631c01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff631c01640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:47:30.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff631400640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff631400640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff631400640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:47:30.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff632402640 1 --2- 172.21.15.12:0/3424312083 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff618008b90 0x7ff61800b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff632402640 10 monclient(hunting): get_auth_request con 0x7ff618008b90 auth_method 0 2024-09-11T01:47:30.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff632402640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff632402640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff632402640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:47:30.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff631c01640 10 monclient(hunting): handle_auth_done global_id 204099 payload 293 2024-09-11T01:47:30.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff631c01640 1 -- 172.21.15.12:0/3424312083 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff618008b90 msgr2=0x7ff61800b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:47:30.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff631c01640 1 --2- 172.21.15.12:0/3424312083 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff618008b90 0x7ff61800b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff631c01640 1 -- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff618005ec0 msgr2=0x7ff618008350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:47:30.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff631c01640 1 --2- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff618005ec0 0x7ff618008350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff631c01640 10 monclient: _finish_hunting 0 2024-09-11T01:47:30.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff631c01640 1 monclient: found mon.c 2024-09-11T01:47:30.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff631c01640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:30.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff631c01640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:47:30.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff631c01640 1 -- 172.21.15.12:0/3424312083 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=2+}) -- 0x7ff61800bfd0 con 0x7ff62c138230 2024-09-11T01:47:30.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff631c01640 10 monclient: _finish_auth 0 2024-09-11T01:47:30.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff631c01640 10 monclient: _check_auth_tickets 2024-09-11T01:47:30.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff631c01640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:30.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff633e65640 5 monclient: authenticate success, global_id 204099 2024-09-11T01:47:30.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff633e65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:47:30.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.006+0000 7ff631400640 1 --2- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff618005ec0 0x7ff618008350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:47:30.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.008+0000 7ff631c01640 1 -- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c138230 msgr2=0x7ff618005760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:30.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.008+0000 7ff631c01640 1 --2- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c138230 0x7ff618005760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff628004650 tx=0x7ff61c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb0215499f82d900 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:30.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.008+0000 7ff630bff640 1 -- 172.21.15.12:0/3424312083 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff628019070 con 0x7ff62c138230 2024-09-11T01:47:30.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.008+0000 7ff630bff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:30.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.008+0000 7ff630bff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:47:30.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.008+0000 7ff630bff640 10 monclient: dump: 2024-09-11T01:47:30.017 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:30.017 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.017 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.017 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.017 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:30.017 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.017 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:30.018 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:30.018 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:30.018 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.008+0000 7ff630bff640 1 -- 172.21.15.12:0/3424312083 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff62800d860 con 0x7ff62c138230 2024-09-11T01:47:30.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.008+0000 7ff633e65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:47:30.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.008+0000 7ff630bff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:47:30.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.008+0000 7ff633e65640 10 monclient: get_monmap_and_config success 2024-09-11T01:47:30.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.008+0000 7ff633e65640 4 set_mon_vals no callback set 2024-09-11T01:47:30.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.009+0000 7ff633e65640 10 monclient: shutdown 2024-09-11T01:47:30.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.009+0000 7ff633e65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:47:30.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.009+0000 7ff633e65640 1 -- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c138230 msgr2=0x7ff618005760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.009+0000 7ff633e65640 1 --2- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c138230 0x7ff618005760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff628004650 tx=0x7ff61c008040 comp rx=0 tx=0).stop 2024-09-11T01:47:30.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.009+0000 7ff632402640 1 -- 172.21.15.12:0/3424312083 reap_dead start 2024-09-11T01:47:30.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.009+0000 7ff633e65640 1 -- 172.21.15.12:0/3424312083 shutdown_connections 2024-09-11T01:47:30.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.009+0000 7ff633e65640 1 --2- 172.21.15.12:0/3424312083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c09d030 0x7ff62c0a5020 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.009+0000 7ff633e65640 1 -- 172.21.15.12:0/3424312083 >> 172.21.15.12:0/3424312083 conn(0x7ff62c0a7120 msgr2=0x7ff62c0a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:47:30.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.009+0000 7ff633e65640 1 -- 172.21.15.12:0/3424312083 shutdown_connections 2024-09-11T01:47:30.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.009+0000 7ff633e65640 1 -- 172.21.15.12:0/3424312083 wait complete. 2024-09-11T01:47:30.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.010+0000 7ff633e65640 10 monclient: build_initial_monmap 2024-09-11T01:47:30.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.010+0000 7ff633e65640 10 monclient: monmap: 2024-09-11T01:47:30.025 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:47:30.025 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:47:30.025 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:47:30.025 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:47:30.025 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:47:30.025 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.025 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:47:30.025 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:47:30.025 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:47:30.025 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.011+0000 7ff633e65640 1 Processor -- start 2024-09-11T01:47:30.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.011+0000 7ff633e65640 1 -- start start 2024-09-11T01:47:30.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.011+0000 7ff633e65640 10 monclient: init 2024-09-11T01:47:30.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.011+0000 7ff633e65640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:30.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.011+0000 7ff633e65640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:47:30.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.011+0000 7ff633e65640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c09d030 0x7ff62c137040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.011+0000 7ff633e65640 10 monclient(hunting): picked mon.noname-a con 0x7ff62c09d030 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:47:30.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.011+0000 7ff633e65640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c138230 0x7ff62c137580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.011+0000 7ff633e65640 10 monclient(hunting): picked mon.noname-c con 0x7ff62c138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:47:30.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.011+0000 7ff633e65640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff62c1cd0f0 0x7ff62c120c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.011+0000 7ff633e65640 10 monclient(hunting): picked mon.noname-b con 0x7ff62c1cd0f0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:47:30.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.011+0000 7ff633e65640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.011+0000 7ff633e65640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff62c0a5c20 con 0x7ff62c09d030 2024-09-11T01:47:30.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.011+0000 7ff633e65640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.011+0000 7ff633e65640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff62c0a5aa0 con 0x7ff62c1cd0f0 2024-09-11T01:47:30.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.011+0000 7ff633e65640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.011+0000 7ff633e65640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff62c0a5da0 con 0x7ff62c138230 2024-09-11T01:47:30.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.011+0000 7ff633e65640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:30.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.011+0000 7ff633e65640 10 monclient(hunting): authenticate will time out at 1856.453832s 2024-09-11T01:47:30.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.011+0000 7ff631c01640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c09d030 0x7ff62c137040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.011+0000 7ff631400640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c138230 0x7ff62c137580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.011+0000 7ff631c01640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c09d030 0x7ff62c137040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38454/0 (socket says 172.21.15.12:38454) 2024-09-11T01:47:30.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.011+0000 7ff631c01640 1 -- 172.21.15.12:0/1035780994 learned_addr learned my addr 172.21.15.12:0/1035780994 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:47:30.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.011+0000 7ff631c01640 10 monclient(hunting): get_auth_request con 0x7ff62c09d030 auth_method 0 2024-09-11T01:47:30.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff631c01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff631c01640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff631c01640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff631400640 10 monclient(hunting): get_auth_request con 0x7ff62c138230 auth_method 0 2024-09-11T01:47:30.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff631400640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff631400640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff631400640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff631c01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff631c01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff631c01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:30.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff631400640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff631400640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff631400640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:30.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff632402640 1 --2- 172.21.15.12:0/1035780994 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff62c1cd0f0 0x7ff62c120c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff632402640 10 monclient(hunting): get_auth_request con 0x7ff62c1cd0f0 auth_method 0 2024-09-11T01:47:30.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff632402640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff632402640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff632402640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff631c01640 10 monclient(hunting): handle_auth_done global_id 204102 payload 824 2024-09-11T01:47:30.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff631c01640 1 -- 172.21.15.12:0/1035780994 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c138230 msgr2=0x7ff62c137580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff631c01640 1 --2- 172.21.15.12:0/1035780994 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c138230 0x7ff62c137580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff631c01640 1 -- 172.21.15.12:0/1035780994 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff62c1cd0f0 msgr2=0x7ff62c120c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff631c01640 1 --2- 172.21.15.12:0/1035780994 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff62c1cd0f0 0x7ff62c120c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff631c01640 10 monclient: _finish_hunting 0 2024-09-11T01:47:30.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff631c01640 1 monclient: found mon.noname-a 2024-09-11T01:47:30.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff631c01640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:30.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff631c01640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:47:30.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.012+0000 7ff631c01640 1 -- 172.21.15.12:0/1035780994 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff62c1226a0 con 0x7ff62c09d030 2024-09-11T01:47:30.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.014+0000 7ff631c01640 1 -- 172.21.15.12:0/1035780994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c09d030 msgr2=0x7ff62c137040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:30.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.014+0000 7ff631c01640 1 --2- 172.21.15.12:0/1035780994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c09d030 0x7ff62c137040 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff628008010 tx=0x7ff628004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:30.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.014+0000 7ff6227fc640 1 -- 172.21.15.12:0/1035780994 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62802f070 con 0x7ff62c09d030 2024-09-11T01:47:30.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.014+0000 7ff6227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:30.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.014+0000 7ff6227fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:47:30.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.014+0000 7ff6227fc640 10 monclient: dump: 2024-09-11T01:47:30.030 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:30.030 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.030 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.030 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.030 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:30.030 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.030 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:30.030 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:30.031 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.014+0000 7ff6227fc640 10 monclient: _finish_auth 0 2024-09-11T01:47:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.014+0000 7ff6227fc640 10 monclient: _check_auth_tickets 2024-09-11T01:47:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.014+0000 7ff6227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.014+0000 7ff6227fc640 1 -- 172.21.15.12:0/1035780994 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff62800de40 con 0x7ff62c09d030 2024-09-11T01:47:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.014+0000 7ff6227fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:47:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.014+0000 7ff6227fc640 1 -- 172.21.15.12:0/1035780994 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff628019040 con 0x7ff62c09d030 2024-09-11T01:47:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.014+0000 7ff633e65640 5 monclient: authenticate success, global_id 204102 2024-09-11T01:47:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.014+0000 7ff633e65640 10 monclient: _renew_subs 2024-09-11T01:47:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.014+0000 7ff633e65640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:47:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.014+0000 7ff633e65640 1 -- 172.21.15.12:0/1035780994 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff62c0a1810 con 0x7ff62c09d030 2024-09-11T01:47:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.014+0000 7ff623fff640 4 set_mon_vals no callback set 2024-09-11T01:47:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.015+0000 7ff6227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:30.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.015+0000 7ff6227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:47:30.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.015+0000 7ff6227fc640 10 monclient: dump: 2024-09-11T01:47:30.032 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:30.032 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.032 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.032 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.032 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:30.032 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.032 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:30.032 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:30.032 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:30.032 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.015+0000 7ff633e65640 10 monclient: _renew_subs 2024-09-11T01:47:30.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.015+0000 7ff633e65640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:47:30.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.015+0000 7ff633e65640 1 -- 172.21.15.12:0/1035780994 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff62c124160 con 0x7ff62c09d030 2024-09-11T01:47:30.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.016+0000 7ff6227fc640 1 -- 172.21.15.12:0/1035780994 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff62801fbe0 con 0x7ff62c09d030 2024-09-11T01:47:30.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.016+0000 7ff6227fc640 1 --2- 172.21.15.12:0/1035780994 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff61003e260 0x7ff610040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.016+0000 7ff633e65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:30.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.016+0000 7ff633e65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:30.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.016+0000 7ff633e65640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:47:30.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.016+0000 7ff633e65640 1 -- 172.21.15.12:0/1035780994 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff62c138610 con 0x7ff62c09d030 2024-09-11T01:47:30.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.016+0000 7ff6227fc640 1 -- 172.21.15.12:0/1035780994 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(616..616 src has 507..616) ==== 4831+0+0 (secure 0 0 0) 0x7ff628054540 con 0x7ff62c09d030 2024-09-11T01:47:30.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.016+0000 7ff631400640 1 --2- 172.21.15.12:0/1035780994 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff61003e260 0x7ff610040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.016+0000 7ff631400640 10 monclient: get_auth_request con 0x7ff61003e260 auth_method 0 2024-09-11T01:47:30.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.019+0000 7ff631400640 1 --2- 172.21.15.12:0/1035780994 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff61003e260 0x7ff610040720 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ff62c123880 tx=0x7ff61c02e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:30.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.021+0000 7ff6227fc640 1 -- 172.21.15.12:0/1035780994 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff628039080 con 0x7ff62c09d030 2024-09-11T01:47:30.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.021+0000 7ff6227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:30.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.021+0000 7ff6227fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:47:30.053 INFO:tasks.rados.rados.0.smithi071.stdout:2751: done (2 left) 2024-09-11T01:47:30.053 INFO:tasks.rados.rados.0.smithi071.stdout:2753: done (1 left) 2024-09-11T01:47:30.053 INFO:tasks.rados.rados.0.smithi071.stdout:2755: done (0 left) 2024-09-11T01:47:30.053 INFO:tasks.rados.rados.0.smithi071.stdout:2756: snap_create 2024-09-11T01:47:30.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.285+0000 7ff633e65640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:47:30.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.285+0000 7ff633e65640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:47:30.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.285+0000 7ff633e65640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:47:30.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.285+0000 7ff633e65640 1 -- 172.21.15.12:0/1035780994 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff62c0698e0 con 0x7ff62c09d030 2024-09-11T01:47:30.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.285+0000 7ff6227fc640 1 -- 172.21.15.12:0/1035780994 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7ff628032360 con 0x7ff62c09d030 2024-09-11T01:47:30.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.285+0000 7ff6227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:47:30.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.285+0000 7ff6227fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:47:30.287 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":172,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:47:30.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.287+0000 7ff633e65640 1 -- 172.21.15.12:0/1035780994 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff61003e260 msgr2=0x7ff610040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.287+0000 7ff633e65640 1 --2- 172.21.15.12:0/1035780994 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff61003e260 0x7ff610040720 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ff62c123880 tx=0x7ff61c02e000 comp rx=0 tx=0).stop 2024-09-11T01:47:30.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.287+0000 7ff633e65640 10 monclient: shutdown 2024-09-11T01:47:30.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.287+0000 7ff633e65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:47:30.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.287+0000 7ff633e65640 1 -- 172.21.15.12:0/1035780994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c09d030 msgr2=0x7ff62c137040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.287+0000 7ff633e65640 1 --2- 172.21.15.12:0/1035780994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c09d030 0x7ff62c137040 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff628008010 tx=0x7ff628004730 comp rx=0 tx=0).stop 2024-09-11T01:47:30.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.287+0000 7ff633e65640 1 -- 172.21.15.12:0/1035780994 shutdown_connections 2024-09-11T01:47:30.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.287+0000 7ff633e65640 1 --2- 172.21.15.12:0/1035780994 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff62c1cd0f0 0x7ff62c120c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.287+0000 7ff633e65640 1 --2- 172.21.15.12:0/1035780994 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff62c138230 0x7ff62c137580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.287+0000 7ff633e65640 1 --2- 172.21.15.12:0/1035780994 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff61003e260 0x7ff610040720 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.287+0000 7ff633e65640 1 --2- 172.21.15.12:0/1035780994 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff62c09d030 0x7ff62c137040 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.287+0000 7ff633e65640 1 -- 172.21.15.12:0/1035780994 >> 172.21.15.12:0/1035780994 conn(0x7ff62c0a7120 msgr2=0x7ff62c0a28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:47:30.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.287+0000 7ff633e65640 1 -- 172.21.15.12:0/1035780994 shutdown_connections 2024-09-11T01:47:30.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.287+0000 7ff633e65640 1 -- 172.21.15.12:0/1035780994 wait complete. 2024-09-11T01:47:30.299 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:47:30.299 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:47:30.300 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:47:30.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.399+0000 7fbfb7857640 10 monclient: get_monmap_and_config 2024-09-11T01:47:30.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.399+0000 7fbfb7857640 10 monclient: build_initial_monmap 2024-09-11T01:47:30.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.399+0000 7fbfb7857640 10 monclient: monmap: 2024-09-11T01:47:30.402 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:47:30.402 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.402 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:47:30.400635+0000 2024-09-11T01:47:30.402 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:47:30.400635+0000 2024-09-11T01:47:30.402 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:47:30.402 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.402 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:47:30.402 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:47:30.402 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:47:30.403 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:47:30.403 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:47:30.403 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.400+0000 7fbfb7857640 1 Processor -- start 2024-09-11T01:47:30.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.400+0000 7fbfb7857640 1 -- start start 2024-09-11T01:47:30.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.400+0000 7fbfb7857640 10 monclient: init 2024-09-11T01:47:30.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb7857640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:30.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb7857640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-11T01:47:30.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb7857640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbfb0137e50 0x7fbfb009d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb7857640 10 monclient(hunting): picked mon.noname-a con 0x7fbfb0137e50 addr v2:172.21.15.12:3300/0 2024-09-11T01:47:30.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb7857640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfb0138230 0x7fbfb0138660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb7857640 10 monclient(hunting): picked mon.noname-b con 0x7fbfb0138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:47:30.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb7857640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbfb0134cb0 addr v1:172.21.15.12:6789/0 2024-09-11T01:47:30.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb7857640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfb00a6580 con 0x7fbfb0134cb0 2024-09-11T01:47:30.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb7857640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb7857640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fbfb00a62d0 con 0x7fbfb0137e50 2024-09-11T01:47:30.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb7857640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb7857640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fbfb009bb40 con 0x7fbfb0138230 2024-09-11T01:47:30.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb7857640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:30.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb7857640 10 monclient(hunting): authenticate will time out at 1856.843268s 2024-09-11T01:47:30.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb55f3640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbfb0137e50 0x7fbfb009d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb5df4640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbfb0134cb0 0x7fbfb0135120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36308/0 (socket says 172.21.15.12:36308) 2024-09-11T01:47:30.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb5df4640 1 -- 172.21.15.12:0/710603206 learned_addr learned my addr 172.21.15.12:0/710603206 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:47:30.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb4df2640 1 --2- 172.21.15.12:0/710603206 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfb0138230 0x7fbfb0138660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb55f3640 10 monclient(hunting): get_auth_request con 0x7fbfb0137e50 auth_method 0 2024-09-11T01:47:30.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb55f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb55f3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb55f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb4df2640 10 monclient(hunting): get_auth_request con 0x7fbfb0138230 auth_method 0 2024-09-11T01:47:30.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb4df2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb4df2640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb4df2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb55f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb55f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb55f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:30.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb4df2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb4df2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb4df2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:30.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb4df2640 10 monclient(hunting): handle_auth_done global_id 184274 payload 274 2024-09-11T01:47:30.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb4df2640 1 -- 172.21.15.12:0/710603206 >> v2:172.21.15.12:3300/0 conn(0x7fbfb0137e50 msgr2=0x7fbfb009d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:47:30.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb4df2640 1 --2- 172.21.15.12:0/710603206 >> v2:172.21.15.12:3300/0 conn(0x7fbfb0137e50 0x7fbfb009d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb4df2640 1 -- 172.21.15.12:0/710603206 >> v1:172.21.15.12:6789/0 conn(0x7fbfb0134cb0 legacy=0x7fbfb0135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:47:30.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb4df2640 10 monclient: _finish_hunting 0 2024-09-11T01:47:30.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb4df2640 1 monclient: found mon.noname-b 2024-09-11T01:47:30.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb4df2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:30.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb4df2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:47:30.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb4df2640 1 -- 172.21.15.12:0/710603206 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfb009c4a0 con 0x7fbfb0138230 2024-09-11T01:47:30.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.401+0000 7fbfb55f3640 1 --2- 172.21.15.12:0/710603206 >> v2:172.21.15.12:3300/0 conn(0x7fbfb0137e50 0x7fbfb009d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:47:30.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.404+0000 7fbfb4df2640 1 -- 172.21.15.12:0/710603206 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfb0138230 msgr2=0x7fbfb0138660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:30.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.404+0000 7fbfb4df2640 1 --2- 172.21.15.12:0/710603206 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfb0138230 0x7fbfb0138660 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbfa00097a0 tx=0x7fbfa0029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3283e97f1a9e32d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:30.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.404+0000 7fbfa7fff640 1 -- 172.21.15.12:0/710603206 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0037070 con 0x7fbfb0138230 2024-09-11T01:47:30.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.404+0000 7fbfa7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:30.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.404+0000 7fbfa7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:47:30.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.404+0000 7fbfa7fff640 10 monclient: dump: 2024-09-11T01:47:30.416 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:30.416 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.416 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.416 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.416 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:30.416 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.416 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:30.416 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:30.416 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:30.416 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.404+0000 7fbfa7fff640 10 monclient: _finish_auth 0 2024-09-11T01:47:30.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.404+0000 7fbfa7fff640 10 monclient: _check_auth_tickets 2024-09-11T01:47:30.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.404+0000 7fbfa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:30.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.404+0000 7fbfa7fff640 1 -- 172.21.15.12:0/710603206 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfa0032440 con 0x7fbfb0138230 2024-09-11T01:47:30.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.404+0000 7fbfb7857640 5 monclient: authenticate success, global_id 184274 2024-09-11T01:47:30.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.404+0000 7fbfb7857640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:47:30.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.404+0000 7fbfa7fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:47:30.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.404+0000 7fbfa7fff640 1 -- 172.21.15.12:0/710603206 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0032810 con 0x7fbfb0138230 2024-09-11T01:47:30.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.404+0000 7fbfb7857640 10 monclient: get_monmap_and_config success 2024-09-11T01:47:30.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.404+0000 7fbfb7857640 4 set_mon_vals no callback set 2024-09-11T01:47:30.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.404+0000 7fbfa7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:30.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.404+0000 7fbfa7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:47:30.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.404+0000 7fbfa7fff640 10 monclient: dump: 2024-09-11T01:47:30.418 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:30.418 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.418 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.418 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.418 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:30.418 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.418 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:30.418 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:30.418 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:30.418 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.405+0000 7fbfb7857640 10 monclient: shutdown 2024-09-11T01:47:30.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.405+0000 7fbfb7857640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:47:30.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.405+0000 7fbfb7857640 1 -- 172.21.15.12:0/710603206 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfb0138230 msgr2=0x7fbfb0138660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.405+0000 7fbfb7857640 1 --2- 172.21.15.12:0/710603206 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfb0138230 0x7fbfb0138660 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbfa00097a0 tx=0x7fbfa0029fd0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.405+0000 7fbfb7857640 1 -- 172.21.15.12:0/710603206 shutdown_connections 2024-09-11T01:47:30.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.405+0000 7fbfb7857640 1 --2- 172.21.15.12:0/710603206 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfb0138230 0x7fbfb0138660 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.405+0000 7fbfb7857640 1 --2- 172.21.15.12:0/710603206 >> v2:172.21.15.12:3300/0 conn(0x7fbfb0137e50 0x7fbfb009d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.405+0000 7fbfb7857640 1 -- 172.21.15.12:0/710603206 >> 172.21.15.12:0/710603206 conn(0x7fbfb00a7020 msgr2=0x7fbfb00a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:47:30.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.405+0000 7fbfb7857640 1 -- 172.21.15.12:0/710603206 shutdown_connections 2024-09-11T01:47:30.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.405+0000 7fbfb7857640 1 -- 172.21.15.12:0/710603206 wait complete. 2024-09-11T01:47:30.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.405+0000 7fbfb7857640 10 monclient: build_initial_monmap 2024-09-11T01:47:30.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.405+0000 7fbfb7857640 10 monclient: monmap: 2024-09-11T01:47:30.420 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:47:30.420 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:47:30.420 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:47:30.420 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:47:30.420 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:47:30.420 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.420 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:47:30.420 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:47:30.420 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:47:30.420 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb7857640 1 Processor -- start 2024-09-11T01:47:30.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb7857640 1 -- start start 2024-09-11T01:47:30.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb7857640 10 monclient: init 2024-09-11T01:47:30.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb7857640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:30.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb7857640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:47:30.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb7857640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfb0134cb0 0x7fbfb01cd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb7857640 10 monclient(hunting): picked mon.noname-a con 0x7fbfb0134cb0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:47:30.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb7857640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfb0137e50 0x7fbfb0133e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb7857640 10 monclient(hunting): picked mon.noname-c con 0x7fbfb0137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:47:30.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb7857640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfb0138230 0x7fbfb0134370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb7857640 10 monclient(hunting): picked mon.noname-b con 0x7fbfb0138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:47:30.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb7857640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb7857640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb006f710 con 0x7fbfb0134cb0 2024-09-11T01:47:30.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb7857640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb7857640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb0138ba0 con 0x7fbfb0138230 2024-09-11T01:47:30.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb7857640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb7857640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fbfb006f890 con 0x7fbfb0137e50 2024-09-11T01:47:30.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb7857640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:30.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb7857640 10 monclient(hunting): authenticate will time out at 1856.848741s 2024-09-11T01:47:30.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb55f3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfb0134cb0 0x7fbfb01cd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb4df2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfb0137e50 0x7fbfb0133e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb4df2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfb0137e50 0x7fbfb0133e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:33044/0 (socket says 172.21.15.12:33044) 2024-09-11T01:47:30.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb4df2640 1 -- 172.21.15.12:0/1872241417 learned_addr learned my addr 172.21.15.12:0/1872241417 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:47:30.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb4df2640 10 monclient(hunting): get_auth_request con 0x7fbfb0137e50 auth_method 0 2024-09-11T01:47:30.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb4df2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb4df2640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb4df2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb55f3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfb0134cb0 0x7fbfb01cd5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38462/0 (socket says 172.21.15.12:38462) 2024-09-11T01:47:30.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb55f3640 10 monclient(hunting): get_auth_request con 0x7fbfb0134cb0 auth_method 0 2024-09-11T01:47:30.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb55f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb55f3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb55f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb55f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb55f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb55f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:30.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb4df2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb4df2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.406+0000 7fbfb4df2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:30.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.407+0000 7fbfb5df4640 1 --2- 172.21.15.12:0/1872241417 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfb0138230 0x7fbfb0134370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.407+0000 7fbfb5df4640 10 monclient(hunting): get_auth_request con 0x7fbfb0138230 auth_method 0 2024-09-11T01:47:30.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.407+0000 7fbfb5df4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.407+0000 7fbfb5df4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.407+0000 7fbfb5df4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.407+0000 7fbfb55f3640 10 monclient(hunting): handle_auth_done global_id 204108 payload 824 2024-09-11T01:47:30.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.407+0000 7fbfb55f3640 1 -- 172.21.15.12:0/1872241417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfb0137e50 msgr2=0x7fbfb0133e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.407+0000 7fbfb55f3640 1 --2- 172.21.15.12:0/1872241417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfb0137e50 0x7fbfb0133e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.407+0000 7fbfb55f3640 1 -- 172.21.15.12:0/1872241417 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfb0138230 msgr2=0x7fbfb0134370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.407+0000 7fbfb55f3640 1 --2- 172.21.15.12:0/1872241417 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfb0138230 0x7fbfb0134370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.407+0000 7fbfb55f3640 10 monclient: _finish_hunting 0 2024-09-11T01:47:30.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.407+0000 7fbfb55f3640 1 monclient: found mon.noname-a 2024-09-11T01:47:30.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.407+0000 7fbfb55f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:30.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.407+0000 7fbfb55f3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:47:30.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.407+0000 7fbfb55f3640 1 -- 172.21.15.12:0/1872241417 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfb0124fb0 con 0x7fbfb0134cb0 2024-09-11T01:47:30.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.407+0000 7fbfb4df2640 1 --2- 172.21.15.12:0/1872241417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfb0137e50 0x7fbfb0133e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:47:30.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.407+0000 7fbfb5df4640 1 --2- 172.21.15.12:0/1872241417 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfb0138230 0x7fbfb0134370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:47:30.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.409+0000 7fbfb55f3640 1 -- 172.21.15.12:0/1872241417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfb0134cb0 msgr2=0x7fbfb01cd5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:30.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.409+0000 7fbfb55f3640 1 --2- 172.21.15.12:0/1872241417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfb0134cb0 0x7fbfb01cd5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbfac00e900 tx=0x7fbfac00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:30.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.409+0000 7fbfa5ffb640 1 -- 172.21.15.12:0/1872241417 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfac017070 con 0x7fbfb0134cb0 2024-09-11T01:47:30.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.409+0000 7fbfa5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:30.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.409+0000 7fbfa5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:47:30.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.409+0000 7fbfa5ffb640 10 monclient: dump: 2024-09-11T01:47:30.426 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:30.426 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.426 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.426 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.426 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:30.426 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.426 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:30.426 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:30.426 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:30.426 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.409+0000 7fbfa5ffb640 10 monclient: _finish_auth 0 2024-09-11T01:47:30.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.409+0000 7fbfa5ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:47:30.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.409+0000 7fbfa5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:30.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.409+0000 7fbfa5ffb640 1 -- 172.21.15.12:0/1872241417 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfac00cc10 con 0x7fbfb0134cb0 2024-09-11T01:47:30.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.409+0000 7fbfa5ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:47:30.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.409+0000 7fbfa5ffb640 1 -- 172.21.15.12:0/1872241417 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfac013470 con 0x7fbfb0134cb0 2024-09-11T01:47:30.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.409+0000 7fbfb7857640 5 monclient: authenticate success, global_id 204108 2024-09-11T01:47:30.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.409+0000 7fbfa5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:30.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.409+0000 7fbfa5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:47:30.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.409+0000 7fbfa5ffb640 10 monclient: dump: 2024-09-11T01:47:30.427 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:30.427 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.427 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.427 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.427 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:30.427 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.427 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:30.428 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:30.428 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:30.428 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.409+0000 7fbfb7857640 10 monclient: _renew_subs 2024-09-11T01:47:30.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.409+0000 7fbfb7857640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:47:30.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.409+0000 7fbfb7857640 1 -- 172.21.15.12:0/1872241417 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfb01234f0 con 0x7fbfb0134cb0 2024-09-11T01:47:30.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.409+0000 7fbfa77fe640 4 set_mon_vals no callback set 2024-09-11T01:47:30.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.409+0000 7fbfb7857640 10 monclient: _renew_subs 2024-09-11T01:47:30.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.409+0000 7fbfb7857640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:47:30.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.409+0000 7fbfb7857640 1 -- 172.21.15.12:0/1872241417 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfb0125850 con 0x7fbfb0134cb0 2024-09-11T01:47:30.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.410+0000 7fbfb7857640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:30.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.410+0000 7fbfb7857640 1 --2- 172.21.15.12:0/1872241417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfb0121a30 0x7fbfb0121e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.410+0000 7fbfb7857640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.410+0000 7fbfb7857640 1 -- 172.21.15.12:0/1872241417 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb006a060 con 0x7fbfb0121a30 2024-09-11T01:47:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.410+0000 7fbfa5ffb640 1 -- 172.21.15.12:0/1872241417 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fbfac024060 con 0x7fbfb0134cb0 2024-09-11T01:47:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.410+0000 7fbfb4df2640 1 --2- 172.21.15.12:0/1872241417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfb0121a30 0x7fbfb0121e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.410+0000 7fbfb4df2640 10 monclient: get_auth_request con 0x7fbfb0121a30 auth_method 0 2024-09-11T01:47:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.410+0000 7fbfb4df2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.410+0000 7fbfb4df2640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.410+0000 7fbfb4df2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.410+0000 7fbfb4df2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.410+0000 7fbfb4df2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.410+0000 7fbfb4df2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.410+0000 7fbfb4df2640 10 monclient(hunting): handle_auth_done global_id 204111 payload 824 2024-09-11T01:47:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.410+0000 7fbfb4df2640 1 -- 172.21.15.12:0/1872241417 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbfb00a8f50 con 0x7fbfb0121a30 2024-09-11T01:47:30.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.411+0000 7fbfa5ffb640 1 --2- 172.21.15.12:0/1872241417 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbf9403e210 0x7fbf940406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.411+0000 7fbfa5ffb640 1 -- 172.21.15.12:0/1872241417 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(617..617 src has 507..617) ==== 4803+0+0 (secure 0 0 0) 0x7fbfac052c20 con 0x7fbfb0134cb0 2024-09-11T01:47:30.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.411+0000 7fbfb5df4640 1 --2- 172.21.15.12:0/1872241417 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbf9403e210 0x7fbf940406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.411+0000 7fbfb5df4640 10 monclient: get_auth_request con 0x7fbf9403e210 auth_method 0 2024-09-11T01:47:30.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.413+0000 7fbfb4df2640 1 -- 172.21.15.12:0/1872241417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfb0121a30 msgr2=0x7fbfb0121e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:30.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.413+0000 7fbfb4df2640 1 --2- 172.21.15.12:0/1872241417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfb0121a30 0x7fbfb0121e10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbfa00097a0 tx=0x7fbfa00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:30.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.413+0000 7fbfa5ffb640 1 -- 172.21.15.12:0/1872241417 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0043070 con 0x7fbfb0121a30 2024-09-11T01:47:30.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.413+0000 7fbfa5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:30.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.413+0000 7fbfa5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:47:30.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.413+0000 7fbfa5ffb640 10 monclient: dump: 2024-09-11T01:47:30.430 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:30.430 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.430 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.430 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.430 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:30.431 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.431 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:30.431 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:30.431 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:30.431 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.413+0000 7fbfa5ffb640 1 -- 172.21.15.12:0/1872241417 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbfa003ca60 con 0x7fbfb0121a30 2024-09-11T01:47:30.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.413+0000 7fbfa5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:30.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.413+0000 7fbfa5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:47:30.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.413+0000 7fbfa5ffb640 1 -- 172.21.15.12:0/1872241417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfb0121a30 msgr2=0x7fbfb0121e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.413+0000 7fbfa5ffb640 1 --2- 172.21.15.12:0/1872241417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfb0121a30 0x7fbfb0121e10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbfa00097a0 tx=0x7fbfa00046c0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.413+0000 7fbfa5ffb640 1 -- 172.21.15.12:0/1872241417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfb0121a30 msgr2=0x7fbfb0121e10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:47:30.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.413+0000 7fbfa5ffb640 1 --2- 172.21.15.12:0/1872241417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfb0121a30 0x7fbfb0121e10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.414+0000 7fbfb5df4640 1 --2- 172.21.15.12:0/1872241417 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbf9403e210 0x7fbf940406d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0123b10 tx=0x7fbfa8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:30.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.419+0000 7fbfb7857640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:47:30.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.419+0000 7fbfb7857640 1 --2- 172.21.15.12:0/1872241417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfb0122350 0x7fbfb00a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.419+0000 7fbfb7857640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.419+0000 7fbfb7857640 1 -- 172.21.15.12:0/1872241417 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb0125400 con 0x7fbfb0122350 2024-09-11T01:47:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.419+0000 7fbfb4df2640 1 --2- 172.21.15.12:0/1872241417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfb0122350 0x7fbfb00a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.420+0000 7fbfb4df2640 10 monclient: get_auth_request con 0x7fbfb0122350 auth_method 0 2024-09-11T01:47:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.420+0000 7fbfb4df2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.420+0000 7fbfb4df2640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.420+0000 7fbfb4df2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.420+0000 7fbfb4df2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.420+0000 7fbfb4df2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.420+0000 7fbfb4df2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.420+0000 7fbfb4df2640 10 monclient(hunting): handle_auth_done global_id 204114 payload 824 2024-09-11T01:47:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.420+0000 7fbfb4df2640 1 -- 172.21.15.12:0/1872241417 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbfb009d4b0 con 0x7fbfb0122350 2024-09-11T01:47:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.422+0000 7fbfb4df2640 1 -- 172.21.15.12:0/1872241417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfb0122350 msgr2=0x7fbfb00a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.422+0000 7fbfb4df2640 1 --2- 172.21.15.12:0/1872241417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfb0122350 0x7fbfb00a1920 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbfb00a9540 tx=0x7fbfa0009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:30.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.422+0000 7fbfa5ffb640 1 -- 172.21.15.12:0/1872241417 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0037040 con 0x7fbfb0122350 2024-09-11T01:47:30.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.422+0000 7fbfa5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:30.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.422+0000 7fbfa5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:47:30.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.422+0000 7fbfa5ffb640 10 monclient: dump: 2024-09-11T01:47:30.435 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:30.435 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.435 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.435 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.435 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:30.435 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.435 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:30.435 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:30.435 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:30.435 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.422+0000 7fbfa5ffb640 1 -- 172.21.15.12:0/1872241417 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7fbfa003b7b0 con 0x7fbfb0122350 2024-09-11T01:47:30.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.422+0000 7fbfa5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:47:30.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.422+0000 7fbfa5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:47:30.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.422+0000 7fbfa5ffb640 1 -- 172.21.15.12:0/1872241417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfb0122350 msgr2=0x7fbfb00a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.422+0000 7fbfa5ffb640 1 --2- 172.21.15.12:0/1872241417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfb0122350 0x7fbfb00a1920 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbfb00a9540 tx=0x7fbfa0009190 comp rx=0 tx=0).stop 2024-09-11T01:47:30.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.422+0000 7fbfa5ffb640 1 -- 172.21.15.12:0/1872241417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfb0122350 msgr2=0x7fbfb00a1920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:47:30.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.422+0000 7fbfa5ffb640 1 --2- 172.21.15.12:0/1872241417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfb0122350 0x7fbfb00a1920 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbfb00a9540 tx=0x7fbfa0009190 comp rx=0 tx=0).stop 2024-09-11T01:47:30.436 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:47:30.436 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:47:30.436 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:47:30.437 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:47:30.437 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 172, 2024-09-11T01:47:30.437 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 16047, 2024-09-11T01:47:30.437 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:47:30.437 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:47:30.437 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:47:30.437 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:47:30.437 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:30.437 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:47:30.437 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:47:30.437 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:47:30.437 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:47:30.437 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:47:30.437 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:47:30.438 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:47:30.438 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:47:30.438 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:47:30.438 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:47:30.438 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:47:30.438 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:47:30.438 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:47:30.438 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:47:30.438 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:47:30.439 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:30.439 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:47:30.439 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:47:30.439 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:47:30.439 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:47:30.439 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:47:30.439 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:47:30.439 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:47:30.439 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:47:30.440 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:47:30.440 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:47:30.440 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:47:30.440 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:47:30.440 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:47:30.440 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:30.440 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.440 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:47:30.440 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:47:30.440 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:47:30.440 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:47:30.441 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:47:30.441 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:47:30.441 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:47:30.441 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:47:30.441 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:47:30.441 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:47:30.441 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:47:30.441 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:47:30.441 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:47:30.441 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:47:30.441 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:47:30.441 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:47:30.441 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:47:30.442 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:47:30.442 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:47:30.442 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:47:30.442 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:47:30.442 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:47:30.442 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:47:30.442 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:47:30.442 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:47:30.442 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:47:30.442 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:47:30.442 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:47:30.442 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:30.442 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:47:30.442 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.442 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:47:30.443 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.443 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:47:30.443 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:47:30.443 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:47:30.443 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:47:30.443 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.443 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:47:30.443 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:47:30.443 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:30.443 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.443 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.443 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:47:30.443 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:47:30.444 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:30.444 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:30.444 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:30.444 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.444 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:47:30.444 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:47:30.444 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:47:30.444 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:47:30.444 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:47:30.444 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.444 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.444 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:47:30.445 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:47:30.445 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:47:30.445 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:47:30.445 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.445 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:47:30.445 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:47:30.445 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:30.445 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.445 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.445 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:47:30.445 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:47:30.445 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:30.446 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:30.446 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:30.446 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.446 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:47:30.446 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:47:30.446 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:47:30.446 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:47:30.446 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:47:30.446 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.446 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.446 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:47:30.446 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:47:30.446 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:47:30.447 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:47:30.447 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.447 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:47:30.447 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:47:30.447 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:30.447 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.447 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.447 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:47:30.447 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:47:30.447 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:30.447 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:30.447 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:30.447 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.447 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:47:30.447 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:47:30.448 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:47:30.448 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:47:30.448 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:47:30.448 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:30.448 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:30.448 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.448 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:47:30.448 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:47:30.448 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.448 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:47:30.448 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:47:30.448 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:47:30.448 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:30.449 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:30.449 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:47:30.449 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.449 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:47:30.449 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:47:30.449 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:47:30.449 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:30.449 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:30.449 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.449 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:47:30.450 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:47:30.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.423+0000 7fbfb7857640 1 -- 172.21.15.12:0/1872241417 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbf9403e210 msgr2=0x7fbf940406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.423+0000 7fbfb7857640 1 --2- 172.21.15.12:0/1872241417 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbf9403e210 0x7fbf940406d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0123b10 tx=0x7fbfa8009040 comp rx=0 tx=0).stop 2024-09-11T01:47:30.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.423+0000 7fbfb7857640 10 monclient: shutdown 2024-09-11T01:47:30.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.423+0000 7fbfb7857640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:47:30.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.423+0000 7fbfb7857640 1 -- 172.21.15.12:0/1872241417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfb0134cb0 msgr2=0x7fbfb01cd5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.423+0000 7fbfb7857640 1 --2- 172.21.15.12:0/1872241417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfb0134cb0 0x7fbfb01cd5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbfac00e900 tx=0x7fbfac00edc0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.423+0000 7fbfb5df4640 1 -- 172.21.15.12:0/1872241417 reap_dead start 2024-09-11T01:47:30.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.423+0000 7fbfb5df4640 1 -- 172.21.15.12:0/1872241417 reap_dead start 2024-09-11T01:47:30.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.423+0000 7fbfb7857640 1 -- 172.21.15.12:0/1872241417 shutdown_connections 2024-09-11T01:47:30.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.423+0000 7fbfb7857640 1 -- 172.21.15.12:0/1872241417 >> 172.21.15.12:0/1872241417 conn(0x7fbfb00a7020 msgr2=0x7fbfb0137690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:47:30.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.423+0000 7fbfb7857640 1 -- 172.21.15.12:0/1872241417 shutdown_connections 2024-09-11T01:47:30.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.423+0000 7fbfb7857640 1 -- 172.21.15.12:0/1872241417 wait complete. 2024-09-11T01:47:30.452 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:47:30.452 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:47:30.452 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:47:30.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.553+0000 7f2ce1641640 10 monclient: get_monmap_and_config 2024-09-11T01:47:30.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.553+0000 7f2ce1641640 10 monclient: build_initial_monmap 2024-09-11T01:47:30.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.553+0000 7f2ce1641640 10 monclient: monmap: 2024-09-11T01:47:30.555 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:47:30.555 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.555 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:47:30.554369+0000 2024-09-11T01:47:30.555 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:47:30.554369+0000 2024-09-11T01:47:30.555 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:47:30.556 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.556 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:47:30.556 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:47:30.556 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:47:30.556 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:47:30.556 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:47:30.556 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.554+0000 7f2ce1641640 1 Processor -- start 2024-09-11T01:47:30.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.554+0000 7f2ce1641640 1 -- start start 2024-09-11T01:47:30.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.554+0000 7f2ce1641640 10 monclient: init 2024-09-11T01:47:30.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.554+0000 7f2ce1641640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:30.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.554+0000 7f2ce1641640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-11T01:47:30.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.554+0000 7f2ce1641640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2cdc0a2e20 addr v1:172.21.15.12:6789/0 2024-09-11T01:47:30.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.554+0000 7f2ce1641640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2cdc0a4970 0x7f2cdc06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.554+0000 7f2ce1641640 10 monclient(hunting): picked mon.noname-c con 0x7f2cdc0a4970 addr v2:172.21.15.71:3300/0 2024-09-11T01:47:30.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.554+0000 7f2ce1641640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2cdc06f910 0x7f2cdc13f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.554+0000 7f2ce1641640 10 monclient(hunting): picked mon.noname-a con 0x7f2cdc06f910 addr v2:172.21.15.12:3300/0 2024-09-11T01:47:30.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.554+0000 7f2ce1641640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cdc06f4f0 con 0x7f2cdc0a2e20 2024-09-11T01:47:30.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.554+0000 7f2ce1641640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.554+0000 7f2ce1641640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f2cdc06ef30 con 0x7f2cdc06f910 2024-09-11T01:47:30.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.554+0000 7f2ce1641640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.554+0000 7f2ce1641640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f2cdc06ec50 con 0x7f2cdc0a4970 2024-09-11T01:47:30.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.554+0000 7f2ce1641640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:30.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.554+0000 7f2ce1641640 10 monclient(hunting): authenticate will time out at 1856.996948s 2024-09-11T01:47:30.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.555+0000 7f2cdb7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2cdc06f910 0x7f2cdc13f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.555+0000 7f2cdaffd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2cdc0a2e20 0x7f2cdc0a3200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36320/0 (socket says 172.21.15.12:36320) 2024-09-11T01:47:30.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.555+0000 7f2cdb7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2cdc06f910 0x7f2cdc13f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38488/0 (socket says 172.21.15.12:38488) 2024-09-11T01:47:30.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.555+0000 7f2cdaffd640 1 -- 172.21.15.12:0/4176665052 learned_addr learned my addr 172.21.15.12:0/4176665052 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:47:30.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.555+0000 7f2cdb7fe640 10 monclient(hunting): get_auth_request con 0x7f2cdc06f910 auth_method 0 2024-09-11T01:47:30.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.555+0000 7f2cdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.555+0000 7f2cdb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.555+0000 7f2cdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.555+0000 7f2cdb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.555+0000 7f2cdb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.555+0000 7f2cdb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:30.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.555+0000 7f2cd3fff640 1 --2- 172.21.15.12:0/4176665052 >> v2:172.21.15.71:3300/0 conn(0x7f2cdc0a4970 0x7f2cdc06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.555+0000 7f2cd3fff640 10 monclient(hunting): get_auth_request con 0x7f2cdc0a4970 auth_method 0 2024-09-11T01:47:30.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.555+0000 7f2cd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.555+0000 7f2cd3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.555+0000 7f2cd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.555+0000 7f2cdb7fe640 10 monclient(hunting): handle_auth_done global_id 204117 payload 274 2024-09-11T01:47:30.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.555+0000 7f2cdb7fe640 1 -- 172.21.15.12:0/4176665052 >> v2:172.21.15.71:3300/0 conn(0x7f2cdc0a4970 msgr2=0x7f2cdc06ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:47:30.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.555+0000 7f2cdb7fe640 1 --2- 172.21.15.12:0/4176665052 >> v2:172.21.15.71:3300/0 conn(0x7f2cdc0a4970 0x7f2cdc06ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.555+0000 7f2cdb7fe640 1 -- 172.21.15.12:0/4176665052 >> v1:172.21.15.12:6789/0 conn(0x7f2cdc0a2e20 legacy=0x7f2cdc0a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:47:30.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.555+0000 7f2cdb7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:47:30.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.555+0000 7f2cdb7fe640 1 monclient: found mon.noname-a 2024-09-11T01:47:30.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.555+0000 7f2cdb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:30.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.555+0000 7f2cdb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:47:30.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.555+0000 7f2cdb7fe640 1 -- 172.21.15.12:0/4176665052 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cdc0a3910 con 0x7f2cdc06f910 2024-09-11T01:47:30.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.555+0000 7f2cd3fff640 1 --2- 172.21.15.12:0/4176665052 >> v2:172.21.15.71:3300/0 conn(0x7f2cdc0a4970 0x7f2cdc06ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:47:30.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2cdb7fe640 1 -- 172.21.15.12:0/4176665052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cdc06f910 msgr2=0x7f2cdc13f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:30.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2cdb7fe640 1 --2- 172.21.15.12:0/4176665052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cdc06f910 0x7f2cdc13f430 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc074f40 tx=0x7f2cd402a4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39bb8e38a9ff05ff server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:30.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2cda7fc640 1 -- 172.21.15.12:0/4176665052 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd4007950 con 0x7f2cdc06f910 2024-09-11T01:47:30.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2cda7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:30.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2cda7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:47:30.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2cda7fc640 10 monclient: dump: 2024-09-11T01:47:30.569 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:30.569 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.569 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.570 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.570 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:30.570 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.570 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:30.570 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:30.570 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:30.570 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2cda7fc640 10 monclient: _finish_auth 0 2024-09-11T01:47:30.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2cda7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:47:30.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2cda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:30.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2cda7fc640 1 -- 172.21.15.12:0/4176665052 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cd4007af0 con 0x7f2cdc06f910 2024-09-11T01:47:30.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2ce1641640 5 monclient: authenticate success, global_id 204117 2024-09-11T01:47:30.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2ce1641640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:47:30.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2cda7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:47:30.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2ce1641640 10 monclient: get_monmap_and_config success 2024-09-11T01:47:30.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2cda7fc640 1 -- 172.21.15.12:0/4176665052 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd4031400 con 0x7f2cdc06f910 2024-09-11T01:47:30.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2cda7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:30.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2ce1641640 4 set_mon_vals no callback set 2024-09-11T01:47:30.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2cda7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:47:30.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2cda7fc640 10 monclient: dump: 2024-09-11T01:47:30.571 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:30.571 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.571 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.572 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.572 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:30.572 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.572 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:30.572 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:30.572 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:30.572 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2ce1641640 10 monclient: shutdown 2024-09-11T01:47:30.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2ce1641640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:47:30.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2ce1641640 1 -- 172.21.15.12:0/4176665052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cdc06f910 msgr2=0x7f2cdc13f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2ce1641640 1 --2- 172.21.15.12:0/4176665052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cdc06f910 0x7f2cdc13f430 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc074f40 tx=0x7f2cd402a4b0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2ce1641640 1 -- 172.21.15.12:0/4176665052 shutdown_connections 2024-09-11T01:47:30.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2ce1641640 1 --2- 172.21.15.12:0/4176665052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cdc06f910 0x7f2cdc13f430 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2ce1641640 1 --2- 172.21.15.12:0/4176665052 >> v2:172.21.15.71:3300/0 conn(0x7f2cdc0a4970 0x7f2cdc06ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2ce1641640 1 -- 172.21.15.12:0/4176665052 >> 172.21.15.12:0/4176665052 conn(0x7f2cdc0a8600 msgr2=0x7f2cdc0a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:47:30.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2ce1641640 1 -- 172.21.15.12:0/4176665052 shutdown_connections 2024-09-11T01:47:30.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.558+0000 7f2ce1641640 1 -- 172.21.15.12:0/4176665052 wait complete. 2024-09-11T01:47:30.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.559+0000 7f2ce1641640 10 monclient: build_initial_monmap 2024-09-11T01:47:30.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.559+0000 7f2ce1641640 10 monclient: monmap: 2024-09-11T01:47:30.573 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:47:30.573 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:47:30.573 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:47:30.573 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:47:30.574 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:47:30.574 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.574 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:47:30.574 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:47:30.574 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:47:30.574 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.559+0000 7f2ce1641640 1 Processor -- start 2024-09-11T01:47:30.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.559+0000 7f2ce1641640 1 -- start start 2024-09-11T01:47:30.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.559+0000 7f2ce1641640 10 monclient: init 2024-09-11T01:47:30.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2ce1641640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:30.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2ce1641640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:47:30.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2ce1641640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2cdc06f910 0x7f2cdc13e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2ce1641640 10 monclient(hunting): picked mon.noname-c con 0x7f2cdc06f910 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:47:30.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2ce1641640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2cdc0a2e20 0x7f2cdc13eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2ce1641640 10 monclient(hunting): picked mon.noname-b con 0x7f2cdc0a2e20 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:47:30.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2ce1641640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cdc0a4970 0x7f2cdc139400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2ce1641640 10 monclient(hunting): picked mon.noname-a con 0x7f2cdc0a4970 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:47:30.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2ce1641640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2ce1641640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2cdc13fd00 con 0x7f2cdc0a4970 2024-09-11T01:47:30.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2ce1641640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2ce1641640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f2cdc13fb80 con 0x7f2cdc0a2e20 2024-09-11T01:47:30.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2ce1641640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2ce1641640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2cdc13fe80 con 0x7f2cdc06f910 2024-09-11T01:47:30.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2ce1641640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:30.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2ce1641640 10 monclient(hunting): authenticate will time out at 1857.002364s 2024-09-11T01:47:30.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdaffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2cdc06f910 0x7f2cdc13e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdb7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cdc0a4970 0x7f2cdc139400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdb7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cdc0a4970 0x7f2cdc139400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38490/0 (socket says 172.21.15.12:38490) 2024-09-11T01:47:30.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdaffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2cdc06f910 0x7f2cdc13e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:33054/0 (socket says 172.21.15.12:33054) 2024-09-11T01:47:30.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdb7fe640 1 -- 172.21.15.12:0/2684207119 learned_addr learned my addr 172.21.15.12:0/2684207119 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:47:30.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdb7fe640 10 monclient(hunting): get_auth_request con 0x7f2cdc0a4970 auth_method 0 2024-09-11T01:47:30.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdaffd640 10 monclient(hunting): get_auth_request con 0x7f2cdc06f910 auth_method 0 2024-09-11T01:47:30.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:30.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:30.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cd3fff640 1 --2- 172.21.15.12:0/2684207119 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2cdc0a2e20 0x7f2cdc13eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cd3fff640 10 monclient(hunting): get_auth_request con 0x7f2cdc0a2e20 auth_method 0 2024-09-11T01:47:30.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cd3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:30.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdb7fe640 10 monclient(hunting): handle_auth_done global_id 204120 payload 824 2024-09-11T01:47:30.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdb7fe640 1 -- 172.21.15.12:0/2684207119 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2cdc06f910 msgr2=0x7f2cdc13e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdb7fe640 1 --2- 172.21.15.12:0/2684207119 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2cdc06f910 0x7f2cdc13e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdb7fe640 1 -- 172.21.15.12:0/2684207119 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2cdc0a2e20 msgr2=0x7f2cdc13eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdb7fe640 1 --2- 172.21.15.12:0/2684207119 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2cdc0a2e20 0x7f2cdc13eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdb7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:47:30.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdb7fe640 1 monclient: found mon.noname-a 2024-09-11T01:47:30.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:30.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:47:30.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdb7fe640 1 -- 172.21.15.12:0/2684207119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cdc09eea0 con 0x7f2cdc0a4970 2024-09-11T01:47:30.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.560+0000 7f2cdaffd640 1 --2- 172.21.15.12:0/2684207119 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2cdc06f910 0x7f2cdc13e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:47:30.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.563+0000 7f2cdb7fe640 1 -- 172.21.15.12:0/2684207119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cdc0a4970 msgr2=0x7f2cdc139400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:30.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.563+0000 7f2cdb7fe640 1 --2- 172.21.15.12:0/2684207119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cdc0a4970 0x7f2cdc139400 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2cd402aaa0 tx=0x7f2cd4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:30.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.563+0000 7f2cd37fe640 1 -- 172.21.15.12:0/2684207119 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd4040070 con 0x7f2cdc0a4970 2024-09-11T01:47:30.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.563+0000 7f2cd37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:30.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.563+0000 7f2cd37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:47:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.563+0000 7f2cd37fe640 10 monclient: dump: 2024-09-11T01:47:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:30.580 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:30.581 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.563+0000 7f2cd37fe640 10 monclient: _finish_auth 0 2024-09-11T01:47:30.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.563+0000 7f2cd37fe640 10 monclient: _check_auth_tickets 2024-09-11T01:47:30.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.563+0000 7f2cd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:30.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.563+0000 7f2cd37fe640 1 -- 172.21.15.12:0/2684207119 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cd4045660 con 0x7f2cdc0a4970 2024-09-11T01:47:30.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.563+0000 7f2cd37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:47:30.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.563+0000 7f2cd37fe640 1 -- 172.21.15.12:0/2684207119 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd4045a30 con 0x7f2cdc0a4970 2024-09-11T01:47:30.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.563+0000 7f2ce1641640 5 monclient: authenticate success, global_id 204120 2024-09-11T01:47:30.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.563+0000 7f2ce1641640 10 monclient: _renew_subs 2024-09-11T01:47:30.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.563+0000 7f2ce1641640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:47:30.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.563+0000 7f2ce1641640 1 -- 172.21.15.12:0/2684207119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cdc09f610 con 0x7f2cdc0a4970 2024-09-11T01:47:30.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.563+0000 7f2cd37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:30.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.563+0000 7f2cd37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:47:30.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.563+0000 7f2cd37fe640 10 monclient: dump: 2024-09-11T01:47:30.582 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:30.582 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.582 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.582 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.582 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:30.582 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.582 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:30.582 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:30.582 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:30.582 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.563+0000 7f2cd9ffb640 4 set_mon_vals no callback set 2024-09-11T01:47:30.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.563+0000 7f2ce1641640 10 monclient: _renew_subs 2024-09-11T01:47:30.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.563+0000 7f2ce1641640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:47:30.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.563+0000 7f2ce1641640 1 -- 172.21.15.12:0/2684207119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cdc13c0d0 con 0x7f2cdc0a4970 2024-09-11T01:47:30.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.564+0000 7f2ce1641640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:30.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.564+0000 7f2ce1641640 1 --2- 172.21.15.12:0/2684207119 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2cdc02d440 0x7f2cdc02d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.564+0000 7f2cdb7fe640 1 --2- 172.21.15.12:0/2684207119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cdc0a4970 0x7f2cdc139400 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2cd402aaa0 tx=0x7f2cd4004a40 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7f2cd4074e90 osd_map(617..617 src has 507..617) 2024-09-11T01:47:30.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.564+0000 7f2ce1641640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.564+0000 7f2ce1641640 1 -- 172.21.15.12:0/2684207119 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2cdc06a060 con 0x7f2cdc02d440 2024-09-11T01:47:30.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.564+0000 7f2cd37fe640 1 -- 172.21.15.12:0/2684207119 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f2cd4007960 con 0x7f2cdc0a4970 2024-09-11T01:47:30.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.564+0000 7f2cdaffd640 1 --2- 172.21.15.12:0/2684207119 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2cdc02d440 0x7f2cdc02d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.564+0000 7f2cdaffd640 10 monclient: get_auth_request con 0x7f2cdc02d440 auth_method 0 2024-09-11T01:47:30.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.564+0000 7f2cdaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.564+0000 7f2cdaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.564+0000 7f2cdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.564+0000 7f2cdaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.564+0000 7f2cdaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.564+0000 7f2cdaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.564+0000 7f2cd37fe640 1 --2- 172.21.15.12:0/2684207119 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2cbc03e210 0x7f2cbc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.564+0000 7f2cdaffd640 10 monclient(hunting): handle_auth_done global_id 184283 payload 824 2024-09-11T01:47:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.564+0000 7f2cdaffd640 1 -- 172.21.15.12:0/2684207119 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2cdc13f740 con 0x7f2cdc02d440 2024-09-11T01:47:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.564+0000 7f2cd3fff640 1 --2- 172.21.15.12:0/2684207119 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2cbc03e210 0x7f2cbc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.564+0000 7f2cd3fff640 10 monclient: get_auth_request con 0x7f2cbc03e210 auth_method 0 2024-09-11T01:47:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.566+0000 7f2cdaffd640 1 -- 172.21.15.12:0/2684207119 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2cdc02d440 msgr2=0x7f2cdc02d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.566+0000 7f2cdaffd640 1 --2- 172.21.15.12:0/2684207119 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2cdc02d440 0x7f2cdc02d820 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc007fd0 tx=0x7f2ccc007700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.566+0000 7f2cd37fe640 1 -- 172.21.15.12:0/2684207119 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc017030 con 0x7f2cdc02d440 2024-09-11T01:47:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.566+0000 7f2cd37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.566+0000 7f2cd37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:47:30.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.566+0000 7f2cd37fe640 10 monclient: dump: 2024-09-11T01:47:30.585 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:30.585 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.585 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.585 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.585 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:30.585 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.585 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:30.585 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:30.585 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:30.585 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.567+0000 7f2cd37fe640 1 -- 172.21.15.12:0/2684207119 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2ccc00fc50 con 0x7f2cdc02d440 2024-09-11T01:47:30.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.567+0000 7f2cd37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:30.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.567+0000 7f2cd37fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:47:30.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.567+0000 7f2cd37fe640 1 -- 172.21.15.12:0/2684207119 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2cdc02d440 msgr2=0x7f2cdc02d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.567+0000 7f2cd37fe640 1 --2- 172.21.15.12:0/2684207119 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2cdc02d440 0x7f2cdc02d820 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc007fd0 tx=0x7f2ccc007700 comp rx=0 tx=0).stop 2024-09-11T01:47:30.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.567+0000 7f2cd37fe640 1 -- 172.21.15.12:0/2684207119 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2cdc02d440 msgr2=0x7f2cdc02d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:47:30.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.567+0000 7f2cd37fe640 1 --2- 172.21.15.12:0/2684207119 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2cdc02d440 0x7f2cdc02d820 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc007fd0 tx=0x7f2ccc007700 comp rx=0 tx=0).stop 2024-09-11T01:47:30.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.567+0000 7f2cd3fff640 1 --2- 172.21.15.12:0/2684207119 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2cbc03e210 0x7f2cbc0406d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc13aeb0 tx=0x7f2cc4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:30.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.573+0000 7f2ce1641640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:47:30.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.573+0000 7f2ce1641640 1 --2- 172.21.15.12:0/2684207119 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2cdc02dd60 0x7f2cdc02e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.573+0000 7f2ce1641640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.573+0000 7f2ce1641640 1 -- 172.21.15.12:0/2684207119 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2cdc06ded0 con 0x7f2cdc02dd60 2024-09-11T01:47:30.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.573+0000 7f2cdaffd640 1 --2- 172.21.15.12:0/2684207119 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2cdc02dd60 0x7f2cdc02e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.573+0000 7f2cdaffd640 10 monclient: get_auth_request con 0x7f2cdc02dd60 auth_method 0 2024-09-11T01:47:30.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.573+0000 7f2cdaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.573+0000 7f2cdaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.573+0000 7f2cdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.573+0000 7f2cdaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.573+0000 7f2cdaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.573+0000 7f2cdaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:30.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.573+0000 7f2cdaffd640 10 monclient(hunting): handle_auth_done global_id 184286 payload 824 2024-09-11T01:47:30.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.573+0000 7f2cdaffd640 1 -- 172.21.15.12:0/2684207119 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2cdc1378d0 con 0x7f2cdc02dd60 2024-09-11T01:47:30.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.575+0000 7f2cdaffd640 1 -- 172.21.15.12:0/2684207119 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2cdc02dd60 msgr2=0x7f2cdc02e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:30.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.575+0000 7f2cdaffd640 1 --2- 172.21.15.12:0/2684207119 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2cdc02dd60 0x7f2cdc02e140 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc007fd0 tx=0x7f2ccc004940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:30.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.576+0000 7f2cd37fe640 1 -- 172.21.15.12:0/2684207119 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc006000 con 0x7f2cdc02dd60 2024-09-11T01:47:30.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.576+0000 7f2cd37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:30.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.576+0000 7f2cd37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:47:30.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.576+0000 7f2cd37fe640 10 monclient: dump: 2024-09-11T01:47:30.589 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:30.589 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.589 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.589 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.592 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:30.592 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.592 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:30.592 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:30.592 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:30.592 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.576+0000 7f2cd37fe640 1 -- 172.21.15.12:0/2684207119 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4542 (secure 0 0 0) 0x7f2ccc006000 con 0x7f2cdc02dd60 2024-09-11T01:47:30.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.576+0000 7f2cd37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:47:30.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.576+0000 7f2cd37fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:47:30.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.576+0000 7f2cd37fe640 1 -- 172.21.15.12:0/2684207119 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2cdc02dd60 msgr2=0x7f2cdc02e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.576+0000 7f2cd37fe640 1 --2- 172.21.15.12:0/2684207119 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2cdc02dd60 0x7f2cdc02e140 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc007fd0 tx=0x7f2ccc004940 comp rx=0 tx=0).stop 2024-09-11T01:47:30.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.576+0000 7f2cd37fe640 1 -- 172.21.15.12:0/2684207119 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2cdc02dd60 msgr2=0x7f2cdc02e140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:47:30.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.576+0000 7f2cd37fe640 1 --2- 172.21.15.12:0/2684207119 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2cdc02dd60 0x7f2cdc02e140 secure :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc007fd0 tx=0x7f2ccc004940 comp rx=0 tx=0).stop 2024-09-11T01:47:30.593 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:47:30.593 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:47:30.593 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:47:30.593 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:47:30.593 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 172, 2024-09-11T01:47:30.593 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 111169, 2024-09-11T01:47:30.593 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:47:30.593 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:47:30.594 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:47:30.594 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:47:30.594 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:30.594 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:47:30.594 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:47:30.594 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:47:30.594 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:47:30.594 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:47:30.594 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:47:30.594 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:47:30.594 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:47:30.594 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:47:30.594 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:47:30.594 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:47:30.595 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:47:30.595 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:47:30.595 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:47:30.595 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:47:30.595 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:30.595 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:47:30.595 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:47:30.595 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:47:30.595 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:47:30.595 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:47:30.595 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:47:30.595 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:47:30.595 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:47:30.596 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:47:30.596 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:47:30.596 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:47:30.596 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:47:30.596 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:47:30.596 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:30.596 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.596 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:47:30.596 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:47:30.596 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:47:30.596 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:47:30.596 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:47:30.596 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:47:30.596 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:47:30.597 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:47:30.597 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:47:30.597 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:47:30.597 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:47:30.597 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:47:30.597 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:47:30.597 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:47:30.597 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:47:30.597 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:47:30.597 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:47:30.597 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:47:30.597 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:47:30.597 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:47:30.598 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:47:30.598 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:47:30.598 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:47:30.598 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:47:30.598 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:47:30.598 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:47:30.598 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:47:30.598 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:47:30.598 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:30.598 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:47:30.598 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.598 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:47:30.598 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.599 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:47:30.599 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:47:30.599 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:47:30.599 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:47:30.599 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.599 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:47:30.599 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:47:30.599 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:30.599 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.599 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.599 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:47:30.599 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:47:30.599 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:30.599 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:30.600 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:30.600 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.600 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:47:30.600 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:47:30.600 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:47:30.600 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:47:30.600 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:47:30.600 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.600 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.600 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:47:30.600 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:47:30.600 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:47:30.600 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:47:30.600 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.601 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:47:30.601 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:47:30.601 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:30.601 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.601 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.601 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:47:30.601 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:47:30.601 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:30.601 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:30.601 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:30.601 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.601 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:47:30.601 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:47:30.601 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:47:30.602 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:47:30.602 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:47:30.602 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.602 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.602 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:47:30.602 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:47:30.602 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:47:30.602 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:47:30.602 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.602 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:47:30.602 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:47:30.602 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:30.602 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.602 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.602 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:47:30.603 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:47:30.603 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:30.603 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:30.603 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:30.603 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.603 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:47:30.603 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:47:30.603 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:47:30.603 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:47:30.603 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:47:30.603 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:30.603 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:30.603 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.603 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:47:30.603 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:47:30.604 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.604 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:47:30.604 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:47:30.604 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:47:30.604 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:30.604 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:30.604 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:47:30.604 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.604 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:47:30.604 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:47:30.604 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:47:30.604 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:30.604 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:30.604 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.605 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:47:30.605 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:47:30.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.576+0000 7f2ce1641640 1 -- 172.21.15.12:0/2684207119 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2cbc03e210 msgr2=0x7f2cbc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.576+0000 7f2ce1641640 1 --2- 172.21.15.12:0/2684207119 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2cbc03e210 0x7f2cbc0406d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc13aeb0 tx=0x7f2cc4008040 comp rx=0 tx=0).stop 2024-09-11T01:47:30.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.576+0000 7f2ce1641640 10 monclient: shutdown 2024-09-11T01:47:30.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.576+0000 7f2ce1641640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:47:30.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.576+0000 7f2ce1641640 1 -- 172.21.15.12:0/2684207119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cdc0a4970 msgr2=0x7f2cdc139400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.576+0000 7f2ce1641640 1 --2- 172.21.15.12:0/2684207119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cdc0a4970 0x7f2cdc139400 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2cd402aaa0 tx=0x7f2cd4004a40 comp rx=0 tx=0).stop 2024-09-11T01:47:30.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.576+0000 7f2cdb7fe640 1 -- 172.21.15.12:0/2684207119 reap_dead start 2024-09-11T01:47:30.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.576+0000 7f2cdb7fe640 1 -- 172.21.15.12:0/2684207119 reap_dead start 2024-09-11T01:47:30.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.576+0000 7f2cd37fe640 1 -- 172.21.15.12:0/2684207119 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(617..617 src has 507..617) ==== 4803+0+0 (unknown 0 0 0) 0x7f2cd4074e90 con 0x7f2cdc0a4970 2024-09-11T01:47:30.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.576+0000 7f2ce1641640 1 -- 172.21.15.12:0/2684207119 shutdown_connections 2024-09-11T01:47:30.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.576+0000 7f2ce1641640 1 -- 172.21.15.12:0/2684207119 >> 172.21.15.12:0/2684207119 conn(0x7f2cdc0a8600 msgr2=0x7f2cdc0a1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:47:30.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.576+0000 7f2ce1641640 1 -- 172.21.15.12:0/2684207119 shutdown_connections 2024-09-11T01:47:30.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.576+0000 7f2ce1641640 1 -- 172.21.15.12:0/2684207119 wait complete. 2024-09-11T01:47:30.607 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:47:30.607 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:47:30.608 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:47:30.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.708+0000 7f05e15a0640 10 monclient: get_monmap_and_config 2024-09-11T01:47:30.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.708+0000 7f05e15a0640 10 monclient: build_initial_monmap 2024-09-11T01:47:30.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.708+0000 7f05e15a0640 10 monclient: monmap: 2024-09-11T01:47:30.710 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:47:30.710 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.710 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:47:30.708969+0000 2024-09-11T01:47:30.710 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:47:30.708969+0000 2024-09-11T01:47:30.710 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:47:30.710 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.710 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:47:30.710 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:47:30.710 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:47:30.711 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:47:30.711 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:47:30.711 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.708+0000 7f05e15a0640 1 Processor -- start 2024-09-11T01:47:30.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.708+0000 7f05e15a0640 1 -- start start 2024-09-11T01:47:30.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.708+0000 7f05e15a0640 10 monclient: init 2024-09-11T01:47:30.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05e15a0640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:30.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05e15a0640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-11T01:47:30.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05e15a0640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f05dc137e50 0x7f05dc09d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05e15a0640 10 monclient(hunting): picked mon.noname-c con 0x7f05dc137e50 addr v2:172.21.15.71:3300/0 2024-09-11T01:47:30.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05e15a0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f05dc138230 addr v1:172.21.15.71:6789/0 2024-09-11T01:47:30.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05e15a0640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f05dc13d380 0x7f05dc13f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05e15a0640 10 monclient(hunting): picked mon.noname-a con 0x7f05dc13d380 addr v2:172.21.15.12:3300/0 2024-09-11T01:47:30.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05e15a0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05dc0a62d0 con 0x7f05dc138230 2024-09-11T01:47:30.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05e15a0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05e15a0640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f05dc09bb40 con 0x7f05dc13d380 2024-09-11T01:47:30.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05e15a0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05e15a0640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f05dc09c4a0 con 0x7f05dc137e50 2024-09-11T01:47:30.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05e15a0640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:30.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05e15a0640 10 monclient(hunting): authenticate will time out at 1857.151413s 2024-09-11T01:47:30.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05db7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f05dc13d380 0x7f05dc13f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05db7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f05dc13d380 0x7f05dc13f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38494/0 (socket says 172.21.15.12:38494) 2024-09-11T01:47:30.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05db7fe640 1 -- 172.21.15.12:0/312719405 learned_addr learned my addr 172.21.15.12:0/312719405 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:47:30.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05db7fe640 10 monclient(hunting): get_auth_request con 0x7f05dc13d380 auth_method 0 2024-09-11T01:47:30.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:30.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05daffd640 1 --2- 172.21.15.12:0/312719405 >> v2:172.21.15.71:3300/0 conn(0x7f05dc137e50 0x7f05dc09d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05daffd640 10 monclient(hunting): get_auth_request con 0x7f05dc137e50 auth_method 0 2024-09-11T01:47:30.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05daffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05db7fe640 10 monclient(hunting): handle_auth_done global_id 204123 payload 274 2024-09-11T01:47:30.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05db7fe640 1 -- 172.21.15.12:0/312719405 >> v2:172.21.15.71:3300/0 conn(0x7f05dc137e50 msgr2=0x7f05dc09d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:47:30.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05db7fe640 1 --2- 172.21.15.12:0/312719405 >> v2:172.21.15.71:3300/0 conn(0x7f05dc137e50 0x7f05dc09d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05db7fe640 1 -- 172.21.15.12:0/312719405 >> v1:172.21.15.71:6789/0 conn(0x7f05dc138230 legacy=0x7f05dc138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:47:30.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05db7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:47:30.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05db7fe640 1 monclient: found mon.noname-a 2024-09-11T01:47:30.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:30.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05db7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:47:30.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.709+0000 7f05db7fe640 1 -- 172.21.15.12:0/312719405 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05dc09c780 con 0x7f05dc13d380 2024-09-11T01:47:30.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.712+0000 7f05db7fe640 1 -- 172.21.15.12:0/312719405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05dc13d380 msgr2=0x7f05dc13f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:30.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.712+0000 7f05db7fe640 1 --2- 172.21.15.12:0/312719405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05dc13d380 0x7f05dc13f770 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f05dc138d80 tx=0x7f05d402a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de931fd89fd6b486 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:30.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.712+0000 7f05d9ffb640 1 -- 172.21.15.12:0/312719405 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d4037070 con 0x7f05dc13d380 2024-09-11T01:47:30.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.712+0000 7f05d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:30.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.712+0000 7f05d9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:47:30.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.712+0000 7f05d9ffb640 10 monclient: dump: 2024-09-11T01:47:30.724 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:30.724 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.724 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.724 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.725 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:30.725 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.725 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:30.725 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:30.725 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:30.725 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.712+0000 7f05d9ffb640 10 monclient: _finish_auth 0 2024-09-11T01:47:30.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.712+0000 7f05d9ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:47:30.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.712+0000 7f05d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:30.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.712+0000 7f05e15a0640 5 monclient: authenticate success, global_id 204123 2024-09-11T01:47:30.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.712+0000 7f05e15a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:47:30.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.712+0000 7f05d9ffb640 1 -- 172.21.15.12:0/312719405 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05d4032440 con 0x7f05dc13d380 2024-09-11T01:47:30.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.712+0000 7f05d9ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:47:30.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.712+0000 7f05d9ffb640 1 -- 172.21.15.12:0/312719405 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d40327f0 con 0x7f05dc13d380 2024-09-11T01:47:30.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.712+0000 7f05d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:30.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.712+0000 7f05d9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:47:30.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.712+0000 7f05d9ffb640 10 monclient: dump: 2024-09-11T01:47:30.726 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:30.726 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.726 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.726 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.726 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:30.726 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.726 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:30.726 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:30.726 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:30.726 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.712+0000 7f05e15a0640 10 monclient: get_monmap_and_config success 2024-09-11T01:47:30.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.712+0000 7f05e15a0640 4 set_mon_vals no callback set 2024-09-11T01:47:30.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.713+0000 7f05e15a0640 10 monclient: shutdown 2024-09-11T01:47:30.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.713+0000 7f05e15a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:47:30.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.713+0000 7f05e15a0640 1 -- 172.21.15.12:0/312719405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05dc13d380 msgr2=0x7f05dc13f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.713+0000 7f05e15a0640 1 --2- 172.21.15.12:0/312719405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05dc13d380 0x7f05dc13f770 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f05dc138d80 tx=0x7f05d402a080 comp rx=0 tx=0).stop 2024-09-11T01:47:30.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.713+0000 7f05e15a0640 1 -- 172.21.15.12:0/312719405 shutdown_connections 2024-09-11T01:47:30.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.713+0000 7f05e15a0640 1 --2- 172.21.15.12:0/312719405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05dc13d380 0x7f05dc13f770 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.713+0000 7f05e15a0640 1 --2- 172.21.15.12:0/312719405 >> v2:172.21.15.71:3300/0 conn(0x7f05dc137e50 0x7f05dc09d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.713+0000 7f05e15a0640 1 -- 172.21.15.12:0/312719405 >> 172.21.15.12:0/312719405 conn(0x7f05dc0a7020 msgr2=0x7f05dc0a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:47:30.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.713+0000 7f05e15a0640 1 -- 172.21.15.12:0/312719405 shutdown_connections 2024-09-11T01:47:30.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.713+0000 7f05e15a0640 1 -- 172.21.15.12:0/312719405 wait complete. 2024-09-11T01:47:30.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.714+0000 7f05e15a0640 10 monclient: build_initial_monmap 2024-09-11T01:47:30.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.714+0000 7f05e15a0640 10 monclient: monmap: 2024-09-11T01:47:30.728 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:47:30.728 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:47:30.728 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:47:30.728 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:47:30.728 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:47:30.728 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.728 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:47:30.728 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:47:30.728 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:47:30.728 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.714+0000 7f05e15a0640 1 Processor -- start 2024-09-11T01:47:30.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.714+0000 7f05e15a0640 1 -- start start 2024-09-11T01:47:30.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.714+0000 7f05e15a0640 10 monclient: init 2024-09-11T01:47:30.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05e15a0640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:30.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05e15a0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:47:30.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05e15a0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f05dc137e50 0x7f05dc1cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05e15a0640 10 monclient(hunting): picked mon.noname-c con 0x7f05dc137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:47:30.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05e15a0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05dc138230 0x7f05dc1370c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05e15a0640 10 monclient(hunting): picked mon.noname-b con 0x7f05dc138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:47:30.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05e15a0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05dc13d380 0x7f05dc13a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05e15a0640 10 monclient(hunting): picked mon.noname-a con 0x7f05dc13d380 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:47:30.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05e15a0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05e15a0640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f05dc06f890 con 0x7f05dc13d380 2024-09-11T01:47:30.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05e15a0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05e15a0640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f05dc06f710 con 0x7f05dc138230 2024-09-11T01:47:30.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05e15a0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05e15a0640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f05dc13fcb0 con 0x7f05dc137e50 2024-09-11T01:47:30.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05e15a0640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:30.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05e15a0640 10 monclient(hunting): authenticate will time out at 1857.157132s 2024-09-11T01:47:30.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05daffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f05dc137e50 0x7f05dc1cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05db7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05dc13d380 0x7f05dc13a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05daffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f05dc137e50 0x7f05dc1cd020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:33078/0 (socket says 172.21.15.12:33078) 2024-09-11T01:47:30.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05daffd640 1 -- 172.21.15.12:0/511170154 learned_addr learned my addr 172.21.15.12:0/511170154 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:47:30.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05daffd640 10 monclient(hunting): get_auth_request con 0x7f05dc137e50 auth_method 0 2024-09-11T01:47:30.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05daffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05db7fe640 10 monclient(hunting): get_auth_request con 0x7f05dc13d380 auth_method 0 2024-09-11T01:47:30.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:30.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:30.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05da7fc640 1 --2- 172.21.15.12:0/511170154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05dc138230 0x7f05dc1370c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05db7fe640 10 monclient(hunting): handle_auth_done global_id 204126 payload 824 2024-09-11T01:47:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05db7fe640 1 -- 172.21.15.12:0/511170154 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f05dc137e50 msgr2=0x7f05dc1cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05db7fe640 1 --2- 172.21.15.12:0/511170154 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f05dc137e50 0x7f05dc1cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05db7fe640 1 -- 172.21.15.12:0/511170154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05dc138230 msgr2=0x7f05dc1370c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05db7fe640 1 --2- 172.21.15.12:0/511170154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05dc138230 0x7f05dc1370c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05db7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:47:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05db7fe640 1 monclient: found mon.noname-a 2024-09-11T01:47:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05db7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:47:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05db7fe640 1 -- 172.21.15.12:0/511170154 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05dc121770 con 0x7f05dc13d380 2024-09-11T01:47:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05da7fc640 10 monclient: get_auth_request con 0x7f05dc138230 auth_method 0 2024-09-11T01:47:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.715+0000 7f05da7fc640 1 --2- 172.21.15.12:0/511170154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05dc138230 0x7f05dc1370c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:47:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.717+0000 7f05db7fe640 1 -- 172.21.15.12:0/511170154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05dc13d380 msgr2=0x7f05dc13a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:30.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.717+0000 7f05db7fe640 1 --2- 172.21.15.12:0/511170154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05dc13d380 0x7f05dc13a430 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f05d4031000 tx=0x7f05d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:30.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.717+0000 7f05d37fe640 1 -- 172.21.15.12:0/511170154 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d4041030 con 0x7f05dc13d380 2024-09-11T01:47:30.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.717+0000 7f05d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:30.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.717+0000 7f05d37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:47:30.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.717+0000 7f05d37fe640 10 monclient: dump: 2024-09-11T01:47:30.733 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:30.733 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.733 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.733 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.733 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:30.733 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.733 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:30.733 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:30.733 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:30.734 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.717+0000 7f05d37fe640 10 monclient: _finish_auth 0 2024-09-11T01:47:30.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.717+0000 7f05d37fe640 10 monclient: _check_auth_tickets 2024-09-11T01:47:30.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.717+0000 7f05d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:30.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.717+0000 7f05d37fe640 1 -- 172.21.15.12:0/511170154 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05d403d630 con 0x7f05dc13d380 2024-09-11T01:47:30.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.717+0000 7f05e15a0640 5 monclient: authenticate success, global_id 204126 2024-09-11T01:47:30.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.717+0000 7f05e15a0640 10 monclient: _renew_subs 2024-09-11T01:47:30.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.717+0000 7f05e15a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:47:30.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.717+0000 7f05e15a0640 1 -- 172.21.15.12:0/511170154 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05dc13ca50 con 0x7f05dc13d380 2024-09-11T01:47:30.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.717+0000 7f05d37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:47:30.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.718+0000 7f05d37fe640 1 -- 172.21.15.12:0/511170154 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d4037040 con 0x7f05dc13d380 2024-09-11T01:47:30.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.718+0000 7f05d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:30.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.718+0000 7f05d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:47:30.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.718+0000 7f05d37fe640 10 monclient: dump: 2024-09-11T01:47:30.735 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:30.735 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.735 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.735 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.735 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:30.735 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.735 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:30.735 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:30.735 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:30.735 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.718+0000 7f05d8ff9640 4 set_mon_vals no callback set 2024-09-11T01:47:30.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.718+0000 7f05e15a0640 10 monclient: _renew_subs 2024-09-11T01:47:30.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.718+0000 7f05e15a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:47:30.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.718+0000 7f05e15a0640 1 -- 172.21.15.12:0/511170154 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05dc13bef0 con 0x7f05dc13d380 2024-09-11T01:47:30.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.718+0000 7f05e15a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:30.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.718+0000 7f05e15a0640 1 --2- 172.21.15.12:0/511170154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05dc1211b0 0x7f05dc13a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.718+0000 7f05e15a0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.718+0000 7f05e15a0640 1 -- 172.21.15.12:0/511170154 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f05dc06a060 con 0x7f05dc1211b0 2024-09-11T01:47:30.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.718+0000 7f05d37fe640 1 -- 172.21.15.12:0/511170154 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f05d403a030 con 0x7f05dc13d380 2024-09-11T01:47:30.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.719+0000 7f05d37fe640 1 --2- 172.21.15.12:0/511170154 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f05bc03e240 0x7f05bc040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.719+0000 7f05daffd640 1 --2- 172.21.15.12:0/511170154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05dc1211b0 0x7f05dc13a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.719+0000 7f05daffd640 10 monclient: get_auth_request con 0x7f05dc1211b0 auth_method 0 2024-09-11T01:47:30.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.719+0000 7f05daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.719+0000 7f05daffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.719+0000 7f05daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.719+0000 7f05d37fe640 1 -- 172.21.15.12:0/511170154 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(617..617 src has 507..617) ==== 4803+0+0 (secure 0 0 0) 0x7f05d4072910 con 0x7f05dc13d380 2024-09-11T01:47:30.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.719+0000 7f05daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.719+0000 7f05daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.719+0000 7f05daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.719+0000 7f05da7fc640 1 --2- 172.21.15.12:0/511170154 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f05bc03e240 0x7f05bc040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.719+0000 7f05da7fc640 10 monclient: get_auth_request con 0x7f05bc03e240 auth_method 0 2024-09-11T01:47:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.719+0000 7f05daffd640 10 monclient(hunting): handle_auth_done global_id 134704 payload 824 2024-09-11T01:47:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.719+0000 7f05daffd640 1 -- 172.21.15.12:0/511170154 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f05dc0a1600 con 0x7f05dc1211b0 2024-09-11T01:47:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.721+0000 7f05daffd640 1 -- 172.21.15.12:0/511170154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05dc1211b0 msgr2=0x7f05dc13a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.721+0000 7f05daffd640 1 --2- 172.21.15.12:0/511170154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05dc1211b0 0x7f05dc13a970 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f05cc004980 tx=0x7f05cc007700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.721+0000 7f05da7fc640 1 --2- 172.21.15.12:0/511170154 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f05bc03e240 0x7f05bc040700 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f05dc120ef0 tx=0x7f05c4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.721+0000 7f05d37fe640 1 -- 172.21.15.12:0/511170154 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05cc0147a0 con 0x7f05dc1211b0 2024-09-11T01:47:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.721+0000 7f05d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.721+0000 7f05d37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:47:30.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.721+0000 7f05d37fe640 10 monclient: dump: 2024-09-11T01:47:30.738 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:30.738 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.738 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.738 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.738 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:30.738 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.738 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:30.738 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:30.738 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:30.738 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.722+0000 7f05d37fe640 1 -- 172.21.15.12:0/511170154 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f05cc01a070 con 0x7f05dc1211b0 2024-09-11T01:47:30.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.722+0000 7f05d37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:30.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.722+0000 7f05d37fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:47:30.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.722+0000 7f05d37fe640 1 -- 172.21.15.12:0/511170154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05dc1211b0 msgr2=0x7f05dc13a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.722+0000 7f05d37fe640 1 --2- 172.21.15.12:0/511170154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05dc1211b0 0x7f05dc13a970 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f05cc004980 tx=0x7f05cc007700 comp rx=0 tx=0).stop 2024-09-11T01:47:30.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.722+0000 7f05d37fe640 1 -- 172.21.15.12:0/511170154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05dc1211b0 msgr2=0x7f05dc13a970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:47:30.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.722+0000 7f05d37fe640 1 --2- 172.21.15.12:0/511170154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05dc1211b0 0x7f05dc13a970 secure :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f05cc004980 tx=0x7f05cc007700 comp rx=0 tx=0).stop 2024-09-11T01:47:30.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.728+0000 7f05e15a0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:47:30.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.728+0000 7f05e15a0640 1 --2- 172.21.15.12:0/511170154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05dc13aeb0 0x7f05dc0a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.728+0000 7f05e15a0640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.728+0000 7f05e15a0640 1 -- 172.21.15.12:0/511170154 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f05dc13b2c0 con 0x7f05dc13aeb0 2024-09-11T01:47:30.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.728+0000 7f05daffd640 1 --2- 172.21.15.12:0/511170154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05dc13aeb0 0x7f05dc0a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.728+0000 7f05daffd640 10 monclient: get_auth_request con 0x7f05dc13aeb0 auth_method 0 2024-09-11T01:47:30.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.728+0000 7f05daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.728+0000 7f05daffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.728+0000 7f05daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.728+0000 7f05daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.728+0000 7f05daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.728+0000 7f05daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:30.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.729+0000 7f05daffd640 10 monclient(hunting): handle_auth_done global_id 134707 payload 824 2024-09-11T01:47:30.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.729+0000 7f05daffd640 1 -- 172.21.15.12:0/511170154 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f05dc137bb0 con 0x7f05dc13aeb0 2024-09-11T01:47:30.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.731+0000 7f05daffd640 1 -- 172.21.15.12:0/511170154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05dc13aeb0 msgr2=0x7f05dc0a1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:30.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.731+0000 7f05daffd640 1 --2- 172.21.15.12:0/511170154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05dc13aeb0 0x7f05dc0a1880 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f05cc004980 tx=0x7f05cc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:30.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.731+0000 7f05d37fe640 1 -- 172.21.15.12:0/511170154 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05cc014bb0 con 0x7f05dc13aeb0 2024-09-11T01:47:30.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.731+0000 7f05d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:30.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.731+0000 7f05d37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:47:30.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.731+0000 7f05d37fe640 10 monclient: dump: 2024-09-11T01:47:30.743 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:30.743 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.743 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.743 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.743 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:30.743 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.743 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:30.743 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:30.743 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:30.743 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.731+0000 7f05d37fe640 1 -- 172.21.15.12:0/511170154 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f05cc014bb0 con 0x7f05dc13aeb0 2024-09-11T01:47:30.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.731+0000 7f05d37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:47:30.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.731+0000 7f05d37fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:47:30.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.731+0000 7f05d37fe640 1 -- 172.21.15.12:0/511170154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05dc13aeb0 msgr2=0x7f05dc0a1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.731+0000 7f05d37fe640 1 --2- 172.21.15.12:0/511170154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05dc13aeb0 0x7f05dc0a1880 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f05cc004980 tx=0x7f05cc004a40 comp rx=0 tx=0).stop 2024-09-11T01:47:30.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.731+0000 7f05d37fe640 1 -- 172.21.15.12:0/511170154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05dc13aeb0 msgr2=0x7f05dc0a1880 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:47:30.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.731+0000 7f05d37fe640 1 --2- 172.21.15.12:0/511170154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05dc13aeb0 0x7f05dc0a1880 secure :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f05cc004980 tx=0x7f05cc004a40 comp rx=0 tx=0).stop 2024-09-11T01:47:30.744 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:47:30.744 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:47:30.744 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:47:30.744 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:47:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 172, 2024-09-11T01:47:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 383693, 2024-09-11T01:47:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:47:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:47:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:47:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:47:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:47:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:47:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:47:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:47:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:47:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:47:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:47:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:47:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:47:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:47:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:47:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:47:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:47:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:47:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:47:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:47:30.747 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:47:30.747 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:47:30.747 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:47:30.747 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:47:30.747 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:47:30.747 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:47:30.747 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:47:30.747 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:47:30.747 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:47:30.747 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:47:30.747 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:47:30.747 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:47:30.747 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:30.748 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.748 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:47:30.748 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:47:30.748 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:47:30.748 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:47:30.748 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:47:30.748 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:47:30.748 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:47:30.748 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:47:30.748 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:47:30.748 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:47:30.748 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:47:30.748 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:47:30.748 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:47:30.749 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:47:30.749 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:47:30.749 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:47:30.749 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:47:30.749 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:47:30.749 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:47:30.749 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:47:30.749 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:47:30.749 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:47:30.749 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:47:30.749 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:47:30.749 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:47:30.749 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:47:30.750 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:47:30.750 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:47:30.750 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:30.750 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:47:30.750 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.750 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:47:30.750 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.750 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:47:30.750 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:47:30.750 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:47:30.750 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:47:30.750 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.750 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:47:30.750 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:47:30.751 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:30.751 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.751 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.751 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:47:30.751 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:47:30.751 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:30.751 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:30.751 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:30.751 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.751 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:47:30.751 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:47:30.752 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:47:30.752 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:47:30.752 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:47:30.752 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.752 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.752 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:47:30.752 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:47:30.752 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:47:30.752 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:47:30.752 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.752 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:47:30.753 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:47:30.753 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:30.753 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.753 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.753 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:47:30.753 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:47:30.753 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:30.753 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:30.753 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:30.753 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.753 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:47:30.754 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:47:30.754 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:47:30.754 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:47:30.754 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:47:30.754 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.754 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.754 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:47:30.754 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:47:30.754 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:47:30.754 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:47:30.754 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.754 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:47:30.754 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:47:30.754 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:30.755 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.755 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.755 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:47:30.755 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:47:30.755 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:30.755 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:30.755 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:30.755 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.755 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:47:30.755 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:47:30.755 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:47:30.755 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:47:30.755 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:47:30.755 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:30.756 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:30.756 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.756 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:47:30.756 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:47:30.756 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.756 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:47:30.756 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:47:30.756 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:47:30.756 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:30.756 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:30.756 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:47:30.756 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.756 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:47:30.757 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:47:30.757 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:47:30.757 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:30.757 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:30.757 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:47:30.757 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.757 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:47:30.757 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:47:30.757 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:47:30.757 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:30.757 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:30.757 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:47:30.758 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:30.758 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:47:30.758 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:47:30.758 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:47:30.758 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:30.758 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:30.758 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:30.758 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:47:30.758 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:47:30.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.732+0000 7f05e15a0640 1 -- 172.21.15.12:0/511170154 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f05bc03e240 msgr2=0x7f05bc040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.732+0000 7f05e15a0640 1 --2- 172.21.15.12:0/511170154 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f05bc03e240 0x7f05bc040700 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f05dc120ef0 tx=0x7f05c4009040 comp rx=0 tx=0).stop 2024-09-11T01:47:30.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.732+0000 7f05e15a0640 10 monclient: shutdown 2024-09-11T01:47:30.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.732+0000 7f05e15a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:47:30.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.732+0000 7f05e15a0640 1 -- 172.21.15.12:0/511170154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05dc13d380 msgr2=0x7f05dc13a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.732+0000 7f05e15a0640 1 --2- 172.21.15.12:0/511170154 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f05dc13d380 0x7f05dc13a430 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f05d4031000 tx=0x7f05d40046c0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.732+0000 7f05db7fe640 1 -- 172.21.15.12:0/511170154 reap_dead start 2024-09-11T01:47:30.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.732+0000 7f05db7fe640 1 -- 172.21.15.12:0/511170154 reap_dead start 2024-09-11T01:47:30.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.732+0000 7f05e15a0640 1 -- 172.21.15.12:0/511170154 shutdown_connections 2024-09-11T01:47:30.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.732+0000 7f05e15a0640 1 -- 172.21.15.12:0/511170154 >> 172.21.15.12:0/511170154 conn(0x7f05dc0a7020 msgr2=0x7f05dc0a0430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:47:30.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.732+0000 7f05e15a0640 1 -- 172.21.15.12:0/511170154 shutdown_connections 2024-09-11T01:47:30.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.732+0000 7f05e15a0640 1 -- 172.21.15.12:0/511170154 wait complete. 2024-09-11T01:47:30.761 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:47:30.761 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:47:30.761 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:47:30.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.860+0000 7f52d129b640 10 monclient: get_monmap_and_config 2024-09-11T01:47:30.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.860+0000 7f52d129b640 10 monclient: build_initial_monmap 2024-09-11T01:47:30.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.860+0000 7f52d129b640 10 monclient: monmap: 2024-09-11T01:47:30.862 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:47:30.862 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.862 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:47:30.860849+0000 2024-09-11T01:47:30.862 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:47:30.860849+0000 2024-09-11T01:47:30.862 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:47:30.862 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.862 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:47:30.862 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:47:30.862 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:47:30.862 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:47:30.862 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:47:30.863 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.860+0000 7f52d129b640 1 Processor -- start 2024-09-11T01:47:30.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.860+0000 7f52d129b640 1 -- start start 2024-09-11T01:47:30.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.860+0000 7f52d129b640 10 monclient: init 2024-09-11T01:47:30.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52d129b640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:30.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52d129b640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-11T01:47:30.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52d129b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f52cc137d50 0x7f52cc09d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52d129b640 10 monclient(hunting): picked mon.noname-b con 0x7f52cc137d50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:47:30.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52d129b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f52cc138130 addr v1:172.21.15.71:6789/0 2024-09-11T01:47:30.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52d129b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f52cc13d280 0x7f52cc13f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52d129b640 10 monclient(hunting): picked mon.noname-c con 0x7f52cc13d280 addr v2:172.21.15.71:3300/0 2024-09-11T01:47:30.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52d129b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52cc09ba20 con 0x7f52cc138130 2024-09-11T01:47:30.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52d129b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52d129b640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f52cc09c650 con 0x7f52cc13d280 2024-09-11T01:47:30.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52d129b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52d129b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f52cc09c370 con 0x7f52cc137d50 2024-09-11T01:47:30.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52d129b640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:30.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52d129b640 10 monclient(hunting): authenticate will time out at 1857.303300s 2024-09-11T01:47:30.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52caffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f52cc137d50 0x7f52cc09d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52caffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f52cc137d50 0x7f52cc09d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:33090/0 (socket says 172.21.15.12:33090) 2024-09-11T01:47:30.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52caffd640 1 -- 172.21.15.12:0/1270072072 learned_addr learned my addr 172.21.15.12:0/1270072072 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:47:30.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52caffd640 10 monclient(hunting): get_auth_request con 0x7f52cc137d50 auth_method 0 2024-09-11T01:47:30.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52caffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:30.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52caffd640 10 monclient(hunting): handle_auth_done global_id 184292 payload 274 2024-09-11T01:47:30.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52caffd640 1 -- 172.21.15.12:0/1270072072 >> v2:172.21.15.71:3300/0 conn(0x7f52cc13d280 msgr2=0x7f52cc13f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:47:30.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52cb7fe640 1 --2- 172.21.15.12:0/1270072072 >> v2:172.21.15.71:3300/0 conn(0x7f52cc13d280 0x7f52cc13f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52caffd640 1 --2- 172.21.15.12:0/1270072072 >> v2:172.21.15.71:3300/0 conn(0x7f52cc13d280 0x7f52cc13f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52caffd640 1 -- 172.21.15.12:0/1270072072 >> v1:172.21.15.71:6789/0 conn(0x7f52cc138130 legacy=0x7f52cc138550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:47:30.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52caffd640 10 monclient: _finish_hunting 0 2024-09-11T01:47:30.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52caffd640 1 monclient: found mon.noname-b 2024-09-11T01:47:30.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:30.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52caffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:47:30.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52caffd640 1 -- 172.21.15.12:0/1270072072 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52cc0a6ce0 con 0x7f52cc137d50 2024-09-11T01:47:30.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52cb7fe640 10 monclient: get_auth_request con 0x7f52cc13d280 auth_method 0 2024-09-11T01:47:30.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.861+0000 7f52cb7fe640 1 --2- 172.21.15.12:0/1270072072 >> v2:172.21.15.71:3300/0 conn(0x7f52cc13d280 0x7f52cc13f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:47:30.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.864+0000 7f52caffd640 1 -- 172.21.15.12:0/1270072072 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f52cc137d50 msgr2=0x7f52cc09d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:30.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.864+0000 7f52caffd640 1 --2- 172.21.15.12:0/1270072072 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f52cc137d50 0x7f52cc09d110 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f52cc09c7d0 tx=0x7f52c402a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f36911b3208cf2f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:30.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.864+0000 7f52c9ffb640 1 -- 172.21.15.12:0/1270072072 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c4037070 con 0x7f52cc137d50 2024-09-11T01:47:30.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.864+0000 7f52c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:30.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.864+0000 7f52c9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:47:30.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.864+0000 7f52c9ffb640 10 monclient: dump: 2024-09-11T01:47:30.876 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:30.876 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.876 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.876 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.876 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:30.876 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.876 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:30.876 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:30.876 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:30.877 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.864+0000 7f52c9ffb640 10 monclient: _finish_auth 0 2024-09-11T01:47:30.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.864+0000 7f52c9ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:47:30.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.864+0000 7f52c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:30.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.864+0000 7f52c9ffb640 1 -- 172.21.15.12:0/1270072072 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52c4032640 con 0x7f52cc137d50 2024-09-11T01:47:30.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.864+0000 7f52d129b640 5 monclient: authenticate success, global_id 184292 2024-09-11T01:47:30.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.864+0000 7f52d129b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:47:30.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.864+0000 7f52c9ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:47:30.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.864+0000 7f52c9ffb640 1 -- 172.21.15.12:0/1270072072 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c4032a10 con 0x7f52cc137d50 2024-09-11T01:47:30.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.864+0000 7f52c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:30.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.864+0000 7f52c9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:47:30.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.864+0000 7f52c9ffb640 10 monclient: dump: 2024-09-11T01:47:30.877 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:30.877 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.878 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.878 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.878 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:30.878 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.878 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:30.878 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:30.878 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:30.878 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.864+0000 7f52d129b640 10 monclient: get_monmap_and_config success 2024-09-11T01:47:30.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.864+0000 7f52d129b640 4 set_mon_vals no callback set 2024-09-11T01:47:30.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.865+0000 7f52d129b640 10 monclient: shutdown 2024-09-11T01:47:30.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.865+0000 7f52d129b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:47:30.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.865+0000 7f52d129b640 1 -- 172.21.15.12:0/1270072072 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f52cc137d50 msgr2=0x7f52cc09d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.865+0000 7f52d129b640 1 --2- 172.21.15.12:0/1270072072 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f52cc137d50 0x7f52cc09d110 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f52cc09c7d0 tx=0x7f52c402a110 comp rx=0 tx=0).stop 2024-09-11T01:47:30.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.865+0000 7f52d129b640 1 -- 172.21.15.12:0/1270072072 shutdown_connections 2024-09-11T01:47:30.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.865+0000 7f52d129b640 1 --2- 172.21.15.12:0/1270072072 >> v2:172.21.15.71:3300/0 conn(0x7f52cc13d280 0x7f52cc13f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.865+0000 7f52d129b640 1 --2- 172.21.15.12:0/1270072072 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f52cc137d50 0x7f52cc09d110 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.865+0000 7f52d129b640 1 -- 172.21.15.12:0/1270072072 >> 172.21.15.12:0/1270072072 conn(0x7f52cc0a7020 msgr2=0x7f52cc0a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:47:30.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.865+0000 7f52d129b640 1 -- 172.21.15.12:0/1270072072 shutdown_connections 2024-09-11T01:47:30.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.865+0000 7f52d129b640 1 -- 172.21.15.12:0/1270072072 wait complete. 2024-09-11T01:47:30.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.866+0000 7f52d129b640 10 monclient: build_initial_monmap 2024-09-11T01:47:30.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.866+0000 7f52d129b640 10 monclient: monmap: 2024-09-11T01:47:30.879 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:47:30.879 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:47:30.879 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:47:30.879 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:47:30.879 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:47:30.880 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.880 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:47:30.880 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:47:30.880 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:47:30.880 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.866+0000 7f52d129b640 1 Processor -- start 2024-09-11T01:47:30.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.866+0000 7f52d129b640 1 -- start start 2024-09-11T01:47:30.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.866+0000 7f52d129b640 10 monclient: init 2024-09-11T01:47:30.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.866+0000 7f52d129b640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:30.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.866+0000 7f52d129b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:47:30.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52d129b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52cc137d50 0x7f52cc1cce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52d129b640 10 monclient(hunting): picked mon.noname-a con 0x7f52cc137d50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:47:30.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52d129b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f52cc138130 0x7f52cc136f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52d129b640 10 monclient(hunting): picked mon.noname-c con 0x7f52cc138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:47:30.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52d129b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52cc13d280 0x7f52cc13a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52d129b640 10 monclient(hunting): picked mon.noname-b con 0x7f52cc13d280 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:47:30.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52d129b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52d129b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f52cc09d7d0 con 0x7f52cc137d50 2024-09-11T01:47:30.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52d129b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52d129b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f52cc09d650 con 0x7f52cc13d280 2024-09-11T01:47:30.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52d129b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:30.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52d129b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f52cc13fbb0 con 0x7f52cc138130 2024-09-11T01:47:30.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52d129b640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:30.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52d129b640 10 monclient(hunting): authenticate will time out at 1857.309075s 2024-09-11T01:47:30.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52ca7fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f52cc138130 0x7f52cc136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52caffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52cc137d50 0x7f52cc1cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52caffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52cc137d50 0x7f52cc1cce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38500/0 (socket says 172.21.15.12:38500) 2024-09-11T01:47:30.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52ca7fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f52cc138130 0x7f52cc136f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:33100/0 (socket says 172.21.15.12:33100) 2024-09-11T01:47:30.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52caffd640 1 -- 172.21.15.12:0/1406705634 learned_addr learned my addr 172.21.15.12:0/1406705634 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:47:30.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52caffd640 10 monclient(hunting): get_auth_request con 0x7f52cc137d50 auth_method 0 2024-09-11T01:47:30.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52caffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52ca7fc640 10 monclient(hunting): get_auth_request con 0x7f52cc138130 auth_method 0 2024-09-11T01:47:30.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52ca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:30.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52ca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52ca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52ca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:30.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52cb7fe640 1 --2- 172.21.15.12:0/1406705634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52cc13d280 0x7f52cc13a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52cb7fe640 10 monclient(hunting): get_auth_request con 0x7f52cc13d280 auth_method 0 2024-09-11T01:47:30.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:30.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:30.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:30.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:30.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:30.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:30.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52ca7fc640 10 monclient(hunting): handle_auth_done global_id 184295 payload 824 2024-09-11T01:47:30.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52ca7fc640 1 -- 172.21.15.12:0/1406705634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52cc13d280 msgr2=0x7f52cc13a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52ca7fc640 1 --2- 172.21.15.12:0/1406705634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52cc13d280 0x7f52cc13a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52ca7fc640 1 -- 172.21.15.12:0/1406705634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52cc137d50 msgr2=0x7f52cc1cce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:30.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52ca7fc640 1 --2- 172.21.15.12:0/1406705634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52cc137d50 0x7f52cc1cce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:30.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52ca7fc640 10 monclient: _finish_hunting 0 2024-09-11T01:47:30.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52ca7fc640 1 monclient: found mon.noname-c 2024-09-11T01:47:30.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:30.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52ca7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:47:30.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52ca7fc640 1 -- 172.21.15.12:0/1406705634 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52cc13bc50 con 0x7f52cc138130 2024-09-11T01:47:30.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.867+0000 7f52caffd640 1 --2- 172.21.15.12:0/1406705634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52cc137d50 0x7f52cc1cce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:47:30.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.869+0000 7f52ca7fc640 1 -- 172.21.15.12:0/1406705634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f52cc138130 msgr2=0x7f52cc136f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:30.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.869+0000 7f52ca7fc640 1 --2- 172.21.15.12:0/1406705634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f52cc138130 0x7f52cc136f40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f52b400c440 tx=0x7f52b400c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:30.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.869+0000 7f52c37fe640 1 -- 172.21.15.12:0/1406705634 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b40119a0 con 0x7f52cc138130 2024-09-11T01:47:30.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.869+0000 7f52c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:30.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.869+0000 7f52c37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:47:30.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.869+0000 7f52c37fe640 10 monclient: dump: 2024-09-11T01:47:30.885 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:30.885 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.885 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.885 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.885 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:30.885 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.885 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:30.886 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:30.886 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:30.886 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.869+0000 7f52c37fe640 10 monclient: _finish_auth 0 2024-09-11T01:47:30.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.869+0000 7f52c37fe640 10 monclient: _check_auth_tickets 2024-09-11T01:47:30.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.869+0000 7f52c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:30.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.869+0000 7f52c37fe640 1 -- 172.21.15.12:0/1406705634 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52b4017070 con 0x7f52cc138130 2024-09-11T01:47:30.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.869+0000 7f52c37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:47:30.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.869+0000 7f52c37fe640 1 -- 172.21.15.12:0/1406705634 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b4011d80 con 0x7f52cc138130 2024-09-11T01:47:30.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.869+0000 7f52d129b640 5 monclient: authenticate success, global_id 184295 2024-09-11T01:47:30.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.869+0000 7f52d129b640 10 monclient: _renew_subs 2024-09-11T01:47:30.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.869+0000 7f52d129b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:47:30.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.869+0000 7f52d129b640 1 -- 172.21.15.12:0/1406705634 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52cc0a1710 con 0x7f52cc138130 2024-09-11T01:47:30.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.869+0000 7f52c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:30.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.869+0000 7f52c97fa640 4 set_mon_vals no callback set 2024-09-11T01:47:30.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.870+0000 7f52c37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:47:30.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.870+0000 7f52c37fe640 10 monclient: dump: 2024-09-11T01:47:30.887 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:30.887 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:30.887 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.887 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:30.887 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:30.887 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:30.887 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:30.887 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:30.887 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:30.887 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:30.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.870+0000 7f52d129b640 10 monclient: _renew_subs 2024-09-11T01:47:30.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.870+0000 7f52d129b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:47:30.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.870+0000 7f52d129b640 1 -- 172.21.15.12:0/1406705634 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f52cc13c7b0 con 0x7f52cc138130 2024-09-11T01:47:30.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.870+0000 7f52c37fe640 1 -- 172.21.15.12:0/1406705634 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f52b401c460 con 0x7f52cc138130 2024-09-11T01:47:30.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.870+0000 7f52d129b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:30.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.870+0000 7f52d129b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:30.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.871+0000 7f52d129b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:47:30.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.871+0000 7f52d129b640 1 -- 172.21.15.12:0/1406705634 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52cc09d110 con 0x7f52cc138130 2024-09-11T01:47:30.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.871+0000 7f52c37fe640 1 --2- 172.21.15.12:0/1406705634 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f52ac03e260 0x7f52ac040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:30.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.871+0000 7f52c37fe640 1 -- 172.21.15.12:0/1406705634 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(617..617 src has 507..617) ==== 4803+0+0 (secure 0 0 0) 0x7f52b4052af0 con 0x7f52cc138130 2024-09-11T01:47:30.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.871+0000 7f52caffd640 1 --2- 172.21.15.12:0/1406705634 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f52ac03e260 0x7f52ac040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:30.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.871+0000 7f52caffd640 10 monclient: get_auth_request con 0x7f52ac03e260 auth_method 0 2024-09-11T01:47:30.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.874+0000 7f52caffd640 1 --2- 172.21.15.12:0/1406705634 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f52ac03e260 0x7f52ac040720 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f52c402a150 tx=0x7f52c402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:30.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.876+0000 7f52c37fe640 1 -- 172.21.15.12:0/1406705634 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f52b40514d0 con 0x7f52cc138130 2024-09-11T01:47:30.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.876+0000 7f52c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:30.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:30.876+0000 7f52c37fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:47:31.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:31.132+0000 7f52d129b640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:47:31.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:31.132+0000 7f52d129b640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:47:31.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:31.132+0000 7f52d129b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:47:31.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:31.132+0000 7f52d129b640 1 -- 172.21.15.12:0/1406705634 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f52cc121270 con 0x7f52cc138130 2024-09-11T01:47:31.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:31.133+0000 7f52c37fe640 1 -- 172.21.15.12:0/1406705634 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f52b401ac80 con 0x7f52cc138130 2024-09-11T01:47:31.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:31.133+0000 7f52c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:47:31.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:31.133+0000 7f52c37fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:47:31.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:31.135+0000 7f52d129b640 1 -- 172.21.15.12:0/1406705634 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f52ac03e260 msgr2=0x7f52ac040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:31.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:31.135+0000 7f52d129b640 1 --2- 172.21.15.12:0/1406705634 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f52ac03e260 0x7f52ac040720 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f52c402a150 tx=0x7f52c402e000 comp rx=0 tx=0).stop 2024-09-11T01:47:31.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:31.135+0000 7f52d129b640 10 monclient: shutdown 2024-09-11T01:47:31.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:31.135+0000 7f52d129b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:47:31.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:31.135+0000 7f52d129b640 1 -- 172.21.15.12:0/1406705634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f52cc138130 msgr2=0x7f52cc136f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:31.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:31.135+0000 7f52d129b640 1 --2- 172.21.15.12:0/1406705634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f52cc138130 0x7f52cc136f40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f52b400c440 tx=0x7f52b400c900 comp rx=0 tx=0).stop 2024-09-11T01:47:31.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:31.135+0000 7f52d129b640 1 -- 172.21.15.12:0/1406705634 shutdown_connections 2024-09-11T01:47:31.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:31.135+0000 7f52d129b640 1 --2- 172.21.15.12:0/1406705634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52cc13d280 0x7f52cc13a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:31.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:31.135+0000 7f52d129b640 1 --2- 172.21.15.12:0/1406705634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f52cc138130 0x7f52cc136f40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:31.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:31.135+0000 7f52d129b640 1 --2- 172.21.15.12:0/1406705634 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f52ac03e260 0x7f52ac040720 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:31.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:31.135+0000 7f52d129b640 1 --2- 172.21.15.12:0/1406705634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f52cc137d50 0x7f52cc1cce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:31.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:31.135+0000 7f52d129b640 1 -- 172.21.15.12:0/1406705634 >> 172.21.15.12:0/1406705634 conn(0x7f52cc0a7020 msgr2=0x7f52cc13e7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:47:31.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:31.135+0000 7f52d129b640 1 -- 172.21.15.12:0/1406705634 shutdown_connections 2024-09-11T01:47:31.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:31.135+0000 7f52d129b640 1 -- 172.21.15.12:0/1406705634 wait complete. 2024-09-11T01:47:31.138 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.136+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.139 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.137+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=12,kv=11,logm=44} crc {auth=124044233,config=2586702516,health=2689222117,kv=287466125,logm=525113813}) 2024-09-11T01:47:31.139 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.137+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=31,config=2,health=11,kv=11,logm=45} crc {auth=1803553868,config=2586702516,health=3333571077,kv=3263697420,logm=967794051}) 2024-09-11T01:47:31.139 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.137+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.139 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.137+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=12,kv=11,logm=44} crc {auth=124044233,config=2586702516,health=2689222117,kv=287466125,logm=525113813}) 2024-09-11T01:47:31.139 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.137+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=31,config=2,health=12,kv=12,logm=43} crc {auth=1917966744,config=2586702516,health=1862816756,kv=3263697420,logm=605637052}) 2024-09-11T01:47:31.148 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:47:31.148 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.146+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.148 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.146+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1911550069}) 2024-09-11T01:47:31.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.146+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=445781405}) 2024-09-11T01:47:31.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.146+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.146+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1911550069}) 2024-09-11T01:47:31.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.146+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1751054478}) 2024-09-11T01:47:31.149 INFO:tasks.rados.rados.0.smithi071.stdout:2756: done (0 left) 2024-09-11T01:47:31.150 INFO:tasks.rados.rados.0.smithi071.stdout:2757: write oid 35 current snap is 308 2024-09-11T01:47:31.150 INFO:tasks.rados.rados.0.smithi071.stdout:2757: seq_num 981 ranges {487667=428806,1655659=770995} 2024-09-11T01:47:31.154 INFO:tasks.rados.rados.0.smithi071.stdout:2757: writing smithi07134777-35 from 487667 to 916473 tid 1 2024-09-11T01:47:31.162 INFO:tasks.rados.rados.0.smithi071.stdout:2757: writing smithi07134777-35 from 1655659 to 2426654 tid 2 2024-09-11T01:47:31.162 INFO:tasks.rados.rados.0.smithi071.stdout:2758: snap_create 2024-09-11T01:47:31.163 INFO:tasks.rados.rados.0.smithi071.stdout:2757: finishing write tid 1 to smithi07134777-35 2024-09-11T01:47:31.167 INFO:tasks.rados.rados.0.smithi071.stdout:2757: finishing write tid 2 to smithi07134777-35 2024-09-11T01:47:31.167 INFO:tasks.rados.rados.0.smithi071.stdout:2757: finishing write tid 3 to smithi07134777-35 2024-09-11T01:47:31.168 INFO:tasks.rados.rados.0.smithi071.stdout:2757: finishing write tid 4 to smithi07134777-35 2024-09-11T01:47:31.168 INFO:tasks.rados.rados.0.smithi071.stdout:2757: oid 35 updating version 0 to 1036 2024-09-11T01:47:31.168 INFO:tasks.rados.rados.0.smithi071.stdout:2757: oid 35 version 1036 is already newer than 1035 2024-09-11T01:47:31.168 INFO:tasks.rados.rados.0.smithi071.stdout:2757: oid 35 version 1036 is already newer than 1034 2024-09-11T01:47:31.168 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 1036 (ObjNum 981 snap 308 seq_num 981) dirty exists 2024-09-11T01:47:31.168 INFO:tasks.rados.rados.0.smithi071.stdout:2757: left oid 35 (ObjNum 981 snap 308 seq_num 981) 2024-09-11T01:47:31.556 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.555+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.556 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.555+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1154863217}) 2024-09-11T01:47:31.557 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.555+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1244882435}) 2024-09-11T01:47:31.557 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.555+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.557 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.555+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1154863217}) 2024-09-11T01:47:31.557 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.555+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1448001383}) 2024-09-11T01:47:31.558 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.556+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.558 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.556+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867607997}) 2024-09-11T01:47:31.558 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.556+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3189300713}) 2024-09-11T01:47:31.558 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.556+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.558 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.556+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867607997}) 2024-09-11T01:47:31.558 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.556+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=113375332}) 2024-09-11T01:47:31.559 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.558+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.559 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.558+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1934051421}) 2024-09-11T01:47:31.559 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.558+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2174894126}) 2024-09-11T01:47:31.560 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.558+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.560 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.558+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1934051421}) 2024-09-11T01:47:31.560 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.558+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3764159987}) 2024-09-11T01:47:31.560 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.559+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.561 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.559+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3687311816}) 2024-09-11T01:47:31.561 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.559+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4034031527}) 2024-09-11T01:47:31.561 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.559+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.561 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.559+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3687311816}) 2024-09-11T01:47:31.561 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.559+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1336426883}) 2024-09-11T01:47:31.561 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.560+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.561 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.560+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1043862456}) 2024-09-11T01:47:31.562 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.560+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2603696735}) 2024-09-11T01:47:31.562 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.560+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.562 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.560+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1043862456}) 2024-09-11T01:47:31.562 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.560+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2216594348}) 2024-09-11T01:47:31.563 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.561+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.563 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.561+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=599388254}) 2024-09-11T01:47:31.563 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.561+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3990652517}) 2024-09-11T01:47:31.563 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.561+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.563 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.561+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=599388254}) 2024-09-11T01:47:31.563 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.561+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3308537076}) 2024-09-11T01:47:31.564 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.562+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.564 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.562+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1957481805}) 2024-09-11T01:47:31.564 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.562+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3002899298}) 2024-09-11T01:47:31.564 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.562+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.564 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.562+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1957481805}) 2024-09-11T01:47:31.564 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.562+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3018844119}) 2024-09-11T01:47:31.565 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.563+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.565 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.563+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619711448}) 2024-09-11T01:47:31.565 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.563+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1662490898}) 2024-09-11T01:47:31.565 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.563+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.565 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.563+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619711448}) 2024-09-11T01:47:31.565 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.563+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=505027126}) 2024-09-11T01:47:31.566 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.564+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.566 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.564+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3879470460}) 2024-09-11T01:47:31.566 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.564+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2612973291}) 2024-09-11T01:47:31.566 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.564+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.566 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.564+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3879470460}) 2024-09-11T01:47:31.566 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.564+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2869284626}) 2024-09-11T01:47:31.567 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.565+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.567 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.565+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3494830472}) 2024-09-11T01:47:31.567 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.565+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2534785155}) 2024-09-11T01:47:31.567 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.565+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.567 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.565+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3494830472}) 2024-09-11T01:47:31.567 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.565+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1580300519}) 2024-09-11T01:47:31.568 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.566+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.568 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.566+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1857116110}) 2024-09-11T01:47:31.568 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.566+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3066630381}) 2024-09-11T01:47:31.568 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.566+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.569 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.567+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1857116110}) 2024-09-11T01:47:31.569 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.567+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2947525320}) 2024-09-11T01:47:31.569 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.568+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.569 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.568+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1516303874}) 2024-09-11T01:47:31.569 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.568+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4007986566}) 2024-09-11T01:47:31.569 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.568+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.570 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.568+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1516303874}) 2024-09-11T01:47:31.570 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.568+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=36870076}) 2024-09-11T01:47:31.570 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.569+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.570 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.569+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1967194667}) 2024-09-11T01:47:31.570 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.569+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=293527858}) 2024-09-11T01:47:31.571 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.569+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.571 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.569+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1967194667}) 2024-09-11T01:47:31.571 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.569+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4025746810}) 2024-09-11T01:47:31.571 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.570+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.571 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.570+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4234848041}) 2024-09-11T01:47:31.571 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.570+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1041371025}) 2024-09-11T01:47:31.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.570+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.570+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4234848041}) 2024-09-11T01:47:31.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.570+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1006917793}) 2024-09-11T01:47:31.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.571+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.571+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3679659797}) 2024-09-11T01:47:31.573 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.571+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3586844924}) 2024-09-11T01:47:31.573 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.571+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.573 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.571+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3679659797}) 2024-09-11T01:47:31.573 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.571+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2847860164}) 2024-09-11T01:47:31.573 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.572+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.574 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.572+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3555364291}) 2024-09-11T01:47:31.574 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.572+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3087982721}) 2024-09-11T01:47:31.574 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.572+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.574 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.572+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3555364291}) 2024-09-11T01:47:31.574 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.572+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=387133303}) 2024-09-11T01:47:31.575 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.573+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.575 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.573+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2895400077}) 2024-09-11T01:47:31.575 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.573+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=417126514}) 2024-09-11T01:47:31.575 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.573+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.575 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.573+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2895400077}) 2024-09-11T01:47:31.575 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.573+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4066024715}) 2024-09-11T01:47:31.576 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.574+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.576 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.574+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2299582112}) 2024-09-11T01:47:31.576 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.574+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1663383665}) 2024-09-11T01:47:31.576 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.574+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.576 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.574+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2299582112}) 2024-09-11T01:47:31.576 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.574+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3532274441}) 2024-09-11T01:47:31.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.575+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.575+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3864339780}) 2024-09-11T01:47:31.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.575+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1091051901}) 2024-09-11T01:47:31.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.575+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.575+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3864339780}) 2024-09-11T01:47:31.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.575+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1399051597}) 2024-09-11T01:47:31.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.576+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.576+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1553006702}) 2024-09-11T01:47:31.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.576+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1513198423}) 2024-09-11T01:47:31.579 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.576+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.579 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.576+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1553006702}) 2024-09-11T01:47:31.579 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.576+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=719359181}) 2024-09-11T01:47:31.579 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.577+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.579 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.577+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3784013144}) 2024-09-11T01:47:31.580 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.577+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2273030030}) 2024-09-11T01:47:31.580 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.577+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.580 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.577+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3784013144}) 2024-09-11T01:47:31.580 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.577+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=693051129}) 2024-09-11T01:47:31.580 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.578+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.580 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.578+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2882622981}) 2024-09-11T01:47:31.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.578+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1893402366}) 2024-09-11T01:47:31.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.578+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.578+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2882622981}) 2024-09-11T01:47:31.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.578+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4041104004}) 2024-09-11T01:47:31.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.580+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.580+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4220037568}) 2024-09-11T01:47:31.582 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.580+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=778830171}) 2024-09-11T01:47:31.582 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.580+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.582 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.580+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4220037568}) 2024-09-11T01:47:31.582 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.580+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1560008670}) 2024-09-11T01:47:31.582 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.581+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.582 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.581+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=95481510}) 2024-09-11T01:47:31.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.581+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=852888540}) 2024-09-11T01:47:31.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.581+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.581+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=95481510}) 2024-09-11T01:47:31.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.581+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2530371276}) 2024-09-11T01:47:31.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.582+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.582+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2320844249}) 2024-09-11T01:47:31.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.582+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1561883209}) 2024-09-11T01:47:31.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.582+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.582+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2320844249}) 2024-09-11T01:47:31.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.582+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1715442146}) 2024-09-11T01:47:31.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.583+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.583+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3469691103}) 2024-09-11T01:47:31.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.583+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1587246548}) 2024-09-11T01:47:31.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.583+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.583+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3469691103}) 2024-09-11T01:47:31.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.583+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1712519937}) 2024-09-11T01:47:31.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.584+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.586 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.584+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1080297011}) 2024-09-11T01:47:31.586 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.584+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2463304091}) 2024-09-11T01:47:31.586 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.584+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.586 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.584+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1080297011}) 2024-09-11T01:47:31.586 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.584+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2633069606}) 2024-09-11T01:47:31.701 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.699+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.701 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.699+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=265594328}) 2024-09-11T01:47:31.701 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.699+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1625430046}) 2024-09-11T01:47:31.701 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.699+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:31.701 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.699+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=265594328}) 2024-09-11T01:47:31.701 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:31.699+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3800748434}) 2024-09-11T01:47:32.137 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.135+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.137 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.135+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2963461270}) 2024-09-11T01:47:32.137 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.135+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=512498008}) 2024-09-11T01:47:32.137 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.135+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.137 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.135+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2963461270}) 2024-09-11T01:47:32.137 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.135+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3114460779}) 2024-09-11T01:47:32.141 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.139+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.141 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.139+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2957035321}) 2024-09-11T01:47:32.141 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.139+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1582394303}) 2024-09-11T01:47:32.141 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.139+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.141 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.139+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2957035321}) 2024-09-11T01:47:32.141 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.139+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1986449849}) 2024-09-11T01:47:32.149 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:47:32.149 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:47:32.150 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.148+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.151 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.148+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1157821443}) 2024-09-11T01:47:32.151 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.148+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3981981221}) 2024-09-11T01:47:32.151 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.148+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.151 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.148+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1157821443}) 2024-09-11T01:47:32.151 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.148+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=301268170}) 2024-09-11T01:47:32.153 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.151+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.153 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.151+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3028664676}) 2024-09-11T01:47:32.153 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.151+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3666220687}) 2024-09-11T01:47:32.154 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.151+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.154 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.151+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3028664676}) 2024-09-11T01:47:32.154 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.151+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1044746535}) 2024-09-11T01:47:32.155 INFO:tasks.rados.rados.0.smithi071.stdout:2757: done (1 left) 2024-09-11T01:47:32.156 INFO:tasks.rados.rados.0.smithi071.stdout:2758: done (0 left) 2024-09-11T01:47:32.156 INFO:tasks.rados.rados.0.smithi071.stdout:2759: read oid 45 snap -1 2024-09-11T01:47:32.156 INFO:tasks.rados.rados.0.smithi071.stdout:2759: expect (ObjNum 932 snap 293 seq_num 932) 2024-09-11T01:47:32.156 INFO:tasks.rados.rados.0.smithi071.stdout:2760: read oid 23 snap -1 2024-09-11T01:47:32.156 INFO:tasks.rados.rados.0.smithi071.stdout:2760: expect (ObjNum 689 snap 204 seq_num 689) 2024-09-11T01:47:32.156 INFO:tasks.rados.rados.0.smithi071.stdout:2761: rollback oid 5 current snap is 309 2024-09-11T01:47:32.156 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 5 to 274 2024-09-11T01:47:32.156 INFO:tasks.rados.rados.0.smithi071.stdout:2762: read oid 35 snap -1 2024-09-11T01:47:32.156 INFO:tasks.rados.rados.0.smithi071.stdout:2762: expect (ObjNum 981 snap 308 seq_num 981) 2024-09-11T01:47:32.156 INFO:tasks.rados.rados.0.smithi071.stdout:2763: snap_remove snap 308 2024-09-11T01:47:32.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.260+0000 7f9aefeef640 10 monclient: get_monmap_and_config 2024-09-11T01:47:32.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.260+0000 7f9aefeef640 10 monclient: build_initial_monmap 2024-09-11T01:47:32.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.260+0000 7f9aefeef640 10 monclient: monmap: 2024-09-11T01:47:32.262 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:47:32.262 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:32.262 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:47:32.261337+0000 2024-09-11T01:47:32.262 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:47:32.261337+0000 2024-09-11T01:47:32.262 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:47:32.262 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:32.262 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:47:32.263 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:47:32.263 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:47:32.263 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:47:32.263 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:47:32.263 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:32.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.261+0000 7f9aefeef640 1 Processor -- start 2024-09-11T01:47:32.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.261+0000 7f9aefeef640 1 -- start start 2024-09-11T01:47:32.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.261+0000 7f9aefeef640 10 monclient: init 2024-09-11T01:47:32.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aefeef640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:32.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aefeef640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-11T01:47:32.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aefeef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9ae80a2e90 addr v1:172.21.15.12:6789/0 2024-09-11T01:47:32.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aefeef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9ae80a3270 addr v1:172.21.15.71:6789/0 2024-09-11T01:47:32.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aefeef640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9ae813e4a0 0x7f9ae8140890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:32.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aefeef640 10 monclient(hunting): picked mon.noname-a con 0x7f9ae813e4a0 addr v2:172.21.15.12:3300/0 2024-09-11T01:47:32.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aefeef640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ae80a46c0 con 0x7f9ae80a2e90 2024-09-11T01:47:32.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aefeef640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ae806ee50 con 0x7f9ae80a3270 2024-09-11T01:47:32.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aefeef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:32.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aefeef640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f9ae80a4900 con 0x7f9ae813e4a0 2024-09-11T01:47:32.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aefeef640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:32.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aefeef640 10 monclient(hunting): authenticate will time out at 1858.704423s 2024-09-11T01:47:32.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aedc8b640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9ae80a2e90 0x7f9ae806dc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36326/0 (socket says 172.21.15.12:36326) 2024-09-11T01:47:32.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aee48c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9ae813e4a0 0x7f9ae8140890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:32.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aedc8b640 1 -- 172.21.15.12:0/2209990789 learned_addr learned my addr 172.21.15.12:0/2209990789 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:47:32.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aee48c640 10 monclient(hunting): get_auth_request con 0x7f9ae813e4a0 auth_method 0 2024-09-11T01:47:32.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aee48c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:32.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aee48c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:32.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aee48c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:32.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aee48c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:32.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aee48c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:32.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aee48c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:32.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aee48c640 10 monclient(hunting): handle_auth_done global_id 204132 payload 274 2024-09-11T01:47:32.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aee48c640 1 -- 172.21.15.12:0/2209990789 >> v1:172.21.15.71:6789/0 conn(0x7f9ae80a3270 legacy=0x7f9ae80a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:47:32.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aee48c640 1 -- 172.21.15.12:0/2209990789 >> v1:172.21.15.12:6789/0 conn(0x7f9ae80a2e90 legacy=0x7f9ae806dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:47:32.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aee48c640 10 monclient: _finish_hunting 0 2024-09-11T01:47:32.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aee48c640 1 monclient: found mon.noname-a 2024-09-11T01:47:32.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aee48c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:32.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aee48c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:47:32.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.262+0000 7f9aee48c640 1 -- 172.21.15.12:0/2209990789 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ae806eb70 con 0x7f9ae813e4a0 2024-09-11T01:47:32.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.265+0000 7f9aee48c640 1 -- 172.21.15.12:0/2209990789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ae813e4a0 msgr2=0x7f9ae8140890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:32.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.265+0000 7f9aee48c640 1 --2- 172.21.15.12:0/2209990789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ae813e4a0 0x7f9ae8140890 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8071820 tx=0x7f9ae002a460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a17238f91e2e95c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:32.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.266+0000 7f9aecc89640 1 -- 172.21.15.12:0/2209990789 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae0007a30 con 0x7f9ae813e4a0 2024-09-11T01:47:32.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.266+0000 7f9aecc89640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:32.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.266+0000 7f9aecc89640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:47:32.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.266+0000 7f9aecc89640 10 monclient: dump: 2024-09-11T01:47:32.272 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:32.272 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:32.272 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.272 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.272 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:32.272 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:32.272 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:32.272 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:32.272 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:32.272 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:32.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.266+0000 7f9aecc89640 10 monclient: _finish_auth 0 2024-09-11T01:47:32.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.266+0000 7f9aecc89640 10 monclient: _check_auth_tickets 2024-09-11T01:47:32.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.266+0000 7f9aecc89640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:32.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.266+0000 7f9aecc89640 1 -- 172.21.15.12:0/2209990789 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ae0007bd0 con 0x7f9ae813e4a0 2024-09-11T01:47:32.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.266+0000 7f9aefeef640 5 monclient: authenticate success, global_id 204132 2024-09-11T01:47:32.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.266+0000 7f9aefeef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:47:32.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.266+0000 7f9aecc89640 10 monclient: handle_config config(0 keys) 2024-09-11T01:47:32.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.266+0000 7f9aefeef640 10 monclient: get_monmap_and_config success 2024-09-11T01:47:32.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.266+0000 7f9aecc89640 1 -- 172.21.15.12:0/2209990789 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae0031400 con 0x7f9ae813e4a0 2024-09-11T01:47:32.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.266+0000 7f9aefeef640 4 set_mon_vals no callback set 2024-09-11T01:47:32.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.266+0000 7f9aecc89640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:32.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.266+0000 7f9aecc89640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:47:32.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.266+0000 7f9aecc89640 10 monclient: dump: 2024-09-11T01:47:32.274 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:32.274 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:32.274 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.274 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.274 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:32.274 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:32.274 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:32.275 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:32.275 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:32.275 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:32.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.266+0000 7f9aefeef640 10 monclient: shutdown 2024-09-11T01:47:32.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.266+0000 7f9aefeef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:47:32.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.266+0000 7f9aefeef640 1 -- 172.21.15.12:0/2209990789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ae813e4a0 msgr2=0x7f9ae8140890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:32.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.266+0000 7f9aefeef640 1 --2- 172.21.15.12:0/2209990789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ae813e4a0 0x7f9ae8140890 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8071820 tx=0x7f9ae002a460 comp rx=0 tx=0).stop 2024-09-11T01:47:32.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.266+0000 7f9aefeef640 1 -- 172.21.15.12:0/2209990789 shutdown_connections 2024-09-11T01:47:32.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.266+0000 7f9aefeef640 1 --2- 172.21.15.12:0/2209990789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ae813e4a0 0x7f9ae8140890 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.266+0000 7f9aefeef640 1 -- 172.21.15.12:0/2209990789 >> 172.21.15.12:0/2209990789 conn(0x7f9ae80a8490 msgr2=0x7f9ae80a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:47:32.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.266+0000 7f9aefeef640 1 -- 172.21.15.12:0/2209990789 shutdown_connections 2024-09-11T01:47:32.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.266+0000 7f9aefeef640 1 -- 172.21.15.12:0/2209990789 wait complete. 2024-09-11T01:47:32.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.267+0000 7f9aefeef640 10 monclient: build_initial_monmap 2024-09-11T01:47:32.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.267+0000 7f9aefeef640 10 monclient: monmap: 2024-09-11T01:47:32.276 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:47:32.276 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:47:32.276 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:47:32.276 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:47:32.277 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:47:32.277 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:32.277 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:47:32.277 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:47:32.277 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:47:32.277 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:32.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.267+0000 7f9aefeef640 1 Processor -- start 2024-09-11T01:47:32.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.267+0000 7f9aefeef640 1 -- start start 2024-09-11T01:47:32.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.267+0000 7f9aefeef640 10 monclient: init 2024-09-11T01:47:32.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aefeef640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:32.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aefeef640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:47:32.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aefeef640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ae80a2e90 0x7f9ae813cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:32.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aefeef640 10 monclient(hunting): picked mon.noname-b con 0x7f9ae80a2e90 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:47:32.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aefeef640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ae80a3270 0x7f9ae8138590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:32.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aefeef640 10 monclient(hunting): picked mon.noname-a con 0x7f9ae80a3270 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:47:32.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aefeef640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ae813e4a0 0x7f9ae8138ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:32.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aefeef640 10 monclient(hunting): picked mon.noname-c con 0x7f9ae813e4a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:47:32.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aefeef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:32.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aefeef640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae8140f50 con 0x7f9ae80a3270 2024-09-11T01:47:32.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aefeef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:32.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aefeef640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae8140dd0 con 0x7f9ae80a2e90 2024-09-11T01:47:32.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aefeef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:32.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aefeef640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f9ae81410d0 con 0x7f9ae813e4a0 2024-09-11T01:47:32.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aefeef640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:32.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aefeef640 10 monclient(hunting): authenticate will time out at 1858.710226s 2024-09-11T01:47:32.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aed48a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ae80a3270 0x7f9ae8138590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:32.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aee48c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ae813e4a0 0x7f9ae8138ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:32.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aed48a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ae80a3270 0x7f9ae8138590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38530/0 (socket says 172.21.15.12:38530) 2024-09-11T01:47:32.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aed48a640 1 -- 172.21.15.12:0/1718814017 learned_addr learned my addr 172.21.15.12:0/1718814017 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:47:32.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aee48c640 1 --2- 172.21.15.12:0/1718814017 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ae813e4a0 0x7f9ae8138ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:33106/0 (socket says 172.21.15.12:33106) 2024-09-11T01:47:32.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aed48a640 10 monclient(hunting): get_auth_request con 0x7f9ae80a3270 auth_method 0 2024-09-11T01:47:32.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aed48a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:32.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aed48a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:32.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aed48a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:32.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aee48c640 10 monclient(hunting): get_auth_request con 0x7f9ae813e4a0 auth_method 0 2024-09-11T01:47:32.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aee48c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:32.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aee48c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:32.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aee48c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:32.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aed48a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:32.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aed48a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:32.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aed48a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:32.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aee48c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:32.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aee48c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:32.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aedc8b640 1 --2- 172.21.15.12:0/1718814017 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ae80a2e90 0x7f9ae813cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:32.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aee48c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:32.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aedc8b640 10 monclient(hunting): get_auth_request con 0x7f9ae80a2e90 auth_method 0 2024-09-11T01:47:32.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aedc8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:32.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aedc8b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:32.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aedc8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:32.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aed48a640 10 monclient(hunting): handle_auth_done global_id 204135 payload 824 2024-09-11T01:47:32.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aed48a640 1 -- 172.21.15.12:0/1718814017 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ae813e4a0 msgr2=0x7f9ae8138ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:32.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aed48a640 1 --2- 172.21.15.12:0/1718814017 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ae813e4a0 0x7f9ae8138ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aed48a640 1 -- 172.21.15.12:0/1718814017 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ae80a2e90 msgr2=0x7f9ae813cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:32.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aed48a640 1 --2- 172.21.15.12:0/1718814017 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ae80a2e90 0x7f9ae813cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aed48a640 10 monclient: _finish_hunting 0 2024-09-11T01:47:32.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aed48a640 1 monclient: found mon.noname-a 2024-09-11T01:47:32.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aed48a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:32.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aed48a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:47:32.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aed48a640 1 -- 172.21.15.12:0/1718814017 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ae809f000 con 0x7f9ae80a3270 2024-09-11T01:47:32.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.268+0000 7f9aedc8b640 1 --2- 172.21.15.12:0/1718814017 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ae80a2e90 0x7f9ae813cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:47:32.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.270+0000 7f9aed48a640 1 -- 172.21.15.12:0/1718814017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ae80a3270 msgr2=0x7f9ae8138590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:32.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.270+0000 7f9aed48a640 1 --2- 172.21.15.12:0/1718814017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ae80a3270 0x7f9ae8138590 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9ad800c430 tx=0x7f9ad800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:32.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.271+0000 7f9ade7fc640 1 -- 172.21.15.12:0/1718814017 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad80119a0 con 0x7f9ae80a3270 2024-09-11T01:47:32.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.271+0000 7f9ade7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:32.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.271+0000 7f9ade7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:47:32.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.271+0000 7f9ade7fc640 10 monclient: dump: 2024-09-11T01:47:32.284 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:32.284 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:32.285 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.285 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.285 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:32.285 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:32.285 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:32.285 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:32.285 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:32.285 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:32.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.271+0000 7f9ade7fc640 10 monclient: _finish_auth 0 2024-09-11T01:47:32.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.271+0000 7f9ade7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:47:32.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.271+0000 7f9ade7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:32.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.271+0000 7f9ade7fc640 1 -- 172.21.15.12:0/1718814017 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ad8017070 con 0x7f9ae80a3270 2024-09-11T01:47:32.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.271+0000 7f9ade7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:47:32.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.271+0000 7f9ade7fc640 1 -- 172.21.15.12:0/1718814017 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad8011d80 con 0x7f9ae80a3270 2024-09-11T01:47:32.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.271+0000 7f9aefeef640 5 monclient: authenticate success, global_id 204135 2024-09-11T01:47:32.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.271+0000 7f9ade7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:32.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.271+0000 7f9ade7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:47:32.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.271+0000 7f9ade7fc640 10 monclient: dump: 2024-09-11T01:47:32.286 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:32.286 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:32.286 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.287 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.287 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:32.287 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:32.287 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:32.287 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:32.287 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:32.287 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:32.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.271+0000 7f9aefeef640 10 monclient: _renew_subs 2024-09-11T01:47:32.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.271+0000 7f9aefeef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:47:32.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.271+0000 7f9adffff640 4 set_mon_vals no callback set 2024-09-11T01:47:32.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.271+0000 7f9aefeef640 1 -- 172.21.15.12:0/1718814017 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ae8139010 con 0x7f9ae80a3270 2024-09-11T01:47:32.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.271+0000 7f9aefeef640 10 monclient: _renew_subs 2024-09-11T01:47:32.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.271+0000 7f9aefeef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:47:32.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.271+0000 7f9aefeef640 1 -- 172.21.15.12:0/1718814017 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ae813bcb0 con 0x7f9ae80a3270 2024-09-11T01:47:32.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.271+0000 7f9ade7fc640 1 -- 172.21.15.12:0/1718814017 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f9ad801c460 con 0x7f9ae80a3270 2024-09-11T01:47:32.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.272+0000 7f9aefeef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:32.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.272+0000 7f9aefeef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:32.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.272+0000 7f9aefeef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:47:32.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.272+0000 7f9aefeef640 1 -- 172.21.15.12:0/1718814017 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ae80a5e40 con 0x7f9ae80a3270 2024-09-11T01:47:32.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.272+0000 7f9ade7fc640 1 --2- 172.21.15.12:0/1718814017 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9acc03e240 0x7f9acc040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:32.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.272+0000 7f9ade7fc640 1 -- 172.21.15.12:0/1718814017 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(619..619 src has 507..619) ==== 4803+0+0 (secure 0 0 0) 0x7f9ad8052b70 con 0x7f9ae80a3270 2024-09-11T01:47:32.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.272+0000 7f9aedc8b640 1 --2- 172.21.15.12:0/1718814017 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9acc03e240 0x7f9acc040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:32.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.272+0000 7f9aedc8b640 10 monclient: get_auth_request con 0x7f9acc03e240 auth_method 0 2024-09-11T01:47:32.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.275+0000 7f9aedc8b640 1 --2- 172.21.15.12:0/1718814017 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9acc03e240 0x7f9acc040700 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4007c90 tx=0x7f9ae4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:32.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.277+0000 7f9ade7fc640 1 -- 172.21.15.12:0/1718814017 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9ad8012420 con 0x7f9ae80a3270 2024-09-11T01:47:32.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.277+0000 7f9ade7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:32.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.277+0000 7f9ade7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:47:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.531+0000 7f9aefeef640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:47:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.531+0000 7f9aefeef640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:47:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.531+0000 7f9aefeef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:47:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.532+0000 7f9aefeef640 1 -- 172.21.15.12:0/1718814017 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9ae806dc60 con 0x7f9ae80a3270 2024-09-11T01:47:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.532+0000 7f9ade7fc640 1 -- 172.21.15.12:0/1718814017 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f9ad8051550 con 0x7f9ae80a3270 2024-09-11T01:47:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.532+0000 7f9ade7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:47:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.532+0000 7f9ade7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:47:32.534 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":172,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:47:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.534+0000 7f9aefeef640 1 -- 172.21.15.12:0/1718814017 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9acc03e240 msgr2=0x7f9acc040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.534+0000 7f9aefeef640 1 --2- 172.21.15.12:0/1718814017 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9acc03e240 0x7f9acc040700 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4007c90 tx=0x7f9ae4009040 comp rx=0 tx=0).stop 2024-09-11T01:47:32.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.534+0000 7f9aefeef640 10 monclient: shutdown 2024-09-11T01:47:32.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.534+0000 7f9aefeef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:47:32.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.534+0000 7f9aefeef640 1 -- 172.21.15.12:0/1718814017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ae80a3270 msgr2=0x7f9ae8138590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:32.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.534+0000 7f9aefeef640 1 --2- 172.21.15.12:0/1718814017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ae80a3270 0x7f9ae8138590 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9ad800c430 tx=0x7f9ad800c8f0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.534+0000 7f9aefeef640 1 -- 172.21.15.12:0/1718814017 shutdown_connections 2024-09-11T01:47:32.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.534+0000 7f9aefeef640 1 --2- 172.21.15.12:0/1718814017 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ae813e4a0 0x7f9ae8138ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.534+0000 7f9aefeef640 1 --2- 172.21.15.12:0/1718814017 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9acc03e240 0x7f9acc040700 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.534+0000 7f9aefeef640 1 --2- 172.21.15.12:0/1718814017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ae80a3270 0x7f9ae8138590 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.534+0000 7f9aefeef640 1 --2- 172.21.15.12:0/1718814017 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ae80a2e90 0x7f9ae813cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.534+0000 7f9aefeef640 1 -- 172.21.15.12:0/1718814017 >> 172.21.15.12:0/1718814017 conn(0x7f9ae80a8490 msgr2=0x7f9ae80a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:47:32.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.534+0000 7f9aefeef640 1 -- 172.21.15.12:0/1718814017 shutdown_connections 2024-09-11T01:47:32.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.534+0000 7f9aefeef640 1 -- 172.21.15.12:0/1718814017 wait complete. 2024-09-11T01:47:32.546 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:47:32.546 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:47:32.546 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:47:32.547 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:47:32.550 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.548+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.550 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.548+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4195000861}) 2024-09-11T01:47:32.550 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.548+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2285687303}) 2024-09-11T01:47:32.550 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.548+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.550 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.548+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4195000861}) 2024-09-11T01:47:32.550 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.548+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4154645960}) 2024-09-11T01:47:32.551 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.549+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.551 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.549+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3022076517}) 2024-09-11T01:47:32.551 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.549+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1205728089}) 2024-09-11T01:47:32.551 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.549+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.551 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.549+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3022076517}) 2024-09-11T01:47:32.552 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.549+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=134197693}) 2024-09-11T01:47:32.552 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.550+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.552 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.550+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=48241432}) 2024-09-11T01:47:32.552 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.550+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3363731753}) 2024-09-11T01:47:32.552 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.550+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.553 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.550+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=48241432}) 2024-09-11T01:47:32.553 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.550+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2672619120}) 2024-09-11T01:47:32.553 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.551+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.553 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.551+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3030167559}) 2024-09-11T01:47:32.553 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.551+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3007736330}) 2024-09-11T01:47:32.553 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.551+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.554 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.551+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3030167559}) 2024-09-11T01:47:32.554 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.551+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1488732945}) 2024-09-11T01:47:32.554 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.552+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.554 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.552+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3036828541}) 2024-09-11T01:47:32.555 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.552+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1692801185}) 2024-09-11T01:47:32.555 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.552+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.555 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.552+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3036828541}) 2024-09-11T01:47:32.555 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.553+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=827153455}) 2024-09-11T01:47:32.556 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.554+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.556 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.554+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2076905887}) 2024-09-11T01:47:32.556 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.554+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1665044734}) 2024-09-11T01:47:32.556 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.554+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.556 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.554+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2076905887}) 2024-09-11T01:47:32.556 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.554+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=566168626}) 2024-09-11T01:47:32.557 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.555+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.557 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.555+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589485659}) 2024-09-11T01:47:32.557 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.555+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2997683174}) 2024-09-11T01:47:32.557 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.555+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.557 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.555+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589485659}) 2024-09-11T01:47:32.557 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.555+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4234149704}) 2024-09-11T01:47:32.558 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.556+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.558 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.556+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3644154681}) 2024-09-11T01:47:32.558 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.556+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1382263802}) 2024-09-11T01:47:32.558 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.556+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.558 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.556+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3644154681}) 2024-09-11T01:47:32.558 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.556+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=477056623}) 2024-09-11T01:47:32.559 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.557+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.559 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.557+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=804179458}) 2024-09-11T01:47:32.559 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.557+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2544422301}) 2024-09-11T01:47:32.559 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.557+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.559 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.557+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=804179458}) 2024-09-11T01:47:32.559 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.557+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=509199232}) 2024-09-11T01:47:32.560 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.558+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.560 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.558+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=861361171}) 2024-09-11T01:47:32.560 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.558+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=796664856}) 2024-09-11T01:47:32.560 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.558+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.560 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.558+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=861361171}) 2024-09-11T01:47:32.560 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.558+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3957464505}) 2024-09-11T01:47:32.561 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.559+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.561 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.559+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311554131}) 2024-09-11T01:47:32.561 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.559+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3844562011}) 2024-09-11T01:47:32.561 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.559+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.561 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.559+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311554131}) 2024-09-11T01:47:32.561 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.559+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3004917093}) 2024-09-11T01:47:32.562 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.560+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.562 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.560+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2850425892}) 2024-09-11T01:47:32.562 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.560+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2870714904}) 2024-09-11T01:47:32.562 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.560+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.562 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.560+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2850425892}) 2024-09-11T01:47:32.562 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.560+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3587404299}) 2024-09-11T01:47:32.563 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.561+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.563 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.561+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4292088666}) 2024-09-11T01:47:32.563 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.561+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=123474590}) 2024-09-11T01:47:32.563 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.561+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.564 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.561+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4292088666}) 2024-09-11T01:47:32.564 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.561+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3056652720}) 2024-09-11T01:47:32.564 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.562+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.564 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.562+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=544173441}) 2024-09-11T01:47:32.564 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.562+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=325246229}) 2024-09-11T01:47:32.564 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.562+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.564 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.562+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=544173441}) 2024-09-11T01:47:32.564 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.562+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2021901488}) 2024-09-11T01:47:32.565 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.563+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.565 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.563+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1700477571}) 2024-09-11T01:47:32.565 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.563+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=206369094}) 2024-09-11T01:47:32.565 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.563+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.565 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.563+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1700477571}) 2024-09-11T01:47:32.566 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.563+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=389819144}) 2024-09-11T01:47:32.566 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.563+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.566 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.563+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4280888013}) 2024-09-11T01:47:32.566 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.563+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3253899298}) 2024-09-11T01:47:32.566 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.563+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.566 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.564+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4280888013}) 2024-09-11T01:47:32.566 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.564+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3032082369}) 2024-09-11T01:47:32.567 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.564+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.567 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.564+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2569340705}) 2024-09-11T01:47:32.567 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.564+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3790086988}) 2024-09-11T01:47:32.567 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.564+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.567 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.564+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2569340705}) 2024-09-11T01:47:32.567 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.564+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1642780117}) 2024-09-11T01:47:32.567 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.565+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.568 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.565+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1587866995}) 2024-09-11T01:47:32.568 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.565+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4044893653}) 2024-09-11T01:47:32.568 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.565+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.568 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.565+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1587866995}) 2024-09-11T01:47:32.568 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.565+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1645940331}) 2024-09-11T01:47:32.568 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.566+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.568 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.566+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1215141390}) 2024-09-11T01:47:32.569 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.566+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4180033080}) 2024-09-11T01:47:32.569 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.566+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.569 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.566+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1215141390}) 2024-09-11T01:47:32.569 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.566+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1664490154}) 2024-09-11T01:47:32.570 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.567+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.570 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.567+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3864290272}) 2024-09-11T01:47:32.570 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.567+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3053935792}) 2024-09-11T01:47:32.570 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.567+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.570 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.567+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3864290272}) 2024-09-11T01:47:32.570 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.567+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2373288932}) 2024-09-11T01:47:32.570 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.568+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.570 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.568+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=163424339}) 2024-09-11T01:47:32.570 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.568+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2596882915}) 2024-09-11T01:47:32.570 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.568+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.571 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.568+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=163424339}) 2024-09-11T01:47:32.571 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.568+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4250761680}) 2024-09-11T01:47:32.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.569+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.569+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=426231002}) 2024-09-11T01:47:32.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.569+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3244280110}) 2024-09-11T01:47:32.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.569+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.569+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=426231002}) 2024-09-11T01:47:32.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.569+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4219168862}) 2024-09-11T01:47:32.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.570+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.570+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4285250931}) 2024-09-11T01:47:32.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.570+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2101059852}) 2024-09-11T01:47:32.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.570+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.570+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4285250931}) 2024-09-11T01:47:32.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.570+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=440601529}) 2024-09-11T01:47:32.573 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.571+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.573 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.571+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1101245120}) 2024-09-11T01:47:32.574 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.571+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=61253562}) 2024-09-11T01:47:32.574 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.571+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.574 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.571+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1101245120}) 2024-09-11T01:47:32.574 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.571+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1407736353}) 2024-09-11T01:47:32.574 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.571+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.574 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.571+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3749399007}) 2024-09-11T01:47:32.574 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.571+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=961319783}) 2024-09-11T01:47:32.574 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.571+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.574 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.571+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3749399007}) 2024-09-11T01:47:32.574 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.571+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3933972372}) 2024-09-11T01:47:32.576 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.572+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.572+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=235391358}) 2024-09-11T01:47:32.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.572+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=105952436}) 2024-09-11T01:47:32.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.572+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.572+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=235391358}) 2024-09-11T01:47:32.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.572+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2876947455}) 2024-09-11T01:47:32.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.573+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.573+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1937221972}) 2024-09-11T01:47:32.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.573+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1343799511}) 2024-09-11T01:47:32.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.573+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.573+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1937221972}) 2024-09-11T01:47:32.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.573+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3297158783}) 2024-09-11T01:47:32.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.573+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.573+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2764364029}) 2024-09-11T01:47:32.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.573+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1925000119}) 2024-09-11T01:47:32.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.573+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.573+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2764364029}) 2024-09-11T01:47:32.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.573+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1084614434}) 2024-09-11T01:47:32.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.574+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.574+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1434736916}) 2024-09-11T01:47:32.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.574+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1360989318}) 2024-09-11T01:47:32.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.574+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.574+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1434736916}) 2024-09-11T01:47:32.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.574+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1423538259}) 2024-09-11T01:47:32.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.575+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.575+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=47,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=5,osd_snap=2} crc {logm=1694408290,mdsmap=3140514009,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=144757030,mon_config_key=4195014768,monmap=2933275274,nvmeofgw=598337686,osd_metadata=4228979228,osd_snap=4089070047}) 2024-09-11T01:47:32.578 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.575+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=45,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=5} crc {logm=1455863660,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2171431985,mon_config_key=3330371543,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2239697408}) 2024-09-11T01:47:32.579 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.575+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.579 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.575+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=47,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=5,osd_snap=2} crc {logm=1694408290,mdsmap=3140514009,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=144757030,mon_config_key=4195014768,monmap=2933275274,nvmeofgw=598337686,osd_metadata=4228979228,osd_snap=4089070047}) 2024-09-11T01:47:32.579 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.575+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=49,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=5} crc {logm=1632282099,mdsmap=3140514009,mgr=4080174641,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1614759922,mon_config_key=1752117452,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2239697408}) 2024-09-11T01:47:32.580 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.576+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.580 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.576+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=2963811323}) 2024-09-11T01:47:32.580 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.576+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=100} crc {osd_snap=374818126}) 2024-09-11T01:47:32.580 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.576+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.580 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.576+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=2963811323}) 2024-09-11T01:47:32.580 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.576+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=100} crc {osd_snap=557645153}) 2024-09-11T01:47:32.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.576+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.576+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=27,osdmap=73} crc {osd_snap=1364656806,osdmap=591808331}) 2024-09-11T01:47:32.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.576+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=29,osdmap=71} crc {osd_snap=402484204,osdmap=4212816806}) 2024-09-11T01:47:32.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.576+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.576+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=27,osdmap=73} crc {osd_snap=1364656806,osdmap=591808331}) 2024-09-11T01:47:32.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.576+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=29,osdmap=71} crc {osd_snap=402484204,osdmap=646393515}) 2024-09-11T01:47:32.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.578+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.578+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4059418525}) 2024-09-11T01:47:32.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.578+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2225470049}) 2024-09-11T01:47:32.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.578+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:47:32.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.578+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4059418525}) 2024-09-11T01:47:32.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:47:32.578+0000 7f02f71fb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2836135268}) 2024-09-11T01:47:32.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.650+0000 7fba2f100640 10 monclient: get_monmap_and_config 2024-09-11T01:47:32.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.650+0000 7fba2f100640 10 monclient: build_initial_monmap 2024-09-11T01:47:32.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.650+0000 7fba2f100640 10 monclient: monmap: 2024-09-11T01:47:32.652 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:47:32.652 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:32.652 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:47:32.651468+0000 2024-09-11T01:47:32.653 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:47:32.651468+0000 2024-09-11T01:47:32.653 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:47:32.653 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:32.653 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:47:32.653 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:47:32.653 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:47:32.653 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:47:32.653 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:47:32.653 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:32.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.651+0000 7fba2f100640 1 Processor -- start 2024-09-11T01:47:32.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.651+0000 7fba2f100640 1 -- start start 2024-09-11T01:47:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.651+0000 7fba2f100640 10 monclient: init 2024-09-11T01:47:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.651+0000 7fba2f100640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.651+0000 7fba2f100640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-11T01:47:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.651+0000 7fba2f100640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fba281381c0 0x7fba281385a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.651+0000 7fba2f100640 10 monclient(hunting): picked mon.noname-c con 0x7fba281381c0 addr v2:172.21.15.71:3300/0 2024-09-11T01:47:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.651+0000 7fba2f100640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fba280a6aa0 0x7fba2809cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2f100640 10 monclient(hunting): picked mon.noname-b con 0x7fba280a6aa0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:47:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2f100640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fba280a4f20 0x7fba280a5390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2f100640 10 monclient(hunting): picked mon.noname-a con 0x7fba280a4f20 addr v2:172.21.15.12:3300/0 2024-09-11T01:47:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2f100640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2f100640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fba2806fc90 con 0x7fba280a4f20 2024-09-11T01:47:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2f100640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:32.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2f100640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fba2809c5f0 con 0x7fba281381c0 2024-09-11T01:47:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2f100640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2f100640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fba2809c310 con 0x7fba280a6aa0 2024-09-11T01:47:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2f100640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2f100640 10 monclient(hunting): authenticate will time out at 1859.094060s 2024-09-11T01:47:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2d69d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fba280a4f20 0x7fba280a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba27fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fba280a6aa0 0x7fba2809cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2d69d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fba280a4f20 0x7fba280a5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38538/0 (socket says 172.21.15.12:38538) 2024-09-11T01:47:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2d69d640 1 -- 172.21.15.12:0/3875566807 learned_addr learned my addr 172.21.15.12:0/3875566807 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:47:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2d69d640 10 monclient(hunting): get_auth_request con 0x7fba280a4f20 auth_method 0 2024-09-11T01:47:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2d69d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba27fff640 1 --2- 172.21.15.12:0/3875566807 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fba280a6aa0 0x7fba2809cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:33108/0 (socket says 172.21.15.12:33108) 2024-09-11T01:47:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2d69d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:32.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2d69d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba27fff640 10 monclient(hunting): get_auth_request con 0x7fba280a6aa0 auth_method 0 2024-09-11T01:47:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba27fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba27fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2d69d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2d69d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2d69d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2d69d640 10 monclient(hunting): handle_auth_done global_id 204138 payload 274 2024-09-11T01:47:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2ce9c640 1 --2- 172.21.15.12:0/3875566807 >> v2:172.21.15.71:3300/0 conn(0x7fba281381c0 0x7fba281385a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2d69d640 1 -- 172.21.15.12:0/3875566807 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fba280a6aa0 msgr2=0x7fba2809cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:47:32.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2d69d640 1 --2- 172.21.15.12:0/3875566807 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fba280a6aa0 0x7fba2809cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2d69d640 1 -- 172.21.15.12:0/3875566807 >> v2:172.21.15.71:3300/0 conn(0x7fba281381c0 msgr2=0x7fba281385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:47:32.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2d69d640 1 --2- 172.21.15.12:0/3875566807 >> v2:172.21.15.71:3300/0 conn(0x7fba281381c0 0x7fba281385a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2d69d640 10 monclient: _finish_hunting 0 2024-09-11T01:47:32.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2d69d640 1 monclient: found mon.noname-a 2024-09-11T01:47:32.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2d69d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:32.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2d69d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:47:32.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2d69d640 1 -- 172.21.15.12:0/3875566807 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba2809c8d0 con 0x7fba280a4f20 2024-09-11T01:47:32.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2ce9c640 10 monclient: get_auth_request con 0x7fba281381c0 auth_method 0 2024-09-11T01:47:32.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba27fff640 1 --2- 172.21.15.12:0/3875566807 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fba280a6aa0 0x7fba2809cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:47:32.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.652+0000 7fba2ce9c640 1 --2- 172.21.15.12:0/3875566807 >> v2:172.21.15.71:3300/0 conn(0x7fba281381c0 0x7fba281385a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:47:32.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.655+0000 7fba2d69d640 1 -- 172.21.15.12:0/3875566807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba280a4f20 msgr2=0x7fba280a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:32.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.655+0000 7fba2d69d640 1 --2- 172.21.15.12:0/3875566807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba280a4f20 0x7fba280a5390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fba2806fab0 tx=0x7fba2002a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7ff2c9bc39a899d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:32.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.655+0000 7fba277fe640 1 -- 172.21.15.12:0/3875566807 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba20036070 con 0x7fba280a4f20 2024-09-11T01:47:32.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.655+0000 7fba277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:32.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.655+0000 7fba277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:47:32.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.655+0000 7fba277fe640 10 monclient: dump: 2024-09-11T01:47:32.667 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:32.667 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:32.667 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.667 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.667 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:32.667 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:32.667 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:32.667 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:32.667 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:32.667 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:32.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.655+0000 7fba277fe640 10 monclient: _finish_auth 0 2024-09-11T01:47:32.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.655+0000 7fba277fe640 10 monclient: _check_auth_tickets 2024-09-11T01:47:32.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.655+0000 7fba277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:32.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.655+0000 7fba277fe640 1 -- 172.21.15.12:0/3875566807 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba20007800 con 0x7fba280a4f20 2024-09-11T01:47:32.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.655+0000 7fba2f100640 5 monclient: authenticate success, global_id 204138 2024-09-11T01:47:32.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.655+0000 7fba2f100640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:47:32.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.655+0000 7fba277fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:47:32.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.655+0000 7fba2f100640 10 monclient: get_monmap_and_config success 2024-09-11T01:47:32.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.655+0000 7fba277fe640 1 -- 172.21.15.12:0/3875566807 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba20007bd0 con 0x7fba280a4f20 2024-09-11T01:47:32.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.655+0000 7fba2f100640 4 set_mon_vals no callback set 2024-09-11T01:47:32.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.655+0000 7fba277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:32.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.655+0000 7fba277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:47:32.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.655+0000 7fba277fe640 10 monclient: dump: 2024-09-11T01:47:32.668 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:32.668 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:32.668 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.669 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.669 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:32.669 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:32.669 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:32.669 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:32.669 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:32.669 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:32.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.655+0000 7fba2f100640 10 monclient: shutdown 2024-09-11T01:47:32.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.655+0000 7fba2f100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:47:32.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.655+0000 7fba2f100640 1 -- 172.21.15.12:0/3875566807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba280a4f20 msgr2=0x7fba280a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:32.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.655+0000 7fba2f100640 1 --2- 172.21.15.12:0/3875566807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba280a4f20 0x7fba280a5390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fba2806fab0 tx=0x7fba2002a640 comp rx=0 tx=0).stop 2024-09-11T01:47:32.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.656+0000 7fba2f100640 1 -- 172.21.15.12:0/3875566807 shutdown_connections 2024-09-11T01:47:32.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.656+0000 7fba2f100640 1 --2- 172.21.15.12:0/3875566807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba280a4f20 0x7fba280a5390 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.656+0000 7fba2f100640 1 --2- 172.21.15.12:0/3875566807 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fba280a6aa0 0x7fba2809cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.656+0000 7fba2f100640 1 --2- 172.21.15.12:0/3875566807 >> v2:172.21.15.71:3300/0 conn(0x7fba281381c0 0x7fba281385a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.656+0000 7fba2f100640 1 -- 172.21.15.12:0/3875566807 >> 172.21.15.12:0/3875566807 conn(0x7fba280a7440 msgr2=0x7fba280a7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:47:32.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.656+0000 7fba2f100640 1 -- 172.21.15.12:0/3875566807 shutdown_connections 2024-09-11T01:47:32.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.656+0000 7fba2f100640 1 -- 172.21.15.12:0/3875566807 wait complete. 2024-09-11T01:47:32.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.656+0000 7fba2f100640 10 monclient: build_initial_monmap 2024-09-11T01:47:32.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.656+0000 7fba2f100640 10 monclient: monmap: 2024-09-11T01:47:32.670 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:47:32.670 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:47:32.670 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:47:32.670 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:47:32.670 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:47:32.670 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:32.670 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:47:32.670 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:47:32.670 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:47:32.671 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:32.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.656+0000 7fba2f100640 1 Processor -- start 2024-09-11T01:47:32.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2f100640 1 -- start start 2024-09-11T01:47:32.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2f100640 10 monclient: init 2024-09-11T01:47:32.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2f100640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:32.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2f100640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:47:32.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2f100640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fba280a4f20 0x7fba28120b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:32.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2f100640 10 monclient(hunting): picked mon.noname-c con 0x7fba280a4f20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:47:32.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2f100640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba280a6aa0 0x7fba281210b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:32.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2f100640 10 monclient(hunting): picked mon.noname-b con 0x7fba280a6aa0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:47:32.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2f100640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba281381c0 0x7fba281215f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:32.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2f100640 10 monclient(hunting): picked mon.noname-a con 0x7fba281381c0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:47:32.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2f100640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:32.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2f100640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fba2809d690 con 0x7fba281381c0 2024-09-11T01:47:32.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2f100640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:32.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2f100640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fba2809d510 con 0x7fba280a6aa0 2024-09-11T01:47:32.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2f100640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:32.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2f100640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fba280a58d0 con 0x7fba280a4f20 2024-09-11T01:47:32.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2f100640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:32.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2f100640 10 monclient(hunting): authenticate will time out at 1859.099673s 2024-09-11T01:47:32.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2ce9c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fba280a4f20 0x7fba28120b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:32.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2d69d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba281381c0 0x7fba281215f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:32.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2ce9c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fba280a4f20 0x7fba28120b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:33116/0 (socket says 172.21.15.12:33116) 2024-09-11T01:47:32.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2ce9c640 1 -- 172.21.15.12:0/3917755973 learned_addr learned my addr 172.21.15.12:0/3917755973 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:47:32.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2ce9c640 10 monclient(hunting): get_auth_request con 0x7fba280a4f20 auth_method 0 2024-09-11T01:47:32.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2ce9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:32.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2ce9c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:32.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2ce9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:32.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2d69d640 10 monclient(hunting): get_auth_request con 0x7fba281381c0 auth_method 0 2024-09-11T01:47:32.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2d69d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:32.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2d69d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:32.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2d69d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:32.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2ce9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:32.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2ce9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:32.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2ce9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:32.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2d69d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:32.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2d69d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:32.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba2d69d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:32.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba27fff640 1 --2- 172.21.15.12:0/3917755973 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba280a6aa0 0x7fba281210b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:32.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba27fff640 10 monclient(hunting): get_auth_request con 0x7fba280a6aa0 auth_method 0 2024-09-11T01:47:32.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:32.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba27fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:32.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.657+0000 7fba27fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:32.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.658+0000 7fba2ce9c640 10 monclient(hunting): handle_auth_done global_id 184304 payload 824 2024-09-11T01:47:32.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.658+0000 7fba2ce9c640 1 -- 172.21.15.12:0/3917755973 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba280a6aa0 msgr2=0x7fba281210b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:32.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.658+0000 7fba2ce9c640 1 --2- 172.21.15.12:0/3917755973 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba280a6aa0 0x7fba281210b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.658+0000 7fba2ce9c640 1 -- 172.21.15.12:0/3917755973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba281381c0 msgr2=0x7fba281215f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:32.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.658+0000 7fba2ce9c640 1 --2- 172.21.15.12:0/3917755973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba281381c0 0x7fba281215f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.658+0000 7fba2ce9c640 10 monclient: _finish_hunting 0 2024-09-11T01:47:32.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.658+0000 7fba2ce9c640 1 monclient: found mon.noname-c 2024-09-11T01:47:32.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.658+0000 7fba2ce9c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:32.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.658+0000 7fba2ce9c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:47:32.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.658+0000 7fba2ce9c640 1 -- 172.21.15.12:0/3917755973 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba281364e0 con 0x7fba280a4f20 2024-09-11T01:47:32.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.658+0000 7fba27fff640 1 --2- 172.21.15.12:0/3917755973 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba280a6aa0 0x7fba281210b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:47:32.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.660+0000 7fba2ce9c640 1 -- 172.21.15.12:0/3917755973 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fba280a4f20 msgr2=0x7fba28120b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:32.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.660+0000 7fba2ce9c640 1 --2- 172.21.15.12:0/3917755973 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fba280a4f20 0x7fba28120b70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fba1c00c7f0 tx=0x7fba1c00cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:32.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.660+0000 7fba257fa640 1 -- 172.21.15.12:0/3917755973 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba1c015070 con 0x7fba280a4f20 2024-09-11T01:47:32.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.660+0000 7fba257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:32.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.660+0000 7fba257fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:47:32.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.660+0000 7fba257fa640 10 monclient: dump: 2024-09-11T01:47:32.675 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:32.675 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:32.675 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.675 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.675 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:32.675 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:32.675 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:32.675 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:32.676 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:32.676 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:32.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.660+0000 7fba257fa640 10 monclient: _finish_auth 0 2024-09-11T01:47:32.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.660+0000 7fba257fa640 10 monclient: _check_auth_tickets 2024-09-11T01:47:32.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.660+0000 7fba257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:32.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.660+0000 7fba257fa640 1 -- 172.21.15.12:0/3917755973 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba1c007c40 con 0x7fba280a4f20 2024-09-11T01:47:32.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.660+0000 7fba257fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:47:32.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.660+0000 7fba257fa640 1 -- 172.21.15.12:0/3917755973 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba1c00ce40 con 0x7fba280a4f20 2024-09-11T01:47:32.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.660+0000 7fba2f100640 5 monclient: authenticate success, global_id 184304 2024-09-11T01:47:32.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.660+0000 7fba2f100640 10 monclient: _renew_subs 2024-09-11T01:47:32.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.660+0000 7fba2f100640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:47:32.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.660+0000 7fba2f100640 1 -- 172.21.15.12:0/3917755973 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba280a1710 con 0x7fba280a4f20 2024-09-11T01:47:32.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.660+0000 7fba257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:32.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.660+0000 7fba257fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:47:32.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.660+0000 7fba257fa640 10 monclient: dump: 2024-09-11T01:47:32.677 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:32.677 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:32.677 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.677 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.677 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:32.677 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:32.677 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:32.677 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:32.677 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:32.677 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:32.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.660+0000 7fba26ffd640 4 set_mon_vals no callback set 2024-09-11T01:47:32.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.660+0000 7fba2f100640 10 monclient: _renew_subs 2024-09-11T01:47:32.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.660+0000 7fba2f100640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:47:32.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.660+0000 7fba2f100640 1 -- 172.21.15.12:0/3917755973 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fba281232c0 con 0x7fba280a4f20 2024-09-11T01:47:32.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.661+0000 7fba2f100640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:32.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.661+0000 7fba2f100640 1 --2- 172.21.15.12:0/3917755973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba281c0a50 0x7fba28135450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:32.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.661+0000 7fba2f100640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:32.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.661+0000 7fba2f100640 1 -- 172.21.15.12:0/3917755973 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fba2809cfd0 con 0x7fba281c0a50 2024-09-11T01:47:32.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.661+0000 7fba27fff640 1 --2- 172.21.15.12:0/3917755973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba281c0a50 0x7fba28135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:32.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.661+0000 7fba27fff640 10 monclient: get_auth_request con 0x7fba281c0a50 auth_method 0 2024-09-11T01:47:32.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.661+0000 7fba27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:32.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.661+0000 7fba27fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:32.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.661+0000 7fba27fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:32.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.661+0000 7fba2ce9c640 1 --2- 172.21.15.12:0/3917755973 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fba280a4f20 0x7fba28120b70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fba1c00c7f0 tx=0x7fba1c00cbf0 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7fba1c0113f0 mgrmap(e 5) 2024-09-11T01:47:32.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.661+0000 7fba257fa640 1 -- 172.21.15.12:0/3917755973 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fba1c0113f0 con 0x7fba280a4f20 2024-09-11T01:47:32.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.661+0000 7fba27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:32.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.661+0000 7fba27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:32.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.661+0000 7fba27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:32.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.661+0000 7fba27fff640 10 monclient(hunting): handle_auth_done global_id 204144 payload 824 2024-09-11T01:47:32.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.661+0000 7fba27fff640 1 -- 172.21.15.12:0/3917755973 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fba281229b0 con 0x7fba281c0a50 2024-09-11T01:47:32.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.661+0000 7fba257fa640 1 --2- 172.21.15.12:0/3917755973 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fba0c03e260 0x7fba0c040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:32.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.662+0000 7fba2d69d640 1 --2- 172.21.15.12:0/3917755973 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fba0c03e260 0x7fba0c040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:32.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.662+0000 7fba2d69d640 10 monclient: get_auth_request con 0x7fba0c03e260 auth_method 0 2024-09-11T01:47:32.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.664+0000 7fba27fff640 1 -- 172.21.15.12:0/3917755973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba281c0a50 msgr2=0x7fba28135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:32.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.664+0000 7fba27fff640 1 --2- 172.21.15.12:0/3917755973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba281c0a50 0x7fba28135450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fba28121d30 tx=0x7fba14011c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:32.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.664+0000 7fba257fa640 1 -- 172.21.15.12:0/3917755973 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba1400fc10 con 0x7fba281c0a50 2024-09-11T01:47:32.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.664+0000 7fba257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:32.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.664+0000 7fba257fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:47:32.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.664+0000 7fba257fa640 10 monclient: dump: 2024-09-11T01:47:32.679 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:32.680 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:32.680 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.680 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.680 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:32.680 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:32.680 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:32.680 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:32.680 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:32.680 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:32.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.664+0000 7fba257fa640 1 -- 172.21.15.12:0/3917755973 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fba1400fc10 con 0x7fba281c0a50 2024-09-11T01:47:32.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.664+0000 7fba257fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:32.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.664+0000 7fba257fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:47:32.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.664+0000 7fba257fa640 1 -- 172.21.15.12:0/3917755973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba281c0a50 msgr2=0x7fba28135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:32.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.664+0000 7fba257fa640 1 --2- 172.21.15.12:0/3917755973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba281c0a50 0x7fba28135450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fba28121d30 tx=0x7fba14011c90 comp rx=0 tx=0).stop 2024-09-11T01:47:32.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.664+0000 7fba257fa640 1 -- 172.21.15.12:0/3917755973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba281c0a50 msgr2=0x7fba28135450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:47:32.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.664+0000 7fba257fa640 1 --2- 172.21.15.12:0/3917755973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba281c0a50 0x7fba28135450 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fba28121d30 tx=0x7fba14011c90 comp rx=0 tx=0).stop 2024-09-11T01:47:32.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.664+0000 7fba2d69d640 1 --2- 172.21.15.12:0/3917755973 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fba0c03e260 0x7fba0c040720 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fba2809bb60 tx=0x7fba20009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:32.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.670+0000 7fba2f100640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:47:32.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.670+0000 7fba2f100640 1 --2- 172.21.15.12:0/3917755973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba28135990 0x7fba28135da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:32.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.670+0000 7fba2f100640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:32.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.670+0000 7fba2f100640 1 -- 172.21.15.12:0/3917755973 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fba281362e0 con 0x7fba28135990 2024-09-11T01:47:32.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.670+0000 7fba27fff640 1 --2- 172.21.15.12:0/3917755973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba28135990 0x7fba28135da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:32.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.670+0000 7fba27fff640 10 monclient: get_auth_request con 0x7fba28135990 auth_method 0 2024-09-11T01:47:32.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.670+0000 7fba27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:32.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.670+0000 7fba27fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:32.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.670+0000 7fba27fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:32.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.670+0000 7fba27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:32.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.670+0000 7fba27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:32.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.670+0000 7fba27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:32.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.670+0000 7fba27fff640 10 monclient(hunting): handle_auth_done global_id 204147 payload 824 2024-09-11T01:47:32.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.670+0000 7fba27fff640 1 -- 172.21.15.12:0/3917755973 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fba28123600 con 0x7fba28135990 2024-09-11T01:47:32.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.673+0000 7fba27fff640 1 -- 172.21.15.12:0/3917755973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba28135990 msgr2=0x7fba28135da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:32.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.673+0000 7fba27fff640 1 --2- 172.21.15.12:0/3917755973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba28135990 0x7fba28135da0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fba1401b000 tx=0x7fba140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:32.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.673+0000 7fba257fa640 1 -- 172.21.15.12:0/3917755973 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba14006250 con 0x7fba28135990 2024-09-11T01:47:32.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.673+0000 7fba257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:32.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.673+0000 7fba257fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:47:32.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.673+0000 7fba257fa640 10 monclient: dump: 2024-09-11T01:47:32.684 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:32.684 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:32.684 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.684 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.684 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:32.684 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:32.684 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:32.685 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:32.685 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:32.685 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:32.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.673+0000 7fba257fa640 1 -- 172.21.15.12:0/3917755973 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7fba14006250 con 0x7fba28135990 2024-09-11T01:47:32.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.673+0000 7fba257fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:47:32.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.673+0000 7fba257fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:47:32.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.673+0000 7fba257fa640 1 -- 172.21.15.12:0/3917755973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba28135990 msgr2=0x7fba28135da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:32.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.673+0000 7fba257fa640 1 --2- 172.21.15.12:0/3917755973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba28135990 0x7fba28135da0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fba1401b000 tx=0x7fba140046c0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.673+0000 7fba257fa640 1 -- 172.21.15.12:0/3917755973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba28135990 msgr2=0x7fba28135da0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:47:32.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.673+0000 7fba257fa640 1 --2- 172.21.15.12:0/3917755973 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba28135990 0x7fba28135da0 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fba1401b000 tx=0x7fba140046c0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.685 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:47:32.686 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:47:32.686 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:47:32.686 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:47:32.686 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 172, 2024-09-11T01:47:32.686 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 18298, 2024-09-11T01:47:32.686 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:47:32.686 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:47:32.686 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:47:32.686 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:47:32.686 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:32.686 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:47:32.686 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:47:32.686 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:47:32.687 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:47:32.687 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:47:32.687 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:47:32.687 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:47:32.687 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:47:32.687 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:47:32.687 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:47:32.687 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:47:32.687 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:47:32.687 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:47:32.687 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:47:32.687 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:47:32.687 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:32.687 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:47:32.688 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:47:32.688 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:47:32.688 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:47:32.688 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:47:32.688 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:47:32.688 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:47:32.688 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:47:32.688 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:47:32.688 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:47:32.688 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:47:32.689 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:47:32.689 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:47:32.689 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:32.689 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:32.689 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:47:32.689 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:47:32.689 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:47:32.689 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:47:32.689 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:47:32.689 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:47:32.689 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:47:32.689 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:47:32.689 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:47:32.689 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:47:32.690 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:47:32.690 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:47:32.690 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:47:32.690 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:47:32.690 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:47:32.690 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:47:32.690 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:47:32.690 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:47:32.690 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:47:32.690 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:47:32.690 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:47:32.690 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:47:32.690 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:47:32.691 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:47:32.691 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:47:32.691 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:47:32.691 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:47:32.691 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:47:32.691 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:32.691 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:47:32.691 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:32.691 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:47:32.691 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:32.691 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:47:32.691 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:47:32.691 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:47:32.692 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:47:32.692 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:32.692 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:47:32.692 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:47:32.692 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:32.692 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:32.692 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:32.692 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:47:32.692 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:47:32.692 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:32.692 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:32.692 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:32.692 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:32.692 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:47:32.693 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:47:32.693 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:47:32.693 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:47:32.693 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:47:32.693 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:32.693 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:32.693 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:47:32.693 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:47:32.693 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:47:32.693 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:47:32.693 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:32.693 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:47:32.693 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:47:32.694 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:32.694 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:32.694 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:32.694 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:47:32.694 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:47:32.694 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:32.694 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:32.694 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:32.694 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:32.694 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:47:32.694 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:47:32.694 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:47:32.694 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:47:32.695 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:47:32.695 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:32.695 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:32.695 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:47:32.695 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:47:32.695 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:47:32.695 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:47:32.695 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:32.695 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:47:32.695 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:47:32.695 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:32.695 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:32.695 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:32.695 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:47:32.696 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:47:32.696 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:32.696 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:32.696 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:32.696 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:32.696 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:47:32.696 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:47:32.696 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:47:32.696 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:47:32.696 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:47:32.696 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:32.696 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:32.696 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:32.696 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:47:32.697 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:47:32.697 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:32.697 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:47:32.697 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:47:32.697 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:47:32.697 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:32.697 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:32.697 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:47:32.697 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:32.697 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:47:32.697 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:47:32.697 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:47:32.697 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:32.697 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:32.698 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:32.698 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:47:32.698 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:47:32.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.673+0000 7fba2f100640 1 -- 172.21.15.12:0/3917755973 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fba0c03e260 msgr2=0x7fba0c040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:32.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.673+0000 7fba2f100640 1 --2- 172.21.15.12:0/3917755973 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fba0c03e260 0x7fba0c040720 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fba2809bb60 tx=0x7fba20009000 comp rx=0 tx=0).stop 2024-09-11T01:47:32.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.673+0000 7fba2f100640 10 monclient: shutdown 2024-09-11T01:47:32.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.673+0000 7fba2f100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:47:32.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.673+0000 7fba2f100640 1 -- 172.21.15.12:0/3917755973 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fba280a4f20 msgr2=0x7fba28120b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:32.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.673+0000 7fba2f100640 1 --2- 172.21.15.12:0/3917755973 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fba280a4f20 0x7fba28120b70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fba1c00c7f0 tx=0x7fba1c00cbf0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.673+0000 7fba2d69d640 1 -- 172.21.15.12:0/3917755973 reap_dead start 2024-09-11T01:47:32.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.673+0000 7fba257fa640 1 -- 172.21.15.12:0/3917755973 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(619..619 src has 507..619) ==== 4803+0+0 (unknown 0 0 0) 0x7fba1c0530e0 con 0x7fba280a4f20 2024-09-11T01:47:32.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.673+0000 7fba2d69d640 1 -- 172.21.15.12:0/3917755973 reap_dead start 2024-09-11T01:47:32.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.673+0000 7fba2f100640 1 -- 172.21.15.12:0/3917755973 shutdown_connections 2024-09-11T01:47:32.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.673+0000 7fba2f100640 1 -- 172.21.15.12:0/3917755973 >> 172.21.15.12:0/3917755973 conn(0x7fba280a7440 msgr2=0x7fba2809ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:47:32.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.674+0000 7fba2f100640 1 -- 172.21.15.12:0/3917755973 shutdown_connections 2024-09-11T01:47:32.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.674+0000 7fba2f100640 1 -- 172.21.15.12:0/3917755973 wait complete. 2024-09-11T01:47:32.700 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:47:32.700 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:47:32.701 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:47:32.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.839+0000 7fa698652640 10 monclient: get_monmap_and_config 2024-09-11T01:47:32.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.839+0000 7fa698652640 10 monclient: build_initial_monmap 2024-09-11T01:47:32.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.839+0000 7fa698652640 10 monclient: monmap: 2024-09-11T01:47:32.841 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:47:32.842 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:32.842 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:47:32.840564+0000 2024-09-11T01:47:32.842 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:47:32.840564+0000 2024-09-11T01:47:32.842 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:47:32.842 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:32.842 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:47:32.842 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:47:32.842 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:47:32.842 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:47:32.842 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:47:32.842 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:32.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.840+0000 7fa698652640 1 Processor -- start 2024-09-11T01:47:32.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.840+0000 7fa698652640 1 -- start start 2024-09-11T01:47:32.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.840+0000 7fa698652640 10 monclient: init 2024-09-11T01:47:32.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa698652640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:32.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa698652640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-11T01:47:32.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa698652640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa690135fc0 addr v1:172.21.15.12:6789/0 2024-09-11T01:47:32.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa698652640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa690074000 0x7fa690074410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:32.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa698652640 10 monclient(hunting): picked mon.noname-b con 0x7fa690074000 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:47:32.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa698652640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa69013ae40 0x7fa69013d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:32.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa698652640 10 monclient(hunting): picked mon.noname-a con 0x7fa69013ae40 addr v2:172.21.15.12:3300/0 2024-09-11T01:47:32.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa698652640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6900732d0 con 0x7fa690135fc0 2024-09-11T01:47:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa698652640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa698652640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fa690073890 con 0x7fa69013ae40 2024-09-11T01:47:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa698652640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa698652640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa690072000 con 0x7fa690074000 2024-09-11T01:47:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa698652640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa698652640 10 monclient(hunting): authenticate will time out at 1859.283373s 2024-09-11T01:47:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa696bef640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa69013ae40 0x7fa69013d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa695bed640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa690074000 0x7fa690074410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa6963ee640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa690135fc0 0x7fa6901363a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36332/0 (socket says 172.21.15.12:36332) 2024-09-11T01:47:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa6963ee640 1 -- 172.21.15.12:0/2929250745 learned_addr learned my addr 172.21.15.12:0/2929250745 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:47:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa695bed640 10 monclient(hunting): get_auth_request con 0x7fa690074000 auth_method 0 2024-09-11T01:47:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa695bed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa695bed640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:32.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa695bed640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa696bef640 10 monclient(hunting): get_auth_request con 0x7fa69013ae40 auth_method 0 2024-09-11T01:47:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa696bef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa696bef640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa696bef640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa695bed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa695bed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa695bed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa696bef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa696bef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa696bef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa695bed640 10 monclient(hunting): handle_auth_done global_id 184307 payload 274 2024-09-11T01:47:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa695bed640 1 -- 172.21.15.12:0/2929250745 >> v2:172.21.15.12:3300/0 conn(0x7fa69013ae40 msgr2=0x7fa69013d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:47:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa695bed640 1 --2- 172.21.15.12:0/2929250745 >> v2:172.21.15.12:3300/0 conn(0x7fa69013ae40 0x7fa69013d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa695bed640 1 -- 172.21.15.12:0/2929250745 >> v1:172.21.15.12:6789/0 conn(0x7fa690135fc0 legacy=0x7fa6901363a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:47:32.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa695bed640 10 monclient: _finish_hunting 0 2024-09-11T01:47:32.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa695bed640 1 monclient: found mon.noname-b 2024-09-11T01:47:32.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa695bed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:32.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa695bed640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:47:32.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa695bed640 1 -- 172.21.15.12:0/2929250745 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6900a4970 con 0x7fa690074000 2024-09-11T01:47:32.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.841+0000 7fa696bef640 1 --2- 172.21.15.12:0/2929250745 >> v2:172.21.15.12:3300/0 conn(0x7fa69013ae40 0x7fa69013d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:47:32.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.844+0000 7fa695bed640 1 -- 172.21.15.12:0/2929250745 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa690074000 msgr2=0x7fa690074410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:32.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.844+0000 7fa695bed640 1 --2- 172.21.15.12:0/2929250745 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa690074000 0x7fa690074410 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa6840097a0 tx=0x7fa684029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b2b87b5c107ca8b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:32.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.844+0000 7fa6953ec640 1 -- 172.21.15.12:0/2929250745 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa684037070 con 0x7fa690074000 2024-09-11T01:47:32.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.844+0000 7fa6953ec640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:32.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.844+0000 7fa6953ec640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:47:32.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.844+0000 7fa6953ec640 10 monclient: dump: 2024-09-11T01:47:32.854 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:32.854 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:32.854 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.854 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.854 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:32.854 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:32.854 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:32.854 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:32.854 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:32.854 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:32.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.844+0000 7fa6953ec640 10 monclient: _finish_auth 0 2024-09-11T01:47:32.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.844+0000 7fa6953ec640 10 monclient: _check_auth_tickets 2024-09-11T01:47:32.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.844+0000 7fa6953ec640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:32.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.844+0000 7fa6953ec640 1 -- 172.21.15.12:0/2929250745 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa684032440 con 0x7fa690074000 2024-09-11T01:47:32.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.844+0000 7fa698652640 5 monclient: authenticate success, global_id 184307 2024-09-11T01:47:32.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.844+0000 7fa698652640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:47:32.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.844+0000 7fa6953ec640 10 monclient: handle_config config(0 keys) 2024-09-11T01:47:32.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.844+0000 7fa698652640 10 monclient: get_monmap_and_config success 2024-09-11T01:47:32.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.844+0000 7fa6953ec640 1 -- 172.21.15.12:0/2929250745 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa684032810 con 0x7fa690074000 2024-09-11T01:47:32.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.844+0000 7fa6953ec640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:32.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.844+0000 7fa698652640 4 set_mon_vals no callback set 2024-09-11T01:47:32.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.844+0000 7fa6953ec640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:47:32.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.844+0000 7fa6953ec640 10 monclient: dump: 2024-09-11T01:47:32.855 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:32.856 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:32.856 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.856 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.856 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:32.856 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:32.856 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:32.856 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:32.856 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:32.856 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:32.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.845+0000 7fa698652640 10 monclient: shutdown 2024-09-11T01:47:32.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.845+0000 7fa698652640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:47:32.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.845+0000 7fa698652640 1 -- 172.21.15.12:0/2929250745 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa690074000 msgr2=0x7fa690074410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:32.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.845+0000 7fa698652640 1 --2- 172.21.15.12:0/2929250745 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa690074000 0x7fa690074410 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa6840097a0 tx=0x7fa684029fd0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.845+0000 7fa698652640 1 -- 172.21.15.12:0/2929250745 shutdown_connections 2024-09-11T01:47:32.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.845+0000 7fa698652640 1 --2- 172.21.15.12:0/2929250745 >> v2:172.21.15.12:3300/0 conn(0x7fa69013ae40 0x7fa69013d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.845+0000 7fa698652640 1 --2- 172.21.15.12:0/2929250745 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa690074000 0x7fa690074410 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.845+0000 7fa698652640 1 -- 172.21.15.12:0/2929250745 >> 172.21.15.12:0/2929250745 conn(0x7fa6900a8600 msgr2=0x7fa6900a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:47:32.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.845+0000 7fa698652640 1 -- 172.21.15.12:0/2929250745 shutdown_connections 2024-09-11T01:47:32.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.845+0000 7fa698652640 1 -- 172.21.15.12:0/2929250745 wait complete. 2024-09-11T01:47:32.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.846+0000 7fa698652640 10 monclient: build_initial_monmap 2024-09-11T01:47:32.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.846+0000 7fa698652640 10 monclient: monmap: 2024-09-11T01:47:32.857 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:47:32.857 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:47:32.857 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:47:32.857 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:47:32.857 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:47:32.858 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:32.858 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:47:32.858 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:47:32.858 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:47:32.858 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:32.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.846+0000 7fa698652640 1 Processor -- start 2024-09-11T01:47:32.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.846+0000 7fa698652640 1 -- start start 2024-09-11T01:47:32.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.846+0000 7fa698652640 10 monclient: init 2024-09-11T01:47:32.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa698652640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:32.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa698652640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:47:32.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa698652640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa690074000 0x7fa6901cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:32.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa698652640 10 monclient(hunting): picked mon.noname-c con 0x7fa690074000 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:47:32.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa698652640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa690135fc0 0x7fa690134c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:32.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa698652640 10 monclient(hunting): picked mon.noname-a con 0x7fa690135fc0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:47:32.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa698652640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa69013ae40 0x7fa690137bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:32.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa698652640 10 monclient(hunting): picked mon.noname-b con 0x7fa69013ae40 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:47:32.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa698652640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:32.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa698652640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa69013d930 con 0x7fa690135fc0 2024-09-11T01:47:32.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa698652640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:32.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa698652640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa69013d7b0 con 0x7fa69013ae40 2024-09-11T01:47:32.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa698652640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:32.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa698652640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa69013dab0 con 0x7fa690074000 2024-09-11T01:47:32.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa698652640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:32.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa698652640 10 monclient(hunting): authenticate will time out at 1859.289129s 2024-09-11T01:47:32.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa6963ee640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa690074000 0x7fa6901cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:32.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa695bed640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa690135fc0 0x7fa690134c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:32.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa6963ee640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa690074000 0x7fa6901cad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:33134/0 (socket says 172.21.15.12:33134) 2024-09-11T01:47:32.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa695bed640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa690135fc0 0x7fa690134c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38582/0 (socket says 172.21.15.12:38582) 2024-09-11T01:47:32.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa6963ee640 1 -- 172.21.15.12:0/3949331230 learned_addr learned my addr 172.21.15.12:0/3949331230 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:47:32.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa6963ee640 10 monclient(hunting): get_auth_request con 0x7fa690074000 auth_method 0 2024-09-11T01:47:32.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa6963ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:32.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa6963ee640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:32.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa6963ee640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:32.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa695bed640 10 monclient(hunting): get_auth_request con 0x7fa690135fc0 auth_method 0 2024-09-11T01:47:32.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa695bed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:32.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa695bed640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:32.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa695bed640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:32.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa6963ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:32.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa6963ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:32.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa6963ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:32.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa695bed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:32.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa695bed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:32.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa695bed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:32.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa6963ee640 10 monclient(hunting): handle_auth_done global_id 184310 payload 824 2024-09-11T01:47:32.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa6963ee640 1 -- 172.21.15.12:0/3949331230 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa69013ae40 msgr2=0x7fa690137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:32.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa696bef640 1 --2- 172.21.15.12:0/3949331230 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa69013ae40 0x7fa690137bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:32.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa6963ee640 1 --2- 172.21.15.12:0/3949331230 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa69013ae40 0x7fa690137bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa6963ee640 1 -- 172.21.15.12:0/3949331230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa690135fc0 msgr2=0x7fa690134c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:32.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa6963ee640 1 --2- 172.21.15.12:0/3949331230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa690135fc0 0x7fa690134c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa6963ee640 10 monclient: _finish_hunting 0 2024-09-11T01:47:32.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa6963ee640 1 monclient: found mon.noname-c 2024-09-11T01:47:32.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa6963ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:32.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa6963ee640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:47:32.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa6963ee640 1 -- 172.21.15.12:0/3949331230 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6901cbc70 con 0x7fa690074000 2024-09-11T01:47:32.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa696bef640 10 monclient: get_auth_request con 0x7fa69013ae40 auth_method 0 2024-09-11T01:47:32.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa695bed640 1 --2- 172.21.15.12:0/3949331230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa690135fc0 0x7fa690134c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:47:32.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.847+0000 7fa696bef640 1 --2- 172.21.15.12:0/3949331230 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa69013ae40 0x7fa690137bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:47:32.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.849+0000 7fa6963ee640 1 -- 172.21.15.12:0/3949331230 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa690074000 msgr2=0x7fa6901cad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:32.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.849+0000 7fa6963ee640 1 --2- 172.21.15.12:0/3949331230 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa690074000 0x7fa6901cad50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa68000c460 tx=0x7fa68000c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:32.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.849+0000 7fa68effd640 1 -- 172.21.15.12:0/3949331230 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa680007070 con 0x7fa690074000 2024-09-11T01:47:32.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.849+0000 7fa68effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:32.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.849+0000 7fa68effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:47:32.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.849+0000 7fa68effd640 10 monclient: dump: 2024-09-11T01:47:32.863 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:32.863 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:32.863 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.863 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.863 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:32.863 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:32.863 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:32.863 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:32.863 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:32.863 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:32.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa68effd640 10 monclient: _finish_auth 0 2024-09-11T01:47:32.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa68effd640 10 monclient: _check_auth_tickets 2024-09-11T01:47:32.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa68effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:32.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa68effd640 1 -- 172.21.15.12:0/3949331230 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa680007230 con 0x7fa690074000 2024-09-11T01:47:32.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa698652640 5 monclient: authenticate success, global_id 184310 2024-09-11T01:47:32.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa698652640 10 monclient: _renew_subs 2024-09-11T01:47:32.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa698652640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:47:32.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa698652640 1 -- 172.21.15.12:0/3949331230 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa69011f330 con 0x7fa690074000 2024-09-11T01:47:32.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa68effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:47:32.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa68effd640 1 -- 172.21.15.12:0/3949331230 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa680007620 con 0x7fa690074000 2024-09-11T01:47:32.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa68effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:32.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa68effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:47:32.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa68effd640 10 monclient: dump: 2024-09-11T01:47:32.864 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:32.864 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:32.864 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.865 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.865 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:32.865 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:32.865 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:32.865 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:32.865 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:32.865 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:32.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa698652640 10 monclient: _renew_subs 2024-09-11T01:47:32.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa698652640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:47:32.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa698652640 1 -- 172.21.15.12:0/3949331230 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6900a3940 con 0x7fa690074000 2024-09-11T01:47:32.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa694beb640 4 set_mon_vals no callback set 2024-09-11T01:47:32.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa698652640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:32.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa698652640 1 --2- 172.21.15.12:0/3949331230 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa6901351c0 0x7fa690138110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:32.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa698652640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:32.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa698652640 1 -- 172.21.15.12:0/3949331230 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa69006a060 con 0x7fa6901351c0 2024-09-11T01:47:32.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa695bed640 1 --2- 172.21.15.12:0/3949331230 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa6901351c0 0x7fa690138110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:32.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa695bed640 10 monclient: get_auth_request con 0x7fa6901351c0 auth_method 0 2024-09-11T01:47:32.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa695bed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:32.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa695bed640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:32.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa695bed640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:32.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa695bed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:32.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.850+0000 7fa695bed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:32.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.851+0000 7fa695bed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:32.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.851+0000 7fa68effd640 1 -- 172.21.15.12:0/3949331230 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fa68001b400 con 0x7fa690074000 2024-09-11T01:47:32.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.851+0000 7fa695bed640 10 monclient(hunting): handle_auth_done global_id 184313 payload 824 2024-09-11T01:47:32.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.851+0000 7fa695bed640 1 -- 172.21.15.12:0/3949331230 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa69011ea70 con 0x7fa6901351c0 2024-09-11T01:47:32.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.851+0000 7fa68effd640 1 --2- 172.21.15.12:0/3949331230 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa67c03e260 0x7fa67c040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:32.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.851+0000 7fa68effd640 1 -- 172.21.15.12:0/3949331230 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(619..619 src has 507..619) ==== 4803+0+0 (secure 0 0 0) 0x7fa6800529a0 con 0x7fa690074000 2024-09-11T01:47:32.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.851+0000 7fa696bef640 1 --2- 172.21.15.12:0/3949331230 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa67c03e260 0x7fa67c040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:32.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.851+0000 7fa696bef640 10 monclient: get_auth_request con 0x7fa67c03e260 auth_method 0 2024-09-11T01:47:32.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.853+0000 7fa695bed640 1 -- 172.21.15.12:0/3949331230 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa6901351c0 msgr2=0x7fa690138110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:32.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.853+0000 7fa695bed640 1 --2- 172.21.15.12:0/3949331230 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa6901351c0 0x7fa690138110 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa684034000 tx=0x7fa6840046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:32.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.853+0000 7fa68effd640 1 -- 172.21.15.12:0/3949331230 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa684043070 con 0x7fa6901351c0 2024-09-11T01:47:32.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.853+0000 7fa68effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:32.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.853+0000 7fa68effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:47:32.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.853+0000 7fa68effd640 10 monclient: dump: 2024-09-11T01:47:32.870 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:32.870 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:32.870 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.870 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.870 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:32.870 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:32.871 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:32.871 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:32.871 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:32.871 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:32.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.854+0000 7fa68effd640 1 -- 172.21.15.12:0/3949331230 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa684038040 con 0x7fa6901351c0 2024-09-11T01:47:32.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.854+0000 7fa68effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:32.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.854+0000 7fa68effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:47:32.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.854+0000 7fa68effd640 1 -- 172.21.15.12:0/3949331230 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa6901351c0 msgr2=0x7fa690138110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:32.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.854+0000 7fa68effd640 1 --2- 172.21.15.12:0/3949331230 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa6901351c0 0x7fa690138110 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa684034000 tx=0x7fa6840046c0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.854+0000 7fa68effd640 1 -- 172.21.15.12:0/3949331230 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa6901351c0 msgr2=0x7fa690138110 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:47:32.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.854+0000 7fa68effd640 1 --2- 172.21.15.12:0/3949331230 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa6901351c0 0x7fa690138110 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa684034000 tx=0x7fa6840046c0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.854+0000 7fa696bef640 1 --2- 172.21.15.12:0/3949331230 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa67c03e260 0x7fa67c040720 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa690074cd0 tx=0x7fa68800b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:32.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.860+0000 7fa698652640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:47:32.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.860+0000 7fa698652640 1 --2- 172.21.15.12:0/3949331230 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa690138650 0x7fa69009c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:32.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.860+0000 7fa698652640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:32.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.860+0000 7fa698652640 1 -- 172.21.15.12:0/3949331230 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa690138a30 con 0x7fa690138650 2024-09-11T01:47:32.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.860+0000 7fa695bed640 1 --2- 172.21.15.12:0/3949331230 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa690138650 0x7fa69009c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:32.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.860+0000 7fa695bed640 10 monclient: get_auth_request con 0x7fa690138650 auth_method 0 2024-09-11T01:47:32.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.860+0000 7fa695bed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:32.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.860+0000 7fa695bed640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:32.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.860+0000 7fa695bed640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:32.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.860+0000 7fa695bed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:32.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.860+0000 7fa695bed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:32.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.860+0000 7fa695bed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:32.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.860+0000 7fa695bed640 10 monclient(hunting): handle_auth_done global_id 184316 payload 824 2024-09-11T01:47:32.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.860+0000 7fa695bed640 1 -- 172.21.15.12:0/3949331230 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa69009f610 con 0x7fa690138650 2024-09-11T01:47:32.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.862+0000 7fa695bed640 1 -- 172.21.15.12:0/3949331230 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa690138650 msgr2=0x7fa69009c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:32.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.862+0000 7fa695bed640 1 --2- 172.21.15.12:0/3949331230 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa690138650 0x7fa69009c320 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa68402a010 tx=0x7fa684008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:32.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.863+0000 7fa68effd640 1 -- 172.21.15.12:0/3949331230 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa684037040 con 0x7fa690138650 2024-09-11T01:47:32.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.863+0000 7fa68effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:32.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.863+0000 7fa68effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:47:32.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.863+0000 7fa68effd640 10 monclient: dump: 2024-09-11T01:47:32.876 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:32.876 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:32.876 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.876 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:32.876 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:32.877 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:32.877 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:32.877 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:32.877 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:32.877 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:32.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.863+0000 7fa68effd640 1 -- 172.21.15.12:0/3949331230 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4542 (secure 0 0 0) 0x7fa684003a30 con 0x7fa690138650 2024-09-11T01:47:32.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.863+0000 7fa68effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:47:32.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.863+0000 7fa68effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:47:32.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.863+0000 7fa68effd640 1 -- 172.21.15.12:0/3949331230 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa690138650 msgr2=0x7fa69009c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:32.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.863+0000 7fa68effd640 1 --2- 172.21.15.12:0/3949331230 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa690138650 0x7fa69009c320 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa68402a010 tx=0x7fa684008030 comp rx=0 tx=0).stop 2024-09-11T01:47:32.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.863+0000 7fa68effd640 1 -- 172.21.15.12:0/3949331230 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa690138650 msgr2=0x7fa69009c320 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:47:32.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.863+0000 7fa68effd640 1 --2- 172.21.15.12:0/3949331230 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa690138650 0x7fa69009c320 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa68402a010 tx=0x7fa684008030 comp rx=0 tx=0).stop 2024-09-11T01:47:32.879 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:47:32.879 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:47:32.879 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:47:32.879 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:47:32.880 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 172, 2024-09-11T01:47:32.880 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 113456, 2024-09-11T01:47:32.880 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:47:32.880 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:47:32.880 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:47:32.881 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:47:32.881 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:32.881 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:47:32.881 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:47:32.881 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:47:32.881 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:47:32.881 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:47:32.882 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:47:32.882 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:47:32.882 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:47:32.882 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:47:32.882 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:47:32.883 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:47:32.883 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:47:32.883 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:47:32.883 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:47:32.883 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:47:32.883 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:32.883 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:47:32.884 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:47:32.884 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:47:32.884 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:47:32.884 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:47:32.884 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:47:32.884 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:47:32.885 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:47:32.885 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:47:32.885 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:47:32.885 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:47:32.885 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:47:32.886 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:47:32.886 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:32.886 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:32.886 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:47:32.886 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:47:32.886 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:47:32.886 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:47:32.887 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:47:32.887 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:47:32.887 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:47:32.887 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:47:32.887 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:47:32.887 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:47:32.887 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:47:32.888 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:47:32.888 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:47:32.888 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:47:32.888 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:47:32.888 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:47:32.888 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:47:32.888 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:47:32.889 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:47:32.889 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:47:32.889 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:47:32.889 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:47:32.889 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:47:32.889 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:47:32.890 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:47:32.890 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:47:32.890 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:47:32.890 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:47:32.890 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:32.890 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:47:32.891 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:32.891 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:47:32.891 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:32.891 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:47:32.891 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:47:32.891 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:47:32.892 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:47:32.892 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:32.892 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:47:32.892 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:47:32.892 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:32.892 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:32.893 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:32.893 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:47:32.893 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:47:32.893 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:32.893 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:32.893 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:32.894 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:32.894 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:47:32.894 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:47:32.894 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:47:32.894 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:47:32.894 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:47:32.895 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:32.895 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:32.895 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:47:32.895 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:47:32.895 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:47:32.895 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:47:32.896 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:32.896 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:47:32.896 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:47:32.896 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:32.896 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:32.896 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:32.897 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:47:32.897 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:47:32.897 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:32.897 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:32.897 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:32.897 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:32.898 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:47:32.898 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:47:32.898 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:47:32.898 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:47:32.898 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:47:32.898 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:32.898 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:32.899 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:47:32.899 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:47:32.899 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:47:32.899 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:47:32.899 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:32.899 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:47:32.900 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:47:32.900 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:32.900 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:32.900 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:32.900 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:47:32.900 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:47:32.901 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:32.901 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:32.901 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:32.901 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:32.901 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:47:32.902 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:47:32.902 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:47:32.902 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:47:32.902 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:47:32.902 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:32.902 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:32.903 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:32.903 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:47:32.903 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:47:32.903 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:32.903 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:47:32.904 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:47:32.904 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:47:32.904 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:32.904 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:32.904 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:47:32.904 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:32.905 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:47:32.905 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:47:32.905 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:47:32.905 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:32.905 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:32.906 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:32.906 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:47:32.906 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:47:32.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.863+0000 7fa698652640 1 -- 172.21.15.12:0/3949331230 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa67c03e260 msgr2=0x7fa67c040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:32.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.863+0000 7fa698652640 1 --2- 172.21.15.12:0/3949331230 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa67c03e260 0x7fa67c040720 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa690074cd0 tx=0x7fa68800b040 comp rx=0 tx=0).stop 2024-09-11T01:47:32.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.863+0000 7fa698652640 10 monclient: shutdown 2024-09-11T01:47:32.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.863+0000 7fa698652640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:47:32.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.863+0000 7fa698652640 1 -- 172.21.15.12:0/3949331230 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa690074000 msgr2=0x7fa6901cad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:32.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.863+0000 7fa698652640 1 --2- 172.21.15.12:0/3949331230 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa690074000 0x7fa6901cad50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa68000c460 tx=0x7fa68000c920 comp rx=0 tx=0).stop 2024-09-11T01:47:32.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.863+0000 7fa696bef640 1 -- 172.21.15.12:0/3949331230 reap_dead start 2024-09-11T01:47:32.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.863+0000 7fa698652640 1 -- 172.21.15.12:0/3949331230 shutdown_connections 2024-09-11T01:47:32.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.863+0000 7fa698652640 1 --2- 172.21.15.12:0/3949331230 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa690074000 0x7fa6901cad50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:32.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.863+0000 7fa698652640 1 -- 172.21.15.12:0/3949331230 >> 172.21.15.12:0/3949331230 conn(0x7fa6900a8600 msgr2=0x7fa69013c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:47:32.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.863+0000 7fa698652640 1 -- 172.21.15.12:0/3949331230 shutdown_connections 2024-09-11T01:47:32.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:32.863+0000 7fa698652640 1 -- 172.21.15.12:0/3949331230 wait complete. 2024-09-11T01:47:32.911 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:47:32.911 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:47:32.911 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:47:33.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.011+0000 7f5ddcdec640 10 monclient: get_monmap_and_config 2024-09-11T01:47:33.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.011+0000 7f5ddcdec640 10 monclient: build_initial_monmap 2024-09-11T01:47:33.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.012+0000 7f5ddcdec640 10 monclient: monmap: 2024-09-11T01:47:33.014 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:47:33.014 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:33.014 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:47:33.012744+0000 2024-09-11T01:47:33.014 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:47:33.012744+0000 2024-09-11T01:47:33.014 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:47:33.014 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:33.014 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:47:33.014 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:47:33.014 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:47:33.014 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:47:33.014 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:47:33.014 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:33.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.012+0000 7f5ddcdec640 1 Processor -- start 2024-09-11T01:47:33.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.012+0000 7f5ddcdec640 1 -- start start 2024-09-11T01:47:33.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.012+0000 7f5ddcdec640 10 monclient: init 2024-09-11T01:47:33.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5ddcdec640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:33.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5ddcdec640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-11T01:47:33.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5ddcdec640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5dd81381c0 0x7f5dd81385a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:33.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5ddcdec640 10 monclient(hunting): picked mon.noname-a con 0x7f5dd81381c0 addr v2:172.21.15.12:3300/0 2024-09-11T01:47:33.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5ddcdec640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5dd80a6aa0 0x7f5dd809cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:33.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5ddcdec640 10 monclient(hunting): picked mon.noname-c con 0x7f5dd80a6aa0 addr v2:172.21.15.71:3300/0 2024-09-11T01:47:33.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5ddcdec640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5dd80a4f20 0x7f5dd80a5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:33.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5ddcdec640 10 monclient(hunting): picked mon.noname-b con 0x7f5dd80a4f20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:47:33.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5ddcdec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:33.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5ddcdec640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f5dd806fc90 con 0x7f5dd81381c0 2024-09-11T01:47:33.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5ddcdec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:33.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5ddcdec640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f5dd809c5f0 con 0x7f5dd80a6aa0 2024-09-11T01:47:33.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5ddcdec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:33.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5ddcdec640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5dd809c310 con 0x7f5dd80a4f20 2024-09-11T01:47:33.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5ddcdec640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:33.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5ddcdec640 10 monclient(hunting): authenticate will time out at 1859.455384s 2024-09-11T01:47:33.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd659c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5dd81381c0 0x7f5dd81385a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:33.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd6d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5dd80a4f20 0x7f5dd80a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:33.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd659c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5dd81381c0 0x7f5dd81385a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38596/0 (socket says 172.21.15.12:38596) 2024-09-11T01:47:33.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd659c640 1 -- 172.21.15.12:0/1713716291 learned_addr learned my addr 172.21.15.12:0/1713716291 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:47:33.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd659c640 10 monclient(hunting): get_auth_request con 0x7f5dd81381c0 auth_method 0 2024-09-11T01:47:33.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:33.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:33.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:33.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd6d9d640 10 monclient(hunting): get_auth_request con 0x7f5dd80a4f20 auth_method 0 2024-09-11T01:47:33.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:33.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:33.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:33.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dcffff640 1 --2- 172.21.15.12:0/1713716291 >> v2:172.21.15.71:3300/0 conn(0x7f5dd80a6aa0 0x7f5dd809cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dcffff640 10 monclient(hunting): get_auth_request con 0x7f5dd80a6aa0 auth_method 0 2024-09-11T01:47:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dcffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dcffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dcffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dcffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dcffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dcffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd6d9d640 10 monclient(hunting): handle_auth_done global_id 184319 payload 274 2024-09-11T01:47:33.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd6d9d640 1 -- 172.21.15.12:0/1713716291 >> v2:172.21.15.71:3300/0 conn(0x7f5dd80a6aa0 msgr2=0x7f5dd809cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:47:33.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd6d9d640 1 --2- 172.21.15.12:0/1713716291 >> v2:172.21.15.71:3300/0 conn(0x7f5dd80a6aa0 0x7f5dd809cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:33.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd6d9d640 1 -- 172.21.15.12:0/1713716291 >> v2:172.21.15.12:3300/0 conn(0x7f5dd81381c0 msgr2=0x7f5dd81385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:47:33.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd6d9d640 1 --2- 172.21.15.12:0/1713716291 >> v2:172.21.15.12:3300/0 conn(0x7f5dd81381c0 0x7f5dd81385a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:33.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd6d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:47:33.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd6d9d640 1 monclient: found mon.noname-b 2024-09-11T01:47:33.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:33.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd6d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:47:33.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd6d9d640 1 -- 172.21.15.12:0/1713716291 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd809c8d0 con 0x7f5dd80a4f20 2024-09-11T01:47:33.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.013+0000 7f5dd659c640 1 --2- 172.21.15.12:0/1713716291 >> v2:172.21.15.12:3300/0 conn(0x7f5dd81381c0 0x7f5dd81385a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:47:33.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.016+0000 7f5dd6d9d640 1 -- 172.21.15.12:0/1713716291 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5dd80a4f20 msgr2=0x7f5dd80a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:33.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.016+0000 7f5dd6d9d640 1 --2- 172.21.15.12:0/1713716291 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5dd80a4f20 0x7f5dd80a5370 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5dd80a4be0 tx=0x7f5dd002a4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=420e1d60512d4c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:33.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.016+0000 7f5dd5d9b640 1 -- 172.21.15.12:0/1713716291 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0007950 con 0x7f5dd80a4f20 2024-09-11T01:47:33.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.016+0000 7f5dd5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:33.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.016+0000 7f5dd5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:47:33.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.016+0000 7f5dd5d9b640 10 monclient: dump: 2024-09-11T01:47:33.028 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:33.028 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:33.028 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:33.028 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:33.028 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:33.029 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:33.029 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:33.029 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:33.029 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:33.029 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:33.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.016+0000 7f5dd5d9b640 10 monclient: _finish_auth 0 2024-09-11T01:47:33.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.016+0000 7f5dd5d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:47:33.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.016+0000 7f5dd5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:33.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.016+0000 7f5dd5d9b640 1 -- 172.21.15.12:0/1713716291 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dd0007af0 con 0x7f5dd80a4f20 2024-09-11T01:47:33.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.016+0000 7f5ddcdec640 5 monclient: authenticate success, global_id 184319 2024-09-11T01:47:33.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.016+0000 7f5ddcdec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:47:33.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.016+0000 7f5dd5d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:47:33.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.016+0000 7f5dd5d9b640 1 -- 172.21.15.12:0/1713716291 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0031400 con 0x7f5dd80a4f20 2024-09-11T01:47:33.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.016+0000 7f5ddcdec640 10 monclient: get_monmap_and_config success 2024-09-11T01:47:33.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.016+0000 7f5ddcdec640 4 set_mon_vals no callback set 2024-09-11T01:47:33.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.016+0000 7f5dd5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:33.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.016+0000 7f5dd5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:47:33.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.016+0000 7f5dd5d9b640 10 monclient: dump: 2024-09-11T01:47:33.030 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:33.030 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:33.030 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:33.030 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:33.030 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:33.030 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:33.030 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:33.030 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:33.030 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:33.031 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:33.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.017+0000 7f5ddcdec640 10 monclient: shutdown 2024-09-11T01:47:33.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.017+0000 7f5ddcdec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:47:33.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.017+0000 7f5ddcdec640 1 -- 172.21.15.12:0/1713716291 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5dd80a4f20 msgr2=0x7f5dd80a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:33.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.017+0000 7f5ddcdec640 1 --2- 172.21.15.12:0/1713716291 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5dd80a4f20 0x7f5dd80a5370 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5dd80a4be0 tx=0x7f5dd002a4d0 comp rx=0 tx=0).stop 2024-09-11T01:47:33.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.017+0000 7f5ddcdec640 1 -- 172.21.15.12:0/1713716291 shutdown_connections 2024-09-11T01:47:33.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.017+0000 7f5ddcdec640 1 --2- 172.21.15.12:0/1713716291 >> v2:172.21.15.71:3300/0 conn(0x7f5dd80a6aa0 0x7f5dd809cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:33.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.017+0000 7f5ddcdec640 1 --2- 172.21.15.12:0/1713716291 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5dd80a4f20 0x7f5dd80a5370 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:33.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.017+0000 7f5ddcdec640 1 --2- 172.21.15.12:0/1713716291 >> v2:172.21.15.12:3300/0 conn(0x7f5dd81381c0 0x7f5dd81385a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:33.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.017+0000 7f5ddcdec640 1 -- 172.21.15.12:0/1713716291 >> 172.21.15.12:0/1713716291 conn(0x7f5dd80a7440 msgr2=0x7f5dd80a7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:47:33.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.017+0000 7f5ddcdec640 1 -- 172.21.15.12:0/1713716291 shutdown_connections 2024-09-11T01:47:33.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.017+0000 7f5ddcdec640 1 -- 172.21.15.12:0/1713716291 wait complete. 2024-09-11T01:47:33.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.018+0000 7f5ddcdec640 10 monclient: build_initial_monmap 2024-09-11T01:47:33.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.018+0000 7f5ddcdec640 10 monclient: monmap: 2024-09-11T01:47:33.032 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:47:33.032 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:47:33.032 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:47:33.032 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:47:33.032 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:47:33.032 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:33.032 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:47:33.032 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:47:33.032 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:47:33.032 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:33.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.018+0000 7f5ddcdec640 1 Processor -- start 2024-09-11T01:47:33.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.018+0000 7f5ddcdec640 1 -- start start 2024-09-11T01:47:33.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.018+0000 7f5ddcdec640 10 monclient: init 2024-09-11T01:47:33.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5ddcdec640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:33.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5ddcdec640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:47:33.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5ddcdec640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dd80a4f20 0x7f5dd8135e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:33.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5ddcdec640 10 monclient(hunting): picked mon.noname-a con 0x7f5dd80a4f20 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:47:33.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5ddcdec640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dd80a6aa0 0x7f5dd8136350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:33.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5ddcdec640 10 monclient(hunting): picked mon.noname-b con 0x7f5dd80a6aa0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:47:33.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5ddcdec640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5dd81381c0 0x7f5dd81239f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:33.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5ddcdec640 10 monclient(hunting): picked mon.noname-c con 0x7f5dd81381c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:47:33.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5ddcdec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:33.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5ddcdec640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd809d690 con 0x7f5dd80a4f20 2024-09-11T01:47:33.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5ddcdec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:33.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5ddcdec640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd809d510 con 0x7f5dd80a6aa0 2024-09-11T01:47:33.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5ddcdec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:33.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5ddcdec640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5dd80a58b0 con 0x7f5dd81381c0 2024-09-11T01:47:33.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5ddcdec640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:33.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5ddcdec640 10 monclient(hunting): authenticate will time out at 1859.461210s 2024-09-11T01:47:33.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dd80a4f20 0x7f5dd8135e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:33.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dd80a4f20 0x7f5dd8135e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38612/0 (socket says 172.21.15.12:38612) 2024-09-11T01:47:33.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd659c640 1 -- 172.21.15.12:0/4177105577 learned_addr learned my addr 172.21.15.12:0/4177105577 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:47:33.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd659c640 10 monclient(hunting): get_auth_request con 0x7f5dd80a4f20 auth_method 0 2024-09-11T01:47:33.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:33.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:33.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:33.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd6d9d640 1 --2- 172.21.15.12:0/4177105577 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5dd81381c0 0x7f5dd81239f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:33.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd6d9d640 10 monclient(hunting): get_auth_request con 0x7f5dd81381c0 auth_method 0 2024-09-11T01:47:33.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:33.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:33.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:33.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:33.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:33.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:33.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:33.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:33.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:33.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dcffff640 1 --2- 172.21.15.12:0/4177105577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dd80a6aa0 0x7f5dd8136350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:33.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd659c640 10 monclient(hunting): handle_auth_done global_id 204159 payload 824 2024-09-11T01:47:33.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd659c640 1 -- 172.21.15.12:0/4177105577 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5dd81381c0 msgr2=0x7f5dd81239f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:33.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd659c640 1 --2- 172.21.15.12:0/4177105577 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5dd81381c0 0x7f5dd81239f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:33.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd659c640 1 -- 172.21.15.12:0/4177105577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dd80a6aa0 msgr2=0x7f5dd8136350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:33.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd659c640 1 --2- 172.21.15.12:0/4177105577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dd80a6aa0 0x7f5dd8136350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:33.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd659c640 10 monclient: _finish_hunting 0 2024-09-11T01:47:33.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd659c640 1 monclient: found mon.noname-a 2024-09-11T01:47:33.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:33.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:47:33.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dd659c640 1 -- 172.21.15.12:0/4177105577 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd8127530 con 0x7f5dd80a4f20 2024-09-11T01:47:33.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dcffff640 10 monclient: get_auth_request con 0x7f5dd80a6aa0 auth_method 0 2024-09-11T01:47:33.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.019+0000 7f5dcffff640 1 --2- 172.21.15.12:0/4177105577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dd80a6aa0 0x7f5dd8136350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:47:33.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.021+0000 7f5dd659c640 1 -- 172.21.15.12:0/4177105577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dd80a4f20 msgr2=0x7f5dd8135e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:33.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.021+0000 7f5dd659c640 1 --2- 172.21.15.12:0/4177105577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dd80a4f20 0x7f5dd8135e10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5dc800e9d0 tx=0x7f5dc800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:33.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.021+0000 7f5dceffd640 1 -- 172.21.15.12:0/4177105577 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc8017070 con 0x7f5dd80a4f20 2024-09-11T01:47:33.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.021+0000 7f5dceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:33.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.022+0000 7f5dceffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:47:33.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.022+0000 7f5dceffd640 10 monclient: dump: 2024-09-11T01:47:33.036 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:33.036 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:33.037 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:33.037 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:33.037 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:33.037 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:33.037 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:33.037 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:33.037 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:33.037 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:33.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.022+0000 7f5dceffd640 10 monclient: _finish_auth 0 2024-09-11T01:47:33.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.022+0000 7f5dceffd640 10 monclient: _check_auth_tickets 2024-09-11T01:47:33.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.022+0000 7f5dceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.022+0000 7f5ddcdec640 5 monclient: authenticate success, global_id 204159 2024-09-11T01:47:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.022+0000 7f5dceffd640 1 -- 172.21.15.12:0/4177105577 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dc800cc70 con 0x7f5dd80a4f20 2024-09-11T01:47:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.022+0000 7f5dceffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:47:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.022+0000 7f5dceffd640 1 -- 172.21.15.12:0/4177105577 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc8013670 con 0x7f5dd80a4f20 2024-09-11T01:47:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.022+0000 7f5ddcdec640 10 monclient: _renew_subs 2024-09-11T01:47:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.022+0000 7f5ddcdec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:47:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.022+0000 7f5ddcdec640 1 -- 172.21.15.12:0/4177105577 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dd8127810 con 0x7f5dd80a4f20 2024-09-11T01:47:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.022+0000 7f5dceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.022+0000 7f5dceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:47:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.022+0000 7f5dceffd640 10 monclient: dump: 2024-09-11T01:47:33.042 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:33.043 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:33.043 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:33.043 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:33.043 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:33.043 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:33.043 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:33.043 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:33.043 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:33.043 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:33.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.022+0000 7f5dd559a640 4 set_mon_vals no callback set 2024-09-11T01:47:33.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.022+0000 7f5ddcdec640 10 monclient: _renew_subs 2024-09-11T01:47:33.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.022+0000 7f5ddcdec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:47:33.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.022+0000 7f5ddcdec640 1 -- 172.21.15.12:0/4177105577 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dd81263c0 con 0x7f5dd80a4f20 2024-09-11T01:47:33.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.022+0000 7f5ddcdec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:33.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.022+0000 7f5ddcdec640 1 --2- 172.21.15.12:0/4177105577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dd81ccfb0 0x7f5dd8123f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:33.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.022+0000 7f5ddcdec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:33.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.022+0000 7f5ddcdec640 1 -- 172.21.15.12:0/4177105577 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd81271b0 con 0x7f5dd81ccfb0 2024-09-11T01:47:33.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.022+0000 7f5dceffd640 1 -- 172.21.15.12:0/4177105577 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f5dc8024060 con 0x7f5dd80a4f20 2024-09-11T01:47:33.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.023+0000 7f5dcffff640 1 --2- 172.21.15.12:0/4177105577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dd81ccfb0 0x7f5dd8123f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:33.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.023+0000 7f5dcffff640 10 monclient: get_auth_request con 0x7f5dd81ccfb0 auth_method 0 2024-09-11T01:47:33.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.023+0000 7f5dcffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:33.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.023+0000 7f5dcffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:33.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.023+0000 7f5dcffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:33.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.023+0000 7f5dceffd640 1 --2- 172.21.15.12:0/4177105577 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5db803e240 0x7f5db8040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:33.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.023+0000 7f5dcffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:33.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.023+0000 7f5dcffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:33.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.023+0000 7f5dcffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:33.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.023+0000 7f5dceffd640 1 -- 172.21.15.12:0/4177105577 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(619..619 src has 507..619) ==== 4803+0+0 (secure 0 0 0) 0x7f5dc8052c60 con 0x7f5dd80a4f20 2024-09-11T01:47:33.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.023+0000 7f5dd6d9d640 1 --2- 172.21.15.12:0/4177105577 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5db803e240 0x7f5db8040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:33.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.023+0000 7f5dcffff640 10 monclient(hunting): handle_auth_done global_id 134722 payload 824 2024-09-11T01:47:33.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.023+0000 7f5dcffff640 1 -- 172.21.15.12:0/4177105577 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5dd81388c0 con 0x7f5dd81ccfb0 2024-09-11T01:47:33.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.023+0000 7f5dd6d9d640 10 monclient: get_auth_request con 0x7f5db803e240 auth_method 0 2024-09-11T01:47:33.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.025+0000 7f5dcffff640 1 -- 172.21.15.12:0/4177105577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dd81ccfb0 msgr2=0x7f5dd8123f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:33.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.025+0000 7f5dcffff640 1 --2- 172.21.15.12:0/4177105577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dd81ccfb0 0x7f5dd8123f30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0009fd0 tx=0x7f5dc0010c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:33.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.025+0000 7f5dceffd640 1 -- 172.21.15.12:0/4177105577 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc000f040 con 0x7f5dd81ccfb0 2024-09-11T01:47:33.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.025+0000 7f5dceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:33.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.025+0000 7f5dceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:47:33.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.025+0000 7f5dceffd640 10 monclient: dump: 2024-09-11T01:47:33.046 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:33.046 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:33.046 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:33.046 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:33.046 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:33.046 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:33.046 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:33.046 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:33.046 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:33.046 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:33.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.026+0000 7f5dd6d9d640 1 --2- 172.21.15.12:0/4177105577 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5db803e240 0x7f5db8040700 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5dd806fab0 tx=0x7f5dd002f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:33.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.026+0000 7f5dceffd640 1 -- 172.21.15.12:0/4177105577 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5dc0004020 con 0x7f5dd81ccfb0 2024-09-11T01:47:33.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.026+0000 7f5dceffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:33.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.026+0000 7f5dceffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:47:33.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.026+0000 7f5dceffd640 1 -- 172.21.15.12:0/4177105577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dd81ccfb0 msgr2=0x7f5dd8123f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:33.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.026+0000 7f5dceffd640 1 --2- 172.21.15.12:0/4177105577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dd81ccfb0 0x7f5dd8123f30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0009fd0 tx=0x7f5dc0010c90 comp rx=0 tx=0).stop 2024-09-11T01:47:33.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.026+0000 7f5dceffd640 1 -- 172.21.15.12:0/4177105577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dd81ccfb0 msgr2=0x7f5dd8123f30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:47:33.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.026+0000 7f5dceffd640 1 --2- 172.21.15.12:0/4177105577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dd81ccfb0 0x7f5dd8123f30 secure :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0009fd0 tx=0x7f5dc0010c90 comp rx=0 tx=0).stop 2024-09-11T01:47:33.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.032+0000 7f5ddcdec640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:47:33.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.032+0000 7f5ddcdec640 1 --2- 172.21.15.12:0/4177105577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dd8124470 0x7f5dd80a2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:33.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.032+0000 7f5ddcdec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:33.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.032+0000 7f5ddcdec640 1 -- 172.21.15.12:0/4177105577 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd8124880 con 0x7f5dd8124470 2024-09-11T01:47:33.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.032+0000 7f5dcffff640 1 --2- 172.21.15.12:0/4177105577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dd8124470 0x7f5dd80a2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:33.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.032+0000 7f5dcffff640 10 monclient: get_auth_request con 0x7f5dd8124470 auth_method 0 2024-09-11T01:47:33.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.032+0000 7f5dcffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:33.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.032+0000 7f5dcffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:33.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.032+0000 7f5dcffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:33.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.033+0000 7f5dcffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:33.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.033+0000 7f5dcffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:33.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.033+0000 7f5dcffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:33.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.033+0000 7f5dcffff640 10 monclient(hunting): handle_auth_done global_id 134725 payload 824 2024-09-11T01:47:33.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.033+0000 7f5dcffff640 1 -- 172.21.15.12:0/4177105577 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5dd81254b0 con 0x7f5dd8124470 2024-09-11T01:47:33.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.035+0000 7f5dcffff640 1 -- 172.21.15.12:0/4177105577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dd8124470 msgr2=0x7f5dd80a2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:33.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.035+0000 7f5dcffff640 1 --2- 172.21.15.12:0/4177105577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dd8124470 0x7f5dd80a2c00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0008010 tx=0x7f5dc00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:33.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.035+0000 7f5dceffd640 1 -- 172.21.15.12:0/4177105577 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0003d90 con 0x7f5dd8124470 2024-09-11T01:47:33.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.035+0000 7f5dceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:33.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.035+0000 7f5dceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:47:33.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.035+0000 7f5dceffd640 10 monclient: dump: 2024-09-11T01:47:33.050 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:33.050 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:33.050 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:33.050 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:33.050 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:33.050 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:33.050 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:33.050 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:33.050 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:33.051 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:33.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.036+0000 7f5dceffd640 1 -- 172.21.15.12:0/4177105577 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f5dc0003d90 con 0x7f5dd8124470 2024-09-11T01:47:33.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.036+0000 7f5dceffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:47:33.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.036+0000 7f5dceffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:47:33.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.036+0000 7f5dceffd640 1 -- 172.21.15.12:0/4177105577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dd8124470 msgr2=0x7f5dd80a2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:33.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.036+0000 7f5dceffd640 1 --2- 172.21.15.12:0/4177105577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dd8124470 0x7f5dd80a2c00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0008010 tx=0x7f5dc00046c0 comp rx=0 tx=0).stop 2024-09-11T01:47:33.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.036+0000 7f5dceffd640 1 -- 172.21.15.12:0/4177105577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dd8124470 msgr2=0x7f5dd80a2c00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:47:33.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.036+0000 7f5dceffd640 1 --2- 172.21.15.12:0/4177105577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dd8124470 0x7f5dd80a2c00 secure :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0008010 tx=0x7f5dc00046c0 comp rx=0 tx=0).stop 2024-09-11T01:47:33.051 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:47:33.051 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:47:33.052 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:47:33.052 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:47:33.052 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 172, 2024-09-11T01:47:33.052 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 385998, 2024-09-11T01:47:33.052 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:47:33.052 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:47:33.052 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:47:33.052 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:47:33.052 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:33.052 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:47:33.052 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:47:33.052 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:47:33.052 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:47:33.053 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:47:33.053 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:47:33.053 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:47:33.053 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:47:33.053 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:47:33.053 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:47:33.053 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:47:33.053 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:47:33.053 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:47:33.053 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:47:33.053 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:47:33.053 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:33.053 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:47:33.053 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:47:33.054 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:47:33.054 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:47:33.054 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:47:33.054 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:47:33.054 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:47:33.054 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:47:33.054 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:47:33.054 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:47:33.054 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:47:33.054 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:47:33.054 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:47:33.054 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:33.054 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:33.054 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:47:33.055 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:47:33.055 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:47:33.055 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:47:33.055 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:47:33.055 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:47:33.055 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:47:33.055 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:47:33.055 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:47:33.055 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:47:33.055 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:47:33.055 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:47:33.055 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:47:33.055 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:47:33.056 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:47:33.056 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:47:33.056 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:47:33.056 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:47:33.056 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:47:33.056 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:47:33.056 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:47:33.056 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:47:33.056 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:47:33.056 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:47:33.056 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:47:33.057 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:47:33.057 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:47:33.057 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:47:33.057 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:33.057 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:47:33.057 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:33.057 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:47:33.057 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:33.057 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:47:33.057 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:47:33.057 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:47:33.058 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:47:33.058 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:33.058 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:47:33.058 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:47:33.058 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:33.058 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:33.058 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:33.058 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:47:33.058 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:47:33.058 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:33.058 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:33.058 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:33.058 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:33.058 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:47:33.058 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:47:33.059 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:47:33.059 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:47:33.059 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:47:33.059 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:33.059 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:33.059 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:47:33.059 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:47:33.059 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:47:33.059 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:47:33.059 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:33.059 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:47:33.060 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:47:33.060 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:33.060 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:33.060 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:33.060 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:47:33.060 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:47:33.060 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:33.060 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:33.060 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:33.060 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:33.060 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:47:33.060 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:47:33.060 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:47:33.061 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:47:33.061 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:47:33.061 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:33.061 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:33.061 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:47:33.061 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:47:33.061 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:47:33.061 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:47:33.061 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:33.061 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:47:33.061 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:47:33.061 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:33.062 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:33.062 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:33.062 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:47:33.062 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:47:33.062 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:33.062 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:33.062 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:33.062 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:33.062 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:47:33.062 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:47:33.062 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:47:33.062 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:47:33.063 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:47:33.063 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:33.063 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:33.063 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:33.063 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:47:33.063 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:47:33.063 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:33.063 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:47:33.063 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:47:33.063 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:47:33.063 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:33.063 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:33.063 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:47:33.063 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:33.064 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:47:33.064 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:47:33.064 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:47:33.064 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:33.064 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:33.064 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:47:33.064 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:33.064 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:47:33.064 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:47:33.064 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:47:33.064 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:33.064 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:33.064 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:47:33.065 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:33.065 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:47:33.065 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:47:33.065 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:47:33.065 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:33.065 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:33.065 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:33.065 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:47:33.065 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:47:33.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.036+0000 7f5ddcdec640 1 -- 172.21.15.12:0/4177105577 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5db803e240 msgr2=0x7f5db8040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:33.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.036+0000 7f5ddcdec640 1 --2- 172.21.15.12:0/4177105577 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5db803e240 0x7f5db8040700 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5dd806fab0 tx=0x7f5dd002f000 comp rx=0 tx=0).stop 2024-09-11T01:47:33.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.036+0000 7f5ddcdec640 10 monclient: shutdown 2024-09-11T01:47:33.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.036+0000 7f5ddcdec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:47:33.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.036+0000 7f5ddcdec640 1 -- 172.21.15.12:0/4177105577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dd80a4f20 msgr2=0x7f5dd8135e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:33.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.036+0000 7f5ddcdec640 1 --2- 172.21.15.12:0/4177105577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5dd80a4f20 0x7f5dd8135e10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5dc800e9d0 tx=0x7f5dc800ee90 comp rx=0 tx=0).stop 2024-09-11T01:47:33.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.036+0000 7f5dd6d9d640 1 -- 172.21.15.12:0/4177105577 reap_dead start 2024-09-11T01:47:33.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.036+0000 7f5dd6d9d640 1 -- 172.21.15.12:0/4177105577 reap_dead start 2024-09-11T01:47:33.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.036+0000 7f5ddcdec640 1 -- 172.21.15.12:0/4177105577 shutdown_connections 2024-09-11T01:47:33.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.036+0000 7f5ddcdec640 1 -- 172.21.15.12:0/4177105577 >> 172.21.15.12:0/4177105577 conn(0x7f5dd80a7440 msgr2=0x7f5dd809ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:47:33.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.036+0000 7f5ddcdec640 1 -- 172.21.15.12:0/4177105577 shutdown_connections 2024-09-11T01:47:33.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:33.036+0000 7f5ddcdec640 1 -- 172.21.15.12:0/4177105577 wait complete. 2024-09-11T01:47:33.067 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:47:33.068 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-11T01:47:33.068 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-11T01:47:33.068 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-11T01:47:33.068 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-11T01:47:33.068 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T01:47:33.068 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:47:33.077 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T01:47:33.077 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-11T01:47:33.078 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-11T01:47:33.078 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T01:47:48.078 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-11T01:47:48.079 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-11T01:47:48.079 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:47:48.080 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:47:48.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.191+0000 7eff95d2b640 10 monclient: get_monmap_and_config 2024-09-11T01:47:48.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.191+0000 7eff95d2b640 10 monclient: build_initial_monmap 2024-09-11T01:47:48.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.191+0000 7eff95d2b640 10 monclient: monmap: 2024-09-11T01:47:48.193 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:47:48.193 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:48.193 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:47:48.192339+0000 2024-09-11T01:47:48.193 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:47:48.192339+0000 2024-09-11T01:47:48.193 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:47:48.194 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:48.194 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:47:48.194 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:47:48.194 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:47:48.194 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:47:48.194 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:47:48.194 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:48.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.192+0000 7eff95d2b640 1 Processor -- start 2024-09-11T01:47:48.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.192+0000 7eff95d2b640 1 -- start start 2024-09-11T01:47:48.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.192+0000 7eff95d2b640 10 monclient: init 2024-09-11T01:47:48.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff95d2b640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:48.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff95d2b640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-11T01:47:48.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff95d2b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7eff900a1380 0x7eff900a1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:48.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff95d2b640 10 monclient(hunting): picked mon.noname-c con 0x7eff900a1380 addr v2:172.21.15.71:3300/0 2024-09-11T01:47:48.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff95d2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7eff900a0c90 addr v1:172.21.15.12:6789/0 2024-09-11T01:47:48.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff95d2b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 0x7eff9013de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:48.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff95d2b640 10 monclient(hunting): picked mon.noname-b con 0x7eff901361f0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:47:48.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff95d2b640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff900a2b40 con 0x7eff900a0c90 2024-09-11T01:47:48.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff95d2b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:48.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff95d2b640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7eff9009f9a0 con 0x7eff900a1380 2024-09-11T01:47:48.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff95d2b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:48.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff95d2b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7eff900a2230 con 0x7eff901361f0 2024-09-11T01:47:48.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff95d2b640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:48.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff95d2b640 10 monclient(hunting): authenticate will time out at 1874.635182s 2024-09-11T01:47:48.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff8effd640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7eff900a0c90 legacy=0x7eff90135ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:47:48.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff8ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 0x7eff9013de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:48.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff8ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 0x7eff9013de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55124/0 (socket says 172.21.15.12:55124) 2024-09-11T01:47:48.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff8ffff640 1 -- 172.21.15.12:0/27444542 learned_addr learned my addr 172.21.15.12:0/27444542 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:47:48.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff8ffff640 10 monclient(hunting): get_auth_request con 0x7eff901361f0 auth_method 0 2024-09-11T01:47:48.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:48.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:48.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:48.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff8ffff640 1 -- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 msgr2=0x7eff9013de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:47:48.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff8ffff640 1 -- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 msgr2=0x7eff9013de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:47:48.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff8ffff640 1 --2- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 0x7eff9013de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:47:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff8f7fe640 1 --2- 172.21.15.12:0/27444542 >> v2:172.21.15.71:3300/0 conn(0x7eff900a1380 0x7eff900a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff8f7fe640 10 monclient(hunting): get_auth_request con 0x7eff900a1380 auth_method 0 2024-09-11T01:47:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff8ffff640 1 --2- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 0x7eff9013de20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:47:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff8f7fe640 1 -- 172.21.15.12:0/27444542 >> v2:172.21.15.71:3300/0 conn(0x7eff900a1380 msgr2=0x7eff900a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:47:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff8f7fe640 1 -- 172.21.15.12:0/27444542 >> v2:172.21.15.71:3300/0 conn(0x7eff900a1380 msgr2=0x7eff900a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:47:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff8f7fe640 1 --2- 172.21.15.12:0/27444542 >> v2:172.21.15.71:3300/0 conn(0x7eff900a1380 0x7eff900a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:47:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.193+0000 7eff8f7fe640 1 --2- 172.21.15.12:0/27444542 >> v2:172.21.15.71:3300/0 conn(0x7eff900a1380 0x7eff900a1760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:47:48.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.394+0000 7eff8effd640 1 -- 172.21.15.12:0/27444542 >> v1:172.21.15.12:6789/0 conn(0x7eff900a0c90 legacy=0x7eff90135ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:47:48.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.394+0000 7eff8ffff640 1 --2- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 0x7eff9013de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:48.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.394+0000 7eff8ffff640 10 monclient(hunting): get_auth_request con 0x7eff901361f0 auth_method 0 2024-09-11T01:47:48.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.394+0000 7eff8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:48.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.394+0000 7eff8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:48.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.394+0000 7eff8ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:47:48.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.394+0000 7eff8ffff640 1 -- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 msgr2=0x7eff9013de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:47:48.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.394+0000 7eff8ffff640 1 -- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 msgr2=0x7eff9013de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:47:48.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.394+0000 7eff8ffff640 1 --2- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 0x7eff9013de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:47:48.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.394+0000 7eff8ffff640 1 --2- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 0x7eff9013de20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:47:48.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.394+0000 7eff8f7fe640 1 --2- 172.21.15.12:0/27444542 >> v2:172.21.15.71:3300/0 conn(0x7eff900a1380 0x7eff900a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:48.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.394+0000 7eff8f7fe640 10 monclient(hunting): get_auth_request con 0x7eff900a1380 auth_method 0 2024-09-11T01:47:48.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.394+0000 7eff8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:48.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.394+0000 7eff8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:48.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.394+0000 7eff8f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:47:48.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.394+0000 7eff8f7fe640 1 -- 172.21.15.12:0/27444542 >> v2:172.21.15.71:3300/0 conn(0x7eff900a1380 msgr2=0x7eff900a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:47:48.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.394+0000 7eff8f7fe640 1 -- 172.21.15.12:0/27444542 >> v2:172.21.15.71:3300/0 conn(0x7eff900a1380 msgr2=0x7eff900a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:47:48.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.394+0000 7eff8f7fe640 1 --2- 172.21.15.12:0/27444542 >> v2:172.21.15.71:3300/0 conn(0x7eff900a1380 0x7eff900a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:47:48.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.394+0000 7eff8f7fe640 1 --2- 172.21.15.12:0/27444542 >> v2:172.21.15.71:3300/0 conn(0x7eff900a1380 0x7eff900a1760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:47:48.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.795+0000 7eff8effd640 1 -- 172.21.15.12:0/27444542 >> v1:172.21.15.12:6789/0 conn(0x7eff900a0c90 legacy=0x7eff90135ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:47:48.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.795+0000 7eff8ffff640 1 --2- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 0x7eff9013de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:48.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.795+0000 7eff8ffff640 10 monclient(hunting): get_auth_request con 0x7eff901361f0 auth_method 0 2024-09-11T01:47:48.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.795+0000 7eff8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:48.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.795+0000 7eff8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:48.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.795+0000 7eff8ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:47:48.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.795+0000 7eff8ffff640 1 -- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 msgr2=0x7eff9013de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:47:48.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.795+0000 7eff8ffff640 1 -- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 msgr2=0x7eff9013de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:47:48.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.795+0000 7eff8ffff640 1 --2- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 0x7eff9013de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:47:48.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.796+0000 7eff8ffff640 1 --2- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 0x7eff9013de20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:47:48.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.796+0000 7eff8f7fe640 1 --2- 172.21.15.12:0/27444542 >> v2:172.21.15.71:3300/0 conn(0x7eff900a1380 0x7eff900a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:48.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.796+0000 7eff8f7fe640 10 monclient(hunting): get_auth_request con 0x7eff900a1380 auth_method 0 2024-09-11T01:47:48.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.796+0000 7eff8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:48.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.796+0000 7eff8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:48.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.796+0000 7eff8f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:47:48.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.796+0000 7eff8f7fe640 1 -- 172.21.15.12:0/27444542 >> v2:172.21.15.71:3300/0 conn(0x7eff900a1380 msgr2=0x7eff900a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:47:48.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.796+0000 7eff8f7fe640 1 -- 172.21.15.12:0/27444542 >> v2:172.21.15.71:3300/0 conn(0x7eff900a1380 msgr2=0x7eff900a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:47:48.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.796+0000 7eff8f7fe640 1 --2- 172.21.15.12:0/27444542 >> v2:172.21.15.71:3300/0 conn(0x7eff900a1380 0x7eff900a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:47:48.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:48.796+0000 7eff8f7fe640 1 --2- 172.21.15.12:0/27444542 >> v2:172.21.15.71:3300/0 conn(0x7eff900a1380 0x7eff900a1760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:47:49.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:49.596+0000 7eff8effd640 1 -- 172.21.15.12:0/27444542 >> v1:172.21.15.12:6789/0 conn(0x7eff900a0c90 legacy=0x7eff90135ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:47:49.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:49.597+0000 7eff8ffff640 1 --2- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 0x7eff9013de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:49.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:49.597+0000 7eff8ffff640 10 monclient(hunting): get_auth_request con 0x7eff901361f0 auth_method 0 2024-09-11T01:47:49.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:49.597+0000 7eff8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:49.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:49.597+0000 7eff8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:49.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:49.597+0000 7eff8ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:47:49.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:49.597+0000 7eff8ffff640 1 -- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 msgr2=0x7eff9013de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:47:49.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:49.597+0000 7eff8ffff640 1 -- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 msgr2=0x7eff9013de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:47:49.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:49.597+0000 7eff8ffff640 1 --2- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 0x7eff9013de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:47:49.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:49.597+0000 7eff8ffff640 1 --2- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 0x7eff9013de20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:47:49.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:49.597+0000 7eff8f7fe640 1 --2- 172.21.15.12:0/27444542 >> v2:172.21.15.71:3300/0 conn(0x7eff900a1380 0x7eff900a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:49.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:49.597+0000 7eff8f7fe640 10 monclient(hunting): get_auth_request con 0x7eff900a1380 auth_method 0 2024-09-11T01:47:49.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:49.597+0000 7eff8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:49.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:49.597+0000 7eff8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:49.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:49.597+0000 7eff8f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:47:49.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:49.597+0000 7eff8f7fe640 1 -- 172.21.15.12:0/27444542 >> v2:172.21.15.71:3300/0 conn(0x7eff900a1380 msgr2=0x7eff900a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:47:49.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:49.597+0000 7eff8f7fe640 1 -- 172.21.15.12:0/27444542 >> v2:172.21.15.71:3300/0 conn(0x7eff900a1380 msgr2=0x7eff900a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:47:49.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:49.597+0000 7eff8f7fe640 1 --2- 172.21.15.12:0/27444542 >> v2:172.21.15.71:3300/0 conn(0x7eff900a1380 0x7eff900a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:47:49.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:49.597+0000 7eff8f7fe640 1 --2- 172.21.15.12:0/27444542 >> v2:172.21.15.71:3300/0 conn(0x7eff900a1380 0x7eff900a1760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:47:51.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:51.192+0000 7eff8d7fa640 10 monclient(hunting): tick 2024-09-11T01:47:51.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:51.192+0000 7eff8d7fa640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:47:51.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:51.192+0000 7eff8d7fa640 1 monclient(hunting): continuing hunt 2024-09-11T01:47:51.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:51.192+0000 7eff8d7fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:47:51.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:51.192+0000 7eff8d7fa640 1 -- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 msgr2=0x7eff9013de20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:47:51.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:51.192+0000 7eff8d7fa640 1 --2- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 0x7eff9013de20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:51.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:51.192+0000 7eff8d7fa640 1 -- 172.21.15.12:0/27444542 >> v2:172.21.15.71:3300/0 conn(0x7eff900a1380 msgr2=0x7eff900a1760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:47:51.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:51.192+0000 7eff8d7fa640 1 --2- 172.21.15.12:0/27444542 >> v2:172.21.15.71:3300/0 conn(0x7eff900a1380 0x7eff900a1760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:51.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:51.192+0000 7eff8d7fa640 1 -- 172.21.15.12:0/27444542 >> v1:172.21.15.12:6789/0 conn(0x7eff900a0c90 legacy=0x7eff90135ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:47:51.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:51.192+0000 7eff8d7fa640 10 monclient: _add_conns ranks=[1,3] 2024-09-11T01:47:51.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:51.192+0000 7eff8d7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7eff7c0036a0 addr v1:172.21.15.71:6789/0 2024-09-11T01:47:51.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:51.192+0000 7eff8d7fa640 1 --2- 172.21.15.12:0/27444542 >> v2:172.21.15.12:3300/0 conn(0x7eff7c0073d0 0x7eff7c009890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:51.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:51.192+0000 7eff8d7fa640 10 monclient(hunting): picked mon.noname-a con 0x7eff7c0073d0 addr v2:172.21.15.12:3300/0 2024-09-11T01:47:51.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:51.192+0000 7eff8d7fa640 1 -- 172.21.15.12:0/27444542 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff900a2510 con 0x7eff7c0036a0 2024-09-11T01:47:51.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:51.192+0000 7eff8d7fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:51.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:51.192+0000 7eff8d7fa640 1 -- 172.21.15.12:0/27444542 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7eff9009f9a0 con 0x7eff7c0073d0 2024-09-11T01:47:51.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:51.192+0000 7eff8d7fa640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:51.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:51.192+0000 7eff8effd640 1 -- 172.21.15.12:0/27444542 >> v2:172.21.15.12:3300/0 conn(0x7eff7c0073d0 msgr2=0x7eff7c009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:47:51.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:51.192+0000 7eff8effd640 1 --2- 172.21.15.12:0/27444542 >> v2:172.21.15.12:3300/0 conn(0x7eff7c0073d0 0x7eff7c009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:47:51.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:51.195+0000 7eff8f7fe640 1 -- 172.21.15.12:0/27444542 >> v1:172.21.15.71:6789/0 conn(0x7eff7c0036a0 legacy=0x7eff7c005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:51.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:51.393+0000 7eff8effd640 1 -- 172.21.15.12:0/27444542 >> v2:172.21.15.12:3300/0 conn(0x7eff7c0073d0 msgr2=0x7eff7c009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:47:51.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:51.393+0000 7eff8effd640 1 --2- 172.21.15.12:0/27444542 >> v2:172.21.15.12:3300/0 conn(0x7eff7c0073d0 0x7eff7c009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:47:51.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:51.794+0000 7eff8effd640 1 -- 172.21.15.12:0/27444542 >> v2:172.21.15.12:3300/0 conn(0x7eff7c0073d0 msgr2=0x7eff7c009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:47:51.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:51.794+0000 7eff8effd640 1 --2- 172.21.15.12:0/27444542 >> v2:172.21.15.12:3300/0 conn(0x7eff7c0073d0 0x7eff7c009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:47:52.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:52.595+0000 7eff8effd640 1 -- 172.21.15.12:0/27444542 >> v2:172.21.15.12:3300/0 conn(0x7eff7c0073d0 msgr2=0x7eff7c009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:47:52.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:52.595+0000 7eff8effd640 1 --2- 172.21.15.12:0/27444542 >> v2:172.21.15.12:3300/0 conn(0x7eff7c0073d0 0x7eff7c009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:47:53.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.089+0000 7eff8e7fc640 1 -- 172.21.15.12:0/27444542 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2756302147 0 0) 0x7eff900a2510 con 0x7eff7c0036a0 2024-09-11T01:47:53.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.089+0000 7eff8e7fc640 10 monclient(hunting): _negotiate 2024-09-11T01:47:53.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.089+0000 7eff8e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:53.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.089+0000 7eff8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:53.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.089+0000 7eff8e7fc640 10 monclient(hunting): my global_id is 134728 2024-09-11T01:47:53.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.089+0000 7eff8e7fc640 1 -- 172.21.15.12:0/27444542 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff78003520 con 0x7eff7c0036a0 2024-09-11T01:47:53.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 1 -- 172.21.15.12:0/27444542 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3681338034 0 0) 0x7eff78003520 con 0x7eff7c0036a0 2024-09-11T01:47:53.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 1 -- 172.21.15.12:0/27444542 >> v2:172.21.15.12:3300/0 conn(0x7eff7c0073d0 msgr2=0x7eff7c009890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:47:53.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 1 --2- 172.21.15.12:0/27444542 >> v2:172.21.15.12:3300/0 conn(0x7eff7c0073d0 0x7eff7c009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 10 monclient: _finish_hunting 0 2024-09-11T01:47:53.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 1 monclient: found mon.noname-c-legacy 2024-09-11T01:47:53.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:53.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-09-11T01:47:53.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 1 -- 172.21.15.12:0/27444542 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff900a2b40 con 0x7eff7c0036a0 2024-09-11T01:47:53.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 10 monclient: _finish_auth 0 2024-09-11T01:47:53.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:47:53.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:53.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 1 -- 172.21.15.12:0/27444542 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7eff8000d680 con 0x7eff7c0036a0 2024-09-11T01:47:53.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:53.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-11T01:47:53.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 10 monclient: dump: 2024-09-11T01:47:53.099 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:53.099 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:53.099 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.099 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.099 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:53.099 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:53.099 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:53.099 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:53.099 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:53.099 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:53.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-09-11T01:47:53.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:53.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 1 -- 172.21.15.12:0/27444542 >> v1:172.21.15.71:6789/0 conn(0x7eff7c0036a0 legacy=0x7eff7c005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:53.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:47:53.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8ffff640 1 -- 172.21.15.12:0/27444542 reap_dead start 2024-09-11T01:47:53.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 1 --2- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 0x7eff780039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 10 monclient(hunting): picked mon.c con 0x7eff901361f0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:47:53.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 1 --2- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff780040a0 0x7eff78004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 10 monclient(hunting): picked mon.a con 0x7eff780040a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:47:53.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 1 --2- 172.21.15.12:0/27444542 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff78004e10 0x7eff78005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 10 monclient(hunting): picked mon.b con 0x7eff78004e10 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:47:53.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:53.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 1 -- 172.21.15.12:0/27444542 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7eff9009f9a0 con 0x7eff780040a0 2024-09-11T01:47:53.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:53.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 1 -- 172.21.15.12:0/27444542 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7eff78010130 con 0x7eff78004e10 2024-09-11T01:47:53.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:53.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 1 -- 172.21.15.12:0/27444542 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7eff78010330 con 0x7eff901361f0 2024-09-11T01:47:53.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:53.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:47:53.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8effd640 1 -- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff780040a0 msgr2=0x7eff78004800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:47:53.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8effd640 1 --2- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff780040a0 0x7eff78004800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:47:53.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.092+0000 7eff8f7fe640 1 --2- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 0x7eff780039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:53.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8f7fe640 10 monclient(hunting): get_auth_request con 0x7eff901361f0 auth_method 0 2024-09-11T01:47:53.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:53.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:53.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8ffff640 1 --2- 172.21.15.12:0/27444542 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff78004e10 0x7eff78005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:53.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:47:53.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8ffff640 10 monclient(hunting): get_auth_request con 0x7eff78004e10 auth_method 0 2024-09-11T01:47:53.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:53.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:53.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:47:53.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:53.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:53.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:47:53.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:53.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:53.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:47:53.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8f7fe640 10 monclient(hunting): handle_auth_done global_id 134728 payload 293 2024-09-11T01:47:53.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8f7fe640 1 -- 172.21.15.12:0/27444542 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff78004e10 msgr2=0x7eff78005290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:47:53.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8f7fe640 1 --2- 172.21.15.12:0/27444542 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff78004e10 0x7eff78005290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8f7fe640 1 -- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff780040a0 msgr2=0x7eff78004800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:47:53.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8f7fe640 1 --2- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff780040a0 0x7eff78004800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8f7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:47:53.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8f7fe640 1 monclient: found mon.c 2024-09-11T01:47:53.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:53.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8f7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:47:53.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8f7fe640 1 -- 172.21.15.12:0/27444542 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff78003520 con 0x7eff901361f0 2024-09-11T01:47:53.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8f7fe640 10 monclient: _finish_auth 0 2024-09-11T01:47:53.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8f7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:47:53.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:53.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff95d2b640 5 monclient: authenticate success, global_id 134728 2024-09-11T01:47:53.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff95d2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:47:53.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.093+0000 7eff8ffff640 1 --2- 172.21.15.12:0/27444542 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff78004e10 0x7eff78005290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:47:53.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.096+0000 7eff8f7fe640 1 -- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 msgr2=0x7eff780039b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:53.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.096+0000 7eff8f7fe640 1 --2- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 0x7eff780039b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7eff80006fd0 tx=0x7eff78005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c96bda086a03a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:53.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.096+0000 7eff8e7fc640 1 -- 172.21.15.12:0/27444542 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff80031070 con 0x7eff901361f0 2024-09-11T01:47:53.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.096+0000 7eff8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:53.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.096+0000 7eff8e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:47:53.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.096+0000 7eff8e7fc640 10 monclient: dump: 2024-09-11T01:47:53.110 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:53.110 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:53.110 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.110 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.110 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:53.110 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:53.111 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:53.111 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:53.111 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:53.111 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:53.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.096+0000 7eff8e7fc640 1 -- 172.21.15.12:0/27444542 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff8002bb40 con 0x7eff901361f0 2024-09-11T01:47:53.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.096+0000 7eff95d2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:47:53.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.096+0000 7eff8e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:47:53.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.096+0000 7eff8e7fc640 1 -- 172.21.15.12:0/27444542 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8000dd70 con 0x7eff901361f0 2024-09-11T01:47:53.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.096+0000 7eff95d2b640 10 monclient: get_monmap_and_config success 2024-09-11T01:47:53.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.096+0000 7eff95d2b640 4 set_mon_vals no callback set 2024-09-11T01:47:53.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.096+0000 7eff8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:53.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.096+0000 7eff8e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:47:53.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.096+0000 7eff8e7fc640 10 monclient: dump: 2024-09-11T01:47:53.111 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:53.112 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:53.112 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.112 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.112 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:53.112 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:53.112 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:53.112 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:53.112 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:53.112 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:53.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.097+0000 7eff95d2b640 10 monclient: shutdown 2024-09-11T01:47:53.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.097+0000 7eff95d2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:47:53.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.097+0000 7eff95d2b640 1 -- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 msgr2=0x7eff780039b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:53.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.097+0000 7eff95d2b640 1 --2- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 0x7eff780039b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7eff80006fd0 tx=0x7eff78005830 comp rx=0 tx=0).stop 2024-09-11T01:47:53.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.097+0000 7eff95d2b640 1 -- 172.21.15.12:0/27444542 shutdown_connections 2024-09-11T01:47:53.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.097+0000 7eff95d2b640 1 --2- 172.21.15.12:0/27444542 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff78004e10 0x7eff78005290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.097+0000 7eff95d2b640 1 --2- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff780040a0 0x7eff78004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.097+0000 7eff95d2b640 1 --2- 172.21.15.12:0/27444542 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff901361f0 0x7eff780039b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.097+0000 7eff95d2b640 1 -- 172.21.15.12:0/27444542 >> 172.21.15.12:0/27444542 conn(0x7eff900a2d90 msgr2=0x7eff900a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:47:53.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.097+0000 7eff95d2b640 1 -- 172.21.15.12:0/27444542 shutdown_connections 2024-09-11T01:47:53.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.097+0000 7eff95d2b640 1 -- 172.21.15.12:0/27444542 wait complete. 2024-09-11T01:47:53.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.098+0000 7eff95d2b640 10 monclient: build_initial_monmap 2024-09-11T01:47:53.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.098+0000 7eff95d2b640 10 monclient: monmap: 2024-09-11T01:47:53.113 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:47:53.113 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:47:53.113 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:47:53.113 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:47:53.113 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:47:53.114 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:53.114 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:47:53.114 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:47:53.114 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:47:53.114 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:53.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.099+0000 7eff95d2b640 1 Processor -- start 2024-09-11T01:47:53.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.099+0000 7eff95d2b640 1 -- start start 2024-09-11T01:47:53.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.099+0000 7eff95d2b640 10 monclient: init 2024-09-11T01:47:53.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.099+0000 7eff95d2b640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:53.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.099+0000 7eff95d2b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:47:53.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.099+0000 7eff95d2b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff780040a0 0x7eff901c8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.099+0000 7eff95d2b640 10 monclient(hunting): picked mon.noname-a con 0x7eff780040a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:47:53.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.099+0000 7eff95d2b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff78004e10 0x7eff901c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.099+0000 7eff95d2b640 10 monclient(hunting): picked mon.noname-c con 0x7eff78004e10 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:47:53.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.099+0000 7eff95d2b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff901361f0 0x7eff9011d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.099+0000 7eff95d2b640 10 monclient(hunting): picked mon.noname-b con 0x7eff901361f0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:47:53.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.099+0000 7eff95d2b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:53.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.099+0000 7eff8f7fe640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff780040a0 msgr2=0x7eff901c8b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:47:53.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff95d2b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7eff900a0510 con 0x7eff780040a0 2024-09-11T01:47:53.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff8f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff780040a0 0x7eff901c8b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:47:53.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff95d2b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:53.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff95d2b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7eff900a0390 con 0x7eff901361f0 2024-09-11T01:47:53.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff95d2b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:53.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff95d2b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7eff901366a0 con 0x7eff78004e10 2024-09-11T01:47:53.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff95d2b640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:53.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff95d2b640 10 monclient(hunting): authenticate will time out at 1879.543095s 2024-09-11T01:47:53.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff8effd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff78004e10 0x7eff901c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:53.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff8effd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff78004e10 0x7eff901c90b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55178/0 (socket says 172.21.15.12:55178) 2024-09-11T01:47:53.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff8effd640 1 -- 172.21.15.12:0/525703580 learned_addr learned my addr 172.21.15.12:0/525703580 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:47:53.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff8effd640 10 monclient(hunting): get_auth_request con 0x7eff78004e10 auth_method 0 2024-09-11T01:47:53.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff8effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:53.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff8effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:53.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:53.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff8effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:53.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff8effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:53.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff8effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:53.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff8ffff640 1 --2- 172.21.15.12:0/525703580 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff901361f0 0x7eff9011d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:53.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff8ffff640 10 monclient(hunting): get_auth_request con 0x7eff901361f0 auth_method 0 2024-09-11T01:47:53.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:53.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:53.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:53.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff8effd640 10 monclient(hunting): handle_auth_done global_id 184325 payload 824 2024-09-11T01:47:53.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff8effd640 1 -- 172.21.15.12:0/525703580 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff901361f0 msgr2=0x7eff9011d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:53.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff8effd640 1 --2- 172.21.15.12:0/525703580 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff901361f0 0x7eff9011d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff8effd640 1 -- 172.21.15.12:0/525703580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff780040a0 msgr2=0x7eff901c8b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:47:53.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff8effd640 1 --2- 172.21.15.12:0/525703580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff780040a0 0x7eff901c8b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff8effd640 10 monclient: _finish_hunting 0 2024-09-11T01:47:53.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff8effd640 1 monclient: found mon.noname-c 2024-09-11T01:47:53.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:53.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff8effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:47:53.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.100+0000 7eff8effd640 1 -- 172.21.15.12:0/525703580 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff90132730 con 0x7eff78004e10 2024-09-11T01:47:53.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.102+0000 7eff8effd640 1 -- 172.21.15.12:0/525703580 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff78004e10 msgr2=0x7eff901c90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:53.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.102+0000 7eff8effd640 1 --2- 172.21.15.12:0/525703580 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff78004e10 0x7eff901c90b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7eff8400b780 tx=0x7eff8400bc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:53.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.103+0000 7eff77fff640 1 -- 172.21.15.12:0/525703580 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8400cb40 con 0x7eff78004e10 2024-09-11T01:47:53.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.103+0000 7eff77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:53.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.103+0000 7eff77fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:47:53.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.103+0000 7eff77fff640 10 monclient: dump: 2024-09-11T01:47:53.118 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:53.118 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:53.118 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.118 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.118 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:53.118 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:53.118 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:53.118 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:53.118 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:53.118 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:53.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.103+0000 7eff77fff640 10 monclient: _finish_auth 0 2024-09-11T01:47:53.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.103+0000 7eff77fff640 10 monclient: _check_auth_tickets 2024-09-11T01:47:53.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.103+0000 7eff77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:53.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.103+0000 7eff77fff640 1 -- 172.21.15.12:0/525703580 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff84010030 con 0x7eff78004e10 2024-09-11T01:47:53.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.103+0000 7eff95d2b640 5 monclient: authenticate success, global_id 184325 2024-09-11T01:47:53.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.103+0000 7eff77fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:47:53.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.103+0000 7eff95d2b640 10 monclient: _renew_subs 2024-09-11T01:47:53.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.103+0000 7eff77fff640 1 -- 172.21.15.12:0/525703580 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff84016430 con 0x7eff78004e10 2024-09-11T01:47:53.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.103+0000 7eff95d2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:47:53.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.103+0000 7eff95d2b640 1 -- 172.21.15.12:0/525703580 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff9009d480 con 0x7eff78004e10 2024-09-11T01:47:53.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.103+0000 7eff77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:53.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.103+0000 7eff77fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:47:53.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.103+0000 7eff77fff640 10 monclient: dump: 2024-09-11T01:47:53.119 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:53.120 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:53.120 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.120 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.120 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:53.120 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:53.120 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:53.120 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:53.120 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:53.120 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:53.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.103+0000 7eff8dffb640 4 set_mon_vals no callback set 2024-09-11T01:47:53.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.103+0000 7eff95d2b640 10 monclient: _renew_subs 2024-09-11T01:47:53.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.103+0000 7eff95d2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:47:53.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.103+0000 7eff95d2b640 1 -- 172.21.15.12:0/525703580 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7eff9011f950 con 0x7eff78004e10 2024-09-11T01:47:53.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.104+0000 7eff77fff640 1 -- 172.21.15.12:0/525703580 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7eff84016650 con 0x7eff78004e10 2024-09-11T01:47:53.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.104+0000 7eff77fff640 1 --2- 172.21.15.12:0/525703580 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7eff7c03e9a0 0x7eff7c040e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.104+0000 7eff77fff640 1 -- 172.21.15.12:0/525703580 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(619..619 src has 507..619) ==== 4803+0+0 (secure 0 0 0) 0x7eff84052a00 con 0x7eff78004e10 2024-09-11T01:47:53.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.104+0000 7eff95d2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:53.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.104+0000 7eff95d2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:53.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.104+0000 7eff95d2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:47:53.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.104+0000 7eff95d2b640 1 -- 172.21.15.12:0/525703580 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff90135ae0 con 0x7eff78004e10 2024-09-11T01:47:53.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.105+0000 7eff8f7fe640 1 --2- 172.21.15.12:0/525703580 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7eff7c03e9a0 0x7eff7c040e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:53.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.105+0000 7eff8f7fe640 10 monclient: get_auth_request con 0x7eff7c03e9a0 auth_method 0 2024-09-11T01:47:53.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.107+0000 7eff8f7fe640 1 --2- 172.21.15.12:0/525703580 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7eff7c03e9a0 0x7eff7c040e60 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7eff78010500 tx=0x7eff80031000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:53.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.110+0000 7eff77fff640 1 -- 172.21.15.12:0/525703580 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7eff8401b020 con 0x7eff78004e10 2024-09-11T01:47:53.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.110+0000 7eff77fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:53.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.110+0000 7eff77fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:47:53.157 INFO:tasks.rados.rados.0.smithi071.stdout:2763: done (4 left) 2024-09-11T01:47:53.157 INFO:tasks.rados.rados.0.smithi071.stdout:2764: read oid 33 snap -1 2024-09-11T01:47:53.157 INFO:tasks.rados.rados.0.smithi071.stdout:2764: expect (ObjNum 979 snap 306 seq_num 979) 2024-09-11T01:47:53.157 INFO:tasks.rados.rados.0.smithi071.stdout:2765: snap_remove snap 301 2024-09-11T01:47:53.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.374+0000 7eff95d2b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:47:53.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.374+0000 7eff95d2b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:47:53.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.374+0000 7eff95d2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:47:53.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.374+0000 7eff95d2b640 1 -- 172.21.15.12:0/525703580 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7eff9011edf0 con 0x7eff78004e10 2024-09-11T01:47:53.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.374+0000 7eff77fff640 1 -- 172.21.15.12:0/525703580 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7eff84012db0 con 0x7eff78004e10 2024-09-11T01:47:53.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.374+0000 7eff77fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:47:53.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.374+0000 7eff77fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:47:53.377 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":174,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:47:53.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.376+0000 7eff95d2b640 1 -- 172.21.15.12:0/525703580 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7eff7c03e9a0 msgr2=0x7eff7c040e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:53.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.376+0000 7eff95d2b640 1 --2- 172.21.15.12:0/525703580 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7eff7c03e9a0 0x7eff7c040e60 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7eff78010500 tx=0x7eff80031000 comp rx=0 tx=0).stop 2024-09-11T01:47:53.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.376+0000 7eff95d2b640 10 monclient: shutdown 2024-09-11T01:47:53.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.376+0000 7eff95d2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:47:53.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.376+0000 7eff95d2b640 1 -- 172.21.15.12:0/525703580 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff78004e10 msgr2=0x7eff901c90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:53.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.376+0000 7eff95d2b640 1 --2- 172.21.15.12:0/525703580 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff78004e10 0x7eff901c90b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7eff8400b780 tx=0x7eff8400bc40 comp rx=0 tx=0).stop 2024-09-11T01:47:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.376+0000 7eff95d2b640 1 -- 172.21.15.12:0/525703580 shutdown_connections 2024-09-11T01:47:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.376+0000 7eff95d2b640 1 --2- 172.21.15.12:0/525703580 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff901361f0 0x7eff9011d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.376+0000 7eff95d2b640 1 --2- 172.21.15.12:0/525703580 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff78004e10 0x7eff901c90b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.376+0000 7eff95d2b640 1 --2- 172.21.15.12:0/525703580 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7eff7c03e9a0 0x7eff7c040e60 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.376+0000 7eff95d2b640 1 --2- 172.21.15.12:0/525703580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff780040a0 0x7eff901c8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.376+0000 7eff95d2b640 1 -- 172.21.15.12:0/525703580 >> 172.21.15.12:0/525703580 conn(0x7eff900a2d90 msgr2=0x7eff9009e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:47:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.376+0000 7eff95d2b640 1 -- 172.21.15.12:0/525703580 shutdown_connections 2024-09-11T01:47:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.376+0000 7eff95d2b640 1 -- 172.21.15.12:0/525703580 wait complete. 2024-09-11T01:47:53.390 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:47:53.390 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T01:47:53.390 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:47:53.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.490+0000 7f13a15a1640 10 monclient: get_monmap_and_config 2024-09-11T01:47:53.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.490+0000 7f13a15a1640 10 monclient: build_initial_monmap 2024-09-11T01:47:53.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.490+0000 7f13a15a1640 10 monclient: monmap: 2024-09-11T01:47:53.493 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:47:53.493 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:53.493 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:47:53.492213+0000 2024-09-11T01:47:53.493 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:47:53.492213+0000 2024-09-11T01:47:53.493 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:47:53.493 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:53.493 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:47:53.494 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:47:53.494 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:47:53.494 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:47:53.494 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:47:53.494 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:53.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.491+0000 7f13a15a1640 1 Processor -- start 2024-09-11T01:47:53.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.491+0000 7f13a15a1640 1 -- start start 2024-09-11T01:47:53.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.491+0000 7f13a15a1640 10 monclient: init 2024-09-11T01:47:53.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.491+0000 7f13a15a1640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:53.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.491+0000 7f13a15a1640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-11T01:47:53.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.491+0000 7f13a15a1640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f139c136fb0 0x7f139c137390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.491+0000 7f13a15a1640 10 monclient(hunting): picked mon.noname-a con 0x7f139c136fb0 addr v2:172.21.15.12:3300/0 2024-09-11T01:47:53.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.491+0000 7f13a15a1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f139c09cff0 addr v1:172.21.15.71:6789/0 2024-09-11T01:47:53.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.491+0000 7f13a15a1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f139c09d460 addr v1:172.21.15.12:6789/0 2024-09-11T01:47:53.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.491+0000 7f13a15a1640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f139c138b00 con 0x7f139c09d460 2024-09-11T01:47:53.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.491+0000 7f13a15a1640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f139c0a64e0 con 0x7f139c09cff0 2024-09-11T01:47:53.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.491+0000 7f13a15a1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:53.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.491+0000 7f13a15a1640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f139c09c330 con 0x7f139c136fb0 2024-09-11T01:47:53.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.491+0000 7f13a15a1640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:53.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.491+0000 7f13a15a1640 10 monclient(hunting): authenticate will time out at 1879.934808s 2024-09-11T01:47:53.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.491+0000 7f139bfff640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7f139c136fb0 msgr2=0x7f139c137390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:47:53.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.491+0000 7f13a0da0640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7f139c09d460 legacy=0x7f139c1383f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:47:53.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.491+0000 7f139bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f139c136fb0 0x7f139c137390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:47:53.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.492+0000 7f139b7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f139c09cff0 0x7f139c137ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.12:47522/0 (socket says 172.21.15.12:47522) 2024-09-11T01:47:53.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.492+0000 7f139b7fe640 1 -- 172.21.15.12:0/3281377181 learned_addr learned my addr 172.21.15.12:0/3281377181 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:47:53.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.494+0000 7f139b7fe640 1 -- 172.21.15.12:0/3281377181 >> v1:172.21.15.71:6789/0 conn(0x7f139c09cff0 legacy=0x7f139c137ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:53.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.494+0000 7f139affd640 1 -- 172.21.15.12:0/3281377181 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1400820355 0 0) 0x7f139c0a64e0 con 0x7f139c09cff0 2024-09-11T01:47:53.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.494+0000 7f139affd640 10 monclient(hunting): _negotiate 2024-09-11T01:47:53.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.494+0000 7f139affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:53.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.494+0000 7f139affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:53.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.494+0000 7f139affd640 10 monclient(hunting): my global_id is 134734 2024-09-11T01:47:53.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.494+0000 7f139affd640 1 -- 172.21.15.12:0/3281377181 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1384003520 con 0x7f139c09cff0 2024-09-11T01:47:53.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 1 -- 172.21.15.12:0/3281377181 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 588944324 0 0) 0x7f1384003520 con 0x7f139c09cff0 2024-09-11T01:47:53.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 1 -- 172.21.15.12:0/3281377181 >> v2:172.21.15.12:3300/0 conn(0x7f139c136fb0 msgr2=0x7f139c137390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:47:53.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 1 --2- 172.21.15.12:0/3281377181 >> v2:172.21.15.12:3300/0 conn(0x7f139c136fb0 0x7f139c137390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 1 -- 172.21.15.12:0/3281377181 >> v1:172.21.15.12:6789/0 conn(0x7f139c09d460 legacy=0x7f139c1383f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:47:53.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 10 monclient: _finish_hunting 0 2024-09-11T01:47:53.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 1 monclient: found mon.noname-c-legacy 2024-09-11T01:47:53.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:53.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-09-11T01:47:53.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 1 -- 172.21.15.12:0/3281377181 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f139c09c610 con 0x7f139c09cff0 2024-09-11T01:47:53.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 10 monclient: _finish_auth 0 2024-09-11T01:47:53.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 10 monclient: _check_auth_tickets 2024-09-11T01:47:53.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:53.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 1 -- 172.21.15.12:0/3281377181 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f1390001a60 con 0x7f139c09cff0 2024-09-11T01:47:53.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f13a15a1640 5 monclient: authenticate success, global_id 134734 2024-09-11T01:47:53.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f13a15a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:47:53.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:53.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-11T01:47:53.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 10 monclient: dump: 2024-09-11T01:47:53.502 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:53.502 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:53.502 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.502 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.502 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:53.502 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:53.502 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:53.502 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:53.502 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:53.502 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:53.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-09-11T01:47:53.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:53.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 1 -- 172.21.15.12:0/3281377181 >> v1:172.21.15.71:6789/0 conn(0x7f139c09cff0 legacy=0x7f139c137ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:53.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:47:53.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 1 --2- 172.21.15.12:0/3281377181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13840039b0 0x7f1384003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 10 monclient(hunting): picked mon.c con 0x7f13840039b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:47:53.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 1 --2- 172.21.15.12:0/3281377181 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1384004800 0x7f1384004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 10 monclient(hunting): picked mon.b con 0x7f1384004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:47:53.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 1 --2- 172.21.15.12:0/3281377181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13840053c0 0x7f138400ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 10 monclient(hunting): picked mon.a con 0x7f13840053c0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:47:53.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:53.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 1 -- 172.21.15.12:0/3281377181 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f139c09c330 con 0x7f13840053c0 2024-09-11T01:47:53.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:53.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 1 -- 172.21.15.12:0/3281377181 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f138400c300 con 0x7f1384004800 2024-09-11T01:47:53.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:53.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 1 -- 172.21.15.12:0/3281377181 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f138400c500 con 0x7f13840039b0 2024-09-11T01:47:53.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:53.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139affd640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:47:53.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f13a15a1640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:47:53.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f13a0da0640 1 -- 172.21.15.12:0/3281377181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13840053c0 msgr2=0x7f138400ba20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:47:53.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f13a0da0640 1 --2- 172.21.15.12:0/3281377181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13840053c0 0x7f138400ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:47:53.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139bfff640 1 --2- 172.21.15.12:0/3281377181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13840039b0 0x7f1384003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:53.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139bfff640 10 monclient(hunting): get_auth_request con 0x7f13840039b0 auth_method 0 2024-09-11T01:47:53.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:53.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:53.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:47:53.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139b7fe640 1 --2- 172.21.15.12:0/3281377181 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1384004800 0x7f1384004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:53.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139b7fe640 10 monclient(hunting): get_auth_request con 0x7f1384004800 auth_method 0 2024-09-11T01:47:53.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:53.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:53.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:47:53.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:53.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:53.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:47:53.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:53.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:53.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:47:53.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139bfff640 10 monclient(hunting): handle_auth_done global_id 134734 payload 293 2024-09-11T01:47:53.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139bfff640 1 -- 172.21.15.12:0/3281377181 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1384004800 msgr2=0x7f1384004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:47:53.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139bfff640 1 --2- 172.21.15.12:0/3281377181 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1384004800 0x7f1384004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139bfff640 1 -- 172.21.15.12:0/3281377181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13840053c0 msgr2=0x7f138400ba20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:47:53.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139bfff640 1 --2- 172.21.15.12:0/3281377181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13840053c0 0x7f138400ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:47:53.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139bfff640 1 monclient: found mon.c 2024-09-11T01:47:53.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f13a0da0640 1 -- 172.21.15.12:0/3281377181 reap_dead start 2024-09-11T01:47:53.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:53.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139bfff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:47:53.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139bfff640 1 -- 172.21.15.12:0/3281377181 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1384003520 con 0x7f13840039b0 2024-09-11T01:47:53.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139bfff640 10 monclient: _finish_auth 0 2024-09-11T01:47:53.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:47:53.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.495+0000 7f139bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:53.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.498+0000 7f139bfff640 1 -- 172.21.15.12:0/3281377181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13840039b0 msgr2=0x7f1384003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:53.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.498+0000 7f139bfff640 1 --2- 172.21.15.12:0/3281377181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13840039b0 0x7f1384003db0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f138c009900 tx=0x7f1384005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61adee4e3d3a78fc server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:53.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.498+0000 7f139affd640 1 -- 172.21.15.12:0/3281377181 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138c009500 con 0x7f13840039b0 2024-09-11T01:47:53.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.498+0000 7f139affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:53.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.498+0000 7f139affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:47:53.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.498+0000 7f139affd640 10 monclient: dump: 2024-09-11T01:47:53.514 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:53.514 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:53.514 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.514 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.514 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:53.514 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:53.514 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:53.514 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:53.514 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:53.514 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:53.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.498+0000 7f139affd640 1 -- 172.21.15.12:0/3281377181 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f138c02e070 con 0x7f13840039b0 2024-09-11T01:47:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.498+0000 7f139affd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:47:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.498+0000 7f139affd640 1 -- 172.21.15.12:0/3281377181 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138c02a910 con 0x7f13840039b0 2024-09-11T01:47:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.498+0000 7f139affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.498+0000 7f139affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:47:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.498+0000 7f139affd640 10 monclient: dump: 2024-09-11T01:47:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:53.516 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:53.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.498+0000 7f13a15a1640 10 monclient: get_monmap_and_config success 2024-09-11T01:47:53.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.498+0000 7f13a15a1640 4 set_mon_vals no callback set 2024-09-11T01:47:53.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.499+0000 7f13a15a1640 10 monclient: shutdown 2024-09-11T01:47:53.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.499+0000 7f13a15a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:47:53.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.499+0000 7f13a15a1640 1 -- 172.21.15.12:0/3281377181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13840039b0 msgr2=0x7f1384003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:53.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.499+0000 7f13a15a1640 1 --2- 172.21.15.12:0/3281377181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13840039b0 0x7f1384003db0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f138c009900 tx=0x7f1384005830 comp rx=0 tx=0).stop 2024-09-11T01:47:53.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.499+0000 7f13a15a1640 1 -- 172.21.15.12:0/3281377181 shutdown_connections 2024-09-11T01:47:53.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.499+0000 7f13a15a1640 1 --2- 172.21.15.12:0/3281377181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f13840039b0 0x7f1384003db0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.499+0000 7f13a15a1640 1 -- 172.21.15.12:0/3281377181 >> 172.21.15.12:0/3281377181 conn(0x7f139c0ab190 msgr2=0x7f139c0a6fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:47:53.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.499+0000 7f13a15a1640 1 -- 172.21.15.12:0/3281377181 shutdown_connections 2024-09-11T01:47:53.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.499+0000 7f13a15a1640 1 -- 172.21.15.12:0/3281377181 wait complete. 2024-09-11T01:47:53.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.499+0000 7f13a15a1640 10 monclient: build_initial_monmap 2024-09-11T01:47:53.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.499+0000 7f13a15a1640 10 monclient: monmap: 2024-09-11T01:47:53.517 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:47:53.517 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:47:53.517 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:47:53.517 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:47:53.517 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:47:53.517 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:53.517 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:47:53.517 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:47:53.517 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:47:53.517 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:53.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f13a15a1640 1 Processor -- start 2024-09-11T01:47:53.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f13a15a1640 1 -- start start 2024-09-11T01:47:53.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f13a15a1640 10 monclient: init 2024-09-11T01:47:53.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f13a15a1640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:53.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f13a15a1640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:47:53.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f13a15a1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13840039b0 0x7f139c135d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f13a15a1640 10 monclient(hunting): picked mon.noname-b con 0x7f13840039b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:47:53.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f13a15a1640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f139c1ccfb0 0x7f139c136290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f13a15a1640 10 monclient(hunting): picked mon.noname-c con 0x7f139c1ccfb0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:47:53.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f13a15a1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f139c123970 0x7f139c123e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f13a15a1640 10 monclient(hunting): picked mon.noname-a con 0x7f139c123970 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:47:53.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f13a15a1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:53.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f13a15a1640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f139c0a7b50 con 0x7f139c123970 2024-09-11T01:47:53.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f13a15a1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:53.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f13a15a1640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f139c0a79d0 con 0x7f13840039b0 2024-09-11T01:47:53.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f13a15a1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:53.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f13a15a1640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f139c137390 con 0x7f139c1ccfb0 2024-09-11T01:47:53.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f13a15a1640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:53.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f13a15a1640 10 monclient(hunting): authenticate will time out at 1879.943821s 2024-09-11T01:47:53.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f13a0da0640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f139c123970 msgr2=0x7f139c123e00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:47:53.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f13a0da0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f139c123970 0x7f139c123e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:47:53.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f139b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f139c1ccfb0 0x7f139c136290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:53.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f139b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f139c1ccfb0 0x7f139c136290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55200/0 (socket says 172.21.15.12:55200) 2024-09-11T01:47:53.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f139b7fe640 1 -- 172.21.15.12:0/2298910405 learned_addr learned my addr 172.21.15.12:0/2298910405 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:47:53.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f139b7fe640 10 monclient(hunting): get_auth_request con 0x7f139c1ccfb0 auth_method 0 2024-09-11T01:47:53.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f139b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:53.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f139b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:53.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.500+0000 7f139b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:53.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.501+0000 7f139b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:53.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.501+0000 7f139b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.501+0000 7f139b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.501+0000 7f139bfff640 1 --2- 172.21.15.12:0/2298910405 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13840039b0 0x7f139c135d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.501+0000 7f139b7fe640 10 monclient(hunting): handle_auth_done global_id 184328 payload 824 2024-09-11T01:47:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.501+0000 7f139b7fe640 1 -- 172.21.15.12:0/2298910405 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13840039b0 msgr2=0x7f139c135d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.501+0000 7f139b7fe640 1 --2- 172.21.15.12:0/2298910405 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13840039b0 0x7f139c135d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.501+0000 7f139b7fe640 1 -- 172.21.15.12:0/2298910405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f139c123970 msgr2=0x7f139c123e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:47:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.501+0000 7f139b7fe640 1 --2- 172.21.15.12:0/2298910405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f139c123970 0x7f139c123e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.501+0000 7f139b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:47:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.501+0000 7f139b7fe640 1 monclient: found mon.noname-c 2024-09-11T01:47:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.501+0000 7f139b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.501+0000 7f139b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:47:53.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.501+0000 7f139b7fe640 1 -- 172.21.15.12:0/2298910405 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f139c127750 con 0x7f139c1ccfb0 2024-09-11T01:47:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.501+0000 7f139bfff640 10 monclient: get_auth_request con 0x7f13840039b0 auth_method 0 2024-09-11T01:47:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.501+0000 7f139bfff640 1 --2- 172.21.15.12:0/2298910405 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13840039b0 0x7f139c135d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:47:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.503+0000 7f139b7fe640 1 -- 172.21.15.12:0/2298910405 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f139c1ccfb0 msgr2=0x7f139c136290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.503+0000 7f139b7fe640 1 --2- 172.21.15.12:0/2298910405 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f139c1ccfb0 0x7f139c136290 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1390007310 tx=0x7f13900077d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.503+0000 7f139b7fe640 1 --2- 172.21.15.12:0/2298910405 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f139c1ccfb0 0x7f139c136290 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1390007310 tx=0x7f13900077d0 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7f1390013630 mon_map magic: 0 2024-09-11T01:47:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.503+0000 7f1398ff9640 1 -- 172.21.15.12:0/2298910405 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139000fcb0 con 0x7f139c1ccfb0 2024-09-11T01:47:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.503+0000 7f1398ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.503+0000 7f1398ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:47:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.503+0000 7f1398ff9640 10 monclient: dump: 2024-09-11T01:47:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.521 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:53.522 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.503+0000 7f1398ff9640 10 monclient: _finish_auth 0 2024-09-11T01:47:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.503+0000 7f1398ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:47:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.503+0000 7f1398ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.503+0000 7f1398ff9640 1 -- 172.21.15.12:0/2298910405 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1390018070 con 0x7f139c1ccfb0 2024-09-11T01:47:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.503+0000 7f13a15a1640 5 monclient: authenticate success, global_id 184328 2024-09-11T01:47:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.503+0000 7f13a15a1640 10 monclient: _renew_subs 2024-09-11T01:47:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.503+0000 7f13a15a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:47:53.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.503+0000 7f13a15a1640 1 -- 172.21.15.12:0/2298910405 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f139c1367d0 con 0x7f139c1ccfb0 2024-09-11T01:47:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.503+0000 7f1398ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:47:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.503+0000 7f13a15a1640 10 monclient: _renew_subs 2024-09-11T01:47:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.503+0000 7f13a15a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:47:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.503+0000 7f13a15a1640 1 -- 172.21.15.12:0/2298910405 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f139c125f50 con 0x7f139c1ccfb0 2024-09-11T01:47:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.503+0000 7f1399ffb640 4 set_mon_vals no callback set 2024-09-11T01:47:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.504+0000 7f13a15a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.504+0000 7f13a15a1640 1 --2- 172.21.15.12:0/2298910405 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f139c124340 0x7f139c0a2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.504+0000 7f13a15a1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.504+0000 7f13a15a1640 1 -- 172.21.15.12:0/2298910405 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f139c137710 con 0x7f139c124340 2024-09-11T01:47:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.504+0000 7f139bfff640 1 --2- 172.21.15.12:0/2298910405 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f139c124340 0x7f139c0a2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.504+0000 7f139bfff640 10 monclient: get_auth_request con 0x7f139c124340 auth_method 0 2024-09-11T01:47:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.504+0000 7f139bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.504+0000 7f139bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:53.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.504+0000 7f139bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:53.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.504+0000 7f1398ff9640 1 -- 172.21.15.12:0/2298910405 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1390013630 con 0x7f139c1ccfb0 2024-09-11T01:47:53.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.504+0000 7f1398ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:53.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.504+0000 7f1398ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:47:53.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.504+0000 7f1398ff9640 10 monclient: dump: 2024-09-11T01:47:53.524 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:53.524 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:53.524 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.524 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.524 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:53.524 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:53.524 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:53.524 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:53.524 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:53.524 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:53.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.504+0000 7f1398ff9640 1 -- 172.21.15.12:0/2298910405 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f13900137d0 con 0x7f139c1ccfb0 2024-09-11T01:47:53.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.504+0000 7f139bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:53.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.504+0000 7f139bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:53.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.504+0000 7f139bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:53.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.504+0000 7f139bfff640 10 monclient(hunting): handle_auth_done global_id 184331 payload 824 2024-09-11T01:47:53.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.504+0000 7f139bfff640 1 -- 172.21.15.12:0/2298910405 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f139c0a16a0 con 0x7f139c124340 2024-09-11T01:47:53.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.504+0000 7f1398ff9640 1 --2- 172.21.15.12:0/2298910405 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f138803e1c0 0x7f1388040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.505+0000 7f13a0da0640 1 --2- 172.21.15.12:0/2298910405 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f138803e1c0 0x7f1388040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:53.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.505+0000 7f13a0da0640 10 monclient: get_auth_request con 0x7f138803e1c0 auth_method 0 2024-09-11T01:47:53.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.507+0000 7f139bfff640 1 -- 172.21.15.12:0/2298910405 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f139c124340 msgr2=0x7f139c0a2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:53.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.507+0000 7f139bfff640 1 --2- 172.21.15.12:0/2298910405 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f139c124340 0x7f139c0a2250 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f138c032000 tx=0x7f138c005a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:53.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.507+0000 7f1398ff9640 1 -- 172.21.15.12:0/2298910405 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138c042020 con 0x7f139c124340 2024-09-11T01:47:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.507+0000 7f1398ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.507+0000 7f1398ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:47:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.507+0000 7f1398ff9640 10 monclient: dump: 2024-09-11T01:47:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:53.526 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:53.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.507+0000 7f1398ff9640 1 -- 172.21.15.12:0/2298910405 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f138c02e040 con 0x7f139c124340 2024-09-11T01:47:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.507+0000 7f1398ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.507+0000 7f1398ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:47:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.507+0000 7f1398ff9640 1 -- 172.21.15.12:0/2298910405 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f139c124340 msgr2=0x7f139c0a2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.507+0000 7f1398ff9640 1 --2- 172.21.15.12:0/2298910405 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f139c124340 0x7f139c0a2250 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f138c032000 tx=0x7f138c005a10 comp rx=0 tx=0).stop 2024-09-11T01:47:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.507+0000 7f1398ff9640 1 -- 172.21.15.12:0/2298910405 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f139c124340 msgr2=0x7f139c0a2250 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:47:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.507+0000 7f1398ff9640 1 --2- 172.21.15.12:0/2298910405 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f139c124340 0x7f139c0a2250 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f138c032000 tx=0x7f138c005a10 comp rx=0 tx=0).stop 2024-09-11T01:47:53.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.507+0000 7f13a0da0640 1 --2- 172.21.15.12:0/2298910405 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f138803e1c0 0x7f1388040680 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f139c09d840 tx=0x7f1394009170 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:53.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.513+0000 7f13a15a1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:47:53.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.513+0000 7f13a15a1640 1 --2- 172.21.15.12:0/2298910405 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f139c0a2790 0x7f139c0a2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.513+0000 7f13a15a1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:53.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.513+0000 7f13a15a1640 1 -- 172.21.15.12:0/2298910405 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f139c1273d0 con 0x7f139c0a2790 2024-09-11T01:47:53.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.513+0000 7f139bfff640 1 --2- 172.21.15.12:0/2298910405 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f139c0a2790 0x7f139c0a2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:53.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.513+0000 7f139bfff640 10 monclient: get_auth_request con 0x7f139c0a2790 auth_method 0 2024-09-11T01:47:53.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.513+0000 7f139bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:53.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.513+0000 7f139bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:53.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.513+0000 7f139bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:53.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.513+0000 7f139bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:53.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.513+0000 7f139bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:53.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.513+0000 7f139bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:53.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.514+0000 7f139bfff640 10 monclient(hunting): handle_auth_done global_id 184334 payload 824 2024-09-11T01:47:53.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.514+0000 7f139bfff640 1 -- 172.21.15.12:0/2298910405 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f139c124720 con 0x7f139c0a2790 2024-09-11T01:47:53.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.516+0000 7f139bfff640 1 -- 172.21.15.12:0/2298910405 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f139c0a2790 msgr2=0x7f139c0a2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:53.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.516+0000 7f139bfff640 1 --2- 172.21.15.12:0/2298910405 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f139c0a2790 0x7f139c0a2ba0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f138c009900 tx=0x7f138c008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:53.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.516+0000 7f1398ff9640 1 -- 172.21.15.12:0/2298910405 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138c03e670 con 0x7f139c0a2790 2024-09-11T01:47:53.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.516+0000 7f1398ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.516+0000 7f1398ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:47:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.516+0000 7f1398ff9640 10 monclient: dump: 2024-09-11T01:47:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:53.531 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.516+0000 7f1398ff9640 1 -- 172.21.15.12:0/2298910405 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4531 (secure 0 0 0) 0x7f138c03e670 con 0x7f139c0a2790 2024-09-11T01:47:53.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.516+0000 7f1398ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:47:53.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.516+0000 7f1398ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:47:53.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.516+0000 7f1398ff9640 1 -- 172.21.15.12:0/2298910405 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f139c0a2790 msgr2=0x7f139c0a2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:53.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.516+0000 7f1398ff9640 1 --2- 172.21.15.12:0/2298910405 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f139c0a2790 0x7f139c0a2ba0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f138c009900 tx=0x7f138c008030 comp rx=0 tx=0).stop 2024-09-11T01:47:53.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.516+0000 7f1398ff9640 1 -- 172.21.15.12:0/2298910405 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f139c0a2790 msgr2=0x7f139c0a2ba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:47:53.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.516+0000 7f1398ff9640 1 --2- 172.21.15.12:0/2298910405 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f139c0a2790 0x7f139c0a2ba0 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f138c009900 tx=0x7f138c008030 comp rx=0 tx=0).stop 2024-09-11T01:47:53.532 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:47:53.532 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:47:53.532 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:47:53.532 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:47:53.533 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 174, 2024-09-11T01:47:53.533 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 134110, 2024-09-11T01:47:53.533 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:47:53.533 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:47:53.533 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:47:53.533 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:53.533 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:47:53.533 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:47:53.533 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:47:53.533 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:47:53.533 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:47:53.533 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:47:53.533 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:47:53.534 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:47:53.534 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:47:53.534 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:47:53.534 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:47:53.534 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:47:53.534 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:47:53.534 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:47:53.534 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:47:53.534 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:53.534 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:47:53.534 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:47:53.534 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:47:53.534 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:47:53.534 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:47:53.535 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:47:53.535 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:47:53.535 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:47:53.535 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:47:53.535 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:47:53.535 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:47:53.535 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:47:53.535 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:47:53.535 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:53.535 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:53.535 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:47:53.535 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:47:53.535 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:47:53.536 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:47:53.536 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:47:53.536 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:47:53.536 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:47:53.536 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:47:53.536 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:47:53.536 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:47:53.536 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:47:53.536 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:47:53.536 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:47:53.536 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:47:53.536 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:47:53.536 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:47:53.537 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:47:53.537 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:47:53.537 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:47:53.537 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:47:53.537 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:47:53.537 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:47:53.537 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:47:53.537 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:47:53.537 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:47:53.537 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:47:53.537 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:47:53.537 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:47:53.537 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:53.538 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:47:53.538 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:53.538 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:47:53.538 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:53.538 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:47:53.538 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:47:53.538 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:47:53.538 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:47:53.538 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:53.538 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:47:53.538 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:47:53.538 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:53.538 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:53.538 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:53.539 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:47:53.539 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:47:53.539 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:53.539 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:53.539 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:53.539 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:53.539 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:47:53.539 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:47:53.539 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:47:53.539 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:47:53.539 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:47:53.539 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:53.539 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:53.540 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:47:53.540 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:47:53.540 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:47:53.540 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:47:53.540 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:53.540 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:47:53.540 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:47:53.540 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:53.540 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:53.540 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:53.540 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:47:53.540 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:47:53.540 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:53.541 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:53.541 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:53.541 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:53.541 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:47:53.541 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:47:53.541 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:47:53.541 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:47:53.541 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:47:53.541 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:53.541 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:53.541 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:47:53.541 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:47:53.541 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:47:53.541 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:47:53.542 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:53.542 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:47:53.542 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:47:53.542 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:53.542 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:53.542 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:53.542 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:47:53.542 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:47:53.542 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:53.542 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:53.542 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:53.542 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:53.542 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:47:53.542 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:47:53.543 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:47:53.543 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:47:53.543 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:47:53.543 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:53.543 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:53.543 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:53.543 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:47:53.543 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:47:53.543 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:53.543 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:47:53.543 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:47:53.543 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:47:53.543 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:53.544 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:53.544 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:47:53.544 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:53.544 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:47:53.544 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:47:53.544 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:47:53.544 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:53.544 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:53.544 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:53.544 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:47:53.544 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:47:53.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.516+0000 7f13a15a1640 1 -- 172.21.15.12:0/2298910405 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f138803e1c0 msgr2=0x7f1388040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:53.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.517+0000 7f13a15a1640 1 --2- 172.21.15.12:0/2298910405 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f138803e1c0 0x7f1388040680 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f139c09d840 tx=0x7f1394009170 comp rx=0 tx=0).stop 2024-09-11T01:47:53.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.517+0000 7f13a15a1640 10 monclient: shutdown 2024-09-11T01:47:53.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.517+0000 7f13a15a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:47:53.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.517+0000 7f13a15a1640 1 -- 172.21.15.12:0/2298910405 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f139c1ccfb0 msgr2=0x7f139c136290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:53.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.517+0000 7f13a15a1640 1 --2- 172.21.15.12:0/2298910405 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f139c1ccfb0 0x7f139c136290 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1390007310 tx=0x7f13900077d0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.517+0000 7f13a0da0640 1 -- 172.21.15.12:0/2298910405 reap_dead start 2024-09-11T01:47:53.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.517+0000 7f1398ff9640 1 -- 172.21.15.12:0/2298910405 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(620..620 src has 507..620) ==== 4847+0+0 (unknown 0 0 0) 0x7f1390052f90 con 0x7f139c1ccfb0 2024-09-11T01:47:53.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.517+0000 7f13a15a1640 1 -- 172.21.15.12:0/2298910405 shutdown_connections 2024-09-11T01:47:53.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.517+0000 7f13a15a1640 1 --2- 172.21.15.12:0/2298910405 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f139c1ccfb0 0x7f139c136290 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.517+0000 7f13a15a1640 1 -- 172.21.15.12:0/2298910405 >> 172.21.15.12:0/2298910405 conn(0x7f139c0ab190 msgr2=0x7f139c137920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:47:53.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.517+0000 7f13a15a1640 1 -- 172.21.15.12:0/2298910405 shutdown_connections 2024-09-11T01:47:53.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.517+0000 7f13a15a1640 1 -- 172.21.15.12:0/2298910405 wait complete. 2024-09-11T01:47:53.547 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:47:53.547 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T01:47:53.547 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:47:53.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.646+0000 7fcbb95da640 10 monclient: get_monmap_and_config 2024-09-11T01:47:53.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.646+0000 7fcbb95da640 10 monclient: build_initial_monmap 2024-09-11T01:47:53.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.646+0000 7fcbb95da640 10 monclient: monmap: 2024-09-11T01:47:53.649 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:47:53.649 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:53.650 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:47:53.648526+0000 2024-09-11T01:47:53.650 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:47:53.648526+0000 2024-09-11T01:47:53.650 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:47:53.650 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:53.650 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:47:53.650 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:47:53.650 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:47:53.650 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:47:53.650 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:47:53.650 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:53.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.647+0000 7fcbb95da640 1 Processor -- start 2024-09-11T01:47:53.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.647+0000 7fcbb95da640 1 -- start start 2024-09-11T01:47:53.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.647+0000 7fcbb95da640 10 monclient: init 2024-09-11T01:47:53.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.647+0000 7fcbb95da640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:53.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.647+0000 7fcbb95da640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-11T01:47:53.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.647+0000 7fcbb95da640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcbb4134ad0 addr v1:172.21.15.71:6789/0 2024-09-11T01:47:53.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.648+0000 7fcbb95da640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcbb40a2200 addr v1:172.21.15.12:6789/0 2024-09-11T01:47:53.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.648+0000 7fcbb95da640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcbb41355c0 0x7fcbb413d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.648+0000 7fcbb95da640 10 monclient(hunting): picked mon.noname-a con 0x7fcbb41355c0 addr v2:172.21.15.12:3300/0 2024-09-11T01:47:53.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.648+0000 7fcbb95da640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbb409ed30 con 0x7fcbb40a2200 2024-09-11T01:47:53.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.648+0000 7fcbb27fc640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7fcbb40a2200 legacy=0x7fcbb413a000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:47:53.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.648+0000 7fcbb95da640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbb409f010 con 0x7fcbb4134ad0 2024-09-11T01:47:53.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.648+0000 7fcbb95da640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:53.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.648+0000 7fcbb95da640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fcbb409f5d0 con 0x7fcbb41355c0 2024-09-11T01:47:53.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.648+0000 7fcbb95da640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:53.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.648+0000 7fcbb95da640 10 monclient(hunting): authenticate will time out at 1880.091124s 2024-09-11T01:47:53.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.648+0000 7fcbb37fe640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7fcbb41355c0 msgr2=0x7fcbb413d730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:47:53.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.648+0000 7fcbb37fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fcbb41355c0 0x7fcbb413d730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:47:53.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.648+0000 7fcbb2ffd640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fcbb4134ad0 0x7fcbb4134eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.12:47526/0 (socket says 172.21.15.12:47526) 2024-09-11T01:47:53.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.648+0000 7fcbb2ffd640 1 -- 172.21.15.12:0/1581036705 learned_addr learned my addr 172.21.15.12:0/1581036705 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:47:53.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.650+0000 7fcbb2ffd640 1 -- 172.21.15.12:0/1581036705 >> v1:172.21.15.71:6789/0 conn(0x7fcbb4134ad0 legacy=0x7fcbb4134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:53.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.650+0000 7fcbb1ffb640 1 -- 172.21.15.12:0/1581036705 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3368900694 0 0) 0x7fcbb409f010 con 0x7fcbb4134ad0 2024-09-11T01:47:53.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.650+0000 7fcbb1ffb640 10 monclient(hunting): _negotiate 2024-09-11T01:47:53.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.650+0000 7fcbb1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:53.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.650+0000 7fcbb1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:53.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.650+0000 7fcbb1ffb640 10 monclient(hunting): my global_id is 134737 2024-09-11T01:47:53.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.650+0000 7fcbb1ffb640 1 -- 172.21.15.12:0/1581036705 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcba0003520 con 0x7fcbb4134ad0 2024-09-11T01:47:53.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 1 -- 172.21.15.12:0/1581036705 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1771376854 0 0) 0x7fcba0003520 con 0x7fcbb4134ad0 2024-09-11T01:47:53.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 1 -- 172.21.15.12:0/1581036705 >> v2:172.21.15.12:3300/0 conn(0x7fcbb41355c0 msgr2=0x7fcbb413d730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:47:53.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 1 --2- 172.21.15.12:0/1581036705 >> v2:172.21.15.12:3300/0 conn(0x7fcbb41355c0 0x7fcbb413d730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 1 -- 172.21.15.12:0/1581036705 >> v1:172.21.15.12:6789/0 conn(0x7fcbb40a2200 legacy=0x7fcbb413a000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:47:53.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 10 monclient: _finish_hunting 0 2024-09-11T01:47:53.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 1 monclient: found mon.noname-c-legacy 2024-09-11T01:47:53.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:53.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-09-11T01:47:53.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 1 -- 172.21.15.12:0/1581036705 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbb409f2f0 con 0x7fcbb4134ad0 2024-09-11T01:47:53.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 10 monclient: _finish_auth 0 2024-09-11T01:47:53.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:47:53.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:53.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 1 -- 172.21.15.12:0/1581036705 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7fcba4001d20 con 0x7fcbb4134ad0 2024-09-11T01:47:53.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:53.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-11T01:47:53.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 10 monclient: dump: 2024-09-11T01:47:53.660 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:53.660 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:53.660 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.660 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.660 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:53.660 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:53.660 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:53.660 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:53.661 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:53.661 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:53.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-09-11T01:47:53.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:53.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 1 -- 172.21.15.12:0/1581036705 >> v1:172.21.15.71:6789/0 conn(0x7fcbb4134ad0 legacy=0x7fcbb4134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:53.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:47:53.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 1 --2- 172.21.15.12:0/1581036705 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcba00039b0 0x7fcba0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 10 monclient(hunting): picked mon.c con 0x7fcba00039b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:47:53.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 1 --2- 172.21.15.12:0/1581036705 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcba0004800 0x7fcba0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 10 monclient(hunting): picked mon.b con 0x7fcba0004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:47:53.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 1 --2- 172.21.15.12:0/1581036705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcba0005410 0x7fcba000ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 10 monclient(hunting): picked mon.a con 0x7fcba0005410 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:47:53.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:53.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 1 -- 172.21.15.12:0/1581036705 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb409f5d0 con 0x7fcba0005410 2024-09-11T01:47:53.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:53.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 1 -- 172.21.15.12:0/1581036705 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fcba000c320 con 0x7fcba0004800 2024-09-11T01:47:53.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:53.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 1 -- 172.21.15.12:0/1581036705 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcba000c520 con 0x7fcba00039b0 2024-09-11T01:47:53.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:53.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:47:53.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb37fe640 1 -- 172.21.15.12:0/1581036705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcba0005410 msgr2=0x7fcba000ba40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:47:53.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb37fe640 1 --2- 172.21.15.12:0/1581036705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcba0005410 0x7fcba000ba40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:47:53.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb2ffd640 1 --2- 172.21.15.12:0/1581036705 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcba00039b0 0x7fcba0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:53.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb2ffd640 10 monclient(hunting): get_auth_request con 0x7fcba00039b0 auth_method 0 2024-09-11T01:47:53.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:53.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:53.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:47:53.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb27fc640 1 --2- 172.21.15.12:0/1581036705 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcba0004800 0x7fcba0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:53.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb27fc640 10 monclient(hunting): get_auth_request con 0x7fcba0004800 auth_method 0 2024-09-11T01:47:53.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:53.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb27fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:53.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb27fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:47:53.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:53.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:53.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:47:53.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:53.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:53.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb27fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:47:53.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb2ffd640 10 monclient(hunting): handle_auth_done global_id 134737 payload 293 2024-09-11T01:47:53.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb2ffd640 1 -- 172.21.15.12:0/1581036705 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcba0004800 msgr2=0x7fcba0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:47:53.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb2ffd640 1 --2- 172.21.15.12:0/1581036705 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcba0004800 0x7fcba0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb2ffd640 1 -- 172.21.15.12:0/1581036705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcba0005410 msgr2=0x7fcba000ba40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:47:53.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb2ffd640 1 --2- 172.21.15.12:0/1581036705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcba0005410 0x7fcba000ba40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb2ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:47:53.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb2ffd640 1 monclient: found mon.c 2024-09-11T01:47:53.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:53.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb2ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:47:53.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb37fe640 1 -- 172.21.15.12:0/1581036705 reap_dead start 2024-09-11T01:47:53.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb2ffd640 1 -- 172.21.15.12:0/1581036705 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcba0003520 con 0x7fcba00039b0 2024-09-11T01:47:53.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb2ffd640 10 monclient: _finish_auth 0 2024-09-11T01:47:53.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb2ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:47:53.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:53.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb95da640 5 monclient: authenticate success, global_id 134737 2024-09-11T01:47:53.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.651+0000 7fcbb95da640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:47:53.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.654+0000 7fcbb2ffd640 1 -- 172.21.15.12:0/1581036705 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcba00039b0 msgr2=0x7fcba0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:53.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.654+0000 7fcbb2ffd640 1 --2- 172.21.15.12:0/1581036705 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcba00039b0 0x7fcba0003dd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcba4001850 tx=0x7fcba0005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf4d86467821b54c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:53.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.654+0000 7fcbb1ffb640 1 -- 172.21.15.12:0/1581036705 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba400a690 con 0x7fcba00039b0 2024-09-11T01:47:53.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.654+0000 7fcbb1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:53.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.654+0000 7fcbb1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:47:53.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.654+0000 7fcbb1ffb640 10 monclient: dump: 2024-09-11T01:47:53.665 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:53.665 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:53.665 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.665 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.665 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:53.665 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:53.665 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:53.665 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:53.666 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:53.666 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:53.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.654+0000 7fcbb95da640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:47:53.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.654+0000 7fcbb1ffb640 1 -- 172.21.15.12:0/1581036705 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcba402f070 con 0x7fcba00039b0 2024-09-11T01:47:53.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.654+0000 7fcbb1ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:47:53.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.654+0000 7fcbb1ffb640 1 -- 172.21.15.12:0/1581036705 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba402bdc0 con 0x7fcba00039b0 2024-09-11T01:47:53.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.654+0000 7fcbb95da640 10 monclient: get_monmap_and_config success 2024-09-11T01:47:53.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.654+0000 7fcbb95da640 4 set_mon_vals no callback set 2024-09-11T01:47:53.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.654+0000 7fcbb1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:53.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.654+0000 7fcbb1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:47:53.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.654+0000 7fcbb1ffb640 10 monclient: dump: 2024-09-11T01:47:53.666 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:53.666 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:53.667 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.667 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.667 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:53.667 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:53.667 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:53.667 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:53.667 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:53.667 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:53.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.655+0000 7fcbb95da640 10 monclient: shutdown 2024-09-11T01:47:53.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.655+0000 7fcbb95da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:47:53.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.655+0000 7fcbb95da640 1 -- 172.21.15.12:0/1581036705 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcba00039b0 msgr2=0x7fcba0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:53.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.655+0000 7fcbb95da640 1 --2- 172.21.15.12:0/1581036705 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcba00039b0 0x7fcba0003dd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcba4001850 tx=0x7fcba0005830 comp rx=0 tx=0).stop 2024-09-11T01:47:53.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.655+0000 7fcbb95da640 1 -- 172.21.15.12:0/1581036705 shutdown_connections 2024-09-11T01:47:53.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.655+0000 7fcbb95da640 1 --2- 172.21.15.12:0/1581036705 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcba00039b0 0x7fcba0003dd0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.655+0000 7fcbb95da640 1 -- 172.21.15.12:0/1581036705 >> 172.21.15.12:0/1581036705 conn(0x7fcbb40a2f70 msgr2=0x7fcbb40a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:47:53.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.655+0000 7fcbb95da640 1 -- 172.21.15.12:0/1581036705 shutdown_connections 2024-09-11T01:47:53.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.655+0000 7fcbb95da640 1 -- 172.21.15.12:0/1581036705 wait complete. 2024-09-11T01:47:53.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.655+0000 7fcbb95da640 10 monclient: build_initial_monmap 2024-09-11T01:47:53.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.655+0000 7fcbb95da640 10 monclient: monmap: 2024-09-11T01:47:53.668 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:47:53.668 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:47:53.668 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:47:53.668 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:47:53.668 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:47:53.668 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:53.668 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:47:53.668 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:47:53.668 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:47:53.669 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:53.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.656+0000 7fcbb95da640 1 Processor -- start 2024-09-11T01:47:53.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.656+0000 7fcbb95da640 1 -- start start 2024-09-11T01:47:53.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.656+0000 7fcbb95da640 10 monclient: init 2024-09-11T01:47:53.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.656+0000 7fcbb95da640 10 monclient: _reopen_session rank -1 2024-09-11T01:47:53.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.656+0000 7fcbb95da640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:47:53.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.656+0000 7fcbb95da640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcba00039b0 0x7fcbb41cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.656+0000 7fcbb95da640 10 monclient(hunting): picked mon.noname-b con 0x7fcba00039b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:47:53.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.656+0000 7fcbb95da640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcbb4126d00 0x7fcbb41cd910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.656+0000 7fcbb95da640 10 monclient(hunting): picked mon.noname-c con 0x7fcbb4126d00 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:47:53.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.656+0000 7fcbb95da640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbb41246e0 0x7fcbb4124b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.656+0000 7fcbb95da640 10 monclient(hunting): picked mon.noname-a con 0x7fcbb41246e0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:47:53.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.656+0000 7fcbb95da640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:53.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.656+0000 7fcbb95da640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb40a0150 con 0x7fcbb41246e0 2024-09-11T01:47:53.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.656+0000 7fcbb95da640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:53.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.656+0000 7fcbb95da640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb409ffd0 con 0x7fcba00039b0 2024-09-11T01:47:53.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.656+0000 7fcbb95da640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:53.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.656+0000 7fcbb95da640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcbb4134eb0 con 0x7fcbb4126d00 2024-09-11T01:47:53.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.656+0000 7fcbb95da640 10 monclient(hunting): _renew_subs 2024-09-11T01:47:53.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.656+0000 7fcbb95da640 10 monclient(hunting): authenticate will time out at 1880.099950s 2024-09-11T01:47:53.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.656+0000 7fcbb37fe640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbb41246e0 msgr2=0x7fcbb4124b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:47:53.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.656+0000 7fcbb37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbb41246e0 0x7fcbb4124b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:47:53.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb27fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcbb4126d00 0x7fcbb41cd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:53.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb27fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcbb4126d00 0x7fcbb41cd910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55222/0 (socket says 172.21.15.12:55222) 2024-09-11T01:47:53.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb27fc640 1 -- 172.21.15.12:0/2278114036 learned_addr learned my addr 172.21.15.12:0/2278114036 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:47:53.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb27fc640 10 monclient(hunting): get_auth_request con 0x7fcbb4126d00 auth_method 0 2024-09-11T01:47:53.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:53.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb27fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:53.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:53.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:53.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:53.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:53.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb2ffd640 1 --2- 172.21.15.12:0/2278114036 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcba00039b0 0x7fcbb41cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:53.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb2ffd640 10 monclient(hunting): get_auth_request con 0x7fcba00039b0 auth_method 0 2024-09-11T01:47:53.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:53.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:53.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:53.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:53.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:53.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:53.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb27fc640 10 monclient(hunting): handle_auth_done global_id 184337 payload 824 2024-09-11T01:47:53.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb27fc640 1 -- 172.21.15.12:0/2278114036 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcba00039b0 msgr2=0x7fcbb41cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:53.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb27fc640 1 --2- 172.21.15.12:0/2278114036 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcba00039b0 0x7fcbb41cd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb27fc640 1 -- 172.21.15.12:0/2278114036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbb41246e0 msgr2=0x7fcbb4124b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:47:53.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb27fc640 1 --2- 172.21.15.12:0/2278114036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbb41246e0 0x7fcbb4124b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb27fc640 10 monclient: _finish_hunting 0 2024-09-11T01:47:53.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb27fc640 1 monclient: found mon.noname-c 2024-09-11T01:47:53.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:47:53.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:47:53.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.657+0000 7fcbb27fc640 1 -- 172.21.15.12:0/2278114036 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbb4128500 con 0x7fcbb4126d00 2024-09-11T01:47:53.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.659+0000 7fcbb27fc640 1 -- 172.21.15.12:0/2278114036 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcbb4126d00 msgr2=0x7fcbb41cd910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:53.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.659+0000 7fcbb27fc640 1 --2- 172.21.15.12:0/2278114036 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcbb4126d00 0x7fcbb41cd910 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c003fd0 tx=0x7fcb9c009a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:53.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.659+0000 7fcbab7fe640 1 -- 172.21.15.12:0/2278114036 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c016070 con 0x7fcbb4126d00 2024-09-11T01:47:53.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.659+0000 7fcbab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:53.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.659+0000 7fcbab7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:47:53.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.659+0000 7fcbab7fe640 10 monclient: dump: 2024-09-11T01:47:53.679 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:53.679 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:53.679 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.679 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.679 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:53.679 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:53.679 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:53.679 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:53.679 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:53.679 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:53.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.659+0000 7fcbab7fe640 10 monclient: _finish_auth 0 2024-09-11T01:47:53.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.659+0000 7fcbab7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:47:53.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.659+0000 7fcbab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:47:53.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.659+0000 7fcbab7fe640 1 -- 172.21.15.12:0/2278114036 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb9c012810 con 0x7fcbb4126d00 2024-09-11T01:47:53.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.659+0000 7fcbb95da640 5 monclient: authenticate success, global_id 184337 2024-09-11T01:47:53.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.659+0000 7fcbb95da640 10 monclient: _renew_subs 2024-09-11T01:47:53.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.659+0000 7fcbb95da640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:47:53.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.659+0000 7fcbb95da640 1 -- 172.21.15.12:0/2278114036 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbb41261a0 con 0x7fcbb4126d00 2024-09-11T01:47:53.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.659+0000 7fcbab7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:47:53.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.659+0000 7fcbab7fe640 1 -- 172.21.15.12:0/2278114036 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c012c00 con 0x7fcbb4126d00 2024-09-11T01:47:53.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.659+0000 7fcbab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:53.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.659+0000 7fcbab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:47:53.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.659+0000 7fcbab7fe640 10 monclient: dump: 2024-09-11T01:47:53.680 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:53.681 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:53.681 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.681 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.681 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:53.681 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:53.681 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:53.681 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:53.681 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:53.681 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:53.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.659+0000 7fcbb17fa640 4 set_mon_vals no callback set 2024-09-11T01:47:53.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.659+0000 7fcbb95da640 10 monclient: _renew_subs 2024-09-11T01:47:53.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.659+0000 7fcbb95da640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:47:53.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.659+0000 7fcbb95da640 1 -- 172.21.15.12:0/2278114036 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbb4128ec0 con 0x7fcbb4126d00 2024-09-11T01:47:53.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.660+0000 7fcbb95da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:53.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.660+0000 7fcbb95da640 1 --2- 172.21.15.12:0/2278114036 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbb41250b0 0x7fcbb4125490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.660+0000 7fcbb95da640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:53.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.660+0000 7fcbb95da640 1 -- 172.21.15.12:0/2278114036 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb4128180 con 0x7fcbb41250b0 2024-09-11T01:47:53.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.660+0000 7fcbab7fe640 1 -- 172.21.15.12:0/2278114036 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fcb9c023060 con 0x7fcbb4126d00 2024-09-11T01:47:53.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.661+0000 7fcbab7fe640 1 --2- 172.21.15.12:0/2278114036 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcb9403e1f0 0x7fcb940406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.661+0000 7fcbb2ffd640 1 --2- 172.21.15.12:0/2278114036 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbb41250b0 0x7fcbb4125490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:53.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.661+0000 7fcbab7fe640 1 -- 172.21.15.12:0/2278114036 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(620..620 src has 507..620) ==== 4847+0+0 (secure 0 0 0) 0x7fcb9c0526d0 con 0x7fcbb4126d00 2024-09-11T01:47:53.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.661+0000 7fcbb2ffd640 10 monclient: get_auth_request con 0x7fcbb41250b0 auth_method 0 2024-09-11T01:47:53.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.661+0000 7fcbb2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:53.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.661+0000 7fcbb2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:53.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.661+0000 7fcbb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:53.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.661+0000 7fcbb2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:53.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.661+0000 7fcbb2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:53.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.661+0000 7fcbb2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:53.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.661+0000 7fcbb37fe640 1 --2- 172.21.15.12:0/2278114036 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcb9403e1f0 0x7fcb940406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:53.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.661+0000 7fcbb37fe640 10 monclient: get_auth_request con 0x7fcb9403e1f0 auth_method 0 2024-09-11T01:47:53.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.661+0000 7fcbb2ffd640 10 monclient(hunting): handle_auth_done global_id 134743 payload 824 2024-09-11T01:47:53.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.661+0000 7fcbb2ffd640 1 -- 172.21.15.12:0/2278114036 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcbb4127d80 con 0x7fcbb41250b0 2024-09-11T01:47:53.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.663+0000 7fcbb2ffd640 1 -- 172.21.15.12:0/2278114036 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbb41250b0 msgr2=0x7fcbb4125490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:53.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.663+0000 7fcbb2ffd640 1 --2- 172.21.15.12:0/2278114036 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbb41250b0 0x7fcbb4125490 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fcba4033000 tx=0x7fcbb409ef00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:53.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.663+0000 7fcbb37fe640 1 --2- 172.21.15.12:0/2278114036 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcb9403e1f0 0x7fcb940406b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fcbb406cdd0 tx=0x7fcbac006bb0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:53.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.663+0000 7fcbab7fe640 1 -- 172.21.15.12:0/2278114036 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba400a690 con 0x7fcbb41250b0 2024-09-11T01:47:53.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.663+0000 7fcbab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:53.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.663+0000 7fcbab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:47:53.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.663+0000 7fcbab7fe640 10 monclient: dump: 2024-09-11T01:47:53.683 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:53.684 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:53.684 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.684 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.684 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:53.684 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:53.684 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:53.684 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:53.684 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:53.684 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:53.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.664+0000 7fcbab7fe640 1 -- 172.21.15.12:0/2278114036 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcba402f050 con 0x7fcbb41250b0 2024-09-11T01:47:53.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.664+0000 7fcbab7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:47:53.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.664+0000 7fcbab7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:47:53.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.664+0000 7fcbab7fe640 1 -- 172.21.15.12:0/2278114036 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbb41250b0 msgr2=0x7fcbb4125490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:53.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.664+0000 7fcbab7fe640 1 --2- 172.21.15.12:0/2278114036 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbb41250b0 0x7fcbb4125490 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fcba4033000 tx=0x7fcbb409ef00 comp rx=0 tx=0).stop 2024-09-11T01:47:53.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.664+0000 7fcbab7fe640 1 -- 172.21.15.12:0/2278114036 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbb41250b0 msgr2=0x7fcbb4125490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:47:53.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.664+0000 7fcbab7fe640 1 --2- 172.21.15.12:0/2278114036 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbb41250b0 0x7fcbb4125490 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.670+0000 7fcbb95da640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:47:53.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.670+0000 7fcbb95da640 1 --2- 172.21.15.12:0/2278114036 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbb411c8e0 0x7fcbb411ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:47:53.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.670+0000 7fcbb95da640 10 monclient(hunting): start opening mon connection 2024-09-11T01:47:53.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.670+0000 7fcbb95da640 1 -- 172.21.15.12:0/2278114036 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb406d480 con 0x7fcbb411c8e0 2024-09-11T01:47:53.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.671+0000 7fcbb2ffd640 1 --2- 172.21.15.12:0/2278114036 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbb411c8e0 0x7fcbb411ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:47:53.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.671+0000 7fcbb2ffd640 10 monclient: get_auth_request con 0x7fcbb411c8e0 auth_method 0 2024-09-11T01:47:53.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.671+0000 7fcbb2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:47:53.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.671+0000 7fcbb2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:47:53.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.671+0000 7fcbb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:47:53.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.671+0000 7fcbb2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:47:53.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.671+0000 7fcbb2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:47:53.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.671+0000 7fcbb2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:47:53.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.671+0000 7fcbb2ffd640 10 monclient(hunting): handle_auth_done global_id 134746 payload 824 2024-09-11T01:47:53.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.671+0000 7fcbb2ffd640 1 -- 172.21.15.12:0/2278114036 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcbb41259d0 con 0x7fcbb411c8e0 2024-09-11T01:47:53.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.673+0000 7fcbb2ffd640 1 -- 172.21.15.12:0/2278114036 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbb411c8e0 msgr2=0x7fcbb411ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:47:53.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.673+0000 7fcbb2ffd640 1 --2- 172.21.15.12:0/2278114036 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbb411c8e0 0x7fcbb411ccf0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fcba40017a0 tx=0x7fcba400a290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:47:53.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.674+0000 7fcbab7fe640 1 -- 172.21.15.12:0/2278114036 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba4034e00 con 0x7fcbb411c8e0 2024-09-11T01:47:53.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.674+0000 7fcbab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:47:53.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.674+0000 7fcbab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:47:53.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.674+0000 7fcbab7fe640 10 monclient: dump: 2024-09-11T01:47:53.687 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:47:53.687 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:47:53.687 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.687 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:47:53.687 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:47:53.687 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:47:53.687 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:47:53.688 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:47:53.688 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:47:53.688 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:47:53.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.674+0000 7fcbab7fe640 1 -- 172.21.15.12:0/2278114036 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4867 (secure 0 0 0) 0x7fcba4034e00 con 0x7fcbb411c8e0 2024-09-11T01:47:53.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.674+0000 7fcbab7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:47:53.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.674+0000 7fcbab7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:47:53.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.674+0000 7fcbab7fe640 1 -- 172.21.15.12:0/2278114036 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbb411c8e0 msgr2=0x7fcbb411ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:53.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.674+0000 7fcbab7fe640 1 --2- 172.21.15.12:0/2278114036 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbb411c8e0 0x7fcbb411ccf0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fcba40017a0 tx=0x7fcba400a290 comp rx=0 tx=0).stop 2024-09-11T01:47:53.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.674+0000 7fcbab7fe640 1 -- 172.21.15.12:0/2278114036 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbb411c8e0 msgr2=0x7fcbb411ccf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:47:53.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.674+0000 7fcbab7fe640 1 --2- 172.21.15.12:0/2278114036 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbb411c8e0 0x7fcbb411ccf0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.689 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:47:53.689 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:47:53.689 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:47:53.689 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:47:53.689 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 174, 2024-09-11T01:47:53.689 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 406637, 2024-09-11T01:47:53.689 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:47:53.689 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:47:53.689 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:47:53.689 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:53.689 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:47:53.690 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:47:53.690 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:47:53.690 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:47:53.690 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:47:53.690 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:47:53.690 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:47:53.690 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:47:53.690 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:47:53.690 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:47:53.690 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:47:53.690 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:47:53.690 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:47:53.690 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:47:53.691 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:47:53.691 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:53.691 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:47:53.691 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:47:53.691 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:47:53.691 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:47:53.691 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:47:53.691 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:47:53.691 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:47:53.691 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:47:53.691 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:47:53.691 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:47:53.691 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:47:53.691 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:47:53.692 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:47:53.692 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:53.692 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:53.692 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:47:53.692 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:47:53.692 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:47:53.692 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:47:53.692 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:47:53.692 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:47:53.692 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:47:53.692 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:47:53.692 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:47:53.692 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:47:53.693 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:47:53.693 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:47:53.693 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:47:53.693 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:47:53.693 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:47:53.693 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:47:53.693 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:47:53.693 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:47:53.693 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:47:53.693 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:47:53.693 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:47:53.693 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:47:53.693 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:47:53.693 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:47:53.694 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:47:53.694 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:47:53.694 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:47:53.694 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:47:53.694 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:53.694 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:47:53.694 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:53.694 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:47:53.694 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:53.694 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:47:53.694 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:47:53.694 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:47:53.694 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:47:53.694 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:53.695 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:47:53.695 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:47:53.695 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:53.695 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:53.695 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:53.695 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:47:53.695 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:47:53.695 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:53.695 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:53.695 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:53.695 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:53.695 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:47:53.695 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:47:53.696 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:47:53.696 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:47:53.696 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:47:53.696 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:53.696 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:53.696 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:47:53.696 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:47:53.696 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:47:53.696 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:47:53.696 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:53.696 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:47:53.696 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:47:53.696 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:53.697 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:53.697 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:53.697 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:47:53.697 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:47:53.697 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:53.697 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:53.697 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:53.697 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:53.697 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:47:53.697 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:47:53.697 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:47:53.697 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:47:53.697 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:47:53.697 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:53.698 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:53.698 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:47:53.698 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:47:53.698 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:47:53.698 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:47:53.698 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:53.698 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:47:53.698 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:47:53.698 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:53.698 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:53.698 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:53.698 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:47:53.698 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:47:53.698 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:47:53.699 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:53.699 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:53.699 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:53.699 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:47:53.699 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:47:53.699 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:47:53.699 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:47:53.699 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:47:53.699 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:53.699 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:53.699 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:53.699 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:47:53.699 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:47:53.699 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:53.700 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:47:53.700 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:47:53.700 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:47:53.700 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:53.700 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:53.700 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:47:53.700 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:53.700 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:47:53.700 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:47:53.700 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:47:53.700 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:53.700 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:53.700 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:47:53.701 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:53.701 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:47:53.701 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:47:53.701 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:47:53.701 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:53.701 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:47:53.701 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:47:53.701 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:47:53.701 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:47:53.701 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:47:53.701 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:47:53.701 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:47:53.701 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:47:53.701 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:47:53.702 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:47:53.702 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:47:53.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.674+0000 7fcbb95da640 1 -- 172.21.15.12:0/2278114036 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcb9403e1f0 msgr2=0x7fcb940406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:53.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.674+0000 7fcbb95da640 1 --2- 172.21.15.12:0/2278114036 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcb9403e1f0 0x7fcb940406b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fcbb406cdd0 tx=0x7fcbac006bb0 comp rx=0 tx=0).stop 2024-09-11T01:47:53.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.674+0000 7fcbb95da640 10 monclient: shutdown 2024-09-11T01:47:53.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.674+0000 7fcbb95da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:47:53.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.674+0000 7fcbb95da640 1 -- 172.21.15.12:0/2278114036 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcbb4126d00 msgr2=0x7fcbb41cd910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:47:53.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.674+0000 7fcbb95da640 1 --2- 172.21.15.12:0/2278114036 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcbb4126d00 0x7fcbb41cd910 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c003fd0 tx=0x7fcb9c009a70 comp rx=0 tx=0).stop 2024-09-11T01:47:53.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.674+0000 7fcbb37fe640 1 -- 172.21.15.12:0/2278114036 reap_dead start 2024-09-11T01:47:53.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.674+0000 7fcbb37fe640 1 -- 172.21.15.12:0/2278114036 reap_dead start 2024-09-11T01:47:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.674+0000 7fcbb95da640 1 -- 172.21.15.12:0/2278114036 shutdown_connections 2024-09-11T01:47:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.674+0000 7fcbb95da640 1 -- 172.21.15.12:0/2278114036 >> 172.21.15.12:0/2278114036 conn(0x7fcbb40a2f70 msgr2=0x7fcbb40a4c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:47:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.674+0000 7fcbb95da640 1 -- 172.21.15.12:0/2278114036 shutdown_connections 2024-09-11T01:47:53.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:47:53.674+0000 7fcbb95da640 1 -- 172.21.15.12:0/2278114036 wait complete. 2024-09-11T01:47:53.704 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:47:53.704 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:47:54.339 INFO:tasks.rados.rados.0.smithi071.stdout:2765: done (5 left) 2024-09-11T01:47:54.339 INFO:tasks.rados.rados.0.smithi071.stdout:2766: copy_from oid 48 from oid 27 current snap is 309 2024-09-11T01:47:54.339 INFO:tasks.rados.rados.0.smithi071.stdout:2761: finishing rollback tid 0 to smithi07134777-5 2024-09-11T01:47:54.339 INFO:tasks.rados.rados.0.smithi071.stdout:2760: expect (ObjNum 689 snap 204 seq_num 689) 2024-09-11T01:47:54.398 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1326006 2024-09-11T01:47:54.431 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:47:54.483 INFO:tasks.rados.rados.0.smithi071.stdout:2762: expect (ObjNum 981 snap 308 seq_num 981) 2024-09-11T01:47:54.500 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1951687 2024-09-11T01:47:54.602 INFO:tasks.rados.rados.0.smithi071.stdout:2761: finishing rollback tid 1 to smithi07134777-5 2024-09-11T01:47:54.602 INFO:tasks.rados.rados.0.smithi071.stdout:2761: finishing rollback tid 2 to smithi07134777-5 2024-09-11T01:47:54.602 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 1173 (ObjNum 886 snap 272 seq_num 886) dirty exists 2024-09-11T01:47:54.602 INFO:tasks.rados.rados.0.smithi071.stdout:2759: expect (ObjNum 932 snap 293 seq_num 932) 2024-09-11T01:47:54.770 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3233828 2024-09-11T01:47:54.810 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2304000 2024-09-11T01:47:54.862 INFO:tasks.rados.rados.0.smithi071.stdout:2764: expect (ObjNum 979 snap 306 seq_num 979) 2024-09-11T01:47:54.862 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:47:54.896 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 325023 2024-09-11T01:47:54.958 INFO:tasks.rados.rados.0.smithi071.stdout:2766: finishing copy_from racing read to smithi07134777-48 2024-09-11T01:47:54.958 INFO:tasks.rados.rados.0.smithi071.stdout:2766: finishing copy_from to smithi07134777-48 2024-09-11T01:47:54.958 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 854 (ObjNum 875 snap 267 seq_num 875) dirty exists 2024-09-11T01:47:54.958 INFO:tasks.rados.rados.0.smithi071.stdout:2759: done (5 left) 2024-09-11T01:47:54.958 INFO:tasks.rados.rados.0.smithi071.stdout:2760: done (4 left) 2024-09-11T01:47:54.959 INFO:tasks.rados.rados.0.smithi071.stdout:2761: done (3 left) 2024-09-11T01:47:54.959 INFO:tasks.rados.rados.0.smithi071.stdout:2762: done (2 left) 2024-09-11T01:47:54.959 INFO:tasks.rados.rados.0.smithi071.stdout:2764: done (1 left) 2024-09-11T01:47:54.959 INFO:tasks.rados.rados.0.smithi071.stdout:2766: done (0 left) 2024-09-11T01:47:54.959 INFO:tasks.rados.rados.0.smithi071.stdout:2767: read oid 14 snap -1 2024-09-11T01:47:54.959 INFO:tasks.rados.rados.0.smithi071.stdout:2767: expect (ObjNum 942 snap 296 seq_num 942) 2024-09-11T01:47:54.959 INFO:tasks.rados.rados.0.smithi071.stdout:2768: snap_create 2024-09-11T01:47:54.966 INFO:tasks.rados.rados.0.smithi071.stdout:2767: expect (ObjNum 942 snap 296 seq_num 942) 2024-09-11T01:47:54.976 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1439758 2024-09-11T01:47:55.033 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:47:55.342 INFO:tasks.rados.rados.0.smithi071.stdout:2767: done (1 left) 2024-09-11T01:47:55.342 INFO:tasks.rados.rados.0.smithi071.stdout:2768: done (0 left) 2024-09-11T01:47:55.342 INFO:tasks.rados.rados.0.smithi071.stdout:2769: snap_create 2024-09-11T01:47:56.345 INFO:tasks.rados.rados.0.smithi071.stdout:2769: done (0 left) 2024-09-11T01:47:56.345 INFO:tasks.rados.rados.0.smithi071.stdout:2770: snap_create 2024-09-11T01:47:57.346 INFO:tasks.rados.rados.0.smithi071.stdout:2770: done (0 left) 2024-09-11T01:47:57.347 INFO:tasks.rados.rados.0.smithi071.stdout:2771: snap_create 2024-09-11T01:47:58.086 INFO:tasks.rados.rados.0.smithi071.stdout:2771: done (0 left) 2024-09-11T01:47:58.086 INFO:tasks.rados.rados.0.smithi071.stdout:2772: copy_from oid 3 from oid 1 current snap is 313 2024-09-11T01:47:58.087 INFO:tasks.rados.rados.0.smithi071.stdout:2773: delete oid 6 current snap is 313 2024-09-11T01:47:58.096 INFO:tasks.rados.rados.0.smithi071.stdout:2773: done (1 left) 2024-09-11T01:47:58.097 INFO:tasks.rados.rados.0.smithi071.stdout:2774: snap_create 2024-09-11T01:47:58.103 INFO:tasks.rados.rados.0.smithi071.stdout:2772: finishing copy_from racing read to smithi07134777-3 2024-09-11T01:47:58.112 INFO:tasks.rados.rados.0.smithi071.stdout:2772: finishing copy_from to smithi07134777-3 2024-09-11T01:47:58.112 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 1757 (ObjNum 688 snap 204 seq_num 688) dirty exists 2024-09-11T01:47:59.091 INFO:tasks.rados.rados.0.smithi071.stdout:2772: done (1 left) 2024-09-11T01:47:59.091 INFO:tasks.rados.rados.0.smithi071.stdout:2774: done (0 left) 2024-09-11T01:47:59.091 INFO:tasks.rados.rados.0.smithi071.stdout:2775: read oid 6 snap -1 2024-09-11T01:47:59.092 INFO:tasks.rados.rados.0.smithi071.stdout:2775: expect deleted 2024-09-11T01:47:59.092 INFO:tasks.rados.rados.0.smithi071.stdout:2776: rollback oid 15 current snap is 314 2024-09-11T01:47:59.092 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 15 to 306 2024-09-11T01:47:59.092 INFO:tasks.rados.rados.0.smithi071.stdout:2777: write oid 42 current snap is 314 2024-09-11T01:47:59.092 INFO:tasks.rados.rados.0.smithi071.stdout:2777: seq_num 983 ranges {521281=579304,1656270=774664} 2024-09-11T01:47:59.097 INFO:tasks.rados.rados.0.smithi071.stdout:2777: writing smithi07134777-42 from 521281 to 1100585 tid 1 2024-09-11T01:47:59.105 INFO:tasks.rados.rados.0.smithi071.stdout:2777: writing smithi07134777-42 from 1656270 to 2430934 tid 2 2024-09-11T01:47:59.105 INFO:tasks.rados.rados.0.smithi071.stdout:2778: write oid 7 current snap is 314 2024-09-11T01:47:59.105 INFO:tasks.rados.rados.0.smithi071.stdout:2778: seq_num 984 ranges {538088=654553,1714752=634675} 2024-09-11T01:47:59.111 INFO:tasks.rados.rados.0.smithi071.stdout:2778: writing smithi07134777-7 from 538088 to 1192641 tid 1 2024-09-11T01:47:59.119 INFO:tasks.rados.rados.0.smithi071.stdout:2778: writing smithi07134777-7 from 1714752 to 2349427 tid 2 2024-09-11T01:47:59.119 INFO:tasks.rados.rados.0.smithi071.stdout:2779: snap_create 2024-09-11T01:47:59.119 INFO:tasks.rados.rados.0.smithi071.stdout:2776: finishing rollback tid 0 to smithi07134777-15 2024-09-11T01:47:59.119 INFO:tasks.rados.rados.0.smithi071.stdout:2776: finishing rollback tid 1 to smithi07134777-15 2024-09-11T01:47:59.119 INFO:tasks.rados.rados.0.smithi071.stdout:2776: finishing rollback tid 2 to smithi07134777-15 2024-09-11T01:47:59.119 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 1913 (ObjNum 975 snap 305 seq_num 975) dirty exists 2024-09-11T01:47:59.119 INFO:tasks.rados.rados.0.smithi071.stdout:2777: finishing write tid 1 to smithi07134777-42 2024-09-11T01:47:59.119 INFO:tasks.rados.rados.0.smithi071.stdout:2777: finishing write tid 2 to smithi07134777-42 2024-09-11T01:47:59.119 INFO:tasks.rados.rados.0.smithi071.stdout:2777: finishing write tid 3 to smithi07134777-42 2024-09-11T01:47:59.119 INFO:tasks.rados.rados.0.smithi071.stdout:2777: finishing write tid 4 to smithi07134777-42 2024-09-11T01:47:59.120 INFO:tasks.rados.rados.0.smithi071.stdout:2777: oid 42 updating version 0 to 869 2024-09-11T01:47:59.120 INFO:tasks.rados.rados.0.smithi071.stdout:2777: oid 42 version 869 is already newer than 868 2024-09-11T01:47:59.120 INFO:tasks.rados.rados.0.smithi071.stdout:2777: oid 42 updating version 869 to 870 2024-09-11T01:47:59.120 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 870 (ObjNum 983 snap 314 seq_num 983) dirty exists 2024-09-11T01:47:59.120 INFO:tasks.rados.rados.0.smithi071.stdout:2777: left oid 42 (ObjNum 983 snap 314 seq_num 983) 2024-09-11T01:47:59.123 INFO:tasks.rados.rados.0.smithi071.stdout:2778: finishing write tid 1 to smithi07134777-7 2024-09-11T01:47:59.123 INFO:tasks.rados.rados.0.smithi071.stdout:2778: finishing write tid 2 to smithi07134777-7 2024-09-11T01:47:59.124 INFO:tasks.rados.rados.0.smithi071.stdout:2778: finishing write tid 3 to smithi07134777-7 2024-09-11T01:47:59.125 INFO:tasks.rados.rados.0.smithi071.stdout:2778: finishing write tid 4 to smithi07134777-7 2024-09-11T01:47:59.125 INFO:tasks.rados.rados.0.smithi071.stdout:2778: oid 7 updating version 0 to 1916 2024-09-11T01:47:59.125 INFO:tasks.rados.rados.0.smithi071.stdout:2778: oid 7 version 1916 is already newer than 1915 2024-09-11T01:47:59.125 INFO:tasks.rados.rados.0.smithi071.stdout:2778: oid 7 updating version 1916 to 1917 2024-09-11T01:47:59.125 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 1917 (ObjNum 984 snap 314 seq_num 984) dirty exists 2024-09-11T01:47:59.125 INFO:tasks.rados.rados.0.smithi071.stdout:2778: left oid 7 (ObjNum 984 snap 314 seq_num 984) 2024-09-11T01:48:00.092 INFO:tasks.rados.rados.0.smithi071.stdout:2775: done (4 left) 2024-09-11T01:48:00.093 INFO:tasks.rados.rados.0.smithi071.stdout:2776: done (3 left) 2024-09-11T01:48:00.093 INFO:tasks.rados.rados.0.smithi071.stdout:2777: done (2 left) 2024-09-11T01:48:00.093 INFO:tasks.rados.rados.0.smithi071.stdout:2778: done (1 left) 2024-09-11T01:48:00.093 INFO:tasks.rados.rados.0.smithi071.stdout:2779: done (0 left) 2024-09-11T01:48:00.093 INFO:tasks.rados.rados.0.smithi071.stdout:2780: rollback oid 44 current snap is 315 2024-09-11T01:48:00.093 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 44 to 283 2024-09-11T01:48:00.093 INFO:tasks.rados.rados.0.smithi071.stdout:2781: copy_from oid 6 from oid 10 current snap is 315 2024-09-11T01:48:00.093 INFO:tasks.rados.rados.0.smithi071.stdout:2782: snap_create 2024-09-11T01:48:00.100 INFO:tasks.rados.rados.0.smithi071.stdout:2780: finishing rollback tid 1 to smithi07134777-44 2024-09-11T01:48:00.100 INFO:tasks.rados.rados.0.smithi071.stdout:2780: finishing rollback tid 2 to smithi07134777-44 2024-09-11T01:48:00.100 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 1045 (ObjNum 779 snap 236 seq_num 779) dirty exists 2024-09-11T01:48:00.110 INFO:tasks.rados.rados.0.smithi071.stdout:2781: finishing copy_from racing read to smithi07134777-6 2024-09-11T01:48:00.111 INFO:tasks.rados.rados.0.smithi071.stdout:2781: finishing copy_from to smithi07134777-6 2024-09-11T01:48:00.111 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 475 (ObjNum 601 snap 172 seq_num 601) dirty exists 2024-09-11T01:48:01.097 INFO:tasks.rados.rados.0.smithi071.stdout:2780: done (2 left) 2024-09-11T01:48:01.097 INFO:tasks.rados.rados.0.smithi071.stdout:2781: done (1 left) 2024-09-11T01:48:01.097 INFO:tasks.rados.rados.0.smithi071.stdout:2782: done (0 left) 2024-09-11T01:48:01.098 INFO:tasks.rados.rados.0.smithi071.stdout:2783: read oid 21 snap -1 2024-09-11T01:48:01.098 INFO:tasks.rados.rados.0.smithi071.stdout:2783: expect (ObjNum 938 snap 294 seq_num 938) 2024-09-11T01:48:01.098 INFO:tasks.rados.rados.0.smithi071.stdout:2784: read oid 43 snap -1 2024-09-11T01:48:01.098 INFO:tasks.rados.rados.0.smithi071.stdout:2784: expect deleted 2024-09-11T01:48:01.098 INFO:tasks.rados.rados.0.smithi071.stdout:2785: snap_remove snap 267 2024-09-11T01:48:02.096 INFO:tasks.rados.rados.0.smithi071.stdout:2785: done (2 left) 2024-09-11T01:48:02.096 INFO:tasks.rados.rados.0.smithi071.stdout:2786: delete oid 50 current snap is 316 2024-09-11T01:48:02.096 INFO:tasks.rados.rados.0.smithi071.stdout:2783: expect (ObjNum 938 snap 294 seq_num 938) 2024-09-11T01:48:02.183 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:48:02.224 INFO:tasks.rados.rados.0.smithi071.stdout:2783: done (2 left) 2024-09-11T01:48:02.224 INFO:tasks.rados.rados.0.smithi071.stdout:2784: done (1 left) 2024-09-11T01:48:02.224 INFO:tasks.rados.rados.0.smithi071.stdout:2786: done (0 left) 2024-09-11T01:48:02.224 INFO:tasks.rados.rados.0.smithi071.stdout:2787: copy_from oid 29 from oid 36 current snap is 316 2024-09-11T01:48:02.224 INFO:tasks.rados.rados.0.smithi071.stdout:2788: rollback oid 11 current snap is 316 2024-09-11T01:48:02.224 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 11 to 271 2024-09-11T01:48:02.225 INFO:tasks.rados.rados.0.smithi071.stdout:2789: snap_create 2024-09-11T01:48:02.227 INFO:tasks.rados.rados.0.smithi071.stdout:2788: finishing rollback tid 1 to smithi07134777-11 2024-09-11T01:48:02.227 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 1290 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-11T01:48:02.236 INFO:tasks.rados.rados.0.smithi071.stdout:2787: finishing copy_from racing read to smithi07134777-29 2024-09-11T01:48:02.242 INFO:tasks.rados.rados.0.smithi071.stdout:2787: finishing copy_from to smithi07134777-29 2024-09-11T01:48:02.242 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 1066 (ObjNum 829 snap 252 seq_num 829) dirty exists 2024-09-11T01:48:03.090 INFO:tasks.rados.rados.0.smithi071.stdout:2787: done (2 left) 2024-09-11T01:48:03.090 INFO:tasks.rados.rados.0.smithi071.stdout:2788: done (1 left) 2024-09-11T01:48:03.090 INFO:tasks.rados.rados.0.smithi071.stdout:2789: done (0 left) 2024-09-11T01:48:03.090 INFO:tasks.rados.rados.0.smithi071.stdout:2790: write (excl) oid 29 current snap is 317 2024-09-11T01:48:03.090 INFO:tasks.rados.rados.0.smithi071.stdout:2790: seq_num 986 ranges {571702=405051,1431716=638344,2560365=11337} 2024-09-11T01:48:03.095 INFO:tasks.rados.rados.0.smithi071.stdout:2790: writing smithi07134777-29 from 571702 to 976753 tid 1 2024-09-11T01:48:03.101 INFO:tasks.rados.rados.0.smithi071.stdout:2790: writing smithi07134777-29 from 1431716 to 2070060 tid 2 2024-09-11T01:48:03.104 INFO:tasks.rados.rados.0.smithi071.stdout:2790: writing smithi07134777-29 from 2560365 to 2571702 tid 3 2024-09-11T01:48:03.105 INFO:tasks.rados.rados.0.smithi071.stdout:2791: write (excl) oid 11 current snap is 317 2024-09-11T01:48:03.105 INFO:tasks.rados.rados.0.smithi071.stdout:2790: finishing write tid 1 to smithi07134777-29 2024-09-11T01:48:03.105 INFO:tasks.rados.rados.0.smithi071.stdout:2791: seq_num 987 ranges {588509=480300,1773845=782002} 2024-09-11T01:48:03.110 INFO:tasks.rados.rados.0.smithi071.stdout:2791: writing smithi07134777-11 from 588509 to 1068809 tid 1 2024-09-11T01:48:03.119 INFO:tasks.rados.rados.0.smithi071.stdout:2791: writing smithi07134777-11 from 1773845 to 2555847 tid 2 2024-09-11T01:48:03.119 INFO:tasks.rados.rados.0.smithi071.stdout:2792: read oid 45 snap 251 2024-09-11T01:48:03.119 INFO:tasks.rados.rados.0.smithi071.stdout:2792: expect (ObjNum 793 snap 243 seq_num 793) 2024-09-11T01:48:03.120 INFO:tasks.rados.rados.0.smithi071.stdout:2793: read oid 14 snap -1 2024-09-11T01:48:03.120 INFO:tasks.rados.rados.0.smithi071.stdout:2793: expect (ObjNum 942 snap 296 seq_num 942) 2024-09-11T01:48:03.120 INFO:tasks.rados.rados.0.smithi071.stdout:2794: write (excl) oid 31 current snap is 317 2024-09-11T01:48:03.120 INFO:tasks.rados.rados.0.smithi071.stdout:2790: finishing write tid 2 to smithi07134777-29 2024-09-11T01:48:03.120 INFO:tasks.rados.rados.0.smithi071.stdout:2794: seq_num 988 ranges {605316=555549,1832327=642013} 2024-09-11T01:48:03.125 INFO:tasks.rados.rados.0.smithi071.stdout:2794: writing smithi07134777-31 from 605316 to 1160865 tid 1 2024-09-11T01:48:03.134 INFO:tasks.rados.rados.0.smithi071.stdout:2794: writing smithi07134777-31 from 1832327 to 2474340 tid 2 2024-09-11T01:48:03.134 INFO:tasks.rados.rados.0.smithi071.stdout:2795: rollback oid 20 current snap is 317 2024-09-11T01:48:03.134 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 20 to 311 2024-09-11T01:48:03.134 INFO:tasks.rados.rados.0.smithi071.stdout:2790: finishing write tid 3 to smithi07134777-29 2024-09-11T01:48:03.134 INFO:tasks.rados.rados.0.smithi071.stdout:2790: finishing write tid 4 to smithi07134777-29 2024-09-11T01:48:03.134 INFO:tasks.rados.rados.0.smithi071.stdout:2790: finishing write tid 5 to smithi07134777-29 2024-09-11T01:48:03.134 INFO:tasks.rados.rados.0.smithi071.stdout:2790: oid 29 updating version 0 to 1932 2024-09-11T01:48:03.134 INFO:tasks.rados.rados.0.smithi071.stdout:2790: oid 29 version 1932 is already newer than 1931 2024-09-11T01:48:03.134 INFO:tasks.rados.rados.0.smithi071.stdout:2790: oid 29 updating version 1932 to 1933 2024-09-11T01:48:03.134 INFO:tasks.rados.rados.0.smithi071.stdout:2790: oid 29 updating version 1933 to 1934 2024-09-11T01:48:03.134 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 1934 (ObjNum 986 snap 317 seq_num 986) dirty exists 2024-09-11T01:48:03.134 INFO:tasks.rados.rados.0.smithi071.stdout:2790: left oid 29 (ObjNum 986 snap 317 seq_num 986) 2024-09-11T01:48:03.135 INFO:tasks.rados.rados.0.smithi071.stdout:2791: finishing write tid 1 to smithi07134777-11 2024-09-11T01:48:03.135 INFO:tasks.rados.rados.0.smithi071.stdout:2793: expect (ObjNum 942 snap 296 seq_num 942) 2024-09-11T01:48:03.135 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:48:03.239 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2982726 2024-09-11T01:48:03.270 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:48:03.344 INFO:tasks.rados.rados.0.smithi071.stdout:2791: finishing write tid 2 to smithi07134777-11 2024-09-11T01:48:03.345 INFO:tasks.rados.rados.0.smithi071.stdout:2791: finishing write tid 3 to smithi07134777-11 2024-09-11T01:48:03.345 INFO:tasks.rados.rados.0.smithi071.stdout:2791: finishing write tid 4 to smithi07134777-11 2024-09-11T01:48:03.345 INFO:tasks.rados.rados.0.smithi071.stdout:2791: oid 11 updating version 0 to 1292 2024-09-11T01:48:03.345 INFO:tasks.rados.rados.0.smithi071.stdout:2791: oid 11 updating version 1292 to 1293 2024-09-11T01:48:03.345 INFO:tasks.rados.rados.0.smithi071.stdout:2791: oid 11 version 1293 is already newer than 1291 2024-09-11T01:48:03.345 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 1293 (ObjNum 987 snap 317 seq_num 987) dirty exists 2024-09-11T01:48:03.345 INFO:tasks.rados.rados.0.smithi071.stdout:2791: left oid 11 (ObjNum 987 snap 317 seq_num 987) 2024-09-11T01:48:03.345 INFO:tasks.rados.rados.0.smithi071.stdout:2792: expect (ObjNum 793 snap 243 seq_num 793) 2024-09-11T01:48:03.401 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 66581 2024-09-11T01:48:03.515 INFO:tasks.rados.rados.0.smithi071.stdout:2794: finishing write tid 1 to smithi07134777-31 2024-09-11T01:48:03.515 INFO:tasks.rados.rados.0.smithi071.stdout:2794: finishing write tid 2 to smithi07134777-31 2024-09-11T01:48:03.515 INFO:tasks.rados.rados.0.smithi071.stdout:2795: finishing rollback tid 0 to smithi07134777-20 2024-09-11T01:48:03.515 INFO:tasks.rados.rados.0.smithi071.stdout:2794: finishing write tid 3 to smithi07134777-31 2024-09-11T01:48:03.515 INFO:tasks.rados.rados.0.smithi071.stdout:2795: finishing rollback tid 1 to smithi07134777-20 2024-09-11T01:48:03.515 INFO:tasks.rados.rados.0.smithi071.stdout:2795: finishing rollback tid 2 to smithi07134777-20 2024-09-11T01:48:03.515 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 1048 (ObjNum 958 snap 304 seq_num 958) dirty exists 2024-09-11T01:48:03.515 INFO:tasks.rados.rados.0.smithi071.stdout:2790: done (5 left) 2024-09-11T01:48:03.515 INFO:tasks.rados.rados.0.smithi071.stdout:2791: done (4 left) 2024-09-11T01:48:03.515 INFO:tasks.rados.rados.0.smithi071.stdout:2792: done (3 left) 2024-09-11T01:48:03.515 INFO:tasks.rados.rados.0.smithi071.stdout:2793: done (2 left) 2024-09-11T01:48:03.515 INFO:tasks.rados.rados.0.smithi071.stdout:2795: done (1 left) 2024-09-11T01:48:03.516 INFO:tasks.rados.rados.0.smithi071.stdout:2796: read oid 44 snap -1 2024-09-11T01:48:03.516 INFO:tasks.rados.rados.0.smithi071.stdout:2796: expect (ObjNum 779 snap 236 seq_num 779) 2024-09-11T01:48:03.516 INFO:tasks.rados.rados.0.smithi071.stdout:2797: rollback oid 16 current snap is 317 2024-09-11T01:48:03.516 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 16 to 250 2024-09-11T01:48:03.516 INFO:tasks.rados.rados.0.smithi071.stdout:2794: finishing write tid 4 to smithi07134777-31 2024-09-11T01:48:03.516 INFO:tasks.rados.rados.0.smithi071.stdout:2794: oid 31 updating version 0 to 1760 2024-09-11T01:48:03.516 INFO:tasks.rados.rados.0.smithi071.stdout:2794: oid 31 version 1760 is already newer than 1759 2024-09-11T01:48:03.516 INFO:tasks.rados.rados.0.smithi071.stdout:2794: oid 31 version 1760 is already newer than 1758 2024-09-11T01:48:03.516 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 1760 (ObjNum 988 snap 317 seq_num 988) dirty exists 2024-09-11T01:48:03.516 INFO:tasks.rados.rados.0.smithi071.stdout:2794: left oid 31 (ObjNum 988 snap 317 seq_num 988) 2024-09-11T01:48:03.516 INFO:tasks.rados.rados.0.smithi071.stdout:2794: done (2 left) 2024-09-11T01:48:03.516 INFO:tasks.rados.rados.0.smithi071.stdout:2798: snap_remove snap 302 2024-09-11T01:48:04.093 INFO:tasks.rados.rados.0.smithi071.stdout:2798: done (2 left) 2024-09-11T01:48:04.093 INFO:tasks.rados.rados.0.smithi071.stdout:2797: finishing rollback tid 0 to smithi07134777-16 2024-09-11T01:48:04.093 INFO:tasks.rados.rados.0.smithi071.stdout:2797: finishing rollback tid 1 to smithi07134777-16 2024-09-11T01:48:04.094 INFO:tasks.rados.rados.0.smithi071.stdout:2797: finishing rollback tid 2 to smithi07134777-16 2024-09-11T01:48:04.094 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 1632 (ObjNum 411 snap 115 seq_num 411) dirty exists 2024-09-11T01:48:04.094 INFO:tasks.rados.rados.0.smithi071.stdout:2796: expect (ObjNum 779 snap 236 seq_num 779) 2024-09-11T01:48:04.138 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3024298 2024-09-11T01:48:04.139 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:48:04.246 INFO:tasks.rados.rados.0.smithi071.stdout:2799: read oid 48 snap -1 2024-09-11T01:48:04.246 INFO:tasks.rados.rados.0.smithi071.stdout:2799: expect (ObjNum 875 snap 267 seq_num 875) 2024-09-11T01:48:04.246 INFO:tasks.rados.rados.0.smithi071.stdout:2796: done (2 left) 2024-09-11T01:48:04.246 INFO:tasks.rados.rados.0.smithi071.stdout:2797: done (1 left) 2024-09-11T01:48:04.246 INFO:tasks.rados.rados.0.smithi071.stdout:2800: delete oid 29 current snap is 317 2024-09-11T01:48:04.250 INFO:tasks.rados.rados.0.smithi071.stdout:2799: expect (ObjNum 875 snap 267 seq_num 875) 2024-09-11T01:48:04.251 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:48:04.280 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 365335 2024-09-11T01:48:04.332 INFO:tasks.rados.rados.0.smithi071.stdout:2799: done (1 left) 2024-09-11T01:48:04.332 INFO:tasks.rados.rados.0.smithi071.stdout:2800: done (0 left) 2024-09-11T01:48:04.332 INFO:tasks.rados.rados.0.smithi071.stdout:2801: write (excl) oid 19 current snap is 317 2024-09-11T01:48:04.333 INFO:tasks.rados.rados.0.smithi071.stdout:2801: seq_num 990 ranges {638930=422400,1665644=645682} 2024-09-11T01:48:04.338 INFO:tasks.rados.rados.0.smithi071.stdout:2801: writing smithi07134777-19 from 638930 to 1061330 tid 1 2024-09-11T01:48:04.345 INFO:tasks.rados.rados.0.smithi071.stdout:2801: writing smithi07134777-19 from 1665644 to 2311326 tid 2 2024-09-11T01:48:04.345 INFO:tasks.rados.rados.0.smithi071.stdout:2802: copy_from oid 35 from oid 49 current snap is 317 2024-09-11T01:48:04.345 INFO:tasks.rados.rados.0.smithi071.stdout:2801: finishing write tid 1 to smithi07134777-19 2024-09-11T01:48:04.345 INFO:tasks.rados.rados.0.smithi071.stdout:2803: delete oid 34 current snap is 317 2024-09-11T01:48:04.355 INFO:tasks.rados.rados.0.smithi071.stdout:2801: finishing write tid 2 to smithi07134777-19 2024-09-11T01:48:04.355 INFO:tasks.rados.rados.0.smithi071.stdout:2801: finishing write tid 3 to smithi07134777-19 2024-09-11T01:48:04.355 INFO:tasks.rados.rados.0.smithi071.stdout:2801: finishing write tid 4 to smithi07134777-19 2024-09-11T01:48:04.355 INFO:tasks.rados.rados.0.smithi071.stdout:2801: oid 19 updating version 0 to 1074 2024-09-11T01:48:04.355 INFO:tasks.rados.rados.0.smithi071.stdout:2801: oid 19 updating version 1074 to 1076 2024-09-11T01:48:04.355 INFO:tasks.rados.rados.0.smithi071.stdout:2801: oid 19 version 1076 is already newer than 1075 2024-09-11T01:48:04.355 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 1076 (ObjNum 990 snap 317 seq_num 990) dirty exists 2024-09-11T01:48:04.355 INFO:tasks.rados.rados.0.smithi071.stdout:2801: left oid 19 (ObjNum 990 snap 317 seq_num 990) 2024-09-11T01:48:04.355 INFO:tasks.rados.rados.0.smithi071.stdout:2801: done (2 left) 2024-09-11T01:48:04.356 INFO:tasks.rados.rados.0.smithi071.stdout:2803: done (1 left) 2024-09-11T01:48:04.356 INFO:tasks.rados.rados.0.smithi071.stdout:2804: write (excl) oid 34 current snap is 317 2024-09-11T01:48:04.356 INFO:tasks.rados.rados.0.smithi071.stdout:2804: seq_num 992 ranges {672544=572898,1666255=532998} 2024-09-11T01:48:04.359 INFO:tasks.rados.rados.0.smithi071.stdout:2804: writing smithi07134777-34 from 672544 to 1245442 tid 1 2024-09-11T01:48:04.365 INFO:tasks.rados.rados.0.smithi071.stdout:2804: writing smithi07134777-34 from 1666255 to 2199253 tid 2 2024-09-11T01:48:04.365 INFO:tasks.rados.rados.0.smithi071.stdout:2805: read oid 14 snap -1 2024-09-11T01:48:04.365 INFO:tasks.rados.rados.0.smithi071.stdout:2805: expect (ObjNum 942 snap 296 seq_num 942) 2024-09-11T01:48:04.365 INFO:tasks.rados.rados.0.smithi071.stdout:2806: read oid 7 snap -1 2024-09-11T01:48:04.365 INFO:tasks.rados.rados.0.smithi071.stdout:2806: expect (ObjNum 984 snap 314 seq_num 984) 2024-09-11T01:48:04.365 INFO:tasks.rados.rados.0.smithi071.stdout:2807: read oid 23 snap -1 2024-09-11T01:48:04.365 INFO:tasks.rados.rados.0.smithi071.stdout:2807: expect (ObjNum 689 snap 204 seq_num 689) 2024-09-11T01:48:04.365 INFO:tasks.rados.rados.0.smithi071.stdout:2808: snap_create 2024-09-11T01:48:04.365 INFO:tasks.rados.rados.0.smithi071.stdout:2802: finishing copy_from racing read to smithi07134777-35 2024-09-11T01:48:04.369 INFO:tasks.rados.rados.0.smithi071.stdout:2805: expect (ObjNum 942 snap 296 seq_num 942) 2024-09-11T01:48:04.516 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:48:04.590 INFO:tasks.rados.rados.0.smithi071.stdout:2807: expect (ObjNum 689 snap 204 seq_num 689) 2024-09-11T01:48:04.692 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:48:04.745 INFO:tasks.rados.rados.0.smithi071.stdout:2804: finishing write tid 1 to smithi07134777-34 2024-09-11T01:48:04.745 INFO:tasks.rados.rados.0.smithi071.stdout:2802: finishing copy_from to smithi07134777-35 2024-09-11T01:48:04.745 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 384 (ObjNum 949 snap 300 seq_num 949) dirty exists 2024-09-11T01:48:04.745 INFO:tasks.rados.rados.0.smithi071.stdout:2804: finishing write tid 2 to smithi07134777-34 2024-09-11T01:48:04.746 INFO:tasks.rados.rados.0.smithi071.stdout:2804: finishing write tid 3 to smithi07134777-34 2024-09-11T01:48:04.746 INFO:tasks.rados.rados.0.smithi071.stdout:2804: finishing write tid 4 to smithi07134777-34 2024-09-11T01:48:04.746 INFO:tasks.rados.rados.0.smithi071.stdout:2804: oid 34 updating version 0 to 1773 2024-09-11T01:48:04.746 INFO:tasks.rados.rados.0.smithi071.stdout:2804: oid 34 version 1773 is already newer than 1772 2024-09-11T01:48:04.746 INFO:tasks.rados.rados.0.smithi071.stdout:2804: oid 34 version 1773 is already newer than 1771 2024-09-11T01:48:04.746 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 1773 (ObjNum 992 snap 317 seq_num 992) dirty exists 2024-09-11T01:48:04.746 INFO:tasks.rados.rados.0.smithi071.stdout:2804: left oid 34 (ObjNum 992 snap 317 seq_num 992) 2024-09-11T01:48:04.746 INFO:tasks.rados.rados.0.smithi071.stdout:2806: expect (ObjNum 984 snap 314 seq_num 984) 2024-09-11T01:48:04.802 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:48:05.096 INFO:tasks.rados.rados.0.smithi071.stdout:2802: done (5 left) 2024-09-11T01:48:05.097 INFO:tasks.rados.rados.0.smithi071.stdout:2804: done (4 left) 2024-09-11T01:48:05.097 INFO:tasks.rados.rados.0.smithi071.stdout:2805: done (3 left) 2024-09-11T01:48:05.097 INFO:tasks.rados.rados.0.smithi071.stdout:2806: done (2 left) 2024-09-11T01:48:05.097 INFO:tasks.rados.rados.0.smithi071.stdout:2807: done (1 left) 2024-09-11T01:48:05.097 INFO:tasks.rados.rados.0.smithi071.stdout:2808: done (0 left) 2024-09-11T01:48:05.097 INFO:tasks.rados.rados.0.smithi071.stdout:2809: copy_from oid 43 from oid 15 current snap is 318 2024-09-11T01:48:05.097 INFO:tasks.rados.rados.0.smithi071.stdout:2810: read oid 24 snap -1 2024-09-11T01:48:05.097 INFO:tasks.rados.rados.0.smithi071.stdout:2810: expect (ObjNum 918 snap 290 seq_num 918) 2024-09-11T01:48:05.097 INFO:tasks.rados.rados.0.smithi071.stdout:2811: write oid 1 current snap is 318 2024-09-11T01:48:05.097 INFO:tasks.rados.rados.0.smithi071.stdout:2811: seq_num 993 ranges {689351=648147,2124737=564614} 2024-09-11T01:48:05.103 INFO:tasks.rados.rados.0.smithi071.stdout:2811: writing smithi07134777-1 from 689351 to 1337498 tid 1 2024-09-11T01:48:05.111 INFO:tasks.rados.rados.0.smithi071.stdout:2811: writing smithi07134777-1 from 2124737 to 2689351 tid 2 2024-09-11T01:48:05.111 INFO:tasks.rados.rados.0.smithi071.stdout:2812: write (excl) oid 39 current snap is 318 2024-09-11T01:48:05.111 INFO:tasks.rados.rados.0.smithi071.stdout:2811: finishing write tid 1 to smithi07134777-1 2024-09-11T01:48:05.111 INFO:tasks.rados.rados.0.smithi071.stdout:2812: seq_num 994 ranges {706158=723396,2183219=522939} 2024-09-11T01:48:05.119 INFO:tasks.rados.rados.0.smithi071.stdout:2812: writing smithi07134777-39 from 706158 to 1429554 tid 1 2024-09-11T01:48:05.126 INFO:tasks.rados.rados.0.smithi071.stdout:2812: writing smithi07134777-39 from 2183219 to 2706158 tid 2 2024-09-11T01:48:05.126 INFO:tasks.rados.rados.0.smithi071.stdout:2813: write oid 14 current snap is 318 2024-09-11T01:48:05.126 INFO:tasks.rados.rados.0.smithi071.stdout:2813: seq_num 995 ranges {722965=798645,2125348=597617} 2024-09-11T01:48:05.134 INFO:tasks.rados.rados.0.smithi071.stdout:2813: writing smithi07134777-14 from 722965 to 1521610 tid 1 2024-09-11T01:48:05.142 INFO:tasks.rados.rados.0.smithi071.stdout:2813: writing smithi07134777-14 from 2125348 to 2722965 tid 2 2024-09-11T01:48:05.142 INFO:tasks.rados.rados.0.smithi071.stdout:2814: read oid 41 snap -1 2024-09-11T01:48:05.142 INFO:tasks.rados.rados.0.smithi071.stdout:2814: expect (ObjNum 960 snap 304 seq_num 960) 2024-09-11T01:48:05.142 INFO:tasks.rados.rados.0.smithi071.stdout:2815: read oid 31 snap -1 2024-09-11T01:48:05.142 INFO:tasks.rados.rados.0.smithi071.stdout:2815: expect (ObjNum 988 snap 317 seq_num 988) 2024-09-11T01:48:05.142 INFO:tasks.rados.rados.0.smithi071.stdout:2816: rollback oid 29 current snap is 318 2024-09-11T01:48:05.142 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 29 to 305 2024-09-11T01:48:05.142 INFO:tasks.rados.rados.0.smithi071.stdout:2810: expect (ObjNum 918 snap 290 seq_num 918) 2024-09-11T01:48:05.142 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:48:05.219 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:48:05.370 INFO:tasks.rados.rados.0.smithi071.stdout:2809: finishing copy_from racing read to smithi07134777-43 2024-09-11T01:48:05.370 INFO:tasks.rados.rados.0.smithi071.stdout:2811: finishing write tid 2 to smithi07134777-1 2024-09-11T01:48:05.370 INFO:tasks.rados.rados.0.smithi071.stdout:2811: finishing write tid 3 to smithi07134777-1 2024-09-11T01:48:05.370 INFO:tasks.rados.rados.0.smithi071.stdout:2809: finishing copy_from to smithi07134777-43 2024-09-11T01:48:05.370 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 1913 (ObjNum 975 snap 305 seq_num 975) dirty exists 2024-09-11T01:48:05.370 INFO:tasks.rados.rados.0.smithi071.stdout:2811: finishing write tid 4 to smithi07134777-1 2024-09-11T01:48:05.370 INFO:tasks.rados.rados.0.smithi071.stdout:2811: oid 1 updating version 0 to 1761 2024-09-11T01:48:05.370 INFO:tasks.rados.rados.0.smithi071.stdout:2811: oid 1 updating version 1761 to 1762 2024-09-11T01:48:05.370 INFO:tasks.rados.rados.0.smithi071.stdout:2811: oid 1 updating version 1762 to 1763 2024-09-11T01:48:05.370 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 1763 (ObjNum 993 snap 318 seq_num 993) dirty exists 2024-09-11T01:48:05.371 INFO:tasks.rados.rados.0.smithi071.stdout:2811: left oid 1 (ObjNum 993 snap 318 seq_num 993) 2024-09-11T01:48:05.371 INFO:tasks.rados.rados.0.smithi071.stdout:2809: done (7 left) 2024-09-11T01:48:05.371 INFO:tasks.rados.rados.0.smithi071.stdout:2810: done (6 left) 2024-09-11T01:48:05.371 INFO:tasks.rados.rados.0.smithi071.stdout:2811: done (5 left) 2024-09-11T01:48:05.371 INFO:tasks.rados.rados.0.smithi071.stdout:2817: delete oid 26 current snap is 318 2024-09-11T01:48:05.371 INFO:tasks.rados.rados.0.smithi071.stdout:2812: finishing write tid 1 to smithi07134777-39 2024-09-11T01:48:05.371 INFO:tasks.rados.rados.0.smithi071.stdout:2812: finishing write tid 2 to smithi07134777-39 2024-09-11T01:48:05.371 INFO:tasks.rados.rados.0.smithi071.stdout:2812: finishing write tid 3 to smithi07134777-39 2024-09-11T01:48:05.371 INFO:tasks.rados.rados.0.smithi071.stdout:2812: finishing write tid 4 to smithi07134777-39 2024-09-11T01:48:05.371 INFO:tasks.rados.rados.0.smithi071.stdout:2812: oid 39 updating version 0 to 1040 2024-09-11T01:48:05.371 INFO:tasks.rados.rados.0.smithi071.stdout:2812: oid 39 version 1040 is already newer than 1038 2024-09-11T01:48:05.371 INFO:tasks.rados.rados.0.smithi071.stdout:2812: oid 39 version 1040 is already newer than 1039 2024-09-11T01:48:05.371 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 1040 (ObjNum 994 snap 318 seq_num 994) dirty exists 2024-09-11T01:48:05.371 INFO:tasks.rados.rados.0.smithi071.stdout:2812: left oid 39 (ObjNum 994 snap 318 seq_num 994) 2024-09-11T01:48:05.372 INFO:tasks.rados.rados.0.smithi071.stdout:2813: finishing write tid 1 to smithi07134777-14 2024-09-11T01:48:05.372 INFO:tasks.rados.rados.0.smithi071.stdout:2814: expect (ObjNum 960 snap 304 seq_num 960) 2024-09-11T01:48:05.372 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 520206 2024-09-11T01:48:05.409 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:48:05.431 INFO:tasks.rados.rados.0.smithi071.stdout:2815: expect (ObjNum 988 snap 317 seq_num 988) 2024-09-11T01:48:05.579 INFO:tasks.rados.rados.0.smithi071.stdout:2813: finishing write tid 2 to smithi07134777-14 2024-09-11T01:48:05.579 INFO:tasks.rados.rados.0.smithi071.stdout:2813: finishing write tid 3 to smithi07134777-14 2024-09-11T01:48:05.579 INFO:tasks.rados.rados.0.smithi071.stdout:2813: finishing write tid 4 to smithi07134777-14 2024-09-11T01:48:05.579 INFO:tasks.rados.rados.0.smithi071.stdout:2813: oid 14 updating version 0 to 1950 2024-09-11T01:48:05.579 INFO:tasks.rados.rados.0.smithi071.stdout:2813: oid 14 updating version 1950 to 1951 2024-09-11T01:48:05.579 INFO:tasks.rados.rados.0.smithi071.stdout:2813: oid 14 version 1951 is already newer than 1949 2024-09-11T01:48:05.579 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 1951 (ObjNum 995 snap 318 seq_num 995) dirty exists 2024-09-11T01:48:05.579 INFO:tasks.rados.rados.0.smithi071.stdout:2813: left oid 14 (ObjNum 995 snap 318 seq_num 995) 2024-09-11T01:48:05.579 INFO:tasks.rados.rados.0.smithi071.stdout:2816: finishing rollback tid 1 to smithi07134777-29 2024-09-11T01:48:05.579 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 1947 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-11T01:48:05.579 INFO:tasks.rados.rados.0.smithi071.stdout:2812: done (5 left) 2024-09-11T01:48:05.580 INFO:tasks.rados.rados.0.smithi071.stdout:2813: done (4 left) 2024-09-11T01:48:05.580 INFO:tasks.rados.rados.0.smithi071.stdout:2814: done (3 left) 2024-09-11T01:48:05.580 INFO:tasks.rados.rados.0.smithi071.stdout:2815: done (2 left) 2024-09-11T01:48:05.580 INFO:tasks.rados.rados.0.smithi071.stdout:2816: done (1 left) 2024-09-11T01:48:05.580 INFO:tasks.rados.rados.0.smithi071.stdout:2817: done (0 left) 2024-09-11T01:48:05.580 INFO:tasks.rados.rados.0.smithi071.stdout:2818: snap_remove snap 303 2024-09-11T01:48:06.101 INFO:tasks.rados.rados.0.smithi071.stdout:2818: done (0 left) 2024-09-11T01:48:06.101 INFO:tasks.rados.rados.0.smithi071.stdout:2819: copy_from oid 7 from oid 10 current snap is 318 2024-09-11T01:48:06.101 INFO:tasks.rados.rados.0.smithi071.stdout:2820: copy_from oid 15 from oid 23 current snap is 318 2024-09-11T01:48:06.101 INFO:tasks.rados.rados.0.smithi071.stdout:2821: read oid 25 snap 310 2024-09-11T01:48:06.101 INFO:tasks.rados.rados.0.smithi071.stdout:2821: expect (ObjNum 723 snap 224 seq_num 723) 2024-09-11T01:48:06.101 INFO:tasks.rados.rados.0.smithi071.stdout:2822: copy_from oid 24 from oid 8 current snap is 318 2024-09-11T01:48:06.102 INFO:tasks.rados.rados.0.smithi071.stdout:2823: snap_create 2024-09-11T01:48:06.114 INFO:tasks.rados.rados.0.smithi071.stdout:2821: expect (ObjNum 723 snap 224 seq_num 723) 2024-09-11T01:48:06.114 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:48:06.166 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:48:06.190 INFO:tasks.rados.rados.0.smithi071.stdout:2822: finishing copy_from racing read to smithi07134777-24 2024-09-11T01:48:06.190 INFO:tasks.rados.rados.0.smithi071.stdout:2820: finishing copy_from racing read to smithi07134777-15 2024-09-11T01:48:06.191 INFO:tasks.rados.rados.0.smithi071.stdout:2819: finishing copy_from to smithi07134777-7 2024-09-11T01:48:06.191 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 475 (ObjNum 601 snap 172 seq_num 601) dirty exists 2024-09-11T01:48:06.191 INFO:tasks.rados.rados.0.smithi071.stdout:2819: finishing copy_from racing read to smithi07134777-7 2024-09-11T01:48:06.191 INFO:tasks.rados.rados.0.smithi071.stdout:2822: finishing copy_from to smithi07134777-24 2024-09-11T01:48:06.191 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 1070 (ObjNum 937 snap 293 seq_num 937) dirty exists 2024-09-11T01:48:06.191 INFO:tasks.rados.rados.0.smithi071.stdout:2820: finishing copy_from to smithi07134777-15 2024-09-11T01:48:06.191 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 1039 (ObjNum 689 snap 204 seq_num 689) dirty exists 2024-09-11T01:48:07.105 INFO:tasks.rados.rados.0.smithi071.stdout:2819: done (4 left) 2024-09-11T01:48:07.105 INFO:tasks.rados.rados.0.smithi071.stdout:2820: done (3 left) 2024-09-11T01:48:07.106 INFO:tasks.rados.rados.0.smithi071.stdout:2821: done (2 left) 2024-09-11T01:48:07.106 INFO:tasks.rados.rados.0.smithi071.stdout:2822: done (1 left) 2024-09-11T01:48:07.106 INFO:tasks.rados.rados.0.smithi071.stdout:2823: done (0 left) 2024-09-11T01:48:07.106 INFO:tasks.rados.rados.0.smithi071.stdout:2824: read oid 48 snap 315 2024-09-11T01:48:07.106 INFO:tasks.rados.rados.0.smithi071.stdout:2824: expect (ObjNum 875 snap 267 seq_num 875) 2024-09-11T01:48:07.106 INFO:tasks.rados.rados.0.smithi071.stdout:2825: snap_create 2024-09-11T01:48:07.110 INFO:tasks.rados.rados.0.smithi071.stdout:2824: expect (ObjNum 875 snap 267 seq_num 875) 2024-09-11T01:48:07.110 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:48:07.139 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 75709 2024-09-11T01:48:08.088 INFO:tasks.rados.rados.0.smithi071.stdout:2824: done (1 left) 2024-09-11T01:48:08.089 INFO:tasks.rados.rados.0.smithi071.stdout:2825: done (0 left) 2024-09-11T01:48:08.089 INFO:tasks.rados.rados.0.smithi071.stdout:2826: snap_create 2024-09-11T01:48:09.094 INFO:tasks.rados.rados.0.smithi071.stdout:2826: done (0 left) 2024-09-11T01:48:09.094 INFO:tasks.rados.rados.0.smithi071.stdout:2827: read oid 48 snap 299 2024-09-11T01:48:09.094 INFO:tasks.rados.rados.0.smithi071.stdout:2827: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:48:09.095 INFO:tasks.rados.rados.0.smithi071.stdout:2828: snap_remove snap 286 2024-09-11T01:48:10.102 INFO:tasks.rados.rados.0.smithi071.stdout:2828: done (1 left) 2024-09-11T01:48:10.102 INFO:tasks.rados.rados.0.smithi071.stdout:2829: write (excl) oid 30 current snap is 321 2024-09-11T01:48:10.102 INFO:tasks.rados.rados.0.smithi071.stdout:2829: seq_num 997 ranges {756579=665496,1958665=400347} 2024-09-11T01:48:10.110 INFO:tasks.rados.rados.0.smithi071.stdout:2829: writing smithi07134777-30 from 756579 to 1422075 tid 1 2024-09-11T01:48:10.116 INFO:tasks.rados.rados.0.smithi071.stdout:2829: writing smithi07134777-30 from 1958665 to 2359012 tid 2 2024-09-11T01:48:10.116 INFO:tasks.rados.rados.0.smithi071.stdout:2830: read oid 47 snap 271 2024-09-11T01:48:10.116 INFO:tasks.rados.rados.0.smithi071.stdout:2830: expect (ObjNum 882 snap 271 seq_num 882) 2024-09-11T01:48:10.116 INFO:tasks.rados.rados.0.smithi071.stdout:2831: delete oid 39 current snap is 321 2024-09-11T01:48:10.116 INFO:tasks.rados.rados.0.smithi071.stdout:2827: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:48:10.146 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3257671 2024-09-11T01:48:10.186 INFO:tasks.rados.rados.0.smithi071.stdout:2829: finishing write tid 1 to smithi07134777-30 2024-09-11T01:48:10.186 INFO:tasks.rados.rados.0.smithi071.stdout:2829: finishing write tid 2 to smithi07134777-30 2024-09-11T01:48:10.186 INFO:tasks.rados.rados.0.smithi071.stdout:2829: finishing write tid 3 to smithi07134777-30 2024-09-11T01:48:10.187 INFO:tasks.rados.rados.0.smithi071.stdout:2829: finishing write tid 4 to smithi07134777-30 2024-09-11T01:48:10.187 INFO:tasks.rados.rados.0.smithi071.stdout:2829: oid 30 updating version 0 to 1296 2024-09-11T01:48:10.187 INFO:tasks.rados.rados.0.smithi071.stdout:2829: oid 30 version 1296 is already newer than 1295 2024-09-11T01:48:10.187 INFO:tasks.rados.rados.0.smithi071.stdout:2829: oid 30 version 1296 is already newer than 1294 2024-09-11T01:48:10.187 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 1296 (ObjNum 997 snap 321 seq_num 997) dirty exists 2024-09-11T01:48:10.187 INFO:tasks.rados.rados.0.smithi071.stdout:2829: left oid 30 (ObjNum 997 snap 321 seq_num 997) 2024-09-11T01:48:10.187 INFO:tasks.rados.rados.0.smithi071.stdout:2830: expect (ObjNum 882 snap 271 seq_num 882) 2024-09-11T01:48:10.292 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:48:10.346 INFO:tasks.rados.rados.0.smithi071.stdout:2827: done (3 left) 2024-09-11T01:48:10.346 INFO:tasks.rados.rados.0.smithi071.stdout:2829: done (2 left) 2024-09-11T01:48:10.346 INFO:tasks.rados.rados.0.smithi071.stdout:2830: done (1 left) 2024-09-11T01:48:10.346 INFO:tasks.rados.rados.0.smithi071.stdout:2831: done (0 left) 2024-09-11T01:48:10.346 INFO:tasks.rados.rados.0.smithi071.stdout:2832: snap_remove snap 276 2024-09-11T01:48:11.112 INFO:tasks.rados.rados.0.smithi071.stdout:2832: done (0 left) 2024-09-11T01:48:11.112 INFO:tasks.rados.rados.0.smithi071.stdout:2833: snap_create 2024-09-11T01:48:12.113 INFO:tasks.rados.rados.0.smithi071.stdout:2833: done (0 left) 2024-09-11T01:48:12.114 INFO:tasks.rados.rados.0.smithi071.stdout:2834: write (excl) oid 32 current snap is 322 2024-09-11T01:48:12.114 INFO:tasks.rados.rados.0.smithi071.stdout:2834: seq_num 999 ranges {790193=415994,1959276=404016} 2024-09-11T01:48:12.120 INFO:tasks.rados.rados.0.smithi071.stdout:2834: writing smithi07134777-32 from 790193 to 1206187 tid 1 2024-09-11T01:48:12.126 INFO:tasks.rados.rados.0.smithi071.stdout:2834: writing smithi07134777-32 from 1959276 to 2363292 tid 2 2024-09-11T01:48:12.126 INFO:tasks.rados.rados.0.smithi071.stdout:2835: snap_remove snap 311 2024-09-11T01:48:13.092 INFO:tasks.rados.rados.0.smithi071.stdout:2835: done (1 left) 2024-09-11T01:48:13.092 INFO:tasks.rados.rados.0.smithi071.stdout:2836: read oid 9 snap -1 2024-09-11T01:48:13.092 INFO:tasks.rados.rados.0.smithi071.stdout:2836: expect (ObjNum 972 snap 305 seq_num 972) 2024-09-11T01:48:13.093 INFO:tasks.rados.rados.0.smithi071.stdout:2837: snap_remove snap 259 2024-09-11T01:48:13.705 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-11T01:48:13.705 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T01:48:13.705 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T01:48:13.709 INFO:tasks.ceph.mon.a:Started 2024-09-11T01:48:13.709 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-11T01:48:13.709 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-11T01:48:13.709 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T01:48:28.710 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-11T01:48:28.711 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-11T01:48:28.711 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:48:28.712 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:48:28.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.822+0000 7fd6fab53640 10 monclient: get_monmap_and_config 2024-09-11T01:48:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.822+0000 7fd6fab53640 10 monclient: build_initial_monmap 2024-09-11T01:48:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.822+0000 7fd6fab53640 10 monclient: monmap: 2024-09-11T01:48:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:48:28.823085+0000 2024-09-11T01:48:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:48:28.823085+0000 2024-09-11T01:48:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:28.824 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:48:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:48:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:48:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:48:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:48:28.825 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:28.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.823+0000 7fd6fab53640 1 Processor -- start 2024-09-11T01:48:28.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.823+0000 7fd6fab53640 1 -- start start 2024-09-11T01:48:28.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.823+0000 7fd6fab53640 10 monclient: init 2024-09-11T01:48:28.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.823+0000 7fd6fab53640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:28.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.823+0000 7fd6fab53640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-11T01:48:28.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.823+0000 7fd6fab53640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6f409d790 addr v1:172.21.15.71:6789/0 2024-09-11T01:48:28.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.823+0000 7fd6fab53640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6f409e280 addr v1:172.21.15.12:6789/0 2024-09-11T01:48:28.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.823+0000 7fd6fab53640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd6f409e6d0 0x7fd6f413d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:28.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.823+0000 7fd6fab53640 10 monclient(hunting): picked mon.noname-a con 0x7fd6f409e6d0 addr v2:172.21.15.12:3300/0 2024-09-11T01:48:28.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.823+0000 7fd6fab53640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6f40a1da0 con 0x7fd6f409e280 2024-09-11T01:48:28.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.823+0000 7fd6fab53640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6f40a0760 con 0x7fd6f409d790 2024-09-11T01:48:28.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.823+0000 7fd6fab53640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:28.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.823+0000 7fd6fab53640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fd6f40a1fa0 con 0x7fd6f409e6d0 2024-09-11T01:48:28.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.823+0000 7fd6fab53640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:28.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.823+0000 7fd6fab53640 10 monclient(hunting): authenticate will time out at 1915.265902s 2024-09-11T01:48:28.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.823+0000 7fd6f90f0640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd6f409e6d0 0x7fd6f413d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:28.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.824+0000 7fd6f3fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fd6f409e280 0x7fd6f4139f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46168/0 (socket says 172.21.15.12:46168) 2024-09-11T01:48:28.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.824+0000 7fd6f3fff640 1 -- 172.21.15.12:0/4013474181 learned_addr learned my addr 172.21.15.12:0/4013474181 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:48:28.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.824+0000 7fd6f90f0640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd6f409e6d0 0x7fd6f413d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36652/0 (socket says 172.21.15.12:36652) 2024-09-11T01:48:28.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.824+0000 7fd6f90f0640 10 monclient(hunting): get_auth_request con 0x7fd6f409e6d0 auth_method 0 2024-09-11T01:48:28.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.824+0000 7fd6f90f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:28.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.824+0000 7fd6f90f0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:28.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.824+0000 7fd6f90f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:28.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.824+0000 7fd6f90f0640 1 -- 172.21.15.12:0/4013474181 >> v2:172.21.15.12:3300/0 conn(0x7fd6f409e6d0 msgr2=0x7fd6f413d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:48:28.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.824+0000 7fd6f90f0640 1 -- 172.21.15.12:0/4013474181 >> v2:172.21.15.12:3300/0 conn(0x7fd6f409e6d0 msgr2=0x7fd6f413d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:48:28.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.824+0000 7fd6f90f0640 1 --2- 172.21.15.12:0/4013474181 >> v2:172.21.15.12:3300/0 conn(0x7fd6f409e6d0 0x7fd6f413d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:48:28.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.824+0000 7fd6f90f0640 1 --2- 172.21.15.12:0/4013474181 >> v2:172.21.15.12:3300/0 conn(0x7fd6f409e6d0 0x7fd6f413d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:48:28.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.826+0000 7fd6f3fff640 1 -- 172.21.15.12:0/4013474181 >> v1:172.21.15.12:6789/0 conn(0x7fd6f409e280 legacy=0x7fd6f4139f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:28.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:28.826+0000 7fd6f88ef640 1 -- 172.21.15.12:0/4013474181 >> v1:172.21.15.71:6789/0 conn(0x7fd6f409d790 legacy=0x7fd6f409db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:29.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.025+0000 7fd6f90f0640 1 --2- 172.21.15.12:0/4013474181 >> v2:172.21.15.12:3300/0 conn(0x7fd6f409e6d0 0x7fd6f413d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.025+0000 7fd6f90f0640 10 monclient(hunting): get_auth_request con 0x7fd6f409e6d0 auth_method 0 2024-09-11T01:48:29.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.025+0000 7fd6f90f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:29.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.025+0000 7fd6f90f0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:29.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.025+0000 7fd6f90f0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:48:29.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.025+0000 7fd6f90f0640 1 -- 172.21.15.12:0/4013474181 >> v2:172.21.15.12:3300/0 conn(0x7fd6f409e6d0 msgr2=0x7fd6f413d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:48:29.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.025+0000 7fd6f90f0640 1 -- 172.21.15.12:0/4013474181 >> v2:172.21.15.12:3300/0 conn(0x7fd6f409e6d0 msgr2=0x7fd6f413d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:48:29.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.025+0000 7fd6f90f0640 1 --2- 172.21.15.12:0/4013474181 >> v2:172.21.15.12:3300/0 conn(0x7fd6f409e6d0 0x7fd6f413d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:48:29.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.025+0000 7fd6f90f0640 1 --2- 172.21.15.12:0/4013474181 >> v2:172.21.15.12:3300/0 conn(0x7fd6f409e6d0 0x7fd6f413d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:48:29.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.132+0000 7fd6f37fe640 1 -- 172.21.15.12:0/4013474181 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2502271387 0 0) 0x7fd6f40a1da0 con 0x7fd6f409e280 2024-09-11T01:48:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.132+0000 7fd6f37fe640 10 monclient(hunting): _negotiate 2024-09-11T01:48:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.132+0000 7fd6f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.132+0000 7fd6f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.132+0000 7fd6f37fe640 10 monclient(hunting): my global_id is 214098 2024-09-11T01:48:29.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.132+0000 7fd6f37fe640 1 -- 172.21.15.12:0/4013474181 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6e0003520 con 0x7fd6f409e280 2024-09-11T01:48:29.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 1 -- 172.21.15.12:0/4013474181 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4076198712 0 0) 0x7fd6e0003520 con 0x7fd6f409e280 2024-09-11T01:48:29.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 1 -- 172.21.15.12:0/4013474181 >> v2:172.21.15.12:3300/0 conn(0x7fd6f409e6d0 msgr2=0x7fd6f413d650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:48:29.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 1 --2- 172.21.15.12:0/4013474181 >> v2:172.21.15.12:3300/0 conn(0x7fd6f409e6d0 0x7fd6f413d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 1 -- 172.21.15.12:0/4013474181 >> v1:172.21.15.71:6789/0 conn(0x7fd6f409d790 legacy=0x7fd6f409db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 10 monclient: _finish_hunting 0 2024-09-11T01:48:29.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:48:29.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:29.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:48:29.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 1 -- 172.21.15.12:0/4013474181 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f409eda0 con 0x7fd6f409e280 2024-09-11T01:48:29.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 10 monclient: _finish_auth 0 2024-09-11T01:48:29.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 10 monclient: _check_auth_tickets 2024-09-11T01:48:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 1 -- 172.21.15.12:0/4013474181 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7fd6e40019a0 con 0x7fd6f409e280 2024-09-11T01:48:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6fab53640 5 monclient: authenticate success, global_id 214098 2024-09-11T01:48:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6fab53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:48:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:48:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 10 monclient: dump: 2024-09-11T01:48:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:29.139 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:48:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 1 -- 172.21.15.12:0/4013474181 >> v1:172.21.15.12:6789/0 conn(0x7fd6f409e280 legacy=0x7fd6f4139f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:48:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 1 --2- 172.21.15.12:0/4013474181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6e0003b20 0x7fd6e0003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 10 monclient(hunting): picked mon.c con 0x7fd6e0003b20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 1 --2- 172.21.15.12:0/4013474181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6e0004800 0x7fd6e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 10 monclient(hunting): picked mon.a con 0x7fd6e0004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:48:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 1 --2- 172.21.15.12:0/4013474181 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6e000bbc0 0x7fd6e000dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 10 monclient(hunting): picked mon.b con 0x7fd6e000bbc0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:48:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 1 -- 172.21.15.12:0/4013474181 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f40a1fa0 con 0x7fd6e0004800 2024-09-11T01:48:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 1 -- 172.21.15.12:0/4013474181 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e000e5d0 con 0x7fd6e000bbc0 2024-09-11T01:48:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 1 -- 172.21.15.12:0/4013474181 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd6e000e7d0 con 0x7fd6e0003b20 2024-09-11T01:48:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f37fe640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:48:29.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6fab53640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:48:29.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f88ef640 1 --2- 172.21.15.12:0/4013474181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6e0003b20 0x7fd6e0003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f3fff640 1 --2- 172.21.15.12:0/4013474181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6e0004800 0x7fd6e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f88ef640 10 monclient(hunting): get_auth_request con 0x7fd6e0003b20 auth_method 0 2024-09-11T01:48:29.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f88ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:29.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f88ef640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:29.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f88ef640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:48:29.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f3fff640 10 monclient(hunting): get_auth_request con 0x7fd6e0004800 auth_method 0 2024-09-11T01:48:29.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:29.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:29.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:48:29.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f90f0640 1 --2- 172.21.15.12:0/4013474181 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6e000bbc0 0x7fd6e000dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f90f0640 10 monclient(hunting): get_auth_request con 0x7fd6e000bbc0 auth_method 0 2024-09-11T01:48:29.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f90f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:29.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f90f0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:29.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f90f0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:48:29.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f88ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:29.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f88ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:29.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f88ef640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:48:29.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:29.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:29.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:48:29.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f90f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:29.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f90f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:29.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f90f0640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:48:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f3fff640 10 monclient(hunting): handle_auth_done global_id 214098 payload 293 2024-09-11T01:48:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f3fff640 1 -- 172.21.15.12:0/4013474181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6e0003b20 msgr2=0x7fd6e0003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:48:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f3fff640 1 --2- 172.21.15.12:0/4013474181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6e0003b20 0x7fd6e0003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f3fff640 1 -- 172.21.15.12:0/4013474181 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6e000bbc0 msgr2=0x7fd6e000dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:48:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f3fff640 1 --2- 172.21.15.12:0/4013474181 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6e000bbc0 0x7fd6e000dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f3fff640 10 monclient: _finish_hunting 0 2024-09-11T01:48:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f3fff640 1 monclient: found mon.a 2024-09-11T01:48:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f3fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f3fff640 1 -- 172.21.15.12:0/4013474181 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6e0003520 con 0x7fd6e0004800 2024-09-11T01:48:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f3fff640 10 monclient: _finish_auth 0 2024-09-11T01:48:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f3fff640 10 monclient: _check_auth_tickets 2024-09-11T01:48:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:29.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f90f0640 1 -- 172.21.15.12:0/4013474181 reap_dead start 2024-09-11T01:48:29.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.133+0000 7fd6f88ef640 1 --2- 172.21.15.12:0/4013474181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6e0003b20 0x7fd6e0003f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:48:29.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.137+0000 7fd6f3fff640 1 -- 172.21.15.12:0/4013474181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6e0004800 msgr2=0x7fd6e0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:29.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.137+0000 7fd6f3fff640 1 --2- 172.21.15.12:0/4013474181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6e0004800 0x7fd6e0004c80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4001bc0 tx=0x7fd6e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2513b85dcf42ef5 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:29.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.137+0000 7fd6f37fe640 1 -- 172.21.15.12:0/4013474181 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e400c0e0 con 0x7fd6e0004800 2024-09-11T01:48:29.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.137+0000 7fd6f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:29.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.137+0000 7fd6f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:48:29.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.137+0000 7fd6f37fe640 10 monclient: dump: 2024-09-11T01:48:29.149 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:29.149 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:29.149 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.149 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.149 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:29.150 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.137+0000 7fd6fab53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:48:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.137+0000 7fd6f37fe640 1 -- 172.21.15.12:0/4013474181 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6e4033070 con 0x7fd6e0004800 2024-09-11T01:48:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.137+0000 7fd6f37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.137+0000 7fd6f37fe640 1 -- 172.21.15.12:0/4013474181 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e4005d30 con 0x7fd6e0004800 2024-09-11T01:48:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.137+0000 7fd6fab53640 10 monclient: get_monmap_and_config success 2024-09-11T01:48:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.137+0000 7fd6fab53640 4 set_mon_vals no callback set 2024-09-11T01:48:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.137+0000 7fd6f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.137+0000 7fd6f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:48:29.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.137+0000 7fd6f37fe640 10 monclient: dump: 2024-09-11T01:48:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:29.151 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.137+0000 7fd6fab53640 10 monclient: shutdown 2024-09-11T01:48:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.137+0000 7fd6fab53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.137+0000 7fd6fab53640 1 -- 172.21.15.12:0/4013474181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6e0004800 msgr2=0x7fd6e0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.137+0000 7fd6fab53640 1 --2- 172.21.15.12:0/4013474181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6e0004800 0x7fd6e0004c80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4001bc0 tx=0x7fd6e0005830 comp rx=0 tx=0).stop 2024-09-11T01:48:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.138+0000 7fd6fab53640 1 -- 172.21.15.12:0/4013474181 shutdown_connections 2024-09-11T01:48:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.138+0000 7fd6fab53640 1 --2- 172.21.15.12:0/4013474181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6e0004800 0x7fd6e0004c80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.138+0000 7fd6fab53640 1 -- 172.21.15.12:0/4013474181 >> 172.21.15.12:0/4013474181 conn(0x7fd6f40a2d90 msgr2=0x7fd6f40a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.138+0000 7fd6fab53640 1 -- 172.21.15.12:0/4013474181 shutdown_connections 2024-09-11T01:48:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.138+0000 7fd6fab53640 1 -- 172.21.15.12:0/4013474181 wait complete. 2024-09-11T01:48:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.138+0000 7fd6fab53640 10 monclient: build_initial_monmap 2024-09-11T01:48:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.138+0000 7fd6fab53640 10 monclient: monmap: 2024-09-11T01:48:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:48:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:48:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:48:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.152 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:48:29.153 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:48:29.153 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:48:29.153 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.139+0000 7fd6fab53640 1 Processor -- start 2024-09-11T01:48:29.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.139+0000 7fd6fab53640 1 -- start start 2024-09-11T01:48:29.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.139+0000 7fd6fab53640 10 monclient: init 2024-09-11T01:48:29.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6fab53640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:29.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6fab53640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:48:29.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6fab53640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6e0004800 0x7fd6f4134820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6fab53640 10 monclient(hunting): picked mon.noname-c con 0x7fd6e0004800 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:29.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6fab53640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6f4134d60 0x7fd6f41214e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6fab53640 10 monclient(hunting): picked mon.noname-b con 0x7fd6f4134d60 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:48:29.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6fab53640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6f4121a20 0x7fd6f4121eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6fab53640 10 monclient(hunting): picked mon.noname-a con 0x7fd6f4121a20 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:48:29.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6fab53640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:29.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6fab53640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f409ce20 con 0x7fd6f4121a20 2024-09-11T01:48:29.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6fab53640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:29.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6fab53640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f409cca0 con 0x7fd6f4134d60 2024-09-11T01:48:29.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6fab53640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:29.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6fab53640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd6f409db70 con 0x7fd6e0004800 2024-09-11T01:48:29.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6fab53640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:29.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6fab53640 10 monclient(hunting): authenticate will time out at 1915.582131s 2024-09-11T01:48:29.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6f88ef640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6e0004800 0x7fd6f4134820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6f88ef640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6e0004800 0x7fd6f4134820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:44094/0 (socket says 172.21.15.12:44094) 2024-09-11T01:48:29.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6f88ef640 1 -- 172.21.15.12:0/2838699099 learned_addr learned my addr 172.21.15.12:0/2838699099 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:48:29.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6f88ef640 10 monclient(hunting): get_auth_request con 0x7fd6e0004800 auth_method 0 2024-09-11T01:48:29.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6f88ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:29.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6f88ef640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:29.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6f88ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:29.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6f88ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:29.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6f88ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:29.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6f88ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:29.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6f3fff640 1 --2- 172.21.15.12:0/2838699099 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6f4134d60 0x7fd6f41214e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6f88ef640 10 monclient(hunting): handle_auth_done global_id 184340 payload 824 2024-09-11T01:48:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6f88ef640 1 -- 172.21.15.12:0/2838699099 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6f4134d60 msgr2=0x7fd6f41214e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6f88ef640 1 --2- 172.21.15.12:0/2838699099 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6f4134d60 0x7fd6f41214e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6f88ef640 1 -- 172.21.15.12:0/2838699099 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6f4121a20 msgr2=0x7fd6f4121eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6f88ef640 1 --2- 172.21.15.12:0/2838699099 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6f4121a20 0x7fd6f4121eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6f88ef640 10 monclient: _finish_hunting 0 2024-09-11T01:48:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6f88ef640 1 monclient: found mon.noname-c 2024-09-11T01:48:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6f88ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6f88ef640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:48:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.140+0000 7fd6f88ef640 1 -- 172.21.15.12:0/2838699099 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f4122eb0 con 0x7fd6e0004800 2024-09-11T01:48:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.142+0000 7fd6f88ef640 1 -- 172.21.15.12:0/2838699099 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6e0004800 msgr2=0x7fd6f4134820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:29.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.142+0000 7fd6f88ef640 1 --2- 172.21.15.12:0/2838699099 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6e0004800 0x7fd6f4134820 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc00f7a0 tx=0x7fd6dc00fc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:29.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.143+0000 7fd6f17fa640 1 -- 172.21.15.12:0/2838699099 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc001690 con 0x7fd6e0004800 2024-09-11T01:48:29.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.143+0000 7fd6f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:29.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.143+0000 7fd6f17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:48:29.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.143+0000 7fd6f17fa640 10 monclient: dump: 2024-09-11T01:48:29.157 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:29.157 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:29.157 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.157 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.157 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:29.157 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.157 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:29.157 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:29.158 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:29.158 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.143+0000 7fd6f17fa640 10 monclient: _finish_auth 0 2024-09-11T01:48:29.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.143+0000 7fd6f17fa640 10 monclient: _check_auth_tickets 2024-09-11T01:48:29.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.143+0000 7fd6f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:29.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.143+0000 7fd6f17fa640 1 -- 172.21.15.12:0/2838699099 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6dc018070 con 0x7fd6e0004800 2024-09-11T01:48:29.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.143+0000 7fd6f17fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:29.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.143+0000 7fd6f17fa640 1 -- 172.21.15.12:0/2838699099 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc0057c0 con 0x7fd6e0004800 2024-09-11T01:48:29.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.143+0000 7fd6fab53640 5 monclient: authenticate success, global_id 184340 2024-09-11T01:48:29.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.143+0000 7fd6fab53640 10 monclient: _renew_subs 2024-09-11T01:48:29.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.143+0000 7fd6fab53640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:48:29.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.143+0000 7fd6fab53640 1 -- 172.21.15.12:0/2838699099 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6f41357e0 con 0x7fd6e0004800 2024-09-11T01:48:29.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.143+0000 7fd6f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:29.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.143+0000 7fd6f2ffd640 4 set_mon_vals no callback set 2024-09-11T01:48:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.143+0000 7fd6f17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:48:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.143+0000 7fd6f17fa640 10 monclient: dump: 2024-09-11T01:48:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:29.159 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.143+0000 7fd6fab53640 10 monclient: _renew_subs 2024-09-11T01:48:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.143+0000 7fd6fab53640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:48:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.143+0000 7fd6fab53640 1 -- 172.21.15.12:0/2838699099 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6f4124c50 con 0x7fd6e0004800 2024-09-11T01:48:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.144+0000 7fd6f17fa640 1 -- 172.21.15.12:0/2838699099 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fd6dc005960 con 0x7fd6e0004800 2024-09-11T01:48:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.144+0000 7fd6fab53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.144+0000 7fd6fab53640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.144+0000 7fd6fab53640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:48:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.144+0000 7fd6fab53640 1 -- 172.21.15.12:0/2838699099 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6f409e000 con 0x7fd6e0004800 2024-09-11T01:48:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.144+0000 7fd6f17fa640 1 --2- 172.21.15.12:0/2838699099 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd6d403e260 0x7fd6d4040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.144+0000 7fd6f17fa640 1 -- 172.21.15.12:0/2838699099 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(640..640 src has 507..640) ==== 4827+0+0 (secure 0 0 0) 0x7fd6dc052a30 con 0x7fd6e0004800 2024-09-11T01:48:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.144+0000 7fd6f3fff640 1 --2- 172.21.15.12:0/2838699099 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd6d403e260 0x7fd6d4040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.145+0000 7fd6f3fff640 10 monclient: get_auth_request con 0x7fd6d403e260 auth_method 0 2024-09-11T01:48:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.147+0000 7fd6f3fff640 1 --2- 172.21.15.12:0/2838699099 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd6d403e260 0x7fd6d4040720 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4001390 tx=0x7fd6e4007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:29.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.151+0000 7fd6f17fa640 1 -- 172.21.15.12:0/2838699099 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd6dc051400 con 0x7fd6e0004800 2024-09-11T01:48:29.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.151+0000 7fd6f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:29.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.151+0000 7fd6f17fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:48:29.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.410+0000 7fd6fab53640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:48:29.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.410+0000 7fd6fab53640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:48:29.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.410+0000 7fd6fab53640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:48:29.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.410+0000 7fd6fab53640 1 -- 172.21.15.12:0/2838699099 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd6f40a4f60 con 0x7fd6e0004800 2024-09-11T01:48:29.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.411+0000 7fd6f17fa640 1 -- 172.21.15.12:0/2838699099 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fd6dc00db70 con 0x7fd6e0004800 2024-09-11T01:48:29.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.411+0000 7fd6f17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:48:29.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.411+0000 7fd6f17fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:48:29.413 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":182,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:48:29.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.413+0000 7fd6fab53640 1 -- 172.21.15.12:0/2838699099 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd6d403e260 msgr2=0x7fd6d4040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.413+0000 7fd6fab53640 1 --2- 172.21.15.12:0/2838699099 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd6d403e260 0x7fd6d4040720 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4001390 tx=0x7fd6e4007000 comp rx=0 tx=0).stop 2024-09-11T01:48:29.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.413+0000 7fd6fab53640 10 monclient: shutdown 2024-09-11T01:48:29.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.413+0000 7fd6fab53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:29.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.413+0000 7fd6fab53640 1 -- 172.21.15.12:0/2838699099 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6e0004800 msgr2=0x7fd6f4134820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.413+0000 7fd6fab53640 1 --2- 172.21.15.12:0/2838699099 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6e0004800 0x7fd6f4134820 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc00f7a0 tx=0x7fd6dc00fc60 comp rx=0 tx=0).stop 2024-09-11T01:48:29.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.413+0000 7fd6fab53640 1 -- 172.21.15.12:0/2838699099 shutdown_connections 2024-09-11T01:48:29.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.413+0000 7fd6fab53640 1 --2- 172.21.15.12:0/2838699099 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd6d403e260 0x7fd6d4040720 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.413+0000 7fd6fab53640 1 --2- 172.21.15.12:0/2838699099 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd6f4121a20 0x7fd6f4121eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.413+0000 7fd6fab53640 1 --2- 172.21.15.12:0/2838699099 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6f4134d60 0x7fd6f41214e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.413+0000 7fd6fab53640 1 --2- 172.21.15.12:0/2838699099 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd6e0004800 0x7fd6f4134820 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.413+0000 7fd6fab53640 1 -- 172.21.15.12:0/2838699099 >> 172.21.15.12:0/2838699099 conn(0x7fd6f40a2d90 msgr2=0x7fd6f40a4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:29.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.413+0000 7fd6fab53640 1 -- 172.21.15.12:0/2838699099 shutdown_connections 2024-09-11T01:48:29.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.413+0000 7fd6fab53640 1 -- 172.21.15.12:0/2838699099 wait complete. 2024-09-11T01:48:29.426 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:48:29.426 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:48:29.426 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:48:29.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.528+0000 7f960caef640 10 monclient: get_monmap_and_config 2024-09-11T01:48:29.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.528+0000 7f960caef640 10 monclient: build_initial_monmap 2024-09-11T01:48:29.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.528+0000 7f960caef640 10 monclient: monmap: 2024-09-11T01:48:29.530 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:29.530 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:29.530 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:48:29.528994+0000 2024-09-11T01:48:29.530 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:48:29.528994+0000 2024-09-11T01:48:29.531 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:29.531 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.531 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:48:29.531 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:48:29.531 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:48:29.531 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:48:29.531 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:48:29.531 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.528+0000 7f960caef640 1 Processor -- start 2024-09-11T01:48:29.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f960caef640 1 -- start start 2024-09-11T01:48:29.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f960caef640 10 monclient: init 2024-09-11T01:48:29.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f960caef640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:29.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f960caef640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-11T01:48:29.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f960caef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9608137e50 addr v1:172.21.15.71:6789/0 2024-09-11T01:48:29.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f960caef640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9608138230 0x7f9608138660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f960caef640 10 monclient(hunting): picked mon.noname-b con 0x7f9608138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:29.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f960caef640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f960813d380 0x7f960813f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f960caef640 10 monclient(hunting): picked mon.noname-c con 0x7f960813d380 addr v2:172.21.15.71:3300/0 2024-09-11T01:48:29.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f960caef640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96080a6580 con 0x7f9608137e50 2024-09-11T01:48:29.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f960caef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:29.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f960caef640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f96080a62d0 con 0x7f960813d380 2024-09-11T01:48:29.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f960caef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:29.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f960caef640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f960809bb40 con 0x7f9608138230 2024-09-11T01:48:29.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f960caef640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:29.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f960caef640 10 monclient(hunting): authenticate will time out at 1915.971537s 2024-09-11T01:48:29.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f95fffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9608138230 0x7f9608138660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f95fffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9608138230 0x7f9608138660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:44098/0 (socket says 172.21.15.12:44098) 2024-09-11T01:48:29.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f95fffff640 1 -- 172.21.15.12:0/3618183637 learned_addr learned my addr 172.21.15.12:0/3618183637 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:48:29.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f95fffff640 10 monclient(hunting): get_auth_request con 0x7f9608138230 auth_method 0 2024-09-11T01:48:29.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f95fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:29.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f95fffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:29.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f95fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:29.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f95fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:29.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f95fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:29.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f95fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:29.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f9606d9d640 1 --2- 172.21.15.12:0/3618183637 >> v2:172.21.15.71:3300/0 conn(0x7f960813d380 0x7f960813f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f9606d9d640 10 monclient(hunting): get_auth_request con 0x7f960813d380 auth_method 0 2024-09-11T01:48:29.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f9606d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:29.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f9606d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:29.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f9606d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:29.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f95fffff640 10 monclient(hunting): handle_auth_done global_id 184343 payload 274 2024-09-11T01:48:29.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f95fffff640 1 -- 172.21.15.12:0/3618183637 >> v2:172.21.15.71:3300/0 conn(0x7f960813d380 msgr2=0x7f960813f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:48:29.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f95fffff640 1 --2- 172.21.15.12:0/3618183637 >> v2:172.21.15.71:3300/0 conn(0x7f960813d380 0x7f960813f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f95fffff640 1 -- 172.21.15.12:0/3618183637 >> v1:172.21.15.71:6789/0 conn(0x7f9608137e50 legacy=0x7f960809d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:48:29.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f95fffff640 10 monclient: _finish_hunting 0 2024-09-11T01:48:29.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f95fffff640 1 monclient: found mon.noname-b 2024-09-11T01:48:29.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f95fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:29.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f95fffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:48:29.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.529+0000 7f95fffff640 1 -- 172.21.15.12:0/3618183637 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f960809c4a0 con 0x7f9608138230 2024-09-11T01:48:29.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.532+0000 7f95fffff640 1 -- 172.21.15.12:0/3618183637 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9608138230 msgr2=0x7f9608138660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:29.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.532+0000 7f95fffff640 1 --2- 172.21.15.12:0/3618183637 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9608138230 0x7f9608138660 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f95f0009870 tx=0x7f95f002a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d2676721bbd4157 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:29.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.532+0000 7f9605d9b640 1 -- 172.21.15.12:0/3618183637 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f0037070 con 0x7f9608138230 2024-09-11T01:48:29.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.532+0000 7f9605d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:29.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.532+0000 7f9605d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:48:29.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.532+0000 7f9605d9b640 10 monclient: dump: 2024-09-11T01:48:29.547 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:29.547 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:29.547 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.547 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.547 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:29.547 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.547 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:29.547 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:29.547 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:29.548 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.532+0000 7f9605d9b640 10 monclient: _finish_auth 0 2024-09-11T01:48:29.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.532+0000 7f9605d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:48:29.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.532+0000 7f9605d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:29.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.532+0000 7f9605d9b640 1 -- 172.21.15.12:0/3618183637 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95f0032440 con 0x7f9608138230 2024-09-11T01:48:29.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.532+0000 7f9605d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:29.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.532+0000 7f9605d9b640 1 -- 172.21.15.12:0/3618183637 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f0032810 con 0x7f9608138230 2024-09-11T01:48:29.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.532+0000 7f960caef640 5 monclient: authenticate success, global_id 184343 2024-09-11T01:48:29.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.532+0000 7f960caef640 10 monclient: get_monmap_and_config success 2024-09-11T01:48:29.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.532+0000 7f960caef640 4 set_mon_vals no callback set 2024-09-11T01:48:29.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.532+0000 7f9605d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:29.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.532+0000 7f9605d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:48:29.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.532+0000 7f9605d9b640 10 monclient: dump: 2024-09-11T01:48:29.549 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:29.550 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:29.550 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.550 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.550 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:29.550 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.551 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:29.551 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:29.551 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:29.551 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.533+0000 7f960caef640 10 monclient: shutdown 2024-09-11T01:48:29.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.533+0000 7f960caef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:29.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.533+0000 7f960caef640 1 -- 172.21.15.12:0/3618183637 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9608138230 msgr2=0x7f9608138660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.533+0000 7f960caef640 1 --2- 172.21.15.12:0/3618183637 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9608138230 0x7f9608138660 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f95f0009870 tx=0x7f95f002a0a0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.533+0000 7f960caef640 1 -- 172.21.15.12:0/3618183637 shutdown_connections 2024-09-11T01:48:29.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.533+0000 7f960caef640 1 --2- 172.21.15.12:0/3618183637 >> v2:172.21.15.71:3300/0 conn(0x7f960813d380 0x7f960813f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.533+0000 7f960caef640 1 --2- 172.21.15.12:0/3618183637 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9608138230 0x7f9608138660 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.533+0000 7f960caef640 1 -- 172.21.15.12:0/3618183637 >> 172.21.15.12:0/3618183637 conn(0x7f96080a7020 msgr2=0x7f96080a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:29.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.533+0000 7f960caef640 1 -- 172.21.15.12:0/3618183637 shutdown_connections 2024-09-11T01:48:29.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.533+0000 7f960caef640 1 -- 172.21.15.12:0/3618183637 wait complete. 2024-09-11T01:48:29.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.534+0000 7f960caef640 10 monclient: build_initial_monmap 2024-09-11T01:48:29.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.534+0000 7f960caef640 10 monclient: monmap: 2024-09-11T01:48:29.553 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:29.553 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:48:29.553 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:48:29.553 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:48:29.553 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:29.554 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.554 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:48:29.554 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:48:29.554 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:48:29.554 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.534+0000 7f960caef640 1 Processor -- start 2024-09-11T01:48:29.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.534+0000 7f960caef640 1 -- start start 2024-09-11T01:48:29.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.534+0000 7f960caef640 10 monclient: init 2024-09-11T01:48:29.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f960caef640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:29.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f960caef640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:48:29.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f960caef640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9608137e50 0x7f96081c9e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f960caef640 10 monclient(hunting): picked mon.noname-b con 0x7f9608137e50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:48:29.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f960caef640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9608138230 0x7f96080a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f960caef640 10 monclient(hunting): picked mon.noname-a con 0x7f9608138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:48:29.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f960caef640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f960813d380 0x7f96081cd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f960caef640 10 monclient(hunting): picked mon.noname-c con 0x7f960813d380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f960caef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f960caef640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f960806f890 con 0x7f9608138230 2024-09-11T01:48:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f960caef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f960caef640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f960806f710 con 0x7f9608137e50 2024-09-11T01:48:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f960caef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f960caef640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f960813fcb0 con 0x7f960813d380 2024-09-11T01:48:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f960caef640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f960caef640 10 monclient(hunting): authenticate will time out at 1915.977260s 2024-09-11T01:48:29.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f95fffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9608138230 0x7f96080a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f95fffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9608138230 0x7f96080a42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36702/0 (socket says 172.21.15.12:36702) 2024-09-11T01:48:29.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f95fffff640 1 -- 172.21.15.12:0/1565340093 learned_addr learned my addr 172.21.15.12:0/1565340093 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:48:29.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f95fffff640 10 monclient(hunting): get_auth_request con 0x7f9608138230 auth_method 0 2024-09-11T01:48:29.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f95fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:29.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f95fffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:29.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f95fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:29.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f9606d9d640 1 --2- 172.21.15.12:0/1565340093 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f960813d380 0x7f96081cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f9606d9d640 10 monclient(hunting): get_auth_request con 0x7f960813d380 auth_method 0 2024-09-11T01:48:29.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f9606d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:29.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f9606d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f9606d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f95fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f95fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f95fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f960659c640 1 --2- 172.21.15.12:0/1565340093 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9608137e50 0x7f96081c9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f9606d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f9606d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f9606d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f960659c640 10 monclient(hunting): get_auth_request con 0x7f9608137e50 auth_method 0 2024-09-11T01:48:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f960659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f960659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f960659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:29.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f960659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:29.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f960659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:29.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f960659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:29.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f95fffff640 10 monclient(hunting): handle_auth_done global_id 214101 payload 824 2024-09-11T01:48:29.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f95fffff640 1 -- 172.21.15.12:0/1565340093 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f960813d380 msgr2=0x7f96081cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f95fffff640 1 --2- 172.21.15.12:0/1565340093 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f960813d380 0x7f96081cd390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f95fffff640 1 -- 172.21.15.12:0/1565340093 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9608137e50 msgr2=0x7f96081c9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f95fffff640 1 --2- 172.21.15.12:0/1565340093 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9608137e50 0x7f96081c9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f95fffff640 10 monclient: _finish_hunting 0 2024-09-11T01:48:29.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f95fffff640 1 monclient: found mon.noname-a 2024-09-11T01:48:29.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f95fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:29.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f95fffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:48:29.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.535+0000 7f95fffff640 1 -- 172.21.15.12:0/1565340093 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f960813b6d0 con 0x7f9608138230 2024-09-11T01:48:29.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.537+0000 7f95fffff640 1 -- 172.21.15.12:0/1565340093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9608138230 msgr2=0x7f96080a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:29.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.537+0000 7f95fffff640 1 --2- 172.21.15.12:0/1565340093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9608138230 0x7f96080a42c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f95f00090b0 tx=0x7f95f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:29.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.538+0000 7f95feffd640 1 -- 172.21.15.12:0/1565340093 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f0041020 con 0x7f9608138230 2024-09-11T01:48:29.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.538+0000 7f95feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:29.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.538+0000 7f95feffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:48:29.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.538+0000 7f95feffd640 10 monclient: dump: 2024-09-11T01:48:29.561 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:29.561 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:29.561 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.561 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.561 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:29.561 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.562 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:29.562 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:29.562 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:29.562 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.538+0000 7f95feffd640 10 monclient: _finish_auth 0 2024-09-11T01:48:29.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.538+0000 7f95feffd640 10 monclient: _check_auth_tickets 2024-09-11T01:48:29.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.538+0000 7f95feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:29.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.538+0000 7f95feffd640 1 -- 172.21.15.12:0/1565340093 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95f003d630 con 0x7f9608138230 2024-09-11T01:48:29.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.538+0000 7f95feffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:29.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.538+0000 7f95feffd640 1 -- 172.21.15.12:0/1565340093 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f0037040 con 0x7f9608138230 2024-09-11T01:48:29.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.538+0000 7f960caef640 5 monclient: authenticate success, global_id 214101 2024-09-11T01:48:29.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.538+0000 7f960caef640 10 monclient: _renew_subs 2024-09-11T01:48:29.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.538+0000 7f960caef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:29.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.538+0000 7f960caef640 1 -- 172.21.15.12:0/1565340093 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f960813faa0 con 0x7f9608138230 2024-09-11T01:48:29.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.538+0000 7f95feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:29.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.538+0000 7f95feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:48:29.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.538+0000 7f95feffd640 10 monclient: dump: 2024-09-11T01:48:29.564 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:29.564 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:29.564 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.564 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.564 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:29.564 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.564 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:29.564 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:29.564 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:29.565 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.538+0000 7f960559a640 4 set_mon_vals no callback set 2024-09-11T01:48:29.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.538+0000 7f960caef640 10 monclient: _renew_subs 2024-09-11T01:48:29.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.538+0000 7f960caef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:29.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.538+0000 7f960caef640 1 -- 172.21.15.12:0/1565340093 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96080a27d0 con 0x7f9608138230 2024-09-11T01:48:29.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.539+0000 7f95feffd640 1 -- 172.21.15.12:0/1565340093 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f95f003a030 con 0x7f9608138230 2024-09-11T01:48:29.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.539+0000 7f960caef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:29.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.539+0000 7f960caef640 1 --2- 172.21.15.12:0/1565340093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96080a4800 0x7f96081cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.539+0000 7f960caef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:29.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.539+0000 7f960caef640 1 -- 172.21.15.12:0/1565340093 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f960806a060 con 0x7f96080a4800 2024-09-11T01:48:29.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.539+0000 7f960659c640 1 --2- 172.21.15.12:0/1565340093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96080a4800 0x7f96081cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.539+0000 7f960659c640 10 monclient: get_auth_request con 0x7f96080a4800 auth_method 0 2024-09-11T01:48:29.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.539+0000 7f960659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:29.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.539+0000 7f960659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:29.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.539+0000 7f960659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:29.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.539+0000 7f960659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:29.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.539+0000 7f960659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:29.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.539+0000 7f960659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:29.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.539+0000 7f95feffd640 1 --2- 172.21.15.12:0/1565340093 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f95e803e260 0x7f95e8040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.539+0000 7f95feffd640 1 -- 172.21.15.12:0/1565340093 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(640..640 src has 507..640) ==== 4827+0+0 (secure 0 0 0) 0x7f95f0072ea0 con 0x7f9608138230 2024-09-11T01:48:29.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.539+0000 7f960659c640 10 monclient(hunting): handle_auth_done global_id 214104 payload 824 2024-09-11T01:48:29.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.539+0000 7f960659c640 1 -- 172.21.15.12:0/1565340093 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9608138660 con 0x7f96080a4800 2024-09-11T01:48:29.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.539+0000 7f9606d9d640 1 --2- 172.21.15.12:0/1565340093 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f95e803e260 0x7f95e8040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.539+0000 7f9606d9d640 10 monclient: get_auth_request con 0x7f95e803e260 auth_method 0 2024-09-11T01:48:29.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.541+0000 7f960659c640 1 -- 172.21.15.12:0/1565340093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96080a4800 msgr2=0x7f96081cd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:29.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.541+0000 7f960659c640 1 --2- 172.21.15.12:0/1565340093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96080a4800 0x7f96081cd8d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f95f800e770 tx=0x7f95f800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:29.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.541+0000 7f95feffd640 1 -- 172.21.15.12:0/1565340093 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f8017070 con 0x7f96080a4800 2024-09-11T01:48:29.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.541+0000 7f95feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:29.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.541+0000 7f95feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:48:29.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.541+0000 7f95feffd640 10 monclient: dump: 2024-09-11T01:48:29.571 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:29.571 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:29.571 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.571 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.571 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:29.571 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.571 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:29.571 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:29.571 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:29.571 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.542+0000 7f95feffd640 1 -- 172.21.15.12:0/1565340093 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f95f800c9b0 con 0x7f96080a4800 2024-09-11T01:48:29.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.542+0000 7f95feffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:29.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.542+0000 7f95feffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:48:29.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.542+0000 7f95feffd640 1 -- 172.21.15.12:0/1565340093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96080a4800 msgr2=0x7f96081cd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.542+0000 7f95feffd640 1 --2- 172.21.15.12:0/1565340093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96080a4800 0x7f96081cd8d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f95f800e770 tx=0x7f95f800ec30 comp rx=0 tx=0).stop 2024-09-11T01:48:29.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.542+0000 7f95feffd640 1 -- 172.21.15.12:0/1565340093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96080a4800 msgr2=0x7f96081cd8d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:48:29.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.542+0000 7f95feffd640 1 --2- 172.21.15.12:0/1565340093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96080a4800 0x7f96081cd8d0 secure :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f95f800e770 tx=0x7f95f800ec30 comp rx=0 tx=0).stop 2024-09-11T01:48:29.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.542+0000 7f9606d9d640 1 --2- 172.21.15.12:0/1565340093 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f95e803e260 0x7f95e8040720 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f9600003aa0 tx=0x7f9600009820 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:29.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.548+0000 7f960caef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:48:29.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.548+0000 7f960caef640 1 --2- 172.21.15.12:0/1565340093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96080a1600 0x7f96081cde10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.548+0000 7f960caef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:29.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.548+0000 7f960caef640 1 -- 172.21.15.12:0/1565340093 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f96080a2bb0 con 0x7f96080a1600 2024-09-11T01:48:29.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.548+0000 7f960659c640 1 --2- 172.21.15.12:0/1565340093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96080a1600 0x7f96081cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.548+0000 7f960659c640 10 monclient: get_auth_request con 0x7f96080a1600 auth_method 0 2024-09-11T01:48:29.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.548+0000 7f960659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:29.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.548+0000 7f960659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:29.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.548+0000 7f960659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:29.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.548+0000 7f960659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:29.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.548+0000 7f960659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:29.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.548+0000 7f960659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:29.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.548+0000 7f960659c640 10 monclient(hunting): handle_auth_done global_id 214107 payload 824 2024-09-11T01:48:29.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.548+0000 7f960659c640 1 -- 172.21.15.12:0/1565340093 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9608069950 con 0x7f96080a1600 2024-09-11T01:48:29.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.550+0000 7f960659c640 1 -- 172.21.15.12:0/1565340093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96080a1600 msgr2=0x7f96081cde10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:29.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.550+0000 7f960659c640 1 --2- 172.21.15.12:0/1565340093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96080a1600 0x7f96081cde10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f95f8019000 tx=0x7f95f80046f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:29.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.551+0000 7f95feffd640 1 -- 172.21.15.12:0/1565340093 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f8017070 con 0x7f96080a1600 2024-09-11T01:48:29.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.551+0000 7f95feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:29.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.551+0000 7f95feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:48:29.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.551+0000 7f95feffd640 10 monclient: dump: 2024-09-11T01:48:29.574 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:29.574 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:29.574 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.574 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.574 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:29.574 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.574 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:29.575 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:29.575 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:29.575 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.551+0000 7f95feffd640 1 -- 172.21.15.12:0/1565340093 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7f95f8006540 con 0x7f96080a1600 2024-09-11T01:48:29.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.551+0000 7f95feffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:48:29.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.551+0000 7f95feffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:48:29.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.551+0000 7f95feffd640 1 -- 172.21.15.12:0/1565340093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96080a1600 msgr2=0x7f96081cde10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.551+0000 7f95feffd640 1 --2- 172.21.15.12:0/1565340093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96080a1600 0x7f96081cde10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f95f8019000 tx=0x7f95f80046f0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.551+0000 7f95feffd640 1 -- 172.21.15.12:0/1565340093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96080a1600 msgr2=0x7f96081cde10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:48:29.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.551+0000 7f95feffd640 1 --2- 172.21.15.12:0/1565340093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f96080a1600 0x7f96081cde10 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f95f8019000 tx=0x7f95f80046f0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.576 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:48:29.576 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:48:29.576 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:48:29.576 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:48:29.576 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 182, 2024-09-11T01:48:29.576 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 15751, 2024-09-11T01:48:29.576 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:48:29.576 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:48:29.576 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:48:29.576 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:48:29.577 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:29.577 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:48:29.577 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:48:29.577 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:48:29.577 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:48:29.577 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:48:29.577 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:48:29.577 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:48:29.577 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:48:29.577 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:48:29.577 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:48:29.577 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:48:29.578 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:48:29.578 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:48:29.578 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:48:29.578 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:48:29.578 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:29.578 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:48:29.578 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:48:29.578 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:48:29.578 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:48:29.578 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:48:29.578 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:48:29.578 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:48:29.579 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:48:29.579 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:48:29.579 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:48:29.579 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:48:29.579 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:48:29.579 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:48:29.579 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:29.579 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.580 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:48:29.580 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:48:29.580 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:48:29.580 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:48:29.580 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:48:29.580 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:48:29.580 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:48:29.580 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:48:29.580 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:48:29.580 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:48:29.581 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:48:29.581 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:48:29.581 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:48:29.581 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:48:29.581 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:48:29.581 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:48:29.581 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:48:29.581 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:48:29.581 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:48:29.581 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:48:29.581 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:48:29.582 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:48:29.582 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:48:29.582 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:48:29.582 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:48:29.582 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:48:29.582 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:48:29.582 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:48:29.582 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:29.582 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:48:29.583 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.583 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:48:29.583 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.583 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:48:29.583 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:48:29.583 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:48:29.583 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:48:29.583 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.583 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:48:29.583 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:48:29.584 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:29.584 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.584 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.584 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:48:29.584 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:48:29.585 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:29.585 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:29.585 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:29.585 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.585 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:48:29.585 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:48:29.585 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:48:29.585 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:48:29.585 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:48:29.585 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.586 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.586 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:48:29.586 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:48:29.586 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:48:29.586 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:48:29.586 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.586 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:48:29.587 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:48:29.587 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:29.587 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.587 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.587 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:48:29.587 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:48:29.587 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:29.587 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:29.587 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:29.588 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.588 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:48:29.588 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:48:29.588 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:48:29.588 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:48:29.588 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:48:29.589 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.589 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.589 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:48:29.589 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:48:29.589 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:48:29.589 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:48:29.589 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.589 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:48:29.589 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:48:29.590 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:29.590 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.590 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.590 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:48:29.590 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:48:29.590 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:29.590 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:29.590 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:29.591 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.591 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:48:29.591 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:48:29.591 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:48:29.591 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:48:29.591 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:48:29.591 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:29.592 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:29.592 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.592 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:48:29.592 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:48:29.592 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.592 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:48:29.593 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:48:29.593 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:48:29.593 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:29.593 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:29.593 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:48:29.593 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.593 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:48:29.594 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:48:29.594 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:48:29.594 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:29.594 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:29.594 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.594 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:48:29.594 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:48:29.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.551+0000 7f960caef640 1 -- 172.21.15.12:0/1565340093 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f95e803e260 msgr2=0x7f95e8040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.551+0000 7f960caef640 1 --2- 172.21.15.12:0/1565340093 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f95e803e260 0x7f95e8040720 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f9600003aa0 tx=0x7f9600009820 comp rx=0 tx=0).stop 2024-09-11T01:48:29.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.551+0000 7f960caef640 10 monclient: shutdown 2024-09-11T01:48:29.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.551+0000 7f960caef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:29.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.551+0000 7f960caef640 1 -- 172.21.15.12:0/1565340093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9608138230 msgr2=0x7f96080a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.551+0000 7f960caef640 1 --2- 172.21.15.12:0/1565340093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9608138230 0x7f96080a42c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f95f00090b0 tx=0x7f95f00046c0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.551+0000 7f9606d9d640 1 -- 172.21.15.12:0/1565340093 reap_dead start 2024-09-11T01:48:29.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.551+0000 7f9606d9d640 1 -- 172.21.15.12:0/1565340093 reap_dead start 2024-09-11T01:48:29.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.551+0000 7f960caef640 1 -- 172.21.15.12:0/1565340093 shutdown_connections 2024-09-11T01:48:29.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.551+0000 7f960caef640 1 -- 172.21.15.12:0/1565340093 >> 172.21.15.12:0/1565340093 conn(0x7f96080a7020 msgr2=0x7f9608136ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:29.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.551+0000 7f960caef640 1 -- 172.21.15.12:0/1565340093 shutdown_connections 2024-09-11T01:48:29.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.551+0000 7f960caef640 1 -- 172.21.15.12:0/1565340093 wait complete. 2024-09-11T01:48:29.598 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:48:29.598 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:48:29.598 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:48:29.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.698+0000 7fb0d8e85640 10 monclient: get_monmap_and_config 2024-09-11T01:48:29.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.698+0000 7fb0d8e85640 10 monclient: build_initial_monmap 2024-09-11T01:48:29.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.698+0000 7fb0d8e85640 10 monclient: monmap: 2024-09-11T01:48:29.700 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:29.700 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:29.701 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:48:29.699220+0000 2024-09-11T01:48:29.701 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:48:29.699220+0000 2024-09-11T01:48:29.701 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:29.701 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.701 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:48:29.701 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:48:29.701 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:48:29.701 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:48:29.701 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:48:29.701 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.699+0000 7fb0d8e85640 1 Processor -- start 2024-09-11T01:48:29.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.699+0000 7fb0d8e85640 1 -- start start 2024-09-11T01:48:29.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.699+0000 7fb0d8e85640 10 monclient: init 2024-09-11T01:48:29.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.699+0000 7fb0d8e85640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:29.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.699+0000 7fb0d8e85640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-11T01:48:29.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.699+0000 7fb0d8e85640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb0d4135f70 addr v1:172.21.15.71:6789/0 2024-09-11T01:48:29.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.699+0000 7fb0d8e85640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb0d409e020 0x7fb0d413ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.699+0000 7fb0d8e85640 10 monclient(hunting): picked mon.noname-a con 0x7fb0d409e020 addr v2:172.21.15.12:3300/0 2024-09-11T01:48:29.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.699+0000 7fb0d8e85640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb0d413b320 addr v1:172.21.15.12:6789/0 2024-09-11T01:48:29.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.699+0000 7fb0d8e85640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0d409da70 con 0x7fb0d413b320 2024-09-11T01:48:29.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.699+0000 7fb0d8e85640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0d409dde0 con 0x7fb0d4135f70 2024-09-11T01:48:29.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.699+0000 7fb0d8e85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:29.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.699+0000 7fb0d8e85640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fb0d406cc50 con 0x7fb0d409e020 2024-09-11T01:48:29.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.699+0000 7fb0d8e85640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:29.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.699+0000 7fb0d8e85640 10 monclient(hunting): authenticate will time out at 1916.141810s 2024-09-11T01:48:29.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.699+0000 7fb0cbfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb0d409e020 0x7fb0d413ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.699+0000 7fb0cbfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb0d409e020 0x7fb0d413ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36732/0 (socket says 172.21.15.12:36732) 2024-09-11T01:48:29.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.699+0000 7fb0d2d9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb0d413b320 0x7fb0d413d710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46184/0 (socket says 172.21.15.12:46184) 2024-09-11T01:48:29.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.699+0000 7fb0cbfff640 1 -- 172.21.15.12:0/3339813837 learned_addr learned my addr 172.21.15.12:0/3339813837 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:48:29.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.699+0000 7fb0cbfff640 10 monclient(hunting): get_auth_request con 0x7fb0d409e020 auth_method 0 2024-09-11T01:48:29.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.699+0000 7fb0cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:29.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.699+0000 7fb0cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:29.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.699+0000 7fb0cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:29.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.700+0000 7fb0cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:29.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.700+0000 7fb0cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:29.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.700+0000 7fb0cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:29.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.700+0000 7fb0cbfff640 10 monclient(hunting): handle_auth_done global_id 214110 payload 274 2024-09-11T01:48:29.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.700+0000 7fb0cbfff640 1 -- 172.21.15.12:0/3339813837 >> v1:172.21.15.71:6789/0 conn(0x7fb0d4135f70 legacy=0x7fb0d4136350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:48:29.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.700+0000 7fb0cbfff640 1 -- 172.21.15.12:0/3339813837 >> v1:172.21.15.12:6789/0 conn(0x7fb0d413b320 legacy=0x7fb0d413d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:48:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.700+0000 7fb0cbfff640 10 monclient: _finish_hunting 0 2024-09-11T01:48:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.700+0000 7fb0cbfff640 1 monclient: found mon.noname-a 2024-09-11T01:48:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.700+0000 7fb0cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.700+0000 7fb0cbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:48:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.700+0000 7fb0cbfff640 1 -- 172.21.15.12:0/3339813837 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d406d3b0 con 0x7fb0d409e020 2024-09-11T01:48:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0cbfff640 1 -- 172.21.15.12:0/3339813837 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0d409e020 msgr2=0x7fb0d413ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0cbfff640 1 --2- 172.21.15.12:0/3339813837 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0d409e020 0x7fb0d413ade0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc009870 tx=0x7fb0bc02a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e040b197eda04c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0d1d9b640 1 -- 172.21.15.12:0/3339813837 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc037070 con 0x7fb0d409e020 2024-09-11T01:48:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0d1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0d1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:48:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0d1d9b640 10 monclient: dump: 2024-09-11T01:48:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:29.716 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0d1d9b640 10 monclient: _finish_auth 0 2024-09-11T01:48:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0d1d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:48:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0d1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0d1d9b640 1 -- 172.21.15.12:0/3339813837 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0bc032640 con 0x7fb0d409e020 2024-09-11T01:48:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0d8e85640 5 monclient: authenticate success, global_id 214110 2024-09-11T01:48:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0d8e85640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:48:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0d1d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0d8e85640 10 monclient: get_monmap_and_config success 2024-09-11T01:48:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0d1d9b640 1 -- 172.21.15.12:0/3339813837 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc032a10 con 0x7fb0d409e020 2024-09-11T01:48:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0d8e85640 4 set_mon_vals no callback set 2024-09-11T01:48:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0d1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0d1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:48:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0d1d9b640 10 monclient: dump: 2024-09-11T01:48:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:29.718 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0d8e85640 10 monclient: shutdown 2024-09-11T01:48:29.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0d8e85640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:29.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0d8e85640 1 -- 172.21.15.12:0/3339813837 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0d409e020 msgr2=0x7fb0d413ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0d8e85640 1 --2- 172.21.15.12:0/3339813837 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0d409e020 0x7fb0d413ade0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc009870 tx=0x7fb0bc02a0a0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0d8e85640 1 -- 172.21.15.12:0/3339813837 shutdown_connections 2024-09-11T01:48:29.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0d8e85640 1 --2- 172.21.15.12:0/3339813837 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0d409e020 0x7fb0d413ade0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0d8e85640 1 -- 172.21.15.12:0/3339813837 >> 172.21.15.12:0/3339813837 conn(0x7fb0d40a3e30 msgr2=0x7fb0d40a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:29.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0d8e85640 1 -- 172.21.15.12:0/3339813837 shutdown_connections 2024-09-11T01:48:29.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.703+0000 7fb0d8e85640 1 -- 172.21.15.12:0/3339813837 wait complete. 2024-09-11T01:48:29.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.704+0000 7fb0d8e85640 10 monclient: build_initial_monmap 2024-09-11T01:48:29.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.704+0000 7fb0d8e85640 10 monclient: monmap: 2024-09-11T01:48:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:48:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:48:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:48:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:48:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:48:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:48:29.720 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.704+0000 7fb0d8e85640 1 Processor -- start 2024-09-11T01:48:29.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.704+0000 7fb0d8e85640 1 -- start start 2024-09-11T01:48:29.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.704+0000 7fb0d8e85640 10 monclient: init 2024-09-11T01:48:29.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d8e85640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:29.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d8e85640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:48:29.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d8e85640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0d409e020 0x7fb0d4096640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d8e85640 10 monclient(hunting): picked mon.noname-a con 0x7fb0d409e020 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:48:29.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d8e85640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0d4135f70 0x7fb0d409b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d8e85640 10 monclient(hunting): picked mon.noname-b con 0x7fb0d4135f70 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:48:29.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d8e85640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb0d413b320 0x7fb0d409b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d8e85640 10 monclient(hunting): picked mon.noname-c con 0x7fb0d413b320 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d8e85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d8e85640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d413ee30 con 0x7fb0d409e020 2024-09-11T01:48:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d8e85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d8e85640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d409f020 con 0x7fb0d4135f70 2024-09-11T01:48:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d8e85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:29.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d8e85640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb0d413efb0 con 0x7fb0d413b320 2024-09-11T01:48:29.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d8e85640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:29.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d8e85640 10 monclient(hunting): authenticate will time out at 1916.147464s 2024-09-11T01:48:29.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0d409e020 0x7fb0d4096640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0d409e020 0x7fb0d4096640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36746/0 (socket says 172.21.15.12:36746) 2024-09-11T01:48:29.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d259c640 1 -- 172.21.15.12:0/238369008 learned_addr learned my addr 172.21.15.12:0/238369008 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:48:29.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d259c640 10 monclient(hunting): get_auth_request con 0x7fb0d409e020 auth_method 0 2024-09-11T01:48:29.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:29.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:29.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:29.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d2d9d640 1 --2- 172.21.15.12:0/238369008 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb0d413b320 0x7fb0d409b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d2d9d640 10 monclient(hunting): get_auth_request con 0x7fb0d413b320 auth_method 0 2024-09-11T01:48:29.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:29.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:29.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:29.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0cbfff640 1 --2- 172.21.15.12:0/238369008 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0d4135f70 0x7fb0d409b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0cbfff640 10 monclient(hunting): get_auth_request con 0x7fb0d4135f70 auth_method 0 2024-09-11T01:48:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d259c640 10 monclient(hunting): handle_auth_done global_id 214113 payload 824 2024-09-11T01:48:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d259c640 1 -- 172.21.15.12:0/238369008 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb0d413b320 msgr2=0x7fb0d409b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d259c640 1 --2- 172.21.15.12:0/238369008 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb0d413b320 0x7fb0d409b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d259c640 1 -- 172.21.15.12:0/238369008 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0d4135f70 msgr2=0x7fb0d409b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d259c640 1 --2- 172.21.15.12:0/238369008 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0d4135f70 0x7fb0d409b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d259c640 10 monclient: _finish_hunting 0 2024-09-11T01:48:29.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d259c640 1 monclient: found mon.noname-a 2024-09-11T01:48:29.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:29.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:48:29.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.705+0000 7fb0d259c640 1 -- 172.21.15.12:0/238369008 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d409e4e0 con 0x7fb0d409e020 2024-09-11T01:48:29.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.708+0000 7fb0d259c640 1 -- 172.21.15.12:0/238369008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0d409e020 msgr2=0x7fb0d4096640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:29.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.708+0000 7fb0d259c640 1 --2- 172.21.15.12:0/238369008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0d409e020 0x7fb0d4096640 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb0c400c490 tx=0x7fb0c400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:29.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.708+0000 7fb0caffd640 1 -- 172.21.15.12:0/238369008 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c4016020 con 0x7fb0d409e020 2024-09-11T01:48:29.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.708+0000 7fb0caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:29.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.708+0000 7fb0caffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:48:29.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.708+0000 7fb0caffd640 10 monclient: dump: 2024-09-11T01:48:29.728 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:29.728 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:29.728 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.728 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.728 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:29.728 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.728 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:29.728 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:29.729 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:29.729 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.708+0000 7fb0caffd640 10 monclient: _finish_auth 0 2024-09-11T01:48:29.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.708+0000 7fb0caffd640 10 monclient: _check_auth_tickets 2024-09-11T01:48:29.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.708+0000 7fb0caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:29.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.708+0000 7fb0caffd640 1 -- 172.21.15.12:0/238369008 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0c4012a40 con 0x7fb0d409e020 2024-09-11T01:48:29.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.708+0000 7fb0d8e85640 5 monclient: authenticate success, global_id 214113 2024-09-11T01:48:29.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.708+0000 7fb0d8e85640 10 monclient: _renew_subs 2024-09-11T01:48:29.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.708+0000 7fb0d8e85640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:29.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.708+0000 7fb0d8e85640 1 -- 172.21.15.12:0/238369008 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0d413ea50 con 0x7fb0d409e020 2024-09-11T01:48:29.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.708+0000 7fb0caffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:29.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.708+0000 7fb0caffd640 1 -- 172.21.15.12:0/238369008 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c4012e30 con 0x7fb0d409e020 2024-09-11T01:48:29.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.708+0000 7fb0caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:29.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.708+0000 7fb0caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:48:29.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.708+0000 7fb0caffd640 10 monclient: dump: 2024-09-11T01:48:29.731 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:29.731 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:29.731 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.731 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.731 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:29.732 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.732 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:29.732 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:29.732 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:29.732 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.708+0000 7fb0d159a640 4 set_mon_vals no callback set 2024-09-11T01:48:29.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.708+0000 7fb0d8e85640 10 monclient: _renew_subs 2024-09-11T01:48:29.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.708+0000 7fb0d8e85640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:29.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.708+0000 7fb0d8e85640 1 -- 172.21.15.12:0/238369008 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0d4097300 con 0x7fb0d409e020 2024-09-11T01:48:29.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.709+0000 7fb0d8e85640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:29.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.709+0000 7fb0d8e85640 1 --2- 172.21.15.12:0/238369008 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb0d4096b80 0x7fb0d409be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.709+0000 7fb0d8e85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:29.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.709+0000 7fb0d8e85640 1 -- 172.21.15.12:0/238369008 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb0d4136350 con 0x7fb0d4096b80 2024-09-11T01:48:29.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.709+0000 7fb0caffd640 1 -- 172.21.15.12:0/238369008 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fb0c401b400 con 0x7fb0d409e020 2024-09-11T01:48:29.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.709+0000 7fb0cbfff640 1 --2- 172.21.15.12:0/238369008 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb0d4096b80 0x7fb0d409be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.709+0000 7fb0cbfff640 10 monclient: get_auth_request con 0x7fb0d4096b80 auth_method 0 2024-09-11T01:48:29.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.709+0000 7fb0cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:29.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.709+0000 7fb0cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:29.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.709+0000 7fb0cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:29.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.709+0000 7fb0cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:29.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.709+0000 7fb0cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:29.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.709+0000 7fb0cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.709+0000 7fb0caffd640 1 --2- 172.21.15.12:0/238369008 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb0b403e210 0x7fb0b40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.709+0000 7fb0cbfff640 10 monclient(hunting): handle_auth_done global_id 184352 payload 824 2024-09-11T01:48:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.709+0000 7fb0cbfff640 1 -- 172.21.15.12:0/238369008 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb0d409c660 con 0x7fb0d4096b80 2024-09-11T01:48:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.709+0000 7fb0caffd640 1 -- 172.21.15.12:0/238369008 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(640..640 src has 507..640) ==== 4827+0+0 (secure 0 0 0) 0x7fb0c4052d80 con 0x7fb0d409e020 2024-09-11T01:48:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.709+0000 7fb0d2d9d640 1 --2- 172.21.15.12:0/238369008 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb0b403e210 0x7fb0b40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.709+0000 7fb0d2d9d640 10 monclient: get_auth_request con 0x7fb0b403e210 auth_method 0 2024-09-11T01:48:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.711+0000 7fb0cbfff640 1 -- 172.21.15.12:0/238369008 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb0d4096b80 msgr2=0x7fb0d409be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.711+0000 7fb0cbfff640 1 --2- 172.21.15.12:0/238369008 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb0d4096b80 0x7fb0d409be20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc034000 tx=0x7fb0bc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.711+0000 7fb0caffd640 1 -- 172.21.15.12:0/238369008 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc037040 con 0x7fb0d4096b80 2024-09-11T01:48:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.711+0000 7fb0caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.711+0000 7fb0caffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:48:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.711+0000 7fb0caffd640 10 monclient: dump: 2024-09-11T01:48:29.739 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:29.739 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:29.739 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.739 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.739 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:29.739 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.739 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:29.739 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:29.739 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:29.739 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.712+0000 7fb0d2d9d640 1 --2- 172.21.15.12:0/238369008 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb0b403e210 0x7fb0b40406d0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4098070 tx=0x7fb0cc004ee0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:29.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.712+0000 7fb0caffd640 1 -- 172.21.15.12:0/238369008 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb0bc03c820 con 0x7fb0d4096b80 2024-09-11T01:48:29.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.712+0000 7fb0caffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:29.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.712+0000 7fb0caffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:48:29.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.712+0000 7fb0caffd640 1 -- 172.21.15.12:0/238369008 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb0d4096b80 msgr2=0x7fb0d409be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.712+0000 7fb0caffd640 1 --2- 172.21.15.12:0/238369008 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb0d4096b80 0x7fb0d409be20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc034000 tx=0x7fb0bc0046c0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.712+0000 7fb0caffd640 1 -- 172.21.15.12:0/238369008 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb0d4096b80 msgr2=0x7fb0d409be20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:48:29.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.712+0000 7fb0caffd640 1 --2- 172.21.15.12:0/238369008 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb0d4096b80 0x7fb0d409be20 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc034000 tx=0x7fb0bc0046c0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.718+0000 7fb0d8e85640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:48:29.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.718+0000 7fb0d8e85640 1 --2- 172.21.15.12:0/238369008 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb0d4131540 0x7fb0d4131950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.718+0000 7fb0d8e85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.718+0000 7fb0d8e85640 1 -- 172.21.15.12:0/238369008 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb0d406e520 con 0x7fb0d4131540 2024-09-11T01:48:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.718+0000 7fb0cbfff640 1 --2- 172.21.15.12:0/238369008 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb0d4131540 0x7fb0d4131950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.718+0000 7fb0cbfff640 10 monclient: get_auth_request con 0x7fb0d4131540 auth_method 0 2024-09-11T01:48:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.718+0000 7fb0cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.718+0000 7fb0cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.718+0000 7fb0cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.718+0000 7fb0cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.718+0000 7fb0cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.718+0000 7fb0cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.718+0000 7fb0cbfff640 10 monclient(hunting): handle_auth_done global_id 184355 payload 824 2024-09-11T01:48:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.718+0000 7fb0cbfff640 1 -- 172.21.15.12:0/238369008 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb0d413d790 con 0x7fb0d4131540 2024-09-11T01:48:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.720+0000 7fb0cbfff640 1 -- 172.21.15.12:0/238369008 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb0d4131540 msgr2=0x7fb0d4131950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.720+0000 7fb0cbfff640 1 --2- 172.21.15.12:0/238369008 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb0d4131540 0x7fb0d4131950 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb0d41364d0 tx=0x7fb0bc008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.721+0000 7fb0caffd640 1 -- 172.21.15.12:0/238369008 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc037040 con 0x7fb0d4131540 2024-09-11T01:48:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.721+0000 7fb0caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.721+0000 7fb0caffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:48:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.721+0000 7fb0caffd640 10 monclient: dump: 2024-09-11T01:48:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:29.744 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:29.744 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.721+0000 7fb0caffd640 1 -- 172.21.15.12:0/238369008 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4542 (secure 0 0 0) 0x7fb0bc0044d0 con 0x7fb0d4131540 2024-09-11T01:48:29.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.721+0000 7fb0caffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:48:29.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.721+0000 7fb0caffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:48:29.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.721+0000 7fb0caffd640 1 -- 172.21.15.12:0/238369008 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb0d4131540 msgr2=0x7fb0d4131950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.721+0000 7fb0caffd640 1 --2- 172.21.15.12:0/238369008 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb0d4131540 0x7fb0d4131950 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb0d41364d0 tx=0x7fb0bc008030 comp rx=0 tx=0).stop 2024-09-11T01:48:29.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.721+0000 7fb0caffd640 1 -- 172.21.15.12:0/238369008 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb0d4131540 msgr2=0x7fb0d4131950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:48:29.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.721+0000 7fb0caffd640 1 --2- 172.21.15.12:0/238369008 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb0d4131540 0x7fb0d4131950 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb0d41364d0 tx=0x7fb0bc008030 comp rx=0 tx=0).stop 2024-09-11T01:48:29.745 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:48:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:48:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:48:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:48:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 182, 2024-09-11T01:48:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 170315, 2024-09-11T01:48:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:48:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:48:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:48:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:48:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:48:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:48:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:48:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:48:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:48:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:48:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:48:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:48:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:48:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:48:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:48:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:48:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:48:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:48:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:48:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:48:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:48:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:48:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:48:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:48:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:48:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:48:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:48:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:48:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:48:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:48:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:48:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:48:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:48:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:48:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:48:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:48:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:48:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:48:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:48:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:48:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:48:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:48:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:48:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:48:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:48:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:48:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:48:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:48:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:48:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:48:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:48:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:48:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:48:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:48:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:48:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:48:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:48:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:48:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:48:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:48:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:48:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:48:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:48:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:48:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:48:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:48:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:48:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:48:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:48:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:48:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:48:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:48:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:48:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:48:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:48:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:48:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:48:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:48:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:48:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:48:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:48:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:48:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:48:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:29.757 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.757 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:48:29.757 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:48:29.757 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:48:29.757 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:48:29.757 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:48:29.757 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.757 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.757 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:48:29.757 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:48:29.757 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:48:29.758 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:48:29.758 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.758 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:48:29.758 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:48:29.758 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:29.758 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.758 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.758 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:48:29.758 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:48:29.758 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:29.758 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:29.758 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:29.759 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.759 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:48:29.759 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:48:29.759 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:48:29.759 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:48:29.759 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:48:29.759 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:29.760 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:29.760 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.760 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:48:29.760 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:48:29.760 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.760 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:48:29.760 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:48:29.760 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:48:29.760 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:29.760 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:29.760 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:48:29.760 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.761 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:48:29.761 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:48:29.761 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:48:29.761 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:29.761 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:29.761 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.761 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:48:29.761 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:48:29.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.721+0000 7fb0d8e85640 1 -- 172.21.15.12:0/238369008 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb0b403e210 msgr2=0x7fb0b40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.721+0000 7fb0d8e85640 1 --2- 172.21.15.12:0/238369008 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb0b403e210 0x7fb0b40406d0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4098070 tx=0x7fb0cc004ee0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.721+0000 7fb0d8e85640 10 monclient: shutdown 2024-09-11T01:48:29.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.721+0000 7fb0d8e85640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:29.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.721+0000 7fb0d8e85640 1 -- 172.21.15.12:0/238369008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0d409e020 msgr2=0x7fb0d4096640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.721+0000 7fb0d8e85640 1 --2- 172.21.15.12:0/238369008 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb0d409e020 0x7fb0d4096640 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb0c400c490 tx=0x7fb0c400c950 comp rx=0 tx=0).stop 2024-09-11T01:48:29.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.721+0000 7fb0d2d9d640 1 -- 172.21.15.12:0/238369008 reap_dead start 2024-09-11T01:48:29.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.721+0000 7fb0d2d9d640 1 -- 172.21.15.12:0/238369008 reap_dead start 2024-09-11T01:48:29.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.721+0000 7fb0d8e85640 1 -- 172.21.15.12:0/238369008 shutdown_connections 2024-09-11T01:48:29.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.721+0000 7fb0d8e85640 1 -- 172.21.15.12:0/238369008 >> 172.21.15.12:0/238369008 conn(0x7fb0d40a3e30 msgr2=0x7fb0d409f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:29.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.721+0000 7fb0d8e85640 1 -- 172.21.15.12:0/238369008 shutdown_connections 2024-09-11T01:48:29.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.721+0000 7fb0d8e85640 1 -- 172.21.15.12:0/238369008 wait complete. 2024-09-11T01:48:29.766 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:48:29.766 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:48:29.766 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:48:29.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.866+0000 7f436d952640 10 monclient: get_monmap_and_config 2024-09-11T01:48:29.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.866+0000 7f436d952640 10 monclient: build_initial_monmap 2024-09-11T01:48:29.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.866+0000 7f436d952640 10 monclient: monmap: 2024-09-11T01:48:29.869 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:29.869 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:29.869 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:48:29.867601+0000 2024-09-11T01:48:29.869 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:48:29.867601+0000 2024-09-11T01:48:29.869 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:29.869 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.870 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:48:29.870 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:48:29.870 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:48:29.870 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:48:29.870 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:48:29.870 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.867+0000 7f436d952640 1 Processor -- start 2024-09-11T01:48:29.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.867+0000 7f436d952640 1 -- start start 2024-09-11T01:48:29.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.867+0000 7f436d952640 10 monclient: init 2024-09-11T01:48:29.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f436d952640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:29.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f436d952640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-11T01:48:29.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f436d952640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f43680a2e20 addr v1:172.21.15.71:6789/0 2024-09-11T01:48:29.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f436d952640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f43680a4970 0x7f436806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f436d952640 10 monclient(hunting): picked mon.noname-a con 0x7f43680a4970 addr v2:172.21.15.12:3300/0 2024-09-11T01:48:29.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f436d952640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f436806f910 addr v1:172.21.15.12:6789/0 2024-09-11T01:48:29.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f436d952640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f436806f4f0 con 0x7f436806f910 2024-09-11T01:48:29.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f436d952640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43680a3910 con 0x7f43680a2e20 2024-09-11T01:48:29.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f436d952640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:29.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f436d952640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f436806ef30 con 0x7f43680a4970 2024-09-11T01:48:29.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f436d952640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:29.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f436d952640 10 monclient(hunting): authenticate will time out at 1916.310170s 2024-09-11T01:48:29.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f435ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f43680a4970 0x7f436806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f43677fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f436806f910 0x7f436813f430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46186/0 (socket says 172.21.15.12:46186) 2024-09-11T01:48:29.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f43677fe640 1 -- 172.21.15.12:0/3345379442 learned_addr learned my addr 172.21.15.12:0/3345379442 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:48:29.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f435ffff640 10 monclient(hunting): get_auth_request con 0x7f43680a4970 auth_method 0 2024-09-11T01:48:29.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f435ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:29.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f435ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:29.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f435ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:29.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f435ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:29.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f435ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:29.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f435ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:29.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f435ffff640 10 monclient(hunting): handle_auth_done global_id 214116 payload 274 2024-09-11T01:48:29.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f435ffff640 1 -- 172.21.15.12:0/3345379442 >> v1:172.21.15.71:6789/0 conn(0x7f43680a2e20 legacy=0x7f43680a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:48:29.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f435ffff640 1 -- 172.21.15.12:0/3345379442 >> v1:172.21.15.12:6789/0 conn(0x7f436806f910 legacy=0x7f436813f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:48:29.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f435ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:48:29.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f435ffff640 1 monclient: found mon.noname-a 2024-09-11T01:48:29.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f435ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:29.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f435ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:48:29.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.868+0000 7f435ffff640 1 -- 172.21.15.12:0/3345379442 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43680a3ae0 con 0x7f43680a4970 2024-09-11T01:48:29.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.871+0000 7f435ffff640 1 -- 172.21.15.12:0/3345379442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43680a4970 msgr2=0x7f436806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:29.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.871+0000 7f435ffff640 1 --2- 172.21.15.12:0/3345379442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43680a4970 0x7f436806ded0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f43500097a0 tx=0x7f4350029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bff9aa64405f3f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:29.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.871+0000 7f43667fc640 1 -- 172.21.15.12:0/3345379442 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4350037070 con 0x7f43680a4970 2024-09-11T01:48:29.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.871+0000 7f43667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:29.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.871+0000 7f43667fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:48:29.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.871+0000 7f43667fc640 10 monclient: dump: 2024-09-11T01:48:29.884 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:29.884 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:29.885 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.885 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.886 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:29.886 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.886 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:29.886 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:29.886 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:29.886 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.871+0000 7f43667fc640 10 monclient: _finish_auth 0 2024-09-11T01:48:29.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.871+0000 7f43667fc640 10 monclient: _check_auth_tickets 2024-09-11T01:48:29.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.871+0000 7f43667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:29.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.871+0000 7f43667fc640 1 -- 172.21.15.12:0/3345379442 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4350032640 con 0x7f43680a4970 2024-09-11T01:48:29.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.871+0000 7f436d952640 5 monclient: authenticate success, global_id 214116 2024-09-11T01:48:29.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.871+0000 7f436d952640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:48:29.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.871+0000 7f43667fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:29.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.871+0000 7f436d952640 10 monclient: get_monmap_and_config success 2024-09-11T01:48:29.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.871+0000 7f43667fc640 1 -- 172.21.15.12:0/3345379442 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4350032a10 con 0x7f43680a4970 2024-09-11T01:48:29.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.871+0000 7f43667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:29.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.871+0000 7f436d952640 4 set_mon_vals no callback set 2024-09-11T01:48:29.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.871+0000 7f43667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:48:29.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.871+0000 7f43667fc640 10 monclient: dump: 2024-09-11T01:48:29.888 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:29.888 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:29.888 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.888 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.888 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:29.888 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.888 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:29.888 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:29.889 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:29.889 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.872+0000 7f436d952640 10 monclient: shutdown 2024-09-11T01:48:29.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.872+0000 7f436d952640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:29.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.872+0000 7f436d952640 1 -- 172.21.15.12:0/3345379442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43680a4970 msgr2=0x7f436806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.872+0000 7f436d952640 1 --2- 172.21.15.12:0/3345379442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43680a4970 0x7f436806ded0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f43500097a0 tx=0x7f4350029fd0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.872+0000 7f436d952640 1 -- 172.21.15.12:0/3345379442 shutdown_connections 2024-09-11T01:48:29.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.872+0000 7f436d952640 1 --2- 172.21.15.12:0/3345379442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43680a4970 0x7f436806ded0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.872+0000 7f436d952640 1 -- 172.21.15.12:0/3345379442 >> 172.21.15.12:0/3345379442 conn(0x7f43680a8600 msgr2=0x7f43680a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:29.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.872+0000 7f436d952640 1 -- 172.21.15.12:0/3345379442 shutdown_connections 2024-09-11T01:48:29.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.872+0000 7f436d952640 1 -- 172.21.15.12:0/3345379442 wait complete. 2024-09-11T01:48:29.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.872+0000 7f436d952640 10 monclient: build_initial_monmap 2024-09-11T01:48:29.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.872+0000 7f436d952640 10 monclient: monmap: 2024-09-11T01:48:29.890 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:29.890 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:48:29.890 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:48:29.890 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:48:29.890 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:29.890 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.890 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:48:29.890 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:48:29.891 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:48:29.891 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f436d952640 1 Processor -- start 2024-09-11T01:48:29.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f436d952640 1 -- start start 2024-09-11T01:48:29.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f436d952640 10 monclient: init 2024-09-11T01:48:29.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f436d952640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:29.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f436d952640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:48:29.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f436d952640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f436806f910 0x7f4368137140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f436d952640 10 monclient(hunting): picked mon.noname-c con 0x7f436806f910 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:29.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f436d952640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43680a2e20 0x7f4368137680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f436d952640 10 monclient(hunting): picked mon.noname-a con 0x7f43680a2e20 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:48:29.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f436d952640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43680a4970 0x7f4368137bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f436d952640 10 monclient(hunting): picked mon.noname-b con 0x7f43680a4970 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:48:29.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f436d952640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:29.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f436d952640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4368140d60 con 0x7f43680a2e20 2024-09-11T01:48:29.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f436d952640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:29.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f436d952640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f436806ec50 con 0x7f43680a4970 2024-09-11T01:48:29.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f436d952640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:29.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f436d952640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4368140ee0 con 0x7f436806f910 2024-09-11T01:48:29.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f436d952640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:29.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f436d952640 10 monclient(hunting): authenticate will time out at 1916.315821s 2024-09-11T01:48:29.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f435ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43680a2e20 0x7f4368137680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f4366ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f436806f910 0x7f4368137140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f435ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43680a2e20 0x7f4368137680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36766/0 (socket says 172.21.15.12:36766) 2024-09-11T01:48:29.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f4366ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f436806f910 0x7f4368137140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:44144/0 (socket says 172.21.15.12:44144) 2024-09-11T01:48:29.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f4366ffd640 1 -- 172.21.15.12:0/3890171953 learned_addr learned my addr 172.21.15.12:0/3890171953 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:48:29.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f4366ffd640 10 monclient(hunting): get_auth_request con 0x7f436806f910 auth_method 0 2024-09-11T01:48:29.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f4366ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:29.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f4366ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:29.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f4366ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:29.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f435ffff640 10 monclient(hunting): get_auth_request con 0x7f43680a2e20 auth_method 0 2024-09-11T01:48:29.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f435ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:29.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f435ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:29.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.873+0000 7f435ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:29.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.874+0000 7f4366ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:29.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.874+0000 7f4366ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:29.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.874+0000 7f4366ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:29.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.874+0000 7f435ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:29.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.874+0000 7f435ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:29.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.874+0000 7f435ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:29.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.874+0000 7f43677fe640 1 --2- 172.21.15.12:0/3890171953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43680a4970 0x7f4368137bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.874+0000 7f4366ffd640 10 monclient(hunting): handle_auth_done global_id 184358 payload 824 2024-09-11T01:48:29.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.874+0000 7f4366ffd640 1 -- 172.21.15.12:0/3890171953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43680a4970 msgr2=0x7f4368137bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.874+0000 7f4366ffd640 1 --2- 172.21.15.12:0/3890171953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43680a4970 0x7f4368137bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.874+0000 7f4366ffd640 1 -- 172.21.15.12:0/3890171953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43680a2e20 msgr2=0x7f4368137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.874+0000 7f4366ffd640 1 --2- 172.21.15.12:0/3890171953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43680a2e20 0x7f4368137680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.874+0000 7f4366ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:48:29.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.874+0000 7f4366ffd640 1 monclient: found mon.noname-c 2024-09-11T01:48:29.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.874+0000 7f4366ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:29.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.874+0000 7f4366ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:48:29.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.874+0000 7f4366ffd640 1 -- 172.21.15.12:0/3890171953 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f436813abe0 con 0x7f436806f910 2024-09-11T01:48:29.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.874+0000 7f43677fe640 10 monclient: get_auth_request con 0x7f43680a4970 auth_method 0 2024-09-11T01:48:29.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.874+0000 7f43677fe640 1 --2- 172.21.15.12:0/3890171953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43680a4970 0x7f4368137bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:48:29.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.874+0000 7f435ffff640 1 --2- 172.21.15.12:0/3890171953 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43680a2e20 0x7f4368137680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:48:29.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.876+0000 7f4366ffd640 1 -- 172.21.15.12:0/3890171953 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f436806f910 msgr2=0x7f4368137140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:29.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.876+0000 7f4366ffd640 1 --2- 172.21.15.12:0/3890171953 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f436806f910 0x7f4368137140 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f435800c430 tx=0x7f435800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:29.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.876+0000 7f4364ff9640 1 -- 172.21.15.12:0/3890171953 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43580119a0 con 0x7f436806f910 2024-09-11T01:48:29.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.876+0000 7f4364ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:29.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.876+0000 7f4364ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:48:29.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.876+0000 7f4364ff9640 10 monclient: dump: 2024-09-11T01:48:29.897 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:29.897 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:29.897 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.897 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.897 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:29.897 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.897 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:29.897 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:29.897 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:29.897 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.876+0000 7f4364ff9640 10 monclient: _finish_auth 0 2024-09-11T01:48:29.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.876+0000 7f4364ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:48:29.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.876+0000 7f4364ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:29.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.876+0000 7f4364ff9640 1 -- 172.21.15.12:0/3890171953 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4358017070 con 0x7f436806f910 2024-09-11T01:48:29.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.876+0000 7f4364ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:29.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.876+0000 7f4364ff9640 1 -- 172.21.15.12:0/3890171953 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4358011d80 con 0x7f436806f910 2024-09-11T01:48:29.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.876+0000 7f436d952640 5 monclient: authenticate success, global_id 184358 2024-09-11T01:48:29.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.876+0000 7f436d952640 10 monclient: _renew_subs 2024-09-11T01:48:29.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.876+0000 7f436d952640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:48:29.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.876+0000 7f436d952640 1 -- 172.21.15.12:0/3890171953 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f436813aec0 con 0x7f436806f910 2024-09-11T01:48:29.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.876+0000 7f4364ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:29.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.876+0000 7f4364ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:48:29.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.876+0000 7f4364ff9640 10 monclient: dump: 2024-09-11T01:48:29.899 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:29.899 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:29.899 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.899 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.899 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:29.899 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.899 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:29.899 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:29.899 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:29.899 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.876+0000 7f4365ffb640 4 set_mon_vals no callback set 2024-09-11T01:48:29.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.876+0000 7f436d952640 10 monclient: _renew_subs 2024-09-11T01:48:29.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.876+0000 7f436d952640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:48:29.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.876+0000 7f436d952640 1 -- 172.21.15.12:0/3890171953 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4368140800 con 0x7f436806f910 2024-09-11T01:48:29.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.877+0000 7f436d952640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:29.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.877+0000 7f436d952640 1 --2- 172.21.15.12:0/3890171953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43681cda30 0x7f436802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.877+0000 7f436d952640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:29.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.877+0000 7f436d952640 1 -- 172.21.15.12:0/3890171953 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f436806a060 con 0x7f43681cda30 2024-09-11T01:48:29.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.877+0000 7f435ffff640 1 --2- 172.21.15.12:0/3890171953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43681cda30 0x7f436802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.877+0000 7f4364ff9640 1 -- 172.21.15.12:0/3890171953 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f435801c460 con 0x7f436806f910 2024-09-11T01:48:29.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.877+0000 7f435ffff640 10 monclient: get_auth_request con 0x7f43681cda30 auth_method 0 2024-09-11T01:48:29.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.877+0000 7f435ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:29.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.877+0000 7f435ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:29.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.877+0000 7f435ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:29.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.877+0000 7f435ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:29.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.877+0000 7f435ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:29.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.877+0000 7f435ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:29.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.877+0000 7f4364ff9640 1 --2- 172.21.15.12:0/3890171953 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f434803e240 0x7f4348040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.877+0000 7f435ffff640 10 monclient(hunting): handle_auth_done global_id 134758 payload 824 2024-09-11T01:48:29.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.878+0000 7f4364ff9640 1 -- 172.21.15.12:0/3890171953 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(640..640 src has 507..640) ==== 4827+0+0 (secure 0 0 0) 0x7f4358052bd0 con 0x7f436806f910 2024-09-11T01:48:29.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.878+0000 7f435ffff640 1 -- 172.21.15.12:0/3890171953 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4368072350 con 0x7f43681cda30 2024-09-11T01:48:29.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.878+0000 7f43677fe640 1 --2- 172.21.15.12:0/3890171953 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f434803e240 0x7f4348040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.878+0000 7f43677fe640 10 monclient: get_auth_request con 0x7f434803e240 auth_method 0 2024-09-11T01:48:29.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.880+0000 7f435ffff640 1 -- 172.21.15.12:0/3890171953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43681cda30 msgr2=0x7f436802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:29.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.880+0000 7f435ffff640 1 --2- 172.21.15.12:0/3890171953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43681cda30 0x7f436802d440 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4350034000 tx=0x7f43500046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:29.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.880+0000 7f4364ff9640 1 -- 172.21.15.12:0/3890171953 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4350037040 con 0x7f43681cda30 2024-09-11T01:48:29.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.880+0000 7f4364ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:29.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.880+0000 7f4364ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:48:29.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.880+0000 7f4364ff9640 10 monclient: dump: 2024-09-11T01:48:29.905 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:29.905 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:29.906 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.906 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.906 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:29.906 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.906 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:29.906 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:29.906 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:29.906 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.880+0000 7f43677fe640 1 --2- 172.21.15.12:0/3890171953 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f434803e240 0x7f4348040700 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4360005ac0 tx=0x7f4360009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:29.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.880+0000 7f4364ff9640 1 -- 172.21.15.12:0/3890171953 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f435003ea80 con 0x7f43681cda30 2024-09-11T01:48:29.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.880+0000 7f4364ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:29.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.880+0000 7f4364ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:48:29.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.880+0000 7f4364ff9640 1 -- 172.21.15.12:0/3890171953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43681cda30 msgr2=0x7f436802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.880+0000 7f4364ff9640 1 --2- 172.21.15.12:0/3890171953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43681cda30 0x7f436802d440 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4350034000 tx=0x7f43500046c0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.880+0000 7f4364ff9640 1 -- 172.21.15.12:0/3890171953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43681cda30 msgr2=0x7f436802d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:48:29.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.880+0000 7f4364ff9640 1 --2- 172.21.15.12:0/3890171953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43681cda30 0x7f436802d440 secure :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4350034000 tx=0x7f43500046c0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.887+0000 7f436d952640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:48:29.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.887+0000 7f436d952640 1 --2- 172.21.15.12:0/3890171953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43681cde10 0x7f436802d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:29.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.887+0000 7f436d952640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:29.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.887+0000 7f436d952640 1 -- 172.21.15.12:0/3890171953 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f436813a900 con 0x7f43681cde10 2024-09-11T01:48:29.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.887+0000 7f435ffff640 1 --2- 172.21.15.12:0/3890171953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43681cde10 0x7f436802d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:29.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.887+0000 7f435ffff640 10 monclient: get_auth_request con 0x7f43681cde10 auth_method 0 2024-09-11T01:48:29.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.887+0000 7f435ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:29.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.887+0000 7f435ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:29.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.887+0000 7f435ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:29.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.887+0000 7f435ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:29.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.887+0000 7f435ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:29.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.887+0000 7f435ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:29.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.887+0000 7f435ffff640 10 monclient(hunting): handle_auth_done global_id 134761 payload 824 2024-09-11T01:48:29.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.887+0000 7f435ffff640 1 -- 172.21.15.12:0/3890171953 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f436806ded0 con 0x7f43681cde10 2024-09-11T01:48:29.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.889+0000 7f435ffff640 1 -- 172.21.15.12:0/3890171953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43681cde10 msgr2=0x7f436802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:29.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.889+0000 7f435ffff640 1 --2- 172.21.15.12:0/3890171953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43681cde10 0x7f436802d980 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f435002a010 tx=0x7f4350008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:29.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.890+0000 7f4364ff9640 1 -- 172.21.15.12:0/3890171953 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4350037040 con 0x7f43681cde10 2024-09-11T01:48:29.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.890+0000 7f4364ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:29.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.890+0000 7f4364ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:48:29.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.890+0000 7f4364ff9640 10 monclient: dump: 2024-09-11T01:48:29.909 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:29.909 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:29.910 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.910 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:29.910 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:29.910 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:29.910 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:29.910 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:29.910 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:29.911 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:29.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.890+0000 7f4364ff9640 1 -- 172.21.15.12:0/3890171953 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f435004e070 con 0x7f43681cde10 2024-09-11T01:48:29.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.890+0000 7f4364ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:48:29.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.890+0000 7f4364ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:48:29.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.890+0000 7f4364ff9640 1 -- 172.21.15.12:0/3890171953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43681cde10 msgr2=0x7f436802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.890+0000 7f4364ff9640 1 --2- 172.21.15.12:0/3890171953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43681cde10 0x7f436802d980 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f435002a010 tx=0x7f4350008030 comp rx=0 tx=0).stop 2024-09-11T01:48:29.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.890+0000 7f4364ff9640 1 -- 172.21.15.12:0/3890171953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43681cde10 msgr2=0x7f436802d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:48:29.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.890+0000 7f4364ff9640 1 --2- 172.21.15.12:0/3890171953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43681cde10 0x7f436802d980 secure :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f435002a010 tx=0x7f4350008030 comp rx=0 tx=0).stop 2024-09-11T01:48:29.913 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:48:29.913 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:48:29.913 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:48:29.913 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:48:29.913 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 182, 2024-09-11T01:48:29.914 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 442852, 2024-09-11T01:48:29.914 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:48:29.914 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:48:29.914 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:48:29.914 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:48:29.914 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:29.915 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:48:29.915 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:48:29.915 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:48:29.915 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:48:29.915 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:48:29.915 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:48:29.916 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:48:29.916 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:48:29.916 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:48:29.916 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:48:29.916 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:48:29.916 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:48:29.917 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:48:29.917 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:48:29.917 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:48:29.917 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:29.917 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:48:29.918 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:48:29.918 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:48:29.918 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:48:29.918 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:48:29.918 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:48:29.918 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:48:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:48:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:48:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:48:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:48:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:48:29.920 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:48:29.920 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:29.920 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.920 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:48:29.921 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:48:29.921 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:48:29.921 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:48:29.921 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:48:29.921 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:48:29.922 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:48:29.922 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:48:29.922 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:48:29.922 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:48:29.922 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:48:29.923 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:48:29.923 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:48:29.923 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:48:29.923 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:48:29.923 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:48:29.924 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:48:29.924 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:48:29.924 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:48:29.924 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:48:29.924 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:48:29.925 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:48:29.925 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:48:29.925 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:48:29.925 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:48:29.925 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:48:29.926 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:48:29.926 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:48:29.926 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:29.926 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:48:29.926 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.926 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:48:29.927 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.927 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:48:29.927 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:48:29.927 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:48:29.927 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:48:29.927 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.928 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:48:29.928 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:48:29.928 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:29.928 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.928 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.929 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:48:29.929 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:48:29.929 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:29.929 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:29.929 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:29.930 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.930 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:48:29.930 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:48:29.930 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:48:29.930 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:48:29.930 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:48:29.931 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.931 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.931 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:48:29.931 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:48:29.931 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:48:29.931 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:48:29.932 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.932 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:48:29.932 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:48:29.932 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:29.932 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.932 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.933 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:48:29.933 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:48:29.933 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:29.933 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:29.933 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:48:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:48:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:48:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:48:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:48:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:48:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:48:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:48:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:48:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:48:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:48:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.937 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.937 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:48:29.937 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:48:29.937 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:29.938 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:29.938 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:29.938 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.938 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:48:29.939 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:48:29.939 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:48:29.939 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:48:29.939 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:48:29.939 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:29.940 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:29.940 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.940 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:48:29.940 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:48:29.941 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.941 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:48:29.941 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:48:29.941 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:48:29.941 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:29.941 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:29.941 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:48:29.942 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.942 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:48:29.942 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:48:29.942 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:48:29.942 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:29.943 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:29.943 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:48:29.943 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.943 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:48:29.943 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:48:29.943 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:48:29.944 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:29.944 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:29.944 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:48:29.944 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:29.944 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:48:29.944 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:48:29.945 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:48:29.945 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:29.945 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:29.945 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:29.945 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:48:29.946 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:48:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.890+0000 7f436d952640 1 -- 172.21.15.12:0/3890171953 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f434803e240 msgr2=0x7f4348040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.890+0000 7f436d952640 1 --2- 172.21.15.12:0/3890171953 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f434803e240 0x7f4348040700 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4360005ac0 tx=0x7f4360009040 comp rx=0 tx=0).stop 2024-09-11T01:48:29.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.890+0000 7f436d952640 10 monclient: shutdown 2024-09-11T01:48:29.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.890+0000 7f436d952640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:29.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.890+0000 7f436d952640 1 -- 172.21.15.12:0/3890171953 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f436806f910 msgr2=0x7f4368137140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:29.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.890+0000 7f436d952640 1 --2- 172.21.15.12:0/3890171953 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f436806f910 0x7f4368137140 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f435800c430 tx=0x7f435800c8f0 comp rx=0 tx=0).stop 2024-09-11T01:48:29.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.890+0000 7f43677fe640 1 -- 172.21.15.12:0/3890171953 reap_dead start 2024-09-11T01:48:29.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.890+0000 7f43677fe640 1 -- 172.21.15.12:0/3890171953 reap_dead start 2024-09-11T01:48:29.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.890+0000 7f436d952640 1 -- 172.21.15.12:0/3890171953 shutdown_connections 2024-09-11T01:48:29.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.890+0000 7f436d952640 1 -- 172.21.15.12:0/3890171953 >> 172.21.15.12:0/3890171953 conn(0x7f43680a8600 msgr2=0x7f436809e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:29.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.890+0000 7f436d952640 1 -- 172.21.15.12:0/3890171953 shutdown_connections 2024-09-11T01:48:29.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:29.890+0000 7f436d952640 1 -- 172.21.15.12:0/3890171953 wait complete. 2024-09-11T01:48:29.949 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:48:29.949 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:48:29.949 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:48:30.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.048+0000 7fde36c62640 10 monclient: get_monmap_and_config 2024-09-11T01:48:30.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.048+0000 7fde36c62640 10 monclient: build_initial_monmap 2024-09-11T01:48:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.048+0000 7fde36c62640 10 monclient: monmap: 2024-09-11T01:48:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:48:30.049471+0000 2024-09-11T01:48:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:48:30.049471+0000 2024-09-11T01:48:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:48:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:48:30.051 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:48:30.052 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:48:30.052 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:48:30.052 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:30.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.049+0000 7fde36c62640 1 Processor -- start 2024-09-11T01:48:30.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.049+0000 7fde36c62640 1 -- start start 2024-09-11T01:48:30.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.049+0000 7fde36c62640 10 monclient: init 2024-09-11T01:48:30.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde36c62640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:30.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde36c62640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-11T01:48:30.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde36c62640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fde30134ad0 0x7fde30134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:30.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde36c62640 10 monclient(hunting): picked mon.noname-c con 0x7fde30134ad0 addr v2:172.21.15.71:3300/0 2024-09-11T01:48:30.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde36c62640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fde300a2200 0x7fde301353f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:30.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde36c62640 10 monclient(hunting): picked mon.noname-b con 0x7fde300a2200 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:30.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde36c62640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fde30133a50 addr v1:172.21.15.71:6789/0 2024-09-11T01:48:30.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde36c62640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde3009ed30 con 0x7fde30133a50 2024-09-11T01:48:30.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde36c62640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:30.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde36c62640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fde3009f5d0 con 0x7fde30134ad0 2024-09-11T01:48:30.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde36c62640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:30.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde36c62640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fde30135930 con 0x7fde300a2200 2024-09-11T01:48:30.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde36c62640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:30.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde36c62640 10 monclient(hunting): authenticate will time out at 1916.492217s 2024-09-11T01:48:30.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde2ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fde300a2200 0x7fde301353f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:30.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde2ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fde300a2200 0x7fde301353f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:44154/0 (socket says 172.21.15.12:44154) 2024-09-11T01:48:30.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde2ffff640 1 -- 172.21.15.12:0/3058540533 learned_addr learned my addr 172.21.15.12:0/3058540533 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:48:30.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde2ffff640 10 monclient(hunting): get_auth_request con 0x7fde300a2200 auth_method 0 2024-09-11T01:48:30.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:30.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:30.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:30.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:30.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:30.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:30.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde349fe640 1 --2- 172.21.15.12:0/3058540533 >> v2:172.21.15.71:3300/0 conn(0x7fde30134ad0 0x7fde30134eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:30.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde2ffff640 10 monclient(hunting): handle_auth_done global_id 184361 payload 274 2024-09-11T01:48:30.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde2ffff640 1 -- 172.21.15.12:0/3058540533 >> v2:172.21.15.71:3300/0 conn(0x7fde30134ad0 msgr2=0x7fde30134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:48:30.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde2ffff640 1 --2- 172.21.15.12:0/3058540533 >> v2:172.21.15.71:3300/0 conn(0x7fde30134ad0 0x7fde30134eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:30.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde2ffff640 1 -- 172.21.15.12:0/3058540533 >> v1:172.21.15.71:6789/0 conn(0x7fde30133a50 legacy=0x7fde30133ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:48:30.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde2ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:48:30.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde2ffff640 1 monclient: found mon.noname-b 2024-09-11T01:48:30.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:30.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde2ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:48:30.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde2ffff640 1 -- 172.21.15.12:0/3058540533 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde3009f010 con 0x7fde300a2200 2024-09-11T01:48:30.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde349fe640 10 monclient: get_auth_request con 0x7fde30134ad0 auth_method 0 2024-09-11T01:48:30.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.050+0000 7fde349fe640 1 --2- 172.21.15.12:0/3058540533 >> v2:172.21.15.71:3300/0 conn(0x7fde30134ad0 0x7fde30134eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:48:30.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.053+0000 7fde2ffff640 1 -- 172.21.15.12:0/3058540533 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fde300a2200 msgr2=0x7fde301353f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:30.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.053+0000 7fde2ffff640 1 --2- 172.21.15.12:0/3058540533 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fde300a2200 0x7fde301353f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fde24009870 tx=0x7fde2402a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d3fed574aeed1dc server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.053+0000 7fde2f7fe640 1 -- 172.21.15.12:0/3058540533 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde24037070 con 0x7fde300a2200 2024-09-11T01:48:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.053+0000 7fde2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.053+0000 7fde2f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:48:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.053+0000 7fde2f7fe640 10 monclient: dump: 2024-09-11T01:48:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:30.061 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:30.062 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.053+0000 7fde2f7fe640 10 monclient: _finish_auth 0 2024-09-11T01:48:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.053+0000 7fde2f7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:48:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.053+0000 7fde2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.053+0000 7fde2f7fe640 1 -- 172.21.15.12:0/3058540533 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde24032640 con 0x7fde300a2200 2024-09-11T01:48:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.053+0000 7fde36c62640 5 monclient: authenticate success, global_id 184361 2024-09-11T01:48:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.053+0000 7fde2f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.053+0000 7fde36c62640 10 monclient: get_monmap_and_config success 2024-09-11T01:48:30.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.053+0000 7fde2f7fe640 1 -- 172.21.15.12:0/3058540533 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde24032a10 con 0x7fde300a2200 2024-09-11T01:48:30.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.053+0000 7fde36c62640 4 set_mon_vals no callback set 2024-09-11T01:48:30.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.053+0000 7fde2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:30.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.053+0000 7fde2f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:48:30.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.053+0000 7fde2f7fe640 10 monclient: dump: 2024-09-11T01:48:30.063 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:30.063 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:30.063 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:30.063 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:30.063 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:30.063 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:30.064 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.054+0000 7fde36c62640 10 monclient: shutdown 2024-09-11T01:48:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.054+0000 7fde36c62640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.054+0000 7fde36c62640 1 -- 172.21.15.12:0/3058540533 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fde300a2200 msgr2=0x7fde301353f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.054+0000 7fde36c62640 1 --2- 172.21.15.12:0/3058540533 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fde300a2200 0x7fde301353f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fde24009870 tx=0x7fde2402a0a0 comp rx=0 tx=0).stop 2024-09-11T01:48:30.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.054+0000 7fde36c62640 1 -- 172.21.15.12:0/3058540533 shutdown_connections 2024-09-11T01:48:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.054+0000 7fde36c62640 1 --2- 172.21.15.12:0/3058540533 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fde300a2200 0x7fde301353f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.054+0000 7fde36c62640 1 --2- 172.21.15.12:0/3058540533 >> v2:172.21.15.71:3300/0 conn(0x7fde30134ad0 0x7fde30134eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.054+0000 7fde36c62640 1 -- 172.21.15.12:0/3058540533 >> 172.21.15.12:0/3058540533 conn(0x7fde300a2f70 msgr2=0x7fde300a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.054+0000 7fde36c62640 1 -- 172.21.15.12:0/3058540533 shutdown_connections 2024-09-11T01:48:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.054+0000 7fde36c62640 1 -- 172.21.15.12:0/3058540533 wait complete. 2024-09-11T01:48:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.055+0000 7fde36c62640 10 monclient: build_initial_monmap 2024-09-11T01:48:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.055+0000 7fde36c62640 10 monclient: monmap: 2024-09-11T01:48:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:48:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:48:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:48:30.065 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:30.066 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:30.066 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:48:30.066 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:48:30.066 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:48:30.066 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:30.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.055+0000 7fde36c62640 1 Processor -- start 2024-09-11T01:48:30.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.055+0000 7fde36c62640 1 -- start start 2024-09-11T01:48:30.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.055+0000 7fde36c62640 10 monclient: init 2024-09-11T01:48:30.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde36c62640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:30.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde36c62640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:48:30.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde36c62640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde300a2200 0x7fde3013f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:30.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde36c62640 10 monclient(hunting): picked mon.noname-b con 0x7fde300a2200 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:48:30.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde36c62640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde30133a50 0x7fde301cd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:30.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde36c62640 10 monclient(hunting): picked mon.noname-a con 0x7fde30133a50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:48:30.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde36c62640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fde30134ad0 0x7fde301cd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:30.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde36c62640 10 monclient(hunting): picked mon.noname-c con 0x7fde30134ad0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:30.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde36c62640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:30.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde36c62640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fde300a0150 con 0x7fde30133a50 2024-09-11T01:48:30.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde36c62640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:30.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde36c62640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fde3009ffd0 con 0x7fde300a2200 2024-09-11T01:48:30.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde36c62640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:30.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde36c62640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fde301345f0 con 0x7fde30134ad0 2024-09-11T01:48:30.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde36c62640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:30.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde36c62640 10 monclient(hunting): authenticate will time out at 1916.498372s 2024-09-11T01:48:30.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde351ff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fde30134ad0 0x7fde301cd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:30.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde2ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde30133a50 0x7fde301cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:30.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde2ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde30133a50 0x7fde301cd490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36770/0 (socket says 172.21.15.12:36770) 2024-09-11T01:48:30.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde351ff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fde30134ad0 0x7fde301cd9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:44164/0 (socket says 172.21.15.12:44164) 2024-09-11T01:48:30.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde2ffff640 1 -- 172.21.15.12:0/1621946709 learned_addr learned my addr 172.21.15.12:0/1621946709 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:48:30.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde2ffff640 10 monclient(hunting): get_auth_request con 0x7fde30133a50 auth_method 0 2024-09-11T01:48:30.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:30.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:30.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:30.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde351ff640 10 monclient(hunting): get_auth_request con 0x7fde30134ad0 auth_method 0 2024-09-11T01:48:30.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde351ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:30.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde351ff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:30.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde351ff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:30.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde349fe640 1 --2- 172.21.15.12:0/1621946709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde300a2200 0x7fde3013f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:30.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde351ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:30.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde351ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:30.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde351ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:30.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:30.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:30.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:30.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde349fe640 10 monclient(hunting): get_auth_request con 0x7fde300a2200 auth_method 0 2024-09-11T01:48:30.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde349fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:30.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde349fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:30.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde349fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:30.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde349fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:30.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde349fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:30.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde349fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:30.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde2ffff640 10 monclient(hunting): handle_auth_done global_id 214122 payload 824 2024-09-11T01:48:30.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde2ffff640 1 -- 172.21.15.12:0/1621946709 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fde30134ad0 msgr2=0x7fde301cd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:30.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde2ffff640 1 --2- 172.21.15.12:0/1621946709 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fde30134ad0 0x7fde301cd9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:30.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde2ffff640 1 -- 172.21.15.12:0/1621946709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde300a2200 msgr2=0x7fde3013f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:30.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde2ffff640 1 --2- 172.21.15.12:0/1621946709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde300a2200 0x7fde3013f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:30.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde2ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:48:30.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde2ffff640 1 monclient: found mon.noname-a 2024-09-11T01:48:30.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:30.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde2ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:48:30.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde2ffff640 1 -- 172.21.15.12:0/1621946709 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde30133870 con 0x7fde30133a50 2024-09-11T01:48:30.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.056+0000 7fde351ff640 1 --2- 172.21.15.12:0/1621946709 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fde30134ad0 0x7fde301cd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:48:30.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.059+0000 7fde2ffff640 1 -- 172.21.15.12:0/1621946709 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde30133a50 msgr2=0x7fde301cd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:30.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.059+0000 7fde2ffff640 1 --2- 172.21.15.12:0/1621946709 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde30133a50 0x7fde301cd490 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fde240090b0 tx=0x7fde240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:30.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.059+0000 7fde2d7fa640 1 -- 172.21.15.12:0/1621946709 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde24041020 con 0x7fde30133a50 2024-09-11T01:48:30.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.059+0000 7fde2d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:30.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.059+0000 7fde2d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:48:30.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.059+0000 7fde2d7fa640 10 monclient: dump: 2024-09-11T01:48:30.078 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:30.078 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:30.078 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:30.078 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:30.078 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:30.078 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:30.078 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:30.079 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:30.079 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:30.079 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:30.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.059+0000 7fde2d7fa640 10 monclient: _finish_auth 0 2024-09-11T01:48:30.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.059+0000 7fde2d7fa640 10 monclient: _check_auth_tickets 2024-09-11T01:48:30.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.059+0000 7fde2d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:30.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.059+0000 7fde2d7fa640 1 -- 172.21.15.12:0/1621946709 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde2403d830 con 0x7fde30133a50 2024-09-11T01:48:30.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.059+0000 7fde2d7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:30.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.059+0000 7fde2d7fa640 1 -- 172.21.15.12:0/1621946709 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde24037040 con 0x7fde30133a50 2024-09-11T01:48:30.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.059+0000 7fde2d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:30.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.059+0000 7fde2d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:48:30.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.059+0000 7fde2d7fa640 10 monclient: dump: 2024-09-11T01:48:30.080 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:30.080 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:30.080 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:30.080 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:30.080 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:30.080 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:30.080 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:30.080 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:30.080 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:30.080 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:30.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.059+0000 7fde36c62640 5 monclient: authenticate success, global_id 214122 2024-09-11T01:48:30.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.059+0000 7fde36c62640 10 monclient: _renew_subs 2024-09-11T01:48:30.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.059+0000 7fde36c62640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:30.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.059+0000 7fde36c62640 1 -- 172.21.15.12:0/1621946709 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde3011cdf0 con 0x7fde30133a50 2024-09-11T01:48:30.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.059+0000 7fde2effd640 4 set_mon_vals no callback set 2024-09-11T01:48:30.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.059+0000 7fde36c62640 10 monclient: _renew_subs 2024-09-11T01:48:30.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.059+0000 7fde36c62640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:30.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.059+0000 7fde36c62640 1 -- 172.21.15.12:0/1621946709 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde300a1710 con 0x7fde30133a50 2024-09-11T01:48:30.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.060+0000 7fde2d7fa640 1 -- 172.21.15.12:0/1621946709 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fde2403a030 con 0x7fde30133a50 2024-09-11T01:48:30.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.060+0000 7fde36c62640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:30.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.060+0000 7fde36c62640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:30.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.060+0000 7fde36c62640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:30.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.060+0000 7fde36c62640 1 -- 172.21.15.12:0/1621946709 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde30133ee0 con 0x7fde30133a50 2024-09-11T01:48:30.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.060+0000 7fde2d7fa640 1 --2- 172.21.15.12:0/1621946709 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fde1c03e210 0x7fde1c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:30.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.060+0000 7fde2d7fa640 1 -- 172.21.15.12:0/1621946709 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(640..640 src has 507..640) ==== 4827+0+0 (secure 0 0 0) 0x7fde240731a0 con 0x7fde30133a50 2024-09-11T01:48:30.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.060+0000 7fde349fe640 1 --2- 172.21.15.12:0/1621946709 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fde1c03e210 0x7fde1c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:30.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.060+0000 7fde349fe640 10 monclient: get_auth_request con 0x7fde1c03e210 auth_method 0 2024-09-11T01:48:30.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.063+0000 7fde349fe640 1 --2- 172.21.15.12:0/1621946709 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fde1c03e210 0x7fde1c0406d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fde2000ab60 tx=0x7fde20008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:30.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.065+0000 7fde2d7fa640 1 -- 172.21.15.12:0/1621946709 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fde2404b030 con 0x7fde30133a50 2024-09-11T01:48:30.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.065+0000 7fde2d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:30.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.065+0000 7fde2d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:48:30.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.319+0000 7fde36c62640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:48:30.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.319+0000 7fde36c62640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:48:30.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.319+0000 7fde36c62640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:30.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.319+0000 7fde36c62640 1 -- 172.21.15.12:0/1621946709 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fde3006d480 con 0x7fde30133a50 2024-09-11T01:48:30.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.319+0000 7fde2d7fa640 1 -- 172.21.15.12:0/1621946709 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fde30133870 con 0x7fde30133a50 2024-09-11T01:48:30.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.320+0000 7fde2d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:48:30.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.320+0000 7fde2d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:48:30.323 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.322+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.324 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.322+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=1,health=11,kv=12,logm=45} crc {auth=2841893604,config=1228700967,health=3285666041,kv=3263697420,logm=2974606316}) 2024-09-11T01:48:30.324 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.322+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=30,config=2,health=12,kv=12,logm=44} crc {auth=2217510499,config=2586702516,health=3041450585,kv=631696822,logm=877735308}) 2024-09-11T01:48:30.324 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.322+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.324 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.322+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=1,health=11,kv=12,logm=45} crc {auth=2841893604,config=1228700967,health=3285666041,kv=3263697420,logm=2974606316}) 2024-09-11T01:48:30.324 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.322+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=31,config=2,health=12,kv=12,logm=43} crc {auth=2540406501,config=2586702516,health=3041450585,kv=3263697420,logm=710370781}) 2024-09-11T01:48:30.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.322+0000 7fde36c62640 1 -- 172.21.15.12:0/1621946709 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fde1c03e210 msgr2=0x7fde1c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:30.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.322+0000 7fde36c62640 1 --2- 172.21.15.12:0/1621946709 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fde1c03e210 0x7fde1c0406d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fde2000ab60 tx=0x7fde20008040 comp rx=0 tx=0).stop 2024-09-11T01:48:30.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.322+0000 7fde36c62640 10 monclient: shutdown 2024-09-11T01:48:30.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.322+0000 7fde36c62640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:30.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.322+0000 7fde36c62640 1 -- 172.21.15.12:0/1621946709 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde30133a50 msgr2=0x7fde301cd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:30.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.322+0000 7fde36c62640 1 --2- 172.21.15.12:0/1621946709 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde30133a50 0x7fde301cd490 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fde240090b0 tx=0x7fde240046c0 comp rx=0 tx=0).stop 2024-09-11T01:48:30.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.322+0000 7fde36c62640 1 -- 172.21.15.12:0/1621946709 shutdown_connections 2024-09-11T01:48:30.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.322+0000 7fde36c62640 1 --2- 172.21.15.12:0/1621946709 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fde30134ad0 0x7fde301cd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:30.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.322+0000 7fde36c62640 1 --2- 172.21.15.12:0/1621946709 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fde1c03e210 0x7fde1c0406d0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:30.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.322+0000 7fde36c62640 1 --2- 172.21.15.12:0/1621946709 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde30133a50 0x7fde301cd490 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:30.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.322+0000 7fde36c62640 1 --2- 172.21.15.12:0/1621946709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde300a2200 0x7fde3013f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:30.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.322+0000 7fde36c62640 1 -- 172.21.15.12:0/1621946709 >> 172.21.15.12:0/1621946709 conn(0x7fde300a2f70 msgr2=0x7fde3009d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:30.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.322+0000 7fde36c62640 1 -- 172.21.15.12:0/1621946709 shutdown_connections 2024-09-11T01:48:30.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:30.322+0000 7fde36c62640 1 -- 172.21.15.12:0/1621946709 wait complete. 2024-09-11T01:48:30.327 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.326+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.328 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.326+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1212239581}) 2024-09-11T01:48:30.328 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.326+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3133336024}) 2024-09-11T01:48:30.328 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.326+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.328 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.326+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1212239581}) 2024-09-11T01:48:30.328 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.326+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=853840334}) 2024-09-11T01:48:30.329 INFO:tasks.rados.rados.0.smithi071.stdout:2837: done (2 left) 2024-09-11T01:48:30.329 INFO:tasks.rados.rados.0.smithi071.stdout:2838: snap_remove snap 251 2024-09-11T01:48:30.340 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:48:30.477 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.474+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.477 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.475+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4870088}) 2024-09-11T01:48:30.477 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.475+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3415050444}) 2024-09-11T01:48:30.477 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.475+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.477 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.475+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4870088}) 2024-09-11T01:48:30.478 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.475+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1414944021}) 2024-09-11T01:48:30.480 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.478+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.480 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.479+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=590450709}) 2024-09-11T01:48:30.481 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.479+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3376041596}) 2024-09-11T01:48:30.481 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.479+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.481 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.479+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=590450709}) 2024-09-11T01:48:30.481 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.479+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3041596130}) 2024-09-11T01:48:30.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.570+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.570+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=65358837}) 2024-09-11T01:48:30.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.570+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4237119654}) 2024-09-11T01:48:30.572 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.570+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.573 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.570+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=65358837}) 2024-09-11T01:48:30.573 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.570+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3226007561}) 2024-09-11T01:48:30.575 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.571+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.575 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.571+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=658331162}) 2024-09-11T01:48:30.575 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.571+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1059032172}) 2024-09-11T01:48:30.575 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.571+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.575 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.571+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=658331162}) 2024-09-11T01:48:30.575 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.571+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2963445388}) 2024-09-11T01:48:30.575 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.572+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.575 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.572+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2526055455}) 2024-09-11T01:48:30.575 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.572+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1242591490}) 2024-09-11T01:48:30.576 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.572+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.576 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.572+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2526055455}) 2024-09-11T01:48:30.576 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.572+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2481465126}) 2024-09-11T01:48:30.576 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.573+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.576 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.573+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2077448540}) 2024-09-11T01:48:30.576 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.573+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3413858125}) 2024-09-11T01:48:30.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.573+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.573+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2077448540}) 2024-09-11T01:48:30.577 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.573+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3188646394}) 2024-09-11T01:48:30.582 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.574+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.582 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.574+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3665008831}) 2024-09-11T01:48:30.582 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.574+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2183369806}) 2024-09-11T01:48:30.582 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.574+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.582 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.574+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3665008831}) 2024-09-11T01:48:30.582 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.574+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=168519276}) 2024-09-11T01:48:30.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.575+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.575+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3951892824}) 2024-09-11T01:48:30.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.575+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1471101457}) 2024-09-11T01:48:30.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.575+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.575+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3951892824}) 2024-09-11T01:48:30.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.575+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2370327827}) 2024-09-11T01:48:30.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.576+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.576+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2323473282}) 2024-09-11T01:48:30.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.576+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3700807984}) 2024-09-11T01:48:30.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.576+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.576+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2323473282}) 2024-09-11T01:48:30.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.576+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3095977634}) 2024-09-11T01:48:30.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.576+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.576+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2266041948}) 2024-09-11T01:48:30.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.576+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1038563015}) 2024-09-11T01:48:30.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.576+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.576+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2266041948}) 2024-09-11T01:48:30.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.576+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=560601023}) 2024-09-11T01:48:30.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.577+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.577+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613909689}) 2024-09-11T01:48:30.586 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.577+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=23784436}) 2024-09-11T01:48:30.586 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.577+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.586 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.577+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613909689}) 2024-09-11T01:48:30.586 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.577+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=777821204}) 2024-09-11T01:48:30.586 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.578+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.586 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.578+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1990795194}) 2024-09-11T01:48:30.586 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.578+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3561183650}) 2024-09-11T01:48:30.587 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.578+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.587 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.578+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1990795194}) 2024-09-11T01:48:30.587 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.578+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1159851296}) 2024-09-11T01:48:30.587 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.579+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.587 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.579+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=325107225}) 2024-09-11T01:48:30.587 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.579+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3358302572}) 2024-09-11T01:48:30.588 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.579+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.588 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.579+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=325107225}) 2024-09-11T01:48:30.588 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.579+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2460294875}) 2024-09-11T01:48:30.589 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.580+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.589 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.580+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=881635033}) 2024-09-11T01:48:30.589 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.580+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=760006114}) 2024-09-11T01:48:30.589 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.580+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.589 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.580+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=881635033}) 2024-09-11T01:48:30.590 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.580+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2599619710}) 2024-09-11T01:48:30.627 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.581+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.627 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.581+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619137415}) 2024-09-11T01:48:30.628 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.581+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1456717462}) 2024-09-11T01:48:30.628 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.581+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.628 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.581+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619137415}) 2024-09-11T01:48:30.628 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.581+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3876926668}) 2024-09-11T01:48:30.628 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.581+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.628 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.581+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2804351575}) 2024-09-11T01:48:30.628 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.581+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1906392471}) 2024-09-11T01:48:30.628 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.581+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.628 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.581+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2804351575}) 2024-09-11T01:48:30.628 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.582+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1719385327}) 2024-09-11T01:48:30.628 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.583+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.629 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.583+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4164600535}) 2024-09-11T01:48:30.629 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.583+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2916623137}) 2024-09-11T01:48:30.629 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.583+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.629 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.583+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4164600535}) 2024-09-11T01:48:30.629 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.583+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=619327021}) 2024-09-11T01:48:30.629 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.583+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.629 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.583+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821182847}) 2024-09-11T01:48:30.629 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.583+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2639106735}) 2024-09-11T01:48:30.629 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.583+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.629 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.583+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821182847}) 2024-09-11T01:48:30.629 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.583+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=938666963}) 2024-09-11T01:48:30.629 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.584+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.630 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.584+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=388022958}) 2024-09-11T01:48:30.630 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.584+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1409299604}) 2024-09-11T01:48:30.630 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.584+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.630 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.584+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=388022958}) 2024-09-11T01:48:30.630 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.584+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=269445834}) 2024-09-11T01:48:30.630 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.585+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.630 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.585+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=42340165}) 2024-09-11T01:48:30.630 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.585+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2779144020}) 2024-09-11T01:48:30.630 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.585+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.631 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.585+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=42340165}) 2024-09-11T01:48:30.631 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.585+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=154886647}) 2024-09-11T01:48:30.631 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.586+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.631 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.586+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2662264059}) 2024-09-11T01:48:30.631 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.586+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2312079012}) 2024-09-11T01:48:30.631 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.586+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.631 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.586+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2662264059}) 2024-09-11T01:48:30.631 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.586+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2032653288}) 2024-09-11T01:48:30.631 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.587+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.632 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.587+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1236769918}) 2024-09-11T01:48:30.632 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.587+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1088043556}) 2024-09-11T01:48:30.632 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.587+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.632 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.587+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1236769918}) 2024-09-11T01:48:30.632 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.587+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2809629628}) 2024-09-11T01:48:30.632 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.588+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.632 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.588+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=73841050}) 2024-09-11T01:48:30.632 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.588+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3842906205}) 2024-09-11T01:48:30.633 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.588+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.633 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.588+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=73841050}) 2024-09-11T01:48:30.633 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.588+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=235038963}) 2024-09-11T01:48:30.633 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.589+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.633 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.589+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1704692795}) 2024-09-11T01:48:30.633 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.589+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2237652446}) 2024-09-11T01:48:30.633 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.589+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.633 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.589+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1704692795}) 2024-09-11T01:48:30.633 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.589+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1622824204}) 2024-09-11T01:48:30.633 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.589+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.633 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.589+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364879886}) 2024-09-11T01:48:30.634 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.589+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4135260352}) 2024-09-11T01:48:30.634 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.589+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.634 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.589+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364879886}) 2024-09-11T01:48:30.634 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.589+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=21522820}) 2024-09-11T01:48:30.634 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.590+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.634 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.590+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1800719543}) 2024-09-11T01:48:30.634 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.590+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1423592129}) 2024-09-11T01:48:30.634 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.590+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.635 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.590+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1800719543}) 2024-09-11T01:48:30.635 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.590+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2951994069}) 2024-09-11T01:48:30.635 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.591+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.635 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.591+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=610213806}) 2024-09-11T01:48:30.635 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.591+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2977489278}) 2024-09-11T01:48:30.635 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.591+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.635 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.591+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=610213806}) 2024-09-11T01:48:30.635 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.591+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=536146595}) 2024-09-11T01:48:30.635 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.592+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.635 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.592+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921608294}) 2024-09-11T01:48:30.636 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.592+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1889856178}) 2024-09-11T01:48:30.636 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.592+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.636 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.592+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921608294}) 2024-09-11T01:48:30.636 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.592+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1643089693}) 2024-09-11T01:48:30.636 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.593+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.636 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.593+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=817423719}) 2024-09-11T01:48:30.636 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.593+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2747914522}) 2024-09-11T01:48:30.636 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.593+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.636 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.593+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=817423719}) 2024-09-11T01:48:30.636 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.593+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=593676225}) 2024-09-11T01:48:30.636 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.594+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.637 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.594+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1500033456}) 2024-09-11T01:48:30.637 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.594+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=577884529}) 2024-09-11T01:48:30.637 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.594+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.637 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.594+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1500033456}) 2024-09-11T01:48:30.637 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.594+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4050049256}) 2024-09-11T01:48:30.637 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.594+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.637 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.594+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1297931656}) 2024-09-11T01:48:30.637 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.594+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2481589123}) 2024-09-11T01:48:30.637 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.594+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.637 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.595+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1297931656}) 2024-09-11T01:48:30.638 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.595+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1679702290}) 2024-09-11T01:48:30.638 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.595+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.638 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.595+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329189300}) 2024-09-11T01:48:30.638 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.595+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1749682815}) 2024-09-11T01:48:30.638 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.595+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.638 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.595+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329189300}) 2024-09-11T01:48:30.638 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.595+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3628379775}) 2024-09-11T01:48:30.638 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.596+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.638 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.596+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431728373}) 2024-09-11T01:48:30.638 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.596+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2069134904}) 2024-09-11T01:48:30.638 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.596+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.638 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.596+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431728373}) 2024-09-11T01:48:30.639 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.596+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1677590796}) 2024-09-11T01:48:30.639 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.597+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.639 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.597+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3901524179}) 2024-09-11T01:48:30.639 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.597+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3107338225}) 2024-09-11T01:48:30.639 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.597+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.639 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.597+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3901524179}) 2024-09-11T01:48:30.639 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.597+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2632898698}) 2024-09-11T01:48:30.639 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.598+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.639 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.598+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696348869}) 2024-09-11T01:48:30.639 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.598+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3877836734}) 2024-09-11T01:48:30.639 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.598+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.639 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.598+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696348869}) 2024-09-11T01:48:30.640 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.598+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1562596021}) 2024-09-11T01:48:30.640 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.599+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.640 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.599+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2530793821}) 2024-09-11T01:48:30.640 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.599+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1352840600}) 2024-09-11T01:48:30.640 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.599+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.640 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.599+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2530793821}) 2024-09-11T01:48:30.640 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.599+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2523365328}) 2024-09-11T01:48:30.640 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.599+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.640 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.599+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2724663984}) 2024-09-11T01:48:30.640 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.600+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3016889958}) 2024-09-11T01:48:30.640 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.600+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.640 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.600+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2724663984}) 2024-09-11T01:48:30.641 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.600+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1317476354}) 2024-09-11T01:48:30.641 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.600+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.641 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.600+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2264745996}) 2024-09-11T01:48:30.641 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.600+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1974984152}) 2024-09-11T01:48:30.641 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.600+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.641 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.600+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2264745996}) 2024-09-11T01:48:30.641 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.600+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3444379154}) 2024-09-11T01:48:30.641 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.601+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.641 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.601+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1433104319}) 2024-09-11T01:48:30.641 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.601+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=96634216}) 2024-09-11T01:48:30.641 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.601+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.642 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.601+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1433104319}) 2024-09-11T01:48:30.642 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.601+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3731097215}) 2024-09-11T01:48:30.642 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.602+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.642 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.602+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1223851802}) 2024-09-11T01:48:30.642 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.602+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1527180435}) 2024-09-11T01:48:30.642 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.602+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.642 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.602+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1223851802}) 2024-09-11T01:48:30.642 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.602+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1252689281}) 2024-09-11T01:48:30.642 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.603+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.642 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.603+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1215041312}) 2024-09-11T01:48:30.642 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.603+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3344380338}) 2024-09-11T01:48:30.643 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.603+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.643 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.603+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1215041312}) 2024-09-11T01:48:30.643 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.603+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3246946119}) 2024-09-11T01:48:30.643 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.604+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.643 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.604+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2810229075}) 2024-09-11T01:48:30.643 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.604+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1701012931}) 2024-09-11T01:48:30.643 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.604+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.643 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.604+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2810229075}) 2024-09-11T01:48:30.643 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.604+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=778931547}) 2024-09-11T01:48:30.643 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.604+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.644 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.604+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3663506043}) 2024-09-11T01:48:30.644 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.604+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2643959073}) 2024-09-11T01:48:30.644 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.604+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.644 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.604+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3663506043}) 2024-09-11T01:48:30.644 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.604+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2531408930}) 2024-09-11T01:48:30.644 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.605+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.644 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.605+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=367394391}) 2024-09-11T01:48:30.644 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.605+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1587096168}) 2024-09-11T01:48:30.644 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.605+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.644 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.605+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=367394391}) 2024-09-11T01:48:30.644 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.605+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1470077318}) 2024-09-11T01:48:30.644 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.606+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.645 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.606+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1533633740}) 2024-09-11T01:48:30.645 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.606+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3676865533}) 2024-09-11T01:48:30.645 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.606+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.645 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.606+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1533633740}) 2024-09-11T01:48:30.645 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.606+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2884965139}) 2024-09-11T01:48:30.645 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.607+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.645 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.607+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=766230853}) 2024-09-11T01:48:30.645 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.607+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=256436546}) 2024-09-11T01:48:30.645 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.607+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.645 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.607+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=766230853}) 2024-09-11T01:48:30.646 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.607+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2076719047}) 2024-09-11T01:48:30.646 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.607+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.646 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.607+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=882493316}) 2024-09-11T01:48:30.646 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.607+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1281888333}) 2024-09-11T01:48:30.646 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.607+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.646 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.607+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=882493316}) 2024-09-11T01:48:30.646 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.607+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=926648492}) 2024-09-11T01:48:30.646 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.608+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.646 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.608+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1426067698}) 2024-09-11T01:48:30.646 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.608+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=385758468}) 2024-09-11T01:48:30.646 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.608+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.647 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.608+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1426067698}) 2024-09-11T01:48:30.647 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.608+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4088357680}) 2024-09-11T01:48:30.647 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.609+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.647 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.609+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4137610030}) 2024-09-11T01:48:30.647 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.609+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3205858927}) 2024-09-11T01:48:30.647 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.609+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.647 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.609+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4137610030}) 2024-09-11T01:48:30.647 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.609+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1462526093}) 2024-09-11T01:48:30.647 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.610+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.647 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.610+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1218147569}) 2024-09-11T01:48:30.647 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.610+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=599726244}) 2024-09-11T01:48:30.648 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.610+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.648 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.610+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1218147569}) 2024-09-11T01:48:30.648 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.610+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3579370255}) 2024-09-11T01:48:30.648 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.610+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.648 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.610+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1635414784}) 2024-09-11T01:48:30.648 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.610+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2198676005}) 2024-09-11T01:48:30.648 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.610+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.648 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.610+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1635414784}) 2024-09-11T01:48:30.648 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.611+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1370430123}) 2024-09-11T01:48:30.648 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.618+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.649 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.618+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1186026377}) 2024-09-11T01:48:30.649 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.618+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3537550243}) 2024-09-11T01:48:30.649 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.618+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.649 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.618+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1186026377}) 2024-09-11T01:48:30.649 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.618+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3793441889}) 2024-09-11T01:48:30.649 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.619+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.649 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.619+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1923140781}) 2024-09-11T01:48:30.649 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.619+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2920975132}) 2024-09-11T01:48:30.649 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.619+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.649 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.619+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1923140781}) 2024-09-11T01:48:30.649 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.619+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1822806135}) 2024-09-11T01:48:30.649 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.620+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.650 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.620+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843979737}) 2024-09-11T01:48:30.650 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.620+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2608006815}) 2024-09-11T01:48:30.650 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.620+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:30.650 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.620+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843979737}) 2024-09-11T01:48:30.650 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:30.620+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2807053004}) 2024-09-11T01:48:31.039 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.037+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:31.039 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.037+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1395875096}) 2024-09-11T01:48:31.039 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.037+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1707775760}) 2024-09-11T01:48:31.039 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.037+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:31.039 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.037+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1395875096}) 2024-09-11T01:48:31.039 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.037+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4211280332}) 2024-09-11T01:48:31.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.183+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:31.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.183+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4276734597}) 2024-09-11T01:48:31.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.183+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1828633830}) 2024-09-11T01:48:31.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.183+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:31.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.183+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4276734597}) 2024-09-11T01:48:31.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.183+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3523455100}) 2024-09-11T01:48:31.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.184+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:31.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.184+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3490336743}) 2024-09-11T01:48:31.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.184+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=598746722}) 2024-09-11T01:48:31.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.185+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:31.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.185+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3490336743}) 2024-09-11T01:48:31.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.185+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2373332694}) 2024-09-11T01:48:31.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.185+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:31.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.185+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1750757823}) 2024-09-11T01:48:31.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.185+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=209276043}) 2024-09-11T01:48:31.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.185+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:31.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.185+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1750757823}) 2024-09-11T01:48:31.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.185+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1318880486}) 2024-09-11T01:48:31.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.186+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:31.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.186+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4293799631}) 2024-09-11T01:48:31.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.186+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1925649037}) 2024-09-11T01:48:31.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.186+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:31.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.186+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4293799631}) 2024-09-11T01:48:31.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.186+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2302478215}) 2024-09-11T01:48:31.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.187+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:31.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.187+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1473923160}) 2024-09-11T01:48:31.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.187+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=163315694}) 2024-09-11T01:48:31.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.187+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:31.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.187+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1473923160}) 2024-09-11T01:48:31.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.187+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3563805676}) 2024-09-11T01:48:31.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.188+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:31.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.188+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3933995132}) 2024-09-11T01:48:31.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.188+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2510222683}) 2024-09-11T01:48:31.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.188+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:31.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.188+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3933995132}) 2024-09-11T01:48:31.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.188+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2932043447}) 2024-09-11T01:48:31.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.189+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:31.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.189+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4099464351}) 2024-09-11T01:48:31.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.189+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1752929571}) 2024-09-11T01:48:31.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.189+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:31.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.189+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4099464351}) 2024-09-11T01:48:31.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.189+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1139549415}) 2024-09-11T01:48:31.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.190+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:31.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.190+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=60,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=11,mon_config_key=16,monmap=1} crc {logm=3618779324,mdsmap=3760429986,mgr=4124675564,mgr_metadata=3338432716,mgrstat=3650483383,mon_config_key=3552379192,monmap=4199999934}) 2024-09-11T01:48:31.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.190+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=57,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=15,monmap=3,nvmeofgw=2,osd_metadata=2} crc {logm=3340805313,mdsmap=3760429986,mgr=1465092206,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1487211059,mon_config_key=3341644531,monmap=2933275274,nvmeofgw=3140514009,osd_metadata=1622464708}) 2024-09-11T01:48:31.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.190+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:31.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.190+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=60,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=11,mon_config_key=16,monmap=1} crc {logm=3618779324,mdsmap=3760429986,mgr=4124675564,mgr_metadata=3338432716,mgrstat=3650483383,mon_config_key=3552379192,monmap=4199999934}) 2024-09-11T01:48:31.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.190+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=60,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3} crc {logm=2660309740,mdsmap=161958762,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1579492110,mon_config_key=2466682877,monmap=2933275274}) 2024-09-11T01:48:31.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.191+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:31.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.191+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=88} crc {monmap=3140514009,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=411286724,osd_snap=2365507841}) 2024-09-11T01:48:31.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.191+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=88} crc {monmap=3140514009,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=411286724,osd_snap=4062934974}) 2024-09-11T01:48:31.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.191+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:31.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.191+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=88} crc {monmap=3140514009,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=411286724,osd_snap=2365507841}) 2024-09-11T01:48:31.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.191+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=88} crc {monmap=3140514009,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=411286724,osd_snap=4100543748}) 2024-09-11T01:48:31.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.192+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:31.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.192+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=44,osdmap=56} crc {osd_snap=3373304931,osdmap=4005906092}) 2024-09-11T01:48:31.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.192+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=44,osdmap=56} crc {osd_snap=2032718454,osdmap=2803123383}) 2024-09-11T01:48:31.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.192+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:31.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.192+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=44,osdmap=56} crc {osd_snap=3373304931,osdmap=4005906092}) 2024-09-11T01:48:31.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.192+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=44,osdmap=56} crc {osd_snap=931242530,osdmap=2673317110}) 2024-09-11T01:48:31.196 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.194+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:31.196 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.194+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4075077037}) 2024-09-11T01:48:31.196 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.194+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1351161973}) 2024-09-11T01:48:31.196 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.194+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:31.197 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.194+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4075077037}) 2024-09-11T01:48:31.197 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.194+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1848862313}) 2024-09-11T01:48:31.202 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.200+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:31.202 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.200+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=57} crc {osdmap=1913269161}) 2024-09-11T01:48:31.202 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.200+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=58} crc {osdmap=1875530155}) 2024-09-11T01:48:31.202 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.200+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:48:31.202 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.200+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=57} crc {osdmap=1913269161}) 2024-09-11T01:48:31.202 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:48:31.200+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=57} crc {osdmap=927955325}) 2024-09-11T01:48:31.330 INFO:tasks.rados.rados.0.smithi071.stdout:2838: done (2 left) 2024-09-11T01:48:31.330 INFO:tasks.rados.rados.0.smithi071.stdout:2839: copy_from oid 23 from oid 6 current snap is 322 2024-09-11T01:48:31.331 INFO:tasks.rados.rados.0.smithi071.stdout:2834: finishing write tid 1 to smithi07134777-32 2024-09-11T01:48:31.331 INFO:tasks.rados.rados.0.smithi071.stdout:2834: finishing write tid 2 to smithi07134777-32 2024-09-11T01:48:31.331 INFO:tasks.rados.rados.0.smithi071.stdout:2834: finishing write tid 3 to smithi07134777-32 2024-09-11T01:48:31.331 INFO:tasks.rados.rados.0.smithi071.stdout:2834: finishing write tid 4 to smithi07134777-32 2024-09-11T01:48:31.331 INFO:tasks.rados.rados.0.smithi071.stdout:2834: oid 32 updating version 0 to 1059 2024-09-11T01:48:31.331 INFO:tasks.rados.rados.0.smithi071.stdout:2834: oid 32 updating version 1059 to 1060 2024-09-11T01:48:31.331 INFO:tasks.rados.rados.0.smithi071.stdout:2834: oid 32 version 1060 is already newer than 1058 2024-09-11T01:48:31.331 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 1060 (ObjNum 999 snap 322 seq_num 999) dirty exists 2024-09-11T01:48:31.331 INFO:tasks.rados.rados.0.smithi071.stdout:2834: left oid 32 (ObjNum 999 snap 322 seq_num 999) 2024-09-11T01:48:31.331 INFO:tasks.rados.rados.0.smithi071.stdout:2836: expect (ObjNum 972 snap 305 seq_num 972) 2024-09-11T01:48:31.341 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:48:31.341 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:48:31.362 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 728514 2024-09-11T01:48:31.407 INFO:tasks.rados.rados.0.smithi071.stdout:2834: done (2 left) 2024-09-11T01:48:31.407 INFO:tasks.rados.rados.0.smithi071.stdout:2836: done (1 left) 2024-09-11T01:48:31.407 INFO:tasks.rados.rados.0.smithi071.stdout:2840: delete oid 10 current snap is 322 2024-09-11T01:48:31.407 INFO:tasks.rados.rados.0.smithi071.stdout:2839: finishing copy_from racing read to smithi07134777-23 2024-09-11T01:48:31.407 INFO:tasks.rados.rados.0.smithi071.stdout:2839: finishing copy_from to smithi07134777-23 2024-09-11T01:48:31.408 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 475 (ObjNum 601 snap 172 seq_num 601) dirty exists 2024-09-11T01:48:31.410 INFO:tasks.rados.rados.0.smithi071.stdout:2839: done (1 left) 2024-09-11T01:48:31.410 INFO:tasks.rados.rados.0.smithi071.stdout:2840: done (0 left) 2024-09-11T01:48:31.410 INFO:tasks.rados.rados.0.smithi071.stdout:2841: read oid 5 snap -1 2024-09-11T01:48:31.410 INFO:tasks.rados.rados.0.smithi071.stdout:2841: expect (ObjNum 886 snap 272 seq_num 886) 2024-09-11T01:48:31.410 INFO:tasks.rados.rados.0.smithi071.stdout:2842: delete oid 40 current snap is 322 2024-09-11T01:48:31.412 INFO:tasks.rados.rados.0.smithi071.stdout:2842: done (1 left) 2024-09-11T01:48:31.412 INFO:tasks.rados.rados.0.smithi071.stdout:2843: write oid 9 current snap is 322 2024-09-11T01:48:31.412 INFO:tasks.rados.rados.0.smithi071.stdout:2843: seq_num 1002 ranges {440614=641741,1734722=551343} 2024-09-11T01:48:31.418 INFO:tasks.rados.rados.0.smithi071.stdout:2843: writing smithi07134777-9 from 440614 to 1082355 tid 1 2024-09-11T01:48:31.424 INFO:tasks.rados.rados.0.smithi071.stdout:2843: writing smithi07134777-9 from 1734722 to 2286065 tid 2 2024-09-11T01:48:31.425 INFO:tasks.rados.rados.0.smithi071.stdout:2844: write oid 30 current snap is 322 2024-09-11T01:48:31.425 INFO:tasks.rados.rados.0.smithi071.stdout:2844: seq_num 1003 ranges {457421=716990,1676851=411354,2677497=179924} 2024-09-11T01:48:31.430 INFO:tasks.rados.rados.0.smithi071.stdout:2844: writing smithi07134777-30 from 457421 to 1174411 tid 1 2024-09-11T01:48:31.435 INFO:tasks.rados.rados.0.smithi071.stdout:2844: writing smithi07134777-30 from 1676851 to 2088205 tid 2 2024-09-11T01:48:31.440 INFO:tasks.rados.rados.0.smithi071.stdout:2844: writing smithi07134777-30 from 2677497 to 2857421 tid 3 2024-09-11T01:48:31.440 INFO:tasks.rados.rados.0.smithi071.stdout:2845: write (excl) oid 10 current snap is 322 2024-09-11T01:48:31.440 INFO:tasks.rados.rados.0.smithi071.stdout:2845: seq_num 1004 ranges {474228=508592,1451686=555012,2704920=169308} 2024-09-11T01:48:31.445 INFO:tasks.rados.rados.0.smithi071.stdout:2845: writing smithi07134777-10 from 474228 to 982820 tid 1 2024-09-11T01:48:31.450 INFO:tasks.rados.rados.0.smithi071.stdout:2845: writing smithi07134777-10 from 1451686 to 2006698 tid 2 2024-09-11T01:48:31.455 INFO:tasks.rados.rados.0.smithi071.stdout:2845: writing smithi07134777-10 from 2704920 to 2874228 tid 3 2024-09-11T01:48:31.455 INFO:tasks.rados.rados.0.smithi071.stdout:2846: write oid 36 current snap is 322 2024-09-11T01:48:31.456 INFO:tasks.rados.rados.0.smithi071.stdout:2846: seq_num 1005 ranges {491035=583841,1510168=698670,2732343=158692} 2024-09-11T01:48:31.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.456+0000 7fdfe4216640 10 monclient: get_monmap_and_config 2024-09-11T01:48:31.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.456+0000 7fdfe4216640 10 monclient: build_initial_monmap 2024-09-11T01:48:31.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.456+0000 7fdfe4216640 10 monclient: monmap: 2024-09-11T01:48:31.458 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:31.458 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:31.458 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:48:31.457330+0000 2024-09-11T01:48:31.458 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:48:31.457330+0000 2024-09-11T01:48:31.458 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:31.458 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:31.458 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:48:31.458 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:48:31.458 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:48:31.459 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:48:31.459 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:48:31.459 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:31.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.457+0000 7fdfe4216640 1 Processor -- start 2024-09-11T01:48:31.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.457+0000 7fdfe4216640 1 -- start start 2024-09-11T01:48:31.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.457+0000 7fdfe4216640 10 monclient: init 2024-09-11T01:48:31.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.457+0000 7fdfe4216640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:31.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.457+0000 7fdfe4216640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-11T01:48:31.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe4216640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdfdc138230 0x7fdfdc138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:31.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe4216640 10 monclient(hunting): picked mon.noname-a con 0x7fdfdc138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:48:31.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe4216640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdfdc09d030 0x7fdfdc0a5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:31.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe4216640 10 monclient(hunting): picked mon.noname-b con 0x7fdfdc09d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:31.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe4216640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdfdc09d460 addr v1:172.21.15.12:6789/0 2024-09-11T01:48:31.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe4216640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfdc0a6e30 con 0x7fdfdc09d460 2024-09-11T01:48:31.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe4216640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:31.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe4216640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fdfdc09c9c0 con 0x7fdfdc138230 2024-09-11T01:48:31.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe4216640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:31.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe4216640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fdfdc09ba20 con 0x7fdfdc09d030 2024-09-11T01:48:31.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe4216640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:31.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe4216640 10 monclient(hunting): authenticate will time out at 1917.900135s 2024-09-11T01:48:31.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe17b1640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdfdc09d030 0x7fdfdc0a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:31.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe17b1640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdfdc09d030 0x7fdfdc0a5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:44166/0 (socket says 172.21.15.12:44166) 2024-09-11T01:48:31.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe1fb2640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdfdc138230 0x7fdfdc138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:31.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe17b1640 1 -- 172.21.15.12:0/2402067866 learned_addr learned my addr 172.21.15.12:0/2402067866 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:48:31.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe17b1640 10 monclient(hunting): get_auth_request con 0x7fdfdc09d030 auth_method 0 2024-09-11T01:48:31.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe17b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:31.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe17b1640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:31.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe17b1640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:31.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe27b3640 1 --1- 172.21.15.12:0/2402067866 >> v1:172.21.15.12:6789/0 conn(0x7fdfdc09d460 0x7fdfdc0a5560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46196/0 (socket says 172.21.15.12:46196) 2024-09-11T01:48:31.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe1fb2640 10 monclient(hunting): get_auth_request con 0x7fdfdc138230 auth_method 0 2024-09-11T01:48:31.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe1fb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:31.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe1fb2640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:31.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe1fb2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:31.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe17b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:31.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe17b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:31.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe17b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:31.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe1fb2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:31.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe1fb2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:31.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe1fb2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:31.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe1fb2640 10 monclient(hunting): handle_auth_done global_id 214125 payload 274 2024-09-11T01:48:31.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe1fb2640 1 -- 172.21.15.12:0/2402067866 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdfdc09d030 msgr2=0x7fdfdc0a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:48:31.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe1fb2640 1 --2- 172.21.15.12:0/2402067866 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdfdc09d030 0x7fdfdc0a5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:31.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe1fb2640 1 -- 172.21.15.12:0/2402067866 >> v1:172.21.15.12:6789/0 conn(0x7fdfdc09d460 legacy=0x7fdfdc0a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:48:31.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe1fb2640 10 monclient: _finish_hunting 0 2024-09-11T01:48:31.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe1fb2640 1 monclient: found mon.noname-a 2024-09-11T01:48:31.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe1fb2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:31.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe1fb2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:48:31.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe1fb2640 1 -- 172.21.15.12:0/2402067866 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfdc0a66a0 con 0x7fdfdc138230 2024-09-11T01:48:31.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.458+0000 7fdfe17b1640 1 --2- 172.21.15.12:0/2402067866 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdfdc09d030 0x7fdfdc0a5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:48:31.465 INFO:tasks.rados.rados.0.smithi071.stdout:2846: writing smithi07134777-36 from 491035 to 1074876 tid 1 2024-09-11T01:48:31.470 INFO:tasks.rados.rados.0.smithi071.stdout:2846: writing smithi07134777-36 from 1510168 to 2208838 tid 2 2024-09-11T01:48:31.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.461+0000 7fdfe1fb2640 1 -- 172.21.15.12:0/2402067866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfdc138230 msgr2=0x7fdfdc138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:31.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.461+0000 7fdfe1fb2640 1 --2- 172.21.15.12:0/2402067866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfdc138230 0x7fdfdc138610 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc0a17e0 tx=0x7fdfd8029fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa136285950786bc server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:31.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.461+0000 7fdfe0fb0640 1 -- 172.21.15.12:0/2402067866 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd8037070 con 0x7fdfdc138230 2024-09-11T01:48:31.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.461+0000 7fdfe0fb0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:31.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.461+0000 7fdfe0fb0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:48:31.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.461+0000 7fdfe0fb0640 10 monclient: dump: 2024-09-11T01:48:31.472 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:31.472 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:31.472 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:31.472 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:31.472 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:31.472 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:31.472 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:31.472 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:31.472 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:31.472 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:31.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.461+0000 7fdfe0fb0640 10 monclient: _finish_auth 0 2024-09-11T01:48:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.461+0000 7fdfe0fb0640 10 monclient: _check_auth_tickets 2024-09-11T01:48:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.461+0000 7fdfe0fb0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.461+0000 7fdfe0fb0640 1 -- 172.21.15.12:0/2402067866 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfd8032440 con 0x7fdfdc138230 2024-09-11T01:48:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.461+0000 7fdfe4216640 5 monclient: authenticate success, global_id 214125 2024-09-11T01:48:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.461+0000 7fdfe4216640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:48:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.461+0000 7fdfe0fb0640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.461+0000 7fdfe4216640 10 monclient: get_monmap_and_config success 2024-09-11T01:48:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.461+0000 7fdfe4216640 4 set_mon_vals no callback set 2024-09-11T01:48:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.461+0000 7fdfe0fb0640 1 -- 172.21.15.12:0/2402067866 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd8032810 con 0x7fdfdc138230 2024-09-11T01:48:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.461+0000 7fdfe0fb0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.461+0000 7fdfe0fb0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:48:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.461+0000 7fdfe0fb0640 10 monclient: dump: 2024-09-11T01:48:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:31.473 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:31.474 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:31.474 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:31.474 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:31.474 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:31.474 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:31.474 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:31.474 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:31.474 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:31.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.462+0000 7fdfe4216640 10 monclient: shutdown 2024-09-11T01:48:31.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.462+0000 7fdfe4216640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:31.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.462+0000 7fdfe4216640 1 -- 172.21.15.12:0/2402067866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfdc138230 msgr2=0x7fdfdc138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:31.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.462+0000 7fdfe4216640 1 --2- 172.21.15.12:0/2402067866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfdc138230 0x7fdfdc138610 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc0a17e0 tx=0x7fdfd8029fb0 comp rx=0 tx=0).stop 2024-09-11T01:48:31.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.462+0000 7fdfe4216640 1 -- 172.21.15.12:0/2402067866 shutdown_connections 2024-09-11T01:48:31.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.462+0000 7fdfe4216640 1 --2- 172.21.15.12:0/2402067866 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdfdc09d030 0x7fdfdc0a5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.462+0000 7fdfe4216640 1 --2- 172.21.15.12:0/2402067866 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfdc138230 0x7fdfdc138610 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.462+0000 7fdfe4216640 1 -- 172.21.15.12:0/2402067866 >> 172.21.15.12:0/2402067866 conn(0x7fdfdc0a7120 msgr2=0x7fdfdc0a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.462+0000 7fdfe4216640 1 -- 172.21.15.12:0/2402067866 shutdown_connections 2024-09-11T01:48:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.462+0000 7fdfe4216640 1 -- 172.21.15.12:0/2402067866 wait complete. 2024-09-11T01:48:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.463+0000 7fdfe4216640 10 monclient: build_initial_monmap 2024-09-11T01:48:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.463+0000 7fdfe4216640 10 monclient: monmap: 2024-09-11T01:48:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:48:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:48:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:48:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:48:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:48:31.475 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:48:31.476 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.463+0000 7fdfe4216640 1 Processor -- start 2024-09-11T01:48:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.463+0000 7fdfe4216640 1 -- start start 2024-09-11T01:48:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.463+0000 7fdfe4216640 10 monclient: init 2024-09-11T01:48:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe4216640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe4216640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:48:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe4216640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfdc09d030 0x7fdfdc137240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe4216640 10 monclient(hunting): picked mon.noname-b con 0x7fdfdc09d030 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:48:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe4216640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdfdc09d460 0x7fdfdc137780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe4216640 10 monclient(hunting): picked mon.noname-c con 0x7fdfdc09d460 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe4216640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfdc138230 0x7fdfdc124cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe4216640 10 monclient(hunting): picked mon.noname-a con 0x7fdfdc138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:48:31.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe4216640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe4216640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc06f970 con 0x7fdfdc138230 2024-09-11T01:48:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe4216640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe4216640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc06f7f0 con 0x7fdfdc09d030 2024-09-11T01:48:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe4216640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe4216640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fdfdc0a5c70 con 0x7fdfdc09d460 2024-09-11T01:48:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe4216640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe4216640 10 monclient(hunting): authenticate will time out at 1917.906602s 2024-09-11T01:48:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe17b1640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdfdc09d460 0x7fdfdc137780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe17b1640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdfdc09d460 0x7fdfdc137780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:44170/0 (socket says 172.21.15.12:44170) 2024-09-11T01:48:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe17b1640 1 -- 172.21.15.12:0/1146817783 learned_addr learned my addr 172.21.15.12:0/1146817783 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:48:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe17b1640 10 monclient(hunting): get_auth_request con 0x7fdfdc09d460 auth_method 0 2024-09-11T01:48:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe17b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe17b1640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:31.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe17b1640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe27b3640 1 --2- 172.21.15.12:0/1146817783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfdc138230 0x7fdfdc124cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe1fb2640 1 --2- 172.21.15.12:0/1146817783 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfdc09d030 0x7fdfdc137240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe1fb2640 10 monclient(hunting): get_auth_request con 0x7fdfdc09d030 auth_method 0 2024-09-11T01:48:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe1fb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe1fb2640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe1fb2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe27b3640 10 monclient(hunting): get_auth_request con 0x7fdfdc138230 auth_method 0 2024-09-11T01:48:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe27b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe27b3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe27b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe17b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe17b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe17b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:31.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe1fb2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:31.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe1fb2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:31.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe1fb2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:31.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe27b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:31.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe27b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:31.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe27b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:31.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.464+0000 7fdfe17b1640 10 monclient(hunting): handle_auth_done global_id 184370 payload 824 2024-09-11T01:48:31.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.465+0000 7fdfe17b1640 1 -- 172.21.15.12:0/1146817783 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfdc09d030 msgr2=0x7fdfdc137240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:31.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.465+0000 7fdfe17b1640 1 --2- 172.21.15.12:0/1146817783 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfdc09d030 0x7fdfdc137240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:31.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.465+0000 7fdfe17b1640 1 -- 172.21.15.12:0/1146817783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfdc138230 msgr2=0x7fdfdc124cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:31.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.465+0000 7fdfe17b1640 1 --2- 172.21.15.12:0/1146817783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfdc138230 0x7fdfdc124cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:31.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.465+0000 7fdfe17b1640 10 monclient: _finish_hunting 0 2024-09-11T01:48:31.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.465+0000 7fdfe17b1640 1 monclient: found mon.noname-c 2024-09-11T01:48:31.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.465+0000 7fdfe17b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:31.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.465+0000 7fdfe17b1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:48:31.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.465+0000 7fdfe17b1640 1 -- 172.21.15.12:0/1146817783 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfdc1267b0 con 0x7fdfdc09d460 2024-09-11T01:48:31.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.465+0000 7fdfe27b3640 1 --2- 172.21.15.12:0/1146817783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfdc138230 0x7fdfdc124cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:48:31.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.465+0000 7fdfe1fb2640 1 --2- 172.21.15.12:0/1146817783 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfdc09d030 0x7fdfdc137240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:48:31.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.467+0000 7fdfe17b1640 1 -- 172.21.15.12:0/1146817783 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdfdc09d460 msgr2=0x7fdfdc137780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:31.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.467+0000 7fdfe17b1640 1 --2- 172.21.15.12:0/1146817783 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdfdc09d460 0x7fdfdc137780 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc00e9d0 tx=0x7fdfcc00ee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:31.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.467+0000 7fdfd27fc640 1 -- 172.21.15.12:0/1146817783 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc017070 con 0x7fdfdc09d460 2024-09-11T01:48:31.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.467+0000 7fdfd27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:31.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.467+0000 7fdfd27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:48:31.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.467+0000 7fdfd27fc640 10 monclient: dump: 2024-09-11T01:48:31.480 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:31.480 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:31.480 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:31.480 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:31.480 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:31.480 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:31.481 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:31.481 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:31.481 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:31.481 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:31.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.467+0000 7fdfd27fc640 10 monclient: _finish_auth 0 2024-09-11T01:48:31.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.467+0000 7fdfd27fc640 10 monclient: _check_auth_tickets 2024-09-11T01:48:31.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.467+0000 7fdfd27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:31.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.467+0000 7fdfd27fc640 1 -- 172.21.15.12:0/1146817783 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfcc00cc70 con 0x7fdfdc09d460 2024-09-11T01:48:31.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.467+0000 7fdfe4216640 5 monclient: authenticate success, global_id 184370 2024-09-11T01:48:31.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.467+0000 7fdfd27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:31.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.467+0000 7fdfd27fc640 1 -- 172.21.15.12:0/1146817783 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc013670 con 0x7fdfdc09d460 2024-09-11T01:48:31.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.467+0000 7fdfe4216640 10 monclient: _renew_subs 2024-09-11T01:48:31.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.467+0000 7fdfe4216640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:48:31.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.467+0000 7fdfe4216640 1 -- 172.21.15.12:0/1146817783 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfdc0a1810 con 0x7fdfdc09d460 2024-09-11T01:48:31.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.467+0000 7fdfd27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:31.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.467+0000 7fdfd27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:48:31.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.467+0000 7fdfd27fc640 10 monclient: dump: 2024-09-11T01:48:31.484 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:31.484 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:31.484 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:31.484 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:31.485 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:31.485 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:31.485 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:31.485 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:31.485 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:31.485 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:31.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.467+0000 7fdfd3fff640 4 set_mon_vals no callback set 2024-09-11T01:48:31.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.467+0000 7fdfe4216640 10 monclient: _renew_subs 2024-09-11T01:48:31.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.467+0000 7fdfe4216640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:48:31.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.467+0000 7fdfe4216640 1 -- 172.21.15.12:0/1146817783 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfdc128b10 con 0x7fdfdc09d460 2024-09-11T01:48:31.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.468+0000 7fdfd27fc640 1 -- 172.21.15.12:0/1146817783 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fdfcc024060 con 0x7fdfdc09d460 2024-09-11T01:48:31.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.469+0000 7fdfd27fc640 1 --2- 172.21.15.12:0/1146817783 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdfc003e240 0x7fdfc0040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:31.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.469+0000 7fdfd27fc640 1 -- 172.21.15.12:0/1146817783 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(642..642 src has 507..642) ==== 4831+0+0 (secure 0 0 0) 0x7fdfcc052ca0 con 0x7fdfdc09d460 2024-09-11T01:48:31.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.469+0000 7fdfe4216640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:31.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.469+0000 7fdfe4216640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:31.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.469+0000 7fdfe4216640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:48:31.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.469+0000 7fdfe4216640 1 -- 172.21.15.12:0/1146817783 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfdc09bd90 con 0x7fdfdc09d460 2024-09-11T01:48:31.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.469+0000 7fdfe1fb2640 1 --2- 172.21.15.12:0/1146817783 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdfc003e240 0x7fdfc0040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:31.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.469+0000 7fdfe1fb2640 10 monclient: get_auth_request con 0x7fdfc003e240 auth_method 0 2024-09-11T01:48:31.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.472+0000 7fdfe1fb2640 1 --2- 172.21.15.12:0/1146817783 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdfc003e240 0x7fdfc0040700 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8031000 tx=0x7fdfd802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:31.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.475+0000 7fdfd27fc640 1 -- 172.21.15.12:0/1146817783 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fdfcc00ce20 con 0x7fdfdc09d460 2024-09-11T01:48:31.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.475+0000 7fdfd27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:31.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.475+0000 7fdfd27fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:48:31.488 INFO:tasks.rados.rados.0.smithi071.stdout:2846: writing smithi07134777-36 from 2732343 to 2891035 tid 3 2024-09-11T01:48:31.488 INFO:tasks.rados.rados.0.smithi071.stdout:2847: write oid 13 current snap is 322 2024-09-11T01:48:31.489 INFO:tasks.rados.rados.0.smithi071.stdout:2847: seq_num 1006 ranges {507842=659090,1568650=558681,2759766=148076} 2024-09-11T01:48:31.489 INFO:tasks.rados.rados.0.smithi071.stdout:2847: writing smithi07134777-13 from 507842 to 1166932 tid 1 2024-09-11T01:48:31.489 INFO:tasks.rados.rados.0.smithi071.stdout:2847: writing smithi07134777-13 from 1568650 to 2127331 tid 2 2024-09-11T01:48:31.489 INFO:tasks.rados.rados.0.smithi071.stdout:2847: writing smithi07134777-13 from 2759766 to 2907842 tid 3 2024-09-11T01:48:31.489 INFO:tasks.rados.rados.0.smithi071.stdout:2848: read oid 46 snap -1 2024-09-11T01:48:31.489 INFO:tasks.rados.rados.0.smithi071.stdout:2848: expect (ObjNum 841 snap 255 seq_num 841) 2024-09-11T01:48:31.489 INFO:tasks.rados.rados.0.smithi071.stdout:2849: read oid 6 snap 294 2024-09-11T01:48:31.489 INFO:tasks.rados.rados.0.smithi071.stdout:2849: expect (ObjNum 926 snap 292 seq_num 926) 2024-09-11T01:48:31.489 INFO:tasks.rados.rados.0.smithi071.stdout:2850: delete oid 8 current snap is 322 2024-09-11T01:48:31.489 INFO:tasks.rados.rados.0.smithi071.stdout:2841: expect (ObjNum 886 snap 272 seq_num 886) 2024-09-11T01:48:31.501 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1384874 2024-09-11T01:48:31.540 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:48:31.654 INFO:tasks.rados.rados.0.smithi071.stdout:2843: finishing write tid 1 to smithi07134777-9 2024-09-11T01:48:31.654 INFO:tasks.rados.rados.0.smithi071.stdout:2843: finishing write tid 2 to smithi07134777-9 2024-09-11T01:48:31.654 INFO:tasks.rados.rados.0.smithi071.stdout:2843: finishing write tid 3 to smithi07134777-9 2024-09-11T01:48:31.654 INFO:tasks.rados.rados.0.smithi071.stdout:2843: finishing write tid 4 to smithi07134777-9 2024-09-11T01:48:31.654 INFO:tasks.rados.rados.0.smithi071.stdout:2843: oid 9 updating version 0 to 1014 2024-09-11T01:48:31.654 INFO:tasks.rados.rados.0.smithi071.stdout:2843: oid 9 updating version 1014 to 1016 2024-09-11T01:48:31.654 INFO:tasks.rados.rados.0.smithi071.stdout:2843: oid 9 version 1016 is already newer than 1015 2024-09-11T01:48:31.654 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 1016 (ObjNum 1002 snap 322 seq_num 1002) dirty exists 2024-09-11T01:48:31.654 INFO:tasks.rados.rados.0.smithi071.stdout:2843: left oid 9 (ObjNum 1002 snap 322 seq_num 1002) 2024-09-11T01:48:31.654 INFO:tasks.rados.rados.0.smithi071.stdout:2844: finishing write tid 1 to smithi07134777-30 2024-09-11T01:48:31.655 INFO:tasks.rados.rados.0.smithi071.stdout:2841: done (8 left) 2024-09-11T01:48:31.655 INFO:tasks.rados.rados.0.smithi071.stdout:2843: done (7 left) 2024-09-11T01:48:31.655 INFO:tasks.rados.rados.0.smithi071.stdout:2850: done (6 left) 2024-09-11T01:48:31.655 INFO:tasks.rados.rados.0.smithi071.stdout:2851: write (excl) oid 17 current snap is 322 2024-09-11T01:48:31.655 INFO:tasks.rados.rados.0.smithi071.stdout:2844: finishing write tid 2 to smithi07134777-30 2024-09-11T01:48:31.655 INFO:tasks.rados.rados.0.smithi071.stdout:2844: finishing write tid 3 to smithi07134777-30 2024-09-11T01:48:31.655 INFO:tasks.rados.rados.0.smithi071.stdout:2844: finishing write tid 4 to smithi07134777-30 2024-09-11T01:48:31.655 INFO:tasks.rados.rados.0.smithi071.stdout:2851: seq_num 1008 ranges {541456=409588,1569261=562350,2814612=126844} 2024-09-11T01:48:31.658 INFO:tasks.rados.rados.0.smithi071.stdout:2851: writing smithi07134777-17 from 541456 to 951044 tid 1 2024-09-11T01:48:31.665 INFO:tasks.rados.rados.0.smithi071.stdout:2851: writing smithi07134777-17 from 1569261 to 2131611 tid 2 2024-09-11T01:48:31.671 INFO:tasks.rados.rados.0.smithi071.stdout:2851: writing smithi07134777-17 from 2814612 to 2941456 tid 3 2024-09-11T01:48:31.671 INFO:tasks.rados.rados.0.smithi071.stdout:2852: copy_from oid 21 from oid 24 current snap is 322 2024-09-11T01:48:31.671 INFO:tasks.rados.rados.0.smithi071.stdout:2844: finishing write tid 5 to smithi07134777-30 2024-09-11T01:48:31.671 INFO:tasks.rados.rados.0.smithi071.stdout:2844: oid 30 updating version 0 to 1298 2024-09-11T01:48:31.671 INFO:tasks.rados.rados.0.smithi071.stdout:2844: oid 30 updating version 1298 to 1299 2024-09-11T01:48:31.671 INFO:tasks.rados.rados.0.smithi071.stdout:2844: oid 30 version 1299 is already newer than 1297 2024-09-11T01:48:31.671 INFO:tasks.rados.rados.0.smithi071.stdout:2844: oid 30 updating version 1299 to 1300 2024-09-11T01:48:31.671 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 1300 (ObjNum 1003 snap 322 seq_num 1003) dirty exists 2024-09-11T01:48:31.672 INFO:tasks.rados.rados.0.smithi071.stdout:2844: left oid 30 (ObjNum 1003 snap 322 seq_num 1003) 2024-09-11T01:48:31.672 INFO:tasks.rados.rados.0.smithi071.stdout:2845: finishing write tid 1 to smithi07134777-10 2024-09-11T01:48:31.672 INFO:tasks.rados.rados.0.smithi071.stdout:2845: finishing write tid 2 to smithi07134777-10 2024-09-11T01:48:31.672 INFO:tasks.rados.rados.0.smithi071.stdout:2845: finishing write tid 3 to smithi07134777-10 2024-09-11T01:48:31.672 INFO:tasks.rados.rados.0.smithi071.stdout:2845: finishing write tid 4 to smithi07134777-10 2024-09-11T01:48:31.672 INFO:tasks.rados.rados.0.smithi071.stdout:2845: finishing write tid 5 to smithi07134777-10 2024-09-11T01:48:31.672 INFO:tasks.rados.rados.0.smithi071.stdout:2845: oid 10 updating version 0 to 527 2024-09-11T01:48:31.672 INFO:tasks.rados.rados.0.smithi071.stdout:2845: oid 10 version 527 is already newer than 526 2024-09-11T01:48:31.672 INFO:tasks.rados.rados.0.smithi071.stdout:2845: oid 10 updating version 527 to 528 2024-09-11T01:48:31.672 INFO:tasks.rados.rados.0.smithi071.stdout:2845: oid 10 version 528 is already newer than 525 2024-09-11T01:48:31.672 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 528 (ObjNum 1004 snap 322 seq_num 1004) dirty exists 2024-09-11T01:48:31.672 INFO:tasks.rados.rados.0.smithi071.stdout:2845: left oid 10 (ObjNum 1004 snap 322 seq_num 1004) 2024-09-11T01:48:31.672 INFO:tasks.rados.rados.0.smithi071.stdout:2846: finishing write tid 1 to smithi07134777-36 2024-09-11T01:48:31.673 INFO:tasks.rados.rados.0.smithi071.stdout:2846: finishing write tid 2 to smithi07134777-36 2024-09-11T01:48:31.673 INFO:tasks.rados.rados.0.smithi071.stdout:2846: finishing write tid 3 to smithi07134777-36 2024-09-11T01:48:31.673 INFO:tasks.rados.rados.0.smithi071.stdout:2846: finishing write tid 4 to smithi07134777-36 2024-09-11T01:48:31.673 INFO:tasks.rados.rados.0.smithi071.stdout:2844: done (7 left) 2024-09-11T01:48:31.673 INFO:tasks.rados.rados.0.smithi071.stdout:2845: done (6 left) 2024-09-11T01:48:31.673 INFO:tasks.rados.rados.0.smithi071.stdout:2853: copy_from oid 10 from oid 19 current snap is 322 2024-09-11T01:48:31.673 INFO:tasks.rados.rados.0.smithi071.stdout:2846: finishing write tid 5 to smithi07134777-36 2024-09-11T01:48:31.673 INFO:tasks.rados.rados.0.smithi071.stdout:2846: oid 36 updating version 0 to 1077 2024-09-11T01:48:31.673 INFO:tasks.rados.rados.0.smithi071.stdout:2846: oid 36 updating version 1077 to 1078 2024-09-11T01:48:31.673 INFO:tasks.rados.rados.0.smithi071.stdout:2846: oid 36 updating version 1078 to 1079 2024-09-11T01:48:31.673 INFO:tasks.rados.rados.0.smithi071.stdout:2846: oid 36 updating version 1079 to 1080 2024-09-11T01:48:31.673 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 1080 (ObjNum 1005 snap 322 seq_num 1005) dirty exists 2024-09-11T01:48:31.673 INFO:tasks.rados.rados.0.smithi071.stdout:2846: left oid 36 (ObjNum 1005 snap 322 seq_num 1005) 2024-09-11T01:48:31.673 INFO:tasks.rados.rados.0.smithi071.stdout:2847: finishing write tid 1 to smithi07134777-13 2024-09-11T01:48:31.674 INFO:tasks.rados.rados.0.smithi071.stdout:2847: finishing write tid 2 to smithi07134777-13 2024-09-11T01:48:31.674 INFO:tasks.rados.rados.0.smithi071.stdout:2847: finishing write tid 3 to smithi07134777-13 2024-09-11T01:48:31.674 INFO:tasks.rados.rados.0.smithi071.stdout:2847: finishing write tid 4 to smithi07134777-13 2024-09-11T01:48:31.674 INFO:tasks.rados.rados.0.smithi071.stdout:2847: finishing write tid 5 to smithi07134777-13 2024-09-11T01:48:31.674 INFO:tasks.rados.rados.0.smithi071.stdout:2847: oid 13 updating version 0 to 1764 2024-09-11T01:48:31.674 INFO:tasks.rados.rados.0.smithi071.stdout:2847: oid 13 updating version 1764 to 1765 2024-09-11T01:48:31.674 INFO:tasks.rados.rados.0.smithi071.stdout:2847: oid 13 updating version 1765 to 1766 2024-09-11T01:48:31.674 INFO:tasks.rados.rados.0.smithi071.stdout:2847: oid 13 updating version 1766 to 1767 2024-09-11T01:48:31.674 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 1767 (ObjNum 1006 snap 322 seq_num 1006) dirty exists 2024-09-11T01:48:31.674 INFO:tasks.rados.rados.0.smithi071.stdout:2847: left oid 13 (ObjNum 1006 snap 322 seq_num 1006) 2024-09-11T01:48:31.674 INFO:tasks.rados.rados.0.smithi071.stdout:2846: done (6 left) 2024-09-11T01:48:31.674 INFO:tasks.rados.rados.0.smithi071.stdout:2847: done (5 left) 2024-09-11T01:48:31.674 INFO:tasks.rados.rados.0.smithi071.stdout:2854: write oid 44 current snap is 322 2024-09-11T01:48:31.674 INFO:tasks.rados.rados.0.smithi071.stdout:2854: seq_num 1009 ranges {558263=484837,1627743=706008} 2024-09-11T01:48:31.675 INFO:tasks.rados.rados.0.smithi071.stdout:2854: writing smithi07134777-44 from 558263 to 1043100 tid 1 2024-09-11T01:48:31.684 INFO:tasks.rados.rados.0.smithi071.stdout:2854: writing smithi07134777-44 from 1627743 to 2333751 tid 2 2024-09-11T01:48:31.684 INFO:tasks.rados.rados.0.smithi071.stdout:2855: write (excl) oid 18 current snap is 322 2024-09-11T01:48:31.684 INFO:tasks.rados.rados.0.smithi071.stdout:2848: expect (ObjNum 841 snap 255 seq_num 841) 2024-09-11T01:48:31.684 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 204577 2024-09-11T01:48:31.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.744+0000 7fdfe4216640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:48:31.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.744+0000 7fdfe4216640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:48:31.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.744+0000 7fdfe4216640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:48:31.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.744+0000 7fdfe4216640 1 -- 172.21.15.12:0/1146817783 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdfdc138610 con 0x7fdfdc09d460 2024-09-11T01:48:31.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.744+0000 7fdfd27fc640 1 -- 172.21.15.12:0/1146817783 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fdfcc013d30 con 0x7fdfdc09d460 2024-09-11T01:48:31.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.744+0000 7fdfd27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:48:31.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.744+0000 7fdfd27fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:48:31.889 INFO:tasks.rados.rados.0.smithi071.stdout:2849: expect (ObjNum 926 snap 292 seq_num 926) 2024-09-11T01:48:31.889 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:48:31.889 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":182,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:48:31.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.747+0000 7fdfe4216640 1 -- 172.21.15.12:0/1146817783 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdfc003e240 msgr2=0x7fdfc0040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:31.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.747+0000 7fdfe4216640 1 --2- 172.21.15.12:0/1146817783 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdfc003e240 0x7fdfc0040700 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8031000 tx=0x7fdfd802e000 comp rx=0 tx=0).stop 2024-09-11T01:48:31.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.747+0000 7fdfe4216640 10 monclient: shutdown 2024-09-11T01:48:31.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.747+0000 7fdfe4216640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:31.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.747+0000 7fdfe4216640 1 -- 172.21.15.12:0/1146817783 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdfdc09d460 msgr2=0x7fdfdc137780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:31.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.747+0000 7fdfe4216640 1 --2- 172.21.15.12:0/1146817783 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdfdc09d460 0x7fdfdc137780 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc00e9d0 tx=0x7fdfcc00ee90 comp rx=0 tx=0).stop 2024-09-11T01:48:31.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.747+0000 7fdfe4216640 1 -- 172.21.15.12:0/1146817783 shutdown_connections 2024-09-11T01:48:31.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.747+0000 7fdfe4216640 1 --2- 172.21.15.12:0/1146817783 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdfc003e240 0x7fdfc0040700 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:31.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.747+0000 7fdfe4216640 1 --2- 172.21.15.12:0/1146817783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfdc138230 0x7fdfdc124cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:31.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.747+0000 7fdfe4216640 1 --2- 172.21.15.12:0/1146817783 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdfdc09d460 0x7fdfdc137780 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:31.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.747+0000 7fdfe4216640 1 --2- 172.21.15.12:0/1146817783 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfdc09d030 0x7fdfdc137240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:31.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.747+0000 7fdfe4216640 1 -- 172.21.15.12:0/1146817783 >> 172.21.15.12:0/1146817783 conn(0x7fdfdc0a7120 msgr2=0x7fdfdc0a3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:31.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.747+0000 7fdfe4216640 1 -- 172.21.15.12:0/1146817783 shutdown_connections 2024-09-11T01:48:31.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.747+0000 7fdfe4216640 1 -- 172.21.15.12:0/1146817783 wait complete. 2024-09-11T01:48:31.892 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:48:31.892 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:48:31.892 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:48:31.892 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:48:31.897 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:48:31.949 INFO:tasks.rados.rados.0.smithi071.stdout:2851: finishing write tid 1 to smithi07134777-17 2024-09-11T01:48:31.950 INFO:tasks.rados.rados.0.smithi071.stdout:2851: finishing write tid 2 to smithi07134777-17 2024-09-11T01:48:31.950 INFO:tasks.rados.rados.0.smithi071.stdout:2851: finishing write tid 3 to smithi07134777-17 2024-09-11T01:48:31.950 INFO:tasks.rados.rados.0.smithi071.stdout:2851: finishing write tid 4 to smithi07134777-17 2024-09-11T01:48:31.950 INFO:tasks.rados.rados.0.smithi071.stdout:2851: finishing write tid 5 to smithi07134777-17 2024-09-11T01:48:31.950 INFO:tasks.rados.rados.0.smithi071.stdout:2851: oid 17 updating version 0 to 1769 2024-09-11T01:48:31.950 INFO:tasks.rados.rados.0.smithi071.stdout:2851: oid 17 updating version 1769 to 1771 2024-09-11T01:48:31.950 INFO:tasks.rados.rados.0.smithi071.stdout:2851: oid 17 version 1771 is already newer than 1770 2024-09-11T01:48:31.950 INFO:tasks.rados.rados.0.smithi071.stdout:2851: oid 17 version 1771 is already newer than 1768 2024-09-11T01:48:31.950 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 1771 (ObjNum 1008 snap 322 seq_num 1008) dirty exists 2024-09-11T01:48:31.950 INFO:tasks.rados.rados.0.smithi071.stdout:2851: left oid 17 (ObjNum 1008 snap 322 seq_num 1008) 2024-09-11T01:48:31.950 INFO:tasks.rados.rados.0.smithi071.stdout:2853: finishing copy_from racing read to smithi07134777-10 2024-09-11T01:48:31.950 INFO:tasks.rados.rados.0.smithi071.stdout:2852: finishing copy_from racing read to smithi07134777-21 2024-09-11T01:48:31.950 INFO:tasks.rados.rados.0.smithi071.stdout:2853: finishing copy_from to smithi07134777-10 2024-09-11T01:48:31.951 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 1076 (ObjNum 990 snap 317 seq_num 990) dirty exists 2024-09-11T01:48:31.951 INFO:tasks.rados.rados.0.smithi071.stdout:2854: finishing write tid 1 to smithi07134777-44 2024-09-11T01:48:31.951 INFO:tasks.rados.rados.0.smithi071.stdout:2854: finishing write tid 2 to smithi07134777-44 2024-09-11T01:48:31.951 INFO:tasks.rados.rados.0.smithi071.stdout:2855: seq_num 1010 ranges {575070=560086,1686225=566019,2869458=105612} 2024-09-11T01:48:31.955 INFO:tasks.rados.rados.0.smithi071.stdout:2855: writing smithi07134777-18 from 575070 to 1135156 tid 1 2024-09-11T01:48:31.960 INFO:tasks.rados.rados.0.smithi071.stdout:2855: writing smithi07134777-18 from 1686225 to 2252244 tid 2 2024-09-11T01:48:31.965 INFO:tasks.rados.rados.0.smithi071.stdout:2855: writing smithi07134777-18 from 2869458 to 2975070 tid 3 2024-09-11T01:48:31.965 INFO:tasks.rados.rados.0.smithi071.stdout:2848: done (6 left) 2024-09-11T01:48:31.965 INFO:tasks.rados.rados.0.smithi071.stdout:2849: done (5 left) 2024-09-11T01:48:31.965 INFO:tasks.rados.rados.0.smithi071.stdout:2851: done (4 left) 2024-09-11T01:48:31.965 INFO:tasks.rados.rados.0.smithi071.stdout:2853: done (3 left) 2024-09-11T01:48:31.965 INFO:tasks.rados.rados.0.smithi071.stdout:2856: snap_remove snap 306 2024-09-11T01:48:31.965 INFO:tasks.rados.rados.0.smithi071.stdout:2854: finishing write tid 3 to smithi07134777-44 2024-09-11T01:48:31.966 INFO:tasks.rados.rados.0.smithi071.stdout:2852: finishing copy_from to smithi07134777-21 2024-09-11T01:48:31.966 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 1070 (ObjNum 937 snap 293 seq_num 937) dirty exists 2024-09-11T01:48:31.966 INFO:tasks.rados.rados.0.smithi071.stdout:2854: finishing write tid 4 to smithi07134777-44 2024-09-11T01:48:31.966 INFO:tasks.rados.rados.0.smithi071.stdout:2854: oid 44 updating version 0 to 1050 2024-09-11T01:48:31.966 INFO:tasks.rados.rados.0.smithi071.stdout:2854: oid 44 updating version 1050 to 1051 2024-09-11T01:48:31.966 INFO:tasks.rados.rados.0.smithi071.stdout:2854: oid 44 version 1051 is already newer than 1049 2024-09-11T01:48:31.966 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 1051 (ObjNum 1009 snap 322 seq_num 1009) dirty exists 2024-09-11T01:48:31.966 INFO:tasks.rados.rados.0.smithi071.stdout:2854: left oid 44 (ObjNum 1009 snap 322 seq_num 1009) 2024-09-11T01:48:31.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.995+0000 7fd75b2cd640 10 monclient: get_monmap_and_config 2024-09-11T01:48:31.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.995+0000 7fd75b2cd640 10 monclient: build_initial_monmap 2024-09-11T01:48:31.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.995+0000 7fd75b2cd640 10 monclient: monmap: 2024-09-11T01:48:31.997 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:31.997 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:31.997 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:48:31.996522+0000 2024-09-11T01:48:31.997 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:48:31.996522+0000 2024-09-11T01:48:31.998 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:31.998 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:31.998 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:48:31.998 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:48:31.998 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:48:31.998 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:48:31.998 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:48:31.998 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.996+0000 7fd75b2cd640 1 Processor -- start 2024-09-11T01:48:32.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.996+0000 7fd75b2cd640 1 -- start start 2024-09-11T01:48:32.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.996+0000 7fd75b2cd640 10 monclient: init 2024-09-11T01:48:32.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.996+0000 7fd75b2cd640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:32.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd75b2cd640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-11T01:48:32.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd75b2cd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd7540a08c0 0x7fd754137be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd75b2cd640 10 monclient(hunting): picked mon.noname-b con 0x7fd7540a08c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:32.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd75b2cd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd754073ef0 addr v1:172.21.15.71:6789/0 2024-09-11T01:48:32.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd75b2cd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd754074330 0x7fd7540a5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd75b2cd640 10 monclient(hunting): picked mon.noname-c con 0x7fd754074330 addr v2:172.21.15.71:3300/0 2024-09-11T01:48:32.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd75b2cd640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd754138a80 con 0x7fd754073ef0 2024-09-11T01:48:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd75b2cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd75b2cd640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fd7540a06f0 con 0x7fd754074330 2024-09-11T01:48:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd75b2cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd75b2cd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd7540a0ca0 con 0x7fd7540a08c0 2024-09-11T01:48:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd75b2cd640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd75b2cd640 10 monclient(hunting): authenticate will time out at 1918.439097s 2024-09-11T01:48:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd759069640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd7540a08c0 0x7fd754137be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd759069640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd7540a08c0 0x7fd754137be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:44172/0 (socket says 172.21.15.12:44172) 2024-09-11T01:48:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd759069640 1 -- 172.21.15.12:0/4292978353 learned_addr learned my addr 172.21.15.12:0/4292978353 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:48:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd759069640 10 monclient(hunting): get_auth_request con 0x7fd7540a08c0 auth_method 0 2024-09-11T01:48:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd759069640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd759069640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:32.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd759069640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:32.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd759069640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:32.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd759069640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:32.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd75986a640 1 --2- 172.21.15.12:0/4292978353 >> v2:172.21.15.71:3300/0 conn(0x7fd754074330 0x7fd7540a5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd759069640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:32.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd75986a640 10 monclient(hunting): get_auth_request con 0x7fd754074330 auth_method 0 2024-09-11T01:48:32.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd75986a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:32.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd75986a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:32.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd75986a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:32.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd75986a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:32.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd75986a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:32.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd75986a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:32.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd759069640 10 monclient(hunting): handle_auth_done global_id 184373 payload 274 2024-09-11T01:48:32.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd759069640 1 -- 172.21.15.12:0/4292978353 >> v2:172.21.15.71:3300/0 conn(0x7fd754074330 msgr2=0x7fd7540a5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:48:32.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd759069640 1 --2- 172.21.15.12:0/4292978353 >> v2:172.21.15.71:3300/0 conn(0x7fd754074330 0x7fd7540a5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd759069640 1 -- 172.21.15.12:0/4292978353 >> v1:172.21.15.71:6789/0 conn(0x7fd754073ef0 legacy=0x7fd7540a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:48:32.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd759069640 10 monclient: _finish_hunting 0 2024-09-11T01:48:32.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd759069640 1 monclient: found mon.noname-b 2024-09-11T01:48:32.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd759069640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:32.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd759069640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:48:32.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:31.997+0000 7fd759069640 1 -- 172.21.15.12:0/4292978353 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd754070100 con 0x7fd7540a08c0 2024-09-11T01:48:32.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.000+0000 7fd759069640 1 -- 172.21.15.12:0/4292978353 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd7540a08c0 msgr2=0x7fd754137be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:32.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.000+0000 7fd759069640 1 --2- 172.21.15.12:0/4292978353 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd7540a08c0 0x7fd754137be0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd748009870 tx=0x7fd74802a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bb438b1bb1cde1e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:32.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.000+0000 7fd747fff640 1 -- 172.21.15.12:0/4292978353 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd748037070 con 0x7fd7540a08c0 2024-09-11T01:48:32.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.000+0000 7fd747fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:32.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.000+0000 7fd747fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:48:32.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.000+0000 7fd747fff640 10 monclient: dump: 2024-09-11T01:48:32.010 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:32.010 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:32.010 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.010 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.010 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:32.010 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.010 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:32.010 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:32.010 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:32.011 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.000+0000 7fd747fff640 10 monclient: _finish_auth 0 2024-09-11T01:48:32.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.000+0000 7fd747fff640 10 monclient: _check_auth_tickets 2024-09-11T01:48:32.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.000+0000 7fd747fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:32.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.000+0000 7fd747fff640 1 -- 172.21.15.12:0/4292978353 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd748032440 con 0x7fd7540a08c0 2024-09-11T01:48:32.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.000+0000 7fd747fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:32.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.000+0000 7fd747fff640 1 -- 172.21.15.12:0/4292978353 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd748032810 con 0x7fd7540a08c0 2024-09-11T01:48:32.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.000+0000 7fd75b2cd640 5 monclient: authenticate success, global_id 184373 2024-09-11T01:48:32.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.000+0000 7fd75b2cd640 10 monclient: get_monmap_and_config success 2024-09-11T01:48:32.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.000+0000 7fd75b2cd640 4 set_mon_vals no callback set 2024-09-11T01:48:32.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.000+0000 7fd747fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:32.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.000+0000 7fd747fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:48:32.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.000+0000 7fd747fff640 10 monclient: dump: 2024-09-11T01:48:32.011 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:32.012 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.000+0000 7fd75b2cd640 10 monclient: shutdown 2024-09-11T01:48:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.001+0000 7fd75b2cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.001+0000 7fd75b2cd640 1 -- 172.21.15.12:0/4292978353 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd7540a08c0 msgr2=0x7fd754137be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.001+0000 7fd75b2cd640 1 --2- 172.21.15.12:0/4292978353 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd7540a08c0 0x7fd754137be0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd748009870 tx=0x7fd74802a0a0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.001+0000 7fd75b2cd640 1 -- 172.21.15.12:0/4292978353 shutdown_connections 2024-09-11T01:48:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.001+0000 7fd75b2cd640 1 --2- 172.21.15.12:0/4292978353 >> v2:172.21.15.71:3300/0 conn(0x7fd754074330 0x7fd7540a5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.001+0000 7fd75b2cd640 1 --2- 172.21.15.12:0/4292978353 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd7540a08c0 0x7fd754137be0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.001+0000 7fd75b2cd640 1 -- 172.21.15.12:0/4292978353 >> 172.21.15.12:0/4292978353 conn(0x7fd754068950 msgr2=0x7fd754068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.001+0000 7fd75b2cd640 1 -- 172.21.15.12:0/4292978353 shutdown_connections 2024-09-11T01:48:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.001+0000 7fd75b2cd640 1 -- 172.21.15.12:0/4292978353 wait complete. 2024-09-11T01:48:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.001+0000 7fd75b2cd640 10 monclient: build_initial_monmap 2024-09-11T01:48:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.001+0000 7fd75b2cd640 10 monclient: monmap: 2024-09-11T01:48:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:48:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:48:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:48:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:32.013 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.014 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:48:32.014 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:48:32.014 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:48:32.014 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75b2cd640 1 Processor -- start 2024-09-11T01:48:32.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75b2cd640 1 -- start start 2024-09-11T01:48:32.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75b2cd640 10 monclient: init 2024-09-11T01:48:32.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75b2cd640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:32.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75b2cd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:48:32.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75b2cd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd754073ef0 0x7fd75413f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75b2cd640 10 monclient(hunting): picked mon.noname-b con 0x7fd754073ef0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:48:32.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75b2cd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd754074330 0x7fd7541312f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75b2cd640 10 monclient(hunting): picked mon.noname-c con 0x7fd754074330 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:32.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75b2cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7540a08c0 0x7fd754131830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75b2cd640 10 monclient(hunting): picked mon.noname-a con 0x7fd7540a08c0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:48:32.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75b2cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:32.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75b2cd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd7541382a0 con 0x7fd7540a08c0 2024-09-11T01:48:32.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75b2cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:32.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75b2cd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd754138120 con 0x7fd754073ef0 2024-09-11T01:48:32.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75b2cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:32.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75b2cd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd7540a62e0 con 0x7fd754074330 2024-09-11T01:48:32.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75b2cd640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:32.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75b2cd640 10 monclient(hunting): authenticate will time out at 1918.444677s 2024-09-11T01:48:32.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd758868640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd754074330 0x7fd7541312f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd759069640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd754073ef0 0x7fd75413f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75986a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7540a08c0 0x7fd754131830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd758868640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd754074330 0x7fd7541312f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:44184/0 (socket says 172.21.15.12:44184) 2024-09-11T01:48:32.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd758868640 1 -- 172.21.15.12:0/1084872094 learned_addr learned my addr 172.21.15.12:0/1084872094 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:48:32.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd758868640 10 monclient(hunting): get_auth_request con 0x7fd754074330 auth_method 0 2024-09-11T01:48:32.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd758868640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:32.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd758868640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:32.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd758868640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:32.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd759069640 10 monclient(hunting): get_auth_request con 0x7fd754073ef0 auth_method 0 2024-09-11T01:48:32.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd759069640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:32.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd759069640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:32.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd759069640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:32.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75986a640 10 monclient(hunting): get_auth_request con 0x7fd7540a08c0 auth_method 0 2024-09-11T01:48:32.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75986a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:32.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75986a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:32.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75986a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:32.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd758868640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:32.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd758868640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:32.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd758868640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:32.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd759069640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:32.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd759069640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:32.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd759069640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:32.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75986a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:32.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75986a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:32.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.002+0000 7fd75986a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:32.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.003+0000 7fd758868640 10 monclient(hunting): handle_auth_done global_id 184376 payload 824 2024-09-11T01:48:32.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.003+0000 7fd758868640 1 -- 172.21.15.12:0/1084872094 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd754073ef0 msgr2=0x7fd75413f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:32.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.003+0000 7fd758868640 1 --2- 172.21.15.12:0/1084872094 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd754073ef0 0x7fd75413f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.003+0000 7fd758868640 1 -- 172.21.15.12:0/1084872094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7540a08c0 msgr2=0x7fd754131830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:32.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.003+0000 7fd758868640 1 --2- 172.21.15.12:0/1084872094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7540a08c0 0x7fd754131830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.003+0000 7fd758868640 10 monclient: _finish_hunting 0 2024-09-11T01:48:32.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.003+0000 7fd758868640 1 monclient: found mon.noname-c 2024-09-11T01:48:32.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.003+0000 7fd758868640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:32.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.003+0000 7fd758868640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:48:32.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.003+0000 7fd758868640 1 -- 172.21.15.12:0/1084872094 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd75413cc30 con 0x7fd754074330 2024-09-11T01:48:32.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.003+0000 7fd759069640 1 --2- 172.21.15.12:0/1084872094 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd754073ef0 0x7fd75413f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:48:32.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.005+0000 7fd758868640 1 -- 172.21.15.12:0/1084872094 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd754074330 msgr2=0x7fd7541312f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:32.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.005+0000 7fd758868640 1 --2- 172.21.15.12:0/1084872094 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd754074330 0x7fd7541312f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd75000b3f0 tx=0x7fd75000b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:32.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.005+0000 7fd745ffb640 1 -- 172.21.15.12:0/1084872094 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd750015020 con 0x7fd754074330 2024-09-11T01:48:32.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.005+0000 7fd745ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:32.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.005+0000 7fd745ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:48:32.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.005+0000 7fd745ffb640 10 monclient: dump: 2024-09-11T01:48:32.020 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:32.020 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:32.020 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.020 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.020 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:32.020 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.020 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:32.020 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:32.020 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:32.020 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.005+0000 7fd745ffb640 10 monclient: _finish_auth 0 2024-09-11T01:48:32.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.005+0000 7fd745ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:48:32.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.005+0000 7fd745ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:32.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.005+0000 7fd75b2cd640 5 monclient: authenticate success, global_id 184376 2024-09-11T01:48:32.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.005+0000 7fd75b2cd640 10 monclient: _renew_subs 2024-09-11T01:48:32.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.005+0000 7fd75b2cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:48:32.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.005+0000 7fd75b2cd640 1 -- 172.21.15.12:0/1084872094 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd75413fc40 con 0x7fd754074330 2024-09-11T01:48:32.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.005+0000 7fd745ffb640 1 -- 172.21.15.12:0/1084872094 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd750011a40 con 0x7fd754074330 2024-09-11T01:48:32.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.005+0000 7fd745ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:32.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.005+0000 7fd745ffb640 1 -- 172.21.15.12:0/1084872094 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd750011e30 con 0x7fd754074330 2024-09-11T01:48:32.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.005+0000 7fd745ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:32.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.005+0000 7fd745ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:48:32.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.005+0000 7fd745ffb640 10 monclient: dump: 2024-09-11T01:48:32.021 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:32.021 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:32.022 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.022 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.022 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:32.022 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.022 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:32.022 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:32.022 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:32.022 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.005+0000 7fd746ffd640 4 set_mon_vals no callback set 2024-09-11T01:48:32.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.005+0000 7fd75b2cd640 10 monclient: _renew_subs 2024-09-11T01:48:32.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.005+0000 7fd75b2cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:48:32.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.005+0000 7fd75b2cd640 1 -- 172.21.15.12:0/1084872094 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd754131d70 con 0x7fd754074330 2024-09-11T01:48:32.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.006+0000 7fd75b2cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:32.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.006+0000 7fd75b2cd640 1 --2- 172.21.15.12:0/1084872094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7541ccf90 0x7fd7541cd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.006+0000 7fd75b2cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:32.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.006+0000 7fd75b2cd640 1 -- 172.21.15.12:0/1084872094 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd75413eef0 con 0x7fd7541ccf90 2024-09-11T01:48:32.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.006+0000 7fd759069640 1 --2- 172.21.15.12:0/1084872094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7541ccf90 0x7fd7541cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.006+0000 7fd759069640 10 monclient: get_auth_request con 0x7fd7541ccf90 auth_method 0 2024-09-11T01:48:32.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.006+0000 7fd759069640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:32.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.006+0000 7fd759069640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:32.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.006+0000 7fd759069640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:32.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.006+0000 7fd759069640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:32.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.006+0000 7fd759069640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:32.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.006+0000 7fd745ffb640 1 -- 172.21.15.12:0/1084872094 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fd75001a400 con 0x7fd754074330 2024-09-11T01:48:32.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.006+0000 7fd759069640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:32.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.006+0000 7fd759069640 10 monclient(hunting): handle_auth_done global_id 214134 payload 824 2024-09-11T01:48:32.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.006+0000 7fd759069640 1 -- 172.21.15.12:0/1084872094 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd754137be0 con 0x7fd7541ccf90 2024-09-11T01:48:32.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.007+0000 7fd745ffb640 1 --2- 172.21.15.12:0/1084872094 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd73803e210 0x7fd7380406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.007+0000 7fd745ffb640 1 -- 172.21.15.12:0/1084872094 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(642..642 src has 507..642) ==== 4831+0+0 (secure 0 0 0) 0x7fd750052cd0 con 0x7fd754074330 2024-09-11T01:48:32.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.007+0000 7fd75986a640 1 --2- 172.21.15.12:0/1084872094 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd73803e210 0x7fd7380406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.007+0000 7fd75986a640 10 monclient: get_auth_request con 0x7fd73803e210 auth_method 0 2024-09-11T01:48:32.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.009+0000 7fd759069640 1 -- 172.21.15.12:0/1084872094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7541ccf90 msgr2=0x7fd7541cd370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:32.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.009+0000 7fd759069640 1 --2- 172.21.15.12:0/1084872094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7541ccf90 0x7fd7541cd370 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd75406a2a0 tx=0x7fd7480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:32.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.009+0000 7fd745ffb640 1 -- 172.21.15.12:0/1084872094 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd748037040 con 0x7fd7541ccf90 2024-09-11T01:48:32.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.009+0000 7fd745ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:32.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.009+0000 7fd745ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:48:32.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.009+0000 7fd745ffb640 10 monclient: dump: 2024-09-11T01:48:32.026 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:32.026 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:32.027 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.027 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.027 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:32.027 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.027 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:32.027 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:32.027 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:32.027 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.009+0000 7fd745ffb640 1 -- 172.21.15.12:0/1084872094 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd748038040 con 0x7fd7541ccf90 2024-09-11T01:48:32.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.009+0000 7fd745ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:32.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.009+0000 7fd745ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:48:32.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.009+0000 7fd745ffb640 1 -- 172.21.15.12:0/1084872094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7541ccf90 msgr2=0x7fd7541cd370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:32.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.009+0000 7fd745ffb640 1 --2- 172.21.15.12:0/1084872094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7541ccf90 0x7fd7541cd370 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd75406a2a0 tx=0x7fd7480046c0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.009+0000 7fd745ffb640 1 -- 172.21.15.12:0/1084872094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7541ccf90 msgr2=0x7fd7541cd370 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:48:32.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.009+0000 7fd745ffb640 1 --2- 172.21.15.12:0/1084872094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7541ccf90 0x7fd7541cd370 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd75406a2a0 tx=0x7fd7480046c0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.009+0000 7fd75986a640 1 --2- 172.21.15.12:0/1084872094 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd73803e210 0x7fd7380406d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd75413d110 tx=0x7fd74c00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:32.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.015+0000 7fd75b2cd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:48:32.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.015+0000 7fd75b2cd640 1 --2- 172.21.15.12:0/1084872094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7541cd8b0 0x7fd754071680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.015+0000 7fd75b2cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:32.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.015+0000 7fd75b2cd640 1 -- 172.21.15.12:0/1084872094 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd75406d8c0 con 0x7fd7541cd8b0 2024-09-11T01:48:32.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.015+0000 7fd759069640 1 --2- 172.21.15.12:0/1084872094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7541cd8b0 0x7fd754071680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.015+0000 7fd759069640 10 monclient: get_auth_request con 0x7fd7541cd8b0 auth_method 0 2024-09-11T01:48:32.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.015+0000 7fd759069640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:32.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.015+0000 7fd759069640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:32.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.015+0000 7fd759069640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:32.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.016+0000 7fd759069640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:32.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.016+0000 7fd759069640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:32.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.016+0000 7fd759069640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:32.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.016+0000 7fd759069640 10 monclient(hunting): handle_auth_done global_id 214137 payload 824 2024-09-11T01:48:32.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.016+0000 7fd759069640 1 -- 172.21.15.12:0/1084872094 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd7541cdcc0 con 0x7fd7541cd8b0 2024-09-11T01:48:32.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.018+0000 7fd759069640 1 -- 172.21.15.12:0/1084872094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7541cd8b0 msgr2=0x7fd754071680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:32.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.018+0000 7fd759069640 1 --2- 172.21.15.12:0/1084872094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7541cd8b0 0x7fd754071680 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd74802a0e0 tx=0x7fd748008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:32.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.018+0000 7fd745ffb640 1 -- 172.21.15.12:0/1084872094 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd748037050 con 0x7fd7541cd8b0 2024-09-11T01:48:32.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.018+0000 7fd745ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:32.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.018+0000 7fd745ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:48:32.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.018+0000 7fd745ffb640 10 monclient: dump: 2024-09-11T01:48:32.029 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:32.029 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:32.029 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.029 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.029 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:32.030 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.030 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:32.030 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:32.030 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:32.030 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.018+0000 7fd745ffb640 1 -- 172.21.15.12:0/1084872094 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7fd748003820 con 0x7fd7541cd8b0 2024-09-11T01:48:32.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.018+0000 7fd745ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:48:32.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.018+0000 7fd745ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:48:32.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.018+0000 7fd745ffb640 1 -- 172.21.15.12:0/1084872094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7541cd8b0 msgr2=0x7fd754071680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:32.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.018+0000 7fd745ffb640 1 --2- 172.21.15.12:0/1084872094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7541cd8b0 0x7fd754071680 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd74802a0e0 tx=0x7fd748008030 comp rx=0 tx=0).stop 2024-09-11T01:48:32.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.018+0000 7fd745ffb640 1 -- 172.21.15.12:0/1084872094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7541cd8b0 msgr2=0x7fd754071680 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:48:32.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.018+0000 7fd745ffb640 1 --2- 172.21.15.12:0/1084872094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7541cd8b0 0x7fd754071680 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd74802a0e0 tx=0x7fd748008030 comp rx=0 tx=0).stop 2024-09-11T01:48:32.031 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:48:32.031 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:48:32.031 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:48:32.031 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:48:32.031 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 182, 2024-09-11T01:48:32.031 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 18218, 2024-09-11T01:48:32.031 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:48:32.031 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:48:32.031 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:48:32.031 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:48:32.031 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:32.031 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:48:32.032 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:48:32.032 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:48:32.032 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:48:32.032 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:48:32.032 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:48:32.032 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:48:32.032 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:48:32.032 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:48:32.032 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:48:32.032 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:48:32.032 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:48:32.032 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:48:32.032 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:48:32.032 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:48:32.033 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:32.033 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:48:32.033 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:48:32.033 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:48:32.033 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:48:32.033 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:48:32.033 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:48:32.033 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:48:32.033 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:48:32.033 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:48:32.033 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:48:32.033 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:48:32.033 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:48:32.033 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:48:32.034 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:32.034 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.034 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:48:32.034 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:48:32.034 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:48:32.034 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:48:32.034 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:48:32.034 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:48:32.034 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:48:32.034 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:48:32.034 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:48:32.034 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:48:32.034 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:48:32.035 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:48:32.035 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:48:32.035 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:48:32.035 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:48:32.035 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:48:32.035 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:48:32.035 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:48:32.035 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:48:32.035 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:48:32.035 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:48:32.035 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:48:32.035 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:48:32.035 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:48:32.035 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:48:32.035 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:48:32.036 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:48:32.036 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:48:32.036 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:32.036 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:48:32.036 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.036 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:48:32.036 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.037 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:48:32.037 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:48:32.037 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:48:32.037 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:48:32.037 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.037 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:48:32.037 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:48:32.037 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:32.038 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.038 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.038 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:48:32.038 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:48:32.038 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:32.038 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:32.038 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:32.038 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.038 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:48:32.038 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:48:32.038 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:48:32.038 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:48:32.038 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:48:32.038 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.039 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.039 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:48:32.039 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:48:32.039 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:48:32.039 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:48:32.039 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.039 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:48:32.039 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:48:32.039 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:32.039 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.039 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.039 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:48:32.039 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:48:32.039 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:32.040 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:32.040 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:32.040 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.040 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:48:32.040 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:48:32.040 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:48:32.040 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:48:32.040 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:48:32.040 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.040 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.040 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:48:32.040 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:48:32.040 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:48:32.041 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:48:32.041 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.041 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:48:32.041 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:48:32.041 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:32.041 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.041 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.041 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:48:32.041 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:48:32.041 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:32.041 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:32.041 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:32.041 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.041 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:48:32.042 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:48:32.042 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:48:32.042 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:48:32.042 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:48:32.042 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:32.042 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:32.042 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.042 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:48:32.042 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:48:32.042 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.042 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:48:32.042 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:48:32.042 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:48:32.042 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:32.043 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:32.043 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:48:32.043 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.043 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:48:32.043 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:48:32.043 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:48:32.043 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:32.043 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:32.043 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.043 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:48:32.043 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:48:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.019+0000 7fd75b2cd640 1 -- 172.21.15.12:0/1084872094 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd73803e210 msgr2=0x7fd7380406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.019+0000 7fd75b2cd640 1 --2- 172.21.15.12:0/1084872094 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd73803e210 0x7fd7380406d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd75413d110 tx=0x7fd74c00b040 comp rx=0 tx=0).stop 2024-09-11T01:48:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.019+0000 7fd75b2cd640 10 monclient: shutdown 2024-09-11T01:48:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.019+0000 7fd75b2cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.019+0000 7fd75b2cd640 1 -- 172.21.15.12:0/1084872094 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd754074330 msgr2=0x7fd7541312f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.019+0000 7fd75b2cd640 1 --2- 172.21.15.12:0/1084872094 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd754074330 0x7fd7541312f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd75000b3f0 tx=0x7fd75000b8b0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.019+0000 7fd75986a640 1 -- 172.21.15.12:0/1084872094 reap_dead start 2024-09-11T01:48:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.019+0000 7fd75986a640 1 -- 172.21.15.12:0/1084872094 reap_dead start 2024-09-11T01:48:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.019+0000 7fd75b2cd640 1 -- 172.21.15.12:0/1084872094 shutdown_connections 2024-09-11T01:48:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.019+0000 7fd75b2cd640 1 -- 172.21.15.12:0/1084872094 >> 172.21.15.12:0/1084872094 conn(0x7fd754068950 msgr2=0x7fd7540733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:32.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.019+0000 7fd75b2cd640 1 -- 172.21.15.12:0/1084872094 shutdown_connections 2024-09-11T01:48:32.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.019+0000 7fd75b2cd640 1 -- 172.21.15.12:0/1084872094 wait complete. 2024-09-11T01:48:32.046 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:48:32.046 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:48:32.046 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:48:32.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.146+0000 7f7c7e862640 10 monclient: get_monmap_and_config 2024-09-11T01:48:32.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.146+0000 7f7c7e862640 10 monclient: build_initial_monmap 2024-09-11T01:48:32.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.146+0000 7f7c7e862640 10 monclient: monmap: 2024-09-11T01:48:32.149 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:32.149 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:32.149 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:48:32.147608+0000 2024-09-11T01:48:32.149 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:48:32.147608+0000 2024-09-11T01:48:32.149 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:32.149 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.149 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:48:32.149 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:48:32.149 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:48:32.149 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:48:32.149 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:48:32.149 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.147+0000 7f7c7e862640 1 Processor -- start 2024-09-11T01:48:32.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.147+0000 7f7c7e862640 1 -- start start 2024-09-11T01:48:32.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.147+0000 7f7c7e862640 10 monclient: init 2024-09-11T01:48:32.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c7e862640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:32.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c7e862640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-11T01:48:32.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c7e862640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c78134ad0 0x7f7c78134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c7e862640 10 monclient(hunting): picked mon.noname-b con 0x7f7c78134ad0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:32.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c7e862640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7c780a2200 0x7f7c781353f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c7e862640 10 monclient(hunting): picked mon.noname-c con 0x7f7c780a2200 addr v2:172.21.15.71:3300/0 2024-09-11T01:48:32.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c7e862640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c78133a50 addr v1:172.21.15.12:6789/0 2024-09-11T01:48:32.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c7e862640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c7809ed30 con 0x7f7c78133a50 2024-09-11T01:48:32.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c7e862640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:32.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c7e862640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f7c7809f5d0 con 0x7f7c780a2200 2024-09-11T01:48:32.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c7e862640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:32.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c7e862640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7c78135930 con 0x7f7c78134ad0 2024-09-11T01:48:32.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c7e862640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:32.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c7e862640 10 monclient(hunting): authenticate will time out at 1918.590359s 2024-09-11T01:48:32.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c77fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c78134ad0 0x7f7c78134eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c77fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c78134ad0 0x7f7c78134eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:44188/0 (socket says 172.21.15.12:44188) 2024-09-11T01:48:32.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c7cdff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7c78133a50 0x7f7c78133f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46212/0 (socket says 172.21.15.12:46212) 2024-09-11T01:48:32.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c77fff640 1 -- 172.21.15.12:0/3317659176 learned_addr learned my addr 172.21.15.12:0/3317659176 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:48:32.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c77fff640 10 monclient(hunting): get_auth_request con 0x7f7c78134ad0 auth_method 0 2024-09-11T01:48:32.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:32.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c77fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:32.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:32.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c777fe640 1 --2- 172.21.15.12:0/3317659176 >> v2:172.21.15.71:3300/0 conn(0x7f7c780a2200 0x7f7c781353f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:32.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:32.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:32.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c777fe640 10 monclient(hunting): get_auth_request con 0x7f7c780a2200 auth_method 0 2024-09-11T01:48:32.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:32.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c777fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:32.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:32.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:32.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:32.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:32.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c77fff640 10 monclient(hunting): handle_auth_done global_id 184379 payload 274 2024-09-11T01:48:32.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c77fff640 1 -- 172.21.15.12:0/3317659176 >> v2:172.21.15.71:3300/0 conn(0x7f7c780a2200 msgr2=0x7f7c781353f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:48:32.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c77fff640 1 --2- 172.21.15.12:0/3317659176 >> v2:172.21.15.71:3300/0 conn(0x7f7c780a2200 0x7f7c781353f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c77fff640 1 -- 172.21.15.12:0/3317659176 >> v1:172.21.15.12:6789/0 conn(0x7f7c78133a50 legacy=0x7f7c78133f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:48:32.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c77fff640 10 monclient: _finish_hunting 0 2024-09-11T01:48:32.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c77fff640 1 monclient: found mon.noname-b 2024-09-11T01:48:32.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:32.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c77fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:48:32.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.148+0000 7f7c77fff640 1 -- 172.21.15.12:0/3317659176 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c7809f010 con 0x7f7c78134ad0 2024-09-11T01:48:32.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.151+0000 7f7c77fff640 1 -- 172.21.15.12:0/3317659176 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c78134ad0 msgr2=0x7f7c78134eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:32.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.151+0000 7f7c77fff640 1 --2- 172.21.15.12:0/3317659176 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c78134ad0 0x7f7c78134eb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7c68009870 tx=0x7f7c6802a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a532014bd720b8f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:32.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.151+0000 7f7c76ffd640 1 -- 172.21.15.12:0/3317659176 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c68037070 con 0x7f7c78134ad0 2024-09-11T01:48:32.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.152+0000 7f7c76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:32.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.152+0000 7f7c76ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:48:32.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.152+0000 7f7c76ffd640 10 monclient: dump: 2024-09-11T01:48:32.163 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:32.163 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:32.163 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.163 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.163 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:32.163 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.163 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:32.163 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:32.163 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:32.164 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.152+0000 7f7c76ffd640 10 monclient: _finish_auth 0 2024-09-11T01:48:32.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.152+0000 7f7c76ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:48:32.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.152+0000 7f7c76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:32.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.152+0000 7f7c7e862640 5 monclient: authenticate success, global_id 184379 2024-09-11T01:48:32.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.152+0000 7f7c76ffd640 1 -- 172.21.15.12:0/3317659176 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c68032440 con 0x7f7c78134ad0 2024-09-11T01:48:32.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.152+0000 7f7c76ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:32.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.152+0000 7f7c7e862640 10 monclient: get_monmap_and_config success 2024-09-11T01:48:32.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.152+0000 7f7c7e862640 4 set_mon_vals no callback set 2024-09-11T01:48:32.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.152+0000 7f7c76ffd640 1 -- 172.21.15.12:0/3317659176 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c68032810 con 0x7f7c78134ad0 2024-09-11T01:48:32.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.152+0000 7f7c76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:32.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.152+0000 7f7c76ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:48:32.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.152+0000 7f7c76ffd640 10 monclient: dump: 2024-09-11T01:48:32.165 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:32.165 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:32.165 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.165 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.165 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:32.165 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.165 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:32.165 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:32.165 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:32.165 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.152+0000 7f7c7e862640 10 monclient: shutdown 2024-09-11T01:48:32.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.152+0000 7f7c7e862640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:32.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.152+0000 7f7c7e862640 1 -- 172.21.15.12:0/3317659176 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c78134ad0 msgr2=0x7f7c78134eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:32.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.152+0000 7f7c7e862640 1 --2- 172.21.15.12:0/3317659176 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c78134ad0 0x7f7c78134eb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7c68009870 tx=0x7f7c6802a0a0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.152+0000 7f7c7e862640 1 -- 172.21.15.12:0/3317659176 shutdown_connections 2024-09-11T01:48:32.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.152+0000 7f7c7e862640 1 --2- 172.21.15.12:0/3317659176 >> v2:172.21.15.71:3300/0 conn(0x7f7c780a2200 0x7f7c781353f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.152+0000 7f7c7e862640 1 --2- 172.21.15.12:0/3317659176 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c78134ad0 0x7f7c78134eb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.152+0000 7f7c7e862640 1 -- 172.21.15.12:0/3317659176 >> 172.21.15.12:0/3317659176 conn(0x7f7c780a2f70 msgr2=0x7f7c780a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:32.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.152+0000 7f7c7e862640 1 -- 172.21.15.12:0/3317659176 shutdown_connections 2024-09-11T01:48:32.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.152+0000 7f7c7e862640 1 -- 172.21.15.12:0/3317659176 wait complete. 2024-09-11T01:48:32.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.153+0000 7f7c7e862640 10 monclient: build_initial_monmap 2024-09-11T01:48:32.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.153+0000 7f7c7e862640 10 monclient: monmap: 2024-09-11T01:48:32.166 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:32.166 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:48:32.166 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:48:32.166 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:48:32.166 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:32.166 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.167 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:48:32.167 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:48:32.167 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:48:32.167 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.153+0000 7f7c7e862640 1 Processor -- start 2024-09-11T01:48:32.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.153+0000 7f7c7e862640 1 -- start start 2024-09-11T01:48:32.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.153+0000 7f7c7e862640 10 monclient: init 2024-09-11T01:48:32.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7e862640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:32.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7e862640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:48:32.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7e862640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c780a2200 0x7f7c78132a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7e862640 10 monclient(hunting): picked mon.noname-b con 0x7f7c780a2200 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:48:32.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7e862640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c78133a50 0x7f7c78132fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7e862640 10 monclient(hunting): picked mon.noname-a con 0x7f7c78133a50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:48:32.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7e862640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c78134ad0 0x7f7c781217f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7e862640 10 monclient(hunting): picked mon.noname-c con 0x7f7c78134ad0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7e862640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7e862640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7c780a0150 con 0x7f7c78133a50 2024-09-11T01:48:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7e862640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7e862640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7809ffd0 con 0x7f7c780a2200 2024-09-11T01:48:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7e862640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7e862640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7c78134610 con 0x7f7c78134ad0 2024-09-11T01:48:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7e862640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7e862640 10 monclient(hunting): authenticate will time out at 1918.596278s 2024-09-11T01:48:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7cdff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c78134ad0 0x7f7c781217f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c777fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c78133a50 0x7f7c78132fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7cdff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c78134ad0 0x7f7c781217f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:44196/0 (socket says 172.21.15.12:44196) 2024-09-11T01:48:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7cdff640 1 -- 172.21.15.12:0/4039728454 learned_addr learned my addr 172.21.15.12:0/4039728454 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:48:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7cdff640 10 monclient(hunting): get_auth_request con 0x7f7c78134ad0 auth_method 0 2024-09-11T01:48:32.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7cdff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:32.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7cdff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:32.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7cdff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:32.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c777fe640 10 monclient(hunting): get_auth_request con 0x7f7c78133a50 auth_method 0 2024-09-11T01:48:32.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:32.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c777fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:32.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:32.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c77fff640 1 --2- 172.21.15.12:0/4039728454 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c780a2200 0x7f7c78132a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c77fff640 10 monclient(hunting): get_auth_request con 0x7f7c780a2200 auth_method 0 2024-09-11T01:48:32.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:32.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c77fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:32.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:32.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7cdff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:32.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7cdff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:32.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7cdff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:32.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:32.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:32.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:32.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:32.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:32.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:32.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7cdff640 10 monclient(hunting): handle_auth_done global_id 184382 payload 824 2024-09-11T01:48:32.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7cdff640 1 -- 172.21.15.12:0/4039728454 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c780a2200 msgr2=0x7f7c78132a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:32.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7cdff640 1 --2- 172.21.15.12:0/4039728454 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c780a2200 0x7f7c78132a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7cdff640 1 -- 172.21.15.12:0/4039728454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c78133a50 msgr2=0x7f7c78132fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:32.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7cdff640 1 --2- 172.21.15.12:0/4039728454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c78133a50 0x7f7c78132fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7cdff640 10 monclient: _finish_hunting 0 2024-09-11T01:48:32.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7cdff640 1 monclient: found mon.noname-c 2024-09-11T01:48:32.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7cdff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:32.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7cdff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:48:32.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.154+0000 7f7c7cdff640 1 -- 172.21.15.12:0/4039728454 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c781cbd80 con 0x7f7c78134ad0 2024-09-11T01:48:32.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.156+0000 7f7c7cdff640 1 -- 172.21.15.12:0/4039728454 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c78134ad0 msgr2=0x7f7c781217f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:32.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.156+0000 7f7c7cdff640 1 --2- 172.21.15.12:0/4039728454 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c78134ad0 0x7f7c781217f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7c7000c780 tx=0x7f7c7000cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:32.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.157+0000 7f7c74ff9640 1 -- 172.21.15.12:0/4039728454 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c70017070 con 0x7f7c78134ad0 2024-09-11T01:48:32.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.157+0000 7f7c74ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:32.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.157+0000 7f7c74ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:48:32.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.157+0000 7f7c74ff9640 10 monclient: dump: 2024-09-11T01:48:32.171 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:32.171 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:32.171 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.171 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.171 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:32.172 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.157+0000 7f7c74ff9640 10 monclient: _finish_auth 0 2024-09-11T01:48:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.157+0000 7f7c74ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:48:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.157+0000 7f7c74ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.157+0000 7f7c7e862640 5 monclient: authenticate success, global_id 184382 2024-09-11T01:48:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.157+0000 7f7c74ff9640 1 -- 172.21.15.12:0/4039728454 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c7000a860 con 0x7f7c78134ad0 2024-09-11T01:48:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.157+0000 7f7c74ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.157+0000 7f7c74ff9640 1 -- 172.21.15.12:0/4039728454 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c7000ac50 con 0x7f7c78134ad0 2024-09-11T01:48:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.157+0000 7f7c7e862640 10 monclient: _renew_subs 2024-09-11T01:48:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.157+0000 7f7c7e862640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:48:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.157+0000 7f7c7e862640 1 -- 172.21.15.12:0/4039728454 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c78133510 con 0x7f7c78134ad0 2024-09-11T01:48:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.157+0000 7f7c74ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.157+0000 7f7c74ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:48:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.157+0000 7f7c74ff9640 10 monclient: dump: 2024-09-11T01:48:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:32.173 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.157+0000 7f7c767fc640 4 set_mon_vals no callback set 2024-09-11T01:48:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.157+0000 7f7c7e862640 10 monclient: _renew_subs 2024-09-11T01:48:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.157+0000 7f7c7e862640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:48:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.157+0000 7f7c7e862640 1 -- 172.21.15.12:0/4039728454 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c78123190 con 0x7f7c78134ad0 2024-09-11T01:48:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.157+0000 7f7c7e862640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.158+0000 7f7c7e862640 1 --2- 172.21.15.12:0/4039728454 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c781cb0d0 0x7f7c7802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.158+0000 7f7c7e862640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.158+0000 7f7c7e862640 1 -- 172.21.15.12:0/4039728454 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7c78133f00 con 0x7f7c781cb0d0 2024-09-11T01:48:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.158+0000 7f7c77fff640 1 --2- 172.21.15.12:0/4039728454 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c781cb0d0 0x7f7c7802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.158+0000 7f7c77fff640 10 monclient: get_auth_request con 0x7f7c781cb0d0 auth_method 0 2024-09-11T01:48:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.158+0000 7f7c77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.158+0000 7f7c77fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.158+0000 7f7c77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.158+0000 7f7c77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.158+0000 7f7c77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.158+0000 7f7c77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.158+0000 7f7c74ff9640 1 -- 172.21.15.12:0/4039728454 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f7c70007070 con 0x7f7c78134ad0 2024-09-11T01:48:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.158+0000 7f7c77fff640 10 monclient(hunting): handle_auth_done global_id 184385 payload 824 2024-09-11T01:48:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.158+0000 7f7c77fff640 1 -- 172.21.15.12:0/4039728454 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7c78124cc0 con 0x7f7c781cb0d0 2024-09-11T01:48:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.158+0000 7f7c74ff9640 1 --2- 172.21.15.12:0/4039728454 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7c6403e210 0x7f7c640406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.158+0000 7f7c74ff9640 1 -- 172.21.15.12:0/4039728454 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(642..642 src has 507..642) ==== 4831+0+0 (secure 0 0 0) 0x7f7c70052750 con 0x7f7c78134ad0 2024-09-11T01:48:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.158+0000 7f7c777fe640 1 --2- 172.21.15.12:0/4039728454 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7c6403e210 0x7f7c640406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.158+0000 7f7c777fe640 10 monclient: get_auth_request con 0x7f7c6403e210 auth_method 0 2024-09-11T01:48:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.160+0000 7f7c77fff640 1 -- 172.21.15.12:0/4039728454 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c781cb0d0 msgr2=0x7f7c7802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.160+0000 7f7c77fff640 1 --2- 172.21.15.12:0/4039728454 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c781cb0d0 0x7f7c7802d440 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7c68034000 tx=0x7f7c680046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.160+0000 7f7c74ff9640 1 -- 172.21.15.12:0/4039728454 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c68043070 con 0x7f7c781cb0d0 2024-09-11T01:48:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.160+0000 7f7c74ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.160+0000 7f7c74ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:48:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.160+0000 7f7c74ff9640 10 monclient: dump: 2024-09-11T01:48:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:32.176 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.160+0000 7f7c74ff9640 1 -- 172.21.15.12:0/4039728454 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7c68048450 con 0x7f7c781cb0d0 2024-09-11T01:48:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.160+0000 7f7c74ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.160+0000 7f7c74ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:48:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.160+0000 7f7c74ff9640 1 -- 172.21.15.12:0/4039728454 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c781cb0d0 msgr2=0x7f7c7802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.160+0000 7f7c74ff9640 1 --2- 172.21.15.12:0/4039728454 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c781cb0d0 0x7f7c7802d440 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7c68034000 tx=0x7f7c680046c0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.160+0000 7f7c74ff9640 1 -- 172.21.15.12:0/4039728454 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c781cb0d0 msgr2=0x7f7c7802d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:48:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.160+0000 7f7c74ff9640 1 --2- 172.21.15.12:0/4039728454 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c781cb0d0 0x7f7c7802d440 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7c68034000 tx=0x7f7c680046c0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.161+0000 7f7c777fe640 1 --2- 172.21.15.12:0/4039728454 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7c6403e210 0x7f7c640406d0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7c780a1820 tx=0x7f7c6c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:32.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.167+0000 7f7c7e862640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:48:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.167+0000 7f7c7e862640 1 --2- 172.21.15.12:0/4039728454 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c78123ca0 0x7f7c7802d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.167+0000 7f7c7e862640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.167+0000 7f7c7e862640 1 -- 172.21.15.12:0/4039728454 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7c7811dff0 con 0x7f7c78123ca0 2024-09-11T01:48:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.167+0000 7f7c77fff640 1 --2- 172.21.15.12:0/4039728454 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c78123ca0 0x7f7c7802d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.167+0000 7f7c77fff640 10 monclient: get_auth_request con 0x7f7c78123ca0 auth_method 0 2024-09-11T01:48:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.167+0000 7f7c77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.167+0000 7f7c77fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.167+0000 7f7c77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.167+0000 7f7c77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.167+0000 7f7c77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.167+0000 7f7c77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.167+0000 7f7c77fff640 10 monclient(hunting): handle_auth_done global_id 184388 payload 824 2024-09-11T01:48:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.167+0000 7f7c77fff640 1 -- 172.21.15.12:0/4039728454 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7c78124fe0 con 0x7f7c78123ca0 2024-09-11T01:48:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.169+0000 7f7c77fff640 1 -- 172.21.15.12:0/4039728454 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c78123ca0 msgr2=0x7f7c7802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.169+0000 7f7c77fff640 1 --2- 172.21.15.12:0/4039728454 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c78123ca0 0x7f7c7802d980 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7c68000c00 tx=0x7f7c68008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.169+0000 7f7c74ff9640 1 -- 172.21.15.12:0/4039728454 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c68037040 con 0x7f7c78123ca0 2024-09-11T01:48:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.169+0000 7f7c74ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.169+0000 7f7c74ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:48:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.169+0000 7f7c74ff9640 10 monclient: dump: 2024-09-11T01:48:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:32.181 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.170+0000 7f7c74ff9640 1 -- 172.21.15.12:0/4039728454 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4542 (secure 0 0 0) 0x7f7c68003cf0 con 0x7f7c78123ca0 2024-09-11T01:48:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.170+0000 7f7c74ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:48:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.170+0000 7f7c74ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:48:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.170+0000 7f7c74ff9640 1 -- 172.21.15.12:0/4039728454 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c78123ca0 msgr2=0x7f7c7802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.170+0000 7f7c74ff9640 1 --2- 172.21.15.12:0/4039728454 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c78123ca0 0x7f7c7802d980 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7c68000c00 tx=0x7f7c68008030 comp rx=0 tx=0).stop 2024-09-11T01:48:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.170+0000 7f7c74ff9640 1 -- 172.21.15.12:0/4039728454 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c78123ca0 msgr2=0x7f7c7802d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:48:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.170+0000 7f7c74ff9640 1 --2- 172.21.15.12:0/4039728454 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c78123ca0 0x7f7c7802d980 secure :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7c68000c00 tx=0x7f7c68008030 comp rx=0 tx=0).stop 2024-09-11T01:48:32.181 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:48:32.181 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:48:32.182 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:48:32.182 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:48:32.182 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 182, 2024-09-11T01:48:32.182 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 172763, 2024-09-11T01:48:32.182 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:48:32.182 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:48:32.182 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:48:32.182 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:48:32.182 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:32.182 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:48:32.182 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:48:32.182 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:48:32.182 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:48:32.183 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:48:32.183 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:48:32.183 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:48:32.183 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:48:32.183 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:48:32.183 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:48:32.183 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:48:32.183 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:48:32.183 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:48:32.183 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:48:32.183 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:48:32.183 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:32.183 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:48:32.183 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:48:32.184 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:48:32.184 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:48:32.184 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:48:32.184 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:48:32.184 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:48:32.184 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:48:32.184 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:48:32.184 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:48:32.184 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:48:32.184 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:48:32.184 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:48:32.184 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:32.184 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.184 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:48:32.185 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:48:32.185 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:48:32.185 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:48:32.185 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:48:32.185 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:48:32.185 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:48:32.185 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:48:32.185 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:48:32.185 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:48:32.185 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:48:32.185 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:48:32.185 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:48:32.185 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:48:32.185 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:48:32.186 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:48:32.186 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:48:32.186 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:48:32.186 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:48:32.186 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:48:32.186 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:48:32.186 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:48:32.186 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:48:32.186 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:48:32.186 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:48:32.186 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:48:32.186 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:48:32.186 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:48:32.186 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:32.187 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:48:32.187 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.187 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:48:32.187 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.187 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:48:32.187 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:48:32.187 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:48:32.187 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:48:32.188 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.188 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:48:32.188 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:48:32.188 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:32.188 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.188 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.188 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:48:32.188 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:48:32.188 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:32.188 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:32.188 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:32.188 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.188 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:48:32.188 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:48:32.189 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:48:32.189 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:48:32.189 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:48:32.189 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.189 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.189 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:48:32.189 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:48:32.189 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:48:32.189 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:48:32.189 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.189 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:48:32.189 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:48:32.189 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:32.189 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.190 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.190 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:48:32.190 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:48:32.190 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:32.190 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:32.190 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:32.190 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.190 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:48:32.190 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:48:32.190 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:48:32.190 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:48:32.190 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:48:32.190 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.190 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.191 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:48:32.191 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:48:32.191 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:48:32.191 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:48:32.191 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.191 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:48:32.191 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:48:32.191 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:32.191 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.191 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.191 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:48:32.191 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:48:32.191 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:32.191 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:32.192 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:32.192 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.192 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:48:32.192 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:48:32.192 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:48:32.192 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:48:32.192 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:48:32.192 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:32.192 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:32.192 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.192 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:48:32.192 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:48:32.192 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.192 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:48:32.193 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:48:32.193 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:48:32.193 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:32.193 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:32.193 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:48:32.193 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.193 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:48:32.193 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:48:32.193 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:48:32.193 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:32.193 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:32.193 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.193 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:48:32.193 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:48:32.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.170+0000 7f7c7e862640 1 -- 172.21.15.12:0/4039728454 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7c6403e210 msgr2=0x7f7c640406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:32.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.170+0000 7f7c7e862640 1 --2- 172.21.15.12:0/4039728454 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7c6403e210 0x7f7c640406d0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7c780a1820 tx=0x7f7c6c009190 comp rx=0 tx=0).stop 2024-09-11T01:48:32.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.170+0000 7f7c7e862640 10 monclient: shutdown 2024-09-11T01:48:32.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.170+0000 7f7c7e862640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:32.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.170+0000 7f7c7e862640 1 -- 172.21.15.12:0/4039728454 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c78134ad0 msgr2=0x7f7c781217f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:32.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.170+0000 7f7c7e862640 1 --2- 172.21.15.12:0/4039728454 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c78134ad0 0x7f7c781217f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7c7000c780 tx=0x7f7c7000cc40 comp rx=0 tx=0).stop 2024-09-11T01:48:32.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.170+0000 7f7c7cdff640 1 -- 172.21.15.12:0/4039728454 reap_dead start 2024-09-11T01:48:32.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.170+0000 7f7c7e862640 1 -- 172.21.15.12:0/4039728454 shutdown_connections 2024-09-11T01:48:32.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.170+0000 7f7c7e862640 1 --2- 172.21.15.12:0/4039728454 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c78134ad0 0x7f7c781217f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.170+0000 7f7c7e862640 1 -- 172.21.15.12:0/4039728454 >> 172.21.15.12:0/4039728454 conn(0x7f7c780a2f70 msgr2=0x7f7c78073f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:32.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.170+0000 7f7c7e862640 1 -- 172.21.15.12:0/4039728454 shutdown_connections 2024-09-11T01:48:32.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.170+0000 7f7c7e862640 1 -- 172.21.15.12:0/4039728454 wait complete. 2024-09-11T01:48:32.196 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:48:32.196 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:48:32.196 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:48:32.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.295+0000 7f4fa2952640 10 monclient: get_monmap_and_config 2024-09-11T01:48:32.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.295+0000 7f4fa2952640 10 monclient: build_initial_monmap 2024-09-11T01:48:32.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.295+0000 7f4fa2952640 10 monclient: monmap: 2024-09-11T01:48:32.297 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:32.298 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:32.298 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:48:32.296638+0000 2024-09-11T01:48:32.298 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:48:32.296638+0000 2024-09-11T01:48:32.298 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:32.298 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.298 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:48:32.298 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:48:32.298 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:48:32.298 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:48:32.298 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:48:32.299 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.296+0000 7f4fa2952640 1 Processor -- start 2024-09-11T01:48:32.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.296+0000 7f4fa2952640 1 -- start start 2024-09-11T01:48:32.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.296+0000 7f4fa2952640 10 monclient: init 2024-09-11T01:48:32.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa2952640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:32.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa2952640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-11T01:48:32.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa2952640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4f9c0a2e20 addr v1:172.21.15.71:6789/0 2024-09-11T01:48:32.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa2952640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f9c0a4970 0x7f4f9c06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa2952640 10 monclient(hunting): picked mon.noname-b con 0x7f4f9c0a4970 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:32.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa2952640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4f9c06f910 0x7f4f9c13f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa2952640 10 monclient(hunting): picked mon.noname-a con 0x7f4f9c06f910 addr v2:172.21.15.12:3300/0 2024-09-11T01:48:32.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa2952640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f9c06f4f0 con 0x7f4f9c0a2e20 2024-09-11T01:48:32.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa2952640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:32.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa2952640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f4f9c06ef30 con 0x7f4f9c06f910 2024-09-11T01:48:32.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa2952640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:32.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa2952640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4f9c06ec50 con 0x7f4f9c0a4970 2024-09-11T01:48:32.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa2952640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:32.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa2952640 10 monclient(hunting): authenticate will time out at 1918.739281s 2024-09-11T01:48:32.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa0eef640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4f9c06f910 0x7f4f9c13f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4f9b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f9c0a4970 0x7f4f9c06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa0eef640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4f9c06f910 0x7f4f9c13f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36830/0 (socket says 172.21.15.12:36830) 2024-09-11T01:48:32.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4f9b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f9c0a4970 0x7f4f9c06ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:44208/0 (socket says 172.21.15.12:44208) 2024-09-11T01:48:32.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa0eef640 1 -- 172.21.15.12:0/3226816116 learned_addr learned my addr 172.21.15.12:0/3226816116 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:48:32.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa0eef640 10 monclient(hunting): get_auth_request con 0x7f4f9c06f910 auth_method 0 2024-09-11T01:48:32.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa0eef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:32.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa0eef640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:32.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa0eef640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:32.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4f9b7fe640 10 monclient(hunting): get_auth_request con 0x7f4f9c0a4970 auth_method 0 2024-09-11T01:48:32.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4f9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:32.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4f9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:32.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4f9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:32.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa0eef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:32.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa0eef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:32.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa0eef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:32.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4f9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:32.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4f9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:32.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4f9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:32.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa0eef640 10 monclient(hunting): handle_auth_done global_id 214143 payload 274 2024-09-11T01:48:32.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa0eef640 1 -- 172.21.15.12:0/3226816116 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f9c0a4970 msgr2=0x7f4f9c06ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:48:32.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa0eef640 1 --2- 172.21.15.12:0/3226816116 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f9c0a4970 0x7f4f9c06ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa0eef640 1 -- 172.21.15.12:0/3226816116 >> v1:172.21.15.71:6789/0 conn(0x7f4f9c0a2e20 legacy=0x7f4f9c0a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:48:32.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa0eef640 10 monclient: _finish_hunting 0 2024-09-11T01:48:32.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa0eef640 1 monclient: found mon.noname-a 2024-09-11T01:48:32.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa0eef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:32.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa0eef640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:48:32.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4fa0eef640 1 -- 172.21.15.12:0/3226816116 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f9c0a3910 con 0x7f4f9c06f910 2024-09-11T01:48:32.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.297+0000 7f4f9b7fe640 1 --2- 172.21.15.12:0/3226816116 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f9c0a4970 0x7f4f9c06ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:48:32.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.300+0000 7f4fa0eef640 1 -- 172.21.15.12:0/3226816116 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f9c06f910 msgr2=0x7f4f9c13f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:32.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.300+0000 7f4fa0eef640 1 --2- 172.21.15.12:0/3226816116 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f9c06f910 0x7f4f9c13f430 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4f94009e50 tx=0x7f4f9402a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b0c08c699f9d2ed server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:32.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.300+0000 7f4f9affd640 1 -- 172.21.15.12:0/3226816116 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f94036070 con 0x7f4f9c06f910 2024-09-11T01:48:32.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.300+0000 7f4f9affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:32.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.300+0000 7f4f9affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:48:32.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.300+0000 7f4f9affd640 10 monclient: dump: 2024-09-11T01:48:32.313 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:32.313 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:32.313 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.314 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.314 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:32.314 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.314 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:32.314 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:32.314 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:32.314 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.300+0000 7f4f9affd640 10 monclient: _finish_auth 0 2024-09-11T01:48:32.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.300+0000 7f4f9affd640 10 monclient: _check_auth_tickets 2024-09-11T01:48:32.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.300+0000 7f4f9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:32.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.300+0000 7f4f9affd640 1 -- 172.21.15.12:0/3226816116 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f94007800 con 0x7f4f9c06f910 2024-09-11T01:48:32.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.300+0000 7f4fa2952640 5 monclient: authenticate success, global_id 214143 2024-09-11T01:48:32.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.300+0000 7f4fa2952640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:48:32.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.300+0000 7f4f9affd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:32.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.300+0000 7f4fa2952640 10 monclient: get_monmap_and_config success 2024-09-11T01:48:32.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.300+0000 7f4f9affd640 1 -- 172.21.15.12:0/3226816116 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f94007bd0 con 0x7f4f9c06f910 2024-09-11T01:48:32.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.300+0000 7f4f9affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:32.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.300+0000 7f4fa2952640 4 set_mon_vals no callback set 2024-09-11T01:48:32.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.300+0000 7f4f9affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:48:32.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.300+0000 7f4f9affd640 10 monclient: dump: 2024-09-11T01:48:32.316 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:32.316 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:32.316 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.316 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.316 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:32.316 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.316 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:32.316 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:32.316 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:32.317 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.301+0000 7f4fa2952640 10 monclient: shutdown 2024-09-11T01:48:32.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.301+0000 7f4fa2952640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:32.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.301+0000 7f4fa2952640 1 -- 172.21.15.12:0/3226816116 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f9c06f910 msgr2=0x7f4f9c13f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:32.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.301+0000 7f4fa2952640 1 --2- 172.21.15.12:0/3226816116 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f9c06f910 0x7f4f9c13f430 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4f94009e50 tx=0x7f4f9402a560 comp rx=0 tx=0).stop 2024-09-11T01:48:32.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.301+0000 7f4fa2952640 1 -- 172.21.15.12:0/3226816116 shutdown_connections 2024-09-11T01:48:32.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.301+0000 7f4fa2952640 1 --2- 172.21.15.12:0/3226816116 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f9c06f910 0x7f4f9c13f430 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.301+0000 7f4fa2952640 1 --2- 172.21.15.12:0/3226816116 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f9c0a4970 0x7f4f9c06ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.301+0000 7f4fa2952640 1 -- 172.21.15.12:0/3226816116 >> 172.21.15.12:0/3226816116 conn(0x7f4f9c0a8600 msgr2=0x7f4f9c0a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:32.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.301+0000 7f4fa2952640 1 -- 172.21.15.12:0/3226816116 shutdown_connections 2024-09-11T01:48:32.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.301+0000 7f4fa2952640 1 -- 172.21.15.12:0/3226816116 wait complete. 2024-09-11T01:48:32.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.301+0000 7f4fa2952640 10 monclient: build_initial_monmap 2024-09-11T01:48:32.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.301+0000 7f4fa2952640 10 monclient: monmap: 2024-09-11T01:48:32.318 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:32.318 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:48:32.318 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:48:32.318 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:48:32.318 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:32.318 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.318 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:48:32.318 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:48:32.318 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:48:32.318 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.302+0000 7f4fa2952640 1 Processor -- start 2024-09-11T01:48:32.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.302+0000 7f4fa2952640 1 -- start start 2024-09-11T01:48:32.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.302+0000 7f4fa2952640 10 monclient: init 2024-09-11T01:48:32.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.302+0000 7f4fa2952640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:32.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.302+0000 7f4fa2952640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:48:32.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.302+0000 7f4fa2952640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f9c06f910 0x7f4f9c1398c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.302+0000 7f4fa2952640 10 monclient(hunting): picked mon.noname-c con 0x7f4f9c06f910 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:32.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4fa2952640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f9c0a2e20 0x7f4f9c139e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4fa2952640 10 monclient(hunting): picked mon.noname-b con 0x7f4f9c0a2e20 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:48:32.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4fa2952640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f9c0a4970 0x7f4f9c1ccf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4fa2952640 10 monclient(hunting): picked mon.noname-a con 0x7f4f9c0a4970 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:48:32.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4fa2952640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:32.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4fa2952640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9c13fd00 con 0x7f4f9c0a4970 2024-09-11T01:48:32.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4fa2952640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:32.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4fa2952640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9c13fb80 con 0x7f4f9c0a2e20 2024-09-11T01:48:32.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4fa2952640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:32.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4fa2952640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4f9c13fe80 con 0x7f4f9c06f910 2024-09-11T01:48:32.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4fa2952640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:32.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4fa2952640 10 monclient(hunting): authenticate will time out at 1918.745083s 2024-09-11T01:48:32.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4f9bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f9c06f910 0x7f4f9c1398c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4fa0eef640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f9c0a4970 0x7f4f9c1ccf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4f9bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f9c06f910 0x7f4f9c1398c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:44210/0 (socket says 172.21.15.12:44210) 2024-09-11T01:48:32.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4f9bfff640 1 -- 172.21.15.12:0/1417824283 learned_addr learned my addr 172.21.15.12:0/1417824283 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:48:32.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4f9bfff640 10 monclient(hunting): get_auth_request con 0x7f4f9c06f910 auth_method 0 2024-09-11T01:48:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4f9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4f9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4f9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4fa0eef640 10 monclient(hunting): get_auth_request con 0x7f4f9c0a4970 auth_method 0 2024-09-11T01:48:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4fa0eef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4fa0eef640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4fa0eef640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4f9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4f9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4f9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4fa0eef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:32.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4fa0eef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:32.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4fa0eef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:32.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4f9b7fe640 1 --2- 172.21.15.12:0/1417824283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f9c0a2e20 0x7f4f9c139e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4f9bfff640 10 monclient(hunting): handle_auth_done global_id 184394 payload 824 2024-09-11T01:48:32.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4f9bfff640 1 -- 172.21.15.12:0/1417824283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f9c0a2e20 msgr2=0x7f4f9c139e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:32.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4f9bfff640 1 --2- 172.21.15.12:0/1417824283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f9c0a2e20 0x7f4f9c139e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4f9bfff640 1 -- 172.21.15.12:0/1417824283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f9c0a4970 msgr2=0x7f4f9c1ccf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:32.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4f9bfff640 1 --2- 172.21.15.12:0/1417824283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f9c0a4970 0x7f4f9c1ccf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4f9bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:48:32.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4f9bfff640 1 monclient: found mon.noname-c 2024-09-11T01:48:32.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4f9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:32.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4f9bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:48:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4f9bfff640 1 -- 172.21.15.12:0/1417824283 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f9c13c070 con 0x7f4f9c06f910 2024-09-11T01:48:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4f9b7fe640 10 monclient: get_auth_request con 0x7f4f9c0a2e20 auth_method 0 2024-09-11T01:48:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4f9b7fe640 1 --2- 172.21.15.12:0/1417824283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f9c0a2e20 0x7f4f9c139e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:48:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.303+0000 7f4fa0eef640 1 --2- 172.21.15.12:0/1417824283 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f9c0a4970 0x7f4f9c1ccf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:48:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.305+0000 7f4f9bfff640 1 -- 172.21.15.12:0/1417824283 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f9c06f910 msgr2=0x7f4f9c1398c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.305+0000 7f4f9bfff640 1 --2- 172.21.15.12:0/1417824283 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f9c06f910 0x7f4f9c1398c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c00c430 tx=0x7f4f8c00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.305+0000 7f4f98ff9640 1 -- 172.21.15.12:0/1417824283 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c0119a0 con 0x7f4f9c06f910 2024-09-11T01:48:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.305+0000 7f4f98ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.305+0000 7f4f98ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:48:32.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.305+0000 7f4f98ff9640 10 monclient: dump: 2024-09-11T01:48:32.324 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:32.324 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:32.324 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.324 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.324 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:32.324 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.324 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:32.324 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:32.324 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:32.324 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.305+0000 7f4f98ff9640 10 monclient: _finish_auth 0 2024-09-11T01:48:32.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.305+0000 7f4f98ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:48:32.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.305+0000 7f4f98ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:32.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.305+0000 7f4f98ff9640 1 -- 172.21.15.12:0/1417824283 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f8c017070 con 0x7f4f9c06f910 2024-09-11T01:48:32.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.305+0000 7f4fa2952640 5 monclient: authenticate success, global_id 184394 2024-09-11T01:48:32.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.305+0000 7f4fa2952640 10 monclient: _renew_subs 2024-09-11T01:48:32.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.305+0000 7f4fa2952640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:48:32.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.305+0000 7f4fa2952640 1 -- 172.21.15.12:0/1417824283 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f9c0a6340 con 0x7f4f9c06f910 2024-09-11T01:48:32.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.305+0000 7f4f98ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:32.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.305+0000 7f4f98ff9640 1 -- 172.21.15.12:0/1417824283 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c011d80 con 0x7f4f9c06f910 2024-09-11T01:48:32.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.305+0000 7f4f98ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:32.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.305+0000 7f4f98ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:48:32.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.305+0000 7f4f98ff9640 10 monclient: dump: 2024-09-11T01:48:32.326 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:32.326 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:32.326 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.326 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.326 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:32.326 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.326 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:32.326 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:32.326 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:32.327 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.305+0000 7f4f9a7fc640 4 set_mon_vals no callback set 2024-09-11T01:48:32.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.305+0000 7f4fa2952640 10 monclient: _renew_subs 2024-09-11T01:48:32.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.305+0000 7f4fa2952640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:48:32.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.305+0000 7f4fa2952640 1 -- 172.21.15.12:0/1417824283 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f9c13bd90 con 0x7f4f9c06f910 2024-09-11T01:48:32.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.306+0000 7f4fa2952640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:32.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.306+0000 7f4fa2952640 1 --2- 172.21.15.12:0/1417824283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f9c1cd4d0 0x7f4f9c1cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.306+0000 7f4fa2952640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:32.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.306+0000 7f4fa2952640 1 -- 172.21.15.12:0/1417824283 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9c06a060 con 0x7f4f9c1cd4d0 2024-09-11T01:48:32.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.307+0000 7f4f98ff9640 1 -- 172.21.15.12:0/1417824283 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f4f8c01c460 con 0x7f4f9c06f910 2024-09-11T01:48:32.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.307+0000 7f4f9b7fe640 1 --2- 172.21.15.12:0/1417824283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f9c1cd4d0 0x7f4f9c1cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.307+0000 7f4f9b7fe640 10 monclient: get_auth_request con 0x7f4f9c1cd4d0 auth_method 0 2024-09-11T01:48:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.307+0000 7f4f9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.307+0000 7f4f9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.307+0000 7f4f9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.307+0000 7f4f9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.307+0000 7f4f9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.307+0000 7f4f9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.307+0000 7f4f98ff9640 1 --2- 172.21.15.12:0/1417824283 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4f8803e210 0x7f4f880406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.307+0000 7f4f98ff9640 1 -- 172.21.15.12:0/1417824283 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(642..642 src has 507..642) ==== 4831+0+0 (secure 0 0 0) 0x7f4f8c052c00 con 0x7f4f9c06f910 2024-09-11T01:48:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.307+0000 7f4f9b7fe640 10 monclient(hunting): handle_auth_done global_id 134782 payload 824 2024-09-11T01:48:32.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.307+0000 7f4f9b7fe640 1 -- 172.21.15.12:0/1417824283 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4f9c1cddf0 con 0x7f4f9c1cd4d0 2024-09-11T01:48:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.307+0000 7f4fa0eef640 1 --2- 172.21.15.12:0/1417824283 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4f8803e210 0x7f4f880406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.307+0000 7f4fa0eef640 10 monclient: get_auth_request con 0x7f4f8803e210 auth_method 0 2024-09-11T01:48:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.309+0000 7f4f9b7fe640 1 -- 172.21.15.12:0/1417824283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f9c1cd4d0 msgr2=0x7f4f9c1cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.309+0000 7f4f9b7fe640 1 --2- 172.21.15.12:0/1417824283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f9c1cd4d0 0x7f4f9c1cd8b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0a72d0 tx=0x7f4f90010bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.309+0000 7f4f98ff9640 1 -- 172.21.15.12:0/1417824283 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9000f040 con 0x7f4f9c1cd4d0 2024-09-11T01:48:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.309+0000 7f4f98ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.309+0000 7f4f98ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:48:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.309+0000 7f4f98ff9640 10 monclient: dump: 2024-09-11T01:48:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.329 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.330 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:32.330 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.330 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:32.330 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:32.330 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:32.330 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.310+0000 7f4fa0eef640 1 --2- 172.21.15.12:0/1417824283 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4f8803e210 0x7f4f880406d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4f94033000 tx=0x7f4f94009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:32.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.310+0000 7f4f98ff9640 1 -- 172.21.15.12:0/1417824283 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4f9001c400 con 0x7f4f9c1cd4d0 2024-09-11T01:48:32.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.310+0000 7f4f98ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:32.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.310+0000 7f4f98ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:48:32.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.310+0000 7f4f98ff9640 1 -- 172.21.15.12:0/1417824283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f9c1cd4d0 msgr2=0x7f4f9c1cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:32.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.310+0000 7f4f98ff9640 1 --2- 172.21.15.12:0/1417824283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f9c1cd4d0 0x7f4f9c1cd8b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0a72d0 tx=0x7f4f90010bb0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.310+0000 7f4f98ff9640 1 -- 172.21.15.12:0/1417824283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f9c1cd4d0 msgr2=0x7f4f9c1cd8b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:48:32.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.310+0000 7f4f98ff9640 1 --2- 172.21.15.12:0/1417824283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f9c1cd4d0 0x7f4f9c1cd8b0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.316+0000 7f4fa2952640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:48:32.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.316+0000 7f4fa2952640 1 --2- 172.21.15.12:0/1417824283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f9c0a0830 0x7f4f9c0a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.316+0000 7f4fa2952640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.316+0000 7f4fa2952640 1 -- 172.21.15.12:0/1417824283 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9c06ded0 con 0x7f4f9c0a0830 2024-09-11T01:48:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.316+0000 7f4f9b7fe640 1 --2- 172.21.15.12:0/1417824283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f9c0a0830 0x7f4f9c0a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.316+0000 7f4f9b7fe640 10 monclient: get_auth_request con 0x7f4f9c0a0830 auth_method 0 2024-09-11T01:48:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.316+0000 7f4f9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.316+0000 7f4f9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.316+0000 7f4f9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.316+0000 7f4f9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.316+0000 7f4f9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.316+0000 7f4f9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:32.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.316+0000 7f4f9b7fe640 10 monclient(hunting): handle_auth_done global_id 134785 payload 824 2024-09-11T01:48:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.316+0000 7f4f9b7fe640 1 -- 172.21.15.12:0/1417824283 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4f9c13f000 con 0x7f4f9c0a0830 2024-09-11T01:48:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.319+0000 7f4f9b7fe640 1 -- 172.21.15.12:0/1417824283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f9c0a0830 msgr2=0x7f4f9c0a0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.319+0000 7f4f9b7fe640 1 --2- 172.21.15.12:0/1417824283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f9c0a0830 0x7f4f9c0a0c40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0a72d0 tx=0x7f4f900046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.319+0000 7f4f98ff9640 1 -- 172.21.15.12:0/1417824283 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f90006450 con 0x7f4f9c0a0830 2024-09-11T01:48:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.319+0000 7f4f98ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.319+0000 7f4f98ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:48:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.319+0000 7f4f98ff9640 10 monclient: dump: 2024-09-11T01:48:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:32.334 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:32.335 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.319+0000 7f4f98ff9640 1 -- 172.21.15.12:0/1417824283 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f4f90006450 con 0x7f4f9c0a0830 2024-09-11T01:48:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.319+0000 7f4f98ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:48:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.319+0000 7f4f98ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:48:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.319+0000 7f4f98ff9640 1 -- 172.21.15.12:0/1417824283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f9c0a0830 msgr2=0x7f4f9c0a0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.319+0000 7f4f98ff9640 1 --2- 172.21.15.12:0/1417824283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f9c0a0830 0x7f4f9c0a0c40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0a72d0 tx=0x7f4f900046c0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.319+0000 7f4f98ff9640 1 -- 172.21.15.12:0/1417824283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f9c0a0830 msgr2=0x7f4f9c0a0c40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:48:32.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.319+0000 7f4f98ff9640 1 --2- 172.21.15.12:0/1417824283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f9c0a0830 0x7f4f9c0a0c40 secure :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0a72d0 tx=0x7f4f900046c0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.336 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:48:32.336 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:48:32.336 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:48:32.336 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:48:32.336 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 182, 2024-09-11T01:48:32.336 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 445281, 2024-09-11T01:48:32.336 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:48:32.336 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:48:32.336 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:48:32.337 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:48:32.337 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:32.337 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:48:32.337 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:48:32.337 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:48:32.337 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:48:32.337 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:48:32.337 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:48:32.337 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:48:32.337 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:48:32.337 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:48:32.337 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:48:32.338 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:48:32.338 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:48:32.338 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:48:32.338 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:48:32.338 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:48:32.338 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:32.338 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:48:32.338 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:48:32.338 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:48:32.338 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:48:32.338 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:48:32.339 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:48:32.339 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:48:32.339 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:48:32.339 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:48:32.339 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:48:32.339 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:48:32.339 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:48:32.339 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:48:32.339 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:32.339 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.340 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:48:32.340 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:48:32.340 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:48:32.340 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:48:32.340 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:48:32.340 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:48:32.340 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:48:32.340 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:48:32.340 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:48:32.340 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:48:32.340 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:48:32.341 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:48:32.341 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:48:32.341 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:48:32.341 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:48:32.341 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:48:32.341 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:48:32.341 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:48:32.341 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:48:32.341 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:48:32.341 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:48:32.341 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:48:32.342 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:48:32.342 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:48:32.342 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:48:32.342 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:48:32.342 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:48:32.342 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:48:32.342 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:32.342 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:48:32.342 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.342 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:48:32.342 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.343 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:48:32.343 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:48:32.343 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:48:32.343 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:48:32.343 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.343 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:48:32.343 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:48:32.343 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:32.344 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.344 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.344 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:48:32.344 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:48:32.344 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:32.344 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:32.344 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:32.344 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.344 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:48:32.344 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:48:32.344 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:48:32.344 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:48:32.345 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:48:32.345 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.345 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.345 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:48:32.345 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:48:32.345 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:48:32.345 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:48:32.346 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.346 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:48:32.346 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:48:32.346 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:32.346 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.346 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.346 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:48:32.346 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:48:32.346 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:32.346 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:32.346 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:32.347 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.347 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:48:32.347 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:48:32.347 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:48:32.347 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:48:32.347 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:48:32.347 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.347 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.347 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:48:32.348 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:48:32.348 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:48:32.348 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:48:32.348 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.348 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:48:32.348 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:48:32.348 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:32.348 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.348 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.348 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:48:32.348 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:48:32.349 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:32.349 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:32.349 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:32.349 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.349 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:48:32.349 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:48:32.349 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:48:32.349 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:48:32.350 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:48:32.350 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:32.350 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:32.350 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.350 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:48:32.350 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:48:32.350 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.350 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:48:32.350 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:48:32.350 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:48:32.350 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:32.351 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:32.351 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:48:32.351 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.351 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:48:32.351 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:48:32.351 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:48:32.351 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:32.351 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:32.351 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:48:32.352 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.352 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:48:32.352 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:48:32.352 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:48:32.352 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:32.352 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:32.352 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:48:32.352 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:32.353 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:48:32.353 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:48:32.353 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:48:32.353 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:32.353 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:32.353 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:32.353 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:48:32.353 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:48:32.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.319+0000 7f4fa2952640 1 -- 172.21.15.12:0/1417824283 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4f8803e210 msgr2=0x7f4f880406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:32.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.319+0000 7f4fa2952640 1 --2- 172.21.15.12:0/1417824283 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4f8803e210 0x7f4f880406d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4f94033000 tx=0x7f4f94009000 comp rx=0 tx=0).stop 2024-09-11T01:48:32.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.319+0000 7f4fa2952640 10 monclient: shutdown 2024-09-11T01:48:32.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.319+0000 7f4fa2952640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:32.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.319+0000 7f4fa2952640 1 -- 172.21.15.12:0/1417824283 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f9c06f910 msgr2=0x7f4f9c1398c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:32.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.319+0000 7f4fa2952640 1 --2- 172.21.15.12:0/1417824283 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f9c06f910 0x7f4f9c1398c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c00c430 tx=0x7f4f8c00c8f0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.319+0000 7f4fa0eef640 1 -- 172.21.15.12:0/1417824283 reap_dead start 2024-09-11T01:48:32.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.319+0000 7f4fa0eef640 1 -- 172.21.15.12:0/1417824283 reap_dead start 2024-09-11T01:48:32.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.319+0000 7f4fa2952640 1 -- 172.21.15.12:0/1417824283 shutdown_connections 2024-09-11T01:48:32.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.319+0000 7f4fa2952640 1 -- 172.21.15.12:0/1417824283 >> 172.21.15.12:0/1417824283 conn(0x7f4f9c0a8600 msgr2=0x7f4f9c1381d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:32.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.320+0000 7f4fa2952640 1 -- 172.21.15.12:0/1417824283 shutdown_connections 2024-09-11T01:48:32.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.320+0000 7f4fa2952640 1 -- 172.21.15.12:0/1417824283 wait complete. 2024-09-11T01:48:32.356 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:48:32.356 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-11T01:48:32.356 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T01:48:32.356 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-11T01:48:32.356 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-11T01:48:32.357 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-11T01:48:32.357 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:48:32.369 INFO:tasks.ceph.mon.c:Stopped 2024-09-11T01:48:32.369 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:48:32.369 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:48:32.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.508+0000 7f0ae930e640 10 monclient: get_monmap_and_config 2024-09-11T01:48:32.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.508+0000 7f0ae930e640 10 monclient: build_initial_monmap 2024-09-11T01:48:32.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.508+0000 7f0ae930e640 10 monclient: monmap: 2024-09-11T01:48:32.510 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:32.510 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:32.510 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:48:32.509156+0000 2024-09-11T01:48:32.510 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:48:32.509156+0000 2024-09-11T01:48:32.510 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:32.511 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.511 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:48:32.511 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:48:32.511 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:48:32.511 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:48:32.511 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:48:32.511 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.508+0000 7f0ae930e640 1 Processor -- start 2024-09-11T01:48:32.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae930e640 1 -- start start 2024-09-11T01:48:32.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae930e640 10 monclient: init 2024-09-11T01:48:32.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae930e640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:32.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae930e640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-11T01:48:32.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae930e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ae40a03a0 addr v1:172.21.15.12:6789/0 2024-09-11T01:48:32.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae930e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0ae40a13a0 0x7f0ae4139a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae930e640 10 monclient(hunting): picked mon.noname-c con 0x7f0ae40a13a0 addr v2:172.21.15.71:3300/0 2024-09-11T01:48:32.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae930e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ae40a17d0 0x7f0ae413bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae930e640 10 monclient(hunting): picked mon.noname-b con 0x7f0ae40a17d0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:32.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae930e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ae409fa00 con 0x7f0ae40a03a0 2024-09-11T01:48:32.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae930e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:32.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae930e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f0ae409ead0 con 0x7f0ae40a13a0 2024-09-11T01:48:32.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae930e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:32.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae930e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f0ae40a0780 con 0x7f0ae40a17d0 2024-09-11T01:48:32.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae930e640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:32.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae930e640 10 monclient(hunting): authenticate will time out at 1918.951576s 2024-09-11T01:48:32.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae37fe640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ae40a17d0 msgr2=0x7f0ae413bfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:48:32.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae37fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ae40a17d0 0x7f0ae413bfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:48:32.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae2ffd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0ae40a03a0 0x7f0ae40a0c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46224/0 (socket says 172.21.15.12:46224) 2024-09-11T01:48:32.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae2ffd640 1 -- 172.21.15.12:0/375816983 learned_addr learned my addr 172.21.15.12:0/375816983 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:48:32.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae27fc640 1 --2- 172.21.15.12:0/375816983 >> v2:172.21.15.71:3300/0 conn(0x7f0ae40a13a0 0x7f0ae4139a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae27fc640 10 monclient(hunting): get_auth_request con 0x7f0ae40a13a0 auth_method 0 2024-09-11T01:48:32.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:32.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae27fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:32.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:32.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:32.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:32.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.509+0000 7f0ae27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:32.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.510+0000 7f0ae27fc640 10 monclient(hunting): handle_auth_done global_id 134788 payload 274 2024-09-11T01:48:32.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.510+0000 7f0ae27fc640 1 -- 172.21.15.12:0/375816983 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ae40a17d0 msgr2=0x7f0ae413bfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:48:32.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.510+0000 7f0ae27fc640 1 --2- 172.21.15.12:0/375816983 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ae40a17d0 0x7f0ae413bfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.510+0000 7f0ae27fc640 1 -- 172.21.15.12:0/375816983 >> v1:172.21.15.12:6789/0 conn(0x7f0ae40a03a0 legacy=0x7f0ae40a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:48:32.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.510+0000 7f0ae27fc640 10 monclient: _finish_hunting 0 2024-09-11T01:48:32.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.510+0000 7f0ae27fc640 1 monclient: found mon.noname-c 2024-09-11T01:48:32.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.510+0000 7f0ae27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:32.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.510+0000 7f0ae27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:48:32.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.510+0000 7f0ae27fc640 1 -- 172.21.15.12:0/375816983 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae40a2a00 con 0x7f0ae40a13a0 2024-09-11T01:48:32.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.513+0000 7f0ae27fc640 1 -- 172.21.15.12:0/375816983 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ae40a13a0 msgr2=0x7f0ae4139a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:32.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.513+0000 7f0ae27fc640 1 --2- 172.21.15.12:0/375816983 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ae40a13a0 0x7f0ae4139a90 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0ad80097a0 tx=0x7f0ad8029fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b641db2ee23530ba server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:32.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.513+0000 7f0ae1ffb640 1 -- 172.21.15.12:0/375816983 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad8037070 con 0x7f0ae40a13a0 2024-09-11T01:48:32.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.513+0000 7f0ae1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:32.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.513+0000 7f0ae1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:48:32.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.513+0000 7f0ae1ffb640 10 monclient: dump: 2024-09-11T01:48:32.523 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:32.523 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:32.523 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.523 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.524 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:32.524 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.524 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:32.524 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:32.524 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:32.524 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.513+0000 7f0ae1ffb640 10 monclient: _finish_auth 0 2024-09-11T01:48:32.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.513+0000 7f0ae1ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:48:32.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.513+0000 7f0ae1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:32.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.513+0000 7f0ae1ffb640 1 -- 172.21.15.12:0/375816983 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ad8032640 con 0x7f0ae40a13a0 2024-09-11T01:48:32.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.513+0000 7f0ae1ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:32.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.513+0000 7f0ae1ffb640 1 -- 172.21.15.12:0/375816983 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad8032a10 con 0x7f0ae40a13a0 2024-09-11T01:48:32.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.513+0000 7f0ae1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:32.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.513+0000 7f0ae1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:48:32.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.513+0000 7f0ae1ffb640 10 monclient: dump: 2024-09-11T01:48:32.525 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:32.525 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:32.525 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.525 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.525 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:32.526 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.513+0000 7f0ae930e640 5 monclient: authenticate success, global_id 134788 2024-09-11T01:48:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.513+0000 7f0ae930e640 10 monclient: get_monmap_and_config success 2024-09-11T01:48:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.513+0000 7f0ae930e640 4 set_mon_vals no callback set 2024-09-11T01:48:32.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.514+0000 7f0ae930e640 10 monclient: shutdown 2024-09-11T01:48:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.514+0000 7f0ae930e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.514+0000 7f0ae930e640 1 -- 172.21.15.12:0/375816983 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ae40a13a0 msgr2=0x7f0ae4139a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.514+0000 7f0ae930e640 1 --2- 172.21.15.12:0/375816983 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ae40a13a0 0x7f0ae4139a90 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0ad80097a0 tx=0x7f0ad8029fe0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.514+0000 7f0ae930e640 1 -- 172.21.15.12:0/375816983 shutdown_connections 2024-09-11T01:48:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.514+0000 7f0ae930e640 1 --2- 172.21.15.12:0/375816983 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ae40a17d0 0x7f0ae413bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.514+0000 7f0ae930e640 1 --2- 172.21.15.12:0/375816983 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ae40a13a0 0x7f0ae4139a90 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.514+0000 7f0ae930e640 1 -- 172.21.15.12:0/375816983 >> 172.21.15.12:0/375816983 conn(0x7f0ae40a2d90 msgr2=0x7f0ae40a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.514+0000 7f0ae930e640 1 -- 172.21.15.12:0/375816983 shutdown_connections 2024-09-11T01:48:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.514+0000 7f0ae930e640 1 -- 172.21.15.12:0/375816983 wait complete. 2024-09-11T01:48:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.514+0000 7f0ae930e640 10 monclient: build_initial_monmap 2024-09-11T01:48:32.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.514+0000 7f0ae930e640 10 monclient: monmap: 2024-09-11T01:48:32.528 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:32.528 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:48:32.528 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:48:32.528 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:48:32.528 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:32.528 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.528 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:48:32.528 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:48:32.528 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:48:32.528 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae930e640 1 Processor -- start 2024-09-11T01:48:32.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae930e640 1 -- start start 2024-09-11T01:48:32.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae930e640 10 monclient: init 2024-09-11T01:48:32.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae930e640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:32.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae930e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:48:32.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae930e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ae40a03a0 0x7f0ae41c9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae930e640 10 monclient(hunting): picked mon.noname-b con 0x7f0ae40a03a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:48:32.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae930e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ae40a13a0 0x7f0ae4135e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae930e640 10 monclient(hunting): picked mon.noname-a con 0x7f0ae40a13a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:48:32.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae930e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ae40a17d0 0x7f0ae41cce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae930e640 10 monclient(hunting): picked mon.noname-c con 0x7f0ae40a17d0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:32.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae930e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:32.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae930e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae409f1d0 con 0x7f0ae40a13a0 2024-09-11T01:48:32.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae930e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:32.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae930e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae40a26f0 con 0x7f0ae40a03a0 2024-09-11T01:48:32.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae930e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:32.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae930e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f0ae409f350 con 0x7f0ae40a17d0 2024-09-11T01:48:32.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae930e640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:32.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae930e640 10 monclient(hunting): authenticate will time out at 1918.957754s 2024-09-11T01:48:32.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae37fe640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ae40a17d0 msgr2=0x7f0ae41cce00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:48:32.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae37fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ae40a17d0 0x7f0ae41cce00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:48:32.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae27fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ae40a13a0 0x7f0ae4135e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae27fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ae40a13a0 0x7f0ae4135e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36846/0 (socket says 172.21.15.12:36846) 2024-09-11T01:48:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae27fc640 1 -- 172.21.15.12:0/31932406 learned_addr learned my addr 172.21.15.12:0/31932406 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:48:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae27fc640 10 monclient(hunting): get_auth_request con 0x7f0ae40a13a0 auth_method 0 2024-09-11T01:48:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae27fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.515+0000 7f0ae2ffd640 1 --2- 172.21.15.12:0/31932406 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ae40a03a0 0x7f0ae41c9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.516+0000 7f0ae27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.516+0000 7f0ae2ffd640 10 monclient(hunting): get_auth_request con 0x7f0ae40a03a0 auth_method 0 2024-09-11T01:48:32.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.516+0000 7f0ae2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.516+0000 7f0ae2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.516+0000 7f0ae2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.516+0000 7f0ae2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.516+0000 7f0ae2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.516+0000 7f0ae2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.516+0000 7f0ae27fc640 10 monclient(hunting): handle_auth_done global_id 214149 payload 824 2024-09-11T01:48:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.516+0000 7f0ae27fc640 1 -- 172.21.15.12:0/31932406 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ae40a17d0 msgr2=0x7f0ae41cce00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:48:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.516+0000 7f0ae27fc640 1 --2- 172.21.15.12:0/31932406 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ae40a17d0 0x7f0ae41cce00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.516+0000 7f0ae27fc640 1 -- 172.21.15.12:0/31932406 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ae40a03a0 msgr2=0x7f0ae41c9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.516+0000 7f0ae27fc640 1 --2- 172.21.15.12:0/31932406 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ae40a03a0 0x7f0ae41c9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.516+0000 7f0ae27fc640 10 monclient: _finish_hunting 0 2024-09-11T01:48:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.516+0000 7f0ae27fc640 1 monclient: found mon.noname-a 2024-09-11T01:48:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.516+0000 7f0ae27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.516+0000 7f0ae27fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:48:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.516+0000 7f0ae27fc640 1 -- 172.21.15.12:0/31932406 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae41ca800 con 0x7f0ae40a13a0 2024-09-11T01:48:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.518+0000 7f0ae27fc640 1 -- 172.21.15.12:0/31932406 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ae40a13a0 msgr2=0x7f0ae4135e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.518+0000 7f0ae27fc640 1 --2- 172.21.15.12:0/31932406 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ae40a13a0 0x7f0ae4135e60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0ad80098c0 tx=0x7f0ad80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.518+0000 7f0acb7fe640 1 -- 172.21.15.12:0/31932406 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad8042020 con 0x7f0ae40a13a0 2024-09-11T01:48:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.518+0000 7f0acb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.518+0000 7f0acb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:48:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.518+0000 7f0acb7fe640 10 monclient: dump: 2024-09-11T01:48:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:32.533 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:32.534 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.518+0000 7f0acb7fe640 10 monclient: _finish_auth 0 2024-09-11T01:48:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.518+0000 7f0acb7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:48:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.518+0000 7f0acb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.518+0000 7f0acb7fe640 1 -- 172.21.15.12:0/31932406 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ad803d880 con 0x7f0ae40a13a0 2024-09-11T01:48:32.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.518+0000 7f0ae930e640 5 monclient: authenticate success, global_id 214149 2024-09-11T01:48:32.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.518+0000 7f0ae930e640 10 monclient: _renew_subs 2024-09-11T01:48:32.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.518+0000 7f0ae930e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:32.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.518+0000 7f0ae930e640 1 -- 172.21.15.12:0/31932406 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ae409d480 con 0x7f0ae40a13a0 2024-09-11T01:48:32.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.518+0000 7f0acb7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:32.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.518+0000 7f0acb7fe640 1 -- 172.21.15.12:0/31932406 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad8037040 con 0x7f0ae40a13a0 2024-09-11T01:48:32.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.518+0000 7f0acb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:32.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.518+0000 7f0acb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:48:32.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.518+0000 7f0acb7fe640 10 monclient: dump: 2024-09-11T01:48:32.535 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:32.535 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:32.535 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.535 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:32.535 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:32.535 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:32.536 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.518+0000 7f0ae930e640 10 monclient: _renew_subs 2024-09-11T01:48:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.518+0000 7f0ae930e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.518+0000 7f0ae930e640 1 -- 172.21.15.12:0/31932406 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ae4140410 con 0x7f0ae40a13a0 2024-09-11T01:48:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.518+0000 7f0ae17fa640 4 set_mon_vals no callback set 2024-09-11T01:48:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.519+0000 7f0acb7fe640 1 -- 172.21.15.12:0/31932406 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f0ad803dc90 con 0x7f0ae40a13a0 2024-09-11T01:48:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.519+0000 7f0ae930e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.519+0000 7f0ae930e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.519+0000 7f0ae930e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:32.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.519+0000 7f0ae930e640 1 -- 172.21.15.12:0/31932406 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ae40a0c90 con 0x7f0ae40a13a0 2024-09-11T01:48:32.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.519+0000 7f0acb7fe640 1 --2- 172.21.15.12:0/31932406 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0ad003e1c0 0x7f0ad0040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:32.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.519+0000 7f0acb7fe640 1 -- 172.21.15.12:0/31932406 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(642..642 src has 507..642) ==== 4831+0+0 (secure 0 0 0) 0x7f0ad8073780 con 0x7f0ae40a13a0 2024-09-11T01:48:32.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.520+0000 7f0ae2ffd640 1 --2- 172.21.15.12:0/31932406 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0ad003e1c0 0x7f0ad0040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:32.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.520+0000 7f0ae2ffd640 10 monclient: get_auth_request con 0x7f0ad003e1c0 auth_method 0 2024-09-11T01:48:32.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.522+0000 7f0ae2ffd640 1 --2- 172.21.15.12:0/31932406 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0ad003e1c0 0x7f0ad0040680 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0ad40092a0 tx=0x7f0ad4009230 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:32.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.524+0000 7f0acb7fe640 1 -- 172.21.15.12:0/31932406 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0ad804c070 con 0x7f0ae40a13a0 2024-09-11T01:48:32.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.524+0000 7f0acb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:32.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.524+0000 7f0acb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:48:32.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.778+0000 7f0ae930e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:48:32.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.778+0000 7f0ae930e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:48:32.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.778+0000 7f0ae930e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:32.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.778+0000 7f0ae930e640 1 -- 172.21.15.12:0/31932406 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0ae411cfa0 con 0x7f0ae40a13a0 2024-09-11T01:48:32.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.779+0000 7f0acb7fe640 1 -- 172.21.15.12:0/31932406 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f0ad802ae20 con 0x7f0ae40a13a0 2024-09-11T01:48:32.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.779+0000 7f0acb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:48:32.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.779+0000 7f0acb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:48:32.781 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":182,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:48:32.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.781+0000 7f0ae930e640 1 -- 172.21.15.12:0/31932406 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0ad003e1c0 msgr2=0x7f0ad0040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:32.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.781+0000 7f0ae930e640 1 --2- 172.21.15.12:0/31932406 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0ad003e1c0 0x7f0ad0040680 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0ad40092a0 tx=0x7f0ad4009230 comp rx=0 tx=0).stop 2024-09-11T01:48:32.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.781+0000 7f0ae930e640 10 monclient: shutdown 2024-09-11T01:48:32.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.781+0000 7f0ae930e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:32.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.781+0000 7f0ae930e640 1 -- 172.21.15.12:0/31932406 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ae40a13a0 msgr2=0x7f0ae4135e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:32.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.781+0000 7f0ae930e640 1 --2- 172.21.15.12:0/31932406 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ae40a13a0 0x7f0ae4135e60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0ad80098c0 tx=0x7f0ad80046c0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.781+0000 7f0ae930e640 1 -- 172.21.15.12:0/31932406 shutdown_connections 2024-09-11T01:48:32.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.781+0000 7f0ae930e640 1 --2- 172.21.15.12:0/31932406 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ae40a17d0 0x7f0ae41cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.781+0000 7f0ae930e640 1 --2- 172.21.15.12:0/31932406 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0ad003e1c0 0x7f0ad0040680 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.781+0000 7f0ae930e640 1 --2- 172.21.15.12:0/31932406 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ae40a13a0 0x7f0ae4135e60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.781+0000 7f0ae930e640 1 --2- 172.21.15.12:0/31932406 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ae40a03a0 0x7f0ae41c9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:32.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.781+0000 7f0ae930e640 1 -- 172.21.15.12:0/31932406 >> 172.21.15.12:0/31932406 conn(0x7f0ae40a2d90 msgr2=0x7f0ae40a4cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:32.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.781+0000 7f0ae930e640 1 -- 172.21.15.12:0/31932406 shutdown_connections 2024-09-11T01:48:32.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:32.781+0000 7f0ae930e640 1 -- 172.21.15.12:0/31932406 wait complete. 2024-09-11T01:48:35.794 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:48:35.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.906+0000 7fda6b0c6640 10 monclient: get_monmap_and_config 2024-09-11T01:48:35.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.906+0000 7fda6b0c6640 10 monclient: build_initial_monmap 2024-09-11T01:48:35.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.906+0000 7fda6b0c6640 10 monclient: monmap: 2024-09-11T01:48:35.908 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:35.908 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:35.909 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:48:35.907479+0000 2024-09-11T01:48:35.909 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:48:35.907479+0000 2024-09-11T01:48:35.909 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:35.909 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:35.909 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:48:35.909 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:48:35.909 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:48:35.909 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:48:35.909 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:48:35.909 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:35.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.907+0000 7fda6b0c6640 1 Processor -- start 2024-09-11T01:48:35.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.907+0000 7fda6b0c6640 1 -- start start 2024-09-11T01:48:35.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.907+0000 7fda6b0c6640 10 monclient: init 2024-09-11T01:48:35.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda6b0c6640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:35.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda6b0c6640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-11T01:48:35.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda6b0c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda640a2e90 0x7fda6406dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:35.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda6b0c6640 10 monclient(hunting): picked mon.noname-b con 0x7fda640a2e90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:35.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda6b0c6640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fda640a3270 0x7fda640a3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:35.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda6b0c6640 10 monclient(hunting): picked mon.noname-a con 0x7fda640a3270 addr v2:172.21.15.12:3300/0 2024-09-11T01:48:35.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda6b0c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fda64138040 addr v1:172.21.15.12:6789/0 2024-09-11T01:48:35.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda6b0c6640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda640a46c0 con 0x7fda64138040 2024-09-11T01:48:35.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda6b0c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:35.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda6b0c6640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fda640a4900 con 0x7fda640a3270 2024-09-11T01:48:35.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda6b0c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:35.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda6b0c6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fda6406ee50 con 0x7fda640a2e90 2024-09-11T01:48:35.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda6b0c6640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:35.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda6b0c6640 10 monclient(hunting): authenticate will time out at 1922.350227s 2024-09-11T01:48:35.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda68e62640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda640a2e90 msgr2=0x7fda6406dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:48:35.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda68e62640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda640a2e90 0x7fda6406dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:48:35.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda63fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fda640a3270 0x7fda640a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:35.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda63fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fda640a3270 0x7fda640a3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36860/0 (socket says 172.21.15.12:36860) 2024-09-11T01:48:35.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda69663640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fda64138040 0x7fda64138460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46240/0 (socket says 172.21.15.12:46240) 2024-09-11T01:48:35.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda63fff640 1 -- 172.21.15.12:0/3901832548 learned_addr learned my addr 172.21.15.12:0/3901832548 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:48:35.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda63fff640 10 monclient(hunting): get_auth_request con 0x7fda640a3270 auth_method 0 2024-09-11T01:48:35.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:35.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda63fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:35.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:35.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:35.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:35.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:35.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda63fff640 10 monclient(hunting): handle_auth_done global_id 214152 payload 274 2024-09-11T01:48:35.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda63fff640 1 -- 172.21.15.12:0/3901832548 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda640a2e90 msgr2=0x7fda6406dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:48:35.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda63fff640 1 --2- 172.21.15.12:0/3901832548 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda640a2e90 0x7fda6406dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:35.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda63fff640 1 -- 172.21.15.12:0/3901832548 >> v1:172.21.15.12:6789/0 conn(0x7fda64138040 legacy=0x7fda64138460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:48:35.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda63fff640 10 monclient: _finish_hunting 0 2024-09-11T01:48:35.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda63fff640 1 monclient: found mon.noname-a 2024-09-11T01:48:35.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:35.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda63fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:48:35.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.908+0000 7fda63fff640 1 -- 172.21.15.12:0/3901832548 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda6406eb70 con 0x7fda640a3270 2024-09-11T01:48:35.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.911+0000 7fda63fff640 1 -- 172.21.15.12:0/3901832548 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda640a3270 msgr2=0x7fda640a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:35.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.911+0000 7fda63fff640 1 --2- 172.21.15.12:0/3901832548 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda640a3270 0x7fda640a3650 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fda640a4a80 tx=0x7fda5002a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=782a27714e603ed0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:35.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.911+0000 7fda637fe640 1 -- 172.21.15.12:0/3901832548 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda50037070 con 0x7fda640a3270 2024-09-11T01:48:35.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.911+0000 7fda637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:35.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.911+0000 7fda637fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:48:35.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.911+0000 7fda637fe640 10 monclient: dump: 2024-09-11T01:48:35.922 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:35.922 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:35.922 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:35.922 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:35.922 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:35.923 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.911+0000 7fda637fe640 10 monclient: _finish_auth 0 2024-09-11T01:48:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.911+0000 7fda637fe640 10 monclient: _check_auth_tickets 2024-09-11T01:48:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.911+0000 7fda637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.911+0000 7fda6b0c6640 5 monclient: authenticate success, global_id 214152 2024-09-11T01:48:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.911+0000 7fda637fe640 1 -- 172.21.15.12:0/3901832548 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda50032640 con 0x7fda640a3270 2024-09-11T01:48:35.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.911+0000 7fda6b0c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:48:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.911+0000 7fda637fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.911+0000 7fda6b0c6640 10 monclient: get_monmap_and_config success 2024-09-11T01:48:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.911+0000 7fda6b0c6640 4 set_mon_vals no callback set 2024-09-11T01:48:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.911+0000 7fda637fe640 1 -- 172.21.15.12:0/3901832548 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda50032a10 con 0x7fda640a3270 2024-09-11T01:48:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.911+0000 7fda637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.911+0000 7fda637fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:48:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.911+0000 7fda637fe640 10 monclient: dump: 2024-09-11T01:48:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:35.924 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:35.925 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:35.925 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:35.925 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:35.925 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:35.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.912+0000 7fda6b0c6640 10 monclient: shutdown 2024-09-11T01:48:35.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.912+0000 7fda6b0c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:35.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.912+0000 7fda6b0c6640 1 -- 172.21.15.12:0/3901832548 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda640a3270 msgr2=0x7fda640a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:35.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.912+0000 7fda6b0c6640 1 --2- 172.21.15.12:0/3901832548 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda640a3270 0x7fda640a3650 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fda640a4a80 tx=0x7fda5002a0a0 comp rx=0 tx=0).stop 2024-09-11T01:48:35.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.912+0000 7fda6b0c6640 1 -- 172.21.15.12:0/3901832548 shutdown_connections 2024-09-11T01:48:35.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.912+0000 7fda6b0c6640 1 --2- 172.21.15.12:0/3901832548 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda640a3270 0x7fda640a3650 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:35.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.912+0000 7fda6b0c6640 1 --2- 172.21.15.12:0/3901832548 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda640a2e90 0x7fda6406dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:35.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.912+0000 7fda6b0c6640 1 -- 172.21.15.12:0/3901832548 >> 172.21.15.12:0/3901832548 conn(0x7fda640a8490 msgr2=0x7fda640a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:35.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.912+0000 7fda6b0c6640 1 -- 172.21.15.12:0/3901832548 shutdown_connections 2024-09-11T01:48:35.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.912+0000 7fda6b0c6640 1 -- 172.21.15.12:0/3901832548 wait complete. 2024-09-11T01:48:35.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.913+0000 7fda6b0c6640 10 monclient: build_initial_monmap 2024-09-11T01:48:35.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.913+0000 7fda6b0c6640 10 monclient: monmap: 2024-09-11T01:48:35.926 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:35.926 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:48:35.926 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:48:35.926 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:48:35.926 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:35.926 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:35.926 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:48:35.926 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:48:35.926 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:48:35.926 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:35.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.913+0000 7fda6b0c6640 1 Processor -- start 2024-09-11T01:48:35.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.913+0000 7fda6b0c6640 1 -- start start 2024-09-11T01:48:35.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.913+0000 7fda6b0c6640 10 monclient: init 2024-09-11T01:48:35.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda6b0c6640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:35.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda6b0c6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:48:35.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda6b0c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda640a2e90 0x7fda641cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:35.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda6b0c6640 10 monclient(hunting): picked mon.noname-c con 0x7fda640a2e90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:35.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda6b0c6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda640a3270 0x7fda641cda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:35.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda6b0c6640 10 monclient(hunting): picked mon.noname-a con 0x7fda640a3270 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:48:35.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda6b0c6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda64138040 0x7fda64124910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:35.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda6b0c6640 10 monclient(hunting): picked mon.noname-b con 0x7fda64138040 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:48:35.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda6b0c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:35.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda6b0c6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fda640a3b90 con 0x7fda640a3270 2024-09-11T01:48:35.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda6b0c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:35.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda6b0c6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fda6406fba0 con 0x7fda64138040 2024-09-11T01:48:35.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda6b0c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:35.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda6b0c6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fda6409f680 con 0x7fda640a2e90 2024-09-11T01:48:35.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda6b0c6640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:35.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda6b0c6640 10 monclient(hunting): authenticate will time out at 1922.356140s 2024-09-11T01:48:35.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda68e62640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda640a2e90 msgr2=0x7fda641cd4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:48:35.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda68e62640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda640a2e90 0x7fda641cd4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:48:35.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda63fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda640a3270 0x7fda641cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:35.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda63fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda640a3270 0x7fda641cda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36866/0 (socket says 172.21.15.12:36866) 2024-09-11T01:48:35.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda63fff640 1 -- 172.21.15.12:0/678157052 learned_addr learned my addr 172.21.15.12:0/678157052 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:48:35.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda63fff640 10 monclient(hunting): get_auth_request con 0x7fda640a3270 auth_method 0 2024-09-11T01:48:35.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:35.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda63fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:35.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:35.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:35.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:35.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:35.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda63fff640 10 monclient(hunting): handle_auth_done global_id 214155 payload 824 2024-09-11T01:48:35.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda63fff640 1 -- 172.21.15.12:0/678157052 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda640a2e90 msgr2=0x7fda641cd4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:48:35.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda63fff640 1 --2- 172.21.15.12:0/678157052 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda640a2e90 0x7fda641cd4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:35.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda69663640 1 --2- 172.21.15.12:0/678157052 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda64138040 0x7fda64124910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:35.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda63fff640 1 -- 172.21.15.12:0/678157052 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda64138040 msgr2=0x7fda64124910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:35.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda63fff640 1 --2- 172.21.15.12:0/678157052 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda64138040 0x7fda64124910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:35.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda63fff640 10 monclient: _finish_hunting 0 2024-09-11T01:48:35.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda63fff640 1 monclient: found mon.noname-a 2024-09-11T01:48:35.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:35.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda63fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:48:35.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.914+0000 7fda63fff640 1 -- 172.21.15.12:0/678157052 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda64128450 con 0x7fda640a3270 2024-09-11T01:48:35.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.916+0000 7fda63fff640 1 -- 172.21.15.12:0/678157052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda640a3270 msgr2=0x7fda641cda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:35.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.916+0000 7fda63fff640 1 --2- 172.21.15.12:0/678157052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda640a3270 0x7fda641cda20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fda500090b0 tx=0x7fda500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:35.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.917+0000 7fda617fa640 1 -- 172.21.15.12:0/678157052 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda50041070 con 0x7fda640a3270 2024-09-11T01:48:35.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.917+0000 7fda617fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:35.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.917+0000 7fda617fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:48:35.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.917+0000 7fda617fa640 10 monclient: dump: 2024-09-11T01:48:35.930 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:35.930 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:35.930 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:35.930 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:35.930 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:35.930 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:35.931 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:35.931 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:35.931 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:35.931 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:35.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.917+0000 7fda617fa640 10 monclient: _finish_auth 0 2024-09-11T01:48:35.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.917+0000 7fda617fa640 10 monclient: _check_auth_tickets 2024-09-11T01:48:35.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.917+0000 7fda617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:35.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.917+0000 7fda617fa640 1 -- 172.21.15.12:0/678157052 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda5003caa0 con 0x7fda640a3270 2024-09-11T01:48:35.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.917+0000 7fda617fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:35.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.917+0000 7fda617fa640 1 -- 172.21.15.12:0/678157052 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda50037040 con 0x7fda640a3270 2024-09-11T01:48:35.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.917+0000 7fda617fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:35.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.917+0000 7fda617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:48:35.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.917+0000 7fda617fa640 10 monclient: dump: 2024-09-11T01:48:35.932 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:35.932 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:35.932 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:35.932 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:35.932 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:35.932 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:35.932 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:35.932 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:35.932 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:35.932 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:35.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.917+0000 7fda6b0c6640 5 monclient: authenticate success, global_id 214155 2024-09-11T01:48:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.917+0000 7fda6b0c6640 10 monclient: _renew_subs 2024-09-11T01:48:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.917+0000 7fda627fc640 4 set_mon_vals no callback set 2024-09-11T01:48:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.917+0000 7fda6b0c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.917+0000 7fda6b0c6640 1 -- 172.21.15.12:0/678157052 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda64128730 con 0x7fda640a3270 2024-09-11T01:48:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.917+0000 7fda6b0c6640 10 monclient: _renew_subs 2024-09-11T01:48:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.917+0000 7fda6b0c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.917+0000 7fda6b0c6640 1 -- 172.21.15.12:0/678157052 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda641263d0 con 0x7fda640a3270 2024-09-11T01:48:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.918+0000 7fda617fa640 1 -- 172.21.15.12:0/678157052 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fda5003d3f0 con 0x7fda640a3270 2024-09-11T01:48:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.918+0000 7fda6b0c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.918+0000 7fda6b0c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.918+0000 7fda6b0c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.918+0000 7fda6b0c6640 1 -- 172.21.15.12:0/678157052 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda64127fe0 con 0x7fda640a3270 2024-09-11T01:48:35.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.918+0000 7fda617fa640 1 --2- 172.21.15.12:0/678157052 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda4803e210 0x7fda480406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:35.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.918+0000 7fda617fa640 1 -- 172.21.15.12:0/678157052 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(642..642 src has 507..642) ==== 4831+0+0 (secure 0 0 0) 0x7fda50073890 con 0x7fda640a3270 2024-09-11T01:48:35.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.918+0000 7fda68e62640 1 --2- 172.21.15.12:0/678157052 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda4803e210 0x7fda480406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:35.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.918+0000 7fda68e62640 10 monclient: get_auth_request con 0x7fda4803e210 auth_method 0 2024-09-11T01:48:35.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.921+0000 7fda68e62640 1 --2- 172.21.15.12:0/678157052 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda4803e210 0x7fda480406d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fda58003980 tx=0x7fda58006d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:35.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.922+0000 7fda617fa640 1 -- 172.21.15.12:0/678157052 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fda50072260 con 0x7fda640a3270 2024-09-11T01:48:35.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.922+0000 7fda617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:35.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:35.922+0000 7fda617fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:48:36.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:36.181+0000 7fda6b0c6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:48:36.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:36.181+0000 7fda6b0c6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:48:36.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:36.181+0000 7fda6b0c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:36.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:36.181+0000 7fda6b0c6640 1 -- 172.21.15.12:0/678157052 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fda64127bc0 con 0x7fda640a3270 2024-09-11T01:48:36.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:36.181+0000 7fda617fa640 1 -- 172.21.15.12:0/678157052 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fda50032520 con 0x7fda640a3270 2024-09-11T01:48:36.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:36.181+0000 7fda617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:48:36.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:36.181+0000 7fda617fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:48:36.184 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":182,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:48:36.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:36.184+0000 7fda6b0c6640 1 -- 172.21.15.12:0/678157052 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda4803e210 msgr2=0x7fda480406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:36.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:36.184+0000 7fda6b0c6640 1 --2- 172.21.15.12:0/678157052 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda4803e210 0x7fda480406d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fda58003980 tx=0x7fda58006d30 comp rx=0 tx=0).stop 2024-09-11T01:48:36.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:36.184+0000 7fda6b0c6640 10 monclient: shutdown 2024-09-11T01:48:36.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:36.184+0000 7fda6b0c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:36.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:36.184+0000 7fda6b0c6640 1 -- 172.21.15.12:0/678157052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda640a3270 msgr2=0x7fda641cda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:36.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:36.184+0000 7fda6b0c6640 1 --2- 172.21.15.12:0/678157052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda640a3270 0x7fda641cda20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fda500090b0 tx=0x7fda500046c0 comp rx=0 tx=0).stop 2024-09-11T01:48:36.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:36.184+0000 7fda6b0c6640 1 -- 172.21.15.12:0/678157052 shutdown_connections 2024-09-11T01:48:36.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:36.184+0000 7fda6b0c6640 1 --2- 172.21.15.12:0/678157052 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda64138040 0x7fda64124910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:36.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:36.184+0000 7fda6b0c6640 1 --2- 172.21.15.12:0/678157052 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fda4803e210 0x7fda480406d0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:36.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:36.184+0000 7fda6b0c6640 1 --2- 172.21.15.12:0/678157052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda640a3270 0x7fda641cda20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:36.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:36.184+0000 7fda6b0c6640 1 --2- 172.21.15.12:0/678157052 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fda640a2e90 0x7fda641cd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:36.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:36.184+0000 7fda6b0c6640 1 -- 172.21.15.12:0/678157052 >> 172.21.15.12:0/678157052 conn(0x7fda640a8490 msgr2=0x7fda64072090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:36.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:36.184+0000 7fda6b0c6640 1 -- 172.21.15.12:0/678157052 shutdown_connections 2024-09-11T01:48:36.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:36.184+0000 7fda6b0c6640 1 -- 172.21.15.12:0/678157052 wait complete. 2024-09-11T01:48:39.197 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:48:39.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.309+0000 7f4cc9f64640 10 monclient: get_monmap_and_config 2024-09-11T01:48:39.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.309+0000 7f4cc9f64640 10 monclient: build_initial_monmap 2024-09-11T01:48:39.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.309+0000 7f4cc9f64640 10 monclient: monmap: 2024-09-11T01:48:39.311 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:39.311 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:39.311 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:48:39.310312+0000 2024-09-11T01:48:39.311 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:48:39.310312+0000 2024-09-11T01:48:39.311 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:39.311 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:39.312 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:48:39.312 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:48:39.312 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:48:39.312 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:48:39.312 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:48:39.312 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:39.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.310+0000 7f4cc9f64640 1 Processor -- start 2024-09-11T01:48:39.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.310+0000 7f4cc9f64640 1 -- start start 2024-09-11T01:48:39.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.310+0000 7f4cc9f64640 10 monclient: init 2024-09-11T01:48:39.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc9f64640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:39.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc9f64640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-11T01:48:39.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc9f64640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4cc40a2e90 0x7f4cc406dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:39.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc9f64640 10 monclient(hunting): picked mon.noname-c con 0x7f4cc40a2e90 addr v2:172.21.15.71:3300/0 2024-09-11T01:48:39.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc9f64640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4cc40a3270 0x7f4cc40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:39.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc9f64640 10 monclient(hunting): picked mon.noname-b con 0x7f4cc40a3270 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:39.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc9f64640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4cc4138040 addr v1:172.21.15.71:6789/0 2024-09-11T01:48:39.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc9f64640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cc40a46c0 con 0x7f4cc4138040 2024-09-11T01:48:39.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc9f64640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:39.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc9f64640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f4cc40a4900 con 0x7f4cc40a2e90 2024-09-11T01:48:39.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc9f64640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:39.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc9f64640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4cc406ee50 con 0x7f4cc40a3270 2024-09-11T01:48:39.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc9f64640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:39.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc9f64640 10 monclient(hunting): authenticate will time out at 1925.753158s 2024-09-11T01:48:39.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc2ffd640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4cc40a3270 msgr2=0x7f4cc40a3680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:48:39.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc2ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4cc40a3270 0x7f4cc40a3680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:48:39.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc37fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4cc40a2e90 0x7f4cc406dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:39.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc3fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f4cc4138040 0x7f4cc4138490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.12:54034/0 (socket says 172.21.15.12:54034) 2024-09-11T01:48:39.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc3fff640 1 -- 172.21.15.12:0/3993878676 learned_addr learned my addr 172.21.15.12:0/3993878676 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:48:39.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc37fe640 10 monclient(hunting): get_auth_request con 0x7f4cc40a2e90 auth_method 0 2024-09-11T01:48:39.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:39.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:39.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:39.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:39.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:39.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:39.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc37fe640 10 monclient(hunting): handle_auth_done global_id 134794 payload 274 2024-09-11T01:48:39.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc37fe640 1 -- 172.21.15.12:0/3993878676 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4cc40a3270 msgr2=0x7f4cc40a3680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:48:39.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc37fe640 1 --2- 172.21.15.12:0/3993878676 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4cc40a3270 0x7f4cc40a3680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:39.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc37fe640 1 -- 172.21.15.12:0/3993878676 >> v1:172.21.15.71:6789/0 conn(0x7f4cc4138040 legacy=0x7f4cc4138490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:48:39.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc37fe640 10 monclient: _finish_hunting 0 2024-09-11T01:48:39.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc37fe640 1 monclient: found mon.noname-c 2024-09-11T01:48:39.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:39.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:48:39.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.311+0000 7f4cc37fe640 1 -- 172.21.15.12:0/3993878676 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cc406eb70 con 0x7f4cc40a2e90 2024-09-11T01:48:39.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.314+0000 7f4cc37fe640 1 -- 172.21.15.12:0/3993878676 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4cc40a2e90 msgr2=0x7f4cc406dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:39.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.314+0000 7f4cc37fe640 1 --2- 172.21.15.12:0/3993878676 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4cc40a2e90 0x7f4cc406dc60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4cb40097a0 tx=0x7f4cb4029fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31923d44a9a07972 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:39.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc27fc640 1 -- 172.21.15.12:0/3993878676 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4037070 con 0x7f4cc40a2e90 2024-09-11T01:48:39.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:39.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:48:39.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc27fc640 10 monclient: dump: 2024-09-11T01:48:39.325 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:39.325 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:39.325 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:39.325 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:39.325 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:39.325 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:39.325 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:39.325 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:39.326 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:39.326 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:39.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc27fc640 10 monclient: _finish_auth 0 2024-09-11T01:48:39.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc27fc640 10 monclient: _check_auth_tickets 2024-09-11T01:48:39.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:39.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc9f64640 5 monclient: authenticate success, global_id 134794 2024-09-11T01:48:39.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc27fc640 1 -- 172.21.15.12:0/3993878676 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cb4032640 con 0x7f4cc40a2e90 2024-09-11T01:48:39.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc9f64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:48:39.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:39.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc9f64640 10 monclient: get_monmap_and_config success 2024-09-11T01:48:39.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc9f64640 4 set_mon_vals no callback set 2024-09-11T01:48:39.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc27fc640 1 -- 172.21.15.12:0/3993878676 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4032a10 con 0x7f4cc40a2e90 2024-09-11T01:48:39.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:39.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:48:39.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc27fc640 10 monclient: dump: 2024-09-11T01:48:39.327 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:39.327 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:39.327 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:39.327 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:39.327 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:39.327 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:39.327 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:39.327 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:39.327 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:39.327 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:39.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc9f64640 10 monclient: shutdown 2024-09-11T01:48:39.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc9f64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:39.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc9f64640 1 -- 172.21.15.12:0/3993878676 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4cc40a2e90 msgr2=0x7f4cc406dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:39.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc9f64640 1 --2- 172.21.15.12:0/3993878676 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4cc40a2e90 0x7f4cc406dc60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4cb40097a0 tx=0x7f4cb4029fe0 comp rx=0 tx=0).stop 2024-09-11T01:48:39.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc9f64640 1 -- 172.21.15.12:0/3993878676 shutdown_connections 2024-09-11T01:48:39.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc9f64640 1 --2- 172.21.15.12:0/3993878676 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4cc40a3270 0x7f4cc40a3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:39.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc9f64640 1 --2- 172.21.15.12:0/3993878676 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4cc40a2e90 0x7f4cc406dc60 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:39.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc9f64640 1 -- 172.21.15.12:0/3993878676 >> 172.21.15.12:0/3993878676 conn(0x7f4cc40a8490 msgr2=0x7f4cc40a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:39.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc9f64640 1 -- 172.21.15.12:0/3993878676 shutdown_connections 2024-09-11T01:48:39.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.315+0000 7f4cc9f64640 1 -- 172.21.15.12:0/3993878676 wait complete. 2024-09-11T01:48:39.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.316+0000 7f4cc9f64640 10 monclient: build_initial_monmap 2024-09-11T01:48:39.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.316+0000 7f4cc9f64640 10 monclient: monmap: 2024-09-11T01:48:39.328 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:39.329 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:48:39.329 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:48:39.329 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:48:39.329 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:39.329 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:39.329 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:48:39.329 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:48:39.329 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:48:39.329 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:39.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.316+0000 7f4cc9f64640 1 Processor -- start 2024-09-11T01:48:39.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.316+0000 7f4cc9f64640 1 -- start start 2024-09-11T01:48:39.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.316+0000 7f4cc9f64640 10 monclient: init 2024-09-11T01:48:39.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc9f64640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:39.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc9f64640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:48:39.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc9f64640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4cc40a2e90 0x7f4cc41cd830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:39.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc9f64640 10 monclient(hunting): picked mon.noname-b con 0x7f4cc40a2e90 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:48:39.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc9f64640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4cc40a3270 0x7f4cc4124a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:39.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc9f64640 10 monclient(hunting): picked mon.noname-a con 0x7f4cc40a3270 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:48:39.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc9f64640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4cc4138040 0x7f4cc4124f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:39.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc9f64640 10 monclient(hunting): picked mon.noname-c con 0x7f4cc4138040 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:39.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc9f64640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:39.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc9f64640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc406fba0 con 0x7f4cc40a3270 2024-09-11T01:48:39.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc9f64640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:39.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc9f64640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc40a3bc0 con 0x7f4cc40a2e90 2024-09-11T01:48:39.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc9f64640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:39.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc9f64640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4cc406fd20 con 0x7f4cc4138040 2024-09-11T01:48:39.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc9f64640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:39.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc9f64640 10 monclient(hunting): authenticate will time out at 1925.759466s 2024-09-11T01:48:39.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc3fff640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4cc4138040 msgr2=0x7f4cc4124f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:48:39.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc3fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4cc4138040 0x7f4cc4124f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:48:39.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc2ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4cc40a3270 0x7f4cc4124a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:39.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc2ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4cc40a3270 0x7f4cc4124a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38090/0 (socket says 172.21.15.12:38090) 2024-09-11T01:48:39.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc2ffd640 1 -- 172.21.15.12:0/3789087657 learned_addr learned my addr 172.21.15.12:0/3789087657 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:48:39.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc2ffd640 10 monclient(hunting): get_auth_request con 0x7f4cc40a3270 auth_method 0 2024-09-11T01:48:39.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:39.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:39.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:39.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc37fe640 1 --2- 172.21.15.12:0/3789087657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4cc40a2e90 0x7f4cc41cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:39.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc37fe640 10 monclient(hunting): get_auth_request con 0x7f4cc40a2e90 auth_method 0 2024-09-11T01:48:39.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:39.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:39.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:39.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:39.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:39.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:39.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:39.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:39.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:39.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc2ffd640 10 monclient(hunting): handle_auth_done global_id 214158 payload 824 2024-09-11T01:48:39.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc2ffd640 1 -- 172.21.15.12:0/3789087657 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4cc4138040 msgr2=0x7f4cc4124f80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:48:39.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc2ffd640 1 --2- 172.21.15.12:0/3789087657 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4cc4138040 0x7f4cc4124f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:39.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc2ffd640 1 -- 172.21.15.12:0/3789087657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4cc40a2e90 msgr2=0x7f4cc41cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:39.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc2ffd640 1 --2- 172.21.15.12:0/3789087657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4cc40a2e90 0x7f4cc41cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:39.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc2ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:48:39.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc2ffd640 1 monclient: found mon.noname-a 2024-09-11T01:48:39.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:39.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:48:39.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.317+0000 7f4cc2ffd640 1 -- 172.21.15.12:0/3789087657 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cc4137e20 con 0x7f4cc40a3270 2024-09-11T01:48:39.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.320+0000 7f4cc2ffd640 1 -- 172.21.15.12:0/3789087657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4cc40a3270 msgr2=0x7f4cc4124a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:39.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.320+0000 7f4cc2ffd640 1 --2- 172.21.15.12:0/3789087657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4cc40a3270 0x7f4cc4124a40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4cb800b550 tx=0x7f4cb800ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:39.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.320+0000 7f4cc2ffd640 1 --2- 172.21.15.12:0/3789087657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4cc40a3270 0x7f4cc4124a40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4cb800b550 tx=0x7f4cb800ba10 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7f4cb8005070 mon_map magic: 0 2024-09-11T01:48:39.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.320+0000 7f4cabfff640 1 -- 172.21.15.12:0/3789087657 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8005070 con 0x7f4cc40a3270 2024-09-11T01:48:39.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.320+0000 7f4cabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:39.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.320+0000 7f4cabfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:48:39.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.320+0000 7f4cabfff640 10 monclient: dump: 2024-09-11T01:48:39.334 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:39.334 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:39.334 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:39.334 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:39.334 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:39.334 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:39.334 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:39.334 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:39.334 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:39.334 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:39.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.320+0000 7f4cabfff640 10 monclient: _finish_auth 0 2024-09-11T01:48:39.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.320+0000 7f4cabfff640 10 monclient: _check_auth_tickets 2024-09-11T01:48:39.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.320+0000 7f4cabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:39.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.320+0000 7f4cabfff640 1 -- 172.21.15.12:0/3789087657 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cb8015070 con 0x7f4cc40a3270 2024-09-11T01:48:39.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.320+0000 7f4cc9f64640 5 monclient: authenticate success, global_id 214158 2024-09-11T01:48:39.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.320+0000 7f4cc9f64640 10 monclient: _renew_subs 2024-09-11T01:48:39.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.320+0000 7f4cc9f64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:39.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.320+0000 7f4cc9f64640 1 -- 172.21.15.12:0/3789087657 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cc409f680 con 0x7f4cc40a3270 2024-09-11T01:48:39.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.320+0000 7f4cabfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:39.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.320+0000 7f4cc1ffb640 4 set_mon_vals no callback set 2024-09-11T01:48:39.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.320+0000 7f4cc9f64640 10 monclient: _renew_subs 2024-09-11T01:48:39.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.320+0000 7f4cc9f64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:39.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.320+0000 7f4cc9f64640 1 -- 172.21.15.12:0/3789087657 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cc4128580 con 0x7f4cc40a3270 2024-09-11T01:48:39.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.321+0000 7f4cabfff640 1 -- 172.21.15.12:0/3789087657 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8005450 con 0x7f4cc40a3270 2024-09-11T01:48:39.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.321+0000 7f4cabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:39.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.321+0000 7f4cabfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:48:39.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.321+0000 7f4cabfff640 10 monclient: dump: 2024-09-11T01:48:39.336 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:39.336 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:39.336 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:39.336 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:39.336 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:39.336 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:39.336 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:39.336 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:39.336 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:39.337 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:39.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.321+0000 7f4cabfff640 1 -- 172.21.15.12:0/3789087657 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f4cb8005630 con 0x7f4cc40a3270 2024-09-11T01:48:39.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.321+0000 7f4cc9f64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:39.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.321+0000 7f4cc9f64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:39.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.321+0000 7f4cc9f64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:39.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.321+0000 7f4cc9f64640 1 -- 172.21.15.12:0/3789087657 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cc4138c30 con 0x7f4cc40a3270 2024-09-11T01:48:39.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.322+0000 7f4cabfff640 1 --2- 172.21.15.12:0/3789087657 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4cb003e1c0 0x7f4cb0040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:39.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.322+0000 7f4cc37fe640 1 --2- 172.21.15.12:0/3789087657 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4cb003e1c0 0x7f4cb0040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:39.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.322+0000 7f4cc37fe640 10 monclient: get_auth_request con 0x7f4cb003e1c0 auth_method 0 2024-09-11T01:48:39.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.325+0000 7f4cc37fe640 1 --2- 172.21.15.12:0/3789087657 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4cb003e1c0 0x7f4cb0040680 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4031000 tx=0x7f4cb402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:39.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.326+0000 7f4cabfff640 1 -- 172.21.15.12:0/3789087657 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(642..642 src has 507..642) ==== 4831+0+0 (secure 0 0 0) 0x7f4cb8052b30 con 0x7f4cc40a3270 2024-09-11T01:48:39.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.943+0000 7f4cabfff640 1 -- 172.21.15.12:0/3789087657 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4cb8054e20 con 0x7f4cc40a3270 2024-09-11T01:48:39.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.943+0000 7f4cabfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:39.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:39.943+0000 7f4cabfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:48:40.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:40.208+0000 7f4cc9f64640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:48:40.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:40.208+0000 7f4cc9f64640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:48:40.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:40.208+0000 7f4cc9f64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:40.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:40.208+0000 7f4cc9f64640 1 -- 172.21.15.12:0/3789087657 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4cc4127060 con 0x7f4cc40a3270 2024-09-11T01:48:40.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:40.209+0000 7f4cabfff640 1 -- 172.21.15.12:0/3789087657 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7f4cb8051500 con 0x7f4cc40a3270 2024-09-11T01:48:40.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:40.209+0000 7f4cabfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:48:40.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:40.209+0000 7f4cabfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:48:40.211 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":182,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:48:40.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:40.211+0000 7f4cc9f64640 1 -- 172.21.15.12:0/3789087657 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4cb003e1c0 msgr2=0x7f4cb0040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:40.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:40.211+0000 7f4cc9f64640 1 --2- 172.21.15.12:0/3789087657 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4cb003e1c0 0x7f4cb0040680 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4031000 tx=0x7f4cb402e000 comp rx=0 tx=0).stop 2024-09-11T01:48:40.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:40.211+0000 7f4cc9f64640 10 monclient: shutdown 2024-09-11T01:48:40.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:40.211+0000 7f4cc9f64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:40.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:40.211+0000 7f4cc9f64640 1 -- 172.21.15.12:0/3789087657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4cc40a3270 msgr2=0x7f4cc4124a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:40.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:40.211+0000 7f4cc9f64640 1 --2- 172.21.15.12:0/3789087657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4cc40a3270 0x7f4cc4124a40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4cb800b550 tx=0x7f4cb800ba10 comp rx=0 tx=0).stop 2024-09-11T01:48:40.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:40.211+0000 7f4cc9f64640 1 -- 172.21.15.12:0/3789087657 shutdown_connections 2024-09-11T01:48:40.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:40.211+0000 7f4cc9f64640 1 --2- 172.21.15.12:0/3789087657 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4cc4138040 0x7f4cc4124f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:40.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:40.211+0000 7f4cc9f64640 1 --2- 172.21.15.12:0/3789087657 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4cb003e1c0 0x7f4cb0040680 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:40.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:40.211+0000 7f4cc9f64640 1 --2- 172.21.15.12:0/3789087657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4cc40a3270 0x7f4cc4124a40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:40.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:40.211+0000 7f4cc9f64640 1 --2- 172.21.15.12:0/3789087657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4cc40a2e90 0x7f4cc41cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:40.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:40.211+0000 7f4cc9f64640 1 -- 172.21.15.12:0/3789087657 >> 172.21.15.12:0/3789087657 conn(0x7f4cc40a8490 msgr2=0x7f4cc4072090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:40.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:40.211+0000 7f4cc9f64640 1 -- 172.21.15.12:0/3789087657 shutdown_connections 2024-09-11T01:48:40.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:40.211+0000 7f4cc9f64640 1 -- 172.21.15.12:0/3789087657 wait complete. 2024-09-11T01:48:43.224 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:48:43.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.335+0000 7f88226b6640 10 monclient: get_monmap_and_config 2024-09-11T01:48:43.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.335+0000 7f88226b6640 10 monclient: build_initial_monmap 2024-09-11T01:48:43.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.335+0000 7f88226b6640 10 monclient: monmap: 2024-09-11T01:48:43.337 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:43.337 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:43.338 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:48:43.336420+0000 2024-09-11T01:48:43.338 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:48:43.336420+0000 2024-09-11T01:48:43.338 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:43.338 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:43.338 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:48:43.338 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:48:43.338 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:48:43.339 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:48:43.339 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:48:43.339 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:43.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.336+0000 7f88226b6640 1 Processor -- start 2024-09-11T01:48:43.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.336+0000 7f88226b6640 1 -- start start 2024-09-11T01:48:43.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.336+0000 7f88226b6640 10 monclient: init 2024-09-11T01:48:43.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f88226b6640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:43.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f88226b6640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-11T01:48:43.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f88226b6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f881c09d790 0x7f881c09db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:43.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f88226b6640 10 monclient(hunting): picked mon.noname-b con 0x7f881c09d790 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:43.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f88226b6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f881c09e0b0 addr v1:172.21.15.12:6789/0 2024-09-11T01:48:43.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f88226b6640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f881c09e4d0 0x7f881c135170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:43.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f88226b6640 10 monclient(hunting): picked mon.noname-a con 0x7f881c09e4d0 addr v2:172.21.15.12:3300/0 2024-09-11T01:48:43.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f88226b6640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f881c0a1da0 con 0x7f881c09e0b0 2024-09-11T01:48:43.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f88226b6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:43.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f881bfff640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f881c09d790 msgr2=0x7f881c09db70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:48:43.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f88226b6640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f881c0a0760 con 0x7f881c09e4d0 2024-09-11T01:48:43.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f88226b6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:43.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f88226b6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f881c09e8f0 con 0x7f881c09d790 2024-09-11T01:48:43.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f881bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f881c09d790 0x7f881c09db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:48:43.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f88226b6640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:43.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f88226b6640 10 monclient(hunting): authenticate will time out at 1929.779246s 2024-09-11T01:48:43.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f8820c53640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f881c09e4d0 0x7f881c135170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:43.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f881b7fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f881c09e0b0 0x7f881c134a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:34758/0 (socket says 172.21.15.12:34758) 2024-09-11T01:48:43.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f881b7fe640 1 -- 172.21.15.12:0/4056102057 learned_addr learned my addr 172.21.15.12:0/4056102057 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:48:43.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f8820c53640 10 monclient(hunting): get_auth_request con 0x7f881c09e4d0 auth_method 0 2024-09-11T01:48:43.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f8820c53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:43.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f8820c53640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:43.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f8820c53640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:43.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f8820c53640 1 -- 172.21.15.12:0/4056102057 >> v2:172.21.15.12:3300/0 conn(0x7f881c09e4d0 msgr2=0x7f881c135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:48:43.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f8820c53640 1 -- 172.21.15.12:0/4056102057 >> v2:172.21.15.12:3300/0 conn(0x7f881c09e4d0 msgr2=0x7f881c135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:48:43.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f8820c53640 1 --2- 172.21.15.12:0/4056102057 >> v2:172.21.15.12:3300/0 conn(0x7f881c09e4d0 0x7f881c135170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:48:43.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.337+0000 7f8820c53640 1 --2- 172.21.15.12:0/4056102057 >> v2:172.21.15.12:3300/0 conn(0x7f881c09e4d0 0x7f881c135170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:48:43.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.339+0000 7f881b7fe640 1 -- 172.21.15.12:0/4056102057 >> v1:172.21.15.12:6789/0 conn(0x7f881c09e0b0 legacy=0x7f881c134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:43.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.538+0000 7f881bfff640 1 -- 172.21.15.12:0/4056102057 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f881c09d790 msgr2=0x7f881c09db70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:48:43.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.538+0000 7f881bfff640 1 --2- 172.21.15.12:0/4056102057 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f881c09d790 0x7f881c09db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:48:43.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.539+0000 7f8820c53640 1 --2- 172.21.15.12:0/4056102057 >> v2:172.21.15.12:3300/0 conn(0x7f881c09e4d0 0x7f881c135170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:43.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.539+0000 7f8820c53640 10 monclient(hunting): get_auth_request con 0x7f881c09e4d0 auth_method 0 2024-09-11T01:48:43.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.539+0000 7f8820c53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:43.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.539+0000 7f8820c53640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:43.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.539+0000 7f8820c53640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:48:43.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.539+0000 7f8820c53640 1 -- 172.21.15.12:0/4056102057 >> v2:172.21.15.12:3300/0 conn(0x7f881c09e4d0 msgr2=0x7f881c135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:48:43.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.539+0000 7f8820c53640 1 -- 172.21.15.12:0/4056102057 >> v2:172.21.15.12:3300/0 conn(0x7f881c09e4d0 msgr2=0x7f881c135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:48:43.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.539+0000 7f8820c53640 1 --2- 172.21.15.12:0/4056102057 >> v2:172.21.15.12:3300/0 conn(0x7f881c09e4d0 0x7f881c135170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:48:43.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.539+0000 7f8820c53640 1 --2- 172.21.15.12:0/4056102057 >> v2:172.21.15.12:3300/0 conn(0x7f881c09e4d0 0x7f881c135170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:48:43.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.939+0000 7f881bfff640 1 -- 172.21.15.12:0/4056102057 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f881c09d790 msgr2=0x7f881c09db70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:48:43.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.939+0000 7f881bfff640 1 --2- 172.21.15.12:0/4056102057 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f881c09d790 0x7f881c09db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:48:43.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.941+0000 7f8820c53640 1 --2- 172.21.15.12:0/4056102057 >> v2:172.21.15.12:3300/0 conn(0x7f881c09e4d0 0x7f881c135170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:43.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.941+0000 7f8820c53640 10 monclient(hunting): get_auth_request con 0x7f881c09e4d0 auth_method 0 2024-09-11T01:48:43.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.941+0000 7f8820c53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:43.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.941+0000 7f8820c53640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:43.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.941+0000 7f8820c53640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:48:43.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.941+0000 7f8820c53640 1 -- 172.21.15.12:0/4056102057 >> v2:172.21.15.12:3300/0 conn(0x7f881c09e4d0 msgr2=0x7f881c135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:48:43.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.941+0000 7f8820c53640 1 -- 172.21.15.12:0/4056102057 >> v2:172.21.15.12:3300/0 conn(0x7f881c09e4d0 msgr2=0x7f881c135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:48:43.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.941+0000 7f8820c53640 1 --2- 172.21.15.12:0/4056102057 >> v2:172.21.15.12:3300/0 conn(0x7f881c09e4d0 0x7f881c135170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:48:43.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:43.941+0000 7f8820c53640 1 --2- 172.21.15.12:0/4056102057 >> v2:172.21.15.12:3300/0 conn(0x7f881c09e4d0 0x7f881c135170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:48:44.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:44.740+0000 7f881bfff640 1 -- 172.21.15.12:0/4056102057 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f881c09d790 msgr2=0x7f881c09db70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:48:44.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:44.740+0000 7f881bfff640 1 --2- 172.21.15.12:0/4056102057 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f881c09d790 0x7f881c09db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:48:44.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:44.742+0000 7f8820c53640 1 --2- 172.21.15.12:0/4056102057 >> v2:172.21.15.12:3300/0 conn(0x7f881c09e4d0 0x7f881c135170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:44.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:44.742+0000 7f8820c53640 10 monclient(hunting): get_auth_request con 0x7f881c09e4d0 auth_method 0 2024-09-11T01:48:44.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:44.742+0000 7f8820c53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:44.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:44.742+0000 7f8820c53640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:44.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:44.742+0000 7f8820c53640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:48:44.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:44.742+0000 7f8820c53640 1 -- 172.21.15.12:0/4056102057 >> v2:172.21.15.12:3300/0 conn(0x7f881c09e4d0 msgr2=0x7f881c135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:48:44.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:44.742+0000 7f8820c53640 1 -- 172.21.15.12:0/4056102057 >> v2:172.21.15.12:3300/0 conn(0x7f881c09e4d0 msgr2=0x7f881c135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:48:44.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:44.742+0000 7f8820c53640 1 --2- 172.21.15.12:0/4056102057 >> v2:172.21.15.12:3300/0 conn(0x7f881c09e4d0 0x7f881c135170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:48:44.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:44.742+0000 7f8820c53640 1 --2- 172.21.15.12:0/4056102057 >> v2:172.21.15.12:3300/0 conn(0x7f881c09e4d0 0x7f881c135170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:48:46.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.337+0000 7f8819ffb640 10 monclient(hunting): tick 2024-09-11T01:48:46.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.337+0000 7f8819ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:48:46.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.337+0000 7f8819ffb640 1 monclient(hunting): continuing hunt 2024-09-11T01:48:46.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.337+0000 7f8819ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:48:46.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.337+0000 7f8819ffb640 1 -- 172.21.15.12:0/4056102057 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f881c09d790 msgr2=0x7f881c09db70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:48:46.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.337+0000 7f8819ffb640 1 --2- 172.21.15.12:0/4056102057 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f881c09d790 0x7f881c09db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:46.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.337+0000 7f8819ffb640 1 -- 172.21.15.12:0/4056102057 >> v2:172.21.15.12:3300/0 conn(0x7f881c09e4d0 msgr2=0x7f881c135170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:48:46.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.337+0000 7f8819ffb640 1 --2- 172.21.15.12:0/4056102057 >> v2:172.21.15.12:3300/0 conn(0x7f881c09e4d0 0x7f881c135170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:46.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.337+0000 7f8819ffb640 1 -- 172.21.15.12:0/4056102057 >> v1:172.21.15.12:6789/0 conn(0x7f881c09e0b0 legacy=0x7f881c134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:46.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.337+0000 7f8819ffb640 10 monclient: _add_conns ranks=[1,4] 2024-09-11T01:48:46.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.337+0000 7f8819ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8808005970 addr v1:172.21.15.71:6789/0 2024-09-11T01:48:46.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.337+0000 7f8819ffb640 1 --2- 172.21.15.12:0/4056102057 >> v2:172.21.15.71:3300/0 conn(0x7f88080096a0 0x7f880800bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:46.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.337+0000 7f8819ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f88080096a0 addr v2:172.21.15.71:3300/0 2024-09-11T01:48:46.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.337+0000 7f8819ffb640 1 -- 172.21.15.12:0/4056102057 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f881c0a1fa0 con 0x7f8808005970 2024-09-11T01:48:46.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.337+0000 7f8819ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:46.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.337+0000 7f8819ffb640 1 -- 172.21.15.12:0/4056102057 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f881c0a0760 con 0x7f88080096a0 2024-09-11T01:48:46.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.337+0000 7f8819ffb640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:46.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.337+0000 7f881b7fe640 1 --2- 172.21.15.12:0/4056102057 >> v2:172.21.15.71:3300/0 conn(0x7f88080096a0 0x7f880800bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:46.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.337+0000 7f881b7fe640 10 monclient(hunting): get_auth_request con 0x7f88080096a0 auth_method 0 2024-09-11T01:48:46.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.337+0000 7f881b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:46.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.337+0000 7f881b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:46.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.337+0000 7f881b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:46.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.338+0000 7f881b7fe640 1 -- 172.21.15.12:0/4056102057 >> v2:172.21.15.71:3300/0 conn(0x7f88080096a0 msgr2=0x7f880800bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:48:46.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.338+0000 7f881b7fe640 1 -- 172.21.15.12:0/4056102057 >> v2:172.21.15.71:3300/0 conn(0x7f88080096a0 msgr2=0x7f880800bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:48:46.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.338+0000 7f881b7fe640 1 --2- 172.21.15.12:0/4056102057 >> v2:172.21.15.71:3300/0 conn(0x7f88080096a0 0x7f880800bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:48:46.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.338+0000 7f881b7fe640 1 --2- 172.21.15.12:0/4056102057 >> v2:172.21.15.71:3300/0 conn(0x7f88080096a0 0x7f880800bb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:48:46.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.340+0000 7f881bfff640 1 -- 172.21.15.12:0/4056102057 >> v1:172.21.15.71:6789/0 conn(0x7f8808005970 legacy=0x7f8808007df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:46.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.511+0000 7f881affd640 1 -- 172.21.15.12:0/4056102057 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 671635102 0 0) 0x7f881c0a1fa0 con 0x7f8808005970 2024-09-11T01:48:46.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.511+0000 7f881affd640 10 monclient(hunting): _negotiate 2024-09-11T01:48:46.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.511+0000 7f881affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:46.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.511+0000 7f881affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:46.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.511+0000 7f881affd640 10 monclient(hunting): my global_id is 134800 2024-09-11T01:48:46.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.511+0000 7f881affd640 1 -- 172.21.15.12:0/4056102057 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8804003520 con 0x7f8808005970 2024-09-11T01:48:46.514 INFO:tasks.rados.rados.0.smithi071.stdout:2852: done (3 left) 2024-09-11T01:48:46.514 INFO:tasks.rados.rados.0.smithi071.stdout:2854: done (2 left) 2024-09-11T01:48:46.514 INFO:tasks.rados.rados.0.smithi071.stdout:2856: done (1 left) 2024-09-11T01:48:46.514 INFO:tasks.rados.rados.0.smithi071.stdout:2857: rollback oid 17 current snap is 322 2024-09-11T01:48:46.514 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 17 to 294 2024-09-11T01:48:46.514 INFO:tasks.rados.rados.0.smithi071.stdout:2855: finishing write tid 1 to smithi07134777-18 2024-09-11T01:48:46.514 INFO:tasks.rados.rados.0.smithi071.stdout:2855: finishing write tid 2 to smithi07134777-18 2024-09-11T01:48:46.514 INFO:tasks.rados.rados.0.smithi071.stdout:2855: finishing write tid 3 to smithi07134777-18 2024-09-11T01:48:46.515 INFO:tasks.rados.rados.0.smithi071.stdout:2855: finishing write tid 4 to smithi07134777-18 2024-09-11T01:48:46.515 INFO:tasks.rados.rados.0.smithi071.stdout:2855: finishing write tid 5 to smithi07134777-18 2024-09-11T01:48:46.515 INFO:tasks.rados.rados.0.smithi071.stdout:2855: oid 18 updating version 0 to 1304 2024-09-11T01:48:46.515 INFO:tasks.rados.rados.0.smithi071.stdout:2855: oid 18 version 1304 is already newer than 1302 2024-09-11T01:48:46.515 INFO:tasks.rados.rados.0.smithi071.stdout:2855: oid 18 version 1304 is already newer than 1301 2024-09-11T01:48:46.515 INFO:tasks.rados.rados.0.smithi071.stdout:2855: oid 18 version 1304 is already newer than 1303 2024-09-11T01:48:46.515 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 1304 (ObjNum 1010 snap 322 seq_num 1010) dirty exists 2024-09-11T01:48:46.515 INFO:tasks.rados.rados.0.smithi071.stdout:2855: left oid 18 (ObjNum 1010 snap 322 seq_num 1010) 2024-09-11T01:48:46.515 INFO:tasks.rados.rados.0.smithi071.stdout:2855: done (1 left) 2024-09-11T01:48:46.515 INFO:tasks.rados.rados.0.smithi071.stdout:2858: read oid 25 snap -1 2024-09-11T01:48:46.515 INFO:tasks.rados.rados.0.smithi071.stdout:2858: expect (ObjNum 723 snap 224 seq_num 723) 2024-09-11T01:48:46.515 INFO:tasks.rados.rados.0.smithi071.stdout:2859: write oid 34 current snap is 322 2024-09-11T01:48:46.515 INFO:tasks.rados.rados.0.smithi071.stdout:2859: seq_num 1011 ranges {591877=635335,1628354=709677} 2024-09-11T01:48:46.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.511+0000 7f881affd640 1 -- 172.21.15.12:0/4056102057 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3397794369 0 0) 0x7f8804003520 con 0x7f8808005970 2024-09-11T01:48:46.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.512+0000 7f881affd640 1 -- 172.21.15.12:0/4056102057 >> v2:172.21.15.71:3300/0 conn(0x7f88080096a0 msgr2=0x7f880800bb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:48:46.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.512+0000 7f881affd640 1 --2- 172.21.15.12:0/4056102057 >> v2:172.21.15.71:3300/0 conn(0x7f88080096a0 0x7f880800bb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:46.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.512+0000 7f881affd640 10 monclient: _finish_hunting 0 2024-09-11T01:48:46.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.512+0000 7f881affd640 1 monclient: found mon.noname-c-legacy 2024-09-11T01:48:46.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.512+0000 7f881affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:46.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.512+0000 7f881affd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-09-11T01:48:46.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.512+0000 7f881affd640 1 -- 172.21.15.12:0/4056102057 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f880800d2f0 con 0x7f8808005970 2024-09-11T01:48:46.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.512+0000 7f881affd640 10 monclient: _finish_auth 0 2024-09-11T01:48:46.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.512+0000 7f881affd640 10 monclient: _check_auth_tickets 2024-09-11T01:48:46.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.512+0000 7f881affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:46.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.512+0000 7f881affd640 1 -- 172.21.15.12:0/4056102057 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f880c004600 con 0x7f8808005970 2024-09-11T01:48:46.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.512+0000 7f88226b6640 5 monclient: authenticate success, global_id 134800 2024-09-11T01:48:46.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.512+0000 7f88226b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:48:46.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.512+0000 7f881affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:46.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.512+0000 7f881affd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-11T01:48:46.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.512+0000 7f881affd640 10 monclient: dump: 2024-09-11T01:48:46.518 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:46.518 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:46.518 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:46.518 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:46.518 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:46.518 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:46.519 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:46.519 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:46.519 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:46.519 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:46.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.512+0000 7f881affd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-09-11T01:48:46.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.512+0000 7f881affd640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:46.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.512+0000 7f881affd640 1 -- 172.21.15.12:0/4056102057 >> v1:172.21.15.71:6789/0 conn(0x7f8808005970 legacy=0x7f8808007df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:46.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.512+0000 7f8820c53640 1 -- 172.21.15.12:0/4056102057 reap_dead start 2024-09-11T01:48:46.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.512+0000 7f881affd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:48:46.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.512+0000 7f881affd640 1 --2- 172.21.15.12:0/4056102057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88040039b0 0x7f8804003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:46.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.512+0000 7f881affd640 10 monclient(hunting): picked mon.a con 0x7f88040039b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:48:46.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.513+0000 7f881affd640 1 --2- 172.21.15.12:0/4056102057 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8804004800 0x7f8804004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:46.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.513+0000 7f881b7fe640 1 -- 172.21.15.12:0/4056102057 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8804004800 msgr2=0x7f8804004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:48:46.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.513+0000 7f881b7fe640 1 --2- 172.21.15.12:0/4056102057 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8804004800 0x7f8804004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:48:46.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.513+0000 7f881affd640 10 monclient(hunting): picked mon.c con 0x7f8804004800 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:46.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.513+0000 7f881affd640 1 --2- 172.21.15.12:0/4056102057 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8804005230 0x7f880400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:46.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.513+0000 7f881affd640 10 monclient(hunting): picked mon.b con 0x7f8804005230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:48:46.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.513+0000 7f881affd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:46.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.513+0000 7f881affd640 1 -- 172.21.15.12:0/4056102057 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f881c0a0760 con 0x7f88040039b0 2024-09-11T01:48:46.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.513+0000 7f881affd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:46.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.513+0000 7f881affd640 1 -- 172.21.15.12:0/4056102057 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f880400c190 con 0x7f8804005230 2024-09-11T01:48:46.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.513+0000 7f881affd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:46.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.513+0000 7f881affd640 1 -- 172.21.15.12:0/4056102057 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f880400c390 con 0x7f8804004800 2024-09-11T01:48:46.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.513+0000 7f881affd640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:46.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.513+0000 7f881affd640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:48:46.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.513+0000 7f88226b6640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:48:46.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.514+0000 7f8820c53640 1 --2- 172.21.15.12:0/4056102057 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8804005230 0x7f880400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:46.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.514+0000 7f8820c53640 10 monclient(hunting): get_auth_request con 0x7f8804005230 auth_method 0 2024-09-11T01:48:46.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.514+0000 7f8820c53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:46.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.514+0000 7f8820c53640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:46.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.514+0000 7f8820c53640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:48:46.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.514+0000 7f8820c53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:46.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.514+0000 7f8820c53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:46.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.514+0000 7f8820c53640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:48:46.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.514+0000 7f881bfff640 1 --2- 172.21.15.12:0/4056102057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88040039b0 0x7f8804003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:46.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.514+0000 7f881bfff640 10 monclient(hunting): get_auth_request con 0x7f88040039b0 auth_method 0 2024-09-11T01:48:46.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.514+0000 7f881bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:46.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.514+0000 7f881bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:46.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.514+0000 7f881bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:48:46.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.515+0000 7f881bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:46.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.515+0000 7f881bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:46.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.515+0000 7f881bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:48:46.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.515+0000 7f8820c53640 10 monclient(hunting): handle_auth_done global_id 134800 payload 293 2024-09-11T01:48:46.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.515+0000 7f8820c53640 1 -- 172.21.15.12:0/4056102057 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8804004800 msgr2=0x7f8804004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:48:46.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.515+0000 7f8820c53640 1 --2- 172.21.15.12:0/4056102057 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8804004800 0x7f8804004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:46.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.515+0000 7f8820c53640 1 -- 172.21.15.12:0/4056102057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88040039b0 msgr2=0x7f8804003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:48:46.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.515+0000 7f8820c53640 1 --2- 172.21.15.12:0/4056102057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88040039b0 0x7f8804003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:46.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.515+0000 7f8820c53640 10 monclient: _finish_hunting 0 2024-09-11T01:48:46.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.515+0000 7f8820c53640 1 monclient: found mon.b 2024-09-11T01:48:46.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.515+0000 7f8820c53640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:46.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.515+0000 7f8820c53640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:48:46.523 INFO:tasks.rados.rados.0.smithi071.stdout:2859: writing smithi07134777-34 from 591877 to 1227212 tid 1 2024-09-11T01:48:46.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.515+0000 7f8820c53640 1 -- 172.21.15.12:0/4056102057 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8804003520 con 0x7f8804005230 2024-09-11T01:48:46.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.515+0000 7f8820c53640 10 monclient: _finish_auth 0 2024-09-11T01:48:46.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.515+0000 7f8820c53640 10 monclient: _check_auth_tickets 2024-09-11T01:48:46.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.515+0000 7f8820c53640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:46.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.519+0000 7f8820c53640 1 -- 172.21.15.12:0/4056102057 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8804005230 msgr2=0x7f880400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:46.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.519+0000 7f8820c53640 1 --2- 172.21.15.12:0/4056102057 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8804005230 0x7f880400b9c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f881c0a05a0 tx=0x7f8804005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9dd7876eea9d6089 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:46.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.521+0000 7f881affd640 1 -- 172.21.15.12:0/4056102057 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8814031070 con 0x7f8804005230 2024-09-11T01:48:46.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.521+0000 7f881affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:46.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.521+0000 7f881affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:48:46.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.521+0000 7f881affd640 10 monclient: dump: 2024-09-11T01:48:46.525 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:46.526 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:46.526 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:46.526 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:46.526 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:46.526 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:46.526 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:46.526 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:46.526 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:46.526 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:46.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.521+0000 7f88226b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:48:46.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.521+0000 7f881affd640 1 -- 172.21.15.12:0/4056102057 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f881402b960 con 0x7f8804005230 2024-09-11T01:48:46.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.521+0000 7f881affd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:46.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.521+0000 7f881affd640 1 -- 172.21.15.12:0/4056102057 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8814004040 con 0x7f8804005230 2024-09-11T01:48:46.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.521+0000 7f88226b6640 10 monclient: get_monmap_and_config success 2024-09-11T01:48:46.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.522+0000 7f88226b6640 4 set_mon_vals no callback set 2024-09-11T01:48:46.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.522+0000 7f881affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:46.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.522+0000 7f881affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:48:46.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.522+0000 7f881affd640 10 monclient: dump: 2024-09-11T01:48:46.527 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:46.527 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:46.527 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:46.527 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:46.527 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:46.527 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:46.527 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:46.527 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:46.527 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:46.528 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:46.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.523+0000 7f88226b6640 10 monclient: shutdown 2024-09-11T01:48:46.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.523+0000 7f88226b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:46.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.523+0000 7f88226b6640 1 -- 172.21.15.12:0/4056102057 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8804005230 msgr2=0x7f880400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:46.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.523+0000 7f88226b6640 1 --2- 172.21.15.12:0/4056102057 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8804005230 0x7f880400b9c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f881c0a05a0 tx=0x7f8804005830 comp rx=0 tx=0).stop 2024-09-11T01:48:46.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.523+0000 7f88226b6640 1 -- 172.21.15.12:0/4056102057 shutdown_connections 2024-09-11T01:48:46.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.523+0000 7f88226b6640 1 --2- 172.21.15.12:0/4056102057 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8804005230 0x7f880400b9c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:46.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.523+0000 7f88226b6640 1 --2- 172.21.15.12:0/4056102057 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8804004800 0x7f8804004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:46.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.523+0000 7f88226b6640 1 --2- 172.21.15.12:0/4056102057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88040039b0 0x7f8804003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:46.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.523+0000 7f88226b6640 1 -- 172.21.15.12:0/4056102057 >> 172.21.15.12:0/4056102057 conn(0x7f881c0a2d90 msgr2=0x7f881c0a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:46.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.523+0000 7f88226b6640 1 -- 172.21.15.12:0/4056102057 shutdown_connections 2024-09-11T01:48:46.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.523+0000 7f88226b6640 1 -- 172.21.15.12:0/4056102057 wait complete. 2024-09-11T01:48:46.529 INFO:tasks.rados.rados.0.smithi071.stdout:2859: writing smithi07134777-34 from 1628354 to 2338031 tid 2 2024-09-11T01:48:46.529 INFO:tasks.rados.rados.0.smithi071.stdout:2860: delete oid 15 current snap is 322 2024-09-11T01:48:46.530 INFO:tasks.rados.rados.0.smithi071.stdout:2858: expect (ObjNum 723 snap 224 seq_num 723) 2024-09-11T01:48:46.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.524+0000 7f88226b6640 10 monclient: build_initial_monmap 2024-09-11T01:48:46.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.524+0000 7f88226b6640 10 monclient: monmap: 2024-09-11T01:48:46.532 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:46.533 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:48:46.533 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:48:46.533 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:48:46.533 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:46.533 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:46.533 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:48:46.533 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:48:46.536 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:48:46.536 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:46.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.525+0000 7f88226b6640 1 Processor -- start 2024-09-11T01:48:46.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.525+0000 7f88226b6640 1 -- start start 2024-09-11T01:48:46.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.525+0000 7f88226b6640 10 monclient: init 2024-09-11T01:48:46.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.526+0000 7f88226b6640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:46.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.526+0000 7f88226b6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:48:46.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.526+0000 7f88226b6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88040039b0 0x7f881c1ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:46.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.526+0000 7f88226b6640 10 monclient(hunting): picked mon.noname-a con 0x7f88040039b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:48:46.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.526+0000 7f88226b6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8804004800 0x7f881c1cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:46.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.526+0000 7f88226b6640 10 monclient(hunting): picked mon.noname-c con 0x7f8804004800 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:46.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.526+0000 7f88226b6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8804005230 0x7f881c124640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:46.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.526+0000 7f881b7fe640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8804004800 msgr2=0x7f881c1cd030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:48:46.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.526+0000 7f881b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8804004800 0x7f881c1cd030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:48:46.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.526+0000 7f88226b6640 10 monclient(hunting): picked mon.noname-b con 0x7f8804005230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:48:46.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.526+0000 7f88226b6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:46.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.526+0000 7f88226b6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f881c0a23a0 con 0x7f88040039b0 2024-09-11T01:48:46.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.526+0000 7f88226b6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:46.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.526+0000 7f88226b6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f881c0a2220 con 0x7f8804005230 2024-09-11T01:48:46.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.526+0000 7f88226b6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:46.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.526+0000 7f88226b6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f881c09cca0 con 0x7f8804004800 2024-09-11T01:48:46.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.526+0000 7f88226b6640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:46.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.526+0000 7f88226b6640 10 monclient(hunting): authenticate will time out at 1932.968864s 2024-09-11T01:48:46.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f881bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88040039b0 0x7f881c1ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:46.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f881bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88040039b0 0x7f881c1ccaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42266/0 (socket says 172.21.15.12:42266) 2024-09-11T01:48:46.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f881bfff640 1 -- 172.21.15.12:0/3415743366 learned_addr learned my addr 172.21.15.12:0/3415743366 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:48:46.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f881bfff640 10 monclient(hunting): get_auth_request con 0x7f88040039b0 auth_method 0 2024-09-11T01:48:46.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f881bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:46.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f881bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:46.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f881bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:46.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f8820c53640 1 --2- 172.21.15.12:0/3415743366 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8804005230 0x7f881c124640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:46.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f8820c53640 10 monclient(hunting): get_auth_request con 0x7f8804005230 auth_method 0 2024-09-11T01:48:46.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f8820c53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:46.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f8820c53640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:46.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f8820c53640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:46.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f881bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:46.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f881bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:46.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f881bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:46.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f8820c53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:46.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f8820c53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:46.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f8820c53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:46.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f881bfff640 10 monclient(hunting): handle_auth_done global_id 214161 payload 824 2024-09-11T01:48:46.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f881bfff640 1 -- 172.21.15.12:0/3415743366 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8804004800 msgr2=0x7f881c1cd030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:48:46.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f881bfff640 1 --2- 172.21.15.12:0/3415743366 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8804004800 0x7f881c1cd030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:46.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f881bfff640 1 -- 172.21.15.12:0/3415743366 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8804005230 msgr2=0x7f881c124640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:46.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f881bfff640 1 --2- 172.21.15.12:0/3415743366 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8804005230 0x7f881c124640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:46.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f881bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:48:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f881bfff640 1 monclient: found mon.noname-a 2024-09-11T01:48:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f881bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f881bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:48:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.527+0000 7f881bfff640 1 -- 172.21.15.12:0/3415743366 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f881c126100 con 0x7f88040039b0 2024-09-11T01:48:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.530+0000 7f881bfff640 1 -- 172.21.15.12:0/3415743366 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88040039b0 msgr2=0x7f881c1ccaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.530+0000 7f881bfff640 1 --2- 172.21.15.12:0/3415743366 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88040039b0 0x7f881c1ccaf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f880c003e70 tx=0x7f880c00ae70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.530+0000 7f8818ff9640 1 -- 172.21.15.12:0/3415743366 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c015070 con 0x7f88040039b0 2024-09-11T01:48:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.530+0000 7f8818ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.530+0000 7f8818ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:48:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.530+0000 7f8818ff9640 10 monclient: dump: 2024-09-11T01:48:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:46.540 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:46.541 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.530+0000 7f8818ff9640 10 monclient: _finish_auth 0 2024-09-11T01:48:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.530+0000 7f8818ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:48:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.530+0000 7f8818ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.530+0000 7f88226b6640 5 monclient: authenticate success, global_id 214161 2024-09-11T01:48:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.530+0000 7f88226b6640 10 monclient: _renew_subs 2024-09-11T01:48:46.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.530+0000 7f88226b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.530+0000 7f88226b6640 1 -- 172.21.15.12:0/3415743366 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f881c135a00 con 0x7f88040039b0 2024-09-11T01:48:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.530+0000 7f8818ff9640 1 -- 172.21.15.12:0/3415743366 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f880c00fd20 con 0x7f88040039b0 2024-09-11T01:48:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.530+0000 7f8818ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.530+0000 7f8818ff9640 1 -- 172.21.15.12:0/3415743366 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c011660 con 0x7f88040039b0 2024-09-11T01:48:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.530+0000 7f8818ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.530+0000 7f8818ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:48:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.530+0000 7f8818ff9640 10 monclient: dump: 2024-09-11T01:48:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:46.542 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:46.543 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:46.543 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:46.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.530+0000 7f881a7fc640 4 set_mon_vals no callback set 2024-09-11T01:48:46.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.530+0000 7f88226b6640 10 monclient: _renew_subs 2024-09-11T01:48:46.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.530+0000 7f88226b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:46.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.530+0000 7f88226b6640 1 -- 172.21.15.12:0/3415743366 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f881c126c60 con 0x7f88040039b0 2024-09-11T01:48:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.532+0000 7f8818ff9640 1 -- 172.21.15.12:0/3415743366 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f880c011800 con 0x7f88040039b0 2024-09-11T01:48:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.532+0000 7f88226b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.532+0000 7f88226b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:46.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.532+0000 7f88226b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.532+0000 7f88226b6640 1 -- 172.21.15.12:0/3415743366 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f881c0a0980 con 0x7f88040039b0 2024-09-11T01:48:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.532+0000 7f8818ff9640 1 --2- 172.21.15.12:0/3415743366 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f880803eb70 0x7f8808041030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.532+0000 7f8818ff9640 1 -- 172.21.15.12:0/3415743366 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(643..643 src has 507..643) ==== 4847+0+0 (secure 0 0 0) 0x7f880c0537e0 con 0x7f88040039b0 2024-09-11T01:48:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.532+0000 7f881b7fe640 1 --2- 172.21.15.12:0/3415743366 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f880803eb70 0x7f8808041030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.532+0000 7f881b7fe640 10 monclient: get_auth_request con 0x7f880803eb70 auth_method 0 2024-09-11T01:48:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.535+0000 7f881b7fe640 1 --2- 172.21.15.12:0/3415743366 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f880803eb70 0x7f8808041030 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8810007fd0 tx=0x7f8810000e40 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.537+0000 7f8818ff9640 1 -- 172.21.15.12:0/3415743366 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f880c0103b0 con 0x7f88040039b0 2024-09-11T01:48:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.537+0000 7f8818ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:46.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.537+0000 7f8818ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:48:46.549 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1586114 2024-09-11T01:48:46.585 INFO:tasks.rados.rados.0.smithi071.stdout:2857: finishing rollback tid 0 to smithi07134777-17 2024-09-11T01:48:46.585 INFO:tasks.rados.rados.0.smithi071.stdout:2857: finishing rollback tid 1 to smithi07134777-17 2024-09-11T01:48:46.585 INFO:tasks.rados.rados.0.smithi071.stdout:2857: finishing rollback tid 2 to smithi07134777-17 2024-09-11T01:48:46.585 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 1774 (ObjNum 931 snap 293 seq_num 931) dirty exists 2024-09-11T01:48:46.585 INFO:tasks.rados.rados.0.smithi071.stdout:2859: finishing write tid 1 to smithi07134777-34 2024-09-11T01:48:46.585 INFO:tasks.rados.rados.0.smithi071.stdout:2859: finishing write tid 2 to smithi07134777-34 2024-09-11T01:48:46.585 INFO:tasks.rados.rados.0.smithi071.stdout:2859: finishing write tid 3 to smithi07134777-34 2024-09-11T01:48:46.585 INFO:tasks.rados.rados.0.smithi071.stdout:2859: finishing write tid 4 to smithi07134777-34 2024-09-11T01:48:46.585 INFO:tasks.rados.rados.0.smithi071.stdout:2859: oid 34 updating version 0 to 1915 2024-09-11T01:48:46.585 INFO:tasks.rados.rados.0.smithi071.stdout:2859: oid 34 updating version 1915 to 1916 2024-09-11T01:48:46.586 INFO:tasks.rados.rados.0.smithi071.stdout:2859: oid 34 updating version 1916 to 1917 2024-09-11T01:48:46.586 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 1917 (ObjNum 1011 snap 322 seq_num 1011) dirty exists 2024-09-11T01:48:46.586 INFO:tasks.rados.rados.0.smithi071.stdout:2859: left oid 34 (ObjNum 1011 snap 322 seq_num 1011) 2024-09-11T01:48:46.586 INFO:tasks.rados.rados.0.smithi071.stdout:2857: done (3 left) 2024-09-11T01:48:46.586 INFO:tasks.rados.rados.0.smithi071.stdout:2858: done (2 left) 2024-09-11T01:48:46.586 INFO:tasks.rados.rados.0.smithi071.stdout:2859: done (1 left) 2024-09-11T01:48:46.586 INFO:tasks.rados.rados.0.smithi071.stdout:2860: done (0 left) 2024-09-11T01:48:46.586 INFO:tasks.rados.rados.0.smithi071.stdout:2861: write oid 34 current snap is 322 2024-09-11T01:48:46.586 INFO:tasks.rados.rados.0.smithi071.stdout:2861: seq_num 1013 ranges {625491=502186,1861671=713346} 2024-09-11T01:48:46.590 INFO:tasks.rados.rados.0.smithi071.stdout:2861: writing smithi07134777-34 from 625491 to 1127677 tid 1 2024-09-11T01:48:46.598 INFO:tasks.rados.rados.0.smithi071.stdout:2861: writing smithi07134777-34 from 1861671 to 2575017 tid 2 2024-09-11T01:48:46.598 INFO:tasks.rados.rados.0.smithi071.stdout:2862: write (excl) oid 7 current snap is 322 2024-09-11T01:48:46.598 INFO:tasks.rados.rados.0.smithi071.stdout:2862: seq_num 1014 ranges {642298=577435,1920153=573357,2979150=63148} 2024-09-11T01:48:46.605 INFO:tasks.rados.rados.0.smithi071.stdout:2862: writing smithi07134777-7 from 642298 to 1219733 tid 1 2024-09-11T01:48:46.612 INFO:tasks.rados.rados.0.smithi071.stdout:2862: writing smithi07134777-7 from 1920153 to 2493510 tid 2 2024-09-11T01:48:46.615 INFO:tasks.rados.rados.0.smithi071.stdout:2862: writing smithi07134777-7 from 2979150 to 3042298 tid 3 2024-09-11T01:48:46.615 INFO:tasks.rados.rados.0.smithi071.stdout:2863: write oid 19 current snap is 322 2024-09-11T01:48:46.616 INFO:tasks.rados.rados.0.smithi071.stdout:2861: finishing write tid 1 to smithi07134777-34 2024-09-11T01:48:46.616 INFO:tasks.rados.rados.0.smithi071.stdout:2863: seq_num 1015 ranges {659105=652684,1862282=717015} 2024-09-11T01:48:46.622 INFO:tasks.rados.rados.0.smithi071.stdout:2863: writing smithi07134777-19 from 659105 to 1311789 tid 1 2024-09-11T01:48:46.630 INFO:tasks.rados.rados.0.smithi071.stdout:2863: writing smithi07134777-19 from 1862282 to 2579297 tid 2 2024-09-11T01:48:46.630 INFO:tasks.rados.rados.0.smithi071.stdout:2864: rollback oid 47 current snap is 322 2024-09-11T01:48:46.630 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 47 to 313 2024-09-11T01:48:46.630 INFO:tasks.rados.rados.0.smithi071.stdout:2861: finishing write tid 2 to smithi07134777-34 2024-09-11T01:48:46.630 INFO:tasks.rados.rados.0.smithi071.stdout:2861: finishing write tid 3 to smithi07134777-34 2024-09-11T01:48:46.630 INFO:tasks.rados.rados.0.smithi071.stdout:2861: finishing write tid 4 to smithi07134777-34 2024-09-11T01:48:46.631 INFO:tasks.rados.rados.0.smithi071.stdout:2861: oid 34 updating version 0 to 1918 2024-09-11T01:48:46.631 INFO:tasks.rados.rados.0.smithi071.stdout:2861: oid 34 updating version 1918 to 1919 2024-09-11T01:48:46.631 INFO:tasks.rados.rados.0.smithi071.stdout:2861: oid 34 updating version 1919 to 1920 2024-09-11T01:48:46.631 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 1920 (ObjNum 1013 snap 322 seq_num 1013) dirty exists 2024-09-11T01:48:46.631 INFO:tasks.rados.rados.0.smithi071.stdout:2861: left oid 34 (ObjNum 1013 snap 322 seq_num 1013) 2024-09-11T01:48:46.631 INFO:tasks.rados.rados.0.smithi071.stdout:2862: finishing write tid 1 to smithi07134777-7 2024-09-11T01:48:46.631 INFO:tasks.rados.rados.0.smithi071.stdout:2862: finishing write tid 2 to smithi07134777-7 2024-09-11T01:48:46.631 INFO:tasks.rados.rados.0.smithi071.stdout:2862: finishing write tid 3 to smithi07134777-7 2024-09-11T01:48:46.631 INFO:tasks.rados.rados.0.smithi071.stdout:2862: finishing write tid 4 to smithi07134777-7 2024-09-11T01:48:46.631 INFO:tasks.rados.rados.0.smithi071.stdout:2862: finishing write tid 5 to smithi07134777-7 2024-09-11T01:48:46.631 INFO:tasks.rados.rados.0.smithi071.stdout:2862: oid 7 updating version 0 to 2023 2024-09-11T01:48:46.631 INFO:tasks.rados.rados.0.smithi071.stdout:2862: oid 7 version 2023 is already newer than 2022 2024-09-11T01:48:46.631 INFO:tasks.rados.rados.0.smithi071.stdout:2862: oid 7 updating version 2023 to 2024 2024-09-11T01:48:46.631 INFO:tasks.rados.rados.0.smithi071.stdout:2862: oid 7 updating version 2024 to 2025 2024-09-11T01:48:46.632 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 2025 (ObjNum 1014 snap 322 seq_num 1014) dirty exists 2024-09-11T01:48:46.632 INFO:tasks.rados.rados.0.smithi071.stdout:2862: left oid 7 (ObjNum 1014 snap 322 seq_num 1014) 2024-09-11T01:48:46.632 INFO:tasks.rados.rados.0.smithi071.stdout:2863: finishing write tid 1 to smithi07134777-19 2024-09-11T01:48:46.632 INFO:tasks.rados.rados.0.smithi071.stdout:2861: done (3 left) 2024-09-11T01:48:46.632 INFO:tasks.rados.rados.0.smithi071.stdout:2862: done (2 left) 2024-09-11T01:48:46.632 INFO:tasks.rados.rados.0.smithi071.stdout:2865: read oid 35 snap -1 2024-09-11T01:48:46.632 INFO:tasks.rados.rados.0.smithi071.stdout:2865: expect (ObjNum 949 snap 300 seq_num 949) 2024-09-11T01:48:46.632 INFO:tasks.rados.rados.0.smithi071.stdout:2866: delete oid 14 current snap is 322 2024-09-11T01:48:46.634 INFO:tasks.rados.rados.0.smithi071.stdout:2863: finishing write tid 2 to smithi07134777-19 2024-09-11T01:48:46.634 INFO:tasks.rados.rados.0.smithi071.stdout:2863: finishing write tid 3 to smithi07134777-19 2024-09-11T01:48:46.634 INFO:tasks.rados.rados.0.smithi071.stdout:2863: finishing write tid 4 to smithi07134777-19 2024-09-11T01:48:46.634 INFO:tasks.rados.rados.0.smithi071.stdout:2863: oid 19 updating version 0 to 1083 2024-09-11T01:48:46.634 INFO:tasks.rados.rados.0.smithi071.stdout:2863: oid 19 updating version 1083 to 1084 2024-09-11T01:48:46.634 INFO:tasks.rados.rados.0.smithi071.stdout:2863: oid 19 version 1084 is already newer than 1082 2024-09-11T01:48:46.634 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 1084 (ObjNum 1015 snap 322 seq_num 1015) dirty exists 2024-09-11T01:48:46.635 INFO:tasks.rados.rados.0.smithi071.stdout:2863: left oid 19 (ObjNum 1015 snap 322 seq_num 1015) 2024-09-11T01:48:46.635 INFO:tasks.rados.rados.0.smithi071.stdout:2863: done (3 left) 2024-09-11T01:48:46.635 INFO:tasks.rados.rados.0.smithi071.stdout:2866: done (2 left) 2024-09-11T01:48:46.635 INFO:tasks.rados.rados.0.smithi071.stdout:2867: copy_from oid 11 from oid 29 current snap is 322 2024-09-11T01:48:46.635 INFO:tasks.rados.rados.0.smithi071.stdout:2868: write (excl) oid 49 current snap is 322 2024-09-11T01:48:46.635 INFO:tasks.rados.rados.0.smithi071.stdout:2868: seq_num 1017 ranges {692719=403182,1579246=720684,2828713=264006} 2024-09-11T01:48:46.640 INFO:tasks.rados.rados.0.smithi071.stdout:2868: writing smithi07134777-49 from 692719 to 1095901 tid 1 2024-09-11T01:48:46.646 INFO:tasks.rados.rados.0.smithi071.stdout:2868: writing smithi07134777-49 from 1579246 to 2299930 tid 2 2024-09-11T01:48:46.651 INFO:tasks.rados.rados.0.smithi071.stdout:2868: writing smithi07134777-49 from 2828713 to 3092719 tid 3 2024-09-11T01:48:46.651 INFO:tasks.rados.rados.0.smithi071.stdout:2869: copy_from oid 12 from oid 40 current snap is 322 2024-09-11T01:48:46.651 INFO:tasks.rados.rados.0.smithi071.stdout:2864: finishing rollback tid 0 to smithi07134777-47 2024-09-11T01:48:46.652 INFO:tasks.rados.rados.0.smithi071.stdout:2864: finishing rollback tid 1 to smithi07134777-47 2024-09-11T01:48:46.652 INFO:tasks.rados.rados.0.smithi071.stdout:2867: finishing copy_from to smithi07134777-11 2024-09-11T01:48:46.652 INFO:tasks.rados.rados.0.smithi071.stdout:2867: got expected ENOENT (src dne) 2024-09-11T01:48:46.652 INFO:tasks.rados.rados.0.smithi071.stdout:2867: finishing copy_from racing read to smithi07134777-11 2024-09-11T01:48:46.652 INFO:tasks.rados.rados.0.smithi071.stdout:2864: finishing rollback tid 2 to smithi07134777-47 2024-09-11T01:48:46.652 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 1777 (ObjNum 715 snap 221 seq_num 715) dirty exists 2024-09-11T01:48:46.652 INFO:tasks.rados.rados.0.smithi071.stdout:2865: expect (ObjNum 949 snap 300 seq_num 949) 2024-09-11T01:48:46.652 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:48:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.804+0000 7f88226b6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:48:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.804+0000 7f88226b6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:48:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.804+0000 7f88226b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.804+0000 7f88226b6640 1 -- 172.21.15.12:0/3415743366 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f881c06d460 con 0x7f88040039b0 2024-09-11T01:48:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.804+0000 7f8818ff9640 1 -- 172.21.15.12:0/3415743366 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f880c0521a0 con 0x7f88040039b0 2024-09-11T01:48:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.804+0000 7f8818ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:48:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.804+0000 7f8818ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:48:46.807 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":186,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:48:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.807+0000 7f88226b6640 1 -- 172.21.15.12:0/3415743366 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f880803eb70 msgr2=0x7f8808041030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.807+0000 7f88226b6640 1 --2- 172.21.15.12:0/3415743366 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f880803eb70 0x7f8808041030 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8810007fd0 tx=0x7f8810000e40 comp rx=0 tx=0).stop 2024-09-11T01:48:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.807+0000 7f88226b6640 10 monclient: shutdown 2024-09-11T01:48:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.807+0000 7f88226b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.807+0000 7f88226b6640 1 -- 172.21.15.12:0/3415743366 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88040039b0 msgr2=0x7f881c1ccaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.807+0000 7f88226b6640 1 --2- 172.21.15.12:0/3415743366 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88040039b0 0x7f881c1ccaf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f880c003e70 tx=0x7f880c00ae70 comp rx=0 tx=0).stop 2024-09-11T01:48:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.807+0000 7f88226b6640 1 -- 172.21.15.12:0/3415743366 shutdown_connections 2024-09-11T01:48:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.807+0000 7f88226b6640 1 --2- 172.21.15.12:0/3415743366 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8804005230 0x7f881c124640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.807+0000 7f88226b6640 1 --2- 172.21.15.12:0/3415743366 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8804004800 0x7f881c1cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.807+0000 7f88226b6640 1 --2- 172.21.15.12:0/3415743366 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f880803eb70 0x7f8808041030 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.807+0000 7f88226b6640 1 --2- 172.21.15.12:0/3415743366 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f88040039b0 0x7f881c1ccaf0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.807+0000 7f88226b6640 1 -- 172.21.15.12:0/3415743366 >> 172.21.15.12:0/3415743366 conn(0x7f881c0a2d90 msgr2=0x7f881c0a4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.807+0000 7f88226b6640 1 -- 172.21.15.12:0/3415743366 shutdown_connections 2024-09-11T01:48:46.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.807+0000 7f88226b6640 1 -- 172.21.15.12:0/3415743366 wait complete. 2024-09-11T01:48:46.819 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:48:46.819 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-11T01:48:46.819 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:48:46.851 INFO:tasks.rados.rados.0.smithi071.stdout:2868: finishing write tid 1 to smithi07134777-49 2024-09-11T01:48:46.851 INFO:tasks.rados.rados.0.smithi071.stdout:2864: done (4 left) 2024-09-11T01:48:46.851 INFO:tasks.rados.rados.0.smithi071.stdout:2865: done (3 left) 2024-09-11T01:48:46.851 INFO:tasks.rados.rados.0.smithi071.stdout:2867: done (2 left) 2024-09-11T01:48:46.851 INFO:tasks.rados.rados.0.smithi071.stdout:2870: read oid 46 snap -1 2024-09-11T01:48:46.851 INFO:tasks.rados.rados.0.smithi071.stdout:2870: expect (ObjNum 841 snap 255 seq_num 841) 2024-09-11T01:48:46.851 INFO:tasks.rados.rados.0.smithi071.stdout:2871: snap_create 2024-09-11T01:48:46.852 INFO:tasks.rados.rados.0.smithi071.stdout:2869: finishing copy_from to smithi07134777-12 2024-09-11T01:48:46.852 INFO:tasks.rados.rados.0.smithi071.stdout:2869: got expected ENOENT (src dne) 2024-09-11T01:48:46.852 INFO:tasks.rados.rados.0.smithi071.stdout:2869: finishing copy_from racing read to smithi07134777-12 2024-09-11T01:48:46.852 INFO:tasks.rados.rados.0.smithi071.stdout:2868: finishing write tid 2 to smithi07134777-49 2024-09-11T01:48:46.852 INFO:tasks.rados.rados.0.smithi071.stdout:2868: finishing write tid 3 to smithi07134777-49 2024-09-11T01:48:46.852 INFO:tasks.rados.rados.0.smithi071.stdout:2868: finishing write tid 4 to smithi07134777-49 2024-09-11T01:48:46.852 INFO:tasks.rados.rados.0.smithi071.stdout:2868: finishing write tid 5 to smithi07134777-49 2024-09-11T01:48:46.852 INFO:tasks.rados.rados.0.smithi071.stdout:2868: oid 49 updating version 0 to 387 2024-09-11T01:48:46.852 INFO:tasks.rados.rados.0.smithi071.stdout:2868: oid 49 version 387 is already newer than 385 2024-09-11T01:48:46.852 INFO:tasks.rados.rados.0.smithi071.stdout:2868: oid 49 version 387 is already newer than 386 2024-09-11T01:48:46.852 INFO:tasks.rados.rados.0.smithi071.stdout:2868: oid 49 updating version 387 to 388 2024-09-11T01:48:46.852 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 388 (ObjNum 1017 snap 322 seq_num 1017) dirty exists 2024-09-11T01:48:46.852 INFO:tasks.rados.rados.0.smithi071.stdout:2868: left oid 49 (ObjNum 1017 snap 322 seq_num 1017) 2024-09-11T01:48:46.853 INFO:tasks.rados.rados.0.smithi071.stdout:2870: expect (ObjNum 841 snap 255 seq_num 841) 2024-09-11T01:48:46.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.920+0000 7f3b5e12c640 10 monclient: get_monmap_and_config 2024-09-11T01:48:46.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.920+0000 7f3b5e12c640 10 monclient: build_initial_monmap 2024-09-11T01:48:46.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.920+0000 7f3b5e12c640 10 monclient: monmap: 2024-09-11T01:48:46.922 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:46.922 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:46.922 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:48:46.921205+0000 2024-09-11T01:48:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:48:46.921205+0000 2024-09-11T01:48:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:48:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:48:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:48:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:48:46.923 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:48:46.923 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:46.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.921+0000 7f3b5e12c640 1 Processor -- start 2024-09-11T01:48:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.921+0000 7f3b5e12c640 1 -- start start 2024-09-11T01:48:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.921+0000 7f3b5e12c640 10 monclient: init 2024-09-11T01:48:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.921+0000 7f3b5e12c640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.921+0000 7f3b5e12c640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-11T01:48:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.921+0000 7f3b5e12c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3b580a2db0 0x7f3b5806f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.921+0000 7f3b5e12c640 10 monclient(hunting): picked mon.noname-c con 0x7f3b580a2db0 addr v2:172.21.15.71:3300/0 2024-09-11T01:48:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.921+0000 7f3b5e12c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3b5806ded0 addr v1:172.21.15.12:6789/0 2024-09-11T01:48:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.921+0000 7f3b5e12c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3b580a38a0 0x7f3b5813f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.921+0000 7f3b5e12c640 10 monclient(hunting): picked mon.noname-b con 0x7f3b580a38a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.921+0000 7f3b5e12c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b580a4650 con 0x7f3b5806ded0 2024-09-11T01:48:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.921+0000 7f3b5e12c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:46.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.921+0000 7f3b5e12c640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f3b5806f1a0 con 0x7f3b580a2db0 2024-09-11T01:48:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.921+0000 7f3b5e12c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.921+0000 7f3b5e12c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f3b5806eec0 con 0x7f3b580a38a0 2024-09-11T01:48:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.921+0000 7f3b5e12c640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.921+0000 7f3b5e12c640 10 monclient(hunting): authenticate will time out at 1933.363752s 2024-09-11T01:48:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.921+0000 7f3b57fff640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3b580a38a0 msgr2=0x7f3b5813f3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:48:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.921+0000 7f3b57fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3b580a38a0 0x7f3b5813f3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:48:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.921+0000 7f3b56ffd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3b5806ded0 0x7f3b580a3190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37256/0 (socket says 172.21.15.12:37256) 2024-09-11T01:48:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.921+0000 7f3b56ffd640 1 -- 172.21.15.12:0/2697886209 learned_addr learned my addr 172.21.15.12:0/2697886209 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:48:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.921+0000 7f3b577fe640 1 --2- 172.21.15.12:0/2697886209 >> v2:172.21.15.71:3300/0 conn(0x7f3b580a2db0 0x7f3b5806f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.922+0000 7f3b577fe640 10 monclient(hunting): get_auth_request con 0x7f3b580a2db0 auth_method 0 2024-09-11T01:48:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.922+0000 7f3b577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.922+0000 7f3b577fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:46.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.922+0000 7f3b577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:46.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.922+0000 7f3b577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:46.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.922+0000 7f3b577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:46.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.922+0000 7f3b577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:46.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.922+0000 7f3b577fe640 10 monclient(hunting): handle_auth_done global_id 134806 payload 274 2024-09-11T01:48:46.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.922+0000 7f3b577fe640 1 -- 172.21.15.12:0/2697886209 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3b580a38a0 msgr2=0x7f3b5813f3c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:48:46.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.922+0000 7f3b577fe640 1 --2- 172.21.15.12:0/2697886209 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3b580a38a0 0x7f3b5813f3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:46.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.922+0000 7f3b577fe640 1 -- 172.21.15.12:0/2697886209 >> v1:172.21.15.12:6789/0 conn(0x7f3b5806ded0 legacy=0x7f3b580a3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:48:46.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.922+0000 7f3b577fe640 10 monclient: _finish_hunting 0 2024-09-11T01:48:46.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.922+0000 7f3b577fe640 1 monclient: found mon.noname-c 2024-09-11T01:48:46.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.922+0000 7f3b577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:46.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.922+0000 7f3b577fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:48:46.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.922+0000 7f3b577fe640 1 -- 172.21.15.12:0/2697886209 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b5806e2e0 con 0x7f3b580a2db0 2024-09-11T01:48:46.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.925+0000 7f3b577fe640 1 -- 172.21.15.12:0/2697886209 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b580a2db0 msgr2=0x7f3b5806f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:46.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.925+0000 7f3b577fe640 1 --2- 172.21.15.12:0/2697886209 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b580a2db0 0x7f3b5806f8a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3b480097a0 tx=0x7f3b4802a080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89e5a4785f8f9a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.925+0000 7f3b567fc640 1 -- 172.21.15.12:0/2697886209 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b48037070 con 0x7f3b580a2db0 2024-09-11T01:48:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.925+0000 7f3b567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.925+0000 7f3b567fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:48:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.925+0000 7f3b567fc640 10 monclient: dump: 2024-09-11T01:48:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:46.935 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:46.936 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:46.936 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:46.936 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:46.936 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:46.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.925+0000 7f3b567fc640 10 monclient: _finish_auth 0 2024-09-11T01:48:46.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.925+0000 7f3b567fc640 10 monclient: _check_auth_tickets 2024-09-11T01:48:46.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.925+0000 7f3b567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:46.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.925+0000 7f3b567fc640 1 -- 172.21.15.12:0/2697886209 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b48032640 con 0x7f3b580a2db0 2024-09-11T01:48:46.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.925+0000 7f3b567fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:46.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.925+0000 7f3b567fc640 1 -- 172.21.15.12:0/2697886209 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b48032a10 con 0x7f3b580a2db0 2024-09-11T01:48:46.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.925+0000 7f3b567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:46.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.925+0000 7f3b567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:48:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.925+0000 7f3b567fc640 10 monclient: dump: 2024-09-11T01:48:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:46.937 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.925+0000 7f3b5e12c640 5 monclient: authenticate success, global_id 134806 2024-09-11T01:48:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.925+0000 7f3b5e12c640 10 monclient: get_monmap_and_config success 2024-09-11T01:48:46.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.925+0000 7f3b5e12c640 4 set_mon_vals no callback set 2024-09-11T01:48:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.926+0000 7f3b5e12c640 10 monclient: shutdown 2024-09-11T01:48:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.926+0000 7f3b5e12c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.926+0000 7f3b5e12c640 1 -- 172.21.15.12:0/2697886209 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b580a2db0 msgr2=0x7f3b5806f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.926+0000 7f3b5e12c640 1 --2- 172.21.15.12:0/2697886209 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b580a2db0 0x7f3b5806f8a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3b480097a0 tx=0x7f3b4802a080 comp rx=0 tx=0).stop 2024-09-11T01:48:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.926+0000 7f3b5e12c640 1 -- 172.21.15.12:0/2697886209 shutdown_connections 2024-09-11T01:48:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.926+0000 7f3b5e12c640 1 --2- 172.21.15.12:0/2697886209 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3b580a38a0 0x7f3b5813f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.926+0000 7f3b5e12c640 1 --2- 172.21.15.12:0/2697886209 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b580a2db0 0x7f3b5806f8a0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.926+0000 7f3b5e12c640 1 -- 172.21.15.12:0/2697886209 >> 172.21.15.12:0/2697886209 conn(0x7f3b580a83b0 msgr2=0x7f3b580a87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:46.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.926+0000 7f3b5e12c640 1 -- 172.21.15.12:0/2697886209 shutdown_connections 2024-09-11T01:48:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.926+0000 7f3b5e12c640 1 -- 172.21.15.12:0/2697886209 wait complete. 2024-09-11T01:48:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.926+0000 7f3b5e12c640 10 monclient: build_initial_monmap 2024-09-11T01:48:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.926+0000 7f3b5e12c640 10 monclient: monmap: 2024-09-11T01:48:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:48:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:48:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:48:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:48:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:48:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:48:46.939 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:46.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.927+0000 7f3b5e12c640 1 Processor -- start 2024-09-11T01:48:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.927+0000 7f3b5e12c640 1 -- start start 2024-09-11T01:48:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.927+0000 7f3b5e12c640 10 monclient: init 2024-09-11T01:48:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.927+0000 7f3b5e12c640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.927+0000 7f3b5e12c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:48:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.927+0000 7f3b5e12c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3b5806ded0 0x7f3b581cd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.927+0000 7f3b5e12c640 10 monclient(hunting): picked mon.noname-c con 0x7f3b5806ded0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.927+0000 7f3b5e12c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b580a2db0 0x7f3b58138340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.927+0000 7f3b5e12c640 10 monclient(hunting): picked mon.noname-b con 0x7f3b580a2db0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:48:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.927+0000 7f3b5e12c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b580a38a0 0x7f3b5813b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.927+0000 7f3b5e12c640 10 monclient(hunting): picked mon.noname-a con 0x7f3b580a38a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:48:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.927+0000 7f3b5e12c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.927+0000 7f3b5e12c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5813fc90 con 0x7f3b580a38a0 2024-09-11T01:48:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.927+0000 7f3b5e12c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:46.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.927+0000 7f3b5e12c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5813fb10 con 0x7f3b580a2db0 2024-09-11T01:48:46.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.927+0000 7f3b5e12c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:46.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.927+0000 7f3b5e12c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f3b5813fe10 con 0x7f3b5806ded0 2024-09-11T01:48:46.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.927+0000 7f3b5e12c640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:46.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.927+0000 7f3b5e12c640 10 monclient(hunting): authenticate will time out at 1933.369943s 2024-09-11T01:48:46.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.927+0000 7f3b577fe640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3b5806ded0 msgr2=0x7f3b581cd0e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:48:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.927+0000 7f3b577fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3b5806ded0 0x7f3b581cd0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:48:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b57fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b580a38a0 0x7f3b5813b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b57fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b580a38a0 0x7f3b5813b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42276/0 (socket says 172.21.15.12:42276) 2024-09-11T01:48:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b57fff640 1 -- 172.21.15.12:0/4000551411 learned_addr learned my addr 172.21.15.12:0/4000551411 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:48:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b57fff640 10 monclient(hunting): get_auth_request con 0x7f3b580a38a0 auth_method 0 2024-09-11T01:48:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b57fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b57fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b56ffd640 1 --2- 172.21.15.12:0/4000551411 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b580a2db0 0x7f3b58138340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b56ffd640 10 monclient(hunting): get_auth_request con 0x7f3b580a2db0 auth_method 0 2024-09-11T01:48:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:46.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b56ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b57fff640 10 monclient(hunting): handle_auth_done global_id 214164 payload 824 2024-09-11T01:48:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b57fff640 1 -- 172.21.15.12:0/4000551411 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3b5806ded0 msgr2=0x7f3b581cd0e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:48:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b57fff640 1 --2- 172.21.15.12:0/4000551411 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3b5806ded0 0x7f3b581cd0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b57fff640 1 -- 172.21.15.12:0/4000551411 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b580a2db0 msgr2=0x7f3b58138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b57fff640 1 --2- 172.21.15.12:0/4000551411 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b580a2db0 0x7f3b58138340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b57fff640 10 monclient: _finish_hunting 0 2024-09-11T01:48:46.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b57fff640 1 monclient: found mon.noname-a 2024-09-11T01:48:46.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:46.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b57fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:48:46.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.928+0000 7f3b57fff640 1 -- 172.21.15.12:0/4000551411 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b5813d7c0 con 0x7f3b580a38a0 2024-09-11T01:48:46.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.930+0000 7f3b57fff640 1 -- 172.21.15.12:0/4000551411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b580a38a0 msgr2=0x7f3b5813b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:46.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.930+0000 7f3b57fff640 1 --2- 172.21.15.12:0/4000551411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b580a38a0 0x7f3b5813b5b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3b5000be90 tx=0x7f3b5000bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:46.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.930+0000 7f3b3ffff640 1 -- 172.21.15.12:0/4000551411 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b50016630 con 0x7f3b580a38a0 2024-09-11T01:48:46.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.930+0000 7f3b3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:46.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.930+0000 7f3b3ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:48:46.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.930+0000 7f3b3ffff640 10 monclient: dump: 2024-09-11T01:48:46.946 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:46.946 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:46.946 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:46.946 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:46.946 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:46.947 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:46.947 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:46.947 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:46.947 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:46.947 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:46.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.930+0000 7f3b3ffff640 10 monclient: _finish_auth 0 2024-09-11T01:48:46.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.930+0000 7f3b3ffff640 10 monclient: _check_auth_tickets 2024-09-11T01:48:46.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.930+0000 7f3b3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:46.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.930+0000 7f3b3ffff640 1 -- 172.21.15.12:0/4000551411 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b50010030 con 0x7f3b580a38a0 2024-09-11T01:48:46.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.930+0000 7f3b5e12c640 5 monclient: authenticate success, global_id 214164 2024-09-11T01:48:46.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.930+0000 7f3b3ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:46.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.930+0000 7f3b3ffff640 1 -- 172.21.15.12:0/4000551411 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b50016a10 con 0x7f3b580a38a0 2024-09-11T01:48:46.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.930+0000 7f3b5e12c640 10 monclient: _renew_subs 2024-09-11T01:48:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.930+0000 7f3b5e12c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.930+0000 7f3b5e12c640 1 -- 172.21.15.12:0/4000551411 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b5813ee10 con 0x7f3b580a38a0 2024-09-11T01:48:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.930+0000 7f3b3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.930+0000 7f3b3ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:48:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.930+0000 7f3b3ffff640 10 monclient: dump: 2024-09-11T01:48:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:46.948 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:46.949 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:46.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.930+0000 7f3b557fa640 4 set_mon_vals no callback set 2024-09-11T01:48:46.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.930+0000 7f3b5e12c640 10 monclient: _renew_subs 2024-09-11T01:48:46.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.930+0000 7f3b5e12c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:46.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.930+0000 7f3b5e12c640 1 -- 172.21.15.12:0/4000551411 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b580a6730 con 0x7f3b580a38a0 2024-09-11T01:48:46.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.931+0000 7f3b5e12c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:46.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.931+0000 7f3b5e12c640 1 --2- 172.21.15.12:0/4000551411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b58138880 0x7f3b5802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:46.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.931+0000 7f3b5e12c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:46.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.931+0000 7f3b5e12c640 1 -- 172.21.15.12:0/4000551411 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5806a060 con 0x7f3b58138880 2024-09-11T01:48:46.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.931+0000 7f3b3ffff640 1 -- 172.21.15.12:0/4000551411 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f3b50008070 con 0x7f3b580a38a0 2024-09-11T01:48:46.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.931+0000 7f3b577fe640 1 --2- 172.21.15.12:0/4000551411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b58138880 0x7f3b5802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:46.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.931+0000 7f3b577fe640 10 monclient: get_auth_request con 0x7f3b58138880 auth_method 0 2024-09-11T01:48:46.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.931+0000 7f3b577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:46.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.931+0000 7f3b577fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:46.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.931+0000 7f3b577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:46.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.931+0000 7f3b577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:46.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.931+0000 7f3b577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:46.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.931+0000 7f3b577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:46.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.931+0000 7f3b3ffff640 1 --2- 172.21.15.12:0/4000551411 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3b4403e1c0 0x7f3b44040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:46.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.932+0000 7f3b577fe640 10 monclient(hunting): handle_auth_done global_id 214167 payload 824 2024-09-11T01:48:46.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.932+0000 7f3b577fe640 1 -- 172.21.15.12:0/4000551411 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3b580a6320 con 0x7f3b58138880 2024-09-11T01:48:46.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.932+0000 7f3b3ffff640 1 -- 172.21.15.12:0/4000551411 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(643..643 src has 507..643) ==== 4847+0+0 (secure 0 0 0) 0x7f3b500535b0 con 0x7f3b580a38a0 2024-09-11T01:48:46.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.932+0000 7f3b56ffd640 1 --2- 172.21.15.12:0/4000551411 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3b4403e1c0 0x7f3b44040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:46.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.932+0000 7f3b56ffd640 10 monclient: get_auth_request con 0x7f3b4403e1c0 auth_method 0 2024-09-11T01:48:46.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.934+0000 7f3b577fe640 1 -- 172.21.15.12:0/4000551411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b58138880 msgr2=0x7f3b5802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:46.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.934+0000 7f3b577fe640 1 --2- 172.21.15.12:0/4000551411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b58138880 0x7f3b5802d440 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3b58072030 tx=0x7f3b480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:46.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.934+0000 7f3b3ffff640 1 -- 172.21.15.12:0/4000551411 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b48042020 con 0x7f3b58138880 2024-09-11T01:48:46.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.934+0000 7f3b3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:46.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.934+0000 7f3b3ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:48:46.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.934+0000 7f3b3ffff640 10 monclient: dump: 2024-09-11T01:48:46.953 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:46.953 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:46.953 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:46.953 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:46.954 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:46.954 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:46.954 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:46.954 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:46.954 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:46.954 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:46.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.934+0000 7f3b56ffd640 1 --2- 172.21.15.12:0/4000551411 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3b4403e1c0 0x7f3b44040680 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c007c90 tx=0x7f3b4c007be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:46.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.934+0000 7f3b3ffff640 1 -- 172.21.15.12:0/4000551411 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3b48038040 con 0x7f3b58138880 2024-09-11T01:48:46.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.934+0000 7f3b3ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:46.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.934+0000 7f3b3ffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:48:46.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.934+0000 7f3b3ffff640 1 -- 172.21.15.12:0/4000551411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b58138880 msgr2=0x7f3b5802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:46.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.934+0000 7f3b3ffff640 1 --2- 172.21.15.12:0/4000551411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b58138880 0x7f3b5802d440 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3b58072030 tx=0x7f3b480046c0 comp rx=0 tx=0).stop 2024-09-11T01:48:46.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.934+0000 7f3b3ffff640 1 -- 172.21.15.12:0/4000551411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b58138880 msgr2=0x7f3b5802d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:48:46.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.934+0000 7f3b3ffff640 1 --2- 172.21.15.12:0/4000551411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b58138880 0x7f3b5802d440 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3b58072030 tx=0x7f3b480046c0 comp rx=0 tx=0).stop 2024-09-11T01:48:46.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.940+0000 7f3b5e12c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:48:46.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.940+0000 7f3b5e12c640 1 --2- 172.21.15.12:0/4000551411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b5802d980 0x7f3b5802dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:46.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.940+0000 7f3b5e12c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:46.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.940+0000 7f3b5e12c640 1 -- 172.21.15.12:0/4000551411 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5806f8a0 con 0x7f3b5802d980 2024-09-11T01:48:46.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.941+0000 7f3b577fe640 1 --2- 172.21.15.12:0/4000551411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b5802d980 0x7f3b5802dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:46.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.941+0000 7f3b577fe640 10 monclient: get_auth_request con 0x7f3b5802d980 auth_method 0 2024-09-11T01:48:46.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.941+0000 7f3b577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:46.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.941+0000 7f3b577fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:46.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.941+0000 7f3b577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:46.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.941+0000 7f3b577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:46.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.941+0000 7f3b577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:46.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.941+0000 7f3b577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:46.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.941+0000 7f3b577fe640 10 monclient(hunting): handle_auth_done global_id 214170 payload 824 2024-09-11T01:48:46.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.941+0000 7f3b577fe640 1 -- 172.21.15.12:0/4000551411 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3b58069990 con 0x7f3b5802d980 2024-09-11T01:48:46.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.943+0000 7f3b577fe640 1 -- 172.21.15.12:0/4000551411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b5802d980 msgr2=0x7f3b5802dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:46.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.943+0000 7f3b577fe640 1 --2- 172.21.15.12:0/4000551411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b5802d980 0x7f3b5802dd90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3b48000c00 tx=0x7f3b48009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:46.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.943+0000 7f3b3ffff640 1 -- 172.21.15.12:0/4000551411 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b48042050 con 0x7f3b5802d980 2024-09-11T01:48:46.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.943+0000 7f3b3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:46.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.943+0000 7f3b3ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:48:46.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.943+0000 7f3b3ffff640 10 monclient: dump: 2024-09-11T01:48:46.957 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:46.957 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:46.957 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:46.957 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:46.957 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:46.957 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:46.957 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:46.958 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:46.958 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:46.958 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:46.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.943+0000 7f3b3ffff640 1 -- 172.21.15.12:0/4000551411 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4532 (secure 0 0 0) 0x7f3b48004450 con 0x7f3b5802d980 2024-09-11T01:48:46.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.943+0000 7f3b3ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:48:46.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.943+0000 7f3b3ffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:48:46.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.943+0000 7f3b3ffff640 1 -- 172.21.15.12:0/4000551411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b5802d980 msgr2=0x7f3b5802dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:46.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.943+0000 7f3b3ffff640 1 --2- 172.21.15.12:0/4000551411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b5802d980 0x7f3b5802dd90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3b48000c00 tx=0x7f3b48009190 comp rx=0 tx=0).stop 2024-09-11T01:48:46.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.943+0000 7f3b3ffff640 1 -- 172.21.15.12:0/4000551411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b5802d980 msgr2=0x7f3b5802dd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:48:46.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.943+0000 7f3b3ffff640 1 --2- 172.21.15.12:0/4000551411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b5802d980 0x7f3b5802dd90 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:46.958 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:48:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:48:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:48:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:48:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 186, 2024-09-11T01:48:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 33143, 2024-09-11T01:48:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:48:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:48:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: 1 2024-09-11T01:48:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:48:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:48:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:48:46.959 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:48:46.960 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:48:46.960 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:48:46.960 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:48:46.960 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:48:46.960 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:48:46.960 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:48:46.960 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:48:46.960 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:48:46.960 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:48:46.960 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:48:46.960 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:48:46.960 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:46.960 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:48:46.961 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:48:46.961 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:48:46.961 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:48:46.961 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:48:46.961 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:48:46.961 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:48:46.961 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:48:46.961 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:48:46.961 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:48:46.961 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:48:46.961 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:48:46.961 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:48:46.961 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:46.962 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:46.962 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:48:46.962 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:48:46.962 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:48:46.962 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:48:46.962 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:48:46.962 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:48:46.962 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:48:46.962 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:48:46.962 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:48:46.962 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:48:46.962 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:48:46.963 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:48:46.963 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:48:46.963 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:48:46.963 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:48:46.963 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:48:46.963 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:48:46.963 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:48:46.963 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:48:46.963 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:48:46.963 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:48:46.963 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:48:46.963 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:48:46.963 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:48:46.964 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:48:46.964 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:48:46.964 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:48:46.964 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:48:46.964 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:46.964 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:48:46.964 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:46.964 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:48:46.964 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:46.964 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:48:46.964 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:48:46.965 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:48:46.965 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:48:46.965 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:46.965 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:48:46.965 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:48:46.965 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:46.965 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:46.965 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:46.965 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:48:46.965 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:48:46.965 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:46.965 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:46.965 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:46.966 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:46.966 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:48:46.966 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:48:46.966 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:48:46.966 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:48:46.966 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:48:46.966 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:46.966 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:46.966 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:48:46.966 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:48:46.966 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:48:46.966 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:48:46.966 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:46.967 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:48:46.967 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:48:46.967 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:46.967 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:46.967 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:46.967 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:48:46.967 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:48:46.967 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:46.967 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:46.967 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:46.967 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:46.967 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:48:46.967 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:48:46.967 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:48:46.968 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:48:46.968 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:48:46.968 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:46.968 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:46.968 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:48:46.968 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:48:46.968 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:48:46.968 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:48:46.968 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:46.968 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:48:46.968 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:48:46.968 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:46.968 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:46.968 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:46.969 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:48:46.969 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:48:46.969 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:46.969 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:46.969 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:46.969 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:46.969 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:48:46.969 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:48:46.969 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:48:46.969 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:48:46.969 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:48:46.969 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:46.969 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:46.969 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:46.970 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:48:46.970 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:48:46.970 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:46.970 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:48:46.970 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:48:46.970 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:48:46.970 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:46.970 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:46.970 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:48:46.970 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:46.970 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:48:46.970 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:48:46.970 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:48:46.970 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:46.971 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:46.971 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:46.971 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:48:46.971 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:48:46.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.944+0000 7f3b5e12c640 1 -- 172.21.15.12:0/4000551411 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3b4403e1c0 msgr2=0x7f3b44040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:46.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.944+0000 7f3b5e12c640 1 --2- 172.21.15.12:0/4000551411 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3b4403e1c0 0x7f3b44040680 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c007c90 tx=0x7f3b4c007be0 comp rx=0 tx=0).stop 2024-09-11T01:48:46.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.944+0000 7f3b5e12c640 10 monclient: shutdown 2024-09-11T01:48:46.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.944+0000 7f3b5e12c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:46.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.944+0000 7f3b5e12c640 1 -- 172.21.15.12:0/4000551411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b580a38a0 msgr2=0x7f3b5813b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:46.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.944+0000 7f3b57fff640 1 -- 172.21.15.12:0/4000551411 reap_dead start 2024-09-11T01:48:46.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.944+0000 7f3b5e12c640 1 --2- 172.21.15.12:0/4000551411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b580a38a0 0x7f3b5813b5b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3b5000be90 tx=0x7f3b5000bf80 comp rx=0 tx=0).stop 2024-09-11T01:48:46.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.944+0000 7f3b5e12c640 1 -- 172.21.15.12:0/4000551411 shutdown_connections 2024-09-11T01:48:46.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.944+0000 7f3b5e12c640 1 --2- 172.21.15.12:0/4000551411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3b580a38a0 0x7f3b5813b5b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:46.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.944+0000 7f3b5e12c640 1 -- 172.21.15.12:0/4000551411 >> 172.21.15.12:0/4000551411 conn(0x7f3b580a83b0 msgr2=0x7f3b5809e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:46.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.944+0000 7f3b5e12c640 1 -- 172.21.15.12:0/4000551411 shutdown_connections 2024-09-11T01:48:46.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:46.944+0000 7f3b5e12c640 1 -- 172.21.15.12:0/4000551411 wait complete. 2024-09-11T01:48:46.974 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:48:46.974 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T01:48:46.974 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:48:47.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.074+0000 7f7eebe9b640 10 monclient: get_monmap_and_config 2024-09-11T01:48:47.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.074+0000 7f7eebe9b640 10 monclient: build_initial_monmap 2024-09-11T01:48:47.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.075+0000 7f7eebe9b640 10 monclient: monmap: 2024-09-11T01:48:47.076 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:47.077 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:47.077 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:48:47.075714+0000 2024-09-11T01:48:47.077 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:48:47.075714+0000 2024-09-11T01:48:47.077 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:47.077 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:47.077 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:48:47.077 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:48:47.077 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:48:47.077 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:48:47.077 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:48:47.077 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:47.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.075+0000 7f7eebe9b640 1 Processor -- start 2024-09-11T01:48:47.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.075+0000 7f7eebe9b640 1 -- start start 2024-09-11T01:48:47.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.075+0000 7f7eebe9b640 10 monclient: init 2024-09-11T01:48:47.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7eebe9b640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:47.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7eebe9b640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-11T01:48:47.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7eebe9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7ee4135f70 0x7f7ee4136350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:47.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7eebe9b640 10 monclient(hunting): picked mon.noname-a con 0x7f7ee4135f70 addr v2:172.21.15.12:3300/0 2024-09-11T01:48:47.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7eebe9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7ee409e020 0x7f7ee40a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:47.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7eebe9b640 10 monclient(hunting): picked mon.noname-b con 0x7f7ee409e020 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:47.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7eebe9b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7ee40a3280 addr v1:172.21.15.71:6789/0 2024-09-11T01:48:47.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7eebe9b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ee409da70 con 0x7f7ee40a3280 2024-09-11T01:48:47.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7eebe9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:47.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7eebe9b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f7ee409dde0 con 0x7f7ee4135f70 2024-09-11T01:48:47.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7eebe9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:47.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7eebe9b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7ee406cc50 con 0x7f7ee409e020 2024-09-11T01:48:47.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7eebe9b640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:47.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7eebe9b640 10 monclient(hunting): authenticate will time out at 1933.518371s 2024-09-11T01:48:47.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7ee9436640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7ee409e020 msgr2=0x7f7ee40a2d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:48:47.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7ee9436640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7ee409e020 0x7f7ee40a2d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:48:47.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7ee9c37640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7ee4135f70 0x7f7ee4136350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:47.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7ee9c37640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7ee4135f70 0x7f7ee4136350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42306/0 (socket says 172.21.15.12:42306) 2024-09-11T01:48:47.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7ee9c37640 1 -- 172.21.15.12:0/1257792352 learned_addr learned my addr 172.21.15.12:0/1257792352 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:48:47.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7ee9c37640 10 monclient(hunting): get_auth_request con 0x7f7ee4135f70 auth_method 0 2024-09-11T01:48:47.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7ee9c37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:47.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7ee9c37640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:47.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7ee9c37640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:47.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7ee9c37640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:47.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7ee9c37640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:47.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7ee9c37640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:47.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7ee9c37640 10 monclient(hunting): handle_auth_done global_id 214173 payload 274 2024-09-11T01:48:47.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7ee9c37640 1 -- 172.21.15.12:0/1257792352 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7ee409e020 msgr2=0x7f7ee40a2d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:48:47.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7ee9c37640 1 --2- 172.21.15.12:0/1257792352 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7ee409e020 0x7f7ee40a2d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:47.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7ee9c37640 1 -- 172.21.15.12:0/1257792352 >> v1:172.21.15.71:6789/0 conn(0x7f7ee40a3280 legacy=0x7f7ee413cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:48:47.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7ee9c37640 10 monclient: _finish_hunting 0 2024-09-11T01:48:47.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7ee9c37640 1 monclient: found mon.noname-a 2024-09-11T01:48:47.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7ee9c37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:47.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7ee9c37640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:48:47.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.076+0000 7f7ee9c37640 1 -- 172.21.15.12:0/1257792352 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ee406d3b0 con 0x7f7ee4135f70 2024-09-11T01:48:47.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.079+0000 7f7ee9c37640 1 -- 172.21.15.12:0/1257792352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ee4135f70 msgr2=0x7f7ee4136350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:47.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.079+0000 7f7ee9c37640 1 --2- 172.21.15.12:0/1257792352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ee4135f70 0x7f7ee4136350 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0009870 tx=0x7f7ee002a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f61e19d98bf193f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:47.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.079+0000 7f7ee8c35640 1 -- 172.21.15.12:0/1257792352 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0037070 con 0x7f7ee4135f70 2024-09-11T01:48:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.079+0000 7f7ee8c35640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.079+0000 7f7ee8c35640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:48:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.079+0000 7f7ee8c35640 10 monclient: dump: 2024-09-11T01:48:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:47.090 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.079+0000 7f7ee8c35640 10 monclient: _finish_auth 0 2024-09-11T01:48:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.079+0000 7f7ee8c35640 10 monclient: _check_auth_tickets 2024-09-11T01:48:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.079+0000 7f7ee8c35640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.079+0000 7f7ee8c35640 1 -- 172.21.15.12:0/1257792352 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ee0032640 con 0x7f7ee4135f70 2024-09-11T01:48:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.079+0000 7f7eebe9b640 5 monclient: authenticate success, global_id 214173 2024-09-11T01:48:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.079+0000 7f7eebe9b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:48:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.079+0000 7f7ee8c35640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.079+0000 7f7eebe9b640 10 monclient: get_monmap_and_config success 2024-09-11T01:48:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.079+0000 7f7ee8c35640 1 -- 172.21.15.12:0/1257792352 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0032a10 con 0x7f7ee4135f70 2024-09-11T01:48:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.079+0000 7f7ee8c35640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.079+0000 7f7eebe9b640 4 set_mon_vals no callback set 2024-09-11T01:48:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.079+0000 7f7ee8c35640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:48:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.079+0000 7f7ee8c35640 10 monclient: dump: 2024-09-11T01:48:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:47.091 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:47.092 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.080+0000 7f7eebe9b640 10 monclient: shutdown 2024-09-11T01:48:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.080+0000 7f7eebe9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.080+0000 7f7eebe9b640 1 -- 172.21.15.12:0/1257792352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ee4135f70 msgr2=0x7f7ee4136350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.080+0000 7f7eebe9b640 1 --2- 172.21.15.12:0/1257792352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ee4135f70 0x7f7ee4136350 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0009870 tx=0x7f7ee002a0d0 comp rx=0 tx=0).stop 2024-09-11T01:48:47.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.080+0000 7f7eebe9b640 1 -- 172.21.15.12:0/1257792352 shutdown_connections 2024-09-11T01:48:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.080+0000 7f7eebe9b640 1 --2- 172.21.15.12:0/1257792352 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7ee409e020 0x7f7ee40a2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.080+0000 7f7eebe9b640 1 --2- 172.21.15.12:0/1257792352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ee4135f70 0x7f7ee4136350 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.080+0000 7f7eebe9b640 1 -- 172.21.15.12:0/1257792352 >> 172.21.15.12:0/1257792352 conn(0x7f7ee40a3e30 msgr2=0x7f7ee40a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.080+0000 7f7eebe9b640 1 -- 172.21.15.12:0/1257792352 shutdown_connections 2024-09-11T01:48:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.080+0000 7f7eebe9b640 1 -- 172.21.15.12:0/1257792352 wait complete. 2024-09-11T01:48:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.081+0000 7f7eebe9b640 10 monclient: build_initial_monmap 2024-09-11T01:48:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.081+0000 7f7eebe9b640 10 monclient: monmap: 2024-09-11T01:48:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:48:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:48:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:48:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:48:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:48:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:48:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:48:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:48:47.094 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.081+0000 7f7eebe9b640 1 Processor -- start 2024-09-11T01:48:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.081+0000 7f7eebe9b640 1 -- start start 2024-09-11T01:48:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.081+0000 7f7eebe9b640 10 monclient: init 2024-09-11T01:48:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.081+0000 7f7eebe9b640 10 monclient: _reopen_session rank -1 2024-09-11T01:48:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.081+0000 7f7eebe9b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:48:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.081+0000 7f7eebe9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7ee409e020 0x7f7ee412f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.081+0000 7f7eebe9b640 10 monclient(hunting): picked mon.noname-c con 0x7f7ee409e020 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:48:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.081+0000 7f7eebe9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee40a3280 0x7f7ee409b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.081+0000 7f7eebe9b640 10 monclient(hunting): picked mon.noname-b con 0x7f7ee40a3280 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:48:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.081+0000 7f7eebe9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ee4135f70 0x7f7ee409b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.081+0000 7f7eebe9b640 10 monclient(hunting): picked mon.noname-a con 0x7f7ee4135f70 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:48:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.081+0000 7f7eebe9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.081+0000 7f7eebe9b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee4136890 con 0x7f7ee4135f70 2024-09-11T01:48:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.081+0000 7f7eebe9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.081+0000 7f7eebe9b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee409f020 con 0x7f7ee40a3280 2024-09-11T01:48:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.081+0000 7f7eebe9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.081+0000 7f7eebe9b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7ee4136a10 con 0x7f7ee409e020 2024-09-11T01:48:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.081+0000 7f7eebe9b640 10 monclient(hunting): _renew_subs 2024-09-11T01:48:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.081+0000 7f7ee9c37640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7ee409e020 msgr2=0x7f7ee412f7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:48:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.081+0000 7f7eebe9b640 10 monclient(hunting): authenticate will time out at 1933.523998s 2024-09-11T01:48:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7ee9c37640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7ee409e020 0x7f7ee412f7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:48:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7eea438640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ee4135f70 0x7f7ee409b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7eea438640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ee4135f70 0x7f7ee409b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42320/0 (socket says 172.21.15.12:42320) 2024-09-11T01:48:47.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7eea438640 1 -- 172.21.15.12:0/1881620889 learned_addr learned my addr 172.21.15.12:0/1881620889 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:48:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7eea438640 10 monclient(hunting): get_auth_request con 0x7f7ee4135f70 auth_method 0 2024-09-11T01:48:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7eea438640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7eea438640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7eea438640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7eea438640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7eea438640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7ee9436640 1 --2- 172.21.15.12:0/1881620889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee40a3280 0x7f7ee409b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7eea438640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7ee9436640 10 monclient(hunting): get_auth_request con 0x7f7ee40a3280 auth_method 0 2024-09-11T01:48:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7ee9436640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7ee9436640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7ee9436640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:47.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7ee9436640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:47.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7ee9436640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:47.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7ee9436640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:47.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7eea438640 10 monclient(hunting): handle_auth_done global_id 214176 payload 824 2024-09-11T01:48:47.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7eea438640 1 -- 172.21.15.12:0/1881620889 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7ee409e020 msgr2=0x7f7ee412f7a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:48:47.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7eea438640 1 --2- 172.21.15.12:0/1881620889 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7ee409e020 0x7f7ee412f7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:47.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7eea438640 1 -- 172.21.15.12:0/1881620889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee40a3280 msgr2=0x7f7ee409b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:47.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7eea438640 1 --2- 172.21.15.12:0/1881620889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee40a3280 0x7f7ee409b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:48:47.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7eea438640 10 monclient: _finish_hunting 0 2024-09-11T01:48:47.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7eea438640 1 monclient: found mon.noname-a 2024-09-11T01:48:47.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7eea438640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:48:47.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7eea438640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:48:47.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.082+0000 7f7eea438640 1 -- 172.21.15.12:0/1881620889 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ee4071a30 con 0x7f7ee4135f70 2024-09-11T01:48:47.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.084+0000 7f7eea438640 1 -- 172.21.15.12:0/1881620889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ee4135f70 msgr2=0x7f7ee409b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:47.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.084+0000 7f7eea438640 1 --2- 172.21.15.12:0/1881620889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ee4135f70 0x7f7ee409b8e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7edc00c780 tx=0x7f7edc00cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:47.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.084+0000 7f7ed67fc640 1 -- 172.21.15.12:0/1881620889 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7edc004b60 con 0x7f7ee4135f70 2024-09-11T01:48:47.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.084+0000 7f7ed67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:47.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.084+0000 7f7ed67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:48:47.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.084+0000 7f7ed67fc640 10 monclient: dump: 2024-09-11T01:48:47.098 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:47.098 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:47.098 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:47.098 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:47.098 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:47.099 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:47.099 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:47.099 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:47.099 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:47.099 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:47.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.084+0000 7f7ed67fc640 10 monclient: _finish_auth 0 2024-09-11T01:48:47.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.084+0000 7f7ed67fc640 10 monclient: _check_auth_tickets 2024-09-11T01:48:47.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.084+0000 7f7ed67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:48:47.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.084+0000 7f7eebe9b640 5 monclient: authenticate success, global_id 214176 2024-09-11T01:48:47.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.084+0000 7f7ed67fc640 1 -- 172.21.15.12:0/1881620889 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7edc004d20 con 0x7f7ee4135f70 2024-09-11T01:48:47.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.084+0000 7f7ed67fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:48:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.084+0000 7f7ed67fc640 1 -- 172.21.15.12:0/1881620889 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7edc00aa90 con 0x7f7ee4135f70 2024-09-11T01:48:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.084+0000 7f7ed67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.084+0000 7f7ed67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:48:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.084+0000 7f7ed67fc640 10 monclient: dump: 2024-09-11T01:48:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:47.100 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:47.101 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.084+0000 7f7ed77fe640 4 set_mon_vals no callback set 2024-09-11T01:48:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.084+0000 7f7eebe9b640 10 monclient: _renew_subs 2024-09-11T01:48:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.084+0000 7f7eebe9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.084+0000 7f7eebe9b640 1 -- 172.21.15.12:0/1881620889 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ee409c910 con 0x7f7ee4135f70 2024-09-11T01:48:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.084+0000 7f7eebe9b640 10 monclient: _renew_subs 2024-09-11T01:48:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.084+0000 7f7eebe9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:48:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.084+0000 7f7eebe9b640 1 -- 172.21.15.12:0/1881620889 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ee40a5580 con 0x7f7ee4135f70 2024-09-11T01:48:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.085+0000 7f7ed67fc640 1 -- 172.21.15.12:0/1881620889 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f7edc00acb0 con 0x7f7ee4135f70 2024-09-11T01:48:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.085+0000 7f7eebe9b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.085+0000 7f7eebe9b640 1 --2- 172.21.15.12:0/1881620889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee4071630 0x7f7ee402e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.085+0000 7f7eebe9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:47.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.085+0000 7f7eebe9b640 1 -- 172.21.15.12:0/1881620889 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee40a5130 con 0x7f7ee4071630 2024-09-11T01:48:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.085+0000 7f7ed67fc640 1 --2- 172.21.15.12:0/1881620889 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7ec803e210 0x7f7ec80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.086+0000 7f7ee9c37640 1 --2- 172.21.15.12:0/1881620889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee4071630 0x7f7ee402e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.086+0000 7f7ee9c37640 10 monclient: get_auth_request con 0x7f7ee4071630 auth_method 0 2024-09-11T01:48:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.086+0000 7f7ee9c37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.086+0000 7f7ee9c37640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.086+0000 7f7ee9c37640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.086+0000 7f7ed67fc640 1 -- 172.21.15.12:0/1881620889 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(643..643 src has 507..643) ==== 4847+0+0 (secure 0 0 0) 0x7f7edc052f70 con 0x7f7ee4135f70 2024-09-11T01:48:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.086+0000 7f7ee9c37640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.086+0000 7f7ee9c37640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.086+0000 7f7ee9c37640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.086+0000 7f7ee9436640 1 --2- 172.21.15.12:0/1881620889 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7ec803e210 0x7f7ec80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.086+0000 7f7ee9436640 10 monclient: get_auth_request con 0x7f7ec803e210 auth_method 0 2024-09-11T01:48:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.086+0000 7f7ee9c37640 10 monclient(hunting): handle_auth_done global_id 134815 payload 824 2024-09-11T01:48:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.086+0000 7f7ee9c37640 1 -- 172.21.15.12:0/1881620889 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7ee40a38f0 con 0x7f7ee4071630 2024-09-11T01:48:47.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.088+0000 7f7ee9c37640 1 -- 172.21.15.12:0/1881620889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee4071630 msgr2=0x7f7ee402e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:47.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.088+0000 7f7ee9c37640 1 --2- 172.21.15.12:0/1881620889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee4071630 0x7f7ee402e5c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0034000 tx=0x7f7ee00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:47.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.088+0000 7f7ee9436640 1 --2- 172.21.15.12:0/1881620889 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7ec803e210 0x7f7ec80406d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7ee40a2780 tx=0x7f7ed8009110 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:47.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.088+0000 7f7ed67fc640 1 -- 172.21.15.12:0/1881620889 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0042070 con 0x7f7ee4071630 2024-09-11T01:48:47.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.088+0000 7f7ed67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:47.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.088+0000 7f7ed67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:48:47.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.088+0000 7f7ed67fc640 10 monclient: dump: 2024-09-11T01:48:47.105 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:47.105 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:47.105 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:47.106 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:47.106 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:47.106 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:47.106 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:47.106 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:47.106 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:47.106 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:47.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.089+0000 7f7ed67fc640 1 -- 172.21.15.12:0/1881620889 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7ee0032a00 con 0x7f7ee4071630 2024-09-11T01:48:47.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.089+0000 7f7ed67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:48:47.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.089+0000 7f7ed67fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:48:47.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.089+0000 7f7ed67fc640 1 -- 172.21.15.12:0/1881620889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee4071630 msgr2=0x7f7ee402e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:47.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.089+0000 7f7ed67fc640 1 --2- 172.21.15.12:0/1881620889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee4071630 0x7f7ee402e5c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0034000 tx=0x7f7ee00046c0 comp rx=0 tx=0).stop 2024-09-11T01:48:47.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.089+0000 7f7ed67fc640 1 -- 172.21.15.12:0/1881620889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee4071630 msgr2=0x7f7ee402e5c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:48:47.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.089+0000 7f7ed67fc640 1 --2- 172.21.15.12:0/1881620889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee4071630 0x7f7ee402e5c0 secure :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0034000 tx=0x7f7ee00046c0 comp rx=0 tx=0).stop 2024-09-11T01:48:47.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.095+0000 7f7eebe9b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:48:47.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.095+0000 7f7eebe9b640 1 --2- 172.21.15.12:0/1881620889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee409be20 0x7f7ee4063d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:48:47.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.095+0000 7f7eebe9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:48:47.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.095+0000 7f7eebe9b640 1 -- 172.21.15.12:0/1881620889 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee409c200 con 0x7f7ee409be20 2024-09-11T01:48:47.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.095+0000 7f7ee9c37640 1 --2- 172.21.15.12:0/1881620889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee409be20 0x7f7ee4063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:48:47.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.095+0000 7f7ee9c37640 10 monclient: get_auth_request con 0x7f7ee409be20 auth_method 0 2024-09-11T01:48:47.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.095+0000 7f7ee9c37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:48:47.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.095+0000 7f7ee9c37640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:48:47.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.095+0000 7f7ee9c37640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:48:47.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.095+0000 7f7ee9c37640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:48:47.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.095+0000 7f7ee9c37640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:48:47.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.095+0000 7f7ee9c37640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:48:47.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.095+0000 7f7ee9c37640 10 monclient(hunting): handle_auth_done global_id 134818 payload 824 2024-09-11T01:48:47.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.095+0000 7f7ee9c37640 1 -- 172.21.15.12:0/1881620889 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7ee409e520 con 0x7f7ee409be20 2024-09-11T01:48:47.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.097+0000 7f7ee9c37640 1 -- 172.21.15.12:0/1881620889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee409be20 msgr2=0x7f7ee4063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:48:47.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.097+0000 7f7ee9c37640 1 --2- 172.21.15.12:0/1881620889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee409be20 0x7f7ee4063d50 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0000c00 tx=0x7f7ee0009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:48:47.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.098+0000 7f7ed67fc640 1 -- 172.21.15.12:0/1881620889 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0037040 con 0x7f7ee409be20 2024-09-11T01:48:47.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.098+0000 7f7ed67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:48:47.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.098+0000 7f7ed67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:48:47.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.098+0000 7f7ed67fc640 10 monclient: dump: 2024-09-11T01:48:47.109 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:48:47.109 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:48:47.109 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:47.110 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:48:47.110 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:48:47.110 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:48:47.110 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:48:47.110 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:48:47.110 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:48:47.111 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:48:47.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.098+0000 7f7ed67fc640 1 -- 172.21.15.12:0/1881620889 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4865 (secure 0 0 0) 0x7f7ee0003610 con 0x7f7ee409be20 2024-09-11T01:48:47.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.098+0000 7f7ed67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:48:47.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.098+0000 7f7ed67fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:48:47.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.098+0000 7f7ed67fc640 1 -- 172.21.15.12:0/1881620889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee409be20 msgr2=0x7f7ee4063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:47.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.098+0000 7f7ed67fc640 1 --2- 172.21.15.12:0/1881620889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee409be20 0x7f7ee4063d50 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0000c00 tx=0x7f7ee0009240 comp rx=0 tx=0).stop 2024-09-11T01:48:47.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.098+0000 7f7ed67fc640 1 -- 172.21.15.12:0/1881620889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee409be20 msgr2=0x7f7ee4063d50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:48:47.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.098+0000 7f7ed67fc640 1 --2- 172.21.15.12:0/1881620889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee409be20 0x7f7ee4063d50 secure :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0000c00 tx=0x7f7ee0009240 comp rx=0 tx=0).stop 2024-09-11T01:48:47.112 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:48:47.112 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:48:47.112 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:48:47.112 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:48:47.112 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 186, 2024-09-11T01:48:47.112 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 460059, 2024-09-11T01:48:47.113 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:48:47.113 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:48:47.113 INFO:teuthology.orchestra.run.smithi012.stdout: 1 2024-09-11T01:48:47.113 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:47.113 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:48:47.113 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:48:47.113 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:48:47.113 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:48:47.114 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:48:47.114 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:48:47.114 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:48:47.114 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:48:47.114 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:48:47.114 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:48:47.114 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:48:47.114 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:48:47.114 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:48:47.115 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:48:47.115 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:48:47.115 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:47.115 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:48:47.115 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:48:47.115 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:48:47.115 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:48:47.115 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:48:47.116 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:48:47.116 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:48:47.116 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:48:47.116 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:48:47.116 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:48:47.116 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:48:47.116 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:48:47.116 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:48:47.116 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:47.117 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:47.117 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:48:47.117 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:48:47.117 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:48:47.117 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:48:47.117 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:48:47.117 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:48:47.117 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:48:47.118 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:48:47.118 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:48:47.118 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:48:47.118 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:48:47.118 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:48:47.118 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:48:47.118 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:48:47.118 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:48:47.118 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:48:47.119 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:48:47.119 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:48:47.119 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:48:47.119 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:48:47.119 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:48:47.119 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:48:47.119 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:48:47.119 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:48:47.119 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:48:47.120 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:48:47.120 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:48:47.120 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:48:47.120 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:47.120 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:48:47.120 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:47.120 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:48:47.120 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:47.121 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:48:47.121 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:48:47.121 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:48:47.121 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:48:47.121 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:47.121 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:48:47.121 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:48:47.121 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:47.121 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:47.121 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:47.121 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:48:47.121 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:48:47.121 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:47.122 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:47.122 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:47.122 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:47.122 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:48:47.122 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:48:47.122 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:48:47.122 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:48:47.122 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:48:47.122 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:47.122 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:47.122 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:48:47.122 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:48:47.122 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:48:47.123 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:48:47.123 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:47.123 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:48:47.123 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:48:47.123 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:47.123 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:47.123 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:47.123 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:48:47.123 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:48:47.123 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:47.123 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:47.123 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:47.123 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:47.123 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:48:47.124 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:48:47.124 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:48:47.124 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:48:47.124 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:48:47.124 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:47.124 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:47.124 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:48:47.124 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:48:47.124 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:48:47.124 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:48:47.124 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:47.124 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:48:47.124 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:48:47.125 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:47.125 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:47.125 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:47.125 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:48:47.125 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:48:47.125 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:48:47.125 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:47.125 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:47.125 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:47.125 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:48:47.125 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:48:47.125 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:48:47.125 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:48:47.125 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:48:47.126 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:47.126 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:47.126 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:47.126 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:48:47.126 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:48:47.126 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:47.126 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:48:47.126 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:48:47.126 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:48:47.126 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:47.126 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:47.126 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:48:47.126 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:47.126 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:48:47.127 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:48:47.127 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:48:47.127 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:47.127 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:47.127 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:48:47.127 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:47.127 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:48:47.127 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:48:47.127 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:48:47.127 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:47.127 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:48:47.127 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:48:47.127 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:48:47.127 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:48:47.127 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:48:47.128 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:48:47.128 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:48:47.128 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:48:47.128 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:48:47.128 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:48:47.128 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:48:47.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.098+0000 7f7eebe9b640 1 -- 172.21.15.12:0/1881620889 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7ec803e210 msgr2=0x7f7ec80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:47.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.098+0000 7f7eebe9b640 1 --2- 172.21.15.12:0/1881620889 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7ec803e210 0x7f7ec80406d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7ee40a2780 tx=0x7f7ed8009110 comp rx=0 tx=0).stop 2024-09-11T01:48:47.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.098+0000 7f7eebe9b640 10 monclient: shutdown 2024-09-11T01:48:47.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.098+0000 7f7eebe9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:48:47.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.098+0000 7f7eebe9b640 1 -- 172.21.15.12:0/1881620889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ee4135f70 msgr2=0x7f7ee409b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:48:47.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.098+0000 7f7eebe9b640 1 --2- 172.21.15.12:0/1881620889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7ee4135f70 0x7f7ee409b8e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7edc00c780 tx=0x7f7edc00cb80 comp rx=0 tx=0).stop 2024-09-11T01:48:47.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.098+0000 7f7eea438640 1 -- 172.21.15.12:0/1881620889 reap_dead start 2024-09-11T01:48:47.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.098+0000 7f7eea438640 1 -- 172.21.15.12:0/1881620889 reap_dead start 2024-09-11T01:48:47.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.098+0000 7f7eebe9b640 1 -- 172.21.15.12:0/1881620889 shutdown_connections 2024-09-11T01:48:47.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.098+0000 7f7eebe9b640 1 -- 172.21.15.12:0/1881620889 >> 172.21.15.12:0/1881620889 conn(0x7f7ee40a3e30 msgr2=0x7f7ee4130530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:48:47.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.098+0000 7f7eebe9b640 1 -- 172.21.15.12:0/1881620889 shutdown_connections 2024-09-11T01:48:47.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:48:47.098+0000 7f7eebe9b640 1 -- 172.21.15.12:0/1881620889 wait complete. 2024-09-11T01:48:47.130 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:48:47.131 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:48:47.523 INFO:tasks.rados.rados.0.smithi071.stdout:2868: done (3 left) 2024-09-11T01:48:47.523 INFO:tasks.rados.rados.0.smithi071.stdout:2869: done (2 left) 2024-09-11T01:48:47.523 INFO:tasks.rados.rados.0.smithi071.stdout:2870: done (1 left) 2024-09-11T01:48:47.523 INFO:tasks.rados.rados.0.smithi071.stdout:2871: done (0 left) 2024-09-11T01:48:47.523 INFO:tasks.rados.rados.0.smithi071.stdout:2872: snap_remove snap 305 2024-09-11T01:48:48.525 INFO:tasks.rados.rados.0.smithi071.stdout:2872: done (0 left) 2024-09-11T01:48:48.526 INFO:tasks.rados.rados.0.smithi071.stdout:2873: write (excl) oid 43 current snap is 323 2024-09-11T01:48:48.526 INFO:tasks.rados.rados.0.smithi071.stdout:2873: seq_num 1018 ranges {709526=478431,1637728=464342,2856136=253390} 2024-09-11T01:48:48.532 INFO:tasks.rados.rados.0.smithi071.stdout:2873: writing smithi07134777-43 from 709526 to 1187957 tid 1 2024-09-11T01:48:48.538 INFO:tasks.rados.rados.0.smithi071.stdout:2873: writing smithi07134777-43 from 1637728 to 2102070 tid 2 2024-09-11T01:48:48.544 INFO:tasks.rados.rados.0.smithi071.stdout:2873: writing smithi07134777-43 from 2856136 to 3109526 tid 3 2024-09-11T01:48:48.544 INFO:tasks.rados.rados.0.smithi071.stdout:2874: write oid 14 current snap is 323 2024-09-11T01:48:48.544 INFO:tasks.rados.rados.0.smithi071.stdout:2873: finishing write tid 1 to smithi07134777-43 2024-09-11T01:48:48.544 INFO:tasks.rados.rados.0.smithi071.stdout:2874: seq_num 1019 ranges {726333=670033,2096210=724353} 2024-09-11T01:48:48.551 INFO:tasks.rados.rados.0.smithi071.stdout:2874: writing smithi07134777-14 from 726333 to 1396366 tid 1 2024-09-11T01:48:48.560 INFO:tasks.rados.rados.0.smithi071.stdout:2874: writing smithi07134777-14 from 2096210 to 2820563 tid 2 2024-09-11T01:48:48.560 INFO:tasks.rados.rados.0.smithi071.stdout:2875: read oid 42 snap -1 2024-09-11T01:48:48.560 INFO:tasks.rados.rados.0.smithi071.stdout:2875: expect (ObjNum 983 snap 314 seq_num 983) 2024-09-11T01:48:48.560 INFO:tasks.rados.rados.0.smithi071.stdout:2876: copy_from oid 19 from oid 31 current snap is 323 2024-09-11T01:48:48.560 INFO:tasks.rados.rados.0.smithi071.stdout:2873: finishing write tid 2 to smithi07134777-43 2024-09-11T01:48:48.560 INFO:tasks.rados.rados.0.smithi071.stdout:2873: finishing write tid 3 to smithi07134777-43 2024-09-11T01:48:48.560 INFO:tasks.rados.rados.0.smithi071.stdout:2873: finishing write tid 4 to smithi07134777-43 2024-09-11T01:48:48.560 INFO:tasks.rados.rados.0.smithi071.stdout:2873: finishing write tid 5 to smithi07134777-43 2024-09-11T01:48:48.560 INFO:tasks.rados.rados.0.smithi071.stdout:2873: oid 43 updating version 0 to 1921 2024-09-11T01:48:48.561 INFO:tasks.rados.rados.0.smithi071.stdout:2873: oid 43 updating version 1921 to 1924 2024-09-11T01:48:48.561 INFO:tasks.rados.rados.0.smithi071.stdout:2873: oid 43 version 1924 is already newer than 1922 2024-09-11T01:48:48.561 INFO:tasks.rados.rados.0.smithi071.stdout:2873: oid 43 version 1924 is already newer than 1923 2024-09-11T01:48:48.561 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 1924 (ObjNum 1018 snap 323 seq_num 1018) dirty exists 2024-09-11T01:48:48.561 INFO:tasks.rados.rados.0.smithi071.stdout:2873: left oid 43 (ObjNum 1018 snap 323 seq_num 1018) 2024-09-11T01:48:48.561 INFO:tasks.rados.rados.0.smithi071.stdout:2874: finishing write tid 1 to smithi07134777-14 2024-09-11T01:48:48.561 INFO:tasks.rados.rados.0.smithi071.stdout:2873: done (3 left) 2024-09-11T01:48:48.561 INFO:tasks.rados.rados.0.smithi071.stdout:2877: snap_create 2024-09-11T01:48:48.565 INFO:tasks.rados.rados.0.smithi071.stdout:2875: expect (ObjNum 983 snap 314 seq_num 983) 2024-09-11T01:48:48.565 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:48:48.620 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:48:48.719 INFO:tasks.rados.rados.0.smithi071.stdout:2874: finishing write tid 2 to smithi07134777-14 2024-09-11T01:48:48.719 INFO:tasks.rados.rados.0.smithi071.stdout:2874: finishing write tid 3 to smithi07134777-14 2024-09-11T01:48:48.719 INFO:tasks.rados.rados.0.smithi071.stdout:2874: finishing write tid 4 to smithi07134777-14 2024-09-11T01:48:48.719 INFO:tasks.rados.rados.0.smithi071.stdout:2874: oid 14 updating version 0 to 2037 2024-09-11T01:48:48.719 INFO:tasks.rados.rados.0.smithi071.stdout:2874: oid 14 version 2037 is already newer than 2036 2024-09-11T01:48:48.719 INFO:tasks.rados.rados.0.smithi071.stdout:2874: oid 14 updating version 2037 to 2038 2024-09-11T01:48:48.719 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 2038 (ObjNum 1019 snap 323 seq_num 1019) dirty exists 2024-09-11T01:48:48.719 INFO:tasks.rados.rados.0.smithi071.stdout:2874: left oid 14 (ObjNum 1019 snap 323 seq_num 1019) 2024-09-11T01:48:48.720 INFO:tasks.rados.rados.0.smithi071.stdout:2876: finishing copy_from racing read to smithi07134777-19 2024-09-11T01:48:48.720 INFO:tasks.rados.rados.0.smithi071.stdout:2876: finishing copy_from to smithi07134777-19 2024-09-11T01:48:48.720 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 1760 (ObjNum 988 snap 317 seq_num 988) dirty exists 2024-09-11T01:48:48.816 INFO:tasks.rados.rados.0.smithi071.stdout:2874: done (3 left) 2024-09-11T01:48:48.816 INFO:tasks.rados.rados.0.smithi071.stdout:2875: done (2 left) 2024-09-11T01:48:48.817 INFO:tasks.rados.rados.0.smithi071.stdout:2876: done (1 left) 2024-09-11T01:48:48.817 INFO:tasks.rados.rados.0.smithi071.stdout:2877: done (0 left) 2024-09-11T01:48:48.817 INFO:tasks.rados.rados.0.smithi071.stdout:2878: delete oid 39 current snap is 324 2024-09-11T01:48:48.821 INFO:tasks.rados.rados.0.smithi071.stdout:2878: done (0 left) 2024-09-11T01:48:48.822 INFO:tasks.rados.rados.0.smithi071.stdout:2879: snap_create 2024-09-11T01:48:49.824 INFO:tasks.rados.rados.0.smithi071.stdout:2879: done (0 left) 2024-09-11T01:48:49.824 INFO:tasks.rados.rados.0.smithi071.stdout:2880: copy_from oid 22 from oid 10 current snap is 325 2024-09-11T01:48:49.824 INFO:tasks.rados.rados.0.smithi071.stdout:2881: read oid 7 snap -1 2024-09-11T01:48:49.824 INFO:tasks.rados.rados.0.smithi071.stdout:2881: expect (ObjNum 1014 snap 322 seq_num 1014) 2024-09-11T01:48:49.824 INFO:tasks.rados.rados.0.smithi071.stdout:2882: snap_create 2024-09-11T01:48:49.832 INFO:tasks.rados.rados.0.smithi071.stdout:2881: expect (ObjNum 1014 snap 322 seq_num 1014) 2024-09-11T01:48:49.832 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 153968 2024-09-11T01:48:49.912 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2499170 2024-09-11T01:48:49.961 INFO:tasks.rados.rados.0.smithi071.stdout:2880: finishing copy_from racing read to smithi07134777-22 2024-09-11T01:48:49.961 INFO:tasks.rados.rados.0.smithi071.stdout:2880: finishing copy_from to smithi07134777-22 2024-09-11T01:48:49.961 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 1076 (ObjNum 990 snap 317 seq_num 990) dirty exists 2024-09-11T01:48:50.825 INFO:tasks.rados.rados.0.smithi071.stdout:2880: done (2 left) 2024-09-11T01:48:50.825 INFO:tasks.rados.rados.0.smithi071.stdout:2881: done (1 left) 2024-09-11T01:48:50.825 INFO:tasks.rados.rados.0.smithi071.stdout:2882: done (0 left) 2024-09-11T01:48:50.825 INFO:tasks.rados.rados.0.smithi071.stdout:2883: rollback oid 4 current snap is 326 2024-09-11T01:48:50.825 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 4 to 274 2024-09-11T01:48:50.826 INFO:tasks.rados.rados.0.smithi071.stdout:2884: snap_create 2024-09-11T01:48:50.833 INFO:tasks.rados.rados.0.smithi071.stdout:2883: finishing rollback tid 0 to smithi07134777-4 2024-09-11T01:48:50.836 INFO:tasks.rados.rados.0.smithi071.stdout:2883: finishing rollback tid 1 to smithi07134777-4 2024-09-11T01:48:50.836 INFO:tasks.rados.rados.0.smithi071.stdout:2883: finishing rollback tid 2 to smithi07134777-4 2024-09-11T01:48:50.836 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 1307 (ObjNum 828 snap 252 seq_num 828) dirty exists 2024-09-11T01:48:51.830 INFO:tasks.rados.rados.0.smithi071.stdout:2883: done (1 left) 2024-09-11T01:48:51.830 INFO:tasks.rados.rados.0.smithi071.stdout:2884: done (0 left) 2024-09-11T01:48:51.830 INFO:tasks.rados.rados.0.smithi071.stdout:2885: snap_remove snap 287 2024-09-11T01:48:52.832 INFO:tasks.rados.rados.0.smithi071.stdout:2885: done (0 left) 2024-09-11T01:48:52.832 INFO:tasks.rados.rados.0.smithi071.stdout:2886: read oid 46 snap 315 2024-09-11T01:48:52.832 INFO:tasks.rados.rados.0.smithi071.stdout:2886: expect (ObjNum 841 snap 255 seq_num 841) 2024-09-11T01:48:52.832 INFO:tasks.rados.rados.0.smithi071.stdout:2887: read oid 12 snap -1 2024-09-11T01:48:52.832 INFO:tasks.rados.rados.0.smithi071.stdout:2887: expect deleted 2024-09-11T01:48:52.833 INFO:tasks.rados.rados.0.smithi071.stdout:2888: snap_create 2024-09-11T01:48:52.841 INFO:tasks.rados.rados.0.smithi071.stdout:2886: expect (ObjNum 841 snap 255 seq_num 841) 2024-09-11T01:48:52.852 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1436050 2024-09-11T01:48:52.890 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1181790 2024-09-11T01:48:52.916 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:48:53.816 INFO:tasks.rados.rados.0.smithi071.stdout:2886: done (2 left) 2024-09-11T01:48:53.817 INFO:tasks.rados.rados.0.smithi071.stdout:2887: done (1 left) 2024-09-11T01:48:53.817 INFO:tasks.rados.rados.0.smithi071.stdout:2888: done (0 left) 2024-09-11T01:48:53.817 INFO:tasks.rados.rados.0.smithi071.stdout:2889: read oid 26 snap -1 2024-09-11T01:48:53.817 INFO:tasks.rados.rados.0.smithi071.stdout:2889: expect deleted 2024-09-11T01:48:53.817 INFO:tasks.rados.rados.0.smithi071.stdout:2890: read oid 22 snap 324 2024-09-11T01:48:53.817 INFO:tasks.rados.rados.0.smithi071.stdout:2890: expect deleted 2024-09-11T01:48:53.817 INFO:tasks.rados.rados.0.smithi071.stdout:2891: copy_from oid 44 from oid 41 current snap is 328 2024-09-11T01:48:53.817 INFO:tasks.rados.rados.0.smithi071.stdout:2892: read oid 39 snap -1 2024-09-11T01:48:53.817 INFO:tasks.rados.rados.0.smithi071.stdout:2892: expect deleted 2024-09-11T01:48:53.817 INFO:tasks.rados.rados.0.smithi071.stdout:2893: delete oid 34 current snap is 328 2024-09-11T01:48:53.825 INFO:tasks.rados.rados.0.smithi071.stdout:2889: done (4 left) 2024-09-11T01:48:53.825 INFO:tasks.rados.rados.0.smithi071.stdout:2890: done (3 left) 2024-09-11T01:48:53.825 INFO:tasks.rados.rados.0.smithi071.stdout:2892: done (2 left) 2024-09-11T01:48:53.825 INFO:tasks.rados.rados.0.smithi071.stdout:2893: done (1 left) 2024-09-11T01:48:53.825 INFO:tasks.rados.rados.0.smithi071.stdout:2894: delete oid 47 current snap is 328 2024-09-11T01:48:53.827 INFO:tasks.rados.rados.0.smithi071.stdout:2891: finishing copy_from racing read to smithi07134777-44 2024-09-11T01:48:53.831 INFO:tasks.rados.rados.0.smithi071.stdout:2891: finishing copy_from to smithi07134777-44 2024-09-11T01:48:53.831 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 484 (ObjNum 960 snap 304 seq_num 960) dirty exists 2024-09-11T01:48:53.831 INFO:tasks.rados.rados.0.smithi071.stdout:2891: done (1 left) 2024-09-11T01:48:53.832 INFO:tasks.rados.rados.0.smithi071.stdout:2894: done (0 left) 2024-09-11T01:48:53.832 INFO:tasks.rados.rados.0.smithi071.stdout:2895: copy_from oid 22 from oid 7 current snap is 328 2024-09-11T01:48:53.832 INFO:tasks.rados.rados.0.smithi071.stdout:2896: write (excl) oid 25 current snap is 328 2024-09-11T01:48:53.832 INFO:tasks.rados.rados.0.smithi071.stdout:2896: seq_num 1023 ranges {793561=571029,1813785=731691,2993251=200310} 2024-09-11T01:48:53.839 INFO:tasks.rados.rados.0.smithi071.stdout:2896: writing smithi07134777-25 from 793561 to 1364590 tid 1 2024-09-11T01:48:53.845 INFO:tasks.rados.rados.0.smithi071.stdout:2896: writing smithi07134777-25 from 1813785 to 2545476 tid 2 2024-09-11T01:48:53.850 INFO:tasks.rados.rados.0.smithi071.stdout:2896: writing smithi07134777-25 from 2993251 to 3193561 tid 3 2024-09-11T01:48:53.850 INFO:tasks.rados.rados.0.smithi071.stdout:2897: read oid 2 snap -1 2024-09-11T01:48:53.850 INFO:tasks.rados.rados.0.smithi071.stdout:2897: expect (ObjNum 978 snap 306 seq_num 978) 2024-09-11T01:48:53.850 INFO:tasks.rados.rados.0.smithi071.stdout:2898: write oid 3 current snap is 328 2024-09-11T01:48:53.850 INFO:tasks.rados.rados.0.smithi071.stdout:2895: finishing copy_from racing read to smithi07134777-22 2024-09-11T01:48:53.850 INFO:tasks.rados.rados.0.smithi071.stdout:2895: finishing copy_from to smithi07134777-22 2024-09-11T01:48:53.851 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 2025 (ObjNum 1014 snap 322 seq_num 1014) dirty exists 2024-09-11T01:48:53.851 INFO:tasks.rados.rados.0.smithi071.stdout:2896: finishing write tid 1 to smithi07134777-25 2024-09-11T01:48:53.851 INFO:tasks.rados.rados.0.smithi071.stdout:2898: seq_num 1024 ranges {410368=646278,1472267=475349,2504321=405600} 2024-09-11T01:48:53.855 INFO:tasks.rados.rados.0.smithi071.stdout:2898: writing smithi07134777-3 from 410368 to 1056646 tid 1 2024-09-11T01:48:53.860 INFO:tasks.rados.rados.0.smithi071.stdout:2898: writing smithi07134777-3 from 1472267 to 1947616 tid 2 2024-09-11T01:48:53.866 INFO:tasks.rados.rados.0.smithi071.stdout:2898: writing smithi07134777-3 from 2504321 to 2909921 tid 3 2024-09-11T01:48:53.866 INFO:tasks.rados.rados.0.smithi071.stdout:2895: done (3 left) 2024-09-11T01:48:53.866 INFO:tasks.rados.rados.0.smithi071.stdout:2899: copy_from oid 10 from oid 35 current snap is 328 2024-09-11T01:48:53.866 INFO:tasks.rados.rados.0.smithi071.stdout:2896: finishing write tid 2 to smithi07134777-25 2024-09-11T01:48:53.866 INFO:tasks.rados.rados.0.smithi071.stdout:2897: expect (ObjNum 978 snap 306 seq_num 978) 2024-09-11T01:48:53.875 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1684616 2024-09-11T01:48:53.921 INFO:tasks.rados.rados.0.smithi071.stdout:2896: finishing write tid 3 to smithi07134777-25 2024-09-11T01:48:53.921 INFO:tasks.rados.rados.0.smithi071.stdout:2896: finishing write tid 4 to smithi07134777-25 2024-09-11T01:48:53.921 INFO:tasks.rados.rados.0.smithi071.stdout:2896: finishing write tid 5 to smithi07134777-25 2024-09-11T01:48:53.921 INFO:tasks.rados.rados.0.smithi071.stdout:2896: oid 25 updating version 0 to 1017 2024-09-11T01:48:53.921 INFO:tasks.rados.rados.0.smithi071.stdout:2896: oid 25 updating version 1017 to 1018 2024-09-11T01:48:53.921 INFO:tasks.rados.rados.0.smithi071.stdout:2896: oid 25 updating version 1018 to 1019 2024-09-11T01:48:53.922 INFO:tasks.rados.rados.0.smithi071.stdout:2896: oid 25 updating version 1019 to 1020 2024-09-11T01:48:53.922 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 1020 (ObjNum 1023 snap 328 seq_num 1023) dirty exists 2024-09-11T01:48:53.922 INFO:tasks.rados.rados.0.smithi071.stdout:2896: left oid 25 (ObjNum 1023 snap 328 seq_num 1023) 2024-09-11T01:48:53.922 INFO:tasks.rados.rados.0.smithi071.stdout:2898: finishing write tid 1 to smithi07134777-3 2024-09-11T01:48:53.922 INFO:tasks.rados.rados.0.smithi071.stdout:2898: finishing write tid 2 to smithi07134777-3 2024-09-11T01:48:53.922 INFO:tasks.rados.rados.0.smithi071.stdout:2896: done (3 left) 2024-09-11T01:48:53.922 INFO:tasks.rados.rados.0.smithi071.stdout:2897: done (2 left) 2024-09-11T01:48:53.922 INFO:tasks.rados.rados.0.smithi071.stdout:2900: snap_create 2024-09-11T01:48:53.922 INFO:tasks.rados.rados.0.smithi071.stdout:2898: finishing write tid 3 to smithi07134777-3 2024-09-11T01:48:53.922 INFO:tasks.rados.rados.0.smithi071.stdout:2898: finishing write tid 4 to smithi07134777-3 2024-09-11T01:48:53.922 INFO:tasks.rados.rados.0.smithi071.stdout:2898: finishing write tid 5 to smithi07134777-3 2024-09-11T01:48:53.922 INFO:tasks.rados.rados.0.smithi071.stdout:2898: oid 3 updating version 0 to 2029 2024-09-11T01:48:53.922 INFO:tasks.rados.rados.0.smithi071.stdout:2898: oid 3 version 2029 is already newer than 2026 2024-09-11T01:48:53.922 INFO:tasks.rados.rados.0.smithi071.stdout:2898: oid 3 version 2029 is already newer than 2028 2024-09-11T01:48:53.923 INFO:tasks.rados.rados.0.smithi071.stdout:2898: oid 3 version 2029 is already newer than 2027 2024-09-11T01:48:53.923 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 2029 (ObjNum 1024 snap 328 seq_num 1024) dirty exists 2024-09-11T01:48:53.923 INFO:tasks.rados.rados.0.smithi071.stdout:2898: left oid 3 (ObjNum 1024 snap 328 seq_num 1024) 2024-09-11T01:48:53.923 INFO:tasks.rados.rados.0.smithi071.stdout:2899: finishing copy_from racing read to smithi07134777-10 2024-09-11T01:48:53.923 INFO:tasks.rados.rados.0.smithi071.stdout:2899: finishing copy_from to smithi07134777-10 2024-09-11T01:48:53.923 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 384 (ObjNum 949 snap 300 seq_num 949) dirty exists 2024-09-11T01:48:54.843 INFO:tasks.rados.rados.0.smithi071.stdout:2898: done (2 left) 2024-09-11T01:48:54.843 INFO:tasks.rados.rados.0.smithi071.stdout:2899: done (1 left) 2024-09-11T01:48:54.843 INFO:tasks.rados.rados.0.smithi071.stdout:2900: done (0 left) 2024-09-11T01:48:54.843 INFO:tasks.rados.rados.0.smithi071.stdout:2901: snap_create 2024-09-11T01:48:55.847 INFO:tasks.rados.rados.0.smithi071.stdout:2901: done (0 left) 2024-09-11T01:48:55.847 INFO:tasks.rados.rados.0.smithi071.stdout:2902: rollback oid 7 current snap is 330 2024-09-11T01:48:55.847 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 7 to 324 2024-09-11T01:48:55.847 INFO:tasks.rados.rados.0.smithi071.stdout:2903: delete oid 37 current snap is 330 2024-09-11T01:48:55.850 INFO:tasks.rados.rados.0.smithi071.stdout:2903: done (1 left) 2024-09-11T01:48:55.850 INFO:tasks.rados.rados.0.smithi071.stdout:2904: delete oid 23 current snap is 330 2024-09-11T01:48:55.855 INFO:tasks.rados.rados.0.smithi071.stdout:2904: done (1 left) 2024-09-11T01:48:55.855 INFO:tasks.rados.rados.0.smithi071.stdout:2905: copy_from oid 24 from oid 3 current snap is 330 2024-09-11T01:48:55.855 INFO:tasks.rados.rados.0.smithi071.stdout:2906: write oid 42 current snap is 330 2024-09-11T01:48:55.855 INFO:tasks.rados.rados.0.smithi071.stdout:2906: seq_num 1027 ranges {460789=588378,1764066=739029,3219296=41493} 2024-09-11T01:48:55.861 INFO:tasks.rados.rados.0.smithi071.stdout:2906: writing smithi07134777-42 from 460789 to 1049167 tid 1 2024-09-11T01:48:55.869 INFO:tasks.rados.rados.0.smithi071.stdout:2906: writing smithi07134777-42 from 1764066 to 2503095 tid 2 2024-09-11T01:48:55.873 INFO:tasks.rados.rados.0.smithi071.stdout:2906: writing smithi07134777-42 from 3219296 to 3260789 tid 3 2024-09-11T01:48:55.874 INFO:tasks.rados.rados.0.smithi071.stdout:2907: snap_create 2024-09-11T01:48:55.874 INFO:tasks.rados.rados.0.smithi071.stdout:2902: finishing rollback tid 0 to smithi07134777-7 2024-09-11T01:48:55.874 INFO:tasks.rados.rados.0.smithi071.stdout:2902: finishing rollback tid 1 to smithi07134777-7 2024-09-11T01:48:55.874 INFO:tasks.rados.rados.0.smithi071.stdout:2902: finishing rollback tid 2 to smithi07134777-7 2024-09-11T01:48:55.874 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 2050 (ObjNum 1014 snap 322 seq_num 1014) dirty exists 2024-09-11T01:48:55.874 INFO:tasks.rados.rados.0.smithi071.stdout:2906: finishing write tid 1 to smithi07134777-42 2024-09-11T01:48:55.874 INFO:tasks.rados.rados.0.smithi071.stdout:2906: finishing write tid 2 to smithi07134777-42 2024-09-11T01:48:55.874 INFO:tasks.rados.rados.0.smithi071.stdout:2905: finishing copy_from racing read to smithi07134777-24 2024-09-11T01:48:55.875 INFO:tasks.rados.rados.0.smithi071.stdout:2906: finishing write tid 3 to smithi07134777-42 2024-09-11T01:48:55.875 INFO:tasks.rados.rados.0.smithi071.stdout:2906: finishing write tid 4 to smithi07134777-42 2024-09-11T01:48:55.876 INFO:tasks.rados.rados.0.smithi071.stdout:2906: finishing write tid 5 to smithi07134777-42 2024-09-11T01:48:55.876 INFO:tasks.rados.rados.0.smithi071.stdout:2906: oid 42 updating version 0 to 872 2024-09-11T01:48:55.876 INFO:tasks.rados.rados.0.smithi071.stdout:2906: oid 42 version 872 is already newer than 871 2024-09-11T01:48:55.876 INFO:tasks.rados.rados.0.smithi071.stdout:2906: oid 42 updating version 872 to 874 2024-09-11T01:48:55.876 INFO:tasks.rados.rados.0.smithi071.stdout:2906: oid 42 version 874 is already newer than 873 2024-09-11T01:48:55.876 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 874 (ObjNum 1027 snap 330 seq_num 1027) dirty exists 2024-09-11T01:48:55.876 INFO:tasks.rados.rados.0.smithi071.stdout:2906: left oid 42 (ObjNum 1027 snap 330 seq_num 1027) 2024-09-11T01:48:55.877 INFO:tasks.rados.rados.0.smithi071.stdout:2905: finishing copy_from to smithi07134777-24 2024-09-11T01:48:55.877 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 2029 (ObjNum 1024 snap 328 seq_num 1024) dirty exists 2024-09-11T01:48:56.860 INFO:tasks.rados.rados.0.smithi071.stdout:2902: done (3 left) 2024-09-11T01:48:56.860 INFO:tasks.rados.rados.0.smithi071.stdout:2905: done (2 left) 2024-09-11T01:48:56.860 INFO:tasks.rados.rados.0.smithi071.stdout:2906: done (1 left) 2024-09-11T01:48:56.860 INFO:tasks.rados.rados.0.smithi071.stdout:2907: done (0 left) 2024-09-11T01:48:56.860 INFO:tasks.rados.rados.0.smithi071.stdout:2908: write oid 35 current snap is 331 2024-09-11T01:48:56.860 INFO:tasks.rados.rados.0.smithi071.stdout:2908: seq_num 1028 ranges {477596=663627,1706195=482687,2614013=567041} 2024-09-11T01:48:56.865 INFO:tasks.rados.rados.0.smithi071.stdout:2908: writing smithi07134777-35 from 477596 to 1141223 tid 1 2024-09-11T01:48:56.871 INFO:tasks.rados.rados.0.smithi071.stdout:2908: writing smithi07134777-35 from 1706195 to 2188882 tid 2 2024-09-11T01:48:56.876 INFO:tasks.rados.rados.0.smithi071.stdout:2908: writing smithi07134777-35 from 2614013 to 3181054 tid 3 2024-09-11T01:48:56.877 INFO:tasks.rados.rados.0.smithi071.stdout:2909: write oid 39 current snap is 331 2024-09-11T01:48:56.877 INFO:tasks.rados.rados.0.smithi071.stdout:2909: seq_num 1029 ranges {494403=738876,1764677=626345,3041436=252967} 2024-09-11T01:48:56.883 INFO:tasks.rados.rados.0.smithi071.stdout:2909: writing smithi07134777-39 from 494403 to 1233279 tid 1 2024-09-11T01:48:56.889 INFO:tasks.rados.rados.0.smithi071.stdout:2909: writing smithi07134777-39 from 1764677 to 2391022 tid 2 2024-09-11T01:48:56.894 INFO:tasks.rados.rados.0.smithi071.stdout:2909: writing smithi07134777-39 from 3041436 to 3294403 tid 3 2024-09-11T01:48:56.894 INFO:tasks.rados.rados.0.smithi071.stdout:2910: write oid 19 current snap is 331 2024-09-11T01:48:56.895 INFO:tasks.rados.rados.0.smithi071.stdout:2908: finishing write tid 1 to smithi07134777-35 2024-09-11T01:48:56.895 INFO:tasks.rados.rados.0.smithi071.stdout:2908: finishing write tid 2 to smithi07134777-35 2024-09-11T01:48:56.895 INFO:tasks.rados.rados.0.smithi071.stdout:2908: finishing write tid 3 to smithi07134777-35 2024-09-11T01:48:56.895 INFO:tasks.rados.rados.0.smithi071.stdout:2910: seq_num 1030 ranges {511210=414125,1423159=486356,2668859=589585} 2024-09-11T01:48:56.899 INFO:tasks.rados.rados.0.smithi071.stdout:2910: writing smithi07134777-19 from 511210 to 925335 tid 1 2024-09-11T01:48:56.904 INFO:tasks.rados.rados.0.smithi071.stdout:2910: writing smithi07134777-19 from 1423159 to 1909515 tid 2 2024-09-11T01:48:56.912 INFO:tasks.rados.rados.0.smithi071.stdout:2910: writing smithi07134777-19 from 2668859 to 3258444 tid 3 2024-09-11T01:48:56.912 INFO:tasks.rados.rados.0.smithi071.stdout:2911: delete oid 28 current snap is 331 2024-09-11T01:48:56.912 INFO:tasks.rados.rados.0.smithi071.stdout:2908: finishing write tid 4 to smithi07134777-35 2024-09-11T01:48:56.912 INFO:tasks.rados.rados.0.smithi071.stdout:2908: finishing write tid 5 to smithi07134777-35 2024-09-11T01:48:56.912 INFO:tasks.rados.rados.0.smithi071.stdout:2908: oid 35 updating version 0 to 1097 2024-09-11T01:48:56.912 INFO:tasks.rados.rados.0.smithi071.stdout:2908: oid 35 version 1097 is already newer than 1096 2024-09-11T01:48:56.912 INFO:tasks.rados.rados.0.smithi071.stdout:2908: oid 35 updating version 1097 to 1098 2024-09-11T01:48:56.913 INFO:tasks.rados.rados.0.smithi071.stdout:2908: oid 35 version 1098 is already newer than 1095 2024-09-11T01:48:56.913 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 1098 (ObjNum 1028 snap 331 seq_num 1028) dirty exists 2024-09-11T01:48:56.913 INFO:tasks.rados.rados.0.smithi071.stdout:2908: left oid 35 (ObjNum 1028 snap 331 seq_num 1028) 2024-09-11T01:48:56.913 INFO:tasks.rados.rados.0.smithi071.stdout:2909: finishing write tid 1 to smithi07134777-39 2024-09-11T01:48:56.913 INFO:tasks.rados.rados.0.smithi071.stdout:2909: finishing write tid 2 to smithi07134777-39 2024-09-11T01:48:56.913 INFO:tasks.rados.rados.0.smithi071.stdout:2909: finishing write tid 3 to smithi07134777-39 2024-09-11T01:48:56.913 INFO:tasks.rados.rados.0.smithi071.stdout:2909: finishing write tid 4 to smithi07134777-39 2024-09-11T01:48:56.913 INFO:tasks.rados.rados.0.smithi071.stdout:2909: finishing write tid 5 to smithi07134777-39 2024-09-11T01:48:56.913 INFO:tasks.rados.rados.0.smithi071.stdout:2909: oid 39 updating version 0 to 1102 2024-09-11T01:48:56.913 INFO:tasks.rados.rados.0.smithi071.stdout:2909: oid 39 version 1102 is already newer than 1101 2024-09-11T01:48:56.913 INFO:tasks.rados.rados.0.smithi071.stdout:2909: oid 39 version 1102 is already newer than 1099 2024-09-11T01:48:56.913 INFO:tasks.rados.rados.0.smithi071.stdout:2909: oid 39 version 1102 is already newer than 1100 2024-09-11T01:48:56.913 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 1102 (ObjNum 1029 snap 331 seq_num 1029) dirty exists 2024-09-11T01:48:56.913 INFO:tasks.rados.rados.0.smithi071.stdout:2909: left oid 39 (ObjNum 1029 snap 331 seq_num 1029) 2024-09-11T01:48:56.914 INFO:tasks.rados.rados.0.smithi071.stdout:2910: finishing write tid 1 to smithi07134777-19 2024-09-11T01:48:56.914 INFO:tasks.rados.rados.0.smithi071.stdout:2910: finishing write tid 2 to smithi07134777-19 2024-09-11T01:48:56.915 INFO:tasks.rados.rados.0.smithi071.stdout:2908: done (3 left) 2024-09-11T01:48:56.915 INFO:tasks.rados.rados.0.smithi071.stdout:2909: done (2 left) 2024-09-11T01:48:56.915 INFO:tasks.rados.rados.0.smithi071.stdout:2911: done (1 left) 2024-09-11T01:48:56.915 INFO:tasks.rados.rados.0.smithi071.stdout:2912: rollback oid 22 current snap is 331 2024-09-11T01:48:56.915 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 22 to 299 2024-09-11T01:48:56.915 INFO:tasks.rados.rados.0.smithi071.stdout:2913: read oid 49 snap -1 2024-09-11T01:48:56.915 INFO:tasks.rados.rados.0.smithi071.stdout:2913: expect (ObjNum 1017 snap 322 seq_num 1017) 2024-09-11T01:48:56.915 INFO:tasks.rados.rados.0.smithi071.stdout:2914: write oid 3 current snap is 331 2024-09-11T01:48:56.916 INFO:tasks.rados.rados.0.smithi071.stdout:2914: seq_num 1032 ranges {544824=564623,1823770=490025,3007352=337472} 2024-09-11T01:48:56.921 INFO:tasks.rados.rados.0.smithi071.stdout:2914: writing smithi07134777-3 from 544824 to 1109447 tid 1 2024-09-11T01:48:56.928 INFO:tasks.rados.rados.0.smithi071.stdout:2914: writing smithi07134777-3 from 1823770 to 2313795 tid 2 2024-09-11T01:48:56.934 INFO:tasks.rados.rados.0.smithi071.stdout:2914: writing smithi07134777-3 from 3007352 to 3344824 tid 3 2024-09-11T01:48:56.934 INFO:tasks.rados.rados.0.smithi071.stdout:2915: read oid 2 snap -1 2024-09-11T01:48:56.934 INFO:tasks.rados.rados.0.smithi071.stdout:2915: expect (ObjNum 978 snap 306 seq_num 978) 2024-09-11T01:48:56.934 INFO:tasks.rados.rados.0.smithi071.stdout:2916: snap_remove snap 304 2024-09-11T01:48:57.863 INFO:tasks.rados.rados.0.smithi071.stdout:2916: done (5 left) 2024-09-11T01:48:57.863 INFO:tasks.rados.rados.0.smithi071.stdout:2917: snap_remove snap 244 2024-09-11T01:48:58.817 INFO:tasks.rados.rados.0.smithi071.stdout:2917: done (5 left) 2024-09-11T01:48:58.817 INFO:tasks.rados.rados.0.smithi071.stdout:2918: read oid 29 snap -1 2024-09-11T01:48:58.817 INFO:tasks.rados.rados.0.smithi071.stdout:2918: expect deleted 2024-09-11T01:48:58.817 INFO:tasks.rados.rados.0.smithi071.stdout:2919: read oid 46 snap -1 2024-09-11T01:48:58.818 INFO:tasks.rados.rados.0.smithi071.stdout:2919: expect (ObjNum 841 snap 255 seq_num 841) 2024-09-11T01:48:58.818 INFO:tasks.rados.rados.0.smithi071.stdout:2920: snap_remove snap 273 2024-09-11T01:48:59.822 INFO:tasks.rados.rados.0.smithi071.stdout:2920: done (7 left) 2024-09-11T01:48:59.822 INFO:tasks.rados.rados.0.smithi071.stdout:2921: snap_create 2024-09-11T01:48:59.822 INFO:tasks.rados.rados.0.smithi071.stdout:2910: finishing write tid 3 to smithi07134777-19 2024-09-11T01:48:59.822 INFO:tasks.rados.rados.0.smithi071.stdout:2910: finishing write tid 4 to smithi07134777-19 2024-09-11T01:48:59.823 INFO:tasks.rados.rados.0.smithi071.stdout:2910: finishing write tid 5 to smithi07134777-19 2024-09-11T01:48:59.823 INFO:tasks.rados.rados.0.smithi071.stdout:2910: oid 19 updating version 0 to 1761 2024-09-11T01:48:59.823 INFO:tasks.rados.rados.0.smithi071.stdout:2910: oid 19 updating version 1761 to 1764 2024-09-11T01:48:59.823 INFO:tasks.rados.rados.0.smithi071.stdout:2910: oid 19 version 1764 is already newer than 1763 2024-09-11T01:48:59.823 INFO:tasks.rados.rados.0.smithi071.stdout:2910: oid 19 version 1764 is already newer than 1762 2024-09-11T01:48:59.823 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 1764 (ObjNum 1030 snap 331 seq_num 1030) dirty exists 2024-09-11T01:48:59.823 INFO:tasks.rados.rados.0.smithi071.stdout:2910: left oid 19 (ObjNum 1030 snap 331 seq_num 1030) 2024-09-11T01:48:59.823 INFO:tasks.rados.rados.0.smithi071.stdout:2912: finishing rollback tid 0 to smithi07134777-22 2024-09-11T01:48:59.823 INFO:tasks.rados.rados.0.smithi071.stdout:2912: finishing rollback tid 1 to smithi07134777-22 2024-09-11T01:48:59.823 INFO:tasks.rados.rados.0.smithi071.stdout:2912: finishing rollback tid 2 to smithi07134777-22 2024-09-11T01:48:59.823 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 2032 (ObjNum 745 snap 227 seq_num 745) dirty exists 2024-09-11T01:48:59.823 INFO:tasks.rados.rados.0.smithi071.stdout:2914: finishing write tid 1 to smithi07134777-3 2024-09-11T01:48:59.823 INFO:tasks.rados.rados.0.smithi071.stdout:2913: expect (ObjNum 1017 snap 322 seq_num 1017) 2024-09-11T01:48:59.823 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 442924 2024-09-11T01:48:59.886 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:00.009 INFO:tasks.rados.rados.0.smithi071.stdout:2914: finishing write tid 2 to smithi07134777-3 2024-09-11T01:49:00.009 INFO:tasks.rados.rados.0.smithi071.stdout:2914: finishing write tid 3 to smithi07134777-3 2024-09-11T01:49:00.009 INFO:tasks.rados.rados.0.smithi071.stdout:2914: finishing write tid 4 to smithi07134777-3 2024-09-11T01:49:00.009 INFO:tasks.rados.rados.0.smithi071.stdout:2914: finishing write tid 5 to smithi07134777-3 2024-09-11T01:49:00.009 INFO:tasks.rados.rados.0.smithi071.stdout:2914: oid 3 updating version 0 to 2036 2024-09-11T01:49:00.009 INFO:tasks.rados.rados.0.smithi071.stdout:2914: oid 3 version 2036 is already newer than 2033 2024-09-11T01:49:00.009 INFO:tasks.rados.rados.0.smithi071.stdout:2914: oid 3 version 2036 is already newer than 2034 2024-09-11T01:49:00.009 INFO:tasks.rados.rados.0.smithi071.stdout:2914: oid 3 version 2036 is already newer than 2035 2024-09-11T01:49:00.009 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 2036 (ObjNum 1032 snap 331 seq_num 1032) dirty exists 2024-09-11T01:49:00.009 INFO:tasks.rados.rados.0.smithi071.stdout:2914: left oid 3 (ObjNum 1032 snap 331 seq_num 1032) 2024-09-11T01:49:00.009 INFO:tasks.rados.rados.0.smithi071.stdout:2915: expect (ObjNum 978 snap 306 seq_num 978) 2024-09-11T01:49:00.010 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:00.061 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1719636 2024-09-11T01:49:00.068 INFO:tasks.rados.rados.0.smithi071.stdout:2919: expect (ObjNum 841 snap 255 seq_num 841) 2024-09-11T01:49:00.084 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1633932 2024-09-11T01:49:00.103 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:00.826 INFO:tasks.rados.rados.0.smithi071.stdout:2910: done (7 left) 2024-09-11T01:49:00.826 INFO:tasks.rados.rados.0.smithi071.stdout:2912: done (6 left) 2024-09-11T01:49:00.826 INFO:tasks.rados.rados.0.smithi071.stdout:2913: done (5 left) 2024-09-11T01:49:00.826 INFO:tasks.rados.rados.0.smithi071.stdout:2914: done (4 left) 2024-09-11T01:49:00.826 INFO:tasks.rados.rados.0.smithi071.stdout:2915: done (3 left) 2024-09-11T01:49:00.827 INFO:tasks.rados.rados.0.smithi071.stdout:2918: done (2 left) 2024-09-11T01:49:00.827 INFO:tasks.rados.rados.0.smithi071.stdout:2919: done (1 left) 2024-09-11T01:49:00.827 INFO:tasks.rados.rados.0.smithi071.stdout:2921: done (0 left) 2024-09-11T01:49:00.827 INFO:tasks.rados.rados.0.smithi071.stdout:2922: read oid 20 snap 271 2024-09-11T01:49:00.827 INFO:tasks.rados.rados.0.smithi071.stdout:2922: expect (ObjNum 832 snap 254 seq_num 832) 2024-09-11T01:49:00.827 INFO:tasks.rados.rados.0.smithi071.stdout:2923: snap_create 2024-09-11T01:49:00.832 INFO:tasks.rados.rados.0.smithi071.stdout:2922: expect (ObjNum 832 snap 254 seq_num 832) 2024-09-11T01:49:01.828 INFO:tasks.rados.rados.0.smithi071.stdout:2922: done (1 left) 2024-09-11T01:49:01.828 INFO:tasks.rados.rados.0.smithi071.stdout:2923: done (0 left) 2024-09-11T01:49:01.828 INFO:tasks.rados.rados.0.smithi071.stdout:2924: copy_from oid 17 from oid 40 current snap is 333 2024-09-11T01:49:01.829 INFO:tasks.rados.rados.0.smithi071.stdout:2925: write oid 44 current snap is 333 2024-09-11T01:49:01.829 INFO:tasks.rados.rados.0.smithi071.stdout:2925: seq_num 1033 ranges {561631=639872,1882252=633683,3034775=326856} 2024-09-11T01:49:01.834 INFO:tasks.rados.rados.0.smithi071.stdout:2925: writing smithi07134777-44 from 561631 to 1201503 tid 1 2024-09-11T01:49:01.841 INFO:tasks.rados.rados.0.smithi071.stdout:2925: writing smithi07134777-44 from 1882252 to 2515935 tid 2 2024-09-11T01:49:01.846 INFO:tasks.rados.rados.0.smithi071.stdout:2925: writing smithi07134777-44 from 3034775 to 3361631 tid 3 2024-09-11T01:49:01.846 INFO:tasks.rados.rados.0.smithi071.stdout:2926: snap_create 2024-09-11T01:49:01.846 INFO:tasks.rados.rados.0.smithi071.stdout:2924: finishing copy_from to smithi07134777-17 2024-09-11T01:49:01.846 INFO:tasks.rados.rados.0.smithi071.stdout:2924: got expected ENOENT (src dne) 2024-09-11T01:49:01.846 INFO:tasks.rados.rados.0.smithi071.stdout:2924: finishing copy_from racing read to smithi07134777-17 2024-09-11T01:49:01.847 INFO:tasks.rados.rados.0.smithi071.stdout:2925: finishing write tid 1 to smithi07134777-44 2024-09-11T01:49:01.847 INFO:tasks.rados.rados.0.smithi071.stdout:2925: finishing write tid 2 to smithi07134777-44 2024-09-11T01:49:01.852 INFO:tasks.rados.rados.0.smithi071.stdout:2925: finishing write tid 3 to smithi07134777-44 2024-09-11T01:49:01.852 INFO:tasks.rados.rados.0.smithi071.stdout:2925: finishing write tid 4 to smithi07134777-44 2024-09-11T01:49:01.853 INFO:tasks.rados.rados.0.smithi071.stdout:2925: finishing write tid 5 to smithi07134777-44 2024-09-11T01:49:01.853 INFO:tasks.rados.rados.0.smithi071.stdout:2925: oid 44 updating version 0 to 1055 2024-09-11T01:49:01.853 INFO:tasks.rados.rados.0.smithi071.stdout:2925: oid 44 version 1055 is already newer than 1053 2024-09-11T01:49:01.853 INFO:tasks.rados.rados.0.smithi071.stdout:2925: oid 44 updating version 1055 to 1056 2024-09-11T01:49:01.853 INFO:tasks.rados.rados.0.smithi071.stdout:2925: oid 44 version 1056 is already newer than 1054 2024-09-11T01:49:01.853 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 1056 (ObjNum 1033 snap 333 seq_num 1033) dirty exists 2024-09-11T01:49:01.853 INFO:tasks.rados.rados.0.smithi071.stdout:2925: left oid 44 (ObjNum 1033 snap 333 seq_num 1033) 2024-09-11T01:49:02.832 INFO:tasks.rados.rados.0.smithi071.stdout:2924: done (2 left) 2024-09-11T01:49:02.833 INFO:tasks.rados.rados.0.smithi071.stdout:2925: done (1 left) 2024-09-11T01:49:02.833 INFO:tasks.rados.rados.0.smithi071.stdout:2926: done (0 left) 2024-09-11T01:49:02.833 INFO:tasks.rados.rados.0.smithi071.stdout:2927: delete oid 10 current snap is 334 2024-09-11T01:49:02.839 INFO:tasks.rados.rados.0.smithi071.stdout:2927: done (0 left) 2024-09-11T01:49:02.839 INFO:tasks.rados.rados.0.smithi071.stdout:2928: rollback oid 50 current snap is 334 2024-09-11T01:49:02.839 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 50 to 321 2024-09-11T01:49:02.839 INFO:tasks.rados.rados.0.smithi071.stdout:2929: read oid 35 snap -1 2024-09-11T01:49:02.839 INFO:tasks.rados.rados.0.smithi071.stdout:2929: expect (ObjNum 1028 snap 331 seq_num 1028) 2024-09-11T01:49:02.839 INFO:tasks.rados.rados.0.smithi071.stdout:2930: read oid 37 snap -1 2024-09-11T01:49:02.839 INFO:tasks.rados.rados.0.smithi071.stdout:2930: expect deleted 2024-09-11T01:49:02.840 INFO:tasks.rados.rados.0.smithi071.stdout:2931: rollback oid 41 current snap is 334 2024-09-11T01:49:02.840 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 41 to 284 2024-09-11T01:49:02.840 INFO:tasks.rados.rados.0.smithi071.stdout:2932: snap_remove snap 314 2024-09-11T01:49:03.819 INFO:tasks.rados.rados.0.smithi071.stdout:2932: done (4 left) 2024-09-11T01:49:03.819 INFO:tasks.rados.rados.0.smithi071.stdout:2933: read oid 19 snap -1 2024-09-11T01:49:03.819 INFO:tasks.rados.rados.0.smithi071.stdout:2933: expect (ObjNum 1030 snap 331 seq_num 1030) 2024-09-11T01:49:03.819 INFO:tasks.rados.rados.0.smithi071.stdout:2934: write (excl) oid 2 current snap is 334 2024-09-11T01:49:03.819 INFO:tasks.rados.rados.0.smithi071.stdout:2934: seq_num 1035 ranges {595245=506723,1715569=637352,2805974=562298} 2024-09-11T01:49:03.825 INFO:tasks.rados.rados.0.smithi071.stdout:2934: writing smithi07134777-2 from 595245 to 1101968 tid 1 2024-09-11T01:49:03.833 INFO:tasks.rados.rados.0.smithi071.stdout:2934: writing smithi07134777-2 from 1715569 to 2352921 tid 2 2024-09-11T01:49:03.840 INFO:tasks.rados.rados.0.smithi071.stdout:2934: writing smithi07134777-2 from 2805974 to 3368272 tid 3 2024-09-11T01:49:03.840 INFO:tasks.rados.rados.0.smithi071.stdout:2935: read oid 21 snap -1 2024-09-11T01:49:03.840 INFO:tasks.rados.rados.0.smithi071.stdout:2935: expect (ObjNum 937 snap 293 seq_num 937) 2024-09-11T01:49:03.840 INFO:tasks.rados.rados.0.smithi071.stdout:2936: copy_from oid 13 from oid 29 current snap is 334 2024-09-11T01:49:03.840 INFO:tasks.rados.rados.0.smithi071.stdout:2931: finishing rollback tid 0 to smithi07134777-41 2024-09-11T01:49:03.840 INFO:tasks.rados.rados.0.smithi071.stdout:2937: rollback oid 36 current snap is 334 2024-09-11T01:49:03.840 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 36 to 325 2024-09-11T01:49:03.840 INFO:tasks.rados.rados.0.smithi071.stdout:2938: write oid 44 current snap is 334 2024-09-11T01:49:03.840 INFO:tasks.rados.rados.0.smithi071.stdout:2938: seq_num 1036 ranges {612052=581972,1657698=497363,2717044=695008} 2024-09-11T01:49:03.846 INFO:tasks.rados.rados.0.smithi071.stdout:2938: writing smithi07134777-44 from 612052 to 1194024 tid 1 2024-09-11T01:49:03.852 INFO:tasks.rados.rados.0.smithi071.stdout:2938: writing smithi07134777-44 from 1657698 to 2155061 tid 2 2024-09-11T01:49:03.859 INFO:tasks.rados.rados.0.smithi071.stdout:2938: writing smithi07134777-44 from 2717044 to 3412052 tid 3 2024-09-11T01:49:03.859 INFO:tasks.rados.rados.0.smithi071.stdout:2939: write oid 9 current snap is 334 2024-09-11T01:49:03.859 INFO:tasks.rados.rados.0.smithi071.stdout:2939: seq_num 1037 ranges {628859=657221,1716180=641021,3144467=284392} 2024-09-11T01:49:03.866 INFO:tasks.rados.rados.0.smithi071.stdout:2939: writing smithi07134777-9 from 628859 to 1286080 tid 1 2024-09-11T01:49:03.872 INFO:tasks.rados.rados.0.smithi071.stdout:2939: writing smithi07134777-9 from 1716180 to 2357201 tid 2 2024-09-11T01:49:03.878 INFO:tasks.rados.rados.0.smithi071.stdout:2939: writing smithi07134777-9 from 3144467 to 3428859 tid 3 2024-09-11T01:49:03.878 INFO:tasks.rados.rados.0.smithi071.stdout:2940: rollback oid 39 current snap is 334 2024-09-11T01:49:03.878 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 39 to 310 2024-09-11T01:49:03.878 INFO:tasks.rados.rados.0.smithi071.stdout:2931: finishing rollback tid 1 to smithi07134777-41 2024-09-11T01:49:03.879 INFO:tasks.rados.rados.0.smithi071.stdout:2931: finishing rollback tid 2 to smithi07134777-41 2024-09-11T01:49:03.879 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 1080 (ObjNum 832 snap 254 seq_num 832) dirty exists 2024-09-11T01:49:03.879 INFO:tasks.rados.rados.0.smithi071.stdout:2928: finishing rollback tid 1 to smithi07134777-50 2024-09-11T01:49:03.879 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 1861 (ObjNum 65 snap 0 seq_num 3079915) dirty dne 2024-09-11T01:49:03.879 INFO:tasks.rados.rados.0.smithi071.stdout:2929: expect (ObjNum 1028 snap 331 seq_num 1028) 2024-09-11T01:49:04.013 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1743937 2024-09-11T01:49:04.054 INFO:tasks.rados.rados.0.smithi071.stdout:2933: expect (ObjNum 1030 snap 331 seq_num 1030) 2024-09-11T01:49:04.096 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3079915 2024-09-11T01:49:04.180 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2451387 2024-09-11T01:49:04.205 INFO:tasks.rados.rados.0.smithi071.stdout:2934: finishing write tid 1 to smithi07134777-2 2024-09-11T01:49:04.205 INFO:tasks.rados.rados.0.smithi071.stdout:2936: finishing copy_from to smithi07134777-13 2024-09-11T01:49:04.205 INFO:tasks.rados.rados.0.smithi071.stdout:2936: got expected ENOENT (src dne) 2024-09-11T01:49:04.205 INFO:tasks.rados.rados.0.smithi071.stdout:2936: finishing copy_from racing read to smithi07134777-13 2024-09-11T01:49:04.205 INFO:tasks.rados.rados.0.smithi071.stdout:2934: finishing write tid 2 to smithi07134777-2 2024-09-11T01:49:04.205 INFO:tasks.rados.rados.0.smithi071.stdout:2934: finishing write tid 3 to smithi07134777-2 2024-09-11T01:49:04.205 INFO:tasks.rados.rados.0.smithi071.stdout:2937: finishing rollback tid 0 to smithi07134777-36 2024-09-11T01:49:04.205 INFO:tasks.rados.rados.0.smithi071.stdout:2937: finishing rollback tid 1 to smithi07134777-36 2024-09-11T01:49:04.205 INFO:tasks.rados.rados.0.smithi071.stdout:2937: finishing rollback tid 2 to smithi07134777-36 2024-09-11T01:49:04.206 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 1767 (ObjNum 1005 snap 322 seq_num 1005) dirty exists 2024-09-11T01:49:04.206 INFO:tasks.rados.rados.0.smithi071.stdout:2935: expect (ObjNum 937 snap 293 seq_num 937) 2024-09-11T01:49:04.219 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3512436 2024-09-11T01:49:04.230 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 898545 2024-09-11T01:49:04.284 INFO:tasks.rados.rados.0.smithi071.stdout:2934: finishing write tid 4 to smithi07134777-2 2024-09-11T01:49:04.285 INFO:tasks.rados.rados.0.smithi071.stdout:2934: finishing write tid 5 to smithi07134777-2 2024-09-11T01:49:04.285 INFO:tasks.rados.rados.0.smithi071.stdout:2934: oid 2 updating version 0 to 878 2024-09-11T01:49:04.285 INFO:tasks.rados.rados.0.smithi071.stdout:2934: oid 2 version 878 is already newer than 876 2024-09-11T01:49:04.285 INFO:tasks.rados.rados.0.smithi071.stdout:2934: oid 2 version 878 is already newer than 875 2024-09-11T01:49:04.285 INFO:tasks.rados.rados.0.smithi071.stdout:2934: oid 2 version 878 is already newer than 877 2024-09-11T01:49:04.285 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 878 (ObjNum 1035 snap 334 seq_num 1035) dirty exists 2024-09-11T01:49:04.285 INFO:tasks.rados.rados.0.smithi071.stdout:2934: left oid 2 (ObjNum 1035 snap 334 seq_num 1035) 2024-09-11T01:49:04.285 INFO:tasks.rados.rados.0.smithi071.stdout:2938: finishing write tid 1 to smithi07134777-44 2024-09-11T01:49:04.285 INFO:tasks.rados.rados.0.smithi071.stdout:2938: finishing write tid 2 to smithi07134777-44 2024-09-11T01:49:04.285 INFO:tasks.rados.rados.0.smithi071.stdout:2938: finishing write tid 3 to smithi07134777-44 2024-09-11T01:49:04.285 INFO:tasks.rados.rados.0.smithi071.stdout:2938: finishing write tid 4 to smithi07134777-44 2024-09-11T01:49:04.285 INFO:tasks.rados.rados.0.smithi071.stdout:2938: finishing write tid 5 to smithi07134777-44 2024-09-11T01:49:04.285 INFO:tasks.rados.rados.0.smithi071.stdout:2938: oid 44 updating version 0 to 1057 2024-09-11T01:49:04.286 INFO:tasks.rados.rados.0.smithi071.stdout:2938: oid 44 updating version 1057 to 1058 2024-09-11T01:49:04.286 INFO:tasks.rados.rados.0.smithi071.stdout:2938: oid 44 updating version 1058 to 1060 2024-09-11T01:49:04.286 INFO:tasks.rados.rados.0.smithi071.stdout:2938: oid 44 version 1060 is already newer than 1059 2024-09-11T01:49:04.286 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 1060 (ObjNum 1036 snap 334 seq_num 1036) dirty exists 2024-09-11T01:49:04.286 INFO:tasks.rados.rados.0.smithi071.stdout:2938: left oid 44 (ObjNum 1036 snap 334 seq_num 1036) 2024-09-11T01:49:04.286 INFO:tasks.rados.rados.0.smithi071.stdout:2939: finishing write tid 1 to smithi07134777-9 2024-09-11T01:49:04.286 INFO:tasks.rados.rados.0.smithi071.stdout:2939: finishing write tid 2 to smithi07134777-9 2024-09-11T01:49:04.286 INFO:tasks.rados.rados.0.smithi071.stdout:2928: done (11 left) 2024-09-11T01:49:04.286 INFO:tasks.rados.rados.0.smithi071.stdout:2929: done (10 left) 2024-09-11T01:49:04.286 INFO:tasks.rados.rados.0.smithi071.stdout:2930: done (9 left) 2024-09-11T01:49:04.286 INFO:tasks.rados.rados.0.smithi071.stdout:2931: done (8 left) 2024-09-11T01:49:04.286 INFO:tasks.rados.rados.0.smithi071.stdout:2933: done (7 left) 2024-09-11T01:49:04.286 INFO:tasks.rados.rados.0.smithi071.stdout:2934: done (6 left) 2024-09-11T01:49:04.287 INFO:tasks.rados.rados.0.smithi071.stdout:2935: done (5 left) 2024-09-11T01:49:04.287 INFO:tasks.rados.rados.0.smithi071.stdout:2936: done (4 left) 2024-09-11T01:49:04.287 INFO:tasks.rados.rados.0.smithi071.stdout:2937: done (3 left) 2024-09-11T01:49:04.287 INFO:tasks.rados.rados.0.smithi071.stdout:2938: done (2 left) 2024-09-11T01:49:04.287 INFO:tasks.rados.rados.0.smithi071.stdout:2941: copy_from oid 11 from oid 26 current snap is 334 2024-09-11T01:49:04.287 INFO:tasks.rados.rados.0.smithi071.stdout:2939: finishing write tid 3 to smithi07134777-9 2024-09-11T01:49:04.287 INFO:tasks.rados.rados.0.smithi071.stdout:2939: finishing write tid 4 to smithi07134777-9 2024-09-11T01:49:04.287 INFO:tasks.rados.rados.0.smithi071.stdout:2939: finishing write tid 5 to smithi07134777-9 2024-09-11T01:49:04.287 INFO:tasks.rados.rados.0.smithi071.stdout:2939: oid 9 updating version 0 to 1023 2024-09-11T01:49:04.287 INFO:tasks.rados.rados.0.smithi071.stdout:2939: oid 9 version 1023 is already newer than 1021 2024-09-11T01:49:04.287 INFO:tasks.rados.rados.0.smithi071.stdout:2939: oid 9 updating version 1023 to 1024 2024-09-11T01:49:04.287 INFO:tasks.rados.rados.0.smithi071.stdout:2939: oid 9 version 1024 is already newer than 1022 2024-09-11T01:49:04.287 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 1024 (ObjNum 1037 snap 334 seq_num 1037) dirty exists 2024-09-11T01:49:04.287 INFO:tasks.rados.rados.0.smithi071.stdout:2939: left oid 9 (ObjNum 1037 snap 334 seq_num 1037) 2024-09-11T01:49:04.287 INFO:tasks.rados.rados.0.smithi071.stdout:2940: finishing rollback tid 0 to smithi07134777-39 2024-09-11T01:49:04.287 INFO:tasks.rados.rados.0.smithi071.stdout:2940: finishing rollback tid 1 to smithi07134777-39 2024-09-11T01:49:04.288 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 1131 (ObjNum 33 snap 0 seq_num 3922517824) dirty dne 2024-09-11T01:49:04.288 INFO:tasks.rados.rados.0.smithi071.stdout:2939: done (2 left) 2024-09-11T01:49:04.288 INFO:tasks.rados.rados.0.smithi071.stdout:2940: done (1 left) 2024-09-11T01:49:04.288 INFO:tasks.rados.rados.0.smithi071.stdout:2942: read oid 22 snap -1 2024-09-11T01:49:04.288 INFO:tasks.rados.rados.0.smithi071.stdout:2942: expect (ObjNum 745 snap 227 seq_num 745) 2024-09-11T01:49:04.288 INFO:tasks.rados.rados.0.smithi071.stdout:2943: delete oid 23 current snap is 334 2024-09-11T01:49:04.289 INFO:tasks.rados.rados.0.smithi071.stdout:2941: finishing copy_from to smithi07134777-11 2024-09-11T01:49:04.289 INFO:tasks.rados.rados.0.smithi071.stdout:2941: got expected ENOENT (src dne) 2024-09-11T01:49:04.289 INFO:tasks.rados.rados.0.smithi071.stdout:2941: finishing copy_from racing read to smithi07134777-11 2024-09-11T01:49:04.289 INFO:tasks.rados.rados.0.smithi071.stdout:2941: done (2 left) 2024-09-11T01:49:04.289 INFO:tasks.rados.rados.0.smithi071.stdout:2943: done (1 left) 2024-09-11T01:49:04.289 INFO:tasks.rados.rados.0.smithi071.stdout:2944: write (excl) oid 44 current snap is 334 2024-09-11T01:49:04.289 INFO:tasks.rados.rados.0.smithi071.stdout:2944: seq_num 1039 ranges {662473=407719,1833144=644690,3199313=263160} 2024-09-11T01:49:04.291 INFO:tasks.rados.rados.0.smithi071.stdout:2944: writing smithi07134777-44 from 662473 to 1070192 tid 1 2024-09-11T01:49:04.299 INFO:tasks.rados.rados.0.smithi071.stdout:2944: writing smithi07134777-44 from 1833144 to 2477834 tid 2 2024-09-11T01:49:04.304 INFO:tasks.rados.rados.0.smithi071.stdout:2944: writing smithi07134777-44 from 3199313 to 3462473 tid 3 2024-09-11T01:49:04.305 INFO:tasks.rados.rados.0.smithi071.stdout:2945: write (excl) oid 45 current snap is 334 2024-09-11T01:49:04.305 INFO:tasks.rados.rados.0.smithi071.stdout:2945: seq_num 1040 ranges {679280=482968,1775273=788348,3110383=368897} 2024-09-11T01:49:04.311 INFO:tasks.rados.rados.0.smithi071.stdout:2945: writing smithi07134777-45 from 679280 to 1162248 tid 1 2024-09-11T01:49:04.319 INFO:tasks.rados.rados.0.smithi071.stdout:2945: writing smithi07134777-45 from 1775273 to 2563621 tid 2 2024-09-11T01:49:04.324 INFO:tasks.rados.rados.0.smithi071.stdout:2945: writing smithi07134777-45 from 3110383 to 3479280 tid 3 2024-09-11T01:49:04.324 INFO:tasks.rados.rados.0.smithi071.stdout:2946: rollback oid 50 current snap is 334 2024-09-11T01:49:04.325 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 50 to 284 2024-09-11T01:49:04.325 INFO:tasks.rados.rados.0.smithi071.stdout:2942: expect (ObjNum 745 snap 227 seq_num 745) 2024-09-11T01:49:04.453 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:04.517 INFO:tasks.rados.rados.0.smithi071.stdout:2944: finishing write tid 1 to smithi07134777-44 2024-09-11T01:49:04.517 INFO:tasks.rados.rados.0.smithi071.stdout:2944: finishing write tid 2 to smithi07134777-44 2024-09-11T01:49:04.517 INFO:tasks.rados.rados.0.smithi071.stdout:2942: done (3 left) 2024-09-11T01:49:04.517 INFO:tasks.rados.rados.0.smithi071.stdout:2947: snap_create 2024-09-11T01:49:04.518 INFO:tasks.rados.rados.0.smithi071.stdout:2944: finishing write tid 3 to smithi07134777-44 2024-09-11T01:49:04.518 INFO:tasks.rados.rados.0.smithi071.stdout:2944: finishing write tid 4 to smithi07134777-44 2024-09-11T01:49:04.518 INFO:tasks.rados.rados.0.smithi071.stdout:2944: finishing write tid 5 to smithi07134777-44 2024-09-11T01:49:04.518 INFO:tasks.rados.rados.0.smithi071.stdout:2944: oid 44 updating version 0 to 1064 2024-09-11T01:49:04.518 INFO:tasks.rados.rados.0.smithi071.stdout:2944: oid 44 version 1064 is already newer than 1061 2024-09-11T01:49:04.518 INFO:tasks.rados.rados.0.smithi071.stdout:2944: oid 44 version 1064 is already newer than 1062 2024-09-11T01:49:04.518 INFO:tasks.rados.rados.0.smithi071.stdout:2944: oid 44 version 1064 is already newer than 1063 2024-09-11T01:49:04.518 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 1064 (ObjNum 1039 snap 334 seq_num 1039) dirty exists 2024-09-11T01:49:04.518 INFO:tasks.rados.rados.0.smithi071.stdout:2944: left oid 44 (ObjNum 1039 snap 334 seq_num 1039) 2024-09-11T01:49:04.519 INFO:tasks.rados.rados.0.smithi071.stdout:2945: finishing write tid 1 to smithi07134777-45 2024-09-11T01:49:04.519 INFO:tasks.rados.rados.0.smithi071.stdout:2945: finishing write tid 2 to smithi07134777-45 2024-09-11T01:49:04.519 INFO:tasks.rados.rados.0.smithi071.stdout:2946: finishing rollback tid 1 to smithi07134777-50 2024-09-11T01:49:04.519 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 1861 (ObjNum 65 snap 0 seq_num 3923014226) dirty dne 2024-09-11T01:49:04.519 INFO:tasks.rados.rados.0.smithi071.stdout:2945: finishing write tid 3 to smithi07134777-45 2024-09-11T01:49:04.519 INFO:tasks.rados.rados.0.smithi071.stdout:2945: finishing write tid 4 to smithi07134777-45 2024-09-11T01:49:04.519 INFO:tasks.rados.rados.0.smithi071.stdout:2945: finishing write tid 5 to smithi07134777-45 2024-09-11T01:49:04.519 INFO:tasks.rados.rados.0.smithi071.stdout:2945: oid 45 updating version 0 to 2033 2024-09-11T01:49:04.519 INFO:tasks.rados.rados.0.smithi071.stdout:2945: oid 45 updating version 2033 to 2034 2024-09-11T01:49:04.519 INFO:tasks.rados.rados.0.smithi071.stdout:2945: oid 45 version 2034 is already newer than 2032 2024-09-11T01:49:04.519 INFO:tasks.rados.rados.0.smithi071.stdout:2945: oid 45 version 2034 is already newer than 2031 2024-09-11T01:49:04.519 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 2034 (ObjNum 1040 snap 334 seq_num 1040) dirty exists 2024-09-11T01:49:04.519 INFO:tasks.rados.rados.0.smithi071.stdout:2945: left oid 45 (ObjNum 1040 snap 334 seq_num 1040) 2024-09-11T01:49:04.841 INFO:tasks.rados.rados.0.smithi071.stdout:2944: done (3 left) 2024-09-11T01:49:04.842 INFO:tasks.rados.rados.0.smithi071.stdout:2945: done (2 left) 2024-09-11T01:49:04.842 INFO:tasks.rados.rados.0.smithi071.stdout:2946: done (1 left) 2024-09-11T01:49:04.842 INFO:tasks.rados.rados.0.smithi071.stdout:2947: done (0 left) 2024-09-11T01:49:04.842 INFO:tasks.rados.rados.0.smithi071.stdout:2948: copy_from oid 9 from oid 6 current snap is 335 2024-09-11T01:49:04.842 INFO:tasks.rados.rados.0.smithi071.stdout:2949: read oid 23 snap -1 2024-09-11T01:49:04.842 INFO:tasks.rados.rados.0.smithi071.stdout:2949: expect deleted 2024-09-11T01:49:04.842 INFO:tasks.rados.rados.0.smithi071.stdout:2950: write oid 34 current snap is 335 2024-09-11T01:49:04.842 INFO:tasks.rados.rados.0.smithi071.stdout:2950: seq_num 1041 ranges {696087=558217,1833755=648359,3137806=358281} 2024-09-11T01:49:04.848 INFO:tasks.rados.rados.0.smithi071.stdout:2950: writing smithi07134777-34 from 696087 to 1254304 tid 1 2024-09-11T01:49:04.854 INFO:tasks.rados.rados.0.smithi071.stdout:2950: writing smithi07134777-34 from 1833755 to 2482114 tid 2 2024-09-11T01:49:04.860 INFO:tasks.rados.rados.0.smithi071.stdout:2950: writing smithi07134777-34 from 3137806 to 3496087 tid 3 2024-09-11T01:49:04.861 INFO:tasks.rados.rados.0.smithi071.stdout:2951: rollback oid 17 current snap is 335 2024-09-11T01:49:04.861 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 17 to 328 2024-09-11T01:49:04.861 INFO:tasks.rados.rados.0.smithi071.stdout:2948: finishing copy_from racing read to smithi07134777-9 2024-09-11T01:49:04.861 INFO:tasks.rados.rados.0.smithi071.stdout:2948: finishing copy_from to smithi07134777-9 2024-09-11T01:49:04.861 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 475 (ObjNum 601 snap 172 seq_num 601) dirty exists 2024-09-11T01:49:04.861 INFO:tasks.rados.rados.0.smithi071.stdout:2950: finishing write tid 1 to smithi07134777-34 2024-09-11T01:49:04.861 INFO:tasks.rados.rados.0.smithi071.stdout:2950: finishing write tid 2 to smithi07134777-34 2024-09-11T01:49:04.861 INFO:tasks.rados.rados.0.smithi071.stdout:2948: done (3 left) 2024-09-11T01:49:04.861 INFO:tasks.rados.rados.0.smithi071.stdout:2949: done (2 left) 2024-09-11T01:49:04.861 INFO:tasks.rados.rados.0.smithi071.stdout:2952: rollback oid 33 current snap is 335 2024-09-11T01:49:04.861 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 33 to 331 2024-09-11T01:49:04.861 INFO:tasks.rados.rados.0.smithi071.stdout:2953: read oid 43 snap 294 2024-09-11T01:49:04.861 INFO:tasks.rados.rados.0.smithi071.stdout:2953: expect deleted 2024-09-11T01:49:04.862 INFO:tasks.rados.rados.0.smithi071.stdout:2954: snap_create 2024-09-11T01:49:04.867 INFO:tasks.rados.rados.0.smithi071.stdout:2951: finishing rollback tid 0 to smithi07134777-17 2024-09-11T01:49:04.867 INFO:tasks.rados.rados.0.smithi071.stdout:2951: finishing rollback tid 1 to smithi07134777-17 2024-09-11T01:49:04.867 INFO:tasks.rados.rados.0.smithi071.stdout:2951: finishing rollback tid 2 to smithi07134777-17 2024-09-11T01:49:04.867 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 1781 (ObjNum 931 snap 293 seq_num 931) dirty exists 2024-09-11T01:49:04.868 INFO:tasks.rados.rados.0.smithi071.stdout:2950: finishing write tid 3 to smithi07134777-34 2024-09-11T01:49:04.868 INFO:tasks.rados.rados.0.smithi071.stdout:2950: finishing write tid 4 to smithi07134777-34 2024-09-11T01:49:04.869 INFO:tasks.rados.rados.0.smithi071.stdout:2950: finishing write tid 5 to smithi07134777-34 2024-09-11T01:49:04.869 INFO:tasks.rados.rados.0.smithi071.stdout:2950: oid 34 updating version 0 to 1927 2024-09-11T01:49:04.869 INFO:tasks.rados.rados.0.smithi071.stdout:2950: oid 34 version 1927 is already newer than 1926 2024-09-11T01:49:04.869 INFO:tasks.rados.rados.0.smithi071.stdout:2950: oid 34 updating version 1927 to 1928 2024-09-11T01:49:04.869 INFO:tasks.rados.rados.0.smithi071.stdout:2950: oid 34 updating version 1928 to 1929 2024-09-11T01:49:04.869 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 1929 (ObjNum 1041 snap 335 seq_num 1041) dirty exists 2024-09-11T01:49:04.870 INFO:tasks.rados.rados.0.smithi071.stdout:2950: left oid 34 (ObjNum 1041 snap 335 seq_num 1041) 2024-09-11T01:49:04.870 INFO:tasks.rados.rados.0.smithi071.stdout:2952: finishing rollback tid 0 to smithi07134777-33 2024-09-11T01:49:04.870 INFO:tasks.rados.rados.0.smithi071.stdout:2952: finishing rollback tid 1 to smithi07134777-33 2024-09-11T01:49:04.870 INFO:tasks.rados.rados.0.smithi071.stdout:2952: finishing rollback tid 2 to smithi07134777-33 2024-09-11T01:49:04.870 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 1932 (ObjNum 979 snap 306 seq_num 979) dirty exists 2024-09-11T01:49:05.858 INFO:tasks.rados.rados.0.smithi071.stdout:2950: done (4 left) 2024-09-11T01:49:05.858 INFO:tasks.rados.rados.0.smithi071.stdout:2951: done (3 left) 2024-09-11T01:49:05.858 INFO:tasks.rados.rados.0.smithi071.stdout:2952: done (2 left) 2024-09-11T01:49:05.858 INFO:tasks.rados.rados.0.smithi071.stdout:2953: done (1 left) 2024-09-11T01:49:05.858 INFO:tasks.rados.rados.0.smithi071.stdout:2954: done (0 left) 2024-09-11T01:49:05.858 INFO:tasks.rados.rados.0.smithi071.stdout:2955: read oid 14 snap -1 2024-09-11T01:49:05.858 INFO:tasks.rados.rados.0.smithi071.stdout:2955: expect (ObjNum 1019 snap 323 seq_num 1019) 2024-09-11T01:49:05.858 INFO:tasks.rados.rados.0.smithi071.stdout:2956: write oid 13 current snap is 336 2024-09-11T01:49:05.858 INFO:tasks.rados.rados.0.smithi071.stdout:2956: seq_num 1042 ranges {712894=749819,2008590=792017,3281582=231312} 2024-09-11T01:49:05.865 INFO:tasks.rados.rados.0.smithi071.stdout:2956: writing smithi07134777-13 from 712894 to 1462713 tid 1 2024-09-11T01:49:05.873 INFO:tasks.rados.rados.0.smithi071.stdout:2956: writing smithi07134777-13 from 2008590 to 2800607 tid 2 2024-09-11T01:49:05.878 INFO:tasks.rados.rados.0.smithi071.stdout:2956: writing smithi07134777-13 from 3281582 to 3512894 tid 3 2024-09-11T01:49:05.878 INFO:tasks.rados.rados.0.smithi071.stdout:2957: snap_remove snap 333 2024-09-11T01:49:06.851 INFO:tasks.rados.rados.0.smithi071.stdout:2957: done (2 left) 2024-09-11T01:49:06.851 INFO:tasks.rados.rados.0.smithi071.stdout:2958: rollback oid 50 current snap is 336 2024-09-11T01:49:06.851 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 50 to 307 2024-09-11T01:49:06.852 INFO:tasks.rados.rados.0.smithi071.stdout:2955: expect (ObjNum 1019 snap 323 seq_num 1019) 2024-09-11T01:49:06.939 INFO:tasks.rados.rados.0.smithi071.stdout:2956: finishing write tid 1 to smithi07134777-13 2024-09-11T01:49:06.939 INFO:tasks.rados.rados.0.smithi071.stdout:2956: finishing write tid 2 to smithi07134777-13 2024-09-11T01:49:06.939 INFO:tasks.rados.rados.0.smithi071.stdout:2956: finishing write tid 3 to smithi07134777-13 2024-09-11T01:49:06.939 INFO:tasks.rados.rados.0.smithi071.stdout:2956: finishing write tid 4 to smithi07134777-13 2024-09-11T01:49:06.940 INFO:tasks.rados.rados.0.smithi071.stdout:2956: finishing write tid 5 to smithi07134777-13 2024-09-11T01:49:06.940 INFO:tasks.rados.rados.0.smithi071.stdout:2956: oid 13 updating version 0 to 1783 2024-09-11T01:49:06.940 INFO:tasks.rados.rados.0.smithi071.stdout:2956: oid 13 updating version 1783 to 1784 2024-09-11T01:49:06.940 INFO:tasks.rados.rados.0.smithi071.stdout:2956: oid 13 version 1784 is already newer than 1782 2024-09-11T01:49:06.940 INFO:tasks.rados.rados.0.smithi071.stdout:2956: oid 13 updating version 1784 to 1785 2024-09-11T01:49:06.940 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 1785 (ObjNum 1042 snap 336 seq_num 1042) dirty exists 2024-09-11T01:49:06.940 INFO:tasks.rados.rados.0.smithi071.stdout:2956: left oid 13 (ObjNum 1042 snap 336 seq_num 1042) 2024-09-11T01:49:06.940 INFO:tasks.rados.rados.0.smithi071.stdout:2955: done (2 left) 2024-09-11T01:49:06.940 INFO:tasks.rados.rados.0.smithi071.stdout:2956: done (1 left) 2024-09-11T01:49:06.940 INFO:tasks.rados.rados.0.smithi071.stdout:2959: delete oid 2 current snap is 336 2024-09-11T01:49:06.940 INFO:tasks.rados.rados.0.smithi071.stdout:2958: finishing rollback tid 1 to smithi07134777-50 2024-09-11T01:49:06.940 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 1861 (ObjNum 97 snap 0 seq_num 2371409728) dirty dne 2024-09-11T01:49:06.942 INFO:tasks.rados.rados.0.smithi071.stdout:2958: done (1 left) 2024-09-11T01:49:06.942 INFO:tasks.rados.rados.0.smithi071.stdout:2959: done (0 left) 2024-09-11T01:49:06.942 INFO:tasks.rados.rados.0.smithi071.stdout:2960: write (excl) oid 41 current snap is 336 2024-09-11T01:49:06.942 INFO:tasks.rados.rados.0.smithi071.stdout:2960: seq_num 1044 ranges {746508=500317,2009201=795686,3220075=326433} 2024-09-11T01:49:06.948 INFO:tasks.rados.rados.0.smithi071.stdout:2960: writing smithi07134777-41 from 746508 to 1246825 tid 1 2024-09-11T01:49:06.957 INFO:tasks.rados.rados.0.smithi071.stdout:2960: writing smithi07134777-41 from 2009201 to 2804887 tid 2 2024-09-11T01:49:06.961 INFO:tasks.rados.rados.0.smithi071.stdout:2960: writing smithi07134777-41 from 3220075 to 3546508 tid 3 2024-09-11T01:49:06.961 INFO:tasks.rados.rados.0.smithi071.stdout:2961: snap_remove snap 261 2024-09-11T01:49:07.132 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-11T01:49:07.132 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-11T01:49:07.132 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-11T01:49:07.135 INFO:tasks.ceph.mon.c:Started 2024-09-11T01:49:07.135 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:49:07.135 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:49:07.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.246+0000 7f225f3a9640 10 monclient: get_monmap_and_config 2024-09-11T01:49:07.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.246+0000 7f225f3a9640 10 monclient: build_initial_monmap 2024-09-11T01:49:07.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.246+0000 7f225f3a9640 10 monclient: monmap: 2024-09-11T01:49:07.248 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:07.248 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:07.248 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:49:07.247307+0000 2024-09-11T01:49:07.248 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:49:07.247307+0000 2024-09-11T01:49:07.248 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:07.248 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:07.249 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:49:07.249 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:49:07.249 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:49:07.249 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:49:07.249 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:49:07.249 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:07.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.247+0000 7f225f3a9640 1 Processor -- start 2024-09-11T01:49:07.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.247+0000 7f225f3a9640 1 -- start start 2024-09-11T01:49:07.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.247+0000 7f225f3a9640 10 monclient: init 2024-09-11T01:49:07.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.247+0000 7f225f3a9640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:07.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.247+0000 7f225f3a9640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-11T01:49:07.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.247+0000 7f225f3a9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2258138210 0x7f22581385f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:07.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.247+0000 7f225f3a9640 10 monclient(hunting): picked mon.noname-b con 0x7f2258138210 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:07.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.247+0000 7f225f3a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f22580a6810 addr v1:172.21.15.12:6789/0 2024-09-11T01:49:07.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.247+0000 7f225f3a9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f22580a57a0 0x7f2258140120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:07.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.247+0000 7f225f3a9640 10 monclient(hunting): picked mon.noname-a con 0x7f22580a57a0 addr v2:172.21.15.12:3300/0 2024-09-11T01:49:07.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.247+0000 7f225f3a9640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f225809ba20 con 0x7f22580a6810 2024-09-11T01:49:07.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.247+0000 7f225f3a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:07.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.247+0000 7f225f3a9640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f225809c7a0 con 0x7f22580a57a0 2024-09-11T01:49:07.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.247+0000 7f225f3a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:07.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.247+0000 7f225f3a9640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f225809c4c0 con 0x7f2258138210 2024-09-11T01:49:07.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.247+0000 7f225f3a9640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:07.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.247+0000 7f225f3a9640 10 monclient(hunting): authenticate will time out at 1953.689937s 2024-09-11T01:49:07.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225d145640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2258138210 0x7f22581385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225c944640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f22580a6810 0x7f22580a5090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42200/0 (socket says 172.21.15.12:42200) 2024-09-11T01:49:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225c944640 1 -- 172.21.15.12:0/897730185 learned_addr learned my addr 172.21.15.12:0/897730185 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:49:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225d145640 10 monclient(hunting): get_auth_request con 0x7f2258138210 auth_method 0 2024-09-11T01:49:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225d145640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225d145640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225d145640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225d946640 1 --2- 172.21.15.12:0/897730185 >> v2:172.21.15.12:3300/0 conn(0x7f22580a57a0 0x7f2258140120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225d946640 10 monclient(hunting): get_auth_request con 0x7f22580a57a0 auth_method 0 2024-09-11T01:49:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225d946640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225d946640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225d946640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225d145640 1 -- 172.21.15.12:0/897730185 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2258138210 msgr2=0x7f22581385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:49:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225d145640 1 -- 172.21.15.12:0/897730185 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2258138210 msgr2=0x7f22581385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:49:07.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225d145640 1 --2- 172.21.15.12:0/897730185 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2258138210 0x7f22581385f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:49:07.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225d946640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:07.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225d946640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:07.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225d145640 1 --2- 172.21.15.12:0/897730185 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2258138210 0x7f22581385f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:07.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225d946640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:07.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225d946640 10 monclient(hunting): handle_auth_done global_id 214179 payload 274 2024-09-11T01:49:07.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225d946640 1 -- 172.21.15.12:0/897730185 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2258138210 msgr2=0x7f22581385f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:49:07.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225d946640 1 --2- 172.21.15.12:0/897730185 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2258138210 0x7f22581385f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:07.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225d946640 1 -- 172.21.15.12:0/897730185 >> v1:172.21.15.12:6789/0 conn(0x7f22580a6810 legacy=0x7f22580a5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:49:07.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225d946640 10 monclient: _finish_hunting 0 2024-09-11T01:49:07.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225d946640 1 monclient: found mon.noname-a 2024-09-11T01:49:07.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225d946640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:07.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225d946640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:49:07.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.248+0000 7f225d946640 1 -- 172.21.15.12:0/897730185 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22580a6590 con 0x7f22580a57a0 2024-09-11T01:49:07.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.251+0000 7f225d946640 1 -- 172.21.15.12:0/897730185 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22580a57a0 msgr2=0x7f2258140120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.251+0000 7f225d946640 1 --2- 172.21.15.12:0/897730185 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22580a57a0 0x7f2258140120 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2250009d90 tx=0x7f225002a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d336fe3db2025f6b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.251+0000 7f224ffff640 1 -- 172.21.15.12:0/897730185 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2250007950 con 0x7f22580a57a0 2024-09-11T01:49:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.251+0000 7f224ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.251+0000 7f224ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:49:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.251+0000 7f224ffff640 10 monclient: dump: 2024-09-11T01:49:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:07.260 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:07.261 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:07.261 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:07.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.251+0000 7f224ffff640 10 monclient: _finish_auth 0 2024-09-11T01:49:07.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.251+0000 7f224ffff640 10 monclient: _check_auth_tickets 2024-09-11T01:49:07.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.251+0000 7f224ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:07.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.251+0000 7f225f3a9640 5 monclient: authenticate success, global_id 214179 2024-09-11T01:49:07.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.251+0000 7f225f3a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:49:07.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.251+0000 7f224ffff640 1 -- 172.21.15.12:0/897730185 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2250007af0 con 0x7f22580a57a0 2024-09-11T01:49:07.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.251+0000 7f224ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:07.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.251+0000 7f224ffff640 1 -- 172.21.15.12:0/897730185 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2250031400 con 0x7f22580a57a0 2024-09-11T01:49:07.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.251+0000 7f225f3a9640 10 monclient: get_monmap_and_config success 2024-09-11T01:49:07.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.251+0000 7f225f3a9640 4 set_mon_vals no callback set 2024-09-11T01:49:07.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.251+0000 7f224ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:07.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.251+0000 7f224ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:07.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.251+0000 7f224ffff640 10 monclient: dump: 2024-09-11T01:49:07.262 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:07.262 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:07.262 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:07.262 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:07.262 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:07.262 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:07.262 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:07.262 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:07.262 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:07.262 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:07.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.251+0000 7f225f3a9640 10 monclient: shutdown 2024-09-11T01:49:07.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.251+0000 7f225f3a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:07.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.251+0000 7f225f3a9640 1 -- 172.21.15.12:0/897730185 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22580a57a0 msgr2=0x7f2258140120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:07.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.251+0000 7f225f3a9640 1 --2- 172.21.15.12:0/897730185 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22580a57a0 0x7f2258140120 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2250009d90 tx=0x7f225002a490 comp rx=0 tx=0).stop 2024-09-11T01:49:07.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.252+0000 7f225f3a9640 1 -- 172.21.15.12:0/897730185 shutdown_connections 2024-09-11T01:49:07.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.252+0000 7f225f3a9640 1 --2- 172.21.15.12:0/897730185 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f22580a57a0 0x7f2258140120 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:07.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.252+0000 7f225f3a9640 1 --2- 172.21.15.12:0/897730185 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2258138210 0x7f22581385f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:07.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.252+0000 7f225f3a9640 1 -- 172.21.15.12:0/897730185 >> 172.21.15.12:0/897730185 conn(0x7f22580aaa80 msgr2=0x7f22580aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:07.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.252+0000 7f225f3a9640 1 -- 172.21.15.12:0/897730185 shutdown_connections 2024-09-11T01:49:07.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.252+0000 7f225f3a9640 1 -- 172.21.15.12:0/897730185 wait complete. 2024-09-11T01:49:07.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.252+0000 7f225f3a9640 10 monclient: build_initial_monmap 2024-09-11T01:49:07.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.252+0000 7f225f3a9640 10 monclient: monmap: 2024-09-11T01:49:07.263 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:07.263 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:49:07.263 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:49:07.263 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:49:07.264 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:07.264 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:07.264 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:49:07.264 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:49:07.264 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:49:07.264 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:07.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225f3a9640 1 Processor -- start 2024-09-11T01:49:07.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225f3a9640 1 -- start start 2024-09-11T01:49:07.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225f3a9640 10 monclient: init 2024-09-11T01:49:07.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225f3a9640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:07.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225f3a9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:49:07.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225f3a9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f22580a57a0 0x7f2258136df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:07.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225f3a9640 10 monclient(hunting): picked mon.noname-c con 0x7f22580a57a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:07.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225f3a9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22580a6810 0x7f2258137330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:07.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225f3a9640 10 monclient(hunting): picked mon.noname-b con 0x7f22580a6810 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:49:07.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225f3a9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2258138210 0x7f2258120ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:07.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225f3a9640 10 monclient(hunting): picked mon.noname-a con 0x7f2258138210 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:49:07.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225f3a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:07.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225f3a9640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f225806f670 con 0x7f2258138210 2024-09-11T01:49:07.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225f3a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:07.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225f3a9640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f225806f4f0 con 0x7f22580a6810 2024-09-11T01:49:07.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225f3a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:07.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225f3a9640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2258138b30 con 0x7f22580a57a0 2024-09-11T01:49:07.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225f3a9640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:07.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225f3a9640 10 monclient(hunting): authenticate will time out at 1953.695666s 2024-09-11T01:49:07.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225d145640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f22580a57a0 0x7f2258136df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:07.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225d946640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2258138210 0x7f2258120ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:07.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225d145640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f22580a57a0 0x7f2258136df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52540/0 (socket says 172.21.15.12:52540) 2024-09-11T01:49:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225d145640 1 -- 172.21.15.12:0/2367938467 learned_addr learned my addr 172.21.15.12:0/2367938467 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:49:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225d145640 10 monclient(hunting): get_auth_request con 0x7f22580a57a0 auth_method 0 2024-09-11T01:49:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225d145640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225d145640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225d145640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225d946640 10 monclient(hunting): get_auth_request con 0x7f2258138210 auth_method 0 2024-09-11T01:49:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225d946640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225d946640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225d946640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225d946640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225d946640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225d145640 1 -- 172.21.15.12:0/2367938467 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f22580a57a0 msgr2=0x7f2258136df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:49:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225d145640 1 -- 172.21.15.12:0/2367938467 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f22580a57a0 msgr2=0x7f2258136df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:49:07.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225d145640 1 --2- 172.21.15.12:0/2367938467 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f22580a57a0 0x7f2258136df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:49:07.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225d946640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:07.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225d145640 1 --2- 172.21.15.12:0/2367938467 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f22580a57a0 0x7f2258136df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:07.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225c944640 1 --2- 172.21.15.12:0/2367938467 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22580a6810 0x7f2258137330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:07.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225c944640 10 monclient(hunting): get_auth_request con 0x7f22580a6810 auth_method 0 2024-09-11T01:49:07.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225c944640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:07.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225c944640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:07.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.253+0000 7f225c944640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:07.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.254+0000 7f225d946640 10 monclient(hunting): handle_auth_done global_id 214182 payload 824 2024-09-11T01:49:07.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.254+0000 7f225d946640 1 -- 172.21.15.12:0/2367938467 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f22580a57a0 msgr2=0x7f2258136df0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:49:07.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.254+0000 7f225d946640 1 --2- 172.21.15.12:0/2367938467 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f22580a57a0 0x7f2258136df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:07.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.254+0000 7f225d946640 1 -- 172.21.15.12:0/2367938467 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22580a6810 msgr2=0x7f2258137330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:07.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.254+0000 7f225d946640 1 --2- 172.21.15.12:0/2367938467 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22580a6810 0x7f2258137330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:07.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.254+0000 7f225d946640 10 monclient: _finish_hunting 0 2024-09-11T01:49:07.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.254+0000 7f225d946640 1 monclient: found mon.noname-a 2024-09-11T01:49:07.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.254+0000 7f225d946640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:07.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.254+0000 7f225d946640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:49:07.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.254+0000 7f225d946640 1 -- 172.21.15.12:0/2367938467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22580a1880 con 0x7f2258138210 2024-09-11T01:49:07.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.254+0000 7f225c944640 1 --2- 172.21.15.12:0/2367938467 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22580a6810 0x7f2258137330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:49:07.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.256+0000 7f225d946640 1 -- 172.21.15.12:0/2367938467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2258138210 msgr2=0x7f2258120ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:07.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.256+0000 7f225d946640 1 --2- 172.21.15.12:0/2367938467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2258138210 0x7f2258120ce0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2250031000 tx=0x7f2250004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:07.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.256+0000 7f224dffb640 1 -- 172.21.15.12:0/2367938467 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2250040070 con 0x7f2258138210 2024-09-11T01:49:07.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.256+0000 7f224dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:07.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.256+0000 7f224dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:49:07.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.256+0000 7f224dffb640 10 monclient: dump: 2024-09-11T01:49:07.268 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:07.268 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:07.269 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:07.269 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:07.269 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:07.269 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:07.269 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:07.269 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:07.269 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:07.269 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:07.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.256+0000 7f224dffb640 10 monclient: _finish_auth 0 2024-09-11T01:49:07.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.256+0000 7f224dffb640 10 monclient: _check_auth_tickets 2024-09-11T01:49:07.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.256+0000 7f224dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:07.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.256+0000 7f224dffb640 1 -- 172.21.15.12:0/2367938467 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2250045660 con 0x7f2258138210 2024-09-11T01:49:07.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.256+0000 7f224dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:07.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.256+0000 7f224dffb640 1 -- 172.21.15.12:0/2367938467 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22500314c0 con 0x7f2258138210 2024-09-11T01:49:07.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.256+0000 7f224dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:07.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.256+0000 7f224dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:07.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.256+0000 7f224dffb640 10 monclient: dump: 2024-09-11T01:49:07.270 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:07.270 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:07.270 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:07.270 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:07.270 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:07.270 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:07.270 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:07.270 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:07.270 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:07.270 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:07.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.256+0000 7f225f3a9640 5 monclient: authenticate success, global_id 214182 2024-09-11T01:49:07.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.256+0000 7f225f3a9640 10 monclient: _renew_subs 2024-09-11T01:49:07.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.256+0000 7f225f3a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:07.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.256+0000 7f225f3a9640 1 -- 172.21.15.12:0/2367938467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2258122450 con 0x7f2258138210 2024-09-11T01:49:07.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.256+0000 7f224f7fe640 4 set_mon_vals no callback set 2024-09-11T01:49:07.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.256+0000 7f225f3a9640 10 monclient: _renew_subs 2024-09-11T01:49:07.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.256+0000 7f225f3a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:07.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.256+0000 7f225f3a9640 1 -- 172.21.15.12:0/2367938467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2258122fb0 con 0x7f2258138210 2024-09-11T01:49:07.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.257+0000 7f224dffb640 1 -- 172.21.15.12:0/2367938467 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f2250007960 con 0x7f2258138210 2024-09-11T01:49:07.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.257+0000 7f225f3a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:07.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.257+0000 7f225f3a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:07.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.257+0000 7f225f3a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:07.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.257+0000 7f225f3a9640 1 -- 172.21.15.12:0/2367938467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22580a5bd0 con 0x7f2258138210 2024-09-11T01:49:07.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.258+0000 7f224dffb640 1 --2- 172.21.15.12:0/2367938467 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f223c03e210 0x7f223c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:07.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.258+0000 7f224dffb640 1 -- 172.21.15.12:0/2367938467 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(664..664 src has 507..664) ==== 4831+0+0 (secure 0 0 0) 0x7f22500749d0 con 0x7f2258138210 2024-09-11T01:49:07.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.258+0000 7f225d145640 1 --2- 172.21.15.12:0/2367938467 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f223c03e210 0x7f223c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:07.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.258+0000 7f225d145640 10 monclient: get_auth_request con 0x7f223c03e210 auth_method 0 2024-09-11T01:49:07.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.261+0000 7f225d145640 1 --2- 172.21.15.12:0/2367938467 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f223c03e210 0x7f223c0406d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2258122f40 tx=0x7f2254008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:07.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.262+0000 7f224dffb640 1 -- 172.21.15.12:0/2367938467 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f225809ba20 con 0x7f2258138210 2024-09-11T01:49:07.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.262+0000 7f224dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:07.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.262+0000 7f224dffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:49:07.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.517+0000 7f225f3a9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:49:07.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.517+0000 7f225f3a9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:49:07.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.517+0000 7f225f3a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:07.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.517+0000 7f225f3a9640 1 -- 172.21.15.12:0/2367938467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2258124130 con 0x7f2258138210 2024-09-11T01:49:07.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.517+0000 7f224dffb640 1 -- 172.21.15.12:0/2367938467 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f225003b3b0 con 0x7f2258138210 2024-09-11T01:49:07.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.517+0000 7f224dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:49:07.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.517+0000 7f224dffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:49:07.519 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":186,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:49:07.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.519+0000 7f225f3a9640 1 -- 172.21.15.12:0/2367938467 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f223c03e210 msgr2=0x7f223c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:07.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.519+0000 7f225f3a9640 1 --2- 172.21.15.12:0/2367938467 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f223c03e210 0x7f223c0406d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2258122f40 tx=0x7f2254008040 comp rx=0 tx=0).stop 2024-09-11T01:49:07.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.519+0000 7f225f3a9640 10 monclient: shutdown 2024-09-11T01:49:07.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.519+0000 7f225f3a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:07.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.519+0000 7f225f3a9640 1 -- 172.21.15.12:0/2367938467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2258138210 msgr2=0x7f2258120ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:07.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.519+0000 7f225f3a9640 1 --2- 172.21.15.12:0/2367938467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2258138210 0x7f2258120ce0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2250031000 tx=0x7f2250004a40 comp rx=0 tx=0).stop 2024-09-11T01:49:07.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.519+0000 7f225f3a9640 1 -- 172.21.15.12:0/2367938467 shutdown_connections 2024-09-11T01:49:07.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.519+0000 7f225f3a9640 1 --2- 172.21.15.12:0/2367938467 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f223c03e210 0x7f223c0406d0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:07.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.519+0000 7f225f3a9640 1 --2- 172.21.15.12:0/2367938467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2258138210 0x7f2258120ce0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:07.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.519+0000 7f225f3a9640 1 --2- 172.21.15.12:0/2367938467 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22580a6810 0x7f2258137330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:07.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.519+0000 7f225f3a9640 1 --2- 172.21.15.12:0/2367938467 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f22580a57a0 0x7f2258136df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:07.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.519+0000 7f225f3a9640 1 -- 172.21.15.12:0/2367938467 >> 172.21.15.12:0/2367938467 conn(0x7f22580aaa80 msgr2=0x7f22580a0200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:07.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.519+0000 7f225f3a9640 1 -- 172.21.15.12:0/2367938467 shutdown_connections 2024-09-11T01:49:07.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:07.519+0000 7f225f3a9640 1 -- 172.21.15.12:0/2367938467 wait complete. 2024-09-11T01:49:07.864 INFO:tasks.rados.rados.0.smithi071.stdout:2961: done (1 left) 2024-09-11T01:49:07.864 INFO:tasks.rados.rados.0.smithi071.stdout:2962: snap_create 2024-09-11T01:49:07.864 INFO:tasks.rados.rados.0.smithi071.stdout:2960: finishing write tid 1 to smithi07134777-41 2024-09-11T01:49:07.864 INFO:tasks.rados.rados.0.smithi071.stdout:2960: finishing write tid 2 to smithi07134777-41 2024-09-11T01:49:07.864 INFO:tasks.rados.rados.0.smithi071.stdout:2960: finishing write tid 3 to smithi07134777-41 2024-09-11T01:49:07.864 INFO:tasks.rados.rados.0.smithi071.stdout:2960: finishing write tid 4 to smithi07134777-41 2024-09-11T01:49:07.865 INFO:tasks.rados.rados.0.smithi071.stdout:2960: finishing write tid 5 to smithi07134777-41 2024-09-11T01:49:07.865 INFO:tasks.rados.rados.0.smithi071.stdout:2960: oid 41 updating version 0 to 1084 2024-09-11T01:49:07.865 INFO:tasks.rados.rados.0.smithi071.stdout:2960: oid 41 version 1084 is already newer than 1083 2024-09-11T01:49:07.865 INFO:tasks.rados.rados.0.smithi071.stdout:2960: oid 41 version 1084 is already newer than 1081 2024-09-11T01:49:07.865 INFO:tasks.rados.rados.0.smithi071.stdout:2960: oid 41 version 1084 is already newer than 1082 2024-09-11T01:49:07.865 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 1084 (ObjNum 1044 snap 336 seq_num 1044) dirty exists 2024-09-11T01:49:07.865 INFO:tasks.rados.rados.0.smithi071.stdout:2960: left oid 41 (ObjNum 1044 snap 336 seq_num 1044) 2024-09-11T01:49:08.819 INFO:tasks.rados.rados.0.smithi071.stdout:2960: done (1 left) 2024-09-11T01:49:08.819 INFO:tasks.rados.rados.0.smithi071.stdout:2962: done (0 left) 2024-09-11T01:49:08.819 INFO:tasks.rados.rados.0.smithi071.stdout:2963: snap_remove snap 318 2024-09-11T01:49:09.826 INFO:tasks.rados.rados.0.smithi071.stdout:2963: done (0 left) 2024-09-11T01:49:09.827 INFO:tasks.rados.rados.0.smithi071.stdout:2964: read oid 23 snap -1 2024-09-11T01:49:09.827 INFO:tasks.rados.rados.0.smithi071.stdout:2964: expect deleted 2024-09-11T01:49:09.827 INFO:tasks.rados.rados.0.smithi071.stdout:2965: write oid 5 current snap is 337 2024-09-11T01:49:09.827 INFO:tasks.rados.rados.0.smithi071.stdout:2965: seq_num 1045 ranges {763315=575566,2067683=655697,3247498=315817} 2024-09-11T01:49:09.834 INFO:tasks.rados.rados.0.smithi071.stdout:2965: writing smithi07134777-5 from 763315 to 1338881 tid 1 2024-09-11T01:49:09.842 INFO:tasks.rados.rados.0.smithi071.stdout:2965: writing smithi07134777-5 from 2067683 to 2723380 tid 2 2024-09-11T01:49:09.848 INFO:tasks.rados.rados.0.smithi071.stdout:2965: writing smithi07134777-5 from 3247498 to 3563315 tid 3 2024-09-11T01:49:09.848 INFO:tasks.rados.rados.0.smithi071.stdout:2966: write oid 1 current snap is 337 2024-09-11T01:49:09.848 INFO:tasks.rados.rados.0.smithi071.stdout:2966: seq_num 1046 ranges {780122=650815,2126165=799355} 2024-09-11T01:49:09.855 INFO:tasks.rados.rados.0.smithi071.stdout:2966: writing smithi07134777-1 from 780122 to 1430937 tid 1 2024-09-11T01:49:09.864 INFO:tasks.rados.rados.0.smithi071.stdout:2966: writing smithi07134777-1 from 2126165 to 2925520 tid 2 2024-09-11T01:49:09.864 INFO:tasks.rados.rados.0.smithi071.stdout:2967: snap_create 2024-09-11T01:49:09.865 INFO:tasks.rados.rados.0.smithi071.stdout:2965: finishing write tid 1 to smithi07134777-5 2024-09-11T01:49:09.865 INFO:tasks.rados.rados.0.smithi071.stdout:2965: finishing write tid 2 to smithi07134777-5 2024-09-11T01:49:09.865 INFO:tasks.rados.rados.0.smithi071.stdout:2965: finishing write tid 3 to smithi07134777-5 2024-09-11T01:49:09.865 INFO:tasks.rados.rados.0.smithi071.stdout:2965: finishing write tid 4 to smithi07134777-5 2024-09-11T01:49:09.865 INFO:tasks.rados.rados.0.smithi071.stdout:2965: finishing write tid 5 to smithi07134777-5 2024-09-11T01:49:09.865 INFO:tasks.rados.rados.0.smithi071.stdout:2965: oid 5 updating version 0 to 1176 2024-09-11T01:49:09.865 INFO:tasks.rados.rados.0.smithi071.stdout:2965: oid 5 version 1176 is already newer than 1175 2024-09-11T01:49:09.865 INFO:tasks.rados.rados.0.smithi071.stdout:2965: oid 5 updating version 1176 to 1177 2024-09-11T01:49:09.865 INFO:tasks.rados.rados.0.smithi071.stdout:2965: oid 5 version 1177 is already newer than 1174 2024-09-11T01:49:09.865 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 1177 (ObjNum 1045 snap 337 seq_num 1045) dirty exists 2024-09-11T01:49:09.865 INFO:tasks.rados.rados.0.smithi071.stdout:2965: left oid 5 (ObjNum 1045 snap 337 seq_num 1045) 2024-09-11T01:49:09.865 INFO:tasks.rados.rados.0.smithi071.stdout:2966: finishing write tid 1 to smithi07134777-1 2024-09-11T01:49:09.869 INFO:tasks.rados.rados.0.smithi071.stdout:2966: finishing write tid 2 to smithi07134777-1 2024-09-11T01:49:09.869 INFO:tasks.rados.rados.0.smithi071.stdout:2966: finishing write tid 3 to smithi07134777-1 2024-09-11T01:49:09.869 INFO:tasks.rados.rados.0.smithi071.stdout:2966: finishing write tid 4 to smithi07134777-1 2024-09-11T01:49:09.869 INFO:tasks.rados.rados.0.smithi071.stdout:2966: oid 1 updating version 0 to 1786 2024-09-11T01:49:09.869 INFO:tasks.rados.rados.0.smithi071.stdout:2966: oid 1 updating version 1786 to 1787 2024-09-11T01:49:09.869 INFO:tasks.rados.rados.0.smithi071.stdout:2966: oid 1 updating version 1787 to 1788 2024-09-11T01:49:09.870 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 1788 (ObjNum 1046 snap 337 seq_num 1046) dirty exists 2024-09-11T01:49:09.870 INFO:tasks.rados.rados.0.smithi071.stdout:2966: left oid 1 (ObjNum 1046 snap 337 seq_num 1046) 2024-09-11T01:49:10.532 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:49:10.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.644+0000 7f65bcf82640 10 monclient: get_monmap_and_config 2024-09-11T01:49:10.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.644+0000 7f65bcf82640 10 monclient: build_initial_monmap 2024-09-11T01:49:10.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.644+0000 7f65bcf82640 10 monclient: monmap: 2024-09-11T01:49:10.646 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:10.646 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:10.646 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:49:10.644823+0000 2024-09-11T01:49:10.646 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:49:10.644823+0000 2024-09-11T01:49:10.646 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:10.646 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:10.646 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:49:10.646 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:49:10.647 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:49:10.647 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:49:10.647 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:49:10.647 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:10.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.644+0000 7f65bcf82640 1 Processor -- start 2024-09-11T01:49:10.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.644+0000 7f65bcf82640 1 -- start start 2024-09-11T01:49:10.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.644+0000 7f65bcf82640 10 monclient: init 2024-09-11T01:49:10.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.645+0000 7f65bcf82640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:10.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.645+0000 7f65bcf82640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-11T01:49:10.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.645+0000 7f65bcf82640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f65b8138230 0x7f65b8138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:10.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.645+0000 7f65bcf82640 10 monclient(hunting): picked mon.noname-a con 0x7f65b8138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:49:10.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.645+0000 7f65bcf82640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f65b809d030 0x7f65b80a5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:10.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.645+0000 7f65bcf82640 10 monclient(hunting): picked mon.noname-c con 0x7f65b809d030 addr v2:172.21.15.71:3300/0 2024-09-11T01:49:10.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.645+0000 7f65bcf82640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f65b809d460 addr v1:172.21.15.71:6789/0 2024-09-11T01:49:10.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.645+0000 7f65bcf82640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65b80a6e30 con 0x7f65b809d460 2024-09-11T01:49:10.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.645+0000 7f65bcf82640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:10.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.645+0000 7f65bcf82640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f65b809c9c0 con 0x7f65b8138230 2024-09-11T01:49:10.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.645+0000 7f65bcf82640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:10.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.645+0000 7f65bcf82640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f65b809ba20 con 0x7f65b809d030 2024-09-11T01:49:10.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.645+0000 7f65bcf82640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:10.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.645+0000 7f65bcf82640 10 monclient(hunting): authenticate will time out at 1957.087611s 2024-09-11T01:49:10.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.645+0000 7f65b659c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f65b8138230 0x7f65b8138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:10.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.645+0000 7f65b659c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f65b8138230 0x7f65b8138610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34558/0 (socket says 172.21.15.12:34558) 2024-09-11T01:49:10.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.645+0000 7f65b659c640 1 -- 172.21.15.12:0/1114332911 learned_addr learned my addr 172.21.15.12:0/1114332911 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:49:10.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.645+0000 7f65b659c640 10 monclient(hunting): get_auth_request con 0x7f65b8138230 auth_method 0 2024-09-11T01:49:10.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.645+0000 7f65b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:10.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.645+0000 7f65b659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:10.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.645+0000 7f65b659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:10.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.645+0000 7f65b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:10.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.645+0000 7f65b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:10.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.645+0000 7f65b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:10.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.645+0000 7f65b5d9b640 1 --2- 172.21.15.12:0/1114332911 >> v2:172.21.15.71:3300/0 conn(0x7f65b809d030 0x7f65b80a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:10.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.645+0000 7f65b659c640 10 monclient(hunting): handle_auth_done global_id 214185 payload 274 2024-09-11T01:49:10.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.646+0000 7f65b659c640 1 -- 172.21.15.12:0/1114332911 >> v2:172.21.15.71:3300/0 conn(0x7f65b809d030 msgr2=0x7f65b80a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:49:10.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.646+0000 7f65b659c640 1 --2- 172.21.15.12:0/1114332911 >> v2:172.21.15.71:3300/0 conn(0x7f65b809d030 0x7f65b80a5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:10.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.646+0000 7f65b659c640 1 -- 172.21.15.12:0/1114332911 >> v1:172.21.15.71:6789/0 conn(0x7f65b809d460 legacy=0x7f65b80a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:49:10.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.646+0000 7f65b659c640 10 monclient: _finish_hunting 0 2024-09-11T01:49:10.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.646+0000 7f65b659c640 1 monclient: found mon.noname-a 2024-09-11T01:49:10.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.646+0000 7f65b659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.646+0000 7f65b659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:49:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.646+0000 7f65b659c640 1 -- 172.21.15.12:0/1114332911 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65b80a66a0 con 0x7f65b8138230 2024-09-11T01:49:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.646+0000 7f65b5d9b640 10 monclient: get_auth_request con 0x7f65b809d030 auth_method 0 2024-09-11T01:49:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.646+0000 7f65b5d9b640 1 --2- 172.21.15.12:0/1114332911 >> v2:172.21.15.71:3300/0 conn(0x7f65b809d030 0x7f65b80a5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:49:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.648+0000 7f65b659c640 1 -- 172.21.15.12:0/1114332911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65b8138230 msgr2=0x7f65b8138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.648+0000 7f65b659c640 1 --2- 172.21.15.12:0/1114332911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65b8138230 0x7f65b8138610 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f65b809dae0 tx=0x7f65a802a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6f6ed2e9ec319fd server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.649+0000 7f65b559a640 1 -- 172.21.15.12:0/1114332911 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a8036070 con 0x7f65b8138230 2024-09-11T01:49:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.649+0000 7f65b559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.649+0000 7f65b559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:49:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.649+0000 7f65b559a640 10 monclient: dump: 2024-09-11T01:49:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:10.656 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:10.656 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:10.656 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:10.656 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:10.656 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:10.656 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:10.656 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:10.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.649+0000 7f65b559a640 10 monclient: _finish_auth 0 2024-09-11T01:49:10.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.649+0000 7f65b559a640 10 monclient: _check_auth_tickets 2024-09-11T01:49:10.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.649+0000 7f65b559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:10.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.649+0000 7f65b559a640 1 -- 172.21.15.12:0/1114332911 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65a8007800 con 0x7f65b8138230 2024-09-11T01:49:10.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.649+0000 7f65b559a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:10.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.649+0000 7f65b559a640 1 -- 172.21.15.12:0/1114332911 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a8007bd0 con 0x7f65b8138230 2024-09-11T01:49:10.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.649+0000 7f65bcf82640 5 monclient: authenticate success, global_id 214185 2024-09-11T01:49:10.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.649+0000 7f65bcf82640 10 monclient: get_monmap_and_config success 2024-09-11T01:49:10.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.649+0000 7f65b559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:10.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.649+0000 7f65bcf82640 4 set_mon_vals no callback set 2024-09-11T01:49:10.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.649+0000 7f65b559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:10.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.649+0000 7f65b559a640 10 monclient: dump: 2024-09-11T01:49:10.657 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:10.657 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:10.657 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:10.657 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:10.657 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:10.657 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:10.657 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:10.658 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:10.658 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:10.658 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:10.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.649+0000 7f65bcf82640 10 monclient: shutdown 2024-09-11T01:49:10.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.649+0000 7f65bcf82640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:10.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.649+0000 7f65bcf82640 1 -- 172.21.15.12:0/1114332911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65b8138230 msgr2=0x7f65b8138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:10.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.649+0000 7f65bcf82640 1 --2- 172.21.15.12:0/1114332911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65b8138230 0x7f65b8138610 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f65b809dae0 tx=0x7f65a802a570 comp rx=0 tx=0).stop 2024-09-11T01:49:10.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.649+0000 7f65bcf82640 1 -- 172.21.15.12:0/1114332911 shutdown_connections 2024-09-11T01:49:10.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.649+0000 7f65bcf82640 1 --2- 172.21.15.12:0/1114332911 >> v2:172.21.15.71:3300/0 conn(0x7f65b809d030 0x7f65b80a5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:10.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.649+0000 7f65bcf82640 1 --2- 172.21.15.12:0/1114332911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65b8138230 0x7f65b8138610 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:10.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.649+0000 7f65bcf82640 1 -- 172.21.15.12:0/1114332911 >> 172.21.15.12:0/1114332911 conn(0x7f65b80a7120 msgr2=0x7f65b80a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:10.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.650+0000 7f65bcf82640 1 -- 172.21.15.12:0/1114332911 shutdown_connections 2024-09-11T01:49:10.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.650+0000 7f65bcf82640 1 -- 172.21.15.12:0/1114332911 wait complete. 2024-09-11T01:49:10.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.650+0000 7f65bcf82640 10 monclient: build_initial_monmap 2024-09-11T01:49:10.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.650+0000 7f65bcf82640 10 monclient: monmap: 2024-09-11T01:49:10.664 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:10.664 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:49:10.664 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:49:10.664 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:49:10.664 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:10.664 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:10.665 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:49:10.665 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:49:10.665 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:49:10.665 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:10.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65bcf82640 1 Processor -- start 2024-09-11T01:49:10.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65bcf82640 1 -- start start 2024-09-11T01:49:10.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65bcf82640 10 monclient: init 2024-09-11T01:49:10.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65bcf82640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:10.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65bcf82640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:49:10.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65bcf82640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65b809d030 0x7f65b81cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:10.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65bcf82640 10 monclient(hunting): picked mon.noname-b con 0x7f65b809d030 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:49:10.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65bcf82640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65b809d460 0x7f65b81371e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:10.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65bcf82640 10 monclient(hunting): picked mon.noname-a con 0x7f65b809d460 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:49:10.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65bcf82640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f65b8138230 0x7f65b8137720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:10.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65bcf82640 10 monclient(hunting): picked mon.noname-c con 0x7f65b8138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:10.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65bcf82640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:10.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65bcf82640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f65b806f970 con 0x7f65b809d460 2024-09-11T01:49:10.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65bcf82640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:10.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65bcf82640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f65b806f7f0 con 0x7f65b809d030 2024-09-11T01:49:10.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65bcf82640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:10.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65bcf82640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f65b80a5c70 con 0x7f65b8138230 2024-09-11T01:49:10.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65bcf82640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:10.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65bcf82640 10 monclient(hunting): authenticate will time out at 1957.093790s 2024-09-11T01:49:10.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65b5d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65b809d460 0x7f65b81371e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:10.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65b5d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65b809d460 0x7f65b81371e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34562/0 (socket says 172.21.15.12:34562) 2024-09-11T01:49:10.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65b5d9b640 1 -- 172.21.15.12:0/3731196248 learned_addr learned my addr 172.21.15.12:0/3731196248 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:49:10.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65b5d9b640 10 monclient(hunting): get_auth_request con 0x7f65b809d460 auth_method 0 2024-09-11T01:49:10.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65b5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:10.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65b5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:10.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65b5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:10.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65b6d9d640 1 --2- 172.21.15.12:0/3731196248 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f65b8138230 0x7f65b8137720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:10.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65b6d9d640 10 monclient(hunting): get_auth_request con 0x7f65b8138230 auth_method 0 2024-09-11T01:49:10.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65b6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:10.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65b6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:10.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65b6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:10.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65b5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:10.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65b5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:10.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.651+0000 7f65b5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:10.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.652+0000 7f65b6d9d640 1 -- 172.21.15.12:0/3731196248 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f65b8138230 msgr2=0x7f65b8137720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:49:10.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.652+0000 7f65b6d9d640 1 -- 172.21.15.12:0/3731196248 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f65b8138230 msgr2=0x7f65b8137720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:49:10.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.652+0000 7f65b6d9d640 1 --2- 172.21.15.12:0/3731196248 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f65b8138230 0x7f65b8137720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:49:10.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.652+0000 7f65b659c640 1 --2- 172.21.15.12:0/3731196248 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65b809d030 0x7f65b81cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:10.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.652+0000 7f65b659c640 10 monclient(hunting): get_auth_request con 0x7f65b809d030 auth_method 0 2024-09-11T01:49:10.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.652+0000 7f65b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:10.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.652+0000 7f65b659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:10.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.652+0000 7f65b659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:10.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.652+0000 7f65b6d9d640 1 --2- 172.21.15.12:0/3731196248 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f65b8138230 0x7f65b8137720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:10.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.652+0000 7f65b5d9b640 10 monclient(hunting): handle_auth_done global_id 214188 payload 824 2024-09-11T01:49:10.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.652+0000 7f65b5d9b640 1 -- 172.21.15.12:0/3731196248 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f65b8138230 msgr2=0x7f65b8137720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:49:10.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.652+0000 7f65b5d9b640 1 --2- 172.21.15.12:0/3731196248 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f65b8138230 0x7f65b8137720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:10.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.652+0000 7f65b5d9b640 1 -- 172.21.15.12:0/3731196248 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65b809d030 msgr2=0x7f65b81cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:10.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.652+0000 7f65b5d9b640 1 --2- 172.21.15.12:0/3731196248 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65b809d030 0x7f65b81cd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:10.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.652+0000 7f65b5d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:49:10.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.652+0000 7f65b5d9b640 1 monclient: found mon.noname-a 2024-09-11T01:49:10.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.652+0000 7f65b5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:10.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.652+0000 7f65b5d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:49:10.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.652+0000 7f65b5d9b640 1 -- 172.21.15.12:0/3731196248 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65b81cd730 con 0x7f65b809d460 2024-09-11T01:49:10.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.654+0000 7f65b5d9b640 1 -- 172.21.15.12:0/3731196248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65b809d460 msgr2=0x7f65b81371e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:10.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.654+0000 7f65b5d9b640 1 --2- 172.21.15.12:0/3731196248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65b809d460 0x7f65b81371e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f65ac00c820 tx=0x7f65ac00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:10.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.654+0000 7f65a6ffd640 1 -- 172.21.15.12:0/3731196248 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ac015070 con 0x7f65b809d460 2024-09-11T01:49:10.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.654+0000 7f65a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:10.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.654+0000 7f65a6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:49:10.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.654+0000 7f65a6ffd640 10 monclient: dump: 2024-09-11T01:49:10.673 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:10.673 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:10.673 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:10.673 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:10.673 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:10.673 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:10.674 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:10.674 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:10.674 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:10.674 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:10.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.654+0000 7f65a6ffd640 10 monclient: _finish_auth 0 2024-09-11T01:49:10.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.654+0000 7f65a6ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:49:10.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.654+0000 7f65a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:10.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.654+0000 7f65bcf82640 5 monclient: authenticate success, global_id 214188 2024-09-11T01:49:10.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.654+0000 7f65a6ffd640 1 -- 172.21.15.12:0/3731196248 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65ac007c40 con 0x7f65b809d460 2024-09-11T01:49:10.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.654+0000 7f65bcf82640 10 monclient: _renew_subs 2024-09-11T01:49:10.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.654+0000 7f65bcf82640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:10.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.654+0000 7f65bcf82640 1 -- 172.21.15.12:0/3731196248 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65b80a1550 con 0x7f65b809d460 2024-09-11T01:49:10.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.654+0000 7f65a6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:10.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.654+0000 7f65a6ffd640 1 -- 172.21.15.12:0/3731196248 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ac0113f0 con 0x7f65b809d460 2024-09-11T01:49:10.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.654+0000 7f65a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:10.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.654+0000 7f65a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:10.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.654+0000 7f65a6ffd640 10 monclient: dump: 2024-09-11T01:49:10.675 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:10.675 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:10.675 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:10.675 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:10.675 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:10.675 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:10.675 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:10.675 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:10.675 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:10.675 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:10.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.654+0000 7f65a7fff640 4 set_mon_vals no callback set 2024-09-11T01:49:10.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.654+0000 7f65bcf82640 10 monclient: _renew_subs 2024-09-11T01:49:10.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.654+0000 7f65bcf82640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:10.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.654+0000 7f65bcf82640 1 -- 172.21.15.12:0/3731196248 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65b8124300 con 0x7f65b809d460 2024-09-11T01:49:10.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.655+0000 7f65a6ffd640 1 -- 172.21.15.12:0/3731196248 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f65ac011590 con 0x7f65b809d460 2024-09-11T01:49:10.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.655+0000 7f65bcf82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:10.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.655+0000 7f65bcf82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:10.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.655+0000 7f65bcf82640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:10.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.655+0000 7f65bcf82640 1 -- 172.21.15.12:0/3731196248 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65b80a9c40 con 0x7f65b809d460 2024-09-11T01:49:10.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.655+0000 7f65a6ffd640 1 --2- 172.21.15.12:0/3731196248 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f659803e260 0x7f6598040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:10.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.656+0000 7f65a6ffd640 1 -- 172.21.15.12:0/3731196248 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(667..667 src has 507..667) ==== 4815+0+0 (secure 0 0 0) 0x7f65ac053110 con 0x7f65b809d460 2024-09-11T01:49:10.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.656+0000 7f65b659c640 1 --2- 172.21.15.12:0/3731196248 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f659803e260 0x7f6598040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:10.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.656+0000 7f65b659c640 10 monclient: get_auth_request con 0x7f659803e260 auth_method 0 2024-09-11T01:49:10.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.659+0000 7f65b659c640 1 --2- 172.21.15.12:0/3731196248 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f659803e260 0x7f6598040720 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f65b809bff0 tx=0x7f65a8009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:10.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.660+0000 7f65a6ffd640 1 -- 172.21.15.12:0/3731196248 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f65b80a9c40 con 0x7f65b809d460 2024-09-11T01:49:10.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.660+0000 7f65a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:10.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.660+0000 7f65a6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:49:10.870 INFO:tasks.rados.rados.0.smithi071.stdout:2964: done (3 left) 2024-09-11T01:49:10.870 INFO:tasks.rados.rados.0.smithi071.stdout:2965: done (2 left) 2024-09-11T01:49:10.870 INFO:tasks.rados.rados.0.smithi071.stdout:2966: done (1 left) 2024-09-11T01:49:10.871 INFO:tasks.rados.rados.0.smithi071.stdout:2967: done (0 left) 2024-09-11T01:49:10.871 INFO:tasks.rados.rados.0.smithi071.stdout:2968: rollback oid 19 current snap is 338 2024-09-11T01:49:10.871 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 19 to 300 2024-09-11T01:49:10.871 INFO:tasks.rados.rados.0.smithi071.stdout:2969: write oid 20 current snap is 338 2024-09-11T01:49:10.871 INFO:tasks.rados.rados.0.smithi071.stdout:2969: seq_num 1047 ranges {796929=726064,2184647=659366,3302344=294585} 2024-09-11T01:49:10.878 INFO:tasks.rados.rados.0.smithi071.stdout:2969: writing smithi07134777-20 from 796929 to 1522993 tid 1 2024-09-11T01:49:10.886 INFO:tasks.rados.rados.0.smithi071.stdout:2969: writing smithi07134777-20 from 2184647 to 2844013 tid 2 2024-09-11T01:49:10.890 INFO:tasks.rados.rados.0.smithi071.stdout:2969: writing smithi07134777-20 from 3302344 to 3596929 tid 3 2024-09-11T01:49:10.890 INFO:tasks.rados.rados.0.smithi071.stdout:2970: snap_remove snap 316 2024-09-11T01:49:10.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.918+0000 7f65bcf82640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:49:10.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.918+0000 7f65bcf82640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:49:10.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.918+0000 7f65bcf82640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:10.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.918+0000 7f65bcf82640 1 -- 172.21.15.12:0/3731196248 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f65b8138610 con 0x7f65b809d460 2024-09-11T01:49:10.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.918+0000 7f65a6ffd640 1 -- 172.21.15.12:0/3731196248 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f65ac051af0 con 0x7f65b809d460 2024-09-11T01:49:10.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.918+0000 7f65a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:49:10.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.918+0000 7f65a6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:49:10.921 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":186,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:49:10.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.920+0000 7f65bcf82640 1 -- 172.21.15.12:0/3731196248 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f659803e260 msgr2=0x7f6598040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:10.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.920+0000 7f65bcf82640 1 --2- 172.21.15.12:0/3731196248 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f659803e260 0x7f6598040720 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f65b809bff0 tx=0x7f65a8009000 comp rx=0 tx=0).stop 2024-09-11T01:49:10.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.920+0000 7f65bcf82640 10 monclient: shutdown 2024-09-11T01:49:10.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.920+0000 7f65bcf82640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:10.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.920+0000 7f65bcf82640 1 -- 172.21.15.12:0/3731196248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65b809d460 msgr2=0x7f65b81371e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:10.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.920+0000 7f65bcf82640 1 --2- 172.21.15.12:0/3731196248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65b809d460 0x7f65b81371e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f65ac00c820 tx=0x7f65ac00cc20 comp rx=0 tx=0).stop 2024-09-11T01:49:10.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.920+0000 7f65bcf82640 1 -- 172.21.15.12:0/3731196248 shutdown_connections 2024-09-11T01:49:10.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.920+0000 7f65bcf82640 1 --2- 172.21.15.12:0/3731196248 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f65b8138230 0x7f65b8137720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:10.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.921+0000 7f65bcf82640 1 --2- 172.21.15.12:0/3731196248 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f659803e260 0x7f6598040720 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:10.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.921+0000 7f65bcf82640 1 --2- 172.21.15.12:0/3731196248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65b809d460 0x7f65b81371e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:10.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.921+0000 7f65bcf82640 1 --2- 172.21.15.12:0/3731196248 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65b809d030 0x7f65b81cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:10.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.921+0000 7f65bcf82640 1 -- 172.21.15.12:0/3731196248 >> 172.21.15.12:0/3731196248 conn(0x7f65b80a7120 msgr2=0x7f65b80a3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:10.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.921+0000 7f65bcf82640 1 -- 172.21.15.12:0/3731196248 shutdown_connections 2024-09-11T01:49:10.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:10.921+0000 7f65bcf82640 1 -- 172.21.15.12:0/3731196248 wait complete. 2024-09-11T01:49:11.874 INFO:tasks.rados.rados.0.smithi071.stdout:2970: done (2 left) 2024-09-11T01:49:11.874 INFO:tasks.rados.rados.0.smithi071.stdout:2971: read oid 9 snap -1 2024-09-11T01:49:11.874 INFO:tasks.rados.rados.0.smithi071.stdout:2971: expect (ObjNum 601 snap 172 seq_num 601) 2024-09-11T01:49:11.875 INFO:tasks.rados.rados.0.smithi071.stdout:2972: write oid 21 current snap is 338 2024-09-11T01:49:11.875 INFO:tasks.rados.rados.0.smithi071.stdout:2968: finishing rollback tid 0 to smithi07134777-19 2024-09-11T01:49:11.875 INFO:tasks.rados.rados.0.smithi071.stdout:2972: seq_num 1048 ranges {413736=401313,1326776=403024,2413414=457893,3438716=175020} 2024-09-11T01:49:11.878 INFO:tasks.rados.rados.0.smithi071.stdout:2972: writing smithi07134777-21 from 413736 to 815049 tid 1 2024-09-11T01:49:11.884 INFO:tasks.rados.rados.0.smithi071.stdout:2972: writing smithi07134777-21 from 1326776 to 1729800 tid 2 2024-09-11T01:49:11.891 INFO:tasks.rados.rados.0.smithi071.stdout:2972: writing smithi07134777-21 from 2413414 to 2871307 tid 3 2024-09-11T01:49:11.898 INFO:tasks.rados.rados.0.smithi071.stdout:2972: writing smithi07134777-21 from 3438716 to 3613736 tid 4 2024-09-11T01:49:11.898 INFO:tasks.rados.rados.0.smithi071.stdout:2973: snap_create 2024-09-11T01:49:11.898 INFO:tasks.rados.rados.0.smithi071.stdout:2968: finishing rollback tid 1 to smithi07134777-19 2024-09-11T01:49:11.898 INFO:tasks.rados.rados.0.smithi071.stdout:2968: finishing rollback tid 2 to smithi07134777-19 2024-09-11T01:49:11.898 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 1770 (ObjNum 598 snap 170 seq_num 598) dirty exists 2024-09-11T01:49:11.898 INFO:tasks.rados.rados.0.smithi071.stdout:2969: finishing write tid 1 to smithi07134777-20 2024-09-11T01:49:11.898 INFO:tasks.rados.rados.0.smithi071.stdout:2969: finishing write tid 2 to smithi07134777-20 2024-09-11T01:49:11.898 INFO:tasks.rados.rados.0.smithi071.stdout:2969: finishing write tid 3 to smithi07134777-20 2024-09-11T01:49:11.898 INFO:tasks.rados.rados.0.smithi071.stdout:2969: finishing write tid 4 to smithi07134777-20 2024-09-11T01:49:11.898 INFO:tasks.rados.rados.0.smithi071.stdout:2969: finishing write tid 5 to smithi07134777-20 2024-09-11T01:49:11.898 INFO:tasks.rados.rados.0.smithi071.stdout:2969: oid 20 updating version 0 to 1068 2024-09-11T01:49:11.898 INFO:tasks.rados.rados.0.smithi071.stdout:2969: oid 20 version 1068 is already newer than 1067 2024-09-11T01:49:11.899 INFO:tasks.rados.rados.0.smithi071.stdout:2969: oid 20 version 1068 is already newer than 1065 2024-09-11T01:49:11.899 INFO:tasks.rados.rados.0.smithi071.stdout:2969: oid 20 version 1068 is already newer than 1066 2024-09-11T01:49:11.899 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 1068 (ObjNum 1047 snap 338 seq_num 1047) dirty exists 2024-09-11T01:49:11.899 INFO:tasks.rados.rados.0.smithi071.stdout:2969: left oid 20 (ObjNum 1047 snap 338 seq_num 1047) 2024-09-11T01:49:11.899 INFO:tasks.rados.rados.0.smithi071.stdout:2971: expect (ObjNum 601 snap 172 seq_num 601) 2024-09-11T01:49:11.929 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 849226 2024-09-11T01:49:11.975 INFO:tasks.rados.rados.0.smithi071.stdout:2972: finishing write tid 1 to smithi07134777-21 2024-09-11T01:49:11.975 INFO:tasks.rados.rados.0.smithi071.stdout:2972: finishing write tid 2 to smithi07134777-21 2024-09-11T01:49:11.975 INFO:tasks.rados.rados.0.smithi071.stdout:2972: finishing write tid 3 to smithi07134777-21 2024-09-11T01:49:11.975 INFO:tasks.rados.rados.0.smithi071.stdout:2972: finishing write tid 4 to smithi07134777-21 2024-09-11T01:49:11.975 INFO:tasks.rados.rados.0.smithi071.stdout:2972: finishing write tid 5 to smithi07134777-21 2024-09-11T01:49:11.976 INFO:tasks.rados.rados.0.smithi071.stdout:2972: finishing write tid 6 to smithi07134777-21 2024-09-11T01:49:11.976 INFO:tasks.rados.rados.0.smithi071.stdout:2972: oid 21 updating version 0 to 2038 2024-09-11T01:49:11.976 INFO:tasks.rados.rados.0.smithi071.stdout:2972: oid 21 updating version 2038 to 2039 2024-09-11T01:49:11.976 INFO:tasks.rados.rados.0.smithi071.stdout:2972: oid 21 version 2039 is already newer than 2036 2024-09-11T01:49:11.976 INFO:tasks.rados.rados.0.smithi071.stdout:2972: oid 21 version 2039 is already newer than 2035 2024-09-11T01:49:11.976 INFO:tasks.rados.rados.0.smithi071.stdout:2972: oid 21 version 2039 is already newer than 2037 2024-09-11T01:49:11.976 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 2039 (ObjNum 1048 snap 338 seq_num 1048) dirty exists 2024-09-11T01:49:11.976 INFO:tasks.rados.rados.0.smithi071.stdout:2972: left oid 21 (ObjNum 1048 snap 338 seq_num 1048) 2024-09-11T01:49:12.877 INFO:tasks.rados.rados.0.smithi071.stdout:2968: done (4 left) 2024-09-11T01:49:12.877 INFO:tasks.rados.rados.0.smithi071.stdout:2969: done (3 left) 2024-09-11T01:49:12.877 INFO:tasks.rados.rados.0.smithi071.stdout:2971: done (2 left) 2024-09-11T01:49:12.877 INFO:tasks.rados.rados.0.smithi071.stdout:2972: done (1 left) 2024-09-11T01:49:12.877 INFO:tasks.rados.rados.0.smithi071.stdout:2973: done (0 left) 2024-09-11T01:49:12.877 INFO:tasks.rados.rados.0.smithi071.stdout:2974: snap_remove snap 315 2024-09-11T01:49:13.823 INFO:tasks.rados.rados.0.smithi071.stdout:2974: done (0 left) 2024-09-11T01:49:13.823 INFO:tasks.rados.rados.0.smithi071.stdout:2975: write oid 12 current snap is 339 2024-09-11T01:49:13.824 INFO:tasks.rados.rados.0.smithi071.stdout:2975: seq_num 1049 ranges {430543=476562,1385258=663035,2840837=669165} 2024-09-11T01:49:13.828 INFO:tasks.rados.rados.0.smithi071.stdout:2975: writing smithi07134777-12 from 430543 to 907105 tid 1 2024-09-11T01:49:13.834 INFO:tasks.rados.rados.0.smithi071.stdout:2975: writing smithi07134777-12 from 1385258 to 2048293 tid 2 2024-09-11T01:49:13.842 INFO:tasks.rados.rados.0.smithi071.stdout:2975: writing smithi07134777-12 from 2840837 to 3510002 tid 3 2024-09-11T01:49:13.842 INFO:tasks.rados.rados.0.smithi071.stdout:2976: snap_remove snap 336 2024-09-11T01:49:13.934 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:49:14.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.041+0000 7f5b6c8ef640 10 monclient: get_monmap_and_config 2024-09-11T01:49:14.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.041+0000 7f5b6c8ef640 10 monclient: build_initial_monmap 2024-09-11T01:49:14.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.041+0000 7f5b6c8ef640 10 monclient: monmap: 2024-09-11T01:49:14.043 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:14.043 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:14.043 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:49:14.042280+0000 2024-09-11T01:49:14.043 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:49:14.042280+0000 2024-09-11T01:49:14.043 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:14.044 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:14.044 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:49:14.044 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:49:14.044 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:49:14.044 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:49:14.044 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:49:14.044 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:14.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.042+0000 7f5b6c8ef640 1 Processor -- start 2024-09-11T01:49:14.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.042+0000 7f5b6c8ef640 1 -- start start 2024-09-11T01:49:14.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.042+0000 7f5b6c8ef640 10 monclient: init 2024-09-11T01:49:14.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.042+0000 7f5b6c8ef640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:14.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.042+0000 7f5b6c8ef640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-11T01:49:14.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.042+0000 7f5b6c8ef640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b68138230 0x7f5b68138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:14.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.042+0000 7f5b6c8ef640 10 monclient(hunting): picked mon.noname-b con 0x7f5b68138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:14.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.042+0000 7f5b6c8ef640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5b6809d030 0x7f5b680a5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:14.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.042+0000 7f5b6c8ef640 10 monclient(hunting): picked mon.noname-a con 0x7f5b6809d030 addr v2:172.21.15.12:3300/0 2024-09-11T01:49:14.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.042+0000 7f5b6c8ef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5b6809d430 addr v1:172.21.15.12:6789/0 2024-09-11T01:49:14.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.042+0000 7f5b6c8ef640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b680a6e30 con 0x7f5b6809d430 2024-09-11T01:49:14.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.042+0000 7f5b6c8ef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:14.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.042+0000 7f5b6c8ef640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f5b6809c9c0 con 0x7f5b6809d030 2024-09-11T01:49:14.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.042+0000 7f5b6c8ef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:14.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.042+0000 7f5b6c8ef640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5b6809ba20 con 0x7f5b68138230 2024-09-11T01:49:14.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.042+0000 7f5b6c8ef640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:14.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.042+0000 7f5b6c8ef640 10 monclient(hunting): authenticate will time out at 1960.484994s 2024-09-11T01:49:14.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b5ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5b6809d030 0x7f5b680a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:14.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b6659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b68138230 0x7f5b68138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:14.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b66d9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5b6809d430 0x7f5b680a5560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42216/0 (socket says 172.21.15.12:42216) 2024-09-11T01:49:14.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b66d9d640 1 -- 172.21.15.12:0/1155784441 learned_addr learned my addr 172.21.15.12:0/1155784441 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:49:14.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b6659c640 10 monclient(hunting): get_auth_request con 0x7f5b68138230 auth_method 0 2024-09-11T01:49:14.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b6659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:14.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b6659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:14.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b6659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:14.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b5ffff640 10 monclient(hunting): get_auth_request con 0x7f5b6809d030 auth_method 0 2024-09-11T01:49:14.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:14.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:14.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:14.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:14.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:14.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b6659c640 1 -- 172.21.15.12:0/1155784441 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b68138230 msgr2=0x7f5b68138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:49:14.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b6659c640 1 -- 172.21.15.12:0/1155784441 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b68138230 msgr2=0x7f5b68138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:49:14.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b6659c640 1 --2- 172.21.15.12:0/1155784441 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b68138230 0x7f5b68138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:49:14.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:14.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b6659c640 1 --2- 172.21.15.12:0/1155784441 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b68138230 0x7f5b68138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:14.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b5ffff640 10 monclient(hunting): handle_auth_done global_id 214191 payload 274 2024-09-11T01:49:14.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b5ffff640 1 -- 172.21.15.12:0/1155784441 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b68138230 msgr2=0x7f5b68138610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:49:14.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b5ffff640 1 --2- 172.21.15.12:0/1155784441 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b68138230 0x7f5b68138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b5ffff640 1 -- 172.21.15.12:0/1155784441 >> v1:172.21.15.12:6789/0 conn(0x7f5b6809d430 legacy=0x7f5b680a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:49:14.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b5ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:49:14.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b5ffff640 1 monclient: found mon.noname-a 2024-09-11T01:49:14.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:14.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b5ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:49:14.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.043+0000 7f5b5ffff640 1 -- 172.21.15.12:0/1155784441 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b680a66a0 con 0x7f5b6809d030 2024-09-11T01:49:14.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.046+0000 7f5b5ffff640 1 -- 172.21.15.12:0/1155784441 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b6809d030 msgr2=0x7f5b680a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:14.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.046+0000 7f5b5ffff640 1 --2- 172.21.15.12:0/1155784441 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b6809d030 0x7f5b680a5020 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5b6806d650 tx=0x7f5b50029fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86e058454a0728d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:14.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.046+0000 7f5b65d9b640 1 -- 172.21.15.12:0/1155784441 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b50037070 con 0x7f5b6809d030 2024-09-11T01:49:14.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.046+0000 7f5b65d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:14.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.046+0000 7f5b65d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:49:14.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.046+0000 7f5b65d9b640 10 monclient: dump: 2024-09-11T01:49:14.052 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:14.052 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:14.052 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.052 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.052 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:14.052 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:14.052 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:14.052 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:14.053 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:14.053 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:14.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.046+0000 7f5b65d9b640 10 monclient: _finish_auth 0 2024-09-11T01:49:14.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.046+0000 7f5b65d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:49:14.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.046+0000 7f5b65d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:14.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.046+0000 7f5b65d9b640 1 -- 172.21.15.12:0/1155784441 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b50032440 con 0x7f5b6809d030 2024-09-11T01:49:14.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.046+0000 7f5b6c8ef640 5 monclient: authenticate success, global_id 214191 2024-09-11T01:49:14.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.046+0000 7f5b6c8ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:49:14.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.046+0000 7f5b65d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:14.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.046+0000 7f5b6c8ef640 10 monclient: get_monmap_and_config success 2024-09-11T01:49:14.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.046+0000 7f5b65d9b640 1 -- 172.21.15.12:0/1155784441 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b50032810 con 0x7f5b6809d030 2024-09-11T01:49:14.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.046+0000 7f5b65d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:14.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.046+0000 7f5b6c8ef640 4 set_mon_vals no callback set 2024-09-11T01:49:14.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.046+0000 7f5b65d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:14.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.046+0000 7f5b65d9b640 10 monclient: dump: 2024-09-11T01:49:14.054 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:14.054 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:14.054 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.054 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.054 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:14.054 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:14.054 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:14.054 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:14.054 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:14.054 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:14.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.047+0000 7f5b6c8ef640 10 monclient: shutdown 2024-09-11T01:49:14.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.047+0000 7f5b6c8ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:14.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.047+0000 7f5b6c8ef640 1 -- 172.21.15.12:0/1155784441 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b6809d030 msgr2=0x7f5b680a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:14.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.047+0000 7f5b6c8ef640 1 --2- 172.21.15.12:0/1155784441 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b6809d030 0x7f5b680a5020 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5b6806d650 tx=0x7f5b50029fb0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.047+0000 7f5b6c8ef640 1 -- 172.21.15.12:0/1155784441 shutdown_connections 2024-09-11T01:49:14.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.047+0000 7f5b6c8ef640 1 --2- 172.21.15.12:0/1155784441 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b6809d030 0x7f5b680a5020 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.047+0000 7f5b6c8ef640 1 --2- 172.21.15.12:0/1155784441 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b68138230 0x7f5b68138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.047+0000 7f5b6c8ef640 1 -- 172.21.15.12:0/1155784441 >> 172.21.15.12:0/1155784441 conn(0x7f5b680a7120 msgr2=0x7f5b680a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:14.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.047+0000 7f5b6c8ef640 1 -- 172.21.15.12:0/1155784441 shutdown_connections 2024-09-11T01:49:14.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.047+0000 7f5b6c8ef640 1 -- 172.21.15.12:0/1155784441 wait complete. 2024-09-11T01:49:14.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.047+0000 7f5b6c8ef640 10 monclient: build_initial_monmap 2024-09-11T01:49:14.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.047+0000 7f5b6c8ef640 10 monclient: monmap: 2024-09-11T01:49:14.061 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:14.061 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:49:14.061 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:49:14.061 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:49:14.061 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:14.061 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:14.061 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:49:14.061 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:49:14.061 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:49:14.061 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:14.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b6c8ef640 1 Processor -- start 2024-09-11T01:49:14.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b6c8ef640 1 -- start start 2024-09-11T01:49:14.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b6c8ef640 10 monclient: init 2024-09-11T01:49:14.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b6c8ef640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:14.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b6c8ef640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:49:14.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b6c8ef640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b6809d030 0x7f5b681cd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:14.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b6c8ef640 10 monclient(hunting): picked mon.noname-c con 0x7f5b6809d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:14.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b6c8ef640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b6809d430 0x7f5b6813f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:14.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b6c8ef640 10 monclient(hunting): picked mon.noname-a con 0x7f5b6809d430 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:49:14.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b6c8ef640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b68138230 0x7f5b6813f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:14.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b6c8ef640 10 monclient(hunting): picked mon.noname-b con 0x7f5b68138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:49:14.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b6c8ef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:14.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b6c8ef640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6806f970 con 0x7f5b6809d430 2024-09-11T01:49:14.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b6c8ef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:14.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b6c8ef640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6806f7f0 con 0x7f5b68138230 2024-09-11T01:49:14.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b6c8ef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:14.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b6c8ef640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5b680a5c70 con 0x7f5b6809d030 2024-09-11T01:49:14.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b6c8ef640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:14.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b6c8ef640 10 monclient(hunting): authenticate will time out at 1960.490823s 2024-09-11T01:49:14.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b5ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b6809d430 0x7f5b6813f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:14.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b6659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b6809d030 0x7f5b681cd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:14.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b5ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b6809d430 0x7f5b6813f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34568/0 (socket says 172.21.15.12:34568) 2024-09-11T01:49:14.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b5ffff640 1 -- 172.21.15.12:0/1375622441 learned_addr learned my addr 172.21.15.12:0/1375622441 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:49:14.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b6659c640 10 monclient(hunting): get_auth_request con 0x7f5b6809d030 auth_method 0 2024-09-11T01:49:14.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b6659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:14.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b6659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:14.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b6659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:14.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b5ffff640 10 monclient(hunting): get_auth_request con 0x7f5b6809d430 auth_method 0 2024-09-11T01:49:14.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:14.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:14.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.048+0000 7f5b5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:14.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.049+0000 7f5b6659c640 1 -- 172.21.15.12:0/1375622441 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b6809d030 msgr2=0x7f5b681cd4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:49:14.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.049+0000 7f5b6659c640 1 -- 172.21.15.12:0/1375622441 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b6809d030 msgr2=0x7f5b681cd4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:49:14.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.049+0000 7f5b5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:14.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.049+0000 7f5b6659c640 1 --2- 172.21.15.12:0/1375622441 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b6809d030 0x7f5b681cd4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:49:14.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.049+0000 7f5b5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:14.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.049+0000 7f5b5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:14.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.049+0000 7f5b6659c640 1 --2- 172.21.15.12:0/1375622441 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b6809d030 0x7f5b681cd4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:14.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.049+0000 7f5b66d9d640 1 --2- 172.21.15.12:0/1375622441 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b68138230 0x7f5b6813f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:14.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.049+0000 7f5b66d9d640 10 monclient(hunting): get_auth_request con 0x7f5b68138230 auth_method 0 2024-09-11T01:49:14.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.049+0000 7f5b66d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:14.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.049+0000 7f5b66d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:14.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.049+0000 7f5b66d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:14.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.049+0000 7f5b5ffff640 10 monclient(hunting): handle_auth_done global_id 214194 payload 824 2024-09-11T01:49:14.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.049+0000 7f5b5ffff640 1 -- 172.21.15.12:0/1375622441 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b6809d030 msgr2=0x7f5b681cd4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:49:14.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.049+0000 7f5b5ffff640 1 --2- 172.21.15.12:0/1375622441 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b6809d030 0x7f5b681cd4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.049+0000 7f5b5ffff640 1 -- 172.21.15.12:0/1375622441 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b68138230 msgr2=0x7f5b6813f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:14.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.049+0000 7f5b5ffff640 1 --2- 172.21.15.12:0/1375622441 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b68138230 0x7f5b6813f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.049+0000 7f5b5ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:49:14.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.049+0000 7f5b5ffff640 1 monclient: found mon.noname-a 2024-09-11T01:49:14.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.049+0000 7f5b5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:14.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.049+0000 7f5b5ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:49:14.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.049+0000 7f5b5ffff640 1 -- 172.21.15.12:0/1375622441 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b680a1810 con 0x7f5b6809d430 2024-09-11T01:49:14.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.051+0000 7f5b5ffff640 1 -- 172.21.15.12:0/1375622441 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b6809d430 msgr2=0x7f5b6813f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:14.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.051+0000 7f5b5ffff640 1 --2- 172.21.15.12:0/1375622441 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b6809d430 0x7f5b6813f360 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5b50009890 tx=0x7f5b500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:14.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.051+0000 7f5b5effd640 1 -- 172.21.15.12:0/1375622441 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b50041070 con 0x7f5b6809d430 2024-09-11T01:49:14.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.051+0000 7f5b5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:14.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.051+0000 7f5b5effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:49:14.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.051+0000 7f5b5effd640 10 monclient: dump: 2024-09-11T01:49:14.066 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:14.066 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:14.066 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.066 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.066 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:14.066 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:14.066 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:14.066 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:14.067 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:14.067 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:14.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.051+0000 7f5b5effd640 10 monclient: _finish_auth 0 2024-09-11T01:49:14.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.051+0000 7f5b5effd640 10 monclient: _check_auth_tickets 2024-09-11T01:49:14.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.051+0000 7f5b5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:14.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.051+0000 7f5b6c8ef640 5 monclient: authenticate success, global_id 214194 2024-09-11T01:49:14.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.051+0000 7f5b5effd640 1 -- 172.21.15.12:0/1375622441 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b5003caa0 con 0x7f5b6809d430 2024-09-11T01:49:14.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.051+0000 7f5b5effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:14.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.051+0000 7f5b5effd640 1 -- 172.21.15.12:0/1375622441 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b50037040 con 0x7f5b6809d430 2024-09-11T01:49:14.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.051+0000 7f5b5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:14.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.051+0000 7f5b5effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:14.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.051+0000 7f5b5effd640 10 monclient: dump: 2024-09-11T01:49:14.067 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:14.067 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:14.068 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.068 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.068 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:14.068 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:14.068 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:14.068 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:14.068 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:14.068 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:14.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.051+0000 7f5b6c8ef640 10 monclient: _renew_subs 2024-09-11T01:49:14.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.051+0000 7f5b6559a640 4 set_mon_vals no callback set 2024-09-11T01:49:14.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.051+0000 7f5b6c8ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:14.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.051+0000 7f5b6c8ef640 1 -- 172.21.15.12:0/1375622441 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b6813bad0 con 0x7f5b6809d430 2024-09-11T01:49:14.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.051+0000 7f5b6c8ef640 10 monclient: _renew_subs 2024-09-11T01:49:14.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.051+0000 7f5b6c8ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:14.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.051+0000 7f5b6c8ef640 1 -- 172.21.15.12:0/1375622441 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b6813d590 con 0x7f5b6809d430 2024-09-11T01:49:14.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.052+0000 7f5b6c8ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:14.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.052+0000 7f5b6c8ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:14.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.052+0000 7f5b6c8ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:14.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.052+0000 7f5b5effd640 1 -- 172.21.15.12:0/1375622441 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f5b5003d3f0 con 0x7f5b6809d430 2024-09-11T01:49:14.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.052+0000 7f5b6c8ef640 1 -- 172.21.15.12:0/1375622441 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b68138610 con 0x7f5b6809d430 2024-09-11T01:49:14.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.052+0000 7f5b5effd640 1 --2- 172.21.15.12:0/1375622441 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5b4803e290 0x7f5b48040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:14.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.053+0000 7f5b5effd640 1 -- 172.21.15.12:0/1375622441 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(671..671 src has 507..671) ==== 4843+0+0 (secure 0 0 0) 0x7f5b50073650 con 0x7f5b6809d430 2024-09-11T01:49:14.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.053+0000 7f5b6659c640 1 --2- 172.21.15.12:0/1375622441 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5b4803e290 0x7f5b48040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:14.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.053+0000 7f5b6659c640 10 monclient: get_auth_request con 0x7f5b4803e290 auth_method 0 2024-09-11T01:49:14.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.053+0000 7f5b6659c640 1 -- 172.21.15.12:0/1375622441 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5b4803e290 msgr2=0x7f5b48040750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:49:14.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.053+0000 7f5b6659c640 1 -- 172.21.15.12:0/1375622441 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5b4803e290 msgr2=0x7f5b48040750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:49:14.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.053+0000 7f5b6659c640 1 --2- 172.21.15.12:0/1375622441 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5b4803e290 0x7f5b48040750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:49:14.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.053+0000 7f5b6659c640 1 --2- 172.21.15.12:0/1375622441 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5b4803e290 0x7f5b48040750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:14.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.057+0000 7f5b5effd640 1 -- 172.21.15.12:0/1375622441 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5b5004b070 con 0x7f5b6809d430 2024-09-11T01:49:14.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.057+0000 7f5b5effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:14.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.057+0000 7f5b5effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:49:14.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.255+0000 7f5b6659c640 1 --2- 172.21.15.12:0/1375622441 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5b4803e290 0x7f5b48040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:14.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.255+0000 7f5b6659c640 10 monclient: get_auth_request con 0x7f5b4803e290 auth_method 0 2024-09-11T01:49:14.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.257+0000 7f5b6659c640 1 --2- 172.21.15.12:0/1375622441 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5b4803e290 0x7f5b48040750 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5b58006fd0 tx=0x7f5b58008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:14.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.311+0000 7f5b6c8ef640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:49:14.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.311+0000 7f5b6c8ef640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:49:14.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.311+0000 7f5b6c8ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:14.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.311+0000 7f5b6c8ef640 1 -- 172.21.15.12:0/1375622441 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5b680aa090 con 0x7f5b6809d430 2024-09-11T01:49:14.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.312+0000 7f5b5effd640 1 -- 172.21.15.12:0/1375622441 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f5b50072010 con 0x7f5b6809d430 2024-09-11T01:49:14.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.312+0000 7f5b5effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:49:14.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.312+0000 7f5b5effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:49:14.314 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":188,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:49:14.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.313+0000 7f5b6c8ef640 1 -- 172.21.15.12:0/1375622441 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5b4803e290 msgr2=0x7f5b48040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:14.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.313+0000 7f5b6c8ef640 1 --2- 172.21.15.12:0/1375622441 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5b4803e290 0x7f5b48040750 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5b58006fd0 tx=0x7f5b58008040 comp rx=0 tx=0).stop 2024-09-11T01:49:14.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.313+0000 7f5b6c8ef640 10 monclient: shutdown 2024-09-11T01:49:14.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.313+0000 7f5b6c8ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:14.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.313+0000 7f5b6c8ef640 1 -- 172.21.15.12:0/1375622441 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b6809d430 msgr2=0x7f5b6813f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:14.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.313+0000 7f5b6c8ef640 1 --2- 172.21.15.12:0/1375622441 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b6809d430 0x7f5b6813f360 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5b50009890 tx=0x7f5b500046c0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.314+0000 7f5b6c8ef640 1 -- 172.21.15.12:0/1375622441 shutdown_connections 2024-09-11T01:49:14.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.314+0000 7f5b6c8ef640 1 --2- 172.21.15.12:0/1375622441 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b68138230 0x7f5b6813f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.314+0000 7f5b6c8ef640 1 --2- 172.21.15.12:0/1375622441 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5b4803e290 0x7f5b48040750 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.314+0000 7f5b6c8ef640 1 --2- 172.21.15.12:0/1375622441 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b6809d430 0x7f5b6813f360 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.314+0000 7f5b6c8ef640 1 --2- 172.21.15.12:0/1375622441 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b6809d030 0x7f5b681cd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.314+0000 7f5b6c8ef640 1 -- 172.21.15.12:0/1375622441 >> 172.21.15.12:0/1375622441 conn(0x7f5b680a7120 msgr2=0x7f5b680a3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:14.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.314+0000 7f5b6c8ef640 1 -- 172.21.15.12:0/1375622441 shutdown_connections 2024-09-11T01:49:14.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.314+0000 7f5b6c8ef640 1 -- 172.21.15.12:0/1375622441 wait complete. 2024-09-11T01:49:14.326 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:49:14.326 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:49:14.326 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:49:14.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.426+0000 7f8ef9ba9640 10 monclient: get_monmap_and_config 2024-09-11T01:49:14.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.426+0000 7f8ef9ba9640 10 monclient: build_initial_monmap 2024-09-11T01:49:14.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.426+0000 7f8ef9ba9640 10 monclient: monmap: 2024-09-11T01:49:14.429 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:14.429 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:14.429 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:49:14.427537+0000 2024-09-11T01:49:14.429 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:49:14.427537+0000 2024-09-11T01:49:14.429 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:14.429 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:14.429 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:49:14.429 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:49:14.429 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:49:14.429 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:49:14.429 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:49:14.430 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:14.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.427+0000 7f8ef9ba9640 1 Processor -- start 2024-09-11T01:49:14.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.427+0000 7f8ef9ba9640 1 -- start start 2024-09-11T01:49:14.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.427+0000 7f8ef9ba9640 10 monclient: init 2024-09-11T01:49:14.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef9ba9640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:14.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef9ba9640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-11T01:49:14.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef9ba9640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f8ef4134ad0 0x7f8ef4134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:14.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef9ba9640 10 monclient(hunting): picked mon.noname-c con 0x7f8ef4134ad0 addr v2:172.21.15.71:3300/0 2024-09-11T01:49:14.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef9ba9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ef40a2200 addr v1:172.21.15.12:6789/0 2024-09-11T01:49:14.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef9ba9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8ef41353f0 0x7f8ef4134160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:14.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef9ba9640 10 monclient(hunting): picked mon.noname-b con 0x7f8ef41353f0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:14.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef9ba9640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ef409ed30 con 0x7f8ef40a2200 2024-09-11T01:49:14.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef9ba9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:14.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef9ba9640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f8ef409f5d0 con 0x7f8ef4134ad0 2024-09-11T01:49:14.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef9ba9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:14.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef9ba9640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8ef409f010 con 0x7f8ef41353f0 2024-09-11T01:49:14.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef9ba9640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:14.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef9ba9640 10 monclient(hunting): authenticate will time out at 1960.870182s 2024-09-11T01:49:14.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef2ffd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f8ef40a2200 0x7f8ef4133a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42218/0 (socket says 172.21.15.12:42218) 2024-09-11T01:49:14.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef3fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8ef41353f0 0x7f8ef4134160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:14.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef2ffd640 1 -- 172.21.15.12:0/2524893474 learned_addr learned my addr 172.21.15.12:0/2524893474 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:49:14.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef3fff640 10 monclient(hunting): get_auth_request con 0x7f8ef41353f0 auth_method 0 2024-09-11T01:49:14.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:14.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:14.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:14.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef3fff640 1 -- 172.21.15.12:0/2524893474 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8ef41353f0 msgr2=0x7f8ef4134160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:49:14.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef3fff640 1 -- 172.21.15.12:0/2524893474 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8ef41353f0 msgr2=0x7f8ef4134160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:49:14.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef3fff640 1 --2- 172.21.15.12:0/2524893474 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8ef41353f0 0x7f8ef4134160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:49:14.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef37fe640 1 --2- 172.21.15.12:0/2524893474 >> v2:172.21.15.71:3300/0 conn(0x7f8ef4134ad0 0x7f8ef4134eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:14.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef3fff640 1 --2- 172.21.15.12:0/2524893474 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8ef41353f0 0x7f8ef4134160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:14.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef37fe640 10 monclient(hunting): get_auth_request con 0x7f8ef4134ad0 auth_method 0 2024-09-11T01:49:14.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:14.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:14.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:14.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:14.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:14.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef37fe640 10 monclient(hunting): handle_auth_done global_id 134830 payload 274 2024-09-11T01:49:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef37fe640 1 -- 172.21.15.12:0/2524893474 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8ef41353f0 msgr2=0x7f8ef4134160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:49:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef37fe640 1 --2- 172.21.15.12:0/2524893474 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8ef41353f0 0x7f8ef4134160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef37fe640 1 -- 172.21.15.12:0/2524893474 >> v1:172.21.15.12:6789/0 conn(0x7f8ef40a2200 legacy=0x7f8ef4133a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:49:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef37fe640 10 monclient: _finish_hunting 0 2024-09-11T01:49:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef37fe640 1 monclient: found mon.noname-c 2024-09-11T01:49:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:49:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.428+0000 7f8ef37fe640 1 -- 172.21.15.12:0/2524893474 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ef409f2f0 con 0x7f8ef4134ad0 2024-09-11T01:49:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.431+0000 7f8ef37fe640 1 -- 172.21.15.12:0/2524893474 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ef4134ad0 msgr2=0x7f8ef4134eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.431+0000 7f8ef37fe640 1 --2- 172.21.15.12:0/2524893474 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ef4134ad0 0x7f8ef4134eb0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40097d0 tx=0x7f8ee402a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd3b7c31e3c2efdf server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.432+0000 7f8ef27fc640 1 -- 172.21.15.12:0/2524893474 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee4037070 con 0x7f8ef4134ad0 2024-09-11T01:49:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.432+0000 7f8ef27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:14.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.432+0000 7f8ef27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:49:14.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.432+0000 7f8ef27fc640 10 monclient: dump: 2024-09-11T01:49:14.440 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:14.440 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:14.440 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.440 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.440 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:14.440 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:14.440 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:14.441 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:14.441 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:14.441 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:14.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.432+0000 7f8ef27fc640 10 monclient: _finish_auth 0 2024-09-11T01:49:14.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.432+0000 7f8ef27fc640 10 monclient: _check_auth_tickets 2024-09-11T01:49:14.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.432+0000 7f8ef27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:14.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.432+0000 7f8ef9ba9640 5 monclient: authenticate success, global_id 134830 2024-09-11T01:49:14.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.432+0000 7f8ef27fc640 1 -- 172.21.15.12:0/2524893474 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ee4032440 con 0x7f8ef4134ad0 2024-09-11T01:49:14.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.432+0000 7f8ef27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:14.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.432+0000 7f8ef9ba9640 10 monclient: get_monmap_and_config success 2024-09-11T01:49:14.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.432+0000 7f8ef9ba9640 4 set_mon_vals no callback set 2024-09-11T01:49:14.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.432+0000 7f8ef27fc640 1 -- 172.21.15.12:0/2524893474 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee4032810 con 0x7f8ef4134ad0 2024-09-11T01:49:14.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.432+0000 7f8ef27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:14.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.432+0000 7f8ef27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:49:14.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.432+0000 7f8ef27fc640 10 monclient: dump: 2024-09-11T01:49:14.443 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:14.443 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:14.443 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.443 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.443 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:14.443 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:14.443 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:14.443 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:14.443 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:14.443 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:14.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.432+0000 7f8ef9ba9640 10 monclient: shutdown 2024-09-11T01:49:14.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.432+0000 7f8ef9ba9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:14.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.432+0000 7f8ef9ba9640 1 -- 172.21.15.12:0/2524893474 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ef4134ad0 msgr2=0x7f8ef4134eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:14.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.432+0000 7f8ef9ba9640 1 --2- 172.21.15.12:0/2524893474 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ef4134ad0 0x7f8ef4134eb0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40097d0 tx=0x7f8ee402a000 comp rx=0 tx=0).stop 2024-09-11T01:49:14.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.432+0000 7f8ef9ba9640 1 -- 172.21.15.12:0/2524893474 shutdown_connections 2024-09-11T01:49:14.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.432+0000 7f8ef9ba9640 1 --2- 172.21.15.12:0/2524893474 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8ef41353f0 0x7f8ef4134160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.432+0000 7f8ef9ba9640 1 --2- 172.21.15.12:0/2524893474 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ef4134ad0 0x7f8ef4134eb0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.432+0000 7f8ef9ba9640 1 -- 172.21.15.12:0/2524893474 >> 172.21.15.12:0/2524893474 conn(0x7f8ef40a2f70 msgr2=0x7f8ef40a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:14.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.432+0000 7f8ef9ba9640 1 -- 172.21.15.12:0/2524893474 shutdown_connections 2024-09-11T01:49:14.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.432+0000 7f8ef9ba9640 1 -- 172.21.15.12:0/2524893474 wait complete. 2024-09-11T01:49:14.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.433+0000 7f8ef9ba9640 10 monclient: build_initial_monmap 2024-09-11T01:49:14.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.433+0000 7f8ef9ba9640 10 monclient: monmap: 2024-09-11T01:49:14.444 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:14.444 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:49:14.445 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:49:14.445 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:49:14.445 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:14.445 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:14.445 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:49:14.445 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:49:14.445 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:49:14.445 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:14.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.433+0000 7f8ef9ba9640 1 Processor -- start 2024-09-11T01:49:14.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.433+0000 7f8ef9ba9640 1 -- start start 2024-09-11T01:49:14.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.433+0000 7f8ef9ba9640 10 monclient: init 2024-09-11T01:49:14.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef9ba9640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:14.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef9ba9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:49:14.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef9ba9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8ef40a2200 0x7f8ef413b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:14.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef9ba9640 10 monclient(hunting): picked mon.noname-c con 0x7f8ef40a2200 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:14.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef9ba9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef4134ad0 0x7f8ef413d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:14.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef9ba9640 10 monclient(hunting): picked mon.noname-a con 0x7f8ef4134ad0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:49:14.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef9ba9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ef41353f0 0x7f8ef413d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:14.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef9ba9640 10 monclient(hunting): picked mon.noname-b con 0x7f8ef41353f0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:49:14.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef9ba9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:14.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef9ba9640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef4134820 con 0x7f8ef4134ad0 2024-09-11T01:49:14.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef9ba9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:14.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef9ba9640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef41346a0 con 0x7f8ef41353f0 2024-09-11T01:49:14.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef9ba9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:14.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef9ba9640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8ef409ffd0 con 0x7f8ef40a2200 2024-09-11T01:49:14.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef9ba9640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:14.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef9ba9640 10 monclient(hunting): authenticate will time out at 1960.876292s 2024-09-11T01:49:14.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef37fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8ef40a2200 0x7f8ef413b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:14.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef37fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8ef40a2200 0x7f8ef413b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52576/0 (socket says 172.21.15.12:52576) 2024-09-11T01:49:14.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef37fe640 1 -- 172.21.15.12:0/2950328002 learned_addr learned my addr 172.21.15.12:0/2950328002 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:49:14.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef37fe640 10 monclient(hunting): get_auth_request con 0x7f8ef40a2200 auth_method 0 2024-09-11T01:49:14.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:14.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:14.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:14.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef2ffd640 1 --2- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef4134ad0 0x7f8ef413d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:14.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef2ffd640 10 monclient(hunting): get_auth_request con 0x7f8ef4134ad0 auth_method 0 2024-09-11T01:49:14.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:14.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:14.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:14.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef37fe640 1 -- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8ef40a2200 msgr2=0x7f8ef413b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:49:14.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef37fe640 1 -- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8ef40a2200 msgr2=0x7f8ef413b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:49:14.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef37fe640 1 --2- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8ef40a2200 0x7f8ef413b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:49:14.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef3fff640 1 --2- 172.21.15.12:0/2950328002 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ef41353f0 0x7f8ef413d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:14.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef37fe640 1 --2- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8ef40a2200 0x7f8ef413b8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:14.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef3fff640 10 monclient(hunting): get_auth_request con 0x7f8ef41353f0 auth_method 0 2024-09-11T01:49:14.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:14.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:14.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:14.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:14.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:14.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:14.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:14.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:14.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:14.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef2ffd640 10 monclient(hunting): handle_auth_done global_id 214197 payload 824 2024-09-11T01:49:14.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef2ffd640 1 -- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8ef40a2200 msgr2=0x7f8ef413b8b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:49:14.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef2ffd640 1 --2- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8ef40a2200 0x7f8ef413b8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef2ffd640 1 -- 172.21.15.12:0/2950328002 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ef41353f0 msgr2=0x7f8ef413d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:14.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef2ffd640 1 --2- 172.21.15.12:0/2950328002 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ef41353f0 0x7f8ef413d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef2ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:49:14.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef2ffd640 1 monclient: found mon.noname-a 2024-09-11T01:49:14.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:14.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:49:14.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.434+0000 7f8ef2ffd640 1 -- 172.21.15.12:0/2950328002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ef413a0e0 con 0x7f8ef4134ad0 2024-09-11T01:49:14.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.437+0000 7f8ef2ffd640 1 -- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef4134ad0 msgr2=0x7f8ef413d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:14.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.437+0000 7f8ef2ffd640 1 --2- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef4134ad0 0x7f8ef413d050 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8ee800b410 tx=0x7f8ee800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:14.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.437+0000 7f8edbfff640 1 -- 172.21.15.12:0/2950328002 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee8015020 con 0x7f8ef4134ad0 2024-09-11T01:49:14.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.437+0000 7f8edbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:14.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.437+0000 7f8edbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:49:14.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.437+0000 7f8edbfff640 10 monclient: dump: 2024-09-11T01:49:14.461 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:14.461 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:14.461 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.461 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.462 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:14.462 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:14.462 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:14.462 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:14.462 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:14.462 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:14.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.437+0000 7f8edbfff640 10 monclient: _finish_auth 0 2024-09-11T01:49:14.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.437+0000 7f8edbfff640 10 monclient: _check_auth_tickets 2024-09-11T01:49:14.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.437+0000 7f8edbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:14.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.437+0000 7f8edbfff640 1 -- 172.21.15.12:0/2950328002 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ee8011a60 con 0x7f8ef4134ad0 2024-09-11T01:49:14.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.437+0000 7f8edbfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:14.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.437+0000 7f8edbfff640 1 -- 172.21.15.12:0/2950328002 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee8011e50 con 0x7f8ef4134ad0 2024-09-11T01:49:14.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.437+0000 7f8edbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:14.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.437+0000 7f8edbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:14.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.437+0000 7f8edbfff640 10 monclient: dump: 2024-09-11T01:49:14.464 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:14.464 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:14.465 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.465 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.465 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:14.465 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:14.465 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:14.466 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:14.466 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:14.466 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:14.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.437+0000 7f8ef9ba9640 5 monclient: authenticate success, global_id 214197 2024-09-11T01:49:14.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.437+0000 7f8ef9ba9640 10 monclient: _renew_subs 2024-09-11T01:49:14.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.437+0000 7f8ef9ba9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:14.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.437+0000 7f8ef9ba9640 1 -- 172.21.15.12:0/2950328002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ef413a3c0 con 0x7f8ef4134ad0 2024-09-11T01:49:14.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.437+0000 7f8ef1ffb640 4 set_mon_vals no callback set 2024-09-11T01:49:14.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.437+0000 7f8ef9ba9640 10 monclient: _renew_subs 2024-09-11T01:49:14.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.437+0000 7f8ef9ba9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:14.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.437+0000 7f8ef9ba9640 1 -- 172.21.15.12:0/2950328002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ef413eb10 con 0x7f8ef4134ad0 2024-09-11T01:49:14.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.438+0000 7f8ef9ba9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:14.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.438+0000 7f8ef9ba9640 1 --2- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef413ab30 0x7f8ef402e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:14.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.438+0000 7f8ef9ba9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:14.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.438+0000 7f8ef9ba9640 1 -- 172.21.15.12:0/2950328002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef4134eb0 con 0x7f8ef413ab30 2024-09-11T01:49:14.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.438+0000 7f8ef37fe640 1 --2- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef413ab30 0x7f8ef402e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:14.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.438+0000 7f8ef37fe640 10 monclient: get_auth_request con 0x7f8ef413ab30 auth_method 0 2024-09-11T01:49:14.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.438+0000 7f8ef37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:14.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.438+0000 7f8ef37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:14.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.438+0000 7f8ef37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:14.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.438+0000 7f8ef37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:14.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.438+0000 7f8ef37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:14.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.438+0000 7f8ef37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:14.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.438+0000 7f8edbfff640 1 -- 172.21.15.12:0/2950328002 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f8ee801a410 con 0x7f8ef4134ad0 2024-09-11T01:49:14.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.438+0000 7f8ef37fe640 10 monclient(hunting): handle_auth_done global_id 214200 payload 824 2024-09-11T01:49:14.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.438+0000 7f8ef37fe640 1 -- 172.21.15.12:0/2950328002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8ef4072db0 con 0x7f8ef413ab30 2024-09-11T01:49:14.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.438+0000 7f8edbfff640 1 --2- 172.21.15.12:0/2950328002 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8ee003e3b0 0x7f8ee0040870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:14.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.438+0000 7f8edbfff640 1 -- 172.21.15.12:0/2950328002 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(671..671 src has 507..671) ==== 4843+0+0 (secure 0 0 0) 0x7f8ee8052940 con 0x7f8ef4134ad0 2024-09-11T01:49:14.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.438+0000 7f8ef3fff640 1 --2- 172.21.15.12:0/2950328002 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8ee003e3b0 0x7f8ee0040870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:14.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.439+0000 7f8ef3fff640 10 monclient: get_auth_request con 0x7f8ee003e3b0 auth_method 0 2024-09-11T01:49:14.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.440+0000 7f8ef37fe640 1 -- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef413ab30 msgr2=0x7f8ef402e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:14.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.440+0000 7f8ef37fe640 1 --2- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef413ab30 0x7f8ef402e5c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8ee402a040 tx=0x7f8ee40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:14.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.440+0000 7f8edbfff640 1 -- 172.21.15.12:0/2950328002 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee4043070 con 0x7f8ef413ab30 2024-09-11T01:49:14.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.440+0000 7f8edbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:14.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.440+0000 7f8edbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:14.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.440+0000 7f8edbfff640 10 monclient: dump: 2024-09-11T01:49:14.469 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:14.469 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:14.469 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.470 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.470 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:14.470 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:14.470 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:14.470 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:14.470 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:14.470 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:14.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.441+0000 7f8edbfff640 1 -- 172.21.15.12:0/2950328002 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8ee403c850 con 0x7f8ef413ab30 2024-09-11T01:49:14.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.441+0000 7f8edbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:14.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.441+0000 7f8edbfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:49:14.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.441+0000 7f8edbfff640 1 -- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef413ab30 msgr2=0x7f8ef402e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:14.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.441+0000 7f8edbfff640 1 --2- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef413ab30 0x7f8ef402e5c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8ee402a040 tx=0x7f8ee40046c0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.441+0000 7f8edbfff640 1 -- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef413ab30 msgr2=0x7f8ef402e5c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:49:14.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.441+0000 7f8edbfff640 1 --2- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef413ab30 0x7f8ef402e5c0 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8ee402a040 tx=0x7f8ee40046c0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.441+0000 7f8ef3fff640 1 --2- 172.21.15.12:0/2950328002 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8ee003e3b0 0x7f8ee0040870 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f8eec009610 tx=0x7f8eec0097d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:14.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.447+0000 7f8ef9ba9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:49:14.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.447+0000 7f8ef9ba9640 1 --2- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef413bdf0 0x7f8ef413dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:14.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.447+0000 7f8ef9ba9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:14.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.447+0000 7f8ef9ba9640 1 -- 172.21.15.12:0/2950328002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef4139c70 con 0x7f8ef413bdf0 2024-09-11T01:49:14.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.447+0000 7f8ef37fe640 1 --2- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef413bdf0 0x7f8ef413dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:14.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.447+0000 7f8ef37fe640 10 monclient: get_auth_request con 0x7f8ef413bdf0 auth_method 0 2024-09-11T01:49:14.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.447+0000 7f8ef37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:14.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.447+0000 7f8ef37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:14.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.447+0000 7f8ef37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:14.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.447+0000 7f8ef37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:14.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.447+0000 7f8ef37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:14.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.447+0000 7f8ef37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:14.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.447+0000 7f8ef37fe640 10 monclient(hunting): handle_auth_done global_id 214203 payload 824 2024-09-11T01:49:14.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.447+0000 7f8ef37fe640 1 -- 172.21.15.12:0/2950328002 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8ef4138bc0 con 0x7f8ef413bdf0 2024-09-11T01:49:14.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.450+0000 7f8ef37fe640 1 -- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef413bdf0 msgr2=0x7f8ef413dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:14.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.450+0000 7f8ef37fe640 1 --2- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef413bdf0 0x7f8ef413dad0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8ef409d660 tx=0x7f8ee4009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:14.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.450+0000 7f8edbfff640 1 -- 172.21.15.12:0/2950328002 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee4037040 con 0x7f8ef413bdf0 2024-09-11T01:49:14.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.450+0000 7f8edbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:14.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.450+0000 7f8edbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:14.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.450+0000 7f8edbfff640 10 monclient: dump: 2024-09-11T01:49:14.474 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:14.474 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:14.474 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.474 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.474 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:14.475 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:14.475 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:14.475 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:14.475 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:14.475 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:14.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.450+0000 7f8edbfff640 1 -- 172.21.15.12:0/2950328002 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7f8ee403b3a0 con 0x7f8ef413bdf0 2024-09-11T01:49:14.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.450+0000 7f8edbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:49:14.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.450+0000 7f8edbfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:49:14.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.450+0000 7f8edbfff640 1 -- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef413bdf0 msgr2=0x7f8ef413dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:14.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.450+0000 7f8edbfff640 1 --2- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef413bdf0 0x7f8ef413dad0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8ef409d660 tx=0x7f8ee4009190 comp rx=0 tx=0).stop 2024-09-11T01:49:14.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.450+0000 7f8edbfff640 1 -- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef413bdf0 msgr2=0x7f8ef413dad0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:49:14.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.450+0000 7f8edbfff640 1 --2- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef413bdf0 0x7f8ef413dad0 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8ef409d660 tx=0x7f8ee4009190 comp rx=0 tx=0).stop 2024-09-11T01:49:14.476 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:49:14.477 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:49:14.477 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:49:14.477 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:49:14.477 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 188, 2024-09-11T01:49:14.477 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 60650, 2024-09-11T01:49:14.477 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:49:14.477 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:49:14.477 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:49:14.477 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:49:14.477 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:14.477 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:49:14.477 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:49:14.477 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:49:14.478 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:49:14.478 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:49:14.478 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:49:14.478 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:49:14.478 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:49:14.478 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:49:14.478 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:49:14.478 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:49:14.478 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:49:14.478 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:49:14.478 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:49:14.478 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:49:14.479 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:14.479 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:49:14.479 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:49:14.479 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:49:14.479 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:49:14.479 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:49:14.479 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:49:14.479 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:49:14.479 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:49:14.479 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:49:14.479 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:49:14.479 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:49:14.479 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:49:14.479 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:49:14.480 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:14.480 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:14.480 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:49:14.480 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:49:14.480 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:49:14.480 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:49:14.480 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:49:14.480 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:49:14.480 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:49:14.480 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:49:14.480 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:49:14.480 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:49:14.480 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:49:14.481 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:49:14.481 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:49:14.481 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:49:14.481 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:49:14.481 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:49:14.481 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:49:14.481 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:49:14.481 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:49:14.481 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:49:14.481 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:49:14.481 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:49:14.481 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:49:14.481 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:49:14.482 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:49:14.482 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:49:14.482 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:49:14.482 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:49:14.482 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:14.482 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:49:14.482 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:14.482 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:49:14.482 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:14.482 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:49:14.482 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:49:14.482 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:49:14.482 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:49:14.483 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:14.483 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:49:14.483 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:49:14.483 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:14.483 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:14.483 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:14.483 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:49:14.483 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:49:14.483 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:14.483 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:14.483 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:14.483 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:14.483 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:49:14.483 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:49:14.484 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:49:14.484 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:49:14.484 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:49:14.484 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:14.484 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:14.484 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:49:14.484 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:49:14.484 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:49:14.484 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:49:14.484 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:14.484 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:49:14.484 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:49:14.484 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:14.484 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:14.485 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:14.485 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:49:14.485 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:49:14.485 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:14.485 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:14.485 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:14.485 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:14.485 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:49:14.485 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:49:14.486 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:49:14.486 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:49:14.486 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:49:14.486 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:14.486 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:14.486 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:49:14.486 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:49:14.486 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:49:14.486 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:49:14.486 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:14.486 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:49:14.486 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:49:14.486 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:14.487 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:14.487 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:14.487 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:49:14.487 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:49:14.487 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:14.487 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:14.487 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:14.487 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:14.487 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:49:14.487 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:49:14.487 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:49:14.487 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:49:14.487 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:49:14.487 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:14.487 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:14.488 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:14.488 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:49:14.488 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:49:14.488 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:14.488 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:49:14.488 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:49:14.488 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:49:14.488 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:14.488 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:14.488 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:49:14.488 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:14.488 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:49:14.488 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:49:14.489 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:49:14.489 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:14.489 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:14.489 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:14.489 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:49:14.489 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:49:14.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.450+0000 7f8ef9ba9640 1 -- 172.21.15.12:0/2950328002 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8ee003e3b0 msgr2=0x7f8ee0040870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:14.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.450+0000 7f8ef9ba9640 1 --2- 172.21.15.12:0/2950328002 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8ee003e3b0 0x7f8ee0040870 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f8eec009610 tx=0x7f8eec0097d0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.450+0000 7f8ef9ba9640 10 monclient: shutdown 2024-09-11T01:49:14.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.450+0000 7f8ef9ba9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:14.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.450+0000 7f8ef9ba9640 1 -- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef4134ad0 msgr2=0x7f8ef413d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:14.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.450+0000 7f8ef9ba9640 1 --2- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef4134ad0 0x7f8ef413d050 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8ee800b410 tx=0x7f8ee800b8d0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.450+0000 7f8ef3fff640 1 -- 172.21.15.12:0/2950328002 reap_dead start 2024-09-11T01:49:14.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.450+0000 7f8ef9ba9640 1 -- 172.21.15.12:0/2950328002 shutdown_connections 2024-09-11T01:49:14.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.450+0000 7f8ef9ba9640 1 --2- 172.21.15.12:0/2950328002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ef4134ad0 0x7f8ef413d050 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.450+0000 7f8ef9ba9640 1 -- 172.21.15.12:0/2950328002 >> 172.21.15.12:0/2950328002 conn(0x7f8ef40a2f70 msgr2=0x7f8ef40a4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:14.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.450+0000 7f8ef9ba9640 1 -- 172.21.15.12:0/2950328002 shutdown_connections 2024-09-11T01:49:14.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.450+0000 7f8ef9ba9640 1 -- 172.21.15.12:0/2950328002 wait complete. 2024-09-11T01:49:14.493 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:49:14.493 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:49:14.493 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:49:14.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.594+0000 7f2dea260640 10 monclient: get_monmap_and_config 2024-09-11T01:49:14.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.594+0000 7f2dea260640 10 monclient: build_initial_monmap 2024-09-11T01:49:14.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.594+0000 7f2dea260640 10 monclient: monmap: 2024-09-11T01:49:14.596 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:14.596 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:14.596 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:49:14.595038+0000 2024-09-11T01:49:14.596 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:49:14.595038+0000 2024-09-11T01:49:14.597 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:14.597 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:14.597 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:49:14.597 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:49:14.597 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:49:14.597 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:49:14.597 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:49:14.597 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:14.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2dea260640 1 Processor -- start 2024-09-11T01:49:14.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2dea260640 1 -- start start 2024-09-11T01:49:14.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2dea260640 10 monclient: init 2024-09-11T01:49:14.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2dea260640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:14.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2dea260640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-11T01:49:14.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2dea260640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2de4135fc0 0x7f2de41363a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:14.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2dea260640 10 monclient(hunting): picked mon.noname-a con 0x7f2de4135fc0 addr v2:172.21.15.12:3300/0 2024-09-11T01:49:14.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2dea260640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de4074000 0x7f2de4074410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:14.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2dea260640 10 monclient(hunting): picked mon.noname-b con 0x7f2de4074000 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:14.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2dea260640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2de40a2e20 addr v1:172.21.15.71:6789/0 2024-09-11T01:49:14.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2dea260640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2de40732d0 con 0x7f2de40a2e20 2024-09-11T01:49:14.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2dea260640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:14.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2dea260640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f2de4073890 con 0x7f2de4135fc0 2024-09-11T01:49:14.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2dea260640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:14.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2dea260640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2de4072000 con 0x7f2de4074000 2024-09-11T01:49:14.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2dea260640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:14.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2dea260640 10 monclient(hunting): authenticate will time out at 1961.037630s 2024-09-11T01:49:14.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2de2ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de4074000 0x7f2de4074410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:14.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2de37fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2de4135fc0 0x7f2de41363a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:14.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2de2ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de4074000 0x7f2de4074410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52584/0 (socket says 172.21.15.12:52584) 2024-09-11T01:49:14.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2de37fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2de4135fc0 0x7f2de41363a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34608/0 (socket says 172.21.15.12:34608) 2024-09-11T01:49:14.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2de2ffd640 1 -- 172.21.15.12:0/583846597 learned_addr learned my addr 172.21.15.12:0/583846597 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:49:14.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2de2ffd640 10 monclient(hunting): get_auth_request con 0x7f2de4074000 auth_method 0 2024-09-11T01:49:14.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2de2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:14.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2de2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:14.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2de2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:14.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2de37fe640 10 monclient(hunting): get_auth_request con 0x7f2de4135fc0 auth_method 0 2024-09-11T01:49:14.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2de37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:14.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2de37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:14.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2de37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:14.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2de2ffd640 1 -- 172.21.15.12:0/583846597 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de4074000 msgr2=0x7f2de4074410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:49:14.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2de2ffd640 1 -- 172.21.15.12:0/583846597 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de4074000 msgr2=0x7f2de4074410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:49:14.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2de37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:14.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2de37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:14.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2de2ffd640 1 --2- 172.21.15.12:0/583846597 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de4074000 0x7f2de4074410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:49:14.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2de37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:14.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.595+0000 7f2de2ffd640 1 --2- 172.21.15.12:0/583846597 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de4074000 0x7f2de4074410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:14.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.596+0000 7f2de37fe640 10 monclient(hunting): handle_auth_done global_id 214206 payload 274 2024-09-11T01:49:14.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.596+0000 7f2de37fe640 1 -- 172.21.15.12:0/583846597 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de4074000 msgr2=0x7f2de4074410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:49:14.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.596+0000 7f2de37fe640 1 --2- 172.21.15.12:0/583846597 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de4074000 0x7f2de4074410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.596+0000 7f2de37fe640 1 -- 172.21.15.12:0/583846597 >> v1:172.21.15.71:6789/0 conn(0x7f2de40a2e20 legacy=0x7f2de40a3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:49:14.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.596+0000 7f2de37fe640 10 monclient: _finish_hunting 0 2024-09-11T01:49:14.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.596+0000 7f2de37fe640 1 monclient: found mon.noname-a 2024-09-11T01:49:14.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.596+0000 7f2de37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:14.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.596+0000 7f2de37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:49:14.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.596+0000 7f2de37fe640 1 -- 172.21.15.12:0/583846597 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de40a4970 con 0x7f2de4135fc0 2024-09-11T01:49:14.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.598+0000 7f2de37fe640 1 -- 172.21.15.12:0/583846597 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2de4135fc0 msgr2=0x7f2de41363a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:14.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.598+0000 7f2de37fe640 1 --2- 172.21.15.12:0/583846597 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2de4135fc0 0x7f2de41363a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2dd40097a0 tx=0x7f2dd4029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=149335dd471f47f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:14.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.599+0000 7f2de27fc640 1 -- 172.21.15.12:0/583846597 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4037070 con 0x7f2de4135fc0 2024-09-11T01:49:14.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.599+0000 7f2de27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:14.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.599+0000 7f2de27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:49:14.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.599+0000 7f2de27fc640 10 monclient: dump: 2024-09-11T01:49:14.615 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:14.616 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:14.616 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.616 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.616 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:14.616 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:14.616 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:14.616 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:14.616 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:14.616 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:14.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.599+0000 7f2de27fc640 10 monclient: _finish_auth 0 2024-09-11T01:49:14.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.599+0000 7f2de27fc640 10 monclient: _check_auth_tickets 2024-09-11T01:49:14.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.599+0000 7f2de27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:14.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.599+0000 7f2de27fc640 1 -- 172.21.15.12:0/583846597 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dd4032440 con 0x7f2de4135fc0 2024-09-11T01:49:14.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.599+0000 7f2dea260640 5 monclient: authenticate success, global_id 214206 2024-09-11T01:49:14.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.599+0000 7f2de27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:14.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.599+0000 7f2de27fc640 1 -- 172.21.15.12:0/583846597 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4032810 con 0x7f2de4135fc0 2024-09-11T01:49:14.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.599+0000 7f2de27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:14.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.599+0000 7f2de27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:14.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.599+0000 7f2de27fc640 10 monclient: dump: 2024-09-11T01:49:14.618 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:14.618 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:14.618 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.619 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.619 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:14.619 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:14.619 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:14.619 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:14.619 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:14.619 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:14.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.599+0000 7f2dea260640 10 monclient: get_monmap_and_config success 2024-09-11T01:49:14.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.599+0000 7f2dea260640 4 set_mon_vals no callback set 2024-09-11T01:49:14.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.599+0000 7f2dea260640 10 monclient: shutdown 2024-09-11T01:49:14.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.599+0000 7f2dea260640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:14.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.599+0000 7f2dea260640 1 -- 172.21.15.12:0/583846597 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2de4135fc0 msgr2=0x7f2de41363a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:14.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.599+0000 7f2dea260640 1 --2- 172.21.15.12:0/583846597 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2de4135fc0 0x7f2de41363a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2dd40097a0 tx=0x7f2dd4029fd0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.599+0000 7f2dea260640 1 -- 172.21.15.12:0/583846597 shutdown_connections 2024-09-11T01:49:14.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.599+0000 7f2dea260640 1 --2- 172.21.15.12:0/583846597 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de4074000 0x7f2de4074410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.599+0000 7f2dea260640 1 --2- 172.21.15.12:0/583846597 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2de4135fc0 0x7f2de41363a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.599+0000 7f2dea260640 1 -- 172.21.15.12:0/583846597 >> 172.21.15.12:0/583846597 conn(0x7f2de40a8600 msgr2=0x7f2de40a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:14.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.599+0000 7f2dea260640 1 -- 172.21.15.12:0/583846597 shutdown_connections 2024-09-11T01:49:14.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.599+0000 7f2dea260640 1 -- 172.21.15.12:0/583846597 wait complete. 2024-09-11T01:49:14.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.600+0000 7f2dea260640 10 monclient: build_initial_monmap 2024-09-11T01:49:14.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.600+0000 7f2dea260640 10 monclient: monmap: 2024-09-11T01:49:14.622 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:14.622 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:49:14.623 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:49:14.623 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:49:14.623 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:14.623 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:14.623 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:49:14.623 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:49:14.623 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:49:14.624 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:14.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.600+0000 7f2dea260640 1 Processor -- start 2024-09-11T01:49:14.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.600+0000 7f2dea260640 1 -- start start 2024-09-11T01:49:14.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.600+0000 7f2dea260640 10 monclient: init 2024-09-11T01:49:14.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2dea260640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:14.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2dea260640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:49:14.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2dea260640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2de4074000 0x7f2de411eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:14.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2dea260640 10 monclient(hunting): picked mon.noname-a con 0x7f2de4074000 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:49:14.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2dea260640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de40a2e20 0x7f2de4134d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:14.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2dea260640 10 monclient(hunting): picked mon.noname-c con 0x7f2de40a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:14.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2dea260640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2de4135fc0 0x7f2de4135270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:14.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2dea260640 10 monclient(hunting): picked mon.noname-b con 0x7f2de4135fc0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:49:14.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2dea260640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:14.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2dea260640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2de40a3ac0 con 0x7f2de4074000 2024-09-11T01:49:14.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2dea260640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:14.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2dea260640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f2de40a3940 con 0x7f2de4135fc0 2024-09-11T01:49:14.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2dea260640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:14.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2dea260640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2de40a3c40 con 0x7f2de40a2e20 2024-09-11T01:49:14.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2dea260640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:14.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2dea260640 10 monclient(hunting): authenticate will time out at 1961.043218s 2024-09-11T01:49:14.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de2ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de40a2e20 0x7f2de4134d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:14.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2de4074000 0x7f2de411eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:14.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2de4074000 0x7f2de411eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34612/0 (socket says 172.21.15.12:34612) 2024-09-11T01:49:14.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de2ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de40a2e20 0x7f2de4134d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52586/0 (socket says 172.21.15.12:52586) 2024-09-11T01:49:14.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de37fe640 1 -- 172.21.15.12:0/607852556 learned_addr learned my addr 172.21.15.12:0/607852556 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:49:14.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de37fe640 10 monclient(hunting): get_auth_request con 0x7f2de4074000 auth_method 0 2024-09-11T01:49:14.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:14.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:14.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:14.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de2ffd640 10 monclient(hunting): get_auth_request con 0x7f2de40a2e20 auth_method 0 2024-09-11T01:49:14.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:14.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:14.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:14.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:14.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:14.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:14.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de2ffd640 1 -- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de40a2e20 msgr2=0x7f2de4134d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:49:14.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de2ffd640 1 -- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de40a2e20 msgr2=0x7f2de4134d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:49:14.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de2ffd640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de40a2e20 0x7f2de4134d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:49:14.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de2ffd640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de40a2e20 0x7f2de4134d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:14.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de3fff640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2de4135fc0 0x7f2de4135270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:14.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de3fff640 10 monclient(hunting): get_auth_request con 0x7f2de4135fc0 auth_method 0 2024-09-11T01:49:14.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:14.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:14.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:14.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de37fe640 10 monclient(hunting): handle_auth_done global_id 214209 payload 824 2024-09-11T01:49:14.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de37fe640 1 -- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de40a2e20 msgr2=0x7f2de4134d30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:49:14.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de37fe640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de40a2e20 0x7f2de4134d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de37fe640 1 -- 172.21.15.12:0/607852556 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2de4135fc0 msgr2=0x7f2de4135270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:14.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de37fe640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2de4135fc0 0x7f2de4135270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:14.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de37fe640 10 monclient: _finish_hunting 0 2024-09-11T01:49:14.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de37fe640 1 monclient: found mon.noname-a 2024-09-11T01:49:14.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:14.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:49:14.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.601+0000 7f2de37fe640 1 -- 172.21.15.12:0/607852556 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de40a2350 con 0x7f2de4074000 2024-09-11T01:49:14.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.603+0000 7f2de37fe640 1 -- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2de4074000 msgr2=0x7f2de411eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:14.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.603+0000 7f2de37fe640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2de4074000 0x7f2de411eb70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2dd40098c0 tx=0x7f2dd40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:14.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.604+0000 7f2dcbfff640 1 -- 172.21.15.12:0/607852556 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4041020 con 0x7f2de4074000 2024-09-11T01:49:14.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.604+0000 7f2dcbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:14.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.604+0000 7f2dcbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:49:14.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.604+0000 7f2dcbfff640 10 monclient: dump: 2024-09-11T01:49:14.635 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:14.635 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:14.635 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.635 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.636 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:14.636 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:14.636 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:14.636 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:14.636 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:14.636 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:14.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.604+0000 7f2dcbfff640 10 monclient: _finish_auth 0 2024-09-11T01:49:14.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.604+0000 7f2dcbfff640 10 monclient: _check_auth_tickets 2024-09-11T01:49:14.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.604+0000 7f2dcbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:14.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.604+0000 7f2dcbfff640 1 -- 172.21.15.12:0/607852556 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dd403d8f0 con 0x7f2de4074000 2024-09-11T01:49:14.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.604+0000 7f2dcbfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:14.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.604+0000 7f2dcbfff640 1 -- 172.21.15.12:0/607852556 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4037040 con 0x7f2de4074000 2024-09-11T01:49:14.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.604+0000 7f2dea260640 5 monclient: authenticate success, global_id 214209 2024-09-11T01:49:14.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.604+0000 7f2dea260640 10 monclient: _renew_subs 2024-09-11T01:49:14.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.604+0000 7f2dea260640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:14.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.604+0000 7f2dea260640 1 -- 172.21.15.12:0/607852556 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2de4120e40 con 0x7f2de4074000 2024-09-11T01:49:14.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.604+0000 7f2dcbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:14.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.604+0000 7f2dcbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:14.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.604+0000 7f2dcbfff640 10 monclient: dump: 2024-09-11T01:49:14.639 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:14.639 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:14.639 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.639 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:14.640 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:14.640 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:14.640 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:14.640 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:14.640 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:14.641 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:14.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.604+0000 7f2de1ffb640 4 set_mon_vals no callback set 2024-09-11T01:49:14.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.604+0000 7f2dea260640 10 monclient: _renew_subs 2024-09-11T01:49:14.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.604+0000 7f2dea260640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:14.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.604+0000 7f2dea260640 1 -- 172.21.15.12:0/607852556 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2de40a1b10 con 0x7f2de4074000 2024-09-11T01:49:14.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.604+0000 7f2dcbfff640 1 -- 172.21.15.12:0/607852556 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f2dd403a030 con 0x7f2de4074000 2024-09-11T01:49:14.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.605+0000 7f2dea260640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:14.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.605+0000 7f2dea260640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 0x7f2de402d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:14.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.605+0000 7f2dea260640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:14.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.605+0000 7f2dea260640 1 -- 172.21.15.12:0/607852556 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2de406a060 con 0x7f2de41cd5d0 2024-09-11T01:49:14.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.605+0000 7f2dcbfff640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2dd003e210 0x7f2dd00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:14.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.605+0000 7f2de2ffd640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 0x7f2de402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:14.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.605+0000 7f2dcbfff640 1 -- 172.21.15.12:0/607852556 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(671..671 src has 507..671) ==== 4843+0+0 (secure 0 0 0) 0x7f2dd4073480 con 0x7f2de4074000 2024-09-11T01:49:14.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.605+0000 7f2de2ffd640 10 monclient: get_auth_request con 0x7f2de41cd5d0 auth_method 0 2024-09-11T01:49:14.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.605+0000 7f2de2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:14.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.605+0000 7f2de2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:14.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.605+0000 7f2de2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:14.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.605+0000 7f2de2ffd640 1 -- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 msgr2=0x7f2de402d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:49:14.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.605+0000 7f2de2ffd640 1 -- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 msgr2=0x7f2de402d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:49:14.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.605+0000 7f2de2ffd640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 0x7f2de402d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:49:14.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.605+0000 7f2de2ffd640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 0x7f2de402d440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:14.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.605+0000 7f2de3fff640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2dd003e210 0x7f2dd00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:14.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.605+0000 7f2de3fff640 10 monclient: get_auth_request con 0x7f2dd003e210 auth_method 0 2024-09-11T01:49:14.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.608+0000 7f2de3fff640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2dd003e210 0x7f2dd00406d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc005520 tx=0x7f2ddc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:14.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.807+0000 7f2de2ffd640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 0x7f2de402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:14.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.807+0000 7f2de2ffd640 10 monclient: get_auth_request con 0x7f2de41cd5d0 auth_method 0 2024-09-11T01:49:14.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.807+0000 7f2de2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:14.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.807+0000 7f2de2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:14.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.807+0000 7f2de2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:49:14.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.807+0000 7f2de2ffd640 1 -- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 msgr2=0x7f2de402d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:49:14.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.807+0000 7f2de2ffd640 1 -- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 msgr2=0x7f2de402d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:49:14.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.807+0000 7f2de2ffd640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 0x7f2de402d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:49:14.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:14.807+0000 7f2de2ffd640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 0x7f2de402d440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:49:14.831 INFO:tasks.rados.rados.0.smithi071.stdout:2976: done (1 left) 2024-09-11T01:49:14.831 INFO:tasks.rados.rados.0.smithi071.stdout:2977: snap_remove snap 294 2024-09-11T01:49:15.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:15.208+0000 7f2de2ffd640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 0x7f2de402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:15.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:15.208+0000 7f2de2ffd640 10 monclient: get_auth_request con 0x7f2de41cd5d0 auth_method 0 2024-09-11T01:49:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:15.208+0000 7f2de2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:15.208+0000 7f2de2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:15.208+0000 7f2de2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:49:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:15.208+0000 7f2de2ffd640 1 -- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 msgr2=0x7f2de402d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:49:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:15.208+0000 7f2de2ffd640 1 -- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 msgr2=0x7f2de402d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:49:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:15.208+0000 7f2de2ffd640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 0x7f2de402d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:49:15.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:15.208+0000 7f2de2ffd640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 0x7f2de402d440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:49:15.834 INFO:tasks.rados.rados.0.smithi071.stdout:2977: done (1 left) 2024-09-11T01:49:15.834 INFO:tasks.rados.rados.0.smithi071.stdout:2978: copy_from oid 31 from oid 23 current snap is 339 2024-09-11T01:49:15.835 INFO:tasks.rados.rados.0.smithi071.stdout:2975: finishing write tid 1 to smithi07134777-12 2024-09-11T01:49:15.835 INFO:tasks.rados.rados.0.smithi071.stdout:2975: finishing write tid 2 to smithi07134777-12 2024-09-11T01:49:15.835 INFO:tasks.rados.rados.0.smithi071.stdout:2975: finishing write tid 3 to smithi07134777-12 2024-09-11T01:49:15.835 INFO:tasks.rados.rados.0.smithi071.stdout:2975: finishing write tid 4 to smithi07134777-12 2024-09-11T01:49:15.835 INFO:tasks.rados.rados.0.smithi071.stdout:2975: finishing write tid 5 to smithi07134777-12 2024-09-11T01:49:15.835 INFO:tasks.rados.rados.0.smithi071.stdout:2975: oid 12 updating version 0 to 1934 2024-09-11T01:49:15.835 INFO:tasks.rados.rados.0.smithi071.stdout:2975: oid 12 version 1934 is already newer than 1933 2024-09-11T01:49:15.835 INFO:tasks.rados.rados.0.smithi071.stdout:2975: oid 12 updating version 1934 to 1936 2024-09-11T01:49:15.835 INFO:tasks.rados.rados.0.smithi071.stdout:2975: oid 12 version 1936 is already newer than 1935 2024-09-11T01:49:15.835 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 1936 (ObjNum 1049 snap 339 seq_num 1049) dirty exists 2024-09-11T01:49:15.835 INFO:tasks.rados.rados.0.smithi071.stdout:2975: left oid 12 (ObjNum 1049 snap 339 seq_num 1049) 2024-09-11T01:49:15.835 INFO:tasks.rados.rados.0.smithi071.stdout:2975: done (1 left) 2024-09-11T01:49:15.835 INFO:tasks.rados.rados.0.smithi071.stdout:2979: write oid 4 current snap is 339 2024-09-11T01:49:15.835 INFO:tasks.rados.rados.0.smithi071.stdout:2979: seq_num 1050 ranges {447350=668164,1560093=406693,2584613=480437} 2024-09-11T01:49:15.841 INFO:tasks.rados.rados.0.smithi071.stdout:2979: writing smithi07134777-4 from 447350 to 1115514 tid 1 2024-09-11T01:49:15.846 INFO:tasks.rados.rados.0.smithi071.stdout:2979: writing smithi07134777-4 from 1560093 to 1966786 tid 2 2024-09-11T01:49:15.853 INFO:tasks.rados.rados.0.smithi071.stdout:2979: writing smithi07134777-4 from 2584613 to 3065050 tid 3 2024-09-11T01:49:15.853 INFO:tasks.rados.rados.0.smithi071.stdout:2980: read oid 50 snap -1 2024-09-11T01:49:15.853 INFO:tasks.rados.rados.0.smithi071.stdout:2980: expect deleted 2024-09-11T01:49:15.853 INFO:tasks.rados.rados.0.smithi071.stdout:2981: snap_remove snap 326 2024-09-11T01:49:16.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:16.010+0000 7f2de2ffd640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 0x7f2de402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:16.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:16.010+0000 7f2de2ffd640 10 monclient: get_auth_request con 0x7f2de41cd5d0 auth_method 0 2024-09-11T01:49:16.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:16.010+0000 7f2de2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:16.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:16.010+0000 7f2de2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:16.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:16.010+0000 7f2de2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:49:16.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:16.010+0000 7f2de2ffd640 1 -- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 msgr2=0x7f2de402d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:49:16.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:16.010+0000 7f2de2ffd640 1 -- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 msgr2=0x7f2de402d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:49:16.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:16.010+0000 7f2de2ffd640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 0x7f2de402d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:49:16.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:16.010+0000 7f2de2ffd640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 0x7f2de402d440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:49:16.836 INFO:tasks.rados.rados.0.smithi071.stdout:2981: done (3 left) 2024-09-11T01:49:16.836 INFO:tasks.rados.rados.0.smithi071.stdout:2982: delete oid 8 current snap is 339 2024-09-11T01:49:16.836 INFO:tasks.rados.rados.0.smithi071.stdout:2978: finishing copy_from to smithi07134777-31 2024-09-11T01:49:16.836 INFO:tasks.rados.rados.0.smithi071.stdout:2978: got expected ENOENT (src dne) 2024-09-11T01:49:16.836 INFO:tasks.rados.rados.0.smithi071.stdout:2978: finishing copy_from racing read to smithi07134777-31 2024-09-11T01:49:16.836 INFO:tasks.rados.rados.0.smithi071.stdout:2979: finishing write tid 1 to smithi07134777-4 2024-09-11T01:49:16.837 INFO:tasks.rados.rados.0.smithi071.stdout:2979: finishing write tid 2 to smithi07134777-4 2024-09-11T01:49:16.837 INFO:tasks.rados.rados.0.smithi071.stdout:2979: finishing write tid 3 to smithi07134777-4 2024-09-11T01:49:16.837 INFO:tasks.rados.rados.0.smithi071.stdout:2979: finishing write tid 4 to smithi07134777-4 2024-09-11T01:49:16.837 INFO:tasks.rados.rados.0.smithi071.stdout:2979: finishing write tid 5 to smithi07134777-4 2024-09-11T01:49:16.837 INFO:tasks.rados.rados.0.smithi071.stdout:2979: oid 4 updating version 0 to 1310 2024-09-11T01:49:16.837 INFO:tasks.rados.rados.0.smithi071.stdout:2979: oid 4 updating version 1310 to 1311 2024-09-11T01:49:16.837 INFO:tasks.rados.rados.0.smithi071.stdout:2979: oid 4 version 1311 is already newer than 1309 2024-09-11T01:49:16.837 INFO:tasks.rados.rados.0.smithi071.stdout:2979: oid 4 version 1311 is already newer than 1308 2024-09-11T01:49:16.837 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 1311 (ObjNum 1050 snap 339 seq_num 1050) dirty exists 2024-09-11T01:49:16.837 INFO:tasks.rados.rados.0.smithi071.stdout:2979: left oid 4 (ObjNum 1050 snap 339 seq_num 1050) 2024-09-11T01:49:16.851 INFO:tasks.rados.rados.0.smithi071.stdout:2978: done (3 left) 2024-09-11T01:49:16.851 INFO:tasks.rados.rados.0.smithi071.stdout:2979: done (2 left) 2024-09-11T01:49:16.851 INFO:tasks.rados.rados.0.smithi071.stdout:2980: done (1 left) 2024-09-11T01:49:16.852 INFO:tasks.rados.rados.0.smithi071.stdout:2982: done (0 left) 2024-09-11T01:49:16.852 INFO:tasks.rados.rados.0.smithi071.stdout:2983: write (excl) oid 22 current snap is 339 2024-09-11T01:49:16.852 INFO:tasks.rados.rados.0.smithi071.stdout:2983: seq_num 1052 ranges {480964=418662,1560704=410362,2523106=619334} 2024-09-11T01:49:16.856 INFO:tasks.rados.rados.0.smithi071.stdout:2983: writing smithi07134777-22 from 480964 to 899626 tid 1 2024-09-11T01:49:16.862 INFO:tasks.rados.rados.0.smithi071.stdout:2983: writing smithi07134777-22 from 1560704 to 1971066 tid 2 2024-09-11T01:49:16.869 INFO:tasks.rados.rados.0.smithi071.stdout:2983: writing smithi07134777-22 from 2523106 to 3142440 tid 3 2024-09-11T01:49:16.869 INFO:tasks.rados.rados.0.smithi071.stdout:2984: write oid 19 current snap is 339 2024-09-11T01:49:16.869 INFO:tasks.rados.rados.0.smithi071.stdout:2983: finishing write tid 1 to smithi07134777-22 2024-09-11T01:49:16.869 INFO:tasks.rados.rados.0.smithi071.stdout:2984: seq_num 1053 ranges {497771=493911,1619186=554020,2950529=430606} 2024-09-11T01:49:16.874 INFO:tasks.rados.rados.0.smithi071.stdout:2984: writing smithi07134777-19 from 497771 to 991682 tid 1 2024-09-11T01:49:16.881 INFO:tasks.rados.rados.0.smithi071.stdout:2984: writing smithi07134777-19 from 1619186 to 2173206 tid 2 2024-09-11T01:49:16.888 INFO:tasks.rados.rados.0.smithi071.stdout:2984: writing smithi07134777-19 from 2950529 to 3381135 tid 3 2024-09-11T01:49:16.888 INFO:tasks.rados.rados.0.smithi071.stdout:2985: read oid 10 snap -1 2024-09-11T01:49:16.889 INFO:tasks.rados.rados.0.smithi071.stdout:2985: expect deleted 2024-09-11T01:49:16.889 INFO:tasks.rados.rados.0.smithi071.stdout:2986: copy_from oid 25 from oid 39 current snap is 339 2024-09-11T01:49:16.889 INFO:tasks.rados.rados.0.smithi071.stdout:2983: finishing write tid 2 to smithi07134777-22 2024-09-11T01:49:16.889 INFO:tasks.rados.rados.0.smithi071.stdout:2983: finishing write tid 3 to smithi07134777-22 2024-09-11T01:49:16.889 INFO:tasks.rados.rados.0.smithi071.stdout:2983: finishing write tid 4 to smithi07134777-22 2024-09-11T01:49:16.889 INFO:tasks.rados.rados.0.smithi071.stdout:2983: finishing write tid 5 to smithi07134777-22 2024-09-11T01:49:16.889 INFO:tasks.rados.rados.0.smithi071.stdout:2983: oid 22 updating version 0 to 2040 2024-09-11T01:49:16.889 INFO:tasks.rados.rados.0.smithi071.stdout:2983: oid 22 version 2040 is already newer than 2038 2024-09-11T01:49:16.889 INFO:tasks.rados.rados.0.smithi071.stdout:2983: oid 22 version 2040 is already newer than 2037 2024-09-11T01:49:16.889 INFO:tasks.rados.rados.0.smithi071.stdout:2983: oid 22 version 2040 is already newer than 2039 2024-09-11T01:49:16.889 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 2040 (ObjNum 1052 snap 339 seq_num 1052) dirty exists 2024-09-11T01:49:16.889 INFO:tasks.rados.rados.0.smithi071.stdout:2983: left oid 22 (ObjNum 1052 snap 339 seq_num 1052) 2024-09-11T01:49:16.889 INFO:tasks.rados.rados.0.smithi071.stdout:2984: finishing write tid 1 to smithi07134777-19 2024-09-11T01:49:16.890 INFO:tasks.rados.rados.0.smithi071.stdout:2984: finishing write tid 2 to smithi07134777-19 2024-09-11T01:49:16.890 INFO:tasks.rados.rados.0.smithi071.stdout:2983: done (3 left) 2024-09-11T01:49:16.890 INFO:tasks.rados.rados.0.smithi071.stdout:2987: read oid 38 snap -1 2024-09-11T01:49:16.890 INFO:tasks.rados.rados.0.smithi071.stdout:2987: expect (ObjNum 961 snap 305 seq_num 961) 2024-09-11T01:49:16.890 INFO:tasks.rados.rados.0.smithi071.stdout:2988: read oid 49 snap -1 2024-09-11T01:49:16.890 INFO:tasks.rados.rados.0.smithi071.stdout:2988: expect (ObjNum 1017 snap 322 seq_num 1017) 2024-09-11T01:49:16.890 INFO:tasks.rados.rados.0.smithi071.stdout:2989: delete oid 42 current snap is 339 2024-09-11T01:49:16.891 INFO:tasks.rados.rados.0.smithi071.stdout:2986: finishing copy_from to smithi07134777-25 2024-09-11T01:49:16.891 INFO:tasks.rados.rados.0.smithi071.stdout:2986: got expected ENOENT (src dne) 2024-09-11T01:49:16.891 INFO:tasks.rados.rados.0.smithi071.stdout:2986: finishing copy_from racing read to smithi07134777-25 2024-09-11T01:49:16.893 INFO:tasks.rados.rados.0.smithi071.stdout:2984: finishing write tid 3 to smithi07134777-19 2024-09-11T01:49:16.893 INFO:tasks.rados.rados.0.smithi071.stdout:2985: done (5 left) 2024-09-11T01:49:16.893 INFO:tasks.rados.rados.0.smithi071.stdout:2986: done (4 left) 2024-09-11T01:49:16.893 INFO:tasks.rados.rados.0.smithi071.stdout:2989: done (3 left) 2024-09-11T01:49:16.893 INFO:tasks.rados.rados.0.smithi071.stdout:2990: rollback oid 14 current snap is 339 2024-09-11T01:49:16.893 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 14 to 307 2024-09-11T01:49:16.893 INFO:tasks.rados.rados.0.smithi071.stdout:2984: finishing write tid 4 to smithi07134777-19 2024-09-11T01:49:16.893 INFO:tasks.rados.rados.0.smithi071.stdout:2991: read oid 31 snap -1 2024-09-11T01:49:16.893 INFO:tasks.rados.rados.0.smithi071.stdout:2991: expect (ObjNum 988 snap 317 seq_num 988) 2024-09-11T01:49:16.893 INFO:tasks.rados.rados.0.smithi071.stdout:2992: snap_create 2024-09-11T01:49:16.893 INFO:tasks.rados.rados.0.smithi071.stdout:2984: finishing write tid 5 to smithi07134777-19 2024-09-11T01:49:16.893 INFO:tasks.rados.rados.0.smithi071.stdout:2984: oid 19 updating version 0 to 1772 2024-09-11T01:49:16.894 INFO:tasks.rados.rados.0.smithi071.stdout:2984: oid 19 updating version 1772 to 1773 2024-09-11T01:49:16.894 INFO:tasks.rados.rados.0.smithi071.stdout:2984: oid 19 version 1773 is already newer than 1771 2024-09-11T01:49:16.894 INFO:tasks.rados.rados.0.smithi071.stdout:2984: oid 19 updating version 1773 to 1774 2024-09-11T01:49:16.894 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 1774 (ObjNum 1053 snap 339 seq_num 1053) dirty exists 2024-09-11T01:49:16.894 INFO:tasks.rados.rados.0.smithi071.stdout:2984: left oid 19 (ObjNum 1053 snap 339 seq_num 1053) 2024-09-11T01:49:16.894 INFO:tasks.rados.rados.0.smithi071.stdout:2987: expect (ObjNum 961 snap 305 seq_num 961) 2024-09-11T01:49:16.917 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1931053 2024-09-11T01:49:16.952 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:16.988 INFO:tasks.rados.rados.0.smithi071.stdout:2988: expect (ObjNum 1017 snap 322 seq_num 1017) 2024-09-11T01:49:17.008 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2156101 2024-09-11T01:49:17.042 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 662342 2024-09-11T01:49:17.156 INFO:tasks.rados.rados.0.smithi071.stdout:2991: expect (ObjNum 988 snap 317 seq_num 988) 2024-09-11T01:49:17.157 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:17.224 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1307933 2024-09-11T01:49:17.253 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:17.303 INFO:tasks.rados.rados.0.smithi071.stdout:2990: finishing rollback tid 0 to smithi07134777-14 2024-09-11T01:49:17.304 INFO:tasks.rados.rados.0.smithi071.stdout:2990: finishing rollback tid 1 to smithi07134777-14 2024-09-11T01:49:17.304 INFO:tasks.rados.rados.0.smithi071.stdout:2990: finishing rollback tid 2 to smithi07134777-14 2024-09-11T01:49:17.304 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 2139 (ObjNum 942 snap 296 seq_num 942) dirty exists 2024-09-11T01:49:17.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.601+0000 7f2dcaffd640 10 monclient: tick 2024-09-11T01:49:17.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.601+0000 7f2dcaffd640 10 monclient: _check_auth_tickets 2024-09-11T01:49:17.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.601+0000 7f2dcaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:17.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.601+0000 7f2dcaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:17.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.612+0000 7f2de2ffd640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 0x7f2de402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:17.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.612+0000 7f2de2ffd640 10 monclient: get_auth_request con 0x7f2de41cd5d0 auth_method 0 2024-09-11T01:49:17.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.612+0000 7f2de2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:17.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.612+0000 7f2de2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:17.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.612+0000 7f2de2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:49:17.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.612+0000 7f2de2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:17.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.612+0000 7f2de2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:17.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.612+0000 7f2de2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:17.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.612+0000 7f2de2ffd640 10 monclient(hunting): handle_auth_done global_id 224099 payload 824 2024-09-11T01:49:17.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.612+0000 7f2de2ffd640 1 -- 172.21.15.12:0/607852556 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2de4121500 con 0x7f2de41cd5d0 2024-09-11T01:49:17.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.615+0000 7f2de2ffd640 1 -- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 msgr2=0x7f2de402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:17.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.615+0000 7f2de2ffd640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 0x7f2de402d440 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2de4120940 tx=0x7f2dd8018e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:17.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.615+0000 7f2dcbfff640 1 -- 172.21.15.12:0/607852556 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd800f040 con 0x7f2de41cd5d0 2024-09-11T01:49:17.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.615+0000 7f2dcbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:17.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.615+0000 7f2dcbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:49:17.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.615+0000 7f2dcbfff640 10 monclient: dump: 2024-09-11T01:49:17.617 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:17.617 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:17.617 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:17.617 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:17.617 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:17.617 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:17.617 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:17.618 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:17.618 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:17.618 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:17.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.615+0000 7f2dcbfff640 1 -- 172.21.15.12:0/607852556 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2dd8004020 con 0x7f2de41cd5d0 2024-09-11T01:49:17.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.615+0000 7f2dcbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:17.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.615+0000 7f2dcbfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:49:17.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.615+0000 7f2dcbfff640 1 -- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 msgr2=0x7f2de402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:17.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.615+0000 7f2dcbfff640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 0x7f2de402d440 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2de4120940 tx=0x7f2dd8018e20 comp rx=0 tx=0).stop 2024-09-11T01:49:17.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.615+0000 7f2dcbfff640 1 -- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 msgr2=0x7f2de402d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:49:17.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.615+0000 7f2dcbfff640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd5d0 0x7f2de402d440 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2de4120940 tx=0x7f2dd8018e20 comp rx=0 tx=0).stop 2024-09-11T01:49:17.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.622+0000 7f2dea260640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:49:17.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.622+0000 7f2dea260640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd9b0 0x7f2de402d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:17.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.622+0000 7f2dea260640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:17.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.622+0000 7f2dea260640 1 -- 172.21.15.12:0/607852556 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2de4074410 con 0x7f2de41cd9b0 2024-09-11T01:49:17.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.623+0000 7f2de2ffd640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd9b0 0x7f2de402d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:17.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.623+0000 7f2de2ffd640 10 monclient: get_auth_request con 0x7f2de41cd9b0 auth_method 0 2024-09-11T01:49:17.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.623+0000 7f2de2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:17.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.623+0000 7f2de2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:17.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.623+0000 7f2de2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:17.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.623+0000 7f2de2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:17.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.623+0000 7f2de2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:17.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.623+0000 7f2de2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:17.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.623+0000 7f2de2ffd640 10 monclient(hunting): handle_auth_done global_id 224102 payload 824 2024-09-11T01:49:17.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.623+0000 7f2de2ffd640 1 -- 172.21.15.12:0/607852556 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2de409da30 con 0x7f2de41cd9b0 2024-09-11T01:49:17.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.625+0000 7f2de2ffd640 1 -- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd9b0 msgr2=0x7f2de402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:17.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.625+0000 7f2de2ffd640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd9b0 0x7f2de402d980 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8021000 tx=0x7f2dd80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:17.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.625+0000 7f2dcbfff640 1 -- 172.21.15.12:0/607852556 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd8003e60 con 0x7f2de41cd9b0 2024-09-11T01:49:17.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.625+0000 7f2dcbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:17.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.625+0000 7f2dcbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:49:17.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.625+0000 7f2dcbfff640 10 monclient: dump: 2024-09-11T01:49:17.628 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:17.628 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:17.628 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:17.628 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:17.628 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:17.628 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:17.629 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:17.629 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:17.629 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:17.629 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:17.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.626+0000 7f2dcbfff640 1 -- 172.21.15.12:0/607852556 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7f2dd8003e60 con 0x7f2de41cd9b0 2024-09-11T01:49:17.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.626+0000 7f2dcbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:49:17.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.626+0000 7f2dcbfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:49:17.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.626+0000 7f2dcbfff640 1 -- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd9b0 msgr2=0x7f2de402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:17.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.626+0000 7f2dcbfff640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd9b0 0x7f2de402d980 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8021000 tx=0x7f2dd80046c0 comp rx=0 tx=0).stop 2024-09-11T01:49:17.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.626+0000 7f2dcbfff640 1 -- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd9b0 msgr2=0x7f2de402d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:49:17.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.626+0000 7f2dcbfff640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2de41cd9b0 0x7f2de402d980 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8021000 tx=0x7f2dd80046c0 comp rx=0 tx=0).stop 2024-09-11T01:49:17.630 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:49:17.630 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:49:17.630 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:49:17.630 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:49:17.630 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 188, 2024-09-11T01:49:17.630 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 10397, 2024-09-11T01:49:17.630 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:49:17.630 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:49:17.630 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:49:17.630 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:49:17.630 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:17.630 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 4, 2024-09-11T01:49:17.631 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:49:17.631 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:49:17.631 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:49:17.631 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:49:17.631 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:49:17.631 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:49:17.631 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:49:17.631 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:49:17.631 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:49:17.631 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:49:17.631 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:49:17.631 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:49:17.631 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:49:17.632 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:49:17.632 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:17.632 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:49:17.632 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:49:17.632 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:49:17.632 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:49:17.632 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:49:17.632 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:49:17.632 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:49:17.632 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:49:17.632 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:49:17.632 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:49:17.632 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:49:17.633 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:49:17.633 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:49:17.633 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:17.633 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:17.633 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:49:17.633 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:49:17.633 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:49:17.633 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:49:17.633 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:49:17.633 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:49:17.633 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:49:17.633 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:49:17.633 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:49:17.634 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:49:17.634 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:49:17.634 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:49:17.634 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:49:17.634 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:49:17.634 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:49:17.634 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:49:17.634 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:49:17.634 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:49:17.634 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:49:17.634 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:49:17.634 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:49:17.634 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:49:17.634 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:49:17.635 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:49:17.635 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:49:17.635 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:49:17.635 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:49:17.635 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:49:17.635 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:17.635 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:49:17.635 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:17.635 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:49:17.635 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:17.635 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:49:17.635 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:49:17.635 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:49:17.635 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:49:17.636 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:17.636 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:49:17.636 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:49:17.636 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:17.636 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:17.636 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:17.636 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:49:17.636 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:49:17.636 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:17.636 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:17.636 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:17.636 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:17.636 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:49:17.637 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:49:17.637 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:49:17.637 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:49:17.637 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:49:17.637 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:17.637 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:17.637 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:49:17.637 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:49:17.637 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:49:17.637 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:49:17.637 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:17.637 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:49:17.637 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:49:17.637 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:17.638 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:17.638 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:17.638 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:49:17.638 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:49:17.638 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:17.638 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:17.638 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:17.638 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:17.638 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:49:17.638 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:49:17.638 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:49:17.638 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:49:17.638 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:49:17.638 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:17.639 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:17.639 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:49:17.639 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:49:17.639 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:49:17.639 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:49:17.639 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:17.639 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:49:17.639 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:49:17.639 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:17.639 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:17.639 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:17.639 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:49:17.639 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:49:17.639 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:17.640 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:17.640 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:17.640 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:17.640 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:49:17.640 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:49:17.640 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:49:17.640 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:49:17.640 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:49:17.640 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:17.640 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:17.640 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:17.640 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:49:17.640 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:49:17.640 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:17.641 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:49:17.641 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:49:17.641 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:49:17.641 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:17.641 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:17.641 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:49:17.641 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:17.641 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:49:17.641 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:49:17.641 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:49:17.641 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:17.641 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:17.641 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:17.641 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:49:17.642 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:49:17.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.626+0000 7f2dea260640 1 -- 172.21.15.12:0/607852556 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2dd003e210 msgr2=0x7f2dd00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:17.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.626+0000 7f2dea260640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2dd003e210 0x7f2dd00406d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc005520 tx=0x7f2ddc009040 comp rx=0 tx=0).stop 2024-09-11T01:49:17.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.626+0000 7f2dea260640 10 monclient: shutdown 2024-09-11T01:49:17.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.626+0000 7f2dea260640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:17.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.626+0000 7f2dea260640 1 -- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2de4074000 msgr2=0x7f2de411eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:17.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.626+0000 7f2dea260640 1 --2- 172.21.15.12:0/607852556 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2de4074000 0x7f2de411eb70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2dd40098c0 tx=0x7f2dd40046c0 comp rx=0 tx=0).stop 2024-09-11T01:49:17.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.626+0000 7f2de3fff640 1 -- 172.21.15.12:0/607852556 reap_dead start 2024-09-11T01:49:17.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.626+0000 7f2de3fff640 1 -- 172.21.15.12:0/607852556 reap_dead start 2024-09-11T01:49:17.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.626+0000 7f2dea260640 1 -- 172.21.15.12:0/607852556 shutdown_connections 2024-09-11T01:49:17.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.626+0000 7f2dea260640 1 -- 172.21.15.12:0/607852556 >> 172.21.15.12:0/607852556 conn(0x7f2de40a8600 msgr2=0x7f2de413cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:17.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.626+0000 7f2dea260640 1 -- 172.21.15.12:0/607852556 shutdown_connections 2024-09-11T01:49:17.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.626+0000 7f2dea260640 1 -- 172.21.15.12:0/607852556 wait complete. 2024-09-11T01:49:17.644 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:49:17.644 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:49:17.644 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:49:17.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.754+0000 7fed5db52640 10 monclient: get_monmap_and_config 2024-09-11T01:49:17.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.754+0000 7fed5db52640 10 monclient: build_initial_monmap 2024-09-11T01:49:17.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.754+0000 7fed5db52640 10 monclient: monmap: 2024-09-11T01:49:17.756 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:17.756 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:17.756 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:49:17.755417+0000 2024-09-11T01:49:17.756 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:49:17.755417+0000 2024-09-11T01:49:17.756 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:17.757 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:17.757 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:49:17.757 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:49:17.757 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:49:17.757 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:49:17.757 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:49:17.757 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:17.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.755+0000 7fed5db52640 1 Processor -- start 2024-09-11T01:49:17.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.755+0000 7fed5db52640 1 -- start start 2024-09-11T01:49:17.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.755+0000 7fed5db52640 10 monclient: init 2024-09-11T01:49:17.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.755+0000 7fed5db52640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:17.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.755+0000 7fed5db52640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-11T01:49:17.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed5db52640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fed58134ad0 0x7fed58134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:17.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed5db52640 10 monclient(hunting): picked mon.noname-a con 0x7fed58134ad0 addr v2:172.21.15.12:3300/0 2024-09-11T01:49:17.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed5db52640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fed580a2200 addr v1:172.21.15.71:6789/0 2024-09-11T01:49:17.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed5db52640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fed581353f0 addr v1:172.21.15.12:6789/0 2024-09-11T01:49:17.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed5db52640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed58134870 con 0x7fed581353f0 2024-09-11T01:49:17.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed5db52640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed5809ed30 con 0x7fed580a2200 2024-09-11T01:49:17.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed5db52640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:17.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed5db52640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fed5809f5d0 con 0x7fed58134ad0 2024-09-11T01:49:17.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed5db52640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:17.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed5db52640 10 monclient(hunting): authenticate will time out at 1964.198089s 2024-09-11T01:49:17.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed577fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fed58134ad0 0x7fed58134eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:17.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed57fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fed581353f0 0x7fed58134160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59884/0 (socket says 172.21.15.12:59884) 2024-09-11T01:49:17.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed577fe640 10 monclient(hunting): get_auth_request con 0x7fed58134ad0 auth_method 0 2024-09-11T01:49:17.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed57fff640 1 -- 172.21.15.12:0/3437261741 learned_addr learned my addr 172.21.15.12:0/3437261741 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:49:17.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:17.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed577fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:17.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:17.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:17.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:17.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:17.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed577fe640 10 monclient(hunting): handle_auth_done global_id 214212 payload 274 2024-09-11T01:49:17.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed577fe640 1 -- 172.21.15.12:0/3437261741 >> v1:172.21.15.71:6789/0 conn(0x7fed580a2200 legacy=0x7fed58133a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:49:17.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed577fe640 1 -- 172.21.15.12:0/3437261741 >> v1:172.21.15.12:6789/0 conn(0x7fed581353f0 legacy=0x7fed58134160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:49:17.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed577fe640 10 monclient: _finish_hunting 0 2024-09-11T01:49:17.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed577fe640 1 monclient: found mon.noname-a 2024-09-11T01:49:17.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:17.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed577fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:49:17.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.756+0000 7fed577fe640 1 -- 172.21.15.12:0/3437261741 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed5809f010 con 0x7fed58134ad0 2024-09-11T01:49:17.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.759+0000 7fed577fe640 1 -- 172.21.15.12:0/3437261741 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed58134ad0 msgr2=0x7fed58134eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:17.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.759+0000 7fed577fe640 1 --2- 172.21.15.12:0/3437261741 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed58134ad0 0x7fed58134eb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fed480097d0 tx=0x7fed4802a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d30c3bce444b86a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:17.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.759+0000 7fed567fc640 1 -- 172.21.15.12:0/3437261741 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed48037070 con 0x7fed58134ad0 2024-09-11T01:49:17.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.759+0000 7fed567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:17.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.759+0000 7fed567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:49:17.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.759+0000 7fed567fc640 10 monclient: dump: 2024-09-11T01:49:17.765 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:17.766 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:17.766 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:17.766 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:17.766 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:17.766 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:17.766 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:17.766 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:17.766 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:17.766 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:17.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.759+0000 7fed567fc640 10 monclient: _finish_auth 0 2024-09-11T01:49:17.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.759+0000 7fed567fc640 10 monclient: _check_auth_tickets 2024-09-11T01:49:17.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.759+0000 7fed567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:17.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.759+0000 7fed567fc640 1 -- 172.21.15.12:0/3437261741 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed48032640 con 0x7fed58134ad0 2024-09-11T01:49:17.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.759+0000 7fed5db52640 5 monclient: authenticate success, global_id 214212 2024-09-11T01:49:17.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.759+0000 7fed5db52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:49:17.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.759+0000 7fed567fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:17.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.759+0000 7fed5db52640 10 monclient: get_monmap_and_config success 2024-09-11T01:49:17.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.759+0000 7fed567fc640 1 -- 172.21.15.12:0/3437261741 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed48032a10 con 0x7fed58134ad0 2024-09-11T01:49:17.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.759+0000 7fed5db52640 4 set_mon_vals no callback set 2024-09-11T01:49:17.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.759+0000 7fed567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:17.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.759+0000 7fed567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:17.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.759+0000 7fed567fc640 10 monclient: dump: 2024-09-11T01:49:17.767 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:17.767 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:17.767 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:17.767 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:17.768 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:17.768 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:17.768 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:17.768 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:17.768 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:17.768 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:17.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.760+0000 7fed5db52640 10 monclient: shutdown 2024-09-11T01:49:17.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.760+0000 7fed5db52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:17.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.760+0000 7fed5db52640 1 -- 172.21.15.12:0/3437261741 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed58134ad0 msgr2=0x7fed58134eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:17.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.760+0000 7fed5db52640 1 --2- 172.21.15.12:0/3437261741 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed58134ad0 0x7fed58134eb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fed480097d0 tx=0x7fed4802a080 comp rx=0 tx=0).stop 2024-09-11T01:49:17.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.760+0000 7fed5db52640 1 -- 172.21.15.12:0/3437261741 shutdown_connections 2024-09-11T01:49:17.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.760+0000 7fed5db52640 1 --2- 172.21.15.12:0/3437261741 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed58134ad0 0x7fed58134eb0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:17.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.760+0000 7fed5db52640 1 -- 172.21.15.12:0/3437261741 >> 172.21.15.12:0/3437261741 conn(0x7fed580a2f70 msgr2=0x7fed580a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:17.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.760+0000 7fed5db52640 1 -- 172.21.15.12:0/3437261741 shutdown_connections 2024-09-11T01:49:17.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.760+0000 7fed5db52640 1 -- 172.21.15.12:0/3437261741 wait complete. 2024-09-11T01:49:17.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.760+0000 7fed5db52640 10 monclient: build_initial_monmap 2024-09-11T01:49:17.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.760+0000 7fed5db52640 10 monclient: monmap: 2024-09-11T01:49:17.769 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:17.769 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:49:17.769 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:49:17.769 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:49:17.769 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:17.769 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:17.769 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:49:17.769 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:49:17.769 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:49:17.769 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:17.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed5db52640 1 Processor -- start 2024-09-11T01:49:17.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed5db52640 1 -- start start 2024-09-11T01:49:17.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed5db52640 10 monclient: init 2024-09-11T01:49:17.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed5db52640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:17.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed5db52640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:49:17.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed5db52640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fed580a2200 0x7fed581cd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:17.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed5db52640 10 monclient(hunting): picked mon.noname-c con 0x7fed580a2200 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:17.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed5db52640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed58134ad0 0x7fed58132d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:17.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed5db52640 10 monclient(hunting): picked mon.noname-b con 0x7fed58134ad0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:49:17.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed5db52640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed581353f0 0x7fed581332d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:17.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed5db52640 10 monclient(hunting): picked mon.noname-a con 0x7fed581353f0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:49:17.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed5db52640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:17.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed5db52640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fed5809ce20 con 0x7fed581353f0 2024-09-11T01:49:17.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed5db52640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:17.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed5db52640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fed5809cca0 con 0x7fed58134ad0 2024-09-11T01:49:17.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed5db52640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:17.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed5db52640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fed5809ffd0 con 0x7fed580a2200 2024-09-11T01:49:17.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed5db52640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:17.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed5db52640 10 monclient(hunting): authenticate will time out at 1964.203788s 2024-09-11T01:49:17.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed57fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed581353f0 0x7fed581332d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:17.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed577fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fed580a2200 0x7fed581cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:17.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed57fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed581353f0 0x7fed581332d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43756/0 (socket says 172.21.15.12:43756) 2024-09-11T01:49:17.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed57fff640 1 -- 172.21.15.12:0/21796722 learned_addr learned my addr 172.21.15.12:0/21796722 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:49:17.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed577fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fed580a2200 0x7fed581cd190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42222/0 (socket says 172.21.15.12:42222) 2024-09-11T01:49:17.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed57fff640 10 monclient(hunting): get_auth_request con 0x7fed581353f0 auth_method 0 2024-09-11T01:49:17.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:17.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed57fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:17.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed57fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:17.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed577fe640 10 monclient(hunting): get_auth_request con 0x7fed580a2200 auth_method 0 2024-09-11T01:49:17.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:17.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed577fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:17.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.761+0000 7fed577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:17.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.762+0000 7fed57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:17.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.762+0000 7fed57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:17.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.762+0000 7fed57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:17.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.762+0000 7fed577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:17.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.762+0000 7fed577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:17.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.762+0000 7fed577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:17.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.762+0000 7fed56ffd640 1 --2- 172.21.15.12:0/21796722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed58134ad0 0x7fed58132d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:17.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.762+0000 7fed56ffd640 10 monclient(hunting): get_auth_request con 0x7fed58134ad0 auth_method 0 2024-09-11T01:49:17.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.762+0000 7fed56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:17.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.762+0000 7fed56ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:17.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.762+0000 7fed56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:17.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.762+0000 7fed56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:17.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.762+0000 7fed56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:17.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.762+0000 7fed56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:17.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.762+0000 7fed57fff640 10 monclient(hunting): handle_auth_done global_id 214215 payload 824 2024-09-11T01:49:17.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.762+0000 7fed57fff640 1 -- 172.21.15.12:0/21796722 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fed580a2200 msgr2=0x7fed581cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:17.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.762+0000 7fed57fff640 1 --2- 172.21.15.12:0/21796722 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fed580a2200 0x7fed581cd190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:17.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.762+0000 7fed57fff640 1 -- 172.21.15.12:0/21796722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed58134ad0 msgr2=0x7fed58132d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:17.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.762+0000 7fed57fff640 1 --2- 172.21.15.12:0/21796722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed58134ad0 0x7fed58132d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:17.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.762+0000 7fed57fff640 10 monclient: _finish_hunting 0 2024-09-11T01:49:17.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.762+0000 7fed57fff640 1 monclient: found mon.noname-a 2024-09-11T01:49:17.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.762+0000 7fed57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:17.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.762+0000 7fed57fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:49:17.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.762+0000 7fed57fff640 1 -- 172.21.15.12:0/21796722 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed581274e0 con 0x7fed581353f0 2024-09-11T01:49:17.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.764+0000 7fed57fff640 1 -- 172.21.15.12:0/21796722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed581353f0 msgr2=0x7fed581332d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:17.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.764+0000 7fed57fff640 1 --2- 172.21.15.12:0/21796722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed581353f0 0x7fed581332d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fed5000caf0 tx=0x7fed5000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:17.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.764+0000 7fed3ffff640 1 -- 172.21.15.12:0/21796722 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed5000ae50 con 0x7fed581353f0 2024-09-11T01:49:17.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.764+0000 7fed3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:17.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.764+0000 7fed3ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:49:17.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.764+0000 7fed3ffff640 10 monclient: dump: 2024-09-11T01:49:17.778 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:17.778 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:17.778 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:17.778 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:17.778 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:17.778 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:17.778 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:17.778 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:17.778 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:17.778 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:17.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.764+0000 7fed3ffff640 10 monclient: _finish_auth 0 2024-09-11T01:49:17.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.764+0000 7fed3ffff640 10 monclient: _check_auth_tickets 2024-09-11T01:49:17.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.764+0000 7fed3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:17.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.764+0000 7fed5db52640 5 monclient: authenticate success, global_id 214215 2024-09-11T01:49:17.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.764+0000 7fed3ffff640 1 -- 172.21.15.12:0/21796722 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed50012400 con 0x7fed581353f0 2024-09-11T01:49:17.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.764+0000 7fed5db52640 10 monclient: _renew_subs 2024-09-11T01:49:17.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.764+0000 7fed5db52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:17.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.764+0000 7fed5db52640 1 -- 172.21.15.12:0/21796722 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed58125fc0 con 0x7fed581353f0 2024-09-11T01:49:17.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.764+0000 7fed3ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:17.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.764+0000 7fed3ffff640 1 -- 172.21.15.12:0/21796722 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed500127d0 con 0x7fed581353f0 2024-09-11T01:49:17.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.764+0000 7fed3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:17.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.764+0000 7fed3ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:17.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.764+0000 7fed3ffff640 10 monclient: dump: 2024-09-11T01:49:17.779 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:17.780 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:17.780 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:17.780 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:17.780 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:17.780 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:17.780 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:17.780 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:17.780 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:17.780 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:17.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.764+0000 7fed5db52640 10 monclient: _renew_subs 2024-09-11T01:49:17.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.764+0000 7fed5db52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:17.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.764+0000 7fed5db52640 1 -- 172.21.15.12:0/21796722 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed58125460 con 0x7fed581353f0 2024-09-11T01:49:17.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.764+0000 7fed55ffb640 4 set_mon_vals no callback set 2024-09-11T01:49:17.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.765+0000 7fed5db52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:17.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.765+0000 7fed5db52640 1 --2- 172.21.15.12:0/21796722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed58124160 0x7fed5802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:17.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.765+0000 7fed5db52640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:17.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.765+0000 7fed5db52640 1 -- 172.21.15.12:0/21796722 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fed58134eb0 con 0x7fed58124160 2024-09-11T01:49:17.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.765+0000 7fed3ffff640 1 -- 172.21.15.12:0/21796722 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fed50012a30 con 0x7fed581353f0 2024-09-11T01:49:17.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.765+0000 7fed577fe640 1 --2- 172.21.15.12:0/21796722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed58124160 0x7fed5802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:17.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.765+0000 7fed577fe640 10 monclient: get_auth_request con 0x7fed58124160 auth_method 0 2024-09-11T01:49:17.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.765+0000 7fed577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:17.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.765+0000 7fed577fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:17.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.765+0000 7fed577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:17.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.766+0000 7fed577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:17.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.766+0000 7fed577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:17.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.766+0000 7fed577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:17.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.766+0000 7fed3ffff640 1 --2- 172.21.15.12:0/21796722 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fed4403e3b0 0x7fed44040870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:17.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.766+0000 7fed3ffff640 1 -- 172.21.15.12:0/21796722 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(674..674 src has 507..674) ==== 4847+0+0 (secure 0 0 0) 0x7fed50053430 con 0x7fed581353f0 2024-09-11T01:49:17.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.766+0000 7fed577fe640 10 monclient(hunting): handle_auth_done global_id 134842 payload 824 2024-09-11T01:49:17.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.766+0000 7fed577fe640 1 -- 172.21.15.12:0/21796722 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fed58072db0 con 0x7fed58124160 2024-09-11T01:49:17.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.766+0000 7fed56ffd640 1 --2- 172.21.15.12:0/21796722 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fed4403e3b0 0x7fed44040870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:17.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.766+0000 7fed56ffd640 10 monclient: get_auth_request con 0x7fed4403e3b0 auth_method 0 2024-09-11T01:49:17.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.768+0000 7fed577fe640 1 -- 172.21.15.12:0/21796722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed58124160 msgr2=0x7fed5802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:17.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.768+0000 7fed577fe640 1 --2- 172.21.15.12:0/21796722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed58124160 0x7fed5802d440 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fed580a2110 tx=0x7fed480046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.768+0000 7fed56ffd640 1 --2- 172.21.15.12:0/21796722 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fed4403e3b0 0x7fed44040870 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fed580a2630 tx=0x7fed4c007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.768+0000 7fed3ffff640 1 -- 172.21.15.12:0/21796722 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed48044070 con 0x7fed58124160 2024-09-11T01:49:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.768+0000 7fed3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.768+0000 7fed3ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:49:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.768+0000 7fed3ffff640 10 monclient: dump: 2024-09-11T01:49:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:17.784 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:17.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.769+0000 7fed3ffff640 1 -- 172.21.15.12:0/21796722 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fed48032e40 con 0x7fed58124160 2024-09-11T01:49:17.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.769+0000 7fed3ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:17.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.769+0000 7fed3ffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:49:17.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.769+0000 7fed3ffff640 1 -- 172.21.15.12:0/21796722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed58124160 msgr2=0x7fed5802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:17.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.769+0000 7fed3ffff640 1 --2- 172.21.15.12:0/21796722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed58124160 0x7fed5802d440 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fed580a2110 tx=0x7fed480046c0 comp rx=0 tx=0).stop 2024-09-11T01:49:17.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.769+0000 7fed3ffff640 1 -- 172.21.15.12:0/21796722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed58124160 msgr2=0x7fed5802d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:49:17.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.769+0000 7fed3ffff640 1 --2- 172.21.15.12:0/21796722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed58124160 0x7fed5802d440 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:17.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.775+0000 7fed5db52640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:49:17.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.775+0000 7fed5db52640 1 --2- 172.21.15.12:0/21796722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed58124540 0x7fed5802d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:17.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.775+0000 7fed5db52640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:17.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.775+0000 7fed5db52640 1 -- 172.21.15.12:0/21796722 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fed581cd6d0 con 0x7fed58124540 2024-09-11T01:49:17.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.775+0000 7fed577fe640 1 --2- 172.21.15.12:0/21796722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed58124540 0x7fed5802d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:17.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.775+0000 7fed577fe640 10 monclient: get_auth_request con 0x7fed58124540 auth_method 0 2024-09-11T01:49:17.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.775+0000 7fed577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:17.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.775+0000 7fed577fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:17.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.775+0000 7fed577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:17.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.775+0000 7fed577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:17.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.775+0000 7fed577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:17.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.775+0000 7fed577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:17.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.776+0000 7fed577fe640 10 monclient(hunting): handle_auth_done global_id 134845 payload 824 2024-09-11T01:49:17.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.776+0000 7fed577fe640 1 -- 172.21.15.12:0/21796722 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fed5806d480 con 0x7fed58124540 2024-09-11T01:49:17.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.778+0000 7fed577fe640 1 -- 172.21.15.12:0/21796722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed58124540 msgr2=0x7fed5802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:17.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.778+0000 7fed577fe640 1 --2- 172.21.15.12:0/21796722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed58124540 0x7fed5802d980 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fed4802a0c0 tx=0x7fed48009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:17.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.779+0000 7fed3ffff640 1 -- 172.21.15.12:0/21796722 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed48037040 con 0x7fed58124540 2024-09-11T01:49:17.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.779+0000 7fed3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:17.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.779+0000 7fed3ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:49:17.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.779+0000 7fed3ffff640 10 monclient: dump: 2024-09-11T01:49:17.787 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:17.787 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:17.787 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:17.787 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:17.787 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:17.787 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:17.787 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:17.787 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:17.787 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:17.787 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:17.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.779+0000 7fed3ffff640 1 -- 172.21.15.12:0/21796722 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7fed4803d3c0 con 0x7fed58124540 2024-09-11T01:49:17.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.779+0000 7fed3ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:49:17.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.779+0000 7fed3ffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:49:17.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.779+0000 7fed3ffff640 1 -- 172.21.15.12:0/21796722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed58124540 msgr2=0x7fed5802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:17.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.779+0000 7fed3ffff640 1 --2- 172.21.15.12:0/21796722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed58124540 0x7fed5802d980 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fed4802a0c0 tx=0x7fed48009190 comp rx=0 tx=0).stop 2024-09-11T01:49:17.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.779+0000 7fed3ffff640 1 -- 172.21.15.12:0/21796722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed58124540 msgr2=0x7fed5802d980 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:49:17.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.779+0000 7fed3ffff640 1 --2- 172.21.15.12:0/21796722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed58124540 0x7fed5802d980 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:17.788 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:49:17.788 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:49:17.788 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:49:17.788 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:49:17.788 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 188, 2024-09-11T01:49:17.788 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 490740, 2024-09-11T01:49:17.788 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:49:17.789 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:49:17.789 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:49:17.789 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:49:17.789 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:17.789 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 4, 2024-09-11T01:49:17.789 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:49:17.789 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:49:17.789 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:49:17.789 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:49:17.789 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:49:17.789 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:49:17.789 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:49:17.789 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:49:17.790 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:49:17.790 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:49:17.790 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:49:17.790 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:49:17.790 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:49:17.790 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:49:17.790 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:17.790 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:49:17.790 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:49:17.790 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:49:17.790 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:49:17.790 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:49:17.790 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:49:17.790 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:49:17.791 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:49:17.791 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:49:17.791 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:49:17.791 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:49:17.791 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:49:17.791 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:49:17.791 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:17.791 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:17.791 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:49:17.791 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:49:17.791 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:49:17.791 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:49:17.791 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:49:17.792 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:49:17.792 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:49:17.792 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:49:17.792 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:49:17.792 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:49:17.792 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:49:17.792 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:49:17.792 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:49:17.792 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:49:17.792 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:49:17.792 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:49:17.792 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:49:17.792 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:49:17.792 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:49:17.793 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:49:17.793 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:49:17.793 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:49:17.793 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:49:17.793 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:49:17.793 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:49:17.793 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:49:17.793 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:49:17.793 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:49:17.793 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:17.793 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:49:17.793 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:17.793 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:49:17.793 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:17.794 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:49:17.794 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:49:17.794 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:49:17.794 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:49:17.794 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:17.794 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:49:17.794 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:49:17.794 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:17.794 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:17.794 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:17.794 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:49:17.794 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:49:17.794 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:17.794 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:17.795 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:17.795 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:17.795 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:49:17.795 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:49:17.795 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:49:17.795 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:49:17.795 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:49:17.795 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:17.795 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:17.795 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:49:17.795 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:49:17.795 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:49:17.795 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:49:17.795 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:17.796 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:49:17.796 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:49:17.796 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:17.796 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:17.796 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:17.796 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:49:17.796 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:49:17.796 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:17.796 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:17.796 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:17.796 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:17.796 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:49:17.796 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:49:17.796 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:49:17.797 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:49:17.797 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:49:17.797 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:17.797 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:17.797 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:49:17.797 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:49:17.797 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:49:17.797 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:49:17.797 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:17.797 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:49:17.797 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:49:17.797 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:17.797 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:17.797 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:17.798 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:49:17.798 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:49:17.798 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:17.798 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:17.798 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:17.798 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:17.798 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:49:17.798 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:49:17.798 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:49:17.798 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:49:17.798 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:49:17.798 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:17.798 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:17.798 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:17.799 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:49:17.799 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:49:17.799 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:17.799 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:49:17.799 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:49:17.799 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:49:17.799 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:17.799 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:17.799 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:49:17.799 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:17.799 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:49:17.799 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:49:17.799 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:49:17.799 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:17.800 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:17.800 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:49:17.800 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:17.800 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:49:17.800 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:49:17.800 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:49:17.800 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:17.800 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:17.800 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:49:17.800 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:17.800 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:49:17.800 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:49:17.800 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:49:17.800 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:17.801 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:17.801 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:17.801 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:49:17.801 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:49:17.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.779+0000 7fed5db52640 1 -- 172.21.15.12:0/21796722 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fed4403e3b0 msgr2=0x7fed44040870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:17.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.779+0000 7fed5db52640 1 --2- 172.21.15.12:0/21796722 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fed4403e3b0 0x7fed44040870 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fed580a2630 tx=0x7fed4c007e30 comp rx=0 tx=0).stop 2024-09-11T01:49:17.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.779+0000 7fed5db52640 10 monclient: shutdown 2024-09-11T01:49:17.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.779+0000 7fed5db52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:17.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.779+0000 7fed5db52640 1 -- 172.21.15.12:0/21796722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed581353f0 msgr2=0x7fed581332d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:17.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.779+0000 7fed57fff640 1 -- 172.21.15.12:0/21796722 reap_dead start 2024-09-11T01:49:17.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.779+0000 7fed5db52640 1 --2- 172.21.15.12:0/21796722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed581353f0 0x7fed581332d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fed5000caf0 tx=0x7fed5000a060 comp rx=0 tx=0).stop 2024-09-11T01:49:17.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.779+0000 7fed5db52640 1 -- 172.21.15.12:0/21796722 shutdown_connections 2024-09-11T01:49:17.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.779+0000 7fed5db52640 1 --2- 172.21.15.12:0/21796722 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed581353f0 0x7fed581332d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:17.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.779+0000 7fed5db52640 1 -- 172.21.15.12:0/21796722 >> 172.21.15.12:0/21796722 conn(0x7fed580a2f70 msgr2=0x7fed58133fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:17.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.779+0000 7fed5db52640 1 -- 172.21.15.12:0/21796722 shutdown_connections 2024-09-11T01:49:17.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.779+0000 7fed5db52640 1 -- 172.21.15.12:0/21796722 wait complete. 2024-09-11T01:49:17.803 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:49:17.803 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:49:17.803 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:49:17.840 INFO:tasks.rados.rados.0.smithi071.stdout:2984: done (5 left) 2024-09-11T01:49:17.840 INFO:tasks.rados.rados.0.smithi071.stdout:2987: done (4 left) 2024-09-11T01:49:17.841 INFO:tasks.rados.rados.0.smithi071.stdout:2988: done (3 left) 2024-09-11T01:49:17.841 INFO:tasks.rados.rados.0.smithi071.stdout:2990: done (2 left) 2024-09-11T01:49:17.841 INFO:tasks.rados.rados.0.smithi071.stdout:2991: done (1 left) 2024-09-11T01:49:17.841 INFO:tasks.rados.rados.0.smithi071.stdout:2992: done (0 left) 2024-09-11T01:49:17.841 INFO:tasks.rados.rados.0.smithi071.stdout:2993: write oid 31 current snap is 340 2024-09-11T01:49:17.841 INFO:tasks.rados.rados.0.smithi071.stdout:2993: seq_num 1055 ranges {531385=644409,1736150=557689,3005375=569503} 2024-09-11T01:49:17.846 INFO:tasks.rados.rados.0.smithi071.stdout:2993: writing smithi07134777-31 from 531385 to 1175794 tid 1 2024-09-11T01:49:17.853 INFO:tasks.rados.rados.0.smithi071.stdout:2993: writing smithi07134777-31 from 1736150 to 2293839 tid 2 2024-09-11T01:49:17.860 INFO:tasks.rados.rados.0.smithi071.stdout:2993: writing smithi07134777-31 from 3005375 to 3574878 tid 3 2024-09-11T01:49:17.860 INFO:tasks.rados.rados.0.smithi071.stdout:2994: delete oid 30 current snap is 340 2024-09-11T01:49:17.860 INFO:tasks.rados.rados.0.smithi071.stdout:2993: finishing write tid 1 to smithi07134777-31 2024-09-11T01:49:17.860 INFO:tasks.rados.rados.0.smithi071.stdout:2993: finishing write tid 2 to smithi07134777-31 2024-09-11T01:49:17.864 INFO:tasks.rados.rados.0.smithi071.stdout:2993: finishing write tid 3 to smithi07134777-31 2024-09-11T01:49:17.866 INFO:tasks.rados.rados.0.smithi071.stdout:2994: done (1 left) 2024-09-11T01:49:17.866 INFO:tasks.rados.rados.0.smithi071.stdout:2995: copy_from oid 14 from oid 37 current snap is 340 2024-09-11T01:49:17.866 INFO:tasks.rados.rados.0.smithi071.stdout:2996: write (excl) oid 50 current snap is 340 2024-09-11T01:49:17.866 INFO:tasks.rados.rados.0.smithi071.stdout:2996: seq_num 1057 ranges {564999=511260,1853114=561358,3060221=592047} 2024-09-11T01:49:17.872 INFO:tasks.rados.rados.0.smithi071.stdout:2996: writing smithi07134777-50 from 564999 to 1076259 tid 1 2024-09-11T01:49:17.879 INFO:tasks.rados.rados.0.smithi071.stdout:2996: writing smithi07134777-50 from 1853114 to 2414472 tid 2 2024-09-11T01:49:17.887 INFO:tasks.rados.rados.0.smithi071.stdout:2996: writing smithi07134777-50 from 3060221 to 3652268 tid 3 2024-09-11T01:49:17.887 INFO:tasks.rados.rados.0.smithi071.stdout:2997: delete oid 9 current snap is 340 2024-09-11T01:49:17.887 INFO:tasks.rados.rados.0.smithi071.stdout:2993: finishing write tid 4 to smithi07134777-31 2024-09-11T01:49:17.887 INFO:tasks.rados.rados.0.smithi071.stdout:2993: finishing write tid 5 to smithi07134777-31 2024-09-11T01:49:17.887 INFO:tasks.rados.rados.0.smithi071.stdout:2993: oid 31 updating version 0 to 1791 2024-09-11T01:49:17.887 INFO:tasks.rados.rados.0.smithi071.stdout:2993: oid 31 updating version 1791 to 1792 2024-09-11T01:49:17.887 INFO:tasks.rados.rados.0.smithi071.stdout:2993: oid 31 version 1792 is already newer than 1790 2024-09-11T01:49:17.887 INFO:tasks.rados.rados.0.smithi071.stdout:2993: oid 31 version 1792 is already newer than 1789 2024-09-11T01:49:17.887 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 1792 (ObjNum 1055 snap 340 seq_num 1055) dirty exists 2024-09-11T01:49:17.887 INFO:tasks.rados.rados.0.smithi071.stdout:2993: left oid 31 (ObjNum 1055 snap 340 seq_num 1055) 2024-09-11T01:49:17.887 INFO:tasks.rados.rados.0.smithi071.stdout:2995: finishing copy_from to smithi07134777-14 2024-09-11T01:49:17.887 INFO:tasks.rados.rados.0.smithi071.stdout:2995: got expected ENOENT (src dne) 2024-09-11T01:49:17.888 INFO:tasks.rados.rados.0.smithi071.stdout:2995: finishing copy_from racing read to smithi07134777-14 2024-09-11T01:49:17.888 INFO:tasks.rados.rados.0.smithi071.stdout:2996: finishing write tid 1 to smithi07134777-50 2024-09-11T01:49:17.888 INFO:tasks.rados.rados.0.smithi071.stdout:2996: finishing write tid 2 to smithi07134777-50 2024-09-11T01:49:17.890 INFO:tasks.rados.rados.0.smithi071.stdout:2993: done (3 left) 2024-09-11T01:49:17.890 INFO:tasks.rados.rados.0.smithi071.stdout:2995: done (2 left) 2024-09-11T01:49:17.890 INFO:tasks.rados.rados.0.smithi071.stdout:2997: done (1 left) 2024-09-11T01:49:17.890 INFO:tasks.rados.rados.0.smithi071.stdout:2998: rollback oid 10 current snap is 340 2024-09-11T01:49:17.890 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 10 to 332 2024-09-11T01:49:17.890 INFO:tasks.rados.rados.0.smithi071.stdout:2999: write oid 42 current snap is 340 2024-09-11T01:49:17.890 INFO:tasks.rados.rados.0.smithi071.stdout:2999: seq_num 1059 ranges {598613=661758,1970078=565027,3115067=614591} 2024-09-11T01:49:17.897 INFO:tasks.rados.rados.0.smithi071.stdout:2999: writing smithi07134777-42 from 598613 to 1260371 tid 1 2024-09-11T01:49:17.904 INFO:tasks.rados.rados.0.smithi071.stdout:2999: writing smithi07134777-42 from 1970078 to 2535105 tid 2 2024-09-11T01:49:17.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.906+0000 7f32cf516640 10 monclient: get_monmap_and_config 2024-09-11T01:49:17.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.906+0000 7f32cf516640 10 monclient: build_initial_monmap 2024-09-11T01:49:17.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.906+0000 7f32cf516640 10 monclient: monmap: 2024-09-11T01:49:17.907 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:17.907 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:17.908 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:49:17.906927+0000 2024-09-11T01:49:17.908 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:49:17.906927+0000 2024-09-11T01:49:17.908 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:17.908 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:17.908 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:49:17.908 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:49:17.908 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:49:17.908 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:49:17.908 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:49:17.908 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:17.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.906+0000 7f32cf516640 1 Processor -- start 2024-09-11T01:49:17.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.906+0000 7f32cf516640 1 -- start start 2024-09-11T01:49:17.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.906+0000 7f32cf516640 10 monclient: init 2024-09-11T01:49:17.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cf516640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:17.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cf516640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-11T01:49:17.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cf516640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f32c8137d50 0x7f32c809d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:17.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cf516640 10 monclient(hunting): picked mon.noname-b con 0x7f32c8137d50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:17.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cf516640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f32c8138130 addr v1:172.21.15.71:6789/0 2024-09-11T01:49:17.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cf516640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f32c813d280 0x7f32c813f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:17.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cf516640 10 monclient(hunting): picked mon.noname-a con 0x7f32c813d280 addr v2:172.21.15.12:3300/0 2024-09-11T01:49:17.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cf516640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32c809ba20 con 0x7f32c8138130 2024-09-11T01:49:17.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cf516640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:17.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cf516640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f32c809c650 con 0x7f32c813d280 2024-09-11T01:49:17.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cf516640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:17.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cf516640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f32c809c370 con 0x7f32c8137d50 2024-09-11T01:49:17.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cf516640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:17.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cf516640 10 monclient(hunting): authenticate will time out at 1964.349504s 2024-09-11T01:49:17.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cdab3640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f32c813d280 0x7f32c813f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:17.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cdab3640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f32c813d280 0x7f32c813f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43758/0 (socket says 172.21.15.12:43758) 2024-09-11T01:49:17.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cdab3640 1 -- 172.21.15.12:0/2170649194 learned_addr learned my addr 172.21.15.12:0/2170649194 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:49:17.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cdab3640 10 monclient(hunting): get_auth_request con 0x7f32c813d280 auth_method 0 2024-09-11T01:49:17.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cdab3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:17.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cdab3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:17.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cdab3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:17.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cd2b2640 1 --2- 172.21.15.12:0/2170649194 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f32c8137d50 0x7f32c809d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:17.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cd2b2640 10 monclient(hunting): get_auth_request con 0x7f32c8137d50 auth_method 0 2024-09-11T01:49:17.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cd2b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:17.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cd2b2640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:17.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cd2b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:17.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cdab3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:17.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cdab3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:17.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cdab3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:17.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cd2b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:17.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cd2b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:17.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cd2b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:17.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cdab3640 10 monclient(hunting): handle_auth_done global_id 214218 payload 274 2024-09-11T01:49:17.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cdab3640 1 -- 172.21.15.12:0/2170649194 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f32c8137d50 msgr2=0x7f32c809d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:49:17.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cdab3640 1 --2- 172.21.15.12:0/2170649194 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f32c8137d50 0x7f32c809d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:17.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cdab3640 1 -- 172.21.15.12:0/2170649194 >> v1:172.21.15.71:6789/0 conn(0x7f32c8138130 legacy=0x7f32c8138550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:49:17.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cdab3640 10 monclient: _finish_hunting 0 2024-09-11T01:49:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cdab3640 1 monclient: found mon.noname-a 2024-09-11T01:49:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cdab3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cdab3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:49:17.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.907+0000 7f32cdab3640 1 -- 172.21.15.12:0/2170649194 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32c80a6ce0 con 0x7f32c813d280 2024-09-11T01:49:17.916 INFO:tasks.rados.rados.0.smithi071.stdout:2999: writing smithi07134777-42 from 3115067 to 3729658 tid 3 2024-09-11T01:49:17.916 INFO:tasks.rados.rados.0.smithi071.stdout:2996: finishing write tid 3 to smithi07134777-50 2024-09-11T01:49:17.916 INFO:tasks.rados.rados.0.smithi071.stdout:2996: finishing write tid 4 to smithi07134777-50 2024-09-11T01:49:17.916 INFO:tasks.rados.rados.0.smithi071.stdout:2996: finishing write tid 5 to smithi07134777-50 2024-09-11T01:49:17.916 INFO:tasks.rados.rados.0.smithi071.stdout:2996: oid 50 updating version 0 to 2141 2024-09-11T01:49:17.916 INFO:tasks.rados.rados.0.smithi071.stdout:2996: oid 50 version 2141 is already newer than 2140 2024-09-11T01:49:17.916 INFO:tasks.rados.rados.0.smithi071.stdout:2996: oid 50 updating version 2141 to 2142 2024-09-11T01:49:17.916 INFO:tasks.rados.rados.0.smithi071.stdout:2996: oid 50 updating version 2142 to 2143 2024-09-11T01:49:17.916 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 2143 (ObjNum 1057 snap 340 seq_num 1057) dirty exists 2024-09-11T01:49:17.916 INFO:tasks.rados.rados.0.smithi071.stdout:2996: left oid 50 (ObjNum 1057 snap 340 seq_num 1057) 2024-09-11T01:49:17.916 INFO:tasks.rados.rados.0.smithi071.stdout:2998: finishing rollback tid 1 to smithi07134777-10 2024-09-11T01:49:17.917 INFO:tasks.rados.rados.0.smithi071.stdout:2998: finishing rollback tid 2 to smithi07134777-10 2024-09-11T01:49:17.917 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 1086 (ObjNum 949 snap 300 seq_num 949) dirty exists 2024-09-11T01:49:17.917 INFO:tasks.rados.rados.0.smithi071.stdout:2999: finishing write tid 1 to smithi07134777-42 2024-09-11T01:49:17.917 INFO:tasks.rados.rados.0.smithi071.stdout:2999: finishing write tid 2 to smithi07134777-42 2024-09-11T01:49:17.917 INFO:tasks.rados.rados.0.smithi071.stdout:3000: read oid 4 snap 330 2024-09-11T01:49:17.917 INFO:tasks.rados.rados.0.smithi071.stdout:3000: expect (ObjNum 828 snap 252 seq_num 828) 2024-09-11T01:49:17.917 INFO:tasks.rados.rados.0.smithi071.stdout:2996: done (3 left) 2024-09-11T01:49:17.917 INFO:tasks.rados.rados.0.smithi071.stdout:2998: done (2 left) 2024-09-11T01:49:17.917 INFO:tasks.rados.rados.0.smithi071.stdout:3001: write oid 24 current snap is 340 2024-09-11T01:49:17.917 INFO:tasks.rados.rados.0.smithi071.stdout:3001: seq_num 1060 ranges {615420=737007,1912207=425038,3026137=542216} 2024-09-11T01:49:17.921 INFO:tasks.rados.rados.0.smithi071.stdout:3001: writing smithi07134777-24 from 615420 to 1352427 tid 1 2024-09-11T01:49:17.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.910+0000 7f32cdab3640 1 -- 172.21.15.12:0/2170649194 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32c813d280 msgr2=0x7f32c813f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:17.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.910+0000 7f32cdab3640 1 --2- 172.21.15.12:0/2170649194 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32c813d280 0x7f32c813f670 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f32c809c7d0 tx=0x7f32c002a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad327952e4c452 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:17.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.910+0000 7f32bffff640 1 -- 172.21.15.12:0/2170649194 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c0036070 con 0x7f32c813d280 2024-09-11T01:49:17.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.910+0000 7f32bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:17.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.910+0000 7f32bffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:49:17.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.910+0000 7f32bffff640 10 monclient: dump: 2024-09-11T01:49:17.923 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:17.923 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:17.923 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:17.923 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:17.923 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:17.923 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:17.923 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:17.923 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:17.923 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:17.924 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:17.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.910+0000 7f32bffff640 10 monclient: _finish_auth 0 2024-09-11T01:49:17.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.910+0000 7f32bffff640 10 monclient: _check_auth_tickets 2024-09-11T01:49:17.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.910+0000 7f32bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:17.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.910+0000 7f32bffff640 1 -- 172.21.15.12:0/2170649194 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32c0031600 con 0x7f32c813d280 2024-09-11T01:49:17.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.910+0000 7f32cf516640 5 monclient: authenticate success, global_id 214218 2024-09-11T01:49:17.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.910+0000 7f32cf516640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:49:17.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.910+0000 7f32bffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:17.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.910+0000 7f32bffff640 1 -- 172.21.15.12:0/2170649194 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c00319d0 con 0x7f32c813d280 2024-09-11T01:49:17.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.910+0000 7f32bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:17.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.910+0000 7f32bffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:17.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.910+0000 7f32bffff640 10 monclient: dump: 2024-09-11T01:49:17.924 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:17.925 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:17.925 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:17.925 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:17.925 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:17.925 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:17.925 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:17.925 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:17.925 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:17.925 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:17.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.910+0000 7f32cf516640 10 monclient: get_monmap_and_config success 2024-09-11T01:49:17.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.910+0000 7f32cf516640 4 set_mon_vals no callback set 2024-09-11T01:49:17.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.911+0000 7f32cf516640 10 monclient: shutdown 2024-09-11T01:49:17.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.911+0000 7f32cf516640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.911+0000 7f32cf516640 1 -- 172.21.15.12:0/2170649194 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32c813d280 msgr2=0x7f32c813f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.911+0000 7f32cf516640 1 --2- 172.21.15.12:0/2170649194 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32c813d280 0x7f32c813f670 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f32c809c7d0 tx=0x7f32c002a5d0 comp rx=0 tx=0).stop 2024-09-11T01:49:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.911+0000 7f32cf516640 1 -- 172.21.15.12:0/2170649194 shutdown_connections 2024-09-11T01:49:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.911+0000 7f32cf516640 1 --2- 172.21.15.12:0/2170649194 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32c813d280 0x7f32c813f670 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.911+0000 7f32cf516640 1 --2- 172.21.15.12:0/2170649194 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f32c8137d50 0x7f32c809d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.911+0000 7f32cf516640 1 -- 172.21.15.12:0/2170649194 >> 172.21.15.12:0/2170649194 conn(0x7f32c80a7020 msgr2=0x7f32c80a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.912+0000 7f32cf516640 1 -- 172.21.15.12:0/2170649194 shutdown_connections 2024-09-11T01:49:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.912+0000 7f32cf516640 1 -- 172.21.15.12:0/2170649194 wait complete. 2024-09-11T01:49:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.913+0000 7f32cf516640 10 monclient: build_initial_monmap 2024-09-11T01:49:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.913+0000 7f32cf516640 10 monclient: monmap: 2024-09-11T01:49:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:49:17.926 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:49:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:49:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:49:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:49:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:49:17.927 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.913+0000 7f32cf516640 1 Processor -- start 2024-09-11T01:49:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.913+0000 7f32cf516640 1 -- start start 2024-09-11T01:49:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.913+0000 7f32cf516640 10 monclient: init 2024-09-11T01:49:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cf516640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cf516640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:49:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cf516640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32c8137d50 0x7f32c8136be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cf516640 10 monclient(hunting): picked mon.noname-a con 0x7f32c8137d50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:49:17.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cf516640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f32c8138130 0x7f32c8139f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:17.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cf516640 10 monclient(hunting): picked mon.noname-c con 0x7f32c8138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:17.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cf516640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f32c813d280 0x7f32c813a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:17.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cf516640 10 monclient(hunting): picked mon.noname-b con 0x7f32c813d280 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:49:17.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cf516640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:17.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cf516640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f32c809d7d0 con 0x7f32c8137d50 2024-09-11T01:49:17.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cf516640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:17.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cf516640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f32c809d650 con 0x7f32c813d280 2024-09-11T01:49:17.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cf516640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:17.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cf516640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f32c813fbb0 con 0x7f32c8138130 2024-09-11T01:49:17.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cf516640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:17.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cf516640 10 monclient(hunting): authenticate will time out at 1964.356389s 2024-09-11T01:49:17.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cd2b2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32c8137d50 0x7f32c8136be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cd2b2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32c8137d50 0x7f32c8136be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43772/0 (socket says 172.21.15.12:43772) 2024-09-11T01:49:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cd2b2640 1 -- 172.21.15.12:0/56445919 learned_addr learned my addr 172.21.15.12:0/56445919 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:49:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cd2b2640 10 monclient(hunting): get_auth_request con 0x7f32c8137d50 auth_method 0 2024-09-11T01:49:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cd2b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cd2b2640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cd2b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32ccab1640 1 --2- 172.21.15.12:0/56445919 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f32c8138130 0x7f32c8139f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32ccab1640 10 monclient(hunting): get_auth_request con 0x7f32c8138130 auth_method 0 2024-09-11T01:49:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32ccab1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32ccab1640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32ccab1640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cdab3640 1 --2- 172.21.15.12:0/56445919 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f32c813d280 0x7f32c813a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:17.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cdab3640 10 monclient(hunting): get_auth_request con 0x7f32c813d280 auth_method 0 2024-09-11T01:49:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cdab3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cdab3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cdab3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cd2b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cd2b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cd2b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32ccab1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32ccab1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32ccab1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cdab3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cdab3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cdab3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cd2b2640 10 monclient(hunting): handle_auth_done global_id 214221 payload 824 2024-09-11T01:49:17.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cd2b2640 1 -- 172.21.15.12:0/56445919 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f32c8138130 msgr2=0x7f32c8139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cd2b2640 1 --2- 172.21.15.12:0/56445919 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f32c8138130 0x7f32c8139f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.914+0000 7f32cd2b2640 1 -- 172.21.15.12:0/56445919 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f32c813d280 msgr2=0x7f32c813a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.915+0000 7f32cd2b2640 1 --2- 172.21.15.12:0/56445919 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f32c813d280 0x7f32c813a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.915+0000 7f32cd2b2640 10 monclient: _finish_hunting 0 2024-09-11T01:49:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.915+0000 7f32cd2b2640 1 monclient: found mon.noname-a 2024-09-11T01:49:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.915+0000 7f32cd2b2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.915+0000 7f32cd2b2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:49:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.915+0000 7f32cd2b2640 1 -- 172.21.15.12:0/56445919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32c80a9430 con 0x7f32c8137d50 2024-09-11T01:49:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.915+0000 7f32ccab1640 1 --2- 172.21.15.12:0/56445919 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f32c8138130 0x7f32c8139f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:49:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.917+0000 7f32cd2b2640 1 -- 172.21.15.12:0/56445919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32c8137d50 msgr2=0x7f32c8136be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.917+0000 7f32cd2b2640 1 --2- 172.21.15.12:0/56445919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32c8137d50 0x7f32c8136be0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f32c400e9b0 tx=0x7f32c400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.917+0000 7f32bdffb640 1 -- 172.21.15.12:0/56445919 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c4017070 con 0x7f32c8137d50 2024-09-11T01:49:17.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.917+0000 7f32bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.917+0000 7f32bdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:49:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.917+0000 7f32bdffb640 10 monclient: dump: 2024-09-11T01:49:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:17.932 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.917+0000 7f32bdffb640 10 monclient: _finish_auth 0 2024-09-11T01:49:17.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.917+0000 7f32bdffb640 10 monclient: _check_auth_tickets 2024-09-11T01:49:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.917+0000 7f32bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.917+0000 7f32bdffb640 1 -- 172.21.15.12:0/56445919 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32c400cbb0 con 0x7f32c8137d50 2024-09-11T01:49:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.917+0000 7f32bdffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.917+0000 7f32bdffb640 1 -- 172.21.15.12:0/56445919 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c4013470 con 0x7f32c8137d50 2024-09-11T01:49:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.917+0000 7f32cf516640 5 monclient: authenticate success, global_id 214221 2024-09-11T01:49:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.917+0000 7f32cf516640 10 monclient: _renew_subs 2024-09-11T01:49:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.917+0000 7f32cf516640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.917+0000 7f32cf516640 1 -- 172.21.15.12:0/56445919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32c81213f0 con 0x7f32c8137d50 2024-09-11T01:49:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.917+0000 7f32bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.917+0000 7f32bdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.917+0000 7f32bdffb640 10 monclient: dump: 2024-09-11T01:49:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:17.933 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:17.934 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.917+0000 7f32bf7fe640 4 set_mon_vals no callback set 2024-09-11T01:49:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.917+0000 7f32cf516640 10 monclient: _renew_subs 2024-09-11T01:49:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.917+0000 7f32cf516640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.917+0000 7f32cf516640 1 -- 172.21.15.12:0/56445919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32c80a1710 con 0x7f32c8137d50 2024-09-11T01:49:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.918+0000 7f32cf516640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.918+0000 7f32cf516640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:17.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.918+0000 7f32cf516640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:17.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.918+0000 7f32bdffb640 1 -- 172.21.15.12:0/56445919 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f32c4024060 con 0x7f32c8137d50 2024-09-11T01:49:17.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.918+0000 7f32cf516640 1 -- 172.21.15.12:0/56445919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32c809d110 con 0x7f32c8137d50 2024-09-11T01:49:17.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.919+0000 7f32bdffb640 1 --2- 172.21.15.12:0/56445919 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f32ac03e260 0x7f32ac040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:17.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.919+0000 7f32bdffb640 1 -- 172.21.15.12:0/56445919 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(675..675 src has 507..675) ==== 4819+0+0 (secure 0 0 0) 0x7f32c4052db0 con 0x7f32c8137d50 2024-09-11T01:49:17.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.919+0000 7f32ccab1640 1 --2- 172.21.15.12:0/56445919 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f32ac03e260 0x7f32ac040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:17.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.919+0000 7f32ccab1640 10 monclient: get_auth_request con 0x7f32ac03e260 auth_method 0 2024-09-11T01:49:17.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.922+0000 7f32ccab1640 1 --2- 172.21.15.12:0/56445919 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f32ac03e260 0x7f32ac040720 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f32b8004320 tx=0x7f32b8007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:17.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.924+0000 7f32bdffb640 1 -- 172.21.15.12:0/56445919 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f32c80a9430 con 0x7f32c8137d50 2024-09-11T01:49:17.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.924+0000 7f32bdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:17.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:17.924+0000 7f32bdffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:49:17.936 INFO:tasks.rados.rados.0.smithi071.stdout:3001: writing smithi07134777-24 from 1912207 to 2337245 tid 2 2024-09-11T01:49:17.937 INFO:tasks.rados.rados.0.smithi071.stdout:3001: writing smithi07134777-24 from 3026137 to 3568353 tid 3 2024-09-11T01:49:17.937 INFO:tasks.rados.rados.0.smithi071.stdout:3002: write oid 44 current snap is 340 2024-09-11T01:49:17.937 INFO:tasks.rados.rados.0.smithi071.stdout:2999: finishing write tid 3 to smithi07134777-42 2024-09-11T01:49:17.937 INFO:tasks.rados.rados.0.smithi071.stdout:3002: seq_num 1061 ranges {632227=412256,1570689=568696,2653560=753488} 2024-09-11T01:49:17.937 INFO:tasks.rados.rados.0.smithi071.stdout:3002: writing smithi07134777-44 from 632227 to 1044483 tid 1 2024-09-11T01:49:17.943 INFO:tasks.rados.rados.0.smithi071.stdout:3002: writing smithi07134777-44 from 1570689 to 2139385 tid 2 2024-09-11T01:49:17.950 INFO:tasks.rados.rados.0.smithi071.stdout:3002: writing smithi07134777-44 from 2653560 to 3407048 tid 3 2024-09-11T01:49:17.950 INFO:tasks.rados.rados.0.smithi071.stdout:3003: read oid 49 snap -1 2024-09-11T01:49:17.950 INFO:tasks.rados.rados.0.smithi071.stdout:3003: expect (ObjNum 1017 snap 322 seq_num 1017) 2024-09-11T01:49:17.950 INFO:tasks.rados.rados.0.smithi071.stdout:3004: read oid 2 snap 337 2024-09-11T01:49:17.951 INFO:tasks.rados.rados.0.smithi071.stdout:3004: expect deleted 2024-09-11T01:49:17.951 INFO:tasks.rados.rados.0.smithi071.stdout:3005: write oid 16 current snap is 340 2024-09-11T01:49:17.951 INFO:tasks.rados.rados.0.smithi071.stdout:2999: finishing write tid 4 to smithi07134777-42 2024-09-11T01:49:17.951 INFO:tasks.rados.rados.0.smithi071.stdout:3005: seq_num 1062 ranges {649034=487505,1629171=428707,2680983=564760,3798768=50266} 2024-09-11T01:49:17.957 INFO:tasks.rados.rados.0.smithi071.stdout:3005: writing smithi07134777-16 from 649034 to 1136539 tid 1 2024-09-11T01:49:17.963 INFO:tasks.rados.rados.0.smithi071.stdout:3005: writing smithi07134777-16 from 1629171 to 2057878 tid 2 2024-09-11T01:49:17.969 INFO:tasks.rados.rados.0.smithi071.stdout:3005: writing smithi07134777-16 from 2680983 to 3245743 tid 3 2024-09-11T01:49:17.973 INFO:tasks.rados.rados.0.smithi071.stdout:3005: writing smithi07134777-16 from 3798768 to 3849034 tid 4 2024-09-11T01:49:17.973 INFO:tasks.rados.rados.0.smithi071.stdout:3006: read oid 37 snap 250 2024-09-11T01:49:17.973 INFO:tasks.rados.rados.0.smithi071.stdout:3006: expect (ObjNum 723 snap 224 seq_num 723) 2024-09-11T01:49:17.973 INFO:tasks.rados.rados.0.smithi071.stdout:3007: read oid 9 snap -1 2024-09-11T01:49:17.973 INFO:tasks.rados.rados.0.smithi071.stdout:3007: expect deleted 2024-09-11T01:49:17.974 INFO:tasks.rados.rados.0.smithi071.stdout:3008: snap_remove snap 320 2024-09-11T01:49:18.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:18.191+0000 7f32cf516640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:49:18.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:18.191+0000 7f32cf516640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:49:18.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:18.191+0000 7f32cf516640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:18.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:18.191+0000 7f32cf516640 1 -- 172.21.15.12:0/56445919 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f32c8074a40 con 0x7f32c8137d50 2024-09-11T01:49:18.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:18.192+0000 7f32bdffb640 1 -- 172.21.15.12:0/56445919 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f32c4051780 con 0x7f32c8137d50 2024-09-11T01:49:18.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:18.192+0000 7f32bdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:49:18.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:18.192+0000 7f32bdffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:49:18.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.192+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.192+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=11,logm=50} crc {auth=3611130140,config=2586702516,health=2091932314,kv=1918975338,logm=1661014372}) 2024-09-11T01:49:18.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.192+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=12,kv=11,logm=48} crc {auth=2799074894,config=2586702516,health=3041450585,kv=2372998920,logm=2341450654}) 2024-09-11T01:49:18.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.192+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.192+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=11,logm=50} crc {auth=3611130140,config=2586702516,health=2091932314,kv=1918975338,logm=1661014372}) 2024-09-11T01:49:18.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.192+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=12,logm=46} crc {auth=3125129841,config=2586702516,health=3041450585,kv=3263697420,logm=3109365365}) 2024-09-11T01:49:18.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.193+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.193+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=401729266}) 2024-09-11T01:49:18.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.193+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3462919363}) 2024-09-11T01:49:18.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.193+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.193+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=401729266}) 2024-09-11T01:49:18.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.193+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3522214364}) 2024-09-11T01:49:18.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:18.194+0000 7f32cf516640 1 -- 172.21.15.12:0/56445919 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f32ac03e260 msgr2=0x7f32ac040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:18.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:18.194+0000 7f32cf516640 1 --2- 172.21.15.12:0/56445919 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f32ac03e260 0x7f32ac040720 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f32b8004320 tx=0x7f32b8007e30 comp rx=0 tx=0).stop 2024-09-11T01:49:18.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:18.194+0000 7f32cf516640 10 monclient: shutdown 2024-09-11T01:49:18.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:18.194+0000 7f32cf516640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:18.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:18.194+0000 7f32cf516640 1 -- 172.21.15.12:0/56445919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32c8137d50 msgr2=0x7f32c8136be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:18.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:18.194+0000 7f32cf516640 1 --2- 172.21.15.12:0/56445919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32c8137d50 0x7f32c8136be0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f32c400e9b0 tx=0x7f32c400ee70 comp rx=0 tx=0).stop 2024-09-11T01:49:18.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:18.195+0000 7f32cf516640 1 -- 172.21.15.12:0/56445919 shutdown_connections 2024-09-11T01:49:18.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:18.195+0000 7f32cf516640 1 --2- 172.21.15.12:0/56445919 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f32c813d280 0x7f32c813a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:18.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:18.195+0000 7f32cf516640 1 --2- 172.21.15.12:0/56445919 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f32c8138130 0x7f32c8139f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:18.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:18.195+0000 7f32cf516640 1 --2- 172.21.15.12:0/56445919 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f32ac03e260 0x7f32ac040720 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:18.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:18.195+0000 7f32cf516640 1 --2- 172.21.15.12:0/56445919 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f32c8137d50 0x7f32c8136be0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:18.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:18.195+0000 7f32cf516640 1 -- 172.21.15.12:0/56445919 >> 172.21.15.12:0/56445919 conn(0x7f32c80a7020 msgr2=0x7f32c813e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:18.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:18.195+0000 7f32cf516640 1 -- 172.21.15.12:0/56445919 shutdown_connections 2024-09-11T01:49:18.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:18.195+0000 7f32cf516640 1 -- 172.21.15.12:0/56445919 wait complete. 2024-09-11T01:49:18.207 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:49:18.246 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.244+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.247 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.244+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=218836379}) 2024-09-11T01:49:18.247 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.244+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1036543794}) 2024-09-11T01:49:18.247 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.245+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.247 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.245+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=218836379}) 2024-09-11T01:49:18.247 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.245+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3420788226}) 2024-09-11T01:49:18.248 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.245+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.248 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.246+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2014399978}) 2024-09-11T01:49:18.248 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.246+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=400820715}) 2024-09-11T01:49:18.248 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.246+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.248 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.246+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2014399978}) 2024-09-11T01:49:18.248 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.246+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2890571113}) 2024-09-11T01:49:18.249 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.247+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.249 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.247+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3897329694}) 2024-09-11T01:49:18.249 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.247+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2310775270}) 2024-09-11T01:49:18.249 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.247+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.249 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.247+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3897329694}) 2024-09-11T01:49:18.249 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.247+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=850251416}) 2024-09-11T01:49:18.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.817+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.817+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=95020114}) 2024-09-11T01:49:18.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.817+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2150481667}) 2024-09-11T01:49:18.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.817+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.817+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=95020114}) 2024-09-11T01:49:18.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.817+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2685047055}) 2024-09-11T01:49:18.822 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.820+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.822 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.820+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1548384263}) 2024-09-11T01:49:18.822 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.820+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3523078889}) 2024-09-11T01:49:18.822 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.820+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.822 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.820+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1548384263}) 2024-09-11T01:49:18.822 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.820+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1007694342}) 2024-09-11T01:49:18.826 INFO:tasks.rados.rados.0.smithi071.stdout:3008: done (9 left) 2024-09-11T01:49:18.826 INFO:tasks.rados.rados.0.smithi071.stdout:3009: read oid 11 snap -1 2024-09-11T01:49:18.826 INFO:tasks.rados.rados.0.smithi071.stdout:3009: expect (ObjNum 987 snap 317 seq_num 987) 2024-09-11T01:49:18.827 INFO:tasks.rados.rados.0.smithi071.stdout:3010: read oid 1 snap 252 2024-09-11T01:49:18.827 INFO:tasks.rados.rados.0.smithi071.stdout:3010: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:49:18.827 INFO:tasks.rados.rados.0.smithi071.stdout:3011: rollback oid 46 current snap is 340 2024-09-11T01:49:18.827 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 46 to 299 2024-09-11T01:49:18.827 INFO:tasks.rados.rados.0.smithi071.stdout:2999: finishing write tid 5 to smithi07134777-42 2024-09-11T01:49:18.827 INFO:tasks.rados.rados.0.smithi071.stdout:2999: oid 42 updating version 0 to 882 2024-09-11T01:49:18.827 INFO:tasks.rados.rados.0.smithi071.stdout:2999: oid 42 updating version 882 to 884 2024-09-11T01:49:18.827 INFO:tasks.rados.rados.0.smithi071.stdout:2999: oid 42 version 884 is already newer than 883 2024-09-11T01:49:18.827 INFO:tasks.rados.rados.0.smithi071.stdout:2999: oid 42 version 884 is already newer than 881 2024-09-11T01:49:18.827 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 884 (ObjNum 1059 snap 340 seq_num 1059) dirty exists 2024-09-11T01:49:18.827 INFO:tasks.rados.rados.0.smithi071.stdout:2999: left oid 42 (ObjNum 1059 snap 340 seq_num 1059) 2024-09-11T01:49:18.827 INFO:tasks.rados.rados.0.smithi071.stdout:3000: expect (ObjNum 828 snap 252 seq_num 828) 2024-09-11T01:49:18.827 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:18.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.829+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.829+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1080571418}) 2024-09-11T01:49:18.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.829+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3681352476}) 2024-09-11T01:49:18.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.829+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.829+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1080571418}) 2024-09-11T01:49:18.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.829+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1950967735}) 2024-09-11T01:49:18.841 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.839+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.842 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.840+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1393202289}) 2024-09-11T01:49:18.843 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.840+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=973421606}) 2024-09-11T01:49:18.843 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.840+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.843 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.840+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1393202289}) 2024-09-11T01:49:18.843 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.840+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=519131690}) 2024-09-11T01:49:18.860 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.856+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.860 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.857+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=174098585}) 2024-09-11T01:49:18.860 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.857+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1833646817}) 2024-09-11T01:49:18.860 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.857+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.860 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.857+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=174098585}) 2024-09-11T01:49:18.860 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.857+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=798266591}) 2024-09-11T01:49:18.866 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.865+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.866 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.865+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2111140200}) 2024-09-11T01:49:18.866 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.865+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3937064255}) 2024-09-11T01:49:18.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.865+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.865+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2111140200}) 2024-09-11T01:49:18.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.865+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2811957265}) 2024-09-11T01:49:18.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.866+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.866+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3242552723}) 2024-09-11T01:49:18.868 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.866+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=62142818}) 2024-09-11T01:49:18.868 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.866+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.868 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.866+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3242552723}) 2024-09-11T01:49:18.868 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.866+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4292210191}) 2024-09-11T01:49:18.869 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.867+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.869 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.867+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1127979999}) 2024-09-11T01:49:18.869 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.867+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1418830854}) 2024-09-11T01:49:18.869 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.867+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.869 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.867+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1127979999}) 2024-09-11T01:49:18.869 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.867+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4036750290}) 2024-09-11T01:49:18.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.868+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.868+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1748896222}) 2024-09-11T01:49:18.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.868+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3357475553}) 2024-09-11T01:49:18.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.868+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.868+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1748896222}) 2024-09-11T01:49:18.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.868+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1739882021}) 2024-09-11T01:49:18.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.869+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.869+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2894688961}) 2024-09-11T01:49:18.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.869+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1176998694}) 2024-09-11T01:49:18.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.869+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.869+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2894688961}) 2024-09-11T01:49:18.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.869+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1088835565}) 2024-09-11T01:49:18.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.870+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.870+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2245531429}) 2024-09-11T01:49:18.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.870+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1315392699}) 2024-09-11T01:49:18.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.870+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.870+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2245531429}) 2024-09-11T01:49:18.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.870+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2215543187}) 2024-09-11T01:49:18.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.870+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.870+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4220628666}) 2024-09-11T01:49:18.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.870+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=918729416}) 2024-09-11T01:49:18.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.870+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.870+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4220628666}) 2024-09-11T01:49:18.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.870+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4086001316}) 2024-09-11T01:49:18.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.871+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.871+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=152881034}) 2024-09-11T01:49:18.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.871+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1772111583}) 2024-09-11T01:49:18.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.871+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.871+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=152881034}) 2024-09-11T01:49:18.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.871+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4243569282}) 2024-09-11T01:49:18.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.872+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.872+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1603710069}) 2024-09-11T01:49:18.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.872+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=672566536}) 2024-09-11T01:49:18.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.872+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.875 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.872+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1603710069}) 2024-09-11T01:49:18.875 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.872+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2374141182}) 2024-09-11T01:49:18.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.873+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.873+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1160305847}) 2024-09-11T01:49:18.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.873+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=309937137}) 2024-09-11T01:49:18.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.873+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.873+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1160305847}) 2024-09-11T01:49:18.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.873+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=424397577}) 2024-09-11T01:49:18.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.873+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.873+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3059852588}) 2024-09-11T01:49:18.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.873+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=992475296}) 2024-09-11T01:49:18.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.873+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.873+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3059852588}) 2024-09-11T01:49:18.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.873+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2527364151}) 2024-09-11T01:49:18.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.874+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.874+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4221245415}) 2024-09-11T01:49:18.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.874+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1606242306}) 2024-09-11T01:49:18.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.874+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.874+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4221245415}) 2024-09-11T01:49:18.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.874+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1663162667}) 2024-09-11T01:49:18.879 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.875+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.879 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.875+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2493197783}) 2024-09-11T01:49:18.879 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.875+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2478751621}) 2024-09-11T01:49:18.879 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.875+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.879 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.875+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2493197783}) 2024-09-11T01:49:18.879 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.875+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2936632364}) 2024-09-11T01:49:18.879 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.876+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.879 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.876+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2200587311}) 2024-09-11T01:49:18.879 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.876+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3873259527}) 2024-09-11T01:49:18.879 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.876+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.879 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.876+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2200587311}) 2024-09-11T01:49:18.879 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.876+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3243781736}) 2024-09-11T01:49:18.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.876+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.876+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=857805382}) 2024-09-11T01:49:18.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.876+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1128198546}) 2024-09-11T01:49:18.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.876+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.876+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=857805382}) 2024-09-11T01:49:18.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.876+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1076184378}) 2024-09-11T01:49:18.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.877+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.877+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178734805}) 2024-09-11T01:49:18.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.877+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1379454969}) 2024-09-11T01:49:18.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.877+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.877+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178734805}) 2024-09-11T01:49:18.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.877+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3713530476}) 2024-09-11T01:49:18.882 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.878+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.882 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.878+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1297632169}) 2024-09-11T01:49:18.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.878+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1120933597}) 2024-09-11T01:49:18.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.878+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.878+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1297632169}) 2024-09-11T01:49:18.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.878+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=943451686}) 2024-09-11T01:49:18.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.879+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.879+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=40986353}) 2024-09-11T01:49:18.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.879+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3158043943}) 2024-09-11T01:49:18.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.879+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.879+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=40986353}) 2024-09-11T01:49:18.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.879+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3721378941}) 2024-09-11T01:49:18.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.879+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.879+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=734813925}) 2024-09-11T01:49:18.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.879+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3888328353}) 2024-09-11T01:49:18.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.879+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.879+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=734813925}) 2024-09-11T01:49:18.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.879+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3302306957}) 2024-09-11T01:49:18.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.880+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.880+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3207819196}) 2024-09-11T01:49:18.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.880+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=935654404}) 2024-09-11T01:49:18.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.880+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.880+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3207819196}) 2024-09-11T01:49:18.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.880+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=304989627}) 2024-09-11T01:49:18.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.881+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.881+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=126911060}) 2024-09-11T01:49:18.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.881+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2888961789}) 2024-09-11T01:49:18.885 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.881+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.885 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.881+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=126911060}) 2024-09-11T01:49:18.885 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.881+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1671900271}) 2024-09-11T01:49:18.887 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.882+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.887 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.882+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2116365652}) 2024-09-11T01:49:18.887 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.882+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2976438310}) 2024-09-11T01:49:18.887 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.882+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.887 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.882+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2116365652}) 2024-09-11T01:49:18.887 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.882+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3400548634}) 2024-09-11T01:49:18.887 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.882+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.887 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.882+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2232995312}) 2024-09-11T01:49:18.887 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.882+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4136336535}) 2024-09-11T01:49:18.887 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.882+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.887 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.882+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2232995312}) 2024-09-11T01:49:18.887 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.882+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1011042361}) 2024-09-11T01:49:18.888 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.883+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.888 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.883+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3954241480}) 2024-09-11T01:49:18.888 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.883+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4275315829}) 2024-09-11T01:49:18.888 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.883+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.888 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.883+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3954241480}) 2024-09-11T01:49:18.888 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.883+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4097493087}) 2024-09-11T01:49:18.888 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.884+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.888 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.884+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1890492}) 2024-09-11T01:49:18.888 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.884+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4245973121}) 2024-09-11T01:49:18.888 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.884+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.888 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.884+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1890492}) 2024-09-11T01:49:18.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.884+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2970700256}) 2024-09-11T01:49:18.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.885+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.885+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1070931083}) 2024-09-11T01:49:18.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.885+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3499985640}) 2024-09-11T01:49:18.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.885+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.885+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1070931083}) 2024-09-11T01:49:18.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.885+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3095800325}) 2024-09-11T01:49:18.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.885+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.885+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644291173}) 2024-09-11T01:49:18.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.885+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1149230267}) 2024-09-11T01:49:18.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.885+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.885+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644291173}) 2024-09-11T01:49:18.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.885+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=502839900}) 2024-09-11T01:49:18.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.886+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.886+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4077278566}) 2024-09-11T01:49:18.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.886+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3883342368}) 2024-09-11T01:49:18.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.886+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.886+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4077278566}) 2024-09-11T01:49:18.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.886+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2245659866}) 2024-09-11T01:49:18.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.887+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.887+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=424391543}) 2024-09-11T01:49:18.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.887+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3545351031}) 2024-09-11T01:49:18.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.887+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.887+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=424391543}) 2024-09-11T01:49:18.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.887+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4221934791}) 2024-09-11T01:49:18.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.888+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.888+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3130293903}) 2024-09-11T01:49:18.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.888+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1036522271}) 2024-09-11T01:49:18.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.888+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.888+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3130293903}) 2024-09-11T01:49:18.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.888+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1570665908}) 2024-09-11T01:49:18.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.889+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.889+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3076200132}) 2024-09-11T01:49:18.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.889+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=199855117}) 2024-09-11T01:49:18.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.889+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.889+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3076200132}) 2024-09-11T01:49:18.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.889+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=472763685}) 2024-09-11T01:49:18.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.889+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.889+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4169033109}) 2024-09-11T01:49:18.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.889+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=259093048}) 2024-09-11T01:49:18.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.889+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.889+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4169033109}) 2024-09-11T01:49:18.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.889+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1635650750}) 2024-09-11T01:49:18.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.890+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.895 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.890+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3014969002}) 2024-09-11T01:49:18.895 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.890+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2979042451}) 2024-09-11T01:49:18.895 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.890+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.895 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.890+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3014969002}) 2024-09-11T01:49:18.895 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.890+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3519356968}) 2024-09-11T01:49:18.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.891+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.891+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3881704739}) 2024-09-11T01:49:18.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.891+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3413318459}) 2024-09-11T01:49:18.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.891+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.891+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3881704739}) 2024-09-11T01:49:18.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.891+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4095500347}) 2024-09-11T01:49:18.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.892+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.892+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=296943017}) 2024-09-11T01:49:18.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.892+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=432548836}) 2024-09-11T01:49:18.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.892+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.892+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=296943017}) 2024-09-11T01:49:18.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.892+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3918231601}) 2024-09-11T01:49:18.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.892+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.892+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1592942139}) 2024-09-11T01:49:18.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.892+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=827571101}) 2024-09-11T01:49:18.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.892+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.892+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1592942139}) 2024-09-11T01:49:18.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.892+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1319281450}) 2024-09-11T01:49:18.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.893+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.893+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3322472452}) 2024-09-11T01:49:18.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.893+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2221499406}) 2024-09-11T01:49:18.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.893+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.893+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3322472452}) 2024-09-11T01:49:18.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.893+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3319148423}) 2024-09-11T01:49:18.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.894+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.894+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4144295375}) 2024-09-11T01:49:18.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.894+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1104859167}) 2024-09-11T01:49:18.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.894+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.894+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4144295375}) 2024-09-11T01:49:18.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.894+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4229100380}) 2024-09-11T01:49:18.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.895+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.895+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1683671965}) 2024-09-11T01:49:18.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.895+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2353637454}) 2024-09-11T01:49:18.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.895+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.895+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1683671965}) 2024-09-11T01:49:18.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.895+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2917626453}) 2024-09-11T01:49:18.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.895+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.895+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3941189030}) 2024-09-11T01:49:18.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.895+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2561791730}) 2024-09-11T01:49:18.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.895+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.895+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3941189030}) 2024-09-11T01:49:18.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.895+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3698690404}) 2024-09-11T01:49:18.903 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.896+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.903 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.896+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1677310581}) 2024-09-11T01:49:18.903 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.896+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=7229542}) 2024-09-11T01:49:18.903 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.896+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.896+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1677310581}) 2024-09-11T01:49:18.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.896+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2527328292}) 2024-09-11T01:49:18.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.897+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.897+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1529632062}) 2024-09-11T01:49:18.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.897+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1936970409}) 2024-09-11T01:49:18.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.897+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.897+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1529632062}) 2024-09-11T01:49:18.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.897+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2779529381}) 2024-09-11T01:49:18.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.898+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.898+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4282755249}) 2024-09-11T01:49:18.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.898+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1179828150}) 2024-09-11T01:49:18.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.898+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.898+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4282755249}) 2024-09-11T01:49:18.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.898+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3022903558}) 2024-09-11T01:49:18.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.898+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.898+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3587466781}) 2024-09-11T01:49:18.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.898+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=394445501}) 2024-09-11T01:49:18.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.898+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.899+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3587466781}) 2024-09-11T01:49:18.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.899+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3380377943}) 2024-09-11T01:49:18.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.899+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.899+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2529833339}) 2024-09-11T01:49:18.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.899+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1131447518}) 2024-09-11T01:49:18.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.899+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.899+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2529833339}) 2024-09-11T01:49:18.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.899+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=956518361}) 2024-09-11T01:49:18.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.900+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.900+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3002058435}) 2024-09-11T01:49:18.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.900+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1270710555}) 2024-09-11T01:49:18.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.900+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.900+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3002058435}) 2024-09-11T01:49:18.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.900+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3630727699}) 2024-09-11T01:49:18.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.901+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.901+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=237199572}) 2024-09-11T01:49:18.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.901+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=996170729}) 2024-09-11T01:49:18.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.901+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.901+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=237199572}) 2024-09-11T01:49:18.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.901+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4028175473}) 2024-09-11T01:49:18.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.902+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.902+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1296921723}) 2024-09-11T01:49:18.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.902+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=573799978}) 2024-09-11T01:49:18.907 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.902+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.907 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.902+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1296921723}) 2024-09-11T01:49:18.907 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.902+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=18644122}) 2024-09-11T01:49:18.908 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.902+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.908 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.902+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2359471261}) 2024-09-11T01:49:18.908 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.902+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=63632486}) 2024-09-11T01:49:18.908 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.902+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.908 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.902+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2359471261}) 2024-09-11T01:49:18.908 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.902+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=535708374}) 2024-09-11T01:49:18.908 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.903+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.908 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.903+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2090628385}) 2024-09-11T01:49:18.908 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.903+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2386147939}) 2024-09-11T01:49:18.909 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.903+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.909 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.903+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2090628385}) 2024-09-11T01:49:18.909 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.903+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=246064851}) 2024-09-11T01:49:18.909 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.904+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.909 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.904+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839100070}) 2024-09-11T01:49:18.909 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.904+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=601043172}) 2024-09-11T01:49:18.909 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.904+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.909 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.904+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839100070}) 2024-09-11T01:49:18.909 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.904+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2450424704}) 2024-09-11T01:49:18.958 INFO:tasks.rados.rados.0.smithi071.stdout:3001: finishing write tid 1 to smithi07134777-24 2024-09-11T01:49:18.958 INFO:tasks.rados.rados.0.smithi071.stdout:3001: finishing write tid 2 to smithi07134777-24 2024-09-11T01:49:18.958 INFO:tasks.rados.rados.0.smithi071.stdout:2999: done (11 left) 2024-09-11T01:49:18.959 INFO:tasks.rados.rados.0.smithi071.stdout:3000: done (10 left) 2024-09-11T01:49:18.959 INFO:tasks.rados.rados.0.smithi071.stdout:3012: snap_create 2024-09-11T01:49:18.959 INFO:tasks.rados.rados.0.smithi071.stdout:3001: finishing write tid 3 to smithi07134777-24 2024-09-11T01:49:18.959 INFO:tasks.rados.rados.0.smithi071.stdout:3001: finishing write tid 4 to smithi07134777-24 2024-09-11T01:49:18.959 INFO:tasks.rados.rados.0.smithi071.stdout:3001: finishing write tid 5 to smithi07134777-24 2024-09-11T01:49:18.959 INFO:tasks.rados.rados.0.smithi071.stdout:3001: oid 24 updating version 0 to 2042 2024-09-11T01:49:18.959 INFO:tasks.rados.rados.0.smithi071.stdout:3001: oid 24 version 2042 is already newer than 2040 2024-09-11T01:49:18.959 INFO:tasks.rados.rados.0.smithi071.stdout:3001: oid 24 updating version 2042 to 2043 2024-09-11T01:49:18.959 INFO:tasks.rados.rados.0.smithi071.stdout:3001: oid 24 version 2043 is already newer than 2041 2024-09-11T01:49:18.959 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 2043 (ObjNum 1060 snap 340 seq_num 1060) dirty exists 2024-09-11T01:49:18.959 INFO:tasks.rados.rados.0.smithi071.stdout:3001: left oid 24 (ObjNum 1060 snap 340 seq_num 1060) 2024-09-11T01:49:18.959 INFO:tasks.rados.rados.0.smithi071.stdout:3002: finishing write tid 1 to smithi07134777-44 2024-09-11T01:49:18.960 INFO:tasks.rados.rados.0.smithi071.stdout:3002: finishing write tid 2 to smithi07134777-44 2024-09-11T01:49:18.960 INFO:tasks.rados.rados.0.smithi071.stdout:3002: finishing write tid 3 to smithi07134777-44 2024-09-11T01:49:18.960 INFO:tasks.rados.rados.0.smithi071.stdout:3002: finishing write tid 4 to smithi07134777-44 2024-09-11T01:49:18.960 INFO:tasks.rados.rados.0.smithi071.stdout:3002: finishing write tid 5 to smithi07134777-44 2024-09-11T01:49:18.960 INFO:tasks.rados.rados.0.smithi071.stdout:3002: oid 44 updating version 0 to 1069 2024-09-11T01:49:18.960 INFO:tasks.rados.rados.0.smithi071.stdout:3002: oid 44 updating version 1069 to 1072 2024-09-11T01:49:18.960 INFO:tasks.rados.rados.0.smithi071.stdout:3002: oid 44 version 1072 is already newer than 1070 2024-09-11T01:49:18.960 INFO:tasks.rados.rados.0.smithi071.stdout:3002: oid 44 version 1072 is already newer than 1071 2024-09-11T01:49:18.960 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 1072 (ObjNum 1061 snap 340 seq_num 1061) dirty exists 2024-09-11T01:49:18.960 INFO:tasks.rados.rados.0.smithi071.stdout:3002: left oid 44 (ObjNum 1061 snap 340 seq_num 1061) 2024-09-11T01:49:18.960 INFO:tasks.rados.rados.0.smithi071.stdout:3003: expect (ObjNum 1017 snap 322 seq_num 1017) 2024-09-11T01:49:18.960 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:18.961 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.957+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.961 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.957+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=155705133}) 2024-09-11T01:49:18.961 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.957+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4076483446}) 2024-09-11T01:49:18.961 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.957+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:18.961 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.957+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=155705133}) 2024-09-11T01:49:18.961 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:18.957+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3599399492}) 2024-09-11T01:49:19.021 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:19.105 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.103+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:19.105 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.103+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3923742477}) 2024-09-11T01:49:19.105 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.103+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1847363567}) 2024-09-11T01:49:19.105 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.103+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:19.105 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.103+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3923742477}) 2024-09-11T01:49:19.105 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.103+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3399208340}) 2024-09-11T01:49:19.133 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.131+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:19.133 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.131+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598001557}) 2024-09-11T01:49:19.133 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.131+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=753638034}) 2024-09-11T01:49:19.133 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.131+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:19.133 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.131+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598001557}) 2024-09-11T01:49:19.133 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.131+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=655184214}) 2024-09-11T01:49:19.143 INFO:tasks.rados.rados.0.smithi071.stdout:3005: finishing write tid 1 to smithi07134777-16 2024-09-11T01:49:19.143 INFO:tasks.rados.rados.0.smithi071.stdout:3005: finishing write tid 2 to smithi07134777-16 2024-09-11T01:49:19.143 INFO:tasks.rados.rados.0.smithi071.stdout:3005: finishing write tid 3 to smithi07134777-16 2024-09-11T01:49:19.143 INFO:tasks.rados.rados.0.smithi071.stdout:3005: finishing write tid 4 to smithi07134777-16 2024-09-11T01:49:19.143 INFO:tasks.rados.rados.0.smithi071.stdout:3005: finishing write tid 5 to smithi07134777-16 2024-09-11T01:49:19.143 INFO:tasks.rados.rados.0.smithi071.stdout:3005: finishing write tid 6 to smithi07134777-16 2024-09-11T01:49:19.143 INFO:tasks.rados.rados.0.smithi071.stdout:3005: oid 16 updating version 0 to 2047 2024-09-11T01:49:19.144 INFO:tasks.rados.rados.0.smithi071.stdout:3005: oid 16 updating version 2047 to 2048 2024-09-11T01:49:19.144 INFO:tasks.rados.rados.0.smithi071.stdout:3005: oid 16 version 2048 is already newer than 2045 2024-09-11T01:49:19.144 INFO:tasks.rados.rados.0.smithi071.stdout:3005: oid 16 version 2048 is already newer than 2046 2024-09-11T01:49:19.144 INFO:tasks.rados.rados.0.smithi071.stdout:3005: oid 16 version 2048 is already newer than 2044 2024-09-11T01:49:19.144 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 2048 (ObjNum 1062 snap 340 seq_num 1062) dirty exists 2024-09-11T01:49:19.144 INFO:tasks.rados.rados.0.smithi071.stdout:3005: left oid 16 (ObjNum 1062 snap 340 seq_num 1062) 2024-09-11T01:49:19.144 INFO:tasks.rados.rados.0.smithi071.stdout:3006: expect (ObjNum 723 snap 224 seq_num 723) 2024-09-11T01:49:19.159 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:19.183 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:19.206 INFO:tasks.rados.rados.0.smithi071.stdout:3010: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:49:19.208 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:49:19.208 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:49:19.211 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.209+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:19.211 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.209+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=44704779}) 2024-09-11T01:49:19.211 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.209+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2654322645}) 2024-09-11T01:49:19.211 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.209+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:19.211 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.209+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=44704779}) 2024-09-11T01:49:19.212 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.209+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1689263327}) 2024-09-11T01:49:19.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.313+0000 7f962bce6640 10 monclient: get_monmap_and_config 2024-09-11T01:49:19.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.313+0000 7f962bce6640 10 monclient: build_initial_monmap 2024-09-11T01:49:19.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.313+0000 7f962bce6640 10 monclient: monmap: 2024-09-11T01:49:19.315 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:19.315 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:19.315 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:49:19.313794+0000 2024-09-11T01:49:19.315 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:49:19.313794+0000 2024-09-11T01:49:19.315 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:19.315 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:19.315 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:49:19.315 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:49:19.315 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:49:19.315 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:49:19.315 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:49:19.315 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:19.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.313+0000 7f962bce6640 1 Processor -- start 2024-09-11T01:49:19.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.313+0000 7f962bce6640 1 -- start start 2024-09-11T01:49:19.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.313+0000 7f962bce6640 10 monclient: init 2024-09-11T01:49:19.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962bce6640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:19.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962bce6640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-11T01:49:19.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962bce6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9624138230 0x7f9624138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:19.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962bce6640 10 monclient(hunting): picked mon.noname-c con 0x7f9624138230 addr v2:172.21.15.71:3300/0 2024-09-11T01:49:19.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962bce6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f962409d030 addr v1:172.21.15.12:6789/0 2024-09-11T01:49:19.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962bce6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f962409d460 0x7f96240a5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:19.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962bce6640 10 monclient(hunting): picked mon.noname-b con 0x7f962409d460 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:19.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962bce6640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96240a6e30 con 0x7f962409d030 2024-09-11T01:49:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962bce6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962bce6640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f962409c9c0 con 0x7f9624138230 2024-09-11T01:49:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962bce6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962bce6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f962409ba20 con 0x7f962409d460 2024-09-11T01:49:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962bce6640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962bce6640 10 monclient(hunting): authenticate will time out at 1965.756398s 2024-09-11T01:49:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962a283640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f962409d460 0x7f96240a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f9629281640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f962409d030 0x7f96240a5020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59888/0 (socket says 172.21.15.12:59888) 2024-09-11T01:49:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f9629281640 1 -- 172.21.15.12:0/1429117059 learned_addr learned my addr 172.21.15.12:0/1429117059 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:49:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962a283640 10 monclient(hunting): get_auth_request con 0x7f962409d460 auth_method 0 2024-09-11T01:49:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962a283640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962a283640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962a283640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f9629a82640 1 --2- 172.21.15.12:0/1429117059 >> v2:172.21.15.71:3300/0 conn(0x7f9624138230 0x7f9624138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f9629a82640 10 monclient(hunting): get_auth_request con 0x7f9624138230 auth_method 0 2024-09-11T01:49:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f9629a82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f9629a82640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f9629a82640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962a283640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962a283640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962a283640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f9629a82640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f9629a82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f9629a82640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962a283640 10 monclient(hunting): handle_auth_done global_id 224114 payload 274 2024-09-11T01:49:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962a283640 1 -- 172.21.15.12:0/1429117059 >> v2:172.21.15.71:3300/0 conn(0x7f9624138230 msgr2=0x7f9624138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:49:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962a283640 1 --2- 172.21.15.12:0/1429117059 >> v2:172.21.15.71:3300/0 conn(0x7f9624138230 0x7f9624138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:19.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962a283640 1 -- 172.21.15.12:0/1429117059 >> v1:172.21.15.12:6789/0 conn(0x7f962409d030 legacy=0x7f96240a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:49:19.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962a283640 10 monclient: _finish_hunting 0 2024-09-11T01:49:19.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962a283640 1 monclient: found mon.noname-b 2024-09-11T01:49:19.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962a283640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:19.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962a283640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:49:19.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.314+0000 7f962a283640 1 -- 172.21.15.12:0/1429117059 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96240a66a0 con 0x7f962409d460 2024-09-11T01:49:19.321 INFO:tasks.rados.rados.0.smithi071.stdout:3009: expect (ObjNum 987 snap 317 seq_num 987) 2024-09-11T01:49:19.327 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1665166 2024-09-11T01:49:19.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.317+0000 7f962a283640 1 -- 172.21.15.12:0/1429117059 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f962409d460 msgr2=0x7f96240a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:19.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.317+0000 7f962a283640 1 --2- 172.21.15.12:0/1429117059 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f962409d460 0x7f96240a5730 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f96240a17e0 tx=0x7f961c02a460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bc3677e0471c28d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:19.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.317+0000 7f9628a80640 1 -- 172.21.15.12:0/1429117059 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961c007950 con 0x7f962409d460 2024-09-11T01:49:19.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.317+0000 7f9628a80640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:19.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.317+0000 7f9628a80640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:49:19.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.317+0000 7f9628a80640 10 monclient: dump: 2024-09-11T01:49:19.328 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:19.328 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:19.328 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:19.328 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:19.328 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:19.328 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:19.328 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:19.328 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:19.328 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:19.328 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:19.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.317+0000 7f9628a80640 10 monclient: _finish_auth 0 2024-09-11T01:49:19.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.317+0000 7f9628a80640 10 monclient: _check_auth_tickets 2024-09-11T01:49:19.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.317+0000 7f9628a80640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:19.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.317+0000 7f9628a80640 1 -- 172.21.15.12:0/1429117059 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f961c007af0 con 0x7f962409d460 2024-09-11T01:49:19.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.317+0000 7f9628a80640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:19.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.317+0000 7f9628a80640 1 -- 172.21.15.12:0/1429117059 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961c031400 con 0x7f962409d460 2024-09-11T01:49:19.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.317+0000 7f962bce6640 5 monclient: authenticate success, global_id 224114 2024-09-11T01:49:19.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.317+0000 7f962bce6640 10 monclient: get_monmap_and_config success 2024-09-11T01:49:19.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.317+0000 7f962bce6640 4 set_mon_vals no callback set 2024-09-11T01:49:19.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.317+0000 7f9628a80640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:19.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.317+0000 7f9628a80640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:49:19.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.317+0000 7f9628a80640 10 monclient: dump: 2024-09-11T01:49:19.329 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:19.329 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:19.329 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:19.329 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:19.329 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:19.330 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:19.330 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:19.330 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:19.330 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:19.330 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:19.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.318+0000 7f962bce6640 10 monclient: shutdown 2024-09-11T01:49:19.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.318+0000 7f962bce6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:19.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.318+0000 7f962bce6640 1 -- 172.21.15.12:0/1429117059 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f962409d460 msgr2=0x7f96240a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:19.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.318+0000 7f962bce6640 1 --2- 172.21.15.12:0/1429117059 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f962409d460 0x7f96240a5730 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f96240a17e0 tx=0x7f961c02a460 comp rx=0 tx=0).stop 2024-09-11T01:49:19.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.318+0000 7f962bce6640 1 -- 172.21.15.12:0/1429117059 shutdown_connections 2024-09-11T01:49:19.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.318+0000 7f962bce6640 1 --2- 172.21.15.12:0/1429117059 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f962409d460 0x7f96240a5730 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:19.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.318+0000 7f962bce6640 1 --2- 172.21.15.12:0/1429117059 >> v2:172.21.15.71:3300/0 conn(0x7f9624138230 0x7f9624138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:19.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.318+0000 7f962bce6640 1 -- 172.21.15.12:0/1429117059 >> 172.21.15.12:0/1429117059 conn(0x7f96240a7120 msgr2=0x7f96240a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:19.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.318+0000 7f962bce6640 1 -- 172.21.15.12:0/1429117059 shutdown_connections 2024-09-11T01:49:19.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.318+0000 7f962bce6640 1 -- 172.21.15.12:0/1429117059 wait complete. 2024-09-11T01:49:19.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.319+0000 7f962bce6640 10 monclient: build_initial_monmap 2024-09-11T01:49:19.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.319+0000 7f962bce6640 10 monclient: monmap: 2024-09-11T01:49:19.331 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:19.331 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:49:19.331 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:49:19.331 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:49:19.331 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:19.331 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:19.332 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:49:19.332 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:49:19.332 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:49:19.332 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:19.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.319+0000 7f962bce6640 1 Processor -- start 2024-09-11T01:49:19.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.319+0000 7f962bce6640 1 -- start start 2024-09-11T01:49:19.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.319+0000 7f962bce6640 10 monclient: init 2024-09-11T01:49:19.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.319+0000 7f962bce6640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:19.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.319+0000 7f962bce6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:49:19.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.319+0000 7f962bce6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f962409d030 0x7f9624136d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:19.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.319+0000 7f962bce6640 10 monclient(hunting): picked mon.noname-b con 0x7f962409d030 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:49:19.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.319+0000 7f962bce6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f962409d460 0x7f9624137260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:19.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.319+0000 7f962bce6640 10 monclient(hunting): picked mon.noname-a con 0x7f962409d460 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:49:19.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.319+0000 7f962bce6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9624138230 0x7f9624120c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:19.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.319+0000 7f962bce6640 10 monclient(hunting): picked mon.noname-c con 0x7f9624138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:19.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.319+0000 7f962bce6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:19.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.319+0000 7f962bce6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f962406f970 con 0x7f962409d460 2024-09-11T01:49:19.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.319+0000 7f962bce6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:19.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.319+0000 7f962bce6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f962406f7f0 con 0x7f962409d030 2024-09-11T01:49:19.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.319+0000 7f962bce6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:19.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.319+0000 7f962bce6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f96240a5c70 con 0x7f9624138230 2024-09-11T01:49:19.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.319+0000 7f962bce6640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:19.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.319+0000 7f962bce6640 10 monclient(hunting): authenticate will time out at 1965.761982s 2024-09-11T01:49:19.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629281640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f962409d460 0x7f9624137260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:19.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629281640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f962409d460 0x7f9624137260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43780/0 (socket says 172.21.15.12:43780) 2024-09-11T01:49:19.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629281640 1 -- 172.21.15.12:0/3191232598 learned_addr learned my addr 172.21.15.12:0/3191232598 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:49:19.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f962a283640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9624138230 0x7f9624120c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:19.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629281640 10 monclient(hunting): get_auth_request con 0x7f962409d460 auth_method 0 2024-09-11T01:49:19.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629281640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:19.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629281640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:19.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629281640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:19.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f962a283640 10 monclient(hunting): get_auth_request con 0x7f9624138230 auth_method 0 2024-09-11T01:49:19.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f962a283640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:19.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f962a283640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:19.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f962a283640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:19.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629a82640 1 --2- 172.21.15.12:0/3191232598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f962409d030 0x7f9624136d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:19.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629a82640 10 monclient(hunting): get_auth_request con 0x7f962409d030 auth_method 0 2024-09-11T01:49:19.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629a82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:19.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629a82640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:19.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629a82640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:19.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629281640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:19.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629281640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:19.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629281640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:19.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f962a283640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:19.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f962a283640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:19.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f962a283640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:19.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629a82640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:19.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629a82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:19.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629a82640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:19.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629281640 10 monclient(hunting): handle_auth_done global_id 214224 payload 824 2024-09-11T01:49:19.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629281640 1 -- 172.21.15.12:0/3191232598 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9624138230 msgr2=0x7f9624120c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:19.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629281640 1 --2- 172.21.15.12:0/3191232598 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9624138230 0x7f9624120c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:19.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629281640 1 -- 172.21.15.12:0/3191232598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f962409d030 msgr2=0x7f9624136d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:19.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629281640 1 --2- 172.21.15.12:0/3191232598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f962409d030 0x7f9624136d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:19.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629281640 10 monclient: _finish_hunting 0 2024-09-11T01:49:19.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629281640 1 monclient: found mon.noname-a 2024-09-11T01:49:19.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629281640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:19.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629281640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:49:19.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629281640 1 -- 172.21.15.12:0/3191232598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96240a12a0 con 0x7f962409d460 2024-09-11T01:49:19.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f962a283640 1 --2- 172.21.15.12:0/3191232598 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9624138230 0x7f9624120c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:49:19.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.320+0000 7f9629a82640 1 --2- 172.21.15.12:0/3191232598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f962409d030 0x7f9624136d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:49:19.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.322+0000 7f9629281640 1 -- 172.21.15.12:0/3191232598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f962409d460 msgr2=0x7f9624137260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:19.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.322+0000 7f9629281640 1 --2- 172.21.15.12:0/3191232598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f962409d460 0x7f9624137260 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f961400c460 tx=0x7f961400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:19.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.322+0000 7f961a7fc640 1 -- 172.21.15.12:0/3191232598 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9614007070 con 0x7f962409d460 2024-09-11T01:49:19.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.322+0000 7f961a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:19.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.322+0000 7f961a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:49:19.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.322+0000 7f961a7fc640 10 monclient: dump: 2024-09-11T01:49:19.337 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:19.337 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:19.337 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:19.337 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:19.337 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:19.337 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:19.337 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:19.337 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:19.337 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:19.337 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:19.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.322+0000 7f961a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:49:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.322+0000 7f961a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:49:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.322+0000 7f961a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.322+0000 7f961a7fc640 1 -- 172.21.15.12:0/3191232598 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9614007230 con 0x7f962409d460 2024-09-11T01:49:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.322+0000 7f962bce6640 5 monclient: authenticate success, global_id 214224 2024-09-11T01:49:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.322+0000 7f962bce6640 10 monclient: _renew_subs 2024-09-11T01:49:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.322+0000 7f962bce6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.322+0000 7f962bce6640 1 -- 172.21.15.12:0/3191232598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9624122380 con 0x7f962409d460 2024-09-11T01:49:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.322+0000 7f961a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.322+0000 7f961a7fc640 1 -- 172.21.15.12:0/3191232598 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9614007620 con 0x7f962409d460 2024-09-11T01:49:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.322+0000 7f961a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.322+0000 7f961a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.322+0000 7f961a7fc640 10 monclient: dump: 2024-09-11T01:49:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:19.338 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:19.339 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:19.339 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:19.339 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:19.339 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:19.339 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:19.339 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:19.339 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:19.339 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:19.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.322+0000 7f962bce6640 10 monclient: _renew_subs 2024-09-11T01:49:19.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.322+0000 7f961bfff640 4 set_mon_vals no callback set 2024-09-11T01:49:19.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.322+0000 7f962bce6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:19.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.322+0000 7f962bce6640 1 -- 172.21.15.12:0/3191232598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9624122ee0 con 0x7f962409d460 2024-09-11T01:49:19.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.323+0000 7f962bce6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:19.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.323+0000 7f962bce6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:19.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.323+0000 7f962bce6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:19.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.323+0000 7f962bce6640 1 -- 172.21.15.12:0/3191232598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9624138610 con 0x7f962409d460 2024-09-11T01:49:19.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.323+0000 7f9629281640 1 --2- 172.21.15.12:0/3191232598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f962409d460 0x7f9624137260 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f961400c460 tx=0x7f961400c920 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7f9614052a70 osd_map(676..676 src has 507..676) 2024-09-11T01:49:19.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.323+0000 7f961a7fc640 1 -- 172.21.15.12:0/3191232598 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f961401b400 con 0x7f962409d460 2024-09-11T01:49:19.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.324+0000 7f961a7fc640 1 --2- 172.21.15.12:0/3191232598 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f960803e210 0x7f96080406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:19.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.324+0000 7f9629a82640 1 --2- 172.21.15.12:0/3191232598 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f960803e210 0x7f96080406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:19.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.324+0000 7f9629a82640 10 monclient: get_auth_request con 0x7f960803e210 auth_method 0 2024-09-11T01:49:19.341 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:19.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.326+0000 7f9629a82640 1 --2- 172.21.15.12:0/3191232598 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f960803e210 0x7f96080406d0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f9620009f30 tx=0x7f9620009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:19.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:19.328+0000 7f961a7fc640 1 -- 172.21.15.12:0/3191232598 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(676..676 src has 507..676) ==== 4891+0+0 (secure 0 0 0) 0x7f9614052a70 con 0x7f962409d460 2024-09-11T01:49:19.372 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 986111 2024-09-11T01:49:19.389 INFO:tasks.rados.rados.0.smithi071.stdout:3011: finishing rollback tid 0 to smithi07134777-46 2024-09-11T01:49:19.389 INFO:tasks.rados.rados.0.smithi071.stdout:3011: finishing rollback tid 1 to smithi07134777-46 2024-09-11T01:49:19.389 INFO:tasks.rados.rados.0.smithi071.stdout:3011: finishing rollback tid 2 to smithi07134777-46 2024-09-11T01:49:19.389 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 1554 (ObjNum 841 snap 255 seq_num 841) dirty exists 2024-09-11T01:49:19.541 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.539+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:19.541 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.539+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3160252406}) 2024-09-11T01:49:19.541 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.539+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3818055659}) 2024-09-11T01:49:19.541 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.539+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:19.541 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.539+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3160252406}) 2024-09-11T01:49:19.542 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.539+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=644948371}) 2024-09-11T01:49:19.543 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.540+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:19.543 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.540+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2564730437}) 2024-09-11T01:49:19.543 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.540+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3161770213}) 2024-09-11T01:49:19.543 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.540+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:19.543 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.540+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2564730437}) 2024-09-11T01:49:19.543 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.540+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=607988623}) 2024-09-11T01:49:19.543 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.541+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:19.543 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.541+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2416522398}) 2024-09-11T01:49:19.543 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.541+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4060600802}) 2024-09-11T01:49:19.543 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.541+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:19.543 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.541+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2416522398}) 2024-09-11T01:49:19.544 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.541+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1836919713}) 2024-09-11T01:49:19.546 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.542+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:19.546 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.542+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2105536409}) 2024-09-11T01:49:19.546 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.542+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=337089250}) 2024-09-11T01:49:19.546 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.542+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:19.546 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.542+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2105536409}) 2024-09-11T01:49:19.546 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.542+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1575911814}) 2024-09-11T01:49:19.546 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.543+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:19.546 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.543+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=75,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=4} crc {logm=1508991642,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=3367178313,mon_config_key=4137481205}) 2024-09-11T01:49:19.546 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.543+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=78,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=2} crc {logm=1633668165,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2450298824,mon_config_key=243754935}) 2024-09-11T01:49:19.546 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.543+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:19.546 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.543+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=75,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=4} crc {logm=1508991642,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=3367178313,mon_config_key=4137481205}) 2024-09-11T01:49:19.547 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.543+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=75,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6,mon_config_key=6} crc {logm=418502168,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=307938839,mon_config_key=3410560155}) 2024-09-11T01:49:19.547 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.543+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:19.547 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.543+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=76} crc {mon_config_key=3874621322,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=3166507939,osd_snap=2154587959}) 2024-09-11T01:49:19.547 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.543+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=76} crc {mon_config_key=1104985114,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=3166507939,osd_snap=494056148}) 2024-09-11T01:49:19.547 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.543+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:19.547 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.543+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=76} crc {mon_config_key=3874621322,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=3166507939,osd_snap=2154587959}) 2024-09-11T01:49:19.547 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.543+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=78} crc {mon_config_key=1596791814,monmap=2933275274,nvmeofgw=598337686,osd_metadata=1246036448,osd_pg_creating=3166507939,osd_snap=3052961946}) 2024-09-11T01:49:19.547 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.544+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:19.547 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.544+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=65,osdmap=35} crc {osd_snap=2863704522,osdmap=1842354535}) 2024-09-11T01:49:19.547 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.544+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=66,osdmap=34} crc {osd_snap=818541222,osdmap=1416386931}) 2024-09-11T01:49:19.547 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.544+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:19.547 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.544+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=65,osdmap=35} crc {osd_snap=2863704522,osdmap=1842354535}) 2024-09-11T01:49:19.547 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.544+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=68,osdmap=32} crc {osd_snap=1894635132,osdmap=3532929081}) 2024-09-11T01:49:19.549 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.545+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:19.549 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.545+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2868667540}) 2024-09-11T01:49:19.549 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.545+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=249755579}) 2024-09-11T01:49:19.549 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.545+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:19.549 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.545+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2868667540}) 2024-09-11T01:49:19.549 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.545+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3240534417}) 2024-09-11T01:49:19.549 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.546+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:19.549 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.546+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2670036698}) 2024-09-11T01:49:19.549 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.546+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1074549236}) 2024-09-11T01:49:19.549 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.546+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:19.549 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.546+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2670036698}) 2024-09-11T01:49:19.549 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.546+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=371863460}) 2024-09-11T01:49:19.552 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.550+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:19.552 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.550+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=23} crc {osdmap=416264801}) 2024-09-11T01:49:19.552 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.550+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=22} crc {osdmap=803565605}) 2024-09-11T01:49:19.552 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.550+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:49:19.552 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.550+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=23} crc {osdmap=416264801}) 2024-09-11T01:49:19.552 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:49:19.550+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=22} crc {osdmap=3486116197}) 2024-09-11T01:49:19.874 INFO:tasks.rados.rados.0.smithi071.stdout:3001: done (10 left) 2024-09-11T01:49:19.874 INFO:tasks.rados.rados.0.smithi071.stdout:3002: done (9 left) 2024-09-11T01:49:19.874 INFO:tasks.rados.rados.0.smithi071.stdout:3003: done (8 left) 2024-09-11T01:49:19.874 INFO:tasks.rados.rados.0.smithi071.stdout:3004: done (7 left) 2024-09-11T01:49:19.874 INFO:tasks.rados.rados.0.smithi071.stdout:3005: done (6 left) 2024-09-11T01:49:19.874 INFO:tasks.rados.rados.0.smithi071.stdout:3006: done (5 left) 2024-09-11T01:49:19.874 INFO:tasks.rados.rados.0.smithi071.stdout:3007: done (4 left) 2024-09-11T01:49:19.874 INFO:tasks.rados.rados.0.smithi071.stdout:3009: done (3 left) 2024-09-11T01:49:19.874 INFO:tasks.rados.rados.0.smithi071.stdout:3010: done (2 left) 2024-09-11T01:49:19.874 INFO:tasks.rados.rados.0.smithi071.stdout:3011: done (1 left) 2024-09-11T01:49:19.874 INFO:tasks.rados.rados.0.smithi071.stdout:3012: done (0 left) 2024-09-11T01:49:19.874 INFO:tasks.rados.rados.0.smithi071.stdout:3013: copy_from oid 17 from oid 13 current snap is 341 2024-09-11T01:49:19.874 INFO:tasks.rados.rados.0.smithi071.stdout:3014: read oid 1 snap -1 2024-09-11T01:49:19.875 INFO:tasks.rados.rados.0.smithi071.stdout:3014: expect (ObjNum 1046 snap 337 seq_num 1046) 2024-09-11T01:49:19.875 INFO:tasks.rados.rados.0.smithi071.stdout:3015: copy_from oid 30 from oid 23 current snap is 341 2024-09-11T01:49:19.875 INFO:tasks.rados.rados.0.smithi071.stdout:3016: rollback oid 28 current snap is 341 2024-09-11T01:49:19.875 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 28 to 327 2024-09-11T01:49:19.875 INFO:tasks.rados.rados.0.smithi071.stdout:3017: rollback oid 14 current snap is 341 2024-09-11T01:49:19.875 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 14 to 300 2024-09-11T01:49:19.875 INFO:tasks.rados.rados.0.smithi071.stdout:3018: rollback oid 2 current snap is 341 2024-09-11T01:49:19.875 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 2 to 236 2024-09-11T01:49:19.875 INFO:tasks.rados.rados.0.smithi071.stdout:3019: snap_remove snap 331 2024-09-11T01:49:20.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.151+0000 7f961a7fc640 1 -- 172.21.15.12:0/3191232598 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f96240a12a0 con 0x7f962409d460 2024-09-11T01:49:20.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.151+0000 7f961a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:20.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.151+0000 7f961a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:49:20.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.411+0000 7f962bce6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:49:20.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.411+0000 7f962bce6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:49:20.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.411+0000 7f962bce6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:20.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.411+0000 7f962bce6640 1 -- 172.21.15.12:0/3191232598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f96240698e0 con 0x7f962409d460 2024-09-11T01:49:20.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.412+0000 7f961a7fc640 1 -- 172.21.15.12:0/3191232598 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f9614010d10 con 0x7f962409d460 2024-09-11T01:49:20.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.412+0000 7f961a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:49:20.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.412+0000 7f961a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:49:20.414 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":188,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:49:20.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.414+0000 7f962bce6640 1 -- 172.21.15.12:0/3191232598 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f960803e210 msgr2=0x7f96080406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:20.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.414+0000 7f962bce6640 1 --2- 172.21.15.12:0/3191232598 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f960803e210 0x7f96080406d0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f9620009f30 tx=0x7f9620009190 comp rx=0 tx=0).stop 2024-09-11T01:49:20.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.414+0000 7f962bce6640 10 monclient: shutdown 2024-09-11T01:49:20.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.414+0000 7f962bce6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:20.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.414+0000 7f962bce6640 1 -- 172.21.15.12:0/3191232598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f962409d460 msgr2=0x7f9624137260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:20.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.414+0000 7f962bce6640 1 --2- 172.21.15.12:0/3191232598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f962409d460 0x7f9624137260 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f961400c460 tx=0x7f961400c920 comp rx=0 tx=0).stop 2024-09-11T01:49:20.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.414+0000 7f962bce6640 1 -- 172.21.15.12:0/3191232598 shutdown_connections 2024-09-11T01:49:20.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.414+0000 7f962bce6640 1 --2- 172.21.15.12:0/3191232598 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9624138230 0x7f9624120c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.414+0000 7f962bce6640 1 --2- 172.21.15.12:0/3191232598 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f960803e210 0x7f96080406d0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.414+0000 7f962bce6640 1 --2- 172.21.15.12:0/3191232598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f962409d460 0x7f9624137260 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.414+0000 7f962bce6640 1 --2- 172.21.15.12:0/3191232598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f962409d030 0x7f9624136d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.414+0000 7f962bce6640 1 -- 172.21.15.12:0/3191232598 >> 172.21.15.12:0/3191232598 conn(0x7f96240a7120 msgr2=0x7f96240a3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:20.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.414+0000 7f962bce6640 1 -- 172.21.15.12:0/3191232598 shutdown_connections 2024-09-11T01:49:20.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.414+0000 7f962bce6640 1 -- 172.21.15.12:0/3191232598 wait complete. 2024-09-11T01:49:20.425 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:49:20.426 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:49:20.426 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:49:20.426 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:49:20.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.526+0000 7f015b3b4640 10 monclient: get_monmap_and_config 2024-09-11T01:49:20.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.526+0000 7f015b3b4640 10 monclient: build_initial_monmap 2024-09-11T01:49:20.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.526+0000 7f015b3b4640 10 monclient: monmap: 2024-09-11T01:49:20.528 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:20.528 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:20.528 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:49:20.526938+0000 2024-09-11T01:49:20.528 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:49:20.526938+0000 2024-09-11T01:49:20.528 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:20.528 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:20.528 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:49:20.528 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:49:20.528 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:49:20.528 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:49:20.528 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:49:20.529 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:20.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.526+0000 7f015b3b4640 1 Processor -- start 2024-09-11T01:49:20.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015b3b4640 1 -- start start 2024-09-11T01:49:20.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015b3b4640 10 monclient: init 2024-09-11T01:49:20.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015b3b4640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:20.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015b3b4640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-11T01:49:20.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015b3b4640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0154137e50 0x7f015409d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:20.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015b3b4640 10 monclient(hunting): picked mon.noname-c con 0x7f0154137e50 addr v2:172.21.15.71:3300/0 2024-09-11T01:49:20.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015b3b4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0154138230 0x7f0154138640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:20.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015b3b4640 10 monclient(hunting): picked mon.noname-a con 0x7f0154138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:49:20.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015b3b4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0154134cb0 addr v1:172.21.15.71:6789/0 2024-09-11T01:49:20.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015b3b4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01540a62d0 con 0x7f0154134cb0 2024-09-11T01:49:20.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015b3b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:20.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015b3b4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f015409bb40 con 0x7f0154138230 2024-09-11T01:49:20.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015b3b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:20.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015b3b4640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f015409c4a0 con 0x7f0154137e50 2024-09-11T01:49:20.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015b3b4640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:20.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015b3b4640 10 monclient(hunting): authenticate will time out at 1966.969551s 2024-09-11T01:49:20.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015894f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0154138230 0x7f0154138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:20.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015894f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0154138230 0x7f0154138640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43782/0 (socket says 172.21.15.12:43782) 2024-09-11T01:49:20.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015894f640 1 -- 172.21.15.12:0/4009718687 learned_addr learned my addr 172.21.15.12:0/4009718687 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:49:20.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015894f640 10 monclient(hunting): get_auth_request con 0x7f0154138230 auth_method 0 2024-09-11T01:49:20.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015894f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:20.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015894f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:20.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015894f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:20.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015894f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:20.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015894f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:20.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015894f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:20.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015894f640 10 monclient(hunting): handle_auth_done global_id 214227 payload 274 2024-09-11T01:49:20.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015894f640 1 -- 172.21.15.12:0/4009718687 >> v2:172.21.15.71:3300/0 conn(0x7f0154137e50 msgr2=0x7f015409d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:49:20.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f0159150640 1 --2- 172.21.15.12:0/4009718687 >> v2:172.21.15.71:3300/0 conn(0x7f0154137e50 0x7f015409d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:20.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015894f640 1 --2- 172.21.15.12:0/4009718687 >> v2:172.21.15.71:3300/0 conn(0x7f0154137e50 0x7f015409d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015894f640 1 -- 172.21.15.12:0/4009718687 >> v1:172.21.15.71:6789/0 conn(0x7f0154134cb0 legacy=0x7f0154135160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:49:20.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015894f640 10 monclient: _finish_hunting 0 2024-09-11T01:49:20.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015894f640 1 monclient: found mon.noname-a 2024-09-11T01:49:20.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015894f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:20.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015894f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:49:20.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.527+0000 7f015894f640 1 -- 172.21.15.12:0/4009718687 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f015409c780 con 0x7f0154138230 2024-09-11T01:49:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.530+0000 7f015894f640 1 -- 172.21.15.12:0/4009718687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0154138230 msgr2=0x7f0154138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.530+0000 7f015894f640 1 --2- 172.21.15.12:0/4009718687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0154138230 0x7f0154138640 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f015409cf30 tx=0x7f014402a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=682d15122ed2004e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.531+0000 7f014bfff640 1 -- 172.21.15.12:0/4009718687 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0144037070 con 0x7f0154138230 2024-09-11T01:49:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.531+0000 7f014bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.531+0000 7f014bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:49:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.531+0000 7f014bfff640 10 monclient: dump: 2024-09-11T01:49:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:20.540 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:20.541 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.531+0000 7f014bfff640 10 monclient: _finish_auth 0 2024-09-11T01:49:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.531+0000 7f014bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:49:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.531+0000 7f014bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.531+0000 7f014bfff640 1 -- 172.21.15.12:0/4009718687 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0144032640 con 0x7f0154138230 2024-09-11T01:49:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.531+0000 7f015b3b4640 5 monclient: authenticate success, global_id 214227 2024-09-11T01:49:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.531+0000 7f015b3b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:49:20.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.531+0000 7f014bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.531+0000 7f015b3b4640 10 monclient: get_monmap_and_config success 2024-09-11T01:49:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.531+0000 7f014bfff640 1 -- 172.21.15.12:0/4009718687 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0144032a10 con 0x7f0154138230 2024-09-11T01:49:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.531+0000 7f015b3b4640 4 set_mon_vals no callback set 2024-09-11T01:49:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.531+0000 7f014bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.531+0000 7f014bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.531+0000 7f014bfff640 10 monclient: dump: 2024-09-11T01:49:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:20.542 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:20.543 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:20.543 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:20.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.531+0000 7f015b3b4640 10 monclient: shutdown 2024-09-11T01:49:20.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.531+0000 7f015b3b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:20.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.531+0000 7f015b3b4640 1 -- 172.21.15.12:0/4009718687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0154138230 msgr2=0x7f0154138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:20.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.531+0000 7f015b3b4640 1 --2- 172.21.15.12:0/4009718687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0154138230 0x7f0154138640 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f015409cf30 tx=0x7f014402a080 comp rx=0 tx=0).stop 2024-09-11T01:49:20.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.531+0000 7f015b3b4640 1 -- 172.21.15.12:0/4009718687 shutdown_connections 2024-09-11T01:49:20.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.531+0000 7f015b3b4640 1 --2- 172.21.15.12:0/4009718687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0154138230 0x7f0154138640 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.531+0000 7f015b3b4640 1 --2- 172.21.15.12:0/4009718687 >> v2:172.21.15.71:3300/0 conn(0x7f0154137e50 0x7f015409d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.531+0000 7f015b3b4640 1 -- 172.21.15.12:0/4009718687 >> 172.21.15.12:0/4009718687 conn(0x7f01540a7020 msgr2=0x7f01540a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:20.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.532+0000 7f015b3b4640 1 -- 172.21.15.12:0/4009718687 shutdown_connections 2024-09-11T01:49:20.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.532+0000 7f015b3b4640 1 -- 172.21.15.12:0/4009718687 wait complete. 2024-09-11T01:49:20.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.532+0000 7f015b3b4640 10 monclient: build_initial_monmap 2024-09-11T01:49:20.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.532+0000 7f015b3b4640 10 monclient: monmap: 2024-09-11T01:49:20.544 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:20.544 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:49:20.544 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:49:20.544 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:49:20.544 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:20.544 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:20.544 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:49:20.544 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:49:20.544 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:49:20.544 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:20.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.532+0000 7f015b3b4640 1 Processor -- start 2024-09-11T01:49:20.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.532+0000 7f015b3b4640 1 -- start start 2024-09-11T01:49:20.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.532+0000 7f015b3b4640 10 monclient: init 2024-09-11T01:49:20.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015b3b4640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:20.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015b3b4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:49:20.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015b3b4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0154134cb0 0x7f01541cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:20.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015b3b4640 10 monclient(hunting): picked mon.noname-b con 0x7f0154134cb0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:49:20.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015b3b4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0154137e50 0x7f01541cd910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:20.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015b3b4640 10 monclient(hunting): picked mon.noname-c con 0x7f0154137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:20.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015b3b4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0154138230 0x7f01541216d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:20.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015b3b4640 10 monclient(hunting): picked mon.noname-a con 0x7f0154138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:49:20.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015b3b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:20.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015b3b4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f015406f710 con 0x7f0154138230 2024-09-11T01:49:20.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015b3b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:20.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015b3b4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0154138b80 con 0x7f0154134cb0 2024-09-11T01:49:20.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015b3b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:20.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015b3b4640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f015406f890 con 0x7f0154137e50 2024-09-11T01:49:20.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015b3b4640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:20.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015b3b4640 10 monclient(hunting): authenticate will time out at 1966.975352s 2024-09-11T01:49:20.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015894f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0154137e50 0x7f01541cd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:20.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f0159951640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0154138230 0x7f01541216d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:20.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015894f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0154137e50 0x7f01541cd910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42278/0 (socket says 172.21.15.12:42278) 2024-09-11T01:49:20.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015894f640 1 -- 172.21.15.12:0/290478577 learned_addr learned my addr 172.21.15.12:0/290478577 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:49:20.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015894f640 10 monclient(hunting): get_auth_request con 0x7f0154137e50 auth_method 0 2024-09-11T01:49:20.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015894f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:20.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015894f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:20.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015894f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:20.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f0159951640 10 monclient(hunting): get_auth_request con 0x7f0154138230 auth_method 0 2024-09-11T01:49:20.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f0159951640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:20.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f0159951640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:20.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f0159951640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:20.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015894f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:20.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015894f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:20.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015894f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:20.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f0159951640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:20.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f0159951640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:20.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f0159951640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:20.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f0159150640 1 --2- 172.21.15.12:0/290478577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0154134cb0 0x7f01541cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:20.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f0159150640 10 monclient(hunting): get_auth_request con 0x7f0154134cb0 auth_method 0 2024-09-11T01:49:20.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f0159150640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:20.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f0159150640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:20.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f0159150640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:20.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015894f640 10 monclient(hunting): handle_auth_done global_id 224120 payload 824 2024-09-11T01:49:20.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015894f640 1 -- 172.21.15.12:0/290478577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0154134cb0 msgr2=0x7f01541cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:20.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015894f640 1 --2- 172.21.15.12:0/290478577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0154134cb0 0x7f01541cd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015894f640 1 -- 172.21.15.12:0/290478577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0154138230 msgr2=0x7f01541216d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:20.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015894f640 1 --2- 172.21.15.12:0/290478577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0154138230 0x7f01541216d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015894f640 10 monclient: _finish_hunting 0 2024-09-11T01:49:20.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015894f640 1 monclient: found mon.noname-c 2024-09-11T01:49:20.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015894f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:20.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015894f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:49:20.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f015894f640 1 -- 172.21.15.12:0/290478577 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01540a8e90 con 0x7f0154137e50 2024-09-11T01:49:20.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.533+0000 7f0159951640 1 --2- 172.21.15.12:0/290478577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0154138230 0x7f01541216d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:49:20.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.536+0000 7f015894f640 1 -- 172.21.15.12:0/290478577 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0154137e50 msgr2=0x7f01541cd910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:20.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.536+0000 7f015894f640 1 --2- 172.21.15.12:0/290478577 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0154137e50 0x7f01541cd910 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f01440090b0 tx=0x7f01440046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:20.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.536+0000 7f0149ffb640 1 -- 172.21.15.12:0/290478577 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0144037050 con 0x7f0154137e50 2024-09-11T01:49:20.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.536+0000 7f0149ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:20.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.536+0000 7f0149ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:49:20.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.536+0000 7f0149ffb640 10 monclient: dump: 2024-09-11T01:49:20.549 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:20.549 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:20.549 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.549 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.549 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:20.549 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:20.549 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:20.549 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:20.549 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:20.549 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:20.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.536+0000 7f0149ffb640 10 monclient: _finish_auth 0 2024-09-11T01:49:20.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.536+0000 7f0149ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:49:20.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.536+0000 7f0149ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:20.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.536+0000 7f0149ffb640 1 -- 172.21.15.12:0/290478577 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0144043070 con 0x7f0154137e50 2024-09-11T01:49:20.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.536+0000 7f015b3b4640 5 monclient: authenticate success, global_id 224120 2024-09-11T01:49:20.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.536+0000 7f015b3b4640 10 monclient: _renew_subs 2024-09-11T01:49:20.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.536+0000 7f015b3b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:49:20.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.536+0000 7f015b3b4640 1 -- 172.21.15.12:0/290478577 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0154137280 con 0x7f0154137e50 2024-09-11T01:49:20.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.536+0000 7f0149ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:20.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.536+0000 7f0149ffb640 1 -- 172.21.15.12:0/290478577 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014403cc60 con 0x7f0154137e50 2024-09-11T01:49:20.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.536+0000 7f0149ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:20.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.536+0000 7f0149ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:49:20.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.536+0000 7f0149ffb640 10 monclient: dump: 2024-09-11T01:49:20.550 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:20.551 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:20.551 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.551 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.551 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:20.551 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:20.551 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:20.551 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:20.551 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:20.551 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:20.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.536+0000 7f015b3b4640 10 monclient: _renew_subs 2024-09-11T01:49:20.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.536+0000 7f015b3b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:49:20.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.536+0000 7f015b3b4640 1 -- 172.21.15.12:0/290478577 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f01541254f0 con 0x7f0154137e50 2024-09-11T01:49:20.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.536+0000 7f014affd640 4 set_mon_vals no callback set 2024-09-11T01:49:20.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.537+0000 7f015b3b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:20.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.537+0000 7f015b3b4640 1 --2- 172.21.15.12:0/290478577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0154121c10 0x7f0154121ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:20.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.537+0000 7f015b3b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:20.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.537+0000 7f015b3b4640 1 -- 172.21.15.12:0/290478577 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f015406a060 con 0x7f0154121c10 2024-09-11T01:49:20.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.537+0000 7f0159150640 1 --2- 172.21.15.12:0/290478577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0154121c10 0x7f0154121ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:20.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.537+0000 7f0149ffb640 1 -- 172.21.15.12:0/290478577 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f0144048460 con 0x7f0154137e50 2024-09-11T01:49:20.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.537+0000 7f0159150640 10 monclient: get_auth_request con 0x7f0154121c10 auth_method 0 2024-09-11T01:49:20.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.537+0000 7f0159150640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:20.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.537+0000 7f0159150640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:20.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.537+0000 7f0159150640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:20.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.537+0000 7f0159150640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:20.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.537+0000 7f0159150640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:20.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.537+0000 7f0159150640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:20.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.537+0000 7f0159150640 10 monclient(hunting): handle_auth_done global_id 214233 payload 824 2024-09-11T01:49:20.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.537+0000 7f0159150640 1 -- 172.21.15.12:0/290478577 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0154135900 con 0x7f0154121c10 2024-09-11T01:49:20.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.537+0000 7f0149ffb640 1 --2- 172.21.15.12:0/290478577 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f013803e210 0x7f01380406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:20.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.537+0000 7f0149ffb640 1 -- 172.21.15.12:0/290478577 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(677..677 src has 507..677) ==== 4803+0+0 (secure 0 0 0) 0x7f01440738d0 con 0x7f0154137e50 2024-09-11T01:49:20.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.538+0000 7f0159951640 1 --2- 172.21.15.12:0/290478577 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f013803e210 0x7f01380406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:20.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.538+0000 7f0159951640 10 monclient: get_auth_request con 0x7f013803e210 auth_method 0 2024-09-11T01:49:20.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.540+0000 7f0159150640 1 -- 172.21.15.12:0/290478577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0154121c10 msgr2=0x7f0154121ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:20.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.540+0000 7f0159150640 1 --2- 172.21.15.12:0/290478577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0154121c10 0x7f0154121ff0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f015000e770 tx=0x7f015000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:20.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.540+0000 7f0149ffb640 1 -- 172.21.15.12:0/290478577 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0150019020 con 0x7f0154121c10 2024-09-11T01:49:20.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.540+0000 7f0149ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:20.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.540+0000 7f0149ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:20.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.540+0000 7f0149ffb640 10 monclient: dump: 2024-09-11T01:49:20.555 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:20.556 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:20.556 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.556 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.556 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:20.556 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:20.556 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:20.556 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:20.556 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:20.556 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:20.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.540+0000 7f0149ffb640 1 -- 172.21.15.12:0/290478577 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f015000aba0 con 0x7f0154121c10 2024-09-11T01:49:20.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.540+0000 7f0149ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:20.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.540+0000 7f0149ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:49:20.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.540+0000 7f0149ffb640 1 -- 172.21.15.12:0/290478577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0154121c10 msgr2=0x7f0154121ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:20.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.540+0000 7f0149ffb640 1 --2- 172.21.15.12:0/290478577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0154121c10 0x7f0154121ff0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f015000e770 tx=0x7f015000ec30 comp rx=0 tx=0).stop 2024-09-11T01:49:20.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.540+0000 7f0149ffb640 1 -- 172.21.15.12:0/290478577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0154121c10 msgr2=0x7f0154121ff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:49:20.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.540+0000 7f0149ffb640 1 --2- 172.21.15.12:0/290478577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0154121c10 0x7f0154121ff0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.540+0000 7f0159951640 1 --2- 172.21.15.12:0/290478577 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f013803e210 0x7f01380406d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0154123a80 tx=0x7f014c003d10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:20.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.546+0000 7f015b3b4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:49:20.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.546+0000 7f015b3b4640 1 --2- 172.21.15.12:0/290478577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0154122530 0x7f01540a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:20.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.546+0000 7f015b3b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:20.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.546+0000 7f015b3b4640 1 -- 172.21.15.12:0/290478577 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0154125270 con 0x7f0154122530 2024-09-11T01:49:20.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.546+0000 7f0159150640 1 --2- 172.21.15.12:0/290478577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0154122530 0x7f01540a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:20.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.546+0000 7f0159150640 10 monclient: get_auth_request con 0x7f0154122530 auth_method 0 2024-09-11T01:49:20.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.546+0000 7f0159150640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:20.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.546+0000 7f0159150640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:20.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.546+0000 7f0159150640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:20.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.546+0000 7f0159150640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:20.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.546+0000 7f0159150640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:20.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.546+0000 7f0159150640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:20.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.547+0000 7f0159150640 10 monclient(hunting): handle_auth_done global_id 214236 payload 824 2024-09-11T01:49:20.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.547+0000 7f0159150640 1 -- 172.21.15.12:0/290478577 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0154069950 con 0x7f0154122530 2024-09-11T01:49:20.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.549+0000 7f0159150640 1 -- 172.21.15.12:0/290478577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0154122530 msgr2=0x7f01540a0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:20.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.549+0000 7f0159150640 1 --2- 172.21.15.12:0/290478577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0154122530 0x7f01540a0430 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f015001a000 tx=0x7f01500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:20.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.549+0000 7f0149ffb640 1 -- 172.21.15.12:0/290478577 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0150019070 con 0x7f0154122530 2024-09-11T01:49:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.549+0000 7f0149ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.549+0000 7f0149ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.549+0000 7f0149ffb640 10 monclient: dump: 2024-09-11T01:49:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:20.559 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:20.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.549+0000 7f0149ffb640 1 -- 172.21.15.12:0/290478577 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7f01500202b0 con 0x7f0154122530 2024-09-11T01:49:20.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.549+0000 7f0149ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:49:20.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.549+0000 7f0149ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:49:20.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.549+0000 7f0149ffb640 1 -- 172.21.15.12:0/290478577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0154122530 msgr2=0x7f01540a0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:20.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.549+0000 7f0149ffb640 1 --2- 172.21.15.12:0/290478577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0154122530 0x7f01540a0430 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f015001a000 tx=0x7f01500046c0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.549+0000 7f0149ffb640 1 -- 172.21.15.12:0/290478577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0154122530 msgr2=0x7f01540a0430 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:49:20.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.549+0000 7f0149ffb640 1 --2- 172.21.15.12:0/290478577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0154122530 0x7f01540a0430 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f015001a000 tx=0x7f01500046c0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.560 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:49:20.560 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:49:20.560 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:49:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:49:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 188, 2024-09-11T01:49:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 66749, 2024-09-11T01:49:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:49:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:49:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:49:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:49:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 7, 2024-09-11T01:49:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:49:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:49:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:49:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:49:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:49:20.562 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:49:20.562 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:49:20.562 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:49:20.562 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:49:20.562 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:49:20.562 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:49:20.562 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:49:20.562 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:49:20.562 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:49:20.562 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:20.562 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:49:20.562 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:49:20.562 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:49:20.562 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:49:20.563 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:49:20.563 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:49:20.563 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:49:20.563 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:49:20.563 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:49:20.563 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:49:20.563 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:49:20.563 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:49:20.563 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:49:20.563 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:20.563 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.563 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:49:20.563 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:49:20.564 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:49:20.564 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:49:20.564 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:49:20.564 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:49:20.564 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:49:20.564 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:49:20.564 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:49:20.564 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:49:20.564 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:49:20.564 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:49:20.564 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:49:20.564 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:49:20.564 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:49:20.564 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:49:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:49:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:49:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:49:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:49:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:49:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:49:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:49:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:49:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:49:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:49:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:49:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:49:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:20.565 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:49:20.566 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.566 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:49:20.566 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.566 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:49:20.566 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:49:20.566 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:49:20.566 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:49:20.566 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.566 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:49:20.566 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:49:20.566 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:20.566 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.566 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.566 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:49:20.567 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:49:20.567 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:20.567 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:20.567 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:20.567 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.567 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:49:20.567 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:49:20.567 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:49:20.567 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:49:20.567 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:49:20.567 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.567 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.567 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:49:20.567 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:49:20.567 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:49:20.568 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:49:20.568 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.568 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:49:20.568 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:49:20.568 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:20.568 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.568 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.568 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:49:20.568 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:49:20.568 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:20.568 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:20.568 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:20.568 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.568 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:49:20.569 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:49:20.569 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:49:20.569 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:49:20.569 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:49:20.569 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.569 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.569 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:49:20.569 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:49:20.569 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:49:20.569 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:49:20.569 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.569 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:49:20.569 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:49:20.570 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:20.570 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.570 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.570 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:49:20.570 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:49:20.570 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:20.570 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:20.570 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:20.570 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.570 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:49:20.570 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:49:20.570 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:49:20.570 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:49:20.570 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:49:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:49:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:49:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:49:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:49:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:49:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:49:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:49:20.571 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:49:20.572 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:49:20.572 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:20.572 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:20.572 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.572 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:49:20.572 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:49:20.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.550+0000 7f015b3b4640 1 -- 172.21.15.12:0/290478577 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f013803e210 msgr2=0x7f01380406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:20.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.550+0000 7f015b3b4640 1 --2- 172.21.15.12:0/290478577 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f013803e210 0x7f01380406d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0154123a80 tx=0x7f014c003d10 comp rx=0 tx=0).stop 2024-09-11T01:49:20.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.550+0000 7f015b3b4640 10 monclient: shutdown 2024-09-11T01:49:20.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.550+0000 7f015b3b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:20.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.550+0000 7f015b3b4640 1 -- 172.21.15.12:0/290478577 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0154137e50 msgr2=0x7f01541cd910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:20.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.550+0000 7f015b3b4640 1 --2- 172.21.15.12:0/290478577 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0154137e50 0x7f01541cd910 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f01440090b0 tx=0x7f01440046c0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.550+0000 7f0159951640 1 -- 172.21.15.12:0/290478577 reap_dead start 2024-09-11T01:49:20.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.550+0000 7f015b3b4640 1 -- 172.21.15.12:0/290478577 shutdown_connections 2024-09-11T01:49:20.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.550+0000 7f015b3b4640 1 --2- 172.21.15.12:0/290478577 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0154137e50 0x7f01541cd910 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.550+0000 7f015b3b4640 1 -- 172.21.15.12:0/290478577 >> 172.21.15.12:0/290478577 conn(0x7f01540a7020 msgr2=0x7f0154137620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:20.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.550+0000 7f015b3b4640 1 -- 172.21.15.12:0/290478577 shutdown_connections 2024-09-11T01:49:20.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.550+0000 7f015b3b4640 1 -- 172.21.15.12:0/290478577 wait complete. 2024-09-11T01:49:20.574 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:49:20.575 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:49:20.575 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:49:20.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.675+0000 7faaa6218640 10 monclient: get_monmap_and_config 2024-09-11T01:49:20.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.675+0000 7faaa6218640 10 monclient: build_initial_monmap 2024-09-11T01:49:20.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.675+0000 7faaa6218640 10 monclient: monmap: 2024-09-11T01:49:20.677 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:20.677 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:20.677 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:49:20.676208+0000 2024-09-11T01:49:20.677 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:49:20.676208+0000 2024-09-11T01:49:20.677 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:20.677 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:20.678 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:49:20.678 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:49:20.678 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:49:20.678 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:49:20.678 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:49:20.678 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:20.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.676+0000 7faaa6218640 1 Processor -- start 2024-09-11T01:49:20.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.676+0000 7faaa6218640 1 -- start start 2024-09-11T01:49:20.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.676+0000 7faaa6218640 10 monclient: init 2024-09-11T01:49:20.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.676+0000 7faaa6218640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:20.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.676+0000 7faaa6218640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-11T01:49:20.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.676+0000 7faaa6218640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7faaa00a1380 0x7faaa00a1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:20.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.676+0000 7faaa6218640 10 monclient(hunting): picked mon.noname-a con 0x7faaa00a1380 addr v2:172.21.15.12:3300/0 2024-09-11T01:49:20.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.676+0000 7faaa6218640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faaa00a0370 addr v1:172.21.15.12:6789/0 2024-09-11T01:49:20.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.676+0000 7faaa6218640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faaa00a0c90 addr v1:172.21.15.71:6789/0 2024-09-11T01:49:20.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.676+0000 7faaa6218640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faaa0136900 con 0x7faaa00a0370 2024-09-11T01:49:20.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.676+0000 7faaa6218640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faaa00a1ca0 con 0x7faaa00a0c90 2024-09-11T01:49:20.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.676+0000 7faaa6218640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:20.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.676+0000 7faaa6218640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7faaa00a21c0 con 0x7faaa00a1380 2024-09-11T01:49:20.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.676+0000 7faaa6218640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:20.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.676+0000 7faaa6218640 10 monclient(hunting): authenticate will time out at 1967.118930s 2024-09-11T01:49:20.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.676+0000 7faa9f7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7faaa00a1380 0x7faaa00a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:20.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.676+0000 7faa9effd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7faaa00a0370 0x7faaa0135ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59898/0 (socket says 172.21.15.12:59898) 2024-09-11T01:49:20.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.677+0000 7faa9effd640 1 -- 172.21.15.12:0/3239886091 learned_addr learned my addr 172.21.15.12:0/3239886091 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:49:20.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.677+0000 7faa9f7fe640 10 monclient(hunting): get_auth_request con 0x7faaa00a1380 auth_method 0 2024-09-11T01:49:20.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.677+0000 7faa9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:20.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.677+0000 7faa9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:20.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.677+0000 7faa9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:20.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.677+0000 7faa9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:20.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.677+0000 7faa9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:20.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.677+0000 7faa9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:20.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.677+0000 7faa9f7fe640 10 monclient(hunting): handle_auth_done global_id 214239 payload 274 2024-09-11T01:49:20.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.677+0000 7faa9f7fe640 1 -- 172.21.15.12:0/3239886091 >> v1:172.21.15.71:6789/0 conn(0x7faaa00a0c90 legacy=0x7faaa01361f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:49:20.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.677+0000 7faa9f7fe640 1 -- 172.21.15.12:0/3239886091 >> v1:172.21.15.12:6789/0 conn(0x7faaa00a0370 legacy=0x7faaa0135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:49:20.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.677+0000 7faa9f7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:49:20.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.677+0000 7faa9f7fe640 1 monclient: found mon.noname-a 2024-09-11T01:49:20.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.677+0000 7faa9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:20.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.677+0000 7faa9f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:49:20.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.677+0000 7faa9f7fe640 1 -- 172.21.15.12:0/3239886091 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaa00a24a0 con 0x7faaa00a1380 2024-09-11T01:49:20.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faa9f7fe640 1 -- 172.21.15.12:0/3239886091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faaa00a1380 msgr2=0x7faaa00a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:20.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faa9f7fe640 1 --2- 172.21.15.12:0/3239886091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faaa00a1380 0x7faaa00a1760 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7faa900097a0 tx=0x7faa90029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3f902dfd276ef15 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:20.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faa9e7fc640 1 -- 172.21.15.12:0/3239886091 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa90037070 con 0x7faaa00a1380 2024-09-11T01:49:20.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faa9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:20.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faa9e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:49:20.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faa9e7fc640 10 monclient: dump: 2024-09-11T01:49:20.689 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:20.689 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:20.689 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.689 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.689 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:20.689 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:20.689 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:20.689 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:20.690 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:20.690 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:20.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faa9e7fc640 10 monclient: _finish_auth 0 2024-09-11T01:49:20.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faa9e7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:49:20.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faa9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:20.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faa9e7fc640 1 -- 172.21.15.12:0/3239886091 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa90032640 con 0x7faaa00a1380 2024-09-11T01:49:20.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faaa6218640 5 monclient: authenticate success, global_id 214239 2024-09-11T01:49:20.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faaa6218640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:49:20.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faa9e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:20.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faaa6218640 10 monclient: get_monmap_and_config success 2024-09-11T01:49:20.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faa9e7fc640 1 -- 172.21.15.12:0/3239886091 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa90032a10 con 0x7faaa00a1380 2024-09-11T01:49:20.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faa9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:20.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faaa6218640 4 set_mon_vals no callback set 2024-09-11T01:49:20.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faa9e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:20.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faa9e7fc640 10 monclient: dump: 2024-09-11T01:49:20.691 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:20.691 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:20.691 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.691 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.691 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:20.691 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:20.691 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:20.691 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:20.691 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:20.691 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:20.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faaa6218640 10 monclient: shutdown 2024-09-11T01:49:20.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faaa6218640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:20.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faaa6218640 1 -- 172.21.15.12:0/3239886091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faaa00a1380 msgr2=0x7faaa00a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:20.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faaa6218640 1 --2- 172.21.15.12:0/3239886091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faaa00a1380 0x7faaa00a1760 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7faa900097a0 tx=0x7faa90029fd0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faaa6218640 1 -- 172.21.15.12:0/3239886091 shutdown_connections 2024-09-11T01:49:20.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faaa6218640 1 --2- 172.21.15.12:0/3239886091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faaa00a1380 0x7faaa00a1760 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faaa6218640 1 -- 172.21.15.12:0/3239886091 >> 172.21.15.12:0/3239886091 conn(0x7faaa00a2f70 msgr2=0x7faaa00a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:20.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faaa6218640 1 -- 172.21.15.12:0/3239886091 shutdown_connections 2024-09-11T01:49:20.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.680+0000 7faaa6218640 1 -- 172.21.15.12:0/3239886091 wait complete. 2024-09-11T01:49:20.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.681+0000 7faaa6218640 10 monclient: build_initial_monmap 2024-09-11T01:49:20.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.681+0000 7faaa6218640 10 monclient: monmap: 2024-09-11T01:49:20.692 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:20.692 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:49:20.692 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:49:20.692 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:49:20.692 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:20.693 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:20.693 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:49:20.693 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:49:20.693 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:49:20.693 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:20.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.681+0000 7faaa6218640 1 Processor -- start 2024-09-11T01:49:20.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faaa6218640 1 -- start start 2024-09-11T01:49:20.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faaa6218640 10 monclient: init 2024-09-11T01:49:20.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faaa6218640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:20.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faaa6218640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:49:20.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faaa6218640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faaa00a0370 0x7faaa013db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:20.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faaa6218640 10 monclient(hunting): picked mon.noname-b con 0x7faaa00a0370 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:49:20.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faaa6218640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faaa00a0c90 0x7faaa0135c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:20.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faaa6218640 10 monclient(hunting): picked mon.noname-c con 0x7faaa00a0c90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:20.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faaa6218640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faaa00a1380 0x7faaa009d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:20.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faaa6218640 10 monclient(hunting): picked mon.noname-a con 0x7faaa00a1380 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:49:20.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faaa6218640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:20.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faaa6218640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7faaa009cca0 con 0x7faaa00a1380 2024-09-11T01:49:20.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faaa6218640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:20.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faaa6218640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7faaa009fb60 con 0x7faaa00a0370 2024-09-11T01:49:20.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faaa6218640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:20.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faaa6218640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7faaa009ce20 con 0x7faaa00a0c90 2024-09-11T01:49:20.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faaa6218640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:20.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faaa6218640 10 monclient(hunting): authenticate will time out at 1967.124477s 2024-09-11T01:49:20.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faaa00a1380 0x7faaa009d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:20.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9effd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faaa00a0c90 0x7faaa0135c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:20.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faaa00a1380 0x7faaa009d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43826/0 (socket says 172.21.15.12:43826) 2024-09-11T01:49:20.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9ffff640 1 -- 172.21.15.12:0/1332340025 learned_addr learned my addr 172.21.15.12:0/1332340025 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:49:20.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9ffff640 10 monclient(hunting): get_auth_request con 0x7faaa00a1380 auth_method 0 2024-09-11T01:49:20.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:20.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:20.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:20.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9effd640 10 monclient(hunting): get_auth_request con 0x7faaa00a0c90 auth_method 0 2024-09-11T01:49:20.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:20.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:20.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:20.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:20.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:20.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9f7fe640 1 --2- 172.21.15.12:0/1332340025 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faaa00a0370 0x7faaa013db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:20.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:20.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9f7fe640 10 monclient(hunting): get_auth_request con 0x7faaa00a0370 auth_method 0 2024-09-11T01:49:20.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:20.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:20.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:20.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:20.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:20.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:20.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:20.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:20.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:20.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9ffff640 10 monclient(hunting): handle_auth_done global_id 214242 payload 824 2024-09-11T01:49:20.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9ffff640 1 -- 172.21.15.12:0/1332340025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faaa00a0c90 msgr2=0x7faaa0135c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:20.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9ffff640 1 --2- 172.21.15.12:0/1332340025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faaa00a0c90 0x7faaa0135c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9ffff640 1 -- 172.21.15.12:0/1332340025 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faaa00a0370 msgr2=0x7faaa013db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:20.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9ffff640 1 --2- 172.21.15.12:0/1332340025 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faaa00a0370 0x7faaa013db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:49:20.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9ffff640 1 monclient: found mon.noname-a 2024-09-11T01:49:20.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:20.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:49:20.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.682+0000 7faa9ffff640 1 -- 172.21.15.12:0/1332340025 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaa0120d90 con 0x7faaa00a1380 2024-09-11T01:49:20.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.685+0000 7faa9ffff640 1 -- 172.21.15.12:0/1332340025 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faaa00a1380 msgr2=0x7faaa009d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:20.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.685+0000 7faa9ffff640 1 --2- 172.21.15.12:0/1332340025 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faaa00a1380 0x7faaa009d790 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7faa9800c700 tx=0x7faa9800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:20.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.685+0000 7faa87fff640 1 -- 172.21.15.12:0/1332340025 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa98016020 con 0x7faaa00a1380 2024-09-11T01:49:20.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.685+0000 7faa87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:20.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.685+0000 7faa87fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:49:20.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.685+0000 7faa87fff640 10 monclient: dump: 2024-09-11T01:49:20.698 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:20.698 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:20.698 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.698 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.698 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:20.698 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:20.698 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:20.698 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:20.698 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:20.698 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:20.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.685+0000 7faa87fff640 10 monclient: _finish_auth 0 2024-09-11T01:49:20.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.685+0000 7faa87fff640 10 monclient: _check_auth_tickets 2024-09-11T01:49:20.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.685+0000 7faa87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:20.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.685+0000 7faa87fff640 1 -- 172.21.15.12:0/1332340025 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa9800ac60 con 0x7faaa00a1380 2024-09-11T01:49:20.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.685+0000 7faaa6218640 5 monclient: authenticate success, global_id 214242 2024-09-11T01:49:20.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.685+0000 7faaa6218640 10 monclient: _renew_subs 2024-09-11T01:49:20.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.685+0000 7faaa6218640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:20.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.685+0000 7faaa6218640 1 -- 172.21.15.12:0/1332340025 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaa009d480 con 0x7faaa00a1380 2024-09-11T01:49:20.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.685+0000 7faa87fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:20.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.685+0000 7faa87fff640 1 -- 172.21.15.12:0/1332340025 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa98012660 con 0x7faaa00a1380 2024-09-11T01:49:20.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.685+0000 7faa87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:20.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.685+0000 7faa87fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:20.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.685+0000 7faa87fff640 10 monclient: dump: 2024-09-11T01:49:20.699 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:20.699 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:20.699 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.699 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.700 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:20.700 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:20.700 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:20.700 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:20.700 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:20.700 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:20.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.685+0000 7faaa6218640 10 monclient: _renew_subs 2024-09-11T01:49:20.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.685+0000 7faaa6218640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:20.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.685+0000 7faaa6218640 1 -- 172.21.15.12:0/1332340025 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaa0121630 con 0x7faaa00a1380 2024-09-11T01:49:20.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.685+0000 7faa9dffb640 4 set_mon_vals no callback set 2024-09-11T01:49:20.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.686+0000 7faaa6218640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:20.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.686+0000 7faaa6218640 1 --2- 172.21.15.12:0/1332340025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faaa009eb00 0x7faaa002d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:20.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.686+0000 7faaa6218640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:20.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.686+0000 7faaa6218640 1 -- 172.21.15.12:0/1332340025 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7faaa00a1760 con 0x7faaa009eb00 2024-09-11T01:49:20.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.686+0000 7faa87fff640 1 -- 172.21.15.12:0/1332340025 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7faa980128c0 con 0x7faaa00a1380 2024-09-11T01:49:20.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.686+0000 7faa9f7fe640 1 --2- 172.21.15.12:0/1332340025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faaa009eb00 0x7faaa002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:20.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.686+0000 7faa9f7fe640 10 monclient: get_auth_request con 0x7faaa009eb00 auth_method 0 2024-09-11T01:49:20.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.686+0000 7faa9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:20.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.686+0000 7faa9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:20.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.686+0000 7faa9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:20.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.686+0000 7faa9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:20.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.686+0000 7faa9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:20.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.686+0000 7faa9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:20.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.686+0000 7faa9f7fe640 10 monclient(hunting): handle_auth_done global_id 224126 payload 824 2024-09-11T01:49:20.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.686+0000 7faa9f7fe640 1 -- 172.21.15.12:0/1332340025 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faaa013a1f0 con 0x7faaa009eb00 2024-09-11T01:49:20.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.686+0000 7faa87fff640 1 --2- 172.21.15.12:0/1332340025 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7faa8c03e260 0x7faa8c040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:20.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.686+0000 7faa87fff640 1 -- 172.21.15.12:0/1332340025 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(677..677 src has 507..677) ==== 4803+0+0 (secure 0 0 0) 0x7faa98052f60 con 0x7faaa00a1380 2024-09-11T01:49:20.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.686+0000 7faa9effd640 1 --2- 172.21.15.12:0/1332340025 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7faa8c03e260 0x7faa8c040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:20.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.686+0000 7faa9effd640 10 monclient: get_auth_request con 0x7faa8c03e260 auth_method 0 2024-09-11T01:49:20.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.688+0000 7faa9f7fe640 1 -- 172.21.15.12:0/1332340025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faaa009eb00 msgr2=0x7faaa002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:20.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.688+0000 7faa9f7fe640 1 --2- 172.21.15.12:0/1332340025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faaa009eb00 0x7faaa002d440 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7faa90034000 tx=0x7faa900046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:20.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.688+0000 7faa87fff640 1 -- 172.21.15.12:0/1332340025 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa90037040 con 0x7faaa009eb00 2024-09-11T01:49:20.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.688+0000 7faa87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:20.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.688+0000 7faa87fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:49:20.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.688+0000 7faa87fff640 10 monclient: dump: 2024-09-11T01:49:20.705 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:20.705 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:20.705 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.705 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.705 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:20.705 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:20.705 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:20.705 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:20.705 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:20.705 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:20.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.689+0000 7faa87fff640 1 -- 172.21.15.12:0/1332340025 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7faa9003c820 con 0x7faaa009eb00 2024-09-11T01:49:20.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.689+0000 7faa87fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:20.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.689+0000 7faa87fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:49:20.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.689+0000 7faa87fff640 1 -- 172.21.15.12:0/1332340025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faaa009eb00 msgr2=0x7faaa002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:20.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.689+0000 7faa87fff640 1 --2- 172.21.15.12:0/1332340025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faaa009eb00 0x7faaa002d440 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7faa90034000 tx=0x7faa900046c0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.689+0000 7faa87fff640 1 -- 172.21.15.12:0/1332340025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faaa009eb00 msgr2=0x7faaa002d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:49:20.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.689+0000 7faa87fff640 1 --2- 172.21.15.12:0/1332340025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faaa009eb00 0x7faaa002d440 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7faa90034000 tx=0x7faa900046c0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.689+0000 7faa9effd640 1 --2- 172.21.15.12:0/1332340025 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7faa8c03e260 0x7faa8c040720 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7faa94007c90 tx=0x7faa94007be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:20.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.695+0000 7faaa6218640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:49:20.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.695+0000 7faaa6218640 1 --2- 172.21.15.12:0/1332340025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faaa01361d0 0x7faaa002d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:20.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.695+0000 7faaa6218640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:20.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.695+0000 7faaa6218640 1 -- 172.21.15.12:0/1332340025 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7faaa006d480 con 0x7faaa01361d0 2024-09-11T01:49:20.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.695+0000 7faa9f7fe640 1 --2- 172.21.15.12:0/1332340025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faaa01361d0 0x7faaa002d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:20.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.695+0000 7faa9f7fe640 10 monclient: get_auth_request con 0x7faaa01361d0 auth_method 0 2024-09-11T01:49:20.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.695+0000 7faa9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:20.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.695+0000 7faa9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:20.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.695+0000 7faa9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.695+0000 7faa9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.695+0000 7faa9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.695+0000 7faa9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.695+0000 7faa9f7fe640 10 monclient(hunting): handle_auth_done global_id 224129 payload 824 2024-09-11T01:49:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.695+0000 7faa9f7fe640 1 -- 172.21.15.12:0/1332340025 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faaa009e490 con 0x7faaa01361d0 2024-09-11T01:49:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.698+0000 7faa9f7fe640 1 -- 172.21.15.12:0/1332340025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faaa01361d0 msgr2=0x7faaa002d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.698+0000 7faa9f7fe640 1 --2- 172.21.15.12:0/1332340025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faaa01361d0 0x7faaa002d980 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7faa9002a010 tx=0x7faa90008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.698+0000 7faa87fff640 1 -- 172.21.15.12:0/1332340025 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa90050070 con 0x7faaa01361d0 2024-09-11T01:49:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.698+0000 7faa87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.698+0000 7faa87fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:49:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.698+0000 7faa87fff640 10 monclient: dump: 2024-09-11T01:49:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:20.707 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:20.708 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.698+0000 7faa87fff640 1 -- 172.21.15.12:0/1332340025 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7faa9004c810 con 0x7faaa01361d0 2024-09-11T01:49:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.698+0000 7faa87fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:49:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.698+0000 7faa87fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:49:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.698+0000 7faa87fff640 1 -- 172.21.15.12:0/1332340025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faaa01361d0 msgr2=0x7faaa002d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:20.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.698+0000 7faa87fff640 1 --2- 172.21.15.12:0/1332340025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faaa01361d0 0x7faaa002d980 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7faa9002a010 tx=0x7faa90008030 comp rx=0 tx=0).stop 2024-09-11T01:49:20.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.698+0000 7faa87fff640 1 -- 172.21.15.12:0/1332340025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faaa01361d0 msgr2=0x7faaa002d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:49:20.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.698+0000 7faa87fff640 1 --2- 172.21.15.12:0/1332340025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faaa01361d0 0x7faaa002d980 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7faa9002a010 tx=0x7faa90008030 comp rx=0 tx=0).stop 2024-09-11T01:49:20.709 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:49:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:49:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:49:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:49:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 188, 2024-09-11T01:49:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 13470, 2024-09-11T01:49:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:49:20.709 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:49:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:49:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:49:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 7, 2024-09-11T01:49:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:49:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:49:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:49:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:49:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:49:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:49:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:49:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:49:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:49:20.710 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:49:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:49:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:49:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:49:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:49:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:49:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:49:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:49:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:49:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:49:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:49:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:49:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:49:20.711 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:49:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:49:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:49:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:49:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:49:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:49:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:49:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:49:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:49:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:49:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:49:20.712 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:49:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:49:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:49:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:49:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:49:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:49:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:49:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:49:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:49:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:49:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:49:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:49:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:49:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:49:20.713 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:49:20.714 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:49:20.714 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:49:20.714 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:49:20.714 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:49:20.714 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:49:20.714 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:49:20.714 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:49:20.714 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:20.714 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:49:20.714 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.714 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:49:20.714 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.714 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:49:20.714 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:49:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:49:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:49:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:49:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:49:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:49:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:49:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:20.715 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:49:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:49:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:49:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:49:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:49:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:49:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:49:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:49:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:49:20.716 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:49:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:49:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:49:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:49:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:49:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:49:20.717 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:49:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:49:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:49:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:49:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:49:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:49:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:49:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:49:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:49:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.718 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:49:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:49:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:49:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:49:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:49:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:49:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:49:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:20.719 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:49:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:49:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:49:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:49:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:49:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:49:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:49:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:49:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:49:20.720 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.721 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:49:20.721 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:49:20.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.698+0000 7faaa6218640 1 -- 172.21.15.12:0/1332340025 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7faa8c03e260 msgr2=0x7faa8c040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:20.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.698+0000 7faaa6218640 1 --2- 172.21.15.12:0/1332340025 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7faa8c03e260 0x7faa8c040720 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7faa94007c90 tx=0x7faa94007be0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.698+0000 7faaa6218640 10 monclient: shutdown 2024-09-11T01:49:20.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.698+0000 7faaa6218640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:20.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.698+0000 7faaa6218640 1 -- 172.21.15.12:0/1332340025 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faaa00a1380 msgr2=0x7faaa009d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:20.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.698+0000 7faaa6218640 1 --2- 172.21.15.12:0/1332340025 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faaa00a1380 0x7faaa009d790 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7faa9800c700 tx=0x7faa9800cbc0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.698+0000 7faa9ffff640 1 -- 172.21.15.12:0/1332340025 reap_dead start 2024-09-11T01:49:20.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.698+0000 7faa9ffff640 1 -- 172.21.15.12:0/1332340025 reap_dead start 2024-09-11T01:49:20.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.698+0000 7faaa6218640 1 -- 172.21.15.12:0/1332340025 shutdown_connections 2024-09-11T01:49:20.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.698+0000 7faaa6218640 1 -- 172.21.15.12:0/1332340025 >> 172.21.15.12:0/1332340025 conn(0x7faaa00a2f70 msgr2=0x7faaa0071e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:20.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.699+0000 7faaa6218640 1 -- 172.21.15.12:0/1332340025 shutdown_connections 2024-09-11T01:49:20.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.699+0000 7faaa6218640 1 -- 172.21.15.12:0/1332340025 wait complete. 2024-09-11T01:49:20.723 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:49:20.723 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:49:20.723 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:49:20.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.824+0000 7fb4afe47640 10 monclient: get_monmap_and_config 2024-09-11T01:49:20.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.824+0000 7fb4afe47640 10 monclient: build_initial_monmap 2024-09-11T01:49:20.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.824+0000 7fb4afe47640 10 monclient: monmap: 2024-09-11T01:49:20.826 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:20.826 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:20.826 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:49:20.824785+0000 2024-09-11T01:49:20.826 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:49:20.824785+0000 2024-09-11T01:49:20.827 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:20.827 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:20.827 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:49:20.827 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:49:20.827 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:49:20.827 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:49:20.827 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:49:20.827 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:20.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.824+0000 7fb4afe47640 1 Processor -- start 2024-09-11T01:49:20.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.824+0000 7fb4afe47640 1 -- start start 2024-09-11T01:49:20.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.824+0000 7fb4afe47640 10 monclient: init 2024-09-11T01:49:20.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4afe47640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:20.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4afe47640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-11T01:49:20.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4afe47640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb4a8137e50 addr v1:172.21.15.12:6789/0 2024-09-11T01:49:20.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4afe47640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb4a8138230 0x7fb4a8138640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:20.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4afe47640 10 monclient(hunting): picked mon.noname-a con 0x7fb4a8138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:49:20.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4afe47640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb4a813d380 0x7fb4a813f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:20.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4afe47640 10 monclient(hunting): picked mon.noname-c con 0x7fb4a813d380 addr v2:172.21.15.71:3300/0 2024-09-11T01:49:20.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4afe47640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4a80a62d0 con 0x7fb4a8137e50 2024-09-11T01:49:20.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4afe47640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:20.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4afe47640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fb4a809bb40 con 0x7fb4a8138230 2024-09-11T01:49:20.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4afe47640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:20.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4afe47640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fb4a809c4a0 con 0x7fb4a813d380 2024-09-11T01:49:20.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4afe47640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:20.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4afe47640 10 monclient(hunting): authenticate will time out at 1967.267391s 2024-09-11T01:49:20.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4ad3e2640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb4a8138230 0x7fb4a8138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:20.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4adbe3640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb4a8137e50 0x7fb4a809d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59906/0 (socket says 172.21.15.12:59906) 2024-09-11T01:49:20.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4adbe3640 1 -- 172.21.15.12:0/3508343547 learned_addr learned my addr 172.21.15.12:0/3508343547 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:49:20.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4ad3e2640 10 monclient(hunting): get_auth_request con 0x7fb4a8138230 auth_method 0 2024-09-11T01:49:20.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4ad3e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:20.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4ad3e2640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:20.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4ad3e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:20.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4ad3e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:20.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4ad3e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:20.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4ad3e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:20.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4ad3e2640 10 monclient(hunting): handle_auth_done global_id 214245 payload 274 2024-09-11T01:49:20.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4ad3e2640 1 -- 172.21.15.12:0/3508343547 >> v2:172.21.15.71:3300/0 conn(0x7fb4a813d380 msgr2=0x7fb4a813f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:49:20.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4ae3e4640 1 --2- 172.21.15.12:0/3508343547 >> v2:172.21.15.71:3300/0 conn(0x7fb4a813d380 0x7fb4a813f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:20.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4ad3e2640 1 --2- 172.21.15.12:0/3508343547 >> v2:172.21.15.71:3300/0 conn(0x7fb4a813d380 0x7fb4a813f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4ad3e2640 1 -- 172.21.15.12:0/3508343547 >> v1:172.21.15.12:6789/0 conn(0x7fb4a8137e50 legacy=0x7fb4a809d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:49:20.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4ad3e2640 10 monclient: _finish_hunting 0 2024-09-11T01:49:20.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4ad3e2640 1 monclient: found mon.noname-a 2024-09-11T01:49:20.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4ad3e2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:20.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4ad3e2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:49:20.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4ad3e2640 1 -- 172.21.15.12:0/3508343547 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4a809c780 con 0x7fb4a8138230 2024-09-11T01:49:20.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4ae3e4640 10 monclient: get_auth_request con 0x7fb4a813d380 auth_method 0 2024-09-11T01:49:20.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.825+0000 7fb4ae3e4640 1 --2- 172.21.15.12:0/3508343547 >> v2:172.21.15.71:3300/0 conn(0x7fb4a813d380 0x7fb4a813f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:49:20.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.828+0000 7fb4ad3e2640 1 -- 172.21.15.12:0/3508343547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4a8138230 msgr2=0x7fb4a8138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:20.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.828+0000 7fb4ad3e2640 1 --2- 172.21.15.12:0/3508343547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4a8138230 0x7fb4a8138640 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb4a809cf30 tx=0x7fb498029fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ae29545b7e462a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:20.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.828+0000 7fb4acbe1640 1 -- 172.21.15.12:0/3508343547 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb498037070 con 0x7fb4a8138230 2024-09-11T01:49:20.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.828+0000 7fb4acbe1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:20.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.828+0000 7fb4acbe1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:49:20.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.828+0000 7fb4acbe1640 10 monclient: dump: 2024-09-11T01:49:20.845 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:20.845 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:20.845 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.845 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.845 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:20.845 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:20.845 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:20.846 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:20.846 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:20.846 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:20.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.828+0000 7fb4acbe1640 10 monclient: _finish_auth 0 2024-09-11T01:49:20.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.828+0000 7fb4acbe1640 10 monclient: _check_auth_tickets 2024-09-11T01:49:20.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.828+0000 7fb4acbe1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:20.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.828+0000 7fb4acbe1640 1 -- 172.21.15.12:0/3508343547 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb498032440 con 0x7fb4a8138230 2024-09-11T01:49:20.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.828+0000 7fb4afe47640 5 monclient: authenticate success, global_id 214245 2024-09-11T01:49:20.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.828+0000 7fb4afe47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:49:20.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.828+0000 7fb4acbe1640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:20.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.828+0000 7fb4afe47640 10 monclient: get_monmap_and_config success 2024-09-11T01:49:20.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.828+0000 7fb4acbe1640 1 -- 172.21.15.12:0/3508343547 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb498032810 con 0x7fb4a8138230 2024-09-11T01:49:20.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.828+0000 7fb4acbe1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:20.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.828+0000 7fb4afe47640 4 set_mon_vals no callback set 2024-09-11T01:49:20.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.828+0000 7fb4acbe1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:20.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.828+0000 7fb4acbe1640 10 monclient: dump: 2024-09-11T01:49:20.847 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:20.847 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:20.847 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.847 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.847 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:20.847 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:20.847 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:20.847 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:20.848 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:20.848 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:20.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.829+0000 7fb4afe47640 10 monclient: shutdown 2024-09-11T01:49:20.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.829+0000 7fb4afe47640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:20.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.829+0000 7fb4afe47640 1 -- 172.21.15.12:0/3508343547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4a8138230 msgr2=0x7fb4a8138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:20.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.829+0000 7fb4afe47640 1 --2- 172.21.15.12:0/3508343547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4a8138230 0x7fb4a8138640 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb4a809cf30 tx=0x7fb498029fb0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.829+0000 7fb4afe47640 1 -- 172.21.15.12:0/3508343547 shutdown_connections 2024-09-11T01:49:20.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.829+0000 7fb4afe47640 1 --2- 172.21.15.12:0/3508343547 >> v2:172.21.15.71:3300/0 conn(0x7fb4a813d380 0x7fb4a813f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.829+0000 7fb4afe47640 1 --2- 172.21.15.12:0/3508343547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4a8138230 0x7fb4a8138640 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.829+0000 7fb4afe47640 1 -- 172.21.15.12:0/3508343547 >> 172.21.15.12:0/3508343547 conn(0x7fb4a80a7020 msgr2=0x7fb4a80a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:20.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.829+0000 7fb4afe47640 1 -- 172.21.15.12:0/3508343547 shutdown_connections 2024-09-11T01:49:20.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.829+0000 7fb4afe47640 1 -- 172.21.15.12:0/3508343547 wait complete. 2024-09-11T01:49:20.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.830+0000 7fb4afe47640 10 monclient: build_initial_monmap 2024-09-11T01:49:20.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.830+0000 7fb4afe47640 10 monclient: monmap: 2024-09-11T01:49:20.849 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:20.849 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:49:20.849 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:49:20.849 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:49:20.849 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:20.849 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:20.849 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:49:20.850 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:49:20.850 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:49:20.850 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:20.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.830+0000 7fb4afe47640 1 Processor -- start 2024-09-11T01:49:20.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.830+0000 7fb4afe47640 1 -- start start 2024-09-11T01:49:20.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.830+0000 7fb4afe47640 10 monclient: init 2024-09-11T01:49:20.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.830+0000 7fb4afe47640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:20.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.830+0000 7fb4afe47640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:49:20.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.830+0000 7fb4afe47640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8137e50 0x7fb4a81ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:20.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.830+0000 7fb4afe47640 10 monclient(hunting): picked mon.noname-b con 0x7fb4a8137e50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:49:20.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.830+0000 7fb4afe47640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4a8138230 0x7fb4a8136dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:20.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.830+0000 7fb4afe47640 10 monclient(hunting): picked mon.noname-a con 0x7fb4a8138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:49:20.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.830+0000 7fb4afe47640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4a813d380 0x7fb4a80a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:20.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.830+0000 7fb4afe47640 10 monclient(hunting): picked mon.noname-c con 0x7fb4a813d380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:20.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.830+0000 7fb4afe47640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:20.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.830+0000 7fb4afe47640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a806f890 con 0x7fb4a8138230 2024-09-11T01:49:20.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.830+0000 7fb4afe47640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:20.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.830+0000 7fb4afe47640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a806f710 con 0x7fb4a8137e50 2024-09-11T01:49:20.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.830+0000 7fb4afe47640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:20.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.830+0000 7fb4afe47640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb4a813fcb0 con 0x7fb4a813d380 2024-09-11T01:49:20.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.830+0000 7fb4afe47640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:20.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.830+0000 7fb4afe47640 10 monclient(hunting): authenticate will time out at 1967.272976s 2024-09-11T01:49:20.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ad3e2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4a8138230 0x7fb4a8136dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:20.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ae3e4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4a813d380 0x7fb4a80a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:20.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ad3e2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4a8138230 0x7fb4a8136dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43846/0 (socket says 172.21.15.12:43846) 2024-09-11T01:49:20.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ad3e2640 1 -- 172.21.15.12:0/4155348080 learned_addr learned my addr 172.21.15.12:0/4155348080 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:49:20.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ad3e2640 10 monclient(hunting): get_auth_request con 0x7fb4a8138230 auth_method 0 2024-09-11T01:49:20.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ad3e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:20.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ad3e2640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:20.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ad3e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:20.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ae3e4640 10 monclient(hunting): get_auth_request con 0x7fb4a813d380 auth_method 0 2024-09-11T01:49:20.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ae3e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:20.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ae3e4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:20.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ae3e4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:20.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ad3e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:20.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ad3e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:20.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ad3e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:20.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ae3e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:20.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ae3e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:20.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ae3e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:20.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4adbe3640 1 --2- 172.21.15.12:0/4155348080 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8137e50 0x7fb4a81ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:20.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4adbe3640 10 monclient(hunting): get_auth_request con 0x7fb4a8137e50 auth_method 0 2024-09-11T01:49:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4adbe3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4adbe3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4adbe3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ae3e4640 10 monclient(hunting): handle_auth_done global_id 224132 payload 824 2024-09-11T01:49:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ae3e4640 1 -- 172.21.15.12:0/4155348080 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8137e50 msgr2=0x7fb4a81ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ae3e4640 1 --2- 172.21.15.12:0/4155348080 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8137e50 0x7fb4a81ccdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ae3e4640 1 -- 172.21.15.12:0/4155348080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4a8138230 msgr2=0x7fb4a8136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ae3e4640 1 --2- 172.21.15.12:0/4155348080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4a8138230 0x7fb4a8136dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ae3e4640 10 monclient: _finish_hunting 0 2024-09-11T01:49:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ae3e4640 1 monclient: found mon.noname-c 2024-09-11T01:49:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ae3e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:20.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ae3e4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:49:20.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ae3e4640 1 -- 172.21.15.12:0/4155348080 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4a81cdd10 con 0x7fb4a813d380 2024-09-11T01:49:20.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.831+0000 7fb4ad3e2640 1 --2- 172.21.15.12:0/4155348080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb4a8138230 0x7fb4a8136dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:49:20.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.833+0000 7fb4ae3e4640 1 -- 172.21.15.12:0/4155348080 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4a813d380 msgr2=0x7fb4a80a19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:20.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.833+0000 7fb4ae3e4640 1 --2- 172.21.15.12:0/4155348080 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4a813d380 0x7fb4a80a19b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb4a000ce20 tx=0x7fb4a000a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:20.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.833+0000 7fb49e7fc640 1 -- 172.21.15.12:0/4155348080 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a001a070 con 0x7fb4a813d380 2024-09-11T01:49:20.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.833+0000 7fb49e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:20.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.833+0000 7fb49e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:49:20.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.833+0000 7fb49e7fc640 10 monclient: dump: 2024-09-11T01:49:20.855 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:20.855 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:20.855 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.855 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.856 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:20.856 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:20.856 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:20.856 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:20.856 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:20.856 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:20.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.833+0000 7fb49e7fc640 10 monclient: _finish_auth 0 2024-09-11T01:49:20.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.833+0000 7fb49e7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:49:20.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.833+0000 7fb49e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:20.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.833+0000 7fb49e7fc640 1 -- 172.21.15.12:0/4155348080 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4a000da50 con 0x7fb4a813d380 2024-09-11T01:49:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.833+0000 7fb49e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.833+0000 7fb49e7fc640 1 -- 172.21.15.12:0/4155348080 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a000de40 con 0x7fb4a813d380 2024-09-11T01:49:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.833+0000 7fb49e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.833+0000 7fb49e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:49:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.833+0000 7fb49e7fc640 10 monclient: dump: 2024-09-11T01:49:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:20.857 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:20.858 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:20.858 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:20.858 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:20.858 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:20.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.833+0000 7fb4afe47640 5 monclient: authenticate success, global_id 224132 2024-09-11T01:49:20.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.833+0000 7fb4afe47640 10 monclient: _renew_subs 2024-09-11T01:49:20.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.833+0000 7fb4afe47640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:49:20.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.833+0000 7fb4afe47640 1 -- 172.21.15.12:0/4155348080 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4a8121480 con 0x7fb4a813d380 2024-09-11T01:49:20.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.833+0000 7fb49ffff640 4 set_mon_vals no callback set 2024-09-11T01:49:20.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.833+0000 7fb4afe47640 10 monclient: _renew_subs 2024-09-11T01:49:20.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.833+0000 7fb4afe47640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:49:20.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.833+0000 7fb4afe47640 1 -- 172.21.15.12:0/4155348080 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4a80a3470 con 0x7fb4a813d380 2024-09-11T01:49:20.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.834+0000 7fb4afe47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:20.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.834+0000 7fb4afe47640 1 --2- 172.21.15.12:0/4155348080 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8120ec0 0x7fb4a80a1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:20.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.834+0000 7fb4afe47640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:20.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.834+0000 7fb4afe47640 1 -- 172.21.15.12:0/4155348080 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a806a060 con 0x7fb4a8120ec0 2024-09-11T01:49:20.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.834+0000 7fb49e7fc640 1 -- 172.21.15.12:0/4155348080 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fb4a001e3b0 con 0x7fb4a813d380 2024-09-11T01:49:20.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.835+0000 7fb4adbe3640 1 --2- 172.21.15.12:0/4155348080 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8120ec0 0x7fb4a80a1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:20.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.835+0000 7fb4adbe3640 10 monclient: get_auth_request con 0x7fb4a8120ec0 auth_method 0 2024-09-11T01:49:20.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.835+0000 7fb4adbe3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:20.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.835+0000 7fb4adbe3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:20.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.835+0000 7fb4adbe3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:20.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.835+0000 7fb49e7fc640 1 --2- 172.21.15.12:0/4155348080 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb48c03e260 0x7fb48c040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:20.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.835+0000 7fb4adbe3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:20.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.835+0000 7fb4adbe3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:20.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.835+0000 7fb4adbe3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:20.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.835+0000 7fb49e7fc640 1 -- 172.21.15.12:0/4155348080 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(677..677 src has 507..677) ==== 4803+0+0 (secure 0 0 0) 0x7fb4a0053830 con 0x7fb4a813d380 2024-09-11T01:49:20.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.835+0000 7fb4ad3e2640 1 --2- 172.21.15.12:0/4155348080 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb48c03e260 0x7fb48c040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:20.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.835+0000 7fb4adbe3640 10 monclient(hunting): handle_auth_done global_id 134866 payload 824 2024-09-11T01:49:20.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.835+0000 7fb4adbe3640 1 -- 172.21.15.12:0/4155348080 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb4a8138640 con 0x7fb4a8120ec0 2024-09-11T01:49:20.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.835+0000 7fb4ad3e2640 10 monclient: get_auth_request con 0x7fb48c03e260 auth_method 0 2024-09-11T01:49:20.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.837+0000 7fb4adbe3640 1 -- 172.21.15.12:0/4155348080 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8120ec0 msgr2=0x7fb4a80a1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:20.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.837+0000 7fb4adbe3640 1 --2- 172.21.15.12:0/4155348080 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8120ec0 0x7fb4a80a1ef0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb4a813f890 tx=0x7fb4a400e960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:20.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.837+0000 7fb49e7fc640 1 -- 172.21.15.12:0/4155348080 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a4005cf0 con 0x7fb4a8120ec0 2024-09-11T01:49:20.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.837+0000 7fb49e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:20.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.837+0000 7fb4ad3e2640 1 --2- 172.21.15.12:0/4155348080 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb48c03e260 0x7fb48c040720 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8137e20 tx=0x7fb49802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:20.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.837+0000 7fb49e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:49:20.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.837+0000 7fb49e7fc640 10 monclient: dump: 2024-09-11T01:49:20.861 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:20.861 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:20.862 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.862 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.862 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:20.862 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:20.862 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:20.862 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:20.862 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:20.862 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:20.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.838+0000 7fb49e7fc640 1 -- 172.21.15.12:0/4155348080 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb4a4018070 con 0x7fb4a8120ec0 2024-09-11T01:49:20.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.838+0000 7fb49e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:20.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.838+0000 7fb49e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:49:20.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.838+0000 7fb49e7fc640 1 -- 172.21.15.12:0/4155348080 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8120ec0 msgr2=0x7fb4a80a1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:20.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.838+0000 7fb49e7fc640 1 --2- 172.21.15.12:0/4155348080 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8120ec0 0x7fb4a80a1ef0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb4a813f890 tx=0x7fb4a400e960 comp rx=0 tx=0).stop 2024-09-11T01:49:20.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.838+0000 7fb49e7fc640 1 -- 172.21.15.12:0/4155348080 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8120ec0 msgr2=0x7fb4a80a1ef0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:49:20.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.838+0000 7fb49e7fc640 1 --2- 172.21.15.12:0/4155348080 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8120ec0 0x7fb4a80a1ef0 secure :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb4a813f890 tx=0x7fb4a400e960 comp rx=0 tx=0).stop 2024-09-11T01:49:20.864 INFO:tasks.rados.rados.0.smithi071.stdout:3019: done (6 left) 2024-09-11T01:49:20.864 INFO:tasks.rados.rados.0.smithi071.stdout:3020: snap_remove snap 274 2024-09-11T01:49:20.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.844+0000 7fb4afe47640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:49:20.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.844+0000 7fb4afe47640 1 --2- 172.21.15.12:0/4155348080 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8137310 0x7fb4a80a2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:20.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.844+0000 7fb4afe47640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:20.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.844+0000 7fb4afe47640 1 -- 172.21.15.12:0/4155348080 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a8069950 con 0x7fb4a8137310 2024-09-11T01:49:20.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.844+0000 7fb4adbe3640 1 --2- 172.21.15.12:0/4155348080 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8137310 0x7fb4a80a2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:20.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.844+0000 7fb4adbe3640 10 monclient: get_auth_request con 0x7fb4a8137310 auth_method 0 2024-09-11T01:49:20.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.844+0000 7fb4adbe3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:20.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.844+0000 7fb4adbe3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:20.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.844+0000 7fb4adbe3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:20.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.845+0000 7fb4adbe3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:20.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.845+0000 7fb4adbe3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:20.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.845+0000 7fb4adbe3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:20.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.845+0000 7fb4adbe3640 10 monclient(hunting): handle_auth_done global_id 134869 payload 824 2024-09-11T01:49:20.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.845+0000 7fb4adbe3640 1 -- 172.21.15.12:0/4155348080 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb4a80a1600 con 0x7fb4a8137310 2024-09-11T01:49:20.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.847+0000 7fb4adbe3640 1 -- 172.21.15.12:0/4155348080 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8137310 msgr2=0x7fb4a80a2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:20.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.847+0000 7fb4adbe3640 1 --2- 172.21.15.12:0/4155348080 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8137310 0x7fb4a80a2430 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4017000 tx=0x7fb4a4004940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:20.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.847+0000 7fb49e7fc640 1 -- 172.21.15.12:0/4155348080 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a40074e0 con 0x7fb4a8137310 2024-09-11T01:49:20.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.847+0000 7fb49e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:20.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.847+0000 7fb49e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:49:20.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.847+0000 7fb49e7fc640 10 monclient: dump: 2024-09-11T01:49:20.867 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:20.867 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:20.868 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.868 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:20.868 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:20.868 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:20.868 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:20.868 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:20.868 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:20.868 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:20.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.848+0000 7fb49e7fc640 1 -- 172.21.15.12:0/4155348080 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7fb4a40074e0 con 0x7fb4a8137310 2024-09-11T01:49:20.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.848+0000 7fb49e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:49:20.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.848+0000 7fb49e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:49:20.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.848+0000 7fb49e7fc640 1 -- 172.21.15.12:0/4155348080 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8137310 msgr2=0x7fb4a80a2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:20.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.848+0000 7fb49e7fc640 1 --2- 172.21.15.12:0/4155348080 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8137310 0x7fb4a80a2430 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4017000 tx=0x7fb4a4004940 comp rx=0 tx=0).stop 2024-09-11T01:49:20.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.848+0000 7fb49e7fc640 1 -- 172.21.15.12:0/4155348080 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8137310 msgr2=0x7fb4a80a2430 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:49:20.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.848+0000 7fb49e7fc640 1 --2- 172.21.15.12:0/4155348080 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8137310 0x7fb4a80a2430 secure :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4017000 tx=0x7fb4a4004940 comp rx=0 tx=0).stop 2024-09-11T01:49:20.869 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:49:20.869 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:49:20.870 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:49:20.870 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:49:20.870 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 188, 2024-09-11T01:49:20.870 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 493810, 2024-09-11T01:49:20.870 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:49:20.870 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:49:20.870 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:49:20.870 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:49:20.870 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:20.871 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 7, 2024-09-11T01:49:20.871 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:49:20.871 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:49:20.871 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:49:20.871 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:49:20.871 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:49:20.871 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:49:20.871 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:49:20.871 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:49:20.871 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:49:20.871 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:49:20.872 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:49:20.872 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:49:20.872 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:49:20.872 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:49:20.872 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:20.872 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:49:20.872 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:49:20.872 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:49:20.872 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:49:20.872 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:49:20.872 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:49:20.873 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:49:20.873 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:49:20.873 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:49:20.873 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:49:20.873 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:49:20.873 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:49:20.873 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:49:20.873 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:20.873 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.873 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:49:20.873 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:49:20.874 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:49:20.874 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:49:20.874 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:49:20.874 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:49:20.874 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:49:20.874 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:49:20.874 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:49:20.874 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:49:20.874 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:49:20.874 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:49:20.875 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:49:20.875 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:49:20.875 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:49:20.875 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:49:20.875 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:49:20.875 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:49:20.875 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:49:20.875 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:49:20.875 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:49:20.876 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:49:20.876 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:49:20.876 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:49:20.876 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:49:20.876 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:49:20.876 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:49:20.876 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:49:20.876 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:20.876 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:49:20.876 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.876 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:49:20.877 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.877 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:49:20.877 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:49:20.877 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:49:20.877 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:49:20.877 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.877 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:49:20.877 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:49:20.877 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:20.877 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.878 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.878 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:49:20.878 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:49:20.878 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:20.878 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:20.878 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:20.878 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.878 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:49:20.878 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:49:20.878 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:49:20.878 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:49:20.879 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:49:20.879 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.879 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.879 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:49:20.879 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:49:20.879 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:49:20.879 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:49:20.879 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.879 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:49:20.879 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:49:20.879 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:20.879 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.880 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.880 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:49:20.880 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:49:20.880 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:20.880 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:20.880 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:20.880 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.881 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:49:20.881 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:49:20.881 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:49:20.881 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:49:20.881 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:49:20.881 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.881 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.881 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:49:20.881 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:49:20.881 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:49:20.881 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:49:20.881 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.882 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:49:20.882 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:49:20.882 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:20.882 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.882 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.882 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:49:20.882 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:49:20.882 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:20.882 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:20.882 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:20.882 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.883 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:49:20.883 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:49:20.883 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:49:20.883 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:49:20.883 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:49:20.883 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:20.883 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:20.883 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.883 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:49:20.883 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:49:20.884 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.884 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:49:20.884 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:49:20.884 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:49:20.884 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:20.884 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:20.884 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:49:20.884 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.884 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:49:20.885 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:49:20.885 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:49:20.885 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:20.885 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:20.885 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:49:20.885 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.885 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:49:20.885 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:49:20.885 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:49:20.885 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:20.885 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:20.885 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:49:20.886 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:20.886 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:49:20.886 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:49:20.886 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:49:20.886 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:20.886 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:20.886 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:20.886 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:49:20.886 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:49:20.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.848+0000 7fb4afe47640 1 -- 172.21.15.12:0/4155348080 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb48c03e260 msgr2=0x7fb48c040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:20.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.848+0000 7fb4afe47640 1 --2- 172.21.15.12:0/4155348080 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb48c03e260 0x7fb48c040720 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8137e20 tx=0x7fb49802e000 comp rx=0 tx=0).stop 2024-09-11T01:49:20.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.848+0000 7fb4afe47640 10 monclient: shutdown 2024-09-11T01:49:20.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.848+0000 7fb4afe47640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:20.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.848+0000 7fb4afe47640 1 -- 172.21.15.12:0/4155348080 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4a813d380 msgr2=0x7fb4a80a19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:20.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.848+0000 7fb4afe47640 1 --2- 172.21.15.12:0/4155348080 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4a813d380 0x7fb4a80a19b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb4a000ce20 tx=0x7fb4a000a710 comp rx=0 tx=0).stop 2024-09-11T01:49:20.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.848+0000 7fb4ae3e4640 1 -- 172.21.15.12:0/4155348080 reap_dead start 2024-09-11T01:49:20.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.848+0000 7fb4afe47640 1 -- 172.21.15.12:0/4155348080 shutdown_connections 2024-09-11T01:49:20.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.848+0000 7fb4afe47640 1 --2- 172.21.15.12:0/4155348080 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4a813d380 0x7fb4a80a19b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:20.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.848+0000 7fb4afe47640 1 -- 172.21.15.12:0/4155348080 >> 172.21.15.12:0/4155348080 conn(0x7fb4a80a7020 msgr2=0x7fb4a813c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:20.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.848+0000 7fb4afe47640 1 -- 172.21.15.12:0/4155348080 shutdown_connections 2024-09-11T01:49:20.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:20.848+0000 7fb4afe47640 1 -- 172.21.15.12:0/4155348080 wait complete. 2024-09-11T01:49:20.889 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:49:20.889 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-11T01:49:20.889 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T01:49:20.889 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-11T01:49:20.890 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-11T01:49:20.890 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-11T01:49:20.890 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:49:20.907 INFO:tasks.ceph.mon.b:Stopped 2024-09-11T01:49:20.907 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:49:20.908 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:49:21.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.005+0000 7f58863e8640 10 monclient: get_monmap_and_config 2024-09-11T01:49:21.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.005+0000 7f58863e8640 10 monclient: build_initial_monmap 2024-09-11T01:49:21.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.005+0000 7f58863e8640 10 monclient: monmap: 2024-09-11T01:49:21.008 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:21.008 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:21.008 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:49:21.006542+0000 2024-09-11T01:49:21.008 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:49:21.006542+0000 2024-09-11T01:49:21.008 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:21.008 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:21.008 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:49:21.008 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:49:21.008 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:49:21.008 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:49:21.008 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:49:21.009 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:21.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.006+0000 7f58863e8640 1 Processor -- start 2024-09-11T01:49:21.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.006+0000 7f58863e8640 1 -- start start 2024-09-11T01:49:21.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.006+0000 7f58863e8640 10 monclient: init 2024-09-11T01:49:21.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f58863e8640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:21.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f58863e8640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-11T01:49:21.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f58863e8640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f58800a2e90 0x7f588006dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:21.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f58863e8640 10 monclient(hunting): picked mon.noname-c con 0x7f58800a2e90 addr v2:172.21.15.71:3300/0 2024-09-11T01:49:21.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f58863e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f58800a3270 addr v1:172.21.15.71:6789/0 2024-09-11T01:49:21.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f58863e8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f588013d490 0x7f588013f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:21.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f58863e8640 10 monclient(hunting): picked mon.noname-b con 0x7f588013d490 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:21.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f58863e8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58800a46c0 con 0x7f58800a3270 2024-09-11T01:49:21.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f58863e8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:21.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f58863e8640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f58800a4900 con 0x7f58800a2e90 2024-09-11T01:49:21.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f58863e8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:21.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f58863e8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f588006ee50 con 0x7f588013d490 2024-09-11T01:49:21.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f58863e8640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:21.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f58863e8640 10 monclient(hunting): authenticate will time out at 1967.449263s 2024-09-11T01:49:21.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f587ffff640 1 -- >> v2:172.21.15.71:3300/0 conn(0x7f58800a2e90 msgr2=0x7f588006dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:49:21.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f5877fff640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f58800a3270 legacy=0x7f58800a3680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:49:21.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f5884985640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f588013d490 0x7f588013f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:21.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f587ffff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f58800a2e90 0x7f588006dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:21.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f5884985640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f588013d490 0x7f588013f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42336/0 (socket says 172.21.15.12:42336) 2024-09-11T01:49:21.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f5884985640 1 -- 172.21.15.12:0/3738804811 learned_addr learned my addr 172.21.15.12:0/3738804811 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:49:21.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f5884985640 10 monclient(hunting): get_auth_request con 0x7f588013d490 auth_method 0 2024-09-11T01:49:21.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f5884985640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:21.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f5884985640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:21.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f5884985640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:21.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f5884985640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:21.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f5884985640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:21.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f5884985640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:21.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f5884985640 10 monclient(hunting): handle_auth_done global_id 224135 payload 274 2024-09-11T01:49:21.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f5884985640 1 -- 172.21.15.12:0/3738804811 >> v2:172.21.15.71:3300/0 conn(0x7f58800a2e90 msgr2=0x7f588006dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:49:21.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f5884985640 1 --2- 172.21.15.12:0/3738804811 >> v2:172.21.15.71:3300/0 conn(0x7f58800a2e90 0x7f588006dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:21.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f5884985640 1 -- 172.21.15.12:0/3738804811 >> v1:172.21.15.71:6789/0 conn(0x7f58800a3270 legacy=0x7f58800a3680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:49:21.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f5884985640 10 monclient: _finish_hunting 0 2024-09-11T01:49:21.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f5884985640 1 monclient: found mon.noname-b 2024-09-11T01:49:21.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f5884985640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:21.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f5884985640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:49:21.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.007+0000 7f5884985640 1 -- 172.21.15.12:0/3738804811 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f588006eb70 con 0x7f588013d490 2024-09-11T01:49:21.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.010+0000 7f5884985640 1 -- 172.21.15.12:0/3738804811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f588013d490 msgr2=0x7f588013f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:21.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.010+0000 7f5884985640 1 --2- 172.21.15.12:0/3738804811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f588013d490 0x7f588013f880 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5880071140 tx=0x7f587802a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6185b19b9abbd8e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:21.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.010+0000 7f587f7fe640 1 -- 172.21.15.12:0/3738804811 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5878036070 con 0x7f588013d490 2024-09-11T01:49:21.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.010+0000 7f587f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:21.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.010+0000 7f587f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:49:21.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.010+0000 7f587f7fe640 10 monclient: dump: 2024-09-11T01:49:21.020 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:21.020 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:21.020 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:21.021 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:21.021 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:21.021 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:21.021 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:21.021 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:21.021 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:21.022 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:21.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.010+0000 7f587f7fe640 10 monclient: _finish_auth 0 2024-09-11T01:49:21.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.010+0000 7f587f7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:49:21.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.010+0000 7f587f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:21.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.010+0000 7f587f7fe640 1 -- 172.21.15.12:0/3738804811 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5878007800 con 0x7f588013d490 2024-09-11T01:49:21.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.010+0000 7f58863e8640 5 monclient: authenticate success, global_id 224135 2024-09-11T01:49:21.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.010+0000 7f58863e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:49:21.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.010+0000 7f587f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:21.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.010+0000 7f587f7fe640 1 -- 172.21.15.12:0/3738804811 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5878007bd0 con 0x7f588013d490 2024-09-11T01:49:21.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.010+0000 7f587f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:21.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.010+0000 7f587f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:49:21.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.010+0000 7f587f7fe640 10 monclient: dump: 2024-09-11T01:49:21.023 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:21.023 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:21.023 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:21.023 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:21.023 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:21.023 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:21.023 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:21.023 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:21.023 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:21.023 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:21.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.010+0000 7f58863e8640 10 monclient: get_monmap_and_config success 2024-09-11T01:49:21.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.010+0000 7f58863e8640 4 set_mon_vals no callback set 2024-09-11T01:49:21.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.011+0000 7f58863e8640 10 monclient: shutdown 2024-09-11T01:49:21.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.011+0000 7f58863e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:21.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.011+0000 7f58863e8640 1 -- 172.21.15.12:0/3738804811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f588013d490 msgr2=0x7f588013f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:21.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.011+0000 7f58863e8640 1 --2- 172.21.15.12:0/3738804811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f588013d490 0x7f588013f880 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5880071140 tx=0x7f587802a560 comp rx=0 tx=0).stop 2024-09-11T01:49:21.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.011+0000 7f58863e8640 1 -- 172.21.15.12:0/3738804811 shutdown_connections 2024-09-11T01:49:21.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.011+0000 7f58863e8640 1 --2- 172.21.15.12:0/3738804811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f588013d490 0x7f588013f880 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:21.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.011+0000 7f58863e8640 1 --2- 172.21.15.12:0/3738804811 >> v2:172.21.15.71:3300/0 conn(0x7f58800a2e90 0x7f588006dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:21.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.011+0000 7f58863e8640 1 -- 172.21.15.12:0/3738804811 >> 172.21.15.12:0/3738804811 conn(0x7f58800a8490 msgr2=0x7f58800a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:21.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.011+0000 7f58863e8640 1 -- 172.21.15.12:0/3738804811 shutdown_connections 2024-09-11T01:49:21.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.011+0000 7f58863e8640 1 -- 172.21.15.12:0/3738804811 wait complete. 2024-09-11T01:49:21.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.011+0000 7f58863e8640 10 monclient: build_initial_monmap 2024-09-11T01:49:21.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.011+0000 7f58863e8640 10 monclient: monmap: 2024-09-11T01:49:21.025 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:21.025 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:49:21.025 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:49:21.025 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:49:21.025 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:21.025 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:21.025 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:49:21.025 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:49:21.025 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:49:21.025 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:21.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.012+0000 7f58863e8640 1 Processor -- start 2024-09-11T01:49:21.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.012+0000 7f58863e8640 1 -- start start 2024-09-11T01:49:21.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.012+0000 7f58863e8640 10 monclient: init 2024-09-11T01:49:21.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.012+0000 7f58863e8640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:21.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.012+0000 7f58863e8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:49:21.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.012+0000 7f58863e8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58800a2e90 0x7f5880137e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:21.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.012+0000 7f58863e8640 10 monclient(hunting): picked mon.noname-a con 0x7f58800a2e90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:49:21.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.012+0000 7f58863e8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f58800a3270 0x7f5880138360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:21.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.012+0000 7f58863e8640 10 monclient(hunting): picked mon.noname-b con 0x7f58800a3270 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:49:21.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.012+0000 7f58863e8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f588013d490 0x7f58801cce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:21.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.012+0000 7f58863e8640 10 monclient(hunting): picked mon.noname-c con 0x7f588013d490 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:21.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.012+0000 7f58863e8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.012+0000 7f58863e8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f588006fd20 con 0x7f58800a2e90 2024-09-11T01:49:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.012+0000 7f58863e8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.012+0000 7f58863e8640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f588006fba0 con 0x7f58800a3270 2024-09-11T01:49:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.012+0000 7f58863e8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.012+0000 7f58863e8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f588013fdc0 con 0x7f588013d490 2024-09-11T01:49:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.012+0000 7f58863e8640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.012+0000 7f58863e8640 10 monclient(hunting): authenticate will time out at 1967.454900s 2024-09-11T01:49:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.012+0000 7f5884985640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f588013d490 0x7f58801cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.012+0000 7f587ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58800a2e90 0x7f5880137e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f5877fff640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f58800a3270 msgr2=0x7f5880138360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:49:21.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f5884985640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f588013d490 0x7f58801cce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42342/0 (socket says 172.21.15.12:42342) 2024-09-11T01:49:21.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f5884985640 1 -- 172.21.15.12:0/1659428141 learned_addr learned my addr 172.21.15.12:0/1659428141 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:49:21.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f5884985640 10 monclient(hunting): get_auth_request con 0x7f588013d490 auth_method 0 2024-09-11T01:49:21.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f587ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58800a2e90 0x7f5880137e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43880/0 (socket says 172.21.15.12:43880) 2024-09-11T01:49:21.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f5884985640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:21.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f5884985640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:21.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f5884985640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:21.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f5877fff640 1 --2- 172.21.15.12:0/1659428141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f58800a3270 0x7f5880138360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:21.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f587ffff640 10 monclient(hunting): get_auth_request con 0x7f58800a2e90 auth_method 0 2024-09-11T01:49:21.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f587ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:21.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f587ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:21.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f587ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:21.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f5884985640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:21.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f5884985640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:21.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f5884985640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:21.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f587ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:21.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f587ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:21.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f587ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:21.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f5884985640 10 monclient(hunting): handle_auth_done global_id 224138 payload 824 2024-09-11T01:49:21.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f5884985640 1 -- 172.21.15.12:0/1659428141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f58800a3270 msgr2=0x7f5880138360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:49:21.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f5884985640 1 --2- 172.21.15.12:0/1659428141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f58800a3270 0x7f5880138360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:21.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f5884985640 1 -- 172.21.15.12:0/1659428141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58800a2e90 msgr2=0x7f5880137e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:21.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f5884985640 1 --2- 172.21.15.12:0/1659428141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58800a2e90 0x7f5880137e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:21.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f5884985640 10 monclient: _finish_hunting 0 2024-09-11T01:49:21.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f5884985640 1 monclient: found mon.noname-c 2024-09-11T01:49:21.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f5884985640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:21.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f5884985640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:49:21.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f5884985640 1 -- 172.21.15.12:0/1659428141 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58800a2510 con 0x7f588013d490 2024-09-11T01:49:21.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.013+0000 7f587ffff640 1 --2- 172.21.15.12:0/1659428141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58800a2e90 0x7f5880137e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:49:21.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.015+0000 7f5884985640 1 -- 172.21.15.12:0/1659428141 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f588013d490 msgr2=0x7f58801cce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:21.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.015+0000 7f5884985640 1 --2- 172.21.15.12:0/1659428141 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f588013d490 0x7f58801cce60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5878009f40 tx=0x7f5878004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:21.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.015+0000 7f587d7fa640 1 -- 172.21.15.12:0/1659428141 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5878041030 con 0x7f588013d490 2024-09-11T01:49:21.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.015+0000 7f587d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:21.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.015+0000 7f587d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:49:21.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.015+0000 7f587d7fa640 10 monclient: dump: 2024-09-11T01:49:21.031 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:21.031 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:21.031 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:21.032 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:21.032 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:21.032 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:21.032 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:21.032 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:21.032 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:21.032 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:21.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.015+0000 7f587d7fa640 10 monclient: _finish_auth 0 2024-09-11T01:49:21.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.015+0000 7f587d7fa640 10 monclient: _check_auth_tickets 2024-09-11T01:49:21.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.015+0000 7f587d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:21.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.015+0000 7f587d7fa640 1 -- 172.21.15.12:0/1659428141 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5878037040 con 0x7f588013d490 2024-09-11T01:49:21.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.015+0000 7f587d7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:21.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.015+0000 7f587d7fa640 1 -- 172.21.15.12:0/1659428141 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5878036070 con 0x7f588013d490 2024-09-11T01:49:21.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.015+0000 7f587d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:21.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.015+0000 7f587d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:49:21.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.015+0000 7f587d7fa640 10 monclient: dump: 2024-09-11T01:49:21.033 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:21.033 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:21.033 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:21.033 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:21.033 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:21.034 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:21.034 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:21.034 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:21.034 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:21.034 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:21.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.015+0000 7f58863e8640 5 monclient: authenticate success, global_id 224138 2024-09-11T01:49:21.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.015+0000 7f58863e8640 10 monclient: _renew_subs 2024-09-11T01:49:21.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.015+0000 7f58863e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:49:21.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.015+0000 7f58863e8640 1 -- 172.21.15.12:0/1659428141 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f588009f1e0 con 0x7f588013d490 2024-09-11T01:49:21.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.015+0000 7f587effd640 4 set_mon_vals no callback set 2024-09-11T01:49:21.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.015+0000 7f58863e8640 10 monclient: _renew_subs 2024-09-11T01:49:21.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.015+0000 7f58863e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:49:21.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.015+0000 7f58863e8640 1 -- 172.21.15.12:0/1659428141 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f588009f680 con 0x7f588013d490 2024-09-11T01:49:21.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.016+0000 7f587d7fa640 1 -- 172.21.15.12:0/1659428141 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f587804e060 con 0x7f588013d490 2024-09-11T01:49:21.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.016+0000 7f58863e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:21.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.016+0000 7f58863e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:21.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.016+0000 7f58863e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:49:21.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.016+0000 7f58863e8640 1 -- 172.21.15.12:0/1659428141 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f588006dc60 con 0x7f588013d490 2024-09-11T01:49:21.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.016+0000 7f587d7fa640 1 --2- 172.21.15.12:0/1659428141 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f586003e210 0x7f58600406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:21.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.017+0000 7f587d7fa640 1 -- 172.21.15.12:0/1659428141 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(678..678 src has 507..678) ==== 4847+0+0 (secure 0 0 0) 0x7f5878073a50 con 0x7f588013d490 2024-09-11T01:49:21.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.017+0000 7f587ffff640 1 --2- 172.21.15.12:0/1659428141 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f586003e210 0x7f58600406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:21.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.017+0000 7f587ffff640 10 monclient: get_auth_request con 0x7f586003e210 auth_method 0 2024-09-11T01:49:21.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.019+0000 7f587ffff640 1 --2- 172.21.15.12:0/1659428141 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f586003e210 0x7f58600406d0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f5870006e70 tx=0x7f5870006dc0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:21.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.021+0000 7f587d7fa640 1 -- 172.21.15.12:0/1659428141 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f587803d480 con 0x7f588013d490 2024-09-11T01:49:21.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.021+0000 7f587d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:21.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.021+0000 7f587d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:49:21.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.282+0000 7f58863e8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:49:21.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.282+0000 7f58863e8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:49:21.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.282+0000 7f58863e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:49:21.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.282+0000 7f58863e8640 1 -- 172.21.15.12:0/1659428141 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f58800698e0 con 0x7f588013d490 2024-09-11T01:49:21.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.282+0000 7f587d7fa640 1 -- 172.21.15.12:0/1659428141 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f5878072410 con 0x7f588013d490 2024-09-11T01:49:21.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.282+0000 7f587d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:49:21.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.282+0000 7f587d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:49:21.286 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":188,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:49:21.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.285+0000 7f58863e8640 1 -- 172.21.15.12:0/1659428141 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f586003e210 msgr2=0x7f58600406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:21.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.285+0000 7f58863e8640 1 --2- 172.21.15.12:0/1659428141 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f586003e210 0x7f58600406d0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f5870006e70 tx=0x7f5870006dc0 comp rx=0 tx=0).stop 2024-09-11T01:49:21.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.285+0000 7f58863e8640 10 monclient: shutdown 2024-09-11T01:49:21.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.285+0000 7f58863e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:21.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.285+0000 7f58863e8640 1 -- 172.21.15.12:0/1659428141 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f588013d490 msgr2=0x7f58801cce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:21.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.285+0000 7f58863e8640 1 --2- 172.21.15.12:0/1659428141 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f588013d490 0x7f58801cce60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5878009f40 tx=0x7f5878004a40 comp rx=0 tx=0).stop 2024-09-11T01:49:21.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.285+0000 7f58863e8640 1 -- 172.21.15.12:0/1659428141 shutdown_connections 2024-09-11T01:49:21.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.285+0000 7f58863e8640 1 --2- 172.21.15.12:0/1659428141 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f588013d490 0x7f58801cce60 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:21.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.285+0000 7f58863e8640 1 --2- 172.21.15.12:0/1659428141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f58800a3270 0x7f5880138360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:21.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.285+0000 7f58863e8640 1 --2- 172.21.15.12:0/1659428141 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f586003e210 0x7f58600406d0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:21.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.285+0000 7f58863e8640 1 --2- 172.21.15.12:0/1659428141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58800a2e90 0x7f5880137e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:21.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.285+0000 7f58863e8640 1 -- 172.21.15.12:0/1659428141 >> 172.21.15.12:0/1659428141 conn(0x7f58800a8490 msgr2=0x7f588013c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:21.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.285+0000 7f58863e8640 1 -- 172.21.15.12:0/1659428141 shutdown_connections 2024-09-11T01:49:21.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:21.285+0000 7f58863e8640 1 -- 172.21.15.12:0/1659428141 wait complete. 2024-09-11T01:49:24.297 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:49:24.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.409+0000 7fcbede71640 10 monclient: get_monmap_and_config 2024-09-11T01:49:24.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.409+0000 7fcbede71640 10 monclient: build_initial_monmap 2024-09-11T01:49:24.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.409+0000 7fcbede71640 10 monclient: monmap: 2024-09-11T01:49:24.411 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:24.411 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:24.411 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:49:24.410133+0000 2024-09-11T01:49:24.411 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:49:24.410133+0000 2024-09-11T01:49:24.411 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:24.411 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:24.411 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:49:24.411 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:49:24.412 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:49:24.412 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:49:24.412 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:49:24.412 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:24.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.410+0000 7fcbede71640 1 Processor -- start 2024-09-11T01:49:24.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.410+0000 7fcbede71640 1 -- start start 2024-09-11T01:49:24.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.410+0000 7fcbede71640 10 monclient: init 2024-09-11T01:49:24.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbede71640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:24.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbede71640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-11T01:49:24.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbede71640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcbe8138230 0x7fcbe8138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:24.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbede71640 10 monclient(hunting): picked mon.noname-b con 0x7fcbe8138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:24.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbede71640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcbe809d030 addr v1:172.21.15.12:6789/0 2024-09-11T01:49:24.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbede71640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcbe809d430 0x7fcbe80a5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:24.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbede71640 10 monclient(hunting): picked mon.noname-c con 0x7fcbe809d430 addr v2:172.21.15.71:3300/0 2024-09-11T01:49:24.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbede71640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbe80a6e30 con 0x7fcbe809d030 2024-09-11T01:49:24.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbede71640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:24.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbede71640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fcbe809c9c0 con 0x7fcbe809d430 2024-09-11T01:49:24.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbede71640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:24.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbede71640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcbe809ba20 con 0x7fcbe8138230 2024-09-11T01:49:24.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbede71640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:24.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbede71640 10 monclient(hunting): authenticate will time out at 1970.853206s 2024-09-11T01:49:24.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbe77fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcbe8138230 0x7fcbe8138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:24.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbdffff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fcbe809d030 0x7fcbe80a5020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59920/0 (socket says 172.21.15.12:59920) 2024-09-11T01:49:24.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbdffff640 1 -- 172.21.15.12:0/2068135021 learned_addr learned my addr 172.21.15.12:0/2068135021 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:49:24.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbe7fff640 1 -- 172.21.15.12:0/2068135021 >> v2:172.21.15.71:3300/0 conn(0x7fcbe809d430 msgr2=0x7fcbe80a5730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:49:24.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbe77fe640 10 monclient(hunting): get_auth_request con 0x7fcbe8138230 auth_method 0 2024-09-11T01:49:24.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbe77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:24.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbe77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:24.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbe77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:24.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbe7fff640 1 --2- 172.21.15.12:0/2068135021 >> v2:172.21.15.71:3300/0 conn(0x7fcbe809d430 0x7fcbe80a5730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:24.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbe77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:24.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbe77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:24.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbe77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:24.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbe77fe640 10 monclient(hunting): handle_auth_done global_id 224141 payload 274 2024-09-11T01:49:24.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbe77fe640 1 -- 172.21.15.12:0/2068135021 >> v2:172.21.15.71:3300/0 conn(0x7fcbe809d430 msgr2=0x7fcbe80a5730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:49:24.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbe77fe640 1 --2- 172.21.15.12:0/2068135021 >> v2:172.21.15.71:3300/0 conn(0x7fcbe809d430 0x7fcbe80a5730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:24.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbe77fe640 1 -- 172.21.15.12:0/2068135021 >> v1:172.21.15.12:6789/0 conn(0x7fcbe809d030 legacy=0x7fcbe80a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:49:24.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbe77fe640 10 monclient: _finish_hunting 0 2024-09-11T01:49:24.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbe77fe640 1 monclient: found mon.noname-b 2024-09-11T01:49:24.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbe77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:24.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbe77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:49:24.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.411+0000 7fcbe77fe640 1 -- 172.21.15.12:0/2068135021 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe80a66a0 con 0x7fcbe8138230 2024-09-11T01:49:24.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.414+0000 7fcbe77fe640 1 -- 172.21.15.12:0/2068135021 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcbe8138230 msgr2=0x7fcbe8138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:24.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.414+0000 7fcbe77fe640 1 --2- 172.21.15.12:0/2068135021 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcbe8138230 0x7fcbe8138610 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcbe809cf80 tx=0x7fcbd8029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15c7787ade501c8c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:24.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.414+0000 7fcbe6ffd640 1 -- 172.21.15.12:0/2068135021 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd8037070 con 0x7fcbe8138230 2024-09-11T01:49:24.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.414+0000 7fcbe6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:24.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.414+0000 7fcbe6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:49:24.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.414+0000 7fcbe6ffd640 10 monclient: dump: 2024-09-11T01:49:24.423 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:24.423 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:24.423 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:24.423 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:24.423 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:24.423 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:24.423 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:24.423 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:24.423 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:24.423 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:24.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.414+0000 7fcbe6ffd640 10 monclient: _finish_auth 0 2024-09-11T01:49:24.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.414+0000 7fcbe6ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:49:24.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.414+0000 7fcbe6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:24.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.414+0000 7fcbe6ffd640 1 -- 172.21.15.12:0/2068135021 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbd8032440 con 0x7fcbe8138230 2024-09-11T01:49:24.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.414+0000 7fcbe6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:24.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.414+0000 7fcbe6ffd640 1 -- 172.21.15.12:0/2068135021 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd8032810 con 0x7fcbe8138230 2024-09-11T01:49:24.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.414+0000 7fcbede71640 5 monclient: authenticate success, global_id 224141 2024-09-11T01:49:24.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.414+0000 7fcbede71640 10 monclient: get_monmap_and_config success 2024-09-11T01:49:24.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.414+0000 7fcbede71640 4 set_mon_vals no callback set 2024-09-11T01:49:24.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.414+0000 7fcbe6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:24.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.414+0000 7fcbe6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:49:24.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.414+0000 7fcbe6ffd640 10 monclient: dump: 2024-09-11T01:49:24.424 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:24.424 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:24.424 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:24.424 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:24.425 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.415+0000 7fcbede71640 10 monclient: shutdown 2024-09-11T01:49:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.415+0000 7fcbede71640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.415+0000 7fcbede71640 1 -- 172.21.15.12:0/2068135021 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcbe8138230 msgr2=0x7fcbe8138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.415+0000 7fcbede71640 1 --2- 172.21.15.12:0/2068135021 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcbe8138230 0x7fcbe8138610 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcbe809cf80 tx=0x7fcbd8029fb0 comp rx=0 tx=0).stop 2024-09-11T01:49:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.415+0000 7fcbede71640 1 -- 172.21.15.12:0/2068135021 shutdown_connections 2024-09-11T01:49:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.415+0000 7fcbede71640 1 --2- 172.21.15.12:0/2068135021 >> v2:172.21.15.71:3300/0 conn(0x7fcbe809d430 0x7fcbe80a5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:24.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.415+0000 7fcbede71640 1 --2- 172.21.15.12:0/2068135021 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcbe8138230 0x7fcbe8138610 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.415+0000 7fcbede71640 1 -- 172.21.15.12:0/2068135021 >> 172.21.15.12:0/2068135021 conn(0x7fcbe80a7120 msgr2=0x7fcbe80a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.415+0000 7fcbede71640 1 -- 172.21.15.12:0/2068135021 shutdown_connections 2024-09-11T01:49:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.415+0000 7fcbede71640 1 -- 172.21.15.12:0/2068135021 wait complete. 2024-09-11T01:49:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.415+0000 7fcbede71640 10 monclient: build_initial_monmap 2024-09-11T01:49:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.415+0000 7fcbede71640 10 monclient: monmap: 2024-09-11T01:49:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:49:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:49:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:49:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:49:24.426 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:49:24.427 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:49:24.427 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:24.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.416+0000 7fcbede71640 1 Processor -- start 2024-09-11T01:49:24.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.416+0000 7fcbede71640 1 -- start start 2024-09-11T01:49:24.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.416+0000 7fcbede71640 10 monclient: init 2024-09-11T01:49:24.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.416+0000 7fcbede71640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:24.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.416+0000 7fcbede71640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:49:24.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.416+0000 7fcbede71640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbe809d030 0x7fcbe81cd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:24.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.416+0000 7fcbede71640 10 monclient(hunting): picked mon.noname-a con 0x7fcbe809d030 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:49:24.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.416+0000 7fcbede71640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcbe809d430 0x7fcbe813f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:24.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.416+0000 7fcbede71640 10 monclient(hunting): picked mon.noname-c con 0x7fcbe809d430 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:24.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.416+0000 7fcbede71640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbe8138230 0x7fcbe813f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:24.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.416+0000 7fcbede71640 10 monclient(hunting): picked mon.noname-b con 0x7fcbe8138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:49:24.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.416+0000 7fcbede71640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:24.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.416+0000 7fcbede71640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe806f970 con 0x7fcbe809d030 2024-09-11T01:49:24.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.416+0000 7fcbede71640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:24.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.416+0000 7fcbede71640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe806f7f0 con 0x7fcbe8138230 2024-09-11T01:49:24.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.416+0000 7fcbede71640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:24.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.416+0000 7fcbede71640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcbe80a5c70 con 0x7fcbe809d430 2024-09-11T01:49:24.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.416+0000 7fcbede71640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:24.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.416+0000 7fcbede71640 10 monclient(hunting): authenticate will time out at 1970.858892s 2024-09-11T01:49:24.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.416+0000 7fcbe77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbe809d030 0x7fcbe81cd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:24.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.416+0000 7fcbe77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbe809d030 0x7fcbe81cd2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43896/0 (socket says 172.21.15.12:43896) 2024-09-11T01:49:24.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbe77fe640 1 -- 172.21.15.12:0/2951342972 learned_addr learned my addr 172.21.15.12:0/2951342972 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:49:24.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbe77fe640 10 monclient(hunting): get_auth_request con 0x7fcbe809d030 auth_method 0 2024-09-11T01:49:24.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.416+0000 7fcbdffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcbe809d430 0x7fcbe813f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:24.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbe77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:24.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbe77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:24.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbe77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:24.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbdffff640 10 monclient(hunting): get_auth_request con 0x7fcbe809d430 auth_method 0 2024-09-11T01:49:24.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbdffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:24.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbdffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:24.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbdffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:24.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbe7fff640 1 -- 172.21.15.12:0/2951342972 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbe8138230 msgr2=0x7fcbe813f970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:49:24.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbe7fff640 1 --2- 172.21.15.12:0/2951342972 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbe8138230 0x7fcbe813f970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:24.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbe77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:24.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbe77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:24.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbe77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:24.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbdffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:24.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbdffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:24.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbdffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:24.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbdffff640 10 monclient(hunting): handle_auth_done global_id 224144 payload 824 2024-09-11T01:49:24.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbdffff640 1 -- 172.21.15.12:0/2951342972 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbe8138230 msgr2=0x7fcbe813f970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:49:24.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbdffff640 1 --2- 172.21.15.12:0/2951342972 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbe8138230 0x7fcbe813f970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:24.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbdffff640 1 -- 172.21.15.12:0/2951342972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbe809d030 msgr2=0x7fcbe81cd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:24.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbdffff640 1 --2- 172.21.15.12:0/2951342972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbe809d030 0x7fcbe81cd2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:24.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbdffff640 10 monclient: _finish_hunting 0 2024-09-11T01:49:24.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbdffff640 1 monclient: found mon.noname-c 2024-09-11T01:49:24.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:24.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbdffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:49:24.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbdffff640 1 -- 172.21.15.12:0/2951342972 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe813bba0 con 0x7fcbe809d430 2024-09-11T01:49:24.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.417+0000 7fcbe77fe640 1 --2- 172.21.15.12:0/2951342972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbe809d030 0x7fcbe81cd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:49:24.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.419+0000 7fcbdffff640 1 -- 172.21.15.12:0/2951342972 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcbe809d430 msgr2=0x7fcbe813f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:24.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.419+0000 7fcbdffff640 1 --2- 172.21.15.12:0/2951342972 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcbe809d430 0x7fcbe813f430 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcbd000c3e0 tx=0x7fcbd000c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:24.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.419+0000 7fcbe4ff9640 1 -- 172.21.15.12:0/2951342972 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd0017070 con 0x7fcbe809d430 2024-09-11T01:49:24.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.419+0000 7fcbe4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:24.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.419+0000 7fcbe4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:49:24.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.419+0000 7fcbe4ff9640 10 monclient: dump: 2024-09-11T01:49:24.431 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:24.431 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:24.431 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:24.431 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:24.431 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:24.431 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:24.431 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:24.432 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:24.432 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:24.432 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:24.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.419+0000 7fcbe4ff9640 10 monclient: _finish_auth 0 2024-09-11T01:49:24.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.419+0000 7fcbe4ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:49:24.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.419+0000 7fcbe4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:24.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.419+0000 7fcbe4ff9640 1 -- 172.21.15.12:0/2951342972 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbd00070d0 con 0x7fcbe809d430 2024-09-11T01:49:24.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.419+0000 7fcbe4ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:24.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.419+0000 7fcbe4ff9640 1 -- 172.21.15.12:0/2951342972 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd00074c0 con 0x7fcbe809d430 2024-09-11T01:49:24.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.419+0000 7fcbede71640 5 monclient: authenticate success, global_id 224144 2024-09-11T01:49:24.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.419+0000 7fcbede71640 10 monclient: _renew_subs 2024-09-11T01:49:24.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.419+0000 7fcbede71640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:49:24.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.419+0000 7fcbede71640 1 -- 172.21.15.12:0/2951342972 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbe813d660 con 0x7fcbe809d430 2024-09-11T01:49:24.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.419+0000 7fcbe4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:24.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.419+0000 7fcbe4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:49:24.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.419+0000 7fcbe4ff9640 10 monclient: dump: 2024-09-11T01:49:24.433 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:24.433 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:24.433 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:24.433 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:24.433 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:24.433 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:24.433 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:24.433 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:24.433 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:24.434 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:24.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.419+0000 7fcbe5ffb640 4 set_mon_vals no callback set 2024-09-11T01:49:24.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.419+0000 7fcbede71640 10 monclient: _renew_subs 2024-09-11T01:49:24.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.419+0000 7fcbede71640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:49:24.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.419+0000 7fcbede71640 1 -- 172.21.15.12:0/2951342972 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbe80a1810 con 0x7fcbe809d430 2024-09-11T01:49:24.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.420+0000 7fcbede71640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:24.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.420+0000 7fcbede71640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:24.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.420+0000 7fcbede71640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:49:24.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.420+0000 7fcbede71640 1 -- 172.21.15.12:0/2951342972 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbe8138610 con 0x7fcbe809d430 2024-09-11T01:49:24.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.420+0000 7fcbe4ff9640 1 -- 172.21.15.12:0/2951342972 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fcbd001b3b0 con 0x7fcbe809d430 2024-09-11T01:49:24.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.421+0000 7fcbe4ff9640 1 --2- 172.21.15.12:0/2951342972 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcbc803e210 0x7fcbc80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:24.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.421+0000 7fcbe4ff9640 1 -- 172.21.15.12:0/2951342972 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(678..678 src has 507..678) ==== 4847+0+0 (secure 0 0 0) 0x7fcbd00524e0 con 0x7fcbe809d430 2024-09-11T01:49:24.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.421+0000 7fcbe77fe640 1 --2- 172.21.15.12:0/2951342972 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcbc803e210 0x7fcbc80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:24.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.421+0000 7fcbe77fe640 10 monclient: get_auth_request con 0x7fcbc803e210 auth_method 0 2024-09-11T01:49:24.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.424+0000 7fcbe77fe640 1 --2- 172.21.15.12:0/2951342972 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcbc803e210 0x7fcbc80406d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8009560 tx=0x7fcbd802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:24.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.424+0000 7fcbe4ff9640 1 -- 172.21.15.12:0/2951342972 DispatchQueue::entry inject delay of 0.002000 2024-09-11T01:49:24.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.426+0000 7fcbe4ff9640 1 -- 172.21.15.12:0/2951342972 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fcbd001a360 con 0x7fcbe809d430 2024-09-11T01:49:24.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.426+0000 7fcbe4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:24.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.426+0000 7fcbe4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:49:24.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.688+0000 7fcbede71640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:49:24.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.688+0000 7fcbede71640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:49:24.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.688+0000 7fcbede71640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:49:24.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.688+0000 7fcbede71640 1 -- 172.21.15.12:0/2951342972 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcbe80a4750 con 0x7fcbe809d430 2024-09-11T01:49:24.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.688+0000 7fcbe4ff9640 1 -- 172.21.15.12:0/2951342972 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7fcbd000ec90 con 0x7fcbe809d430 2024-09-11T01:49:24.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.688+0000 7fcbe4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:49:24.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.688+0000 7fcbe4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:49:24.690 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":188,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:49:24.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.690+0000 7fcbede71640 1 -- 172.21.15.12:0/2951342972 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcbc803e210 msgr2=0x7fcbc80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:24.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.690+0000 7fcbede71640 1 --2- 172.21.15.12:0/2951342972 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcbc803e210 0x7fcbc80406d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8009560 tx=0x7fcbd802e000 comp rx=0 tx=0).stop 2024-09-11T01:49:24.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.690+0000 7fcbede71640 10 monclient: shutdown 2024-09-11T01:49:24.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.690+0000 7fcbede71640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:24.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.690+0000 7fcbede71640 1 -- 172.21.15.12:0/2951342972 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcbe809d430 msgr2=0x7fcbe813f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:24.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.690+0000 7fcbede71640 1 --2- 172.21.15.12:0/2951342972 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcbe809d430 0x7fcbe813f430 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcbd000c3e0 tx=0x7fcbd000c8a0 comp rx=0 tx=0).stop 2024-09-11T01:49:24.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.690+0000 7fcbede71640 1 -- 172.21.15.12:0/2951342972 shutdown_connections 2024-09-11T01:49:24.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.691+0000 7fcbede71640 1 --2- 172.21.15.12:0/2951342972 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbe8138230 0x7fcbe813f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:24.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.691+0000 7fcbede71640 1 --2- 172.21.15.12:0/2951342972 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcbe809d430 0x7fcbe813f430 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:24.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.691+0000 7fcbede71640 1 --2- 172.21.15.12:0/2951342972 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcbc803e210 0x7fcbc80406d0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:24.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.691+0000 7fcbede71640 1 --2- 172.21.15.12:0/2951342972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcbe809d030 0x7fcbe81cd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:24.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.691+0000 7fcbede71640 1 -- 172.21.15.12:0/2951342972 >> 172.21.15.12:0/2951342972 conn(0x7fcbe80a7120 msgr2=0x7fcbe80a3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:24.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.691+0000 7fcbede71640 1 -- 172.21.15.12:0/2951342972 shutdown_connections 2024-09-11T01:49:24.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:24.691+0000 7fcbede71640 1 -- 172.21.15.12:0/2951342972 wait complete. 2024-09-11T01:49:27.703 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:49:27.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.814+0000 7f8cf62ef640 10 monclient: get_monmap_and_config 2024-09-11T01:49:27.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.814+0000 7f8cf62ef640 10 monclient: build_initial_monmap 2024-09-11T01:49:27.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.814+0000 7f8cf62ef640 10 monclient: monmap: 2024-09-11T01:49:27.816 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:27.817 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:27.817 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:49:27.815625+0000 2024-09-11T01:49:27.817 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:49:27.815625+0000 2024-09-11T01:49:27.817 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:27.817 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:27.817 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:49:27.817 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:49:27.817 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:49:27.817 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:49:27.817 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:49:27.817 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:27.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.815+0000 7f8cf62ef640 1 Processor -- start 2024-09-11T01:49:27.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.815+0000 7f8cf62ef640 1 -- start start 2024-09-11T01:49:27.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.815+0000 7f8cf62ef640 10 monclient: init 2024-09-11T01:49:27.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf62ef640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:27.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf62ef640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-11T01:49:27.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf62ef640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f8cf00a2e90 0x7f8cf006dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:27.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf62ef640 10 monclient(hunting): picked mon.noname-c con 0x7f8cf00a2e90 addr v2:172.21.15.71:3300/0 2024-09-11T01:49:27.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf62ef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8cf00a3270 addr v1:172.21.15.71:6789/0 2024-09-11T01:49:27.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf62ef640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8cf013d490 0x7f8cf013f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:27.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf62ef640 10 monclient(hunting): picked mon.noname-a con 0x7f8cf013d490 addr v2:172.21.15.12:3300/0 2024-09-11T01:49:27.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf62ef640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cf00a46c0 con 0x7f8cf00a3270 2024-09-11T01:49:27.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf62ef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:27.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf62ef640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f8cf00a4900 con 0x7f8cf013d490 2024-09-11T01:49:27.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf62ef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:27.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf62ef640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f8cf006ee50 con 0x7f8cf00a2e90 2024-09-11T01:49:27.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf62ef640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:27.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf62ef640 10 monclient(hunting): authenticate will time out at 1974.258325s 2024-09-11T01:49:27.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8ceffff640 1 -- >> v2:172.21.15.71:3300/0 conn(0x7f8cf00a2e90 msgr2=0x7f8cf006dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:49:27.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cef7fe640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f8cf00a3270 legacy=0x7f8cf00a3680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:49:27.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8ceffff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f8cf00a2e90 0x7f8cf006dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:27.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf488c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8cf013d490 0x7f8cf013f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:27.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf488c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8cf013d490 0x7f8cf013f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60178/0 (socket says 172.21.15.12:60178) 2024-09-11T01:49:27.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf488c640 1 -- 172.21.15.12:0/1444382459 learned_addr learned my addr 172.21.15.12:0/1444382459 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:49:27.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf488c640 10 monclient(hunting): get_auth_request con 0x7f8cf013d490 auth_method 0 2024-09-11T01:49:27.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf488c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:27.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf488c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:27.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf488c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:27.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf488c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:27.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf488c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:27.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf488c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:27.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf488c640 10 monclient(hunting): handle_auth_done global_id 214257 payload 274 2024-09-11T01:49:27.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf488c640 1 -- 172.21.15.12:0/1444382459 >> v2:172.21.15.71:3300/0 conn(0x7f8cf00a2e90 msgr2=0x7f8cf006dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:49:27.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf488c640 1 --2- 172.21.15.12:0/1444382459 >> v2:172.21.15.71:3300/0 conn(0x7f8cf00a2e90 0x7f8cf006dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:27.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf488c640 1 -- 172.21.15.12:0/1444382459 >> v1:172.21.15.71:6789/0 conn(0x7f8cf00a3270 legacy=0x7f8cf00a3680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:49:27.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf488c640 10 monclient: _finish_hunting 0 2024-09-11T01:49:27.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf488c640 1 monclient: found mon.noname-a 2024-09-11T01:49:27.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf488c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:27.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf488c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:49:27.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.816+0000 7f8cf488c640 1 -- 172.21.15.12:0/1444382459 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cf006eb70 con 0x7f8cf013d490 2024-09-11T01:49:27.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.819+0000 7f8cf488c640 1 -- 172.21.15.12:0/1444382459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cf013d490 msgr2=0x7f8cf013f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:27.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.819+0000 7f8cf488c640 1 --2- 172.21.15.12:0/1444382459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cf013d490 0x7f8cf013f880 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8cf00a94a0 tx=0x7f8ce802a580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0df98ad8bc902c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:27.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.819+0000 7f8ceeffd640 1 -- 172.21.15.12:0/1444382459 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8036070 con 0x7f8cf013d490 2024-09-11T01:49:27.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.819+0000 7f8ceeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:27.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.819+0000 7f8ceeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:49:27.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.819+0000 7f8ceeffd640 10 monclient: dump: 2024-09-11T01:49:27.827 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:27.827 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:27.827 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:27.827 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:27.827 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:27.828 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:27.828 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:27.828 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:27.828 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:27.828 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:27.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.819+0000 7f8ceeffd640 10 monclient: _finish_auth 0 2024-09-11T01:49:27.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.819+0000 7f8ceeffd640 10 monclient: _check_auth_tickets 2024-09-11T01:49:27.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.819+0000 7f8ceeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:27.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.819+0000 7f8ceeffd640 1 -- 172.21.15.12:0/1444382459 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ce8007800 con 0x7f8cf013d490 2024-09-11T01:49:27.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.819+0000 7f8cf62ef640 5 monclient: authenticate success, global_id 214257 2024-09-11T01:49:27.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.819+0000 7f8cf62ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:49:27.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.819+0000 7f8ceeffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:27.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.819+0000 7f8cf62ef640 10 monclient: get_monmap_and_config success 2024-09-11T01:49:27.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.819+0000 7f8ceeffd640 1 -- 172.21.15.12:0/1444382459 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8007bd0 con 0x7f8cf013d490 2024-09-11T01:49:27.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.819+0000 7f8ceeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:27.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.819+0000 7f8cf62ef640 4 set_mon_vals no callback set 2024-09-11T01:49:27.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.819+0000 7f8ceeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:27.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.819+0000 7f8ceeffd640 10 monclient: dump: 2024-09-11T01:49:27.829 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:27.829 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:27.829 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:27.829 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:27.829 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:27.829 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:27.829 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:27.829 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:27.830 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:27.830 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:27.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.820+0000 7f8cf62ef640 10 monclient: shutdown 2024-09-11T01:49:27.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.820+0000 7f8cf62ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:27.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.820+0000 7f8cf62ef640 1 -- 172.21.15.12:0/1444382459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cf013d490 msgr2=0x7f8cf013f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:27.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.820+0000 7f8cf62ef640 1 --2- 172.21.15.12:0/1444382459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cf013d490 0x7f8cf013f880 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8cf00a94a0 tx=0x7f8ce802a580 comp rx=0 tx=0).stop 2024-09-11T01:49:27.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.820+0000 7f8cf62ef640 1 -- 172.21.15.12:0/1444382459 shutdown_connections 2024-09-11T01:49:27.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.820+0000 7f8cf62ef640 1 --2- 172.21.15.12:0/1444382459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cf013d490 0x7f8cf013f880 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:27.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.820+0000 7f8cf62ef640 1 --2- 172.21.15.12:0/1444382459 >> v2:172.21.15.71:3300/0 conn(0x7f8cf00a2e90 0x7f8cf006dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:27.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.820+0000 7f8cf62ef640 1 -- 172.21.15.12:0/1444382459 >> 172.21.15.12:0/1444382459 conn(0x7f8cf00a8490 msgr2=0x7f8cf00a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:27.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.820+0000 7f8cf62ef640 1 -- 172.21.15.12:0/1444382459 shutdown_connections 2024-09-11T01:49:27.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.820+0000 7f8cf62ef640 1 -- 172.21.15.12:0/1444382459 wait complete. 2024-09-11T01:49:27.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.821+0000 7f8cf62ef640 10 monclient: build_initial_monmap 2024-09-11T01:49:27.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.821+0000 7f8cf62ef640 10 monclient: monmap: 2024-09-11T01:49:27.831 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:27.831 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:49:27.831 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:49:27.831 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:49:27.831 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:27.831 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:27.831 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:49:27.831 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:49:27.831 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:49:27.831 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:27.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.821+0000 7f8cf62ef640 1 Processor -- start 2024-09-11T01:49:27.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.821+0000 7f8cf62ef640 1 -- start start 2024-09-11T01:49:27.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.821+0000 7f8cf62ef640 10 monclient: init 2024-09-11T01:49:27.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8cf62ef640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:27.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8cf62ef640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:49:27.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8cf62ef640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cf00a2e90 0x7f8cf0139c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:27.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8cf62ef640 10 monclient(hunting): picked mon.noname-a con 0x7f8cf00a2e90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:49:27.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8cf62ef640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8cf00a3270 0x7f8cf013bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:27.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8cf62ef640 10 monclient(hunting): picked mon.noname-b con 0x7f8cf00a3270 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:49:27.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8cf62ef640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8cf013d490 0x7f8cf0138520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:27.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8cf62ef640 10 monclient(hunting): picked mon.noname-c con 0x7f8cf013d490 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:27.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8cf62ef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:27.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8cf62ef640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf006fd20 con 0x7f8cf00a2e90 2024-09-11T01:49:27.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8cf62ef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:27.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8cf62ef640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf006fba0 con 0x7f8cf00a3270 2024-09-11T01:49:27.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8cf62ef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:27.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8cf62ef640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8cf013fdc0 con 0x7f8cf013d490 2024-09-11T01:49:27.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8cf62ef640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:27.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8cf62ef640 10 monclient(hunting): authenticate will time out at 1974.264132s 2024-09-11T01:49:27.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8ceffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cf00a2e90 0x7f8cf0139c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:27.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8ceffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cf00a2e90 0x7f8cf0139c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60192/0 (socket says 172.21.15.12:60192) 2024-09-11T01:49:27.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8cf488c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8cf013d490 0x7f8cf0138520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:27.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8ceffff640 1 -- 172.21.15.12:0/1261831693 learned_addr learned my addr 172.21.15.12:0/1261831693 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:49:27.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8ceffff640 10 monclient(hunting): get_auth_request con 0x7f8cf00a2e90 auth_method 0 2024-09-11T01:49:27.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8ceffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:27.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8ceffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:27.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8cef7fe640 1 -- 172.21.15.12:0/1261831693 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8cf00a3270 msgr2=0x7f8cf013bc50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:49:27.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8ceffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:27.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8cef7fe640 1 --2- 172.21.15.12:0/1261831693 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8cf00a3270 0x7f8cf013bc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:27.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8cf488c640 10 monclient(hunting): get_auth_request con 0x7f8cf013d490 auth_method 0 2024-09-11T01:49:27.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8cf488c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:27.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8cf488c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:27.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8cf488c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:27.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8ceffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:27.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8ceffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:27.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8ceffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:27.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8cf488c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:27.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8cf488c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:27.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8cf488c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:27.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8ceffff640 10 monclient(hunting): handle_auth_done global_id 214260 payload 824 2024-09-11T01:49:27.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8ceffff640 1 -- 172.21.15.12:0/1261831693 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8cf013d490 msgr2=0x7f8cf0138520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:27.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8ceffff640 1 --2- 172.21.15.12:0/1261831693 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8cf013d490 0x7f8cf0138520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:27.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8ceffff640 1 -- 172.21.15.12:0/1261831693 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8cf00a3270 msgr2=0x7f8cf013bc50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:49:27.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8ceffff640 1 --2- 172.21.15.12:0/1261831693 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8cf00a3270 0x7f8cf013bc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:27.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8ceffff640 10 monclient: _finish_hunting 0 2024-09-11T01:49:27.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8ceffff640 1 monclient: found mon.noname-a 2024-09-11T01:49:27.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8ceffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:27.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8ceffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:49:27.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.822+0000 7f8ceffff640 1 -- 172.21.15.12:0/1261831693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cf0122ef0 con 0x7f8cf00a2e90 2024-09-11T01:49:27.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.824+0000 7f8ceffff640 1 -- 172.21.15.12:0/1261831693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cf00a2e90 msgr2=0x7f8cf0139c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:27.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.824+0000 7f8ceffff640 1 --2- 172.21.15.12:0/1261831693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cf00a2e90 0x7f8cf0139c40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8ce000b550 tx=0x7f8ce000ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:27.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.824+0000 7f8cecff9640 1 -- 172.21.15.12:0/1261831693 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce0013020 con 0x7f8cf00a2e90 2024-09-11T01:49:27.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.825+0000 7f8cecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:27.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.825+0000 7f8cecff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:49:27.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.825+0000 7f8cecff9640 10 monclient: dump: 2024-09-11T01:49:27.836 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:27.836 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:27.836 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:27.836 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:27.836 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:27.836 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:27.836 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:27.836 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:27.836 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:27.836 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:27.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.825+0000 7f8cecff9640 10 monclient: _finish_auth 0 2024-09-11T01:49:27.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.825+0000 7f8cecff9640 10 monclient: _check_auth_tickets 2024-09-11T01:49:27.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.825+0000 7f8cecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:27.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.825+0000 7f8cecff9640 1 -- 172.21.15.12:0/1261831693 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ce0005070 con 0x7f8cf00a2e90 2024-09-11T01:49:27.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.825+0000 7f8cecff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:27.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.825+0000 7f8cecff9640 1 -- 172.21.15.12:0/1261831693 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce0005480 con 0x7f8cf00a2e90 2024-09-11T01:49:27.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.825+0000 7f8cf62ef640 5 monclient: authenticate success, global_id 214260 2024-09-11T01:49:27.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.825+0000 7f8cecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:27.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.825+0000 7f8cecff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:27.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.825+0000 7f8cecff9640 10 monclient: dump: 2024-09-11T01:49:27.837 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:27.837 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:27.837 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:27.837 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:27.838 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:27.838 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:27.838 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:27.838 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:27.838 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:27.838 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:27.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.825+0000 7f8cf62ef640 10 monclient: _renew_subs 2024-09-11T01:49:27.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.825+0000 7f8cf62ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:27.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.825+0000 7f8cf62ef640 1 -- 172.21.15.12:0/1261831693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cf013b110 con 0x7f8cf00a2e90 2024-09-11T01:49:27.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.825+0000 7f8cee7fc640 4 set_mon_vals no callback set 2024-09-11T01:49:27.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.825+0000 7f8cf62ef640 10 monclient: _renew_subs 2024-09-11T01:49:27.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.825+0000 7f8cf62ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:27.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.825+0000 7f8cf62ef640 1 -- 172.21.15.12:0/1261831693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cf009f680 con 0x7f8cf00a2e90 2024-09-11T01:49:27.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.826+0000 7f8cf62ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:27.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.826+0000 7f8cf62ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:27.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.826+0000 7f8cf62ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:27.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.826+0000 7f8cf62ef640 1 -- 172.21.15.12:0/1261831693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cf00a7590 con 0x7f8cf00a2e90 2024-09-11T01:49:27.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.826+0000 7f8cecff9640 1 -- 172.21.15.12:0/1261831693 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f8ce0005620 con 0x7f8cf00a2e90 2024-09-11T01:49:27.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.826+0000 7f8cecff9640 1 --2- 172.21.15.12:0/1261831693 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8cdc03e1f0 0x7f8cdc0406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:27.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.826+0000 7f8cecff9640 1 -- 172.21.15.12:0/1261831693 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(678..678 src has 507..678) ==== 4847+0+0 (secure 0 0 0) 0x7f8ce0052cd0 con 0x7f8cf00a2e90 2024-09-11T01:49:27.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.827+0000 7f8cef7fe640 1 --2- 172.21.15.12:0/1261831693 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8cdc03e1f0 0x7f8cdc0406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:27.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.827+0000 7f8cef7fe640 10 monclient: get_auth_request con 0x7f8cdc03e1f0 auth_method 0 2024-09-11T01:49:27.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.829+0000 7f8cef7fe640 1 --2- 172.21.15.12:0/1261831693 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8cdc03e1f0 0x7f8cdc0406b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f8cf013c380 tx=0x7f8ce4006d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:27.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.831+0000 7f8cecff9640 1 -- 172.21.15.12:0/1261831693 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8cf00a7590 con 0x7f8cf00a2e90 2024-09-11T01:49:27.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.831+0000 7f8cecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:27.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:27.831+0000 7f8cecff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:49:28.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:28.090+0000 7f8cf62ef640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:49:28.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:28.090+0000 7f8cf62ef640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:49:28.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:28.090+0000 7f8cf62ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:28.090+0000 7f8cf62ef640 1 -- 172.21.15.12:0/1261831693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8cf006dc60 con 0x7f8cf00a2e90 2024-09-11T01:49:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:28.091+0000 7f8cecff9640 1 -- 172.21.15.12:0/1261831693 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7f8ce001d020 con 0x7f8cf00a2e90 2024-09-11T01:49:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:28.091+0000 7f8cecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:49:28.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:28.091+0000 7f8cecff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:49:28.093 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":188,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:49:28.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:28.093+0000 7f8cf62ef640 1 -- 172.21.15.12:0/1261831693 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8cdc03e1f0 msgr2=0x7f8cdc0406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:28.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:28.093+0000 7f8cf62ef640 1 --2- 172.21.15.12:0/1261831693 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8cdc03e1f0 0x7f8cdc0406b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f8cf013c380 tx=0x7f8ce4006d30 comp rx=0 tx=0).stop 2024-09-11T01:49:28.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:28.093+0000 7f8cf62ef640 10 monclient: shutdown 2024-09-11T01:49:28.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:28.093+0000 7f8cf62ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:28.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:28.093+0000 7f8cf62ef640 1 -- 172.21.15.12:0/1261831693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cf00a2e90 msgr2=0x7f8cf0139c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:28.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:28.093+0000 7f8cf62ef640 1 --2- 172.21.15.12:0/1261831693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cf00a2e90 0x7f8cf0139c40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8ce000b550 tx=0x7f8ce000ba10 comp rx=0 tx=0).stop 2024-09-11T01:49:28.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:28.093+0000 7f8cf62ef640 1 -- 172.21.15.12:0/1261831693 shutdown_connections 2024-09-11T01:49:28.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:28.093+0000 7f8cf62ef640 1 --2- 172.21.15.12:0/1261831693 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8cf013d490 0x7f8cf0138520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:28.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:28.093+0000 7f8cf62ef640 1 --2- 172.21.15.12:0/1261831693 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8cf00a3270 0x7f8cf013bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:28.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:28.093+0000 7f8cf62ef640 1 --2- 172.21.15.12:0/1261831693 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8cdc03e1f0 0x7f8cdc0406b0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:28.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:28.093+0000 7f8cf62ef640 1 --2- 172.21.15.12:0/1261831693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cf00a2e90 0x7f8cf0139c40 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:28.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:28.093+0000 7f8cf62ef640 1 -- 172.21.15.12:0/1261831693 >> 172.21.15.12:0/1261831693 conn(0x7f8cf00a8490 msgr2=0x7f8cf00a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:28.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:28.093+0000 7f8cf62ef640 1 -- 172.21.15.12:0/1261831693 shutdown_connections 2024-09-11T01:49:28.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:28.093+0000 7f8cf62ef640 1 -- 172.21.15.12:0/1261831693 wait complete. 2024-09-11T01:49:31.106 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:49:31.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.217+0000 7fe4996ed640 10 monclient: get_monmap_and_config 2024-09-11T01:49:31.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.217+0000 7fe4996ed640 10 monclient: build_initial_monmap 2024-09-11T01:49:31.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.217+0000 7fe4996ed640 10 monclient: monmap: 2024-09-11T01:49:31.219 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:31.219 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:31.220 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:49:31.218602+0000 2024-09-11T01:49:31.220 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:49:31.218602+0000 2024-09-11T01:49:31.220 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:31.220 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:31.220 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:49:31.220 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:49:31.220 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:49:31.220 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:49:31.220 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:49:31.220 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:31.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.218+0000 7fe4996ed640 1 Processor -- start 2024-09-11T01:49:31.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.218+0000 7fe4996ed640 1 -- start start 2024-09-11T01:49:31.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.218+0000 7fe4996ed640 10 monclient: init 2024-09-11T01:49:31.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe4996ed640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:31.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe4996ed640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-11T01:49:31.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe4996ed640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe49409d790 0x7fe49409db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:31.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe4996ed640 10 monclient(hunting): picked mon.noname-b con 0x7fe49409d790 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:31.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe4996ed640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe49409e0b0 addr v1:172.21.15.12:6789/0 2024-09-11T01:49:31.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe4996ed640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe49409e4d0 0x7fe494135170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:31.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe4996ed640 10 monclient(hunting): picked mon.noname-c con 0x7fe49409e4d0 addr v2:172.21.15.71:3300/0 2024-09-11T01:49:31.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe4996ed640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4940a1da0 con 0x7fe49409e0b0 2024-09-11T01:49:31.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe4996ed640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:31.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe4996ed640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fe4940a0760 con 0x7fe49409e4d0 2024-09-11T01:49:31.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe4996ed640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:31.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe4996ed640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe49409e8f0 con 0x7fe49409d790 2024-09-11T01:49:31.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe4996ed640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:31.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe4996ed640 10 monclient(hunting): authenticate will time out at 1977.661237s 2024-09-11T01:49:31.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe492ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe49409d790 0x7fe49409db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:31.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe48bfff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe49409e0b0 0x7fe494134a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:51300/0 (socket says 172.21.15.12:51300) 2024-09-11T01:49:31.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe48bfff640 1 -- 172.21.15.12:0/2961582046 learned_addr learned my addr 172.21.15.12:0/2961582046 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:49:31.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe4937fe640 1 -- 172.21.15.12:0/2961582046 >> v2:172.21.15.71:3300/0 conn(0x7fe49409e4d0 msgr2=0x7fe494135170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:49:31.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe492ffd640 10 monclient(hunting): get_auth_request con 0x7fe49409d790 auth_method 0 2024-09-11T01:49:31.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe492ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:31.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe492ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:31.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe492ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:31.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe4937fe640 1 --2- 172.21.15.12:0/2961582046 >> v2:172.21.15.71:3300/0 conn(0x7fe49409e4d0 0x7fe494135170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:31.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe492ffd640 1 -- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe49409d790 msgr2=0x7fe49409db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:49:31.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe492ffd640 1 -- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe49409d790 msgr2=0x7fe49409db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:49:31.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe492ffd640 1 --2- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe49409d790 0x7fe49409db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:49:31.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.219+0000 7fe492ffd640 1 --2- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe49409d790 0x7fe49409db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:31.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.221+0000 7fe48bfff640 1 -- 172.21.15.12:0/2961582046 >> v1:172.21.15.12:6789/0 conn(0x7fe49409e0b0 legacy=0x7fe494134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:31.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.420+0000 7fe4937fe640 1 -- 172.21.15.12:0/2961582046 >> v2:172.21.15.71:3300/0 conn(0x7fe49409e4d0 msgr2=0x7fe494135170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:49:31.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.420+0000 7fe4937fe640 1 --2- 172.21.15.12:0/2961582046 >> v2:172.21.15.71:3300/0 conn(0x7fe49409e4d0 0x7fe494135170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:49:31.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.420+0000 7fe492ffd640 1 --2- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe49409d790 0x7fe49409db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:31.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.420+0000 7fe492ffd640 10 monclient(hunting): get_auth_request con 0x7fe49409d790 auth_method 0 2024-09-11T01:49:31.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.420+0000 7fe492ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:31.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.420+0000 7fe492ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:31.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.420+0000 7fe492ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:49:31.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.420+0000 7fe492ffd640 1 -- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe49409d790 msgr2=0x7fe49409db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:49:31.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.420+0000 7fe492ffd640 1 -- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe49409d790 msgr2=0x7fe49409db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:49:31.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.420+0000 7fe492ffd640 1 --2- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe49409d790 0x7fe49409db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:49:31.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.420+0000 7fe492ffd640 1 --2- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe49409d790 0x7fe49409db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:49:31.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.821+0000 7fe4937fe640 1 -- 172.21.15.12:0/2961582046 >> v2:172.21.15.71:3300/0 conn(0x7fe49409e4d0 msgr2=0x7fe494135170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:49:31.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.821+0000 7fe4937fe640 1 --2- 172.21.15.12:0/2961582046 >> v2:172.21.15.71:3300/0 conn(0x7fe49409e4d0 0x7fe494135170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:49:31.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.821+0000 7fe492ffd640 1 --2- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe49409d790 0x7fe49409db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:31.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.821+0000 7fe492ffd640 10 monclient(hunting): get_auth_request con 0x7fe49409d790 auth_method 0 2024-09-11T01:49:31.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.821+0000 7fe492ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:31.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.821+0000 7fe492ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:31.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.821+0000 7fe492ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:49:31.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.822+0000 7fe492ffd640 1 -- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe49409d790 msgr2=0x7fe49409db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:49:31.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.822+0000 7fe492ffd640 1 -- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe49409d790 msgr2=0x7fe49409db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:49:31.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.822+0000 7fe492ffd640 1 --2- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe49409d790 0x7fe49409db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:49:31.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:31.822+0000 7fe492ffd640 1 --2- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe49409d790 0x7fe49409db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:49:32.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:32.622+0000 7fe4937fe640 1 -- 172.21.15.12:0/2961582046 >> v2:172.21.15.71:3300/0 conn(0x7fe49409e4d0 msgr2=0x7fe494135170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:49:32.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:32.622+0000 7fe4937fe640 1 --2- 172.21.15.12:0/2961582046 >> v2:172.21.15.71:3300/0 conn(0x7fe49409e4d0 0x7fe494135170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:49:32.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:32.624+0000 7fe492ffd640 1 --2- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe49409d790 0x7fe49409db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:32.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:32.624+0000 7fe492ffd640 10 monclient(hunting): get_auth_request con 0x7fe49409d790 auth_method 0 2024-09-11T01:49:32.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:32.624+0000 7fe492ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:32.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:32.624+0000 7fe492ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:32.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:32.624+0000 7fe492ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:49:32.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:32.624+0000 7fe492ffd640 1 -- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe49409d790 msgr2=0x7fe49409db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:49:32.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:32.624+0000 7fe492ffd640 1 -- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe49409d790 msgr2=0x7fe49409db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:49:32.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:32.624+0000 7fe492ffd640 1 --2- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe49409d790 0x7fe49409db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:49:32.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:32.624+0000 7fe492ffd640 1 --2- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe49409d790 0x7fe49409db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:49:34.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe4917fa640 10 monclient(hunting): tick 2024-09-11T01:49:34.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe4917fa640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:49:34.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe4917fa640 1 monclient(hunting): continuing hunt 2024-09-11T01:49:34.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe4917fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:49:34.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe4917fa640 1 -- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe49409d790 msgr2=0x7fe49409db70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:49:34.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe4917fa640 1 --2- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe49409d790 0x7fe49409db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:34.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe4917fa640 1 -- 172.21.15.12:0/2961582046 >> v2:172.21.15.71:3300/0 conn(0x7fe49409e4d0 msgr2=0x7fe494135170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:49:34.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe4917fa640 1 --2- 172.21.15.12:0/2961582046 >> v2:172.21.15.71:3300/0 conn(0x7fe49409e4d0 0x7fe494135170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:34.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe4917fa640 1 -- 172.21.15.12:0/2961582046 >> v1:172.21.15.12:6789/0 conn(0x7fe49409e0b0 legacy=0x7fe494134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:34.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe4917fa640 10 monclient: _add_conns ranks=[1,3] 2024-09-11T01:49:34.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe4917fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe474005950 addr v1:172.21.15.71:6789/0 2024-09-11T01:49:34.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe4917fa640 1 --2- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740096a0 0x7fe47400bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:34.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe4917fa640 10 monclient(hunting): picked mon.noname-a con 0x7fe4740096a0 addr v2:172.21.15.12:3300/0 2024-09-11T01:49:34.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe4917fa640 1 -- 172.21.15.12:0/2961582046 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4940a1fa0 con 0x7fe474005950 2024-09-11T01:49:34.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe4917fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:34.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe4917fa640 1 -- 172.21.15.12:0/2961582046 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fe4940a0760 con 0x7fe4740096a0 2024-09-11T01:49:34.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe4917fa640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:34.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe48bfff640 1 --2- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740096a0 0x7fe47400bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:34.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe492ffd640 1 -- 172.21.15.12:0/2961582046 >> v1:172.21.15.71:6789/0 conn(0x7fe474005950 legacy=0x7fe474007dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:49:34.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe48bfff640 10 monclient(hunting): get_auth_request con 0x7fe4740096a0 auth_method 0 2024-09-11T01:49:34.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe48bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:34.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe48bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:34.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe48bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:34.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe48bfff640 1 -- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740096a0 msgr2=0x7fe47400bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:49:34.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe48bfff640 1 -- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740096a0 msgr2=0x7fe47400bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:49:34.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe48bfff640 1 --2- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740096a0 0x7fe47400bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:49:34.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.219+0000 7fe48bfff640 1 --2- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740096a0 0x7fe47400bb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:34.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.420+0000 7fe492ffd640 1 -- 172.21.15.12:0/2961582046 >> v1:172.21.15.71:6789/0 conn(0x7fe474005950 legacy=0x7fe474007dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:49:34.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.420+0000 7fe48bfff640 1 --2- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740096a0 0x7fe47400bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:34.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.420+0000 7fe48bfff640 10 monclient(hunting): get_auth_request con 0x7fe4740096a0 auth_method 0 2024-09-11T01:49:34.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.420+0000 7fe48bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:34.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.420+0000 7fe48bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:34.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.420+0000 7fe48bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:49:34.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.420+0000 7fe48bfff640 1 -- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740096a0 msgr2=0x7fe47400bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:49:34.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.420+0000 7fe48bfff640 1 -- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740096a0 msgr2=0x7fe47400bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:49:34.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.420+0000 7fe48bfff640 1 --2- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740096a0 0x7fe47400bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:49:34.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.420+0000 7fe48bfff640 1 --2- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740096a0 0x7fe47400bb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:49:34.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.821+0000 7fe48bfff640 1 --2- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740096a0 0x7fe47400bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:34.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.821+0000 7fe48bfff640 10 monclient(hunting): get_auth_request con 0x7fe4740096a0 auth_method 0 2024-09-11T01:49:34.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.821+0000 7fe48bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:34.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.821+0000 7fe48bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:34.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.821+0000 7fe48bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:49:34.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.821+0000 7fe48bfff640 1 -- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740096a0 msgr2=0x7fe47400bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:49:34.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.821+0000 7fe48bfff640 1 -- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740096a0 msgr2=0x7fe47400bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:49:34.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.821+0000 7fe48bfff640 1 --2- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740096a0 0x7fe47400bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:49:34.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.821+0000 7fe48bfff640 1 --2- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740096a0 0x7fe47400bb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:49:34.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:34.821+0000 7fe492ffd640 1 -- 172.21.15.12:0/2961582046 >> v1:172.21.15.71:6789/0 conn(0x7fe474005950 legacy=0x7fe474007dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:49:35.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:35.622+0000 7fe48bfff640 1 --2- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740096a0 0x7fe47400bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:35.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:35.622+0000 7fe48bfff640 10 monclient(hunting): get_auth_request con 0x7fe4740096a0 auth_method 0 2024-09-11T01:49:35.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:35.622+0000 7fe48bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:35.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:35.622+0000 7fe48bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:35.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:35.622+0000 7fe48bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:49:35.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:35.622+0000 7fe48bfff640 1 -- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740096a0 msgr2=0x7fe47400bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:49:35.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:35.622+0000 7fe48bfff640 1 -- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740096a0 msgr2=0x7fe47400bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:49:35.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:35.622+0000 7fe48bfff640 1 --2- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740096a0 0x7fe47400bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:49:35.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:35.622+0000 7fe48bfff640 1 --2- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740096a0 0x7fe47400bb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:49:35.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:35.622+0000 7fe492ffd640 1 -- 172.21.15.12:0/2961582046 >> v1:172.21.15.71:6789/0 conn(0x7fe474005950 legacy=0x7fe474007dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:49:35.893 INFO:tasks.rados.rados.0.smithi071.stdout:3020: done (6 left) 2024-09-11T01:49:35.893 INFO:tasks.rados.rados.0.smithi071.stdout:3021: copy_from oid 37 from oid 12 current snap is 341 2024-09-11T01:49:35.893 INFO:tasks.rados.rados.0.smithi071.stdout:3015: finishing copy_from to smithi07134777-30 2024-09-11T01:49:35.893 INFO:tasks.rados.rados.0.smithi071.stdout:3015: got expected ENOENT (src dne) 2024-09-11T01:49:35.893 INFO:tasks.rados.rados.0.smithi071.stdout:3015: finishing copy_from racing read to smithi07134777-30 2024-09-11T01:49:35.893 INFO:tasks.rados.rados.0.smithi071.stdout:3018: finishing rollback tid 1 to smithi07134777-2 2024-09-11T01:49:35.893 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 879 (ObjNum 129 snap 0 seq_num 2370491280) dirty dne 2024-09-11T01:49:35.893 INFO:tasks.rados.rados.0.smithi071.stdout:3016: finishing rollback tid 1 to smithi07134777-28 2024-09-11T01:49:35.893 INFO:tasks.rados.rados.0.smithi071.stdout:3016: finishing rollback tid 2 to smithi07134777-28 2024-09-11T01:49:35.893 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 2050 (ObjNum 965 snap 305 seq_num 965) dirty exists 2024-09-11T01:49:35.893 INFO:tasks.rados.rados.0.smithi071.stdout:3014: expect (ObjNum 1046 snap 337 seq_num 1046) 2024-09-11T01:49:35.999 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 11758 2024-09-11T01:49:36.064 INFO:tasks.rados.rados.0.smithi071.stdout:3017: finishing rollback tid 0 to smithi07134777-14 2024-09-11T01:49:36.064 INFO:tasks.rados.rados.0.smithi071.stdout:3017: finishing rollback tid 1 to smithi07134777-14 2024-09-11T01:49:36.064 INFO:tasks.rados.rados.0.smithi071.stdout:3014: done (6 left) 2024-09-11T01:49:36.064 INFO:tasks.rados.rados.0.smithi071.stdout:3015: done (5 left) 2024-09-11T01:49:36.064 INFO:tasks.rados.rados.0.smithi071.stdout:3016: done (4 left) 2024-09-11T01:49:36.064 INFO:tasks.rados.rados.0.smithi071.stdout:3018: done (3 left) 2024-09-11T01:49:36.064 INFO:tasks.rados.rados.0.smithi071.stdout:3022: rollback oid 32 current snap is 341 2024-09-11T01:49:36.064 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 32 to 307 2024-09-11T01:49:36.064 INFO:tasks.rados.rados.0.smithi071.stdout:3017: finishing rollback tid 2 to smithi07134777-14 2024-09-11T01:49:36.065 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 2153 (ObjNum 942 snap 296 seq_num 942) dirty exists 2024-09-11T01:49:36.065 INFO:tasks.rados.rados.0.smithi071.stdout:3013: finishing copy_from racing read to smithi07134777-17 2024-09-11T01:49:36.065 INFO:tasks.rados.rados.0.smithi071.stdout:3013: finishing copy_from to smithi07134777-17 2024-09-11T01:49:36.065 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 1785 (ObjNum 1042 snap 336 seq_num 1042) dirty exists 2024-09-11T01:49:36.065 INFO:tasks.rados.rados.0.smithi071.stdout:3021: finishing copy_from racing read to smithi07134777-37 2024-09-11T01:49:36.065 INFO:tasks.rados.rados.0.smithi071.stdout:3021: finishing copy_from to smithi07134777-37 2024-09-11T01:49:36.065 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 1936 (ObjNum 1049 snap 339 seq_num 1049) dirty exists 2024-09-11T01:49:36.065 INFO:tasks.rados.rados.0.smithi071.stdout:3013: done (3 left) 2024-09-11T01:49:36.065 INFO:tasks.rados.rados.0.smithi071.stdout:3017: done (2 left) 2024-09-11T01:49:36.065 INFO:tasks.rados.rados.0.smithi071.stdout:3021: done (1 left) 2024-09-11T01:49:36.065 INFO:tasks.rados.rados.0.smithi071.stdout:3023: write oid 36 current snap is 341 2024-09-11T01:49:36.066 INFO:tasks.rados.rados.0.smithi071.stdout:3023: seq_num 1063 ranges {665841=562754,1687653=572365,2708406=776032} 2024-09-11T01:49:36.069 INFO:tasks.rados.rados.0.smithi071.stdout:3023: writing smithi07134777-36 from 665841 to 1228595 tid 1 2024-09-11T01:49:36.074 INFO:tasks.rados.rados.0.smithi071.stdout:3023: writing smithi07134777-36 from 1687653 to 2260018 tid 2 2024-09-11T01:49:36.081 INFO:tasks.rados.rados.0.smithi071.stdout:3023: writing smithi07134777-36 from 2708406 to 3484438 tid 3 2024-09-11T01:49:36.082 INFO:tasks.rados.rados.0.smithi071.stdout:3024: snap_create 2024-09-11T01:49:36.082 INFO:tasks.rados.rados.0.smithi071.stdout:3022: finishing rollback tid 0 to smithi07134777-32 2024-09-11T01:49:36.082 INFO:tasks.rados.rados.0.smithi071.stdout:3022: finishing rollback tid 1 to smithi07134777-32 2024-09-11T01:49:36.082 INFO:tasks.rados.rados.0.smithi071.stdout:3022: finishing rollback tid 2 to smithi07134777-32 2024-09-11T01:49:36.082 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 1939 (ObjNum 954 snap 302 seq_num 954) dirty exists 2024-09-11T01:49:36.082 INFO:tasks.rados.rados.0.smithi071.stdout:3023: finishing write tid 1 to smithi07134777-36 2024-09-11T01:49:36.082 INFO:tasks.rados.rados.0.smithi071.stdout:3023: finishing write tid 2 to smithi07134777-36 2024-09-11T01:49:36.086 INFO:tasks.rados.rados.0.smithi071.stdout:3023: finishing write tid 3 to smithi07134777-36 2024-09-11T01:49:36.088 INFO:tasks.rados.rados.0.smithi071.stdout:3023: finishing write tid 4 to smithi07134777-36 2024-09-11T01:49:36.088 INFO:tasks.rados.rados.0.smithi071.stdout:3023: finishing write tid 5 to smithi07134777-36 2024-09-11T01:49:36.088 INFO:tasks.rados.rados.0.smithi071.stdout:3023: oid 36 updating version 0 to 1778 2024-09-11T01:49:36.088 INFO:tasks.rados.rados.0.smithi071.stdout:3023: oid 36 version 1778 is already newer than 1776 2024-09-11T01:49:36.088 INFO:tasks.rados.rados.0.smithi071.stdout:3023: oid 36 version 1778 is already newer than 1775 2024-09-11T01:49:36.088 INFO:tasks.rados.rados.0.smithi071.stdout:3023: oid 36 version 1778 is already newer than 1777 2024-09-11T01:49:36.089 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 1778 (ObjNum 1063 snap 341 seq_num 1063) dirty exists 2024-09-11T01:49:36.089 INFO:tasks.rados.rados.0.smithi071.stdout:3023: left oid 36 (ObjNum 1063 snap 341 seq_num 1063) 2024-09-11T01:49:37.218 INFO:tasks.rados.rados.0.smithi071.stdout:3022: done (2 left) 2024-09-11T01:49:37.218 INFO:tasks.rados.rados.0.smithi071.stdout:3023: done (1 left) 2024-09-11T01:49:37.218 INFO:tasks.rados.rados.0.smithi071.stdout:3024: done (0 left) 2024-09-11T01:49:37.218 INFO:tasks.rados.rados.0.smithi071.stdout:3025: copy_from oid 6 from oid 19 current snap is 342 2024-09-11T01:49:37.218 INFO:tasks.rados.rados.0.smithi071.stdout:3026: rollback oid 38 current snap is 342 2024-09-11T01:49:37.219 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 38 to 334 2024-09-11T01:49:37.219 INFO:tasks.rados.rados.0.smithi071.stdout:3027: copy_from oid 28 from oid 21 current snap is 342 2024-09-11T01:49:37.219 INFO:tasks.rados.rados.0.smithi071.stdout:3028: read oid 44 snap -1 2024-09-11T01:49:37.219 INFO:tasks.rados.rados.0.smithi071.stdout:3028: expect (ObjNum 1061 snap 340 seq_num 1061) 2024-09-11T01:49:37.219 INFO:tasks.rados.rados.0.smithi071.stdout:3029: write (excl) oid 32 current snap is 342 2024-09-11T01:49:37.219 INFO:tasks.rados.rados.0.smithi071.stdout:3029: seq_num 1064 ranges {682648=638003,2029782=432376,3019476=587304} 2024-09-11T01:49:37.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.219+0000 7fe4917fa640 10 monclient(hunting): tick 2024-09-11T01:49:37.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.219+0000 7fe4917fa640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:49:37.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.219+0000 7fe4917fa640 1 monclient(hunting): continuing hunt 2024-09-11T01:49:37.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.219+0000 7fe4917fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:49:37.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.219+0000 7fe4917fa640 1 -- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740096a0 msgr2=0x7fe47400bb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:49:37.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.219+0000 7fe4917fa640 1 --2- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740096a0 0x7fe47400bb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:37.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.219+0000 7fe4917fa640 1 -- 172.21.15.12:0/2961582046 >> v1:172.21.15.71:6789/0 conn(0x7fe474005950 legacy=0x7fe474007dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:49:37.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.219+0000 7fe4917fa640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-11T01:49:37.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.219+0000 7fe4937fe640 1 -- 172.21.15.12:0/2961582046 reap_dead start 2024-09-11T01:49:37.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.219+0000 7fe4917fa640 1 --2- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740085a0 0x7fe47400bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:37.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.219+0000 7fe4917fa640 10 monclient(hunting): picked mon.noname-a con 0x7fe4740085a0 addr v2:172.21.15.12:3300/0 2024-09-11T01:49:37.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.219+0000 7fe4917fa640 1 --2- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe474008a20 0x7fe474008e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:37.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.219+0000 7fe4917fa640 10 monclient(hunting): picked mon.noname-b con 0x7fe474008a20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:37.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.219+0000 7fe4917fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe474007d40 addr v1:172.21.15.71:6789/0 2024-09-11T01:49:37.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.219+0000 7fe4917fa640 1 -- 172.21.15.12:0/2961582046 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe47400d2f0 con 0x7fe474007d40 2024-09-11T01:49:37.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.219+0000 7fe4917fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:37.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.219+0000 7fe4917fa640 1 -- 172.21.15.12:0/2961582046 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fe4940a0760 con 0x7fe4740085a0 2024-09-11T01:49:37.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.219+0000 7fe4917fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:37.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.219+0000 7fe4917fa640 1 -- 172.21.15.12:0/2961582046 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe49409e8f0 con 0x7fe474008a20 2024-09-11T01:49:37.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.219+0000 7fe4917fa640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:37.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.219+0000 7fe4937fe640 1 -- 172.21.15.12:0/2961582046 >> v1:172.21.15.71:6789/0 conn(0x7fe474007d40 legacy=0x7fe47400c1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:49:37.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.220+0000 7fe48bfff640 1 --2- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe474008a20 0x7fe474008e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:37.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.220+0000 7fe48bfff640 10 monclient(hunting): get_auth_request con 0x7fe474008a20 auth_method 0 2024-09-11T01:49:37.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.220+0000 7fe48bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:37.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.220+0000 7fe48bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:37.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.220+0000 7fe48bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:37.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.220+0000 7fe492ffd640 1 --2- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740085a0 0x7fe47400bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:37.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.220+0000 7fe492ffd640 10 monclient(hunting): get_auth_request con 0x7fe4740085a0 auth_method 0 2024-09-11T01:49:37.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.220+0000 7fe492ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:37.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.220+0000 7fe492ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:37.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.220+0000 7fe492ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:37.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.220+0000 7fe48bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:37.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.220+0000 7fe48bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:37.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.220+0000 7fe48bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:37.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.220+0000 7fe492ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:37.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.220+0000 7fe492ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:37.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.220+0000 7fe492ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:37.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.220+0000 7fe48bfff640 10 monclient(hunting): handle_auth_done global_id 224150 payload 274 2024-09-11T01:49:37.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.220+0000 7fe48bfff640 1 -- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740085a0 msgr2=0x7fe47400bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:49:37.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.220+0000 7fe48bfff640 1 --2- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740085a0 0x7fe47400bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:37.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.220+0000 7fe48bfff640 1 -- 172.21.15.12:0/2961582046 >> v1:172.21.15.71:6789/0 conn(0x7fe474007d40 legacy=0x7fe47400c1f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:49:37.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.220+0000 7fe48bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:49:37.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.220+0000 7fe48bfff640 1 monclient: found mon.noname-b 2024-09-11T01:49:37.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.220+0000 7fe48bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:37.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.220+0000 7fe48bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:49:37.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.220+0000 7fe48bfff640 1 -- 172.21.15.12:0/2961582046 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4940a1fa0 con 0x7fe474008a20 2024-09-11T01:49:37.226 INFO:tasks.rados.rados.0.smithi071.stdout:3029: writing smithi07134777-32 from 682648 to 1320651 tid 1 2024-09-11T01:49:37.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.223+0000 7fe48bfff640 1 -- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe474008a20 msgr2=0x7fe474008e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:37.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.223+0000 7fe48bfff640 1 --2- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe474008a20 0x7fe474008e80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe47c007c00 tx=0x7fe47c02add0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66420c3a5b5ae7e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:37.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.223+0000 7fe4927fc640 1 -- 172.21.15.12:0/2961582046 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47c031070 con 0x7fe474008a20 2024-09-11T01:49:37.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.223+0000 7fe4927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:37.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.223+0000 7fe4927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:49:37.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.223+0000 7fe4927fc640 10 monclient: dump: 2024-09-11T01:49:37.228 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:37.228 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:37.228 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.228 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.228 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:37.228 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:37.229 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:37.229 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:37.229 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:37.229 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:37.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.224+0000 7fe4927fc640 10 monclient: _finish_auth 0 2024-09-11T01:49:37.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.224+0000 7fe4927fc640 10 monclient: _check_auth_tickets 2024-09-11T01:49:37.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.224+0000 7fe4927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:37.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.224+0000 7fe4927fc640 1 -- 172.21.15.12:0/2961582046 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe47c02b8d0 con 0x7fe474008a20 2024-09-11T01:49:37.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.224+0000 7fe4996ed640 5 monclient: authenticate success, global_id 224150 2024-09-11T01:49:37.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.224+0000 7fe4996ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:49:37.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.224+0000 7fe4927fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:37.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.224+0000 7fe4996ed640 10 monclient: get_monmap_and_config success 2024-09-11T01:49:37.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.224+0000 7fe4927fc640 1 -- 172.21.15.12:0/2961582046 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47c02bca0 con 0x7fe474008a20 2024-09-11T01:49:37.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.224+0000 7fe4927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:37.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.224+0000 7fe4996ed640 4 set_mon_vals no callback set 2024-09-11T01:49:37.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.224+0000 7fe4927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:49:37.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.224+0000 7fe4927fc640 10 monclient: dump: 2024-09-11T01:49:37.230 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:37.230 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:37.230 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.230 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.230 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:37.230 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:37.230 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:37.230 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:37.230 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:37.230 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:37.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.225+0000 7fe4996ed640 10 monclient: shutdown 2024-09-11T01:49:37.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.225+0000 7fe4996ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:37.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.225+0000 7fe4996ed640 1 -- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe474008a20 msgr2=0x7fe474008e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:37.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.225+0000 7fe4996ed640 1 --2- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe474008a20 0x7fe474008e80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe47c007c00 tx=0x7fe47c02add0 comp rx=0 tx=0).stop 2024-09-11T01:49:37.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.225+0000 7fe4996ed640 1 -- 172.21.15.12:0/2961582046 shutdown_connections 2024-09-11T01:49:37.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.225+0000 7fe4996ed640 1 --2- 172.21.15.12:0/2961582046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe474008a20 0x7fe474008e80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:37.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.225+0000 7fe4996ed640 1 --2- 172.21.15.12:0/2961582046 >> v2:172.21.15.12:3300/0 conn(0x7fe4740085a0 0x7fe47400bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:37.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.225+0000 7fe4996ed640 1 -- 172.21.15.12:0/2961582046 >> 172.21.15.12:0/2961582046 conn(0x7fe4940a2d90 msgr2=0x7fe4940a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:37.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.225+0000 7fe4996ed640 1 -- 172.21.15.12:0/2961582046 shutdown_connections 2024-09-11T01:49:37.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.225+0000 7fe4996ed640 1 -- 172.21.15.12:0/2961582046 wait complete. 2024-09-11T01:49:37.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.226+0000 7fe4996ed640 10 monclient: build_initial_monmap 2024-09-11T01:49:37.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.226+0000 7fe4996ed640 10 monclient: monmap: 2024-09-11T01:49:37.231 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:37.231 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:49:37.232 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:49:37.232 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:49:37.232 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:37.232 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:37.232 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:49:37.232 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:49:37.232 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:49:37.232 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:37.233 INFO:tasks.rados.rados.0.smithi071.stdout:3029: writing smithi07134777-32 from 2029782 to 2462158 tid 2 2024-09-11T01:49:37.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.226+0000 7fe4996ed640 1 Processor -- start 2024-09-11T01:49:37.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.227+0000 7fe4996ed640 1 -- start start 2024-09-11T01:49:37.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.227+0000 7fe4996ed640 10 monclient: init 2024-09-11T01:49:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.227+0000 7fe4996ed640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.227+0000 7fe4996ed640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:49:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.227+0000 7fe4996ed640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe474007d40 0x7fe494134c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.227+0000 7fe4996ed640 10 monclient(hunting): picked mon.noname-a con 0x7fe474007d40 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:49:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.227+0000 7fe4996ed640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4740085a0 0x7fe494121560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.227+0000 7fe4996ed640 10 monclient(hunting): picked mon.noname-c con 0x7fe4740085a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.227+0000 7fe4996ed640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe474008a20 0x7fe494121aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.227+0000 7fe4996ed640 10 monclient(hunting): picked mon.noname-b con 0x7fe474008a20 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:49:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.227+0000 7fe4996ed640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.227+0000 7fe4996ed640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe4940a23a0 con 0x7fe474007d40 2024-09-11T01:49:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.227+0000 7fe4996ed640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.227+0000 7fe4996ed640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe4940a2220 con 0x7fe474008a20 2024-09-11T01:49:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.227+0000 7fe4996ed640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:37.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.227+0000 7fe4996ed640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe49409cca0 con 0x7fe4740085a0 2024-09-11T01:49:37.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe4996ed640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:37.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe4996ed640 10 monclient(hunting): authenticate will time out at 1983.670019s 2024-09-11T01:49:37.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe48bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4740085a0 0x7fe494121560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:37.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe4937fe640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe474008a20 msgr2=0x7fe494121aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:49:37.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe48bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4740085a0 0x7fe494121560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45950/0 (socket says 172.21.15.12:45950) 2024-09-11T01:49:37.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe48bfff640 1 -- 172.21.15.12:0/3834586459 learned_addr learned my addr 172.21.15.12:0/3834586459 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:49:37.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe48bfff640 10 monclient(hunting): get_auth_request con 0x7fe4740085a0 auth_method 0 2024-09-11T01:49:37.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe48bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:37.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe48bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:37.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe48bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:37.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe4937fe640 1 --2- 172.21.15.12:0/3834586459 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe474008a20 0x7fe494121aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:37.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe492ffd640 1 --2- 172.21.15.12:0/3834586459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe474007d40 0x7fe494134c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:37.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe492ffd640 10 monclient(hunting): get_auth_request con 0x7fe474007d40 auth_method 0 2024-09-11T01:49:37.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe492ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:37.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe492ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:37.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe492ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:37.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe48bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:37.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe48bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:37.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe48bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:37.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe492ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:37.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe492ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:37.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe492ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:37.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe48bfff640 10 monclient(hunting): handle_auth_done global_id 224153 payload 824 2024-09-11T01:49:37.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe48bfff640 1 -- 172.21.15.12:0/3834586459 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe474008a20 msgr2=0x7fe494121aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:49:37.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe48bfff640 1 --2- 172.21.15.12:0/3834586459 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe474008a20 0x7fe494121aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:37.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe48bfff640 1 -- 172.21.15.12:0/3834586459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe474007d40 msgr2=0x7fe494134c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:37.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe48bfff640 1 --2- 172.21.15.12:0/3834586459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe474007d40 0x7fe494134c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:37.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe48bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:49:37.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe48bfff640 1 monclient: found mon.noname-c 2024-09-11T01:49:37.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe48bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:37.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe48bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:49:37.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.228+0000 7fe48bfff640 1 -- 172.21.15.12:0/3834586459 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe494123020 con 0x7fe4740085a0 2024-09-11T01:49:37.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.231+0000 7fe48bfff640 1 -- 172.21.15.12:0/3834586459 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4740085a0 msgr2=0x7fe494121560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:37.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.231+0000 7fe48bfff640 1 --2- 172.21.15.12:0/3834586459 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4740085a0 0x7fe494121560 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe47c0010e0 tx=0x7fe47c001460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:37.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.231+0000 7fe48b7fe640 1 -- 172.21.15.12:0/3834586459 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47c044030 con 0x7fe4740085a0 2024-09-11T01:49:37.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.231+0000 7fe48b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:37.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.231+0000 7fe48b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:49:37.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.231+0000 7fe48b7fe640 10 monclient: dump: 2024-09-11T01:49:37.239 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:37.240 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:37.240 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.240 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.240 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:37.240 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:37.240 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:37.240 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:37.240 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:37.240 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:37.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.231+0000 7fe48b7fe640 10 monclient: _finish_auth 0 2024-09-11T01:49:37.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.231+0000 7fe48b7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:49:37.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.231+0000 7fe48b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:37.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.231+0000 7fe48b7fe640 1 -- 172.21.15.12:0/3834586459 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe47c02b6d0 con 0x7fe4740085a0 2024-09-11T01:49:37.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.231+0000 7fe48b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:37.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.231+0000 7fe48b7fe640 1 -- 172.21.15.12:0/3834586459 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47c040ab0 con 0x7fe4740085a0 2024-09-11T01:49:37.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.231+0000 7fe48b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:37.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.231+0000 7fe48b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:49:37.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.231+0000 7fe48b7fe640 10 monclient: dump: 2024-09-11T01:49:37.241 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:37.241 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:37.241 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.241 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.241 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:37.241 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:37.241 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:37.241 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:37.241 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:37.241 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:37.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.231+0000 7fe4996ed640 5 monclient: authenticate success, global_id 224153 2024-09-11T01:49:37.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.231+0000 7fe4996ed640 10 monclient: _renew_subs 2024-09-11T01:49:37.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.231+0000 7fe4996ed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:49:37.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.231+0000 7fe4996ed640 1 -- 172.21.15.12:0/3834586459 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe49409d480 con 0x7fe4740085a0 2024-09-11T01:49:37.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.231+0000 7fe491ffb640 4 set_mon_vals no callback set 2024-09-11T01:49:37.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.232+0000 7fe4996ed640 10 monclient: _renew_subs 2024-09-11T01:49:37.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.232+0000 7fe4996ed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:49:37.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.232+0000 7fe4996ed640 1 -- 172.21.15.12:0/3834586459 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe494124ae0 con 0x7fe4740085a0 2024-09-11T01:49:37.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.234+0000 7fe4996ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:37.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.234+0000 7fe4996ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:37.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.234+0000 7fe4996ed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:49:37.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.234+0000 7fe48b7fe640 1 -- 172.21.15.12:0/3834586459 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fe47c040cd0 con 0x7fe4740085a0 2024-09-11T01:49:37.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.234+0000 7fe4996ed640 1 -- 172.21.15.12:0/3834586459 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe49409db70 con 0x7fe4740085a0 2024-09-11T01:49:37.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.234+0000 7fe48b7fe640 1 --2- 172.21.15.12:0/3834586459 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe474042240 0x7fe474044700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:37.243 INFO:tasks.rados.rados.0.smithi071.stdout:3029: writing smithi07134777-32 from 3019476 to 3606780 tid 3 2024-09-11T01:49:37.243 INFO:tasks.rados.rados.0.smithi071.stdout:3030: snap_remove snap 328 2024-09-11T01:49:37.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.234+0000 7fe48b7fe640 1 -- 172.21.15.12:0/3834586459 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(680..680 src has 507..680) ==== 4835+0+0 (secure 0 0 0) 0x7fe47c076030 con 0x7fe4740085a0 2024-09-11T01:49:37.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.234+0000 7fe492ffd640 1 --2- 172.21.15.12:0/3834586459 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe474042240 0x7fe474044700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:37.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.234+0000 7fe492ffd640 10 monclient: get_auth_request con 0x7fe474042240 auth_method 0 2024-09-11T01:49:37.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.237+0000 7fe492ffd640 1 --2- 172.21.15.12:0/3834586459 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe474042240 0x7fe474044700 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe484008010 tx=0x7fe484008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:37.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.241+0000 7fe48b7fe640 1 -- 172.21.15.12:0/3834586459 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe47c03ee30 con 0x7fe4740085a0 2024-09-11T01:49:37.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.242+0000 7fe48b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:37.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.242+0000 7fe48b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:49:37.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.505+0000 7fe4996ed640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:49:37.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.505+0000 7fe4996ed640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:49:37.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.505+0000 7fe4996ed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:49:37.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.505+0000 7fe4996ed640 1 -- 172.21.15.12:0/3834586459 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe49406fb50 con 0x7fe4740085a0 2024-09-11T01:49:37.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.505+0000 7fe48b7fe640 1 -- 172.21.15.12:0/3834586459 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fe47c03d580 con 0x7fe4740085a0 2024-09-11T01:49:37.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.505+0000 7fe48b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:49:37.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.505+0000 7fe48b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:49:37.507 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":192,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:49:37.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.508+0000 7fe4996ed640 1 -- 172.21.15.12:0/3834586459 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe474042240 msgr2=0x7fe474044700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:37.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.508+0000 7fe4996ed640 1 --2- 172.21.15.12:0/3834586459 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe474042240 0x7fe474044700 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe484008010 tx=0x7fe484008040 comp rx=0 tx=0).stop 2024-09-11T01:49:37.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.508+0000 7fe4996ed640 10 monclient: shutdown 2024-09-11T01:49:37.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.508+0000 7fe4996ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:37.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.508+0000 7fe4996ed640 1 -- 172.21.15.12:0/3834586459 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4740085a0 msgr2=0x7fe494121560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:37.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.508+0000 7fe4996ed640 1 --2- 172.21.15.12:0/3834586459 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4740085a0 0x7fe494121560 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe47c0010e0 tx=0x7fe47c001460 comp rx=0 tx=0).stop 2024-09-11T01:49:37.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.508+0000 7fe4996ed640 1 -- 172.21.15.12:0/3834586459 shutdown_connections 2024-09-11T01:49:37.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.508+0000 7fe4996ed640 1 --2- 172.21.15.12:0/3834586459 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe474008a20 0x7fe494121aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:37.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.508+0000 7fe4996ed640 1 --2- 172.21.15.12:0/3834586459 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4740085a0 0x7fe494121560 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:37.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.508+0000 7fe4996ed640 1 --2- 172.21.15.12:0/3834586459 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe474042240 0x7fe474044700 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:37.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.508+0000 7fe4996ed640 1 --2- 172.21.15.12:0/3834586459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe474007d40 0x7fe494134c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:37.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.508+0000 7fe4996ed640 1 -- 172.21.15.12:0/3834586459 >> 172.21.15.12:0/3834586459 conn(0x7fe4940a2d90 msgr2=0x7fe494072500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:37.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.508+0000 7fe4996ed640 1 -- 172.21.15.12:0/3834586459 shutdown_connections 2024-09-11T01:49:37.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.508+0000 7fe4996ed640 1 -- 172.21.15.12:0/3834586459 wait complete. 2024-09-11T01:49:37.520 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:49:37.520 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-11T01:49:37.520 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:49:37.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.620+0000 7f6b53fb4640 10 monclient: get_monmap_and_config 2024-09-11T01:49:37.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.620+0000 7f6b53fb4640 10 monclient: build_initial_monmap 2024-09-11T01:49:37.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.620+0000 7f6b53fb4640 10 monclient: monmap: 2024-09-11T01:49:37.622 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:37.623 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:37.623 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:49:37.621654+0000 2024-09-11T01:49:37.623 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:49:37.621654+0000 2024-09-11T01:49:37.623 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:37.623 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:37.623 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:49:37.623 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:49:37.623 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:49:37.623 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:49:37.623 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:49:37.623 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:37.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.621+0000 7f6b53fb4640 1 Processor -- start 2024-09-11T01:49:37.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.621+0000 7f6b53fb4640 1 -- start start 2024-09-11T01:49:37.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.621+0000 7f6b53fb4640 10 monclient: init 2024-09-11T01:49:37.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.622+0000 7f6b53fb4640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:37.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.622+0000 7f6b53fb4640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-11T01:49:37.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.622+0000 7f6b53fb4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6b4c0a2e20 addr v1:172.21.15.12:6789/0 2024-09-11T01:49:37.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.622+0000 7f6b53fb4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6b4c0a4970 addr v1:172.21.15.71:6789/0 2024-09-11T01:49:37.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.622+0000 7f6b53fb4640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f6b4c06f910 0x7f6b4c06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:37.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.622+0000 7f6b53fb4640 10 monclient(hunting): picked mon.noname-c con 0x7f6b4c06f910 addr v2:172.21.15.71:3300/0 2024-09-11T01:49:37.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.622+0000 7f6b53fb4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b4c06f4f0 con 0x7f6b4c0a2e20 2024-09-11T01:49:37.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.622+0000 7f6b53fb4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b4c0a3910 con 0x7f6b4c0a4970 2024-09-11T01:49:37.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.622+0000 7f6b53fb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:37.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.622+0000 7f6b53fb4640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f6b4c06ef30 con 0x7f6b4c06f910 2024-09-11T01:49:37.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.622+0000 7f6b53fb4640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:37.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.622+0000 7f6b53fb4640 10 monclient(hunting): authenticate will time out at 1984.064285s 2024-09-11T01:49:37.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.622+0000 7f6b52551640 1 -- >> v2:172.21.15.71:3300/0 conn(0x7f6b4c06f910 msgr2=0x7f6b4c06ded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:49:37.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.622+0000 7f6b5154f640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f6b4c0a4970 legacy=0x7f6b4c13d420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:49:37.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.622+0000 7f6b52551640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f6b4c06f910 0x7f6b4c06ded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:37.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.622+0000 7f6b51d50640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6b4c0a2e20 0x7f6b4c0a3200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:32822/0 (socket says 172.21.15.12:32822) 2024-09-11T01:49:37.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.622+0000 7f6b51d50640 1 -- 172.21.15.12:0/4251403869 learned_addr learned my addr 172.21.15.12:0/4251403869 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:49:37.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.624+0000 7f6b51d50640 1 -- 172.21.15.12:0/4251403869 >> v1:172.21.15.12:6789/0 conn(0x7f6b4c0a2e20 legacy=0x7f6b4c0a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:37.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.624+0000 7f6b50d4e640 1 -- 172.21.15.12:0/4251403869 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 378634465 0 0) 0x7f6b4c06f4f0 con 0x7f6b4c0a2e20 2024-09-11T01:49:37.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.624+0000 7f6b50d4e640 10 monclient(hunting): _negotiate 2024-09-11T01:49:37.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.624+0000 7f6b50d4e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:37.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.624+0000 7f6b50d4e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:37.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.624+0000 7f6b50d4e640 10 monclient(hunting): my global_id is 214269 2024-09-11T01:49:37.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.624+0000 7f6b50d4e640 1 -- 172.21.15.12:0/4251403869 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b38003520 con 0x7f6b4c0a2e20 2024-09-11T01:49:37.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.624+0000 7f6b50d4e640 1 -- 172.21.15.12:0/4251403869 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3389927212 0 0) 0x7f6b38003520 con 0x7f6b4c0a2e20 2024-09-11T01:49:37.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 1 -- 172.21.15.12:0/4251403869 >> v2:172.21.15.71:3300/0 conn(0x7f6b4c06f910 msgr2=0x7f6b4c06ded0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:49:37.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 1 --2- 172.21.15.12:0/4251403869 >> v2:172.21.15.71:3300/0 conn(0x7f6b4c06f910 0x7f6b4c06ded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:37.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 1 -- 172.21.15.12:0/4251403869 >> v1:172.21.15.71:6789/0 conn(0x7f6b4c0a4970 legacy=0x7f6b4c13d420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:49:37.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 10 monclient: _finish_hunting 0 2024-09-11T01:49:37.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:49:37.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:37.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:49:37.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 1 -- 172.21.15.12:0/4251403869 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b4c0a3ae0 con 0x7f6b4c0a2e20 2024-09-11T01:49:37.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 10 monclient: _finish_auth 0 2024-09-11T01:49:37.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 10 monclient: _check_auth_tickets 2024-09-11T01:49:37.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:37.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 1 -- 172.21.15.12:0/4251403869 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f6b480019a0 con 0x7f6b4c0a2e20 2024-09-11T01:49:37.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b53fb4640 5 monclient: authenticate success, global_id 214269 2024-09-11T01:49:37.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b53fb4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:49:37.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:37.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:49:37.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 10 monclient: dump: 2024-09-11T01:49:37.632 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:37.632 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:37.632 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.632 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.632 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:37.632 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:37.632 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:37.632 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:37.632 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:37.632 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:37.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:49:37.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:37.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 1 -- 172.21.15.12:0/4251403869 >> v1:172.21.15.12:6789/0 conn(0x7f6b4c0a2e20 legacy=0x7f6b4c0a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:37.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:49:37.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 1 --2- 172.21.15.12:0/4251403869 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6b38003a00 0x7f6b38003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:37.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 10 monclient(hunting): picked mon.c con 0x7f6b38003a00 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:37.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 1 --2- 172.21.15.12:0/4251403869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b38004800 0x7f6b38004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:37.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 10 monclient(hunting): picked mon.a con 0x7f6b38004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:49:37.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 1 --2- 172.21.15.12:0/4251403869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b38005410 0x7f6b3800ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:37.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 10 monclient(hunting): picked mon.b con 0x7f6b38005410 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:49:37.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:37.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 1 -- 172.21.15.12:0/4251403869 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4c06ef30 con 0x7f6b38004800 2024-09-11T01:49:37.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:37.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 1 -- 172.21.15.12:0/4251403869 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3800c350 con 0x7f6b38005410 2024-09-11T01:49:37.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:37.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 1 -- 172.21.15.12:0/4251403869 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6b3800c550 con 0x7f6b38003a00 2024-09-11T01:49:37.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:37.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b50d4e640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:49:37.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b53fb4640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:49:37.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b51d50640 1 --2- 172.21.15.12:0/4251403869 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6b38003a00 0x7f6b38003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:37.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b51d50640 10 monclient(hunting): get_auth_request con 0x7f6b38003a00 auth_method 0 2024-09-11T01:49:37.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b51d50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:37.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b51d50640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:37.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b51d50640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:49:37.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b5154f640 1 --2- 172.21.15.12:0/4251403869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b38004800 0x7f6b38004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:37.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b5154f640 10 monclient(hunting): get_auth_request con 0x7f6b38004800 auth_method 0 2024-09-11T01:49:37.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b5154f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:37.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b5154f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:37.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b5154f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:49:37.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b52551640 1 -- 172.21.15.12:0/4251403869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b38005410 msgr2=0x7f6b3800ba70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:49:37.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b52551640 1 --2- 172.21.15.12:0/4251403869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b38005410 0x7f6b3800ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:37.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b51d50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:37.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b51d50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:37.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b51d50640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:49:37.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b5154f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:37.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b5154f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:37.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b5154f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:49:37.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b5154f640 10 monclient(hunting): handle_auth_done global_id 214269 payload 293 2024-09-11T01:49:37.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b5154f640 1 -- 172.21.15.12:0/4251403869 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6b38003a00 msgr2=0x7f6b38003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:49:37.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b5154f640 1 --2- 172.21.15.12:0/4251403869 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6b38003a00 0x7f6b38003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:37.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b5154f640 1 -- 172.21.15.12:0/4251403869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b38005410 msgr2=0x7f6b3800ba70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:49:37.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b5154f640 1 --2- 172.21.15.12:0/4251403869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b38005410 0x7f6b3800ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:37.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b5154f640 10 monclient: _finish_hunting 0 2024-09-11T01:49:37.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b5154f640 1 monclient: found mon.a 2024-09-11T01:49:37.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b5154f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:37.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b52551640 1 -- 172.21.15.12:0/4251403869 reap_dead start 2024-09-11T01:49:37.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b5154f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:37.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b5154f640 1 -- 172.21.15.12:0/4251403869 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b38003520 con 0x7f6b38004800 2024-09-11T01:49:37.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b5154f640 10 monclient: _finish_auth 0 2024-09-11T01:49:37.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b5154f640 10 monclient: _check_auth_tickets 2024-09-11T01:49:37.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b5154f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:37.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.625+0000 7f6b51d50640 1 --2- 172.21.15.12:0/4251403869 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6b38003a00 0x7f6b38003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:49:37.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.628+0000 7f6b5154f640 1 -- 172.21.15.12:0/4251403869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b38004800 msgr2=0x7f6b38004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:37.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.628+0000 7f6b5154f640 1 --2- 172.21.15.12:0/4251403869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b38004800 0x7f6b38004c80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c009830 tx=0x7f6b38005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23fb44b87accef0a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:37.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.628+0000 7f6b50d4e640 1 -- 172.21.15.12:0/4251403869 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c009430 con 0x7f6b38004800 2024-09-11T01:49:37.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.628+0000 7f6b50d4e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:37.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.628+0000 7f6b50d4e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:37.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.628+0000 7f6b50d4e640 10 monclient: dump: 2024-09-11T01:49:37.637 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:37.637 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:37.637 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.637 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.637 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:37.637 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:37.637 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:37.637 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:37.637 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:37.637 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:37.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.628+0000 7f6b5154f640 1 --2- 172.21.15.12:0/4251403869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b38004800 0x7f6b38004c80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c009830 tx=0x7f6b38005830 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7f6b3c02a850 mon_map magic: 0 2024-09-11T01:49:37.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.628+0000 7f6b53fb4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:49:37.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.628+0000 7f6b50d4e640 1 -- 172.21.15.12:0/4251403869 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b3c02e070 con 0x7f6b38004800 2024-09-11T01:49:37.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.628+0000 7f6b50d4e640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:37.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.628+0000 7f6b53fb4640 10 monclient: get_monmap_and_config success 2024-09-11T01:49:37.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.628+0000 7f6b53fb4640 4 set_mon_vals no callback set 2024-09-11T01:49:37.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.629+0000 7f6b53fb4640 10 monclient: shutdown 2024-09-11T01:49:37.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.629+0000 7f6b53fb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:37.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.629+0000 7f6b53fb4640 1 -- 172.21.15.12:0/4251403869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b38004800 msgr2=0x7f6b38004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:37.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.629+0000 7f6b53fb4640 1 --2- 172.21.15.12:0/4251403869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b38004800 0x7f6b38004c80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c009830 tx=0x7f6b38005830 comp rx=0 tx=0).stop 2024-09-11T01:49:37.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.629+0000 7f6b50d4e640 1 -- 172.21.15.12:0/4251403869 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 0 0 0) 0x7f6b3c02a850 con 0x7f6b38004800 2024-09-11T01:49:37.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.629+0000 7f6b50d4e640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-09-11T01:49:37.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.629+0000 7f6b53fb4640 1 -- 172.21.15.12:0/4251403869 shutdown_connections 2024-09-11T01:49:37.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.629+0000 7f6b53fb4640 1 --2- 172.21.15.12:0/4251403869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b38004800 0x7f6b38004c80 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:37.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.629+0000 7f6b53fb4640 1 -- 172.21.15.12:0/4251403869 >> 172.21.15.12:0/4251403869 conn(0x7f6b4c0a8600 msgr2=0x7f6b4c0a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:37.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.629+0000 7f6b53fb4640 1 -- 172.21.15.12:0/4251403869 shutdown_connections 2024-09-11T01:49:37.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.629+0000 7f6b53fb4640 1 -- 172.21.15.12:0/4251403869 wait complete. 2024-09-11T01:49:37.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.629+0000 7f6b53fb4640 10 monclient: build_initial_monmap 2024-09-11T01:49:37.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.629+0000 7f6b53fb4640 10 monclient: monmap: 2024-09-11T01:49:37.644 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:37.644 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:49:37.644 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:49:37.644 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:49:37.644 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:37.645 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:37.645 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:49:37.645 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:49:37.645 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:49:37.645 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:37.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b53fb4640 1 Processor -- start 2024-09-11T01:49:37.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b53fb4640 1 -- start start 2024-09-11T01:49:37.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b53fb4640 10 monclient: init 2024-09-11T01:49:37.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b53fb4640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:37.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b53fb4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:49:37.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b53fb4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6b38004800 0x7f6b4c1cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:37.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b53fb4640 10 monclient(hunting): picked mon.noname-c con 0x7f6b38004800 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:37.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b53fb4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b4c1cd9b0 0x7f6b4c125a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:37.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b53fb4640 10 monclient(hunting): picked mon.noname-b con 0x7f6b4c1cd9b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:49:37.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b53fb4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c1cde10 0x7f6b4c125fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:37.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b53fb4640 10 monclient(hunting): picked mon.noname-a con 0x7f6b4c1cde10 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:49:37.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b53fb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:37.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b53fb4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4c06e260 con 0x7f6b4c1cde10 2024-09-11T01:49:37.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b53fb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:37.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b53fb4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4c06e0e0 con 0x7f6b4c1cd9b0 2024-09-11T01:49:37.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b53fb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:37.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b53fb4640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6b4c06ec50 con 0x7f6b38004800 2024-09-11T01:49:37.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b53fb4640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:37.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b53fb4640 10 monclient(hunting): authenticate will time out at 1984.072604s 2024-09-11T01:49:37.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b51d50640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6b38004800 0x7f6b4c1cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:37.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b51d50640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6b38004800 0x7f6b4c1cd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45966/0 (socket says 172.21.15.12:45966) 2024-09-11T01:49:37.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b51d50640 1 -- 172.21.15.12:0/417759769 learned_addr learned my addr 172.21.15.12:0/417759769 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:49:37.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b51d50640 10 monclient(hunting): get_auth_request con 0x7f6b38004800 auth_method 0 2024-09-11T01:49:37.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b52551640 1 --2- 172.21.15.12:0/417759769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c1cde10 0x7f6b4c125fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:37.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b51d50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:37.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b51d50640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:37.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b51d50640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:37.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b52551640 10 monclient(hunting): get_auth_request con 0x7f6b4c1cde10 auth_method 0 2024-09-11T01:49:37.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b52551640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:37.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b52551640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:37.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b52551640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:37.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b5154f640 1 -- 172.21.15.12:0/417759769 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b4c1cd9b0 msgr2=0x7f6b4c125a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:49:37.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b5154f640 1 --2- 172.21.15.12:0/417759769 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b4c1cd9b0 0x7f6b4c125a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:37.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b51d50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:37.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b51d50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:37.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b51d50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:37.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b52551640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:37.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b52551640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:37.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b52551640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:37.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b52551640 10 monclient(hunting): handle_auth_done global_id 214272 payload 824 2024-09-11T01:49:37.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b52551640 1 -- 172.21.15.12:0/417759769 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6b38004800 msgr2=0x7f6b4c1cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:37.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b52551640 1 --2- 172.21.15.12:0/417759769 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6b38004800 0x7f6b4c1cd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:37.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b52551640 1 -- 172.21.15.12:0/417759769 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b4c1cd9b0 msgr2=0x7f6b4c125a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:49:37.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b52551640 1 --2- 172.21.15.12:0/417759769 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b4c1cd9b0 0x7f6b4c125a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:37.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b52551640 10 monclient: _finish_hunting 0 2024-09-11T01:49:37.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b52551640 1 monclient: found mon.noname-a 2024-09-11T01:49:37.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b52551640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:37.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b52551640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:49:37.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b52551640 1 -- 172.21.15.12:0/417759769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b4c127540 con 0x7f6b4c1cde10 2024-09-11T01:49:37.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.630+0000 7f6b51d50640 1 --2- 172.21.15.12:0/417759769 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6b38004800 0x7f6b4c1cd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:49:37.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.633+0000 7f6b52551640 1 -- 172.21.15.12:0/417759769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c1cde10 msgr2=0x7f6b4c125fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:37.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.633+0000 7f6b52551640 1 --2- 172.21.15.12:0/417759769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c1cde10 0x7f6b4c125fc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c0a62d0 tx=0x7f6b44009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:37.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.633+0000 7f6b427fc640 1 -- 172.21.15.12:0/417759769 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b44015470 con 0x7f6b4c1cde10 2024-09-11T01:49:37.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.633+0000 7f6b427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:37.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.633+0000 7f6b427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:49:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.633+0000 7f6b427fc640 10 monclient: dump: 2024-09-11T01:49:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:37.650 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.633+0000 7f6b427fc640 10 monclient: _finish_auth 0 2024-09-11T01:49:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.633+0000 7f6b427fc640 10 monclient: _check_auth_tickets 2024-09-11T01:49:37.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.633+0000 7f6b427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:37.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.633+0000 7f6b427fc640 1 -- 172.21.15.12:0/417759769 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b44010070 con 0x7f6b4c1cde10 2024-09-11T01:49:37.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.633+0000 7f6b53fb4640 5 monclient: authenticate success, global_id 214272 2024-09-11T01:49:37.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.633+0000 7f6b427fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:37.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.633+0000 7f6b427fc640 1 -- 172.21.15.12:0/417759769 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b44015850 con 0x7f6b4c1cde10 2024-09-11T01:49:37.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.633+0000 7f6b53fb4640 10 monclient: _renew_subs 2024-09-11T01:49:37.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.633+0000 7f6b53fb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:37.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.633+0000 7f6b53fb4640 1 -- 172.21.15.12:0/417759769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b4c12a280 con 0x7f6b4c1cde10 2024-09-11T01:49:37.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.633+0000 7f6b427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:37.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.633+0000 7f6b427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:37.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.633+0000 7f6b427fc640 10 monclient: dump: 2024-09-11T01:49:37.651 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:37.651 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:37.652 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.652 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.652 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:37.652 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:37.652 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:37.652 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:37.652 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:37.652 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:37.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.633+0000 7f6b43fff640 4 set_mon_vals no callback set 2024-09-11T01:49:37.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.633+0000 7f6b53fb4640 10 monclient: _renew_subs 2024-09-11T01:49:37.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.633+0000 7f6b53fb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:37.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.633+0000 7f6b53fb4640 1 -- 172.21.15.12:0/417759769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b4c1280a0 con 0x7f6b4c1cde10 2024-09-11T01:49:37.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.634+0000 7f6b53fb4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:37.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.634+0000 7f6b53fb4640 1 --2- 172.21.15.12:0/417759769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c126500 0x7f6b4c1268e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:37.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.634+0000 7f6b53fb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:37.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.634+0000 7f6b53fb4640 1 -- 172.21.15.12:0/417759769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4c06a060 con 0x7f6b4c126500 2024-09-11T01:49:37.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.634+0000 7f6b427fc640 1 -- 172.21.15.12:0/417759769 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f6b4401d030 con 0x7f6b4c1cde10 2024-09-11T01:49:37.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.634+0000 7f6b51d50640 1 --2- 172.21.15.12:0/417759769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c126500 0x7f6b4c1268e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:37.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.634+0000 7f6b51d50640 10 monclient: get_auth_request con 0x7f6b4c126500 auth_method 0 2024-09-11T01:49:37.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.634+0000 7f6b51d50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:37.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.634+0000 7f6b51d50640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:37.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.634+0000 7f6b51d50640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:37.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.634+0000 7f6b51d50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:37.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.634+0000 7f6b51d50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:37.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.634+0000 7f6b51d50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:37.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.634+0000 7f6b51d50640 10 monclient(hunting): handle_auth_done global_id 214275 payload 824 2024-09-11T01:49:37.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.634+0000 7f6b427fc640 1 --2- 172.21.15.12:0/417759769 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6b3003e1f0 0x7f6b300406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:37.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.634+0000 7f6b51d50640 1 -- 172.21.15.12:0/417759769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6b4c0a3320 con 0x7f6b4c126500 2024-09-11T01:49:37.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.634+0000 7f6b427fc640 1 -- 172.21.15.12:0/417759769 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(680..680 src has 507..680) ==== 4835+0+0 (secure 0 0 0) 0x7f6b44052b00 con 0x7f6b4c1cde10 2024-09-11T01:49:37.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.635+0000 7f6b5154f640 1 --2- 172.21.15.12:0/417759769 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6b3003e1f0 0x7f6b300406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:37.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.635+0000 7f6b5154f640 10 monclient: get_auth_request con 0x7f6b3003e1f0 auth_method 0 2024-09-11T01:49:37.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.636+0000 7f6b51d50640 1 -- 172.21.15.12:0/417759769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c126500 msgr2=0x7f6b4c1268e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:37.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.636+0000 7f6b51d50640 1 --2- 172.21.15.12:0/417759769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c126500 0x7f6b4c1268e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6b480015b0 tx=0x7f6b48012690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:37.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.637+0000 7f6b427fc640 1 -- 172.21.15.12:0/417759769 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b4800c0e0 con 0x7f6b4c126500 2024-09-11T01:49:37.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.637+0000 7f6b427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:37.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.637+0000 7f6b427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:37.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.637+0000 7f6b427fc640 10 monclient: dump: 2024-09-11T01:49:37.654 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:37.654 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:37.654 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.654 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.655 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:37.655 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:37.655 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:37.655 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:37.655 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:37.655 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:37.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.637+0000 7f6b427fc640 1 -- 172.21.15.12:0/417759769 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6b4801a070 con 0x7f6b4c126500 2024-09-11T01:49:37.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.637+0000 7f6b427fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:37.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.637+0000 7f6b427fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:49:37.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.637+0000 7f6b427fc640 1 -- 172.21.15.12:0/417759769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c126500 msgr2=0x7f6b4c1268e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:37.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.637+0000 7f6b427fc640 1 --2- 172.21.15.12:0/417759769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c126500 0x7f6b4c1268e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6b480015b0 tx=0x7f6b48012690 comp rx=0 tx=0).stop 2024-09-11T01:49:37.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.637+0000 7f6b427fc640 1 -- 172.21.15.12:0/417759769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c126500 msgr2=0x7f6b4c1268e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:49:37.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.637+0000 7f6b427fc640 1 --2- 172.21.15.12:0/417759769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c126500 0x7f6b4c1268e0 secure :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6b480015b0 tx=0x7f6b48012690 comp rx=0 tx=0).stop 2024-09-11T01:49:37.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.637+0000 7f6b5154f640 1 --2- 172.21.15.12:0/417759769 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6b3003e1f0 0x7f6b300406b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c0038b0 tx=0x7f6b3c005000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:37.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.643+0000 7f6b53fb4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:49:37.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.643+0000 7f6b53fb4640 1 --2- 172.21.15.12:0/417759769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c09c5f0 0x7f6b4c09c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:37.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.643+0000 7f6b53fb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:37.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.643+0000 7f6b53fb4640 1 -- 172.21.15.12:0/417759769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4c126e20 con 0x7f6b4c09c5f0 2024-09-11T01:49:37.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.643+0000 7f6b51d50640 1 --2- 172.21.15.12:0/417759769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c09c5f0 0x7f6b4c09c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:37.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.643+0000 7f6b51d50640 10 monclient: get_auth_request con 0x7f6b4c09c5f0 auth_method 0 2024-09-11T01:49:37.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.643+0000 7f6b51d50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:37.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.643+0000 7f6b51d50640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:37.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.643+0000 7f6b51d50640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:37.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.643+0000 7f6b51d50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:37.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.643+0000 7f6b51d50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:37.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.643+0000 7f6b51d50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:37.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.643+0000 7f6b51d50640 10 monclient(hunting): handle_auth_done global_id 214278 payload 824 2024-09-11T01:49:37.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.643+0000 7f6b51d50640 1 -- 172.21.15.12:0/417759769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6b4c09f590 con 0x7f6b4c09c5f0 2024-09-11T01:49:37.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.646+0000 7f6b51d50640 1 -- 172.21.15.12:0/417759769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c09c5f0 msgr2=0x7f6b4c09c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:37.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.646+0000 7f6b51d50640 1 --2- 172.21.15.12:0/417759769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c09c5f0 0x7f6b4c09c9d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6b48019000 tx=0x7f6b48007b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:37.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.646+0000 7f6b427fc640 1 -- 172.21.15.12:0/417759769 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b48015d10 con 0x7f6b4c09c5f0 2024-09-11T01:49:37.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.646+0000 7f6b427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:37.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.646+0000 7f6b427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:37.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.646+0000 7f6b427fc640 10 monclient: dump: 2024-09-11T01:49:37.662 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:37.662 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:37.662 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.662 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.662 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:37.662 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:37.662 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:37.663 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:37.663 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:37.663 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:37.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.646+0000 7f6b427fc640 1 -- 172.21.15.12:0/417759769 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4699 (secure 0 0 0) 0x7f6b48015d10 con 0x7f6b4c09c5f0 2024-09-11T01:49:37.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.646+0000 7f6b427fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:49:37.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.646+0000 7f6b427fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:49:37.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.646+0000 7f6b427fc640 1 -- 172.21.15.12:0/417759769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c09c5f0 msgr2=0x7f6b4c09c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:37.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.646+0000 7f6b427fc640 1 --2- 172.21.15.12:0/417759769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c09c5f0 0x7f6b4c09c9d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6b48019000 tx=0x7f6b48007b60 comp rx=0 tx=0).stop 2024-09-11T01:49:37.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.646+0000 7f6b427fc640 1 -- 172.21.15.12:0/417759769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c09c5f0 msgr2=0x7f6b4c09c9d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:49:37.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.646+0000 7f6b427fc640 1 --2- 172.21.15.12:0/417759769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c09c5f0 0x7f6b4c09c9d0 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6b48019000 tx=0x7f6b48007b60 comp rx=0 tx=0).stop 2024-09-11T01:49:37.663 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:49:37.664 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:49:37.664 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:49:37.664 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:49:37.664 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 192, 2024-09-11T01:49:37.664 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 83846, 2024-09-11T01:49:37.664 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:49:37.664 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:49:37.664 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:49:37.664 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:37.664 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:49:37.664 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:49:37.664 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:49:37.665 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:49:37.665 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:49:37.665 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:49:37.665 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:49:37.665 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:49:37.665 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:49:37.665 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:49:37.665 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:49:37.665 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:49:37.665 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:49:37.665 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:49:37.665 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:49:37.665 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:37.666 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:49:37.666 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:49:37.666 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:49:37.666 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:49:37.666 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:49:37.666 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:49:37.666 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:49:37.666 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:49:37.666 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:49:37.666 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:49:37.666 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:49:37.666 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:49:37.666 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:49:37.666 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:37.667 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:37.667 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:49:37.667 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:49:37.667 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:49:37.667 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:49:37.667 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:49:37.667 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:49:37.667 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:49:37.667 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:49:37.667 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:49:37.667 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:49:37.667 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:49:37.667 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:49:37.667 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:49:37.668 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:49:37.668 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:49:37.668 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:49:37.668 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:49:37.668 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:49:37.668 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:49:37.668 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:49:37.668 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:49:37.668 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:49:37.668 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:49:37.668 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:49:37.668 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:49:37.668 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:49:37.669 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:49:37.669 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:49:37.669 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:37.669 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:49:37.669 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:37.669 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:49:37.669 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:37.669 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:49:37.669 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:49:37.669 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:49:37.669 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:49:37.669 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:37.669 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:49:37.669 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:49:37.670 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:37.670 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:37.670 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:37.670 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:49:37.670 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:49:37.670 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:37.670 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:37.670 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:37.670 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:37.670 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:49:37.670 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:49:37.670 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:49:37.670 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:49:37.670 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:49:37.671 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:37.671 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:37.671 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:49:37.671 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:49:37.671 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:49:37.671 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:49:37.671 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:37.671 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:49:37.671 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:49:37.671 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:37.671 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:37.671 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:37.671 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:49:37.671 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:49:37.672 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:37.672 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:37.672 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:37.672 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:37.672 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:49:37.672 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:49:37.672 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:49:37.672 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:49:37.672 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:49:37.672 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:37.672 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:37.672 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:49:37.672 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:49:37.672 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:49:37.673 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:49:37.673 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:37.673 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:49:37.673 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:49:37.673 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:37.673 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:37.673 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:37.673 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:49:37.673 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:49:37.673 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:37.673 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:37.673 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:37.673 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:37.674 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:49:37.674 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:49:37.674 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:49:37.674 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:49:37.674 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:49:37.674 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:37.674 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:37.674 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:37.674 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:49:37.674 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:49:37.674 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:37.674 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:49:37.674 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:49:37.674 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:49:37.675 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:37.675 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:37.675 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:49:37.675 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:37.675 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:49:37.675 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:49:37.675 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:49:37.675 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:37.675 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:37.675 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:49:37.675 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:37.675 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:49:37.675 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:49:37.675 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:49:37.676 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:37.676 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:37.676 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:37.676 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:49:37.676 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:49:37.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.646+0000 7f6b53fb4640 1 -- 172.21.15.12:0/417759769 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6b3003e1f0 msgr2=0x7f6b300406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:37.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.646+0000 7f6b53fb4640 1 --2- 172.21.15.12:0/417759769 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6b3003e1f0 0x7f6b300406b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c0038b0 tx=0x7f6b3c005000 comp rx=0 tx=0).stop 2024-09-11T01:49:37.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.646+0000 7f6b53fb4640 10 monclient: shutdown 2024-09-11T01:49:37.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.646+0000 7f6b53fb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:37.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.646+0000 7f6b53fb4640 1 -- 172.21.15.12:0/417759769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c1cde10 msgr2=0x7f6b4c125fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:37.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.646+0000 7f6b53fb4640 1 --2- 172.21.15.12:0/417759769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b4c1cde10 0x7f6b4c125fc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c0a62d0 tx=0x7f6b44009e30 comp rx=0 tx=0).stop 2024-09-11T01:49:37.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.646+0000 7f6b52551640 1 -- 172.21.15.12:0/417759769 reap_dead start 2024-09-11T01:49:37.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.646+0000 7f6b52551640 1 -- 172.21.15.12:0/417759769 reap_dead start 2024-09-11T01:49:37.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.647+0000 7f6b53fb4640 1 -- 172.21.15.12:0/417759769 shutdown_connections 2024-09-11T01:49:37.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.647+0000 7f6b53fb4640 1 -- 172.21.15.12:0/417759769 >> 172.21.15.12:0/417759769 conn(0x7f6b4c0a8600 msgr2=0x7f6b4c0a71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:37.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.647+0000 7f6b53fb4640 1 -- 172.21.15.12:0/417759769 shutdown_connections 2024-09-11T01:49:37.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.647+0000 7f6b53fb4640 1 -- 172.21.15.12:0/417759769 wait complete. 2024-09-11T01:49:37.678 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:49:37.678 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T01:49:37.679 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:49:37.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.779+0000 7ff0b1fb4640 10 monclient: get_monmap_and_config 2024-09-11T01:49:37.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.779+0000 7ff0b1fb4640 10 monclient: build_initial_monmap 2024-09-11T01:49:37.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.779+0000 7ff0b1fb4640 10 monclient: monmap: 2024-09-11T01:49:37.781 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:37.781 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:37.781 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:49:37.780613+0000 2024-09-11T01:49:37.782 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:49:37.780613+0000 2024-09-11T01:49:37.782 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:37.782 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:37.782 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:49:37.782 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:49:37.782 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:49:37.782 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:49:37.782 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:49:37.782 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:37.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.780+0000 7ff0b1fb4640 1 Processor -- start 2024-09-11T01:49:37.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.780+0000 7ff0b1fb4640 1 -- start start 2024-09-11T01:49:37.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.780+0000 7ff0b1fb4640 10 monclient: init 2024-09-11T01:49:37.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0b1fb4640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:37.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0b1fb4640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-11T01:49:37.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0b1fb4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac0a1380 0x7ff0ac0a1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:37.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0b1fb4640 10 monclient(hunting): picked mon.noname-b con 0x7ff0ac0a1380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:37.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0b1fb4640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff0ac0a0370 0x7ff0ac135ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:37.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0b1fb4640 10 monclient(hunting): picked mon.noname-c con 0x7ff0ac0a0370 addr v2:172.21.15.71:3300/0 2024-09-11T01:49:37.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0b1fb4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff0ac0a0c90 0x7ff0ac136020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:37.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0b1fb4640 10 monclient(hunting): picked mon.noname-a con 0x7ff0ac0a0c90 addr v2:172.21.15.12:3300/0 2024-09-11T01:49:37.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0b1fb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:37.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0b1fb4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff0ac0a1ca0 con 0x7ff0ac0a0c90 2024-09-11T01:49:37.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0b1fb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:37.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0b1fb4640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7ff0ac0a21c0 con 0x7ff0ac0a0370 2024-09-11T01:49:37.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0b1fb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:37.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0b1fb4640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff0ac0a24a0 con 0x7ff0ac0a1380 2024-09-11T01:49:37.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0b1fb4640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:37.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0b1fb4640 10 monclient(hunting): authenticate will time out at 1984.223198s 2024-09-11T01:49:37.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0aaffd640 1 -- >> v2:172.21.15.71:3300/0 conn(0x7ff0ac0a0370 msgr2=0x7ff0ac135ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:49:37.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0ab7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac0a1380 0x7ff0ac0a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:37.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0abfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff0ac0a0c90 0x7ff0ac136020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0aaffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff0ac0a0370 0x7ff0ac135ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0ab7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac0a1380 0x7ff0ac0a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45980/0 (socket says 172.21.15.12:45980) 2024-09-11T01:49:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0abfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff0ac0a0c90 0x7ff0ac136020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50516/0 (socket says 172.21.15.12:50516) 2024-09-11T01:49:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0ab7fe640 1 -- 172.21.15.12:0/2929607602 learned_addr learned my addr 172.21.15.12:0/2929607602 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:49:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0ab7fe640 10 monclient(hunting): get_auth_request con 0x7ff0ac0a1380 auth_method 0 2024-09-11T01:49:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0abfff640 10 monclient(hunting): get_auth_request con 0x7ff0ac0a0c90 auth_method 0 2024-09-11T01:49:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0abfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:37.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0ab7fe640 10 monclient(hunting): handle_auth_done global_id 224159 payload 274 2024-09-11T01:49:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0ab7fe640 1 -- 172.21.15.12:0/2929607602 >> v2:172.21.15.71:3300/0 conn(0x7ff0ac0a0370 msgr2=0x7ff0ac135ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:49:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0ab7fe640 1 --2- 172.21.15.12:0/2929607602 >> v2:172.21.15.71:3300/0 conn(0x7ff0ac0a0370 0x7ff0ac135ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0ab7fe640 1 -- 172.21.15.12:0/2929607602 >> v2:172.21.15.12:3300/0 conn(0x7ff0ac0a0c90 msgr2=0x7ff0ac136020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:49:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0ab7fe640 1 --2- 172.21.15.12:0/2929607602 >> v2:172.21.15.12:3300/0 conn(0x7ff0ac0a0c90 0x7ff0ac136020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0ab7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:49:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0ab7fe640 1 monclient: found mon.noname-b 2024-09-11T01:49:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:37.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0ab7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:49:37.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.781+0000 7ff0ab7fe640 1 -- 172.21.15.12:0/2929607602 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0ac0a2780 con 0x7ff0ac0a1380 2024-09-11T01:49:37.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.784+0000 7ff0ab7fe640 1 -- 172.21.15.12:0/2929607602 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac0a1380 msgr2=0x7ff0ac0a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:37.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.784+0000 7ff0ab7fe640 1 --2- 172.21.15.12:0/2929607602 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac0a1380 0x7ff0ac0a1760 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac0a23d0 tx=0x7ff09c02a580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e08b071c0cdc7d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:37.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.784+0000 7ff0aa7fc640 1 -- 172.21.15.12:0/2929607602 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09c036070 con 0x7ff0ac0a1380 2024-09-11T01:49:37.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.784+0000 7ff0aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:37.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.784+0000 7ff0aa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:49:37.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.784+0000 7ff0aa7fc640 10 monclient: dump: 2024-09-11T01:49:37.796 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:37.796 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:37.796 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.797 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.797 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:37.797 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:37.797 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:37.797 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:37.797 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:37.797 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:37.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.784+0000 7ff0aa7fc640 10 monclient: _finish_auth 0 2024-09-11T01:49:37.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.784+0000 7ff0aa7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:49:37.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.784+0000 7ff0aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:37.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.784+0000 7ff0b1fb4640 5 monclient: authenticate success, global_id 224159 2024-09-11T01:49:37.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.784+0000 7ff0aa7fc640 1 -- 172.21.15.12:0/2929607602 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff09c007800 con 0x7ff0ac0a1380 2024-09-11T01:49:37.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.784+0000 7ff0b1fb4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:49:37.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.784+0000 7ff0aa7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:37.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.784+0000 7ff0b1fb4640 10 monclient: get_monmap_and_config success 2024-09-11T01:49:37.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.784+0000 7ff0b1fb4640 4 set_mon_vals no callback set 2024-09-11T01:49:37.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.784+0000 7ff0aa7fc640 1 -- 172.21.15.12:0/2929607602 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09c007bd0 con 0x7ff0ac0a1380 2024-09-11T01:49:37.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.784+0000 7ff0aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:37.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.784+0000 7ff0aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:49:37.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.784+0000 7ff0aa7fc640 10 monclient: dump: 2024-09-11T01:49:37.798 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:37.798 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:37.798 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.798 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.798 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:37.798 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:37.799 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:37.799 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:37.799 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:37.799 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:37.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.785+0000 7ff0b1fb4640 10 monclient: shutdown 2024-09-11T01:49:37.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.785+0000 7ff0b1fb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:37.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.785+0000 7ff0b1fb4640 1 -- 172.21.15.12:0/2929607602 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac0a1380 msgr2=0x7ff0ac0a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:37.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.785+0000 7ff0b1fb4640 1 --2- 172.21.15.12:0/2929607602 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac0a1380 0x7ff0ac0a1760 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac0a23d0 tx=0x7ff09c02a580 comp rx=0 tx=0).stop 2024-09-11T01:49:37.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.785+0000 7ff0b1fb4640 1 -- 172.21.15.12:0/2929607602 shutdown_connections 2024-09-11T01:49:37.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.785+0000 7ff0b1fb4640 1 --2- 172.21.15.12:0/2929607602 >> v2:172.21.15.71:3300/0 conn(0x7ff0ac0a0370 0x7ff0ac135ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:37.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.785+0000 7ff0b1fb4640 1 --2- 172.21.15.12:0/2929607602 >> v2:172.21.15.12:3300/0 conn(0x7ff0ac0a0c90 0x7ff0ac136020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:37.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.785+0000 7ff0b1fb4640 1 --2- 172.21.15.12:0/2929607602 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac0a1380 0x7ff0ac0a1760 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:37.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.785+0000 7ff0b1fb4640 1 -- 172.21.15.12:0/2929607602 >> 172.21.15.12:0/2929607602 conn(0x7ff0ac0a2f70 msgr2=0x7ff0ac0a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:37.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.785+0000 7ff0b1fb4640 1 -- 172.21.15.12:0/2929607602 shutdown_connections 2024-09-11T01:49:37.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.785+0000 7ff0b1fb4640 1 -- 172.21.15.12:0/2929607602 wait complete. 2024-09-11T01:49:37.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.785+0000 7ff0b1fb4640 10 monclient: build_initial_monmap 2024-09-11T01:49:37.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.785+0000 7ff0b1fb4640 10 monclient: monmap: 2024-09-11T01:49:37.800 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:37.800 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:49:37.800 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:49:37.800 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:49:37.800 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:37.801 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:37.801 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:49:37.801 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:49:37.801 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:49:37.801 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:37.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.786+0000 7ff0b1fb4640 1 Processor -- start 2024-09-11T01:49:37.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.786+0000 7ff0b1fb4640 1 -- start start 2024-09-11T01:49:37.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.786+0000 7ff0b1fb4640 10 monclient: init 2024-09-11T01:49:37.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.786+0000 7ff0b1fb4640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:37.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.786+0000 7ff0b1fb4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:49:37.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.786+0000 7ff0b1fb4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac0a0370 0x7ff0ac1c8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:37.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.786+0000 7ff0b1fb4640 10 monclient(hunting): picked mon.noname-c con 0x7ff0ac0a0370 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:37.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.786+0000 7ff0b1fb4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0ac0a0c90 0x7ff0ac135980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:37.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.786+0000 7ff0b1fb4640 10 monclient(hunting): picked mon.noname-a con 0x7ff0ac0a0c90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:49:37.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.786+0000 7ff0b1fb4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0ac0a1380 0x7ff0ac135ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:37.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.786+0000 7ff0b1fb4640 10 monclient(hunting): picked mon.noname-b con 0x7ff0ac0a1380 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:49:37.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.786+0000 7ff0b1fb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:37.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.786+0000 7ff0b1fb4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff0ac09ce20 con 0x7ff0ac0a0c90 2024-09-11T01:49:37.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.786+0000 7ff0b1fb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:37.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.786+0000 7ff0b1fb4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff0ac09cca0 con 0x7ff0ac0a1380 2024-09-11T01:49:37.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.786+0000 7ff0b1fb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:37.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.786+0000 7ff0b1fb4640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff0ac136560 con 0x7ff0ac0a0370 2024-09-11T01:49:37.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.786+0000 7ff0b1fb4640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:37.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.786+0000 7ff0b1fb4640 10 monclient(hunting): authenticate will time out at 1984.228913s 2024-09-11T01:49:37.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.786+0000 7ff0ab7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac0a0370 0x7ff0ac1c8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:37.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.786+0000 7ff0aaffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0ac0a0c90 0x7ff0ac135980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:37.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0ab7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac0a0370 0x7ff0ac1c8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45986/0 (socket says 172.21.15.12:45986) 2024-09-11T01:49:37.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0ab7fe640 1 -- 172.21.15.12:0/2081740790 learned_addr learned my addr 172.21.15.12:0/2081740790 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:49:37.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0aaffd640 10 monclient(hunting): get_auth_request con 0x7ff0ac0a0c90 auth_method 0 2024-09-11T01:49:37.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:37.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:37.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:37.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0ab7fe640 10 monclient(hunting): get_auth_request con 0x7ff0ac0a0370 auth_method 0 2024-09-11T01:49:37.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:37.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:37.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:37.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0abfff640 1 -- 172.21.15.12:0/2081740790 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0ac0a1380 msgr2=0x7ff0ac135ec0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:49:37.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0abfff640 1 --2- 172.21.15.12:0/2081740790 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0ac0a1380 0x7ff0ac135ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:37.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0aaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:37.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0aaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:37.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0aaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:37.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:37.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:37.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:37.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0aaffd640 10 monclient(hunting): handle_auth_done global_id 214284 payload 824 2024-09-11T01:49:37.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0aaffd640 1 -- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac0a0370 msgr2=0x7ff0ac1c8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:37.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0aaffd640 1 --2- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac0a0370 0x7ff0ac1c8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:37.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0aaffd640 1 -- 172.21.15.12:0/2081740790 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0ac0a1380 msgr2=0x7ff0ac135ec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:49:37.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0aaffd640 1 --2- 172.21.15.12:0/2081740790 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0ac0a1380 0x7ff0ac135ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:37.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0aaffd640 10 monclient: _finish_hunting 0 2024-09-11T01:49:37.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0aaffd640 1 monclient: found mon.noname-a 2024-09-11T01:49:37.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:37.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0aaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:49:37.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0aaffd640 1 -- 172.21.15.12:0/2081740790 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0ac125080 con 0x7ff0ac0a0c90 2024-09-11T01:49:37.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.787+0000 7ff0ab7fe640 1 --2- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac0a0370 0x7ff0ac1c8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:49:37.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.789+0000 7ff0aaffd640 1 -- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0ac0a0c90 msgr2=0x7ff0ac135980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:37.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.789+0000 7ff0aaffd640 1 --2- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0ac0a0c90 0x7ff0ac135980 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff0a000b520 tx=0x7ff0a000b9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:37.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.789+0000 7ff093fff640 1 -- 172.21.15.12:0/2081740790 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a0003dd0 con 0x7ff0ac0a0c90 2024-09-11T01:49:37.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.789+0000 7ff093fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:37.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.789+0000 7ff093fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:49:37.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.789+0000 7ff093fff640 10 monclient: dump: 2024-09-11T01:49:37.805 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:37.805 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:37.805 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.805 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.805 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:37.805 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:37.806 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:37.806 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:37.806 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:37.806 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:37.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.789+0000 7ff093fff640 10 monclient: _finish_auth 0 2024-09-11T01:49:37.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.789+0000 7ff093fff640 10 monclient: _check_auth_tickets 2024-09-11T01:49:37.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.789+0000 7ff093fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:37.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.789+0000 7ff093fff640 1 -- 172.21.15.12:0/2081740790 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0a0015070 con 0x7ff0ac0a0c90 2024-09-11T01:49:37.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.789+0000 7ff0b1fb4640 5 monclient: authenticate success, global_id 214284 2024-09-11T01:49:37.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.789+0000 7ff0b1fb4640 10 monclient: _renew_subs 2024-09-11T01:49:37.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.789+0000 7ff0b1fb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:37.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.789+0000 7ff0b1fb4640 1 -- 172.21.15.12:0/2081740790 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0ac125360 con 0x7ff0ac0a0c90 2024-09-11T01:49:37.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.789+0000 7ff093fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:37.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.789+0000 7ff093fff640 1 -- 172.21.15.12:0/2081740790 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a0005290 con 0x7ff0ac0a0c90 2024-09-11T01:49:37.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.789+0000 7ff093fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:37.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.789+0000 7ff093fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:37.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.789+0000 7ff093fff640 10 monclient: dump: 2024-09-11T01:49:37.807 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:37.807 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:37.807 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.807 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.807 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:37.807 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:37.807 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:37.807 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:37.807 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:37.807 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:37.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.789+0000 7ff0b1fb4640 10 monclient: _renew_subs 2024-09-11T01:49:37.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.789+0000 7ff0a9ffb640 4 set_mon_vals no callback set 2024-09-11T01:49:37.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.789+0000 7ff0b1fb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:37.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.789+0000 7ff0b1fb4640 1 -- 172.21.15.12:0/2081740790 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0ac09e130 con 0x7ff0ac0a0c90 2024-09-11T01:49:37.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.790+0000 7ff0b1fb4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:37.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.790+0000 7ff0b1fb4640 1 --2- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac121f00 0x7ff0ac02d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:37.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.790+0000 7ff0b1fb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:37.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.790+0000 7ff0b1fb4640 1 -- 172.21.15.12:0/2081740790 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff0ac0a1760 con 0x7ff0ac121f00 2024-09-11T01:49:37.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.790+0000 7ff0ab7fe640 1 --2- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac121f00 0x7ff0ac02d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:37.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.790+0000 7ff0ab7fe640 10 monclient: get_auth_request con 0x7ff0ac121f00 auth_method 0 2024-09-11T01:49:37.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.790+0000 7ff0ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:37.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.790+0000 7ff0ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:37.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.790+0000 7ff0ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:37.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.790+0000 7ff0ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:37.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.790+0000 7ff0ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:37.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.790+0000 7ff0ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:37.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.790+0000 7ff093fff640 1 -- 172.21.15.12:0/2081740790 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff0a0005470 con 0x7ff0ac0a0c90 2024-09-11T01:49:37.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.790+0000 7ff0ab7fe640 10 monclient(hunting): handle_auth_done global_id 224165 payload 824 2024-09-11T01:49:37.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.790+0000 7ff0ab7fe640 1 -- 172.21.15.12:0/2081740790 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff0ac123000 con 0x7ff0ac121f00 2024-09-11T01:49:37.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.791+0000 7ff093fff640 1 --2- 172.21.15.12:0/2081740790 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff09803e240 0x7ff098040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:37.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.791+0000 7ff093fff640 1 -- 172.21.15.12:0/2081740790 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(680..680 src has 507..680) ==== 4835+0+0 (secure 0 0 0) 0x7ff0a0052610 con 0x7ff0ac0a0c90 2024-09-11T01:49:37.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.791+0000 7ff0abfff640 1 --2- 172.21.15.12:0/2081740790 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff09803e240 0x7ff098040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:37.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.791+0000 7ff0abfff640 10 monclient: get_auth_request con 0x7ff09803e240 auth_method 0 2024-09-11T01:49:37.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.793+0000 7ff0ab7fe640 1 -- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac121f00 msgr2=0x7ff0ac02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:37.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.793+0000 7ff0ab7fe640 1 --2- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac121f00 0x7ff0ac02d440 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff09c02a5c0 tx=0x7ff09c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:37.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.793+0000 7ff093fff640 1 -- 172.21.15.12:0/2081740790 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09c042070 con 0x7ff0ac121f00 2024-09-11T01:49:37.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.793+0000 7ff093fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:37.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.793+0000 7ff093fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:49:37.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.793+0000 7ff093fff640 10 monclient: dump: 2024-09-11T01:49:37.810 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:37.810 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:37.810 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.810 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:37.810 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:37.810 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:37.810 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:37.810 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:37.810 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:37.811 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:37.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.793+0000 7ff093fff640 1 -- 172.21.15.12:0/2081740790 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff09c037040 con 0x7ff0ac121f00 2024-09-11T01:49:37.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.793+0000 7ff093fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:37.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.793+0000 7ff093fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:49:37.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.793+0000 7ff093fff640 1 -- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac121f00 msgr2=0x7ff0ac02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:37.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.793+0000 7ff093fff640 1 --2- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac121f00 0x7ff0ac02d440 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff09c02a5c0 tx=0x7ff09c004a40 comp rx=0 tx=0).stop 2024-09-11T01:49:37.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.793+0000 7ff093fff640 1 -- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac121f00 msgr2=0x7ff0ac02d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:49:37.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.793+0000 7ff093fff640 1 --2- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac121f00 0x7ff0ac02d440 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff09c02a5c0 tx=0x7ff09c004a40 comp rx=0 tx=0).stop 2024-09-11T01:49:37.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.793+0000 7ff0abfff640 1 --2- 172.21.15.12:0/2081740790 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff09803e240 0x7ff098040700 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac0a1120 tx=0x7ff0a400f040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:37.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.799+0000 7ff0b1fb4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:49:37.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.799+0000 7ff0b1fb4640 1 --2- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac1222e0 0x7ff0ac02d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:37.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.799+0000 7ff0b1fb4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:37.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.799+0000 7ff0b1fb4640 1 -- 172.21.15.12:0/2081740790 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff0ac11dff0 con 0x7ff0ac1222e0 2024-09-11T01:49:37.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.799+0000 7ff0ab7fe640 1 --2- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac1222e0 0x7ff0ac02d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:37.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.799+0000 7ff0ab7fe640 10 monclient: get_auth_request con 0x7ff0ac1222e0 auth_method 0 2024-09-11T01:49:37.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.799+0000 7ff0ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:37.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.799+0000 7ff0ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:37.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.799+0000 7ff0ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:37.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.799+0000 7ff0ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:37.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.799+0000 7ff0ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:37.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.800+0000 7ff0ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:37.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.800+0000 7ff0ab7fe640 10 monclient(hunting): handle_auth_done global_id 224168 payload 824 2024-09-11T01:49:37.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.800+0000 7ff0ab7fe640 1 -- 172.21.15.12:0/2081740790 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff0ac06d480 con 0x7ff0ac1222e0 2024-09-11T01:49:37.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.800+0000 7ff0ab7fe640 1 -- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac1222e0 msgr2=0x7ff0ac02d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send send error: (32) Broken pipe 2024-09-11T01:49:37.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.800+0000 7ff0ab7fe640 1 --2- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac1222e0 0x7ff0ac02d980 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7ff09c041000 tx=0x7ff09c02f000 comp rx=0 tx=0).write compression request write failed r=-32 ((32) Broken pipe) 2024-09-11T01:49:37.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:37.800+0000 7ff0ab7fe640 1 --2- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac1222e0 0x7ff0ac02d980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:38.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.001+0000 7ff0ab7fe640 1 --2- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac1222e0 0x7ff0ac02d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:38.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.001+0000 7ff0ab7fe640 10 monclient: get_auth_request con 0x7ff0ac1222e0 auth_method 0 2024-09-11T01:49:38.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.001+0000 7ff0ab7fe640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:38.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.001+0000 7ff0ab7fe640 10 monclient: _init_auth method 2 2024-09-11T01:49:38.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.001+0000 7ff0ab7fe640 10 monclient: _init_auth already have auth, reseting 2024-09-11T01:49:38.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.001+0000 7ff0ab7fe640 10 monclient: handle_auth_reply_more payload 9 2024-09-11T01:49:38.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.001+0000 7ff0ab7fe640 10 monclient: handle_auth_reply_more payload_len 9 2024-09-11T01:49:38.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.001+0000 7ff0ab7fe640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-09-11T01:49:38.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.001+0000 7ff0ab7fe640 10 monclient: handle_auth_done global_id 224168 payload 293 2024-09-11T01:49:38.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.003+0000 7ff0ab7fe640 1 -- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac1222e0 msgr2=0x7ff0ac02d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:38.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.003+0000 7ff0ab7fe640 1 --2- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac1222e0 0x7ff0ac02d980 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff09c003fd0 tx=0x7ff09c03e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:38.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.003+0000 7ff093fff640 1 -- 172.21.15.12:0/2081740790 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09c036040 con 0x7ff0ac1222e0 2024-09-11T01:49:38.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.003+0000 7ff093fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:38.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.003+0000 7ff093fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:49:38.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.003+0000 7ff093fff640 10 monclient: dump: 2024-09-11T01:49:38.006 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:38.006 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:38.006 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:38.006 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:38.006 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:38.006 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:38.006 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:38.006 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:38.006 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:38.006 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:38.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.003+0000 7ff093fff640 1 -- 172.21.15.12:0/2081740790 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4864 (secure 0 0 0) 0x7ff09c004840 con 0x7ff0ac1222e0 2024-09-11T01:49:38.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.003+0000 7ff093fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:49:38.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.003+0000 7ff093fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:49:38.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.003+0000 7ff093fff640 1 -- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac1222e0 msgr2=0x7ff0ac02d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:38.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.003+0000 7ff093fff640 1 --2- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac1222e0 0x7ff0ac02d980 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff09c003fd0 tx=0x7ff09c03e000 comp rx=0 tx=0).stop 2024-09-11T01:49:38.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.003+0000 7ff093fff640 1 -- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac1222e0 msgr2=0x7ff0ac02d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:49:38.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.003+0000 7ff093fff640 1 --2- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff0ac1222e0 0x7ff0ac02d980 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff09c003fd0 tx=0x7ff09c03e000 comp rx=0 tx=0).stop 2024-09-11T01:49:38.007 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:49:38.007 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:49:38.008 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:49:38.008 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:49:38.008 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 192, 2024-09-11T01:49:38.008 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 30775, 2024-09-11T01:49:38.008 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:49:38.008 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:49:38.008 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:49:38.008 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:38.008 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:49:38.008 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:49:38.008 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:49:38.008 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:49:38.008 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:49:38.008 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:49:38.009 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:49:38.009 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:49:38.009 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:49:38.009 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:49:38.009 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:49:38.009 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:49:38.009 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:49:38.009 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:49:38.009 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:49:38.009 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:38.009 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:49:38.009 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:49:38.009 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:49:38.010 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:49:38.010 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:49:38.010 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:49:38.010 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:49:38.010 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:49:38.010 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:49:38.010 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:49:38.010 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:49:38.010 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:49:38.010 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:49:38.010 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:38.010 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:38.010 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:49:38.011 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:49:38.011 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:49:38.011 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:49:38.011 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:49:38.011 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:49:38.011 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:49:38.011 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:49:38.011 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:49:38.011 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:49:38.011 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:49:38.011 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:49:38.011 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:49:38.011 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:49:38.011 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:49:38.012 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:49:38.012 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:49:38.012 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:49:38.012 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:49:38.012 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:49:38.012 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:49:38.012 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:49:38.012 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:49:38.012 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:49:38.012 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:49:38.012 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:49:38.012 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:49:38.012 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:49:38.013 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:38.013 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:49:38.013 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:38.013 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:49:38.013 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:38.013 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:49:38.013 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:49:38.013 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:49:38.013 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:49:38.013 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:38.013 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:49:38.013 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:49:38.013 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:38.014 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:38.014 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:38.014 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:49:38.014 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:49:38.014 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:38.014 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:38.014 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:38.014 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:38.014 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:49:38.014 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:49:38.014 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:49:38.014 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:49:38.014 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:49:38.014 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:38.015 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:38.015 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:49:38.015 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:49:38.015 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:49:38.015 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:49:38.015 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:38.015 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:49:38.015 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:49:38.015 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:38.015 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:38.015 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:38.015 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:49:38.015 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:49:38.015 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:38.016 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:38.016 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:38.016 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:38.016 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:49:38.016 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:49:38.016 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:49:38.016 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:49:38.016 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:49:38.016 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:38.016 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:38.016 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:49:38.016 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:49:38.016 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:49:38.017 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:49:38.017 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:38.017 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:49:38.017 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:49:38.017 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:38.017 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:38.017 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:38.017 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:49:38.017 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:49:38.017 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:49:38.017 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:38.017 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:38.017 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:38.017 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:49:38.018 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:49:38.018 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:49:38.018 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:49:38.018 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:49:38.018 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:38.018 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:38.018 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:38.018 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:49:38.018 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:49:38.018 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:38.018 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:49:38.018 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:49:38.018 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:49:38.018 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:38.018 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:38.019 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:49:38.019 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:38.019 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:49:38.019 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:49:38.019 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:49:38.019 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:38.019 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:38.019 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:49:38.019 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:38.019 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:49:38.019 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:49:38.019 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:49:38.019 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:38.019 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:49:38.019 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:49:38.020 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:49:38.020 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:49:38.020 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:49:38.020 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:49:38.020 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:49:38.020 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:49:38.020 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:49:38.020 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:49:38.020 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:49:38.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.004+0000 7ff0b1fb4640 1 -- 172.21.15.12:0/2081740790 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff09803e240 msgr2=0x7ff098040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:38.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.004+0000 7ff0b1fb4640 1 --2- 172.21.15.12:0/2081740790 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff09803e240 0x7ff098040700 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac0a1120 tx=0x7ff0a400f040 comp rx=0 tx=0).stop 2024-09-11T01:49:38.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.004+0000 7ff0b1fb4640 10 monclient: shutdown 2024-09-11T01:49:38.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.004+0000 7ff0b1fb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:38.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.004+0000 7ff0b1fb4640 1 -- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0ac0a0c90 msgr2=0x7ff0ac135980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:38.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.004+0000 7ff0b1fb4640 1 --2- 172.21.15.12:0/2081740790 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff0ac0a0c90 0x7ff0ac135980 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff0a000b520 tx=0x7ff0a000b9e0 comp rx=0 tx=0).stop 2024-09-11T01:49:38.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.004+0000 7ff0abfff640 1 -- 172.21.15.12:0/2081740790 reap_dead start 2024-09-11T01:49:38.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.004+0000 7ff0abfff640 1 -- 172.21.15.12:0/2081740790 reap_dead start 2024-09-11T01:49:38.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.004+0000 7ff0b1fb4640 1 -- 172.21.15.12:0/2081740790 shutdown_connections 2024-09-11T01:49:38.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.004+0000 7ff0b1fb4640 1 -- 172.21.15.12:0/2081740790 >> 172.21.15.12:0/2081740790 conn(0x7ff0ac0a2f70 msgr2=0x7ff0ac09e6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:38.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.004+0000 7ff0b1fb4640 1 -- 172.21.15.12:0/2081740790 shutdown_connections 2024-09-11T01:49:38.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:38.004+0000 7ff0b1fb4640 1 -- 172.21.15.12:0/2081740790 wait complete. 2024-09-11T01:49:38.023 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:49:38.023 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:49:38.936 INFO:tasks.rados.rados.0.smithi071.stdout:3030: done (5 left) 2024-09-11T01:49:38.937 INFO:tasks.rados.rados.0.smithi071.stdout:3031: snap_remove snap 337 2024-09-11T01:49:39.939 INFO:tasks.rados.rados.0.smithi071.stdout:3031: done (5 left) 2024-09-11T01:49:39.939 INFO:tasks.rados.rados.0.smithi071.stdout:3032: read oid 1 snap -1 2024-09-11T01:49:39.939 INFO:tasks.rados.rados.0.smithi071.stdout:3032: expect (ObjNum 1046 snap 337 seq_num 1046) 2024-09-11T01:49:39.939 INFO:tasks.rados.rados.0.smithi071.stdout:3033: read oid 37 snap -1 2024-09-11T01:49:39.940 INFO:tasks.rados.rados.0.smithi071.stdout:3033: expect (ObjNum 1049 snap 339 seq_num 1049) 2024-09-11T01:49:39.940 INFO:tasks.rados.rados.0.smithi071.stdout:3034: write (excl) oid 42 current snap is 342 2024-09-11T01:49:39.940 INFO:tasks.rados.rados.0.smithi071.stdout:3034: seq_num 1065 ranges {699455=429605,1804617=576034,3163252=514929} 2024-09-11T01:49:39.946 INFO:tasks.rados.rados.0.smithi071.stdout:3034: writing smithi07134777-42 from 699455 to 1129060 tid 1 2024-09-11T01:49:39.954 INFO:tasks.rados.rados.0.smithi071.stdout:3034: writing smithi07134777-42 from 1804617 to 2380651 tid 2 2024-09-11T01:49:39.962 INFO:tasks.rados.rados.0.smithi071.stdout:3034: writing smithi07134777-42 from 3163252 to 3678181 tid 3 2024-09-11T01:49:39.962 INFO:tasks.rados.rados.0.smithi071.stdout:3035: copy_from oid 43 from oid 16 current snap is 342 2024-09-11T01:49:39.962 INFO:tasks.rados.rados.0.smithi071.stdout:3026: finishing rollback tid 0 to smithi07134777-38 2024-09-11T01:49:39.962 INFO:tasks.rados.rados.0.smithi071.stdout:3026: finishing rollback tid 1 to smithi07134777-38 2024-09-11T01:49:39.962 INFO:tasks.rados.rados.0.smithi071.stdout:3026: finishing rollback tid 2 to smithi07134777-38 2024-09-11T01:49:39.962 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 2169 (ObjNum 961 snap 305 seq_num 961) dirty exists 2024-09-11T01:49:39.962 INFO:tasks.rados.rados.0.smithi071.stdout:3028: expect (ObjNum 1061 snap 340 seq_num 1061) 2024-09-11T01:49:40.138 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:40.226 INFO:tasks.rados.rados.0.smithi071.stdout:3029: finishing write tid 1 to smithi07134777-32 2024-09-11T01:49:40.227 INFO:tasks.rados.rados.0.smithi071.stdout:3029: finishing write tid 2 to smithi07134777-32 2024-09-11T01:49:40.227 INFO:tasks.rados.rados.0.smithi071.stdout:3027: finishing copy_from racing read to smithi07134777-28 2024-09-11T01:49:40.227 INFO:tasks.rados.rados.0.smithi071.stdout:3027: finishing copy_from to smithi07134777-28 2024-09-11T01:49:40.227 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 2039 (ObjNum 1048 snap 338 seq_num 1048) dirty exists 2024-09-11T01:49:40.227 INFO:tasks.rados.rados.0.smithi071.stdout:3029: finishing write tid 3 to smithi07134777-32 2024-09-11T01:49:40.227 INFO:tasks.rados.rados.0.smithi071.stdout:3029: finishing write tid 4 to smithi07134777-32 2024-09-11T01:49:40.227 INFO:tasks.rados.rados.0.smithi071.stdout:3026: done (8 left) 2024-09-11T01:49:40.227 INFO:tasks.rados.rados.0.smithi071.stdout:3027: done (7 left) 2024-09-11T01:49:40.227 INFO:tasks.rados.rados.0.smithi071.stdout:3028: done (6 left) 2024-09-11T01:49:40.227 INFO:tasks.rados.rados.0.smithi071.stdout:3036: delete oid 28 current snap is 342 2024-09-11T01:49:40.227 INFO:tasks.rados.rados.0.smithi071.stdout:3029: finishing write tid 5 to smithi07134777-32 2024-09-11T01:49:40.227 INFO:tasks.rados.rados.0.smithi071.stdout:3029: oid 32 updating version 0 to 1943 2024-09-11T01:49:40.228 INFO:tasks.rados.rados.0.smithi071.stdout:3029: oid 32 version 1943 is already newer than 1941 2024-09-11T01:49:40.228 INFO:tasks.rados.rados.0.smithi071.stdout:3029: oid 32 version 1943 is already newer than 1942 2024-09-11T01:49:40.228 INFO:tasks.rados.rados.0.smithi071.stdout:3029: oid 32 version 1943 is already newer than 1940 2024-09-11T01:49:40.228 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 1943 (ObjNum 1064 snap 342 seq_num 1064) dirty exists 2024-09-11T01:49:40.228 INFO:tasks.rados.rados.0.smithi071.stdout:3029: left oid 32 (ObjNum 1064 snap 342 seq_num 1064) 2024-09-11T01:49:40.228 INFO:tasks.rados.rados.0.smithi071.stdout:3025: finishing copy_from racing read to smithi07134777-6 2024-09-11T01:49:40.228 INFO:tasks.rados.rados.0.smithi071.stdout:3025: finishing copy_from to smithi07134777-6 2024-09-11T01:49:40.228 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 1774 (ObjNum 1053 snap 339 seq_num 1053) dirty exists 2024-09-11T01:49:40.228 INFO:tasks.rados.rados.0.smithi071.stdout:3032: expect (ObjNum 1046 snap 337 seq_num 1046) 2024-09-11T01:49:40.353 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:40.417 INFO:tasks.rados.rados.0.smithi071.stdout:3033: expect (ObjNum 1049 snap 339 seq_num 1049) 2024-09-11T01:49:40.449 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 804072 2024-09-11T01:49:40.493 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2877036 2024-09-11T01:49:40.501 INFO:tasks.rados.rados.0.smithi071.stdout:3034: finishing write tid 1 to smithi07134777-42 2024-09-11T01:49:40.501 INFO:tasks.rados.rados.0.smithi071.stdout:3034: finishing write tid 2 to smithi07134777-42 2024-09-11T01:49:40.501 INFO:tasks.rados.rados.0.smithi071.stdout:3034: finishing write tid 3 to smithi07134777-42 2024-09-11T01:49:40.501 INFO:tasks.rados.rados.0.smithi071.stdout:3034: finishing write tid 4 to smithi07134777-42 2024-09-11T01:49:40.501 INFO:tasks.rados.rados.0.smithi071.stdout:3034: finishing write tid 5 to smithi07134777-42 2024-09-11T01:49:40.502 INFO:tasks.rados.rados.0.smithi071.stdout:3034: oid 42 updating version 0 to 887 2024-09-11T01:49:40.502 INFO:tasks.rados.rados.0.smithi071.stdout:3034: oid 42 version 887 is already newer than 885 2024-09-11T01:49:40.502 INFO:tasks.rados.rados.0.smithi071.stdout:3034: oid 42 version 887 is already newer than 886 2024-09-11T01:49:40.502 INFO:tasks.rados.rados.0.smithi071.stdout:3034: oid 42 updating version 887 to 888 2024-09-11T01:49:40.502 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 888 (ObjNum 1065 snap 342 seq_num 1065) dirty exists 2024-09-11T01:49:40.502 INFO:tasks.rados.rados.0.smithi071.stdout:3034: left oid 42 (ObjNum 1065 snap 342 seq_num 1065) 2024-09-11T01:49:40.502 INFO:tasks.rados.rados.0.smithi071.stdout:3035: finishing copy_from racing read to smithi07134777-43 2024-09-11T01:49:40.502 INFO:tasks.rados.rados.0.smithi071.stdout:3035: finishing copy_from to smithi07134777-43 2024-09-11T01:49:40.502 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 2048 (ObjNum 1062 snap 340 seq_num 1062) dirty exists 2024-09-11T01:49:40.502 INFO:tasks.rados.rados.0.smithi071.stdout:3025: done (6 left) 2024-09-11T01:49:40.502 INFO:tasks.rados.rados.0.smithi071.stdout:3029: done (5 left) 2024-09-11T01:49:40.502 INFO:tasks.rados.rados.0.smithi071.stdout:3032: done (4 left) 2024-09-11T01:49:40.502 INFO:tasks.rados.rados.0.smithi071.stdout:3033: done (3 left) 2024-09-11T01:49:40.502 INFO:tasks.rados.rados.0.smithi071.stdout:3034: done (2 left) 2024-09-11T01:49:40.503 INFO:tasks.rados.rados.0.smithi071.stdout:3035: done (1 left) 2024-09-11T01:49:40.503 INFO:tasks.rados.rados.0.smithi071.stdout:3036: done (0 left) 2024-09-11T01:49:40.503 INFO:tasks.rados.rados.0.smithi071.stdout:3037: read oid 28 snap -1 2024-09-11T01:49:40.503 INFO:tasks.rados.rados.0.smithi071.stdout:3037: expect deleted 2024-09-11T01:49:40.503 INFO:tasks.rados.rados.0.smithi071.stdout:3038: snap_remove snap 335 2024-09-11T01:49:41.005 INFO:tasks.rados.rados.0.smithi071.stdout:3038: done (1 left) 2024-09-11T01:49:41.006 INFO:tasks.rados.rados.0.smithi071.stdout:3039: read oid 11 snap -1 2024-09-11T01:49:41.006 INFO:tasks.rados.rados.0.smithi071.stdout:3039: expect (ObjNum 987 snap 317 seq_num 987) 2024-09-11T01:49:41.006 INFO:tasks.rados.rados.0.smithi071.stdout:3040: snap_create 2024-09-11T01:49:41.017 INFO:tasks.rados.rados.0.smithi071.stdout:3039: expect (ObjNum 987 snap 317 seq_num 987) 2024-09-11T01:49:41.046 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:41.074 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 23873 2024-09-11T01:49:42.013 INFO:tasks.rados.rados.0.smithi071.stdout:3037: done (2 left) 2024-09-11T01:49:42.013 INFO:tasks.rados.rados.0.smithi071.stdout:3039: done (1 left) 2024-09-11T01:49:42.013 INFO:tasks.rados.rados.0.smithi071.stdout:3040: done (0 left) 2024-09-11T01:49:42.014 INFO:tasks.rados.rados.0.smithi071.stdout:3041: copy_from oid 34 from oid 14 current snap is 343 2024-09-11T01:49:42.014 INFO:tasks.rados.rados.0.smithi071.stdout:3042: copy_from oid 46 from oid 19 current snap is 343 2024-09-11T01:49:42.014 INFO:tasks.rados.rados.0.smithi071.stdout:3043: snap_create 2024-09-11T01:49:42.028 INFO:tasks.rados.rados.0.smithi071.stdout:3042: finishing copy_from racing read to smithi07134777-46 2024-09-11T01:49:42.041 INFO:tasks.rados.rados.0.smithi071.stdout:3041: finishing copy_from racing read to smithi07134777-34 2024-09-11T01:49:42.045 INFO:tasks.rados.rados.0.smithi071.stdout:3042: finishing copy_from to smithi07134777-46 2024-09-11T01:49:42.045 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 1774 (ObjNum 1053 snap 339 seq_num 1053) dirty exists 2024-09-11T01:49:42.045 INFO:tasks.rados.rados.0.smithi071.stdout:3041: finishing copy_from to smithi07134777-34 2024-09-11T01:49:42.045 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 2153 (ObjNum 942 snap 296 seq_num 942) dirty exists 2024-09-11T01:49:43.020 INFO:tasks.rados.rados.0.smithi071.stdout:3041: done (2 left) 2024-09-11T01:49:43.020 INFO:tasks.rados.rados.0.smithi071.stdout:3042: done (1 left) 2024-09-11T01:49:43.020 INFO:tasks.rados.rados.0.smithi071.stdout:3043: done (0 left) 2024-09-11T01:49:43.020 INFO:tasks.rados.rados.0.smithi071.stdout:3044: snap_remove snap 327 2024-09-11T01:49:43.831 INFO:tasks.rados.rados.0.smithi071.stdout:3044: done (0 left) 2024-09-11T01:49:43.831 INFO:tasks.rados.rados.0.smithi071.stdout:3045: read oid 32 snap -1 2024-09-11T01:49:43.831 INFO:tasks.rados.rados.0.smithi071.stdout:3045: expect (ObjNum 1064 snap 342 seq_num 1064) 2024-09-11T01:49:43.831 INFO:tasks.rados.rados.0.smithi071.stdout:3046: write (excl) oid 19 current snap is 344 2024-09-11T01:49:43.831 INFO:tasks.rados.rados.0.smithi071.stdout:3046: seq_num 1067 ranges {733069=580103,1921581=579703,3218098=537473} 2024-09-11T01:49:43.839 INFO:tasks.rados.rados.0.smithi071.stdout:3046: writing smithi07134777-19 from 733069 to 1313172 tid 1 2024-09-11T01:49:43.845 INFO:tasks.rados.rados.0.smithi071.stdout:3046: writing smithi07134777-19 from 1921581 to 2501284 tid 2 2024-09-11T01:49:43.854 INFO:tasks.rados.rados.0.smithi071.stdout:3046: writing smithi07134777-19 from 3218098 to 3755571 tid 3 2024-09-11T01:49:43.854 INFO:tasks.rados.rados.0.smithi071.stdout:3047: read oid 36 snap -1 2024-09-11T01:49:43.854 INFO:tasks.rados.rados.0.smithi071.stdout:3047: expect (ObjNum 1063 snap 341 seq_num 1063) 2024-09-11T01:49:43.854 INFO:tasks.rados.rados.0.smithi071.stdout:3048: copy_from oid 49 from oid 39 current snap is 344 2024-09-11T01:49:43.854 INFO:tasks.rados.rados.0.smithi071.stdout:3049: rollback oid 30 current snap is 344 2024-09-11T01:49:43.854 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 30 to 324 2024-09-11T01:49:43.854 INFO:tasks.rados.rados.0.smithi071.stdout:3050: read oid 31 snap -1 2024-09-11T01:49:43.854 INFO:tasks.rados.rados.0.smithi071.stdout:3050: expect (ObjNum 1055 snap 340 seq_num 1055) 2024-09-11T01:49:43.854 INFO:tasks.rados.rados.0.smithi071.stdout:3051: rollback oid 28 current snap is 344 2024-09-11T01:49:43.855 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 28 to 310 2024-09-11T01:49:43.855 INFO:tasks.rados.rados.0.smithi071.stdout:3052: read oid 21 snap -1 2024-09-11T01:49:43.855 INFO:tasks.rados.rados.0.smithi071.stdout:3052: expect (ObjNum 1048 snap 338 seq_num 1048) 2024-09-11T01:49:43.855 INFO:tasks.rados.rados.0.smithi071.stdout:3053: rollback oid 33 current snap is 344 2024-09-11T01:49:43.855 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 33 to 236 2024-09-11T01:49:43.855 INFO:tasks.rados.rados.0.smithi071.stdout:3054: snap_create 2024-09-11T01:49:43.855 INFO:tasks.rados.rados.0.smithi071.stdout:3048: finishing copy_from to smithi07134777-49 2024-09-11T01:49:43.855 INFO:tasks.rados.rados.0.smithi071.stdout:3048: got expected ENOENT (src dne) 2024-09-11T01:49:43.856 INFO:tasks.rados.rados.0.smithi071.stdout:3048: finishing copy_from racing read to smithi07134777-49 2024-09-11T01:49:43.857 INFO:tasks.rados.rados.0.smithi071.stdout:3045: expect (ObjNum 1064 snap 342 seq_num 1064) 2024-09-11T01:49:43.857 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:44.151 INFO:tasks.rados.rados.0.smithi071.stdout:3049: finishing rollback tid 1 to smithi07134777-30 2024-09-11T01:49:44.151 INFO:tasks.rados.rados.0.smithi071.stdout:3050: expect (ObjNum 1055 snap 340 seq_num 1055) 2024-09-11T01:49:44.154 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 703009 2024-09-11T01:49:44.301 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:44.379 INFO:tasks.rados.rados.0.smithi071.stdout:3049: finishing rollback tid 2 to smithi07134777-30 2024-09-11T01:49:44.379 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 1314 (ObjNum 1003 snap 322 seq_num 1003) dirty exists 2024-09-11T01:49:44.379 INFO:tasks.rados.rados.0.smithi071.stdout:3052: expect (ObjNum 1048 snap 338 seq_num 1048) 2024-09-11T01:49:44.422 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:44.474 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:44.526 INFO:tasks.rados.rados.0.smithi071.stdout:3046: finishing write tid 1 to smithi07134777-19 2024-09-11T01:49:44.526 INFO:tasks.rados.rados.0.smithi071.stdout:3051: finishing rollback tid 1 to smithi07134777-28 2024-09-11T01:49:44.526 INFO:tasks.rados.rados.0.smithi071.stdout:3051: finishing rollback tid 2 to smithi07134777-28 2024-09-11T01:49:44.526 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 2053 (ObjNum 965 snap 305 seq_num 965) dirty exists 2024-09-11T01:49:44.526 INFO:tasks.rados.rados.0.smithi071.stdout:3047: expect (ObjNum 1063 snap 341 seq_num 1063) 2024-09-11T01:49:44.551 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1602337 2024-09-11T01:49:44.676 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 815464 2024-09-11T01:49:44.747 INFO:tasks.rados.rados.0.smithi071.stdout:3046: finishing write tid 2 to smithi07134777-19 2024-09-11T01:49:44.747 INFO:tasks.rados.rados.0.smithi071.stdout:3046: finishing write tid 3 to smithi07134777-19 2024-09-11T01:49:44.747 INFO:tasks.rados.rados.0.smithi071.stdout:3046: finishing write tid 4 to smithi07134777-19 2024-09-11T01:49:44.747 INFO:tasks.rados.rados.0.smithi071.stdout:3046: finishing write tid 5 to smithi07134777-19 2024-09-11T01:49:44.747 INFO:tasks.rados.rados.0.smithi071.stdout:3046: oid 19 updating version 0 to 1780 2024-09-11T01:49:44.747 INFO:tasks.rados.rados.0.smithi071.stdout:3046: oid 19 updating version 1780 to 1781 2024-09-11T01:49:44.747 INFO:tasks.rados.rados.0.smithi071.stdout:3046: oid 19 version 1781 is already newer than 1779 2024-09-11T01:49:44.747 INFO:tasks.rados.rados.0.smithi071.stdout:3046: oid 19 updating version 1781 to 1782 2024-09-11T01:49:44.747 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 1782 (ObjNum 1067 snap 344 seq_num 1067) dirty exists 2024-09-11T01:49:44.747 INFO:tasks.rados.rados.0.smithi071.stdout:3046: left oid 19 (ObjNum 1067 snap 344 seq_num 1067) 2024-09-11T01:49:44.747 INFO:tasks.rados.rados.0.smithi071.stdout:3053: finishing rollback tid 0 to smithi07134777-33 2024-09-11T01:49:44.747 INFO:tasks.rados.rados.0.smithi071.stdout:3053: finishing rollback tid 1 to smithi07134777-33 2024-09-11T01:49:44.748 INFO:tasks.rados.rados.0.smithi071.stdout:3053: finishing rollback tid 2 to smithi07134777-33 2024-09-11T01:49:44.748 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 2156 (ObjNum 250 snap 63 seq_num 250) dirty exists 2024-09-11T01:49:44.839 INFO:tasks.rados.rados.0.smithi071.stdout:3045: done (9 left) 2024-09-11T01:49:44.839 INFO:tasks.rados.rados.0.smithi071.stdout:3046: done (8 left) 2024-09-11T01:49:44.839 INFO:tasks.rados.rados.0.smithi071.stdout:3047: done (7 left) 2024-09-11T01:49:44.839 INFO:tasks.rados.rados.0.smithi071.stdout:3048: done (6 left) 2024-09-11T01:49:44.839 INFO:tasks.rados.rados.0.smithi071.stdout:3049: done (5 left) 2024-09-11T01:49:44.839 INFO:tasks.rados.rados.0.smithi071.stdout:3050: done (4 left) 2024-09-11T01:49:44.839 INFO:tasks.rados.rados.0.smithi071.stdout:3051: done (3 left) 2024-09-11T01:49:44.839 INFO:tasks.rados.rados.0.smithi071.stdout:3052: done (2 left) 2024-09-11T01:49:44.839 INFO:tasks.rados.rados.0.smithi071.stdout:3053: done (1 left) 2024-09-11T01:49:44.839 INFO:tasks.rados.rados.0.smithi071.stdout:3054: done (0 left) 2024-09-11T01:49:44.839 INFO:tasks.rados.rados.0.smithi071.stdout:3055: delete oid 4 current snap is 345 2024-09-11T01:49:44.844 INFO:tasks.rados.rados.0.smithi071.stdout:3055: done (0 left) 2024-09-11T01:49:44.844 INFO:tasks.rados.rados.0.smithi071.stdout:3056: write (excl) oid 34 current snap is 345 2024-09-11T01:49:44.844 INFO:tasks.rados.rados.0.smithi071.stdout:3056: seq_num 1069 ranges {766683=730601,1922192=583372,3156591=676370} 2024-09-11T01:49:44.852 INFO:tasks.rados.rados.0.smithi071.stdout:3056: writing smithi07134777-34 from 766683 to 1497284 tid 1 2024-09-11T01:49:44.857 INFO:tasks.rados.rados.0.smithi071.stdout:3056: writing smithi07134777-34 from 1922192 to 2505564 tid 2 2024-09-11T01:49:44.865 INFO:tasks.rados.rados.0.smithi071.stdout:3056: writing smithi07134777-34 from 3156591 to 3832961 tid 3 2024-09-11T01:49:44.865 INFO:tasks.rados.rados.0.smithi071.stdout:3057: write oid 37 current snap is 345 2024-09-11T01:49:44.865 INFO:tasks.rados.rados.0.smithi071.stdout:3056: finishing write tid 1 to smithi07134777-34 2024-09-11T01:49:44.865 INFO:tasks.rados.rados.0.smithi071.stdout:3056: finishing write tid 2 to smithi07134777-34 2024-09-11T01:49:44.865 INFO:tasks.rados.rados.0.smithi071.stdout:3057: seq_num 1070 ranges {783490=405850,1980674=727030,3184014=487642} 2024-09-11T01:49:44.871 INFO:tasks.rados.rados.0.smithi071.stdout:3057: writing smithi07134777-37 from 783490 to 1189340 tid 1 2024-09-11T01:49:44.879 INFO:tasks.rados.rados.0.smithi071.stdout:3057: writing smithi07134777-37 from 1980674 to 2707704 tid 2 2024-09-11T01:49:44.884 INFO:tasks.rados.rados.0.smithi071.stdout:3057: writing smithi07134777-37 from 3184014 to 3671656 tid 3 2024-09-11T01:49:44.885 INFO:tasks.rados.rados.0.smithi071.stdout:3058: snap_remove snap 313 2024-09-11T01:49:45.843 INFO:tasks.rados.rados.0.smithi071.stdout:3058: done (2 left) 2024-09-11T01:49:45.843 INFO:tasks.rados.rados.0.smithi071.stdout:3059: read oid 47 snap -1 2024-09-11T01:49:45.843 INFO:tasks.rados.rados.0.smithi071.stdout:3059: expect deleted 2024-09-11T01:49:45.843 INFO:tasks.rados.rados.0.smithi071.stdout:3060: read oid 25 snap -1 2024-09-11T01:49:45.843 INFO:tasks.rados.rados.0.smithi071.stdout:3060: expect (ObjNum 1023 snap 328 seq_num 1023) 2024-09-11T01:49:45.843 INFO:tasks.rados.rados.0.smithi071.stdout:3061: delete oid 44 current snap is 345 2024-09-11T01:49:45.843 INFO:tasks.rados.rados.0.smithi071.stdout:3056: finishing write tid 3 to smithi07134777-34 2024-09-11T01:49:45.843 INFO:tasks.rados.rados.0.smithi071.stdout:3056: finishing write tid 4 to smithi07134777-34 2024-09-11T01:49:45.844 INFO:tasks.rados.rados.0.smithi071.stdout:3056: finishing write tid 5 to smithi07134777-34 2024-09-11T01:49:45.844 INFO:tasks.rados.rados.0.smithi071.stdout:3056: oid 34 updating version 0 to 2157 2024-09-11T01:49:45.844 INFO:tasks.rados.rados.0.smithi071.stdout:3056: oid 34 updating version 2157 to 2159 2024-09-11T01:49:45.844 INFO:tasks.rados.rados.0.smithi071.stdout:3056: oid 34 updating version 2159 to 2160 2024-09-11T01:49:45.844 INFO:tasks.rados.rados.0.smithi071.stdout:3056: oid 34 version 2160 is already newer than 2158 2024-09-11T01:49:45.844 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 2160 (ObjNum 1069 snap 345 seq_num 1069) dirty exists 2024-09-11T01:49:45.844 INFO:tasks.rados.rados.0.smithi071.stdout:3056: left oid 34 (ObjNum 1069 snap 345 seq_num 1069) 2024-09-11T01:49:45.844 INFO:tasks.rados.rados.0.smithi071.stdout:3057: finishing write tid 1 to smithi07134777-37 2024-09-11T01:49:45.844 INFO:tasks.rados.rados.0.smithi071.stdout:3057: finishing write tid 2 to smithi07134777-37 2024-09-11T01:49:45.844 INFO:tasks.rados.rados.0.smithi071.stdout:3057: finishing write tid 3 to smithi07134777-37 2024-09-11T01:49:45.844 INFO:tasks.rados.rados.0.smithi071.stdout:3057: finishing write tid 4 to smithi07134777-37 2024-09-11T01:49:45.844 INFO:tasks.rados.rados.0.smithi071.stdout:3057: finishing write tid 5 to smithi07134777-37 2024-09-11T01:49:45.844 INFO:tasks.rados.rados.0.smithi071.stdout:3057: oid 37 updating version 0 to 1944 2024-09-11T01:49:45.845 INFO:tasks.rados.rados.0.smithi071.stdout:3057: oid 37 updating version 1944 to 1945 2024-09-11T01:49:45.845 INFO:tasks.rados.rados.0.smithi071.stdout:3057: oid 37 updating version 1945 to 1947 2024-09-11T01:49:45.845 INFO:tasks.rados.rados.0.smithi071.stdout:3057: oid 37 version 1947 is already newer than 1946 2024-09-11T01:49:45.845 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 1947 (ObjNum 1070 snap 345 seq_num 1070) dirty exists 2024-09-11T01:49:45.845 INFO:tasks.rados.rados.0.smithi071.stdout:3057: left oid 37 (ObjNum 1070 snap 345 seq_num 1070) 2024-09-11T01:49:45.848 INFO:tasks.rados.rados.0.smithi071.stdout:3056: done (4 left) 2024-09-11T01:49:45.849 INFO:tasks.rados.rados.0.smithi071.stdout:3057: done (3 left) 2024-09-11T01:49:45.849 INFO:tasks.rados.rados.0.smithi071.stdout:3059: done (2 left) 2024-09-11T01:49:45.849 INFO:tasks.rados.rados.0.smithi071.stdout:3061: done (1 left) 2024-09-11T01:49:45.849 INFO:tasks.rados.rados.0.smithi071.stdout:3062: read oid 30 snap 300 2024-09-11T01:49:45.849 INFO:tasks.rados.rados.0.smithi071.stdout:3062: expect deleted 2024-09-11T01:49:45.849 INFO:tasks.rados.rados.0.smithi071.stdout:3063: rollback oid 47 current snap is 345 2024-09-11T01:49:45.849 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 47 to 317 2024-09-11T01:49:45.849 INFO:tasks.rados.rados.0.smithi071.stdout:3064: delete oid 39 current snap is 345 2024-09-11T01:49:45.852 INFO:tasks.rados.rados.0.smithi071.stdout:3063: finishing rollback tid 1 to smithi07134777-47 2024-09-11T01:49:45.852 INFO:tasks.rados.rados.0.smithi071.stdout:3062: done (3 left) 2024-09-11T01:49:45.852 INFO:tasks.rados.rados.0.smithi071.stdout:3064: done (2 left) 2024-09-11T01:49:45.852 INFO:tasks.rados.rados.0.smithi071.stdout:3065: snap_create 2024-09-11T01:49:45.854 INFO:tasks.rados.rados.0.smithi071.stdout:3063: finishing rollback tid 2 to smithi07134777-47 2024-09-11T01:49:45.855 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 1794 (ObjNum 715 snap 221 seq_num 715) dirty exists 2024-09-11T01:49:45.858 INFO:tasks.rados.rados.0.smithi071.stdout:3060: expect (ObjNum 1023 snap 328 seq_num 1023) 2024-09-11T01:49:45.861 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 574110 2024-09-11T01:49:46.848 INFO:tasks.rados.rados.0.smithi071.stdout:3060: done (2 left) 2024-09-11T01:49:46.848 INFO:tasks.rados.rados.0.smithi071.stdout:3063: done (1 left) 2024-09-11T01:49:46.848 INFO:tasks.rados.rados.0.smithi071.stdout:3065: done (0 left) 2024-09-11T01:49:46.849 INFO:tasks.rados.rados.0.smithi071.stdout:3066: write oid 32 current snap is 346 2024-09-11T01:49:46.849 INFO:tasks.rados.rados.0.smithi071.stdout:3066: seq_num 1073 ranges {433911=747950,1756120=277791} 2024-09-11T01:49:46.854 INFO:tasks.rados.rados.0.smithi071.stdout:3066: writing smithi07134777-32 from 433911 to 1181861 tid 1 2024-09-11T01:49:46.860 INFO:tasks.rados.rados.0.smithi071.stdout:3066: writing smithi07134777-32 from 1756120 to 2033911 tid 2 2024-09-11T01:49:46.860 INFO:tasks.rados.rados.0.smithi071.stdout:3067: write (excl) oid 19 current snap is 346 2024-09-11T01:49:46.860 INFO:tasks.rados.rados.0.smithi071.stdout:3067: seq_num 1074 ranges {450718=423199,1414602=636116} 2024-09-11T01:49:46.863 INFO:tasks.rados.rados.0.smithi071.stdout:3067: writing smithi07134777-19 from 450718 to 873917 tid 1 2024-09-11T01:49:46.870 INFO:tasks.rados.rados.0.smithi071.stdout:3067: writing smithi07134777-19 from 1414602 to 2050718 tid 2 2024-09-11T01:49:46.870 INFO:tasks.rados.rados.0.smithi071.stdout:3068: write (excl) oid 37 current snap is 346 2024-09-11T01:49:46.870 INFO:tasks.rados.rados.0.smithi071.stdout:3068: seq_num 1075 ranges {467525=498448,1473084=594379} 2024-09-11T01:49:46.875 INFO:tasks.rados.rados.0.smithi071.stdout:3068: writing smithi07134777-37 from 467525 to 965973 tid 1 2024-09-11T01:49:46.882 INFO:tasks.rados.rados.0.smithi071.stdout:3068: writing smithi07134777-37 from 1473084 to 2067463 tid 2 2024-09-11T01:49:46.882 INFO:tasks.rados.rados.0.smithi071.stdout:3069: delete oid 14 current snap is 346 2024-09-11T01:49:46.882 INFO:tasks.rados.rados.0.smithi071.stdout:3067: finishing write tid 1 to smithi07134777-19 2024-09-11T01:49:46.882 INFO:tasks.rados.rados.0.smithi071.stdout:3066: finishing write tid 1 to smithi07134777-32 2024-09-11T01:49:46.882 INFO:tasks.rados.rados.0.smithi071.stdout:3067: finishing write tid 2 to smithi07134777-19 2024-09-11T01:49:46.882 INFO:tasks.rados.rados.0.smithi071.stdout:3066: finishing write tid 2 to smithi07134777-32 2024-09-11T01:49:46.882 INFO:tasks.rados.rados.0.smithi071.stdout:3066: finishing write tid 3 to smithi07134777-32 2024-09-11T01:49:46.882 INFO:tasks.rados.rados.0.smithi071.stdout:3066: finishing write tid 4 to smithi07134777-32 2024-09-11T01:49:46.882 INFO:tasks.rados.rados.0.smithi071.stdout:3066: oid 32 updating version 0 to 1950 2024-09-11T01:49:46.882 INFO:tasks.rados.rados.0.smithi071.stdout:3066: oid 32 version 1950 is already newer than 1949 2024-09-11T01:49:46.882 INFO:tasks.rados.rados.0.smithi071.stdout:3066: oid 32 version 1950 is already newer than 1948 2024-09-11T01:49:46.882 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 1950 (ObjNum 1073 snap 346 seq_num 1073) dirty exists 2024-09-11T01:49:46.882 INFO:tasks.rados.rados.0.smithi071.stdout:3066: left oid 32 (ObjNum 1073 snap 346 seq_num 1073) 2024-09-11T01:49:46.883 INFO:tasks.rados.rados.0.smithi071.stdout:3067: finishing write tid 3 to smithi07134777-19 2024-09-11T01:49:46.883 INFO:tasks.rados.rados.0.smithi071.stdout:3068: finishing write tid 1 to smithi07134777-37 2024-09-11T01:49:46.883 INFO:tasks.rados.rados.0.smithi071.stdout:3067: finishing write tid 4 to smithi07134777-19 2024-09-11T01:49:46.883 INFO:tasks.rados.rados.0.smithi071.stdout:3067: oid 19 updating version 0 to 1783 2024-09-11T01:49:46.883 INFO:tasks.rados.rados.0.smithi071.stdout:3067: oid 19 updating version 1783 to 1784 2024-09-11T01:49:46.883 INFO:tasks.rados.rados.0.smithi071.stdout:3067: oid 19 updating version 1784 to 1785 2024-09-11T01:49:46.883 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 1785 (ObjNum 1074 snap 346 seq_num 1074) dirty exists 2024-09-11T01:49:46.883 INFO:tasks.rados.rados.0.smithi071.stdout:3067: left oid 19 (ObjNum 1074 snap 346 seq_num 1074) 2024-09-11T01:49:46.886 INFO:tasks.rados.rados.0.smithi071.stdout:3068: finishing write tid 2 to smithi07134777-37 2024-09-11T01:49:46.886 INFO:tasks.rados.rados.0.smithi071.stdout:3068: finishing write tid 3 to smithi07134777-37 2024-09-11T01:49:46.886 INFO:tasks.rados.rados.0.smithi071.stdout:3068: finishing write tid 4 to smithi07134777-37 2024-09-11T01:49:46.886 INFO:tasks.rados.rados.0.smithi071.stdout:3068: oid 37 updating version 0 to 1952 2024-09-11T01:49:46.886 INFO:tasks.rados.rados.0.smithi071.stdout:3068: oid 37 version 1952 is already newer than 1951 2024-09-11T01:49:46.886 INFO:tasks.rados.rados.0.smithi071.stdout:3068: oid 37 updating version 1952 to 1953 2024-09-11T01:49:46.886 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 1953 (ObjNum 1075 snap 346 seq_num 1075) dirty exists 2024-09-11T01:49:46.887 INFO:tasks.rados.rados.0.smithi071.stdout:3068: left oid 37 (ObjNum 1075 snap 346 seq_num 1075) 2024-09-11T01:49:46.888 INFO:tasks.rados.rados.0.smithi071.stdout:3066: done (3 left) 2024-09-11T01:49:46.888 INFO:tasks.rados.rados.0.smithi071.stdout:3067: done (2 left) 2024-09-11T01:49:46.888 INFO:tasks.rados.rados.0.smithi071.stdout:3068: done (1 left) 2024-09-11T01:49:46.888 INFO:tasks.rados.rados.0.smithi071.stdout:3069: done (0 left) 2024-09-11T01:49:46.888 INFO:tasks.rados.rados.0.smithi071.stdout:3070: copy_from oid 8 from oid 9 current snap is 346 2024-09-11T01:49:46.888 INFO:tasks.rados.rados.0.smithi071.stdout:3071: delete oid 11 current snap is 346 2024-09-11T01:49:46.889 INFO:tasks.rados.rados.0.smithi071.stdout:3070: finishing copy_from to smithi07134777-8 2024-09-11T01:49:46.889 INFO:tasks.rados.rados.0.smithi071.stdout:3070: got expected ENOENT (src dne) 2024-09-11T01:49:46.889 INFO:tasks.rados.rados.0.smithi071.stdout:3070: finishing copy_from racing read to smithi07134777-8 2024-09-11T01:49:46.891 INFO:tasks.rados.rados.0.smithi071.stdout:3070: done (1 left) 2024-09-11T01:49:46.891 INFO:tasks.rados.rados.0.smithi071.stdout:3071: done (0 left) 2024-09-11T01:49:46.891 INFO:tasks.rados.rados.0.smithi071.stdout:3072: copy_from oid 48 from oid 32 current snap is 346 2024-09-11T01:49:46.891 INFO:tasks.rados.rados.0.smithi071.stdout:3073: snap_remove snap 252 2024-09-11T01:49:47.854 INFO:tasks.rados.rados.0.smithi071.stdout:3073: done (1 left) 2024-09-11T01:49:47.854 INFO:tasks.rados.rados.0.smithi071.stdout:3074: delete oid 17 current snap is 346 2024-09-11T01:49:47.854 INFO:tasks.rados.rados.0.smithi071.stdout:3072: finishing copy_from racing read to smithi07134777-48 2024-09-11T01:49:47.854 INFO:tasks.rados.rados.0.smithi071.stdout:3072: finishing copy_from to smithi07134777-48 2024-09-11T01:49:47.854 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 1950 (ObjNum 1073 snap 346 seq_num 1073) dirty exists 2024-09-11T01:49:47.864 INFO:tasks.rados.rados.0.smithi071.stdout:3072: done (1 left) 2024-09-11T01:49:47.864 INFO:tasks.rados.rados.0.smithi071.stdout:3074: done (0 left) 2024-09-11T01:49:47.864 INFO:tasks.rados.rados.0.smithi071.stdout:3075: delete oid 1 current snap is 346 2024-09-11T01:49:47.880 INFO:tasks.rados.rados.0.smithi071.stdout:3075: done (0 left) 2024-09-11T01:49:47.880 INFO:tasks.rados.rados.0.smithi071.stdout:3076: write oid 21 current snap is 346 2024-09-11T01:49:47.881 INFO:tasks.rados.rados.0.smithi071.stdout:3076: seq_num 1080 ranges {551560=591046,1765494=386066} 2024-09-11T01:49:47.887 INFO:tasks.rados.rados.0.smithi071.stdout:3076: writing smithi07134777-21 from 551560 to 1142606 tid 1 2024-09-11T01:49:47.893 INFO:tasks.rados.rados.0.smithi071.stdout:3076: writing smithi07134777-21 from 1765494 to 2151560 tid 2 2024-09-11T01:49:47.893 INFO:tasks.rados.rados.0.smithi071.stdout:3077: delete oid 17 current snap is 346 2024-09-11T01:49:47.895 INFO:tasks.rados.rados.0.smithi071.stdout:3077: done (1 left) 2024-09-11T01:49:47.895 INFO:tasks.rados.rados.0.smithi071.stdout:3078: write oid 9 current snap is 346 2024-09-11T01:49:47.895 INFO:tasks.rados.rados.0.smithi071.stdout:3078: seq_num 1082 ranges {585174=741544,1766105=419069} 2024-09-11T01:49:47.903 INFO:tasks.rados.rados.0.smithi071.stdout:3078: writing smithi07134777-9 from 585174 to 1326718 tid 1 2024-09-11T01:49:47.909 INFO:tasks.rados.rados.0.smithi071.stdout:3078: writing smithi07134777-9 from 1766105 to 2185174 tid 2 2024-09-11T01:49:47.909 INFO:tasks.rados.rados.0.smithi071.stdout:3079: read oid 35 snap 307 2024-09-11T01:49:47.909 INFO:tasks.rados.rados.0.smithi071.stdout:3079: expect (ObjNum 870 snap 267 seq_num 870) 2024-09-11T01:49:47.909 INFO:tasks.rados.rados.0.smithi071.stdout:3080: copy_from oid 37 from oid 27 current snap is 346 2024-09-11T01:49:47.909 INFO:tasks.rados.rados.0.smithi071.stdout:3076: finishing write tid 1 to smithi07134777-21 2024-09-11T01:49:47.910 INFO:tasks.rados.rados.0.smithi071.stdout:3076: finishing write tid 2 to smithi07134777-21 2024-09-11T01:49:47.910 INFO:tasks.rados.rados.0.smithi071.stdout:3076: finishing write tid 3 to smithi07134777-21 2024-09-11T01:49:47.910 INFO:tasks.rados.rados.0.smithi071.stdout:3076: finishing write tid 4 to smithi07134777-21 2024-09-11T01:49:47.910 INFO:tasks.rados.rados.0.smithi071.stdout:3076: oid 21 updating version 0 to 2056 2024-09-11T01:49:47.910 INFO:tasks.rados.rados.0.smithi071.stdout:3076: oid 21 version 2056 is already newer than 2054 2024-09-11T01:49:47.910 INFO:tasks.rados.rados.0.smithi071.stdout:3076: oid 21 version 2056 is already newer than 2055 2024-09-11T01:49:47.910 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 2056 (ObjNum 1080 snap 346 seq_num 1080) dirty exists 2024-09-11T01:49:47.910 INFO:tasks.rados.rados.0.smithi071.stdout:3076: left oid 21 (ObjNum 1080 snap 346 seq_num 1080) 2024-09-11T01:49:47.910 INFO:tasks.rados.rados.0.smithi071.stdout:3078: finishing write tid 1 to smithi07134777-9 2024-09-11T01:49:47.910 INFO:tasks.rados.rados.0.smithi071.stdout:3076: done (3 left) 2024-09-11T01:49:47.911 INFO:tasks.rados.rados.0.smithi071.stdout:3081: rollback oid 34 current snap is 346 2024-09-11T01:49:47.911 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 34 to 296 2024-09-11T01:49:47.911 INFO:tasks.rados.rados.0.smithi071.stdout:3082: copy_from oid 42 from oid 36 current snap is 346 2024-09-11T01:49:47.911 INFO:tasks.rados.rados.0.smithi071.stdout:3083: write (excl) oid 30 current snap is 346 2024-09-11T01:49:47.911 INFO:tasks.rados.rados.0.smithi071.stdout:3083: seq_num 1083 ranges {601981=416793,1424587=492702} 2024-09-11T01:49:47.913 INFO:tasks.rados.rados.0.smithi071.stdout:3083: writing smithi07134777-30 from 601981 to 1018774 tid 1 2024-09-11T01:49:47.920 INFO:tasks.rados.rados.0.smithi071.stdout:3083: writing smithi07134777-30 from 1424587 to 1917289 tid 2 2024-09-11T01:49:47.920 INFO:tasks.rados.rados.0.smithi071.stdout:3084: snap_create 2024-09-11T01:49:47.920 INFO:tasks.rados.rados.0.smithi071.stdout:3078: finishing write tid 2 to smithi07134777-9 2024-09-11T01:49:47.920 INFO:tasks.rados.rados.0.smithi071.stdout:3078: finishing write tid 3 to smithi07134777-9 2024-09-11T01:49:47.920 INFO:tasks.rados.rados.0.smithi071.stdout:3078: finishing write tid 4 to smithi07134777-9 2024-09-11T01:49:47.920 INFO:tasks.rados.rados.0.smithi071.stdout:3078: oid 9 updating version 0 to 1028 2024-09-11T01:49:47.920 INFO:tasks.rados.rados.0.smithi071.stdout:3078: oid 9 version 1028 is already newer than 1027 2024-09-11T01:49:47.921 INFO:tasks.rados.rados.0.smithi071.stdout:3078: oid 9 version 1028 is already newer than 1026 2024-09-11T01:49:47.921 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 1028 (ObjNum 1082 snap 346 seq_num 1082) dirty exists 2024-09-11T01:49:47.921 INFO:tasks.rados.rados.0.smithi071.stdout:3078: left oid 9 (ObjNum 1082 snap 346 seq_num 1082) 2024-09-11T01:49:47.921 INFO:tasks.rados.rados.0.smithi071.stdout:3081: finishing rollback tid 0 to smithi07134777-34 2024-09-11T01:49:47.921 INFO:tasks.rados.rados.0.smithi071.stdout:3081: finishing rollback tid 1 to smithi07134777-34 2024-09-11T01:49:47.921 INFO:tasks.rados.rados.0.smithi071.stdout:3081: finishing rollback tid 2 to smithi07134777-34 2024-09-11T01:49:47.921 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 2163 (ObjNum 857 snap 263 seq_num 857) dirty exists 2024-09-11T01:49:47.922 INFO:tasks.rados.rados.0.smithi071.stdout:3083: finishing write tid 1 to smithi07134777-30 2024-09-11T01:49:47.923 INFO:tasks.rados.rados.0.smithi071.stdout:3082: finishing copy_from racing read to smithi07134777-42 2024-09-11T01:49:47.925 INFO:tasks.rados.rados.0.smithi071.stdout:3079: expect (ObjNum 870 snap 267 seq_num 870) 2024-09-11T01:49:47.925 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:47.973 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:48.021 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:48.069 INFO:tasks.rados.rados.0.smithi071.stdout:3083: finishing write tid 2 to smithi07134777-30 2024-09-11T01:49:48.069 INFO:tasks.rados.rados.0.smithi071.stdout:3083: finishing write tid 3 to smithi07134777-30 2024-09-11T01:49:48.069 INFO:tasks.rados.rados.0.smithi071.stdout:3083: finishing write tid 4 to smithi07134777-30 2024-09-11T01:49:48.069 INFO:tasks.rados.rados.0.smithi071.stdout:3083: oid 30 updating version 0 to 1317 2024-09-11T01:49:48.069 INFO:tasks.rados.rados.0.smithi071.stdout:3083: oid 30 updating version 1317 to 1318 2024-09-11T01:49:48.069 INFO:tasks.rados.rados.0.smithi071.stdout:3083: oid 30 updating version 1318 to 1319 2024-09-11T01:49:48.070 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 1319 (ObjNum 1083 snap 346 seq_num 1083) dirty exists 2024-09-11T01:49:48.070 INFO:tasks.rados.rados.0.smithi071.stdout:3083: left oid 30 (ObjNum 1083 snap 346 seq_num 1083) 2024-09-11T01:49:48.070 INFO:tasks.rados.rados.0.smithi071.stdout:3080: finishing copy_from racing read to smithi07134777-37 2024-09-11T01:49:48.070 INFO:tasks.rados.rados.0.smithi071.stdout:3080: finishing copy_from to smithi07134777-37 2024-09-11T01:49:48.070 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 854 (ObjNum 875 snap 267 seq_num 875) dirty exists 2024-09-11T01:49:48.070 INFO:tasks.rados.rados.0.smithi071.stdout:3082: finishing copy_from to smithi07134777-42 2024-09-11T01:49:48.070 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 1778 (ObjNum 1063 snap 341 seq_num 1063) dirty exists 2024-09-11T01:49:48.827 INFO:tasks.rados.rados.0.smithi071.stdout:3078: done (6 left) 2024-09-11T01:49:48.827 INFO:tasks.rados.rados.0.smithi071.stdout:3079: done (5 left) 2024-09-11T01:49:48.827 INFO:tasks.rados.rados.0.smithi071.stdout:3080: done (4 left) 2024-09-11T01:49:48.827 INFO:tasks.rados.rados.0.smithi071.stdout:3081: done (3 left) 2024-09-11T01:49:48.828 INFO:tasks.rados.rados.0.smithi071.stdout:3082: done (2 left) 2024-09-11T01:49:48.828 INFO:tasks.rados.rados.0.smithi071.stdout:3083: done (1 left) 2024-09-11T01:49:48.828 INFO:tasks.rados.rados.0.smithi071.stdout:3084: done (0 left) 2024-09-11T01:49:48.828 INFO:tasks.rados.rados.0.smithi071.stdout:3085: write (excl) oid 45 current snap is 347 2024-09-11T01:49:48.828 INFO:tasks.rados.rados.0.smithi071.stdout:3085: seq_num 1084 ranges {618788=492042,1883069=335719} 2024-09-11T01:49:48.833 INFO:tasks.rados.rados.0.smithi071.stdout:3085: writing smithi07134777-45 from 618788 to 1110830 tid 1 2024-09-11T01:49:48.839 INFO:tasks.rados.rados.0.smithi071.stdout:3085: writing smithi07134777-45 from 1883069 to 2218788 tid 2 2024-09-11T01:49:48.839 INFO:tasks.rados.rados.0.smithi071.stdout:3086: delete oid 6 current snap is 347 2024-09-11T01:49:48.845 INFO:tasks.rados.rados.0.smithi071.stdout:3086: done (1 left) 2024-09-11T01:49:48.845 INFO:tasks.rados.rados.0.smithi071.stdout:3087: read oid 49 snap 338 2024-09-11T01:49:48.845 INFO:tasks.rados.rados.0.smithi071.stdout:3087: expect (ObjNum 1017 snap 322 seq_num 1017) 2024-09-11T01:49:48.845 INFO:tasks.rados.rados.0.smithi071.stdout:3088: write oid 46 current snap is 347 2024-09-11T01:49:48.845 INFO:tasks.rados.rados.0.smithi071.stdout:3088: seq_num 1086 ranges {652402=642540,1883680=368722} 2024-09-11T01:49:48.852 INFO:tasks.rados.rados.0.smithi071.stdout:3088: writing smithi07134777-46 from 652402 to 1294942 tid 1 2024-09-11T01:49:48.860 INFO:tasks.rados.rados.0.smithi071.stdout:3088: writing smithi07134777-46 from 1883680 to 2252402 tid 2 2024-09-11T01:49:48.860 INFO:tasks.rados.rados.0.smithi071.stdout:3089: rollback oid 31 current snap is 347 2024-09-11T01:49:48.860 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 31 to 296 2024-09-11T01:49:48.861 INFO:tasks.rados.rados.0.smithi071.stdout:3085: finishing write tid 1 to smithi07134777-45 2024-09-11T01:49:48.861 INFO:tasks.rados.rados.0.smithi071.stdout:3085: finishing write tid 2 to smithi07134777-45 2024-09-11T01:49:48.861 INFO:tasks.rados.rados.0.smithi071.stdout:3085: finishing write tid 3 to smithi07134777-45 2024-09-11T01:49:48.861 INFO:tasks.rados.rados.0.smithi071.stdout:3085: finishing write tid 4 to smithi07134777-45 2024-09-11T01:49:48.861 INFO:tasks.rados.rados.0.smithi071.stdout:3085: oid 45 updating version 0 to 2059 2024-09-11T01:49:48.861 INFO:tasks.rados.rados.0.smithi071.stdout:3085: oid 45 version 2059 is already newer than 2058 2024-09-11T01:49:48.861 INFO:tasks.rados.rados.0.smithi071.stdout:3085: oid 45 version 2059 is already newer than 2057 2024-09-11T01:49:48.862 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 2059 (ObjNum 1084 snap 347 seq_num 1084) dirty exists 2024-09-11T01:49:48.862 INFO:tasks.rados.rados.0.smithi071.stdout:3085: left oid 45 (ObjNum 1084 snap 347 seq_num 1084) 2024-09-11T01:49:48.862 INFO:tasks.rados.rados.0.smithi071.stdout:3087: expect (ObjNum 1017 snap 322 seq_num 1017) 2024-09-11T01:49:48.910 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:48.984 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1476387 2024-09-11T01:49:49.026 INFO:tasks.rados.rados.0.smithi071.stdout:3088: finishing write tid 1 to smithi07134777-46 2024-09-11T01:49:49.026 INFO:tasks.rados.rados.0.smithi071.stdout:3088: finishing write tid 2 to smithi07134777-46 2024-09-11T01:49:49.026 INFO:tasks.rados.rados.0.smithi071.stdout:3088: finishing write tid 3 to smithi07134777-46 2024-09-11T01:49:49.026 INFO:tasks.rados.rados.0.smithi071.stdout:3088: finishing write tid 4 to smithi07134777-46 2024-09-11T01:49:49.026 INFO:tasks.rados.rados.0.smithi071.stdout:3088: oid 46 updating version 0 to 1776 2024-09-11T01:49:49.026 INFO:tasks.rados.rados.0.smithi071.stdout:3088: oid 46 updating version 1776 to 1777 2024-09-11T01:49:49.026 INFO:tasks.rados.rados.0.smithi071.stdout:3088: oid 46 version 1777 is already newer than 1775 2024-09-11T01:49:49.026 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 1777 (ObjNum 1086 snap 347 seq_num 1086) dirty exists 2024-09-11T01:49:49.027 INFO:tasks.rados.rados.0.smithi071.stdout:3088: left oid 46 (ObjNum 1086 snap 347 seq_num 1086) 2024-09-11T01:49:49.027 INFO:tasks.rados.rados.0.smithi071.stdout:3089: finishing rollback tid 0 to smithi07134777-31 2024-09-11T01:49:49.027 INFO:tasks.rados.rados.0.smithi071.stdout:3089: finishing rollback tid 1 to smithi07134777-31 2024-09-11T01:49:49.027 INFO:tasks.rados.rados.0.smithi071.stdout:3089: finishing rollback tid 2 to smithi07134777-31 2024-09-11T01:49:49.027 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 1799 (ObjNum 737 snap 226 seq_num 737) dirty exists 2024-09-11T01:49:49.027 INFO:tasks.rados.rados.0.smithi071.stdout:3085: done (3 left) 2024-09-11T01:49:49.027 INFO:tasks.rados.rados.0.smithi071.stdout:3087: done (2 left) 2024-09-11T01:49:49.027 INFO:tasks.rados.rados.0.smithi071.stdout:3088: done (1 left) 2024-09-11T01:49:49.027 INFO:tasks.rados.rados.0.smithi071.stdout:3089: done (0 left) 2024-09-11T01:49:49.027 INFO:tasks.rados.rados.0.smithi071.stdout:3090: read oid 6 snap 339 2024-09-11T01:49:49.027 INFO:tasks.rados.rados.0.smithi071.stdout:3090: expect (ObjNum 601 snap 172 seq_num 601) 2024-09-11T01:49:49.027 INFO:tasks.rados.rados.0.smithi071.stdout:3091: read oid 19 snap -1 2024-09-11T01:49:49.027 INFO:tasks.rados.rados.0.smithi071.stdout:3091: expect (ObjNum 1074 snap 346 seq_num 1074) 2024-09-11T01:49:49.028 INFO:tasks.rados.rados.0.smithi071.stdout:3092: read oid 34 snap -1 2024-09-11T01:49:49.028 INFO:tasks.rados.rados.0.smithi071.stdout:3092: expect (ObjNum 857 snap 263 seq_num 857) 2024-09-11T01:49:49.028 INFO:tasks.rados.rados.0.smithi071.stdout:3093: write (excl) oid 7 current snap is 347 2024-09-11T01:49:49.028 INFO:tasks.rados.rados.0.smithi071.stdout:3093: seq_num 1087 ranges {669209=717789,1942162=327047} 2024-09-11T01:49:49.032 INFO:tasks.rados.rados.0.smithi071.stdout:3093: writing smithi07134777-7 from 669209 to 1386998 tid 1 2024-09-11T01:49:49.038 INFO:tasks.rados.rados.0.smithi071.stdout:3093: writing smithi07134777-7 from 1942162 to 2269209 tid 2 2024-09-11T01:49:49.038 INFO:tasks.rados.rados.0.smithi071.stdout:3094: snap_create 2024-09-11T01:49:49.038 INFO:tasks.rados.rados.0.smithi071.stdout:3090: expect (ObjNum 601 snap 172 seq_num 601) 2024-09-11T01:49:49.123 INFO:tasks.rados.rados.0.smithi071.stdout:3091: expect (ObjNum 1074 snap 346 seq_num 1074) 2024-09-11T01:49:49.123 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:49.217 INFO:tasks.rados.rados.0.smithi071.stdout:3092: expect (ObjNum 857 snap 263 seq_num 857) 2024-09-11T01:49:49.297 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2006663 2024-09-11T01:49:49.307 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:49.361 INFO:tasks.rados.rados.0.smithi071.stdout:3093: finishing write tid 1 to smithi07134777-7 2024-09-11T01:49:49.361 INFO:tasks.rados.rados.0.smithi071.stdout:3093: finishing write tid 2 to smithi07134777-7 2024-09-11T01:49:49.361 INFO:tasks.rados.rados.0.smithi071.stdout:3093: finishing write tid 3 to smithi07134777-7 2024-09-11T01:49:49.361 INFO:tasks.rados.rados.0.smithi071.stdout:3093: finishing write tid 4 to smithi07134777-7 2024-09-11T01:49:49.361 INFO:tasks.rados.rados.0.smithi071.stdout:3093: oid 7 updating version 0 to 2213 2024-09-11T01:49:49.361 INFO:tasks.rados.rados.0.smithi071.stdout:3093: oid 7 version 2213 is already newer than 2212 2024-09-11T01:49:49.361 INFO:tasks.rados.rados.0.smithi071.stdout:3093: oid 7 version 2213 is already newer than 2211 2024-09-11T01:49:49.362 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 2213 (ObjNum 1087 snap 347 seq_num 1087) dirty exists 2024-09-11T01:49:49.362 INFO:tasks.rados.rados.0.smithi071.stdout:3093: left oid 7 (ObjNum 1087 snap 347 seq_num 1087) 2024-09-11T01:49:49.858 INFO:tasks.rados.rados.0.smithi071.stdout:3090: done (4 left) 2024-09-11T01:49:49.858 INFO:tasks.rados.rados.0.smithi071.stdout:3091: done (3 left) 2024-09-11T01:49:49.859 INFO:tasks.rados.rados.0.smithi071.stdout:3092: done (2 left) 2024-09-11T01:49:49.859 INFO:tasks.rados.rados.0.smithi071.stdout:3093: done (1 left) 2024-09-11T01:49:49.859 INFO:tasks.rados.rados.0.smithi071.stdout:3094: done (0 left) 2024-09-11T01:49:49.859 INFO:tasks.rados.rados.0.smithi071.stdout:3095: read oid 44 snap -1 2024-09-11T01:49:49.859 INFO:tasks.rados.rados.0.smithi071.stdout:3095: expect deleted 2024-09-11T01:49:49.859 INFO:tasks.rados.rados.0.smithi071.stdout:3096: read oid 31 snap -1 2024-09-11T01:49:49.859 INFO:tasks.rados.rados.0.smithi071.stdout:3096: expect (ObjNum 737 snap 226 seq_num 737) 2024-09-11T01:49:49.859 INFO:tasks.rados.rados.0.smithi071.stdout:3097: write oid 47 current snap is 348 2024-09-11T01:49:49.859 INFO:tasks.rados.rados.0.smithi071.stdout:3097: seq_num 1088 ranges {686016=509391,1716997=569019} 2024-09-11T01:49:49.865 INFO:tasks.rados.rados.0.smithi071.stdout:3097: writing smithi07134777-47 from 686016 to 1195407 tid 1 2024-09-11T01:49:49.871 INFO:tasks.rados.rados.0.smithi071.stdout:3097: writing smithi07134777-47 from 1716997 to 2286016 tid 2 2024-09-11T01:49:49.871 INFO:tasks.rados.rados.0.smithi071.stdout:3098: snap_remove snap 250 2024-09-11T01:49:50.859 INFO:tasks.rados.rados.0.smithi071.stdout:3098: done (3 left) 2024-09-11T01:49:50.859 INFO:tasks.rados.rados.0.smithi071.stdout:3099: copy_from oid 41 from oid 32 current snap is 348 2024-09-11T01:49:50.860 INFO:tasks.rados.rados.0.smithi071.stdout:3096: expect (ObjNum 737 snap 226 seq_num 737) 2024-09-11T01:49:50.910 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 783665 2024-09-11T01:49:50.942 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:50.983 INFO:tasks.rados.rados.0.smithi071.stdout:3097: finishing write tid 1 to smithi07134777-47 2024-09-11T01:49:50.983 INFO:tasks.rados.rados.0.smithi071.stdout:3097: finishing write tid 2 to smithi07134777-47 2024-09-11T01:49:50.983 INFO:tasks.rados.rados.0.smithi071.stdout:3097: finishing write tid 3 to smithi07134777-47 2024-09-11T01:49:50.983 INFO:tasks.rados.rados.0.smithi071.stdout:3097: finishing write tid 4 to smithi07134777-47 2024-09-11T01:49:50.983 INFO:tasks.rados.rados.0.smithi071.stdout:3097: oid 47 updating version 0 to 1800 2024-09-11T01:49:50.983 INFO:tasks.rados.rados.0.smithi071.stdout:3097: oid 47 updating version 1800 to 1801 2024-09-11T01:49:50.983 INFO:tasks.rados.rados.0.smithi071.stdout:3097: oid 47 updating version 1801 to 1802 2024-09-11T01:49:50.983 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 1802 (ObjNum 1088 snap 348 seq_num 1088) dirty exists 2024-09-11T01:49:50.983 INFO:tasks.rados.rados.0.smithi071.stdout:3097: left oid 47 (ObjNum 1088 snap 348 seq_num 1088) 2024-09-11T01:49:50.983 INFO:tasks.rados.rados.0.smithi071.stdout:3095: done (3 left) 2024-09-11T01:49:50.984 INFO:tasks.rados.rados.0.smithi071.stdout:3096: done (2 left) 2024-09-11T01:49:50.984 INFO:tasks.rados.rados.0.smithi071.stdout:3097: done (1 left) 2024-09-11T01:49:50.984 INFO:tasks.rados.rados.0.smithi071.stdout:3100: delete oid 35 current snap is 348 2024-09-11T01:49:50.984 INFO:tasks.rados.rados.0.smithi071.stdout:3099: finishing copy_from racing read to smithi07134777-41 2024-09-11T01:49:50.984 INFO:tasks.rados.rados.0.smithi071.stdout:3099: finishing copy_from to smithi07134777-41 2024-09-11T01:49:50.984 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 1950 (ObjNum 1073 snap 346 seq_num 1073) dirty exists 2024-09-11T01:49:50.986 INFO:tasks.rados.rados.0.smithi071.stdout:3099: done (1 left) 2024-09-11T01:49:50.986 INFO:tasks.rados.rados.0.smithi071.stdout:3100: done (0 left) 2024-09-11T01:49:50.987 INFO:tasks.rados.rados.0.smithi071.stdout:3101: snap_remove snap 342 2024-09-11T01:49:51.875 INFO:tasks.rados.rados.0.smithi071.stdout:3101: done (0 left) 2024-09-11T01:49:51.876 INFO:tasks.rados.rados.0.smithi071.stdout:3102: read oid 6 snap -1 2024-09-11T01:49:51.876 INFO:tasks.rados.rados.0.smithi071.stdout:3102: expect deleted 2024-09-11T01:49:51.876 INFO:tasks.rados.rados.0.smithi071.stdout:3103: rollback oid 48 current snap is 348 2024-09-11T01:49:51.876 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 48 to 283 2024-09-11T01:49:51.876 INFO:tasks.rados.rados.0.smithi071.stdout:3104: write oid 19 current snap is 348 2024-09-11T01:49:51.876 INFO:tasks.rados.rados.0.smithi071.stdout:3104: seq_num 1090 ranges {719630=659889,2117608=202022} 2024-09-11T01:49:51.884 INFO:tasks.rados.rados.0.smithi071.stdout:3104: writing smithi07134777-19 from 719630 to 1379519 tid 1 2024-09-11T01:49:51.889 INFO:tasks.rados.rados.0.smithi071.stdout:3104: writing smithi07134777-19 from 2117608 to 2319630 tid 2 2024-09-11T01:49:51.889 INFO:tasks.rados.rados.0.smithi071.stdout:3105: write oid 9 current snap is 348 2024-09-11T01:49:51.889 INFO:tasks.rados.rados.0.smithi071.stdout:3105: seq_num 1091 ranges {736437=735138,2176090=160347} 2024-09-11T01:49:51.897 INFO:tasks.rados.rados.0.smithi071.stdout:3105: writing smithi07134777-9 from 736437 to 1471575 tid 1 2024-09-11T01:49:51.902 INFO:tasks.rados.rados.0.smithi071.stdout:3105: writing smithi07134777-9 from 2176090 to 2336437 tid 2 2024-09-11T01:49:51.902 INFO:tasks.rados.rados.0.smithi071.stdout:3106: read oid 37 snap -1 2024-09-11T01:49:51.902 INFO:tasks.rados.rados.0.smithi071.stdout:3106: expect (ObjNum 875 snap 267 seq_num 875) 2024-09-11T01:49:51.902 INFO:tasks.rados.rados.0.smithi071.stdout:3107: snap_remove snap 332 2024-09-11T01:49:52.874 INFO:tasks.rados.rados.0.smithi071.stdout:3107: done (5 left) 2024-09-11T01:49:52.874 INFO:tasks.rados.rados.0.smithi071.stdout:3108: write oid 32 current snap is 348 2024-09-11T01:49:52.875 INFO:tasks.rados.rados.0.smithi071.stdout:3108: seq_num 1092 ranges {753244=410387,1834572=518672} 2024-09-11T01:49:52.881 INFO:tasks.rados.rados.0.smithi071.stdout:3108: writing smithi07134777-32 from 753244 to 1163631 tid 1 2024-09-11T01:49:52.888 INFO:tasks.rados.rados.0.smithi071.stdout:3108: writing smithi07134777-32 from 1834572 to 2353244 tid 2 2024-09-11T01:49:52.888 INFO:tasks.rados.rados.0.smithi071.stdout:3109: snap_create 2024-09-11T01:49:52.888 INFO:tasks.rados.rados.0.smithi071.stdout:3103: finishing rollback tid 0 to smithi07134777-48 2024-09-11T01:49:52.888 INFO:tasks.rados.rados.0.smithi071.stdout:3103: finishing rollback tid 1 to smithi07134777-48 2024-09-11T01:49:52.889 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 2228 (ObjNum 129 snap 0 seq_num 2371176376) dirty dne 2024-09-11T01:49:52.889 INFO:tasks.rados.rados.0.smithi071.stdout:3106: expect (ObjNum 875 snap 267 seq_num 875) 2024-09-11T01:49:52.889 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:52.920 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:49:52.974 INFO:tasks.rados.rados.0.smithi071.stdout:3104: finishing write tid 1 to smithi07134777-19 2024-09-11T01:49:52.974 INFO:tasks.rados.rados.0.smithi071.stdout:3104: finishing write tid 2 to smithi07134777-19 2024-09-11T01:49:52.974 INFO:tasks.rados.rados.0.smithi071.stdout:3104: finishing write tid 3 to smithi07134777-19 2024-09-11T01:49:52.975 INFO:tasks.rados.rados.0.smithi071.stdout:3104: finishing write tid 4 to smithi07134777-19 2024-09-11T01:49:52.975 INFO:tasks.rados.rados.0.smithi071.stdout:3104: oid 19 updating version 0 to 1787 2024-09-11T01:49:52.975 INFO:tasks.rados.rados.0.smithi071.stdout:3104: oid 19 version 1787 is already newer than 1786 2024-09-11T01:49:52.975 INFO:tasks.rados.rados.0.smithi071.stdout:3104: oid 19 updating version 1787 to 1788 2024-09-11T01:49:52.975 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 1788 (ObjNum 1090 snap 348 seq_num 1090) dirty exists 2024-09-11T01:49:52.975 INFO:tasks.rados.rados.0.smithi071.stdout:3104: left oid 19 (ObjNum 1090 snap 348 seq_num 1090) 2024-09-11T01:49:52.975 INFO:tasks.rados.rados.0.smithi071.stdout:3105: finishing write tid 1 to smithi07134777-9 2024-09-11T01:49:52.975 INFO:tasks.rados.rados.0.smithi071.stdout:3105: finishing write tid 2 to smithi07134777-9 2024-09-11T01:49:52.975 INFO:tasks.rados.rados.0.smithi071.stdout:3105: finishing write tid 3 to smithi07134777-9 2024-09-11T01:49:52.975 INFO:tasks.rados.rados.0.smithi071.stdout:3105: finishing write tid 4 to smithi07134777-9 2024-09-11T01:49:52.975 INFO:tasks.rados.rados.0.smithi071.stdout:3105: oid 9 updating version 0 to 1030 2024-09-11T01:49:52.976 INFO:tasks.rados.rados.0.smithi071.stdout:3105: oid 9 version 1030 is already newer than 1029 2024-09-11T01:49:52.976 INFO:tasks.rados.rados.0.smithi071.stdout:3105: oid 9 updating version 1030 to 1031 2024-09-11T01:49:52.976 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 1031 (ObjNum 1091 snap 348 seq_num 1091) dirty exists 2024-09-11T01:49:52.976 INFO:tasks.rados.rados.0.smithi071.stdout:3105: left oid 9 (ObjNum 1091 snap 348 seq_num 1091) 2024-09-11T01:49:52.976 INFO:tasks.rados.rados.0.smithi071.stdout:3108: finishing write tid 1 to smithi07134777-32 2024-09-11T01:49:52.976 INFO:tasks.rados.rados.0.smithi071.stdout:3108: finishing write tid 2 to smithi07134777-32 2024-09-11T01:49:52.976 INFO:tasks.rados.rados.0.smithi071.stdout:3108: finishing write tid 3 to smithi07134777-32 2024-09-11T01:49:52.976 INFO:tasks.rados.rados.0.smithi071.stdout:3108: finishing write tid 4 to smithi07134777-32 2024-09-11T01:49:52.976 INFO:tasks.rados.rados.0.smithi071.stdout:3108: oid 32 updating version 0 to 1957 2024-09-11T01:49:52.976 INFO:tasks.rados.rados.0.smithi071.stdout:3108: oid 32 version 1957 is already newer than 1955 2024-09-11T01:49:52.976 INFO:tasks.rados.rados.0.smithi071.stdout:3108: oid 32 version 1957 is already newer than 1956 2024-09-11T01:49:52.976 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 1957 (ObjNum 1092 snap 348 seq_num 1092) dirty exists 2024-09-11T01:49:52.977 INFO:tasks.rados.rados.0.smithi071.stdout:3108: left oid 32 (ObjNum 1092 snap 348 seq_num 1092) 2024-09-11T01:49:53.828 INFO:tasks.rados.rados.0.smithi071.stdout:3102: done (6 left) 2024-09-11T01:49:53.828 INFO:tasks.rados.rados.0.smithi071.stdout:3103: done (5 left) 2024-09-11T01:49:53.828 INFO:tasks.rados.rados.0.smithi071.stdout:3104: done (4 left) 2024-09-11T01:49:53.829 INFO:tasks.rados.rados.0.smithi071.stdout:3105: done (3 left) 2024-09-11T01:49:53.829 INFO:tasks.rados.rados.0.smithi071.stdout:3106: done (2 left) 2024-09-11T01:49:53.829 INFO:tasks.rados.rados.0.smithi071.stdout:3108: done (1 left) 2024-09-11T01:49:53.829 INFO:tasks.rados.rados.0.smithi071.stdout:3109: done (0 left) 2024-09-11T01:49:53.829 INFO:tasks.rados.rados.0.smithi071.stdout:3110: rollback oid 6 current snap is 349 2024-09-11T01:49:53.829 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 6 to 312 2024-09-11T01:49:53.829 INFO:tasks.rados.rados.0.smithi071.stdout:3111: snap_create 2024-09-11T01:49:53.836 INFO:tasks.rados.rados.0.smithi071.stdout:3110: finishing rollback tid 1 to smithi07134777-6 2024-09-11T01:49:53.836 INFO:tasks.rados.rados.0.smithi071.stdout:3110: finishing rollback tid 2 to smithi07134777-6 2024-09-11T01:49:53.837 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 2166 (ObjNum 926 snap 292 seq_num 926) dirty exists 2024-09-11T01:49:54.835 INFO:tasks.rados.rados.0.smithi071.stdout:3110: done (1 left) 2024-09-11T01:49:54.835 INFO:tasks.rados.rados.0.smithi071.stdout:3111: done (0 left) 2024-09-11T01:49:54.835 INFO:tasks.rados.rados.0.smithi071.stdout:3112: copy_from oid 15 from oid 49 current snap is 350 2024-09-11T01:49:54.835 INFO:tasks.rados.rados.0.smithi071.stdout:3113: rollback oid 47 current snap is 350 2024-09-11T01:49:54.835 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 47 to 338 2024-09-11T01:49:54.835 INFO:tasks.rados.rados.0.smithi071.stdout:3114: write oid 7 current snap is 350 2024-09-11T01:49:54.835 INFO:tasks.rados.rados.0.smithi071.stdout:3114: seq_num 1093 ranges {770051=485636,1776701=511047} 2024-09-11T01:49:54.842 INFO:tasks.rados.rados.0.smithi071.stdout:3114: writing smithi07134777-7 from 770051 to 1255687 tid 1 2024-09-11T01:49:54.848 INFO:tasks.rados.rados.0.smithi071.stdout:3114: writing smithi07134777-7 from 1776701 to 2287748 tid 2 2024-09-11T01:49:54.848 INFO:tasks.rados.rados.0.smithi071.stdout:3115: rollback oid 29 current snap is 350 2024-09-11T01:49:54.848 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 29 to 340 2024-09-11T01:49:54.849 INFO:tasks.rados.rados.0.smithi071.stdout:3113: finishing rollback tid 0 to smithi07134777-47 2024-09-11T01:49:54.849 INFO:tasks.rados.rados.0.smithi071.stdout:3113: finishing rollback tid 1 to smithi07134777-47 2024-09-11T01:49:54.849 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 1804 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-11T01:49:54.849 INFO:tasks.rados.rados.0.smithi071.stdout:3113: done (3 left) 2024-09-11T01:49:54.849 INFO:tasks.rados.rados.0.smithi071.stdout:3116: snap_remove snap 283 2024-09-11T01:49:55.839 INFO:tasks.rados.rados.0.smithi071.stdout:3116: done (3 left) 2024-09-11T01:49:55.839 INFO:tasks.rados.rados.0.smithi071.stdout:3117: read oid 48 snap -1 2024-09-11T01:49:55.840 INFO:tasks.rados.rados.0.smithi071.stdout:3117: expect deleted 2024-09-11T01:49:55.840 INFO:tasks.rados.rados.0.smithi071.stdout:3118: rollback oid 13 current snap is 350 2024-09-11T01:49:55.840 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 13 to 321 2024-09-11T01:49:55.840 INFO:tasks.rados.rados.0.smithi071.stdout:3112: finishing copy_from racing read to smithi07134777-15 2024-09-11T01:49:55.840 INFO:tasks.rados.rados.0.smithi071.stdout:3114: finishing write tid 1 to smithi07134777-7 2024-09-11T01:49:55.840 INFO:tasks.rados.rados.0.smithi071.stdout:3112: finishing copy_from to smithi07134777-15 2024-09-11T01:49:55.840 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 388 (ObjNum 1017 snap 322 seq_num 1017) dirty exists 2024-09-11T01:49:55.840 INFO:tasks.rados.rados.0.smithi071.stdout:3114: finishing write tid 2 to smithi07134777-7 2024-09-11T01:49:55.840 INFO:tasks.rados.rados.0.smithi071.stdout:3114: finishing write tid 3 to smithi07134777-7 2024-09-11T01:49:55.840 INFO:tasks.rados.rados.0.smithi071.stdout:3115: finishing rollback tid 1 to smithi07134777-29 2024-09-11T01:49:55.840 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 1947 (ObjNum 129 snap 0 seq_num 2371133488) dirty dne 2024-09-11T01:49:55.840 INFO:tasks.rados.rados.0.smithi071.stdout:3114: finishing write tid 4 to smithi07134777-7 2024-09-11T01:49:55.840 INFO:tasks.rados.rados.0.smithi071.stdout:3114: oid 7 updating version 0 to 2246 2024-09-11T01:49:55.844 INFO:tasks.rados.rados.0.smithi071.stdout:3114: oid 7 updating version 2246 to 2247 2024-09-11T01:49:55.844 INFO:tasks.rados.rados.0.smithi071.stdout:3114: oid 7 version 2247 is already newer than 2244 2024-09-11T01:49:55.844 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 2247 (ObjNum 1093 snap 350 seq_num 1093) dirty exists 2024-09-11T01:49:55.844 INFO:tasks.rados.rados.0.smithi071.stdout:3114: left oid 7 (ObjNum 1093 snap 350 seq_num 1093) 2024-09-11T01:49:55.844 INFO:tasks.rados.rados.0.smithi071.stdout:3112: done (4 left) 2024-09-11T01:49:55.844 INFO:tasks.rados.rados.0.smithi071.stdout:3114: done (3 left) 2024-09-11T01:49:55.844 INFO:tasks.rados.rados.0.smithi071.stdout:3115: done (2 left) 2024-09-11T01:49:55.844 INFO:tasks.rados.rados.0.smithi071.stdout:3119: rollback oid 12 current snap is 350 2024-09-11T01:49:55.844 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 12 to 323 2024-09-11T01:49:55.844 INFO:tasks.rados.rados.0.smithi071.stdout:3120: write (excl) oid 49 current snap is 350 2024-09-11T01:49:55.844 INFO:tasks.rados.rados.0.smithi071.stdout:3120: seq_num 1094 ranges {786858=560885,1835183=551675} 2024-09-11T01:49:55.848 INFO:tasks.rados.rados.0.smithi071.stdout:3120: writing smithi07134777-49 from 786858 to 1347743 tid 1 2024-09-11T01:49:55.854 INFO:tasks.rados.rados.0.smithi071.stdout:3120: writing smithi07134777-49 from 1835183 to 2386858 tid 2 2024-09-11T01:49:55.855 INFO:tasks.rados.rados.0.smithi071.stdout:3121: copy_from oid 33 from oid 37 current snap is 350 2024-09-11T01:49:55.855 INFO:tasks.rados.rados.0.smithi071.stdout:3118: finishing rollback tid 0 to smithi07134777-13 2024-09-11T01:49:55.855 INFO:tasks.rados.rados.0.smithi071.stdout:3117: done (4 left) 2024-09-11T01:49:55.855 INFO:tasks.rados.rados.0.smithi071.stdout:3122: read oid 35 snap 325 2024-09-11T01:49:55.855 INFO:tasks.rados.rados.0.smithi071.stdout:3122: expect (ObjNum 949 snap 300 seq_num 949) 2024-09-11T01:49:55.855 INFO:tasks.rados.rados.0.smithi071.stdout:3123: snap_remove snap 299 2024-09-11T01:49:57.704 INFO:tasks.rados.rados.0.smithi071.stdout:3123: done (5 left) 2024-09-11T01:49:57.704 INFO:tasks.rados.rados.0.smithi071.stdout:3124: snap_remove snap 348 2024-09-11T01:49:57.847 INFO:tasks.rados.rados.0.smithi071.stdout:3124: done (5 left) 2024-09-11T01:49:57.847 INFO:tasks.rados.rados.0.smithi071.stdout:3125: copy_from oid 31 from oid 40 current snap is 350 2024-09-11T01:49:57.847 INFO:tasks.rados.rados.0.smithi071.stdout:3118: finishing rollback tid 1 to smithi07134777-13 2024-09-11T01:49:57.847 INFO:tasks.rados.rados.0.smithi071.stdout:3119: finishing rollback tid 0 to smithi07134777-12 2024-09-11T01:49:57.847 INFO:tasks.rados.rados.0.smithi071.stdout:3119: finishing rollback tid 1 to smithi07134777-12 2024-09-11T01:49:57.847 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 2168 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-11T01:49:57.847 INFO:tasks.rados.rados.0.smithi071.stdout:3118: finishing rollback tid 2 to smithi07134777-13 2024-09-11T01:49:57.847 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 1807 (ObjNum 808 snap 246 seq_num 808) dirty exists 2024-09-11T01:49:57.848 INFO:tasks.rados.rados.0.smithi071.stdout:3120: finishing write tid 1 to smithi07134777-49 2024-09-11T01:49:57.848 INFO:tasks.rados.rados.0.smithi071.stdout:3120: finishing write tid 2 to smithi07134777-49 2024-09-11T01:49:57.848 INFO:tasks.rados.rados.0.smithi071.stdout:3120: finishing write tid 3 to smithi07134777-49 2024-09-11T01:49:57.848 INFO:tasks.rados.rados.0.smithi071.stdout:3120: finishing write tid 4 to smithi07134777-49 2024-09-11T01:49:57.848 INFO:tasks.rados.rados.0.smithi071.stdout:3120: oid 49 updating version 0 to 391 2024-09-11T01:49:57.848 INFO:tasks.rados.rados.0.smithi071.stdout:3120: oid 49 version 391 is already newer than 390 2024-09-11T01:49:57.848 INFO:tasks.rados.rados.0.smithi071.stdout:3120: oid 49 version 391 is already newer than 389 2024-09-11T01:49:57.848 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 391 (ObjNum 1094 snap 350 seq_num 1094) dirty exists 2024-09-11T01:49:57.848 INFO:tasks.rados.rados.0.smithi071.stdout:3120: left oid 49 (ObjNum 1094 snap 350 seq_num 1094) 2024-09-11T01:49:57.848 INFO:tasks.rados.rados.0.smithi071.stdout:3121: finishing copy_from racing read to smithi07134777-33 2024-09-11T01:49:57.848 INFO:tasks.rados.rados.0.smithi071.stdout:3121: finishing copy_from to smithi07134777-33 2024-09-11T01:49:57.848 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 854 (ObjNum 875 snap 267 seq_num 875) dirty exists 2024-09-11T01:49:57.848 INFO:tasks.rados.rados.0.smithi071.stdout:3122: expect (ObjNum 949 snap 300 seq_num 949) 2024-09-11T01:49:58.023 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-11T01:49:58.023 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-11T01:49:58.024 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-11T01:49:58.026 INFO:tasks.ceph.mon.b:Started 2024-09-11T01:49:58.026 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:49:58.026 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:49:58.069 INFO:tasks.rados.rados.0.smithi071.stdout:3125: finishing copy_from to smithi07134777-31 2024-09-11T01:49:58.069 INFO:tasks.rados.rados.0.smithi071.stdout:3125: got expected ENOENT (src dne) 2024-09-11T01:49:58.069 INFO:tasks.rados.rados.0.smithi071.stdout:3125: finishing copy_from racing read to smithi07134777-31 2024-09-11T01:49:58.069 INFO:tasks.rados.rados.0.smithi071.stdout:3118: done (5 left) 2024-09-11T01:49:58.069 INFO:tasks.rados.rados.0.smithi071.stdout:3119: done (4 left) 2024-09-11T01:49:58.069 INFO:tasks.rados.rados.0.smithi071.stdout:3120: done (3 left) 2024-09-11T01:49:58.069 INFO:tasks.rados.rados.0.smithi071.stdout:3121: done (2 left) 2024-09-11T01:49:58.069 INFO:tasks.rados.rados.0.smithi071.stdout:3122: done (1 left) 2024-09-11T01:49:58.069 INFO:tasks.rados.rados.0.smithi071.stdout:3125: done (0 left) 2024-09-11T01:49:58.069 INFO:tasks.rados.rados.0.smithi071.stdout:3126: copy_from oid 22 from oid 40 current snap is 350 2024-09-11T01:49:58.069 INFO:tasks.rados.rados.0.smithi071.stdout:3127: rollback oid 10 current snap is 350 2024-09-11T01:49:58.070 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 10 to 323 2024-09-11T01:49:58.070 INFO:tasks.rados.rados.0.smithi071.stdout:3128: delete oid 19 current snap is 350 2024-09-11T01:49:58.070 INFO:tasks.rados.rados.0.smithi071.stdout:3126: finishing copy_from to smithi07134777-22 2024-09-11T01:49:58.070 INFO:tasks.rados.rados.0.smithi071.stdout:3126: got expected ENOENT (src dne) 2024-09-11T01:49:58.070 INFO:tasks.rados.rados.0.smithi071.stdout:3126: finishing copy_from racing read to smithi07134777-22 2024-09-11T01:49:58.073 INFO:tasks.rados.rados.0.smithi071.stdout:3127: finishing rollback tid 0 to smithi07134777-10 2024-09-11T01:49:58.073 INFO:tasks.rados.rados.0.smithi071.stdout:3127: finishing rollback tid 1 to smithi07134777-10 2024-09-11T01:49:58.073 INFO:tasks.rados.rados.0.smithi071.stdout:3127: finishing rollback tid 2 to smithi07134777-10 2024-09-11T01:49:58.073 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 1953 (ObjNum 990 snap 317 seq_num 990) dirty exists 2024-09-11T01:49:58.075 INFO:tasks.rados.rados.0.smithi071.stdout:3126: done (2 left) 2024-09-11T01:49:58.075 INFO:tasks.rados.rados.0.smithi071.stdout:3127: done (1 left) 2024-09-11T01:49:58.075 INFO:tasks.rados.rados.0.smithi071.stdout:3128: done (0 left) 2024-09-11T01:49:58.075 INFO:tasks.rados.rados.0.smithi071.stdout:3129: write oid 35 current snap is 350 2024-09-11T01:49:58.075 INFO:tasks.rados.rados.0.smithi071.stdout:3129: seq_num 1096 ranges {420472=427736,1268500=658374,2347953=72519} 2024-09-11T01:49:58.080 INFO:tasks.rados.rados.0.smithi071.stdout:3129: writing smithi07134777-35 from 420472 to 848208 tid 1 2024-09-11T01:49:58.085 INFO:tasks.rados.rados.0.smithi071.stdout:3129: writing smithi07134777-35 from 1268500 to 1926874 tid 2 2024-09-11T01:49:58.088 INFO:tasks.rados.rados.0.smithi071.stdout:3129: writing smithi07134777-35 from 2347953 to 2420472 tid 3 2024-09-11T01:49:58.089 INFO:tasks.rados.rados.0.smithi071.stdout:3130: write (excl) oid 32 current snap is 350 2024-09-11T01:49:58.089 INFO:tasks.rados.rados.0.smithi071.stdout:3129: finishing write tid 1 to smithi07134777-35 2024-09-11T01:49:58.089 INFO:tasks.rados.rados.0.smithi071.stdout:3130: seq_num 1097 ranges {437279=502985,1610629=518385} 2024-09-11T01:49:58.093 INFO:tasks.rados.rados.0.smithi071.stdout:3130: writing smithi07134777-32 from 437279 to 940264 tid 1 2024-09-11T01:49:58.100 INFO:tasks.rados.rados.0.smithi071.stdout:3130: writing smithi07134777-32 from 1610629 to 2129014 tid 2 2024-09-11T01:49:58.100 INFO:tasks.rados.rados.0.smithi071.stdout:3131: write (excl) oid 22 current snap is 350 2024-09-11T01:49:58.100 INFO:tasks.rados.rados.0.smithi071.stdout:3129: finishing write tid 2 to smithi07134777-35 2024-09-11T01:49:58.100 INFO:tasks.rados.rados.0.smithi071.stdout:3131: seq_num 1098 ranges {454086=578234,1669111=662043} 2024-09-11T01:49:58.105 INFO:tasks.rados.rados.0.smithi071.stdout:3131: writing smithi07134777-22 from 454086 to 1032320 tid 1 2024-09-11T01:49:58.113 INFO:tasks.rados.rados.0.smithi071.stdout:3131: writing smithi07134777-22 from 1669111 to 2331154 tid 2 2024-09-11T01:49:58.113 INFO:tasks.rados.rados.0.smithi071.stdout:3132: write oid 37 current snap is 350 2024-09-11T01:49:58.113 INFO:tasks.rados.rados.0.smithi071.stdout:3129: finishing write tid 3 to smithi07134777-35 2024-09-11T01:49:58.113 INFO:tasks.rados.rados.0.smithi071.stdout:3129: finishing write tid 4 to smithi07134777-35 2024-09-11T01:49:58.114 INFO:tasks.rados.rados.0.smithi071.stdout:3129: finishing write tid 5 to smithi07134777-35 2024-09-11T01:49:58.114 INFO:tasks.rados.rados.0.smithi071.stdout:3129: oid 35 updating version 0 to 1958 2024-09-11T01:49:58.114 INFO:tasks.rados.rados.0.smithi071.stdout:3129: oid 35 updating version 1958 to 1959 2024-09-11T01:49:58.114 INFO:tasks.rados.rados.0.smithi071.stdout:3129: oid 35 updating version 1959 to 1961 2024-09-11T01:49:58.114 INFO:tasks.rados.rados.0.smithi071.stdout:3129: oid 35 version 1961 is already newer than 1960 2024-09-11T01:49:58.114 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 1961 (ObjNum 1096 snap 350 seq_num 1096) dirty exists 2024-09-11T01:49:58.114 INFO:tasks.rados.rados.0.smithi071.stdout:3129: left oid 35 (ObjNum 1096 snap 350 seq_num 1096) 2024-09-11T01:49:58.114 INFO:tasks.rados.rados.0.smithi071.stdout:3130: finishing write tid 1 to smithi07134777-32 2024-09-11T01:49:58.114 INFO:tasks.rados.rados.0.smithi071.stdout:3132: seq_num 1099 ranges {470893=653483,1727593=522054} 2024-09-11T01:49:58.119 INFO:tasks.rados.rados.0.smithi071.stdout:3132: writing smithi07134777-37 from 470893 to 1124376 tid 1 2024-09-11T01:49:58.127 INFO:tasks.rados.rados.0.smithi071.stdout:3132: writing smithi07134777-37 from 1727593 to 2249647 tid 2 2024-09-11T01:49:58.127 INFO:tasks.rados.rados.0.smithi071.stdout:3129: done (3 left) 2024-09-11T01:49:58.127 INFO:tasks.rados.rados.0.smithi071.stdout:3133: copy_from oid 34 from oid 36 current snap is 350 2024-09-11T01:49:58.127 INFO:tasks.rados.rados.0.smithi071.stdout:3130: finishing write tid 2 to smithi07134777-32 2024-09-11T01:49:58.127 INFO:tasks.rados.rados.0.smithi071.stdout:3130: finishing write tid 3 to smithi07134777-32 2024-09-11T01:49:58.128 INFO:tasks.rados.rados.0.smithi071.stdout:3130: finishing write tid 4 to smithi07134777-32 2024-09-11T01:49:58.128 INFO:tasks.rados.rados.0.smithi071.stdout:3130: oid 32 updating version 0 to 1964 2024-09-11T01:49:58.128 INFO:tasks.rados.rados.0.smithi071.stdout:3130: oid 32 version 1964 is already newer than 1962 2024-09-11T01:49:58.128 INFO:tasks.rados.rados.0.smithi071.stdout:3130: oid 32 version 1964 is already newer than 1963 2024-09-11T01:49:58.128 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 1964 (ObjNum 1097 snap 350 seq_num 1097) dirty exists 2024-09-11T01:49:58.128 INFO:tasks.rados.rados.0.smithi071.stdout:3130: left oid 32 (ObjNum 1097 snap 350 seq_num 1097) 2024-09-11T01:49:58.128 INFO:tasks.rados.rados.0.smithi071.stdout:3131: finishing write tid 1 to smithi07134777-22 2024-09-11T01:49:58.128 INFO:tasks.rados.rados.0.smithi071.stdout:3130: done (3 left) 2024-09-11T01:49:58.128 INFO:tasks.rados.rados.0.smithi071.stdout:3134: write oid 7 current snap is 350 2024-09-11T01:49:58.128 INFO:tasks.rados.rados.0.smithi071.stdout:3131: finishing write tid 2 to smithi07134777-22 2024-09-11T01:49:58.128 INFO:tasks.rados.rados.0.smithi071.stdout:3131: finishing write tid 3 to smithi07134777-22 2024-09-11T01:49:58.128 INFO:tasks.rados.rados.0.smithi071.stdout:3131: finishing write tid 4 to smithi07134777-22 2024-09-11T01:49:58.128 INFO:tasks.rados.rados.0.smithi071.stdout:3131: oid 22 updating version 0 to 2041 2024-09-11T01:49:58.128 INFO:tasks.rados.rados.0.smithi071.stdout:3131: oid 22 updating version 2041 to 2042 2024-09-11T01:49:58.129 INFO:tasks.rados.rados.0.smithi071.stdout:3131: oid 22 updating version 2042 to 2043 2024-09-11T01:49:58.129 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 2043 (ObjNum 1098 snap 350 seq_num 1098) dirty exists 2024-09-11T01:49:58.129 INFO:tasks.rados.rados.0.smithi071.stdout:3131: left oid 22 (ObjNum 1098 snap 350 seq_num 1098) 2024-09-11T01:49:58.129 INFO:tasks.rados.rados.0.smithi071.stdout:3132: finishing write tid 1 to smithi07134777-37 2024-09-11T01:49:58.129 INFO:tasks.rados.rados.0.smithi071.stdout:3134: seq_num 1100 ranges {487700=728732,1786075=665712} 2024-09-11T01:49:58.132 INFO:tasks.rados.rados.0.smithi071.stdout:3134: writing smithi07134777-7 from 487700 to 1216432 tid 1 2024-09-11T01:49:58.140 INFO:tasks.rados.rados.0.smithi071.stdout:3134: writing smithi07134777-7 from 1786075 to 2451787 tid 2 2024-09-11T01:49:58.140 INFO:tasks.rados.rados.0.smithi071.stdout:3131: done (3 left) 2024-09-11T01:49:58.140 INFO:tasks.rados.rados.0.smithi071.stdout:3135: snap_create 2024-09-11T01:49:58.140 INFO:tasks.rados.rados.0.smithi071.stdout:3132: finishing write tid 2 to smithi07134777-37 2024-09-11T01:49:58.140 INFO:tasks.rados.rados.0.smithi071.stdout:3132: finishing write tid 3 to smithi07134777-37 2024-09-11T01:49:58.140 INFO:tasks.rados.rados.0.smithi071.stdout:3132: finishing write tid 4 to smithi07134777-37 2024-09-11T01:49:58.140 INFO:tasks.rados.rados.0.smithi071.stdout:3132: oid 37 updating version 0 to 1965 2024-09-11T01:49:58.140 INFO:tasks.rados.rados.0.smithi071.stdout:3132: oid 37 updating version 1965 to 1967 2024-09-11T01:49:58.140 INFO:tasks.rados.rados.0.smithi071.stdout:3132: oid 37 version 1967 is already newer than 1966 2024-09-11T01:49:58.140 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 1967 (ObjNum 1099 snap 350 seq_num 1099) dirty exists 2024-09-11T01:49:58.140 INFO:tasks.rados.rados.0.smithi071.stdout:3132: left oid 37 (ObjNum 1099 snap 350 seq_num 1099) 2024-09-11T01:49:58.141 INFO:tasks.rados.rados.0.smithi071.stdout:3134: finishing write tid 1 to smithi07134777-7 2024-09-11T01:49:58.141 INFO:tasks.rados.rados.0.smithi071.stdout:3133: finishing copy_from racing read to smithi07134777-34 2024-09-11T01:49:58.144 INFO:tasks.rados.rados.0.smithi071.stdout:3133: finishing copy_from to smithi07134777-34 2024-09-11T01:49:58.144 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 1778 (ObjNum 1063 snap 341 seq_num 1063) dirty exists 2024-09-11T01:49:58.144 INFO:tasks.rados.rados.0.smithi071.stdout:3134: finishing write tid 2 to smithi07134777-7 2024-09-11T01:49:58.144 INFO:tasks.rados.rados.0.smithi071.stdout:3134: finishing write tid 3 to smithi07134777-7 2024-09-11T01:49:58.145 INFO:tasks.rados.rados.0.smithi071.stdout:3134: finishing write tid 4 to smithi07134777-7 2024-09-11T01:49:58.145 INFO:tasks.rados.rados.0.smithi071.stdout:3134: oid 7 updating version 0 to 2274 2024-09-11T01:49:58.145 INFO:tasks.rados.rados.0.smithi071.stdout:3134: oid 7 version 2274 is already newer than 2273 2024-09-11T01:49:58.145 INFO:tasks.rados.rados.0.smithi071.stdout:3134: oid 7 updating version 2274 to 2275 2024-09-11T01:49:58.145 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 2275 (ObjNum 1100 snap 350 seq_num 1100) dirty exists 2024-09-11T01:49:58.145 INFO:tasks.rados.rados.0.smithi071.stdout:3134: left oid 7 (ObjNum 1100 snap 350 seq_num 1100) 2024-09-11T01:49:58.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.152+0000 7f75d4dbc640 10 monclient: get_monmap_and_config 2024-09-11T01:49:58.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.152+0000 7f75d4dbc640 10 monclient: build_initial_monmap 2024-09-11T01:49:58.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.152+0000 7f75d4dbc640 10 monclient: monmap: 2024-09-11T01:49:58.154 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:58.154 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:58.154 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:49:58.153200+0000 2024-09-11T01:49:58.154 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:49:58.153200+0000 2024-09-11T01:49:58.154 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:58.154 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:58.154 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:49:58.154 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:49:58.154 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:49:58.154 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:49:58.155 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:49:58.155 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.153+0000 7f75d4dbc640 1 Processor -- start 2024-09-11T01:49:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.153+0000 7f75d4dbc640 1 -- start start 2024-09-11T01:49:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.153+0000 7f75d4dbc640 10 monclient: init 2024-09-11T01:49:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.153+0000 7f75d4dbc640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.153+0000 7f75d4dbc640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-11T01:49:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.153+0000 7f75d4dbc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f75d009d790 addr v1:172.21.15.12:6789/0 2024-09-11T01:49:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.153+0000 7f75d4dbc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f75d009e280 0x7f75d0139f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.153+0000 7f75d4dbc640 10 monclient(hunting): picked mon.noname-b con 0x7f75d009e280 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.153+0000 7f75d4dbc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f75d009e6a0 addr v1:172.21.15.71:6789/0 2024-09-11T01:49:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.153+0000 7f75d4dbc640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75d00a1da0 con 0x7f75d009d790 2024-09-11T01:49:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.153+0000 7f75d4dbc640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75d00a0760 con 0x7f75d009e6a0 2024-09-11T01:49:58.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.153+0000 7f75d4dbc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.153+0000 7f75d4dbc640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f75d00a1fa0 con 0x7f75d009e280 2024-09-11T01:49:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.153+0000 7f75d4dbc640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.153+0000 7f75d4dbc640 10 monclient(hunting): authenticate will time out at 2004.595830s 2024-09-11T01:49:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.153+0000 7f75cdd9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f75d009e280 0x7f75d0139f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.153+0000 7f75ce59c640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f75d009d790 0x7f75d009db70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43850/0 (socket says 172.21.15.12:43850) 2024-09-11T01:49:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.153+0000 7f75cdd9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f75d009e280 0x7f75d0139f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45228/0 (socket says 172.21.15.12:45228) 2024-09-11T01:49:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.153+0000 7f75ce59c640 1 -- 172.21.15.12:0/483232828 learned_addr learned my addr 172.21.15.12:0/483232828 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:49:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.153+0000 7f75cdd9b640 10 monclient(hunting): get_auth_request con 0x7f75d009e280 auth_method 0 2024-09-11T01:49:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.153+0000 7f75cdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.153+0000 7f75cdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.153+0000 7f75cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.154+0000 7f75cdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.154+0000 7f75cdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:58.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.154+0000 7f75cdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:58.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.154+0000 7f75cdd9b640 10 monclient(hunting): handle_auth_done global_id 224171 payload 274 2024-09-11T01:49:58.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.154+0000 7f75cdd9b640 1 -- 172.21.15.12:0/483232828 >> v1:172.21.15.71:6789/0 conn(0x7f75d009e6a0 legacy=0x7f75d013c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:49:58.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.154+0000 7f75cdd9b640 1 -- 172.21.15.12:0/483232828 >> v1:172.21.15.12:6789/0 conn(0x7f75d009d790 legacy=0x7f75d009db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:49:58.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.154+0000 7f75cdd9b640 10 monclient: _finish_hunting 0 2024-09-11T01:49:58.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.154+0000 7f75cdd9b640 1 monclient: found mon.noname-b 2024-09-11T01:49:58.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.154+0000 7f75cdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:58.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.154+0000 7f75cdd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:49:58.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.154+0000 7f75cdd9b640 1 -- 172.21.15.12:0/483232828 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75d009eda0 con 0x7f75d009e280 2024-09-11T01:49:58.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.157+0000 7f75cdd9b640 1 -- 172.21.15.12:0/483232828 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f75d009e280 msgr2=0x7f75d0139f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:58.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.157+0000 7f75cdd9b640 1 --2- 172.21.15.12:0/483232828 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f75d009e280 0x7f75d0139f20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f75c40097d0 tx=0x7f75c402a030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ed96565e9bb4540 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:58.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.157+0000 7f75cd59a640 1 -- 172.21.15.12:0/483232828 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c4037070 con 0x7f75d009e280 2024-09-11T01:49:58.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.157+0000 7f75cd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:58.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.157+0000 7f75cd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:49:58.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.157+0000 7f75cd59a640 10 monclient: dump: 2024-09-11T01:49:58.165 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:58.165 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:58.165 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:58.165 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:58.165 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:58.165 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:58.165 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:58.165 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:58.165 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:58.165 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:58.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.157+0000 7f75cd59a640 10 monclient: _finish_auth 0 2024-09-11T01:49:58.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.157+0000 7f75cd59a640 10 monclient: _check_auth_tickets 2024-09-11T01:49:58.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.157+0000 7f75cd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:58.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.157+0000 7f75cd59a640 1 -- 172.21.15.12:0/483232828 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75c4032640 con 0x7f75d009e280 2024-09-11T01:49:58.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.157+0000 7f75cd59a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:58.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.157+0000 7f75cd59a640 1 -- 172.21.15.12:0/483232828 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c4032a10 con 0x7f75d009e280 2024-09-11T01:49:58.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.157+0000 7f75d4dbc640 5 monclient: authenticate success, global_id 224171 2024-09-11T01:49:58.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.157+0000 7f75d4dbc640 10 monclient: get_monmap_and_config success 2024-09-11T01:49:58.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.157+0000 7f75d4dbc640 4 set_mon_vals no callback set 2024-09-11T01:49:58.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.157+0000 7f75cd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:58.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.157+0000 7f75cd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:49:58.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.157+0000 7f75cd59a640 10 monclient: dump: 2024-09-11T01:49:58.167 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:58.167 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:58.167 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:58.167 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:58.167 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:58.167 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:58.167 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:58.167 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:58.167 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:58.167 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:58.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.157+0000 7f75d4dbc640 10 monclient: shutdown 2024-09-11T01:49:58.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.157+0000 7f75d4dbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:58.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.157+0000 7f75d4dbc640 1 -- 172.21.15.12:0/483232828 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f75d009e280 msgr2=0x7f75d0139f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:58.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.157+0000 7f75d4dbc640 1 --2- 172.21.15.12:0/483232828 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f75d009e280 0x7f75d0139f20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f75c40097d0 tx=0x7f75c402a030 comp rx=0 tx=0).stop 2024-09-11T01:49:58.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.157+0000 7f75d4dbc640 1 -- 172.21.15.12:0/483232828 shutdown_connections 2024-09-11T01:49:58.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.157+0000 7f75d4dbc640 1 --2- 172.21.15.12:0/483232828 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f75d009e280 0x7f75d0139f20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:58.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.157+0000 7f75d4dbc640 1 -- 172.21.15.12:0/483232828 >> 172.21.15.12:0/483232828 conn(0x7f75d00a2d90 msgr2=0x7f75d00a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:58.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.157+0000 7f75d4dbc640 1 -- 172.21.15.12:0/483232828 shutdown_connections 2024-09-11T01:49:58.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.158+0000 7f75d4dbc640 1 -- 172.21.15.12:0/483232828 wait complete. 2024-09-11T01:49:58.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.158+0000 7f75d4dbc640 10 monclient: build_initial_monmap 2024-09-11T01:49:58.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.158+0000 7f75d4dbc640 10 monclient: monmap: 2024-09-11T01:49:58.168 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:49:58.168 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:49:58.169 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:49:58.169 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:49:58.169 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:49:58.169 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:58.169 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:49:58.169 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:49:58.169 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:49:58.169 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:58.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75d4dbc640 1 Processor -- start 2024-09-11T01:49:58.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75d4dbc640 1 -- start start 2024-09-11T01:49:58.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75d4dbc640 10 monclient: init 2024-09-11T01:49:58.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75d4dbc640 10 monclient: _reopen_session rank -1 2024-09-11T01:49:58.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75d4dbc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:49:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75d4dbc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f75d009d790 0x7f75d01cd180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75d4dbc640 10 monclient(hunting): picked mon.noname-c con 0x7f75d009d790 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:49:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75d4dbc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75d009e280 0x7f75d0134ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75d4dbc640 10 monclient(hunting): picked mon.noname-b con 0x7f75d009e280 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:49:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75d4dbc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75d009e6a0 0x7f75d0135430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75d4dbc640 10 monclient(hunting): picked mon.noname-a con 0x7f75d009e6a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:49:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75d4dbc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75d4dbc640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f75d009ce20 con 0x7f75d009e6a0 2024-09-11T01:49:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75d4dbc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75d4dbc640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f75d009cca0 con 0x7f75d009e280 2024-09-11T01:49:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75d4dbc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:49:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75d4dbc640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f75d013db90 con 0x7f75d009d790 2024-09-11T01:49:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75d4dbc640 10 monclient(hunting): _renew_subs 2024-09-11T01:49:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75d4dbc640 10 monclient(hunting): authenticate will time out at 2004.601714s 2024-09-11T01:49:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75ce59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f75d009d790 0x7f75d01cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75ce59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f75d009d790 0x7f75d01cd180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45230/0 (socket says 172.21.15.12:45230) 2024-09-11T01:49:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75ced9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75d009e6a0 0x7f75d0135430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75ce59c640 1 -- 172.21.15.12:0/397169434 learned_addr learned my addr 172.21.15.12:0/397169434 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:49:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75ce59c640 10 monclient(hunting): get_auth_request con 0x7f75d009d790 auth_method 0 2024-09-11T01:49:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75ce59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75ced9d640 10 monclient(hunting): get_auth_request con 0x7f75d009e6a0 auth_method 0 2024-09-11T01:49:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75ced9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75ced9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75ce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75ce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75ce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75ced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:49:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75ced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:49:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.159+0000 7f75ced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:49:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.160+0000 7f75cdd9b640 1 --2- 172.21.15.12:0/397169434 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75d009e280 0x7f75d0134ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.160+0000 7f75cdd9b640 10 monclient(hunting): get_auth_request con 0x7f75d009e280 auth_method 0 2024-09-11T01:49:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.160+0000 7f75cdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:49:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.160+0000 7f75cdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:49:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.160+0000 7f75cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:49:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.160+0000 7f75ced9d640 10 monclient(hunting): handle_auth_done global_id 214287 payload 824 2024-09-11T01:49:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.160+0000 7f75ced9d640 1 -- 172.21.15.12:0/397169434 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f75d009d790 msgr2=0x7f75d01cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.160+0000 7f75ced9d640 1 --2- 172.21.15.12:0/397169434 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f75d009d790 0x7f75d01cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:58.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.160+0000 7f75cdd9b640 1 -- 172.21.15.12:0/397169434 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75d009e280 msgr2=0x7f75d0134ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:49:58.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.160+0000 7f75ced9d640 1 -- 172.21.15.12:0/397169434 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75d009e280 msgr2=0x7f75d0134ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:58.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.160+0000 7f75cdd9b640 1 -- 172.21.15.12:0/397169434 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75d009e280 msgr2=0x7f75d0134ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:49:58.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.160+0000 7f75cdd9b640 1 --2- 172.21.15.12:0/397169434 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75d009e280 0x7f75d0134ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:49:58.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.160+0000 7f75cdd9b640 1 --2- 172.21.15.12:0/397169434 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75d009e280 0x7f75d0134ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:49:58.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.160+0000 7f75ced9d640 1 --2- 172.21.15.12:0/397169434 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75d009e280 0x7f75d0134ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:58.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.160+0000 7f75ced9d640 10 monclient: _finish_hunting 0 2024-09-11T01:49:58.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.160+0000 7f75ced9d640 1 monclient: found mon.noname-a 2024-09-11T01:49:58.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.160+0000 7f75ced9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:49:58.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.160+0000 7f75ced9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:49:58.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.160+0000 7f75ced9d640 1 -- 172.21.15.12:0/397169434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75d00a5aa0 con 0x7f75d009e6a0 2024-09-11T01:49:58.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.160+0000 7f75ce59c640 1 --2- 172.21.15.12:0/397169434 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f75d009d790 0x7f75d01cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:49:58.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.162+0000 7f75ced9d640 1 -- 172.21.15.12:0/397169434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75d009e6a0 msgr2=0x7f75d0135430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:49:58.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.162+0000 7f75ced9d640 1 --2- 172.21.15.12:0/397169434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75d009e6a0 0x7f75d0135430 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f75c800c780 tx=0x7f75c800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:58.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.162+0000 7f75beffd640 1 -- 172.21.15.12:0/397169434 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c8016020 con 0x7f75d009e6a0 2024-09-11T01:49:58.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.162+0000 7f75beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:58.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.162+0000 7f75beffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:49:58.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.162+0000 7f75beffd640 10 monclient: dump: 2024-09-11T01:49:58.174 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:58.174 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:58.174 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:58.174 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:58.175 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:58.175 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:58.175 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:58.175 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:58.175 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:58.175 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:58.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.162+0000 7f75beffd640 10 monclient: _finish_auth 0 2024-09-11T01:49:58.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.162+0000 7f75beffd640 10 monclient: _check_auth_tickets 2024-09-11T01:49:58.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.162+0000 7f75beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:49:58.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.162+0000 7f75beffd640 1 -- 172.21.15.12:0/397169434 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75c800ac60 con 0x7f75d009e6a0 2024-09-11T01:49:58.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.162+0000 7f75d4dbc640 5 monclient: authenticate success, global_id 214287 2024-09-11T01:49:58.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.162+0000 7f75d4dbc640 10 monclient: _renew_subs 2024-09-11T01:49:58.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.162+0000 7f75d4dbc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:58.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.162+0000 7f75d4dbc640 1 -- 172.21.15.12:0/397169434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75d011f540 con 0x7f75d009e6a0 2024-09-11T01:49:58.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.162+0000 7f75beffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:49:58.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.162+0000 7f75beffd640 1 -- 172.21.15.12:0/397169434 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c8012660 con 0x7f75d009e6a0 2024-09-11T01:49:58.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.162+0000 7f75beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:49:58.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.162+0000 7f75beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:49:58.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.162+0000 7f75beffd640 10 monclient: dump: 2024-09-11T01:49:58.177 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:49:58.177 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:49:58.177 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:58.177 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:49:58.177 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:49:58.177 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:49:58.177 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:49:58.177 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:49:58.177 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:49:58.178 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:49:58.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.162+0000 7f75d4dbc640 10 monclient: _renew_subs 2024-09-11T01:49:58.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.162+0000 7f75d4dbc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:58.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.162+0000 7f75bffff640 4 set_mon_vals no callback set 2024-09-11T01:49:58.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.162+0000 7f75d4dbc640 1 -- 172.21.15.12:0/397169434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75d01200a0 con 0x7f75d009e6a0 2024-09-11T01:49:58.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.163+0000 7f75d4dbc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:58.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.163+0000 7f75beffd640 1 -- 172.21.15.12:0/397169434 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f75c8012840 con 0x7f75d009e6a0 2024-09-11T01:49:58.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.163+0000 7f75d4dbc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:58.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.163+0000 7f75d4dbc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:58.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.163+0000 7f75d4dbc640 1 -- 172.21.15.12:0/397169434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75d00a2660 con 0x7f75d009e6a0 2024-09-11T01:49:58.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.163+0000 7f75beffd640 1 --2- 172.21.15.12:0/397169434 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f75b003e210 0x7f75b00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:49:58.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.164+0000 7f75beffd640 1 -- 172.21.15.12:0/397169434 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(700..700 src has 507..700) ==== 4863+0+0 (secure 0 0 0) 0x7f75c80531b0 con 0x7f75d009e6a0 2024-09-11T01:49:58.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.164+0000 7f75ce59c640 1 --2- 172.21.15.12:0/397169434 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f75b003e210 0x7f75b00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:49:58.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.164+0000 7f75ce59c640 10 monclient: get_auth_request con 0x7f75b003e210 auth_method 0 2024-09-11T01:49:58.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.166+0000 7f75ce59c640 1 --2- 172.21.15.12:0/397169434 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f75b003e210 0x7f75b00406d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f75c0007d50 tx=0x7f75c0007ca0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:49:58.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.168+0000 7f75beffd640 1 -- 172.21.15.12:0/397169434 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f75d00a2660 con 0x7f75d009e6a0 2024-09-11T01:49:58.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.168+0000 7f75beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:49:58.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.168+0000 7f75beffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:49:58.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.426+0000 7f75d4dbc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:49:58.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.426+0000 7f75d4dbc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:49:58.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.426+0000 7f75d4dbc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:49:58.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.426+0000 7f75d4dbc640 1 -- 172.21.15.12:0/397169434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f75d009dbf0 con 0x7f75d009e6a0 2024-09-11T01:49:58.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.426+0000 7f75beffd640 1 -- 172.21.15.12:0/397169434 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f75c8020020 con 0x7f75d009e6a0 2024-09-11T01:49:58.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.426+0000 7f75beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:49:58.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.426+0000 7f75beffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:49:58.428 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":192,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:49:58.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.428+0000 7f75d4dbc640 1 -- 172.21.15.12:0/397169434 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f75b003e210 msgr2=0x7f75b00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:58.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.428+0000 7f75d4dbc640 1 --2- 172.21.15.12:0/397169434 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f75b003e210 0x7f75b00406d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f75c0007d50 tx=0x7f75c0007ca0 comp rx=0 tx=0).stop 2024-09-11T01:49:58.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.428+0000 7f75d4dbc640 10 monclient: shutdown 2024-09-11T01:49:58.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.428+0000 7f75d4dbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:49:58.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.428+0000 7f75d4dbc640 1 -- 172.21.15.12:0/397169434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75d009e6a0 msgr2=0x7f75d0135430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:49:58.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.428+0000 7f75d4dbc640 1 --2- 172.21.15.12:0/397169434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75d009e6a0 0x7f75d0135430 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f75c800c780 tx=0x7f75c800cc40 comp rx=0 tx=0).stop 2024-09-11T01:49:58.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.428+0000 7f75d4dbc640 1 -- 172.21.15.12:0/397169434 shutdown_connections 2024-09-11T01:49:58.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.428+0000 7f75d4dbc640 1 --2- 172.21.15.12:0/397169434 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f75b003e210 0x7f75b00406d0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:58.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.428+0000 7f75d4dbc640 1 --2- 172.21.15.12:0/397169434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75d009e6a0 0x7f75d0135430 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:58.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.428+0000 7f75d4dbc640 1 --2- 172.21.15.12:0/397169434 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75d009e280 0x7f75d0134ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:58.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.428+0000 7f75d4dbc640 1 --2- 172.21.15.12:0/397169434 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f75d009d790 0x7f75d01cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:49:58.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.428+0000 7f75d4dbc640 1 -- 172.21.15.12:0/397169434 >> 172.21.15.12:0/397169434 conn(0x7f75d00a2d90 msgr2=0x7f75d0074100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:49:58.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.428+0000 7f75d4dbc640 1 -- 172.21.15.12:0/397169434 shutdown_connections 2024-09-11T01:49:58.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:49:58.428+0000 7f75d4dbc640 1 -- 172.21.15.12:0/397169434 wait complete. 2024-09-11T01:49:58.828 INFO:tasks.rados.rados.0.smithi071.stdout:3132: done (3 left) 2024-09-11T01:49:58.828 INFO:tasks.rados.rados.0.smithi071.stdout:3133: done (2 left) 2024-09-11T01:49:58.828 INFO:tasks.rados.rados.0.smithi071.stdout:3134: done (1 left) 2024-09-11T01:49:58.828 INFO:tasks.rados.rados.0.smithi071.stdout:3135: done (0 left) 2024-09-11T01:49:58.829 INFO:tasks.rados.rados.0.smithi071.stdout:3136: write (excl) oid 9 current snap is 351 2024-09-11T01:49:58.829 INFO:tasks.rados.rados.0.smithi071.stdout:3136: seq_num 1101 ranges {504507=403981,1328204=409370,2252362=252145} 2024-09-11T01:49:58.833 INFO:tasks.rados.rados.0.smithi071.stdout:3136: writing smithi07134777-9 from 504507 to 908488 tid 1 2024-09-11T01:49:58.837 INFO:tasks.rados.rados.0.smithi071.stdout:3136: writing smithi07134777-9 from 1328204 to 1737574 tid 2 2024-09-11T01:49:58.841 INFO:tasks.rados.rados.0.smithi071.stdout:3136: writing smithi07134777-9 from 2252362 to 2504507 tid 3 2024-09-11T01:49:58.842 INFO:tasks.rados.rados.0.smithi071.stdout:3137: delete oid 1 current snap is 351 2024-09-11T01:49:58.842 INFO:tasks.rados.rados.0.smithi071.stdout:3136: finishing write tid 1 to smithi07134777-9 2024-09-11T01:49:58.842 INFO:tasks.rados.rados.0.smithi071.stdout:3136: finishing write tid 2 to smithi07134777-9 2024-09-11T01:49:58.844 INFO:tasks.rados.rados.0.smithi071.stdout:3136: finishing write tid 3 to smithi07134777-9 2024-09-11T01:49:58.844 INFO:tasks.rados.rados.0.smithi071.stdout:3136: finishing write tid 4 to smithi07134777-9 2024-09-11T01:49:58.845 INFO:tasks.rados.rados.0.smithi071.stdout:3136: finishing write tid 5 to smithi07134777-9 2024-09-11T01:49:58.845 INFO:tasks.rados.rados.0.smithi071.stdout:3136: oid 9 updating version 0 to 1034 2024-09-11T01:49:58.845 INFO:tasks.rados.rados.0.smithi071.stdout:3136: oid 9 version 1034 is already newer than 1032 2024-09-11T01:49:58.845 INFO:tasks.rados.rados.0.smithi071.stdout:3136: oid 9 version 1034 is already newer than 1033 2024-09-11T01:49:58.845 INFO:tasks.rados.rados.0.smithi071.stdout:3136: oid 9 updating version 1034 to 1035 2024-09-11T01:49:58.845 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 1035 (ObjNum 1101 snap 351 seq_num 1101) dirty exists 2024-09-11T01:49:58.845 INFO:tasks.rados.rados.0.smithi071.stdout:3136: left oid 9 (ObjNum 1101 snap 351 seq_num 1101) 2024-09-11T01:49:58.846 INFO:tasks.rados.rados.0.smithi071.stdout:3136: done (1 left) 2024-09-11T01:49:58.846 INFO:tasks.rados.rados.0.smithi071.stdout:3137: done (0 left) 2024-09-11T01:49:58.846 INFO:tasks.rados.rados.0.smithi071.stdout:3138: delete oid 37 current snap is 351 2024-09-11T01:49:58.853 INFO:tasks.rados.rados.0.smithi071.stdout:3138: done (0 left) 2024-09-11T01:49:58.853 INFO:tasks.rados.rados.0.smithi071.stdout:3139: write (excl) oid 14 current snap is 351 2024-09-11T01:49:58.853 INFO:tasks.rados.rados.0.smithi071.stdout:3139: seq_num 1104 ranges {554928=746081,2020003=534925} 2024-09-11T01:49:58.859 INFO:tasks.rados.rados.0.smithi071.stdout:3139: writing smithi07134777-14 from 554928 to 1301009 tid 1 2024-09-11T01:49:58.867 INFO:tasks.rados.rados.0.smithi071.stdout:3139: writing smithi07134777-14 from 2020003 to 2554928 tid 2 2024-09-11T01:49:58.867 INFO:tasks.rados.rados.0.smithi071.stdout:3140: snap_create 2024-09-11T01:49:58.867 INFO:tasks.rados.rados.0.smithi071.stdout:3139: finishing write tid 1 to smithi07134777-14 2024-09-11T01:49:58.872 INFO:tasks.rados.rados.0.smithi071.stdout:3139: finishing write tid 2 to smithi07134777-14 2024-09-11T01:49:58.872 INFO:tasks.rados.rados.0.smithi071.stdout:3139: finishing write tid 3 to smithi07134777-14 2024-09-11T01:49:58.872 INFO:tasks.rados.rados.0.smithi071.stdout:3139: finishing write tid 4 to smithi07134777-14 2024-09-11T01:49:58.872 INFO:tasks.rados.rados.0.smithi071.stdout:3139: oid 14 updating version 0 to 2277 2024-09-11T01:49:58.872 INFO:tasks.rados.rados.0.smithi071.stdout:3139: oid 14 version 2277 is already newer than 2276 2024-09-11T01:49:58.872 INFO:tasks.rados.rados.0.smithi071.stdout:3139: oid 14 updating version 2277 to 2278 2024-09-11T01:49:58.872 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 2278 (ObjNum 1104 snap 351 seq_num 1104) dirty exists 2024-09-11T01:49:58.873 INFO:tasks.rados.rados.0.smithi071.stdout:3139: left oid 14 (ObjNum 1104 snap 351 seq_num 1104) 2024-09-11T01:49:59.859 INFO:tasks.rados.rados.0.smithi071.stdout:3139: done (1 left) 2024-09-11T01:49:59.859 INFO:tasks.rados.rados.0.smithi071.stdout:3140: done (0 left) 2024-09-11T01:49:59.859 INFO:tasks.rados.rados.0.smithi071.stdout:3141: snap_remove snap 300 2024-09-11T01:50:00.861 INFO:tasks.rados.rados.0.smithi071.stdout:3141: done (0 left) 2024-09-11T01:50:00.861 INFO:tasks.rados.rados.0.smithi071.stdout:3142: read oid 24 snap -1 2024-09-11T01:50:00.861 INFO:tasks.rados.rados.0.smithi071.stdout:3142: expect (ObjNum 1060 snap 340 seq_num 1060) 2024-09-11T01:50:00.862 INFO:tasks.rados.rados.0.smithi071.stdout:3143: snap_remove snap 319 2024-09-11T01:50:01.442 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:50:01.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.553+0000 7f6260433640 10 monclient: get_monmap_and_config 2024-09-11T01:50:01.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.553+0000 7f6260433640 10 monclient: build_initial_monmap 2024-09-11T01:50:01.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.553+0000 7f6260433640 10 monclient: monmap: 2024-09-11T01:50:01.557 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:50:01.557 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:01.557 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:50:01.555613+0000 2024-09-11T01:50:01.557 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:50:01.555613+0000 2024-09-11T01:50:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:50:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:50:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:50:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:50:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:50:01.558 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:50:01.559 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:01.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.554+0000 7f6260433640 1 Processor -- start 2024-09-11T01:50:01.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.554+0000 7f6260433640 1 -- start start 2024-09-11T01:50:01.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.554+0000 7f6260433640 10 monclient: init 2024-09-11T01:50:01.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f6260433640 10 monclient: _reopen_session rank -1 2024-09-11T01:50:01.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f6260433640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-11T01:50:01.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f6260433640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6258138230 0x7f6258138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:01.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f6260433640 10 monclient(hunting): picked mon.noname-a con 0x7f6258138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:50:01.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f6260433640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f625809d030 0x7f62580a5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:01.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f6260433640 10 monclient(hunting): picked mon.noname-b con 0x7f625809d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:50:01.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f6260433640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f625809d460 0x7f62580a5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:01.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f6260433640 10 monclient(hunting): picked mon.noname-c con 0x7f625809d460 addr v2:172.21.15.71:3300/0 2024-09-11T01:50:01.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f6260433640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:01.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f6260433640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f625809c9c0 con 0x7f6258138230 2024-09-11T01:50:01.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f6260433640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:01.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f6260433640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f62580a6e30 con 0x7f625809d460 2024-09-11T01:50:01.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f6260433640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:01.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f6260433640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f625809ba20 con 0x7f625809d030 2024-09-11T01:50:01.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f6260433640 10 monclient(hunting): _renew_subs 2024-09-11T01:50:01.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f6260433640 10 monclient(hunting): authenticate will time out at 2007.998375s 2024-09-11T01:50:01.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625d9ce640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f625809d030 0x7f62580a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:01.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625e1cf640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6258138230 0x7f6258138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:01.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625d9ce640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f625809d030 0x7f62580a5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45236/0 (socket says 172.21.15.12:45236) 2024-09-11T01:50:01.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625e1cf640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6258138230 0x7f6258138610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40000/0 (socket says 172.21.15.12:40000) 2024-09-11T01:50:01.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625d9ce640 1 -- 172.21.15.12:0/2642818235 learned_addr learned my addr 172.21.15.12:0/2642818235 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:50:01.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625d9ce640 10 monclient(hunting): get_auth_request con 0x7f625809d030 auth_method 0 2024-09-11T01:50:01.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625d9ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:01.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625d9ce640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:01.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625d9ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:01.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625e1cf640 10 monclient(hunting): get_auth_request con 0x7f6258138230 auth_method 0 2024-09-11T01:50:01.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625e1cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:01.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625e1cf640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:01.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625e1cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:01.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625e9d0640 1 --2- 172.21.15.12:0/2642818235 >> v2:172.21.15.71:3300/0 conn(0x7f625809d460 0x7f62580a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:01.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625e9d0640 10 monclient(hunting): get_auth_request con 0x7f625809d460 auth_method 0 2024-09-11T01:50:01.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625e9d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:01.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625e9d0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:01.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625e9d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:01.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625d9ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:01.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625d9ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:01.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625d9ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:01.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625e1cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:01.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625e1cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:01.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625e1cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:01.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625e9d0640 1 -- 172.21.15.12:0/2642818235 >> v2:172.21.15.71:3300/0 conn(0x7f625809d460 msgr2=0x7f62580a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:50:01.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625e9d0640 1 -- 172.21.15.12:0/2642818235 >> v2:172.21.15.71:3300/0 conn(0x7f625809d460 msgr2=0x7f62580a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:50:01.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625e9d0640 1 --2- 172.21.15.12:0/2642818235 >> v2:172.21.15.71:3300/0 conn(0x7f625809d460 0x7f62580a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:50:01.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625e9d0640 1 --2- 172.21.15.12:0/2642818235 >> v2:172.21.15.71:3300/0 conn(0x7f625809d460 0x7f62580a5560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:50:01.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625d9ce640 10 monclient(hunting): handle_auth_done global_id 224177 payload 274 2024-09-11T01:50:01.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625d9ce640 1 -- 172.21.15.12:0/2642818235 >> v2:172.21.15.71:3300/0 conn(0x7f625809d460 msgr2=0x7f62580a5560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:50:01.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625d9ce640 1 --2- 172.21.15.12:0/2642818235 >> v2:172.21.15.71:3300/0 conn(0x7f625809d460 0x7f62580a5560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:01.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625d9ce640 1 -- 172.21.15.12:0/2642818235 >> v2:172.21.15.12:3300/0 conn(0x7f6258138230 msgr2=0x7f6258138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:50:01.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625d9ce640 1 --2- 172.21.15.12:0/2642818235 >> v2:172.21.15.12:3300/0 conn(0x7f6258138230 0x7f6258138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:01.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625d9ce640 10 monclient: _finish_hunting 0 2024-09-11T01:50:01.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625d9ce640 1 monclient: found mon.noname-b 2024-09-11T01:50:01.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625d9ce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:50:01.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625d9ce640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:50:01.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625d9ce640 1 -- 172.21.15.12:0/2642818235 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62580a66a0 con 0x7f625809d030 2024-09-11T01:50:01.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.555+0000 7f625e1cf640 1 --2- 172.21.15.12:0/2642818235 >> v2:172.21.15.12:3300/0 conn(0x7f6258138230 0x7f6258138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:50:01.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.558+0000 7f625d9ce640 1 -- 172.21.15.12:0/2642818235 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f625809d030 msgr2=0x7f62580a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:01.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.558+0000 7f625d9ce640 1 --2- 172.21.15.12:0/2642818235 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f625809d030 0x7f62580a5020 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f625806e500 tx=0x7f624802a0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b5c3a57da85ac38 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:01.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.558+0000 7f625d1cd640 1 -- 172.21.15.12:0/2642818235 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6248037070 con 0x7f625809d030 2024-09-11T01:50:01.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.558+0000 7f625d1cd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:01.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.558+0000 7f625d1cd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:50:01.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.558+0000 7f625d1cd640 10 monclient: dump: 2024-09-11T01:50:01.576 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:01.577 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:01.577 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:01.577 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:01.577 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:01.577 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:01.577 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:01.577 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:01.577 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:01.577 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:01.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.558+0000 7f625d1cd640 10 monclient: _finish_auth 0 2024-09-11T01:50:01.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.558+0000 7f625d1cd640 10 monclient: _check_auth_tickets 2024-09-11T01:50:01.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.558+0000 7f625d1cd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:50:01.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.558+0000 7f625d1cd640 1 -- 172.21.15.12:0/2642818235 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6248032640 con 0x7f625809d030 2024-09-11T01:50:01.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.558+0000 7f625d1cd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:50:01.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.558+0000 7f625d1cd640 1 -- 172.21.15.12:0/2642818235 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6248032a10 con 0x7f625809d030 2024-09-11T01:50:01.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.558+0000 7f6260433640 5 monclient: authenticate success, global_id 224177 2024-09-11T01:50:01.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.558+0000 7f6260433640 10 monclient: get_monmap_and_config success 2024-09-11T01:50:01.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.558+0000 7f625d1cd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:01.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.558+0000 7f6260433640 4 set_mon_vals no callback set 2024-09-11T01:50:01.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.558+0000 7f625d1cd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:50:01.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.558+0000 7f625d1cd640 10 monclient: dump: 2024-09-11T01:50:01.578 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:01.579 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:01.579 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:01.579 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:01.579 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:01.579 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:01.579 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:01.579 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:01.579 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:01.579 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:01.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.559+0000 7f6260433640 10 monclient: shutdown 2024-09-11T01:50:01.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.559+0000 7f6260433640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:50:01.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.559+0000 7f6260433640 1 -- 172.21.15.12:0/2642818235 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f625809d030 msgr2=0x7f62580a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:01.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.559+0000 7f6260433640 1 --2- 172.21.15.12:0/2642818235 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f625809d030 0x7f62580a5020 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f625806e500 tx=0x7f624802a0b0 comp rx=0 tx=0).stop 2024-09-11T01:50:01.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.559+0000 7f6260433640 1 -- 172.21.15.12:0/2642818235 shutdown_connections 2024-09-11T01:50:01.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.559+0000 7f6260433640 1 --2- 172.21.15.12:0/2642818235 >> v2:172.21.15.71:3300/0 conn(0x7f625809d460 0x7f62580a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:01.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.559+0000 7f6260433640 1 --2- 172.21.15.12:0/2642818235 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f625809d030 0x7f62580a5020 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:01.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.559+0000 7f6260433640 1 --2- 172.21.15.12:0/2642818235 >> v2:172.21.15.12:3300/0 conn(0x7f6258138230 0x7f6258138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:01.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.559+0000 7f6260433640 1 -- 172.21.15.12:0/2642818235 >> 172.21.15.12:0/2642818235 conn(0x7f62580a7120 msgr2=0x7f62580a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:50:01.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.559+0000 7f6260433640 1 -- 172.21.15.12:0/2642818235 shutdown_connections 2024-09-11T01:50:01.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.559+0000 7f6260433640 1 -- 172.21.15.12:0/2642818235 wait complete. 2024-09-11T01:50:01.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.560+0000 7f6260433640 10 monclient: build_initial_monmap 2024-09-11T01:50:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.560+0000 7f6260433640 10 monclient: monmap: 2024-09-11T01:50:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:50:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:50:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:50:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:50:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:50:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:01.582 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:50:01.582 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:50:01.582 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:50:01.582 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:01.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.560+0000 7f6260433640 1 Processor -- start 2024-09-11T01:50:01.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.560+0000 7f6260433640 1 -- start start 2024-09-11T01:50:01.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.560+0000 7f6260433640 10 monclient: init 2024-09-11T01:50:01.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.560+0000 7f6260433640 10 monclient: _reopen_session rank -1 2024-09-11T01:50:01.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f6260433640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:50:01.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f6260433640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f625809d030 0x7f6258136da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:01.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f6260433640 10 monclient(hunting): picked mon.noname-c con 0x7f625809d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:50:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f6260433640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f625809d460 0x7f62581372e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f6260433640 10 monclient(hunting): picked mon.noname-b con 0x7f625809d460 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:50:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f6260433640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6258138230 0x7f6258120c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f6260433640 10 monclient(hunting): picked mon.noname-a con 0x7f6258138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:50:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f6260433640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f6260433640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f625806f970 con 0x7f6258138230 2024-09-11T01:50:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f6260433640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f6260433640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f625806f7f0 con 0x7f625809d460 2024-09-11T01:50:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f6260433640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f6260433640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f62580a5aa0 con 0x7f625809d030 2024-09-11T01:50:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f6260433640 10 monclient(hunting): _renew_subs 2024-09-11T01:50:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f6260433640 10 monclient(hunting): authenticate will time out at 2008.004092s 2024-09-11T01:50:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e1cf640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f625809d030 0x7f6258136da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e1cf640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f625809d030 0x7f6258136da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45252/0 (socket says 172.21.15.12:45252) 2024-09-11T01:50:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e1cf640 1 -- 172.21.15.12:0/2013615426 learned_addr learned my addr 172.21.15.12:0/2013615426 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:50:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e1cf640 10 monclient(hunting): get_auth_request con 0x7f625809d030 auth_method 0 2024-09-11T01:50:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e1cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e1cf640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e1cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e9d0640 1 --2- 172.21.15.12:0/2013615426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6258138230 0x7f6258120c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e9d0640 10 monclient(hunting): get_auth_request con 0x7f6258138230 auth_method 0 2024-09-11T01:50:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e9d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e9d0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e9d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e1cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e1cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e1cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625d9ce640 1 --2- 172.21.15.12:0/2013615426 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f625809d460 0x7f62581372e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625d9ce640 10 monclient(hunting): get_auth_request con 0x7f625809d460 auth_method 0 2024-09-11T01:50:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625d9ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625d9ce640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:01.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625d9ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:01.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e9d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:01.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e9d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:01.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e9d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:01.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625d9ce640 1 -- 172.21.15.12:0/2013615426 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f625809d460 msgr2=0x7f62581372e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:50:01.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625d9ce640 1 -- 172.21.15.12:0/2013615426 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f625809d460 msgr2=0x7f62581372e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:50:01.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625d9ce640 1 --2- 172.21.15.12:0/2013615426 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f625809d460 0x7f62581372e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:50:01.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625d9ce640 1 --2- 172.21.15.12:0/2013615426 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f625809d460 0x7f62581372e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:50:01.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e1cf640 10 monclient(hunting): handle_auth_done global_id 224180 payload 824 2024-09-11T01:50:01.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e1cf640 1 -- 172.21.15.12:0/2013615426 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f625809d460 msgr2=0x7f62581372e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:50:01.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e1cf640 1 --2- 172.21.15.12:0/2013615426 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f625809d460 0x7f62581372e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:01.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e1cf640 1 -- 172.21.15.12:0/2013615426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6258138230 msgr2=0x7f6258120c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:01.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e1cf640 1 --2- 172.21.15.12:0/2013615426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6258138230 0x7f6258120c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:01.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e1cf640 10 monclient: _finish_hunting 0 2024-09-11T01:50:01.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e1cf640 1 monclient: found mon.noname-c 2024-09-11T01:50:01.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e1cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:50:01.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e1cf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:50:01.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e1cf640 1 -- 172.21.15.12:0/2013615426 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6258122400 con 0x7f625809d030 2024-09-11T01:50:01.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.561+0000 7f625e9d0640 1 --2- 172.21.15.12:0/2013615426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6258138230 0x7f6258120c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:50:01.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.563+0000 7f625e1cf640 1 -- 172.21.15.12:0/2013615426 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f625809d030 msgr2=0x7f6258136da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:01.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.563+0000 7f625e1cf640 1 --2- 172.21.15.12:0/2013615426 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f625809d030 0x7f6258136da0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f625400ec10 tx=0x7f625400c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:01.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.563+0000 7f624effd640 1 -- 172.21.15.12:0/2013615426 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6254017070 con 0x7f625809d030 2024-09-11T01:50:01.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.563+0000 7f624effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:01.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.563+0000 7f624effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:50:01.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.563+0000 7f624effd640 10 monclient: dump: 2024-09-11T01:50:01.587 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:01.588 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:01.588 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:01.588 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:01.588 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:01.588 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:01.588 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:01.588 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:01.588 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:01.588 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:01.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.564+0000 7f624effd640 10 monclient: _finish_auth 0 2024-09-11T01:50:01.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.564+0000 7f624effd640 10 monclient: _check_auth_tickets 2024-09-11T01:50:01.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.564+0000 7f624effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:50:01.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.564+0000 7f624effd640 1 -- 172.21.15.12:0/2013615426 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f625400ce40 con 0x7f625809d030 2024-09-11T01:50:01.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.564+0000 7f6260433640 5 monclient: authenticate success, global_id 224180 2024-09-11T01:50:01.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.564+0000 7f6260433640 10 monclient: _renew_subs 2024-09-11T01:50:01.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.564+0000 7f6260433640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:50:01.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.564+0000 7f6260433640 1 -- 172.21.15.12:0/2013615426 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62580a1410 con 0x7f625809d030 2024-09-11T01:50:01.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.564+0000 7f624effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:50:01.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.564+0000 7f624effd640 1 -- 172.21.15.12:0/2013615426 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6254013690 con 0x7f625809d030 2024-09-11T01:50:01.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.564+0000 7f624effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:01.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.564+0000 7f624effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:50:01.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.564+0000 7f624effd640 10 monclient: dump: 2024-09-11T01:50:01.590 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:01.590 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:01.590 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:01.590 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:01.590 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:01.590 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:01.590 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:01.590 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:01.590 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:01.590 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:01.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.564+0000 7f6260433640 10 monclient: _renew_subs 2024-09-11T01:50:01.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.564+0000 7f6260433640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:50:01.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.564+0000 7f6260433640 1 -- 172.21.15.12:0/2013615426 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6258123ec0 con 0x7f625809d030 2024-09-11T01:50:01.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.564+0000 7f625c9cc640 4 set_mon_vals no callback set 2024-09-11T01:50:01.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.565+0000 7f6260433640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:01.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.565+0000 7f6260433640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:01.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.565+0000 7f6260433640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:50:01.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.565+0000 7f6260433640 1 -- 172.21.15.12:0/2013615426 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6258124580 con 0x7f625809d030 2024-09-11T01:50:01.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.565+0000 7f625e1cf640 1 --2- 172.21.15.12:0/2013615426 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f625809d030 0x7f6258136da0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f625400ec10 tx=0x7f625400c520 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7f6254052c30 osd_map(703..703 src has 507..703) 2024-09-11T01:50:01.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.565+0000 7f624effd640 1 -- 172.21.15.12:0/2013615426 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f6254024060 con 0x7f625809d030 2024-09-11T01:50:01.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.565+0000 7f624effd640 1 --2- 172.21.15.12:0/2013615426 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f623c03e260 0x7f623c040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:01.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.566+0000 7f625d9ce640 1 --2- 172.21.15.12:0/2013615426 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f623c03e260 0x7f623c040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:01.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.566+0000 7f625d9ce640 10 monclient: get_auth_request con 0x7f623c03e260 auth_method 0 2024-09-11T01:50:01.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.568+0000 7f625d9ce640 1 --2- 172.21.15.12:0/2013615426 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f623c03e260 0x7f623c040720 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f62580a17e0 tx=0x7f624802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:01.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:01.570+0000 7f624effd640 1 -- 172.21.15.12:0/2013615426 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(703..703 src has 507..703) ==== 4815+0+0 (secure 0 0 0) 0x7f6254052c30 con 0x7f625809d030 2024-09-11T01:50:01.877 INFO:tasks.rados.rados.0.smithi071.stdout:3143: done (1 left) 2024-09-11T01:50:01.877 INFO:tasks.rados.rados.0.smithi071.stdout:3144: read oid 47 snap -1 2024-09-11T01:50:01.877 INFO:tasks.rados.rados.0.smithi071.stdout:3144: expect deleted 2024-09-11T01:50:01.877 INFO:tasks.rados.rados.0.smithi071.stdout:3145: write (excl) oid 12 current snap is 352 2024-09-11T01:50:01.877 INFO:tasks.rados.rados.0.smithi071.stdout:3145: seq_num 1105 ranges {571735=421330,1562132=416708} 2024-09-11T01:50:01.882 INFO:tasks.rados.rados.0.smithi071.stdout:3145: writing smithi07134777-12 from 571735 to 993065 tid 1 2024-09-11T01:50:01.889 INFO:tasks.rados.rados.0.smithi071.stdout:3145: writing smithi07134777-12 from 1562132 to 1978840 tid 2 2024-09-11T01:50:01.889 INFO:tasks.rados.rados.0.smithi071.stdout:3146: read oid 43 snap 312 2024-09-11T01:50:01.889 INFO:tasks.rados.rados.0.smithi071.stdout:3146: expect deleted 2024-09-11T01:50:01.889 INFO:tasks.rados.rados.0.smithi071.stdout:3147: rollback oid 7 current snap is 352 2024-09-11T01:50:01.889 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 7 to 325 2024-09-11T01:50:01.889 INFO:tasks.rados.rados.0.smithi071.stdout:3142: expect (ObjNum 1060 snap 340 seq_num 1060) 2024-09-11T01:50:01.889 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:50:02.000 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3039287 2024-09-11T01:50:02.080 INFO:tasks.rados.rados.0.smithi071.stdout:3145: finishing write tid 1 to smithi07134777-12 2024-09-11T01:50:02.080 INFO:tasks.rados.rados.0.smithi071.stdout:3145: finishing write tid 2 to smithi07134777-12 2024-09-11T01:50:02.080 INFO:tasks.rados.rados.0.smithi071.stdout:3145: finishing write tid 3 to smithi07134777-12 2024-09-11T01:50:02.080 INFO:tasks.rados.rados.0.smithi071.stdout:3145: finishing write tid 4 to smithi07134777-12 2024-09-11T01:50:02.080 INFO:tasks.rados.rados.0.smithi071.stdout:3145: oid 12 updating version 0 to 2169 2024-09-11T01:50:02.080 INFO:tasks.rados.rados.0.smithi071.stdout:3145: oid 12 updating version 2169 to 2171 2024-09-11T01:50:02.080 INFO:tasks.rados.rados.0.smithi071.stdout:3145: oid 12 version 2171 is already newer than 2170 2024-09-11T01:50:02.080 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 2171 (ObjNum 1105 snap 352 seq_num 1105) dirty exists 2024-09-11T01:50:02.081 INFO:tasks.rados.rados.0.smithi071.stdout:3145: left oid 12 (ObjNum 1105 snap 352 seq_num 1105) 2024-09-11T01:50:02.081 INFO:tasks.rados.rados.0.smithi071.stdout:3142: done (4 left) 2024-09-11T01:50:02.081 INFO:tasks.rados.rados.0.smithi071.stdout:3144: done (3 left) 2024-09-11T01:50:02.081 INFO:tasks.rados.rados.0.smithi071.stdout:3145: done (2 left) 2024-09-11T01:50:02.081 INFO:tasks.rados.rados.0.smithi071.stdout:3146: done (1 left) 2024-09-11T01:50:02.081 INFO:tasks.rados.rados.0.smithi071.stdout:3148: write oid 1 current snap is 352 2024-09-11T01:50:02.081 INFO:tasks.rados.rados.0.smithi071.stdout:3147: finishing rollback tid 0 to smithi07134777-7 2024-09-11T01:50:02.081 INFO:tasks.rados.rados.0.smithi071.stdout:3147: finishing rollback tid 1 to smithi07134777-7 2024-09-11T01:50:02.081 INFO:tasks.rados.rados.0.smithi071.stdout:3147: finishing rollback tid 2 to smithi07134777-7 2024-09-11T01:50:02.081 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 2302 (ObjNum 1014 snap 322 seq_num 1014) dirty exists 2024-09-11T01:50:02.081 INFO:tasks.rados.rados.0.smithi071.stdout:3148: seq_num 1106 ranges {588542=496579,1620614=676719} 2024-09-11T01:50:02.085 INFO:tasks.rados.rados.0.smithi071.stdout:3148: writing smithi07134777-1 from 588542 to 1085121 tid 1 2024-09-11T01:50:02.092 INFO:tasks.rados.rados.0.smithi071.stdout:3148: writing smithi07134777-1 from 1620614 to 2297333 tid 2 2024-09-11T01:50:02.092 INFO:tasks.rados.rados.0.smithi071.stdout:3147: done (1 left) 2024-09-11T01:50:02.092 INFO:tasks.rados.rados.0.smithi071.stdout:3149: delete oid 49 current snap is 352 2024-09-11T01:50:02.093 INFO:tasks.rados.rados.0.smithi071.stdout:3148: finishing write tid 1 to smithi07134777-1 2024-09-11T01:50:02.095 INFO:tasks.rados.rados.0.smithi071.stdout:3149: done (1 left) 2024-09-11T01:50:02.095 INFO:tasks.rados.rados.0.smithi071.stdout:3150: copy_from oid 39 from oid 27 current snap is 352 2024-09-11T01:50:02.096 INFO:tasks.rados.rados.0.smithi071.stdout:3151: copy_from oid 6 from oid 2 current snap is 352 2024-09-11T01:50:02.096 INFO:tasks.rados.rados.0.smithi071.stdout:3152: snap_remove snap 277 2024-09-11T01:50:02.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.393+0000 7f624effd640 1 -- 172.21.15.12:0/2013615426 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6258124580 con 0x7f625809d030 2024-09-11T01:50:02.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.393+0000 7f624effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:02.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.393+0000 7f624effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:50:02.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.652+0000 7f6260433640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:50:02.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.652+0000 7f6260433640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:50:02.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.652+0000 7f6260433640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:50:02.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.652+0000 7f6260433640 1 -- 172.21.15.12:0/2013615426 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f62580a5e20 con 0x7f625809d030 2024-09-11T01:50:02.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.652+0000 7f624effd640 1 -- 172.21.15.12:0/2013615426 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f62540123d0 con 0x7f625809d030 2024-09-11T01:50:02.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.652+0000 7f624effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:50:02.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.652+0000 7f624effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:50:02.656 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":194,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:50:02.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.655+0000 7f6260433640 1 -- 172.21.15.12:0/2013615426 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f623c03e260 msgr2=0x7f623c040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:02.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.655+0000 7f6260433640 1 --2- 172.21.15.12:0/2013615426 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f623c03e260 0x7f623c040720 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f62580a17e0 tx=0x7f624802e000 comp rx=0 tx=0).stop 2024-09-11T01:50:02.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.655+0000 7f6260433640 10 monclient: shutdown 2024-09-11T01:50:02.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.655+0000 7f6260433640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:50:02.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.655+0000 7f6260433640 1 -- 172.21.15.12:0/2013615426 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f625809d030 msgr2=0x7f6258136da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:02.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.655+0000 7f6260433640 1 --2- 172.21.15.12:0/2013615426 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f625809d030 0x7f6258136da0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f625400ec10 tx=0x7f625400c520 comp rx=0 tx=0).stop 2024-09-11T01:50:02.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.655+0000 7f6260433640 1 -- 172.21.15.12:0/2013615426 shutdown_connections 2024-09-11T01:50:02.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.655+0000 7f6260433640 1 --2- 172.21.15.12:0/2013615426 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f623c03e260 0x7f623c040720 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:02.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.655+0000 7f6260433640 1 --2- 172.21.15.12:0/2013615426 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6258138230 0x7f6258120c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:02.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.655+0000 7f6260433640 1 --2- 172.21.15.12:0/2013615426 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f625809d460 0x7f62581372e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:02.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.655+0000 7f6260433640 1 --2- 172.21.15.12:0/2013615426 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f625809d030 0x7f6258136da0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:02.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.655+0000 7f6260433640 1 -- 172.21.15.12:0/2013615426 >> 172.21.15.12:0/2013615426 conn(0x7f62580a7120 msgr2=0x7f62580a8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:50:02.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.655+0000 7f6260433640 1 -- 172.21.15.12:0/2013615426 shutdown_connections 2024-09-11T01:50:02.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.655+0000 7f6260433640 1 -- 172.21.15.12:0/2013615426 wait complete. 2024-09-11T01:50:02.668 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:50:02.668 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:50:02.668 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:50:02.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.767+0000 7fbc83910640 10 monclient: get_monmap_and_config 2024-09-11T01:50:02.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.767+0000 7fbc83910640 10 monclient: build_initial_monmap 2024-09-11T01:50:02.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.767+0000 7fbc83910640 10 monclient: monmap: 2024-09-11T01:50:02.770 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:50:02.770 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:02.770 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:50:02.769300+0000 2024-09-11T01:50:02.770 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:50:02.769300+0000 2024-09-11T01:50:02.771 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:50:02.771 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:02.771 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:50:02.771 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:50:02.771 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:50:02.771 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:50:02.771 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:50:02.771 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:02.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.768+0000 7fbc83910640 1 Processor -- start 2024-09-11T01:50:02.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.768+0000 7fbc83910640 1 -- start start 2024-09-11T01:50:02.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.768+0000 7fbc83910640 10 monclient: init 2024-09-11T01:50:02.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.768+0000 7fbc83910640 10 monclient: _reopen_session rank -1 2024-09-11T01:50:02.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.768+0000 7fbc83910640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-11T01:50:02.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.768+0000 7fbc83910640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbc7c0a2d40 0x7fbc7c06cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:02.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.768+0000 7fbc83910640 10 monclient(hunting): picked mon.noname-b con 0x7fbc7c0a2d40 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:50:02.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.768+0000 7fbc83910640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc7c06d090 addr v1:172.21.15.12:6789/0 2024-09-11T01:50:02.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.768+0000 7fbc83910640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbc7c06d470 addr v1:172.21.15.71:6789/0 2024-09-11T01:50:02.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.768+0000 7fbc83910640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc7c09aed0 con 0x7fbc7c06d090 2024-09-11T01:50:02.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.768+0000 7fbc83910640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc7c0a3920 con 0x7fbc7c06d470 2024-09-11T01:50:02.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.768+0000 7fbc83910640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:02.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.768+0000 7fbc83910640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fbc7c06dd10 con 0x7fbc7c0a2d40 2024-09-11T01:50:02.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.768+0000 7fbc83910640 10 monclient(hunting): _renew_subs 2024-09-11T01:50:02.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.769+0000 7fbc83910640 10 monclient(hunting): authenticate will time out at 2009.212011s 2024-09-11T01:50:02.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.769+0000 7fbc816ac640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbc7c0a2d40 0x7fbc7c06cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:02.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.769+0000 7fbc816ac640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbc7c0a2d40 0x7fbc7c06cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45268/0 (socket says 172.21.15.12:45268) 2024-09-11T01:50:02.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.769+0000 7fbc80eab640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbc7c06d090 0x7fbc7c13c4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43854/0 (socket says 172.21.15.12:43854) 2024-09-11T01:50:02.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.769+0000 7fbc816ac640 1 -- 172.21.15.12:0/812139537 learned_addr learned my addr 172.21.15.12:0/812139537 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:50:02.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.769+0000 7fbc816ac640 10 monclient(hunting): get_auth_request con 0x7fbc7c0a2d40 auth_method 0 2024-09-11T01:50:02.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.769+0000 7fbc816ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:02.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.769+0000 7fbc816ac640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:02.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.769+0000 7fbc816ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:02.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.769+0000 7fbc816ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:02.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.769+0000 7fbc816ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:02.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.769+0000 7fbc816ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:02.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.769+0000 7fbc816ac640 10 monclient(hunting): handle_auth_done global_id 224183 payload 274 2024-09-11T01:50:02.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.769+0000 7fbc816ac640 1 -- 172.21.15.12:0/812139537 >> v1:172.21.15.71:6789/0 conn(0x7fbc7c06d470 legacy=0x7fbc7c13fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:50:02.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.769+0000 7fbc816ac640 1 -- 172.21.15.12:0/812139537 >> v1:172.21.15.12:6789/0 conn(0x7fbc7c06d090 legacy=0x7fbc7c13c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:50:02.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.769+0000 7fbc816ac640 10 monclient: _finish_hunting 0 2024-09-11T01:50:02.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.769+0000 7fbc816ac640 1 monclient: found mon.noname-b 2024-09-11T01:50:02.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.769+0000 7fbc816ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:50:02.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.769+0000 7fbc816ac640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:50:02.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.769+0000 7fbc816ac640 1 -- 172.21.15.12:0/812139537 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc7c0a3320 con 0x7fbc7c0a2d40 2024-09-11T01:50:02.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc816ac640 1 -- 172.21.15.12:0/812139537 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbc7c0a2d40 msgr2=0x7fbc7c06cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:02.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc816ac640 1 --2- 172.21.15.12:0/812139537 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbc7c0a2d40 0x7fbc7c06cb50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c075000 tx=0x7fbc7802a050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b8982e5d3bcecfa server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:02.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc73fff640 1 -- 172.21.15.12:0/812139537 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc78037070 con 0x7fbc7c0a2d40 2024-09-11T01:50:02.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:02.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc73fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:50:02.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc73fff640 10 monclient: dump: 2024-09-11T01:50:02.783 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:02.783 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:02.783 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:02.783 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:02.784 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:02.784 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:02.784 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:02.784 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:02.784 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:02.784 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:02.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc73fff640 10 monclient: _finish_auth 0 2024-09-11T01:50:02.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc73fff640 10 monclient: _check_auth_tickets 2024-09-11T01:50:02.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:50:02.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc73fff640 1 -- 172.21.15.12:0/812139537 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc78032640 con 0x7fbc7c0a2d40 2024-09-11T01:50:02.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc83910640 5 monclient: authenticate success, global_id 224183 2024-09-11T01:50:02.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc83910640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:50:02.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc73fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:50:02.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc83910640 10 monclient: get_monmap_and_config success 2024-09-11T01:50:02.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc83910640 4 set_mon_vals no callback set 2024-09-11T01:50:02.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc73fff640 1 -- 172.21.15.12:0/812139537 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc78032a10 con 0x7fbc7c0a2d40 2024-09-11T01:50:02.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:02.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc73fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:50:02.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc73fff640 10 monclient: dump: 2024-09-11T01:50:02.785 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:02.785 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:02.785 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:02.786 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:02.786 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:02.786 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:02.786 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:02.786 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:02.786 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:02.786 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:02.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc83910640 10 monclient: shutdown 2024-09-11T01:50:02.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc83910640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:50:02.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc83910640 1 -- 172.21.15.12:0/812139537 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbc7c0a2d40 msgr2=0x7fbc7c06cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:02.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc83910640 1 --2- 172.21.15.12:0/812139537 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbc7c0a2d40 0x7fbc7c06cb50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c075000 tx=0x7fbc7802a050 comp rx=0 tx=0).stop 2024-09-11T01:50:02.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc83910640 1 -- 172.21.15.12:0/812139537 shutdown_connections 2024-09-11T01:50:02.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc83910640 1 --2- 172.21.15.12:0/812139537 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbc7c0a2d40 0x7fbc7c06cb50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:02.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc83910640 1 -- 172.21.15.12:0/812139537 >> 172.21.15.12:0/812139537 conn(0x7fbc7c0a3e30 msgr2=0x7fbc7c0a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:50:02.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc83910640 1 -- 172.21.15.12:0/812139537 shutdown_connections 2024-09-11T01:50:02.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.772+0000 7fbc83910640 1 -- 172.21.15.12:0/812139537 wait complete. 2024-09-11T01:50:02.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.773+0000 7fbc83910640 10 monclient: build_initial_monmap 2024-09-11T01:50:02.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.773+0000 7fbc83910640 10 monclient: monmap: 2024-09-11T01:50:02.788 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:50:02.789 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:50:02.789 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:50:02.789 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:50:02.789 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:50:02.789 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:02.790 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:50:02.790 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:50:02.790 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:50:02.790 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:02.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.773+0000 7fbc83910640 1 Processor -- start 2024-09-11T01:50:02.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.773+0000 7fbc83910640 1 -- start start 2024-09-11T01:50:02.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.773+0000 7fbc83910640 10 monclient: init 2024-09-11T01:50:02.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc83910640 10 monclient: _reopen_session rank -1 2024-09-11T01:50:02.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc83910640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:50:02.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc83910640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbc7c06d090 0x7fbc7c0994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:02.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc83910640 10 monclient(hunting): picked mon.noname-c con 0x7fbc7c06d090 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:50:02.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc83910640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc7c06d470 0x7fbc7c093ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:02.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc83910640 10 monclient(hunting): picked mon.noname-b con 0x7fbc7c06d470 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:50:02.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc83910640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc7c0a2d40 0x7fbc7c094410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:02.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc83910640 10 monclient(hunting): picked mon.noname-a con 0x7fbc7c0a2d40 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:50:02.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc83910640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:02.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc83910640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c09ba10 con 0x7fbc7c0a2d40 2024-09-11T01:50:02.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc83910640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:02.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc83910640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c0a3120 con 0x7fbc7c06d470 2024-09-11T01:50:02.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc83910640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:02.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc83910640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fbc7c09b3a0 con 0x7fbc7c06d090 2024-09-11T01:50:02.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc83910640 10 monclient(hunting): _renew_subs 2024-09-11T01:50:02.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc83910640 10 monclient(hunting): authenticate will time out at 2009.217352s 2024-09-11T01:50:02.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc816ac640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbc7c06d090 0x7fbc7c0994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:02.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc816ac640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbc7c06d090 0x7fbc7c0994e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45276/0 (socket says 172.21.15.12:45276) 2024-09-11T01:50:02.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc816ac640 1 -- 172.21.15.12:0/4222289773 learned_addr learned my addr 172.21.15.12:0/4222289773 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:50:02.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc816ac640 10 monclient(hunting): get_auth_request con 0x7fbc7c06d090 auth_method 0 2024-09-11T01:50:02.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc816ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:02.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc816ac640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:02.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc816ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:02.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc81ead640 1 --2- 172.21.15.12:0/4222289773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc7c0a2d40 0x7fbc7c094410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:02.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc81ead640 10 monclient(hunting): get_auth_request con 0x7fbc7c0a2d40 auth_method 0 2024-09-11T01:50:02.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc81ead640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:02.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc81ead640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:02.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc81ead640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:02.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc816ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:02.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc816ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:02.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc816ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:02.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc80eab640 1 --2- 172.21.15.12:0/4222289773 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc7c06d470 0x7fbc7c093ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:02.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc80eab640 10 monclient(hunting): get_auth_request con 0x7fbc7c06d470 auth_method 0 2024-09-11T01:50:02.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc80eab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:02.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc80eab640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:02.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc80eab640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:02.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc81ead640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:02.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc81ead640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:02.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc81ead640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:02.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc80eab640 1 -- 172.21.15.12:0/4222289773 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc7c06d470 msgr2=0x7fbc7c093ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:50:02.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc80eab640 1 -- 172.21.15.12:0/4222289773 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc7c06d470 msgr2=0x7fbc7c093ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:50:02.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc80eab640 1 --2- 172.21.15.12:0/4222289773 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc7c06d470 0x7fbc7c093ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:50:02.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc80eab640 1 --2- 172.21.15.12:0/4222289773 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc7c06d470 0x7fbc7c093ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:50:02.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc816ac640 10 monclient(hunting): handle_auth_done global_id 224186 payload 824 2024-09-11T01:50:02.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc816ac640 1 -- 172.21.15.12:0/4222289773 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc7c06d470 msgr2=0x7fbc7c093ed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:50:02.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc816ac640 1 --2- 172.21.15.12:0/4222289773 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc7c06d470 0x7fbc7c093ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:02.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc816ac640 1 -- 172.21.15.12:0/4222289773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc7c0a2d40 msgr2=0x7fbc7c094410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:02.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc816ac640 1 --2- 172.21.15.12:0/4222289773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc7c0a2d40 0x7fbc7c094410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:02.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc816ac640 10 monclient: _finish_hunting 0 2024-09-11T01:50:02.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc816ac640 1 monclient: found mon.noname-c 2024-09-11T01:50:02.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc816ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:50:02.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc816ac640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:50:02.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.774+0000 7fbc816ac640 1 -- 172.21.15.12:0/4222289773 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc7c095990 con 0x7fbc7c06d090 2024-09-11T01:50:02.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.777+0000 7fbc816ac640 1 -- 172.21.15.12:0/4222289773 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbc7c06d090 msgr2=0x7fbc7c0994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:02.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.777+0000 7fbc816ac640 1 --2- 172.21.15.12:0/4222289773 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbc7c06d090 0x7fbc7c0994e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbc78009990 tx=0x7fbc780046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:02.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.777+0000 7fbc71ffb640 1 -- 172.21.15.12:0/4222289773 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc78037050 con 0x7fbc7c06d090 2024-09-11T01:50:02.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.777+0000 7fbc71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:02.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.777+0000 7fbc71ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:50:02.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.777+0000 7fbc71ffb640 10 monclient: dump: 2024-09-11T01:50:02.802 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:02.802 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:02.802 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:02.803 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:02.803 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:02.803 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:02.803 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:02.803 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:02.803 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:02.804 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:02.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.777+0000 7fbc71ffb640 10 monclient: _finish_auth 0 2024-09-11T01:50:02.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.777+0000 7fbc71ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:50:02.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.777+0000 7fbc71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:50:02.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.777+0000 7fbc71ffb640 1 -- 172.21.15.12:0/4222289773 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc78038040 con 0x7fbc7c06d090 2024-09-11T01:50:02.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.777+0000 7fbc71ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:50:02.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.777+0000 7fbc71ffb640 1 -- 172.21.15.12:0/4222289773 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc7803ca60 con 0x7fbc7c06d090 2024-09-11T01:50:02.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.777+0000 7fbc71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:02.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.777+0000 7fbc71ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:50:02.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.777+0000 7fbc71ffb640 10 monclient: dump: 2024-09-11T01:50:02.805 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:02.805 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:02.806 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:02.806 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:02.806 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:02.806 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:02.806 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:02.806 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:02.806 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:02.807 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:02.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.777+0000 7fbc83910640 5 monclient: authenticate success, global_id 224186 2024-09-11T01:50:02.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.777+0000 7fbc83910640 10 monclient: _renew_subs 2024-09-11T01:50:02.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.777+0000 7fbc83910640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:50:02.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.777+0000 7fbc737fe640 4 set_mon_vals no callback set 2024-09-11T01:50:02.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.777+0000 7fbc83910640 1 -- 172.21.15.12:0/4222289773 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc7c097450 con 0x7fbc7c06d090 2024-09-11T01:50:02.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.777+0000 7fbc83910640 10 monclient: _renew_subs 2024-09-11T01:50:02.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.777+0000 7fbc83910640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:50:02.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.777+0000 7fbc83910640 1 -- 172.21.15.12:0/4222289773 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc7c0964f0 con 0x7fbc7c06d090 2024-09-11T01:50:02.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.778+0000 7fbc83910640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:02.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.778+0000 7fbc83910640 1 --2- 172.21.15.12:0/4222289773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc7c099a20 0x7fbc7c094950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:02.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.778+0000 7fbc83910640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:02.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.778+0000 7fbc83910640 1 -- 172.21.15.12:0/4222289773 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c06cbd0 con 0x7fbc7c099a20 2024-09-11T01:50:02.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.778+0000 7fbc80eab640 1 --2- 172.21.15.12:0/4222289773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc7c099a20 0x7fbc7c094950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:02.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.778+0000 7fbc80eab640 10 monclient: get_auth_request con 0x7fbc7c099a20 auth_method 0 2024-09-11T01:50:02.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.778+0000 7fbc80eab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:02.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.778+0000 7fbc80eab640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:02.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.778+0000 7fbc80eab640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:02.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.778+0000 7fbc71ffb640 1 -- 172.21.15.12:0/4222289773 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fbc7803cc00 con 0x7fbc7c06d090 2024-09-11T01:50:02.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.778+0000 7fbc80eab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:02.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.778+0000 7fbc80eab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:02.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.778+0000 7fbc80eab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:02.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.778+0000 7fbc80eab640 10 monclient(hunting): handle_auth_done global_id 214299 payload 824 2024-09-11T01:50:02.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.778+0000 7fbc80eab640 1 -- 172.21.15.12:0/4222289773 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbc7c09e520 con 0x7fbc7c099a20 2024-09-11T01:50:02.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.778+0000 7fbc71ffb640 1 --2- 172.21.15.12:0/4222289773 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbc6003e210 0x7fbc600406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:02.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.778+0000 7fbc71ffb640 1 -- 172.21.15.12:0/4222289773 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(704..704 src has 507..704) ==== 4831+0+0 (secure 0 0 0) 0x7fbc78073120 con 0x7fbc7c06d090 2024-09-11T01:50:02.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.779+0000 7fbc81ead640 1 --2- 172.21.15.12:0/4222289773 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbc6003e210 0x7fbc600406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:02.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.779+0000 7fbc81ead640 10 monclient: get_auth_request con 0x7fbc6003e210 auth_method 0 2024-09-11T01:50:02.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.781+0000 7fbc80eab640 1 -- 172.21.15.12:0/4222289773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc7c099a20 msgr2=0x7fbc7c094950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:02.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.781+0000 7fbc80eab640 1 --2- 172.21.15.12:0/4222289773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc7c099a20 0x7fbc7c094950 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c097c20 tx=0x7fbc6c00e890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:02.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.781+0000 7fbc71ffb640 1 -- 172.21.15.12:0/4222289773 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c019070 con 0x7fbc7c099a20 2024-09-11T01:50:02.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.781+0000 7fbc71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:02.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.781+0000 7fbc71ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:50:02.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.781+0000 7fbc71ffb640 10 monclient: dump: 2024-09-11T01:50:02.813 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:02.813 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:02.813 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:02.813 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:02.813 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:02.813 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:02.814 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:02.814 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:02.814 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:02.814 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:02.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.781+0000 7fbc81ead640 1 --2- 172.21.15.12:0/4222289773 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbc6003e210 0x7fbc600406d0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c097940 tx=0x7fbc74003d10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:02.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.781+0000 7fbc71ffb640 1 -- 172.21.15.12:0/4222289773 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbc6c004e20 con 0x7fbc7c099a20 2024-09-11T01:50:02.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.781+0000 7fbc71ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:02.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.781+0000 7fbc71ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:50:02.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.781+0000 7fbc71ffb640 1 -- 172.21.15.12:0/4222289773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc7c099a20 msgr2=0x7fbc7c094950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:02.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.781+0000 7fbc71ffb640 1 --2- 172.21.15.12:0/4222289773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc7c099a20 0x7fbc7c094950 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c097c20 tx=0x7fbc6c00e890 comp rx=0 tx=0).stop 2024-09-11T01:50:02.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.781+0000 7fbc71ffb640 1 -- 172.21.15.12:0/4222289773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc7c099a20 msgr2=0x7fbc7c094950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:50:02.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.781+0000 7fbc71ffb640 1 --2- 172.21.15.12:0/4222289773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc7c099a20 0x7fbc7c094950 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:02.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.787+0000 7fbc83910640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:50:02.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.787+0000 7fbc83910640 1 --2- 172.21.15.12:0/4222289773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc7c0913c0 0x7fbc7c0917d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:02.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.787+0000 7fbc83910640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:02.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.787+0000 7fbc83910640 1 -- 172.21.15.12:0/4222289773 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c06e520 con 0x7fbc7c0913c0 2024-09-11T01:50:02.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.787+0000 7fbc80eab640 1 --2- 172.21.15.12:0/4222289773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc7c0913c0 0x7fbc7c0917d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:02.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.787+0000 7fbc80eab640 10 monclient: get_auth_request con 0x7fbc7c0913c0 auth_method 0 2024-09-11T01:50:02.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.787+0000 7fbc80eab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:02.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.787+0000 7fbc80eab640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:02.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.787+0000 7fbc80eab640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:02.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.787+0000 7fbc80eab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:02.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.787+0000 7fbc80eab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:02.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.787+0000 7fbc80eab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:02.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.788+0000 7fbc80eab640 10 monclient(hunting): handle_auth_done global_id 214302 payload 824 2024-09-11T01:50:02.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.788+0000 7fbc80eab640 1 -- 172.21.15.12:0/4222289773 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbc7c1409b0 con 0x7fbc7c0913c0 2024-09-11T01:50:02.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.790+0000 7fbc80eab640 1 -- 172.21.15.12:0/4222289773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc7c0913c0 msgr2=0x7fbc7c0917d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:02.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.790+0000 7fbc80eab640 1 --2- 172.21.15.12:0/4222289773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc7c0913c0 0x7fbc7c0917d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c097c20 tx=0x7fbc6c004930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:02.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.790+0000 7fbc71ffb640 1 -- 172.21.15.12:0/4222289773 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c019040 con 0x7fbc7c0913c0 2024-09-11T01:50:02.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.790+0000 7fbc71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:02.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.790+0000 7fbc71ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:50:02.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.790+0000 7fbc71ffb640 10 monclient: dump: 2024-09-11T01:50:02.816 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:02.817 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:02.817 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:02.817 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:02.817 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:02.817 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:02.817 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:02.817 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:02.817 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:02.817 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:02.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.790+0000 7fbc71ffb640 1 -- 172.21.15.12:0/4222289773 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7fbc6c01b620 con 0x7fbc7c0913c0 2024-09-11T01:50:02.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.790+0000 7fbc71ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:50:02.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.790+0000 7fbc71ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:50:02.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.790+0000 7fbc71ffb640 1 -- 172.21.15.12:0/4222289773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc7c0913c0 msgr2=0x7fbc7c0917d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:02.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.790+0000 7fbc71ffb640 1 --2- 172.21.15.12:0/4222289773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc7c0913c0 0x7fbc7c0917d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c097c20 tx=0x7fbc6c004930 comp rx=0 tx=0).stop 2024-09-11T01:50:02.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.790+0000 7fbc71ffb640 1 -- 172.21.15.12:0/4222289773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc7c0913c0 msgr2=0x7fbc7c0917d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:50:02.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.790+0000 7fbc71ffb640 1 --2- 172.21.15.12:0/4222289773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc7c0913c0 0x7fbc7c0917d0 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c097c20 tx=0x7fbc6c004930 comp rx=0 tx=0).stop 2024-09-11T01:50:02.818 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:50:02.818 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:50:02.818 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:50:02.818 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:50:02.818 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 194, 2024-09-11T01:50:02.818 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 108991, 2024-09-11T01:50:02.819 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:50:02.819 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:50:02.819 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:50:02.819 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:50:02.819 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:02.819 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:50:02.819 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:50:02.819 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:50:02.819 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:50:02.819 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:50:02.819 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:50:02.819 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:50:02.819 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:50:02.819 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:50:02.820 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:50:02.820 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:50:02.820 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:50:02.820 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:50:02.820 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:50:02.820 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:50:02.820 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:02.820 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:50:02.820 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:50:02.820 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:50:02.820 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:50:02.820 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:50:02.820 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:50:02.821 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:50:02.821 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:50:02.821 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:50:02.821 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:50:02.821 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:50:02.821 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:50:02.821 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:50:02.821 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:02.821 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:02.821 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:50:02.821 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:50:02.821 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:50:02.821 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:50:02.821 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:50:02.822 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:50:02.822 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:50:02.822 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:50:02.822 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:50:02.822 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:50:02.822 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:50:02.822 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:50:02.822 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:50:02.822 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:50:02.822 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:50:02.822 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:50:02.822 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:50:02.822 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:50:02.822 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:50:02.823 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:50:02.823 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:50:02.823 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:50:02.823 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:50:02.823 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:50:02.823 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:50:02.823 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:50:02.823 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:50:02.823 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:50:02.823 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:02.823 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:50:02.823 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:02.823 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:50:02.824 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:02.824 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:50:02.824 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:50:02.824 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:50:02.824 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:50:02.824 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:02.824 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:50:02.824 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:50:02.824 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:02.824 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:02.824 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:02.824 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:50:02.825 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:50:02.825 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:02.825 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:02.825 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:02.825 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:02.825 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:50:02.825 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:50:02.825 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:50:02.825 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:50:02.825 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:50:02.825 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:02.825 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:02.825 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:50:02.825 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:50:02.826 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:50:02.826 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:50:02.826 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:02.826 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:50:02.826 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:50:02.826 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:02.826 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:02.826 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:02.826 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:50:02.826 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:50:02.826 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:02.826 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:02.826 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:02.826 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:02.827 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:50:02.827 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:50:02.827 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:50:02.827 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:50:02.827 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:50:02.827 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:02.827 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:02.827 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:50:02.827 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:50:02.827 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:50:02.827 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:50:02.827 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:02.827 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:50:02.827 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:50:02.828 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:02.828 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:02.828 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:02.828 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:50:02.828 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:50:02.828 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:02.828 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:02.828 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:02.828 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:02.828 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:50:02.828 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:50:02.828 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:50:02.828 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:50:02.829 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:50:02.829 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:02.829 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:02.829 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:02.829 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:50:02.829 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:50:02.829 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:02.829 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:50:02.829 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:50:02.829 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:50:02.829 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:02.829 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:02.829 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:50:02.830 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:02.830 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:50:02.830 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:50:02.830 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:50:02.830 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:02.830 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:02.830 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:50:02.830 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:02.830 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:50:02.830 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:50:02.830 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:50:02.830 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:02.830 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:02.830 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:02.831 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:50:02.831 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:50:02.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.791+0000 7fbc83910640 1 -- 172.21.15.12:0/4222289773 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbc6003e210 msgr2=0x7fbc600406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:02.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.791+0000 7fbc83910640 1 --2- 172.21.15.12:0/4222289773 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbc6003e210 0x7fbc600406d0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c097940 tx=0x7fbc74003d10 comp rx=0 tx=0).stop 2024-09-11T01:50:02.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.791+0000 7fbc83910640 10 monclient: shutdown 2024-09-11T01:50:02.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.791+0000 7fbc83910640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:50:02.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.791+0000 7fbc83910640 1 -- 172.21.15.12:0/4222289773 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbc7c06d090 msgr2=0x7fbc7c0994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:02.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.791+0000 7fbc83910640 1 --2- 172.21.15.12:0/4222289773 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbc7c06d090 0x7fbc7c0994e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbc78009990 tx=0x7fbc780046c0 comp rx=0 tx=0).stop 2024-09-11T01:50:02.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.791+0000 7fbc81ead640 1 -- 172.21.15.12:0/4222289773 reap_dead start 2024-09-11T01:50:02.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.791+0000 7fbc83910640 1 -- 172.21.15.12:0/4222289773 shutdown_connections 2024-09-11T01:50:02.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.791+0000 7fbc83910640 1 --2- 172.21.15.12:0/4222289773 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbc7c06d090 0x7fbc7c0994e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:02.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.791+0000 7fbc83910640 1 -- 172.21.15.12:0/4222289773 >> 172.21.15.12:0/4222289773 conn(0x7fbc7c0a3e30 msgr2=0x7fbc7c09f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:50:02.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.791+0000 7fbc83910640 1 -- 172.21.15.12:0/4222289773 shutdown_connections 2024-09-11T01:50:02.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.791+0000 7fbc83910640 1 -- 172.21.15.12:0/4222289773 wait complete. 2024-09-11T01:50:02.833 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:50:02.833 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:50:02.833 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:50:02.880 INFO:tasks.rados.rados.0.smithi071.stdout:3152: done (3 left) 2024-09-11T01:50:02.880 INFO:tasks.rados.rados.0.smithi071.stdout:3153: write (excl) oid 5 current snap is 352 2024-09-11T01:50:02.880 INFO:tasks.rados.rados.0.smithi071.stdout:3148: finishing write tid 2 to smithi07134777-1 2024-09-11T01:50:02.880 INFO:tasks.rados.rados.0.smithi071.stdout:3153: seq_num 1108 ranges {622156=647077,1737578=680388} 2024-09-11T01:50:02.886 INFO:tasks.rados.rados.0.smithi071.stdout:3153: writing smithi07134777-5 from 622156 to 1269233 tid 1 2024-09-11T01:50:02.894 INFO:tasks.rados.rados.0.smithi071.stdout:3153: writing smithi07134777-5 from 1737578 to 2417966 tid 2 2024-09-11T01:50:02.894 INFO:tasks.rados.rados.0.smithi071.stdout:3154: read oid 41 snap 236 2024-09-11T01:50:02.894 INFO:tasks.rados.rados.0.smithi071.stdout:3154: expect (ObjNum 753 snap 229 seq_num 753) 2024-09-11T01:50:02.894 INFO:tasks.rados.rados.0.smithi071.stdout:3155: write oid 10 current snap is 352 2024-09-11T01:50:02.894 INFO:tasks.rados.rados.0.smithi071.stdout:3148: finishing write tid 3 to smithi07134777-1 2024-09-11T01:50:02.894 INFO:tasks.rados.rados.0.smithi071.stdout:3148: finishing write tid 4 to smithi07134777-1 2024-09-11T01:50:02.894 INFO:tasks.rados.rados.0.smithi071.stdout:3148: oid 1 updating version 0 to 1809 2024-09-11T01:50:02.895 INFO:tasks.rados.rados.0.smithi071.stdout:3148: oid 1 updating version 1809 to 1810 2024-09-11T01:50:02.895 INFO:tasks.rados.rados.0.smithi071.stdout:3148: oid 1 version 1810 is already newer than 1808 2024-09-11T01:50:02.895 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 1810 (ObjNum 1106 snap 352 seq_num 1106) dirty exists 2024-09-11T01:50:02.895 INFO:tasks.rados.rados.0.smithi071.stdout:3148: left oid 1 (ObjNum 1106 snap 352 seq_num 1106) 2024-09-11T01:50:02.895 INFO:tasks.rados.rados.0.smithi071.stdout:3151: finishing copy_from to smithi07134777-6 2024-09-11T01:50:02.895 INFO:tasks.rados.rados.0.smithi071.stdout:3151: got expected ENOENT (src dne) 2024-09-11T01:50:02.895 INFO:tasks.rados.rados.0.smithi071.stdout:3151: finishing copy_from racing read to smithi07134777-6 2024-09-11T01:50:02.895 INFO:tasks.rados.rados.0.smithi071.stdout:3150: finishing copy_from racing read to smithi07134777-39 2024-09-11T01:50:02.895 INFO:tasks.rados.rados.0.smithi071.stdout:3150: finishing copy_from to smithi07134777-39 2024-09-11T01:50:02.895 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 854 (ObjNum 875 snap 267 seq_num 875) dirty exists 2024-09-11T01:50:02.895 INFO:tasks.rados.rados.0.smithi071.stdout:3155: seq_num 1109 ranges {638963=722326,2079707=424046} 2024-09-11T01:50:02.902 INFO:tasks.rados.rados.0.smithi071.stdout:3155: writing smithi07134777-10 from 638963 to 1361289 tid 1 2024-09-11T01:50:02.910 INFO:tasks.rados.rados.0.smithi071.stdout:3155: writing smithi07134777-10 from 2079707 to 2503753 tid 2 2024-09-11T01:50:02.910 INFO:tasks.rados.rados.0.smithi071.stdout:3148: done (5 left) 2024-09-11T01:50:02.910 INFO:tasks.rados.rados.0.smithi071.stdout:3150: done (4 left) 2024-09-11T01:50:02.910 INFO:tasks.rados.rados.0.smithi071.stdout:3151: done (3 left) 2024-09-11T01:50:02.910 INFO:tasks.rados.rados.0.smithi071.stdout:3156: rollback oid 36 current snap is 352 2024-09-11T01:50:02.910 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 36 to 340 2024-09-11T01:50:02.910 INFO:tasks.rados.rados.0.smithi071.stdout:3153: finishing write tid 1 to smithi07134777-5 2024-09-11T01:50:02.910 INFO:tasks.rados.rados.0.smithi071.stdout:3153: finishing write tid 2 to smithi07134777-5 2024-09-11T01:50:02.910 INFO:tasks.rados.rados.0.smithi071.stdout:3153: finishing write tid 3 to smithi07134777-5 2024-09-11T01:50:02.910 INFO:tasks.rados.rados.0.smithi071.stdout:3153: finishing write tid 4 to smithi07134777-5 2024-09-11T01:50:02.910 INFO:tasks.rados.rados.0.smithi071.stdout:3153: oid 5 updating version 0 to 1180 2024-09-11T01:50:02.910 INFO:tasks.rados.rados.0.smithi071.stdout:3153: oid 5 version 1180 is already newer than 1178 2024-09-11T01:50:02.910 INFO:tasks.rados.rados.0.smithi071.stdout:3153: oid 5 version 1180 is already newer than 1179 2024-09-11T01:50:02.911 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 1180 (ObjNum 1108 snap 352 seq_num 1108) dirty exists 2024-09-11T01:50:02.911 INFO:tasks.rados.rados.0.smithi071.stdout:3153: left oid 5 (ObjNum 1108 snap 352 seq_num 1108) 2024-09-11T01:50:02.911 INFO:tasks.rados.rados.0.smithi071.stdout:3154: expect (ObjNum 753 snap 229 seq_num 753) 2024-09-11T01:50:02.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.943+0000 7f03c3af6640 10 monclient: get_monmap_and_config 2024-09-11T01:50:02.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.943+0000 7f03c3af6640 10 monclient: build_initial_monmap 2024-09-11T01:50:02.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.943+0000 7f03c3af6640 10 monclient: monmap: 2024-09-11T01:50:02.946 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:50:02.946 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:02.946 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:50:02.945020+0000 2024-09-11T01:50:02.946 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:50:02.945020+0000 2024-09-11T01:50:02.946 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:50:02.946 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:02.946 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:50:02.946 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:50:02.946 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:50:02.947 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:50:02.947 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:50:02.947 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:02.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.943+0000 7f03c3af6640 1 Processor -- start 2024-09-11T01:50:02.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c3af6640 1 -- start start 2024-09-11T01:50:02.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c3af6640 10 monclient: init 2024-09-11T01:50:02.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c3af6640 10 monclient: _reopen_session rank -1 2024-09-11T01:50:02.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c3af6640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-11T01:50:02.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c3af6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc137e50 0x7f03bc09d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:02.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c3af6640 10 monclient(hunting): picked mon.noname-b con 0x7f03bc137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:50:02.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c3af6640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f03bc138230 0x7f03bc138630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:02.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c3af6640 10 monclient(hunting): picked mon.noname-a con 0x7f03bc138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:50:02.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c3af6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f03bc134cb0 addr v1:172.21.15.12:6789/0 2024-09-11T01:50:02.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c3af6640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03bc0a62d0 con 0x7f03bc134cb0 2024-09-11T01:50:02.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c3af6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:02.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c3af6640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f03bc0a65b0 con 0x7f03bc138230 2024-09-11T01:50:02.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c3af6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:02.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c3af6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f03bc09bb90 con 0x7f03bc137e50 2024-09-11T01:50:02.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c3af6640 10 monclient(hunting): _renew_subs 2024-09-11T01:50:02.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c3af6640 10 monclient(hunting): authenticate will time out at 2009.387590s 2024-09-11T01:50:02.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c1892640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc137e50 0x7f03bc09d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:02.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c1091640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f03bc138230 0x7f03bc138630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:02.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c2093640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f03bc134cb0 0x7f03bc135120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43864/0 (socket says 172.21.15.12:43864) 2024-09-11T01:50:02.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c2093640 1 -- 172.21.15.12:0/1679726906 learned_addr learned my addr 172.21.15.12:0/1679726906 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:50:02.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c1892640 1 --2- 172.21.15.12:0/1679726906 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc137e50 0x7f03bc09d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45278/0 (socket says 172.21.15.12:45278) 2024-09-11T01:50:02.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c1091640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f03bc138230 0x7f03bc138630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40042/0 (socket says 172.21.15.12:40042) 2024-09-11T01:50:02.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c1892640 10 monclient(hunting): get_auth_request con 0x7f03bc137e50 auth_method 0 2024-09-11T01:50:02.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c1892640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c1892640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c1892640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c1091640 10 monclient(hunting): get_auth_request con 0x7f03bc138230 auth_method 0 2024-09-11T01:50:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c1091640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c1091640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c1091640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c1892640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c1892640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c1892640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c1091640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c1091640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c1091640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c1892640 10 monclient(hunting): handle_auth_done global_id 224189 payload 274 2024-09-11T01:50:02.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c1892640 1 -- 172.21.15.12:0/1679726906 >> v2:172.21.15.12:3300/0 conn(0x7f03bc138230 msgr2=0x7f03bc138630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:50:02.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c1892640 1 --2- 172.21.15.12:0/1679726906 >> v2:172.21.15.12:3300/0 conn(0x7f03bc138230 0x7f03bc138630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:02.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.944+0000 7f03c1892640 1 -- 172.21.15.12:0/1679726906 >> v1:172.21.15.12:6789/0 conn(0x7f03bc134cb0 legacy=0x7f03bc135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:50:02.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.945+0000 7f03c1892640 10 monclient: _finish_hunting 0 2024-09-11T01:50:02.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.945+0000 7f03c1892640 1 monclient: found mon.noname-b 2024-09-11T01:50:02.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.945+0000 7f03c1892640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:50:02.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.945+0000 7f03c1892640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:50:02.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.945+0000 7f03c1892640 1 -- 172.21.15.12:0/1679726906 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03bc09c4f0 con 0x7f03bc137e50 2024-09-11T01:50:02.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.945+0000 7f03c1091640 1 --2- 172.21.15.12:0/1679726906 >> v2:172.21.15.12:3300/0 conn(0x7f03bc138230 0x7f03bc138630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:50:02.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.947+0000 7f03c1892640 1 -- 172.21.15.12:0/1679726906 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc137e50 msgr2=0x7f03bc09d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:02.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.947+0000 7f03c1892640 1 --2- 172.21.15.12:0/1679726906 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc137e50 0x7f03bc09d160 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f03b80097a0 tx=0x7f03b8029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3291419f72924dc6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:02.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.947+0000 7f03c0890640 1 -- 172.21.15.12:0/1679726906 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b8037070 con 0x7f03bc137e50 2024-09-11T01:50:02.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.947+0000 7f03c0890640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:02.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.948+0000 7f03c0890640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:50:02.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.948+0000 7f03c0890640 10 monclient: dump: 2024-09-11T01:50:02.959 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:02.959 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:02.959 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:02.959 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:02.959 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:02.959 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:02.959 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:02.959 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:02.960 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:02.960 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:02.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.948+0000 7f03c0890640 10 monclient: _finish_auth 0 2024-09-11T01:50:02.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.948+0000 7f03c0890640 10 monclient: _check_auth_tickets 2024-09-11T01:50:02.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.948+0000 7f03c0890640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:50:02.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.948+0000 7f03c3af6640 5 monclient: authenticate success, global_id 224189 2024-09-11T01:50:02.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.948+0000 7f03c3af6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:50:02.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.948+0000 7f03c0890640 1 -- 172.21.15.12:0/1679726906 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03b8032440 con 0x7f03bc137e50 2024-09-11T01:50:02.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.948+0000 7f03c0890640 10 monclient: handle_config config(0 keys) 2024-09-11T01:50:02.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.948+0000 7f03c3af6640 10 monclient: get_monmap_and_config success 2024-09-11T01:50:02.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.948+0000 7f03c3af6640 4 set_mon_vals no callback set 2024-09-11T01:50:02.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.948+0000 7f03c0890640 1 -- 172.21.15.12:0/1679726906 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b8032810 con 0x7f03bc137e50 2024-09-11T01:50:02.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.948+0000 7f03c0890640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:02.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.948+0000 7f03c0890640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:50:02.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.948+0000 7f03c0890640 10 monclient: dump: 2024-09-11T01:50:02.961 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:02.961 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:02.961 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:02.961 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:02.961 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:02.961 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:02.961 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:02.961 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:02.961 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:02.961 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:02.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.948+0000 7f03c3af6640 10 monclient: shutdown 2024-09-11T01:50:02.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.948+0000 7f03c3af6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:50:02.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.948+0000 7f03c3af6640 1 -- 172.21.15.12:0/1679726906 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc137e50 msgr2=0x7f03bc09d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:02.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.948+0000 7f03c3af6640 1 --2- 172.21.15.12:0/1679726906 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc137e50 0x7f03bc09d160 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f03b80097a0 tx=0x7f03b8029fd0 comp rx=0 tx=0).stop 2024-09-11T01:50:02.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.948+0000 7f03c3af6640 1 -- 172.21.15.12:0/1679726906 shutdown_connections 2024-09-11T01:50:02.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.948+0000 7f03c3af6640 1 --2- 172.21.15.12:0/1679726906 >> v2:172.21.15.12:3300/0 conn(0x7f03bc138230 0x7f03bc138630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:02.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.948+0000 7f03c3af6640 1 --2- 172.21.15.12:0/1679726906 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc137e50 0x7f03bc09d160 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:02.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.948+0000 7f03c3af6640 1 -- 172.21.15.12:0/1679726906 >> 172.21.15.12:0/1679726906 conn(0x7f03bc0a7020 msgr2=0x7f03bc0a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:50:02.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.948+0000 7f03c3af6640 1 -- 172.21.15.12:0/1679726906 shutdown_connections 2024-09-11T01:50:02.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.948+0000 7f03c3af6640 1 -- 172.21.15.12:0/1679726906 wait complete. 2024-09-11T01:50:02.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.949+0000 7f03c3af6640 10 monclient: build_initial_monmap 2024-09-11T01:50:02.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.949+0000 7f03c3af6640 10 monclient: monmap: 2024-09-11T01:50:02.962 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:50:02.963 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:50:02.963 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:50:02.963 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:50:02.963 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:50:02.963 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:02.963 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:50:02.963 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:50:02.963 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:50:02.963 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:02.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.949+0000 7f03c3af6640 1 Processor -- start 2024-09-11T01:50:02.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.949+0000 7f03c3af6640 1 -- start start 2024-09-11T01:50:02.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.949+0000 7f03c3af6640 10 monclient: init 2024-09-11T01:50:02.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c3af6640 10 monclient: _reopen_session rank -1 2024-09-11T01:50:02.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c3af6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:50:02.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c3af6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03bc134cb0 0x7f03bc133d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:02.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c3af6640 10 monclient(hunting): picked mon.noname-b con 0x7f03bc134cb0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:50:02.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c3af6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc137e50 0x7f03bc1342d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:02.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c3af6640 10 monclient(hunting): picked mon.noname-c con 0x7f03bc137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:50:02.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c3af6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03bc138230 0x7f03bc1219c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:02.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c3af6640 10 monclient(hunting): picked mon.noname-a con 0x7f03bc138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:50:02.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c3af6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:02.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c3af6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f03bc06f710 con 0x7f03bc138230 2024-09-11T01:50:02.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c3af6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:02.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c3af6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f03bc138b70 con 0x7f03bc134cb0 2024-09-11T01:50:02.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c3af6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:02.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c3af6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f03bc06f890 con 0x7f03bc137e50 2024-09-11T01:50:02.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c3af6640 10 monclient(hunting): _renew_subs 2024-09-11T01:50:02.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c3af6640 10 monclient(hunting): authenticate will time out at 2009.393206s 2024-09-11T01:50:02.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1091640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc137e50 0x7f03bc1342d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:02.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1091640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc137e50 0x7f03bc1342d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45286/0 (socket says 172.21.15.12:45286) 2024-09-11T01:50:02.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1091640 1 -- 172.21.15.12:0/3069445165 learned_addr learned my addr 172.21.15.12:0/3069445165 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:50:02.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1091640 10 monclient(hunting): get_auth_request con 0x7f03bc137e50 auth_method 0 2024-09-11T01:50:02.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1091640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:02.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1091640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:02.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1091640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:02.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c2093640 1 --2- 172.21.15.12:0/3069445165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03bc138230 0x7f03bc1219c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:02.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c2093640 10 monclient(hunting): get_auth_request con 0x7f03bc138230 auth_method 0 2024-09-11T01:50:02.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c2093640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:02.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c2093640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:02.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c2093640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:02.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1892640 1 --2- 172.21.15.12:0/3069445165 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03bc134cb0 0x7f03bc133d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:02.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1091640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:02.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1091640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:02.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1091640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:02.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1892640 10 monclient(hunting): get_auth_request con 0x7f03bc134cb0 auth_method 0 2024-09-11T01:50:02.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1892640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:02.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1892640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:02.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1892640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:02.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c2093640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:02.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c2093640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:02.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c2093640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:02.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1892640 1 -- 172.21.15.12:0/3069445165 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03bc134cb0 msgr2=0x7f03bc133d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:50:02.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1892640 1 -- 172.21.15.12:0/3069445165 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03bc134cb0 msgr2=0x7f03bc133d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:50:02.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1892640 1 --2- 172.21.15.12:0/3069445165 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03bc134cb0 0x7f03bc133d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:50:02.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1892640 1 --2- 172.21.15.12:0/3069445165 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03bc134cb0 0x7f03bc133d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:50:02.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1091640 10 monclient(hunting): handle_auth_done global_id 224192 payload 824 2024-09-11T01:50:02.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1091640 1 -- 172.21.15.12:0/3069445165 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03bc134cb0 msgr2=0x7f03bc133d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:50:02.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1091640 1 --2- 172.21.15.12:0/3069445165 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03bc134cb0 0x7f03bc133d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:02.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1091640 1 -- 172.21.15.12:0/3069445165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03bc138230 msgr2=0x7f03bc1219c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:02.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1091640 1 --2- 172.21.15.12:0/3069445165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03bc138230 0x7f03bc1219c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:02.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1091640 10 monclient: _finish_hunting 0 2024-09-11T01:50:02.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1091640 1 monclient: found mon.noname-c 2024-09-11T01:50:02.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1091640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:50:02.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1091640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:50:02.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.950+0000 7f03c1091640 1 -- 172.21.15.12:0/3069445165 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03bc1ce1f0 con 0x7f03bc137e50 2024-09-11T01:50:02.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.952+0000 7f03c1091640 1 -- 172.21.15.12:0/3069445165 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc137e50 msgr2=0x7f03bc1342d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:02.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.952+0000 7f03c1091640 1 --2- 172.21.15.12:0/3069445165 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc137e50 0x7f03bc1342d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f03ac00ebe0 tx=0x7f03ac00c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:02.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.952+0000 7f03b27fc640 1 -- 172.21.15.12:0/3069445165 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ac017070 con 0x7f03bc137e50 2024-09-11T01:50:02.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:02.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03b27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:50:02.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03b27fc640 10 monclient: dump: 2024-09-11T01:50:02.968 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:02.968 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:02.968 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:02.968 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:02.968 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:02.968 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:02.968 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:02.968 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:02.968 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:02.968 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:02.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03b27fc640 10 monclient: _finish_auth 0 2024-09-11T01:50:02.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03b27fc640 10 monclient: _check_auth_tickets 2024-09-11T01:50:02.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:50:02.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03b27fc640 1 -- 172.21.15.12:0/3069445165 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03ac00ce40 con 0x7f03bc137e50 2024-09-11T01:50:02.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03b27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:50:02.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03b27fc640 1 -- 172.21.15.12:0/3069445165 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ac013690 con 0x7f03bc137e50 2024-09-11T01:50:02.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:02.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:50:02.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03b27fc640 10 monclient: dump: 2024-09-11T01:50:02.969 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:02.969 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:02.969 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:02.969 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:02.969 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:02.970 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:02.970 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:02.970 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:02.970 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:02.970 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:02.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03c3af6640 5 monclient: authenticate success, global_id 224192 2024-09-11T01:50:02.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03c3af6640 10 monclient: _renew_subs 2024-09-11T01:50:02.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03c3af6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:50:02.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03c3af6640 1 -- 172.21.15.12:0/3069445165 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03bc125790 con 0x7f03bc137e50 2024-09-11T01:50:02.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03b3fff640 4 set_mon_vals no callback set 2024-09-11T01:50:02.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03c3af6640 10 monclient: _renew_subs 2024-09-11T01:50:02.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03c3af6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:50:02.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03c3af6640 1 -- 172.21.15.12:0/3069445165 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f03bc134810 con 0x7f03bc137e50 2024-09-11T01:50:02.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03c3af6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:02.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03c3af6640 1 --2- 172.21.15.12:0/3069445165 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc1cd540 0x7f03bc121f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:02.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03c3af6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:02.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03c3af6640 1 -- 172.21.15.12:0/3069445165 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f03bc06a060 con 0x7f03bc1cd540 2024-09-11T01:50:02.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03b27fc640 1 -- 172.21.15.12:0/3069445165 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f03ac024060 con 0x7f03bc137e50 2024-09-11T01:50:02.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03c1892640 1 --2- 172.21.15.12:0/3069445165 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc1cd540 0x7f03bc121f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:02.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03c1892640 10 monclient: get_auth_request con 0x7f03bc1cd540 auth_method 0 2024-09-11T01:50:02.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03c1892640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:02.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03c1892640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:02.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.953+0000 7f03c1892640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:02.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.954+0000 7f03c1892640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:02.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.954+0000 7f03c1892640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:02.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.954+0000 7f03c1892640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:02.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.954+0000 7f03b27fc640 1 --2- 172.21.15.12:0/3069445165 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f03a003e260 0x7f03a0040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:02.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.954+0000 7f03c1892640 10 monclient(hunting): handle_auth_done global_id 224195 payload 824 2024-09-11T01:50:02.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.954+0000 7f03c1892640 1 -- 172.21.15.12:0/3069445165 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f03bc125080 con 0x7f03bc1cd540 2024-09-11T01:50:02.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.954+0000 7f03b27fc640 1 -- 172.21.15.12:0/3069445165 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(705..705 src has 507..705) ==== 4847+0+0 (secure 0 0 0) 0x7f03ac052d40 con 0x7f03bc137e50 2024-09-11T01:50:02.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.954+0000 7f03c2093640 1 --2- 172.21.15.12:0/3069445165 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f03a003e260 0x7f03a0040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:02.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.954+0000 7f03c2093640 10 monclient: get_auth_request con 0x7f03a003e260 auth_method 0 2024-09-11T01:50:02.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.956+0000 7f03c1892640 1 -- 172.21.15.12:0/3069445165 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc1cd540 msgr2=0x7f03bc121f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:02.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.956+0000 7f03c1892640 1 --2- 172.21.15.12:0/3069445165 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc1cd540 0x7f03bc121f00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f03b8034000 tx=0x7f03b80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:02.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.956+0000 7f03b27fc640 1 -- 172.21.15.12:0/3069445165 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b8037040 con 0x7f03bc1cd540 2024-09-11T01:50:02.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.956+0000 7f03b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:02.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.956+0000 7f03b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:50:02.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.956+0000 7f03b27fc640 10 monclient: dump: 2024-09-11T01:50:02.975 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:02.975 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:02.975 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:02.975 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:02.975 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:02.975 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:02.975 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:02.976 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:02.976 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:02.976 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:02.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.956+0000 7f03c2093640 1 --2- 172.21.15.12:0/3069445165 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f03a003e260 0x7f03a0040720 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f03bc123860 tx=0x7f03b4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:02.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.957+0000 7f03b27fc640 1 -- 172.21.15.12:0/3069445165 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f03b803c660 con 0x7f03bc1cd540 2024-09-11T01:50:02.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.957+0000 7f03b27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:02.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.957+0000 7f03b27fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:50:02.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.957+0000 7f03b27fc640 1 -- 172.21.15.12:0/3069445165 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc1cd540 msgr2=0x7f03bc121f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:02.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.957+0000 7f03b27fc640 1 --2- 172.21.15.12:0/3069445165 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc1cd540 0x7f03bc121f00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f03b8034000 tx=0x7f03b80046c0 comp rx=0 tx=0).stop 2024-09-11T01:50:02.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.957+0000 7f03b27fc640 1 -- 172.21.15.12:0/3069445165 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc1cd540 msgr2=0x7f03bc121f00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:50:02.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.957+0000 7f03b27fc640 1 --2- 172.21.15.12:0/3069445165 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc1cd540 0x7f03bc121f00 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f03b8034000 tx=0x7f03b80046c0 comp rx=0 tx=0).stop 2024-09-11T01:50:02.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.963+0000 7f03c3af6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:50:02.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.963+0000 7f03c3af6640 1 --2- 172.21.15.12:0/3069445165 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc122440 0x7f03bc0a1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:02.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.963+0000 7f03c3af6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:02.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.963+0000 7f03c3af6640 1 -- 172.21.15.12:0/3069445165 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f03bc122850 con 0x7f03bc122440 2024-09-11T01:50:02.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.963+0000 7f03c1892640 1 --2- 172.21.15.12:0/3069445165 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc122440 0x7f03bc0a1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:02.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.963+0000 7f03c1892640 10 monclient: get_auth_request con 0x7f03bc122440 auth_method 0 2024-09-11T01:50:02.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.963+0000 7f03c1892640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:02.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.963+0000 7f03c1892640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:02.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.963+0000 7f03c1892640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:02.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.963+0000 7f03c1892640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:02.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.963+0000 7f03c1892640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:02.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.963+0000 7f03c1892640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:02.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.963+0000 7f03c1892640 10 monclient(hunting): handle_auth_done global_id 224198 payload 824 2024-09-11T01:50:02.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.963+0000 7f03c1892640 1 -- 172.21.15.12:0/3069445165 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f03bc0a1600 con 0x7f03bc122440 2024-09-11T01:50:02.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.965+0000 7f03c1892640 1 -- 172.21.15.12:0/3069445165 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc122440 msgr2=0x7f03bc0a1890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:02.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.965+0000 7f03c1892640 1 --2- 172.21.15.12:0/3069445165 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc122440 0x7f03bc0a1890 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f03b80097a0 tx=0x7f03b8008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:02.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.965+0000 7f03b27fc640 1 -- 172.21.15.12:0/3069445165 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b8037050 con 0x7f03bc122440 2024-09-11T01:50:02.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.965+0000 7f03b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:02.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.965+0000 7f03b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:50:02.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.965+0000 7f03b27fc640 10 monclient: dump: 2024-09-11T01:50:02.978 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:02.978 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:02.978 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:02.978 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:02.978 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:02.978 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:02.978 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:02.978 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:02.979 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:02.979 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:02.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.966+0000 7f03b27fc640 1 -- 172.21.15.12:0/3069445165 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7f03b8047850 con 0x7f03bc122440 2024-09-11T01:50:02.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.966+0000 7f03b27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:50:02.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.966+0000 7f03b27fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:50:02.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.966+0000 7f03b27fc640 1 -- 172.21.15.12:0/3069445165 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc122440 msgr2=0x7f03bc0a1890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:02.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.966+0000 7f03b27fc640 1 --2- 172.21.15.12:0/3069445165 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc122440 0x7f03bc0a1890 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f03b80097a0 tx=0x7f03b8008030 comp rx=0 tx=0).stop 2024-09-11T01:50:02.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.966+0000 7f03b27fc640 1 -- 172.21.15.12:0/3069445165 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc122440 msgr2=0x7f03bc0a1890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:50:02.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.966+0000 7f03b27fc640 1 --2- 172.21.15.12:0/3069445165 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc122440 0x7f03bc0a1890 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f03b80097a0 tx=0x7f03b8008030 comp rx=0 tx=0).stop 2024-09-11T01:50:02.979 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:50:02.980 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:50:02.980 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:50:02.980 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:50:02.980 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 194, 2024-09-11T01:50:02.980 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 55738, 2024-09-11T01:50:02.980 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:50:02.980 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:50:02.980 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:50:02.980 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:50:02.980 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:02.980 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:50:02.980 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:50:02.981 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:50:02.981 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:50:02.981 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:50:02.981 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:50:02.981 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:50:02.981 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:50:02.981 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:50:02.981 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:50:02.981 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:50:02.981 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:50:02.981 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:50:02.981 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:50:02.981 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:50:02.981 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:02.982 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:50:02.982 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:50:02.982 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:50:02.982 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:50:02.982 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:50:02.982 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:50:02.982 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:50:02.982 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:50:02.982 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:50:02.982 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:50:02.982 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:50:02.982 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:50:02.982 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:50:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:50:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:50:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:50:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:50:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:50:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:50:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:50:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:50:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:50:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:50:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:50:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:50:02.984 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:50:02.984 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:50:02.984 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:50:02.984 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:50:02.984 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:50:02.984 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:50:02.984 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:50:02.984 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:50:02.984 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:50:02.984 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:50:02.984 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:50:02.984 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:50:02.985 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:50:02.985 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:50:02.985 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:50:02.985 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:50:02.985 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:02.985 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:50:02.985 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:02.985 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:50:02.985 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:02.985 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:50:02.985 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:50:02.985 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:50:02.985 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:50:02.986 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:02.986 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:50:02.986 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:50:02.986 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:02.986 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:02.986 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:02.986 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:50:02.986 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:50:02.986 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:02.986 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:02.986 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:02.986 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:02.986 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:50:02.986 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:50:02.987 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:50:02.987 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:50:02.987 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:50:02.987 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:02.987 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:02.987 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:50:02.987 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:50:02.987 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:50:02.987 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:50:02.987 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:02.987 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:50:02.987 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:50:02.987 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:02.988 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:02.988 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:02.988 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:50:02.988 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:50:02.988 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:02.988 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:02.988 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:02.988 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:02.988 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:50:02.988 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:50:02.988 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:50:02.988 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:50:02.988 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:50:02.989 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:02.989 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:02.989 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:50:02.989 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:50:02.989 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:50:02.989 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:50:02.989 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:02.989 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:50:02.989 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:50:02.989 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:02.989 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:02.989 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:02.992 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:50:02.992 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:50:02.992 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:02.993 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:02.993 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:02.993 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:02.993 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:50:02.993 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:50:02.993 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:50:02.993 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:50:02.993 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:50:02.993 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:02.993 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:02.993 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:02.993 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:50:02.993 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:50:02.993 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:02.994 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:50:02.994 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:50:02.994 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:50:02.994 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:02.994 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:02.994 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:50:02.994 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:02.994 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:50:02.994 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:50:02.994 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:50:02.994 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:02.994 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:02.994 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:50:02.994 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:02.994 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:50:02.995 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:50:02.995 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:50:02.995 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:02.995 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:02.995 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:02.995 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:50:02.995 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:50:02.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.966+0000 7f03c3af6640 1 -- 172.21.15.12:0/3069445165 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f03a003e260 msgr2=0x7f03a0040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:02.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.966+0000 7f03c3af6640 1 --2- 172.21.15.12:0/3069445165 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f03a003e260 0x7f03a0040720 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f03bc123860 tx=0x7f03b4009040 comp rx=0 tx=0).stop 2024-09-11T01:50:02.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.966+0000 7f03c3af6640 10 monclient: shutdown 2024-09-11T01:50:02.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.966+0000 7f03c3af6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:50:02.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.966+0000 7f03c3af6640 1 -- 172.21.15.12:0/3069445165 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc137e50 msgr2=0x7f03bc1342d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:02.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.966+0000 7f03c3af6640 1 --2- 172.21.15.12:0/3069445165 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc137e50 0x7f03bc1342d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f03ac00ebe0 tx=0x7f03ac00c520 comp rx=0 tx=0).stop 2024-09-11T01:50:02.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.966+0000 7f03c2093640 1 -- 172.21.15.12:0/3069445165 reap_dead start 2024-09-11T01:50:02.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.966+0000 7f03c3af6640 1 -- 172.21.15.12:0/3069445165 shutdown_connections 2024-09-11T01:50:02.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.966+0000 7f03c3af6640 1 --2- 172.21.15.12:0/3069445165 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03bc137e50 0x7f03bc1342d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:02.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.966+0000 7f03c3af6640 1 -- 172.21.15.12:0/3069445165 >> 172.21.15.12:0/3069445165 conn(0x7f03bc0a7020 msgr2=0x7f03bc136220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:50:02.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.966+0000 7f03c3af6640 1 -- 172.21.15.12:0/3069445165 shutdown_connections 2024-09-11T01:50:02.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:02.966+0000 7f03c3af6640 1 -- 172.21.15.12:0/3069445165 wait complete. 2024-09-11T01:50:02.997 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:50:02.997 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:50:02.998 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:50:03.026 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2310849 2024-09-11T01:50:03.030 INFO:tasks.rados.rados.0.smithi071.stdout:3155: finishing write tid 1 to smithi07134777-10 2024-09-11T01:50:03.030 INFO:tasks.rados.rados.0.smithi071.stdout:3155: finishing write tid 2 to smithi07134777-10 2024-09-11T01:50:03.030 INFO:tasks.rados.rados.0.smithi071.stdout:3155: finishing write tid 3 to smithi07134777-10 2024-09-11T01:50:03.030 INFO:tasks.rados.rados.0.smithi071.stdout:3155: finishing write tid 4 to smithi07134777-10 2024-09-11T01:50:03.030 INFO:tasks.rados.rados.0.smithi071.stdout:3155: oid 10 updating version 0 to 1955 2024-09-11T01:50:03.030 INFO:tasks.rados.rados.0.smithi071.stdout:3155: oid 10 updating version 1955 to 1956 2024-09-11T01:50:03.030 INFO:tasks.rados.rados.0.smithi071.stdout:3155: oid 10 version 1956 is already newer than 1954 2024-09-11T01:50:03.030 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 1956 (ObjNum 1109 snap 352 seq_num 1109) dirty exists 2024-09-11T01:50:03.030 INFO:tasks.rados.rados.0.smithi071.stdout:3155: left oid 10 (ObjNum 1109 snap 352 seq_num 1109) 2024-09-11T01:50:03.030 INFO:tasks.rados.rados.0.smithi071.stdout:3153: done (3 left) 2024-09-11T01:50:03.030 INFO:tasks.rados.rados.0.smithi071.stdout:3154: done (2 left) 2024-09-11T01:50:03.030 INFO:tasks.rados.rados.0.smithi071.stdout:3155: done (1 left) 2024-09-11T01:50:03.031 INFO:tasks.rados.rados.0.smithi071.stdout:3157: read oid 39 snap 343 2024-09-11T01:50:03.031 INFO:tasks.rados.rados.0.smithi071.stdout:3157: expect deleted 2024-09-11T01:50:03.031 INFO:tasks.rados.rados.0.smithi071.stdout:3158: read oid 13 snap -1 2024-09-11T01:50:03.031 INFO:tasks.rados.rados.0.smithi071.stdout:3158: expect (ObjNum 808 snap 246 seq_num 808) 2024-09-11T01:50:03.031 INFO:tasks.rados.rados.0.smithi071.stdout:3159: copy_from oid 11 from oid 12 current snap is 352 2024-09-11T01:50:03.031 INFO:tasks.rados.rados.0.smithi071.stdout:3156: finishing rollback tid 0 to smithi07134777-36 2024-09-11T01:50:03.031 INFO:tasks.rados.rados.0.smithi071.stdout:3156: finishing rollback tid 1 to smithi07134777-36 2024-09-11T01:50:03.031 INFO:tasks.rados.rados.0.smithi071.stdout:3156: finishing rollback tid 2 to smithi07134777-36 2024-09-11T01:50:03.031 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 1792 (ObjNum 1005 snap 322 seq_num 1005) dirty exists 2024-09-11T01:50:03.031 INFO:tasks.rados.rados.0.smithi071.stdout:3156: done (3 left) 2024-09-11T01:50:03.031 INFO:tasks.rados.rados.0.smithi071.stdout:3160: snap_create 2024-09-11T01:50:03.035 INFO:tasks.rados.rados.0.smithi071.stdout:3158: expect (ObjNum 808 snap 246 seq_num 808) 2024-09-11T01:50:03.047 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1738463 2024-09-11T01:50:03.080 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:50:03.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.098+0000 7f9aabedd640 10 monclient: get_monmap_and_config 2024-09-11T01:50:03.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.098+0000 7f9aabedd640 10 monclient: build_initial_monmap 2024-09-11T01:50:03.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.098+0000 7f9aabedd640 10 monclient: monmap: 2024-09-11T01:50:03.101 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:50:03.101 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:03.101 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:50:03.100514+0000 2024-09-11T01:50:03.101 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:50:03.100514+0000 2024-09-11T01:50:03.101 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:50:03.102 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:03.102 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:50:03.102 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:50:03.102 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:50:03.102 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:50:03.102 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:50:03.102 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:03.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.099+0000 7f9aabedd640 1 Processor -- start 2024-09-11T01:50:03.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.099+0000 7f9aabedd640 1 -- start start 2024-09-11T01:50:03.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.099+0000 7f9aabedd640 10 monclient: init 2024-09-11T01:50:03.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aabedd640 10 monclient: _reopen_session rank -1 2024-09-11T01:50:03.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aabedd640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-11T01:50:03.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aabedd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9aa40a2db0 0x7f9aa406f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:03.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aabedd640 10 monclient(hunting): picked mon.noname-a con 0x7f9aa40a2db0 addr v2:172.21.15.12:3300/0 2024-09-11T01:50:03.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aabedd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9aa406ded0 0x7f9aa40a3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:03.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aabedd640 10 monclient(hunting): picked mon.noname-b con 0x7f9aa406ded0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:50:03.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aabedd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9aa40a36d0 addr v1:172.21.15.71:6789/0 2024-09-11T01:50:03.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aabedd640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9aa40a4650 con 0x7f9aa40a36d0 2024-09-11T01:50:03.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aabedd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:03.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aabedd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f9aa406f1a0 con 0x7f9aa40a2db0 2024-09-11T01:50:03.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aabedd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:03.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aabedd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f9aa406eec0 con 0x7f9aa406ded0 2024-09-11T01:50:03.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aabedd640 10 monclient(hunting): _renew_subs 2024-09-11T01:50:03.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aabedd640 10 monclient(hunting): authenticate will time out at 2009.543147s 2024-09-11T01:50:03.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9478640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9aa406ded0 0x7f9aa40a3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:03.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9c79640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9aa40a2db0 0x7f9aa406f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:03.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9478640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9aa406ded0 0x7f9aa40a3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45322/0 (socket says 172.21.15.12:45322) 2024-09-11T01:50:03.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9c79640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9aa40a2db0 0x7f9aa406f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40058/0 (socket says 172.21.15.12:40058) 2024-09-11T01:50:03.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9478640 1 -- 172.21.15.12:0/3487857634 learned_addr learned my addr 172.21.15.12:0/3487857634 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:50:03.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9478640 10 monclient(hunting): get_auth_request con 0x7f9aa406ded0 auth_method 0 2024-09-11T01:50:03.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9478640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:03.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9478640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:03.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9478640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:03.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9c79640 10 monclient(hunting): get_auth_request con 0x7f9aa40a2db0 auth_method 0 2024-09-11T01:50:03.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:03.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9c79640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:03.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9c79640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:03.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9478640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:03.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9478640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:03.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9478640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:03.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9c79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:03.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9c79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:03.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9c79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:03.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9c79640 10 monclient(hunting): handle_auth_done global_id 214311 payload 274 2024-09-11T01:50:03.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9c79640 1 -- 172.21.15.12:0/3487857634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9aa406ded0 msgr2=0x7f9aa40a3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:50:03.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9c79640 1 --2- 172.21.15.12:0/3487857634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9aa406ded0 0x7f9aa40a3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:03.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9c79640 1 -- 172.21.15.12:0/3487857634 >> v1:172.21.15.71:6789/0 conn(0x7f9aa40a36d0 legacy=0x7f9aa4137f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:50:03.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9c79640 10 monclient: _finish_hunting 0 2024-09-11T01:50:03.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9c79640 1 monclient: found mon.noname-a 2024-09-11T01:50:03.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9c79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:50:03.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9c79640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:50:03.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9c79640 1 -- 172.21.15.12:0/3487857634 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa406ebe0 con 0x7f9aa40a2db0 2024-09-11T01:50:03.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.100+0000 7f9aa9478640 1 --2- 172.21.15.12:0/3487857634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9aa406ded0 0x7f9aa40a3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:50:03.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.103+0000 7f9aa9c79640 1 -- 172.21.15.12:0/3487857634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9aa40a2db0 msgr2=0x7f9aa406f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:03.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.103+0000 7f9aa9c79640 1 --2- 172.21.15.12:0/3487857634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9aa40a2db0 0x7f9aa406f8a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9aa00097a0 tx=0x7f9aa002a050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1cc0c7acf6256b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:03.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.103+0000 7f9aa8c77640 1 -- 172.21.15.12:0/3487857634 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa0037070 con 0x7f9aa40a2db0 2024-09-11T01:50:03.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.103+0000 7f9aa8c77640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:03.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.103+0000 7f9aa8c77640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:50:03.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.103+0000 7f9aa8c77640 10 monclient: dump: 2024-09-11T01:50:03.113 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:03.114 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:03.114 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:03.114 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:03.114 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:03.114 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:03.114 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:03.114 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:03.114 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:03.114 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:03.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.103+0000 7f9aa8c77640 10 monclient: _finish_auth 0 2024-09-11T01:50:03.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.103+0000 7f9aa8c77640 10 monclient: _check_auth_tickets 2024-09-11T01:50:03.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.103+0000 7f9aa8c77640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:50:03.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.103+0000 7f9aa8c77640 1 -- 172.21.15.12:0/3487857634 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aa0032440 con 0x7f9aa40a2db0 2024-09-11T01:50:03.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.103+0000 7f9aabedd640 5 monclient: authenticate success, global_id 214311 2024-09-11T01:50:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.103+0000 7f9aabedd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:50:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.103+0000 7f9aa8c77640 10 monclient: handle_config config(0 keys) 2024-09-11T01:50:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.103+0000 7f9aabedd640 10 monclient: get_monmap_and_config success 2024-09-11T01:50:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.103+0000 7f9aa8c77640 1 -- 172.21.15.12:0/3487857634 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa0032810 con 0x7f9aa40a2db0 2024-09-11T01:50:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.103+0000 7f9aabedd640 4 set_mon_vals no callback set 2024-09-11T01:50:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.103+0000 7f9aa8c77640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.103+0000 7f9aa8c77640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:50:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.103+0000 7f9aa8c77640 10 monclient: dump: 2024-09-11T01:50:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:03.115 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:03.116 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:03.116 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:03.116 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:03.116 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:03.116 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:03.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.104+0000 7f9aabedd640 10 monclient: shutdown 2024-09-11T01:50:03.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.104+0000 7f9aabedd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:50:03.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.104+0000 7f9aabedd640 1 -- 172.21.15.12:0/3487857634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9aa40a2db0 msgr2=0x7f9aa406f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:03.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.104+0000 7f9aabedd640 1 --2- 172.21.15.12:0/3487857634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9aa40a2db0 0x7f9aa406f8a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9aa00097a0 tx=0x7f9aa002a050 comp rx=0 tx=0).stop 2024-09-11T01:50:03.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.104+0000 7f9aabedd640 1 -- 172.21.15.12:0/3487857634 shutdown_connections 2024-09-11T01:50:03.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.104+0000 7f9aabedd640 1 --2- 172.21.15.12:0/3487857634 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9aa406ded0 0x7f9aa40a3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:03.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.104+0000 7f9aabedd640 1 --2- 172.21.15.12:0/3487857634 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9aa40a2db0 0x7f9aa406f8a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:03.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.104+0000 7f9aabedd640 1 -- 172.21.15.12:0/3487857634 >> 172.21.15.12:0/3487857634 conn(0x7f9aa40a83b0 msgr2=0x7f9aa40a87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:50:03.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.104+0000 7f9aabedd640 1 -- 172.21.15.12:0/3487857634 shutdown_connections 2024-09-11T01:50:03.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.104+0000 7f9aabedd640 1 -- 172.21.15.12:0/3487857634 wait complete. 2024-09-11T01:50:03.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.104+0000 7f9aabedd640 10 monclient: build_initial_monmap 2024-09-11T01:50:03.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.104+0000 7f9aabedd640 10 monclient: monmap: 2024-09-11T01:50:03.117 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:50:03.117 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:50:03.117 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:50:03.117 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:50:03.117 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:50:03.117 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:03.117 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:50:03.117 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:50:03.117 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:50:03.118 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:03.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aabedd640 1 Processor -- start 2024-09-11T01:50:03.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aabedd640 1 -- start start 2024-09-11T01:50:03.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aabedd640 10 monclient: init 2024-09-11T01:50:03.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aabedd640 10 monclient: _reopen_session rank -1 2024-09-11T01:50:03.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aabedd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:50:03.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aabedd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa406ded0 0x7f9aa41cd400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:03.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aabedd640 10 monclient(hunting): picked mon.noname-b con 0x7f9aa406ded0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:50:03.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aabedd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9aa40a2db0 0x7f9aa41cd940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:03.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aabedd640 10 monclient(hunting): picked mon.noname-c con 0x7f9aa40a2db0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:50:03.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aabedd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9aa40a36d0 0x7f9aa4124fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:03.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aabedd640 10 monclient(hunting): picked mon.noname-a con 0x7f9aa40a36d0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:50:03.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aabedd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:03.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aabedd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa41387f0 con 0x7f9aa40a36d0 2024-09-11T01:50:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aabedd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aabedd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa4138670 con 0x7f9aa406ded0 2024-09-11T01:50:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aabedd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aabedd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f9aa4138970 con 0x7f9aa40a2db0 2024-09-11T01:50:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aabedd640 10 monclient(hunting): _renew_subs 2024-09-11T01:50:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aabedd640 10 monclient(hunting): authenticate will time out at 2009.548851s 2024-09-11T01:50:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aa9478640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9aa40a2db0 0x7f9aa41cd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aa9478640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9aa40a2db0 0x7f9aa41cd940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45326/0 (socket says 172.21.15.12:45326) 2024-09-11T01:50:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aa9478640 1 -- 172.21.15.12:0/4289207184 learned_addr learned my addr 172.21.15.12:0/4289207184 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:50:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aa9478640 10 monclient(hunting): get_auth_request con 0x7f9aa40a2db0 auth_method 0 2024-09-11T01:50:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aa9478640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aa9478640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aa9478640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aaa47a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9aa40a36d0 0x7f9aa4124fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aaa47a640 10 monclient(hunting): get_auth_request con 0x7f9aa40a36d0 auth_method 0 2024-09-11T01:50:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aaa47a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aaa47a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.105+0000 7f9aaa47a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.106+0000 7f9aa9c79640 1 --2- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa406ded0 0x7f9aa41cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.106+0000 7f9aa9478640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.106+0000 7f9aa9478640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.106+0000 7f9aa9478640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.106+0000 7f9aa9c79640 10 monclient(hunting): get_auth_request con 0x7f9aa406ded0 auth_method 0 2024-09-11T01:50:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.106+0000 7f9aa9c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.106+0000 7f9aa9c79640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.106+0000 7f9aa9c79640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:03.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.106+0000 7f9aaa47a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:03.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.106+0000 7f9aaa47a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:03.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.106+0000 7f9aaa47a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:03.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.106+0000 7f9aa9c79640 1 -- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa406ded0 msgr2=0x7f9aa41cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:50:03.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.106+0000 7f9aa9c79640 1 -- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa406ded0 msgr2=0x7f9aa41cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:50:03.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.106+0000 7f9aa9c79640 1 --2- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa406ded0 0x7f9aa41cd400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:50:03.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.106+0000 7f9aa9c79640 1 --2- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa406ded0 0x7f9aa41cd400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:50:03.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.106+0000 7f9aa9478640 10 monclient(hunting): handle_auth_done global_id 224204 payload 824 2024-09-11T01:50:03.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.106+0000 7f9aa9478640 1 -- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa406ded0 msgr2=0x7f9aa41cd400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:50:03.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.106+0000 7f9aa9478640 1 --2- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa406ded0 0x7f9aa41cd400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:03.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.106+0000 7f9aa9478640 1 -- 172.21.15.12:0/4289207184 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9aa40a36d0 msgr2=0x7f9aa4124fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:03.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.106+0000 7f9aa9478640 1 --2- 172.21.15.12:0/4289207184 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9aa40a36d0 0x7f9aa4124fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:03.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.106+0000 7f9aa9478640 10 monclient: _finish_hunting 0 2024-09-11T01:50:03.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.106+0000 7f9aa9478640 1 monclient: found mon.noname-c 2024-09-11T01:50:03.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.106+0000 7f9aa9478640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:50:03.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.106+0000 7f9aa9478640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:50:03.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.106+0000 7f9aa9478640 1 -- 172.21.15.12:0/4289207184 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa40a5820 con 0x7f9aa40a2db0 2024-09-11T01:50:03.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.106+0000 7f9aaa47a640 1 --2- 172.21.15.12:0/4289207184 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9aa40a36d0 0x7f9aa4124fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:50:03.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.108+0000 7f9aa9478640 1 -- 172.21.15.12:0/4289207184 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9aa40a2db0 msgr2=0x7f9aa41cd940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:03.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.108+0000 7f9aa9478640 1 --2- 172.21.15.12:0/4289207184 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9aa40a2db0 0x7f9aa41cd940 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9a9400ea30 tx=0x7f9a9400eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:03.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.108+0000 7f9a9a7fc640 1 -- 172.21.15.12:0/4289207184 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a94017070 con 0x7f9aa40a2db0 2024-09-11T01:50:03.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.108+0000 7f9a9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:03.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.108+0000 7f9a9a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:50:03.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.108+0000 7f9a9a7fc640 10 monclient: dump: 2024-09-11T01:50:03.122 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:03.122 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:03.122 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:03.123 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:03.123 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:03.123 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:03.123 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:03.123 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:03.123 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:03.123 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:03.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.108+0000 7f9a9a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:50:03.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.108+0000 7f9a9a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:50:03.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.108+0000 7f9a9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:50:03.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.108+0000 7f9aabedd640 5 monclient: authenticate success, global_id 224204 2024-09-11T01:50:03.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.108+0000 7f9a9a7fc640 1 -- 172.21.15.12:0/4289207184 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a9400cc70 con 0x7f9aa40a2db0 2024-09-11T01:50:03.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.108+0000 7f9aabedd640 10 monclient: _renew_subs 2024-09-11T01:50:03.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.108+0000 7f9aabedd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:50:03.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.108+0000 7f9aabedd640 1 -- 172.21.15.12:0/4289207184 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aa409f5a0 con 0x7f9aa40a2db0 2024-09-11T01:50:03.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.108+0000 7f9a9a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:50:03.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.108+0000 7f9a9a7fc640 1 -- 172.21.15.12:0/4289207184 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a94013470 con 0x7f9aa40a2db0 2024-09-11T01:50:03.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.108+0000 7f9a9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:03.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.108+0000 7f9a9a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:50:03.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.108+0000 7f9a9a7fc640 10 monclient: dump: 2024-09-11T01:50:03.124 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:03.124 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:03.124 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:03.125 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:03.125 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:03.125 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:03.125 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:03.125 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:03.125 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:03.125 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:03.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.108+0000 7f9a9bfff640 4 set_mon_vals no callback set 2024-09-11T01:50:03.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.108+0000 7f9aabedd640 10 monclient: _renew_subs 2024-09-11T01:50:03.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.108+0000 7f9aabedd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:50:03.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.108+0000 7f9aabedd640 1 -- 172.21.15.12:0/4289207184 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aa4128d70 con 0x7f9aa40a2db0 2024-09-11T01:50:03.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.109+0000 7f9aabedd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:03.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.109+0000 7f9aabedd640 1 --2- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa4125ca0 0x7f9aa402d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:03.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.109+0000 7f9aabedd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:03.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.109+0000 7f9aabedd640 1 -- 172.21.15.12:0/4289207184 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa406a060 con 0x7f9aa4125ca0 2024-09-11T01:50:03.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.109+0000 7f9aa9478640 1 --2- 172.21.15.12:0/4289207184 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9aa40a2db0 0x7f9aa41cd940 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9a9400ea30 tx=0x7f9a9400eef0 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7f9a94052cd0 osd_map(705..705 src has 507..705) 2024-09-11T01:50:03.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.109+0000 7f9a9a7fc640 1 -- 172.21.15.12:0/4289207184 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f9a94024060 con 0x7f9aa40a2db0 2024-09-11T01:50:03.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.109+0000 7f9aa9c79640 1 --2- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa4125ca0 0x7f9aa402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:03.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.109+0000 7f9aa9c79640 10 monclient: get_auth_request con 0x7f9aa4125ca0 auth_method 0 2024-09-11T01:50:03.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.109+0000 7f9aa9c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:03.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.109+0000 7f9aa9c79640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:03.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.109+0000 7f9aa9c79640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:03.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.109+0000 7f9aa9c79640 1 -- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa4125ca0 msgr2=0x7f9aa402d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:50:03.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.109+0000 7f9aa9c79640 1 -- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa4125ca0 msgr2=0x7f9aa402d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:50:03.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.109+0000 7f9aa9c79640 1 --2- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa4125ca0 0x7f9aa402d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:50:03.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.109+0000 7f9aa9c79640 1 --2- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa4125ca0 0x7f9aa402d440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:50:03.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.110+0000 7f9a9a7fc640 1 --2- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9a8803e210 0x7f9a880406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:03.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.110+0000 7f9aaa47a640 1 --2- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9a8803e210 0x7f9a880406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:03.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.110+0000 7f9aaa47a640 10 monclient: get_auth_request con 0x7f9a8803e210 auth_method 0 2024-09-11T01:50:03.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.112+0000 7f9aaa47a640 1 --2- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9a8803e210 0x7f9a880406d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c00f1d0 tx=0x7f9a9c003d10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:03.175 INFO:tasks.rados.rados.0.smithi071.stdout:3159: finishing copy_from racing read to smithi07134777-11 2024-09-11T01:50:03.175 INFO:tasks.rados.rados.0.smithi071.stdout:3159: finishing copy_from to smithi07134777-11 2024-09-11T01:50:03.175 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 2171 (ObjNum 1105 snap 352 seq_num 1105) dirty exists 2024-09-11T01:50:03.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.310+0000 7f9aa9c79640 1 --2- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa4125ca0 0x7f9aa402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:03.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.310+0000 7f9aa9c79640 10 monclient: get_auth_request con 0x7f9aa4125ca0 auth_method 0 2024-09-11T01:50:03.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.310+0000 7f9aa9c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:03.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.310+0000 7f9aa9c79640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:03.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.310+0000 7f9aa9c79640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:50:03.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.310+0000 7f9aa9c79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:03.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.310+0000 7f9aa9c79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:03.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.310+0000 7f9aa9c79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:03.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.310+0000 7f9aa9c79640 10 monclient(hunting): handle_auth_done global_id 234100 payload 824 2024-09-11T01:50:03.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.311+0000 7f9aa9c79640 1 -- 172.21.15.12:0/4289207184 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9aa4138d40 con 0x7f9aa4125ca0 2024-09-11T01:50:03.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.313+0000 7f9aa9c79640 1 -- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa4125ca0 msgr2=0x7f9aa402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:03.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.313+0000 7f9aa9c79640 1 --2- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa4125ca0 0x7f9aa402d440 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9aa406e860 tx=0x7f9aa00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:03.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.313+0000 7f9a9a7fc640 1 -- 172.21.15.12:0/4289207184 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa0042070 con 0x7f9aa4125ca0 2024-09-11T01:50:03.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.313+0000 7f9a9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:03.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.314+0000 7f9a9a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:50:03.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.314+0000 7f9a9a7fc640 10 monclient: dump: 2024-09-11T01:50:03.316 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:03.316 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:03.317 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:03.317 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:03.317 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:03.317 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:03.317 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:03.317 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:03.317 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:03.317 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:03.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.314+0000 7f9a9a7fc640 1 -- 172.21.15.12:0/4289207184 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9aa003e660 con 0x7f9aa4125ca0 2024-09-11T01:50:03.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.314+0000 7f9a9a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:03.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.314+0000 7f9a9a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:50:03.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.314+0000 7f9a9a7fc640 1 -- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa4125ca0 msgr2=0x7f9aa402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:03.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.314+0000 7f9a9a7fc640 1 --2- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa4125ca0 0x7f9aa402d440 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9aa406e860 tx=0x7f9aa00046c0 comp rx=0 tx=0).stop 2024-09-11T01:50:03.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.314+0000 7f9a9a7fc640 1 -- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa4125ca0 msgr2=0x7f9aa402d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:50:03.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.314+0000 7f9a9a7fc640 1 --2- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa4125ca0 0x7f9aa402d440 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9aa406e860 tx=0x7f9aa00046c0 comp rx=0 tx=0).stop 2024-09-11T01:50:03.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.320+0000 7f9aabedd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:50:03.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.320+0000 7f9aabedd640 1 --2- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa4126080 0x7f9aa402d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:03.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.320+0000 7f9aabedd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:03.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.320+0000 7f9aabedd640 1 -- 172.21.15.12:0/4289207184 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa409d9c0 con 0x7f9aa4126080 2024-09-11T01:50:03.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.321+0000 7f9aa9c79640 1 --2- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa4126080 0x7f9aa402d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:03.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.321+0000 7f9aa9c79640 10 monclient: get_auth_request con 0x7f9aa4126080 auth_method 0 2024-09-11T01:50:03.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.321+0000 7f9aa9c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:03.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.321+0000 7f9aa9c79640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:03.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.321+0000 7f9aa9c79640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:03.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.321+0000 7f9aa9c79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:03.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.321+0000 7f9aa9c79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:03.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.321+0000 7f9aa9c79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:03.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.321+0000 7f9aa9c79640 10 monclient(hunting): handle_auth_done global_id 234103 payload 824 2024-09-11T01:50:03.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.321+0000 7f9aa9c79640 1 -- 172.21.15.12:0/4289207184 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9aa4127570 con 0x7f9aa4126080 2024-09-11T01:50:03.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.323+0000 7f9aa9c79640 1 -- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa4126080 msgr2=0x7f9aa402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:03.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.323+0000 7f9aa9c79640 1 --2- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa4126080 0x7f9aa402d980 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0000c00 tx=0x7f9aa0009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:03.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.324+0000 7f9a9a7fc640 1 -- 172.21.15.12:0/4289207184 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa0037040 con 0x7f9aa4126080 2024-09-11T01:50:03.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.324+0000 7f9a9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:03.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.324+0000 7f9a9a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:50:03.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.324+0000 7f9a9a7fc640 10 monclient: dump: 2024-09-11T01:50:03.327 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:03.327 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:03.327 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:03.327 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:03.327 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:03.327 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:03.327 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:03.327 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:03.327 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:03.328 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:03.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.324+0000 7f9a9a7fc640 1 -- 172.21.15.12:0/4289207184 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4707 (secure 0 0 0) 0x7f9aa0048850 con 0x7f9aa4126080 2024-09-11T01:50:03.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.324+0000 7f9a9a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:50:03.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.324+0000 7f9a9a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:50:03.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.324+0000 7f9a9a7fc640 1 -- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa4126080 msgr2=0x7f9aa402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:03.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.324+0000 7f9a9a7fc640 1 --2- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa4126080 0x7f9aa402d980 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0000c00 tx=0x7f9aa0009190 comp rx=0 tx=0).stop 2024-09-11T01:50:03.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.324+0000 7f9a9a7fc640 1 -- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa4126080 msgr2=0x7f9aa402d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:50:03.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.324+0000 7f9a9a7fc640 1 --2- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9aa4126080 0x7f9aa402d980 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0000c00 tx=0x7f9aa0009190 comp rx=0 tx=0).stop 2024-09-11T01:50:03.329 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:50:03.329 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:50:03.329 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:50:03.329 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:50:03.329 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 194, 2024-09-11T01:50:03.329 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 5216, 2024-09-11T01:50:03.329 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:50:03.330 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:50:03.330 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:50:03.330 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:50:03.330 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:03.330 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:50:03.330 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:50:03.330 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:50:03.330 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:50:03.330 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:50:03.330 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:50:03.330 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:50:03.330 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:50:03.330 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:50:03.330 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:50:03.331 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:50:03.331 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:50:03.331 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:50:03.331 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:50:03.331 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:50:03.331 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:03.331 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:50:03.331 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:50:03.331 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:50:03.331 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:50:03.332 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:50:03.332 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:50:03.332 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:50:03.332 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:50:03.332 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:50:03.332 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:50:03.332 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:50:03.332 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:50:03.332 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:50:03.333 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:03.333 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:03.333 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:50:03.333 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:50:03.333 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:50:03.333 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:50:03.333 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:50:03.333 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:50:03.333 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:50:03.333 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:50:03.333 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:50:03.334 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:50:03.334 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:50:03.334 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:50:03.334 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:50:03.334 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:50:03.334 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:50:03.334 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:50:03.334 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:50:03.334 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:50:03.334 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:50:03.334 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:50:03.334 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:50:03.334 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:50:03.334 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:50:03.335 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:50:03.335 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:50:03.335 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:50:03.335 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:50:03.335 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:50:03.335 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:03.335 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:50:03.335 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:03.335 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:50:03.335 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:03.335 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:50:03.335 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:50:03.335 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:50:03.336 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:50:03.336 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:03.336 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:50:03.336 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:50:03.336 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:03.336 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:03.336 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:03.336 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:50:03.336 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:50:03.337 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:03.337 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:03.337 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:03.337 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:03.337 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:50:03.337 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:50:03.337 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:50:03.337 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:50:03.337 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:50:03.337 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:03.337 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:03.337 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:50:03.337 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:50:03.337 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:50:03.337 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:50:03.338 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:03.338 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:50:03.338 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:50:03.338 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:03.338 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:03.338 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:03.338 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:50:03.338 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:50:03.338 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:03.338 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:03.338 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:03.338 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:03.338 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:50:03.338 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:50:03.339 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:50:03.339 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:50:03.339 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:50:03.339 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:03.339 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:03.339 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:50:03.339 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:50:03.339 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:50:03.339 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:50:03.339 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:03.339 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:50:03.340 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:50:03.340 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:03.340 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:03.340 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:03.340 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:50:03.340 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:50:03.340 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:03.340 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:03.340 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:03.340 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:03.341 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:50:03.341 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:50:03.341 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:50:03.341 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:50:03.341 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:50:03.341 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:03.341 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:03.341 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:03.342 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:50:03.342 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:50:03.342 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:03.342 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:50:03.342 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:50:03.342 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:50:03.342 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:03.342 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:03.342 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:50:03.342 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:03.342 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:50:03.343 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:50:03.343 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:50:03.343 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:03.343 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:03.343 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:50:03.343 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:03.343 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:50:03.343 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:50:03.343 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:50:03.343 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:03.343 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:03.343 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:03.344 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:50:03.344 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:50:03.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.324+0000 7f9aabedd640 1 -- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9a8803e210 msgr2=0x7f9a880406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:03.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.324+0000 7f9aabedd640 1 --2- 172.21.15.12:0/4289207184 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9a8803e210 0x7f9a880406d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c00f1d0 tx=0x7f9a9c003d10 comp rx=0 tx=0).stop 2024-09-11T01:50:03.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.324+0000 7f9aabedd640 10 monclient: shutdown 2024-09-11T01:50:03.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.324+0000 7f9aabedd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:50:03.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.324+0000 7f9aabedd640 1 -- 172.21.15.12:0/4289207184 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9aa40a2db0 msgr2=0x7f9aa41cd940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:03.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.324+0000 7f9aabedd640 1 --2- 172.21.15.12:0/4289207184 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9aa40a2db0 0x7f9aa41cd940 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9a9400ea30 tx=0x7f9a9400eef0 comp rx=0 tx=0).stop 2024-09-11T01:50:03.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.324+0000 7f9aaa47a640 1 -- 172.21.15.12:0/4289207184 reap_dead start 2024-09-11T01:50:03.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.324+0000 7f9a9a7fc640 1 -- 172.21.15.12:0/4289207184 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(705..705 src has 507..705) ==== 4847+0+0 (unknown 0 0 0) 0x7f9a94052cd0 con 0x7f9aa40a2db0 2024-09-11T01:50:03.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.324+0000 7f9aaa47a640 1 -- 172.21.15.12:0/4289207184 reap_dead start 2024-09-11T01:50:03.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.324+0000 7f9aabedd640 1 -- 172.21.15.12:0/4289207184 shutdown_connections 2024-09-11T01:50:03.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.324+0000 7f9aabedd640 1 -- 172.21.15.12:0/4289207184 >> 172.21.15.12:0/4289207184 conn(0x7f9aa40a83b0 msgr2=0x7f9aa40a17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:50:03.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.324+0000 7f9aabedd640 1 -- 172.21.15.12:0/4289207184 shutdown_connections 2024-09-11T01:50:03.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.324+0000 7f9aabedd640 1 -- 172.21.15.12:0/4289207184 wait complete. 2024-09-11T01:50:03.346 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:50:03.346 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:50:03.347 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:50:03.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.445+0000 7fbabbb16640 10 monclient: get_monmap_and_config 2024-09-11T01:50:03.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.445+0000 7fbabbb16640 10 monclient: build_initial_monmap 2024-09-11T01:50:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.445+0000 7fbabbb16640 10 monclient: monmap: 2024-09-11T01:50:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:50:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:50:03.447333+0000 2024-09-11T01:50:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:50:03.447333+0000 2024-09-11T01:50:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:50:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:50:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:50:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:50:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:50:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:50:03.449 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:03.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.446+0000 7fbabbb16640 1 Processor -- start 2024-09-11T01:50:03.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.446+0000 7fbabbb16640 1 -- start start 2024-09-11T01:50:03.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.446+0000 7fbabbb16640 10 monclient: init 2024-09-11T01:50:03.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.446+0000 7fbabbb16640 10 monclient: _reopen_session rank -1 2024-09-11T01:50:03.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.446+0000 7fbabbb16640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-11T01:50:03.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.446+0000 7fbabbb16640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbab4137e50 addr v1:172.21.15.12:6789/0 2024-09-11T01:50:03.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.446+0000 7fbabbb16640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbab4138230 0x7fbab4138640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:03.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.446+0000 7fbabbb16640 10 monclient(hunting): picked mon.noname-c con 0x7fbab4138230 addr v2:172.21.15.71:3300/0 2024-09-11T01:50:03.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.446+0000 7fbabbb16640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbab413d380 0x7fbab413f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:03.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.446+0000 7fbabbb16640 10 monclient(hunting): picked mon.noname-a con 0x7fbab413d380 addr v2:172.21.15.12:3300/0 2024-09-11T01:50:03.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbabbb16640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbab40a62d0 con 0x7fbab4137e50 2024-09-11T01:50:03.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbabbb16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:03.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbabbb16640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fbab409bb40 con 0x7fbab413d380 2024-09-11T01:50:03.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbabbb16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:03.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbabbb16640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fbab409c4a0 con 0x7fbab4138230 2024-09-11T01:50:03.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbabbb16640 10 monclient(hunting): _renew_subs 2024-09-11T01:50:03.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbabbb16640 10 monclient(hunting): authenticate will time out at 2009.890041s 2024-09-11T01:50:03.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbaba0b3640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbab413d380 0x7fbab413f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:03.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbab98b2640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbab4137e50 0x7fbab409d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43874/0 (socket says 172.21.15.12:43874) 2024-09-11T01:50:03.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbab98b2640 1 -- 172.21.15.12:0/1872931521 learned_addr learned my addr 172.21.15.12:0/1872931521 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:50:03.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbaba0b3640 10 monclient(hunting): get_auth_request con 0x7fbab413d380 auth_method 0 2024-09-11T01:50:03.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbaba0b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:03.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbaba0b3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:03.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbaba0b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:03.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbaba0b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:03.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbaba0b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:03.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbaba0b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:03.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbaba0b3640 10 monclient(hunting): handle_auth_done global_id 214317 payload 274 2024-09-11T01:50:03.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbaba0b3640 1 -- 172.21.15.12:0/1872931521 >> v2:172.21.15.71:3300/0 conn(0x7fbab4138230 msgr2=0x7fbab4138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:50:03.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbab90b1640 1 --2- 172.21.15.12:0/1872931521 >> v2:172.21.15.71:3300/0 conn(0x7fbab4138230 0x7fbab4138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:03.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbaba0b3640 1 --2- 172.21.15.12:0/1872931521 >> v2:172.21.15.71:3300/0 conn(0x7fbab4138230 0x7fbab4138640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:03.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbaba0b3640 1 -- 172.21.15.12:0/1872931521 >> v1:172.21.15.12:6789/0 conn(0x7fbab4137e50 legacy=0x7fbab409d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:50:03.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbaba0b3640 10 monclient: _finish_hunting 0 2024-09-11T01:50:03.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbaba0b3640 1 monclient: found mon.noname-a 2024-09-11T01:50:03.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbaba0b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:50:03.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbaba0b3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:50:03.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbaba0b3640 1 -- 172.21.15.12:0/1872931521 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbab409c780 con 0x7fbab413d380 2024-09-11T01:50:03.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbab90b1640 10 monclient: get_auth_request con 0x7fbab4138230 auth_method 0 2024-09-11T01:50:03.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.447+0000 7fbab90b1640 1 --2- 172.21.15.12:0/1872931521 >> v2:172.21.15.71:3300/0 conn(0x7fbab4138230 0x7fbab4138640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:50:03.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbaba0b3640 1 -- 172.21.15.12:0/1872931521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbab413d380 msgr2=0x7fbab413f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:03.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbaba0b3640 1 --2- 172.21.15.12:0/1872931521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbab413d380 0x7fbab413f770 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbab40a6f80 tx=0x7fbaac02a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22bf1b36d62b6831 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:03.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbab88b0640 1 -- 172.21.15.12:0/1872931521 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaac007a30 con 0x7fbab413d380 2024-09-11T01:50:03.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbab88b0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:03.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbab88b0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:50:03.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbab88b0640 10 monclient: dump: 2024-09-11T01:50:03.460 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:03.460 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:03.460 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:03.460 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:03.460 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:03.460 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:03.460 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:03.460 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:03.461 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:03.461 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:03.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbab88b0640 10 monclient: _finish_auth 0 2024-09-11T01:50:03.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbab88b0640 10 monclient: _check_auth_tickets 2024-09-11T01:50:03.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbab88b0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:50:03.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbab88b0640 1 -- 172.21.15.12:0/1872931521 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaac007bd0 con 0x7fbab413d380 2024-09-11T01:50:03.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbabbb16640 5 monclient: authenticate success, global_id 214317 2024-09-11T01:50:03.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbab88b0640 10 monclient: handle_config config(0 keys) 2024-09-11T01:50:03.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbabbb16640 10 monclient: get_monmap_and_config success 2024-09-11T01:50:03.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbabbb16640 4 set_mon_vals no callback set 2024-09-11T01:50:03.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbab88b0640 1 -- 172.21.15.12:0/1872931521 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaac031400 con 0x7fbab413d380 2024-09-11T01:50:03.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbab88b0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:03.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbab88b0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:50:03.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbab88b0640 10 monclient: dump: 2024-09-11T01:50:03.462 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:03.462 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:03.462 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:03.462 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:03.462 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:03.462 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:03.462 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:03.462 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:03.462 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:03.462 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:03.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbabbb16640 10 monclient: shutdown 2024-09-11T01:50:03.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbabbb16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:50:03.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbabbb16640 1 -- 172.21.15.12:0/1872931521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbab413d380 msgr2=0x7fbab413f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbabbb16640 1 --2- 172.21.15.12:0/1872931521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbab413d380 0x7fbab413f770 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbab40a6f80 tx=0x7fbaac02a480 comp rx=0 tx=0).stop 2024-09-11T01:50:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbabbb16640 1 -- 172.21.15.12:0/1872931521 shutdown_connections 2024-09-11T01:50:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbabbb16640 1 --2- 172.21.15.12:0/1872931521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbab413d380 0x7fbab413f770 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbabbb16640 1 --2- 172.21.15.12:0/1872931521 >> v2:172.21.15.71:3300/0 conn(0x7fbab4138230 0x7fbab4138640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbabbb16640 1 -- 172.21.15.12:0/1872931521 >> 172.21.15.12:0/1872931521 conn(0x7fbab40a7020 msgr2=0x7fbab40a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:50:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbabbb16640 1 -- 172.21.15.12:0/1872931521 shutdown_connections 2024-09-11T01:50:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.450+0000 7fbabbb16640 1 -- 172.21.15.12:0/1872931521 wait complete. 2024-09-11T01:50:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.451+0000 7fbabbb16640 10 monclient: build_initial_monmap 2024-09-11T01:50:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.451+0000 7fbabbb16640 10 monclient: monmap: 2024-09-11T01:50:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:50:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:50:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:50:03.463 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:50:03.464 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:50:03.464 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:03.464 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:50:03.464 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:50:03.464 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:50:03.464 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:03.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.451+0000 7fbabbb16640 1 Processor -- start 2024-09-11T01:50:03.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.451+0000 7fbabbb16640 1 -- start start 2024-09-11T01:50:03.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.451+0000 7fbabbb16640 10 monclient: init 2024-09-11T01:50:03.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbabbb16640 10 monclient: _reopen_session rank -1 2024-09-11T01:50:03.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbabbb16640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:50:03.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbabbb16640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbab4137e50 0x7fbab41ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:03.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbabbb16640 10 monclient(hunting): picked mon.noname-b con 0x7fbab4137e50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:50:03.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbabbb16640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbab4138230 0x7fbab4136e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:03.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbabbb16640 10 monclient(hunting): picked mon.noname-a con 0x7fbab4138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:50:03.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbabbb16640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbab413d380 0x7fbab40a1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:03.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbabbb16640 10 monclient(hunting): picked mon.noname-c con 0x7fbab413d380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:50:03.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbabbb16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:03.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbabbb16640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbab406f890 con 0x7fbab4138230 2024-09-11T01:50:03.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbabbb16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:03.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbabbb16640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbab406f710 con 0x7fbab4137e50 2024-09-11T01:50:03.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbabbb16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:03.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbabbb16640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fbab413fcb0 con 0x7fbab413d380 2024-09-11T01:50:03.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbabbb16640 10 monclient(hunting): _renew_subs 2024-09-11T01:50:03.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbabbb16640 10 monclient(hunting): authenticate will time out at 2009.895333s 2024-09-11T01:50:03.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbab90b1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbab4138230 0x7fbab4136e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:03.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbab90b1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbab4138230 0x7fbab4136e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40082/0 (socket says 172.21.15.12:40082) 2024-09-11T01:50:03.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbab90b1640 1 -- 172.21.15.12:0/4001251664 learned_addr learned my addr 172.21.15.12:0/4001251664 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:50:03.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbab90b1640 10 monclient(hunting): get_auth_request con 0x7fbab4138230 auth_method 0 2024-09-11T01:50:03.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbab90b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:03.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbab90b1640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:03.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbab90b1640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:03.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbaba0b3640 1 --2- 172.21.15.12:0/4001251664 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbab413d380 0x7fbab40a1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:03.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbaba0b3640 10 monclient(hunting): get_auth_request con 0x7fbab413d380 auth_method 0 2024-09-11T01:50:03.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbaba0b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:03.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbaba0b3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:03.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbaba0b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:03.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbab98b2640 1 --2- 172.21.15.12:0/4001251664 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbab4137e50 0x7fbab41ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:03.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbab90b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:03.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbab90b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:03.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbab90b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:03.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbab98b2640 1 -- 172.21.15.12:0/4001251664 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbab4137e50 msgr2=0x7fbab41ccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:50:03.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbab98b2640 1 -- 172.21.15.12:0/4001251664 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbab4137e50 msgr2=0x7fbab41ccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:50:03.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbab98b2640 1 --2- 172.21.15.12:0/4001251664 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbab4137e50 0x7fbab41ccda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:50:03.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbab98b2640 1 --2- 172.21.15.12:0/4001251664 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbab4137e50 0x7fbab41ccda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:50:03.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbaba0b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:03.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbaba0b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:03.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbaba0b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:03.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbab90b1640 10 monclient(hunting): handle_auth_done global_id 214320 payload 824 2024-09-11T01:50:03.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbab90b1640 1 -- 172.21.15.12:0/4001251664 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbab413d380 msgr2=0x7fbab40a1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:03.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbab90b1640 1 --2- 172.21.15.12:0/4001251664 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbab413d380 0x7fbab40a1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:03.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbab90b1640 1 -- 172.21.15.12:0/4001251664 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbab4137e50 msgr2=0x7fbab41ccda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:50:03.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbab90b1640 1 --2- 172.21.15.12:0/4001251664 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbab4137e50 0x7fbab41ccda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:03.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbab90b1640 10 monclient: _finish_hunting 0 2024-09-11T01:50:03.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbab90b1640 1 monclient: found mon.noname-a 2024-09-11T01:50:03.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbab90b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:50:03.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbab90b1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:50:03.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbab90b1640 1 -- 172.21.15.12:0/4001251664 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbab40a1710 con 0x7fbab4138230 2024-09-11T01:50:03.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.452+0000 7fbaba0b3640 1 --2- 172.21.15.12:0/4001251664 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbab413d380 0x7fbab40a1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:50:03.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.455+0000 7fbab90b1640 1 -- 172.21.15.12:0/4001251664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbab4138230 msgr2=0x7fbab4136e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:03.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.455+0000 7fbab90b1640 1 --2- 172.21.15.12:0/4001251664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbab4138230 0x7fbab4136e60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbaa400c7f0 tx=0x7fbaa400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:03.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.455+0000 7fbaaa7fc640 1 -- 172.21.15.12:0/4001251664 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa4015070 con 0x7fbab4138230 2024-09-11T01:50:03.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.455+0000 7fbaaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:03.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.455+0000 7fbaaa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:50:03.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.455+0000 7fbaaa7fc640 10 monclient: dump: 2024-09-11T01:50:03.468 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:03.468 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:03.469 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:03.469 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:03.469 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:03.469 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:03.469 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:03.469 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:03.469 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:03.469 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:03.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.455+0000 7fbaaa7fc640 10 monclient: _finish_auth 0 2024-09-11T01:50:03.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.455+0000 7fbaaa7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:50:03.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.455+0000 7fbaaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:50:03.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.455+0000 7fbaaa7fc640 1 -- 172.21.15.12:0/4001251664 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaa4007c40 con 0x7fbab4138230 2024-09-11T01:50:03.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.455+0000 7fbabbb16640 5 monclient: authenticate success, global_id 214320 2024-09-11T01:50:03.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.455+0000 7fbabbb16640 10 monclient: _renew_subs 2024-09-11T01:50:03.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.455+0000 7fbabbb16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:50:03.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.455+0000 7fbabbb16640 1 -- 172.21.15.12:0/4001251664 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbab4137b00 con 0x7fbab4138230 2024-09-11T01:50:03.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.455+0000 7fbaaa7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:50:03.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.455+0000 7fbaaa7fc640 1 -- 172.21.15.12:0/4001251664 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa400ce40 con 0x7fbab4138230 2024-09-11T01:50:03.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.455+0000 7fbaaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:03.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.455+0000 7fbaaa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:50:03.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.455+0000 7fbaaa7fc640 10 monclient: dump: 2024-09-11T01:50:03.470 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:03.470 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:03.470 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:03.470 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:03.470 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:03.470 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:03.470 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:03.471 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:03.471 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:03.471 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:03.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.455+0000 7fbabbb16640 10 monclient: _renew_subs 2024-09-11T01:50:03.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.455+0000 7fbabbb16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:50:03.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.455+0000 7fbaab7fe640 4 set_mon_vals no callback set 2024-09-11T01:50:03.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.455+0000 7fbabbb16640 1 -- 172.21.15.12:0/4001251664 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbab40a4010 con 0x7fbab4138230 2024-09-11T01:50:03.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.456+0000 7fbabbb16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:03.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.456+0000 7fbabbb16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:03.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.456+0000 7fbabbb16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:50:03.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.456+0000 7fbabbb16640 1 -- 172.21.15.12:0/4001251664 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbab4138640 con 0x7fbab4138230 2024-09-11T01:50:03.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.456+0000 7fbaaa7fc640 1 -- 172.21.15.12:0/4001251664 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fbaa40113f0 con 0x7fbab4138230 2024-09-11T01:50:03.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.456+0000 7fbaaa7fc640 1 --2- 172.21.15.12:0/4001251664 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fba9803e210 0x7fba980406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:03.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.456+0000 7fbaaa7fc640 1 -- 172.21.15.12:0/4001251664 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(705..705 src has 507..705) ==== 4847+0+0 (secure 0 0 0) 0x7fbaa4053170 con 0x7fbab4138230 2024-09-11T01:50:03.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.456+0000 7fbab98b2640 1 --2- 172.21.15.12:0/4001251664 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fba9803e210 0x7fba980406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:03.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.456+0000 7fbab98b2640 10 monclient: get_auth_request con 0x7fba9803e210 auth_method 0 2024-09-11T01:50:03.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.459+0000 7fbab98b2640 1 --2- 172.21.15.12:0/4001251664 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fba9803e210 0x7fba980406d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fbab0007950 tx=0x7fbab0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:03.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.461+0000 7fbaaa7fc640 1 -- 172.21.15.12:0/4001251664 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbaa4051b30 con 0x7fbab4138230 2024-09-11T01:50:03.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.461+0000 7fbaaa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:03.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.461+0000 7fbaaa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:50:03.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.717+0000 7fbabbb16640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:50:03.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.717+0000 7fbabbb16640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:50:03.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.717+0000 7fbabbb16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:50:03.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.717+0000 7fbabbb16640 1 -- 172.21.15.12:0/4001251664 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fbab41cdc30 con 0x7fbab4138230 2024-09-11T01:50:03.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.718+0000 7fbaaa7fc640 1 -- 172.21.15.12:0/4001251664 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fbaa4021cf0 con 0x7fbab4138230 2024-09-11T01:50:03.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.718+0000 7fbaaa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:50:03.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.718+0000 7fbaaa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:50:03.722 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.718+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.723 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.718+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=10,kv=12,logm=48} crc {auth=734937575,config=2586702516,health=1040225209,kv=3263697420,logm=3035137882}) 2024-09-11T01:50:03.723 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.718+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=11,kv=10,logm=49} crc {auth=3093544250,config=2586702516,health=2246248209,kv=3209825898,logm=3544317574}) 2024-09-11T01:50:03.723 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.718+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.723 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.718+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=10,kv=12,logm=48} crc {auth=734937575,config=2586702516,health=1040225209,kv=3263697420,logm=3035137882}) 2024-09-11T01:50:03.723 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.718+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=10,kv=12,logm=47} crc {auth=2979844504,config=2586702516,health=3028765855,kv=3263697420,logm=1272706610}) 2024-09-11T01:50:03.723 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.720+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.723 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.720+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=76530039}) 2024-09-11T01:50:03.723 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.720+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4173268768}) 2024-09-11T01:50:03.723 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.720+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.723 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.720+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=76530039}) 2024-09-11T01:50:03.723 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.720+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2130256868}) 2024-09-11T01:50:03.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.721+0000 7fbabbb16640 1 -- 172.21.15.12:0/4001251664 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fba9803e210 msgr2=0x7fba980406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:03.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.721+0000 7fbabbb16640 1 --2- 172.21.15.12:0/4001251664 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fba9803e210 0x7fba980406d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fbab0007950 tx=0x7fbab0009040 comp rx=0 tx=0).stop 2024-09-11T01:50:03.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.721+0000 7fbabbb16640 10 monclient: shutdown 2024-09-11T01:50:03.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.721+0000 7fbabbb16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:50:03.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.721+0000 7fbabbb16640 1 -- 172.21.15.12:0/4001251664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbab4138230 msgr2=0x7fbab4136e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:03.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.721+0000 7fbabbb16640 1 --2- 172.21.15.12:0/4001251664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbab4138230 0x7fbab4136e60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbaa400c7f0 tx=0x7fbaa400cbf0 comp rx=0 tx=0).stop 2024-09-11T01:50:03.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.721+0000 7fbabbb16640 1 -- 172.21.15.12:0/4001251664 shutdown_connections 2024-09-11T01:50:03.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.721+0000 7fbabbb16640 1 --2- 172.21.15.12:0/4001251664 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbab413d380 0x7fbab40a1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:03.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.721+0000 7fbabbb16640 1 --2- 172.21.15.12:0/4001251664 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fba9803e210 0x7fba980406d0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:03.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.721+0000 7fbabbb16640 1 --2- 172.21.15.12:0/4001251664 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbab4138230 0x7fbab4136e60 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:03.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.721+0000 7fbabbb16640 1 --2- 172.21.15.12:0/4001251664 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbab4137e50 0x7fbab41ccda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:03.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.721+0000 7fbabbb16640 1 -- 172.21.15.12:0/4001251664 >> 172.21.15.12:0/4001251664 conn(0x7fbab40a7020 msgr2=0x7fbab413c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:50:03.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.721+0000 7fbabbb16640 1 -- 172.21.15.12:0/4001251664 shutdown_connections 2024-09-11T01:50:03.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:03.721+0000 7fbabbb16640 1 -- 172.21.15.12:0/4001251664 wait complete. 2024-09-11T01:50:03.726 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.721+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.726 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.721+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2163373983}) 2024-09-11T01:50:03.726 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.721+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1623314105}) 2024-09-11T01:50:03.726 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.721+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.727 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.721+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2163373983}) 2024-09-11T01:50:03.727 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.721+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2418857436}) 2024-09-11T01:50:03.727 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.722+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.727 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.722+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=32321106}) 2024-09-11T01:50:03.727 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.722+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1189029778}) 2024-09-11T01:50:03.727 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.722+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.727 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.722+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=32321106}) 2024-09-11T01:50:03.727 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.722+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1767005279}) 2024-09-11T01:50:03.727 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.724+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.727 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.724+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1939116925}) 2024-09-11T01:50:03.727 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.724+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3060220862}) 2024-09-11T01:50:03.727 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.724+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.727 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.724+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1939116925}) 2024-09-11T01:50:03.727 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.724+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1419440652}) 2024-09-11T01:50:03.734 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:50:03.734 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.732+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.735 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.732+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3788993053}) 2024-09-11T01:50:03.735 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.732+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2892580689}) 2024-09-11T01:50:03.735 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.732+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.735 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.732+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3788993053}) 2024-09-11T01:50:03.735 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.732+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2956458542}) 2024-09-11T01:50:03.735 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.733+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.736 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.733+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2635295078}) 2024-09-11T01:50:03.736 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.733+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=470453812}) 2024-09-11T01:50:03.736 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.733+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.736 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.733+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2635295078}) 2024-09-11T01:50:03.736 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.733+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4017744576}) 2024-09-11T01:50:03.736 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.734+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.737 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.734+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3840975001}) 2024-09-11T01:50:03.737 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.734+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3143293916}) 2024-09-11T01:50:03.737 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.734+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.737 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.734+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3840975001}) 2024-09-11T01:50:03.737 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.734+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=118078567}) 2024-09-11T01:50:03.737 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.735+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.738 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.735+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1126481032}) 2024-09-11T01:50:03.738 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.735+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4161363238}) 2024-09-11T01:50:03.738 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.735+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.738 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.735+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1126481032}) 2024-09-11T01:50:03.738 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.735+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3791672983}) 2024-09-11T01:50:03.738 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.736+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.738 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.736+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1094533857}) 2024-09-11T01:50:03.739 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.736+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2120632132}) 2024-09-11T01:50:03.739 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.736+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.739 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.736+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1094533857}) 2024-09-11T01:50:03.739 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.736+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4150636572}) 2024-09-11T01:50:03.739 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.737+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.739 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.737+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4255295755}) 2024-09-11T01:50:03.739 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.737+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3692287002}) 2024-09-11T01:50:03.740 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.737+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.740 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.737+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4255295755}) 2024-09-11T01:50:03.740 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.737+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3873131554}) 2024-09-11T01:50:03.740 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.738+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.740 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.738+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=988263145}) 2024-09-11T01:50:03.741 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.738+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1937329929}) 2024-09-11T01:50:03.741 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.738+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.741 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.738+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=988263145}) 2024-09-11T01:50:03.741 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.738+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3430176077}) 2024-09-11T01:50:03.741 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.739+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.741 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.739+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2182891712}) 2024-09-11T01:50:03.742 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.739+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1492768115}) 2024-09-11T01:50:03.742 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.739+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.742 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.739+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2182891712}) 2024-09-11T01:50:03.742 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.739+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=87244685}) 2024-09-11T01:50:03.742 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.740+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.742 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.740+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=16335343}) 2024-09-11T01:50:03.743 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.740+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3118406786}) 2024-09-11T01:50:03.743 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.740+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.743 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.740+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=16335343}) 2024-09-11T01:50:03.743 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.740+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2781850982}) 2024-09-11T01:50:03.743 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.741+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.743 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.741+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2419250400}) 2024-09-11T01:50:03.743 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.741+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3788739030}) 2024-09-11T01:50:03.744 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.741+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.744 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.741+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2419250400}) 2024-09-11T01:50:03.744 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.741+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1393684349}) 2024-09-11T01:50:03.745 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.742+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.745 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.742+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=76990634}) 2024-09-11T01:50:03.745 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.742+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4177289407}) 2024-09-11T01:50:03.745 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.742+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.745 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.742+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=76990634}) 2024-09-11T01:50:03.745 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.742+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1535347195}) 2024-09-11T01:50:03.745 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.743+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.746 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.743+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2266172156}) 2024-09-11T01:50:03.746 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.743+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2797542447}) 2024-09-11T01:50:03.746 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.743+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.746 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.743+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2266172156}) 2024-09-11T01:50:03.746 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.743+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4126876188}) 2024-09-11T01:50:03.746 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.744+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.747 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.744+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=269176428}) 2024-09-11T01:50:03.747 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.744+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2190206790}) 2024-09-11T01:50:03.747 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.744+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.747 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.744+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=269176428}) 2024-09-11T01:50:03.747 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.744+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=483259014}) 2024-09-11T01:50:03.747 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.745+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.748 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.745+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=709095951}) 2024-09-11T01:50:03.748 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.745+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3295589422}) 2024-09-11T01:50:03.748 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.745+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.748 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.745+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=709095951}) 2024-09-11T01:50:03.748 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.745+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3569881789}) 2024-09-11T01:50:03.748 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.746+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.749 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.746+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3208504471}) 2024-09-11T01:50:03.749 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.746+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=27719971}) 2024-09-11T01:50:03.749 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.746+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.749 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.746+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3208504471}) 2024-09-11T01:50:03.749 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.746+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1521478448}) 2024-09-11T01:50:03.749 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.747+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.750 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.747+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=270858872}) 2024-09-11T01:50:03.750 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.747+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4265942191}) 2024-09-11T01:50:03.750 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.747+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.750 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.747+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=270858872}) 2024-09-11T01:50:03.750 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.747+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=331182672}) 2024-09-11T01:50:03.750 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.748+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.751 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.748+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=187242472}) 2024-09-11T01:50:03.751 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.748+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3164743278}) 2024-09-11T01:50:03.751 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.748+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.751 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.748+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=187242472}) 2024-09-11T01:50:03.751 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.748+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2738532325}) 2024-09-11T01:50:03.751 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.749+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.752 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.749+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=948218348}) 2024-09-11T01:50:03.752 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.749+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2056229047}) 2024-09-11T01:50:03.752 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.749+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.752 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.749+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=948218348}) 2024-09-11T01:50:03.752 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.749+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1226839797}) 2024-09-11T01:50:03.752 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.750+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.753 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.750+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=782919722}) 2024-09-11T01:50:03.753 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.750+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2582663986}) 2024-09-11T01:50:03.753 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.750+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.753 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.750+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=782919722}) 2024-09-11T01:50:03.753 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.750+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=531748907}) 2024-09-11T01:50:03.754 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.750+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.754 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.751+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3646122973}) 2024-09-11T01:50:03.754 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.751+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2591990722}) 2024-09-11T01:50:03.754 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.751+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.754 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.751+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3646122973}) 2024-09-11T01:50:03.754 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.751+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3359574149}) 2024-09-11T01:50:03.755 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.751+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.755 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.751+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=219981504}) 2024-09-11T01:50:03.755 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.752+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3768929559}) 2024-09-11T01:50:03.755 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.752+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.755 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.752+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=219981504}) 2024-09-11T01:50:03.755 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.752+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=929959826}) 2024-09-11T01:50:03.755 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.752+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.755 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.752+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4180882854}) 2024-09-11T01:50:03.756 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.752+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=749569946}) 2024-09-11T01:50:03.756 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.752+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.756 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.753+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4180882854}) 2024-09-11T01:50:03.756 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.753+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1479801020}) 2024-09-11T01:50:03.756 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.753+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.757 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.753+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2780058767}) 2024-09-11T01:50:03.757 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.753+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1861692763}) 2024-09-11T01:50:03.757 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.753+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.757 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.754+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2780058767}) 2024-09-11T01:50:03.757 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.754+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1426852467}) 2024-09-11T01:50:03.757 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.754+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.757 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.754+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2194538809}) 2024-09-11T01:50:03.758 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.754+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3399491611}) 2024-09-11T01:50:03.758 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.754+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.758 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.754+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2194538809}) 2024-09-11T01:50:03.758 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.754+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2268058660}) 2024-09-11T01:50:03.758 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.755+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.759 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.755+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=10909665}) 2024-09-11T01:50:03.759 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.755+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2883220087}) 2024-09-11T01:50:03.759 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.755+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.759 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.755+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=10909665}) 2024-09-11T01:50:03.759 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.755+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2510768045}) 2024-09-11T01:50:03.759 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.756+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.759 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.756+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2628633757}) 2024-09-11T01:50:03.760 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.756+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=678149300}) 2024-09-11T01:50:03.760 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.756+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.760 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.756+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2628633757}) 2024-09-11T01:50:03.760 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.756+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1943888480}) 2024-09-11T01:50:03.760 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.757+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.761 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.757+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=692718913}) 2024-09-11T01:50:03.761 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.757+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3555393249}) 2024-09-11T01:50:03.761 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.757+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.761 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.757+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=692718913}) 2024-09-11T01:50:03.761 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.757+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4263084733}) 2024-09-11T01:50:03.761 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.759+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.762 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.759+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1392291851}) 2024-09-11T01:50:03.762 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.759+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1956138689}) 2024-09-11T01:50:03.762 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.759+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.762 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.759+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1392291851}) 2024-09-11T01:50:03.762 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.759+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3602865238}) 2024-09-11T01:50:03.763 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.760+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.763 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.760+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2820100078}) 2024-09-11T01:50:03.763 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.760+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3540998948}) 2024-09-11T01:50:03.763 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.760+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.763 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.760+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2820100078}) 2024-09-11T01:50:03.763 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.760+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=501600948}) 2024-09-11T01:50:03.764 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.761+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.764 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.761+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3290649199}) 2024-09-11T01:50:03.764 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.761+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3633940903}) 2024-09-11T01:50:03.764 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.761+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.764 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.761+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3290649199}) 2024-09-11T01:50:03.764 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.761+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3279325899}) 2024-09-11T01:50:03.764 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.761+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.765 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.761+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1730068725}) 2024-09-11T01:50:03.765 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.762+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1054680611}) 2024-09-11T01:50:03.765 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.762+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.765 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.762+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1730068725}) 2024-09-11T01:50:03.765 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.762+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=8549772}) 2024-09-11T01:50:03.765 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.762+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.765 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.762+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=179051877}) 2024-09-11T01:50:03.766 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.762+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=876032843}) 2024-09-11T01:50:03.766 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.763+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.766 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.763+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=179051877}) 2024-09-11T01:50:03.766 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.763+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2311755327}) 2024-09-11T01:50:03.766 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.764+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.766 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.764+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2537412139}) 2024-09-11T01:50:03.767 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.764+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1230888592}) 2024-09-11T01:50:03.767 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.764+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.767 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.764+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2537412139}) 2024-09-11T01:50:03.767 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.764+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3229199870}) 2024-09-11T01:50:03.767 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.765+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.767 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.765+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2739158060}) 2024-09-11T01:50:03.768 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.765+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=29296599}) 2024-09-11T01:50:03.768 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.765+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.768 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.765+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2739158060}) 2024-09-11T01:50:03.768 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.765+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3360769541}) 2024-09-11T01:50:03.768 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.766+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.769 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.766+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1232483534}) 2024-09-11T01:50:03.769 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.766+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1119247922}) 2024-09-11T01:50:03.769 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.766+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.769 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.766+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1232483534}) 2024-09-11T01:50:03.769 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.766+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2782361265}) 2024-09-11T01:50:03.770 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.767+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.770 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.767+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=474978346}) 2024-09-11T01:50:03.770 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.767+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4094580613}) 2024-09-11T01:50:03.770 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.767+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.770 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.767+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=474978346}) 2024-09-11T01:50:03.770 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.767+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3913440212}) 2024-09-11T01:50:03.771 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.768+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.771 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.768+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=799326977}) 2024-09-11T01:50:03.771 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.768+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1281026516}) 2024-09-11T01:50:03.771 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.768+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.771 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.768+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=799326977}) 2024-09-11T01:50:03.771 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.768+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3596229376}) 2024-09-11T01:50:03.772 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.769+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.772 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.769+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350318980}) 2024-09-11T01:50:03.772 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.769+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=75523304}) 2024-09-11T01:50:03.772 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.769+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.772 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.769+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350318980}) 2024-09-11T01:50:03.772 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.769+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2537574308}) 2024-09-11T01:50:03.772 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.770+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.773 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.770+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3398393317}) 2024-09-11T01:50:03.773 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.770+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2759217031}) 2024-09-11T01:50:03.773 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.770+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.773 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.770+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3398393317}) 2024-09-11T01:50:03.773 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.770+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=526319521}) 2024-09-11T01:50:03.774 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.771+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.774 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.771+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3914063561}) 2024-09-11T01:50:03.774 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.771+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3112438217}) 2024-09-11T01:50:03.774 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.771+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.774 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.771+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3914063561}) 2024-09-11T01:50:03.774 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.771+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=368848750}) 2024-09-11T01:50:03.775 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.772+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.775 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.772+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=984767467}) 2024-09-11T01:50:03.775 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.772+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1608549249}) 2024-09-11T01:50:03.775 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.772+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.775 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.772+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=984767467}) 2024-09-11T01:50:03.775 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.772+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3896580739}) 2024-09-11T01:50:03.776 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.773+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.776 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.773+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1209987887}) 2024-09-11T01:50:03.776 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.773+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2703105230}) 2024-09-11T01:50:03.776 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.773+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.776 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.773+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1209987887}) 2024-09-11T01:50:03.776 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.773+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4017469029}) 2024-09-11T01:50:03.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.774+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.774+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=834108527}) 2024-09-11T01:50:03.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.774+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3785485636}) 2024-09-11T01:50:03.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.774+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.774+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=834108527}) 2024-09-11T01:50:03.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.774+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2989553434}) 2024-09-11T01:50:03.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.775+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.775+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1058190423}) 2024-09-11T01:50:03.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.775+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=642860132}) 2024-09-11T01:50:03.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.775+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.775+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1058190423}) 2024-09-11T01:50:03.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.775+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=16704954}) 2024-09-11T01:50:03.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.776+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.776+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2294490631}) 2024-09-11T01:50:03.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.776+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2364029722}) 2024-09-11T01:50:03.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.776+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.776+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2294490631}) 2024-09-11T01:50:03.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.776+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1173641211}) 2024-09-11T01:50:03.780 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.777+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.780 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.777+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2357727188}) 2024-09-11T01:50:03.780 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.777+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3318058211}) 2024-09-11T01:50:03.780 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.777+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.780 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.777+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2357727188}) 2024-09-11T01:50:03.780 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.777+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3252289128}) 2024-09-11T01:50:03.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.778+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.778+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4108407650}) 2024-09-11T01:50:03.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.778+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4275237407}) 2024-09-11T01:50:03.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.778+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.778+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4108407650}) 2024-09-11T01:50:03.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.778+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1799741966}) 2024-09-11T01:50:03.782 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.779+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.782 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.779+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1230953287}) 2024-09-11T01:50:03.782 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.779+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2146436747}) 2024-09-11T01:50:03.782 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.779+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.783 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.779+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1230953287}) 2024-09-11T01:50:03.783 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.779+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=896370113}) 2024-09-11T01:50:03.783 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.780+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.783 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.780+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1860845184}) 2024-09-11T01:50:03.783 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.780+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2786454975}) 2024-09-11T01:50:03.783 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.780+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.783 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.780+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1860845184}) 2024-09-11T01:50:03.783 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.780+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1429663764}) 2024-09-11T01:50:03.784 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.781+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.784 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.781+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1544568532}) 2024-09-11T01:50:03.784 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.781+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2807545434}) 2024-09-11T01:50:03.784 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.781+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.784 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.781+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1544568532}) 2024-09-11T01:50:03.784 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.781+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2212906816}) 2024-09-11T01:50:03.785 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.782+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.785 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.782+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3744608833}) 2024-09-11T01:50:03.785 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.782+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1034242937}) 2024-09-11T01:50:03.785 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.782+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.785 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.782+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3744608833}) 2024-09-11T01:50:03.785 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.782+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1217340604}) 2024-09-11T01:50:03.797 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.795+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.797 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.795+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=749432428}) 2024-09-11T01:50:03.798 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.795+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3844586318}) 2024-09-11T01:50:03.798 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.795+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.798 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.795+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=749432428}) 2024-09-11T01:50:03.798 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.795+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1127305160}) 2024-09-11T01:50:03.799 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.796+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.799 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.796+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2325276061}) 2024-09-11T01:50:03.799 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.796+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1272041161}) 2024-09-11T01:50:03.799 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.796+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.799 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.796+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2325276061}) 2024-09-11T01:50:03.799 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.796+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1383650923}) 2024-09-11T01:50:03.800 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.797+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.800 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.797+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4244299584}) 2024-09-11T01:50:03.800 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.797+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=208535599}) 2024-09-11T01:50:03.800 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.797+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.800 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.797+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4244299584}) 2024-09-11T01:50:03.800 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.797+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=46054627}) 2024-09-11T01:50:03.801 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.798+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.801 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.798+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=484390677}) 2024-09-11T01:50:03.801 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.798+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3689925135}) 2024-09-11T01:50:03.801 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.798+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.801 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.798+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=484390677}) 2024-09-11T01:50:03.801 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.798+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1825969864}) 2024-09-11T01:50:03.802 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.799+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.802 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.799+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3744063552}) 2024-09-11T01:50:03.802 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.799+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=565571072}) 2024-09-11T01:50:03.802 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.799+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.802 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.799+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3744063552}) 2024-09-11T01:50:03.802 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.799+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3223639803}) 2024-09-11T01:50:03.803 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.800+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.803 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.800+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3954197427}) 2024-09-11T01:50:03.803 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.800+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1298218766}) 2024-09-11T01:50:03.803 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.800+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.803 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.800+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3954197427}) 2024-09-11T01:50:03.803 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.800+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4017337164}) 2024-09-11T01:50:03.804 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.801+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.804 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.801+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3006881255}) 2024-09-11T01:50:03.804 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.801+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1710928551}) 2024-09-11T01:50:03.804 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.801+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.804 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.801+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3006881255}) 2024-09-11T01:50:03.804 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.801+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1251430072}) 2024-09-11T01:50:03.805 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.802+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.805 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.802+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2062091839}) 2024-09-11T01:50:03.805 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.802+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3060326008}) 2024-09-11T01:50:03.805 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.802+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.805 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.802+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2062091839}) 2024-09-11T01:50:03.805 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.802+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3697188945}) 2024-09-11T01:50:03.806 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.803+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.807 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.803+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2249444300}) 2024-09-11T01:50:03.807 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.803+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3754233760}) 2024-09-11T01:50:03.807 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.803+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.807 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.803+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2249444300}) 2024-09-11T01:50:03.807 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.803+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=959250566}) 2024-09-11T01:50:03.807 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.804+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.807 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.804+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1007656177}) 2024-09-11T01:50:03.807 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.804+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2089232681}) 2024-09-11T01:50:03.807 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.804+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.807 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.804+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1007656177}) 2024-09-11T01:50:03.807 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.804+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=841939009}) 2024-09-11T01:50:03.808 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.805+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.808 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.805+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2954401137}) 2024-09-11T01:50:03.808 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.805+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1006559764}) 2024-09-11T01:50:03.808 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.805+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.808 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.805+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2954401137}) 2024-09-11T01:50:03.808 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.805+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=337322487}) 2024-09-11T01:50:03.809 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.806+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.810 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.806+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=40591228}) 2024-09-11T01:50:03.810 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.806+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=835548577}) 2024-09-11T01:50:03.810 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.806+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.810 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.806+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=40591228}) 2024-09-11T01:50:03.810 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.806+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2348311237}) 2024-09-11T01:50:03.810 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.807+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.810 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.807+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1350505285}) 2024-09-11T01:50:03.810 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.807+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3853258970}) 2024-09-11T01:50:03.810 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.807+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.810 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.807+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1350505285}) 2024-09-11T01:50:03.810 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.807+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1496351989}) 2024-09-11T01:50:03.811 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.808+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.811 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.808+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=409639462}) 2024-09-11T01:50:03.811 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.808+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1942104562}) 2024-09-11T01:50:03.811 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.808+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.811 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.808+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=409639462}) 2024-09-11T01:50:03.811 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.808+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3306643533}) 2024-09-11T01:50:03.812 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.809+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.812 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.809+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2367346212}) 2024-09-11T01:50:03.812 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.809+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=722517521}) 2024-09-11T01:50:03.812 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.809+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.812 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.809+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2367346212}) 2024-09-11T01:50:03.812 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.809+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3144291712}) 2024-09-11T01:50:03.813 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.810+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.813 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.810+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=365074233}) 2024-09-11T01:50:03.813 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.810+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=859576731}) 2024-09-11T01:50:03.813 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.810+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.813 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.810+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=365074233}) 2024-09-11T01:50:03.813 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.810+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1511284699}) 2024-09-11T01:50:03.814 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.811+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.814 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.811+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2326693251}) 2024-09-11T01:50:03.814 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.811+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2220784824}) 2024-09-11T01:50:03.814 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.811+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.814 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.811+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2326693251}) 2024-09-11T01:50:03.814 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.811+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3179438130}) 2024-09-11T01:50:03.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.812+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.812+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399317799}) 2024-09-11T01:50:03.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.812+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2165525942}) 2024-09-11T01:50:03.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.812+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.812+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399317799}) 2024-09-11T01:50:03.816 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.812+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1818309677}) 2024-09-11T01:50:03.816 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.813+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.816 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.813+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=10} crc {logm=3939867590,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1058971,mon_config_key=236727234}) 2024-09-11T01:50:03.816 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.813+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=66,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=8} crc {logm=3516165792,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=3987684566,mon_config_key=3957700848}) 2024-09-11T01:50:03.816 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.813+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.816 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.813+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=10} crc {logm=3939867590,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1058971,mon_config_key=236727234}) 2024-09-11T01:50:03.817 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.813+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=10} crc {logm=3535049829,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=3987684566,mon_config_key=3213767696}) 2024-09-11T01:50:03.817 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.814+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.814+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=5,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=82} crc {mon_config_key=3322174277,monmap=1972551139,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=2454325219,osd_snap=538270600}) 2024-09-11T01:50:03.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.814+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=6,monmap=2,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=83} crc {mon_config_key=622617680,monmap=3140514009,nvmeofgw=1596038039,osd_metadata=2942852742,osd_pg_creating=2454325219,osd_snap=3753215272}) 2024-09-11T01:50:03.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.814+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.814+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=5,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=82} crc {mon_config_key=3322174277,monmap=1972551139,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=2454325219,osd_snap=538270600}) 2024-09-11T01:50:03.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.814+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=6,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=81} crc {mon_config_key=622617680,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=2454325219,osd_snap=2940949775}) 2024-09-11T01:50:03.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.815+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.815+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=60,osdmap=40} crc {osd_snap=4121809690,osdmap=1129542577}) 2024-09-11T01:50:03.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.815+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=62,osdmap=38} crc {osd_snap=1410105848,osdmap=3045260326}) 2024-09-11T01:50:03.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.815+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.815+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=60,osdmap=40} crc {osd_snap=4121809690,osdmap=1129542577}) 2024-09-11T01:50:03.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.815+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=62,osdmap=38} crc {osd_snap=1209158914,osdmap=1730943591}) 2024-09-11T01:50:03.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.816+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.816+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1508585385}) 2024-09-11T01:50:03.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.816+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3300165824}) 2024-09-11T01:50:03.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.816+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.816+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1508585385}) 2024-09-11T01:50:03.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.816+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=425072544}) 2024-09-11T01:50:03.820 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.817+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.820 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.817+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1126003668}) 2024-09-11T01:50:03.820 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.817+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2355393318}) 2024-09-11T01:50:03.820 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.817+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.820 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.817+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1126003668}) 2024-09-11T01:50:03.820 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.817+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4131799627}) 2024-09-11T01:50:03.824 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.821+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.824 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.822+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=61} crc {osdmap=1605928763}) 2024-09-11T01:50:03.824 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.822+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=60} crc {osdmap=1136798768}) 2024-09-11T01:50:03.824 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.822+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:50:03.824 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.822+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=61} crc {osdmap=1605928763}) 2024-09-11T01:50:03.824 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:50:03.822+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=62} crc {osdmap=1993989296}) 2024-09-11T01:50:03.833 INFO:tasks.rados.rados.0.smithi071.stdout:3157: done (3 left) 2024-09-11T01:50:03.833 INFO:tasks.rados.rados.0.smithi071.stdout:3158: done (2 left) 2024-09-11T01:50:03.833 INFO:tasks.rados.rados.0.smithi071.stdout:3159: done (1 left) 2024-09-11T01:50:03.833 INFO:tasks.rados.rados.0.smithi071.stdout:3160: done (0 left) 2024-09-11T01:50:03.833 INFO:tasks.rados.rados.0.smithi071.stdout:3161: rollback oid 46 current snap is 353 2024-09-11T01:50:03.833 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 46 to 324 2024-09-11T01:50:03.833 INFO:tasks.rados.rados.0.smithi071.stdout:3162: snap_create 2024-09-11T01:50:03.841 INFO:tasks.rados.rados.0.smithi071.stdout:3161: finishing rollback tid 0 to smithi07134777-46 2024-09-11T01:50:03.841 INFO:tasks.rados.rados.0.smithi071.stdout:3161: finishing rollback tid 1 to smithi07134777-46 2024-09-11T01:50:03.841 INFO:tasks.rados.rados.0.smithi071.stdout:3161: finishing rollback tid 2 to smithi07134777-46 2024-09-11T01:50:03.841 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 1780 (ObjNum 841 snap 255 seq_num 841) dirty exists 2024-09-11T01:50:04.735 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:50:04.735 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:50:04.842 INFO:tasks.rados.rados.0.smithi071.stdout:3161: done (1 left) 2024-09-11T01:50:04.843 INFO:tasks.rados.rados.0.smithi071.stdout:3162: done (0 left) 2024-09-11T01:50:04.843 INFO:tasks.rados.rados.0.smithi071.stdout:3163: write (excl) oid 27 current snap is 354 2024-09-11T01:50:04.843 INFO:tasks.rados.rados.0.smithi071.stdout:3163: seq_num 1110 ranges {655770=513928,1854542=684057} 2024-09-11T01:50:04.848 INFO:tasks.rados.rados.0.smithi071.stdout:3163: writing smithi07134777-27 from 655770 to 1169698 tid 1 2024-09-11T01:50:04.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.847+0000 7f43d68a2640 10 monclient: get_monmap_and_config 2024-09-11T01:50:04.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.847+0000 7f43d68a2640 10 monclient: build_initial_monmap 2024-09-11T01:50:04.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.847+0000 7f43d68a2640 10 monclient: monmap: 2024-09-11T01:50:04.850 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:50:04.850 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:04.850 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:50:04.849336+0000 2024-09-11T01:50:04.850 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:50:04.849336+0000 2024-09-11T01:50:04.850 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:50:04.850 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:04.850 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:50:04.850 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:50:04.850 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:50:04.851 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:50:04.851 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:50:04.851 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:04.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.848+0000 7f43d68a2640 1 Processor -- start 2024-09-11T01:50:04.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.848+0000 7f43d68a2640 1 -- start start 2024-09-11T01:50:04.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.848+0000 7f43d68a2640 10 monclient: init 2024-09-11T01:50:04.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.849+0000 7f43d68a2640 10 monclient: _reopen_session rank -1 2024-09-11T01:50:04.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.849+0000 7f43d68a2640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-11T01:50:04.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.849+0000 7f43d68a2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f43d00a03a0 addr v1:172.21.15.12:6789/0 2024-09-11T01:50:04.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.849+0000 7f43d68a2640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f43d00a13a0 0x7f43d0139a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:04.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.849+0000 7f43d68a2640 10 monclient(hunting): picked mon.noname-a con 0x7f43d00a13a0 addr v2:172.21.15.12:3300/0 2024-09-11T01:50:04.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.849+0000 7f43d68a2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f43d00a17d0 addr v1:172.21.15.71:6789/0 2024-09-11T01:50:04.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.849+0000 7f43d68a2640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43d009fa00 con 0x7f43d00a03a0 2024-09-11T01:50:04.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.849+0000 7f43d68a2640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43d00a29b0 con 0x7f43d00a17d0 2024-09-11T01:50:04.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.849+0000 7f43d68a2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:04.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.849+0000 7f43d68a2640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f43d00a0780 con 0x7f43d00a13a0 2024-09-11T01:50:04.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.849+0000 7f43d68a2640 10 monclient(hunting): _renew_subs 2024-09-11T01:50:04.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.849+0000 7f43d68a2640 10 monclient(hunting): authenticate will time out at 2011.292468s 2024-09-11T01:50:04.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.849+0000 7f43cffff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f43d00a03a0 0x7f43d00a0c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43876/0 (socket says 172.21.15.12:43876) 2024-09-11T01:50:04.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.849+0000 7f43cffff640 1 -- 172.21.15.12:0/3447014931 learned_addr learned my addr 172.21.15.12:0/3447014931 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:50:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.851+0000 7f43cffff640 1 -- 172.21.15.12:0/3447014931 >> v1:172.21.15.12:6789/0 conn(0x7f43d00a03a0 legacy=0x7f43d00a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.851+0000 7f43cf7fe640 1 --2- 172.21.15.12:0/3447014931 >> v2:172.21.15.12:3300/0 conn(0x7f43d00a13a0 0x7f43d0139a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.851+0000 7f43cf7fe640 10 monclient(hunting): get_auth_request con 0x7f43d00a13a0 auth_method 0 2024-09-11T01:50:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.851+0000 7f43cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.851+0000 7f43cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.851+0000 7f43cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.851+0000 7f43d4e3f640 1 -- 172.21.15.12:0/3447014931 >> v1:172.21.15.71:6789/0 conn(0x7f43d00a17d0 legacy=0x7f43d013bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.851+0000 7f43cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.851+0000 7f43cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:04.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.851+0000 7f43ceffd640 1 -- 172.21.15.12:0/3447014931 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2569829992 0 0) 0x7f43d009fa00 con 0x7f43d00a03a0 2024-09-11T01:50:04.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.851+0000 7f43cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:04.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.852+0000 7f43ceffd640 10 monclient(hunting): _negotiate 2024-09-11T01:50:04.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.852+0000 7f43ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:04.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.852+0000 7f43ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:04.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.852+0000 7f43ceffd640 10 monclient(hunting): my global_id is 214323 2024-09-11T01:50:04.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.852+0000 7f43ceffd640 1 -- 172.21.15.12:0/3447014931 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43b8003520 con 0x7f43d00a03a0 2024-09-11T01:50:04.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.852+0000 7f43ceffd640 1 -- 172.21.15.12:0/3447014931 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3366406074 0 0) 0x7f43d00a29b0 con 0x7f43d00a17d0 2024-09-11T01:50:04.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.852+0000 7f43ceffd640 10 monclient(hunting): _negotiate 2024-09-11T01:50:04.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.852+0000 7f43ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:04.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.852+0000 7f43ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:04.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.852+0000 7f43ceffd640 10 monclient(hunting): my global_id is 234106 2024-09-11T01:50:04.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.852+0000 7f43ceffd640 1 -- 172.21.15.12:0/3447014931 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43d009fa00 con 0x7f43d00a17d0 2024-09-11T01:50:04.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.852+0000 7f43cf7fe640 10 monclient(hunting): handle_auth_done global_id 214326 payload 274 2024-09-11T01:50:04.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.852+0000 7f43cf7fe640 1 -- 172.21.15.12:0/3447014931 >> v1:172.21.15.71:6789/0 conn(0x7f43d00a17d0 legacy=0x7f43d013bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:04.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.852+0000 7f43cf7fe640 1 -- 172.21.15.12:0/3447014931 >> v1:172.21.15.12:6789/0 conn(0x7f43d00a03a0 legacy=0x7f43d00a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:04.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.852+0000 7f43cf7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:50:04.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.852+0000 7f43cf7fe640 1 monclient: found mon.noname-a 2024-09-11T01:50:04.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.852+0000 7f43cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:50:04.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.852+0000 7f43cf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:50:04.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.852+0000 7f43cf7fe640 1 -- 172.21.15.12:0/3447014931 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43d009e510 con 0x7f43d00a13a0 2024-09-11T01:50:04.858 INFO:tasks.rados.rados.0.smithi071.stdout:3163: writing smithi07134777-27 from 1854542 to 2538599 tid 2 2024-09-11T01:50:04.859 INFO:tasks.rados.rados.0.smithi071.stdout:3164: rollback oid 7 current snap is 354 2024-09-11T01:50:04.859 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 7 to 353 2024-09-11T01:50:04.859 INFO:tasks.rados.rados.0.smithi071.stdout:3163: finishing write tid 1 to smithi07134777-27 2024-09-11T01:50:04.859 INFO:tasks.rados.rados.0.smithi071.stdout:3165: delete oid 18 current snap is 354 2024-09-11T01:50:04.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.855+0000 7f43cf7fe640 1 -- 172.21.15.12:0/3447014931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43d00a13a0 msgr2=0x7f43d0139a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:04.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.855+0000 7f43cf7fe640 1 --2- 172.21.15.12:0/3447014931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43d00a13a0 0x7f43d0139a90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f43c40098a0 tx=0x7f43c402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f600cb94046b6341 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:04.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.855+0000 7f43ceffd640 1 -- 172.21.15.12:0/3447014931 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c4037070 con 0x7f43d00a13a0 2024-09-11T01:50:04.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.855+0000 7f43ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:04.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.855+0000 7f43ceffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:50:04.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.855+0000 7f43ceffd640 10 monclient: dump: 2024-09-11T01:50:04.861 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:04.861 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:04.861 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:04.861 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:04.861 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:04.861 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:04.861 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:04.861 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:04.861 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:04.861 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:04.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.855+0000 7f43ceffd640 10 monclient: _finish_auth 0 2024-09-11T01:50:04.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.855+0000 7f43ceffd640 10 monclient: _check_auth_tickets 2024-09-11T01:50:04.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.855+0000 7f43ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:50:04.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.855+0000 7f43ceffd640 1 -- 172.21.15.12:0/3447014931 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43c4032640 con 0x7f43d00a13a0 2024-09-11T01:50:04.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.855+0000 7f43d68a2640 5 monclient: authenticate success, global_id 214326 2024-09-11T01:50:04.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.855+0000 7f43d68a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:50:04.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.855+0000 7f43ceffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:50:04.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.855+0000 7f43d68a2640 10 monclient: get_monmap_and_config success 2024-09-11T01:50:04.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.855+0000 7f43d68a2640 4 set_mon_vals no callback set 2024-09-11T01:50:04.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.855+0000 7f43ceffd640 1 -- 172.21.15.12:0/3447014931 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c4032a10 con 0x7f43d00a13a0 2024-09-11T01:50:04.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.855+0000 7f43ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:04.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.855+0000 7f43ceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:50:04.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.855+0000 7f43ceffd640 10 monclient: dump: 2024-09-11T01:50:04.862 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:04.862 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:04.862 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:04.862 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:04.863 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:04.863 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:04.863 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:04.863 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:04.863 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:04.863 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:04.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.856+0000 7f43d68a2640 10 monclient: shutdown 2024-09-11T01:50:04.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.856+0000 7f43d68a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:50:04.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.856+0000 7f43d68a2640 1 -- 172.21.15.12:0/3447014931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43d00a13a0 msgr2=0x7f43d0139a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:04.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.856+0000 7f43d68a2640 1 --2- 172.21.15.12:0/3447014931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43d00a13a0 0x7f43d0139a90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f43c40098a0 tx=0x7f43c402a210 comp rx=0 tx=0).stop 2024-09-11T01:50:04.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.856+0000 7f43d68a2640 1 -- 172.21.15.12:0/3447014931 shutdown_connections 2024-09-11T01:50:04.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.856+0000 7f43d68a2640 1 --2- 172.21.15.12:0/3447014931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43d00a13a0 0x7f43d0139a90 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f43c40098a0 tx=0x7f43c402a210 comp rx=0 tx=0).stop 2024-09-11T01:50:04.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.856+0000 7f43d68a2640 1 -- 172.21.15.12:0/3447014931 >> 172.21.15.12:0/3447014931 conn(0x7f43d00a2d90 msgr2=0x7f43d00a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:50:04.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.856+0000 7f43d68a2640 1 -- 172.21.15.12:0/3447014931 shutdown_connections 2024-09-11T01:50:04.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.856+0000 7f43d68a2640 1 -- 172.21.15.12:0/3447014931 wait complete. 2024-09-11T01:50:04.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.857+0000 7f43d68a2640 10 monclient: build_initial_monmap 2024-09-11T01:50:04.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.857+0000 7f43d68a2640 10 monclient: monmap: 2024-09-11T01:50:04.864 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:50:04.864 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:50:04.864 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:50:04.864 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:50:04.864 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:50:04.864 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:04.864 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:50:04.864 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:50:04.864 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:50:04.864 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:04.865 INFO:tasks.rados.rados.0.smithi071.stdout:3165: done (2 left) 2024-09-11T01:50:04.865 INFO:tasks.rados.rados.0.smithi071.stdout:3166: snap_remove snap 310 2024-09-11T01:50:04.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.858+0000 7f43d68a2640 1 Processor -- start 2024-09-11T01:50:04.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.858+0000 7f43d68a2640 1 -- start start 2024-09-11T01:50:04.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.858+0000 7f43d68a2640 10 monclient: init 2024-09-11T01:50:04.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43d68a2640 10 monclient: _reopen_session rank -1 2024-09-11T01:50:04.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43d68a2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:50:04.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43d68a2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43d00a03a0 0x7f43d01cd830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:04.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43d68a2640 10 monclient(hunting): picked mon.noname-b con 0x7f43d00a03a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:50:04.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43d68a2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f43d00a17d0 0x7f43d01cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:04.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43d68a2640 10 monclient(hunting): picked mon.noname-c con 0x7f43d00a17d0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:50:04.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43d68a2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43d006f5a0 0x7f43d006fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:04.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43d68a2640 10 monclient(hunting): picked mon.noname-a con 0x7f43d006f5a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:50:04.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43d68a2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43d68a2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f43d009f1d0 con 0x7f43d006f5a0 2024-09-11T01:50:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43d68a2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43d68a2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f43d009f350 con 0x7f43d00a03a0 2024-09-11T01:50:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43d68a2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43d68a2640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f43d0074ba0 con 0x7f43d00a17d0 2024-09-11T01:50:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43d68a2640 10 monclient(hunting): _renew_subs 2024-09-11T01:50:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43d68a2640 10 monclient(hunting): authenticate will time out at 2011.302469s 2024-09-11T01:50:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43cf7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f43d00a17d0 0x7f43d01cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43d4e3f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43d006f5a0 0x7f43d006fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43cf7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f43d00a17d0 0x7f43d01cdd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45346/0 (socket says 172.21.15.12:45346) 2024-09-11T01:50:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43cf7fe640 1 -- 172.21.15.12:0/2657911929 learned_addr learned my addr 172.21.15.12:0/2657911929 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:50:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43cf7fe640 10 monclient(hunting): get_auth_request con 0x7f43d00a17d0 auth_method 0 2024-09-11T01:50:04.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43d4e3f640 10 monclient(hunting): get_auth_request con 0x7f43d006f5a0 auth_method 0 2024-09-11T01:50:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43d4e3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43d4e3f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43d4e3f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43cffff640 1 --2- 172.21.15.12:0/2657911929 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43d00a03a0 0x7f43d01cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43cffff640 10 monclient(hunting): get_auth_request con 0x7f43d00a03a0 auth_method 0 2024-09-11T01:50:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43cffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43d4e3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:04.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43d4e3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:04.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43d4e3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:04.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:04.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:04.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:04.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:04.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:04.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.859+0000 7f43cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:04.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.860+0000 7f43d4e3f640 10 monclient(hunting): handle_auth_done global_id 214329 payload 824 2024-09-11T01:50:04.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.860+0000 7f43d4e3f640 1 -- 172.21.15.12:0/2657911929 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f43d00a17d0 msgr2=0x7f43d01cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:04.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.860+0000 7f43d4e3f640 1 --2- 172.21.15.12:0/2657911929 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f43d00a17d0 0x7f43d01cdd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:04.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.860+0000 7f43d4e3f640 1 -- 172.21.15.12:0/2657911929 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43d00a03a0 msgr2=0x7f43d01cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:04.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.860+0000 7f43d4e3f640 1 --2- 172.21.15.12:0/2657911929 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43d00a03a0 0x7f43d01cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:04.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.860+0000 7f43d4e3f640 10 monclient: _finish_hunting 0 2024-09-11T01:50:04.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.860+0000 7f43d4e3f640 1 monclient: found mon.noname-a 2024-09-11T01:50:04.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.860+0000 7f43d4e3f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:50:04.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.860+0000 7f43d4e3f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:50:04.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.860+0000 7f43d4e3f640 1 -- 172.21.15.12:0/2657911929 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43d0127be0 con 0x7f43d006f5a0 2024-09-11T01:50:04.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.862+0000 7f43d4e3f640 1 -- 172.21.15.12:0/2657911929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43d006f5a0 msgr2=0x7f43d006fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:04.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.862+0000 7f43d4e3f640 1 --2- 172.21.15.12:0/2657911929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43d006f5a0 0x7f43d006fa10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f43c8001960 tx=0x7f43c800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:04.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.862+0000 7f43ccff9640 1 -- 172.21.15.12:0/2657911929 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c8017070 con 0x7f43d006f5a0 2024-09-11T01:50:04.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.862+0000 7f43ccff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:04.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.862+0000 7f43ccff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:50:04.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.862+0000 7f43ccff9640 10 monclient: dump: 2024-09-11T01:50:04.874 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:04.874 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:04.874 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:04.874 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:04.874 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:04.875 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:04.875 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:04.875 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:04.875 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:04.875 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:04.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.862+0000 7f43ccff9640 10 monclient: _finish_auth 0 2024-09-11T01:50:04.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.862+0000 7f43ccff9640 10 monclient: _check_auth_tickets 2024-09-11T01:50:04.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.862+0000 7f43ccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:50:04.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.862+0000 7f43ccff9640 1 -- 172.21.15.12:0/2657911929 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43c800ce50 con 0x7f43d006f5a0 2024-09-11T01:50:04.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.862+0000 7f43d68a2640 5 monclient: authenticate success, global_id 214329 2024-09-11T01:50:04.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.862+0000 7f43d68a2640 10 monclient: _renew_subs 2024-09-11T01:50:04.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.862+0000 7f43d68a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:50:04.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.862+0000 7f43d68a2640 1 -- 172.21.15.12:0/2657911929 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43d0128fa0 con 0x7f43d006f5a0 2024-09-11T01:50:04.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.862+0000 7f43ccff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:50:04.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.862+0000 7f43ccff9640 1 -- 172.21.15.12:0/2657911929 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c800a9b0 con 0x7f43d006f5a0 2024-09-11T01:50:04.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.862+0000 7f43ccff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:04.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.862+0000 7f43ccff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:50:04.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.862+0000 7f43ccff9640 10 monclient: dump: 2024-09-11T01:50:04.876 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:04.876 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:04.876 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:04.876 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:04.876 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:04.876 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:04.876 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:04.876 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:04.876 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:04.876 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:04.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.862+0000 7f43ce7fc640 4 set_mon_vals no callback set 2024-09-11T01:50:04.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.862+0000 7f43d68a2640 10 monclient: _renew_subs 2024-09-11T01:50:04.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.862+0000 7f43d68a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:50:04.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.862+0000 7f43d68a2640 1 -- 172.21.15.12:0/2657911929 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43d0071060 con 0x7f43d006f5a0 2024-09-11T01:50:04.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.863+0000 7f43ccff9640 1 -- 172.21.15.12:0/2657911929 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f43c800ac10 con 0x7f43d006f5a0 2024-09-11T01:50:04.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.863+0000 7f43d68a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:04.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.863+0000 7f43d68a2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:04.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.863+0000 7f43d68a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:50:04.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.863+0000 7f43d68a2640 1 -- 172.21.15.12:0/2657911929 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43d00a0c90 con 0x7f43d006f5a0 2024-09-11T01:50:04.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.863+0000 7f43ccff9640 1 --2- 172.21.15.12:0/2657911929 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f43bc03e290 0x7f43bc040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:04.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.864+0000 7f43ccff9640 1 -- 172.21.15.12:0/2657911929 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(707..707 src has 507..707) ==== 4787+0+0 (secure 0 0 0) 0x7f43c80524c0 con 0x7f43d006f5a0 2024-09-11T01:50:04.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.864+0000 7f43cffff640 1 --2- 172.21.15.12:0/2657911929 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f43bc03e290 0x7f43bc040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:04.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.864+0000 7f43cffff640 10 monclient: get_auth_request con 0x7f43bc03e290 auth_method 0 2024-09-11T01:50:04.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.866+0000 7f43cffff640 1 --2- 172.21.15.12:0/2657911929 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f43bc03e290 0x7f43bc040750 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f43c0001590 tx=0x7f43c0006980 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:04.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.868+0000 7f43ccff9640 1 -- 172.21.15.12:0/2657911929 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f43c8050eb0 con 0x7f43d006f5a0 2024-09-11T01:50:04.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.868+0000 7f43ccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:04.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:04.868+0000 7f43ccff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:50:05.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.127+0000 7f43d68a2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:50:05.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.127+0000 7f43d68a2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:50:05.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.127+0000 7f43d68a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:50:05.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.127+0000 7f43d68a2640 1 -- 172.21.15.12:0/2657911929 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f43d0128d80 con 0x7f43d006f5a0 2024-09-11T01:50:05.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.127+0000 7f43ccff9640 1 -- 172.21.15.12:0/2657911929 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f43c8052190 con 0x7f43d006f5a0 2024-09-11T01:50:05.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.127+0000 7f43ccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:50:05.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.127+0000 7f43ccff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:50:05.130 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":194,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:50:05.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.129+0000 7f43d68a2640 1 -- 172.21.15.12:0/2657911929 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f43bc03e290 msgr2=0x7f43bc040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:05.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.129+0000 7f43d68a2640 1 --2- 172.21.15.12:0/2657911929 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f43bc03e290 0x7f43bc040750 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f43c0001590 tx=0x7f43c0006980 comp rx=0 tx=0).stop 2024-09-11T01:50:05.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.129+0000 7f43d68a2640 10 monclient: shutdown 2024-09-11T01:50:05.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.129+0000 7f43d68a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:50:05.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.129+0000 7f43d68a2640 1 -- 172.21.15.12:0/2657911929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43d006f5a0 msgr2=0x7f43d006fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:05.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.129+0000 7f43d68a2640 1 --2- 172.21.15.12:0/2657911929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43d006f5a0 0x7f43d006fa10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f43c8001960 tx=0x7f43c800cc00 comp rx=0 tx=0).stop 2024-09-11T01:50:05.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.129+0000 7f43d68a2640 1 -- 172.21.15.12:0/2657911929 shutdown_connections 2024-09-11T01:50:05.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.129+0000 7f43d68a2640 1 --2- 172.21.15.12:0/2657911929 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f43bc03e290 0x7f43bc040750 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:05.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.129+0000 7f43d68a2640 1 --2- 172.21.15.12:0/2657911929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43d006f5a0 0x7f43d006fa10 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:05.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.129+0000 7f43d68a2640 1 --2- 172.21.15.12:0/2657911929 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f43d00a17d0 0x7f43d01cdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:05.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.129+0000 7f43d68a2640 1 --2- 172.21.15.12:0/2657911929 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43d00a03a0 0x7f43d01cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:05.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.129+0000 7f43d68a2640 1 -- 172.21.15.12:0/2657911929 >> 172.21.15.12:0/2657911929 conn(0x7f43d00a2d90 msgr2=0x7f43d013dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:50:05.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.129+0000 7f43d68a2640 1 -- 172.21.15.12:0/2657911929 shutdown_connections 2024-09-11T01:50:05.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.129+0000 7f43d68a2640 1 -- 172.21.15.12:0/2657911929 wait complete. 2024-09-11T01:50:05.142 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:50:05.142 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:50:05.142 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:50:05.142 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:50:05.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.242+0000 7f721e4b8640 10 monclient: get_monmap_and_config 2024-09-11T01:50:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.242+0000 7f721e4b8640 10 monclient: build_initial_monmap 2024-09-11T01:50:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.242+0000 7f721e4b8640 10 monclient: monmap: 2024-09-11T01:50:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:50:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:50:05.243992+0000 2024-09-11T01:50:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:50:05.243992+0000 2024-09-11T01:50:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:50:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:50:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:50:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:50:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:50:05.246 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:50:05.246 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:05.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.242+0000 7f721e4b8640 1 Processor -- start 2024-09-11T01:50:05.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.243+0000 7f721e4b8640 1 -- start start 2024-09-11T01:50:05.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.243+0000 7f721e4b8640 10 monclient: init 2024-09-11T01:50:05.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.243+0000 7f721e4b8640 10 monclient: _reopen_session rank -1 2024-09-11T01:50:05.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.243+0000 7f721e4b8640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-11T01:50:05.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.243+0000 7f721e4b8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7218137e50 addr v1:172.21.15.71:6789/0 2024-09-11T01:50:05.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.243+0000 7f721e4b8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7218138230 0x7f7218138660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:05.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.243+0000 7f721e4b8640 10 monclient(hunting): picked mon.noname-b con 0x7f7218138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:50:05.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.243+0000 7f721e4b8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f721813d380 addr v1:172.21.15.12:6789/0 2024-09-11T01:50:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.243+0000 7f721e4b8640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72180a6580 con 0x7f721813d380 2024-09-11T01:50:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.243+0000 7f721e4b8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72180a62d0 con 0x7f7218137e50 2024-09-11T01:50:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.243+0000 7f721e4b8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.243+0000 7f721e4b8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f721809bb90 con 0x7f7218138230 2024-09-11T01:50:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.243+0000 7f721e4b8640 10 monclient(hunting): _renew_subs 2024-09-11T01:50:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.243+0000 7f721e4b8640 10 monclient(hunting): authenticate will time out at 2011.686560s 2024-09-11T01:50:05.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.243+0000 7f721ca55640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f721813d380 0x7f721813f770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43878/0 (socket says 172.21.15.12:43878) 2024-09-11T01:50:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.243+0000 7f721ca55640 1 -- 172.21.15.12:0/4116414936 learned_addr learned my addr 172.21.15.12:0/4116414936 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:50:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.243+0000 7f72177fe640 1 --2- 172.21.15.12:0/4116414936 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7218138230 0x7f7218138660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.243+0000 7f72177fe640 10 monclient(hunting): get_auth_request con 0x7f7218138230 auth_method 0 2024-09-11T01:50:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.243+0000 7f72177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.243+0000 7f72177fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.243+0000 7f72177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.243+0000 7f72177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.243+0000 7f72177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.243+0000 7f72177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.244+0000 7f72177fe640 10 monclient(hunting): handle_auth_done global_id 224213 payload 274 2024-09-11T01:50:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.244+0000 7f72177fe640 1 -- 172.21.15.12:0/4116414936 >> v1:172.21.15.71:6789/0 conn(0x7f7218137e50 legacy=0x7f721809d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:50:05.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.244+0000 7f72177fe640 1 -- 172.21.15.12:0/4116414936 >> v1:172.21.15.12:6789/0 conn(0x7f721813d380 legacy=0x7f721813f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:50:05.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.244+0000 7f72177fe640 10 monclient: _finish_hunting 0 2024-09-11T01:50:05.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.244+0000 7f72177fe640 1 monclient: found mon.noname-b 2024-09-11T01:50:05.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.244+0000 7f72177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:50:05.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.244+0000 7f72177fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:50:05.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.244+0000 7f72177fe640 1 -- 172.21.15.12:0/4116414936 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f721809c4f0 con 0x7f7218138230 2024-09-11T01:50:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.246+0000 7f72177fe640 1 -- 172.21.15.12:0/4116414936 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7218138230 msgr2=0x7f7218138660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.246+0000 7f72177fe640 1 --2- 172.21.15.12:0/4116414936 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7218138230 0x7f7218138660 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f720c0097d0 tx=0x7f720c029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=426a98ca3f00d31d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.247+0000 7f7216ffd640 1 -- 172.21.15.12:0/4116414936 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f720c037070 con 0x7f7218138230 2024-09-11T01:50:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.247+0000 7f7216ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.247+0000 7f7216ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:50:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.247+0000 7f7216ffd640 10 monclient: dump: 2024-09-11T01:50:05.257 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:05.258 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.247+0000 7f7216ffd640 10 monclient: _finish_auth 0 2024-09-11T01:50:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.247+0000 7f7216ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:50:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.247+0000 7f7216ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:50:05.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.247+0000 7f721e4b8640 5 monclient: authenticate success, global_id 224213 2024-09-11T01:50:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.247+0000 7f721e4b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:50:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.247+0000 7f7216ffd640 1 -- 172.21.15.12:0/4116414936 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f720c032440 con 0x7f7218138230 2024-09-11T01:50:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.247+0000 7f7216ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:50:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.247+0000 7f721e4b8640 10 monclient: get_monmap_and_config success 2024-09-11T01:50:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.247+0000 7f7216ffd640 1 -- 172.21.15.12:0/4116414936 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f720c032810 con 0x7f7218138230 2024-09-11T01:50:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.247+0000 7f7216ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.247+0000 7f721e4b8640 4 set_mon_vals no callback set 2024-09-11T01:50:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.247+0000 7f7216ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:50:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.247+0000 7f7216ffd640 10 monclient: dump: 2024-09-11T01:50:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:05.259 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:05.260 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:05.260 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:05.260 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:05.260 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:05.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.247+0000 7f721e4b8640 10 monclient: shutdown 2024-09-11T01:50:05.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.247+0000 7f721e4b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:50:05.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.247+0000 7f721e4b8640 1 -- 172.21.15.12:0/4116414936 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7218138230 msgr2=0x7f7218138660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:05.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.247+0000 7f721e4b8640 1 --2- 172.21.15.12:0/4116414936 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7218138230 0x7f7218138660 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f720c0097d0 tx=0x7f720c029fb0 comp rx=0 tx=0).stop 2024-09-11T01:50:05.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.247+0000 7f721e4b8640 1 -- 172.21.15.12:0/4116414936 shutdown_connections 2024-09-11T01:50:05.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.247+0000 7f721e4b8640 1 --2- 172.21.15.12:0/4116414936 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7218138230 0x7f7218138660 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:05.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.247+0000 7f721e4b8640 1 -- 172.21.15.12:0/4116414936 >> 172.21.15.12:0/4116414936 conn(0x7f72180a7020 msgr2=0x7f72180a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:50:05.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.247+0000 7f721e4b8640 1 -- 172.21.15.12:0/4116414936 shutdown_connections 2024-09-11T01:50:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.247+0000 7f721e4b8640 1 -- 172.21.15.12:0/4116414936 wait complete. 2024-09-11T01:50:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.248+0000 7f721e4b8640 10 monclient: build_initial_monmap 2024-09-11T01:50:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.248+0000 7f721e4b8640 10 monclient: monmap: 2024-09-11T01:50:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:50:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:50:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:50:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:50:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:50:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:50:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:50:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:50:05.261 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:05.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.248+0000 7f721e4b8640 1 Processor -- start 2024-09-11T01:50:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.248+0000 7f721e4b8640 1 -- start start 2024-09-11T01:50:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.248+0000 7f721e4b8640 10 monclient: init 2024-09-11T01:50:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f721e4b8640 10 monclient: _reopen_session rank -1 2024-09-11T01:50:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f721e4b8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:50:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f721e4b8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7218137e50 0x7f721813b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f721e4b8640 10 monclient(hunting): picked mon.noname-b con 0x7f7218137e50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:50:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f721e4b8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7218138230 0x7f7218136fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f721e4b8640 10 monclient(hunting): picked mon.noname-c con 0x7f7218138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:50:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f721e4b8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721813d380 0x7f72181cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f721e4b8640 10 monclient(hunting): picked mon.noname-a con 0x7f721813d380 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:50:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f721e4b8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f721e4b8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f721806f890 con 0x7f721813d380 2024-09-11T01:50:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f721e4b8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f721e4b8640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f721806f710 con 0x7f7218137e50 2024-09-11T01:50:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f721e4b8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f721e4b8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7218140e90 con 0x7f7218138230 2024-09-11T01:50:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f721e4b8640 10 monclient(hunting): _renew_subs 2024-09-11T01:50:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f721e4b8640 10 monclient(hunting): authenticate will time out at 2011.692371s 2024-09-11T01:50:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f72177fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7218138230 0x7f7218136fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f721ca55640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721813d380 0x7f72181cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f72177fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7218138230 0x7f7218136fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45370/0 (socket says 172.21.15.12:45370) 2024-09-11T01:50:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f72177fe640 1 -- 172.21.15.12:0/3683986363 learned_addr learned my addr 172.21.15.12:0/3683986363 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:50:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f721ca55640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721813d380 0x7f72181cd0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40116/0 (socket says 172.21.15.12:40116) 2024-09-11T01:50:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f72177fe640 10 monclient(hunting): get_auth_request con 0x7f7218138230 auth_method 0 2024-09-11T01:50:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f72177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:05.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f72177fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:05.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f72177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:05.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f721ca55640 10 monclient(hunting): get_auth_request con 0x7f721813d380 auth_method 0 2024-09-11T01:50:05.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f721ca55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:05.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f721ca55640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:05.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f721ca55640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:05.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f72177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:05.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f72177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:05.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f72177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:05.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f721ca55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:05.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f721ca55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:05.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f721ca55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:05.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f7217fff640 1 --2- 172.21.15.12:0/3683986363 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7218137e50 0x7f721813b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:05.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f7217fff640 10 monclient(hunting): get_auth_request con 0x7f7218137e50 auth_method 0 2024-09-11T01:50:05.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f7217fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:05.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f7217fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:05.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f7217fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:05.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f72177fe640 10 monclient(hunting): handle_auth_done global_id 224216 payload 824 2024-09-11T01:50:05.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f72177fe640 1 -- 172.21.15.12:0/3683986363 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7218137e50 msgr2=0x7f721813b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:05.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f72177fe640 1 --2- 172.21.15.12:0/3683986363 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7218137e50 0x7f721813b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:05.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f72177fe640 1 -- 172.21.15.12:0/3683986363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721813d380 msgr2=0x7f72181cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:05.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f72177fe640 1 --2- 172.21.15.12:0/3683986363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721813d380 0x7f72181cd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:05.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f72177fe640 10 monclient: _finish_hunting 0 2024-09-11T01:50:05.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f72177fe640 1 monclient: found mon.noname-c 2024-09-11T01:50:05.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f72177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:50:05.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f72177fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:50:05.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f72177fe640 1 -- 172.21.15.12:0/3683986363 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f721813cba0 con 0x7f7218138230 2024-09-11T01:50:05.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.249+0000 7f721ca55640 1 --2- 172.21.15.12:0/3683986363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721813d380 0x7f72181cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:50:05.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.252+0000 7f72177fe640 1 -- 172.21.15.12:0/3683986363 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7218138230 msgr2=0x7f7218136fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:05.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.252+0000 7f72177fe640 1 --2- 172.21.15.12:0/3683986363 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7218138230 0x7f7218136fd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f720c009720 tx=0x7f720c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:05.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.252+0000 7f7214ff9640 1 -- 172.21.15.12:0/3683986363 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f720c041070 con 0x7f7218138230 2024-09-11T01:50:05.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.252+0000 7f7214ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:05.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.252+0000 7f7214ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:50:05.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.252+0000 7f7214ff9640 10 monclient: dump: 2024-09-11T01:50:05.266 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:05.266 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:05.266 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.266 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.267 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:05.267 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:05.267 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:05.267 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:05.267 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:05.267 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:05.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.252+0000 7f7214ff9640 10 monclient: _finish_auth 0 2024-09-11T01:50:05.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.252+0000 7f7214ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:50:05.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.252+0000 7f7214ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:50:05.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.252+0000 7f7214ff9640 1 -- 172.21.15.12:0/3683986363 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f720c03c870 con 0x7f7218138230 2024-09-11T01:50:05.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.252+0000 7f721e4b8640 5 monclient: authenticate success, global_id 224216 2024-09-11T01:50:05.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.252+0000 7f721e4b8640 10 monclient: _renew_subs 2024-09-11T01:50:05.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.252+0000 7f721e4b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:50:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.252+0000 7f721e4b8640 1 -- 172.21.15.12:0/3683986363 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f721813ce80 con 0x7f7218138230 2024-09-11T01:50:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.252+0000 7f7214ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:50:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.252+0000 7f7214ff9640 1 -- 172.21.15.12:0/3683986363 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f720c037040 con 0x7f7218138230 2024-09-11T01:50:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.252+0000 7f7214ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.252+0000 7f7214ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:50:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.252+0000 7f7214ff9640 10 monclient: dump: 2024-09-11T01:50:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:05.268 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:05.269 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:05.269 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:05.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.252+0000 7f721e4b8640 10 monclient: _renew_subs 2024-09-11T01:50:05.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.252+0000 7f721e4b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:50:05.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.252+0000 7f721e4b8640 1 -- 172.21.15.12:0/3683986363 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f721813c2f0 con 0x7f7218138230 2024-09-11T01:50:05.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.252+0000 7f72167fc640 4 set_mon_vals no callback set 2024-09-11T01:50:05.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.253+0000 7f721e4b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:05.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.253+0000 7f721e4b8640 1 --2- 172.21.15.12:0/3683986363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7218139820 0x7f721802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:05.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.253+0000 7f721e4b8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:05.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.253+0000 7f721e4b8640 1 -- 172.21.15.12:0/3683986363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f721813d1c0 con 0x7f7218139820 2024-09-11T01:50:05.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.253+0000 7f7217fff640 1 --2- 172.21.15.12:0/3683986363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7218139820 0x7f721802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:05.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.253+0000 7f7217fff640 10 monclient: get_auth_request con 0x7f7218139820 auth_method 0 2024-09-11T01:50:05.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.253+0000 7f7217fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:05.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.253+0000 7f7217fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:05.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.253+0000 7f7217fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:05.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.253+0000 7f7217fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:05.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.253+0000 7f7217fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:05.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.253+0000 7f7217fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:05.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.253+0000 7f7214ff9640 1 -- 172.21.15.12:0/3683986363 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f720c03cc60 con 0x7f7218138230 2024-09-11T01:50:05.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.253+0000 7f7217fff640 10 monclient(hunting): handle_auth_done global_id 214335 payload 824 2024-09-11T01:50:05.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.253+0000 7f7217fff640 1 -- 172.21.15.12:0/3683986363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7218138660 con 0x7f7218139820 2024-09-11T01:50:05.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.253+0000 7f7214ff9640 1 --2- 172.21.15.12:0/3683986363 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f720403e210 0x7f72040406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:05.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.253+0000 7f7214ff9640 1 -- 172.21.15.12:0/3683986363 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(707..707 src has 507..707) ==== 4787+0+0 (secure 0 0 0) 0x7f720c073800 con 0x7f7218138230 2024-09-11T01:50:05.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.253+0000 7f721ca55640 1 --2- 172.21.15.12:0/3683986363 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f720403e210 0x7f72040406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:05.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.253+0000 7f721ca55640 10 monclient: get_auth_request con 0x7f720403e210 auth_method 0 2024-09-11T01:50:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.255+0000 7f7217fff640 1 -- 172.21.15.12:0/3683986363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7218139820 msgr2=0x7f721802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.255+0000 7f7217fff640 1 --2- 172.21.15.12:0/3683986363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7218139820 0x7f721802d440 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f72181ce170 tx=0x7f720800e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.255+0000 7f7214ff9640 1 -- 172.21.15.12:0/3683986363 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7208007450 con 0x7f7218139820 2024-09-11T01:50:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.255+0000 7f7214ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.255+0000 7f7214ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:50:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.255+0000 7f7214ff9640 10 monclient: dump: 2024-09-11T01:50:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:05.273 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.274 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.274 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:05.274 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:05.274 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:05.274 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:05.274 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:05.274 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:05.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.256+0000 7f7214ff9640 1 -- 172.21.15.12:0/3683986363 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7208007450 con 0x7f7218139820 2024-09-11T01:50:05.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.256+0000 7f7214ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:05.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.256+0000 7f7214ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:50:05.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.256+0000 7f7214ff9640 1 -- 172.21.15.12:0/3683986363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7218139820 msgr2=0x7f721802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.256+0000 7f7214ff9640 1 --2- 172.21.15.12:0/3683986363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7218139820 0x7f721802d440 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f72181ce170 tx=0x7f720800e960 comp rx=0 tx=0).stop 2024-09-11T01:50:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.256+0000 7f7214ff9640 1 -- 172.21.15.12:0/3683986363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7218139820 msgr2=0x7f721802d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:50:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.256+0000 7f7214ff9640 1 --2- 172.21.15.12:0/3683986363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7218139820 0x7f721802d440 secure :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f72181ce170 tx=0x7f720800e960 comp rx=0 tx=0).stop 2024-09-11T01:50:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.256+0000 7f721ca55640 1 --2- 172.21.15.12:0/3683986363 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f720403e210 0x7f72040406d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7210005c00 tx=0x7f7210009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.262+0000 7f721e4b8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:50:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.262+0000 7f721e4b8640 1 --2- 172.21.15.12:0/3683986363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721802d980 0x7f721802dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.262+0000 7f721e4b8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.262+0000 7f721e4b8640 1 -- 172.21.15.12:0/3683986363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7218140590 con 0x7f721802d980 2024-09-11T01:50:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.262+0000 7f7217fff640 1 --2- 172.21.15.12:0/3683986363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721802d980 0x7f721802dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.262+0000 7f7217fff640 10 monclient: get_auth_request con 0x7f721802d980 auth_method 0 2024-09-11T01:50:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.262+0000 7f7217fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:05.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.262+0000 7f7217fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:05.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.262+0000 7f7217fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:05.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.262+0000 7f7217fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:05.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.262+0000 7f7217fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:05.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.262+0000 7f7217fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:05.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.262+0000 7f7217fff640 10 monclient(hunting): handle_auth_done global_id 214338 payload 824 2024-09-11T01:50:05.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.262+0000 7f7217fff640 1 -- 172.21.15.12:0/3683986363 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f72181cddb0 con 0x7f721802d980 2024-09-11T01:50:05.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.264+0000 7f7217fff640 1 -- 172.21.15.12:0/3683986363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721802d980 msgr2=0x7f721802dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:05.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.264+0000 7f7217fff640 1 --2- 172.21.15.12:0/3683986363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721802d980 0x7f721802dd90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7208009010 tx=0x7f7208003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:05.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.264+0000 7f7214ff9640 1 -- 172.21.15.12:0/3683986363 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f720801eb70 con 0x7f721802d980 2024-09-11T01:50:05.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.264+0000 7f7214ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:05.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.264+0000 7f7214ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:50:05.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.264+0000 7f7214ff9640 10 monclient: dump: 2024-09-11T01:50:05.276 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:05.277 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.265+0000 7f7214ff9640 1 -- 172.21.15.12:0/3683986363 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f720801eb70 con 0x7f721802d980 2024-09-11T01:50:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.265+0000 7f7214ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:50:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.265+0000 7f7214ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:50:05.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.265+0000 7f7214ff9640 1 -- 172.21.15.12:0/3683986363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721802d980 msgr2=0x7f721802dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:05.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.265+0000 7f7214ff9640 1 --2- 172.21.15.12:0/3683986363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721802d980 0x7f721802dd90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7208009010 tx=0x7f7208003760 comp rx=0 tx=0).stop 2024-09-11T01:50:05.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.265+0000 7f7214ff9640 1 -- 172.21.15.12:0/3683986363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721802d980 msgr2=0x7f721802dd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:50:05.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.265+0000 7f7214ff9640 1 --2- 172.21.15.12:0/3683986363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f721802d980 0x7f721802dd90 secure :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7208009010 tx=0x7f7208003760 comp rx=0 tx=0).stop 2024-09-11T01:50:05.278 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:50:05.278 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:50:05.278 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:50:05.278 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:50:05.278 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 194, 2024-09-11T01:50:05.278 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 111466, 2024-09-11T01:50:05.278 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:50:05.279 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:50:05.279 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:50:05.279 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:50:05.279 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:05.279 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:50:05.279 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:50:05.279 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:50:05.279 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:50:05.279 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:50:05.279 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:50:05.279 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:50:05.279 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:50:05.280 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:50:05.280 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:50:05.280 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:50:05.280 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:50:05.280 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:50:05.280 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:50:05.280 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:50:05.280 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:05.280 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:50:05.280 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:50:05.280 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:50:05.280 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:50:05.280 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:50:05.281 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:50:05.281 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:50:05.281 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:50:05.281 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:50:05.281 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:50:05.281 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:50:05.281 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:50:05.281 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:50:05.281 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:05.281 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.282 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:50:05.282 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:50:05.282 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:50:05.282 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:50:05.282 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:50:05.282 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:50:05.282 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:50:05.282 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:50:05.282 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:50:05.282 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:50:05.282 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:50:05.282 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:50:05.282 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:50:05.282 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:50:05.283 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:50:05.283 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:50:05.283 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:50:05.283 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:50:05.283 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:50:05.283 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:50:05.283 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:50:05.283 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:50:05.283 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:50:05.283 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:50:05.283 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:50:05.283 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:50:05.283 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:50:05.284 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:50:05.284 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:05.284 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:50:05.284 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.284 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:50:05.284 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.284 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:50:05.284 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:50:05.284 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:50:05.284 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:50:05.284 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.284 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:50:05.284 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:50:05.285 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:05.285 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.285 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.285 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:50:05.285 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:50:05.285 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:05.285 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:05.285 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:05.285 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.285 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:50:05.285 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:50:05.285 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:50:05.286 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:50:05.286 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:50:05.286 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.286 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.286 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:50:05.286 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:50:05.286 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:50:05.286 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:50:05.286 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.286 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:50:05.286 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:50:05.286 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:05.286 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.287 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.287 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:50:05.287 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:50:05.287 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:05.287 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:05.287 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:05.287 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.287 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:50:05.287 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:50:05.287 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:50:05.287 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:50:05.287 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:50:05.287 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.287 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.288 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:50:05.288 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:50:05.288 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:50:05.288 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:50:05.288 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.288 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:50:05.288 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:50:05.288 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:05.288 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.288 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.288 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:50:05.288 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:50:05.289 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:05.289 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:05.289 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:05.289 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.289 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:50:05.289 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:50:05.289 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:50:05.289 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:50:05.289 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:50:05.289 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:05.289 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:05.289 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.289 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:50:05.289 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:50:05.290 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.290 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:50:05.290 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:50:05.290 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:50:05.290 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:05.290 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:05.290 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:50:05.290 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.290 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:50:05.290 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:50:05.290 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:50:05.290 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:05.290 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:05.291 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:50:05.291 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.291 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:50:05.291 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:50:05.291 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:50:05.291 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:05.291 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:05.291 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.291 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:50:05.291 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:50:05.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.265+0000 7f721e4b8640 1 -- 172.21.15.12:0/3683986363 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f720403e210 msgr2=0x7f72040406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:05.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.265+0000 7f721e4b8640 1 --2- 172.21.15.12:0/3683986363 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f720403e210 0x7f72040406d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7210005c00 tx=0x7f7210009040 comp rx=0 tx=0).stop 2024-09-11T01:50:05.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.265+0000 7f721e4b8640 10 monclient: shutdown 2024-09-11T01:50:05.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.265+0000 7f721e4b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:50:05.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.265+0000 7f721e4b8640 1 -- 172.21.15.12:0/3683986363 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7218138230 msgr2=0x7f7218136fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:05.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.265+0000 7f721e4b8640 1 --2- 172.21.15.12:0/3683986363 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7218138230 0x7f7218136fd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f720c009720 tx=0x7f720c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:50:05.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.265+0000 7f721ca55640 1 -- 172.21.15.12:0/3683986363 reap_dead start 2024-09-11T01:50:05.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.265+0000 7f721ca55640 1 -- 172.21.15.12:0/3683986363 reap_dead start 2024-09-11T01:50:05.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.265+0000 7f721e4b8640 1 -- 172.21.15.12:0/3683986363 shutdown_connections 2024-09-11T01:50:05.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.265+0000 7f721e4b8640 1 -- 172.21.15.12:0/3683986363 >> 172.21.15.12:0/3683986363 conn(0x7f72180a7020 msgr2=0x7f721813d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:50:05.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.265+0000 7f721e4b8640 1 -- 172.21.15.12:0/3683986363 shutdown_connections 2024-09-11T01:50:05.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.265+0000 7f721e4b8640 1 -- 172.21.15.12:0/3683986363 wait complete. 2024-09-11T01:50:05.294 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:50:05.294 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:50:05.294 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:50:05.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.394+0000 7f3f51d3d640 10 monclient: get_monmap_and_config 2024-09-11T01:50:05.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.394+0000 7f3f51d3d640 10 monclient: build_initial_monmap 2024-09-11T01:50:05.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.394+0000 7f3f51d3d640 10 monclient: monmap: 2024-09-11T01:50:05.631 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:50:05.631 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:05.631 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:50:05.395804+0000 2024-09-11T01:50:05.632 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:50:05.395804+0000 2024-09-11T01:50:05.632 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:50:05.632 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:05.632 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:50:05.632 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:50:05.632 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:50:05.632 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:50:05.632 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:50:05.632 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:05.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.394+0000 7f3f51d3d640 1 Processor -- start 2024-09-11T01:50:05.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.394+0000 7f3f51d3d640 1 -- start start 2024-09-11T01:50:05.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.394+0000 7f3f51d3d640 10 monclient: init 2024-09-11T01:50:05.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f51d3d640 10 monclient: _reopen_session rank -1 2024-09-11T01:50:05.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f51d3d640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-11T01:50:05.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f51d3d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c137e50 0x7f3f4c09d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:05.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f51d3d640 10 monclient(hunting): picked mon.noname-b con 0x7f3f4c137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:50:05.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f51d3d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3f4c138230 addr v1:172.21.15.12:6789/0 2024-09-11T01:50:05.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f51d3d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3f4c13d380 0x7f3f4c13f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:05.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f51d3d640 10 monclient(hunting): picked mon.noname-a con 0x7f3f4c13d380 addr v2:172.21.15.12:3300/0 2024-09-11T01:50:05.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f51d3d640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f4c0a6580 con 0x7f3f4c138230 2024-09-11T01:50:05.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f51d3d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:05.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f51d3d640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f3f4c0a62d0 con 0x7f3f4c13d380 2024-09-11T01:50:05.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f51d3d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:05.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f51d3d640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f3f4c09bb40 con 0x7f3f4c137e50 2024-09-11T01:50:05.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f51d3d640 10 monclient(hunting): _renew_subs 2024-09-11T01:50:05.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f51d3d640 10 monclient(hunting): authenticate will time out at 2011.838497s 2024-09-11T01:50:05.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f4b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c137e50 0x7f3f4c09d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:05.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f4bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3f4c13d380 0x7f3f4c13f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:05.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f4bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3f4c13d380 0x7f3f4c13f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40148/0 (socket says 172.21.15.12:40148) 2024-09-11T01:50:05.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f4b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c137e50 0x7f3f4c09d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45378/0 (socket says 172.21.15.12:45378) 2024-09-11T01:50:05.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f4bfff640 1 -- 172.21.15.12:0/2129298897 learned_addr learned my addr 172.21.15.12:0/2129298897 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:50:05.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f4bfff640 10 monclient(hunting): get_auth_request con 0x7f3f4c13d380 auth_method 0 2024-09-11T01:50:05.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:05.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:05.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:05.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f4b7fe640 10 monclient(hunting): get_auth_request con 0x7f3f4c137e50 auth_method 0 2024-09-11T01:50:05.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:05.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:05.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:05.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:05.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:05.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:05.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f4b7fe640 10 monclient(hunting): handle_auth_done global_id 224219 payload 274 2024-09-11T01:50:05.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f4b7fe640 1 -- 172.21.15.12:0/2129298897 >> v2:172.21.15.12:3300/0 conn(0x7f3f4c13d380 msgr2=0x7f3f4c13f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:50:05.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f4b7fe640 1 --2- 172.21.15.12:0/2129298897 >> v2:172.21.15.12:3300/0 conn(0x7f3f4c13d380 0x7f3f4c13f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:05.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f4b7fe640 1 -- 172.21.15.12:0/2129298897 >> v1:172.21.15.12:6789/0 conn(0x7f3f4c138230 legacy=0x7f3f4c138630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:50:05.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f4b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:50:05.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f4b7fe640 1 monclient: found mon.noname-b 2024-09-11T01:50:05.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:50:05.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f4b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:50:05.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.395+0000 7f3f4b7fe640 1 -- 172.21.15.12:0/2129298897 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f4c09c4a0 con 0x7f3f4c137e50 2024-09-11T01:50:05.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.398+0000 7f3f4b7fe640 1 -- 172.21.15.12:0/2129298897 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c137e50 msgr2=0x7f3f4c09d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:05.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.398+0000 7f3f4b7fe640 1 --2- 172.21.15.12:0/2129298897 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c137e50 0x7f3f4c09d160 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c0097a0 tx=0x7f3f3c029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9aed679044a14bbb server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:05.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.398+0000 7f3f4affd640 1 -- 172.21.15.12:0/2129298897 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c037070 con 0x7f3f4c137e50 2024-09-11T01:50:05.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.398+0000 7f3f4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:05.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.398+0000 7f3f4affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:50:05.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.398+0000 7f3f4affd640 10 monclient: dump: 2024-09-11T01:50:05.647 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:05.647 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:05.647 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.647 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.647 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:05.647 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:05.647 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:05.648 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:05.648 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:05.648 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:05.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.398+0000 7f3f4affd640 10 monclient: _finish_auth 0 2024-09-11T01:50:05.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.398+0000 7f3f4affd640 10 monclient: _check_auth_tickets 2024-09-11T01:50:05.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.398+0000 7f3f4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:50:05.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.398+0000 7f3f4affd640 1 -- 172.21.15.12:0/2129298897 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f3c032440 con 0x7f3f4c137e50 2024-09-11T01:50:05.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.398+0000 7f3f51d3d640 5 monclient: authenticate success, global_id 224219 2024-09-11T01:50:05.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.398+0000 7f3f51d3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:50:05.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.398+0000 7f3f4affd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:50:05.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.398+0000 7f3f51d3d640 10 monclient: get_monmap_and_config success 2024-09-11T01:50:05.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.398+0000 7f3f51d3d640 4 set_mon_vals no callback set 2024-09-11T01:50:05.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.398+0000 7f3f4affd640 1 -- 172.21.15.12:0/2129298897 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c032810 con 0x7f3f4c137e50 2024-09-11T01:50:05.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.398+0000 7f3f4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:05.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.398+0000 7f3f4affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:50:05.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.398+0000 7f3f4affd640 10 monclient: dump: 2024-09-11T01:50:05.649 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:05.649 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:05.649 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.649 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.649 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:05.649 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:05.649 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:05.649 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:05.649 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:05.650 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:05.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.399+0000 7f3f51d3d640 10 monclient: shutdown 2024-09-11T01:50:05.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.399+0000 7f3f51d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:50:05.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.399+0000 7f3f51d3d640 1 -- 172.21.15.12:0/2129298897 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c137e50 msgr2=0x7f3f4c09d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:05.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.399+0000 7f3f51d3d640 1 --2- 172.21.15.12:0/2129298897 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c137e50 0x7f3f4c09d160 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c0097a0 tx=0x7f3f3c029fd0 comp rx=0 tx=0).stop 2024-09-11T01:50:05.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.399+0000 7f3f51d3d640 1 -- 172.21.15.12:0/2129298897 shutdown_connections 2024-09-11T01:50:05.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.399+0000 7f3f51d3d640 1 --2- 172.21.15.12:0/2129298897 >> v2:172.21.15.12:3300/0 conn(0x7f3f4c13d380 0x7f3f4c13f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:05.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.399+0000 7f3f51d3d640 1 --2- 172.21.15.12:0/2129298897 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c137e50 0x7f3f4c09d160 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:05.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.399+0000 7f3f51d3d640 1 -- 172.21.15.12:0/2129298897 >> 172.21.15.12:0/2129298897 conn(0x7f3f4c0a7020 msgr2=0x7f3f4c0a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:50:05.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.399+0000 7f3f51d3d640 1 -- 172.21.15.12:0/2129298897 shutdown_connections 2024-09-11T01:50:05.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.399+0000 7f3f51d3d640 1 -- 172.21.15.12:0/2129298897 wait complete. 2024-09-11T01:50:05.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.400+0000 7f3f51d3d640 10 monclient: build_initial_monmap 2024-09-11T01:50:05.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.400+0000 7f3f51d3d640 10 monclient: monmap: 2024-09-11T01:50:05.651 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:50:05.651 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:50:05.651 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:50:05.651 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:50:05.651 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:50:05.651 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:05.651 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:50:05.652 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:50:05.652 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:50:05.652 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:05.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.400+0000 7f3f51d3d640 1 Processor -- start 2024-09-11T01:50:05.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.400+0000 7f3f51d3d640 1 -- start start 2024-09-11T01:50:05.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.400+0000 7f3f51d3d640 10 monclient: init 2024-09-11T01:50:05.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f51d3d640 10 monclient: _reopen_session rank -1 2024-09-11T01:50:05.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f51d3d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:50:05.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f51d3d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f4c137e50 0x7f3f4c1cd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:05.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f51d3d640 10 monclient(hunting): picked mon.noname-a con 0x7f3f4c137e50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:50:05.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f51d3d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c138230 0x7f3f4c1370a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:05.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f51d3d640 10 monclient(hunting): picked mon.noname-c con 0x7f3f4c138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:50:05.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f51d3d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f4c13d380 0x7f3f4c0a1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:05.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f51d3d640 10 monclient(hunting): picked mon.noname-b con 0x7f3f4c13d380 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:50:05.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f51d3d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:05.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f51d3d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c06f890 con 0x7f3f4c137e50 2024-09-11T01:50:05.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f51d3d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:05.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f51d3d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c06f710 con 0x7f3f4c13d380 2024-09-11T01:50:05.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f51d3d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:05.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f51d3d640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f3f4c13fcb0 con 0x7f3f4c138230 2024-09-11T01:50:05.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f51d3d640 10 monclient(hunting): _renew_subs 2024-09-11T01:50:05.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f51d3d640 10 monclient(hunting): authenticate will time out at 2011.844210s 2024-09-11T01:50:05.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f4b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f4c137e50 0x7f3f4c1cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:05.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f4b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f4c137e50 0x7f3f4c1cd190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40150/0 (socket says 172.21.15.12:40150) 2024-09-11T01:50:05.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f43fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c138230 0x7f3f4c1370a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:05.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f4b7fe640 1 -- 172.21.15.12:0/3265737939 learned_addr learned my addr 172.21.15.12:0/3265737939 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:50:05.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f4b7fe640 10 monclient(hunting): get_auth_request con 0x7f3f4c137e50 auth_method 0 2024-09-11T01:50:05.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:05.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:05.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:05.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f43fff640 10 monclient(hunting): get_auth_request con 0x7f3f4c138230 auth_method 0 2024-09-11T01:50:05.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:05.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f43fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:05.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:05.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:05.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:05.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:05.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:05.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:05.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:05.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f43fff640 10 monclient(hunting): handle_auth_done global_id 224222 payload 824 2024-09-11T01:50:05.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f4bfff640 1 --2- 172.21.15.12:0/3265737939 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f4c13d380 0x7f3f4c0a1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:05.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f43fff640 1 -- 172.21.15.12:0/3265737939 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f4c13d380 msgr2=0x7f3f4c0a1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:05.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f43fff640 1 --2- 172.21.15.12:0/3265737939 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f4c13d380 0x7f3f4c0a1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:05.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f43fff640 1 -- 172.21.15.12:0/3265737939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f4c137e50 msgr2=0x7f3f4c1cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:05.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f43fff640 1 --2- 172.21.15.12:0/3265737939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f4c137e50 0x7f3f4c1cd190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:05.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f43fff640 10 monclient: _finish_hunting 0 2024-09-11T01:50:05.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f43fff640 1 monclient: found mon.noname-c 2024-09-11T01:50:05.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:50:05.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f43fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:50:05.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f43fff640 1 -- 172.21.15.12:0/3265737939 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f4c121770 con 0x7f3f4c138230 2024-09-11T01:50:05.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f4b7fe640 1 --2- 172.21.15.12:0/3265737939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3f4c137e50 0x7f3f4c1cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:50:05.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f4bfff640 10 monclient: get_auth_request con 0x7f3f4c13d380 auth_method 0 2024-09-11T01:50:05.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.401+0000 7f3f4bfff640 1 --2- 172.21.15.12:0/3265737939 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f4c13d380 0x7f3f4c0a1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:50:05.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.403+0000 7f3f43fff640 1 -- 172.21.15.12:0/3265737939 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c138230 msgr2=0x7f3f4c1370a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:05.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.403+0000 7f3f43fff640 1 --2- 172.21.15.12:0/3265737939 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c138230 0x7f3f4c1370a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3f3400c440 tx=0x7f3f3400c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:05.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.404+0000 7f3f48ff9640 1 -- 172.21.15.12:0/3265737939 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f340119a0 con 0x7f3f4c138230 2024-09-11T01:50:05.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.404+0000 7f3f48ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:05.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.404+0000 7f3f48ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:50:05.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.404+0000 7f3f48ff9640 10 monclient: dump: 2024-09-11T01:50:05.656 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:05.656 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:05.656 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.656 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.656 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:05.657 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:05.657 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:05.657 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:05.657 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:05.657 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:05.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.404+0000 7f3f48ff9640 10 monclient: _finish_auth 0 2024-09-11T01:50:05.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.404+0000 7f3f48ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:50:05.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.404+0000 7f3f48ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:50:05.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.404+0000 7f3f51d3d640 5 monclient: authenticate success, global_id 224222 2024-09-11T01:50:05.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.404+0000 7f3f48ff9640 1 -- 172.21.15.12:0/3265737939 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f34017070 con 0x7f3f4c138230 2024-09-11T01:50:05.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.404+0000 7f3f48ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:50:05.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.404+0000 7f3f48ff9640 1 -- 172.21.15.12:0/3265737939 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f34011d80 con 0x7f3f4c138230 2024-09-11T01:50:05.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.404+0000 7f3f51d3d640 10 monclient: _renew_subs 2024-09-11T01:50:05.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.404+0000 7f3f51d3d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:50:05.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.404+0000 7f3f51d3d640 1 -- 172.21.15.12:0/3265737939 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f4c121a50 con 0x7f3f4c138230 2024-09-11T01:50:05.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.404+0000 7f3f4a7fc640 4 set_mon_vals no callback set 2024-09-11T01:50:05.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.404+0000 7f3f48ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:05.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.404+0000 7f3f48ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:50:05.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.404+0000 7f3f48ff9640 10 monclient: dump: 2024-09-11T01:50:05.658 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:05.658 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:05.658 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.658 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.658 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:05.658 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:05.658 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:05.659 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:05.659 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:05.659 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:05.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.404+0000 7f3f51d3d640 10 monclient: _renew_subs 2024-09-11T01:50:05.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.404+0000 7f3f51d3d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:50:05.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.404+0000 7f3f51d3d640 1 -- 172.21.15.12:0/3265737939 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f4c0a1710 con 0x7f3f4c138230 2024-09-11T01:50:05.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.404+0000 7f3f51d3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:05.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.404+0000 7f3f51d3d640 1 --2- 172.21.15.12:0/3265737939 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c0a22e0 0x7f3f4c0a26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:05.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.404+0000 7f3f51d3d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:05.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.404+0000 7f3f51d3d640 1 -- 172.21.15.12:0/3265737939 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f3f4c06a060 con 0x7f3f4c0a22e0 2024-09-11T01:50:05.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.405+0000 7f3f48ff9640 1 -- 172.21.15.12:0/3265737939 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f3f3401c460 con 0x7f3f4c138230 2024-09-11T01:50:05.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.405+0000 7f3f4b7fe640 1 --2- 172.21.15.12:0/3265737939 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c0a22e0 0x7f3f4c0a26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:05.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.405+0000 7f3f4b7fe640 10 monclient: get_auth_request con 0x7f3f4c0a22e0 auth_method 0 2024-09-11T01:50:05.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.405+0000 7f3f4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:05.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.405+0000 7f3f4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:05.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.405+0000 7f3f4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:05.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.405+0000 7f3f4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:05.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.405+0000 7f3f4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:05.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.405+0000 7f3f4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:05.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.405+0000 7f3f48ff9640 1 --2- 172.21.15.12:0/3265737939 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3f2c03e260 0x7f3f2c040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:05.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.405+0000 7f3f4b7fe640 10 monclient(hunting): handle_auth_done global_id 224225 payload 824 2024-09-11T01:50:05.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.405+0000 7f3f4b7fe640 1 -- 172.21.15.12:0/3265737939 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3f4c0a2c00 con 0x7f3f4c0a22e0 2024-09-11T01:50:05.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.405+0000 7f3f48ff9640 1 -- 172.21.15.12:0/3265737939 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(707..707 src has 507..707) ==== 4787+0+0 (secure 0 0 0) 0x7f3f34052ae0 con 0x7f3f4c138230 2024-09-11T01:50:05.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.405+0000 7f3f4bfff640 1 --2- 172.21.15.12:0/3265737939 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3f2c03e260 0x7f3f2c040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:05.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.405+0000 7f3f4bfff640 10 monclient: get_auth_request con 0x7f3f2c03e260 auth_method 0 2024-09-11T01:50:05.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.407+0000 7f3f4b7fe640 1 -- 172.21.15.12:0/3265737939 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c0a22e0 msgr2=0x7f3f4c0a26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:05.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.407+0000 7f3f4b7fe640 1 --2- 172.21.15.12:0/3265737939 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c0a22e0 0x7f3f4c0a26c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c0097a0 tx=0x7f3f3c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:05.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.407+0000 7f3f48ff9640 1 -- 172.21.15.12:0/3265737939 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c043070 con 0x7f3f4c0a22e0 2024-09-11T01:50:05.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.407+0000 7f3f48ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:05.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.407+0000 7f3f48ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:50:05.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.407+0000 7f3f48ff9640 10 monclient: dump: 2024-09-11T01:50:05.661 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:05.661 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:05.661 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.661 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.661 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:05.661 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:05.661 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:05.661 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:05.662 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.407+0000 7f3f48ff9640 1 -- 172.21.15.12:0/3265737939 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3f3c038040 con 0x7f3f4c0a22e0 2024-09-11T01:50:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.408+0000 7f3f48ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.408+0000 7f3f48ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:50:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.408+0000 7f3f48ff9640 1 -- 172.21.15.12:0/3265737939 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c0a22e0 msgr2=0x7f3f4c0a26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.408+0000 7f3f48ff9640 1 --2- 172.21.15.12:0/3265737939 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c0a22e0 0x7f3f4c0a26c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c0097a0 tx=0x7f3f3c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:50:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.408+0000 7f3f48ff9640 1 -- 172.21.15.12:0/3265737939 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c0a22e0 msgr2=0x7f3f4c0a26c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:50:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.408+0000 7f3f48ff9640 1 --2- 172.21.15.12:0/3265737939 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c0a22e0 0x7f3f4c0a26c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.408+0000 7f3f4bfff640 1 --2- 172.21.15.12:0/3265737939 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3f2c03e260 0x7f3f2c040720 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3f44009be0 tx=0x7f3f44009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.414+0000 7f3f51d3d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:50:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.414+0000 7f3f51d3d640 1 --2- 172.21.15.12:0/3265737939 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c09e420 0x7f3f4c09e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:05.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.414+0000 7f3f51d3d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.414+0000 7f3f51d3d640 1 -- 172.21.15.12:0/3265737939 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f3f4c09d160 con 0x7f3f4c09e420 2024-09-11T01:50:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.414+0000 7f3f4b7fe640 1 --2- 172.21.15.12:0/3265737939 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c09e420 0x7f3f4c09e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.414+0000 7f3f4b7fe640 10 monclient: get_auth_request con 0x7f3f4c09e420 auth_method 0 2024-09-11T01:50:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.414+0000 7f3f4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.414+0000 7f3f4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.414+0000 7f3f4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.414+0000 7f3f4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.414+0000 7f3f4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.414+0000 7f3f4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.414+0000 7f3f4b7fe640 10 monclient(hunting): handle_auth_done global_id 224228 payload 824 2024-09-11T01:50:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.414+0000 7f3f4b7fe640 1 -- 172.21.15.12:0/3265737939 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3f4c120d00 con 0x7f3f4c09e420 2024-09-11T01:50:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.416+0000 7f3f4b7fe640 1 -- 172.21.15.12:0/3265737939 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c09e420 msgr2=0x7f3f4c09e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.416+0000 7f3f4b7fe640 1 --2- 172.21.15.12:0/3265737939 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c09e420 0x7f3f4c09e800 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c042000 tx=0x7f3f3c008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:05.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.416+0000 7f3f48ff9640 1 -- 172.21.15.12:0/3265737939 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c037040 con 0x7f3f4c09e420 2024-09-11T01:50:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.416+0000 7f3f48ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.416+0000 7f3f48ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:50:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.416+0000 7f3f48ff9640 10 monclient: dump: 2024-09-11T01:50:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:05.664 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:05.664 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:05.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.417+0000 7f3f48ff9640 1 -- 172.21.15.12:0/3265737939 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7f3f3c003de0 con 0x7f3f4c09e420 2024-09-11T01:50:05.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.417+0000 7f3f48ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:50:05.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.417+0000 7f3f48ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:50:05.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.417+0000 7f3f48ff9640 1 -- 172.21.15.12:0/3265737939 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c09e420 msgr2=0x7f3f4c09e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:05.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.417+0000 7f3f48ff9640 1 --2- 172.21.15.12:0/3265737939 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c09e420 0x7f3f4c09e800 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c042000 tx=0x7f3f3c008030 comp rx=0 tx=0).stop 2024-09-11T01:50:05.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.417+0000 7f3f48ff9640 1 -- 172.21.15.12:0/3265737939 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c09e420 msgr2=0x7f3f4c09e800 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:50:05.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.417+0000 7f3f48ff9640 1 --2- 172.21.15.12:0/3265737939 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c09e420 0x7f3f4c09e800 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c042000 tx=0x7f3f3c008030 comp rx=0 tx=0).stop 2024-09-11T01:50:05.665 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:50:05.666 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:50:05.666 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:50:05.666 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:50:05.666 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 194, 2024-09-11T01:50:05.666 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 58189, 2024-09-11T01:50:05.666 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:50:05.666 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:50:05.666 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:50:05.666 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:50:05.666 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:05.666 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:50:05.666 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:50:05.666 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:50:05.667 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:50:05.667 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:50:05.667 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:50:05.667 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:50:05.667 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:50:05.667 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:50:05.667 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:50:05.667 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:50:05.667 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:50:05.667 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:50:05.667 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:50:05.667 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:50:05.667 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:05.668 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:50:05.668 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:50:05.668 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:50:05.668 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:50:05.668 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:50:05.668 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:50:05.668 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:50:05.668 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:50:05.668 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:50:05.668 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:50:05.668 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:50:05.668 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:50:05.669 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:50:05.669 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:05.669 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.669 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:50:05.669 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:50:05.669 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:50:05.669 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:50:05.669 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:50:05.669 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:50:05.669 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:50:05.669 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:50:05.669 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:50:05.670 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:50:05.670 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:50:05.670 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:50:05.670 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:50:05.670 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:50:05.670 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:50:05.670 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:50:05.670 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:50:05.670 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:50:05.670 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:50:05.670 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:50:05.670 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:50:05.670 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:50:05.670 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:50:05.671 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:50:05.671 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:50:05.671 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:50:05.671 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:50:05.671 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:50:05.671 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:05.671 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:50:05.671 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.671 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:50:05.671 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.671 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:50:05.671 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:50:05.671 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:50:05.672 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:50:05.672 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.672 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:50:05.672 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:50:05.672 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:05.672 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.672 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.672 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:50:05.672 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:50:05.672 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:05.672 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:05.672 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:05.673 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.673 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:50:05.673 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:50:05.673 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:50:05.673 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:50:05.673 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:50:05.673 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.673 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.673 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:50:05.673 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:50:05.673 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:50:05.673 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:50:05.673 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.673 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:50:05.674 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:50:05.674 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:05.674 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.674 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.674 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:50:05.674 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:50:05.674 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:05.674 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:05.674 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:05.674 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.674 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:50:05.674 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:50:05.674 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:50:05.675 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:50:05.675 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:50:05.675 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.675 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.675 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:50:05.675 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:50:05.675 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:50:05.675 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:50:05.675 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.675 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:50:05.675 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:50:05.675 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:05.675 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.675 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.676 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:50:05.676 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:50:05.676 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:05.676 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:05.676 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:05.676 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.676 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:50:05.676 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:50:05.676 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:50:05.676 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:50:05.676 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:50:05.676 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:05.676 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:05.677 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.677 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:50:05.677 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:50:05.677 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.677 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:50:05.677 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:50:05.677 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:50:05.677 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:05.677 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:05.677 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:50:05.677 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.677 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:50:05.677 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:50:05.677 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:50:05.678 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:05.678 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:05.678 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:50:05.678 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.678 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:50:05.678 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:50:05.678 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:50:05.678 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:05.678 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:05.678 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.678 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:50:05.678 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:50:05.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.417+0000 7f3f51d3d640 1 -- 172.21.15.12:0/3265737939 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3f2c03e260 msgr2=0x7f3f2c040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:05.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.417+0000 7f3f51d3d640 1 --2- 172.21.15.12:0/3265737939 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3f2c03e260 0x7f3f2c040720 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3f44009be0 tx=0x7f3f44009040 comp rx=0 tx=0).stop 2024-09-11T01:50:05.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.417+0000 7f3f51d3d640 10 monclient: shutdown 2024-09-11T01:50:05.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.417+0000 7f3f51d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:50:05.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.417+0000 7f3f51d3d640 1 -- 172.21.15.12:0/3265737939 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c138230 msgr2=0x7f3f4c1370a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:05.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.417+0000 7f3f51d3d640 1 --2- 172.21.15.12:0/3265737939 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3f4c138230 0x7f3f4c1370a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3f3400c440 tx=0x7f3f3400c900 comp rx=0 tx=0).stop 2024-09-11T01:50:05.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.417+0000 7f3f4bfff640 1 -- 172.21.15.12:0/3265737939 reap_dead start 2024-09-11T01:50:05.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.417+0000 7f3f4bfff640 1 -- 172.21.15.12:0/3265737939 reap_dead start 2024-09-11T01:50:05.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.417+0000 7f3f51d3d640 1 -- 172.21.15.12:0/3265737939 shutdown_connections 2024-09-11T01:50:05.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.417+0000 7f3f51d3d640 1 -- 172.21.15.12:0/3265737939 >> 172.21.15.12:0/3265737939 conn(0x7f3f4c0a7020 msgr2=0x7f3f4c13c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:50:05.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.417+0000 7f3f51d3d640 1 -- 172.21.15.12:0/3265737939 shutdown_connections 2024-09-11T01:50:05.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.417+0000 7f3f51d3d640 1 -- 172.21.15.12:0/3265737939 wait complete. 2024-09-11T01:50:05.681 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:50:05.681 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:50:05.681 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:50:05.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.784+0000 7f68e8bfb640 10 monclient: get_monmap_and_config 2024-09-11T01:50:05.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.784+0000 7f68e8bfb640 10 monclient: build_initial_monmap 2024-09-11T01:50:05.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.784+0000 7f68e8bfb640 10 monclient: monmap: 2024-09-11T01:50:05.787 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:50:05.787 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:05.787 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:50:05.786510+0000 2024-09-11T01:50:05.787 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:50:05.786510+0000 2024-09-11T01:50:05.787 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:50:05.788 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:05.788 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:50:05.788 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:50:05.788 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:50:05.788 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:50:05.788 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:50:05.788 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:05.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.785+0000 7f68e8bfb640 1 Processor -- start 2024-09-11T01:50:05.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.785+0000 7f68e8bfb640 1 -- start start 2024-09-11T01:50:05.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.785+0000 7f68e8bfb640 10 monclient: init 2024-09-11T01:50:05.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.785+0000 7f68e8bfb640 10 monclient: _reopen_session rank -1 2024-09-11T01:50:05.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.785+0000 7f68e8bfb640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-11T01:50:05.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.785+0000 7f68e8bfb640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f68e40a1380 0x7f68e40a1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:05.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.785+0000 7f68e8bfb640 10 monclient(hunting): picked mon.noname-a con 0x7f68e40a1380 addr v2:172.21.15.12:3300/0 2024-09-11T01:50:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.785+0000 7f68e8bfb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f68e40a0370 addr v1:172.21.15.12:6789/0 2024-09-11T01:50:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.785+0000 7f68e8bfb640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68e40a0c90 0x7f68e41361f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.785+0000 7f68e8bfb640 10 monclient(hunting): picked mon.noname-b con 0x7f68e40a0c90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:50:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.785+0000 7f68e8bfb640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68e40a1ca0 con 0x7f68e40a0370 2024-09-11T01:50:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.785+0000 7f68e8bfb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.785+0000 7f68e8bfb640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f68e40a21c0 con 0x7f68e40a1380 2024-09-11T01:50:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.785+0000 7f68e8bfb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.785+0000 7f68e8bfb640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f68e40a24a0 con 0x7f68e40a0c90 2024-09-11T01:50:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.785+0000 7f68e8bfb640 10 monclient(hunting): _renew_subs 2024-09-11T01:50:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.785+0000 7f68e8bfb640 10 monclient(hunting): authenticate will time out at 2012.228949s 2024-09-11T01:50:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.786+0000 7f68e2d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68e40a0c90 0x7f68e41361f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.786+0000 7f68e259c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f68e40a1380 0x7f68e40a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:05.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.786+0000 7f68e2d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68e40a0c90 0x7f68e41361f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:45434/0 (socket says 172.21.15.12:45434) 2024-09-11T01:50:05.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.786+0000 7f68e2d9d640 1 -- 172.21.15.12:0/2782250350 learned_addr learned my addr 172.21.15.12:0/2782250350 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:50:05.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.786+0000 7f68e259c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f68e40a1380 0x7f68e40a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40152/0 (socket says 172.21.15.12:40152) 2024-09-11T01:50:05.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.786+0000 7f68e2d9d640 10 monclient(hunting): get_auth_request con 0x7f68e40a0c90 auth_method 0 2024-09-11T01:50:05.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.786+0000 7f68e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:05.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.786+0000 7f68e2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:05.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.786+0000 7f68e2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:05.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.786+0000 7f68e259c640 10 monclient(hunting): get_auth_request con 0x7f68e40a1380 auth_method 0 2024-09-11T01:50:05.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.786+0000 7f68e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:05.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.786+0000 7f68e259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:05.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.786+0000 7f68e259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:05.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.786+0000 7f68e2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:05.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.786+0000 7f68e2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:05.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.786+0000 7f68e2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:05.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.786+0000 7f68e2d9d640 10 monclient(hunting): handle_auth_done global_id 224231 payload 274 2024-09-11T01:50:05.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.786+0000 7f68e2d9d640 1 -- 172.21.15.12:0/2782250350 >> v2:172.21.15.12:3300/0 conn(0x7f68e40a1380 msgr2=0x7f68e40a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:50:05.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.786+0000 7f68e2d9d640 1 --2- 172.21.15.12:0/2782250350 >> v2:172.21.15.12:3300/0 conn(0x7f68e40a1380 0x7f68e40a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:05.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.786+0000 7f68e2d9d640 1 -- 172.21.15.12:0/2782250350 >> v1:172.21.15.12:6789/0 conn(0x7f68e40a0370 legacy=0x7f68e4135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:50:05.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.786+0000 7f68e2d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:50:05.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.786+0000 7f68e2d9d640 1 monclient: found mon.noname-b 2024-09-11T01:50:05.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.786+0000 7f68e2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:50:05.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.786+0000 7f68e2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:50:05.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.786+0000 7f68e2d9d640 1 -- 172.21.15.12:0/2782250350 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68e40a2780 con 0x7f68e40a0c90 2024-09-11T01:50:05.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e2d9d640 1 -- 172.21.15.12:0/2782250350 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68e40a0c90 msgr2=0x7f68e41361f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:05.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e2d9d640 1 --2- 172.21.15.12:0/2782250350 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68e40a0c90 0x7f68e41361f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f68dc009e50 tx=0x7f68dc02a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3268195fd73fc50 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:05.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e1d9b640 1 -- 172.21.15.12:0/2782250350 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68dc036070 con 0x7f68e40a0c90 2024-09-11T01:50:05.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:05.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:50:05.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e1d9b640 10 monclient: dump: 2024-09-11T01:50:05.800 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:05.800 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:05.800 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.800 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.800 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:05.800 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:05.800 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:05.800 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:05.800 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:05.800 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:05.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e1d9b640 10 monclient: _finish_auth 0 2024-09-11T01:50:05.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e1d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:50:05.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:50:05.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e1d9b640 1 -- 172.21.15.12:0/2782250350 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68dc007800 con 0x7f68e40a0c90 2024-09-11T01:50:05.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e8bfb640 5 monclient: authenticate success, global_id 224231 2024-09-11T01:50:05.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e8bfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:50:05.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e1d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:50:05.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e8bfb640 10 monclient: get_monmap_and_config success 2024-09-11T01:50:05.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e1d9b640 1 -- 172.21.15.12:0/2782250350 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68dc007bd0 con 0x7f68e40a0c90 2024-09-11T01:50:05.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:05.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e8bfb640 4 set_mon_vals no callback set 2024-09-11T01:50:05.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:50:05.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e1d9b640 10 monclient: dump: 2024-09-11T01:50:05.801 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:05.801 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:05.802 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.802 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.802 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:05.802 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:05.802 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:05.802 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:05.802 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:05.802 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:05.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e8bfb640 10 monclient: shutdown 2024-09-11T01:50:05.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e8bfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:50:05.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e8bfb640 1 -- 172.21.15.12:0/2782250350 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68e40a0c90 msgr2=0x7f68e41361f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:05.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e8bfb640 1 --2- 172.21.15.12:0/2782250350 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68e40a0c90 0x7f68e41361f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f68dc009e50 tx=0x7f68dc02a560 comp rx=0 tx=0).stop 2024-09-11T01:50:05.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e8bfb640 1 -- 172.21.15.12:0/2782250350 shutdown_connections 2024-09-11T01:50:05.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e8bfb640 1 --2- 172.21.15.12:0/2782250350 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68e40a0c90 0x7f68e41361f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:05.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e8bfb640 1 --2- 172.21.15.12:0/2782250350 >> v2:172.21.15.12:3300/0 conn(0x7f68e40a1380 0x7f68e40a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:05.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e8bfb640 1 -- 172.21.15.12:0/2782250350 >> 172.21.15.12:0/2782250350 conn(0x7f68e40a2f70 msgr2=0x7f68e40a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:50:05.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e8bfb640 1 -- 172.21.15.12:0/2782250350 shutdown_connections 2024-09-11T01:50:05.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.789+0000 7f68e8bfb640 1 -- 172.21.15.12:0/2782250350 wait complete. 2024-09-11T01:50:05.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.790+0000 7f68e8bfb640 10 monclient: build_initial_monmap 2024-09-11T01:50:05.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.790+0000 7f68e8bfb640 10 monclient: monmap: 2024-09-11T01:50:05.804 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:50:05.804 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:50:05.804 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:50:05.804 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:50:05.804 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:50:05.804 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:05.804 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:50:05.804 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:50:05.804 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:50:05.804 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:05.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e8bfb640 1 Processor -- start 2024-09-11T01:50:05.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e8bfb640 1 -- start start 2024-09-11T01:50:05.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e8bfb640 10 monclient: init 2024-09-11T01:50:05.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e8bfb640 10 monclient: _reopen_session rank -1 2024-09-11T01:50:05.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e8bfb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:50:05.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e8bfb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68e40a0370 0x7f68e41c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:05.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e8bfb640 10 monclient(hunting): picked mon.noname-a con 0x7f68e40a0370 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:50:05.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e8bfb640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68e40a0c90 0x7f68e4135c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:05.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e8bfb640 10 monclient(hunting): picked mon.noname-b con 0x7f68e40a0c90 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:50:05.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e8bfb640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68e40a1380 0x7f68e41215e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:05.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e8bfb640 10 monclient(hunting): picked mon.noname-c con 0x7f68e40a1380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:50:05.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e8bfb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:05.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e8bfb640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f68e409ce20 con 0x7f68e40a0370 2024-09-11T01:50:05.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e8bfb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:05.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e8bfb640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f68e409cca0 con 0x7f68e40a0c90 2024-09-11T01:50:05.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e8bfb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e8bfb640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f68e4136730 con 0x7f68e40a1380 2024-09-11T01:50:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e8bfb640 10 monclient(hunting): _renew_subs 2024-09-11T01:50:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e8bfb640 10 monclient(hunting): authenticate will time out at 2012.234587s 2024-09-11T01:50:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68e40a0370 0x7f68e41c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e2d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68e40a1380 0x7f68e41215e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68e40a0370 0x7f68e41c8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40168/0 (socket says 172.21.15.12:40168) 2024-09-11T01:50:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e259c640 1 -- 172.21.15.12:0/2625729250 learned_addr learned my addr 172.21.15.12:0/2625729250 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:50:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e259c640 10 monclient(hunting): get_auth_request con 0x7f68e40a0370 auth_method 0 2024-09-11T01:50:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e2d9d640 10 monclient(hunting): get_auth_request con 0x7f68e40a1380 auth_method 0 2024-09-11T01:50:05.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:05.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:05.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:05.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:05.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:05.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:05.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:05.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:05.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:05.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e259c640 10 monclient(hunting): handle_auth_done global_id 214350 payload 824 2024-09-11T01:50:05.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68d9d9b640 1 --2- 172.21.15.12:0/2625729250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68e40a0c90 0x7f68e4135c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:05.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e259c640 1 -- 172.21.15.12:0/2625729250 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68e40a1380 msgr2=0x7f68e41215e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:05.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e259c640 1 --2- 172.21.15.12:0/2625729250 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68e40a1380 0x7f68e41215e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:05.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e259c640 1 -- 172.21.15.12:0/2625729250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68e40a0c90 msgr2=0x7f68e4135c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:05.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e259c640 1 --2- 172.21.15.12:0/2625729250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68e40a0c90 0x7f68e4135c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:05.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e259c640 10 monclient: _finish_hunting 0 2024-09-11T01:50:05.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e259c640 1 monclient: found mon.noname-a 2024-09-11T01:50:05.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:50:05.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:50:05.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e259c640 1 -- 172.21.15.12:0/2625729250 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68e41c9ab0 con 0x7f68e40a0370 2024-09-11T01:50:05.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68d9d9b640 10 monclient: get_auth_request con 0x7f68e40a0c90 auth_method 0 2024-09-11T01:50:05.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.791+0000 7f68e2d9d640 1 --2- 172.21.15.12:0/2625729250 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68e40a1380 0x7f68e41215e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:50:05.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.792+0000 7f68d9d9b640 1 --2- 172.21.15.12:0/2625729250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68e40a0c90 0x7f68e4135c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:50:05.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.794+0000 7f68e259c640 1 -- 172.21.15.12:0/2625729250 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68e40a0370 msgr2=0x7f68e41c8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:05.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.794+0000 7f68e259c640 1 --2- 172.21.15.12:0/2625729250 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68e40a0370 0x7f68e41c8dc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f68d400d830 tx=0x7f68d400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:05.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.794+0000 7f68db7fe640 1 -- 172.21.15.12:0/2625729250 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d400b870 con 0x7f68e40a0370 2024-09-11T01:50:05.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.794+0000 7f68db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:05.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.794+0000 7f68db7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:50:05.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.794+0000 7f68db7fe640 10 monclient: dump: 2024-09-11T01:50:05.809 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:05.809 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:05.809 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.809 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.809 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:05.809 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:05.809 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:05.809 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:05.809 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:05.809 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:05.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.794+0000 7f68db7fe640 10 monclient: _finish_auth 0 2024-09-11T01:50:05.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.794+0000 7f68db7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:50:05.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.794+0000 7f68db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:50:05.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.794+0000 7f68db7fe640 1 -- 172.21.15.12:0/2625729250 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68d4019070 con 0x7f68e40a0370 2024-09-11T01:50:05.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.794+0000 7f68e8bfb640 5 monclient: authenticate success, global_id 214350 2024-09-11T01:50:05.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.794+0000 7f68e8bfb640 10 monclient: _renew_subs 2024-09-11T01:50:05.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.794+0000 7f68e8bfb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:50:05.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.794+0000 7f68e8bfb640 1 -- 172.21.15.12:0/2625729250 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68e409d480 con 0x7f68e40a0370 2024-09-11T01:50:05.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.794+0000 7f68db7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:50:05.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.794+0000 7f68db7fe640 1 -- 172.21.15.12:0/2625729250 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d400bc30 con 0x7f68e40a0370 2024-09-11T01:50:05.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.794+0000 7f68db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:05.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.794+0000 7f68db7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:50:05.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.794+0000 7f68db7fe640 10 monclient: dump: 2024-09-11T01:50:05.810 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:05.810 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:05.810 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.810 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.811 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:05.811 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:05.811 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:05.811 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:05.811 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:05.811 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:05.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.794+0000 7f68e159a640 4 set_mon_vals no callback set 2024-09-11T01:50:05.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.794+0000 7f68e8bfb640 10 monclient: _renew_subs 2024-09-11T01:50:05.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.794+0000 7f68e8bfb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:50:05.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.794+0000 7f68e8bfb640 1 -- 172.21.15.12:0/2625729250 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68e41c9300 con 0x7f68e40a0370 2024-09-11T01:50:05.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.795+0000 7f68e8bfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:05.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.795+0000 7f68e8bfb640 1 --2- 172.21.15.12:0/2625729250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68e4136190 0x7f68e402d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:05.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.795+0000 7f68e8bfb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:05.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.795+0000 7f68e8bfb640 1 -- 172.21.15.12:0/2625729250 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f68e4136570 con 0x7f68e4136190 2024-09-11T01:50:05.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.795+0000 7f68db7fe640 1 -- 172.21.15.12:0/2625729250 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f68d4013070 con 0x7f68e40a0370 2024-09-11T01:50:05.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.795+0000 7f68db7fe640 1 --2- 172.21.15.12:0/2625729250 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f68c403e240 0x7f68c4040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:05.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.795+0000 7f68d9d9b640 1 --2- 172.21.15.12:0/2625729250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68e4136190 0x7f68e402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:05.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.795+0000 7f68db7fe640 1 -- 172.21.15.12:0/2625729250 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(707..707 src has 507..707) ==== 4787+0+0 (secure 0 0 0) 0x7f68d4052eb0 con 0x7f68e40a0370 2024-09-11T01:50:05.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.795+0000 7f68d9d9b640 10 monclient: get_auth_request con 0x7f68e4136190 auth_method 0 2024-09-11T01:50:05.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.795+0000 7f68d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:05.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.795+0000 7f68d9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:05.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.795+0000 7f68d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:05.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.795+0000 7f68d9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:05.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.795+0000 7f68d9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:05.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.795+0000 7f68d9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:05.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.795+0000 7f68e2d9d640 1 --2- 172.21.15.12:0/2625729250 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f68c403e240 0x7f68c4040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:05.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.795+0000 7f68e2d9d640 10 monclient: get_auth_request con 0x7f68c403e240 auth_method 0 2024-09-11T01:50:05.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.796+0000 7f68d9d9b640 10 monclient(hunting): handle_auth_done global_id 234115 payload 824 2024-09-11T01:50:05.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.796+0000 7f68d9d9b640 1 -- 172.21.15.12:0/2625729250 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f68e409d740 con 0x7f68e4136190 2024-09-11T01:50:05.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.798+0000 7f68d9d9b640 1 -- 172.21.15.12:0/2625729250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68e4136190 msgr2=0x7f68e402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:05.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.798+0000 7f68d9d9b640 1 --2- 172.21.15.12:0/2625729250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68e4136190 0x7f68e402d440 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f68e4072e80 tx=0x7f68cc00d950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:05.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.798+0000 7f68e2d9d640 1 --2- 172.21.15.12:0/2625729250 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f68c403e240 0x7f68c4040700 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f68e40a4140 tx=0x7f68dc009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:05.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.798+0000 7f68db7fe640 1 -- 172.21.15.12:0/2625729250 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68cc012a10 con 0x7f68e4136190 2024-09-11T01:50:05.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.798+0000 7f68db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:05.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.798+0000 7f68db7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:50:05.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.798+0000 7f68db7fe640 10 monclient: dump: 2024-09-11T01:50:05.816 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:05.816 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:05.816 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.816 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.816 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:05.816 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:05.816 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:05.816 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:05.817 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:05.817 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:05.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.798+0000 7f68db7fe640 1 -- 172.21.15.12:0/2625729250 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f68cc012a10 con 0x7f68e4136190 2024-09-11T01:50:05.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.798+0000 7f68db7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:05.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.798+0000 7f68db7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:50:05.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.798+0000 7f68db7fe640 1 -- 172.21.15.12:0/2625729250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68e4136190 msgr2=0x7f68e402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:05.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.798+0000 7f68db7fe640 1 --2- 172.21.15.12:0/2625729250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68e4136190 0x7f68e402d440 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f68e4072e80 tx=0x7f68cc00d950 comp rx=0 tx=0).stop 2024-09-11T01:50:05.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.798+0000 7f68db7fe640 1 -- 172.21.15.12:0/2625729250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68e4136190 msgr2=0x7f68e402d440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:50:05.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.798+0000 7f68db7fe640 1 --2- 172.21.15.12:0/2625729250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68e4136190 0x7f68e402d440 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:05.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.805+0000 7f68e8bfb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:50:05.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.805+0000 7f68e8bfb640 1 --2- 172.21.15.12:0/2625729250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68e402d980 0x7f68e402dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:05.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.805+0000 7f68e8bfb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:05.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.805+0000 7f68e8bfb640 1 -- 172.21.15.12:0/2625729250 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f68e40a1760 con 0x7f68e402d980 2024-09-11T01:50:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.805+0000 7f68d9d9b640 1 --2- 172.21.15.12:0/2625729250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68e402d980 0x7f68e402dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.805+0000 7f68d9d9b640 10 monclient: get_auth_request con 0x7f68e402d980 auth_method 0 2024-09-11T01:50:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.805+0000 7f68d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.805+0000 7f68d9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.805+0000 7f68d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.805+0000 7f68d9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.805+0000 7f68d9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.805+0000 7f68d9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.805+0000 7f68d9d9b640 10 monclient(hunting): handle_auth_done global_id 234118 payload 824 2024-09-11T01:50:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.805+0000 7f68d9d9b640 1 -- 172.21.15.12:0/2625729250 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f68e411dff0 con 0x7f68e402d980 2024-09-11T01:50:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.808+0000 7f68d9d9b640 1 -- 172.21.15.12:0/2625729250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68e402d980 msgr2=0x7f68e402dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.808+0000 7f68d9d9b640 1 --2- 172.21.15.12:0/2625729250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68e402d980 0x7f68e402dd90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f68e4123e80 tx=0x7f68cc003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:05.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.808+0000 7f68db7fe640 1 -- 172.21.15.12:0/2625729250 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68cc006df0 con 0x7f68e402d980 2024-09-11T01:50:05.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.808+0000 7f68db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:05.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.808+0000 7f68db7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:50:05.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.808+0000 7f68db7fe640 10 monclient: dump: 2024-09-11T01:50:05.819 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:05.819 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:05.819 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.819 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:05.819 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:05.819 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:05.819 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:05.819 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:05.819 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:05.819 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:05.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.808+0000 7f68db7fe640 1 -- 172.21.15.12:0/2625729250 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4707 (secure 0 0 0) 0x7f68cc006df0 con 0x7f68e402d980 2024-09-11T01:50:05.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.808+0000 7f68db7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:50:05.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.808+0000 7f68db7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:50:05.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.808+0000 7f68db7fe640 1 -- 172.21.15.12:0/2625729250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68e402d980 msgr2=0x7f68e402dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:05.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.808+0000 7f68db7fe640 1 --2- 172.21.15.12:0/2625729250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68e402d980 0x7f68e402dd90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f68e4123e80 tx=0x7f68cc003760 comp rx=0 tx=0).stop 2024-09-11T01:50:05.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.808+0000 7f68db7fe640 1 -- 172.21.15.12:0/2625729250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68e402d980 msgr2=0x7f68e402dd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:50:05.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.808+0000 7f68db7fe640 1 --2- 172.21.15.12:0/2625729250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68e402d980 0x7f68e402dd90 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f68e4123e80 tx=0x7f68cc003760 comp rx=0 tx=0).stop 2024-09-11T01:50:05.820 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:50:05.820 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:50:05.820 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:50:05.821 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:50:05.821 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 194, 2024-09-11T01:50:05.821 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 7701, 2024-09-11T01:50:05.821 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:50:05.821 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:50:05.821 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:50:05.821 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:50:05.821 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:05.821 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 4, 2024-09-11T01:50:05.821 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:50:05.821 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:50:05.821 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:50:05.822 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:50:05.822 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:50:05.822 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:50:05.822 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:50:05.822 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:50:05.822 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:50:05.822 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:50:05.822 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:50:05.822 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:50:05.822 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:50:05.822 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:50:05.822 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:05.822 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:50:05.822 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:50:05.823 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:50:05.823 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:50:05.823 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:50:05.823 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:50:05.823 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:50:05.823 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:50:05.823 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:50:05.823 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:50:05.823 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:50:05.823 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:50:05.823 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:50:05.823 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:05.823 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.823 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:50:05.824 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:50:05.824 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:50:05.824 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:50:05.824 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:50:05.824 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:50:05.824 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:50:05.824 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:50:05.824 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:50:05.824 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:50:05.824 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:50:05.824 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:50:05.824 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:50:05.824 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:50:05.825 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:50:05.825 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:50:05.825 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:50:05.825 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:50:05.825 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:50:05.825 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:50:05.825 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:50:05.825 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:50:05.825 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:50:05.825 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:50:05.825 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:50:05.825 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:50:05.825 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:50:05.826 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:50:05.826 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:05.826 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:50:05.826 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.826 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:50:05.826 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.826 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:50:05.826 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:50:05.826 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:50:05.826 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:50:05.826 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.826 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:50:05.826 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:50:05.826 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:05.826 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.827 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.827 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:50:05.827 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:50:05.827 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:05.827 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:05.827 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:05.827 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.827 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:50:05.827 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:50:05.827 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:50:05.827 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:50:05.827 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:50:05.827 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.828 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.828 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:50:05.828 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:50:05.828 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:50:05.828 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:50:05.828 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.828 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:50:05.828 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:50:05.828 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:05.828 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.828 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.828 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:50:05.828 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:50:05.828 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:05.829 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:05.829 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:05.829 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.829 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:50:05.829 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:50:05.829 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:50:05.829 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:50:05.829 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:50:05.829 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.829 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.829 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:50:05.829 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:50:05.830 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:50:05.830 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:50:05.830 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.830 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:50:05.830 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:50:05.830 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:05.830 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.830 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.830 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:50:05.830 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:50:05.830 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:05.830 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:05.830 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:05.830 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.831 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:50:05.831 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:50:05.831 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:50:05.831 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:50:05.831 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:50:05.831 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:05.831 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:05.831 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.831 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:50:05.831 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:50:05.831 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.831 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:50:05.831 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:50:05.832 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:50:05.832 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:05.832 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:05.832 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:50:05.832 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.832 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:50:05.832 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:50:05.832 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:50:05.832 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:05.832 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:05.832 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:50:05.832 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:05.832 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:50:05.833 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:50:05.833 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:50:05.833 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:05.833 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:05.833 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:05.833 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:50:05.833 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:50:05.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.809+0000 7f68e8bfb640 1 -- 172.21.15.12:0/2625729250 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f68c403e240 msgr2=0x7f68c4040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:05.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.809+0000 7f68e8bfb640 1 --2- 172.21.15.12:0/2625729250 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f68c403e240 0x7f68c4040700 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f68e40a4140 tx=0x7f68dc009000 comp rx=0 tx=0).stop 2024-09-11T01:50:05.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.809+0000 7f68e8bfb640 10 monclient: shutdown 2024-09-11T01:50:05.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.809+0000 7f68e8bfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:50:05.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.809+0000 7f68e8bfb640 1 -- 172.21.15.12:0/2625729250 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68e40a0370 msgr2=0x7f68e41c8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:05.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.809+0000 7f68e8bfb640 1 --2- 172.21.15.12:0/2625729250 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68e40a0370 0x7f68e41c8dc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f68d400d830 tx=0x7f68d400dcf0 comp rx=0 tx=0).stop 2024-09-11T01:50:05.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.809+0000 7f68e2d9d640 1 -- 172.21.15.12:0/2625729250 reap_dead start 2024-09-11T01:50:05.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.809+0000 7f68e2d9d640 1 -- 172.21.15.12:0/2625729250 reap_dead start 2024-09-11T01:50:05.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.809+0000 7f68e8bfb640 1 -- 172.21.15.12:0/2625729250 shutdown_connections 2024-09-11T01:50:05.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.809+0000 7f68e8bfb640 1 -- 172.21.15.12:0/2625729250 >> 172.21.15.12:0/2625729250 conn(0x7f68e40a2f70 msgr2=0x7f68e4071950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:50:05.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.809+0000 7f68e8bfb640 1 -- 172.21.15.12:0/2625729250 shutdown_connections 2024-09-11T01:50:05.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:05.809+0000 7f68e8bfb640 1 -- 172.21.15.12:0/2625729250 wait complete. 2024-09-11T01:50:05.835 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:50:05.836 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-11T01:50:05.836 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-11T01:50:05.836 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-11T01:50:05.836 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-11T01:50:05.836 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-11T01:50:05.836 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:50:05.848 INFO:tasks.ceph.mon.c:Stopped 2024-09-11T01:50:05.848 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-11T01:50:05.848 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-11T01:50:05.848 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T01:50:20.849 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-11T01:50:20.850 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-11T01:50:20.851 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:50:20.851 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:50:20.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.968+0000 7f314c573640 10 monclient: get_monmap_and_config 2024-09-11T01:50:20.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.968+0000 7f314c573640 10 monclient: build_initial_monmap 2024-09-11T01:50:20.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.968+0000 7f314c573640 10 monclient: monmap: 2024-09-11T01:50:20.977 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:50:20.977 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:20.977 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:50:20.970608+0000 2024-09-11T01:50:20.977 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:50:20.970608+0000 2024-09-11T01:50:20.978 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:50:20.978 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:20.978 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:50:20.978 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:50:20.978 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:50:20.978 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:50:20.978 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:50:20.979 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:20.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.969+0000 7f314c573640 1 Processor -- start 2024-09-11T01:50:20.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.969+0000 7f314c573640 1 -- start start 2024-09-11T01:50:20.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.969+0000 7f314c573640 10 monclient: init 2024-09-11T01:50:20.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314c573640 10 monclient: _reopen_session rank -1 2024-09-11T01:50:20.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314c573640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-11T01:50:20.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314c573640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f31440a2db0 0x7f314406cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:20.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314c573640 10 monclient(hunting): picked mon.noname-a con 0x7f31440a2db0 addr v2:172.21.15.12:3300/0 2024-09-11T01:50:20.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314c573640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f314409fbd0 0x7f314406d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:20.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314c573640 10 monclient(hunting): picked mon.noname-b con 0x7f314409fbd0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:50:20.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314c573640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f314406d5d0 0x7f314413e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:20.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314c573640 10 monclient(hunting): picked mon.noname-c con 0x7f314406d5d0 addr v2:172.21.15.71:3300/0 2024-09-11T01:50:20.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314c573640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:20.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314c573640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f31440a3850 con 0x7f31440a2db0 2024-09-11T01:50:20.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314c573640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:20.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314c573640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f31440a3190 con 0x7f314406d5d0 2024-09-11T01:50:20.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314c573640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:20.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314c573640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f314409eea0 con 0x7f314409fbd0 2024-09-11T01:50:20.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314c573640 10 monclient(hunting): _renew_subs 2024-09-11T01:50:20.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314c573640 10 monclient(hunting): authenticate will time out at 2027.413310s 2024-09-11T01:50:20.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f3149b0e640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f314409fbd0 msgr2=0x7f314406d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:50:20.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f3149b0e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f314409fbd0 0x7f314406d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:50:20.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314a30f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f31440a2db0 0x7f314406cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:20.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314a30f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f31440a2db0 0x7f314406cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42802/0 (socket says 172.21.15.12:42802) 2024-09-11T01:50:20.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314a30f640 1 -- 172.21.15.12:0/970996451 learned_addr learned my addr 172.21.15.12:0/970996451 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:50:20.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314a30f640 10 monclient(hunting): get_auth_request con 0x7f31440a2db0 auth_method 0 2024-09-11T01:50:20.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314a30f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:20.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314a30f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:20.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314a30f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:20.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314a30f640 1 -- 172.21.15.12:0/970996451 >> v2:172.21.15.12:3300/0 conn(0x7f31440a2db0 msgr2=0x7f314406cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:50:20.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314a30f640 1 -- 172.21.15.12:0/970996451 >> v2:172.21.15.12:3300/0 conn(0x7f31440a2db0 msgr2=0x7f314406cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:50:20.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314a30f640 1 --2- 172.21.15.12:0/970996451 >> v2:172.21.15.12:3300/0 conn(0x7f31440a2db0 0x7f314406cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:50:20.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314a30f640 1 --2- 172.21.15.12:0/970996451 >> v2:172.21.15.12:3300/0 conn(0x7f31440a2db0 0x7f314406cb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:50:20.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314ab10640 1 --2- 172.21.15.12:0/970996451 >> v2:172.21.15.71:3300/0 conn(0x7f314406d5d0 0x7f314413e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:20.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314ab10640 10 monclient(hunting): get_auth_request con 0x7f314406d5d0 auth_method 0 2024-09-11T01:50:20.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314ab10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:20.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314ab10640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:20.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314ab10640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:20.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314ab10640 1 -- 172.21.15.12:0/970996451 >> v2:172.21.15.71:3300/0 conn(0x7f314406d5d0 msgr2=0x7f314413e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:50:20.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314ab10640 1 -- 172.21.15.12:0/970996451 >> v2:172.21.15.71:3300/0 conn(0x7f314406d5d0 msgr2=0x7f314413e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:50:20.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314ab10640 1 --2- 172.21.15.12:0/970996451 >> v2:172.21.15.71:3300/0 conn(0x7f314406d5d0 0x7f314413e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:50:20.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:20.970+0000 7f314ab10640 1 --2- 172.21.15.12:0/970996451 >> v2:172.21.15.71:3300/0 conn(0x7f314406d5d0 0x7f314413e550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:50:21.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.171+0000 7f3149b0e640 1 -- 172.21.15.12:0/970996451 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f314409fbd0 msgr2=0x7f314406d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:50:21.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.171+0000 7f3149b0e640 1 --2- 172.21.15.12:0/970996451 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f314409fbd0 0x7f314406d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:50:21.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.171+0000 7f314a30f640 1 --2- 172.21.15.12:0/970996451 >> v2:172.21.15.12:3300/0 conn(0x7f31440a2db0 0x7f314406cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:21.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.171+0000 7f314a30f640 10 monclient(hunting): get_auth_request con 0x7f31440a2db0 auth_method 0 2024-09-11T01:50:21.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.171+0000 7f314a30f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:21.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.171+0000 7f314a30f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:21.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.171+0000 7f314a30f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:50:21.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.171+0000 7f314a30f640 1 -- 172.21.15.12:0/970996451 >> v2:172.21.15.12:3300/0 conn(0x7f31440a2db0 msgr2=0x7f314406cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:50:21.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.171+0000 7f314a30f640 1 -- 172.21.15.12:0/970996451 >> v2:172.21.15.12:3300/0 conn(0x7f31440a2db0 msgr2=0x7f314406cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:50:21.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.171+0000 7f314a30f640 1 --2- 172.21.15.12:0/970996451 >> v2:172.21.15.12:3300/0 conn(0x7f31440a2db0 0x7f314406cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:50:21.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.171+0000 7f314a30f640 1 --2- 172.21.15.12:0/970996451 >> v2:172.21.15.12:3300/0 conn(0x7f31440a2db0 0x7f314406cb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:50:21.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.171+0000 7f314ab10640 1 --2- 172.21.15.12:0/970996451 >> v2:172.21.15.71:3300/0 conn(0x7f314406d5d0 0x7f314413e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:21.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.171+0000 7f314ab10640 10 monclient(hunting): get_auth_request con 0x7f314406d5d0 auth_method 0 2024-09-11T01:50:21.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.171+0000 7f314ab10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:21.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.171+0000 7f314ab10640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:21.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.171+0000 7f314ab10640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:50:21.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.171+0000 7f314ab10640 1 -- 172.21.15.12:0/970996451 >> v2:172.21.15.71:3300/0 conn(0x7f314406d5d0 msgr2=0x7f314413e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:50:21.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.171+0000 7f314ab10640 1 -- 172.21.15.12:0/970996451 >> v2:172.21.15.71:3300/0 conn(0x7f314406d5d0 msgr2=0x7f314413e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:50:21.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.171+0000 7f314ab10640 1 --2- 172.21.15.12:0/970996451 >> v2:172.21.15.71:3300/0 conn(0x7f314406d5d0 0x7f314413e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:50:21.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.171+0000 7f314ab10640 1 --2- 172.21.15.12:0/970996451 >> v2:172.21.15.71:3300/0 conn(0x7f314406d5d0 0x7f314413e550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:50:21.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.572+0000 7f3149b0e640 1 -- 172.21.15.12:0/970996451 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f314409fbd0 msgr2=0x7f314406d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:50:21.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.572+0000 7f3149b0e640 1 --2- 172.21.15.12:0/970996451 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f314409fbd0 0x7f314406d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:50:21.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.572+0000 7f314a30f640 1 --2- 172.21.15.12:0/970996451 >> v2:172.21.15.12:3300/0 conn(0x7f31440a2db0 0x7f314406cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:21.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.572+0000 7f314a30f640 10 monclient(hunting): get_auth_request con 0x7f31440a2db0 auth_method 0 2024-09-11T01:50:21.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.572+0000 7f314a30f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:21.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.572+0000 7f314a30f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:21.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.572+0000 7f314a30f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:50:21.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.572+0000 7f314a30f640 1 -- 172.21.15.12:0/970996451 >> v2:172.21.15.12:3300/0 conn(0x7f31440a2db0 msgr2=0x7f314406cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:50:21.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.572+0000 7f314a30f640 1 -- 172.21.15.12:0/970996451 >> v2:172.21.15.12:3300/0 conn(0x7f31440a2db0 msgr2=0x7f314406cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:50:21.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.572+0000 7f314a30f640 1 --2- 172.21.15.12:0/970996451 >> v2:172.21.15.12:3300/0 conn(0x7f31440a2db0 0x7f314406cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:50:21.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.572+0000 7f314a30f640 1 --2- 172.21.15.12:0/970996451 >> v2:172.21.15.12:3300/0 conn(0x7f31440a2db0 0x7f314406cb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:50:21.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.572+0000 7f314ab10640 1 --2- 172.21.15.12:0/970996451 >> v2:172.21.15.71:3300/0 conn(0x7f314406d5d0 0x7f314413e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:21.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.572+0000 7f314ab10640 10 monclient(hunting): get_auth_request con 0x7f314406d5d0 auth_method 0 2024-09-11T01:50:21.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.572+0000 7f314ab10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:21.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.572+0000 7f314ab10640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:21.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.572+0000 7f314ab10640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:50:21.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.572+0000 7f314ab10640 1 -- 172.21.15.12:0/970996451 >> v2:172.21.15.71:3300/0 conn(0x7f314406d5d0 msgr2=0x7f314413e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:50:21.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.572+0000 7f314ab10640 1 -- 172.21.15.12:0/970996451 >> v2:172.21.15.71:3300/0 conn(0x7f314406d5d0 msgr2=0x7f314413e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:50:21.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.572+0000 7f314ab10640 1 --2- 172.21.15.12:0/970996451 >> v2:172.21.15.71:3300/0 conn(0x7f314406d5d0 0x7f314413e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:50:21.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:21.572+0000 7f314ab10640 1 --2- 172.21.15.12:0/970996451 >> v2:172.21.15.71:3300/0 conn(0x7f314406d5d0 0x7f314413e550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:50:22.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:22.373+0000 7f3149b0e640 1 -- 172.21.15.12:0/970996451 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f314409fbd0 msgr2=0x7f314406d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:50:22.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:22.373+0000 7f3149b0e640 1 --2- 172.21.15.12:0/970996451 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f314409fbd0 0x7f314406d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:50:22.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:22.373+0000 7f314ab10640 1 --2- 172.21.15.12:0/970996451 >> v2:172.21.15.71:3300/0 conn(0x7f314406d5d0 0x7f314413e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:22.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:22.373+0000 7f314a30f640 1 --2- 172.21.15.12:0/970996451 >> v2:172.21.15.12:3300/0 conn(0x7f31440a2db0 0x7f314406cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:22.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:22.373+0000 7f314a30f640 10 monclient(hunting): get_auth_request con 0x7f31440a2db0 auth_method 0 2024-09-11T01:50:22.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:22.373+0000 7f314a30f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:22.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:22.373+0000 7f314a30f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:22.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:22.373+0000 7f314a30f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:50:22.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:22.373+0000 7f314ab10640 10 monclient(hunting): get_auth_request con 0x7f314406d5d0 auth_method 0 2024-09-11T01:50:22.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:22.373+0000 7f314ab10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:22.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:22.373+0000 7f314ab10640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:22.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:22.373+0000 7f314ab10640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:50:22.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:22.374+0000 7f314a30f640 1 -- 172.21.15.12:0/970996451 >> v2:172.21.15.12:3300/0 conn(0x7f31440a2db0 msgr2=0x7f314406cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:50:22.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:22.374+0000 7f314a30f640 1 -- 172.21.15.12:0/970996451 >> v2:172.21.15.12:3300/0 conn(0x7f31440a2db0 msgr2=0x7f314406cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:50:22.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:22.374+0000 7f314a30f640 1 --2- 172.21.15.12:0/970996451 >> v2:172.21.15.12:3300/0 conn(0x7f31440a2db0 0x7f314406cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:50:22.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:22.374+0000 7f314a30f640 1 --2- 172.21.15.12:0/970996451 >> v2:172.21.15.12:3300/0 conn(0x7f31440a2db0 0x7f314406cb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:50:22.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:22.374+0000 7f314ab10640 1 -- 172.21.15.12:0/970996451 >> v2:172.21.15.71:3300/0 conn(0x7f314406d5d0 msgr2=0x7f314413e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:50:22.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:22.374+0000 7f314ab10640 1 -- 172.21.15.12:0/970996451 >> v2:172.21.15.71:3300/0 conn(0x7f314406d5d0 msgr2=0x7f314413e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:50:22.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:22.374+0000 7f314ab10640 1 --2- 172.21.15.12:0/970996451 >> v2:172.21.15.71:3300/0 conn(0x7f314406d5d0 0x7f314413e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:50:22.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:22.374+0000 7f314ab10640 1 --2- 172.21.15.12:0/970996451 >> v2:172.21.15.71:3300/0 conn(0x7f314406d5d0 0x7f314413e550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:50:23.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:23.971+0000 7f313bfff640 10 monclient(hunting): tick 2024-09-11T01:50:23.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:23.971+0000 7f313bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:50:23.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:23.971+0000 7f313bfff640 1 monclient(hunting): continuing hunt 2024-09-11T01:50:23.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:23.971+0000 7f313bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:50:23.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:23.971+0000 7f313bfff640 1 -- 172.21.15.12:0/970996451 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f314409fbd0 msgr2=0x7f314406d090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:50:23.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:23.971+0000 7f313bfff640 1 --2- 172.21.15.12:0/970996451 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f314409fbd0 0x7f314406d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:23.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:23.971+0000 7f313bfff640 1 -- 172.21.15.12:0/970996451 >> v2:172.21.15.71:3300/0 conn(0x7f314406d5d0 msgr2=0x7f314413e550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:50:23.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:23.971+0000 7f313bfff640 1 --2- 172.21.15.12:0/970996451 >> v2:172.21.15.71:3300/0 conn(0x7f314406d5d0 0x7f314413e550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:23.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:23.971+0000 7f313bfff640 1 -- 172.21.15.12:0/970996451 >> v2:172.21.15.12:3300/0 conn(0x7f31440a2db0 msgr2=0x7f314406cb50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:50:23.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:23.971+0000 7f313bfff640 1 --2- 172.21.15.12:0/970996451 >> v2:172.21.15.12:3300/0 conn(0x7f31440a2db0 0x7f314406cb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:23.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:23.971+0000 7f313bfff640 10 monclient: _add_conns ranks=[0,1] 2024-09-11T01:50:23.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:23.971+0000 7f313bfff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f31280036e0 addr v1:172.21.15.12:6789/0 2024-09-11T01:50:23.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:23.971+0000 7f313bfff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f31280073f0 addr v1:172.21.15.71:6789/0 2024-09-11T01:50:23.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:23.971+0000 7f313bfff640 1 -- 172.21.15.12:0/970996451 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f314409f180 con 0x7f31280036e0 2024-09-11T01:50:23.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:23.971+0000 7f313bfff640 1 -- 172.21.15.12:0/970996451 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f312800c130 con 0x7f31280073f0 2024-09-11T01:50:23.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:23.971+0000 7f313bfff640 10 monclient(hunting): _renew_subs 2024-09-11T01:50:23.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:23.972+0000 7f314a30f640 1 -- 172.21.15.12:0/970996451 >> v1:172.21.15.12:6789/0 conn(0x7f31280036e0 legacy=0x7f3128005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:23.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:23.973+0000 7f3149b0e640 1 -- 172.21.15.12:0/970996451 >> v1:172.21.15.71:6789/0 conn(0x7f31280073f0 legacy=0x7f31280098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:25.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 1 -- 172.21.15.12:0/970996451 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 449084893 0 0) 0x7f314409f180 con 0x7f31280036e0 2024-09-11T01:50:25.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 10 monclient(hunting): _negotiate 2024-09-11T01:50:25.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:25.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:25.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 10 monclient(hunting): my global_id is 214353 2024-09-11T01:50:25.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 1 -- 172.21.15.12:0/970996451 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3130003520 con 0x7f31280036e0 2024-09-11T01:50:25.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 1 -- 172.21.15.12:0/970996451 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1040617115 0 0) 0x7f3130003520 con 0x7f31280036e0 2024-09-11T01:50:25.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 1 -- 172.21.15.12:0/970996451 >> v1:172.21.15.71:6789/0 conn(0x7f31280073f0 legacy=0x7f31280098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:25.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 10 monclient: _finish_hunting 0 2024-09-11T01:50:25.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:50:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:50:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:50:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 1 -- 172.21.15.12:0/970996451 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f312800d3f0 con 0x7f31280036e0 2024-09-11T01:50:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 10 monclient: _finish_auth 0 2024-09-11T01:50:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 10 monclient: _check_auth_tickets 2024-09-11T01:50:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:50:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 1 -- 172.21.15.12:0/970996451 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f314000f040 con 0x7f31280036e0 2024-09-11T01:50:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314c573640 5 monclient: authenticate success, global_id 214353 2024-09-11T01:50:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314c573640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:50:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:50:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 10 monclient: dump: 2024-09-11T01:50:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:25.867 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:25.867 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:25.867 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:25.867 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:25.867 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:25.867 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:25.867 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:25.867 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:25.867 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:25.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:50:25.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 10 monclient: _reopen_session rank -1 2024-09-11T01:50:25.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 1 -- 172.21.15.12:0/970996451 >> v1:172.21.15.12:6789/0 conn(0x7f31280036e0 legacy=0x7f3128005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:25.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:50:25.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314ab10640 1 -- 172.21.15.12:0/970996451 reap_dead start 2024-09-11T01:50:25.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 1 --2- 172.21.15.12:0/970996451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31300039e0 0x7f3130003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:25.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 10 monclient(hunting): picked mon.a con 0x7f31300039e0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:50:25.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 1 --2- 172.21.15.12:0/970996451 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3130004800 0x7f3130004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:25.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.859+0000 7f314930d640 10 monclient(hunting): picked mon.b con 0x7f3130004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:50:25.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314930d640 1 --2- 172.21.15.12:0/970996451 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f31300053f0 0x7f313000ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:25.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314930d640 10 monclient(hunting): picked mon.c con 0x7f31300053f0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:50:25.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314930d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:25.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314930d640 1 -- 172.21.15.12:0/970996451 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f313000c2d0 con 0x7f31300039e0 2024-09-11T01:50:25.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314930d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:25.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314930d640 1 -- 172.21.15.12:0/970996451 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f313000c480 con 0x7f3130004800 2024-09-11T01:50:25.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314930d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:25.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314930d640 1 -- 172.21.15.12:0/970996451 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f313000c680 con 0x7f31300053f0 2024-09-11T01:50:25.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314930d640 10 monclient(hunting): _renew_subs 2024-09-11T01:50:25.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314930d640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:50:25.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314c573640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:50:25.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314ab10640 1 -- 172.21.15.12:0/970996451 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f31300053f0 msgr2=0x7f313000ba20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:50:25.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314ab10640 1 --2- 172.21.15.12:0/970996451 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f31300053f0 0x7f313000ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:50:25.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314a30f640 1 --2- 172.21.15.12:0/970996451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31300039e0 0x7f3130003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:25.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314a30f640 10 monclient(hunting): get_auth_request con 0x7f31300039e0 auth_method 0 2024-09-11T01:50:25.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314a30f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:25.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314a30f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:25.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314a30f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:50:25.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f3149b0e640 1 --2- 172.21.15.12:0/970996451 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3130004800 0x7f3130004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:25.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314a30f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:25.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314a30f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:25.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314a30f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:50:25.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f3149b0e640 10 monclient(hunting): get_auth_request con 0x7f3130004800 auth_method 0 2024-09-11T01:50:25.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f3149b0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:25.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f3149b0e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:25.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f3149b0e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:50:25.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f3149b0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:25.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f3149b0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:25.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f3149b0e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:50:25.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314a30f640 10 monclient(hunting): handle_auth_done global_id 214353 payload 293 2024-09-11T01:50:25.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314a30f640 1 -- 172.21.15.12:0/970996451 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f31300053f0 msgr2=0x7f313000ba20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:50:25.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314a30f640 1 --2- 172.21.15.12:0/970996451 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f31300053f0 0x7f313000ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:25.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314a30f640 1 -- 172.21.15.12:0/970996451 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3130004800 msgr2=0x7f3130004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:50:25.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314a30f640 1 --2- 172.21.15.12:0/970996451 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3130004800 0x7f3130004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:25.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314a30f640 10 monclient: _finish_hunting 0 2024-09-11T01:50:25.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314a30f640 1 monclient: found mon.a 2024-09-11T01:50:25.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314a30f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:50:25.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314a30f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:50:25.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314a30f640 1 -- 172.21.15.12:0/970996451 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3130003520 con 0x7f31300039e0 2024-09-11T01:50:25.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314a30f640 10 monclient: _finish_auth 0 2024-09-11T01:50:25.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314a30f640 10 monclient: _check_auth_tickets 2024-09-11T01:50:25.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.860+0000 7f314a30f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:50:25.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.863+0000 7f314a30f640 1 -- 172.21.15.12:0/970996451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31300039e0 msgr2=0x7f3130003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:25.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.863+0000 7f314a30f640 1 --2- 172.21.15.12:0/970996451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31300039e0 0x7f3130003de0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3140009100 tx=0x7f3130005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9faed7052094676e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:25.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.863+0000 7f314930d640 1 -- 172.21.15.12:0/970996451 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3140039070 con 0x7f31300039e0 2024-09-11T01:50:25.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.863+0000 7f314930d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:25.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.863+0000 7f314930d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:50:25.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.863+0000 7f314930d640 10 monclient: dump: 2024-09-11T01:50:25.876 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:25.876 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:25.877 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:25.877 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:25.877 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:25.877 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:25.877 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:25.877 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:25.877 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:25.877 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:25.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.863+0000 7f314930d640 1 -- 172.21.15.12:0/970996451 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3140034410 con 0x7f31300039e0 2024-09-11T01:50:25.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.863+0000 7f314930d640 10 monclient: handle_config config(0 keys) 2024-09-11T01:50:25.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.863+0000 7f314930d640 1 -- 172.21.15.12:0/970996451 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314000ce00 con 0x7f31300039e0 2024-09-11T01:50:25.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.863+0000 7f314c573640 10 monclient: get_monmap_and_config success 2024-09-11T01:50:25.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.863+0000 7f314c573640 4 set_mon_vals no callback set 2024-09-11T01:50:25.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.863+0000 7f314930d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:25.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.863+0000 7f314930d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:50:25.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.863+0000 7f314930d640 10 monclient: dump: 2024-09-11T01:50:25.878 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:25.878 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:25.878 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:25.878 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:25.878 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:25.878 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:25.878 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:25.878 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:25.878 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:25.878 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:25.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.864+0000 7f314c573640 10 monclient: shutdown 2024-09-11T01:50:25.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.864+0000 7f314c573640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:50:25.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.864+0000 7f314c573640 1 -- 172.21.15.12:0/970996451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31300039e0 msgr2=0x7f3130003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:25.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.864+0000 7f314c573640 1 --2- 172.21.15.12:0/970996451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31300039e0 0x7f3130003de0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3140009100 tx=0x7f3130005830 comp rx=0 tx=0).stop 2024-09-11T01:50:25.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.864+0000 7f314c573640 1 -- 172.21.15.12:0/970996451 shutdown_connections 2024-09-11T01:50:25.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.864+0000 7f314c573640 1 --2- 172.21.15.12:0/970996451 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f31300053f0 0x7f313000ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:25.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.864+0000 7f314c573640 1 --2- 172.21.15.12:0/970996451 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3130004800 0x7f3130004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:25.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.864+0000 7f314c573640 1 --2- 172.21.15.12:0/970996451 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31300039e0 0x7f3130003de0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:25.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.864+0000 7f314c573640 1 -- 172.21.15.12:0/970996451 >> 172.21.15.12:0/970996451 conn(0x7f31440a3ea0 msgr2=0x7f31440a42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:50:25.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.864+0000 7f314c573640 1 -- 172.21.15.12:0/970996451 shutdown_connections 2024-09-11T01:50:25.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.864+0000 7f314c573640 1 -- 172.21.15.12:0/970996451 wait complete. 2024-09-11T01:50:25.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.865+0000 7f314c573640 10 monclient: build_initial_monmap 2024-09-11T01:50:25.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.865+0000 7f314c573640 10 monclient: monmap: 2024-09-11T01:50:25.879 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:50:25.880 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:50:25.880 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:50:25.880 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:50:25.880 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:50:25.880 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:25.880 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:50:25.880 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:50:25.880 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:50:25.880 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:25.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.866+0000 7f314c573640 1 Processor -- start 2024-09-11T01:50:25.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.866+0000 7f314c573640 1 -- start start 2024-09-11T01:50:25.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.866+0000 7f314c573640 10 monclient: init 2024-09-11T01:50:25.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f314c573640 10 monclient: _reopen_session rank -1 2024-09-11T01:50:25.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f314c573640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:50:25.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f314c573640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f31300039e0 0x7f3144099500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:25.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f314c573640 10 monclient(hunting): picked mon.noname-c con 0x7f31300039e0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:50:25.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f314c573640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3130004800 0x7f3144099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:25.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f314c573640 10 monclient(hunting): picked mon.noname-a con 0x7f3130004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:50:25.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f314c573640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31300053f0 0x7f31440931f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:25.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f314c573640 10 monclient(hunting): picked mon.noname-b con 0x7f31300053f0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:50:25.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f314c573640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:25.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f314c573640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f314409ea20 con 0x7f3130004800 2024-09-11T01:50:25.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f314c573640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:25.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f314c573640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f314409e8a0 con 0x7f31300053f0 2024-09-11T01:50:25.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f314c573640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:25.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f314c573640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f314406cb50 con 0x7f31300039e0 2024-09-11T01:50:25.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f314c573640 10 monclient(hunting): _renew_subs 2024-09-11T01:50:25.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f314c573640 10 monclient(hunting): authenticate will time out at 2032.310113s 2024-09-11T01:50:25.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f314a30f640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f31300039e0 msgr2=0x7f3144099500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:50:25.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f314a30f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f31300039e0 0x7f3144099500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:50:25.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f3149b0e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3130004800 0x7f3144099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:25.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f3149b0e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3130004800 0x7f3144099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42848/0 (socket says 172.21.15.12:42848) 2024-09-11T01:50:25.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f3149b0e640 1 -- 172.21.15.12:0/3595868740 learned_addr learned my addr 172.21.15.12:0/3595868740 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:50:25.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f3149b0e640 10 monclient(hunting): get_auth_request con 0x7f3130004800 auth_method 0 2024-09-11T01:50:25.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f3149b0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:25.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f3149b0e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:25.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f3149b0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:25.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f314ab10640 1 --2- 172.21.15.12:0/3595868740 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31300053f0 0x7f31440931f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:25.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f314ab10640 10 monclient(hunting): get_auth_request con 0x7f31300053f0 auth_method 0 2024-09-11T01:50:25.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f314ab10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:25.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f314ab10640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:25.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f314ab10640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:25.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f3149b0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:25.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f3149b0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:25.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f3149b0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:25.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f314ab10640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:25.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f314ab10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:25.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f314ab10640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:25.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f3149b0e640 10 monclient(hunting): handle_auth_done global_id 214356 payload 824 2024-09-11T01:50:25.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f3149b0e640 1 -- 172.21.15.12:0/3595868740 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f31300039e0 msgr2=0x7f3144099500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:50:25.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f3149b0e640 1 --2- 172.21.15.12:0/3595868740 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f31300039e0 0x7f3144099500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:25.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f3149b0e640 1 -- 172.21.15.12:0/3595868740 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31300053f0 msgr2=0x7f31440931f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:25.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f3149b0e640 1 --2- 172.21.15.12:0/3595868740 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31300053f0 0x7f31440931f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:25.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f3149b0e640 10 monclient: _finish_hunting 0 2024-09-11T01:50:25.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f3149b0e640 1 monclient: found mon.noname-a 2024-09-11T01:50:25.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f3149b0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:50:25.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f3149b0e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:50:25.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.867+0000 7f3149b0e640 1 -- 172.21.15.12:0/3595868740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f314409e590 con 0x7f3130004800 2024-09-11T01:50:25.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.870+0000 7f3149b0e640 1 -- 172.21.15.12:0/3595868740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3130004800 msgr2=0x7f3144099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:25.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.870+0000 7f3149b0e640 1 --2- 172.21.15.12:0/3595868740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3130004800 0x7f3144099a40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3134007540 tx=0x7f313400ff80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:25.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.870+0000 7f313affd640 1 -- 172.21.15.12:0/3595868740 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3134018640 con 0x7f3130004800 2024-09-11T01:50:25.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.870+0000 7f313affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:25.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.870+0000 7f313affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:50:25.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.870+0000 7f313affd640 10 monclient: dump: 2024-09-11T01:50:25.884 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:25.885 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:25.885 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:25.885 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:25.885 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:25.885 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:25.885 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:25.885 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:25.885 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:25.885 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:25.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.870+0000 7f313affd640 10 monclient: _finish_auth 0 2024-09-11T01:50:25.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.870+0000 7f313affd640 10 monclient: _check_auth_tickets 2024-09-11T01:50:25.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.870+0000 7f313affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:50:25.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.870+0000 7f314c573640 5 monclient: authenticate success, global_id 214356 2024-09-11T01:50:25.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.870+0000 7f314c573640 10 monclient: _renew_subs 2024-09-11T01:50:25.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.870+0000 7f314c573640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:50:25.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.870+0000 7f314c573640 1 -- 172.21.15.12:0/3595868740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3144099f80 con 0x7f3130004800 2024-09-11T01:50:25.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.870+0000 7f314c573640 10 monclient: _renew_subs 2024-09-11T01:50:25.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.870+0000 7f313affd640 1 -- 172.21.15.12:0/3595868740 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31340187e0 con 0x7f3130004800 2024-09-11T01:50:25.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.870+0000 7f314c573640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:50:25.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.870+0000 7f314c573640 1 -- 172.21.15.12:0/3595868740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3144096a00 con 0x7f3130004800 2024-09-11T01:50:25.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.870+0000 7f313affd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:50:25.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.870+0000 7f313affd640 1 -- 172.21.15.12:0/3595868740 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3134018bd0 con 0x7f3130004800 2024-09-11T01:50:25.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.870+0000 7f313affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:25.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.870+0000 7f313affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:50:25.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.870+0000 7f313affd640 10 monclient: dump: 2024-09-11T01:50:25.886 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:25.886 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:25.886 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:25.887 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:25.887 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:25.887 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:25.887 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:25.887 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:25.887 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:25.887 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:25.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.870+0000 7f3148b0c640 4 set_mon_vals no callback set 2024-09-11T01:50:25.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.871+0000 7f314c573640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:25.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.871+0000 7f314c573640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:25.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.871+0000 7f314c573640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:50:25.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.871+0000 7f314c573640 1 -- 172.21.15.12:0/3595868740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f314406cf30 con 0x7f3130004800 2024-09-11T01:50:25.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.872+0000 7f313affd640 1 -- 172.21.15.12:0/3595868740 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f3134016070 con 0x7f3130004800 2024-09-11T01:50:25.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.872+0000 7f313affd640 1 --2- 172.21.15.12:0/3595868740 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f312803ed80 0x7f3128041240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:25.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.872+0000 7f313affd640 1 -- 172.21.15.12:0/3595868740 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(707..707 src has 507..707) ==== 4787+0+0 (secure 0 0 0) 0x7f3134054180 con 0x7f3130004800 2024-09-11T01:50:25.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.872+0000 7f314a30f640 1 --2- 172.21.15.12:0/3595868740 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f312803ed80 0x7f3128041240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:25.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.872+0000 7f314a30f640 10 monclient: get_auth_request con 0x7f312803ed80 auth_method 0 2024-09-11T01:50:25.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.875+0000 7f314a30f640 1 --2- 172.21.15.12:0/3595868740 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f312803ed80 0x7f3128041240 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3140036000 tx=0x7f314002f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:25.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.877+0000 7f313affd640 1 -- 172.21.15.12:0/3595868740 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3134017e20 con 0x7f3130004800 2024-09-11T01:50:25.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.877+0000 7f313affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:25.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:25.877+0000 7f313affd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:50:26.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.137+0000 7f314c573640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:50:26.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.137+0000 7f314c573640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:50:26.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.137+0000 7f314c573640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:50:26.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.137+0000 7f314c573640 1 -- 172.21.15.12:0/3595868740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f314406e570 con 0x7f3130004800 2024-09-11T01:50:26.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.138+0000 7f313affd640 1 -- 172.21.15.12:0/3595868740 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f3134009d50 con 0x7f3130004800 2024-09-11T01:50:26.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.138+0000 7f313affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:50:26.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.138+0000 7f313affd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:50:26.141 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":196,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:50:26.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.140+0000 7f314c573640 1 -- 172.21.15.12:0/3595868740 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f312803ed80 msgr2=0x7f3128041240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:26.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.140+0000 7f314c573640 1 --2- 172.21.15.12:0/3595868740 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f312803ed80 0x7f3128041240 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3140036000 tx=0x7f314002f000 comp rx=0 tx=0).stop 2024-09-11T01:50:26.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.140+0000 7f314c573640 10 monclient: shutdown 2024-09-11T01:50:26.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.140+0000 7f314c573640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:50:26.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.140+0000 7f314c573640 1 -- 172.21.15.12:0/3595868740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3130004800 msgr2=0x7f3144099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:26.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.140+0000 7f314c573640 1 --2- 172.21.15.12:0/3595868740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3130004800 0x7f3144099a40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3134007540 tx=0x7f313400ff80 comp rx=0 tx=0).stop 2024-09-11T01:50:26.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.140+0000 7f314c573640 1 -- 172.21.15.12:0/3595868740 shutdown_connections 2024-09-11T01:50:26.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.140+0000 7f314c573640 1 --2- 172.21.15.12:0/3595868740 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31300053f0 0x7f31440931f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:26.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.140+0000 7f314c573640 1 --2- 172.21.15.12:0/3595868740 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f312803ed80 0x7f3128041240 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:26.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.140+0000 7f314c573640 1 --2- 172.21.15.12:0/3595868740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3130004800 0x7f3144099a40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:26.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.140+0000 7f314c573640 1 --2- 172.21.15.12:0/3595868740 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f31300039e0 0x7f3144099500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:26.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.140+0000 7f314c573640 1 -- 172.21.15.12:0/3595868740 >> 172.21.15.12:0/3595868740 conn(0x7f31440a3ea0 msgr2=0x7f3144070f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:50:26.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.140+0000 7f314c573640 1 -- 172.21.15.12:0/3595868740 shutdown_connections 2024-09-11T01:50:26.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.140+0000 7f314c573640 1 -- 172.21.15.12:0/3595868740 wait complete. 2024-09-11T01:50:26.154 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:50:26.154 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-11T01:50:26.154 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:50:26.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.253+0000 7fe4cbe2b640 10 monclient: get_monmap_and_config 2024-09-11T01:50:26.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.253+0000 7fe4cbe2b640 10 monclient: build_initial_monmap 2024-09-11T01:50:26.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.253+0000 7fe4cbe2b640 10 monclient: monmap: 2024-09-11T01:50:26.256 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:50:26.256 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:26.256 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:50:26.255044+0000 2024-09-11T01:50:26.256 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:50:26.255044+0000 2024-09-11T01:50:26.256 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:50:26.256 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:26.256 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:50:26.256 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:50:26.257 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:50:26.257 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:50:26.257 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:50:26.257 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:26.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.253+0000 7fe4cbe2b640 1 Processor -- start 2024-09-11T01:50:26.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.254+0000 7fe4cbe2b640 1 -- start start 2024-09-11T01:50:26.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.254+0000 7fe4cbe2b640 10 monclient: init 2024-09-11T01:50:26.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.254+0000 7fe4cbe2b640 10 monclient: _reopen_session rank -1 2024-09-11T01:50:26.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.254+0000 7fe4cbe2b640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-11T01:50:26.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.254+0000 7fe4cbe2b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe4c40a2e20 0x7fe4c406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:26.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.254+0000 7fe4cbe2b640 10 monclient(hunting): picked mon.noname-a con 0x7fe4c40a2e20 addr v2:172.21.15.12:3300/0 2024-09-11T01:50:26.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.254+0000 7fe4cbe2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe4c40a4970 addr v1:172.21.15.12:6789/0 2024-09-11T01:50:26.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.254+0000 7fe4cbe2b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe4c406f910 addr v1:172.21.15.71:6789/0 2024-09-11T01:50:26.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.254+0000 7fe4cbe2b640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4c406f4f0 con 0x7fe4c40a4970 2024-09-11T01:50:26.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.254+0000 7fe4cbe2b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4c40a3910 con 0x7fe4c406f910 2024-09-11T01:50:26.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.254+0000 7fe4cbe2b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:26.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.254+0000 7fe4cbe2b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fe4c406ef30 con 0x7fe4c40a2e20 2024-09-11T01:50:26.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.254+0000 7fe4cbe2b640 10 monclient(hunting): _renew_subs 2024-09-11T01:50:26.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.254+0000 7fe4cbe2b640 10 monclient(hunting): authenticate will time out at 2032.697751s 2024-09-11T01:50:26.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.254+0000 7fe4c9bc7640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe4c40a2e20 0x7fe4c406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:26.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.254+0000 7fe4c9bc7640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe4c40a2e20 0x7fe4c406ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42864/0 (socket says 172.21.15.12:42864) 2024-09-11T01:50:26.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.254+0000 7fe4c9bc7640 1 -- 172.21.15.12:0/183505691 learned_addr learned my addr 172.21.15.12:0/183505691 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:50:26.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.254+0000 7fe4c9bc7640 10 monclient(hunting): get_auth_request con 0x7fe4c40a2e20 auth_method 0 2024-09-11T01:50:26.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.254+0000 7fe4c9bc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:26.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.254+0000 7fe4c9bc7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:26.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.254+0000 7fe4c9bc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:26.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.254+0000 7fe4c9bc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:26.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.254+0000 7fe4c9bc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:26.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.254+0000 7fe4c9bc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:26.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.257+0000 7fe4c93c6640 1 -- 172.21.15.12:0/183505691 >> v1:172.21.15.12:6789/0 conn(0x7fe4c40a4970 legacy=0x7fe4c40a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:26.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.257+0000 7fe4c9bc7640 10 monclient(hunting): handle_auth_done global_id 214359 payload 274 2024-09-11T01:50:26.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.257+0000 7fe4c9bc7640 1 -- 172.21.15.12:0/183505691 >> v1:172.21.15.71:6789/0 conn(0x7fe4c406f910 legacy=0x7fe4c413f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:50:26.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.257+0000 7fe4c9bc7640 1 -- 172.21.15.12:0/183505691 >> v1:172.21.15.12:6789/0 conn(0x7fe4c40a4970 legacy=0x7fe4c40a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:26.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.257+0000 7fe4c9bc7640 10 monclient: _finish_hunting 0 2024-09-11T01:50:26.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.257+0000 7fe4c9bc7640 1 monclient: found mon.noname-a 2024-09-11T01:50:26.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.257+0000 7fe4c9bc7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:50:26.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.257+0000 7fe4c9bc7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:50:26.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.257+0000 7fe4c9bc7640 1 -- 172.21.15.12:0/183505691 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c40a3ae0 con 0x7fe4c40a2e20 2024-09-11T01:50:26.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.260+0000 7fe4c9bc7640 1 -- 172.21.15.12:0/183505691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c40a2e20 msgr2=0x7fe4c406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:26.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.260+0000 7fe4c9bc7640 1 --2- 172.21.15.12:0/183505691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c40a2e20 0x7fe4c406ded0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe4c00098f0 tx=0x7fe4c002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88283ed873cd4b4a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:26.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.260+0000 7fe4c8bc5640 1 -- 172.21.15.12:0/183505691 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0037070 con 0x7fe4c40a2e20 2024-09-11T01:50:26.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.260+0000 7fe4c8bc5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:26.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.260+0000 7fe4c8bc5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:50:26.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.260+0000 7fe4c8bc5640 10 monclient: dump: 2024-09-11T01:50:26.264 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:26.264 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:26.264 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:26.264 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:26.264 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:26.264 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:26.264 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:26.264 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:26.264 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:26.264 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:26.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.260+0000 7fe4c8bc5640 10 monclient: _finish_auth 0 2024-09-11T01:50:26.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.260+0000 7fe4c8bc5640 10 monclient: _check_auth_tickets 2024-09-11T01:50:26.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.260+0000 7fe4c8bc5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:50:26.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.260+0000 7fe4c8bc5640 1 -- 172.21.15.12:0/183505691 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4c0032640 con 0x7fe4c40a2e20 2024-09-11T01:50:26.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.260+0000 7fe4c8bc5640 10 monclient: handle_config config(0 keys) 2024-09-11T01:50:26.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.260+0000 7fe4c8bc5640 1 -- 172.21.15.12:0/183505691 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0032a10 con 0x7fe4c40a2e20 2024-09-11T01:50:26.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.260+0000 7fe4cbe2b640 5 monclient: authenticate success, global_id 214359 2024-09-11T01:50:26.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.260+0000 7fe4cbe2b640 10 monclient: get_monmap_and_config success 2024-09-11T01:50:26.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.260+0000 7fe4cbe2b640 4 set_mon_vals no callback set 2024-09-11T01:50:26.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.260+0000 7fe4c8bc5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:26.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.260+0000 7fe4c8bc5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:50:26.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.260+0000 7fe4c8bc5640 10 monclient: dump: 2024-09-11T01:50:26.266 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:26.266 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:26.266 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:26.266 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:26.266 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:26.266 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:26.266 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:26.266 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:26.266 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:26.266 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:26.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.260+0000 7fe4cbe2b640 10 monclient: shutdown 2024-09-11T01:50:26.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.260+0000 7fe4cbe2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:50:26.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.260+0000 7fe4cbe2b640 1 -- 172.21.15.12:0/183505691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c40a2e20 msgr2=0x7fe4c406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:26.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.260+0000 7fe4cbe2b640 1 --2- 172.21.15.12:0/183505691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c40a2e20 0x7fe4c406ded0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe4c00098f0 tx=0x7fe4c002a1c0 comp rx=0 tx=0).stop 2024-09-11T01:50:26.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.260+0000 7fe4cbe2b640 1 -- 172.21.15.12:0/183505691 shutdown_connections 2024-09-11T01:50:26.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.260+0000 7fe4cbe2b640 1 --2- 172.21.15.12:0/183505691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c40a2e20 0x7fe4c406ded0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:26.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.260+0000 7fe4cbe2b640 1 -- 172.21.15.12:0/183505691 >> 172.21.15.12:0/183505691 conn(0x7fe4c40a8600 msgr2=0x7fe4c40a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:50:26.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.261+0000 7fe4cbe2b640 1 -- 172.21.15.12:0/183505691 shutdown_connections 2024-09-11T01:50:26.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.261+0000 7fe4cbe2b640 1 -- 172.21.15.12:0/183505691 wait complete. 2024-09-11T01:50:26.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.261+0000 7fe4cbe2b640 10 monclient: build_initial_monmap 2024-09-11T01:50:26.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.261+0000 7fe4cbe2b640 10 monclient: monmap: 2024-09-11T01:50:26.272 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:50:26.272 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:50:26.273 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:50:26.273 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:50:26.273 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:50:26.273 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:26.273 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:50:26.273 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:50:26.273 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:50:26.273 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:26.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.261+0000 7fe4cbe2b640 1 Processor -- start 2024-09-11T01:50:26.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.261+0000 7fe4cbe2b640 1 -- start start 2024-09-11T01:50:26.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.261+0000 7fe4cbe2b640 10 monclient: init 2024-09-11T01:50:26.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4cbe2b640 10 monclient: _reopen_session rank -1 2024-09-11T01:50:26.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4cbe2b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:50:26.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4cbe2b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4c406f910 0x7fe4c41370a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:26.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4cbe2b640 10 monclient(hunting): picked mon.noname-c con 0x7fe4c406f910 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:50:26.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4cbe2b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c40a2e20 0x7fe4c41375e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:26.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4cbe2b640 10 monclient(hunting): picked mon.noname-a con 0x7fe4c40a2e20 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:50:26.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4cbe2b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe4c40a4970 0x7fe4c4137b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:26.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4cbe2b640 10 monclient(hunting): picked mon.noname-b con 0x7fe4c40a4970 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:50:26.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4cbe2b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:26.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4cbe2b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c4140d60 con 0x7fe4c40a2e20 2024-09-11T01:50:26.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4cbe2b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:26.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4cbe2b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c406ec50 con 0x7fe4c40a4970 2024-09-11T01:50:26.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4cbe2b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:26.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4cbe2b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe4c4140ee0 con 0x7fe4c406f910 2024-09-11T01:50:26.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4cbe2b640 10 monclient(hunting): _renew_subs 2024-09-11T01:50:26.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4cbe2b640 10 monclient(hunting): authenticate will time out at 2032.705438s 2024-09-11T01:50:26.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4c9bc7640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4c406f910 msgr2=0x7fe4c41370a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:50:26.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4c9bc7640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4c406f910 0x7fe4c41370a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:50:26.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4c93c6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c40a2e20 0x7fe4c41375e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:26.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4c93c6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c40a2e20 0x7fe4c41375e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42866/0 (socket says 172.21.15.12:42866) 2024-09-11T01:50:26.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4c93c6640 1 -- 172.21.15.12:0/2135995896 learned_addr learned my addr 172.21.15.12:0/2135995896 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:50:26.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4c93c6640 10 monclient(hunting): get_auth_request con 0x7fe4c40a2e20 auth_method 0 2024-09-11T01:50:26.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4c93c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:26.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4c93c6640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:26.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4c93c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:26.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4c93c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:26.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4c93c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:26.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4c93c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:26.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4ca3c8640 1 --2- 172.21.15.12:0/2135995896 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe4c40a4970 0x7fe4c4137b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:26.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4c93c6640 10 monclient(hunting): handle_auth_done global_id 214362 payload 824 2024-09-11T01:50:26.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4c93c6640 1 -- 172.21.15.12:0/2135995896 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4c406f910 msgr2=0x7fe4c41370a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:50:26.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4c93c6640 1 --2- 172.21.15.12:0/2135995896 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4c406f910 0x7fe4c41370a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:26.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4c93c6640 1 -- 172.21.15.12:0/2135995896 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe4c40a4970 msgr2=0x7fe4c4137b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:26.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4c93c6640 1 --2- 172.21.15.12:0/2135995896 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe4c40a4970 0x7fe4c4137b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:26.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4c93c6640 10 monclient: _finish_hunting 0 2024-09-11T01:50:26.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4c93c6640 1 monclient: found mon.noname-a 2024-09-11T01:50:26.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4c93c6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:50:26.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4c93c6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:50:26.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4c93c6640 1 -- 172.21.15.12:0/2135995896 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c413ad80 con 0x7fe4c40a2e20 2024-09-11T01:50:26.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4ca3c8640 10 monclient: get_auth_request con 0x7fe4c40a4970 auth_method 0 2024-09-11T01:50:26.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.262+0000 7fe4ca3c8640 1 --2- 172.21.15.12:0/2135995896 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe4c40a4970 0x7fe4c4137b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:50:26.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.265+0000 7fe4c93c6640 1 -- 172.21.15.12:0/2135995896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c40a2e20 msgr2=0x7fe4c41375e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:26.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.265+0000 7fe4c93c6640 1 --2- 172.21.15.12:0/2135995896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c40a2e20 0x7fe4c41375e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe4b400c450 tx=0x7fe4b400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:26.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.265+0000 7fe4ba7fc640 1 -- 172.21.15.12:0/2135995896 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b4017070 con 0x7fe4c40a2e20 2024-09-11T01:50:26.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.265+0000 7fe4ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:26.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.265+0000 7fe4ba7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:50:26.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.265+0000 7fe4ba7fc640 10 monclient: dump: 2024-09-11T01:50:26.277 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:26.277 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:26.277 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:26.277 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:26.277 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:26.277 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:26.277 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:26.277 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:26.277 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:26.278 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:26.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.265+0000 7fe4ba7fc640 10 monclient: _finish_auth 0 2024-09-11T01:50:26.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.265+0000 7fe4ba7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:50:26.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.265+0000 7fe4ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:50:26.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.265+0000 7fe4ba7fc640 1 -- 172.21.15.12:0/2135995896 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4b40057c0 con 0x7fe4c40a2e20 2024-09-11T01:50:26.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.265+0000 7fe4cbe2b640 5 monclient: authenticate success, global_id 214362 2024-09-11T01:50:26.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.265+0000 7fe4cbe2b640 10 monclient: _renew_subs 2024-09-11T01:50:26.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.265+0000 7fe4cbe2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:50:26.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.265+0000 7fe4cbe2b640 1 -- 172.21.15.12:0/2135995896 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4c413b060 con 0x7fe4c40a2e20 2024-09-11T01:50:26.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.265+0000 7fe4ba7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:50:26.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.265+0000 7fe4ba7fc640 1 -- 172.21.15.12:0/2135995896 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b4005b90 con 0x7fe4c40a2e20 2024-09-11T01:50:26.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.265+0000 7fe4ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:26.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.265+0000 7fe4ba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:50:26.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.265+0000 7fe4ba7fc640 10 monclient: dump: 2024-09-11T01:50:26.279 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:26.279 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:26.279 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:26.279 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:26.279 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:26.279 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:26.279 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:26.279 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:26.279 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:26.279 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:26.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.265+0000 7fe4bbfff640 4 set_mon_vals no callback set 2024-09-11T01:50:26.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.265+0000 7fe4cbe2b640 10 monclient: _renew_subs 2024-09-11T01:50:26.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.265+0000 7fe4cbe2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:50:26.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.265+0000 7fe4cbe2b640 1 -- 172.21.15.12:0/2135995896 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4c41408d0 con 0x7fe4c40a2e20 2024-09-11T01:50:26.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.266+0000 7fe4cbe2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:26.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.266+0000 7fe4cbe2b640 1 --2- 172.21.15.12:0/2135995896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c41cd8d0 0x7fe4c41cdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:26.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.266+0000 7fe4ba7fc640 1 -- 172.21.15.12:0/2135995896 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fe4b401b3b0 con 0x7fe4c40a2e20 2024-09-11T01:50:26.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.266+0000 7fe4cbe2b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:26.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.266+0000 7fe4cbe2b640 1 -- 172.21.15.12:0/2135995896 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c406a060 con 0x7fe4c41cd8d0 2024-09-11T01:50:26.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.266+0000 7fe4c9bc7640 1 --2- 172.21.15.12:0/2135995896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c41cd8d0 0x7fe4c41cdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:26.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.266+0000 7fe4c9bc7640 10 monclient: get_auth_request con 0x7fe4c41cd8d0 auth_method 0 2024-09-11T01:50:26.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.266+0000 7fe4c9bc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:26.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.266+0000 7fe4c9bc7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:26.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.266+0000 7fe4c9bc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:26.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.266+0000 7fe4c9bc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:26.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.266+0000 7fe4c9bc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:26.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.266+0000 7fe4c9bc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:26.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.266+0000 7fe4ba7fc640 1 --2- 172.21.15.12:0/2135995896 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe4a803e240 0x7fe4a8040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:26.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.266+0000 7fe4ba7fc640 1 -- 172.21.15.12:0/2135995896 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(707..707 src has 507..707) ==== 4787+0+0 (secure 0 0 0) 0x7fe4b40524b0 con 0x7fe4c40a2e20 2024-09-11T01:50:26.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.266+0000 7fe4c9bc7640 10 monclient(hunting): handle_auth_done global_id 214365 payload 824 2024-09-11T01:50:26.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.266+0000 7fe4c9bc7640 1 -- 172.21.15.12:0/2135995896 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe4c406ded0 con 0x7fe4c41cd8d0 2024-09-11T01:50:26.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.266+0000 7fe4ca3c8640 1 --2- 172.21.15.12:0/2135995896 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe4a803e240 0x7fe4a8040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:26.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.266+0000 7fe4ca3c8640 10 monclient: get_auth_request con 0x7fe4a803e240 auth_method 0 2024-09-11T01:50:26.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.268+0000 7fe4c9bc7640 1 -- 172.21.15.12:0/2135995896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c41cd8d0 msgr2=0x7fe4c41cdcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:26.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.268+0000 7fe4c9bc7640 1 --2- 172.21.15.12:0/2135995896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c41cd8d0 0x7fe4c41cdcb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0034000 tx=0x7fe4c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:26.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.268+0000 7fe4ba7fc640 1 -- 172.21.15.12:0/2135995896 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0041070 con 0x7fe4c41cd8d0 2024-09-11T01:50:26.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.268+0000 7fe4ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:26.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.268+0000 7fe4ba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:50:26.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.268+0000 7fe4ba7fc640 10 monclient: dump: 2024-09-11T01:50:26.281 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:26.285 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:26.285 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:26.285 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:26.285 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:26.285 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:26.285 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:26.285 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:26.285 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:26.285 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:26.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.269+0000 7fe4ba7fc640 1 -- 172.21.15.12:0/2135995896 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe4c003ca80 con 0x7fe4c41cd8d0 2024-09-11T01:50:26.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.269+0000 7fe4ba7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:26.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.269+0000 7fe4ba7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:50:26.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.269+0000 7fe4ba7fc640 1 -- 172.21.15.12:0/2135995896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c41cd8d0 msgr2=0x7fe4c41cdcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:26.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.269+0000 7fe4ba7fc640 1 --2- 172.21.15.12:0/2135995896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c41cd8d0 0x7fe4c41cdcb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0034000 tx=0x7fe4c00046c0 comp rx=0 tx=0).stop 2024-09-11T01:50:26.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.269+0000 7fe4ba7fc640 1 -- 172.21.15.12:0/2135995896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c41cd8d0 msgr2=0x7fe4c41cdcb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:50:26.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.269+0000 7fe4ba7fc640 1 --2- 172.21.15.12:0/2135995896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c41cd8d0 0x7fe4c41cdcb0 secure :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0034000 tx=0x7fe4c00046c0 comp rx=0 tx=0).stop 2024-09-11T01:50:26.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.269+0000 7fe4ca3c8640 1 --2- 172.21.15.12:0/2135995896 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe4a803e240 0x7fe4a8040700 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc0044a0 tx=0x7fe4bc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:26.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.275+0000 7fe4cbe2b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:50:26.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.275+0000 7fe4cbe2b640 1 --2- 172.21.15.12:0/2135995896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c402d440 0x7fe4c402d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:26.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.275+0000 7fe4cbe2b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:26.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.275+0000 7fe4cbe2b640 1 -- 172.21.15.12:0/2135995896 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c409da30 con 0x7fe4c402d440 2024-09-11T01:50:26.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.275+0000 7fe4c9bc7640 1 --2- 172.21.15.12:0/2135995896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c402d440 0x7fe4c402d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:26.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.275+0000 7fe4c9bc7640 10 monclient: get_auth_request con 0x7fe4c402d440 auth_method 0 2024-09-11T01:50:26.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.275+0000 7fe4c9bc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:26.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.275+0000 7fe4c9bc7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:26.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.275+0000 7fe4c9bc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:26.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.275+0000 7fe4c9bc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:26.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.275+0000 7fe4c9bc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:26.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.275+0000 7fe4c9bc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:26.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.275+0000 7fe4c9bc7640 10 monclient(hunting): handle_auth_done global_id 214368 payload 824 2024-09-11T01:50:26.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.275+0000 7fe4c9bc7640 1 -- 172.21.15.12:0/2135995896 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe4c4140010 con 0x7fe4c402d440 2024-09-11T01:50:26.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.278+0000 7fe4c9bc7640 1 -- 172.21.15.12:0/2135995896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c402d440 msgr2=0x7fe4c402d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:26.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.278+0000 7fe4c9bc7640 1 --2- 172.21.15.12:0/2135995896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c402d440 0x7fe4c402d820 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe4c00098f0 tx=0x7fe4c0009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:26.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.278+0000 7fe4ba7fc640 1 -- 172.21.15.12:0/2135995896 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0037040 con 0x7fe4c402d440 2024-09-11T01:50:26.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.278+0000 7fe4ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:26.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.278+0000 7fe4ba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:50:26.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.278+0000 7fe4ba7fc640 10 monclient: dump: 2024-09-11T01:50:26.290 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:26.290 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:26.290 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:26.290 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:26.290 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:26.290 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:26.290 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:26.290 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:26.290 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:26.290 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:26.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.278+0000 7fe4ba7fc640 1 -- 172.21.15.12:0/2135995896 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4700 (secure 0 0 0) 0x7fe4c0003a30 con 0x7fe4c402d440 2024-09-11T01:50:26.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.278+0000 7fe4ba7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:50:26.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.278+0000 7fe4ba7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:50:26.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.278+0000 7fe4ba7fc640 1 -- 172.21.15.12:0/2135995896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c402d440 msgr2=0x7fe4c402d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:26.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.278+0000 7fe4ba7fc640 1 --2- 172.21.15.12:0/2135995896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c402d440 0x7fe4c402d820 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe4c00098f0 tx=0x7fe4c0009240 comp rx=0 tx=0).stop 2024-09-11T01:50:26.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.278+0000 7fe4ba7fc640 1 -- 172.21.15.12:0/2135995896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c402d440 msgr2=0x7fe4c402d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:50:26.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.278+0000 7fe4ba7fc640 1 --2- 172.21.15.12:0/2135995896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c402d440 0x7fe4c402d820 secure :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe4c00098f0 tx=0x7fe4c0009240 comp rx=0 tx=0).stop 2024-09-11T01:50:26.291 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:50:26.291 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:50:26.291 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:50:26.291 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:50:26.291 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 196, 2024-09-11T01:50:26.292 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 132478, 2024-09-11T01:50:26.292 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:50:26.292 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:50:26.292 INFO:teuthology.orchestra.run.smithi012.stdout: 1 2024-09-11T01:50:26.292 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:26.292 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:50:26.292 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:50:26.292 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:50:26.292 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:50:26.292 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:50:26.292 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:50:26.292 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:50:26.292 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:50:26.293 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:50:26.293 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:50:26.293 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:50:26.293 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:50:26.293 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:50:26.293 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:50:26.293 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:50:26.293 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:26.293 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:50:26.293 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:50:26.293 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:50:26.293 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:50:26.293 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:50:26.294 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:50:26.294 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:50:26.294 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:50:26.294 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:50:26.294 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:50:26.294 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:50:26.294 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:50:26.294 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:50:26.294 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:26.294 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:26.294 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:50:26.294 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:50:26.294 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:50:26.295 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:50:26.295 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:50:26.295 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:50:26.295 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:50:26.295 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:50:26.295 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:50:26.295 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:50:26.295 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:50:26.295 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:50:26.295 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:50:26.295 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:50:26.295 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:50:26.295 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:50:26.295 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:50:26.296 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:50:26.296 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:50:26.296 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:50:26.296 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:50:26.296 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:50:26.296 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:50:26.296 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:50:26.296 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:50:26.296 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:50:26.296 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:50:26.296 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:50:26.296 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:26.296 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:50:26.296 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:26.297 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:50:26.297 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:26.297 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:50:26.297 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:50:26.297 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:50:26.297 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:50:26.297 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:26.297 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:50:26.297 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:50:26.297 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:26.297 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:26.297 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:26.297 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:50:26.298 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:50:26.298 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:26.298 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:26.298 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:26.298 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:26.298 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:50:26.298 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:50:26.298 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:50:26.298 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:50:26.298 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:50:26.298 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:26.298 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:26.298 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:50:26.298 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:50:26.299 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:50:26.299 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:50:26.299 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:26.299 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:50:26.299 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:50:26.299 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:26.299 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:26.299 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:26.299 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:50:26.299 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:50:26.299 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:26.299 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:26.299 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:26.299 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:26.300 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:50:26.300 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:50:26.300 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:50:26.300 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:50:26.300 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:50:26.300 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:26.300 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:26.300 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:50:26.300 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:50:26.300 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:50:26.300 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:50:26.300 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:26.301 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:50:26.301 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:50:26.301 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:26.301 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:26.301 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:26.301 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:50:26.301 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:50:26.301 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:26.301 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:26.301 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:26.301 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:26.301 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:50:26.301 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:50:26.301 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:50:26.302 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:50:26.302 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:50:26.302 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:26.302 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:26.302 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:26.302 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:50:26.302 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:50:26.302 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:26.302 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:50:26.302 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:50:26.302 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:50:26.302 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:26.302 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:26.303 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:50:26.303 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:26.303 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:50:26.303 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:50:26.303 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:50:26.303 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:26.303 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:26.303 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:50:26.303 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:26.303 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:50:26.303 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:50:26.303 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:50:26.303 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:26.303 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:26.304 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:26.304 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:50:26.304 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:50:26.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.278+0000 7fe4cbe2b640 1 -- 172.21.15.12:0/2135995896 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe4a803e240 msgr2=0x7fe4a8040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:26.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.278+0000 7fe4cbe2b640 1 --2- 172.21.15.12:0/2135995896 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe4a803e240 0x7fe4a8040700 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc0044a0 tx=0x7fe4bc009040 comp rx=0 tx=0).stop 2024-09-11T01:50:26.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.278+0000 7fe4cbe2b640 10 monclient: shutdown 2024-09-11T01:50:26.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.278+0000 7fe4cbe2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:50:26.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.278+0000 7fe4cbe2b640 1 -- 172.21.15.12:0/2135995896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c40a2e20 msgr2=0x7fe4c41375e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:26.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.278+0000 7fe4cbe2b640 1 --2- 172.21.15.12:0/2135995896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4c40a2e20 0x7fe4c41375e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe4b400c450 tx=0x7fe4b400c910 comp rx=0 tx=0).stop 2024-09-11T01:50:26.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.278+0000 7fe4ca3c8640 1 -- 172.21.15.12:0/2135995896 reap_dead start 2024-09-11T01:50:26.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.278+0000 7fe4ca3c8640 1 -- 172.21.15.12:0/2135995896 reap_dead start 2024-09-11T01:50:26.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.278+0000 7fe4cbe2b640 1 -- 172.21.15.12:0/2135995896 shutdown_connections 2024-09-11T01:50:26.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.278+0000 7fe4cbe2b640 1 -- 172.21.15.12:0/2135995896 >> 172.21.15.12:0/2135995896 conn(0x7fe4c40a8600 msgr2=0x7fe4c40a1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:50:26.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.278+0000 7fe4cbe2b640 1 -- 172.21.15.12:0/2135995896 shutdown_connections 2024-09-11T01:50:26.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.278+0000 7fe4cbe2b640 1 -- 172.21.15.12:0/2135995896 wait complete. 2024-09-11T01:50:26.306 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:50:26.306 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T01:50:26.306 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:50:26.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.406+0000 7fdce45ec640 10 monclient: get_monmap_and_config 2024-09-11T01:50:26.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.406+0000 7fdce45ec640 10 monclient: build_initial_monmap 2024-09-11T01:50:26.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.406+0000 7fdce45ec640 10 monclient: monmap: 2024-09-11T01:50:26.409 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:50:26.409 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:26.409 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:50:26.407925+0000 2024-09-11T01:50:26.409 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:50:26.407925+0000 2024-09-11T01:50:26.409 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:50:26.409 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:26.409 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:50:26.409 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:50:26.409 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:50:26.410 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:50:26.410 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:50:26.410 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:26.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.406+0000 7fdce45ec640 1 Processor -- start 2024-09-11T01:50:26.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.406+0000 7fdce45ec640 1 -- start start 2024-09-11T01:50:26.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.406+0000 7fdce45ec640 10 monclient: init 2024-09-11T01:50:26.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce45ec640 10 monclient: _reopen_session rank -1 2024-09-11T01:50:26.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce45ec640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-11T01:50:26.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce45ec640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdcdc137e50 0x7fdcdc09d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:26.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce45ec640 10 monclient(hunting): picked mon.noname-b con 0x7fdcdc137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:50:26.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce45ec640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdcdc138230 0x7fdcdc138630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:26.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce45ec640 10 monclient(hunting): picked mon.noname-a con 0x7fdcdc138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:50:26.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce45ec640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fdcdc134cb0 0x7fdcdc135120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:26.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce45ec640 10 monclient(hunting): picked mon.noname-c con 0x7fdcdc134cb0 addr v2:172.21.15.71:3300/0 2024-09-11T01:50:26.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce45ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:26.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce45ec640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fdcdc0a65b0 con 0x7fdcdc138230 2024-09-11T01:50:26.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce45ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:26.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce45ec640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fdcdc0a62d0 con 0x7fdcdc134cb0 2024-09-11T01:50:26.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce45ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:26.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce45ec640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fdcdc09bb40 con 0x7fdcdc137e50 2024-09-11T01:50:26.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce45ec640 10 monclient(hunting): _renew_subs 2024-09-11T01:50:26.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce45ec640 10 monclient(hunting): authenticate will time out at 2032.850479s 2024-09-11T01:50:26.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce2388640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdcdc137e50 msgr2=0x7fdcdc09d160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:50:26.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce2388640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdcdc137e50 0x7fdcdc09d160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:50:26.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce1b87640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdcdc138230 0x7fdcdc138630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:26.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce1b87640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdcdc138230 0x7fdcdc138630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41470/0 (socket says 172.21.15.12:41470) 2024-09-11T01:50:26.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce1b87640 1 -- 172.21.15.12:0/2789998368 learned_addr learned my addr 172.21.15.12:0/2789998368 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:50:26.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce1b87640 10 monclient(hunting): get_auth_request con 0x7fdcdc138230 auth_method 0 2024-09-11T01:50:26.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce1b87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce1b87640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce1b87640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce1b87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce1b87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce1b87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce2b89640 1 --2- 172.21.15.12:0/2789998368 >> v2:172.21.15.71:3300/0 conn(0x7fdcdc134cb0 0x7fdcdc135120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce1b87640 10 monclient(hunting): handle_auth_done global_id 214371 payload 274 2024-09-11T01:50:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce1b87640 1 -- 172.21.15.12:0/2789998368 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdcdc137e50 msgr2=0x7fdcdc09d160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:50:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce1b87640 1 --2- 172.21.15.12:0/2789998368 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdcdc137e50 0x7fdcdc09d160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce1b87640 1 -- 172.21.15.12:0/2789998368 >> v2:172.21.15.71:3300/0 conn(0x7fdcdc134cb0 msgr2=0x7fdcdc135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:50:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce1b87640 1 --2- 172.21.15.12:0/2789998368 >> v2:172.21.15.71:3300/0 conn(0x7fdcdc134cb0 0x7fdcdc135120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce1b87640 10 monclient: _finish_hunting 0 2024-09-11T01:50:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce1b87640 1 monclient: found mon.noname-a 2024-09-11T01:50:26.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce1b87640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:50:26.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce1b87640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:50:26.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce1b87640 1 -- 172.21.15.12:0/2789998368 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcdc09c4a0 con 0x7fdcdc138230 2024-09-11T01:50:26.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce2b89640 10 monclient: get_auth_request con 0x7fdcdc134cb0 auth_method 0 2024-09-11T01:50:26.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.407+0000 7fdce2b89640 1 --2- 172.21.15.12:0/2789998368 >> v2:172.21.15.71:3300/0 conn(0x7fdcdc134cb0 0x7fdcdc135120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:50:26.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.410+0000 7fdce1b87640 1 -- 172.21.15.12:0/2789998368 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcdc138230 msgr2=0x7fdcdc138630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:26.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.410+0000 7fdce1b87640 1 --2- 172.21.15.12:0/2789998368 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcdc138230 0x7fdcdc138630 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc0a8360 tx=0x7fdcd802a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b538bf626c803a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:26.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.410+0000 7fdce1386640 1 -- 172.21.15.12:0/2789998368 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8037070 con 0x7fdcdc138230 2024-09-11T01:50:26.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.410+0000 7fdce1386640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:26.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.410+0000 7fdce1386640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:50:26.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.410+0000 7fdce1386640 10 monclient: dump: 2024-09-11T01:50:26.423 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:26.423 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:26.423 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:26.423 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:26.423 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:26.423 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:26.424 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:26.424 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:26.424 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:26.424 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:26.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.411+0000 7fdce1386640 10 monclient: _finish_auth 0 2024-09-11T01:50:26.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.411+0000 7fdce1386640 10 monclient: _check_auth_tickets 2024-09-11T01:50:26.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.411+0000 7fdce1386640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:50:26.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.411+0000 7fdce1386640 1 -- 172.21.15.12:0/2789998368 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcd8032640 con 0x7fdcdc138230 2024-09-11T01:50:26.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.411+0000 7fdce45ec640 5 monclient: authenticate success, global_id 214371 2024-09-11T01:50:26.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.411+0000 7fdce45ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:50:26.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.411+0000 7fdce1386640 10 monclient: handle_config config(0 keys) 2024-09-11T01:50:26.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.411+0000 7fdce45ec640 10 monclient: get_monmap_and_config success 2024-09-11T01:50:26.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.411+0000 7fdce1386640 1 -- 172.21.15.12:0/2789998368 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8032a10 con 0x7fdcdc138230 2024-09-11T01:50:26.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.411+0000 7fdce1386640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:26.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.411+0000 7fdce45ec640 4 set_mon_vals no callback set 2024-09-11T01:50:26.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.411+0000 7fdce1386640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:50:26.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.411+0000 7fdce1386640 10 monclient: dump: 2024-09-11T01:50:26.425 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:26.425 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:26.425 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:26.425 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:26.425 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:26.425 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:26.425 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:26.425 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:26.425 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:26.425 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:26.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.411+0000 7fdce45ec640 10 monclient: shutdown 2024-09-11T01:50:26.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.411+0000 7fdce45ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:50:26.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.411+0000 7fdce45ec640 1 -- 172.21.15.12:0/2789998368 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcdc138230 msgr2=0x7fdcdc138630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:26.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.411+0000 7fdce45ec640 1 --2- 172.21.15.12:0/2789998368 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcdc138230 0x7fdcdc138630 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc0a8360 tx=0x7fdcd802a0d0 comp rx=0 tx=0).stop 2024-09-11T01:50:26.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.411+0000 7fdce45ec640 1 -- 172.21.15.12:0/2789998368 shutdown_connections 2024-09-11T01:50:26.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.411+0000 7fdce45ec640 1 --2- 172.21.15.12:0/2789998368 >> v2:172.21.15.71:3300/0 conn(0x7fdcdc134cb0 0x7fdcdc135120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:26.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.411+0000 7fdce45ec640 1 --2- 172.21.15.12:0/2789998368 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcdc138230 0x7fdcdc138630 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:26.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.411+0000 7fdce45ec640 1 --2- 172.21.15.12:0/2789998368 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdcdc137e50 0x7fdcdc09d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:26.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.411+0000 7fdce45ec640 1 -- 172.21.15.12:0/2789998368 >> 172.21.15.12:0/2789998368 conn(0x7fdcdc0a7020 msgr2=0x7fdcdc0a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:50:26.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.411+0000 7fdce45ec640 1 -- 172.21.15.12:0/2789998368 shutdown_connections 2024-09-11T01:50:26.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.411+0000 7fdce45ec640 1 -- 172.21.15.12:0/2789998368 wait complete. 2024-09-11T01:50:26.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.412+0000 7fdce45ec640 10 monclient: build_initial_monmap 2024-09-11T01:50:26.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.412+0000 7fdce45ec640 10 monclient: monmap: 2024-09-11T01:50:26.426 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:50:26.427 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:50:26.427 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:50:26.427 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:50:26.427 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:50:26.427 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:26.427 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:50:26.427 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:50:26.427 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:50:26.427 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:26.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.412+0000 7fdce45ec640 1 Processor -- start 2024-09-11T01:50:26.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.412+0000 7fdce45ec640 1 -- start start 2024-09-11T01:50:26.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.412+0000 7fdce45ec640 10 monclient: init 2024-09-11T01:50:26.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.412+0000 7fdce45ec640 10 monclient: _reopen_session rank -1 2024-09-11T01:50:26.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.412+0000 7fdce45ec640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:50:26.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.412+0000 7fdce45ec640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdcdc134cb0 0x7fdcdc1cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:26.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.412+0000 7fdce45ec640 10 monclient(hunting): picked mon.noname-c con 0x7fdcdc134cb0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:50:26.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.412+0000 7fdce45ec640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcdc137e50 0x7fdcdc1cd910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:26.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.412+0000 7fdce45ec640 10 monclient(hunting): picked mon.noname-a con 0x7fdcdc137e50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:50:26.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.412+0000 7fdce45ec640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcdc138230 0x7fdcdc121740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:26.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.412+0000 7fdce45ec640 10 monclient(hunting): picked mon.noname-b con 0x7fdcdc138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:50:26.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.412+0000 7fdce45ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:26.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.412+0000 7fdce45ec640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdcdc06f710 con 0x7fdcdc137e50 2024-09-11T01:50:26.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.412+0000 7fdce45ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:26.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.412+0000 7fdce45ec640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fdcdc138b70 con 0x7fdcdc138230 2024-09-11T01:50:26.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.412+0000 7fdce45ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:26.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.412+0000 7fdce45ec640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fdcdc06f890 con 0x7fdcdc134cb0 2024-09-11T01:50:26.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce45ec640 10 monclient(hunting): _renew_subs 2024-09-11T01:50:26.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce45ec640 10 monclient(hunting): authenticate will time out at 2032.856014s 2024-09-11T01:50:26.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce2388640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdcdc134cb0 msgr2=0x7fdcdc1cd3d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:50:26.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce2388640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdcdc134cb0 0x7fdcdc1cd3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:50:26.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce1b87640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcdc137e50 0x7fdcdc1cd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:26.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce1b87640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcdc137e50 0x7fdcdc1cd910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41484/0 (socket says 172.21.15.12:41484) 2024-09-11T01:50:26.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce1b87640 1 -- 172.21.15.12:0/569182470 learned_addr learned my addr 172.21.15.12:0/569182470 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:50:26.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce1b87640 10 monclient(hunting): get_auth_request con 0x7fdcdc137e50 auth_method 0 2024-09-11T01:50:26.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce1b87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:26.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce1b87640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:26.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce1b87640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:26.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce1b87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:26.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce1b87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:26.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce1b87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:26.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce2b89640 1 --2- 172.21.15.12:0/569182470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcdc138230 0x7fdcdc121740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:26.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce2b89640 10 monclient(hunting): get_auth_request con 0x7fdcdc138230 auth_method 0 2024-09-11T01:50:26.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce2b89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:26.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce2b89640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:26.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce2b89640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:26.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce1b87640 10 monclient(hunting): handle_auth_done global_id 214374 payload 824 2024-09-11T01:50:26.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce1b87640 1 -- 172.21.15.12:0/569182470 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdcdc134cb0 msgr2=0x7fdcdc1cd3d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:50:26.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce1b87640 1 --2- 172.21.15.12:0/569182470 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdcdc134cb0 0x7fdcdc1cd3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:26.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce1b87640 1 -- 172.21.15.12:0/569182470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcdc138230 msgr2=0x7fdcdc121740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:26.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce1b87640 1 --2- 172.21.15.12:0/569182470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcdc138230 0x7fdcdc121740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:26.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce1b87640 10 monclient: _finish_hunting 0 2024-09-11T01:50:26.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce1b87640 1 monclient: found mon.noname-a 2024-09-11T01:50:26.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce1b87640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:50:26.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce1b87640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:50:26.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.413+0000 7fdce1b87640 1 -- 172.21.15.12:0/569182470 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcdc123200 con 0x7fdcdc137e50 2024-09-11T01:50:26.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.415+0000 7fdce1b87640 1 -- 172.21.15.12:0/569182470 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcdc137e50 msgr2=0x7fdcdc1cd910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:26.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.415+0000 7fdce1b87640 1 --2- 172.21.15.12:0/569182470 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcdc137e50 0x7fdcdc1cd910 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8000c00 tx=0x7fdcd80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:26.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.415+0000 7fdcceffd640 1 -- 172.21.15.12:0/569182470 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8041030 con 0x7fdcdc137e50 2024-09-11T01:50:26.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.415+0000 7fdcceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:26.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.415+0000 7fdcceffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:50:26.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.415+0000 7fdcceffd640 10 monclient: dump: 2024-09-11T01:50:26.431 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:26.431 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:26.431 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:26.431 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:26.431 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:26.431 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:26.431 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:26.431 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:26.432 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:26.432 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:26.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.415+0000 7fdcceffd640 10 monclient: _finish_auth 0 2024-09-11T01:50:26.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.415+0000 7fdcceffd640 10 monclient: _check_auth_tickets 2024-09-11T01:50:26.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.415+0000 7fdcceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:50:26.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.415+0000 7fdcceffd640 1 -- 172.21.15.12:0/569182470 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcd803d6f0 con 0x7fdcdc137e50 2024-09-11T01:50:26.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.415+0000 7fdce45ec640 5 monclient: authenticate success, global_id 214374 2024-09-11T01:50:26.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.415+0000 7fdcceffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:50:26.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.415+0000 7fdcceffd640 1 -- 172.21.15.12:0/569182470 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8037040 con 0x7fdcdc137e50 2024-09-11T01:50:26.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.415+0000 7fdce45ec640 10 monclient: _renew_subs 2024-09-11T01:50:26.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.415+0000 7fdce45ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:50:26.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.415+0000 7fdce45ec640 1 -- 172.21.15.12:0/569182470 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcdc0a1710 con 0x7fdcdc137e50 2024-09-11T01:50:26.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.415+0000 7fdcceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:26.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.415+0000 7fdcceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:50:26.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.415+0000 7fdcceffd640 10 monclient: dump: 2024-09-11T01:50:26.433 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:26.433 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:26.433 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:26.433 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:26.433 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:26.433 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:26.433 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:26.433 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:26.433 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:26.433 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:26.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.415+0000 7fdce0b85640 4 set_mon_vals no callback set 2024-09-11T01:50:26.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.415+0000 7fdce45ec640 10 monclient: _renew_subs 2024-09-11T01:50:26.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.415+0000 7fdce45ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:50:26.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.415+0000 7fdce45ec640 1 -- 172.21.15.12:0/569182470 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcdc125f40 con 0x7fdcdc137e50 2024-09-11T01:50:26.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.416+0000 7fdce45ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:26.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.416+0000 7fdce45ec640 1 --2- 172.21.15.12:0/569182470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcdc121c80 0x7fdcdc122060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:26.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.416+0000 7fdce45ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:26.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.416+0000 7fdce45ec640 1 -- 172.21.15.12:0/569182470 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fdcdc06a060 con 0x7fdcdc121c80 2024-09-11T01:50:26.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.416+0000 7fdcceffd640 1 -- 172.21.15.12:0/569182470 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fdcd804e060 con 0x7fdcdc137e50 2024-09-11T01:50:26.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.416+0000 7fdce2388640 1 --2- 172.21.15.12:0/569182470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcdc121c80 0x7fdcdc122060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:26.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.416+0000 7fdce2388640 10 monclient: get_auth_request con 0x7fdcdc121c80 auth_method 0 2024-09-11T01:50:26.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.416+0000 7fdce2388640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:26.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.416+0000 7fdce2388640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:26.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.416+0000 7fdce2388640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:26.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.416+0000 7fdce2388640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:26.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.416+0000 7fdce2388640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:26.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.416+0000 7fdce2388640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:26.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.417+0000 7fdcceffd640 1 --2- 172.21.15.12:0/569182470 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdcc003e210 0x7fdcc00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:26.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.417+0000 7fdcceffd640 1 -- 172.21.15.12:0/569182470 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(707..707 src has 507..707) ==== 4787+0+0 (secure 0 0 0) 0x7fdcd8073210 con 0x7fdcdc137e50 2024-09-11T01:50:26.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.417+0000 7fdce2388640 10 monclient(hunting): handle_auth_done global_id 234127 payload 824 2024-09-11T01:50:26.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.417+0000 7fdce2388640 1 -- 172.21.15.12:0/569182470 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdcdc138630 con 0x7fdcdc121c80 2024-09-11T01:50:26.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.417+0000 7fdce2b89640 1 --2- 172.21.15.12:0/569182470 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdcc003e210 0x7fdcc00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:26.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.417+0000 7fdce2b89640 10 monclient: get_auth_request con 0x7fdcc003e210 auth_method 0 2024-09-11T01:50:26.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.419+0000 7fdce2388640 1 -- 172.21.15.12:0/569182470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcdc121c80 msgr2=0x7fdcdc122060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:26.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.419+0000 7fdce2388640 1 --2- 172.21.15.12:0/569182470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcdc121c80 0x7fdcdc122060 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0003980 tx=0x7fdcd000b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:26.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.419+0000 7fdcceffd640 1 -- 172.21.15.12:0/569182470 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0016070 con 0x7fdcdc121c80 2024-09-11T01:50:26.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.419+0000 7fdcceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:26.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.419+0000 7fdcceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:50:26.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.419+0000 7fdcceffd640 10 monclient: dump: 2024-09-11T01:50:26.435 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:26.435 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:26.436 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:26.436 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:26.436 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:26.436 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:26.436 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:26.436 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:26.436 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:26.436 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:26.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.419+0000 7fdce2b89640 1 --2- 172.21.15.12:0/569182470 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdcc003e210 0x7fdcc00406d0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc124ed0 tx=0x7fdcd40074a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:26.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.419+0000 7fdcceffd640 1 -- 172.21.15.12:0/569182470 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdcd0005270 con 0x7fdcdc121c80 2024-09-11T01:50:26.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.419+0000 7fdcceffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:50:26.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.419+0000 7fdcceffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:50:26.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.419+0000 7fdcceffd640 1 -- 172.21.15.12:0/569182470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcdc121c80 msgr2=0x7fdcdc122060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:26.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.419+0000 7fdcceffd640 1 --2- 172.21.15.12:0/569182470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcdc121c80 0x7fdcdc122060 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0003980 tx=0x7fdcd000b8e0 comp rx=0 tx=0).stop 2024-09-11T01:50:26.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.419+0000 7fdcceffd640 1 -- 172.21.15.12:0/569182470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcdc121c80 msgr2=0x7fdcdc122060 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:50:26.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.419+0000 7fdcceffd640 1 --2- 172.21.15.12:0/569182470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcdc121c80 0x7fdcdc122060 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0003980 tx=0x7fdcd000b8e0 comp rx=0 tx=0).stop 2024-09-11T01:50:26.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.425+0000 7fdce45ec640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:50:26.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.425+0000 7fdce45ec640 1 --2- 172.21.15.12:0/569182470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcdc1225a0 0x7fdcdc0a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:50:26.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.426+0000 7fdce45ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:50:26.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.426+0000 7fdce45ec640 1 -- 172.21.15.12:0/569182470 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fdcdc069950 con 0x7fdcdc1225a0 2024-09-11T01:50:26.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.426+0000 7fdce2388640 1 --2- 172.21.15.12:0/569182470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcdc1225a0 0x7fdcdc0a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:50:26.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.426+0000 7fdce2388640 10 monclient: get_auth_request con 0x7fdcdc1225a0 auth_method 0 2024-09-11T01:50:26.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.426+0000 7fdce2388640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:50:26.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.426+0000 7fdce2388640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:50:26.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.426+0000 7fdce2388640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:50:26.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.426+0000 7fdce2388640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:50:26.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.426+0000 7fdce2388640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:50:26.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.426+0000 7fdce2388640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:50:26.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.426+0000 7fdce2388640 10 monclient(hunting): handle_auth_done global_id 234130 payload 824 2024-09-11T01:50:26.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.426+0000 7fdce2388640 1 -- 172.21.15.12:0/569182470 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdcdc1cde50 con 0x7fdcdc1225a0 2024-09-11T01:50:26.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.428+0000 7fdce2388640 1 -- 172.21.15.12:0/569182470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcdc1225a0 msgr2=0x7fdcdc0a18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:50:26.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.428+0000 7fdce2388640 1 --2- 172.21.15.12:0/569182470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcdc1225a0 0x7fdcdc0a18e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0003980 tx=0x7fdcd00059d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:50:26.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.429+0000 7fdcceffd640 1 -- 172.21.15.12:0/569182470 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0016040 con 0x7fdcdc1225a0 2024-09-11T01:50:26.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.429+0000 7fdcceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:50:26.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.429+0000 7fdcceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:50:26.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.429+0000 7fdcceffd640 10 monclient: dump: 2024-09-11T01:50:26.440 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:50:26.440 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:50:26.440 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:26.440 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:50:26.440 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:50:26.440 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:50:26.440 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:50:26.440 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:50:26.440 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:50:26.440 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:50:26.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.429+0000 7fdcceffd640 1 -- 172.21.15.12:0/569182470 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4864 (secure 0 0 0) 0x7fdcd0021450 con 0x7fdcdc1225a0 2024-09-11T01:50:26.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.429+0000 7fdcceffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:50:26.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.429+0000 7fdcceffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:50:26.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.429+0000 7fdcceffd640 1 -- 172.21.15.12:0/569182470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcdc1225a0 msgr2=0x7fdcdc0a18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:26.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.429+0000 7fdcceffd640 1 --2- 172.21.15.12:0/569182470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcdc1225a0 0x7fdcdc0a18e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0003980 tx=0x7fdcd00059d0 comp rx=0 tx=0).stop 2024-09-11T01:50:26.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.429+0000 7fdcceffd640 1 -- 172.21.15.12:0/569182470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcdc1225a0 msgr2=0x7fdcdc0a18e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:50:26.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.429+0000 7fdcceffd640 1 --2- 172.21.15.12:0/569182470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcdc1225a0 0x7fdcdc0a18e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:50:26.441 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:50:26.441 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:50:26.441 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:50:26.441 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:50:26.441 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 196, 2024-09-11T01:50:26.441 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 28322, 2024-09-11T01:50:26.442 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:50:26.442 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:50:26.442 INFO:teuthology.orchestra.run.smithi012.stdout: 1 2024-09-11T01:50:26.442 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:26.442 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:50:26.442 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:50:26.442 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:50:26.442 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:50:26.442 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:50:26.442 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:50:26.442 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:50:26.442 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:50:26.442 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:50:26.442 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:50:26.443 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:50:26.443 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:50:26.443 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:50:26.443 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:50:26.443 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:50:26.443 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:26.443 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:50:26.443 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:50:26.443 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:50:26.443 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:50:26.443 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:50:26.444 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:50:26.444 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:50:26.444 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:50:26.444 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:50:26.444 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:50:26.444 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:50:26.444 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:50:26.444 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:50:26.444 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:26.444 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:26.444 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:50:26.444 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:50:26.444 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:50:26.445 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:50:26.445 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:50:26.445 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:50:26.445 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:50:26.445 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:50:26.445 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:50:26.445 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:50:26.445 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:50:26.445 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:50:26.445 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:50:26.445 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:50:26.445 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:50:26.445 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:50:26.445 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:50:26.446 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:50:26.446 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:50:26.446 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:50:26.446 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:50:26.446 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:50:26.446 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:50:26.446 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:50:26.446 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:50:26.446 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:50:26.446 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:50:26.446 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:50:26.446 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:26.446 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:50:26.446 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:26.446 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:50:26.447 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:26.447 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:50:26.447 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:50:26.447 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:50:26.447 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:50:26.447 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:26.447 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:50:26.447 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:50:26.447 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:26.447 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:26.447 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:26.447 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:50:26.447 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:50:26.447 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:26.448 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:26.448 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:26.448 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:26.448 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:50:26.448 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:50:26.448 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:50:26.448 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:50:26.448 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:50:26.448 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:26.448 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:26.448 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:50:26.448 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:50:26.448 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:50:26.448 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:50:26.449 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:26.449 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:50:26.449 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:50:26.449 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:26.449 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:26.449 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:26.449 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:50:26.449 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:50:26.449 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:26.449 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:26.450 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:26.450 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:26.450 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:50:26.450 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:50:26.450 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:50:26.450 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:50:26.450 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:50:26.450 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:26.450 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:26.450 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:50:26.450 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:50:26.450 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:50:26.450 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:50:26.450 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:26.450 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:50:26.451 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:50:26.451 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:26.451 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:26.451 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:26.451 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:50:26.451 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:50:26.451 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:50:26.451 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:26.451 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:26.451 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:26.451 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:50:26.451 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:50:26.451 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:50:26.451 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:50:26.452 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:50:26.452 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:26.452 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:26.452 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:26.452 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:50:26.452 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:50:26.452 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:26.452 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:50:26.452 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:50:26.452 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:50:26.452 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:26.452 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:26.452 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:50:26.453 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:26.453 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:50:26.453 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:50:26.453 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:50:26.453 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:26.453 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:26.453 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:50:26.453 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:26.453 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:50:26.453 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:50:26.453 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:50:26.453 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:26.453 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:50:26.453 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:50:26.453 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:50:26.454 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:50:26.454 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:50:26.454 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:50:26.454 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:50:26.454 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:50:26.454 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:50:26.454 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:50:26.454 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:50:26.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.429+0000 7fdce45ec640 1 -- 172.21.15.12:0/569182470 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdcc003e210 msgr2=0x7fdcc00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:26.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.429+0000 7fdce45ec640 1 --2- 172.21.15.12:0/569182470 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fdcc003e210 0x7fdcc00406d0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc124ed0 tx=0x7fdcd40074a0 comp rx=0 tx=0).stop 2024-09-11T01:50:26.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.429+0000 7fdce45ec640 10 monclient: shutdown 2024-09-11T01:50:26.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.429+0000 7fdce45ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:50:26.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.429+0000 7fdce45ec640 1 -- 172.21.15.12:0/569182470 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcdc137e50 msgr2=0x7fdcdc1cd910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:50:26.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.429+0000 7fdce45ec640 1 --2- 172.21.15.12:0/569182470 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdcdc137e50 0x7fdcdc1cd910 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8000c00 tx=0x7fdcd80046c0 comp rx=0 tx=0).stop 2024-09-11T01:50:26.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.429+0000 7fdce2b89640 1 -- 172.21.15.12:0/569182470 reap_dead start 2024-09-11T01:50:26.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.429+0000 7fdce2b89640 1 -- 172.21.15.12:0/569182470 reap_dead start 2024-09-11T01:50:26.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.429+0000 7fdce45ec640 1 -- 172.21.15.12:0/569182470 shutdown_connections 2024-09-11T01:50:26.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.429+0000 7fdce45ec640 1 -- 172.21.15.12:0/569182470 >> 172.21.15.12:0/569182470 conn(0x7fdcdc0a7020 msgr2=0x7fdcdc135a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:50:26.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.430+0000 7fdce45ec640 1 -- 172.21.15.12:0/569182470 shutdown_connections 2024-09-11T01:50:26.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:50:26.430+0000 7fdce45ec640 1 -- 172.21.15.12:0/569182470 wait complete. 2024-09-11T01:50:26.457 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:50:26.457 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:50:26.870 INFO:tasks.rados.rados.0.smithi071.stdout:3166: done (2 left) 2024-09-11T01:50:26.870 INFO:tasks.rados.rados.0.smithi071.stdout:3167: write oid 26 current snap is 354 2024-09-11T01:50:26.870 INFO:tasks.rados.rados.0.smithi071.stdout:3167: seq_num 1112 ranges {689384=664426,1971506=687726} 2024-09-11T01:50:26.877 INFO:tasks.rados.rados.0.smithi071.stdout:3167: writing smithi07134777-26 from 689384 to 1353810 tid 1 2024-09-11T01:50:26.885 INFO:tasks.rados.rados.0.smithi071.stdout:3167: writing smithi07134777-26 from 1971506 to 2659232 tid 2 2024-09-11T01:50:26.885 INFO:tasks.rados.rados.0.smithi071.stdout:3168: snap_create 2024-09-11T01:50:26.885 INFO:tasks.rados.rados.0.smithi071.stdout:3163: finishing write tid 2 to smithi07134777-27 2024-09-11T01:50:26.885 INFO:tasks.rados.rados.0.smithi071.stdout:3163: finishing write tid 3 to smithi07134777-27 2024-09-11T01:50:26.885 INFO:tasks.rados.rados.0.smithi071.stdout:3163: finishing write tid 4 to smithi07134777-27 2024-09-11T01:50:26.885 INFO:tasks.rados.rados.0.smithi071.stdout:3163: oid 27 updating version 0 to 2174 2024-09-11T01:50:26.885 INFO:tasks.rados.rados.0.smithi071.stdout:3163: oid 27 version 2174 is already newer than 2172 2024-09-11T01:50:26.885 INFO:tasks.rados.rados.0.smithi071.stdout:3163: oid 27 version 2174 is already newer than 2173 2024-09-11T01:50:26.885 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 2174 (ObjNum 1110 snap 354 seq_num 1110) dirty exists 2024-09-11T01:50:26.885 INFO:tasks.rados.rados.0.smithi071.stdout:3163: left oid 27 (ObjNum 1110 snap 354 seq_num 1110) 2024-09-11T01:50:26.885 INFO:tasks.rados.rados.0.smithi071.stdout:3164: finishing rollback tid 0 to smithi07134777-7 2024-09-11T01:50:26.885 INFO:tasks.rados.rados.0.smithi071.stdout:3164: finishing rollback tid 1 to smithi07134777-7 2024-09-11T01:50:26.886 INFO:tasks.rados.rados.0.smithi071.stdout:3164: finishing rollback tid 2 to smithi07134777-7 2024-09-11T01:50:26.886 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 2318 (ObjNum 1014 snap 322 seq_num 1014) dirty exists 2024-09-11T01:50:26.892 INFO:tasks.rados.rados.0.smithi071.stdout:3167: finishing write tid 1 to smithi07134777-26 2024-09-11T01:50:26.892 INFO:tasks.rados.rados.0.smithi071.stdout:3167: finishing write tid 2 to smithi07134777-26 2024-09-11T01:50:26.893 INFO:tasks.rados.rados.0.smithi071.stdout:3167: finishing write tid 3 to smithi07134777-26 2024-09-11T01:50:26.893 INFO:tasks.rados.rados.0.smithi071.stdout:3167: finishing write tid 4 to smithi07134777-26 2024-09-11T01:50:26.893 INFO:tasks.rados.rados.0.smithi071.stdout:3167: oid 26 updating version 0 to 2176 2024-09-11T01:50:26.893 INFO:tasks.rados.rados.0.smithi071.stdout:3167: oid 26 version 2176 is already newer than 2175 2024-09-11T01:50:26.893 INFO:tasks.rados.rados.0.smithi071.stdout:3167: oid 26 updating version 2176 to 2177 2024-09-11T01:50:26.893 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 2177 (ObjNum 1112 snap 354 seq_num 1112) dirty exists 2024-09-11T01:50:26.893 INFO:tasks.rados.rados.0.smithi071.stdout:3167: left oid 26 (ObjNum 1112 snap 354 seq_num 1112) 2024-09-11T01:50:27.876 INFO:tasks.rados.rados.0.smithi071.stdout:3163: done (3 left) 2024-09-11T01:50:27.876 INFO:tasks.rados.rados.0.smithi071.stdout:3164: done (2 left) 2024-09-11T01:50:27.876 INFO:tasks.rados.rados.0.smithi071.stdout:3167: done (1 left) 2024-09-11T01:50:27.876 INFO:tasks.rados.rados.0.smithi071.stdout:3168: done (0 left) 2024-09-11T01:50:27.876 INFO:tasks.rados.rados.0.smithi071.stdout:3169: read oid 30 snap 354 2024-09-11T01:50:27.876 INFO:tasks.rados.rados.0.smithi071.stdout:3169: expect (ObjNum 1083 snap 346 seq_num 1083) 2024-09-11T01:50:27.876 INFO:tasks.rados.rados.0.smithi071.stdout:3170: snap_remove snap 329 2024-09-11T01:50:28.880 INFO:tasks.rados.rados.0.smithi071.stdout:3170: done (1 left) 2024-09-11T01:50:28.880 INFO:tasks.rados.rados.0.smithi071.stdout:3171: read oid 16 snap -1 2024-09-11T01:50:28.880 INFO:tasks.rados.rados.0.smithi071.stdout:3171: expect (ObjNum 1062 snap 340 seq_num 1062) 2024-09-11T01:50:28.880 INFO:tasks.rados.rados.0.smithi071.stdout:3172: write (excl) oid 49 current snap is 355 2024-09-11T01:50:28.880 INFO:tasks.rados.rados.0.smithi071.stdout:3172: seq_num 1113 ranges {706191=739675,1913635=431384} 2024-09-11T01:50:28.888 INFO:tasks.rados.rados.0.smithi071.stdout:3172: writing smithi07134777-49 from 706191 to 1445866 tid 1 2024-09-11T01:50:28.893 INFO:tasks.rados.rados.0.smithi071.stdout:3172: writing smithi07134777-49 from 1913635 to 2345019 tid 2 2024-09-11T01:50:28.894 INFO:tasks.rados.rados.0.smithi071.stdout:3173: delete oid 46 current snap is 355 2024-09-11T01:50:28.894 INFO:tasks.rados.rados.0.smithi071.stdout:3169: expect (ObjNum 1083 snap 346 seq_num 1083) 2024-09-11T01:50:28.894 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 403477 2024-09-11T01:50:28.900 INFO:tasks.ceph.osd.1.smithi012.stderr:2024-09-11T01:50:28.897+0000 7f11d54b0640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 152 after trimming was started 2024-09-11T01:50:28.943 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 622847 2024-09-11T01:50:28.976 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:50:29.018 INFO:tasks.rados.rados.0.smithi071.stdout:3171: expect (ObjNum 1062 snap 340 seq_num 1062) 2024-09-11T01:50:29.024 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 589569 2024-09-11T01:50:29.132 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3094192 2024-09-11T01:50:29.218 INFO:tasks.rados.rados.0.smithi071.stdout:3172: finishing write tid 1 to smithi07134777-49 2024-09-11T01:50:29.218 INFO:tasks.rados.rados.0.smithi071.stdout:3172: finishing write tid 2 to smithi07134777-49 2024-09-11T01:50:29.218 INFO:tasks.rados.rados.0.smithi071.stdout:3172: finishing write tid 3 to smithi07134777-49 2024-09-11T01:50:29.218 INFO:tasks.rados.rados.0.smithi071.stdout:3172: finishing write tid 4 to smithi07134777-49 2024-09-11T01:50:29.218 INFO:tasks.rados.rados.0.smithi071.stdout:3172: oid 49 updating version 0 to 394 2024-09-11T01:50:29.218 INFO:tasks.rados.rados.0.smithi071.stdout:3172: oid 49 updating version 394 to 395 2024-09-11T01:50:29.218 INFO:tasks.rados.rados.0.smithi071.stdout:3172: oid 49 version 395 is already newer than 393 2024-09-11T01:50:29.218 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 395 (ObjNum 1113 snap 355 seq_num 1113) dirty exists 2024-09-11T01:50:29.218 INFO:tasks.rados.rados.0.smithi071.stdout:3172: left oid 49 (ObjNum 1113 snap 355 seq_num 1113) 2024-09-11T01:50:29.219 INFO:tasks.rados.rados.0.smithi071.stdout:3169: done (3 left) 2024-09-11T01:50:29.219 INFO:tasks.rados.rados.0.smithi071.stdout:3171: done (2 left) 2024-09-11T01:50:29.219 INFO:tasks.rados.rados.0.smithi071.stdout:3172: done (1 left) 2024-09-11T01:50:29.219 INFO:tasks.rados.rados.0.smithi071.stdout:3173: done (0 left) 2024-09-11T01:50:29.219 INFO:tasks.rados.rados.0.smithi071.stdout:3174: read oid 21 snap -1 2024-09-11T01:50:29.219 INFO:tasks.rados.rados.0.smithi071.stdout:3174: expect (ObjNum 1080 snap 346 seq_num 1080) 2024-09-11T01:50:29.219 INFO:tasks.rados.rados.0.smithi071.stdout:3175: snap_create 2024-09-11T01:50:29.224 INFO:tasks.rados.rados.0.smithi071.stdout:3174: expect (ObjNum 1080 snap 346 seq_num 1080) 2024-09-11T01:50:29.224 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 8963 2024-09-11T01:50:29.273 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1477322 2024-09-11T01:50:29.302 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1128729 2024-09-11T01:50:29.885 INFO:tasks.rados.rados.0.smithi071.stdout:3174: done (1 left) 2024-09-11T01:50:29.885 INFO:tasks.rados.rados.0.smithi071.stdout:3175: done (0 left) 2024-09-11T01:50:29.885 INFO:tasks.rados.rados.0.smithi071.stdout:3176: read oid 40 snap -1 2024-09-11T01:50:29.886 INFO:tasks.rados.rados.0.smithi071.stdout:3176: expect deleted 2024-09-11T01:50:29.886 INFO:tasks.rados.rados.0.smithi071.stdout:3177: copy_from oid 33 from oid 17 current snap is 356 2024-09-11T01:50:29.886 INFO:tasks.rados.rados.0.smithi071.stdout:3178: snap_remove snap 346 2024-09-11T01:50:30.858 INFO:tasks.rados.rados.0.smithi071.stdout:3178: done (2 left) 2024-09-11T01:50:30.858 INFO:tasks.rados.rados.0.smithi071.stdout:3179: rollback oid 29 current snap is 356 2024-09-11T01:50:30.859 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 29 to 343 2024-09-11T01:50:30.859 INFO:tasks.rados.rados.0.smithi071.stdout:3180: write oid 11 current snap is 356 2024-09-11T01:50:30.859 INFO:tasks.rados.rados.0.smithi071.stdout:3180: seq_num 1115 ranges {739805=490173,1630599=435053,2636284=103521} 2024-09-11T01:50:30.866 INFO:tasks.rados.rados.0.smithi071.stdout:3180: writing smithi07134777-11 from 739805 to 1229978 tid 1 2024-09-11T01:50:30.871 INFO:tasks.rados.rados.0.smithi071.stdout:3180: writing smithi07134777-11 from 1630599 to 2065652 tid 2 2024-09-11T01:50:30.876 INFO:tasks.rados.rados.0.smithi071.stdout:3180: writing smithi07134777-11 from 2636284 to 2739805 tid 3 2024-09-11T01:50:30.876 INFO:tasks.rados.rados.0.smithi071.stdout:3181: delete oid 14 current snap is 356 2024-09-11T01:50:30.876 INFO:tasks.rados.rados.0.smithi071.stdout:3177: finishing copy_from to smithi07134777-33 2024-09-11T01:50:30.876 INFO:tasks.rados.rados.0.smithi071.stdout:3177: got expected ENOENT (src dne) 2024-09-11T01:50:30.876 INFO:tasks.rados.rados.0.smithi071.stdout:3177: finishing copy_from racing read to smithi07134777-33 2024-09-11T01:50:30.876 INFO:tasks.rados.rados.0.smithi071.stdout:3179: finishing rollback tid 1 to smithi07134777-29 2024-09-11T01:50:30.876 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 1947 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-11T01:50:30.876 INFO:tasks.rados.rados.0.smithi071.stdout:3180: finishing write tid 1 to smithi07134777-11 2024-09-11T01:50:30.879 INFO:tasks.rados.rados.0.smithi071.stdout:3176: done (4 left) 2024-09-11T01:50:30.879 INFO:tasks.rados.rados.0.smithi071.stdout:3177: done (3 left) 2024-09-11T01:50:30.879 INFO:tasks.rados.rados.0.smithi071.stdout:3179: done (2 left) 2024-09-11T01:50:30.879 INFO:tasks.rados.rados.0.smithi071.stdout:3181: done (1 left) 2024-09-11T01:50:30.879 INFO:tasks.rados.rados.0.smithi071.stdout:3182: rollback oid 31 current snap is 356 2024-09-11T01:50:30.879 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 31 to 271 2024-09-11T01:50:30.879 INFO:tasks.rados.rados.0.smithi071.stdout:3183: copy_from oid 18 from oid 37 current snap is 356 2024-09-11T01:50:30.879 INFO:tasks.rados.rados.0.smithi071.stdout:3184: delete oid 5 current snap is 356 2024-09-11T01:50:30.880 INFO:tasks.rados.rados.0.smithi071.stdout:3183: finishing copy_from to smithi07134777-18 2024-09-11T01:50:30.880 INFO:tasks.rados.rados.0.smithi071.stdout:3183: got expected ENOENT (src dne) 2024-09-11T01:50:30.880 INFO:tasks.rados.rados.0.smithi071.stdout:3183: finishing copy_from racing read to smithi07134777-18 2024-09-11T01:50:30.882 INFO:tasks.rados.rados.0.smithi071.stdout:3183: done (3 left) 2024-09-11T01:50:30.882 INFO:tasks.rados.rados.0.smithi071.stdout:3184: done (2 left) 2024-09-11T01:50:30.883 INFO:tasks.rados.rados.0.smithi071.stdout:3185: read oid 9 snap 338 2024-09-11T01:50:30.883 INFO:tasks.rados.rados.0.smithi071.stdout:3185: expect (ObjNum 601 snap 172 seq_num 601) 2024-09-11T01:50:30.883 INFO:tasks.rados.rados.0.smithi071.stdout:3186: read oid 8 snap 271 2024-09-11T01:50:30.883 INFO:tasks.rados.rados.0.smithi071.stdout:3186: expect deleted 2024-09-11T01:50:30.883 INFO:tasks.rados.rados.0.smithi071.stdout:3187: rollback oid 12 current snap is 356 2024-09-11T01:50:30.883 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 12 to 352 2024-09-11T01:50:30.883 INFO:tasks.rados.rados.0.smithi071.stdout:3188: read oid 42 snap -1 2024-09-11T01:50:30.883 INFO:tasks.rados.rados.0.smithi071.stdout:3188: expect (ObjNum 1063 snap 341 seq_num 1063) 2024-09-11T01:50:30.883 INFO:tasks.rados.rados.0.smithi071.stdout:3189: read oid 43 snap 323 2024-09-11T01:50:30.883 INFO:tasks.rados.rados.0.smithi071.stdout:3189: expect (ObjNum 1018 snap 323 seq_num 1018) 2024-09-11T01:50:30.883 INFO:tasks.rados.rados.0.smithi071.stdout:3190: snap_create 2024-09-11T01:50:30.883 INFO:tasks.rados.rados.0.smithi071.stdout:3180: finishing write tid 2 to smithi07134777-11 2024-09-11T01:50:30.884 INFO:tasks.rados.rados.0.smithi071.stdout:3180: finishing write tid 3 to smithi07134777-11 2024-09-11T01:50:30.885 INFO:tasks.rados.rados.0.smithi071.stdout:3182: finishing rollback tid 0 to smithi07134777-31 2024-09-11T01:50:30.885 INFO:tasks.rados.rados.0.smithi071.stdout:3180: finishing write tid 4 to smithi07134777-11 2024-09-11T01:50:30.886 INFO:tasks.rados.rados.0.smithi071.stdout:3180: finishing write tid 5 to smithi07134777-11 2024-09-11T01:50:30.886 INFO:tasks.rados.rados.0.smithi071.stdout:3180: oid 11 updating version 0 to 2176 2024-09-11T01:50:30.886 INFO:tasks.rados.rados.0.smithi071.stdout:3180: oid 11 version 2176 is already newer than 2174 2024-09-11T01:50:30.886 INFO:tasks.rados.rados.0.smithi071.stdout:3180: oid 11 version 2176 is already newer than 2175 2024-09-11T01:50:30.886 INFO:tasks.rados.rados.0.smithi071.stdout:3180: oid 11 version 2176 is already newer than 2173 2024-09-11T01:50:30.886 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 2176 (ObjNum 1115 snap 356 seq_num 1115) dirty exists 2024-09-11T01:50:30.887 INFO:tasks.rados.rados.0.smithi071.stdout:3180: left oid 11 (ObjNum 1115 snap 356 seq_num 1115) 2024-09-11T01:50:30.887 INFO:tasks.rados.rados.0.smithi071.stdout:3182: finishing rollback tid 1 to smithi07134777-31 2024-09-11T01:50:30.887 INFO:tasks.rados.rados.0.smithi071.stdout:3182: finishing rollback tid 2 to smithi07134777-31 2024-09-11T01:50:30.887 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 1813 (ObjNum 737 snap 226 seq_num 737) dirty exists 2024-09-11T01:50:30.892 INFO:tasks.rados.rados.0.smithi071.stdout:3188: expect (ObjNum 1063 snap 341 seq_num 1063) 2024-09-11T01:50:31.136 INFO:tasks.rados.rados.0.smithi071.stdout:3185: expect (ObjNum 601 snap 172 seq_num 601) 2024-09-11T01:50:31.143 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1416142 2024-09-11T01:50:31.204 INFO:tasks.rados.rados.0.smithi071.stdout:3187: finishing rollback tid 0 to smithi07134777-12 2024-09-11T01:50:31.205 INFO:tasks.rados.rados.0.smithi071.stdout:3187: finishing rollback tid 1 to smithi07134777-12 2024-09-11T01:50:31.205 INFO:tasks.rados.rados.0.smithi071.stdout:3187: finishing rollback tid 2 to smithi07134777-12 2024-09-11T01:50:31.205 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 2180 (ObjNum 1105 snap 352 seq_num 1105) dirty exists 2024-09-11T01:50:31.205 INFO:tasks.rados.rados.0.smithi071.stdout:3189: expect (ObjNum 1018 snap 323 seq_num 1018) 2024-09-11T01:50:31.243 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3004646 2024-09-11T01:50:31.283 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3007843 2024-09-11T01:50:31.291 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 929372 2024-09-11T01:50:31.890 INFO:tasks.rados.rados.0.smithi071.stdout:3180: done (7 left) 2024-09-11T01:50:31.891 INFO:tasks.rados.rados.0.smithi071.stdout:3182: done (6 left) 2024-09-11T01:50:31.891 INFO:tasks.rados.rados.0.smithi071.stdout:3185: done (5 left) 2024-09-11T01:50:31.891 INFO:tasks.rados.rados.0.smithi071.stdout:3186: done (4 left) 2024-09-11T01:50:31.891 INFO:tasks.rados.rados.0.smithi071.stdout:3187: done (3 left) 2024-09-11T01:50:31.891 INFO:tasks.rados.rados.0.smithi071.stdout:3188: done (2 left) 2024-09-11T01:50:31.891 INFO:tasks.rados.rados.0.smithi071.stdout:3189: done (1 left) 2024-09-11T01:50:31.891 INFO:tasks.rados.rados.0.smithi071.stdout:3190: done (0 left) 2024-09-11T01:50:31.891 INFO:tasks.rados.rados.0.smithi071.stdout:3191: write oid 24 current snap is 357 2024-09-11T01:50:31.891 INFO:tasks.rados.rados.0.smithi071.stdout:3191: seq_num 1118 ranges {790226=432273,1806045=582380} 2024-09-11T01:50:31.897 INFO:tasks.rados.rados.0.smithi071.stdout:3191: writing smithi07134777-24 from 790226 to 1222499 tid 1 2024-09-11T01:50:31.904 INFO:tasks.rados.rados.0.smithi071.stdout:3191: writing smithi07134777-24 from 1806045 to 2388425 tid 2 2024-09-11T01:50:31.904 INFO:tasks.rados.rados.0.smithi071.stdout:3192: rollback oid 44 current snap is 357 2024-09-11T01:50:31.904 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 44 to 307 2024-09-11T01:50:31.904 INFO:tasks.rados.rados.0.smithi071.stdout:3193: write oid 29 current snap is 357 2024-09-11T01:50:31.904 INFO:tasks.rados.rados.0.smithi071.stdout:3193: seq_num 1119 ranges {407033=507522,1464527=442391,2345976=461057} 2024-09-11T01:50:31.909 INFO:tasks.rados.rados.0.smithi071.stdout:3193: writing smithi07134777-29 from 407033 to 914555 tid 1 2024-09-11T01:50:31.914 INFO:tasks.rados.rados.0.smithi071.stdout:3193: writing smithi07134777-29 from 1464527 to 1906918 tid 2 2024-09-11T01:50:31.920 INFO:tasks.rados.rados.0.smithi071.stdout:3193: writing smithi07134777-29 from 2345976 to 2807033 tid 3 2024-09-11T01:50:31.921 INFO:tasks.rados.rados.0.smithi071.stdout:3194: copy_from oid 6 from oid 8 current snap is 357 2024-09-11T01:50:31.921 INFO:tasks.rados.rados.0.smithi071.stdout:3192: finishing rollback tid 1 to smithi07134777-44 2024-09-11T01:50:31.921 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 1073 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-11T01:50:31.921 INFO:tasks.rados.rados.0.smithi071.stdout:3191: finishing write tid 1 to smithi07134777-24 2024-09-11T01:50:31.921 INFO:tasks.rados.rados.0.smithi071.stdout:3191: finishing write tid 2 to smithi07134777-24 2024-09-11T01:50:31.921 INFO:tasks.rados.rados.0.smithi071.stdout:3191: finishing write tid 3 to smithi07134777-24 2024-09-11T01:50:31.921 INFO:tasks.rados.rados.0.smithi071.stdout:3191: finishing write tid 4 to smithi07134777-24 2024-09-11T01:50:31.921 INFO:tasks.rados.rados.0.smithi071.stdout:3191: oid 24 updating version 0 to 2061 2024-09-11T01:50:31.921 INFO:tasks.rados.rados.0.smithi071.stdout:3191: oid 24 updating version 2061 to 2062 2024-09-11T01:50:31.921 INFO:tasks.rados.rados.0.smithi071.stdout:3191: oid 24 version 2062 is already newer than 2060 2024-09-11T01:50:31.922 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 2062 (ObjNum 1118 snap 357 seq_num 1118) dirty exists 2024-09-11T01:50:31.922 INFO:tasks.rados.rados.0.smithi071.stdout:3191: left oid 24 (ObjNum 1118 snap 357 seq_num 1118) 2024-09-11T01:50:31.922 INFO:tasks.rados.rados.0.smithi071.stdout:3193: finishing write tid 1 to smithi07134777-29 2024-09-11T01:50:31.922 INFO:tasks.rados.rados.0.smithi071.stdout:3193: finishing write tid 2 to smithi07134777-29 2024-09-11T01:50:31.922 INFO:tasks.rados.rados.0.smithi071.stdout:3191: done (3 left) 2024-09-11T01:50:31.922 INFO:tasks.rados.rados.0.smithi071.stdout:3192: done (2 left) 2024-09-11T01:50:31.922 INFO:tasks.rados.rados.0.smithi071.stdout:3195: copy_from oid 42 from oid 44 current snap is 357 2024-09-11T01:50:31.922 INFO:tasks.rados.rados.0.smithi071.stdout:3196: read oid 4 snap -1 2024-09-11T01:50:31.922 INFO:tasks.rados.rados.0.smithi071.stdout:3196: expect deleted 2024-09-11T01:50:31.922 INFO:tasks.rados.rados.0.smithi071.stdout:3197: delete oid 28 current snap is 357 2024-09-11T01:50:31.923 INFO:tasks.rados.rados.0.smithi071.stdout:3195: finishing copy_from to smithi07134777-42 2024-09-11T01:50:31.923 INFO:tasks.rados.rados.0.smithi071.stdout:3195: got expected ENOENT (src dne) 2024-09-11T01:50:31.923 INFO:tasks.rados.rados.0.smithi071.stdout:3195: finishing copy_from racing read to smithi07134777-42 2024-09-11T01:50:31.924 INFO:tasks.rados.rados.0.smithi071.stdout:3194: finishing copy_from to smithi07134777-6 2024-09-11T01:50:31.924 INFO:tasks.rados.rados.0.smithi071.stdout:3194: got expected ENOENT (src dne) 2024-09-11T01:50:31.924 INFO:tasks.rados.rados.0.smithi071.stdout:3194: finishing copy_from racing read to smithi07134777-6 2024-09-11T01:50:31.924 INFO:tasks.rados.rados.0.smithi071.stdout:3194: done (4 left) 2024-09-11T01:50:31.924 INFO:tasks.rados.rados.0.smithi071.stdout:3195: done (3 left) 2024-09-11T01:50:31.924 INFO:tasks.rados.rados.0.smithi071.stdout:3196: done (2 left) 2024-09-11T01:50:31.924 INFO:tasks.rados.rados.0.smithi071.stdout:3197: done (1 left) 2024-09-11T01:50:31.924 INFO:tasks.rados.rados.0.smithi071.stdout:3198: read oid 21 snap -1 2024-09-11T01:50:31.924 INFO:tasks.rados.rados.0.smithi071.stdout:3198: expect (ObjNum 1080 snap 346 seq_num 1080) 2024-09-11T01:50:31.924 INFO:tasks.rados.rados.0.smithi071.stdout:3199: write (excl) oid 5 current snap is 357 2024-09-11T01:50:31.924 INFO:tasks.rados.rados.0.smithi071.stdout:3199: seq_num 1121 ranges {440647=658020,1581491=446060,2800822=39825} 2024-09-11T01:50:31.929 INFO:tasks.rados.rados.0.smithi071.stdout:3199: writing smithi07134777-5 from 440647 to 1098667 tid 1 2024-09-11T01:50:31.934 INFO:tasks.rados.rados.0.smithi071.stdout:3199: writing smithi07134777-5 from 1581491 to 2027551 tid 2 2024-09-11T01:50:31.941 INFO:tasks.rados.rados.0.smithi071.stdout:3199: writing smithi07134777-5 from 2800822 to 2840647 tid 3 2024-09-11T01:50:31.941 INFO:tasks.rados.rados.0.smithi071.stdout:3200: write (excl) oid 20 current snap is 357 2024-09-11T01:50:31.941 INFO:tasks.rados.rados.0.smithi071.stdout:3200: seq_num 1122 ranges {457454=733269,1923620=589718} 2024-09-11T01:50:31.945 INFO:tasks.rados.rados.0.smithi071.stdout:3200: writing smithi07134777-20 from 457454 to 1190723 tid 1 2024-09-11T01:50:31.953 INFO:tasks.rados.rados.0.smithi071.stdout:3200: writing smithi07134777-20 from 1923620 to 2513338 tid 2 2024-09-11T01:50:31.953 INFO:tasks.rados.rados.0.smithi071.stdout:3201: write (excl) oid 34 current snap is 357 2024-09-11T01:50:31.953 INFO:tasks.rados.rados.0.smithi071.stdout:3193: finishing write tid 3 to smithi07134777-29 2024-09-11T01:50:31.953 INFO:tasks.rados.rados.0.smithi071.stdout:3201: seq_num 1123 ranges {474261=408518,1582102=449729,2739315=134946} 2024-09-11T01:50:31.958 INFO:tasks.rados.rados.0.smithi071.stdout:3201: writing smithi07134777-34 from 474261 to 882779 tid 1 2024-09-11T01:50:31.964 INFO:tasks.rados.rados.0.smithi071.stdout:3201: writing smithi07134777-34 from 1582102 to 2031831 tid 2 2024-09-11T01:50:31.969 INFO:tasks.rados.rados.0.smithi071.stdout:3201: writing smithi07134777-34 from 2739315 to 2874261 tid 3 2024-09-11T01:50:31.969 INFO:tasks.rados.rados.0.smithi071.stdout:3202: write (excl) oid 25 current snap is 357 2024-09-11T01:50:31.969 INFO:tasks.rados.rados.0.smithi071.stdout:3193: finishing write tid 4 to smithi07134777-29 2024-09-11T01:50:31.969 INFO:tasks.rados.rados.0.smithi071.stdout:3193: finishing write tid 5 to smithi07134777-29 2024-09-11T01:50:31.969 INFO:tasks.rados.rados.0.smithi071.stdout:3193: oid 29 updating version 0 to 2345 2024-09-11T01:50:31.970 INFO:tasks.rados.rados.0.smithi071.stdout:3193: oid 29 version 2345 is already newer than 2344 2024-09-11T01:50:31.970 INFO:tasks.rados.rados.0.smithi071.stdout:3193: oid 29 updating version 2345 to 2346 2024-09-11T01:50:31.970 INFO:tasks.rados.rados.0.smithi071.stdout:3193: oid 29 updating version 2346 to 2347 2024-09-11T01:50:31.970 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 2347 (ObjNum 1119 snap 357 seq_num 1119) dirty exists 2024-09-11T01:50:31.970 INFO:tasks.rados.rados.0.smithi071.stdout:3193: left oid 29 (ObjNum 1119 snap 357 seq_num 1119) 2024-09-11T01:50:31.970 INFO:tasks.rados.rados.0.smithi071.stdout:3202: seq_num 1124 ranges {491068=483767,1640584=593387,2650385=240683} 2024-09-11T01:50:31.974 INFO:tasks.rados.rados.0.smithi071.stdout:3202: writing smithi07134777-25 from 491068 to 974835 tid 1 2024-09-11T01:50:31.981 INFO:tasks.rados.rados.0.smithi071.stdout:3202: writing smithi07134777-25 from 1640584 to 2233971 tid 2 2024-09-11T01:50:31.984 INFO:tasks.rados.rados.0.smithi071.stdout:3202: writing smithi07134777-25 from 2650385 to 2891068 tid 3 2024-09-11T01:50:31.984 INFO:tasks.rados.rados.0.smithi071.stdout:3193: done (5 left) 2024-09-11T01:50:31.985 INFO:tasks.rados.rados.0.smithi071.stdout:3203: delete oid 45 current snap is 357 2024-09-11T01:50:31.985 INFO:tasks.rados.rados.0.smithi071.stdout:3198: expect (ObjNum 1080 snap 346 seq_num 1080) 2024-09-11T01:50:31.994 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1340442 2024-09-11T01:50:32.021 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1257447 2024-09-11T01:50:32.049 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:50:32.088 INFO:tasks.rados.rados.0.smithi071.stdout:3199: finishing write tid 1 to smithi07134777-5 2024-09-11T01:50:32.088 INFO:tasks.rados.rados.0.smithi071.stdout:3198: done (5 left) 2024-09-11T01:50:32.088 INFO:tasks.rados.rados.0.smithi071.stdout:3203: done (4 left) 2024-09-11T01:50:32.088 INFO:tasks.rados.rados.0.smithi071.stdout:3204: snap_remove snap 355 2024-09-11T01:50:32.892 INFO:tasks.rados.rados.0.smithi071.stdout:3204: done (4 left) 2024-09-11T01:50:32.893 INFO:tasks.rados.rados.0.smithi071.stdout:3205: copy_from oid 11 from oid 48 current snap is 357 2024-09-11T01:50:32.893 INFO:tasks.rados.rados.0.smithi071.stdout:3199: finishing write tid 2 to smithi07134777-5 2024-09-11T01:50:32.893 INFO:tasks.rados.rados.0.smithi071.stdout:3199: finishing write tid 3 to smithi07134777-5 2024-09-11T01:50:32.893 INFO:tasks.rados.rados.0.smithi071.stdout:3199: finishing write tid 4 to smithi07134777-5 2024-09-11T01:50:32.893 INFO:tasks.rados.rados.0.smithi071.stdout:3199: finishing write tid 5 to smithi07134777-5 2024-09-11T01:50:32.893 INFO:tasks.rados.rados.0.smithi071.stdout:3199: oid 5 updating version 0 to 1182 2024-09-11T01:50:32.893 INFO:tasks.rados.rados.0.smithi071.stdout:3199: oid 5 updating version 1182 to 1184 2024-09-11T01:50:32.893 INFO:tasks.rados.rados.0.smithi071.stdout:3199: oid 5 version 1184 is already newer than 1183 2024-09-11T01:50:32.893 INFO:tasks.rados.rados.0.smithi071.stdout:3199: oid 5 updating version 1184 to 1185 2024-09-11T01:50:32.893 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 1185 (ObjNum 1121 snap 357 seq_num 1121) dirty exists 2024-09-11T01:50:32.893 INFO:tasks.rados.rados.0.smithi071.stdout:3199: left oid 5 (ObjNum 1121 snap 357 seq_num 1121) 2024-09-11T01:50:32.893 INFO:tasks.rados.rados.0.smithi071.stdout:3200: finishing write tid 1 to smithi07134777-20 2024-09-11T01:50:32.894 INFO:tasks.rados.rados.0.smithi071.stdout:3200: finishing write tid 2 to smithi07134777-20 2024-09-11T01:50:32.894 INFO:tasks.rados.rados.0.smithi071.stdout:3200: finishing write tid 3 to smithi07134777-20 2024-09-11T01:50:32.894 INFO:tasks.rados.rados.0.smithi071.stdout:3200: finishing write tid 4 to smithi07134777-20 2024-09-11T01:50:32.894 INFO:tasks.rados.rados.0.smithi071.stdout:3200: oid 20 updating version 0 to 1075 2024-09-11T01:50:32.894 INFO:tasks.rados.rados.0.smithi071.stdout:3200: oid 20 version 1075 is already newer than 1074 2024-09-11T01:50:32.894 INFO:tasks.rados.rados.0.smithi071.stdout:3200: oid 20 updating version 1075 to 1076 2024-09-11T01:50:32.894 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 1076 (ObjNum 1122 snap 357 seq_num 1122) dirty exists 2024-09-11T01:50:32.894 INFO:tasks.rados.rados.0.smithi071.stdout:3200: left oid 20 (ObjNum 1122 snap 357 seq_num 1122) 2024-09-11T01:50:32.894 INFO:tasks.rados.rados.0.smithi071.stdout:3201: finishing write tid 1 to smithi07134777-34 2024-09-11T01:50:32.894 INFO:tasks.rados.rados.0.smithi071.stdout:3201: finishing write tid 2 to smithi07134777-34 2024-09-11T01:50:32.894 INFO:tasks.rados.rados.0.smithi071.stdout:3201: finishing write tid 3 to smithi07134777-34 2024-09-11T01:50:32.894 INFO:tasks.rados.rados.0.smithi071.stdout:3201: finishing write tid 4 to smithi07134777-34 2024-09-11T01:50:32.894 INFO:tasks.rados.rados.0.smithi071.stdout:3201: finishing write tid 5 to smithi07134777-34 2024-09-11T01:50:32.894 INFO:tasks.rados.rados.0.smithi071.stdout:3201: oid 34 updating version 0 to 2184 2024-09-11T01:50:32.895 INFO:tasks.rados.rados.0.smithi071.stdout:3201: oid 34 version 2184 is already newer than 2181 2024-09-11T01:50:32.895 INFO:tasks.rados.rados.0.smithi071.stdout:3201: oid 34 version 2184 is already newer than 2183 2024-09-11T01:50:32.895 INFO:tasks.rados.rados.0.smithi071.stdout:3201: oid 34 version 2184 is already newer than 2182 2024-09-11T01:50:32.895 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 2184 (ObjNum 1123 snap 357 seq_num 1123) dirty exists 2024-09-11T01:50:32.895 INFO:tasks.rados.rados.0.smithi071.stdout:3201: left oid 34 (ObjNum 1123 snap 357 seq_num 1123) 2024-09-11T01:50:32.895 INFO:tasks.rados.rados.0.smithi071.stdout:3202: finishing write tid 1 to smithi07134777-25 2024-09-11T01:50:32.895 INFO:tasks.rados.rados.0.smithi071.stdout:3202: finishing write tid 2 to smithi07134777-25 2024-09-11T01:50:32.895 INFO:tasks.rados.rados.0.smithi071.stdout:3202: finishing write tid 3 to smithi07134777-25 2024-09-11T01:50:32.895 INFO:tasks.rados.rados.0.smithi071.stdout:3202: finishing write tid 4 to smithi07134777-25 2024-09-11T01:50:32.895 INFO:tasks.rados.rados.0.smithi071.stdout:3202: finishing write tid 5 to smithi07134777-25 2024-09-11T01:50:32.895 INFO:tasks.rados.rados.0.smithi071.stdout:3202: oid 25 updating version 0 to 1038 2024-09-11T01:50:32.895 INFO:tasks.rados.rados.0.smithi071.stdout:3202: oid 25 version 1038 is already newer than 1037 2024-09-11T01:50:32.895 INFO:tasks.rados.rados.0.smithi071.stdout:3202: oid 25 version 1038 is already newer than 1036 2024-09-11T01:50:32.895 INFO:tasks.rados.rados.0.smithi071.stdout:3202: oid 25 updating version 1038 to 1039 2024-09-11T01:50:32.896 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 1039 (ObjNum 1124 snap 357 seq_num 1124) dirty exists 2024-09-11T01:50:32.896 INFO:tasks.rados.rados.0.smithi071.stdout:3202: left oid 25 (ObjNum 1124 snap 357 seq_num 1124) 2024-09-11T01:50:32.896 INFO:tasks.rados.rados.0.smithi071.stdout:3199: done (4 left) 2024-09-11T01:50:32.896 INFO:tasks.rados.rados.0.smithi071.stdout:3200: done (3 left) 2024-09-11T01:50:32.896 INFO:tasks.rados.rados.0.smithi071.stdout:3201: done (2 left) 2024-09-11T01:50:32.896 INFO:tasks.rados.rados.0.smithi071.stdout:3202: done (1 left) 2024-09-11T01:50:32.896 INFO:tasks.rados.rados.0.smithi071.stdout:3206: rollback oid 24 current snap is 357 2024-09-11T01:50:32.896 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 24 to 347 2024-09-11T01:50:32.896 INFO:tasks.rados.rados.0.smithi071.stdout:3207: rollback oid 1 current snap is 357 2024-09-11T01:50:32.896 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 1 to 307 2024-09-11T01:50:32.896 INFO:tasks.rados.rados.0.smithi071.stdout:3208: snap_remove snap 330 2024-09-11T01:50:33.903 INFO:tasks.rados.rados.0.smithi071.stdout:3208: done (3 left) 2024-09-11T01:50:33.903 INFO:tasks.rados.rados.0.smithi071.stdout:3209: copy_from oid 41 from oid 4 current snap is 357 2024-09-11T01:50:33.903 INFO:tasks.rados.rados.0.smithi071.stdout:3205: finishing copy_from to smithi07134777-11 2024-09-11T01:50:33.903 INFO:tasks.rados.rados.0.smithi071.stdout:3205: got expected ENOENT (src dne) 2024-09-11T01:50:33.903 INFO:tasks.rados.rados.0.smithi071.stdout:3205: finishing copy_from racing read to smithi07134777-11 2024-09-11T01:50:33.903 INFO:tasks.rados.rados.0.smithi071.stdout:3207: finishing rollback tid 0 to smithi07134777-1 2024-09-11T01:50:33.903 INFO:tasks.rados.rados.0.smithi071.stdout:3207: finishing rollback tid 1 to smithi07134777-1 2024-09-11T01:50:33.903 INFO:tasks.rados.rados.0.smithi071.stdout:3206: finishing rollback tid 0 to smithi07134777-24 2024-09-11T01:50:33.903 INFO:tasks.rados.rados.0.smithi071.stdout:3207: finishing rollback tid 2 to smithi07134777-1 2024-09-11T01:50:33.903 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 1816 (ObjNum 688 snap 204 seq_num 688) dirty exists 2024-09-11T01:50:33.904 INFO:tasks.rados.rados.0.smithi071.stdout:3206: finishing rollback tid 1 to smithi07134777-24 2024-09-11T01:50:33.904 INFO:tasks.rados.rados.0.smithi071.stdout:3206: finishing rollback tid 2 to smithi07134777-24 2024-09-11T01:50:33.904 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 2067 (ObjNum 1060 snap 340 seq_num 1060) dirty exists 2024-09-11T01:50:33.904 INFO:tasks.rados.rados.0.smithi071.stdout:3205: done (3 left) 2024-09-11T01:50:33.904 INFO:tasks.rados.rados.0.smithi071.stdout:3206: done (2 left) 2024-09-11T01:50:33.904 INFO:tasks.rados.rados.0.smithi071.stdout:3207: done (1 left) 2024-09-11T01:50:33.904 INFO:tasks.rados.rados.0.smithi071.stdout:3210: delete oid 26 current snap is 357 2024-09-11T01:50:33.906 INFO:tasks.rados.rados.0.smithi071.stdout:3209: finishing copy_from to smithi07134777-41 2024-09-11T01:50:33.906 INFO:tasks.rados.rados.0.smithi071.stdout:3209: got expected ENOENT (src dne) 2024-09-11T01:50:33.906 INFO:tasks.rados.rados.0.smithi071.stdout:3209: finishing copy_from racing read to smithi07134777-41 2024-09-11T01:50:33.914 INFO:tasks.rados.rados.0.smithi071.stdout:3209: done (1 left) 2024-09-11T01:50:33.914 INFO:tasks.rados.rados.0.smithi071.stdout:3210: done (0 left) 2024-09-11T01:50:33.914 INFO:tasks.rados.rados.0.smithi071.stdout:3211: write (excl) oid 4 current snap is 357 2024-09-11T01:50:33.915 INFO:tasks.rados.rados.0.smithi071.stdout:3211: seq_num 1127 ranges {541489=425867,1416030=740714,2732654=208835} 2024-09-11T01:50:33.919 INFO:tasks.rados.rados.0.smithi071.stdout:3211: writing smithi07134777-4 from 541489 to 967356 tid 1 2024-09-11T01:50:33.926 INFO:tasks.rados.rados.0.smithi071.stdout:3211: writing smithi07134777-4 from 1416030 to 2156744 tid 2 2024-09-11T01:50:33.931 INFO:tasks.rados.rados.0.smithi071.stdout:3211: writing smithi07134777-4 from 2732654 to 2941489 tid 3 2024-09-11T01:50:33.931 INFO:tasks.rados.rados.0.smithi071.stdout:3212: write oid 27 current snap is 357 2024-09-11T01:50:33.931 INFO:tasks.rados.rados.0.smithi071.stdout:3212: seq_num 1128 ranges {558296=501116,1474512=600725,2760077=198219} 2024-09-11T01:50:33.936 INFO:tasks.rados.rados.0.smithi071.stdout:3212: writing smithi07134777-27 from 558296 to 1059412 tid 1 2024-09-11T01:50:33.941 INFO:tasks.rados.rados.0.smithi071.stdout:3212: writing smithi07134777-27 from 1474512 to 2075237 tid 2 2024-09-11T01:50:33.946 INFO:tasks.rados.rados.0.smithi071.stdout:3212: writing smithi07134777-27 from 2760077 to 2958296 tid 3 2024-09-11T01:50:33.946 INFO:tasks.rados.rados.0.smithi071.stdout:3213: write oid 22 current snap is 357 2024-09-11T01:50:33.947 INFO:tasks.rados.rados.0.smithi071.stdout:3211: finishing write tid 1 to smithi07134777-4 2024-09-11T01:50:33.947 INFO:tasks.rados.rados.0.smithi071.stdout:3211: finishing write tid 2 to smithi07134777-4 2024-09-11T01:50:33.947 INFO:tasks.rados.rados.0.smithi071.stdout:3213: seq_num 1129 ranges {575103=576365,1932994=744383} 2024-09-11T01:50:33.952 INFO:tasks.rados.rados.0.smithi071.stdout:3213: writing smithi07134777-22 from 575103 to 1151468 tid 1 2024-09-11T01:50:33.963 INFO:tasks.rados.rados.0.smithi071.stdout:3213: writing smithi07134777-22 from 1932994 to 2677377 tid 2 2024-09-11T01:50:33.963 INFO:tasks.rados.rados.0.smithi071.stdout:3214: rollback oid 30 current snap is 357 2024-09-11T01:50:33.963 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 30 to 323 2024-09-11T01:50:33.963 INFO:tasks.rados.rados.0.smithi071.stdout:3211: finishing write tid 3 to smithi07134777-4 2024-09-11T01:50:33.963 INFO:tasks.rados.rados.0.smithi071.stdout:3211: finishing write tid 4 to smithi07134777-4 2024-09-11T01:50:33.963 INFO:tasks.rados.rados.0.smithi071.stdout:3211: finishing write tid 5 to smithi07134777-4 2024-09-11T01:50:33.964 INFO:tasks.rados.rados.0.smithi071.stdout:3211: oid 4 updating version 0 to 2179 2024-09-11T01:50:33.964 INFO:tasks.rados.rados.0.smithi071.stdout:3211: oid 4 version 2179 is already newer than 2178 2024-09-11T01:50:33.964 INFO:tasks.rados.rados.0.smithi071.stdout:3211: oid 4 updating version 2179 to 2180 2024-09-11T01:50:33.964 INFO:tasks.rados.rados.0.smithi071.stdout:3211: oid 4 version 2180 is already newer than 2177 2024-09-11T01:50:33.964 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 2180 (ObjNum 1127 snap 357 seq_num 1127) dirty exists 2024-09-11T01:50:33.964 INFO:tasks.rados.rados.0.smithi071.stdout:3211: left oid 4 (ObjNum 1127 snap 357 seq_num 1127) 2024-09-11T01:50:33.964 INFO:tasks.rados.rados.0.smithi071.stdout:3212: finishing write tid 1 to smithi07134777-27 2024-09-11T01:50:33.964 INFO:tasks.rados.rados.0.smithi071.stdout:3212: finishing write tid 2 to smithi07134777-27 2024-09-11T01:50:33.964 INFO:tasks.rados.rados.0.smithi071.stdout:3212: finishing write tid 3 to smithi07134777-27 2024-09-11T01:50:33.964 INFO:tasks.rados.rados.0.smithi071.stdout:3212: finishing write tid 4 to smithi07134777-27 2024-09-11T01:50:33.964 INFO:tasks.rados.rados.0.smithi071.stdout:3212: finishing write tid 5 to smithi07134777-27 2024-09-11T01:50:33.964 INFO:tasks.rados.rados.0.smithi071.stdout:3212: oid 27 updating version 0 to 2186 2024-09-11T01:50:33.964 INFO:tasks.rados.rados.0.smithi071.stdout:3212: oid 27 updating version 2186 to 2187 2024-09-11T01:50:33.964 INFO:tasks.rados.rados.0.smithi071.stdout:3212: oid 27 updating version 2187 to 2189 2024-09-11T01:50:33.965 INFO:tasks.rados.rados.0.smithi071.stdout:3212: oid 27 version 2189 is already newer than 2188 2024-09-11T01:50:33.965 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 2189 (ObjNum 1128 snap 357 seq_num 1128) dirty exists 2024-09-11T01:50:33.965 INFO:tasks.rados.rados.0.smithi071.stdout:3212: left oid 27 (ObjNum 1128 snap 357 seq_num 1128) 2024-09-11T01:50:33.965 INFO:tasks.rados.rados.0.smithi071.stdout:3213: finishing write tid 1 to smithi07134777-22 2024-09-11T01:50:33.965 INFO:tasks.rados.rados.0.smithi071.stdout:3211: done (3 left) 2024-09-11T01:50:33.965 INFO:tasks.rados.rados.0.smithi071.stdout:3212: done (2 left) 2024-09-11T01:50:33.965 INFO:tasks.rados.rados.0.smithi071.stdout:3215: write (excl) oid 33 current snap is 357 2024-09-11T01:50:33.965 INFO:tasks.rados.rados.0.smithi071.stdout:3215: seq_num 1130 ranges {591910=651614,1875123=604394} 2024-09-11T01:50:33.968 INFO:tasks.rados.rados.0.smithi071.stdout:3215: writing smithi07134777-33 from 591910 to 1243524 tid 1 2024-09-11T01:50:33.975 INFO:tasks.rados.rados.0.smithi071.stdout:3215: writing smithi07134777-33 from 1875123 to 2479517 tid 2 2024-09-11T01:50:33.975 INFO:tasks.rados.rados.0.smithi071.stdout:3216: write oid 2 current snap is 357 2024-09-11T01:50:33.975 INFO:tasks.rados.rados.0.smithi071.stdout:3216: seq_num 1131 ranges {608717=726863,1933605=748052} 2024-09-11T01:50:33.982 INFO:tasks.rados.rados.0.smithi071.stdout:3216: writing smithi07134777-2 from 608717 to 1335580 tid 1 2024-09-11T01:50:33.991 INFO:tasks.rados.rados.0.smithi071.stdout:3216: writing smithi07134777-2 from 1933605 to 2681657 tid 2 2024-09-11T01:50:33.991 INFO:tasks.rados.rados.0.smithi071.stdout:3217: copy_from oid 34 from oid 3 current snap is 357 2024-09-11T01:50:33.991 INFO:tasks.rados.rados.0.smithi071.stdout:3213: finishing write tid 2 to smithi07134777-22 2024-09-11T01:50:33.992 INFO:tasks.rados.rados.0.smithi071.stdout:3213: finishing write tid 3 to smithi07134777-22 2024-09-11T01:50:33.992 INFO:tasks.rados.rados.0.smithi071.stdout:3213: finishing write tid 4 to smithi07134777-22 2024-09-11T01:50:33.992 INFO:tasks.rados.rados.0.smithi071.stdout:3213: oid 22 updating version 0 to 2045 2024-09-11T01:50:33.992 INFO:tasks.rados.rados.0.smithi071.stdout:3213: oid 22 updating version 2045 to 2046 2024-09-11T01:50:33.992 INFO:tasks.rados.rados.0.smithi071.stdout:3213: oid 22 version 2046 is already newer than 2044 2024-09-11T01:50:33.992 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 2046 (ObjNum 1129 snap 357 seq_num 1129) dirty exists 2024-09-11T01:50:33.992 INFO:tasks.rados.rados.0.smithi071.stdout:3213: left oid 22 (ObjNum 1129 snap 357 seq_num 1129) 2024-09-11T01:50:33.992 INFO:tasks.rados.rados.0.smithi071.stdout:3214: finishing rollback tid 0 to smithi07134777-30 2024-09-11T01:50:33.992 INFO:tasks.rados.rados.0.smithi071.stdout:3214: finishing rollback tid 1 to smithi07134777-30 2024-09-11T01:50:33.992 INFO:tasks.rados.rados.0.smithi071.stdout:3214: finishing rollback tid 2 to smithi07134777-30 2024-09-11T01:50:33.992 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 2183 (ObjNum 1003 snap 322 seq_num 1003) dirty exists 2024-09-11T01:50:33.992 INFO:tasks.rados.rados.0.smithi071.stdout:3215: finishing write tid 1 to smithi07134777-33 2024-09-11T01:50:33.993 INFO:tasks.rados.rados.0.smithi071.stdout:3215: finishing write tid 2 to smithi07134777-33 2024-09-11T01:50:33.993 INFO:tasks.rados.rados.0.smithi071.stdout:3215: finishing write tid 3 to smithi07134777-33 2024-09-11T01:50:33.993 INFO:tasks.rados.rados.0.smithi071.stdout:3215: finishing write tid 4 to smithi07134777-33 2024-09-11T01:50:33.993 INFO:tasks.rados.rados.0.smithi071.stdout:3215: oid 33 updating version 0 to 2192 2024-09-11T01:50:33.993 INFO:tasks.rados.rados.0.smithi071.stdout:3215: oid 33 version 2192 is already newer than 2191 2024-09-11T01:50:33.993 INFO:tasks.rados.rados.0.smithi071.stdout:3215: oid 33 version 2192 is already newer than 2190 2024-09-11T01:50:33.993 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 2192 (ObjNum 1130 snap 357 seq_num 1130) dirty exists 2024-09-11T01:50:33.993 INFO:tasks.rados.rados.0.smithi071.stdout:3215: left oid 33 (ObjNum 1130 snap 357 seq_num 1130) 2024-09-11T01:50:33.993 INFO:tasks.rados.rados.0.smithi071.stdout:3216: finishing write tid 1 to smithi07134777-2 2024-09-11T01:50:33.993 INFO:tasks.rados.rados.0.smithi071.stdout:3213: done (4 left) 2024-09-11T01:50:33.993 INFO:tasks.rados.rados.0.smithi071.stdout:3214: done (3 left) 2024-09-11T01:50:33.993 INFO:tasks.rados.rados.0.smithi071.stdout:3215: done (2 left) 2024-09-11T01:50:33.993 INFO:tasks.rados.rados.0.smithi071.stdout:3218: read oid 42 snap -1 2024-09-11T01:50:33.993 INFO:tasks.rados.rados.0.smithi071.stdout:3218: expect (ObjNum 1063 snap 341 seq_num 1063) 2024-09-11T01:50:33.994 INFO:tasks.rados.rados.0.smithi071.stdout:3219: write (excl) oid 28 current snap is 357 2024-09-11T01:50:33.994 INFO:tasks.rados.rados.0.smithi071.stdout:3219: seq_num 1132 ranges {625524=402112,1592087=608063,2869769=155755} 2024-09-11T01:50:33.995 INFO:tasks.rados.rados.0.smithi071.stdout:3219: writing smithi07134777-28 from 625524 to 1027636 tid 1 2024-09-11T01:50:34.002 INFO:tasks.rados.rados.0.smithi071.stdout:3219: writing smithi07134777-28 from 1592087 to 2200150 tid 2 2024-09-11T01:50:34.007 INFO:tasks.rados.rados.0.smithi071.stdout:3219: writing smithi07134777-28 from 2869769 to 3025524 tid 3 2024-09-11T01:50:34.007 INFO:tasks.rados.rados.0.smithi071.stdout:3220: copy_from oid 22 from oid 27 current snap is 357 2024-09-11T01:50:34.007 INFO:tasks.rados.rados.0.smithi071.stdout:3216: finishing write tid 2 to smithi07134777-2 2024-09-11T01:50:34.007 INFO:tasks.rados.rados.0.smithi071.stdout:3218: expect (ObjNum 1063 snap 341 seq_num 1063) 2024-09-11T01:50:34.057 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3169524 2024-09-11T01:50:34.075 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:50:34.155 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:50:34.237 INFO:tasks.rados.rados.0.smithi071.stdout:3216: finishing write tid 3 to smithi07134777-2 2024-09-11T01:50:34.237 INFO:tasks.rados.rados.0.smithi071.stdout:3216: finishing write tid 4 to smithi07134777-2 2024-09-11T01:50:34.238 INFO:tasks.rados.rados.0.smithi071.stdout:3216: oid 2 updating version 0 to 1780 2024-09-11T01:50:34.238 INFO:tasks.rados.rados.0.smithi071.stdout:3216: oid 2 version 1780 is already newer than 1779 2024-09-11T01:50:34.238 INFO:tasks.rados.rados.0.smithi071.stdout:3216: oid 2 updating version 1780 to 1781 2024-09-11T01:50:34.238 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 1781 (ObjNum 1131 snap 357 seq_num 1131) dirty exists 2024-09-11T01:50:34.238 INFO:tasks.rados.rados.0.smithi071.stdout:3216: left oid 2 (ObjNum 1131 snap 357 seq_num 1131) 2024-09-11T01:50:34.238 INFO:tasks.rados.rados.0.smithi071.stdout:3219: finishing write tid 1 to smithi07134777-28 2024-09-11T01:50:34.238 INFO:tasks.rados.rados.0.smithi071.stdout:3217: finishing copy_from racing read to smithi07134777-34 2024-09-11T01:50:34.238 INFO:tasks.rados.rados.0.smithi071.stdout:3216: done (4 left) 2024-09-11T01:50:34.238 INFO:tasks.rados.rados.0.smithi071.stdout:3218: done (3 left) 2024-09-11T01:50:34.238 INFO:tasks.rados.rados.0.smithi071.stdout:3221: read oid 6 snap -1 2024-09-11T01:50:34.238 INFO:tasks.rados.rados.0.smithi071.stdout:3221: expect (ObjNum 926 snap 292 seq_num 926) 2024-09-11T01:50:34.238 INFO:tasks.rados.rados.0.smithi071.stdout:3222: copy_from oid 35 from oid 20 current snap is 357 2024-09-11T01:50:34.238 INFO:tasks.rados.rados.0.smithi071.stdout:3219: finishing write tid 2 to smithi07134777-28 2024-09-11T01:50:34.238 INFO:tasks.rados.rados.0.smithi071.stdout:3219: finishing write tid 3 to smithi07134777-28 2024-09-11T01:50:34.239 INFO:tasks.rados.rados.0.smithi071.stdout:3219: finishing write tid 4 to smithi07134777-28 2024-09-11T01:50:34.239 INFO:tasks.rados.rados.0.smithi071.stdout:3219: finishing write tid 5 to smithi07134777-28 2024-09-11T01:50:34.239 INFO:tasks.rados.rados.0.smithi071.stdout:3219: oid 28 updating version 0 to 2070 2024-09-11T01:50:34.239 INFO:tasks.rados.rados.0.smithi071.stdout:3219: oid 28 version 2070 is already newer than 2069 2024-09-11T01:50:34.239 INFO:tasks.rados.rados.0.smithi071.stdout:3219: oid 28 version 2070 is already newer than 2068 2024-09-11T01:50:34.239 INFO:tasks.rados.rados.0.smithi071.stdout:3219: oid 28 updating version 2070 to 2071 2024-09-11T01:50:34.239 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 2071 (ObjNum 1132 snap 357 seq_num 1132) dirty exists 2024-09-11T01:50:34.239 INFO:tasks.rados.rados.0.smithi071.stdout:3219: left oid 28 (ObjNum 1132 snap 357 seq_num 1132) 2024-09-11T01:50:34.239 INFO:tasks.rados.rados.0.smithi071.stdout:3217: finishing copy_from to smithi07134777-34 2024-09-11T01:50:34.239 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 2036 (ObjNum 1032 snap 331 seq_num 1032) dirty exists 2024-09-11T01:50:34.239 INFO:tasks.rados.rados.0.smithi071.stdout:3220: finishing copy_from racing read to smithi07134777-22 2024-09-11T01:50:34.239 INFO:tasks.rados.rados.0.smithi071.stdout:3220: finishing copy_from to smithi07134777-22 2024-09-11T01:50:34.239 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 2189 (ObjNum 1128 snap 357 seq_num 1128) dirty exists 2024-09-11T01:50:34.239 INFO:tasks.rados.rados.0.smithi071.stdout:3217: done (4 left) 2024-09-11T01:50:34.239 INFO:tasks.rados.rados.0.smithi071.stdout:3219: done (3 left) 2024-09-11T01:50:34.240 INFO:tasks.rados.rados.0.smithi071.stdout:3220: done (2 left) 2024-09-11T01:50:34.240 INFO:tasks.rados.rados.0.smithi071.stdout:3223: snap_remove snap 324 2024-09-11T01:50:34.912 INFO:tasks.rados.rados.0.smithi071.stdout:3223: done (2 left) 2024-09-11T01:50:34.912 INFO:tasks.rados.rados.0.smithi071.stdout:3224: read oid 24 snap 350 2024-09-11T01:50:34.912 INFO:tasks.rados.rados.0.smithi071.stdout:3224: expect (ObjNum 1060 snap 340 seq_num 1060) 2024-09-11T01:50:34.913 INFO:tasks.rados.rados.0.smithi071.stdout:3225: delete oid 25 current snap is 357 2024-09-11T01:50:34.913 INFO:tasks.rados.rados.0.smithi071.stdout:3221: expect (ObjNum 926 snap 292 seq_num 926) 2024-09-11T01:50:34.913 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:50:35.062 INFO:tasks.rados.rados.0.smithi071.stdout:3222: finishing copy_from racing read to smithi07134777-35 2024-09-11T01:50:35.062 INFO:tasks.rados.rados.0.smithi071.stdout:3222: finishing copy_from to smithi07134777-35 2024-09-11T01:50:35.062 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 1076 (ObjNum 1122 snap 357 seq_num 1122) dirty exists 2024-09-11T01:50:35.062 INFO:tasks.rados.rados.0.smithi071.stdout:3224: expect (ObjNum 1060 snap 340 seq_num 1060) 2024-09-11T01:50:35.218 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3581830 2024-09-11T01:50:35.224 INFO:tasks.rados.rados.0.smithi071.stdout:3221: done (3 left) 2024-09-11T01:50:35.224 INFO:tasks.rados.rados.0.smithi071.stdout:3222: done (2 left) 2024-09-11T01:50:35.224 INFO:tasks.rados.rados.0.smithi071.stdout:3224: done (1 left) 2024-09-11T01:50:35.225 INFO:tasks.rados.rados.0.smithi071.stdout:3225: done (0 left) 2024-09-11T01:50:35.225 INFO:tasks.rados.rados.0.smithi071.stdout:3226: write (excl) oid 6 current snap is 357 2024-09-11T01:50:35.225 INFO:tasks.rados.rados.0.smithi071.stdout:3226: seq_num 1134 ranges {659138=668963,2109051=611732} 2024-09-11T01:50:35.231 INFO:tasks.rados.rados.0.smithi071.stdout:3226: writing smithi07134777-6 from 659138 to 1328101 tid 1 2024-09-11T01:50:35.239 INFO:tasks.rados.rados.0.smithi071.stdout:3226: writing smithi07134777-6 from 2109051 to 2720783 tid 2 2024-09-11T01:50:35.239 INFO:tasks.rados.rados.0.smithi071.stdout:3227: snap_remove snap 349 2024-09-11T01:50:35.239 INFO:tasks.rados.rados.0.smithi071.stdout:3226: finishing write tid 1 to smithi07134777-6 2024-09-11T01:50:35.861 INFO:tasks.rados.rados.0.smithi071.stdout:3227: done (1 left) 2024-09-11T01:50:35.861 INFO:tasks.rados.rados.0.smithi071.stdout:3228: delete oid 24 current snap is 357 2024-09-11T01:50:35.861 INFO:tasks.rados.rados.0.smithi071.stdout:3226: finishing write tid 2 to smithi07134777-6 2024-09-11T01:50:35.861 INFO:tasks.rados.rados.0.smithi071.stdout:3226: finishing write tid 3 to smithi07134777-6 2024-09-11T01:50:35.861 INFO:tasks.rados.rados.0.smithi071.stdout:3226: finishing write tid 4 to smithi07134777-6 2024-09-11T01:50:35.861 INFO:tasks.rados.rados.0.smithi071.stdout:3226: oid 6 updating version 0 to 2194 2024-09-11T01:50:35.862 INFO:tasks.rados.rados.0.smithi071.stdout:3226: oid 6 version 2194 is already newer than 2193 2024-09-11T01:50:35.862 INFO:tasks.rados.rados.0.smithi071.stdout:3226: oid 6 updating version 2194 to 2195 2024-09-11T01:50:35.862 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 2195 (ObjNum 1134 snap 357 seq_num 1134) dirty exists 2024-09-11T01:50:35.862 INFO:tasks.rados.rados.0.smithi071.stdout:3226: left oid 6 (ObjNum 1134 snap 357 seq_num 1134) 2024-09-11T01:50:35.874 INFO:tasks.rados.rados.0.smithi071.stdout:3226: done (1 left) 2024-09-11T01:50:35.874 INFO:tasks.rados.rados.0.smithi071.stdout:3228: done (0 left) 2024-09-11T01:50:35.874 INFO:tasks.rados.rados.0.smithi071.stdout:3229: snap_create 2024-09-11T01:50:36.866 INFO:tasks.rados.rados.0.smithi071.stdout:3229: done (0 left) 2024-09-11T01:50:36.866 INFO:tasks.rados.rados.0.smithi071.stdout:3230: snap_create 2024-09-11T01:50:37.870 INFO:tasks.rados.rados.0.smithi071.stdout:3230: done (0 left) 2024-09-11T01:50:37.870 INFO:tasks.rados.rados.0.smithi071.stdout:3231: snap_create 2024-09-11T01:50:38.872 INFO:tasks.rados.rados.0.smithi071.stdout:3231: done (0 left) 2024-09-11T01:50:38.872 INFO:tasks.rados.rados.0.smithi071.stdout:3232: read oid 5 snap -1 2024-09-11T01:50:38.873 INFO:tasks.rados.rados.0.smithi071.stdout:3232: expect (ObjNum 1121 snap 357 seq_num 1121) 2024-09-11T01:50:38.873 INFO:tasks.rados.rados.0.smithi071.stdout:3233: write (excl) oid 19 current snap is 360 2024-09-11T01:50:38.873 INFO:tasks.rados.rados.0.smithi071.stdout:3233: seq_num 1136 ranges {692752=419461,1826015=615401,2979461=113291} 2024-09-11T01:50:38.878 INFO:tasks.rados.rados.0.smithi071.stdout:3233: writing smithi07134777-19 from 692752 to 1112213 tid 1 2024-09-11T01:50:38.885 INFO:tasks.rados.rados.0.smithi071.stdout:3233: writing smithi07134777-19 from 1826015 to 2441416 tid 2 2024-09-11T01:50:38.889 INFO:tasks.rados.rados.0.smithi071.stdout:3233: writing smithi07134777-19 from 2979461 to 3092752 tid 3 2024-09-11T01:50:38.889 INFO:tasks.rados.rados.0.smithi071.stdout:3234: snap_remove snap 345 2024-09-11T01:50:39.880 INFO:tasks.rados.rados.0.smithi071.stdout:3234: done (2 left) 2024-09-11T01:50:39.880 INFO:tasks.rados.rados.0.smithi071.stdout:3235: delete oid 32 current snap is 360 2024-09-11T01:50:39.880 INFO:tasks.rados.rados.0.smithi071.stdout:3232: expect (ObjNum 1121 snap 357 seq_num 1121) 2024-09-11T01:50:39.885 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 958443 2024-09-11T01:50:39.918 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2099355 2024-09-11T01:50:39.952 INFO:tasks.rados.rados.0.smithi071.stdout:3233: finishing write tid 1 to smithi07134777-19 2024-09-11T01:50:39.952 INFO:tasks.rados.rados.0.smithi071.stdout:3233: finishing write tid 2 to smithi07134777-19 2024-09-11T01:50:39.952 INFO:tasks.rados.rados.0.smithi071.stdout:3233: finishing write tid 3 to smithi07134777-19 2024-09-11T01:50:39.952 INFO:tasks.rados.rados.0.smithi071.stdout:3233: finishing write tid 4 to smithi07134777-19 2024-09-11T01:50:39.952 INFO:tasks.rados.rados.0.smithi071.stdout:3233: finishing write tid 5 to smithi07134777-19 2024-09-11T01:50:39.952 INFO:tasks.rados.rados.0.smithi071.stdout:3233: oid 19 updating version 0 to 1794 2024-09-11T01:50:39.952 INFO:tasks.rados.rados.0.smithi071.stdout:3233: oid 19 updating version 1794 to 1795 2024-09-11T01:50:39.952 INFO:tasks.rados.rados.0.smithi071.stdout:3233: oid 19 version 1795 is already newer than 1793 2024-09-11T01:50:39.953 INFO:tasks.rados.rados.0.smithi071.stdout:3233: oid 19 updating version 1795 to 1796 2024-09-11T01:50:39.953 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 1796 (ObjNum 1136 snap 360 seq_num 1136) dirty exists 2024-09-11T01:50:39.953 INFO:tasks.rados.rados.0.smithi071.stdout:3233: left oid 19 (ObjNum 1136 snap 360 seq_num 1136) 2024-09-11T01:50:39.953 INFO:tasks.rados.rados.0.smithi071.stdout:3232: done (2 left) 2024-09-11T01:50:39.953 INFO:tasks.rados.rados.0.smithi071.stdout:3233: done (1 left) 2024-09-11T01:50:39.953 INFO:tasks.rados.rados.0.smithi071.stdout:3235: done (0 left) 2024-09-11T01:50:39.953 INFO:tasks.rados.rados.0.smithi071.stdout:3236: read oid 23 snap -1 2024-09-11T01:50:39.953 INFO:tasks.rados.rados.0.smithi071.stdout:3236: expect deleted 2024-09-11T01:50:39.953 INFO:tasks.rados.rados.0.smithi071.stdout:3237: delete oid 13 current snap is 360 2024-09-11T01:50:39.955 INFO:tasks.rados.rados.0.smithi071.stdout:3236: done (1 left) 2024-09-11T01:50:39.955 INFO:tasks.rados.rados.0.smithi071.stdout:3237: done (0 left) 2024-09-11T01:50:39.955 INFO:tasks.rados.rados.0.smithi071.stdout:3238: read oid 30 snap -1 2024-09-11T01:50:39.955 INFO:tasks.rados.rados.0.smithi071.stdout:3238: expect (ObjNum 1003 snap 322 seq_num 1003) 2024-09-11T01:50:39.955 INFO:tasks.rados.rados.0.smithi071.stdout:3239: snap_remove snap 312 2024-09-11T01:50:40.864 INFO:tasks.rados.rados.0.smithi071.stdout:3239: done (1 left) 2024-09-11T01:50:40.864 INFO:tasks.rados.rados.0.smithi071.stdout:3240: write oid 9 current snap is 360 2024-09-11T01:50:40.864 INFO:tasks.rados.rados.0.smithi071.stdout:3240: seq_num 1139 ranges {743173=645208,1885108=762728} 2024-09-11T01:50:40.873 INFO:tasks.rados.rados.0.smithi071.stdout:3240: writing smithi07134777-9 from 743173 to 1388381 tid 1 2024-09-11T01:50:40.879 INFO:tasks.rados.rados.0.smithi071.stdout:3240: writing smithi07134777-9 from 1885108 to 2647836 tid 2 2024-09-11T01:50:40.879 INFO:tasks.rados.rados.0.smithi071.stdout:3241: read oid 33 snap 356 2024-09-11T01:50:40.880 INFO:tasks.rados.rados.0.smithi071.stdout:3241: expect (ObjNum 875 snap 267 seq_num 875) 2024-09-11T01:50:40.880 INFO:tasks.rados.rados.0.smithi071.stdout:3242: rollback oid 39 current snap is 360 2024-09-11T01:50:40.880 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 39 to 353 2024-09-11T01:50:40.880 INFO:tasks.rados.rados.0.smithi071.stdout:3238: expect (ObjNum 1003 snap 322 seq_num 1003) 2024-09-11T01:50:40.885 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1092914 2024-09-11T01:50:40.987 INFO:tasks.rados.rados.0.smithi071.stdout:3240: finishing write tid 1 to smithi07134777-9 2024-09-11T01:50:40.987 INFO:tasks.rados.rados.0.smithi071.stdout:3242: finishing rollback tid 0 to smithi07134777-39 2024-09-11T01:50:40.987 INFO:tasks.rados.rados.0.smithi071.stdout:3242: finishing rollback tid 1 to smithi07134777-39 2024-09-11T01:50:40.987 INFO:tasks.rados.rados.0.smithi071.stdout:3242: finishing rollback tid 2 to smithi07134777-39 2024-09-11T01:50:40.987 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 1972 (ObjNum 875 snap 267 seq_num 875) dirty exists 2024-09-11T01:50:40.987 INFO:tasks.rados.rados.0.smithi071.stdout:3241: expect (ObjNum 875 snap 267 seq_num 875) 2024-09-11T01:50:40.987 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:50:41.017 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:50:41.045 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:50:41.074 INFO:tasks.rados.rados.0.smithi071.stdout:3240: finishing write tid 2 to smithi07134777-9 2024-09-11T01:50:41.074 INFO:tasks.rados.rados.0.smithi071.stdout:3240: finishing write tid 3 to smithi07134777-9 2024-09-11T01:50:41.074 INFO:tasks.rados.rados.0.smithi071.stdout:3240: finishing write tid 4 to smithi07134777-9 2024-09-11T01:50:41.074 INFO:tasks.rados.rados.0.smithi071.stdout:3240: oid 9 updating version 0 to 1042 2024-09-11T01:50:41.074 INFO:tasks.rados.rados.0.smithi071.stdout:3240: oid 9 updating version 1042 to 1043 2024-09-11T01:50:41.074 INFO:tasks.rados.rados.0.smithi071.stdout:3240: oid 9 version 1043 is already newer than 1041 2024-09-11T01:50:41.074 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 1043 (ObjNum 1139 snap 360 seq_num 1139) dirty exists 2024-09-11T01:50:41.075 INFO:tasks.rados.rados.0.smithi071.stdout:3240: left oid 9 (ObjNum 1139 snap 360 seq_num 1139) 2024-09-11T01:50:41.075 INFO:tasks.rados.rados.0.smithi071.stdout:3238: done (3 left) 2024-09-11T01:50:41.075 INFO:tasks.rados.rados.0.smithi071.stdout:3240: done (2 left) 2024-09-11T01:50:41.075 INFO:tasks.rados.rados.0.smithi071.stdout:3241: done (1 left) 2024-09-11T01:50:41.075 INFO:tasks.rados.rados.0.smithi071.stdout:3242: done (0 left) 2024-09-11T01:50:41.075 INFO:tasks.rados.rados.0.smithi071.stdout:3243: snap_remove snap 354 2024-09-11T01:50:41.886 INFO:tasks.rados.rados.0.smithi071.stdout:3243: done (0 left) 2024-09-11T01:50:41.886 INFO:tasks.rados.rados.0.smithi071.stdout:3244: read oid 32 snap -1 2024-09-11T01:50:41.887 INFO:tasks.rados.rados.0.smithi071.stdout:3244: expect deleted 2024-09-11T01:50:41.887 INFO:tasks.rados.rados.0.smithi071.stdout:3245: snap_remove snap 225 2024-09-11T01:50:42.891 INFO:tasks.rados.rados.0.smithi071.stdout:3245: done (1 left) 2024-09-11T01:50:42.891 INFO:tasks.rados.rados.0.smithi071.stdout:3246: read oid 37 snap -1 2024-09-11T01:50:42.891 INFO:tasks.rados.rados.0.smithi071.stdout:3246: expect deleted 2024-09-11T01:50:42.891 INFO:tasks.rados.rados.0.smithi071.stdout:3247: rollback oid 34 current snap is 360 2024-09-11T01:50:42.891 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 34 to 323 2024-09-11T01:50:42.891 INFO:tasks.rados.rados.0.smithi071.stdout:3248: read oid 12 snap -1 2024-09-11T01:50:42.892 INFO:tasks.rados.rados.0.smithi071.stdout:3248: expect (ObjNum 1105 snap 352 seq_num 1105) 2024-09-11T01:50:42.892 INFO:tasks.rados.rados.0.smithi071.stdout:3249: delete oid 4 current snap is 360 2024-09-11T01:50:42.904 INFO:tasks.rados.rados.0.smithi071.stdout:3244: done (4 left) 2024-09-11T01:50:42.905 INFO:tasks.rados.rados.0.smithi071.stdout:3246: done (3 left) 2024-09-11T01:50:42.905 INFO:tasks.rados.rados.0.smithi071.stdout:3249: done (2 left) 2024-09-11T01:50:42.905 INFO:tasks.rados.rados.0.smithi071.stdout:3250: snap_create 2024-09-11T01:50:42.915 INFO:tasks.rados.rados.0.smithi071.stdout:3248: expect (ObjNum 1105 snap 352 seq_num 1105) 2024-09-11T01:50:42.968 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:50:42.995 INFO:tasks.rados.rados.0.smithi071.stdout:3247: finishing rollback tid 0 to smithi07134777-34 2024-09-11T01:50:42.996 INFO:tasks.rados.rados.0.smithi071.stdout:3247: finishing rollback tid 1 to smithi07134777-34 2024-09-11T01:50:42.996 INFO:tasks.rados.rados.0.smithi071.stdout:3247: finishing rollback tid 2 to smithi07134777-34 2024-09-11T01:50:42.996 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 2198 (ObjNum 1013 snap 322 seq_num 1013) dirty exists 2024-09-11T01:50:43.897 INFO:tasks.rados.rados.0.smithi071.stdout:3247: done (2 left) 2024-09-11T01:50:44.917 INFO:tasks.rados.rados.0.smithi071.stdout:3248: done (1 left) 2024-09-11T01:50:44.917 INFO:tasks.rados.rados.0.smithi071.stdout:3250: done (0 left) 2024-09-11T01:50:44.917 INFO:tasks.rados.rados.0.smithi071.stdout:3251: rollback oid 12 current snap is 361 2024-09-11T01:50:44.917 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 12 to 360 2024-09-11T01:50:44.917 INFO:tasks.rados.rados.0.smithi071.stdout:3252: snap_create 2024-09-11T01:50:44.920 INFO:tasks.rados.rados.0.smithi071.stdout:3251: finishing rollback tid 0 to smithi07134777-12 2024-09-11T01:50:44.920 INFO:tasks.rados.rados.0.smithi071.stdout:3251: finishing rollback tid 1 to smithi07134777-12 2024-09-11T01:50:44.920 INFO:tasks.rados.rados.0.smithi071.stdout:3251: finishing rollback tid 2 to smithi07134777-12 2024-09-11T01:50:44.920 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 2201 (ObjNum 1105 snap 352 seq_num 1105) dirty exists 2024-09-11T01:50:44.920 INFO:tasks.rados.rados.0.smithi071.stdout:3251: done (1 left) 2024-09-11T01:50:44.920 INFO:tasks.rados.rados.0.smithi071.stdout:3252: done (0 left) 2024-09-11T01:50:44.920 INFO:tasks.rados.rados.0.smithi071.stdout:3253: write (excl) oid 47 current snap is 362 2024-09-11T01:50:44.921 INFO:tasks.rados.rados.0.smithi071.stdout:3253: seq_num 1141 ranges {776787=512059,1718425=766397,3000223=176564} 2024-09-11T01:50:44.921 INFO:tasks.rados.rados.0.smithi071.stdout:3253: writing smithi07134777-47 from 776787 to 1288846 tid 1 2024-09-11T01:50:44.921 INFO:tasks.rados.rados.0.smithi071.stdout:3253: writing smithi07134777-47 from 1718425 to 2484822 tid 2 2024-09-11T01:50:44.921 INFO:tasks.rados.rados.0.smithi071.stdout:3253: writing smithi07134777-47 from 3000223 to 3176787 tid 3 2024-09-11T01:50:44.921 INFO:tasks.rados.rados.0.smithi071.stdout:3254: copy_from oid 30 from oid 48 current snap is 362 2024-09-11T01:50:44.921 INFO:tasks.rados.rados.0.smithi071.stdout:3253: finishing write tid 1 to smithi07134777-47 2024-09-11T01:50:44.921 INFO:tasks.rados.rados.0.smithi071.stdout:3255: snap_create 2024-09-11T01:50:44.921 INFO:tasks.rados.rados.0.smithi071.stdout:3253: finishing write tid 2 to smithi07134777-47 2024-09-11T01:50:44.921 INFO:tasks.rados.rados.0.smithi071.stdout:3254: finishing copy_from to smithi07134777-30 2024-09-11T01:50:44.921 INFO:tasks.rados.rados.0.smithi071.stdout:3254: got expected ENOENT (src dne) 2024-09-11T01:50:44.921 INFO:tasks.rados.rados.0.smithi071.stdout:3253: finishing write tid 3 to smithi07134777-47 2024-09-11T01:50:44.921 INFO:tasks.rados.rados.0.smithi071.stdout:3254: finishing copy_from racing read to smithi07134777-30 2024-09-11T01:50:44.921 INFO:tasks.rados.rados.0.smithi071.stdout:3253: finishing write tid 4 to smithi07134777-47 2024-09-11T01:50:44.922 INFO:tasks.rados.rados.0.smithi071.stdout:3253: finishing write tid 5 to smithi07134777-47 2024-09-11T01:50:44.922 INFO:tasks.rados.rados.0.smithi071.stdout:3253: oid 47 updating version 0 to 1879 2024-09-11T01:50:44.922 INFO:tasks.rados.rados.0.smithi071.stdout:3253: oid 47 version 1879 is already newer than 1878 2024-09-11T01:50:44.922 INFO:tasks.rados.rados.0.smithi071.stdout:3253: oid 47 version 1879 is already newer than 1877 2024-09-11T01:50:44.922 INFO:tasks.rados.rados.0.smithi071.stdout:3253: oid 47 updating version 1879 to 1880 2024-09-11T01:50:44.922 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 1880 (ObjNum 1141 snap 362 seq_num 1141) dirty exists 2024-09-11T01:50:44.922 INFO:tasks.rados.rados.0.smithi071.stdout:3253: left oid 47 (ObjNum 1141 snap 362 seq_num 1141) 2024-09-11T01:50:45.864 INFO:tasks.rados.rados.0.smithi071.stdout:3253: done (2 left) 2024-09-11T01:50:45.864 INFO:tasks.rados.rados.0.smithi071.stdout:3254: done (1 left) 2024-09-11T01:50:45.865 INFO:tasks.rados.rados.0.smithi071.stdout:3255: done (0 left) 2024-09-11T01:50:45.865 INFO:tasks.rados.rados.0.smithi071.stdout:3256: read oid 45 snap -1 2024-09-11T01:50:45.865 INFO:tasks.rados.rados.0.smithi071.stdout:3256: expect deleted 2024-09-11T01:50:45.865 INFO:tasks.rados.rados.0.smithi071.stdout:3257: read oid 5 snap -1 2024-09-11T01:50:45.865 INFO:tasks.rados.rados.0.smithi071.stdout:3257: expect (ObjNum 1121 snap 357 seq_num 1121) 2024-09-11T01:50:45.865 INFO:tasks.rados.rados.0.smithi071.stdout:3258: read oid 24 snap 357 2024-09-11T01:50:45.865 INFO:tasks.rados.rados.0.smithi071.stdout:3258: expect deleted 2024-09-11T01:50:45.865 INFO:tasks.rados.rados.0.smithi071.stdout:3259: read oid 18 snap -1 2024-09-11T01:50:45.865 INFO:tasks.rados.rados.0.smithi071.stdout:3259: expect deleted 2024-09-11T01:50:45.865 INFO:tasks.rados.rados.0.smithi071.stdout:3260: delete oid 49 current snap is 363 2024-09-11T01:50:45.872 INFO:tasks.rados.rados.0.smithi071.stdout:3256: done (4 left) 2024-09-11T01:50:45.872 INFO:tasks.rados.rados.0.smithi071.stdout:3258: done (3 left) 2024-09-11T01:50:45.872 INFO:tasks.rados.rados.0.smithi071.stdout:3259: done (2 left) 2024-09-11T01:50:45.872 INFO:tasks.rados.rados.0.smithi071.stdout:3260: done (1 left) 2024-09-11T01:50:45.872 INFO:tasks.rados.rados.0.smithi071.stdout:3261: snap_create 2024-09-11T01:50:45.873 INFO:tasks.rados.rados.0.smithi071.stdout:3257: expect (ObjNum 1121 snap 357 seq_num 1121) 2024-09-11T01:50:45.873 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 211810 2024-09-11T01:50:46.457 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-11T01:50:46.458 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-11T01:50:46.458 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-11T01:50:46.461 INFO:tasks.ceph.mon.c:Started 2024-09-11T01:50:46.461 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-11T01:50:46.461 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-11T01:50:46.461 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T01:51:01.461 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-11T01:51:01.462 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-11T01:51:01.462 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:51:01.462 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:51:01.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.574+0000 7f34370b8640 10 monclient: get_monmap_and_config 2024-09-11T01:51:01.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.574+0000 7f34370b8640 10 monclient: build_initial_monmap 2024-09-11T01:51:01.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.574+0000 7f34370b8640 10 monclient: monmap: 2024-09-11T01:51:01.576 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:01.577 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:01.577 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:51:01.575670+0000 2024-09-11T01:51:01.577 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:51:01.575670+0000 2024-09-11T01:51:01.577 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:01.577 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:01.577 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:51:01.577 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:51:01.577 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:51:01.577 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:51:01.577 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:51:01.577 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:01.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.575+0000 7f34370b8640 1 Processor -- start 2024-09-11T01:51:01.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.575+0000 7f34370b8640 1 -- start start 2024-09-11T01:51:01.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.575+0000 7f34370b8640 10 monclient: init 2024-09-11T01:51:01.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f34370b8640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:01.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f34370b8640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-11T01:51:01.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f34370b8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343009da80 0x7f3430136010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:01.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f34370b8640 10 monclient(hunting): picked mon.noname-b con 0x7f343009da80 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:01.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f34370b8640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f343009de60 0x7f34300a2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:01.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f34370b8640 10 monclient(hunting): picked mon.noname-c con 0x7f343009de60 addr v2:172.21.15.71:3300/0 2024-09-11T01:51:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f34370b8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f34300a32f0 0x7f34300a3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f34370b8640 10 monclient(hunting): picked mon.noname-a con 0x7f34300a32f0 addr v2:172.21.15.12:3300/0 2024-09-11T01:51:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f34370b8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f34370b8640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f34300a1590 con 0x7f34300a32f0 2024-09-11T01:51:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f34370b8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f34370b8640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f343006d5b0 con 0x7f343009de60 2024-09-11T01:51:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f34370b8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f34370b8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f343006d890 con 0x7f343009da80 2024-09-11T01:51:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f34370b8640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f34370b8640 10 monclient(hunting): authenticate will time out at 2068.018386s 2024-09-11T01:51:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f3434e54640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343009da80 0x7f3430136010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f3435655640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f34300a32f0 0x7f34300a3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:01.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f3434e54640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343009da80 0x7f3430136010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52392/0 (socket says 172.21.15.12:52392) 2024-09-11T01:51:01.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f3435655640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f34300a32f0 0x7f34300a3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46758/0 (socket says 172.21.15.12:46758) 2024-09-11T01:51:01.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f3434e54640 1 -- 172.21.15.12:0/3047612433 learned_addr learned my addr 172.21.15.12:0/3047612433 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:01.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f3434e54640 10 monclient(hunting): get_auth_request con 0x7f343009da80 auth_method 0 2024-09-11T01:51:01.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f3434e54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:01.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f3434e54640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:01.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f3434e54640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:01.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f3435655640 10 monclient(hunting): get_auth_request con 0x7f34300a32f0 auth_method 0 2024-09-11T01:51:01.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f3435655640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:01.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f3435655640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:01.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f3435655640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:01.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f3435655640 1 -- 172.21.15.12:0/3047612433 >> v2:172.21.15.12:3300/0 conn(0x7f34300a32f0 msgr2=0x7f34300a3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:51:01.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f3434e54640 1 -- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343009da80 msgr2=0x7f3430136010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:51:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f3435655640 1 -- 172.21.15.12:0/3047612433 >> v2:172.21.15.12:3300/0 conn(0x7f34300a32f0 msgr2=0x7f34300a3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f3434e54640 1 -- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343009da80 msgr2=0x7f3430136010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f3435655640 1 --2- 172.21.15.12:0/3047612433 >> v2:172.21.15.12:3300/0 conn(0x7f34300a32f0 0x7f34300a3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f3434e54640 1 --2- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343009da80 0x7f3430136010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f3434e54640 1 --2- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343009da80 0x7f3430136010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:51:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f3435655640 1 --2- 172.21.15.12:0/3047612433 >> v2:172.21.15.12:3300/0 conn(0x7f34300a32f0 0x7f34300a3720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:51:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f342ffff640 1 --2- 172.21.15.12:0/3047612433 >> v2:172.21.15.71:3300/0 conn(0x7f343009de60 0x7f34300a2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f342ffff640 10 monclient(hunting): get_auth_request con 0x7f343009de60 auth_method 0 2024-09-11T01:51:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f342ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f342ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f342ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f342ffff640 1 -- 172.21.15.12:0/3047612433 >> v2:172.21.15.71:3300/0 conn(0x7f343009de60 msgr2=0x7f34300a2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:51:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f342ffff640 1 -- 172.21.15.12:0/3047612433 >> v2:172.21.15.71:3300/0 conn(0x7f343009de60 msgr2=0x7f34300a2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f342ffff640 1 --2- 172.21.15.12:0/3047612433 >> v2:172.21.15.71:3300/0 conn(0x7f343009de60 0x7f34300a2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.576+0000 7f342ffff640 1 --2- 172.21.15.12:0/3047612433 >> v2:172.21.15.71:3300/0 conn(0x7f343009de60 0x7f34300a2db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:51:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f3434e54640 1 --2- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343009da80 0x7f3430136010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f3435655640 1 --2- 172.21.15.12:0/3047612433 >> v2:172.21.15.12:3300/0 conn(0x7f34300a32f0 0x7f34300a3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f3434e54640 10 monclient(hunting): get_auth_request con 0x7f343009da80 auth_method 0 2024-09-11T01:51:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f3434e54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f3434e54640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f3434e54640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f3435655640 10 monclient(hunting): get_auth_request con 0x7f34300a32f0 auth_method 0 2024-09-11T01:51:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f3435655640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f3435655640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f3435655640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f3434e54640 1 -- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343009da80 msgr2=0x7f3430136010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:51:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f3435655640 1 -- 172.21.15.12:0/3047612433 >> v2:172.21.15.12:3300/0 conn(0x7f34300a32f0 msgr2=0x7f34300a3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:51:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f3434e54640 1 -- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343009da80 msgr2=0x7f3430136010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f3435655640 1 -- 172.21.15.12:0/3047612433 >> v2:172.21.15.12:3300/0 conn(0x7f34300a32f0 msgr2=0x7f34300a3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f3434e54640 1 --2- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343009da80 0x7f3430136010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f3435655640 1 --2- 172.21.15.12:0/3047612433 >> v2:172.21.15.12:3300/0 conn(0x7f34300a32f0 0x7f34300a3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f3435655640 1 --2- 172.21.15.12:0/3047612433 >> v2:172.21.15.12:3300/0 conn(0x7f34300a32f0 0x7f34300a3720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:51:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f3434e54640 1 --2- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343009da80 0x7f3430136010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:51:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f342ffff640 1 --2- 172.21.15.12:0/3047612433 >> v2:172.21.15.71:3300/0 conn(0x7f343009de60 0x7f34300a2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f342ffff640 10 monclient(hunting): get_auth_request con 0x7f343009de60 auth_method 0 2024-09-11T01:51:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f342ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f342ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f342ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:01.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f342ffff640 1 -- 172.21.15.12:0/3047612433 >> v2:172.21.15.71:3300/0 conn(0x7f343009de60 msgr2=0x7f34300a2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:51:01.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f342ffff640 1 -- 172.21.15.12:0/3047612433 >> v2:172.21.15.71:3300/0 conn(0x7f343009de60 msgr2=0x7f34300a2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:01.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f342ffff640 1 --2- 172.21.15.12:0/3047612433 >> v2:172.21.15.71:3300/0 conn(0x7f343009de60 0x7f34300a2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:01.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:01.777+0000 7f342ffff640 1 --2- 172.21.15.12:0/3047612433 >> v2:172.21.15.71:3300/0 conn(0x7f343009de60 0x7f34300a2db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:51:02.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f3435655640 1 --2- 172.21.15.12:0/3047612433 >> v2:172.21.15.12:3300/0 conn(0x7f34300a32f0 0x7f34300a3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:02.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f3434e54640 1 --2- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343009da80 0x7f3430136010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:02.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f3435655640 10 monclient(hunting): get_auth_request con 0x7f34300a32f0 auth_method 0 2024-09-11T01:51:02.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f3435655640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:02.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f3435655640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:02.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f3435655640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:02.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f3434e54640 10 monclient(hunting): get_auth_request con 0x7f343009da80 auth_method 0 2024-09-11T01:51:02.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f3434e54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:02.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f3434e54640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:02.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f3434e54640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:02.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f3435655640 1 -- 172.21.15.12:0/3047612433 >> v2:172.21.15.12:3300/0 conn(0x7f34300a32f0 msgr2=0x7f34300a3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:51:02.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f3435655640 1 -- 172.21.15.12:0/3047612433 >> v2:172.21.15.12:3300/0 conn(0x7f34300a32f0 msgr2=0x7f34300a3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:02.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f3434e54640 1 -- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343009da80 msgr2=0x7f3430136010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:51:02.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f3435655640 1 --2- 172.21.15.12:0/3047612433 >> v2:172.21.15.12:3300/0 conn(0x7f34300a32f0 0x7f34300a3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:02.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f3434e54640 1 -- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343009da80 msgr2=0x7f3430136010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:02.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f3434e54640 1 --2- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343009da80 0x7f3430136010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:02.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f3435655640 1 --2- 172.21.15.12:0/3047612433 >> v2:172.21.15.12:3300/0 conn(0x7f34300a32f0 0x7f34300a3720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:51:02.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f3434e54640 1 --2- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343009da80 0x7f3430136010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:51:02.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f342ffff640 1 --2- 172.21.15.12:0/3047612433 >> v2:172.21.15.71:3300/0 conn(0x7f343009de60 0x7f34300a2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:02.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f342ffff640 10 monclient(hunting): get_auth_request con 0x7f343009de60 auth_method 0 2024-09-11T01:51:02.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f342ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:02.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f342ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:02.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f342ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:02.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f342ffff640 1 -- 172.21.15.12:0/3047612433 >> v2:172.21.15.71:3300/0 conn(0x7f343009de60 msgr2=0x7f34300a2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:51:02.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f342ffff640 1 -- 172.21.15.12:0/3047612433 >> v2:172.21.15.71:3300/0 conn(0x7f343009de60 msgr2=0x7f34300a2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:02.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f342ffff640 1 --2- 172.21.15.12:0/3047612433 >> v2:172.21.15.71:3300/0 conn(0x7f343009de60 0x7f34300a2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:02.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.178+0000 7f342ffff640 1 --2- 172.21.15.12:0/3047612433 >> v2:172.21.15.71:3300/0 conn(0x7f343009de60 0x7f34300a2db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:51:02.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.979+0000 7f3435655640 1 --2- 172.21.15.12:0/3047612433 >> v2:172.21.15.12:3300/0 conn(0x7f34300a32f0 0x7f34300a3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:02.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.979+0000 7f3435655640 10 monclient(hunting): get_auth_request con 0x7f34300a32f0 auth_method 0 2024-09-11T01:51:02.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.979+0000 7f3435655640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:02.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.979+0000 7f3435655640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:02.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.979+0000 7f3435655640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:02.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.979+0000 7f3435655640 1 -- 172.21.15.12:0/3047612433 >> v2:172.21.15.12:3300/0 conn(0x7f34300a32f0 msgr2=0x7f34300a3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:51:02.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.979+0000 7f3435655640 1 -- 172.21.15.12:0/3047612433 >> v2:172.21.15.12:3300/0 conn(0x7f34300a32f0 msgr2=0x7f34300a3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:02.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.979+0000 7f3435655640 1 --2- 172.21.15.12:0/3047612433 >> v2:172.21.15.12:3300/0 conn(0x7f34300a32f0 0x7f34300a3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:02.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.979+0000 7f3435655640 1 --2- 172.21.15.12:0/3047612433 >> v2:172.21.15.12:3300/0 conn(0x7f34300a32f0 0x7f34300a3720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:51:02.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.979+0000 7f3434e54640 1 --2- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343009da80 0x7f3430136010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:02.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.979+0000 7f3434e54640 10 monclient(hunting): get_auth_request con 0x7f343009da80 auth_method 0 2024-09-11T01:51:02.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.979+0000 7f3434e54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:02.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.979+0000 7f3434e54640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:02.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.979+0000 7f3434e54640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:02.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.979+0000 7f3434e54640 1 -- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343009da80 msgr2=0x7f3430136010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:51:02.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.979+0000 7f3434e54640 1 -- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343009da80 msgr2=0x7f3430136010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:02.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.979+0000 7f3434e54640 1 --2- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343009da80 0x7f3430136010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:02.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.979+0000 7f3434e54640 1 --2- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343009da80 0x7f3430136010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:51:02.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.980+0000 7f342ffff640 1 --2- 172.21.15.12:0/3047612433 >> v2:172.21.15.71:3300/0 conn(0x7f343009de60 0x7f34300a2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:02.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.980+0000 7f342ffff640 10 monclient(hunting): get_auth_request con 0x7f343009de60 auth_method 0 2024-09-11T01:51:02.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.980+0000 7f342ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:02.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.980+0000 7f342ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:02.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.980+0000 7f342ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:02.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.980+0000 7f342ffff640 1 -- 172.21.15.12:0/3047612433 >> v2:172.21.15.71:3300/0 conn(0x7f343009de60 msgr2=0x7f34300a2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:51:02.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.980+0000 7f342ffff640 1 -- 172.21.15.12:0/3047612433 >> v2:172.21.15.71:3300/0 conn(0x7f343009de60 msgr2=0x7f34300a2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:02.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.980+0000 7f342ffff640 1 --2- 172.21.15.12:0/3047612433 >> v2:172.21.15.71:3300/0 conn(0x7f343009de60 0x7f34300a2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:02.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:02.980+0000 7f342ffff640 1 --2- 172.21.15.12:0/3047612433 >> v2:172.21.15.71:3300/0 conn(0x7f343009de60 0x7f34300a2db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:51:04.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:04.576+0000 7f342e7fc640 10 monclient(hunting): tick 2024-09-11T01:51:04.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:04.576+0000 7f342e7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:51:04.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:04.576+0000 7f342e7fc640 1 monclient(hunting): continuing hunt 2024-09-11T01:51:04.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:04.576+0000 7f342e7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:51:04.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:04.576+0000 7f342e7fc640 1 -- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343009da80 msgr2=0x7f3430136010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:51:04.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:04.576+0000 7f342e7fc640 1 --2- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343009da80 0x7f3430136010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:04.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:04.576+0000 7f342e7fc640 1 -- 172.21.15.12:0/3047612433 >> v2:172.21.15.71:3300/0 conn(0x7f343009de60 msgr2=0x7f34300a2db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:51:04.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:04.576+0000 7f342e7fc640 1 --2- 172.21.15.12:0/3047612433 >> v2:172.21.15.71:3300/0 conn(0x7f343009de60 0x7f34300a2db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:04.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:04.576+0000 7f342e7fc640 1 -- 172.21.15.12:0/3047612433 >> v2:172.21.15.12:3300/0 conn(0x7f34300a32f0 msgr2=0x7f34300a3720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:51:04.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:04.576+0000 7f342e7fc640 1 --2- 172.21.15.12:0/3047612433 >> v2:172.21.15.12:3300/0 conn(0x7f34300a32f0 0x7f34300a3720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:04.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:04.576+0000 7f342e7fc640 10 monclient: _add_conns ranks=[0,1] 2024-09-11T01:51:04.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:04.576+0000 7f342e7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f34200036b0 addr v1:172.21.15.12:6789/0 2024-09-11T01:51:04.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:04.576+0000 7f342e7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f34200073a0 addr v1:172.21.15.71:6789/0 2024-09-11T01:51:04.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:04.576+0000 7f342e7fc640 1 -- 172.21.15.12:0/3047612433 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f343006d2d0 con 0x7f34200036b0 2024-09-11T01:51:04.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:04.576+0000 7f342e7fc640 1 -- 172.21.15.12:0/3047612433 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f342000c0e0 con 0x7f34200073a0 2024-09-11T01:51:04.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:04.576+0000 7f342e7fc640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:04.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:04.578+0000 7f3434e54640 1 -- 172.21.15.12:0/3047612433 >> v1:172.21.15.12:6789/0 conn(0x7f34200036b0 legacy=0x7f3420005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:04.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:04.579+0000 7f342ffff640 1 -- 172.21.15.12:0/3047612433 >> v1:172.21.15.71:6789/0 conn(0x7f34200073a0 legacy=0x7f3420009860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:06.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.854+0000 7f342f7fe640 1 -- 172.21.15.12:0/3047612433 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3706573016 0 0) 0x7f342000c0e0 con 0x7f34200073a0 2024-09-11T01:51:06.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.854+0000 7f342f7fe640 10 monclient(hunting): _negotiate 2024-09-11T01:51:06.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.854+0000 7f342f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:06.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.854+0000 7f342f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:06.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.854+0000 7f342f7fe640 10 monclient(hunting): my global_id is 234133 2024-09-11T01:51:06.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.854+0000 7f342f7fe640 1 -- 172.21.15.12:0/3047612433 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3414003520 con 0x7f34200073a0 2024-09-11T01:51:06.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 1 -- 172.21.15.12:0/3047612433 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3056862733 0 0) 0x7f3414003520 con 0x7f34200073a0 2024-09-11T01:51:06.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 1 -- 172.21.15.12:0/3047612433 >> v1:172.21.15.12:6789/0 conn(0x7f34200036b0 legacy=0x7f3420005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:06.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:51:06.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 1 monclient: found mon.noname-c-legacy 2024-09-11T01:51:06.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:06.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-09-11T01:51:06.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 1 -- 172.21.15.12:0/3047612433 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f342000d330 con 0x7f34200073a0 2024-09-11T01:51:06.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 10 monclient: _finish_auth 0 2024-09-11T01:51:06.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:51:06.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:06.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 1 -- 172.21.15.12:0/3047612433 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f341c00d680 con 0x7f34200073a0 2024-09-11T01:51:06.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:06.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-11T01:51:06.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 10 monclient: dump: 2024-09-11T01:51:06.861 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:06.862 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:06.862 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:06.862 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:06.862 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:06.862 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:06.862 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:06.862 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:06.862 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:06.862 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:06.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-09-11T01:51:06.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:06.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 1 -- 172.21.15.12:0/3047612433 >> v1:172.21.15.71:6789/0 conn(0x7f34200073a0 legacy=0x7f3420009860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f3435655640 1 -- 172.21.15.12:0/3047612433 reap_dead start 2024-09-11T01:51:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:51:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 1 --2- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3414003af0 0x7f3414003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 10 monclient(hunting): picked mon.c con 0x7f3414003af0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 1 --2- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3414004800 0x7f3414004c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 10 monclient(hunting): picked mon.a con 0x7f3414004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:51:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 1 --2- 172.21.15.12:0/3047612433 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f34140099b0 0x7f341400bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 10 monclient(hunting): picked mon.b con 0x7f34140099b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:51:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 1 -- 172.21.15.12:0/3047612433 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f341400c410 con 0x7f3414004800 2024-09-11T01:51:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 1 -- 172.21.15.12:0/3047612433 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f341400c5c0 con 0x7f34140099b0 2024-09-11T01:51:06.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:06.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 1 -- 172.21.15.12:0/3047612433 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f341400c7c0 con 0x7f3414003af0 2024-09-11T01:51:06.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:06.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:51:06.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f3434e54640 1 --2- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3414003af0 0x7f3414003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:06.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f3434e54640 10 monclient(hunting): get_auth_request con 0x7f3414003af0 auth_method 0 2024-09-11T01:51:06.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f3434e54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:06.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f3434e54640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:06.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f3434e54640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:06.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f3435655640 1 --2- 172.21.15.12:0/3047612433 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f34140099b0 0x7f341400bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:06.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f3435655640 10 monclient(hunting): get_auth_request con 0x7f34140099b0 auth_method 0 2024-09-11T01:51:06.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f3435655640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:06.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f3435655640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:06.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f3435655640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:06.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f3434e54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:06.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f3434e54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:06.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342ffff640 1 --2- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3414004800 0x7f3414004c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:06.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f3434e54640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:51:06.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342ffff640 10 monclient(hunting): get_auth_request con 0x7f3414004800 auth_method 0 2024-09-11T01:51:06.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:06.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:06.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:06.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f3435655640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:06.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f3435655640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:06.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f3435655640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:51:06.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:06.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:06.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.855+0000 7f342ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:51:06.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.856+0000 7f3434e54640 10 monclient(hunting): handle_auth_done global_id 234133 payload 293 2024-09-11T01:51:06.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.856+0000 7f3434e54640 1 -- 172.21.15.12:0/3047612433 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f34140099b0 msgr2=0x7f341400bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:06.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.856+0000 7f3434e54640 1 --2- 172.21.15.12:0/3047612433 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f34140099b0 0x7f341400bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:06.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.856+0000 7f3434e54640 1 -- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3414004800 msgr2=0x7f3414004c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:06.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.856+0000 7f3434e54640 1 --2- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3414004800 0x7f3414004c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:06.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.856+0000 7f3434e54640 10 monclient: _finish_hunting 0 2024-09-11T01:51:06.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.856+0000 7f3434e54640 1 monclient: found mon.c 2024-09-11T01:51:06.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.856+0000 7f3434e54640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:06.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.856+0000 7f3434e54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:06.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.856+0000 7f3434e54640 1 -- 172.21.15.12:0/3047612433 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3414003520 con 0x7f3414003af0 2024-09-11T01:51:06.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.856+0000 7f3434e54640 10 monclient: _finish_auth 0 2024-09-11T01:51:06.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.856+0000 7f3434e54640 10 monclient: _check_auth_tickets 2024-09-11T01:51:06.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.856+0000 7f3434e54640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:06.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.856+0000 7f34370b8640 5 monclient: authenticate success, global_id 234133 2024-09-11T01:51:06.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.856+0000 7f34370b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:51:06.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.856+0000 7f3435655640 1 --2- 172.21.15.12:0/3047612433 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f34140099b0 0x7f341400bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:51:06.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.859+0000 7f3434e54640 1 -- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3414003af0 msgr2=0x7f3414003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:06.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.859+0000 7f3434e54640 1 --2- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3414003af0 0x7f3414003ef0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3424009100 tx=0x7f3414005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87ae861b0714eb86 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:06.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.859+0000 7f342f7fe640 1 -- 172.21.15.12:0/3047612433 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3424031070 con 0x7f3414003af0 2024-09-11T01:51:06.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.859+0000 7f342f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:06.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.859+0000 7f342f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:06.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.859+0000 7f342f7fe640 10 monclient: dump: 2024-09-11T01:51:06.873 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:06.874 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:06.874 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:06.874 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:06.874 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:06.874 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:06.874 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:06.874 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:06.874 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:06.874 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:06.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.859+0000 7f34370b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:51:06.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.859+0000 7f342f7fe640 1 -- 172.21.15.12:0/3047612433 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f342402ba00 con 0x7f3414003af0 2024-09-11T01:51:06.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.859+0000 7f342f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:06.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.859+0000 7f342f7fe640 1 -- 172.21.15.12:0/3047612433 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342400f040 con 0x7f3414003af0 2024-09-11T01:51:06.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.859+0000 7f342f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:06.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.859+0000 7f342f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:06.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.859+0000 7f342f7fe640 10 monclient: dump: 2024-09-11T01:51:06.875 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:06.875 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:06.875 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:06.875 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:06.875 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:06.875 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:06.875 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:06.875 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:06.875 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:06.875 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:06.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.859+0000 7f34370b8640 10 monclient: get_monmap_and_config success 2024-09-11T01:51:06.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.859+0000 7f34370b8640 4 set_mon_vals no callback set 2024-09-11T01:51:06.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.860+0000 7f34370b8640 10 monclient: shutdown 2024-09-11T01:51:06.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.860+0000 7f34370b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:06.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.860+0000 7f34370b8640 1 -- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3414003af0 msgr2=0x7f3414003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:06.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.860+0000 7f34370b8640 1 --2- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3414003af0 0x7f3414003ef0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3424009100 tx=0x7f3414005830 comp rx=0 tx=0).stop 2024-09-11T01:51:06.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.860+0000 7f34370b8640 1 -- 172.21.15.12:0/3047612433 shutdown_connections 2024-09-11T01:51:06.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.860+0000 7f34370b8640 1 --2- 172.21.15.12:0/3047612433 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f34140099b0 0x7f341400bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:06.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.860+0000 7f34370b8640 1 --2- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3414004800 0x7f3414004c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:06.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.860+0000 7f34370b8640 1 --2- 172.21.15.12:0/3047612433 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3414003af0 0x7f3414003ef0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:06.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.860+0000 7f34370b8640 1 -- 172.21.15.12:0/3047612433 >> 172.21.15.12:0/3047612433 conn(0x7f34300a3ea0 msgr2=0x7f34300a42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:06.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.860+0000 7f34370b8640 1 -- 172.21.15.12:0/3047612433 shutdown_connections 2024-09-11T01:51:06.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.860+0000 7f34370b8640 1 -- 172.21.15.12:0/3047612433 wait complete. 2024-09-11T01:51:06.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.861+0000 7f34370b8640 10 monclient: build_initial_monmap 2024-09-11T01:51:06.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.861+0000 7f34370b8640 10 monclient: monmap: 2024-09-11T01:51:06.877 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:06.877 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:51:06.877 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:51:06.877 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:51:06.877 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:06.877 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:06.877 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:51:06.877 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:51:06.877 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:51:06.877 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:06.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.862+0000 7f34370b8640 1 Processor -- start 2024-09-11T01:51:06.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.862+0000 7f34370b8640 1 -- start start 2024-09-11T01:51:06.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.862+0000 7f34370b8640 10 monclient: init 2024-09-11T01:51:06.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.862+0000 7f34370b8640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:06.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.862+0000 7f34370b8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:51:06.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.862+0000 7f34370b8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3414003af0 0x7f3430073cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:06.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.862+0000 7f34370b8640 10 monclient(hunting): picked mon.noname-c con 0x7f3414003af0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:06.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.862+0000 7f34370b8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3414004800 0x7f343012ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:06.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.862+0000 7f34370b8640 10 monclient(hunting): picked mon.noname-b con 0x7f3414004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:51:06.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.862+0000 7f34370b8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34140099b0 0x7f343012d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:06.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.862+0000 7f34370b8640 10 monclient(hunting): picked mon.noname-a con 0x7f34140099b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:51:06.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.862+0000 7f34370b8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:06.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.862+0000 7f34370b8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f34300a0ea0 con 0x7f34140099b0 2024-09-11T01:51:06.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.862+0000 7f34370b8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:06.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.862+0000 7f34370b8640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f34300a0d20 con 0x7f3414004800 2024-09-11T01:51:06.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.862+0000 7f34370b8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:06.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.862+0000 7f34370b8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f343009e240 con 0x7f3414003af0 2024-09-11T01:51:06.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.862+0000 7f34370b8640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:06.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.862+0000 7f34370b8640 10 monclient(hunting): authenticate will time out at 2073.304945s 2024-09-11T01:51:06.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3434e54640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3414003af0 0x7f3430073cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:06.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3435655640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34140099b0 0x7f343012d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:06.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3434e54640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3414003af0 0x7f3430073cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:47878/0 (socket says 172.21.15.12:47878) 2024-09-11T01:51:06.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3434e54640 1 -- 172.21.15.12:0/3975729829 learned_addr learned my addr 172.21.15.12:0/3975729829 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:06.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3435655640 10 monclient(hunting): get_auth_request con 0x7f34140099b0 auth_method 0 2024-09-11T01:51:06.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3435655640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:06.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3435655640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:06.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3435655640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:06.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3434e54640 10 monclient(hunting): get_auth_request con 0x7f3414003af0 auth_method 0 2024-09-11T01:51:06.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3434e54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:06.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3434e54640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:06.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3434e54640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:06.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f342ffff640 1 --2- 172.21.15.12:0/3975729829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3414004800 0x7f343012ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:06.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f342ffff640 10 monclient(hunting): get_auth_request con 0x7f3414004800 auth_method 0 2024-09-11T01:51:06.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f342ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:06.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f342ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:06.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f342ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:06.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3435655640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:06.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3435655640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:06.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3435655640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:06.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f342ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:06.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f342ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:06.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3434e54640 1 -- 172.21.15.12:0/3975729829 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3414003af0 msgr2=0x7f3430073cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:51:06.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3434e54640 1 -- 172.21.15.12:0/3975729829 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3414003af0 msgr2=0x7f3430073cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:51:06.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3434e54640 1 --2- 172.21.15.12:0/3975729829 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3414003af0 0x7f3430073cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:06.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f342ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:06.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3434e54640 1 --2- 172.21.15.12:0/3975729829 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3414003af0 0x7f3430073cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:51:06.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3435655640 10 monclient(hunting): handle_auth_done global_id 214377 payload 824 2024-09-11T01:51:06.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3435655640 1 -- 172.21.15.12:0/3975729829 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3414003af0 msgr2=0x7f3430073cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:51:06.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3435655640 1 --2- 172.21.15.12:0/3975729829 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3414003af0 0x7f3430073cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:06.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3435655640 1 -- 172.21.15.12:0/3975729829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3414004800 msgr2=0x7f343012ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:06.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3435655640 1 --2- 172.21.15.12:0/3975729829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3414004800 0x7f343012ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:06.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3435655640 10 monclient: _finish_hunting 0 2024-09-11T01:51:06.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3435655640 1 monclient: found mon.noname-a 2024-09-11T01:51:06.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3435655640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:06.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3435655640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:51:06.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f3435655640 1 -- 172.21.15.12:0/3975729829 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34300714b0 con 0x7f34140099b0 2024-09-11T01:51:06.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.863+0000 7f342ffff640 1 --2- 172.21.15.12:0/3975729829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3414004800 0x7f343012ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:51:06.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.865+0000 7f3435655640 1 -- 172.21.15.12:0/3975729829 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34140099b0 msgr2=0x7f343012d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:06.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.865+0000 7f3435655640 1 --2- 172.21.15.12:0/3975729829 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34140099b0 0x7f343012d3f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f343009b680 tx=0x7f3428016710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:06.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.866+0000 7f342d7fa640 1 -- 172.21.15.12:0/3975729829 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342801c070 con 0x7f34140099b0 2024-09-11T01:51:06.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.866+0000 7f342d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:06.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.866+0000 7f342d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:51:06.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.866+0000 7f342d7fa640 10 monclient: dump: 2024-09-11T01:51:06.882 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:06.882 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:06.882 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:06.882 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:06.883 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:06.883 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:06.883 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:06.883 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:06.883 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:06.883 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:06.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.866+0000 7f342d7fa640 10 monclient: _finish_auth 0 2024-09-11T01:51:06.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.866+0000 7f342d7fa640 10 monclient: _check_auth_tickets 2024-09-11T01:51:06.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.866+0000 7f342d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:06.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.866+0000 7f34370b8640 5 monclient: authenticate success, global_id 214377 2024-09-11T01:51:06.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.866+0000 7f342d7fa640 1 -- 172.21.15.12:0/3975729829 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3428021460 con 0x7f34140099b0 2024-09-11T01:51:06.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.866+0000 7f342d7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:06.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.866+0000 7f342d7fa640 1 -- 172.21.15.12:0/3975729829 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3428021830 con 0x7f34140099b0 2024-09-11T01:51:06.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.866+0000 7f34370b8640 10 monclient: _renew_subs 2024-09-11T01:51:06.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.866+0000 7f34370b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:51:06.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.866+0000 7f34370b8640 1 -- 172.21.15.12:0/3975729829 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f343012f200 con 0x7f34140099b0 2024-09-11T01:51:06.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.866+0000 7f342d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:06.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.866+0000 7f342d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:51:06.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.866+0000 7f342d7fa640 10 monclient: dump: 2024-09-11T01:51:06.884 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:06.884 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:06.884 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:06.884 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:06.884 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:06.884 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:06.884 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:06.884 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:06.885 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:06.885 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:06.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.866+0000 7f342effd640 4 set_mon_vals no callback set 2024-09-11T01:51:06.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.866+0000 7f34370b8640 10 monclient: _renew_subs 2024-09-11T01:51:06.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.866+0000 7f34370b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:51:06.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.866+0000 7f34370b8640 1 -- 172.21.15.12:0/3975729829 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f343009cd80 con 0x7f34140099b0 2024-09-11T01:51:06.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.867+0000 7f342d7fa640 1 -- 172.21.15.12:0/3975729829 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f3428018420 con 0x7f34140099b0 2024-09-11T01:51:06.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.867+0000 7f34370b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:06.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.867+0000 7f34370b8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:06.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.867+0000 7f34370b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:51:06.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.867+0000 7f34370b8640 1 -- 172.21.15.12:0/3975729829 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f343009e590 con 0x7f34140099b0 2024-09-11T01:51:06.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.867+0000 7f342d7fa640 1 --2- 172.21.15.12:0/3975729829 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f342003eb20 0x7f3420040fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:06.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.867+0000 7f342d7fa640 1 -- 172.21.15.12:0/3975729829 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(729..729 src has 507..729) ==== 4771+0+0 (secure 0 0 0) 0x7f3428054110 con 0x7f34140099b0 2024-09-11T01:51:06.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.868+0000 7f3434e54640 1 --2- 172.21.15.12:0/3975729829 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f342003eb20 0x7f3420040fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:06.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.868+0000 7f3434e54640 10 monclient: get_auth_request con 0x7f342003eb20 auth_method 0 2024-09-11T01:51:06.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.870+0000 7f3434e54640 1 --2- 172.21.15.12:0/3975729829 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f342003eb20 0x7f3420040fe0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f3424009100 tx=0x7f3424031000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:06.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.872+0000 7f342d7fa640 1 -- 172.21.15.12:0/3975729829 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3428026060 con 0x7f34140099b0 2024-09-11T01:51:06.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.872+0000 7f342d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:06.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:06.872+0000 7f342d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:51:06.914 INFO:tasks.rados.rados.0.smithi071.stdout:3257: done (1 left) 2024-09-11T01:51:06.914 INFO:tasks.rados.rados.0.smithi071.stdout:3261: done (0 left) 2024-09-11T01:51:06.914 INFO:tasks.rados.rados.0.smithi071.stdout:3262: read oid 24 snap 353 2024-09-11T01:51:06.914 INFO:tasks.rados.rados.0.smithi071.stdout:3262: expect (ObjNum 1060 snap 340 seq_num 1060) 2024-09-11T01:51:06.914 INFO:tasks.rados.rados.0.smithi071.stdout:3263: snap_remove snap 347 2024-09-11T01:51:07.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:07.136+0000 7f34370b8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:51:07.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:07.136+0000 7f34370b8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:51:07.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:07.136+0000 7f34370b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:51:07.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:07.136+0000 7f34370b8640 1 -- 172.21.15.12:0/3975729829 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f343006e570 con 0x7f34140099b0 2024-09-11T01:51:07.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:07.137+0000 7f342d7fa640 1 -- 172.21.15.12:0/3975729829 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f3428052b10 con 0x7f34140099b0 2024-09-11T01:51:07.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:07.137+0000 7f342d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:51:07.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:07.137+0000 7f342d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:51:07.139 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":202,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:51:07.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:07.139+0000 7f34370b8640 1 -- 172.21.15.12:0/3975729829 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f342003eb20 msgr2=0x7f3420040fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:07.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:07.139+0000 7f34370b8640 1 --2- 172.21.15.12:0/3975729829 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f342003eb20 0x7f3420040fe0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f3424009100 tx=0x7f3424031000 comp rx=0 tx=0).stop 2024-09-11T01:51:07.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:07.139+0000 7f34370b8640 10 monclient: shutdown 2024-09-11T01:51:07.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:07.139+0000 7f34370b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:07.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:07.139+0000 7f34370b8640 1 -- 172.21.15.12:0/3975729829 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34140099b0 msgr2=0x7f343012d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:07.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:07.139+0000 7f34370b8640 1 --2- 172.21.15.12:0/3975729829 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34140099b0 0x7f343012d3f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f343009b680 tx=0x7f3428016710 comp rx=0 tx=0).stop 2024-09-11T01:51:07.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:07.139+0000 7f34370b8640 1 -- 172.21.15.12:0/3975729829 shutdown_connections 2024-09-11T01:51:07.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:07.139+0000 7f34370b8640 1 --2- 172.21.15.12:0/3975729829 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f342003eb20 0x7f3420040fe0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:07.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:07.139+0000 7f34370b8640 1 --2- 172.21.15.12:0/3975729829 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34140099b0 0x7f343012d3f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:07.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:07.139+0000 7f34370b8640 1 --2- 172.21.15.12:0/3975729829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3414004800 0x7f343012ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:07.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:07.139+0000 7f34370b8640 1 --2- 172.21.15.12:0/3975729829 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3414003af0 0x7f3430073cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:07.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:07.139+0000 7f34370b8640 1 -- 172.21.15.12:0/3975729829 >> 172.21.15.12:0/3975729829 conn(0x7f34300a3ea0 msgr2=0x7f343009e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:07.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:07.139+0000 7f34370b8640 1 -- 172.21.15.12:0/3975729829 shutdown_connections 2024-09-11T01:51:07.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:07.139+0000 7f34370b8640 1 -- 172.21.15.12:0/3975729829 wait complete. 2024-09-11T01:51:10.151 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:51:10.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.263+0000 7fc1a9b24640 10 monclient: get_monmap_and_config 2024-09-11T01:51:10.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.263+0000 7fc1a9b24640 10 monclient: build_initial_monmap 2024-09-11T01:51:10.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.263+0000 7fc1a9b24640 10 monclient: monmap: 2024-09-11T01:51:10.265 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:10.265 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:10.265 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:51:10.263967+0000 2024-09-11T01:51:10.265 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:51:10.263967+0000 2024-09-11T01:51:10.265 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:10.265 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:10.265 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:51:10.265 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:51:10.265 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:51:10.266 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:51:10.266 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:51:10.266 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:10.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a9b24640 1 Processor -- start 2024-09-11T01:51:10.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a9b24640 1 -- start start 2024-09-11T01:51:10.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a9b24640 10 monclient: init 2024-09-11T01:51:10.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a9b24640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:10.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a9b24640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-11T01:51:10.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a9b24640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1a4138230 0x7fc1a4138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:10.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a9b24640 10 monclient(hunting): picked mon.noname-b con 0x7fc1a4138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:10.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a9b24640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc1a409d030 0x7fc1a40a5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:10.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a9b24640 10 monclient(hunting): picked mon.noname-c con 0x7fc1a409d030 addr v2:172.21.15.71:3300/0 2024-09-11T01:51:10.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a9b24640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc1a409d430 0x7fc1a40a5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:10.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a9b24640 10 monclient(hunting): picked mon.noname-a con 0x7fc1a409d430 addr v2:172.21.15.12:3300/0 2024-09-11T01:51:10.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a9b24640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:10.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a9b24640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc1a409c9c0 con 0x7fc1a409d430 2024-09-11T01:51:10.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a9b24640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:10.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a9b24640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fc1a40a6e30 con 0x7fc1a409d030 2024-09-11T01:51:10.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a9b24640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:10.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a9b24640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc1a409ba20 con 0x7fc1a4138230 2024-09-11T01:51:10.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a9b24640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:10.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a9b24640 10 monclient(hunting): authenticate will time out at 2076.706676s 2024-09-11T01:51:10.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a37fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1a4138230 0x7fc1a4138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:10.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a3fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc1a409d430 0x7fc1a40a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:10.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a37fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1a4138230 0x7fc1a4138610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:47888/0 (socket says 172.21.15.12:47888) 2024-09-11T01:51:10.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a37fe640 1 -- 172.21.15.12:0/1076163856 learned_addr learned my addr 172.21.15.12:0/1076163856 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:10.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a37fe640 10 monclient(hunting): get_auth_request con 0x7fc1a4138230 auth_method 0 2024-09-11T01:51:10.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:10.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:10.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:10.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a3fff640 10 monclient(hunting): get_auth_request con 0x7fc1a409d430 auth_method 0 2024-09-11T01:51:10.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:10.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:10.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:10.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a37fe640 1 -- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1a4138230 msgr2=0x7fc1a4138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:51:10.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a3fff640 1 -- 172.21.15.12:0/1076163856 >> v2:172.21.15.12:3300/0 conn(0x7fc1a409d430 msgr2=0x7fc1a40a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:51:10.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.264+0000 7fc1a37fe640 1 -- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1a4138230 msgr2=0x7fc1a4138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:10.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.265+0000 7fc1a3fff640 1 -- 172.21.15.12:0/1076163856 >> v2:172.21.15.12:3300/0 conn(0x7fc1a409d430 msgr2=0x7fc1a40a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:10.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.265+0000 7fc1a37fe640 1 --2- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1a4138230 0x7fc1a4138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:10.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.265+0000 7fc1a3fff640 1 --2- 172.21.15.12:0/1076163856 >> v2:172.21.15.12:3300/0 conn(0x7fc1a409d430 0x7fc1a40a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:10.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.265+0000 7fc1a3fff640 1 --2- 172.21.15.12:0/1076163856 >> v2:172.21.15.12:3300/0 conn(0x7fc1a409d430 0x7fc1a40a5560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:51:10.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.265+0000 7fc1a37fe640 1 --2- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1a4138230 0x7fc1a4138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:51:10.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.265+0000 7fc1a2ffd640 1 --2- 172.21.15.12:0/1076163856 >> v2:172.21.15.71:3300/0 conn(0x7fc1a409d030 0x7fc1a40a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:10.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.265+0000 7fc1a2ffd640 10 monclient(hunting): get_auth_request con 0x7fc1a409d030 auth_method 0 2024-09-11T01:51:10.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.265+0000 7fc1a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:10.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.265+0000 7fc1a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:10.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.265+0000 7fc1a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:10.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.265+0000 7fc1a2ffd640 1 -- 172.21.15.12:0/1076163856 >> v2:172.21.15.71:3300/0 conn(0x7fc1a409d030 msgr2=0x7fc1a40a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:51:10.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.265+0000 7fc1a2ffd640 1 -- 172.21.15.12:0/1076163856 >> v2:172.21.15.71:3300/0 conn(0x7fc1a409d030 msgr2=0x7fc1a40a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:10.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.265+0000 7fc1a2ffd640 1 --2- 172.21.15.12:0/1076163856 >> v2:172.21.15.71:3300/0 conn(0x7fc1a409d030 0x7fc1a40a5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:10.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.265+0000 7fc1a2ffd640 1 --2- 172.21.15.12:0/1076163856 >> v2:172.21.15.71:3300/0 conn(0x7fc1a409d030 0x7fc1a40a5020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:51:10.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.466+0000 7fc1a37fe640 1 --2- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1a4138230 0x7fc1a4138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:10.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.466+0000 7fc1a3fff640 1 --2- 172.21.15.12:0/1076163856 >> v2:172.21.15.12:3300/0 conn(0x7fc1a409d430 0x7fc1a40a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:10.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.466+0000 7fc1a37fe640 10 monclient(hunting): get_auth_request con 0x7fc1a4138230 auth_method 0 2024-09-11T01:51:10.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.466+0000 7fc1a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:10.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.466+0000 7fc1a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:10.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.466+0000 7fc1a37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:10.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.466+0000 7fc1a3fff640 10 monclient(hunting): get_auth_request con 0x7fc1a409d430 auth_method 0 2024-09-11T01:51:10.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.466+0000 7fc1a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:10.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.466+0000 7fc1a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:10.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.466+0000 7fc1a3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:10.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.466+0000 7fc1a37fe640 1 -- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1a4138230 msgr2=0x7fc1a4138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:51:10.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.466+0000 7fc1a37fe640 1 -- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1a4138230 msgr2=0x7fc1a4138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:10.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.466+0000 7fc1a37fe640 1 --2- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1a4138230 0x7fc1a4138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:10.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.466+0000 7fc1a3fff640 1 -- 172.21.15.12:0/1076163856 >> v2:172.21.15.12:3300/0 conn(0x7fc1a409d430 msgr2=0x7fc1a40a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:51:10.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.466+0000 7fc1a3fff640 1 -- 172.21.15.12:0/1076163856 >> v2:172.21.15.12:3300/0 conn(0x7fc1a409d430 msgr2=0x7fc1a40a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:10.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.466+0000 7fc1a3fff640 1 --2- 172.21.15.12:0/1076163856 >> v2:172.21.15.12:3300/0 conn(0x7fc1a409d430 0x7fc1a40a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:10.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.466+0000 7fc1a37fe640 1 --2- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1a4138230 0x7fc1a4138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:51:10.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.466+0000 7fc1a3fff640 1 --2- 172.21.15.12:0/1076163856 >> v2:172.21.15.12:3300/0 conn(0x7fc1a409d430 0x7fc1a40a5560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:51:10.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.466+0000 7fc1a2ffd640 1 --2- 172.21.15.12:0/1076163856 >> v2:172.21.15.71:3300/0 conn(0x7fc1a409d030 0x7fc1a40a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:10.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.466+0000 7fc1a2ffd640 10 monclient(hunting): get_auth_request con 0x7fc1a409d030 auth_method 0 2024-09-11T01:51:10.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.466+0000 7fc1a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:10.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.466+0000 7fc1a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:10.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.466+0000 7fc1a2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:10.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.467+0000 7fc1a2ffd640 1 -- 172.21.15.12:0/1076163856 >> v2:172.21.15.71:3300/0 conn(0x7fc1a409d030 msgr2=0x7fc1a40a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:51:10.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.467+0000 7fc1a2ffd640 1 -- 172.21.15.12:0/1076163856 >> v2:172.21.15.71:3300/0 conn(0x7fc1a409d030 msgr2=0x7fc1a40a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:10.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.467+0000 7fc1a2ffd640 1 --2- 172.21.15.12:0/1076163856 >> v2:172.21.15.71:3300/0 conn(0x7fc1a409d030 0x7fc1a40a5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:10.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.467+0000 7fc1a2ffd640 1 --2- 172.21.15.12:0/1076163856 >> v2:172.21.15.71:3300/0 conn(0x7fc1a409d030 0x7fc1a40a5020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:51:10.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.867+0000 7fc1a37fe640 1 --2- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1a4138230 0x7fc1a4138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.867+0000 7fc1a3fff640 1 --2- 172.21.15.12:0/1076163856 >> v2:172.21.15.12:3300/0 conn(0x7fc1a409d430 0x7fc1a40a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.867+0000 7fc1a37fe640 10 monclient(hunting): get_auth_request con 0x7fc1a4138230 auth_method 0 2024-09-11T01:51:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.867+0000 7fc1a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.867+0000 7fc1a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.867+0000 7fc1a37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.867+0000 7fc1a3fff640 10 monclient(hunting): get_auth_request con 0x7fc1a409d430 auth_method 0 2024-09-11T01:51:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.867+0000 7fc1a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.867+0000 7fc1a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.867+0000 7fc1a3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.867+0000 7fc1a37fe640 1 -- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1a4138230 msgr2=0x7fc1a4138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:51:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.867+0000 7fc1a37fe640 1 -- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1a4138230 msgr2=0x7fc1a4138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.867+0000 7fc1a37fe640 1 --2- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1a4138230 0x7fc1a4138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:10.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.867+0000 7fc1a3fff640 1 -- 172.21.15.12:0/1076163856 >> v2:172.21.15.12:3300/0 conn(0x7fc1a409d430 msgr2=0x7fc1a40a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:51:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.867+0000 7fc1a3fff640 1 -- 172.21.15.12:0/1076163856 >> v2:172.21.15.12:3300/0 conn(0x7fc1a409d430 msgr2=0x7fc1a40a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.867+0000 7fc1a3fff640 1 --2- 172.21.15.12:0/1076163856 >> v2:172.21.15.12:3300/0 conn(0x7fc1a409d430 0x7fc1a40a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.867+0000 7fc1a37fe640 1 --2- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1a4138230 0x7fc1a4138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:51:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.867+0000 7fc1a3fff640 1 --2- 172.21.15.12:0/1076163856 >> v2:172.21.15.12:3300/0 conn(0x7fc1a409d430 0x7fc1a40a5560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:51:10.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.869+0000 7fc1a2ffd640 1 --2- 172.21.15.12:0/1076163856 >> v2:172.21.15.71:3300/0 conn(0x7fc1a409d030 0x7fc1a40a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.869+0000 7fc1a2ffd640 10 monclient(hunting): get_auth_request con 0x7fc1a409d030 auth_method 0 2024-09-11T01:51:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.869+0000 7fc1a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.869+0000 7fc1a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.869+0000 7fc1a2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.869+0000 7fc1a2ffd640 1 -- 172.21.15.12:0/1076163856 >> v2:172.21.15.71:3300/0 conn(0x7fc1a409d030 msgr2=0x7fc1a40a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:51:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.869+0000 7fc1a2ffd640 1 -- 172.21.15.12:0/1076163856 >> v2:172.21.15.71:3300/0 conn(0x7fc1a409d030 msgr2=0x7fc1a40a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.869+0000 7fc1a2ffd640 1 --2- 172.21.15.12:0/1076163856 >> v2:172.21.15.71:3300/0 conn(0x7fc1a409d030 0x7fc1a40a5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:10.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:10.869+0000 7fc1a2ffd640 1 --2- 172.21.15.12:0/1076163856 >> v2:172.21.15.71:3300/0 conn(0x7fc1a409d030 0x7fc1a40a5020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:51:11.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.668+0000 7fc1a3fff640 1 --2- 172.21.15.12:0/1076163856 >> v2:172.21.15.12:3300/0 conn(0x7fc1a409d430 0x7fc1a40a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.668+0000 7fc1a3fff640 10 monclient(hunting): get_auth_request con 0x7fc1a409d430 auth_method 0 2024-09-11T01:51:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.668+0000 7fc1a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.668+0000 7fc1a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.668+0000 7fc1a3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.668+0000 7fc1a3fff640 1 -- 172.21.15.12:0/1076163856 >> v2:172.21.15.12:3300/0 conn(0x7fc1a409d430 msgr2=0x7fc1a40a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:51:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.668+0000 7fc1a3fff640 1 -- 172.21.15.12:0/1076163856 >> v2:172.21.15.12:3300/0 conn(0x7fc1a409d430 msgr2=0x7fc1a40a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.668+0000 7fc1a3fff640 1 --2- 172.21.15.12:0/1076163856 >> v2:172.21.15.12:3300/0 conn(0x7fc1a409d430 0x7fc1a40a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.668+0000 7fc1a3fff640 1 --2- 172.21.15.12:0/1076163856 >> v2:172.21.15.12:3300/0 conn(0x7fc1a409d430 0x7fc1a40a5560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:51:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.668+0000 7fc1a37fe640 1 --2- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1a4138230 0x7fc1a4138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.668+0000 7fc1a37fe640 10 monclient(hunting): get_auth_request con 0x7fc1a4138230 auth_method 0 2024-09-11T01:51:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.668+0000 7fc1a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:11.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.668+0000 7fc1a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:11.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.668+0000 7fc1a37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:11.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.668+0000 7fc1a37fe640 1 -- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1a4138230 msgr2=0x7fc1a4138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:51:11.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.668+0000 7fc1a37fe640 1 -- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1a4138230 msgr2=0x7fc1a4138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:11.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.668+0000 7fc1a37fe640 1 --2- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1a4138230 0x7fc1a4138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:11.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.668+0000 7fc1a37fe640 1 --2- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1a4138230 0x7fc1a4138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:51:11.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.670+0000 7fc1a2ffd640 1 --2- 172.21.15.12:0/1076163856 >> v2:172.21.15.71:3300/0 conn(0x7fc1a409d030 0x7fc1a40a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:11.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.670+0000 7fc1a2ffd640 10 monclient(hunting): get_auth_request con 0x7fc1a409d030 auth_method 0 2024-09-11T01:51:11.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.670+0000 7fc1a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:11.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.670+0000 7fc1a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:11.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.670+0000 7fc1a2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:11.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.670+0000 7fc1a2ffd640 1 -- 172.21.15.12:0/1076163856 >> v2:172.21.15.71:3300/0 conn(0x7fc1a409d030 msgr2=0x7fc1a40a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:51:11.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.670+0000 7fc1a2ffd640 1 -- 172.21.15.12:0/1076163856 >> v2:172.21.15.71:3300/0 conn(0x7fc1a409d030 msgr2=0x7fc1a40a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:11.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.670+0000 7fc1a2ffd640 1 --2- 172.21.15.12:0/1076163856 >> v2:172.21.15.71:3300/0 conn(0x7fc1a409d030 0x7fc1a40a5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:11.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:11.670+0000 7fc1a2ffd640 1 --2- 172.21.15.12:0/1076163856 >> v2:172.21.15.71:3300/0 conn(0x7fc1a409d030 0x7fc1a40a5020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:51:12.538 INFO:tasks.rados.rados.0.smithi071.stdout:3263: done (1 left) 2024-09-11T01:51:12.538 INFO:tasks.rados.rados.0.smithi071.stdout:3264: write (excl) oid 42 current snap is 364 2024-09-11T01:51:12.538 INFO:tasks.rados.rados.0.smithi071.stdout:3264: seq_num 1143 ranges {410401=662557,1719036=770066,2938716=271685} 2024-09-11T01:51:12.544 INFO:tasks.rados.rados.0.smithi071.stdout:3264: writing smithi07134777-42 from 410401 to 1072958 tid 1 2024-09-11T01:51:12.553 INFO:tasks.rados.rados.0.smithi071.stdout:3264: writing smithi07134777-42 from 1719036 to 2489102 tid 2 2024-09-11T01:51:12.557 INFO:tasks.rados.rados.0.smithi071.stdout:3264: writing smithi07134777-42 from 2938716 to 3210401 tid 3 2024-09-11T01:51:12.557 INFO:tasks.rados.rados.0.smithi071.stdout:3265: delete oid 28 current snap is 364 2024-09-11T01:51:12.558 INFO:tasks.rados.rados.0.smithi071.stdout:3262: expect (ObjNum 1060 snap 340 seq_num 1060) 2024-09-11T01:51:12.558 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:51:12.694 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:51:12.761 INFO:tasks.rados.rados.0.smithi071.stdout:3264: finishing write tid 1 to smithi07134777-42 2024-09-11T01:51:12.761 INFO:tasks.rados.rados.0.smithi071.stdout:3264: finishing write tid 2 to smithi07134777-42 2024-09-11T01:51:12.761 INFO:tasks.rados.rados.0.smithi071.stdout:3264: finishing write tid 3 to smithi07134777-42 2024-09-11T01:51:12.761 INFO:tasks.rados.rados.0.smithi071.stdout:3264: finishing write tid 4 to smithi07134777-42 2024-09-11T01:51:12.761 INFO:tasks.rados.rados.0.smithi071.stdout:3264: finishing write tid 5 to smithi07134777-42 2024-09-11T01:51:12.762 INFO:tasks.rados.rados.0.smithi071.stdout:3264: oid 42 updating version 0 to 1782 2024-09-11T01:51:12.762 INFO:tasks.rados.rados.0.smithi071.stdout:3264: oid 42 updating version 1782 to 1785 2024-09-11T01:51:12.762 INFO:tasks.rados.rados.0.smithi071.stdout:3264: oid 42 version 1785 is already newer than 1784 2024-09-11T01:51:12.762 INFO:tasks.rados.rados.0.smithi071.stdout:3264: oid 42 version 1785 is already newer than 1783 2024-09-11T01:51:12.762 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 1785 (ObjNum 1143 snap 364 seq_num 1143) dirty exists 2024-09-11T01:51:12.762 INFO:tasks.rados.rados.0.smithi071.stdout:3264: left oid 42 (ObjNum 1143 snap 364 seq_num 1143) 2024-09-11T01:51:12.762 INFO:tasks.rados.rados.0.smithi071.stdout:3262: done (2 left) 2024-09-11T01:51:12.762 INFO:tasks.rados.rados.0.smithi071.stdout:3264: done (1 left) 2024-09-11T01:51:12.762 INFO:tasks.rados.rados.0.smithi071.stdout:3265: done (0 left) 2024-09-11T01:51:12.762 INFO:tasks.rados.rados.0.smithi071.stdout:3266: copy_from oid 17 from oid 14 current snap is 364 2024-09-11T01:51:12.762 INFO:tasks.rados.rados.0.smithi071.stdout:3267: delete oid 31 current snap is 364 2024-09-11T01:51:12.763 INFO:tasks.rados.rados.0.smithi071.stdout:3266: finishing copy_from to smithi07134777-17 2024-09-11T01:51:12.763 INFO:tasks.rados.rados.0.smithi071.stdout:3266: got expected ENOENT (src dne) 2024-09-11T01:51:12.763 INFO:tasks.rados.rados.0.smithi071.stdout:3266: finishing copy_from racing read to smithi07134777-17 2024-09-11T01:51:12.764 INFO:tasks.rados.rados.0.smithi071.stdout:3266: done (1 left) 2024-09-11T01:51:12.764 INFO:tasks.rados.rados.0.smithi071.stdout:3267: done (0 left) 2024-09-11T01:51:12.764 INFO:tasks.rados.rados.0.smithi071.stdout:3268: delete oid 41 current snap is 364 2024-09-11T01:51:12.767 INFO:tasks.rados.rados.0.smithi071.stdout:3268: done (0 left) 2024-09-11T01:51:12.767 INFO:tasks.rados.rados.0.smithi071.stdout:3269: read oid 39 snap 284 2024-09-11T01:51:12.767 INFO:tasks.rados.rados.0.smithi071.stdout:3269: expect deleted 2024-09-11T01:51:12.767 INFO:tasks.rados.rados.0.smithi071.stdout:3270: copy_from oid 14 from oid 19 current snap is 364 2024-09-11T01:51:12.767 INFO:tasks.rados.rados.0.smithi071.stdout:3271: write oid 12 current snap is 364 2024-09-11T01:51:12.767 INFO:tasks.rados.rados.0.smithi071.stdout:3271: seq_num 1147 ranges {477629=563553,1836611=777404,3048408=229221} 2024-09-11T01:51:12.772 INFO:tasks.rados.rados.0.smithi071.stdout:3271: writing smithi07134777-12 from 477629 to 1041182 tid 1 2024-09-11T01:51:12.781 INFO:tasks.rados.rados.0.smithi071.stdout:3271: writing smithi07134777-12 from 1836611 to 2614015 tid 2 2024-09-11T01:51:12.785 INFO:tasks.rados.rados.0.smithi071.stdout:3271: writing smithi07134777-12 from 3048408 to 3277629 tid 3 2024-09-11T01:51:12.785 INFO:tasks.rados.rados.0.smithi071.stdout:3272: rollback oid 41 current snap is 364 2024-09-11T01:51:12.785 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 41 to 284 2024-09-11T01:51:12.785 INFO:tasks.rados.rados.0.smithi071.stdout:3270: finishing copy_from racing read to smithi07134777-14 2024-09-11T01:51:12.785 INFO:tasks.rados.rados.0.smithi071.stdout:3271: finishing write tid 1 to smithi07134777-12 2024-09-11T01:51:12.785 INFO:tasks.rados.rados.0.smithi071.stdout:3270: finishing copy_from to smithi07134777-14 2024-09-11T01:51:12.785 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 1796 (ObjNum 1136 snap 360 seq_num 1136) dirty exists 2024-09-11T01:51:12.786 INFO:tasks.rados.rados.0.smithi071.stdout:3269: done (3 left) 2024-09-11T01:51:12.786 INFO:tasks.rados.rados.0.smithi071.stdout:3270: done (2 left) 2024-09-11T01:51:12.786 INFO:tasks.rados.rados.0.smithi071.stdout:3273: snap_create 2024-09-11T01:51:12.787 INFO:tasks.rados.rados.0.smithi071.stdout:3272: finishing rollback tid 1 to smithi07134777-41 2024-09-11T01:51:12.788 INFO:tasks.rados.rados.0.smithi071.stdout:3272: finishing rollback tid 2 to smithi07134777-41 2024-09-11T01:51:12.788 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 1959 (ObjNum 832 snap 254 seq_num 832) dirty exists 2024-09-11T01:51:12.789 INFO:tasks.rados.rados.0.smithi071.stdout:3271: finishing write tid 2 to smithi07134777-12 2024-09-11T01:51:12.789 INFO:tasks.rados.rados.0.smithi071.stdout:3271: finishing write tid 3 to smithi07134777-12 2024-09-11T01:51:12.789 INFO:tasks.rados.rados.0.smithi071.stdout:3271: finishing write tid 4 to smithi07134777-12 2024-09-11T01:51:12.790 INFO:tasks.rados.rados.0.smithi071.stdout:3271: finishing write tid 5 to smithi07134777-12 2024-09-11T01:51:12.790 INFO:tasks.rados.rados.0.smithi071.stdout:3271: oid 12 updating version 0 to 2202 2024-09-11T01:51:12.790 INFO:tasks.rados.rados.0.smithi071.stdout:3271: oid 12 updating version 2202 to 2204 2024-09-11T01:51:12.790 INFO:tasks.rados.rados.0.smithi071.stdout:3271: oid 12 updating version 2204 to 2205 2024-09-11T01:51:12.790 INFO:tasks.rados.rados.0.smithi071.stdout:3271: oid 12 version 2205 is already newer than 2203 2024-09-11T01:51:12.790 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 2205 (ObjNum 1147 snap 364 seq_num 1147) dirty exists 2024-09-11T01:51:12.790 INFO:tasks.rados.rados.0.smithi071.stdout:3271: left oid 12 (ObjNum 1147 snap 364 seq_num 1147) 2024-09-11T01:51:13.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.264+0000 7fc1a17fa640 10 monclient(hunting): tick 2024-09-11T01:51:13.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.264+0000 7fc1a17fa640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:51:13.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.264+0000 7fc1a17fa640 1 monclient(hunting): continuing hunt 2024-09-11T01:51:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.264+0000 7fc1a17fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:51:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.264+0000 7fc1a17fa640 1 -- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1a4138230 msgr2=0x7fc1a4138610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:51:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.264+0000 7fc1a17fa640 1 --2- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1a4138230 0x7fc1a4138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.264+0000 7fc1a17fa640 1 -- 172.21.15.12:0/1076163856 >> v2:172.21.15.71:3300/0 conn(0x7fc1a409d030 msgr2=0x7fc1a40a5020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:51:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.264+0000 7fc1a17fa640 1 --2- 172.21.15.12:0/1076163856 >> v2:172.21.15.71:3300/0 conn(0x7fc1a409d030 0x7fc1a40a5020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.264+0000 7fc1a17fa640 1 -- 172.21.15.12:0/1076163856 >> v2:172.21.15.12:3300/0 conn(0x7fc1a409d430 msgr2=0x7fc1a40a5560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:51:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.264+0000 7fc1a17fa640 1 --2- 172.21.15.12:0/1076163856 >> v2:172.21.15.12:3300/0 conn(0x7fc1a409d430 0x7fc1a40a5560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.264+0000 7fc1a17fa640 10 monclient: _add_conns ranks=[1,0] 2024-09-11T01:51:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.264+0000 7fc1a17fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc18c0036b0 addr v1:172.21.15.71:6789/0 2024-09-11T01:51:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.264+0000 7fc1a17fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc18c0073c0 addr v1:172.21.15.12:6789/0 2024-09-11T01:51:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.264+0000 7fc1a17fa640 1 -- 172.21.15.12:0/1076163856 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1a40a66a0 con 0x7fc18c0073c0 2024-09-11T01:51:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.264+0000 7fc1a17fa640 1 -- 172.21.15.12:0/1076163856 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc18c00c0e0 con 0x7fc18c0036b0 2024-09-11T01:51:13.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.264+0000 7fc1a17fa640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:13.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a2ffd640 1 -- 172.21.15.12:0/1076163856 >> v1:172.21.15.12:6789/0 conn(0x7fc18c0073c0 legacy=0x7fc18c009860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:13.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a37fe640 1 -- 172.21.15.12:0/1076163856 >> v1:172.21.15.71:6789/0 conn(0x7fc18c0036b0 legacy=0x7fc18c005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:13.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 1 -- 172.21.15.12:0/1076163856 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 845087563 0 0) 0x7fc1a40a66a0 con 0x7fc18c0073c0 2024-09-11T01:51:13.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 10 monclient(hunting): _negotiate 2024-09-11T01:51:13.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:13.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 10 monclient(hunting): my global_id is 214380 2024-09-11T01:51:13.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 1 -- 172.21.15.12:0/1076163856 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc190003520 con 0x7fc18c0073c0 2024-09-11T01:51:13.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 1 -- 172.21.15.12:0/1076163856 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2851924283 0 0) 0x7fc18c00c0e0 con 0x7fc18c0036b0 2024-09-11T01:51:13.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 10 monclient(hunting): _negotiate 2024-09-11T01:51:13.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:13.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 10 monclient(hunting): my global_id is 234139 2024-09-11T01:51:13.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 1 -- 172.21.15.12:0/1076163856 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1a40a66a0 con 0x7fc18c0036b0 2024-09-11T01:51:13.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 1 -- 172.21.15.12:0/1076163856 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2159339103 0 0) 0x7fc190003520 con 0x7fc18c0073c0 2024-09-11T01:51:13.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 1 -- 172.21.15.12:0/1076163856 >> v1:172.21.15.71:6789/0 conn(0x7fc18c0036b0 legacy=0x7fc18c005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:13.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 10 monclient: _finish_hunting 0 2024-09-11T01:51:13.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:51:13.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:13.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:51:13.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 1 -- 172.21.15.12:0/1076163856 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc18c00d2e0 con 0x7fc18c0073c0 2024-09-11T01:51:13.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 10 monclient: _finish_auth 0 2024-09-11T01:51:13.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 10 monclient: _check_auth_tickets 2024-09-11T01:51:13.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:13.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 1 -- 172.21.15.12:0/1076163856 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7fc19800d680 con 0x7fc18c0073c0 2024-09-11T01:51:13.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:13.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:51:13.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 10 monclient: dump: 2024-09-11T01:51:13.274 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:13.274 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:13.274 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.274 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.274 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:13.274 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:13.274 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:13.274 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:13.275 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:13.275 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:13.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:51:13.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:13.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.267+0000 7fc1a27fc640 1 -- 172.21.15.12:0/1076163856 >> v1:172.21.15.12:6789/0 conn(0x7fc18c0073c0 legacy=0x7fc18c009860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:13.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a27fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:51:13.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a27fc640 1 --2- 172.21.15.12:0/1076163856 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc190003d00 0x7fc1900040e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a3fff640 1 -- 172.21.15.12:0/1076163856 reap_dead start 2024-09-11T01:51:13.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a27fc640 10 monclient(hunting): picked mon.b con 0x7fc190003d00 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:51:13.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a27fc640 1 --2- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc190004850 0x7fc190004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a27fc640 10 monclient(hunting): picked mon.a con 0x7fc190004850 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:51:13.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a27fc640 1 --2- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1900099b0 0x7fc19000bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a27fc640 10 monclient(hunting): picked mon.c con 0x7fc1900099b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:13.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a27fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:13.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a27fc640 1 -- 172.21.15.12:0/1076163856 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc19000c470 con 0x7fc190004850 2024-09-11T01:51:13.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a27fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:13.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a27fc640 1 -- 172.21.15.12:0/1076163856 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc19000c620 con 0x7fc190003d00 2024-09-11T01:51:13.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a27fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:13.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a27fc640 1 -- 172.21.15.12:0/1076163856 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc19000c820 con 0x7fc1900099b0 2024-09-11T01:51:13.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a27fc640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:13.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a27fc640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:51:13.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a2ffd640 1 --2- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc190004850 0x7fc190004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a37fe640 1 --2- 172.21.15.12:0/1076163856 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc190003d00 0x7fc1900040e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a37fe640 10 monclient(hunting): get_auth_request con 0x7fc190003d00 auth_method 0 2024-09-11T01:51:13.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:13.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:13.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a2ffd640 10 monclient(hunting): get_auth_request con 0x7fc190004850 auth_method 0 2024-09-11T01:51:13.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:13.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:13.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a3fff640 1 --2- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1900099b0 0x7fc19000bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a3fff640 10 monclient(hunting): get_auth_request con 0x7fc1900099b0 auth_method 0 2024-09-11T01:51:13.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:13.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:13.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:13.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:13.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:51:13.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:13.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:13.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:51:13.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:13.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:13.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:51:13.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a37fe640 10 monclient(hunting): handle_auth_done global_id 214380 payload 293 2024-09-11T01:51:13.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a37fe640 1 -- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1900099b0 msgr2=0x7fc19000bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:13.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a37fe640 1 --2- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1900099b0 0x7fc19000bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a37fe640 1 -- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc190004850 msgr2=0x7fc190004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:13.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a37fe640 1 --2- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc190004850 0x7fc190004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a37fe640 10 monclient: _finish_hunting 0 2024-09-11T01:51:13.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a37fe640 1 monclient: found mon.b 2024-09-11T01:51:13.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:13.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a37fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:51:13.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a37fe640 1 -- 172.21.15.12:0/1076163856 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc190003520 con 0x7fc190003d00 2024-09-11T01:51:13.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a37fe640 10 monclient: _finish_auth 0 2024-09-11T01:51:13.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a37fe640 10 monclient: _check_auth_tickets 2024-09-11T01:51:13.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:13.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a9b24640 5 monclient: authenticate success, global_id 214380 2024-09-11T01:51:13.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.268+0000 7fc1a9b24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:51:13.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.271+0000 7fc1a37fe640 1 -- 172.21.15.12:0/1076163856 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc190003d00 msgr2=0x7fc1900040e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:13.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.271+0000 7fc1a37fe640 1 --2- 172.21.15.12:0/1076163856 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc190003d00 0x7fc1900040e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc194009100 tx=0x7fc190005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8619277fa27dfe73 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:13.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.272+0000 7fc1a27fc640 1 -- 172.21.15.12:0/1076163856 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc194031070 con 0x7fc190003d00 2024-09-11T01:51:13.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.272+0000 7fc1a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:13.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.272+0000 7fc1a27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:51:13.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.272+0000 7fc1a27fc640 10 monclient: dump: 2024-09-11T01:51:13.286 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:13.286 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:13.286 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.286 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.286 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:13.286 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:13.287 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:13.287 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:13.287 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:13.287 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:13.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.272+0000 7fc1a9b24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:51:13.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.272+0000 7fc1a27fc640 1 -- 172.21.15.12:0/1076163856 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc19402bb20 con 0x7fc190003d00 2024-09-11T01:51:13.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.272+0000 7fc1a27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:13.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.272+0000 7fc1a9b24640 10 monclient: get_monmap_and_config success 2024-09-11T01:51:13.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.272+0000 7fc1a27fc640 1 -- 172.21.15.12:0/1076163856 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19400f040 con 0x7fc190003d00 2024-09-11T01:51:13.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.272+0000 7fc1a9b24640 4 set_mon_vals no callback set 2024-09-11T01:51:13.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.272+0000 7fc1a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:13.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.272+0000 7fc1a27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:51:13.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.272+0000 7fc1a27fc640 10 monclient: dump: 2024-09-11T01:51:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:13.288 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:13.289 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.272+0000 7fc1a9b24640 10 monclient: shutdown 2024-09-11T01:51:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.272+0000 7fc1a9b24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.272+0000 7fc1a9b24640 1 -- 172.21.15.12:0/1076163856 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc190003d00 msgr2=0x7fc1900040e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.272+0000 7fc1a9b24640 1 --2- 172.21.15.12:0/1076163856 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc190003d00 0x7fc1900040e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc194009100 tx=0x7fc190005830 comp rx=0 tx=0).stop 2024-09-11T01:51:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.273+0000 7fc1a9b24640 1 -- 172.21.15.12:0/1076163856 shutdown_connections 2024-09-11T01:51:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.273+0000 7fc1a9b24640 1 --2- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1900099b0 0x7fc19000bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.273+0000 7fc1a9b24640 1 --2- 172.21.15.12:0/1076163856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc190004850 0x7fc190004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.273+0000 7fc1a9b24640 1 --2- 172.21.15.12:0/1076163856 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc190003d00 0x7fc1900040e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.273+0000 7fc1a9b24640 1 -- 172.21.15.12:0/1076163856 >> 172.21.15.12:0/1076163856 conn(0x7fc1a40a7120 msgr2=0x7fc1a40a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.273+0000 7fc1a9b24640 1 -- 172.21.15.12:0/1076163856 shutdown_connections 2024-09-11T01:51:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.273+0000 7fc1a9b24640 1 -- 172.21.15.12:0/1076163856 wait complete. 2024-09-11T01:51:13.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.273+0000 7fc1a9b24640 10 monclient: build_initial_monmap 2024-09-11T01:51:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.274+0000 7fc1a9b24640 10 monclient: monmap: 2024-09-11T01:51:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:51:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:51:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:51:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:51:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:51:13.290 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:51:13.290 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.274+0000 7fc1a9b24640 1 Processor -- start 2024-09-11T01:51:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.274+0000 7fc1a9b24640 1 -- start start 2024-09-11T01:51:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.274+0000 7fc1a9b24640 10 monclient: init 2024-09-11T01:51:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a9b24640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a9b24640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:51:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a9b24640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc190003d00 0x7fc1a41ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a9b24640 10 monclient(hunting): picked mon.noname-a con 0x7fc190003d00 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:51:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a9b24640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc190004850 0x7fc1a41cd250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a9b24640 10 monclient(hunting): picked mon.noname-b con 0x7fc190004850 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:51:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a9b24640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1900099b0 0x7fc1a41245b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a9b24640 10 monclient(hunting): picked mon.noname-c con 0x7fc1900099b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a9b24640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a9b24640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a4138790 con 0x7fc190003d00 2024-09-11T01:51:13.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a9b24640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:13.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a9b24640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a4138610 con 0x7fc190004850 2024-09-11T01:51:13.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a9b24640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:13.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a9b24640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc1a4138910 con 0x7fc1900099b0 2024-09-11T01:51:13.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a9b24640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:13.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc190003d00 0x7fc1a41ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a9b24640 10 monclient(hunting): authenticate will time out at 2079.717582s 2024-09-11T01:51:13.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc190003d00 0x7fc1a41ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60200/0 (socket says 172.21.15.12:60200) 2024-09-11T01:51:13.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a37fe640 1 -- 172.21.15.12:0/3465818297 learned_addr learned my addr 172.21.15.12:0/3465818297 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:13.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a37fe640 10 monclient(hunting): get_auth_request con 0x7fc190003d00 auth_method 0 2024-09-11T01:51:13.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:13.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a3fff640 1 --2- 172.21.15.12:0/3465818297 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1900099b0 0x7fc1a41245b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a3fff640 10 monclient(hunting): get_auth_request con 0x7fc1900099b0 auth_method 0 2024-09-11T01:51:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a2ffd640 1 --2- 172.21.15.12:0/3465818297 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc190004850 0x7fc1a41cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a2ffd640 10 monclient(hunting): get_auth_request con 0x7fc190004850 auth_method 0 2024-09-11T01:51:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:13.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a3fff640 1 -- 172.21.15.12:0/3465818297 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1900099b0 msgr2=0x7fc1a41245b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:51:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a3fff640 1 -- 172.21.15.12:0/3465818297 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1900099b0 msgr2=0x7fc1a41245b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:51:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a3fff640 1 --2- 172.21.15.12:0/3465818297 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1900099b0 0x7fc1a41245b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a3fff640 1 --2- 172.21.15.12:0/3465818297 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1900099b0 0x7fc1a41245b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:51:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.275+0000 7fc1a37fe640 10 monclient(hunting): handle_auth_done global_id 214383 payload 824 2024-09-11T01:51:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.276+0000 7fc1a37fe640 1 -- 172.21.15.12:0/3465818297 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1900099b0 msgr2=0x7fc1a41245b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:51:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.276+0000 7fc1a37fe640 1 --2- 172.21.15.12:0/3465818297 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1900099b0 0x7fc1a41245b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.276+0000 7fc1a37fe640 1 -- 172.21.15.12:0/3465818297 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc190004850 msgr2=0x7fc1a41cd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:13.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.276+0000 7fc1a37fe640 1 --2- 172.21.15.12:0/3465818297 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc190004850 0x7fc1a41cd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.276+0000 7fc1a37fe640 10 monclient: _finish_hunting 0 2024-09-11T01:51:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.276+0000 7fc1a37fe640 1 monclient: found mon.noname-a 2024-09-11T01:51:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.276+0000 7fc1a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.276+0000 7fc1a37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:51:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.276+0000 7fc1a37fe640 1 -- 172.21.15.12:0/3465818297 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1a4137990 con 0x7fc190003d00 2024-09-11T01:51:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.278+0000 7fc1a37fe640 1 -- 172.21.15.12:0/3465818297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc190003d00 msgr2=0x7fc1a41ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.278+0000 7fc1a37fe640 1 --2- 172.21.15.12:0/3465818297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc190003d00 0x7fc1a41ccd10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc194037040 tx=0x7fc1940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.278+0000 7fc18bfff640 1 -- 172.21.15.12:0/3465818297 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc194044070 con 0x7fc190003d00 2024-09-11T01:51:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.278+0000 7fc18bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.278+0000 7fc18bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:51:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.278+0000 7fc18bfff640 10 monclient: dump: 2024-09-11T01:51:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:13.296 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.296 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.296 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:13.296 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:13.296 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:13.296 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:13.296 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:13.296 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:13.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.278+0000 7fc18bfff640 10 monclient: _finish_auth 0 2024-09-11T01:51:13.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.278+0000 7fc18bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:51:13.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.278+0000 7fc18bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:13.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.278+0000 7fc18bfff640 1 -- 172.21.15.12:0/3465818297 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc194035d10 con 0x7fc190003d00 2024-09-11T01:51:13.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.278+0000 7fc18bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:13.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.278+0000 7fc18bfff640 1 -- 172.21.15.12:0/3465818297 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19400f040 con 0x7fc190003d00 2024-09-11T01:51:13.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.278+0000 7fc1a9b24640 5 monclient: authenticate success, global_id 214383 2024-09-11T01:51:13.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.278+0000 7fc1a9b24640 10 monclient: _renew_subs 2024-09-11T01:51:13.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.278+0000 7fc1a9b24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:51:13.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.278+0000 7fc1a9b24640 1 -- 172.21.15.12:0/3465818297 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1a4137480 con 0x7fc190003d00 2024-09-11T01:51:13.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.278+0000 7fc18bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:13.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.278+0000 7fc18bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:51:13.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.278+0000 7fc18bfff640 10 monclient: dump: 2024-09-11T01:51:13.297 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:13.297 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:13.297 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.297 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.297 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:13.297 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:13.298 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:13.298 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:13.298 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:13.298 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:13.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.278+0000 7fc1a1ffb640 4 set_mon_vals no callback set 2024-09-11T01:51:13.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.278+0000 7fc1a9b24640 10 monclient: _renew_subs 2024-09-11T01:51:13.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.278+0000 7fc1a9b24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:51:13.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.278+0000 7fc1a9b24640 1 -- 172.21.15.12:0/3465818297 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1a41280f0 con 0x7fc190003d00 2024-09-11T01:51:13.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.279+0000 7fc18bfff640 1 -- 172.21.15.12:0/3465818297 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fc194015040 con 0x7fc190003d00 2024-09-11T01:51:13.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.280+0000 7fc18bfff640 1 --2- 172.21.15.12:0/3465818297 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc18c03eaf0 0x7fc18c040fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.280+0000 7fc1a9b24640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:13.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.280+0000 7fc1a9b24640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:13.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.280+0000 7fc1a9b24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:51:13.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.280+0000 7fc18bfff640 1 -- 172.21.15.12:0/3465818297 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(731..731 src has 507..731) ==== 4767+0+0 (secure 0 0 0) 0x7fc194073d00 con 0x7fc190003d00 2024-09-11T01:51:13.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.280+0000 7fc1a9b24640 1 -- 172.21.15.12:0/3465818297 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1a40a5020 con 0x7fc190003d00 2024-09-11T01:51:13.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.280+0000 7fc1a2ffd640 1 --2- 172.21.15.12:0/3465818297 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc18c03eaf0 0x7fc18c040fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.280+0000 7fc1a2ffd640 10 monclient: get_auth_request con 0x7fc18c03eaf0 auth_method 0 2024-09-11T01:51:13.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.282+0000 7fc1a2ffd640 1 --2- 172.21.15.12:0/3465818297 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc18c03eaf0 0x7fc18c040fb0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4126b60 tx=0x7fc198008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:13.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.285+0000 7fc18bfff640 1 -- 172.21.15.12:0/3465818297 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc1a4137480 con 0x7fc190003d00 2024-09-11T01:51:13.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.285+0000 7fc18bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:13.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.285+0000 7fc18bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:51:13.547 INFO:tasks.rados.rados.0.smithi071.stdout:3271: done (2 left) 2024-09-11T01:51:13.547 INFO:tasks.rados.rados.0.smithi071.stdout:3272: done (1 left) 2024-09-11T01:51:13.547 INFO:tasks.rados.rados.0.smithi071.stdout:3273: done (0 left) 2024-09-11T01:51:13.547 INFO:tasks.rados.rados.0.smithi071.stdout:3274: write (excl) oid 12 current snap is 365 2024-09-11T01:51:13.547 INFO:tasks.rados.rados.0.smithi071.stdout:3274: seq_num 1148 ranges {494436=755155,2011446=521062,3075831=218605} 2024-09-11T01:51:13.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.549+0000 7fc1a9b24640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:51:13.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.549+0000 7fc1a9b24640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:51:13.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.549+0000 7fc1a9b24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:51:13.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.549+0000 7fc1a9b24640 1 -- 172.21.15.12:0/3465818297 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc1a4126bd0 con 0x7fc190003d00 2024-09-11T01:51:13.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.550+0000 7fc18bfff640 1 -- 172.21.15.12:0/3465818297 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fc194040ba0 con 0x7fc190003d00 2024-09-11T01:51:13.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.550+0000 7fc18bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:51:13.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.550+0000 7fc18bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:51:13.552 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":208,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:51:13.554 INFO:tasks.rados.rados.0.smithi071.stdout:3274: writing smithi07134777-12 from 494436 to 1249591 tid 1 2024-09-11T01:51:13.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.552+0000 7fc1a9b24640 1 -- 172.21.15.12:0/3465818297 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc18c03eaf0 msgr2=0x7fc18c040fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:13.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.552+0000 7fc1a9b24640 1 --2- 172.21.15.12:0/3465818297 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc18c03eaf0 0x7fc18c040fb0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4126b60 tx=0x7fc198008040 comp rx=0 tx=0).stop 2024-09-11T01:51:13.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.552+0000 7fc1a9b24640 10 monclient: shutdown 2024-09-11T01:51:13.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.552+0000 7fc1a9b24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:13.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.552+0000 7fc1a9b24640 1 -- 172.21.15.12:0/3465818297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc190003d00 msgr2=0x7fc1a41ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:13.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.552+0000 7fc1a9b24640 1 --2- 172.21.15.12:0/3465818297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc190003d00 0x7fc1a41ccd10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc194037040 tx=0x7fc1940046c0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.552+0000 7fc1a9b24640 1 -- 172.21.15.12:0/3465818297 shutdown_connections 2024-09-11T01:51:13.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.552+0000 7fc1a9b24640 1 --2- 172.21.15.12:0/3465818297 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc1900099b0 0x7fc1a41245b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.552+0000 7fc1a9b24640 1 --2- 172.21.15.12:0/3465818297 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc190004850 0x7fc1a41cd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.552+0000 7fc1a9b24640 1 --2- 172.21.15.12:0/3465818297 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc18c03eaf0 0x7fc18c040fb0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.552+0000 7fc1a9b24640 1 --2- 172.21.15.12:0/3465818297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc190003d00 0x7fc1a41ccd10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.552+0000 7fc1a9b24640 1 -- 172.21.15.12:0/3465818297 >> 172.21.15.12:0/3465818297 conn(0x7fc1a40a7120 msgr2=0x7fc1a40a3c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:13.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.552+0000 7fc1a9b24640 1 -- 172.21.15.12:0/3465818297 shutdown_connections 2024-09-11T01:51:13.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.552+0000 7fc1a9b24640 1 -- 172.21.15.12:0/3465818297 wait complete. 2024-09-11T01:51:13.560 INFO:tasks.rados.rados.0.smithi071.stdout:3274: writing smithi07134777-12 from 2011446 to 2532508 tid 2 2024-09-11T01:51:13.565 INFO:tasks.rados.rados.0.smithi071.stdout:3274: writing smithi07134777-12 from 3075831 to 3294436 tid 3 2024-09-11T01:51:13.565 INFO:tasks.rados.rados.0.smithi071.stdout:3275: copy_from oid 44 from oid 31 current snap is 365 2024-09-11T01:51:13.565 INFO:tasks.rados.rados.0.smithi071.stdout:3274: finishing write tid 1 to smithi07134777-12 2024-09-11T01:51:13.565 INFO:tasks.rados.rados.0.smithi071.stdout:3274: finishing write tid 2 to smithi07134777-12 2024-09-11T01:51:13.565 INFO:tasks.rados.rados.0.smithi071.stdout:3276: snap_create 2024-09-11T01:51:13.567 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:51:13.567 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:51:13.567 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:51:13.568 INFO:tasks.rados.rados.0.smithi071.stdout:3274: finishing write tid 3 to smithi07134777-12 2024-09-11T01:51:13.570 INFO:tasks.rados.rados.0.smithi071.stdout:3274: finishing write tid 4 to smithi07134777-12 2024-09-11T01:51:13.570 INFO:tasks.rados.rados.0.smithi071.stdout:3274: finishing write tid 5 to smithi07134777-12 2024-09-11T01:51:13.571 INFO:tasks.rados.rados.0.smithi071.stdout:3274: oid 12 updating version 0 to 2206 2024-09-11T01:51:13.571 INFO:tasks.rados.rados.0.smithi071.stdout:3274: oid 12 updating version 2206 to 2207 2024-09-11T01:51:13.571 INFO:tasks.rados.rados.0.smithi071.stdout:3274: oid 12 updating version 2207 to 2208 2024-09-11T01:51:13.572 INFO:tasks.rados.rados.0.smithi071.stdout:3274: oid 12 updating version 2208 to 2209 2024-09-11T01:51:13.572 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 2209 (ObjNum 1148 snap 365 seq_num 1148) dirty exists 2024-09-11T01:51:13.572 INFO:tasks.rados.rados.0.smithi071.stdout:3274: left oid 12 (ObjNum 1148 snap 365 seq_num 1148) 2024-09-11T01:51:13.573 INFO:tasks.rados.rados.0.smithi071.stdout:3275: finishing copy_from to smithi07134777-44 2024-09-11T01:51:13.573 INFO:tasks.rados.rados.0.smithi071.stdout:3275: got expected ENOENT (src dne) 2024-09-11T01:51:13.573 INFO:tasks.rados.rados.0.smithi071.stdout:3275: finishing copy_from racing read to smithi07134777-44 2024-09-11T01:51:13.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.667+0000 7f1c5a67f640 10 monclient: get_monmap_and_config 2024-09-11T01:51:13.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.667+0000 7f1c5a67f640 10 monclient: build_initial_monmap 2024-09-11T01:51:13.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.667+0000 7f1c5a67f640 10 monclient: monmap: 2024-09-11T01:51:13.669 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:13.669 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:13.669 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:51:13.668362+0000 2024-09-11T01:51:13.670 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:51:13.668362+0000 2024-09-11T01:51:13.670 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:13.670 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:13.670 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:51:13.670 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:51:13.670 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:51:13.670 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:51:13.670 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:51:13.670 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:13.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.668+0000 7f1c5a67f640 1 Processor -- start 2024-09-11T01:51:13.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.668+0000 7f1c5a67f640 1 -- start start 2024-09-11T01:51:13.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.668+0000 7f1c5a67f640 10 monclient: init 2024-09-11T01:51:13.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.668+0000 7f1c5a67f640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:13.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.668+0000 7f1c5a67f640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-11T01:51:13.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c5a67f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1c540a03a0 addr v1:172.21.15.71:6789/0 2024-09-11T01:51:13.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c5a67f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c540a2530 0x7f1c5409f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c5a67f640 10 monclient(hunting): picked mon.noname-b con 0x7f1c540a2530 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:13.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c5a67f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1c540a13a0 0x7f1c5413baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c5a67f640 10 monclient(hunting): picked mon.noname-a con 0x7f1c540a13a0 addr v2:172.21.15.12:3300/0 2024-09-11T01:51:13.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c5a67f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c5409e510 con 0x7f1c540a03a0 2024-09-11T01:51:13.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c5a67f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:13.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c5a67f640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f1c540a29b0 con 0x7f1c540a13a0 2024-09-11T01:51:13.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c5a67f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:13.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c5a67f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1c540a0780 con 0x7f1c540a2530 2024-09-11T01:51:13.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c5a67f640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:13.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c5a67f640 10 monclient(hunting): authenticate will time out at 2080.111144s 2024-09-11T01:51:13.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c537fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c540a2530 0x7f1c5409f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c58c1c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1c540a13a0 0x7f1c5413baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c537fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c540a2530 0x7f1c5409f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:47930/0 (socket says 172.21.15.12:47930) 2024-09-11T01:51:13.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c537fe640 1 -- 172.21.15.12:0/169629977 learned_addr learned my addr 172.21.15.12:0/169629977 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:13.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c58c1c640 10 monclient(hunting): get_auth_request con 0x7f1c540a13a0 auth_method 0 2024-09-11T01:51:13.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c58c1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:13.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c58c1c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c53fff640 1 --1- 172.21.15.12:0/169629977 >> v1:172.21.15.71:6789/0 conn(0x7f1c540a03a0 0x7f1c540a0c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.12:59060/0 (socket says 172.21.15.12:59060) 2024-09-11T01:51:13.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c58c1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:13.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c537fe640 10 monclient(hunting): get_auth_request con 0x7f1c540a2530 auth_method 0 2024-09-11T01:51:13.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:13.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c537fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:13.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c58c1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:13.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c58c1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:13.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c58c1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:13.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:13.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:13.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:13.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c537fe640 10 monclient(hunting): handle_auth_done global_id 244100 payload 274 2024-09-11T01:51:13.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c537fe640 1 -- 172.21.15.12:0/169629977 >> v2:172.21.15.12:3300/0 conn(0x7f1c540a13a0 msgr2=0x7f1c5413baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:13.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c537fe640 1 --2- 172.21.15.12:0/169629977 >> v2:172.21.15.12:3300/0 conn(0x7f1c540a13a0 0x7f1c5413baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c537fe640 1 -- 172.21.15.12:0/169629977 >> v1:172.21.15.71:6789/0 conn(0x7f1c540a03a0 legacy=0x7f1c540a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:13.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c537fe640 10 monclient: _finish_hunting 0 2024-09-11T01:51:13.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c537fe640 1 monclient: found mon.noname-b 2024-09-11T01:51:13.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:13.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c537fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:51:13.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c537fe640 1 -- 172.21.15.12:0/169629977 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c5409edb0 con 0x7f1c540a2530 2024-09-11T01:51:13.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.669+0000 7f1c58c1c640 1 --2- 172.21.15.12:0/169629977 >> v2:172.21.15.12:3300/0 conn(0x7f1c540a13a0 0x7f1c5413baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:51:13.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.672+0000 7f1c537fe640 1 -- 172.21.15.12:0/169629977 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c540a2530 msgr2=0x7f1c5409f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:13.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.672+0000 7f1c537fe640 1 --2- 172.21.15.12:0/169629977 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c540a2530 0x7f1c5409f1d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1c48009870 tx=0x7f1c4802a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8af2d6456dd9daa9 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:13.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.672+0000 7f1c52ffd640 1 -- 172.21.15.12:0/169629977 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c48037070 con 0x7f1c540a2530 2024-09-11T01:51:13.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.672+0000 7f1c52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:13.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.672+0000 7f1c52ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:51:13.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.672+0000 7f1c52ffd640 10 monclient: dump: 2024-09-11T01:51:13.684 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:13.684 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:13.684 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.684 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.684 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:13.684 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:13.684 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:13.684 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:13.685 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:13.685 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:13.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.672+0000 7f1c52ffd640 10 monclient: _finish_auth 0 2024-09-11T01:51:13.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.672+0000 7f1c52ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:51:13.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.672+0000 7f1c52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:13.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.672+0000 7f1c52ffd640 1 -- 172.21.15.12:0/169629977 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c48032440 con 0x7f1c540a2530 2024-09-11T01:51:13.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.672+0000 7f1c5a67f640 5 monclient: authenticate success, global_id 244100 2024-09-11T01:51:13.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.672+0000 7f1c5a67f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:51:13.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.672+0000 7f1c52ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:13.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.672+0000 7f1c52ffd640 1 -- 172.21.15.12:0/169629977 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c48032810 con 0x7f1c540a2530 2024-09-11T01:51:13.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.672+0000 7f1c5a67f640 10 monclient: get_monmap_and_config success 2024-09-11T01:51:13.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.672+0000 7f1c5a67f640 4 set_mon_vals no callback set 2024-09-11T01:51:13.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.672+0000 7f1c52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:13.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.672+0000 7f1c52ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:13.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.672+0000 7f1c52ffd640 10 monclient: dump: 2024-09-11T01:51:13.686 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:13.686 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:13.686 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.686 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.686 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:13.686 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:13.686 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:13.686 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:13.686 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:13.686 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:13.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.673+0000 7f1c5a67f640 10 monclient: shutdown 2024-09-11T01:51:13.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.673+0000 7f1c5a67f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:13.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.673+0000 7f1c5a67f640 1 -- 172.21.15.12:0/169629977 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c540a2530 msgr2=0x7f1c5409f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:13.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.673+0000 7f1c5a67f640 1 --2- 172.21.15.12:0/169629977 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c540a2530 0x7f1c5409f1d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1c48009870 tx=0x7f1c4802a0a0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.673+0000 7f1c5a67f640 1 -- 172.21.15.12:0/169629977 shutdown_connections 2024-09-11T01:51:13.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.673+0000 7f1c5a67f640 1 --2- 172.21.15.12:0/169629977 >> v2:172.21.15.12:3300/0 conn(0x7f1c540a13a0 0x7f1c5413baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.673+0000 7f1c5a67f640 1 --2- 172.21.15.12:0/169629977 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c540a2530 0x7f1c5409f1d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.673+0000 7f1c5a67f640 1 -- 172.21.15.12:0/169629977 >> 172.21.15.12:0/169629977 conn(0x7f1c540a2f70 msgr2=0x7f1c540a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:13.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.673+0000 7f1c5a67f640 1 -- 172.21.15.12:0/169629977 shutdown_connections 2024-09-11T01:51:13.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.673+0000 7f1c5a67f640 1 -- 172.21.15.12:0/169629977 wait complete. 2024-09-11T01:51:13.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.673+0000 7f1c5a67f640 10 monclient: build_initial_monmap 2024-09-11T01:51:13.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.673+0000 7f1c5a67f640 10 monclient: monmap: 2024-09-11T01:51:13.687 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:13.687 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:51:13.687 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:51:13.687 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:51:13.687 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:13.688 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:13.688 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:51:13.688 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:51:13.688 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:51:13.688 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:13.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c5a67f640 1 Processor -- start 2024-09-11T01:51:13.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c5a67f640 1 -- start start 2024-09-11T01:51:13.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c5a67f640 10 monclient: init 2024-09-11T01:51:13.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c5a67f640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:13.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c5a67f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:51:13.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c5a67f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c540a03a0 0x7f1c541ad010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c5a67f640 10 monclient(hunting): picked mon.noname-c con 0x7f1c540a03a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:13.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c5a67f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c540a13a0 0x7f1c541cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c5a67f640 10 monclient(hunting): picked mon.noname-b con 0x7f1c540a13a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:51:13.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c5a67f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c540a2530 0x7f1c541c6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c5a67f640 10 monclient(hunting): picked mon.noname-a con 0x7f1c540a2530 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:51:13.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c5a67f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:13.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c5a67f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1c5409ca30 con 0x7f1c540a2530 2024-09-11T01:51:13.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c5a67f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:13.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c5a67f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1c540a1a20 con 0x7f1c540a13a0 2024-09-11T01:51:13.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c5a67f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:13.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c5a67f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1c5409cbb0 con 0x7f1c540a03a0 2024-09-11T01:51:13.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c5a67f640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:13.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c5a67f640 10 monclient(hunting): authenticate will time out at 2080.116840s 2024-09-11T01:51:13.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c53fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c540a03a0 0x7f1c541ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c53fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c540a03a0 0x7f1c541ad010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:47944/0 (socket says 172.21.15.12:47944) 2024-09-11T01:51:13.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c53fff640 1 -- 172.21.15.12:0/3536893686 learned_addr learned my addr 172.21.15.12:0/3536893686 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:13.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c53fff640 10 monclient(hunting): get_auth_request con 0x7f1c540a03a0 auth_method 0 2024-09-11T01:51:13.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:13.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c53fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:13.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c58c1c640 1 --2- 172.21.15.12:0/3536893686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c540a2530 0x7f1c541c6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c58c1c640 10 monclient(hunting): get_auth_request con 0x7f1c540a2530 auth_method 0 2024-09-11T01:51:13.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c58c1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:13.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c58c1c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.674+0000 7f1c58c1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:13.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.675+0000 7f1c53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:13.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.675+0000 7f1c53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:13.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.675+0000 7f1c53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:13.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.675+0000 7f1c58c1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:13.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.675+0000 7f1c58c1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:13.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.675+0000 7f1c58c1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:13.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.675+0000 7f1c537fe640 1 --2- 172.21.15.12:0/3536893686 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c540a13a0 0x7f1c541cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.675+0000 7f1c537fe640 10 monclient(hunting): get_auth_request con 0x7f1c540a13a0 auth_method 0 2024-09-11T01:51:13.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.675+0000 7f1c537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:13.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.675+0000 7f1c537fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.675+0000 7f1c537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:13.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.675+0000 7f1c53fff640 10 monclient(hunting): handle_auth_done global_id 244103 payload 824 2024-09-11T01:51:13.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.675+0000 7f1c53fff640 1 -- 172.21.15.12:0/3536893686 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c540a13a0 msgr2=0x7f1c541cd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:13.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.675+0000 7f1c53fff640 1 --2- 172.21.15.12:0/3536893686 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c540a13a0 0x7f1c541cd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.675+0000 7f1c53fff640 1 -- 172.21.15.12:0/3536893686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c540a2530 msgr2=0x7f1c541c6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:13.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.675+0000 7f1c53fff640 1 --2- 172.21.15.12:0/3536893686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c540a2530 0x7f1c541c6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.675+0000 7f1c53fff640 10 monclient: _finish_hunting 0 2024-09-11T01:51:13.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.675+0000 7f1c53fff640 1 monclient: found mon.noname-c 2024-09-11T01:51:13.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.675+0000 7f1c53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:13.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.675+0000 7f1c53fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:51:13.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.675+0000 7f1c53fff640 1 -- 172.21.15.12:0/3536893686 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c541ad9a0 con 0x7f1c540a03a0 2024-09-11T01:51:13.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.675+0000 7f1c58c1c640 1 --2- 172.21.15.12:0/3536893686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c540a2530 0x7f1c541c6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:51:13.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.677+0000 7f1c53fff640 1 -- 172.21.15.12:0/3536893686 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c540a03a0 msgr2=0x7f1c541ad010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:13.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.677+0000 7f1c53fff640 1 --2- 172.21.15.12:0/3536893686 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c540a03a0 0x7f1c541ad010 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1c4400c450 tx=0x7f1c4400c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:13.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.677+0000 7f1c50ff9640 1 -- 172.21.15.12:0/3536893686 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c44017070 con 0x7f1c540a03a0 2024-09-11T01:51:13.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.677+0000 7f1c50ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:13.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.677+0000 7f1c50ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:51:13.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.677+0000 7f1c50ff9640 10 monclient: dump: 2024-09-11T01:51:13.693 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:13.693 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:13.693 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.693 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.693 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:13.693 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:13.693 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:13.693 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:13.693 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:13.693 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:13.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.677+0000 7f1c50ff9640 10 monclient: _finish_auth 0 2024-09-11T01:51:13.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.677+0000 7f1c50ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:51:13.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.677+0000 7f1c50ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:13.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.677+0000 7f1c50ff9640 1 -- 172.21.15.12:0/3536893686 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c44005810 con 0x7f1c540a03a0 2024-09-11T01:51:13.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.677+0000 7f1c5a67f640 5 monclient: authenticate success, global_id 244103 2024-09-11T01:51:13.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.677+0000 7f1c5a67f640 10 monclient: _renew_subs 2024-09-11T01:51:13.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.677+0000 7f1c5a67f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:13.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.677+0000 7f1c5a67f640 1 -- 172.21.15.12:0/3536893686 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c541abea0 con 0x7f1c540a03a0 2024-09-11T01:51:13.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.677+0000 7f1c50ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:13.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.677+0000 7f1c50ff9640 1 -- 172.21.15.12:0/3536893686 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c44005c00 con 0x7f1c540a03a0 2024-09-11T01:51:13.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.677+0000 7f1c50ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:13.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.677+0000 7f1c50ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:13.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.677+0000 7f1c50ff9640 10 monclient: dump: 2024-09-11T01:51:13.694 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:13.694 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:13.695 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.695 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.695 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:13.695 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:13.695 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:13.695 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:13.695 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:13.695 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:13.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.677+0000 7f1c527fc640 4 set_mon_vals no callback set 2024-09-11T01:51:13.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.677+0000 7f1c5a67f640 10 monclient: _renew_subs 2024-09-11T01:51:13.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.677+0000 7f1c5a67f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:13.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.677+0000 7f1c5a67f640 1 -- 172.21.15.12:0/3536893686 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c541ac4b0 con 0x7f1c540a03a0 2024-09-11T01:51:13.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.678+0000 7f1c5a67f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:13.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.678+0000 7f1c5a67f640 1 --2- 172.21.15.12:0/3536893686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c541c9990 0x7f1c541c9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.678+0000 7f1c5a67f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:13.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.678+0000 7f1c5a67f640 1 -- 172.21.15.12:0/3536893686 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1c541ae470 con 0x7f1c541c9990 2024-09-11T01:51:13.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.678+0000 7f1c53fff640 1 --2- 172.21.15.12:0/3536893686 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c540a03a0 0x7f1c541ad010 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1c4400c450 tx=0x7f1c4400c910 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7f1c440524a0 osd_map(732..732 src has 507..732) 2024-09-11T01:51:13.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.678+0000 7f1c50ff9640 1 -- 172.21.15.12:0/3536893686 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f1c4401b3b0 con 0x7f1c540a03a0 2024-09-11T01:51:13.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.678+0000 7f1c537fe640 1 --2- 172.21.15.12:0/3536893686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c541c9990 0x7f1c541c9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.678+0000 7f1c537fe640 10 monclient: get_auth_request con 0x7f1c541c9990 auth_method 0 2024-09-11T01:51:13.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.678+0000 7f1c537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:13.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.678+0000 7f1c537fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.678+0000 7f1c537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:13.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.678+0000 7f1c537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:13.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.678+0000 7f1c537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:13.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.678+0000 7f1c537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:13.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.679+0000 7f1c537fe640 10 monclient(hunting): handle_auth_done global_id 214392 payload 824 2024-09-11T01:51:13.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.679+0000 7f1c537fe640 1 -- 172.21.15.12:0/3536893686 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1c5409cfc0 con 0x7f1c541c9990 2024-09-11T01:51:13.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.679+0000 7f1c50ff9640 1 --2- 172.21.15.12:0/3536893686 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1c4003e240 0x7f1c40040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.679+0000 7f1c58c1c640 1 --2- 172.21.15.12:0/3536893686 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1c4003e240 0x7f1c40040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.679+0000 7f1c58c1c640 10 monclient: get_auth_request con 0x7f1c4003e240 auth_method 0 2024-09-11T01:51:13.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.681+0000 7f1c537fe640 1 -- 172.21.15.12:0/3536893686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c541c9990 msgr2=0x7f1c541c9d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:13.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.681+0000 7f1c537fe640 1 --2- 172.21.15.12:0/3536893686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c541c9990 0x7f1c541c9d70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1c5406cd50 tx=0x7f1c480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:13.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.681+0000 7f1c50ff9640 1 -- 172.21.15.12:0/3536893686 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c48042070 con 0x7f1c541c9990 2024-09-11T01:51:13.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.681+0000 7f1c50ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:13.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.681+0000 7f1c50ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:51:13.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.681+0000 7f1c50ff9640 10 monclient: dump: 2024-09-11T01:51:13.697 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:13.697 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:13.697 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.698 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.698 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:13.698 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:13.698 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:13.698 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:13.698 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:13.698 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:13.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.681+0000 7f1c50ff9640 1 -- 172.21.15.12:0/3536893686 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1c48032860 con 0x7f1c541c9990 2024-09-11T01:51:13.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.681+0000 7f1c50ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:13.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.681+0000 7f1c50ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:51:13.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.681+0000 7f1c50ff9640 1 -- 172.21.15.12:0/3536893686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c541c9990 msgr2=0x7f1c541c9d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:13.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.681+0000 7f1c50ff9640 1 --2- 172.21.15.12:0/3536893686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c541c9990 0x7f1c541c9d70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1c5406cd50 tx=0x7f1c480046c0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.681+0000 7f1c50ff9640 1 -- 172.21.15.12:0/3536893686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c541c9990 msgr2=0x7f1c541c9d70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:51:13.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.681+0000 7f1c50ff9640 1 --2- 172.21.15.12:0/3536893686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c541c9990 0x7f1c541c9d70 secure :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1c5406cd50 tx=0x7f1c480046c0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.681+0000 7f1c58c1c640 1 --2- 172.21.15.12:0/3536893686 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1c4003e240 0x7f1c40040700 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c009430 tx=0x7f1c4c00d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:13.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.687+0000 7f1c5a67f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:51:13.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.687+0000 7f1c5a67f640 1 --2- 172.21.15.12:0/3536893686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c541ca2b0 0x7f1c541a2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.687+0000 7f1c5a67f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:13.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.687+0000 7f1c5a67f640 1 -- 172.21.15.12:0/3536893686 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1c5406d480 con 0x7f1c541ca2b0 2024-09-11T01:51:13.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.688+0000 7f1c537fe640 1 --2- 172.21.15.12:0/3536893686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c541ca2b0 0x7f1c541a2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.688+0000 7f1c537fe640 10 monclient: get_auth_request con 0x7f1c541ca2b0 auth_method 0 2024-09-11T01:51:13.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.688+0000 7f1c537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:13.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.688+0000 7f1c537fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.688+0000 7f1c537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:13.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.688+0000 7f1c537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:13.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.688+0000 7f1c537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:13.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.688+0000 7f1c537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:13.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.688+0000 7f1c537fe640 10 monclient(hunting): handle_auth_done global_id 214395 payload 824 2024-09-11T01:51:13.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.688+0000 7f1c537fe640 1 -- 172.21.15.12:0/3536893686 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1c541ca690 con 0x7f1c541ca2b0 2024-09-11T01:51:13.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.690+0000 7f1c537fe640 1 -- 172.21.15.12:0/3536893686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c541ca2b0 msgr2=0x7f1c541a2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:13.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.690+0000 7f1c537fe640 1 --2- 172.21.15.12:0/3536893686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c541ca2b0 0x7f1c541a2600 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1c480090b0 tx=0x7f1c48008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:13.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.690+0000 7f1c50ff9640 1 -- 172.21.15.12:0/3536893686 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c48037040 con 0x7f1c541ca2b0 2024-09-11T01:51:13.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.690+0000 7f1c50ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:13.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.690+0000 7f1c50ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:51:13.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.690+0000 7f1c50ff9640 10 monclient: dump: 2024-09-11T01:51:13.702 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:13.702 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:13.702 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.702 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.702 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:13.702 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:13.702 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:13.702 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:13.703 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:13.703 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:13.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.690+0000 7f1c50ff9640 1 -- 172.21.15.12:0/3536893686 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f1c48003e70 con 0x7f1c541ca2b0 2024-09-11T01:51:13.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.690+0000 7f1c50ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:51:13.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.690+0000 7f1c50ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:51:13.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.690+0000 7f1c50ff9640 1 -- 172.21.15.12:0/3536893686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c541ca2b0 msgr2=0x7f1c541a2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:13.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.690+0000 7f1c50ff9640 1 --2- 172.21.15.12:0/3536893686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c541ca2b0 0x7f1c541a2600 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1c480090b0 tx=0x7f1c48008030 comp rx=0 tx=0).stop 2024-09-11T01:51:13.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.690+0000 7f1c50ff9640 1 -- 172.21.15.12:0/3536893686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c541ca2b0 msgr2=0x7f1c541a2600 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:51:13.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.690+0000 7f1c50ff9640 1 --2- 172.21.15.12:0/3536893686 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c541ca2b0 0x7f1c541a2600 secure :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1c480090b0 tx=0x7f1c48008030 comp rx=0 tx=0).stop 2024-09-11T01:51:13.703 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:51:13.703 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:51:13.704 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:51:13.704 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:51:13.704 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 208, 2024-09-11T01:51:13.704 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 179890, 2024-09-11T01:51:13.704 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:51:13.704 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:51:13.704 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:51:13.704 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:51:13.704 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:13.704 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:51:13.704 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:51:13.704 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:51:13.704 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:51:13.705 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:13.705 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:13.705 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:13.705 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:13.705 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:13.705 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:13.705 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:13.705 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:13.705 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:13.705 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:13.705 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:13.705 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:13.705 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:51:13.706 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:51:13.706 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:13.706 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:13.706 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:13.706 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:13.706 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:13.706 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:13.706 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:13.706 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:13.706 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:13.706 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:13.706 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:13.706 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:13.706 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:13.707 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:51:13.707 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:51:13.707 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:51:13.707 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:51:13.707 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:51:13.707 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:51:13.707 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:51:13.707 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:51:13.707 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:51:13.707 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:51:13.707 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:51:13.707 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:51:13.707 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:51:13.708 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:51:13.708 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:51:13.708 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:51:13.708 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:51:13.708 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:13.708 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:13.708 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:13.708 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:13.708 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:13.708 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:13.708 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:13.708 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:13.708 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:13.708 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:13.709 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:13.709 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:13.709 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:51:13.709 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:13.709 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:51:13.709 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:13.709 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:51:13.709 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:51:13.709 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:13.709 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:13.709 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:13.709 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:13.709 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:51:13.709 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:13.710 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:13.710 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:13.710 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:13.710 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:51:13.710 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:13.710 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:13.710 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:13.710 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:13.710 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:51:13.710 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:51:13.710 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:13.710 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:13.710 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:13.711 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:13.711 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:13.711 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:51:13.711 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:51:13.711 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:13.711 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:13.711 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:13.711 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:13.711 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:51:13.711 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:13.711 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:13.711 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:13.711 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:13.712 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:51:13.712 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:13.712 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:13.712 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:13.712 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:13.712 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:51:13.712 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:51:13.712 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:13.712 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:13.712 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:13.712 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:13.712 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:13.712 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:51:13.713 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:51:13.713 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:13.713 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:13.713 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:13.713 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:13.713 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:51:13.713 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:13.713 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:13.713 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:13.713 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:13.713 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:51:13.713 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:13.713 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:13.713 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:13.714 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:13.714 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:51:13.714 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:51:13.714 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:13.714 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:13.714 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:13.714 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:13.714 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:13.714 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:13.714 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:51:13.714 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:51:13.714 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:13.714 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:13.714 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:13.715 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:51:13.715 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:13.715 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:13.715 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:51:13.715 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:13.715 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:13.715 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:13.715 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:51:13.715 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:13.715 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:13.715 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:51:13.715 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:13.715 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:13.715 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:13.716 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:51:13.716 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:13.716 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:13.716 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:13.716 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:51:13.716 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:51:13.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.691+0000 7f1c5a67f640 1 -- 172.21.15.12:0/3536893686 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1c4003e240 msgr2=0x7f1c40040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:13.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.691+0000 7f1c5a67f640 1 --2- 172.21.15.12:0/3536893686 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1c4003e240 0x7f1c40040700 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c009430 tx=0x7f1c4c00d040 comp rx=0 tx=0).stop 2024-09-11T01:51:13.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.691+0000 7f1c5a67f640 10 monclient: shutdown 2024-09-11T01:51:13.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.691+0000 7f1c5a67f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:13.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.691+0000 7f1c5a67f640 1 -- 172.21.15.12:0/3536893686 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c540a03a0 msgr2=0x7f1c541ad010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:13.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.691+0000 7f1c5a67f640 1 --2- 172.21.15.12:0/3536893686 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c540a03a0 0x7f1c541ad010 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1c4400c450 tx=0x7f1c4400c910 comp rx=0 tx=0).stop 2024-09-11T01:51:13.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.691+0000 7f1c58c1c640 1 -- 172.21.15.12:0/3536893686 reap_dead start 2024-09-11T01:51:13.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.691+0000 7f1c50ff9640 1 -- 172.21.15.12:0/3536893686 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(732..732 src has 507..732) ==== 4739+0+0 (unknown 0 0 0) 0x7f1c440524a0 con 0x7f1c540a03a0 2024-09-11T01:51:13.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.691+0000 7f1c5a67f640 1 -- 172.21.15.12:0/3536893686 shutdown_connections 2024-09-11T01:51:13.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.691+0000 7f1c5a67f640 1 --2- 172.21.15.12:0/3536893686 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c540a03a0 0x7f1c541ad010 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.691+0000 7f1c5a67f640 1 -- 172.21.15.12:0/3536893686 >> 172.21.15.12:0/3536893686 conn(0x7f1c540a2f70 msgr2=0x7f1c54072440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:13.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.691+0000 7f1c5a67f640 1 -- 172.21.15.12:0/3536893686 shutdown_connections 2024-09-11T01:51:13.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.691+0000 7f1c5a67f640 1 -- 172.21.15.12:0/3536893686 wait complete. 2024-09-11T01:51:13.718 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:51:13.718 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:51:13.719 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:51:13.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.819+0000 7ff28f59e640 10 monclient: get_monmap_and_config 2024-09-11T01:51:13.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.819+0000 7ff28f59e640 10 monclient: build_initial_monmap 2024-09-11T01:51:13.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.819+0000 7ff28f59e640 10 monclient: monmap: 2024-09-11T01:51:13.822 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:13.822 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:13.822 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:51:13.820282+0000 2024-09-11T01:51:13.822 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:51:13.820282+0000 2024-09-11T01:51:13.822 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:13.822 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:13.822 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:51:13.822 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:51:13.822 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:51:13.822 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:51:13.822 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:51:13.823 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:13.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.820+0000 7ff28f59e640 1 Processor -- start 2024-09-11T01:51:13.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.820+0000 7ff28f59e640 1 -- start start 2024-09-11T01:51:13.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.820+0000 7ff28f59e640 10 monclient: init 2024-09-11T01:51:13.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.820+0000 7ff28f59e640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:13.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.820+0000 7ff28f59e640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-11T01:51:13.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.820+0000 7ff28f59e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff2900a01c0 0x7ff29009f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.820+0000 7ff28f59e640 10 monclient(hunting): picked mon.noname-b con 0x7ff2900a01c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:13.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.820+0000 7ff28f59e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff2900a0c20 addr v1:172.21.15.12:6789/0 2024-09-11T01:51:13.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.820+0000 7ff28f59e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff2900a1710 0x7ff29013ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.820+0000 7ff28f59e640 10 monclient(hunting): picked mon.noname-a con 0x7ff2900a1710 addr v2:172.21.15.12:3300/0 2024-09-11T01:51:13.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.820+0000 7ff28f59e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff29009e3c0 con 0x7ff2900a0c20 2024-09-11T01:51:13.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.820+0000 7ff28f59e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:13.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.820+0000 7ff28f59e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff29009ecd0 con 0x7ff2900a1710 2024-09-11T01:51:13.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.820+0000 7ff28f59e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:13.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.820+0000 7ff28f59e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff2900a05a0 con 0x7ff2900a01c0 2024-09-11T01:51:13.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.820+0000 7ff28f59e640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:13.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.820+0000 7ff28f59e640 10 monclient(hunting): authenticate will time out at 2080.262850s 2024-09-11T01:51:13.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.820+0000 7ff28ed9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff2900a1710 0x7ff29013ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.820+0000 7ff28e59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff2900a01c0 0x7ff29009f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.820+0000 7ff28ed9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff2900a1710 0x7ff29013ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60258/0 (socket says 172.21.15.12:60258) 2024-09-11T01:51:13.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.820+0000 7ff28e59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff2900a01c0 0x7ff29009f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:47946/0 (socket says 172.21.15.12:47946) 2024-09-11T01:51:13.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.821+0000 7ff28ed9d640 1 -- 172.21.15.12:0/979221025 learned_addr learned my addr 172.21.15.12:0/979221025 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:13.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.821+0000 7ff28ed9d640 10 monclient(hunting): get_auth_request con 0x7ff2900a1710 auth_method 0 2024-09-11T01:51:13.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.821+0000 7ff28ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:13.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.821+0000 7ff28ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.821+0000 7ff28ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:13.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.821+0000 7ff28e59c640 10 monclient(hunting): get_auth_request con 0x7ff2900a01c0 auth_method 0 2024-09-11T01:51:13.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.821+0000 7ff28e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:13.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.821+0000 7ff28e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.821+0000 7ff28e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:13.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.821+0000 7ff28e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:13.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.821+0000 7ff28e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:13.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.821+0000 7ff28e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:13.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.821+0000 7ff28e59c640 10 monclient(hunting): handle_auth_done global_id 244106 payload 274 2024-09-11T01:51:13.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.821+0000 7ff28e59c640 1 -- 172.21.15.12:0/979221025 >> v2:172.21.15.12:3300/0 conn(0x7ff2900a1710 msgr2=0x7ff29013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:13.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.821+0000 7ff28e59c640 1 --2- 172.21.15.12:0/979221025 >> v2:172.21.15.12:3300/0 conn(0x7ff2900a1710 0x7ff29013ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.821+0000 7ff28e59c640 1 -- 172.21.15.12:0/979221025 >> v1:172.21.15.12:6789/0 conn(0x7ff2900a0c20 legacy=0x7ff2900a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:13.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.821+0000 7ff28e59c640 10 monclient: _finish_hunting 0 2024-09-11T01:51:13.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.821+0000 7ff28e59c640 1 monclient: found mon.noname-b 2024-09-11T01:51:13.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.821+0000 7ff28e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:13.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.821+0000 7ff28e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:51:13.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.821+0000 7ff28e59c640 1 -- 172.21.15.12:0/979221025 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2900a20f0 con 0x7ff2900a01c0 2024-09-11T01:51:13.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28e59c640 1 -- 172.21.15.12:0/979221025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff2900a01c0 msgr2=0x7ff29009f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:13.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28e59c640 1 --2- 172.21.15.12:0/979221025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff2900a01c0 0x7ff29009f080 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff2800097a0 tx=0x7ff280029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c08a15a050fe763c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:13.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28d59a640 1 -- 172.21.15.12:0/979221025 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff280037070 con 0x7ff2900a01c0 2024-09-11T01:51:13.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:13.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:51:13.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28d59a640 10 monclient: dump: 2024-09-11T01:51:13.831 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:13.831 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:13.831 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.831 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.831 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:13.832 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:13.832 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:13.832 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:13.832 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:13.832 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:13.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28d59a640 10 monclient: _finish_auth 0 2024-09-11T01:51:13.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28d59a640 10 monclient: _check_auth_tickets 2024-09-11T01:51:13.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:13.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28d59a640 1 -- 172.21.15.12:0/979221025 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff280032440 con 0x7ff2900a01c0 2024-09-11T01:51:13.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28f59e640 5 monclient: authenticate success, global_id 244106 2024-09-11T01:51:13.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:51:13.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28d59a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:13.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28f59e640 10 monclient: get_monmap_and_config success 2024-09-11T01:51:13.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28d59a640 1 -- 172.21.15.12:0/979221025 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff280032810 con 0x7ff2900a01c0 2024-09-11T01:51:13.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:13.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28f59e640 4 set_mon_vals no callback set 2024-09-11T01:51:13.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:13.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28d59a640 10 monclient: dump: 2024-09-11T01:51:13.833 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:13.833 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:13.834 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.834 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.834 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:13.834 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:13.834 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:13.834 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:13.834 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:13.834 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:13.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28f59e640 10 monclient: shutdown 2024-09-11T01:51:13.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:13.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28f59e640 1 -- 172.21.15.12:0/979221025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff2900a01c0 msgr2=0x7ff29009f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:13.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28f59e640 1 --2- 172.21.15.12:0/979221025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff2900a01c0 0x7ff29009f080 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff2800097a0 tx=0x7ff280029fd0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28f59e640 1 -- 172.21.15.12:0/979221025 shutdown_connections 2024-09-11T01:51:13.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28f59e640 1 --2- 172.21.15.12:0/979221025 >> v2:172.21.15.12:3300/0 conn(0x7ff2900a1710 0x7ff29013ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28f59e640 1 --2- 172.21.15.12:0/979221025 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff2900a01c0 0x7ff29009f080 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28f59e640 1 -- 172.21.15.12:0/979221025 >> 172.21.15.12:0/979221025 conn(0x7ff2900a2f00 msgr2=0x7ff2900a3310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:13.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28f59e640 1 -- 172.21.15.12:0/979221025 shutdown_connections 2024-09-11T01:51:13.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.824+0000 7ff28f59e640 1 -- 172.21.15.12:0/979221025 wait complete. 2024-09-11T01:51:13.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.825+0000 7ff28f59e640 10 monclient: build_initial_monmap 2024-09-11T01:51:13.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.825+0000 7ff28f59e640 10 monclient: monmap: 2024-09-11T01:51:13.835 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:13.835 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:51:13.835 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:51:13.835 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:51:13.835 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:13.836 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:13.836 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:51:13.836 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:51:13.836 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:51:13.836 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:13.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.825+0000 7ff28f59e640 1 Processor -- start 2024-09-11T01:51:13.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28f59e640 1 -- start start 2024-09-11T01:51:13.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28f59e640 10 monclient: init 2024-09-11T01:51:13.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28f59e640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:13.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28f59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:51:13.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28f59e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2900a01c0 0x7ff2901ad080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28f59e640 10 monclient(hunting): picked mon.noname-b con 0x7ff2900a01c0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:51:13.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28f59e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2900a0c20 0x7ff2901cd420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28f59e640 10 monclient(hunting): picked mon.noname-a con 0x7ff2900a0c20 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:51:13.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28f59e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff2900a1710 0x7ff2901c6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28f59e640 10 monclient(hunting): picked mon.noname-c con 0x7ff2900a1710 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:13.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28f59e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:13.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28f59e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff29009ca30 con 0x7ff2900a0c20 2024-09-11T01:51:13.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28f59e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:13.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28f59e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff29009e6a0 con 0x7ff2900a01c0 2024-09-11T01:51:13.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28f59e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:13.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28f59e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff29009cbb0 con 0x7ff2900a1710 2024-09-11T01:51:13.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28f59e640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:13.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28f59e640 10 monclient(hunting): authenticate will time out at 2080.268492s 2024-09-11T01:51:13.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28ed9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff2900a1710 0x7ff2901c6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28dd9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2900a0c20 0x7ff2901cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28ed9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff2900a1710 0x7ff2901c6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:47952/0 (socket says 172.21.15.12:47952) 2024-09-11T01:51:13.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28ed9d640 1 -- 172.21.15.12:0/233129265 learned_addr learned my addr 172.21.15.12:0/233129265 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:13.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28ed9d640 10 monclient(hunting): get_auth_request con 0x7ff2900a1710 auth_method 0 2024-09-11T01:51:13.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:13.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:13.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28dd9b640 10 monclient(hunting): get_auth_request con 0x7ff2900a0c20 auth_method 0 2024-09-11T01:51:13.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:13.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:13.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:13.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:13.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:13.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:13.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28e59c640 1 --2- 172.21.15.12:0/233129265 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2900a01c0 0x7ff2901ad080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28e59c640 10 monclient(hunting): get_auth_request con 0x7ff2900a01c0 auth_method 0 2024-09-11T01:51:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28ed9d640 10 monclient(hunting): handle_auth_done global_id 244109 payload 824 2024-09-11T01:51:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28ed9d640 1 -- 172.21.15.12:0/233129265 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2900a01c0 msgr2=0x7ff2901ad080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28ed9d640 1 --2- 172.21.15.12:0/233129265 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2900a01c0 0x7ff2901ad080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28ed9d640 1 -- 172.21.15.12:0/233129265 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2900a0c20 msgr2=0x7ff2901cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28ed9d640 1 --2- 172.21.15.12:0/233129265 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff2900a0c20 0x7ff2901cd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28ed9d640 10 monclient: _finish_hunting 0 2024-09-11T01:51:13.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28ed9d640 1 monclient: found mon.noname-c 2024-09-11T01:51:13.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:13.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:51:13.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28ed9d640 1 -- 172.21.15.12:0/233129265 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2901ada10 con 0x7ff2900a1710 2024-09-11T01:51:13.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.826+0000 7ff28e59c640 1 --2- 172.21.15.12:0/233129265 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2900a01c0 0x7ff2901ad080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:51:13.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.829+0000 7ff28ed9d640 1 -- 172.21.15.12:0/233129265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff2900a1710 msgr2=0x7ff2901c6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:13.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.829+0000 7ff28ed9d640 1 --2- 172.21.15.12:0/233129265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff2900a1710 0x7ff2901c6e20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff2880042f0 tx=0x7ff2880047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:13.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.829+0000 7ff27effd640 1 -- 172.21.15.12:0/233129265 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff288019070 con 0x7ff2900a1710 2024-09-11T01:51:13.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.829+0000 7ff27effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:13.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.829+0000 7ff27effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:51:13.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.829+0000 7ff27effd640 10 monclient: dump: 2024-09-11T01:51:13.846 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:13.846 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:13.846 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.846 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.847 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:13.847 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:13.847 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:13.847 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:13.847 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:13.847 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:13.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.829+0000 7ff27effd640 10 monclient: _finish_auth 0 2024-09-11T01:51:13.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.829+0000 7ff27effd640 10 monclient: _check_auth_tickets 2024-09-11T01:51:13.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.829+0000 7ff27effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:13.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.829+0000 7ff27effd640 1 -- 172.21.15.12:0/233129265 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff28800ad80 con 0x7ff2900a1710 2024-09-11T01:51:13.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.829+0000 7ff28f59e640 5 monclient: authenticate success, global_id 244109 2024-09-11T01:51:13.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.829+0000 7ff28f59e640 10 monclient: _renew_subs 2024-09-11T01:51:13.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.829+0000 7ff28f59e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.829+0000 7ff28f59e640 1 -- 172.21.15.12:0/233129265 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff29009d410 con 0x7ff2900a1710 2024-09-11T01:51:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.829+0000 7ff27effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.829+0000 7ff27effd640 1 -- 172.21.15.12:0/233129265 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff288014630 con 0x7ff2900a1710 2024-09-11T01:51:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.829+0000 7ff27effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.829+0000 7ff27effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.829+0000 7ff27effd640 10 monclient: dump: 2024-09-11T01:51:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:13.848 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:13.849 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:13.849 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:13.849 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:13.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.829+0000 7ff28f59e640 10 monclient: _renew_subs 2024-09-11T01:51:13.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.829+0000 7ff28f59e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:13.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.829+0000 7ff28f59e640 1 -- 172.21.15.12:0/233129265 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2901adfe0 con 0x7ff2900a1710 2024-09-11T01:51:13.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.829+0000 7ff28cd99640 4 set_mon_vals no callback set 2024-09-11T01:51:13.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.830+0000 7ff28f59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:13.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.830+0000 7ff28f59e640 1 --2- 172.21.15.12:0/233129265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff2901c7360 0x7ff2901ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.830+0000 7ff28f59e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:13.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.830+0000 7ff28f59e640 1 -- 172.21.15.12:0/233129265 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff29006ce90 con 0x7ff2901c7360 2024-09-11T01:51:13.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.830+0000 7ff27effd640 1 -- 172.21.15.12:0/233129265 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff2880083e0 con 0x7ff2900a1710 2024-09-11T01:51:13.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.830+0000 7ff28e59c640 1 --2- 172.21.15.12:0/233129265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff2901c7360 0x7ff2901ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.830+0000 7ff28e59c640 10 monclient: get_auth_request con 0x7ff2901c7360 auth_method 0 2024-09-11T01:51:13.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.830+0000 7ff28e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:13.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.830+0000 7ff28e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.830+0000 7ff28e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:13.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.830+0000 7ff28e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:13.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.830+0000 7ff28e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:13.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.830+0000 7ff28e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:13.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.830+0000 7ff28e59c640 10 monclient(hunting): handle_auth_done global_id 244112 payload 824 2024-09-11T01:51:13.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.830+0000 7ff27effd640 1 --2- 172.21.15.12:0/233129265 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff27003e270 0x7ff270040730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.830+0000 7ff28e59c640 1 -- 172.21.15.12:0/233129265 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff29009d010 con 0x7ff2901c7360 2024-09-11T01:51:13.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.830+0000 7ff27effd640 1 -- 172.21.15.12:0/233129265 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(732..732 src has 507..732) ==== 4739+0+0 (secure 0 0 0) 0x7ff288052ac0 con 0x7ff2900a1710 2024-09-11T01:51:13.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.830+0000 7ff28dd9b640 1 --2- 172.21.15.12:0/233129265 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff27003e270 0x7ff270040730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.830+0000 7ff28dd9b640 10 monclient: get_auth_request con 0x7ff27003e270 auth_method 0 2024-09-11T01:51:13.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.832+0000 7ff28e59c640 1 -- 172.21.15.12:0/233129265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff2901c7360 msgr2=0x7ff2901ca1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:13.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.832+0000 7ff28e59c640 1 --2- 172.21.15.12:0/233129265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff2901c7360 0x7ff2901ca1c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff29009d3e0 tx=0x7ff2800046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:13.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.833+0000 7ff27effd640 1 -- 172.21.15.12:0/233129265 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff280042070 con 0x7ff2901c7360 2024-09-11T01:51:13.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.833+0000 7ff27effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:13.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.833+0000 7ff27effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:13.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.833+0000 7ff27effd640 10 monclient: dump: 2024-09-11T01:51:13.851 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:13.851 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:13.851 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.851 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.851 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:13.851 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:13.852 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:13.852 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:13.852 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:13.852 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:13.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.833+0000 7ff28dd9b640 1 --2- 172.21.15.12:0/233129265 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff27003e270 0x7ff270040730 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff284004f60 tx=0x7ff284007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:13.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.833+0000 7ff27effd640 1 -- 172.21.15.12:0/233129265 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff28003e870 con 0x7ff2901c7360 2024-09-11T01:51:13.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.833+0000 7ff27effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:13.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.833+0000 7ff27effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:51:13.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.833+0000 7ff27effd640 1 -- 172.21.15.12:0/233129265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff2901c7360 msgr2=0x7ff2901ca1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:13.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.833+0000 7ff27effd640 1 --2- 172.21.15.12:0/233129265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff2901c7360 0x7ff2901ca1c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff29009d3e0 tx=0x7ff2800046c0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.833+0000 7ff27effd640 1 -- 172.21.15.12:0/233129265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff2901c7360 msgr2=0x7ff2901ca1c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:51:13.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.833+0000 7ff27effd640 1 --2- 172.21.15.12:0/233129265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff2901c7360 0x7ff2901ca1c0 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff29009d3e0 tx=0x7ff2800046c0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.839+0000 7ff28f59e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:51:13.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.839+0000 7ff28f59e640 1 --2- 172.21.15.12:0/233129265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff29002d440 0x7ff29002d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.839+0000 7ff28f59e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:13.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.839+0000 7ff28f59e640 1 -- 172.21.15.12:0/233129265 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff290073a30 con 0x7ff29002d440 2024-09-11T01:51:13.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.839+0000 7ff28e59c640 1 --2- 172.21.15.12:0/233129265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff29002d440 0x7ff29002d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.839+0000 7ff28e59c640 10 monclient: get_auth_request con 0x7ff29002d440 auth_method 0 2024-09-11T01:51:13.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.839+0000 7ff28e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:13.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.839+0000 7ff28e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.839+0000 7ff28e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:13.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.839+0000 7ff28e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:13.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.839+0000 7ff28e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:13.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.839+0000 7ff28e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:13.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.839+0000 7ff28e59c640 10 monclient(hunting): handle_auth_done global_id 244115 payload 824 2024-09-11T01:51:13.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.839+0000 7ff28e59c640 1 -- 172.21.15.12:0/233129265 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff2901a2c90 con 0x7ff29002d440 2024-09-11T01:51:13.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.842+0000 7ff28e59c640 1 -- 172.21.15.12:0/233129265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff29002d440 msgr2=0x7ff29002d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:13.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.842+0000 7ff28e59c640 1 --2- 172.21.15.12:0/233129265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff29002d440 0x7ff29002d850 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff280000c00 tx=0x7ff280008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:13.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.842+0000 7ff27effd640 1 -- 172.21.15.12:0/233129265 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff280037040 con 0x7ff29002d440 2024-09-11T01:51:13.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.842+0000 7ff27effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:13.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.842+0000 7ff27effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:13.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.842+0000 7ff27effd640 10 monclient: dump: 2024-09-11T01:51:13.855 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:13.855 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:13.855 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.855 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.856 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:13.856 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:13.856 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:13.856 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:13.856 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:13.856 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:13.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.842+0000 7ff27effd640 1 -- 172.21.15.12:0/233129265 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7ff280003820 con 0x7ff29002d440 2024-09-11T01:51:13.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.842+0000 7ff27effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:51:13.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.842+0000 7ff27effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:51:13.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.842+0000 7ff27effd640 1 -- 172.21.15.12:0/233129265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff29002d440 msgr2=0x7ff29002d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:13.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.842+0000 7ff27effd640 1 --2- 172.21.15.12:0/233129265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff29002d440 0x7ff29002d850 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff280000c00 tx=0x7ff280008030 comp rx=0 tx=0).stop 2024-09-11T01:51:13.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.842+0000 7ff27effd640 1 -- 172.21.15.12:0/233129265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff29002d440 msgr2=0x7ff29002d850 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:51:13.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.842+0000 7ff27effd640 1 --2- 172.21.15.12:0/233129265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff29002d440 0x7ff29002d850 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff280000c00 tx=0x7ff280008030 comp rx=0 tx=0).stop 2024-09-11T01:51:13.857 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:51:13.857 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:51:13.857 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:51:13.857 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:51:13.857 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 208, 2024-09-11T01:51:13.857 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 27290, 2024-09-11T01:51:13.857 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:51:13.857 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:51:13.857 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:51:13.858 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:51:13.858 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:13.858 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:51:13.858 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:51:13.858 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:51:13.858 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:51:13.858 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:13.858 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:13.858 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:13.858 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:13.858 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:13.858 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:13.858 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:13.858 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:13.859 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:13.859 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:13.859 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:13.859 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:13.859 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:51:13.859 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:51:13.859 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:13.859 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:13.859 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:13.859 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:13.859 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:13.859 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:13.859 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:13.859 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:13.860 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:13.860 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:13.860 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:13.860 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:13.860 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:13.860 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:51:13.860 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:51:13.860 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:51:13.860 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:51:13.860 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:51:13.860 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:51:13.860 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:51:13.861 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:51:13.861 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:51:13.861 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:51:13.861 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:51:13.861 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:51:13.861 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:51:13.861 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:51:13.861 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:51:13.861 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:51:13.861 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:51:13.861 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:13.861 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:13.861 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:13.862 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:13.862 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:13.862 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:13.862 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:13.862 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:13.862 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:13.862 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:13.862 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:13.862 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:13.862 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:51:13.862 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:13.862 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:51:13.862 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:13.862 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:51:13.863 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:51:13.863 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:13.863 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:13.863 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:13.863 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:13.863 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:51:13.863 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:13.863 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:13.863 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:13.863 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:13.863 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:51:13.864 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:13.864 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:13.864 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:13.864 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:13.864 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:51:13.864 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:51:13.864 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:13.864 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:13.864 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:13.864 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:13.864 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:13.864 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:51:13.865 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:51:13.865 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:13.865 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:13.865 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:13.865 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:13.865 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:51:13.865 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:13.865 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:13.865 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:13.865 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:13.865 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:51:13.865 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:13.865 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:13.866 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:13.866 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:13.866 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:51:13.866 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:51:13.866 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:13.866 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:13.866 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:13.866 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:13.866 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:13.866 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:51:13.866 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:51:13.866 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:13.867 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:13.867 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:13.867 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:13.867 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:51:13.867 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:13.867 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:13.867 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:13.867 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:13.867 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:51:13.867 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:13.867 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:13.868 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:13.868 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:13.868 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:51:13.868 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:51:13.868 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:13.868 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:13.868 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:13.868 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:13.868 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:13.868 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:13.868 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:51:13.868 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:51:13.868 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:13.869 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:13.869 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:13.869 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:51:13.869 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:13.869 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:13.869 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:51:13.869 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:13.869 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:13.869 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:13.869 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:51:13.869 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:13.869 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:13.869 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:13.870 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:51:13.870 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:51:13.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.842+0000 7ff28f59e640 1 -- 172.21.15.12:0/233129265 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff27003e270 msgr2=0x7ff270040730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:13.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.842+0000 7ff28f59e640 1 --2- 172.21.15.12:0/233129265 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff27003e270 0x7ff270040730 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff284004f60 tx=0x7ff284007e30 comp rx=0 tx=0).stop 2024-09-11T01:51:13.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.842+0000 7ff28f59e640 10 monclient: shutdown 2024-09-11T01:51:13.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.842+0000 7ff28f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:13.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.842+0000 7ff28f59e640 1 -- 172.21.15.12:0/233129265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff2900a1710 msgr2=0x7ff2901c6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:13.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.842+0000 7ff28ed9d640 1 -- 172.21.15.12:0/233129265 reap_dead start 2024-09-11T01:51:13.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.842+0000 7ff28f59e640 1 --2- 172.21.15.12:0/233129265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff2900a1710 0x7ff2901c6e20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff2880042f0 tx=0x7ff2880047b0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.842+0000 7ff28f59e640 1 -- 172.21.15.12:0/233129265 shutdown_connections 2024-09-11T01:51:13.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.842+0000 7ff28f59e640 1 --2- 172.21.15.12:0/233129265 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff2900a1710 0x7ff2901c6e20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.842+0000 7ff28f59e640 1 -- 172.21.15.12:0/233129265 >> 172.21.15.12:0/233129265 conn(0x7ff2900a2f00 msgr2=0x7ff2900a5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:13.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.842+0000 7ff28f59e640 1 -- 172.21.15.12:0/233129265 shutdown_connections 2024-09-11T01:51:13.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.843+0000 7ff28f59e640 1 -- 172.21.15.12:0/233129265 wait complete. 2024-09-11T01:51:13.872 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:51:13.872 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:51:13.873 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:51:13.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.973+0000 7f09c52a9640 10 monclient: get_monmap_and_config 2024-09-11T01:51:13.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.973+0000 7f09c52a9640 10 monclient: build_initial_monmap 2024-09-11T01:51:13.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.973+0000 7f09c52a9640 10 monclient: monmap: 2024-09-11T01:51:13.976 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:13.976 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:13.976 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:51:13.974540+0000 2024-09-11T01:51:13.976 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:51:13.974540+0000 2024-09-11T01:51:13.976 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:13.976 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:13.976 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:51:13.976 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:51:13.976 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:51:13.976 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:51:13.977 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:51:13.977 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:13.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.974+0000 7f09c52a9640 1 Processor -- start 2024-09-11T01:51:13.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.974+0000 7f09c52a9640 1 -- start start 2024-09-11T01:51:13.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.974+0000 7f09c52a9640 10 monclient: init 2024-09-11T01:51:13.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.974+0000 7f09c52a9640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:13.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.974+0000 7f09c52a9640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-11T01:51:13.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.974+0000 7f09c52a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f09c00a2e20 addr v1:172.21.15.71:6789/0 2024-09-11T01:51:13.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.974+0000 7f09c52a9640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f09c00a4970 0x7f09c006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.974+0000 7f09c52a9640 10 monclient(hunting): picked mon.noname-a con 0x7f09c00a4970 addr v2:172.21.15.12:3300/0 2024-09-11T01:51:13.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.974+0000 7f09c52a9640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f09c006f910 0x7f09c013f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.974+0000 7f09c52a9640 10 monclient(hunting): picked mon.noname-c con 0x7f09c006f910 addr v2:172.21.15.71:3300/0 2024-09-11T01:51:13.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.974+0000 7f09c52a9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09c006f4f0 con 0x7f09c00a2e20 2024-09-11T01:51:13.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.974+0000 7f09c52a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:13.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.974+0000 7f09c52a9640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f09c006ef30 con 0x7f09c00a4970 2024-09-11T01:51:13.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.974+0000 7f09c52a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:13.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.974+0000 7f09c52a9640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f09c006ec50 con 0x7f09c006f910 2024-09-11T01:51:13.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.974+0000 7f09c52a9640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:13.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09c52a9640 10 monclient(hunting): authenticate will time out at 2080.417008s 2024-09-11T01:51:13.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09b67fc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f09c00a4970 0x7f09c006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09b67fc640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f09c00a4970 0x7f09c006ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60270/0 (socket says 172.21.15.12:60270) 2024-09-11T01:51:13.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09b67fc640 1 -- 172.21.15.12:0/4145315932 learned_addr learned my addr 172.21.15.12:0/4145315932 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:13.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09b67fc640 10 monclient(hunting): get_auth_request con 0x7f09c00a4970 auth_method 0 2024-09-11T01:51:13.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09b67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:13.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:13.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09bf7fe640 1 --2- 172.21.15.12:0/4145315932 >> v2:172.21.15.71:3300/0 conn(0x7f09c006f910 0x7f09c013f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09bf7fe640 10 monclient(hunting): get_auth_request con 0x7f09c006f910 auth_method 0 2024-09-11T01:51:13.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:13.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:13.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09b67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:13.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09b67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:13.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09b67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:13.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:13.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:13.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:13.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09b67fc640 10 monclient(hunting): handle_auth_done global_id 214404 payload 274 2024-09-11T01:51:13.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09b67fc640 1 -- 172.21.15.12:0/4145315932 >> v2:172.21.15.71:3300/0 conn(0x7f09c006f910 msgr2=0x7f09c013f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:13.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09b67fc640 1 --2- 172.21.15.12:0/4145315932 >> v2:172.21.15.71:3300/0 conn(0x7f09c006f910 0x7f09c013f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09b67fc640 1 -- 172.21.15.12:0/4145315932 >> v1:172.21.15.71:6789/0 conn(0x7f09c00a2e20 legacy=0x7f09c00a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:13.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09b67fc640 10 monclient: _finish_hunting 0 2024-09-11T01:51:13.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09b67fc640 1 monclient: found mon.noname-a 2024-09-11T01:51:13.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:13.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09b67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:51:13.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.975+0000 7f09b67fc640 1 -- 172.21.15.12:0/4145315932 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c00a3910 con 0x7f09c00a4970 2024-09-11T01:51:13.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.978+0000 7f09b67fc640 1 -- 172.21.15.12:0/4145315932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c00a4970 msgr2=0x7f09c006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:13.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.978+0000 7f09b67fc640 1 --2- 172.21.15.12:0/4145315932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c00a4970 0x7f09c006ded0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f09c00a62d0 tx=0x7f09a802a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a8a239ffc2ae1c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:13.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.978+0000 7f09be7fc640 1 -- 172.21.15.12:0/4145315932 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a8037070 con 0x7f09c00a4970 2024-09-11T01:51:13.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.978+0000 7f09be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:13.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.978+0000 7f09be7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:51:13.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.978+0000 7f09be7fc640 10 monclient: dump: 2024-09-11T01:51:13.986 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:13.986 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:13.986 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.986 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.986 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:13.986 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:13.986 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:13.986 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:13.986 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:13.986 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:13.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.978+0000 7f09be7fc640 10 monclient: _finish_auth 0 2024-09-11T01:51:13.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.978+0000 7f09be7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:51:13.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.978+0000 7f09be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:13.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.978+0000 7f09be7fc640 1 -- 172.21.15.12:0/4145315932 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09a8032640 con 0x7f09c00a4970 2024-09-11T01:51:13.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.978+0000 7f09c52a9640 5 monclient: authenticate success, global_id 214404 2024-09-11T01:51:13.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.978+0000 7f09c52a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:51:13.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.978+0000 7f09be7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:13.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.978+0000 7f09c52a9640 10 monclient: get_monmap_and_config success 2024-09-11T01:51:13.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.978+0000 7f09be7fc640 1 -- 172.21.15.12:0/4145315932 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a8032a10 con 0x7f09c00a4970 2024-09-11T01:51:13.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.978+0000 7f09c52a9640 4 set_mon_vals no callback set 2024-09-11T01:51:13.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.978+0000 7f09be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:13.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.978+0000 7f09be7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:51:13.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.978+0000 7f09be7fc640 10 monclient: dump: 2024-09-11T01:51:13.987 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:13.987 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:13.987 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.987 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:13.988 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:13.988 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:13.988 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:13.988 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:13.988 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:13.988 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:13.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.978+0000 7f09c52a9640 10 monclient: shutdown 2024-09-11T01:51:13.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.978+0000 7f09c52a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:13.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.978+0000 7f09c52a9640 1 -- 172.21.15.12:0/4145315932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c00a4970 msgr2=0x7f09c006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:13.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.978+0000 7f09c52a9640 1 --2- 172.21.15.12:0/4145315932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c00a4970 0x7f09c006ded0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f09c00a62d0 tx=0x7f09a802a0b0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.979+0000 7f09c52a9640 1 -- 172.21.15.12:0/4145315932 shutdown_connections 2024-09-11T01:51:13.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.979+0000 7f09c52a9640 1 --2- 172.21.15.12:0/4145315932 >> v2:172.21.15.71:3300/0 conn(0x7f09c006f910 0x7f09c013f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.979+0000 7f09c52a9640 1 --2- 172.21.15.12:0/4145315932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c00a4970 0x7f09c006ded0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.979+0000 7f09c52a9640 1 -- 172.21.15.12:0/4145315932 >> 172.21.15.12:0/4145315932 conn(0x7f09c00a8600 msgr2=0x7f09c00a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:13.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.979+0000 7f09c52a9640 1 -- 172.21.15.12:0/4145315932 shutdown_connections 2024-09-11T01:51:13.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.979+0000 7f09c52a9640 1 -- 172.21.15.12:0/4145315932 wait complete. 2024-09-11T01:51:13.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.979+0000 7f09c52a9640 10 monclient: build_initial_monmap 2024-09-11T01:51:13.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.979+0000 7f09c52a9640 10 monclient: monmap: 2024-09-11T01:51:13.989 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:13.989 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:51:13.989 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:51:13.989 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:51:13.989 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:13.989 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:13.990 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:51:13.990 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:51:13.990 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:51:13.990 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:13.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.979+0000 7f09c52a9640 1 Processor -- start 2024-09-11T01:51:13.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09c52a9640 1 -- start start 2024-09-11T01:51:13.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09c52a9640 10 monclient: init 2024-09-11T01:51:13.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09c52a9640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09c52a9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:51:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09c52a9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09c006f910 0x7f09c0138400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09c52a9640 10 monclient(hunting): picked mon.noname-b con 0x7f09c006f910 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:51:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09c52a9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c00a2e20 0x7f09c0138940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09c52a9640 10 monclient(hunting): picked mon.noname-a con 0x7f09c00a2e20 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:51:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09c52a9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09c00a4970 0x7f09c013b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09c52a9640 10 monclient(hunting): picked mon.noname-c con 0x7f09c00a4970 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09c52a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09c52a9640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f09c013fd00 con 0x7f09c00a2e20 2024-09-11T01:51:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09c52a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09c52a9640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f09c013fb80 con 0x7f09c006f910 2024-09-11T01:51:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09c52a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09c52a9640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f09c013fe80 con 0x7f09c00a4970 2024-09-11T01:51:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09c52a9640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09c52a9640 10 monclient(hunting): authenticate will time out at 2080.422572s 2024-09-11T01:51:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09b67fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c00a2e20 0x7f09c0138940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09b67fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c00a2e20 0x7f09c0138940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60278/0 (socket says 172.21.15.12:60278) 2024-09-11T01:51:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09b67fc640 1 -- 172.21.15.12:0/3796784863 learned_addr learned my addr 172.21.15.12:0/3796784863 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09bf7fe640 1 --2- 172.21.15.12:0/3796784863 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09c00a4970 0x7f09c013b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09b67fc640 10 monclient(hunting): get_auth_request con 0x7f09c00a2e20 auth_method 0 2024-09-11T01:51:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09b67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09bf7fe640 10 monclient(hunting): get_auth_request con 0x7f09c00a4970 auth_method 0 2024-09-11T01:51:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:13.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:13.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09beffd640 1 --2- 172.21.15.12:0/3796784863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09c006f910 0x7f09c0138400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:13.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09b67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:13.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09b67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:13.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09b67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:13.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09beffd640 10 monclient(hunting): get_auth_request con 0x7f09c006f910 auth_method 0 2024-09-11T01:51:13.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:13.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09beffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:13.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:13.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:13.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:13.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:13.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:13.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:13.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:13.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09b67fc640 10 monclient(hunting): handle_auth_done global_id 214407 payload 824 2024-09-11T01:51:13.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09b67fc640 1 -- 172.21.15.12:0/3796784863 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09c00a4970 msgr2=0x7f09c013b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:13.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09b67fc640 1 --2- 172.21.15.12:0/3796784863 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f09c00a4970 0x7f09c013b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09b67fc640 1 -- 172.21.15.12:0/3796784863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09c006f910 msgr2=0x7f09c0138400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:13.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09b67fc640 1 --2- 172.21.15.12:0/3796784863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09c006f910 0x7f09c0138400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:13.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09b67fc640 10 monclient: _finish_hunting 0 2024-09-11T01:51:13.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09b67fc640 1 monclient: found mon.noname-a 2024-09-11T01:51:13.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:13.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09b67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:51:14.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.980+0000 7f09b67fc640 1 -- 172.21.15.12:0/3796784863 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c013f210 con 0x7f09c00a2e20 2024-09-11T01:51:14.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.983+0000 7f09b67fc640 1 -- 172.21.15.12:0/3796784863 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c00a2e20 msgr2=0x7f09c0138940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:14.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.983+0000 7f09b67fc640 1 --2- 172.21.15.12:0/3796784863 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c00a2e20 0x7f09c0138940 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f09a80090b0 tx=0x7f09a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:14.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.983+0000 7f09b7fff640 1 -- 172.21.15.12:0/3796784863 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a8041020 con 0x7f09c00a2e20 2024-09-11T01:51:14.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.983+0000 7f09b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:14.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.983+0000 7f09b7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:51:14.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.983+0000 7f09b7fff640 10 monclient: dump: 2024-09-11T01:51:14.000 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:14.000 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:14.000 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:14.000 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:14.000 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:14.000 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:14.001 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:14.001 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:14.001 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:14.001 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:14.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.983+0000 7f09b7fff640 10 monclient: _finish_auth 0 2024-09-11T01:51:14.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.983+0000 7f09b7fff640 10 monclient: _check_auth_tickets 2024-09-11T01:51:14.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.983+0000 7f09b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:14.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.983+0000 7f09b7fff640 1 -- 172.21.15.12:0/3796784863 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09a803d850 con 0x7f09c00a2e20 2024-09-11T01:51:14.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.983+0000 7f09b7fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:14.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.983+0000 7f09b7fff640 1 -- 172.21.15.12:0/3796784863 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a8037040 con 0x7f09c00a2e20 2024-09-11T01:51:14.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.983+0000 7f09c52a9640 5 monclient: authenticate success, global_id 214407 2024-09-11T01:51:14.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.983+0000 7f09c52a9640 10 monclient: _renew_subs 2024-09-11T01:51:14.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.983+0000 7f09c52a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:51:14.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.983+0000 7f09c52a9640 1 -- 172.21.15.12:0/3796784863 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09c009f610 con 0x7f09c00a2e20 2024-09-11T01:51:14.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.983+0000 7f09b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:14.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.983+0000 7f09b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:51:14.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.983+0000 7f09b7fff640 10 monclient: dump: 2024-09-11T01:51:14.002 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:14.002 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:14.002 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:14.002 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:14.002 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:14.002 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:14.002 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:14.002 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:14.002 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:14.002 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:14.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.983+0000 7f09bdffb640 4 set_mon_vals no callback set 2024-09-11T01:51:14.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.983+0000 7f09c52a9640 10 monclient: _renew_subs 2024-09-11T01:51:14.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.983+0000 7f09c52a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:51:14.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.983+0000 7f09c52a9640 1 -- 172.21.15.12:0/3796784863 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09c013ceb0 con 0x7f09c00a2e20 2024-09-11T01:51:14.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.984+0000 7f09c52a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:14.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.984+0000 7f09c52a9640 1 --2- 172.21.15.12:0/3796784863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09c01cd3a0 0x7f09c013bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:14.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.984+0000 7f09c52a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:14.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.984+0000 7f09c52a9640 1 -- 172.21.15.12:0/3796784863 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f09c006a060 con 0x7f09c01cd3a0 2024-09-11T01:51:14.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.984+0000 7f09b7fff640 1 -- 172.21.15.12:0/3796784863 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f09a803a030 con 0x7f09c00a2e20 2024-09-11T01:51:14.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.984+0000 7f09beffd640 1 --2- 172.21.15.12:0/3796784863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09c01cd3a0 0x7f09c013bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:14.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.984+0000 7f09beffd640 10 monclient: get_auth_request con 0x7f09c01cd3a0 auth_method 0 2024-09-11T01:51:14.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.984+0000 7f09beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:14.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.984+0000 7f09beffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:14.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.984+0000 7f09beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:14.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.984+0000 7f09b7fff640 1 --2- 172.21.15.12:0/3796784863 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f09a003e290 0x7f09a0040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:14.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.984+0000 7f09beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:14.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.984+0000 7f09beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:14.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.984+0000 7f09beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:14.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.984+0000 7f09b7fff640 1 -- 172.21.15.12:0/3796784863 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(732..732 src has 507..732) ==== 4739+0+0 (secure 0 0 0) 0x7f09a8072fd0 con 0x7f09c00a2e20 2024-09-11T01:51:14.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.984+0000 7f09beffd640 10 monclient(hunting): handle_auth_done global_id 234157 payload 824 2024-09-11T01:51:14.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.984+0000 7f09beffd640 1 -- 172.21.15.12:0/3796784863 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f09c006ded0 con 0x7f09c01cd3a0 2024-09-11T01:51:14.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.985+0000 7f09bf7fe640 1 --2- 172.21.15.12:0/3796784863 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f09a003e290 0x7f09a0040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:14.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.985+0000 7f09bf7fe640 10 monclient: get_auth_request con 0x7f09a003e290 auth_method 0 2024-09-11T01:51:14.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.987+0000 7f09beffd640 1 -- 172.21.15.12:0/3796784863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09c01cd3a0 msgr2=0x7f09c013bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:14.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.987+0000 7f09beffd640 1 --2- 172.21.15.12:0/3796784863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09c01cd3a0 0x7f09c013bad0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f09b000f7a0 tx=0x7f09b000fc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:14.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.987+0000 7f09b7fff640 1 -- 172.21.15.12:0/3796784863 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b0018030 con 0x7f09c01cd3a0 2024-09-11T01:51:14.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.987+0000 7f09b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:14.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.987+0000 7f09b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:51:14.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.987+0000 7f09b7fff640 10 monclient: dump: 2024-09-11T01:51:14.005 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:14.005 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:14.005 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:14.005 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:14.005 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:14.005 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:14.005 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:14.005 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:14.005 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:14.005 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:14.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.987+0000 7f09bf7fe640 1 --2- 172.21.15.12:0/3796784863 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f09a003e290 0x7f09a0040750 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f09c013e8a0 tx=0x7f09b800b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:14.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.987+0000 7f09b7fff640 1 -- 172.21.15.12:0/3796784863 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f09b000d9b0 con 0x7f09c01cd3a0 2024-09-11T01:51:14.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.987+0000 7f09b7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:14.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.987+0000 7f09b7fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:51:14.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.987+0000 7f09b7fff640 1 -- 172.21.15.12:0/3796784863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09c01cd3a0 msgr2=0x7f09c013bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:14.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.987+0000 7f09b7fff640 1 --2- 172.21.15.12:0/3796784863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09c01cd3a0 0x7f09c013bad0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f09b000f7a0 tx=0x7f09b000fc60 comp rx=0 tx=0).stop 2024-09-11T01:51:14.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.987+0000 7f09b7fff640 1 -- 172.21.15.12:0/3796784863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09c01cd3a0 msgr2=0x7f09c013bad0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:51:14.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.987+0000 7f09b7fff640 1 --2- 172.21.15.12:0/3796784863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09c01cd3a0 0x7f09c013bad0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:14.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.993+0000 7f09c52a9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:51:14.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.993+0000 7f09c52a9640 1 --2- 172.21.15.12:0/3796784863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09c013c010 0x7f09c009d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:14.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.993+0000 7f09c52a9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:14.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.993+0000 7f09c52a9640 1 -- 172.21.15.12:0/3796784863 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f09c013c3f0 con 0x7f09c013c010 2024-09-11T01:51:14.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.994+0000 7f09beffd640 1 --2- 172.21.15.12:0/3796784863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09c013c010 0x7f09c009d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:14.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.994+0000 7f09beffd640 10 monclient: get_auth_request con 0x7f09c013c010 auth_method 0 2024-09-11T01:51:14.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.994+0000 7f09beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:14.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.994+0000 7f09beffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:14.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.994+0000 7f09beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:14.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.994+0000 7f09beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:14.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.994+0000 7f09beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:14.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.994+0000 7f09beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:14.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.994+0000 7f09beffd640 10 monclient(hunting): handle_auth_done global_id 234160 payload 824 2024-09-11T01:51:14.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.994+0000 7f09beffd640 1 -- 172.21.15.12:0/3796784863 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f09c013ded0 con 0x7f09c013c010 2024-09-11T01:51:14.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.996+0000 7f09beffd640 1 -- 172.21.15.12:0/3796784863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09c013c010 msgr2=0x7f09c009d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:14.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.996+0000 7f09beffd640 1 --2- 172.21.15.12:0/3796784863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09c013c010 0x7f09c009d330 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f09b000f7a0 tx=0x7f09b0003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:14.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.997+0000 7f09b7fff640 1 -- 172.21.15.12:0/3796784863 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b001dd70 con 0x7f09c013c010 2024-09-11T01:51:14.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.997+0000 7f09b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:14.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.997+0000 7f09b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:51:14.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.997+0000 7f09b7fff640 10 monclient: dump: 2024-09-11T01:51:14.009 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:14.009 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:14.009 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:14.009 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:14.009 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:14.009 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:14.009 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:14.009 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:14.009 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:14.009 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:14.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.997+0000 7f09b7fff640 1 -- 172.21.15.12:0/3796784863 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7f09b001dd70 con 0x7f09c013c010 2024-09-11T01:51:14.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.997+0000 7f09b7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:51:14.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.997+0000 7f09b7fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:51:14.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.997+0000 7f09b7fff640 1 -- 172.21.15.12:0/3796784863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09c013c010 msgr2=0x7f09c009d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:14.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.997+0000 7f09b7fff640 1 --2- 172.21.15.12:0/3796784863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09c013c010 0x7f09c009d330 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f09b000f7a0 tx=0x7f09b0003760 comp rx=0 tx=0).stop 2024-09-11T01:51:14.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.997+0000 7f09b7fff640 1 -- 172.21.15.12:0/3796784863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09c013c010 msgr2=0x7f09c009d330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:51:14.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.997+0000 7f09b7fff640 1 --2- 172.21.15.12:0/3796784863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09c013c010 0x7f09c009d330 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f09b000f7a0 tx=0x7f09b0003760 comp rx=0 tx=0).stop 2024-09-11T01:51:14.010 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:51:14.010 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:51:14.011 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:51:14.011 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:51:14.011 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 208, 2024-09-11T01:51:14.011 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 75889, 2024-09-11T01:51:14.011 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:51:14.011 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:51:14.011 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:51:14.011 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:51:14.011 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:14.011 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:51:14.011 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:51:14.011 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:51:14.011 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:51:14.011 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:14.012 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:14.012 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:14.012 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:14.012 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:14.012 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:14.012 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:14.012 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:14.012 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:14.012 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:14.012 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:14.012 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:14.012 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:51:14.012 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:51:14.012 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:14.013 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:14.013 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:14.013 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:14.013 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:14.013 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:14.013 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:14.013 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:14.013 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:14.013 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:14.013 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:14.013 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:14.013 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:14.013 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:51:14.013 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:51:14.014 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:51:14.014 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:51:14.014 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:51:14.014 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:51:14.014 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:51:14.014 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:51:14.014 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:51:14.014 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:51:14.014 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:51:14.014 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:51:14.014 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:51:14.014 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:51:14.015 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:51:14.015 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:51:14.015 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:51:14.015 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:14.015 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:14.015 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:14.015 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:14.015 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:14.015 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:14.015 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:14.015 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:14.015 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:14.015 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:14.015 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:14.016 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:14.016 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:51:14.016 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:14.016 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:51:14.016 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:14.016 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:51:14.016 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:51:14.016 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:14.016 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:14.016 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:14.016 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:14.016 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:51:14.016 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:14.017 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:14.017 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:14.017 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:14.017 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:51:14.017 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:14.017 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:14.017 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:14.017 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:14.017 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:51:14.017 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:51:14.017 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:14.017 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:14.017 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:14.017 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:14.018 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:14.018 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:51:14.018 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:51:14.018 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:14.018 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:14.018 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:14.018 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:14.018 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:51:14.018 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:14.018 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:14.018 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:14.018 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:14.018 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:51:14.018 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:14.018 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:14.019 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:14.019 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:14.019 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:51:14.019 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:51:14.019 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:14.019 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:14.019 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:14.019 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:14.019 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:14.019 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:51:14.019 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:51:14.019 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:14.019 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:14.019 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:14.020 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:14.020 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:51:14.020 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:14.020 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:14.020 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:14.020 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:14.020 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:51:14.020 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:14.020 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:14.020 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:14.020 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:14.020 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:51:14.020 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:51:14.020 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:14.020 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:14.021 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:14.021 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:14.021 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:14.021 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:14.021 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:51:14.021 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:51:14.021 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:14.021 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:14.021 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:14.021 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:51:14.021 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:14.021 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:14.021 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:51:14.021 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:14.022 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:14.022 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:14.022 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:51:14.022 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:14.022 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:14.022 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:51:14.022 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:14.022 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:14.022 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:14.022 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:51:14.022 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:14.022 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:14.022 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:51:14.022 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:14.022 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:14.023 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:14.023 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:51:14.023 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:14.023 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:14.023 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:14.023 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:51:14.023 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:51:14.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.997+0000 7f09c52a9640 1 -- 172.21.15.12:0/3796784863 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f09a003e290 msgr2=0x7f09a0040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:14.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.997+0000 7f09c52a9640 1 --2- 172.21.15.12:0/3796784863 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f09a003e290 0x7f09a0040750 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f09c013e8a0 tx=0x7f09b800b040 comp rx=0 tx=0).stop 2024-09-11T01:51:14.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.997+0000 7f09c52a9640 10 monclient: shutdown 2024-09-11T01:51:14.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.997+0000 7f09c52a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:14.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.997+0000 7f09c52a9640 1 -- 172.21.15.12:0/3796784863 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c00a2e20 msgr2=0x7f09c0138940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:14.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.997+0000 7f09c52a9640 1 --2- 172.21.15.12:0/3796784863 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c00a2e20 0x7f09c0138940 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f09a80090b0 tx=0x7f09a80046c0 comp rx=0 tx=0).stop 2024-09-11T01:51:14.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.997+0000 7f09bf7fe640 1 -- 172.21.15.12:0/3796784863 reap_dead start 2024-09-11T01:51:14.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.997+0000 7f09bf7fe640 1 -- 172.21.15.12:0/3796784863 reap_dead start 2024-09-11T01:51:14.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.997+0000 7f09c52a9640 1 -- 172.21.15.12:0/3796784863 shutdown_connections 2024-09-11T01:51:14.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.997+0000 7f09c52a9640 1 -- 172.21.15.12:0/3796784863 >> 172.21.15.12:0/3796784863 conn(0x7f09c00a8600 msgr2=0x7f09c00a71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:14.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.998+0000 7f09c52a9640 1 -- 172.21.15.12:0/3796784863 shutdown_connections 2024-09-11T01:51:14.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:13.998+0000 7f09c52a9640 1 -- 172.21.15.12:0/3796784863 wait complete. 2024-09-11T01:51:14.026 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:51:14.026 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:51:14.026 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:51:14.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.125+0000 7fe4d6239640 10 monclient: get_monmap_and_config 2024-09-11T01:51:14.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.125+0000 7fe4d6239640 10 monclient: build_initial_monmap 2024-09-11T01:51:14.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.125+0000 7fe4d6239640 10 monclient: monmap: 2024-09-11T01:51:14.127 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:14.127 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:14.127 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:51:14.126290+0000 2024-09-11T01:51:14.127 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:51:14.126290+0000 2024-09-11T01:51:14.127 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:14.128 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:14.128 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:51:14.128 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:51:14.128 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:51:14.128 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:51:14.128 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:51:14.128 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:14.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.126+0000 7fe4d6239640 1 Processor -- start 2024-09-11T01:51:14.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.126+0000 7fe4d6239640 1 -- start start 2024-09-11T01:51:14.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.126+0000 7fe4d6239640 10 monclient: init 2024-09-11T01:51:14.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4d6239640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:14.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4d6239640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-11T01:51:14.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4d6239640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4d0135fc0 0x7fe4d01363a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:14.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4d6239640 10 monclient(hunting): picked mon.noname-b con 0x7fe4d0135fc0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:14.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4d6239640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe4d0074000 addr v1:172.21.15.12:6789/0 2024-09-11T01:51:14.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4d6239640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe4d00743e0 0x7fe4d00a3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:14.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4d6239640 10 monclient(hunting): picked mon.noname-a con 0x7fe4d00743e0 addr v2:172.21.15.12:3300/0 2024-09-11T01:51:14.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4d6239640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4d00732d0 con 0x7fe4d0074000 2024-09-11T01:51:14.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4d6239640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:14.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4d6239640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fe4d0073890 con 0x7fe4d00743e0 2024-09-11T01:51:14.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4d6239640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:14.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4d6239640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe4d0072000 con 0x7fe4d0135fc0 2024-09-11T01:51:14.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4d6239640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:14.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4d6239640 10 monclient(hunting): authenticate will time out at 2080.569286s 2024-09-11T01:51:14.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4cf7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4d0135fc0 0x7fe4d01363a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:14.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4cf7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4d0135fc0 0x7fe4d01363a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:48002/0 (socket says 172.21.15.12:48002) 2024-09-11T01:51:14.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4c7fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe4d0074000 0x7fe4d00a2e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36728/0 (socket says 172.21.15.12:36728) 2024-09-11T01:51:14.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4cffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe4d00743e0 0x7fe4d00a3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:14.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4cf7fe640 1 -- 172.21.15.12:0/2328411906 learned_addr learned my addr 172.21.15.12:0/2328411906 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:14.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4cf7fe640 10 monclient(hunting): get_auth_request con 0x7fe4d0135fc0 auth_method 0 2024-09-11T01:51:14.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:14.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:14.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:14.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4cffff640 10 monclient(hunting): get_auth_request con 0x7fe4d00743e0 auth_method 0 2024-09-11T01:51:14.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:14.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4cffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:14.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:14.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:14.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:14.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:14.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4cf7fe640 10 monclient(hunting): handle_auth_done global_id 244121 payload 274 2024-09-11T01:51:14.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4cf7fe640 1 -- 172.21.15.12:0/2328411906 >> v2:172.21.15.12:3300/0 conn(0x7fe4d00743e0 msgr2=0x7fe4d00a3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:14.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4cf7fe640 1 --2- 172.21.15.12:0/2328411906 >> v2:172.21.15.12:3300/0 conn(0x7fe4d00743e0 0x7fe4d00a3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:14.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4cf7fe640 1 -- 172.21.15.12:0/2328411906 >> v1:172.21.15.12:6789/0 conn(0x7fe4d0074000 legacy=0x7fe4d00a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:14.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4cf7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:51:14.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4cf7fe640 1 monclient: found mon.noname-b 2024-09-11T01:51:14.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:14.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4cf7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:51:14.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.127+0000 7fe4cf7fe640 1 -- 172.21.15.12:0/2328411906 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4d00747c0 con 0x7fe4d0135fc0 2024-09-11T01:51:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.130+0000 7fe4cf7fe640 1 -- 172.21.15.12:0/2328411906 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4d0135fc0 msgr2=0x7fe4d01363a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.130+0000 7fe4cf7fe640 1 --2- 172.21.15.12:0/2328411906 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4d0135fc0 0x7fe4d01363a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0009870 tx=0x7fe4c002a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd0e3f097063eaf4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.130+0000 7fe4ceffd640 1 -- 172.21.15.12:0/2328411906 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0037070 con 0x7fe4d0135fc0 2024-09-11T01:51:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.130+0000 7fe4ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.130+0000 7fe4ceffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:51:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.130+0000 7fe4ceffd640 10 monclient: dump: 2024-09-11T01:51:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:14.142 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.130+0000 7fe4ceffd640 10 monclient: _finish_auth 0 2024-09-11T01:51:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.130+0000 7fe4ceffd640 10 monclient: _check_auth_tickets 2024-09-11T01:51:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.130+0000 7fe4ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.130+0000 7fe4d6239640 5 monclient: authenticate success, global_id 244121 2024-09-11T01:51:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.130+0000 7fe4d6239640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:51:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.130+0000 7fe4ceffd640 1 -- 172.21.15.12:0/2328411906 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4c0032440 con 0x7fe4d0135fc0 2024-09-11T01:51:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.130+0000 7fe4ceffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.130+0000 7fe4d6239640 10 monclient: get_monmap_and_config success 2024-09-11T01:51:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.130+0000 7fe4d6239640 4 set_mon_vals no callback set 2024-09-11T01:51:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.130+0000 7fe4ceffd640 1 -- 172.21.15.12:0/2328411906 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0032810 con 0x7fe4d0135fc0 2024-09-11T01:51:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.130+0000 7fe4ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.130+0000 7fe4ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.130+0000 7fe4ceffd640 10 monclient: dump: 2024-09-11T01:51:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:14.144 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.131+0000 7fe4d6239640 10 monclient: shutdown 2024-09-11T01:51:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.131+0000 7fe4d6239640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.131+0000 7fe4d6239640 1 -- 172.21.15.12:0/2328411906 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4d0135fc0 msgr2=0x7fe4d01363a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.131+0000 7fe4d6239640 1 --2- 172.21.15.12:0/2328411906 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4d0135fc0 0x7fe4d01363a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0009870 tx=0x7fe4c002a0a0 comp rx=0 tx=0).stop 2024-09-11T01:51:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.131+0000 7fe4d6239640 1 -- 172.21.15.12:0/2328411906 shutdown_connections 2024-09-11T01:51:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.131+0000 7fe4d6239640 1 --2- 172.21.15.12:0/2328411906 >> v2:172.21.15.12:3300/0 conn(0x7fe4d00743e0 0x7fe4d00a3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.131+0000 7fe4d6239640 1 --2- 172.21.15.12:0/2328411906 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4d0135fc0 0x7fe4d01363a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.131+0000 7fe4d6239640 1 -- 172.21.15.12:0/2328411906 >> 172.21.15.12:0/2328411906 conn(0x7fe4d00a8600 msgr2=0x7fe4d00a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.131+0000 7fe4d6239640 1 -- 172.21.15.12:0/2328411906 shutdown_connections 2024-09-11T01:51:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.131+0000 7fe4d6239640 1 -- 172.21.15.12:0/2328411906 wait complete. 2024-09-11T01:51:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.132+0000 7fe4d6239640 10 monclient: build_initial_monmap 2024-09-11T01:51:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.132+0000 7fe4d6239640 10 monclient: monmap: 2024-09-11T01:51:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:51:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:51:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:51:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:51:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:51:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:51:14.146 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.132+0000 7fe4d6239640 1 Processor -- start 2024-09-11T01:51:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.132+0000 7fe4d6239640 1 -- start start 2024-09-11T01:51:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.132+0000 7fe4d6239640 10 monclient: init 2024-09-11T01:51:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.132+0000 7fe4d6239640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.132+0000 7fe4d6239640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:51:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.132+0000 7fe4d6239640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4d0074000 0x7fe4d01cafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.132+0000 7fe4d6239640 10 monclient(hunting): picked mon.noname-c con 0x7fe4d0074000 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.132+0000 7fe4d6239640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe4d00743e0 0x7fe4d0134ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.132+0000 7fe4d6239640 10 monclient(hunting): picked mon.noname-b con 0x7fe4d00743e0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:51:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.132+0000 7fe4d6239640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4d0135fc0 0x7fe4d011eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:14.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.132+0000 7fe4d6239640 10 monclient(hunting): picked mon.noname-a con 0x7fe4d0135fc0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:51:14.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.132+0000 7fe4d6239640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:14.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.132+0000 7fe4d6239640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d01368e0 con 0x7fe4d0135fc0 2024-09-11T01:51:14.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.132+0000 7fe4d6239640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:14.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.132+0000 7fe4d6239640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d00a3a70 con 0x7fe4d00743e0 2024-09-11T01:51:14.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.132+0000 7fe4d6239640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:14.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.132+0000 7fe4d6239640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe4d01222e0 con 0x7fe4d0074000 2024-09-11T01:51:14.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.132+0000 7fe4d6239640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:14.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.132+0000 7fe4d6239640 10 monclient(hunting): authenticate will time out at 2080.574984s 2024-09-11T01:51:14.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cf7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4d0074000 0x7fe4d01cafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:14.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4d0135fc0 0x7fe4d011eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:14.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cf7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4d0074000 0x7fe4d01cafd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:48014/0 (socket says 172.21.15.12:48014) 2024-09-11T01:51:14.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cf7fe640 1 -- 172.21.15.12:0/4036900326 learned_addr learned my addr 172.21.15.12:0/4036900326 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:14.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cf7fe640 10 monclient(hunting): get_auth_request con 0x7fe4d0074000 auth_method 0 2024-09-11T01:51:14.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:14.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:14.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:14.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cffff640 10 monclient(hunting): get_auth_request con 0x7fe4d0135fc0 auth_method 0 2024-09-11T01:51:14.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:14.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:14.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:14.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:14.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:14.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:14.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:14.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:14.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:14.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4c7fff640 1 --2- 172.21.15.12:0/4036900326 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe4d00743e0 0x7fe4d0134ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:14.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cf7fe640 10 monclient(hunting): handle_auth_done global_id 244124 payload 824 2024-09-11T01:51:14.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cf7fe640 1 -- 172.21.15.12:0/4036900326 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe4d00743e0 msgr2=0x7fe4d0134ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:14.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cf7fe640 1 --2- 172.21.15.12:0/4036900326 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe4d00743e0 0x7fe4d0134ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:14.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cf7fe640 1 -- 172.21.15.12:0/4036900326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4d0135fc0 msgr2=0x7fe4d011eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:14.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cf7fe640 1 --2- 172.21.15.12:0/4036900326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4d0135fc0 0x7fe4d011eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:14.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cf7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:51:14.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cf7fe640 1 monclient: found mon.noname-c 2024-09-11T01:51:14.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:14.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:51:14.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cf7fe640 1 -- 172.21.15.12:0/4036900326 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4d01225c0 con 0x7fe4d0074000 2024-09-11T01:51:14.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4c7fff640 10 monclient: get_auth_request con 0x7fe4d00743e0 auth_method 0 2024-09-11T01:51:14.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4cffff640 1 --2- 172.21.15.12:0/4036900326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4d0135fc0 0x7fe4d011eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:51:14.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.133+0000 7fe4c7fff640 1 --2- 172.21.15.12:0/4036900326 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe4d00743e0 0x7fe4d0134ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:51:14.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.135+0000 7fe4cf7fe640 1 -- 172.21.15.12:0/4036900326 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4d0074000 msgr2=0x7fe4d01cafd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:14.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.135+0000 7fe4cf7fe640 1 --2- 172.21.15.12:0/4036900326 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4d0074000 0x7fe4d01cafd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0009990 tx=0x7fe4c00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:14.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.135+0000 7fe4ccff9640 1 -- 172.21.15.12:0/4036900326 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0037040 con 0x7fe4d0074000 2024-09-11T01:51:14.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.135+0000 7fe4ccff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:14.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.135+0000 7fe4ccff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:51:14.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.135+0000 7fe4ccff9640 10 monclient: dump: 2024-09-11T01:51:14.151 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:14.151 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:14.151 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:14.151 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:14.151 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:14.151 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:14.151 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:14.151 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:14.151 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:14.151 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:14.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.135+0000 7fe4ccff9640 10 monclient: _finish_auth 0 2024-09-11T01:51:14.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.135+0000 7fe4ccff9640 10 monclient: _check_auth_tickets 2024-09-11T01:51:14.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.135+0000 7fe4ccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:14.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.135+0000 7fe4ccff9640 1 -- 172.21.15.12:0/4036900326 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4c0042070 con 0x7fe4d0074000 2024-09-11T01:51:14.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.135+0000 7fe4d6239640 5 monclient: authenticate success, global_id 244124 2024-09-11T01:51:14.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.135+0000 7fe4d6239640 10 monclient: _renew_subs 2024-09-11T01:51:14.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.135+0000 7fe4d6239640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:14.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.135+0000 7fe4d6239640 1 -- 172.21.15.12:0/4036900326 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4d009f610 con 0x7fe4d0074000 2024-09-11T01:51:14.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.135+0000 7fe4ccff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:14.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.135+0000 7fe4ccff9640 1 -- 172.21.15.12:0/4036900326 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c003b820 con 0x7fe4d0074000 2024-09-11T01:51:14.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.135+0000 7fe4ccff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:14.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.135+0000 7fe4ccff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:14.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.135+0000 7fe4ccff9640 10 monclient: dump: 2024-09-11T01:51:14.153 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:14.153 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:14.153 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:14.153 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:14.153 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:14.153 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:14.153 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:14.153 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:14.153 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:14.153 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:14.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.135+0000 7fe4cdffb640 4 set_mon_vals no callback set 2024-09-11T01:51:14.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.135+0000 7fe4d6239640 10 monclient: _renew_subs 2024-09-11T01:51:14.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.135+0000 7fe4d6239640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:14.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.135+0000 7fe4d6239640 1 -- 172.21.15.12:0/4036900326 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4d0120590 con 0x7fe4d0074000 2024-09-11T01:51:14.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.136+0000 7fe4ccff9640 1 -- 172.21.15.12:0/4036900326 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fe4c003a030 con 0x7fe4d0074000 2024-09-11T01:51:14.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.136+0000 7fe4d6239640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:14.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.136+0000 7fe4d6239640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:14.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.136+0000 7fe4d6239640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:14.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.136+0000 7fe4d6239640 1 -- 172.21.15.12:0/4036900326 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4d01363a0 con 0x7fe4d0074000 2024-09-11T01:51:14.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.136+0000 7fe4ccff9640 1 --2- 172.21.15.12:0/4036900326 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe4b003e210 0x7fe4b00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:14.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.137+0000 7fe4ccff9640 1 -- 172.21.15.12:0/4036900326 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(732..732 src has 507..732) ==== 4739+0+0 (secure 0 0 0) 0x7fe4c0072fa0 con 0x7fe4d0074000 2024-09-11T01:51:14.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.137+0000 7fe4c7fff640 1 --2- 172.21.15.12:0/4036900326 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe4b003e210 0x7fe4b00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:14.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.137+0000 7fe4c7fff640 10 monclient: get_auth_request con 0x7fe4b003e210 auth_method 0 2024-09-11T01:51:14.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.139+0000 7fe4c7fff640 1 --2- 172.21.15.12:0/4036900326 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe4b003e210 0x7fe4b00406d0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0121420 tx=0x7fe4b8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:14.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.141+0000 7fe4ccff9640 1 -- 172.21.15.12:0/4036900326 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe4d00732d0 con 0x7fe4d0074000 2024-09-11T01:51:14.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.141+0000 7fe4ccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:14.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.141+0000 7fe4ccff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:51:14.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.398+0000 7fe4d6239640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:51:14.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.398+0000 7fe4d6239640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:51:14.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.398+0000 7fe4d6239640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:14.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.398+0000 7fe4d6239640 1 -- 172.21.15.12:0/4036900326 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fe4d01357e0 con 0x7fe4d0074000 2024-09-11T01:51:14.761 INFO:tasks.rados.rados.0.smithi071.stdout:3274: done (2 left) 2024-09-11T01:51:14.761 INFO:tasks.rados.rados.0.smithi071.stdout:3275: done (1 left) 2024-09-11T01:51:14.761 INFO:tasks.rados.rados.0.smithi071.stdout:3276: done (0 left) 2024-09-11T01:51:14.761 INFO:tasks.rados.rados.0.smithi071.stdout:3277: snap_remove snap 365 2024-09-11T01:51:14.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.399+0000 7fe4ccff9640 1 -- 172.21.15.12:0/4036900326 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fe4c00719c0 con 0x7fe4d0074000 2024-09-11T01:51:14.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.399+0000 7fe4ccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:51:14.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.399+0000 7fe4ccff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:51:14.763 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.400+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.763 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.400+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=11,kv=10,logm=48} crc {auth=3978021606,config=2586702516,health=3184170658,kv=1222170128,logm=2587851520}) 2024-09-11T01:51:14.763 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.400+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=12,kv=11,logm=47} crc {auth=3357837358,config=2586702516,health=2499132454,kv=4243670985,logm=483650922}) 2024-09-11T01:51:14.763 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.400+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.763 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.400+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=11,kv=10,logm=48} crc {auth=3978021606,config=2586702516,health=3184170658,kv=1222170128,logm=2587851520}) 2024-09-11T01:51:14.763 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.400+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=12,kv=11,logm=45} crc {auth=2891375500,config=2586702516,health=2499132454,kv=4243670985,logm=1616099961}) 2024-09-11T01:51:14.763 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.401+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.763 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.401+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=859931443}) 2024-09-11T01:51:14.763 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.401+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=642047827}) 2024-09-11T01:51:14.763 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.401+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.764 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.401+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=859931443}) 2024-09-11T01:51:14.764 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.401+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=837767600}) 2024-09-11T01:51:14.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.402+0000 7fe4d6239640 1 -- 172.21.15.12:0/4036900326 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe4b003e210 msgr2=0x7fe4b00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:14.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.402+0000 7fe4d6239640 1 --2- 172.21.15.12:0/4036900326 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe4b003e210 0x7fe4b00406d0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0121420 tx=0x7fe4b8009040 comp rx=0 tx=0).stop 2024-09-11T01:51:14.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.402+0000 7fe4d6239640 10 monclient: shutdown 2024-09-11T01:51:14.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.402+0000 7fe4d6239640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.402+0000 7fe4d6239640 1 -- 172.21.15.12:0/4036900326 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4d0074000 msgr2=0x7fe4d01cafd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.402+0000 7fe4d6239640 1 --2- 172.21.15.12:0/4036900326 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4d0074000 0x7fe4d01cafd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0009990 tx=0x7fe4c00046c0 comp rx=0 tx=0).stop 2024-09-11T01:51:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.402+0000 7fe4d6239640 1 -- 172.21.15.12:0/4036900326 shutdown_connections 2024-09-11T01:51:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.402+0000 7fe4d6239640 1 --2- 172.21.15.12:0/4036900326 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe4b003e210 0x7fe4b00406d0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.402+0000 7fe4d6239640 1 --2- 172.21.15.12:0/4036900326 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe4d0135fc0 0x7fe4d011eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.402+0000 7fe4d6239640 1 --2- 172.21.15.12:0/4036900326 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe4d00743e0 0x7fe4d0134ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.402+0000 7fe4d6239640 1 --2- 172.21.15.12:0/4036900326 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4d0074000 0x7fe4d01cafd0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:14.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.402+0000 7fe4d6239640 1 -- 172.21.15.12:0/4036900326 >> 172.21.15.12:0/4036900326 conn(0x7fe4d00a8600 msgr2=0x7fe4d009e330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:14.765 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.402+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.766 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.402+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=874199721}) 2024-09-11T01:51:14.766 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.402+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3861896509}) 2024-09-11T01:51:14.766 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.402+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.402+0000 7fe4d6239640 1 -- 172.21.15.12:0/4036900326 shutdown_connections 2024-09-11T01:51:14.766 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.402+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=874199721}) 2024-09-11T01:51:14.766 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.402+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1592487272}) 2024-09-11T01:51:14.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:14.402+0000 7fe4d6239640 1 -- 172.21.15.12:0/4036900326 wait complete. 2024-09-11T01:51:14.773 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.403+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.773 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.403+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3863385160}) 2024-09-11T01:51:14.773 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.403+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4246589106}) 2024-09-11T01:51:14.774 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.403+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.774 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.403+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3863385160}) 2024-09-11T01:51:14.774 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.403+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3006911419}) 2024-09-11T01:51:14.774 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.404+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.774 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.404+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1451267891}) 2024-09-11T01:51:14.774 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.404+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3809200141}) 2024-09-11T01:51:14.774 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.404+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.774 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.404+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1451267891}) 2024-09-11T01:51:14.774 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.404+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3428162839}) 2024-09-11T01:51:14.774 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.405+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.774 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.405+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2753462819}) 2024-09-11T01:51:14.774 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.405+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2187670940}) 2024-09-11T01:51:14.774 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.405+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.775 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.405+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2753462819}) 2024-09-11T01:51:14.775 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.405+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1065675439}) 2024-09-11T01:51:14.775 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.406+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.775 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.406+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018917160}) 2024-09-11T01:51:14.775 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.406+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1400880314}) 2024-09-11T01:51:14.775 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.406+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.775 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.406+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018917160}) 2024-09-11T01:51:14.775 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.406+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=870764949}) 2024-09-11T01:51:14.775 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.407+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.775 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.407+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=332514151}) 2024-09-11T01:51:14.775 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.407+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=78552072}) 2024-09-11T01:51:14.775 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.407+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.775 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.407+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=332514151}) 2024-09-11T01:51:14.775 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.407+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2323691583}) 2024-09-11T01:51:14.776 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.407+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.776 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.407+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4116480148}) 2024-09-11T01:51:14.776 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.407+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3341213091}) 2024-09-11T01:51:14.776 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.407+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.407+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4116480148}) 2024-09-11T01:51:14.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.407+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=9143678}) 2024-09-11T01:51:14.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.408+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.408+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=514654753}) 2024-09-11T01:51:14.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.408+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=470386342}) 2024-09-11T01:51:14.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.408+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.408+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=514654753}) 2024-09-11T01:51:14.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.408+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2911081968}) 2024-09-11T01:51:14.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.409+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.409+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3688232571}) 2024-09-11T01:51:14.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.409+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3747269146}) 2024-09-11T01:51:14.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.409+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.409+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3688232571}) 2024-09-11T01:51:14.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.409+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=740362916}) 2024-09-11T01:51:14.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.410+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.410+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3996674541}) 2024-09-11T01:51:14.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.410+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2668375290}) 2024-09-11T01:51:14.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.410+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.410+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3996674541}) 2024-09-11T01:51:14.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.410+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1121924274}) 2024-09-11T01:51:14.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.411+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.411+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=488058391}) 2024-09-11T01:51:14.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.411+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2588465482}) 2024-09-11T01:51:14.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.411+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.411+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=488058391}) 2024-09-11T01:51:14.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.411+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2949135345}) 2024-09-11T01:51:14.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.411+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.412+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3008163850}) 2024-09-11T01:51:14.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.412+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=529286011}) 2024-09-11T01:51:14.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.412+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.412+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3008163850}) 2024-09-11T01:51:14.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.412+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4068257732}) 2024-09-11T01:51:14.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.412+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.412+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1537204510}) 2024-09-11T01:51:14.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.412+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3217586280}) 2024-09-11T01:51:14.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.412+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.412+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1537204510}) 2024-09-11T01:51:14.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.412+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2136500283}) 2024-09-11T01:51:14.780 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:51:14.829 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.413+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.829 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.413+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3071360134}) 2024-09-11T01:51:14.829 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.413+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3513539853}) 2024-09-11T01:51:14.829 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.413+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.413+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3071360134}) 2024-09-11T01:51:14.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.413+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1781826678}) 2024-09-11T01:51:14.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.414+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.414+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=231094588}) 2024-09-11T01:51:14.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.414+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=612045058}) 2024-09-11T01:51:14.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.414+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.414+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=231094588}) 2024-09-11T01:51:14.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.414+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1744498158}) 2024-09-11T01:51:14.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.415+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.415+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2365602181}) 2024-09-11T01:51:14.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.415+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3396652275}) 2024-09-11T01:51:14.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.415+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.415+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2365602181}) 2024-09-11T01:51:14.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.415+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2151223473}) 2024-09-11T01:51:14.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.415+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.415+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=52523770}) 2024-09-11T01:51:14.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.415+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1057569727}) 2024-09-11T01:51:14.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.415+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.415+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=52523770}) 2024-09-11T01:51:14.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.416+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=604244567}) 2024-09-11T01:51:14.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.416+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.416+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=253331220}) 2024-09-11T01:51:14.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.416+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3150993008}) 2024-09-11T01:51:14.832 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.416+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.832 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.416+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=253331220}) 2024-09-11T01:51:14.832 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.416+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1410153289}) 2024-09-11T01:51:14.832 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.417+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.832 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.417+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3647350156}) 2024-09-11T01:51:14.832 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.417+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1457133251}) 2024-09-11T01:51:14.832 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.417+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.832 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.417+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3647350156}) 2024-09-11T01:51:14.832 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.417+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3734427535}) 2024-09-11T01:51:14.832 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.418+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.832 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.418+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462875339}) 2024-09-11T01:51:14.833 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.418+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=8896889}) 2024-09-11T01:51:14.833 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.418+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.833 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.418+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462875339}) 2024-09-11T01:51:14.833 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.418+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1871741125}) 2024-09-11T01:51:14.833 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.418+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.833 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.418+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3719887666}) 2024-09-11T01:51:14.833 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.418+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3765232595}) 2024-09-11T01:51:14.833 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.419+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.833 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.419+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3719887666}) 2024-09-11T01:51:14.833 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.419+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=194481075}) 2024-09-11T01:51:14.833 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.419+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.833 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.419+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2912582711}) 2024-09-11T01:51:14.834 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.419+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3303746247}) 2024-09-11T01:51:14.834 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.419+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.834 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.419+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2912582711}) 2024-09-11T01:51:14.834 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.419+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3918106816}) 2024-09-11T01:51:14.834 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.420+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.834 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.420+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2819729831}) 2024-09-11T01:51:14.834 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.420+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1749114360}) 2024-09-11T01:51:14.834 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.420+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.834 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.420+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2819729831}) 2024-09-11T01:51:14.834 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.420+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1440988047}) 2024-09-11T01:51:14.834 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.421+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.834 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.421+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3754373184}) 2024-09-11T01:51:14.835 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.421+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2504387463}) 2024-09-11T01:51:14.835 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.421+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.835 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.421+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3754373184}) 2024-09-11T01:51:14.835 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.421+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4193933402}) 2024-09-11T01:51:14.835 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.421+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.835 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.422+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1019022555}) 2024-09-11T01:51:14.835 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.422+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1036907389}) 2024-09-11T01:51:14.835 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.422+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.835 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.422+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1019022555}) 2024-09-11T01:51:14.835 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.422+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1097102948}) 2024-09-11T01:51:14.835 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.422+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.836 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.422+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3736668011}) 2024-09-11T01:51:14.836 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.422+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=652564440}) 2024-09-11T01:51:14.836 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.422+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.836 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.422+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3736668011}) 2024-09-11T01:51:14.836 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.422+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3184953236}) 2024-09-11T01:51:14.837 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.423+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.837 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.423+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1086579239}) 2024-09-11T01:51:14.837 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.423+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2974010618}) 2024-09-11T01:51:14.837 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.423+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.837 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.423+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1086579239}) 2024-09-11T01:51:14.837 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.423+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2733752104}) 2024-09-11T01:51:14.837 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.424+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.837 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.424+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3402103159}) 2024-09-11T01:51:14.837 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.424+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1081252848}) 2024-09-11T01:51:14.837 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.424+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.838 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.424+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3402103159}) 2024-09-11T01:51:14.838 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.424+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1419081673}) 2024-09-11T01:51:14.838 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.424+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.838 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.424+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3551278638}) 2024-09-11T01:51:14.838 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.424+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=436696542}) 2024-09-11T01:51:14.838 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.424+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.838 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.424+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3551278638}) 2024-09-11T01:51:14.838 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.424+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1958124296}) 2024-09-11T01:51:14.838 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.425+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.838 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.425+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=249792337}) 2024-09-11T01:51:14.838 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.425+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3757945294}) 2024-09-11T01:51:14.838 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.425+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.839 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.425+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=249792337}) 2024-09-11T01:51:14.839 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.425+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=820369786}) 2024-09-11T01:51:14.839 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.426+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.839 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.426+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4201103504}) 2024-09-11T01:51:14.839 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.426+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1042705987}) 2024-09-11T01:51:14.839 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.426+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.839 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.426+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4201103504}) 2024-09-11T01:51:14.839 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.426+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1355005080}) 2024-09-11T01:51:14.839 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.427+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.839 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.427+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1706021383}) 2024-09-11T01:51:14.839 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.427+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3596556910}) 2024-09-11T01:51:14.839 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.427+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.840 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.427+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1706021383}) 2024-09-11T01:51:14.840 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.427+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=594604164}) 2024-09-11T01:51:14.840 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.428+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.840 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.428+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4016128403}) 2024-09-11T01:51:14.840 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.428+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4187949020}) 2024-09-11T01:51:14.840 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.428+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.840 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.428+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4016128403}) 2024-09-11T01:51:14.840 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.428+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1978004590}) 2024-09-11T01:51:14.840 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.429+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.840 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.429+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2051991441}) 2024-09-11T01:51:14.840 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.429+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=762519724}) 2024-09-11T01:51:14.841 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.429+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.841 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.429+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2051991441}) 2024-09-11T01:51:14.841 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.429+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3502301173}) 2024-09-11T01:51:14.841 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.430+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.841 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.430+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1434442599}) 2024-09-11T01:51:14.841 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.430+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=812509799}) 2024-09-11T01:51:14.841 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.430+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.841 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.430+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1434442599}) 2024-09-11T01:51:14.841 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.430+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=971373400}) 2024-09-11T01:51:14.841 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.431+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.841 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.431+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018212136}) 2024-09-11T01:51:14.841 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.431+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3464495380}) 2024-09-11T01:51:14.842 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.431+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.842 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.431+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018212136}) 2024-09-11T01:51:14.842 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.431+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2907651488}) 2024-09-11T01:51:14.842 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.431+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.842 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.431+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2899327827}) 2024-09-11T01:51:14.842 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.431+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2123416653}) 2024-09-11T01:51:14.842 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.431+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.842 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.431+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2899327827}) 2024-09-11T01:51:14.842 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.431+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3608512280}) 2024-09-11T01:51:14.842 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.432+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.843 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.432+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3224139413}) 2024-09-11T01:51:14.843 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.432+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=194866545}) 2024-09-11T01:51:14.843 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.432+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.843 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.432+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3224139413}) 2024-09-11T01:51:14.843 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.432+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=363515889}) 2024-09-11T01:51:14.843 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.433+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.843 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.433+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3940754506}) 2024-09-11T01:51:14.843 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.433+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3229688730}) 2024-09-11T01:51:14.843 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.433+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.843 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.433+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3940754506}) 2024-09-11T01:51:14.843 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.433+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3928562028}) 2024-09-11T01:51:14.843 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.434+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.434+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3713951629}) 2024-09-11T01:51:14.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.434+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3180779594}) 2024-09-11T01:51:14.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.434+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.434+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3713951629}) 2024-09-11T01:51:14.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.434+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=33414330}) 2024-09-11T01:51:14.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.434+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.434+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=538284908}) 2024-09-11T01:51:14.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.434+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=497250230}) 2024-09-11T01:51:14.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.434+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.434+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=538284908}) 2024-09-11T01:51:14.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.434+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2298502560}) 2024-09-11T01:51:14.845 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.435+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.845 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.435+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=907621595}) 2024-09-11T01:51:14.845 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.435+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2252785243}) 2024-09-11T01:51:14.845 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.435+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.845 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.435+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=907621595}) 2024-09-11T01:51:14.845 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.435+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2304148348}) 2024-09-11T01:51:14.845 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.436+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.845 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.436+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=355873853}) 2024-09-11T01:51:14.845 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.436+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=155678371}) 2024-09-11T01:51:14.845 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.436+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.845 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.436+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=355873853}) 2024-09-11T01:51:14.845 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.436+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4153364220}) 2024-09-11T01:51:14.846 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.437+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.846 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.437+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=298999102}) 2024-09-11T01:51:14.846 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.437+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3367160876}) 2024-09-11T01:51:14.846 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.437+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.846 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.437+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=298999102}) 2024-09-11T01:51:14.846 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.437+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3322368246}) 2024-09-11T01:51:14.846 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.438+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.846 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.438+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3773065715}) 2024-09-11T01:51:14.846 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.438+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1246544124}) 2024-09-11T01:51:14.846 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.438+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.846 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.438+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3773065715}) 2024-09-11T01:51:14.846 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.438+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1753635374}) 2024-09-11T01:51:14.847 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.438+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.847 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.438+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1410435508}) 2024-09-11T01:51:14.847 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.438+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3776676900}) 2024-09-11T01:51:14.847 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.439+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.847 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.439+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1410435508}) 2024-09-11T01:51:14.847 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.439+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1021593207}) 2024-09-11T01:51:14.847 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.439+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.847 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.439+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3050190385}) 2024-09-11T01:51:14.847 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.439+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4072045629}) 2024-09-11T01:51:14.847 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.439+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.848 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.439+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3050190385}) 2024-09-11T01:51:14.848 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.439+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3301388151}) 2024-09-11T01:51:14.848 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.440+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.848 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.440+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2670721220}) 2024-09-11T01:51:14.848 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.440+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1490845219}) 2024-09-11T01:51:14.848 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.440+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.848 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.440+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2670721220}) 2024-09-11T01:51:14.848 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.440+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=776905160}) 2024-09-11T01:51:14.848 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.441+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.848 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.441+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1057824960}) 2024-09-11T01:51:14.849 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.441+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2671880845}) 2024-09-11T01:51:14.849 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.441+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.849 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.441+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1057824960}) 2024-09-11T01:51:14.849 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.441+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=243568518}) 2024-09-11T01:51:14.849 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.442+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.849 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.442+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3196438004}) 2024-09-11T01:51:14.849 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.442+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=855967958}) 2024-09-11T01:51:14.849 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.442+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.849 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.442+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3196438004}) 2024-09-11T01:51:14.850 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.442+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1062922294}) 2024-09-11T01:51:14.850 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.443+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.850 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.443+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1050417124}) 2024-09-11T01:51:14.850 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.443+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2965581189}) 2024-09-11T01:51:14.850 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.443+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.850 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.443+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1050417124}) 2024-09-11T01:51:14.850 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.443+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3860908943}) 2024-09-11T01:51:14.850 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.443+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.850 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.444+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1955623456}) 2024-09-11T01:51:14.851 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.444+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1055448549}) 2024-09-11T01:51:14.851 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.444+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.851 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.444+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1955623456}) 2024-09-11T01:51:14.851 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.444+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3498598452}) 2024-09-11T01:51:14.851 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.444+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.851 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.444+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2186221833}) 2024-09-11T01:51:14.851 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.444+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2004189053}) 2024-09-11T01:51:14.851 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.444+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.851 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.444+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2186221833}) 2024-09-11T01:51:14.851 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.444+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1605749016}) 2024-09-11T01:51:14.851 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.446+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.852 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.446+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1332985213}) 2024-09-11T01:51:14.852 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.446+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3122235727}) 2024-09-11T01:51:14.852 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.446+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.852 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.446+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1332985213}) 2024-09-11T01:51:14.852 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.446+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3341141931}) 2024-09-11T01:51:14.852 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.447+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.852 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.447+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2912662652}) 2024-09-11T01:51:14.852 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.447+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3146864175}) 2024-09-11T01:51:14.852 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.447+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.852 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.447+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2912662652}) 2024-09-11T01:51:14.852 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.447+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2163679814}) 2024-09-11T01:51:14.853 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.448+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.853 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.448+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1701430416}) 2024-09-11T01:51:14.853 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.448+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=735691227}) 2024-09-11T01:51:14.853 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.448+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.853 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.448+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1701430416}) 2024-09-11T01:51:14.853 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.448+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2723021327}) 2024-09-11T01:51:14.853 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.448+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.853 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.448+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=254326320}) 2024-09-11T01:51:14.853 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.448+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2987380807}) 2024-09-11T01:51:14.853 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.449+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.853 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.449+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=254326320}) 2024-09-11T01:51:14.853 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.449+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3395635237}) 2024-09-11T01:51:14.854 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.449+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.854 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.449+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=884034790}) 2024-09-11T01:51:14.854 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.449+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3768849906}) 2024-09-11T01:51:14.854 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.449+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.854 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.449+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=884034790}) 2024-09-11T01:51:14.854 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.449+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4176540789}) 2024-09-11T01:51:14.854 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.450+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.855 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.450+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=48962274}) 2024-09-11T01:51:14.855 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.450+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4279071412}) 2024-09-11T01:51:14.855 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.450+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.855 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.450+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=48962274}) 2024-09-11T01:51:14.855 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.450+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3700103171}) 2024-09-11T01:51:14.855 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.451+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.855 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.451+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=377232411}) 2024-09-11T01:51:14.855 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.451+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1626112143}) 2024-09-11T01:51:14.855 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.451+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.855 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.451+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=377232411}) 2024-09-11T01:51:14.855 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.451+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2414609731}) 2024-09-11T01:51:14.856 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.451+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.856 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.452+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=689584685}) 2024-09-11T01:51:14.856 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.452+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3216709241}) 2024-09-11T01:51:14.856 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.452+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.856 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.452+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=689584685}) 2024-09-11T01:51:14.856 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.452+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=223422639}) 2024-09-11T01:51:14.856 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.452+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.856 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.452+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=405193842}) 2024-09-11T01:51:14.856 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.452+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=336142787}) 2024-09-11T01:51:14.856 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.452+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.856 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.452+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=405193842}) 2024-09-11T01:51:14.857 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.452+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4107757519}) 2024-09-11T01:51:14.857 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.453+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.857 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.453+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2950788568}) 2024-09-11T01:51:14.857 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.453+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1632637110}) 2024-09-11T01:51:14.857 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.453+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.857 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.453+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2950788568}) 2024-09-11T01:51:14.857 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.453+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3213325542}) 2024-09-11T01:51:14.857 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.454+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.857 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.454+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2371457856}) 2024-09-11T01:51:14.857 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.454+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3705004957}) 2024-09-11T01:51:14.857 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.454+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.857 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.454+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2371457856}) 2024-09-11T01:51:14.858 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.454+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1657045614}) 2024-09-11T01:51:14.858 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.455+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.858 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.455+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3760927555}) 2024-09-11T01:51:14.858 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.455+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3870707213}) 2024-09-11T01:51:14.858 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.455+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.858 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.455+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3760927555}) 2024-09-11T01:51:14.858 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.455+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=841608741}) 2024-09-11T01:51:14.858 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.456+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.858 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.456+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1269187087}) 2024-09-11T01:51:14.858 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.456+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2130198574}) 2024-09-11T01:51:14.858 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.456+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.858 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.456+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1269187087}) 2024-09-11T01:51:14.859 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.456+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2279598006}) 2024-09-11T01:51:14.859 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.456+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.859 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.456+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=751703095}) 2024-09-11T01:51:14.859 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.456+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=723061690}) 2024-09-11T01:51:14.859 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.456+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.859 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.456+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=751703095}) 2024-09-11T01:51:14.859 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.456+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2521194904}) 2024-09-11T01:51:14.859 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.457+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.859 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.457+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1054000484}) 2024-09-11T01:51:14.859 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.457+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3899357145}) 2024-09-11T01:51:14.859 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.457+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.859 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.457+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1054000484}) 2024-09-11T01:51:14.860 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.457+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2621866220}) 2024-09-11T01:51:14.860 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.458+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.860 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.458+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2129254145}) 2024-09-11T01:51:14.860 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.458+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3374084024}) 2024-09-11T01:51:14.860 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.458+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.860 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.458+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2129254145}) 2024-09-11T01:51:14.860 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.458+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1590508781}) 2024-09-11T01:51:14.860 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.458+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.860 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.458+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3497422143}) 2024-09-11T01:51:14.860 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.458+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3984339810}) 2024-09-11T01:51:14.860 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.458+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.861 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.458+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3497422143}) 2024-09-11T01:51:14.861 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.458+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2994939865}) 2024-09-11T01:51:14.861 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.459+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.861 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.459+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=451402007}) 2024-09-11T01:51:14.861 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.459+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2968292829}) 2024-09-11T01:51:14.861 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.459+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.861 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.459+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=451402007}) 2024-09-11T01:51:14.861 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.459+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1959565639}) 2024-09-11T01:51:14.861 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.460+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.861 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.460+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=818105883}) 2024-09-11T01:51:14.861 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.460+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3711911133}) 2024-09-11T01:51:14.861 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.460+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.862 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.460+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=818105883}) 2024-09-11T01:51:14.862 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.460+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=567635416}) 2024-09-11T01:51:14.862 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.461+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.862 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.461+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3524361131}) 2024-09-11T01:51:14.862 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.461+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2320274169}) 2024-09-11T01:51:14.862 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.461+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.862 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.461+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3524361131}) 2024-09-11T01:51:14.862 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.461+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3649562062}) 2024-09-11T01:51:14.862 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.461+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.862 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.461+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3694118730}) 2024-09-11T01:51:14.862 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.461+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=920931833}) 2024-09-11T01:51:14.862 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.461+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.863 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.461+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3694118730}) 2024-09-11T01:51:14.863 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.461+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1224401151}) 2024-09-11T01:51:14.863 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.462+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.863 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.462+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=958499578}) 2024-09-11T01:51:14.863 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.462+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=922427279}) 2024-09-11T01:51:14.863 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.462+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.863 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.462+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=958499578}) 2024-09-11T01:51:14.863 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.462+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=701195490}) 2024-09-11T01:51:14.863 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.463+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.863 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.463+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3475010384}) 2024-09-11T01:51:14.863 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.463+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1900156125}) 2024-09-11T01:51:14.863 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.463+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.864 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.463+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3475010384}) 2024-09-11T01:51:14.864 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.463+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3083667211}) 2024-09-11T01:51:14.864 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.463+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.864 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.463+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3461638458}) 2024-09-11T01:51:14.864 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.463+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2075839091}) 2024-09-11T01:51:14.864 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.463+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.864 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.463+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3461638458}) 2024-09-11T01:51:14.864 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.463+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=905700683}) 2024-09-11T01:51:14.864 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.464+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.864 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.464+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=526059964}) 2024-09-11T01:51:14.864 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.464+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=408649722}) 2024-09-11T01:51:14.864 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.464+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.865 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.464+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=526059964}) 2024-09-11T01:51:14.865 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.464+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2883128140}) 2024-09-11T01:51:14.865 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.465+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.865 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.465+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=74,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=3} crc {logm=3297444967,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1474896599,mon_config_key=2129402939}) 2024-09-11T01:51:14.865 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.465+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=77,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=2} crc {logm=683123527,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432717,mgrstat=651382040,mon_config_key=243754935}) 2024-09-11T01:51:14.865 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.465+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.865 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.465+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=74,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=3} crc {logm=3297444967,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1474896599,mon_config_key=2129402939}) 2024-09-11T01:51:14.865 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.465+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=78,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10} crc {logm=2007203448,mdsmap=3760429986,mgr=3108067518,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1474896599}) 2024-09-11T01:51:14.865 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.466+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.865 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.466+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=14,monmap=1,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=75} crc {mon_config_key=997254776,monmap=1228700967,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=3803224348,osd_snap=552186101}) 2024-09-11T01:51:14.865 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.466+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=74} crc {mon_config_key=997254776,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2607543842,osd_pg_creating=3803224348,osd_snap=2789694319}) 2024-09-11T01:51:14.865 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.466+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.866 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.466+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=14,monmap=1,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=75} crc {mon_config_key=997254776,monmap=1228700967,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=3803224348,osd_snap=552186101}) 2024-09-11T01:51:14.866 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.466+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=14,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=74} crc {mon_config_key=997254776,monmap=2933275274,nvmeofgw=1596038039,osd_metadata=2942852742,osd_pg_creating=3803224348,osd_snap=2920672172}) 2024-09-11T01:51:14.866 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.466+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.866 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.466+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=76,osdmap=24} crc {osd_snap=953117430,osdmap=2732152074}) 2024-09-11T01:51:14.866 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.466+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=73,osdmap=27} crc {osd_snap=3187016202,osdmap=2151920081}) 2024-09-11T01:51:14.866 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.466+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.866 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.466+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=76,osdmap=24} crc {osd_snap=953117430,osdmap=2732152074}) 2024-09-11T01:51:14.866 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.466+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=75,osdmap=25} crc {osd_snap=783152664,osdmap=2790936813}) 2024-09-11T01:51:14.866 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.467+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.866 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.467+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4238572990}) 2024-09-11T01:51:14.866 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.467+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1876799045}) 2024-09-11T01:51:14.866 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.467+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.467+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4238572990}) 2024-09-11T01:51:14.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.467+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1758312761}) 2024-09-11T01:51:14.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.468+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.468+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3900110406}) 2024-09-11T01:51:14.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.468+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1536646331}) 2024-09-11T01:51:14.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.468+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.468+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3900110406}) 2024-09-11T01:51:14.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.468+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1424575883}) 2024-09-11T01:51:14.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.469+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.469+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1405634455}) 2024-09-11T01:51:14.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.469+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3787487382}) 2024-09-11T01:51:14.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.469+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.868 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.469+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1405634455}) 2024-09-11T01:51:14.868 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.469+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2608035378}) 2024-09-11T01:51:14.868 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.474+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.868 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.474+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=24} crc {osdmap=1155247167}) 2024-09-11T01:51:14.868 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.474+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=22} crc {osdmap=1358332490}) 2024-09-11T01:51:14.868 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.474+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:51:14.868 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.474+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=24} crc {osdmap=1155247167}) 2024-09-11T01:51:14.868 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:51:14.474+0000 7fe96fe05640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=24} crc {osdmap=1155247166}) 2024-09-11T01:51:15.573 INFO:tasks.rados.rados.0.smithi071.stdout:3277: done (0 left) 2024-09-11T01:51:15.573 INFO:tasks.rados.rados.0.smithi071.stdout:3278: delete oid 10 current snap is 366 2024-09-11T01:51:15.576 INFO:tasks.rados.rados.0.smithi071.stdout:3278: done (0 left) 2024-09-11T01:51:15.576 INFO:tasks.rados.rados.0.smithi071.stdout:3279: write oid 38 current snap is 366 2024-09-11T01:51:15.576 INFO:tasks.rados.rados.0.smithi071.stdout:3279: seq_num 1150 ranges {528050=505653,1612057=524731,2614324=567403} 2024-09-11T01:51:15.582 INFO:tasks.rados.rados.0.smithi071.stdout:3279: writing smithi07134777-38 from 528050 to 1033703 tid 1 2024-09-11T01:51:15.588 INFO:tasks.rados.rados.0.smithi071.stdout:3279: writing smithi07134777-38 from 1612057 to 2136788 tid 2 2024-09-11T01:51:15.595 INFO:tasks.rados.rados.0.smithi071.stdout:3279: writing smithi07134777-38 from 2614324 to 3181727 tid 3 2024-09-11T01:51:15.595 INFO:tasks.rados.rados.0.smithi071.stdout:3280: rollback oid 25 current snap is 366 2024-09-11T01:51:15.595 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 25 to 338 2024-09-11T01:51:15.596 INFO:tasks.rados.rados.0.smithi071.stdout:3279: finishing write tid 1 to smithi07134777-38 2024-09-11T01:51:15.596 INFO:tasks.rados.rados.0.smithi071.stdout:3279: finishing write tid 2 to smithi07134777-38 2024-09-11T01:51:15.596 INFO:tasks.rados.rados.0.smithi071.stdout:3281: copy_from oid 20 from oid 23 current snap is 366 2024-09-11T01:51:15.596 INFO:tasks.rados.rados.0.smithi071.stdout:3282: write (excl) oid 6 current snap is 366 2024-09-11T01:51:15.596 INFO:tasks.rados.rados.0.smithi071.stdout:3282: seq_num 1151 ranges {544857=580902,1670539=668389,3041747=303110} 2024-09-11T01:51:15.601 INFO:tasks.rados.rados.0.smithi071.stdout:3282: writing smithi07134777-6 from 544857 to 1125759 tid 1 2024-09-11T01:51:15.608 INFO:tasks.rados.rados.0.smithi071.stdout:3282: writing smithi07134777-6 from 1670539 to 2338928 tid 2 2024-09-11T01:51:15.613 INFO:tasks.rados.rados.0.smithi071.stdout:3282: writing smithi07134777-6 from 3041747 to 3344857 tid 3 2024-09-11T01:51:15.614 INFO:tasks.rados.rados.0.smithi071.stdout:3283: delete oid 48 current snap is 366 2024-09-11T01:51:15.614 INFO:tasks.rados.rados.0.smithi071.stdout:3281: finishing copy_from to smithi07134777-20 2024-09-11T01:51:15.614 INFO:tasks.rados.rados.0.smithi071.stdout:3281: got expected ENOENT (src dne) 2024-09-11T01:51:15.614 INFO:tasks.rados.rados.0.smithi071.stdout:3281: finishing copy_from racing read to smithi07134777-20 2024-09-11T01:51:15.614 INFO:tasks.rados.rados.0.smithi071.stdout:3280: finishing rollback tid 1 to smithi07134777-25 2024-09-11T01:51:15.614 INFO:tasks.rados.rados.0.smithi071.stdout:3280: finishing rollback tid 2 to smithi07134777-25 2024-09-11T01:51:15.614 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 1045 (ObjNum 1023 snap 328 seq_num 1023) dirty exists 2024-09-11T01:51:15.614 INFO:tasks.rados.rados.0.smithi071.stdout:3279: finishing write tid 3 to smithi07134777-38 2024-09-11T01:51:15.614 INFO:tasks.rados.rados.0.smithi071.stdout:3279: finishing write tid 4 to smithi07134777-38 2024-09-11T01:51:15.614 INFO:tasks.rados.rados.0.smithi071.stdout:3279: finishing write tid 5 to smithi07134777-38 2024-09-11T01:51:15.614 INFO:tasks.rados.rados.0.smithi071.stdout:3279: oid 38 updating version 0 to 2407 2024-09-11T01:51:15.614 INFO:tasks.rados.rados.0.smithi071.stdout:3279: oid 38 version 2407 is already newer than 2404 2024-09-11T01:51:15.614 INFO:tasks.rados.rados.0.smithi071.stdout:3279: oid 38 version 2407 is already newer than 2406 2024-09-11T01:51:15.615 INFO:tasks.rados.rados.0.smithi071.stdout:3279: oid 38 version 2407 is already newer than 2405 2024-09-11T01:51:15.615 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 2407 (ObjNum 1150 snap 366 seq_num 1150) dirty exists 2024-09-11T01:51:15.615 INFO:tasks.rados.rados.0.smithi071.stdout:3279: left oid 38 (ObjNum 1150 snap 366 seq_num 1150) 2024-09-11T01:51:15.615 INFO:tasks.rados.rados.0.smithi071.stdout:3282: finishing write tid 1 to smithi07134777-6 2024-09-11T01:51:15.616 INFO:tasks.rados.rados.0.smithi071.stdout:3279: done (4 left) 2024-09-11T01:51:15.616 INFO:tasks.rados.rados.0.smithi071.stdout:3280: done (3 left) 2024-09-11T01:51:15.616 INFO:tasks.rados.rados.0.smithi071.stdout:3281: done (2 left) 2024-09-11T01:51:15.616 INFO:tasks.rados.rados.0.smithi071.stdout:3283: done (1 left) 2024-09-11T01:51:15.616 INFO:tasks.rados.rados.0.smithi071.stdout:3284: delete oid 29 current snap is 366 2024-09-11T01:51:15.618 INFO:tasks.rados.rados.0.smithi071.stdout:3284: done (1 left) 2024-09-11T01:51:15.618 INFO:tasks.rados.rados.0.smithi071.stdout:3285: write oid 1 current snap is 366 2024-09-11T01:51:15.619 INFO:tasks.rados.rados.0.smithi071.stdout:3285: seq_num 1154 ranges {595278=406649,1729632=532069,3007663=387615} 2024-09-11T01:51:15.624 INFO:tasks.rados.rados.0.smithi071.stdout:3285: writing smithi07134777-1 from 595278 to 1001927 tid 1 2024-09-11T01:51:15.631 INFO:tasks.rados.rados.0.smithi071.stdout:3285: writing smithi07134777-1 from 1729632 to 2261701 tid 2 2024-09-11T01:51:15.637 INFO:tasks.rados.rados.0.smithi071.stdout:3285: writing smithi07134777-1 from 3007663 to 3395278 tid 3 2024-09-11T01:51:15.637 INFO:tasks.rados.rados.0.smithi071.stdout:3286: snap_remove snap 351 2024-09-11T01:51:15.781 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:51:15.781 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:51:15.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.892+0000 7eff5080f640 10 monclient: get_monmap_and_config 2024-09-11T01:51:15.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.892+0000 7eff5080f640 10 monclient: build_initial_monmap 2024-09-11T01:51:15.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.892+0000 7eff5080f640 10 monclient: monmap: 2024-09-11T01:51:15.895 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:15.895 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:15.895 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:51:15.893426+0000 2024-09-11T01:51:15.895 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:51:15.893426+0000 2024-09-11T01:51:15.895 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:15.895 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:15.895 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:51:15.895 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:51:15.896 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:51:15.896 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:51:15.896 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:51:15.896 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:15.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.893+0000 7eff5080f640 1 Processor -- start 2024-09-11T01:51:15.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.893+0000 7eff5080f640 1 -- start start 2024-09-11T01:51:15.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.893+0000 7eff5080f640 10 monclient: init 2024-09-11T01:51:15.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.893+0000 7eff5080f640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:15.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff5080f640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-11T01:51:15.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff5080f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7eff4c0a2e90 addr v1:172.21.15.71:6789/0 2024-09-11T01:51:15.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff5080f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff4c0a3270 0x7eff4c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:15.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff5080f640 10 monclient(hunting): picked mon.noname-b con 0x7eff4c0a3270 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:15.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff5080f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7eff4c13d490 0x7eff4c13f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:15.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff5080f640 10 monclient(hunting): picked mon.noname-a con 0x7eff4c13d490 addr v2:172.21.15.12:3300/0 2024-09-11T01:51:15.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff5080f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff4c0a46c0 con 0x7eff4c0a2e90 2024-09-11T01:51:15.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff5080f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:15.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff5080f640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7eff4c0a4900 con 0x7eff4c13d490 2024-09-11T01:51:15.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff5080f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:15.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff5080f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7eff4c06ee50 con 0x7eff4c0a3270 2024-09-11T01:51:15.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff5080f640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:15.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff5080f640 10 monclient(hunting): authenticate will time out at 2082.336148s 2024-09-11T01:51:15.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff49d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff4c0a3270 0x7eff4c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:15.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff4ad9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7eff4c13d490 0x7eff4c13f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:15.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff49d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff4c0a3270 0x7eff4c0a3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:48016/0 (socket says 172.21.15.12:48016) 2024-09-11T01:51:15.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff49d9b640 1 -- 172.21.15.12:0/1527427934 learned_addr learned my addr 172.21.15.12:0/1527427934 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:15.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff4ad9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7eff4c13d490 0x7eff4c13f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60300/0 (socket says 172.21.15.12:60300) 2024-09-11T01:51:15.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff49d9b640 10 monclient(hunting): get_auth_request con 0x7eff4c0a3270 auth_method 0 2024-09-11T01:51:15.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:15.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff49d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:15.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:15.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff4ad9d640 10 monclient(hunting): get_auth_request con 0x7eff4c13d490 auth_method 0 2024-09-11T01:51:15.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff4ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:15.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff4ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:15.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff4ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:15.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff49d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:15.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff49d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:15.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff49d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:15.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff4ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:15.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff4ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:15.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff4ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:15.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff4ad9d640 10 monclient(hunting): handle_auth_done global_id 214416 payload 274 2024-09-11T01:51:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff4ad9d640 1 -- 172.21.15.12:0/1527427934 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff4c0a3270 msgr2=0x7eff4c0a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff4ad9d640 1 --2- 172.21.15.12:0/1527427934 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff4c0a3270 0x7eff4c0a3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff4ad9d640 1 -- 172.21.15.12:0/1527427934 >> v1:172.21.15.71:6789/0 conn(0x7eff4c0a2e90 legacy=0x7eff4c06dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff4ad9d640 10 monclient: _finish_hunting 0 2024-09-11T01:51:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff4ad9d640 1 monclient: found mon.noname-a 2024-09-11T01:51:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff4ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff4ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:51:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff4ad9d640 1 -- 172.21.15.12:0/1527427934 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff4c06eb70 con 0x7eff4c13d490 2024-09-11T01:51:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.894+0000 7eff49d9b640 1 --2- 172.21.15.12:0/1527427934 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff4c0a3270 0x7eff4c0a3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:51:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.897+0000 7eff4ad9d640 1 -- 172.21.15.12:0/1527427934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff4c13d490 msgr2=0x7eff4c13f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.897+0000 7eff4ad9d640 1 --2- 172.21.15.12:0/1527427934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff4c13d490 0x7eff4c13f880 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7eff4c0a4a80 tx=0x7eff4402a460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f204cad26cac8c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.897+0000 7eff4959a640 1 -- 172.21.15.12:0/1527427934 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff44007950 con 0x7eff4c13d490 2024-09-11T01:51:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.897+0000 7eff4959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:15.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.897+0000 7eff4959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:51:15.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.897+0000 7eff4959a640 10 monclient: dump: 2024-09-11T01:51:15.912 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:15.912 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:15.912 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:15.912 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:15.912 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:15.912 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:15.912 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:15.912 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:15.912 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:15.912 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:15.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.897+0000 7eff4959a640 10 monclient: _finish_auth 0 2024-09-11T01:51:15.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.897+0000 7eff4959a640 10 monclient: _check_auth_tickets 2024-09-11T01:51:15.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.897+0000 7eff4959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:15.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.897+0000 7eff5080f640 5 monclient: authenticate success, global_id 214416 2024-09-11T01:51:15.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.897+0000 7eff4959a640 1 -- 172.21.15.12:0/1527427934 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff44007af0 con 0x7eff4c13d490 2024-09-11T01:51:15.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.897+0000 7eff5080f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:51:15.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.897+0000 7eff4959a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:15.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.897+0000 7eff5080f640 10 monclient: get_monmap_and_config success 2024-09-11T01:51:15.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.897+0000 7eff4959a640 1 -- 172.21.15.12:0/1527427934 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff44031400 con 0x7eff4c13d490 2024-09-11T01:51:15.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.897+0000 7eff4959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:15.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.897+0000 7eff5080f640 4 set_mon_vals no callback set 2024-09-11T01:51:15.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.897+0000 7eff4959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:51:15.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.897+0000 7eff4959a640 10 monclient: dump: 2024-09-11T01:51:15.914 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:15.914 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:15.914 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:15.914 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:15.914 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:15.914 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:15.915 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.898+0000 7eff5080f640 10 monclient: shutdown 2024-09-11T01:51:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.898+0000 7eff5080f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.898+0000 7eff5080f640 1 -- 172.21.15.12:0/1527427934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff4c13d490 msgr2=0x7eff4c13f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.898+0000 7eff5080f640 1 --2- 172.21.15.12:0/1527427934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff4c13d490 0x7eff4c13f880 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7eff4c0a4a80 tx=0x7eff4402a460 comp rx=0 tx=0).stop 2024-09-11T01:51:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.898+0000 7eff5080f640 1 -- 172.21.15.12:0/1527427934 shutdown_connections 2024-09-11T01:51:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.898+0000 7eff5080f640 1 --2- 172.21.15.12:0/1527427934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff4c13d490 0x7eff4c13f880 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.898+0000 7eff5080f640 1 --2- 172.21.15.12:0/1527427934 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff4c0a3270 0x7eff4c0a3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.898+0000 7eff5080f640 1 -- 172.21.15.12:0/1527427934 >> 172.21.15.12:0/1527427934 conn(0x7eff4c0a8490 msgr2=0x7eff4c0a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:15.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.898+0000 7eff5080f640 1 -- 172.21.15.12:0/1527427934 shutdown_connections 2024-09-11T01:51:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.898+0000 7eff5080f640 1 -- 172.21.15.12:0/1527427934 wait complete. 2024-09-11T01:51:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.899+0000 7eff5080f640 10 monclient: build_initial_monmap 2024-09-11T01:51:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.899+0000 7eff5080f640 10 monclient: monmap: 2024-09-11T01:51:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:51:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:51:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:51:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:51:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:51:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:51:15.916 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:15.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.899+0000 7eff5080f640 1 Processor -- start 2024-09-11T01:51:15.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.899+0000 7eff5080f640 1 -- start start 2024-09-11T01:51:15.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.899+0000 7eff5080f640 10 monclient: init 2024-09-11T01:51:15.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.899+0000 7eff5080f640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:15.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.899+0000 7eff5080f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:51:15.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.899+0000 7eff5080f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff4c0a2e90 0x7eff4c13baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:15.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.899+0000 7eff5080f640 10 monclient(hunting): picked mon.noname-a con 0x7eff4c0a2e90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:51:15.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.899+0000 7eff5080f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff4c0a3270 0x7eff4c1399b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:15.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.899+0000 7eff5080f640 10 monclient(hunting): picked mon.noname-b con 0x7eff4c0a3270 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:51:15.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.899+0000 7eff5080f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff4c13d490 0x7eff4c138370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:15.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.899+0000 7eff5080f640 10 monclient(hunting): picked mon.noname-c con 0x7eff4c13d490 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:15.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.899+0000 7eff5080f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:15.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.899+0000 7eff5080f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7eff4c06fd20 con 0x7eff4c0a2e90 2024-09-11T01:51:15.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.899+0000 7eff5080f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:15.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.899+0000 7eff5080f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7eff4c06fba0 con 0x7eff4c0a3270 2024-09-11T01:51:15.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.899+0000 7eff5080f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:15.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.899+0000 7eff5080f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7eff4c13fdc0 con 0x7eff4c13d490 2024-09-11T01:51:15.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.899+0000 7eff5080f640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:15.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.899+0000 7eff5080f640 10 monclient(hunting): authenticate will time out at 2082.341991s 2024-09-11T01:51:15.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4a59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff4c0a2e90 0x7eff4c13baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:15.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4ad9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff4c13d490 0x7eff4c138370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:15.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4a59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff4c0a2e90 0x7eff4c13baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60308/0 (socket says 172.21.15.12:60308) 2024-09-11T01:51:15.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4ad9d640 10 monclient(hunting): get_auth_request con 0x7eff4c13d490 auth_method 0 2024-09-11T01:51:15.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4a59c640 1 -- 172.21.15.12:0/3577991257 learned_addr learned my addr 172.21.15.12:0/3577991257 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:15.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:15.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:15.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:15.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4a59c640 10 monclient(hunting): get_auth_request con 0x7eff4c0a2e90 auth_method 0 2024-09-11T01:51:15.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:15.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:15.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:15.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:15.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:15.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:15.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:15.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:15.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:15.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff49d9b640 1 --2- 172.21.15.12:0/3577991257 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff4c0a3270 0x7eff4c1399b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:15.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff49d9b640 10 monclient(hunting): get_auth_request con 0x7eff4c0a3270 auth_method 0 2024-09-11T01:51:15.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:15.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff49d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:15.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:15.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4ad9d640 10 monclient(hunting): handle_auth_done global_id 244130 payload 824 2024-09-11T01:51:15.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4ad9d640 1 -- 172.21.15.12:0/3577991257 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff4c0a3270 msgr2=0x7eff4c1399b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:15.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4ad9d640 1 --2- 172.21.15.12:0/3577991257 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff4c0a3270 0x7eff4c1399b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:15.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4ad9d640 1 -- 172.21.15.12:0/3577991257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff4c0a2e90 msgr2=0x7eff4c13baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:15.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4ad9d640 1 --2- 172.21.15.12:0/3577991257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff4c0a2e90 0x7eff4c13baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:15.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4ad9d640 10 monclient: _finish_hunting 0 2024-09-11T01:51:15.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4ad9d640 1 monclient: found mon.noname-c 2024-09-11T01:51:15.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:15.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:51:15.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4ad9d640 1 -- 172.21.15.12:0/3577991257 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff4c13af40 con 0x7eff4c13d490 2024-09-11T01:51:15.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.900+0000 7eff4a59c640 1 --2- 172.21.15.12:0/3577991257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff4c0a2e90 0x7eff4c13baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:51:15.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.902+0000 7eff4ad9d640 1 -- 172.21.15.12:0/3577991257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff4c13d490 msgr2=0x7eff4c138370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:15.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.902+0000 7eff4ad9d640 1 --2- 172.21.15.12:0/3577991257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff4c13d490 0x7eff4c138370 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7eff44031000 tx=0x7eff44004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:15.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.902+0000 7eff3affd640 1 -- 172.21.15.12:0/3577991257 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff44040070 con 0x7eff4c13d490 2024-09-11T01:51:15.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.902+0000 7eff3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:15.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.902+0000 7eff3affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:51:15.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.902+0000 7eff3affd640 10 monclient: dump: 2024-09-11T01:51:15.921 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:15.921 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:15.921 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:15.921 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:15.922 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:15.922 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:15.922 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:15.922 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:15.922 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:15.922 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:15.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.902+0000 7eff3affd640 10 monclient: _finish_auth 0 2024-09-11T01:51:15.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.902+0000 7eff3affd640 10 monclient: _check_auth_tickets 2024-09-11T01:51:15.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.902+0000 7eff3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:15.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.902+0000 7eff3affd640 1 -- 172.21.15.12:0/3577991257 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff44045460 con 0x7eff4c13d490 2024-09-11T01:51:15.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.902+0000 7eff5080f640 5 monclient: authenticate success, global_id 244130 2024-09-11T01:51:15.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.902+0000 7eff5080f640 10 monclient: _renew_subs 2024-09-11T01:51:15.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.902+0000 7eff5080f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.902+0000 7eff5080f640 1 -- 172.21.15.12:0/3577991257 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff4c139010 con 0x7eff4c13d490 2024-09-11T01:51:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.902+0000 7eff3affd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.902+0000 7eff3affd640 1 -- 172.21.15.12:0/3577991257 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff44007950 con 0x7eff4c13d490 2024-09-11T01:51:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.902+0000 7eff3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.903+0000 7eff3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.903+0000 7eff3affd640 10 monclient: dump: 2024-09-11T01:51:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:15.923 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:15.924 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.902+0000 7eff48d99640 4 set_mon_vals no callback set 2024-09-11T01:51:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.903+0000 7eff5080f640 10 monclient: _renew_subs 2024-09-11T01:51:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.903+0000 7eff5080f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.903+0000 7eff5080f640 1 -- 172.21.15.12:0/3577991257 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7eff4c09f680 con 0x7eff4c13d490 2024-09-11T01:51:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.903+0000 7eff3affd640 1 -- 172.21.15.12:0/3577991257 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7eff4404d030 con 0x7eff4c13d490 2024-09-11T01:51:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.903+0000 7eff5080f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.903+0000 7eff5080f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.903+0000 7eff5080f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.904+0000 7eff5080f640 1 -- 172.21.15.12:0/3577991257 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff4c13f370 con 0x7eff4c13d490 2024-09-11T01:51:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.904+0000 7eff3affd640 1 --2- 172.21.15.12:0/3577991257 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7eff2c03e260 0x7eff2c040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:15.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.904+0000 7eff3affd640 1 -- 172.21.15.12:0/3577991257 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(734..734 src has 507..734) ==== 4783+0+0 (secure 0 0 0) 0x7eff440745f0 con 0x7eff4c13d490 2024-09-11T01:51:15.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.904+0000 7eff4a59c640 1 --2- 172.21.15.12:0/3577991257 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7eff2c03e260 0x7eff2c040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:15.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.904+0000 7eff4a59c640 10 monclient: get_auth_request con 0x7eff2c03e260 auth_method 0 2024-09-11T01:51:15.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.907+0000 7eff4a59c640 1 --2- 172.21.15.12:0/3577991257 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7eff2c03e260 0x7eff2c040720 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0043e0 tx=0x7eff3c007ef0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:15.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.908+0000 7eff3affd640 1 -- 172.21.15.12:0/3577991257 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7eff4c13f370 con 0x7eff4c13d490 2024-09-11T01:51:15.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.908+0000 7eff3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:15.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:15.908+0000 7eff3affd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:51:16.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.165+0000 7eff5080f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:51:16.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.165+0000 7eff5080f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:51:16.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.165+0000 7eff5080f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:16.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.165+0000 7eff5080f640 1 -- 172.21.15.12:0/3577991257 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7eff4c0a3680 con 0x7eff4c13d490 2024-09-11T01:51:16.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.165+0000 7eff3affd640 1 -- 172.21.15.12:0/3577991257 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7eff44072ff0 con 0x7eff4c13d490 2024-09-11T01:51:16.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.165+0000 7eff3affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:51:16.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.165+0000 7eff3affd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:51:16.168 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":208,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:51:16.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.168+0000 7eff5080f640 1 -- 172.21.15.12:0/3577991257 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7eff2c03e260 msgr2=0x7eff2c040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:16.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.168+0000 7eff5080f640 1 --2- 172.21.15.12:0/3577991257 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7eff2c03e260 0x7eff2c040720 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0043e0 tx=0x7eff3c007ef0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.168+0000 7eff5080f640 10 monclient: shutdown 2024-09-11T01:51:16.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.168+0000 7eff5080f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:16.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.168+0000 7eff5080f640 1 -- 172.21.15.12:0/3577991257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff4c13d490 msgr2=0x7eff4c138370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:16.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.168+0000 7eff5080f640 1 --2- 172.21.15.12:0/3577991257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff4c13d490 0x7eff4c138370 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7eff44031000 tx=0x7eff44004a40 comp rx=0 tx=0).stop 2024-09-11T01:51:16.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.168+0000 7eff5080f640 1 -- 172.21.15.12:0/3577991257 shutdown_connections 2024-09-11T01:51:16.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.168+0000 7eff5080f640 1 --2- 172.21.15.12:0/3577991257 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7eff4c13d490 0x7eff4c138370 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.168+0000 7eff5080f640 1 --2- 172.21.15.12:0/3577991257 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff4c0a3270 0x7eff4c1399b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.168+0000 7eff5080f640 1 --2- 172.21.15.12:0/3577991257 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7eff2c03e260 0x7eff2c040720 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.168+0000 7eff5080f640 1 --2- 172.21.15.12:0/3577991257 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff4c0a2e90 0x7eff4c13baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.168+0000 7eff5080f640 1 -- 172.21.15.12:0/3577991257 >> 172.21.15.12:0/3577991257 conn(0x7eff4c0a8490 msgr2=0x7eff4c072090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:16.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.168+0000 7eff5080f640 1 -- 172.21.15.12:0/3577991257 shutdown_connections 2024-09-11T01:51:16.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.168+0000 7eff5080f640 1 -- 172.21.15.12:0/3577991257 wait complete. 2024-09-11T01:51:16.180 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:51:16.180 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:51:16.180 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:51:16.180 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:51:16.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.283+0000 7f0692a23640 10 monclient: get_monmap_and_config 2024-09-11T01:51:16.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.283+0000 7f0692a23640 10 monclient: build_initial_monmap 2024-09-11T01:51:16.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.283+0000 7f0692a23640 10 monclient: monmap: 2024-09-11T01:51:16.286 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:16.286 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:16.286 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:51:16.284611+0000 2024-09-11T01:51:16.286 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:51:16.284611+0000 2024-09-11T01:51:16.286 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:16.286 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.286 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:51:16.286 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:51:16.286 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:51:16.286 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:51:16.286 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:51:16.286 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.284+0000 7f0692a23640 1 Processor -- start 2024-09-11T01:51:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.284+0000 7f0692a23640 1 -- start start 2024-09-11T01:51:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.284+0000 7f0692a23640 10 monclient: init 2024-09-11T01:51:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f0692a23640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f0692a23640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-11T01:51:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f0692a23640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f068c0a03a0 0x7f068c09f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f0692a23640 10 monclient(hunting): picked mon.noname-a con 0x7f068c0a03a0 addr v2:172.21.15.12:3300/0 2024-09-11T01:51:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f0692a23640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f068c0a2530 0x7f068c0a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f0692a23640 10 monclient(hunting): picked mon.noname-b con 0x7f068c0a2530 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f0692a23640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f068c0a11d0 0x7f068c0a1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f0692a23640 10 monclient(hunting): picked mon.noname-c con 0x7f068c0a11d0 addr v2:172.21.15.71:3300/0 2024-09-11T01:51:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f0692a23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f0692a23640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f068c0a29b0 con 0x7f068c0a03a0 2024-09-11T01:51:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f0692a23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f0692a23640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f068c0a0780 con 0x7f068c0a11d0 2024-09-11T01:51:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f0692a23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f0692a23640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f068c09e510 con 0x7f068c0a2530 2024-09-11T01:51:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f0692a23640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f0692a23640 10 monclient(hunting): authenticate will time out at 2082.727167s 2024-09-11T01:51:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f068c0a2530 0x7f068c0a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f068c0a03a0 0x7f068c09f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f068c0a03a0 0x7f068c09f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60312/0 (socket says 172.21.15.12:60312) 2024-09-11T01:51:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f068c0a2530 0x7f068c0a0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:48034/0 (socket says 172.21.15.12:48034) 2024-09-11T01:51:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068bfff640 1 -- 172.21.15.12:0/83592309 learned_addr learned my addr 172.21.15.12:0/83592309 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068bfff640 10 monclient(hunting): get_auth_request con 0x7f068c0a03a0 auth_method 0 2024-09-11T01:51:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068b7fe640 10 monclient(hunting): get_auth_request con 0x7f068c0a2530 auth_method 0 2024-09-11T01:51:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:16.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f0690fc0640 1 --2- 172.21.15.12:0/83592309 >> v2:172.21.15.71:3300/0 conn(0x7f068c0a11d0 0x7f068c0a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f0690fc0640 10 monclient(hunting): get_auth_request con 0x7f068c0a11d0 auth_method 0 2024-09-11T01:51:16.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f0690fc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:16.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f0690fc0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:16.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f0690fc0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:16.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068b7fe640 10 monclient(hunting): handle_auth_done global_id 244133 payload 274 2024-09-11T01:51:16.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068b7fe640 1 -- 172.21.15.12:0/83592309 >> v2:172.21.15.71:3300/0 conn(0x7f068c0a11d0 msgr2=0x7f068c0a1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:16.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068b7fe640 1 --2- 172.21.15.12:0/83592309 >> v2:172.21.15.71:3300/0 conn(0x7f068c0a11d0 0x7f068c0a1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068b7fe640 1 -- 172.21.15.12:0/83592309 >> v2:172.21.15.12:3300/0 conn(0x7f068c0a03a0 msgr2=0x7f068c09f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:16.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068b7fe640 1 --2- 172.21.15.12:0/83592309 >> v2:172.21.15.12:3300/0 conn(0x7f068c0a03a0 0x7f068c09f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:51:16.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068b7fe640 1 monclient: found mon.noname-b 2024-09-11T01:51:16.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:16.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:51:16.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068b7fe640 1 -- 172.21.15.12:0/83592309 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f068c09edb0 con 0x7f068c0a2530 2024-09-11T01:51:16.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.285+0000 7f068bfff640 1 --2- 172.21.15.12:0/83592309 >> v2:172.21.15.12:3300/0 conn(0x7f068c0a03a0 0x7f068c09f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:51:16.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.288+0000 7f068b7fe640 1 -- 172.21.15.12:0/83592309 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f068c0a2530 msgr2=0x7f068c0a0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:16.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.288+0000 7f068b7fe640 1 --2- 172.21.15.12:0/83592309 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f068c0a2530 0x7f068c0a0c90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f068c09e880 tx=0x7f068002a010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eda387155f68d29c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:16.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.288+0000 7f068affd640 1 -- 172.21.15.12:0/83592309 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0680037070 con 0x7f068c0a2530 2024-09-11T01:51:16.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.288+0000 7f068affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:16.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.288+0000 7f068affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:51:16.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.288+0000 7f068affd640 10 monclient: dump: 2024-09-11T01:51:16.295 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:16.295 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:16.295 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.295 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.295 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:16.296 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.296 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:16.296 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:16.296 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:16.296 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.288+0000 7f068affd640 10 monclient: _finish_auth 0 2024-09-11T01:51:16.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.288+0000 7f068affd640 10 monclient: _check_auth_tickets 2024-09-11T01:51:16.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.288+0000 7f068affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:16.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.288+0000 7f068affd640 1 -- 172.21.15.12:0/83592309 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0680032440 con 0x7f068c0a2530 2024-09-11T01:51:16.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.288+0000 7f0692a23640 5 monclient: authenticate success, global_id 244133 2024-09-11T01:51:16.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.288+0000 7f0692a23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:51:16.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.288+0000 7f068affd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:16.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.288+0000 7f0692a23640 10 monclient: get_monmap_and_config success 2024-09-11T01:51:16.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.288+0000 7f068affd640 1 -- 172.21.15.12:0/83592309 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0680032810 con 0x7f068c0a2530 2024-09-11T01:51:16.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.288+0000 7f0692a23640 4 set_mon_vals no callback set 2024-09-11T01:51:16.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.288+0000 7f068affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:16.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.288+0000 7f068affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:16.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.288+0000 7f068affd640 10 monclient: dump: 2024-09-11T01:51:16.297 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:16.297 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:16.297 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.297 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.297 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:16.297 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.297 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:16.297 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:16.297 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:16.297 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.289+0000 7f0692a23640 10 monclient: shutdown 2024-09-11T01:51:16.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.289+0000 7f0692a23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:16.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.289+0000 7f0692a23640 1 -- 172.21.15.12:0/83592309 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f068c0a2530 msgr2=0x7f068c0a0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:16.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.289+0000 7f0692a23640 1 --2- 172.21.15.12:0/83592309 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f068c0a2530 0x7f068c0a0c90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f068c09e880 tx=0x7f068002a010 comp rx=0 tx=0).stop 2024-09-11T01:51:16.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.289+0000 7f0692a23640 1 -- 172.21.15.12:0/83592309 shutdown_connections 2024-09-11T01:51:16.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.289+0000 7f0692a23640 1 --2- 172.21.15.12:0/83592309 >> v2:172.21.15.71:3300/0 conn(0x7f068c0a11d0 0x7f068c0a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.289+0000 7f0692a23640 1 --2- 172.21.15.12:0/83592309 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f068c0a2530 0x7f068c0a0c90 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.289+0000 7f0692a23640 1 --2- 172.21.15.12:0/83592309 >> v2:172.21.15.12:3300/0 conn(0x7f068c0a03a0 0x7f068c09f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.289+0000 7f0692a23640 1 -- 172.21.15.12:0/83592309 >> 172.21.15.12:0/83592309 conn(0x7f068c0a2f70 msgr2=0x7f068c0a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:16.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.289+0000 7f0692a23640 1 -- 172.21.15.12:0/83592309 shutdown_connections 2024-09-11T01:51:16.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.289+0000 7f0692a23640 1 -- 172.21.15.12:0/83592309 wait complete. 2024-09-11T01:51:16.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.289+0000 7f0692a23640 10 monclient: build_initial_monmap 2024-09-11T01:51:16.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.289+0000 7f0692a23640 10 monclient: monmap: 2024-09-11T01:51:16.299 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:16.299 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:51:16.299 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:51:16.299 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:51:16.299 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:16.299 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.299 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:51:16.299 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:51:16.299 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:51:16.299 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.290+0000 7f0692a23640 1 Processor -- start 2024-09-11T01:51:16.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.290+0000 7f0692a23640 1 -- start start 2024-09-11T01:51:16.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.290+0000 7f0692a23640 10 monclient: init 2024-09-11T01:51:16.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.290+0000 7f0692a23640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:16.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.290+0000 7f0692a23640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:51:16.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.290+0000 7f0692a23640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f068c0a03a0 0x7f068c1a8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.290+0000 7f0692a23640 10 monclient(hunting): picked mon.noname-b con 0x7f068c0a03a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:51:16.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.290+0000 7f0692a23640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068c0a11d0 0x7f068c1c8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.290+0000 7f0692a23640 10 monclient(hunting): picked mon.noname-a con 0x7f068c0a11d0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:51:16.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.290+0000 7f0692a23640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f068c0a2530 0x7f068c1c28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.290+0000 7f0692a23640 10 monclient(hunting): picked mon.noname-c con 0x7f068c0a2530 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:16.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.290+0000 7f0692a23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.290+0000 7f0692a23640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f068c09ce00 con 0x7f068c0a11d0 2024-09-11T01:51:16.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.290+0000 7f0692a23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.290+0000 7f0692a23640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f068c09cc80 con 0x7f068c0a03a0 2024-09-11T01:51:16.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.290+0000 7f0692a23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.290+0000 7f0692a23640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f068c09cf80 con 0x7f068c0a2530 2024-09-11T01:51:16.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.290+0000 7f0692a23640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:16.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.290+0000 7f0692a23640 10 monclient(hunting): authenticate will time out at 2082.732890s 2024-09-11T01:51:16.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.290+0000 7f0690fc0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f068c0a2530 0x7f068c1c28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.290+0000 7f068b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068c0a11d0 0x7f068c1c8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068c0a11d0 0x7f068c1c8dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60328/0 (socket says 172.21.15.12:60328) 2024-09-11T01:51:16.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f0690fc0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f068c0a2530 0x7f068c1c28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:48048/0 (socket says 172.21.15.12:48048) 2024-09-11T01:51:16.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068b7fe640 1 -- 172.21.15.12:0/136925996 learned_addr learned my addr 172.21.15.12:0/136925996 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:16.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068b7fe640 10 monclient(hunting): get_auth_request con 0x7f068c0a11d0 auth_method 0 2024-09-11T01:51:16.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:16.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:16.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:16.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f0690fc0640 10 monclient(hunting): get_auth_request con 0x7f068c0a2530 auth_method 0 2024-09-11T01:51:16.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f0690fc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:16.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f0690fc0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:16.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f0690fc0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:16.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068bfff640 1 --2- 172.21.15.12:0/136925996 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f068c0a03a0 0x7f068c1a8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068bfff640 10 monclient(hunting): get_auth_request con 0x7f068c0a03a0 auth_method 0 2024-09-11T01:51:16.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:16.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:16.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:16.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:16.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:16.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:16.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f0690fc0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:16.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f0690fc0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:16.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f0690fc0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:16.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:16.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:16.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:16.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068b7fe640 10 monclient(hunting): handle_auth_done global_id 214425 payload 824 2024-09-11T01:51:16.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068b7fe640 1 -- 172.21.15.12:0/136925996 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f068c0a2530 msgr2=0x7f068c1c28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:16.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068b7fe640 1 --2- 172.21.15.12:0/136925996 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f068c0a2530 0x7f068c1c28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068b7fe640 1 -- 172.21.15.12:0/136925996 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f068c0a03a0 msgr2=0x7f068c1a8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:16.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068b7fe640 1 --2- 172.21.15.12:0/136925996 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f068c0a03a0 0x7f068c1a8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:51:16.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068b7fe640 1 monclient: found mon.noname-a 2024-09-11T01:51:16.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:16.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:51:16.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f068b7fe640 1 -- 172.21.15.12:0/136925996 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f068c1a93c0 con 0x7f068c0a11d0 2024-09-11T01:51:16.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.291+0000 7f0690fc0640 1 --2- 172.21.15.12:0/136925996 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f068c0a2530 0x7f068c1c28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:51:16.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.293+0000 7f068b7fe640 1 -- 172.21.15.12:0/136925996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068c0a11d0 msgr2=0x7f068c1c8dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:16.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.293+0000 7f068b7fe640 1 --2- 172.21.15.12:0/136925996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068c0a11d0 0x7f068c1c8dd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f068002a600 tx=0x7f06800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:16.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.293+0000 7f0688ff9640 1 -- 172.21.15.12:0/136925996 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0680040020 con 0x7f068c0a11d0 2024-09-11T01:51:16.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.293+0000 7f0688ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:16.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.293+0000 7f0688ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:51:16.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.293+0000 7f0688ff9640 10 monclient: dump: 2024-09-11T01:51:16.309 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:16.309 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:16.309 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.309 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.309 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:16.310 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.310 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:16.310 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:16.310 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:16.310 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.293+0000 7f0688ff9640 10 monclient: _finish_auth 0 2024-09-11T01:51:16.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.293+0000 7f0688ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:51:16.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.293+0000 7f0688ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:16.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.293+0000 7f0688ff9640 1 -- 172.21.15.12:0/136925996 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0680038040 con 0x7f068c0a11d0 2024-09-11T01:51:16.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.293+0000 7f0692a23640 5 monclient: authenticate success, global_id 214425 2024-09-11T01:51:16.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.293+0000 7f0692a23640 10 monclient: _renew_subs 2024-09-11T01:51:16.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.293+0000 7f0692a23640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:51:16.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.293+0000 7f0692a23640 1 -- 172.21.15.12:0/136925996 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f068c1a78c0 con 0x7f068c0a11d0 2024-09-11T01:51:16.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.293+0000 7f0688ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:16.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.293+0000 7f0688ff9640 1 -- 172.21.15.12:0/136925996 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0680037070 con 0x7f068c0a11d0 2024-09-11T01:51:16.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.293+0000 7f0688ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:16.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.293+0000 7f0688ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:51:16.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.293+0000 7f0688ff9640 10 monclient: dump: 2024-09-11T01:51:16.311 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:16.311 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:16.311 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.311 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.311 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:16.311 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.311 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:16.312 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:16.312 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:16.312 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.293+0000 7f0692a23640 10 monclient: _renew_subs 2024-09-11T01:51:16.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.293+0000 7f0692a23640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:51:16.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.293+0000 7f0692a23640 1 -- 172.21.15.12:0/136925996 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f068c1a7ed0 con 0x7f068c0a11d0 2024-09-11T01:51:16.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.293+0000 7f068a7fc640 4 set_mon_vals no callback set 2024-09-11T01:51:16.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.294+0000 7f068b7fe640 1 --2- 172.21.15.12:0/136925996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068c0a11d0 0x7f068c1c8dd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f068002a600 tx=0x7f06800046c0 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7f06800443b0 mgrmap(e 5) 2024-09-11T01:51:16.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.294+0000 7f0688ff9640 1 -- 172.21.15.12:0/136925996 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f06800443b0 con 0x7f068c0a11d0 2024-09-11T01:51:16.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.294+0000 7f0692a23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:16.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.294+0000 7f0692a23640 1 --2- 172.21.15.12:0/136925996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068c1c5b70 0x7f068c02d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.294+0000 7f0692a23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.294+0000 7f0692a23640 1 -- 172.21.15.12:0/136925996 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f068c1aa350 con 0x7f068c1c5b70 2024-09-11T01:51:16.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.294+0000 7f068bfff640 1 --2- 172.21.15.12:0/136925996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068c1c5b70 0x7f068c02d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.294+0000 7f068bfff640 10 monclient: get_auth_request con 0x7f068c1c5b70 auth_method 0 2024-09-11T01:51:16.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.294+0000 7f068bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:16.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.294+0000 7f068bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:16.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.294+0000 7f068bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:16.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.294+0000 7f068bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:16.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.294+0000 7f068bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:16.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.294+0000 7f068bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:16.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.295+0000 7f0688ff9640 1 --2- 172.21.15.12:0/136925996 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f067803e240 0x7f0678040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.295+0000 7f068bfff640 10 monclient(hunting): handle_auth_done global_id 214428 payload 824 2024-09-11T01:51:16.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.295+0000 7f068bfff640 1 -- 172.21.15.12:0/136925996 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f068c09f1d0 con 0x7f068c1c5b70 2024-09-11T01:51:16.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.295+0000 7f0690fc0640 1 --2- 172.21.15.12:0/136925996 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f067803e240 0x7f0678040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.295+0000 7f0690fc0640 10 monclient: get_auth_request con 0x7f067803e240 auth_method 0 2024-09-11T01:51:16.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.297+0000 7f068bfff640 1 -- 172.21.15.12:0/136925996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068c1c5b70 msgr2=0x7f068c02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:16.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.297+0000 7f068bfff640 1 --2- 172.21.15.12:0/136925996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068c1c5b70 0x7f068c02d440 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f067c0038b0 tx=0x7f067c003d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:16.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.297+0000 7f0688ff9640 1 -- 172.21.15.12:0/136925996 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067c01c020 con 0x7f068c1c5b70 2024-09-11T01:51:16.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.297+0000 7f0688ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:16.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.297+0000 7f0688ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:51:16.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.297+0000 7f0688ff9640 10 monclient: dump: 2024-09-11T01:51:16.314 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:16.314 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:16.314 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.314 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.314 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:16.315 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.315 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:16.315 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:16.315 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:16.315 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.297+0000 7f0688ff9640 1 -- 172.21.15.12:0/136925996 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f067c018630 con 0x7f068c1c5b70 2024-09-11T01:51:16.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.297+0000 7f0688ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:16.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.297+0000 7f0688ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:51:16.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.297+0000 7f0688ff9640 1 -- 172.21.15.12:0/136925996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068c1c5b70 msgr2=0x7f068c02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:16.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.297+0000 7f0688ff9640 1 --2- 172.21.15.12:0/136925996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068c1c5b70 0x7f068c02d440 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f067c0038b0 tx=0x7f067c003d70 comp rx=0 tx=0).stop 2024-09-11T01:51:16.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.297+0000 7f0688ff9640 1 -- 172.21.15.12:0/136925996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068c1c5b70 msgr2=0x7f068c02d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:51:16.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.297+0000 7f0688ff9640 1 --2- 172.21.15.12:0/136925996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068c1c5b70 0x7f068c02d440 secure :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f067c0038b0 tx=0x7f067c003d70 comp rx=0 tx=0).stop 2024-09-11T01:51:16.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.297+0000 7f0690fc0640 1 --2- 172.21.15.12:0/136925996 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f067803e240 0x7f0678040700 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f0684007740 tx=0x7f0684009820 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:16.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.303+0000 7f0692a23640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:51:16.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.303+0000 7f0692a23640 1 --2- 172.21.15.12:0/136925996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068c1c5f50 0x7f068c02d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.303+0000 7f0692a23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.303+0000 7f0692a23640 1 -- 172.21.15.12:0/136925996 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f068c1a2d00 con 0x7f068c1c5f50 2024-09-11T01:51:16.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.304+0000 7f068bfff640 1 --2- 172.21.15.12:0/136925996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068c1c5f50 0x7f068c02d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.304+0000 7f068bfff640 10 monclient: get_auth_request con 0x7f068c1c5f50 auth_method 0 2024-09-11T01:51:16.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.304+0000 7f068bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:16.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.304+0000 7f068bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:16.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.304+0000 7f068bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:16.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.304+0000 7f068bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:16.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.304+0000 7f068bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:16.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.304+0000 7f068bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:16.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.304+0000 7f068bfff640 10 monclient(hunting): handle_auth_done global_id 214431 payload 824 2024-09-11T01:51:16.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.304+0000 7f068bfff640 1 -- 172.21.15.12:0/136925996 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f068c1a9e90 con 0x7f068c1c5f50 2024-09-11T01:51:16.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.306+0000 7f068bfff640 1 -- 172.21.15.12:0/136925996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068c1c5f50 msgr2=0x7f068c02d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:16.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.306+0000 7f068bfff640 1 --2- 172.21.15.12:0/136925996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068c1c5f50 0x7f068c02d980 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f067c0038b0 tx=0x7f067c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:16.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.306+0000 7f0688ff9640 1 -- 172.21.15.12:0/136925996 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067c01c030 con 0x7f068c1c5f50 2024-09-11T01:51:16.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.306+0000 7f0688ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:16.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.306+0000 7f0688ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:51:16.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.306+0000 7f0688ff9640 10 monclient: dump: 2024-09-11T01:51:16.319 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:16.319 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:16.320 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.320 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.320 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:16.320 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.320 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:16.320 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:16.320 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:16.320 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.306+0000 7f0688ff9640 1 -- 172.21.15.12:0/136925996 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4878 (secure 0 0 0) 0x7f067c018ca0 con 0x7f068c1c5f50 2024-09-11T01:51:16.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.306+0000 7f0688ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:51:16.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.306+0000 7f0688ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:51:16.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.306+0000 7f0688ff9640 1 -- 172.21.15.12:0/136925996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068c1c5f50 msgr2=0x7f068c02d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:16.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.306+0000 7f0688ff9640 1 --2- 172.21.15.12:0/136925996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068c1c5f50 0x7f068c02d980 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f067c0038b0 tx=0x7f067c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.306+0000 7f0688ff9640 1 -- 172.21.15.12:0/136925996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068c1c5f50 msgr2=0x7f068c02d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:51:16.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.306+0000 7f0688ff9640 1 --2- 172.21.15.12:0/136925996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068c1c5f50 0x7f068c02d980 secure :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f067c0038b0 tx=0x7f067c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.321 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:51:16.321 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:51:16.321 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:51:16.321 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:51:16.321 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 208, 2024-09-11T01:51:16.322 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 182506, 2024-09-11T01:51:16.322 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:51:16.322 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:51:16.322 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:51:16.322 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:51:16.322 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:16.322 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:51:16.322 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:51:16.322 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:51:16.322 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:51:16.323 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:16.323 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:16.323 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:16.323 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:16.323 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:16.323 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:16.323 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:16.323 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:16.323 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:16.323 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:16.323 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:16.323 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:16.323 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:51:16.324 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:51:16.324 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:16.324 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:16.324 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:16.324 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:16.324 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:16.324 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:16.324 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:16.324 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:16.324 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:16.324 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:16.324 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:16.324 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:16.325 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.325 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:51:16.325 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:51:16.325 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:51:16.325 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:51:16.325 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:51:16.325 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:51:16.325 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:51:16.325 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:51:16.325 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:51:16.325 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:51:16.325 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:51:16.325 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:51:16.326 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:51:16.326 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:51:16.326 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:51:16.326 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:51:16.326 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:51:16.326 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:16.326 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:16.326 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:16.326 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:16.326 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:16.326 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:16.326 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:16.326 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:16.326 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:16.327 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:16.327 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:16.327 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:16.327 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:51:16.327 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.327 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:51:16.327 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.327 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:51:16.327 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:51:16.327 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:16.327 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:16.327 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.328 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:16.328 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:51:16.328 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:16.328 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.328 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.328 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:16.328 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:51:16.328 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:16.328 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:16.328 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:16.328 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.328 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:51:16.328 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:51:16.328 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:16.329 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:16.329 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:16.329 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.329 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.329 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:51:16.329 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:51:16.329 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:16.329 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:16.329 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.329 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:16.329 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:51:16.329 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:16.329 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.329 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.330 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:16.330 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:51:16.330 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:16.330 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:16.330 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:16.330 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.330 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:51:16.330 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:51:16.330 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:16.330 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:16.330 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:16.330 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.330 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.331 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:51:16.331 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:51:16.331 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:16.331 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:16.331 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.331 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:16.331 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:51:16.331 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:16.331 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.331 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.331 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:16.331 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:51:16.331 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:16.331 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:16.332 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:16.332 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.332 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:51:16.332 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:51:16.332 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:16.332 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:16.332 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:16.332 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:16.332 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:16.332 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.333 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:51:16.333 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:51:16.333 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.333 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:16.333 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:16.333 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:51:16.333 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:16.333 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:16.333 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:51:16.333 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.333 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:16.333 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:16.334 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:51:16.334 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:16.334 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:16.334 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:51:16.334 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.334 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:16.334 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:16.334 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:51:16.334 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:16.334 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:16.334 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:51:16.334 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.334 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:16.335 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:16.335 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:51:16.335 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:16.335 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:16.335 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.335 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:51:16.335 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:51:16.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.307+0000 7f0692a23640 1 -- 172.21.15.12:0/136925996 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f067803e240 msgr2=0x7f0678040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:16.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.307+0000 7f0692a23640 1 --2- 172.21.15.12:0/136925996 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f067803e240 0x7f0678040700 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f0684007740 tx=0x7f0684009820 comp rx=0 tx=0).stop 2024-09-11T01:51:16.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.307+0000 7f0692a23640 10 monclient: shutdown 2024-09-11T01:51:16.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.307+0000 7f0692a23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:16.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.307+0000 7f0692a23640 1 -- 172.21.15.12:0/136925996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068c0a11d0 msgr2=0x7f068c1c8dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:16.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.307+0000 7f0692a23640 1 --2- 172.21.15.12:0/136925996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f068c0a11d0 0x7f068c1c8dd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f068002a600 tx=0x7f06800046c0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.307+0000 7f0690fc0640 1 -- 172.21.15.12:0/136925996 reap_dead start 2024-09-11T01:51:16.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.307+0000 7f0690fc0640 1 -- 172.21.15.12:0/136925996 reap_dead start 2024-09-11T01:51:16.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.307+0000 7f0688ff9640 1 -- 172.21.15.12:0/136925996 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(734..734 src has 507..734) ==== 4783+0+0 (unknown 0 0 0) 0x7f0680073300 con 0x7f068c0a11d0 2024-09-11T01:51:16.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.307+0000 7f0692a23640 1 -- 172.21.15.12:0/136925996 shutdown_connections 2024-09-11T01:51:16.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.307+0000 7f0692a23640 1 -- 172.21.15.12:0/136925996 >> 172.21.15.12:0/136925996 conn(0x7f068c0a2f70 msgr2=0x7f068c0a57f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:16.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.307+0000 7f0692a23640 1 -- 172.21.15.12:0/136925996 shutdown_connections 2024-09-11T01:51:16.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.307+0000 7f0692a23640 1 -- 172.21.15.12:0/136925996 wait complete. 2024-09-11T01:51:16.338 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:51:16.338 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:51:16.338 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:51:16.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.438+0000 7fbfff8c6640 10 monclient: get_monmap_and_config 2024-09-11T01:51:16.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.438+0000 7fbfff8c6640 10 monclient: build_initial_monmap 2024-09-11T01:51:16.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.438+0000 7fbfff8c6640 10 monclient: monmap: 2024-09-11T01:51:16.440 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:16.440 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:16.441 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:51:16.439203+0000 2024-09-11T01:51:16.441 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:51:16.439203+0000 2024-09-11T01:51:16.441 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:16.441 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.441 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:51:16.441 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:51:16.441 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:51:16.441 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:51:16.441 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:51:16.441 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.439+0000 7fbfff8c6640 1 Processor -- start 2024-09-11T01:51:16.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.439+0000 7fbfff8c6640 1 -- start start 2024-09-11T01:51:16.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.439+0000 7fbfff8c6640 10 monclient: init 2024-09-11T01:51:16.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.439+0000 7fbfff8c6640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:16.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.439+0000 7fbfff8c6640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-11T01:51:16.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.439+0000 7fbfff8c6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbff80a2e20 0x7fbff806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.439+0000 7fbfff8c6640 10 monclient(hunting): picked mon.noname-c con 0x7fbff80a2e20 addr v2:172.21.15.71:3300/0 2024-09-11T01:51:16.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.439+0000 7fbfff8c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbff80a4970 addr v1:172.21.15.12:6789/0 2024-09-11T01:51:16.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.439+0000 7fbfff8c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff806f910 0x7fbff813f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.439+0000 7fbfff8c6640 10 monclient(hunting): picked mon.noname-b con 0x7fbff806f910 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:16.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.439+0000 7fbfff8c6640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbff806f4f0 con 0x7fbff80a4970 2024-09-11T01:51:16.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.439+0000 7fbfff8c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.439+0000 7fbfff8c6640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fbff806ef30 con 0x7fbff80a2e20 2024-09-11T01:51:16.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.439+0000 7fbfff8c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.439+0000 7fbfff8c6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fbff806ec50 con 0x7fbff806f910 2024-09-11T01:51:16.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.439+0000 7fbfff8c6640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:16.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.439+0000 7fbfff8c6640 10 monclient(hunting): authenticate will time out at 2082.881756s 2024-09-11T01:51:16.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.439+0000 7fbffde63640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff806f910 0x7fbff813f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.439+0000 7fbffce61640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbff80a4970 0x7fbff80a3200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:35290/0 (socket says 172.21.15.12:35290) 2024-09-11T01:51:16.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.439+0000 7fbffce61640 1 -- 172.21.15.12:0/2940964884 learned_addr learned my addr 172.21.15.12:0/2940964884 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:51:16.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.439+0000 7fbffde63640 1 --2- 172.21.15.12:0/2940964884 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff806f910 0x7fbff813f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51460/0 (socket says 172.21.15.12:51460) 2024-09-11T01:51:16.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.439+0000 7fbffde63640 10 monclient(hunting): get_auth_request con 0x7fbff806f910 auth_method 0 2024-09-11T01:51:16.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.439+0000 7fbffde63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:16.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.439+0000 7fbffde63640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:16.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.439+0000 7fbffde63640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:16.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.440+0000 7fbffde63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:16.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.440+0000 7fbffde63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:16.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.440+0000 7fbffde63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:16.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.440+0000 7fbffd662640 1 --2- 172.21.15.12:0/2940964884 >> v2:172.21.15.71:3300/0 conn(0x7fbff80a2e20 0x7fbff806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.440+0000 7fbffd662640 10 monclient(hunting): get_auth_request con 0x7fbff80a2e20 auth_method 0 2024-09-11T01:51:16.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.440+0000 7fbffd662640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:16.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.440+0000 7fbffd662640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:16.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.440+0000 7fbffd662640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:16.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.440+0000 7fbffde63640 10 monclient(hunting): handle_auth_done global_id 244139 payload 274 2024-09-11T01:51:16.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.440+0000 7fbffde63640 1 -- 172.21.15.12:0/2940964884 >> v2:172.21.15.71:3300/0 conn(0x7fbff80a2e20 msgr2=0x7fbff806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:16.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.440+0000 7fbffde63640 1 --2- 172.21.15.12:0/2940964884 >> v2:172.21.15.71:3300/0 conn(0x7fbff80a2e20 0x7fbff806ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.440+0000 7fbffde63640 1 -- 172.21.15.12:0/2940964884 >> v1:172.21.15.12:6789/0 conn(0x7fbff80a4970 legacy=0x7fbff80a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:16.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.440+0000 7fbffde63640 10 monclient: _finish_hunting 0 2024-09-11T01:51:16.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.440+0000 7fbffde63640 1 monclient: found mon.noname-b 2024-09-11T01:51:16.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.440+0000 7fbffde63640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:16.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.440+0000 7fbffde63640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:51:16.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.440+0000 7fbffde63640 1 -- 172.21.15.12:0/2940964884 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff80a3910 con 0x7fbff806f910 2024-09-11T01:51:16.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.442+0000 7fbffde63640 1 -- 172.21.15.12:0/2940964884 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff806f910 msgr2=0x7fbff813f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:16.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.442+0000 7fbffde63640 1 --2- 172.21.15.12:0/2940964884 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff806f910 0x7fbff813f430 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbff0009d90 tx=0x7fbff002a490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea0ce2fb3fe3cfc4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:16.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfeffff640 1 -- 172.21.15.12:0/2940964884 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff0007950 con 0x7fbff806f910 2024-09-11T01:51:16.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfeffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:16.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfeffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:51:16.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfeffff640 10 monclient: dump: 2024-09-11T01:51:16.452 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:16.452 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:16.452 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.452 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.452 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:16.453 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.453 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:16.453 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:16.453 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:16.453 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfeffff640 10 monclient: _finish_auth 0 2024-09-11T01:51:16.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfeffff640 10 monclient: _check_auth_tickets 2024-09-11T01:51:16.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfeffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:16.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfff8c6640 5 monclient: authenticate success, global_id 244139 2024-09-11T01:51:16.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfeffff640 1 -- 172.21.15.12:0/2940964884 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbff0007af0 con 0x7fbff806f910 2024-09-11T01:51:16.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfff8c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:51:16.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfeffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:16.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfff8c6640 10 monclient: get_monmap_and_config success 2024-09-11T01:51:16.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfff8c6640 4 set_mon_vals no callback set 2024-09-11T01:51:16.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfeffff640 1 -- 172.21.15.12:0/2940964884 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff0031400 con 0x7fbff806f910 2024-09-11T01:51:16.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfeffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:16.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfeffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:16.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfeffff640 10 monclient: dump: 2024-09-11T01:51:16.454 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:16.454 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:16.454 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.454 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.454 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:16.454 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.454 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:16.454 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:16.454 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:16.454 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfff8c6640 10 monclient: shutdown 2024-09-11T01:51:16.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfff8c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:16.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfff8c6640 1 -- 172.21.15.12:0/2940964884 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff806f910 msgr2=0x7fbff813f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:16.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfff8c6640 1 --2- 172.21.15.12:0/2940964884 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff806f910 0x7fbff813f430 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbff0009d90 tx=0x7fbff002a490 comp rx=0 tx=0).stop 2024-09-11T01:51:16.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfff8c6640 1 -- 172.21.15.12:0/2940964884 shutdown_connections 2024-09-11T01:51:16.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfff8c6640 1 --2- 172.21.15.12:0/2940964884 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff806f910 0x7fbff813f430 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfff8c6640 1 --2- 172.21.15.12:0/2940964884 >> v2:172.21.15.71:3300/0 conn(0x7fbff80a2e20 0x7fbff806ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfff8c6640 1 -- 172.21.15.12:0/2940964884 >> 172.21.15.12:0/2940964884 conn(0x7fbff80a8600 msgr2=0x7fbff80a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:16.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfff8c6640 1 -- 172.21.15.12:0/2940964884 shutdown_connections 2024-09-11T01:51:16.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.443+0000 7fbfff8c6640 1 -- 172.21.15.12:0/2940964884 wait complete. 2024-09-11T01:51:16.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.444+0000 7fbfff8c6640 10 monclient: build_initial_monmap 2024-09-11T01:51:16.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.444+0000 7fbfff8c6640 10 monclient: monmap: 2024-09-11T01:51:16.455 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:16.456 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:51:16.456 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:51:16.456 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:51:16.456 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:16.456 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.456 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:51:16.456 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:51:16.456 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:51:16.456 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.444+0000 7fbfff8c6640 1 Processor -- start 2024-09-11T01:51:16.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.444+0000 7fbfff8c6640 1 -- start start 2024-09-11T01:51:16.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.444+0000 7fbfff8c6640 10 monclient: init 2024-09-11T01:51:16.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbfff8c6640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:16.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbfff8c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:51:16.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbfff8c6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbff806f910 0x7fbff81398c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbfff8c6640 10 monclient(hunting): picked mon.noname-b con 0x7fbff806f910 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:51:16.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbfff8c6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbff80a2e20 0x7fbff8139e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbfff8c6640 10 monclient(hunting): picked mon.noname-a con 0x7fbff80a2e20 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:51:16.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbfff8c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff80a4970 0x7fbff81ccf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbfff8c6640 10 monclient(hunting): picked mon.noname-c con 0x7fbff80a4970 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:16.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbfff8c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbfff8c6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbff813fd00 con 0x7fbff80a2e20 2024-09-11T01:51:16.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbfff8c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbfff8c6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbff813fb80 con 0x7fbff806f910 2024-09-11T01:51:16.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbfff8c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbfff8c6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fbff813fe80 con 0x7fbff80a4970 2024-09-11T01:51:16.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbfff8c6640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:16.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbfff8c6640 10 monclient(hunting): authenticate will time out at 2082.887225s 2024-09-11T01:51:16.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffce61640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbff80a2e20 0x7fbff8139e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffde63640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff80a4970 0x7fbff81ccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffce61640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbff80a2e20 0x7fbff8139e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34416/0 (socket says 172.21.15.12:34416) 2024-09-11T01:51:16.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffce61640 1 -- 172.21.15.12:0/1815719020 learned_addr learned my addr 172.21.15.12:0/1815719020 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:16.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffce61640 10 monclient(hunting): get_auth_request con 0x7fbff80a2e20 auth_method 0 2024-09-11T01:51:16.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffce61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:16.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffce61640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:16.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffce61640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:16.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffde63640 10 monclient(hunting): get_auth_request con 0x7fbff80a4970 auth_method 0 2024-09-11T01:51:16.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffde63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:16.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffde63640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:16.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffde63640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:16.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffce61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:16.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffce61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffce61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffde63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffde63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffde63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffd662640 1 --2- 172.21.15.12:0/1815719020 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbff806f910 0x7fbff81398c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffd662640 10 monclient(hunting): get_auth_request con 0x7fbff806f910 auth_method 0 2024-09-11T01:51:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffd662640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffd662640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffd662640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffde63640 10 monclient(hunting): handle_auth_done global_id 244142 payload 824 2024-09-11T01:51:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffde63640 1 -- 172.21.15.12:0/1815719020 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbff806f910 msgr2=0x7fbff81398c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffde63640 1 --2- 172.21.15.12:0/1815719020 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbff806f910 0x7fbff81398c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffde63640 1 -- 172.21.15.12:0/1815719020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbff80a2e20 msgr2=0x7fbff8139e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffde63640 1 --2- 172.21.15.12:0/1815719020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbff80a2e20 0x7fbff8139e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffde63640 10 monclient: _finish_hunting 0 2024-09-11T01:51:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffde63640 1 monclient: found mon.noname-c 2024-09-11T01:51:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffde63640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffde63640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:51:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffde63640 1 -- 172.21.15.12:0/1815719020 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff80a5b50 con 0x7fbff80a4970 2024-09-11T01:51:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.445+0000 7fbffce61640 1 --2- 172.21.15.12:0/1815719020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbff80a2e20 0x7fbff8139e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:51:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.447+0000 7fbffde63640 1 -- 172.21.15.12:0/1815719020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff80a4970 msgr2=0x7fbff81ccf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.447+0000 7fbffde63640 1 --2- 172.21.15.12:0/1815719020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff80a4970 0x7fbff81ccf20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbff0031000 tx=0x7fbff0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.448+0000 7fbfedffb640 1 -- 172.21.15.12:0/1815719020 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff0040070 con 0x7fbff80a4970 2024-09-11T01:51:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.448+0000 7fbfedffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.448+0000 7fbfedffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:51:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.448+0000 7fbfedffb640 10 monclient: dump: 2024-09-11T01:51:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:16.461 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.448+0000 7fbfedffb640 10 monclient: _finish_auth 0 2024-09-11T01:51:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.448+0000 7fbfedffb640 10 monclient: _check_auth_tickets 2024-09-11T01:51:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.448+0000 7fbfedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.448+0000 7fbfedffb640 1 -- 172.21.15.12:0/1815719020 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbff0045680 con 0x7fbff80a4970 2024-09-11T01:51:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.448+0000 7fbfedffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:16.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.448+0000 7fbfedffb640 1 -- 172.21.15.12:0/1815719020 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff00314c0 con 0x7fbff80a4970 2024-09-11T01:51:16.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.448+0000 7fbfff8c6640 5 monclient: authenticate success, global_id 244142 2024-09-11T01:51:16.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.448+0000 7fbfff8c6640 10 monclient: _renew_subs 2024-09-11T01:51:16.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.448+0000 7fbfff8c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:16.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.448+0000 7fbfff8c6640 1 -- 172.21.15.12:0/1815719020 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbff809f610 con 0x7fbff80a4970 2024-09-11T01:51:16.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.448+0000 7fbfef7fe640 4 set_mon_vals no callback set 2024-09-11T01:51:16.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.448+0000 7fbfedffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:16.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.448+0000 7fbfedffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:16.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.448+0000 7fbfedffb640 10 monclient: dump: 2024-09-11T01:51:16.462 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:16.462 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:16.463 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.463 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.463 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:16.463 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.463 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:16.463 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:16.463 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:16.463 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.448+0000 7fbfff8c6640 10 monclient: _renew_subs 2024-09-11T01:51:16.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.448+0000 7fbfff8c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:16.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.448+0000 7fbfff8c6640 1 -- 172.21.15.12:0/1815719020 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbff813bd00 con 0x7fbff80a4970 2024-09-11T01:51:16.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.449+0000 7fbfff8c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.449+0000 7fbfedffb640 1 -- 172.21.15.12:0/1815719020 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fbff0007960 con 0x7fbff80a4970 2024-09-11T01:51:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.449+0000 7fbfff8c6640 1 --2- 172.21.15.12:0/1815719020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff813a7e0 0x7fbff802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.449+0000 7fbfff8c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.449+0000 7fbfff8c6640 1 -- 172.21.15.12:0/1815719020 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fbff813b980 con 0x7fbff813a7e0 2024-09-11T01:51:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.449+0000 7fbffd662640 1 --2- 172.21.15.12:0/1815719020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff813a7e0 0x7fbff802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.449+0000 7fbffd662640 10 monclient: get_auth_request con 0x7fbff813a7e0 auth_method 0 2024-09-11T01:51:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.449+0000 7fbffd662640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.449+0000 7fbffd662640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.449+0000 7fbffd662640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.449+0000 7fbffd662640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.449+0000 7fbffd662640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.449+0000 7fbffd662640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.449+0000 7fbfedffb640 1 --2- 172.21.15.12:0/1815719020 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbfdc03e240 0x7fbfdc040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.449+0000 7fbfedffb640 1 -- 172.21.15.12:0/1815719020 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(734..734 src has 507..734) ==== 4783+0+0 (secure 0 0 0) 0x7fbff00749f0 con 0x7fbff80a4970 2024-09-11T01:51:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.449+0000 7fbffd662640 10 monclient(hunting): handle_auth_done global_id 244145 payload 824 2024-09-11T01:51:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.449+0000 7fbffd662640 1 -- 172.21.15.12:0/1815719020 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbff806ded0 con 0x7fbff813a7e0 2024-09-11T01:51:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.449+0000 7fbffce61640 1 --2- 172.21.15.12:0/1815719020 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbfdc03e240 0x7fbfdc040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.449+0000 7fbffce61640 10 monclient: get_auth_request con 0x7fbfdc03e240 auth_method 0 2024-09-11T01:51:16.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.451+0000 7fbffd662640 1 -- 172.21.15.12:0/1815719020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff813a7e0 msgr2=0x7fbff802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:16.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.451+0000 7fbffd662640 1 --2- 172.21.15.12:0/1815719020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff813a7e0 0x7fbff802d440 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbff80a2a20 tx=0x7fbff400f930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:16.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.451+0000 7fbfedffb640 1 -- 172.21.15.12:0/1815719020 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff4019030 con 0x7fbff813a7e0 2024-09-11T01:51:16.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.451+0000 7fbfedffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:16.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.451+0000 7fbfedffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.451+0000 7fbfedffb640 10 monclient: dump: 2024-09-11T01:51:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:16.468 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.452+0000 7fbffce61640 1 --2- 172.21.15.12:0/1815719020 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbfdc03e240 0x7fbfdc040700 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8007d50 tx=0x7fbfe8007ca0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.452+0000 7fbfedffb640 1 -- 172.21.15.12:0/1815719020 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbff4004020 con 0x7fbff813a7e0 2024-09-11T01:51:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.452+0000 7fbfedffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.452+0000 7fbfedffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:51:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.452+0000 7fbfedffb640 1 -- 172.21.15.12:0/1815719020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff813a7e0 msgr2=0x7fbff802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.452+0000 7fbfedffb640 1 --2- 172.21.15.12:0/1815719020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff813a7e0 0x7fbff802d440 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbff80a2a20 tx=0x7fbff400f930 comp rx=0 tx=0).stop 2024-09-11T01:51:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.452+0000 7fbfedffb640 1 -- 172.21.15.12:0/1815719020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff813a7e0 msgr2=0x7fbff802d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:51:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.452+0000 7fbfedffb640 1 --2- 172.21.15.12:0/1815719020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff813a7e0 0x7fbff802d440 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbff80a2a20 tx=0x7fbff400f930 comp rx=0 tx=0).stop 2024-09-11T01:51:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.458+0000 7fbfff8c6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:51:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.458+0000 7fbfff8c6640 1 --2- 172.21.15.12:0/1815719020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff802d980 0x7fbff802dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.458+0000 7fbfff8c6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.458+0000 7fbfff8c6640 1 -- 172.21.15.12:0/1815719020 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fbff806a060 con 0x7fbff802d980 2024-09-11T01:51:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.458+0000 7fbffd662640 1 --2- 172.21.15.12:0/1815719020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff802d980 0x7fbff802dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.458+0000 7fbffd662640 10 monclient: get_auth_request con 0x7fbff802d980 auth_method 0 2024-09-11T01:51:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.458+0000 7fbffd662640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.458+0000 7fbffd662640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.458+0000 7fbffd662640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.458+0000 7fbffd662640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.458+0000 7fbffd662640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.458+0000 7fbffd662640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.458+0000 7fbffd662640 10 monclient(hunting): handle_auth_done global_id 244148 payload 824 2024-09-11T01:51:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.458+0000 7fbffd662640 1 -- 172.21.15.12:0/1815719020 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbff81378d0 con 0x7fbff802d980 2024-09-11T01:51:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.460+0000 7fbffd662640 1 -- 172.21.15.12:0/1815719020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff802d980 msgr2=0x7fbff802dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.460+0000 7fbffd662640 1 --2- 172.21.15.12:0/1815719020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff802d980 0x7fbff802dd60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbff80a2a20 tx=0x7fbff40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.461+0000 7fbfedffb640 1 -- 172.21.15.12:0/1815719020 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff401fe30 con 0x7fbff802d980 2024-09-11T01:51:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.461+0000 7fbfedffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.461+0000 7fbfedffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.461+0000 7fbfedffb640 10 monclient: dump: 2024-09-11T01:51:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:16.472 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.472 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:16.472 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:16.472 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:16.472 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.461+0000 7fbfedffb640 1 -- 172.21.15.12:0/1815719020 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7fbff401fe30 con 0x7fbff802d980 2024-09-11T01:51:16.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.461+0000 7fbfedffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:51:16.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.461+0000 7fbfedffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:51:16.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.461+0000 7fbfedffb640 1 -- 172.21.15.12:0/1815719020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff802d980 msgr2=0x7fbff802dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:16.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.461+0000 7fbfedffb640 1 --2- 172.21.15.12:0/1815719020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff802d980 0x7fbff802dd60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbff80a2a20 tx=0x7fbff40046c0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.461+0000 7fbfedffb640 1 -- 172.21.15.12:0/1815719020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff802d980 msgr2=0x7fbff802dd60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:51:16.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.461+0000 7fbfedffb640 1 --2- 172.21.15.12:0/1815719020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff802d980 0x7fbff802dd60 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbff80a2a20 tx=0x7fbff40046c0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.473 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:51:16.473 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:51:16.473 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:51:16.473 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:51:16.473 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 208, 2024-09-11T01:51:16.473 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 29909, 2024-09-11T01:51:16.473 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:51:16.473 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:51:16.474 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:51:16.474 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:51:16.474 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:16.474 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:51:16.474 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:51:16.474 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:51:16.474 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:51:16.474 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:16.474 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:16.474 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:16.474 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:16.474 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:16.475 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:16.475 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:16.475 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:16.475 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:16.475 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:16.475 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:16.475 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:16.475 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:51:16.475 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:51:16.475 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:16.475 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:16.475 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:16.475 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:16.476 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:16.476 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:16.476 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:16.476 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:16.476 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:16.476 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:16.476 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:16.476 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:16.476 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.476 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:51:16.476 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:51:16.476 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:51:16.476 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:51:16.477 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:51:16.477 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:51:16.477 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:51:16.477 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:51:16.477 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:51:16.477 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:51:16.477 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:51:16.477 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:51:16.477 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:51:16.477 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:51:16.477 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:51:16.477 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:51:16.477 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:51:16.477 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:16.477 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:16.478 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:16.478 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:16.478 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:16.478 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:16.478 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:16.478 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:16.478 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:16.478 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:16.478 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:16.478 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:16.478 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:51:16.479 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.479 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:51:16.479 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.479 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:51:16.479 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:51:16.479 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:16.479 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:16.479 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.479 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:16.479 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:51:16.479 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:16.479 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.479 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.479 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:16.480 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:51:16.480 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:16.480 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:16.480 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:16.480 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.480 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:51:16.480 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:51:16.480 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:16.480 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:16.480 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:16.480 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.480 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.480 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:51:16.481 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:51:16.481 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:16.481 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:16.481 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.481 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:16.481 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:51:16.481 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:16.481 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.481 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.481 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:16.481 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:51:16.482 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:16.482 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:16.482 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:16.482 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.482 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:51:16.482 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:51:16.482 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:16.482 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:16.482 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:16.482 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.482 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.482 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:51:16.482 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:51:16.482 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:16.483 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:16.483 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.483 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:16.483 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:51:16.483 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:16.483 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.483 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.483 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:16.483 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:51:16.483 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:16.483 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:16.483 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:16.483 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.483 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:51:16.484 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:51:16.484 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:16.484 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:16.484 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:16.484 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:16.484 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:16.484 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.484 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:51:16.484 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:51:16.484 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.484 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:16.484 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:16.484 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:51:16.485 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:16.485 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:16.485 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:51:16.485 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.485 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:16.485 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:16.485 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:51:16.485 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:16.485 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:16.485 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.485 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:51:16.485 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:51:16.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.461+0000 7fbfff8c6640 1 -- 172.21.15.12:0/1815719020 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbfdc03e240 msgr2=0x7fbfdc040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:16.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.461+0000 7fbfff8c6640 1 --2- 172.21.15.12:0/1815719020 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbfdc03e240 0x7fbfdc040700 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8007d50 tx=0x7fbfe8007ca0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.461+0000 7fbfff8c6640 10 monclient: shutdown 2024-09-11T01:51:16.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.461+0000 7fbfff8c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:16.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.461+0000 7fbfff8c6640 1 -- 172.21.15.12:0/1815719020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff80a4970 msgr2=0x7fbff81ccf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:16.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.461+0000 7fbfff8c6640 1 --2- 172.21.15.12:0/1815719020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff80a4970 0x7fbff81ccf20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbff0031000 tx=0x7fbff0004a40 comp rx=0 tx=0).stop 2024-09-11T01:51:16.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.461+0000 7fbffde63640 1 -- 172.21.15.12:0/1815719020 reap_dead start 2024-09-11T01:51:16.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.461+0000 7fbfff8c6640 1 -- 172.21.15.12:0/1815719020 shutdown_connections 2024-09-11T01:51:16.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.461+0000 7fbfff8c6640 1 --2- 172.21.15.12:0/1815719020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbff80a4970 0x7fbff81ccf20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.461+0000 7fbfff8c6640 1 -- 172.21.15.12:0/1815719020 >> 172.21.15.12:0/1815719020 conn(0x7fbff80a8600 msgr2=0x7fbff809e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:16.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.461+0000 7fbfff8c6640 1 -- 172.21.15.12:0/1815719020 shutdown_connections 2024-09-11T01:51:16.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.461+0000 7fbfff8c6640 1 -- 172.21.15.12:0/1815719020 wait complete. 2024-09-11T01:51:16.489 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:51:16.490 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:51:16.490 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:51:16.576 INFO:tasks.rados.rados.0.smithi071.stdout:3286: done (2 left) 2024-09-11T01:51:16.576 INFO:tasks.rados.rados.0.smithi071.stdout:3287: read oid 3 snap 322 2024-09-11T01:51:16.576 INFO:tasks.rados.rados.0.smithi071.stdout:3287: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:51:16.576 INFO:tasks.rados.rados.0.smithi071.stdout:3288: copy_from oid 34 from oid 5 current snap is 366 2024-09-11T01:51:16.576 INFO:tasks.rados.rados.0.smithi071.stdout:3282: finishing write tid 2 to smithi07134777-6 2024-09-11T01:51:16.576 INFO:tasks.rados.rados.0.smithi071.stdout:3282: finishing write tid 3 to smithi07134777-6 2024-09-11T01:51:16.576 INFO:tasks.rados.rados.0.smithi071.stdout:3282: finishing write tid 4 to smithi07134777-6 2024-09-11T01:51:16.576 INFO:tasks.rados.rados.0.smithi071.stdout:3282: finishing write tid 5 to smithi07134777-6 2024-09-11T01:51:16.577 INFO:tasks.rados.rados.0.smithi071.stdout:3282: oid 6 updating version 0 to 2210 2024-09-11T01:51:16.577 INFO:tasks.rados.rados.0.smithi071.stdout:3282: oid 6 updating version 2210 to 2212 2024-09-11T01:51:16.577 INFO:tasks.rados.rados.0.smithi071.stdout:3282: oid 6 updating version 2212 to 2213 2024-09-11T01:51:16.577 INFO:tasks.rados.rados.0.smithi071.stdout:3282: oid 6 version 2213 is already newer than 2211 2024-09-11T01:51:16.577 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 2213 (ObjNum 1151 snap 366 seq_num 1151) dirty exists 2024-09-11T01:51:16.577 INFO:tasks.rados.rados.0.smithi071.stdout:3282: left oid 6 (ObjNum 1151 snap 366 seq_num 1151) 2024-09-11T01:51:16.577 INFO:tasks.rados.rados.0.smithi071.stdout:3285: finishing write tid 1 to smithi07134777-1 2024-09-11T01:51:16.577 INFO:tasks.rados.rados.0.smithi071.stdout:3285: finishing write tid 2 to smithi07134777-1 2024-09-11T01:51:16.577 INFO:tasks.rados.rados.0.smithi071.stdout:3285: finishing write tid 3 to smithi07134777-1 2024-09-11T01:51:16.577 INFO:tasks.rados.rados.0.smithi071.stdout:3285: finishing write tid 4 to smithi07134777-1 2024-09-11T01:51:16.577 INFO:tasks.rados.rados.0.smithi071.stdout:3285: finishing write tid 5 to smithi07134777-1 2024-09-11T01:51:16.577 INFO:tasks.rados.rados.0.smithi071.stdout:3285: oid 1 updating version 0 to 1890 2024-09-11T01:51:16.577 INFO:tasks.rados.rados.0.smithi071.stdout:3285: oid 1 updating version 1890 to 1891 2024-09-11T01:51:16.577 INFO:tasks.rados.rados.0.smithi071.stdout:3285: oid 1 updating version 1891 to 1892 2024-09-11T01:51:16.578 INFO:tasks.rados.rados.0.smithi071.stdout:3285: oid 1 updating version 1892 to 1893 2024-09-11T01:51:16.578 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 1893 (ObjNum 1154 snap 366 seq_num 1154) dirty exists 2024-09-11T01:51:16.578 INFO:tasks.rados.rados.0.smithi071.stdout:3285: left oid 1 (ObjNum 1154 snap 366 seq_num 1154) 2024-09-11T01:51:16.578 INFO:tasks.rados.rados.0.smithi071.stdout:3282: done (3 left) 2024-09-11T01:51:16.578 INFO:tasks.rados.rados.0.smithi071.stdout:3285: done (2 left) 2024-09-11T01:51:16.578 INFO:tasks.rados.rados.0.smithi071.stdout:3289: snap_create 2024-09-11T01:51:16.588 INFO:tasks.rados.rados.0.smithi071.stdout:3287: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:51:16.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.601+0000 7f8e0a9ad640 10 monclient: get_monmap_and_config 2024-09-11T01:51:16.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.601+0000 7f8e0a9ad640 10 monclient: build_initial_monmap 2024-09-11T01:51:16.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.601+0000 7f8e0a9ad640 10 monclient: monmap: 2024-09-11T01:51:16.603 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:16.603 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:16.603 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:51:16.602536+0000 2024-09-11T01:51:16.603 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:51:16.602536+0000 2024-09-11T01:51:16.604 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:16.604 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.604 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:51:16.604 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:51:16.604 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:51:16.604 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:51:16.604 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:51:16.604 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.602+0000 7f8e0a9ad640 1 Processor -- start 2024-09-11T01:51:16.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.602+0000 7f8e0a9ad640 1 -- start start 2024-09-11T01:51:16.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.602+0000 7f8e0a9ad640 10 monclient: init 2024-09-11T01:51:16.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e0a9ad640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:16.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e0a9ad640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-11T01:51:16.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e0a9ad640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8e04137e50 0x7f8e0409d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e0a9ad640 10 monclient(hunting): picked mon.noname-b con 0x7f8e04137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:16.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e0a9ad640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8e04138230 0x7f8e04138630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e0a9ad640 10 monclient(hunting): picked mon.noname-a con 0x7f8e04138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:51:16.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e0a9ad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8e04134cb0 addr v1:172.21.15.71:6789/0 2024-09-11T01:51:16.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e0a9ad640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e040a62d0 con 0x7f8e04134cb0 2024-09-11T01:51:16.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e0a9ad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e0a9ad640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f8e040a65b0 con 0x7f8e04138230 2024-09-11T01:51:16.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e0a9ad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e0a9ad640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8e0409bb90 con 0x7f8e04137e50 2024-09-11T01:51:16.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e0a9ad640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:16.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e0a9ad640 10 monclient(hunting): authenticate will time out at 2083.045200s 2024-09-11T01:51:16.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e03fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8e04137e50 0x7f8e0409d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e037fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8e04138230 0x7f8e04138630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e03fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8e04137e50 0x7f8e0409d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51500/0 (socket says 172.21.15.12:51500) 2024-09-11T01:51:16.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e037fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8e04138230 0x7f8e04138630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34422/0 (socket says 172.21.15.12:34422) 2024-09-11T01:51:16.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e03fff640 1 -- 172.21.15.12:0/212985993 learned_addr learned my addr 172.21.15.12:0/212985993 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:16.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e03fff640 10 monclient(hunting): get_auth_request con 0x7f8e04137e50 auth_method 0 2024-09-11T01:51:16.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:16.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e03fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:16.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:16.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e037fe640 10 monclient(hunting): get_auth_request con 0x7f8e04138230 auth_method 0 2024-09-11T01:51:16.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:16.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e037fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:16.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:16.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:16.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:16.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:16.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:16.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:16.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:16.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e037fe640 10 monclient(hunting): handle_auth_done global_id 214437 payload 274 2024-09-11T01:51:16.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e037fe640 1 -- 172.21.15.12:0/212985993 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8e04137e50 msgr2=0x7f8e0409d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:16.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e037fe640 1 --2- 172.21.15.12:0/212985993 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8e04137e50 0x7f8e0409d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e037fe640 1 -- 172.21.15.12:0/212985993 >> v1:172.21.15.71:6789/0 conn(0x7f8e04134cb0 legacy=0x7f8e04135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:16.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e037fe640 10 monclient: _finish_hunting 0 2024-09-11T01:51:16.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e037fe640 1 monclient: found mon.noname-a 2024-09-11T01:51:16.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:16.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e037fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:51:16.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e037fe640 1 -- 172.21.15.12:0/212985993 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e0409c4f0 con 0x7f8e04138230 2024-09-11T01:51:16.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.603+0000 7f8e03fff640 1 --2- 172.21.15.12:0/212985993 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8e04137e50 0x7f8e0409d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:51:16.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.606+0000 7f8e037fe640 1 -- 172.21.15.12:0/212985993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e04138230 msgr2=0x7f8e04138630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:16.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.606+0000 7f8e037fe640 1 --2- 172.21.15.12:0/212985993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e04138230 0x7f8e04138630 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8df80097a0 tx=0x7f8df8029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d599cf6773b3b2a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:16.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.606+0000 7f8e02ffd640 1 -- 172.21.15.12:0/212985993 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df8037070 con 0x7f8e04138230 2024-09-11T01:51:16.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.606+0000 7f8e02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:16.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.606+0000 7f8e02ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:51:16.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.606+0000 7f8e02ffd640 10 monclient: dump: 2024-09-11T01:51:16.616 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:16.616 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:16.616 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.616 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.616 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:16.616 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.616 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:16.616 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:16.617 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:16.617 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.606+0000 7f8e02ffd640 10 monclient: _finish_auth 0 2024-09-11T01:51:16.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.606+0000 7f8e02ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:51:16.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.606+0000 7f8e02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:16.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.606+0000 7f8e0a9ad640 5 monclient: authenticate success, global_id 214437 2024-09-11T01:51:16.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.606+0000 7f8e02ffd640 1 -- 172.21.15.12:0/212985993 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8df8032440 con 0x7f8e04138230 2024-09-11T01:51:16.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.606+0000 7f8e0a9ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:51:16.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.606+0000 7f8e02ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:16.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.606+0000 7f8e0a9ad640 10 monclient: get_monmap_and_config success 2024-09-11T01:51:16.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.606+0000 7f8e0a9ad640 4 set_mon_vals no callback set 2024-09-11T01:51:16.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.606+0000 7f8e02ffd640 1 -- 172.21.15.12:0/212985993 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df8032810 con 0x7f8e04138230 2024-09-11T01:51:16.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.606+0000 7f8e02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:16.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.606+0000 7f8e02ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:51:16.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.606+0000 7f8e02ffd640 10 monclient: dump: 2024-09-11T01:51:16.618 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:16.618 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:16.618 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.618 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.618 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:16.618 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.618 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:16.618 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:16.618 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:16.618 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.607+0000 7f8e0a9ad640 10 monclient: shutdown 2024-09-11T01:51:16.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.607+0000 7f8e0a9ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:16.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.607+0000 7f8e0a9ad640 1 -- 172.21.15.12:0/212985993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e04138230 msgr2=0x7f8e04138630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:16.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.607+0000 7f8e0a9ad640 1 --2- 172.21.15.12:0/212985993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e04138230 0x7f8e04138630 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8df80097a0 tx=0x7f8df8029fd0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.607+0000 7f8e0a9ad640 1 -- 172.21.15.12:0/212985993 shutdown_connections 2024-09-11T01:51:16.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.607+0000 7f8e0a9ad640 1 --2- 172.21.15.12:0/212985993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e04138230 0x7f8e04138630 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.607+0000 7f8e0a9ad640 1 --2- 172.21.15.12:0/212985993 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8e04137e50 0x7f8e0409d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.607+0000 7f8e0a9ad640 1 -- 172.21.15.12:0/212985993 >> 172.21.15.12:0/212985993 conn(0x7f8e040a7020 msgr2=0x7f8e040a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:16.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.607+0000 7f8e0a9ad640 1 -- 172.21.15.12:0/212985993 shutdown_connections 2024-09-11T01:51:16.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.607+0000 7f8e0a9ad640 1 -- 172.21.15.12:0/212985993 wait complete. 2024-09-11T01:51:16.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.607+0000 7f8e0a9ad640 10 monclient: build_initial_monmap 2024-09-11T01:51:16.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.607+0000 7f8e0a9ad640 10 monclient: monmap: 2024-09-11T01:51:16.619 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:16.619 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:51:16.620 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:51:16.620 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:51:16.620 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:16.620 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.620 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:51:16.620 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:51:16.620 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:51:16.620 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.608+0000 7f8e0a9ad640 1 Processor -- start 2024-09-11T01:51:16.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.608+0000 7f8e0a9ad640 1 -- start start 2024-09-11T01:51:16.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.608+0000 7f8e0a9ad640 10 monclient: init 2024-09-11T01:51:16.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.608+0000 7f8e0a9ad640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:16.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.608+0000 7f8e0a9ad640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:51:16.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.608+0000 7f8e0a9ad640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8e04134cb0 0x7f8e041cd750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.608+0000 7f8e0a9ad640 10 monclient(hunting): picked mon.noname-c con 0x7f8e04134cb0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:16.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.608+0000 7f8e0a9ad640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e04137e50 0x7f8e04134020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.608+0000 7f8e0a9ad640 10 monclient(hunting): picked mon.noname-b con 0x7f8e04137e50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:51:16.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.608+0000 7f8e0a9ad640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e04138230 0x7f8e04134560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.608+0000 7f8e0a9ad640 10 monclient(hunting): picked mon.noname-a con 0x7f8e04138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:51:16.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.608+0000 7f8e0a9ad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.608+0000 7f8e0a9ad640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0406f710 con 0x7f8e04138230 2024-09-11T01:51:16.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.608+0000 7f8e0a9ad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.608+0000 7f8e0a9ad640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8e04138b70 con 0x7f8e04137e50 2024-09-11T01:51:16.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.608+0000 7f8e0a9ad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.608+0000 7f8e0a9ad640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8e0406f890 con 0x7f8e04134cb0 2024-09-11T01:51:16.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.608+0000 7f8e0a9ad640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:16.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.608+0000 7f8e0a9ad640 10 monclient(hunting): authenticate will time out at 2083.050887s 2024-09-11T01:51:16.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.608+0000 7f8e03fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8e04134cb0 0x7f8e041cd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e08f4a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e04138230 0x7f8e04134560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e037fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e04137e50 0x7f8e04134020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e03fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8e04134cb0 0x7f8e041cd750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51516/0 (socket says 172.21.15.12:51516) 2024-09-11T01:51:16.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e08f4a640 10 monclient(hunting): get_auth_request con 0x7f8e04138230 auth_method 0 2024-09-11T01:51:16.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e03fff640 1 -- 172.21.15.12:0/1654475606 learned_addr learned my addr 172.21.15.12:0/1654475606 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:16.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e08f4a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:16.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e08f4a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:16.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e08f4a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:16.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e037fe640 10 monclient(hunting): get_auth_request con 0x7f8e04137e50 auth_method 0 2024-09-11T01:51:16.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:16.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e037fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:16.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:16.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e03fff640 10 monclient(hunting): get_auth_request con 0x7f8e04134cb0 auth_method 0 2024-09-11T01:51:16.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:16.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e03fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:16.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:16.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e08f4a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:16.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e08f4a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:16.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e08f4a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:16.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:16.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:16.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:16.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:16.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:16.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:16.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e08f4a640 10 monclient(hunting): handle_auth_done global_id 214440 payload 824 2024-09-11T01:51:16.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e08f4a640 1 -- 172.21.15.12:0/1654475606 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8e04134cb0 msgr2=0x7f8e041cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:16.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e08f4a640 1 --2- 172.21.15.12:0/1654475606 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8e04134cb0 0x7f8e041cd750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e08f4a640 1 -- 172.21.15.12:0/1654475606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e04137e50 msgr2=0x7f8e04134020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:16.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e08f4a640 1 --2- 172.21.15.12:0/1654475606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e04137e50 0x7f8e04134020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e08f4a640 10 monclient: _finish_hunting 0 2024-09-11T01:51:16.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e08f4a640 1 monclient: found mon.noname-a 2024-09-11T01:51:16.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e08f4a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:16.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e08f4a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:51:16.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e08f4a640 1 -- 172.21.15.12:0/1654475606 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e04125740 con 0x7f8e04138230 2024-09-11T01:51:16.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.609+0000 7f8e037fe640 1 --2- 172.21.15.12:0/1654475606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e04137e50 0x7f8e04134020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:51:16.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.611+0000 7f8e08f4a640 1 -- 172.21.15.12:0/1654475606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e04138230 msgr2=0x7f8e04134560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:16.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.611+0000 7f8e08f4a640 1 --2- 172.21.15.12:0/1654475606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e04138230 0x7f8e04134560 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc00c780 tx=0x7f8dfc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:16.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.611+0000 7f8e00ff9640 1 -- 172.21.15.12:0/1654475606 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc016020 con 0x7f8e04138230 2024-09-11T01:51:16.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.611+0000 7f8e00ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:16.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.611+0000 7f8e00ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:51:16.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.611+0000 7f8e00ff9640 10 monclient: dump: 2024-09-11T01:51:16.625 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:16.625 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:16.625 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.625 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.625 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:16.625 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.625 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:16.626 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:16.626 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:16.626 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.611+0000 7f8e00ff9640 10 monclient: _finish_auth 0 2024-09-11T01:51:16.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.611+0000 7f8e00ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:51:16.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.611+0000 7f8e00ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:16.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.611+0000 7f8e0a9ad640 5 monclient: authenticate success, global_id 214440 2024-09-11T01:51:16.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.611+0000 7f8e00ff9640 1 -- 172.21.15.12:0/1654475606 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dfc00ac60 con 0x7f8e04138230 2024-09-11T01:51:16.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.611+0000 7f8e0a9ad640 10 monclient: _renew_subs 2024-09-11T01:51:16.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.611+0000 7f8e0a9ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:51:16.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.611+0000 7f8e0a9ad640 1 -- 172.21.15.12:0/1654475606 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e04124220 con 0x7f8e04138230 2024-09-11T01:51:16.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.611+0000 7f8e00ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:16.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.611+0000 7f8e00ff9640 1 -- 172.21.15.12:0/1654475606 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc012660 con 0x7f8e04138230 2024-09-11T01:51:16.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.611+0000 7f8e00ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:16.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.611+0000 7f8e00ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:51:16.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.611+0000 7f8e00ff9640 10 monclient: dump: 2024-09-11T01:51:16.627 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:16.627 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:16.627 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.627 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.627 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:16.627 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.627 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:16.627 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:16.627 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:16.627 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.611+0000 7f8e0a9ad640 10 monclient: _renew_subs 2024-09-11T01:51:16.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.611+0000 7f8e0a9ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:51:16.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.611+0000 7f8e0a9ad640 1 -- 172.21.15.12:0/1654475606 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e04124800 con 0x7f8e04138230 2024-09-11T01:51:16.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.611+0000 7f8e027fc640 4 set_mon_vals no callback set 2024-09-11T01:51:16.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.612+0000 7f8e0a9ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:16.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.612+0000 7f8e0a9ad640 1 --2- 172.21.15.12:0/1654475606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e04125180 0x7f8e04121c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.612+0000 7f8e0a9ad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.612+0000 7f8e0a9ad640 1 -- 172.21.15.12:0/1654475606 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0406a060 con 0x7f8e04125180 2024-09-11T01:51:16.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.612+0000 7f8e00ff9640 1 -- 172.21.15.12:0/1654475606 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f8dfc0128c0 con 0x7f8e04138230 2024-09-11T01:51:16.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.612+0000 7f8e03fff640 1 --2- 172.21.15.12:0/1654475606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e04125180 0x7f8e04121c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.612+0000 7f8e03fff640 10 monclient: get_auth_request con 0x7f8e04125180 auth_method 0 2024-09-11T01:51:16.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.612+0000 7f8e03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:16.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.612+0000 7f8e03fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:16.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.612+0000 7f8e03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:16.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.613+0000 7f8e03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:16.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.613+0000 7f8e03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:16.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.613+0000 7f8e03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:16.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.613+0000 7f8e00ff9640 1 --2- 172.21.15.12:0/1654475606 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8df003e210 0x7f8df00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.613+0000 7f8e03fff640 10 monclient(hunting): handle_auth_done global_id 234181 payload 824 2024-09-11T01:51:16.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.613+0000 7f8e03fff640 1 -- 172.21.15.12:0/1654475606 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8e040a1600 con 0x7f8e04125180 2024-09-11T01:51:16.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.613+0000 7f8e00ff9640 1 -- 172.21.15.12:0/1654475606 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(735..735 src has 676..735) ==== 4799+0+0 (secure 0 0 0) 0x7f8dfc052f30 con 0x7f8e04138230 2024-09-11T01:51:16.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.613+0000 7f8e037fe640 1 --2- 172.21.15.12:0/1654475606 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8df003e210 0x7f8df00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.613+0000 7f8e037fe640 10 monclient: get_auth_request con 0x7f8df003e210 auth_method 0 2024-09-11T01:51:16.630 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:51:16.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.615+0000 7f8e03fff640 1 -- 172.21.15.12:0/1654475606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e04125180 msgr2=0x7f8e04121c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:16.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.615+0000 7f8e03fff640 1 --2- 172.21.15.12:0/1654475606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e04125180 0x7f8e04121c00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8df40119b0 tx=0x7f8df4011e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:16.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.615+0000 7f8e00ff9640 1 -- 172.21.15.12:0/1654475606 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df400fe50 con 0x7f8e04125180 2024-09-11T01:51:16.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.615+0000 7f8e00ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:16.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.615+0000 7f8e00ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:51:16.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.615+0000 7f8e00ff9640 10 monclient: dump: 2024-09-11T01:51:16.632 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:16.633 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:16.633 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.633 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.633 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:16.633 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.633 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:16.633 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:16.633 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:16.633 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.615+0000 7f8e037fe640 1 --2- 172.21.15.12:0/1654475606 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8df003e210 0x7f8df00406d0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f8e040a8360 tx=0x7f8df802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:16.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.615+0000 7f8e00ff9640 1 -- 172.21.15.12:0/1654475606 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8df400fe50 con 0x7f8e04125180 2024-09-11T01:51:16.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.615+0000 7f8e00ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:16.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.615+0000 7f8e00ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:51:16.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.615+0000 7f8e00ff9640 1 -- 172.21.15.12:0/1654475606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e04125180 msgr2=0x7f8e04121c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:16.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.615+0000 7f8e00ff9640 1 --2- 172.21.15.12:0/1654475606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e04125180 0x7f8e04121c00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8df40119b0 tx=0x7f8df4011e70 comp rx=0 tx=0).stop 2024-09-11T01:51:16.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.615+0000 7f8e00ff9640 1 -- 172.21.15.12:0/1654475606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e04125180 msgr2=0x7f8e04121c00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:51:16.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.615+0000 7f8e00ff9640 1 --2- 172.21.15.12:0/1654475606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e04125180 0x7f8e04121c00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.621+0000 7f8e0a9ad640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:51:16.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.621+0000 7f8e0a9ad640 1 --2- 172.21.15.12:0/1654475606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e04122140 0x7f8e04122550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.621+0000 7f8e0a9ad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.621+0000 7f8e0a9ad640 1 -- 172.21.15.12:0/1654475606 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8e04122a90 con 0x7f8e04122140 2024-09-11T01:51:16.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.622+0000 7f8e03fff640 1 --2- 172.21.15.12:0/1654475606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e04122140 0x7f8e04122550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.622+0000 7f8e03fff640 10 monclient: get_auth_request con 0x7f8e04122140 auth_method 0 2024-09-11T01:51:16.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.622+0000 7f8e03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:16.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.622+0000 7f8e03fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:16.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.622+0000 7f8e03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:16.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.622+0000 7f8e03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:16.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.622+0000 7f8e03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:16.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.622+0000 7f8e03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:16.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.622+0000 7f8e03fff640 10 monclient(hunting): handle_auth_done global_id 234184 payload 824 2024-09-11T01:51:16.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.622+0000 7f8e03fff640 1 -- 172.21.15.12:0/1654475606 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8e041236c0 con 0x7f8e04122140 2024-09-11T01:51:16.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.624+0000 7f8e03fff640 1 -- 172.21.15.12:0/1654475606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e04122140 msgr2=0x7f8e04122550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:16.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.624+0000 7f8e03fff640 1 --2- 172.21.15.12:0/1654475606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e04122140 0x7f8e04122550 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8df401b000 tx=0x7f8df40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:16.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.625+0000 7f8e00ff9640 1 -- 172.21.15.12:0/1654475606 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df4014850 con 0x7f8e04122140 2024-09-11T01:51:16.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.625+0000 7f8e00ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:16.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.625+0000 7f8e00ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:51:16.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.625+0000 7f8e00ff9640 10 monclient: dump: 2024-09-11T01:51:16.636 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:16.636 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:16.636 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.636 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.636 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:16.636 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.636 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:16.636 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:16.636 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:16.636 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.625+0000 7f8e00ff9640 1 -- 172.21.15.12:0/1654475606 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7f8df4028400 con 0x7f8e04122140 2024-09-11T01:51:16.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.625+0000 7f8e00ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:51:16.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.625+0000 7f8e00ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:51:16.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.625+0000 7f8e00ff9640 1 -- 172.21.15.12:0/1654475606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e04122140 msgr2=0x7f8e04122550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:16.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.625+0000 7f8e00ff9640 1 --2- 172.21.15.12:0/1654475606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e04122140 0x7f8e04122550 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8df401b000 tx=0x7f8df40046c0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.625+0000 7f8e00ff9640 1 -- 172.21.15.12:0/1654475606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e04122140 msgr2=0x7f8e04122550 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:51:16.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.625+0000 7f8e00ff9640 1 --2- 172.21.15.12:0/1654475606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e04122140 0x7f8e04122550 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8df401b000 tx=0x7f8df40046c0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.637 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:51:16.637 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:51:16.637 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:51:16.637 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:51:16.637 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 208, 2024-09-11T01:51:16.638 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 78517, 2024-09-11T01:51:16.638 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:51:16.638 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:51:16.638 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:51:16.638 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:51:16.638 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:16.638 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:51:16.638 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:51:16.638 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:51:16.638 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:51:16.638 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:16.638 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:16.638 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:16.638 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:16.639 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:16.639 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:16.639 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:16.639 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:16.639 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:16.639 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:16.639 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:16.639 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:16.639 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:51:16.639 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:51:16.639 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:16.639 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:16.639 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:16.640 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:16.640 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:16.640 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:16.640 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:16.640 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:16.640 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:16.640 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:16.640 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:16.640 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:16.640 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.640 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:51:16.640 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:51:16.640 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:51:16.641 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:51:16.641 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:51:16.641 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:51:16.641 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:51:16.641 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:51:16.641 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:51:16.641 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:51:16.641 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:51:16.641 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:51:16.641 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:51:16.641 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:51:16.641 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:51:16.641 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:51:16.641 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:51:16.642 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:16.642 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:16.642 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:16.642 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:16.642 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:16.642 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:16.642 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:16.642 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:16.642 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:16.642 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:16.642 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:16.642 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:16.642 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:51:16.642 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.643 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:51:16.643 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.643 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:51:16.643 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:51:16.643 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:16.643 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:16.643 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.643 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:16.643 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:51:16.643 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:16.643 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.643 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.643 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:16.643 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:51:16.644 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:16.644 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:16.644 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:16.644 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.644 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:51:16.644 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:51:16.644 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:16.644 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:16.644 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:16.644 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.644 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.644 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:51:16.644 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:51:16.645 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:16.645 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:16.645 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.645 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:16.645 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:51:16.645 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:16.645 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.645 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.645 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:16.645 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:51:16.645 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:16.645 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:16.645 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:16.645 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.646 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:51:16.646 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:51:16.646 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:16.646 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:16.646 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:16.646 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.646 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.646 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:51:16.646 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:51:16.646 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:16.646 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:16.646 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.646 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:16.646 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:51:16.647 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:16.647 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.647 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.647 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:16.647 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:51:16.647 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:16.647 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:16.647 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:16.647 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.647 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:51:16.647 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:51:16.647 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:16.647 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:16.648 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:16.648 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:16.648 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:16.648 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.648 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:51:16.648 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:51:16.648 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.648 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:16.648 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:16.648 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:51:16.648 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:16.648 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:16.648 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:51:16.648 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.649 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:16.649 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:16.649 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:51:16.649 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:16.649 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:16.649 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:51:16.649 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:16.649 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:16.649 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:16.649 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:51:16.649 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:16.649 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:16.649 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:16.650 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:51:16.650 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:51:16.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.625+0000 7f8e0a9ad640 1 -- 172.21.15.12:0/1654475606 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8df003e210 msgr2=0x7f8df00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:16.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.625+0000 7f8e0a9ad640 1 --2- 172.21.15.12:0/1654475606 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8df003e210 0x7f8df00406d0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f8e040a8360 tx=0x7f8df802e000 comp rx=0 tx=0).stop 2024-09-11T01:51:16.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.625+0000 7f8e0a9ad640 10 monclient: shutdown 2024-09-11T01:51:16.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.625+0000 7f8e0a9ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:16.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.625+0000 7f8e0a9ad640 1 -- 172.21.15.12:0/1654475606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e04138230 msgr2=0x7f8e04134560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:16.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.625+0000 7f8e0a9ad640 1 --2- 172.21.15.12:0/1654475606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e04138230 0x7f8e04134560 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc00c780 tx=0x7f8dfc00cc40 comp rx=0 tx=0).stop 2024-09-11T01:51:16.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.625+0000 7f8e08f4a640 1 -- 172.21.15.12:0/1654475606 reap_dead start 2024-09-11T01:51:16.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.625+0000 7f8e0a9ad640 1 -- 172.21.15.12:0/1654475606 shutdown_connections 2024-09-11T01:51:16.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.625+0000 7f8e0a9ad640 1 --2- 172.21.15.12:0/1654475606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8e04138230 0x7f8e04134560 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.625+0000 7f8e0a9ad640 1 -- 172.21.15.12:0/1654475606 >> 172.21.15.12:0/1654475606 conn(0x7f8e040a7020 msgr2=0x7f8e04136000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:16.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.625+0000 7f8e0a9ad640 1 -- 172.21.15.12:0/1654475606 shutdown_connections 2024-09-11T01:51:16.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.625+0000 7f8e0a9ad640 1 -- 172.21.15.12:0/1654475606 wait complete. 2024-09-11T01:51:16.652 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:51:16.652 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-11T01:51:16.652 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T01:51:16.652 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-11T01:51:16.652 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-11T01:51:16.653 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T01:51:16.653 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:51:16.666 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T01:51:16.666 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:51:16.666 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:51:16.678 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2881757 2024-09-11T01:51:16.691 INFO:tasks.rados.rados.0.smithi071.stdout:3288: finishing copy_from racing read to smithi07134777-34 2024-09-11T01:51:16.692 INFO:tasks.rados.rados.0.smithi071.stdout:3288: finishing copy_from to smithi07134777-34 2024-09-11T01:51:16.692 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 1185 (ObjNum 1121 snap 357 seq_num 1121) dirty exists 2024-09-11T01:51:16.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.805+0000 7f53123a1640 10 monclient: get_monmap_and_config 2024-09-11T01:51:16.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.806+0000 7f53123a1640 10 monclient: build_initial_monmap 2024-09-11T01:51:16.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.806+0000 7f53123a1640 10 monclient: monmap: 2024-09-11T01:51:16.808 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:16.808 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:16.808 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:51:16.806753+0000 2024-09-11T01:51:16.808 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:51:16.806753+0000 2024-09-11T01:51:16.808 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:16.808 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.808 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:51:16.809 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:51:16.809 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:51:16.809 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:51:16.809 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:51:16.809 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.806+0000 7f53123a1640 1 Processor -- start 2024-09-11T01:51:16.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.806+0000 7f53123a1640 1 -- start start 2024-09-11T01:51:16.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.806+0000 7f53123a1640 10 monclient: init 2024-09-11T01:51:16.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f53123a1640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:16.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f53123a1640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-11T01:51:16.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f53123a1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f530c0a4dd0 addr v1:172.21.15.12:6789/0 2024-09-11T01:51:16.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f53123a1640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f530c0a51b0 0x7f530c0a5590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f53123a1640 10 monclient(hunting): picked mon.noname-a con 0x7f530c0a51b0 addr v2:172.21.15.12:3300/0 2024-09-11T01:51:16.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f53123a1640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f530c13d1a0 0x7f530c13f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f53123a1640 10 monclient(hunting): picked mon.noname-c con 0x7f530c13d1a0 addr v2:172.21.15.71:3300/0 2024-09-11T01:51:16.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f53123a1640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f530c0a6ca0 con 0x7f530c0a4dd0 2024-09-11T01:51:16.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f53123a1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f53123a1640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f530c09c810 con 0x7f530c0a51b0 2024-09-11T01:51:16.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f53123a1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f53123a1640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f530c09c530 con 0x7f530c13d1a0 2024-09-11T01:51:16.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f53123a1640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:16.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f53123a1640 10 monclient(hunting): authenticate will time out at 2083.249319s 2024-09-11T01:51:16.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f531139f640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7f530c0a4dd0 legacy=0x7f530c09cf10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:51:16.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f5310b9e640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7f530c0a51b0 msgr2=0x7f530c0a5590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:51:16.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f5310b9e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f530c0a51b0 0x7f530c0a5590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:51:16.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f5311ba0640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f530c13d1a0 0x7f530c13f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f5311ba0640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f530c13d1a0 0x7f530c13f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:39876/0 (socket says 172.21.15.12:39876) 2024-09-11T01:51:16.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f5311ba0640 1 -- 172.21.15.12:0/1420883497 learned_addr learned my addr 172.21.15.12:0/1420883497 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:16.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f5311ba0640 10 monclient(hunting): get_auth_request con 0x7f530c13d1a0 auth_method 0 2024-09-11T01:51:16.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f5311ba0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:16.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f5311ba0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:16.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f5311ba0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:16.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f5311ba0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:16.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f5311ba0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:16.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f5311ba0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:16.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f5311ba0640 10 monclient(hunting): handle_auth_done global_id 234187 payload 274 2024-09-11T01:51:16.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f5311ba0640 1 -- 172.21.15.12:0/1420883497 >> v2:172.21.15.12:3300/0 conn(0x7f530c0a51b0 msgr2=0x7f530c0a5590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:51:16.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f5311ba0640 1 --2- 172.21.15.12:0/1420883497 >> v2:172.21.15.12:3300/0 conn(0x7f530c0a51b0 0x7f530c0a5590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f5311ba0640 1 -- 172.21.15.12:0/1420883497 >> v1:172.21.15.12:6789/0 conn(0x7f530c0a4dd0 legacy=0x7f530c09cf10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:51:16.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f5311ba0640 10 monclient: _finish_hunting 0 2024-09-11T01:51:16.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f5311ba0640 1 monclient: found mon.noname-c 2024-09-11T01:51:16.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f5311ba0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:16.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f5311ba0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:51:16.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.807+0000 7f5311ba0640 1 -- 172.21.15.12:0/1420883497 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f530c09c250 con 0x7f530c13d1a0 2024-09-11T01:51:16.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.810+0000 7f5311ba0640 1 -- 172.21.15.12:0/1420883497 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f530c13d1a0 msgr2=0x7f530c13f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:16.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.810+0000 7f5311ba0640 1 --2- 172.21.15.12:0/1420883497 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f530c13d1a0 0x7f530c13f590 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f530c09bf70 tx=0x7f530402a610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d829f15f72a71253 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:16.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.811+0000 7f52fffff640 1 -- 172.21.15.12:0/1420883497 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5304036070 con 0x7f530c13d1a0 2024-09-11T01:51:16.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.811+0000 7f52fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:16.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.811+0000 7f52fffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:51:16.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.811+0000 7f52fffff640 10 monclient: dump: 2024-09-11T01:51:16.828 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:16.828 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:16.828 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.829 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.829 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:16.829 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.829 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:16.829 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:16.829 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:16.829 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.811+0000 7f52fffff640 10 monclient: _finish_auth 0 2024-09-11T01:51:16.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.811+0000 7f52fffff640 10 monclient: _check_auth_tickets 2024-09-11T01:51:16.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.811+0000 7f52fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:16.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.811+0000 7f52fffff640 1 -- 172.21.15.12:0/1420883497 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5304007800 con 0x7f530c13d1a0 2024-09-11T01:51:16.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.811+0000 7f52fffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:16.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.811+0000 7f52fffff640 1 -- 172.21.15.12:0/1420883497 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5304007bd0 con 0x7f530c13d1a0 2024-09-11T01:51:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.811+0000 7f53123a1640 5 monclient: authenticate success, global_id 234187 2024-09-11T01:51:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.811+0000 7f53123a1640 10 monclient: get_monmap_and_config success 2024-09-11T01:51:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.811+0000 7f52fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.811+0000 7f53123a1640 4 set_mon_vals no callback set 2024-09-11T01:51:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.811+0000 7f52fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:51:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.811+0000 7f52fffff640 10 monclient: dump: 2024-09-11T01:51:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:16.830 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:16.831 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.811+0000 7f53123a1640 10 monclient: shutdown 2024-09-11T01:51:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.811+0000 7f53123a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.811+0000 7f53123a1640 1 -- 172.21.15.12:0/1420883497 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f530c13d1a0 msgr2=0x7f530c13f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.811+0000 7f53123a1640 1 --2- 172.21.15.12:0/1420883497 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f530c13d1a0 0x7f530c13f590 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f530c09bf70 tx=0x7f530402a610 comp rx=0 tx=0).stop 2024-09-11T01:51:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.811+0000 7f53123a1640 1 -- 172.21.15.12:0/1420883497 shutdown_connections 2024-09-11T01:51:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.811+0000 7f53123a1640 1 --2- 172.21.15.12:0/1420883497 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f530c13d1a0 0x7f530c13f590 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.811+0000 7f53123a1640 1 --2- 172.21.15.12:0/1420883497 >> v2:172.21.15.12:3300/0 conn(0x7f530c0a51b0 0x7f530c0a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.811+0000 7f53123a1640 1 -- 172.21.15.12:0/1420883497 >> 172.21.15.12:0/1420883497 conn(0x7f530c0aaf90 msgr2=0x7f530c0a6ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.811+0000 7f53123a1640 1 -- 172.21.15.12:0/1420883497 shutdown_connections 2024-09-11T01:51:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.811+0000 7f53123a1640 1 -- 172.21.15.12:0/1420883497 wait complete. 2024-09-11T01:51:16.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.812+0000 7f53123a1640 10 monclient: build_initial_monmap 2024-09-11T01:51:16.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.812+0000 7f53123a1640 10 monclient: monmap: 2024-09-11T01:51:16.832 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:16.832 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:51:16.832 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:51:16.832 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:51:16.832 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:16.832 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.832 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:51:16.832 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:51:16.832 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:51:16.832 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.812+0000 7f53123a1640 1 Processor -- start 2024-09-11T01:51:16.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.812+0000 7f53123a1640 1 -- start start 2024-09-11T01:51:16.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.812+0000 7f53123a1640 10 monclient: init 2024-09-11T01:51:16.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f53123a1640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:16.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f53123a1640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:51:16.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f53123a1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c0a4dd0 0x7f530c134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f53123a1640 10 monclient(hunting): picked mon.noname-a con 0x7f530c0a4dd0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:51:16.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f53123a1640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f530c0a51b0 0x7f530c1353f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f53123a1640 10 monclient(hunting): picked mon.noname-c con 0x7f530c0a51b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:16.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f53123a1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f530c13d1a0 0x7f530c1ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f531139f640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c0a4dd0 msgr2=0x7f530c134eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:51:16.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f53123a1640 10 monclient(hunting): picked mon.noname-b con 0x7f530c13d1a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:51:16.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f53123a1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f53123a1640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f530c0a79d0 con 0x7f530c0a4dd0 2024-09-11T01:51:16.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f531139f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c0a4dd0 0x7f530c134eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:51:16.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f53123a1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f53123a1640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f530c0a7850 con 0x7f530c13d1a0 2024-09-11T01:51:16.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f53123a1640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:16.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f53123a1640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f530c13fad0 con 0x7f530c0a51b0 2024-09-11T01:51:16.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f53123a1640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:16.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f53123a1640 10 monclient(hunting): authenticate will time out at 2083.255245s 2024-09-11T01:51:16.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f5310b9e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f530c0a51b0 0x7f530c1353f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f5310b9e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f530c0a51b0 0x7f530c1353f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51556/0 (socket says 172.21.15.12:51556) 2024-09-11T01:51:16.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f5310b9e640 1 -- 172.21.15.12:0/468603904 learned_addr learned my addr 172.21.15.12:0/468603904 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:16.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f5310b9e640 10 monclient(hunting): get_auth_request con 0x7f530c0a51b0 auth_method 0 2024-09-11T01:51:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f5310b9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f5310b9e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f5310b9e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f5310b9e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f5310b9e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f5310b9e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f5311ba0640 1 --2- 172.21.15.12:0/468603904 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f530c13d1a0 0x7f530c1ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f5311ba0640 10 monclient(hunting): get_auth_request con 0x7f530c13d1a0 auth_method 0 2024-09-11T01:51:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f5311ba0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f5311ba0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f5311ba0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f5310b9e640 10 monclient(hunting): handle_auth_done global_id 244157 payload 824 2024-09-11T01:51:16.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f5310b9e640 1 -- 172.21.15.12:0/468603904 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f530c13d1a0 msgr2=0x7f530c1ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f5310b9e640 1 --2- 172.21.15.12:0/468603904 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f530c13d1a0 0x7f530c1ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f5310b9e640 1 -- 172.21.15.12:0/468603904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c0a4dd0 msgr2=0x7f530c134eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:51:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f5310b9e640 1 --2- 172.21.15.12:0/468603904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c0a4dd0 0x7f530c134eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f5310b9e640 10 monclient: _finish_hunting 0 2024-09-11T01:51:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f5310b9e640 1 monclient: found mon.noname-c 2024-09-11T01:51:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f5310b9e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f5310b9e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:51:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.813+0000 7f5310b9e640 1 -- 172.21.15.12:0/468603904 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f530c13d6c0 con 0x7f530c0a51b0 2024-09-11T01:51:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.815+0000 7f5310b9e640 1 -- 172.21.15.12:0/468603904 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f530c0a51b0 msgr2=0x7f530c1353f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.815+0000 7f5310b9e640 1 --2- 172.21.15.12:0/468603904 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f530c0a51b0 0x7f530c1353f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f530800b580 tx=0x7f530800ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.816+0000 7f52fdffb640 1 -- 172.21.15.12:0/468603904 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53080050f0 con 0x7f530c0a51b0 2024-09-11T01:51:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.816+0000 7f52fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:16.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.816+0000 7f52fdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:51:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.816+0000 7f52fdffb640 10 monclient: dump: 2024-09-11T01:51:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:16.837 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.816+0000 7f52fdffb640 10 monclient: _finish_auth 0 2024-09-11T01:51:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.816+0000 7f52fdffb640 10 monclient: _check_auth_tickets 2024-09-11T01:51:16.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.816+0000 7f52fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:16.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.816+0000 7f53123a1640 5 monclient: authenticate success, global_id 244157 2024-09-11T01:51:16.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.816+0000 7f52fdffb640 1 -- 172.21.15.12:0/468603904 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5308015070 con 0x7f530c0a51b0 2024-09-11T01:51:16.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.816+0000 7f53123a1640 10 monclient: _renew_subs 2024-09-11T01:51:16.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.816+0000 7f53123a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:16.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.816+0000 7f53123a1640 1 -- 172.21.15.12:0/468603904 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f530c0a2140 con 0x7f530c0a51b0 2024-09-11T01:51:16.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.816+0000 7f52fdffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:16.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.816+0000 7f52fdffb640 1 -- 172.21.15.12:0/468603904 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53080054d0 con 0x7f530c0a51b0 2024-09-11T01:51:16.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.816+0000 7f52fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:16.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.816+0000 7f52fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:16.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.816+0000 7f52fdffb640 10 monclient: dump: 2024-09-11T01:51:16.838 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:16.838 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:16.838 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.838 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:16.839 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.816+0000 7f52ff7fe640 4 set_mon_vals no callback set 2024-09-11T01:51:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.816+0000 7f53123a1640 10 monclient: _renew_subs 2024-09-11T01:51:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.816+0000 7f53123a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.816+0000 7f53123a1640 1 -- 172.21.15.12:0/468603904 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f530c135930 con 0x7f530c0a51b0 2024-09-11T01:51:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.817+0000 7f52fdffb640 1 -- 172.21.15.12:0/468603904 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f53080056b0 con 0x7f530c0a51b0 2024-09-11T01:51:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.817+0000 7f53123a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:16.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.817+0000 7f53123a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:16.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.817+0000 7f53123a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:16.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.817+0000 7f53123a1640 1 -- 172.21.15.12:0/468603904 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f530c0a5590 con 0x7f530c0a51b0 2024-09-11T01:51:16.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.817+0000 7f52fdffb640 1 --2- 172.21.15.12:0/468603904 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f52f003e1c0 0x7f52f0040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:16.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.817+0000 7f52fdffb640 1 -- 172.21.15.12:0/468603904 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(735..735 src has 676..735) ==== 4799+0+0 (secure 0 0 0) 0x7f5308052c50 con 0x7f530c0a51b0 2024-09-11T01:51:16.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.817+0000 7f531139f640 1 --2- 172.21.15.12:0/468603904 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f52f003e1c0 0x7f52f0040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:16.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.817+0000 7f531139f640 10 monclient: get_auth_request con 0x7f52f003e1c0 auth_method 0 2024-09-11T01:51:16.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.820+0000 7f531139f640 1 --2- 172.21.15.12:0/468603904 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f52f003e1c0 0x7f52f0040680 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f5300003980 tx=0x7f5300006d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:16.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.822+0000 7f52fdffb640 1 -- 172.21.15.12:0/468603904 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5308051640 con 0x7f530c0a51b0 2024-09-11T01:51:16.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.822+0000 7f52fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:16.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:16.822+0000 7f52fdffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:51:17.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:17.080+0000 7f53123a1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:51:17.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:17.080+0000 7f53123a1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:51:17.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:17.080+0000 7f53123a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:17.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:17.080+0000 7f53123a1640 1 -- 172.21.15.12:0/468603904 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f530c0a5e50 con 0x7f530c0a51b0 2024-09-11T01:51:17.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:17.081+0000 7f52fdffb640 1 -- 172.21.15.12:0/468603904 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f530801a460 con 0x7f530c0a51b0 2024-09-11T01:51:17.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:17.081+0000 7f52fdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:51:17.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:17.081+0000 7f52fdffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:51:17.083 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":208,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:51:17.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:17.083+0000 7f53123a1640 1 -- 172.21.15.12:0/468603904 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f52f003e1c0 msgr2=0x7f52f0040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:17.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:17.083+0000 7f53123a1640 1 --2- 172.21.15.12:0/468603904 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f52f003e1c0 0x7f52f0040680 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f5300003980 tx=0x7f5300006d30 comp rx=0 tx=0).stop 2024-09-11T01:51:17.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:17.083+0000 7f53123a1640 10 monclient: shutdown 2024-09-11T01:51:17.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:17.083+0000 7f53123a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:17.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:17.083+0000 7f53123a1640 1 -- 172.21.15.12:0/468603904 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f530c0a51b0 msgr2=0x7f530c1353f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:17.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:17.083+0000 7f53123a1640 1 --2- 172.21.15.12:0/468603904 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f530c0a51b0 0x7f530c1353f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f530800b580 tx=0x7f530800ba40 comp rx=0 tx=0).stop 2024-09-11T01:51:17.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:17.083+0000 7f53123a1640 1 -- 172.21.15.12:0/468603904 shutdown_connections 2024-09-11T01:51:17.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:17.083+0000 7f53123a1640 1 --2- 172.21.15.12:0/468603904 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f530c13d1a0 0x7f530c1ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:17.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:17.083+0000 7f53123a1640 1 --2- 172.21.15.12:0/468603904 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f530c0a51b0 0x7f530c1353f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:17.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:17.083+0000 7f53123a1640 1 --2- 172.21.15.12:0/468603904 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f52f003e1c0 0x7f52f0040680 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:17.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:17.083+0000 7f53123a1640 1 --2- 172.21.15.12:0/468603904 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c0a4dd0 0x7f530c134eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:17.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:17.083+0000 7f53123a1640 1 -- 172.21.15.12:0/468603904 >> 172.21.15.12:0/468603904 conn(0x7f530c0aaf90 msgr2=0x7f530c13c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:17.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:17.083+0000 7f53123a1640 1 -- 172.21.15.12:0/468603904 shutdown_connections 2024-09-11T01:51:17.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:17.083+0000 7f53123a1640 1 -- 172.21.15.12:0/468603904 wait complete. 2024-09-11T01:51:20.096 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:51:20.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.208+0000 7f6f14b05640 10 monclient: get_monmap_and_config 2024-09-11T01:51:20.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.208+0000 7f6f14b05640 10 monclient: build_initial_monmap 2024-09-11T01:51:20.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.208+0000 7f6f14b05640 10 monclient: monmap: 2024-09-11T01:51:20.210 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:20.210 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:20.210 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:51:20.209503+0000 2024-09-11T01:51:20.210 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:51:20.209503+0000 2024-09-11T01:51:20.211 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:20.211 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:20.211 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:51:20.211 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:51:20.211 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:51:20.211 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:51:20.211 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:51:20.211 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:20.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.209+0000 7f6f14b05640 1 Processor -- start 2024-09-11T01:51:20.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.209+0000 7f6f14b05640 1 -- start start 2024-09-11T01:51:20.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.209+0000 7f6f14b05640 10 monclient: init 2024-09-11T01:51:20.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f14b05640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:20.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f14b05640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-11T01:51:20.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f14b05640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f100a2e90 0x7f6f1006dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:20.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f14b05640 10 monclient(hunting): picked mon.noname-b con 0x7f6f100a2e90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:20.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f14b05640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6f100a3270 0x7f6f100a3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:20.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f14b05640 10 monclient(hunting): picked mon.noname-a con 0x7f6f100a3270 addr v2:172.21.15.12:3300/0 2024-09-11T01:51:20.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f14b05640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6f10138040 addr v1:172.21.15.71:6789/0 2024-09-11T01:51:20.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f14b05640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f100a46c0 con 0x7f6f10138040 2024-09-11T01:51:20.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f14b05640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:20.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f14b05640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f6f100a4900 con 0x7f6f100a3270 2024-09-11T01:51:20.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f14b05640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:20.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f14b05640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6f1006ee50 con 0x7f6f100a2e90 2024-09-11T01:51:20.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f14b05640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:20.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f14b05640 10 monclient(hunting): authenticate will time out at 2086.652158s 2024-09-11T01:51:20.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f0dd9b640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7f6f100a3270 msgr2=0x7f6f100a3650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:51:20.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f0dd9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f6f100a3270 0x7f6f100a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:51:20.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f0e59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f100a2e90 0x7f6f1006dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:20.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f0e59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f100a2e90 0x7f6f1006dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51570/0 (socket says 172.21.15.12:51570) 2024-09-11T01:51:20.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f0e59c640 1 -- 172.21.15.12:0/1603319910 learned_addr learned my addr 172.21.15.12:0/1603319910 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:20.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f0e59c640 10 monclient(hunting): get_auth_request con 0x7f6f100a2e90 auth_method 0 2024-09-11T01:51:20.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:20.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:20.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:20.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:20.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:20.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:20.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f0e59c640 10 monclient(hunting): handle_auth_done global_id 244160 payload 274 2024-09-11T01:51:20.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f0e59c640 1 -- 172.21.15.12:0/1603319910 >> v2:172.21.15.12:3300/0 conn(0x7f6f100a3270 msgr2=0x7f6f100a3650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:51:20.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f0e59c640 1 --2- 172.21.15.12:0/1603319910 >> v2:172.21.15.12:3300/0 conn(0x7f6f100a3270 0x7f6f100a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:20.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f0e59c640 1 -- 172.21.15.12:0/1603319910 >> v1:172.21.15.71:6789/0 conn(0x7f6f10138040 legacy=0x7f6f10138460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:20.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f0e59c640 10 monclient: _finish_hunting 0 2024-09-11T01:51:20.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f0e59c640 1 monclient: found mon.noname-b 2024-09-11T01:51:20.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f0e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:20.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f0e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:51:20.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.210+0000 7f6f0e59c640 1 -- 172.21.15.12:0/1603319910 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f1006eb70 con 0x7f6f100a2e90 2024-09-11T01:51:20.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.213+0000 7f6f0e59c640 1 -- 172.21.15.12:0/1603319910 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f100a2e90 msgr2=0x7f6f1006dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:20.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.213+0000 7f6f0e59c640 1 --2- 172.21.15.12:0/1603319910 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f100a2e90 0x7f6f1006dc60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6f100a4a80 tx=0x7f6f0002a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b755376d1a0195c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:20.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.213+0000 7f6f0d59a640 1 -- 172.21.15.12:0/1603319910 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f00037070 con 0x7f6f100a2e90 2024-09-11T01:51:20.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.213+0000 7f6f0d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:20.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.213+0000 7f6f0d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:51:20.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.213+0000 7f6f0d59a640 10 monclient: dump: 2024-09-11T01:51:20.222 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:20.222 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:20.222 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:20.223 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:20.223 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:20.223 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:20.223 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:20.223 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:20.223 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:20.223 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:20.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.213+0000 7f6f0d59a640 10 monclient: _finish_auth 0 2024-09-11T01:51:20.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.213+0000 7f6f0d59a640 10 monclient: _check_auth_tickets 2024-09-11T01:51:20.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.213+0000 7f6f0d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:20.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.213+0000 7f6f0d59a640 1 -- 172.21.15.12:0/1603319910 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f00032640 con 0x7f6f100a2e90 2024-09-11T01:51:20.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.213+0000 7f6f14b05640 5 monclient: authenticate success, global_id 244160 2024-09-11T01:51:20.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.213+0000 7f6f14b05640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:51:20.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.213+0000 7f6f0d59a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:20.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.213+0000 7f6f0d59a640 1 -- 172.21.15.12:0/1603319910 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f00032a10 con 0x7f6f100a2e90 2024-09-11T01:51:20.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.213+0000 7f6f14b05640 10 monclient: get_monmap_and_config success 2024-09-11T01:51:20.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.213+0000 7f6f14b05640 4 set_mon_vals no callback set 2024-09-11T01:51:20.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.213+0000 7f6f0d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:20.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.213+0000 7f6f0d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:20.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.213+0000 7f6f0d59a640 10 monclient: dump: 2024-09-11T01:51:20.224 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:20.224 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:20.224 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:20.224 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:20.224 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:20.224 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:20.225 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:20.225 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:20.225 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:20.225 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:20.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.214+0000 7f6f14b05640 10 monclient: shutdown 2024-09-11T01:51:20.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.214+0000 7f6f14b05640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:20.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.214+0000 7f6f14b05640 1 -- 172.21.15.12:0/1603319910 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f100a2e90 msgr2=0x7f6f1006dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:20.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.214+0000 7f6f14b05640 1 --2- 172.21.15.12:0/1603319910 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f100a2e90 0x7f6f1006dc60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6f100a4a80 tx=0x7f6f0002a0a0 comp rx=0 tx=0).stop 2024-09-11T01:51:20.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.214+0000 7f6f14b05640 1 -- 172.21.15.12:0/1603319910 shutdown_connections 2024-09-11T01:51:20.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.214+0000 7f6f14b05640 1 --2- 172.21.15.12:0/1603319910 >> v2:172.21.15.12:3300/0 conn(0x7f6f100a3270 0x7f6f100a3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:20.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.214+0000 7f6f14b05640 1 --2- 172.21.15.12:0/1603319910 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f100a2e90 0x7f6f1006dc60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:20.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.214+0000 7f6f14b05640 1 -- 172.21.15.12:0/1603319910 >> 172.21.15.12:0/1603319910 conn(0x7f6f100a8490 msgr2=0x7f6f100a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:20.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.214+0000 7f6f14b05640 1 -- 172.21.15.12:0/1603319910 shutdown_connections 2024-09-11T01:51:20.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.214+0000 7f6f14b05640 1 -- 172.21.15.12:0/1603319910 wait complete. 2024-09-11T01:51:20.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.215+0000 7f6f14b05640 10 monclient: build_initial_monmap 2024-09-11T01:51:20.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.215+0000 7f6f14b05640 10 monclient: monmap: 2024-09-11T01:51:20.226 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:20.226 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:51:20.226 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:51:20.226 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:51:20.226 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:20.226 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:20.226 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:51:20.226 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:51:20.226 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:51:20.226 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:20.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.215+0000 7f6f14b05640 1 Processor -- start 2024-09-11T01:51:20.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.215+0000 7f6f14b05640 1 -- start start 2024-09-11T01:51:20.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.215+0000 7f6f14b05640 10 monclient: init 2024-09-11T01:51:20.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f14b05640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:20.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f14b05640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:51:20.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f14b05640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f100a2e90 0x7f6f101cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:20.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f14b05640 10 monclient(hunting): picked mon.noname-a con 0x7f6f100a2e90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:51:20.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f14b05640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f100a3270 0x7f6f101cda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:20.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f14b05640 10 monclient(hunting): picked mon.noname-c con 0x7f6f100a3270 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:20.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f14b05640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f10138040 0x7f6f10124bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:20.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0e59c640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f100a2e90 msgr2=0x7f6f101cd4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:51:20.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f14b05640 10 monclient(hunting): picked mon.noname-b con 0x7f6f10138040 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:51:20.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f14b05640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:20.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f14b05640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6f100a3b90 con 0x7f6f100a2e90 2024-09-11T01:51:20.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0e59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f100a2e90 0x7f6f101cd4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:51:20.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f14b05640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:20.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f14b05640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1006fba0 con 0x7f6f10138040 2024-09-11T01:51:20.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f14b05640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:20.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f14b05640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6f1009f6b0 con 0x7f6f100a3270 2024-09-11T01:51:20.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f14b05640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:20.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f14b05640 10 monclient(hunting): authenticate will time out at 2086.658381s 2024-09-11T01:51:20.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0dd9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f100a3270 0x7f6f101cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:20.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0dd9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f100a3270 0x7f6f101cda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51576/0 (socket says 172.21.15.12:51576) 2024-09-11T01:51:20.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0dd9b640 1 -- 172.21.15.12:0/1334035007 learned_addr learned my addr 172.21.15.12:0/1334035007 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:20.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0dd9b640 10 monclient(hunting): get_auth_request con 0x7f6f100a3270 auth_method 0 2024-09-11T01:51:20.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:20.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:20.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:20.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:20.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:20.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:20.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0ed9d640 1 --2- 172.21.15.12:0/1334035007 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f10138040 0x7f6f10124bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:20.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0ed9d640 10 monclient(hunting): get_auth_request con 0x7f6f10138040 auth_method 0 2024-09-11T01:51:20.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:20.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:20.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:20.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:20.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:20.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:20.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0dd9b640 10 monclient(hunting): handle_auth_done global_id 244163 payload 824 2024-09-11T01:51:20.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0dd9b640 1 -- 172.21.15.12:0/1334035007 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f10138040 msgr2=0x7f6f10124bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:20.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0dd9b640 1 --2- 172.21.15.12:0/1334035007 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f10138040 0x7f6f10124bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:20.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0dd9b640 1 -- 172.21.15.12:0/1334035007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f100a2e90 msgr2=0x7f6f101cd4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:51:20.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0dd9b640 1 --2- 172.21.15.12:0/1334035007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f100a2e90 0x7f6f101cd4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:20.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0dd9b640 10 monclient: _finish_hunting 0 2024-09-11T01:51:20.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0dd9b640 1 monclient: found mon.noname-c 2024-09-11T01:51:20.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:20.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:51:20.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.216+0000 7f6f0dd9b640 1 -- 172.21.15.12:0/1334035007 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f101265c0 con 0x7f6f100a3270 2024-09-11T01:51:20.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.219+0000 7f6f0dd9b640 1 -- 172.21.15.12:0/1334035007 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f100a3270 msgr2=0x7f6f101cda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:20.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.219+0000 7f6f0dd9b640 1 --2- 172.21.15.12:0/1334035007 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f100a3270 0x7f6f101cda20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6ef800b550 tx=0x7f6ef800ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:20.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.219+0000 7f6f06ffd640 1 -- 172.21.15.12:0/1334035007 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8005070 con 0x7f6f100a3270 2024-09-11T01:51:20.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.219+0000 7f6f06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:20.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.219+0000 7f6f06ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:51:20.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.219+0000 7f6f06ffd640 10 monclient: dump: 2024-09-11T01:51:20.231 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:20.231 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:20.231 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:20.231 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:20.231 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:20.231 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:20.231 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:20.231 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:20.231 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:20.231 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:20.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.219+0000 7f6f06ffd640 10 monclient: _finish_auth 0 2024-09-11T01:51:20.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.219+0000 7f6f06ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:51:20.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.219+0000 7f6f06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:20.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.219+0000 7f6f06ffd640 1 -- 172.21.15.12:0/1334035007 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ef8015070 con 0x7f6f100a3270 2024-09-11T01:51:20.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.219+0000 7f6f06ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:20.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.219+0000 7f6f06ffd640 1 -- 172.21.15.12:0/1334035007 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8005450 con 0x7f6f100a3270 2024-09-11T01:51:20.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.219+0000 7f6f14b05640 5 monclient: authenticate success, global_id 244163 2024-09-11T01:51:20.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.219+0000 7f6f14b05640 10 monclient: _renew_subs 2024-09-11T01:51:20.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.219+0000 7f6f14b05640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:20.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.219+0000 7f6f14b05640 1 -- 172.21.15.12:0/1334035007 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f10073c20 con 0x7f6f100a3270 2024-09-11T01:51:20.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.219+0000 7f6f06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:20.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.219+0000 7f6f0cd99640 4 set_mon_vals no callback set 2024-09-11T01:51:20.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.219+0000 7f6f06ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:20.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.219+0000 7f6f06ffd640 10 monclient: dump: 2024-09-11T01:51:20.233 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:20.233 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:20.233 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:20.233 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:20.233 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:20.233 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:20.233 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:20.234 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:20.234 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:20.234 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:20.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.219+0000 7f6f14b05640 10 monclient: _renew_subs 2024-09-11T01:51:20.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.219+0000 7f6f14b05640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:20.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.219+0000 7f6f14b05640 1 -- 172.21.15.12:0/1334035007 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f101cdf60 con 0x7f6f100a3270 2024-09-11T01:51:20.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.220+0000 7f6f06ffd640 1 -- 172.21.15.12:0/1334035007 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f6ef8005630 con 0x7f6f100a3270 2024-09-11T01:51:20.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.220+0000 7f6f14b05640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:20.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.220+0000 7f6f14b05640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:20.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.220+0000 7f6f14b05640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:20.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.220+0000 7f6f14b05640 1 -- 172.21.15.12:0/1334035007 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f10074ac0 con 0x7f6f100a3270 2024-09-11T01:51:20.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.220+0000 7f6f06ffd640 1 --2- 172.21.15.12:0/1334035007 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6ef003e1f0 0x7f6ef00406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:20.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.220+0000 7f6f06ffd640 1 -- 172.21.15.12:0/1334035007 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(735..735 src has 676..735) ==== 4799+0+0 (secure 0 0 0) 0x7f6ef8052ae0 con 0x7f6f100a3270 2024-09-11T01:51:20.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.220+0000 7f6f0e59c640 1 --2- 172.21.15.12:0/1334035007 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6ef003e1f0 0x7f6ef00406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:20.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.220+0000 7f6f0e59c640 10 monclient: get_auth_request con 0x7f6ef003e1f0 auth_method 0 2024-09-11T01:51:20.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.223+0000 7f6f0e59c640 1 --2- 172.21.15.12:0/1334035007 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6ef003e1f0 0x7f6ef00406b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f6f1006e860 tx=0x7f6f0002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:20.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.225+0000 7f6f06ffd640 1 -- 172.21.15.12:0/1334035007 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6f10074ac0 con 0x7f6f100a3270 2024-09-11T01:51:20.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.225+0000 7f6f06ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:20.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.225+0000 7f6f06ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:51:20.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.480+0000 7f6f14b05640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:51:20.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.480+0000 7f6f14b05640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:51:20.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.480+0000 7f6f14b05640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:20.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.480+0000 7f6f14b05640 1 -- 172.21.15.12:0/1334035007 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6f10138c90 con 0x7f6f100a3270 2024-09-11T01:51:20.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.480+0000 7f6f06ffd640 1 -- 172.21.15.12:0/1334035007 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f6f10073c20 con 0x7f6f100a3270 2024-09-11T01:51:20.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.480+0000 7f6f06ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:51:20.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.480+0000 7f6f06ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:51:20.483 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":208,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:51:20.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.482+0000 7f6f14b05640 1 -- 172.21.15.12:0/1334035007 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6ef003e1f0 msgr2=0x7f6ef00406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:20.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.482+0000 7f6f14b05640 1 --2- 172.21.15.12:0/1334035007 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6ef003e1f0 0x7f6ef00406b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f6f1006e860 tx=0x7f6f0002e000 comp rx=0 tx=0).stop 2024-09-11T01:51:20.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.482+0000 7f6f14b05640 10 monclient: shutdown 2024-09-11T01:51:20.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.482+0000 7f6f14b05640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:20.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.482+0000 7f6f14b05640 1 -- 172.21.15.12:0/1334035007 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f100a3270 msgr2=0x7f6f101cda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:20.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.482+0000 7f6f14b05640 1 --2- 172.21.15.12:0/1334035007 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f100a3270 0x7f6f101cda20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6ef800b550 tx=0x7f6ef800ba10 comp rx=0 tx=0).stop 2024-09-11T01:51:20.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.482+0000 7f6f14b05640 1 -- 172.21.15.12:0/1334035007 shutdown_connections 2024-09-11T01:51:20.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.482+0000 7f6f14b05640 1 --2- 172.21.15.12:0/1334035007 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f10138040 0x7f6f10124bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:20.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.482+0000 7f6f14b05640 1 --2- 172.21.15.12:0/1334035007 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f100a3270 0x7f6f101cda20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:20.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.482+0000 7f6f14b05640 1 --2- 172.21.15.12:0/1334035007 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6ef003e1f0 0x7f6ef00406b0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:20.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.482+0000 7f6f14b05640 1 --2- 172.21.15.12:0/1334035007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f100a2e90 0x7f6f101cd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:20.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.482+0000 7f6f14b05640 1 -- 172.21.15.12:0/1334035007 >> 172.21.15.12:0/1334035007 conn(0x7f6f100a8490 msgr2=0x7f6f100a7be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:20.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.482+0000 7f6f14b05640 1 -- 172.21.15.12:0/1334035007 shutdown_connections 2024-09-11T01:51:20.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:20.482+0000 7f6f14b05640 1 -- 172.21.15.12:0/1334035007 wait complete. 2024-09-11T01:51:23.495 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:51:23.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.607+0000 7f600ad89640 10 monclient: get_monmap_and_config 2024-09-11T01:51:23.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.607+0000 7f600ad89640 10 monclient: build_initial_monmap 2024-09-11T01:51:23.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.607+0000 7f600ad89640 10 monclient: monmap: 2024-09-11T01:51:23.609 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:23.609 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:23.609 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:51:23.607956+0000 2024-09-11T01:51:23.609 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:51:23.607956+0000 2024-09-11T01:51:23.609 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:23.609 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:23.609 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:51:23.609 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:51:23.609 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:51:23.609 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:51:23.609 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:51:23.610 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:23.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.608+0000 7f600ad89640 1 Processor -- start 2024-09-11T01:51:23.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.608+0000 7f600ad89640 1 -- start start 2024-09-11T01:51:23.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.608+0000 7f600ad89640 10 monclient: init 2024-09-11T01:51:23.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.608+0000 7f600ad89640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:23.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.608+0000 7f600ad89640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-11T01:51:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.608+0000 7f600ad89640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6004138230 addr v1:172.21.15.71:6789/0 2024-09-11T01:51:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.608+0000 7f600ad89640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f600409d030 addr v1:172.21.15.12:6789/0 2024-09-11T01:51:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.608+0000 7f600ad89640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f600409d460 0x7f60041407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.608+0000 7f600ad89640 10 monclient(hunting): picked mon.noname-b con 0x7f600409d460 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.608+0000 7f600ad89640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60040a6e30 con 0x7f600409d030 2024-09-11T01:51:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.608+0000 7f600ad89640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f600409ba20 con 0x7f6004138230 2024-09-11T01:51:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.608+0000 7f600ad89640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.608+0000 7f600ad89640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f600409c9c0 con 0x7f600409d460 2024-09-11T01:51:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.608+0000 7f600ad89640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.608+0000 7f600ad89640 10 monclient(hunting): authenticate will time out at 2090.050645s 2024-09-11T01:51:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.608+0000 7f6003fff640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7f600409d030 legacy=0x7f600413d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:51:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.608+0000 7f6009326640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f600409d460 0x7f60041407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.608+0000 7f6009326640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f600409d460 0x7f60041407c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51584/0 (socket says 172.21.15.12:51584) 2024-09-11T01:51:23.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.608+0000 7f6009326640 1 -- 172.21.15.12:0/630106839 learned_addr learned my addr 172.21.15.12:0/630106839 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:23.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.608+0000 7f6009326640 10 monclient(hunting): get_auth_request con 0x7f600409d460 auth_method 0 2024-09-11T01:51:23.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.608+0000 7f6009326640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:23.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.608+0000 7f6009326640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:23.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.608+0000 7f6009326640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:23.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.608+0000 7f6009326640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:23.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.608+0000 7f6009326640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:23.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.608+0000 7f6009326640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:23.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.609+0000 7f6009326640 10 monclient(hunting): handle_auth_done global_id 244166 payload 274 2024-09-11T01:51:23.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.609+0000 7f6009326640 1 -- 172.21.15.12:0/630106839 >> v1:172.21.15.71:6789/0 conn(0x7f6004138230 legacy=0x7f6004138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:23.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.609+0000 7f6009326640 1 -- 172.21.15.12:0/630106839 >> v1:172.21.15.12:6789/0 conn(0x7f600409d030 legacy=0x7f600413d090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:51:23.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.609+0000 7f6009326640 10 monclient: _finish_hunting 0 2024-09-11T01:51:23.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.609+0000 7f6009326640 1 monclient: found mon.noname-b 2024-09-11T01:51:23.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.609+0000 7f6009326640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:23.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.609+0000 7f6009326640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:51:23.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.609+0000 7f6009326640 1 -- 172.21.15.12:0/630106839 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60040a66a0 con 0x7f600409d460 2024-09-11T01:51:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.611+0000 7f6009326640 1 -- 172.21.15.12:0/630106839 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f600409d460 msgr2=0x7f60041407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f6009326640 1 --2- 172.21.15.12:0/630106839 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f600409d460 0x7f60041407c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f60040a17e0 tx=0x7f5ffc02a530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64a551d130a948e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f60037fe640 1 -- 172.21.15.12:0/630106839 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc036070 con 0x7f600409d460 2024-09-11T01:51:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f60037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f60037fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:51:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f60037fe640 10 monclient: dump: 2024-09-11T01:51:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:23.619 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:23.620 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f60037fe640 10 monclient: _finish_auth 0 2024-09-11T01:51:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f60037fe640 10 monclient: _check_auth_tickets 2024-09-11T01:51:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f60037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f60037fe640 1 -- 172.21.15.12:0/630106839 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ffc007800 con 0x7f600409d460 2024-09-11T01:51:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f600ad89640 5 monclient: authenticate success, global_id 244166 2024-09-11T01:51:23.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f600ad89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:51:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f60037fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f600ad89640 10 monclient: get_monmap_and_config success 2024-09-11T01:51:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f60037fe640 1 -- 172.21.15.12:0/630106839 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc007bd0 con 0x7f600409d460 2024-09-11T01:51:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f600ad89640 4 set_mon_vals no callback set 2024-09-11T01:51:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f60037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f60037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f60037fe640 10 monclient: dump: 2024-09-11T01:51:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:23.621 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:23.622 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f600ad89640 10 monclient: shutdown 2024-09-11T01:51:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f600ad89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f600ad89640 1 -- 172.21.15.12:0/630106839 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f600409d460 msgr2=0x7f60041407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f600ad89640 1 --2- 172.21.15.12:0/630106839 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f600409d460 0x7f60041407c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f60040a17e0 tx=0x7f5ffc02a530 comp rx=0 tx=0).stop 2024-09-11T01:51:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f600ad89640 1 -- 172.21.15.12:0/630106839 shutdown_connections 2024-09-11T01:51:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f600ad89640 1 --2- 172.21.15.12:0/630106839 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f600409d460 0x7f60041407c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f600ad89640 1 -- 172.21.15.12:0/630106839 >> 172.21.15.12:0/630106839 conn(0x7f60040a7120 msgr2=0x7f60040a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f600ad89640 1 -- 172.21.15.12:0/630106839 shutdown_connections 2024-09-11T01:51:23.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.612+0000 7f600ad89640 1 -- 172.21.15.12:0/630106839 wait complete. 2024-09-11T01:51:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.613+0000 7f600ad89640 10 monclient: build_initial_monmap 2024-09-11T01:51:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.613+0000 7f600ad89640 10 monclient: monmap: 2024-09-11T01:51:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:51:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:51:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:51:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:51:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:51:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:51:23.623 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.613+0000 7f600ad89640 1 Processor -- start 2024-09-11T01:51:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.613+0000 7f600ad89640 1 -- start start 2024-09-11T01:51:23.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.613+0000 7f600ad89640 10 monclient: init 2024-09-11T01:51:23.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f600ad89640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:23.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f600ad89640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:51:23.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f600ad89640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f600409d030 0x7f600413f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:23.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f600ad89640 10 monclient(hunting): picked mon.noname-c con 0x7f600409d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:23.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f600ad89640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f600409d460 0x7f6004136e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:23.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f600ad89640 10 monclient(hunting): picked mon.noname-a con 0x7f600409d460 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:51:23.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f600ad89640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6004138230 0x7f600413cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:23.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f600ad89640 10 monclient(hunting): picked mon.noname-b con 0x7f6004138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:51:23.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f600ad89640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:23.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f600ad89640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6004140e80 con 0x7f600409d460 2024-09-11T01:51:23.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f600ad89640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:23.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f600ad89640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6004140d00 con 0x7f6004138230 2024-09-11T01:51:23.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f600ad89640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:23.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f600ad89640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6004141000 con 0x7f600409d030 2024-09-11T01:51:23.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f600ad89640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:23.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f600ad89640 10 monclient(hunting): authenticate will time out at 2090.056484s 2024-09-11T01:51:23.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6003fff640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f600409d460 msgr2=0x7f6004136e40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:51:23.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6003fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f600409d460 0x7f6004136e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:51:23.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6008b25640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f600409d030 0x7f600413f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:23.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6008b25640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f600409d030 0x7f600413f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51588/0 (socket says 172.21.15.12:51588) 2024-09-11T01:51:23.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6008b25640 1 -- 172.21.15.12:0/643044713 learned_addr learned my addr 172.21.15.12:0/643044713 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:23.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6008b25640 10 monclient(hunting): get_auth_request con 0x7f600409d030 auth_method 0 2024-09-11T01:51:23.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6008b25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:23.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6008b25640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:23.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6008b25640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:23.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6009326640 1 --2- 172.21.15.12:0/643044713 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6004138230 0x7f600413cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:23.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6009326640 10 monclient(hunting): get_auth_request con 0x7f6004138230 auth_method 0 2024-09-11T01:51:23.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6009326640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:23.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6009326640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:23.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6009326640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:23.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6008b25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:23.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6008b25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:23.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6008b25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:23.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6009326640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:23.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6009326640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:23.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6009326640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:23.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6008b25640 10 monclient(hunting): handle_auth_done global_id 244169 payload 824 2024-09-11T01:51:23.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6008b25640 1 -- 172.21.15.12:0/643044713 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6004138230 msgr2=0x7f600413cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:23.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6008b25640 1 --2- 172.21.15.12:0/643044713 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6004138230 0x7f600413cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:23.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6008b25640 1 -- 172.21.15.12:0/643044713 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f600409d460 msgr2=0x7f6004136e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:51:23.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6008b25640 1 --2- 172.21.15.12:0/643044713 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f600409d460 0x7f6004136e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:23.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6008b25640 10 monclient: _finish_hunting 0 2024-09-11T01:51:23.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6008b25640 1 monclient: found mon.noname-c 2024-09-11T01:51:23.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6008b25640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:23.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6008b25640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:51:23.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.614+0000 7f6008b25640 1 -- 172.21.15.12:0/643044713 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f600413fa00 con 0x7f600409d030 2024-09-11T01:51:23.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.617+0000 7f6008b25640 1 -- 172.21.15.12:0/643044713 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f600409d030 msgr2=0x7f600413f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:23.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.617+0000 7f6008b25640 1 --2- 172.21.15.12:0/643044713 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f600409d030 0x7f600413f070 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5ff400c460 tx=0x7f5ff400c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:23.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.617+0000 7f6008b25640 1 --2- 172.21.15.12:0/643044713 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f600409d030 0x7f600413f070 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5ff400c460 tx=0x7f5ff400c920 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7f5ff4016020 mon_map magic: 0 2024-09-11T01:51:23.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.617+0000 7f60017fa640 1 -- 172.21.15.12:0/643044713 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff4016020 con 0x7f600409d030 2024-09-11T01:51:23.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.617+0000 7f60017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:23.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.617+0000 7f60017fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:51:23.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.617+0000 7f60017fa640 10 monclient: dump: 2024-09-11T01:51:23.628 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:23.628 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:23.628 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:23.628 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:23.628 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:23.628 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:23.628 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:23.628 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:23.628 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:23.628 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:23.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.617+0000 7f60017fa640 10 monclient: _finish_auth 0 2024-09-11T01:51:23.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.617+0000 7f60017fa640 10 monclient: _check_auth_tickets 2024-09-11T01:51:23.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.617+0000 7f60017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:23.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.617+0000 7f60017fa640 1 -- 172.21.15.12:0/643044713 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ff4012a80 con 0x7f600409d030 2024-09-11T01:51:23.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.617+0000 7f600ad89640 5 monclient: authenticate success, global_id 244169 2024-09-11T01:51:23.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.617+0000 7f600ad89640 10 monclient: _renew_subs 2024-09-11T01:51:23.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.617+0000 7f600ad89640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:23.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.617+0000 7f600ad89640 1 -- 172.21.15.12:0/643044713 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60041cde40 con 0x7f600409d030 2024-09-11T01:51:23.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.617+0000 7f60017fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:23.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.617+0000 7f600ad89640 10 monclient: _renew_subs 2024-09-11T01:51:23.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.617+0000 7f6002ffd640 4 set_mon_vals no callback set 2024-09-11T01:51:23.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.617+0000 7f600ad89640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:23.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.617+0000 7f600ad89640 1 -- 172.21.15.12:0/643044713 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f60040a1810 con 0x7f600409d030 2024-09-11T01:51:23.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.618+0000 7f60017fa640 1 -- 172.21.15.12:0/643044713 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff4011380 con 0x7f600409d030 2024-09-11T01:51:23.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.618+0000 7f60017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:23.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.618+0000 7f60017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:23.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.618+0000 7f600ad89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:23.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.618+0000 7f60017fa640 10 monclient: dump: 2024-09-11T01:51:23.631 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:23.631 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:23.631 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:23.631 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:23.631 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:23.631 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:23.631 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:23.631 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:23.631 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:23.631 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:23.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.618+0000 7f60017fa640 1 -- 172.21.15.12:0/643044713 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f5ff4011560 con 0x7f600409d030 2024-09-11T01:51:23.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.618+0000 7f600ad89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:23.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.618+0000 7f600ad89640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:23.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.618+0000 7f600ad89640 1 -- 172.21.15.12:0/643044713 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6004138610 con 0x7f600409d030 2024-09-11T01:51:23.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.618+0000 7f60017fa640 1 --2- 172.21.15.12:0/643044713 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5ff003e1c0 0x7f5ff0040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:23.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.619+0000 7f6003fff640 1 --2- 172.21.15.12:0/643044713 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5ff003e1c0 0x7f5ff0040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:23.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.619+0000 7f6003fff640 10 monclient: get_auth_request con 0x7f5ff003e1c0 auth_method 0 2024-09-11T01:51:23.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.621+0000 7f6003fff640 1 --2- 172.21.15.12:0/643044713 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5ff003e1c0 0x7f5ff0040680 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f600413ff00 tx=0x7f5ff8006d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:23.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:23.623+0000 7f60017fa640 1 -- 172.21.15.12:0/643044713 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(735..735 src has 676..735) ==== 4799+0+0 (secure 0 0 0) 0x7f5ff4052e00 con 0x7f600409d030 2024-09-11T01:51:24.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:24.240+0000 7f60017fa640 1 -- 172.21.15.12:0/643044713 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5ff4020030 con 0x7f600409d030 2024-09-11T01:51:24.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:24.240+0000 7f60017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:24.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:24.240+0000 7f60017fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:51:24.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:24.502+0000 7f600ad89640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:51:24.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:24.502+0000 7f600ad89640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:51:24.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:24.502+0000 7f600ad89640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:24.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:24.502+0000 7f600ad89640 1 -- 172.21.15.12:0/643044713 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f60040698e0 con 0x7f600409d030 2024-09-11T01:51:24.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:24.502+0000 7f60017fa640 1 -- 172.21.15.12:0/643044713 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7f5ff4022df0 con 0x7f600409d030 2024-09-11T01:51:24.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:24.503+0000 7f60017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:51:24.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:24.503+0000 7f60017fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:51:24.505 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":208,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:51:24.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:24.505+0000 7f600ad89640 1 -- 172.21.15.12:0/643044713 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5ff003e1c0 msgr2=0x7f5ff0040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:24.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:24.505+0000 7f600ad89640 1 --2- 172.21.15.12:0/643044713 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5ff003e1c0 0x7f5ff0040680 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f600413ff00 tx=0x7f5ff8006d30 comp rx=0 tx=0).stop 2024-09-11T01:51:24.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:24.505+0000 7f600ad89640 10 monclient: shutdown 2024-09-11T01:51:24.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:24.505+0000 7f600ad89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:24.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:24.505+0000 7f600ad89640 1 -- 172.21.15.12:0/643044713 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f600409d030 msgr2=0x7f600413f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:24.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:24.505+0000 7f600ad89640 1 --2- 172.21.15.12:0/643044713 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f600409d030 0x7f600413f070 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5ff400c460 tx=0x7f5ff400c920 comp rx=0 tx=0).stop 2024-09-11T01:51:24.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:24.505+0000 7f600ad89640 1 -- 172.21.15.12:0/643044713 shutdown_connections 2024-09-11T01:51:24.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:24.505+0000 7f600ad89640 1 --2- 172.21.15.12:0/643044713 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6004138230 0x7f600413cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:24.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:24.505+0000 7f600ad89640 1 --2- 172.21.15.12:0/643044713 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5ff003e1c0 0x7f5ff0040680 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:24.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:24.505+0000 7f600ad89640 1 --2- 172.21.15.12:0/643044713 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f600409d460 0x7f6004136e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:24.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:24.505+0000 7f600ad89640 1 --2- 172.21.15.12:0/643044713 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f600409d030 0x7f600413f070 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:24.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:24.505+0000 7f600ad89640 1 -- 172.21.15.12:0/643044713 >> 172.21.15.12:0/643044713 conn(0x7f60040a7120 msgr2=0x7f60040a44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:24.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:24.505+0000 7f600ad89640 1 -- 172.21.15.12:0/643044713 shutdown_connections 2024-09-11T01:51:24.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:24.505+0000 7f600ad89640 1 -- 172.21.15.12:0/643044713 wait complete. 2024-09-11T01:51:27.518 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:51:27.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.630+0000 7fa6059c2640 10 monclient: get_monmap_and_config 2024-09-11T01:51:27.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.630+0000 7fa6059c2640 10 monclient: build_initial_monmap 2024-09-11T01:51:27.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.630+0000 7fa6059c2640 10 monclient: monmap: 2024-09-11T01:51:27.632 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:27.632 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:27.632 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:51:27.630772+0000 2024-09-11T01:51:27.632 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:51:27.630772+0000 2024-09-11T01:51:27.632 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:27.632 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:27.632 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:51:27.632 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:51:27.632 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:51:27.632 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:51:27.632 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:51:27.633 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:27.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.630+0000 7fa6059c2640 1 Processor -- start 2024-09-11T01:51:27.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.630+0000 7fa6059c2640 1 -- start start 2024-09-11T01:51:27.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.630+0000 7fa6059c2640 10 monclient: init 2024-09-11T01:51:27.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.631+0000 7fa6059c2640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:27.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.631+0000 7fa6059c2640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-11T01:51:27.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.631+0000 7fa6059c2640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa600138230 0x7fa600138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:27.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.631+0000 7fa6059c2640 10 monclient(hunting): picked mon.noname-c con 0x7fa600138230 addr v2:172.21.15.71:3300/0 2024-09-11T01:51:27.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.631+0000 7fa6059c2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa60009d030 addr v1:172.21.15.71:6789/0 2024-09-11T01:51:27.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.631+0000 7fa6059c2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa60009d460 addr v1:172.21.15.12:6789/0 2024-09-11T01:51:27.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.631+0000 7fa6059c2640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6000a5e40 con 0x7fa60009d460 2024-09-11T01:51:27.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.631+0000 7fa6059c2640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6000a6e30 con 0x7fa60009d030 2024-09-11T01:51:27.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.631+0000 7fa6059c2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:27.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.631+0000 7fa6059c2640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fa60009c9c0 con 0x7fa600138230 2024-09-11T01:51:27.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.631+0000 7fa6059c2640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:27.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.631+0000 7fa6059c2640 10 monclient(hunting): authenticate will time out at 2094.073447s 2024-09-11T01:51:27.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.631+0000 7fa5ff7fe640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7fa60009d460 legacy=0x7fa6000a5730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:51:27.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.631+0000 7fa5feffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa600138230 0x7fa600138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:27.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.631+0000 7fa5feffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa600138230 0x7fa600138610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:51214/0 (socket says 172.21.15.12:51214) 2024-09-11T01:51:27.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.631+0000 7fa5feffd640 1 -- 172.21.15.12:0/4149871112 learned_addr learned my addr 172.21.15.12:0/4149871112 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:27.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.631+0000 7fa5feffd640 10 monclient(hunting): get_auth_request con 0x7fa600138230 auth_method 0 2024-09-11T01:51:27.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.631+0000 7fa5feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:27.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.631+0000 7fa5feffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:27.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.631+0000 7fa5feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:27.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.631+0000 7fa5feffd640 1 -- 172.21.15.12:0/4149871112 >> v2:172.21.15.71:3300/0 conn(0x7fa600138230 msgr2=0x7fa600138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:51:27.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.631+0000 7fa5feffd640 1 -- 172.21.15.12:0/4149871112 >> v2:172.21.15.71:3300/0 conn(0x7fa600138230 msgr2=0x7fa600138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:27.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.631+0000 7fa5feffd640 1 --2- 172.21.15.12:0/4149871112 >> v2:172.21.15.71:3300/0 conn(0x7fa600138230 0x7fa600138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:27.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.631+0000 7fa5feffd640 1 --2- 172.21.15.12:0/4149871112 >> v2:172.21.15.71:3300/0 conn(0x7fa600138230 0x7fa600138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:51:27.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.634+0000 7fa5f7fff640 1 -- 172.21.15.12:0/4149871112 >> v1:172.21.15.71:6789/0 conn(0x7fa60009d030 legacy=0x7fa6000a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:27.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.832+0000 7fa5ff7fe640 1 -- 172.21.15.12:0/4149871112 >> v1:172.21.15.12:6789/0 conn(0x7fa60009d460 legacy=0x7fa6000a5730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:51:27.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.832+0000 7fa5feffd640 1 --2- 172.21.15.12:0/4149871112 >> v2:172.21.15.71:3300/0 conn(0x7fa600138230 0x7fa600138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:27.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.832+0000 7fa5feffd640 10 monclient(hunting): get_auth_request con 0x7fa600138230 auth_method 0 2024-09-11T01:51:27.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.832+0000 7fa5feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:27.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.832+0000 7fa5feffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:27.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.832+0000 7fa5feffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:27.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.832+0000 7fa5feffd640 1 -- 172.21.15.12:0/4149871112 >> v2:172.21.15.71:3300/0 conn(0x7fa600138230 msgr2=0x7fa600138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:51:27.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.832+0000 7fa5feffd640 1 -- 172.21.15.12:0/4149871112 >> v2:172.21.15.71:3300/0 conn(0x7fa600138230 msgr2=0x7fa600138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:27.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.832+0000 7fa5feffd640 1 --2- 172.21.15.12:0/4149871112 >> v2:172.21.15.71:3300/0 conn(0x7fa600138230 0x7fa600138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:27.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:27.832+0000 7fa5feffd640 1 --2- 172.21.15.12:0/4149871112 >> v2:172.21.15.71:3300/0 conn(0x7fa600138230 0x7fa600138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:51:28.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:28.233+0000 7fa5ff7fe640 1 -- 172.21.15.12:0/4149871112 >> v1:172.21.15.12:6789/0 conn(0x7fa60009d460 legacy=0x7fa6000a5730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:51:28.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:28.233+0000 7fa5feffd640 1 --2- 172.21.15.12:0/4149871112 >> v2:172.21.15.71:3300/0 conn(0x7fa600138230 0x7fa600138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:28.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:28.233+0000 7fa5feffd640 10 monclient(hunting): get_auth_request con 0x7fa600138230 auth_method 0 2024-09-11T01:51:28.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:28.233+0000 7fa5feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:28.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:28.233+0000 7fa5feffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:28.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:28.233+0000 7fa5feffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:28.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:28.233+0000 7fa5feffd640 1 -- 172.21.15.12:0/4149871112 >> v2:172.21.15.71:3300/0 conn(0x7fa600138230 msgr2=0x7fa600138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:51:28.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:28.233+0000 7fa5feffd640 1 -- 172.21.15.12:0/4149871112 >> v2:172.21.15.71:3300/0 conn(0x7fa600138230 msgr2=0x7fa600138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:28.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:28.233+0000 7fa5feffd640 1 --2- 172.21.15.12:0/4149871112 >> v2:172.21.15.71:3300/0 conn(0x7fa600138230 0x7fa600138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:28.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:28.233+0000 7fa5feffd640 1 --2- 172.21.15.12:0/4149871112 >> v2:172.21.15.71:3300/0 conn(0x7fa600138230 0x7fa600138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:51:29.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:29.034+0000 7fa5ff7fe640 1 -- 172.21.15.12:0/4149871112 >> v1:172.21.15.12:6789/0 conn(0x7fa60009d460 legacy=0x7fa6000a5730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:51:29.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:29.034+0000 7fa5feffd640 1 --2- 172.21.15.12:0/4149871112 >> v2:172.21.15.71:3300/0 conn(0x7fa600138230 0x7fa600138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:29.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:29.034+0000 7fa5feffd640 10 monclient(hunting): get_auth_request con 0x7fa600138230 auth_method 0 2024-09-11T01:51:29.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:29.034+0000 7fa5feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:29.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:29.034+0000 7fa5feffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:29.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:29.034+0000 7fa5feffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:29.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:29.034+0000 7fa5feffd640 1 -- 172.21.15.12:0/4149871112 >> v2:172.21.15.71:3300/0 conn(0x7fa600138230 msgr2=0x7fa600138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:51:29.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:29.034+0000 7fa5feffd640 1 -- 172.21.15.12:0/4149871112 >> v2:172.21.15.71:3300/0 conn(0x7fa600138230 msgr2=0x7fa600138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:29.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:29.034+0000 7fa5feffd640 1 --2- 172.21.15.12:0/4149871112 >> v2:172.21.15.71:3300/0 conn(0x7fa600138230 0x7fa600138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:29.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:29.034+0000 7fa5feffd640 1 --2- 172.21.15.12:0/4149871112 >> v2:172.21.15.71:3300/0 conn(0x7fa600138230 0x7fa600138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:51:30.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5fd7fa640 10 monclient(hunting): tick 2024-09-11T01:51:30.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5fd7fa640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:51:30.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5fd7fa640 1 monclient(hunting): continuing hunt 2024-09-11T01:51:30.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5fd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:51:30.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5fd7fa640 1 -- 172.21.15.12:0/4149871112 >> v2:172.21.15.71:3300/0 conn(0x7fa600138230 msgr2=0x7fa600138610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:51:30.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5fd7fa640 1 --2- 172.21.15.12:0/4149871112 >> v2:172.21.15.71:3300/0 conn(0x7fa600138230 0x7fa600138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:30.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5fd7fa640 1 -- 172.21.15.12:0/4149871112 >> v1:172.21.15.71:6789/0 conn(0x7fa60009d030 legacy=0x7fa6000a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:30.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5fd7fa640 1 -- 172.21.15.12:0/4149871112 >> v1:172.21.15.12:6789/0 conn(0x7fa60009d460 legacy=0x7fa6000a5730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:51:30.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5fd7fa640 10 monclient: _add_conns ranks=[2,3] 2024-09-11T01:51:30.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5fd7fa640 1 --2- 172.21.15.12:0/4149871112 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa5e0003740 0x7fa5e0005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:30.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5fd7fa640 10 monclient(hunting): picked mon.noname-b con 0x7fa5e0003740 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:30.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5fd7fa640 1 --2- 172.21.15.12:0/4149871112 >> v2:172.21.15.12:3300/0 conn(0x7fa5e0006250 0x7fa5e00086f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:30.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5fd7fa640 10 monclient(hunting): picked mon.noname-a con 0x7fa5e0006250 addr v2:172.21.15.12:3300/0 2024-09-11T01:51:30.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5fd7fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:30.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5fd7fa640 1 -- 172.21.15.12:0/4149871112 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fa60009c9c0 con 0x7fa5e0006250 2024-09-11T01:51:30.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5fd7fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:30.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5fd7fa640 1 -- 172.21.15.12:0/4149871112 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa5e0008db0 con 0x7fa5e0003740 2024-09-11T01:51:30.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5fd7fa640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:30.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5f7fff640 1 -- 172.21.15.12:0/4149871112 >> v2:172.21.15.12:3300/0 conn(0x7fa5e0006250 msgr2=0x7fa5e00086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:51:30.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5f7fff640 1 --2- 172.21.15.12:0/4149871112 >> v2:172.21.15.12:3300/0 conn(0x7fa5e0006250 0x7fa5e00086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:51:30.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5feffd640 1 --2- 172.21.15.12:0/4149871112 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa5e0003740 0x7fa5e0005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:30.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5feffd640 10 monclient(hunting): get_auth_request con 0x7fa5e0003740 auth_method 0 2024-09-11T01:51:30.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:30.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5feffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:30.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:30.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5feffd640 1 -- 172.21.15.12:0/4149871112 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa5e0003740 msgr2=0x7fa5e0005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:51:30.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5feffd640 1 -- 172.21.15.12:0/4149871112 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa5e0003740 msgr2=0x7fa5e0005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:30.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5feffd640 1 --2- 172.21.15.12:0/4149871112 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa5e0003740 0x7fa5e0005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:30.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.631+0000 7fa5feffd640 1 --2- 172.21.15.12:0/4149871112 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa5e0003740 0x7fa5e0005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:51:30.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.832+0000 7fa5f7fff640 1 -- 172.21.15.12:0/4149871112 >> v2:172.21.15.12:3300/0 conn(0x7fa5e0006250 msgr2=0x7fa5e00086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:51:30.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.832+0000 7fa5f7fff640 1 --2- 172.21.15.12:0/4149871112 >> v2:172.21.15.12:3300/0 conn(0x7fa5e0006250 0x7fa5e00086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:51:30.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.832+0000 7fa5feffd640 1 --2- 172.21.15.12:0/4149871112 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa5e0003740 0x7fa5e0005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:30.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.832+0000 7fa5feffd640 10 monclient(hunting): get_auth_request con 0x7fa5e0003740 auth_method 0 2024-09-11T01:51:30.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.832+0000 7fa5feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:30.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.832+0000 7fa5feffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:30.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.832+0000 7fa5feffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:30.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.832+0000 7fa5feffd640 1 -- 172.21.15.12:0/4149871112 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa5e0003740 msgr2=0x7fa5e0005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:51:30.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.832+0000 7fa5feffd640 1 -- 172.21.15.12:0/4149871112 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa5e0003740 msgr2=0x7fa5e0005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:30.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.832+0000 7fa5feffd640 1 --2- 172.21.15.12:0/4149871112 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa5e0003740 0x7fa5e0005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:30.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:30.832+0000 7fa5feffd640 1 --2- 172.21.15.12:0/4149871112 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa5e0003740 0x7fa5e0005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:51:31.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:31.233+0000 7fa5f7fff640 1 -- 172.21.15.12:0/4149871112 >> v2:172.21.15.12:3300/0 conn(0x7fa5e0006250 msgr2=0x7fa5e00086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:51:31.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:31.233+0000 7fa5f7fff640 1 --2- 172.21.15.12:0/4149871112 >> v2:172.21.15.12:3300/0 conn(0x7fa5e0006250 0x7fa5e00086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:51:31.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:31.233+0000 7fa5feffd640 1 --2- 172.21.15.12:0/4149871112 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa5e0003740 0x7fa5e0005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:31.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:31.233+0000 7fa5feffd640 10 monclient(hunting): get_auth_request con 0x7fa5e0003740 auth_method 0 2024-09-11T01:51:31.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:31.233+0000 7fa5feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:31.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:31.233+0000 7fa5feffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:31.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:31.233+0000 7fa5feffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:31.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:31.233+0000 7fa5feffd640 1 -- 172.21.15.12:0/4149871112 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa5e0003740 msgr2=0x7fa5e0005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:51:31.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:31.233+0000 7fa5feffd640 1 -- 172.21.15.12:0/4149871112 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa5e0003740 msgr2=0x7fa5e0005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:31.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:31.233+0000 7fa5feffd640 1 --2- 172.21.15.12:0/4149871112 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa5e0003740 0x7fa5e0005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:31.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:31.233+0000 7fa5feffd640 1 --2- 172.21.15.12:0/4149871112 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa5e0003740 0x7fa5e0005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:51:31.689 INFO:tasks.rados.rados.0.smithi071.stdout:3287: done (2 left) 2024-09-11T01:51:31.689 INFO:tasks.rados.rados.0.smithi071.stdout:3288: done (1 left) 2024-09-11T01:51:31.689 INFO:tasks.rados.rados.0.smithi071.stdout:3289: done (0 left) 2024-09-11T01:51:31.689 INFO:tasks.rados.rados.0.smithi071.stdout:3290: rollback oid 45 current snap is 367 2024-09-11T01:51:31.689 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 45 to 363 2024-09-11T01:51:31.689 INFO:tasks.rados.rados.0.smithi071.stdout:3291: rollback oid 12 current snap is 367 2024-09-11T01:51:31.690 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 12 to 350 2024-09-11T01:51:31.690 INFO:tasks.rados.rados.0.smithi071.stdout:3292: read oid 7 snap -1 2024-09-11T01:51:31.690 INFO:tasks.rados.rados.0.smithi071.stdout:3292: expect (ObjNum 1014 snap 322 seq_num 1014) 2024-09-11T01:51:31.690 INFO:tasks.rados.rados.0.smithi071.stdout:3293: snap_create 2024-09-11T01:51:31.694 INFO:tasks.rados.rados.0.smithi071.stdout:3290: finishing rollback tid 1 to smithi07134777-45 2024-09-11T01:51:31.694 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 2064 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-11T01:51:31.697 INFO:tasks.rados.rados.0.smithi071.stdout:3291: finishing rollback tid 0 to smithi07134777-12 2024-09-11T01:51:31.697 INFO:tasks.rados.rados.0.smithi071.stdout:3291: finishing rollback tid 1 to smithi07134777-12 2024-09-11T01:51:31.697 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 2215 (ObjNum 129 snap 0 seq_num 2371311152) dirty dne 2024-09-11T01:51:31.698 INFO:tasks.rados.rados.0.smithi071.stdout:3292: expect (ObjNum 1014 snap 322 seq_num 1014) 2024-09-11T01:51:31.698 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:51:31.751 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:51:32.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.034+0000 7fa5f7fff640 1 -- 172.21.15.12:0/4149871112 >> v2:172.21.15.12:3300/0 conn(0x7fa5e0006250 msgr2=0x7fa5e00086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:51:32.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.034+0000 7fa5f7fff640 1 --2- 172.21.15.12:0/4149871112 >> v2:172.21.15.12:3300/0 conn(0x7fa5e0006250 0x7fa5e00086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:51:32.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.034+0000 7fa5feffd640 1 --2- 172.21.15.12:0/4149871112 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa5e0003740 0x7fa5e0005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.034+0000 7fa5feffd640 10 monclient(hunting): get_auth_request con 0x7fa5e0003740 auth_method 0 2024-09-11T01:51:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.034+0000 7fa5feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.034+0000 7fa5feffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.034+0000 7fa5feffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.034+0000 7fa5feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.034+0000 7fa5feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.034+0000 7fa5feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.034+0000 7fa5feffd640 10 monclient(hunting): handle_auth_done global_id 244172 payload 274 2024-09-11T01:51:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.034+0000 7fa5feffd640 1 -- 172.21.15.12:0/4149871112 >> v2:172.21.15.12:3300/0 conn(0x7fa5e0006250 msgr2=0x7fa5e00086f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:51:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.034+0000 7fa5feffd640 1 --2- 172.21.15.12:0/4149871112 >> v2:172.21.15.12:3300/0 conn(0x7fa5e0006250 0x7fa5e00086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.034+0000 7fa5feffd640 10 monclient: _finish_hunting 0 2024-09-11T01:51:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.034+0000 7fa5feffd640 1 monclient: found mon.noname-b 2024-09-11T01:51:32.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.034+0000 7fa5feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.034+0000 7fa5feffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:51:32.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.034+0000 7fa5feffd640 1 -- 172.21.15.12:0/4149871112 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa60009ba20 con 0x7fa5e0003740 2024-09-11T01:51:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.037+0000 7fa5feffd640 1 -- 172.21.15.12:0/4149871112 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa5e0003740 msgr2=0x7fa5e0005b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.037+0000 7fa5feffd640 1 --2- 172.21.15.12:0/4149871112 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa5e0003740 0x7fa5e0005b70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0009100 tx=0x7fa5f002bec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5fa23d7baa8fe5b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.037+0000 7fa5fe7fc640 1 -- 172.21.15.12:0/4149871112 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f0032070 con 0x7fa5e0003740 2024-09-11T01:51:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.037+0000 7fa5fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.037+0000 7fa5fe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:51:32.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.037+0000 7fa5fe7fc640 10 monclient: dump: 2024-09-11T01:51:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:32.040 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:32.041 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.037+0000 7fa5fe7fc640 10 monclient: _finish_auth 0 2024-09-11T01:51:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.037+0000 7fa5fe7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:51:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.037+0000 7fa5fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.037+0000 7fa6059c2640 5 monclient: authenticate success, global_id 244172 2024-09-11T01:51:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.037+0000 7fa6059c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:51:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.037+0000 7fa5fe7fc640 1 -- 172.21.15.12:0/4149871112 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5f0030070 con 0x7fa5e0003740 2024-09-11T01:51:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.037+0000 7fa5fe7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.037+0000 7fa6059c2640 10 monclient: get_monmap_and_config success 2024-09-11T01:51:32.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.037+0000 7fa6059c2640 4 set_mon_vals no callback set 2024-09-11T01:51:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.037+0000 7fa5fe7fc640 1 -- 172.21.15.12:0/4149871112 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f000f040 con 0x7fa5e0003740 2024-09-11T01:51:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.037+0000 7fa5fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.037+0000 7fa5fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.037+0000 7fa5fe7fc640 10 monclient: dump: 2024-09-11T01:51:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.042 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:32.043 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:32.043 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:32.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.038+0000 7fa6059c2640 10 monclient: shutdown 2024-09-11T01:51:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.038+0000 7fa6059c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.038+0000 7fa6059c2640 1 -- 172.21.15.12:0/4149871112 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa5e0003740 msgr2=0x7fa5e0005b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.038+0000 7fa6059c2640 1 --2- 172.21.15.12:0/4149871112 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa5e0003740 0x7fa5e0005b70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0009100 tx=0x7fa5f002bec0 comp rx=0 tx=0).stop 2024-09-11T01:51:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.038+0000 7fa5ff7fe640 1 -- 172.21.15.12:0/4149871112 reap_dead start 2024-09-11T01:51:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.038+0000 7fa6059c2640 1 -- 172.21.15.12:0/4149871112 shutdown_connections 2024-09-11T01:51:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.038+0000 7fa6059c2640 1 -- 172.21.15.12:0/4149871112 >> 172.21.15.12:0/4149871112 conn(0x7fa6000a7120 msgr2=0x7fa6000a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.038+0000 7fa6059c2640 1 -- 172.21.15.12:0/4149871112 shutdown_connections 2024-09-11T01:51:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.038+0000 7fa6059c2640 1 -- 172.21.15.12:0/4149871112 wait complete. 2024-09-11T01:51:32.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.039+0000 7fa6059c2640 10 monclient: build_initial_monmap 2024-09-11T01:51:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.039+0000 7fa6059c2640 10 monclient: monmap: 2024-09-11T01:51:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:51:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:51:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:51:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:51:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:51:32.050 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:51:32.051 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:32.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.039+0000 7fa6059c2640 1 Processor -- start 2024-09-11T01:51:32.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa6059c2640 1 -- start start 2024-09-11T01:51:32.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa6059c2640 10 monclient: init 2024-09-11T01:51:32.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa6059c2640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:32.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa6059c2640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:51:32.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa6059c2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa60013f570 0x7fa60013d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa6059c2640 10 monclient(hunting): picked mon.noname-b con 0x7fa60013f570 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:51:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa6059c2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa600137010 0x7fa6001cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa6059c2640 10 monclient(hunting): picked mon.noname-a con 0x7fa600137010 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:51:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa6059c2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa600137490 0x7fa6001cd580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa6059c2640 10 monclient(hunting): picked mon.noname-c con 0x7fa600137490 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa6059c2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:32.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa6059c2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa600138b50 con 0x7fa600137010 2024-09-11T01:51:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa6059c2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa6059c2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa60006f7f0 con 0x7fa60013f570 2024-09-11T01:51:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa6059c2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa6059c2640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa60013f950 con 0x7fa600137490 2024-09-11T01:51:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa6059c2640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa6059c2640 10 monclient(hunting): authenticate will time out at 2098.482542s 2024-09-11T01:51:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa5f7fff640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa600137010 msgr2=0x7fa6001cd040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:51:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa5f7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa600137010 0x7fa6001cd040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:51:32.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa5ff7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa600137490 0x7fa6001cd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa5ff7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa600137490 0x7fa6001cd580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41084/0 (socket says 172.21.15.12:41084) 2024-09-11T01:51:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa5ff7fe640 1 -- 172.21.15.12:0/2827250254 learned_addr learned my addr 172.21.15.12:0/2827250254 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa5ff7fe640 10 monclient(hunting): get_auth_request con 0x7fa600137490 auth_method 0 2024-09-11T01:51:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa5ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa5ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa5ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa5ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa5ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa5feffd640 1 --2- 172.21.15.12:0/2827250254 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa60013f570 0x7fa60013d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa5ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa5feffd640 10 monclient(hunting): get_auth_request con 0x7fa60013f570 auth_method 0 2024-09-11T01:51:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa5feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:32.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa5feffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa5feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa5feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.040+0000 7fa5feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.041+0000 7fa5feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.041+0000 7fa5ff7fe640 10 monclient(hunting): handle_auth_done global_id 244175 payload 824 2024-09-11T01:51:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.041+0000 7fa5ff7fe640 1 -- 172.21.15.12:0/2827250254 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa60013f570 msgr2=0x7fa60013d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.041+0000 7fa5ff7fe640 1 --2- 172.21.15.12:0/2827250254 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa60013f570 0x7fa60013d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.041+0000 7fa5ff7fe640 1 -- 172.21.15.12:0/2827250254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa600137010 msgr2=0x7fa6001cd040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:51:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.041+0000 7fa5ff7fe640 1 --2- 172.21.15.12:0/2827250254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa600137010 0x7fa6001cd040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.041+0000 7fa5ff7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:51:32.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.041+0000 7fa5ff7fe640 1 monclient: found mon.noname-c 2024-09-11T01:51:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.041+0000 7fa5ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.041+0000 7fa5ff7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:51:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.041+0000 7fa5ff7fe640 1 -- 172.21.15.12:0/2827250254 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa60013ff00 con 0x7fa600137490 2024-09-11T01:51:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.043+0000 7fa5ff7fe640 1 -- 172.21.15.12:0/2827250254 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa600137490 msgr2=0x7fa6001cd580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.043+0000 7fa5ff7fe640 1 --2- 172.21.15.12:0/2827250254 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa600137490 0x7fa6001cd580 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa60009d860 tx=0x7fa5f800be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.043+0000 7fa5fcff9640 1 -- 172.21.15.12:0/2827250254 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8016630 con 0x7fa600137490 2024-09-11T01:51:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.043+0000 7fa5fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:32.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.043+0000 7fa5fcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:51:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.043+0000 7fa5fcff9640 10 monclient: dump: 2024-09-11T01:51:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:32.057 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:32.057 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.043+0000 7fa5fcff9640 10 monclient: _finish_auth 0 2024-09-11T01:51:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.043+0000 7fa5fcff9640 10 monclient: _check_auth_tickets 2024-09-11T01:51:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.043+0000 7fa5fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.043+0000 7fa5fcff9640 1 -- 172.21.15.12:0/2827250254 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5f8010030 con 0x7fa600137490 2024-09-11T01:51:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.043+0000 7fa6059c2640 5 monclient: authenticate success, global_id 244175 2024-09-11T01:51:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.043+0000 7fa5fcff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.043+0000 7fa5fcff9640 1 -- 172.21.15.12:0/2827250254 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8016a30 con 0x7fa600137490 2024-09-11T01:51:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.043+0000 7fa6059c2640 10 monclient: _renew_subs 2024-09-11T01:51:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.043+0000 7fa6059c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.043+0000 7fa6059c2640 1 -- 172.21.15.12:0/2827250254 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa60013e430 con 0x7fa600137490 2024-09-11T01:51:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.043+0000 7fa5fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.043+0000 7fa5fcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.043+0000 7fa5fcff9640 10 monclient: dump: 2024-09-11T01:51:32.058 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:32.059 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.043+0000 7fa5fdffb640 4 set_mon_vals no callback set 2024-09-11T01:51:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.043+0000 7fa6059c2640 10 monclient: _renew_subs 2024-09-11T01:51:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.043+0000 7fa6059c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:32.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.043+0000 7fa6059c2640 1 -- 172.21.15.12:0/2827250254 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa60013ea10 con 0x7fa600137490 2024-09-11T01:51:32.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.044+0000 7fa5fcff9640 1 -- 172.21.15.12:0/2827250254 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fa5f80153e0 con 0x7fa600137490 2024-09-11T01:51:32.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.045+0000 7fa6059c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:32.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.045+0000 7fa6059c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:32.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.045+0000 7fa6059c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:32.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.045+0000 7fa6059c2640 1 -- 172.21.15.12:0/2827250254 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa600138610 con 0x7fa600137490 2024-09-11T01:51:32.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.045+0000 7fa5fcff9640 1 --2- 172.21.15.12:0/2827250254 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa5e003efd0 0x7fa5e0041490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:32.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.045+0000 7fa5fcff9640 1 -- 172.21.15.12:0/2827250254 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(736..736 src has 676..736) ==== 4771+0+0 (secure 0 0 0) 0x7fa5f8053090 con 0x7fa600137490 2024-09-11T01:51:32.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.045+0000 7fa5feffd640 1 --2- 172.21.15.12:0/2827250254 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa5e003efd0 0x7fa5e0041490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:32.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.045+0000 7fa5feffd640 10 monclient: get_auth_request con 0x7fa5e003efd0 auth_method 0 2024-09-11T01:51:32.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.048+0000 7fa5feffd640 1 --2- 172.21.15.12:0/2827250254 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa5e003efd0 0x7fa5e0041490 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0006220 tx=0x7fa5f0032000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:32.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.050+0000 7fa5fcff9640 1 -- 172.21.15.12:0/2827250254 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa5f800ce20 con 0x7fa600137490 2024-09-11T01:51:32.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.050+0000 7fa5fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:32.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.050+0000 7fa5fcff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:51:32.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.307+0000 7fa6059c2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:51:32.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.307+0000 7fa6059c2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:51:32.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.307+0000 7fa6059c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:32.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.307+0000 7fa6059c2640 1 -- 172.21.15.12:0/2827250254 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa600140dd0 con 0x7fa600137490 2024-09-11T01:51:32.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.307+0000 7fa5fcff9640 1 -- 172.21.15.12:0/2827250254 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fa5f8051a90 con 0x7fa600137490 2024-09-11T01:51:32.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.307+0000 7fa5fcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:51:32.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.307+0000 7fa5fcff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:51:32.311 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":210,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:51:32.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.309+0000 7fa6059c2640 1 -- 172.21.15.12:0/2827250254 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa5e003efd0 msgr2=0x7fa5e0041490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:32.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.309+0000 7fa6059c2640 1 --2- 172.21.15.12:0/2827250254 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa5e003efd0 0x7fa5e0041490 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0006220 tx=0x7fa5f0032000 comp rx=0 tx=0).stop 2024-09-11T01:51:32.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.309+0000 7fa6059c2640 10 monclient: shutdown 2024-09-11T01:51:32.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.309+0000 7fa6059c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:32.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.309+0000 7fa6059c2640 1 -- 172.21.15.12:0/2827250254 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa600137490 msgr2=0x7fa6001cd580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:32.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.309+0000 7fa6059c2640 1 --2- 172.21.15.12:0/2827250254 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa600137490 0x7fa6001cd580 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa60009d860 tx=0x7fa5f800be90 comp rx=0 tx=0).stop 2024-09-11T01:51:32.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.310+0000 7fa6059c2640 1 -- 172.21.15.12:0/2827250254 shutdown_connections 2024-09-11T01:51:32.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.310+0000 7fa6059c2640 1 --2- 172.21.15.12:0/2827250254 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa600137490 0x7fa6001cd580 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:32.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.310+0000 7fa6059c2640 1 --2- 172.21.15.12:0/2827250254 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa5e003efd0 0x7fa5e0041490 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:32.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.310+0000 7fa6059c2640 1 --2- 172.21.15.12:0/2827250254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa600137010 0x7fa6001cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:32.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.310+0000 7fa6059c2640 1 --2- 172.21.15.12:0/2827250254 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa60013f570 0x7fa60013d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:32.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.310+0000 7fa6059c2640 1 -- 172.21.15.12:0/2827250254 >> 172.21.15.12:0/2827250254 conn(0x7fa6000a7120 msgr2=0x7fa6000a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:32.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.310+0000 7fa6059c2640 1 -- 172.21.15.12:0/2827250254 shutdown_connections 2024-09-11T01:51:32.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.310+0000 7fa6059c2640 1 -- 172.21.15.12:0/2827250254 wait complete. 2024-09-11T01:51:32.322 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:51:32.322 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T01:51:32.322 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:51:32.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.422+0000 7f74413c2640 10 monclient: get_monmap_and_config 2024-09-11T01:51:32.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.422+0000 7f74413c2640 10 monclient: build_initial_monmap 2024-09-11T01:51:32.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.422+0000 7f74413c2640 10 monclient: monmap: 2024-09-11T01:51:32.424 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:32.424 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:32.425 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:51:32.423566+0000 2024-09-11T01:51:32.425 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:51:32.423566+0000 2024-09-11T01:51:32.425 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:32.425 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:32.425 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:51:32.425 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:51:32.425 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:51:32.425 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:51:32.425 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:51:32.425 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:32.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.423+0000 7f74413c2640 1 Processor -- start 2024-09-11T01:51:32.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.423+0000 7f74413c2640 1 -- start start 2024-09-11T01:51:32.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.423+0000 7f74413c2640 10 monclient: init 2024-09-11T01:51:32.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.423+0000 7f74413c2640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:32.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.423+0000 7f74413c2640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-11T01:51:32.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.423+0000 7f74413c2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f743c074180 addr v1:172.21.15.12:6789/0 2024-09-11T01:51:32.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.423+0000 7f74413c2640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f743c0a06d0 0x7f743c13cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:32.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.423+0000 7f74413c2640 10 monclient(hunting): picked mon.noname-c con 0x7f743c0a06d0 addr v2:172.21.15.71:3300/0 2024-09-11T01:51:32.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.424+0000 7f74413c2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f743c134ac0 addr v1:172.21.15.71:6789/0 2024-09-11T01:51:32.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.424+0000 7f74413c2640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f743c070160 con 0x7f743c074180 2024-09-11T01:51:32.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.424+0000 7f74413c2640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f743c070440 con 0x7f743c134ac0 2024-09-11T01:51:32.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.424+0000 7f74413c2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:32.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.424+0000 7f74413c2640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f743c135560 con 0x7f743c0a06d0 2024-09-11T01:51:32.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.424+0000 7f74413c2640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:32.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.424+0000 7f74413c2640 10 monclient(hunting): authenticate will time out at 2098.866066s 2024-09-11T01:51:32.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.424+0000 7f743affd640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7f743c074180 legacy=0x7f743c074560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:51:32.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.424+0000 7f743a7fc640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f743c0a06d0 0x7f743c13cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:32.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.424+0000 7f743a7fc640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f743c0a06d0 0x7f743c13cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:51292/0 (socket says 172.21.15.12:51292) 2024-09-11T01:51:32.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.424+0000 7f743a7fc640 1 -- 172.21.15.12:0/3790600299 learned_addr learned my addr 172.21.15.12:0/3790600299 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:32.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.424+0000 7f743a7fc640 10 monclient(hunting): get_auth_request con 0x7f743c0a06d0 auth_method 0 2024-09-11T01:51:32.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.424+0000 7f743a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:32.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.424+0000 7f743a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:32.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.424+0000 7f743a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:32.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.426+0000 7f743a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:32.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.426+0000 7f743b7fe640 1 -- 172.21.15.12:0/3790600299 >> v1:172.21.15.71:6789/0 conn(0x7f743c134ac0 legacy=0x7f743c13f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:32.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.426+0000 7f743a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:32.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.426+0000 7f743a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:32.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.426+0000 7f743a7fc640 10 monclient(hunting): handle_auth_done global_id 234202 payload 274 2024-09-11T01:51:32.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.426+0000 7f743a7fc640 1 -- 172.21.15.12:0/3790600299 >> v1:172.21.15.71:6789/0 conn(0x7f743c134ac0 legacy=0x7f743c13f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:32.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.426+0000 7f743a7fc640 1 -- 172.21.15.12:0/3790600299 >> v1:172.21.15.12:6789/0 conn(0x7f743c074180 legacy=0x7f743c074560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:51:32.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.426+0000 7f743a7fc640 10 monclient: _finish_hunting 0 2024-09-11T01:51:32.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.426+0000 7f743a7fc640 1 monclient: found mon.noname-c 2024-09-11T01:51:32.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.427+0000 7f743a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:32.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.427+0000 7f743a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:51:32.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.426+0000 7f7439ffb640 1 -- 172.21.15.12:0/3790600299 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 591667012 0 0) 0x7f743c070440 con 0x7f743c134ac0 2024-09-11T01:51:32.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.427+0000 7f743a7fc640 1 -- 172.21.15.12:0/3790600299 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f743c068bc0 con 0x7f743c0a06d0 2024-09-11T01:51:32.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.427+0000 7f7439ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T01:51:32.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.429+0000 7f743a7fc640 1 -- 172.21.15.12:0/3790600299 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f743c0a06d0 msgr2=0x7f743c13cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:32.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.429+0000 7f743a7fc640 1 --2- 172.21.15.12:0/3790600299 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f743c0a06d0 0x7f743c13cec0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f74240098f0 tx=0x7f742402a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e31615c7f1ae80c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:32.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.430+0000 7f7439ffb640 1 -- 172.21.15.12:0/3790600299 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7424037070 con 0x7f743c0a06d0 2024-09-11T01:51:32.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.430+0000 7f7439ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:32.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.430+0000 7f7439ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:51:32.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.430+0000 7f7439ffb640 10 monclient: dump: 2024-09-11T01:51:32.436 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:32.436 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:32.436 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.436 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.436 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:32.436 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:32.436 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:32.436 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:32.436 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:32.436 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:32.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.430+0000 7f7439ffb640 10 monclient: _finish_auth 0 2024-09-11T01:51:32.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.430+0000 7f7439ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:51:32.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.430+0000 7f7439ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:32.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.430+0000 7f74413c2640 5 monclient: authenticate success, global_id 234202 2024-09-11T01:51:32.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.430+0000 7f74413c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:51:32.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.430+0000 7f7439ffb640 1 -- 172.21.15.12:0/3790600299 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7424032640 con 0x7f743c0a06d0 2024-09-11T01:51:32.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.430+0000 7f7439ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:32.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.430+0000 7f7439ffb640 1 -- 172.21.15.12:0/3790600299 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7424032a10 con 0x7f743c0a06d0 2024-09-11T01:51:32.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.430+0000 7f74413c2640 10 monclient: get_monmap_and_config success 2024-09-11T01:51:32.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.430+0000 7f74413c2640 4 set_mon_vals no callback set 2024-09-11T01:51:32.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.430+0000 7f7439ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:32.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.430+0000 7f7439ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:51:32.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.430+0000 7f7439ffb640 10 monclient: dump: 2024-09-11T01:51:32.437 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:32.438 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:32.438 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.438 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.438 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:32.438 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:32.438 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:32.438 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:32.438 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:32.438 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:32.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.430+0000 7f74413c2640 10 monclient: shutdown 2024-09-11T01:51:32.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.430+0000 7f74413c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:32.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.430+0000 7f74413c2640 1 -- 172.21.15.12:0/3790600299 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f743c0a06d0 msgr2=0x7f743c13cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:32.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.430+0000 7f74413c2640 1 --2- 172.21.15.12:0/3790600299 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f743c0a06d0 0x7f743c13cec0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f74240098f0 tx=0x7f742402a1c0 comp rx=0 tx=0).stop 2024-09-11T01:51:32.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.430+0000 7f74413c2640 1 -- 172.21.15.12:0/3790600299 shutdown_connections 2024-09-11T01:51:32.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.430+0000 7f74413c2640 1 --2- 172.21.15.12:0/3790600299 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f743c0a06d0 0x7f743c13cec0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:32.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.430+0000 7f74413c2640 1 -- 172.21.15.12:0/3790600299 >> 172.21.15.12:0/3790600299 conn(0x7f743c0a36c0 msgr2=0x7f743c0a3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:32.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.430+0000 7f74413c2640 1 -- 172.21.15.12:0/3790600299 shutdown_connections 2024-09-11T01:51:32.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.430+0000 7f74413c2640 1 -- 172.21.15.12:0/3790600299 wait complete. 2024-09-11T01:51:32.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.431+0000 7f74413c2640 10 monclient: build_initial_monmap 2024-09-11T01:51:32.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.431+0000 7f74413c2640 10 monclient: monmap: 2024-09-11T01:51:32.439 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:32.439 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:51:32.439 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:51:32.439 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:51:32.439 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:32.439 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:32.439 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:51:32.439 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:51:32.440 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:51:32.440 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:32.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.431+0000 7f74413c2640 1 Processor -- start 2024-09-11T01:51:32.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.431+0000 7f74413c2640 1 -- start start 2024-09-11T01:51:32.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.431+0000 7f74413c2640 10 monclient: init 2024-09-11T01:51:32.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f74413c2640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:32.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f74413c2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:51:32.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f74413c2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f743c074180 0x7f743c13f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:32.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f74413c2640 10 monclient(hunting): picked mon.noname-b con 0x7f743c074180 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:51:32.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f74413c2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f743c0a06d0 0x7f743c13a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:32.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f74413c2640 10 monclient(hunting): picked mon.noname-c con 0x7f743c0a06d0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:32.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f74413c2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f743c134ac0 0x7f743c13a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:32.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f74413c2640 10 monclient(hunting): picked mon.noname-a con 0x7f743c134ac0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:51:32.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f74413c2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:32.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f74413c2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f743c140d40 con 0x7f743c134ac0 2024-09-11T01:51:32.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f74413c2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:32.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f74413c2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f743c070740 con 0x7f743c074180 2024-09-11T01:51:32.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f74413c2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:32.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f74413c2640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f743c140ec0 con 0x7f743c0a06d0 2024-09-11T01:51:32.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f74413c2640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:32.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f74413c2640 10 monclient(hunting): authenticate will time out at 2098.874360s 2024-09-11T01:51:32.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743b7fe640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f743c134ac0 msgr2=0x7f743c13a720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:51:32.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f743c134ac0 0x7f743c13a720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:51:32.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743a7fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f743c0a06d0 0x7f743c13a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:32.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743a7fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f743c0a06d0 0x7f743c13a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41100/0 (socket says 172.21.15.12:41100) 2024-09-11T01:51:32.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743a7fc640 1 -- 172.21.15.12:0/2515176716 learned_addr learned my addr 172.21.15.12:0/2515176716 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:32.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743a7fc640 10 monclient(hunting): get_auth_request con 0x7f743c0a06d0 auth_method 0 2024-09-11T01:51:32.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:32.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:32.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:32.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:32.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:32.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:32.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743affd640 1 --2- 172.21.15.12:0/2515176716 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f743c074180 0x7f743c13f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:32.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743affd640 10 monclient(hunting): get_auth_request con 0x7f743c074180 auth_method 0 2024-09-11T01:51:32.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:32.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:32.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:32.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:32.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:32.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:32.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743a7fc640 10 monclient(hunting): handle_auth_done global_id 244178 payload 824 2024-09-11T01:51:32.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743a7fc640 1 -- 172.21.15.12:0/2515176716 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f743c074180 msgr2=0x7f743c13f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:32.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743a7fc640 1 --2- 172.21.15.12:0/2515176716 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f743c074180 0x7f743c13f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:32.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743a7fc640 1 -- 172.21.15.12:0/2515176716 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f743c134ac0 msgr2=0x7f743c13a720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:51:32.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743a7fc640 1 --2- 172.21.15.12:0/2515176716 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f743c134ac0 0x7f743c13a720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:32.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743a7fc640 10 monclient: _finish_hunting 0 2024-09-11T01:51:32.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743a7fc640 1 monclient: found mon.noname-c 2024-09-11T01:51:32.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:32.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:51:32.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.432+0000 7f743a7fc640 1 -- 172.21.15.12:0/2515176716 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f743c13dd20 con 0x7f743c0a06d0 2024-09-11T01:51:32.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.435+0000 7f743a7fc640 1 -- 172.21.15.12:0/2515176716 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f743c0a06d0 msgr2=0x7f743c13a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:32.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.435+0000 7f743a7fc640 1 --2- 172.21.15.12:0/2515176716 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f743c0a06d0 0x7f743c13a1e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f74240090b0 tx=0x7f74240046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:32.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.435+0000 7f74337fe640 1 -- 172.21.15.12:0/2515176716 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7424042070 con 0x7f743c0a06d0 2024-09-11T01:51:32.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.435+0000 7f74337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:32.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.435+0000 7f74337fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:51:32.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.435+0000 7f74337fe640 10 monclient: dump: 2024-09-11T01:51:32.448 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:32.448 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:32.448 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.448 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.448 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:32.448 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:32.448 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:32.448 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:32.448 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:32.448 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:32.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.435+0000 7f74337fe640 10 monclient: _finish_auth 0 2024-09-11T01:51:32.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.435+0000 7f74337fe640 10 monclient: _check_auth_tickets 2024-09-11T01:51:32.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.435+0000 7f74337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:32.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.435+0000 7f74337fe640 1 -- 172.21.15.12:0/2515176716 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f742403e880 con 0x7f743c0a06d0 2024-09-11T01:51:32.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.435+0000 7f74413c2640 5 monclient: authenticate success, global_id 244178 2024-09-11T01:51:32.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.435+0000 7f74413c2640 10 monclient: _renew_subs 2024-09-11T01:51:32.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.435+0000 7f74413c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:32.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.435+0000 7f74413c2640 1 -- 172.21.15.12:0/2515176716 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f743c13e000 con 0x7f743c0a06d0 2024-09-11T01:51:32.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.435+0000 7f74337fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:32.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.435+0000 7f74337fe640 1 -- 172.21.15.12:0/2515176716 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7424037040 con 0x7f743c0a06d0 2024-09-11T01:51:32.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.435+0000 7f74337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:32.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.435+0000 7f74337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:32.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.435+0000 7f74337fe640 10 monclient: dump: 2024-09-11T01:51:32.449 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:32.449 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:32.449 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.449 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.450 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:32.450 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:32.450 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:32.450 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:32.450 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:32.450 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:32.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.435+0000 7f74397fa640 4 set_mon_vals no callback set 2024-09-11T01:51:32.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.435+0000 7f74413c2640 10 monclient: _renew_subs 2024-09-11T01:51:32.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.435+0000 7f74413c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:32.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.435+0000 7f74413c2640 1 -- 172.21.15.12:0/2515176716 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f743c075000 con 0x7f743c0a06d0 2024-09-11T01:51:32.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.436+0000 7f74413c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:32.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.436+0000 7f74413c2640 1 --2- 172.21.15.12:0/2515176716 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f743c13e770 0x7f743c13ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:32.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.436+0000 7f74413c2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:32.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.436+0000 7f74413c2640 1 -- 172.21.15.12:0/2515176716 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f743c133ad0 con 0x7f743c13e770 2024-09-11T01:51:32.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.436+0000 7f743affd640 1 --2- 172.21.15.12:0/2515176716 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f743c13e770 0x7f743c13ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:32.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.436+0000 7f74337fe640 1 -- 172.21.15.12:0/2515176716 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f742403ec70 con 0x7f743c0a06d0 2024-09-11T01:51:32.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.436+0000 7f743affd640 10 monclient: get_auth_request con 0x7f743c13e770 auth_method 0 2024-09-11T01:51:32.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.436+0000 7f743affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:32.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.436+0000 7f743affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:32.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.436+0000 7f743affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:32.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.436+0000 7f743affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:32.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.436+0000 7f743affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:32.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.436+0000 7f743affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:32.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.436+0000 7f743affd640 10 monclient(hunting): handle_auth_done global_id 244181 payload 824 2024-09-11T01:51:32.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.436+0000 7f743affd640 1 -- 172.21.15.12:0/2515176716 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f743c074560 con 0x7f743c13e770 2024-09-11T01:51:32.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.436+0000 7f74337fe640 1 --2- 172.21.15.12:0/2515176716 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f741c03e1c0 0x7f741c040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:32.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.436+0000 7f74337fe640 1 -- 172.21.15.12:0/2515176716 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(736..736 src has 676..736) ==== 4771+0+0 (secure 0 0 0) 0x7f7424073800 con 0x7f743c0a06d0 2024-09-11T01:51:32.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.436+0000 7f743b7fe640 1 --2- 172.21.15.12:0/2515176716 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f741c03e1c0 0x7f741c040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:32.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.436+0000 7f743b7fe640 10 monclient: get_auth_request con 0x7f741c03e1c0 auth_method 0 2024-09-11T01:51:32.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.438+0000 7f743affd640 1 -- 172.21.15.12:0/2515176716 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f743c13e770 msgr2=0x7f743c13ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:32.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.438+0000 7f743affd640 1 --2- 172.21.15.12:0/2515176716 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f743c13e770 0x7f743c13ac60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7434009610 tx=0x7f7434009ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:32.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.438+0000 7f74337fe640 1 -- 172.21.15.12:0/2515176716 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7434016070 con 0x7f743c13e770 2024-09-11T01:51:32.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.438+0000 7f74337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:32.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.438+0000 7f74337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:32.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.438+0000 7f74337fe640 10 monclient: dump: 2024-09-11T01:51:32.453 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:32.453 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:32.453 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.453 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.453 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:32.453 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:32.454 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:32.454 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:32.454 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:32.454 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:32.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.439+0000 7f74337fe640 1 -- 172.21.15.12:0/2515176716 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7434005a40 con 0x7f743c13e770 2024-09-11T01:51:32.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.439+0000 7f74337fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:32.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.439+0000 7f74337fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:51:32.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.439+0000 7f74337fe640 1 -- 172.21.15.12:0/2515176716 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f743c13e770 msgr2=0x7f743c13ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:32.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.439+0000 7f743b7fe640 1 --2- 172.21.15.12:0/2515176716 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f741c03e1c0 0x7f741c040680 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f742c007fd0 tx=0x7f742c007c20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:32.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.439+0000 7f74337fe640 1 --2- 172.21.15.12:0/2515176716 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f743c13e770 0x7f743c13ac60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7434009610 tx=0x7f7434009ad0 comp rx=0 tx=0).stop 2024-09-11T01:51:32.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.439+0000 7f74337fe640 1 -- 172.21.15.12:0/2515176716 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f743c13e770 msgr2=0x7f743c13ac60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:51:32.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.439+0000 7f74337fe640 1 --2- 172.21.15.12:0/2515176716 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f743c13e770 0x7f743c13ac60 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7434009610 tx=0x7f7434009ad0 comp rx=0 tx=0).stop 2024-09-11T01:51:32.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.445+0000 7f74413c2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:51:32.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.445+0000 7f74413c2640 1 --2- 172.21.15.12:0/2515176716 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f743c135710 0x7f743c135af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:32.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.445+0000 7f74413c2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:32.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.445+0000 7f74413c2640 1 -- 172.21.15.12:0/2515176716 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f743c06dc80 con 0x7f743c135710 2024-09-11T01:51:32.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.445+0000 7f743affd640 1 --2- 172.21.15.12:0/2515176716 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f743c135710 0x7f743c135af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:32.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.445+0000 7f743affd640 10 monclient: get_auth_request con 0x7f743c135710 auth_method 0 2024-09-11T01:51:32.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.445+0000 7f743affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:32.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.445+0000 7f743affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:32.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.445+0000 7f743affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:32.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.445+0000 7f743affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:32.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.445+0000 7f743affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:32.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.445+0000 7f743affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:32.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.445+0000 7f743affd640 10 monclient(hunting): handle_auth_done global_id 244184 payload 824 2024-09-11T01:51:32.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.445+0000 7f743affd640 1 -- 172.21.15.12:0/2515176716 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f743c13d9a0 con 0x7f743c135710 2024-09-11T01:51:32.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.448+0000 7f743affd640 1 -- 172.21.15.12:0/2515176716 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f743c135710 msgr2=0x7f743c135af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:32.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.448+0000 7f743affd640 1 --2- 172.21.15.12:0/2515176716 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f743c135710 0x7f743c135af0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7434017000 tx=0x7f74340059d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:32.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.448+0000 7f74337fe640 1 -- 172.21.15.12:0/2515176716 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7434016050 con 0x7f743c135710 2024-09-11T01:51:32.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.448+0000 7f74337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:32.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.448+0000 7f74337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:32.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.448+0000 7f74337fe640 10 monclient: dump: 2024-09-11T01:51:32.457 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:32.457 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:32.457 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.457 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.457 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:32.457 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:32.457 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:32.458 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:32.458 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:32.458 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:32.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.448+0000 7f74337fe640 1 -- 172.21.15.12:0/2515176716 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4864 (secure 0 0 0) 0x7f743400e3f0 con 0x7f743c135710 2024-09-11T01:51:32.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.448+0000 7f74337fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:51:32.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.448+0000 7f74337fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:51:32.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.448+0000 7f74337fe640 1 -- 172.21.15.12:0/2515176716 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f743c135710 msgr2=0x7f743c135af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:32.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.448+0000 7f74337fe640 1 --2- 172.21.15.12:0/2515176716 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f743c135710 0x7f743c135af0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7434017000 tx=0x7f74340059d0 comp rx=0 tx=0).stop 2024-09-11T01:51:32.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.448+0000 7f74337fe640 1 -- 172.21.15.12:0/2515176716 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f743c135710 msgr2=0x7f743c135af0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:51:32.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.448+0000 7f74337fe640 1 --2- 172.21.15.12:0/2515176716 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f743c135710 0x7f743c135af0 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7434017000 tx=0x7f74340059d0 comp rx=0 tx=0).stop 2024-09-11T01:51:32.459 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:51:32.459 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:51:32.459 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:51:32.459 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:51:32.459 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 210, 2024-09-11T01:51:32.459 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 45896, 2024-09-11T01:51:32.459 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:51:32.459 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:51:32.459 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:51:32.459 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:32.459 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:51:32.459 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:51:32.460 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:51:32.460 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:51:32.460 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:32.460 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:32.460 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:32.460 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:32.460 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:32.460 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:32.460 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:32.460 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:32.460 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:32.461 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:32.461 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:32.461 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:32.461 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:51:32.461 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:51:32.461 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:32.461 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:32.461 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:32.461 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:32.461 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:32.461 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:32.461 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:32.462 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:32.462 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:32.462 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:32.462 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:32.462 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:32.462 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:32.462 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:51:32.462 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:51:32.462 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:51:32.462 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:51:32.462 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:51:32.462 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:51:32.462 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:51:32.463 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:51:32.463 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:51:32.463 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:51:32.463 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:51:32.463 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:51:32.463 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:51:32.463 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:51:32.463 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:51:32.463 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:51:32.463 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:51:32.463 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:32.463 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:32.464 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:32.464 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:32.464 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:32.464 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:32.464 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:32.464 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:32.464 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:32.464 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:32.464 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:32.464 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:32.464 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:51:32.465 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:32.465 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:51:32.465 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:32.465 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:51:32.465 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:51:32.465 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:32.465 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:32.465 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:32.465 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:32.465 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:51:32.465 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:32.465 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:32.465 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:32.466 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:32.466 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:51:32.466 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:32.466 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:32.466 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:32.466 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:32.466 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:51:32.466 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:51:32.466 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:32.466 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:32.466 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:32.466 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:32.466 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:32.466 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:51:32.467 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:51:32.467 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:32.467 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:32.467 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:32.467 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:32.467 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:51:32.467 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:32.467 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:32.468 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:32.468 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:32.468 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:51:32.468 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:32.468 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:32.468 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:32.468 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:32.468 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:51:32.468 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:51:32.468 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:32.468 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:32.468 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:32.468 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:32.469 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:32.469 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:51:32.469 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:51:32.469 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:32.469 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:32.469 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:32.469 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:32.469 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:51:32.469 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:32.469 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:32.469 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:32.470 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:32.470 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:51:32.470 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:32.470 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:32.470 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:32.470 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:32.470 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:51:32.470 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:51:32.470 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:32.470 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:32.470 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:32.470 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:32.470 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:32.471 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:32.471 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:51:32.471 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:51:32.471 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:32.471 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:32.471 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:32.471 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:51:32.471 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:32.471 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:32.471 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:51:32.471 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:32.471 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:32.471 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:32.471 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:51:32.472 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:32.472 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:32.472 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:51:32.472 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:32.472 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:32.472 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:32.472 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:51:32.472 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:32.472 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:32.472 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:51:32.472 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:32.472 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:32.472 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:32.473 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:51:32.473 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:32.473 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:32.473 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:32.473 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:51:32.473 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:51:32.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.448+0000 7f74413c2640 1 -- 172.21.15.12:0/2515176716 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f741c03e1c0 msgr2=0x7f741c040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:32.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.448+0000 7f74413c2640 1 --2- 172.21.15.12:0/2515176716 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f741c03e1c0 0x7f741c040680 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f742c007fd0 tx=0x7f742c007c20 comp rx=0 tx=0).stop 2024-09-11T01:51:32.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.448+0000 7f74413c2640 10 monclient: shutdown 2024-09-11T01:51:32.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.448+0000 7f74413c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:32.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.448+0000 7f74413c2640 1 -- 172.21.15.12:0/2515176716 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f743c0a06d0 msgr2=0x7f743c13a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:32.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.448+0000 7f74413c2640 1 --2- 172.21.15.12:0/2515176716 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f743c0a06d0 0x7f743c13a1e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f74240090b0 tx=0x7f74240046c0 comp rx=0 tx=0).stop 2024-09-11T01:51:32.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.448+0000 7f743b7fe640 1 -- 172.21.15.12:0/2515176716 reap_dead start 2024-09-11T01:51:32.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.448+0000 7f74413c2640 1 -- 172.21.15.12:0/2515176716 shutdown_connections 2024-09-11T01:51:32.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.448+0000 7f74413c2640 1 --2- 172.21.15.12:0/2515176716 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f743c0a06d0 0x7f743c13a1e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:32.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.448+0000 7f74413c2640 1 -- 172.21.15.12:0/2515176716 >> 172.21.15.12:0/2515176716 conn(0x7f743c0a36c0 msgr2=0x7f743c0734b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:32.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.449+0000 7f74413c2640 1 -- 172.21.15.12:0/2515176716 shutdown_connections 2024-09-11T01:51:32.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.449+0000 7f74413c2640 1 -- 172.21.15.12:0/2515176716 wait complete. 2024-09-11T01:51:32.475 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:51:32.475 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T01:51:32.476 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:51:32.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.576+0000 7f913f4bd640 10 monclient: get_monmap_and_config 2024-09-11T01:51:32.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.576+0000 7f913f4bd640 10 monclient: build_initial_monmap 2024-09-11T01:51:32.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.576+0000 7f913f4bd640 10 monclient: monmap: 2024-09-11T01:51:32.578 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:32.578 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:32.578 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:51:32.576873+0000 2024-09-11T01:51:32.578 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:51:32.576873+0000 2024-09-11T01:51:32.578 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:32.578 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:32.578 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:51:32.578 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:51:32.578 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:51:32.578 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:51:32.578 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:51:32.578 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:32.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.576+0000 7f913f4bd640 1 Processor -- start 2024-09-11T01:51:32.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.576+0000 7f913f4bd640 1 -- start start 2024-09-11T01:51:32.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.576+0000 7f913f4bd640 10 monclient: init 2024-09-11T01:51:32.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913f4bd640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:32.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913f4bd640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-11T01:51:32.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913f4bd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9138137e50 addr v1:172.21.15.12:6789/0 2024-09-11T01:51:32.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913f4bd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9138138230 0x7f9138138660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:32.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913f4bd640 10 monclient(hunting): picked mon.noname-b con 0x7f9138138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:32.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913f4bd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f913813d380 addr v1:172.21.15.71:6789/0 2024-09-11T01:51:32.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913f4bd640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91380a6580 con 0x7f9138137e50 2024-09-11T01:51:32.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913f4bd640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91380a62d0 con 0x7f913813d380 2024-09-11T01:51:32.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913f4bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:32.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913f4bd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f913809bb90 con 0x7f9138138230 2024-09-11T01:51:32.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913f4bd640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:32.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913f4bd640 10 monclient(hunting): authenticate will time out at 2099.019480s 2024-09-11T01:51:32.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913d259640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7f9138137e50 legacy=0x7f913809d160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:51:32.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913ca58640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9138138230 0x7f9138138660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:32.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913ca58640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9138138230 0x7f9138138660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41108/0 (socket says 172.21.15.12:41108) 2024-09-11T01:51:32.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913ca58640 1 -- 172.21.15.12:0/2763242171 learned_addr learned my addr 172.21.15.12:0/2763242171 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:32.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913ca58640 10 monclient(hunting): get_auth_request con 0x7f9138138230 auth_method 0 2024-09-11T01:51:32.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913ca58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:32.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913ca58640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:32.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913ca58640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:32.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913ca58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:32.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913ca58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:32.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913ca58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:32.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913ca58640 10 monclient(hunting): handle_auth_done global_id 244187 payload 274 2024-09-11T01:51:32.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913ca58640 1 -- 172.21.15.12:0/2763242171 >> v1:172.21.15.71:6789/0 conn(0x7f913813d380 legacy=0x7f913813f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:32.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913ca58640 1 -- 172.21.15.12:0/2763242171 >> v1:172.21.15.12:6789/0 conn(0x7f9138137e50 legacy=0x7f913809d160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:51:32.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913ca58640 10 monclient: _finish_hunting 0 2024-09-11T01:51:32.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913ca58640 1 monclient: found mon.noname-b 2024-09-11T01:51:32.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913ca58640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:32.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913ca58640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:51:32.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.577+0000 7f913ca58640 1 -- 172.21.15.12:0/2763242171 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f913809c4f0 con 0x7f9138138230 2024-09-11T01:51:32.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.580+0000 7f913ca58640 1 -- 172.21.15.12:0/2763242171 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9138138230 msgr2=0x7f9138138660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:32.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.580+0000 7f913ca58640 1 --2- 172.21.15.12:0/2763242171 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9138138230 0x7f9138138660 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f91280098a0 tx=0x7f912802a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=990d69a21af0d75d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:32.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.580+0000 7f912ffff640 1 -- 172.21.15.12:0/2763242171 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9128037070 con 0x7f9138138230 2024-09-11T01:51:32.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.580+0000 7f912ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:32.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.580+0000 7f912ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:51:32.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.580+0000 7f912ffff640 10 monclient: dump: 2024-09-11T01:51:32.588 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:32.588 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:32.588 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.588 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.588 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:32.588 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:32.588 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:32.588 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:32.588 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:32.588 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:32.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.580+0000 7f912ffff640 10 monclient: _finish_auth 0 2024-09-11T01:51:32.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.580+0000 7f912ffff640 10 monclient: _check_auth_tickets 2024-09-11T01:51:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.580+0000 7f912ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.580+0000 7f912ffff640 1 -- 172.21.15.12:0/2763242171 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9128032640 con 0x7f9138138230 2024-09-11T01:51:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.580+0000 7f913f4bd640 5 monclient: authenticate success, global_id 244187 2024-09-11T01:51:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.580+0000 7f913f4bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:51:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.580+0000 7f912ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.580+0000 7f913f4bd640 10 monclient: get_monmap_and_config success 2024-09-11T01:51:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.580+0000 7f912ffff640 1 -- 172.21.15.12:0/2763242171 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9128032a10 con 0x7f9138138230 2024-09-11T01:51:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.580+0000 7f913f4bd640 4 set_mon_vals no callback set 2024-09-11T01:51:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.580+0000 7f912ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.580+0000 7f912ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.580+0000 7f912ffff640 10 monclient: dump: 2024-09-11T01:51:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:32.589 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:32.590 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.581+0000 7f913f4bd640 10 monclient: shutdown 2024-09-11T01:51:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.581+0000 7f913f4bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.581+0000 7f913f4bd640 1 -- 172.21.15.12:0/2763242171 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9138138230 msgr2=0x7f9138138660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.581+0000 7f913f4bd640 1 --2- 172.21.15.12:0/2763242171 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9138138230 0x7f9138138660 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f91280098a0 tx=0x7f912802a080 comp rx=0 tx=0).stop 2024-09-11T01:51:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.581+0000 7f913f4bd640 1 -- 172.21.15.12:0/2763242171 shutdown_connections 2024-09-11T01:51:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.581+0000 7f913f4bd640 1 --2- 172.21.15.12:0/2763242171 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9138138230 0x7f9138138660 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:32.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.581+0000 7f913f4bd640 1 -- 172.21.15.12:0/2763242171 >> 172.21.15.12:0/2763242171 conn(0x7f91380a7020 msgr2=0x7f91380a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.581+0000 7f913f4bd640 1 -- 172.21.15.12:0/2763242171 shutdown_connections 2024-09-11T01:51:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.581+0000 7f913f4bd640 1 -- 172.21.15.12:0/2763242171 wait complete. 2024-09-11T01:51:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.582+0000 7f913f4bd640 10 monclient: build_initial_monmap 2024-09-11T01:51:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.582+0000 7f913f4bd640 10 monclient: monmap: 2024-09-11T01:51:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:51:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:51:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:51:32.591 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:32.592 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:32.592 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:51:32.592 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:51:32.592 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:51:32.592 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:32.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.582+0000 7f913f4bd640 1 Processor -- start 2024-09-11T01:51:32.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.582+0000 7f913f4bd640 1 -- start start 2024-09-11T01:51:32.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.582+0000 7f913f4bd640 10 monclient: init 2024-09-11T01:51:32.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.582+0000 7f913f4bd640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:32.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.582+0000 7f913f4bd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:51:32.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.582+0000 7f913f4bd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9138137e50 0x7f913813ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:32.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.582+0000 7f913f4bd640 10 monclient(hunting): picked mon.noname-b con 0x7f9138137e50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:51:32.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.582+0000 7f913f4bd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9138138230 0x7f9138139900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:32.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.582+0000 7f913f4bd640 10 monclient(hunting): picked mon.noname-c con 0x7f9138138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:32.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.582+0000 7f913f4bd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f913813d380 0x7f91381370b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:32.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.582+0000 7f913f4bd640 10 monclient(hunting): picked mon.noname-a con 0x7f913813d380 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:51:32.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.582+0000 7f913f4bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:32.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.582+0000 7f913f4bd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f913806f890 con 0x7f913813d380 2024-09-11T01:51:32.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.582+0000 7f913f4bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:32.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.582+0000 7f913f4bd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f913806f710 con 0x7f9138137e50 2024-09-11T01:51:32.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.582+0000 7f913f4bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:32.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.582+0000 7f913f4bd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f9138140e90 con 0x7f9138138230 2024-09-11T01:51:32.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.582+0000 7f913f4bd640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:32.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.582+0000 7f913f4bd640 10 monclient(hunting): authenticate will time out at 2099.024935s 2024-09-11T01:51:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.582+0000 7f913da5a640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f913813d380 msgr2=0x7f91381370b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:51:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.582+0000 7f913da5a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f913813d380 0x7f91381370b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:51:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.583+0000 7f913ca58640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9138138230 0x7f9138139900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.583+0000 7f913ca58640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9138138230 0x7f9138139900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41110/0 (socket says 172.21.15.12:41110) 2024-09-11T01:51:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.583+0000 7f913ca58640 1 -- 172.21.15.12:0/3408654138 learned_addr learned my addr 172.21.15.12:0/3408654138 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.583+0000 7f913ca58640 10 monclient(hunting): get_auth_request con 0x7f9138138230 auth_method 0 2024-09-11T01:51:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.583+0000 7f913ca58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.583+0000 7f913ca58640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.583+0000 7f913ca58640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.583+0000 7f913ca58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.583+0000 7f913ca58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.583+0000 7f913ca58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.583+0000 7f913ca58640 10 monclient(hunting): handle_auth_done global_id 244190 payload 824 2024-09-11T01:51:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.583+0000 7f913ca58640 1 -- 172.21.15.12:0/3408654138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9138137e50 msgr2=0x7f913813ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.583+0000 7f913d259640 1 --2- 172.21.15.12:0/3408654138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9138137e50 0x7f913813ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.583+0000 7f913ca58640 1 --2- 172.21.15.12:0/3408654138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9138137e50 0x7f913813ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.583+0000 7f913ca58640 1 -- 172.21.15.12:0/3408654138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f913813d380 msgr2=0x7f91381370b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:51:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.583+0000 7f913ca58640 1 --2- 172.21.15.12:0/3408654138 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f913813d380 0x7f91381370b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.583+0000 7f913ca58640 10 monclient: _finish_hunting 0 2024-09-11T01:51:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.583+0000 7f913ca58640 1 monclient: found mon.noname-c 2024-09-11T01:51:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.583+0000 7f913ca58640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.583+0000 7f913ca58640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:51:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.583+0000 7f913ca58640 1 -- 172.21.15.12:0/3408654138 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f913813c3d0 con 0x7f9138138230 2024-09-11T01:51:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.583+0000 7f913d259640 10 monclient: get_auth_request con 0x7f9138137e50 auth_method 0 2024-09-11T01:51:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.583+0000 7f913d259640 1 --2- 172.21.15.12:0/3408654138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9138137e50 0x7f913813ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:51:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.585+0000 7f913ca58640 1 -- 172.21.15.12:0/3408654138 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9138138230 msgr2=0x7f9138139900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:32.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.585+0000 7f913ca58640 1 --2- 172.21.15.12:0/3408654138 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9138138230 0x7f9138139900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f91280097f0 tx=0x7f91280046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:32.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.585+0000 7f912dffb640 1 -- 172.21.15.12:0/3408654138 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9128041070 con 0x7f9138138230 2024-09-11T01:51:32.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.585+0000 7f912dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:32.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.585+0000 7f913ca58640 1 --2- 172.21.15.12:0/3408654138 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9138138230 0x7f9138139900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f91280097f0 tx=0x7f91280046c0 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7f912803ca80 config(0 keys) 2024-09-11T01:51:32.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.585+0000 7f912dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:51:32.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.585+0000 7f912dffb640 10 monclient: dump: 2024-09-11T01:51:32.596 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:32.596 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:32.596 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.596 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.596 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:32.596 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:32.596 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:32.596 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:32.596 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:32.597 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:32.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.585+0000 7f912dffb640 10 monclient: _finish_auth 0 2024-09-11T01:51:32.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.585+0000 7f912dffb640 10 monclient: _check_auth_tickets 2024-09-11T01:51:32.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.585+0000 7f912dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:32.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.585+0000 7f913f4bd640 5 monclient: authenticate success, global_id 244190 2024-09-11T01:51:32.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.585+0000 7f912dffb640 1 -- 172.21.15.12:0/3408654138 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f912803ca80 con 0x7f9138138230 2024-09-11T01:51:32.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.585+0000 7f913f4bd640 10 monclient: _renew_subs 2024-09-11T01:51:32.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.585+0000 7f913f4bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:32.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.585+0000 7f913f4bd640 1 -- 172.21.15.12:0/3408654138 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f913813a8b0 con 0x7f9138138230 2024-09-11T01:51:32.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.585+0000 7f912dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:32.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.585+0000 7f913f4bd640 10 monclient: _renew_subs 2024-09-11T01:51:32.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.585+0000 7f913f4bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:32.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.586+0000 7f913f4bd640 1 -- 172.21.15.12:0/3408654138 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f913813aee0 con 0x7f9138138230 2024-09-11T01:51:32.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.585+0000 7f912f7fe640 4 set_mon_vals no callback set 2024-09-11T01:51:32.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.586+0000 7f913f4bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:32.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.586+0000 7f913f4bd640 1 --2- 172.21.15.12:0/3408654138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91381cda20 0x7f913802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:32.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.586+0000 7f913f4bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:32.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.586+0000 7f913f4bd640 1 -- 172.21.15.12:0/3408654138 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f9138138660 con 0x7f91381cda20 2024-09-11T01:51:32.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.586+0000 7f912dffb640 1 -- 172.21.15.12:0/3408654138 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9128037040 con 0x7f9138138230 2024-09-11T01:51:32.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.586+0000 7f912dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:32.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.586+0000 7f912dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:32.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.586+0000 7f913d259640 1 --2- 172.21.15.12:0/3408654138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91381cda20 0x7f913802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:32.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.586+0000 7f912dffb640 10 monclient: dump: 2024-09-11T01:51:32.602 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:32.602 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:32.602 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.602 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.602 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:32.602 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:32.602 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:32.602 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:32.602 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:32.602 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:32.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.586+0000 7f912dffb640 1 -- 172.21.15.12:0/3408654138 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f912803d3f0 con 0x7f9138138230 2024-09-11T01:51:32.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.586+0000 7f913d259640 10 monclient: get_auth_request con 0x7f91381cda20 auth_method 0 2024-09-11T01:51:32.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.586+0000 7f913d259640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:32.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.586+0000 7f913d259640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:32.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.586+0000 7f913d259640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:32.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.586+0000 7f913d259640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:32.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.586+0000 7f913d259640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:32.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.586+0000 7f913d259640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:32.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.587+0000 7f913d259640 10 monclient(hunting): handle_auth_done global_id 234211 payload 824 2024-09-11T01:51:32.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.587+0000 7f913d259640 1 -- 172.21.15.12:0/3408654138 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9138140590 con 0x7f91381cda20 2024-09-11T01:51:32.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.587+0000 7f912dffb640 1 --2- 172.21.15.12:0/3408654138 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f911c03e1c0 0x7f911c040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:32.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.587+0000 7f913da5a640 1 --2- 172.21.15.12:0/3408654138 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f911c03e1c0 0x7f911c040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:32.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.587+0000 7f913da5a640 10 monclient: get_auth_request con 0x7f911c03e1c0 auth_method 0 2024-09-11T01:51:32.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.589+0000 7f913d259640 1 -- 172.21.15.12:0/3408654138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91381cda20 msgr2=0x7f913802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:32.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.589+0000 7f913d259640 1 --2- 172.21.15.12:0/3408654138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91381cda20 0x7f913802d440 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f913400c6f0 tx=0x7f913400cbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:32.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.589+0000 7f912dffb640 1 -- 172.21.15.12:0/3408654138 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9134003dc0 con 0x7f91381cda20 2024-09-11T01:51:32.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.589+0000 7f912dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:32.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.589+0000 7f912dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:51:32.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.589+0000 7f912dffb640 10 monclient: dump: 2024-09-11T01:51:32.604 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:32.604 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:32.604 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.604 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.604 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:32.604 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:32.604 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:32.604 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:32.604 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:32.604 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:32.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.589+0000 7f913da5a640 1 --2- 172.21.15.12:0/3408654138 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f911c03e1c0 0x7f911c040680 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f913813b180 tx=0x7f9130007d50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:32.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.589+0000 7f912dffb640 1 -- 172.21.15.12:0/3408654138 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9134003dc0 con 0x7f91381cda20 2024-09-11T01:51:32.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.589+0000 7f912dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:32.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.589+0000 7f912dffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:51:32.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.589+0000 7f912dffb640 1 -- 172.21.15.12:0/3408654138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91381cda20 msgr2=0x7f913802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:32.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.589+0000 7f912dffb640 1 --2- 172.21.15.12:0/3408654138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91381cda20 0x7f913802d440 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f913400c6f0 tx=0x7f913400cbb0 comp rx=0 tx=0).stop 2024-09-11T01:51:32.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.589+0000 7f912dffb640 1 -- 172.21.15.12:0/3408654138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91381cda20 msgr2=0x7f913802d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:51:32.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.589+0000 7f912dffb640 1 --2- 172.21.15.12:0/3408654138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91381cda20 0x7f913802d440 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:32.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.596+0000 7f913f4bd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:51:32.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.596+0000 7f913f4bd640 1 --2- 172.21.15.12:0/3408654138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91381cde00 0x7f913802d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:32.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.596+0000 7f913f4bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:32.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.596+0000 7f913f4bd640 1 -- 172.21.15.12:0/3408654138 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f913813cec0 con 0x7f91381cde00 2024-09-11T01:51:32.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.596+0000 7f913d259640 1 --2- 172.21.15.12:0/3408654138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91381cde00 0x7f913802d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:32.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.596+0000 7f913d259640 10 monclient: get_auth_request con 0x7f91381cde00 auth_method 0 2024-09-11T01:51:32.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.596+0000 7f913d259640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:32.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.596+0000 7f913d259640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:32.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.596+0000 7f913d259640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:32.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.596+0000 7f913d259640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:32.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.596+0000 7f913d259640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:32.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.596+0000 7f913d259640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:32.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.596+0000 7f913d259640 10 monclient(hunting): handle_auth_done global_id 234214 payload 824 2024-09-11T01:51:32.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.596+0000 7f913d259640 1 -- 172.21.15.12:0/3408654138 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f913813d070 con 0x7f91381cde00 2024-09-11T01:51:32.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.598+0000 7f913d259640 1 -- 172.21.15.12:0/3408654138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91381cde00 msgr2=0x7f913802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:32.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.598+0000 7f913d259640 1 --2- 172.21.15.12:0/3408654138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91381cde00 0x7f913802d980 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f913400c6f0 tx=0x7f91340059d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:32.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.599+0000 7f912dffb640 1 -- 172.21.15.12:0/3408654138 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f913401c320 con 0x7f91381cde00 2024-09-11T01:51:32.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.599+0000 7f912dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:32.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.599+0000 7f912dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:51:32.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.599+0000 7f912dffb640 10 monclient: dump: 2024-09-11T01:51:32.607 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:32.607 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:32.607 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.607 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:32.607 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:32.607 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:32.607 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:32.607 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:32.607 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:32.607 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:32.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.599+0000 7f912dffb640 1 -- 172.21.15.12:0/3408654138 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4699 (secure 0 0 0) 0x7f913401c320 con 0x7f91381cde00 2024-09-11T01:51:32.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.599+0000 7f912dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:51:32.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.599+0000 7f912dffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:51:32.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.599+0000 7f912dffb640 1 -- 172.21.15.12:0/3408654138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91381cde00 msgr2=0x7f913802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:32.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.599+0000 7f912dffb640 1 --2- 172.21.15.12:0/3408654138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91381cde00 0x7f913802d980 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f913400c6f0 tx=0x7f91340059d0 comp rx=0 tx=0).stop 2024-09-11T01:51:32.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.599+0000 7f912dffb640 1 -- 172.21.15.12:0/3408654138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91381cde00 msgr2=0x7f913802d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:51:32.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.599+0000 7f912dffb640 1 --2- 172.21.15.12:0/3408654138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91381cde00 0x7f913802d980 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:32.608 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:51:32.608 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:51:32.608 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:51:32.609 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:51:32.609 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 210, 2024-09-11T01:51:32.609 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 94491, 2024-09-11T01:51:32.609 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:51:32.609 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:51:32.609 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:51:32.609 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:32.609 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:51:32.609 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:51:32.609 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:51:32.609 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:51:32.609 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:32.610 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:32.610 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:32.610 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:32.610 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:32.610 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:32.610 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:32.610 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:32.610 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:32.610 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:32.610 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:32.610 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:32.610 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:51:32.610 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:51:32.610 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:32.611 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:32.611 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:32.611 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:32.611 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:32.611 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:32.611 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:32.611 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:32.611 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:32.611 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:32.611 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:32.611 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:32.611 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:32.611 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:51:32.611 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:51:32.612 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:51:32.612 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:51:32.612 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:51:32.612 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:51:32.612 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:51:32.612 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:51:32.612 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:51:32.612 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:51:32.612 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:51:32.612 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:51:32.612 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:51:32.612 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:51:32.612 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:51:32.613 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:51:32.613 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:51:32.613 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:32.613 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:32.613 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:32.613 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:32.613 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:32.613 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:32.613 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:32.613 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:32.613 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:32.613 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:32.613 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:32.613 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:32.614 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:51:32.614 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:32.614 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:51:32.614 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:32.614 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:51:32.614 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:51:32.614 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:32.614 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:32.614 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:32.614 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:32.614 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:51:32.614 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:32.614 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:32.614 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:32.615 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:32.615 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:51:32.615 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:32.615 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:32.615 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:32.615 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:32.615 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:51:32.615 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:51:32.615 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:32.615 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:32.615 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:32.615 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:32.615 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:32.615 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:51:32.616 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:51:32.616 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:32.616 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:32.616 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:32.616 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:32.616 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:51:32.616 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:32.616 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:32.616 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:32.616 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:32.616 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:51:32.616 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:32.616 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:32.617 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:32.617 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:32.617 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:51:32.617 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:51:32.617 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:32.617 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:32.617 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:32.617 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:32.617 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:32.617 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:51:32.617 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:51:32.617 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:32.617 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:32.617 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:32.618 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:32.618 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:51:32.618 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:32.618 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:32.618 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:32.618 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:32.618 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:51:32.618 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:32.618 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:32.618 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:32.618 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:32.618 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:51:32.619 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:51:32.619 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:32.619 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:32.619 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:32.619 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:32.619 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:32.619 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:32.619 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:51:32.619 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:51:32.619 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:32.619 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:32.619 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:32.619 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:51:32.619 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:32.620 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:32.620 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:51:32.620 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:32.620 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:32.620 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:32.620 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:51:32.620 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:32.620 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:32.620 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:51:32.620 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:32.620 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:32.620 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:32.620 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:51:32.621 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:32.621 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:32.621 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:32.621 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:51:32.621 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:51:32.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.599+0000 7f913f4bd640 1 -- 172.21.15.12:0/3408654138 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f911c03e1c0 msgr2=0x7f911c040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:32.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.599+0000 7f913f4bd640 1 --2- 172.21.15.12:0/3408654138 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f911c03e1c0 0x7f911c040680 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f913813b180 tx=0x7f9130007d50 comp rx=0 tx=0).stop 2024-09-11T01:51:32.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.599+0000 7f913f4bd640 10 monclient: shutdown 2024-09-11T01:51:32.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.599+0000 7f913f4bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:32.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.599+0000 7f913f4bd640 1 -- 172.21.15.12:0/3408654138 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9138138230 msgr2=0x7f9138139900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:32.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.599+0000 7f913f4bd640 1 --2- 172.21.15.12:0/3408654138 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9138138230 0x7f9138139900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f91280097f0 tx=0x7f91280046c0 comp rx=0 tx=0).stop 2024-09-11T01:51:32.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.599+0000 7f913da5a640 1 -- 172.21.15.12:0/3408654138 reap_dead start 2024-09-11T01:51:32.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.599+0000 7f912dffb640 1 -- 172.21.15.12:0/3408654138 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(736..736 src has 676..736) ==== 4771+0+0 (unknown 0 0 0) 0x7f9128073aa0 con 0x7f9138138230 2024-09-11T01:51:32.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.599+0000 7f913da5a640 1 -- 172.21.15.12:0/3408654138 reap_dead start 2024-09-11T01:51:32.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.599+0000 7f913f4bd640 1 -- 172.21.15.12:0/3408654138 shutdown_connections 2024-09-11T01:51:32.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.599+0000 7f913f4bd640 1 -- 172.21.15.12:0/3408654138 >> 172.21.15.12:0/3408654138 conn(0x7f91380a7020 msgr2=0x7f913813d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:32.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.599+0000 7f913f4bd640 1 -- 172.21.15.12:0/3408654138 shutdown_connections 2024-09-11T01:51:32.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:32.599+0000 7f913f4bd640 1 -- 172.21.15.12:0/3408654138 wait complete. 2024-09-11T01:51:32.623 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:51:32.624 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:51:32.691 INFO:tasks.rados.rados.0.smithi071.stdout:3290: done (3 left) 2024-09-11T01:51:32.691 INFO:tasks.rados.rados.0.smithi071.stdout:3291: done (2 left) 2024-09-11T01:51:32.691 INFO:tasks.rados.rados.0.smithi071.stdout:3292: done (1 left) 2024-09-11T01:51:32.691 INFO:tasks.rados.rados.0.smithi071.stdout:3293: done (0 left) 2024-09-11T01:51:32.691 INFO:tasks.rados.rados.0.smithi071.stdout:3294: write (excl) oid 50 current snap is 368 2024-09-11T01:51:32.691 INFO:tasks.rados.rados.0.smithi071.stdout:3294: seq_num 1155 ranges {612085=481898,1788114=675727,3035086=376999} 2024-09-11T01:51:32.696 INFO:tasks.rados.rados.0.smithi071.stdout:3294: writing smithi07134777-50 from 612085 to 1093983 tid 1 2024-09-11T01:51:32.705 INFO:tasks.rados.rados.0.smithi071.stdout:3294: writing smithi07134777-50 from 1788114 to 2463841 tid 2 2024-09-11T01:51:32.710 INFO:tasks.rados.rados.0.smithi071.stdout:3294: writing smithi07134777-50 from 3035086 to 3412085 tid 3 2024-09-11T01:51:32.710 INFO:tasks.rados.rados.0.smithi071.stdout:3295: snap_remove snap 358 2024-09-11T01:51:33.128 INFO:tasks.rados.rados.0.smithi071.stdout:3295: done (1 left) 2024-09-11T01:51:33.128 INFO:tasks.rados.rados.0.smithi071.stdout:3296: snap_create 2024-09-11T01:51:33.128 INFO:tasks.rados.rados.0.smithi071.stdout:3294: finishing write tid 1 to smithi07134777-50 2024-09-11T01:51:33.128 INFO:tasks.rados.rados.0.smithi071.stdout:3294: finishing write tid 2 to smithi07134777-50 2024-09-11T01:51:33.128 INFO:tasks.rados.rados.0.smithi071.stdout:3294: finishing write tid 3 to smithi07134777-50 2024-09-11T01:51:33.128 INFO:tasks.rados.rados.0.smithi071.stdout:3294: finishing write tid 4 to smithi07134777-50 2024-09-11T01:51:33.129 INFO:tasks.rados.rados.0.smithi071.stdout:3294: finishing write tid 5 to smithi07134777-50 2024-09-11T01:51:33.129 INFO:tasks.rados.rados.0.smithi071.stdout:3294: oid 50 updating version 0 to 2421 2024-09-11T01:51:33.129 INFO:tasks.rados.rados.0.smithi071.stdout:3294: oid 50 version 2421 is already newer than 2419 2024-09-11T01:51:33.129 INFO:tasks.rados.rados.0.smithi071.stdout:3294: oid 50 version 2421 is already newer than 2420 2024-09-11T01:51:33.129 INFO:tasks.rados.rados.0.smithi071.stdout:3294: oid 50 version 2421 is already newer than 2418 2024-09-11T01:51:33.129 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 2421 (ObjNum 1155 snap 368 seq_num 1155) dirty exists 2024-09-11T01:51:33.129 INFO:tasks.rados.rados.0.smithi071.stdout:3294: left oid 50 (ObjNum 1155 snap 368 seq_num 1155) 2024-09-11T01:51:34.138 INFO:tasks.rados.rados.0.smithi071.stdout:3294: done (1 left) 2024-09-11T01:51:34.138 INFO:tasks.rados.rados.0.smithi071.stdout:3296: done (0 left) 2024-09-11T01:51:34.138 INFO:tasks.rados.rados.0.smithi071.stdout:3297: snap_remove snap 360 2024-09-11T01:51:35.141 INFO:tasks.rados.rados.0.smithi071.stdout:3297: done (0 left) 2024-09-11T01:51:35.141 INFO:tasks.rados.rados.0.smithi071.stdout:3298: read oid 30 snap -1 2024-09-11T01:51:35.141 INFO:tasks.rados.rados.0.smithi071.stdout:3298: expect (ObjNum 1003 snap 322 seq_num 1003) 2024-09-11T01:51:35.141 INFO:tasks.rados.rados.0.smithi071.stdout:3299: write (excl) oid 31 current snap is 369 2024-09-11T01:51:35.141 INFO:tasks.rados.rados.0.smithi071.stdout:3299: seq_num 1156 ranges {628892=673500,1962949=535738,3178862=250030} 2024-09-11T01:51:35.149 INFO:tasks.rados.rados.0.smithi071.stdout:3299: writing smithi07134777-31 from 628892 to 1302392 tid 1 2024-09-11T01:51:35.156 INFO:tasks.rados.rados.0.smithi071.stdout:3299: writing smithi07134777-31 from 1962949 to 2498687 tid 2 2024-09-11T01:51:35.162 INFO:tasks.rados.rados.0.smithi071.stdout:3299: writing smithi07134777-31 from 3178862 to 3428892 tid 3 2024-09-11T01:51:35.163 INFO:tasks.rados.rados.0.smithi071.stdout:3300: delete oid 9 current snap is 369 2024-09-11T01:51:35.163 INFO:tasks.rados.rados.0.smithi071.stdout:3298: expect (ObjNum 1003 snap 322 seq_num 1003) 2024-09-11T01:51:35.163 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:51:35.209 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 804029 2024-09-11T01:51:35.283 INFO:tasks.rados.rados.0.smithi071.stdout:3299: finishing write tid 1 to smithi07134777-31 2024-09-11T01:51:35.284 INFO:tasks.rados.rados.0.smithi071.stdout:3299: finishing write tid 2 to smithi07134777-31 2024-09-11T01:51:35.284 INFO:tasks.rados.rados.0.smithi071.stdout:3299: finishing write tid 3 to smithi07134777-31 2024-09-11T01:51:35.284 INFO:tasks.rados.rados.0.smithi071.stdout:3299: finishing write tid 4 to smithi07134777-31 2024-09-11T01:51:35.284 INFO:tasks.rados.rados.0.smithi071.stdout:3299: finishing write tid 5 to smithi07134777-31 2024-09-11T01:51:35.284 INFO:tasks.rados.rados.0.smithi071.stdout:3299: oid 31 updating version 0 to 1904 2024-09-11T01:51:35.284 INFO:tasks.rados.rados.0.smithi071.stdout:3299: oid 31 updating version 1904 to 1909 2024-09-11T01:51:35.284 INFO:tasks.rados.rados.0.smithi071.stdout:3299: oid 31 updating version 1909 to 1911 2024-09-11T01:51:35.284 INFO:tasks.rados.rados.0.smithi071.stdout:3299: oid 31 version 1911 is already newer than 1910 2024-09-11T01:51:35.285 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 1911 (ObjNum 1156 snap 369 seq_num 1156) dirty exists 2024-09-11T01:51:35.285 INFO:tasks.rados.rados.0.smithi071.stdout:3299: left oid 31 (ObjNum 1156 snap 369 seq_num 1156) 2024-09-11T01:51:35.285 INFO:tasks.rados.rados.0.smithi071.stdout:3298: done (2 left) 2024-09-11T01:51:35.285 INFO:tasks.rados.rados.0.smithi071.stdout:3299: done (1 left) 2024-09-11T01:51:35.285 INFO:tasks.rados.rados.0.smithi071.stdout:3300: done (0 left) 2024-09-11T01:51:35.285 INFO:tasks.rados.rados.0.smithi071.stdout:3301: write (excl) oid 6 current snap is 369 2024-09-11T01:51:35.285 INFO:tasks.rados.rados.0.smithi071.stdout:3301: seq_num 1158 ranges {662506=423998,1563560=539407,2717355=490285} 2024-09-11T01:51:35.288 INFO:tasks.rados.rados.0.smithi071.stdout:3301: writing smithi07134777-6 from 662506 to 1086504 tid 1 2024-09-11T01:51:35.294 INFO:tasks.rados.rados.0.smithi071.stdout:3301: writing smithi07134777-6 from 1563560 to 2102967 tid 2 2024-09-11T01:51:35.300 INFO:tasks.rados.rados.0.smithi071.stdout:3301: writing smithi07134777-6 from 2717355 to 3207640 tid 3 2024-09-11T01:51:35.300 INFO:tasks.rados.rados.0.smithi071.stdout:3302: snap_create 2024-09-11T01:51:35.301 INFO:tasks.rados.rados.0.smithi071.stdout:3301: finishing write tid 1 to smithi07134777-6 2024-09-11T01:51:35.301 INFO:tasks.rados.rados.0.smithi071.stdout:3301: finishing write tid 2 to smithi07134777-6 2024-09-11T01:51:35.305 INFO:tasks.rados.rados.0.smithi071.stdout:3301: finishing write tid 3 to smithi07134777-6 2024-09-11T01:51:35.306 INFO:tasks.rados.rados.0.smithi071.stdout:3301: finishing write tid 4 to smithi07134777-6 2024-09-11T01:51:35.306 INFO:tasks.rados.rados.0.smithi071.stdout:3301: finishing write tid 5 to smithi07134777-6 2024-09-11T01:51:35.306 INFO:tasks.rados.rados.0.smithi071.stdout:3301: oid 6 updating version 0 to 2219 2024-09-11T01:51:35.306 INFO:tasks.rados.rados.0.smithi071.stdout:3301: oid 6 version 2219 is already newer than 2217 2024-09-11T01:51:35.306 INFO:tasks.rados.rados.0.smithi071.stdout:3301: oid 6 version 2219 is already newer than 2216 2024-09-11T01:51:35.306 INFO:tasks.rados.rados.0.smithi071.stdout:3301: oid 6 version 2219 is already newer than 2218 2024-09-11T01:51:35.306 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 2219 (ObjNum 1158 snap 369 seq_num 1158) dirty exists 2024-09-11T01:51:35.306 INFO:tasks.rados.rados.0.smithi071.stdout:3301: left oid 6 (ObjNum 1158 snap 369 seq_num 1158) 2024-09-11T01:51:36.142 INFO:tasks.rados.rados.0.smithi071.stdout:3301: done (1 left) 2024-09-11T01:51:36.142 INFO:tasks.rados.rados.0.smithi071.stdout:3302: done (0 left) 2024-09-11T01:51:36.142 INFO:tasks.rados.rados.0.smithi071.stdout:3303: read oid 17 snap -1 2024-09-11T01:51:36.142 INFO:tasks.rados.rados.0.smithi071.stdout:3303: expect deleted 2024-09-11T01:51:36.142 INFO:tasks.rados.rados.0.smithi071.stdout:3304: copy_from oid 3 from oid 38 current snap is 370 2024-09-11T01:51:36.142 INFO:tasks.rados.rados.0.smithi071.stdout:3305: snap_remove snap 307 2024-09-11T01:51:37.144 INFO:tasks.rados.rados.0.smithi071.stdout:3305: done (2 left) 2024-09-11T01:51:37.144 INFO:tasks.rados.rados.0.smithi071.stdout:3306: write oid 35 current snap is 370 2024-09-11T01:51:37.144 INFO:tasks.rados.rados.0.smithi071.stdout:3306: seq_num 1159 ranges {679313=499247,1622042=683065,2744778=701557} 2024-09-11T01:51:37.150 INFO:tasks.rados.rados.0.smithi071.stdout:3306: writing smithi07134777-35 from 679313 to 1178560 tid 1 2024-09-11T01:51:37.157 INFO:tasks.rados.rados.0.smithi071.stdout:3306: writing smithi07134777-35 from 1622042 to 2305107 tid 2 2024-09-11T01:51:37.164 INFO:tasks.rados.rados.0.smithi071.stdout:3306: writing smithi07134777-35 from 2744778 to 3446335 tid 3 2024-09-11T01:51:37.164 INFO:tasks.rados.rados.0.smithi071.stdout:3307: read oid 41 snap 357 2024-09-11T01:51:37.164 INFO:tasks.rados.rados.0.smithi071.stdout:3307: expect (ObjNum 1073 snap 346 seq_num 1073) 2024-09-11T01:51:37.164 INFO:tasks.rados.rados.0.smithi071.stdout:3308: rollback oid 21 current snap is 370 2024-09-11T01:51:37.164 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 21 to 369 2024-09-11T01:51:37.164 INFO:tasks.rados.rados.0.smithi071.stdout:3309: write oid 23 current snap is 370 2024-09-11T01:51:37.164 INFO:tasks.rados.rados.0.smithi071.stdout:3309: seq_num 1160 ranges {696120=574496,1680524=543076,2888554=512829} 2024-09-11T01:51:37.171 INFO:tasks.rados.rados.0.smithi071.stdout:3309: writing smithi07134777-23 from 696120 to 1270616 tid 1 2024-09-11T01:51:37.177 INFO:tasks.rados.rados.0.smithi071.stdout:3309: writing smithi07134777-23 from 1680524 to 2223600 tid 2 2024-09-11T01:51:37.185 INFO:tasks.rados.rados.0.smithi071.stdout:3309: writing smithi07134777-23 from 2888554 to 3401383 tid 3 2024-09-11T01:51:37.185 INFO:tasks.rados.rados.0.smithi071.stdout:3310: write (excl) oid 36 current snap is 370 2024-09-11T01:51:37.185 INFO:tasks.rados.rados.0.smithi071.stdout:3310: seq_num 1161 ranges {712927=649745,2139006=686734} 2024-09-11T01:51:37.193 INFO:tasks.rados.rados.0.smithi071.stdout:3310: writing smithi07134777-36 from 712927 to 1362672 tid 1 2024-09-11T01:51:37.201 INFO:tasks.rados.rados.0.smithi071.stdout:3310: writing smithi07134777-36 from 2139006 to 2825740 tid 2 2024-09-11T01:51:37.201 INFO:tasks.rados.rados.0.smithi071.stdout:3311: snap_remove snap 356 2024-09-11T01:51:38.163 INFO:tasks.rados.rados.0.smithi071.stdout:3311: done (7 left) 2024-09-11T01:51:38.163 INFO:tasks.rados.rados.0.smithi071.stdout:3312: rollback oid 4 current snap is 370 2024-09-11T01:51:38.164 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 4 to 362 2024-09-11T01:51:38.164 INFO:tasks.rados.rados.0.smithi071.stdout:3304: finishing copy_from racing read to smithi07134777-3 2024-09-11T01:51:38.164 INFO:tasks.rados.rados.0.smithi071.stdout:3304: finishing copy_from to smithi07134777-3 2024-09-11T01:51:38.164 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 2407 (ObjNum 1150 snap 366 seq_num 1150) dirty exists 2024-09-11T01:51:38.164 INFO:tasks.rados.rados.0.smithi071.stdout:3306: finishing write tid 1 to smithi07134777-35 2024-09-11T01:51:38.164 INFO:tasks.rados.rados.0.smithi071.stdout:3306: finishing write tid 2 to smithi07134777-35 2024-09-11T01:51:38.164 INFO:tasks.rados.rados.0.smithi071.stdout:3306: finishing write tid 3 to smithi07134777-35 2024-09-11T01:51:38.164 INFO:tasks.rados.rados.0.smithi071.stdout:3306: finishing write tid 4 to smithi07134777-35 2024-09-11T01:51:38.164 INFO:tasks.rados.rados.0.smithi071.stdout:3306: finishing write tid 5 to smithi07134777-35 2024-09-11T01:51:38.164 INFO:tasks.rados.rados.0.smithi071.stdout:3306: oid 35 updating version 0 to 1974 2024-09-11T01:51:38.164 INFO:tasks.rados.rados.0.smithi071.stdout:3306: oid 35 updating version 1974 to 1975 2024-09-11T01:51:38.164 INFO:tasks.rados.rados.0.smithi071.stdout:3306: oid 35 version 1975 is already newer than 1973 2024-09-11T01:51:38.164 INFO:tasks.rados.rados.0.smithi071.stdout:3306: oid 35 updating version 1975 to 1976 2024-09-11T01:51:38.165 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 1976 (ObjNum 1159 snap 370 seq_num 1159) dirty exists 2024-09-11T01:51:38.165 INFO:tasks.rados.rados.0.smithi071.stdout:3306: left oid 35 (ObjNum 1159 snap 370 seq_num 1159) 2024-09-11T01:51:38.165 INFO:tasks.rados.rados.0.smithi071.stdout:3308: finishing rollback tid 0 to smithi07134777-21 2024-09-11T01:51:38.165 INFO:tasks.rados.rados.0.smithi071.stdout:3308: finishing rollback tid 1 to smithi07134777-21 2024-09-11T01:51:38.165 INFO:tasks.rados.rados.0.smithi071.stdout:3308: finishing rollback tid 2 to smithi07134777-21 2024-09-11T01:51:38.165 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 2076 (ObjNum 1080 snap 346 seq_num 1080) dirty exists 2024-09-11T01:51:38.165 INFO:tasks.rados.rados.0.smithi071.stdout:3307: expect (ObjNum 1073 snap 346 seq_num 1073) 2024-09-11T01:51:38.213 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1617547 2024-09-11T01:51:38.223 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:51:38.290 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:51:38.359 INFO:tasks.rados.rados.0.smithi071.stdout:3309: finishing write tid 1 to smithi07134777-23 2024-09-11T01:51:38.359 INFO:tasks.rados.rados.0.smithi071.stdout:3309: finishing write tid 2 to smithi07134777-23 2024-09-11T01:51:38.359 INFO:tasks.rados.rados.0.smithi071.stdout:3309: finishing write tid 3 to smithi07134777-23 2024-09-11T01:51:38.359 INFO:tasks.rados.rados.0.smithi071.stdout:3303: done (7 left) 2024-09-11T01:51:38.360 INFO:tasks.rados.rados.0.smithi071.stdout:3304: done (6 left) 2024-09-11T01:51:38.360 INFO:tasks.rados.rados.0.smithi071.stdout:3306: done (5 left) 2024-09-11T01:51:38.360 INFO:tasks.rados.rados.0.smithi071.stdout:3307: done (4 left) 2024-09-11T01:51:38.360 INFO:tasks.rados.rados.0.smithi071.stdout:3308: done (3 left) 2024-09-11T01:51:38.360 INFO:tasks.rados.rados.0.smithi071.stdout:3313: copy_from oid 40 from oid 8 current snap is 370 2024-09-11T01:51:38.360 INFO:tasks.rados.rados.0.smithi071.stdout:3309: finishing write tid 4 to smithi07134777-23 2024-09-11T01:51:38.360 INFO:tasks.rados.rados.0.smithi071.stdout:3309: finishing write tid 5 to smithi07134777-23 2024-09-11T01:51:38.361 INFO:tasks.rados.rados.0.smithi071.stdout:3309: oid 23 updating version 0 to 1115 2024-09-11T01:51:38.361 INFO:tasks.rados.rados.0.smithi071.stdout:3309: oid 23 updating version 1115 to 1116 2024-09-11T01:51:38.361 INFO:tasks.rados.rados.0.smithi071.stdout:3309: oid 23 version 1116 is already newer than 1109 2024-09-11T01:51:38.361 INFO:tasks.rados.rados.0.smithi071.stdout:3309: oid 23 version 1116 is already newer than 1114 2024-09-11T01:51:38.361 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 1116 (ObjNum 1160 snap 370 seq_num 1160) dirty exists 2024-09-11T01:51:38.361 INFO:tasks.rados.rados.0.smithi071.stdout:3309: left oid 23 (ObjNum 1160 snap 370 seq_num 1160) 2024-09-11T01:51:38.361 INFO:tasks.rados.rados.0.smithi071.stdout:3310: finishing write tid 1 to smithi07134777-36 2024-09-11T01:51:38.361 INFO:tasks.rados.rados.0.smithi071.stdout:3310: finishing write tid 2 to smithi07134777-36 2024-09-11T01:51:38.361 INFO:tasks.rados.rados.0.smithi071.stdout:3310: finishing write tid 3 to smithi07134777-36 2024-09-11T01:51:38.362 INFO:tasks.rados.rados.0.smithi071.stdout:3310: finishing write tid 4 to smithi07134777-36 2024-09-11T01:51:38.362 INFO:tasks.rados.rados.0.smithi071.stdout:3310: oid 36 updating version 0 to 1797 2024-09-11T01:51:38.362 INFO:tasks.rados.rados.0.smithi071.stdout:3310: oid 36 updating version 1797 to 1798 2024-09-11T01:51:38.362 INFO:tasks.rados.rados.0.smithi071.stdout:3310: oid 36 updating version 1798 to 1799 2024-09-11T01:51:38.362 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 1799 (ObjNum 1161 snap 370 seq_num 1161) dirty exists 2024-09-11T01:51:38.362 INFO:tasks.rados.rados.0.smithi071.stdout:3310: left oid 36 (ObjNum 1161 snap 370 seq_num 1161) 2024-09-11T01:51:38.362 INFO:tasks.rados.rados.0.smithi071.stdout:3312: finishing rollback tid 1 to smithi07134777-4 2024-09-11T01:51:38.362 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 2184 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-11T01:51:38.362 INFO:tasks.rados.rados.0.smithi071.stdout:3309: done (3 left) 2024-09-11T01:51:38.362 INFO:tasks.rados.rados.0.smithi071.stdout:3310: done (2 left) 2024-09-11T01:51:38.362 INFO:tasks.rados.rados.0.smithi071.stdout:3312: done (1 left) 2024-09-11T01:51:38.363 INFO:tasks.rados.rados.0.smithi071.stdout:3314: copy_from oid 5 from oid 49 current snap is 370 2024-09-11T01:51:38.363 INFO:tasks.rados.rados.0.smithi071.stdout:3315: write (excl) oid 38 current snap is 370 2024-09-11T01:51:38.363 INFO:tasks.rados.rados.0.smithi071.stdout:3315: seq_num 1162 ranges {729734=724994,2197488=430392,3227047=302687} 2024-09-11T01:51:38.365 INFO:tasks.rados.rados.0.smithi071.stdout:3315: writing smithi07134777-38 from 729734 to 1454728 tid 1 2024-09-11T01:51:38.372 INFO:tasks.rados.rados.0.smithi071.stdout:3315: writing smithi07134777-38 from 2197488 to 2627880 tid 2 2024-09-11T01:51:38.377 INFO:tasks.rados.rados.0.smithi071.stdout:3315: writing smithi07134777-38 from 3227047 to 3529734 tid 3 2024-09-11T01:51:38.377 INFO:tasks.rados.rados.0.smithi071.stdout:3316: delete oid 15 current snap is 370 2024-09-11T01:51:38.377 INFO:tasks.rados.rados.0.smithi071.stdout:3314: finishing copy_from to smithi07134777-5 2024-09-11T01:51:38.377 INFO:tasks.rados.rados.0.smithi071.stdout:3314: got expected ENOENT (src dne) 2024-09-11T01:51:38.377 INFO:tasks.rados.rados.0.smithi071.stdout:3313: finishing copy_from to smithi07134777-40 2024-09-11T01:51:38.378 INFO:tasks.rados.rados.0.smithi071.stdout:3313: got expected ENOENT (src dne) 2024-09-11T01:51:38.378 INFO:tasks.rados.rados.0.smithi071.stdout:3313: finishing copy_from racing read to smithi07134777-40 2024-09-11T01:51:38.378 INFO:tasks.rados.rados.0.smithi071.stdout:3314: finishing copy_from racing read to smithi07134777-5 2024-09-11T01:51:38.378 INFO:tasks.rados.rados.0.smithi071.stdout:3315: finishing write tid 1 to smithi07134777-38 2024-09-11T01:51:38.378 INFO:tasks.rados.rados.0.smithi071.stdout:3315: finishing write tid 2 to smithi07134777-38 2024-09-11T01:51:38.383 INFO:tasks.rados.rados.0.smithi071.stdout:3315: finishing write tid 3 to smithi07134777-38 2024-09-11T01:51:38.383 INFO:tasks.rados.rados.0.smithi071.stdout:3315: finishing write tid 4 to smithi07134777-38 2024-09-11T01:51:38.383 INFO:tasks.rados.rados.0.smithi071.stdout:3315: finishing write tid 5 to smithi07134777-38 2024-09-11T01:51:38.383 INFO:tasks.rados.rados.0.smithi071.stdout:3315: oid 38 updating version 0 to 2452 2024-09-11T01:51:38.383 INFO:tasks.rados.rados.0.smithi071.stdout:3315: oid 38 version 2452 is already newer than 2451 2024-09-11T01:51:38.383 INFO:tasks.rados.rados.0.smithi071.stdout:3315: oid 38 version 2452 is already newer than 2450 2024-09-11T01:51:38.383 INFO:tasks.rados.rados.0.smithi071.stdout:3315: oid 38 version 2452 is already newer than 2449 2024-09-11T01:51:38.383 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 2452 (ObjNum 1162 snap 370 seq_num 1162) dirty exists 2024-09-11T01:51:38.384 INFO:tasks.rados.rados.0.smithi071.stdout:3315: left oid 38 (ObjNum 1162 snap 370 seq_num 1162) 2024-09-11T01:51:38.384 INFO:tasks.rados.rados.0.smithi071.stdout:3313: done (3 left) 2024-09-11T01:51:38.384 INFO:tasks.rados.rados.0.smithi071.stdout:3314: done (2 left) 2024-09-11T01:51:38.384 INFO:tasks.rados.rados.0.smithi071.stdout:3315: done (1 left) 2024-09-11T01:51:38.384 INFO:tasks.rados.rados.0.smithi071.stdout:3316: done (0 left) 2024-09-11T01:51:38.384 INFO:tasks.rados.rados.0.smithi071.stdout:3317: snap_create 2024-09-11T01:51:39.167 INFO:tasks.rados.rados.0.smithi071.stdout:3317: done (0 left) 2024-09-11T01:51:39.167 INFO:tasks.rados.rados.0.smithi071.stdout:3318: snap_remove snap 321 2024-09-11T01:51:40.169 INFO:tasks.rados.rados.0.smithi071.stdout:3318: done (0 left) 2024-09-11T01:51:40.169 INFO:tasks.rados.rados.0.smithi071.stdout:3319: snap_remove snap 353 2024-09-11T01:51:41.171 INFO:tasks.rados.rados.0.smithi071.stdout:3319: done (0 left) 2024-09-11T01:51:41.171 INFO:tasks.rados.rados.0.smithi071.stdout:3320: snap_remove snap 236 2024-09-11T01:51:42.176 INFO:tasks.rados.rados.0.smithi071.stdout:3320: done (0 left) 2024-09-11T01:51:42.177 INFO:tasks.rados.rados.0.smithi071.stdout:3321: rollback oid 39 current snap is 371 2024-09-11T01:51:42.177 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 39 to 350 2024-09-11T01:51:42.177 INFO:tasks.rados.rados.0.smithi071.stdout:3322: snap_remove snap 296 2024-09-11T01:51:43.132 INFO:tasks.rados.rados.0.smithi071.stdout:3322: done (1 left) 2024-09-11T01:51:43.132 INFO:tasks.rados.rados.0.smithi071.stdout:3323: snap_create 2024-09-11T01:51:43.132 INFO:tasks.rados.rados.0.smithi071.stdout:3321: finishing rollback tid 0 to smithi07134777-39 2024-09-11T01:51:43.133 INFO:tasks.rados.rados.0.smithi071.stdout:3321: finishing rollback tid 1 to smithi07134777-39 2024-09-11T01:51:43.133 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 1978 (ObjNum 129 snap 0 seq_num 2370585504) dirty dne 2024-09-11T01:51:44.136 INFO:tasks.rados.rados.0.smithi071.stdout:3321: done (1 left) 2024-09-11T01:51:44.136 INFO:tasks.rados.rados.0.smithi071.stdout:3323: done (0 left) 2024-09-11T01:51:44.137 INFO:tasks.rados.rados.0.smithi071.stdout:3324: snap_remove snap 357 2024-09-11T01:51:45.139 INFO:tasks.rados.rados.0.smithi071.stdout:3324: done (0 left) 2024-09-11T01:51:45.139 INFO:tasks.rados.rados.0.smithi071.stdout:3325: read oid 3 snap 364 2024-09-11T01:51:45.139 INFO:tasks.rados.rados.0.smithi071.stdout:3325: expect (ObjNum 1032 snap 331 seq_num 1032) 2024-09-11T01:51:45.139 INFO:tasks.rados.rados.0.smithi071.stdout:3326: read oid 17 snap -1 2024-09-11T01:51:45.139 INFO:tasks.rados.rados.0.smithi071.stdout:3326: expect deleted 2024-09-11T01:51:45.140 INFO:tasks.rados.rados.0.smithi071.stdout:3327: snap_remove snap 343 2024-09-11T01:51:46.148 INFO:tasks.rados.rados.0.smithi071.stdout:3327: done (2 left) 2024-09-11T01:51:46.148 INFO:tasks.rados.rados.0.smithi071.stdout:3328: snap_create 2024-09-11T01:51:46.148 INFO:tasks.rados.rados.0.smithi071.stdout:3325: expect (ObjNum 1032 snap 331 seq_num 1032) 2024-09-11T01:51:46.191 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3242263 2024-09-11T01:51:46.199 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 923050 2024-09-11T01:51:47.149 INFO:tasks.rados.rados.0.smithi071.stdout:3325: done (2 left) 2024-09-11T01:51:47.149 INFO:tasks.rados.rados.0.smithi071.stdout:3326: done (1 left) 2024-09-11T01:51:47.149 INFO:tasks.rados.rados.0.smithi071.stdout:3328: done (0 left) 2024-09-11T01:51:47.149 INFO:tasks.rados.rados.0.smithi071.stdout:3329: copy_from oid 43 from oid 6 current snap is 373 2024-09-11T01:51:47.149 INFO:tasks.rados.rados.0.smithi071.stdout:3330: write oid 18 current snap is 373 2024-09-11T01:51:47.149 INFO:tasks.rados.rados.0.smithi071.stdout:3330: seq_num 1164 ranges {763348=591845,1914452=434061,2881893=674270} 2024-09-11T01:51:47.156 INFO:tasks.rados.rados.0.smithi071.stdout:3330: writing smithi07134777-18 from 763348 to 1355193 tid 1 2024-09-11T01:51:47.161 INFO:tasks.rados.rados.0.smithi071.stdout:3330: writing smithi07134777-18 from 1914452 to 2348513 tid 2 2024-09-11T01:51:47.168 INFO:tasks.rados.rados.0.smithi071.stdout:3330: writing smithi07134777-18 from 2881893 to 3556163 tid 3 2024-09-11T01:51:47.168 INFO:tasks.rados.rados.0.smithi071.stdout:3331: write oid 49 current snap is 373 2024-09-11T01:51:47.168 INFO:tasks.rados.rados.0.smithi071.stdout:3330: finishing write tid 1 to smithi07134777-18 2024-09-11T01:51:47.169 INFO:tasks.rados.rados.0.smithi071.stdout:3330: finishing write tid 2 to smithi07134777-18 2024-09-11T01:51:47.169 INFO:tasks.rados.rados.0.smithi071.stdout:3331: seq_num 1165 ranges {780155=667094,1972934=694072,3309316=270839} 2024-09-11T01:51:47.176 INFO:tasks.rados.rados.0.smithi071.stdout:3331: writing smithi07134777-49 from 780155 to 1447249 tid 1 2024-09-11T01:51:47.183 INFO:tasks.rados.rados.0.smithi071.stdout:3331: writing smithi07134777-49 from 1972934 to 2667006 tid 2 2024-09-11T01:51:47.188 INFO:tasks.rados.rados.0.smithi071.stdout:3331: writing smithi07134777-49 from 3309316 to 3580155 tid 3 2024-09-11T01:51:47.188 INFO:tasks.rados.rados.0.smithi071.stdout:3332: delete oid 12 current snap is 373 2024-09-11T01:51:47.188 INFO:tasks.rados.rados.0.smithi071.stdout:3329: finishing copy_from racing read to smithi07134777-43 2024-09-11T01:51:47.188 INFO:tasks.rados.rados.0.smithi071.stdout:3329: finishing copy_from to smithi07134777-43 2024-09-11T01:51:47.188 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 2219 (ObjNum 1158 snap 369 seq_num 1158) dirty exists 2024-09-11T01:51:47.189 INFO:tasks.rados.rados.0.smithi071.stdout:3330: finishing write tid 3 to smithi07134777-18 2024-09-11T01:51:47.189 INFO:tasks.rados.rados.0.smithi071.stdout:3330: finishing write tid 4 to smithi07134777-18 2024-09-11T01:51:47.189 INFO:tasks.rados.rados.0.smithi071.stdout:3330: finishing write tid 5 to smithi07134777-18 2024-09-11T01:51:47.189 INFO:tasks.rados.rados.0.smithi071.stdout:3330: oid 18 updating version 0 to 2185 2024-09-11T01:51:47.189 INFO:tasks.rados.rados.0.smithi071.stdout:3330: oid 18 updating version 2185 to 2187 2024-09-11T01:51:47.189 INFO:tasks.rados.rados.0.smithi071.stdout:3330: oid 18 version 2187 is already newer than 2186 2024-09-11T01:51:47.189 INFO:tasks.rados.rados.0.smithi071.stdout:3330: oid 18 updating version 2187 to 2188 2024-09-11T01:51:47.189 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 2188 (ObjNum 1164 snap 373 seq_num 1164) dirty exists 2024-09-11T01:51:47.189 INFO:tasks.rados.rados.0.smithi071.stdout:3330: left oid 18 (ObjNum 1164 snap 373 seq_num 1164) 2024-09-11T01:51:47.189 INFO:tasks.rados.rados.0.smithi071.stdout:3331: finishing write tid 1 to smithi07134777-49 2024-09-11T01:51:47.189 INFO:tasks.rados.rados.0.smithi071.stdout:3331: finishing write tid 2 to smithi07134777-49 2024-09-11T01:51:47.190 INFO:tasks.rados.rados.0.smithi071.stdout:3331: finishing write tid 3 to smithi07134777-49 2024-09-11T01:51:47.190 INFO:tasks.rados.rados.0.smithi071.stdout:3331: finishing write tid 4 to smithi07134777-49 2024-09-11T01:51:47.191 INFO:tasks.rados.rados.0.smithi071.stdout:3331: finishing write tid 5 to smithi07134777-49 2024-09-11T01:51:47.191 INFO:tasks.rados.rados.0.smithi071.stdout:3331: oid 49 updating version 0 to 400 2024-09-11T01:51:47.191 INFO:tasks.rados.rados.0.smithi071.stdout:3331: oid 49 version 400 is already newer than 398 2024-09-11T01:51:47.191 INFO:tasks.rados.rados.0.smithi071.stdout:3331: oid 49 version 400 is already newer than 397 2024-09-11T01:51:47.191 INFO:tasks.rados.rados.0.smithi071.stdout:3331: oid 49 version 400 is already newer than 399 2024-09-11T01:51:47.191 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 400 (ObjNum 1165 snap 373 seq_num 1165) dirty exists 2024-09-11T01:51:47.191 INFO:tasks.rados.rados.0.smithi071.stdout:3331: left oid 49 (ObjNum 1165 snap 373 seq_num 1165) 2024-09-11T01:51:47.191 INFO:tasks.rados.rados.0.smithi071.stdout:3329: done (3 left) 2024-09-11T01:51:47.192 INFO:tasks.rados.rados.0.smithi071.stdout:3330: done (2 left) 2024-09-11T01:51:47.192 INFO:tasks.rados.rados.0.smithi071.stdout:3331: done (1 left) 2024-09-11T01:51:47.192 INFO:tasks.rados.rados.0.smithi071.stdout:3332: done (0 left) 2024-09-11T01:51:47.192 INFO:tasks.rados.rados.0.smithi071.stdout:3333: rollback oid 49 current snap is 373 2024-09-11T01:51:47.192 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 49 to 361 2024-09-11T01:51:47.192 INFO:tasks.rados.rados.0.smithi071.stdout:3334: snap_remove snap 368 2024-09-11T01:51:48.132 INFO:tasks.rados.rados.0.smithi071.stdout:3334: done (1 left) 2024-09-11T01:51:48.133 INFO:tasks.rados.rados.0.smithi071.stdout:3335: snap_create 2024-09-11T01:51:48.133 INFO:tasks.rados.rados.0.smithi071.stdout:3333: finishing rollback tid 0 to smithi07134777-49 2024-09-11T01:51:48.133 INFO:tasks.rados.rados.0.smithi071.stdout:3333: finishing rollback tid 1 to smithi07134777-49 2024-09-11T01:51:48.133 INFO:tasks.rados.rados.0.smithi071.stdout:3333: finishing rollback tid 2 to smithi07134777-49 2024-09-11T01:51:48.133 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 403 (ObjNum 1113 snap 355 seq_num 1113) dirty exists 2024-09-11T01:51:49.134 INFO:tasks.rados.rados.0.smithi071.stdout:3333: done (1 left) 2024-09-11T01:51:49.135 INFO:tasks.rados.rados.0.smithi071.stdout:3335: done (0 left) 2024-09-11T01:51:49.135 INFO:tasks.rados.rados.0.smithi071.stdout:3336: write (excl) oid 22 current snap is 374 2024-09-11T01:51:49.135 INFO:tasks.rados.rados.0.smithi071.stdout:3336: seq_num 1167 ranges {413769=417592,1573545=697741,2847809=624439} 2024-09-11T01:51:49.139 INFO:tasks.rados.rados.0.smithi071.stdout:3336: writing smithi07134777-22 from 413769 to 831361 tid 1 2024-09-11T01:51:49.149 INFO:tasks.rados.rados.0.smithi071.stdout:3336: writing smithi07134777-22 from 1573545 to 2271286 tid 2 2024-09-11T01:51:49.154 INFO:tasks.rados.rados.0.smithi071.stdout:3336: writing smithi07134777-22 from 2847809 to 3472248 tid 3 2024-09-11T01:51:49.154 INFO:tasks.rados.rados.0.smithi071.stdout:3337: copy_from oid 28 from oid 35 current snap is 374 2024-09-11T01:51:49.154 INFO:tasks.rados.rados.0.smithi071.stdout:3336: finishing write tid 1 to smithi07134777-22 2024-09-11T01:51:49.154 INFO:tasks.rados.rados.0.smithi071.stdout:3336: finishing write tid 2 to smithi07134777-22 2024-09-11T01:51:49.154 INFO:tasks.rados.rados.0.smithi071.stdout:3338: read oid 23 snap 271 2024-09-11T01:51:49.154 INFO:tasks.rados.rados.0.smithi071.stdout:3338: expect (ObjNum 652 snap 199 seq_num 652) 2024-09-11T01:51:49.154 INFO:tasks.rados.rados.0.smithi071.stdout:3339: snap_create 2024-09-11T01:51:49.157 INFO:tasks.rados.rados.0.smithi071.stdout:3338: expect (ObjNum 652 snap 199 seq_num 652) 2024-09-11T01:51:49.240 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1264765 2024-09-11T01:51:49.309 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1946767 2024-09-11T01:51:49.333 INFO:tasks.rados.rados.0.smithi071.stdout:3336: finishing write tid 3 to smithi07134777-22 2024-09-11T01:51:49.334 INFO:tasks.rados.rados.0.smithi071.stdout:3336: finishing write tid 4 to smithi07134777-22 2024-09-11T01:51:49.334 INFO:tasks.rados.rados.0.smithi071.stdout:3336: finishing write tid 5 to smithi07134777-22 2024-09-11T01:51:49.334 INFO:tasks.rados.rados.0.smithi071.stdout:3336: oid 22 updating version 0 to 2411 2024-09-11T01:51:49.334 INFO:tasks.rados.rados.0.smithi071.stdout:3336: oid 22 version 2411 is already newer than 2408 2024-09-11T01:51:49.334 INFO:tasks.rados.rados.0.smithi071.stdout:3336: oid 22 version 2411 is already newer than 2410 2024-09-11T01:51:49.334 INFO:tasks.rados.rados.0.smithi071.stdout:3336: oid 22 version 2411 is already newer than 2409 2024-09-11T01:51:49.334 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 2411 (ObjNum 1167 snap 374 seq_num 1167) dirty exists 2024-09-11T01:51:49.334 INFO:tasks.rados.rados.0.smithi071.stdout:3336: left oid 22 (ObjNum 1167 snap 374 seq_num 1167) 2024-09-11T01:51:49.334 INFO:tasks.rados.rados.0.smithi071.stdout:3337: finishing copy_from racing read to smithi07134777-28 2024-09-11T01:51:49.334 INFO:tasks.rados.rados.0.smithi071.stdout:3337: finishing copy_from to smithi07134777-28 2024-09-11T01:51:49.334 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 1976 (ObjNum 1159 snap 370 seq_num 1159) dirty exists 2024-09-11T01:51:50.143 INFO:tasks.rados.rados.0.smithi071.stdout:3336: done (3 left) 2024-09-11T01:51:50.144 INFO:tasks.rados.rados.0.smithi071.stdout:3337: done (2 left) 2024-09-11T01:51:50.144 INFO:tasks.rados.rados.0.smithi071.stdout:3338: done (1 left) 2024-09-11T01:51:50.144 INFO:tasks.rados.rados.0.smithi071.stdout:3339: done (0 left) 2024-09-11T01:51:50.144 INFO:tasks.rados.rados.0.smithi071.stdout:3340: write (excl) oid 46 current snap is 375 2024-09-11T01:51:50.144 INFO:tasks.rados.rados.0.smithi071.stdout:3340: seq_num 1168 ranges {430576=492841,1632027=441399,2475232=435711,3365397=265179} 2024-09-11T01:51:50.148 INFO:tasks.rados.rados.0.smithi071.stdout:3340: writing smithi07134777-46 from 430576 to 923417 tid 1 2024-09-11T01:51:50.155 INFO:tasks.rados.rados.0.smithi071.stdout:3340: writing smithi07134777-46 from 1632027 to 2073426 tid 2 2024-09-11T01:51:50.159 INFO:tasks.rados.rados.0.smithi071.stdout:3340: writing smithi07134777-46 from 2475232 to 2910943 tid 3 2024-09-11T01:51:50.164 INFO:tasks.rados.rados.0.smithi071.stdout:3340: writing smithi07134777-46 from 3365397 to 3630576 tid 4 2024-09-11T01:51:50.164 INFO:tasks.rados.rados.0.smithi071.stdout:3341: rollback oid 44 current snap is 375 2024-09-11T01:51:50.164 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 44 to 359 2024-09-11T01:51:50.164 INFO:tasks.rados.rados.0.smithi071.stdout:3340: finishing write tid 1 to smithi07134777-46 2024-09-11T01:51:50.164 INFO:tasks.rados.rados.0.smithi071.stdout:3340: finishing write tid 2 to smithi07134777-46 2024-09-11T01:51:50.164 INFO:tasks.rados.rados.0.smithi071.stdout:3340: finishing write tid 3 to smithi07134777-46 2024-09-11T01:51:50.164 INFO:tasks.rados.rados.0.smithi071.stdout:3342: snap_create 2024-09-11T01:51:50.167 INFO:tasks.rados.rados.0.smithi071.stdout:3341: finishing rollback tid 1 to smithi07134777-44 2024-09-11T01:51:50.167 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 1073 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-11T01:51:50.167 INFO:tasks.rados.rados.0.smithi071.stdout:3340: finishing write tid 4 to smithi07134777-46 2024-09-11T01:51:50.167 INFO:tasks.rados.rados.0.smithi071.stdout:3340: finishing write tid 5 to smithi07134777-46 2024-09-11T01:51:50.167 INFO:tasks.rados.rados.0.smithi071.stdout:3340: finishing write tid 6 to smithi07134777-46 2024-09-11T01:51:50.167 INFO:tasks.rados.rados.0.smithi071.stdout:3340: oid 46 updating version 0 to 1784 2024-09-11T01:51:50.167 INFO:tasks.rados.rados.0.smithi071.stdout:3340: oid 46 version 1784 is already newer than 1783 2024-09-11T01:51:50.167 INFO:tasks.rados.rados.0.smithi071.stdout:3340: oid 46 updating version 1784 to 1785 2024-09-11T01:51:50.167 INFO:tasks.rados.rados.0.smithi071.stdout:3340: oid 46 updating version 1785 to 1786 2024-09-11T01:51:50.167 INFO:tasks.rados.rados.0.smithi071.stdout:3340: oid 46 version 1786 is already newer than 1782 2024-09-11T01:51:50.167 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 1786 (ObjNum 1168 snap 375 seq_num 1168) dirty exists 2024-09-11T01:51:50.168 INFO:tasks.rados.rados.0.smithi071.stdout:3340: left oid 46 (ObjNum 1168 snap 375 seq_num 1168) 2024-09-11T01:51:51.145 INFO:tasks.rados.rados.0.smithi071.stdout:3340: done (2 left) 2024-09-11T01:51:51.145 INFO:tasks.rados.rados.0.smithi071.stdout:3341: done (1 left) 2024-09-11T01:51:51.145 INFO:tasks.rados.rados.0.smithi071.stdout:3342: done (0 left) 2024-09-11T01:51:51.145 INFO:tasks.rados.rados.0.smithi071.stdout:3343: delete oid 19 current snap is 376 2024-09-11T01:51:51.149 INFO:tasks.rados.rados.0.smithi071.stdout:3343: done (0 left) 2024-09-11T01:51:51.149 INFO:tasks.rados.rados.0.smithi071.stdout:3344: write oid 28 current snap is 376 2024-09-11T01:51:51.149 INFO:tasks.rados.rados.0.smithi071.stdout:3344: seq_num 1170 ranges {464190=643339,1748991=445068,2930078=574608} 2024-09-11T01:51:51.155 INFO:tasks.rados.rados.0.smithi071.stdout:3344: writing smithi07134777-28 from 464190 to 1107529 tid 1 2024-09-11T01:51:51.161 INFO:tasks.rados.rados.0.smithi071.stdout:3344: writing smithi07134777-28 from 1748991 to 2194059 tid 2 2024-09-11T01:51:51.169 INFO:tasks.rados.rados.0.smithi071.stdout:3344: writing smithi07134777-28 from 2930078 to 3504686 tid 3 2024-09-11T01:51:51.169 INFO:tasks.rados.rados.0.smithi071.stdout:3345: write oid 12 current snap is 376 2024-09-11T01:51:51.169 INFO:tasks.rados.rados.0.smithi071.stdout:3344: finishing write tid 1 to smithi07134777-28 2024-09-11T01:51:51.169 INFO:tasks.rados.rados.0.smithi071.stdout:3344: finishing write tid 2 to smithi07134777-28 2024-09-11T01:51:51.169 INFO:tasks.rados.rados.0.smithi071.stdout:3345: seq_num 1171 ranges {480997=434941,1407473=705079,2557501=785880} 2024-09-11T01:51:51.174 INFO:tasks.rados.rados.0.smithi071.stdout:3345: writing smithi07134777-12 from 480997 to 915938 tid 1 2024-09-11T01:51:51.180 INFO:tasks.rados.rados.0.smithi071.stdout:3345: writing smithi07134777-12 from 1407473 to 2112552 tid 2 2024-09-11T01:51:51.187 INFO:tasks.rados.rados.0.smithi071.stdout:3345: writing smithi07134777-12 from 2557501 to 3343381 tid 3 2024-09-11T01:51:51.188 INFO:tasks.rados.rados.0.smithi071.stdout:3346: delete oid 15 current snap is 376 2024-09-11T01:51:51.188 INFO:tasks.rados.rados.0.smithi071.stdout:3344: finishing write tid 3 to smithi07134777-28 2024-09-11T01:51:51.188 INFO:tasks.rados.rados.0.smithi071.stdout:3344: finishing write tid 4 to smithi07134777-28 2024-09-11T01:51:51.188 INFO:tasks.rados.rados.0.smithi071.stdout:3344: finishing write tid 5 to smithi07134777-28 2024-09-11T01:51:51.188 INFO:tasks.rados.rados.0.smithi071.stdout:3344: oid 28 updating version 0 to 2079 2024-09-11T01:51:51.188 INFO:tasks.rados.rados.0.smithi071.stdout:3344: oid 28 version 2079 is already newer than 2077 2024-09-11T01:51:51.188 INFO:tasks.rados.rados.0.smithi071.stdout:3344: oid 28 version 2079 is already newer than 2078 2024-09-11T01:51:51.188 INFO:tasks.rados.rados.0.smithi071.stdout:3344: oid 28 updating version 2079 to 2080 2024-09-11T01:51:51.188 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 2080 (ObjNum 1170 snap 376 seq_num 1170) dirty exists 2024-09-11T01:51:51.188 INFO:tasks.rados.rados.0.smithi071.stdout:3344: left oid 28 (ObjNum 1170 snap 376 seq_num 1170) 2024-09-11T01:51:51.188 INFO:tasks.rados.rados.0.smithi071.stdout:3345: finishing write tid 1 to smithi07134777-12 2024-09-11T01:51:51.188 INFO:tasks.rados.rados.0.smithi071.stdout:3345: finishing write tid 2 to smithi07134777-12 2024-09-11T01:51:51.194 INFO:tasks.rados.rados.0.smithi071.stdout:3344: done (2 left) 2024-09-11T01:51:51.194 INFO:tasks.rados.rados.0.smithi071.stdout:3346: done (1 left) 2024-09-11T01:51:51.194 INFO:tasks.rados.rados.0.smithi071.stdout:3347: copy_from oid 43 from oid 7 current snap is 376 2024-09-11T01:51:51.194 INFO:tasks.rados.rados.0.smithi071.stdout:3348: delete oid 35 current snap is 376 2024-09-11T01:51:51.194 INFO:tasks.rados.rados.0.smithi071.stdout:3345: finishing write tid 3 to smithi07134777-12 2024-09-11T01:51:51.194 INFO:tasks.rados.rados.0.smithi071.stdout:3345: finishing write tid 4 to smithi07134777-12 2024-09-11T01:51:51.194 INFO:tasks.rados.rados.0.smithi071.stdout:3345: finishing write tid 5 to smithi07134777-12 2024-09-11T01:51:51.194 INFO:tasks.rados.rados.0.smithi071.stdout:3345: oid 12 updating version 0 to 2306 2024-09-11T01:51:51.194 INFO:tasks.rados.rados.0.smithi071.stdout:3345: oid 12 version 2306 is already newer than 2305 2024-09-11T01:51:51.194 INFO:tasks.rados.rados.0.smithi071.stdout:3345: oid 12 version 2306 is already newer than 2304 2024-09-11T01:51:51.195 INFO:tasks.rados.rados.0.smithi071.stdout:3345: oid 12 version 2306 is already newer than 2303 2024-09-11T01:51:51.195 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 2306 (ObjNum 1171 snap 376 seq_num 1171) dirty exists 2024-09-11T01:51:51.195 INFO:tasks.rados.rados.0.smithi071.stdout:3345: left oid 12 (ObjNum 1171 snap 376 seq_num 1171) 2024-09-11T01:51:51.196 INFO:tasks.rados.rados.0.smithi071.stdout:3345: done (2 left) 2024-09-11T01:51:51.197 INFO:tasks.rados.rados.0.smithi071.stdout:3348: done (1 left) 2024-09-11T01:51:51.197 INFO:tasks.rados.rados.0.smithi071.stdout:3349: write (excl) oid 46 current snap is 376 2024-09-11T01:51:51.197 INFO:tasks.rados.rados.0.smithi071.stdout:3349: seq_num 1174 ranges {531418=660688,1982919=452406,3039770=691648} 2024-09-11T01:51:51.203 INFO:tasks.rados.rados.0.smithi071.stdout:3349: writing smithi07134777-46 from 531418 to 1192106 tid 1 2024-09-11T01:51:51.210 INFO:tasks.rados.rados.0.smithi071.stdout:3349: writing smithi07134777-46 from 1982919 to 2435325 tid 2 2024-09-11T01:51:51.218 INFO:tasks.rados.rados.0.smithi071.stdout:3349: writing smithi07134777-46 from 3039770 to 3731418 tid 3 2024-09-11T01:51:51.218 INFO:tasks.rados.rados.0.smithi071.stdout:3350: write oid 34 current snap is 376 2024-09-11T01:51:51.218 INFO:tasks.rados.rados.0.smithi071.stdout:3347: finishing copy_from racing read to smithi07134777-43 2024-09-11T01:51:51.218 INFO:tasks.rados.rados.0.smithi071.stdout:3347: finishing copy_from to smithi07134777-43 2024-09-11T01:51:51.218 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 2318 (ObjNum 1014 snap 322 seq_num 1014) dirty exists 2024-09-11T01:51:51.218 INFO:tasks.rados.rados.0.smithi071.stdout:3349: finishing write tid 1 to smithi07134777-46 2024-09-11T01:51:51.218 INFO:tasks.rados.rados.0.smithi071.stdout:3349: finishing write tid 2 to smithi07134777-46 2024-09-11T01:51:51.218 INFO:tasks.rados.rados.0.smithi071.stdout:3350: seq_num 1175 ranges {548225=735937,1925048=596064,2950840=547321} 2024-09-11T01:51:51.224 INFO:tasks.rados.rados.0.smithi071.stdout:3350: writing smithi07134777-34 from 548225 to 1284162 tid 1 2024-09-11T01:51:51.231 INFO:tasks.rados.rados.0.smithi071.stdout:3350: writing smithi07134777-34 from 1925048 to 2521112 tid 2 2024-09-11T01:51:51.237 INFO:tasks.rados.rados.0.smithi071.stdout:3350: writing smithi07134777-34 from 2950840 to 3498161 tid 3 2024-09-11T01:51:51.237 INFO:tasks.rados.rados.0.smithi071.stdout:3347: done (2 left) 2024-09-11T01:51:51.237 INFO:tasks.rados.rados.0.smithi071.stdout:3351: rollback oid 48 current snap is 376 2024-09-11T01:51:51.237 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 48 to 344 2024-09-11T01:51:51.237 INFO:tasks.rados.rados.0.smithi071.stdout:3349: finishing write tid 3 to smithi07134777-46 2024-09-11T01:51:51.237 INFO:tasks.rados.rados.0.smithi071.stdout:3349: finishing write tid 4 to smithi07134777-46 2024-09-11T01:51:51.237 INFO:tasks.rados.rados.0.smithi071.stdout:3349: finishing write tid 5 to smithi07134777-46 2024-09-11T01:51:51.237 INFO:tasks.rados.rados.0.smithi071.stdout:3349: oid 46 updating version 0 to 1789 2024-09-11T01:51:51.237 INFO:tasks.rados.rados.0.smithi071.stdout:3349: oid 46 version 1789 is already newer than 1787 2024-09-11T01:51:51.237 INFO:tasks.rados.rados.0.smithi071.stdout:3349: oid 46 updating version 1789 to 1790 2024-09-11T01:51:51.238 INFO:tasks.rados.rados.0.smithi071.stdout:3349: oid 46 version 1790 is already newer than 1788 2024-09-11T01:51:51.238 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 1790 (ObjNum 1174 snap 376 seq_num 1174) dirty exists 2024-09-11T01:51:51.238 INFO:tasks.rados.rados.0.smithi071.stdout:3349: left oid 46 (ObjNum 1174 snap 376 seq_num 1174) 2024-09-11T01:51:51.238 INFO:tasks.rados.rados.0.smithi071.stdout:3350: finishing write tid 1 to smithi07134777-34 2024-09-11T01:51:51.238 INFO:tasks.rados.rados.0.smithi071.stdout:3350: finishing write tid 2 to smithi07134777-34 2024-09-11T01:51:51.238 INFO:tasks.rados.rados.0.smithi071.stdout:3349: done (2 left) 2024-09-11T01:51:51.238 INFO:tasks.rados.rados.0.smithi071.stdout:3352: read oid 37 snap 364 2024-09-11T01:51:51.238 INFO:tasks.rados.rados.0.smithi071.stdout:3352: expect deleted 2024-09-11T01:51:51.239 INFO:tasks.rados.rados.0.smithi071.stdout:3353: snap_remove snap 362 2024-09-11T01:51:52.152 INFO:tasks.rados.rados.0.smithi071.stdout:3353: done (3 left) 2024-09-11T01:51:52.153 INFO:tasks.rados.rados.0.smithi071.stdout:3354: read oid 25 snap -1 2024-09-11T01:51:52.153 INFO:tasks.rados.rados.0.smithi071.stdout:3354: expect (ObjNum 1023 snap 328 seq_num 1023) 2024-09-11T01:51:52.153 INFO:tasks.rados.rados.0.smithi071.stdout:3355: read oid 45 snap -1 2024-09-11T01:51:52.153 INFO:tasks.rados.rados.0.smithi071.stdout:3355: expect deleted 2024-09-11T01:51:52.153 INFO:tasks.rados.rados.0.smithi071.stdout:3356: read oid 26 snap -1 2024-09-11T01:51:52.153 INFO:tasks.rados.rados.0.smithi071.stdout:3356: expect deleted 2024-09-11T01:51:52.153 INFO:tasks.rados.rados.0.smithi071.stdout:3357: write (excl) oid 27 current snap is 376 2024-09-11T01:51:52.153 INFO:tasks.rados.rados.0.smithi071.stdout:3357: seq_num 1176 ranges {565032=411186,1583530=456075,2578263=758593} 2024-09-11T01:51:52.157 INFO:tasks.rados.rados.0.smithi071.stdout:3357: writing smithi07134777-27 from 565032 to 976218 tid 1 2024-09-11T01:51:52.163 INFO:tasks.rados.rados.0.smithi071.stdout:3357: writing smithi07134777-27 from 1583530 to 2039605 tid 2 2024-09-11T01:51:52.172 INFO:tasks.rados.rados.0.smithi071.stdout:3357: writing smithi07134777-27 from 2578263 to 3336856 tid 3 2024-09-11T01:51:52.172 INFO:tasks.rados.rados.0.smithi071.stdout:3358: rollback oid 3 current snap is 376 2024-09-11T01:51:52.172 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 3 to 284 2024-09-11T01:51:52.172 INFO:tasks.rados.rados.0.smithi071.stdout:3351: finishing rollback tid 1 to smithi07134777-48 2024-09-11T01:51:52.172 INFO:tasks.rados.rados.0.smithi071.stdout:3351: finishing rollback tid 2 to smithi07134777-48 2024-09-11T01:51:52.173 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 2525 (ObjNum 875 snap 267 seq_num 875) dirty exists 2024-09-11T01:51:52.173 INFO:tasks.rados.rados.0.smithi071.stdout:3350: finishing write tid 3 to smithi07134777-34 2024-09-11T01:51:52.173 INFO:tasks.rados.rados.0.smithi071.stdout:3350: finishing write tid 4 to smithi07134777-34 2024-09-11T01:51:52.173 INFO:tasks.rados.rados.0.smithi071.stdout:3350: finishing write tid 5 to smithi07134777-34 2024-09-11T01:51:52.173 INFO:tasks.rados.rados.0.smithi071.stdout:3350: oid 34 updating version 0 to 2321 2024-09-11T01:51:52.173 INFO:tasks.rados.rados.0.smithi071.stdout:3350: oid 34 version 2321 is already newer than 2319 2024-09-11T01:51:52.173 INFO:tasks.rados.rados.0.smithi071.stdout:3350: oid 34 updating version 2321 to 2322 2024-09-11T01:51:52.173 INFO:tasks.rados.rados.0.smithi071.stdout:3350: oid 34 version 2322 is already newer than 2320 2024-09-11T01:51:52.173 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 2322 (ObjNum 1175 snap 376 seq_num 1175) dirty exists 2024-09-11T01:51:52.173 INFO:tasks.rados.rados.0.smithi071.stdout:3350: left oid 34 (ObjNum 1175 snap 376 seq_num 1175) 2024-09-11T01:51:52.173 INFO:tasks.rados.rados.0.smithi071.stdout:3354: expect (ObjNum 1023 snap 328 seq_num 1023) 2024-09-11T01:51:52.196 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2767328 2024-09-11T01:51:52.252 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 499164 2024-09-11T01:51:52.295 INFO:tasks.rados.rados.0.smithi071.stdout:3357: finishing write tid 1 to smithi07134777-27 2024-09-11T01:51:52.296 INFO:tasks.rados.rados.0.smithi071.stdout:3350: done (7 left) 2024-09-11T01:51:52.296 INFO:tasks.rados.rados.0.smithi071.stdout:3351: done (6 left) 2024-09-11T01:51:52.296 INFO:tasks.rados.rados.0.smithi071.stdout:3352: done (5 left) 2024-09-11T01:51:52.296 INFO:tasks.rados.rados.0.smithi071.stdout:3354: done (4 left) 2024-09-11T01:51:52.296 INFO:tasks.rados.rados.0.smithi071.stdout:3355: done (3 left) 2024-09-11T01:51:52.296 INFO:tasks.rados.rados.0.smithi071.stdout:3356: done (2 left) 2024-09-11T01:51:52.296 INFO:tasks.rados.rados.0.smithi071.stdout:3359: write (excl) oid 29 current snap is 376 2024-09-11T01:51:52.296 INFO:tasks.rados.rados.0.smithi071.stdout:3357: finishing write tid 2 to smithi07134777-27 2024-09-11T01:51:52.296 INFO:tasks.rados.rados.0.smithi071.stdout:3358: finishing rollback tid 0 to smithi07134777-3 2024-09-11T01:51:52.296 INFO:tasks.rados.rados.0.smithi071.stdout:3358: finishing rollback tid 1 to smithi07134777-3 2024-09-11T01:51:52.296 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 2413 (ObjNum 33 snap 0 seq_num 2372150608) dirty dne 2024-09-11T01:51:52.296 INFO:tasks.rados.rados.0.smithi071.stdout:3357: finishing write tid 3 to smithi07134777-27 2024-09-11T01:51:52.297 INFO:tasks.rados.rados.0.smithi071.stdout:3357: finishing write tid 4 to smithi07134777-27 2024-09-11T01:51:52.297 INFO:tasks.rados.rados.0.smithi071.stdout:3357: finishing write tid 5 to smithi07134777-27 2024-09-11T01:51:52.297 INFO:tasks.rados.rados.0.smithi071.stdout:3357: oid 27 updating version 0 to 2325 2024-09-11T01:51:52.297 INFO:tasks.rados.rados.0.smithi071.stdout:3357: oid 27 version 2325 is already newer than 2324 2024-09-11T01:51:52.297 INFO:tasks.rados.rados.0.smithi071.stdout:3357: oid 27 version 2325 is already newer than 2323 2024-09-11T01:51:52.297 INFO:tasks.rados.rados.0.smithi071.stdout:3357: oid 27 updating version 2325 to 2326 2024-09-11T01:51:52.297 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 2326 (ObjNum 1176 snap 376 seq_num 1176) dirty exists 2024-09-11T01:51:52.297 INFO:tasks.rados.rados.0.smithi071.stdout:3357: left oid 27 (ObjNum 1176 snap 376 seq_num 1176) 2024-09-11T01:51:52.297 INFO:tasks.rados.rados.0.smithi071.stdout:3359: seq_num 1177 ranges {581839=486435,1642012=599733,3005686=569865} 2024-09-11T01:51:52.301 INFO:tasks.rados.rados.0.smithi071.stdout:3359: writing smithi07134777-29 from 581839 to 1068274 tid 1 2024-09-11T01:51:52.307 INFO:tasks.rados.rados.0.smithi071.stdout:3359: writing smithi07134777-29 from 1642012 to 2241745 tid 2 2024-09-11T01:51:52.315 INFO:tasks.rados.rados.0.smithi071.stdout:3359: writing smithi07134777-29 from 3005686 to 3575551 tid 3 2024-09-11T01:51:52.315 INFO:tasks.rados.rados.0.smithi071.stdout:3357: done (2 left) 2024-09-11T01:51:52.315 INFO:tasks.rados.rados.0.smithi071.stdout:3358: done (1 left) 2024-09-11T01:51:52.315 INFO:tasks.rados.rados.0.smithi071.stdout:3360: write (excl) oid 48 current snap is 376 2024-09-11T01:51:52.315 INFO:tasks.rados.rados.0.smithi071.stdout:3359: finishing write tid 1 to smithi07134777-29 2024-09-11T01:51:52.315 INFO:tasks.rados.rados.0.smithi071.stdout:3359: finishing write tid 2 to smithi07134777-29 2024-09-11T01:51:52.315 INFO:tasks.rados.rados.0.smithi071.stdout:3360: seq_num 1178 ranges {598646=561684,1700494=459744,2633109=781137} 2024-09-11T01:51:52.321 INFO:tasks.rados.rados.0.smithi071.stdout:3360: writing smithi07134777-48 from 598646 to 1160330 tid 1 2024-09-11T01:51:52.326 INFO:tasks.rados.rados.0.smithi071.stdout:3360: writing smithi07134777-48 from 1700494 to 2160238 tid 2 2024-09-11T01:51:52.333 INFO:tasks.rados.rados.0.smithi071.stdout:3360: writing smithi07134777-48 from 2633109 to 3414246 tid 3 2024-09-11T01:51:52.334 INFO:tasks.rados.rados.0.smithi071.stdout:3361: write (excl) oid 36 current snap is 376 2024-09-11T01:51:52.334 INFO:tasks.rados.rados.0.smithi071.stdout:3361: seq_num 1179 ranges {615453=753286,2158976=603402,3460532=354921} 2024-09-11T01:51:52.341 INFO:tasks.rados.rados.0.smithi071.stdout:3361: writing smithi07134777-36 from 615453 to 1368739 tid 1 2024-09-11T01:51:52.348 INFO:tasks.rados.rados.0.smithi071.stdout:3361: writing smithi07134777-36 from 2158976 to 2762378 tid 2 2024-09-11T01:51:52.355 INFO:tasks.rados.rados.0.smithi071.stdout:3361: writing smithi07134777-36 from 3460532 to 3815453 tid 3 2024-09-11T01:51:52.355 INFO:tasks.rados.rados.0.smithi071.stdout:3362: snap_create 2024-09-11T01:51:52.355 INFO:tasks.rados.rados.0.smithi071.stdout:3359: finishing write tid 3 to smithi07134777-29 2024-09-11T01:51:52.356 INFO:tasks.rados.rados.0.smithi071.stdout:3359: finishing write tid 4 to smithi07134777-29 2024-09-11T01:51:52.356 INFO:tasks.rados.rados.0.smithi071.stdout:3359: finishing write tid 5 to smithi07134777-29 2024-09-11T01:51:52.356 INFO:tasks.rados.rados.0.smithi071.stdout:3359: oid 29 updating version 0 to 2536 2024-09-11T01:51:52.356 INFO:tasks.rados.rados.0.smithi071.stdout:3359: oid 29 version 2536 is already newer than 2534 2024-09-11T01:51:52.356 INFO:tasks.rados.rados.0.smithi071.stdout:3359: oid 29 updating version 2536 to 2537 2024-09-11T01:51:52.356 INFO:tasks.rados.rados.0.smithi071.stdout:3359: oid 29 version 2537 is already newer than 2535 2024-09-11T01:51:52.356 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 2537 (ObjNum 1177 snap 376 seq_num 1177) dirty exists 2024-09-11T01:51:52.356 INFO:tasks.rados.rados.0.smithi071.stdout:3359: left oid 29 (ObjNum 1177 snap 376 seq_num 1177) 2024-09-11T01:51:52.356 INFO:tasks.rados.rados.0.smithi071.stdout:3360: finishing write tid 1 to smithi07134777-48 2024-09-11T01:51:52.356 INFO:tasks.rados.rados.0.smithi071.stdout:3360: finishing write tid 2 to smithi07134777-48 2024-09-11T01:51:52.356 INFO:tasks.rados.rados.0.smithi071.stdout:3360: finishing write tid 3 to smithi07134777-48 2024-09-11T01:51:52.356 INFO:tasks.rados.rados.0.smithi071.stdout:3360: finishing write tid 4 to smithi07134777-48 2024-09-11T01:51:52.356 INFO:tasks.rados.rados.0.smithi071.stdout:3360: finishing write tid 5 to smithi07134777-48 2024-09-11T01:51:52.357 INFO:tasks.rados.rados.0.smithi071.stdout:3360: oid 48 updating version 0 to 2538 2024-09-11T01:51:52.357 INFO:tasks.rados.rados.0.smithi071.stdout:3360: oid 48 updating version 2538 to 2541 2024-09-11T01:51:52.357 INFO:tasks.rados.rados.0.smithi071.stdout:3360: oid 48 version 2541 is already newer than 2540 2024-09-11T01:51:52.357 INFO:tasks.rados.rados.0.smithi071.stdout:3360: oid 48 version 2541 is already newer than 2539 2024-09-11T01:51:52.357 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 2541 (ObjNum 1178 snap 376 seq_num 1178) dirty exists 2024-09-11T01:51:52.357 INFO:tasks.rados.rados.0.smithi071.stdout:3360: left oid 48 (ObjNum 1178 snap 376 seq_num 1178) 2024-09-11T01:51:52.357 INFO:tasks.rados.rados.0.smithi071.stdout:3361: finishing write tid 1 to smithi07134777-36 2024-09-11T01:51:52.357 INFO:tasks.rados.rados.0.smithi071.stdout:3361: finishing write tid 2 to smithi07134777-36 2024-09-11T01:51:52.358 INFO:tasks.rados.rados.0.smithi071.stdout:3361: finishing write tid 3 to smithi07134777-36 2024-09-11T01:51:52.358 INFO:tasks.rados.rados.0.smithi071.stdout:3361: finishing write tid 4 to smithi07134777-36 2024-09-11T01:51:52.358 INFO:tasks.rados.rados.0.smithi071.stdout:3361: finishing write tid 5 to smithi07134777-36 2024-09-11T01:51:52.358 INFO:tasks.rados.rados.0.smithi071.stdout:3361: oid 36 updating version 0 to 1802 2024-09-11T01:51:52.358 INFO:tasks.rados.rados.0.smithi071.stdout:3361: oid 36 updating version 1802 to 1804 2024-09-11T01:51:52.358 INFO:tasks.rados.rados.0.smithi071.stdout:3361: oid 36 version 1804 is already newer than 1801 2024-09-11T01:51:52.358 INFO:tasks.rados.rados.0.smithi071.stdout:3361: oid 36 version 1804 is already newer than 1803 2024-09-11T01:51:52.359 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 1804 (ObjNum 1179 snap 376 seq_num 1179) dirty exists 2024-09-11T01:51:52.359 INFO:tasks.rados.rados.0.smithi071.stdout:3361: left oid 36 (ObjNum 1179 snap 376 seq_num 1179) 2024-09-11T01:51:52.625 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-11T01:51:52.625 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T01:51:52.625 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T01:51:52.629 INFO:tasks.ceph.mon.a:Started 2024-09-11T01:51:52.629 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:51:52.629 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:51:52.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.740+0000 7f262bb7c640 10 monclient: get_monmap_and_config 2024-09-11T01:51:52.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.740+0000 7f262bb7c640 10 monclient: build_initial_monmap 2024-09-11T01:51:52.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.740+0000 7f262bb7c640 10 monclient: monmap: 2024-09-11T01:51:52.742 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:52.742 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:52.742 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:51:52.741230+0000 2024-09-11T01:51:52.742 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:51:52.741230+0000 2024-09-11T01:51:52.742 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:52.742 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:52.742 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:51:52.742 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:51:52.742 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:51:52.743 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:51:52.743 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:51:52.743 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:52.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.741+0000 7f262bb7c640 1 Processor -- start 2024-09-11T01:51:52.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.741+0000 7f262bb7c640 1 -- start start 2024-09-11T01:51:52.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.741+0000 7f262bb7c640 10 monclient: init 2024-09-11T01:51:52.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.741+0000 7f262bb7c640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:52.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.741+0000 7f262bb7c640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-11T01:51:52.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.741+0000 7f262bb7c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2624138230 0x7f2624138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:52.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.741+0000 7f262bb7c640 10 monclient(hunting): picked mon.noname-a con 0x7f2624138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:51:52.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.741+0000 7f262bb7c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f262409d030 addr v1:172.21.15.12:6789/0 2024-09-11T01:51:52.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.741+0000 7f262bb7c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f262409d460 0x7f26240a5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:52.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.741+0000 7f262bb7c640 10 monclient(hunting): picked mon.noname-b con 0x7f262409d460 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:52.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.741+0000 7f262bb7c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26240a6e30 con 0x7f262409d030 2024-09-11T01:51:52.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.741+0000 7f262bb7c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:52.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.741+0000 7f262bb7c640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f262409c9c0 con 0x7f2624138230 2024-09-11T01:51:52.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.741+0000 7f262bb7c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:52.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.741+0000 7f262bb7c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f262409ba20 con 0x7f262409d460 2024-09-11T01:51:52.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.741+0000 7f262bb7c640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:52.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.741+0000 7f262bb7c640 10 monclient(hunting): authenticate will time out at 2119.183954s 2024-09-11T01:51:52.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f2629918640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2624138230 0x7f2624138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:52.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f262a119640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f262409d460 0x7f26240a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:52.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f2629117640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f262409d030 0x7f26240a5020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50834/0 (socket says 172.21.15.12:50834) 2024-09-11T01:51:52.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f2629918640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2624138230 0x7f2624138610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52952/0 (socket says 172.21.15.12:52952) 2024-09-11T01:51:52.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f262a119640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f262409d460 0x7f26240a5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:54866/0 (socket says 172.21.15.12:54866) 2024-09-11T01:51:52.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f2629117640 1 -- 172.21.15.12:0/2822472086 learned_addr learned my addr 172.21.15.12:0/2822472086 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:51:52.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f2629918640 10 monclient(hunting): get_auth_request con 0x7f2624138230 auth_method 0 2024-09-11T01:51:52.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f2629918640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:52.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f2629918640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:52.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f2629918640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:52.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f262a119640 10 monclient(hunting): get_auth_request con 0x7f262409d460 auth_method 0 2024-09-11T01:51:52.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f262a119640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:52.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f262a119640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:52.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f262a119640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:52.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f262a119640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:52.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f2629918640 1 -- 172.21.15.12:0/2822472086 >> v2:172.21.15.12:3300/0 conn(0x7f2624138230 msgr2=0x7f2624138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:51:52.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f262a119640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:52.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f2629918640 1 -- 172.21.15.12:0/2822472086 >> v2:172.21.15.12:3300/0 conn(0x7f2624138230 msgr2=0x7f2624138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:52.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f2629918640 1 --2- 172.21.15.12:0/2822472086 >> v2:172.21.15.12:3300/0 conn(0x7f2624138230 0x7f2624138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:52.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f262a119640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:52.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f2629918640 1 --2- 172.21.15.12:0/2822472086 >> v2:172.21.15.12:3300/0 conn(0x7f2624138230 0x7f2624138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:51:52.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f262a119640 10 monclient(hunting): handle_auth_done global_id 244193 payload 274 2024-09-11T01:51:52.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f262a119640 1 -- 172.21.15.12:0/2822472086 >> v2:172.21.15.12:3300/0 conn(0x7f2624138230 msgr2=0x7f2624138610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:51:52.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f262a119640 1 --2- 172.21.15.12:0/2822472086 >> v2:172.21.15.12:3300/0 conn(0x7f2624138230 0x7f2624138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:52.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f262a119640 1 -- 172.21.15.12:0/2822472086 >> v1:172.21.15.12:6789/0 conn(0x7f262409d030 legacy=0x7f26240a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:52.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f262a119640 10 monclient: _finish_hunting 0 2024-09-11T01:51:52.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f262a119640 1 monclient: found mon.noname-b 2024-09-11T01:51:52.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f262a119640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:52.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f262a119640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:51:52.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.742+0000 7f262a119640 1 -- 172.21.15.12:0/2822472086 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26240a66a0 con 0x7f262409d460 2024-09-11T01:51:52.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.745+0000 7f262a119640 1 -- 172.21.15.12:0/2822472086 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f262409d460 msgr2=0x7f26240a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:52.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.745+0000 7f262a119640 1 --2- 172.21.15.12:0/2822472086 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f262409d460 0x7f26240a5730 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f26240a17e0 tx=0x7f261c02a460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7f0fa7a9bd8747d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:52.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.745+0000 7f2628916640 1 -- 172.21.15.12:0/2822472086 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261c007950 con 0x7f262409d460 2024-09-11T01:51:52.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.745+0000 7f2628916640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:52.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.745+0000 7f2628916640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:51:52.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.745+0000 7f2628916640 10 monclient: dump: 2024-09-11T01:51:52.754 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:52.755 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:52.755 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:52.755 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:52.755 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:52.755 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:52.755 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:52.755 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:52.755 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:52.755 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:52.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.745+0000 7f2628916640 10 monclient: _finish_auth 0 2024-09-11T01:51:52.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.745+0000 7f2628916640 10 monclient: _check_auth_tickets 2024-09-11T01:51:52.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.745+0000 7f2628916640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:52.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.745+0000 7f2628916640 1 -- 172.21.15.12:0/2822472086 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f261c007af0 con 0x7f262409d460 2024-09-11T01:51:52.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.745+0000 7f262bb7c640 5 monclient: authenticate success, global_id 244193 2024-09-11T01:51:52.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.745+0000 7f262bb7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:51:52.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.745+0000 7f2628916640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:52.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.745+0000 7f262bb7c640 10 monclient: get_monmap_and_config success 2024-09-11T01:51:52.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.745+0000 7f2628916640 1 -- 172.21.15.12:0/2822472086 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261c031400 con 0x7f262409d460 2024-09-11T01:51:52.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.745+0000 7f2628916640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:52.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.745+0000 7f262bb7c640 4 set_mon_vals no callback set 2024-09-11T01:51:52.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.745+0000 7f2628916640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:52.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.745+0000 7f2628916640 10 monclient: dump: 2024-09-11T01:51:52.756 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:52.756 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:52.756 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:52.756 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:52.756 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:52.757 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:52.757 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:52.757 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:52.757 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:52.757 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:52.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.746+0000 7f262bb7c640 10 monclient: shutdown 2024-09-11T01:51:52.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.746+0000 7f262bb7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:52.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.746+0000 7f262bb7c640 1 -- 172.21.15.12:0/2822472086 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f262409d460 msgr2=0x7f26240a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:52.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.746+0000 7f262bb7c640 1 --2- 172.21.15.12:0/2822472086 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f262409d460 0x7f26240a5730 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f26240a17e0 tx=0x7f261c02a460 comp rx=0 tx=0).stop 2024-09-11T01:51:52.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.746+0000 7f262bb7c640 1 -- 172.21.15.12:0/2822472086 shutdown_connections 2024-09-11T01:51:52.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.746+0000 7f262bb7c640 1 --2- 172.21.15.12:0/2822472086 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f262409d460 0x7f26240a5730 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:52.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.746+0000 7f262bb7c640 1 --2- 172.21.15.12:0/2822472086 >> v2:172.21.15.12:3300/0 conn(0x7f2624138230 0x7f2624138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:52.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.746+0000 7f262bb7c640 1 -- 172.21.15.12:0/2822472086 >> 172.21.15.12:0/2822472086 conn(0x7f26240a7120 msgr2=0x7f26240a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:52.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.746+0000 7f262bb7c640 1 -- 172.21.15.12:0/2822472086 shutdown_connections 2024-09-11T01:51:52.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.746+0000 7f262bb7c640 1 -- 172.21.15.12:0/2822472086 wait complete. 2024-09-11T01:51:52.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.746+0000 7f262bb7c640 10 monclient: build_initial_monmap 2024-09-11T01:51:52.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.746+0000 7f262bb7c640 10 monclient: monmap: 2024-09-11T01:51:52.758 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:52.758 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:51:52.758 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:51:52.758 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:51:52.758 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:52.758 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:52.758 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:51:52.758 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:51:52.758 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:51:52.758 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:52.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.747+0000 7f262bb7c640 1 Processor -- start 2024-09-11T01:51:52.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.747+0000 7f262bb7c640 1 -- start start 2024-09-11T01:51:52.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.747+0000 7f262bb7c640 10 monclient: init 2024-09-11T01:51:52.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.747+0000 7f262bb7c640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:52.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.747+0000 7f262bb7c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:51:52.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.747+0000 7f262bb7c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f262409d030 0x7f2624137010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:52.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.747+0000 7f262bb7c640 10 monclient(hunting): picked mon.noname-a con 0x7f262409d030 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:51:52.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.747+0000 7f262bb7c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f262409d460 0x7f2624120c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:52.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.747+0000 7f262bb7c640 10 monclient(hunting): picked mon.noname-b con 0x7f262409d460 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:51:52.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.747+0000 7f262bb7c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2624138230 0x7f26241211b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:52.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.747+0000 7f262bb7c640 10 monclient(hunting): picked mon.noname-c con 0x7f2624138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:52.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.747+0000 7f262bb7c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:52.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.747+0000 7f262bb7c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f262406f970 con 0x7f262409d030 2024-09-11T01:51:52.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.747+0000 7f262bb7c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:52.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.747+0000 7f262bb7c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f262406f7f0 con 0x7f262409d460 2024-09-11T01:51:52.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.747+0000 7f262bb7c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:52.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.747+0000 7f262bb7c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f26240a5c70 con 0x7f2624138230 2024-09-11T01:51:52.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.747+0000 7f262bb7c640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:52.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.747+0000 7f262bb7c640 10 monclient(hunting): authenticate will time out at 2119.189897s 2024-09-11T01:51:52.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.747+0000 7f2629918640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f262409d030 0x7f2624137010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:52.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.747+0000 7f262a119640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2624138230 0x7f26241211b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:52.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f2629918640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f262409d030 0x7f2624137010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52968/0 (socket says 172.21.15.12:52968) 2024-09-11T01:51:52.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f2629918640 1 -- 172.21.15.12:0/751711673 learned_addr learned my addr 172.21.15.12:0/751711673 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:52.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f2629918640 10 monclient(hunting): get_auth_request con 0x7f262409d030 auth_method 0 2024-09-11T01:51:52.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f2629918640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:52.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f2629918640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:52.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f2629918640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:52.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f262a119640 10 monclient(hunting): get_auth_request con 0x7f2624138230 auth_method 0 2024-09-11T01:51:52.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f262a119640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:52.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f262a119640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:52.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f262a119640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:52.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f2629918640 1 -- 172.21.15.12:0/751711673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f262409d030 msgr2=0x7f2624137010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:51:52.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f2629918640 1 -- 172.21.15.12:0/751711673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f262409d030 msgr2=0x7f2624137010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:51:52.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f2629918640 1 --2- 172.21.15.12:0/751711673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f262409d030 0x7f2624137010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:52.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f262a119640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:52.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f262a119640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:52.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f262a119640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:52.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f2629918640 1 --2- 172.21.15.12:0/751711673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f262409d030 0x7f2624137010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:51:52.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f2629117640 1 --2- 172.21.15.12:0/751711673 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f262409d460 0x7f2624120c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:52.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f2629117640 10 monclient(hunting): get_auth_request con 0x7f262409d460 auth_method 0 2024-09-11T01:51:52.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f2629117640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:52.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f2629117640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:52.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f2629117640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:52.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f2629117640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:52.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f2629117640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:52.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f2629117640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:52.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f262a119640 10 monclient(hunting): handle_auth_done global_id 244196 payload 824 2024-09-11T01:51:52.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f262a119640 1 -- 172.21.15.12:0/751711673 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f262409d460 msgr2=0x7f2624120c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:52.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f262a119640 1 --2- 172.21.15.12:0/751711673 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f262409d460 0x7f2624120c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:52.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f262a119640 1 -- 172.21.15.12:0/751711673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f262409d030 msgr2=0x7f2624137010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:51:52.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f262a119640 1 --2- 172.21.15.12:0/751711673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f262409d030 0x7f2624137010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:52.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f262a119640 10 monclient: _finish_hunting 0 2024-09-11T01:51:52.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f262a119640 1 monclient: found mon.noname-c 2024-09-11T01:51:52.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f262a119640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:52.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f262a119640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:51:52.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.748+0000 7f262a119640 1 -- 172.21.15.12:0/751711673 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2624122400 con 0x7f2624138230 2024-09-11T01:51:52.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.750+0000 7f262a119640 1 -- 172.21.15.12:0/751711673 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2624138230 msgr2=0x7f26241211b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:52.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.750+0000 7f262a119640 1 --2- 172.21.15.12:0/751711673 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2624138230 0x7f26241211b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f261c031000 tx=0x7f261c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:52.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.751+0000 7f261a7fc640 1 -- 172.21.15.12:0/751711673 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261c040070 con 0x7f2624138230 2024-09-11T01:51:52.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.751+0000 7f261a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:52.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.751+0000 7f261a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:51:52.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.751+0000 7f261a7fc640 10 monclient: dump: 2024-09-11T01:51:52.763 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:52.763 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:52.763 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:52.763 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:52.763 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:52.763 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:52.764 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:52.764 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:52.764 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:52.764 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:52.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.751+0000 7f261a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:51:52.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.751+0000 7f261a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:51:52.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.751+0000 7f261a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:52.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.751+0000 7f261a7fc640 1 -- 172.21.15.12:0/751711673 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f261c045660 con 0x7f2624138230 2024-09-11T01:51:52.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.751+0000 7f262bb7c640 5 monclient: authenticate success, global_id 244196 2024-09-11T01:51:52.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.751+0000 7f262bb7c640 10 monclient: _renew_subs 2024-09-11T01:51:52.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.751+0000 7f262bb7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:52.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.751+0000 7f262bb7c640 1 -- 172.21.15.12:0/751711673 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2624123ec0 con 0x7f2624138230 2024-09-11T01:51:52.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.751+0000 7f261a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:52.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.751+0000 7f261a7fc640 1 -- 172.21.15.12:0/751711673 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261c0314c0 con 0x7f2624138230 2024-09-11T01:51:52.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.751+0000 7f261a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:52.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.751+0000 7f261a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:52.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.751+0000 7f261a7fc640 10 monclient: dump: 2024-09-11T01:51:52.765 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:52.765 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:52.765 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:52.765 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:52.765 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:52.765 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:52.765 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:52.765 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:52.765 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:52.765 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:52.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.751+0000 7f262bb7c640 10 monclient: _renew_subs 2024-09-11T01:51:52.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.751+0000 7f261bfff640 4 set_mon_vals no callback set 2024-09-11T01:51:52.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.751+0000 7f262bb7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:52.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.751+0000 7f262bb7c640 1 -- 172.21.15.12:0/751711673 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2624124480 con 0x7f2624138230 2024-09-11T01:51:52.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.752+0000 7f262bb7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:52.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.752+0000 7f262bb7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:52.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.752+0000 7f262bb7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:52.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.752+0000 7f261a7fc640 1 -- 172.21.15.12:0/751711673 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f261c04d030 con 0x7f2624138230 2024-09-11T01:51:52.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.752+0000 7f262bb7c640 1 -- 172.21.15.12:0/751711673 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2624138610 con 0x7f2624138230 2024-09-11T01:51:52.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.752+0000 7f261a7fc640 1 --2- 172.21.15.12:0/751711673 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f260803e210 0x7f26080406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:52.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.752+0000 7f261a7fc640 1 -- 172.21.15.12:0/751711673 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(757..757 src has 676..757) ==== 4831+0+0 (secure 0 0 0) 0x7f261c0746e0 con 0x7f2624138230 2024-09-11T01:51:52.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.752+0000 7f2629918640 1 --2- 172.21.15.12:0/751711673 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f260803e210 0x7f26080406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:52.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.752+0000 7f2629918640 10 monclient: get_auth_request con 0x7f260803e210 auth_method 0 2024-09-11T01:51:52.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.755+0000 7f2629918640 1 --2- 172.21.15.12:0/751711673 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f260803e210 0x7f26080406d0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f2624137e20 tx=0x7f2620008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:52.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.757+0000 7f261a7fc640 1 -- 172.21.15.12:0/751711673 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f261c0730b0 con 0x7f2624138230 2024-09-11T01:51:52.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.757+0000 7f261a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:52.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:52.757+0000 7f261a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:51:53.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:53.014+0000 7f262bb7c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:51:53.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:53.014+0000 7f262bb7c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:51:53.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:53.014+0000 7f262bb7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:53.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:53.014+0000 7f262bb7c640 1 -- 172.21.15.12:0/751711673 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2624124800 con 0x7f2624138230 2024-09-11T01:51:53.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:53.014+0000 7f261a7fc640 1 -- 172.21.15.12:0/751711673 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f261c074390 con 0x7f2624138230 2024-09-11T01:51:53.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:53.014+0000 7f261a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:51:53.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:53.014+0000 7f261a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:51:53.016 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":210,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:51:53.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:53.016+0000 7f262bb7c640 1 -- 172.21.15.12:0/751711673 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f260803e210 msgr2=0x7f26080406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:53.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:53.016+0000 7f262bb7c640 1 --2- 172.21.15.12:0/751711673 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f260803e210 0x7f26080406d0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f2624137e20 tx=0x7f2620008040 comp rx=0 tx=0).stop 2024-09-11T01:51:53.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:53.016+0000 7f262bb7c640 10 monclient: shutdown 2024-09-11T01:51:53.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:53.016+0000 7f262bb7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:53.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:53.016+0000 7f262bb7c640 1 -- 172.21.15.12:0/751711673 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2624138230 msgr2=0x7f26241211b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:53.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:53.016+0000 7f262bb7c640 1 --2- 172.21.15.12:0/751711673 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2624138230 0x7f26241211b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f261c031000 tx=0x7f261c004a40 comp rx=0 tx=0).stop 2024-09-11T01:51:53.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:53.016+0000 7f262bb7c640 1 -- 172.21.15.12:0/751711673 shutdown_connections 2024-09-11T01:51:53.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:53.016+0000 7f262bb7c640 1 --2- 172.21.15.12:0/751711673 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2624138230 0x7f26241211b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:53.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:53.016+0000 7f262bb7c640 1 --2- 172.21.15.12:0/751711673 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f262409d460 0x7f2624120c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:53.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:53.016+0000 7f262bb7c640 1 --2- 172.21.15.12:0/751711673 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f260803e210 0x7f26080406d0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:53.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:53.016+0000 7f262bb7c640 1 --2- 172.21.15.12:0/751711673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f262409d030 0x7f2624137010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:53.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:53.016+0000 7f262bb7c640 1 -- 172.21.15.12:0/751711673 >> 172.21.15.12:0/751711673 conn(0x7f26240a7120 msgr2=0x7f26240a3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:53.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:53.016+0000 7f262bb7c640 1 -- 172.21.15.12:0/751711673 shutdown_connections 2024-09-11T01:51:53.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:53.016+0000 7f262bb7c640 1 -- 172.21.15.12:0/751711673 wait complete. 2024-09-11T01:51:53.134 INFO:tasks.rados.rados.0.smithi071.stdout:3359: done (3 left) 2024-09-11T01:51:53.134 INFO:tasks.rados.rados.0.smithi071.stdout:3360: done (2 left) 2024-09-11T01:51:53.134 INFO:tasks.rados.rados.0.smithi071.stdout:3361: done (1 left) 2024-09-11T01:51:53.134 INFO:tasks.rados.rados.0.smithi071.stdout:3362: done (0 left) 2024-09-11T01:51:53.134 INFO:tasks.rados.rados.0.smithi071.stdout:3363: read oid 21 snap -1 2024-09-11T01:51:53.134 INFO:tasks.rados.rados.0.smithi071.stdout:3363: expect (ObjNum 1080 snap 346 seq_num 1080) 2024-09-11T01:51:53.134 INFO:tasks.rados.rados.0.smithi071.stdout:3364: read oid 39 snap -1 2024-09-11T01:51:53.134 INFO:tasks.rados.rados.0.smithi071.stdout:3364: expect deleted 2024-09-11T01:51:53.134 INFO:tasks.rados.rados.0.smithi071.stdout:3365: snap_remove snap 334 2024-09-11T01:51:54.148 INFO:tasks.rados.rados.0.smithi071.stdout:3365: done (2 left) 2024-09-11T01:51:54.148 INFO:tasks.rados.rados.0.smithi071.stdout:3366: read oid 49 snap -1 2024-09-11T01:51:54.148 INFO:tasks.rados.rados.0.smithi071.stdout:3366: expect (ObjNum 1113 snap 355 seq_num 1113) 2024-09-11T01:51:54.148 INFO:tasks.rados.rados.0.smithi071.stdout:3367: write oid 3 current snap is 377 2024-09-11T01:51:54.148 INFO:tasks.rados.rados.0.smithi071.stdout:3367: seq_num 1180 ranges {632260=428535,1817458=463413,2687955=520034,3709324=122936} 2024-09-11T01:51:54.154 INFO:tasks.rados.rados.0.smithi071.stdout:3367: writing smithi07134777-3 from 632260 to 1060795 tid 1 2024-09-11T01:51:54.162 INFO:tasks.rados.rados.0.smithi071.stdout:3367: writing smithi07134777-3 from 1817458 to 2280871 tid 2 2024-09-11T01:51:54.167 INFO:tasks.rados.rados.0.smithi071.stdout:3367: writing smithi07134777-3 from 2687955 to 3207989 tid 3 2024-09-11T01:51:54.171 INFO:tasks.rados.rados.0.smithi071.stdout:3367: writing smithi07134777-3 from 3709324 to 3832260 tid 4 2024-09-11T01:51:54.171 INFO:tasks.rados.rados.0.smithi071.stdout:3368: snap_remove snap 373 2024-09-11T01:51:55.167 INFO:tasks.rados.rados.0.smithi071.stdout:3368: done (4 left) 2024-09-11T01:51:55.167 INFO:tasks.rados.rados.0.smithi071.stdout:3369: copy_from oid 31 from oid 18 current snap is 377 2024-09-11T01:51:55.167 INFO:tasks.rados.rados.0.smithi071.stdout:3363: expect (ObjNum 1080 snap 346 seq_num 1080) 2024-09-11T01:51:55.201 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:51:55.279 INFO:tasks.rados.rados.0.smithi071.stdout:3366: expect (ObjNum 1113 snap 355 seq_num 1113) 2024-09-11T01:51:55.291 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2140560 2024-09-11T01:51:55.337 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2357540 2024-09-11T01:51:55.344 INFO:tasks.rados.rados.0.smithi071.stdout:3367: finishing write tid 1 to smithi07134777-3 2024-09-11T01:51:55.344 INFO:tasks.rados.rados.0.smithi071.stdout:3367: finishing write tid 2 to smithi07134777-3 2024-09-11T01:51:55.344 INFO:tasks.rados.rados.0.smithi071.stdout:3367: finishing write tid 3 to smithi07134777-3 2024-09-11T01:51:55.344 INFO:tasks.rados.rados.0.smithi071.stdout:3367: finishing write tid 4 to smithi07134777-3 2024-09-11T01:51:55.344 INFO:tasks.rados.rados.0.smithi071.stdout:3367: finishing write tid 5 to smithi07134777-3 2024-09-11T01:51:55.344 INFO:tasks.rados.rados.0.smithi071.stdout:3367: finishing write tid 6 to smithi07134777-3 2024-09-11T01:51:55.345 INFO:tasks.rados.rados.0.smithi071.stdout:3367: oid 3 updating version 0 to 2417 2024-09-11T01:51:55.345 INFO:tasks.rados.rados.0.smithi071.stdout:3367: oid 3 version 2417 is already newer than 2414 2024-09-11T01:51:55.345 INFO:tasks.rados.rados.0.smithi071.stdout:3367: oid 3 version 2417 is already newer than 2415 2024-09-11T01:51:55.345 INFO:tasks.rados.rados.0.smithi071.stdout:3367: oid 3 updating version 2417 to 2418 2024-09-11T01:51:55.345 INFO:tasks.rados.rados.0.smithi071.stdout:3367: oid 3 version 2418 is already newer than 2416 2024-09-11T01:51:55.345 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 2418 (ObjNum 1180 snap 377 seq_num 1180) dirty exists 2024-09-11T01:51:55.345 INFO:tasks.rados.rados.0.smithi071.stdout:3367: left oid 3 (ObjNum 1180 snap 377 seq_num 1180) 2024-09-11T01:51:55.345 INFO:tasks.rados.rados.0.smithi071.stdout:3369: finishing copy_from racing read to smithi07134777-31 2024-09-11T01:51:55.345 INFO:tasks.rados.rados.0.smithi071.stdout:3369: finishing copy_from to smithi07134777-31 2024-09-11T01:51:55.345 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 2188 (ObjNum 1164 snap 373 seq_num 1164) dirty exists 2024-09-11T01:51:55.345 INFO:tasks.rados.rados.0.smithi071.stdout:3363: done (4 left) 2024-09-11T01:51:55.345 INFO:tasks.rados.rados.0.smithi071.stdout:3364: done (3 left) 2024-09-11T01:51:55.345 INFO:tasks.rados.rados.0.smithi071.stdout:3366: done (2 left) 2024-09-11T01:51:55.346 INFO:tasks.rados.rados.0.smithi071.stdout:3367: done (1 left) 2024-09-11T01:51:55.346 INFO:tasks.rados.rados.0.smithi071.stdout:3369: done (0 left) 2024-09-11T01:51:55.346 INFO:tasks.rados.rados.0.smithi071.stdout:3370: rollback oid 22 current snap is 377 2024-09-11T01:51:55.346 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 22 to 375 2024-09-11T01:51:55.346 INFO:tasks.rados.rados.0.smithi071.stdout:3371: snap_create 2024-09-11T01:51:55.348 INFO:tasks.rados.rados.0.smithi071.stdout:3370: finishing rollback tid 0 to smithi07134777-22 2024-09-11T01:51:55.348 INFO:tasks.rados.rados.0.smithi071.stdout:3370: finishing rollback tid 1 to smithi07134777-22 2024-09-11T01:51:55.348 INFO:tasks.rados.rados.0.smithi071.stdout:3370: finishing rollback tid 2 to smithi07134777-22 2024-09-11T01:51:55.348 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 2421 (ObjNum 1167 snap 374 seq_num 1167) dirty exists 2024-09-11T01:51:56.030 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:51:56.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.141+0000 7fc7c393d640 10 monclient: get_monmap_and_config 2024-09-11T01:51:56.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.141+0000 7fc7c393d640 10 monclient: build_initial_monmap 2024-09-11T01:51:56.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.141+0000 7fc7c393d640 10 monclient: monmap: 2024-09-11T01:51:56.144 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:56.144 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:56.144 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:51:56.142504+0000 2024-09-11T01:51:56.144 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:51:56.142504+0000 2024-09-11T01:51:56.144 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:56.144 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:56.144 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:51:56.144 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:51:56.144 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:51:56.144 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:51:56.144 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:51:56.144 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:56.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.142+0000 7fc7c393d640 1 Processor -- start 2024-09-11T01:51:56.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.142+0000 7fc7c393d640 1 -- start start 2024-09-11T01:51:56.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.142+0000 7fc7c393d640 10 monclient: init 2024-09-11T01:51:56.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c393d640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:56.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c393d640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-11T01:51:56.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c393d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc7bc0a0c90 addr v1:172.21.15.12:6789/0 2024-09-11T01:51:56.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c393d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc7bc0a2120 0x7fc7bc0a1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:56.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c393d640 10 monclient(hunting): picked mon.noname-c con 0x7fc7bc0a2120 addr v2:172.21.15.71:3300/0 2024-09-11T01:51:56.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c393d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7bc141720 0x7fc7bc143b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:56.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c393d640 10 monclient(hunting): picked mon.noname-b con 0x7fc7bc141720 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:56.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c393d640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7bc09d790 con 0x7fc7bc0a0c90 2024-09-11T01:51:56.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c393d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:56.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c393d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fc7bc09e290 con 0x7fc7bc0a2120 2024-09-11T01:51:56.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c393d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:56.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c393d640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc7bc0a1cc0 con 0x7fc7bc141720 2024-09-11T01:51:56.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c393d640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:56.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c393d640 10 monclient(hunting): authenticate will time out at 2122.585210s 2024-09-11T01:51:56.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c16d9640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc7bc0a0c90 0x7fc7bc0a1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50842/0 (socket says 172.21.15.12:50842) 2024-09-11T01:51:56.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c16d9640 1 -- 172.21.15.12:0/545291892 learned_addr learned my addr 172.21.15.12:0/545291892 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:51:56.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c1eda640 1 --2- 172.21.15.12:0/545291892 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7bc141720 0x7fc7bc143b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:56.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c1eda640 10 monclient(hunting): get_auth_request con 0x7fc7bc141720 auth_method 0 2024-09-11T01:51:56.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c1eda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:56.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c1eda640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:56.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c1eda640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:56.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c0ed8640 1 --2- 172.21.15.12:0/545291892 >> v2:172.21.15.71:3300/0 conn(0x7fc7bc0a2120 0x7fc7bc0a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:56.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c0ed8640 10 monclient(hunting): get_auth_request con 0x7fc7bc0a2120 auth_method 0 2024-09-11T01:51:56.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c0ed8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:56.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c0ed8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:56.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c0ed8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:56.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c1eda640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:56.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c1eda640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:56.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c1eda640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:56.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c0ed8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:56.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c0ed8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:56.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c0ed8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:56.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c1eda640 10 monclient(hunting): handle_auth_done global_id 244199 payload 274 2024-09-11T01:51:56.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c1eda640 1 -- 172.21.15.12:0/545291892 >> v2:172.21.15.71:3300/0 conn(0x7fc7bc0a2120 msgr2=0x7fc7bc0a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:56.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c1eda640 1 --2- 172.21.15.12:0/545291892 >> v2:172.21.15.71:3300/0 conn(0x7fc7bc0a2120 0x7fc7bc0a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:56.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c1eda640 1 -- 172.21.15.12:0/545291892 >> v1:172.21.15.12:6789/0 conn(0x7fc7bc0a0c90 legacy=0x7fc7bc0a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:56.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c1eda640 10 monclient: _finish_hunting 0 2024-09-11T01:51:56.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c1eda640 1 monclient: found mon.noname-b 2024-09-11T01:51:56.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c1eda640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:56.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c1eda640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:51:56.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.143+0000 7fc7c1eda640 1 -- 172.21.15.12:0/545291892 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7bc0a1ea0 con 0x7fc7bc141720 2024-09-11T01:51:56.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.146+0000 7fc7c1eda640 1 -- 172.21.15.12:0/545291892 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7bc141720 msgr2=0x7fc7bc143b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:56.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.146+0000 7fc7c1eda640 1 --2- 172.21.15.12:0/545291892 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7bc141720 0x7fc7bc143b10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4009d90 tx=0x7fc7b402a490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3ee70f566bcbb29 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:56.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.146+0000 7fc7b3fff640 1 -- 172.21.15.12:0/545291892 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b4007950 con 0x7fc7bc141720 2024-09-11T01:51:56.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.146+0000 7fc7b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:56.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.146+0000 7fc7b3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:51:56.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.146+0000 7fc7b3fff640 10 monclient: dump: 2024-09-11T01:51:56.157 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:56.157 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:56.157 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:56.157 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:56.157 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:56.157 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:56.157 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:56.157 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:56.158 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:56.158 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:56.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.146+0000 7fc7b3fff640 10 monclient: _finish_auth 0 2024-09-11T01:51:56.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.146+0000 7fc7b3fff640 10 monclient: _check_auth_tickets 2024-09-11T01:51:56.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.146+0000 7fc7b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:56.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.146+0000 7fc7c393d640 5 monclient: authenticate success, global_id 244199 2024-09-11T01:51:56.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.146+0000 7fc7c393d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:51:56.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.146+0000 7fc7b3fff640 1 -- 172.21.15.12:0/545291892 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7b4007af0 con 0x7fc7bc141720 2024-09-11T01:51:56.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.146+0000 7fc7b3fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:56.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.146+0000 7fc7b3fff640 1 -- 172.21.15.12:0/545291892 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b4031400 con 0x7fc7bc141720 2024-09-11T01:51:56.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.146+0000 7fc7c393d640 10 monclient: get_monmap_and_config success 2024-09-11T01:51:56.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.146+0000 7fc7c393d640 4 set_mon_vals no callback set 2024-09-11T01:51:56.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.146+0000 7fc7b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:56.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.146+0000 7fc7b3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:56.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.146+0000 7fc7b3fff640 10 monclient: dump: 2024-09-11T01:51:56.159 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:56.159 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:56.159 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:56.159 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:56.159 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:56.159 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:56.159 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:56.159 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:56.159 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:56.159 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:56.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.147+0000 7fc7c393d640 10 monclient: shutdown 2024-09-11T01:51:56.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.147+0000 7fc7c393d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:56.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.147+0000 7fc7c393d640 1 -- 172.21.15.12:0/545291892 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7bc141720 msgr2=0x7fc7bc143b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:56.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.147+0000 7fc7c393d640 1 --2- 172.21.15.12:0/545291892 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7bc141720 0x7fc7bc143b10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4009d90 tx=0x7fc7b402a490 comp rx=0 tx=0).stop 2024-09-11T01:51:56.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.147+0000 7fc7c393d640 1 -- 172.21.15.12:0/545291892 shutdown_connections 2024-09-11T01:51:56.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.147+0000 7fc7c393d640 1 --2- 172.21.15.12:0/545291892 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7bc141720 0x7fc7bc143b10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:56.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.147+0000 7fc7c393d640 1 --2- 172.21.15.12:0/545291892 >> v2:172.21.15.71:3300/0 conn(0x7fc7bc0a2120 0x7fc7bc0a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:56.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.147+0000 7fc7c393d640 1 -- 172.21.15.12:0/545291892 >> 172.21.15.12:0/545291892 conn(0x7fc7bc0a2d90 msgr2=0x7fc7bc0a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:56.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.147+0000 7fc7c393d640 1 -- 172.21.15.12:0/545291892 shutdown_connections 2024-09-11T01:51:56.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.147+0000 7fc7c393d640 1 -- 172.21.15.12:0/545291892 wait complete. 2024-09-11T01:51:56.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.148+0000 7fc7c393d640 10 monclient: build_initial_monmap 2024-09-11T01:51:56.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.148+0000 7fc7c393d640 10 monclient: monmap: 2024-09-11T01:51:56.160 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:56.160 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:51:56.160 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:51:56.160 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:51:56.160 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:56.161 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:56.161 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:51:56.161 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:51:56.161 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:51:56.161 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:56.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.148+0000 7fc7c393d640 1 Processor -- start 2024-09-11T01:51:56.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.148+0000 7fc7c393d640 1 -- start start 2024-09-11T01:51:56.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.148+0000 7fc7c393d640 10 monclient: init 2024-09-11T01:51:56.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.148+0000 7fc7c393d640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:56.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.148+0000 7fc7c393d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:51:56.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.148+0000 7fc7c393d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7bc0a0c90 0x7fc7bc1259b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:56.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.148+0000 7fc7c393d640 10 monclient(hunting): picked mon.noname-b con 0x7fc7bc0a0c90 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:51:56.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.148+0000 7fc7c393d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7bc0a2120 0x7fc7bc0737b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:56.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.148+0000 7fc7c393d640 10 monclient(hunting): picked mon.noname-a con 0x7fc7bc0a2120 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:51:56.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.148+0000 7fc7c393d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7bc141720 0x7fc7bc1d1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:56.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.148+0000 7fc7c393d640 10 monclient(hunting): picked mon.noname-c con 0x7fc7bc141720 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:56.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.148+0000 7fc7c393d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:56.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.148+0000 7fc7c393d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc09ce20 con 0x7fc7bc0a2120 2024-09-11T01:51:56.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.148+0000 7fc7c393d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:56.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.148+0000 7fc7c393d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc09cca0 con 0x7fc7bc0a0c90 2024-09-11T01:51:56.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.148+0000 7fc7c393d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:56.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.148+0000 7fc7c393d640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc7bc144050 con 0x7fc7bc141720 2024-09-11T01:51:56.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.148+0000 7fc7c393d640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:56.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.148+0000 7fc7c393d640 10 monclient(hunting): authenticate will time out at 2122.590996s 2024-09-11T01:51:56.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c0ed8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7bc0a2120 0x7fc7bc0737b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:56.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c1eda640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7bc141720 0x7fc7bc1d1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:56.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c16d9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7bc0a0c90 0x7fc7bc1259b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:56.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c0ed8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7bc0a2120 0x7fc7bc0737b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52990/0 (socket says 172.21.15.12:52990) 2024-09-11T01:51:56.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c1eda640 10 monclient(hunting): get_auth_request con 0x7fc7bc141720 auth_method 0 2024-09-11T01:51:56.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c0ed8640 1 -- 172.21.15.12:0/357609192 learned_addr learned my addr 172.21.15.12:0/357609192 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:56.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c16d9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7bc0a0c90 0x7fc7bc1259b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:51544/0 (socket says 172.21.15.12:51544) 2024-09-11T01:51:56.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c1eda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:56.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c1eda640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:56.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c1eda640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:56.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c0ed8640 10 monclient(hunting): get_auth_request con 0x7fc7bc0a2120 auth_method 0 2024-09-11T01:51:56.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c0ed8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:56.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c0ed8640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:56.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c0ed8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:56.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c16d9640 10 monclient(hunting): get_auth_request con 0x7fc7bc0a0c90 auth_method 0 2024-09-11T01:51:56.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c16d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:56.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c16d9640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:56.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c16d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:56.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c1eda640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:56.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c1eda640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:56.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c0ed8640 1 -- 172.21.15.12:0/357609192 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7bc0a2120 msgr2=0x7fc7bc0737b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:51:56.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c0ed8640 1 -- 172.21.15.12:0/357609192 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7bc0a2120 msgr2=0x7fc7bc0737b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:51:56.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c0ed8640 1 --2- 172.21.15.12:0/357609192 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7bc0a2120 0x7fc7bc0737b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:56.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c1eda640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:56.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c16d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:56.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c16d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:56.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c0ed8640 1 --2- 172.21.15.12:0/357609192 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7bc0a2120 0x7fc7bc0737b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:51:56.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c16d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:56.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c1eda640 10 monclient(hunting): handle_auth_done global_id 244202 payload 824 2024-09-11T01:51:56.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c1eda640 1 -- 172.21.15.12:0/357609192 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7bc0a0c90 msgr2=0x7fc7bc1259b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:56.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c1eda640 1 --2- 172.21.15.12:0/357609192 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7bc0a0c90 0x7fc7bc1259b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c1eda640 1 -- 172.21.15.12:0/357609192 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7bc0a2120 msgr2=0x7fc7bc0737b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:51:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c1eda640 1 --2- 172.21.15.12:0/357609192 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7bc0a2120 0x7fc7bc0737b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c1eda640 10 monclient: _finish_hunting 0 2024-09-11T01:51:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c1eda640 1 monclient: found mon.noname-c 2024-09-11T01:51:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c1eda640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c1eda640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:51:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c1eda640 1 -- 172.21.15.12:0/357609192 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7bc143e20 con 0x7fc7bc141720 2024-09-11T01:51:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.149+0000 7fc7c16d9640 1 --2- 172.21.15.12:0/357609192 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7bc0a0c90 0x7fc7bc1259b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:51:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.151+0000 7fc7c1eda640 1 -- 172.21.15.12:0/357609192 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7bc141720 msgr2=0x7fc7bc1d1400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.151+0000 7fc7c1eda640 1 --2- 172.21.15.12:0/357609192 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7bc141720 0x7fc7bc1d1400 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4031000 tx=0x7fc7b4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.152+0000 7fc7b1ffb640 1 -- 172.21.15.12:0/357609192 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b4040070 con 0x7fc7bc141720 2024-09-11T01:51:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.152+0000 7fc7b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:56.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.152+0000 7fc7b1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:51:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.152+0000 7fc7b1ffb640 10 monclient: dump: 2024-09-11T01:51:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:56.166 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.152+0000 7fc7b1ffb640 10 monclient: _finish_auth 0 2024-09-11T01:51:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.152+0000 7fc7b1ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:51:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.152+0000 7fc7b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:56.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.152+0000 7fc7b1ffb640 1 -- 172.21.15.12:0/357609192 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7b4045460 con 0x7fc7bc141720 2024-09-11T01:51:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.152+0000 7fc7c393d640 5 monclient: authenticate success, global_id 244202 2024-09-11T01:51:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.152+0000 7fc7c393d640 10 monclient: _renew_subs 2024-09-11T01:51:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.152+0000 7fc7c393d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.152+0000 7fc7c393d640 1 -- 172.21.15.12:0/357609192 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7bc074520 con 0x7fc7bc141720 2024-09-11T01:51:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.152+0000 7fc7b1ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.152+0000 7fc7b1ffb640 1 -- 172.21.15.12:0/357609192 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b4007950 con 0x7fc7bc141720 2024-09-11T01:51:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.152+0000 7fc7b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.152+0000 7fc7b1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.152+0000 7fc7b1ffb640 10 monclient: dump: 2024-09-11T01:51:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:56.167 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:56.168 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:56.168 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:56.168 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:56.168 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:56.168 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:56.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.152+0000 7fc7c393d640 10 monclient: _renew_subs 2024-09-11T01:51:56.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.152+0000 7fc7c393d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:56.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.152+0000 7fc7b37fe640 4 set_mon_vals no callback set 2024-09-11T01:51:56.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.152+0000 7fc7c393d640 1 -- 172.21.15.12:0/357609192 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7bc125ef0 con 0x7fc7bc141720 2024-09-11T01:51:56.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.153+0000 7fc7c393d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:56.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.153+0000 7fc7c393d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:56.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.153+0000 7fc7c393d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:56.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.153+0000 7fc7b1ffb640 1 -- 172.21.15.12:0/357609192 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fc7b4045870 con 0x7fc7bc141720 2024-09-11T01:51:56.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.153+0000 7fc7c393d640 1 -- 172.21.15.12:0/357609192 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7bc0a1070 con 0x7fc7bc141720 2024-09-11T01:51:56.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.153+0000 7fc7b1ffb640 1 --2- 172.21.15.12:0/357609192 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc7a003e260 0x7fc7a0040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:56.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.153+0000 7fc7b1ffb640 1 -- 172.21.15.12:0/357609192 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(760..760 src has 676..760) ==== 4815+0+0 (secure 0 0 0) 0x7fc7b4074970 con 0x7fc7bc141720 2024-09-11T01:51:56.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.153+0000 7fc7c16d9640 1 --2- 172.21.15.12:0/357609192 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc7a003e260 0x7fc7a0040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:56.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.153+0000 7fc7c16d9640 10 monclient: get_auth_request con 0x7fc7a003e260 auth_method 0 2024-09-11T01:51:56.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.156+0000 7fc7c16d9640 1 --2- 172.21.15.12:0/357609192 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc7a003e260 0x7fc7a0040720 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8007c90 tx=0x7fc7b8007be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:56.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.158+0000 7fc7b1ffb640 1 -- 172.21.15.12:0/357609192 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc7b404a060 con 0x7fc7bc141720 2024-09-11T01:51:56.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.158+0000 7fc7b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:56.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.158+0000 7fc7b1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:51:56.170 INFO:tasks.rados.rados.0.smithi071.stdout:3370: done (1 left) 2024-09-11T01:51:56.170 INFO:tasks.rados.rados.0.smithi071.stdout:3371: done (0 left) 2024-09-11T01:51:56.170 INFO:tasks.rados.rados.0.smithi071.stdout:3372: read oid 20 snap -1 2024-09-11T01:51:56.171 INFO:tasks.rados.rados.0.smithi071.stdout:3372: expect (ObjNum 1122 snap 357 seq_num 1122) 2024-09-11T01:51:56.171 INFO:tasks.rados.rados.0.smithi071.stdout:3373: write (excl) oid 21 current snap is 378 2024-09-11T01:51:56.171 INFO:tasks.rados.rados.0.smithi071.stdout:3373: seq_num 1181 ranges {649067=503784,1875940=607071,3115378=731306} 2024-09-11T01:51:56.174 INFO:tasks.rados.rados.0.smithi071.stdout:3373: writing smithi07134777-21 from 649067 to 1152851 tid 1 2024-09-11T01:51:56.182 INFO:tasks.rados.rados.0.smithi071.stdout:3373: writing smithi07134777-21 from 1875940 to 2483011 tid 2 2024-09-11T01:51:56.190 INFO:tasks.rados.rados.0.smithi071.stdout:3373: writing smithi07134777-21 from 3115378 to 3846684 tid 3 2024-09-11T01:51:56.190 INFO:tasks.rados.rados.0.smithi071.stdout:3374: write oid 12 current snap is 378 2024-09-11T01:51:56.191 INFO:tasks.rados.rados.0.smithi071.stdout:3374: seq_num 1182 ranges {665874=579033,1934422=467082,3142801=542578} 2024-09-11T01:51:56.197 INFO:tasks.rados.rados.0.smithi071.stdout:3374: writing smithi07134777-12 from 665874 to 1244907 tid 1 2024-09-11T01:51:56.203 INFO:tasks.rados.rados.0.smithi071.stdout:3374: writing smithi07134777-12 from 1934422 to 2401504 tid 2 2024-09-11T01:51:56.210 INFO:tasks.rados.rados.0.smithi071.stdout:3374: writing smithi07134777-12 from 3142801 to 3685379 tid 3 2024-09-11T01:51:56.210 INFO:tasks.rados.rados.0.smithi071.stdout:3375: write (excl) oid 10 current snap is 378 2024-09-11T01:51:56.210 INFO:tasks.rados.rados.0.smithi071.stdout:3375: seq_num 1183 ranges {682681=654282,1876551=610740,3053871=470203} 2024-09-11T01:51:56.217 INFO:tasks.rados.rados.0.smithi071.stdout:3375: writing smithi07134777-10 from 682681 to 1336963 tid 1 2024-09-11T01:51:56.223 INFO:tasks.rados.rados.0.smithi071.stdout:3375: writing smithi07134777-10 from 1876551 to 2487291 tid 2 2024-09-11T01:51:56.229 INFO:tasks.rados.rados.0.smithi071.stdout:3375: writing smithi07134777-10 from 3053871 to 3524074 tid 3 2024-09-11T01:51:56.229 INFO:tasks.rados.rados.0.smithi071.stdout:3376: copy_from oid 27 from oid 1 current snap is 378 2024-09-11T01:51:56.229 INFO:tasks.rados.rados.0.smithi071.stdout:3372: expect (ObjNum 1122 snap 357 seq_num 1122) 2024-09-11T01:51:56.229 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 106214 2024-09-11T01:51:56.319 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2703167 2024-09-11T01:51:56.384 INFO:tasks.rados.rados.0.smithi071.stdout:3373: finishing write tid 1 to smithi07134777-21 2024-09-11T01:51:56.384 INFO:tasks.rados.rados.0.smithi071.stdout:3373: finishing write tid 2 to smithi07134777-21 2024-09-11T01:51:56.384 INFO:tasks.rados.rados.0.smithi071.stdout:3373: finishing write tid 3 to smithi07134777-21 2024-09-11T01:51:56.384 INFO:tasks.rados.rados.0.smithi071.stdout:3373: finishing write tid 4 to smithi07134777-21 2024-09-11T01:51:56.384 INFO:tasks.rados.rados.0.smithi071.stdout:3373: finishing write tid 5 to smithi07134777-21 2024-09-11T01:51:56.385 INFO:tasks.rados.rados.0.smithi071.stdout:3373: oid 21 updating version 0 to 2084 2024-09-11T01:51:56.385 INFO:tasks.rados.rados.0.smithi071.stdout:3373: oid 21 version 2084 is already newer than 2081 2024-09-11T01:51:56.385 INFO:tasks.rados.rados.0.smithi071.stdout:3373: oid 21 version 2084 is already newer than 2083 2024-09-11T01:51:56.385 INFO:tasks.rados.rados.0.smithi071.stdout:3373: oid 21 version 2084 is already newer than 2082 2024-09-11T01:51:56.385 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 2084 (ObjNum 1181 snap 378 seq_num 1181) dirty exists 2024-09-11T01:51:56.385 INFO:tasks.rados.rados.0.smithi071.stdout:3373: left oid 21 (ObjNum 1181 snap 378 seq_num 1181) 2024-09-11T01:51:56.385 INFO:tasks.rados.rados.0.smithi071.stdout:3374: finishing write tid 1 to smithi07134777-12 2024-09-11T01:51:56.385 INFO:tasks.rados.rados.0.smithi071.stdout:3374: finishing write tid 2 to smithi07134777-12 2024-09-11T01:51:56.385 INFO:tasks.rados.rados.0.smithi071.stdout:3374: finishing write tid 3 to smithi07134777-12 2024-09-11T01:51:56.385 INFO:tasks.rados.rados.0.smithi071.stdout:3372: done (4 left) 2024-09-11T01:51:56.385 INFO:tasks.rados.rados.0.smithi071.stdout:3373: done (3 left) 2024-09-11T01:51:56.385 INFO:tasks.rados.rados.0.smithi071.stdout:3377: read oid 45 snap -1 2024-09-11T01:51:56.385 INFO:tasks.rados.rados.0.smithi071.stdout:3377: expect deleted 2024-09-11T01:51:56.385 INFO:tasks.rados.rados.0.smithi071.stdout:3378: read oid 2 snap -1 2024-09-11T01:51:56.386 INFO:tasks.rados.rados.0.smithi071.stdout:3378: expect (ObjNum 1131 snap 357 seq_num 1131) 2024-09-11T01:51:56.386 INFO:tasks.rados.rados.0.smithi071.stdout:3379: read oid 19 snap 374 2024-09-11T01:51:56.386 INFO:tasks.rados.rados.0.smithi071.stdout:3379: expect (ObjNum 1136 snap 360 seq_num 1136) 2024-09-11T01:51:56.386 INFO:tasks.rados.rados.0.smithi071.stdout:3380: read oid 36 snap -1 2024-09-11T01:51:56.386 INFO:tasks.rados.rados.0.smithi071.stdout:3380: expect (ObjNum 1179 snap 376 seq_num 1179) 2024-09-11T01:51:56.386 INFO:tasks.rados.rados.0.smithi071.stdout:3381: delete oid 31 current snap is 378 2024-09-11T01:51:56.386 INFO:tasks.rados.rados.0.smithi071.stdout:3374: finishing write tid 4 to smithi07134777-12 2024-09-11T01:51:56.386 INFO:tasks.rados.rados.0.smithi071.stdout:3374: finishing write tid 5 to smithi07134777-12 2024-09-11T01:51:56.386 INFO:tasks.rados.rados.0.smithi071.stdout:3374: oid 12 updating version 0 to 2343 2024-09-11T01:51:56.386 INFO:tasks.rados.rados.0.smithi071.stdout:3374: oid 12 updating version 2343 to 2344 2024-09-11T01:51:56.386 INFO:tasks.rados.rados.0.smithi071.stdout:3374: oid 12 updating version 2344 to 2345 2024-09-11T01:51:56.386 INFO:tasks.rados.rados.0.smithi071.stdout:3374: oid 12 version 2345 is already newer than 2342 2024-09-11T01:51:56.386 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 2345 (ObjNum 1182 snap 378 seq_num 1182) dirty exists 2024-09-11T01:51:56.386 INFO:tasks.rados.rados.0.smithi071.stdout:3374: left oid 12 (ObjNum 1182 snap 378 seq_num 1182) 2024-09-11T01:51:56.387 INFO:tasks.rados.rados.0.smithi071.stdout:3375: finishing write tid 1 to smithi07134777-10 2024-09-11T01:51:56.387 INFO:tasks.rados.rados.0.smithi071.stdout:3375: finishing write tid 2 to smithi07134777-10 2024-09-11T01:51:56.387 INFO:tasks.rados.rados.0.smithi071.stdout:3375: finishing write tid 3 to smithi07134777-10 2024-09-11T01:51:56.387 INFO:tasks.rados.rados.0.smithi071.stdout:3375: finishing write tid 4 to smithi07134777-10 2024-09-11T01:51:56.387 INFO:tasks.rados.rados.0.smithi071.stdout:3375: finishing write tid 5 to smithi07134777-10 2024-09-11T01:51:56.387 INFO:tasks.rados.rados.0.smithi071.stdout:3375: oid 10 updating version 0 to 1963 2024-09-11T01:51:56.387 INFO:tasks.rados.rados.0.smithi071.stdout:3375: oid 10 version 1963 is already newer than 1962 2024-09-11T01:51:56.387 INFO:tasks.rados.rados.0.smithi071.stdout:3375: oid 10 version 1963 is already newer than 1961 2024-09-11T01:51:56.387 INFO:tasks.rados.rados.0.smithi071.stdout:3375: oid 10 updating version 1963 to 1964 2024-09-11T01:51:56.387 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 1964 (ObjNum 1183 snap 378 seq_num 1183) dirty exists 2024-09-11T01:51:56.387 INFO:tasks.rados.rados.0.smithi071.stdout:3375: left oid 10 (ObjNum 1183 snap 378 seq_num 1183) 2024-09-11T01:51:56.387 INFO:tasks.rados.rados.0.smithi071.stdout:3376: finishing copy_from racing read to smithi07134777-27 2024-09-11T01:51:56.387 INFO:tasks.rados.rados.0.smithi071.stdout:3376: finishing copy_from to smithi07134777-27 2024-09-11T01:51:56.387 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 1893 (ObjNum 1154 snap 366 seq_num 1154) dirty exists 2024-09-11T01:51:56.388 INFO:tasks.rados.rados.0.smithi071.stdout:3374: done (7 left) 2024-09-11T01:51:56.388 INFO:tasks.rados.rados.0.smithi071.stdout:3375: done (6 left) 2024-09-11T01:51:56.388 INFO:tasks.rados.rados.0.smithi071.stdout:3376: done (5 left) 2024-09-11T01:51:56.388 INFO:tasks.rados.rados.0.smithi071.stdout:3377: done (4 left) 2024-09-11T01:51:56.388 INFO:tasks.rados.rados.0.smithi071.stdout:3381: done (3 left) 2024-09-11T01:51:56.388 INFO:tasks.rados.rados.0.smithi071.stdout:3382: delete oid 16 current snap is 378 2024-09-11T01:51:56.388 INFO:tasks.rados.rados.0.smithi071.stdout:3378: expect (ObjNum 1131 snap 357 seq_num 1131) 2024-09-11T01:51:56.388 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 246999 2024-09-11T01:51:56.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.430+0000 7fc7c393d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:51:56.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.430+0000 7fc7c393d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:51:56.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.430+0000 7fc7c393d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:56.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.430+0000 7fc7c393d640 1 -- 172.21.15.12:0/357609192 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc7bc06d460 con 0x7fc7bc141720 2024-09-11T01:51:56.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.430+0000 7fc7b1ffb640 1 -- 172.21.15.12:0/357609192 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7fc7b40303a0 con 0x7fc7bc141720 2024-09-11T01:51:56.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.430+0000 7fc7b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:51:56.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.430+0000 7fc7b1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:51:56.432 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":210,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:51:56.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.433+0000 7fc7c393d640 1 -- 172.21.15.12:0/357609192 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc7a003e260 msgr2=0x7fc7a0040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:56.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.433+0000 7fc7c393d640 1 --2- 172.21.15.12:0/357609192 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc7a003e260 0x7fc7a0040720 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8007c90 tx=0x7fc7b8007be0 comp rx=0 tx=0).stop 2024-09-11T01:51:56.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.433+0000 7fc7c393d640 10 monclient: shutdown 2024-09-11T01:51:56.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.433+0000 7fc7c393d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:56.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.433+0000 7fc7c393d640 1 -- 172.21.15.12:0/357609192 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7bc141720 msgr2=0x7fc7bc1d1400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:56.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.433+0000 7fc7c393d640 1 --2- 172.21.15.12:0/357609192 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7bc141720 0x7fc7bc1d1400 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4031000 tx=0x7fc7b4004a40 comp rx=0 tx=0).stop 2024-09-11T01:51:56.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.433+0000 7fc7c393d640 1 -- 172.21.15.12:0/357609192 shutdown_connections 2024-09-11T01:51:56.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.433+0000 7fc7c393d640 1 --2- 172.21.15.12:0/357609192 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7bc141720 0x7fc7bc1d1400 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:56.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.433+0000 7fc7c393d640 1 --2- 172.21.15.12:0/357609192 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc7a003e260 0x7fc7a0040720 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:56.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.433+0000 7fc7c393d640 1 --2- 172.21.15.12:0/357609192 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc7bc0a2120 0x7fc7bc0737b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:56.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.433+0000 7fc7c393d640 1 --2- 172.21.15.12:0/357609192 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7bc0a0c90 0x7fc7bc1259b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:56.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.433+0000 7fc7c393d640 1 -- 172.21.15.12:0/357609192 >> 172.21.15.12:0/357609192 conn(0x7fc7bc0a2d90 msgr2=0x7fc7bc142bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:56.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.433+0000 7fc7c393d640 1 -- 172.21.15.12:0/357609192 shutdown_connections 2024-09-11T01:51:56.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:56.433+0000 7fc7c393d640 1 -- 172.21.15.12:0/357609192 wait complete. 2024-09-11T01:51:56.443 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:51:56.474 INFO:tasks.rados.rados.0.smithi071.stdout:3379: expect (ObjNum 1136 snap 360 seq_num 1136) 2024-09-11T01:51:56.519 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:51:56.554 INFO:tasks.rados.rados.0.smithi071.stdout:3380: expect (ObjNum 1179 snap 376 seq_num 1179) 2024-09-11T01:51:56.579 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1544359 2024-09-11T01:51:56.770 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3542348 2024-09-11T01:51:56.773 INFO:tasks.rados.rados.0.smithi071.stdout:3378: done (3 left) 2024-09-11T01:51:56.774 INFO:tasks.rados.rados.0.smithi071.stdout:3379: done (2 left) 2024-09-11T01:51:56.774 INFO:tasks.rados.rados.0.smithi071.stdout:3380: done (1 left) 2024-09-11T01:51:56.774 INFO:tasks.rados.rados.0.smithi071.stdout:3382: done (0 left) 2024-09-11T01:51:56.774 INFO:tasks.rados.rados.0.smithi071.stdout:3383: rollback oid 7 current snap is 378 2024-09-11T01:51:56.774 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 7 to 309 2024-09-11T01:51:56.774 INFO:tasks.rados.rados.0.smithi071.stdout:3384: read oid 25 snap -1 2024-09-11T01:51:56.774 INFO:tasks.rados.rados.0.smithi071.stdout:3384: expect (ObjNum 1023 snap 328 seq_num 1023) 2024-09-11T01:51:56.774 INFO:tasks.rados.rados.0.smithi071.stdout:3385: rollback oid 35 current snap is 378 2024-09-11T01:51:56.774 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 35 to 366 2024-09-11T01:51:56.774 INFO:tasks.rados.rados.0.smithi071.stdout:3386: read oid 6 snap -1 2024-09-11T01:51:56.774 INFO:tasks.rados.rados.0.smithi071.stdout:3386: expect (ObjNum 1158 snap 369 seq_num 1158) 2024-09-11T01:51:56.774 INFO:tasks.rados.rados.0.smithi071.stdout:3387: copy_from oid 2 from oid 31 current snap is 378 2024-09-11T01:51:56.774 INFO:tasks.rados.rados.0.smithi071.stdout:3388: read oid 28 snap -1 2024-09-11T01:51:56.774 INFO:tasks.rados.rados.0.smithi071.stdout:3388: expect (ObjNum 1170 snap 376 seq_num 1170) 2024-09-11T01:51:56.775 INFO:tasks.rados.rados.0.smithi071.stdout:3389: read oid 29 snap -1 2024-09-11T01:51:56.775 INFO:tasks.rados.rados.0.smithi071.stdout:3389: expect (ObjNum 1177 snap 376 seq_num 1177) 2024-09-11T01:51:56.775 INFO:tasks.rados.rados.0.smithi071.stdout:3390: rollback oid 37 current snap is 378 2024-09-11T01:51:56.775 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 37 to 376 2024-09-11T01:51:56.775 INFO:tasks.rados.rados.0.smithi071.stdout:3391: rollback oid 33 current snap is 378 2024-09-11T01:51:56.775 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 33 to 364 2024-09-11T01:51:56.775 INFO:tasks.rados.rados.0.smithi071.stdout:3392: copy_from oid 17 from oid 48 current snap is 378 2024-09-11T01:51:56.775 INFO:tasks.rados.rados.0.smithi071.stdout:3393: write oid 21 current snap is 378 2024-09-11T01:51:56.775 INFO:tasks.rados.rados.0.smithi071.stdout:3393: seq_num 1186 ranges {733102=596382,1768350=758067,3136140=704019} 2024-09-11T01:51:56.781 INFO:tasks.rados.rados.0.smithi071.stdout:3393: writing smithi07134777-21 from 733102 to 1329484 tid 1 2024-09-11T01:51:56.788 INFO:tasks.rados.rados.0.smithi071.stdout:3393: writing smithi07134777-21 from 1768350 to 2526417 tid 2 2024-09-11T01:51:56.796 INFO:tasks.rados.rados.0.smithi071.stdout:3393: writing smithi07134777-21 from 3136140 to 3840159 tid 3 2024-09-11T01:51:56.796 INFO:tasks.rados.rados.0.smithi071.stdout:3394: write oid 16 current snap is 378 2024-09-11T01:51:56.796 INFO:tasks.rados.rados.0.smithi071.stdout:3387: finishing copy_from to smithi07134777-2 2024-09-11T01:51:56.796 INFO:tasks.rados.rados.0.smithi071.stdout:3387: got expected ENOENT (src dne) 2024-09-11T01:51:56.796 INFO:tasks.rados.rados.0.smithi071.stdout:3394: seq_num 1187 ranges {749909=671631,2110479=618078,3163563=515291} 2024-09-11T01:51:56.804 INFO:tasks.rados.rados.0.smithi071.stdout:3394: writing smithi07134777-16 from 749909 to 1421540 tid 1 2024-09-11T01:51:56.811 INFO:tasks.rados.rados.0.smithi071.stdout:3394: writing smithi07134777-16 from 2110479 to 2728557 tid 2 2024-09-11T01:51:56.817 INFO:tasks.rados.rados.0.smithi071.stdout:3394: writing smithi07134777-16 from 3163563 to 3678854 tid 3 2024-09-11T01:51:56.817 INFO:tasks.rados.rados.0.smithi071.stdout:3395: snap_create 2024-09-11T01:51:56.817 INFO:tasks.rados.rados.0.smithi071.stdout:3387: finishing copy_from racing read to smithi07134777-2 2024-09-11T01:51:56.817 INFO:tasks.rados.rados.0.smithi071.stdout:3385: finishing rollback tid 1 to smithi07134777-35 2024-09-11T01:51:56.817 INFO:tasks.rados.rados.0.smithi071.stdout:3385: finishing rollback tid 2 to smithi07134777-35 2024-09-11T01:51:56.817 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 1981 (ObjNum 1122 snap 357 seq_num 1122) dirty exists 2024-09-11T01:51:56.817 INFO:tasks.rados.rados.0.smithi071.stdout:3390: finishing rollback tid 1 to smithi07134777-37 2024-09-11T01:51:56.817 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 1968 (ObjNum 241 snap 0 seq_num 386) dirty dne 2024-09-11T01:51:56.817 INFO:tasks.rados.rados.0.smithi071.stdout:3383: finishing rollback tid 0 to smithi07134777-7 2024-09-11T01:51:56.817 INFO:tasks.rados.rados.0.smithi071.stdout:3383: finishing rollback tid 1 to smithi07134777-7 2024-09-11T01:51:56.817 INFO:tasks.rados.rados.0.smithi071.stdout:3383: finishing rollback tid 2 to smithi07134777-7 2024-09-11T01:51:56.817 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 2553 (ObjNum 924 snap 290 seq_num 924) dirty exists 2024-09-11T01:51:56.818 INFO:tasks.rados.rados.0.smithi071.stdout:3384: expect (ObjNum 1023 snap 328 seq_num 1023) 2024-09-11T01:51:56.898 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:51:56.945 INFO:tasks.rados.rados.0.smithi071.stdout:3389: expect (ObjNum 1177 snap 376 seq_num 1177) 2024-09-11T01:51:56.964 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:51:57.011 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1049435 2024-09-11T01:51:57.040 INFO:tasks.rados.rados.0.smithi071.stdout:3386: expect (ObjNum 1158 snap 369 seq_num 1158) 2024-09-11T01:51:57.041 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:51:57.254 INFO:tasks.rados.rados.0.smithi071.stdout:3383: done (12 left) 2024-09-11T01:51:57.254 INFO:tasks.rados.rados.0.smithi071.stdout:3384: done (11 left) 2024-09-11T01:51:57.254 INFO:tasks.rados.rados.0.smithi071.stdout:3385: done (10 left) 2024-09-11T01:51:57.255 INFO:tasks.rados.rados.0.smithi071.stdout:3386: done (9 left) 2024-09-11T01:51:57.255 INFO:tasks.rados.rados.0.smithi071.stdout:3387: done (8 left) 2024-09-11T01:51:57.255 INFO:tasks.rados.rados.0.smithi071.stdout:3389: done (7 left) 2024-09-11T01:51:57.255 INFO:tasks.rados.rados.0.smithi071.stdout:3390: done (6 left) 2024-09-11T01:51:57.255 INFO:tasks.rados.rados.0.smithi071.stdout:3395: done (5 left) 2024-09-11T01:51:57.255 INFO:tasks.rados.rados.0.smithi071.stdout:3396: read oid 14 snap -1 2024-09-11T01:51:57.255 INFO:tasks.rados.rados.0.smithi071.stdout:3396: expect (ObjNum 1136 snap 360 seq_num 1136) 2024-09-11T01:51:57.255 INFO:tasks.rados.rados.0.smithi071.stdout:3397: rollback oid 2 current snap is 379 2024-09-11T01:51:57.255 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 2 to 339 2024-09-11T01:51:57.255 INFO:tasks.rados.rados.0.smithi071.stdout:3392: finishing copy_from racing read to smithi07134777-17 2024-09-11T01:51:57.255 INFO:tasks.rados.rados.0.smithi071.stdout:3392: finishing copy_from to smithi07134777-17 2024-09-11T01:51:57.255 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 2541 (ObjNum 1178 snap 376 seq_num 1178) dirty exists 2024-09-11T01:51:57.255 INFO:tasks.rados.rados.0.smithi071.stdout:3388: expect (ObjNum 1170 snap 376 seq_num 1170) 2024-09-11T01:51:57.256 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:51:57.418 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 650313 2024-09-11T01:51:57.488 INFO:tasks.rados.rados.0.smithi071.stdout:3391: finishing rollback tid 0 to smithi07134777-33 2024-09-11T01:51:57.488 INFO:tasks.rados.rados.0.smithi071.stdout:3391: finishing rollback tid 1 to smithi07134777-33 2024-09-11T01:51:57.488 INFO:tasks.rados.rados.0.smithi071.stdout:3391: finishing rollback tid 2 to smithi07134777-33 2024-09-11T01:51:57.488 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 2351 (ObjNum 1130 snap 357 seq_num 1130) dirty exists 2024-09-11T01:51:57.488 INFO:tasks.rados.rados.0.smithi071.stdout:3393: finishing write tid 1 to smithi07134777-21 2024-09-11T01:51:57.488 INFO:tasks.rados.rados.0.smithi071.stdout:3393: finishing write tid 2 to smithi07134777-21 2024-09-11T01:51:57.488 INFO:tasks.rados.rados.0.smithi071.stdout:3388: done (6 left) 2024-09-11T01:51:57.488 INFO:tasks.rados.rados.0.smithi071.stdout:3391: done (5 left) 2024-09-11T01:51:57.489 INFO:tasks.rados.rados.0.smithi071.stdout:3392: done (4 left) 2024-09-11T01:51:57.489 INFO:tasks.rados.rados.0.smithi071.stdout:3398: write oid 25 current snap is 379 2024-09-11T01:51:57.489 INFO:tasks.rados.rados.0.smithi071.stdout:3393: finishing write tid 3 to smithi07134777-21 2024-09-11T01:51:57.489 INFO:tasks.rados.rados.0.smithi071.stdout:3398: seq_num 1188 ranges {766716=746880,2168961=761736,3474633=442916} 2024-09-11T01:51:57.495 INFO:tasks.rados.rados.0.smithi071.stdout:3398: writing smithi07134777-25 from 766716 to 1513596 tid 1 2024-09-11T01:51:57.503 INFO:tasks.rados.rados.0.smithi071.stdout:3398: writing smithi07134777-25 from 2168961 to 2930697 tid 2 2024-09-11T01:51:57.508 INFO:tasks.rados.rados.0.smithi071.stdout:3398: writing smithi07134777-25 from 3474633 to 3917549 tid 3 2024-09-11T01:51:57.508 INFO:tasks.rados.rados.0.smithi071.stdout:3399: snap_remove snap 376 2024-09-11T01:51:58.135 INFO:tasks.rados.rados.0.smithi071.stdout:3399: done (5 left) 2024-09-11T01:51:58.135 INFO:tasks.rados.rados.0.smithi071.stdout:3400: rollback oid 50 current snap is 379 2024-09-11T01:51:58.135 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 50 to 344 2024-09-11T01:51:58.135 INFO:tasks.rados.rados.0.smithi071.stdout:3393: finishing write tid 4 to smithi07134777-21 2024-09-11T01:51:58.135 INFO:tasks.rados.rados.0.smithi071.stdout:3393: finishing write tid 5 to smithi07134777-21 2024-09-11T01:51:58.135 INFO:tasks.rados.rados.0.smithi071.stdout:3393: oid 21 updating version 0 to 2088 2024-09-11T01:51:58.135 INFO:tasks.rados.rados.0.smithi071.stdout:3393: oid 21 version 2088 is already newer than 2087 2024-09-11T01:51:58.136 INFO:tasks.rados.rados.0.smithi071.stdout:3393: oid 21 version 2088 is already newer than 2086 2024-09-11T01:51:58.136 INFO:tasks.rados.rados.0.smithi071.stdout:3393: oid 21 updating version 2088 to 2089 2024-09-11T01:51:58.136 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 2089 (ObjNum 1186 snap 378 seq_num 1186) dirty exists 2024-09-11T01:51:58.136 INFO:tasks.rados.rados.0.smithi071.stdout:3393: left oid 21 (ObjNum 1186 snap 378 seq_num 1186) 2024-09-11T01:51:58.136 INFO:tasks.rados.rados.0.smithi071.stdout:3394: finishing write tid 1 to smithi07134777-16 2024-09-11T01:51:58.136 INFO:tasks.rados.rados.0.smithi071.stdout:3394: finishing write tid 2 to smithi07134777-16 2024-09-11T01:51:58.136 INFO:tasks.rados.rados.0.smithi071.stdout:3394: finishing write tid 3 to smithi07134777-16 2024-09-11T01:51:58.136 INFO:tasks.rados.rados.0.smithi071.stdout:3394: finishing write tid 4 to smithi07134777-16 2024-09-11T01:51:58.136 INFO:tasks.rados.rados.0.smithi071.stdout:3394: finishing write tid 5 to smithi07134777-16 2024-09-11T01:51:58.136 INFO:tasks.rados.rados.0.smithi071.stdout:3394: oid 16 updating version 0 to 2091 2024-09-11T01:51:58.136 INFO:tasks.rados.rados.0.smithi071.stdout:3394: oid 16 updating version 2091 to 2093 2024-09-11T01:51:58.136 INFO:tasks.rados.rados.0.smithi071.stdout:3394: oid 16 version 2093 is already newer than 2090 2024-09-11T01:51:58.136 INFO:tasks.rados.rados.0.smithi071.stdout:3394: oid 16 version 2093 is already newer than 2092 2024-09-11T01:51:58.136 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 2093 (ObjNum 1187 snap 378 seq_num 1187) dirty exists 2024-09-11T01:51:58.137 INFO:tasks.rados.rados.0.smithi071.stdout:3394: left oid 16 (ObjNum 1187 snap 378 seq_num 1187) 2024-09-11T01:51:58.137 INFO:tasks.rados.rados.0.smithi071.stdout:3397: finishing rollback tid 0 to smithi07134777-2 2024-09-11T01:51:58.137 INFO:tasks.rados.rados.0.smithi071.stdout:3397: finishing rollback tid 1 to smithi07134777-2 2024-09-11T01:51:58.137 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 1787 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-11T01:51:58.137 INFO:tasks.rados.rados.0.smithi071.stdout:3396: expect (ObjNum 1136 snap 360 seq_num 1136) 2024-09-11T01:51:58.173 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:51:58.232 INFO:tasks.rados.rados.0.smithi071.stdout:3398: finishing write tid 1 to smithi07134777-25 2024-09-11T01:51:58.232 INFO:tasks.rados.rados.0.smithi071.stdout:3398: finishing write tid 2 to smithi07134777-25 2024-09-11T01:51:58.232 INFO:tasks.rados.rados.0.smithi071.stdout:3398: finishing write tid 3 to smithi07134777-25 2024-09-11T01:51:58.232 INFO:tasks.rados.rados.0.smithi071.stdout:3398: finishing write tid 4 to smithi07134777-25 2024-09-11T01:51:58.232 INFO:tasks.rados.rados.0.smithi071.stdout:3398: finishing write tid 5 to smithi07134777-25 2024-09-11T01:51:58.232 INFO:tasks.rados.rados.0.smithi071.stdout:3398: oid 25 updating version 0 to 1050 2024-09-11T01:51:58.232 INFO:tasks.rados.rados.0.smithi071.stdout:3398: oid 25 version 1050 is already newer than 1049 2024-09-11T01:51:58.232 INFO:tasks.rados.rados.0.smithi071.stdout:3398: oid 25 version 1050 is already newer than 1048 2024-09-11T01:51:58.232 INFO:tasks.rados.rados.0.smithi071.stdout:3398: oid 25 version 1050 is already newer than 1047 2024-09-11T01:51:58.232 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 1050 (ObjNum 1188 snap 379 seq_num 1188) dirty exists 2024-09-11T01:51:58.232 INFO:tasks.rados.rados.0.smithi071.stdout:3398: left oid 25 (ObjNum 1188 snap 379 seq_num 1188) 2024-09-11T01:51:58.233 INFO:tasks.rados.rados.0.smithi071.stdout:3400: finishing rollback tid 0 to smithi07134777-50 2024-09-11T01:51:58.233 INFO:tasks.rados.rados.0.smithi071.stdout:3400: finishing rollback tid 1 to smithi07134777-50 2024-09-11T01:51:58.233 INFO:tasks.rados.rados.0.smithi071.stdout:3393: done (5 left) 2024-09-11T01:51:58.233 INFO:tasks.rados.rados.0.smithi071.stdout:3394: done (4 left) 2024-09-11T01:51:58.233 INFO:tasks.rados.rados.0.smithi071.stdout:3396: done (3 left) 2024-09-11T01:51:58.233 INFO:tasks.rados.rados.0.smithi071.stdout:3397: done (2 left) 2024-09-11T01:51:58.233 INFO:tasks.rados.rados.0.smithi071.stdout:3398: done (1 left) 2024-09-11T01:51:58.233 INFO:tasks.rados.rados.0.smithi071.stdout:3401: read oid 25 snap -1 2024-09-11T01:51:58.233 INFO:tasks.rados.rados.0.smithi071.stdout:3401: expect (ObjNum 1188 snap 379 seq_num 1188) 2024-09-11T01:51:58.233 INFO:tasks.rados.rados.0.smithi071.stdout:3402: delete oid 17 current snap is 379 2024-09-11T01:51:58.233 INFO:tasks.rados.rados.0.smithi071.stdout:3400: finishing rollback tid 2 to smithi07134777-50 2024-09-11T01:51:58.233 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 2557 (ObjNum 1057 snap 340 seq_num 1057) dirty exists 2024-09-11T01:51:58.234 INFO:tasks.rados.rados.0.smithi071.stdout:3400: done (2 left) 2024-09-11T01:51:58.235 INFO:tasks.rados.rados.0.smithi071.stdout:3402: done (1 left) 2024-09-11T01:51:58.235 INFO:tasks.rados.rados.0.smithi071.stdout:3403: snap_remove snap 322 2024-09-11T01:51:59.445 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:51:59.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.556+0000 7f2752164640 10 monclient: get_monmap_and_config 2024-09-11T01:51:59.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.556+0000 7f2752164640 10 monclient: build_initial_monmap 2024-09-11T01:51:59.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.556+0000 7f2752164640 10 monclient: monmap: 2024-09-11T01:51:59.558 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:59.558 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:59.559 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:51:59.557674+0000 2024-09-11T01:51:59.559 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:51:59.557674+0000 2024-09-11T01:51:59.559 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:59.559 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:59.559 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:51:59.559 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:51:59.559 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:51:59.559 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:51:59.559 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:51:59.559 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:59.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.557+0000 7f2752164640 1 Processor -- start 2024-09-11T01:51:59.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.557+0000 7f2752164640 1 -- start start 2024-09-11T01:51:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.557+0000 7f2752164640 10 monclient: init 2024-09-11T01:51:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f2752164640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f2752164640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-11T01:51:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f2752164640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f274c073d90 0x7f274c074170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f2752164640 10 monclient(hunting): picked mon.noname-b con 0x7f274c073d90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f2752164640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f274c0a0660 addr v1:172.21.15.12:6789/0 2024-09-11T01:51:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f2752164640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f274c1380d0 0x7f274c0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f2752164640 10 monclient(hunting): picked mon.noname-a con 0x7f274c1380d0 addr v2:172.21.15.12:3300/0 2024-09-11T01:51:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f2752164640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f274c070560 con 0x7f274c0a0660 2024-09-11T01:51:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f2752164640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f2752164640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f274c070310 con 0x7f274c1380d0 2024-09-11T01:51:59.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f2752164640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:59.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f2752164640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f274c070840 con 0x7f274c073d90 2024-09-11T01:51:59.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f2752164640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:59.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f2752164640 10 monclient(hunting): authenticate will time out at 2126.000340s 2024-09-11T01:51:59.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f274c073d90 0x7f274c074170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:59.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f274c1380d0 0x7f274c0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:59.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274affd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f274c0a0660 0x7f274c0a2560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37076/0 (socket says 172.21.15.12:37076) 2024-09-11T01:51:59.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274affd640 1 -- 172.21.15.12:0/1746586049 learned_addr learned my addr 172.21.15.12:0/1746586049 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:51:59.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274b7fe640 10 monclient(hunting): get_auth_request con 0x7f274c073d90 auth_method 0 2024-09-11T01:51:59.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:59.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:59.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:59.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274bfff640 10 monclient(hunting): get_auth_request con 0x7f274c1380d0 auth_method 0 2024-09-11T01:51:59.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274b7fe640 1 -- 172.21.15.12:0/1746586049 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f274c073d90 msgr2=0x7f274c074170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:51:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274b7fe640 1 -- 172.21.15.12:0/1746586049 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f274c073d90 msgr2=0x7f274c074170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274b7fe640 1 --2- 172.21.15.12:0/1746586049 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f274c073d90 0x7f274c074170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274b7fe640 1 --2- 172.21.15.12:0/1746586049 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f274c073d90 0x7f274c074170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:51:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274bfff640 10 monclient(hunting): handle_auth_done global_id 254097 payload 274 2024-09-11T01:51:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274bfff640 1 -- 172.21.15.12:0/1746586049 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f274c073d90 msgr2=0x7f274c074170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:51:59.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274bfff640 1 --2- 172.21.15.12:0/1746586049 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f274c073d90 0x7f274c074170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:59.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274bfff640 1 -- 172.21.15.12:0/1746586049 >> v1:172.21.15.12:6789/0 conn(0x7f274c0a0660 legacy=0x7f274c0a2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:59.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:51:59.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274bfff640 1 monclient: found mon.noname-a 2024-09-11T01:51:59.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:59.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:51:59.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.558+0000 7f274bfff640 1 -- 172.21.15.12:0/1746586049 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f274c09fff0 con 0x7f274c1380d0 2024-09-11T01:51:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.561+0000 7f274bfff640 1 -- 172.21.15.12:0/1746586049 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f274c1380d0 msgr2=0x7f274c0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.561+0000 7f274bfff640 1 --2- 172.21.15.12:0/1746586049 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f274c1380d0 0x7f274c0a2c70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2744009d90 tx=0x7f274402a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d083e0dbc24907c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f274a7fc640 1 -- 172.21.15.12:0/1746586049 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2744007950 con 0x7f274c1380d0 2024-09-11T01:51:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f274a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f274a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:51:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f274a7fc640 10 monclient: dump: 2024-09-11T01:51:59.569 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:59.570 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:59.570 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:59.570 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:59.570 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:59.570 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:59.570 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:59.570 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:59.570 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:59.570 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:59.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f274a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:51:59.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f274a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:51:59.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f274a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:59.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f274a7fc640 1 -- 172.21.15.12:0/1746586049 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2744007af0 con 0x7f274c1380d0 2024-09-11T01:51:59.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f2752164640 5 monclient: authenticate success, global_id 254097 2024-09-11T01:51:59.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f2752164640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:51:59.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f274a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:59.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f2752164640 10 monclient: get_monmap_and_config success 2024-09-11T01:51:59.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f274a7fc640 1 -- 172.21.15.12:0/1746586049 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2744031400 con 0x7f274c1380d0 2024-09-11T01:51:59.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f2752164640 4 set_mon_vals no callback set 2024-09-11T01:51:59.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f274a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:59.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f274a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:51:59.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f274a7fc640 10 monclient: dump: 2024-09-11T01:51:59.571 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:59.571 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:59.571 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:59.572 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:59.572 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:59.572 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:59.572 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:59.572 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:59.572 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:59.572 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:59.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f2752164640 10 monclient: shutdown 2024-09-11T01:51:59.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f2752164640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:59.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f2752164640 1 -- 172.21.15.12:0/1746586049 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f274c1380d0 msgr2=0x7f274c0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:59.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f2752164640 1 --2- 172.21.15.12:0/1746586049 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f274c1380d0 0x7f274c0a2c70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2744009d90 tx=0x7f274402a490 comp rx=0 tx=0).stop 2024-09-11T01:51:59.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f2752164640 1 -- 172.21.15.12:0/1746586049 shutdown_connections 2024-09-11T01:51:59.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f2752164640 1 --2- 172.21.15.12:0/1746586049 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f274c1380d0 0x7f274c0a2c70 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:59.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f2752164640 1 --2- 172.21.15.12:0/1746586049 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f274c073d90 0x7f274c074170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:59.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f2752164640 1 -- 172.21.15.12:0/1746586049 >> 172.21.15.12:0/1746586049 conn(0x7f274c0a3650 msgr2=0x7f274c0a3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:59.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f2752164640 1 -- 172.21.15.12:0/1746586049 shutdown_connections 2024-09-11T01:51:59.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.562+0000 7f2752164640 1 -- 172.21.15.12:0/1746586049 wait complete. 2024-09-11T01:51:59.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.563+0000 7f2752164640 10 monclient: build_initial_monmap 2024-09-11T01:51:59.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.563+0000 7f2752164640 10 monclient: monmap: 2024-09-11T01:51:59.573 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:59.573 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:51:59.573 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:51:59.573 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:51:59.573 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:59.573 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:59.573 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:51:59.574 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:51:59.574 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:51:59.574 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:59.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.563+0000 7f2752164640 1 Processor -- start 2024-09-11T01:51:59.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.563+0000 7f2752164640 1 -- start start 2024-09-11T01:51:59.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f2752164640 10 monclient: init 2024-09-11T01:51:59.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f2752164640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:59.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f2752164640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:51:59.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f2752164640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f274c073d90 0x7f274c1312d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:59.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f2752164640 10 monclient(hunting): picked mon.noname-a con 0x7f274c073d90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:51:59.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f2752164640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f274c0a0660 0x7f274c131810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:59.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f2752164640 10 monclient(hunting): picked mon.noname-c con 0x7f274c0a0660 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:59.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f2752164640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f274c1380d0 0x7f274c1bf850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:59.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f2752164640 10 monclient(hunting): picked mon.noname-b con 0x7f274c1380d0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:51:59.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f2752164640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:59.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f2752164640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f274c0a44d0 con 0x7f274c073d90 2024-09-11T01:51:59.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f2752164640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:59.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f2752164640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f274c0a4350 con 0x7f274c1380d0 2024-09-11T01:51:59.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f2752164640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:59.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f2752164640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f274c0a31b0 con 0x7f274c0a0660 2024-09-11T01:51:59.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f2752164640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:59.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f2752164640 10 monclient(hunting): authenticate will time out at 2126.006479s 2024-09-11T01:51:59.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f274c073d90 0x7f274c1312d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:59.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f274c073d90 0x7f274c1312d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37912/0 (socket says 172.21.15.12:37912) 2024-09-11T01:51:59.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274affd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f274c0a0660 0x7f274c131810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:59.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274b7fe640 1 -- 172.21.15.12:0/3163513789 learned_addr learned my addr 172.21.15.12:0/3163513789 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:59.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274b7fe640 10 monclient(hunting): get_auth_request con 0x7f274c073d90 auth_method 0 2024-09-11T01:51:59.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:59.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:59.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:59.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274affd640 10 monclient(hunting): get_auth_request con 0x7f274c0a0660 auth_method 0 2024-09-11T01:51:59.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:59.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:59.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:59.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:59.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:59.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:59.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:59.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:59.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:59.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274bfff640 1 --2- 172.21.15.12:0/3163513789 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f274c1380d0 0x7f274c1bf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:59.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274b7fe640 10 monclient(hunting): handle_auth_done global_id 254100 payload 824 2024-09-11T01:51:59.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274b7fe640 1 -- 172.21.15.12:0/3163513789 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f274c0a0660 msgr2=0x7f274c131810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:59.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274b7fe640 1 --2- 172.21.15.12:0/3163513789 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f274c0a0660 0x7f274c131810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:59.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274b7fe640 1 -- 172.21.15.12:0/3163513789 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f274c1380d0 msgr2=0x7f274c1bf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:59.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274b7fe640 1 --2- 172.21.15.12:0/3163513789 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f274c1380d0 0x7f274c1bf850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:59.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:51:59.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274b7fe640 1 monclient: found mon.noname-a 2024-09-11T01:51:59.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:59.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:51:59.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274b7fe640 1 -- 172.21.15.12:0/3163513789 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f274c1c1220 con 0x7f274c073d90 2024-09-11T01:51:59.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274affd640 1 --2- 172.21.15.12:0/3163513789 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f274c0a0660 0x7f274c131810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:51:59.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274bfff640 10 monclient: get_auth_request con 0x7f274c1380d0 auth_method 0 2024-09-11T01:51:59.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.564+0000 7f274bfff640 1 --2- 172.21.15.12:0/3163513789 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f274c1380d0 0x7f274c1bf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:51:59.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.567+0000 7f274b7fe640 1 -- 172.21.15.12:0/3163513789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f274c073d90 msgr2=0x7f274c1312d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:59.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.567+0000 7f274b7fe640 1 --2- 172.21.15.12:0/3163513789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f274c073d90 0x7f274c1312d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f273c00d7d0 tx=0x7f273c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:59.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.567+0000 7f2733fff640 1 -- 172.21.15.12:0/3163513789 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273c00b8d0 con 0x7f274c073d90 2024-09-11T01:51:59.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.567+0000 7f2733fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:59.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.567+0000 7f2733fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:51:59.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.567+0000 7f2733fff640 10 monclient: dump: 2024-09-11T01:51:59.578 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:59.578 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:59.578 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:59.579 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:59.579 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:59.579 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:59.579 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:59.579 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:59.579 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:59.579 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:59.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.567+0000 7f2733fff640 10 monclient: _finish_auth 0 2024-09-11T01:51:59.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.567+0000 7f2733fff640 10 monclient: _check_auth_tickets 2024-09-11T01:51:59.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.567+0000 7f2733fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:59.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.567+0000 7f2733fff640 1 -- 172.21.15.12:0/3163513789 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f273c019070 con 0x7f274c073d90 2024-09-11T01:51:59.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.567+0000 7f2733fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:59.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.567+0000 7f2733fff640 1 -- 172.21.15.12:0/3163513789 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273c00bcd0 con 0x7f274c073d90 2024-09-11T01:51:59.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.567+0000 7f2733fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:59.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.567+0000 7f2733fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:51:59.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.567+0000 7f2733fff640 10 monclient: dump: 2024-09-11T01:51:59.580 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:59.580 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:59.580 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:59.580 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:59.580 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:59.580 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:59.580 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:59.580 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:59.580 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:59.581 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:59.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.567+0000 7f2749ffb640 4 set_mon_vals no callback set 2024-09-11T01:51:59.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.567+0000 7f2752164640 5 monclient: authenticate success, global_id 254100 2024-09-11T01:51:59.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.567+0000 7f2752164640 10 monclient: _renew_subs 2024-09-11T01:51:59.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.567+0000 7f2752164640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:51:59.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.567+0000 7f2752164640 1 -- 172.21.15.12:0/3163513789 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f274c0a6410 con 0x7f274c073d90 2024-09-11T01:51:59.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.567+0000 7f2752164640 10 monclient: _renew_subs 2024-09-11T01:51:59.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.567+0000 7f2752164640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:51:59.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.567+0000 7f2752164640 1 -- 172.21.15.12:0/3163513789 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f274c11ec90 con 0x7f274c073d90 2024-09-11T01:51:59.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.568+0000 7f2733fff640 1 -- 172.21.15.12:0/3163513789 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f273c013070 con 0x7f274c073d90 2024-09-11T01:51:59.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.568+0000 7f2752164640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:59.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.568+0000 7f2752164640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:59.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.568+0000 7f2752164640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:51:59.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.568+0000 7f2752164640 1 -- 172.21.15.12:0/3163513789 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f274c074170 con 0x7f274c073d90 2024-09-11T01:51:59.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.568+0000 7f2733fff640 1 --2- 172.21.15.12:0/3163513789 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f273803e210 0x7f27380406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:59.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.568+0000 7f2733fff640 1 -- 172.21.15.12:0/3163513789 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(763..763 src has 676..763) ==== 4843+0+0 (secure 0 0 0) 0x7f273c052f00 con 0x7f274c073d90 2024-09-11T01:51:59.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.569+0000 7f274affd640 1 --2- 172.21.15.12:0/3163513789 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f273803e210 0x7f27380406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:59.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.569+0000 7f274affd640 10 monclient: get_auth_request con 0x7f273803e210 auth_method 0 2024-09-11T01:51:59.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.571+0000 7f274affd640 1 --2- 172.21.15.12:0/3163513789 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f273803e210 0x7f27380406d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2740007fd0 tx=0x7f2740009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:59.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.573+0000 7f2733fff640 1 -- 172.21.15.12:0/3163513789 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f273c0518c0 con 0x7f274c073d90 2024-09-11T01:51:59.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.573+0000 7f2733fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:59.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.573+0000 7f2733fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:51:59.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.828+0000 7f2752164640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:51:59.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.828+0000 7f2752164640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:51:59.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.828+0000 7f2752164640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:51:59.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.828+0000 7f2752164640 1 -- 172.21.15.12:0/3163513789 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f274c11e7d0 con 0x7f274c073d90 2024-09-11T01:51:59.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.829+0000 7f2733fff640 1 -- 172.21.15.12:0/3163513789 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f273c01cd70 con 0x7f274c073d90 2024-09-11T01:51:59.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.829+0000 7f2733fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:51:59.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.829+0000 7f2733fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:51:59.831 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":214,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:51:59.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.831+0000 7f2752164640 1 -- 172.21.15.12:0/3163513789 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f273803e210 msgr2=0x7f27380406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.831+0000 7f2752164640 1 --2- 172.21.15.12:0/3163513789 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f273803e210 0x7f27380406d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2740007fd0 tx=0x7f2740009040 comp rx=0 tx=0).stop 2024-09-11T01:51:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.831+0000 7f2752164640 10 monclient: shutdown 2024-09-11T01:51:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.831+0000 7f2752164640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.831+0000 7f2752164640 1 -- 172.21.15.12:0/3163513789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f274c073d90 msgr2=0x7f274c1312d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.831+0000 7f2752164640 1 --2- 172.21.15.12:0/3163513789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f274c073d90 0x7f274c1312d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f273c00d7d0 tx=0x7f273c00dc90 comp rx=0 tx=0).stop 2024-09-11T01:51:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.831+0000 7f2752164640 1 -- 172.21.15.12:0/3163513789 shutdown_connections 2024-09-11T01:51:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.831+0000 7f2752164640 1 --2- 172.21.15.12:0/3163513789 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f274c1380d0 0x7f274c1bf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.831+0000 7f2752164640 1 --2- 172.21.15.12:0/3163513789 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f274c0a0660 0x7f274c131810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.831+0000 7f2752164640 1 --2- 172.21.15.12:0/3163513789 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f273803e210 0x7f27380406d0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.831+0000 7f2752164640 1 --2- 172.21.15.12:0/3163513789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f274c073d90 0x7f274c1312d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:59.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.831+0000 7f2752164640 1 -- 172.21.15.12:0/3163513789 >> 172.21.15.12:0/3163513789 conn(0x7f274c0a3650 msgr2=0x7f274c073400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:59.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.831+0000 7f2752164640 1 -- 172.21.15.12:0/3163513789 shutdown_connections 2024-09-11T01:51:59.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.831+0000 7f2752164640 1 -- 172.21.15.12:0/3163513789 wait complete. 2024-09-11T01:51:59.845 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:51:59.845 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:51:59.845 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:51:59.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.945+0000 7feb6e460640 10 monclient: get_monmap_and_config 2024-09-11T01:51:59.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.945+0000 7feb6e460640 10 monclient: build_initial_monmap 2024-09-11T01:51:59.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.945+0000 7feb6e460640 10 monclient: monmap: 2024-09-11T01:51:59.947 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:59.947 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:51:59.946090+0000 2024-09-11T01:51:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:51:59.946090+0000 2024-09-11T01:51:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:51:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:51:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:51:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:51:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:51:59.948 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.946+0000 7feb6e460640 1 Processor -- start 2024-09-11T01:51:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.946+0000 7feb6e460640 1 -- start start 2024-09-11T01:51:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.946+0000 7feb6e460640 10 monclient: init 2024-09-11T01:51:59.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.946+0000 7feb6e460640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:59.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.946+0000 7feb6e460640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-11T01:51:59.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.946+0000 7feb6e460640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feb68134ad0 addr v1:172.21.15.12:6789/0 2024-09-11T01:51:59.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.946+0000 7feb6e460640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feb680a2200 addr v1:172.21.15.71:6789/0 2024-09-11T01:51:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.946+0000 7feb6e460640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feb681355c0 0x7feb6813d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.946+0000 7feb6e460640 10 monclient(hunting): picked mon.noname-a con 0x7feb681355c0 addr v2:172.21.15.12:3300/0 2024-09-11T01:51:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.946+0000 7feb6e460640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb6809ed30 con 0x7feb68134ad0 2024-09-11T01:51:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.946+0000 7feb6e460640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb6809f010 con 0x7feb680a2200 2024-09-11T01:51:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.946+0000 7feb6e460640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.946+0000 7feb6e460640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7feb6809f5d0 con 0x7feb681355c0 2024-09-11T01:51:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.946+0000 7feb6e460640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.946+0000 7feb6e460640 10 monclient(hunting): authenticate will time out at 2126.388715s 2024-09-11T01:51:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.946+0000 7feb67fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7feb68134ad0 0x7feb68134eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37078/0 (socket says 172.21.15.12:37078) 2024-09-11T01:51:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.946+0000 7feb6c9fd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feb681355c0 0x7feb6813d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.946+0000 7feb67fff640 1 -- 172.21.15.12:0/2116316656 learned_addr learned my addr 172.21.15.12:0/2116316656 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:51:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.946+0000 7feb6c9fd640 10 monclient(hunting): get_auth_request con 0x7feb681355c0 auth_method 0 2024-09-11T01:51:59.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.946+0000 7feb6c9fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:59.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.946+0000 7feb6c9fd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:59.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.946+0000 7feb6c9fd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:59.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.946+0000 7feb6c9fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:59.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.946+0000 7feb6c9fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:59.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.946+0000 7feb6c9fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:59.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.947+0000 7feb6c9fd640 1 -- 172.21.15.12:0/2116316656 >> v2:172.21.15.12:3300/0 conn(0x7feb681355c0 msgr2=0x7feb6813d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:51:59.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.947+0000 7feb6c9fd640 1 -- 172.21.15.12:0/2116316656 >> v2:172.21.15.12:3300/0 conn(0x7feb681355c0 msgr2=0x7feb6813d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:51:59.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.947+0000 7feb6c9fd640 1 --2- 172.21.15.12:0/2116316656 >> v2:172.21.15.12:3300/0 conn(0x7feb681355c0 0x7feb6813d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:51:59.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.947+0000 7feb6c9fd640 1 --2- 172.21.15.12:0/2116316656 >> v2:172.21.15.12:3300/0 conn(0x7feb681355c0 0x7feb6813d730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:51:59.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.948+0000 7feb67fff640 1 -- 172.21.15.12:0/2116316656 >> v1:172.21.15.12:6789/0 conn(0x7feb68134ad0 legacy=0x7feb68134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:59.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 1 -- 172.21.15.12:0/2116316656 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1697658796 0 0) 0x7feb6809ed30 con 0x7feb68134ad0 2024-09-11T01:51:59.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 10 monclient(hunting): _negotiate 2024-09-11T01:51:59.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:59.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:59.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 10 monclient(hunting): my global_id is 254106 2024-09-11T01:51:59.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 1 -- 172.21.15.12:0/2116316656 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb50003520 con 0x7feb68134ad0 2024-09-11T01:51:59.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb677fe640 1 -- 172.21.15.12:0/2116316656 >> v1:172.21.15.71:6789/0 conn(0x7feb680a2200 legacy=0x7feb6813a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:59.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 1 -- 172.21.15.12:0/2116316656 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2832840005 0 0) 0x7feb50003520 con 0x7feb68134ad0 2024-09-11T01:51:59.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 1 -- 172.21.15.12:0/2116316656 >> v2:172.21.15.12:3300/0 conn(0x7feb681355c0 msgr2=0x7feb6813d730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:51:59.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 1 --2- 172.21.15.12:0/2116316656 >> v2:172.21.15.12:3300/0 conn(0x7feb681355c0 0x7feb6813d730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:59.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 1 -- 172.21.15.12:0/2116316656 >> v1:172.21.15.71:6789/0 conn(0x7feb680a2200 legacy=0x7feb6813a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:59.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:51:59.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:51:59.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:59.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:51:59.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 1 -- 172.21.15.12:0/2116316656 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb6809f2f0 con 0x7feb68134ad0 2024-09-11T01:51:59.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 10 monclient: _finish_auth 0 2024-09-11T01:51:59.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:51:59.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:59.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb6e460640 5 monclient: authenticate success, global_id 254106 2024-09-11T01:51:59.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb6e460640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:51:59.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 1 -- 172.21.15.12:0/2116316656 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7feb580019a0 con 0x7feb68134ad0 2024-09-11T01:51:59.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:59.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:51:59.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 10 monclient: dump: 2024-09-11T01:51:59.957 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:59.957 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:59.957 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:59.957 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:59.958 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:59.958 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:59.958 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:59.958 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:59.958 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:59.958 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:59.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:51:59.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:59.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 1 -- 172.21.15.12:0/2116316656 >> v1:172.21.15.12:6789/0 conn(0x7feb68134ad0 legacy=0x7feb68134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:59.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:51:59.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 1 --2- 172.21.15.12:0/2116316656 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb50003b20 0x7feb50003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:59.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 10 monclient(hunting): picked mon.c con 0x7feb50003b20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:59.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 1 --2- 172.21.15.12:0/2116316656 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb50004800 0x7feb50004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:59.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 10 monclient(hunting): picked mon.b con 0x7feb50004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:51:59.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 1 --2- 172.21.15.12:0/2116316656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb50005410 0x7feb5000ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:59.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 10 monclient(hunting): picked mon.a con 0x7feb50005410 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:51:59.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:59.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 1 -- 172.21.15.12:0/2116316656 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7feb6809f5d0 con 0x7feb50005410 2024-09-11T01:51:59.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:59.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 1 -- 172.21.15.12:0/2116316656 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7feb5000c320 con 0x7feb50004800 2024-09-11T01:51:59.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:59.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 1 -- 172.21.15.12:0/2116316656 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7feb5000c520 con 0x7feb50003b20 2024-09-11T01:51:59.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:59.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb66ffd640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:51:59.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb6e460640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:51:59.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb67fff640 1 --2- 172.21.15.12:0/2116316656 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb50003b20 0x7feb50003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:59.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb6c9fd640 1 --2- 172.21.15.12:0/2116316656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb50005410 0x7feb5000ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb67fff640 10 monclient(hunting): get_auth_request con 0x7feb50003b20 auth_method 0 2024-09-11T01:51:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb67fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb67fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb6c9fd640 10 monclient(hunting): get_auth_request con 0x7feb50005410 auth_method 0 2024-09-11T01:51:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb6c9fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb6c9fd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb6c9fd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb677fe640 1 --2- 172.21.15.12:0/2116316656 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb50004800 0x7feb50004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb677fe640 10 monclient(hunting): get_auth_request con 0x7feb50004800 auth_method 0 2024-09-11T01:51:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:51:59.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:59.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:59.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb67fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:51:59.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb6c9fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:59.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb6c9fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:59.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb6c9fd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:51:59.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:59.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:59.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb677fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:51:59.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb67fff640 10 monclient(hunting): handle_auth_done global_id 254106 payload 293 2024-09-11T01:51:59.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb67fff640 1 -- 172.21.15.12:0/2116316656 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb50004800 msgr2=0x7feb50004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:59.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb67fff640 1 --2- 172.21.15.12:0/2116316656 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb50004800 0x7feb50004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:59.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb67fff640 1 -- 172.21.15.12:0/2116316656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb50005410 msgr2=0x7feb5000ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:51:59.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb67fff640 1 --2- 172.21.15.12:0/2116316656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb50005410 0x7feb5000ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:59.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb67fff640 10 monclient: _finish_hunting 0 2024-09-11T01:51:59.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb67fff640 1 monclient: found mon.c 2024-09-11T01:51:59.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.949+0000 7feb67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:59.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.950+0000 7feb67fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:59.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.950+0000 7feb67fff640 1 -- 172.21.15.12:0/2116316656 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb50003520 con 0x7feb50003b20 2024-09-11T01:51:59.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.950+0000 7feb67fff640 10 monclient: _finish_auth 0 2024-09-11T01:51:59.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.950+0000 7feb67fff640 10 monclient: _check_auth_tickets 2024-09-11T01:51:59.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.950+0000 7feb67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:59.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.950+0000 7feb6c9fd640 1 --2- 172.21.15.12:0/2116316656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb50005410 0x7feb5000ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:51:59.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.950+0000 7feb6c9fd640 1 -- 172.21.15.12:0/2116316656 reap_dead start 2024-09-11T01:51:59.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.952+0000 7feb67fff640 1 -- 172.21.15.12:0/2116316656 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb50003b20 msgr2=0x7feb50003f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:59.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.952+0000 7feb67fff640 1 --2- 172.21.15.12:0/2116316656 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb50003b20 0x7feb50003f60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7feb580014d0 tx=0x7feb50005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c07476483a345eb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:59.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.952+0000 7feb66ffd640 1 -- 172.21.15.12:0/2116316656 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb58036070 con 0x7feb50003b20 2024-09-11T01:51:59.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.952+0000 7feb66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:59.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.952+0000 7feb66ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:59.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.952+0000 7feb66ffd640 10 monclient: dump: 2024-09-11T01:51:59.969 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:59.969 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:59.969 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:59.970 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:59.970 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:59.970 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:59.970 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:59.970 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:59.970 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:59.970 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:59.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.952+0000 7feb66ffd640 1 -- 172.21.15.12:0/2116316656 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb58005af0 con 0x7feb50003b20 2024-09-11T01:51:59.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.952+0000 7feb6e460640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:51:59.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.953+0000 7feb66ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:59.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.953+0000 7feb66ffd640 1 -- 172.21.15.12:0/2116316656 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb5800c0e0 con 0x7feb50003b20 2024-09-11T01:51:59.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.953+0000 7feb6e460640 10 monclient: get_monmap_and_config success 2024-09-11T01:51:59.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.953+0000 7feb6e460640 4 set_mon_vals no callback set 2024-09-11T01:51:59.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.953+0000 7feb66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:59.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.953+0000 7feb66ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:59.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.953+0000 7feb66ffd640 10 monclient: dump: 2024-09-11T01:51:59.971 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:59.971 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:59.971 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:59.971 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:59.971 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:59.971 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:59.971 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:59.971 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:59.971 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:59.971 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:59.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.953+0000 7feb6e460640 10 monclient: shutdown 2024-09-11T01:51:59.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.953+0000 7feb6e460640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:51:59.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.953+0000 7feb6e460640 1 -- 172.21.15.12:0/2116316656 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb50003b20 msgr2=0x7feb50003f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:59.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.953+0000 7feb6e460640 1 --2- 172.21.15.12:0/2116316656 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb50003b20 0x7feb50003f60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7feb580014d0 tx=0x7feb50005830 comp rx=0 tx=0).stop 2024-09-11T01:51:59.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.953+0000 7feb6e460640 1 -- 172.21.15.12:0/2116316656 shutdown_connections 2024-09-11T01:51:59.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.953+0000 7feb6e460640 1 --2- 172.21.15.12:0/2116316656 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb50003b20 0x7feb50003f60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:59.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.953+0000 7feb6e460640 1 -- 172.21.15.12:0/2116316656 >> 172.21.15.12:0/2116316656 conn(0x7feb680a2f70 msgr2=0x7feb680a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:51:59.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.953+0000 7feb6e460640 1 -- 172.21.15.12:0/2116316656 shutdown_connections 2024-09-11T01:51:59.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.953+0000 7feb6e460640 1 -- 172.21.15.12:0/2116316656 wait complete. 2024-09-11T01:51:59.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.954+0000 7feb6e460640 10 monclient: build_initial_monmap 2024-09-11T01:51:59.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.954+0000 7feb6e460640 10 monclient: monmap: 2024-09-11T01:51:59.972 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:51:59.972 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:51:59.972 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:51:59.973 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:51:59.973 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:51:59.973 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:59.973 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:51:59.973 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:51:59.973 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:51:59.973 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:59.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.954+0000 7feb6e460640 1 Processor -- start 2024-09-11T01:51:59.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.954+0000 7feb6e460640 1 -- start start 2024-09-11T01:51:59.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.954+0000 7feb6e460640 10 monclient: init 2024-09-11T01:51:59.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb6e460640 10 monclient: _reopen_session rank -1 2024-09-11T01:51:59.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb6e460640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:51:59.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb6e460640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb50003b20 0x7feb681cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:59.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb6e460640 10 monclient(hunting): picked mon.noname-a con 0x7feb50003b20 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:51:59.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb6e460640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb681cd910 0x7feb681224b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:59.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb6e460640 10 monclient(hunting): picked mon.noname-c con 0x7feb681cd910 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:51:59.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb6e460640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb681229f0 0x7feb68122e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:59.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb6e460640 10 monclient(hunting): picked mon.noname-b con 0x7feb681229f0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:51:59.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb6e460640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:59.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb6e460640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7feb680a0150 con 0x7feb50003b20 2024-09-11T01:51:59.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb6e460640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:59.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb6e460640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7feb6809ffd0 con 0x7feb681229f0 2024-09-11T01:51:59.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb6e460640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:59.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb6e460640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7feb68134eb0 con 0x7feb681cd910 2024-09-11T01:51:59.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb6e460640 10 monclient(hunting): _renew_subs 2024-09-11T01:51:59.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb6e460640 10 monclient(hunting): authenticate will time out at 2126.397322s 2024-09-11T01:51:59.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb67fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb50003b20 0x7feb681cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:59.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb677fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb681cd910 0x7feb681224b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:59.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb67fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb50003b20 0x7feb681cd3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37940/0 (socket says 172.21.15.12:37940) 2024-09-11T01:51:59.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb67fff640 1 -- 172.21.15.12:0/2101973507 learned_addr learned my addr 172.21.15.12:0/2101973507 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:51:59.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb67fff640 10 monclient(hunting): get_auth_request con 0x7feb50003b20 auth_method 0 2024-09-11T01:51:59.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:59.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb67fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:59.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:59.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb677fe640 10 monclient(hunting): get_auth_request con 0x7feb681cd910 auth_method 0 2024-09-11T01:51:59.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:59.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:59.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:59.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:59.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:59.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:59.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:59.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:59.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:59.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb6c9fd640 1 --2- 172.21.15.12:0/2101973507 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb681229f0 0x7feb68122e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:59.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb6c9fd640 10 monclient(hunting): get_auth_request con 0x7feb681229f0 auth_method 0 2024-09-11T01:51:59.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb6c9fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:59.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb6c9fd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:59.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb6c9fd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:59.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb677fe640 10 monclient(hunting): handle_auth_done global_id 244208 payload 824 2024-09-11T01:51:59.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb677fe640 1 -- 172.21.15.12:0/2101973507 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb681229f0 msgr2=0x7feb68122e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:59.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb677fe640 1 --2- 172.21.15.12:0/2101973507 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb681229f0 0x7feb68122e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:59.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb677fe640 1 -- 172.21.15.12:0/2101973507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb50003b20 msgr2=0x7feb681cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb677fe640 1 --2- 172.21.15.12:0/2101973507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb50003b20 0x7feb681cd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb677fe640 10 monclient: _finish_hunting 0 2024-09-11T01:51:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb677fe640 1 monclient: found mon.noname-c 2024-09-11T01:51:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:51:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb677fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:51:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb677fe640 1 -- 172.21.15.12:0/2101973507 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb6809d480 con 0x7feb681cd910 2024-09-11T01:51:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.955+0000 7feb67fff640 1 --2- 172.21.15.12:0/2101973507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb50003b20 0x7feb681cd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:51:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.957+0000 7feb677fe640 1 -- 172.21.15.12:0/2101973507 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb681cd910 msgr2=0x7feb681224b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.957+0000 7feb677fe640 1 --2- 172.21.15.12:0/2101973507 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb681cd910 0x7feb681224b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7feb5c006df0 tx=0x7feb5c0019c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.958+0000 7feb64ff9640 1 -- 172.21.15.12:0/2101973507 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb5c011790 con 0x7feb681cd910 2024-09-11T01:51:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.958+0000 7feb64ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.958+0000 7feb64ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:51:59.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.958+0000 7feb64ff9640 10 monclient: dump: 2024-09-11T01:51:59.978 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:59.978 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:59.978 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:59.978 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:59.978 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:59.978 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:59.978 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:59.978 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:59.978 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:59.978 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:59.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.958+0000 7feb64ff9640 10 monclient: _finish_auth 0 2024-09-11T01:51:59.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.958+0000 7feb64ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:51:59.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.958+0000 7feb64ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:51:59.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.958+0000 7feb64ff9640 1 -- 172.21.15.12:0/2101973507 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb5c018070 con 0x7feb681cd910 2024-09-11T01:51:59.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.958+0000 7feb64ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:51:59.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.958+0000 7feb64ff9640 1 -- 172.21.15.12:0/2101973507 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb5c011b30 con 0x7feb681cd910 2024-09-11T01:51:59.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.958+0000 7feb64ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:59.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.958+0000 7feb64ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:51:59.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.958+0000 7feb64ff9640 10 monclient: dump: 2024-09-11T01:51:59.979 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:59.979 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:59.979 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:59.979 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:59.979 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:59.979 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:59.979 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:59.979 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:59.979 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:59.979 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.958+0000 7feb6e460640 5 monclient: authenticate success, global_id 244208 2024-09-11T01:51:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.958+0000 7feb6e460640 10 monclient: _renew_subs 2024-09-11T01:51:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.958+0000 7feb6e460640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.958+0000 7feb6e460640 1 -- 172.21.15.12:0/2101973507 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb68123f70 con 0x7feb681cd910 2024-09-11T01:51:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.958+0000 7feb667fc640 4 set_mon_vals no callback set 2024-09-11T01:51:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.958+0000 7feb6e460640 10 monclient: _renew_subs 2024-09-11T01:51:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.958+0000 7feb6e460640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:51:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.958+0000 7feb6e460640 1 -- 172.21.15.12:0/2101973507 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7feb68125a30 con 0x7feb681cd910 2024-09-11T01:51:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.959+0000 7feb64ff9640 1 -- 172.21.15.12:0/2101973507 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7feb5c011cd0 con 0x7feb681cd910 2024-09-11T01:51:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.959+0000 7feb6e460640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.959+0000 7feb6e460640 1 --2- 172.21.15.12:0/2101973507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb6802e5c0 0x7feb681233b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.959+0000 7feb6e460640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:59.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.959+0000 7feb6e460640 1 -- 172.21.15.12:0/2101973507 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7feb680a4620 con 0x7feb6802e5c0 2024-09-11T01:51:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.959+0000 7feb67fff640 1 --2- 172.21.15.12:0/2101973507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb6802e5c0 0x7feb681233b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.959+0000 7feb67fff640 10 monclient: get_auth_request con 0x7feb6802e5c0 auth_method 0 2024-09-11T01:51:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.959+0000 7feb67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.959+0000 7feb67fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.959+0000 7feb67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.959+0000 7feb64ff9640 1 --2- 172.21.15.12:0/2101973507 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7feb5403e290 0x7feb54040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.959+0000 7feb67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.959+0000 7feb67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.959+0000 7feb67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.959+0000 7feb64ff9640 1 -- 172.21.15.12:0/2101973507 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(763..763 src has 676..763) ==== 4843+0+0 (secure 0 0 0) 0x7feb5c052ad0 con 0x7feb681cd910 2024-09-11T01:51:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.959+0000 7feb67fff640 10 monclient(hunting): handle_auth_done global_id 254112 payload 824 2024-09-11T01:51:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.959+0000 7feb67fff640 1 -- 172.21.15.12:0/2101973507 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feb6802e9a0 con 0x7feb6802e5c0 2024-09-11T01:51:59.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.959+0000 7feb6c9fd640 1 --2- 172.21.15.12:0/2101973507 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7feb5403e290 0x7feb54040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.959+0000 7feb6c9fd640 10 monclient: get_auth_request con 0x7feb5403e290 auth_method 0 2024-09-11T01:51:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.961+0000 7feb67fff640 1 -- 172.21.15.12:0/2101973507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb6802e5c0 msgr2=0x7feb681233b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.961+0000 7feb67fff640 1 --2- 172.21.15.12:0/2101973507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb6802e5c0 0x7feb681233b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7feb58034040 tx=0x7feb6809ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.961+0000 7feb64ff9640 1 -- 172.21.15.12:0/2101973507 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb58036030 con 0x7feb6802e5c0 2024-09-11T01:51:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.961+0000 7feb64ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.961+0000 7feb64ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:51:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.961+0000 7feb64ff9640 10 monclient: dump: 2024-09-11T01:51:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:59.982 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:59.983 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:59.983 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:59.983 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:59.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.962+0000 7feb6c9fd640 1 --2- 172.21.15.12:0/2101973507 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7feb5403e290 0x7feb54040750 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7feb68135ab0 tx=0x7feb6000b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:59.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.962+0000 7feb64ff9640 1 -- 172.21.15.12:0/2101973507 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7feb58007490 con 0x7feb6802e5c0 2024-09-11T01:51:59.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.962+0000 7feb64ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:51:59.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.962+0000 7feb64ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:51:59.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.962+0000 7feb64ff9640 1 -- 172.21.15.12:0/2101973507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb6802e5c0 msgr2=0x7feb681233b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:59.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.962+0000 7feb64ff9640 1 --2- 172.21.15.12:0/2101973507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb6802e5c0 0x7feb681233b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7feb58034040 tx=0x7feb6809ef00 comp rx=0 tx=0).stop 2024-09-11T01:51:59.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.962+0000 7feb64ff9640 1 -- 172.21.15.12:0/2101973507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb6802e5c0 msgr2=0x7feb681233b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:51:59.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.962+0000 7feb64ff9640 1 --2- 172.21.15.12:0/2101973507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb6802e5c0 0x7feb681233b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:51:59.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.968+0000 7feb6e460640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:51:59.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.968+0000 7feb6e460640 1 --2- 172.21.15.12:0/2101973507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb68063d50 0x7feb68064160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:51:59.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.968+0000 7feb6e460640 10 monclient(hunting): start opening mon connection 2024-09-11T01:51:59.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.968+0000 7feb6e460640 1 -- 172.21.15.12:0/2101973507 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7feb681238f0 con 0x7feb68063d50 2024-09-11T01:51:59.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.968+0000 7feb67fff640 1 --2- 172.21.15.12:0/2101973507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb68063d50 0x7feb68064160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:51:59.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.968+0000 7feb67fff640 10 monclient: get_auth_request con 0x7feb68063d50 auth_method 0 2024-09-11T01:51:59.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.968+0000 7feb67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:51:59.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.968+0000 7feb67fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:51:59.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.968+0000 7feb67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:51:59.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.968+0000 7feb67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:51:59.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.968+0000 7feb67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:51:59.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.968+0000 7feb67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:51:59.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.968+0000 7feb67fff640 10 monclient(hunting): handle_auth_done global_id 254115 payload 824 2024-09-11T01:51:59.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.968+0000 7feb67fff640 1 -- 172.21.15.12:0/2101973507 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7feb681351e0 con 0x7feb68063d50 2024-09-11T01:51:59.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.970+0000 7feb67fff640 1 -- 172.21.15.12:0/2101973507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb68063d50 msgr2=0x7feb68064160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:51:59.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.970+0000 7feb67fff640 1 --2- 172.21.15.12:0/2101973507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb68063d50 0x7feb68064160 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7feb58032000 tx=0x7feb5802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:51:59.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.971+0000 7feb64ff9640 1 -- 172.21.15.12:0/2101973507 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb5803fbe0 con 0x7feb68063d50 2024-09-11T01:51:59.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.971+0000 7feb64ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:51:59.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.971+0000 7feb64ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:51:59.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.971+0000 7feb64ff9640 10 monclient: dump: 2024-09-11T01:51:59.986 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:51:59.986 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:51:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:51:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:51:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:51:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:51:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:51:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:51:59.987 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:51:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.971+0000 7feb64ff9640 1 -- 172.21.15.12:0/2101973507 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4542 (secure 0 0 0) 0x7feb5803fbe0 con 0x7feb68063d50 2024-09-11T01:51:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.971+0000 7feb64ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:51:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.971+0000 7feb64ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:51:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.971+0000 7feb64ff9640 1 -- 172.21.15.12:0/2101973507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb68063d50 msgr2=0x7feb68064160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:51:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.971+0000 7feb64ff9640 1 --2- 172.21.15.12:0/2101973507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb68063d50 0x7feb68064160 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7feb58032000 tx=0x7feb5802e000 comp rx=0 tx=0).stop 2024-09-11T01:51:59.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.971+0000 7feb64ff9640 1 -- 172.21.15.12:0/2101973507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb68063d50 msgr2=0x7feb68064160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:51:59.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.971+0000 7feb64ff9640 1 --2- 172.21.15.12:0/2101973507 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb68063d50 0x7feb68064160 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7feb58032000 tx=0x7feb5802e000 comp rx=0 tx=0).stop 2024-09-11T01:51:59.988 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:51:59.988 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:51:59.988 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:51:59.988 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:51:59.988 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 214, 2024-09-11T01:51:59.988 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 7249, 2024-09-11T01:51:59.988 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:51:59.988 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:51:59.988 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:51:59.989 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:51:59.989 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:59.989 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:51:59.989 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:51:59.989 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:51:59.989 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:51:59.989 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:59.989 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:59.989 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:59.989 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:59.989 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:59.989 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:59.989 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:59.990 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:59.990 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:59.990 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:59.990 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:59.990 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:59.990 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:51:59.990 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:51:59.990 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:59.990 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:59.990 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:59.990 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:59.990 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:59.990 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:59.991 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:59.991 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:59.991 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:59.991 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:59.991 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:59.991 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:59.991 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:59.991 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:51:59.991 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:51:59.991 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:51:59.991 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:51:59.991 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:51:59.991 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:51:59.991 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:51:59.992 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:51:59.992 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:51:59.992 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:51:59.992 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:51:59.992 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:51:59.992 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:51:59.992 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:51:59.992 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:51:59.992 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:51:59.992 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:51:59.992 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:51:59.992 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:51:59.992 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:51:59.992 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:51:59.993 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:51:59.993 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:51:59.993 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:51:59.993 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:51:59.993 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:51:59.993 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:51:59.993 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:51:59.993 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:59.993 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:51:59.993 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:59.993 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:51:59.993 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:59.993 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:51:59.994 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:51:59.994 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:59.994 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:59.994 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:59.994 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:59.994 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:51:59.994 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:59.994 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:59.994 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:59.994 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:59.994 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:51:59.994 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:59.994 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:59.994 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:59.995 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:59.995 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:51:59.995 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:51:59.995 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:59.995 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:59.995 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:59.995 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:59.995 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:59.995 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:51:59.995 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:51:59.995 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:59.995 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:59.995 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:59.995 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:59.996 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:51:59.996 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:59.996 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:59.996 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:59.996 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:59.996 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:51:59.996 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:59.996 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:59.996 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:59.996 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:59.996 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:51:59.996 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:51:59.996 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:59.996 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:59.997 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:59.997 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:59.997 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:59.997 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:51:59.997 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:51:59.997 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:51:59.997 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:51:59.997 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:59.997 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:51:59.997 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:51:59.997 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:59.997 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:59.997 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:59.997 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:51:59.998 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:51:59.998 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:51:59.998 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:59.998 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:59.998 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:59.998 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:51:59.998 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:51:59.998 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:51:59.998 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:51:59.998 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:51:59.998 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:59.998 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:51:59.998 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:51:59.999 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:51:59.999 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:51:59.999 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:59.999 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:59.999 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:59.999 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:51:59.999 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:51:59.999 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:51:59.999 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:51:59.999 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:51:59.999 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:51:59.999 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:51:59.999 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:51:59.999 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:00.000 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:00.000 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:00.000 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:52:00.000 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:52:00.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.971+0000 7feb6e460640 1 -- 172.21.15.12:0/2101973507 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7feb5403e290 msgr2=0x7feb54040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:00.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.971+0000 7feb6e460640 1 --2- 172.21.15.12:0/2101973507 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7feb5403e290 0x7feb54040750 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7feb68135ab0 tx=0x7feb6000b040 comp rx=0 tx=0).stop 2024-09-11T01:52:00.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.971+0000 7feb6e460640 10 monclient: shutdown 2024-09-11T01:52:00.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.971+0000 7feb6e460640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:00.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.971+0000 7feb6e460640 1 -- 172.21.15.12:0/2101973507 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb681cd910 msgr2=0x7feb681224b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:00.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.971+0000 7feb6e460640 1 --2- 172.21.15.12:0/2101973507 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb681cd910 0x7feb681224b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7feb5c006df0 tx=0x7feb5c0019c0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.971+0000 7feb6c9fd640 1 -- 172.21.15.12:0/2101973507 reap_dead start 2024-09-11T01:52:00.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.971+0000 7feb6e460640 1 -- 172.21.15.12:0/2101973507 shutdown_connections 2024-09-11T01:52:00.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.971+0000 7feb6e460640 1 --2- 172.21.15.12:0/2101973507 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb681cd910 0x7feb681224b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.971+0000 7feb6e460640 1 -- 172.21.15.12:0/2101973507 >> 172.21.15.12:0/2101973507 conn(0x7feb680a2f70 msgr2=0x7feb680a0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:00.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.971+0000 7feb6e460640 1 -- 172.21.15.12:0/2101973507 shutdown_connections 2024-09-11T01:52:00.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:51:59.971+0000 7feb6e460640 1 -- 172.21.15.12:0/2101973507 wait complete. 2024-09-11T01:52:00.002 INFO:tasks.rados.rados.0.smithi071.stdout:3401: expect (ObjNum 1188 snap 379 seq_num 1188) 2024-09-11T01:52:00.003 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:52:00.003 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:52:00.003 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:52:00.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.103+0000 7f987667c640 10 monclient: get_monmap_and_config 2024-09-11T01:52:00.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.103+0000 7f987667c640 10 monclient: build_initial_monmap 2024-09-11T01:52:00.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.103+0000 7f987667c640 10 monclient: monmap: 2024-09-11T01:52:00.105 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:00.105 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:00.105 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:00.104381+0000 2024-09-11T01:52:00.105 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:00.104381+0000 2024-09-11T01:52:00.105 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:00.106 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:00.106 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:00.106 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:00.106 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:00.106 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:00.106 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:00.106 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:00.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.104+0000 7f987667c640 1 Processor -- start 2024-09-11T01:52:00.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.104+0000 7f987667c640 1 -- start start 2024-09-11T01:52:00.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.104+0000 7f987667c640 10 monclient: init 2024-09-11T01:52:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f987667c640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f987667c640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-11T01:52:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f987667c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f98700a2e20 0x7f987006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f987667c640 10 monclient(hunting): picked mon.noname-a con 0x7f98700a2e20 addr v2:172.21.15.12:3300/0 2024-09-11T01:52:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f987667c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f98700a4970 0x7f98700a3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f987667c640 10 monclient(hunting): picked mon.noname-c con 0x7f98700a4970 addr v2:172.21.15.71:3300/0 2024-09-11T01:52:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f987667c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f987006f910 addr v1:172.21.15.71:6789/0 2024-09-11T01:52:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f987667c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f987006f4f0 con 0x7f987006f910 2024-09-11T01:52:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f987667c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f987667c640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f987006ef30 con 0x7f98700a2e20 2024-09-11T01:52:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f987667c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f987667c640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f987006ec50 con 0x7f98700a4970 2024-09-11T01:52:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f987667c640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f987667c640 10 monclient(hunting): authenticate will time out at 2126.547119s 2024-09-11T01:52:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f986ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f98700a2e20 0x7f987006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f986ffff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f98700a2e20 0x7f987006ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37962/0 (socket says 172.21.15.12:37962) 2024-09-11T01:52:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f986ffff640 1 -- 172.21.15.12:0/2146142730 learned_addr learned my addr 172.21.15.12:0/2146142730 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f986ffff640 10 monclient(hunting): get_auth_request con 0x7f98700a2e20 auth_method 0 2024-09-11T01:52:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f986ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f986ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f986ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f9867fff640 1 --2- 172.21.15.12:0/2146142730 >> v2:172.21.15.71:3300/0 conn(0x7f98700a4970 0x7f98700a3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f9867fff640 10 monclient(hunting): get_auth_request con 0x7f98700a4970 auth_method 0 2024-09-11T01:52:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f9867fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f9867fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f9867fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f986ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f986ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f986ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f9867fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f9867fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f9867fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f986ffff640 10 monclient(hunting): handle_auth_done global_id 254118 payload 274 2024-09-11T01:52:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f986ffff640 1 -- 172.21.15.12:0/2146142730 >> v2:172.21.15.71:3300/0 conn(0x7f98700a4970 msgr2=0x7f98700a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f986ffff640 1 --2- 172.21.15.12:0/2146142730 >> v2:172.21.15.71:3300/0 conn(0x7f98700a4970 0x7f98700a3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f986ffff640 1 -- 172.21.15.12:0/2146142730 >> v1:172.21.15.71:6789/0 conn(0x7f987006f910 legacy=0x7f9870137fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f986ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:52:00.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f986ffff640 1 monclient: found mon.noname-a 2024-09-11T01:52:00.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f986ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:00.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f986ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:52:00.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.105+0000 7f986ffff640 1 -- 172.21.15.12:0/2146142730 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98700a3740 con 0x7f98700a2e20 2024-09-11T01:52:00.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.108+0000 7f986ffff640 1 -- 172.21.15.12:0/2146142730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98700a2e20 msgr2=0x7f987006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:00.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.108+0000 7f986ffff640 1 --2- 172.21.15.12:0/2146142730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98700a2e20 0x7f987006ded0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9870074f40 tx=0x7f986002a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7080047a074d74e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:00.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.108+0000 7f986f7fe640 1 -- 172.21.15.12:0/2146142730 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9860037070 con 0x7f98700a2e20 2024-09-11T01:52:00.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.108+0000 7f986f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:00.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.108+0000 7f986f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:52:00.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.108+0000 7f986f7fe640 10 monclient: dump: 2024-09-11T01:52:00.119 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:00.119 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:00.120 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.120 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.120 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:00.120 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:00.120 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:00.120 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:00.120 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:00.120 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:00.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.108+0000 7f986f7fe640 10 monclient: _finish_auth 0 2024-09-11T01:52:00.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.108+0000 7f986f7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:52:00.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.108+0000 7f986f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:00.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.108+0000 7f986f7fe640 1 -- 172.21.15.12:0/2146142730 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9860032640 con 0x7f98700a2e20 2024-09-11T01:52:00.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.108+0000 7f987667c640 5 monclient: authenticate success, global_id 254118 2024-09-11T01:52:00.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.108+0000 7f986f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:00.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.108+0000 7f987667c640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:00.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.108+0000 7f986f7fe640 1 -- 172.21.15.12:0/2146142730 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9860032a10 con 0x7f98700a2e20 2024-09-11T01:52:00.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.108+0000 7f987667c640 4 set_mon_vals no callback set 2024-09-11T01:52:00.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.108+0000 7f986f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:00.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.108+0000 7f986f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:52:00.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.108+0000 7f986f7fe640 10 monclient: dump: 2024-09-11T01:52:00.121 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:00.122 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:00.122 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.122 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.122 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:00.122 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:00.122 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:00.122 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:00.122 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:00.122 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:00.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.109+0000 7f987667c640 10 monclient: shutdown 2024-09-11T01:52:00.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.109+0000 7f987667c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:00.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.109+0000 7f987667c640 1 -- 172.21.15.12:0/2146142730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98700a2e20 msgr2=0x7f987006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:00.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.109+0000 7f987667c640 1 --2- 172.21.15.12:0/2146142730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98700a2e20 0x7f987006ded0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9870074f40 tx=0x7f986002a0b0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.109+0000 7f987667c640 1 -- 172.21.15.12:0/2146142730 shutdown_connections 2024-09-11T01:52:00.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.109+0000 7f987667c640 1 --2- 172.21.15.12:0/2146142730 >> v2:172.21.15.71:3300/0 conn(0x7f98700a4970 0x7f98700a3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.109+0000 7f987667c640 1 --2- 172.21.15.12:0/2146142730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98700a2e20 0x7f987006ded0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.109+0000 7f987667c640 1 -- 172.21.15.12:0/2146142730 >> 172.21.15.12:0/2146142730 conn(0x7f98700a8600 msgr2=0x7f98700a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:00.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.109+0000 7f987667c640 1 -- 172.21.15.12:0/2146142730 shutdown_connections 2024-09-11T01:52:00.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.109+0000 7f987667c640 1 -- 172.21.15.12:0/2146142730 wait complete. 2024-09-11T01:52:00.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.109+0000 7f987667c640 10 monclient: build_initial_monmap 2024-09-11T01:52:00.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.109+0000 7f987667c640 10 monclient: monmap: 2024-09-11T01:52:00.123 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:00.123 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:00.123 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:00.123 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:00.123 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:00.123 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:00.124 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f987667c640 1 Processor -- start 2024-09-11T01:52:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f987667c640 1 -- start start 2024-09-11T01:52:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f987667c640 10 monclient: init 2024-09-11T01:52:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f987667c640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f987667c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:52:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f987667c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f987006f910 0x7f98701cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f987667c640 10 monclient(hunting): picked mon.noname-c con 0x7f987006f910 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f987667c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98700a2e20 0x7f98701cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f987667c640 10 monclient(hunting): picked mon.noname-b con 0x7f98700a2e20 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f987667c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98700a4970 0x7f9870125080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f987667c640 10 monclient(hunting): picked mon.noname-a con 0x7f98700a4970 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f987667c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f987667c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9870138860 con 0x7f98700a4970 2024-09-11T01:52:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f987667c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f987667c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f98701386e0 con 0x7f98700a2e20 2024-09-11T01:52:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f987667c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f987667c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f98701389e0 con 0x7f987006f910 2024-09-11T01:52:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f987667c640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f987667c640 10 monclient(hunting): authenticate will time out at 2126.552716s 2024-09-11T01:52:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f986ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f987006f910 0x7f98701cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f9874c19640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98700a4970 0x7f9870125080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f986ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f987006f910 0x7f98701cd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55628/0 (socket says 172.21.15.12:55628) 2024-09-11T01:52:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f986ffff640 1 -- 172.21.15.12:0/3626973067 learned_addr learned my addr 172.21.15.12:0/3626973067 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f986ffff640 10 monclient(hunting): get_auth_request con 0x7f987006f910 auth_method 0 2024-09-11T01:52:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f986ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f986ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f986ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f9874c19640 10 monclient(hunting): get_auth_request con 0x7f98700a4970 auth_method 0 2024-09-11T01:52:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f9874c19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f9874c19640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f9874c19640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f9867fff640 1 --2- 172.21.15.12:0/3626973067 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98700a2e20 0x7f98701cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f9867fff640 10 monclient(hunting): get_auth_request con 0x7f98700a2e20 auth_method 0 2024-09-11T01:52:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f9867fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f9867fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f9867fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f986ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f986ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f986ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f9874c19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f9874c19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f9874c19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f9867fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f9867fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.110+0000 7f9867fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.111+0000 7f986ffff640 10 monclient(hunting): handle_auth_done global_id 244211 payload 824 2024-09-11T01:52:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.111+0000 7f986ffff640 1 -- 172.21.15.12:0/3626973067 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98700a2e20 msgr2=0x7f98701cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.111+0000 7f986ffff640 1 --2- 172.21.15.12:0/3626973067 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f98700a2e20 0x7f98701cd9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.111+0000 7f986ffff640 1 -- 172.21.15.12:0/3626973067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98700a4970 msgr2=0x7f9870125080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.111+0000 7f986ffff640 1 --2- 172.21.15.12:0/3626973067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98700a4970 0x7f9870125080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.111+0000 7f986ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:52:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.111+0000 7f986ffff640 1 monclient: found mon.noname-c 2024-09-11T01:52:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.111+0000 7f986ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.111+0000 7f986ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:52:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.111+0000 7f986ffff640 1 -- 172.21.15.12:0/3626973067 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9870129770 con 0x7f987006f910 2024-09-11T01:52:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.111+0000 7f9874c19640 1 --2- 172.21.15.12:0/3626973067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f98700a4970 0x7f9870125080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:52:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.113+0000 7f986ffff640 1 -- 172.21.15.12:0/3626973067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f987006f910 msgr2=0x7f98701cd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.113+0000 7f986ffff640 1 --2- 172.21.15.12:0/3626973067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f987006f910 0x7f98701cd470 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f98600090b0 tx=0x7f98600046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.113+0000 7f986d7fa640 1 -- 172.21.15.12:0/3626973067 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9860041020 con 0x7f987006f910 2024-09-11T01:52:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.113+0000 7f986d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:00.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.113+0000 7f986d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:00.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.113+0000 7f986d7fa640 10 monclient: dump: 2024-09-11T01:52:00.129 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:00.129 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:00.129 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.129 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.129 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:00.129 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:00.129 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:00.129 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:00.129 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:00.129 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:00.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.113+0000 7f986d7fa640 10 monclient: _finish_auth 0 2024-09-11T01:52:00.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.113+0000 7f986d7fa640 10 monclient: _check_auth_tickets 2024-09-11T01:52:00.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.113+0000 7f986d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:00.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.113+0000 7f986d7fa640 1 -- 172.21.15.12:0/3626973067 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f986003d850 con 0x7f987006f910 2024-09-11T01:52:00.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.113+0000 7f987667c640 5 monclient: authenticate success, global_id 244211 2024-09-11T01:52:00.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.113+0000 7f986d7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:00.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.113+0000 7f986d7fa640 1 -- 172.21.15.12:0/3626973067 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9860037040 con 0x7f987006f910 2024-09-11T01:52:00.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.113+0000 7f987667c640 10 monclient: _renew_subs 2024-09-11T01:52:00.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.113+0000 7f987667c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:00.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.113+0000 7f987667c640 1 -- 172.21.15.12:0/3626973067 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9870138350 con 0x7f987006f910 2024-09-11T01:52:00.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.113+0000 7f986d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:00.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.113+0000 7f986d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:00.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.113+0000 7f986e7fc640 4 set_mon_vals no callback set 2024-09-11T01:52:00.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.113+0000 7f986d7fa640 10 monclient: dump: 2024-09-11T01:52:00.130 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:00.131 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:00.131 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.131 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.131 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:00.131 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:00.131 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:00.131 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:00.131 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:00.131 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:00.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.113+0000 7f987667c640 10 monclient: _renew_subs 2024-09-11T01:52:00.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.113+0000 7f987667c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.114+0000 7f987667c640 1 -- 172.21.15.12:0/3626973067 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9870126a80 con 0x7f987006f910 2024-09-11T01:52:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.114+0000 7f986d7fa640 1 -- 172.21.15.12:0/3626973067 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f986003a030 con 0x7f987006f910 2024-09-11T01:52:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.114+0000 7f987667c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.114+0000 7f987667c640 1 --2- 172.21.15.12:0/3626973067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f98701255c0 0x7f98701259a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.114+0000 7f987667c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.114+0000 7f987667c640 1 -- 172.21.15.12:0/3626973067 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f987006a060 con 0x7f98701255c0 2024-09-11T01:52:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.114+0000 7f9867fff640 1 --2- 172.21.15.12:0/3626973067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f98701255c0 0x7f98701259a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.115+0000 7f9867fff640 10 monclient: get_auth_request con 0x7f98701255c0 auth_method 0 2024-09-11T01:52:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.115+0000 7f9867fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.115+0000 7f9867fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.115+0000 7f9867fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.115+0000 7f9867fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.115+0000 7f9867fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.115+0000 7f9867fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.115+0000 7f986d7fa640 1 --2- 172.21.15.12:0/3626973067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f985003e240 0x7f9850040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.115+0000 7f986d7fa640 1 -- 172.21.15.12:0/3626973067 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(764..764 src has 676..764) ==== 4815+0+0 (secure 0 0 0) 0x7f9860073290 con 0x7f987006f910 2024-09-11T01:52:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.115+0000 7f9867fff640 10 monclient(hunting): handle_auth_done global_id 244214 payload 824 2024-09-11T01:52:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.115+0000 7f9867fff640 1 -- 172.21.15.12:0/3626973067 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9870128a10 con 0x7f98701255c0 2024-09-11T01:52:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.115+0000 7f9874c19640 1 --2- 172.21.15.12:0/3626973067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f985003e240 0x7f9850040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.115+0000 7f9874c19640 10 monclient: get_auth_request con 0x7f985003e240 auth_method 0 2024-09-11T01:52:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.117+0000 7f9867fff640 1 -- 172.21.15.12:0/3626973067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f98701255c0 msgr2=0x7f98701259a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.117+0000 7f9867fff640 1 --2- 172.21.15.12:0/3626973067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f98701255c0 0x7f98701259a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9870127520 tx=0x7f9858011d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.117+0000 7f986d7fa640 1 -- 172.21.15.12:0/3626973067 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9858019070 con 0x7f98701255c0 2024-09-11T01:52:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.117+0000 7f986d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.117+0000 7f986d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:00.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.117+0000 7f986d7fa640 10 monclient: dump: 2024-09-11T01:52:00.134 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:00.134 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:00.134 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.134 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.134 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:00.134 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:00.134 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:00.134 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:00.134 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:00.134 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:00.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.117+0000 7f9874c19640 1 --2- 172.21.15.12:0/3626973067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f985003e240 0x7f9850040700 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f9870127390 tx=0x7f9868003d10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:00.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.117+0000 7f986d7fa640 1 -- 172.21.15.12:0/3626973067 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f985800f890 con 0x7f98701255c0 2024-09-11T01:52:00.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.117+0000 7f986d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:00.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.117+0000 7f986d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:00.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.117+0000 7f986d7fa640 1 -- 172.21.15.12:0/3626973067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f98701255c0 msgr2=0x7f98701259a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:00.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.117+0000 7f986d7fa640 1 --2- 172.21.15.12:0/3626973067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f98701255c0 0x7f98701259a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9870127520 tx=0x7f9858011d40 comp rx=0 tx=0).stop 2024-09-11T01:52:00.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.117+0000 7f986d7fa640 1 -- 172.21.15.12:0/3626973067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f98701255c0 msgr2=0x7f98701259a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:00.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.117+0000 7f986d7fa640 1 --2- 172.21.15.12:0/3626973067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f98701255c0 0x7f98701259a0 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9870127520 tx=0x7f9858011d40 comp rx=0 tx=0).stop 2024-09-11T01:52:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.124+0000 7f987667c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:52:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.124+0000 7f987667c640 1 --2- 172.21.15.12:0/3626973067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9870125ee0 0x7f98700a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.124+0000 7f987667c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.124+0000 7f987667c640 1 -- 172.21.15.12:0/3626973067 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f987006ded0 con 0x7f9870125ee0 2024-09-11T01:52:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.124+0000 7f9867fff640 1 --2- 172.21.15.12:0/3626973067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9870125ee0 0x7f98700a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.124+0000 7f9867fff640 10 monclient: get_auth_request con 0x7f9870125ee0 auth_method 0 2024-09-11T01:52:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.124+0000 7f9867fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.124+0000 7f9867fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.124+0000 7f9867fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.124+0000 7f9867fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.124+0000 7f9867fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.124+0000 7f9867fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.124+0000 7f9867fff640 10 monclient(hunting): handle_auth_done global_id 244217 payload 824 2024-09-11T01:52:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.124+0000 7f9867fff640 1 -- 172.21.15.12:0/3626973067 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9870127590 con 0x7f9870125ee0 2024-09-11T01:52:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.126+0000 7f9867fff640 1 -- 172.21.15.12:0/3626973067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9870125ee0 msgr2=0x7f98700a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.126+0000 7f9867fff640 1 --2- 172.21.15.12:0/3626973067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9870125ee0 0x7f98700a0830 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9870127520 tx=0x7f98580046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.126+0000 7f986d7fa640 1 -- 172.21.15.12:0/3626973067 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9858019040 con 0x7f9870125ee0 2024-09-11T01:52:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.126+0000 7f986d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:00.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.126+0000 7f986d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.126+0000 7f986d7fa640 10 monclient: dump: 2024-09-11T01:52:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:00.139 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.127+0000 7f986d7fa640 1 -- 172.21.15.12:0/3626973067 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7f9858003e50 con 0x7f9870125ee0 2024-09-11T01:52:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.127+0000 7f986d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:52:00.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.127+0000 7f986d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:00.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.127+0000 7f986d7fa640 1 -- 172.21.15.12:0/3626973067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9870125ee0 msgr2=0x7f98700a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:00.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.127+0000 7f986d7fa640 1 --2- 172.21.15.12:0/3626973067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9870125ee0 0x7f98700a0830 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9870127520 tx=0x7f98580046c0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.127+0000 7f986d7fa640 1 -- 172.21.15.12:0/3626973067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9870125ee0 msgr2=0x7f98700a0830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:00.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.127+0000 7f986d7fa640 1 --2- 172.21.15.12:0/3626973067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9870125ee0 0x7f98700a0830 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9870127520 tx=0x7f98580046c0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.140 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:52:00.140 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:52:00.140 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:52:00.140 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:52:00.140 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 214, 2024-09-11T01:52:00.140 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 73574, 2024-09-11T01:52:00.141 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:52:00.141 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:52:00.141 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:52:00.141 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:52:00.141 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:00.141 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:52:00.141 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:52:00.141 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:52:00.141 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:52:00.141 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:00.141 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:00.141 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:00.141 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:00.142 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:00.142 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:00.142 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:00.142 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:00.142 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:00.142 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:00.142 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:00.142 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:00.142 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:52:00.142 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:52:00.143 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:00.143 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:00.143 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:00.143 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:00.143 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:00.143 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:00.143 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:00.143 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:00.143 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:00.143 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:00.143 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:00.143 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:00.143 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:00.144 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:52:00.144 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:52:00.144 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:52:00.144 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:52:00.144 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:52:00.144 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:52:00.144 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:52:00.144 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:52:00.144 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:52:00.144 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:52:00.144 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:52:00.144 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:52:00.144 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:52:00.144 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:52:00.145 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:52:00.145 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:52:00.145 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:52:00.145 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:00.145 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:00.145 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:00.145 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:00.145 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:00.145 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:00.145 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:00.145 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:00.145 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:00.145 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:00.145 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:00.146 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:00.146 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:52:00.146 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:00.146 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:52:00.146 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:00.146 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:52:00.146 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:52:00.146 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:00.146 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:00.146 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:00.146 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:00.146 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:52:00.146 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:00.147 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:00.147 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:00.147 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:00.147 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:52:00.147 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:00.147 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:00.147 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:00.147 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:00.147 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:52:00.147 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:52:00.147 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:00.147 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:00.147 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:00.148 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:00.148 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:00.148 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:52:00.148 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:52:00.148 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:00.148 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:00.148 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:00.148 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:00.148 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:52:00.148 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:00.148 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:00.148 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:00.149 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:00.149 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:52:00.149 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:00.149 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:00.149 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:00.149 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:00.149 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:52:00.149 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:52:00.149 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:00.149 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:00.149 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:00.149 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:00.149 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:00.150 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:52:00.150 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:52:00.150 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:00.150 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:00.150 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:00.150 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:00.150 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:52:00.150 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:00.150 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:00.150 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:00.150 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:00.150 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:52:00.150 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:00.150 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:00.151 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:00.151 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:00.151 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:52:00.151 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:52:00.151 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:00.151 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:00.151 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:00.151 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:00.151 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:00.151 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:00.151 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:52:00.151 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:52:00.151 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:00.152 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:00.152 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:00.152 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:52:00.152 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:00.152 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:00.152 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:52:00.152 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:00.152 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:00.152 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:00.152 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:52:00.152 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:00.152 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:00.152 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:52:00.153 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:00.153 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:00.153 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:00.153 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:52:00.153 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:00.153 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:00.153 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:52:00.153 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:00.153 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:00.153 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:00.153 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:52:00.153 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:00.154 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:00.154 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:00.154 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:52:00.154 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:52:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.127+0000 7f987667c640 1 -- 172.21.15.12:0/3626973067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f985003e240 msgr2=0x7f9850040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.127+0000 7f987667c640 1 --2- 172.21.15.12:0/3626973067 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f985003e240 0x7f9850040700 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f9870127390 tx=0x7f9868003d10 comp rx=0 tx=0).stop 2024-09-11T01:52:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.127+0000 7f987667c640 10 monclient: shutdown 2024-09-11T01:52:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.127+0000 7f987667c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.127+0000 7f987667c640 1 -- 172.21.15.12:0/3626973067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f987006f910 msgr2=0x7f98701cd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.127+0000 7f987667c640 1 --2- 172.21.15.12:0/3626973067 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f987006f910 0x7f98701cd470 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f98600090b0 tx=0x7f98600046c0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.127+0000 7f9874c19640 1 -- 172.21.15.12:0/3626973067 reap_dead start 2024-09-11T01:52:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.127+0000 7f9874c19640 1 -- 172.21.15.12:0/3626973067 reap_dead start 2024-09-11T01:52:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.127+0000 7f987667c640 1 -- 172.21.15.12:0/3626973067 shutdown_connections 2024-09-11T01:52:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.127+0000 7f987667c640 1 -- 172.21.15.12:0/3626973067 >> 172.21.15.12:0/3626973067 conn(0x7f98700a8600 msgr2=0x7f98700a71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.127+0000 7f987667c640 1 -- 172.21.15.12:0/3626973067 shutdown_connections 2024-09-11T01:52:00.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.127+0000 7f987667c640 1 -- 172.21.15.12:0/3626973067 wait complete. 2024-09-11T01:52:00.156 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:52:00.157 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:52:00.157 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:52:00.192 INFO:tasks.rados.rados.0.smithi071.stdout:3401: done (1 left) 2024-09-11T01:52:00.192 INFO:tasks.rados.rados.0.smithi071.stdout:3403: done (0 left) 2024-09-11T01:52:00.192 INFO:tasks.rados.rados.0.smithi071.stdout:3404: write (excl) oid 24 current snap is 379 2024-09-11T01:52:00.192 INFO:tasks.rados.rados.0.smithi071.stdout:3404: seq_num 1190 ranges {400330=497378,1485925=514405} 2024-09-11T01:52:00.197 INFO:tasks.rados.rados.0.smithi071.stdout:3404: writing smithi07134777-24 from 400330 to 897708 tid 1 2024-09-11T01:52:00.203 INFO:tasks.rados.rados.0.smithi071.stdout:3404: writing smithi07134777-24 from 1485925 to 2000330 tid 2 2024-09-11T01:52:00.203 INFO:tasks.rados.rados.0.smithi071.stdout:3405: snap_remove snap 344 2024-09-11T01:52:00.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.259+0000 7f2ae2efe640 10 monclient: get_monmap_and_config 2024-09-11T01:52:00.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.259+0000 7f2ae2efe640 10 monclient: build_initial_monmap 2024-09-11T01:52:00.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.259+0000 7f2ae2efe640 10 monclient: monmap: 2024-09-11T01:52:00.261 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:00.261 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:00.261 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:00.260343+0000 2024-09-11T01:52:00.261 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:00.260343+0000 2024-09-11T01:52:00.262 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:00.262 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:00.262 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:00.262 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:00.262 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:00.262 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:00.262 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:00.262 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:00.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.260+0000 7f2ae2efe640 1 Processor -- start 2024-09-11T01:52:00.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.260+0000 7f2ae2efe640 1 -- start start 2024-09-11T01:52:00.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.260+0000 7f2ae2efe640 10 monclient: init 2024-09-11T01:52:00.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.260+0000 7f2ae2efe640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:00.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.260+0000 7f2ae2efe640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-11T01:52:00.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.260+0000 7f2ae2efe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2adc131710 0x7f2adc131af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:00.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.260+0000 7f2ae2efe640 10 monclient(hunting): picked mon.noname-c con 0x7f2adc131710 addr v2:172.21.15.71:3300/0 2024-09-11T01:52:00.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.260+0000 7f2ae2efe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2adc132030 0x7f2adc0a25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:00.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.260+0000 7f2ae2efe640 10 monclient(hunting): picked mon.noname-b con 0x7f2adc132030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:00.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.260+0000 7f2ae2efe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2adc0a0d70 0x7f2adc0a2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:00.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.260+0000 7f2ae2efe640 10 monclient(hunting): picked mon.noname-a con 0x7f2adc0a0d70 addr v2:172.21.15.12:3300/0 2024-09-11T01:52:00.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.260+0000 7f2ae2efe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:00.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.260+0000 7f2ae2efe640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f2adc074a70 con 0x7f2adc0a0d70 2024-09-11T01:52:00.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.260+0000 7f2ae2efe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:00.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.260+0000 7f2ae2efe640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f2adc073ab0 con 0x7f2adc131710 2024-09-11T01:52:00.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.260+0000 7f2ae2efe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:00.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.260+0000 7f2ae2efe640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2adc073d90 con 0x7f2adc132030 2024-09-11T01:52:00.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.260+0000 7f2ae2efe640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:00.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.260+0000 7f2ae2efe640 10 monclient(hunting): authenticate will time out at 2126.702971s 2024-09-11T01:52:00.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae149b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2adc0a0d70 0x7f2adc0a2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:00.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2adbfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2adc132030 0x7f2adc0a25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:00.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae149b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2adc0a0d70 0x7f2adc0a2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37968/0 (socket says 172.21.15.12:37968) 2024-09-11T01:52:00.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae149b640 1 -- 172.21.15.12:0/1233025056 learned_addr learned my addr 172.21.15.12:0/1233025056 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:00.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2adbfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2adc132030 0x7f2adc0a25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55646/0 (socket says 172.21.15.12:55646) 2024-09-11T01:52:00.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae149b640 10 monclient(hunting): get_auth_request con 0x7f2adc0a0d70 auth_method 0 2024-09-11T01:52:00.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae149b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:00.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae149b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:00.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae149b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:00.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2adbfff640 10 monclient(hunting): get_auth_request con 0x7f2adc132030 auth_method 0 2024-09-11T01:52:00.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2adbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:00.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2adbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:00.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2adbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:00.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae149b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:00.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae149b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:00.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae0c9a640 1 --2- 172.21.15.12:0/1233025056 >> v2:172.21.15.71:3300/0 conn(0x7f2adc131710 0x7f2adc131af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:00.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae149b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:00.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2adbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:00.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2adbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:00.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2adbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:00.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae0c9a640 10 monclient(hunting): get_auth_request con 0x7f2adc131710 auth_method 0 2024-09-11T01:52:00.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae0c9a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:00.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae0c9a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:00.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae0c9a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:00.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae0c9a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:00.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae0c9a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:00.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae0c9a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:00.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae149b640 10 monclient(hunting): handle_auth_done global_id 254124 payload 274 2024-09-11T01:52:00.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae149b640 1 -- 172.21.15.12:0/1233025056 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2adc132030 msgr2=0x7f2adc0a25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:00.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae149b640 1 --2- 172.21.15.12:0/1233025056 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2adc132030 0x7f2adc0a25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae149b640 1 -- 172.21.15.12:0/1233025056 >> v2:172.21.15.71:3300/0 conn(0x7f2adc131710 msgr2=0x7f2adc131af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:00.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae149b640 1 --2- 172.21.15.12:0/1233025056 >> v2:172.21.15.71:3300/0 conn(0x7f2adc131710 0x7f2adc131af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae149b640 10 monclient: _finish_hunting 0 2024-09-11T01:52:00.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae149b640 1 monclient: found mon.noname-a 2024-09-11T01:52:00.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae149b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:00.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae149b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:52:00.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2ae149b640 1 -- 172.21.15.12:0/1233025056 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2adc074070 con 0x7f2adc0a0d70 2024-09-11T01:52:00.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.261+0000 7f2adbfff640 1 --2- 172.21.15.12:0/1233025056 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2adc132030 0x7f2adc0a25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:52:00.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2ae149b640 1 -- 172.21.15.12:0/1233025056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2adc0a0d70 msgr2=0x7f2adc0a2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:00.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2ae149b640 1 --2- 172.21.15.12:0/1233025056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2adc0a0d70 0x7f2adc0a2b10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2adc06ef60 tx=0x7f2ad402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=362828ea348c8e23 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:00.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2adb7fe640 1 -- 172.21.15.12:0/1233025056 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad4036070 con 0x7f2adc0a0d70 2024-09-11T01:52:00.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2adb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:00.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2adb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:52:00.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2adb7fe640 10 monclient: dump: 2024-09-11T01:52:00.277 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:00.277 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:00.277 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.277 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.277 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:00.278 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:00.278 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:00.278 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:00.278 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:00.278 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:00.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2adb7fe640 10 monclient: _finish_auth 0 2024-09-11T01:52:00.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2adb7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:52:00.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2adb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:00.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2adb7fe640 1 -- 172.21.15.12:0/1233025056 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ad4007800 con 0x7f2adc0a0d70 2024-09-11T01:52:00.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2ae2efe640 5 monclient: authenticate success, global_id 254124 2024-09-11T01:52:00.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2ae2efe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:52:00.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2adb7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:00.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2ae2efe640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:00.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2adb7fe640 1 -- 172.21.15.12:0/1233025056 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad4007bd0 con 0x7f2adc0a0d70 2024-09-11T01:52:00.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2adb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:00.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2ae2efe640 4 set_mon_vals no callback set 2024-09-11T01:52:00.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2adb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:52:00.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2adb7fe640 10 monclient: dump: 2024-09-11T01:52:00.279 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:00.279 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:00.279 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.279 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.279 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:00.279 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:00.279 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:00.279 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:00.279 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:00.280 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:00.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2ae2efe640 10 monclient: shutdown 2024-09-11T01:52:00.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2ae2efe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:00.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2ae2efe640 1 -- 172.21.15.12:0/1233025056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2adc0a0d70 msgr2=0x7f2adc0a2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:00.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2ae2efe640 1 --2- 172.21.15.12:0/1233025056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2adc0a0d70 0x7f2adc0a2b10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2adc06ef60 tx=0x7f2ad402a610 comp rx=0 tx=0).stop 2024-09-11T01:52:00.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2ae2efe640 1 -- 172.21.15.12:0/1233025056 shutdown_connections 2024-09-11T01:52:00.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2ae2efe640 1 --2- 172.21.15.12:0/1233025056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2adc0a0d70 0x7f2adc0a2b10 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2ae2efe640 1 --2- 172.21.15.12:0/1233025056 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2adc132030 0x7f2adc0a25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2ae2efe640 1 --2- 172.21.15.12:0/1233025056 >> v2:172.21.15.71:3300/0 conn(0x7f2adc131710 0x7f2adc131af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2ae2efe640 1 -- 172.21.15.12:0/1233025056 >> 172.21.15.12:0/1233025056 conn(0x7f2adc0a36c0 msgr2=0x7f2adc0a3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:00.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2ae2efe640 1 -- 172.21.15.12:0/1233025056 shutdown_connections 2024-09-11T01:52:00.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.264+0000 7f2ae2efe640 1 -- 172.21.15.12:0/1233025056 wait complete. 2024-09-11T01:52:00.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.265+0000 7f2ae2efe640 10 monclient: build_initial_monmap 2024-09-11T01:52:00.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.265+0000 7f2ae2efe640 10 monclient: monmap: 2024-09-11T01:52:00.281 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:00.281 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:00.281 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:00.281 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:00.281 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:00.281 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:00.281 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:00.281 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:00.281 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:00.281 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:00.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.265+0000 7f2ae2efe640 1 Processor -- start 2024-09-11T01:52:00.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.265+0000 7f2ae2efe640 1 -- start start 2024-09-11T01:52:00.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.265+0000 7f2ae2efe640 10 monclient: init 2024-09-11T01:52:00.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae2efe640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:00.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae2efe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:52:00.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae2efe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2adc0a0d70 0x7f2adc1cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:00.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae2efe640 10 monclient(hunting): picked mon.noname-c con 0x7f2adc0a0d70 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:00.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae2efe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2adc131710 0x7f2adc1308f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:00.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae2efe640 10 monclient(hunting): picked mon.noname-a con 0x7f2adc131710 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:00.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae2efe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2adc132030 0x7f2adc130e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:00.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae2efe640 10 monclient(hunting): picked mon.noname-b con 0x7f2adc132030 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:00.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae2efe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:00.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae2efe640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc073350 con 0x7f2adc131710 2024-09-11T01:52:00.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae2efe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:00.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae2efe640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc0731d0 con 0x7f2adc132030 2024-09-11T01:52:00.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae2efe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:00.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae2efe640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2adc0a3050 con 0x7f2adc0a0d70 2024-09-11T01:52:00.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae2efe640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:00.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae2efe640 10 monclient(hunting): authenticate will time out at 2126.708535s 2024-09-11T01:52:00.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae0c9a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2adc0a0d70 0x7f2adc1cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:00.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2adbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2adc131710 0x7f2adc1308f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:00.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae0c9a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2adc0a0d70 0x7f2adc1cace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55656/0 (socket says 172.21.15.12:55656) 2024-09-11T01:52:00.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae0c9a640 1 -- 172.21.15.12:0/3325382359 learned_addr learned my addr 172.21.15.12:0/3325382359 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:00.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae0c9a640 10 monclient(hunting): get_auth_request con 0x7f2adc0a0d70 auth_method 0 2024-09-11T01:52:00.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae0c9a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:00.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae0c9a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:00.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae0c9a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:00.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2adbfff640 1 --2- 172.21.15.12:0/3325382359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2adc131710 0x7f2adc1308f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37982/0 (socket says 172.21.15.12:37982) 2024-09-11T01:52:00.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2adbfff640 10 monclient(hunting): get_auth_request con 0x7f2adc131710 auth_method 0 2024-09-11T01:52:00.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2adbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:00.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2adbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:00.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2adbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:00.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae0c9a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:00.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae0c9a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:00.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae0c9a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:00.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2adbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:00.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2adbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:00.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2adbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:00.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae149b640 1 --2- 172.21.15.12:0/3325382359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2adc132030 0x7f2adc130e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:00.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae149b640 10 monclient(hunting): get_auth_request con 0x7f2adc132030 auth_method 0 2024-09-11T01:52:00.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae149b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:00.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae149b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae149b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae0c9a640 10 monclient(hunting): handle_auth_done global_id 244223 payload 824 2024-09-11T01:52:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae0c9a640 1 -- 172.21.15.12:0/3325382359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2adc132030 msgr2=0x7f2adc130e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae0c9a640 1 --2- 172.21.15.12:0/3325382359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2adc132030 0x7f2adc130e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae0c9a640 1 -- 172.21.15.12:0/3325382359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2adc131710 msgr2=0x7f2adc1308f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae0c9a640 1 --2- 172.21.15.12:0/3325382359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2adc131710 0x7f2adc1308f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae0c9a640 10 monclient: _finish_hunting 0 2024-09-11T01:52:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae0c9a640 1 monclient: found mon.noname-c 2024-09-11T01:52:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae0c9a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae0c9a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:52:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2ae0c9a640 1 -- 172.21.15.12:0/3325382359 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2adc11f3c0 con 0x7f2adc0a0d70 2024-09-11T01:52:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.266+0000 7f2adbfff640 1 --2- 172.21.15.12:0/3325382359 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2adc131710 0x7f2adc1308f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:52:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.269+0000 7f2ae0c9a640 1 -- 172.21.15.12:0/3325382359 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2adc0a0d70 msgr2=0x7f2adc1cace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:00.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.269+0000 7f2ae0c9a640 1 --2- 172.21.15.12:0/3325382359 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2adc0a0d70 0x7f2adc1cace0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000e890 tx=0x7f2ad000ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:00.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.269+0000 7f2ad97fa640 1 -- 172.21.15.12:0/3325382359 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad0017070 con 0x7f2adc0a0d70 2024-09-11T01:52:00.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.269+0000 7f2ad97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:00.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.269+0000 7f2ad97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:00.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.269+0000 7f2ad97fa640 10 monclient: dump: 2024-09-11T01:52:00.286 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:00.286 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:00.286 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.286 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.286 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:00.286 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:00.286 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:00.286 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:00.286 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:00.286 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.269+0000 7f2ad97fa640 10 monclient: _finish_auth 0 2024-09-11T01:52:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.269+0000 7f2ad97fa640 10 monclient: _check_auth_tickets 2024-09-11T01:52:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.269+0000 7f2ad97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.269+0000 7f2ad97fa640 1 -- 172.21.15.12:0/3325382359 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ad000cb50 con 0x7f2adc0a0d70 2024-09-11T01:52:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.269+0000 7f2ae2efe640 5 monclient: authenticate success, global_id 244223 2024-09-11T01:52:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.269+0000 7f2ae2efe640 10 monclient: _renew_subs 2024-09-11T01:52:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.269+0000 7f2ae2efe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.269+0000 7f2ae2efe640 1 -- 172.21.15.12:0/3325382359 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2adc075000 con 0x7f2adc0a0d70 2024-09-11T01:52:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.269+0000 7f2ad97fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.269+0000 7f2ad97fa640 1 -- 172.21.15.12:0/3325382359 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad0013470 con 0x7f2adc0a0d70 2024-09-11T01:52:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.269+0000 7f2ad97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.269+0000 7f2ad97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:00.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.269+0000 7f2ad97fa640 10 monclient: dump: 2024-09-11T01:52:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:00.288 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:00.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.269+0000 7f2ae2efe640 10 monclient: _renew_subs 2024-09-11T01:52:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.269+0000 7f2ae2efe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.269+0000 7f2ae2efe640 1 -- 172.21.15.12:0/3325382359 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2adc11f0e0 con 0x7f2adc0a0d70 2024-09-11T01:52:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.269+0000 7f2ada7fc640 4 set_mon_vals no callback set 2024-09-11T01:52:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.270+0000 7f2ae2efe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.270+0000 7f2ae2efe640 1 --2- 172.21.15.12:0/3325382359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2adc1c77a0 0x7f2adc1c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.270+0000 7f2ae2efe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.270+0000 7f2ae2efe640 1 -- 172.21.15.12:0/3325382359 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc1bde50 con 0x7f2adc1c77a0 2024-09-11T01:52:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.270+0000 7f2ae0c9a640 1 --2- 172.21.15.12:0/3325382359 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2adc0a0d70 0x7f2adc1cace0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000e890 tx=0x7f2ad000ed50 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7f2ad0013610 mgrmap(e 5) 2024-09-11T01:52:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.270+0000 7f2ad97fa640 1 -- 172.21.15.12:0/3325382359 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f2ad0013610 con 0x7f2adc0a0d70 2024-09-11T01:52:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.270+0000 7f2adbfff640 1 --2- 172.21.15.12:0/3325382359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2adc1c77a0 0x7f2adc1c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.270+0000 7f2adbfff640 10 monclient: get_auth_request con 0x7f2adc1c77a0 auth_method 0 2024-09-11T01:52:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.270+0000 7f2adbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.270+0000 7f2adbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:00.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.270+0000 7f2adbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.270+0000 7f2adbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.270+0000 7f2adbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.270+0000 7f2adbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.270+0000 7f2ad97fa640 1 --2- 172.21.15.12:0/3325382359 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2ac003e260 0x7f2ac0040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.270+0000 7f2adbfff640 10 monclient(hunting): handle_auth_done global_id 234244 payload 824 2024-09-11T01:52:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.270+0000 7f2adbfff640 1 -- 172.21.15.12:0/3325382359 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2adc0707c0 con 0x7f2adc1c77a0 2024-09-11T01:52:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.270+0000 7f2ae149b640 1 --2- 172.21.15.12:0/3325382359 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2ac003e260 0x7f2ac0040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.270+0000 7f2ae149b640 10 monclient: get_auth_request con 0x7f2ac003e260 auth_method 0 2024-09-11T01:52:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.273+0000 7f2adbfff640 1 -- 172.21.15.12:0/3325382359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2adc1c77a0 msgr2=0x7f2adc1c7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.273+0000 7f2adbfff640 1 --2- 172.21.15.12:0/3325382359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2adc1c77a0 0x7f2adc1c7b80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2adc132410 tx=0x7f2ac8015510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.273+0000 7f2ad97fa640 1 -- 172.21.15.12:0/3325382359 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac8020070 con 0x7f2adc1c77a0 2024-09-11T01:52:00.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.273+0000 7f2ad97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.273+0000 7f2ad97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:52:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.273+0000 7f2ad97fa640 10 monclient: dump: 2024-09-11T01:52:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:00.291 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.273+0000 7f2ae149b640 1 --2- 172.21.15.12:0/3325382359 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2ac003e260 0x7f2ac0040720 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4030000 tx=0x7f2ad4009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.273+0000 7f2ad97fa640 1 -- 172.21.15.12:0/3325382359 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2ac8004440 con 0x7f2adc1c77a0 2024-09-11T01:52:00.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.273+0000 7f2ad97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:00.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.273+0000 7f2ad97fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:00.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.273+0000 7f2ad97fa640 1 -- 172.21.15.12:0/3325382359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2adc1c77a0 msgr2=0x7f2adc1c7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:00.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.273+0000 7f2ad97fa640 1 --2- 172.21.15.12:0/3325382359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2adc1c77a0 0x7f2adc1c7b80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2adc132410 tx=0x7f2ac8015510 comp rx=0 tx=0).stop 2024-09-11T01:52:00.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.273+0000 7f2ad97fa640 1 -- 172.21.15.12:0/3325382359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2adc1c77a0 msgr2=0x7f2adc1c7b80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:00.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.273+0000 7f2ad97fa640 1 --2- 172.21.15.12:0/3325382359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2adc1c77a0 0x7f2adc1c7b80 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2adc132410 tx=0x7f2ac8015510 comp rx=0 tx=0).stop 2024-09-11T01:52:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.279+0000 7f2ae2efe640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:52:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.279+0000 7f2ae2efe640 1 --2- 172.21.15.12:0/3325382359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2adc1c80c0 0x7f2adc1b8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.279+0000 7f2ae2efe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.279+0000 7f2ae2efe640 1 -- 172.21.15.12:0/3325382359 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc11deb0 con 0x7f2adc1c80c0 2024-09-11T01:52:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.280+0000 7f2adbfff640 1 --2- 172.21.15.12:0/3325382359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2adc1c80c0 0x7f2adc1b8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.280+0000 7f2adbfff640 10 monclient: get_auth_request con 0x7f2adc1c80c0 auth_method 0 2024-09-11T01:52:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.280+0000 7f2adbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.280+0000 7f2adbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.280+0000 7f2adbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.280+0000 7f2adbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.280+0000 7f2adbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:00.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.280+0000 7f2adbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.280+0000 7f2adbfff640 10 monclient(hunting): handle_auth_done global_id 234247 payload 824 2024-09-11T01:52:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.280+0000 7f2adbfff640 1 -- 172.21.15.12:0/3325382359 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2adc11dbc0 con 0x7f2adc1c80c0 2024-09-11T01:52:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.282+0000 7f2adbfff640 1 -- 172.21.15.12:0/3325382359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2adc1c80c0 msgr2=0x7f2adc1b8520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.282+0000 7f2adbfff640 1 --2- 172.21.15.12:0/3325382359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2adc1c80c0 0x7f2adc1b8520 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2adc132410 tx=0x7f2ac80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.283+0000 7f2ad97fa640 1 -- 172.21.15.12:0/3325382359 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac8020040 con 0x7f2adc1c80c0 2024-09-11T01:52:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.283+0000 7f2ad97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.283+0000 7f2ad97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:52:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.283+0000 7f2ad97fa640 10 monclient: dump: 2024-09-11T01:52:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.296 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:00.297 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.283+0000 7f2ad97fa640 1 -- 172.21.15.12:0/3325382359 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7f2ac8003810 con 0x7f2adc1c80c0 2024-09-11T01:52:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.283+0000 7f2ad97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:52:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.283+0000 7f2ad97fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.283+0000 7f2ad97fa640 1 -- 172.21.15.12:0/3325382359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2adc1c80c0 msgr2=0x7f2adc1b8520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.283+0000 7f2ad97fa640 1 --2- 172.21.15.12:0/3325382359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2adc1c80c0 0x7f2adc1b8520 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2adc132410 tx=0x7f2ac80046c0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.283+0000 7f2ad97fa640 1 -- 172.21.15.12:0/3325382359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2adc1c80c0 msgr2=0x7f2adc1b8520 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.283+0000 7f2ad97fa640 1 --2- 172.21.15.12:0/3325382359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2adc1c80c0 0x7f2adc1b8520 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2adc132410 tx=0x7f2ac80046c0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.298 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:52:00.298 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:52:00.298 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:52:00.298 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:52:00.298 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 214, 2024-09-11T01:52:00.298 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 122175, 2024-09-11T01:52:00.298 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:52:00.298 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:52:00.298 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:52:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:52:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:52:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:52:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:52:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:52:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:00.299 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:00.300 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:00.300 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:00.300 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:00.300 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:00.300 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:52:00.300 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:52:00.300 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:00.300 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:00.300 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:00.300 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:00.300 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:00.300 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:00.300 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:00.300 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:52:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:52:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:52:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:52:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:52:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:52:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:52:00.301 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:52:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:52:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:52:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:52:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:52:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:52:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:52:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:52:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:52:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:52:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:00.302 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:52:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:52:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:52:00.303 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:52:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:52:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:52:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:00.304 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:00.305 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:00.305 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:52:00.305 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:52:00.305 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:00.305 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:00.305 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:00.305 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:00.305 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:00.305 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:52:00.306 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:52:00.306 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:00.306 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:00.306 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:00.306 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:00.306 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:52:00.306 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:00.306 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:00.306 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:00.306 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:00.306 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:52:00.306 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:00.306 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:00.307 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:00.307 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:00.307 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:52:00.307 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:52:00.307 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:00.307 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:00.307 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:00.307 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:00.307 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:00.307 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:52:00.307 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:52:00.307 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:00.308 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:00.308 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:00.308 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:00.308 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:52:00.308 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:00.308 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:00.308 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:00.308 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:00.308 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:52:00.308 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:00.308 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:52:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:52:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:00.309 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:52:00.310 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:52:00.310 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:00.310 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:00.310 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:00.310 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:52:00.310 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:00.310 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:00.310 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:52:00.310 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:00.310 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:00.310 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:00.310 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:52:00.311 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:00.311 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:00.311 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:52:00.311 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:00.311 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:00.311 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:00.311 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:52:00.311 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:00.311 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:00.311 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:00.311 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:52:00.311 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:52:00.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.283+0000 7f2ae2efe640 1 -- 172.21.15.12:0/3325382359 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2ac003e260 msgr2=0x7f2ac0040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:00.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.283+0000 7f2ae2efe640 1 --2- 172.21.15.12:0/3325382359 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2ac003e260 0x7f2ac0040720 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4030000 tx=0x7f2ad4009000 comp rx=0 tx=0).stop 2024-09-11T01:52:00.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.283+0000 7f2ae2efe640 10 monclient: shutdown 2024-09-11T01:52:00.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.283+0000 7f2ae2efe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:00.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.283+0000 7f2ae2efe640 1 -- 172.21.15.12:0/3325382359 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2adc0a0d70 msgr2=0x7f2adc1cace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:00.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.283+0000 7f2ae2efe640 1 --2- 172.21.15.12:0/3325382359 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2adc0a0d70 0x7f2adc1cace0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000e890 tx=0x7f2ad000ed50 comp rx=0 tx=0).stop 2024-09-11T01:52:00.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.283+0000 7f2ae149b640 1 -- 172.21.15.12:0/3325382359 reap_dead start 2024-09-11T01:52:00.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.283+0000 7f2ad97fa640 1 -- 172.21.15.12:0/3325382359 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(764..764 src has 676..764) ==== 4815+0+0 (unknown 0 0 0) 0x7f2ad0053460 con 0x7f2adc0a0d70 2024-09-11T01:52:00.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.283+0000 7f2ae149b640 1 -- 172.21.15.12:0/3325382359 reap_dead start 2024-09-11T01:52:00.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.283+0000 7f2ae2efe640 1 -- 172.21.15.12:0/3325382359 shutdown_connections 2024-09-11T01:52:00.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.283+0000 7f2ae2efe640 1 -- 172.21.15.12:0/3325382359 >> 172.21.15.12:0/3325382359 conn(0x7f2adc0a36c0 msgr2=0x7f2adc09dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:00.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.283+0000 7f2ae2efe640 1 -- 172.21.15.12:0/3325382359 shutdown_connections 2024-09-11T01:52:00.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.283+0000 7f2ae2efe640 1 -- 172.21.15.12:0/3325382359 wait complete. 2024-09-11T01:52:00.314 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:52:00.314 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:52:00.315 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:52:00.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.414+0000 7f6e520ef640 10 monclient: get_monmap_and_config 2024-09-11T01:52:00.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.414+0000 7f6e520ef640 10 monclient: build_initial_monmap 2024-09-11T01:52:00.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.414+0000 7f6e520ef640 10 monclient: monmap: 2024-09-11T01:52:00.416 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:00.416 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:00.416 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:00.414918+0000 2024-09-11T01:52:00.416 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:00.414918+0000 2024-09-11T01:52:00.416 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:00.416 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:00.416 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:00.416 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:00.416 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:00.417 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:00.417 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:00.417 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:00.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.414+0000 7f6e520ef640 1 Processor -- start 2024-09-11T01:52:00.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.414+0000 7f6e520ef640 1 -- start start 2024-09-11T01:52:00.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.414+0000 7f6e520ef640 10 monclient: init 2024-09-11T01:52:00.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e520ef640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:00.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e520ef640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-11T01:52:00.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e520ef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6e4c137d50 addr v1:172.21.15.12:6789/0 2024-09-11T01:52:00.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e520ef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6e4c138130 addr v1:172.21.15.71:6789/0 2024-09-11T01:52:00.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e520ef640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6e4c13e290 0x7f6e4c140680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:00.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e520ef640 10 monclient(hunting): picked mon.noname-b con 0x7f6e4c13e290 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:00.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e520ef640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e4c09d820 con 0x7f6e4c137d50 2024-09-11T01:52:00.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e520ef640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e4c09ba20 con 0x7f6e4c138130 2024-09-11T01:52:00.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e520ef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:00.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e520ef640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6e4c09c650 con 0x7f6e4c13e290 2024-09-11T01:52:00.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e520ef640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:00.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e520ef640 10 monclient(hunting): authenticate will time out at 2126.857504s 2024-09-11T01:52:00.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e4bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6e4c13e290 0x7f6e4c140680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:00.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e4b7fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f6e4c137d50 0x7f6e4c09d110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37082/0 (socket says 172.21.15.12:37082) 2024-09-11T01:52:00.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e4b7fe640 1 -- 172.21.15.12:0/1259867991 learned_addr learned my addr 172.21.15.12:0/1259867991 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:52:00.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e4bfff640 10 monclient(hunting): get_auth_request con 0x7f6e4c13e290 auth_method 0 2024-09-11T01:52:00.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:00.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:00.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:00.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:00.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:00.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:00.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e4bfff640 10 monclient(hunting): handle_auth_done global_id 244226 payload 274 2024-09-11T01:52:00.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e4bfff640 1 -- 172.21.15.12:0/1259867991 >> v1:172.21.15.71:6789/0 conn(0x7f6e4c138130 legacy=0x7f6e4c138580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:00.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e4bfff640 1 -- 172.21.15.12:0/1259867991 >> v1:172.21.15.12:6789/0 conn(0x7f6e4c137d50 legacy=0x7f6e4c09d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:00.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e4bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:52:00.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e4bfff640 1 monclient: found mon.noname-b 2024-09-11T01:52:00.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:00.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e4bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:52:00.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.415+0000 7f6e4bfff640 1 -- 172.21.15.12:0/1259867991 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e4c09c370 con 0x7f6e4c13e290 2024-09-11T01:52:00.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.418+0000 7f6e4bfff640 1 -- 172.21.15.12:0/1259867991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6e4c13e290 msgr2=0x7f6e4c140680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:00.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.418+0000 7f6e4bfff640 1 --2- 172.21.15.12:0/1259867991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6e4c13e290 0x7f6e4c140680 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6e44009dc0 tx=0x7f6e4402a510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b22f0a2f93ebeab server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:00.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.419+0000 7f6e4a7fc640 1 -- 172.21.15.12:0/1259867991 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e44007a30 con 0x7f6e4c13e290 2024-09-11T01:52:00.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.419+0000 7f6e4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:00.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.419+0000 7f6e4a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:52:00.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.419+0000 7f6e4a7fc640 10 monclient: dump: 2024-09-11T01:52:00.432 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:00.432 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:00.432 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.432 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.432 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:00.433 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.419+0000 7f6e4a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:52:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.419+0000 7f6e4a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:52:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.419+0000 7f6e4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.419+0000 7f6e4a7fc640 1 -- 172.21.15.12:0/1259867991 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e44007bd0 con 0x7f6e4c13e290 2024-09-11T01:52:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.419+0000 7f6e4a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:00.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.419+0000 7f6e4a7fc640 1 -- 172.21.15.12:0/1259867991 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e44031400 con 0x7f6e4c13e290 2024-09-11T01:52:00.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.419+0000 7f6e520ef640 5 monclient: authenticate success, global_id 244226 2024-09-11T01:52:00.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.419+0000 7f6e520ef640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:00.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.419+0000 7f6e520ef640 4 set_mon_vals no callback set 2024-09-11T01:52:00.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.419+0000 7f6e4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:00.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.419+0000 7f6e4a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:00.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.419+0000 7f6e4a7fc640 10 monclient: dump: 2024-09-11T01:52:00.434 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:00.434 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:00.434 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.434 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.434 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:00.434 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:00.435 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:00.435 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:00.435 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:00.435 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:00.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.419+0000 7f6e520ef640 10 monclient: shutdown 2024-09-11T01:52:00.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.419+0000 7f6e520ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:00.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.419+0000 7f6e520ef640 1 -- 172.21.15.12:0/1259867991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6e4c13e290 msgr2=0x7f6e4c140680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:00.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.419+0000 7f6e520ef640 1 --2- 172.21.15.12:0/1259867991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6e4c13e290 0x7f6e4c140680 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6e44009dc0 tx=0x7f6e4402a510 comp rx=0 tx=0).stop 2024-09-11T01:52:00.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.419+0000 7f6e520ef640 1 -- 172.21.15.12:0/1259867991 shutdown_connections 2024-09-11T01:52:00.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.419+0000 7f6e520ef640 1 --2- 172.21.15.12:0/1259867991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6e4c13e290 0x7f6e4c140680 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.419+0000 7f6e520ef640 1 -- 172.21.15.12:0/1259867991 >> 172.21.15.12:0/1259867991 conn(0x7f6e4c0a7020 msgr2=0x7f6e4c0a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:00.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.419+0000 7f6e520ef640 1 -- 172.21.15.12:0/1259867991 shutdown_connections 2024-09-11T01:52:00.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.419+0000 7f6e520ef640 1 -- 172.21.15.12:0/1259867991 wait complete. 2024-09-11T01:52:00.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.420+0000 7f6e520ef640 10 monclient: build_initial_monmap 2024-09-11T01:52:00.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.420+0000 7f6e520ef640 10 monclient: monmap: 2024-09-11T01:52:00.436 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:00.436 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:00.436 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:00.436 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:00.436 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:00.436 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:00.436 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:00.437 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:00.437 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:00.437 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:00.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.420+0000 7f6e520ef640 1 Processor -- start 2024-09-11T01:52:00.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.420+0000 7f6e520ef640 1 -- start start 2024-09-11T01:52:00.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.420+0000 7f6e520ef640 10 monclient: init 2024-09-11T01:52:00.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e520ef640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:00.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e520ef640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:52:00.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e520ef640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e4c137d50 0x7f6e4c13c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:00.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e520ef640 10 monclient(hunting): picked mon.noname-b con 0x7f6e4c137d50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:00.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e520ef640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6e4c138130 0x7f6e4c13a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:00.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e520ef640 10 monclient(hunting): picked mon.noname-c con 0x7f6e4c138130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:00.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e520ef640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e4c13e290 0x7f6e4c136db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:00.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e520ef640 10 monclient(hunting): picked mon.noname-a con 0x7f6e4c13e290 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:00.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e520ef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:00.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e520ef640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6e4c140bc0 con 0x7f6e4c13e290 2024-09-11T01:52:00.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e520ef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:00.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e520ef640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6e4c0a6da0 con 0x7f6e4c137d50 2024-09-11T01:52:00.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e520ef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:00.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e520ef640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f6e4c140d40 con 0x7f6e4c138130 2024-09-11T01:52:00.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e520ef640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:00.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e520ef640 10 monclient(hunting): authenticate will time out at 2126.863403s 2024-09-11T01:52:00.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4affd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6e4c138130 0x7f6e4c13a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:00.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4affd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6e4c138130 0x7f6e4c13a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55682/0 (socket says 172.21.15.12:55682) 2024-09-11T01:52:00.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4affd640 1 -- 172.21.15.12:0/3665133977 learned_addr learned my addr 172.21.15.12:0/3665133977 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:00.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4affd640 10 monclient(hunting): get_auth_request con 0x7f6e4c138130 auth_method 0 2024-09-11T01:52:00.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:00.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:00.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:00.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4bfff640 1 --2- 172.21.15.12:0/3665133977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e4c13e290 0x7f6e4c136db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:00.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4bfff640 10 monclient(hunting): get_auth_request con 0x7f6e4c13e290 auth_method 0 2024-09-11T01:52:00.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:00.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:00.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:00.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:00.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:00.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:00.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4b7fe640 1 --2- 172.21.15.12:0/3665133977 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e4c137d50 0x7f6e4c13c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:00.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:00.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:00.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:00.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4b7fe640 10 monclient(hunting): get_auth_request con 0x7f6e4c137d50 auth_method 0 2024-09-11T01:52:00.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:00.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:00.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:00.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4affd640 10 monclient(hunting): handle_auth_done global_id 244229 payload 824 2024-09-11T01:52:00.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4affd640 1 -- 172.21.15.12:0/3665133977 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e4c137d50 msgr2=0x7f6e4c13c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:00.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4affd640 1 --2- 172.21.15.12:0/3665133977 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e4c137d50 0x7f6e4c13c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4affd640 1 -- 172.21.15.12:0/3665133977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e4c13e290 msgr2=0x7f6e4c136db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:00.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4affd640 1 --2- 172.21.15.12:0/3665133977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e4c13e290 0x7f6e4c136db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4affd640 10 monclient: _finish_hunting 0 2024-09-11T01:52:00.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4affd640 1 monclient: found mon.noname-c 2024-09-11T01:52:00.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:00.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:52:00.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4affd640 1 -- 172.21.15.12:0/3665133977 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e4c13cfe0 con 0x7f6e4c138130 2024-09-11T01:52:00.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.421+0000 7f6e4b7fe640 1 --2- 172.21.15.12:0/3665133977 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e4c137d50 0x7f6e4c13c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:52:00.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.424+0000 7f6e4affd640 1 -- 172.21.15.12:0/3665133977 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6e4c138130 msgr2=0x7f6e4c13a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:00.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.424+0000 7f6e4affd640 1 --2- 172.21.15.12:0/3665133977 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6e4c138130 0x7f6e4c13a510 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6e4000c430 tx=0x7f6e4000c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:00.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.424+0000 7f6e33fff640 1 -- 172.21.15.12:0/3665133977 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e400119a0 con 0x7f6e4c138130 2024-09-11T01:52:00.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.424+0000 7f6e33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:00.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.424+0000 7f6e33fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:00.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.424+0000 7f6e33fff640 10 monclient: dump: 2024-09-11T01:52:00.442 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:00.442 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:00.442 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.442 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.442 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:00.443 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:00.443 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:00.443 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:00.443 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:00.443 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:00.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.424+0000 7f6e33fff640 10 monclient: _finish_auth 0 2024-09-11T01:52:00.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.424+0000 7f6e33fff640 10 monclient: _check_auth_tickets 2024-09-11T01:52:00.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.424+0000 7f6e33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:00.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.424+0000 7f6e33fff640 1 -- 172.21.15.12:0/3665133977 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e40017070 con 0x7f6e4c138130 2024-09-11T01:52:00.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.424+0000 7f6e520ef640 5 monclient: authenticate success, global_id 244229 2024-09-11T01:52:00.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.424+0000 7f6e520ef640 10 monclient: _renew_subs 2024-09-11T01:52:00.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.424+0000 7f6e520ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:00.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.424+0000 7f6e520ef640 1 -- 172.21.15.12:0/3665133977 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e4c13b4c0 con 0x7f6e4c138130 2024-09-11T01:52:00.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.424+0000 7f6e33fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:00.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.424+0000 7f6e33fff640 1 -- 172.21.15.12:0/3665133977 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e40011d80 con 0x7f6e4c138130 2024-09-11T01:52:00.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.424+0000 7f6e33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:00.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.424+0000 7f6e33fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:00.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.424+0000 7f6e33fff640 10 monclient: dump: 2024-09-11T01:52:00.444 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:00.444 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:00.444 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.444 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:00.445 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:00.445 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:00.445 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:00.445 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:00.445 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:00.445 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:00.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.424+0000 7f6e520ef640 10 monclient: _renew_subs 2024-09-11T01:52:00.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.424+0000 7f6e49ffb640 4 set_mon_vals no callback set 2024-09-11T01:52:00.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.424+0000 7f6e520ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:00.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.424+0000 7f6e520ef640 1 -- 172.21.15.12:0/3665133977 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e4c13bb40 con 0x7f6e4c138130 2024-09-11T01:52:00.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.425+0000 7f6e520ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:00.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.425+0000 7f6e520ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:00.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.425+0000 7f6e520ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:00.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.425+0000 7f6e520ef640 1 -- 172.21.15.12:0/3665133977 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e4c06a390 con 0x7f6e4c138130 2024-09-11T01:52:00.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.425+0000 7f6e33fff640 1 -- 172.21.15.12:0/3665133977 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f6e4001c460 con 0x7f6e4c138130 2024-09-11T01:52:00.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.425+0000 7f6e33fff640 1 --2- 172.21.15.12:0/3665133977 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6e3803e260 0x7f6e38040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:00.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.425+0000 7f6e33fff640 1 -- 172.21.15.12:0/3665133977 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(764..764 src has 676..764) ==== 4815+0+0 (secure 0 0 0) 0x7f6e40052bf0 con 0x7f6e4c138130 2024-09-11T01:52:00.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.425+0000 7f6e4b7fe640 1 --2- 172.21.15.12:0/3665133977 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6e3803e260 0x7f6e38040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:00.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.425+0000 7f6e4b7fe640 10 monclient: get_auth_request con 0x7f6e3803e260 auth_method 0 2024-09-11T01:52:00.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.428+0000 7f6e4b7fe640 1 --2- 172.21.15.12:0/3665133977 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6e3803e260 0x7f6e38040720 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c005160 tx=0x7f6e3c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:00.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.429+0000 7f6e33fff640 1 -- 172.21.15.12:0/3665133977 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6e4001ac60 con 0x7f6e4c138130 2024-09-11T01:52:00.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.429+0000 7f6e33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:00.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.429+0000 7f6e33fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:00.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.687+0000 7f6e520ef640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:52:00.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.687+0000 7f6e520ef640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:52:00.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.687+0000 7f6e520ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:00.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.687+0000 7f6e520ef640 1 -- 172.21.15.12:0/3665133977 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f6e4c09d190 con 0x7f6e4c138130 2024-09-11T01:52:00.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.688+0000 7f6e33fff640 1 -- 172.21.15.12:0/3665133977 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f6e400515d0 con 0x7f6e4c138130 2024-09-11T01:52:00.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.688+0000 7f6e33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:52:00.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.688+0000 7f6e33fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:00.690 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.689+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.690 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.689+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=10,kv=12,logm=50} crc {auth=3397491213,config=2586702517,health=1993185726,kv=3263697420,logm=3755669137}) 2024-09-11T01:52:00.691 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.689+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=23,config=2,health=11,kv=11,logm=53} crc {auth=2160902611,config=2586702516,health=1493617857,kv=58696370,logm=1975837284}) 2024-09-11T01:52:00.691 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.689+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.691 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.689+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=10,kv=12,logm=50} crc {auth=3397491213,config=2586702517,health=1993185726,kv=3263697420,logm=3755669137}) 2024-09-11T01:52:00.691 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.689+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=11,kv=11,logm=50} crc {auth=823747111,config=2586702516,health=1855816637,kv=3049666978,logm=3599828908}) 2024-09-11T01:52:00.691 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.690+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.691 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.690+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1997946892}) 2024-09-11T01:52:00.691 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.690+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2396913409}) 2024-09-11T01:52:00.692 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.690+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.692 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.690+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1997946892}) 2024-09-11T01:52:00.692 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.690+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3903612670}) 2024-09-11T01:52:00.692 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.691+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.692 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.691+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2402570839}) 2024-09-11T01:52:00.693 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.691+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2787127816}) 2024-09-11T01:52:00.693 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.691+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.693 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.691+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2402570839}) 2024-09-11T01:52:00.693 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.691+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3139276787}) 2024-09-11T01:52:00.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.691+0000 7f6e520ef640 1 -- 172.21.15.12:0/3665133977 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6e3803e260 msgr2=0x7f6e38040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:00.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.691+0000 7f6e520ef640 1 --2- 172.21.15.12:0/3665133977 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6e3803e260 0x7f6e38040720 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c005160 tx=0x7f6e3c009040 comp rx=0 tx=0).stop 2024-09-11T01:52:00.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.691+0000 7f6e520ef640 10 monclient: shutdown 2024-09-11T01:52:00.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.691+0000 7f6e520ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:00.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.691+0000 7f6e520ef640 1 -- 172.21.15.12:0/3665133977 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6e4c138130 msgr2=0x7f6e4c13a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:00.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.691+0000 7f6e520ef640 1 --2- 172.21.15.12:0/3665133977 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6e4c138130 0x7f6e4c13a510 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6e4000c430 tx=0x7f6e4000c8f0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.691+0000 7f6e520ef640 1 -- 172.21.15.12:0/3665133977 shutdown_connections 2024-09-11T01:52:00.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.691+0000 7f6e520ef640 1 --2- 172.21.15.12:0/3665133977 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f6e3803e260 0x7f6e38040720 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.691+0000 7f6e520ef640 1 --2- 172.21.15.12:0/3665133977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e4c13e290 0x7f6e4c136db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.691+0000 7f6e520ef640 1 --2- 172.21.15.12:0/3665133977 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6e4c138130 0x7f6e4c13a510 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.691+0000 7f6e520ef640 1 --2- 172.21.15.12:0/3665133977 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6e4c137d50 0x7f6e4c13c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:00.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.691+0000 7f6e520ef640 1 -- 172.21.15.12:0/3665133977 >> 172.21.15.12:0/3665133977 conn(0x7f6e4c0a7020 msgr2=0x7f6e4c13f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:00.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.692+0000 7f6e520ef640 1 -- 172.21.15.12:0/3665133977 shutdown_connections 2024-09-11T01:52:00.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:00.692+0000 7f6e520ef640 1 -- 172.21.15.12:0/3665133977 wait complete. 2024-09-11T01:52:00.696 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.692+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.696 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.692+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2282830426}) 2024-09-11T01:52:00.697 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.692+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3945926666}) 2024-09-11T01:52:00.697 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.692+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.697 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.692+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2282830426}) 2024-09-11T01:52:00.697 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.692+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2813717103}) 2024-09-11T01:52:00.697 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.693+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.697 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.693+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2842954892}) 2024-09-11T01:52:00.697 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.693+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1586726768}) 2024-09-11T01:52:00.697 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.693+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.697 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.693+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2842954892}) 2024-09-11T01:52:00.697 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.693+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3453541834}) 2024-09-11T01:52:00.697 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.694+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.697 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.694+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4111456448}) 2024-09-11T01:52:00.697 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.694+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2819877004}) 2024-09-11T01:52:00.697 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.694+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.698 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.694+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4111456448}) 2024-09-11T01:52:00.698 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.695+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3488253544}) 2024-09-11T01:52:00.699 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.695+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.700 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.696+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3765918989}) 2024-09-11T01:52:00.700 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.696+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1379721069}) 2024-09-11T01:52:00.700 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.696+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.700 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.696+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3765918989}) 2024-09-11T01:52:00.700 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.696+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1899750974}) 2024-09-11T01:52:00.700 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.696+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.700 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.696+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=155040240}) 2024-09-11T01:52:00.700 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.697+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2154310603}) 2024-09-11T01:52:00.700 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.697+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.700 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.697+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=155040240}) 2024-09-11T01:52:00.700 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.697+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3551146167}) 2024-09-11T01:52:00.700 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.697+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.701 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.698+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3780379126}) 2024-09-11T01:52:00.701 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.698+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1261601890}) 2024-09-11T01:52:00.701 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.698+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.701 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.698+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3780379126}) 2024-09-11T01:52:00.701 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.698+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3535412494}) 2024-09-11T01:52:00.703 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.698+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.703 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.699+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2002323413}) 2024-09-11T01:52:00.703 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.699+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=516398131}) 2024-09-11T01:52:00.703 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.699+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.704 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.699+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2002323413}) 2024-09-11T01:52:00.704 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.699+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=564625779}) 2024-09-11T01:52:00.704 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.700+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.704 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.700+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=365312797}) 2024-09-11T01:52:00.704 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.700+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2590828267}) 2024-09-11T01:52:00.704 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.700+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.704 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.700+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=365312797}) 2024-09-11T01:52:00.704 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.700+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1972570322}) 2024-09-11T01:52:00.704 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.701+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.704 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.701+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1960709292}) 2024-09-11T01:52:00.704 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.701+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2155772391}) 2024-09-11T01:52:00.704 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.701+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.704 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.701+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1960709292}) 2024-09-11T01:52:00.705 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.701+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1521974456}) 2024-09-11T01:52:00.705 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.702+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.705 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.702+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2055000283}) 2024-09-11T01:52:00.705 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.702+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=555553846}) 2024-09-11T01:52:00.705 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.702+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.705 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.702+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2055000283}) 2024-09-11T01:52:00.705 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.702+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3779505949}) 2024-09-11T01:52:00.708 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.702+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.708 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.702+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2650437590}) 2024-09-11T01:52:00.708 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.702+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1189682980}) 2024-09-11T01:52:00.708 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.702+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.708 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.702+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2650437590}) 2024-09-11T01:52:00.708 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.702+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1401301597}) 2024-09-11T01:52:00.708 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.703+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.708 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.703+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1243920674}) 2024-09-11T01:52:00.708 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.703+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2136455160}) 2024-09-11T01:52:00.708 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.703+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.709 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.703+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1243920674}) 2024-09-11T01:52:00.709 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.703+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=334897978}) 2024-09-11T01:52:00.709 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.704+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.709 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.704+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949176156}) 2024-09-11T01:52:00.709 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.704+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=234680249}) 2024-09-11T01:52:00.709 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.704+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.709 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.704+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949176156}) 2024-09-11T01:52:00.709 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.704+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1111197368}) 2024-09-11T01:52:00.709 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.705+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.709 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.705+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=524463954}) 2024-09-11T01:52:00.709 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.705+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2996495771}) 2024-09-11T01:52:00.709 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.705+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.709 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.705+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=524463954}) 2024-09-11T01:52:00.709 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.705+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2178044181}) 2024-09-11T01:52:00.710 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.706+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.710 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.706+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2856244446}) 2024-09-11T01:52:00.710 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.706+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3415749042}) 2024-09-11T01:52:00.710 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.706+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.710 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.706+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2856244446}) 2024-09-11T01:52:00.710 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.706+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1859605331}) 2024-09-11T01:52:00.711 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:52:00.713 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.707+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.713 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.707+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3790080}) 2024-09-11T01:52:00.714 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.707+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=78865318}) 2024-09-11T01:52:00.714 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.707+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.714 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.707+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3790080}) 2024-09-11T01:52:00.714 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.707+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2073721152}) 2024-09-11T01:52:00.714 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.708+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.714 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.708+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3847905789}) 2024-09-11T01:52:00.714 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.708+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3431405289}) 2024-09-11T01:52:00.714 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.708+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.714 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.708+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3847905789}) 2024-09-11T01:52:00.714 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.708+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2981193872}) 2024-09-11T01:52:00.714 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.708+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.714 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.708+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3215307973}) 2024-09-11T01:52:00.714 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.708+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=978475869}) 2024-09-11T01:52:00.714 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.708+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.715 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.708+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3215307973}) 2024-09-11T01:52:00.715 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.708+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2623755403}) 2024-09-11T01:52:00.715 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.709+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.715 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.709+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1906399802}) 2024-09-11T01:52:00.715 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.709+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2477374555}) 2024-09-11T01:52:00.715 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.709+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.715 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.709+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1906399802}) 2024-09-11T01:52:00.715 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.709+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2240103598}) 2024-09-11T01:52:00.715 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.710+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.715 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.710+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2071352092}) 2024-09-11T01:52:00.716 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.710+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=274326857}) 2024-09-11T01:52:00.716 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.710+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.716 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.710+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2071352092}) 2024-09-11T01:52:00.716 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.710+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1522928853}) 2024-09-11T01:52:00.716 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.711+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.716 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.711+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2427545422}) 2024-09-11T01:52:00.716 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.711+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2919322108}) 2024-09-11T01:52:00.716 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.711+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.716 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.711+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2427545422}) 2024-09-11T01:52:00.716 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.711+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1390414634}) 2024-09-11T01:52:00.716 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.712+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.716 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.712+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2389246040}) 2024-09-11T01:52:00.716 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.712+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1786085741}) 2024-09-11T01:52:00.717 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.712+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.717 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.712+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2389246040}) 2024-09-11T01:52:00.717 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.712+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4084624464}) 2024-09-11T01:52:00.722 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.712+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.722 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.712+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=997055406}) 2024-09-11T01:52:00.722 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.712+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=284863066}) 2024-09-11T01:52:00.722 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.712+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.722 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.712+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=997055406}) 2024-09-11T01:52:00.722 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.712+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1878594234}) 2024-09-11T01:52:00.722 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.713+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.722 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.713+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1468740825}) 2024-09-11T01:52:00.722 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.713+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2588722448}) 2024-09-11T01:52:00.722 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.713+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.722 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.713+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1468740825}) 2024-09-11T01:52:00.723 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.713+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1871522453}) 2024-09-11T01:52:00.723 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.714+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.723 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.714+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1651139080}) 2024-09-11T01:52:00.723 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.714+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=169820688}) 2024-09-11T01:52:00.723 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.714+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.723 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.714+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1651139080}) 2024-09-11T01:52:00.723 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.714+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=133517946}) 2024-09-11T01:52:00.723 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.715+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.723 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.715+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=39679455}) 2024-09-11T01:52:00.723 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.715+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3130237419}) 2024-09-11T01:52:00.723 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.715+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.723 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.715+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=39679455}) 2024-09-11T01:52:00.723 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.715+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4059116378}) 2024-09-11T01:52:00.723 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.715+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.724 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.715+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3920818749}) 2024-09-11T01:52:00.724 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.715+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=667979359}) 2024-09-11T01:52:00.724 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.715+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.724 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.715+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3920818749}) 2024-09-11T01:52:00.724 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.715+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=877035979}) 2024-09-11T01:52:00.724 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.716+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.724 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.716+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1031955244}) 2024-09-11T01:52:00.724 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.716+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=552490128}) 2024-09-11T01:52:00.724 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.716+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.724 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.716+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1031955244}) 2024-09-11T01:52:00.724 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.716+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3892159492}) 2024-09-11T01:52:00.724 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.717+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.724 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.717+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3181450624}) 2024-09-11T01:52:00.725 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.717+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1233598301}) 2024-09-11T01:52:00.725 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.717+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.725 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.717+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3181450624}) 2024-09-11T01:52:00.725 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.717+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=556671789}) 2024-09-11T01:52:00.725 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.718+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.725 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.718+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2922346696}) 2024-09-11T01:52:00.725 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.718+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=809092075}) 2024-09-11T01:52:00.725 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.718+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.725 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.718+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2922346696}) 2024-09-11T01:52:00.725 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.718+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1981579088}) 2024-09-11T01:52:00.725 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.718+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.725 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.718+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1296696068}) 2024-09-11T01:52:00.725 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.718+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2349707731}) 2024-09-11T01:52:00.725 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.719+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.726 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.719+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1296696068}) 2024-09-11T01:52:00.726 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.719+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=153911736}) 2024-09-11T01:52:00.726 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.719+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.726 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.719+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1328462340}) 2024-09-11T01:52:00.726 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.719+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1788800173}) 2024-09-11T01:52:00.726 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.719+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.726 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.719+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1328462340}) 2024-09-11T01:52:00.726 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.719+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3069931750}) 2024-09-11T01:52:00.726 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.720+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.726 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.720+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2156600157}) 2024-09-11T01:52:00.726 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.720+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=15003042}) 2024-09-11T01:52:00.726 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.720+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.726 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.720+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2156600157}) 2024-09-11T01:52:00.726 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.720+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3668975886}) 2024-09-11T01:52:00.730 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.721+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.730 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.721+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4124235376}) 2024-09-11T01:52:00.731 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.721+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4058207379}) 2024-09-11T01:52:00.731 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.721+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.731 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.721+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4124235376}) 2024-09-11T01:52:00.731 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.721+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1270175734}) 2024-09-11T01:52:00.731 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.722+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.731 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.722+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2117170431}) 2024-09-11T01:52:00.731 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.722+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3222241134}) 2024-09-11T01:52:00.731 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.722+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.731 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.722+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2117170431}) 2024-09-11T01:52:00.731 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.722+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2413795248}) 2024-09-11T01:52:00.731 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.723+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.731 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.723+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3207513209}) 2024-09-11T01:52:00.731 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.723+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1240721993}) 2024-09-11T01:52:00.732 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.723+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.732 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.723+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3207513209}) 2024-09-11T01:52:00.732 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.723+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1769966138}) 2024-09-11T01:52:00.732 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.723+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.732 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.723+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2040466172}) 2024-09-11T01:52:00.732 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.723+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3553585532}) 2024-09-11T01:52:00.732 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.723+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.732 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.723+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2040466172}) 2024-09-11T01:52:00.732 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.723+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3810627399}) 2024-09-11T01:52:00.732 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.724+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.732 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.724+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=987086803}) 2024-09-11T01:52:00.732 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.724+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3247481061}) 2024-09-11T01:52:00.732 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.724+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.733 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.724+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=987086803}) 2024-09-11T01:52:00.733 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.724+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3141716280}) 2024-09-11T01:52:00.733 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.725+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.733 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.725+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2084531197}) 2024-09-11T01:52:00.733 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.725+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1403755044}) 2024-09-11T01:52:00.733 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.725+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.733 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.725+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2084531197}) 2024-09-11T01:52:00.733 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.725+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1040911772}) 2024-09-11T01:52:00.733 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.726+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.733 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.726+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2657698364}) 2024-09-11T01:52:00.733 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.726+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2146680753}) 2024-09-11T01:52:00.733 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.726+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.733 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.726+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2657698364}) 2024-09-11T01:52:00.734 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.726+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2622199728}) 2024-09-11T01:52:00.734 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.727+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.734 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.727+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1445268566}) 2024-09-11T01:52:00.734 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.727+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3904926961}) 2024-09-11T01:52:00.734 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.727+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.734 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.727+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1445268566}) 2024-09-11T01:52:00.734 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.727+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=353623483}) 2024-09-11T01:52:00.734 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.727+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.734 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.727+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3689517741}) 2024-09-11T01:52:00.734 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.727+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=684940777}) 2024-09-11T01:52:00.734 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.727+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.734 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.727+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3689517741}) 2024-09-11T01:52:00.734 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.727+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=628662746}) 2024-09-11T01:52:00.734 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.728+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.735 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.728+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715372167}) 2024-09-11T01:52:00.735 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.728+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1199251221}) 2024-09-11T01:52:00.735 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.728+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.735 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.728+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715372167}) 2024-09-11T01:52:00.735 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.728+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2906611829}) 2024-09-11T01:52:00.735 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.729+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.735 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.729+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2584348179}) 2024-09-11T01:52:00.735 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.729+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=888342524}) 2024-09-11T01:52:00.735 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.729+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.735 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.729+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2584348179}) 2024-09-11T01:52:00.735 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.729+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3615936984}) 2024-09-11T01:52:00.739 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.730+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.739 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.730+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4232146484}) 2024-09-11T01:52:00.739 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.730+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1160629046}) 2024-09-11T01:52:00.739 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.730+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.739 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.730+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4232146484}) 2024-09-11T01:52:00.739 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.730+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4009093512}) 2024-09-11T01:52:00.739 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.731+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.739 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.731+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2739662552}) 2024-09-11T01:52:00.739 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.731+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=8873376}) 2024-09-11T01:52:00.739 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.731+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.740 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.731+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2739662552}) 2024-09-11T01:52:00.740 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.731+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3289315729}) 2024-09-11T01:52:00.740 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.731+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.740 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.731+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1388851868}) 2024-09-11T01:52:00.740 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.731+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2009150000}) 2024-09-11T01:52:00.740 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.732+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.740 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.732+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1388851868}) 2024-09-11T01:52:00.740 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.732+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3633764166}) 2024-09-11T01:52:00.740 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.732+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.740 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.732+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3647954628}) 2024-09-11T01:52:00.740 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.732+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3710493185}) 2024-09-11T01:52:00.740 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.732+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.741 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.732+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3647954628}) 2024-09-11T01:52:00.741 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.732+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=598738768}) 2024-09-11T01:52:00.741 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.733+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.741 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.733+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1408525884}) 2024-09-11T01:52:00.741 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.733+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1722212665}) 2024-09-11T01:52:00.741 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.733+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.741 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.733+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1408525884}) 2024-09-11T01:52:00.741 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.733+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3841128181}) 2024-09-11T01:52:00.741 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.734+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.741 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.734+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=780285943}) 2024-09-11T01:52:00.741 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.734+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3784056723}) 2024-09-11T01:52:00.741 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.734+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.741 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.734+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=780285943}) 2024-09-11T01:52:00.741 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.734+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3756272788}) 2024-09-11T01:52:00.742 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.735+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.742 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.735+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179425160}) 2024-09-11T01:52:00.742 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.735+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2156049122}) 2024-09-11T01:52:00.742 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.735+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.742 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.735+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179425160}) 2024-09-11T01:52:00.742 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.735+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=758861938}) 2024-09-11T01:52:00.742 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.735+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.742 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.735+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=893076530}) 2024-09-11T01:52:00.742 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.735+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2619205446}) 2024-09-11T01:52:00.742 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.735+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.742 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.735+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=893076530}) 2024-09-11T01:52:00.742 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.736+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3271142446}) 2024-09-11T01:52:00.742 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.736+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.742 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.736+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=659481500}) 2024-09-11T01:52:00.743 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.736+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=649521741}) 2024-09-11T01:52:00.743 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.736+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.743 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.736+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=659481500}) 2024-09-11T01:52:00.743 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.736+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2720412876}) 2024-09-11T01:52:00.743 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.737+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.743 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.737+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=368185336}) 2024-09-11T01:52:00.743 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.737+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=535755212}) 2024-09-11T01:52:00.743 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.737+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.743 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.737+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=368185336}) 2024-09-11T01:52:00.743 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.737+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1706306797}) 2024-09-11T01:52:00.747 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.738+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.747 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.738+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2424754257}) 2024-09-11T01:52:00.747 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.738+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1880133259}) 2024-09-11T01:52:00.747 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.738+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.748 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.738+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2424754257}) 2024-09-11T01:52:00.748 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.738+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2890043180}) 2024-09-11T01:52:00.748 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.738+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.748 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.738+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1340801432}) 2024-09-11T01:52:00.748 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.739+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1761747543}) 2024-09-11T01:52:00.748 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.739+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.748 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.739+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1340801432}) 2024-09-11T01:52:00.748 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.739+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2590108324}) 2024-09-11T01:52:00.748 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.739+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.748 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.739+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3844268348}) 2024-09-11T01:52:00.748 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.739+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1416804640}) 2024-09-11T01:52:00.748 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.739+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.748 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.739+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3844268348}) 2024-09-11T01:52:00.749 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.739+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3653218799}) 2024-09-11T01:52:00.749 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.740+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.749 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.740+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598309270}) 2024-09-11T01:52:00.749 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.740+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3903488730}) 2024-09-11T01:52:00.749 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.740+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.749 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.740+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598309270}) 2024-09-11T01:52:00.749 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.740+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2867344364}) 2024-09-11T01:52:00.749 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.741+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.749 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.741+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4096201336}) 2024-09-11T01:52:00.749 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.741+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1750774853}) 2024-09-11T01:52:00.749 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.741+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.749 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.741+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4096201336}) 2024-09-11T01:52:00.749 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.741+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3965850388}) 2024-09-11T01:52:00.750 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.742+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.750 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.742+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1449977514}) 2024-09-11T01:52:00.750 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.742+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=598958453}) 2024-09-11T01:52:00.750 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.742+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.750 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.742+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1449977514}) 2024-09-11T01:52:00.750 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.742+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=350200938}) 2024-09-11T01:52:00.750 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.742+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.750 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.742+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2589589452}) 2024-09-11T01:52:00.750 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.742+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2466012544}) 2024-09-11T01:52:00.750 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.742+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.750 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.743+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2589589452}) 2024-09-11T01:52:00.750 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.743+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3842803230}) 2024-09-11T01:52:00.750 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.743+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.750 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.743+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1632888188}) 2024-09-11T01:52:00.751 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.743+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3179158193}) 2024-09-11T01:52:00.751 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.743+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.751 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.743+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1632888188}) 2024-09-11T01:52:00.751 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.743+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=61758047}) 2024-09-11T01:52:00.751 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.744+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.751 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.744+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3472178555}) 2024-09-11T01:52:00.751 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.744+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3131729798}) 2024-09-11T01:52:00.751 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.744+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.751 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.744+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3472178555}) 2024-09-11T01:52:00.751 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.744+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=548552261}) 2024-09-11T01:52:00.751 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.745+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.751 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.745+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801191627}) 2024-09-11T01:52:00.752 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.745+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2605841150}) 2024-09-11T01:52:00.752 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.745+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.752 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.745+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801191627}) 2024-09-11T01:52:00.752 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.745+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3719287426}) 2024-09-11T01:52:00.752 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.746+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.752 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.746+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1229126406}) 2024-09-11T01:52:00.752 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.746+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1521727809}) 2024-09-11T01:52:00.752 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.746+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.752 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.746+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1229126406}) 2024-09-11T01:52:00.752 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.746+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=543845544}) 2024-09-11T01:52:00.757 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.746+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.757 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.746+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4269947062}) 2024-09-11T01:52:00.757 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.746+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2112642661}) 2024-09-11T01:52:00.757 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.746+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.757 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.746+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4269947062}) 2024-09-11T01:52:00.757 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.746+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4062165501}) 2024-09-11T01:52:00.757 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.747+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.758 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.747+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1035421450}) 2024-09-11T01:52:00.758 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.747+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3292246343}) 2024-09-11T01:52:00.758 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.747+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.758 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.747+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1035421450}) 2024-09-11T01:52:00.758 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.747+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1460797734}) 2024-09-11T01:52:00.758 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.748+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.758 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.748+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=45688476}) 2024-09-11T01:52:00.758 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.748+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3937317585}) 2024-09-11T01:52:00.758 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.748+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.758 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.748+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=45688476}) 2024-09-11T01:52:00.758 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.748+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3114963474}) 2024-09-11T01:52:00.758 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.749+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.758 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.749+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2482083874}) 2024-09-11T01:52:00.758 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.749+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=988311938}) 2024-09-11T01:52:00.759 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.749+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.759 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.749+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2482083874}) 2024-09-11T01:52:00.759 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.749+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2088409103}) 2024-09-11T01:52:00.759 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.749+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.759 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.749+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=825517703}) 2024-09-11T01:52:00.759 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.749+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3270819285}) 2024-09-11T01:52:00.759 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.749+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.759 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.749+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=825517703}) 2024-09-11T01:52:00.759 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.749+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3863468075}) 2024-09-11T01:52:00.759 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.750+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.759 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.750+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2194271908}) 2024-09-11T01:52:00.759 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.750+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3042457711}) 2024-09-11T01:52:00.760 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.750+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.760 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.750+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2194271908}) 2024-09-11T01:52:00.760 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.750+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1644601191}) 2024-09-11T01:52:00.760 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.751+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.760 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.751+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=112555565}) 2024-09-11T01:52:00.760 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.751+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1601410611}) 2024-09-11T01:52:00.760 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.751+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.760 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.751+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=112555565}) 2024-09-11T01:52:00.760 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.751+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2675844963}) 2024-09-11T01:52:00.760 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.752+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.761 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.752+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2303434677}) 2024-09-11T01:52:00.761 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.752+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=853996079}) 2024-09-11T01:52:00.761 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.752+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.761 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.752+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2303434677}) 2024-09-11T01:52:00.761 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.752+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=611233163}) 2024-09-11T01:52:00.761 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.752+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.761 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.752+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1339091449}) 2024-09-11T01:52:00.761 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.752+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2045439259}) 2024-09-11T01:52:00.761 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.752+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.761 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.752+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1339091449}) 2024-09-11T01:52:00.761 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.752+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2737280893}) 2024-09-11T01:52:00.761 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.753+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.761 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.753+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2903399358}) 2024-09-11T01:52:00.761 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.753+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1772848623}) 2024-09-11T01:52:00.762 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.753+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.762 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.753+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2903399358}) 2024-09-11T01:52:00.762 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.753+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=63257078}) 2024-09-11T01:52:00.762 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.754+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.762 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.754+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1135749430}) 2024-09-11T01:52:00.762 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.754+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2034773695}) 2024-09-11T01:52:00.762 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.754+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.762 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.754+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1135749430}) 2024-09-11T01:52:00.762 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.754+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1327346676}) 2024-09-11T01:52:00.762 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.755+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.762 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.755+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3632683762}) 2024-09-11T01:52:00.762 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.755+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=742364303}) 2024-09-11T01:52:00.762 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.755+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.762 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.755+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3632683762}) 2024-09-11T01:52:00.763 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.755+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4029266060}) 2024-09-11T01:52:00.763 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.755+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.763 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.755+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3878712940}) 2024-09-11T01:52:00.763 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.755+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3626443914}) 2024-09-11T01:52:00.763 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.755+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.763 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.755+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3878712940}) 2024-09-11T01:52:00.763 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.755+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3787721743}) 2024-09-11T01:52:00.765 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.756+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.765 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.756+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3294915082}) 2024-09-11T01:52:00.765 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.756+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1694299884}) 2024-09-11T01:52:00.765 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.756+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.765 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.756+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3294915082}) 2024-09-11T01:52:00.765 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.756+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=468876444}) 2024-09-11T01:52:00.766 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.757+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.766 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.757+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1062399468}) 2024-09-11T01:52:00.766 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.757+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=835401434}) 2024-09-11T01:52:00.766 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.757+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.766 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.757+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1062399468}) 2024-09-11T01:52:00.766 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.757+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3076929952}) 2024-09-11T01:52:00.766 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.758+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.766 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.758+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2504079693}) 2024-09-11T01:52:00.766 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.758+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4181708887}) 2024-09-11T01:52:00.766 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.758+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.766 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.758+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2504079693}) 2024-09-11T01:52:00.766 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.758+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3048992601}) 2024-09-11T01:52:00.766 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.758+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.767 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.758+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597487883}) 2024-09-11T01:52:00.767 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.759+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=62044341}) 2024-09-11T01:52:00.767 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.759+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.767 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.759+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597487883}) 2024-09-11T01:52:00.767 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.759+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4151425988}) 2024-09-11T01:52:00.768 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.767+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.768 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.767+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3629226251}) 2024-09-11T01:52:00.768 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.767+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1745668191}) 2024-09-11T01:52:00.769 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.767+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.769 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.767+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3629226251}) 2024-09-11T01:52:00.769 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.767+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2706277784}) 2024-09-11T01:52:00.769 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.767+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.769 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.768+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=90,mdsmap=2,mgr=8} crc {logm=1625155467,mdsmap=3140514009,mgr=4124675564}) 2024-09-11T01:52:00.770 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.768+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=3768912896,mdsmap=3760429986,mgr=96270735}) 2024-09-11T01:52:00.770 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.768+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.770 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.768+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=90,mdsmap=2,mgr=8} crc {logm=1625155467,mdsmap=3140514009,mgr=4124675564}) 2024-09-11T01:52:00.770 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.768+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=92,mdsmap=2,mgr=6} crc {logm=2441011924,mdsmap=161958762,mgr=4190131725}) 2024-09-11T01:52:00.771 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.769+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.771 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.769+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=60} crc {mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=4005664416,mon_config_key=1809882168,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=623017975,osd_snap=3569202569}) 2024-09-11T01:52:00.771 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.769+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_snap=61} crc {mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=4005664416,mon_config_key=3884116196,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_snap=2169157722}) 2024-09-11T01:52:00.771 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.769+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.771 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.769+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=60} crc {mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=4005664416,mon_config_key=1809882168,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=623017975,osd_snap=3569202569}) 2024-09-11T01:52:00.771 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.769+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=2,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=62} crc {mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=4005664416,mon_config_key=1446728439,monmap=3140514009,nvmeofgw=598337686,osd_metadata=1404192910,osd_pg_creating=623017975,osd_snap=3808582955}) 2024-09-11T01:52:00.771 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.769+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.772 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.769+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=93,osdmap=7} crc {osd_snap=357906041,osdmap=3359272811}) 2024-09-11T01:52:00.772 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.769+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=93,osdmap=7} crc {osd_snap=3895812536,osdmap=3359272811}) 2024-09-11T01:52:00.772 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.769+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.772 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.769+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=93,osdmap=7} crc {osd_snap=357906041,osdmap=3359272811}) 2024-09-11T01:52:00.772 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.769+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=95,osdmap=5} crc {osd_snap=4177344778,osdmap=1968204048}) 2024-09-11T01:52:00.772 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.770+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.773 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.770+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2154291976}) 2024-09-11T01:52:00.773 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.770+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1515550530}) 2024-09-11T01:52:00.773 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.770+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.773 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.770+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2154291976}) 2024-09-11T01:52:00.773 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.770+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3375657701}) 2024-09-11T01:52:00.775 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.774+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.775 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.774+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=36} crc {osdmap=452768479}) 2024-09-11T01:52:00.776 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.774+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=39} crc {osdmap=2106957047}) 2024-09-11T01:52:00.776 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.774+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:00.776 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.774+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=36} crc {osdmap=452768479}) 2024-09-11T01:52:00.776 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:00.774+0000 7efd84b6d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=39} crc {osdmap=4087480536}) 2024-09-11T01:52:01.549 INFO:tasks.rados.rados.0.smithi071.stdout:3405: done (1 left) 2024-09-11T01:52:01.550 INFO:tasks.rados.rados.0.smithi071.stdout:3406: write oid 50 current snap is 379 2024-09-11T01:52:01.550 INFO:tasks.rados.rados.0.smithi071.stdout:3404: finishing write tid 1 to smithi07134777-24 2024-09-11T01:52:01.550 INFO:tasks.rados.rados.0.smithi071.stdout:3406: seq_num 1191 ranges {417137=572627,1428054=589083} 2024-09-11T01:52:01.555 INFO:tasks.rados.rados.0.smithi071.stdout:3406: writing smithi07134777-50 from 417137 to 989764 tid 1 2024-09-11T01:52:01.563 INFO:tasks.rados.rados.0.smithi071.stdout:3406: writing smithi07134777-50 from 1428054 to 2017137 tid 2 2024-09-11T01:52:01.563 INFO:tasks.rados.rados.0.smithi071.stdout:3407: read oid 6 snap 366 2024-09-11T01:52:01.563 INFO:tasks.rados.rados.0.smithi071.stdout:3407: expect (ObjNum 1151 snap 366 seq_num 1151) 2024-09-11T01:52:01.563 INFO:tasks.rados.rados.0.smithi071.stdout:3404: finishing write tid 2 to smithi07134777-24 2024-09-11T01:52:01.563 INFO:tasks.rados.rados.0.smithi071.stdout:3404: finishing write tid 3 to smithi07134777-24 2024-09-11T01:52:01.563 INFO:tasks.rados.rados.0.smithi071.stdout:3404: finishing write tid 4 to smithi07134777-24 2024-09-11T01:52:01.563 INFO:tasks.rados.rados.0.smithi071.stdout:3404: oid 24 updating version 0 to 2096 2024-09-11T01:52:01.563 INFO:tasks.rados.rados.0.smithi071.stdout:3404: oid 24 version 2096 is already newer than 2094 2024-09-11T01:52:01.563 INFO:tasks.rados.rados.0.smithi071.stdout:3404: oid 24 version 2096 is already newer than 2095 2024-09-11T01:52:01.563 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 2096 (ObjNum 1190 snap 379 seq_num 1190) dirty exists 2024-09-11T01:52:01.563 INFO:tasks.rados.rados.0.smithi071.stdout:3404: left oid 24 (ObjNum 1190 snap 379 seq_num 1190) 2024-09-11T01:52:01.564 INFO:tasks.rados.rados.0.smithi071.stdout:3404: done (2 left) 2024-09-11T01:52:01.564 INFO:tasks.rados.rados.0.smithi071.stdout:3408: rollback oid 14 current snap is 379 2024-09-11T01:52:01.564 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 14 to 374 2024-09-11T01:52:01.564 INFO:tasks.rados.rados.0.smithi071.stdout:3409: rollback oid 47 current snap is 379 2024-09-11T01:52:01.564 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 47 to 338 2024-09-11T01:52:01.564 INFO:tasks.rados.rados.0.smithi071.stdout:3410: delete oid 41 current snap is 379 2024-09-11T01:52:01.569 INFO:tasks.rados.rados.0.smithi071.stdout:3406: finishing write tid 1 to smithi07134777-50 2024-09-11T01:52:01.571 INFO:tasks.rados.rados.0.smithi071.stdout:3410: done (4 left) 2024-09-11T01:52:01.571 INFO:tasks.rados.rados.0.smithi071.stdout:3411: snap_create 2024-09-11T01:52:01.575 INFO:tasks.rados.rados.0.smithi071.stdout:3409: finishing rollback tid 0 to smithi07134777-47 2024-09-11T01:52:01.575 INFO:tasks.rados.rados.0.smithi071.stdout:3409: finishing rollback tid 1 to smithi07134777-47 2024-09-11T01:52:01.575 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 2544 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-11T01:52:01.578 INFO:tasks.rados.rados.0.smithi071.stdout:3407: expect (ObjNum 1151 snap 366 seq_num 1151) 2024-09-11T01:52:01.712 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:52:01.712 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:52:01.723 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2326712 2024-09-11T01:52:01.750 INFO:tasks.rados.rados.0.smithi071.stdout:3406: finishing write tid 2 to smithi07134777-50 2024-09-11T01:52:01.750 INFO:tasks.rados.rados.0.smithi071.stdout:3406: finishing write tid 3 to smithi07134777-50 2024-09-11T01:52:01.750 INFO:tasks.rados.rados.0.smithi071.stdout:3406: finishing write tid 4 to smithi07134777-50 2024-09-11T01:52:01.750 INFO:tasks.rados.rados.0.smithi071.stdout:3406: oid 50 updating version 0 to 2566 2024-09-11T01:52:01.750 INFO:tasks.rados.rados.0.smithi071.stdout:3406: oid 50 updating version 2566 to 2568 2024-09-11T01:52:01.750 INFO:tasks.rados.rados.0.smithi071.stdout:3406: oid 50 version 2568 is already newer than 2567 2024-09-11T01:52:01.750 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 2568 (ObjNum 1191 snap 379 seq_num 1191) dirty exists 2024-09-11T01:52:01.750 INFO:tasks.rados.rados.0.smithi071.stdout:3406: left oid 50 (ObjNum 1191 snap 379 seq_num 1191) 2024-09-11T01:52:01.750 INFO:tasks.rados.rados.0.smithi071.stdout:3408: finishing rollback tid 0 to smithi07134777-14 2024-09-11T01:52:01.750 INFO:tasks.rados.rados.0.smithi071.stdout:3408: finishing rollback tid 1 to smithi07134777-14 2024-09-11T01:52:01.750 INFO:tasks.rados.rados.0.smithi071.stdout:3408: finishing rollback tid 2 to smithi07134777-14 2024-09-11T01:52:01.751 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 2572 (ObjNum 1136 snap 360 seq_num 1136) dirty exists 2024-09-11T01:52:01.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.824+0000 7f11da12b640 10 monclient: get_monmap_and_config 2024-09-11T01:52:01.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.824+0000 7f11da12b640 10 monclient: build_initial_monmap 2024-09-11T01:52:01.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.824+0000 7f11da12b640 10 monclient: monmap: 2024-09-11T01:52:01.826 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:01.826 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:01.826 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:01.824917+0000 2024-09-11T01:52:01.826 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:01.824917+0000 2024-09-11T01:52:01.826 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:01.826 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:01.826 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:01.827 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:01.827 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:01.827 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:01.827 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:01.827 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:01.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.824+0000 7f11da12b640 1 Processor -- start 2024-09-11T01:52:01.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11da12b640 1 -- start start 2024-09-11T01:52:01.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11da12b640 10 monclient: init 2024-09-11T01:52:01.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11da12b640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:01.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11da12b640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-11T01:52:01.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11da12b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f11d40a2db0 0x7f11d406cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:01.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11da12b640 10 monclient(hunting): picked mon.noname-c con 0x7f11d40a2db0 addr v2:172.21.15.71:3300/0 2024-09-11T01:52:01.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11da12b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f11d409fbd0 0x7f11d406d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:01.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11da12b640 10 monclient(hunting): picked mon.noname-a con 0x7f11d409fbd0 addr v2:172.21.15.12:3300/0 2024-09-11T01:52:01.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11da12b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f11d406d5d0 addr v1:172.21.15.71:6789/0 2024-09-11T01:52:01.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11da12b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11d409eea0 con 0x7f11d406d5d0 2024-09-11T01:52:01.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11da12b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:01.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11da12b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f11d40a3850 con 0x7f11d409fbd0 2024-09-11T01:52:01.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11da12b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:01.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11da12b640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f11d40a3190 con 0x7f11d40a2db0 2024-09-11T01:52:01.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11da12b640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:01.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11da12b640 10 monclient(hunting): authenticate will time out at 2128.267669s 2024-09-11T01:52:01.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11d2ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f11d409fbd0 0x7f11d406d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:01.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11d2ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f11d409fbd0 0x7f11d406d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37998/0 (socket says 172.21.15.12:37998) 2024-09-11T01:52:01.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11d2ffd640 1 -- 172.21.15.12:0/3481342428 learned_addr learned my addr 172.21.15.12:0/3481342428 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:01.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11d2ffd640 10 monclient(hunting): get_auth_request con 0x7f11d409fbd0 auth_method 0 2024-09-11T01:52:01.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:01.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:01.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:01.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:01.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:01.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:01.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.825+0000 7f11d37fe640 1 --2- 172.21.15.12:0/3481342428 >> v2:172.21.15.71:3300/0 conn(0x7f11d40a2db0 0x7f11d406cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:01.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.826+0000 7f11d37fe640 10 monclient(hunting): get_auth_request con 0x7f11d40a2db0 auth_method 0 2024-09-11T01:52:01.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.826+0000 7f11d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:01.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.826+0000 7f11d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:01.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.826+0000 7f11d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:01.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.826+0000 7f11d2ffd640 10 monclient(hunting): handle_auth_done global_id 254133 payload 274 2024-09-11T01:52:01.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.826+0000 7f11d2ffd640 1 -- 172.21.15.12:0/3481342428 >> v2:172.21.15.71:3300/0 conn(0x7f11d40a2db0 msgr2=0x7f11d406cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:01.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.826+0000 7f11d2ffd640 1 --2- 172.21.15.12:0/3481342428 >> v2:172.21.15.71:3300/0 conn(0x7f11d40a2db0 0x7f11d406cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:01.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.826+0000 7f11d2ffd640 1 -- 172.21.15.12:0/3481342428 >> v1:172.21.15.71:6789/0 conn(0x7f11d406d5d0 legacy=0x7f11d413e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:01.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.826+0000 7f11d2ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:52:01.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.826+0000 7f11d2ffd640 1 monclient: found mon.noname-a 2024-09-11T01:52:01.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.826+0000 7f11d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:01.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.826+0000 7f11d2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:52:01.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.826+0000 7f11d2ffd640 1 -- 172.21.15.12:0/3481342428 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d409f180 con 0x7f11d409fbd0 2024-09-11T01:52:01.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.829+0000 7f11d2ffd640 1 -- 172.21.15.12:0/3481342428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11d409fbd0 msgr2=0x7f11d406d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:01.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.829+0000 7f11d2ffd640 1 --2- 172.21.15.12:0/3481342428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11d409fbd0 0x7f11d406d090 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f11d406db60 tx=0x7f11bc02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b19244022d1771e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:01.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.829+0000 7f11d27fc640 1 -- 172.21.15.12:0/3481342428 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11bc037070 con 0x7f11d409fbd0 2024-09-11T01:52:01.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.829+0000 7f11d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:01.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.829+0000 7f11d27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:52:01.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.829+0000 7f11d27fc640 10 monclient: dump: 2024-09-11T01:52:01.845 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:01.845 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:01.845 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:01.845 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:01.845 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:01.845 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:01.846 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:01.846 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:01.846 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:01.846 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:01.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.829+0000 7f11d27fc640 10 monclient: _finish_auth 0 2024-09-11T01:52:01.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.829+0000 7f11d27fc640 10 monclient: _check_auth_tickets 2024-09-11T01:52:01.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.829+0000 7f11d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:01.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.829+0000 7f11da12b640 5 monclient: authenticate success, global_id 254133 2024-09-11T01:52:01.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.829+0000 7f11da12b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:52:01.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.829+0000 7f11d27fc640 1 -- 172.21.15.12:0/3481342428 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11bc032640 con 0x7f11d409fbd0 2024-09-11T01:52:01.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.829+0000 7f11d27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:01.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.829+0000 7f11da12b640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:01.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.829+0000 7f11d27fc640 1 -- 172.21.15.12:0/3481342428 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11bc032a10 con 0x7f11d409fbd0 2024-09-11T01:52:01.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.829+0000 7f11da12b640 4 set_mon_vals no callback set 2024-09-11T01:52:01.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.829+0000 7f11d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:01.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.829+0000 7f11d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:52:01.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.829+0000 7f11d27fc640 10 monclient: dump: 2024-09-11T01:52:01.847 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:01.847 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:01.847 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:01.847 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:01.847 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:01.847 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:01.847 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:01.847 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:01.848 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:01.848 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:01.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.829+0000 7f11da12b640 10 monclient: shutdown 2024-09-11T01:52:01.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.829+0000 7f11da12b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:01.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.829+0000 7f11da12b640 1 -- 172.21.15.12:0/3481342428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11d409fbd0 msgr2=0x7f11d406d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:01.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.829+0000 7f11da12b640 1 --2- 172.21.15.12:0/3481342428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11d409fbd0 0x7f11d406d090 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f11d406db60 tx=0x7f11bc02a100 comp rx=0 tx=0).stop 2024-09-11T01:52:01.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.830+0000 7f11da12b640 1 -- 172.21.15.12:0/3481342428 shutdown_connections 2024-09-11T01:52:01.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.830+0000 7f11da12b640 1 --2- 172.21.15.12:0/3481342428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11d409fbd0 0x7f11d406d090 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:01.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.830+0000 7f11da12b640 1 --2- 172.21.15.12:0/3481342428 >> v2:172.21.15.71:3300/0 conn(0x7f11d40a2db0 0x7f11d406cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:01.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.830+0000 7f11da12b640 1 -- 172.21.15.12:0/3481342428 >> 172.21.15.12:0/3481342428 conn(0x7f11d40a3ea0 msgr2=0x7f11d40a42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:01.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.830+0000 7f11da12b640 1 -- 172.21.15.12:0/3481342428 shutdown_connections 2024-09-11T01:52:01.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.830+0000 7f11da12b640 1 -- 172.21.15.12:0/3481342428 wait complete. 2024-09-11T01:52:01.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.830+0000 7f11da12b640 10 monclient: build_initial_monmap 2024-09-11T01:52:01.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.830+0000 7f11da12b640 10 monclient: monmap: 2024-09-11T01:52:01.849 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:01.849 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:01.849 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:01.849 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:01.849 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:01.849 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:01.849 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:01.849 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:01.849 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:01.849 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:01.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11da12b640 1 Processor -- start 2024-09-11T01:52:01.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11da12b640 1 -- start start 2024-09-11T01:52:01.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11da12b640 10 monclient: init 2024-09-11T01:52:01.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11da12b640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:01.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11da12b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:52:01.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11da12b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11d406d5d0 0x7f11d4098f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:01.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11da12b640 10 monclient(hunting): picked mon.noname-c con 0x7f11d406d5d0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:01.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11da12b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f11d409fbd0 0x7f11d4093a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:01.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11da12b640 10 monclient(hunting): picked mon.noname-b con 0x7f11d409fbd0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:01.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11da12b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11d40a2db0 0x7f11d4093f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:01.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11da12b640 10 monclient(hunting): picked mon.noname-a con 0x7f11d40a2db0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:01.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11da12b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:01.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11da12b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f11d409ea20 con 0x7f11d40a2db0 2024-09-11T01:52:01.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11da12b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:01.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11da12b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f11d409e8a0 con 0x7f11d409fbd0 2024-09-11T01:52:01.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11da12b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:01.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11da12b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f11d413fc70 con 0x7f11d406d5d0 2024-09-11T01:52:01.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11da12b640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:01.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11da12b640 10 monclient(hunting): authenticate will time out at 2128.273838s 2024-09-11T01:52:01.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11d37fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11d406d5d0 0x7f11d4098f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:01.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11d3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11d40a2db0 0x7f11d4093f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:01.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11d37fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11d406d5d0 0x7f11d4098f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55696/0 (socket says 172.21.15.12:55696) 2024-09-11T01:52:01.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11d37fe640 1 -- 172.21.15.12:0/910803755 learned_addr learned my addr 172.21.15.12:0/910803755 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:01.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11d37fe640 10 monclient(hunting): get_auth_request con 0x7f11d406d5d0 auth_method 0 2024-09-11T01:52:01.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:01.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:01.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.831+0000 7f11d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:01.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d3fff640 10 monclient(hunting): get_auth_request con 0x7f11d40a2db0 auth_method 0 2024-09-11T01:52:01.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:01.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:01.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:01.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:01.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:01.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:01.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:01.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:01.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:01.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d2ffd640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f11d409fbd0 0x7f11d4093a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:01.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d2ffd640 10 monclient(hunting): get_auth_request con 0x7f11d409fbd0 auth_method 0 2024-09-11T01:52:01.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:01.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:01.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:01.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:01.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:01.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:01.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d37fe640 10 monclient(hunting): handle_auth_done global_id 244232 payload 824 2024-09-11T01:52:01.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d37fe640 1 -- 172.21.15.12:0/910803755 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f11d409fbd0 msgr2=0x7f11d4093a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:01.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d37fe640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f11d409fbd0 0x7f11d4093a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:01.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d37fe640 1 -- 172.21.15.12:0/910803755 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11d40a2db0 msgr2=0x7f11d4093f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:01.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d37fe640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11d40a2db0 0x7f11d4093f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:01.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d37fe640 10 monclient: _finish_hunting 0 2024-09-11T01:52:01.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d37fe640 1 monclient: found mon.noname-c 2024-09-11T01:52:01.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:01.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:52:01.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d37fe640 1 -- 172.21.15.12:0/910803755 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d413fa50 con 0x7f11d406d5d0 2024-09-11T01:52:01.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.832+0000 7f11d3fff640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11d40a2db0 0x7f11d4093f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:52:01.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.834+0000 7f11d37fe640 1 -- 172.21.15.12:0/910803755 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11d406d5d0 msgr2=0x7f11d4098f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:01.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.834+0000 7f11d37fe640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11d406d5d0 0x7f11d4098f90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f11c400d830 tx=0x7f11c400dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:01.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.834+0000 7f11cbfff640 1 -- 172.21.15.12:0/910803755 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c400b8d0 con 0x7f11d406d5d0 2024-09-11T01:52:01.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.834+0000 7f11cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:01.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.834+0000 7f11cbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:01.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.834+0000 7f11cbfff640 10 monclient: dump: 2024-09-11T01:52:01.861 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:01.861 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:01.861 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:01.861 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:01.861 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:01.861 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:01.862 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:01.862 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:01.862 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:01.862 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:01.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.834+0000 7f11cbfff640 10 monclient: _finish_auth 0 2024-09-11T01:52:01.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.834+0000 7f11cbfff640 10 monclient: _check_auth_tickets 2024-09-11T01:52:01.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.834+0000 7f11cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:01.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.834+0000 7f11cbfff640 1 -- 172.21.15.12:0/910803755 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11c4019070 con 0x7f11d406d5d0 2024-09-11T01:52:01.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.834+0000 7f11da12b640 5 monclient: authenticate success, global_id 244232 2024-09-11T01:52:01.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.834+0000 7f11cbfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:01.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.834+0000 7f11cbfff640 1 -- 172.21.15.12:0/910803755 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c400bc90 con 0x7f11d406d5d0 2024-09-11T01:52:01.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.834+0000 7f11da12b640 10 monclient: _renew_subs 2024-09-11T01:52:01.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.834+0000 7f11da12b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:01.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.834+0000 7f11da12b640 1 -- 172.21.15.12:0/910803755 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11d409e590 con 0x7f11d406d5d0 2024-09-11T01:52:01.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.834+0000 7f11cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:01.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.834+0000 7f11d1ffb640 4 set_mon_vals no callback set 2024-09-11T01:52:01.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.834+0000 7f11cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:01.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.834+0000 7f11cbfff640 10 monclient: dump: 2024-09-11T01:52:01.863 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:01.863 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:01.863 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:01.863 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:01.863 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:01.863 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:01.864 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:01.864 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:01.864 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:01.864 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:01.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.834+0000 7f11da12b640 10 monclient: _renew_subs 2024-09-11T01:52:01.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.834+0000 7f11da12b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:01.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.834+0000 7f11da12b640 1 -- 172.21.15.12:0/910803755 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f11d4097830 con 0x7f11d406d5d0 2024-09-11T01:52:01.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.835+0000 7f11cbfff640 1 -- 172.21.15.12:0/910803755 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f11c4013070 con 0x7f11d406d5d0 2024-09-11T01:52:01.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.835+0000 7f11da12b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:01.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.835+0000 7f11da12b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:01.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.835+0000 7f11da12b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:01.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.835+0000 7f11da12b640 1 -- 172.21.15.12:0/910803755 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11d406cb50 con 0x7f11d406d5d0 2024-09-11T01:52:01.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d37fe640 1 -- 172.21.15.12:0/910803755 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11d406d5d0 msgr2=0x7f11d4098f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:52:01.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d37fe640 1 -- 172.21.15.12:0/910803755 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11d406d5d0 msgr2=0x7f11d4098f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:52:01.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d37fe640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11d406d5d0 0x7f11d4098f90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f11c400d830 tx=0x7f11c400dcf0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:52:01.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d37fe640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11d406d5d0 0x7f11d4098f90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f11c400d830 tx=0x7f11c400dcf0 comp rx=0 tx=0).stop 2024-09-11T01:52:01.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11cbfff640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f11b403e240 0x7f11b4040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:01.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11cbfff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:01.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11cbfff640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:01.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11cbfff640 1 -- 172.21.15.12:0/910803755 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11d406d5d0 msgr2=0x7f11d4098f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:01.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11cbfff640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11d406d5d0 0x7f11d4098f90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:01.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11cbfff640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:52:01.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11cbfff640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11b4040e10 0x7f11b4043200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:01.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11cbfff640 10 monclient(hunting): picked mon.c con 0x7f11b4040e10 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:01.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11cbfff640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f11d409fbd0 0x7f11b4045bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:01.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11cbfff640 10 monclient(hunting): picked mon.b con 0x7f11d409fbd0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:01.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11cbfff640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11d40a2db0 0x7f11b4048570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:01.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11cbfff640 10 monclient(hunting): picked mon.a con 0x7f11d40a2db0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:01.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11cbfff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:01.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11cbfff640 1 -- 172.21.15.12:0/910803755 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f11b4048f80 con 0x7f11d40a2db0 2024-09-11T01:52:01.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11cbfff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:01.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11cbfff640 1 -- 172.21.15.12:0/910803755 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f11b4049100 con 0x7f11d409fbd0 2024-09-11T01:52:01.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11cbfff640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:01.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11cbfff640 1 -- 172.21.15.12:0/910803755 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f11b4049280 con 0x7f11b4040e10 2024-09-11T01:52:01.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11cbfff640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:01.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d2ffd640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11b4040e10 0x7f11b4043200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:01.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d37fe640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f11b403e240 0x7f11b4040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:01.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d2ffd640 10 monclient(hunting): get_auth_request con 0x7f11b4040e10 auth_method 0 2024-09-11T01:52:01.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:01.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:01.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:52:01.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d37fe640 10 monclient(hunting): get_auth_request con 0x7f11b403e240 auth_method 0 2024-09-11T01:52:01.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d3fff640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f11d409fbd0 0x7f11b4045bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:01.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d3fff640 10 monclient(hunting): get_auth_request con 0x7f11d409fbd0 auth_method 0 2024-09-11T01:52:01.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:01.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:01.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:52:01.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d37fe640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11d40a2db0 0x7f11b4048570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:01.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d37fe640 10 monclient(hunting): get_auth_request con 0x7f11d40a2db0 auth_method 0 2024-09-11T01:52:01.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:01.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:01.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:52:01.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:01.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:01.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:52:01.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:01.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:01.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:52:01.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:01.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:01.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:52:01.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d2ffd640 10 monclient(hunting): handle_auth_done global_id 244232 payload 293 2024-09-11T01:52:01.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d2ffd640 1 -- 172.21.15.12:0/910803755 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f11d409fbd0 msgr2=0x7f11b4045bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:01.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d2ffd640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f11d409fbd0 0x7f11b4045bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:01.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d2ffd640 1 -- 172.21.15.12:0/910803755 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11d40a2db0 msgr2=0x7f11b4048570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:01.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d2ffd640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11d40a2db0 0x7f11b4048570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:01.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d2ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:52:01.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d2ffd640 1 monclient: found mon.c 2024-09-11T01:52:01.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:01.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d2ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:01.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d2ffd640 1 -- 172.21.15.12:0/910803755 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f11b4049400 con 0x7f11b4040e10 2024-09-11T01:52:01.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d2ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:01.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d2ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:01.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d2ffd640 1 -- 172.21.15.12:0/910803755 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11d409f180 con 0x7f11b4040e10 2024-09-11T01:52:01.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d2ffd640 10 monclient: _finish_auth 0 2024-09-11T01:52:01.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d2ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:52:01.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:01.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.836+0000 7f11d3fff640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f11d409fbd0 0x7f11b4045bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:52:01.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.838+0000 7f11d37fe640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f11b403e240 0x7f11b4040700 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f11c401ffd0 tx=0x7f11c4011a40 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:01.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.839+0000 7f11d2ffd640 1 -- 172.21.15.12:0/910803755 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11b4040e10 msgr2=0x7f11b4043200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:01.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.839+0000 7f11d2ffd640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11b4040e10 0x7f11b4043200 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f11bc009630 tx=0x7f11c4008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:01.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.841+0000 7f11cbfff640 1 -- 172.21.15.12:0/910803755 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11bc037040 con 0x7f11b4040e10 2024-09-11T01:52:01.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.841+0000 7f11cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:01.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.841+0000 7f11cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:01.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.841+0000 7f11cbfff640 10 monclient: dump: 2024-09-11T01:52:01.871 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:01.871 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:01.871 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:01.871 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:01.871 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:01.871 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:01.871 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:01.871 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:01.871 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:01.872 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:01.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.841+0000 7f11cbfff640 1 -- 172.21.15.12:0/910803755 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11bc0328c0 con 0x7f11b4040e10 2024-09-11T01:52:01.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.841+0000 7f11cbfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:01.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.841+0000 7f11d17fa640 4 set_mon_vals no callback set 2024-09-11T01:52:01.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.841+0000 7f11cbfff640 1 -- 172.21.15.12:0/910803755 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f11bc03c440 con 0x7f11b4040e10 2024-09-11T01:52:01.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.842+0000 7f11cbfff640 1 -- 172.21.15.12:0/910803755 <== mon.2 v2:172.21.15.12:3301/0 4 ==== osd_map(765..765 src has 676..765) ==== 4831+0+0 (secure 0 0 0) 0x7f11bc0713d0 con 0x7f11b4040e10 2024-09-11T01:52:01.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.845+0000 7f11cbfff640 1 -- 172.21.15.12:0/910803755 <== mon.2 v2:172.21.15.12:3301/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f11bc02ae20 con 0x7f11b4040e10 2024-09-11T01:52:01.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.845+0000 7f11cbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:01.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:01.845+0000 7f11cbfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.100+0000 7f11da12b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:52:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.100+0000 7f11da12b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:52:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.100+0000 7f11da12b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.100+0000 7f11da12b640 1 -- 172.21.15.12:0/910803755 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f11d406e570 con 0x7f11b4040e10 2024-09-11T01:52:02.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.100+0000 7f11cbfff640 1 -- 172.21.15.12:0/910803755 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f11bc071080 con 0x7f11b4040e10 2024-09-11T01:52:02.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.100+0000 7f11cbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:52:02.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.100+0000 7f11cbfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:02.103 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":214,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:52:02.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.102+0000 7f11da12b640 1 -- 172.21.15.12:0/910803755 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f11b403e240 msgr2=0x7f11b4040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:02.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.102+0000 7f11da12b640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f11b403e240 0x7f11b4040700 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f11c401ffd0 tx=0x7f11c4011a40 comp rx=0 tx=0).stop 2024-09-11T01:52:02.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.102+0000 7f11da12b640 10 monclient: shutdown 2024-09-11T01:52:02.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.102+0000 7f11da12b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:02.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.102+0000 7f11da12b640 1 -- 172.21.15.12:0/910803755 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11b4040e10 msgr2=0x7f11b4043200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:02.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.102+0000 7f11da12b640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11b4040e10 0x7f11b4043200 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f11bc009630 tx=0x7f11c4008040 comp rx=0 tx=0).stop 2024-09-11T01:52:02.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.102+0000 7f11da12b640 1 -- 172.21.15.12:0/910803755 shutdown_connections 2024-09-11T01:52:02.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.102+0000 7f11da12b640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f11d409fbd0 0x7f11b4045bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.102+0000 7f11da12b640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f11b4040e10 0x7f11b4043200 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.102+0000 7f11da12b640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f11d40a2db0 0x7f11b4048570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.102+0000 7f11da12b640 1 --2- 172.21.15.12:0/910803755 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f11b403e240 0x7f11b4040700 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.102+0000 7f11da12b640 1 -- 172.21.15.12:0/910803755 >> 172.21.15.12:0/910803755 conn(0x7f11d40a3ea0 msgr2=0x7f11d409d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:02.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.103+0000 7f11da12b640 1 -- 172.21.15.12:0/910803755 shutdown_connections 2024-09-11T01:52:02.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.103+0000 7f11da12b640 1 -- 172.21.15.12:0/910803755 wait complete. 2024-09-11T01:52:02.114 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:52:02.115 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:52:02.115 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:52:02.115 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:52:02.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.215+0000 7f0c65f82640 10 monclient: get_monmap_and_config 2024-09-11T01:52:02.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.215+0000 7f0c65f82640 10 monclient: build_initial_monmap 2024-09-11T01:52:02.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.215+0000 7f0c65f82640 10 monclient: monmap: 2024-09-11T01:52:02.217 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:02.217 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:02.217 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:02.215910+0000 2024-09-11T01:52:02.217 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:02.215910+0000 2024-09-11T01:52:02.217 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:02.217 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.217 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:02.217 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:02.217 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:02.217 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:02.217 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:02.218 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.215+0000 7f0c65f82640 1 Processor -- start 2024-09-11T01:52:02.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.215+0000 7f0c65f82640 1 -- start start 2024-09-11T01:52:02.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.215+0000 7f0c65f82640 10 monclient: init 2024-09-11T01:52:02.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c65f82640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:02.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c65f82640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-11T01:52:02.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c65f82640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c600a03a0 0x7f0c6009f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c65f82640 10 monclient(hunting): picked mon.noname-b con 0x7f0c600a03a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:02.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c65f82640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0c600a2530 addr v1:172.21.15.12:6789/0 2024-09-11T01:52:02.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c65f82640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0c600a13a0 0x7f0c6013baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c65f82640 10 monclient(hunting): picked mon.noname-c con 0x7f0c600a13a0 addr v2:172.21.15.71:3300/0 2024-09-11T01:52:02.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c65f82640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c6009e510 con 0x7f0c600a2530 2024-09-11T01:52:02.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c65f82640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c65f82640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f0c600a29b0 con 0x7f0c600a13a0 2024-09-11T01:52:02.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c65f82640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c65f82640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f0c600a0780 con 0x7f0c600a03a0 2024-09-11T01:52:02.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c65f82640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:02.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c65f82640 10 monclient(hunting): authenticate will time out at 2128.658537s 2024-09-11T01:52:02.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5f7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c600a03a0 0x7f0c6009f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5effd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0c600a2530 0x7f0c600a0c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37088/0 (socket says 172.21.15.12:37088) 2024-09-11T01:52:02.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5effd640 1 -- 172.21.15.12:0/57894115 learned_addr learned my addr 172.21.15.12:0/57894115 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:52:02.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5f7fe640 10 monclient(hunting): get_auth_request con 0x7f0c600a03a0 auth_method 0 2024-09-11T01:52:02.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:02.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:02.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:02.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5ffff640 1 --2- 172.21.15.12:0/57894115 >> v2:172.21.15.71:3300/0 conn(0x7f0c600a13a0 0x7f0c6013baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:02.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:02.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:02.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5ffff640 10 monclient(hunting): get_auth_request con 0x7f0c600a13a0 auth_method 0 2024-09-11T01:52:02.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:02.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:02.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:02.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:02.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:02.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:02.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5f7fe640 10 monclient(hunting): handle_auth_done global_id 244235 payload 274 2024-09-11T01:52:02.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5f7fe640 1 -- 172.21.15.12:0/57894115 >> v2:172.21.15.71:3300/0 conn(0x7f0c600a13a0 msgr2=0x7f0c6013baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:02.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5f7fe640 1 --2- 172.21.15.12:0/57894115 >> v2:172.21.15.71:3300/0 conn(0x7f0c600a13a0 0x7f0c6013baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5f7fe640 1 -- 172.21.15.12:0/57894115 >> v1:172.21.15.12:6789/0 conn(0x7f0c600a2530 legacy=0x7f0c600a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:02.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5f7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:52:02.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5f7fe640 1 monclient: found mon.noname-b 2024-09-11T01:52:02.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:02.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:52:02.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.216+0000 7f0c5f7fe640 1 -- 172.21.15.12:0/57894115 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c6009edb0 con 0x7f0c600a03a0 2024-09-11T01:52:02.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.219+0000 7f0c5f7fe640 1 -- 172.21.15.12:0/57894115 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c600a03a0 msgr2=0x7f0c6009f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:02.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.219+0000 7f0c5f7fe640 1 --2- 172.21.15.12:0/57894115 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c600a03a0 0x7f0c6009f1d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0c50009d90 tx=0x7f0c5002a490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37c94ba7a1e79316 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:02.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.219+0000 7f0c5e7fc640 1 -- 172.21.15.12:0/57894115 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c50007950 con 0x7f0c600a03a0 2024-09-11T01:52:02.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.219+0000 7f0c5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:02.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.219+0000 7f0c5e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:52:02.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.219+0000 7f0c5e7fc640 10 monclient: dump: 2024-09-11T01:52:02.226 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:02.226 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:02.226 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.226 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.226 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:02.226 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.226 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:02.227 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:02.227 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:02.227 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.219+0000 7f0c5e7fc640 10 monclient: _finish_auth 0 2024-09-11T01:52:02.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.219+0000 7f0c5e7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:52:02.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.219+0000 7f0c5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:02.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.219+0000 7f0c5e7fc640 1 -- 172.21.15.12:0/57894115 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c50007af0 con 0x7f0c600a03a0 2024-09-11T01:52:02.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.219+0000 7f0c65f82640 5 monclient: authenticate success, global_id 244235 2024-09-11T01:52:02.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.219+0000 7f0c65f82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:52:02.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.220+0000 7f0c5e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:02.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.220+0000 7f0c65f82640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:02.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.220+0000 7f0c5e7fc640 1 -- 172.21.15.12:0/57894115 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c50031400 con 0x7f0c600a03a0 2024-09-11T01:52:02.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.220+0000 7f0c65f82640 4 set_mon_vals no callback set 2024-09-11T01:52:02.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.220+0000 7f0c5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:02.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.220+0000 7f0c5e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:02.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.220+0000 7f0c5e7fc640 10 monclient: dump: 2024-09-11T01:52:02.228 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:02.228 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:02.228 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.228 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.228 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:02.228 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.228 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:02.228 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:02.228 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:02.228 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.220+0000 7f0c65f82640 10 monclient: shutdown 2024-09-11T01:52:02.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.220+0000 7f0c65f82640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:02.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.220+0000 7f0c65f82640 1 -- 172.21.15.12:0/57894115 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c600a03a0 msgr2=0x7f0c6009f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:02.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.220+0000 7f0c65f82640 1 --2- 172.21.15.12:0/57894115 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c600a03a0 0x7f0c6009f1d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0c50009d90 tx=0x7f0c5002a490 comp rx=0 tx=0).stop 2024-09-11T01:52:02.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.220+0000 7f0c65f82640 1 -- 172.21.15.12:0/57894115 shutdown_connections 2024-09-11T01:52:02.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.220+0000 7f0c65f82640 1 --2- 172.21.15.12:0/57894115 >> v2:172.21.15.71:3300/0 conn(0x7f0c600a13a0 0x7f0c6013baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.220+0000 7f0c65f82640 1 --2- 172.21.15.12:0/57894115 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c600a03a0 0x7f0c6009f1d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.220+0000 7f0c65f82640 1 -- 172.21.15.12:0/57894115 >> 172.21.15.12:0/57894115 conn(0x7f0c600a2f70 msgr2=0x7f0c600a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:02.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.220+0000 7f0c65f82640 1 -- 172.21.15.12:0/57894115 shutdown_connections 2024-09-11T01:52:02.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.220+0000 7f0c65f82640 1 -- 172.21.15.12:0/57894115 wait complete. 2024-09-11T01:52:02.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.221+0000 7f0c65f82640 10 monclient: build_initial_monmap 2024-09-11T01:52:02.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.221+0000 7f0c65f82640 10 monclient: monmap: 2024-09-11T01:52:02.230 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:02.230 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:02.230 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:02.230 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:02.230 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:02.230 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.230 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:02.230 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:02.230 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:02.230 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.221+0000 7f0c65f82640 1 Processor -- start 2024-09-11T01:52:02.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.221+0000 7f0c65f82640 1 -- start start 2024-09-11T01:52:02.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.221+0000 7f0c65f82640 10 monclient: init 2024-09-11T01:52:02.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c65f82640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:02.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c65f82640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:52:02.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c65f82640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c600a03a0 0x7f0c60125d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c65f82640 10 monclient(hunting): picked mon.noname-b con 0x7f0c600a03a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:02.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c65f82640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c600a13a0 0x7f0c6013cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c65f82640 10 monclient(hunting): picked mon.noname-a con 0x7f0c600a13a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c65f82640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c600a2530 0x7f0c601237a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c65f82640 10 monclient(hunting): picked mon.noname-c con 0x7f0c600a2530 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c65f82640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c65f82640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6009ca30 con 0x7f0c600a13a0 2024-09-11T01:52:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c65f82640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c65f82640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0c600a19e0 con 0x7f0c600a03a0 2024-09-11T01:52:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c65f82640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c65f82640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f0c6009cbb0 con 0x7f0c600a2530 2024-09-11T01:52:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c65f82640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c65f82640 10 monclient(hunting): authenticate will time out at 2128.664119s 2024-09-11T01:52:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5effd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c600a13a0 0x7f0c6013cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c600a2530 0x7f0c601237a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5effd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c600a13a0 0x7f0c6013cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38022/0 (socket says 172.21.15.12:38022) 2024-09-11T01:52:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5effd640 1 -- 172.21.15.12:0/4290351880 learned_addr learned my addr 172.21.15.12:0/4290351880 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5effd640 10 monclient(hunting): get_auth_request con 0x7f0c600a13a0 auth_method 0 2024-09-11T01:52:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5ffff640 1 --2- 172.21.15.12:0/4290351880 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c600a2530 0x7f0c601237a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55728/0 (socket says 172.21.15.12:55728) 2024-09-11T01:52:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5ffff640 10 monclient(hunting): get_auth_request con 0x7f0c600a2530 auth_method 0 2024-09-11T01:52:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:02.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:02.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:02.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5f7fe640 1 --2- 172.21.15.12:0/4290351880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c600a03a0 0x7f0c60125d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5f7fe640 10 monclient(hunting): get_auth_request con 0x7f0c600a03a0 auth_method 0 2024-09-11T01:52:02.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:02.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:02.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:02.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:02.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:02.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:02.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:02.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:02.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:02.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5ffff640 10 monclient(hunting): handle_auth_done global_id 244238 payload 824 2024-09-11T01:52:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5ffff640 1 -- 172.21.15.12:0/4290351880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c600a03a0 msgr2=0x7f0c60125d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5ffff640 1 --2- 172.21.15.12:0/4290351880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c600a03a0 0x7f0c60125d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5ffff640 1 -- 172.21.15.12:0/4290351880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c600a13a0 msgr2=0x7f0c6013cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5ffff640 1 --2- 172.21.15.12:0/4290351880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c600a13a0 0x7f0c6013cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:52:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5ffff640 1 monclient: found mon.noname-c 2024-09-11T01:52:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:52:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.222+0000 7f0c5ffff640 1 -- 172.21.15.12:0/4290351880 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c601cbcb0 con 0x7f0c600a2530 2024-09-11T01:52:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.224+0000 7f0c5ffff640 1 -- 172.21.15.12:0/4290351880 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c600a2530 msgr2=0x7f0c601237a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.224+0000 7f0c5ffff640 1 --2- 172.21.15.12:0/4290351880 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c600a2530 0x7f0c601237a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0c5800e820 tx=0x7f0c5800ece0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.224+0000 7f0c57fff640 1 -- 172.21.15.12:0/4290351880 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c58017070 con 0x7f0c600a2530 2024-09-11T01:52:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.224+0000 7f0c57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.224+0000 7f0c57fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.224+0000 7f0c57fff640 10 monclient: dump: 2024-09-11T01:52:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:02.241 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.224+0000 7f0c57fff640 10 monclient: _finish_auth 0 2024-09-11T01:52:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.224+0000 7f0c57fff640 10 monclient: _check_auth_tickets 2024-09-11T01:52:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.224+0000 7f0c57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.224+0000 7f0c65f82640 5 monclient: authenticate success, global_id 244238 2024-09-11T01:52:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.224+0000 7f0c57fff640 1 -- 172.21.15.12:0/4290351880 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c5800cb50 con 0x7f0c600a2530 2024-09-11T01:52:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.224+0000 7f0c57fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.224+0000 7f0c57fff640 1 -- 172.21.15.12:0/4290351880 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c58013470 con 0x7f0c600a2530 2024-09-11T01:52:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.224+0000 7f0c57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.224+0000 7f0c57fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.224+0000 7f0c5dffb640 4 set_mon_vals no callback set 2024-09-11T01:52:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.224+0000 7f0c57fff640 10 monclient: dump: 2024-09-11T01:52:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:02.242 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.224+0000 7f0c65f82640 10 monclient: _renew_subs 2024-09-11T01:52:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.224+0000 7f0c65f82640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.224+0000 7f0c65f82640 1 -- 172.21.15.12:0/4290351880 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c60127530 con 0x7f0c600a2530 2024-09-11T01:52:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.225+0000 7f0c65f82640 10 monclient: _renew_subs 2024-09-11T01:52:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.225+0000 7f0c65f82640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:02.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.225+0000 7f0c65f82640 1 -- 172.21.15.12:0/4290351880 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c601266c0 con 0x7f0c600a2530 2024-09-11T01:52:02.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.225+0000 7f0c65f82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:02.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.225+0000 7f0c65f82640 1 --2- 172.21.15.12:0/4290351880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c6002d440 0x7f0c6002d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.225+0000 7f0c65f82640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.225+0000 7f0c65f82640 1 -- 172.21.15.12:0/4290351880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6009d010 con 0x7f0c6002d440 2024-09-11T01:52:02.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.225+0000 7f0c57fff640 1 -- 172.21.15.12:0/4290351880 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f0c58013650 con 0x7f0c600a2530 2024-09-11T01:52:02.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.226+0000 7f0c5f7fe640 1 --2- 172.21.15.12:0/4290351880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c6002d440 0x7f0c6002d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.226+0000 7f0c5f7fe640 10 monclient: get_auth_request con 0x7f0c6002d440 auth_method 0 2024-09-11T01:52:02.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.226+0000 7f0c5f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:02.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.226+0000 7f0c5f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:02.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.226+0000 7f0c5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:02.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.226+0000 7f0c5f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:02.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.226+0000 7f0c5f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:02.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.226+0000 7f0c5f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:02.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.226+0000 7f0c5f7fe640 10 monclient(hunting): handle_auth_done global_id 254142 payload 824 2024-09-11T01:52:02.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.226+0000 7f0c57fff640 1 --2- 172.21.15.12:0/4290351880 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0c4003e260 0x7f0c40040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.226+0000 7f0c5f7fe640 1 -- 172.21.15.12:0/4290351880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0c601251d0 con 0x7f0c6002d440 2024-09-11T01:52:02.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.226+0000 7f0c57fff640 1 -- 172.21.15.12:0/4290351880 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(765..765 src has 676..765) ==== 4831+0+0 (secure 0 0 0) 0x7f0c58052c90 con 0x7f0c600a2530 2024-09-11T01:52:02.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.226+0000 7f0c5effd640 1 --2- 172.21.15.12:0/4290351880 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0c4003e260 0x7f0c40040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.226+0000 7f0c5effd640 10 monclient: get_auth_request con 0x7f0c4003e260 auth_method 0 2024-09-11T01:52:02.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.228+0000 7f0c5f7fe640 1 -- 172.21.15.12:0/4290351880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c6002d440 msgr2=0x7f0c6002d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:02.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.228+0000 7f0c5f7fe640 1 --2- 172.21.15.12:0/4290351880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c6002d440 0x7f0c6002d820 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0c50034000 tx=0x7f0c50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:02.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.228+0000 7f0c57fff640 1 -- 172.21.15.12:0/4290351880 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5004a070 con 0x7f0c6002d440 2024-09-11T01:52:02.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.228+0000 7f0c57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:02.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.228+0000 7f0c57fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:52:02.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.228+0000 7f0c57fff640 10 monclient: dump: 2024-09-11T01:52:02.244 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:02.245 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:02.245 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.245 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.245 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:02.245 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.245 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:02.245 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:02.245 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:02.245 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.228+0000 7f0c5effd640 1 --2- 172.21.15.12:0/4290351880 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0c4003e260 0x7f0c40040720 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f0c48007d10 tx=0x7f0c48007c60 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:02.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.229+0000 7f0c57fff640 1 -- 172.21.15.12:0/4290351880 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0c50007e80 con 0x7f0c6002d440 2024-09-11T01:52:02.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.229+0000 7f0c57fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:02.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.229+0000 7f0c57fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:02.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.229+0000 7f0c57fff640 1 -- 172.21.15.12:0/4290351880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c6002d440 msgr2=0x7f0c6002d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:02.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.229+0000 7f0c57fff640 1 --2- 172.21.15.12:0/4290351880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c6002d440 0x7f0c6002d820 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0c50034000 tx=0x7f0c50004a40 comp rx=0 tx=0).stop 2024-09-11T01:52:02.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.229+0000 7f0c57fff640 1 -- 172.21.15.12:0/4290351880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c6002d440 msgr2=0x7f0c6002d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:02.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.229+0000 7f0c57fff640 1 --2- 172.21.15.12:0/4290351880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c6002d440 0x7f0c6002d820 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0c50034000 tx=0x7f0c50004a40 comp rx=0 tx=0).stop 2024-09-11T01:52:02.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.235+0000 7f0c65f82640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:52:02.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.235+0000 7f0c65f82640 1 --2- 172.21.15.12:0/4290351880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c6002dd60 0x7f0c6002e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.235+0000 7f0c65f82640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.235+0000 7f0c65f82640 1 -- 172.21.15.12:0/4290351880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0c60120000 con 0x7f0c6002dd60 2024-09-11T01:52:02.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.235+0000 7f0c5f7fe640 1 --2- 172.21.15.12:0/4290351880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c6002dd60 0x7f0c6002e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.235+0000 7f0c5f7fe640 10 monclient: get_auth_request con 0x7f0c6002dd60 auth_method 0 2024-09-11T01:52:02.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.235+0000 7f0c5f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:02.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.235+0000 7f0c5f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:02.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.235+0000 7f0c5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:02.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.235+0000 7f0c5f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:02.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.235+0000 7f0c5f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:02.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.235+0000 7f0c5f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:02.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.235+0000 7f0c5f7fe640 10 monclient(hunting): handle_auth_done global_id 254145 payload 824 2024-09-11T01:52:02.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.235+0000 7f0c5f7fe640 1 -- 172.21.15.12:0/4290351880 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0c6006d480 con 0x7f0c6002dd60 2024-09-11T01:52:02.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.237+0000 7f0c5f7fe640 1 -- 172.21.15.12:0/4290351880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c6002dd60 msgr2=0x7f0c6002e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:02.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.237+0000 7f0c5f7fe640 1 --2- 172.21.15.12:0/4290351880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c6002dd60 0x7f0c6002e140 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0c60073b00 tx=0x7f0c50030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:02.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.238+0000 7f0c57fff640 1 -- 172.21.15.12:0/4290351880 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c50050070 con 0x7f0c6002dd60 2024-09-11T01:52:02.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.238+0000 7f0c57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:02.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.238+0000 7f0c57fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:52:02.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.238+0000 7f0c57fff640 10 monclient: dump: 2024-09-11T01:52:02.250 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:02.250 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:02.250 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.250 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.250 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:02.250 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.250 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:02.250 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:02.250 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:02.250 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.238+0000 7f0c57fff640 1 -- 172.21.15.12:0/4290351880 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4542 (secure 0 0 0) 0x7f0c5004b040 con 0x7f0c6002dd60 2024-09-11T01:52:02.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.238+0000 7f0c57fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:52:02.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.238+0000 7f0c57fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:02.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.238+0000 7f0c57fff640 1 -- 172.21.15.12:0/4290351880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c6002dd60 msgr2=0x7f0c6002e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:02.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.238+0000 7f0c57fff640 1 --2- 172.21.15.12:0/4290351880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c6002dd60 0x7f0c6002e140 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0c60073b00 tx=0x7f0c50030000 comp rx=0 tx=0).stop 2024-09-11T01:52:02.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.238+0000 7f0c57fff640 1 -- 172.21.15.12:0/4290351880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c6002dd60 msgr2=0x7f0c6002e140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:02.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.238+0000 7f0c57fff640 1 --2- 172.21.15.12:0/4290351880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c6002dd60 0x7f0c6002e140 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0c60073b00 tx=0x7f0c50030000 comp rx=0 tx=0).stop 2024-09-11T01:52:02.251 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:52:02.251 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:52:02.251 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:52:02.251 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:52:02.251 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 214, 2024-09-11T01:52:02.251 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 9516, 2024-09-11T01:52:02.251 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:52:02.251 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:52:02.252 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:52:02.252 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:52:02.252 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:02.252 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:52:02.252 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:52:02.252 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:52:02.252 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:52:02.252 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:02.252 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:02.252 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:02.252 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:02.252 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:02.252 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:02.253 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:02.253 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:02.253 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:02.253 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:02.253 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:02.253 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:02.253 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:52:02.253 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:52:02.253 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:02.253 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:02.253 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:02.253 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:02.253 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:02.253 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:02.254 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:02.254 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:02.254 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:02.254 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:02.254 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:02.254 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:02.254 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.254 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:52:02.254 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:52:02.254 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:52:02.254 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:52:02.254 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:52:02.254 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:52:02.254 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:52:02.255 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:52:02.255 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:52:02.255 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:52:02.255 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:52:02.255 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:52:02.255 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:52:02.255 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:52:02.255 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:52:02.255 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:52:02.255 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:52:02.255 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:02.255 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:02.255 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:02.255 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:02.256 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:02.256 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:02.256 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:02.256 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:02.256 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:02.256 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:02.256 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:02.256 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:02.256 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:52:02.256 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.256 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:52:02.256 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.256 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:52:02.256 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:52:02.257 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:02.257 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:02.257 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.257 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:02.257 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:52:02.257 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:02.257 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.257 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.257 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:02.257 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:52:02.257 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:02.257 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:02.257 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:02.257 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.258 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:52:02.258 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:52:02.258 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:02.258 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:02.258 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:02.258 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.258 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.258 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:52:02.258 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:52:02.258 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:02.258 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:02.258 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.258 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:02.258 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:52:02.259 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:02.259 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.259 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.259 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:02.259 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:52:02.259 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:02.259 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:02.259 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:02.259 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.259 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:52:02.259 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:52:02.259 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:02.259 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:02.259 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:02.260 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.260 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.260 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:52:02.260 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:52:02.260 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:02.260 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:02.260 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.260 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:02.260 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:52:02.260 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:02.260 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.260 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.261 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:02.261 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:52:02.261 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:02.261 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:02.261 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:02.261 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.261 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:52:02.261 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:52:02.261 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:02.261 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:02.261 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:02.261 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:02.261 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:02.262 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.262 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:52:02.262 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:52:02.262 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.262 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:02.262 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:02.262 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:52:02.262 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:02.262 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:02.262 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:52:02.262 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.262 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:02.262 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:02.263 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:52:02.263 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:02.263 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:02.263 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.263 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:52:02.263 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:52:02.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.238+0000 7f0c65f82640 1 -- 172.21.15.12:0/4290351880 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0c4003e260 msgr2=0x7f0c40040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:02.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.238+0000 7f0c65f82640 1 --2- 172.21.15.12:0/4290351880 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0c4003e260 0x7f0c40040720 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f0c48007d10 tx=0x7f0c48007c60 comp rx=0 tx=0).stop 2024-09-11T01:52:02.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.238+0000 7f0c65f82640 10 monclient: shutdown 2024-09-11T01:52:02.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.238+0000 7f0c65f82640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:02.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.238+0000 7f0c65f82640 1 -- 172.21.15.12:0/4290351880 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c600a2530 msgr2=0x7f0c601237a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:02.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.238+0000 7f0c5ffff640 1 -- 172.21.15.12:0/4290351880 reap_dead start 2024-09-11T01:52:02.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.238+0000 7f0c65f82640 1 --2- 172.21.15.12:0/4290351880 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c600a2530 0x7f0c601237a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0c5800e820 tx=0x7f0c5800ece0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.238+0000 7f0c65f82640 1 -- 172.21.15.12:0/4290351880 shutdown_connections 2024-09-11T01:52:02.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.238+0000 7f0c65f82640 1 --2- 172.21.15.12:0/4290351880 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c600a2530 0x7f0c601237a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.238+0000 7f0c65f82640 1 -- 172.21.15.12:0/4290351880 >> 172.21.15.12:0/4290351880 conn(0x7f0c600a2f70 msgr2=0x7f0c60072e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:02.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.238+0000 7f0c65f82640 1 -- 172.21.15.12:0/4290351880 shutdown_connections 2024-09-11T01:52:02.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.238+0000 7f0c65f82640 1 -- 172.21.15.12:0/4290351880 wait complete. 2024-09-11T01:52:02.269 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:52:02.269 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:52:02.269 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:52:02.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.369+0000 7f1de45ca640 10 monclient: get_monmap_and_config 2024-09-11T01:52:02.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.369+0000 7f1de45ca640 10 monclient: build_initial_monmap 2024-09-11T01:52:02.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.369+0000 7f1de45ca640 10 monclient: monmap: 2024-09-11T01:52:02.371 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:02.371 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:02.371 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:02.370437+0000 2024-09-11T01:52:02.371 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:02.370437+0000 2024-09-11T01:52:02.371 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:02.371 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.372 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:02.372 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:02.372 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:02.372 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:02.372 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:02.372 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.370+0000 7f1de45ca640 1 Processor -- start 2024-09-11T01:52:02.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.370+0000 7f1de45ca640 1 -- start start 2024-09-11T01:52:02.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.370+0000 7f1de45ca640 10 monclient: init 2024-09-11T01:52:02.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.370+0000 7f1de45ca640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:02.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.370+0000 7f1de45ca640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-11T01:52:02.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.370+0000 7f1de45ca640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1ddc0a2e20 0x7f1ddc06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.370+0000 7f1de45ca640 10 monclient(hunting): picked mon.noname-a con 0x7f1ddc0a2e20 addr v2:172.21.15.12:3300/0 2024-09-11T01:52:02.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.370+0000 7f1de45ca640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1ddc0a4970 0x7f1ddc0a3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.370+0000 7f1de45ca640 10 monclient(hunting): picked mon.noname-c con 0x7f1ddc0a4970 addr v2:172.21.15.71:3300/0 2024-09-11T01:52:02.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.370+0000 7f1de45ca640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc06f910 0x7f1ddc0a3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.370+0000 7f1de45ca640 10 monclient(hunting): picked mon.noname-b con 0x7f1ddc06f910 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:02.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.370+0000 7f1de45ca640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.370+0000 7f1de45ca640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f1ddc06ef30 con 0x7f1ddc0a2e20 2024-09-11T01:52:02.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.370+0000 7f1de45ca640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.370+0000 7f1de45ca640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f1ddc0a3c80 con 0x7f1ddc0a4970 2024-09-11T01:52:02.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.370+0000 7f1de45ca640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.370+0000 7f1de45ca640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1ddc06ec50 con 0x7f1ddc06f910 2024-09-11T01:52:02.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.370+0000 7f1de45ca640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:02.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.370+0000 7f1de45ca640 10 monclient(hunting): authenticate will time out at 2128.812985s 2024-09-11T01:52:02.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2366640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1ddc0a2e20 0x7f1ddc06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2366640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1ddc0a2e20 0x7f1ddc06ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38046/0 (socket says 172.21.15.12:38046) 2024-09-11T01:52:02.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2b67640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc06f910 0x7f1ddc0a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2366640 1 -- 172.21.15.12:0/1800258399 learned_addr learned my addr 172.21.15.12:0/1800258399 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:02.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2366640 10 monclient(hunting): get_auth_request con 0x7f1ddc0a2e20 auth_method 0 2024-09-11T01:52:02.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2366640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:02.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2366640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:02.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2366640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:02.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2b67640 10 monclient(hunting): get_auth_request con 0x7f1ddc06f910 auth_method 0 2024-09-11T01:52:02.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2b67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:02.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2b67640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:02.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2b67640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:02.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2366640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:02.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2366640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:02.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2366640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:02.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2b67640 1 -- 172.21.15.12:0/1800258399 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc06f910 msgr2=0x7f1ddc0a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:52:02.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2b67640 1 -- 172.21.15.12:0/1800258399 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc06f910 msgr2=0x7f1ddc0a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:52:02.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2b67640 1 --2- 172.21.15.12:0/1800258399 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc06f910 0x7f1ddc0a3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:52:02.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2b67640 1 --2- 172.21.15.12:0/1800258399 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc06f910 0x7f1ddc0a3740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:02.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de1b65640 1 --2- 172.21.15.12:0/1800258399 >> v2:172.21.15.71:3300/0 conn(0x7f1ddc0a4970 0x7f1ddc0a3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de1b65640 10 monclient(hunting): get_auth_request con 0x7f1ddc0a4970 auth_method 0 2024-09-11T01:52:02.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de1b65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:02.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de1b65640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:02.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de1b65640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:02.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2366640 10 monclient(hunting): handle_auth_done global_id 254148 payload 274 2024-09-11T01:52:02.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2366640 1 -- 172.21.15.12:0/1800258399 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc06f910 msgr2=0x7f1ddc0a3740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:52:02.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2366640 1 --2- 172.21.15.12:0/1800258399 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc06f910 0x7f1ddc0a3740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2366640 1 -- 172.21.15.12:0/1800258399 >> v2:172.21.15.71:3300/0 conn(0x7f1ddc0a4970 msgr2=0x7f1ddc0a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:02.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2366640 1 --2- 172.21.15.12:0/1800258399 >> v2:172.21.15.71:3300/0 conn(0x7f1ddc0a4970 0x7f1ddc0a3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2366640 10 monclient: _finish_hunting 0 2024-09-11T01:52:02.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2366640 1 monclient: found mon.noname-a 2024-09-11T01:52:02.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2366640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:02.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2366640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:52:02.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.371+0000 7f1de2366640 1 -- 172.21.15.12:0/1800258399 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ddc06f4f0 con 0x7f1ddc0a2e20 2024-09-11T01:52:02.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.374+0000 7f1de2366640 1 -- 172.21.15.12:0/1800258399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ddc0a2e20 msgr2=0x7f1ddc06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:02.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.374+0000 7f1de2366640 1 --2- 172.21.15.12:0/1800258399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ddc0a2e20 0x7f1ddc06ded0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc074f40 tx=0x7f1dd802a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bf48ee7b9d36bfc server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:02.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.374+0000 7f1de1364640 1 -- 172.21.15.12:0/1800258399 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8037070 con 0x7f1ddc0a2e20 2024-09-11T01:52:02.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.374+0000 7f1de1364640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:02.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.374+0000 7f1de1364640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:52:02.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.374+0000 7f1de1364640 10 monclient: dump: 2024-09-11T01:52:02.385 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:02.385 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:02.385 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.385 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.386 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:02.386 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.386 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:02.386 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:02.386 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:02.386 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.374+0000 7f1de1364640 10 monclient: _finish_auth 0 2024-09-11T01:52:02.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.374+0000 7f1de1364640 10 monclient: _check_auth_tickets 2024-09-11T01:52:02.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.374+0000 7f1de1364640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:02.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.374+0000 7f1de1364640 1 -- 172.21.15.12:0/1800258399 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dd8032640 con 0x7f1ddc0a2e20 2024-09-11T01:52:02.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.374+0000 7f1de45ca640 5 monclient: authenticate success, global_id 254148 2024-09-11T01:52:02.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.374+0000 7f1de45ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:52:02.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.374+0000 7f1de1364640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:02.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.374+0000 7f1de45ca640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.374+0000 7f1de1364640 1 -- 172.21.15.12:0/1800258399 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8032a10 con 0x7f1ddc0a2e20 2024-09-11T01:52:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.374+0000 7f1de1364640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.374+0000 7f1de45ca640 4 set_mon_vals no callback set 2024-09-11T01:52:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.374+0000 7f1de1364640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:52:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.374+0000 7f1de1364640 10 monclient: dump: 2024-09-11T01:52:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:02.387 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:02.388 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:02.388 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.374+0000 7f1de45ca640 10 monclient: shutdown 2024-09-11T01:52:02.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.374+0000 7f1de45ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:02.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.374+0000 7f1de45ca640 1 -- 172.21.15.12:0/1800258399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ddc0a2e20 msgr2=0x7f1ddc06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:02.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.374+0000 7f1de45ca640 1 --2- 172.21.15.12:0/1800258399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ddc0a2e20 0x7f1ddc06ded0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc074f40 tx=0x7f1dd802a0b0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.375+0000 7f1de45ca640 1 -- 172.21.15.12:0/1800258399 shutdown_connections 2024-09-11T01:52:02.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.375+0000 7f1de45ca640 1 --2- 172.21.15.12:0/1800258399 >> v2:172.21.15.71:3300/0 conn(0x7f1ddc0a4970 0x7f1ddc0a3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.375+0000 7f1de45ca640 1 --2- 172.21.15.12:0/1800258399 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc06f910 0x7f1ddc0a3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.375+0000 7f1de45ca640 1 --2- 172.21.15.12:0/1800258399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ddc0a2e20 0x7f1ddc06ded0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.375+0000 7f1de45ca640 1 -- 172.21.15.12:0/1800258399 >> 172.21.15.12:0/1800258399 conn(0x7f1ddc0a8600 msgr2=0x7f1ddc0a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:02.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.375+0000 7f1de45ca640 1 -- 172.21.15.12:0/1800258399 shutdown_connections 2024-09-11T01:52:02.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.375+0000 7f1de45ca640 1 -- 172.21.15.12:0/1800258399 wait complete. 2024-09-11T01:52:02.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.375+0000 7f1de45ca640 10 monclient: build_initial_monmap 2024-09-11T01:52:02.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.375+0000 7f1de45ca640 10 monclient: monmap: 2024-09-11T01:52:02.389 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:02.389 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:02.389 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:02.389 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:02.389 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:02.389 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.389 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:02.389 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:02.389 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:02.389 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.375+0000 7f1de45ca640 1 Processor -- start 2024-09-11T01:52:02.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de45ca640 1 -- start start 2024-09-11T01:52:02.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de45ca640 10 monclient: init 2024-09-11T01:52:02.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de45ca640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:02.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de45ca640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:52:02.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de45ca640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ddc06f910 0x7f1ddc1cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de45ca640 10 monclient(hunting): picked mon.noname-b con 0x7f1ddc06f910 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:02.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de45ca640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc0a2e20 0x7f1ddc1cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de45ca640 10 monclient(hunting): picked mon.noname-c con 0x7f1ddc0a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:02.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de45ca640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ddc0a4970 0x7f1ddc1247d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de45ca640 10 monclient(hunting): picked mon.noname-a con 0x7f1ddc0a4970 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:02.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de45ca640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de45ca640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1ddc138470 con 0x7f1ddc0a4970 2024-09-11T01:52:02.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de45ca640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de45ca640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1ddc1382f0 con 0x7f1ddc06f910 2024-09-11T01:52:02.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de45ca640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de45ca640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1ddc1385f0 con 0x7f1ddc0a2e20 2024-09-11T01:52:02.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de45ca640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:02.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de45ca640 10 monclient(hunting): authenticate will time out at 2128.818577s 2024-09-11T01:52:02.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de1b65640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc0a2e20 0x7f1ddc1cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de2b67640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ddc0a4970 0x7f1ddc1247d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de1b65640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc0a2e20 0x7f1ddc1cd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55750/0 (socket says 172.21.15.12:55750) 2024-09-11T01:52:02.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de1b65640 1 -- 172.21.15.12:0/2216708160 learned_addr learned my addr 172.21.15.12:0/2216708160 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:02.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de1b65640 10 monclient(hunting): get_auth_request con 0x7f1ddc0a2e20 auth_method 0 2024-09-11T01:52:02.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de1b65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:02.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de1b65640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:02.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de1b65640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:02.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de2b67640 10 monclient(hunting): get_auth_request con 0x7f1ddc0a4970 auth_method 0 2024-09-11T01:52:02.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de2b67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:02.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de2b67640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:02.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de2b67640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:02.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de1b65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:02.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de1b65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:02.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de2366640 1 --2- 172.21.15.12:0/2216708160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ddc06f910 0x7f1ddc1cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de1b65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:02.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de2b67640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:02.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de2b67640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:02.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de2b67640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de2366640 10 monclient(hunting): get_auth_request con 0x7f1ddc06f910 auth_method 0 2024-09-11T01:52:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de2366640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de2366640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de2366640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de2366640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de2366640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de2366640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de1b65640 10 monclient(hunting): handle_auth_done global_id 244244 payload 824 2024-09-11T01:52:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de1b65640 1 -- 172.21.15.12:0/2216708160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ddc06f910 msgr2=0x7f1ddc1cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de1b65640 1 --2- 172.21.15.12:0/2216708160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ddc06f910 0x7f1ddc1cd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de1b65640 1 -- 172.21.15.12:0/2216708160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ddc0a4970 msgr2=0x7f1ddc1247d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de1b65640 1 --2- 172.21.15.12:0/2216708160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ddc0a4970 0x7f1ddc1247d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de1b65640 10 monclient: _finish_hunting 0 2024-09-11T01:52:02.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de1b65640 1 monclient: found mon.noname-c 2024-09-11T01:52:02.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de1b65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:02.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de1b65640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:52:02.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.376+0000 7f1de1b65640 1 -- 172.21.15.12:0/2216708160 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ddc137bb0 con 0x7f1ddc0a2e20 2024-09-11T01:52:02.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.377+0000 7f1de2b67640 1 --2- 172.21.15.12:0/2216708160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1ddc0a4970 0x7f1ddc1247d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:52:02.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.379+0000 7f1de1b65640 1 -- 172.21.15.12:0/2216708160 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc0a2e20 msgr2=0x7f1ddc1cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:02.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.379+0000 7f1de1b65640 1 --2- 172.21.15.12:0/2216708160 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc0a2e20 0x7f1ddc1cd9b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc00d800 tx=0x7f1dcc00dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:02.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.379+0000 7f1dd2ffd640 1 -- 172.21.15.12:0/2216708160 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc00b8d0 con 0x7f1ddc0a2e20 2024-09-11T01:52:02.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.379+0000 7f1dd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:02.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.379+0000 7f1dd2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:02.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.379+0000 7f1dd2ffd640 10 monclient: dump: 2024-09-11T01:52:02.394 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:02.394 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:02.394 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.395 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.395 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:02.395 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.395 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:02.395 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:02.395 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:02.395 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.379+0000 7f1dd2ffd640 10 monclient: _finish_auth 0 2024-09-11T01:52:02.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.379+0000 7f1dd2ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:52:02.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.379+0000 7f1dd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:02.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.379+0000 7f1dd2ffd640 1 -- 172.21.15.12:0/2216708160 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dcc019070 con 0x7f1ddc0a2e20 2024-09-11T01:52:02.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.379+0000 7f1de45ca640 5 monclient: authenticate success, global_id 244244 2024-09-11T01:52:02.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.379+0000 7f1de45ca640 10 monclient: _renew_subs 2024-09-11T01:52:02.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.379+0000 7f1de45ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:02.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.379+0000 7f1de45ca640 1 -- 172.21.15.12:0/2216708160 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ddc1285f0 con 0x7f1ddc0a2e20 2024-09-11T01:52:02.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.379+0000 7f1dd2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:02.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.379+0000 7f1dd2ffd640 1 -- 172.21.15.12:0/2216708160 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc00bc90 con 0x7f1ddc0a2e20 2024-09-11T01:52:02.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.379+0000 7f1dd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:02.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.379+0000 7f1dd2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:02.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.379+0000 7f1dd2ffd640 10 monclient: dump: 2024-09-11T01:52:02.396 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:02.396 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:02.396 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.396 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.396 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:02.396 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.396 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:02.397 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:02.397 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:02.397 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.379+0000 7f1de0b63640 4 set_mon_vals no callback set 2024-09-11T01:52:02.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.379+0000 7f1de45ca640 10 monclient: _renew_subs 2024-09-11T01:52:02.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.379+0000 7f1de45ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:02.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.379+0000 7f1de45ca640 1 -- 172.21.15.12:0/2216708160 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ddc128310 con 0x7f1ddc0a2e20 2024-09-11T01:52:02.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.380+0000 7f1de45ca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:02.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.380+0000 7f1de45ca640 1 --2- 172.21.15.12:0/2216708160 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc1254d0 0x7f1ddc02d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.380+0000 7f1de45ca640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.380+0000 7f1de45ca640 1 -- 172.21.15.12:0/2216708160 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1ddc128190 con 0x7f1ddc1254d0 2024-09-11T01:52:02.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.380+0000 7f1dd2ffd640 1 -- 172.21.15.12:0/2216708160 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f1dcc013070 con 0x7f1ddc0a2e20 2024-09-11T01:52:02.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.380+0000 7f1de2366640 1 --2- 172.21.15.12:0/2216708160 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc1254d0 0x7f1ddc02d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.380+0000 7f1de2366640 10 monclient: get_auth_request con 0x7f1ddc1254d0 auth_method 0 2024-09-11T01:52:02.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.380+0000 7f1de2366640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:02.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.380+0000 7f1de2366640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:02.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.380+0000 7f1de2366640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:02.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.380+0000 7f1de2366640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:02.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.380+0000 7f1de2366640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:02.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.380+0000 7f1de2366640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:02.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.380+0000 7f1de2366640 10 monclient(hunting): handle_auth_done global_id 244247 payload 824 2024-09-11T01:52:02.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.380+0000 7f1de2366640 1 -- 172.21.15.12:0/2216708160 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1ddc06ded0 con 0x7f1ddc1254d0 2024-09-11T01:52:02.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.380+0000 7f1dd2ffd640 1 --2- 172.21.15.12:0/2216708160 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1dc003e210 0x7f1dc00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.380+0000 7f1dd2ffd640 1 -- 172.21.15.12:0/2216708160 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(765..765 src has 676..765) ==== 4831+0+0 (secure 0 0 0) 0x7f1dcc052e50 con 0x7f1ddc0a2e20 2024-09-11T01:52:02.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.380+0000 7f1de2b67640 1 --2- 172.21.15.12:0/2216708160 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1dc003e210 0x7f1dc00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.380+0000 7f1de2b67640 10 monclient: get_auth_request con 0x7f1dc003e210 auth_method 0 2024-09-11T01:52:02.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.382+0000 7f1de2366640 1 -- 172.21.15.12:0/2216708160 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc1254d0 msgr2=0x7f1ddc02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.382+0000 7f1de2366640 1 --2- 172.21.15.12:0/2216708160 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc1254d0 0x7f1ddc02d440 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8034000 tx=0x7f1dd80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.383+0000 7f1dd2ffd640 1 -- 172.21.15.12:0/2216708160 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8037040 con 0x7f1ddc1254d0 2024-09-11T01:52:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.383+0000 7f1dd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.383+0000 7f1dd2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.383+0000 7f1dd2ffd640 10 monclient: dump: 2024-09-11T01:52:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:02.400 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:02.400 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.383+0000 7f1de2b67640 1 --2- 172.21.15.12:0/2216708160 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1dc003e210 0x7f1dc00406d0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc06fd60 tx=0x7f1dd4009780 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:02.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.383+0000 7f1dd2ffd640 1 -- 172.21.15.12:0/2216708160 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1dd8032e40 con 0x7f1ddc1254d0 2024-09-11T01:52:02.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.383+0000 7f1dd2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:02.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.383+0000 7f1dd2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:02.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.383+0000 7f1dd2ffd640 1 -- 172.21.15.12:0/2216708160 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc1254d0 msgr2=0x7f1ddc02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:02.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.383+0000 7f1dd2ffd640 1 --2- 172.21.15.12:0/2216708160 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc1254d0 0x7f1ddc02d440 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8034000 tx=0x7f1dd80046c0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.383+0000 7f1dd2ffd640 1 -- 172.21.15.12:0/2216708160 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc1254d0 msgr2=0x7f1ddc02d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:02.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.383+0000 7f1dd2ffd640 1 --2- 172.21.15.12:0/2216708160 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc1254d0 0x7f1ddc02d440 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8034000 tx=0x7f1dd80046c0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.389+0000 7f1de45ca640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:52:02.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.389+0000 7f1de45ca640 1 --2- 172.21.15.12:0/2216708160 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc1258b0 0x7f1ddc02d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.389+0000 7f1de45ca640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.389+0000 7f1de45ca640 1 -- 172.21.15.12:0/2216708160 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1ddc06a060 con 0x7f1ddc1258b0 2024-09-11T01:52:02.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.389+0000 7f1de2366640 1 --2- 172.21.15.12:0/2216708160 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc1258b0 0x7f1ddc02d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.389+0000 7f1de2366640 10 monclient: get_auth_request con 0x7f1ddc1258b0 auth_method 0 2024-09-11T01:52:02.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.389+0000 7f1de2366640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:02.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.389+0000 7f1de2366640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:02.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.389+0000 7f1de2366640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:02.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.389+0000 7f1de2366640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:02.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.389+0000 7f1de2366640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:02.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.389+0000 7f1de2366640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:02.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.389+0000 7f1de2366640 10 monclient(hunting): handle_auth_done global_id 244250 payload 824 2024-09-11T01:52:02.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.389+0000 7f1de2366640 1 -- 172.21.15.12:0/2216708160 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1ddc09f590 con 0x7f1ddc1258b0 2024-09-11T01:52:02.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.392+0000 7f1de2366640 1 -- 172.21.15.12:0/2216708160 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc1258b0 msgr2=0x7f1ddc02d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:02.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.392+0000 7f1de2366640 1 --2- 172.21.15.12:0/2216708160 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc1258b0 0x7f1ddc02d980 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc02eb00 tx=0x7f1dd8008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:02.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.392+0000 7f1dd2ffd640 1 -- 172.21.15.12:0/2216708160 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8037040 con 0x7f1ddc1258b0 2024-09-11T01:52:02.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.392+0000 7f1dd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:02.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.392+0000 7f1dd2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:02.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.392+0000 7f1dd2ffd640 10 monclient: dump: 2024-09-11T01:52:02.403 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:02.403 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:02.404 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.404 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.404 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:02.404 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.404 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:02.404 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:02.404 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:02.404 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.392+0000 7f1dd2ffd640 1 -- 172.21.15.12:0/2216708160 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7f1dd804c610 con 0x7f1ddc1258b0 2024-09-11T01:52:02.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.392+0000 7f1dd2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:52:02.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.392+0000 7f1dd2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:02.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.392+0000 7f1dd2ffd640 1 -- 172.21.15.12:0/2216708160 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc1258b0 msgr2=0x7f1ddc02d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:02.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.392+0000 7f1dd2ffd640 1 --2- 172.21.15.12:0/2216708160 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc1258b0 0x7f1ddc02d980 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc02eb00 tx=0x7f1dd8008030 comp rx=0 tx=0).stop 2024-09-11T01:52:02.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.392+0000 7f1dd2ffd640 1 -- 172.21.15.12:0/2216708160 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc1258b0 msgr2=0x7f1ddc02d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:02.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.392+0000 7f1dd2ffd640 1 --2- 172.21.15.12:0/2216708160 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc1258b0 0x7f1ddc02d980 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc02eb00 tx=0x7f1dd8008030 comp rx=0 tx=0).stop 2024-09-11T01:52:02.405 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:52:02.405 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:52:02.405 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:52:02.405 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:52:02.405 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 214, 2024-09-11T01:52:02.405 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 75840, 2024-09-11T01:52:02.405 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:52:02.405 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:52:02.406 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:52:02.406 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:52:02.406 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:02.406 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:52:02.406 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:52:02.406 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:52:02.406 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:52:02.406 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:02.406 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:02.406 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:02.406 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:02.406 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:02.406 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:02.407 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:02.407 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:02.407 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:02.407 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:02.407 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:02.407 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:02.407 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:52:02.407 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:52:02.407 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:02.407 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:02.407 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:02.407 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:02.407 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:02.408 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:02.408 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:02.408 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:02.408 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:02.408 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:02.408 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:02.408 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:02.408 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.408 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:52:02.408 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:52:02.408 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:52:02.408 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:52:02.408 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:52:02.409 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:52:02.409 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:52:02.409 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:52:02.409 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:52:02.409 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:52:02.409 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:52:02.409 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:52:02.409 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:52:02.409 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:52:02.409 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:52:02.409 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:52:02.409 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:52:02.409 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:02.410 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:02.410 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:02.410 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:02.410 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:02.410 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:02.410 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:02.410 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:02.410 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:02.410 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:02.410 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:02.410 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:02.410 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:52:02.410 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.411 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:52:02.411 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.411 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:52:02.411 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:52:02.411 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:02.411 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:02.411 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.411 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:02.411 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:52:02.411 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:02.411 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.411 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.411 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:02.411 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:52:02.412 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:02.412 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:02.412 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:02.412 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.412 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:52:02.412 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:52:02.412 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:02.412 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:02.412 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:02.412 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.412 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.412 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:52:02.412 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:52:02.412 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:02.413 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:02.413 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.413 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:02.413 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:52:02.413 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:02.413 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.413 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.413 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:02.413 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:52:02.413 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:02.413 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:02.413 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:02.413 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.413 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:52:02.414 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:52:02.414 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:02.414 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:02.414 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:02.414 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.414 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.414 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:52:02.414 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:52:02.414 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:02.414 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:02.414 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.414 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:02.414 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:52:02.415 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:02.415 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.415 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.415 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:02.415 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:52:02.415 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:02.415 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:02.415 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:02.415 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.415 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:52:02.415 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:52:02.415 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:02.415 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:02.415 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:02.416 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:02.416 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:02.416 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.416 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:52:02.416 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:52:02.416 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.416 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:02.416 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:02.416 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:52:02.416 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:02.416 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:02.416 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:52:02.416 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.417 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:02.417 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:02.417 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:52:02.417 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:02.417 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:02.417 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:52:02.417 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.417 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:02.417 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:02.417 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:52:02.417 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:02.417 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:02.417 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:52:02.417 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.418 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:02.418 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:02.418 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:52:02.418 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:02.418 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:02.418 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.418 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:52:02.418 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:52:02.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.392+0000 7f1de45ca640 1 -- 172.21.15.12:0/2216708160 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1dc003e210 msgr2=0x7f1dc00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:02.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.392+0000 7f1de45ca640 1 --2- 172.21.15.12:0/2216708160 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1dc003e210 0x7f1dc00406d0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc06fd60 tx=0x7f1dd4009780 comp rx=0 tx=0).stop 2024-09-11T01:52:02.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.392+0000 7f1de45ca640 10 monclient: shutdown 2024-09-11T01:52:02.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.392+0000 7f1de45ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:02.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.392+0000 7f1de45ca640 1 -- 172.21.15.12:0/2216708160 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc0a2e20 msgr2=0x7f1ddc1cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:02.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.392+0000 7f1de45ca640 1 --2- 172.21.15.12:0/2216708160 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1ddc0a2e20 0x7f1ddc1cd9b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc00d800 tx=0x7f1dcc00dcc0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.392+0000 7f1de2b67640 1 -- 172.21.15.12:0/2216708160 reap_dead start 2024-09-11T01:52:02.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.392+0000 7f1de2b67640 1 -- 172.21.15.12:0/2216708160 reap_dead start 2024-09-11T01:52:02.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.392+0000 7f1de45ca640 1 -- 172.21.15.12:0/2216708160 shutdown_connections 2024-09-11T01:52:02.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.392+0000 7f1de45ca640 1 -- 172.21.15.12:0/2216708160 >> 172.21.15.12:0/2216708160 conn(0x7f1ddc0a8600 msgr2=0x7f1ddc0a7b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:02.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.392+0000 7f1de45ca640 1 -- 172.21.15.12:0/2216708160 shutdown_connections 2024-09-11T01:52:02.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.392+0000 7f1de45ca640 1 -- 172.21.15.12:0/2216708160 wait complete. 2024-09-11T01:52:02.421 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:52:02.421 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:52:02.421 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:52:02.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.521+0000 7f5818e23640 10 monclient: get_monmap_and_config 2024-09-11T01:52:02.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.521+0000 7f5818e23640 10 monclient: build_initial_monmap 2024-09-11T01:52:02.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.522+0000 7f5818e23640 10 monclient: monmap: 2024-09-11T01:52:02.524 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:02.524 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:02.524 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:02.522736+0000 2024-09-11T01:52:02.524 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:02.522736+0000 2024-09-11T01:52:02.524 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:02.524 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.524 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:02.524 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:02.524 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:02.524 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:02.524 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:02.524 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.522+0000 7f5818e23640 1 Processor -- start 2024-09-11T01:52:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.522+0000 7f5818e23640 1 -- start start 2024-09-11T01:52:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.522+0000 7f5818e23640 10 monclient: init 2024-09-11T01:52:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5818e23640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5818e23640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-11T01:52:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5818e23640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5814134ad0 addr v1:172.21.15.12:6789/0 2024-09-11T01:52:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5818e23640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f58140a2200 0x7f581413a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5818e23640 10 monclient(hunting): picked mon.noname-c con 0x7f58140a2200 addr v2:172.21.15.71:3300/0 2024-09-11T01:52:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5818e23640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f58141355c0 0x7f581413c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5818e23640 10 monclient(hunting): picked mon.noname-b con 0x7f58141355c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5818e23640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f581409ed30 con 0x7f5814134ad0 2024-09-11T01:52:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5818e23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5818e23640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f581409f5d0 con 0x7f58140a2200 2024-09-11T01:52:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5818e23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5818e23640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f581409f010 con 0x7f58141355c0 2024-09-11T01:52:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5818e23640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5818e23640 10 monclient(hunting): authenticate will time out at 2128.965314s 2024-09-11T01:52:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f581259c640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5814134ad0 0x7f5814134eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37090/0 (socket says 172.21.15.12:37090) 2024-09-11T01:52:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5812d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f58141355c0 0x7f581413c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f581259c640 1 -- 172.21.15.12:0/1516083004 learned_addr learned my addr 172.21.15.12:0/1516083004 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:52:02.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5812d9d640 10 monclient(hunting): get_auth_request con 0x7f58141355c0 auth_method 0 2024-09-11T01:52:02.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5812d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:02.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5812d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:02.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5812d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:02.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5812d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:02.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5812d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:02.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5812d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:02.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5811d9b640 1 --2- 172.21.15.12:0/1516083004 >> v2:172.21.15.71:3300/0 conn(0x7f58140a2200 0x7f581413a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5811d9b640 10 monclient(hunting): get_auth_request con 0x7f58140a2200 auth_method 0 2024-09-11T01:52:02.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5811d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:02.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5811d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:02.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5811d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:02.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5812d9d640 10 monclient(hunting): handle_auth_done global_id 244253 payload 274 2024-09-11T01:52:02.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5812d9d640 1 -- 172.21.15.12:0/1516083004 >> v2:172.21.15.71:3300/0 conn(0x7f58140a2200 msgr2=0x7f581413a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:02.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5812d9d640 1 --2- 172.21.15.12:0/1516083004 >> v2:172.21.15.71:3300/0 conn(0x7f58140a2200 0x7f581413a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5812d9d640 1 -- 172.21.15.12:0/1516083004 >> v1:172.21.15.12:6789/0 conn(0x7f5814134ad0 legacy=0x7f5814134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:02.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5812d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:52:02.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5812d9d640 1 monclient: found mon.noname-b 2024-09-11T01:52:02.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5812d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:02.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5812d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:52:02.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.523+0000 7f5812d9d640 1 -- 172.21.15.12:0/1516083004 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f581409f2f0 con 0x7f58141355c0 2024-09-11T01:52:02.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.526+0000 7f5812d9d640 1 -- 172.21.15.12:0/1516083004 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f58141355c0 msgr2=0x7f581413c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:02.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.526+0000 7f5812d9d640 1 --2- 172.21.15.12:0/1516083004 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f58141355c0 0x7f581413c550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f580c009dc0 tx=0x7f580c02a4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7861e132e289f578 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:02.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.526+0000 7f581159a640 1 -- 172.21.15.12:0/1516083004 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580c007950 con 0x7f58141355c0 2024-09-11T01:52:02.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.526+0000 7f581159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:02.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.526+0000 7f581159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:52:02.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.526+0000 7f581159a640 10 monclient: dump: 2024-09-11T01:52:02.537 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:02.537 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:02.537 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.537 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.537 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:02.537 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.537 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:02.537 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:02.537 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:02.538 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.526+0000 7f581159a640 10 monclient: _finish_auth 0 2024-09-11T01:52:02.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.526+0000 7f581159a640 10 monclient: _check_auth_tickets 2024-09-11T01:52:02.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.526+0000 7f581159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:02.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.526+0000 7f581159a640 1 -- 172.21.15.12:0/1516083004 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f580c007af0 con 0x7f58141355c0 2024-09-11T01:52:02.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.526+0000 7f581159a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:02.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.526+0000 7f581159a640 1 -- 172.21.15.12:0/1516083004 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580c031400 con 0x7f58141355c0 2024-09-11T01:52:02.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.526+0000 7f5818e23640 5 monclient: authenticate success, global_id 244253 2024-09-11T01:52:02.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.526+0000 7f5818e23640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:02.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.526+0000 7f5818e23640 4 set_mon_vals no callback set 2024-09-11T01:52:02.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.526+0000 7f581159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:02.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.526+0000 7f581159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:02.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.526+0000 7f581159a640 10 monclient: dump: 2024-09-11T01:52:02.538 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:02.539 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:02.539 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.539 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.539 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:02.539 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.539 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:02.539 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:02.539 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:02.539 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.527+0000 7f5818e23640 10 monclient: shutdown 2024-09-11T01:52:02.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.527+0000 7f5818e23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:02.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.527+0000 7f5818e23640 1 -- 172.21.15.12:0/1516083004 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f58141355c0 msgr2=0x7f581413c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:02.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.527+0000 7f5818e23640 1 --2- 172.21.15.12:0/1516083004 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f58141355c0 0x7f581413c550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f580c009dc0 tx=0x7f580c02a4d0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.527+0000 7f5818e23640 1 -- 172.21.15.12:0/1516083004 shutdown_connections 2024-09-11T01:52:02.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.527+0000 7f5818e23640 1 --2- 172.21.15.12:0/1516083004 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f58141355c0 0x7f581413c550 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.527+0000 7f5818e23640 1 --2- 172.21.15.12:0/1516083004 >> v2:172.21.15.71:3300/0 conn(0x7f58140a2200 0x7f581413a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.527+0000 7f5818e23640 1 -- 172.21.15.12:0/1516083004 >> 172.21.15.12:0/1516083004 conn(0x7f58140a2f70 msgr2=0x7f58140a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:02.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.527+0000 7f5818e23640 1 -- 172.21.15.12:0/1516083004 shutdown_connections 2024-09-11T01:52:02.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.527+0000 7f5818e23640 1 -- 172.21.15.12:0/1516083004 wait complete. 2024-09-11T01:52:02.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.528+0000 7f5818e23640 10 monclient: build_initial_monmap 2024-09-11T01:52:02.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.528+0000 7f5818e23640 10 monclient: monmap: 2024-09-11T01:52:02.540 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:02.540 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:02.540 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:02.540 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:02.540 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:02.541 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.541 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:02.541 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:02.541 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:02.541 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.528+0000 7f5818e23640 1 Processor -- start 2024-09-11T01:52:02.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.528+0000 7f5818e23640 1 -- start start 2024-09-11T01:52:02.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.528+0000 7f5818e23640 10 monclient: init 2024-09-11T01:52:02.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5818e23640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:02.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5818e23640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:52:02.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5818e23640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f58140a2200 0x7f58141333d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5818e23640 10 monclient(hunting): picked mon.noname-c con 0x7f58140a2200 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:02.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5818e23640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5814134ad0 0x7f5814133910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5818e23640 10 monclient(hunting): picked mon.noname-a con 0x7f5814134ad0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:02.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5818e23640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f58141355c0 0x7f581411cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5818e23640 10 monclient(hunting): picked mon.noname-b con 0x7f58141355c0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:02.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5818e23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5818e23640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f58140a0150 con 0x7f5814134ad0 2024-09-11T01:52:02.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5818e23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5818e23640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f581409ffd0 con 0x7f58141355c0 2024-09-11T01:52:02.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5818e23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5818e23640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f58140a0c90 con 0x7f58140a2200 2024-09-11T01:52:02.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5818e23640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:02.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5818e23640 10 monclient(hunting): authenticate will time out at 2128.971261s 2024-09-11T01:52:02.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f581259c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f58140a2200 0x7f58141333d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5811d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5814134ad0 0x7f5814133910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f581259c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f58140a2200 0x7f58141333d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55790/0 (socket says 172.21.15.12:55790) 2024-09-11T01:52:02.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f581259c640 1 -- 172.21.15.12:0/2433870652 learned_addr learned my addr 172.21.15.12:0/2433870652 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:02.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f581259c640 10 monclient(hunting): get_auth_request con 0x7f58140a2200 auth_method 0 2024-09-11T01:52:02.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f581259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:02.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f581259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:02.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f581259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:02.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5811d9b640 10 monclient(hunting): get_auth_request con 0x7f5814134ad0 auth_method 0 2024-09-11T01:52:02.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5811d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:02.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5811d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:02.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5811d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:02.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5812d9d640 1 --2- 172.21.15.12:0/2433870652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f58141355c0 0x7f581411cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f581259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:02.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f581259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:02.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f581259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:02.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5811d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:02.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5811d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:02.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5811d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:02.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5812d9d640 10 monclient(hunting): get_auth_request con 0x7f58141355c0 auth_method 0 2024-09-11T01:52:02.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5812d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:02.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5812d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:02.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5812d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:02.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5812d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:02.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5812d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:02.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5812d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:02.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f581259c640 10 monclient(hunting): handle_auth_done global_id 244256 payload 824 2024-09-11T01:52:02.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f581259c640 1 -- 172.21.15.12:0/2433870652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f58141355c0 msgr2=0x7f581411cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:02.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f581259c640 1 --2- 172.21.15.12:0/2433870652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f58141355c0 0x7f581411cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f581259c640 1 -- 172.21.15.12:0/2433870652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5814134ad0 msgr2=0x7f5814133910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:02.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f581259c640 1 --2- 172.21.15.12:0/2433870652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5814134ad0 0x7f5814133910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f581259c640 10 monclient: _finish_hunting 0 2024-09-11T01:52:02.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f581259c640 1 monclient: found mon.noname-c 2024-09-11T01:52:02.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f581259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:02.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f581259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:52:02.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f581259c640 1 -- 172.21.15.12:0/2433870652 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5814120ab0 con 0x7f58140a2200 2024-09-11T01:52:02.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.529+0000 7f5811d9b640 1 --2- 172.21.15.12:0/2433870652 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5814134ad0 0x7f5814133910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:52:02.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.531+0000 7f581259c640 1 -- 172.21.15.12:0/2433870652 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f58140a2200 msgr2=0x7f58141333d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:02.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.531+0000 7f581259c640 1 --2- 172.21.15.12:0/2433870652 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f58140a2200 0x7f58141333d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f580400c460 tx=0x7f580400c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:02.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.532+0000 7f5802ffd640 1 -- 172.21.15.12:0/2433870652 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5804007070 con 0x7f58140a2200 2024-09-11T01:52:02.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.532+0000 7f5802ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:02.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.532+0000 7f5802ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:02.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.532+0000 7f5802ffd640 10 monclient: dump: 2024-09-11T01:52:02.546 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:02.546 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:02.546 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.546 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.546 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:02.546 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.546 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:02.546 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:02.546 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:02.546 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.532+0000 7f5802ffd640 10 monclient: _finish_auth 0 2024-09-11T01:52:02.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.532+0000 7f5802ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:52:02.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.532+0000 7f5802ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:02.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.532+0000 7f5818e23640 5 monclient: authenticate success, global_id 244256 2024-09-11T01:52:02.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.532+0000 7f5802ffd640 1 -- 172.21.15.12:0/2433870652 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5804007230 con 0x7f58140a2200 2024-09-11T01:52:02.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.532+0000 7f5818e23640 10 monclient: _renew_subs 2024-09-11T01:52:02.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.532+0000 7f5818e23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:02.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.532+0000 7f5818e23640 1 -- 172.21.15.12:0/2433870652 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f581409d480 con 0x7f58140a2200 2024-09-11T01:52:02.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.532+0000 7f5802ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:02.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.532+0000 7f5802ffd640 1 -- 172.21.15.12:0/2433870652 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5804007620 con 0x7f58140a2200 2024-09-11T01:52:02.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.532+0000 7f5802ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:02.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.532+0000 7f5802ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:02.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.532+0000 7f5802ffd640 10 monclient: dump: 2024-09-11T01:52:02.547 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:02.548 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.532+0000 7f5818e23640 10 monclient: _renew_subs 2024-09-11T01:52:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.532+0000 7f5810d99640 4 set_mon_vals no callback set 2024-09-11T01:52:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.532+0000 7f5818e23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:02.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.532+0000 7f5818e23640 1 -- 172.21.15.12:0/2433870652 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5814134290 con 0x7f58140a2200 2024-09-11T01:52:02.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.533+0000 7f5818e23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:02.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.533+0000 7f5818e23640 1 --2- 172.21.15.12:0/2433870652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f581411d530 0x7f581411d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.533+0000 7f5818e23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.533+0000 7f5818e23640 1 -- 172.21.15.12:0/2433870652 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f58140a5d90 con 0x7f581411d530 2024-09-11T01:52:02.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.533+0000 7f5802ffd640 1 -- 172.21.15.12:0/2433870652 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f580401b400 con 0x7f58140a2200 2024-09-11T01:52:02.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.533+0000 7f5811d9b640 1 --2- 172.21.15.12:0/2433870652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f581411d530 0x7f581411d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.533+0000 7f5811d9b640 10 monclient: get_auth_request con 0x7f581411d530 auth_method 0 2024-09-11T01:52:02.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.533+0000 7f5811d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:02.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.533+0000 7f5811d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:02.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.533+0000 7f5811d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:02.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.533+0000 7f5811d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:02.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.533+0000 7f5811d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:02.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.533+0000 7f5811d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:02.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.533+0000 7f5811d9b640 10 monclient(hunting): handle_auth_done global_id 234277 payload 824 2024-09-11T01:52:02.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.533+0000 7f5811d9b640 1 -- 172.21.15.12:0/2433870652 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5814073aa0 con 0x7f581411d530 2024-09-11T01:52:02.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.533+0000 7f5802ffd640 1 --2- 172.21.15.12:0/2433870652 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f57f403e240 0x7f57f4040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.533+0000 7f5802ffd640 1 -- 172.21.15.12:0/2433870652 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(765..765 src has 676..765) ==== 4831+0+0 (secure 0 0 0) 0x7f5804052a00 con 0x7f58140a2200 2024-09-11T01:52:02.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.533+0000 7f5812d9d640 1 --2- 172.21.15.12:0/2433870652 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f57f403e240 0x7f57f4040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.533+0000 7f5812d9d640 10 monclient: get_auth_request con 0x7f57f403e240 auth_method 0 2024-09-11T01:52:02.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.535+0000 7f5811d9b640 1 -- 172.21.15.12:0/2433870652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f581411d530 msgr2=0x7f581411d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:02.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.535+0000 7f5811d9b640 1 --2- 172.21.15.12:0/2433870652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f581411d530 0x7f581411d910 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f581411f520 tx=0x7f580800e5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:02.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.536+0000 7f5802ffd640 1 -- 172.21.15.12:0/2433870652 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580801a020 con 0x7f581411d530 2024-09-11T01:52:02.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.536+0000 7f5802ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:02.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.536+0000 7f5802ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:52:02.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.536+0000 7f5802ffd640 10 monclient: dump: 2024-09-11T01:52:02.553 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:02.553 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:02.553 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.553 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.553 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:02.553 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.553 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:02.553 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:02.553 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:02.553 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.536+0000 7f5812d9d640 1 --2- 172.21.15.12:0/2433870652 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f57f403e240 0x7f57f4040700 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f58140a72b0 tx=0x7f580c02f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:02.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.536+0000 7f5802ffd640 1 -- 172.21.15.12:0/2433870652 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f580800ee80 con 0x7f581411d530 2024-09-11T01:52:02.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.536+0000 7f5802ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:02.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.536+0000 7f5802ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:02.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.536+0000 7f5802ffd640 1 -- 172.21.15.12:0/2433870652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f581411d530 msgr2=0x7f581411d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:02.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.536+0000 7f5802ffd640 1 --2- 172.21.15.12:0/2433870652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f581411d530 0x7f581411d910 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f581411f520 tx=0x7f580800e5a0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.536+0000 7f5802ffd640 1 -- 172.21.15.12:0/2433870652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f581411d530 msgr2=0x7f581411d910 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:02.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.536+0000 7f5802ffd640 1 --2- 172.21.15.12:0/2433870652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f581411d530 0x7f581411d910 secure :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f581411f520 tx=0x7f580800e5a0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.542+0000 7f5818e23640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:52:02.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.542+0000 7f5818e23640 1 --2- 172.21.15.12:0/2433870652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f581411de50 0x7f58141bcd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.542+0000 7f5818e23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.542+0000 7f5818e23640 1 -- 172.21.15.12:0/2433870652 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5814134eb0 con 0x7f581411de50 2024-09-11T01:52:02.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.543+0000 7f5811d9b640 1 --2- 172.21.15.12:0/2433870652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f581411de50 0x7f58141bcd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.543+0000 7f5811d9b640 10 monclient: get_auth_request con 0x7f581411de50 auth_method 0 2024-09-11T01:52:02.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.543+0000 7f5811d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:02.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.543+0000 7f5811d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:02.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.543+0000 7f5811d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:02.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.543+0000 7f5811d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:02.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.543+0000 7f5811d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:02.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.543+0000 7f5811d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:02.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.543+0000 7f5811d9b640 10 monclient(hunting): handle_auth_done global_id 234280 payload 824 2024-09-11T01:52:02.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.543+0000 7f5811d9b640 1 -- 172.21.15.12:0/2433870652 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5814077360 con 0x7f581411de50 2024-09-11T01:52:02.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.545+0000 7f5811d9b640 1 -- 172.21.15.12:0/2433870652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f581411de50 msgr2=0x7f58141bcd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:02.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.545+0000 7f5811d9b640 1 --2- 172.21.15.12:0/2433870652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f581411de50 0x7f58141bcd00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f580801b000 tx=0x7f5808004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:02.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.545+0000 7f5802ffd640 1 -- 172.21.15.12:0/2433870652 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580801a030 con 0x7f581411de50 2024-09-11T01:52:02.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.545+0000 7f5802ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:02.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.545+0000 7f5802ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:52:02.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.545+0000 7f5802ffd640 10 monclient: dump: 2024-09-11T01:52:02.556 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:02.556 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:02.556 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.556 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.556 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:02.556 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.556 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:02.556 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:02.556 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:02.556 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.545+0000 7f5802ffd640 1 -- 172.21.15.12:0/2433870652 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7f580802b450 con 0x7f581411de50 2024-09-11T01:52:02.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.545+0000 7f5802ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:52:02.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.545+0000 7f5802ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:02.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.545+0000 7f5802ffd640 1 -- 172.21.15.12:0/2433870652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f581411de50 msgr2=0x7f58141bcd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:02.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.545+0000 7f5802ffd640 1 --2- 172.21.15.12:0/2433870652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f581411de50 0x7f58141bcd00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f580801b000 tx=0x7f5808004730 comp rx=0 tx=0).stop 2024-09-11T01:52:02.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.545+0000 7f5802ffd640 1 -- 172.21.15.12:0/2433870652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f581411de50 msgr2=0x7f58141bcd00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:02.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.545+0000 7f5802ffd640 1 --2- 172.21.15.12:0/2433870652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f581411de50 0x7f58141bcd00 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.557 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:52:02.557 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:52:02.557 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:52:02.557 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:52:02.557 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 214, 2024-09-11T01:52:02.558 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 124438, 2024-09-11T01:52:02.558 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:52:02.558 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:52:02.558 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:52:02.558 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:52:02.558 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:02.558 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:52:02.558 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:52:02.558 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:52:02.558 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:52:02.558 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:02.558 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:02.558 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:02.558 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:02.559 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:02.559 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:02.559 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:02.559 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:02.559 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:02.559 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:02.559 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:02.559 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:02.559 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:52:02.559 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:52:02.559 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:02.559 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:02.559 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:02.560 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:02.560 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:02.560 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:02.560 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:02.560 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:02.560 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:02.560 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:02.560 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:02.560 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:02.560 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.560 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:52:02.560 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:52:02.560 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:52:02.561 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:52:02.561 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:52:02.561 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:52:02.561 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:52:02.561 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:52:02.561 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:52:02.561 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:52:02.561 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:52:02.561 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:52:02.561 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:52:02.561 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:52:02.561 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:52:02.561 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:52:02.561 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:52:02.562 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:02.562 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:02.562 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:02.562 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:02.562 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:02.562 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:02.562 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:02.562 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:02.562 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:02.562 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:02.562 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:02.562 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:02.562 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:52:02.562 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.563 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:52:02.563 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.563 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:52:02.563 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:52:02.563 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:02.563 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:02.563 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.563 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:02.563 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:52:02.563 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:02.563 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.563 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.563 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:02.564 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:52:02.564 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:02.564 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:02.564 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:02.564 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.564 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:52:02.564 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:52:02.564 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:02.564 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:02.564 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:02.564 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.564 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.564 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:52:02.564 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:52:02.565 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:02.565 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:02.565 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.565 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:02.565 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:52:02.565 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:02.565 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.565 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.565 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:02.565 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:52:02.565 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:02.565 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:02.565 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:02.565 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.566 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:52:02.566 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:52:02.566 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:02.566 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:02.566 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:02.566 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.566 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.566 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:52:02.566 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:52:02.566 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:02.566 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:02.566 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.566 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:02.567 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:52:02.567 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:02.567 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.567 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.567 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:02.567 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:52:02.567 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:02.567 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:02.567 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:02.567 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.567 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:52:02.567 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:52:02.567 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:02.567 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:02.568 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:02.568 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:02.568 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:02.568 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.568 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:52:02.568 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:52:02.568 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.568 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:02.568 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:02.568 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:52:02.568 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:02.568 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:02.568 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:52:02.568 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.569 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:02.569 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:02.569 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:52:02.569 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:02.569 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:02.569 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:52:02.569 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:02.569 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:02.569 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:02.569 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:52:02.569 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:02.569 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:02.569 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:02.570 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:52:02.570 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:52:02.571 INFO:tasks.rados.rados.0.smithi071.stdout:3406: done (4 left) 2024-09-11T01:52:02.571 INFO:tasks.rados.rados.0.smithi071.stdout:3407: done (3 left) 2024-09-11T01:52:02.571 INFO:tasks.rados.rados.0.smithi071.stdout:3408: done (2 left) 2024-09-11T01:52:02.571 INFO:tasks.rados.rados.0.smithi071.stdout:3409: done (1 left) 2024-09-11T01:52:02.572 INFO:tasks.rados.rados.0.smithi071.stdout:3411: done (0 left) 2024-09-11T01:52:02.572 INFO:tasks.rados.rados.0.smithi071.stdout:3412: copy_from oid 48 from oid 37 current snap is 380 2024-09-11T01:52:02.572 INFO:tasks.rados.rados.0.smithi071.stdout:3413: snap_create 2024-09-11T01:52:02.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.546+0000 7f5818e23640 1 -- 172.21.15.12:0/2433870652 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f57f403e240 msgr2=0x7f57f4040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:02.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.546+0000 7f5818e23640 1 --2- 172.21.15.12:0/2433870652 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f57f403e240 0x7f57f4040700 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f58140a72b0 tx=0x7f580c02f000 comp rx=0 tx=0).stop 2024-09-11T01:52:02.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.546+0000 7f5818e23640 10 monclient: shutdown 2024-09-11T01:52:02.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.546+0000 7f5818e23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:02.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.546+0000 7f5818e23640 1 -- 172.21.15.12:0/2433870652 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f58140a2200 msgr2=0x7f58141333d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:02.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.546+0000 7f5818e23640 1 --2- 172.21.15.12:0/2433870652 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f58140a2200 0x7f58141333d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f580400c460 tx=0x7f580400c920 comp rx=0 tx=0).stop 2024-09-11T01:52:02.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.546+0000 7f5812d9d640 1 -- 172.21.15.12:0/2433870652 reap_dead start 2024-09-11T01:52:02.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.546+0000 7f5812d9d640 1 -- 172.21.15.12:0/2433870652 reap_dead start 2024-09-11T01:52:02.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.546+0000 7f5818e23640 1 -- 172.21.15.12:0/2433870652 shutdown_connections 2024-09-11T01:52:02.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.546+0000 7f5818e23640 1 -- 172.21.15.12:0/2433870652 >> 172.21.15.12:0/2433870652 conn(0x7f58140a2f70 msgr2=0x7f581409d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:02.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.546+0000 7f5818e23640 1 -- 172.21.15.12:0/2433870652 shutdown_connections 2024-09-11T01:52:02.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.546+0000 7f5818e23640 1 -- 172.21.15.12:0/2433870652 wait complete. 2024-09-11T01:52:02.574 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:52:02.574 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-11T01:52:02.574 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T01:52:02.574 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-11T01:52:02.574 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-11T01:52:02.575 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T01:52:02.575 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:52:02.575 INFO:tasks.rados.rados.0.smithi071.stdout:3412: finishing copy_from to smithi07134777-48 2024-09-11T01:52:02.575 INFO:tasks.rados.rados.0.smithi071.stdout:3412: got expected ENOENT (src dne) 2024-09-11T01:52:02.575 INFO:tasks.rados.rados.0.smithi071.stdout:3412: finishing copy_from racing read to smithi07134777-48 2024-09-11T01:52:02.585 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T01:52:02.585 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:52:02.585 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:52:02.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.725+0000 7f5b37789640 10 monclient: get_monmap_and_config 2024-09-11T01:52:02.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.725+0000 7f5b37789640 10 monclient: build_initial_monmap 2024-09-11T01:52:02.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.725+0000 7f5b37789640 10 monclient: monmap: 2024-09-11T01:52:02.727 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:02.727 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:02.727 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:02.726350+0000 2024-09-11T01:52:02.727 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:02.726350+0000 2024-09-11T01:52:02.728 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:02.728 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.728 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:02.728 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:02.728 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:02.728 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:02.728 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:02.728 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.726+0000 7f5b37789640 1 Processor -- start 2024-09-11T01:52:02.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.726+0000 7f5b37789640 1 -- start start 2024-09-11T01:52:02.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.726+0000 7f5b37789640 10 monclient: init 2024-09-11T01:52:02.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.726+0000 7f5b37789640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:02.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.726+0000 7f5b37789640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-11T01:52:02.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.726+0000 7f5b37789640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5b300a2db0 addr v1:172.21.15.71:6789/0 2024-09-11T01:52:02.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.726+0000 7f5b37789640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b3009fbd0 0x7f5b3006d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.726+0000 7f5b37789640 10 monclient(hunting): picked mon.noname-b con 0x7f5b3009fbd0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:02.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.726+0000 7f5b37789640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5b3013d550 0x7f5b3013f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.726+0000 7f5b37789640 10 monclient(hunting): picked mon.noname-a con 0x7f5b3013d550 addr v2:172.21.15.12:3300/0 2024-09-11T01:52:02.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.726+0000 7f5b37789640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b3009eea0 con 0x7f5b300a2db0 2024-09-11T01:52:02.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.726+0000 7f5b37789640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.726+0000 7f5b37789640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f5b300a3850 con 0x7f5b3013d550 2024-09-11T01:52:02.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.726+0000 7f5b37789640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.726+0000 7f5b37789640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5b300a3190 con 0x7f5b3009fbd0 2024-09-11T01:52:02.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.726+0000 7f5b37789640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:02.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.726+0000 7f5b37789640 10 monclient(hunting): authenticate will time out at 2129.168906s 2024-09-11T01:52:02.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.726+0000 7f5b35d26640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7f5b3013d550 msgr2=0x7f5b3013f940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:52:02.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.726+0000 7f5b35d26640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f5b3013d550 0x7f5b3013f940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:02.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.727+0000 7f5b34d24640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b3009fbd0 0x7f5b3006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.727+0000 7f5b34d24640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b3009fbd0 0x7f5b3006d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55800/0 (socket says 172.21.15.12:55800) 2024-09-11T01:52:02.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.727+0000 7f5b34d24640 1 -- 172.21.15.12:0/1207112345 learned_addr learned my addr 172.21.15.12:0/1207112345 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:02.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.727+0000 7f5b34d24640 10 monclient(hunting): get_auth_request con 0x7f5b3009fbd0 auth_method 0 2024-09-11T01:52:02.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.727+0000 7f5b34d24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:02.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.727+0000 7f5b34d24640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:02.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.727+0000 7f5b34d24640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:02.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.727+0000 7f5b34d24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:02.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.727+0000 7f5b34d24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:02.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.727+0000 7f5b34d24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:02.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.727+0000 7f5b34d24640 10 monclient(hunting): handle_auth_done global_id 244259 payload 274 2024-09-11T01:52:02.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.727+0000 7f5b34d24640 1 -- 172.21.15.12:0/1207112345 >> v2:172.21.15.12:3300/0 conn(0x7f5b3013d550 msgr2=0x7f5b3013f940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:52:02.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.727+0000 7f5b34d24640 1 --2- 172.21.15.12:0/1207112345 >> v2:172.21.15.12:3300/0 conn(0x7f5b3013d550 0x7f5b3013f940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.727+0000 7f5b34d24640 1 -- 172.21.15.12:0/1207112345 >> v1:172.21.15.71:6789/0 conn(0x7f5b300a2db0 legacy=0x7f5b3006cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:02.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.727+0000 7f5b34d24640 10 monclient: _finish_hunting 0 2024-09-11T01:52:02.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.727+0000 7f5b34d24640 1 monclient: found mon.noname-b 2024-09-11T01:52:02.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.727+0000 7f5b34d24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:02.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.727+0000 7f5b34d24640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:52:02.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.727+0000 7f5b34d24640 1 -- 172.21.15.12:0/1207112345 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b3009f180 con 0x7f5b3009fbd0 2024-09-11T01:52:02.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b34d24640 1 -- 172.21.15.12:0/1207112345 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b3009fbd0 msgr2=0x7f5b3006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:02.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b34d24640 1 --2- 172.21.15.12:0/1207112345 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b3009fbd0 0x7f5b3006d260 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5b20009870 tx=0x7f5b2002a0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3ac210504a9dfef server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:02.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b27fff640 1 -- 172.21.15.12:0/1207112345 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b20037070 con 0x7f5b3009fbd0 2024-09-11T01:52:02.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:02.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b27fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:52:02.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b27fff640 10 monclient: dump: 2024-09-11T01:52:02.739 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:02.739 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:02.739 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.739 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.739 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:02.739 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.739 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:02.740 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:02.740 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:02.740 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b27fff640 10 monclient: _finish_auth 0 2024-09-11T01:52:02.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b27fff640 10 monclient: _check_auth_tickets 2024-09-11T01:52:02.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:02.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b27fff640 1 -- 172.21.15.12:0/1207112345 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b20032640 con 0x7f5b3009fbd0 2024-09-11T01:52:02.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b37789640 5 monclient: authenticate success, global_id 244259 2024-09-11T01:52:02.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b27fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:02.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b27fff640 1 -- 172.21.15.12:0/1207112345 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b20032a10 con 0x7f5b3009fbd0 2024-09-11T01:52:02.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b37789640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:02.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b37789640 4 set_mon_vals no callback set 2024-09-11T01:52:02.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:02.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b27fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:02.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b27fff640 10 monclient: dump: 2024-09-11T01:52:02.741 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:02.741 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:02.741 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.741 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.741 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:02.741 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.741 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:02.741 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:02.741 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:02.741 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b37789640 10 monclient: shutdown 2024-09-11T01:52:02.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b37789640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:02.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b37789640 1 -- 172.21.15.12:0/1207112345 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b3009fbd0 msgr2=0x7f5b3006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:02.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b37789640 1 --2- 172.21.15.12:0/1207112345 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b3009fbd0 0x7f5b3006d260 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5b20009870 tx=0x7f5b2002a0d0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b37789640 1 -- 172.21.15.12:0/1207112345 shutdown_connections 2024-09-11T01:52:02.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b37789640 1 --2- 172.21.15.12:0/1207112345 >> v2:172.21.15.12:3300/0 conn(0x7f5b3013d550 0x7f5b3013f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b37789640 1 --2- 172.21.15.12:0/1207112345 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b3009fbd0 0x7f5b3006d260 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b37789640 1 -- 172.21.15.12:0/1207112345 >> 172.21.15.12:0/1207112345 conn(0x7f5b300a3ea0 msgr2=0x7f5b300a42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:02.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b37789640 1 -- 172.21.15.12:0/1207112345 shutdown_connections 2024-09-11T01:52:02.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.730+0000 7f5b37789640 1 -- 172.21.15.12:0/1207112345 wait complete. 2024-09-11T01:52:02.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.731+0000 7f5b37789640 10 monclient: build_initial_monmap 2024-09-11T01:52:02.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.731+0000 7f5b37789640 10 monclient: monmap: 2024-09-11T01:52:02.742 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:02.742 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:02.743 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:02.743 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:02.743 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:02.743 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.743 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:02.743 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:02.743 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:02.743 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.731+0000 7f5b37789640 1 Processor -- start 2024-09-11T01:52:02.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.731+0000 7f5b37789640 1 -- start start 2024-09-11T01:52:02.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.731+0000 7f5b37789640 10 monclient: init 2024-09-11T01:52:02.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b37789640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:02.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b37789640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:52:02.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b37789640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b3009fbd0 0x7f5b30099770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b37789640 10 monclient(hunting): picked mon.noname-c con 0x7f5b3009fbd0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:02.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b37789640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b300a2db0 0x7f5b300913c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b37789640 10 monclient(hunting): picked mon.noname-a con 0x7f5b300a2db0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:02.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b37789640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b3013d550 0x7f5b30091900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b37789640 10 monclient(hunting): picked mon.noname-b con 0x7f5b3013d550 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:02.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b37789640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b37789640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5b3009ea20 con 0x7f5b300a2db0 2024-09-11T01:52:02.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b37789640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b37789640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5b3009e8a0 con 0x7f5b3013d550 2024-09-11T01:52:02.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b37789640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:02.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b37789640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5b3006d7a0 con 0x7f5b3009fbd0 2024-09-11T01:52:02.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b37789640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:02.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b37789640 10 monclient(hunting): authenticate will time out at 2129.174352s 2024-09-11T01:52:02.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b34d24640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b300a2db0 msgr2=0x7f5b300913c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:52:02.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b34d24640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b300a2db0 0x7f5b300913c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:02.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b35525640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b3009fbd0 0x7f5b30099770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b35525640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b3009fbd0 0x7f5b30099770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55812/0 (socket says 172.21.15.12:55812) 2024-09-11T01:52:02.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b35525640 1 -- 172.21.15.12:0/3893976020 learned_addr learned my addr 172.21.15.12:0/3893976020 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:02.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b35525640 10 monclient(hunting): get_auth_request con 0x7f5b3009fbd0 auth_method 0 2024-09-11T01:52:02.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b35525640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:02.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b35525640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:02.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b35525640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:02.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b35525640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:02.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b35525640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:02.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b35525640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:02.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b35d26640 1 --2- 172.21.15.12:0/3893976020 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b3013d550 0x7f5b30091900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b35525640 10 monclient(hunting): handle_auth_done global_id 244262 payload 824 2024-09-11T01:52:02.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b35525640 1 -- 172.21.15.12:0/3893976020 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b3013d550 msgr2=0x7f5b30091900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:02.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b35525640 1 --2- 172.21.15.12:0/3893976020 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b3013d550 0x7f5b30091900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b35525640 1 -- 172.21.15.12:0/3893976020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b300a2db0 msgr2=0x7f5b300913c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:52:02.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b35525640 1 --2- 172.21.15.12:0/3893976020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b300a2db0 0x7f5b300913c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:02.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b35525640 10 monclient: _finish_hunting 0 2024-09-11T01:52:02.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b35525640 1 monclient: found mon.noname-c 2024-09-11T01:52:02.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b35525640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:02.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b35525640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:52:02.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b35525640 1 -- 172.21.15.12:0/3893976020 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b3013d380 con 0x7f5b3009fbd0 2024-09-11T01:52:02.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b35d26640 10 monclient: get_auth_request con 0x7f5b3013d550 auth_method 0 2024-09-11T01:52:02.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.732+0000 7f5b35d26640 1 --2- 172.21.15.12:0/3893976020 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b3013d550 0x7f5b30091900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:52:02.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.734+0000 7f5b35525640 1 -- 172.21.15.12:0/3893976020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b3009fbd0 msgr2=0x7f5b30099770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:02.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.734+0000 7f5b35525640 1 --2- 172.21.15.12:0/3893976020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b3009fbd0 0x7f5b30099770 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c00c430 tx=0x7f5b2c00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:02.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.735+0000 7f5b25ffb640 1 -- 172.21.15.12:0/3893976020 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c0119a0 con 0x7f5b3009fbd0 2024-09-11T01:52:02.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.735+0000 7f5b25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:02.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.735+0000 7f5b25ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:02.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.735+0000 7f5b25ffb640 10 monclient: dump: 2024-09-11T01:52:02.747 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:02.747 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:02.747 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.747 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.747 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:02.747 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.748 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:02.748 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:02.748 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:02.748 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.735+0000 7f5b25ffb640 10 monclient: _finish_auth 0 2024-09-11T01:52:02.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.735+0000 7f5b25ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:52:02.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.735+0000 7f5b25ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:02.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.735+0000 7f5b37789640 5 monclient: authenticate success, global_id 244262 2024-09-11T01:52:02.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.735+0000 7f5b25ffb640 1 -- 172.21.15.12:0/3893976020 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b2c017070 con 0x7f5b3009fbd0 2024-09-11T01:52:02.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.735+0000 7f5b37789640 10 monclient: _renew_subs 2024-09-11T01:52:02.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.735+0000 7f5b37789640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:02.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.735+0000 7f5b37789640 1 -- 172.21.15.12:0/3893976020 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b30099e40 con 0x7f5b3009fbd0 2024-09-11T01:52:02.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.735+0000 7f5b25ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:02.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.735+0000 7f5b25ffb640 1 -- 172.21.15.12:0/3893976020 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c011d80 con 0x7f5b3009fbd0 2024-09-11T01:52:02.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.735+0000 7f5b25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:02.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.735+0000 7f5b25ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:02.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.735+0000 7f5b25ffb640 10 monclient: dump: 2024-09-11T01:52:02.749 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:02.749 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:02.749 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.749 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:02.749 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:02.749 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:02.749 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:02.750 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:02.750 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:02.750 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:02.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.735+0000 7f5b37789640 10 monclient: _renew_subs 2024-09-11T01:52:02.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.735+0000 7f5b26ffd640 4 set_mon_vals no callback set 2024-09-11T01:52:02.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.735+0000 7f5b37789640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:02.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.735+0000 7f5b37789640 1 -- 172.21.15.12:0/3893976020 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b30092c80 con 0x7f5b3009fbd0 2024-09-11T01:52:02.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.736+0000 7f5b25ffb640 1 -- 172.21.15.12:0/3893976020 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f5b2c01c460 con 0x7f5b3009fbd0 2024-09-11T01:52:02.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.736+0000 7f5b37789640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:02.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.736+0000 7f5b37789640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:02.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.736+0000 7f5b37789640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:02.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.736+0000 7f5b37789640 1 -- 172.21.15.12:0/3893976020 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b3006cbd0 con 0x7f5b3009fbd0 2024-09-11T01:52:02.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.736+0000 7f5b25ffb640 1 --2- 172.21.15.12:0/3893976020 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5b1403e210 0x7f5b140406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:02.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.736+0000 7f5b25ffb640 1 -- 172.21.15.12:0/3893976020 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(766..766 src has 676..766) ==== 4803+0+0 (secure 0 0 0) 0x7f5b2c052d10 con 0x7f5b3009fbd0 2024-09-11T01:52:02.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.736+0000 7f5b34d24640 1 --2- 172.21.15.12:0/3893976020 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5b1403e210 0x7f5b140406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:02.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.736+0000 7f5b34d24640 10 monclient: get_auth_request con 0x7f5b1403e210 auth_method 0 2024-09-11T01:52:02.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.739+0000 7f5b34d24640 1 --2- 172.21.15.12:0/3893976020 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5b1403e210 0x7f5b140406d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f5b20009870 tx=0x7f5b2002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:02.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.740+0000 7f5b25ffb640 1 -- 172.21.15.12:0/3893976020 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5b2c0516f0 con 0x7f5b3009fbd0 2024-09-11T01:52:02.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.740+0000 7f5b25ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:02.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.740+0000 7f5b25ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:02.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.997+0000 7f5b37789640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:52:02.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.997+0000 7f5b37789640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:52:03.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.997+0000 7f5b37789640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:03.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.997+0000 7f5b37789640 1 -- 172.21.15.12:0/3893976020 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5b30094800 con 0x7f5b3009fbd0 2024-09-11T01:52:03.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.997+0000 7f5b25ffb640 1 -- 172.21.15.12:0/3893976020 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f5b2c01c960 con 0x7f5b3009fbd0 2024-09-11T01:52:03.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.997+0000 7f5b25ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:52:03.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:02.997+0000 7f5b25ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:03.001 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":214,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:52:03.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:03.000+0000 7f5b37789640 1 -- 172.21.15.12:0/3893976020 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5b1403e210 msgr2=0x7f5b140406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:03.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:03.000+0000 7f5b37789640 1 --2- 172.21.15.12:0/3893976020 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5b1403e210 0x7f5b140406d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f5b20009870 tx=0x7f5b2002e000 comp rx=0 tx=0).stop 2024-09-11T01:52:03.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:03.000+0000 7f5b37789640 10 monclient: shutdown 2024-09-11T01:52:03.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:03.000+0000 7f5b37789640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:03.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:03.000+0000 7f5b37789640 1 -- 172.21.15.12:0/3893976020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b3009fbd0 msgr2=0x7f5b30099770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:03.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:03.000+0000 7f5b37789640 1 --2- 172.21.15.12:0/3893976020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b3009fbd0 0x7f5b30099770 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c00c430 tx=0x7f5b2c00c8f0 comp rx=0 tx=0).stop 2024-09-11T01:52:03.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:03.000+0000 7f5b37789640 1 -- 172.21.15.12:0/3893976020 shutdown_connections 2024-09-11T01:52:03.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:03.000+0000 7f5b37789640 1 --2- 172.21.15.12:0/3893976020 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b3013d550 0x7f5b30091900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:03.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:03.000+0000 7f5b37789640 1 --2- 172.21.15.12:0/3893976020 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5b1403e210 0x7f5b140406d0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:03.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:03.000+0000 7f5b37789640 1 --2- 172.21.15.12:0/3893976020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b300a2db0 0x7f5b300913c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:03.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:03.000+0000 7f5b37789640 1 --2- 172.21.15.12:0/3893976020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b3009fbd0 0x7f5b30099770 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:03.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:03.000+0000 7f5b37789640 1 -- 172.21.15.12:0/3893976020 >> 172.21.15.12:0/3893976020 conn(0x7f5b300a3ea0 msgr2=0x7f5b3013ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:03.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:03.000+0000 7f5b37789640 1 -- 172.21.15.12:0/3893976020 shutdown_connections 2024-09-11T01:52:03.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:03.000+0000 7f5b37789640 1 -- 172.21.15.12:0/3893976020 wait complete. 2024-09-11T01:52:06.013 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:52:06.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.124+0000 7f7c37f3a640 10 monclient: get_monmap_and_config 2024-09-11T01:52:06.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.124+0000 7f7c37f3a640 10 monclient: build_initial_monmap 2024-09-11T01:52:06.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.124+0000 7f7c37f3a640 10 monclient: monmap: 2024-09-11T01:52:06.126 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:06.126 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:06.126 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:06.125022+0000 2024-09-11T01:52:06.126 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:06.125022+0000 2024-09-11T01:52:06.126 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:06.126 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:06.126 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:06.126 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:06.126 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:06.126 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:06.127 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:06.127 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:06.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.124+0000 7f7c37f3a640 1 Processor -- start 2024-09-11T01:52:06.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c37f3a640 1 -- start start 2024-09-11T01:52:06.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c37f3a640 10 monclient: init 2024-09-11T01:52:06.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c37f3a640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:06.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c37f3a640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-11T01:52:06.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c37f3a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7c30138230 0x7f7c30138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:06.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c37f3a640 10 monclient(hunting): picked mon.noname-a con 0x7f7c30138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:52:06.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c37f3a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c3009d030 0x7f7c300a5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:06.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c37f3a640 10 monclient(hunting): picked mon.noname-b con 0x7f7c3009d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:06.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c37f3a640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7c3009d460 0x7f7c300a5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:06.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c37f3a640 10 monclient(hunting): picked mon.noname-c con 0x7f7c3009d460 addr v2:172.21.15.71:3300/0 2024-09-11T01:52:06.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c37f3a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:06.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c37f3a640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f7c3009c9c0 con 0x7f7c30138230 2024-09-11T01:52:06.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c37f3a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:06.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c37f3a640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f7c300a6e30 con 0x7f7c3009d460 2024-09-11T01:52:06.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c37f3a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:06.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c37f3a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7c3009ba20 con 0x7f7c3009d030 2024-09-11T01:52:06.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c37f3a640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:06.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c37f3a640 10 monclient(hunting): authenticate will time out at 2132.567703s 2024-09-11T01:52:06.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c35cd6640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7f7c30138230 msgr2=0x7f7c30138610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:52:06.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c35cd6640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7c30138230 0x7f7c30138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:06.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c354d5640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c3009d030 0x7f7c300a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:06.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c354d5640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c3009d030 0x7f7c300a5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55814/0 (socket says 172.21.15.12:55814) 2024-09-11T01:52:06.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c354d5640 1 -- 172.21.15.12:0/3154843675 learned_addr learned my addr 172.21.15.12:0/3154843675 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:06.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c354d5640 10 monclient(hunting): get_auth_request con 0x7f7c3009d030 auth_method 0 2024-09-11T01:52:06.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c354d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:06.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c354d5640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:06.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c354d5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:06.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c354d5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:06.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c354d5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:06.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.125+0000 7f7c354d5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:06.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.126+0000 7f7c364d7640 1 --2- 172.21.15.12:0/3154843675 >> v2:172.21.15.71:3300/0 conn(0x7f7c3009d460 0x7f7c300a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:06.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.126+0000 7f7c354d5640 10 monclient(hunting): handle_auth_done global_id 244265 payload 274 2024-09-11T01:52:06.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.126+0000 7f7c354d5640 1 -- 172.21.15.12:0/3154843675 >> v2:172.21.15.71:3300/0 conn(0x7f7c3009d460 msgr2=0x7f7c300a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:06.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.126+0000 7f7c354d5640 1 --2- 172.21.15.12:0/3154843675 >> v2:172.21.15.71:3300/0 conn(0x7f7c3009d460 0x7f7c300a5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:06.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.126+0000 7f7c354d5640 1 -- 172.21.15.12:0/3154843675 >> v2:172.21.15.12:3300/0 conn(0x7f7c30138230 msgr2=0x7f7c30138610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:52:06.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.126+0000 7f7c354d5640 1 --2- 172.21.15.12:0/3154843675 >> v2:172.21.15.12:3300/0 conn(0x7f7c30138230 0x7f7c30138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:06.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.126+0000 7f7c354d5640 10 monclient: _finish_hunting 0 2024-09-11T01:52:06.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.126+0000 7f7c354d5640 1 monclient: found mon.noname-b 2024-09-11T01:52:06.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.126+0000 7f7c354d5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:06.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.126+0000 7f7c354d5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:52:06.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.126+0000 7f7c354d5640 1 -- 172.21.15.12:0/3154843675 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c300a66a0 con 0x7f7c3009d030 2024-09-11T01:52:06.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.128+0000 7f7c354d5640 1 -- 172.21.15.12:0/3154843675 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c3009d030 msgr2=0x7f7c300a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:06.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.128+0000 7f7c354d5640 1 --2- 172.21.15.12:0/3154843675 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c3009d030 0x7f7c300a5020 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7c3006e500 tx=0x7f7c2402a0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cbdf7846493af1e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:06.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c34cd4640 1 -- 172.21.15.12:0/3154843675 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c24037070 con 0x7f7c3009d030 2024-09-11T01:52:06.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c34cd4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:06.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c34cd4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:52:06.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c34cd4640 10 monclient: dump: 2024-09-11T01:52:06.139 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:06.139 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:06.139 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:06.139 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:06.139 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:06.139 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:06.139 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:06.139 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:06.139 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:06.139 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:06.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c34cd4640 10 monclient: _finish_auth 0 2024-09-11T01:52:06.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c34cd4640 10 monclient: _check_auth_tickets 2024-09-11T01:52:06.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c34cd4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:06.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c37f3a640 5 monclient: authenticate success, global_id 244265 2024-09-11T01:52:06.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c34cd4640 1 -- 172.21.15.12:0/3154843675 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c24032640 con 0x7f7c3009d030 2024-09-11T01:52:06.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c37f3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:52:06.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c34cd4640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:06.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c37f3a640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:06.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c37f3a640 4 set_mon_vals no callback set 2024-09-11T01:52:06.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c34cd4640 1 -- 172.21.15.12:0/3154843675 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c24032a10 con 0x7f7c3009d030 2024-09-11T01:52:06.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c34cd4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:06.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c34cd4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:06.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c34cd4640 10 monclient: dump: 2024-09-11T01:52:06.141 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:06.141 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:06.141 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:06.141 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:06.141 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:06.141 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:06.141 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:06.141 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:06.141 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:06.141 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:06.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c37f3a640 10 monclient: shutdown 2024-09-11T01:52:06.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c37f3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:06.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c37f3a640 1 -- 172.21.15.12:0/3154843675 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c3009d030 msgr2=0x7f7c300a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:06.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c37f3a640 1 --2- 172.21.15.12:0/3154843675 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c3009d030 0x7f7c300a5020 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7c3006e500 tx=0x7f7c2402a0d0 comp rx=0 tx=0).stop 2024-09-11T01:52:06.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c37f3a640 1 -- 172.21.15.12:0/3154843675 shutdown_connections 2024-09-11T01:52:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c37f3a640 1 --2- 172.21.15.12:0/3154843675 >> v2:172.21.15.71:3300/0 conn(0x7f7c3009d460 0x7f7c300a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c37f3a640 1 --2- 172.21.15.12:0/3154843675 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c3009d030 0x7f7c300a5020 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c37f3a640 1 --2- 172.21.15.12:0/3154843675 >> v2:172.21.15.12:3300/0 conn(0x7f7c30138230 0x7f7c30138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c37f3a640 1 -- 172.21.15.12:0/3154843675 >> 172.21.15.12:0/3154843675 conn(0x7f7c300a7120 msgr2=0x7f7c300a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c37f3a640 1 -- 172.21.15.12:0/3154843675 shutdown_connections 2024-09-11T01:52:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.129+0000 7f7c37f3a640 1 -- 172.21.15.12:0/3154843675 wait complete. 2024-09-11T01:52:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.130+0000 7f7c37f3a640 10 monclient: build_initial_monmap 2024-09-11T01:52:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.130+0000 7f7c37f3a640 10 monclient: monmap: 2024-09-11T01:52:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:06.145 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:06.146 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:06.146 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:06.146 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:06.146 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:06.146 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:06.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.130+0000 7f7c37f3a640 1 Processor -- start 2024-09-11T01:52:06.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.130+0000 7f7c37f3a640 1 -- start start 2024-09-11T01:52:06.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.130+0000 7f7c37f3a640 10 monclient: init 2024-09-11T01:52:06.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c37f3a640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:06.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c37f3a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:52:06.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c37f3a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c3009d030 0x7f7c30136f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:06.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c37f3a640 10 monclient(hunting): picked mon.noname-c con 0x7f7c3009d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:06.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c37f3a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c3009d460 0x7f7c30137460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:06.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c37f3a640 10 monclient(hunting): picked mon.noname-a con 0x7f7c3009d460 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:06.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c37f3a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c30138230 0x7f7c30124b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:06.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c37f3a640 10 monclient(hunting): picked mon.noname-b con 0x7f7c30138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:06.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c37f3a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:06.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c37f3a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7c3006f970 con 0x7f7c3009d460 2024-09-11T01:52:06.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c37f3a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:06.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c37f3a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f7c3006f7f0 con 0x7f7c30138230 2024-09-11T01:52:06.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c37f3a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:06.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c37f3a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7c300a5aa0 con 0x7f7c3009d030 2024-09-11T01:52:06.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c37f3a640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:06.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c37f3a640 10 monclient(hunting): authenticate will time out at 2132.573349s 2024-09-11T01:52:06.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c354d5640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c3009d460 msgr2=0x7f7c30137460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:52:06.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c354d5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c3009d460 0x7f7c30137460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:06.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c35cd6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c3009d030 0x7f7c30136f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:06.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c35cd6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c3009d030 0x7f7c30136f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55826/0 (socket says 172.21.15.12:55826) 2024-09-11T01:52:06.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c35cd6640 1 -- 172.21.15.12:0/2602061168 learned_addr learned my addr 172.21.15.12:0/2602061168 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:06.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c35cd6640 10 monclient(hunting): get_auth_request con 0x7f7c3009d030 auth_method 0 2024-09-11T01:52:06.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c35cd6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:06.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c35cd6640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:06.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c35cd6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:06.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c364d7640 1 --2- 172.21.15.12:0/2602061168 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c30138230 0x7f7c30124b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:06.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c35cd6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:06.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c35cd6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:06.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c35cd6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:06.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c364d7640 10 monclient(hunting): get_auth_request con 0x7f7c30138230 auth_method 0 2024-09-11T01:52:06.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c364d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:06.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c364d7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:06.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c364d7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:06.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c364d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:06.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c364d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:06.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c364d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:06.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c35cd6640 10 monclient(hunting): handle_auth_done global_id 244268 payload 824 2024-09-11T01:52:06.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c35cd6640 1 -- 172.21.15.12:0/2602061168 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c30138230 msgr2=0x7f7c30124b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:06.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c35cd6640 1 --2- 172.21.15.12:0/2602061168 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c30138230 0x7f7c30124b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:06.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c35cd6640 1 -- 172.21.15.12:0/2602061168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c3009d460 msgr2=0x7f7c30137460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:52:06.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c35cd6640 1 --2- 172.21.15.12:0/2602061168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c3009d460 0x7f7c30137460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:06.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c35cd6640 10 monclient: _finish_hunting 0 2024-09-11T01:52:06.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c35cd6640 1 monclient: found mon.noname-c 2024-09-11T01:52:06.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c35cd6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:06.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c35cd6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:52:06.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.131+0000 7f7c35cd6640 1 -- 172.21.15.12:0/2602061168 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c301cdbf0 con 0x7f7c3009d030 2024-09-11T01:52:06.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.134+0000 7f7c35cd6640 1 -- 172.21.15.12:0/2602061168 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c3009d030 msgr2=0x7f7c30136f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:06.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.134+0000 7f7c35cd6640 1 --2- 172.21.15.12:0/2602061168 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c3009d030 0x7f7c30136f20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7c2000b9e0 tx=0x7f7c2000bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:06.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.134+0000 7f7c2e7fc640 1 -- 172.21.15.12:0/2602061168 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c2000cdb0 con 0x7f7c3009d030 2024-09-11T01:52:06.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.134+0000 7f7c2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:06.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.134+0000 7f7c2e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:06.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.134+0000 7f7c2e7fc640 10 monclient: dump: 2024-09-11T01:52:06.150 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:06.150 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:06.150 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:06.150 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:06.150 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:06.150 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:06.151 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:06.151 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:06.151 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:06.151 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:06.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.134+0000 7f7c2e7fc640 10 monclient: _finish_auth 0 2024-09-11T01:52:06.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.134+0000 7f7c2e7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:52:06.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.134+0000 7f7c2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:06.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.134+0000 7f7c2e7fc640 1 -- 172.21.15.12:0/2602061168 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c20010070 con 0x7f7c3009d030 2024-09-11T01:52:06.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.134+0000 7f7c37f3a640 5 monclient: authenticate success, global_id 244268 2024-09-11T01:52:06.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.134+0000 7f7c37f3a640 10 monclient: _renew_subs 2024-09-11T01:52:06.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.134+0000 7f7c37f3a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:06.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.134+0000 7f7c37f3a640 1 -- 172.21.15.12:0/2602061168 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c301265c0 con 0x7f7c3009d030 2024-09-11T01:52:06.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.134+0000 7f7c2e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:06.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.134+0000 7f7c2e7fc640 1 -- 172.21.15.12:0/2602061168 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c200156b0 con 0x7f7c3009d030 2024-09-11T01:52:06.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.134+0000 7f7c2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:06.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.134+0000 7f7c2e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:06.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.134+0000 7f7c2e7fc640 10 monclient: dump: 2024-09-11T01:52:06.152 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:06.152 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:06.152 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:06.152 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:06.152 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:06.152 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:06.152 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:06.152 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:06.152 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:06.152 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:06.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.134+0000 7f7c37f3a640 10 monclient: _renew_subs 2024-09-11T01:52:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.134+0000 7f7c37f3a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.134+0000 7f7c37f3a640 1 -- 172.21.15.12:0/2602061168 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c30127180 con 0x7f7c3009d030 2024-09-11T01:52:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.134+0000 7f7c2ffff640 4 set_mon_vals no callback set 2024-09-11T01:52:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.135+0000 7f7c2e7fc640 1 -- 172.21.15.12:0/2602061168 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f7c2001e3f0 con 0x7f7c3009d030 2024-09-11T01:52:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.135+0000 7f7c37f3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.135+0000 7f7c37f3a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.135+0000 7f7c37f3a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.135+0000 7f7c37f3a640 1 -- 172.21.15.12:0/2602061168 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c300a5e20 con 0x7f7c3009d030 2024-09-11T01:52:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.135+0000 7f7c2e7fc640 1 --2- 172.21.15.12:0/2602061168 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7c1c03e240 0x7f7c1c040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.135+0000 7f7c2e7fc640 1 -- 172.21.15.12:0/2602061168 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(766..766 src has 676..766) ==== 4803+0+0 (secure 0 0 0) 0x7f7c20052ec0 con 0x7f7c3009d030 2024-09-11T01:52:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.135+0000 7f7c354d5640 1 --2- 172.21.15.12:0/2602061168 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7c1c03e240 0x7f7c1c040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:06.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.135+0000 7f7c354d5640 10 monclient: get_auth_request con 0x7f7c1c03e240 auth_method 0 2024-09-11T01:52:06.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.138+0000 7f7c354d5640 1 --2- 172.21.15.12:0/2602061168 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7c1c03e240 0x7f7c1c040700 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f7c24034000 tx=0x7f7c2402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:06.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.140+0000 7f7c2e7fc640 1 -- 172.21.15.12:0/2602061168 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7c200518a0 con 0x7f7c3009d030 2024-09-11T01:52:06.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.140+0000 7f7c2e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:06.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.140+0000 7f7c2e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:06.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.395+0000 7f7c37f3a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:52:06.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.395+0000 7f7c37f3a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:52:06.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.395+0000 7f7c37f3a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:06.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.395+0000 7f7c37f3a640 1 -- 172.21.15.12:0/2602061168 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7c30128420 con 0x7f7c3009d030 2024-09-11T01:52:06.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.395+0000 7f7c2e7fc640 1 -- 172.21.15.12:0/2602061168 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f7c2001d3a0 con 0x7f7c3009d030 2024-09-11T01:52:06.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.395+0000 7f7c2e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:52:06.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.395+0000 7f7c2e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:06.398 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":214,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:52:06.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.398+0000 7f7c37f3a640 1 -- 172.21.15.12:0/2602061168 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7c1c03e240 msgr2=0x7f7c1c040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:06.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.398+0000 7f7c37f3a640 1 --2- 172.21.15.12:0/2602061168 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7c1c03e240 0x7f7c1c040700 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f7c24034000 tx=0x7f7c2402e000 comp rx=0 tx=0).stop 2024-09-11T01:52:06.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.398+0000 7f7c37f3a640 10 monclient: shutdown 2024-09-11T01:52:06.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.398+0000 7f7c37f3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:06.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.398+0000 7f7c37f3a640 1 -- 172.21.15.12:0/2602061168 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c3009d030 msgr2=0x7f7c30136f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:06.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.398+0000 7f7c37f3a640 1 --2- 172.21.15.12:0/2602061168 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c3009d030 0x7f7c30136f20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7c2000b9e0 tx=0x7f7c2000bea0 comp rx=0 tx=0).stop 2024-09-11T01:52:06.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.398+0000 7f7c37f3a640 1 -- 172.21.15.12:0/2602061168 shutdown_connections 2024-09-11T01:52:06.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.398+0000 7f7c37f3a640 1 --2- 172.21.15.12:0/2602061168 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c30138230 0x7f7c30124b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:06.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.398+0000 7f7c37f3a640 1 --2- 172.21.15.12:0/2602061168 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7c1c03e240 0x7f7c1c040700 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:06.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.398+0000 7f7c37f3a640 1 --2- 172.21.15.12:0/2602061168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c3009d460 0x7f7c30137460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:06.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.398+0000 7f7c37f3a640 1 --2- 172.21.15.12:0/2602061168 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c3009d030 0x7f7c30136f20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:06.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.398+0000 7f7c37f3a640 1 -- 172.21.15.12:0/2602061168 >> 172.21.15.12:0/2602061168 conn(0x7f7c300a7120 msgr2=0x7f7c300a33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:06.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.398+0000 7f7c37f3a640 1 -- 172.21.15.12:0/2602061168 shutdown_connections 2024-09-11T01:52:06.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:06.398+0000 7f7c37f3a640 1 -- 172.21.15.12:0/2602061168 wait complete. 2024-09-11T01:52:09.411 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:52:09.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.523+0000 7ff6479cd640 10 monclient: get_monmap_and_config 2024-09-11T01:52:09.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.523+0000 7ff6479cd640 10 monclient: build_initial_monmap 2024-09-11T01:52:09.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.523+0000 7ff6479cd640 10 monclient: monmap: 2024-09-11T01:52:09.525 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:09.525 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:09.526 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:09.524628+0000 2024-09-11T01:52:09.526 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:09.524628+0000 2024-09-11T01:52:09.526 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:09.526 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:09.526 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:09.526 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:09.526 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:09.526 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:09.526 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:09.526 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.524+0000 7ff6479cd640 1 Processor -- start 2024-09-11T01:52:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.524+0000 7ff6479cd640 1 -- start start 2024-09-11T01:52:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.524+0000 7ff6479cd640 10 monclient: init 2024-09-11T01:52:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff6479cd640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff6479cd640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-11T01:52:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff6479cd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff6400a0440 addr v1:172.21.15.12:6789/0 2024-09-11T01:52:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff6479cd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6400a2910 0x7ff64013d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff6479cd640 10 monclient(hunting): picked mon.noname-b con 0x7ff6400a2910 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff6479cd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff6400a2d30 0x7ff64013f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff6479cd640 10 monclient(hunting): picked mon.noname-a con 0x7ff6400a2d30 addr v2:172.21.15.12:3300/0 2024-09-11T01:52:09.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff6479cd640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6400a0cf0 con 0x7ff6400a0440 2024-09-11T01:52:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff6479cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff6479cd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff6400744a0 con 0x7ff6400a2d30 2024-09-11T01:52:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff6479cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff6479cd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff6400705b0 con 0x7ff6400a2910 2024-09-11T01:52:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff6479cd640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff6479cd640 10 monclient(hunting): authenticate will time out at 2135.967296s 2024-09-11T01:52:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff645769640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7ff6400a0440 legacy=0x7ff6400a2200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:52:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff645f6a640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7ff6400a2d30 msgr2=0x7ff64013f570 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:52:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff645f6a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff6400a2d30 0x7ff64013f570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff644f68640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6400a2910 0x7ff64013d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff644f68640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6400a2910 0x7ff64013d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52914/0 (socket says 172.21.15.12:52914) 2024-09-11T01:52:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff644f68640 1 -- 172.21.15.12:0/1246747348 learned_addr learned my addr 172.21.15.12:0/1246747348 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff644f68640 10 monclient(hunting): get_auth_request con 0x7ff6400a2910 auth_method 0 2024-09-11T01:52:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff644f68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff644f68640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff644f68640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff644f68640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff644f68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff644f68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff644f68640 10 monclient(hunting): handle_auth_done global_id 244271 payload 274 2024-09-11T01:52:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff644f68640 1 -- 172.21.15.12:0/1246747348 >> v2:172.21.15.12:3300/0 conn(0x7ff6400a2d30 msgr2=0x7ff64013f570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:52:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff644f68640 1 --2- 172.21.15.12:0/1246747348 >> v2:172.21.15.12:3300/0 conn(0x7ff6400a2d30 0x7ff64013f570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff644f68640 1 -- 172.21.15.12:0/1246747348 >> v1:172.21.15.12:6789/0 conn(0x7ff6400a0440 legacy=0x7ff6400a2200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:52:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff644f68640 10 monclient: _finish_hunting 0 2024-09-11T01:52:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff644f68640 1 monclient: found mon.noname-b 2024-09-11T01:52:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff644f68640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:09.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff644f68640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:52:09.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.525+0000 7ff644f68640 1 -- 172.21.15.12:0/1246747348 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff640069950 con 0x7ff6400a2910 2024-09-11T01:52:09.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.528+0000 7ff644f68640 1 -- 172.21.15.12:0/1246747348 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6400a2910 msgr2=0x7ff64013d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:09.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.528+0000 7ff644f68640 1 --2- 172.21.15.12:0/1246747348 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6400a2910 0x7ff64013d020 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff6340098a0 tx=0x7ff63402a0e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbe4ccd964f9514b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:09.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.528+0000 7ff63ffff640 1 -- 172.21.15.12:0/1246747348 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff634037070 con 0x7ff6400a2910 2024-09-11T01:52:09.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.528+0000 7ff63ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:09.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.528+0000 7ff63ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:52:09.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.528+0000 7ff63ffff640 10 monclient: dump: 2024-09-11T01:52:09.539 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:09.539 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:09.539 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:09.539 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:09.539 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:09.539 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:09.539 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:09.539 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:09.539 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:09.539 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:09.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.528+0000 7ff63ffff640 10 monclient: _finish_auth 0 2024-09-11T01:52:09.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.528+0000 7ff63ffff640 10 monclient: _check_auth_tickets 2024-09-11T01:52:09.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.528+0000 7ff63ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:09.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.528+0000 7ff6479cd640 5 monclient: authenticate success, global_id 244271 2024-09-11T01:52:09.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.528+0000 7ff6479cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:52:09.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.528+0000 7ff63ffff640 1 -- 172.21.15.12:0/1246747348 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff634032640 con 0x7ff6400a2910 2024-09-11T01:52:09.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.528+0000 7ff63ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:09.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.528+0000 7ff63ffff640 1 -- 172.21.15.12:0/1246747348 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff634032a10 con 0x7ff6400a2910 2024-09-11T01:52:09.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.528+0000 7ff63ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:09.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.528+0000 7ff63ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:09.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.528+0000 7ff63ffff640 10 monclient: dump: 2024-09-11T01:52:09.540 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:09.540 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:09.541 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:09.541 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:09.541 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:09.541 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:09.541 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:09.541 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:09.541 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:09.541 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:09.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.528+0000 7ff6479cd640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:09.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.528+0000 7ff6479cd640 4 set_mon_vals no callback set 2024-09-11T01:52:09.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.529+0000 7ff6479cd640 10 monclient: shutdown 2024-09-11T01:52:09.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.529+0000 7ff6479cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:09.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.529+0000 7ff6479cd640 1 -- 172.21.15.12:0/1246747348 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6400a2910 msgr2=0x7ff64013d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:09.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.529+0000 7ff6479cd640 1 --2- 172.21.15.12:0/1246747348 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6400a2910 0x7ff64013d020 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff6340098a0 tx=0x7ff63402a0e0 comp rx=0 tx=0).stop 2024-09-11T01:52:09.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.529+0000 7ff6479cd640 1 -- 172.21.15.12:0/1246747348 shutdown_connections 2024-09-11T01:52:09.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.529+0000 7ff6479cd640 1 --2- 172.21.15.12:0/1246747348 >> v2:172.21.15.12:3300/0 conn(0x7ff6400a2d30 0x7ff64013f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:09.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.529+0000 7ff6479cd640 1 --2- 172.21.15.12:0/1246747348 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6400a2910 0x7ff64013d020 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:09.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.529+0000 7ff6479cd640 1 -- 172.21.15.12:0/1246747348 >> 172.21.15.12:0/1246747348 conn(0x7ff6400a3de0 msgr2=0x7ff640068950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:09.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.529+0000 7ff6479cd640 1 -- 172.21.15.12:0/1246747348 shutdown_connections 2024-09-11T01:52:09.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.529+0000 7ff6479cd640 1 -- 172.21.15.12:0/1246747348 wait complete. 2024-09-11T01:52:09.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.530+0000 7ff6479cd640 10 monclient: build_initial_monmap 2024-09-11T01:52:09.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.530+0000 7ff6479cd640 10 monclient: monmap: 2024-09-11T01:52:09.542 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:09.542 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:09.542 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:09.542 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:09.542 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:09.543 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:09.543 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:09.543 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:09.543 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:09.543 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:09.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.530+0000 7ff6479cd640 1 Processor -- start 2024-09-11T01:52:09.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.530+0000 7ff6479cd640 1 -- start start 2024-09-11T01:52:09.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.530+0000 7ff6479cd640 10 monclient: init 2024-09-11T01:52:09.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.530+0000 7ff6479cd640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:09.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff6479cd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:52:09.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff6479cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6400a0440 0x7ff6401312d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:09.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff6479cd640 10 monclient(hunting): picked mon.noname-a con 0x7ff6400a0440 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:09.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff6479cd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6400a2910 0x7ff640131810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:09.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff6479cd640 10 monclient(hunting): picked mon.noname-b con 0x7ff6400a2910 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:09.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff6479cd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6400a2d30 0x7ff6401ccf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:09.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff645769640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6400a0440 msgr2=0x7ff6401312d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:52:09.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff6479cd640 10 monclient(hunting): picked mon.noname-c con 0x7ff6400a2d30 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:09.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff6479cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:09.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff6479cd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff64013fab0 con 0x7ff6400a0440 2024-09-11T01:52:09.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff645769640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6400a0440 0x7ff6401312d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:09.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff6479cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:09.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff6479cd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff640070240 con 0x7ff6400a2910 2024-09-11T01:52:09.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff6479cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:09.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff6479cd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff64013fc30 con 0x7ff6400a2d30 2024-09-11T01:52:09.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff6479cd640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:09.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff6479cd640 10 monclient(hunting): authenticate will time out at 2135.973130s 2024-09-11T01:52:09.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff645f6a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6400a2d30 0x7ff6401ccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:09.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff645f6a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6400a2d30 0x7ff6401ccf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52924/0 (socket says 172.21.15.12:52924) 2024-09-11T01:52:09.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff645f6a640 1 -- 172.21.15.12:0/2201793503 learned_addr learned my addr 172.21.15.12:0/2201793503 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:09.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff645f6a640 10 monclient(hunting): get_auth_request con 0x7ff6400a2d30 auth_method 0 2024-09-11T01:52:09.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff645f6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:09.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff645f6a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:09.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff645f6a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:09.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff645f6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:09.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff645f6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:09.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff645f6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:09.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff644f68640 1 --2- 172.21.15.12:0/2201793503 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6400a2910 0x7ff640131810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:09.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff644f68640 10 monclient(hunting): get_auth_request con 0x7ff6400a2910 auth_method 0 2024-09-11T01:52:09.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff644f68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:09.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff644f68640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:09.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff644f68640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:09.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff644f68640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:09.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff644f68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:09.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff644f68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:09.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff645f6a640 10 monclient(hunting): handle_auth_done global_id 244274 payload 824 2024-09-11T01:52:09.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff645f6a640 1 -- 172.21.15.12:0/2201793503 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6400a2910 msgr2=0x7ff640131810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:09.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff645f6a640 1 --2- 172.21.15.12:0/2201793503 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6400a2910 0x7ff640131810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:09.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff645f6a640 1 -- 172.21.15.12:0/2201793503 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6400a0440 msgr2=0x7ff6401312d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:52:09.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff645f6a640 1 --2- 172.21.15.12:0/2201793503 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6400a0440 0x7ff6401312d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:09.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff645f6a640 10 monclient: _finish_hunting 0 2024-09-11T01:52:09.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff645f6a640 1 monclient: found mon.noname-c 2024-09-11T01:52:09.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff645f6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:09.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff645f6a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:52:09.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.531+0000 7ff645f6a640 1 -- 172.21.15.12:0/2201793503 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff64013dc10 con 0x7ff6400a2d30 2024-09-11T01:52:09.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.533+0000 7ff645f6a640 1 -- 172.21.15.12:0/2201793503 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6400a2d30 msgr2=0x7ff6401ccf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:09.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.533+0000 7ff645f6a640 1 --2- 172.21.15.12:0/2201793503 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6400a2d30 0x7ff6401ccf50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff63800b410 tx=0x7ff63800b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:09.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.533+0000 7ff63dffb640 1 -- 172.21.15.12:0/2201793503 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff638017070 con 0x7ff6400a2d30 2024-09-11T01:52:09.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.533+0000 7ff63dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:09.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.533+0000 7ff63dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:09.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.533+0000 7ff63dffb640 10 monclient: dump: 2024-09-11T01:52:09.548 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:09.548 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:09.548 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:09.548 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:09.549 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:09.549 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:09.549 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:09.549 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:09.549 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:09.549 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:09.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.533+0000 7ff63dffb640 10 monclient: _finish_auth 0 2024-09-11T01:52:09.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.533+0000 7ff63dffb640 10 monclient: _check_auth_tickets 2024-09-11T01:52:09.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.533+0000 7ff63dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:09.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.533+0000 7ff63dffb640 1 -- 172.21.15.12:0/2201793503 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6380060d0 con 0x7ff6400a2d30 2024-09-11T01:52:09.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.533+0000 7ff6479cd640 5 monclient: authenticate success, global_id 244274 2024-09-11T01:52:09.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.533+0000 7ff6479cd640 10 monclient: _renew_subs 2024-09-11T01:52:09.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.533+0000 7ff6479cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:09.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.533+0000 7ff6479cd640 1 -- 172.21.15.12:0/2201793503 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff640073770 con 0x7ff6400a2d30 2024-09-11T01:52:09.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.533+0000 7ff63dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:09.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.534+0000 7ff63dffb640 1 -- 172.21.15.12:0/2201793503 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6380064c0 con 0x7ff6400a2d30 2024-09-11T01:52:09.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.534+0000 7ff63dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:09.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.534+0000 7ff63dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:09.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.534+0000 7ff63dffb640 10 monclient: dump: 2024-09-11T01:52:09.550 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:09.550 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:09.550 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:09.550 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:09.550 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:09.551 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:09.551 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:09.551 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:09.551 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:09.551 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:09.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.534+0000 7ff6479cd640 10 monclient: _renew_subs 2024-09-11T01:52:09.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.534+0000 7ff6479cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:09.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.534+0000 7ff6479cd640 1 -- 172.21.15.12:0/2201793503 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff64013c8a0 con 0x7ff6400a2d30 2024-09-11T01:52:09.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.534+0000 7ff63f7fe640 4 set_mon_vals no callback set 2024-09-11T01:52:09.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.534+0000 7ff63dffb640 1 -- 172.21.15.12:0/2201793503 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff6380066a0 con 0x7ff6400a2d30 2024-09-11T01:52:09.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.535+0000 7ff63dffb640 1 --2- 172.21.15.12:0/2201793503 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff62c03e210 0x7ff62c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:09.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.535+0000 7ff63dffb640 1 -- 172.21.15.12:0/2201793503 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(766..766 src has 676..766) ==== 4803+0+0 (secure 0 0 0) 0x7ff638052360 con 0x7ff6400a2d30 2024-09-11T01:52:09.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.535+0000 7ff6479cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:09.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.535+0000 7ff6479cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:09.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.535+0000 7ff6479cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:09.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.535+0000 7ff6479cd640 1 -- 172.21.15.12:0/2201793503 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff640072870 con 0x7ff6400a2d30 2024-09-11T01:52:09.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.535+0000 7ff645769640 1 --2- 172.21.15.12:0/2201793503 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff62c03e210 0x7ff62c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:09.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.535+0000 7ff645769640 10 monclient: get_auth_request con 0x7ff62c03e210 auth_method 0 2024-09-11T01:52:09.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.538+0000 7ff645769640 1 --2- 172.21.15.12:0/2201793503 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff62c03e210 0x7ff62c0406d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff630003d00 tx=0x7ff630009270 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:09.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.540+0000 7ff63dffb640 1 -- 172.21.15.12:0/2201793503 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff640072870 con 0x7ff6400a2d30 2024-09-11T01:52:09.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.540+0000 7ff63dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:09.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.540+0000 7ff63dffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:09.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.797+0000 7ff6479cd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:52:09.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.797+0000 7ff6479cd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:52:09.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.797+0000 7ff6479cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:09.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.797+0000 7ff6479cd640 1 -- 172.21.15.12:0/2201793503 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff6400a2280 con 0x7ff6400a2d30 2024-09-11T01:52:09.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.797+0000 7ff63dffb640 1 -- 172.21.15.12:0/2201793503 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7ff63801bd90 con 0x7ff6400a2d30 2024-09-11T01:52:09.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.797+0000 7ff63dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:52:09.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.797+0000 7ff63dffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:09.799 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":214,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:52:09.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.800+0000 7ff6479cd640 1 -- 172.21.15.12:0/2201793503 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff62c03e210 msgr2=0x7ff62c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:09.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.800+0000 7ff6479cd640 1 --2- 172.21.15.12:0/2201793503 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff62c03e210 0x7ff62c0406d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff630003d00 tx=0x7ff630009270 comp rx=0 tx=0).stop 2024-09-11T01:52:09.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.800+0000 7ff6479cd640 10 monclient: shutdown 2024-09-11T01:52:09.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.800+0000 7ff6479cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:09.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.800+0000 7ff6479cd640 1 -- 172.21.15.12:0/2201793503 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6400a2d30 msgr2=0x7ff6401ccf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:09.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.800+0000 7ff6479cd640 1 --2- 172.21.15.12:0/2201793503 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6400a2d30 0x7ff6401ccf50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff63800b410 tx=0x7ff63800b8d0 comp rx=0 tx=0).stop 2024-09-11T01:52:09.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.800+0000 7ff6479cd640 1 -- 172.21.15.12:0/2201793503 shutdown_connections 2024-09-11T01:52:09.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.800+0000 7ff6479cd640 1 --2- 172.21.15.12:0/2201793503 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6400a2d30 0x7ff6401ccf50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:09.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.800+0000 7ff6479cd640 1 --2- 172.21.15.12:0/2201793503 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6400a2910 0x7ff640131810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:09.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.800+0000 7ff6479cd640 1 --2- 172.21.15.12:0/2201793503 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff62c03e210 0x7ff62c0406d0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:09.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.800+0000 7ff6479cd640 1 --2- 172.21.15.12:0/2201793503 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6400a0440 0x7ff6401312d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:09.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.800+0000 7ff6479cd640 1 -- 172.21.15.12:0/2201793503 >> 172.21.15.12:0/2201793503 conn(0x7ff6400a3de0 msgr2=0x7ff640071ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:09.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.800+0000 7ff6479cd640 1 -- 172.21.15.12:0/2201793503 shutdown_connections 2024-09-11T01:52:09.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:09.800+0000 7ff6479cd640 1 -- 172.21.15.12:0/2201793503 wait complete. 2024-09-11T01:52:12.813 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:52:12.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.923+0000 7fe2f4802640 10 monclient: get_monmap_and_config 2024-09-11T01:52:12.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.923+0000 7fe2f4802640 10 monclient: build_initial_monmap 2024-09-11T01:52:12.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.923+0000 7fe2f4802640 10 monclient: monmap: 2024-09-11T01:52:12.926 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:12.926 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:12.926 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:12.925334+0000 2024-09-11T01:52:12.926 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:12.925334+0000 2024-09-11T01:52:12.927 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:12.927 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:12.927 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:12.927 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:12.927 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:12.927 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:12.927 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:12.927 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:12.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.924+0000 7fe2f4802640 1 Processor -- start 2024-09-11T01:52:12.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.924+0000 7fe2f4802640 1 -- start start 2024-09-11T01:52:12.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.924+0000 7fe2f4802640 10 monclient: init 2024-09-11T01:52:12.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.924+0000 7fe2f4802640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:12.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.924+0000 7fe2f4802640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-11T01:52:12.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.924+0000 7fe2f4802640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe2f0138030 0x7fe2f009ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:12.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.925+0000 7fe2f4802640 10 monclient(hunting): picked mon.noname-a con 0x7fe2f0138030 addr v2:172.21.15.12:3300/0 2024-09-11T01:52:12.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.925+0000 7fe2f4802640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f0138410 0x7fe2f00a4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:12.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.925+0000 7fe2f4802640 10 monclient(hunting): picked mon.noname-b con 0x7fe2f0138410 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:12.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.925+0000 7fe2f4802640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe2f0138830 addr v1:172.21.15.12:6789/0 2024-09-11T01:52:12.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.925+0000 7fe2f4802640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2f009c770 con 0x7fe2f0138830 2024-09-11T01:52:12.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.925+0000 7fe2f4802640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:12.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.925+0000 7fe2f4802640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fe2f00a6710 con 0x7fe2f0138030 2024-09-11T01:52:12.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.925+0000 7fe2f4802640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:12.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.925+0000 7fe2f4802640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe2f00a6df0 con 0x7fe2f0138410 2024-09-11T01:52:12.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.925+0000 7fe2f4802640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:12.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.925+0000 7fe2eed9d640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7fe2f0138030 msgr2=0x7fe2f009ce70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:52:12.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.925+0000 7fe2f4802640 10 monclient(hunting): authenticate will time out at 2139.368095s 2024-09-11T01:52:12.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.925+0000 7fe2ef59e640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7fe2f0138830 legacy=0x7fe2f00a5460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:52:12.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.925+0000 7fe2eed9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe2f0138030 0x7fe2f009ce70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:12.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.925+0000 7fe2ee59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f0138410 0x7fe2f00a4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:12.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.925+0000 7fe2ee59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f0138410 0x7fe2f00a4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52930/0 (socket says 172.21.15.12:52930) 2024-09-11T01:52:12.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.925+0000 7fe2ee59c640 1 -- 172.21.15.12:0/1475787491 learned_addr learned my addr 172.21.15.12:0/1475787491 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:12.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.925+0000 7fe2ee59c640 10 monclient(hunting): get_auth_request con 0x7fe2f0138410 auth_method 0 2024-09-11T01:52:12.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.925+0000 7fe2ee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:12.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.925+0000 7fe2ee59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:12.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.925+0000 7fe2ee59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:12.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.925+0000 7fe2ee59c640 1 -- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f0138410 msgr2=0x7fe2f00a4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:52:12.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.925+0000 7fe2ee59c640 1 -- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f0138410 msgr2=0x7fe2f00a4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:52:12.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.925+0000 7fe2ee59c640 1 --2- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f0138410 0x7fe2f00a4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:52:12.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:12.925+0000 7fe2ee59c640 1 --2- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f0138410 0x7fe2f00a4f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:13.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:13.126+0000 7fe2ef59e640 1 -- 172.21.15.12:0/1475787491 >> v1:172.21.15.12:6789/0 conn(0x7fe2f0138830 legacy=0x7fe2f00a5460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:52:13.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:13.126+0000 7fe2eed9d640 1 -- 172.21.15.12:0/1475787491 >> v2:172.21.15.12:3300/0 conn(0x7fe2f0138030 msgr2=0x7fe2f009ce70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:52:13.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:13.126+0000 7fe2eed9d640 1 --2- 172.21.15.12:0/1475787491 >> v2:172.21.15.12:3300/0 conn(0x7fe2f0138030 0x7fe2f009ce70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:52:13.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:13.126+0000 7fe2ee59c640 1 --2- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f0138410 0x7fe2f00a4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:13.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:13.126+0000 7fe2ee59c640 10 monclient(hunting): get_auth_request con 0x7fe2f0138410 auth_method 0 2024-09-11T01:52:13.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:13.126+0000 7fe2ee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:13.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:13.126+0000 7fe2ee59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:13.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:13.126+0000 7fe2ee59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:52:13.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:13.126+0000 7fe2ee59c640 1 -- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f0138410 msgr2=0x7fe2f00a4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:52:13.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:13.126+0000 7fe2ee59c640 1 -- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f0138410 msgr2=0x7fe2f00a4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:52:13.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:13.126+0000 7fe2ee59c640 1 --2- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f0138410 0x7fe2f00a4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:52:13.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:13.126+0000 7fe2ee59c640 1 --2- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f0138410 0x7fe2f00a4f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:52:13.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:13.527+0000 7fe2eed9d640 1 -- 172.21.15.12:0/1475787491 >> v2:172.21.15.12:3300/0 conn(0x7fe2f0138030 msgr2=0x7fe2f009ce70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:52:13.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:13.527+0000 7fe2ef59e640 1 -- 172.21.15.12:0/1475787491 >> v1:172.21.15.12:6789/0 conn(0x7fe2f0138830 legacy=0x7fe2f00a5460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:52:13.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:13.527+0000 7fe2eed9d640 1 --2- 172.21.15.12:0/1475787491 >> v2:172.21.15.12:3300/0 conn(0x7fe2f0138030 0x7fe2f009ce70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:52:13.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:13.527+0000 7fe2ee59c640 1 --2- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f0138410 0x7fe2f00a4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:13.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:13.527+0000 7fe2ee59c640 10 monclient(hunting): get_auth_request con 0x7fe2f0138410 auth_method 0 2024-09-11T01:52:13.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:13.527+0000 7fe2ee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:13.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:13.527+0000 7fe2ee59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:13.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:13.527+0000 7fe2ee59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:52:13.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:13.527+0000 7fe2ee59c640 1 -- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f0138410 msgr2=0x7fe2f00a4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:52:13.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:13.527+0000 7fe2ee59c640 1 -- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f0138410 msgr2=0x7fe2f00a4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:52:13.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:13.527+0000 7fe2ee59c640 1 --2- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f0138410 0x7fe2f00a4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:52:13.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:13.527+0000 7fe2ee59c640 1 --2- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f0138410 0x7fe2f00a4f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:52:14.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:14.328+0000 7fe2ef59e640 1 -- 172.21.15.12:0/1475787491 >> v1:172.21.15.12:6789/0 conn(0x7fe2f0138830 legacy=0x7fe2f00a5460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:52:14.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:14.328+0000 7fe2eed9d640 1 -- 172.21.15.12:0/1475787491 >> v2:172.21.15.12:3300/0 conn(0x7fe2f0138030 msgr2=0x7fe2f009ce70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:52:14.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:14.328+0000 7fe2eed9d640 1 --2- 172.21.15.12:0/1475787491 >> v2:172.21.15.12:3300/0 conn(0x7fe2f0138030 0x7fe2f009ce70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:52:14.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:14.328+0000 7fe2ee59c640 1 --2- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f0138410 0x7fe2f00a4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:14.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:14.329+0000 7fe2ee59c640 10 monclient(hunting): get_auth_request con 0x7fe2f0138410 auth_method 0 2024-09-11T01:52:14.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:14.329+0000 7fe2ee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:14.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:14.329+0000 7fe2ee59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:14.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:14.329+0000 7fe2ee59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:52:14.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:14.329+0000 7fe2ee59c640 1 -- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f0138410 msgr2=0x7fe2f00a4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:52:14.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:14.329+0000 7fe2ee59c640 1 -- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f0138410 msgr2=0x7fe2f00a4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:52:14.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:14.329+0000 7fe2ee59c640 1 --2- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f0138410 0x7fe2f00a4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:52:14.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:14.329+0000 7fe2ee59c640 1 --2- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f0138410 0x7fe2f00a4f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:52:15.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.925+0000 7fe2ecd99640 10 monclient(hunting): tick 2024-09-11T01:52:15.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.925+0000 7fe2ecd99640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:52:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.925+0000 7fe2ecd99640 1 monclient(hunting): continuing hunt 2024-09-11T01:52:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.925+0000 7fe2ecd99640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:52:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.925+0000 7fe2ecd99640 1 -- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f0138410 msgr2=0x7fe2f00a4f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:52:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.925+0000 7fe2ecd99640 1 --2- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f0138410 0x7fe2f00a4f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.925+0000 7fe2ecd99640 1 -- 172.21.15.12:0/1475787491 >> v2:172.21.15.12:3300/0 conn(0x7fe2f0138030 msgr2=0x7fe2f009ce70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:52:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.925+0000 7fe2ecd99640 1 --2- 172.21.15.12:0/1475787491 >> v2:172.21.15.12:3300/0 conn(0x7fe2f0138030 0x7fe2f009ce70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.925+0000 7fe2ecd99640 1 -- 172.21.15.12:0/1475787491 >> v1:172.21.15.12:6789/0 conn(0x7fe2f0138830 legacy=0x7fe2f00a5460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:52:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.925+0000 7fe2ecd99640 10 monclient: _add_conns ranks=[4,1] 2024-09-11T01:52:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.925+0000 7fe2ecd99640 1 --2- 172.21.15.12:0/1475787491 >> v2:172.21.15.71:3300/0 conn(0x7fe2d00036c0 0x7fe2d0005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.925+0000 7fe2ecd99640 10 monclient(hunting): picked mon.noname-c con 0x7fe2d00036c0 addr v2:172.21.15.71:3300/0 2024-09-11T01:52:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.925+0000 7fe2ecd99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe2d0006210 addr v1:172.21.15.71:6789/0 2024-09-11T01:52:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.925+0000 7fe2ecd99640 1 -- 172.21.15.12:0/1475787491 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2f009ca50 con 0x7fe2d0006210 2024-09-11T01:52:15.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.925+0000 7fe2ecd99640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:15.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.925+0000 7fe2ecd99640 1 -- 172.21.15.12:0/1475787491 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fe2f00a6df0 con 0x7fe2d00036c0 2024-09-11T01:52:15.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.925+0000 7fe2ecd99640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:15.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.925+0000 7fe2eed9d640 1 --2- 172.21.15.12:0/1475787491 >> v2:172.21.15.71:3300/0 conn(0x7fe2d00036c0 0x7fe2d0005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:15.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.925+0000 7fe2eed9d640 10 monclient(hunting): get_auth_request con 0x7fe2d00036c0 auth_method 0 2024-09-11T01:52:15.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.925+0000 7fe2eed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:15.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.925+0000 7fe2eed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:15.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.925+0000 7fe2eed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.925+0000 7fe2eed9d640 1 -- 172.21.15.12:0/1475787491 >> v2:172.21.15.71:3300/0 conn(0x7fe2d00036c0 msgr2=0x7fe2d0005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:52:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.925+0000 7fe2eed9d640 1 -- 172.21.15.12:0/1475787491 >> v2:172.21.15.71:3300/0 conn(0x7fe2d00036c0 msgr2=0x7fe2d0005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:52:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.925+0000 7fe2eed9d640 1 --2- 172.21.15.12:0/1475787491 >> v2:172.21.15.71:3300/0 conn(0x7fe2d00036c0 0x7fe2d0005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:52:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.926+0000 7fe2eed9d640 1 --2- 172.21.15.12:0/1475787491 >> v2:172.21.15.71:3300/0 conn(0x7fe2d00036c0 0x7fe2d0005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:15.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:15.928+0000 7fe2ee59c640 1 -- 172.21.15.12:0/1475787491 >> v1:172.21.15.71:6789/0 conn(0x7fe2d0006210 legacy=0x7fe2d00086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:16.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:16.127+0000 7fe2eed9d640 1 --2- 172.21.15.12:0/1475787491 >> v2:172.21.15.71:3300/0 conn(0x7fe2d00036c0 0x7fe2d0005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:16.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:16.127+0000 7fe2eed9d640 10 monclient(hunting): get_auth_request con 0x7fe2d00036c0 auth_method 0 2024-09-11T01:52:16.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:16.127+0000 7fe2eed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:16.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:16.127+0000 7fe2eed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:16.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:16.127+0000 7fe2eed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:52:16.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:16.127+0000 7fe2eed9d640 1 -- 172.21.15.12:0/1475787491 >> v2:172.21.15.71:3300/0 conn(0x7fe2d00036c0 msgr2=0x7fe2d0005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:52:16.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:16.127+0000 7fe2eed9d640 1 -- 172.21.15.12:0/1475787491 >> v2:172.21.15.71:3300/0 conn(0x7fe2d00036c0 msgr2=0x7fe2d0005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:52:16.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:16.127+0000 7fe2eed9d640 1 --2- 172.21.15.12:0/1475787491 >> v2:172.21.15.71:3300/0 conn(0x7fe2d00036c0 0x7fe2d0005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:52:16.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:16.127+0000 7fe2eed9d640 1 --2- 172.21.15.12:0/1475787491 >> v2:172.21.15.71:3300/0 conn(0x7fe2d00036c0 0x7fe2d0005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:52:16.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:16.528+0000 7fe2eed9d640 1 --2- 172.21.15.12:0/1475787491 >> v2:172.21.15.71:3300/0 conn(0x7fe2d00036c0 0x7fe2d0005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:16.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:16.528+0000 7fe2eed9d640 10 monclient(hunting): get_auth_request con 0x7fe2d00036c0 auth_method 0 2024-09-11T01:52:16.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:16.528+0000 7fe2eed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:16.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:16.528+0000 7fe2eed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:16.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:16.528+0000 7fe2eed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:52:16.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:16.528+0000 7fe2eed9d640 1 -- 172.21.15.12:0/1475787491 >> v2:172.21.15.71:3300/0 conn(0x7fe2d00036c0 msgr2=0x7fe2d0005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:52:16.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:16.528+0000 7fe2eed9d640 1 -- 172.21.15.12:0/1475787491 >> v2:172.21.15.71:3300/0 conn(0x7fe2d00036c0 msgr2=0x7fe2d0005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:52:16.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:16.528+0000 7fe2eed9d640 1 --2- 172.21.15.12:0/1475787491 >> v2:172.21.15.71:3300/0 conn(0x7fe2d00036c0 0x7fe2d0005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:52:16.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:16.528+0000 7fe2eed9d640 1 --2- 172.21.15.12:0/1475787491 >> v2:172.21.15.71:3300/0 conn(0x7fe2d00036c0 0x7fe2d0005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:52:17.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.329+0000 7fe2eed9d640 1 --2- 172.21.15.12:0/1475787491 >> v2:172.21.15.71:3300/0 conn(0x7fe2d00036c0 0x7fe2d0005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:17.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.329+0000 7fe2eed9d640 10 monclient(hunting): get_auth_request con 0x7fe2d00036c0 auth_method 0 2024-09-11T01:52:17.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.329+0000 7fe2eed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:17.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.329+0000 7fe2eed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:17.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.329+0000 7fe2eed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:52:17.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.329+0000 7fe2eed9d640 1 -- 172.21.15.12:0/1475787491 >> v2:172.21.15.71:3300/0 conn(0x7fe2d00036c0 msgr2=0x7fe2d0005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:52:17.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.329+0000 7fe2eed9d640 1 -- 172.21.15.12:0/1475787491 >> v2:172.21.15.71:3300/0 conn(0x7fe2d00036c0 msgr2=0x7fe2d0005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:52:17.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.329+0000 7fe2eed9d640 1 --2- 172.21.15.12:0/1475787491 >> v2:172.21.15.71:3300/0 conn(0x7fe2d00036c0 0x7fe2d0005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:52:17.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.329+0000 7fe2eed9d640 1 --2- 172.21.15.12:0/1475787491 >> v2:172.21.15.71:3300/0 conn(0x7fe2d00036c0 0x7fe2d0005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:52:17.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.578+0000 7fe2edd9b640 1 -- 172.21.15.12:0/1475787491 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1885140187 0 0) 0x7fe2f009ca50 con 0x7fe2d0006210 2024-09-11T01:52:17.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.578+0000 7fe2edd9b640 10 monclient(hunting): _negotiate 2024-09-11T01:52:17.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.578+0000 7fe2edd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:17.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.578+0000 7fe2edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:17.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.578+0000 7fe2edd9b640 10 monclient(hunting): my global_id is 234289 2024-09-11T01:52:17.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.578+0000 7fe2edd9b640 1 -- 172.21.15.12:0/1475787491 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2dc003520 con 0x7fe2d0006210 2024-09-11T01:52:17.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 1 -- 172.21.15.12:0/1475787491 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1036909850 0 0) 0x7fe2dc003520 con 0x7fe2d0006210 2024-09-11T01:52:17.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 1 -- 172.21.15.12:0/1475787491 >> v2:172.21.15.71:3300/0 conn(0x7fe2d00036c0 msgr2=0x7fe2d0005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:52:17.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 1 --2- 172.21.15.12:0/1475787491 >> v2:172.21.15.71:3300/0 conn(0x7fe2d00036c0 0x7fe2d0005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:17.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 10 monclient: _finish_hunting 0 2024-09-11T01:52:17.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 1 monclient: found mon.noname-c-legacy 2024-09-11T01:52:17.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:17.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-09-11T01:52:17.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 1 -- 172.21.15.12:0/1475787491 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f009c770 con 0x7fe2d0006210 2024-09-11T01:52:17.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 10 monclient: _finish_auth 0 2024-09-11T01:52:17.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 10 monclient: _check_auth_tickets 2024-09-11T01:52:17.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:17.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 1 -- 172.21.15.12:0/1475787491 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7fe2d800f040 con 0x7fe2d0006210 2024-09-11T01:52:17.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2f4802640 5 monclient: authenticate success, global_id 234289 2024-09-11T01:52:17.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2f4802640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:52:17.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:17.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-11T01:52:17.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 10 monclient: dump: 2024-09-11T01:52:17.587 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:17.588 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:17.588 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:17.588 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:17.588 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:17.588 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:17.588 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:17.588 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:17.588 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:17.588 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:17.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-09-11T01:52:17.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:17.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 1 -- 172.21.15.12:0/1475787491 >> v1:172.21.15.71:6789/0 conn(0x7fe2d0006210 legacy=0x7fe2d00086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:17.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:52:17.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2ef59e640 1 -- 172.21.15.12:0/1475787491 reap_dead start 2024-09-11T01:52:17.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 1 --2- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2dc0039b0 0x7fe2dc003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:17.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 10 monclient(hunting): picked mon.c con 0x7fe2dc0039b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:17.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 1 --2- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2dc004800 0x7fe2dc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:17.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 10 monclient(hunting): picked mon.a con 0x7fe2dc004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:17.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 1 --2- 172.21.15.12:0/1475787491 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2dc0052e0 0x7fe2dc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:17.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 10 monclient(hunting): picked mon.b con 0x7fe2dc0052e0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:17.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:17.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 1 -- 172.21.15.12:0/1475787491 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f00a6df0 con 0x7fe2dc004800 2024-09-11T01:52:17.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:17.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 1 -- 172.21.15.12:0/1475787491 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe2dc00c230 con 0x7fe2dc0052e0 2024-09-11T01:52:17.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:17.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 1 -- 172.21.15.12:0/1475787491 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe2dc00c430 con 0x7fe2dc0039b0 2024-09-11T01:52:17.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:17.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2edd9b640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:52:17.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2f4802640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:52:17.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2ee59c640 1 -- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2dc004800 msgr2=0x7fe2dc004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:52:17.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2ee59c640 1 --2- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2dc004800 0x7fe2dc004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:17.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.581+0000 7fe2eed9d640 1 --2- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2dc0039b0 0x7fe2dc003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:17.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2ef59e640 1 --2- 172.21.15.12:0/1475787491 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2dc0052e0 0x7fe2dc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:17.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2eed9d640 10 monclient(hunting): get_auth_request con 0x7fe2dc0039b0 auth_method 0 2024-09-11T01:52:17.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2eed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:17.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2eed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:17.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2eed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:52:17.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2ef59e640 10 monclient(hunting): get_auth_request con 0x7fe2dc0052e0 auth_method 0 2024-09-11T01:52:17.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2ef59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:17.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2ef59e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:17.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2ef59e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:52:17.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2eed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:17.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2eed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:17.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2eed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:52:17.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2ef59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:17.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2ef59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:17.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2ef59e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:52:17.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2eed9d640 10 monclient(hunting): handle_auth_done global_id 234289 payload 293 2024-09-11T01:52:17.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2eed9d640 1 -- 172.21.15.12:0/1475787491 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2dc0052e0 msgr2=0x7fe2dc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:17.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2eed9d640 1 --2- 172.21.15.12:0/1475787491 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2dc0052e0 0x7fe2dc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:17.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2eed9d640 1 -- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2dc004800 msgr2=0x7fe2dc004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:52:17.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2eed9d640 1 --2- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2dc004800 0x7fe2dc004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:17.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2eed9d640 10 monclient: _finish_hunting 0 2024-09-11T01:52:17.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2eed9d640 1 monclient: found mon.c 2024-09-11T01:52:17.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2eed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:17.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2eed9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:17.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2eed9d640 1 -- 172.21.15.12:0/1475787491 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2dc003520 con 0x7fe2dc0039b0 2024-09-11T01:52:17.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2eed9d640 10 monclient: _finish_auth 0 2024-09-11T01:52:17.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2eed9d640 10 monclient: _check_auth_tickets 2024-09-11T01:52:17.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2eed9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:17.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.582+0000 7fe2ef59e640 1 --2- 172.21.15.12:0/1475787491 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2dc0052e0 0x7fe2dc00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:52:17.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.585+0000 7fe2eed9d640 1 -- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2dc0039b0 msgr2=0x7fe2dc003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:17.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.585+0000 7fe2eed9d640 1 --2- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2dc0039b0 0x7fe2dc003db0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8003980 tx=0x7fe2dc005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc3bb38af624d573 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:17.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.585+0000 7fe2edd9b640 1 -- 172.21.15.12:0/1475787491 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e8005130 con 0x7fe2dc0039b0 2024-09-11T01:52:17.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.585+0000 7fe2edd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:17.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.585+0000 7fe2edd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:17.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.585+0000 7fe2edd9b640 10 monclient: dump: 2024-09-11T01:52:17.600 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:17.600 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:17.600 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:17.600 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:17.600 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:17.600 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:17.600 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:17.600 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:17.601 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:17.601 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:17.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.585+0000 7fe2edd9b640 1 -- 172.21.15.12:0/1475787491 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2e8035430 con 0x7fe2dc0039b0 2024-09-11T01:52:17.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.585+0000 7fe2edd9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:17.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.585+0000 7fe2edd9b640 1 -- 172.21.15.12:0/1475787491 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e8035820 con 0x7fe2dc0039b0 2024-09-11T01:52:17.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.585+0000 7fe2f4802640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:17.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.585+0000 7fe2f4802640 4 set_mon_vals no callback set 2024-09-11T01:52:17.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.585+0000 7fe2edd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:17.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.585+0000 7fe2edd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:17.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.585+0000 7fe2edd9b640 10 monclient: dump: 2024-09-11T01:52:17.601 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:17.602 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:17.602 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:17.602 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:17.602 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:17.602 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:17.602 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:17.602 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:17.602 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:17.602 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:17.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.586+0000 7fe2f4802640 10 monclient: shutdown 2024-09-11T01:52:17.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.586+0000 7fe2f4802640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:17.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.586+0000 7fe2f4802640 1 -- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2dc0039b0 msgr2=0x7fe2dc003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:17.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.586+0000 7fe2f4802640 1 --2- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2dc0039b0 0x7fe2dc003db0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8003980 tx=0x7fe2dc005830 comp rx=0 tx=0).stop 2024-09-11T01:52:17.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.586+0000 7fe2f4802640 1 -- 172.21.15.12:0/1475787491 shutdown_connections 2024-09-11T01:52:17.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.586+0000 7fe2f4802640 1 --2- 172.21.15.12:0/1475787491 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2dc0052e0 0x7fe2dc00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:17.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.586+0000 7fe2f4802640 1 --2- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2dc004800 0x7fe2dc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:17.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.586+0000 7fe2f4802640 1 --2- 172.21.15.12:0/1475787491 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2dc0039b0 0x7fe2dc003db0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:17.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.586+0000 7fe2f4802640 1 -- 172.21.15.12:0/1475787491 >> 172.21.15.12:0/1475787491 conn(0x7fe2f0068950 msgr2=0x7fe2f0068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:17.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.586+0000 7fe2f4802640 1 -- 172.21.15.12:0/1475787491 shutdown_connections 2024-09-11T01:52:17.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.586+0000 7fe2f4802640 1 -- 172.21.15.12:0/1475787491 wait complete. 2024-09-11T01:52:17.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.587+0000 7fe2f4802640 10 monclient: build_initial_monmap 2024-09-11T01:52:17.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.587+0000 7fe2f4802640 10 monclient: monmap: 2024-09-11T01:52:17.604 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:17.604 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:17.604 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:17.605 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:17.605 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:17.605 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:17.605 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:17.605 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:17.605 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:17.605 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:17.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.588+0000 7fe2f4802640 1 Processor -- start 2024-09-11T01:52:17.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.588+0000 7fe2f4802640 1 -- start start 2024-09-11T01:52:17.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.588+0000 7fe2f4802640 10 monclient: init 2024-09-11T01:52:17.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.588+0000 7fe2f4802640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:17.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.588+0000 7fe2f4802640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:52:17.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.588+0000 7fe2f4802640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2dc0039b0 0x7fe2f01ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:17.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.588+0000 7fe2f4802640 10 monclient(hunting): picked mon.noname-a con 0x7fe2dc0039b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:17.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.588+0000 7fe2f4802640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2dc004800 0x7fe2f01cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:17.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.588+0000 7fe2f4802640 10 monclient(hunting): picked mon.noname-b con 0x7fe2dc004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:17.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.588+0000 7fe2f4802640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2dc0052e0 0x7fe2f01244b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:17.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.588+0000 7fe2f4802640 10 monclient(hunting): picked mon.noname-c con 0x7fe2dc0052e0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:17.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.588+0000 7fe2f4802640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:17.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.588+0000 7fe2f4802640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f006f6d0 con 0x7fe2dc0039b0 2024-09-11T01:52:17.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.588+0000 7fe2f4802640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:17.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.588+0000 7fe2f4802640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f00a60b0 con 0x7fe2dc004800 2024-09-11T01:52:17.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.588+0000 7fe2f4802640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:17.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.588+0000 7fe2f4802640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe2f006f850 con 0x7fe2dc0052e0 2024-09-11T01:52:17.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.588+0000 7fe2f4802640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:17.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.588+0000 7fe2f4802640 10 monclient(hunting): authenticate will time out at 2144.031982s 2024-09-11T01:52:17.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.588+0000 7fe2eed9d640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2dc0039b0 msgr2=0x7fe2f01ccd60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:52:17.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2eed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2dc0039b0 0x7fe2f01ccd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:17.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ef59e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2dc0052e0 0x7fe2f01244b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:17.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ef59e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2dc0052e0 0x7fe2f01244b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:60698/0 (socket says 172.21.15.12:60698) 2024-09-11T01:52:17.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ef59e640 1 -- 172.21.15.12:0/1729981725 learned_addr learned my addr 172.21.15.12:0/1729981725 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:17.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ef59e640 10 monclient(hunting): get_auth_request con 0x7fe2dc0052e0 auth_method 0 2024-09-11T01:52:17.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ef59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:17.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ef59e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:17.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ef59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:17.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ef59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:17.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ef59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:17.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ef59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:17.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ee59c640 1 --2- 172.21.15.12:0/1729981725 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2dc004800 0x7fe2f01cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:17.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ee59c640 10 monclient(hunting): get_auth_request con 0x7fe2dc004800 auth_method 0 2024-09-11T01:52:17.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:17.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ee59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:17.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ee59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:17.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:17.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:17.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:17.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ef59e640 10 monclient(hunting): handle_auth_done global_id 244277 payload 824 2024-09-11T01:52:17.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ef59e640 1 -- 172.21.15.12:0/1729981725 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2dc004800 msgr2=0x7fe2f01cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:17.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ef59e640 1 --2- 172.21.15.12:0/1729981725 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2dc004800 0x7fe2f01cd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:17.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ef59e640 1 -- 172.21.15.12:0/1729981725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2dc0039b0 msgr2=0x7fe2f01ccd60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:52:17.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ef59e640 1 --2- 172.21.15.12:0/1729981725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2dc0039b0 0x7fe2f01ccd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:17.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ef59e640 10 monclient: _finish_hunting 0 2024-09-11T01:52:17.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ef59e640 1 monclient: found mon.noname-c 2024-09-11T01:52:17.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ef59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:17.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ef59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:52:17.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.589+0000 7fe2ef59e640 1 -- 172.21.15.12:0/1729981725 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f0137890 con 0x7fe2dc0052e0 2024-09-11T01:52:17.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.591+0000 7fe2ef59e640 1 -- 172.21.15.12:0/1729981725 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2dc0052e0 msgr2=0x7fe2f01244b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:17.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.591+0000 7fe2ef59e640 1 --2- 172.21.15.12:0/1729981725 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2dc0052e0 0x7fe2f01244b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0006180 tx=0x7fe2e000c730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:17.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.592+0000 7fe2e77fe640 1 -- 172.21.15.12:0/1729981725 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e000f860 con 0x7fe2dc0052e0 2024-09-11T01:52:17.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.592+0000 7fe2e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:17.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.592+0000 7fe2e77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:17.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.592+0000 7fe2e77fe640 10 monclient: dump: 2024-09-11T01:52:17.611 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:17.612 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:17.612 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:17.612 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:17.612 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:17.612 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:17.612 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:17.612 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:17.612 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:17.612 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:17.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.592+0000 7fe2e77fe640 10 monclient: _finish_auth 0 2024-09-11T01:52:17.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.592+0000 7fe2e77fe640 10 monclient: _check_auth_tickets 2024-09-11T01:52:17.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.592+0000 7fe2e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:17.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.592+0000 7fe2e77fe640 1 -- 172.21.15.12:0/1729981725 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2e0013030 con 0x7fe2dc0052e0 2024-09-11T01:52:17.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.592+0000 7fe2f4802640 5 monclient: authenticate success, global_id 244277 2024-09-11T01:52:17.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.592+0000 7fe2f4802640 10 monclient: _renew_subs 2024-09-11T01:52:17.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.592+0000 7fe2f4802640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:17.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.592+0000 7fe2f4802640 1 -- 172.21.15.12:0/1729981725 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2f00a1710 con 0x7fe2dc0052e0 2024-09-11T01:52:17.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.592+0000 7fe2e77fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:17.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.592+0000 7fe2e77fe640 1 -- 172.21.15.12:0/1729981725 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e000fc40 con 0x7fe2dc0052e0 2024-09-11T01:52:17.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.592+0000 7fe2e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:17.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.592+0000 7fe2e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:17.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.592+0000 7fe2e77fe640 10 monclient: dump: 2024-09-11T01:52:17.613 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:17.613 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:17.613 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:17.614 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:17.614 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:17.614 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:17.614 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:17.614 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:17.614 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:17.614 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:17.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.592+0000 7fe2ed59a640 4 set_mon_vals no callback set 2024-09-11T01:52:17.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.592+0000 7fe2f4802640 10 monclient: _renew_subs 2024-09-11T01:52:17.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.592+0000 7fe2f4802640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:17.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.592+0000 7fe2f4802640 1 -- 172.21.15.12:0/1729981725 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2f0127ff0 con 0x7fe2dc0052e0 2024-09-11T01:52:17.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.593+0000 7fe2e77fe640 1 -- 172.21.15.12:0/1729981725 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fe2e000c040 con 0x7fe2dc0052e0 2024-09-11T01:52:17.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.593+0000 7fe2f4802640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:17.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.593+0000 7fe2f4802640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:17.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.593+0000 7fe2f4802640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:17.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.593+0000 7fe2e77fe640 1 --2- 172.21.15.12:0/1729981725 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe2d003e9f0 0x7fe2d0040eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:17.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.593+0000 7fe2f4802640 1 -- 172.21.15.12:0/1729981725 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2f009d120 con 0x7fe2dc0052e0 2024-09-11T01:52:17.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.593+0000 7fe2e77fe640 1 -- 172.21.15.12:0/1729981725 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(766..766 src has 676..766) ==== 4803+0+0 (secure 0 0 0) 0x7fe2e0053450 con 0x7fe2dc0052e0 2024-09-11T01:52:17.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.593+0000 7fe2eed9d640 1 --2- 172.21.15.12:0/1729981725 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe2d003e9f0 0x7fe2d0040eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:17.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.593+0000 7fe2eed9d640 10 monclient: get_auth_request con 0x7fe2d003e9f0 auth_method 0 2024-09-11T01:52:17.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.596+0000 7fe2eed9d640 1 --2- 172.21.15.12:0/1729981725 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe2d003e9f0 0x7fe2d0040eb0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc0057e0 tx=0x7fe2e802d000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:17.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.598+0000 7fe2e77fe640 1 -- 172.21.15.12:0/1729981725 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe2e0009030 con 0x7fe2dc0052e0 2024-09-11T01:52:17.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.598+0000 7fe2e77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:17.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.598+0000 7fe2e77fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:17.640 INFO:tasks.rados.rados.0.smithi071.stdout:3412: done (1 left) 2024-09-11T01:52:17.641 INFO:tasks.rados.rados.0.smithi071.stdout:3413: done (0 left) 2024-09-11T01:52:17.641 INFO:tasks.rados.rados.0.smithi071.stdout:3414: read oid 13 snap 371 2024-09-11T01:52:17.641 INFO:tasks.rados.rados.0.smithi071.stdout:3414: expect deleted 2024-09-11T01:52:17.641 INFO:tasks.rados.rados.0.smithi071.stdout:3415: read oid 27 snap -1 2024-09-11T01:52:17.641 INFO:tasks.rados.rados.0.smithi071.stdout:3415: expect (ObjNum 1154 snap 366 seq_num 1154) 2024-09-11T01:52:17.641 INFO:tasks.rados.rados.0.smithi071.stdout:3416: copy_from oid 8 from oid 16 current snap is 381 2024-09-11T01:52:17.641 INFO:tasks.rados.rados.0.smithi071.stdout:3417: rollback oid 9 current snap is 381 2024-09-11T01:52:17.641 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 9 to 339 2024-09-11T01:52:17.641 INFO:tasks.rados.rados.0.smithi071.stdout:3418: read oid 4 snap 378 2024-09-11T01:52:17.641 INFO:tasks.rados.rados.0.smithi071.stdout:3418: expect deleted 2024-09-11T01:52:17.641 INFO:tasks.rados.rados.0.smithi071.stdout:3419: write oid 46 current snap is 381 2024-09-11T01:52:17.642 INFO:tasks.rados.rados.0.smithi071.stdout:3419: seq_num 1193 ranges {450751=723125,1945018=105733} 2024-09-11T01:52:17.646 INFO:tasks.rados.rados.0.smithi071.stdout:3419: writing smithi07134777-46 from 450751 to 1173876 tid 1 2024-09-11T01:52:17.652 INFO:tasks.rados.rados.0.smithi071.stdout:3419: writing smithi07134777-46 from 1945018 to 2050751 tid 2 2024-09-11T01:52:17.652 INFO:tasks.rados.rados.0.smithi071.stdout:3420: read oid 33 snap 284 2024-09-11T01:52:17.652 INFO:tasks.rados.rados.0.smithi071.stdout:3420: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:52:17.652 INFO:tasks.rados.rados.0.smithi071.stdout:3421: delete oid 3 current snap is 381 2024-09-11T01:52:17.652 INFO:tasks.rados.rados.0.smithi071.stdout:3417: finishing rollback tid 1 to smithi07134777-9 2024-09-11T01:52:17.652 INFO:tasks.rados.rados.0.smithi071.stdout:3417: finishing rollback tid 2 to smithi07134777-9 2024-09-11T01:52:17.652 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 1052 (ObjNum 601 snap 172 seq_num 601) dirty exists 2024-09-11T01:52:17.655 INFO:tasks.rados.rados.0.smithi071.stdout:3419: finishing write tid 1 to smithi07134777-46 2024-09-11T01:52:17.656 INFO:tasks.rados.rados.0.smithi071.stdout:3415: expect (ObjNum 1154 snap 366 seq_num 1154) 2024-09-11T01:52:17.664 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1630904 2024-09-11T01:52:17.726 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 466069 2024-09-11T01:52:17.771 INFO:tasks.rados.rados.0.smithi071.stdout:3419: finishing write tid 2 to smithi07134777-46 2024-09-11T01:52:17.771 INFO:tasks.rados.rados.0.smithi071.stdout:3419: finishing write tid 3 to smithi07134777-46 2024-09-11T01:52:17.771 INFO:tasks.rados.rados.0.smithi071.stdout:3420: expect (ObjNum 688 snap 204 seq_num 688) 2024-09-11T01:52:17.774 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1783660 2024-09-11T01:52:17.838 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:52:17.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.867+0000 7fe2f4802640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:52:17.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.867+0000 7fe2f4802640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:52:17.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.867+0000 7fe2f4802640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:17.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.867+0000 7fe2f4802640 1 -- 172.21.15.12:0/1729981725 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe2f0127ba0 con 0x7fe2dc0052e0 2024-09-11T01:52:17.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.867+0000 7fe2e77fe640 1 -- 172.21.15.12:0/1729981725 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fe2f0127ba0 con 0x7fe2dc0052e0 2024-09-11T01:52:17.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.867+0000 7fe2e77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:52:17.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.867+0000 7fe2e77fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:17.871 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":216,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:52:17.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.870+0000 7fe2f4802640 1 -- 172.21.15.12:0/1729981725 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe2d003e9f0 msgr2=0x7fe2d0040eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:17.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.870+0000 7fe2f4802640 1 --2- 172.21.15.12:0/1729981725 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe2d003e9f0 0x7fe2d0040eb0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc0057e0 tx=0x7fe2e802d000 comp rx=0 tx=0).stop 2024-09-11T01:52:17.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.870+0000 7fe2f4802640 10 monclient: shutdown 2024-09-11T01:52:17.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.870+0000 7fe2f4802640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:17.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.870+0000 7fe2f4802640 1 -- 172.21.15.12:0/1729981725 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2dc0052e0 msgr2=0x7fe2f01244b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:17.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.870+0000 7fe2f4802640 1 --2- 172.21.15.12:0/1729981725 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2dc0052e0 0x7fe2f01244b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0006180 tx=0x7fe2e000c730 comp rx=0 tx=0).stop 2024-09-11T01:52:17.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.870+0000 7fe2f4802640 1 -- 172.21.15.12:0/1729981725 shutdown_connections 2024-09-11T01:52:17.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.870+0000 7fe2f4802640 1 --2- 172.21.15.12:0/1729981725 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2dc0052e0 0x7fe2f01244b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:17.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.870+0000 7fe2f4802640 1 --2- 172.21.15.12:0/1729981725 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2dc004800 0x7fe2f01cd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:17.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.870+0000 7fe2f4802640 1 --2- 172.21.15.12:0/1729981725 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe2d003e9f0 0x7fe2d0040eb0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:17.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.870+0000 7fe2f4802640 1 --2- 172.21.15.12:0/1729981725 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2dc0039b0 0x7fe2f01ccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:17.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.870+0000 7fe2f4802640 1 -- 172.21.15.12:0/1729981725 >> 172.21.15.12:0/1729981725 conn(0x7fe2f0068950 msgr2=0x7fe2f00a28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:17.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.870+0000 7fe2f4802640 1 -- 172.21.15.12:0/1729981725 shutdown_connections 2024-09-11T01:52:17.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.870+0000 7fe2f4802640 1 -- 172.21.15.12:0/1729981725 wait complete. 2024-09-11T01:52:17.877 INFO:tasks.rados.rados.0.smithi071.stdout:3419: finishing write tid 4 to smithi07134777-46 2024-09-11T01:52:17.877 INFO:tasks.rados.rados.0.smithi071.stdout:3419: oid 46 updating version 0 to 1791 2024-09-11T01:52:17.877 INFO:tasks.rados.rados.0.smithi071.stdout:3419: oid 46 updating version 1791 to 1792 2024-09-11T01:52:17.877 INFO:tasks.rados.rados.0.smithi071.stdout:3419: oid 46 updating version 1792 to 1793 2024-09-11T01:52:17.877 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 1793 (ObjNum 1193 snap 381 seq_num 1193) dirty exists 2024-09-11T01:52:17.877 INFO:tasks.rados.rados.0.smithi071.stdout:3419: left oid 46 (ObjNum 1193 snap 381 seq_num 1193) 2024-09-11T01:52:17.878 INFO:tasks.rados.rados.0.smithi071.stdout:3416: finishing copy_from racing read to smithi07134777-8 2024-09-11T01:52:17.878 INFO:tasks.rados.rados.0.smithi071.stdout:3416: finishing copy_from to smithi07134777-8 2024-09-11T01:52:17.878 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 2093 (ObjNum 1187 snap 378 seq_num 1187) dirty exists 2024-09-11T01:52:17.878 INFO:tasks.rados.rados.0.smithi071.stdout:3414: done (7 left) 2024-09-11T01:52:17.878 INFO:tasks.rados.rados.0.smithi071.stdout:3415: done (6 left) 2024-09-11T01:52:17.878 INFO:tasks.rados.rados.0.smithi071.stdout:3416: done (5 left) 2024-09-11T01:52:17.878 INFO:tasks.rados.rados.0.smithi071.stdout:3417: done (4 left) 2024-09-11T01:52:17.878 INFO:tasks.rados.rados.0.smithi071.stdout:3418: done (3 left) 2024-09-11T01:52:17.878 INFO:tasks.rados.rados.0.smithi071.stdout:3419: done (2 left) 2024-09-11T01:52:17.878 INFO:tasks.rados.rados.0.smithi071.stdout:3420: done (1 left) 2024-09-11T01:52:17.878 INFO:tasks.rados.rados.0.smithi071.stdout:3421: done (0 left) 2024-09-11T01:52:17.878 INFO:tasks.rados.rados.0.smithi071.stdout:3422: rollback oid 2 current snap is 381 2024-09-11T01:52:17.878 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 2 to 366 2024-09-11T01:52:17.879 INFO:tasks.rados.rados.0.smithi071.stdout:3423: rollback oid 31 current snap is 381 2024-09-11T01:52:17.879 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 31 to 325 2024-09-11T01:52:17.879 INFO:tasks.rados.rados.0.smithi071.stdout:3424: snap_create 2024-09-11T01:52:17.879 INFO:tasks.rados.rados.0.smithi071.stdout:3422: finishing rollback tid 1 to smithi07134777-2 2024-09-11T01:52:17.879 INFO:tasks.rados.rados.0.smithi071.stdout:3422: finishing rollback tid 2 to smithi07134777-2 2024-09-11T01:52:17.880 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 1789 (ObjNum 1131 snap 357 seq_num 1131) dirty exists 2024-09-11T01:52:17.880 INFO:tasks.rados.rados.0.smithi071.stdout:3423: finishing rollback tid 1 to smithi07134777-31 2024-09-11T01:52:17.880 INFO:tasks.rados.rados.0.smithi071.stdout:3423: finishing rollback tid 2 to smithi07134777-31 2024-09-11T01:52:17.880 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 2546 (ObjNum 988 snap 317 seq_num 988) dirty exists 2024-09-11T01:52:17.884 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:52:17.884 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T01:52:17.884 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:52:17.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.983+0000 7f0c5e394640 10 monclient: get_monmap_and_config 2024-09-11T01:52:17.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.983+0000 7f0c5e394640 10 monclient: build_initial_monmap 2024-09-11T01:52:17.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.983+0000 7f0c5e394640 10 monclient: monmap: 2024-09-11T01:52:17.986 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:17.986 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:17.986 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:17.985064+0000 2024-09-11T01:52:17.986 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:17.985064+0000 2024-09-11T01:52:17.986 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:17.987 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:17.987 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:17.987 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:17.987 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:17.987 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:17.987 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:17.988 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:17.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c5e394640 1 Processor -- start 2024-09-11T01:52:17.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c5e394640 1 -- start start 2024-09-11T01:52:17.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c5e394640 10 monclient: init 2024-09-11T01:52:17.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c5e394640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:17.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c5e394640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-11T01:52:17.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c5e394640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0c58135fc0 0x7f0c581363a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:17.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c5e394640 10 monclient(hunting): picked mon.noname-a con 0x7f0c58135fc0 addr v2:172.21.15.12:3300/0 2024-09-11T01:52:17.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c5e394640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c58073fe0 0x7f0c580743f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:17.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c5e394640 10 monclient(hunting): picked mon.noname-b con 0x7f0c58073fe0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:17.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c5e394640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0c580a2db0 addr v1:172.21.15.71:6789/0 2024-09-11T01:52:17.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c5e394640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c58072fb0 con 0x7f0c580a2db0 2024-09-11T01:52:17.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c5e394640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:17.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c5e394640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f0c58073850 con 0x7f0c58135fc0 2024-09-11T01:52:17.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c5e394640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:17.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c5e394640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f0c58073290 con 0x7f0c58073fe0 2024-09-11T01:52:17.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c5e394640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:17.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c5e394640 10 monclient(hunting): authenticate will time out at 2144.427780s 2024-09-11T01:52:17.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c57fff640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7f0c58135fc0 msgr2=0x7f0c581363a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:52:17.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c57fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0c58135fc0 0x7f0c581363a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:17.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c577fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c58073fe0 0x7f0c580743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:17.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c577fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c58073fe0 0x7f0c580743f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:60708/0 (socket says 172.21.15.12:60708) 2024-09-11T01:52:17.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c577fe640 1 -- 172.21.15.12:0/4198351428 learned_addr learned my addr 172.21.15.12:0/4198351428 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:17.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c577fe640 10 monclient(hunting): get_auth_request con 0x7f0c58073fe0 auth_method 0 2024-09-11T01:52:17.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:17.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c577fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:17.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:17.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:17.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.984+0000 7f0c577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:17.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.985+0000 7f0c577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:17.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.985+0000 7f0c577fe640 10 monclient(hunting): handle_auth_done global_id 244280 payload 274 2024-09-11T01:52:17.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.985+0000 7f0c577fe640 1 -- 172.21.15.12:0/4198351428 >> v2:172.21.15.12:3300/0 conn(0x7f0c58135fc0 msgr2=0x7f0c581363a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:52:17.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.985+0000 7f0c577fe640 1 --2- 172.21.15.12:0/4198351428 >> v2:172.21.15.12:3300/0 conn(0x7f0c58135fc0 0x7f0c581363a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:17.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.985+0000 7f0c577fe640 1 -- 172.21.15.12:0/4198351428 >> v1:172.21.15.71:6789/0 conn(0x7f0c580a2db0 legacy=0x7f0c580a31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:17.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.985+0000 7f0c577fe640 10 monclient: _finish_hunting 0 2024-09-11T01:52:17.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.985+0000 7f0c577fe640 1 monclient: found mon.noname-b 2024-09-11T01:52:17.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.985+0000 7f0c577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:17.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.985+0000 7f0c577fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:52:17.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.985+0000 7f0c577fe640 1 -- 172.21.15.12:0/4198351428 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c58073570 con 0x7f0c58073fe0 2024-09-11T01:52:17.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c577fe640 1 -- 172.21.15.12:0/4198351428 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c58073fe0 msgr2=0x7f0c580743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:17.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c577fe640 1 --2- 172.21.15.12:0/4198351428 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c58073fe0 0x7f0c580743f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c009870 tx=0x7f0c4c02a0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e4a9adc4ed0f025 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:17.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c56ffd640 1 -- 172.21.15.12:0/4198351428 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c037070 con 0x7f0c58073fe0 2024-09-11T01:52:17.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:17.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c56ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:52:17.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c56ffd640 10 monclient: dump: 2024-09-11T01:52:17.995 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:17.995 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:17.995 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:17.995 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:17.996 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:17.996 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:17.996 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:17.996 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:17.996 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:17.996 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:17.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c56ffd640 10 monclient: _finish_auth 0 2024-09-11T01:52:17.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c56ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:52:17.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:17.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c56ffd640 1 -- 172.21.15.12:0/4198351428 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c4c032640 con 0x7f0c58073fe0 2024-09-11T01:52:17.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c5e394640 5 monclient: authenticate success, global_id 244280 2024-09-11T01:52:17.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c5e394640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:52:17.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c56ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:17.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c56ffd640 1 -- 172.21.15.12:0/4198351428 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c032a10 con 0x7f0c58073fe0 2024-09-11T01:52:17.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c5e394640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:17.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c5e394640 4 set_mon_vals no callback set 2024-09-11T01:52:17.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:17.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c56ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:17.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c56ffd640 10 monclient: dump: 2024-09-11T01:52:17.997 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:17.997 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:17.997 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:17.997 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:17.998 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:17.998 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:17.998 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:17.998 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:17.998 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:17.998 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:17.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c5e394640 10 monclient: shutdown 2024-09-11T01:52:17.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c5e394640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:17.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c5e394640 1 -- 172.21.15.12:0/4198351428 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c58073fe0 msgr2=0x7f0c580743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:17.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c5e394640 1 --2- 172.21.15.12:0/4198351428 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c58073fe0 0x7f0c580743f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c009870 tx=0x7f0c4c02a0d0 comp rx=0 tx=0).stop 2024-09-11T01:52:17.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c5e394640 1 -- 172.21.15.12:0/4198351428 shutdown_connections 2024-09-11T01:52:17.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c5e394640 1 --2- 172.21.15.12:0/4198351428 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c58073fe0 0x7f0c580743f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:17.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c5e394640 1 --2- 172.21.15.12:0/4198351428 >> v2:172.21.15.12:3300/0 conn(0x7f0c58135fc0 0x7f0c581363a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:17.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c5e394640 1 -- 172.21.15.12:0/4198351428 >> 172.21.15.12:0/4198351428 conn(0x7f0c580a83b0 msgr2=0x7f0c580a87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:17.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c5e394640 1 -- 172.21.15.12:0/4198351428 shutdown_connections 2024-09-11T01:52:17.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.988+0000 7f0c5e394640 1 -- 172.21.15.12:0/4198351428 wait complete. 2024-09-11T01:52:17.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.989+0000 7f0c5e394640 10 monclient: build_initial_monmap 2024-09-11T01:52:18.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.989+0000 7f0c5e394640 10 monclient: monmap: 2024-09-11T01:52:18.000 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:18.000 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:18.000 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:18.000 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:18.000 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:18.001 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:18.001 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:18.001 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:18.001 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:18.002 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:18.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.989+0000 7f0c5e394640 1 Processor -- start 2024-09-11T01:52:18.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c5e394640 1 -- start start 2024-09-11T01:52:18.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c5e394640 10 monclient: init 2024-09-11T01:52:18.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c5e394640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:18.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c5e394640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:52:18.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c5e394640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c58073fe0 0x7f0c581cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:18.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c5e394640 10 monclient(hunting): picked mon.noname-c con 0x7f0c58073fe0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:18.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c5e394640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c580a2db0 0x7f0c581cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:18.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c5e394640 10 monclient(hunting): picked mon.noname-a con 0x7f0c580a2db0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:18.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c5e394640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c58135fc0 0x7f0c58130e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:18.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c5e394640 10 monclient(hunting): picked mon.noname-b con 0x7f0c58135fc0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:18.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c5e394640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:18.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c5e394640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0c580a3a80 con 0x7f0c580a2db0 2024-09-11T01:52:18.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c5e394640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:18.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c5e394640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0c580a3900 con 0x7f0c58135fc0 2024-09-11T01:52:18.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c5e394640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:18.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c5e394640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f0c580a3c00 con 0x7f0c58073fe0 2024-09-11T01:52:18.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c5e394640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:18.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c5e394640 10 monclient(hunting): authenticate will time out at 2144.433503s 2024-09-11T01:52:18.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c577fe640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c580a2db0 msgr2=0x7f0c581cb5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:52:18.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c577fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c580a2db0 0x7f0c581cb5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:18.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c57fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c58073fe0 0x7f0c581cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:18.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c57fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c58073fe0 0x7f0c581cb0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:60722/0 (socket says 172.21.15.12:60722) 2024-09-11T01:52:18.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c57fff640 1 -- 172.21.15.12:0/3000196304 learned_addr learned my addr 172.21.15.12:0/3000196304 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:18.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c57fff640 10 monclient(hunting): get_auth_request con 0x7f0c58073fe0 auth_method 0 2024-09-11T01:52:18.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:18.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c57fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:18.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c57fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:18.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c5c931640 1 --2- 172.21.15.12:0/3000196304 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c58135fc0 0x7f0c58130e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:18.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c5c931640 10 monclient(hunting): get_auth_request con 0x7f0c58135fc0 auth_method 0 2024-09-11T01:52:18.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c5c931640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:18.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c5c931640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:18.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c5c931640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:18.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:18.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:18.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:18.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c5c931640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:18.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c5c931640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c5c931640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c57fff640 10 monclient(hunting): handle_auth_done global_id 244283 payload 824 2024-09-11T01:52:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c57fff640 1 -- 172.21.15.12:0/3000196304 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c58135fc0 msgr2=0x7f0c58130e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c57fff640 1 --2- 172.21.15.12:0/3000196304 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c58135fc0 0x7f0c58130e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c57fff640 1 -- 172.21.15.12:0/3000196304 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c580a2db0 msgr2=0x7f0c581cb5f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:52:18.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c57fff640 1 --2- 172.21.15.12:0/3000196304 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0c580a2db0 0x7f0c581cb5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c57fff640 10 monclient: _finish_hunting 0 2024-09-11T01:52:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c57fff640 1 monclient: found mon.noname-c 2024-09-11T01:52:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c57fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:52:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.990+0000 7f0c57fff640 1 -- 172.21.15.12:0/3000196304 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c580a7c40 con 0x7f0c58073fe0 2024-09-11T01:52:18.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.993+0000 7f0c57fff640 1 -- 172.21.15.12:0/3000196304 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c58073fe0 msgr2=0x7f0c581cb0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:18.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.993+0000 7f0c57fff640 1 --2- 172.21.15.12:0/3000196304 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c58073fe0 0x7f0c581cb0b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0c4800b580 tx=0x7f0c4800ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:18.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.993+0000 7f0c54ff9640 1 -- 172.21.15.12:0/3000196304 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c48003dd0 con 0x7f0c58073fe0 2024-09-11T01:52:18.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.993+0000 7f0c54ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:18.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.993+0000 7f0c54ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:18.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.993+0000 7f0c54ff9640 10 monclient: dump: 2024-09-11T01:52:18.018 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:18.018 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:18.018 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:18.018 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:18.018 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:18.018 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:18.019 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:18.019 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:18.019 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:18.019 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:18.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.993+0000 7f0c54ff9640 10 monclient: _finish_auth 0 2024-09-11T01:52:18.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.993+0000 7f0c54ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:52:18.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.993+0000 7f0c54ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:18.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.993+0000 7f0c54ff9640 1 -- 172.21.15.12:0/3000196304 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c48015070 con 0x7f0c58073fe0 2024-09-11T01:52:18.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.993+0000 7f0c54ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:18.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.993+0000 7f0c54ff9640 1 -- 172.21.15.12:0/3000196304 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c48005330 con 0x7f0c58073fe0 2024-09-11T01:52:18.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.993+0000 7f0c54ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:18.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.993+0000 7f0c54ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:18.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.993+0000 7f0c54ff9640 10 monclient: dump: 2024-09-11T01:52:18.021 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:18.021 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:18.021 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:18.021 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:18.022 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:18.022 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:18.022 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:18.022 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:18.022 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:18.022 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:18.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.993+0000 7f0c5e394640 5 monclient: authenticate success, global_id 244283 2024-09-11T01:52:18.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.993+0000 7f0c5e394640 10 monclient: _renew_subs 2024-09-11T01:52:18.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.993+0000 7f0c5e394640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:18.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.993+0000 7f0c5e394640 1 -- 172.21.15.12:0/3000196304 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c5809f5a0 con 0x7f0c58073fe0 2024-09-11T01:52:18.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.993+0000 7f0c567fc640 4 set_mon_vals no callback set 2024-09-11T01:52:18.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.993+0000 7f0c5e394640 10 monclient: _renew_subs 2024-09-11T01:52:18.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.993+0000 7f0c5e394640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:18.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.993+0000 7f0c5e394640 1 -- 172.21.15.12:0/3000196304 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c58138a30 con 0x7f0c58073fe0 2024-09-11T01:52:18.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.994+0000 7f0c5e394640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:18.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.994+0000 7f0c5e394640 1 --2- 172.21.15.12:0/3000196304 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c581cbb30 0x7f0c581313a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:18.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.994+0000 7f0c5e394640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.994+0000 7f0c5e394640 1 -- 172.21.15.12:0/3000196304 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f0c5806a060 con 0x7f0c581cbb30 2024-09-11T01:52:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.994+0000 7f0c577fe640 1 --2- 172.21.15.12:0/3000196304 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c581cbb30 0x7f0c581313a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.994+0000 7f0c577fe640 10 monclient: get_auth_request con 0x7f0c581cbb30 auth_method 0 2024-09-11T01:52:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.994+0000 7f0c577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.994+0000 7f0c577fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:18.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.994+0000 7f0c577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:18.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.994+0000 7f0c54ff9640 1 -- 172.21.15.12:0/3000196304 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f0c48005510 con 0x7f0c58073fe0 2024-09-11T01:52:18.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.994+0000 7f0c577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:18.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.994+0000 7f0c577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:18.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.994+0000 7f0c577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:18.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.994+0000 7f0c577fe640 10 monclient(hunting): handle_auth_done global_id 244286 payload 824 2024-09-11T01:52:18.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.994+0000 7f0c577fe640 1 -- 172.21.15.12:0/3000196304 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0c58138320 con 0x7f0c581cbb30 2024-09-11T01:52:18.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.994+0000 7f0c54ff9640 1 --2- 172.21.15.12:0/3000196304 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0c4403e1c0 0x7f0c44040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:18.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.994+0000 7f0c54ff9640 1 -- 172.21.15.12:0/3000196304 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(767..767 src has 676..767) ==== 4803+0+0 (secure 0 0 0) 0x7f0c48052820 con 0x7f0c58073fe0 2024-09-11T01:52:18.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.994+0000 7f0c5c931640 1 --2- 172.21.15.12:0/3000196304 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0c4403e1c0 0x7f0c44040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:18.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.994+0000 7f0c5c931640 10 monclient: get_auth_request con 0x7f0c4403e1c0 auth_method 0 2024-09-11T01:52:18.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.996+0000 7f0c577fe640 1 -- 172.21.15.12:0/3000196304 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c581cbb30 msgr2=0x7f0c581313a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:18.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.996+0000 7f0c577fe640 1 --2- 172.21.15.12:0/3000196304 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c581cbb30 0x7f0c581313a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c034000 tx=0x7f0c4c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:18.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.997+0000 7f0c54ff9640 1 -- 172.21.15.12:0/3000196304 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c037040 con 0x7f0c581cbb30 2024-09-11T01:52:18.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.997+0000 7f0c54ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:18.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.997+0000 7f0c54ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:18.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.997+0000 7f0c54ff9640 10 monclient: dump: 2024-09-11T01:52:18.028 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:18.028 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:18.029 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:18.029 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:18.029 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:18.029 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:18.029 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:18.029 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:18.029 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:18.030 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:18.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.997+0000 7f0c54ff9640 1 -- 172.21.15.12:0/3000196304 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0c4c009490 con 0x7f0c581cbb30 2024-09-11T01:52:18.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.997+0000 7f0c54ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:18.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.997+0000 7f0c54ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:18.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.997+0000 7f0c54ff9640 1 -- 172.21.15.12:0/3000196304 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c581cbb30 msgr2=0x7f0c581313a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:18.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.997+0000 7f0c54ff9640 1 --2- 172.21.15.12:0/3000196304 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c581cbb30 0x7f0c581313a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c034000 tx=0x7f0c4c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:52:18.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.997+0000 7f0c54ff9640 1 -- 172.21.15.12:0/3000196304 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c581cbb30 msgr2=0x7f0c581313a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:18.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.997+0000 7f0c54ff9640 1 --2- 172.21.15.12:0/3000196304 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c581cbb30 0x7f0c581313a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:18.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:17.997+0000 7f0c5c931640 1 --2- 172.21.15.12:0/3000196304 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0c4403e1c0 0x7f0c44040680 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0c500055e0 tx=0x7f0c50003d10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:18.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.003+0000 7f0c5e394640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:52:18.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.003+0000 7f0c5e394640 1 --2- 172.21.15.12:0/3000196304 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c5813d970 0x7f0c581318e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:18.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.003+0000 7f0c5e394640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:18.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.003+0000 7f0c5e394640 1 -- 172.21.15.12:0/3000196304 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f0c580743f0 con 0x7f0c5813d970 2024-09-11T01:52:18.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.003+0000 7f0c577fe640 1 --2- 172.21.15.12:0/3000196304 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c5813d970 0x7f0c581318e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:18.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.003+0000 7f0c577fe640 10 monclient: get_auth_request con 0x7f0c5813d970 auth_method 0 2024-09-11T01:52:18.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.003+0000 7f0c577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:18.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.003+0000 7f0c577fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:18.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.003+0000 7f0c577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:18.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.003+0000 7f0c577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:18.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.003+0000 7f0c577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:18.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.003+0000 7f0c577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:18.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.003+0000 7f0c577fe640 10 monclient(hunting): handle_auth_done global_id 244289 payload 824 2024-09-11T01:52:18.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.003+0000 7f0c577fe640 1 -- 172.21.15.12:0/3000196304 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0c58137230 con 0x7f0c5813d970 2024-09-11T01:52:18.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.006+0000 7f0c577fe640 1 -- 172.21.15.12:0/3000196304 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c5813d970 msgr2=0x7f0c581318e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:18.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.006+0000 7f0c577fe640 1 --2- 172.21.15.12:0/3000196304 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c5813d970 0x7f0c581318e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c009870 tx=0x7f0c4c009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:18.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.006+0000 7f0c54ff9640 1 -- 172.21.15.12:0/3000196304 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c037040 con 0x7f0c5813d970 2024-09-11T01:52:18.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.006+0000 7f0c54ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:18.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.006+0000 7f0c54ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:18.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.006+0000 7f0c54ff9640 10 monclient: dump: 2024-09-11T01:52:18.034 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:18.034 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:18.035 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:18.035 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:18.035 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:18.035 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:18.035 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:18.035 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:18.035 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:18.035 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:18.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.006+0000 7f0c54ff9640 1 -- 172.21.15.12:0/3000196304 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4864 (secure 0 0 0) 0x7f0c4c04c070 con 0x7f0c5813d970 2024-09-11T01:52:18.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.006+0000 7f0c54ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:52:18.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.006+0000 7f0c54ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:18.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.006+0000 7f0c54ff9640 1 -- 172.21.15.12:0/3000196304 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c5813d970 msgr2=0x7f0c581318e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:18.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.006+0000 7f0c54ff9640 1 --2- 172.21.15.12:0/3000196304 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c5813d970 0x7f0c581318e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c009870 tx=0x7f0c4c009240 comp rx=0 tx=0).stop 2024-09-11T01:52:18.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.006+0000 7f0c54ff9640 1 -- 172.21.15.12:0/3000196304 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c5813d970 msgr2=0x7f0c581318e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:18.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.006+0000 7f0c54ff9640 1 --2- 172.21.15.12:0/3000196304 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c5813d970 0x7f0c581318e0 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c009870 tx=0x7f0c4c009240 comp rx=0 tx=0).stop 2024-09-11T01:52:18.036 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:52:18.036 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:52:18.036 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:52:18.036 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:52:18.036 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 216, 2024-09-11T01:52:18.036 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 91455, 2024-09-11T01:52:18.037 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:52:18.037 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:52:18.037 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:52:18.037 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:18.037 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:52:18.037 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:52:18.037 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:52:18.037 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:52:18.037 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:18.037 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:18.037 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:18.037 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:18.037 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:18.037 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:18.038 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:18.038 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:18.038 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:18.038 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:18.038 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:18.038 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:18.038 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:52:18.038 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:52:18.038 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:18.038 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:18.038 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:18.039 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:18.039 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:18.039 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:18.039 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:18.039 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:18.039 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:18.039 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:18.039 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:18.039 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:18.039 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:18.039 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:52:18.039 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:52:18.039 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:52:18.040 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:52:18.040 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:52:18.040 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:52:18.040 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:52:18.040 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:52:18.040 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:52:18.040 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:52:18.040 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:52:18.040 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:52:18.040 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:52:18.040 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:52:18.040 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:52:18.040 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:52:18.040 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:52:18.041 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:18.041 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:18.041 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:18.041 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:18.041 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:18.041 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:18.041 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:18.041 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:18.041 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:18.041 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:18.041 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:18.041 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:18.041 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:52:18.042 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:18.042 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:52:18.042 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:18.042 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:52:18.042 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:52:18.042 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:18.042 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:18.042 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:18.042 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:18.042 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:52:18.042 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:18.042 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:18.042 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:18.042 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:18.043 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:52:18.043 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:18.043 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:18.043 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:18.043 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:18.043 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:52:18.043 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:52:18.043 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:18.043 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:18.043 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:18.043 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:18.043 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:18.043 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:52:18.043 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:52:18.044 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:18.044 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:18.044 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:18.044 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:18.044 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:52:18.044 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:18.044 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:18.044 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:18.044 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:18.044 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:52:18.044 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:18.044 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:18.044 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:18.045 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:18.045 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:52:18.045 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:52:18.045 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:18.045 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:18.045 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:18.045 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:18.045 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:18.045 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:52:18.045 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:52:18.045 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:18.045 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:18.045 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:18.046 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:18.046 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:52:18.046 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:18.046 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:18.046 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:18.046 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:18.046 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:52:18.046 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:18.046 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:18.046 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:18.046 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:18.046 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:52:18.046 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:52:18.046 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:18.047 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:18.047 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:18.047 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:18.047 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:18.047 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:18.047 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:52:18.047 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:52:18.047 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:18.047 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:18.047 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:18.047 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:52:18.047 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:18.047 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:18.047 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:52:18.048 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:18.048 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:18.048 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:18.048 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:52:18.048 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:18.048 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:18.048 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:52:18.048 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:18.048 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:18.048 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:18.048 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:52:18.048 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:18.048 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:18.048 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:52:18.049 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:18.049 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:18.049 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:18.049 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:52:18.049 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:18.049 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:18.049 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:18.049 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:52:18.049 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:52:18.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.006+0000 7f0c5e394640 1 -- 172.21.15.12:0/3000196304 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0c4403e1c0 msgr2=0x7f0c44040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:18.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.006+0000 7f0c5e394640 1 --2- 172.21.15.12:0/3000196304 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0c4403e1c0 0x7f0c44040680 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0c500055e0 tx=0x7f0c50003d10 comp rx=0 tx=0).stop 2024-09-11T01:52:18.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.006+0000 7f0c5e394640 10 monclient: shutdown 2024-09-11T01:52:18.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.006+0000 7f0c5e394640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:18.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.006+0000 7f0c5e394640 1 -- 172.21.15.12:0/3000196304 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c58073fe0 msgr2=0x7f0c581cb0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:18.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.006+0000 7f0c5e394640 1 --2- 172.21.15.12:0/3000196304 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0c58073fe0 0x7f0c581cb0b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0c4800b580 tx=0x7f0c4800ba40 comp rx=0 tx=0).stop 2024-09-11T01:52:18.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.006+0000 7f0c5c931640 1 -- 172.21.15.12:0/3000196304 reap_dead start 2024-09-11T01:52:18.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.006+0000 7f0c5c931640 1 -- 172.21.15.12:0/3000196304 reap_dead start 2024-09-11T01:52:18.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.006+0000 7f0c5e394640 1 -- 172.21.15.12:0/3000196304 shutdown_connections 2024-09-11T01:52:18.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.006+0000 7f0c5e394640 1 -- 172.21.15.12:0/3000196304 >> 172.21.15.12:0/3000196304 conn(0x7f0c580a83b0 msgr2=0x7f0c580a6ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:18.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.006+0000 7f0c5e394640 1 -- 172.21.15.12:0/3000196304 shutdown_connections 2024-09-11T01:52:18.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.006+0000 7f0c5e394640 1 -- 172.21.15.12:0/3000196304 wait complete. 2024-09-11T01:52:18.052 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:52:18.052 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T01:52:18.052 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:52:18.139 INFO:tasks.rados.rados.0.smithi071.stdout:3422: done (2 left) 2024-09-11T01:52:18.139 INFO:tasks.rados.rados.0.smithi071.stdout:3423: done (1 left) 2024-09-11T01:52:18.139 INFO:tasks.rados.rados.0.smithi071.stdout:3424: done (0 left) 2024-09-11T01:52:18.139 INFO:tasks.rados.rados.0.smithi071.stdout:3425: snap_remove snap 323 2024-09-11T01:52:18.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.152+0000 7f231390e640 10 monclient: get_monmap_and_config 2024-09-11T01:52:18.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.152+0000 7f231390e640 10 monclient: build_initial_monmap 2024-09-11T01:52:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.152+0000 7f231390e640 10 monclient: monmap: 2024-09-11T01:52:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:18.153863+0000 2024-09-11T01:52:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:18.153863+0000 2024-09-11T01:52:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:18.155 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:18.155 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.152+0000 7f231390e640 1 Processor -- start 2024-09-11T01:52:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.152+0000 7f231390e640 1 -- start start 2024-09-11T01:52:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.152+0000 7f231390e640 10 monclient: init 2024-09-11T01:52:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.153+0000 7f231390e640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.153+0000 7f231390e640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-11T01:52:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.153+0000 7f231390e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f230c137d50 addr v1:172.21.15.12:6789/0 2024-09-11T01:52:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.153+0000 7f231390e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f230c138130 0x7f230c138580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.153+0000 7f231390e640 10 monclient(hunting): picked mon.noname-c con 0x7f230c138130 addr v2:172.21.15.71:3300/0 2024-09-11T01:52:18.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.153+0000 7f231390e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f230c13d280 addr v1:172.21.15.71:6789/0 2024-09-11T01:52:18.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.153+0000 7f231390e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f230c09d820 con 0x7f230c137d50 2024-09-11T01:52:18.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.153+0000 7f231390e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f230c09ba20 con 0x7f230c13d280 2024-09-11T01:52:18.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.153+0000 7f23116aa640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7f230c137d50 legacy=0x7f230c09d110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:52:18.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.153+0000 7f231390e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:18.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.153+0000 7f231390e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f230c09c650 con 0x7f230c138130 2024-09-11T01:52:18.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.153+0000 7f231390e640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:18.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.153+0000 7f231390e640 10 monclient(hunting): authenticate will time out at 2144.596574s 2024-09-11T01:52:18.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.153+0000 7f2310ea9640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f230c138130 0x7f230c138580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:18.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.153+0000 7f2311eab640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f230c13d280 0x7f230c13f670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.12:52318/0 (socket says 172.21.15.12:52318) 2024-09-11T01:52:18.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.153+0000 7f2310ea9640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f230c138130 0x7f230c138580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:41980/0 (socket says 172.21.15.12:41980) 2024-09-11T01:52:18.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.153+0000 7f2310ea9640 1 -- 172.21.15.12:0/4028432352 learned_addr learned my addr 172.21.15.12:0/4028432352 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:18.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.153+0000 7f2310ea9640 10 monclient(hunting): get_auth_request con 0x7f230c138130 auth_method 0 2024-09-11T01:52:18.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.153+0000 7f2310ea9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:18.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.153+0000 7f2310ea9640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:18.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.153+0000 7f2310ea9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:18.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.153+0000 7f2310ea9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:18.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.153+0000 7f2310ea9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:18.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.153+0000 7f2310ea9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:18.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.154+0000 7f2310ea9640 10 monclient(hunting): handle_auth_done global_id 234298 payload 274 2024-09-11T01:52:18.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.154+0000 7f2310ea9640 1 -- 172.21.15.12:0/4028432352 >> v1:172.21.15.71:6789/0 conn(0x7f230c13d280 legacy=0x7f230c13f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:18.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.154+0000 7f2310ea9640 1 -- 172.21.15.12:0/4028432352 >> v1:172.21.15.12:6789/0 conn(0x7f230c137d50 legacy=0x7f230c09d110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:52:18.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.154+0000 7f2310ea9640 10 monclient: _finish_hunting 0 2024-09-11T01:52:18.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.154+0000 7f2310ea9640 1 monclient: found mon.noname-c 2024-09-11T01:52:18.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.154+0000 7f2310ea9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:18.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.154+0000 7f2310ea9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:52:18.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.154+0000 7f2310ea9640 1 -- 172.21.15.12:0/4028432352 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f230c09c370 con 0x7f230c138130 2024-09-11T01:52:18.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.156+0000 7f2310ea9640 1 -- 172.21.15.12:0/4028432352 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c138130 msgr2=0x7f230c138580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:18.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.156+0000 7f2310ea9640 1 --2- 172.21.15.12:0/4028432352 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c138130 0x7f230c138580 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f22fc009870 tx=0x7f22fc02a130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=471db18b4070c2b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:18.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.157+0000 7f2303fff640 1 -- 172.21.15.12:0/4028432352 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc037070 con 0x7f230c138130 2024-09-11T01:52:18.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.157+0000 7f2303fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:18.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.157+0000 7f2303fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:18.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.157+0000 7f2303fff640 10 monclient: dump: 2024-09-11T01:52:18.166 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:18.166 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:18.166 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:18.166 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:18.166 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:18.166 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:18.166 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:18.166 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:18.166 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:18.166 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:18.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.157+0000 7f2303fff640 10 monclient: _finish_auth 0 2024-09-11T01:52:18.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.157+0000 7f2303fff640 10 monclient: _check_auth_tickets 2024-09-11T01:52:18.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.157+0000 7f2303fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:18.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.157+0000 7f231390e640 5 monclient: authenticate success, global_id 234298 2024-09-11T01:52:18.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.157+0000 7f231390e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:52:18.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.157+0000 7f2303fff640 1 -- 172.21.15.12:0/4028432352 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22fc032640 con 0x7f230c138130 2024-09-11T01:52:18.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.157+0000 7f2303fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:18.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.157+0000 7f231390e640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:18.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.157+0000 7f231390e640 4 set_mon_vals no callback set 2024-09-11T01:52:18.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.157+0000 7f2303fff640 1 -- 172.21.15.12:0/4028432352 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc032a10 con 0x7f230c138130 2024-09-11T01:52:18.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.157+0000 7f2303fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:18.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.157+0000 7f2303fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:52:18.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.157+0000 7f2303fff640 10 monclient: dump: 2024-09-11T01:52:18.167 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:18.167 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:18.167 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:18.167 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:18.168 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.157+0000 7f231390e640 10 monclient: shutdown 2024-09-11T01:52:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.157+0000 7f231390e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.157+0000 7f231390e640 1 -- 172.21.15.12:0/4028432352 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c138130 msgr2=0x7f230c138580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.157+0000 7f231390e640 1 --2- 172.21.15.12:0/4028432352 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c138130 0x7f230c138580 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f22fc009870 tx=0x7f22fc02a130 comp rx=0 tx=0).stop 2024-09-11T01:52:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.157+0000 7f231390e640 1 -- 172.21.15.12:0/4028432352 shutdown_connections 2024-09-11T01:52:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.157+0000 7f231390e640 1 --2- 172.21.15.12:0/4028432352 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c138130 0x7f230c138580 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:18.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.157+0000 7f231390e640 1 -- 172.21.15.12:0/4028432352 >> 172.21.15.12:0/4028432352 conn(0x7f230c0a7020 msgr2=0x7f230c0a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.157+0000 7f231390e640 1 -- 172.21.15.12:0/4028432352 shutdown_connections 2024-09-11T01:52:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.157+0000 7f231390e640 1 -- 172.21.15.12:0/4028432352 wait complete. 2024-09-11T01:52:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.158+0000 7f231390e640 10 monclient: build_initial_monmap 2024-09-11T01:52:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.158+0000 7f231390e640 10 monclient: monmap: 2024-09-11T01:52:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:18.169 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:18.170 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.158+0000 7f231390e640 1 Processor -- start 2024-09-11T01:52:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.158+0000 7f231390e640 1 -- start start 2024-09-11T01:52:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.158+0000 7f231390e640 10 monclient: init 2024-09-11T01:52:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f231390e640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f231390e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:52:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f231390e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c137d50 0x7f230c13b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f231390e640 10 monclient(hunting): picked mon.noname-b con 0x7f230c137d50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f231390e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f230c138130 0x7f230c136df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:18.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f231390e640 10 monclient(hunting): picked mon.noname-a con 0x7f230c138130 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f231390e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f230c13d280 0x7f230c13d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f231390e640 10 monclient(hunting): picked mon.noname-c con 0x7f230c13d280 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f231390e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f231390e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f230c06f850 con 0x7f230c138130 2024-09-11T01:52:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f231390e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f231390e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f230c06f6d0 con 0x7f230c137d50 2024-09-11T01:52:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f231390e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f231390e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f230c140d90 con 0x7f230c13d280 2024-09-11T01:52:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f231390e640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f231390e640 10 monclient(hunting): authenticate will time out at 2144.602332s 2024-09-11T01:52:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f2310ea9640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f230c138130 msgr2=0x7f230c136df0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:52:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f2310ea9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f230c138130 0x7f230c136df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f2311eab640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f230c13d280 0x7f230c13d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f23116aa640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c137d50 0x7f230c13b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f23116aa640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c137d50 0x7f230c13b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:41994/0 (socket says 172.21.15.12:41994) 2024-09-11T01:52:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f2311eab640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f230c13d280 0x7f230c13d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:60770/0 (socket says 172.21.15.12:60770) 2024-09-11T01:52:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f23116aa640 1 -- 172.21.15.12:0/15117306 learned_addr learned my addr 172.21.15.12:0/15117306 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f23116aa640 10 monclient(hunting): get_auth_request con 0x7f230c137d50 auth_method 0 2024-09-11T01:52:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f23116aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f23116aa640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f23116aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f2311eab640 10 monclient(hunting): get_auth_request con 0x7f230c13d280 auth_method 0 2024-09-11T01:52:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f2311eab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f2311eab640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f2311eab640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f23116aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f23116aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f23116aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f2311eab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f2311eab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f2311eab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f23116aa640 10 monclient(hunting): handle_auth_done global_id 234301 payload 824 2024-09-11T01:52:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f23116aa640 1 -- 172.21.15.12:0/15117306 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f230c13d280 msgr2=0x7f230c13d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f23116aa640 1 --2- 172.21.15.12:0/15117306 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f230c13d280 0x7f230c13d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f23116aa640 1 -- 172.21.15.12:0/15117306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f230c138130 msgr2=0x7f230c136df0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:52:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f23116aa640 1 --2- 172.21.15.12:0/15117306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f230c138130 0x7f230c136df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f23116aa640 10 monclient: _finish_hunting 0 2024-09-11T01:52:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f23116aa640 1 monclient: found mon.noname-b 2024-09-11T01:52:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f23116aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f23116aa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-09-11T01:52:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f23116aa640 1 -- 172.21.15.12:0/15117306 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f230c13f050 con 0x7f230c137d50 2024-09-11T01:52:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.159+0000 7f2311eab640 1 --2- 172.21.15.12:0/15117306 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f230c13d280 0x7f230c13d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:52:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.162+0000 7f23116aa640 1 -- 172.21.15.12:0/15117306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c137d50 msgr2=0x7f230c13b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.162+0000 7f23116aa640 1 --2- 172.21.15.12:0/15117306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c137d50 0x7f230c13b210 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f230800b550 tx=0x7f230800ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.162+0000 7f2301ffb640 1 -- 172.21.15.12:0/15117306 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2308005070 con 0x7f230c137d50 2024-09-11T01:52:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.162+0000 7f2301ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.162+0000 7f2301ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:52:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.162+0000 7f2301ffb640 10 monclient: dump: 2024-09-11T01:52:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:18.175 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.162+0000 7f2301ffb640 10 monclient: _finish_auth 0 2024-09-11T01:52:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.162+0000 7f2301ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:52:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.162+0000 7f2301ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.162+0000 7f2301ffb640 1 -- 172.21.15.12:0/15117306 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2308015070 con 0x7f230c137d50 2024-09-11T01:52:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.162+0000 7f2301ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.162+0000 7f2301ffb640 1 -- 172.21.15.12:0/15117306 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2308005450 con 0x7f230c137d50 2024-09-11T01:52:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.162+0000 7f231390e640 5 monclient: authenticate success, global_id 234301 2024-09-11T01:52:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.162+0000 7f231390e640 10 monclient: _renew_subs 2024-09-11T01:52:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.162+0000 7f231390e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:52:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.162+0000 7f231390e640 1 -- 172.21.15.12:0/15117306 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f230c0a1710 con 0x7f230c137d50 2024-09-11T01:52:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.162+0000 7f23037fe640 4 set_mon_vals no callback set 2024-09-11T01:52:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.162+0000 7f2301ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.162+0000 7f2301ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:52:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.162+0000 7f2301ffb640 10 monclient: dump: 2024-09-11T01:52:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:18.177 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:18.177 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:18.177 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:18.177 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:18.177 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:18.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.162+0000 7f231390e640 10 monclient: _renew_subs 2024-09-11T01:52:18.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.162+0000 7f231390e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:52:18.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.162+0000 7f231390e640 1 -- 172.21.15.12:0/15117306 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f230c140410 con 0x7f230c137d50 2024-09-11T01:52:18.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.163+0000 7f2301ffb640 1 -- 172.21.15.12:0/15117306 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f23080055f0 con 0x7f230c137d50 2024-09-11T01:52:18.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.163+0000 7f231390e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:18.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.163+0000 7f231390e640 1 --2- 172.21.15.12:0/15117306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c137330 0x7f230c13dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:18.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.163+0000 7f231390e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:18.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.163+0000 7f231390e640 1 -- 172.21.15.12:0/15117306 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f230c138600 con 0x7f230c137330 2024-09-11T01:52:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.163+0000 7f2301ffb640 1 --2- 172.21.15.12:0/15117306 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f22f003e210 0x7f22f00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.163+0000 7f2301ffb640 1 -- 172.21.15.12:0/15117306 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(768..768 src has 676..768) ==== 4803+0+0 (secure 0 0 0) 0x7f2308052aa0 con 0x7f230c137d50 2024-09-11T01:52:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.163+0000 7f2310ea9640 1 --2- 172.21.15.12:0/15117306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c137330 0x7f230c13dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.163+0000 7f2310ea9640 10 monclient: get_auth_request con 0x7f230c137330 auth_method 0 2024-09-11T01:52:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.163+0000 7f2310ea9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.163+0000 7f2310ea9640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.163+0000 7f2310ea9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.163+0000 7f2311eab640 1 --2- 172.21.15.12:0/15117306 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f22f003e210 0x7f22f00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.163+0000 7f2311eab640 10 monclient: get_auth_request con 0x7f22f003e210 auth_method 0 2024-09-11T01:52:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.163+0000 7f2310ea9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.163+0000 7f2310ea9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:18.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.163+0000 7f2310ea9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:18.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.164+0000 7f2310ea9640 10 monclient(hunting): handle_auth_done global_id 234304 payload 824 2024-09-11T01:52:18.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.164+0000 7f2310ea9640 1 -- 172.21.15.12:0/15117306 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f230c13bcc0 con 0x7f230c137330 2024-09-11T01:52:18.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.166+0000 7f2311eab640 1 --2- 172.21.15.12:0/15117306 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f22f003e210 0x7f22f00406d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f230c1ce0d0 tx=0x7f2304004ee0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:18.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.166+0000 7f2310ea9640 1 -- 172.21.15.12:0/15117306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c137330 msgr2=0x7f230c13dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:18.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.166+0000 7f2310ea9640 1 --2- 172.21.15.12:0/15117306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c137330 0x7f230c13dba0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f22fc034000 tx=0x7f22fc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:18.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.166+0000 7f2301ffb640 1 -- 172.21.15.12:0/15117306 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc042070 con 0x7f230c137330 2024-09-11T01:52:18.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.166+0000 7f2301ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:18.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.166+0000 7f2301ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:52:18.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.166+0000 7f2301ffb640 10 monclient: dump: 2024-09-11T01:52:18.183 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:18.183 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:18.183 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:18.183 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:18.183 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:18.184 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:18.184 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:18.184 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:18.184 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:18.184 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:18.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.166+0000 7f2301ffb640 1 -- 172.21.15.12:0/15117306 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f22fc032a00 con 0x7f230c137330 2024-09-11T01:52:18.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.166+0000 7f2301ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:18.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.166+0000 7f2301ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:18.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.166+0000 7f2301ffb640 1 -- 172.21.15.12:0/15117306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c137330 msgr2=0x7f230c13dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:18.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.166+0000 7f2301ffb640 1 --2- 172.21.15.12:0/15117306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c137330 0x7f230c13dba0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f22fc034000 tx=0x7f22fc0046c0 comp rx=0 tx=0).stop 2024-09-11T01:52:18.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.166+0000 7f2301ffb640 1 -- 172.21.15.12:0/15117306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c137330 msgr2=0x7f230c13dba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:18.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.166+0000 7f2301ffb640 1 --2- 172.21.15.12:0/15117306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c137330 0x7f230c13dba0 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f22fc034000 tx=0x7f22fc0046c0 comp rx=0 tx=0).stop 2024-09-11T01:52:18.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.172+0000 7f231390e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:52:18.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.172+0000 7f231390e640 1 --2- 172.21.15.12:0/15117306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c13e0e0 0x7f230c0a2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:18.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.172+0000 7f231390e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:18.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.172+0000 7f231390e640 1 -- 172.21.15.12:0/15117306 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f230c13e4c0 con 0x7f230c13e0e0 2024-09-11T01:52:18.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.173+0000 7f2310ea9640 1 --2- 172.21.15.12:0/15117306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c13e0e0 0x7f230c0a2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:18.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.173+0000 7f2310ea9640 10 monclient: get_auth_request con 0x7f230c13e0e0 auth_method 0 2024-09-11T01:52:18.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.173+0000 7f2310ea9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:18.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.173+0000 7f2310ea9640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:18.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.173+0000 7f2310ea9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:18.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.173+0000 7f2310ea9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:18.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.173+0000 7f2310ea9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:18.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.173+0000 7f2310ea9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:18.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.173+0000 7f2310ea9640 10 monclient(hunting): handle_auth_done global_id 234307 payload 824 2024-09-11T01:52:18.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.173+0000 7f2310ea9640 1 -- 172.21.15.12:0/15117306 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f230c074a40 con 0x7f230c13e0e0 2024-09-11T01:52:18.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.175+0000 7f2310ea9640 1 -- 172.21.15.12:0/15117306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c13e0e0 msgr2=0x7f230c0a2d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:18.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.175+0000 7f2310ea9640 1 --2- 172.21.15.12:0/15117306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c13e0e0 0x7f230c0a2d20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f22fc009870 tx=0x7f22fc009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:18.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.176+0000 7f2301ffb640 1 -- 172.21.15.12:0/15117306 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc037040 con 0x7f230c13e0e0 2024-09-11T01:52:18.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.176+0000 7f2301ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:18.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.176+0000 7f2301ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:52:18.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.176+0000 7f2301ffb640 10 monclient: dump: 2024-09-11T01:52:18.186 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:18.186 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:18.186 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:18.186 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:18.186 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:18.186 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:18.186 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:18.187 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:18.187 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:18.187 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:18.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.176+0000 7f2301ffb640 1 -- 172.21.15.12:0/15117306 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4700 (secure 0 0 0) 0x7f22fc03ee40 con 0x7f230c13e0e0 2024-09-11T01:52:18.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.176+0000 7f2301ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:52:18.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.176+0000 7f2301ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:18.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.176+0000 7f2301ffb640 1 -- 172.21.15.12:0/15117306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c13e0e0 msgr2=0x7f230c0a2d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:18.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.176+0000 7f2301ffb640 1 --2- 172.21.15.12:0/15117306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c13e0e0 0x7f230c0a2d20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f22fc009870 tx=0x7f22fc009240 comp rx=0 tx=0).stop 2024-09-11T01:52:18.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.176+0000 7f2301ffb640 1 -- 172.21.15.12:0/15117306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c13e0e0 msgr2=0x7f230c0a2d20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:18.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.176+0000 7f2301ffb640 1 --2- 172.21.15.12:0/15117306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c13e0e0 0x7f230c0a2d20 secure :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f22fc009870 tx=0x7f22fc009240 comp rx=0 tx=0).stop 2024-09-11T01:52:18.187 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:52:18.188 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:52:18.188 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:52:18.188 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:52:18.188 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 216, 2024-09-11T01:52:18.188 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 140070, 2024-09-11T01:52:18.188 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:52:18.188 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:52:18.188 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:52:18.188 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:18.188 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:52:18.188 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:52:18.188 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:52:18.189 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:52:18.189 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:18.189 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:18.189 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:18.189 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:18.189 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:18.189 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:18.189 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:18.189 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:18.189 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:18.189 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:18.189 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:18.189 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:18.189 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:52:18.190 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:52:18.190 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:18.190 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:18.190 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:18.190 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:18.190 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:18.190 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:18.190 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:18.190 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:18.190 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:18.190 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:18.190 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:18.190 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:18.190 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:18.191 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:52:18.191 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:52:18.191 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:52:18.191 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:52:18.191 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:52:18.191 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:52:18.191 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:52:18.191 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:52:18.191 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:52:18.191 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:52:18.191 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:52:18.191 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:52:18.191 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:52:18.191 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:52:18.192 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:52:18.192 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:52:18.192 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:52:18.192 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:18.192 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:18.192 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:18.192 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:18.192 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:18.192 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:18.192 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:18.192 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:18.192 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:18.192 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:18.193 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:18.193 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:18.193 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:52:18.193 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:18.193 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:52:18.193 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:18.193 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:52:18.193 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:52:18.193 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:18.193 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:18.193 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:18.193 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:18.193 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:52:18.193 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:18.194 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:18.194 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:18.194 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:18.194 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:52:18.194 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:18.194 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:18.194 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:18.194 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:18.194 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:52:18.194 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:52:18.194 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:18.194 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:18.194 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:18.194 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:18.195 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:18.195 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:52:18.195 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:52:18.195 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:18.195 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:18.195 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:18.195 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:18.195 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:52:18.195 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:18.195 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:18.195 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:18.195 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:18.195 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:52:18.195 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:18.196 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:18.196 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:18.196 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:18.196 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:52:18.196 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:52:18.196 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:18.196 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:18.196 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:18.196 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:18.196 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:18.196 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:52:18.196 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:52:18.196 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:18.197 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:18.197 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:18.197 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:18.197 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:52:18.197 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:18.197 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:18.197 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:18.197 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:18.197 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:52:18.197 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:18.197 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:18.197 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:18.197 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:18.197 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:52:18.198 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:52:18.198 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:18.198 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:18.198 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:18.198 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:18.198 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:18.198 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:18.198 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:52:18.198 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:52:18.198 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:18.198 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:18.198 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:18.198 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:52:18.198 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:18.199 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:18.199 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:52:18.199 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:18.199 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:18.199 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:18.199 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:52:18.199 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:18.199 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:18.199 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:52:18.199 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:18.199 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:18.199 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:18.200 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 7 2024-09-11T01:52:18.200 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:18.200 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:18.200 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:18.200 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:52:18.200 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:52:18.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.177+0000 7f231390e640 1 -- 172.21.15.12:0/15117306 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f22f003e210 msgr2=0x7f22f00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:18.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.177+0000 7f231390e640 1 --2- 172.21.15.12:0/15117306 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f22f003e210 0x7f22f00406d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f230c1ce0d0 tx=0x7f2304004ee0 comp rx=0 tx=0).stop 2024-09-11T01:52:18.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.177+0000 7f231390e640 10 monclient: shutdown 2024-09-11T01:52:18.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.177+0000 7f231390e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:18.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.177+0000 7f231390e640 1 -- 172.21.15.12:0/15117306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c137d50 msgr2=0x7f230c13b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:18.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.177+0000 7f231390e640 1 --2- 172.21.15.12:0/15117306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f230c137d50 0x7f230c13b210 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f230800b550 tx=0x7f230800ba10 comp rx=0 tx=0).stop 2024-09-11T01:52:18.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.177+0000 7f2311eab640 1 -- 172.21.15.12:0/15117306 reap_dead start 2024-09-11T01:52:18.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.177+0000 7f2311eab640 1 -- 172.21.15.12:0/15117306 reap_dead start 2024-09-11T01:52:18.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.177+0000 7f231390e640 1 -- 172.21.15.12:0/15117306 shutdown_connections 2024-09-11T01:52:18.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.177+0000 7f231390e640 1 -- 172.21.15.12:0/15117306 >> 172.21.15.12:0/15117306 conn(0x7f230c0a7020 msgr2=0x7f230c0a4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:18.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.177+0000 7f231390e640 1 -- 172.21.15.12:0/15117306 shutdown_connections 2024-09-11T01:52:18.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:18.177+0000 7f231390e640 1 -- 172.21.15.12:0/15117306 wait complete. 2024-09-11T01:52:18.203 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:52:18.203 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:52:19.138 INFO:tasks.rados.rados.0.smithi071.stdout:3425: done (0 left) 2024-09-11T01:52:19.138 INFO:tasks.rados.rados.0.smithi071.stdout:3426: read oid 19 snap 375 2024-09-11T01:52:19.138 INFO:tasks.rados.rados.0.smithi071.stdout:3426: expect (ObjNum 1136 snap 360 seq_num 1136) 2024-09-11T01:52:19.138 INFO:tasks.rados.rados.0.smithi071.stdout:3427: write (excl) oid 20 current snap is 382 2024-09-11T01:52:19.139 INFO:tasks.rados.rados.0.smithi071.stdout:3427: seq_num 1195 ranges {484365=589976,1661982=422383} 2024-09-11T01:52:19.144 INFO:tasks.rados.rados.0.smithi071.stdout:3427: writing smithi07134777-20 from 484365 to 1074341 tid 1 2024-09-11T01:52:19.151 INFO:tasks.rados.rados.0.smithi071.stdout:3427: writing smithi07134777-20 from 1661982 to 2084365 tid 2 2024-09-11T01:52:19.152 INFO:tasks.rados.rados.0.smithi071.stdout:3428: copy_from oid 36 from oid 42 current snap is 382 2024-09-11T01:52:19.152 INFO:tasks.rados.rados.0.smithi071.stdout:3426: expect (ObjNum 1136 snap 360 seq_num 1136) 2024-09-11T01:52:19.152 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:52:19.195 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1650275 2024-09-11T01:52:19.253 INFO:tasks.rados.rados.0.smithi071.stdout:3427: finishing write tid 1 to smithi07134777-20 2024-09-11T01:52:19.253 INFO:tasks.rados.rados.0.smithi071.stdout:3427: finishing write tid 2 to smithi07134777-20 2024-09-11T01:52:19.253 INFO:tasks.rados.rados.0.smithi071.stdout:3427: finishing write tid 3 to smithi07134777-20 2024-09-11T01:52:19.253 INFO:tasks.rados.rados.0.smithi071.stdout:3427: finishing write tid 4 to smithi07134777-20 2024-09-11T01:52:19.253 INFO:tasks.rados.rados.0.smithi071.stdout:3427: oid 20 updating version 0 to 1162 2024-09-11T01:52:19.254 INFO:tasks.rados.rados.0.smithi071.stdout:3427: oid 20 updating version 1162 to 1163 2024-09-11T01:52:19.254 INFO:tasks.rados.rados.0.smithi071.stdout:3427: oid 20 version 1163 is already newer than 1157 2024-09-11T01:52:19.254 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 1163 (ObjNum 1195 snap 382 seq_num 1195) dirty exists 2024-09-11T01:52:19.254 INFO:tasks.rados.rados.0.smithi071.stdout:3427: left oid 20 (ObjNum 1195 snap 382 seq_num 1195) 2024-09-11T01:52:19.254 INFO:tasks.rados.rados.0.smithi071.stdout:3428: finishing copy_from racing read to smithi07134777-36 2024-09-11T01:52:19.254 INFO:tasks.rados.rados.0.smithi071.stdout:3428: finishing copy_from to smithi07134777-36 2024-09-11T01:52:19.254 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 1785 (ObjNum 1143 snap 364 seq_num 1143) dirty exists 2024-09-11T01:52:19.254 INFO:tasks.rados.rados.0.smithi071.stdout:3426: done (2 left) 2024-09-11T01:52:19.254 INFO:tasks.rados.rados.0.smithi071.stdout:3427: done (1 left) 2024-09-11T01:52:19.254 INFO:tasks.rados.rados.0.smithi071.stdout:3428: done (0 left) 2024-09-11T01:52:19.254 INFO:tasks.rados.rados.0.smithi071.stdout:3429: snap_create 2024-09-11T01:52:20.142 INFO:tasks.rados.rados.0.smithi071.stdout:3429: done (0 left) 2024-09-11T01:52:20.142 INFO:tasks.rados.rados.0.smithi071.stdout:3430: snap_create 2024-09-11T01:52:21.144 INFO:tasks.rados.rados.0.smithi071.stdout:3430: done (0 left) 2024-09-11T01:52:21.144 INFO:tasks.rados.rados.0.smithi071.stdout:3431: write (excl) oid 16 current snap is 384 2024-09-11T01:52:21.144 INFO:tasks.rados.rados.0.smithi071.stdout:3431: seq_num 1196 ranges {501172=665225,1720464=380708} 2024-09-11T01:52:21.150 INFO:tasks.rados.rados.0.smithi071.stdout:3431: writing smithi07134777-16 from 501172 to 1166397 tid 1 2024-09-11T01:52:21.155 INFO:tasks.rados.rados.0.smithi071.stdout:3431: writing smithi07134777-16 from 1720464 to 2101172 tid 2 2024-09-11T01:52:21.156 INFO:tasks.rados.rados.0.smithi071.stdout:3432: delete oid 7 current snap is 384 2024-09-11T01:52:21.163 INFO:tasks.rados.rados.0.smithi071.stdout:3431: finishing write tid 1 to smithi07134777-16 2024-09-11T01:52:21.165 INFO:tasks.rados.rados.0.smithi071.stdout:3432: done (1 left) 2024-09-11T01:52:21.165 INFO:tasks.rados.rados.0.smithi071.stdout:3433: write (excl) oid 39 current snap is 384 2024-09-11T01:52:21.165 INFO:tasks.rados.rados.0.smithi071.stdout:3433: seq_num 1198 ranges {534786=415723,1437428=697358} 2024-09-11T01:52:21.170 INFO:tasks.rados.rados.0.smithi071.stdout:3433: writing smithi07134777-39 from 534786 to 950509 tid 1 2024-09-11T01:52:21.177 INFO:tasks.rados.rados.0.smithi071.stdout:3433: writing smithi07134777-39 from 1437428 to 2134786 tid 2 2024-09-11T01:52:21.177 INFO:tasks.rados.rados.0.smithi071.stdout:3434: snap_create 2024-09-11T01:52:21.177 INFO:tasks.rados.rados.0.smithi071.stdout:3431: finishing write tid 2 to smithi07134777-16 2024-09-11T01:52:21.177 INFO:tasks.rados.rados.0.smithi071.stdout:3431: finishing write tid 3 to smithi07134777-16 2024-09-11T01:52:21.177 INFO:tasks.rados.rados.0.smithi071.stdout:3431: finishing write tid 4 to smithi07134777-16 2024-09-11T01:52:21.177 INFO:tasks.rados.rados.0.smithi071.stdout:3431: oid 16 updating version 0 to 2097 2024-09-11T01:52:21.177 INFO:tasks.rados.rados.0.smithi071.stdout:3431: oid 16 updating version 2097 to 2098 2024-09-11T01:52:21.178 INFO:tasks.rados.rados.0.smithi071.stdout:3431: oid 16 updating version 2098 to 2099 2024-09-11T01:52:21.178 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 2099 (ObjNum 1196 snap 384 seq_num 1196) dirty exists 2024-09-11T01:52:21.178 INFO:tasks.rados.rados.0.smithi071.stdout:3431: left oid 16 (ObjNum 1196 snap 384 seq_num 1196) 2024-09-11T01:52:21.178 INFO:tasks.rados.rados.0.smithi071.stdout:3433: finishing write tid 1 to smithi07134777-39 2024-09-11T01:52:21.181 INFO:tasks.rados.rados.0.smithi071.stdout:3433: finishing write tid 2 to smithi07134777-39 2024-09-11T01:52:21.181 INFO:tasks.rados.rados.0.smithi071.stdout:3433: finishing write tid 3 to smithi07134777-39 2024-09-11T01:52:21.181 INFO:tasks.rados.rados.0.smithi071.stdout:3433: finishing write tid 4 to smithi07134777-39 2024-09-11T01:52:21.181 INFO:tasks.rados.rados.0.smithi071.stdout:3433: oid 39 updating version 0 to 1984 2024-09-11T01:52:21.182 INFO:tasks.rados.rados.0.smithi071.stdout:3433: oid 39 version 1984 is already newer than 1983 2024-09-11T01:52:21.182 INFO:tasks.rados.rados.0.smithi071.stdout:3433: oid 39 version 1984 is already newer than 1982 2024-09-11T01:52:21.182 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 1984 (ObjNum 1198 snap 384 seq_num 1198) dirty exists 2024-09-11T01:52:21.182 INFO:tasks.rados.rados.0.smithi071.stdout:3433: left oid 39 (ObjNum 1198 snap 384 seq_num 1198) 2024-09-11T01:52:22.153 INFO:tasks.rados.rados.0.smithi071.stdout:3431: done (2 left) 2024-09-11T01:52:22.153 INFO:tasks.rados.rados.0.smithi071.stdout:3433: done (1 left) 2024-09-11T01:52:22.153 INFO:tasks.rados.rados.0.smithi071.stdout:3434: done (0 left) 2024-09-11T01:52:22.153 INFO:tasks.rados.rados.0.smithi071.stdout:3435: snap_remove snap 374 2024-09-11T01:52:23.159 INFO:tasks.rados.rados.0.smithi071.stdout:3435: done (0 left) 2024-09-11T01:52:23.159 INFO:tasks.rados.rados.0.smithi071.stdout:3436: snap_remove snap 371 2024-09-11T01:52:24.155 INFO:tasks.rados.rados.0.smithi071.stdout:3436: done (0 left) 2024-09-11T01:52:24.156 INFO:tasks.rados.rados.0.smithi071.stdout:3437: snap_create 2024-09-11T01:52:25.158 INFO:tasks.rados.rados.0.smithi071.stdout:3437: done (0 left) 2024-09-11T01:52:25.158 INFO:tasks.rados.rados.0.smithi071.stdout:3438: write oid 1 current snap is 386 2024-09-11T01:52:25.158 INFO:tasks.rados.rados.0.smithi071.stdout:3438: seq_num 1199 ranges {551593=490972,1779557=372036} 2024-09-11T01:52:25.163 INFO:tasks.rados.rados.0.smithi071.stdout:3438: writing smithi07134777-1 from 551593 to 1042565 tid 1 2024-09-11T01:52:25.169 INFO:tasks.rados.rados.0.smithi071.stdout:3438: writing smithi07134777-1 from 1779557 to 2151593 tid 2 2024-09-11T01:52:25.169 INFO:tasks.rados.rados.0.smithi071.stdout:3439: snap_create 2024-09-11T01:52:25.175 INFO:tasks.rados.rados.0.smithi071.stdout:3438: finishing write tid 1 to smithi07134777-1 2024-09-11T01:52:25.175 INFO:tasks.rados.rados.0.smithi071.stdout:3438: finishing write tid 2 to smithi07134777-1 2024-09-11T01:52:25.175 INFO:tasks.rados.rados.0.smithi071.stdout:3438: finishing write tid 3 to smithi07134777-1 2024-09-11T01:52:25.175 INFO:tasks.rados.rados.0.smithi071.stdout:3438: finishing write tid 4 to smithi07134777-1 2024-09-11T01:52:25.175 INFO:tasks.rados.rados.0.smithi071.stdout:3438: oid 1 updating version 0 to 2549 2024-09-11T01:52:25.175 INFO:tasks.rados.rados.0.smithi071.stdout:3438: oid 1 version 2549 is already newer than 2548 2024-09-11T01:52:25.175 INFO:tasks.rados.rados.0.smithi071.stdout:3438: oid 1 version 2549 is already newer than 2547 2024-09-11T01:52:25.175 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 2549 (ObjNum 1199 snap 386 seq_num 1199) dirty exists 2024-09-11T01:52:25.175 INFO:tasks.rados.rados.0.smithi071.stdout:3438: left oid 1 (ObjNum 1199 snap 386 seq_num 1199) 2024-09-11T01:52:26.171 INFO:tasks.rados.rados.0.smithi071.stdout:3438: done (1 left) 2024-09-11T01:52:26.171 INFO:tasks.rados.rados.0.smithi071.stdout:3439: done (0 left) 2024-09-11T01:52:26.172 INFO:tasks.rados.rados.0.smithi071.stdout:3440: snap_create 2024-09-11T01:52:27.177 INFO:tasks.rados.rados.0.smithi071.stdout:3440: done (0 left) 2024-09-11T01:52:27.177 INFO:tasks.rados.rados.0.smithi071.stdout:3441: snap_remove snap 367 2024-09-11T01:52:28.141 INFO:tasks.rados.rados.0.smithi071.stdout:3441: done (0 left) 2024-09-11T01:52:28.142 INFO:tasks.rados.rados.0.smithi071.stdout:3442: snap_create 2024-09-11T01:52:29.140 INFO:tasks.rados.rados.0.smithi071.stdout:3442: done (0 left) 2024-09-11T01:52:29.141 INFO:tasks.rados.rados.0.smithi071.stdout:3443: snap_remove snap 317 2024-09-11T01:52:30.145 INFO:tasks.rados.rados.0.smithi071.stdout:3443: done (0 left) 2024-09-11T01:52:30.145 INFO:tasks.rados.rados.0.smithi071.stdout:3444: read oid 18 snap -1 2024-09-11T01:52:30.146 INFO:tasks.rados.rados.0.smithi071.stdout:3444: expect (ObjNum 1164 snap 373 seq_num 1164) 2024-09-11T01:52:30.146 INFO:tasks.rados.rados.0.smithi071.stdout:3445: write (excl) oid 21 current snap is 389 2024-09-11T01:52:30.146 INFO:tasks.rados.rados.0.smithi071.stdout:3445: seq_num 1200 ranges {568400=566221,1838039=330361} 2024-09-11T01:52:30.152 INFO:tasks.rados.rados.0.smithi071.stdout:3445: writing smithi07134777-21 from 568400 to 1134621 tid 1 2024-09-11T01:52:30.159 INFO:tasks.rados.rados.0.smithi071.stdout:3445: writing smithi07134777-21 from 1838039 to 2168400 tid 2 2024-09-11T01:52:30.159 INFO:tasks.rados.rados.0.smithi071.stdout:3446: read oid 7 snap 325 2024-09-11T01:52:30.159 INFO:tasks.rados.rados.0.smithi071.stdout:3446: expect (ObjNum 1014 snap 322 seq_num 1014) 2024-09-11T01:52:30.159 INFO:tasks.rados.rados.0.smithi071.stdout:3447: snap_remove snap 383 2024-09-11T01:52:31.147 INFO:tasks.rados.rados.0.smithi071.stdout:3447: done (3 left) 2024-09-11T01:52:31.147 INFO:tasks.rados.rados.0.smithi071.stdout:3448: rollback oid 11 current snap is 389 2024-09-11T01:52:31.147 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 11 to 338 2024-09-11T01:52:31.147 INFO:tasks.rados.rados.0.smithi071.stdout:3444: expect (ObjNum 1164 snap 373 seq_num 1164) 2024-09-11T01:52:31.147 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:52:31.204 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3216047 2024-09-11T01:52:31.217 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1963006 2024-09-11T01:52:31.233 INFO:tasks.rados.rados.0.smithi071.stdout:3445: finishing write tid 1 to smithi07134777-21 2024-09-11T01:52:31.233 INFO:tasks.rados.rados.0.smithi071.stdout:3445: finishing write tid 2 to smithi07134777-21 2024-09-11T01:52:31.233 INFO:tasks.rados.rados.0.smithi071.stdout:3445: finishing write tid 3 to smithi07134777-21 2024-09-11T01:52:31.234 INFO:tasks.rados.rados.0.smithi071.stdout:3445: finishing write tid 4 to smithi07134777-21 2024-09-11T01:52:31.234 INFO:tasks.rados.rados.0.smithi071.stdout:3445: oid 21 updating version 0 to 2113 2024-09-11T01:52:31.234 INFO:tasks.rados.rados.0.smithi071.stdout:3445: oid 21 updating version 2113 to 2115 2024-09-11T01:52:31.234 INFO:tasks.rados.rados.0.smithi071.stdout:3445: oid 21 version 2115 is already newer than 2114 2024-09-11T01:52:31.234 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 2115 (ObjNum 1200 snap 389 seq_num 1200) dirty exists 2024-09-11T01:52:31.234 INFO:tasks.rados.rados.0.smithi071.stdout:3445: left oid 21 (ObjNum 1200 snap 389 seq_num 1200) 2024-09-11T01:52:31.234 INFO:tasks.rados.rados.0.smithi071.stdout:3444: done (3 left) 2024-09-11T01:52:31.234 INFO:tasks.rados.rados.0.smithi071.stdout:3445: done (2 left) 2024-09-11T01:52:31.234 INFO:tasks.rados.rados.0.smithi071.stdout:3449: rollback oid 25 current snap is 389 2024-09-11T01:52:31.234 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 25 to 375 2024-09-11T01:52:31.235 INFO:tasks.rados.rados.0.smithi071.stdout:3446: expect (ObjNum 1014 snap 322 seq_num 1014) 2024-09-11T01:52:31.235 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:52:31.304 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1735485 2024-09-11T01:52:31.377 INFO:tasks.rados.rados.0.smithi071.stdout:3448: finishing rollback tid 0 to smithi07134777-11 2024-09-11T01:52:31.378 INFO:tasks.rados.rados.0.smithi071.stdout:3448: finishing rollback tid 1 to smithi07134777-11 2024-09-11T01:52:31.378 INFO:tasks.rados.rados.0.smithi071.stdout:3448: finishing rollback tid 2 to smithi07134777-11 2024-09-11T01:52:31.378 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 2191 (ObjNum 987 snap 317 seq_num 987) dirty exists 2024-09-11T01:52:31.378 INFO:tasks.rados.rados.0.smithi071.stdout:3449: finishing rollback tid 0 to smithi07134777-25 2024-09-11T01:52:31.378 INFO:tasks.rados.rados.0.smithi071.stdout:3449: finishing rollback tid 1 to smithi07134777-25 2024-09-11T01:52:31.378 INFO:tasks.rados.rados.0.smithi071.stdout:3449: finishing rollback tid 2 to smithi07134777-25 2024-09-11T01:52:31.378 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 1055 (ObjNum 1023 snap 328 seq_num 1023) dirty exists 2024-09-11T01:52:31.378 INFO:tasks.rados.rados.0.smithi071.stdout:3446: done (2 left) 2024-09-11T01:52:31.378 INFO:tasks.rados.rados.0.smithi071.stdout:3448: done (1 left) 2024-09-11T01:52:31.378 INFO:tasks.rados.rados.0.smithi071.stdout:3449: done (0 left) 2024-09-11T01:52:31.378 INFO:tasks.rados.rados.0.smithi071.stdout:3450: read oid 12 snap -1 2024-09-11T01:52:31.379 INFO:tasks.rados.rados.0.smithi071.stdout:3450: expect (ObjNum 1182 snap 378 seq_num 1182) 2024-09-11T01:52:31.379 INFO:tasks.rados.rados.0.smithi071.stdout:3451: snap_create 2024-09-11T01:52:31.382 INFO:tasks.rados.rados.0.smithi071.stdout:3450: expect (ObjNum 1182 snap 378 seq_num 1182) 2024-09-11T01:52:31.408 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2974618 2024-09-11T01:52:31.451 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3084488 2024-09-11T01:52:31.493 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3054053 2024-09-11T01:52:32.155 INFO:tasks.rados.rados.0.smithi071.stdout:3450: done (1 left) 2024-09-11T01:52:32.155 INFO:tasks.rados.rados.0.smithi071.stdout:3451: done (0 left) 2024-09-11T01:52:32.155 INFO:tasks.rados.rados.0.smithi071.stdout:3452: snap_create 2024-09-11T01:52:33.140 INFO:tasks.rados.rados.0.smithi071.stdout:3452: done (0 left) 2024-09-11T01:52:33.140 INFO:tasks.rados.rados.0.smithi071.stdout:3453: delete oid 5 current snap is 391 2024-09-11T01:52:33.148 INFO:tasks.rados.rados.0.smithi071.stdout:3453: done (0 left) 2024-09-11T01:52:33.148 INFO:tasks.rados.rados.0.smithi071.stdout:3454: write oid 32 current snap is 391 2024-09-11T01:52:33.148 INFO:tasks.rados.rados.0.smithi071.stdout:3454: seq_num 1202 ranges {602014=433072,1671356=530658} 2024-09-11T01:52:33.153 INFO:tasks.rados.rados.0.smithi071.stdout:3454: writing smithi07134777-32 from 602014 to 1035086 tid 1 2024-09-11T01:52:33.161 INFO:tasks.rados.rados.0.smithi071.stdout:3454: writing smithi07134777-32 from 1671356 to 2202014 tid 2 2024-09-11T01:52:33.161 INFO:tasks.rados.rados.0.smithi071.stdout:3455: read oid 5 snap -1 2024-09-11T01:52:33.161 INFO:tasks.rados.rados.0.smithi071.stdout:3455: expect deleted 2024-09-11T01:52:33.161 INFO:tasks.rados.rados.0.smithi071.stdout:3456: read oid 48 snap -1 2024-09-11T01:52:33.161 INFO:tasks.rados.rados.0.smithi071.stdout:3456: expect (ObjNum 1178 snap 376 seq_num 1178) 2024-09-11T01:52:33.161 INFO:tasks.rados.rados.0.smithi071.stdout:3457: read oid 47 snap -1 2024-09-11T01:52:33.161 INFO:tasks.rados.rados.0.smithi071.stdout:3457: expect deleted 2024-09-11T01:52:33.161 INFO:tasks.rados.rados.0.smithi071.stdout:3458: snap_create 2024-09-11T01:52:33.161 INFO:tasks.rados.rados.0.smithi071.stdout:3454: finishing write tid 1 to smithi07134777-32 2024-09-11T01:52:33.166 INFO:tasks.rados.rados.0.smithi071.stdout:3456: expect (ObjNum 1178 snap 376 seq_num 1178) 2024-09-11T01:52:33.186 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2035931 2024-09-11T01:52:33.213 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:52:33.326 INFO:tasks.rados.rados.0.smithi071.stdout:3454: finishing write tid 2 to smithi07134777-32 2024-09-11T01:52:33.326 INFO:tasks.rados.rados.0.smithi071.stdout:3454: finishing write tid 3 to smithi07134777-32 2024-09-11T01:52:33.326 INFO:tasks.rados.rados.0.smithi071.stdout:3454: finishing write tid 4 to smithi07134777-32 2024-09-11T01:52:33.326 INFO:tasks.rados.rados.0.smithi071.stdout:3454: oid 32 updating version 0 to 1987 2024-09-11T01:52:33.326 INFO:tasks.rados.rados.0.smithi071.stdout:3454: oid 32 version 1987 is already newer than 1985 2024-09-11T01:52:33.327 INFO:tasks.rados.rados.0.smithi071.stdout:3454: oid 32 version 1987 is already newer than 1986 2024-09-11T01:52:33.327 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 1987 (ObjNum 1202 snap 391 seq_num 1202) dirty exists 2024-09-11T01:52:33.327 INFO:tasks.rados.rados.0.smithi071.stdout:3454: left oid 32 (ObjNum 1202 snap 391 seq_num 1202) 2024-09-11T01:52:34.162 INFO:tasks.rados.rados.0.smithi071.stdout:3454: done (4 left) 2024-09-11T01:52:34.162 INFO:tasks.rados.rados.0.smithi071.stdout:3455: done (3 left) 2024-09-11T01:52:34.162 INFO:tasks.rados.rados.0.smithi071.stdout:3456: done (2 left) 2024-09-11T01:52:34.162 INFO:tasks.rados.rados.0.smithi071.stdout:3457: done (1 left) 2024-09-11T01:52:34.162 INFO:tasks.rados.rados.0.smithi071.stdout:3458: done (0 left) 2024-09-11T01:52:34.162 INFO:tasks.rados.rados.0.smithi071.stdout:3459: write (excl) oid 45 current snap is 392 2024-09-11T01:52:34.162 INFO:tasks.rados.rados.0.smithi071.stdout:3459: seq_num 1203 ranges {618821=508321,1613485=531077} 2024-09-11T01:52:34.167 INFO:tasks.rados.rados.0.smithi071.stdout:3459: writing smithi07134777-45 from 618821 to 1127142 tid 1 2024-09-11T01:52:34.174 INFO:tasks.rados.rados.0.smithi071.stdout:3459: writing smithi07134777-45 from 1613485 to 2144562 tid 2 2024-09-11T01:52:34.174 INFO:tasks.rados.rados.0.smithi071.stdout:3460: copy_from oid 49 from oid 38 current snap is 392 2024-09-11T01:52:34.175 INFO:tasks.rados.rados.0.smithi071.stdout:3459: finishing write tid 1 to smithi07134777-45 2024-09-11T01:52:34.175 INFO:tasks.rados.rados.0.smithi071.stdout:3461: read oid 1 snap -1 2024-09-11T01:52:34.175 INFO:tasks.rados.rados.0.smithi071.stdout:3461: expect (ObjNum 1199 snap 386 seq_num 1199) 2024-09-11T01:52:34.175 INFO:tasks.rados.rados.0.smithi071.stdout:3462: write oid 32 current snap is 392 2024-09-11T01:52:34.175 INFO:tasks.rados.rados.0.smithi071.stdout:3462: seq_num 1204 ranges {635628=583570,1671967=563661} 2024-09-11T01:52:34.181 INFO:tasks.rados.rados.0.smithi071.stdout:3462: writing smithi07134777-32 from 635628 to 1219198 tid 1 2024-09-11T01:52:34.187 INFO:tasks.rados.rados.0.smithi071.stdout:3462: writing smithi07134777-32 from 1671967 to 2235628 tid 2 2024-09-11T01:52:34.187 INFO:tasks.rados.rados.0.smithi071.stdout:3463: write (excl) oid 46 current snap is 392 2024-09-11T01:52:34.187 INFO:tasks.rados.rados.0.smithi071.stdout:3463: seq_num 1205 ranges {652435=658819,1730449=521986} 2024-09-11T01:52:34.194 INFO:tasks.rados.rados.0.smithi071.stdout:3463: writing smithi07134777-46 from 652435 to 1311254 tid 1 2024-09-11T01:52:34.199 INFO:tasks.rados.rados.0.smithi071.stdout:3463: writing smithi07134777-46 from 1730449 to 2252435 tid 2 2024-09-11T01:52:34.199 INFO:tasks.rados.rados.0.smithi071.stdout:3464: read oid 47 snap -1 2024-09-11T01:52:34.199 INFO:tasks.rados.rados.0.smithi071.stdout:3464: expect deleted 2024-09-11T01:52:34.199 INFO:tasks.rados.rados.0.smithi071.stdout:3465: snap_remove snap 388 2024-09-11T01:52:35.166 INFO:tasks.rados.rados.0.smithi071.stdout:3465: done (6 left) 2024-09-11T01:52:35.167 INFO:tasks.rados.rados.0.smithi071.stdout:3466: read oid 44 snap -1 2024-09-11T01:52:35.167 INFO:tasks.rados.rados.0.smithi071.stdout:3466: expect deleted 2024-09-11T01:52:35.167 INFO:tasks.rados.rados.0.smithi071.stdout:3467: rollback oid 31 current snap is 392 2024-09-11T01:52:35.167 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 31 to 361 2024-09-11T01:52:35.167 INFO:tasks.rados.rados.0.smithi071.stdout:3461: expect (ObjNum 1199 snap 386 seq_num 1199) 2024-09-11T01:52:35.254 INFO:tasks.rados.rados.0.smithi071.stdout:3459: finishing write tid 2 to smithi07134777-45 2024-09-11T01:52:35.255 INFO:tasks.rados.rados.0.smithi071.stdout:3459: finishing write tid 3 to smithi07134777-45 2024-09-11T01:52:35.255 INFO:tasks.rados.rados.0.smithi071.stdout:3459: finishing write tid 4 to smithi07134777-45 2024-09-11T01:52:35.255 INFO:tasks.rados.rados.0.smithi071.stdout:3459: oid 45 updating version 0 to 2123 2024-09-11T01:52:35.255 INFO:tasks.rados.rados.0.smithi071.stdout:3459: oid 45 updating version 2123 to 2124 2024-09-11T01:52:35.255 INFO:tasks.rados.rados.0.smithi071.stdout:3459: oid 45 version 2124 is already newer than 2122 2024-09-11T01:52:35.255 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 2124 (ObjNum 1203 snap 392 seq_num 1203) dirty exists 2024-09-11T01:52:35.255 INFO:tasks.rados.rados.0.smithi071.stdout:3459: left oid 45 (ObjNum 1203 snap 392 seq_num 1203) 2024-09-11T01:52:35.255 INFO:tasks.rados.rados.0.smithi071.stdout:3460: finishing copy_from racing read to smithi07134777-49 2024-09-11T01:52:35.255 INFO:tasks.rados.rados.0.smithi071.stdout:3462: finishing write tid 1 to smithi07134777-32 2024-09-11T01:52:35.255 INFO:tasks.rados.rados.0.smithi071.stdout:3460: finishing copy_from to smithi07134777-49 2024-09-11T01:52:35.255 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 2452 (ObjNum 1162 snap 370 seq_num 1162) dirty exists 2024-09-11T01:52:35.255 INFO:tasks.rados.rados.0.smithi071.stdout:3462: finishing write tid 2 to smithi07134777-32 2024-09-11T01:52:35.255 INFO:tasks.rados.rados.0.smithi071.stdout:3459: done (7 left) 2024-09-11T01:52:35.256 INFO:tasks.rados.rados.0.smithi071.stdout:3460: done (6 left) 2024-09-11T01:52:35.256 INFO:tasks.rados.rados.0.smithi071.stdout:3461: done (5 left) 2024-09-11T01:52:35.256 INFO:tasks.rados.rados.0.smithi071.stdout:3468: snap_remove snap 338 2024-09-11T01:52:36.172 INFO:tasks.rados.rados.0.smithi071.stdout:3468: done (5 left) 2024-09-11T01:52:36.172 INFO:tasks.rados.rados.0.smithi071.stdout:3469: snap_create 2024-09-11T01:52:36.172 INFO:tasks.rados.rados.0.smithi071.stdout:3462: finishing write tid 3 to smithi07134777-32 2024-09-11T01:52:36.173 INFO:tasks.rados.rados.0.smithi071.stdout:3462: finishing write tid 4 to smithi07134777-32 2024-09-11T01:52:36.173 INFO:tasks.rados.rados.0.smithi071.stdout:3462: oid 32 updating version 0 to 1988 2024-09-11T01:52:36.173 INFO:tasks.rados.rados.0.smithi071.stdout:3462: oid 32 updating version 1988 to 1989 2024-09-11T01:52:36.173 INFO:tasks.rados.rados.0.smithi071.stdout:3462: oid 32 updating version 1989 to 1990 2024-09-11T01:52:36.173 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 1990 (ObjNum 1204 snap 392 seq_num 1204) dirty exists 2024-09-11T01:52:36.173 INFO:tasks.rados.rados.0.smithi071.stdout:3462: left oid 32 (ObjNum 1204 snap 392 seq_num 1204) 2024-09-11T01:52:36.173 INFO:tasks.rados.rados.0.smithi071.stdout:3463: finishing write tid 1 to smithi07134777-46 2024-09-11T01:52:36.173 INFO:tasks.rados.rados.0.smithi071.stdout:3463: finishing write tid 2 to smithi07134777-46 2024-09-11T01:52:36.173 INFO:tasks.rados.rados.0.smithi071.stdout:3463: finishing write tid 3 to smithi07134777-46 2024-09-11T01:52:36.173 INFO:tasks.rados.rados.0.smithi071.stdout:3463: finishing write tid 4 to smithi07134777-46 2024-09-11T01:52:36.173 INFO:tasks.rados.rados.0.smithi071.stdout:3463: oid 46 updating version 0 to 1794 2024-09-11T01:52:36.173 INFO:tasks.rados.rados.0.smithi071.stdout:3463: oid 46 updating version 1794 to 1796 2024-09-11T01:52:36.173 INFO:tasks.rados.rados.0.smithi071.stdout:3463: oid 46 version 1796 is already newer than 1795 2024-09-11T01:52:36.174 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 1796 (ObjNum 1205 snap 392 seq_num 1205) dirty exists 2024-09-11T01:52:36.174 INFO:tasks.rados.rados.0.smithi071.stdout:3463: left oid 46 (ObjNum 1205 snap 392 seq_num 1205) 2024-09-11T01:52:36.174 INFO:tasks.rados.rados.0.smithi071.stdout:3467: finishing rollback tid 0 to smithi07134777-31 2024-09-11T01:52:36.174 INFO:tasks.rados.rados.0.smithi071.stdout:3467: finishing rollback tid 1 to smithi07134777-31 2024-09-11T01:52:36.174 INFO:tasks.rados.rados.0.smithi071.stdout:3467: finishing rollback tid 2 to smithi07134777-31 2024-09-11T01:52:36.174 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 2552 (ObjNum 737 snap 226 seq_num 737) dirty exists 2024-09-11T01:52:37.179 INFO:tasks.rados.rados.0.smithi071.stdout:3462: done (5 left) 2024-09-11T01:52:37.179 INFO:tasks.rados.rados.0.smithi071.stdout:3463: done (4 left) 2024-09-11T01:52:37.179 INFO:tasks.rados.rados.0.smithi071.stdout:3464: done (3 left) 2024-09-11T01:52:37.179 INFO:tasks.rados.rados.0.smithi071.stdout:3466: done (2 left) 2024-09-11T01:52:37.179 INFO:tasks.rados.rados.0.smithi071.stdout:3467: done (1 left) 2024-09-11T01:52:37.179 INFO:tasks.rados.rados.0.smithi071.stdout:3469: done (0 left) 2024-09-11T01:52:37.180 INFO:tasks.rados.rados.0.smithi071.stdout:3470: read oid 4 snap 372 2024-09-11T01:52:37.180 INFO:tasks.rados.rados.0.smithi071.stdout:3470: expect deleted 2024-09-11T01:52:37.180 INFO:tasks.rados.rados.0.smithi071.stdout:3471: rollback oid 45 current snap is 393 2024-09-11T01:52:37.180 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 45 to 390 2024-09-11T01:52:37.180 INFO:tasks.rados.rados.0.smithi071.stdout:3472: write oid 22 current snap is 393 2024-09-11T01:52:37.180 INFO:tasks.rados.rados.0.smithi071.stdout:3472: seq_num 1206 ranges {669242=734068,2188931=80311} 2024-09-11T01:52:37.186 INFO:tasks.rados.rados.0.smithi071.stdout:3472: writing smithi07134777-22 from 669242 to 1403310 tid 1 2024-09-11T01:52:37.191 INFO:tasks.rados.rados.0.smithi071.stdout:3472: writing smithi07134777-22 from 2188931 to 2269242 tid 2 2024-09-11T01:52:37.192 INFO:tasks.rados.rados.0.smithi071.stdout:3473: snap_remove snap 387 2024-09-11T01:52:38.142 INFO:tasks.rados.rados.0.smithi071.stdout:3473: done (3 left) 2024-09-11T01:52:38.142 INFO:tasks.rados.rados.0.smithi071.stdout:3474: rollback oid 9 current snap is 393 2024-09-11T01:52:38.142 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 9 to 363 2024-09-11T01:52:38.142 INFO:tasks.rados.rados.0.smithi071.stdout:3471: finishing rollback tid 0 to smithi07134777-45 2024-09-11T01:52:38.142 INFO:tasks.rados.rados.0.smithi071.stdout:3471: finishing rollback tid 1 to smithi07134777-45 2024-09-11T01:52:38.142 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 2137 (ObjNum 129 snap 0 seq_num 2371068336) dirty dne 2024-09-11T01:52:38.143 INFO:tasks.rados.rados.0.smithi071.stdout:3472: finishing write tid 1 to smithi07134777-22 2024-09-11T01:52:38.143 INFO:tasks.rados.rados.0.smithi071.stdout:3472: finishing write tid 2 to smithi07134777-22 2024-09-11T01:52:38.143 INFO:tasks.rados.rados.0.smithi071.stdout:3472: finishing write tid 3 to smithi07134777-22 2024-09-11T01:52:38.143 INFO:tasks.rados.rados.0.smithi071.stdout:3472: finishing write tid 4 to smithi07134777-22 2024-09-11T01:52:38.143 INFO:tasks.rados.rados.0.smithi071.stdout:3472: oid 22 updating version 0 to 2423 2024-09-11T01:52:38.143 INFO:tasks.rados.rados.0.smithi071.stdout:3472: oid 22 updating version 2423 to 2425 2024-09-11T01:52:38.143 INFO:tasks.rados.rados.0.smithi071.stdout:3472: oid 22 version 2425 is already newer than 2424 2024-09-11T01:52:38.143 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 2425 (ObjNum 1206 snap 393 seq_num 1206) dirty exists 2024-09-11T01:52:38.143 INFO:tasks.rados.rados.0.smithi071.stdout:3472: left oid 22 (ObjNum 1206 snap 393 seq_num 1206) 2024-09-11T01:52:38.143 INFO:tasks.rados.rados.0.smithi071.stdout:3470: done (3 left) 2024-09-11T01:52:38.143 INFO:tasks.rados.rados.0.smithi071.stdout:3471: done (2 left) 2024-09-11T01:52:38.143 INFO:tasks.rados.rados.0.smithi071.stdout:3472: done (1 left) 2024-09-11T01:52:38.143 INFO:tasks.rados.rados.0.smithi071.stdout:3475: rollback oid 34 current snap is 393 2024-09-11T01:52:38.143 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 34 to 369 2024-09-11T01:52:38.144 INFO:tasks.rados.rados.0.smithi071.stdout:3476: rollback oid 33 current snap is 393 2024-09-11T01:52:38.144 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 33 to 384 2024-09-11T01:52:38.144 INFO:tasks.rados.rados.0.smithi071.stdout:3477: rollback oid 40 current snap is 393 2024-09-11T01:52:38.144 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 40 to 379 2024-09-11T01:52:38.144 INFO:tasks.rados.rados.0.smithi071.stdout:3478: write (excl) oid 20 current snap is 393 2024-09-11T01:52:38.144 INFO:tasks.rados.rados.0.smithi071.stdout:3478: seq_num 1207 ranges {686049=409317,1847413=438636} 2024-09-11T01:52:38.148 INFO:tasks.rados.rados.0.smithi071.stdout:3478: writing smithi07134777-20 from 686049 to 1095366 tid 1 2024-09-11T01:52:38.155 INFO:tasks.rados.rados.0.smithi071.stdout:3478: writing smithi07134777-20 from 1847413 to 2286049 tid 2 2024-09-11T01:52:38.155 INFO:tasks.rados.rados.0.smithi071.stdout:3479: write (excl) oid 12 current snap is 393 2024-09-11T01:52:38.155 INFO:tasks.rados.rados.0.smithi071.stdout:3479: seq_num 1208 ranges {702856=484566,1789542=513314} 2024-09-11T01:52:38.162 INFO:tasks.rados.rados.0.smithi071.stdout:3479: writing smithi07134777-12 from 702856 to 1187422 tid 1 2024-09-11T01:52:38.168 INFO:tasks.rados.rados.0.smithi071.stdout:3479: writing smithi07134777-12 from 1789542 to 2302856 tid 2 2024-09-11T01:52:38.168 INFO:tasks.rados.rados.0.smithi071.stdout:3480: write oid 1 current snap is 393 2024-09-11T01:52:38.168 INFO:tasks.rados.rados.0.smithi071.stdout:3474: finishing rollback tid 0 to smithi07134777-9 2024-09-11T01:52:38.168 INFO:tasks.rados.rados.0.smithi071.stdout:3480: seq_num 1209 ranges {719663=676168,1964377=355286} 2024-09-11T01:52:38.175 INFO:tasks.rados.rados.0.smithi071.stdout:3480: writing smithi07134777-1 from 719663 to 1395831 tid 1 2024-09-11T01:52:38.182 INFO:tasks.rados.rados.0.smithi071.stdout:3480: writing smithi07134777-1 from 1964377 to 2319663 tid 2 2024-09-11T01:52:38.182 INFO:tasks.rados.rados.0.smithi071.stdout:3481: read oid 2 snap -1 2024-09-11T01:52:38.182 INFO:tasks.rados.rados.0.smithi071.stdout:3481: expect (ObjNum 1131 snap 357 seq_num 1131) 2024-09-11T01:52:38.182 INFO:tasks.rados.rados.0.smithi071.stdout:3482: rollback oid 41 current snap is 393 2024-09-11T01:52:38.182 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 41 to 372 2024-09-11T01:52:38.182 INFO:tasks.rados.rados.0.smithi071.stdout:3477: finishing rollback tid 1 to smithi07134777-40 2024-09-11T01:52:38.182 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 1551 (ObjNum 129 snap 0 seq_num 2370786080) dirty dne 2024-09-11T01:52:38.182 INFO:tasks.rados.rados.0.smithi071.stdout:3474: finishing rollback tid 1 to smithi07134777-9 2024-09-11T01:52:38.182 INFO:tasks.rados.rados.0.smithi071.stdout:3474: finishing rollback tid 2 to smithi07134777-9 2024-09-11T01:52:38.182 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 1058 (ObjNum 1139 snap 360 seq_num 1139) dirty exists 2024-09-11T01:52:38.182 INFO:tasks.rados.rados.0.smithi071.stdout:3475: finishing rollback tid 0 to smithi07134777-34 2024-09-11T01:52:38.183 INFO:tasks.rados.rados.0.smithi071.stdout:3478: finishing write tid 1 to smithi07134777-20 2024-09-11T01:52:38.183 INFO:tasks.rados.rados.0.smithi071.stdout:3478: finishing write tid 2 to smithi07134777-20 2024-09-11T01:52:38.183 INFO:tasks.rados.rados.0.smithi071.stdout:3478: finishing write tid 3 to smithi07134777-20 2024-09-11T01:52:38.183 INFO:tasks.rados.rados.0.smithi071.stdout:3475: finishing rollback tid 1 to smithi07134777-34 2024-09-11T01:52:38.183 INFO:tasks.rados.rados.0.smithi071.stdout:3475: finishing rollback tid 2 to smithi07134777-34 2024-09-11T01:52:38.183 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 2439 (ObjNum 1121 snap 357 seq_num 1121) dirty exists 2024-09-11T01:52:38.183 INFO:tasks.rados.rados.0.smithi071.stdout:3478: finishing write tid 4 to smithi07134777-20 2024-09-11T01:52:38.183 INFO:tasks.rados.rados.0.smithi071.stdout:3478: oid 20 updating version 0 to 1184 2024-09-11T01:52:38.183 INFO:tasks.rados.rados.0.smithi071.stdout:3478: oid 20 updating version 1184 to 1185 2024-09-11T01:52:38.183 INFO:tasks.rados.rados.0.smithi071.stdout:3478: oid 20 version 1185 is already newer than 1183 2024-09-11T01:52:38.183 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 1185 (ObjNum 1207 snap 393 seq_num 1207) dirty exists 2024-09-11T01:52:38.183 INFO:tasks.rados.rados.0.smithi071.stdout:3478: left oid 20 (ObjNum 1207 snap 393 seq_num 1207) 2024-09-11T01:52:38.183 INFO:tasks.rados.rados.0.smithi071.stdout:3476: finishing rollback tid 0 to smithi07134777-33 2024-09-11T01:52:38.183 INFO:tasks.rados.rados.0.smithi071.stdout:3474: done (8 left) 2024-09-11T01:52:38.184 INFO:tasks.rados.rados.0.smithi071.stdout:3475: done (7 left) 2024-09-11T01:52:38.184 INFO:tasks.rados.rados.0.smithi071.stdout:3477: done (6 left) 2024-09-11T01:52:38.184 INFO:tasks.rados.rados.0.smithi071.stdout:3478: done (5 left) 2024-09-11T01:52:38.184 INFO:tasks.rados.rados.0.smithi071.stdout:3483: snap_create 2024-09-11T01:52:38.184 INFO:tasks.rados.rados.0.smithi071.stdout:3476: finishing rollback tid 1 to smithi07134777-33 2024-09-11T01:52:38.184 INFO:tasks.rados.rados.0.smithi071.stdout:3476: finishing rollback tid 2 to smithi07134777-33 2024-09-11T01:52:38.184 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 2443 (ObjNum 1130 snap 357 seq_num 1130) dirty exists 2024-09-11T01:52:38.184 INFO:tasks.rados.rados.0.smithi071.stdout:3479: finishing write tid 1 to smithi07134777-12 2024-09-11T01:52:38.184 INFO:tasks.rados.rados.0.smithi071.stdout:3479: finishing write tid 2 to smithi07134777-12 2024-09-11T01:52:38.184 INFO:tasks.rados.rados.0.smithi071.stdout:3479: finishing write tid 3 to smithi07134777-12 2024-09-11T01:52:38.184 INFO:tasks.rados.rados.0.smithi071.stdout:3479: finishing write tid 4 to smithi07134777-12 2024-09-11T01:52:38.184 INFO:tasks.rados.rados.0.smithi071.stdout:3479: oid 12 updating version 0 to 2446 2024-09-11T01:52:38.184 INFO:tasks.rados.rados.0.smithi071.stdout:3479: oid 12 version 2446 is already newer than 2445 2024-09-11T01:52:38.184 INFO:tasks.rados.rados.0.smithi071.stdout:3479: oid 12 updating version 2446 to 2447 2024-09-11T01:52:38.185 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 2447 (ObjNum 1208 snap 393 seq_num 1208) dirty exists 2024-09-11T01:52:38.185 INFO:tasks.rados.rados.0.smithi071.stdout:3479: left oid 12 (ObjNum 1208 snap 393 seq_num 1208) 2024-09-11T01:52:38.185 INFO:tasks.rados.rados.0.smithi071.stdout:3480: finishing write tid 1 to smithi07134777-1 2024-09-11T01:52:38.185 INFO:tasks.rados.rados.0.smithi071.stdout:3482: finishing rollback tid 1 to smithi07134777-41 2024-09-11T01:52:38.185 INFO:tasks.rados.rados.0.smithi071.stdout:3482: finishing rollback tid 2 to smithi07134777-41 2024-09-11T01:52:38.185 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 1967 (ObjNum 832 snap 254 seq_num 832) dirty exists 2024-09-11T01:52:38.186 INFO:tasks.rados.rados.0.smithi071.stdout:3481: expect (ObjNum 1131 snap 357 seq_num 1131) 2024-09-11T01:52:38.186 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:52:38.204 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-11T01:52:38.204 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T01:52:38.205 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T01:52:38.207 INFO:tasks.ceph.mon.a:Started 2024-09-11T01:52:38.208 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:52:38.208 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:52:38.266 INFO:tasks.rados.rados.0.smithi071.stdout:3480: finishing write tid 2 to smithi07134777-1 2024-09-11T01:52:38.266 INFO:tasks.rados.rados.0.smithi071.stdout:3480: finishing write tid 3 to smithi07134777-1 2024-09-11T01:52:38.266 INFO:tasks.rados.rados.0.smithi071.stdout:3480: finishing write tid 4 to smithi07134777-1 2024-09-11T01:52:38.266 INFO:tasks.rados.rados.0.smithi071.stdout:3480: oid 1 updating version 0 to 2553 2024-09-11T01:52:38.266 INFO:tasks.rados.rados.0.smithi071.stdout:3480: oid 1 updating version 2553 to 2554 2024-09-11T01:52:38.266 INFO:tasks.rados.rados.0.smithi071.stdout:3480: oid 1 updating version 2554 to 2555 2024-09-11T01:52:38.266 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 2555 (ObjNum 1209 snap 393 seq_num 1209) dirty exists 2024-09-11T01:52:38.266 INFO:tasks.rados.rados.0.smithi071.stdout:3480: left oid 1 (ObjNum 1209 snap 393 seq_num 1209) 2024-09-11T01:52:38.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.313+0000 7f1a26cb8640 10 monclient: get_monmap_and_config 2024-09-11T01:52:38.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.313+0000 7f1a26cb8640 10 monclient: build_initial_monmap 2024-09-11T01:52:38.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.313+0000 7f1a26cb8640 10 monclient: monmap: 2024-09-11T01:52:38.315 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:38.316 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:38.316 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:38.314783+0000 2024-09-11T01:52:38.316 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:38.314783+0000 2024-09-11T01:52:38.316 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:38.316 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:38.316 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:38.316 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:38.316 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:38.316 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:38.316 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:38.316 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:38.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.313+0000 7f1a26cb8640 1 Processor -- start 2024-09-11T01:52:38.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.313+0000 7f1a26cb8640 1 -- start start 2024-09-11T01:52:38.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.313+0000 7f1a26cb8640 10 monclient: init 2024-09-11T01:52:38.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a26cb8640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:38.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a26cb8640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-11T01:52:38.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a26cb8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1a20138210 addr v1:172.21.15.71:6789/0 2024-09-11T01:52:38.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a26cb8640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1a200a6810 0x7f1a2013d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:38.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a26cb8640 10 monclient(hunting): picked mon.noname-c con 0x7f1a200a6810 addr v2:172.21.15.71:3300/0 2024-09-11T01:52:38.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a26cb8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1a2013d640 0x7f1a2013fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:38.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a26cb8640 10 monclient(hunting): picked mon.noname-a con 0x7f1a2013d640 addr v2:172.21.15.12:3300/0 2024-09-11T01:52:38.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a26cb8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a20138d00 con 0x7f1a20138210 2024-09-11T01:52:38.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a26cb8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:38.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a26cb8640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f1a2009c4c0 con 0x7f1a2013d640 2024-09-11T01:52:38.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a26cb8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:38.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a26cb8640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f1a2009ba20 con 0x7f1a200a6810 2024-09-11T01:52:38.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a26cb8640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:38.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a26cb8640 10 monclient(hunting): authenticate will time out at 2164.757436s 2024-09-11T01:52:38.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a25255640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1a2013d640 0x7f1a2013fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:38.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a25255640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1a2013d640 0x7f1a2013fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50844/0 (socket says 172.21.15.12:50844) 2024-09-11T01:52:38.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a25255640 1 -- 172.21.15.12:0/2136305934 learned_addr learned my addr 172.21.15.12:0/2136305934 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:38.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a25255640 10 monclient(hunting): get_auth_request con 0x7f1a2013d640 auth_method 0 2024-09-11T01:52:38.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a25255640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:38.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a25255640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:38.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a25255640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:38.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a25255640 1 -- 172.21.15.12:0/2136305934 >> v2:172.21.15.12:3300/0 conn(0x7f1a2013d640 msgr2=0x7f1a2013fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:52:38.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a25255640 1 -- 172.21.15.12:0/2136305934 >> v2:172.21.15.12:3300/0 conn(0x7f1a2013d640 msgr2=0x7f1a2013fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:52:38.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a25255640 1 --2- 172.21.15.12:0/2136305934 >> v2:172.21.15.12:3300/0 conn(0x7f1a2013d640 0x7f1a2013fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:52:38.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a25255640 1 --2- 172.21.15.12:0/2136305934 >> v2:172.21.15.12:3300/0 conn(0x7f1a2013d640 0x7f1a2013fa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:38.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a1ffff640 1 --2- 172.21.15.12:0/2136305934 >> v2:172.21.15.71:3300/0 conn(0x7f1a200a6810 0x7f1a2013d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:38.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a1ffff640 10 monclient(hunting): get_auth_request con 0x7f1a200a6810 auth_method 0 2024-09-11T01:52:38.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:38.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:38.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:38.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:38.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:38.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.314+0000 7f1a1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:38.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.315+0000 7f1a1ffff640 10 monclient(hunting): handle_auth_done global_id 234310 payload 274 2024-09-11T01:52:38.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.315+0000 7f1a1ffff640 1 -- 172.21.15.12:0/2136305934 >> v2:172.21.15.12:3300/0 conn(0x7f1a2013d640 msgr2=0x7f1a2013fa30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:52:38.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.315+0000 7f1a1ffff640 1 --2- 172.21.15.12:0/2136305934 >> v2:172.21.15.12:3300/0 conn(0x7f1a2013d640 0x7f1a2013fa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:38.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.315+0000 7f1a1ffff640 1 -- 172.21.15.12:0/2136305934 >> v1:172.21.15.71:6789/0 conn(0x7f1a20138210 legacy=0x7f1a201385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:38.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.315+0000 7f1a1ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:52:38.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.315+0000 7f1a1ffff640 1 monclient: found mon.noname-c 2024-09-11T01:52:38.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.315+0000 7f1a1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:38.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.315+0000 7f1a1ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:52:38.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.315+0000 7f1a1ffff640 1 -- 172.21.15.12:0/2136305934 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a200a6590 con 0x7f1a200a6810 2024-09-11T01:52:38.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a1ffff640 1 -- 172.21.15.12:0/2136305934 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a200a6810 msgr2=0x7f1a2013d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:38.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a1ffff640 1 --2- 172.21.15.12:0/2136305934 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a200a6810 0x7f1a2013d100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1a20074ee0 tx=0x7f1a1002a4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed40c8233b27889d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:38.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a1f7fe640 1 -- 172.21.15.12:0/2136305934 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a10007950 con 0x7f1a200a6810 2024-09-11T01:52:38.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:38.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a1f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:38.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a1f7fe640 10 monclient: dump: 2024-09-11T01:52:38.327 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:38.327 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:38.328 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:38.328 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:38.328 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:38.328 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:38.328 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:38.328 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:38.328 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:38.328 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:38.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a1f7fe640 10 monclient: _finish_auth 0 2024-09-11T01:52:38.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a1f7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:52:38.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:38.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a1f7fe640 1 -- 172.21.15.12:0/2136305934 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a10007af0 con 0x7f1a200a6810 2024-09-11T01:52:38.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a26cb8640 5 monclient: authenticate success, global_id 234310 2024-09-11T01:52:38.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a26cb8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:52:38.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a1f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:38.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a26cb8640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:38.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a26cb8640 4 set_mon_vals no callback set 2024-09-11T01:52:38.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a1f7fe640 1 -- 172.21.15.12:0/2136305934 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a10031400 con 0x7f1a200a6810 2024-09-11T01:52:38.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:38.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a1f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:52:38.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a1f7fe640 10 monclient: dump: 2024-09-11T01:52:38.329 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:38.329 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:38.329 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:38.329 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:38.329 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:38.330 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:38.330 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:38.330 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:38.330 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:38.330 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:38.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a26cb8640 10 monclient: shutdown 2024-09-11T01:52:38.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a26cb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:38.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a26cb8640 1 -- 172.21.15.12:0/2136305934 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a200a6810 msgr2=0x7f1a2013d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:38.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a26cb8640 1 --2- 172.21.15.12:0/2136305934 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a200a6810 0x7f1a2013d100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1a20074ee0 tx=0x7f1a1002a4b0 comp rx=0 tx=0).stop 2024-09-11T01:52:38.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a26cb8640 1 -- 172.21.15.12:0/2136305934 shutdown_connections 2024-09-11T01:52:38.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a26cb8640 1 --2- 172.21.15.12:0/2136305934 >> v2:172.21.15.12:3300/0 conn(0x7f1a2013d640 0x7f1a2013fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:38.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a26cb8640 1 --2- 172.21.15.12:0/2136305934 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a200a6810 0x7f1a2013d100 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:38.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a26cb8640 1 -- 172.21.15.12:0/2136305934 >> 172.21.15.12:0/2136305934 conn(0x7f1a200aaa80 msgr2=0x7f1a200aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:38.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a26cb8640 1 -- 172.21.15.12:0/2136305934 shutdown_connections 2024-09-11T01:52:38.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.318+0000 7f1a26cb8640 1 -- 172.21.15.12:0/2136305934 wait complete. 2024-09-11T01:52:38.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.319+0000 7f1a26cb8640 10 monclient: build_initial_monmap 2024-09-11T01:52:38.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.319+0000 7f1a26cb8640 10 monclient: monmap: 2024-09-11T01:52:38.331 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:38.331 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:38.331 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:38.331 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:38.331 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:38.331 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:38.331 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:38.331 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:38.331 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:38.331 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:38.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.319+0000 7f1a26cb8640 1 Processor -- start 2024-09-11T01:52:38.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.319+0000 7f1a26cb8640 1 -- start start 2024-09-11T01:52:38.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.319+0000 7f1a26cb8640 10 monclient: init 2024-09-11T01:52:38.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a26cb8640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:38.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a26cb8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:52:38.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a26cb8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1a200a6810 0x7f1a20136ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:38.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a26cb8640 10 monclient(hunting): picked mon.noname-c con 0x7f1a200a6810 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:38.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a26cb8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a20138210 0x7f1a20137220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:38.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a26cb8640 10 monclient(hunting): picked mon.noname-b con 0x7f1a20138210 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:38.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a26cb8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a2013d640 0x7f1a20139c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:38.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a26cb8640 10 monclient(hunting): picked mon.noname-a con 0x7f1a2013d640 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:38.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a26cb8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:38.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a26cb8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2006f670 con 0x7f1a2013d640 2024-09-11T01:52:38.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a26cb8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:38.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a26cb8640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2006f4f0 con 0x7f1a20138210 2024-09-11T01:52:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a26cb8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a26cb8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1a2013ff70 con 0x7f1a200a6810 2024-09-11T01:52:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a26cb8640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a26cb8640 10 monclient(hunting): authenticate will time out at 2164.763515s 2024-09-11T01:52:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a24a54640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1a200a6810 0x7f1a20136ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a25255640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a2013d640 0x7f1a20139c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a1ffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a20138210 0x7f1a20137220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a25255640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a2013d640 0x7f1a20139c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50856/0 (socket says 172.21.15.12:50856) 2024-09-11T01:52:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a1ffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a20138210 0x7f1a20137220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:48820/0 (socket says 172.21.15.12:48820) 2024-09-11T01:52:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a25255640 1 -- 172.21.15.12:0/2219637942 learned_addr learned my addr 172.21.15.12:0/2219637942 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a25255640 10 monclient(hunting): get_auth_request con 0x7f1a2013d640 auth_method 0 2024-09-11T01:52:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a25255640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a25255640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:38.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a24a54640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1a200a6810 0x7f1a20136ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:33618/0 (socket says 172.21.15.12:33618) 2024-09-11T01:52:38.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a25255640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:38.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a1ffff640 10 monclient(hunting): get_auth_request con 0x7f1a20138210 auth_method 0 2024-09-11T01:52:38.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:38.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:38.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:38.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a24a54640 10 monclient(hunting): get_auth_request con 0x7f1a200a6810 auth_method 0 2024-09-11T01:52:38.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a24a54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:38.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a24a54640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:38.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a24a54640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:38.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a25255640 1 -- 172.21.15.12:0/2219637942 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a2013d640 msgr2=0x7f1a20139c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:52:38.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a25255640 1 -- 172.21.15.12:0/2219637942 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a2013d640 msgr2=0x7f1a20139c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:52:38.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a25255640 1 --2- 172.21.15.12:0/2219637942 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a2013d640 0x7f1a20139c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:52:38.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a24a54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:38.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a24a54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:38.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a25255640 1 --2- 172.21.15.12:0/2219637942 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a2013d640 0x7f1a20139c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:38.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a24a54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:38.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:38.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:38.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.320+0000 7f1a1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:38.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.321+0000 7f1a24a54640 10 monclient(hunting): handle_auth_done global_id 244295 payload 824 2024-09-11T01:52:38.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.321+0000 7f1a24a54640 1 -- 172.21.15.12:0/2219637942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a20138210 msgr2=0x7f1a20137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:38.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.321+0000 7f1a24a54640 1 --2- 172.21.15.12:0/2219637942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a20138210 0x7f1a20137220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:38.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.321+0000 7f1a24a54640 1 -- 172.21.15.12:0/2219637942 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a2013d640 msgr2=0x7f1a20139c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:52:38.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.321+0000 7f1a24a54640 1 --2- 172.21.15.12:0/2219637942 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a2013d640 0x7f1a20139c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:38.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.321+0000 7f1a24a54640 10 monclient: _finish_hunting 0 2024-09-11T01:52:38.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.321+0000 7f1a24a54640 1 monclient: found mon.noname-c 2024-09-11T01:52:38.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.321+0000 7f1a24a54640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:38.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.321+0000 7f1a24a54640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:52:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.321+0000 7f1a24a54640 1 -- 172.21.15.12:0/2219637942 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a200a99a0 con 0x7f1a200a6810 2024-09-11T01:52:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.323+0000 7f1a24a54640 1 -- 172.21.15.12:0/2219637942 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1a200a6810 msgr2=0x7f1a20136ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.323+0000 7f1a24a54640 1 --2- 172.21.15.12:0/2219637942 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1a200a6810 0x7f1a20136ce0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1a0800c410 tx=0x7f1a0800c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.323+0000 7f1a1d7fa640 1 -- 172.21.15.12:0/2219637942 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a08017070 con 0x7f1a200a6810 2024-09-11T01:52:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.323+0000 7f1a1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.323+0000 7f1a1d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.323+0000 7f1a1d7fa640 10 monclient: dump: 2024-09-11T01:52:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:38.336 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:38.337 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.323+0000 7f1a1d7fa640 10 monclient: _finish_auth 0 2024-09-11T01:52:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.323+0000 7f1a1d7fa640 10 monclient: _check_auth_tickets 2024-09-11T01:52:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.323+0000 7f1a1d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.323+0000 7f1a1d7fa640 1 -- 172.21.15.12:0/2219637942 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a08005230 con 0x7f1a200a6810 2024-09-11T01:52:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.323+0000 7f1a26cb8640 5 monclient: authenticate success, global_id 244295 2024-09-11T01:52:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.323+0000 7f1a26cb8640 10 monclient: _renew_subs 2024-09-11T01:52:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.323+0000 7f1a26cb8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.323+0000 7f1a26cb8640 1 -- 172.21.15.12:0/2219637942 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a20121560 con 0x7f1a200a6810 2024-09-11T01:52:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.323+0000 7f1a1d7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:38.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.323+0000 7f1a1d7fa640 1 -- 172.21.15.12:0/2219637942 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a080117a0 con 0x7f1a200a6810 2024-09-11T01:52:38.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.323+0000 7f1a1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:38.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.323+0000 7f1a1d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:38.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.323+0000 7f1a1d7fa640 10 monclient: dump: 2024-09-11T01:52:38.338 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:38.338 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:38.338 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:38.338 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:38.338 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:38.338 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:38.338 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:38.338 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:38.338 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:38.338 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:38.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.323+0000 7f1a1effd640 4 set_mon_vals no callback set 2024-09-11T01:52:38.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.323+0000 7f1a26cb8640 10 monclient: _renew_subs 2024-09-11T01:52:38.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.323+0000 7f1a26cb8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:38.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.323+0000 7f1a26cb8640 1 -- 172.21.15.12:0/2219637942 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a200a1880 con 0x7f1a200a6810 2024-09-11T01:52:38.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.324+0000 7f1a1d7fa640 1 -- 172.21.15.12:0/2219637942 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f1a08011940 con 0x7f1a200a6810 2024-09-11T01:52:38.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.324+0000 7f1a1d7fa640 1 --2- 172.21.15.12:0/2219637942 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1a0003e210 0x7f1a000406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:38.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.324+0000 7f1a26cb8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:38.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.324+0000 7f1a26cb8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:38.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.324+0000 7f1a26cb8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:38.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.324+0000 7f1a1d7fa640 1 -- 172.21.15.12:0/2219637942 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(788..788 src has 676..788) ==== 4843+0+0 (secure 0 0 0) 0x7f1a080523b0 con 0x7f1a200a6810 2024-09-11T01:52:38.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.324+0000 7f1a26cb8640 1 -- 172.21.15.12:0/2219637942 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a201385f0 con 0x7f1a200a6810 2024-09-11T01:52:38.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.325+0000 7f1a1ffff640 1 --2- 172.21.15.12:0/2219637942 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1a0003e210 0x7f1a000406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:38.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.325+0000 7f1a1ffff640 10 monclient: get_auth_request con 0x7f1a0003e210 auth_method 0 2024-09-11T01:52:38.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.327+0000 7f1a1ffff640 1 --2- 172.21.15.12:0/2219637942 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1a0003e210 0x7f1a000406d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f1a10031000 tx=0x7f1a1002f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:38.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.329+0000 7f1a1d7fa640 1 -- 172.21.15.12:0/2219637942 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1a200a99a0 con 0x7f1a200a6810 2024-09-11T01:52:38.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.329+0000 7f1a1d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:38.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.329+0000 7f1a1d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:38.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.584+0000 7f1a26cb8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:52:38.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.584+0000 7f1a26cb8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:52:38.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.584+0000 7f1a26cb8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:38.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.584+0000 7f1a26cb8640 1 -- 172.21.15.12:0/2219637942 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1a20120ce0 con 0x7f1a200a6810 2024-09-11T01:52:38.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.584+0000 7f1a1d7fa640 1 -- 172.21.15.12:0/2219637942 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f1a08052080 con 0x7f1a200a6810 2024-09-11T01:52:38.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.584+0000 7f1a1d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:52:38.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.584+0000 7f1a1d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:38.588 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":216,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:52:38.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.586+0000 7f1a26cb8640 1 -- 172.21.15.12:0/2219637942 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1a0003e210 msgr2=0x7f1a000406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:38.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.586+0000 7f1a26cb8640 1 --2- 172.21.15.12:0/2219637942 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1a0003e210 0x7f1a000406d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f1a10031000 tx=0x7f1a1002f000 comp rx=0 tx=0).stop 2024-09-11T01:52:38.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.586+0000 7f1a26cb8640 10 monclient: shutdown 2024-09-11T01:52:38.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.586+0000 7f1a26cb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:38.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.586+0000 7f1a26cb8640 1 -- 172.21.15.12:0/2219637942 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1a200a6810 msgr2=0x7f1a20136ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:38.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.586+0000 7f1a26cb8640 1 --2- 172.21.15.12:0/2219637942 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1a200a6810 0x7f1a20136ce0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1a0800c410 tx=0x7f1a0800c8d0 comp rx=0 tx=0).stop 2024-09-11T01:52:38.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.586+0000 7f1a26cb8640 1 -- 172.21.15.12:0/2219637942 shutdown_connections 2024-09-11T01:52:38.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.586+0000 7f1a26cb8640 1 --2- 172.21.15.12:0/2219637942 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1a0003e210 0x7f1a000406d0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:38.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.586+0000 7f1a26cb8640 1 --2- 172.21.15.12:0/2219637942 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a2013d640 0x7f1a20139c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:38.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.586+0000 7f1a26cb8640 1 --2- 172.21.15.12:0/2219637942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a20138210 0x7f1a20137220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:38.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.586+0000 7f1a26cb8640 1 --2- 172.21.15.12:0/2219637942 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1a200a6810 0x7f1a20136ce0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:38.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.586+0000 7f1a26cb8640 1 -- 172.21.15.12:0/2219637942 >> 172.21.15.12:0/2219637942 conn(0x7f1a200aaa80 msgr2=0x7f1a200a05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:38.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.586+0000 7f1a26cb8640 1 -- 172.21.15.12:0/2219637942 shutdown_connections 2024-09-11T01:52:38.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:38.586+0000 7f1a26cb8640 1 -- 172.21.15.12:0/2219637942 wait complete. 2024-09-11T01:52:39.315 INFO:tasks.rados.rados.0.smithi071.stdout:3476: done (5 left) 2024-09-11T01:52:39.315 INFO:tasks.rados.rados.0.smithi071.stdout:3479: done (4 left) 2024-09-11T01:52:39.315 INFO:tasks.rados.rados.0.smithi071.stdout:3480: done (3 left) 2024-09-11T01:52:39.315 INFO:tasks.rados.rados.0.smithi071.stdout:3481: done (2 left) 2024-09-11T01:52:39.315 INFO:tasks.rados.rados.0.smithi071.stdout:3482: done (1 left) 2024-09-11T01:52:39.315 INFO:tasks.rados.rados.0.smithi071.stdout:3483: done (0 left) 2024-09-11T01:52:39.315 INFO:tasks.rados.rados.0.smithi071.stdout:3484: snap_create 2024-09-11T01:52:40.317 INFO:tasks.rados.rados.0.smithi071.stdout:3484: done (0 left) 2024-09-11T01:52:40.317 INFO:tasks.rados.rados.0.smithi071.stdout:3485: copy_from oid 25 from oid 14 current snap is 395 2024-09-11T01:52:40.318 INFO:tasks.rados.rados.0.smithi071.stdout:3486: rollback oid 11 current snap is 395 2024-09-11T01:52:40.318 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 11 to 381 2024-09-11T01:52:40.318 INFO:tasks.rados.rados.0.smithi071.stdout:3487: snap_remove snap 372 2024-09-11T01:52:41.324 INFO:tasks.rados.rados.0.smithi071.stdout:3487: done (2 left) 2024-09-11T01:52:41.324 INFO:tasks.rados.rados.0.smithi071.stdout:3488: read oid 1 snap -1 2024-09-11T01:52:41.324 INFO:tasks.rados.rados.0.smithi071.stdout:3488: expect (ObjNum 1209 snap 393 seq_num 1209) 2024-09-11T01:52:41.324 INFO:tasks.rados.rados.0.smithi071.stdout:3489: rollback oid 30 current snap is 395 2024-09-11T01:52:41.324 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 30 to 325 2024-09-11T01:52:41.324 INFO:tasks.rados.rados.0.smithi071.stdout:3486: finishing rollback tid 0 to smithi07134777-11 2024-09-11T01:52:41.325 INFO:tasks.rados.rados.0.smithi071.stdout:3486: finishing rollback tid 1 to smithi07134777-11 2024-09-11T01:52:41.325 INFO:tasks.rados.rados.0.smithi071.stdout:3486: finishing rollback tid 2 to smithi07134777-11 2024-09-11T01:52:41.325 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 2194 (ObjNum 1115 snap 356 seq_num 1115) dirty exists 2024-09-11T01:52:41.325 INFO:tasks.rados.rados.0.smithi071.stdout:3485: finishing copy_from racing read to smithi07134777-25 2024-09-11T01:52:41.325 INFO:tasks.rados.rados.0.smithi071.stdout:3485: finishing copy_from to smithi07134777-25 2024-09-11T01:52:41.325 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 2572 (ObjNum 1136 snap 360 seq_num 1136) dirty exists 2024-09-11T01:52:41.325 INFO:tasks.rados.rados.0.smithi071.stdout:3485: done (3 left) 2024-09-11T01:52:41.325 INFO:tasks.rados.rados.0.smithi071.stdout:3486: done (2 left) 2024-09-11T01:52:41.325 INFO:tasks.rados.rados.0.smithi071.stdout:3490: read oid 40 snap 284 2024-09-11T01:52:41.325 INFO:tasks.rados.rados.0.smithi071.stdout:3490: expect (ObjNum 865 snap 266 seq_num 865) 2024-09-11T01:52:41.325 INFO:tasks.rados.rados.0.smithi071.stdout:3491: rollback oid 11 current snap is 395 2024-09-11T01:52:41.325 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 11 to 340 2024-09-11T01:52:41.325 INFO:tasks.rados.rados.0.smithi071.stdout:3492: rollback oid 13 current snap is 395 2024-09-11T01:52:41.325 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 13 to 369 2024-09-11T01:52:41.326 INFO:tasks.rados.rados.0.smithi071.stdout:3493: rollback oid 18 current snap is 395 2024-09-11T01:52:41.326 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 18 to 284 2024-09-11T01:52:41.326 INFO:tasks.rados.rados.0.smithi071.stdout:3494: read oid 2 snap -1 2024-09-11T01:52:41.326 INFO:tasks.rados.rados.0.smithi071.stdout:3494: expect (ObjNum 1131 snap 357 seq_num 1131) 2024-09-11T01:52:41.326 INFO:tasks.rados.rados.0.smithi071.stdout:3495: rollback oid 10 current snap is 395 2024-09-11T01:52:41.326 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 10 to 394 2024-09-11T01:52:41.326 INFO:tasks.rados.rados.0.smithi071.stdout:3496: write oid 47 current snap is 395 2024-09-11T01:52:41.326 INFO:tasks.rados.rados.0.smithi071.stdout:3496: seq_num 1210 ranges {736470=751417,2022859=313611} 2024-09-11T01:52:41.333 INFO:tasks.rados.rados.0.smithi071.stdout:3496: writing smithi07134777-47 from 736470 to 1487887 tid 1 2024-09-11T01:52:41.338 INFO:tasks.rados.rados.0.smithi071.stdout:3496: writing smithi07134777-47 from 2022859 to 2336470 tid 2 2024-09-11T01:52:41.338 INFO:tasks.rados.rados.0.smithi071.stdout:3497: copy_from oid 45 from oid 4 current snap is 395 2024-09-11T01:52:41.339 INFO:tasks.rados.rados.0.smithi071.stdout:3494: expect (ObjNum 1131 snap 357 seq_num 1131) 2024-09-11T01:52:41.346 INFO:tasks.ceph.osd.5.smithi071.stderr:2024-09-11T01:52:41.344+0000 7fca4b8ae640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 17d after trimming was started 2024-09-11T01:52:41.361 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:52:41.417 INFO:tasks.rados.rados.0.smithi071.stdout:3495: finishing rollback tid 0 to smithi07134777-10 2024-09-11T01:52:41.418 INFO:tasks.rados.rados.0.smithi071.stdout:3488: expect (ObjNum 1209 snap 393 seq_num 1209) 2024-09-11T01:52:41.512 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:52:41.562 INFO:tasks.rados.rados.0.smithi071.stdout:3490: expect (ObjNum 865 snap 266 seq_num 865) 2024-09-11T01:52:41.562 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 164262 2024-09-11T01:52:41.600 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:52:41.641 INFO:tasks.rados.rados.0.smithi071.stdout:3495: finishing rollback tid 1 to smithi07134777-10 2024-09-11T01:52:41.641 INFO:tasks.rados.rados.0.smithi071.stdout:3495: finishing rollback tid 2 to smithi07134777-10 2024-09-11T01:52:41.641 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 1970 (ObjNum 1183 snap 378 seq_num 1183) dirty exists 2024-09-11T01:52:41.641 INFO:tasks.rados.rados.0.smithi071.stdout:3489: finishing rollback tid 0 to smithi07134777-30 2024-09-11T01:52:41.641 INFO:tasks.rados.rados.0.smithi071.stdout:3492: finishing rollback tid 1 to smithi07134777-13 2024-09-11T01:52:41.641 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 1852 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-11T01:52:41.641 INFO:tasks.rados.rados.0.smithi071.stdout:3488: done (9 left) 2024-09-11T01:52:41.641 INFO:tasks.rados.rados.0.smithi071.stdout:3490: done (8 left) 2024-09-11T01:52:41.641 INFO:tasks.rados.rados.0.smithi071.stdout:3492: done (7 left) 2024-09-11T01:52:41.641 INFO:tasks.rados.rados.0.smithi071.stdout:3494: done (6 left) 2024-09-11T01:52:41.641 INFO:tasks.rados.rados.0.smithi071.stdout:3495: done (5 left) 2024-09-11T01:52:41.641 INFO:tasks.rados.rados.0.smithi071.stdout:3498: snap_create 2024-09-11T01:52:41.642 INFO:tasks.rados.rados.0.smithi071.stdout:3489: finishing rollback tid 1 to smithi07134777-30 2024-09-11T01:52:41.642 INFO:tasks.rados.rados.0.smithi071.stdout:3489: finishing rollback tid 2 to smithi07134777-30 2024-09-11T01:52:41.642 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 2197 (ObjNum 1003 snap 322 seq_num 1003) dirty exists 2024-09-11T01:52:41.642 INFO:tasks.rados.rados.0.smithi071.stdout:3491: finishing rollback tid 0 to smithi07134777-11 2024-09-11T01:52:41.642 INFO:tasks.rados.rados.0.smithi071.stdout:3496: finishing write tid 1 to smithi07134777-47 2024-09-11T01:52:41.642 INFO:tasks.rados.rados.0.smithi071.stdout:3491: finishing rollback tid 1 to smithi07134777-11 2024-09-11T01:52:41.642 INFO:tasks.rados.rados.0.smithi071.stdout:3491: finishing rollback tid 2 to smithi07134777-11 2024-09-11T01:52:41.642 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 2200 (ObjNum 987 snap 317 seq_num 987) dirty exists 2024-09-11T01:52:41.642 INFO:tasks.rados.rados.0.smithi071.stdout:3493: finishing rollback tid 0 to smithi07134777-18 2024-09-11T01:52:41.642 INFO:tasks.rados.rados.0.smithi071.stdout:3493: finishing rollback tid 1 to smithi07134777-18 2024-09-11T01:52:41.642 INFO:tasks.rados.rados.0.smithi071.stdout:3493: finishing rollback tid 2 to smithi07134777-18 2024-09-11T01:52:41.643 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 2203 (ObjNum 618 snap 183 seq_num 618) dirty exists 2024-09-11T01:52:41.643 INFO:tasks.rados.rados.0.smithi071.stdout:3497: finishing copy_from to smithi07134777-45 2024-09-11T01:52:41.643 INFO:tasks.rados.rados.0.smithi071.stdout:3497: got expected ENOENT (src dne) 2024-09-11T01:52:41.643 INFO:tasks.rados.rados.0.smithi071.stdout:3497: finishing copy_from racing read to smithi07134777-45 2024-09-11T01:52:41.643 INFO:tasks.rados.rados.0.smithi071.stdout:3496: finishing write tid 2 to smithi07134777-47 2024-09-11T01:52:41.643 INFO:tasks.rados.rados.0.smithi071.stdout:3496: finishing write tid 3 to smithi07134777-47 2024-09-11T01:52:41.643 INFO:tasks.rados.rados.0.smithi071.stdout:3496: finishing write tid 4 to smithi07134777-47 2024-09-11T01:52:41.643 INFO:tasks.rados.rados.0.smithi071.stdout:3496: oid 47 updating version 0 to 2558 2024-09-11T01:52:41.643 INFO:tasks.rados.rados.0.smithi071.stdout:3496: oid 47 version 2558 is already newer than 2557 2024-09-11T01:52:41.643 INFO:tasks.rados.rados.0.smithi071.stdout:3496: oid 47 version 2558 is already newer than 2556 2024-09-11T01:52:41.643 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 2558 (ObjNum 1210 snap 395 seq_num 1210) dirty exists 2024-09-11T01:52:41.643 INFO:tasks.rados.rados.0.smithi071.stdout:3496: left oid 47 (ObjNum 1210 snap 395 seq_num 1210) 2024-09-11T01:52:41.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.711+0000 7f15b0b75640 10 monclient: get_monmap_and_config 2024-09-11T01:52:41.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.711+0000 7f15b0b75640 10 monclient: build_initial_monmap 2024-09-11T01:52:41.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.711+0000 7f15b0b75640 10 monclient: monmap: 2024-09-11T01:52:41.714 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:41.714 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:41.714 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:41.713304+0000 2024-09-11T01:52:41.714 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:41.713304+0000 2024-09-11T01:52:41.714 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:41.714 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:41.714 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:41.714 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:41.715 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:41.715 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:41.715 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:41.715 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:41.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.712+0000 7f15b0b75640 1 Processor -- start 2024-09-11T01:52:41.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.712+0000 7f15b0b75640 1 -- start start 2024-09-11T01:52:41.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.712+0000 7f15b0b75640 10 monclient: init 2024-09-11T01:52:41.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.712+0000 7f15b0b75640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:41.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.712+0000 7f15b0b75640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-11T01:52:41.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.712+0000 7f15b0b75640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15ac09d790 0x7f15ac09db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:41.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.712+0000 7f15b0b75640 10 monclient(hunting): picked mon.noname-b con 0x7f15ac09d790 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:41.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.712+0000 7f15b0b75640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f15ac09e0b0 0x7f15ac09e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:41.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.712+0000 7f15b0b75640 10 monclient(hunting): picked mon.noname-c con 0x7f15ac09e0b0 addr v2:172.21.15.71:3300/0 2024-09-11T01:52:41.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.712+0000 7f15b0b75640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f15ac134a60 addr v1:172.21.15.71:6789/0 2024-09-11T01:52:41.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.712+0000 7f15b0b75640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15ac0a1da0 con 0x7f15ac134a60 2024-09-11T01:52:41.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.712+0000 7f15b0b75640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:41.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.712+0000 7f15b0b75640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f15ac0a0760 con 0x7f15ac09e0b0 2024-09-11T01:52:41.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.712+0000 7f15b0b75640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:41.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.712+0000 7f15b0b75640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f15ac0a1fa0 con 0x7f15ac09d790 2024-09-11T01:52:41.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.712+0000 7f15b0b75640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:41.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.712+0000 7f15b0b75640 10 monclient(hunting): authenticate will time out at 2168.155827s 2024-09-11T01:52:41.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.712+0000 7f15aa59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15ac09d790 0x7f15ac09db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:41.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.712+0000 7f15aa59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15ac09d790 0x7f15ac09db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:33666/0 (socket says 172.21.15.12:33666) 2024-09-11T01:52:41.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15aa59c640 1 -- 172.21.15.12:0/1535765880 learned_addr learned my addr 172.21.15.12:0/1535765880 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:41.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15aa59c640 10 monclient(hunting): get_auth_request con 0x7f15ac09d790 auth_method 0 2024-09-11T01:52:41.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:41.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15aa59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:41.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:41.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15aa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:41.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15aa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:41.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15aa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:41.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15aa59c640 1 -- 172.21.15.12:0/1535765880 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15ac09d790 msgr2=0x7f15ac09db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:52:41.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15aa59c640 1 -- 172.21.15.12:0/1535765880 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15ac09d790 msgr2=0x7f15ac09db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:52:41.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15aa59c640 1 --2- 172.21.15.12:0/1535765880 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15ac09d790 0x7f15ac09db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:52:41.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15aa59c640 1 --2- 172.21.15.12:0/1535765880 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15ac09d790 0x7f15ac09db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:41.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15a1d9b640 1 --2- 172.21.15.12:0/1535765880 >> v2:172.21.15.71:3300/0 conn(0x7f15ac09e0b0 0x7f15ac09e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:41.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15a1d9b640 10 monclient(hunting): get_auth_request con 0x7f15ac09e0b0 auth_method 0 2024-09-11T01:52:41.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15a1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:41.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15a1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:41.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15a1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:41.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15a1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:41.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15a1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:41.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15a1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:41.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15a1d9b640 10 monclient(hunting): handle_auth_done global_id 234316 payload 274 2024-09-11T01:52:41.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15a1d9b640 1 -- 172.21.15.12:0/1535765880 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15ac09d790 msgr2=0x7f15ac09db70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:52:41.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15a1d9b640 1 --2- 172.21.15.12:0/1535765880 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15ac09d790 0x7f15ac09db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:41.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15a1d9b640 1 -- 172.21.15.12:0/1535765880 >> v1:172.21.15.71:6789/0 conn(0x7f15ac134a60 legacy=0x7f15ac134e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:41.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15a1d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:52:41.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15a1d9b640 1 monclient: found mon.noname-c 2024-09-11T01:52:41.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15a1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:41.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15a1d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:52:41.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.713+0000 7f15a1d9b640 1 -- 172.21.15.12:0/1535765880 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15ac09eda0 con 0x7f15ac09e0b0 2024-09-11T01:52:41.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.716+0000 7f15a1d9b640 1 -- 172.21.15.12:0/1535765880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15ac09e0b0 msgr2=0x7f15ac09e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:41.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.716+0000 7f15a1d9b640 1 --2- 172.21.15.12:0/1535765880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15ac09e0b0 0x7f15ac09e4d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f15940097a0 tx=0x7f1594029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ac9d33e857068b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:41.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.716+0000 7f15a9d9b640 1 -- 172.21.15.12:0/1535765880 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1594037070 con 0x7f15ac09e0b0 2024-09-11T01:52:41.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.716+0000 7f15a9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:41.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.716+0000 7f15a9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:41.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.716+0000 7f15a9d9b640 10 monclient: dump: 2024-09-11T01:52:41.728 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:41.728 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:41.728 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:41.728 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:41.728 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:41.728 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:41.728 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:41.728 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:41.728 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:41.728 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:41.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.716+0000 7f15a9d9b640 10 monclient: _finish_auth 0 2024-09-11T01:52:41.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.716+0000 7f15a9d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:52:41.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.716+0000 7f15a9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:41.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.716+0000 7f15b0b75640 5 monclient: authenticate success, global_id 234316 2024-09-11T01:52:41.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.716+0000 7f15b0b75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:52:41.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.716+0000 7f15a9d9b640 1 -- 172.21.15.12:0/1535765880 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1594032440 con 0x7f15ac09e0b0 2024-09-11T01:52:41.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.716+0000 7f15a9d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:41.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.716+0000 7f15b0b75640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:41.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.716+0000 7f15a9d9b640 1 -- 172.21.15.12:0/1535765880 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1594032810 con 0x7f15ac09e0b0 2024-09-11T01:52:41.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.716+0000 7f15a9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:41.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.716+0000 7f15b0b75640 4 set_mon_vals no callback set 2024-09-11T01:52:41.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.716+0000 7f15a9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:52:41.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.716+0000 7f15a9d9b640 10 monclient: dump: 2024-09-11T01:52:41.729 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:41.730 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:41.730 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:41.730 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:41.730 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:41.730 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:41.730 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:41.730 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:41.730 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:41.730 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:41.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.717+0000 7f15b0b75640 10 monclient: shutdown 2024-09-11T01:52:41.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.717+0000 7f15b0b75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:41.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.717+0000 7f15b0b75640 1 -- 172.21.15.12:0/1535765880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15ac09e0b0 msgr2=0x7f15ac09e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:41.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.717+0000 7f15b0b75640 1 --2- 172.21.15.12:0/1535765880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15ac09e0b0 0x7f15ac09e4d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f15940097a0 tx=0x7f1594029fd0 comp rx=0 tx=0).stop 2024-09-11T01:52:41.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.717+0000 7f15b0b75640 1 -- 172.21.15.12:0/1535765880 shutdown_connections 2024-09-11T01:52:41.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.717+0000 7f15b0b75640 1 --2- 172.21.15.12:0/1535765880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15ac09e0b0 0x7f15ac09e4d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:41.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.717+0000 7f15b0b75640 1 --2- 172.21.15.12:0/1535765880 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15ac09d790 0x7f15ac09db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:41.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.717+0000 7f15b0b75640 1 -- 172.21.15.12:0/1535765880 >> 172.21.15.12:0/1535765880 conn(0x7f15ac0a2d90 msgr2=0x7f15ac0a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:41.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.717+0000 7f15b0b75640 1 -- 172.21.15.12:0/1535765880 shutdown_connections 2024-09-11T01:52:41.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.717+0000 7f15b0b75640 1 -- 172.21.15.12:0/1535765880 wait complete. 2024-09-11T01:52:41.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.717+0000 7f15b0b75640 10 monclient: build_initial_monmap 2024-09-11T01:52:41.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.717+0000 7f15b0b75640 10 monclient: monmap: 2024-09-11T01:52:41.731 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:41.731 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:41.731 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:41.731 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:41.731 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:41.731 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:41.731 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:41.732 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:41.732 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:41.732 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:41.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.718+0000 7f15b0b75640 1 Processor -- start 2024-09-11T01:52:41.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.718+0000 7f15b0b75640 1 -- start start 2024-09-11T01:52:41.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.718+0000 7f15b0b75640 10 monclient: init 2024-09-11T01:52:41.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.718+0000 7f15b0b75640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:41.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.718+0000 7f15b0b75640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:52:41.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.718+0000 7f15b0b75640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15ac09d790 0x7f15ac1cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:41.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.718+0000 7f15b0b75640 10 monclient(hunting): picked mon.noname-a con 0x7f15ac09d790 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:41.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15b0b75640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15ac09e0b0 0x7f15ac06f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:41.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15b0b75640 10 monclient(hunting): picked mon.noname-c con 0x7f15ac09e0b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:41.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15b0b75640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15ac134a60 0x7f15ac06fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:41.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15b0b75640 10 monclient(hunting): picked mon.noname-b con 0x7f15ac134a60 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:41.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15b0b75640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:41.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15b0b75640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac09ce20 con 0x7f15ac09d790 2024-09-11T01:52:41.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15b0b75640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:41.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15b0b75640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac09cca0 con 0x7f15ac134a60 2024-09-11T01:52:41.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15b0b75640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:41.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15b0b75640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f15ac135590 con 0x7f15ac09e0b0 2024-09-11T01:52:41.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15b0b75640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:41.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15b0b75640 10 monclient(hunting): authenticate will time out at 2168.162063s 2024-09-11T01:52:41.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15a1d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15ac09e0b0 0x7f15ac06f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:41.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15aa59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15ac09d790 0x7f15ac1cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:41.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15aa59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15ac09d790 0x7f15ac1cd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50870/0 (socket says 172.21.15.12:50870) 2024-09-11T01:52:41.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15aa59c640 1 -- 172.21.15.12:0/2281332215 learned_addr learned my addr 172.21.15.12:0/2281332215 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:41.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15aa59c640 10 monclient(hunting): get_auth_request con 0x7f15ac09d790 auth_method 0 2024-09-11T01:52:41.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:41.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15aa59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:41.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:41.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15a1d9b640 10 monclient(hunting): get_auth_request con 0x7f15ac09e0b0 auth_method 0 2024-09-11T01:52:41.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15a1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:41.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15a1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:41.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15a1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:41.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15aa59c640 1 -- 172.21.15.12:0/2281332215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15ac09d790 msgr2=0x7f15ac1cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:52:41.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15aa59c640 1 -- 172.21.15.12:0/2281332215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15ac09d790 msgr2=0x7f15ac1cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:52:41.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15aa59c640 1 --2- 172.21.15.12:0/2281332215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15ac09d790 0x7f15ac1cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:52:41.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15aa59c640 1 --2- 172.21.15.12:0/2281332215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15ac09d790 0x7f15ac1cd3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:41.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15aad9d640 1 --2- 172.21.15.12:0/2281332215 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15ac134a60 0x7f15ac06fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15a1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15a1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15a1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15aad9d640 10 monclient(hunting): get_auth_request con 0x7f15ac134a60 auth_method 0 2024-09-11T01:52:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15aad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15a1d9b640 10 monclient(hunting): handle_auth_done global_id 244301 payload 824 2024-09-11T01:52:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15a1d9b640 1 -- 172.21.15.12:0/2281332215 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15ac134a60 msgr2=0x7f15ac06fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15a1d9b640 1 --2- 172.21.15.12:0/2281332215 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15ac134a60 0x7f15ac06fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15a1d9b640 1 -- 172.21.15.12:0/2281332215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15ac09d790 msgr2=0x7f15ac1cd3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:52:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15a1d9b640 1 --2- 172.21.15.12:0/2281332215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15ac09d790 0x7f15ac1cd3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15a1d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:52:41.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15a1d9b640 1 monclient: found mon.noname-c 2024-09-11T01:52:41.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15a1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:41.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15a1d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:52:41.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15a1d9b640 1 -- 172.21.15.12:0/2281332215 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15ac1280a0 con 0x7f15ac09e0b0 2024-09-11T01:52:41.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.719+0000 7f15aad9d640 1 --2- 172.21.15.12:0/2281332215 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15ac134a60 0x7f15ac06fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:52:41.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.721+0000 7f15a1d9b640 1 -- 172.21.15.12:0/2281332215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15ac09e0b0 msgr2=0x7f15ac06f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:41.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.721+0000 7f15a1d9b640 1 --2- 172.21.15.12:0/2281332215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15ac09e0b0 0x7f15ac06f5a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f15ac09eac0 tx=0x7f15940046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:41.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.722+0000 7f15a37fe640 1 -- 172.21.15.12:0/2281332215 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1594037040 con 0x7f15ac09e0b0 2024-09-11T01:52:41.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.722+0000 7f15a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:41.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.722+0000 7f15a37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:41.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.722+0000 7f15a37fe640 10 monclient: dump: 2024-09-11T01:52:41.736 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:41.736 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:41.736 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:41.737 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:41.737 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:41.737 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:41.737 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:41.737 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:41.737 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:41.737 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:41.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.722+0000 7f15a37fe640 10 monclient: _finish_auth 0 2024-09-11T01:52:41.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.722+0000 7f15a37fe640 10 monclient: _check_auth_tickets 2024-09-11T01:52:41.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.722+0000 7f15a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:41.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.722+0000 7f15a37fe640 1 -- 172.21.15.12:0/2281332215 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1594042070 con 0x7f15ac09e0b0 2024-09-11T01:52:41.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.722+0000 7f15b0b75640 5 monclient: authenticate success, global_id 244301 2024-09-11T01:52:41.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.722+0000 7f15b0b75640 10 monclient: _renew_subs 2024-09-11T01:52:41.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.722+0000 7f15b0b75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:41.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.722+0000 7f15b0b75640 1 -- 172.21.15.12:0/2281332215 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15ac0a4930 con 0x7f15ac09e0b0 2024-09-11T01:52:41.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.722+0000 7f15a37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:41.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.722+0000 7f15a37fe640 1 -- 172.21.15.12:0/2281332215 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159403bb00 con 0x7f15ac09e0b0 2024-09-11T01:52:41.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.722+0000 7f15a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:41.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.722+0000 7f15a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:41.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.722+0000 7f15a37fe640 10 monclient: dump: 2024-09-11T01:52:41.738 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:41.738 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:41.738 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:41.738 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:41.738 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:41.738 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:41.739 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:41.739 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:41.739 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:41.739 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:41.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.722+0000 7f15a959a640 4 set_mon_vals no callback set 2024-09-11T01:52:41.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.722+0000 7f15b0b75640 10 monclient: _renew_subs 2024-09-11T01:52:41.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.722+0000 7f15b0b75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:41.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.722+0000 7f15b0b75640 1 -- 172.21.15.12:0/2281332215 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f15ac127dc0 con 0x7f15ac09e0b0 2024-09-11T01:52:41.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.723+0000 7f15a37fe640 1 -- 172.21.15.12:0/2281332215 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f159403a030 con 0x7f15ac09e0b0 2024-09-11T01:52:41.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.723+0000 7f15b0b75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:41.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.723+0000 7f15b0b75640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:41.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.723+0000 7f15b0b75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:41.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.723+0000 7f15b0b75640 1 -- 172.21.15.12:0/2281332215 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15ac09db70 con 0x7f15ac09e0b0 2024-09-11T01:52:41.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.723+0000 7f15a37fe640 1 --2- 172.21.15.12:0/2281332215 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f158c03e210 0x7f158c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:41.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.723+0000 7f15a37fe640 1 -- 172.21.15.12:0/2281332215 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(791..791 src has 676..791) ==== 4799+0+0 (secure 0 0 0) 0x7f1594073150 con 0x7f15ac09e0b0 2024-09-11T01:52:41.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.723+0000 7f15aa59c640 1 --2- 172.21.15.12:0/2281332215 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f158c03e210 0x7f158c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:41.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.723+0000 7f15aa59c640 10 monclient: get_auth_request con 0x7f158c03e210 auth_method 0 2024-09-11T01:52:41.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.726+0000 7f15aa59c640 1 --2- 172.21.15.12:0/2281332215 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f158c03e210 0x7f158c0406d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f159c006a90 tx=0x7f159c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:41.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.728+0000 7f15a37fe640 1 -- 172.21.15.12:0/2281332215 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f159403c3a0 con 0x7f15ac09e0b0 2024-09-11T01:52:41.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.728+0000 7f15a37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:41.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.728+0000 7f15a37fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:41.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.988+0000 7f15b0b75640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:52:41.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.988+0000 7f15b0b75640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:52:41.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.988+0000 7f15b0b75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:41.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.988+0000 7f15b0b75640 1 -- 172.21.15.12:0/2281332215 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f15ac06d460 con 0x7f15ac09e0b0 2024-09-11T01:52:41.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.989+0000 7f15a37fe640 1 -- 172.21.15.12:0/2281332215 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f1594071b40 con 0x7f15ac09e0b0 2024-09-11T01:52:41.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.989+0000 7f15a37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:52:41.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.989+0000 7f15a37fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:41.992 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":216,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:52:41.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.991+0000 7f15b0b75640 1 -- 172.21.15.12:0/2281332215 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f158c03e210 msgr2=0x7f158c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:41.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.991+0000 7f15b0b75640 1 --2- 172.21.15.12:0/2281332215 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f158c03e210 0x7f158c0406d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f159c006a90 tx=0x7f159c008040 comp rx=0 tx=0).stop 2024-09-11T01:52:41.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.991+0000 7f15b0b75640 10 monclient: shutdown 2024-09-11T01:52:41.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.991+0000 7f15b0b75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:41.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.991+0000 7f15b0b75640 1 -- 172.21.15.12:0/2281332215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15ac09e0b0 msgr2=0x7f15ac06f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:41.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.991+0000 7f15b0b75640 1 --2- 172.21.15.12:0/2281332215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15ac09e0b0 0x7f15ac06f5a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f15ac09eac0 tx=0x7f15940046c0 comp rx=0 tx=0).stop 2024-09-11T01:52:41.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.991+0000 7f15b0b75640 1 -- 172.21.15.12:0/2281332215 shutdown_connections 2024-09-11T01:52:41.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.991+0000 7f15b0b75640 1 --2- 172.21.15.12:0/2281332215 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15ac134a60 0x7f15ac06fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:41.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.991+0000 7f15b0b75640 1 --2- 172.21.15.12:0/2281332215 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15ac09e0b0 0x7f15ac06f5a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:41.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.991+0000 7f15b0b75640 1 --2- 172.21.15.12:0/2281332215 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f158c03e210 0x7f158c0406d0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:41.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.991+0000 7f15b0b75640 1 --2- 172.21.15.12:0/2281332215 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15ac09d790 0x7f15ac1cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:41.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.991+0000 7f15b0b75640 1 -- 172.21.15.12:0/2281332215 >> 172.21.15.12:0/2281332215 conn(0x7f15ac0a2d90 msgr2=0x7f15ac11c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:41.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.991+0000 7f15b0b75640 1 -- 172.21.15.12:0/2281332215 shutdown_connections 2024-09-11T01:52:41.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:41.991+0000 7f15b0b75640 1 -- 172.21.15.12:0/2281332215 wait complete. 2024-09-11T01:52:42.344 INFO:tasks.rados.rados.0.smithi071.stdout:3489: done (5 left) 2024-09-11T01:52:42.344 INFO:tasks.rados.rados.0.smithi071.stdout:3491: done (4 left) 2024-09-11T01:52:42.344 INFO:tasks.rados.rados.0.smithi071.stdout:3493: done (3 left) 2024-09-11T01:52:42.344 INFO:tasks.rados.rados.0.smithi071.stdout:3496: done (2 left) 2024-09-11T01:52:42.344 INFO:tasks.rados.rados.0.smithi071.stdout:3497: done (1 left) 2024-09-11T01:52:42.344 INFO:tasks.rados.rados.0.smithi071.stdout:3498: done (0 left) 2024-09-11T01:52:42.344 INFO:tasks.rados.rados.0.smithi071.stdout:3499: rollback oid 11 current snap is 396 2024-09-11T01:52:42.344 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 11 to 384 2024-09-11T01:52:42.344 INFO:tasks.rados.rados.0.smithi071.stdout:3500: delete oid 38 current snap is 396 2024-09-11T01:52:42.349 INFO:tasks.rados.rados.0.smithi071.stdout:3500: done (1 left) 2024-09-11T01:52:42.350 INFO:tasks.rados.rados.0.smithi071.stdout:3501: rollback oid 1 current snap is 396 2024-09-11T01:52:42.350 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 1 to 385 2024-09-11T01:52:42.350 INFO:tasks.rados.rados.0.smithi071.stdout:3502: snap_create 2024-09-11T01:52:42.350 INFO:tasks.rados.rados.0.smithi071.stdout:3499: finishing rollback tid 0 to smithi07134777-11 2024-09-11T01:52:42.350 INFO:tasks.rados.rados.0.smithi071.stdout:3499: finishing rollback tid 1 to smithi07134777-11 2024-09-11T01:52:42.350 INFO:tasks.rados.rados.0.smithi071.stdout:3499: finishing rollback tid 2 to smithi07134777-11 2024-09-11T01:52:42.351 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 2206 (ObjNum 1115 snap 356 seq_num 1115) dirty exists 2024-09-11T01:52:42.354 INFO:tasks.rados.rados.0.smithi071.stdout:3501: finishing rollback tid 0 to smithi07134777-1 2024-09-11T01:52:42.354 INFO:tasks.rados.rados.0.smithi071.stdout:3501: finishing rollback tid 1 to smithi07134777-1 2024-09-11T01:52:42.354 INFO:tasks.rados.rados.0.smithi071.stdout:3501: finishing rollback tid 2 to smithi07134777-1 2024-09-11T01:52:42.354 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 2561 (ObjNum 1154 snap 366 seq_num 1154) dirty exists 2024-09-11T01:52:43.142 INFO:tasks.rados.rados.0.smithi071.stdout:3499: done (2 left) 2024-09-11T01:52:43.142 INFO:tasks.rados.rados.0.smithi071.stdout:3501: done (1 left) 2024-09-11T01:52:43.142 INFO:tasks.rados.rados.0.smithi071.stdout:3502: done (0 left) 2024-09-11T01:52:43.142 INFO:tasks.rados.rados.0.smithi071.stdout:3503: write oid 32 current snap is 397 2024-09-11T01:52:43.143 INFO:tasks.rados.rados.0.smithi071.stdout:3503: seq_num 1212 ranges {770084=501915,2023470=346614} 2024-09-11T01:52:43.149 INFO:tasks.rados.rados.0.smithi071.stdout:3503: writing smithi07134777-32 from 770084 to 1271999 tid 1 2024-09-11T01:52:43.155 INFO:tasks.rados.rados.0.smithi071.stdout:3503: writing smithi07134777-32 from 2023470 to 2370084 tid 2 2024-09-11T01:52:43.155 INFO:tasks.rados.rados.0.smithi071.stdout:3504: snap_remove snap 379 2024-09-11T01:52:44.150 INFO:tasks.rados.rados.0.smithi071.stdout:3504: done (1 left) 2024-09-11T01:52:44.150 INFO:tasks.rados.rados.0.smithi071.stdout:3505: delete oid 29 current snap is 397 2024-09-11T01:52:44.151 INFO:tasks.rados.rados.0.smithi071.stdout:3503: finishing write tid 1 to smithi07134777-32 2024-09-11T01:52:44.151 INFO:tasks.rados.rados.0.smithi071.stdout:3503: finishing write tid 2 to smithi07134777-32 2024-09-11T01:52:44.151 INFO:tasks.rados.rados.0.smithi071.stdout:3503: finishing write tid 3 to smithi07134777-32 2024-09-11T01:52:44.151 INFO:tasks.rados.rados.0.smithi071.stdout:3503: finishing write tid 4 to smithi07134777-32 2024-09-11T01:52:44.151 INFO:tasks.rados.rados.0.smithi071.stdout:3503: oid 32 updating version 0 to 1991 2024-09-11T01:52:44.151 INFO:tasks.rados.rados.0.smithi071.stdout:3503: oid 32 updating version 1991 to 1993 2024-09-11T01:52:44.152 INFO:tasks.rados.rados.0.smithi071.stdout:3503: oid 32 version 1993 is already newer than 1992 2024-09-11T01:52:44.152 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 1993 (ObjNum 1212 snap 397 seq_num 1212) dirty exists 2024-09-11T01:52:44.152 INFO:tasks.rados.rados.0.smithi071.stdout:3503: left oid 32 (ObjNum 1212 snap 397 seq_num 1212) 2024-09-11T01:52:44.159 INFO:tasks.rados.rados.0.smithi071.stdout:3503: done (1 left) 2024-09-11T01:52:44.159 INFO:tasks.rados.rados.0.smithi071.stdout:3505: done (0 left) 2024-09-11T01:52:44.159 INFO:tasks.rados.rados.0.smithi071.stdout:3506: write oid 28 current snap is 397 2024-09-11T01:52:44.159 INFO:tasks.rados.rados.0.smithi071.stdout:3506: seq_num 1214 ranges {403698=652413,1740434=663264} 2024-09-11T01:52:44.164 INFO:tasks.rados.rados.0.smithi071.stdout:3506: writing smithi07134777-28 from 403698 to 1056111 tid 1 2024-09-11T01:52:44.172 INFO:tasks.rados.rados.0.smithi071.stdout:3506: writing smithi07134777-28 from 1740434 to 2403698 tid 2 2024-09-11T01:52:44.173 INFO:tasks.rados.rados.0.smithi071.stdout:3507: delete oid 46 current snap is 397 2024-09-11T01:52:44.173 INFO:tasks.rados.rados.0.smithi071.stdout:3506: finishing write tid 1 to smithi07134777-28 2024-09-11T01:52:44.176 INFO:tasks.rados.rados.0.smithi071.stdout:3507: done (1 left) 2024-09-11T01:52:44.176 INFO:tasks.rados.rados.0.smithi071.stdout:3508: rollback oid 43 current snap is 397 2024-09-11T01:52:44.176 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 43 to 386 2024-09-11T01:52:44.177 INFO:tasks.rados.rados.0.smithi071.stdout:3509: write oid 20 current snap is 397 2024-09-11T01:52:44.177 INFO:tasks.rados.rados.0.smithi071.stdout:3509: seq_num 1216 ranges {437312=402911,1341045=696749} 2024-09-11T01:52:44.181 INFO:tasks.rados.rados.0.smithi071.stdout:3509: writing smithi07134777-20 from 437312 to 840223 tid 1 2024-09-11T01:52:44.190 INFO:tasks.rados.rados.0.smithi071.stdout:3509: writing smithi07134777-20 from 1341045 to 2037794 tid 2 2024-09-11T01:52:44.190 INFO:tasks.rados.rados.0.smithi071.stdout:3510: copy_from oid 9 from oid 41 current snap is 397 2024-09-11T01:52:44.190 INFO:tasks.rados.rados.0.smithi071.stdout:3506: finishing write tid 2 to smithi07134777-28 2024-09-11T01:52:44.190 INFO:tasks.rados.rados.0.smithi071.stdout:3506: finishing write tid 3 to smithi07134777-28 2024-09-11T01:52:44.190 INFO:tasks.rados.rados.0.smithi071.stdout:3506: finishing write tid 4 to smithi07134777-28 2024-09-11T01:52:44.191 INFO:tasks.rados.rados.0.smithi071.stdout:3506: oid 28 updating version 0 to 2150 2024-09-11T01:52:44.191 INFO:tasks.rados.rados.0.smithi071.stdout:3506: oid 28 updating version 2150 to 2151 2024-09-11T01:52:44.191 INFO:tasks.rados.rados.0.smithi071.stdout:3506: oid 28 version 2151 is already newer than 2149 2024-09-11T01:52:44.191 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 2151 (ObjNum 1214 snap 397 seq_num 1214) dirty exists 2024-09-11T01:52:44.191 INFO:tasks.rados.rados.0.smithi071.stdout:3506: left oid 28 (ObjNum 1214 snap 397 seq_num 1214) 2024-09-11T01:52:44.191 INFO:tasks.rados.rados.0.smithi071.stdout:3508: finishing rollback tid 0 to smithi07134777-43 2024-09-11T01:52:44.191 INFO:tasks.rados.rados.0.smithi071.stdout:3508: finishing rollback tid 1 to smithi07134777-43 2024-09-11T01:52:44.192 INFO:tasks.rados.rados.0.smithi071.stdout:3509: finishing write tid 1 to smithi07134777-20 2024-09-11T01:52:44.192 INFO:tasks.rados.rados.0.smithi071.stdout:3506: done (3 left) 2024-09-11T01:52:44.192 INFO:tasks.rados.rados.0.smithi071.stdout:3511: rollback oid 45 current snap is 397 2024-09-11T01:52:44.192 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 45 to 309 2024-09-11T01:52:44.192 INFO:tasks.rados.rados.0.smithi071.stdout:3512: write oid 30 current snap is 397 2024-09-11T01:52:44.192 INFO:tasks.rados.rados.0.smithi071.stdout:3512: seq_num 1217 ranges {454119=594513,1515880=556760} 2024-09-11T01:52:44.194 INFO:tasks.rados.rados.0.smithi071.stdout:3512: writing smithi07134777-30 from 454119 to 1048632 tid 1 2024-09-11T01:52:44.200 INFO:tasks.rados.rados.0.smithi071.stdout:3512: writing smithi07134777-30 from 1515880 to 2072640 tid 2 2024-09-11T01:52:44.200 INFO:tasks.rados.rados.0.smithi071.stdout:3513: write oid 22 current snap is 397 2024-09-11T01:52:44.200 INFO:tasks.rados.rados.0.smithi071.stdout:3508: finishing rollback tid 2 to smithi07134777-43 2024-09-11T01:52:44.200 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 2465 (ObjNum 1014 snap 322 seq_num 1014) dirty exists 2024-09-11T01:52:44.200 INFO:tasks.rados.rados.0.smithi071.stdout:3513: seq_num 1218 ranges {470926=669762,1574362=700418} 2024-09-11T01:52:44.206 INFO:tasks.rados.rados.0.smithi071.stdout:3513: writing smithi07134777-22 from 470926 to 1140688 tid 1 2024-09-11T01:52:44.213 INFO:tasks.rados.rados.0.smithi071.stdout:3513: writing smithi07134777-22 from 1574362 to 2274780 tid 2 2024-09-11T01:52:44.214 INFO:tasks.rados.rados.0.smithi071.stdout:3508: done (5 left) 2024-09-11T01:52:44.214 INFO:tasks.rados.rados.0.smithi071.stdout:3514: write (excl) oid 2 current snap is 397 2024-09-11T01:52:44.214 INFO:tasks.rados.rados.0.smithi071.stdout:3509: finishing write tid 2 to smithi07134777-20 2024-09-11T01:52:44.214 INFO:tasks.rados.rados.0.smithi071.stdout:3509: finishing write tid 3 to smithi07134777-20 2024-09-11T01:52:44.214 INFO:tasks.rados.rados.0.smithi071.stdout:3509: finishing write tid 4 to smithi07134777-20 2024-09-11T01:52:44.214 INFO:tasks.rados.rados.0.smithi071.stdout:3509: oid 20 updating version 0 to 1192 2024-09-11T01:52:44.214 INFO:tasks.rados.rados.0.smithi071.stdout:3509: oid 20 updating version 1192 to 1193 2024-09-11T01:52:44.214 INFO:tasks.rados.rados.0.smithi071.stdout:3509: oid 20 version 1193 is already newer than 1191 2024-09-11T01:52:44.214 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 1193 (ObjNum 1216 snap 397 seq_num 1216) dirty exists 2024-09-11T01:52:44.214 INFO:tasks.rados.rados.0.smithi071.stdout:3509: left oid 20 (ObjNum 1216 snap 397 seq_num 1216) 2024-09-11T01:52:44.214 INFO:tasks.rados.rados.0.smithi071.stdout:3511: finishing rollback tid 1 to smithi07134777-45 2024-09-11T01:52:44.214 INFO:tasks.rados.rados.0.smithi071.stdout:3511: finishing rollback tid 2 to smithi07134777-45 2024-09-11T01:52:44.214 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 2153 (ObjNum 932 snap 293 seq_num 932) dirty exists 2024-09-11T01:52:44.215 INFO:tasks.rados.rados.0.smithi071.stdout:3512: finishing write tid 1 to smithi07134777-30 2024-09-11T01:52:44.215 INFO:tasks.rados.rados.0.smithi071.stdout:3514: seq_num 1219 ranges {487733=745011,1632844=560429} 2024-09-11T01:52:44.219 INFO:tasks.rados.rados.0.smithi071.stdout:3514: writing smithi07134777-2 from 487733 to 1232744 tid 1 2024-09-11T01:52:44.225 INFO:tasks.rados.rados.0.smithi071.stdout:3514: writing smithi07134777-2 from 1632844 to 2193273 tid 2 2024-09-11T01:52:44.226 INFO:tasks.rados.rados.0.smithi071.stdout:3509: done (5 left) 2024-09-11T01:52:44.226 INFO:tasks.rados.rados.0.smithi071.stdout:3511: done (4 left) 2024-09-11T01:52:44.226 INFO:tasks.rados.rados.0.smithi071.stdout:3515: read oid 18 snap 339 2024-09-11T01:52:44.226 INFO:tasks.rados.rados.0.smithi071.stdout:3515: expect (ObjNum 1010 snap 322 seq_num 1010) 2024-09-11T01:52:44.226 INFO:tasks.rados.rados.0.smithi071.stdout:3516: copy_from oid 24 from oid 8 current snap is 397 2024-09-11T01:52:44.226 INFO:tasks.rados.rados.0.smithi071.stdout:3512: finishing write tid 2 to smithi07134777-30 2024-09-11T01:52:44.226 INFO:tasks.rados.rados.0.smithi071.stdout:3512: finishing write tid 3 to smithi07134777-30 2024-09-11T01:52:44.226 INFO:tasks.rados.rados.0.smithi071.stdout:3512: finishing write tid 4 to smithi07134777-30 2024-09-11T01:52:44.226 INFO:tasks.rados.rados.0.smithi071.stdout:3512: oid 30 updating version 0 to 2208 2024-09-11T01:52:44.226 INFO:tasks.rados.rados.0.smithi071.stdout:3512: oid 30 updating version 2208 to 2209 2024-09-11T01:52:44.226 INFO:tasks.rados.rados.0.smithi071.stdout:3512: oid 30 version 2209 is already newer than 2207 2024-09-11T01:52:44.226 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 2209 (ObjNum 1217 snap 397 seq_num 1217) dirty exists 2024-09-11T01:52:44.226 INFO:tasks.rados.rados.0.smithi071.stdout:3512: left oid 30 (ObjNum 1217 snap 397 seq_num 1217) 2024-09-11T01:52:44.227 INFO:tasks.rados.rados.0.smithi071.stdout:3510: finishing copy_from racing read to smithi07134777-9 2024-09-11T01:52:44.227 INFO:tasks.rados.rados.0.smithi071.stdout:3510: finishing copy_from to smithi07134777-9 2024-09-11T01:52:44.227 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 1967 (ObjNum 832 snap 254 seq_num 832) dirty exists 2024-09-11T01:52:44.227 INFO:tasks.rados.rados.0.smithi071.stdout:3510: done (5 left) 2024-09-11T01:52:44.227 INFO:tasks.rados.rados.0.smithi071.stdout:3512: done (4 left) 2024-09-11T01:52:44.227 INFO:tasks.rados.rados.0.smithi071.stdout:3517: snap_remove snap 390 2024-09-11T01:52:45.005 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:52:45.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.116+0000 7fb49e440640 10 monclient: get_monmap_and_config 2024-09-11T01:52:45.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.116+0000 7fb49e440640 10 monclient: build_initial_monmap 2024-09-11T01:52:45.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.116+0000 7fb49e440640 10 monclient: monmap: 2024-09-11T01:52:45.119 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:45.119 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:45.119 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:45.118037+0000 2024-09-11T01:52:45.119 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:45.118037+0000 2024-09-11T01:52:45.119 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:45.119 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:45.119 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:45.119 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:45.119 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:45.119 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:45.120 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:45.120 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:45.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49e440640 1 Processor -- start 2024-09-11T01:52:45.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49e440640 1 -- start start 2024-09-11T01:52:45.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49e440640 10 monclient: init 2024-09-11T01:52:45.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49e440640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:45.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49e440640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-11T01:52:45.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49e440640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb49809d790 addr v1:172.21.15.71:6789/0 2024-09-11T01:52:45.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49e440640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb49809e280 0x7fb498139f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:45.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49e440640 10 monclient(hunting): picked mon.noname-c con 0x7fb49809e280 addr v2:172.21.15.71:3300/0 2024-09-11T01:52:45.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49e440640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb49809e6d0 0x7fb49813c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:45.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49e440640 10 monclient(hunting): picked mon.noname-a con 0x7fb49809e6d0 addr v2:172.21.15.12:3300/0 2024-09-11T01:52:45.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49e440640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4980a1da0 con 0x7fb49809d790 2024-09-11T01:52:45.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49e440640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:45.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49e440640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fb4980a0760 con 0x7fb49809e6d0 2024-09-11T01:52:45.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49e440640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:45.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49e440640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fb4980a1fa0 con 0x7fb49809e280 2024-09-11T01:52:45.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49e440640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:45.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49e440640 10 monclient(hunting): authenticate will time out at 2171.560681s 2024-09-11T01:52:45.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49c9dd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb49809e6d0 0x7fb49813c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:45.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49c9dd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb49809e6d0 0x7fb49813c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50878/0 (socket says 172.21.15.12:50878) 2024-09-11T01:52:45.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49c9dd640 1 -- 172.21.15.12:0/2148275889 learned_addr learned my addr 172.21.15.12:0/2148275889 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:45.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49c9dd640 10 monclient(hunting): get_auth_request con 0x7fb49809e6d0 auth_method 0 2024-09-11T01:52:45.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49c9dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:45.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49c9dd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:45.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49c9dd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:45.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49c9dd640 1 -- 172.21.15.12:0/2148275889 >> v2:172.21.15.12:3300/0 conn(0x7fb49809e6d0 msgr2=0x7fb49813c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:52:45.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49c9dd640 1 -- 172.21.15.12:0/2148275889 >> v2:172.21.15.12:3300/0 conn(0x7fb49809e6d0 msgr2=0x7fb49813c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:52:45.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb4977fe640 1 --2- 172.21.15.12:0/2148275889 >> v2:172.21.15.71:3300/0 conn(0x7fb49809e280 0x7fb498139f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:45.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49c9dd640 1 --2- 172.21.15.12:0/2148275889 >> v2:172.21.15.12:3300/0 conn(0x7fb49809e6d0 0x7fb49813c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:52:45.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.117+0000 7fb49c9dd640 1 --2- 172.21.15.12:0/2148275889 >> v2:172.21.15.12:3300/0 conn(0x7fb49809e6d0 0x7fb49813c470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.118+0000 7fb4977fe640 10 monclient(hunting): get_auth_request con 0x7fb49809e280 auth_method 0 2024-09-11T01:52:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.118+0000 7fb4977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.118+0000 7fb4977fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.118+0000 7fb4977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.118+0000 7fb4977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.118+0000 7fb4977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.118+0000 7fb4977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.118+0000 7fb4977fe640 10 monclient(hunting): handle_auth_done global_id 234322 payload 274 2024-09-11T01:52:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.118+0000 7fb4977fe640 1 -- 172.21.15.12:0/2148275889 >> v2:172.21.15.12:3300/0 conn(0x7fb49809e6d0 msgr2=0x7fb49813c470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:52:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.118+0000 7fb4977fe640 1 --2- 172.21.15.12:0/2148275889 >> v2:172.21.15.12:3300/0 conn(0x7fb49809e6d0 0x7fb49813c470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.118+0000 7fb4977fe640 1 -- 172.21.15.12:0/2148275889 >> v1:172.21.15.71:6789/0 conn(0x7fb49809d790 legacy=0x7fb49809db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.118+0000 7fb4977fe640 10 monclient: _finish_hunting 0 2024-09-11T01:52:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.118+0000 7fb4977fe640 1 monclient: found mon.noname-c 2024-09-11T01:52:45.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.118+0000 7fb4977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:45.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.118+0000 7fb4977fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:52:45.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.118+0000 7fb4977fe640 1 -- 172.21.15.12:0/2148275889 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb49809eda0 con 0x7fb49809e280 2024-09-11T01:52:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb4977fe640 1 -- 172.21.15.12:0/2148275889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb49809e280 msgr2=0x7fb498139f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb4977fe640 1 --2- 172.21.15.12:0/2148275889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb49809e280 0x7fb498139f20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb49809eca0 tx=0x7fb48c02a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73513715e4abece4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb496ffd640 1 -- 172.21.15.12:0/2148275889 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb48c037070 con 0x7fb49809e280 2024-09-11T01:52:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb496ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb496ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb496ffd640 10 monclient: dump: 2024-09-11T01:52:45.131 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:45.132 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb496ffd640 10 monclient: _finish_auth 0 2024-09-11T01:52:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb496ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:52:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb496ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:45.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb49e440640 5 monclient: authenticate success, global_id 234322 2024-09-11T01:52:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb49e440640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:52:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb496ffd640 1 -- 172.21.15.12:0/2148275889 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb48c032440 con 0x7fb49809e280 2024-09-11T01:52:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb496ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb496ffd640 1 -- 172.21.15.12:0/2148275889 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb48c032810 con 0x7fb49809e280 2024-09-11T01:52:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb496ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb496ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:52:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb496ffd640 10 monclient: dump: 2024-09-11T01:52:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:45.133 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:45.134 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb49e440640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb49e440640 4 set_mon_vals no callback set 2024-09-11T01:52:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb49e440640 10 monclient: shutdown 2024-09-11T01:52:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb49e440640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb49e440640 1 -- 172.21.15.12:0/2148275889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb49809e280 msgr2=0x7fb498139f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb49e440640 1 --2- 172.21.15.12:0/2148275889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb49809e280 0x7fb498139f20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb49809eca0 tx=0x7fb48c02a000 comp rx=0 tx=0).stop 2024-09-11T01:52:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb49e440640 1 -- 172.21.15.12:0/2148275889 shutdown_connections 2024-09-11T01:52:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb49e440640 1 --2- 172.21.15.12:0/2148275889 >> v2:172.21.15.12:3300/0 conn(0x7fb49809e6d0 0x7fb49813c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb49e440640 1 --2- 172.21.15.12:0/2148275889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb49809e280 0x7fb498139f20 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:45.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb49e440640 1 -- 172.21.15.12:0/2148275889 >> 172.21.15.12:0/2148275889 conn(0x7fb4980a2d90 msgr2=0x7fb4980a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:45.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb49e440640 1 -- 172.21.15.12:0/2148275889 shutdown_connections 2024-09-11T01:52:45.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.122+0000 7fb49e440640 1 -- 172.21.15.12:0/2148275889 wait complete. 2024-09-11T01:52:45.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.123+0000 7fb49e440640 10 monclient: build_initial_monmap 2024-09-11T01:52:45.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.123+0000 7fb49e440640 10 monclient: monmap: 2024-09-11T01:52:45.135 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:45.135 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:45.135 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:45.135 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:45.135 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:45.135 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:45.135 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:45.135 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:45.135 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:45.135 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:45.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.123+0000 7fb49e440640 1 Processor -- start 2024-09-11T01:52:45.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.123+0000 7fb49e440640 1 -- start start 2024-09-11T01:52:45.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.123+0000 7fb49e440640 10 monclient: init 2024-09-11T01:52:45.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb49e440640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:45.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb49e440640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:52:45.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb49e440640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb49809d790 0x7fb4981c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:45.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb49e440640 10 monclient(hunting): picked mon.noname-a con 0x7fb49809d790 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:45.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb49e440640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb49809e280 0x7fb4981330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:45.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb49e440640 10 monclient(hunting): picked mon.noname-c con 0x7fb49809e280 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:45.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb49e440640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb49809e6d0 0x7fb4981335f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:45.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb49e440640 10 monclient(hunting): picked mon.noname-b con 0x7fb49809e6d0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:45.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb49e440640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:45.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb49e440640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb49809ce20 con 0x7fb49809d790 2024-09-11T01:52:45.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb49e440640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:45.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb49e440640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb49809cca0 con 0x7fb49809e6d0 2024-09-11T01:52:45.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb49e440640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:45.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb49e440640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb49813c9b0 con 0x7fb49809e280 2024-09-11T01:52:45.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb49e440640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:45.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb49e440640 10 monclient(hunting): authenticate will time out at 2171.567509s 2024-09-11T01:52:45.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb497fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb49809d790 0x7fb4981c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:45.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb4977fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb49809e280 0x7fb4981330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:45.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb497fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb49809d790 0x7fb4981c99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50894/0 (socket says 172.21.15.12:50894) 2024-09-11T01:52:45.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb497fff640 1 -- 172.21.15.12:0/10416316 learned_addr learned my addr 172.21.15.12:0/10416316 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:45.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb497fff640 10 monclient(hunting): get_auth_request con 0x7fb49809d790 auth_method 0 2024-09-11T01:52:45.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb497fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:45.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb497fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:45.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb497fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:45.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb4977fe640 10 monclient(hunting): get_auth_request con 0x7fb49809e280 auth_method 0 2024-09-11T01:52:45.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb4977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:45.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb4977fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:45.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb4977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:45.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb497fff640 1 -- 172.21.15.12:0/10416316 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb49809d790 msgr2=0x7fb4981c99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:52:45.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb497fff640 1 -- 172.21.15.12:0/10416316 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb49809d790 msgr2=0x7fb4981c99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:52:45.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb497fff640 1 --2- 172.21.15.12:0/10416316 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb49809d790 0x7fb4981c99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:52:45.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb4977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:45.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb4977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:45.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb49c9dd640 1 --2- 172.21.15.12:0/10416316 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb49809e6d0 0x7fb4981335f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:45.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb497fff640 1 --2- 172.21.15.12:0/10416316 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb49809d790 0x7fb4981c99b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:45.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb4977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:45.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb49c9dd640 10 monclient(hunting): get_auth_request con 0x7fb49809e6d0 auth_method 0 2024-09-11T01:52:45.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb49c9dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:45.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb49c9dd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:45.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb49c9dd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:45.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb49c9dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:45.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb49c9dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:45.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.124+0000 7fb49c9dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:45.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.125+0000 7fb4977fe640 10 monclient(hunting): handle_auth_done global_id 244304 payload 824 2024-09-11T01:52:45.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.125+0000 7fb4977fe640 1 -- 172.21.15.12:0/10416316 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb49809e6d0 msgr2=0x7fb4981335f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:45.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.125+0000 7fb4977fe640 1 --2- 172.21.15.12:0/10416316 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb49809e6d0 0x7fb4981335f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:45.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.125+0000 7fb4977fe640 1 -- 172.21.15.12:0/10416316 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb49809d790 msgr2=0x7fb4981c99b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:52:45.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.125+0000 7fb4977fe640 1 --2- 172.21.15.12:0/10416316 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb49809d790 0x7fb4981c99b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:45.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.125+0000 7fb4977fe640 10 monclient: _finish_hunting 0 2024-09-11T01:52:45.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.125+0000 7fb4977fe640 1 monclient: found mon.noname-c 2024-09-11T01:52:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.125+0000 7fb4977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.125+0000 7fb4977fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:52:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.125+0000 7fb4977fe640 1 -- 172.21.15.12:0/10416316 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4980a5540 con 0x7fb49809e280 2024-09-11T01:52:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.127+0000 7fb4977fe640 1 -- 172.21.15.12:0/10416316 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb49809e280 msgr2=0x7fb4981330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.127+0000 7fb4977fe640 1 --2- 172.21.15.12:0/10416316 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb49809e280 0x7fb4981330b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb4980a2120 tx=0x7fb48c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.127+0000 7fb494ff9640 1 -- 172.21.15.12:0/10416316 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb48c041020 con 0x7fb49809e280 2024-09-11T01:52:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.127+0000 7fb494ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.127+0000 7fb494ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.127+0000 7fb494ff9640 10 monclient: dump: 2024-09-11T01:52:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:45.140 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:45.141 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.127+0000 7fb494ff9640 10 monclient: _finish_auth 0 2024-09-11T01:52:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.127+0000 7fb494ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:52:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.127+0000 7fb494ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.127+0000 7fb494ff9640 1 -- 172.21.15.12:0/10416316 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb48c03caa0 con 0x7fb49809e280 2024-09-11T01:52:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.127+0000 7fb49e440640 5 monclient: authenticate success, global_id 244304 2024-09-11T01:52:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.127+0000 7fb49e440640 10 monclient: _renew_subs 2024-09-11T01:52:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.127+0000 7fb49e440640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:45.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.127+0000 7fb49e440640 1 -- 172.21.15.12:0/10416316 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb49813c5a0 con 0x7fb49809e280 2024-09-11T01:52:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.127+0000 7fb494ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.127+0000 7fb494ff9640 1 -- 172.21.15.12:0/10416316 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb48c037040 con 0x7fb49809e280 2024-09-11T01:52:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.127+0000 7fb494ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.127+0000 7fb494ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.127+0000 7fb494ff9640 10 monclient: dump: 2024-09-11T01:52:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:45.142 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:45.143 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.127+0000 7fb49e440640 10 monclient: _renew_subs 2024-09-11T01:52:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.127+0000 7fb4967fc640 4 set_mon_vals no callback set 2024-09-11T01:52:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.127+0000 7fb49e440640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.127+0000 7fb49e440640 1 -- 172.21.15.12:0/10416316 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb498071270 con 0x7fb49809e280 2024-09-11T01:52:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.128+0000 7fb494ff9640 1 -- 172.21.15.12:0/10416316 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fb48c03d410 con 0x7fb49809e280 2024-09-11T01:52:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.128+0000 7fb49e440640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.128+0000 7fb49e440640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.128+0000 7fb49e440640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.128+0000 7fb49e440640 1 -- 172.21.15.12:0/10416316 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb498074aa0 con 0x7fb49809e280 2024-09-11T01:52:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.128+0000 7fb494ff9640 1 --2- 172.21.15.12:0/10416316 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb48403e210 0x7fb4840406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.128+0000 7fb494ff9640 1 -- 172.21.15.12:0/10416316 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(794..794 src has 676..794) ==== 4799+0+0 (secure 0 0 0) 0x7fb48c073720 con 0x7fb49809e280 2024-09-11T01:52:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.129+0000 7fb497fff640 1 --2- 172.21.15.12:0/10416316 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb48403e210 0x7fb4840406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:45.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.129+0000 7fb497fff640 10 monclient: get_auth_request con 0x7fb48403e210 auth_method 0 2024-09-11T01:52:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.131+0000 7fb497fff640 1 --2- 172.21.15.12:0/10416316 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb48403e210 0x7fb4840406d0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fb488007d10 tx=0x7fb488009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.133+0000 7fb494ff9640 1 -- 172.21.15.12:0/10416316 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb498074aa0 con 0x7fb49809e280 2024-09-11T01:52:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.133+0000 7fb494ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:45.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.133+0000 7fb494ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:45.152 INFO:tasks.rados.rados.0.smithi071.stdout:3513: finishing write tid 1 to smithi07134777-22 2024-09-11T01:52:45.152 INFO:tasks.rados.rados.0.smithi071.stdout:3513: finishing write tid 2 to smithi07134777-22 2024-09-11T01:52:45.152 INFO:tasks.rados.rados.0.smithi071.stdout:3513: finishing write tid 3 to smithi07134777-22 2024-09-11T01:52:45.152 INFO:tasks.rados.rados.0.smithi071.stdout:3513: finishing write tid 4 to smithi07134777-22 2024-09-11T01:52:45.152 INFO:tasks.rados.rados.0.smithi071.stdout:3513: oid 22 updating version 0 to 2427 2024-09-11T01:52:45.152 INFO:tasks.rados.rados.0.smithi071.stdout:3513: oid 22 updating version 2427 to 2428 2024-09-11T01:52:45.152 INFO:tasks.rados.rados.0.smithi071.stdout:3513: oid 22 version 2428 is already newer than 2426 2024-09-11T01:52:45.153 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 2428 (ObjNum 1218 snap 397 seq_num 1218) dirty exists 2024-09-11T01:52:45.153 INFO:tasks.rados.rados.0.smithi071.stdout:3513: left oid 22 (ObjNum 1218 snap 397 seq_num 1218) 2024-09-11T01:52:45.153 INFO:tasks.rados.rados.0.smithi071.stdout:3514: finishing write tid 1 to smithi07134777-2 2024-09-11T01:52:45.153 INFO:tasks.rados.rados.0.smithi071.stdout:3514: finishing write tid 2 to smithi07134777-2 2024-09-11T01:52:45.153 INFO:tasks.rados.rados.0.smithi071.stdout:3514: finishing write tid 3 to smithi07134777-2 2024-09-11T01:52:45.153 INFO:tasks.rados.rados.0.smithi071.stdout:3514: finishing write tid 4 to smithi07134777-2 2024-09-11T01:52:45.153 INFO:tasks.rados.rados.0.smithi071.stdout:3514: oid 2 updating version 0 to 1791 2024-09-11T01:52:45.153 INFO:tasks.rados.rados.0.smithi071.stdout:3514: oid 2 version 1791 is already newer than 1790 2024-09-11T01:52:45.153 INFO:tasks.rados.rados.0.smithi071.stdout:3514: oid 2 updating version 1791 to 1792 2024-09-11T01:52:45.153 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 1792 (ObjNum 1219 snap 397 seq_num 1219) dirty exists 2024-09-11T01:52:45.153 INFO:tasks.rados.rados.0.smithi071.stdout:3514: left oid 2 (ObjNum 1219 snap 397 seq_num 1219) 2024-09-11T01:52:45.153 INFO:tasks.rados.rados.0.smithi071.stdout:3515: expect (ObjNum 1010 snap 322 seq_num 1010) 2024-09-11T01:52:45.279 INFO:tasks.rados.rados.0.smithi071.stdout:3516: finishing copy_from racing read to smithi07134777-24 2024-09-11T01:52:45.279 INFO:tasks.rados.rados.0.smithi071.stdout:3516: finishing copy_from to smithi07134777-24 2024-09-11T01:52:45.279 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 2093 (ObjNum 1187 snap 378 seq_num 1187) dirty exists 2024-09-11T01:52:45.279 INFO:tasks.rados.rados.0.smithi071.stdout:3513: done (4 left) 2024-09-11T01:52:45.279 INFO:tasks.rados.rados.0.smithi071.stdout:3514: done (3 left) 2024-09-11T01:52:45.280 INFO:tasks.rados.rados.0.smithi071.stdout:3515: done (2 left) 2024-09-11T01:52:45.280 INFO:tasks.rados.rados.0.smithi071.stdout:3516: done (1 left) 2024-09-11T01:52:45.280 INFO:tasks.rados.rados.0.smithi071.stdout:3517: done (0 left) 2024-09-11T01:52:45.280 INFO:tasks.rados.rados.0.smithi071.stdout:3518: write oid 2 current snap is 397 2024-09-11T01:52:45.280 INFO:tasks.rados.rados.0.smithi071.stdout:3518: seq_num 1220 ranges {504540=420260,1574973=704087} 2024-09-11T01:52:45.283 INFO:tasks.rados.rados.0.smithi071.stdout:3518: writing smithi07134777-2 from 504540 to 924800 tid 1 2024-09-11T01:52:45.291 INFO:tasks.rados.rados.0.smithi071.stdout:3518: writing smithi07134777-2 from 1574973 to 2279060 tid 2 2024-09-11T01:52:45.291 INFO:tasks.rados.rados.0.smithi071.stdout:3519: snap_remove snap 309 2024-09-11T01:52:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.397+0000 7fb49e440640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:52:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.397+0000 7fb49e440640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:52:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.397+0000 7fb49e440640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.397+0000 7fb49e440640 1 -- 172.21.15.12:0/10416316 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb49809db70 con 0x7fb49809e280 2024-09-11T01:52:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.397+0000 7fb494ff9640 1 -- 172.21.15.12:0/10416316 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7fb48c04b030 con 0x7fb49809e280 2024-09-11T01:52:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.397+0000 7fb494ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:52:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.397+0000 7fb494ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:45.401 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":216,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":27,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:52:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.399+0000 7fb49e440640 1 -- 172.21.15.12:0/10416316 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb48403e210 msgr2=0x7fb4840406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.399+0000 7fb49e440640 1 --2- 172.21.15.12:0/10416316 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb48403e210 0x7fb4840406d0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fb488007d10 tx=0x7fb488009040 comp rx=0 tx=0).stop 2024-09-11T01:52:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.399+0000 7fb49e440640 10 monclient: shutdown 2024-09-11T01:52:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.399+0000 7fb49e440640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.399+0000 7fb49e440640 1 -- 172.21.15.12:0/10416316 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb49809e280 msgr2=0x7fb4981330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.399+0000 7fb49e440640 1 --2- 172.21.15.12:0/10416316 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb49809e280 0x7fb4981330b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb4980a2120 tx=0x7fb48c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:52:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.399+0000 7fb49e440640 1 -- 172.21.15.12:0/10416316 shutdown_connections 2024-09-11T01:52:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.399+0000 7fb49e440640 1 --2- 172.21.15.12:0/10416316 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb49809e6d0 0x7fb4981335f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.399+0000 7fb49e440640 1 --2- 172.21.15.12:0/10416316 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb49809e280 0x7fb4981330b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.399+0000 7fb49e440640 1 --2- 172.21.15.12:0/10416316 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb48403e210 0x7fb4840406d0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.399+0000 7fb49e440640 1 --2- 172.21.15.12:0/10416316 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb49809d790 0x7fb4981c99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.399+0000 7fb49e440640 1 -- 172.21.15.12:0/10416316 >> 172.21.15.12:0/10416316 conn(0x7fb4980a2d90 msgr2=0x7fb4980a5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.400+0000 7fb49e440640 1 -- 172.21.15.12:0/10416316 shutdown_connections 2024-09-11T01:52:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:45.400+0000 7fb49e440640 1 -- 172.21.15.12:0/10416316 wait complete. 2024-09-11T01:52:46.155 INFO:tasks.rados.rados.0.smithi071.stdout:3519: done (1 left) 2024-09-11T01:52:46.155 INFO:tasks.rados.rados.0.smithi071.stdout:3520: read oid 49 snap -1 2024-09-11T01:52:46.155 INFO:tasks.rados.rados.0.smithi071.stdout:3520: expect (ObjNum 1162 snap 370 seq_num 1162) 2024-09-11T01:52:46.156 INFO:tasks.rados.rados.0.smithi071.stdout:3518: finishing write tid 1 to smithi07134777-2 2024-09-11T01:52:46.156 INFO:tasks.rados.rados.0.smithi071.stdout:3518: finishing write tid 2 to smithi07134777-2 2024-09-11T01:52:46.156 INFO:tasks.rados.rados.0.smithi071.stdout:3518: finishing write tid 3 to smithi07134777-2 2024-09-11T01:52:46.156 INFO:tasks.rados.rados.0.smithi071.stdout:3518: finishing write tid 4 to smithi07134777-2 2024-09-11T01:52:46.156 INFO:tasks.rados.rados.0.smithi071.stdout:3518: oid 2 updating version 0 to 1795 2024-09-11T01:52:46.156 INFO:tasks.rados.rados.0.smithi071.stdout:3518: oid 2 version 1795 is already newer than 1794 2024-09-11T01:52:46.156 INFO:tasks.rados.rados.0.smithi071.stdout:3518: oid 2 version 1795 is already newer than 1793 2024-09-11T01:52:46.156 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 1795 (ObjNum 1220 snap 397 seq_num 1220) dirty exists 2024-09-11T01:52:46.156 INFO:tasks.rados.rados.0.smithi071.stdout:3518: left oid 2 (ObjNum 1220 snap 397 seq_num 1220) 2024-09-11T01:52:46.156 INFO:tasks.rados.rados.0.smithi071.stdout:3518: done (1 left) 2024-09-11T01:52:46.156 INFO:tasks.rados.rados.0.smithi071.stdout:3521: rollback oid 31 current snap is 397 2024-09-11T01:52:46.156 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 31 to 382 2024-09-11T01:52:46.156 INFO:tasks.rados.rados.0.smithi071.stdout:3522: write (excl) oid 41 current snap is 397 2024-09-11T01:52:46.157 INFO:tasks.rados.rados.0.smithi071.stdout:3522: seq_num 1221 ranges {521347=495509,1633455=564098} 2024-09-11T01:52:46.161 INFO:tasks.rados.rados.0.smithi071.stdout:3522: writing smithi07134777-41 from 521347 to 1016856 tid 1 2024-09-11T01:52:46.168 INFO:tasks.rados.rados.0.smithi071.stdout:3522: writing smithi07134777-41 from 1633455 to 2197553 tid 2 2024-09-11T01:52:46.168 INFO:tasks.rados.rados.0.smithi071.stdout:3523: rollback oid 27 current snap is 397 2024-09-11T01:52:46.169 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 27 to 389 2024-09-11T01:52:46.169 INFO:tasks.rados.rados.0.smithi071.stdout:3521: finishing rollback tid 0 to smithi07134777-31 2024-09-11T01:52:46.169 INFO:tasks.rados.rados.0.smithi071.stdout:3521: finishing rollback tid 1 to smithi07134777-31 2024-09-11T01:52:46.169 INFO:tasks.rados.rados.0.smithi071.stdout:3524: read oid 39 snap -1 2024-09-11T01:52:46.169 INFO:tasks.rados.rados.0.smithi071.stdout:3524: expect (ObjNum 1198 snap 384 seq_num 1198) 2024-09-11T01:52:46.169 INFO:tasks.rados.rados.0.smithi071.stdout:3525: copy_from oid 46 from oid 50 current snap is 397 2024-09-11T01:52:46.169 INFO:tasks.rados.rados.0.smithi071.stdout:3526: write oid 40 current snap is 397 2024-09-11T01:52:46.169 INFO:tasks.rados.rados.0.smithi071.stdout:3526: seq_num 1222 ranges {538154=570758,1691937=707756} 2024-09-11T01:52:46.176 INFO:tasks.rados.rados.0.smithi071.stdout:3526: writing smithi07134777-40 from 538154 to 1108912 tid 1 2024-09-11T01:52:46.184 INFO:tasks.rados.rados.0.smithi071.stdout:3526: writing smithi07134777-40 from 1691937 to 2399693 tid 2 2024-09-11T01:52:46.184 INFO:tasks.rados.rados.0.smithi071.stdout:3527: read oid 24 snap -1 2024-09-11T01:52:46.184 INFO:tasks.rados.rados.0.smithi071.stdout:3527: expect (ObjNum 1187 snap 378 seq_num 1187) 2024-09-11T01:52:46.185 INFO:tasks.rados.rados.0.smithi071.stdout:3528: delete oid 37 current snap is 397 2024-09-11T01:52:46.186 INFO:tasks.rados.rados.0.smithi071.stdout:3521: finishing rollback tid 2 to smithi07134777-31 2024-09-11T01:52:46.186 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 2564 (ObjNum 988 snap 317 seq_num 988) dirty exists 2024-09-11T01:52:46.186 INFO:tasks.rados.rados.0.smithi071.stdout:3522: finishing write tid 1 to smithi07134777-41 2024-09-11T01:52:46.187 INFO:tasks.rados.rados.0.smithi071.stdout:3522: finishing write tid 2 to smithi07134777-41 2024-09-11T01:52:46.187 INFO:tasks.rados.rados.0.smithi071.stdout:3522: finishing write tid 3 to smithi07134777-41 2024-09-11T01:52:46.187 INFO:tasks.rados.rados.0.smithi071.stdout:3522: finishing write tid 4 to smithi07134777-41 2024-09-11T01:52:46.187 INFO:tasks.rados.rados.0.smithi071.stdout:3522: oid 41 updating version 0 to 1971 2024-09-11T01:52:46.187 INFO:tasks.rados.rados.0.smithi071.stdout:3522: oid 41 updating version 1971 to 1973 2024-09-11T01:52:46.187 INFO:tasks.rados.rados.0.smithi071.stdout:3522: oid 41 version 1973 is already newer than 1972 2024-09-11T01:52:46.187 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 1973 (ObjNum 1221 snap 397 seq_num 1221) dirty exists 2024-09-11T01:52:46.187 INFO:tasks.rados.rados.0.smithi071.stdout:3522: left oid 41 (ObjNum 1221 snap 397 seq_num 1221) 2024-09-11T01:52:46.187 INFO:tasks.rados.rados.0.smithi071.stdout:3524: expect (ObjNum 1198 snap 384 seq_num 1198) 2024-09-11T01:52:46.189 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1103949 2024-09-11T01:52:46.226 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1988238 2024-09-11T01:52:46.227 INFO:tasks.rados.rados.0.smithi071.stdout:3520: expect (ObjNum 1162 snap 370 seq_num 1162) 2024-09-11T01:52:46.239 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1205624 2024-09-11T01:52:46.367 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:52:46.441 INFO:tasks.rados.rados.0.smithi071.stdout:3525: finishing copy_from racing read to smithi07134777-46 2024-09-11T01:52:46.441 INFO:tasks.rados.rados.0.smithi071.stdout:3526: finishing write tid 1 to smithi07134777-40 2024-09-11T01:52:46.441 INFO:tasks.rados.rados.0.smithi071.stdout:3523: finishing rollback tid 0 to smithi07134777-27 2024-09-11T01:52:46.441 INFO:tasks.rados.rados.0.smithi071.stdout:3523: finishing rollback tid 1 to smithi07134777-27 2024-09-11T01:52:46.441 INFO:tasks.rados.rados.0.smithi071.stdout:3523: finishing rollback tid 2 to smithi07134777-27 2024-09-11T01:52:46.441 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 2477 (ObjNum 1154 snap 366 seq_num 1154) dirty exists 2024-09-11T01:52:46.441 INFO:tasks.rados.rados.0.smithi071.stdout:3520: done (8 left) 2024-09-11T01:52:46.441 INFO:tasks.rados.rados.0.smithi071.stdout:3521: done (7 left) 2024-09-11T01:52:46.442 INFO:tasks.rados.rados.0.smithi071.stdout:3522: done (6 left) 2024-09-11T01:52:46.442 INFO:tasks.rados.rados.0.smithi071.stdout:3523: done (5 left) 2024-09-11T01:52:46.442 INFO:tasks.rados.rados.0.smithi071.stdout:3524: done (4 left) 2024-09-11T01:52:46.442 INFO:tasks.rados.rados.0.smithi071.stdout:3528: done (3 left) 2024-09-11T01:52:46.442 INFO:tasks.rados.rados.0.smithi071.stdout:3529: copy_from oid 16 from oid 30 current snap is 397 2024-09-11T01:52:46.442 INFO:tasks.rados.rados.0.smithi071.stdout:3525: finishing copy_from to smithi07134777-46 2024-09-11T01:52:46.442 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 2568 (ObjNum 1191 snap 379 seq_num 1191) dirty exists 2024-09-11T01:52:46.442 INFO:tasks.rados.rados.0.smithi071.stdout:3526: finishing write tid 2 to smithi07134777-40 2024-09-11T01:52:46.442 INFO:tasks.rados.rados.0.smithi071.stdout:3526: finishing write tid 3 to smithi07134777-40 2024-09-11T01:52:46.442 INFO:tasks.rados.rados.0.smithi071.stdout:3526: finishing write tid 4 to smithi07134777-40 2024-09-11T01:52:46.442 INFO:tasks.rados.rados.0.smithi071.stdout:3526: oid 40 updating version 0 to 1798 2024-09-11T01:52:46.442 INFO:tasks.rados.rados.0.smithi071.stdout:3526: oid 40 updating version 1798 to 2570 2024-09-11T01:52:46.442 INFO:tasks.rados.rados.0.smithi071.stdout:3526: oid 40 version 2570 is already newer than 2569 2024-09-11T01:52:46.442 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 2570 (ObjNum 1222 snap 397 seq_num 1222) dirty exists 2024-09-11T01:52:46.443 INFO:tasks.rados.rados.0.smithi071.stdout:3526: left oid 40 (ObjNum 1222 snap 397 seq_num 1222) 2024-09-11T01:52:46.443 INFO:tasks.rados.rados.0.smithi071.stdout:3527: expect (ObjNum 1187 snap 378 seq_num 1187) 2024-09-11T01:52:46.443 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:52:46.558 INFO:tasks.rados.rados.0.smithi071.stdout:3529: finishing copy_from racing read to smithi07134777-16 2024-09-11T01:52:46.558 INFO:tasks.rados.rados.0.smithi071.stdout:3529: finishing copy_from to smithi07134777-16 2024-09-11T01:52:46.558 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 2209 (ObjNum 1217 snap 397 seq_num 1217) dirty exists 2024-09-11T01:52:46.558 INFO:tasks.rados.rados.0.smithi071.stdout:3525: done (3 left) 2024-09-11T01:52:46.558 INFO:tasks.rados.rados.0.smithi071.stdout:3526: done (2 left) 2024-09-11T01:52:46.558 INFO:tasks.rados.rados.0.smithi071.stdout:3527: done (1 left) 2024-09-11T01:52:46.558 INFO:tasks.rados.rados.0.smithi071.stdout:3529: done (0 left) 2024-09-11T01:52:46.558 INFO:tasks.rados.rados.0.smithi071.stdout:3530: rollback oid 23 current snap is 397 2024-09-11T01:52:46.559 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 23 to 385 2024-09-11T01:52:46.559 INFO:tasks.rados.rados.0.smithi071.stdout:3531: delete oid 43 current snap is 397 2024-09-11T01:52:46.561 INFO:tasks.rados.rados.0.smithi071.stdout:3531: done (1 left) 2024-09-11T01:52:46.561 INFO:tasks.rados.rados.0.smithi071.stdout:3532: write oid 6 current snap is 397 2024-09-11T01:52:46.561 INFO:tasks.rados.rados.0.smithi071.stdout:3532: seq_num 1225 ranges {588575=512858,1867383=455083} 2024-09-11T01:52:46.567 INFO:tasks.rados.rados.0.smithi071.stdout:3532: writing smithi07134777-6 from 588575 to 1101433 tid 1 2024-09-11T01:52:46.573 INFO:tasks.rados.rados.0.smithi071.stdout:3532: writing smithi07134777-6 from 1867383 to 2322466 tid 2 2024-09-11T01:52:46.574 INFO:tasks.rados.rados.0.smithi071.stdout:3533: snap_remove snap 359 2024-09-11T01:52:47.164 INFO:tasks.rados.rados.0.smithi071.stdout:3533: done (2 left) 2024-09-11T01:52:47.165 INFO:tasks.rados.rados.0.smithi071.stdout:3534: read oid 29 snap -1 2024-09-11T01:52:47.165 INFO:tasks.rados.rados.0.smithi071.stdout:3534: expect deleted 2024-09-11T01:52:47.165 INFO:tasks.rados.rados.0.smithi071.stdout:3535: snap_create 2024-09-11T01:52:47.165 INFO:tasks.rados.rados.0.smithi071.stdout:3530: finishing rollback tid 0 to smithi07134777-23 2024-09-11T01:52:47.165 INFO:tasks.rados.rados.0.smithi071.stdout:3530: finishing rollback tid 1 to smithi07134777-23 2024-09-11T01:52:47.165 INFO:tasks.rados.rados.0.smithi071.stdout:3530: finishing rollback tid 2 to smithi07134777-23 2024-09-11T01:52:47.165 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 1203 (ObjNum 1160 snap 370 seq_num 1160) dirty exists 2024-09-11T01:52:47.165 INFO:tasks.rados.rados.0.smithi071.stdout:3532: finishing write tid 1 to smithi07134777-6 2024-09-11T01:52:47.165 INFO:tasks.rados.rados.0.smithi071.stdout:3532: finishing write tid 2 to smithi07134777-6 2024-09-11T01:52:47.165 INFO:tasks.rados.rados.0.smithi071.stdout:3532: finishing write tid 3 to smithi07134777-6 2024-09-11T01:52:47.165 INFO:tasks.rados.rados.0.smithi071.stdout:3532: finishing write tid 4 to smithi07134777-6 2024-09-11T01:52:47.166 INFO:tasks.rados.rados.0.smithi071.stdout:3532: oid 6 updating version 0 to 2490 2024-09-11T01:52:47.166 INFO:tasks.rados.rados.0.smithi071.stdout:3532: oid 6 updating version 2490 to 2491 2024-09-11T01:52:47.166 INFO:tasks.rados.rados.0.smithi071.stdout:3532: oid 6 updating version 2491 to 2492 2024-09-11T01:52:47.166 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 2492 (ObjNum 1225 snap 397 seq_num 1225) dirty exists 2024-09-11T01:52:47.166 INFO:tasks.rados.rados.0.smithi071.stdout:3532: left oid 6 (ObjNum 1225 snap 397 seq_num 1225) 2024-09-11T01:52:48.144 INFO:tasks.rados.rados.0.smithi071.stdout:3530: done (3 left) 2024-09-11T01:52:48.144 INFO:tasks.rados.rados.0.smithi071.stdout:3532: done (2 left) 2024-09-11T01:52:48.145 INFO:tasks.rados.rados.0.smithi071.stdout:3534: done (1 left) 2024-09-11T01:52:48.145 INFO:tasks.rados.rados.0.smithi071.stdout:3535: done (0 left) 2024-09-11T01:52:48.145 INFO:tasks.rados.rados.0.smithi071.stdout:3536: snap_create 2024-09-11T01:52:48.413 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:52:48.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.524+0000 7fe84d060640 10 monclient: get_monmap_and_config 2024-09-11T01:52:48.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.524+0000 7fe84d060640 10 monclient: build_initial_monmap 2024-09-11T01:52:48.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.524+0000 7fe84d060640 10 monclient: monmap: 2024-09-11T01:52:48.526 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:48.526 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:48.527 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:48.525513+0000 2024-09-11T01:52:48.527 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:48.525513+0000 2024-09-11T01:52:48.527 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:48.527 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:48.527 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:48.527 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:48.527 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:48.527 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:48.527 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:48.527 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:48.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.524+0000 7fe84d060640 1 Processor -- start 2024-09-11T01:52:48.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.524+0000 7fe84d060640 1 -- start start 2024-09-11T01:52:48.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.524+0000 7fe84d060640 10 monclient: init 2024-09-11T01:52:48.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.524+0000 7fe84d060640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:48.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.524+0000 7fe84d060640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-11T01:52:48.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.524+0000 7fe84d060640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe8480a1380 0x7fe8480a1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:48.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.524+0000 7fe84d060640 10 monclient(hunting): picked mon.noname-b con 0x7fe8480a1380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:48.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe84d060640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe8480a0c90 addr v1:172.21.15.12:6789/0 2024-09-11T01:52:48.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe84d060640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe8481361f0 0x7fe84813de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:48.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe84d060640 10 monclient(hunting): picked mon.noname-a con 0x7fe8481361f0 addr v2:172.21.15.12:3300/0 2024-09-11T01:52:48.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe84d060640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8480a2b40 con 0x7fe8480a0c90 2024-09-11T01:52:48.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe84d060640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:48.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe84d060640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fe84809f9a0 con 0x7fe8481361f0 2024-09-11T01:52:48.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe84d060640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:48.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe84d060640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe8480a2230 con 0x7fe8480a1380 2024-09-11T01:52:48.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe84d060640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:48.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe84d060640 10 monclient(hunting): authenticate will time out at 2174.968084s 2024-09-11T01:52:48.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe846d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe8480a1380 0x7fe8480a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:48.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe84759e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe8481361f0 0x7fe84813de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:48.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe846d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe8480a1380 0x7fe8480a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:43576/0 (socket says 172.21.15.12:43576) 2024-09-11T01:52:48.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe84659c640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe8480a0c90 0x7fe848135ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59100/0 (socket says 172.21.15.12:59100) 2024-09-11T01:52:48.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe846d9d640 1 -- 172.21.15.12:0/471497619 learned_addr learned my addr 172.21.15.12:0/471497619 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:48.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe846d9d640 10 monclient(hunting): get_auth_request con 0x7fe8480a1380 auth_method 0 2024-09-11T01:52:48.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe846d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:48.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe846d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:48.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe846d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:48.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe84759e640 10 monclient(hunting): get_auth_request con 0x7fe8481361f0 auth_method 0 2024-09-11T01:52:48.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe84759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:48.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe84759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:48.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe84759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:48.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe846d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe846d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe846d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe84759e640 1 -- 172.21.15.12:0/471497619 >> v2:172.21.15.12:3300/0 conn(0x7fe8481361f0 msgr2=0x7fe84813de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:52:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe84759e640 1 -- 172.21.15.12:0/471497619 >> v2:172.21.15.12:3300/0 conn(0x7fe8481361f0 msgr2=0x7fe84813de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:52:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe84759e640 1 --2- 172.21.15.12:0/471497619 >> v2:172.21.15.12:3300/0 conn(0x7fe8481361f0 0x7fe84813de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:52:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe84759e640 1 --2- 172.21.15.12:0/471497619 >> v2:172.21.15.12:3300/0 conn(0x7fe8481361f0 0x7fe84813de20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe846d9d640 10 monclient(hunting): handle_auth_done global_id 244307 payload 274 2024-09-11T01:52:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe846d9d640 1 -- 172.21.15.12:0/471497619 >> v2:172.21.15.12:3300/0 conn(0x7fe8481361f0 msgr2=0x7fe84813de20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:52:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe846d9d640 1 --2- 172.21.15.12:0/471497619 >> v2:172.21.15.12:3300/0 conn(0x7fe8481361f0 0x7fe84813de20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe846d9d640 1 -- 172.21.15.12:0/471497619 >> v1:172.21.15.12:6789/0 conn(0x7fe8480a0c90 legacy=0x7fe848135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe846d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:52:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe846d9d640 1 monclient: found mon.noname-b 2024-09-11T01:52:48.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe846d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:48.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe846d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:52:48.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.525+0000 7fe846d9d640 1 -- 172.21.15.12:0/471497619 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8480a2510 con 0x7fe8480a1380 2024-09-11T01:52:48.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.528+0000 7fe846d9d640 1 -- 172.21.15.12:0/471497619 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe8480a1380 msgr2=0x7fe8480a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:48.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.528+0000 7fe846d9d640 1 --2- 172.21.15.12:0/471497619 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe8480a1380 0x7fe8480a1760 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe838009870 tx=0x7fe83802a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e11341ddea6c7f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:48.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.528+0000 7fe845d9b640 1 -- 172.21.15.12:0/471497619 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe838037070 con 0x7fe8480a1380 2024-09-11T01:52:48.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.528+0000 7fe845d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:48.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.528+0000 7fe845d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:52:48.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.528+0000 7fe845d9b640 10 monclient: dump: 2024-09-11T01:52:48.541 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:48.541 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:48.541 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:48.541 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:48.541 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:48.541 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:48.541 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:48.541 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:48.542 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:48.542 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:48.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.528+0000 7fe845d9b640 10 monclient: _finish_auth 0 2024-09-11T01:52:48.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.528+0000 7fe845d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:52:48.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.528+0000 7fe845d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:48.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.528+0000 7fe845d9b640 1 -- 172.21.15.12:0/471497619 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe838032440 con 0x7fe8480a1380 2024-09-11T01:52:48.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.528+0000 7fe84d060640 5 monclient: authenticate success, global_id 244307 2024-09-11T01:52:48.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.528+0000 7fe84d060640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:52:48.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.528+0000 7fe845d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:48.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.528+0000 7fe845d9b640 1 -- 172.21.15.12:0/471497619 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe838032810 con 0x7fe8480a1380 2024-09-11T01:52:48.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.528+0000 7fe845d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:48.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.528+0000 7fe845d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:48.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.528+0000 7fe845d9b640 10 monclient: dump: 2024-09-11T01:52:48.543 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:48.543 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:48.543 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:48.543 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:48.543 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:48.543 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:48.543 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:48.543 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:48.543 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:48.543 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:48.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.528+0000 7fe84d060640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:48.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.528+0000 7fe84d060640 4 set_mon_vals no callback set 2024-09-11T01:52:48.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.528+0000 7fe84d060640 10 monclient: shutdown 2024-09-11T01:52:48.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.528+0000 7fe84d060640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:48.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.528+0000 7fe84d060640 1 -- 172.21.15.12:0/471497619 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe8480a1380 msgr2=0x7fe8480a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:48.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.528+0000 7fe84d060640 1 --2- 172.21.15.12:0/471497619 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe8480a1380 0x7fe8480a1760 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe838009870 tx=0x7fe83802a0a0 comp rx=0 tx=0).stop 2024-09-11T01:52:48.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.529+0000 7fe84d060640 1 -- 172.21.15.12:0/471497619 shutdown_connections 2024-09-11T01:52:48.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.529+0000 7fe84d060640 1 --2- 172.21.15.12:0/471497619 >> v2:172.21.15.12:3300/0 conn(0x7fe8481361f0 0x7fe84813de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:48.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.529+0000 7fe84d060640 1 --2- 172.21.15.12:0/471497619 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe8480a1380 0x7fe8480a1760 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:48.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.529+0000 7fe84d060640 1 -- 172.21.15.12:0/471497619 >> 172.21.15.12:0/471497619 conn(0x7fe8480a2d90 msgr2=0x7fe8480a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:48.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.529+0000 7fe84d060640 1 -- 172.21.15.12:0/471497619 shutdown_connections 2024-09-11T01:52:48.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.529+0000 7fe84d060640 1 -- 172.21.15.12:0/471497619 wait complete. 2024-09-11T01:52:48.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.529+0000 7fe84d060640 10 monclient: build_initial_monmap 2024-09-11T01:52:48.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.529+0000 7fe84d060640 10 monclient: monmap: 2024-09-11T01:52:48.544 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:48.544 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:48.544 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:48.544 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:48.545 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:48.545 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:48.545 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:48.545 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:48.545 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:48.545 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:48.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.530+0000 7fe84d060640 1 Processor -- start 2024-09-11T01:52:48.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.530+0000 7fe84d060640 1 -- start start 2024-09-11T01:52:48.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.530+0000 7fe84d060640 10 monclient: init 2024-09-11T01:52:48.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.530+0000 7fe84d060640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:48.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.530+0000 7fe84d060640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:52:48.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.530+0000 7fe84d060640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe8480a0c90 0x7fe8481cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:48.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.530+0000 7fe84d060640 10 monclient(hunting): picked mon.noname-c con 0x7fe8480a0c90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:48.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.530+0000 7fe84d060640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8480a1380 0x7fe848134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:48.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.530+0000 7fe84d060640 10 monclient(hunting): picked mon.noname-a con 0x7fe8480a1380 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:48.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.530+0000 7fe84d060640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe8481361f0 0x7fe8481353f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:48.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.530+0000 7fe84d060640 10 monclient(hunting): picked mon.noname-b con 0x7fe8481361f0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:48.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.530+0000 7fe84d060640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:48.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.530+0000 7fe84d060640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe84809ce20 con 0x7fe8480a1380 2024-09-11T01:52:48.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.530+0000 7fe84d060640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:48.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.530+0000 7fe84d060640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe84809cca0 con 0x7fe8481361f0 2024-09-11T01:52:48.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.530+0000 7fe84d060640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:48.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.530+0000 7fe84d060640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe8480a0390 con 0x7fe8480a0c90 2024-09-11T01:52:48.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe84d060640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:48.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe84d060640 10 monclient(hunting): authenticate will time out at 2174.974046s 2024-09-11T01:52:48.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe846d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe8480a0c90 0x7fe8481cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:48.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe84659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8480a1380 0x7fe848134eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:48.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe846d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe8480a0c90 0x7fe8481cae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:43590/0 (socket says 172.21.15.12:43590) 2024-09-11T01:52:48.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe846d9d640 1 -- 172.21.15.12:0/1734040170 learned_addr learned my addr 172.21.15.12:0/1734040170 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:48.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe846d9d640 10 monclient(hunting): get_auth_request con 0x7fe8480a0c90 auth_method 0 2024-09-11T01:52:48.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe846d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:48.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe846d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:48.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe846d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:48.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe84659c640 10 monclient(hunting): get_auth_request con 0x7fe8480a1380 auth_method 0 2024-09-11T01:52:48.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe84659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:48.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe84659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:48.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe84659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:48.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe846d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:48.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe846d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:48.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe84659c640 1 -- 172.21.15.12:0/1734040170 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8480a1380 msgr2=0x7fe848134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:52:48.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe84659c640 1 -- 172.21.15.12:0/1734040170 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8480a1380 msgr2=0x7fe848134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:52:48.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe84659c640 1 --2- 172.21.15.12:0/1734040170 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8480a1380 0x7fe848134eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:52:48.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe846d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:48.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe84659c640 1 --2- 172.21.15.12:0/1734040170 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8480a1380 0x7fe848134eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:48.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe84759e640 1 --2- 172.21.15.12:0/1734040170 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe8481361f0 0x7fe8481353f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:48.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe846d9d640 10 monclient(hunting): handle_auth_done global_id 244310 payload 824 2024-09-11T01:52:48.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe846d9d640 1 -- 172.21.15.12:0/1734040170 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe8481361f0 msgr2=0x7fe8481353f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:48.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe846d9d640 1 --2- 172.21.15.12:0/1734040170 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe8481361f0 0x7fe8481353f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:48.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe846d9d640 1 -- 172.21.15.12:0/1734040170 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8480a1380 msgr2=0x7fe848134eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:52:48.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe846d9d640 1 --2- 172.21.15.12:0/1734040170 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8480a1380 0x7fe848134eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:48.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe846d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:52:48.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe846d9d640 1 monclient: found mon.noname-c 2024-09-11T01:52:48.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe846d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:48.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe846d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:52:48.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.531+0000 7fe846d9d640 1 -- 172.21.15.12:0/1734040170 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe84809d480 con 0x7fe8480a0c90 2024-09-11T01:52:48.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.534+0000 7fe846d9d640 1 -- 172.21.15.12:0/1734040170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe8480a0c90 msgr2=0x7fe8481cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:48.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.534+0000 7fe846d9d640 1 --2- 172.21.15.12:0/1734040170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe8480a0c90 0x7fe8481cae10 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe8380090b0 tx=0x7fe8380046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:48.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.534+0000 7fe82f7fe640 1 -- 172.21.15.12:0/1734040170 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe838041020 con 0x7fe8480a0c90 2024-09-11T01:52:48.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.534+0000 7fe82f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:48.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.534+0000 7fe82f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:48.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.534+0000 7fe82f7fe640 10 monclient: dump: 2024-09-11T01:52:48.549 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:48.549 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:48.549 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:48.549 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:48.549 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:48.550 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:48.550 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:48.550 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:48.550 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:48.550 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:48.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.534+0000 7fe82f7fe640 10 monclient: _finish_auth 0 2024-09-11T01:52:48.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.534+0000 7fe82f7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:52:48.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.534+0000 7fe82f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:48.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.534+0000 7fe82f7fe640 1 -- 172.21.15.12:0/1734040170 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe83803d630 con 0x7fe8480a0c90 2024-09-11T01:52:48.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.534+0000 7fe84d060640 5 monclient: authenticate success, global_id 244310 2024-09-11T01:52:48.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.534+0000 7fe82f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:48.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.534+0000 7fe84d060640 10 monclient: _renew_subs 2024-09-11T01:52:48.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.534+0000 7fe82f7fe640 1 -- 172.21.15.12:0/1734040170 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe838037040 con 0x7fe8480a0c90 2024-09-11T01:52:48.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.534+0000 7fe84d060640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:48.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.534+0000 7fe84d060640 1 -- 172.21.15.12:0/1734040170 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe848120510 con 0x7fe8480a0c90 2024-09-11T01:52:48.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.534+0000 7fe82f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:48.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.534+0000 7fe82f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:48.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.534+0000 7fe82f7fe640 10 monclient: dump: 2024-09-11T01:52:48.551 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:48.551 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:48.551 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:48.551 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:48.552 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:48.552 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:48.552 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:48.552 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:48.552 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:48.552 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:48.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.534+0000 7fe84559a640 4 set_mon_vals no callback set 2024-09-11T01:52:48.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.535+0000 7fe84d060640 10 monclient: _renew_subs 2024-09-11T01:52:48.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.535+0000 7fe84d060640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:48.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.535+0000 7fe84d060640 1 -- 172.21.15.12:0/1734040170 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8481cb350 con 0x7fe8480a0c90 2024-09-11T01:52:48.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.535+0000 7fe82f7fe640 1 -- 172.21.15.12:0/1734040170 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fe83803a030 con 0x7fe8480a0c90 2024-09-11T01:52:48.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.535+0000 7fe84d060640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:48.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.535+0000 7fe84d060640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:48.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.535+0000 7fe84d060640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:48.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.535+0000 7fe84d060640 1 -- 172.21.15.12:0/1734040170 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe848136640 con 0x7fe8480a0c90 2024-09-11T01:52:48.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.535+0000 7fe82f7fe640 1 --2- 172.21.15.12:0/1734040170 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe83403e210 0x7fe8340406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:48.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.535+0000 7fe82f7fe640 1 -- 172.21.15.12:0/1734040170 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(798..798 src has 676..798) ==== 4831+0+0 (secure 0 0 0) 0x7fe838072eb0 con 0x7fe8480a0c90 2024-09-11T01:52:48.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.535+0000 7fe84659c640 1 --2- 172.21.15.12:0/1734040170 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe83403e210 0x7fe8340406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:48.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.535+0000 7fe84659c640 10 monclient: get_auth_request con 0x7fe83403e210 auth_method 0 2024-09-11T01:52:48.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.538+0000 7fe84659c640 1 --2- 172.21.15.12:0/1734040170 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe83403e210 0x7fe8340406d0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe848121920 tx=0x7fe83c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:48.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.540+0000 7fe82f7fe640 1 -- 172.21.15.12:0/1734040170 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe83804b070 con 0x7fe8480a0c90 2024-09-11T01:52:48.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.540+0000 7fe82f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:48.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.540+0000 7fe82f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:48.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.798+0000 7fe84d060640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:52:48.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.798+0000 7fe84d060640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:52:48.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.798+0000 7fe84d060640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:48.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.798+0000 7fe84d060640 1 -- 172.21.15.12:0/1734040170 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe84806d460 con 0x7fe8480a0c90 2024-09-11T01:52:48.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.799+0000 7fe82f7fe640 1 -- 172.21.15.12:0/1734040170 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7fe83803b3f0 con 0x7fe8480a0c90 2024-09-11T01:52:48.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.799+0000 7fe82f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:52:48.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.799+0000 7fe82f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:48.802 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":216,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":31,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:52:48.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.801+0000 7fe84d060640 1 -- 172.21.15.12:0/1734040170 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe83403e210 msgr2=0x7fe8340406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:48.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.801+0000 7fe84d060640 1 --2- 172.21.15.12:0/1734040170 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe83403e210 0x7fe8340406d0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe848121920 tx=0x7fe83c008040 comp rx=0 tx=0).stop 2024-09-11T01:52:48.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.801+0000 7fe84d060640 10 monclient: shutdown 2024-09-11T01:52:48.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.801+0000 7fe84d060640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:48.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.801+0000 7fe84d060640 1 -- 172.21.15.12:0/1734040170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe8480a0c90 msgr2=0x7fe8481cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:48.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.801+0000 7fe84d060640 1 --2- 172.21.15.12:0/1734040170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe8480a0c90 0x7fe8481cae10 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe8380090b0 tx=0x7fe8380046c0 comp rx=0 tx=0).stop 2024-09-11T01:52:48.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.801+0000 7fe84d060640 1 -- 172.21.15.12:0/1734040170 shutdown_connections 2024-09-11T01:52:48.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.801+0000 7fe84d060640 1 --2- 172.21.15.12:0/1734040170 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe8481361f0 0x7fe8481353f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:48.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.801+0000 7fe84d060640 1 --2- 172.21.15.12:0/1734040170 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe83403e210 0x7fe8340406d0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:48.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.801+0000 7fe84d060640 1 --2- 172.21.15.12:0/1734040170 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8480a1380 0x7fe848134eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:48.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.801+0000 7fe84d060640 1 --2- 172.21.15.12:0/1734040170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe8480a0c90 0x7fe8481cae10 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:48.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.801+0000 7fe84d060640 1 -- 172.21.15.12:0/1734040170 >> 172.21.15.12:0/1734040170 conn(0x7fe8480a2d90 msgr2=0x7fe8480a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:48.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.801+0000 7fe84d060640 1 -- 172.21.15.12:0/1734040170 shutdown_connections 2024-09-11T01:52:48.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:48.801+0000 7fe84d060640 1 -- 172.21.15.12:0/1734040170 wait complete. 2024-09-11T01:52:49.145 INFO:tasks.rados.rados.0.smithi071.stdout:3536: done (0 left) 2024-09-11T01:52:49.145 INFO:tasks.rados.rados.0.smithi071.stdout:3537: read oid 49 snap -1 2024-09-11T01:52:49.145 INFO:tasks.rados.rados.0.smithi071.stdout:3537: expect (ObjNum 1162 snap 370 seq_num 1162) 2024-09-11T01:52:49.145 INFO:tasks.rados.rados.0.smithi071.stdout:3538: snap_create 2024-09-11T01:52:49.153 INFO:tasks.rados.rados.0.smithi071.stdout:3537: expect (ObjNum 1162 snap 370 seq_num 1162) 2024-09-11T01:52:49.306 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 945522 2024-09-11T01:52:50.156 INFO:tasks.rados.rados.0.smithi071.stdout:3537: done (1 left) 2024-09-11T01:52:50.156 INFO:tasks.rados.rados.0.smithi071.stdout:3538: done (0 left) 2024-09-11T01:52:50.156 INFO:tasks.rados.rados.0.smithi071.stdout:3539: snap_create 2024-09-11T01:52:51.815 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:52:51.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.927+0000 7fe9d4a40640 10 monclient: get_monmap_and_config 2024-09-11T01:52:51.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.927+0000 7fe9d4a40640 10 monclient: build_initial_monmap 2024-09-11T01:52:51.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.927+0000 7fe9d4a40640 10 monclient: monmap: 2024-09-11T01:52:51.929 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:51.929 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:51.929 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:51.928614+0000 2024-09-11T01:52:51.929 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:51.928614+0000 2024-09-11T01:52:51.930 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:51.930 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:51.930 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:51.930 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:51.930 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:51.930 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:51.930 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:51.930 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:51.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.928+0000 7fe9d4a40640 1 Processor -- start 2024-09-11T01:52:51.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.928+0000 7fe9d4a40640 1 -- start start 2024-09-11T01:52:51.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.928+0000 7fe9d4a40640 10 monclient: init 2024-09-11T01:52:51.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9d4a40640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:51.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9d4a40640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-11T01:52:51.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9d4a40640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe9d00a2e90 0x7fe9d006dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:51.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9d4a40640 10 monclient(hunting): picked mon.noname-c con 0x7fe9d00a2e90 addr v2:172.21.15.71:3300/0 2024-09-11T01:52:51.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9d4a40640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe9d00a3270 0x7fe9d00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:51.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9d4a40640 10 monclient(hunting): picked mon.noname-a con 0x7fe9d00a3270 addr v2:172.21.15.12:3300/0 2024-09-11T01:52:51.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9d4a40640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe9d0138040 addr v1:172.21.15.12:6789/0 2024-09-11T01:52:51.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9d4a40640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9d00a46c0 con 0x7fe9d0138040 2024-09-11T01:52:51.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9d4a40640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:51.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9d4a40640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fe9d00a4900 con 0x7fe9d00a3270 2024-09-11T01:52:51.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9d4a40640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:51.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9d4a40640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fe9d006ee50 con 0x7fe9d00a2e90 2024-09-11T01:52:51.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9d4a40640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:51.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9d4a40640 10 monclient(hunting): authenticate will time out at 2178.371429s 2024-09-11T01:52:51.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9cdd9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe9d00a3270 0x7fe9d00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:51.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9cdd9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe9d00a3270 0x7fe9d00a3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59976/0 (socket says 172.21.15.12:59976) 2024-09-11T01:52:51.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9cdd9b640 1 -- 172.21.15.12:0/4028024554 learned_addr learned my addr 172.21.15.12:0/4028024554 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:51.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9cdd9b640 10 monclient(hunting): get_auth_request con 0x7fe9d00a3270 auth_method 0 2024-09-11T01:52:51.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9cdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:51.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9cdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:51.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:51.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9ced9d640 1 --1- 172.21.15.12:0/4028024554 >> v1:172.21.15.12:6789/0 conn(0x7fe9d0138040 0x7fe9d01384d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59110/0 (socket says 172.21.15.12:59110) 2024-09-11T01:52:51.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9ce59c640 1 --2- 172.21.15.12:0/4028024554 >> v2:172.21.15.71:3300/0 conn(0x7fe9d00a2e90 0x7fe9d006dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:51.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9cdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:51.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9cdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:51.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9cdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:51.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9ce59c640 10 monclient(hunting): get_auth_request con 0x7fe9d00a2e90 auth_method 0 2024-09-11T01:52:51.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:51.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9ce59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:51.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:51.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9ce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:51.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9ce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:51.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.929+0000 7fe9ce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:51.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.930+0000 7fe9ce59c640 10 monclient(hunting): handle_auth_done global_id 234328 payload 274 2024-09-11T01:52:51.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.930+0000 7fe9ce59c640 1 -- 172.21.15.12:0/4028024554 >> v2:172.21.15.12:3300/0 conn(0x7fe9d00a3270 msgr2=0x7fe9d00a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:51.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.930+0000 7fe9ce59c640 1 --2- 172.21.15.12:0/4028024554 >> v2:172.21.15.12:3300/0 conn(0x7fe9d00a3270 0x7fe9d00a3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:51.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.930+0000 7fe9ce59c640 1 -- 172.21.15.12:0/4028024554 >> v1:172.21.15.12:6789/0 conn(0x7fe9d0138040 legacy=0x7fe9d01384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:51.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.930+0000 7fe9ce59c640 10 monclient: _finish_hunting 0 2024-09-11T01:52:51.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.930+0000 7fe9ce59c640 1 monclient: found mon.noname-c 2024-09-11T01:52:51.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.930+0000 7fe9ce59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:51.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.930+0000 7fe9ce59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:52:51.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.930+0000 7fe9ce59c640 1 -- 172.21.15.12:0/4028024554 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9d006eb70 con 0x7fe9d00a2e90 2024-09-11T01:52:51.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.933+0000 7fe9ce59c640 1 -- 172.21.15.12:0/4028024554 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe9d00a2e90 msgr2=0x7fe9d006dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:51.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.933+0000 7fe9ce59c640 1 --2- 172.21.15.12:0/4028024554 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe9d00a2e90 0x7fe9d006dc60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0071140 tx=0x7fe9c002a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10b8d996ecc77d74 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:51.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.933+0000 7fe9cd59a640 1 -- 172.21.15.12:0/4028024554 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c0037070 con 0x7fe9d00a2e90 2024-09-11T01:52:51.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.933+0000 7fe9cd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:51.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.933+0000 7fe9cd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:51.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.933+0000 7fe9cd59a640 10 monclient: dump: 2024-09-11T01:52:51.940 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:51.940 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:51.940 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:51.940 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:51.940 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:51.940 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:51.940 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:51.940 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:51.940 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:51.940 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:51.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.933+0000 7fe9cd59a640 10 monclient: _finish_auth 0 2024-09-11T01:52:51.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.933+0000 7fe9cd59a640 10 monclient: _check_auth_tickets 2024-09-11T01:52:51.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.933+0000 7fe9cd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:51.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.933+0000 7fe9cd59a640 1 -- 172.21.15.12:0/4028024554 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9c0032440 con 0x7fe9d00a2e90 2024-09-11T01:52:51.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.933+0000 7fe9cd59a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:51.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.933+0000 7fe9cd59a640 1 -- 172.21.15.12:0/4028024554 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c0032810 con 0x7fe9d00a2e90 2024-09-11T01:52:51.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.933+0000 7fe9cd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:51.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.933+0000 7fe9cd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:52:51.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.933+0000 7fe9cd59a640 10 monclient: dump: 2024-09-11T01:52:51.941 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:51.941 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:51.941 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:51.941 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:51.941 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:51.941 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:51.941 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:51.942 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:51.942 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:51.942 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:51.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.933+0000 7fe9d4a40640 5 monclient: authenticate success, global_id 234328 2024-09-11T01:52:51.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.933+0000 7fe9d4a40640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:51.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.933+0000 7fe9d4a40640 4 set_mon_vals no callback set 2024-09-11T01:52:51.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.934+0000 7fe9d4a40640 10 monclient: shutdown 2024-09-11T01:52:51.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.934+0000 7fe9d4a40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:51.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.934+0000 7fe9d4a40640 1 -- 172.21.15.12:0/4028024554 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe9d00a2e90 msgr2=0x7fe9d006dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:51.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.934+0000 7fe9d4a40640 1 --2- 172.21.15.12:0/4028024554 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe9d00a2e90 0x7fe9d006dc60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0071140 tx=0x7fe9c002a000 comp rx=0 tx=0).stop 2024-09-11T01:52:51.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.934+0000 7fe9d4a40640 1 -- 172.21.15.12:0/4028024554 shutdown_connections 2024-09-11T01:52:51.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.934+0000 7fe9d4a40640 1 --2- 172.21.15.12:0/4028024554 >> v2:172.21.15.12:3300/0 conn(0x7fe9d00a3270 0x7fe9d00a3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:51.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.934+0000 7fe9d4a40640 1 --2- 172.21.15.12:0/4028024554 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe9d00a2e90 0x7fe9d006dc60 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:51.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.934+0000 7fe9d4a40640 1 -- 172.21.15.12:0/4028024554 >> 172.21.15.12:0/4028024554 conn(0x7fe9d00a8490 msgr2=0x7fe9d00a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:51.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.934+0000 7fe9d4a40640 1 -- 172.21.15.12:0/4028024554 shutdown_connections 2024-09-11T01:52:51.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.934+0000 7fe9d4a40640 1 -- 172.21.15.12:0/4028024554 wait complete. 2024-09-11T01:52:51.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.935+0000 7fe9d4a40640 10 monclient: build_initial_monmap 2024-09-11T01:52:51.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.935+0000 7fe9d4a40640 10 monclient: monmap: 2024-09-11T01:52:51.943 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:51.943 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:51.943 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:51.943 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:51.943 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:51.943 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:51.943 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:51.943 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:51.943 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:51.943 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:51.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.935+0000 7fe9d4a40640 1 Processor -- start 2024-09-11T01:52:51.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.935+0000 7fe9d4a40640 1 -- start start 2024-09-11T01:52:51.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.935+0000 7fe9d4a40640 10 monclient: init 2024-09-11T01:52:51.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.935+0000 7fe9d4a40640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:51.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.935+0000 7fe9d4a40640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:52:51.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.935+0000 7fe9d4a40640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9d00a2e90 0x7fe9d01cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:51.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.935+0000 7fe9d4a40640 10 monclient(hunting): picked mon.noname-a con 0x7fe9d00a2e90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:51.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.935+0000 7fe9d4a40640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe9d00a3270 0x7fe9d01cda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:51.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.935+0000 7fe9d4a40640 10 monclient(hunting): picked mon.noname-c con 0x7fe9d00a3270 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:51.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.935+0000 7fe9d4a40640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe9d0138040 0x7fe9d01248d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:51.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.935+0000 7fe9d4a40640 10 monclient(hunting): picked mon.noname-b con 0x7fe9d0138040 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:51.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.935+0000 7fe9d4a40640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:51.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.935+0000 7fe9d4a40640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d006fba0 con 0x7fe9d00a2e90 2024-09-11T01:52:51.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.935+0000 7fe9d4a40640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:51.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.935+0000 7fe9d4a40640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d00a3bc0 con 0x7fe9d0138040 2024-09-11T01:52:51.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.935+0000 7fe9d4a40640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:51.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.935+0000 7fe9d4a40640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe9d006fd20 con 0x7fe9d00a3270 2024-09-11T01:52:51.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.935+0000 7fe9d4a40640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:51.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.935+0000 7fe9d4a40640 10 monclient(hunting): authenticate will time out at 2178.377892s 2024-09-11T01:52:51.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.935+0000 7fe9ce59c640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9d00a2e90 msgr2=0x7fe9d01cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:52:51.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.935+0000 7fe9ce59c640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9d00a2e90 msgr2=0x7fe9d01cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:52:51.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.935+0000 7fe9ce59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9d00a2e90 0x7fe9d01cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2024-09-11T01:52:51.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9ce59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9d00a2e90 0x7fe9d01cd4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:51.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9cdd9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe9d00a3270 0x7fe9d01cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:51.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9cdd9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe9d00a3270 0x7fe9d01cda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:43610/0 (socket says 172.21.15.12:43610) 2024-09-11T01:52:51.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9cdd9b640 1 -- 172.21.15.12:0/4058384260 learned_addr learned my addr 172.21.15.12:0/4058384260 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:51.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9cdd9b640 10 monclient(hunting): get_auth_request con 0x7fe9d00a3270 auth_method 0 2024-09-11T01:52:51.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9cdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:51.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9cdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:51.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:51.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9cdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:51.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9cdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:51.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9cdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:51.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9ced9d640 1 --2- 172.21.15.12:0/4058384260 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe9d0138040 0x7fe9d01248d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:51.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9ced9d640 10 monclient(hunting): get_auth_request con 0x7fe9d0138040 auth_method 0 2024-09-11T01:52:51.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:51.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9ced9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:51.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9ced9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:51.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9ced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:51.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9ced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:51.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9ced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:51.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9cdd9b640 10 monclient(hunting): handle_auth_done global_id 244313 payload 824 2024-09-11T01:52:51.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9cdd9b640 1 -- 172.21.15.12:0/4058384260 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe9d0138040 msgr2=0x7fe9d01248d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:51.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9cdd9b640 1 --2- 172.21.15.12:0/4058384260 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe9d0138040 0x7fe9d01248d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:51.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9cdd9b640 1 -- 172.21.15.12:0/4058384260 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9d00a2e90 msgr2=0x7fe9d01cd4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:52:51.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9cdd9b640 1 --2- 172.21.15.12:0/4058384260 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9d00a2e90 0x7fe9d01cd4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:51.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9cdd9b640 10 monclient: _finish_hunting 0 2024-09-11T01:52:51.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9cdd9b640 1 monclient: found mon.noname-c 2024-09-11T01:52:51.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9cdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:51.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9cdd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:52:51.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.936+0000 7fe9cdd9b640 1 -- 172.21.15.12:0/4058384260 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9d0137c20 con 0x7fe9d00a3270 2024-09-11T01:52:51.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.938+0000 7fe9cdd9b640 1 -- 172.21.15.12:0/4058384260 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe9d00a3270 msgr2=0x7fe9d01cda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:51.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.938+0000 7fe9cdd9b640 1 --2- 172.21.15.12:0/4058384260 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe9d00a3270 0x7fe9d01cda20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe9c4009fd0 tx=0x7fe9c400ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:51.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.938+0000 7fe9beffd640 1 -- 172.21.15.12:0/4058384260 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c4017070 con 0x7fe9d00a3270 2024-09-11T01:52:51.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.938+0000 7fe9beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:51.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.938+0000 7fe9beffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:51.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.938+0000 7fe9beffd640 10 monclient: dump: 2024-09-11T01:52:51.950 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:51.950 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:51.950 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:51.950 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:51.950 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:51.950 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:51.950 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:51.950 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:51.950 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:51.950 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:51.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.938+0000 7fe9beffd640 10 monclient: _finish_auth 0 2024-09-11T01:52:51.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.938+0000 7fe9beffd640 10 monclient: _check_auth_tickets 2024-09-11T01:52:51.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.938+0000 7fe9beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:51.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.938+0000 7fe9beffd640 1 -- 172.21.15.12:0/4058384260 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9c400cad0 con 0x7fe9d00a3270 2024-09-11T01:52:51.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.938+0000 7fe9d4a40640 5 monclient: authenticate success, global_id 244313 2024-09-11T01:52:51.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.938+0000 7fe9d4a40640 10 monclient: _renew_subs 2024-09-11T01:52:51.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.938+0000 7fe9d4a40640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:51.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.938+0000 7fe9d4a40640 1 -- 172.21.15.12:0/4058384260 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9d0126300 con 0x7fe9d00a3270 2024-09-11T01:52:51.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.938+0000 7fe9beffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:51.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.938+0000 7fe9beffd640 1 -- 172.21.15.12:0/4058384260 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c400cea0 con 0x7fe9d00a3270 2024-09-11T01:52:51.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.938+0000 7fe9beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:51.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.938+0000 7fe9beffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:51.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.938+0000 7fe9beffd640 10 monclient: dump: 2024-09-11T01:52:51.951 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:51.951 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:51.952 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:51.952 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:51.952 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:51.952 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:51.952 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:51.952 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:51.952 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:51.952 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:51.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.938+0000 7fe9d4a40640 10 monclient: _renew_subs 2024-09-11T01:52:51.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.938+0000 7fe9ccd99640 4 set_mon_vals no callback set 2024-09-11T01:52:51.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.939+0000 7fe9d4a40640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:51.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.939+0000 7fe9d4a40640 1 -- 172.21.15.12:0/4058384260 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9d0128660 con 0x7fe9d00a3270 2024-09-11T01:52:51.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.939+0000 7fe9d4a40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:51.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.939+0000 7fe9d4a40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:51.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.939+0000 7fe9d4a40640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:51.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.939+0000 7fe9d4a40640 1 -- 172.21.15.12:0/4058384260 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9d006dc60 con 0x7fe9d00a3270 2024-09-11T01:52:51.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.940+0000 7fe9beffd640 1 -- 172.21.15.12:0/4058384260 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fe9c4024060 con 0x7fe9d00a3270 2024-09-11T01:52:51.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.940+0000 7fe9beffd640 1 --2- 172.21.15.12:0/4058384260 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe9b003e240 0x7fe9b0040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:51.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.940+0000 7fe9beffd640 1 -- 172.21.15.12:0/4058384260 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(800..800 src has 676..800) ==== 4787+0+0 (secure 0 0 0) 0x7fe9c4052c70 con 0x7fe9d00a3270 2024-09-11T01:52:51.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.940+0000 7fe9ce59c640 1 --2- 172.21.15.12:0/4058384260 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe9b003e240 0x7fe9b0040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:51.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.940+0000 7fe9ce59c640 10 monclient: get_auth_request con 0x7fe9b003e240 auth_method 0 2024-09-11T01:52:51.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.943+0000 7fe9ce59c640 1 --2- 172.21.15.12:0/4058384260 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe9b003e240 0x7fe9b0040700 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0031000 tx=0x7fe9c002c000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:51.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.944+0000 7fe9beffd640 1 -- 172.21.15.12:0/4058384260 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe9c4051660 con 0x7fe9d00a3270 2024-09-11T01:52:51.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.944+0000 7fe9beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:51.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:51.944+0000 7fe9beffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:51.987 INFO:tasks.rados.rados.0.smithi071.stdout:3539: done (0 left) 2024-09-11T01:52:51.987 INFO:tasks.rados.rados.0.smithi071.stdout:3540: write oid 28 current snap is 401 2024-09-11T01:52:51.987 INFO:tasks.rados.rados.0.smithi071.stdout:3540: seq_num 1226 ranges {605382=588107,1925865=679517} 2024-09-11T01:52:51.993 INFO:tasks.rados.rados.0.smithi071.stdout:3540: writing smithi07134777-28 from 605382 to 1193489 tid 1 2024-09-11T01:52:52.001 INFO:tasks.rados.rados.0.smithi071.stdout:3540: writing smithi07134777-28 from 1925865 to 2605382 tid 2 2024-09-11T01:52:52.002 INFO:tasks.rados.rados.0.smithi071.stdout:3541: copy_from oid 32 from oid 8 current snap is 401 2024-09-11T01:52:52.002 INFO:tasks.rados.rados.0.smithi071.stdout:3540: finishing write tid 1 to smithi07134777-28 2024-09-11T01:52:52.002 INFO:tasks.rados.rados.0.smithi071.stdout:3542: rollback oid 23 current snap is 401 2024-09-11T01:52:52.002 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 23 to 386 2024-09-11T01:52:52.002 INFO:tasks.rados.rados.0.smithi071.stdout:3543: snap_remove snap 363 2024-09-11T01:52:52.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.213+0000 7fe9d4a40640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:52:52.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.213+0000 7fe9d4a40640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:52:52.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.213+0000 7fe9d4a40640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:52.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.213+0000 7fe9d4a40640 1 -- 172.21.15.12:0/4058384260 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe9d0126e60 con 0x7fe9d00a3270 2024-09-11T01:52:52.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.214+0000 7fe9beffd640 1 -- 172.21.15.12:0/4058384260 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fe9c4013cc0 con 0x7fe9d00a3270 2024-09-11T01:52:52.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.214+0000 7fe9beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:52:52.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.214+0000 7fe9beffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:52.216 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":220,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:52:52.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.216+0000 7fe9d4a40640 1 -- 172.21.15.12:0/4058384260 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe9b003e240 msgr2=0x7fe9b0040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.216+0000 7fe9d4a40640 1 --2- 172.21.15.12:0/4058384260 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe9b003e240 0x7fe9b0040700 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0031000 tx=0x7fe9c002c000 comp rx=0 tx=0).stop 2024-09-11T01:52:52.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.216+0000 7fe9d4a40640 10 monclient: shutdown 2024-09-11T01:52:52.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.216+0000 7fe9d4a40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:52.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.216+0000 7fe9d4a40640 1 -- 172.21.15.12:0/4058384260 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe9d00a3270 msgr2=0x7fe9d01cda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.216+0000 7fe9d4a40640 1 --2- 172.21.15.12:0/4058384260 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe9d00a3270 0x7fe9d01cda20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe9c4009fd0 tx=0x7fe9c400ed40 comp rx=0 tx=0).stop 2024-09-11T01:52:52.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.216+0000 7fe9d4a40640 1 -- 172.21.15.12:0/4058384260 shutdown_connections 2024-09-11T01:52:52.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.216+0000 7fe9d4a40640 1 --2- 172.21.15.12:0/4058384260 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe9d0138040 0x7fe9d01248d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.216+0000 7fe9d4a40640 1 --2- 172.21.15.12:0/4058384260 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe9d00a3270 0x7fe9d01cda20 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.216+0000 7fe9d4a40640 1 --2- 172.21.15.12:0/4058384260 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe9b003e240 0x7fe9b0040700 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.216+0000 7fe9d4a40640 1 --2- 172.21.15.12:0/4058384260 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9d00a2e90 0x7fe9d01cd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.216+0000 7fe9d4a40640 1 -- 172.21.15.12:0/4058384260 >> 172.21.15.12:0/4058384260 conn(0x7fe9d00a8490 msgr2=0x7fe9d0072090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:52.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.216+0000 7fe9d4a40640 1 -- 172.21.15.12:0/4058384260 shutdown_connections 2024-09-11T01:52:52.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.216+0000 7fe9d4a40640 1 -- 172.21.15.12:0/4058384260 wait complete. 2024-09-11T01:52:52.228 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:52:52.228 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:52:52.229 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:52:52.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.329+0000 7faa49608640 10 monclient: get_monmap_and_config 2024-09-11T01:52:52.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.329+0000 7faa49608640 10 monclient: build_initial_monmap 2024-09-11T01:52:52.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.329+0000 7faa49608640 10 monclient: monmap: 2024-09-11T01:52:52.331 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:52.331 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:52.331 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:52.330323+0000 2024-09-11T01:52:52.331 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:52.330323+0000 2024-09-11T01:52:52.331 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:52.331 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.332 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:52.332 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:52.332 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:52.332 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:52.332 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:52.332 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.330+0000 7faa49608640 1 Processor -- start 2024-09-11T01:52:52.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.330+0000 7faa49608640 1 -- start start 2024-09-11T01:52:52.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.330+0000 7faa49608640 10 monclient: init 2024-09-11T01:52:52.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.330+0000 7faa49608640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:52.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.330+0000 7faa49608640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-11T01:52:52.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.330+0000 7faa49608640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faa44131710 addr v1:172.21.15.71:6789/0 2024-09-11T01:52:52.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.330+0000 7faa49608640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7faa440a0d70 0x7faa4413adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:52.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.330+0000 7faa49608640 10 monclient(hunting): picked mon.noname-a con 0x7faa440a0d70 addr v2:172.21.15.12:3300/0 2024-09-11T01:52:52.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.330+0000 7faa49608640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faa4413b2f0 addr v1:172.21.15.12:6789/0 2024-09-11T01:52:52.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.330+0000 7faa49608640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa44074a70 con 0x7faa4413b2f0 2024-09-11T01:52:52.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.331+0000 7faa49608640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa44073af0 con 0x7faa44131710 2024-09-11T01:52:52.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.331+0000 7faa49608640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:52.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.331+0000 7faa49608640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7faa44073d90 con 0x7faa440a0d70 2024-09-11T01:52:52.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.331+0000 7faa49608640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:52.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.331+0000 7faa49608640 10 monclient(hunting): authenticate will time out at 2178.773037s 2024-09-11T01:52:52.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.331+0000 7faa3bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7faa440a0d70 0x7faa4413adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:52.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.331+0000 7faa3bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7faa440a0d70 0x7faa4413adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59994/0 (socket says 172.21.15.12:59994) 2024-09-11T01:52:52.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.331+0000 7faa3bfff640 1 -- 172.21.15.12:0/2450421875 learned_addr learned my addr 172.21.15.12:0/2450421875 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:52.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.331+0000 7faa3bfff640 10 monclient(hunting): get_auth_request con 0x7faa440a0d70 auth_method 0 2024-09-11T01:52:52.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.331+0000 7faa3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:52.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.331+0000 7faa3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:52.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.331+0000 7faa3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:52.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.331+0000 7faa3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:52.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.331+0000 7faa3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:52.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.331+0000 7faa3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:52.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.333+0000 7faa437fe640 1 -- 172.21.15.12:0/2450421875 >> v1:172.21.15.12:6789/0 conn(0x7faa4413b2f0 legacy=0x7faa4413d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:52.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.333+0000 7faa42ffd640 1 -- 172.21.15.12:0/2450421875 >> v1:172.21.15.71:6789/0 conn(0x7faa44131710 legacy=0x7faa44131af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:52.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.333+0000 7faa3bfff640 10 monclient(hunting): handle_auth_done global_id 264102 payload 274 2024-09-11T01:52:52.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.333+0000 7faa3bfff640 1 -- 172.21.15.12:0/2450421875 >> v1:172.21.15.71:6789/0 conn(0x7faa44131710 legacy=0x7faa44131af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.333+0000 7faa3bfff640 1 -- 172.21.15.12:0/2450421875 >> v1:172.21.15.12:6789/0 conn(0x7faa4413b2f0 legacy=0x7faa4413d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.333+0000 7faa3bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:52:52.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.333+0000 7faa3bfff640 1 monclient: found mon.noname-a 2024-09-11T01:52:52.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.333+0000 7faa3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:52.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.333+0000 7faa3bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:52:52.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.333+0000 7faa3bfff640 1 -- 172.21.15.12:0/2450421875 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa44074070 con 0x7faa440a0d70 2024-09-11T01:52:52.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.336+0000 7faa3bfff640 1 -- 172.21.15.12:0/2450421875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa440a0d70 msgr2=0x7faa4413adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:52.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.336+0000 7faa3bfff640 1 --2- 172.21.15.12:0/2450421875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa440a0d70 0x7faa4413adb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7faa2c009970 tx=0x7faa2c02a400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5345ed2edaccd2d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:52.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.336+0000 7faa427fc640 1 -- 172.21.15.12:0/2450421875 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2c036070 con 0x7faa440a0d70 2024-09-11T01:52:52.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.336+0000 7faa427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:52.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.336+0000 7faa427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:52:52.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.336+0000 7faa427fc640 10 monclient: dump: 2024-09-11T01:52:52.339 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:52.339 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:52.339 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.339 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.339 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:52.339 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.339 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:52.340 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:52.340 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:52.340 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.336+0000 7faa427fc640 10 monclient: _finish_auth 0 2024-09-11T01:52:52.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.336+0000 7faa427fc640 10 monclient: _check_auth_tickets 2024-09-11T01:52:52.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.336+0000 7faa427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:52.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.336+0000 7faa49608640 5 monclient: authenticate success, global_id 264102 2024-09-11T01:52:52.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.336+0000 7faa427fc640 1 -- 172.21.15.12:0/2450421875 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa2c02fa00 con 0x7faa440a0d70 2024-09-11T01:52:52.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.336+0000 7faa49608640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:52:52.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.336+0000 7faa427fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:52.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.336+0000 7faa49608640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:52.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.336+0000 7faa49608640 4 set_mon_vals no callback set 2024-09-11T01:52:52.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.336+0000 7faa427fc640 1 -- 172.21.15.12:0/2450421875 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2c02fdd0 con 0x7faa440a0d70 2024-09-11T01:52:52.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.336+0000 7faa427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:52.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.336+0000 7faa427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:52:52.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.336+0000 7faa427fc640 10 monclient: dump: 2024-09-11T01:52:52.341 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:52.341 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:52.341 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.341 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.341 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:52.341 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.341 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:52.341 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:52.341 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:52.341 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.337+0000 7faa49608640 10 monclient: shutdown 2024-09-11T01:52:52.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.337+0000 7faa49608640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:52.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.337+0000 7faa49608640 1 -- 172.21.15.12:0/2450421875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa440a0d70 msgr2=0x7faa4413adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.337+0000 7faa49608640 1 --2- 172.21.15.12:0/2450421875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa440a0d70 0x7faa4413adb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7faa2c009970 tx=0x7faa2c02a400 comp rx=0 tx=0).stop 2024-09-11T01:52:52.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.337+0000 7faa49608640 1 -- 172.21.15.12:0/2450421875 shutdown_connections 2024-09-11T01:52:52.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.337+0000 7faa49608640 1 --2- 172.21.15.12:0/2450421875 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa440a0d70 0x7faa4413adb0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.337+0000 7faa49608640 1 -- 172.21.15.12:0/2450421875 >> 172.21.15.12:0/2450421875 conn(0x7faa440a36c0 msgr2=0x7faa440a3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:52.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.337+0000 7faa49608640 1 -- 172.21.15.12:0/2450421875 shutdown_connections 2024-09-11T01:52:52.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.337+0000 7faa49608640 1 -- 172.21.15.12:0/2450421875 wait complete. 2024-09-11T01:52:52.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.338+0000 7faa49608640 10 monclient: build_initial_monmap 2024-09-11T01:52:52.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.338+0000 7faa49608640 10 monclient: monmap: 2024-09-11T01:52:52.347 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:52.348 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:52.348 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:52.348 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:52.348 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:52.348 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.348 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:52.348 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:52.348 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:52.348 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.338+0000 7faa49608640 1 Processor -- start 2024-09-11T01:52:52.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.338+0000 7faa49608640 1 -- start start 2024-09-11T01:52:52.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.338+0000 7faa49608640 10 monclient: init 2024-09-11T01:52:52.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.338+0000 7faa49608640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:52.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.338+0000 7faa49608640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:52:52.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.338+0000 7faa49608640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa440a0d70 0x7faa44130900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:52.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.338+0000 7faa49608640 10 monclient(hunting): picked mon.noname-a con 0x7faa440a0d70 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:52.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.338+0000 7faa49608640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa44131710 0x7faa44130e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:52.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.338+0000 7faa49608640 10 monclient(hunting): picked mon.noname-b con 0x7faa44131710 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:52.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.338+0000 7faa49608640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa4413b2f0 0x7faa44133600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:52.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.338+0000 7faa49608640 10 monclient(hunting): picked mon.noname-c con 0x7faa4413b2f0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:52.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.338+0000 7faa49608640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:52.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.338+0000 7faa49608640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7faa44073350 con 0x7faa440a0d70 2024-09-11T01:52:52.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.338+0000 7faa49608640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:52.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.338+0000 7faa49608640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7faa440731d0 con 0x7faa44131710 2024-09-11T01:52:52.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.338+0000 7faa49608640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:52.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.338+0000 7faa49608640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7faa4413ee00 con 0x7faa4413b2f0 2024-09-11T01:52:52.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.338+0000 7faa49608640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:52.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.338+0000 7faa49608640 10 monclient(hunting): authenticate will time out at 2178.780918s 2024-09-11T01:52:52.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.338+0000 7faa42ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa440a0d70 0x7faa44130900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:52.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa42ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa440a0d70 0x7faa44130900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60000/0 (socket says 172.21.15.12:60000) 2024-09-11T01:52:52.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa42ffd640 1 -- 172.21.15.12:0/1096130371 learned_addr learned my addr 172.21.15.12:0/1096130371 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:52.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa42ffd640 10 monclient(hunting): get_auth_request con 0x7faa440a0d70 auth_method 0 2024-09-11T01:52:52.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa42ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:52.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa42ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:52.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:52.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa437fe640 1 --2- 172.21.15.12:0/1096130371 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa4413b2f0 0x7faa44133600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa437fe640 10 monclient(hunting): get_auth_request con 0x7faa4413b2f0 auth_method 0 2024-09-11T01:52:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa437fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa42ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa42ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa42ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa3bfff640 1 --2- 172.21.15.12:0/1096130371 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa44131710 0x7faa44130e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa3bfff640 10 monclient(hunting): get_auth_request con 0x7faa44131710 auth_method 0 2024-09-11T01:52:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:52.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa42ffd640 10 monclient(hunting): handle_auth_done global_id 264105 payload 824 2024-09-11T01:52:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa42ffd640 1 -- 172.21.15.12:0/1096130371 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa4413b2f0 msgr2=0x7faa44133600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa42ffd640 1 --2- 172.21.15.12:0/1096130371 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa4413b2f0 0x7faa44133600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa42ffd640 1 -- 172.21.15.12:0/1096130371 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa44131710 msgr2=0x7faa44130e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa42ffd640 1 --2- 172.21.15.12:0/1096130371 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa44131710 0x7faa44130e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa42ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:52:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa42ffd640 1 monclient: found mon.noname-a 2024-09-11T01:52:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa42ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:52:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa42ffd640 1 -- 172.21.15.12:0/1096130371 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4413ac30 con 0x7faa440a0d70 2024-09-11T01:52:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa437fe640 1 --2- 172.21.15.12:0/1096130371 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faa4413b2f0 0x7faa44133600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:52:52.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.339+0000 7faa3bfff640 1 --2- 172.21.15.12:0/1096130371 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa44131710 0x7faa44130e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:52:52.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.341+0000 7faa42ffd640 1 -- 172.21.15.12:0/1096130371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa440a0d70 msgr2=0x7faa44130900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:52.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.341+0000 7faa42ffd640 1 --2- 172.21.15.12:0/1096130371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa440a0d70 0x7faa44130900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7faa3400c490 tx=0x7faa3400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:52.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.341+0000 7faa3b7fe640 1 -- 172.21.15.12:0/1096130371 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa34017070 con 0x7faa440a0d70 2024-09-11T01:52:52.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.341+0000 7faa3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:52.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.341+0000 7faa3b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:52:52.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.341+0000 7faa3b7fe640 10 monclient: dump: 2024-09-11T01:52:52.353 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:52.353 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:52.353 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.353 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.353 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:52.353 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.353 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:52.353 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:52.354 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:52.354 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.341+0000 7faa3b7fe640 10 monclient: _finish_auth 0 2024-09-11T01:52:52.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.341+0000 7faa3b7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:52:52.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.341+0000 7faa3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:52.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.341+0000 7faa3b7fe640 1 -- 172.21.15.12:0/1096130371 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa340057c0 con 0x7faa440a0d70 2024-09-11T01:52:52.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.341+0000 7faa49608640 5 monclient: authenticate success, global_id 264105 2024-09-11T01:52:52.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.341+0000 7faa3b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:52.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.341+0000 7faa3b7fe640 1 -- 172.21.15.12:0/1096130371 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa34005bb0 con 0x7faa440a0d70 2024-09-11T01:52:52.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.341+0000 7faa49608640 10 monclient: _renew_subs 2024-09-11T01:52:52.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.341+0000 7faa49608640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:52:52.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.341+0000 7faa49608640 1 -- 172.21.15.12:0/1096130371 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa44075000 con 0x7faa440a0d70 2024-09-11T01:52:52.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.341+0000 7faa41ffb640 4 set_mon_vals no callback set 2024-09-11T01:52:52.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.341+0000 7faa3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:52.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.341+0000 7faa3b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:52:52.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.341+0000 7faa3b7fe640 10 monclient: dump: 2024-09-11T01:52:52.355 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:52.355 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:52.355 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.355 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.355 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:52.355 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.355 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:52.355 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:52.355 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:52.355 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.341+0000 7faa49608640 10 monclient: _renew_subs 2024-09-11T01:52:52.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.341+0000 7faa49608640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:52:52.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.341+0000 7faa49608640 1 -- 172.21.15.12:0/1096130371 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa44137ca0 con 0x7faa440a0d70 2024-09-11T01:52:52.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.342+0000 7faa49608640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:52.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.342+0000 7faa3b7fe640 1 -- 172.21.15.12:0/1096130371 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7faa3401b3b0 con 0x7faa440a0d70 2024-09-11T01:52:52.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.342+0000 7faa49608640 1 --2- 172.21.15.12:0/1096130371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa44134300 0x7faa4402d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:52.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.342+0000 7faa49608640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:52.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.342+0000 7faa49608640 1 -- 172.21.15.12:0/1096130371 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7faa4413e1e0 con 0x7faa44134300 2024-09-11T01:52:52.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.342+0000 7faa3bfff640 1 --2- 172.21.15.12:0/1096130371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa44134300 0x7faa4402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:52.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.342+0000 7faa3bfff640 10 monclient: get_auth_request con 0x7faa44134300 auth_method 0 2024-09-11T01:52:52.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.342+0000 7faa3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:52.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.342+0000 7faa3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:52.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.342+0000 7faa3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:52.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.342+0000 7faa3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:52.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.342+0000 7faa3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:52.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.342+0000 7faa3b7fe640 1 --2- 172.21.15.12:0/1096130371 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7faa2403e260 0x7faa24040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:52.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.342+0000 7faa3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:52.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.342+0000 7faa3b7fe640 1 -- 172.21.15.12:0/1096130371 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(801..801 src has 676..801) ==== 4787+0+0 (secure 0 0 0) 0x7faa34052330 con 0x7faa440a0d70 2024-09-11T01:52:52.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.343+0000 7faa3bfff640 10 monclient(hunting): handle_auth_done global_id 264108 payload 824 2024-09-11T01:52:52.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.343+0000 7faa3bfff640 1 -- 172.21.15.12:0/1096130371 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faa44131af0 con 0x7faa44134300 2024-09-11T01:52:52.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.343+0000 7faa437fe640 1 --2- 172.21.15.12:0/1096130371 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7faa2403e260 0x7faa24040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:52.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.343+0000 7faa437fe640 10 monclient: get_auth_request con 0x7faa2403e260 auth_method 0 2024-09-11T01:52:52.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.345+0000 7faa3bfff640 1 -- 172.21.15.12:0/1096130371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa44134300 msgr2=0x7faa4402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:52.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.345+0000 7faa3bfff640 1 --2- 172.21.15.12:0/1096130371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa44134300 0x7faa4402d440 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faa2c033000 tx=0x7faa2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:52.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.345+0000 7faa3b7fe640 1 -- 172.21.15.12:0/1096130371 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2c036040 con 0x7faa44134300 2024-09-11T01:52:52.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.345+0000 7faa3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:52.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.345+0000 7faa3b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:52:52.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.345+0000 7faa3b7fe640 10 monclient: dump: 2024-09-11T01:52:52.358 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:52.358 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:52.358 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.358 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.358 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:52.358 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.358 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:52.358 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:52.358 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:52.358 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.345+0000 7faa3b7fe640 1 -- 172.21.15.12:0/1096130371 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7faa2c009490 con 0x7faa44134300 2024-09-11T01:52:52.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.345+0000 7faa3b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:52.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.345+0000 7faa3b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:52.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.345+0000 7faa3b7fe640 1 -- 172.21.15.12:0/1096130371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa44134300 msgr2=0x7faa4402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.345+0000 7faa3b7fe640 1 --2- 172.21.15.12:0/1096130371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa44134300 0x7faa4402d440 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faa2c033000 tx=0x7faa2c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.345+0000 7faa3b7fe640 1 -- 172.21.15.12:0/1096130371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa44134300 msgr2=0x7faa4402d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:52.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.345+0000 7faa3b7fe640 1 --2- 172.21.15.12:0/1096130371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa44134300 0x7faa4402d440 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faa2c033000 tx=0x7faa2c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.345+0000 7faa437fe640 1 --2- 172.21.15.12:0/1096130371 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7faa2403e260 0x7faa24040720 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7faa44077230 tx=0x7faa3c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:52.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.352+0000 7faa49608640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:52:52.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.352+0000 7faa49608640 1 --2- 172.21.15.12:0/1096130371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa441346e0 0x7faa4402d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:52.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.352+0000 7faa49608640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:52.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.352+0000 7faa49608640 1 -- 172.21.15.12:0/1096130371 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7faa4409d130 con 0x7faa441346e0 2024-09-11T01:52:52.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.352+0000 7faa3bfff640 1 --2- 172.21.15.12:0/1096130371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa441346e0 0x7faa4402d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:52.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.352+0000 7faa3bfff640 10 monclient: get_auth_request con 0x7faa441346e0 auth_method 0 2024-09-11T01:52:52.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.352+0000 7faa3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:52.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.352+0000 7faa3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:52.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.352+0000 7faa3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:52.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.352+0000 7faa3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:52.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.352+0000 7faa3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:52.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.352+0000 7faa3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:52.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.352+0000 7faa3bfff640 10 monclient(hunting): handle_auth_done global_id 264111 payload 824 2024-09-11T01:52:52.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.352+0000 7faa3bfff640 1 -- 172.21.15.12:0/1096130371 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faa4406dc80 con 0x7faa441346e0 2024-09-11T01:52:52.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.354+0000 7faa3bfff640 1 -- 172.21.15.12:0/1096130371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa441346e0 msgr2=0x7faa4402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:52.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.354+0000 7faa3bfff640 1 --2- 172.21.15.12:0/1096130371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa441346e0 0x7faa4402d980 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faa2c009970 tx=0x7faa2c009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:52.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.355+0000 7faa3b7fe640 1 -- 172.21.15.12:0/1096130371 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2c036040 con 0x7faa441346e0 2024-09-11T01:52:52.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.355+0000 7faa3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:52.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.355+0000 7faa3b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:52:52.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.355+0000 7faa3b7fe640 10 monclient: dump: 2024-09-11T01:52:52.362 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:52.362 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:52.362 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.362 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.362 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:52.362 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.362 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:52.362 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:52.363 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:52.363 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.355+0000 7faa3b7fe640 1 -- 172.21.15.12:0/1096130371 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7faa2c048c70 con 0x7faa441346e0 2024-09-11T01:52:52.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.355+0000 7faa3b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:52:52.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.355+0000 7faa3b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:52.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.355+0000 7faa3b7fe640 1 -- 172.21.15.12:0/1096130371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa441346e0 msgr2=0x7faa4402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.355+0000 7faa3b7fe640 1 --2- 172.21.15.12:0/1096130371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa441346e0 0x7faa4402d980 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faa2c009970 tx=0x7faa2c009240 comp rx=0 tx=0).stop 2024-09-11T01:52:52.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.355+0000 7faa3b7fe640 1 -- 172.21.15.12:0/1096130371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa441346e0 msgr2=0x7faa4402d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:52.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.355+0000 7faa3b7fe640 1 --2- 172.21.15.12:0/1096130371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa441346e0 0x7faa4402d980 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faa2c009970 tx=0x7faa2c009240 comp rx=0 tx=0).stop 2024-09-11T01:52:52.363 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:52:52.364 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:52:52.364 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:52:52.364 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:52:52.364 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 220, 2024-09-11T01:52:52.364 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 14061, 2024-09-11T01:52:52.364 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:52:52.364 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:52:52.364 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:52:52.364 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:52:52.364 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:52.364 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:52:52.364 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:52:52.364 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:52:52.364 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:52:52.365 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:52.365 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:52.365 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:52.365 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:52.365 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:52.365 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:52.365 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:52.365 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:52.365 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:52.365 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:52.365 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:52.365 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:52.365 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:52:52.366 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:52:52.366 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:52.366 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:52.366 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:52.366 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:52.366 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:52.366 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:52.366 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:52.366 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:52.366 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:52.366 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:52.366 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:52.366 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:52.366 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.367 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:52:52.367 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:52:52.367 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:52:52.367 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:52:52.367 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:52:52.367 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:52:52.367 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:52:52.367 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:52:52.367 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:52:52.367 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:52:52.367 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:52:52.367 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:52:52.367 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:52:52.367 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:52:52.368 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:52:52.368 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:52:52.368 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:52:52.368 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:52.368 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:52.368 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:52.368 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:52.368 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:52.368 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:52.368 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:52.368 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:52.368 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:52.368 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:52.369 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:52.369 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:52.369 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:52:52.369 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.369 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:52:52.369 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.369 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:52:52.369 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:52:52.369 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:52.369 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:52.369 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.369 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:52.369 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:52:52.369 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:52.370 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.370 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.370 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:52.370 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:52:52.370 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:52.370 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:52.370 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:52.370 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.370 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:52:52.370 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:52:52.370 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:52.370 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:52.370 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:52.370 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.371 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.371 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:52:52.371 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:52:52.371 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:52.371 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:52.371 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.371 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:52.371 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:52:52.371 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:52.371 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.371 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.371 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:52.371 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:52:52.371 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:52.372 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:52.372 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:52.372 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.372 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:52:52.372 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:52:52.372 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:52.372 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:52.372 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:52.372 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.372 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.372 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:52:52.372 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:52:52.372 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:52.372 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:52.372 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.373 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:52.373 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:52:52.373 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:52.373 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.373 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.373 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:52.373 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:52:52.373 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:52.373 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:52.373 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:52.373 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.373 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:52:52.373 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:52:52.373 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:52.374 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:52.374 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:52.374 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:52.374 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:52.374 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.374 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:52:52.374 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:52:52.374 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.374 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:52.374 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:52.374 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:52:52.374 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:52.374 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:52.374 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:52:52.374 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.375 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:52.375 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:52.375 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:52:52.375 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:52.375 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:52.375 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.375 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:52:52.375 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:52:52.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.355+0000 7faa49608640 1 -- 172.21.15.12:0/1096130371 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7faa2403e260 msgr2=0x7faa24040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.355+0000 7faa49608640 1 --2- 172.21.15.12:0/1096130371 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7faa2403e260 0x7faa24040720 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7faa44077230 tx=0x7faa3c009040 comp rx=0 tx=0).stop 2024-09-11T01:52:52.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.355+0000 7faa49608640 10 monclient: shutdown 2024-09-11T01:52:52.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.355+0000 7faa49608640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:52.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.355+0000 7faa49608640 1 -- 172.21.15.12:0/1096130371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa440a0d70 msgr2=0x7faa44130900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.355+0000 7faa49608640 1 --2- 172.21.15.12:0/1096130371 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa440a0d70 0x7faa44130900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7faa3400c490 tx=0x7faa3400c950 comp rx=0 tx=0).stop 2024-09-11T01:52:52.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.355+0000 7faa437fe640 1 -- 172.21.15.12:0/1096130371 reap_dead start 2024-09-11T01:52:52.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.355+0000 7faa437fe640 1 -- 172.21.15.12:0/1096130371 reap_dead start 2024-09-11T01:52:52.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.355+0000 7faa49608640 1 -- 172.21.15.12:0/1096130371 shutdown_connections 2024-09-11T01:52:52.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.355+0000 7faa49608640 1 -- 172.21.15.12:0/1096130371 >> 172.21.15.12:0/1096130371 conn(0x7faa440a36c0 msgr2=0x7faa4413dad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:52.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.355+0000 7faa49608640 1 -- 172.21.15.12:0/1096130371 shutdown_connections 2024-09-11T01:52:52.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.355+0000 7faa49608640 1 -- 172.21.15.12:0/1096130371 wait complete. 2024-09-11T01:52:52.377 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:52:52.378 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:52:52.378 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:52:52.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.477+0000 7f722612b640 10 monclient: get_monmap_and_config 2024-09-11T01:52:52.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.477+0000 7f722612b640 10 monclient: build_initial_monmap 2024-09-11T01:52:52.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.477+0000 7f722612b640 10 monclient: monmap: 2024-09-11T01:52:52.479 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:52.479 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:52.478690+0000 2024-09-11T01:52:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:52.478690+0000 2024-09-11T01:52:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:52.480 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.478+0000 7f722612b640 1 Processor -- start 2024-09-11T01:52:52.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.478+0000 7f722612b640 1 -- start start 2024-09-11T01:52:52.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.478+0000 7f722612b640 10 monclient: init 2024-09-11T01:52:52.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.479+0000 7f722612b640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:52.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.479+0000 7f722612b640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-11T01:52:52.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.479+0000 7f722612b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f72200a2d40 0x7f722006cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:52.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.479+0000 7f722612b640 10 monclient(hunting): picked mon.noname-a con 0x7f72200a2d40 addr v2:172.21.15.12:3300/0 2024-09-11T01:52:52.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.479+0000 7f722612b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f722006d090 addr v1:172.21.15.71:6789/0 2024-09-11T01:52:52.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.479+0000 7f722612b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f722006d4a0 addr v1:172.21.15.12:6789/0 2024-09-11T01:52:52.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.479+0000 7f722612b640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f722009aed0 con 0x7f722006d4a0 2024-09-11T01:52:52.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.479+0000 7f722612b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72200a3920 con 0x7f722006d090 2024-09-11T01:52:52.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.479+0000 7f722612b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:52.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.479+0000 7f722612b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f722006dd10 con 0x7f72200a2d40 2024-09-11T01:52:52.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.479+0000 7f722612b640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:52.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.479+0000 7f722612b640 10 monclient(hunting): authenticate will time out at 2178.921323s 2024-09-11T01:52:52.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.479+0000 7f721f7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f72200a2d40 0x7f722006cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:52.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.479+0000 7f721f7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f72200a2d40 0x7f722006cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60016/0 (socket says 172.21.15.12:60016) 2024-09-11T01:52:52.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.479+0000 7f721ffff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f722006d4a0 0x7f722013fc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59138/0 (socket says 172.21.15.12:59138) 2024-09-11T01:52:52.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.479+0000 7f721f7fe640 1 -- 172.21.15.12:0/2914215078 learned_addr learned my addr 172.21.15.12:0/2914215078 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:52.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.479+0000 7f721f7fe640 10 monclient(hunting): get_auth_request con 0x7f72200a2d40 auth_method 0 2024-09-11T01:52:52.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.479+0000 7f721f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:52.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.479+0000 7f721f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:52.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.479+0000 7f721f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:52.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.481+0000 7f721ffff640 1 -- 172.21.15.12:0/2914215078 >> v1:172.21.15.12:6789/0 conn(0x7f722006d4a0 legacy=0x7f722013fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:52.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.481+0000 7f721f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:52.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.481+0000 7f721f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:52.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.481+0000 7f721f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:52.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.481+0000 7f721effd640 1 -- 172.21.15.12:0/2914215078 >> v1:172.21.15.71:6789/0 conn(0x7f722006d090 legacy=0x7f722013c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:52.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.481+0000 7f721f7fe640 10 monclient(hunting): handle_auth_done global_id 264114 payload 274 2024-09-11T01:52:52.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.481+0000 7f721f7fe640 1 -- 172.21.15.12:0/2914215078 >> v1:172.21.15.71:6789/0 conn(0x7f722006d090 legacy=0x7f722013c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.481+0000 7f721f7fe640 1 -- 172.21.15.12:0/2914215078 >> v1:172.21.15.12:6789/0 conn(0x7f722006d4a0 legacy=0x7f722013fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.481+0000 7f721f7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:52:52.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.481+0000 7f721f7fe640 1 monclient: found mon.noname-a 2024-09-11T01:52:52.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.481+0000 7f721f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:52.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.481+0000 7f721f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:52:52.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.481+0000 7f721f7fe640 1 -- 172.21.15.12:0/2914215078 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72200a3320 con 0x7f72200a2d40 2024-09-11T01:52:52.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.484+0000 7f721f7fe640 1 -- 172.21.15.12:0/2914215078 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72200a2d40 msgr2=0x7f722006cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:52.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.484+0000 7f721f7fe640 1 --2- 172.21.15.12:0/2914215078 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72200a2d40 0x7f722006cb50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7210009970 tx=0x7f721002a2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17a4ac7019db5b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:52.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.484+0000 7f721e7fc640 1 -- 172.21.15.12:0/2914215078 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7210037070 con 0x7f72200a2d40 2024-09-11T01:52:52.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.484+0000 7f721e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:52.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.484+0000 7f721e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:52:52.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.484+0000 7f721e7fc640 10 monclient: dump: 2024-09-11T01:52:52.487 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:52.488 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:52.488 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.488 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.488 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:52.488 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.488 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:52.488 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:52.488 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:52.488 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.484+0000 7f721e7fc640 10 monclient: _finish_auth 0 2024-09-11T01:52:52.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.484+0000 7f721e7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:52:52.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.484+0000 7f721e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:52.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.484+0000 7f721e7fc640 1 -- 172.21.15.12:0/2914215078 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7210032640 con 0x7f72200a2d40 2024-09-11T01:52:52.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.484+0000 7f721e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:52.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.484+0000 7f721e7fc640 1 -- 172.21.15.12:0/2914215078 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7210032a10 con 0x7f72200a2d40 2024-09-11T01:52:52.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.484+0000 7f722612b640 5 monclient: authenticate success, global_id 264114 2024-09-11T01:52:52.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.484+0000 7f722612b640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:52.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.484+0000 7f722612b640 4 set_mon_vals no callback set 2024-09-11T01:52:52.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.484+0000 7f721e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:52.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.484+0000 7f721e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:52:52.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.484+0000 7f721e7fc640 10 monclient: dump: 2024-09-11T01:52:52.489 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:52.489 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:52.489 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.489 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.489 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:52.489 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.489 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:52.490 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:52.490 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:52.490 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.485+0000 7f722612b640 10 monclient: shutdown 2024-09-11T01:52:52.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.485+0000 7f722612b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:52.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.485+0000 7f722612b640 1 -- 172.21.15.12:0/2914215078 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72200a2d40 msgr2=0x7f722006cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.485+0000 7f722612b640 1 --2- 172.21.15.12:0/2914215078 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72200a2d40 0x7f722006cb50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7210009970 tx=0x7f721002a2e0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.485+0000 7f722612b640 1 -- 172.21.15.12:0/2914215078 shutdown_connections 2024-09-11T01:52:52.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.485+0000 7f722612b640 1 --2- 172.21.15.12:0/2914215078 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72200a2d40 0x7f722006cb50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.485+0000 7f722612b640 1 -- 172.21.15.12:0/2914215078 >> 172.21.15.12:0/2914215078 conn(0x7f72200a3e30 msgr2=0x7f72200a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:52.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.485+0000 7f722612b640 1 -- 172.21.15.12:0/2914215078 shutdown_connections 2024-09-11T01:52:52.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.485+0000 7f722612b640 1 -- 172.21.15.12:0/2914215078 wait complete. 2024-09-11T01:52:52.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.486+0000 7f722612b640 10 monclient: build_initial_monmap 2024-09-11T01:52:52.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.486+0000 7f722612b640 10 monclient: monmap: 2024-09-11T01:52:52.491 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:52.491 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:52.491 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:52.491 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:52.491 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:52.491 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.491 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:52.491 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:52.491 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:52.491 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.486+0000 7f722612b640 1 Processor -- start 2024-09-11T01:52:52.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.486+0000 7f722612b640 1 -- start start 2024-09-11T01:52:52.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.486+0000 7f722612b640 10 monclient: init 2024-09-11T01:52:52.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f722612b640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:52.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f722612b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:52:52.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f722612b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f722006d090 0x7f7220099400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:52.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f722612b640 10 monclient(hunting): picked mon.noname-c con 0x7f722006d090 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:52.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f722612b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f722006d4a0 0x7f7220093df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:52.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f722612b640 10 monclient(hunting): picked mon.noname-a con 0x7f722006d4a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:52.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f722612b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72200a2d40 0x7f7220094330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:52.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f722612b640 10 monclient(hunting): picked mon.noname-b con 0x7f72200a2d40 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:52.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f722612b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:52.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f722612b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f722009ba10 con 0x7f722006d4a0 2024-09-11T01:52:52.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f722612b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:52.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f722612b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f72200a3120 con 0x7f72200a2d40 2024-09-11T01:52:52.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f722612b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:52.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f722612b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f722009b3a0 con 0x7f722006d090 2024-09-11T01:52:52.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f722612b640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:52.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f722612b640 10 monclient(hunting): authenticate will time out at 2178.929148s 2024-09-11T01:52:52.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721effd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f722006d4a0 0x7f7220093df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:52.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721f7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f722006d090 0x7f7220099400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:52.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721f7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f722006d090 0x7f7220099400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:43626/0 (socket says 172.21.15.12:43626) 2024-09-11T01:52:52.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721effd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f722006d4a0 0x7f7220093df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60018/0 (socket says 172.21.15.12:60018) 2024-09-11T01:52:52.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721f7fe640 1 -- 172.21.15.12:0/3157895397 learned_addr learned my addr 172.21.15.12:0/3157895397 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:52.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721f7fe640 10 monclient(hunting): get_auth_request con 0x7f722006d090 auth_method 0 2024-09-11T01:52:52.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:52.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:52.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:52.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721effd640 10 monclient(hunting): get_auth_request con 0x7f722006d4a0 auth_method 0 2024-09-11T01:52:52.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:52.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:52.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:52.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:52.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:52.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:52.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:52.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:52.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:52.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721ffff640 1 --2- 172.21.15.12:0/3157895397 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72200a2d40 0x7f7220094330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:52.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721ffff640 10 monclient(hunting): get_auth_request con 0x7f72200a2d40 auth_method 0 2024-09-11T01:52:52.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:52.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:52.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:52.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721f7fe640 10 monclient(hunting): handle_auth_done global_id 244319 payload 824 2024-09-11T01:52:52.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721f7fe640 1 -- 172.21.15.12:0/3157895397 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72200a2d40 msgr2=0x7f7220094330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721f7fe640 1 --2- 172.21.15.12:0/3157895397 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72200a2d40 0x7f7220094330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721f7fe640 1 -- 172.21.15.12:0/3157895397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f722006d4a0 msgr2=0x7f7220093df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721f7fe640 1 --2- 172.21.15.12:0/3157895397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f722006d4a0 0x7f7220093df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721f7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:52:52.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721f7fe640 1 monclient: found mon.noname-c 2024-09-11T01:52:52.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:52.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:52:52.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721f7fe640 1 -- 172.21.15.12:0/3157895397 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f722009a100 con 0x7f722006d090 2024-09-11T01:52:52.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.487+0000 7f721effd640 1 --2- 172.21.15.12:0/3157895397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f722006d4a0 0x7f7220093df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:52:52.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.489+0000 7f721f7fe640 1 -- 172.21.15.12:0/3157895397 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f722006d090 msgr2=0x7f7220099400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:52.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.489+0000 7f721f7fe640 1 --2- 172.21.15.12:0/3157895397 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f722006d090 0x7f7220099400 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f72100090b0 tx=0x7f72100046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:52.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.489+0000 7f7217fff640 1 -- 172.21.15.12:0/3157895397 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7210041030 con 0x7f722006d090 2024-09-11T01:52:52.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.489+0000 7f7217fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:52.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.489+0000 7f7217fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:52.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.489+0000 7f7217fff640 10 monclient: dump: 2024-09-11T01:52:52.501 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:52.501 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:52.501 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.501 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.501 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:52.501 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.501 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:52.501 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:52.501 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:52.501 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.490+0000 7f7217fff640 10 monclient: _finish_auth 0 2024-09-11T01:52:52.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.490+0000 7f7217fff640 10 monclient: _check_auth_tickets 2024-09-11T01:52:52.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.490+0000 7f7217fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:52.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.490+0000 7f722612b640 5 monclient: authenticate success, global_id 244319 2024-09-11T01:52:52.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.490+0000 7f7217fff640 1 -- 172.21.15.12:0/3157895397 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f721002f780 con 0x7f722006d090 2024-09-11T01:52:52.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.490+0000 7f7217fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:52.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.490+0000 7f7217fff640 1 -- 172.21.15.12:0/3157895397 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7210037040 con 0x7f722006d090 2024-09-11T01:52:52.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.490+0000 7f722612b640 10 monclient: _renew_subs 2024-09-11T01:52:52.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.490+0000 7f722612b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:52.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.490+0000 7f722612b640 1 -- 172.21.15.12:0/3157895397 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72200958b0 con 0x7f722006d090 2024-09-11T01:52:52.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.490+0000 7f7217fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:52.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.490+0000 7f721d7fa640 4 set_mon_vals no callback set 2024-09-11T01:52:52.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.490+0000 7f7217fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:52.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.490+0000 7f7217fff640 10 monclient: dump: 2024-09-11T01:52:52.502 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:52.502 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:52.502 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.502 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.503 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:52.503 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.503 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:52.503 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:52.503 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:52.503 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.490+0000 7f722612b640 10 monclient: _renew_subs 2024-09-11T01:52:52.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.490+0000 7f722612b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:52.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.490+0000 7f722612b640 1 -- 172.21.15.12:0/3157895397 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7220097370 con 0x7f722006d090 2024-09-11T01:52:52.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.490+0000 7f722612b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:52.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.490+0000 7f722612b640 1 --2- 172.21.15.12:0/3157895397 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7220099940 0x7f7220094870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:52.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.490+0000 7f722612b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:52.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.490+0000 7f722612b640 1 -- 172.21.15.12:0/3157895397 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f72200977e0 con 0x7f7220099940 2024-09-11T01:52:52.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.491+0000 7f721effd640 1 --2- 172.21.15.12:0/3157895397 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7220099940 0x7f7220094870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:52.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.491+0000 7f721effd640 10 monclient: get_auth_request con 0x7f7220099940 auth_method 0 2024-09-11T01:52:52.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.491+0000 7f721effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:52.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.491+0000 7f721effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:52.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.491+0000 7f721effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:52.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.491+0000 7f7217fff640 1 -- 172.21.15.12:0/3157895397 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f721003a030 con 0x7f722006d090 2024-09-11T01:52:52.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.491+0000 7f721effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:52.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.491+0000 7f721effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:52.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.491+0000 7f721effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:52.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.491+0000 7f721effd640 10 monclient(hunting): handle_auth_done global_id 244322 payload 824 2024-09-11T01:52:52.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.491+0000 7f721effd640 1 -- 172.21.15.12:0/3157895397 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f722006cbd0 con 0x7f7220099940 2024-09-11T01:52:52.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.491+0000 7f7217fff640 1 --2- 172.21.15.12:0/3157895397 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f720003e210 0x7f72000406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:52.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.491+0000 7f7217fff640 1 -- 172.21.15.12:0/3157895397 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(801..801 src has 676..801) ==== 4787+0+0 (secure 0 0 0) 0x7f72100730f0 con 0x7f722006d090 2024-09-11T01:52:52.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.491+0000 7f721ffff640 1 --2- 172.21.15.12:0/3157895397 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f720003e210 0x7f72000406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:52.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.491+0000 7f721ffff640 10 monclient: get_auth_request con 0x7f720003e210 auth_method 0 2024-09-11T01:52:52.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.493+0000 7f721effd640 1 -- 172.21.15.12:0/3157895397 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7220099940 msgr2=0x7f7220094870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:52.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.493+0000 7f721effd640 1 --2- 172.21.15.12:0/3157895397 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7220099940 0x7f7220094870 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7208000f60 tx=0x7f720800fea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:52.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.493+0000 7f7217fff640 1 -- 172.21.15.12:0/3157895397 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72080074f0 con 0x7f7220099940 2024-09-11T01:52:52.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.493+0000 7f7217fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:52.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.493+0000 7f7217fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:52.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.493+0000 7f7217fff640 10 monclient: dump: 2024-09-11T01:52:52.505 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:52.505 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:52.505 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.505 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.505 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:52.505 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.505 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:52.505 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:52.506 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:52.506 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.493+0000 7f7217fff640 1 -- 172.21.15.12:0/3157895397 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7208018070 con 0x7f7220099940 2024-09-11T01:52:52.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.493+0000 7f7217fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:52.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.493+0000 7f7217fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:52.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.493+0000 7f7217fff640 1 -- 172.21.15.12:0/3157895397 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7220099940 msgr2=0x7f7220094870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.493+0000 7f7217fff640 1 --2- 172.21.15.12:0/3157895397 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7220099940 0x7f7220094870 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7208000f60 tx=0x7f720800fea0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.493+0000 7f7217fff640 1 -- 172.21.15.12:0/3157895397 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7220099940 msgr2=0x7f7220094870 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:52.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.493+0000 7f7217fff640 1 --2- 172.21.15.12:0/3157895397 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7220099940 0x7f7220094870 secure :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7208000f60 tx=0x7f720800fea0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.494+0000 7f721ffff640 1 --2- 172.21.15.12:0/3157895397 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f720003e210 0x7f72000406d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7218005fd0 tx=0x7f7218007e60 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:52.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.500+0000 7f722612b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:52:52.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.500+0000 7f722612b640 1 --2- 172.21.15.12:0/3157895397 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72200913c0 0x7f72200917d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:52.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.500+0000 7f722612b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:52.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.500+0000 7f722612b640 1 -- 172.21.15.12:0/3157895397 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7220097040 con 0x7f72200913c0 2024-09-11T01:52:52.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.500+0000 7f721effd640 1 --2- 172.21.15.12:0/3157895397 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72200913c0 0x7f72200917d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:52.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.500+0000 7f721effd640 10 monclient: get_auth_request con 0x7f72200913c0 auth_method 0 2024-09-11T01:52:52.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.500+0000 7f721effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:52.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.500+0000 7f721effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:52.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.500+0000 7f721effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:52.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.500+0000 7f721effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:52.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.500+0000 7f721effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:52.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.500+0000 7f721effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:52.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.500+0000 7f721effd640 10 monclient(hunting): handle_auth_done global_id 244325 payload 824 2024-09-11T01:52:52.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.500+0000 7f721effd640 1 -- 172.21.15.12:0/3157895397 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7220096410 con 0x7f72200913c0 2024-09-11T01:52:52.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.502+0000 7f721effd640 1 -- 172.21.15.12:0/3157895397 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72200913c0 msgr2=0x7f72200917d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:52.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.502+0000 7f721effd640 1 --2- 172.21.15.12:0/3157895397 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72200913c0 0x7f72200917d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7208017000 tx=0x7f7208001410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:52.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.502+0000 7f7217fff640 1 -- 172.21.15.12:0/3157895397 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7208005be0 con 0x7f72200913c0 2024-09-11T01:52:52.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.502+0000 7f7217fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:52.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.502+0000 7f7217fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:52.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.502+0000 7f7217fff640 10 monclient: dump: 2024-09-11T01:52:52.510 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:52.510 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:52.510 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.510 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.510 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:52.512 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.512 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:52.512 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:52.512 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:52.513 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.503+0000 7f7217fff640 1 -- 172.21.15.12:0/3157895397 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f7208005be0 con 0x7f72200913c0 2024-09-11T01:52:52.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.503+0000 7f7217fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:52:52.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.503+0000 7f7217fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:52.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.503+0000 7f7217fff640 1 -- 172.21.15.12:0/3157895397 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72200913c0 msgr2=0x7f72200917d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.503+0000 7f7217fff640 1 --2- 172.21.15.12:0/3157895397 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72200913c0 0x7f72200917d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7208017000 tx=0x7f7208001410 comp rx=0 tx=0).stop 2024-09-11T01:52:52.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.503+0000 7f7217fff640 1 -- 172.21.15.12:0/3157895397 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72200913c0 msgr2=0x7f72200917d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:52.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.503+0000 7f7217fff640 1 --2- 172.21.15.12:0/3157895397 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72200913c0 0x7f72200917d0 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7208017000 tx=0x7f7208001410 comp rx=0 tx=0).stop 2024-09-11T01:52:52.513 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:52:52.514 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:52:52.514 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:52:52.514 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:52:52.514 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 220, 2024-09-11T01:52:52.514 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 125951, 2024-09-11T01:52:52.514 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:52:52.514 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:52:52.514 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:52:52.514 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:52:52.514 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:52.514 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:52:52.514 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:52:52.514 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:52:52.514 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:52:52.515 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:52.515 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:52.515 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:52.515 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:52.515 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:52.515 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:52.515 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:52.515 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:52.515 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:52.515 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:52.515 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:52.515 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:52.515 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:52:52.515 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:52:52.516 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:52.516 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:52.516 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:52.516 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:52.516 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:52.516 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:52.516 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:52.516 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:52.516 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:52.516 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:52.516 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:52.516 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:52.516 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.516 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:52:52.517 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:52:52.517 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:52:52.517 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:52:52.517 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:52:52.517 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:52:52.517 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:52:52.517 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:52:52.517 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:52:52.517 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:52:52.517 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:52:52.517 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:52:52.517 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:52:52.517 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:52:52.517 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:52:52.518 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:52:52.518 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:52:52.518 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:52.518 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:52.518 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:52.518 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:52.518 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:52.518 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:52.518 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:52.518 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:52.518 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:52.518 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:52.518 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:52.519 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:52.519 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:52:52.519 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.519 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:52:52.519 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.519 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:52:52.519 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:52:52.519 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:52.519 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:52.519 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.519 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:52.519 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:52:52.519 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:52.519 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.520 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.520 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:52.520 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:52:52.520 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:52.520 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:52.520 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:52.520 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.520 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:52:52.520 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:52:52.520 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:52.520 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:52.520 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:52.520 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.520 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.521 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:52:52.521 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:52:52.521 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:52.521 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:52.521 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.521 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:52.521 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:52:52.521 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:52.521 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.521 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.521 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:52.521 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:52:52.521 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:52.522 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:52.522 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:52.522 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.522 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:52:52.522 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:52:52.522 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:52.522 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:52.522 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:52.522 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.522 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.522 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:52:52.522 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:52:52.522 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:52.522 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:52.523 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.523 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:52.523 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:52:52.523 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:52.523 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.523 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.523 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:52.523 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:52:52.523 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:52.523 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:52.523 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:52.523 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.523 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:52:52.524 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:52:52.524 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:52.524 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:52.524 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:52.524 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:52.524 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:52.524 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.524 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:52:52.524 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:52:52.524 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.524 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:52.524 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:52.524 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:52:52.525 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:52.525 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:52.525 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:52:52.525 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.525 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:52.525 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:52.525 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:52:52.525 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:52.525 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:52.525 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:52:52.525 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.525 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:52.525 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:52.525 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:52:52.526 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:52.526 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:52.526 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:52:52.526 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.526 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:52.526 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:52.526 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:52:52.526 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:52.526 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:52.526 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.526 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:52:52.526 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:52:52.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.503+0000 7f722612b640 1 -- 172.21.15.12:0/3157895397 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f720003e210 msgr2=0x7f72000406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.503+0000 7f722612b640 1 --2- 172.21.15.12:0/3157895397 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f720003e210 0x7f72000406d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7218005fd0 tx=0x7f7218007e60 comp rx=0 tx=0).stop 2024-09-11T01:52:52.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.503+0000 7f722612b640 10 monclient: shutdown 2024-09-11T01:52:52.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.503+0000 7f722612b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:52.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.503+0000 7f722612b640 1 -- 172.21.15.12:0/3157895397 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f722006d090 msgr2=0x7f7220099400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.503+0000 7f722612b640 1 --2- 172.21.15.12:0/3157895397 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f722006d090 0x7f7220099400 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f72100090b0 tx=0x7f72100046c0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.503+0000 7f721ffff640 1 -- 172.21.15.12:0/3157895397 reap_dead start 2024-09-11T01:52:52.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.503+0000 7f721ffff640 1 -- 172.21.15.12:0/3157895397 reap_dead start 2024-09-11T01:52:52.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.503+0000 7f722612b640 1 -- 172.21.15.12:0/3157895397 shutdown_connections 2024-09-11T01:52:52.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.503+0000 7f722612b640 1 -- 172.21.15.12:0/3157895397 >> 172.21.15.12:0/3157895397 conn(0x7f72200a3e30 msgr2=0x7f722009f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:52.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.503+0000 7f722612b640 1 -- 172.21.15.12:0/3157895397 shutdown_connections 2024-09-11T01:52:52.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.503+0000 7f722612b640 1 -- 172.21.15.12:0/3157895397 wait complete. 2024-09-11T01:52:52.529 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:52:52.529 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:52:52.529 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:52:52.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.629+0000 7fbfdfaed640 10 monclient: get_monmap_and_config 2024-09-11T01:52:52.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.629+0000 7fbfdfaed640 10 monclient: build_initial_monmap 2024-09-11T01:52:52.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.630+0000 7fbfdfaed640 10 monclient: monmap: 2024-09-11T01:52:52.631 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:52.632 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:52.632 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:52.630725+0000 2024-09-11T01:52:52.632 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:52.630725+0000 2024-09-11T01:52:52.632 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:52.632 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.632 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:52.632 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:52.632 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:52.632 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:52.632 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:52.632 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.630+0000 7fbfdfaed640 1 Processor -- start 2024-09-11T01:52:52.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.630+0000 7fbfdfaed640 1 -- start start 2024-09-11T01:52:52.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.630+0000 7fbfdfaed640 10 monclient: init 2024-09-11T01:52:52.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdfaed640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:52.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdfaed640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-11T01:52:52.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdfaed640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfd81381c0 0x7fbfd81385a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:52.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdfaed640 10 monclient(hunting): picked mon.noname-b con 0x7fbfd81381c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:52.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdfaed640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbfd80a6aa0 addr v1:172.21.15.12:6789/0 2024-09-11T01:52:52.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdfaed640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbfd813d000 addr v1:172.21.15.71:6789/0 2024-09-11T01:52:52.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdfaed640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfd809c310 con 0x7fbfd80a6aa0 2024-09-11T01:52:52.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdfaed640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfd809c8d0 con 0x7fbfd813d000 2024-09-11T01:52:52.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdfaed640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:52.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdfaed640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fbfd806fc90 con 0x7fbfd81381c0 2024-09-11T01:52:52.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdfaed640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:52.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdfaed640 10 monclient(hunting): authenticate will time out at 2179.073439s 2024-09-11T01:52:52.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdd889640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfd81381c0 0x7fbfd81385a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:52.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdd088640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbfd80a6aa0 0x7fbfd809cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59140/0 (socket says 172.21.15.12:59140) 2024-09-11T01:52:52.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdd088640 1 -- 172.21.15.12:0/3571407740 learned_addr learned my addr 172.21.15.12:0/3571407740 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:52:52.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdd889640 10 monclient(hunting): get_auth_request con 0x7fbfd81381c0 auth_method 0 2024-09-11T01:52:52.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdd889640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:52.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdd889640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:52.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdd889640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:52.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdd889640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:52.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdd889640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:52.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdd889640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:52.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdd889640 10 monclient(hunting): handle_auth_done global_id 244328 payload 274 2024-09-11T01:52:52.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdd889640 1 -- 172.21.15.12:0/3571407740 >> v1:172.21.15.71:6789/0 conn(0x7fbfd813d000 legacy=0x7fbfd813f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:52.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdd889640 1 -- 172.21.15.12:0/3571407740 >> v1:172.21.15.12:6789/0 conn(0x7fbfd80a6aa0 legacy=0x7fbfd809cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:52.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdd889640 10 monclient: _finish_hunting 0 2024-09-11T01:52:52.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdd889640 1 monclient: found mon.noname-b 2024-09-11T01:52:52.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdd889640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:52.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdd889640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:52:52.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.631+0000 7fbfdd889640 1 -- 172.21.15.12:0/3571407740 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd8138ae0 con 0x7fbfd81381c0 2024-09-11T01:52:52.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.634+0000 7fbfdd889640 1 -- 172.21.15.12:0/3571407740 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfd81381c0 msgr2=0x7fbfd81385a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:52.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.634+0000 7fbfdd889640 1 --2- 172.21.15.12:0/3571407740 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfd81381c0 0x7fbfd81385a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbfd40097d0 tx=0x7fbfd402a030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4d2be3dbc47d0a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:52.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.634+0000 7fbfdc887640 1 -- 172.21.15.12:0/3571407740 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4037070 con 0x7fbfd81381c0 2024-09-11T01:52:52.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.634+0000 7fbfdc887640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:52.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.634+0000 7fbfdc887640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:52:52.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.634+0000 7fbfdc887640 10 monclient: dump: 2024-09-11T01:52:52.644 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:52.644 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:52.644 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.644 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.644 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:52.644 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.644 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:52.644 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:52.644 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:52.644 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.634+0000 7fbfdc887640 10 monclient: _finish_auth 0 2024-09-11T01:52:52.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.634+0000 7fbfdc887640 10 monclient: _check_auth_tickets 2024-09-11T01:52:52.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.634+0000 7fbfdc887640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:52.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.634+0000 7fbfdc887640 1 -- 172.21.15.12:0/3571407740 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfd4032640 con 0x7fbfd81381c0 2024-09-11T01:52:52.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.634+0000 7fbfdfaed640 5 monclient: authenticate success, global_id 244328 2024-09-11T01:52:52.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.634+0000 7fbfdfaed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:52:52.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.634+0000 7fbfdc887640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:52.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.634+0000 7fbfdfaed640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:52.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.634+0000 7fbfdfaed640 4 set_mon_vals no callback set 2024-09-11T01:52:52.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.634+0000 7fbfdc887640 1 -- 172.21.15.12:0/3571407740 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4032a10 con 0x7fbfd81381c0 2024-09-11T01:52:52.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.634+0000 7fbfdc887640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:52.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.634+0000 7fbfdc887640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:52.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.634+0000 7fbfdc887640 10 monclient: dump: 2024-09-11T01:52:52.646 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:52.646 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:52.646 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.646 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.646 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:52.646 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.646 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:52.646 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:52.646 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:52.646 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.635+0000 7fbfdfaed640 10 monclient: shutdown 2024-09-11T01:52:52.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.635+0000 7fbfdfaed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:52.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.635+0000 7fbfdfaed640 1 -- 172.21.15.12:0/3571407740 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfd81381c0 msgr2=0x7fbfd81385a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.635+0000 7fbfdfaed640 1 --2- 172.21.15.12:0/3571407740 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfd81381c0 0x7fbfd81385a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbfd40097d0 tx=0x7fbfd402a030 comp rx=0 tx=0).stop 2024-09-11T01:52:52.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.635+0000 7fbfdfaed640 1 -- 172.21.15.12:0/3571407740 shutdown_connections 2024-09-11T01:52:52.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.635+0000 7fbfdfaed640 1 --2- 172.21.15.12:0/3571407740 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfd81381c0 0x7fbfd81385a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.635+0000 7fbfdfaed640 1 -- 172.21.15.12:0/3571407740 >> 172.21.15.12:0/3571407740 conn(0x7fbfd80a7440 msgr2=0x7fbfd80a7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:52.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.635+0000 7fbfdfaed640 1 -- 172.21.15.12:0/3571407740 shutdown_connections 2024-09-11T01:52:52.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.635+0000 7fbfdfaed640 1 -- 172.21.15.12:0/3571407740 wait complete. 2024-09-11T01:52:52.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.636+0000 7fbfdfaed640 10 monclient: build_initial_monmap 2024-09-11T01:52:52.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.636+0000 7fbfdfaed640 10 monclient: monmap: 2024-09-11T01:52:52.647 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:52.647 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:52.647 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:52.647 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:52.648 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:52.648 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.648 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:52.648 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:52.648 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:52.648 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.636+0000 7fbfdfaed640 1 Processor -- start 2024-09-11T01:52:52.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.636+0000 7fbfdfaed640 1 -- start start 2024-09-11T01:52:52.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.636+0000 7fbfdfaed640 10 monclient: init 2024-09-11T01:52:52.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.636+0000 7fbfdfaed640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:52.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.636+0000 7fbfdfaed640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:52:52.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.636+0000 7fbfdfaed640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd80a6aa0 0x7fbfd8136180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:52.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.636+0000 7fbfdfaed640 10 monclient(hunting): picked mon.noname-b con 0x7fbfd80a6aa0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:52.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.636+0000 7fbfdfaed640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfd81381c0 0x7fbfd81366c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:52.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.636+0000 7fbfdfaed640 10 monclient(hunting): picked mon.noname-a con 0x7fbfd81381c0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:52.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.636+0000 7fbfdfaed640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfd813d000 0x7fbfd8138dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:52.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.636+0000 7fbfdfaed640 10 monclient(hunting): picked mon.noname-c con 0x7fbfd813d000 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:52.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.636+0000 7fbfdfaed640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:52.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.636+0000 7fbfdfaed640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd8140d20 con 0x7fbfd81381c0 2024-09-11T01:52:52.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.636+0000 7fbfdfaed640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:52.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.636+0000 7fbfdfaed640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd809c5f0 con 0x7fbfd80a6aa0 2024-09-11T01:52:52.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.636+0000 7fbfdfaed640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:52.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.636+0000 7fbfdfaed640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fbfd8140ea0 con 0x7fbfd813d000 2024-09-11T01:52:52.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.636+0000 7fbfdfaed640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:52.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.636+0000 7fbfdfaed640 10 monclient(hunting): authenticate will time out at 2179.078890s 2024-09-11T01:52:52.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.636+0000 7fbfdd088640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfd81381c0 0x7fbfd81366c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:52.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.636+0000 7fbfde08a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfd813d000 0x7fbfd8138dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:52.649 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfde08a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfd813d000 0x7fbfd8138dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:43664/0 (socket says 172.21.15.12:43664) 2024-09-11T01:52:52.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfde08a640 1 -- 172.21.15.12:0/2206953659 learned_addr learned my addr 172.21.15.12:0/2206953659 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:52.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfde08a640 10 monclient(hunting): get_auth_request con 0x7fbfd813d000 auth_method 0 2024-09-11T01:52:52.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfde08a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:52.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfde08a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:52.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfde08a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:52.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfdd088640 10 monclient(hunting): get_auth_request con 0x7fbfd81381c0 auth_method 0 2024-09-11T01:52:52.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfdd088640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:52.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfdd088640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:52.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfdd088640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:52.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfde08a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:52.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfde08a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:52.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfde08a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:52.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfdd088640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:52.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfdd088640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:52.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfdd088640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:52.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfdd889640 1 --2- 172.21.15.12:0/2206953659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd80a6aa0 0x7fbfd8136180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:52.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfdd889640 10 monclient(hunting): get_auth_request con 0x7fbfd80a6aa0 auth_method 0 2024-09-11T01:52:52.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfdd889640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:52.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfdd889640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:52.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfdd889640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:52.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfdd889640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:52.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfdd889640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:52.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfdd889640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:52.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfde08a640 10 monclient(hunting): handle_auth_done global_id 244331 payload 824 2024-09-11T01:52:52.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfde08a640 1 -- 172.21.15.12:0/2206953659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd80a6aa0 msgr2=0x7fbfd8136180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfde08a640 1 --2- 172.21.15.12:0/2206953659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd80a6aa0 0x7fbfd8136180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfde08a640 1 -- 172.21.15.12:0/2206953659 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfd81381c0 msgr2=0x7fbfd81366c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfde08a640 1 --2- 172.21.15.12:0/2206953659 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfd81381c0 0x7fbfd81366c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfde08a640 10 monclient: _finish_hunting 0 2024-09-11T01:52:52.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfde08a640 1 monclient: found mon.noname-c 2024-09-11T01:52:52.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfde08a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:52.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfde08a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:52:52.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfde08a640 1 -- 172.21.15.12:0/2206953659 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd8136c00 con 0x7fbfd813d000 2024-09-11T01:52:52.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.637+0000 7fbfdd088640 1 --2- 172.21.15.12:0/2206953659 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbfd81381c0 0x7fbfd81366c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:52:52.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.639+0000 7fbfde08a640 1 -- 172.21.15.12:0/2206953659 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfd813d000 msgr2=0x7fbfd8138dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:52.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.639+0000 7fbfde08a640 1 --2- 172.21.15.12:0/2206953659 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfd813d000 0x7fbfd8138dd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbfd000c780 tx=0x7fbfd000cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:52.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.639+0000 7fbfce7fc640 1 -- 172.21.15.12:0/2206953659 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd000ac60 con 0x7fbfd813d000 2024-09-11T01:52:52.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.639+0000 7fbfce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:52.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.639+0000 7fbfce7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:52.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.639+0000 7fbfce7fc640 10 monclient: dump: 2024-09-11T01:52:52.653 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:52.653 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:52.653 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.653 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.653 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:52.653 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.653 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:52.653 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:52.653 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:52.653 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.639+0000 7fbfce7fc640 10 monclient: _finish_auth 0 2024-09-11T01:52:52.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.639+0000 7fbfce7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:52:52.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.639+0000 7fbfce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:52.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.639+0000 7fbfce7fc640 1 -- 172.21.15.12:0/2206953659 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfd000ae00 con 0x7fbfd813d000 2024-09-11T01:52:52.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.639+0000 7fbfdfaed640 5 monclient: authenticate success, global_id 244331 2024-09-11T01:52:52.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.639+0000 7fbfdfaed640 10 monclient: _renew_subs 2024-09-11T01:52:52.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.639+0000 7fbfdfaed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:52.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.639+0000 7fbfdfaed640 1 -- 172.21.15.12:0/2206953659 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfd80a1710 con 0x7fbfd813d000 2024-09-11T01:52:52.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.639+0000 7fbfce7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:52.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.639+0000 7fbfce7fc640 1 -- 172.21.15.12:0/2206953659 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd0004b50 con 0x7fbfd813d000 2024-09-11T01:52:52.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.639+0000 7fbfce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:52.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.639+0000 7fbfce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:52.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.639+0000 7fbfce7fc640 10 monclient: dump: 2024-09-11T01:52:52.654 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:52.654 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:52.654 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.655 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.655 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:52.655 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.655 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:52.655 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:52.655 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:52.655 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.639+0000 7fbfdfaed640 10 monclient: _renew_subs 2024-09-11T01:52:52.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.639+0000 7fbfcffff640 4 set_mon_vals no callback set 2024-09-11T01:52:52.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.639+0000 7fbfdfaed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:52.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.639+0000 7fbfdfaed640 1 -- 172.21.15.12:0/2206953659 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfd8140860 con 0x7fbfd813d000 2024-09-11T01:52:52.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.640+0000 7fbfdfaed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:52.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.640+0000 7fbfdfaed640 1 --2- 172.21.15.12:0/2206953659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd802d440 0x7fbfd802d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:52.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.640+0000 7fbfdfaed640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:52.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.640+0000 7fbfdfaed640 1 -- 172.21.15.12:0/2206953659 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd8077510 con 0x7fbfd802d440 2024-09-11T01:52:52.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.640+0000 7fbfce7fc640 1 -- 172.21.15.12:0/2206953659 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fbfd0006b50 con 0x7fbfd813d000 2024-09-11T01:52:52.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.640+0000 7fbfdd889640 1 --2- 172.21.15.12:0/2206953659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd802d440 0x7fbfd802d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:52.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.640+0000 7fbfdd889640 10 monclient: get_auth_request con 0x7fbfd802d440 auth_method 0 2024-09-11T01:52:52.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.640+0000 7fbfdd889640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:52.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.640+0000 7fbfdd889640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:52.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.640+0000 7fbfdd889640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:52.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.640+0000 7fbfce7fc640 1 --2- 172.21.15.12:0/2206953659 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbfbc03e210 0x7fbfbc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:52.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.641+0000 7fbfdd889640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:52.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.641+0000 7fbfdd889640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:52.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.641+0000 7fbfce7fc640 1 -- 172.21.15.12:0/2206953659 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(801..801 src has 676..801) ==== 4787+0+0 (secure 0 0 0) 0x7fbfd0052f00 con 0x7fbfd813d000 2024-09-11T01:52:52.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.641+0000 7fbfdd889640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:52.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.641+0000 7fbfdd088640 1 --2- 172.21.15.12:0/2206953659 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbfbc03e210 0x7fbfbc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:52.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.641+0000 7fbfdd088640 10 monclient: get_auth_request con 0x7fbfbc03e210 auth_method 0 2024-09-11T01:52:52.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.641+0000 7fbfdd889640 10 monclient(hunting): handle_auth_done global_id 234343 payload 824 2024-09-11T01:52:52.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.641+0000 7fbfdd889640 1 -- 172.21.15.12:0/2206953659 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbfd81385a0 con 0x7fbfd802d440 2024-09-11T01:52:52.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.643+0000 7fbfdd889640 1 -- 172.21.15.12:0/2206953659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd802d440 msgr2=0x7fbfd802d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:52.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.643+0000 7fbfdd889640 1 --2- 172.21.15.12:0/2206953659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd802d440 0x7fbfd802d820 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4034000 tx=0x7fbfd40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:52.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.643+0000 7fbfdd088640 1 --2- 172.21.15.12:0/2206953659 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbfbc03e210 0x7fbfbc0406d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8007c90 tx=0x7fbfc8007be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:52.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.643+0000 7fbfce7fc640 1 -- 172.21.15.12:0/2206953659 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4037040 con 0x7fbfd802d440 2024-09-11T01:52:52.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.643+0000 7fbfce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:52.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.643+0000 7fbfce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:52:52.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.643+0000 7fbfce7fc640 10 monclient: dump: 2024-09-11T01:52:52.660 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:52.660 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:52.660 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.660 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.660 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:52.660 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.660 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:52.660 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:52.660 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:52.660 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.643+0000 7fbfce7fc640 1 -- 172.21.15.12:0/2206953659 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbfd403c820 con 0x7fbfd802d440 2024-09-11T01:52:52.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.643+0000 7fbfce7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:52.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.643+0000 7fbfce7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:52.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.643+0000 7fbfce7fc640 1 -- 172.21.15.12:0/2206953659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd802d440 msgr2=0x7fbfd802d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.643+0000 7fbfce7fc640 1 --2- 172.21.15.12:0/2206953659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd802d440 0x7fbfd802d820 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4034000 tx=0x7fbfd40046c0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.643+0000 7fbfce7fc640 1 -- 172.21.15.12:0/2206953659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd802d440 msgr2=0x7fbfd802d820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:52.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.644+0000 7fbfce7fc640 1 --2- 172.21.15.12:0/2206953659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd802d440 0x7fbfd802d820 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.650+0000 7fbfdfaed640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:52:52.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.650+0000 7fbfdfaed640 1 --2- 172.21.15.12:0/2206953659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd802dd60 0x7fbfd802e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:52.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.650+0000 7fbfdfaed640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:52.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.650+0000 7fbfdfaed640 1 -- 172.21.15.12:0/2206953659 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd813ba70 con 0x7fbfd802dd60 2024-09-11T01:52:52.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.650+0000 7fbfdd889640 1 --2- 172.21.15.12:0/2206953659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd802dd60 0x7fbfd802e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:52.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.650+0000 7fbfdd889640 10 monclient: get_auth_request con 0x7fbfd802dd60 auth_method 0 2024-09-11T01:52:52.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.650+0000 7fbfdd889640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:52.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.650+0000 7fbfdd889640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:52.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.650+0000 7fbfdd889640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:52.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.650+0000 7fbfdd889640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:52.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.650+0000 7fbfdd889640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:52.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.650+0000 7fbfdd889640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:52.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.650+0000 7fbfdd889640 10 monclient(hunting): handle_auth_done global_id 234346 payload 824 2024-09-11T01:52:52.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.650+0000 7fbfdd889640 1 -- 172.21.15.12:0/2206953659 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbfd80a3030 con 0x7fbfd802dd60 2024-09-11T01:52:52.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.652+0000 7fbfdd889640 1 -- 172.21.15.12:0/2206953659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd802dd60 msgr2=0x7fbfd802e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:52.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.653+0000 7fbfdd889640 1 --2- 172.21.15.12:0/2206953659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd802dd60 0x7fbfd802e170 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbfd402a070 tx=0x7fbfd4008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:52.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.653+0000 7fbfce7fc640 1 -- 172.21.15.12:0/2206953659 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd404b040 con 0x7fbfd802dd60 2024-09-11T01:52:52.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.653+0000 7fbfce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:52.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.653+0000 7fbfce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:52:52.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.653+0000 7fbfce7fc640 10 monclient: dump: 2024-09-11T01:52:52.663 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:52.663 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:52.663 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.663 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.663 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:52.663 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.663 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:52.663 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:52.663 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:52.663 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.653+0000 7fbfce7fc640 1 -- 172.21.15.12:0/2206953659 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7fbfd4046d30 con 0x7fbfd802dd60 2024-09-11T01:52:52.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.653+0000 7fbfce7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:52:52.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.653+0000 7fbfce7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:52.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.653+0000 7fbfce7fc640 1 -- 172.21.15.12:0/2206953659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd802dd60 msgr2=0x7fbfd802e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.653+0000 7fbfce7fc640 1 --2- 172.21.15.12:0/2206953659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd802dd60 0x7fbfd802e170 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbfd402a070 tx=0x7fbfd4008030 comp rx=0 tx=0).stop 2024-09-11T01:52:52.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.653+0000 7fbfce7fc640 1 -- 172.21.15.12:0/2206953659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd802dd60 msgr2=0x7fbfd802e170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:52.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.653+0000 7fbfce7fc640 1 --2- 172.21.15.12:0/2206953659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd802dd60 0x7fbfd802e170 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.664 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:52:52.664 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:52:52.664 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:52:52.664 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:52:52.665 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 220, 2024-09-11T01:52:52.665 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 174545, 2024-09-11T01:52:52.665 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:52:52.665 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:52:52.665 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:52:52.665 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:52:52.665 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:52.665 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:52:52.665 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:52:52.665 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:52:52.665 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:52:52.665 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:52.665 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:52.665 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:52.666 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:52.666 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:52.666 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:52.666 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:52.666 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:52.666 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:52.666 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:52.666 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:52.666 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:52.666 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:52:52.666 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:52:52.666 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:52.666 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:52.666 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:52.667 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:52.667 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:52.667 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:52.667 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:52.667 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:52.667 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:52.667 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:52.667 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:52.667 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:52.667 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.667 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:52:52.667 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:52:52.667 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:52:52.668 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:52:52.668 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:52:52.668 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:52:52.668 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:52:52.668 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:52:52.668 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:52:52.668 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:52:52.668 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:52:52.668 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:52:52.668 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:52:52.668 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:52:52.668 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:52:52.669 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:52:52.669 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:52:52.669 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:52.669 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:52.669 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:52.669 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:52.669 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:52.669 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:52.669 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:52.669 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:52.669 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:52.669 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:52.669 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:52.669 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:52.670 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:52:52.670 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.670 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:52:52.670 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.670 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:52:52.670 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:52:52.670 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:52.670 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:52.670 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.670 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:52.670 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:52:52.670 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:52.670 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.670 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.671 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:52.671 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:52:52.671 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:52.671 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:52.671 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:52.671 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.671 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:52:52.671 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:52:52.671 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:52.671 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:52.671 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:52.671 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.671 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.672 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:52:52.672 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:52:52.672 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:52.672 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:52.672 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.672 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:52.672 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:52:52.672 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:52.672 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.672 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.672 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:52.672 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:52:52.672 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:52.672 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:52.673 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:52.673 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.673 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:52:52.673 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:52:52.673 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:52.673 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:52.673 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:52.673 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.673 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.673 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:52:52.673 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:52:52.673 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:52.673 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:52.673 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.674 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:52.674 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:52:52.674 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:52.674 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.674 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.674 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:52.674 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:52:52.674 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:52.674 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:52.674 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:52.674 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.674 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:52:52.674 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:52:52.674 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:52.675 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:52.675 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:52.675 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:52.675 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:52.675 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.675 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:52:52.675 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:52:52.675 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.675 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:52.675 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:52.675 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:52:52.675 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:52.675 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:52.676 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:52:52.676 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.676 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:52.676 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:52.676 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:52:52.676 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:52.676 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:52.676 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:52:52.676 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:52.676 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:52.676 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:52.676 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:52:52.676 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:52.676 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:52.677 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:52.677 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:52:52.677 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:52:52.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.653+0000 7fbfdfaed640 1 -- 172.21.15.12:0/2206953659 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbfbc03e210 msgr2=0x7fbfbc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.653+0000 7fbfdfaed640 1 --2- 172.21.15.12:0/2206953659 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbfbc03e210 0x7fbfbc0406d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8007c90 tx=0x7fbfc8007be0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.653+0000 7fbfdfaed640 10 monclient: shutdown 2024-09-11T01:52:52.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.653+0000 7fbfdfaed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:52.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.653+0000 7fbfdfaed640 1 -- 172.21.15.12:0/2206953659 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfd813d000 msgr2=0x7fbfd8138dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.653+0000 7fbfde08a640 1 -- 172.21.15.12:0/2206953659 reap_dead start 2024-09-11T01:52:52.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.653+0000 7fbfdfaed640 1 --2- 172.21.15.12:0/2206953659 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfd813d000 0x7fbfd8138dd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbfd000c780 tx=0x7fbfd000cc40 comp rx=0 tx=0).stop 2024-09-11T01:52:52.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.653+0000 7fbfdfaed640 1 -- 172.21.15.12:0/2206953659 shutdown_connections 2024-09-11T01:52:52.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.653+0000 7fbfdfaed640 1 --2- 172.21.15.12:0/2206953659 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbfd813d000 0x7fbfd8138dd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.653+0000 7fbfdfaed640 1 -- 172.21.15.12:0/2206953659 >> 172.21.15.12:0/2206953659 conn(0x7fbfd80a7440 msgr2=0x7fbfd813f950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:52.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.653+0000 7fbfdfaed640 1 -- 172.21.15.12:0/2206953659 shutdown_connections 2024-09-11T01:52:52.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.653+0000 7fbfdfaed640 1 -- 172.21.15.12:0/2206953659 wait complete. 2024-09-11T01:52:52.679 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:52:52.679 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:52:52.679 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:52:52.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.778+0000 7f86637d3640 10 monclient: get_monmap_and_config 2024-09-11T01:52:52.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.778+0000 7f86637d3640 10 monclient: build_initial_monmap 2024-09-11T01:52:52.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.778+0000 7f86637d3640 10 monclient: monmap: 2024-09-11T01:52:52.780 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:52.780 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:52.780 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:52.779537+0000 2024-09-11T01:52:52.781 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:52.779537+0000 2024-09-11T01:52:52.781 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:52.781 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.781 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:52.781 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:52.781 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:52.781 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:52.781 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:52.781 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.779+0000 7f86637d3640 1 Processor -- start 2024-09-11T01:52:52.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.779+0000 7f86637d3640 1 -- start start 2024-09-11T01:52:52.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.779+0000 7f86637d3640 10 monclient: init 2024-09-11T01:52:52.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f86637d3640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:52.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f86637d3640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-11T01:52:52.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f86637d3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f865c074160 addr v1:172.21.15.71:6789/0 2024-09-11T01:52:52.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f86637d3640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f865c0a0540 0x7f865c13cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:52.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f86637d3640 10 monclient(hunting): picked mon.noname-b con 0x7f865c0a0540 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:52.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f86637d3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f865c0a2ab0 addr v1:172.21.15.12:6789/0 2024-09-11T01:52:52.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f86637d3640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f865c0a34f0 con 0x7f865c0a2ab0 2024-09-11T01:52:52.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f86637d3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f865c0a0230 con 0x7f865c074160 2024-09-11T01:52:52.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f86637d3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:52.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f86637d3640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f865c070160 con 0x7f865c0a0540 2024-09-11T01:52:52.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f86637d3640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:52.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f86637d3640 10 monclient(hunting): authenticate will time out at 2179.222124s 2024-09-11T01:52:52.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f8660d6e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f865c0a0540 0x7f865c13cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:52.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f8661d70640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f865c0a2ab0 0x7f865c13f4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59148/0 (socket says 172.21.15.12:59148) 2024-09-11T01:52:52.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f8661d70640 1 -- 172.21.15.12:0/2355360198 learned_addr learned my addr 172.21.15.12:0/2355360198 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:52:52.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f8660d6e640 10 monclient(hunting): get_auth_request con 0x7f865c0a0540 auth_method 0 2024-09-11T01:52:52.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f8660d6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:52.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f8660d6e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:52.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f8660d6e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:52.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f8660d6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:52.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f8660d6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:52.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f8660d6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:52.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f8660d6e640 10 monclient(hunting): handle_auth_done global_id 244334 payload 274 2024-09-11T01:52:52.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f8660d6e640 1 -- 172.21.15.12:0/2355360198 >> v1:172.21.15.71:6789/0 conn(0x7f865c074160 legacy=0x7f865c074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:52.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f8660d6e640 1 -- 172.21.15.12:0/2355360198 >> v1:172.21.15.12:6789/0 conn(0x7f865c0a2ab0 legacy=0x7f865c13f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:52.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f8660d6e640 10 monclient: _finish_hunting 0 2024-09-11T01:52:52.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f8660d6e640 1 monclient: found mon.noname-b 2024-09-11T01:52:52.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f8660d6e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:52.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f8660d6e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:52:52.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.780+0000 7f8660d6e640 1 -- 172.21.15.12:0/2355360198 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f865c070440 con 0x7f865c0a0540 2024-09-11T01:52:52.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.783+0000 7f8660d6e640 1 -- 172.21.15.12:0/2355360198 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f865c0a0540 msgr2=0x7f865c13cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:52.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.783+0000 7f8660d6e640 1 --2- 172.21.15.12:0/2355360198 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f865c0a0540 0x7f865c13cfa0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f86580097d0 tx=0x7f8658029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8ccc935b62c9cf4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:52.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.783+0000 7f864ffff640 1 -- 172.21.15.12:0/2355360198 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8658037070 con 0x7f865c0a0540 2024-09-11T01:52:52.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.783+0000 7f864ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:52.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.783+0000 7f864ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:52:52.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.783+0000 7f864ffff640 10 monclient: dump: 2024-09-11T01:52:52.791 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:52.791 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:52.792 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.792 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.792 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:52.792 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.792 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:52.792 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:52.792 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:52.792 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.783+0000 7f864ffff640 10 monclient: _finish_auth 0 2024-09-11T01:52:52.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.783+0000 7f864ffff640 10 monclient: _check_auth_tickets 2024-09-11T01:52:52.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.783+0000 7f864ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:52.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.783+0000 7f86637d3640 5 monclient: authenticate success, global_id 244334 2024-09-11T01:52:52.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.783+0000 7f864ffff640 1 -- 172.21.15.12:0/2355360198 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8658032440 con 0x7f865c0a0540 2024-09-11T01:52:52.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.783+0000 7f86637d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:52:52.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.783+0000 7f864ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:52.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.783+0000 7f86637d3640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:52.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.783+0000 7f86637d3640 4 set_mon_vals no callback set 2024-09-11T01:52:52.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.783+0000 7f864ffff640 1 -- 172.21.15.12:0/2355360198 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8658032810 con 0x7f865c0a0540 2024-09-11T01:52:52.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.783+0000 7f864ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:52.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.783+0000 7f864ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:52.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.783+0000 7f864ffff640 10 monclient: dump: 2024-09-11T01:52:52.793 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:52.793 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:52.793 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.793 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.793 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:52.793 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.796 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:52.796 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:52.796 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:52.796 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.784+0000 7f86637d3640 10 monclient: shutdown 2024-09-11T01:52:52.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.784+0000 7f86637d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:52.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.784+0000 7f86637d3640 1 -- 172.21.15.12:0/2355360198 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f865c0a0540 msgr2=0x7f865c13cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.784+0000 7f86637d3640 1 --2- 172.21.15.12:0/2355360198 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f865c0a0540 0x7f865c13cfa0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f86580097d0 tx=0x7f8658029fb0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.784+0000 7f86637d3640 1 -- 172.21.15.12:0/2355360198 shutdown_connections 2024-09-11T01:52:52.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.784+0000 7f86637d3640 1 --2- 172.21.15.12:0/2355360198 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f865c0a0540 0x7f865c13cfa0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.784+0000 7f86637d3640 1 -- 172.21.15.12:0/2355360198 >> 172.21.15.12:0/2355360198 conn(0x7f865c0a36c0 msgr2=0x7f865c0a3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:52.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.784+0000 7f86637d3640 1 -- 172.21.15.12:0/2355360198 shutdown_connections 2024-09-11T01:52:52.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.784+0000 7f86637d3640 1 -- 172.21.15.12:0/2355360198 wait complete. 2024-09-11T01:52:52.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.784+0000 7f86637d3640 10 monclient: build_initial_monmap 2024-09-11T01:52:52.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.784+0000 7f86637d3640 10 monclient: monmap: 2024-09-11T01:52:52.797 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:52.797 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:52.797 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:52.797 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:52.797 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:52.798 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f86637d3640 1 Processor -- start 2024-09-11T01:52:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f86637d3640 1 -- start start 2024-09-11T01:52:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f86637d3640 10 monclient: init 2024-09-11T01:52:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f86637d3640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f86637d3640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:52:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f86637d3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f865c074160 0x7f865c131650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f86637d3640 10 monclient(hunting): picked mon.noname-a con 0x7f865c074160 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f86637d3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f865c0a0540 0x7f865c131b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:52.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f86637d3640 10 monclient(hunting): picked mon.noname-b con 0x7f865c0a0540 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f86637d3640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f865c0a2ab0 0x7f865c139d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f86637d3640 10 monclient(hunting): picked mon.noname-c con 0x7f865c0a2ab0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f86637d3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f86637d3640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f865c140d90 con 0x7f865c074160 2024-09-11T01:52:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f86637d3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f86637d3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f865c140c10 con 0x7f865c0a0540 2024-09-11T01:52:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f86637d3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f86637d3640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f865c140f10 con 0x7f865c0a2ab0 2024-09-11T01:52:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f86637d3640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f86637d3640 10 monclient(hunting): authenticate will time out at 2179.227560s 2024-09-11T01:52:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f866156f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f865c074160 0x7f865c131650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f8661d70640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f865c0a2ab0 0x7f865c139d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f866156f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f865c074160 0x7f865c131650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60040/0 (socket says 172.21.15.12:60040) 2024-09-11T01:52:52.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f866156f640 1 -- 172.21.15.12:0/3722674563 learned_addr learned my addr 172.21.15.12:0/3722674563 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f8661d70640 10 monclient(hunting): get_auth_request con 0x7f865c0a2ab0 auth_method 0 2024-09-11T01:52:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f8661d70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f8661d70640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f8661d70640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f866156f640 10 monclient(hunting): get_auth_request con 0x7f865c074160 auth_method 0 2024-09-11T01:52:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f866156f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f866156f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f866156f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f8661d70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f8661d70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f8661d70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f866156f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:52.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f866156f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:52.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f866156f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:52.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f8660d6e640 1 --2- 172.21.15.12:0/3722674563 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f865c0a0540 0x7f865c131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:52.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f8660d6e640 10 monclient(hunting): get_auth_request con 0x7f865c0a0540 auth_method 0 2024-09-11T01:52:52.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f8660d6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:52.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f8660d6e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:52.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f8660d6e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:52.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f8661d70640 10 monclient(hunting): handle_auth_done global_id 244337 payload 824 2024-09-11T01:52:52.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f8661d70640 1 -- 172.21.15.12:0/3722674563 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f865c0a0540 msgr2=0x7f865c131b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f8661d70640 1 --2- 172.21.15.12:0/3722674563 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f865c0a0540 0x7f865c131b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f8661d70640 1 -- 172.21.15.12:0/3722674563 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f865c074160 msgr2=0x7f865c131650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:52.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f8661d70640 1 --2- 172.21.15.12:0/3722674563 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f865c074160 0x7f865c131650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:52.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f8661d70640 10 monclient: _finish_hunting 0 2024-09-11T01:52:52.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f8661d70640 1 monclient: found mon.noname-c 2024-09-11T01:52:52.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f8661d70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:52.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f8661d70640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:52:52.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.785+0000 7f8661d70640 1 -- 172.21.15.12:0/3722674563 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f865c075000 con 0x7f865c0a2ab0 2024-09-11T01:52:52.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.788+0000 7f8661d70640 1 -- 172.21.15.12:0/3722674563 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f865c0a2ab0 msgr2=0x7f865c139d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:52.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.788+0000 7f8661d70640 1 --2- 172.21.15.12:0/3722674563 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f865c0a2ab0 0x7f865c139d70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f865400caf0 tx=0x7f865400cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:52.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.788+0000 7f864dffb640 1 -- 172.21.15.12:0/3722674563 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865400ade0 con 0x7f865c0a2ab0 2024-09-11T01:52:52.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.788+0000 7f864dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:52.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.788+0000 7f864dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:52.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.788+0000 7f864dffb640 10 monclient: dump: 2024-09-11T01:52:52.802 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:52.802 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:52.802 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.802 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.802 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:52.802 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.803 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:52.803 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:52.803 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:52.803 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.788+0000 7f864dffb640 10 monclient: _finish_auth 0 2024-09-11T01:52:52.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.788+0000 7f864dffb640 10 monclient: _check_auth_tickets 2024-09-11T01:52:52.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.788+0000 7f864dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:52.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.788+0000 7f86637d3640 5 monclient: authenticate success, global_id 244337 2024-09-11T01:52:52.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.788+0000 7f864dffb640 1 -- 172.21.15.12:0/3722674563 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8654012400 con 0x7f865c0a2ab0 2024-09-11T01:52:52.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.788+0000 7f864dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:52.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.788+0000 7f86637d3640 10 monclient: _renew_subs 2024-09-11T01:52:52.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.788+0000 7f864dffb640 1 -- 172.21.15.12:0/3722674563 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86540127f0 con 0x7f865c0a2ab0 2024-09-11T01:52:52.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.788+0000 7f86637d3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:52.803 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.788+0000 7f86637d3640 1 -- 172.21.15.12:0/3722674563 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f865c13db90 con 0x7f865c0a2ab0 2024-09-11T01:52:52.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.788+0000 7f864dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:52.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.788+0000 7f864dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:52.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.788+0000 7f864f7fe640 4 set_mon_vals no callback set 2024-09-11T01:52:52.804 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.788+0000 7f864dffb640 10 monclient: dump: 2024-09-11T01:52:52.804 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:52.804 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:52.804 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.804 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:52.804 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:52.804 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:52.804 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:52.804 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:52.804 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:52.805 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.788+0000 7f86637d3640 10 monclient: _renew_subs 2024-09-11T01:52:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.788+0000 7f86637d3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.788+0000 7f86637d3640 1 -- 172.21.15.12:0/3722674563 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f865c13b830 con 0x7f865c0a2ab0 2024-09-11T01:52:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.789+0000 7f864dffb640 1 -- 172.21.15.12:0/3722674563 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f8654012990 con 0x7f865c0a2ab0 2024-09-11T01:52:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.789+0000 7f86637d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.789+0000 7f86637d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.789+0000 7f86637d3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.789+0000 7f86637d3640 1 -- 172.21.15.12:0/3722674563 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f865c074540 con 0x7f865c0a2ab0 2024-09-11T01:52:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.789+0000 7f864dffb640 1 --2- 172.21.15.12:0/3722674563 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f864003e260 0x7f8640040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.789+0000 7f864dffb640 1 -- 172.21.15.12:0/3722674563 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(801..801 src has 676..801) ==== 4787+0+0 (secure 0 0 0) 0x7f86540532f0 con 0x7f865c0a2ab0 2024-09-11T01:52:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.789+0000 7f866156f640 1 --2- 172.21.15.12:0/3722674563 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f864003e260 0x7f8640040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:52.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.789+0000 7f866156f640 10 monclient: get_auth_request con 0x7f864003e260 auth_method 0 2024-09-11T01:52:52.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.792+0000 7f866156f640 1 --2- 172.21.15.12:0/3722674563 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f864003e260 0x7f8640040720 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f8650005020 tx=0x7f8650007ef0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:52.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.794+0000 7f864dffb640 1 -- 172.21.15.12:0/3722674563 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f865c075000 con 0x7f865c0a2ab0 2024-09-11T01:52:52.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.794+0000 7f864dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:52.806 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:52.794+0000 7f864dffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:52.992 INFO:tasks.rados.rados.0.smithi071.stdout:3543: done (3 left) 2024-09-11T01:52:52.992 INFO:tasks.rados.rados.0.smithi071.stdout:3544: read oid 44 snap 384 2024-09-11T01:52:52.992 INFO:tasks.rados.rados.0.smithi071.stdout:3544: expect deleted 2024-09-11T01:52:52.992 INFO:tasks.rados.rados.0.smithi071.stdout:3545: read oid 18 snap -1 2024-09-11T01:52:52.992 INFO:tasks.rados.rados.0.smithi071.stdout:3545: expect (ObjNum 618 snap 183 seq_num 618) 2024-09-11T01:52:52.992 INFO:tasks.rados.rados.0.smithi071.stdout:3546: copy_from oid 27 from oid 45 current snap is 401 2024-09-11T01:52:52.992 INFO:tasks.rados.rados.0.smithi071.stdout:3540: finishing write tid 2 to smithi07134777-28 2024-09-11T01:52:52.992 INFO:tasks.rados.rados.0.smithi071.stdout:3540: finishing write tid 3 to smithi07134777-28 2024-09-11T01:52:52.992 INFO:tasks.rados.rados.0.smithi071.stdout:3540: finishing write tid 4 to smithi07134777-28 2024-09-11T01:52:52.993 INFO:tasks.rados.rados.0.smithi071.stdout:3540: oid 28 updating version 0 to 2211 2024-09-11T01:52:52.993 INFO:tasks.rados.rados.0.smithi071.stdout:3540: oid 28 version 2211 is already newer than 2210 2024-09-11T01:52:52.993 INFO:tasks.rados.rados.0.smithi071.stdout:3540: oid 28 updating version 2211 to 2212 2024-09-11T01:52:52.993 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 2212 (ObjNum 1226 snap 401 seq_num 1226) dirty exists 2024-09-11T01:52:52.993 INFO:tasks.rados.rados.0.smithi071.stdout:3540: left oid 28 (ObjNum 1226 snap 401 seq_num 1226) 2024-09-11T01:52:52.993 INFO:tasks.rados.rados.0.smithi071.stdout:3542: finishing rollback tid 0 to smithi07134777-23 2024-09-11T01:52:52.993 INFO:tasks.rados.rados.0.smithi071.stdout:3542: finishing rollback tid 1 to smithi07134777-23 2024-09-11T01:52:52.993 INFO:tasks.rados.rados.0.smithi071.stdout:3542: finishing rollback tid 2 to smithi07134777-23 2024-09-11T01:52:52.993 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 1209 (ObjNum 1160 snap 370 seq_num 1160) dirty exists 2024-09-11T01:52:52.993 INFO:tasks.rados.rados.0.smithi071.stdout:3541: finishing copy_from racing read to smithi07134777-32 2024-09-11T01:52:52.993 INFO:tasks.rados.rados.0.smithi071.stdout:3541: finishing copy_from to smithi07134777-32 2024-09-11T01:52:52.993 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 2093 (ObjNum 1187 snap 378 seq_num 1187) dirty exists 2024-09-11T01:52:52.993 INFO:tasks.rados.rados.0.smithi071.stdout:3540: done (5 left) 2024-09-11T01:52:52.993 INFO:tasks.rados.rados.0.smithi071.stdout:3541: done (4 left) 2024-09-11T01:52:52.994 INFO:tasks.rados.rados.0.smithi071.stdout:3542: done (3 left) 2024-09-11T01:52:52.994 INFO:tasks.rados.rados.0.smithi071.stdout:3547: delete oid 37 current snap is 401 2024-09-11T01:52:53.003 INFO:tasks.rados.rados.0.smithi071.stdout:3544: done (3 left) 2024-09-11T01:52:53.004 INFO:tasks.rados.rados.0.smithi071.stdout:3547: done (2 left) 2024-09-11T01:52:53.004 INFO:tasks.rados.rados.0.smithi071.stdout:3548: delete oid 21 current snap is 401 2024-09-11T01:52:53.004 INFO:tasks.rados.rados.0.smithi071.stdout:3545: expect (ObjNum 618 snap 183 seq_num 618) 2024-09-11T01:52:53.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:53.064+0000 7f86637d3640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:52:53.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:53.064+0000 7f86637d3640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:52:53.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:53.064+0000 7f86637d3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:53.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:53.064+0000 7f86637d3640 1 -- 172.21.15.12:0/3722674563 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f865c13d970 con 0x7f865c0a2ab0 2024-09-11T01:52:53.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:53.066+0000 7f864dffb640 1 -- 172.21.15.12:0/3722674563 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f86540530d0 con 0x7f865c0a2ab0 2024-09-11T01:52:53.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:53.066+0000 7f864dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:52:53.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:53.066+0000 7f864dffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:53.068 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.066+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.069 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.066+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=1,health=11,kv=12,logm=50} crc {auth=2210260778,config=1747058506,health=3724698709,kv=3263697420,logm=1512827944}) 2024-09-11T01:52:53.069 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.066+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=11,kv=11,logm=49} crc {auth=2873855223,config=2586702516,health=3724698709,kv=2372998920,logm=2335812755}) 2024-09-11T01:52:53.069 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.066+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.069 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.066+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=1,health=11,kv=12,logm=50} crc {auth=2210260778,config=1747058506,health=3724698709,kv=3263697420,logm=1512827944}) 2024-09-11T01:52:53.069 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.066+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=12,kv=12,logm=50} crc {auth=2608212000,config=2586702516,health=1443762948,kv=3263697420,logm=4285237752}) 2024-09-11T01:52:53.069 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.067+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.069 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.067+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2943606799}) 2024-09-11T01:52:53.069 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.067+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1550435124}) 2024-09-11T01:52:53.069 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.067+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.069 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.067+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2943606799}) 2024-09-11T01:52:53.069 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.067+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=769663643}) 2024-09-11T01:52:53.070 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.068+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.070 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.068+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=284408139}) 2024-09-11T01:52:53.070 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.068+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1663935450}) 2024-09-11T01:52:53.070 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.068+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.070 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.068+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=284408139}) 2024-09-11T01:52:53.070 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.068+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2520138092}) 2024-09-11T01:52:53.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:53.068+0000 7f86637d3640 1 -- 172.21.15.12:0/3722674563 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f864003e260 msgr2=0x7f8640040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:53.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:53.068+0000 7f86637d3640 1 --2- 172.21.15.12:0/3722674563 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f864003e260 0x7f8640040720 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f8650005020 tx=0x7f8650007ef0 comp rx=0 tx=0).stop 2024-09-11T01:52:53.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:53.068+0000 7f86637d3640 10 monclient: shutdown 2024-09-11T01:52:53.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:53.068+0000 7f86637d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:53.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:53.068+0000 7f86637d3640 1 -- 172.21.15.12:0/3722674563 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f865c0a2ab0 msgr2=0x7f865c139d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:53.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:53.068+0000 7f86637d3640 1 --2- 172.21.15.12:0/3722674563 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f865c0a2ab0 0x7f865c139d70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f865400caf0 tx=0x7f865400cef0 comp rx=0 tx=0).stop 2024-09-11T01:52:53.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:53.068+0000 7f86637d3640 1 -- 172.21.15.12:0/3722674563 shutdown_connections 2024-09-11T01:52:53.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:53.068+0000 7f86637d3640 1 --2- 172.21.15.12:0/3722674563 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f865c0a2ab0 0x7f865c139d70 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:53.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:53.068+0000 7f86637d3640 1 --2- 172.21.15.12:0/3722674563 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f865c0a0540 0x7f865c131b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:53.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:53.068+0000 7f86637d3640 1 --2- 172.21.15.12:0/3722674563 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f864003e260 0x7f8640040720 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:53.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:53.068+0000 7f86637d3640 1 --2- 172.21.15.12:0/3722674563 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f865c074160 0x7f865c131650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:53.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:53.068+0000 7f86637d3640 1 -- 172.21.15.12:0/3722674563 >> 172.21.15.12:0/3722674563 conn(0x7f865c0a36c0 msgr2=0x7f865c071810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:53.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:53.068+0000 7f86637d3640 1 -- 172.21.15.12:0/3722674563 shutdown_connections 2024-09-11T01:52:53.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:53.068+0000 7f86637d3640 1 -- 172.21.15.12:0/3722674563 wait complete. 2024-09-11T01:52:53.073 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.069+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.073 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.069+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2851773363}) 2024-09-11T01:52:53.073 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.069+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=855525444}) 2024-09-11T01:52:53.073 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.069+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.074 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.069+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2851773363}) 2024-09-11T01:52:53.074 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.069+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2705557824}) 2024-09-11T01:52:53.074 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.071+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.074 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.071+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4187504924}) 2024-09-11T01:52:53.074 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.071+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2889361558}) 2024-09-11T01:52:53.074 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.071+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.074 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.071+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4187504924}) 2024-09-11T01:52:53.074 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.071+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=966353853}) 2024-09-11T01:52:53.074 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.072+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.074 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.072+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1661391348}) 2024-09-11T01:52:53.074 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.072+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=385584940}) 2024-09-11T01:52:53.074 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.072+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.074 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.072+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1661391348}) 2024-09-11T01:52:53.075 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.072+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2386202903}) 2024-09-11T01:52:53.075 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.073+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.075 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.073+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3539461473}) 2024-09-11T01:52:53.076 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.073+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3124547367}) 2024-09-11T01:52:53.076 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.073+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.076 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.073+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3539461473}) 2024-09-11T01:52:53.076 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.073+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2999371193}) 2024-09-11T01:52:53.076 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.074+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.076 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.074+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2237927720}) 2024-09-11T01:52:53.076 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.074+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3353947544}) 2024-09-11T01:52:53.076 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.074+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.076 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.074+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2237927720}) 2024-09-11T01:52:53.076 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.074+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4226659086}) 2024-09-11T01:52:53.077 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.075+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.077 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.075+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2117172597}) 2024-09-11T01:52:53.077 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.075+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=733321441}) 2024-09-11T01:52:53.077 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.075+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.077 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.075+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2117172597}) 2024-09-11T01:52:53.077 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.075+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1189348299}) 2024-09-11T01:52:53.078 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.076+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.078 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.076+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4116268358}) 2024-09-11T01:52:53.078 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.076+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=936494828}) 2024-09-11T01:52:53.078 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.076+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.079 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.076+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4116268358}) 2024-09-11T01:52:53.079 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.076+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=75991920}) 2024-09-11T01:52:53.080 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.077+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.080 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.077+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1046963988}) 2024-09-11T01:52:53.080 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.077+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2707072833}) 2024-09-11T01:52:53.080 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.077+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.080 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.077+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1046963988}) 2024-09-11T01:52:53.080 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.077+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3159689713}) 2024-09-11T01:52:53.080 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.078+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.081 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.078+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277067950}) 2024-09-11T01:52:53.081 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.078+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2387948014}) 2024-09-11T01:52:53.081 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.078+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.081 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.078+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277067950}) 2024-09-11T01:52:53.081 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.078+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2447682542}) 2024-09-11T01:52:53.082 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.079+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.082 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.079+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3551876076}) 2024-09-11T01:52:53.082 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.079+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2555706220}) 2024-09-11T01:52:53.082 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.079+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.082 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.079+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3551876076}) 2024-09-11T01:52:53.082 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.079+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3298791600}) 2024-09-11T01:52:53.082 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.080+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.082 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.080+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2571950772}) 2024-09-11T01:52:53.082 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.080+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2525306588}) 2024-09-11T01:52:53.082 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.080+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.083 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.080+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2571950772}) 2024-09-11T01:52:53.083 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.080+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1900047057}) 2024-09-11T01:52:53.084 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:52:53.151 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.149+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.151 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.149+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=844559864}) 2024-09-11T01:52:53.151 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.149+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=383010540}) 2024-09-11T01:52:53.151 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.149+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.151 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.149+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=844559864}) 2024-09-11T01:52:53.151 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.149+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3276049973}) 2024-09-11T01:52:53.173 INFO:tasks.rados.rados.0.smithi071.stdout:3546: finishing copy_from racing read to smithi07134777-27 2024-09-11T01:52:53.173 INFO:tasks.rados.rados.0.smithi071.stdout:3546: finishing copy_from to smithi07134777-27 2024-09-11T01:52:53.173 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 2153 (ObjNum 932 snap 293 seq_num 932) dirty exists 2024-09-11T01:52:53.173 INFO:tasks.rados.rados.0.smithi071.stdout:3545: done (2 left) 2024-09-11T01:52:53.173 INFO:tasks.rados.rados.0.smithi071.stdout:3546: done (1 left) 2024-09-11T01:52:53.173 INFO:tasks.rados.rados.0.smithi071.stdout:3548: done (0 left) 2024-09-11T01:52:53.173 INFO:tasks.rados.rados.0.smithi071.stdout:3549: write oid 26 current snap is 401 2024-09-11T01:52:53.174 INFO:tasks.rados.rados.0.smithi071.stdout:3549: seq_num 1229 ranges {655803=413854,1584958=462421} 2024-09-11T01:52:53.178 INFO:tasks.rados.rados.0.smithi071.stdout:3549: writing smithi07134777-26 from 655803 to 1069657 tid 1 2024-09-11T01:52:53.184 INFO:tasks.rados.rados.0.smithi071.stdout:3549: writing smithi07134777-26 from 1584958 to 2047379 tid 2 2024-09-11T01:52:53.184 INFO:tasks.rados.rados.0.smithi071.stdout:3550: read oid 21 snap -1 2024-09-11T01:52:53.184 INFO:tasks.rados.rados.0.smithi071.stdout:3550: expect deleted 2024-09-11T01:52:53.184 INFO:tasks.rados.rados.0.smithi071.stdout:3551: snap_create 2024-09-11T01:52:53.184 INFO:tasks.rados.rados.0.smithi071.stdout:3549: finishing write tid 1 to smithi07134777-26 2024-09-11T01:52:53.188 INFO:tasks.rados.rados.0.smithi071.stdout:3549: finishing write tid 2 to smithi07134777-26 2024-09-11T01:52:53.188 INFO:tasks.rados.rados.0.smithi071.stdout:3549: finishing write tid 3 to smithi07134777-26 2024-09-11T01:52:53.189 INFO:tasks.rados.rados.0.smithi071.stdout:3549: finishing write tid 4 to smithi07134777-26 2024-09-11T01:52:53.189 INFO:tasks.rados.rados.0.smithi071.stdout:3549: oid 26 updating version 0 to 2521 2024-09-11T01:52:53.189 INFO:tasks.rados.rados.0.smithi071.stdout:3549: oid 26 version 2521 is already newer than 2520 2024-09-11T01:52:53.189 INFO:tasks.rados.rados.0.smithi071.stdout:3549: oid 26 version 2521 is already newer than 2519 2024-09-11T01:52:53.189 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 2521 (ObjNum 1229 snap 401 seq_num 1229) dirty exists 2024-09-11T01:52:53.189 INFO:tasks.rados.rados.0.smithi071.stdout:3549: left oid 26 (ObjNum 1229 snap 401 seq_num 1229) 2024-09-11T01:52:53.277 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.276+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.278 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.276+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582666675}) 2024-09-11T01:52:53.278 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.276+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3680632410}) 2024-09-11T01:52:53.278 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.276+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.278 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.276+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582666675}) 2024-09-11T01:52:53.278 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.276+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=334064223}) 2024-09-11T01:52:53.303 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.301+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.303 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.301+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1782280073}) 2024-09-11T01:52:53.303 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.301+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=684763761}) 2024-09-11T01:52:53.303 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.301+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.303 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.301+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1782280073}) 2024-09-11T01:52:53.303 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.301+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2979146548}) 2024-09-11T01:52:53.305 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.303+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.305 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.303+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1595774910}) 2024-09-11T01:52:53.305 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.304+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1988287189}) 2024-09-11T01:52:53.305 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.304+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.305 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.304+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1595774910}) 2024-09-11T01:52:53.305 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.304+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2162908723}) 2024-09-11T01:52:53.309 INFO:tasks.rados.rados.0.smithi071.stdout:3549: done (2 left) 2024-09-11T01:52:53.310 INFO:tasks.rados.rados.0.smithi071.stdout:3550: done (1 left) 2024-09-11T01:52:53.310 INFO:tasks.rados.rados.0.smithi071.stdout:3551: done (0 left) 2024-09-11T01:52:53.310 INFO:tasks.rados.rados.0.smithi071.stdout:3552: write (excl) oid 35 current snap is 402 2024-09-11T01:52:53.310 INFO:tasks.rados.rados.0.smithi071.stdout:3552: seq_num 1230 ranges {672610=489103,1643440=722432} 2024-09-11T01:52:53.315 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.313+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.315 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.313+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=770532514}) 2024-09-11T01:52:53.315 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.313+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1554113838}) 2024-09-11T01:52:53.315 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.313+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.315 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.313+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=770532514}) 2024-09-11T01:52:53.316 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.313+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2466704347}) 2024-09-11T01:52:53.316 INFO:tasks.rados.rados.0.smithi071.stdout:3552: writing smithi07134777-35 from 672610 to 1161713 tid 1 2024-09-11T01:52:53.322 INFO:tasks.rados.rados.0.smithi071.stdout:3552: writing smithi07134777-35 from 1643440 to 2365872 tid 2 2024-09-11T01:52:53.323 INFO:tasks.rados.rados.0.smithi071.stdout:3553: write oid 37 current snap is 402 2024-09-11T01:52:53.323 INFO:tasks.rados.rados.0.smithi071.stdout:3553: seq_num 1231 ranges {689417=564352,1701922=466090} 2024-09-11T01:52:53.329 INFO:tasks.rados.rados.0.smithi071.stdout:3553: writing smithi07134777-37 from 689417 to 1253769 tid 1 2024-09-11T01:52:53.335 INFO:tasks.rados.rados.0.smithi071.stdout:3553: writing smithi07134777-37 from 1701922 to 2168012 tid 2 2024-09-11T01:52:53.335 INFO:tasks.rados.rados.0.smithi071.stdout:3554: copy_from oid 42 from oid 48 current snap is 402 2024-09-11T01:52:53.335 INFO:tasks.rados.rados.0.smithi071.stdout:3552: finishing write tid 1 to smithi07134777-35 2024-09-11T01:52:53.335 INFO:tasks.rados.rados.0.smithi071.stdout:3552: finishing write tid 2 to smithi07134777-35 2024-09-11T01:52:53.335 INFO:tasks.rados.rados.0.smithi071.stdout:3552: finishing write tid 3 to smithi07134777-35 2024-09-11T01:52:53.335 INFO:tasks.rados.rados.0.smithi071.stdout:3552: finishing write tid 4 to smithi07134777-35 2024-09-11T01:52:53.335 INFO:tasks.rados.rados.0.smithi071.stdout:3552: oid 35 updating version 0 to 2095 2024-09-11T01:52:53.335 INFO:tasks.rados.rados.0.smithi071.stdout:3552: oid 35 updating version 2095 to 2096 2024-09-11T01:52:53.335 INFO:tasks.rados.rados.0.smithi071.stdout:3552: oid 35 version 2096 is already newer than 2094 2024-09-11T01:52:53.335 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 2096 (ObjNum 1230 snap 402 seq_num 1230) dirty exists 2024-09-11T01:52:53.336 INFO:tasks.rados.rados.0.smithi071.stdout:3552: left oid 35 (ObjNum 1230 snap 402 seq_num 1230) 2024-09-11T01:52:53.336 INFO:tasks.rados.rados.0.smithi071.stdout:3552: done (2 left) 2024-09-11T01:52:53.336 INFO:tasks.rados.rados.0.smithi071.stdout:3555: read oid 12 snap -1 2024-09-11T01:52:53.336 INFO:tasks.rados.rados.0.smithi071.stdout:3555: expect (ObjNum 1208 snap 393 seq_num 1208) 2024-09-11T01:52:53.336 INFO:tasks.rados.rados.0.smithi071.stdout:3556: snap_create 2024-09-11T01:52:53.344 INFO:tasks.rados.rados.0.smithi071.stdout:3554: finishing copy_from racing read to smithi07134777-42 2024-09-11T01:52:53.348 INFO:tasks.rados.rados.0.smithi071.stdout:3555: expect (ObjNum 1208 snap 393 seq_num 1208) 2024-09-11T01:52:53.422 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.420+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.422 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.420+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3513722156}) 2024-09-11T01:52:53.422 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.420+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2643855314}) 2024-09-11T01:52:53.422 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.420+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.422 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.420+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3513722156}) 2024-09-11T01:52:53.422 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.420+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2720861334}) 2024-09-11T01:52:53.423 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.421+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.423 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.421+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=537939290}) 2024-09-11T01:52:53.423 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.421+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2658339278}) 2024-09-11T01:52:53.423 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.421+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.423 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.421+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=537939290}) 2024-09-11T01:52:53.423 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.421+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=988666052}) 2024-09-11T01:52:53.424 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.422+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.424 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.422+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2694734381}) 2024-09-11T01:52:53.425 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.422+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2996728312}) 2024-09-11T01:52:53.425 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.422+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.425 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.422+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2694734381}) 2024-09-11T01:52:53.425 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.422+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2770770865}) 2024-09-11T01:52:53.425 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.423+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.425 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.423+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=653959979}) 2024-09-11T01:52:53.425 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.423+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1612299227}) 2024-09-11T01:52:53.425 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.423+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.425 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.423+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=653959979}) 2024-09-11T01:52:53.425 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.423+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1734131653}) 2024-09-11T01:52:53.426 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.424+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.426 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.424+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1539769063}) 2024-09-11T01:52:53.426 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.424+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=312717606}) 2024-09-11T01:52:53.426 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.424+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.426 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.424+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1539769063}) 2024-09-11T01:52:53.427 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.424+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=995133486}) 2024-09-11T01:52:53.427 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.425+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.427 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.425+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=540202306}) 2024-09-11T01:52:53.427 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.425+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=327438523}) 2024-09-11T01:52:53.427 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.425+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.427 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.425+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=540202306}) 2024-09-11T01:52:53.427 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.425+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4074369231}) 2024-09-11T01:52:53.434 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.432+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.434 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.432+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399158280}) 2024-09-11T01:52:53.434 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.432+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1411271284}) 2024-09-11T01:52:53.434 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.432+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.434 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.432+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399158280}) 2024-09-11T01:52:53.434 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.432+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3503820153}) 2024-09-11T01:52:53.435 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.433+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.435 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.433+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2964246818}) 2024-09-11T01:52:53.436 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.433+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=51525787}) 2024-09-11T01:52:53.436 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.433+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.436 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.433+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2964246818}) 2024-09-11T01:52:53.436 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.433+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1483041583}) 2024-09-11T01:52:53.436 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.434+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.436 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.434+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3756092081}) 2024-09-11T01:52:53.436 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.434+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1782656790}) 2024-09-11T01:52:53.436 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.434+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.436 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.434+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3756092081}) 2024-09-11T01:52:53.436 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.434+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3143830804}) 2024-09-11T01:52:53.438 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.434+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.438 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.434+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3412486901}) 2024-09-11T01:52:53.439 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.434+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1206847779}) 2024-09-11T01:52:53.439 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.434+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.439 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.434+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3412486901}) 2024-09-11T01:52:53.439 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.434+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3404990609}) 2024-09-11T01:52:53.439 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.435+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.439 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.435+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3637147730}) 2024-09-11T01:52:53.439 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.435+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=443101864}) 2024-09-11T01:52:53.439 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.435+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.439 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.435+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3637147730}) 2024-09-11T01:52:53.439 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.435+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1996079890}) 2024-09-11T01:52:53.439 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.436+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.439 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.436+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2227698242}) 2024-09-11T01:52:53.440 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.436+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1563724998}) 2024-09-11T01:52:53.440 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.436+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.440 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.436+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2227698242}) 2024-09-11T01:52:53.440 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.436+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=492330079}) 2024-09-11T01:52:53.440 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.436+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.440 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.436+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=412175913}) 2024-09-11T01:52:53.440 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.436+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4126371235}) 2024-09-11T01:52:53.440 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.436+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.440 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.436+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=412175913}) 2024-09-11T01:52:53.440 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.436+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2122724312}) 2024-09-11T01:52:53.440 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.437+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.440 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.437+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3737007544}) 2024-09-11T01:52:53.440 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.437+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1659729135}) 2024-09-11T01:52:53.440 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.437+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.441 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.437+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3737007544}) 2024-09-11T01:52:53.441 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.437+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2434176285}) 2024-09-11T01:52:53.446 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.437+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.446 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.437+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=401533993}) 2024-09-11T01:52:53.446 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.437+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=466075280}) 2024-09-11T01:52:53.447 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.437+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.447 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.437+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=401533993}) 2024-09-11T01:52:53.447 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.437+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1860859106}) 2024-09-11T01:52:53.447 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.438+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.447 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.438+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=550390663}) 2024-09-11T01:52:53.447 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.438+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4209490374}) 2024-09-11T01:52:53.447 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.438+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.447 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.438+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=550390663}) 2024-09-11T01:52:53.447 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.438+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4149717152}) 2024-09-11T01:52:53.447 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.439+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.447 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.439+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1168757830}) 2024-09-11T01:52:53.447 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.439+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2119009248}) 2024-09-11T01:52:53.447 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.439+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.447 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.439+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1168757830}) 2024-09-11T01:52:53.448 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.439+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2339392723}) 2024-09-11T01:52:53.448 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.439+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.448 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.439+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3007514466}) 2024-09-11T01:52:53.448 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.439+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=209260724}) 2024-09-11T01:52:53.448 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.439+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.448 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.439+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3007514466}) 2024-09-11T01:52:53.448 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.439+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4068486489}) 2024-09-11T01:52:53.448 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.440+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.448 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.440+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2602814111}) 2024-09-11T01:52:53.448 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.440+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2558790013}) 2024-09-11T01:52:53.448 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.440+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.448 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.440+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2602814111}) 2024-09-11T01:52:53.449 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.440+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1836181224}) 2024-09-11T01:52:53.449 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.440+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.449 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.440+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3280246016}) 2024-09-11T01:52:53.449 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.440+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3161370496}) 2024-09-11T01:52:53.449 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.440+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.449 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.440+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3280246016}) 2024-09-11T01:52:53.449 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.440+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3864116541}) 2024-09-11T01:52:53.449 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.441+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.449 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.441+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2626382415}) 2024-09-11T01:52:53.449 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.441+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3073064849}) 2024-09-11T01:52:53.449 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.441+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.449 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.441+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2626382415}) 2024-09-11T01:52:53.449 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.441+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3266235374}) 2024-09-11T01:52:53.450 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.442+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.450 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.442+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2947988160}) 2024-09-11T01:52:53.450 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.442+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=371979108}) 2024-09-11T01:52:53.450 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.442+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.450 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.442+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2947988160}) 2024-09-11T01:52:53.450 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.442+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3455712580}) 2024-09-11T01:52:53.450 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.442+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.450 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.442+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3773131805}) 2024-09-11T01:52:53.450 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.442+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2376420710}) 2024-09-11T01:52:53.450 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.442+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.450 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.442+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3773131805}) 2024-09-11T01:52:53.450 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.442+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=402712913}) 2024-09-11T01:52:53.450 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.443+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.450 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.443+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=748334007}) 2024-09-11T01:52:53.451 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.443+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=353484876}) 2024-09-11T01:52:53.451 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.443+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.451 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.443+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=748334007}) 2024-09-11T01:52:53.451 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.443+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1072782873}) 2024-09-11T01:52:53.451 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.443+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.451 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.443+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759219715}) 2024-09-11T01:52:53.451 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.443+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3638989547}) 2024-09-11T01:52:53.451 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.443+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.451 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.443+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759219715}) 2024-09-11T01:52:53.451 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.443+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4258645619}) 2024-09-11T01:52:53.451 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.444+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.451 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.444+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2237031221}) 2024-09-11T01:52:53.451 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.444+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2047913316}) 2024-09-11T01:52:53.452 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.444+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.452 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.444+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2237031221}) 2024-09-11T01:52:53.452 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.444+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=628342700}) 2024-09-11T01:52:53.452 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.445+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.452 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.445+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=953615171}) 2024-09-11T01:52:53.452 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.445+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1082807483}) 2024-09-11T01:52:53.452 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.445+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.452 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.445+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=953615171}) 2024-09-11T01:52:53.452 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.445+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1250407157}) 2024-09-11T01:52:53.460 INFO:tasks.rados.rados.0.smithi071.stdout:3553: finishing write tid 1 to smithi07134777-37 2024-09-11T01:52:53.460 INFO:tasks.rados.rados.0.smithi071.stdout:3553: finishing write tid 2 to smithi07134777-37 2024-09-11T01:52:53.460 INFO:tasks.rados.rados.0.smithi071.stdout:3553: finishing write tid 3 to smithi07134777-37 2024-09-11T01:52:53.460 INFO:tasks.rados.rados.0.smithi071.stdout:3554: finishing copy_from to smithi07134777-42 2024-09-11T01:52:53.460 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 2541 (ObjNum 1178 snap 376 seq_num 1178) dirty exists 2024-09-11T01:52:53.460 INFO:tasks.rados.rados.0.smithi071.stdout:3553: finishing write tid 4 to smithi07134777-37 2024-09-11T01:52:53.460 INFO:tasks.rados.rados.0.smithi071.stdout:3553: oid 37 updating version 0 to 2097 2024-09-11T01:52:53.460 INFO:tasks.rados.rados.0.smithi071.stdout:3553: oid 37 updating version 2097 to 2098 2024-09-11T01:52:53.461 INFO:tasks.rados.rados.0.smithi071.stdout:3553: oid 37 updating version 2098 to 2099 2024-09-11T01:52:53.461 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 2099 (ObjNum 1231 snap 402 seq_num 1231) dirty exists 2024-09-11T01:52:53.461 INFO:tasks.rados.rados.0.smithi071.stdout:3553: left oid 37 (ObjNum 1231 snap 402 seq_num 1231) 2024-09-11T01:52:53.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.445+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.445+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=275709509}) 2024-09-11T01:52:53.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.445+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3136004513}) 2024-09-11T01:52:53.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.445+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.445+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=275709509}) 2024-09-11T01:52:53.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.445+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2511696219}) 2024-09-11T01:52:53.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.446+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.446+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2386107088}) 2024-09-11T01:52:53.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.446+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1979968494}) 2024-09-11T01:52:53.468 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.446+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.469 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.446+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2386107088}) 2024-09-11T01:52:53.469 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.446+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4062881263}) 2024-09-11T01:52:53.469 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.447+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.469 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.447+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=745757863}) 2024-09-11T01:52:53.469 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.447+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2736399165}) 2024-09-11T01:52:53.469 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.447+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.469 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.447+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=745757863}) 2024-09-11T01:52:53.469 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.447+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=583943006}) 2024-09-11T01:52:53.469 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.447+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.469 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.447+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21679126}) 2024-09-11T01:52:53.469 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.447+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3328473886}) 2024-09-11T01:52:53.469 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.447+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.469 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.447+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21679126}) 2024-09-11T01:52:53.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.447+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3014060435}) 2024-09-11T01:52:53.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.448+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.448+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=608508091}) 2024-09-11T01:52:53.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.448+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1886608260}) 2024-09-11T01:52:53.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.448+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.448+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=608508091}) 2024-09-11T01:52:53.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.448+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3802912756}) 2024-09-11T01:52:53.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.448+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.448+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=900424056}) 2024-09-11T01:52:53.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.449+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=62459629}) 2024-09-11T01:52:53.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.449+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.449+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=900424056}) 2024-09-11T01:52:53.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.449+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=654367272}) 2024-09-11T01:52:53.470 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.449+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.471 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.449+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1584012614}) 2024-09-11T01:52:53.471 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.449+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1426759411}) 2024-09-11T01:52:53.471 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.449+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.471 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.449+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1584012614}) 2024-09-11T01:52:53.471 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.449+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1626329477}) 2024-09-11T01:52:53.471 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.450+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.471 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.450+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3445727542}) 2024-09-11T01:52:53.471 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.450+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=735886336}) 2024-09-11T01:52:53.471 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.450+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.471 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.450+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3445727542}) 2024-09-11T01:52:53.471 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.450+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3849946045}) 2024-09-11T01:52:53.471 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.450+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.471 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.450+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921227143}) 2024-09-11T01:52:53.471 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.450+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=546133112}) 2024-09-11T01:52:53.471 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.450+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.472 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.450+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921227143}) 2024-09-11T01:52:53.472 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.450+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1164388918}) 2024-09-11T01:52:53.472 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.451+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.472 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.451+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2911196250}) 2024-09-11T01:52:53.472 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.451+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=535295597}) 2024-09-11T01:52:53.472 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.451+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.472 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.451+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2911196250}) 2024-09-11T01:52:53.472 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.451+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1340762796}) 2024-09-11T01:52:53.472 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.452+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.472 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.452+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3777125692}) 2024-09-11T01:52:53.472 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.452+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=27701347}) 2024-09-11T01:52:53.472 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.452+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.472 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.452+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3777125692}) 2024-09-11T01:52:53.473 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.452+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=476677206}) 2024-09-11T01:52:53.473 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.452+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.473 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.452+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1247913581}) 2024-09-11T01:52:53.473 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.452+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3641191442}) 2024-09-11T01:52:53.473 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.452+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.473 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.452+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1247913581}) 2024-09-11T01:52:53.473 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.452+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=468586008}) 2024-09-11T01:52:53.473 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.453+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.473 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.453+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3468162939}) 2024-09-11T01:52:53.473 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.453+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=890949769}) 2024-09-11T01:52:53.473 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.453+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.473 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.453+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3468162939}) 2024-09-11T01:52:53.473 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.453+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1007521477}) 2024-09-11T01:52:53.473 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.454+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.454+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1624782350}) 2024-09-11T01:52:53.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.454+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3047612299}) 2024-09-11T01:52:53.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.454+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.454+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1624782350}) 2024-09-11T01:52:53.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.454+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2653960001}) 2024-09-11T01:52:53.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.454+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.454+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=916319684}) 2024-09-11T01:52:53.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.454+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1986999271}) 2024-09-11T01:52:53.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.454+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.454+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=916319684}) 2024-09-11T01:52:53.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.454+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=430081486}) 2024-09-11T01:52:53.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.455+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.474 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.455+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3512190412}) 2024-09-11T01:52:53.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.455+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=68710384}) 2024-09-11T01:52:53.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.455+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.455+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3512190412}) 2024-09-11T01:52:53.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.455+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3908019528}) 2024-09-11T01:52:53.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.456+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.456+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=683198831}) 2024-09-11T01:52:53.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.456+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=371518972}) 2024-09-11T01:52:53.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.456+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.456+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=683198831}) 2024-09-11T01:52:53.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.456+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3643683530}) 2024-09-11T01:52:53.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.457+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.457+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3144819648}) 2024-09-11T01:52:53.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.457+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2780137208}) 2024-09-11T01:52:53.475 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.457+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.476 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.457+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3144819648}) 2024-09-11T01:52:53.476 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.457+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2449980368}) 2024-09-11T01:52:53.476 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.457+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.476 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.457+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1816999323}) 2024-09-11T01:52:53.476 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.457+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=51088165}) 2024-09-11T01:52:53.476 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.457+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.476 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.457+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1816999323}) 2024-09-11T01:52:53.476 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.457+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=609858785}) 2024-09-11T01:52:53.476 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.458+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.476 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.458+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2956321232}) 2024-09-11T01:52:53.476 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.458+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2049140583}) 2024-09-11T01:52:53.476 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.458+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.476 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.458+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2956321232}) 2024-09-11T01:52:53.476 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.458+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4206656644}) 2024-09-11T01:52:53.477 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.459+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.477 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.459+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1754065273}) 2024-09-11T01:52:53.477 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.459+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=486302660}) 2024-09-11T01:52:53.477 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.459+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.477 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.459+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1754065273}) 2024-09-11T01:52:53.477 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.459+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3542989432}) 2024-09-11T01:52:53.477 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.460+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.477 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.460+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1063385026}) 2024-09-11T01:52:53.477 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.460+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2565387639}) 2024-09-11T01:52:53.477 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.460+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.477 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.460+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1063385026}) 2024-09-11T01:52:53.477 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.460+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2937855264}) 2024-09-11T01:52:53.477 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.460+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.477 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.460+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2039366535}) 2024-09-11T01:52:53.478 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.460+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1146049171}) 2024-09-11T01:52:53.478 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.460+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.478 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.460+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2039366535}) 2024-09-11T01:52:53.478 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.460+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=319737800}) 2024-09-11T01:52:53.478 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.461+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.478 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.461+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=973024096}) 2024-09-11T01:52:53.478 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.461+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=377622605}) 2024-09-11T01:52:53.478 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.461+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.478 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.461+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=973024096}) 2024-09-11T01:52:53.478 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.461+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2744542202}) 2024-09-11T01:52:53.478 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.461+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.478 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.461+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2361701636}) 2024-09-11T01:52:53.479 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.461+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2929417992}) 2024-09-11T01:52:53.479 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.461+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.479 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.461+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2361701636}) 2024-09-11T01:52:53.479 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.461+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4292039638}) 2024-09-11T01:52:53.479 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.462+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.479 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.462+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2698758075}) 2024-09-11T01:52:53.479 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.462+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1078818930}) 2024-09-11T01:52:53.479 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.462+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.479 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.462+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2698758075}) 2024-09-11T01:52:53.479 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.462+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3369100118}) 2024-09-11T01:52:53.479 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.463+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.479 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.463+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1783261316}) 2024-09-11T01:52:53.479 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.463+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2431488330}) 2024-09-11T01:52:53.479 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.463+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.480 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.463+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1783261316}) 2024-09-11T01:52:53.480 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.463+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3061173457}) 2024-09-11T01:52:53.480 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.463+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.480 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.463+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=115812030}) 2024-09-11T01:52:53.480 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.463+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2101321002}) 2024-09-11T01:52:53.480 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.463+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.480 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.463+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=115812030}) 2024-09-11T01:52:53.480 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.463+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3231954310}) 2024-09-11T01:52:53.480 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.464+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.480 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.464+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=719913295}) 2024-09-11T01:52:53.480 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.464+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1477732932}) 2024-09-11T01:52:53.480 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.464+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.480 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.464+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=719913295}) 2024-09-11T01:52:53.480 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.464+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4213583084}) 2024-09-11T01:52:53.481 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.465+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.481 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.465+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3905797535}) 2024-09-11T01:52:53.481 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.465+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3053173500}) 2024-09-11T01:52:53.481 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.465+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.481 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.465+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3905797535}) 2024-09-11T01:52:53.481 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.465+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2265562933}) 2024-09-11T01:52:53.481 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.466+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.481 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.466+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1964899440}) 2024-09-11T01:52:53.481 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.466+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2450188170}) 2024-09-11T01:52:53.481 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.466+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.481 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.466+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1964899440}) 2024-09-11T01:52:53.481 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.466+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2553635946}) 2024-09-11T01:52:53.483 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.467+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.483 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.467+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=822072596}) 2024-09-11T01:52:53.483 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.467+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2454859892}) 2024-09-11T01:52:53.483 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.467+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.484 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.467+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=822072596}) 2024-09-11T01:52:53.484 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.467+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1977125900}) 2024-09-11T01:52:53.484 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.468+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.484 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.468+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1134460830}) 2024-09-11T01:52:53.484 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.468+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=734075559}) 2024-09-11T01:52:53.484 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.468+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.484 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.468+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1134460830}) 2024-09-11T01:52:53.484 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.468+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=814384773}) 2024-09-11T01:52:53.484 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.468+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.484 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.468+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=951388324}) 2024-09-11T01:52:53.484 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.468+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1449996043}) 2024-09-11T01:52:53.484 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.468+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.484 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.468+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=951388324}) 2024-09-11T01:52:53.484 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.468+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4159852445}) 2024-09-11T01:52:53.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.937+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.937+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1274174553}) 2024-09-11T01:52:53.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.937+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2784484113}) 2024-09-11T01:52:53.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.937+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.937+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1274174553}) 2024-09-11T01:52:53.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.937+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2049749264}) 2024-09-11T01:52:53.985 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.983+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.985 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.984+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3511938179}) 2024-09-11T01:52:53.986 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.984+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3550579482}) 2024-09-11T01:52:53.986 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.984+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.986 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.984+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3511938179}) 2024-09-11T01:52:53.986 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.984+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1625726710}) 2024-09-11T01:52:53.990 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.988+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.990 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.988+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2387216282}) 2024-09-11T01:52:53.990 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.988+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1715168147}) 2024-09-11T01:52:53.990 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.988+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:53.990 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.988+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2387216282}) 2024-09-11T01:52:53.990 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:53.988+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3008215704}) 2024-09-11T01:52:54.008 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.006+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.008 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.006+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3929657982}) 2024-09-11T01:52:54.008 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.006+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=629895697}) 2024-09-11T01:52:54.008 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.006+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.008 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.006+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3929657982}) 2024-09-11T01:52:54.009 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.006+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3540181267}) 2024-09-11T01:52:54.085 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:52:54.085 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:52:54.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.195+0000 7efffa47f640 10 monclient: get_monmap_and_config 2024-09-11T01:52:54.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.195+0000 7efffa47f640 10 monclient: build_initial_monmap 2024-09-11T01:52:54.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.195+0000 7efffa47f640 10 monclient: monmap: 2024-09-11T01:52:54.197 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:54.197 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:54.197 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:54.196320+0000 2024-09-11T01:52:54.197 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:54.196320+0000 2024-09-11T01:52:54.197 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:54.197 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.198 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:54.198 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:54.198 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:54.198 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:54.198 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:54.198 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.196+0000 7efffa47f640 1 Processor -- start 2024-09-11T01:52:54.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.196+0000 7efffa47f640 1 -- start start 2024-09-11T01:52:54.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.196+0000 7efffa47f640 10 monclient: init 2024-09-11T01:52:54.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.196+0000 7efffa47f640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:54.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.196+0000 7efffa47f640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-11T01:52:54.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efffa47f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7efff40a1380 0x7efff40a1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efffa47f640 10 monclient(hunting): picked mon.noname-a con 0x7efff40a1380 addr v2:172.21.15.12:3300/0 2024-09-11T01:52:54.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efffa47f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efff40a0c90 0x7efff4135ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efffa47f640 10 monclient(hunting): picked mon.noname-b con 0x7efff40a0c90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:54.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efffa47f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efff4136020 addr v1:172.21.15.12:6789/0 2024-09-11T01:52:54.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efffa47f640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efff40a2b40 con 0x7efff4136020 2024-09-11T01:52:54.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efffa47f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:54.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efffa47f640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7efff409f9a0 con 0x7efff40a1380 2024-09-11T01:52:54.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efffa47f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:54.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efffa47f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7efff40a2230 con 0x7efff40a0c90 2024-09-11T01:52:54.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efffa47f640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:54.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efffa47f640 10 monclient(hunting): authenticate will time out at 2180.639171s 2024-09-11T01:52:54.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff3fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7efff40a1380 0x7efff40a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff37fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efff40a0c90 0x7efff4135ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff3fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7efff40a1380 0x7efff40a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60050/0 (socket says 172.21.15.12:60050) 2024-09-11T01:52:54.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff3fff640 1 -- 172.21.15.12:0/2006995986 learned_addr learned my addr 172.21.15.12:0/2006995986 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:54.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff3fff640 10 monclient(hunting): get_auth_request con 0x7efff40a1380 auth_method 0 2024-09-11T01:52:54.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:54.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:54.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:54.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff37fe640 10 monclient(hunting): get_auth_request con 0x7efff40a0c90 auth_method 0 2024-09-11T01:52:54.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:54.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:54.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:54.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:54.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:54.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:54.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:54.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:54.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:54.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff37fe640 10 monclient(hunting): handle_auth_done global_id 244340 payload 274 2024-09-11T01:52:54.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff37fe640 1 -- 172.21.15.12:0/2006995986 >> v2:172.21.15.12:3300/0 conn(0x7efff40a1380 msgr2=0x7efff40a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:54.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff37fe640 1 --2- 172.21.15.12:0/2006995986 >> v2:172.21.15.12:3300/0 conn(0x7efff40a1380 0x7efff40a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff37fe640 1 -- 172.21.15.12:0/2006995986 >> v1:172.21.15.12:6789/0 conn(0x7efff4136020 legacy=0x7efff413ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:54.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff37fe640 10 monclient: _finish_hunting 0 2024-09-11T01:52:54.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff37fe640 1 monclient: found mon.noname-b 2024-09-11T01:52:54.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:54.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:52:54.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.197+0000 7efff37fe640 1 -- 172.21.15.12:0/2006995986 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efff40a2510 con 0x7efff40a0c90 2024-09-11T01:52:54.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.200+0000 7efff37fe640 1 -- 172.21.15.12:0/2006995986 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efff40a0c90 msgr2=0x7efff4135ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:54.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.200+0000 7efff37fe640 1 --2- 172.21.15.12:0/2006995986 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efff40a0c90 0x7efff4135ae0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7effe80097a0 tx=0x7effe8029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c72c64ecca7d8ea7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:54.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.200+0000 7efff2ffd640 1 -- 172.21.15.12:0/2006995986 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe8037070 con 0x7efff40a0c90 2024-09-11T01:52:54.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.200+0000 7efff2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:54.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.200+0000 7efff2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:52:54.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.200+0000 7efff2ffd640 10 monclient: dump: 2024-09-11T01:52:54.206 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:54.206 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:54.206 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.206 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.206 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:54.206 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.206 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:54.206 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:54.207 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:54.207 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.200+0000 7efff2ffd640 10 monclient: _finish_auth 0 2024-09-11T01:52:54.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.200+0000 7efff2ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:52:54.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.200+0000 7efff2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:54.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.200+0000 7efff2ffd640 1 -- 172.21.15.12:0/2006995986 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effe8032440 con 0x7efff40a0c90 2024-09-11T01:52:54.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.200+0000 7efffa47f640 5 monclient: authenticate success, global_id 244340 2024-09-11T01:52:54.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.200+0000 7efffa47f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:52:54.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.200+0000 7efff2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:54.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.200+0000 7efff2ffd640 1 -- 172.21.15.12:0/2006995986 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe8032810 con 0x7efff40a0c90 2024-09-11T01:52:54.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.200+0000 7efffa47f640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:54.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.200+0000 7efffa47f640 4 set_mon_vals no callback set 2024-09-11T01:52:54.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.200+0000 7efff2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:54.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.200+0000 7efff2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:54.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.200+0000 7efff2ffd640 10 monclient: dump: 2024-09-11T01:52:54.208 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:54.208 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:54.208 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.208 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.208 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:54.208 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.208 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:54.208 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:54.208 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:54.208 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.201+0000 7efffa47f640 10 monclient: shutdown 2024-09-11T01:52:54.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.201+0000 7efffa47f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:54.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.201+0000 7efffa47f640 1 -- 172.21.15.12:0/2006995986 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efff40a0c90 msgr2=0x7efff4135ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:54.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.201+0000 7efffa47f640 1 --2- 172.21.15.12:0/2006995986 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efff40a0c90 0x7efff4135ae0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7effe80097a0 tx=0x7effe8029fd0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.201+0000 7efffa47f640 1 -- 172.21.15.12:0/2006995986 shutdown_connections 2024-09-11T01:52:54.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.201+0000 7efffa47f640 1 --2- 172.21.15.12:0/2006995986 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efff40a0c90 0x7efff4135ae0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.201+0000 7efffa47f640 1 --2- 172.21.15.12:0/2006995986 >> v2:172.21.15.12:3300/0 conn(0x7efff40a1380 0x7efff40a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.201+0000 7efffa47f640 1 -- 172.21.15.12:0/2006995986 >> 172.21.15.12:0/2006995986 conn(0x7efff40a2d90 msgr2=0x7efff40a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:54.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.201+0000 7efffa47f640 1 -- 172.21.15.12:0/2006995986 shutdown_connections 2024-09-11T01:52:54.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.201+0000 7efffa47f640 1 -- 172.21.15.12:0/2006995986 wait complete. 2024-09-11T01:52:54.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.201+0000 7efffa47f640 10 monclient: build_initial_monmap 2024-09-11T01:52:54.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.201+0000 7efffa47f640 10 monclient: monmap: 2024-09-11T01:52:54.209 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:54.209 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:54.209 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:54.210 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:54.210 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:54.210 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.210 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:54.210 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:54.210 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:54.210 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.211 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.202+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.211 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.202+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2408883105}) 2024-09-11T01:52:54.211 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.202+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2488075946}) 2024-09-11T01:52:54.211 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.202+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.211 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.202+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2408883105}) 2024-09-11T01:52:54.211 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.202+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=405052574}) 2024-09-11T01:52:54.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.202+0000 7efffa47f640 1 Processor -- start 2024-09-11T01:52:54.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.202+0000 7efffa47f640 1 -- start start 2024-09-11T01:52:54.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.202+0000 7efffa47f640 10 monclient: init 2024-09-11T01:52:54.212 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.202+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.212 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.202+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1068254268}) 2024-09-11T01:52:54.213 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.202+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1465890051}) 2024-09-11T01:52:54.213 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.202+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.213 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.202+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1068254268}) 2024-09-11T01:52:54.213 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.202+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1481745827}) 2024-09-11T01:52:54.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efffa47f640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:54.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efffa47f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:52:54.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efffa47f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efff40a0c90 0x7efff4124f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efffa47f640 10 monclient(hunting): picked mon.noname-c con 0x7efff40a0c90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:54.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efffa47f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efff40a1380 0x7efff41254d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efffa47f640 10 monclient(hunting): picked mon.noname-b con 0x7efff40a1380 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:54.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efffa47f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efff4136020 0x7efff4125a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efffa47f640 10 monclient(hunting): picked mon.noname-a con 0x7efff4136020 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:54.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efffa47f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:54.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efffa47f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7efff409ce20 con 0x7efff4136020 2024-09-11T01:52:54.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efffa47f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:54.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efffa47f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7efff409cca0 con 0x7efff40a1380 2024-09-11T01:52:54.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efffa47f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:54.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efffa47f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7efff40a0390 con 0x7efff40a0c90 2024-09-11T01:52:54.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efffa47f640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:54.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efffa47f640 10 monclient(hunting): authenticate will time out at 2180.645306s 2024-09-11T01:52:54.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff3fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efff40a0c90 0x7efff4124f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff3fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efff40a0c90 0x7efff4124f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:43692/0 (socket says 172.21.15.12:43692) 2024-09-11T01:52:54.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff3fff640 1 -- 172.21.15.12:0/2912913700 learned_addr learned my addr 172.21.15.12:0/2912913700 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:54.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff3fff640 10 monclient(hunting): get_auth_request con 0x7efff40a0c90 auth_method 0 2024-09-11T01:52:54.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:54.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:54.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:54.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff8a1c640 1 --2- 172.21.15.12:0/2912913700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efff4136020 0x7efff4125a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff8a1c640 10 monclient(hunting): get_auth_request con 0x7efff4136020 auth_method 0 2024-09-11T01:52:54.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff8a1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:54.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff8a1c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:54.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff8a1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:54.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff37fe640 1 --2- 172.21.15.12:0/2912913700 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efff40a1380 0x7efff41254d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:54.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:54.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:54.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff37fe640 10 monclient(hunting): get_auth_request con 0x7efff40a1380 auth_method 0 2024-09-11T01:52:54.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:54.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:54.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:54.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff8a1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:54.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff8a1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:54.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff8a1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:54.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:54.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:54.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:54.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff3fff640 10 monclient(hunting): handle_auth_done global_id 244343 payload 824 2024-09-11T01:52:54.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff3fff640 1 -- 172.21.15.12:0/2912913700 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efff40a1380 msgr2=0x7efff41254d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:54.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff3fff640 1 --2- 172.21.15.12:0/2912913700 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efff40a1380 0x7efff41254d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff3fff640 1 -- 172.21.15.12:0/2912913700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efff4136020 msgr2=0x7efff4125a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:54.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff3fff640 1 --2- 172.21.15.12:0/2912913700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efff4136020 0x7efff4125a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff3fff640 10 monclient: _finish_hunting 0 2024-09-11T01:52:54.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff3fff640 1 monclient: found mon.noname-c 2024-09-11T01:52:54.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:54.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:52:54.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.203+0000 7efff3fff640 1 -- 172.21.15.12:0/2912913700 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efff4127f40 con 0x7efff40a0c90 2024-09-11T01:52:54.219 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.203+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.219 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.203+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3962255129}) 2024-09-11T01:52:54.219 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.203+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1965516164}) 2024-09-11T01:52:54.219 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.203+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.219 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.203+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3962255129}) 2024-09-11T01:52:54.219 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.203+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2002828296}) 2024-09-11T01:52:54.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.206+0000 7efff3fff640 1 -- 172.21.15.12:0/2912913700 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efff40a0c90 msgr2=0x7efff4124f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:54.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.206+0000 7efff3fff640 1 --2- 172.21.15.12:0/2912913700 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efff40a0c90 0x7efff4124f90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7effe400e8b0 tx=0x7effe400ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:54.220 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.206+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.220 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.206+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3895174718}) 2024-09-11T01:52:54.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.206+0000 7efff0ff9640 1 -- 172.21.15.12:0/2912913700 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe4017070 con 0x7efff40a0c90 2024-09-11T01:52:54.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.206+0000 7efff0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:54.221 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.206+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2834066089}) 2024-09-11T01:52:54.224 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.206+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.206+0000 7efff0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:54.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.206+0000 7efff0ff9640 10 monclient: dump: 2024-09-11T01:52:54.225 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:54.225 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:54.225 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.225 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.225 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:54.225 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.225 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:54.225 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:54.225 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:54.225 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.206+0000 7efff0ff9640 10 monclient: _finish_auth 0 2024-09-11T01:52:54.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.206+0000 7efff0ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:52:54.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.206+0000 7efff0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:54.226 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.206+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3895174718}) 2024-09-11T01:52:54.226 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.206+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=884765237}) 2024-09-11T01:52:54.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.206+0000 7efff0ff9640 1 -- 172.21.15.12:0/2912913700 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effe400cb50 con 0x7efff40a0c90 2024-09-11T01:52:54.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.206+0000 7efffa47f640 5 monclient: authenticate success, global_id 244343 2024-09-11T01:52:54.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.206+0000 7efffa47f640 10 monclient: _renew_subs 2024-09-11T01:52:54.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.206+0000 7efffa47f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:54.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.206+0000 7efffa47f640 1 -- 172.21.15.12:0/2912913700 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efff4126a50 con 0x7efff40a0c90 2024-09-11T01:52:54.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.206+0000 7efff0ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:54.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.206+0000 7efff0ff9640 1 -- 172.21.15.12:0/2912913700 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe4013470 con 0x7efff40a0c90 2024-09-11T01:52:54.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.206+0000 7efff0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:54.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.206+0000 7efff0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:54.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.206+0000 7efff0ff9640 10 monclient: dump: 2024-09-11T01:52:54.228 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:54.228 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:54.228 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.228 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.228 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:54.228 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.228 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:54.228 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:54.228 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:54.228 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.206+0000 7efffa47f640 10 monclient: _renew_subs 2024-09-11T01:52:54.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.206+0000 7efffa47f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:54.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.206+0000 7efffa47f640 1 -- 172.21.15.12:0/2912913700 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efff4128510 con 0x7efff40a0c90 2024-09-11T01:52:54.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.206+0000 7efff27fc640 4 set_mon_vals no callback set 2024-09-11T01:52:54.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.207+0000 7efff0ff9640 1 -- 172.21.15.12:0/2912913700 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7effe4024060 con 0x7efff40a0c90 2024-09-11T01:52:54.229 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.207+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.229 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.207+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2452560950}) 2024-09-11T01:52:54.229 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.207+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1041877891}) 2024-09-11T01:52:54.230 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.207+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.230 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.207+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2452560950}) 2024-09-11T01:52:54.230 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.207+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2382086395}) 2024-09-11T01:52:54.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.208+0000 7efffa47f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:54.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.208+0000 7efffa47f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:54.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.208+0000 7efffa47f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:54.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.208+0000 7efff0ff9640 1 --2- 172.21.15.12:0/2912913700 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7effe003e260 0x7effe0040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.208+0000 7efffa47f640 1 -- 172.21.15.12:0/2912913700 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efff40008d0 con 0x7efff40a0c90 2024-09-11T01:52:54.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.208+0000 7efff0ff9640 1 -- 172.21.15.12:0/2912913700 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(803..803 src has 676..803) ==== 4815+0+0 (secure 0 0 0) 0x7effe4052cb0 con 0x7efff40a0c90 2024-09-11T01:52:54.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.208+0000 7efff37fe640 1 --2- 172.21.15.12:0/2912913700 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7effe003e260 0x7effe0040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.208+0000 7efff37fe640 10 monclient: get_auth_request con 0x7effe003e260 auth_method 0 2024-09-11T01:52:54.232 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.208+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.232 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.208+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4269951003}) 2024-09-11T01:52:54.232 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.208+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1816800187}) 2024-09-11T01:52:54.232 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.208+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.233 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.208+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4269951003}) 2024-09-11T01:52:54.233 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.208+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4237875241}) 2024-09-11T01:52:54.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.211+0000 7efff37fe640 1 --2- 172.21.15.12:0/2912913700 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7effe003e260 0x7effe0040720 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7efff40a1100 tx=0x7effe802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:54.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.213+0000 7efff0ff9640 1 -- 172.21.15.12:0/2912913700 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7effe4051690 con 0x7efff40a0c90 2024-09-11T01:52:54.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.213+0000 7efff0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:54.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.213+0000 7efff0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:54.240 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.213+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.240 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.213+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2486862486}) 2024-09-11T01:52:54.240 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.213+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1499835319}) 2024-09-11T01:52:54.240 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.213+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.240 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.213+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2486862486}) 2024-09-11T01:52:54.240 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.213+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=944013324}) 2024-09-11T01:52:54.240 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.214+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.240 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.214+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3513477874}) 2024-09-11T01:52:54.240 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.214+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3747212749}) 2024-09-11T01:52:54.240 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.214+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.240 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.214+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3513477874}) 2024-09-11T01:52:54.240 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.214+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1108490470}) 2024-09-11T01:52:54.241 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.224+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.241 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.224+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=925215493}) 2024-09-11T01:52:54.241 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.224+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=880085914}) 2024-09-11T01:52:54.241 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.224+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.241 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.224+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=925215493}) 2024-09-11T01:52:54.241 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.224+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1561033283}) 2024-09-11T01:52:54.241 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.225+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.242 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.225+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=740649279}) 2024-09-11T01:52:54.242 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.225+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=81042014}) 2024-09-11T01:52:54.242 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.225+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.242 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.225+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=740649279}) 2024-09-11T01:52:54.242 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.225+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1964220743}) 2024-09-11T01:52:54.242 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.225+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.242 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.225+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3432644742}) 2024-09-11T01:52:54.242 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.225+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3906197699}) 2024-09-11T01:52:54.242 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.225+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.242 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.225+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3432644742}) 2024-09-11T01:52:54.242 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.225+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3152949116}) 2024-09-11T01:52:54.242 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.226+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.243 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.226+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3446550765}) 2024-09-11T01:52:54.243 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.226+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=435009603}) 2024-09-11T01:52:54.243 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.226+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.243 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.226+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3446550765}) 2024-09-11T01:52:54.243 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.226+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3112543253}) 2024-09-11T01:52:54.243 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.226+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.243 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.226+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1501493602}) 2024-09-11T01:52:54.243 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.226+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4238714013}) 2024-09-11T01:52:54.243 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.227+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.243 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.227+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1501493602}) 2024-09-11T01:52:54.243 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.227+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3397366701}) 2024-09-11T01:52:54.243 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.227+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.243 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.228+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=82,mdsmap=1,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7} crc {logm=4166859070,mdsmap=1228700967,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1214808662}) 2024-09-11T01:52:54.243 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.228+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=81,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=1431881383,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=3589741893}) 2024-09-11T01:52:54.244 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.228+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.244 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.228+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=82,mdsmap=1,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7} crc {logm=4166859070,mdsmap=1228700967,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1214808662}) 2024-09-11T01:52:54.244 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.228+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=80,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8} crc {logm=110887520,mdsmap=3760429986,mgr=3108067518,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=3797193364}) 2024-09-11T01:52:54.244 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.229+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.244 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.229+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=2,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=70} crc {mgrstat=4147267112,mon_config_key=1809882168,monmap=2933275274,nvmeofgw=598337686,osd_metadata=825763671,osd_pg_creating=1375471433,osd_snap=2817250644}) 2024-09-11T01:52:54.244 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.229+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=2,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=68} crc {mgrstat=4147267112,mon_config_key=3330371543,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=1375471433,osd_snap=3339508480}) 2024-09-11T01:52:54.244 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.229+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.244 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.229+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=2,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=70} crc {mgrstat=4147267112,mon_config_key=1809882168,monmap=2933275274,nvmeofgw=598337686,osd_metadata=825763671,osd_pg_creating=1375471433,osd_snap=2817250644}) 2024-09-11T01:52:54.244 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.229+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=2,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=69} crc {mgrstat=4147267112,mon_config_key=3148220264,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=1375471433,osd_snap=3350083429}) 2024-09-11T01:52:54.244 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.230+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.244 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.230+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=85,osdmap=15} crc {osd_snap=497674667,osdmap=31708878}) 2024-09-11T01:52:54.244 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.230+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=95,osdmap=5} crc {osd_snap=3002117189,osdmap=1968204048}) 2024-09-11T01:52:54.245 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.230+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.245 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.230+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=85,osdmap=15} crc {osd_snap=497674667,osdmap=31708878}) 2024-09-11T01:52:54.245 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.230+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=94,osdmap=6} crc {osd_snap=1494499895,osdmap=2057132113}) 2024-09-11T01:52:54.245 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.231+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.245 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.231+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=864442828}) 2024-09-11T01:52:54.245 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.231+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4233455131}) 2024-09-11T01:52:54.245 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.231+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.245 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.231+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=864442828}) 2024-09-11T01:52:54.245 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.231+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1479363363}) 2024-09-11T01:52:54.245 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.236+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.245 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.236+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=81} crc {osdmap=2842568781}) 2024-09-11T01:52:54.245 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.236+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=84} crc {osdmap=871565639}) 2024-09-11T01:52:54.245 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.236+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:52:54.245 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.236+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=81} crc {osdmap=2842568781}) 2024-09-11T01:52:54.246 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:52:54.236+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=83} crc {osdmap=1424934451}) 2024-09-11T01:52:54.320 INFO:tasks.rados.rados.0.smithi071.stdout:3553: done (3 left) 2024-09-11T01:52:54.320 INFO:tasks.rados.rados.0.smithi071.stdout:3554: done (2 left) 2024-09-11T01:52:54.320 INFO:tasks.rados.rados.0.smithi071.stdout:3555: done (1 left) 2024-09-11T01:52:54.320 INFO:tasks.rados.rados.0.smithi071.stdout:3556: done (0 left) 2024-09-11T01:52:54.320 INFO:tasks.rados.rados.0.smithi071.stdout:3557: snap_remove snap 380 2024-09-11T01:52:54.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.477+0000 7efffa47f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:52:54.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.477+0000 7efffa47f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:52:54.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.477+0000 7efffa47f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:52:54.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.477+0000 7efffa47f640 1 -- 172.21.15.12:0/2912913700 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7efff4127880 con 0x7efff40a0c90 2024-09-11T01:52:54.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.478+0000 7efff0ff9640 1 -- 172.21.15.12:0/2912913700 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7effe4043630 con 0x7efff40a0c90 2024-09-11T01:52:54.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.478+0000 7efff0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:52:54.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.478+0000 7efff0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:54.480 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":220,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:52:54.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.479+0000 7efffa47f640 1 -- 172.21.15.12:0/2912913700 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7effe003e260 msgr2=0x7effe0040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:54.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.479+0000 7efffa47f640 1 --2- 172.21.15.12:0/2912913700 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7effe003e260 0x7effe0040720 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7efff40a1100 tx=0x7effe802e000 comp rx=0 tx=0).stop 2024-09-11T01:52:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.480+0000 7efffa47f640 10 monclient: shutdown 2024-09-11T01:52:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.480+0000 7efffa47f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.480+0000 7efffa47f640 1 -- 172.21.15.12:0/2912913700 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efff40a0c90 msgr2=0x7efff4124f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.480+0000 7efffa47f640 1 --2- 172.21.15.12:0/2912913700 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efff40a0c90 0x7efff4124f90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7effe400e8b0 tx=0x7effe400ed70 comp rx=0 tx=0).stop 2024-09-11T01:52:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.480+0000 7efffa47f640 1 -- 172.21.15.12:0/2912913700 shutdown_connections 2024-09-11T01:52:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.480+0000 7efffa47f640 1 --2- 172.21.15.12:0/2912913700 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7effe003e260 0x7effe0040720 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.480+0000 7efffa47f640 1 --2- 172.21.15.12:0/2912913700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efff4136020 0x7efff4125a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.480+0000 7efffa47f640 1 --2- 172.21.15.12:0/2912913700 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efff40a1380 0x7efff41254d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.480+0000 7efffa47f640 1 --2- 172.21.15.12:0/2912913700 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efff40a0c90 0x7efff4124f90 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.480+0000 7efffa47f640 1 -- 172.21.15.12:0/2912913700 >> 172.21.15.12:0/2912913700 conn(0x7efff40a2d90 msgr2=0x7efff40a5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.480+0000 7efffa47f640 1 -- 172.21.15.12:0/2912913700 shutdown_connections 2024-09-11T01:52:54.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.480+0000 7efffa47f640 1 -- 172.21.15.12:0/2912913700 wait complete. 2024-09-11T01:52:54.492 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:52:54.492 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:52:54.492 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:52:54.492 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:52:54.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.593+0000 7f86e6dad640 10 monclient: get_monmap_and_config 2024-09-11T01:52:54.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.593+0000 7f86e6dad640 10 monclient: build_initial_monmap 2024-09-11T01:52:54.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.593+0000 7f86e6dad640 10 monclient: monmap: 2024-09-11T01:52:54.595 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:54.595 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:54.595 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:54.594060+0000 2024-09-11T01:52:54.595 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:54.594060+0000 2024-09-11T01:52:54.595 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:54.595 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.595 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:54.595 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:54.595 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:54.595 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:54.596 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:54.596 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e6dad640 1 Processor -- start 2024-09-11T01:52:54.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e6dad640 1 -- start start 2024-09-11T01:52:54.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e6dad640 10 monclient: init 2024-09-11T01:52:54.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e6dad640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:54.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e6dad640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-11T01:52:54.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e6dad640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86e00a5690 0x7f86e00a5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e6dad640 10 monclient(hunting): picked mon.noname-b con 0x7f86e00a5690 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:54.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e6dad640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f86e00742e0 addr v1:172.21.15.12:6789/0 2024-09-11T01:52:54.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e6dad640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f86e00a5fb0 0x7f86e012f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e6dad640 10 monclient(hunting): picked mon.noname-a con 0x7f86e00a5fb0 addr v2:172.21.15.12:3300/0 2024-09-11T01:52:54.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e6dad640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86e0072e80 con 0x7f86e00742e0 2024-09-11T01:52:54.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e6dad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:54.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e6dad640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f86e0070320 con 0x7f86e00a5fb0 2024-09-11T01:52:54.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e6dad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:54.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e6dad640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f86e0073620 con 0x7f86e00a5690 2024-09-11T01:52:54.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e6dad640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:54.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e6dad640 10 monclient(hunting): authenticate will time out at 2181.036782s 2024-09-11T01:52:54.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e4b49640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86e00a5690 0x7f86e00a5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e534a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f86e00a5fb0 0x7f86e012f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e4b49640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86e00a5690 0x7f86e00a5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:43700/0 (socket says 172.21.15.12:43700) 2024-09-11T01:52:54.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e4b49640 1 -- 172.21.15.12:0/1570123569 learned_addr learned my addr 172.21.15.12:0/1570123569 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:54.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e4b49640 10 monclient(hunting): get_auth_request con 0x7f86e00a5690 auth_method 0 2024-09-11T01:52:54.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e4b49640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:54.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e4b49640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:54.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e4b49640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:54.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e534a640 10 monclient(hunting): get_auth_request con 0x7f86e00a5fb0 auth_method 0 2024-09-11T01:52:54.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e534a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:54.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e534a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:54.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e534a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:54.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e534a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:54.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.594+0000 7f86e534a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:54.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.595+0000 7f86e534a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:54.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.595+0000 7f86e4b49640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:54.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.595+0000 7f86e4b49640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:54.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.595+0000 7f86e4b49640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:54.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.595+0000 7f86e4b49640 10 monclient(hunting): handle_auth_done global_id 244346 payload 274 2024-09-11T01:52:54.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.595+0000 7f86e4b49640 1 -- 172.21.15.12:0/1570123569 >> v2:172.21.15.12:3300/0 conn(0x7f86e00a5fb0 msgr2=0x7f86e012f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:54.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.595+0000 7f86e4b49640 1 --2- 172.21.15.12:0/1570123569 >> v2:172.21.15.12:3300/0 conn(0x7f86e00a5fb0 0x7f86e012f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.595+0000 7f86e4b49640 1 -- 172.21.15.12:0/1570123569 >> v1:172.21.15.12:6789/0 conn(0x7f86e00742e0 legacy=0x7f86e012eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:54.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.595+0000 7f86e4b49640 10 monclient: _finish_hunting 0 2024-09-11T01:52:54.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.595+0000 7f86e4b49640 1 monclient: found mon.noname-b 2024-09-11T01:52:54.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.595+0000 7f86e4b49640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:54.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.595+0000 7f86e4b49640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:52:54.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.595+0000 7f86e4b49640 1 -- 172.21.15.12:0/1570123569 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e0073900 con 0x7f86e00a5690 2024-09-11T01:52:54.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.598+0000 7f86e4b49640 1 -- 172.21.15.12:0/1570123569 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86e00a5690 msgr2=0x7f86e00a5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:54.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.598+0000 7f86e4b49640 1 --2- 172.21.15.12:0/1570123569 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86e00a5690 0x7f86e00a5a70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f86d4009870 tx=0x7f86d402a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7f32bf57bcd1586 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:54.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.598+0000 7f86df7fe640 1 -- 172.21.15.12:0/1570123569 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d4037070 con 0x7f86e00a5690 2024-09-11T01:52:54.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.598+0000 7f86df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:54.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.598+0000 7f86df7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:52:54.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.598+0000 7f86df7fe640 10 monclient: dump: 2024-09-11T01:52:54.607 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:54.607 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:54.608 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.608 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.608 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:54.608 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.608 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:54.608 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:54.608 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:54.608 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.598+0000 7f86df7fe640 10 monclient: _finish_auth 0 2024-09-11T01:52:54.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.598+0000 7f86df7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:52:54.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.598+0000 7f86df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:54.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.598+0000 7f86df7fe640 1 -- 172.21.15.12:0/1570123569 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86d4032440 con 0x7f86e00a5690 2024-09-11T01:52:54.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.598+0000 7f86e6dad640 5 monclient: authenticate success, global_id 244346 2024-09-11T01:52:54.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.598+0000 7f86e6dad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:52:54.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.598+0000 7f86df7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:54.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.598+0000 7f86df7fe640 1 -- 172.21.15.12:0/1570123569 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d4032810 con 0x7f86e00a5690 2024-09-11T01:52:54.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.598+0000 7f86e6dad640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:54.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.598+0000 7f86e6dad640 4 set_mon_vals no callback set 2024-09-11T01:52:54.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.598+0000 7f86df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:54.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.598+0000 7f86df7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:54.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.598+0000 7f86df7fe640 10 monclient: dump: 2024-09-11T01:52:54.609 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:54.609 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:54.609 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.609 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.609 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:54.609 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.610 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:54.610 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:54.610 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:54.610 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.598+0000 7f86e6dad640 10 monclient: shutdown 2024-09-11T01:52:54.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.598+0000 7f86e6dad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:54.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.598+0000 7f86e6dad640 1 -- 172.21.15.12:0/1570123569 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86e00a5690 msgr2=0x7f86e00a5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:54.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.598+0000 7f86e6dad640 1 --2- 172.21.15.12:0/1570123569 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86e00a5690 0x7f86e00a5a70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f86d4009870 tx=0x7f86d402a0a0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.599+0000 7f86e6dad640 1 -- 172.21.15.12:0/1570123569 shutdown_connections 2024-09-11T01:52:54.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.599+0000 7f86e6dad640 1 --2- 172.21.15.12:0/1570123569 >> v2:172.21.15.12:3300/0 conn(0x7f86e00a5fb0 0x7f86e012f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.599+0000 7f86e6dad640 1 --2- 172.21.15.12:0/1570123569 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86e00a5690 0x7f86e00a5a70 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.599+0000 7f86e6dad640 1 -- 172.21.15.12:0/1570123569 >> 172.21.15.12:0/1570123569 conn(0x7f86e0068950 msgr2=0x7f86e0068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:54.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.599+0000 7f86e6dad640 1 -- 172.21.15.12:0/1570123569 shutdown_connections 2024-09-11T01:52:54.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.599+0000 7f86e6dad640 1 -- 172.21.15.12:0/1570123569 wait complete. 2024-09-11T01:52:54.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.599+0000 7f86e6dad640 10 monclient: build_initial_monmap 2024-09-11T01:52:54.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.599+0000 7f86e6dad640 10 monclient: monmap: 2024-09-11T01:52:54.611 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:54.611 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:54.611 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:54.611 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:54.611 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:54.611 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.611 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:54.611 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:54.611 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:54.611 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.599+0000 7f86e6dad640 1 Processor -- start 2024-09-11T01:52:54.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86e6dad640 1 -- start start 2024-09-11T01:52:54.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86e6dad640 10 monclient: init 2024-09-11T01:52:54.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86e6dad640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:54.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86e6dad640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:52:54.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86e6dad640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86e00742e0 0x7f86e011cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86e6dad640 10 monclient(hunting): picked mon.noname-b con 0x7f86e00742e0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:54.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86e6dad640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86e00a5690 0x7f86e011d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86e6dad640 10 monclient(hunting): picked mon.noname-a con 0x7f86e00a5690 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:54.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86e6dad640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86e00a5fb0 0x7f86e011d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86e6dad640 10 monclient(hunting): picked mon.noname-c con 0x7f86e00a5fb0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:54.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86e6dad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:54.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86e6dad640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f86e0069b00 con 0x7f86e00a5690 2024-09-11T01:52:54.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86e6dad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:54.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86e6dad640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f86e00a6410 con 0x7f86e00742e0 2024-09-11T01:52:54.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86e6dad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:54.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86e6dad640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f86e0069c80 con 0x7f86e00a5fb0 2024-09-11T01:52:54.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86e6dad640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:54.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86e6dad640 10 monclient(hunting): authenticate will time out at 2181.042728s 2024-09-11T01:52:54.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86dffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86e00a5690 0x7f86e011d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86dffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86e00a5690 0x7f86e011d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60080/0 (socket says 172.21.15.12:60080) 2024-09-11T01:52:54.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86e534a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86e00a5fb0 0x7f86e011d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86dffff640 1 -- 172.21.15.12:0/2042306848 learned_addr learned my addr 172.21.15.12:0/2042306848 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:54.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86dffff640 10 monclient(hunting): get_auth_request con 0x7f86e00a5690 auth_method 0 2024-09-11T01:52:54.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:54.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86dffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:54.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:54.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86e534a640 10 monclient(hunting): get_auth_request con 0x7f86e00a5fb0 auth_method 0 2024-09-11T01:52:54.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86e534a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:54.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86e534a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:54.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86e534a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:54.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:54.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:54.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:54.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86e534a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:54.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86e534a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:54.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.600+0000 7f86e534a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:54.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.601+0000 7f86e4b49640 1 --2- 172.21.15.12:0/2042306848 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86e00742e0 0x7f86e011cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.601+0000 7f86e4b49640 10 monclient(hunting): get_auth_request con 0x7f86e00742e0 auth_method 0 2024-09-11T01:52:54.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.601+0000 7f86e4b49640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:54.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.601+0000 7f86e4b49640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:54.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.601+0000 7f86e4b49640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:54.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.601+0000 7f86dffff640 10 monclient(hunting): handle_auth_done global_id 264138 payload 824 2024-09-11T01:52:54.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.601+0000 7f86dffff640 1 -- 172.21.15.12:0/2042306848 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86e00a5fb0 msgr2=0x7f86e011d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:54.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.601+0000 7f86dffff640 1 --2- 172.21.15.12:0/2042306848 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86e00a5fb0 0x7f86e011d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.601+0000 7f86dffff640 1 -- 172.21.15.12:0/2042306848 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86e00742e0 msgr2=0x7f86e011cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:54.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.601+0000 7f86dffff640 1 --2- 172.21.15.12:0/2042306848 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86e00742e0 0x7f86e011cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.601+0000 7f86dffff640 10 monclient: _finish_hunting 0 2024-09-11T01:52:54.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.601+0000 7f86dffff640 1 monclient: found mon.noname-a 2024-09-11T01:52:54.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.601+0000 7f86dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:54.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.601+0000 7f86dffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:52:54.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.601+0000 7f86dffff640 1 -- 172.21.15.12:0/2042306848 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e011f310 con 0x7f86e00a5690 2024-09-11T01:52:54.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.601+0000 7f86e534a640 1 --2- 172.21.15.12:0/2042306848 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86e00a5fb0 0x7f86e011d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:52:54.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.603+0000 7f86dffff640 1 -- 172.21.15.12:0/2042306848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86e00a5690 msgr2=0x7f86e011d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:54.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.603+0000 7f86dffff640 1 --2- 172.21.15.12:0/2042306848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86e00a5690 0x7f86e011d310 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f86cc00c400 tx=0x7f86cc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:54.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.603+0000 7f86dd7fa640 1 -- 172.21.15.12:0/2042306848 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86cc0119a0 con 0x7f86e00a5690 2024-09-11T01:52:54.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.603+0000 7f86dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:54.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.603+0000 7f86dd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:52:54.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.603+0000 7f86dd7fa640 10 monclient: dump: 2024-09-11T01:52:54.616 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:54.616 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:54.616 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.616 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.616 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:54.616 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.617 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:54.617 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:54.617 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:54.617 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.603+0000 7f86dd7fa640 10 monclient: _finish_auth 0 2024-09-11T01:52:54.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.603+0000 7f86dd7fa640 10 monclient: _check_auth_tickets 2024-09-11T01:52:54.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.603+0000 7f86dd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:54.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.603+0000 7f86e6dad640 5 monclient: authenticate success, global_id 264138 2024-09-11T01:52:54.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.603+0000 7f86e6dad640 10 monclient: _renew_subs 2024-09-11T01:52:54.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.603+0000 7f86dd7fa640 1 -- 172.21.15.12:0/2042306848 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86cc017070 con 0x7f86e00a5690 2024-09-11T01:52:54.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.603+0000 7f86e6dad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:52:54.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.603+0000 7f86e6dad640 1 -- 172.21.15.12:0/2042306848 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86e00a0260 con 0x7f86e00a5690 2024-09-11T01:52:54.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.603+0000 7f86dd7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:54.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.603+0000 7f86dd7fa640 1 -- 172.21.15.12:0/2042306848 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86cc011dc0 con 0x7f86e00a5690 2024-09-11T01:52:54.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.603+0000 7f86dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:54.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.603+0000 7f86dd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:52:54.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.603+0000 7f86dd7fa640 10 monclient: dump: 2024-09-11T01:52:54.618 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:54.618 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:54.618 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.618 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.618 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:54.618 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.618 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:54.618 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:54.618 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:54.618 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.603+0000 7f86e6dad640 10 monclient: _renew_subs 2024-09-11T01:52:54.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.603+0000 7f86deffd640 4 set_mon_vals no callback set 2024-09-11T01:52:54.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.603+0000 7f86e6dad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:52:54.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.603+0000 7f86e6dad640 1 -- 172.21.15.12:0/2042306848 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86e011f8d0 con 0x7f86e00a5690 2024-09-11T01:52:54.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.604+0000 7f86e6dad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:54.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.604+0000 7f86e6dad640 1 --2- 172.21.15.12:0/2042306848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86e01cacc0 0x7f86e01cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.604+0000 7f86e6dad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:54.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.604+0000 7f86e6dad640 1 -- 172.21.15.12:0/2042306848 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f86e00a5a70 con 0x7f86e01cacc0 2024-09-11T01:52:54.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.604+0000 7f86e4b49640 1 --2- 172.21.15.12:0/2042306848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86e01cacc0 0x7f86e01cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.604+0000 7f86e4b49640 10 monclient: get_auth_request con 0x7f86e01cacc0 auth_method 0 2024-09-11T01:52:54.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.604+0000 7f86e4b49640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:54.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.604+0000 7f86e4b49640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:54.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.604+0000 7f86e4b49640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:54.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.604+0000 7f86dd7fa640 1 -- 172.21.15.12:0/2042306848 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f86cc01c460 con 0x7f86e00a5690 2024-09-11T01:52:54.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.604+0000 7f86e4b49640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:54.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.604+0000 7f86e4b49640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:54.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.604+0000 7f86e4b49640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:54.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.604+0000 7f86e4b49640 10 monclient(hunting): handle_auth_done global_id 264141 payload 824 2024-09-11T01:52:54.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.604+0000 7f86e4b49640 1 -- 172.21.15.12:0/2042306848 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f86e012e8d0 con 0x7f86e01cacc0 2024-09-11T01:52:54.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.604+0000 7f86dd7fa640 1 --2- 172.21.15.12:0/2042306848 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f86c403e210 0x7f86c40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.604+0000 7f86dd7fa640 1 -- 172.21.15.12:0/2042306848 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(804..804 src has 676..804) ==== 4739+0+0 (secure 0 0 0) 0x7f86cc052ba0 con 0x7f86e00a5690 2024-09-11T01:52:54.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.605+0000 7f86e534a640 1 --2- 172.21.15.12:0/2042306848 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f86c403e210 0x7f86c40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.605+0000 7f86e534a640 10 monclient: get_auth_request con 0x7f86c403e210 auth_method 0 2024-09-11T01:52:54.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.607+0000 7f86e4b49640 1 -- 172.21.15.12:0/2042306848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86e01cacc0 msgr2=0x7f86e01cb0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:54.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.607+0000 7f86e4b49640 1 --2- 172.21.15.12:0/2042306848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86e01cacc0 0x7f86e01cb0a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f86d4034000 tx=0x7f86d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:54.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.607+0000 7f86dd7fa640 1 -- 172.21.15.12:0/2042306848 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d4037040 con 0x7f86e01cacc0 2024-09-11T01:52:54.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.607+0000 7f86dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:54.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.607+0000 7f86dd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:52:54.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.607+0000 7f86dd7fa640 10 monclient: dump: 2024-09-11T01:52:54.623 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:54.623 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:54.623 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.623 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.623 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:54.623 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.624 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:54.624 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:54.624 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:54.624 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.607+0000 7f86dd7fa640 1 -- 172.21.15.12:0/2042306848 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f86d403e460 con 0x7f86e01cacc0 2024-09-11T01:52:54.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.607+0000 7f86dd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:54.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.607+0000 7f86dd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:54.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.607+0000 7f86dd7fa640 1 -- 172.21.15.12:0/2042306848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86e01cacc0 msgr2=0x7f86e01cb0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:54.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.607+0000 7f86dd7fa640 1 --2- 172.21.15.12:0/2042306848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86e01cacc0 0x7f86e01cb0a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f86d4034000 tx=0x7f86d40046c0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.607+0000 7f86dd7fa640 1 -- 172.21.15.12:0/2042306848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86e01cacc0 msgr2=0x7f86e01cb0a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:54.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.607+0000 7f86dd7fa640 1 --2- 172.21.15.12:0/2042306848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86e01cacc0 0x7f86e01cb0a0 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f86d4034000 tx=0x7f86d40046c0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.607+0000 7f86e534a640 1 --2- 172.21.15.12:0/2042306848 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f86c403e210 0x7f86c40406d0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f86d800e740 tx=0x7f86d800b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:54.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.613+0000 7f86e6dad640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:52:54.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.613+0000 7f86e6dad640 1 --2- 172.21.15.12:0/2042306848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86e01cb5e0 0x7f86e009ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.613+0000 7f86e6dad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:54.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.613+0000 7f86e6dad640 1 -- 172.21.15.12:0/2042306848 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f86e011e0d0 con 0x7f86e01cb5e0 2024-09-11T01:52:54.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.613+0000 7f86e4b49640 1 --2- 172.21.15.12:0/2042306848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86e01cb5e0 0x7f86e009ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.613+0000 7f86e4b49640 10 monclient: get_auth_request con 0x7f86e01cb5e0 auth_method 0 2024-09-11T01:52:54.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.613+0000 7f86e4b49640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:54.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.613+0000 7f86e4b49640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:54.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.613+0000 7f86e4b49640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:54.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.613+0000 7f86e4b49640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:54.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.613+0000 7f86e4b49640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:54.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.613+0000 7f86e4b49640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:54.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.613+0000 7f86e4b49640 10 monclient(hunting): handle_auth_done global_id 264144 payload 824 2024-09-11T01:52:54.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.613+0000 7f86e4b49640 1 -- 172.21.15.12:0/2042306848 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f86e006d8c0 con 0x7f86e01cb5e0 2024-09-11T01:52:54.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.616+0000 7f86e4b49640 1 -- 172.21.15.12:0/2042306848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86e01cb5e0 msgr2=0x7f86e009ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:54.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.616+0000 7f86e4b49640 1 --2- 172.21.15.12:0/2042306848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86e01cb5e0 0x7f86e009ba20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f86d4000c00 tx=0x7f86d4008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:54.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.616+0000 7f86dd7fa640 1 -- 172.21.15.12:0/2042306848 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d4037040 con 0x7f86e01cb5e0 2024-09-11T01:52:54.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.616+0000 7f86dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:54.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.616+0000 7f86dd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:52:54.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.616+0000 7f86dd7fa640 10 monclient: dump: 2024-09-11T01:52:54.626 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:54.626 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:54.626 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.626 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.626 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:54.626 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.626 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:54.627 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:54.627 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:54.627 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.616+0000 7f86dd7fa640 1 -- 172.21.15.12:0/2042306848 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7f86d403cb60 con 0x7f86e01cb5e0 2024-09-11T01:52:54.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.616+0000 7f86dd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:52:54.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.616+0000 7f86dd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:54.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.616+0000 7f86dd7fa640 1 -- 172.21.15.12:0/2042306848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86e01cb5e0 msgr2=0x7f86e009ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:54.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.616+0000 7f86dd7fa640 1 --2- 172.21.15.12:0/2042306848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86e01cb5e0 0x7f86e009ba20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f86d4000c00 tx=0x7f86d4008030 comp rx=0 tx=0).stop 2024-09-11T01:52:54.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.616+0000 7f86dd7fa640 1 -- 172.21.15.12:0/2042306848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86e01cb5e0 msgr2=0x7f86e009ba20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:54.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.616+0000 7f86dd7fa640 1 --2- 172.21.15.12:0/2042306848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86e01cb5e0 0x7f86e009ba20 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f86d4000c00 tx=0x7f86d4008030 comp rx=0 tx=0).stop 2024-09-11T01:52:54.627 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:52:54.628 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:52:54.628 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:52:54.628 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:52:54.628 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 220, 2024-09-11T01:52:54.628 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 16322, 2024-09-11T01:52:54.628 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:52:54.628 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:52:54.628 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:52:54.628 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:52:54.628 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:54.628 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:52:54.628 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:52:54.629 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:52:54.629 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:52:54.629 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:54.629 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:54.629 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:54.629 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:54.629 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:54.629 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:54.629 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:54.629 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:54.629 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:54.629 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:54.629 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:54.630 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:54.630 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:52:54.630 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:52:54.630 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:54.630 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:54.630 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:54.630 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:54.630 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:54.630 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:54.632 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:54.632 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:54.632 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:54.632 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:54.632 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:54.632 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:54.632 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.632 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:52:54.632 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:52:54.633 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:52:54.633 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:52:54.633 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:52:54.633 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:52:54.633 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:52:54.633 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:52:54.633 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:52:54.633 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:52:54.633 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:52:54.633 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:52:54.633 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:52:54.633 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:52:54.633 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:52:54.634 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:52:54.634 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:52:54.634 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:54.634 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:54.634 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:54.634 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:54.634 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:54.634 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:54.634 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:54.634 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:54.634 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:54.634 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:54.634 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:54.634 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:54.635 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:52:54.635 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.635 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:52:54.635 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.635 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:52:54.635 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:52:54.635 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:54.635 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:54.635 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.635 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:54.635 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:52:54.635 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:54.636 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.636 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.636 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:54.636 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:52:54.636 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:54.636 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:54.636 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:54.636 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.636 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:52:54.636 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:52:54.637 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:54.637 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:54.637 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:54.637 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.637 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.637 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:52:54.637 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:52:54.637 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:54.637 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:54.637 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.637 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:54.637 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:52:54.637 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:54.638 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.638 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.638 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:54.638 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:52:54.638 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:54.638 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:54.638 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:54.638 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.638 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:52:54.638 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:52:54.638 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:54.638 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:54.638 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:54.638 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.639 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.639 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:52:54.639 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:52:54.639 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:54.639 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:54.639 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.639 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:54.639 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:52:54.639 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:54.639 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.639 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.639 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:54.639 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:52:54.640 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:54.640 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:54.640 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:54.640 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.640 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:52:54.640 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:52:54.640 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:54.640 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:54.640 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:54.640 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:54.640 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:54.640 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.640 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:52:54.641 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:52:54.641 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.641 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:54.641 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:54.641 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:52:54.641 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:54.641 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:54.641 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:52:54.641 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.641 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:54.641 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:54.641 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:52:54.641 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:54.641 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:54.641 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.642 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:52:54.642 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:52:54.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.616+0000 7f86e6dad640 1 -- 172.21.15.12:0/2042306848 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f86c403e210 msgr2=0x7f86c40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:54.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.616+0000 7f86e6dad640 1 --2- 172.21.15.12:0/2042306848 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f86c403e210 0x7f86c40406d0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f86d800e740 tx=0x7f86d800b040 comp rx=0 tx=0).stop 2024-09-11T01:52:54.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.616+0000 7f86e6dad640 10 monclient: shutdown 2024-09-11T01:52:54.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.616+0000 7f86e6dad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:54.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.616+0000 7f86e6dad640 1 -- 172.21.15.12:0/2042306848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86e00a5690 msgr2=0x7f86e011d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:54.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.616+0000 7f86e534a640 1 -- 172.21.15.12:0/2042306848 reap_dead start 2024-09-11T01:52:54.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.616+0000 7f86e6dad640 1 --2- 172.21.15.12:0/2042306848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86e00a5690 0x7f86e011d310 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f86cc00c400 tx=0x7f86cc00c8c0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.616+0000 7f86e6dad640 1 -- 172.21.15.12:0/2042306848 shutdown_connections 2024-09-11T01:52:54.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.616+0000 7f86e6dad640 1 --2- 172.21.15.12:0/2042306848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86e00a5690 0x7f86e011d310 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.616+0000 7f86e6dad640 1 -- 172.21.15.12:0/2042306848 >> 172.21.15.12:0/2042306848 conn(0x7f86e0068950 msgr2=0x7f86e0131080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:54.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.616+0000 7f86e6dad640 1 -- 172.21.15.12:0/2042306848 shutdown_connections 2024-09-11T01:52:54.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.616+0000 7f86e6dad640 1 -- 172.21.15.12:0/2042306848 wait complete. 2024-09-11T01:52:54.645 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:52:54.645 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:52:54.645 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:52:54.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.745+0000 7fe5e2cef640 10 monclient: get_monmap_and_config 2024-09-11T01:52:54.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.745+0000 7fe5e2cef640 10 monclient: build_initial_monmap 2024-09-11T01:52:54.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.745+0000 7fe5e2cef640 10 monclient: monmap: 2024-09-11T01:52:54.747 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:54.747 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:54.747 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:54.746091+0000 2024-09-11T01:52:54.747 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:54.746091+0000 2024-09-11T01:52:54.747 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:54.747 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.747 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:54.747 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:54.747 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:54.748 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:54.748 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:54.748 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5e2cef640 1 Processor -- start 2024-09-11T01:52:54.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5e2cef640 1 -- start start 2024-09-11T01:52:54.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5e2cef640 10 monclient: init 2024-09-11T01:52:54.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5e2cef640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:54.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5e2cef640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-11T01:52:54.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5e2cef640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe5dc0a03a0 0x7fe5dc09f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5e2cef640 10 monclient(hunting): picked mon.noname-c con 0x7fe5dc0a03a0 addr v2:172.21.15.71:3300/0 2024-09-11T01:52:54.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5e2cef640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe5dc0a2530 0x7fe5dc0a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5e2cef640 10 monclient(hunting): picked mon.noname-a con 0x7fe5dc0a2530 addr v2:172.21.15.12:3300/0 2024-09-11T01:52:54.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5e2cef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe5dc0a11d0 addr v1:172.21.15.12:6789/0 2024-09-11T01:52:54.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5e2cef640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5dc09e510 con 0x7fe5dc0a11d0 2024-09-11T01:52:54.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5e2cef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:54.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5e2cef640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fe5dc0a29b0 con 0x7fe5dc0a2530 2024-09-11T01:52:54.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5e2cef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:54.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5e2cef640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fe5dc0a0780 con 0x7fe5dc0a03a0 2024-09-11T01:52:54.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5e2cef640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:54.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5e2cef640 10 monclient(hunting): authenticate will time out at 2181.188669s 2024-09-11T01:52:54.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5dbfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe5dc0a2530 0x7fe5dc0a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5dbfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe5dc0a2530 0x7fe5dc0a0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60090/0 (socket says 172.21.15.12:60090) 2024-09-11T01:52:54.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5dbfff640 1 -- 172.21.15.12:0/3518403448 learned_addr learned my addr 172.21.15.12:0/3518403448 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:54.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5dbfff640 10 monclient(hunting): get_auth_request con 0x7fe5dc0a2530 auth_method 0 2024-09-11T01:52:54.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:54.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:54.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:54.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5e128c640 1 --1- 172.21.15.12:0/3518403448 >> v1:172.21.15.12:6789/0 conn(0x7fe5dc0a11d0 0x7fe5dc13aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59178/0 (socket says 172.21.15.12:59178) 2024-09-11T01:52:54.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:54.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5e0a8b640 1 --2- 172.21.15.12:0/3518403448 >> v2:172.21.15.71:3300/0 conn(0x7fe5dc0a03a0 0x7fe5dc09f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:54.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:54.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5e0a8b640 10 monclient(hunting): get_auth_request con 0x7fe5dc0a03a0 auth_method 0 2024-09-11T01:52:54.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5e0a8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:54.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5e0a8b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:54.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.746+0000 7fe5e0a8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:54.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.747+0000 7fe5e0a8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:54.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.747+0000 7fe5e0a8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:54.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.747+0000 7fe5e0a8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:54.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.747+0000 7fe5e0a8b640 10 monclient(hunting): handle_auth_done global_id 234358 payload 274 2024-09-11T01:52:54.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.747+0000 7fe5e0a8b640 1 -- 172.21.15.12:0/3518403448 >> v2:172.21.15.12:3300/0 conn(0x7fe5dc0a2530 msgr2=0x7fe5dc0a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:54.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.747+0000 7fe5e0a8b640 1 --2- 172.21.15.12:0/3518403448 >> v2:172.21.15.12:3300/0 conn(0x7fe5dc0a2530 0x7fe5dc0a0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.747+0000 7fe5e0a8b640 1 -- 172.21.15.12:0/3518403448 >> v1:172.21.15.12:6789/0 conn(0x7fe5dc0a11d0 legacy=0x7fe5dc13aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:54.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.747+0000 7fe5e0a8b640 10 monclient: _finish_hunting 0 2024-09-11T01:52:54.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.747+0000 7fe5e0a8b640 1 monclient: found mon.noname-c 2024-09-11T01:52:54.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.747+0000 7fe5e0a8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:54.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.747+0000 7fe5e0a8b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:52:54.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.747+0000 7fe5e0a8b640 1 -- 172.21.15.12:0/3518403448 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5dc09edb0 con 0x7fe5dc0a03a0 2024-09-11T01:52:54.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.750+0000 7fe5e0a8b640 1 -- 172.21.15.12:0/3518403448 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe5dc0a03a0 msgr2=0x7fe5dc09f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:54.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.750+0000 7fe5e0a8b640 1 --2- 172.21.15.12:0/3518403448 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe5dc0a03a0 0x7fe5dc09f1d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc06cd50 tx=0x7fe5d002a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7358b83e3deefc21 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:54.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.750+0000 7fe5db7fe640 1 -- 172.21.15.12:0/3518403448 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d0037070 con 0x7fe5dc0a03a0 2024-09-11T01:52:54.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.750+0000 7fe5db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.750+0000 7fe5db7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:52:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.750+0000 7fe5db7fe640 10 monclient: dump: 2024-09-11T01:52:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:54.760 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.750+0000 7fe5db7fe640 10 monclient: _finish_auth 0 2024-09-11T01:52:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.750+0000 7fe5db7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:52:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.750+0000 7fe5db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.750+0000 7fe5e2cef640 5 monclient: authenticate success, global_id 234358 2024-09-11T01:52:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.750+0000 7fe5e2cef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:52:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.750+0000 7fe5db7fe640 1 -- 172.21.15.12:0/3518403448 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5d0032440 con 0x7fe5dc0a03a0 2024-09-11T01:52:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.750+0000 7fe5db7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.750+0000 7fe5e2cef640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.750+0000 7fe5e2cef640 4 set_mon_vals no callback set 2024-09-11T01:52:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.750+0000 7fe5db7fe640 1 -- 172.21.15.12:0/3518403448 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d0032810 con 0x7fe5dc0a03a0 2024-09-11T01:52:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.750+0000 7fe5db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.750+0000 7fe5db7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:52:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.750+0000 7fe5db7fe640 10 monclient: dump: 2024-09-11T01:52:54.761 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:54.762 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:54.762 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.762 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.762 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:54.762 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.762 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:54.762 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:54.762 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:54.762 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.751+0000 7fe5e2cef640 10 monclient: shutdown 2024-09-11T01:52:54.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.751+0000 7fe5e2cef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:54.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.751+0000 7fe5e2cef640 1 -- 172.21.15.12:0/3518403448 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe5dc0a03a0 msgr2=0x7fe5dc09f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:54.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.751+0000 7fe5e2cef640 1 --2- 172.21.15.12:0/3518403448 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe5dc0a03a0 0x7fe5dc09f1d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc06cd50 tx=0x7fe5d002a000 comp rx=0 tx=0).stop 2024-09-11T01:52:54.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.751+0000 7fe5e2cef640 1 -- 172.21.15.12:0/3518403448 shutdown_connections 2024-09-11T01:52:54.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.751+0000 7fe5e2cef640 1 --2- 172.21.15.12:0/3518403448 >> v2:172.21.15.12:3300/0 conn(0x7fe5dc0a2530 0x7fe5dc0a0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.751+0000 7fe5e2cef640 1 --2- 172.21.15.12:0/3518403448 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe5dc0a03a0 0x7fe5dc09f1d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.751+0000 7fe5e2cef640 1 -- 172.21.15.12:0/3518403448 >> 172.21.15.12:0/3518403448 conn(0x7fe5dc0a2f70 msgr2=0x7fe5dc0a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:54.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.751+0000 7fe5e2cef640 1 -- 172.21.15.12:0/3518403448 shutdown_connections 2024-09-11T01:52:54.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.751+0000 7fe5e2cef640 1 -- 172.21.15.12:0/3518403448 wait complete. 2024-09-11T01:52:54.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.752+0000 7fe5e2cef640 10 monclient: build_initial_monmap 2024-09-11T01:52:54.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.752+0000 7fe5e2cef640 10 monclient: monmap: 2024-09-11T01:52:54.763 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:54.763 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:54.763 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:54.763 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:54.763 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:54.764 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.764 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:54.764 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:54.764 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:54.764 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.752+0000 7fe5e2cef640 1 Processor -- start 2024-09-11T01:52:54.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.752+0000 7fe5e2cef640 1 -- start start 2024-09-11T01:52:54.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.752+0000 7fe5e2cef640 10 monclient: init 2024-09-11T01:52:54.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e2cef640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:54.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e2cef640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:52:54.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e2cef640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe5dc0a03a0 0x7fe5dc1c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e2cef640 10 monclient(hunting): picked mon.noname-b con 0x7fe5dc0a03a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:54.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e2cef640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5dc0a11d0 0x7fe5dc1bc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e2cef640 10 monclient(hunting): picked mon.noname-a con 0x7fe5dc0a11d0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e2cef640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5dc0a2530 0x7fe5dc1bccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e2cef640 10 monclient(hunting): picked mon.noname-c con 0x7fe5dc0a2530 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e2cef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e2cef640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe5dc0a19f0 con 0x7fe5dc0a11d0 2024-09-11T01:52:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e2cef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e2cef640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe5dc0a1870 con 0x7fe5dc0a03a0 2024-09-11T01:52:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e2cef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e2cef640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe5dc09ca30 con 0x7fe5dc0a2530 2024-09-11T01:52:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e2cef640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e2cef640 10 monclient(hunting): authenticate will time out at 2181.195190s 2024-09-11T01:52:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5dbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5dc0a11d0 0x7fe5dc1bc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5dbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5dc0a11d0 0x7fe5dc1bc790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60102/0 (socket says 172.21.15.12:60102) 2024-09-11T01:52:54.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5dbfff640 1 -- 172.21.15.12:0/4017709910 learned_addr learned my addr 172.21.15.12:0/4017709910 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5dbfff640 10 monclient(hunting): get_auth_request con 0x7fe5dc0a11d0 auth_method 0 2024-09-11T01:52:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e128c640 1 --2- 172.21.15.12:0/4017709910 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5dc0a2530 0x7fe5dc1bccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e128c640 10 monclient(hunting): get_auth_request con 0x7fe5dc0a2530 auth_method 0 2024-09-11T01:52:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e128c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e128c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e128c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e128c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:54.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e128c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:54.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e0a8b640 1 --2- 172.21.15.12:0/4017709910 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe5dc0a03a0 0x7fe5dc1c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e128c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:54.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e0a8b640 10 monclient(hunting): get_auth_request con 0x7fe5dc0a03a0 auth_method 0 2024-09-11T01:52:54.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e0a8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:54.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e0a8b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:54.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e0a8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:54.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5dbfff640 10 monclient(hunting): handle_auth_done global_id 264150 payload 824 2024-09-11T01:52:54.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5dbfff640 1 -- 172.21.15.12:0/4017709910 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5dc0a2530 msgr2=0x7fe5dc1bccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:54.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5dbfff640 1 --2- 172.21.15.12:0/4017709910 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5dc0a2530 0x7fe5dc1bccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5dbfff640 1 -- 172.21.15.12:0/4017709910 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe5dc0a03a0 msgr2=0x7fe5dc1c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:54.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5dbfff640 1 --2- 172.21.15.12:0/4017709910 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe5dc0a03a0 0x7fe5dc1c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5dbfff640 10 monclient: _finish_hunting 0 2024-09-11T01:52:54.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5dbfff640 1 monclient: found mon.noname-a 2024-09-11T01:52:54.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:54.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5dbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:52:54.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5dbfff640 1 -- 172.21.15.12:0/4017709910 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5dc1aa140 con 0x7fe5dc0a11d0 2024-09-11T01:52:54.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.753+0000 7fe5e0a8b640 1 --2- 172.21.15.12:0/4017709910 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe5dc0a03a0 0x7fe5dc1c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:52:54.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.755+0000 7fe5dbfff640 1 -- 172.21.15.12:0/4017709910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5dc0a11d0 msgr2=0x7fe5dc1bc790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:54.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.755+0000 7fe5dbfff640 1 --2- 172.21.15.12:0/4017709910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5dc0a11d0 0x7fe5dc1bc790 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe5c800daf0 tx=0x7fe5c8009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:54.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.755+0000 7fe5d97fa640 1 -- 172.21.15.12:0/4017709910 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c801c070 con 0x7fe5dc0a11d0 2024-09-11T01:52:54.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.755+0000 7fe5d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:54.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.755+0000 7fe5d97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:52:54.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.755+0000 7fe5d97fa640 10 monclient: dump: 2024-09-11T01:52:54.768 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:54.768 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:54.768 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.768 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.769 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:54.769 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.769 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:54.769 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:54.769 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:54.769 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.755+0000 7fe5d97fa640 10 monclient: _finish_auth 0 2024-09-11T01:52:54.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.755+0000 7fe5d97fa640 10 monclient: _check_auth_tickets 2024-09-11T01:52:54.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.755+0000 7fe5d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:54.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.756+0000 7fe5e2cef640 5 monclient: authenticate success, global_id 264150 2024-09-11T01:52:54.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.756+0000 7fe5d97fa640 1 -- 172.21.15.12:0/4017709910 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5c8009d70 con 0x7fe5dc0a11d0 2024-09-11T01:52:54.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.756+0000 7fe5d97fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:54.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.756+0000 7fe5d97fa640 1 -- 172.21.15.12:0/4017709910 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c8017610 con 0x7fe5dc0a11d0 2024-09-11T01:52:54.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.756+0000 7fe5e2cef640 10 monclient: _renew_subs 2024-09-11T01:52:54.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.756+0000 7fe5e2cef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:52:54.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.756+0000 7fe5e2cef640 1 -- 172.21.15.12:0/4017709910 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5dc1a8e00 con 0x7fe5dc0a11d0 2024-09-11T01:52:54.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.756+0000 7fe5d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:54.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.756+0000 7fe5d97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:52:54.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.756+0000 7fe5daffd640 4 set_mon_vals no callback set 2024-09-11T01:52:54.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.756+0000 7fe5d97fa640 10 monclient: dump: 2024-09-11T01:52:54.770 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:54.770 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:54.770 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.770 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.770 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:54.770 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.770 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:54.770 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:54.771 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:54.771 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.756+0000 7fe5e2cef640 10 monclient: _renew_subs 2024-09-11T01:52:54.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.756+0000 7fe5e2cef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:52:54.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.756+0000 7fe5e2cef640 1 -- 172.21.15.12:0/4017709910 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5dc1a9960 con 0x7fe5dc0a11d0 2024-09-11T01:52:54.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.756+0000 7fe5e2cef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:54.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.756+0000 7fe5e2cef640 1 --2- 172.21.15.12:0/4017709910 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5dc1ccf10 0x7fe5dc1cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.756+0000 7fe5e2cef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:54.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.756+0000 7fe5e2cef640 1 -- 172.21.15.12:0/4017709910 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe5dc09d4f0 con 0x7fe5dc1ccf10 2024-09-11T01:52:54.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.756+0000 7fe5d97fa640 1 -- 172.21.15.12:0/4017709910 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fe5c80177f0 con 0x7fe5dc0a11d0 2024-09-11T01:52:54.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.757+0000 7fe5e0a8b640 1 --2- 172.21.15.12:0/4017709910 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5dc1ccf10 0x7fe5dc1cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.757+0000 7fe5e0a8b640 10 monclient: get_auth_request con 0x7fe5dc1ccf10 auth_method 0 2024-09-11T01:52:54.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.757+0000 7fe5e0a8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:54.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.757+0000 7fe5e0a8b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:54.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.757+0000 7fe5e0a8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:54.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.757+0000 7fe5e0a8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:54.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.757+0000 7fe5e0a8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:54.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.757+0000 7fe5e0a8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:54.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.757+0000 7fe5d97fa640 1 --2- 172.21.15.12:0/4017709910 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe5c003e210 0x7fe5c00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.757+0000 7fe5d97fa640 1 -- 172.21.15.12:0/4017709910 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(804..804 src has 676..804) ==== 4739+0+0 (secure 0 0 0) 0x7fe5c8052f60 con 0x7fe5dc0a11d0 2024-09-11T01:52:54.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.757+0000 7fe5e0a8b640 10 monclient(hunting): handle_auth_done global_id 244355 payload 824 2024-09-11T01:52:54.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.757+0000 7fe5e0a8b640 1 -- 172.21.15.12:0/4017709910 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe5dc1aade0 con 0x7fe5dc1ccf10 2024-09-11T01:52:54.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.757+0000 7fe5e128c640 1 --2- 172.21.15.12:0/4017709910 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe5c003e210 0x7fe5c00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.757+0000 7fe5e128c640 10 monclient: get_auth_request con 0x7fe5c003e210 auth_method 0 2024-09-11T01:52:54.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.759+0000 7fe5e0a8b640 1 -- 172.21.15.12:0/4017709910 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5dc1ccf10 msgr2=0x7fe5dc1cd2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:54.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.759+0000 7fe5e0a8b640 1 --2- 172.21.15.12:0/4017709910 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5dc1ccf10 0x7fe5dc1cd2f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0034000 tx=0x7fe5d00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:54.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.759+0000 7fe5d97fa640 1 -- 172.21.15.12:0/4017709910 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d0037040 con 0x7fe5dc1ccf10 2024-09-11T01:52:54.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.759+0000 7fe5d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:54.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.759+0000 7fe5d97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:54.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.759+0000 7fe5d97fa640 10 monclient: dump: 2024-09-11T01:52:54.775 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:54.775 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:54.775 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.775 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.775 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:54.775 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.775 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:54.775 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:54.776 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:54.776 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.760+0000 7fe5d97fa640 1 -- 172.21.15.12:0/4017709910 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe5d003c640 con 0x7fe5dc1ccf10 2024-09-11T01:52:54.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.760+0000 7fe5d97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:54.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.760+0000 7fe5d97fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:54.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.760+0000 7fe5d97fa640 1 -- 172.21.15.12:0/4017709910 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5dc1ccf10 msgr2=0x7fe5dc1cd2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:54.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.760+0000 7fe5d97fa640 1 --2- 172.21.15.12:0/4017709910 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5dc1ccf10 0x7fe5dc1cd2f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0034000 tx=0x7fe5d00046c0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.760+0000 7fe5e128c640 1 --2- 172.21.15.12:0/4017709910 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe5c003e210 0x7fe5c00406d0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc074410 tx=0x7fe5d4004ee0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:54.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.760+0000 7fe5d97fa640 1 -- 172.21.15.12:0/4017709910 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5dc1ccf10 msgr2=0x7fe5dc1cd2f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:54.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.760+0000 7fe5d97fa640 1 --2- 172.21.15.12:0/4017709910 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5dc1ccf10 0x7fe5dc1cd2f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.766+0000 7fe5e2cef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:52:54.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.766+0000 7fe5e2cef640 1 --2- 172.21.15.12:0/4017709910 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5dc1cd830 0x7fe5dc1a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.766+0000 7fe5e2cef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:54.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.766+0000 7fe5e2cef640 1 -- 172.21.15.12:0/4017709910 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe5dc1a9bc0 con 0x7fe5dc1cd830 2024-09-11T01:52:54.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.766+0000 7fe5e0a8b640 1 --2- 172.21.15.12:0/4017709910 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5dc1cd830 0x7fe5dc1a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.766+0000 7fe5e0a8b640 10 monclient: get_auth_request con 0x7fe5dc1cd830 auth_method 0 2024-09-11T01:52:54.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.766+0000 7fe5e0a8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:54.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.766+0000 7fe5e0a8b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:54.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.766+0000 7fe5e0a8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:54.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.766+0000 7fe5e0a8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:54.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.766+0000 7fe5e0a8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:54.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.766+0000 7fe5e0a8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:54.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.766+0000 7fe5e0a8b640 10 monclient(hunting): handle_auth_done global_id 244358 payload 824 2024-09-11T01:52:54.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.766+0000 7fe5e0a8b640 1 -- 172.21.15.12:0/4017709910 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe5dc09ce40 con 0x7fe5dc1cd830 2024-09-11T01:52:54.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.768+0000 7fe5e0a8b640 1 -- 172.21.15.12:0/4017709910 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5dc1cd830 msgr2=0x7fe5dc1a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:54.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.768+0000 7fe5e0a8b640 1 --2- 172.21.15.12:0/4017709910 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5dc1cd830 0x7fe5dc1a36e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0000c00 tx=0x7fe5d0008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:54.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.768+0000 7fe5d97fa640 1 -- 172.21.15.12:0/4017709910 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d0049e20 con 0x7fe5dc1cd830 2024-09-11T01:52:54.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.768+0000 7fe5d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:54.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.769+0000 7fe5d97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:52:54.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.769+0000 7fe5d97fa640 10 monclient: dump: 2024-09-11T01:52:54.778 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:54.778 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:54.778 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.778 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.778 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:54.778 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.778 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:54.778 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:54.778 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:54.779 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.769+0000 7fe5d97fa640 1 -- 172.21.15.12:0/4017709910 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7fe5d0049020 con 0x7fe5dc1cd830 2024-09-11T01:52:54.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.769+0000 7fe5d97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:52:54.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.769+0000 7fe5d97fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:54.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.769+0000 7fe5d97fa640 1 -- 172.21.15.12:0/4017709910 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5dc1cd830 msgr2=0x7fe5dc1a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:54.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.769+0000 7fe5d97fa640 1 --2- 172.21.15.12:0/4017709910 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5dc1cd830 0x7fe5dc1a36e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0000c00 tx=0x7fe5d0008030 comp rx=0 tx=0).stop 2024-09-11T01:52:54.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.769+0000 7fe5d97fa640 1 -- 172.21.15.12:0/4017709910 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5dc1cd830 msgr2=0x7fe5dc1a36e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:54.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.769+0000 7fe5d97fa640 1 --2- 172.21.15.12:0/4017709910 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5dc1cd830 0x7fe5dc1a36e0 secure :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0000c00 tx=0x7fe5d0008030 comp rx=0 tx=0).stop 2024-09-11T01:52:54.779 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:52:54.779 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:52:54.780 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:52:54.780 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:52:54.780 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 220, 2024-09-11T01:52:54.780 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 128218, 2024-09-11T01:52:54.780 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:52:54.780 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:52:54.780 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:52:54.780 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:52:54.780 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:54.780 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:52:54.780 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:52:54.780 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:52:54.781 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:52:54.781 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:54.781 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:54.781 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:54.781 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:54.781 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:54.781 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:54.781 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:54.781 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:54.781 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:54.781 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:54.781 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:54.781 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:54.782 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:52:54.782 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:52:54.782 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:54.782 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:54.782 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:54.782 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:54.782 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:54.782 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:54.782 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:54.782 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:54.782 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:54.782 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:54.782 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:54.782 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:54.783 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.783 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:52:54.783 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:52:54.783 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:52:54.783 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:52:54.783 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:52:54.783 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:52:54.783 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:52:54.783 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:52:54.783 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:52:54.783 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:52:54.783 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:52:54.783 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:52:54.784 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:52:54.784 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:52:54.784 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:52:54.784 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:52:54.784 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:52:54.784 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:54.784 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:54.784 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:54.784 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:54.784 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:54.784 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:54.784 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:54.784 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:54.784 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:54.785 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:54.785 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:54.785 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:54.785 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:52:54.785 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.785 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:52:54.785 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.785 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:52:54.785 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:52:54.785 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:54.785 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:54.785 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.785 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:54.786 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:52:54.786 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:54.786 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.786 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.786 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:54.786 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:52:54.786 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:54.786 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:54.786 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:54.786 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.786 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:52:54.786 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:52:54.786 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:54.787 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:54.787 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:54.787 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.787 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.787 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:52:54.787 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:52:54.787 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:54.787 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:54.787 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.787 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:54.787 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:52:54.787 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:54.787 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.788 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.788 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:54.788 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:52:54.788 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:54.788 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:54.788 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:54.788 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.788 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:52:54.788 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:52:54.788 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:54.788 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:54.788 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:54.788 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.788 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.789 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:52:54.789 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:52:54.789 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:54.789 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:54.789 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.789 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:54.789 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:52:54.789 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:54.789 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.789 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.789 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:54.789 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:52:54.789 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:54.789 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:54.790 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:54.790 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.790 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:52:54.790 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:52:54.790 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:54.790 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:54.790 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:54.790 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:54.790 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:54.790 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.790 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:52:54.790 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:52:54.790 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.790 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:54.791 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:54.791 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:52:54.791 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:54.791 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:54.791 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:52:54.791 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.791 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:54.791 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:54.791 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:52:54.791 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:54.791 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:54.791 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:52:54.791 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.791 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:54.792 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:54.792 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:52:54.792 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:54.792 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:54.792 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:52:54.792 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.792 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:54.792 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:54.792 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:52:54.792 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:54.792 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:54.792 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.792 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:52:54.792 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:52:54.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.769+0000 7fe5e2cef640 1 -- 172.21.15.12:0/4017709910 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe5c003e210 msgr2=0x7fe5c00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:54.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.769+0000 7fe5e2cef640 1 --2- 172.21.15.12:0/4017709910 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe5c003e210 0x7fe5c00406d0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc074410 tx=0x7fe5d4004ee0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.769+0000 7fe5e2cef640 10 monclient: shutdown 2024-09-11T01:52:54.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.769+0000 7fe5e2cef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:54.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.769+0000 7fe5e2cef640 1 -- 172.21.15.12:0/4017709910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5dc0a11d0 msgr2=0x7fe5dc1bc790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:54.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.769+0000 7fe5e2cef640 1 --2- 172.21.15.12:0/4017709910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5dc0a11d0 0x7fe5dc1bc790 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe5c800daf0 tx=0x7fe5c8009120 comp rx=0 tx=0).stop 2024-09-11T01:52:54.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.769+0000 7fe5e128c640 1 -- 172.21.15.12:0/4017709910 reap_dead start 2024-09-11T01:52:54.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.769+0000 7fe5e128c640 1 -- 172.21.15.12:0/4017709910 reap_dead start 2024-09-11T01:52:54.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.769+0000 7fe5e2cef640 1 -- 172.21.15.12:0/4017709910 shutdown_connections 2024-09-11T01:52:54.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.769+0000 7fe5e2cef640 1 -- 172.21.15.12:0/4017709910 >> 172.21.15.12:0/4017709910 conn(0x7fe5dc0a2f70 msgr2=0x7fe5dc0a4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:54.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.769+0000 7fe5e2cef640 1 -- 172.21.15.12:0/4017709910 shutdown_connections 2024-09-11T01:52:54.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.769+0000 7fe5e2cef640 1 -- 172.21.15.12:0/4017709910 wait complete. 2024-09-11T01:52:54.795 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:52:54.795 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:52:54.795 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:52:54.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.896+0000 7f3269b73640 10 monclient: get_monmap_and_config 2024-09-11T01:52:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.896+0000 7f3269b73640 10 monclient: build_initial_monmap 2024-09-11T01:52:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.896+0000 7f3269b73640 10 monclient: monmap: 2024-09-11T01:52:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:54.896978+0000 2024-09-11T01:52:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:54.896978+0000 2024-09-11T01:52:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:54.898 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:54.899 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.896+0000 7f3269b73640 1 Processor -- start 2024-09-11T01:52:54.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3269b73640 1 -- start start 2024-09-11T01:52:54.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3269b73640 10 monclient: init 2024-09-11T01:52:54.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3269b73640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:54.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3269b73640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-11T01:52:54.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3269b73640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3264137e50 addr v1:172.21.15.71:6789/0 2024-09-11T01:52:54.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3269b73640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3264138230 0x7f3264138640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3269b73640 10 monclient(hunting): picked mon.noname-a con 0x7f3264138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:52:54.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3269b73640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f326413d380 0x7f326413f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3269b73640 10 monclient(hunting): picked mon.noname-c con 0x7f326413d380 addr v2:172.21.15.71:3300/0 2024-09-11T01:52:54.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3269b73640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32640a62d0 con 0x7f3264137e50 2024-09-11T01:52:54.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3269b73640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:54.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3269b73640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f326409bb40 con 0x7f3264138230 2024-09-11T01:52:54.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3269b73640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:54.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3269b73640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f326409c4a0 con 0x7f326413d380 2024-09-11T01:52:54.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3269b73640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:54.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3269b73640 10 monclient(hunting): authenticate will time out at 2181.339578s 2024-09-11T01:52:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3262ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3264138230 0x7f3264138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3262ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3264138230 0x7f3264138640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60110/0 (socket says 172.21.15.12:60110) 2024-09-11T01:52:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3262ffd640 1 -- 172.21.15.12:0/1289874883 learned_addr learned my addr 172.21.15.12:0/1289874883 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3262ffd640 10 monclient(hunting): get_auth_request con 0x7f3264138230 auth_method 0 2024-09-11T01:52:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3262ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3262ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3262ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3262ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3262ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3263fff640 1 --2- 172.21.15.12:0/1289874883 >> v2:172.21.15.71:3300/0 conn(0x7f326413d380 0x7f326413f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3262ffd640 10 monclient(hunting): handle_auth_done global_id 264153 payload 274 2024-09-11T01:52:54.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3262ffd640 1 -- 172.21.15.12:0/1289874883 >> v2:172.21.15.71:3300/0 conn(0x7f326413d380 msgr2=0x7f326413f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:54.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3262ffd640 1 --2- 172.21.15.12:0/1289874883 >> v2:172.21.15.71:3300/0 conn(0x7f326413d380 0x7f326413f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.897+0000 7f3262ffd640 1 -- 172.21.15.12:0/1289874883 >> v1:172.21.15.71:6789/0 conn(0x7f3264137e50 legacy=0x7f326409d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:54.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.898+0000 7f3262ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:52:54.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.898+0000 7f3262ffd640 1 monclient: found mon.noname-a 2024-09-11T01:52:54.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.898+0000 7f3262ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:54.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.898+0000 7f3262ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:52:54.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.898+0000 7f3262ffd640 1 -- 172.21.15.12:0/1289874883 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f326409c780 con 0x7f3264138230 2024-09-11T01:52:54.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.898+0000 7f3263fff640 10 monclient: get_auth_request con 0x7f326413d380 auth_method 0 2024-09-11T01:52:54.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.898+0000 7f3263fff640 1 --2- 172.21.15.12:0/1289874883 >> v2:172.21.15.71:3300/0 conn(0x7f326413d380 0x7f326413f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:52:54.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.900+0000 7f3262ffd640 1 -- 172.21.15.12:0/1289874883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3264138230 msgr2=0x7f3264138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:54.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.900+0000 7f3262ffd640 1 --2- 172.21.15.12:0/1289874883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3264138230 0x7f3264138640 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f326409cf30 tx=0x7f325802a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ed30eab8985b9ff server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:54.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f32627fc640 1 -- 172.21.15.12:0/1289874883 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258037070 con 0x7f3264138230 2024-09-11T01:52:54.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f32627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:54.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f32627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:52:54.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f32627fc640 10 monclient: dump: 2024-09-11T01:52:54.910 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:54.910 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:54.911 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.911 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.911 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:54.911 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.911 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:54.911 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:54.911 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:54.911 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f32627fc640 10 monclient: _finish_auth 0 2024-09-11T01:52:54.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f32627fc640 10 monclient: _check_auth_tickets 2024-09-11T01:52:54.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f32627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:54.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f3269b73640 5 monclient: authenticate success, global_id 264153 2024-09-11T01:52:54.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f32627fc640 1 -- 172.21.15.12:0/1289874883 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3258032640 con 0x7f3264138230 2024-09-11T01:52:54.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f3269b73640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:52:54.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f32627fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:54.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f3269b73640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:54.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f3269b73640 4 set_mon_vals no callback set 2024-09-11T01:52:54.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f32627fc640 1 -- 172.21.15.12:0/1289874883 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258032a10 con 0x7f3264138230 2024-09-11T01:52:54.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f32627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:54.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f32627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:52:54.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f32627fc640 10 monclient: dump: 2024-09-11T01:52:54.912 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:54.912 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:54.912 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.912 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.912 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:54.912 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.913 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:54.913 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:54.913 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:54.913 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f3269b73640 10 monclient: shutdown 2024-09-11T01:52:54.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f3269b73640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:54.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f3269b73640 1 -- 172.21.15.12:0/1289874883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3264138230 msgr2=0x7f3264138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:54.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f3269b73640 1 --2- 172.21.15.12:0/1289874883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3264138230 0x7f3264138640 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f326409cf30 tx=0x7f325802a080 comp rx=0 tx=0).stop 2024-09-11T01:52:54.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f3269b73640 1 -- 172.21.15.12:0/1289874883 shutdown_connections 2024-09-11T01:52:54.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f3269b73640 1 --2- 172.21.15.12:0/1289874883 >> v2:172.21.15.71:3300/0 conn(0x7f326413d380 0x7f326413f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f3269b73640 1 --2- 172.21.15.12:0/1289874883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3264138230 0x7f3264138640 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f3269b73640 1 -- 172.21.15.12:0/1289874883 >> 172.21.15.12:0/1289874883 conn(0x7f32640a7020 msgr2=0x7f32640a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:54.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f3269b73640 1 -- 172.21.15.12:0/1289874883 shutdown_connections 2024-09-11T01:52:54.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.901+0000 7f3269b73640 1 -- 172.21.15.12:0/1289874883 wait complete. 2024-09-11T01:52:54.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.902+0000 7f3269b73640 10 monclient: build_initial_monmap 2024-09-11T01:52:54.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.902+0000 7f3269b73640 10 monclient: monmap: 2024-09-11T01:52:54.914 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:54.914 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:54.914 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:54.914 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:54.914 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:54.914 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.914 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:54.914 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:54.914 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:54.914 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.902+0000 7f3269b73640 1 Processor -- start 2024-09-11T01:52:54.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.902+0000 7f3269b73640 1 -- start start 2024-09-11T01:52:54.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.902+0000 7f3269b73640 10 monclient: init 2024-09-11T01:52:54.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3269b73640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:54.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3269b73640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:52:54.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3269b73640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3264137e50 0x7f32641ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3269b73640 10 monclient(hunting): picked mon.noname-b con 0x7f3264137e50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:54.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3269b73640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3264138230 0x7f3264136dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3269b73640 10 monclient(hunting): picked mon.noname-a con 0x7f3264138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:54.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3269b73640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f326413d380 0x7f32640a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3269b73640 10 monclient(hunting): picked mon.noname-c con 0x7f326413d380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:54.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3269b73640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:54.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3269b73640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f326406f890 con 0x7f3264138230 2024-09-11T01:52:54.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3269b73640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:54.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3269b73640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f326406f710 con 0x7f3264137e50 2024-09-11T01:52:54.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3269b73640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:54.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3269b73640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f326413fcb0 con 0x7f326413d380 2024-09-11T01:52:54.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3269b73640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:54.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3269b73640 10 monclient(hunting): authenticate will time out at 2181.345179s 2024-09-11T01:52:54.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3262ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3264138230 0x7f3264136dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3262ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3264138230 0x7f3264136dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60122/0 (socket says 172.21.15.12:60122) 2024-09-11T01:52:54.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3262ffd640 1 -- 172.21.15.12:0/68174952 learned_addr learned my addr 172.21.15.12:0/68174952 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:54.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3262ffd640 10 monclient(hunting): get_auth_request con 0x7f3264138230 auth_method 0 2024-09-11T01:52:54.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3262ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:54.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3263fff640 1 --2- 172.21.15.12:0/68174952 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f326413d380 0x7f32640a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3262ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:54.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:54.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3263fff640 10 monclient(hunting): get_auth_request con 0x7f326413d380 auth_method 0 2024-09-11T01:52:54.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3263fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:54.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3263fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:54.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3263fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:54.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3262ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:54.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3262ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:54.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3262ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:54.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3263fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:54.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3263fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:54.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3263fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:54.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f32637fe640 1 --2- 172.21.15.12:0/68174952 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3264137e50 0x7f32641ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f32637fe640 10 monclient(hunting): get_auth_request con 0x7f3264137e50 auth_method 0 2024-09-11T01:52:54.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f32637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:54.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f32637fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:54.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f32637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:54.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3262ffd640 10 monclient(hunting): handle_auth_done global_id 264156 payload 824 2024-09-11T01:52:54.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3262ffd640 1 -- 172.21.15.12:0/68174952 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f326413d380 msgr2=0x7f32640a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:54.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3262ffd640 1 --2- 172.21.15.12:0/68174952 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f326413d380 0x7f32640a19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3262ffd640 1 -- 172.21.15.12:0/68174952 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3264137e50 msgr2=0x7f32641ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:54.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3262ffd640 1 --2- 172.21.15.12:0/68174952 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3264137e50 0x7f32641ccdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3262ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:52:54.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3262ffd640 1 monclient: found mon.noname-a 2024-09-11T01:52:54.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3262ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:54.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3262ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:52:54.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3262ffd640 1 -- 172.21.15.12:0/68174952 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32641cdd10 con 0x7f3264138230 2024-09-11T01:52:54.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.903+0000 7f3263fff640 1 --2- 172.21.15.12:0/68174952 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f326413d380 0x7f32640a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:52:54.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.905+0000 7f3262ffd640 1 -- 172.21.15.12:0/68174952 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3264138230 msgr2=0x7f3264136dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:54.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.905+0000 7f3262ffd640 1 --2- 172.21.15.12:0/68174952 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3264138230 0x7f3264136dd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3258000c00 tx=0x7f32580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:54.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.906+0000 7f324bfff640 1 -- 172.21.15.12:0/68174952 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258041030 con 0x7f3264138230 2024-09-11T01:52:54.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.906+0000 7f324bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:54.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.906+0000 7f324bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:52:54.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.906+0000 7f324bfff640 10 monclient: dump: 2024-09-11T01:52:54.920 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:54.920 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:54.920 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.920 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.920 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:54.920 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.920 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:54.920 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:54.921 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.906+0000 7f324bfff640 10 monclient: _finish_auth 0 2024-09-11T01:52:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.906+0000 7f324bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:52:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.906+0000 7f324bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.906+0000 7f3269b73640 5 monclient: authenticate success, global_id 264156 2024-09-11T01:52:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.906+0000 7f3269b73640 10 monclient: _renew_subs 2024-09-11T01:52:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.906+0000 7f324bfff640 1 -- 172.21.15.12:0/68174952 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f325803d6f0 con 0x7f3264138230 2024-09-11T01:52:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.906+0000 7f3269b73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:52:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.906+0000 7f3269b73640 1 -- 172.21.15.12:0/68174952 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3264121480 con 0x7f3264138230 2024-09-11T01:52:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.906+0000 7f324bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.906+0000 7f324bfff640 1 -- 172.21.15.12:0/68174952 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258037040 con 0x7f3264138230 2024-09-11T01:52:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.906+0000 7f324bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:54.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.906+0000 7f324bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:52:54.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.906+0000 7f324bfff640 10 monclient: dump: 2024-09-11T01:52:54.922 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:54.922 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:54.922 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.922 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.922 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:54.922 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.922 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:54.922 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:54.922 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:54.922 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.906+0000 7f3269b73640 10 monclient: _renew_subs 2024-09-11T01:52:54.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.906+0000 7f3269b73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:52:54.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.906+0000 7f3269b73640 1 -- 172.21.15.12:0/68174952 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32640a3470 con 0x7f3264138230 2024-09-11T01:52:54.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.906+0000 7f3261ffb640 4 set_mon_vals no callback set 2024-09-11T01:52:54.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.906+0000 7f3269b73640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:54.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.906+0000 7f3269b73640 1 --2- 172.21.15.12:0/68174952 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3264120f20 0x7f32640a1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.906+0000 7f3269b73640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:54.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.906+0000 7f3269b73640 1 -- 172.21.15.12:0/68174952 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f326406a060 con 0x7f3264120f20 2024-09-11T01:52:54.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.907+0000 7f324bfff640 1 -- 172.21.15.12:0/68174952 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f325803a030 con 0x7f3264138230 2024-09-11T01:52:54.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.907+0000 7f32637fe640 1 --2- 172.21.15.12:0/68174952 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3264120f20 0x7f32640a1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.907+0000 7f32637fe640 10 monclient: get_auth_request con 0x7f3264120f20 auth_method 0 2024-09-11T01:52:54.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.907+0000 7f32637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:54.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.907+0000 7f32637fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:54.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.907+0000 7f32637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:54.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.907+0000 7f324bfff640 1 --2- 172.21.15.12:0/68174952 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f325003e210 0x7f32500406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.907+0000 7f32637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:54.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.907+0000 7f32637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:54.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.907+0000 7f32637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:54.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.907+0000 7f324bfff640 1 -- 172.21.15.12:0/68174952 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(804..804 src has 676..804) ==== 4739+0+0 (secure 0 0 0) 0x7f3258072ef0 con 0x7f3264138230 2024-09-11T01:52:54.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.907+0000 7f32637fe640 10 monclient(hunting): handle_auth_done global_id 234367 payload 824 2024-09-11T01:52:54.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.907+0000 7f3263fff640 1 --2- 172.21.15.12:0/68174952 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f325003e210 0x7f32500406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.907+0000 7f32637fe640 1 -- 172.21.15.12:0/68174952 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3264138640 con 0x7f3264120f20 2024-09-11T01:52:54.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.907+0000 7f3263fff640 10 monclient: get_auth_request con 0x7f325003e210 auth_method 0 2024-09-11T01:52:54.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.909+0000 7f32637fe640 1 -- 172.21.15.12:0/68174952 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3264120f20 msgr2=0x7f32640a1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:54.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.909+0000 7f32637fe640 1 --2- 172.21.15.12:0/68174952 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3264120f20 0x7f32640a1ef0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f326413f890 tx=0x7f325400e960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:54.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.910+0000 7f3263fff640 1 --2- 172.21.15.12:0/68174952 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f325003e210 0x7f32500406d0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f325c00d8a0 tx=0x7f325c007780 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:54.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.910+0000 7f324bfff640 1 -- 172.21.15.12:0/68174952 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3254007450 con 0x7f3264120f20 2024-09-11T01:52:54.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.910+0000 7f324bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:54.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.910+0000 7f324bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:52:54.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.910+0000 7f324bfff640 10 monclient: dump: 2024-09-11T01:52:54.927 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:54.927 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:54.927 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.927 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.927 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:54.927 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.927 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:54.927 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:54.927 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:54.927 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.910+0000 7f324bfff640 1 -- 172.21.15.12:0/68174952 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3254007450 con 0x7f3264120f20 2024-09-11T01:52:54.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.910+0000 7f324bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:54.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.910+0000 7f324bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:54.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.910+0000 7f324bfff640 1 -- 172.21.15.12:0/68174952 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3264120f20 msgr2=0x7f32640a1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:54.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.910+0000 7f324bfff640 1 --2- 172.21.15.12:0/68174952 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3264120f20 0x7f32640a1ef0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f326413f890 tx=0x7f325400e960 comp rx=0 tx=0).stop 2024-09-11T01:52:54.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.910+0000 7f324bfff640 1 -- 172.21.15.12:0/68174952 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3264120f20 msgr2=0x7f32640a1ef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:54.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.910+0000 7f324bfff640 1 --2- 172.21.15.12:0/68174952 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3264120f20 0x7f32640a1ef0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.916+0000 7f3269b73640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:52:54.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.916+0000 7f3269b73640 1 --2- 172.21.15.12:0/68174952 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3264137310 0x7f32640a2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:54.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.916+0000 7f3269b73640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:54.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.916+0000 7f3269b73640 1 -- 172.21.15.12:0/68174952 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3264069950 con 0x7f3264137310 2024-09-11T01:52:54.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.916+0000 7f32637fe640 1 --2- 172.21.15.12:0/68174952 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3264137310 0x7f32640a2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:54.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.916+0000 7f32637fe640 10 monclient: get_auth_request con 0x7f3264137310 auth_method 0 2024-09-11T01:52:54.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.916+0000 7f32637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:54.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.916+0000 7f32637fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:54.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.916+0000 7f32637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:54.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.916+0000 7f32637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:54.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.916+0000 7f32637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:54.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.916+0000 7f32637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:54.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.917+0000 7f32637fe640 10 monclient(hunting): handle_auth_done global_id 234370 payload 824 2024-09-11T01:52:54.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.917+0000 7f32637fe640 1 -- 172.21.15.12:0/68174952 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f32640a1600 con 0x7f3264137310 2024-09-11T01:52:54.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.919+0000 7f32637fe640 1 -- 172.21.15.12:0/68174952 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3264137310 msgr2=0x7f32640a2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:54.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.919+0000 7f32637fe640 1 --2- 172.21.15.12:0/68174952 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3264137310 0x7f32640a2430 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f326413f890 tx=0x7f3254003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:54.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.919+0000 7f324bfff640 1 -- 172.21.15.12:0/68174952 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325401ed80 con 0x7f3264137310 2024-09-11T01:52:54.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.919+0000 7f324bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:54.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.919+0000 7f324bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:52:54.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.919+0000 7f324bfff640 10 monclient: dump: 2024-09-11T01:52:54.929 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:54.929 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:54.929 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.929 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:54.930 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:54.930 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:54.930 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:54.930 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:54.930 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:54.930 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:54.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.919+0000 7f324bfff640 1 -- 172.21.15.12:0/68174952 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7f325401ed80 con 0x7f3264137310 2024-09-11T01:52:54.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.919+0000 7f324bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:52:54.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.919+0000 7f324bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:54.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.919+0000 7f324bfff640 1 -- 172.21.15.12:0/68174952 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3264137310 msgr2=0x7f32640a2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:54.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.919+0000 7f324bfff640 1 --2- 172.21.15.12:0/68174952 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3264137310 0x7f32640a2430 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f326413f890 tx=0x7f3254003760 comp rx=0 tx=0).stop 2024-09-11T01:52:54.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.919+0000 7f324bfff640 1 -- 172.21.15.12:0/68174952 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3264137310 msgr2=0x7f32640a2430 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:52:54.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.919+0000 7f324bfff640 1 --2- 172.21.15.12:0/68174952 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3264137310 0x7f32640a2430 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f326413f890 tx=0x7f3254003760 comp rx=0 tx=0).stop 2024-09-11T01:52:54.931 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:52:54.931 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:52:54.931 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:52:54.931 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:52:54.931 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 220, 2024-09-11T01:52:54.931 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 176812, 2024-09-11T01:52:54.931 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:52:54.931 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:52:54.931 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:52:54.932 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:52:54.932 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:54.932 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:52:54.932 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:52:54.932 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:52:54.932 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:52:54.932 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:54.932 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:54.932 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:54.932 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:54.932 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:54.932 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:54.932 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:54.932 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:54.933 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:54.933 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:54.933 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:54.933 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:54.933 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:52:54.933 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:52:54.933 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:54.933 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:54.933 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:54.933 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:54.933 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:54.933 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:54.933 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:54.934 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:54.934 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:54.934 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:54.934 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:54.934 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:54.934 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.934 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:52:54.934 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:52:54.934 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:52:54.934 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:52:54.934 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:52:54.934 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:52:54.934 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:52:54.934 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:52:54.935 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:52:54.935 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:52:54.935 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:52:54.935 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:52:54.935 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:52:54.935 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:52:54.935 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:52:54.935 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:52:54.935 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:52:54.935 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:52:54.935 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:52:54.935 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:52:54.935 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:52:54.935 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:52:54.936 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:52:54.936 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:52:54.936 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:52:54.936 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:52:54.936 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:52:54.936 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:52:54.936 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:54.936 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:52:54.936 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.936 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:52:54.937 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.937 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:52:54.937 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:52:54.937 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:54.937 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:54.937 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.937 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:54.937 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:52:54.937 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:54.937 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.937 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.937 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:54.937 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:52:54.938 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:54.938 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:54.938 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:54.938 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.938 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:52:54.938 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:52:54.938 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:54.938 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:54.938 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:54.938 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.938 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.938 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:52:54.938 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:52:54.938 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:54.939 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:54.939 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.939 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:54.939 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:52:54.939 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:54.939 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.939 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.939 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:54.939 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:52:54.939 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:54.939 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:54.939 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:54.939 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.939 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:52:54.940 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:52:54.940 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:54.940 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:54.940 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:54.940 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.940 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.940 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:52:54.940 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:52:54.940 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:52:54.940 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:52:54.940 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.940 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:52:54.940 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:52:54.941 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:54.941 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.941 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.941 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:52:54.941 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:52:54.941 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:52:54.941 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:54.941 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:54.941 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.941 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:52:54.941 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:52:54.941 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:52:54.941 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:52:54.941 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:52:54.942 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:54.942 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:54.942 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.942 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:52:54.942 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:52:54.942 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.942 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:54.942 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:54.942 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:52:54.942 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:54.942 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:54.942 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:52:54.942 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.942 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:54.943 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:54.943 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:52:54.943 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:54.943 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:52:54.943 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:52:54.943 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:52:54.943 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:52:54.943 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:52:54.943 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:52:54.943 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:52:54.943 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:52:54.943 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:52:54.943 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:52:54.943 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:52:54.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.920+0000 7f3269b73640 1 -- 172.21.15.12:0/68174952 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f325003e210 msgr2=0x7f32500406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:54.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.920+0000 7f3269b73640 1 --2- 172.21.15.12:0/68174952 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f325003e210 0x7f32500406d0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f325c00d8a0 tx=0x7f325c007780 comp rx=0 tx=0).stop 2024-09-11T01:52:54.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.920+0000 7f3269b73640 10 monclient: shutdown 2024-09-11T01:52:54.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.920+0000 7f3269b73640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:54.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.920+0000 7f3269b73640 1 -- 172.21.15.12:0/68174952 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3264138230 msgr2=0x7f3264136dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:54.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.920+0000 7f3269b73640 1 --2- 172.21.15.12:0/68174952 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3264138230 0x7f3264136dd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3258000c00 tx=0x7f32580046c0 comp rx=0 tx=0).stop 2024-09-11T01:52:54.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.920+0000 7f3263fff640 1 -- 172.21.15.12:0/68174952 reap_dead start 2024-09-11T01:52:54.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.920+0000 7f3263fff640 1 -- 172.21.15.12:0/68174952 reap_dead start 2024-09-11T01:52:54.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.920+0000 7f3269b73640 1 -- 172.21.15.12:0/68174952 shutdown_connections 2024-09-11T01:52:54.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.920+0000 7f3269b73640 1 -- 172.21.15.12:0/68174952 >> 172.21.15.12:0/68174952 conn(0x7f32640a7020 msgr2=0x7f326413c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:54.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.920+0000 7f3269b73640 1 -- 172.21.15.12:0/68174952 shutdown_connections 2024-09-11T01:52:54.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:54.920+0000 7f3269b73640 1 -- 172.21.15.12:0/68174952 wait complete. 2024-09-11T01:52:54.946 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:52:54.946 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-11T01:52:54.946 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T01:52:54.946 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-11T01:52:54.946 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-11T01:52:54.947 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-11T01:52:54.947 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:52:54.963 INFO:tasks.ceph.mon.b:Stopped 2024-09-11T01:52:54.963 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:52:54.963 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:52:55.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.062+0000 7fa1af9f3640 10 monclient: get_monmap_and_config 2024-09-11T01:52:55.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.062+0000 7fa1af9f3640 10 monclient: build_initial_monmap 2024-09-11T01:52:55.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.062+0000 7fa1af9f3640 10 monclient: monmap: 2024-09-11T01:52:55.064 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:55.064 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:55.064 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:55.062855+0000 2024-09-11T01:52:55.064 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:55.062855+0000 2024-09-11T01:52:55.064 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:55.064 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:55.064 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:55.064 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:55.064 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:55.064 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:55.064 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:55.064 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:55.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.062+0000 7fa1af9f3640 1 Processor -- start 2024-09-11T01:52:55.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.062+0000 7fa1af9f3640 1 -- start start 2024-09-11T01:52:55.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1af9f3640 10 monclient: init 2024-09-11T01:52:55.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1af9f3640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:55.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1af9f3640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-11T01:52:55.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1af9f3640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa1a8138230 0x7fa1a8138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:55.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1af9f3640 10 monclient(hunting): picked mon.noname-c con 0x7fa1a8138230 addr v2:172.21.15.71:3300/0 2024-09-11T01:52:55.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1af9f3640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa1a809d030 0x7fa1a80a5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:55.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1af9f3640 10 monclient(hunting): picked mon.noname-a con 0x7fa1a809d030 addr v2:172.21.15.12:3300/0 2024-09-11T01:52:55.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1af9f3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa1a809d460 addr v1:172.21.15.71:6789/0 2024-09-11T01:52:55.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1af9f3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1a80a6e30 con 0x7fa1a809d460 2024-09-11T01:52:55.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1af9f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:55.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1af9f3640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fa1a809c9c0 con 0x7fa1a809d030 2024-09-11T01:52:55.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1af9f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:55.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1af9f3640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fa1a809ba20 con 0x7fa1a8138230 2024-09-11T01:52:55.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1af9f3640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:55.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1af9f3640 10 monclient(hunting): authenticate will time out at 2181.505594s 2024-09-11T01:52:55.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1ad78f640 1 -- >> v2:172.21.15.71:3300/0 conn(0x7fa1a8138230 msgr2=0x7fa1a8138610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:52:55.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1adf90640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7fa1a809d460 legacy=0x7fa1a80a5560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:52:55.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1ad78f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa1a8138230 0x7fa1a8138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:55.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1acf8e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa1a809d030 0x7fa1a80a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1acf8e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa1a809d030 0x7fa1a80a5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60136/0 (socket says 172.21.15.12:60136) 2024-09-11T01:52:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1acf8e640 1 -- 172.21.15.12:0/2801426354 learned_addr learned my addr 172.21.15.12:0/2801426354 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1acf8e640 10 monclient(hunting): get_auth_request con 0x7fa1a809d030 auth_method 0 2024-09-11T01:52:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1acf8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1acf8e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1acf8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1acf8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1acf8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1acf8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1acf8e640 10 monclient(hunting): handle_auth_done global_id 264159 payload 274 2024-09-11T01:52:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1acf8e640 1 -- 172.21.15.12:0/2801426354 >> v2:172.21.15.71:3300/0 conn(0x7fa1a8138230 msgr2=0x7fa1a8138610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:52:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1acf8e640 1 --2- 172.21.15.12:0/2801426354 >> v2:172.21.15.71:3300/0 conn(0x7fa1a8138230 0x7fa1a8138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1acf8e640 1 -- 172.21.15.12:0/2801426354 >> v1:172.21.15.71:6789/0 conn(0x7fa1a809d460 legacy=0x7fa1a80a5560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:52:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1acf8e640 10 monclient: _finish_hunting 0 2024-09-11T01:52:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1acf8e640 1 monclient: found mon.noname-a 2024-09-11T01:52:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1acf8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1acf8e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:52:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.063+0000 7fa1acf8e640 1 -- 172.21.15.12:0/2801426354 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1a80a66a0 con 0x7fa1a809d030 2024-09-11T01:52:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.066+0000 7fa1acf8e640 1 -- 172.21.15.12:0/2801426354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1a809d030 msgr2=0x7fa1a80a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.066+0000 7fa1acf8e640 1 --2- 172.21.15.12:0/2801426354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1a809d030 0x7fa1a80a5020 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa1a80a17e0 tx=0x7fa19802a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2f32e5c0b0770a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.066+0000 7fa19ffff640 1 -- 172.21.15.12:0/2801426354 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa198037070 con 0x7fa1a809d030 2024-09-11T01:52:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.066+0000 7fa19ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.066+0000 7fa19ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:52:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.066+0000 7fa19ffff640 10 monclient: dump: 2024-09-11T01:52:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:55.076 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:55.077 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:55.077 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:55.077 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:55.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.066+0000 7fa19ffff640 10 monclient: _finish_auth 0 2024-09-11T01:52:55.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.066+0000 7fa19ffff640 10 monclient: _check_auth_tickets 2024-09-11T01:52:55.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.066+0000 7fa19ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:55.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.066+0000 7fa1af9f3640 5 monclient: authenticate success, global_id 264159 2024-09-11T01:52:55.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.066+0000 7fa19ffff640 1 -- 172.21.15.12:0/2801426354 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa198032640 con 0x7fa1a809d030 2024-09-11T01:52:55.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.066+0000 7fa1af9f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:52:55.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.066+0000 7fa19ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:55.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.066+0000 7fa1af9f3640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:55.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.066+0000 7fa1af9f3640 4 set_mon_vals no callback set 2024-09-11T01:52:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.066+0000 7fa19ffff640 1 -- 172.21.15.12:0/2801426354 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa198032a10 con 0x7fa1a809d030 2024-09-11T01:52:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.066+0000 7fa19ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.066+0000 7fa19ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:52:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.066+0000 7fa19ffff640 10 monclient: dump: 2024-09-11T01:52:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:55.078 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:55.079 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.067+0000 7fa1af9f3640 10 monclient: shutdown 2024-09-11T01:52:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.067+0000 7fa1af9f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.067+0000 7fa1af9f3640 1 -- 172.21.15.12:0/2801426354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1a809d030 msgr2=0x7fa1a80a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.067+0000 7fa1af9f3640 1 --2- 172.21.15.12:0/2801426354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1a809d030 0x7fa1a80a5020 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa1a80a17e0 tx=0x7fa19802a0d0 comp rx=0 tx=0).stop 2024-09-11T01:52:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.067+0000 7fa1af9f3640 1 -- 172.21.15.12:0/2801426354 shutdown_connections 2024-09-11T01:52:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.067+0000 7fa1af9f3640 1 --2- 172.21.15.12:0/2801426354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1a809d030 0x7fa1a80a5020 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.067+0000 7fa1af9f3640 1 --2- 172.21.15.12:0/2801426354 >> v2:172.21.15.71:3300/0 conn(0x7fa1a8138230 0x7fa1a8138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.067+0000 7fa1af9f3640 1 -- 172.21.15.12:0/2801426354 >> 172.21.15.12:0/2801426354 conn(0x7fa1a80a7120 msgr2=0x7fa1a80a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.067+0000 7fa1af9f3640 1 -- 172.21.15.12:0/2801426354 shutdown_connections 2024-09-11T01:52:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.067+0000 7fa1af9f3640 1 -- 172.21.15.12:0/2801426354 wait complete. 2024-09-11T01:52:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.068+0000 7fa1af9f3640 10 monclient: build_initial_monmap 2024-09-11T01:52:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.068+0000 7fa1af9f3640 10 monclient: monmap: 2024-09-11T01:52:55.079 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:55.080 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.068+0000 7fa1af9f3640 1 Processor -- start 2024-09-11T01:52:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.068+0000 7fa1af9f3640 1 -- start start 2024-09-11T01:52:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.068+0000 7fa1af9f3640 10 monclient: init 2024-09-11T01:52:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.068+0000 7fa1af9f3640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:55.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.068+0000 7fa1af9f3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:52:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.068+0000 7fa1af9f3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1a809d030 0x7fa1a81ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.068+0000 7fa1af9f3640 10 monclient(hunting): picked mon.noname-b con 0x7fa1a809d030 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.068+0000 7fa1af9f3640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa1a809d460 0x7fa1a81371c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.068+0000 7fa1af9f3640 10 monclient(hunting): picked mon.noname-c con 0x7fa1a809d460 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.068+0000 7fa1af9f3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1a8138230 0x7fa1a8124f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.068+0000 7fa1af9f3640 10 monclient(hunting): picked mon.noname-a con 0x7fa1a8138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.068+0000 7fa1af9f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.068+0000 7fa1af9f3640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a806f970 con 0x7fa1a8138230 2024-09-11T01:52:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.068+0000 7fa1af9f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.068+0000 7fa1af9f3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a806f7f0 con 0x7fa1a809d030 2024-09-11T01:52:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.068+0000 7fa1af9f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.068+0000 7fa1af9f3640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa1a80a5c70 con 0x7fa1a809d460 2024-09-11T01:52:55.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.068+0000 7fa1af9f3640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.068+0000 7fa1af9f3640 10 monclient(hunting): authenticate will time out at 2181.510985s 2024-09-11T01:52:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1ad78f640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1a809d030 msgr2=0x7fa1a81ccfe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:52:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1ad78f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1a809d030 0x7fa1a81ccfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1adf90640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1a8138230 0x7fa1a8124f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1acf8e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa1a809d460 0x7fa1a81371c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1adf90640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1a8138230 0x7fa1a8124f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60146/0 (socket says 172.21.15.12:60146) 2024-09-11T01:52:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1acf8e640 10 monclient(hunting): get_auth_request con 0x7fa1a809d460 auth_method 0 2024-09-11T01:52:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1adf90640 1 -- 172.21.15.12:0/3238767560 learned_addr learned my addr 172.21.15.12:0/3238767560 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1acf8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1acf8e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1acf8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1adf90640 10 monclient(hunting): get_auth_request con 0x7fa1a8138230 auth_method 0 2024-09-11T01:52:55.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1adf90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1adf90640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1adf90640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1acf8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1acf8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1acf8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1adf90640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1adf90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1adf90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1adf90640 10 monclient(hunting): handle_auth_done global_id 264162 payload 824 2024-09-11T01:52:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1adf90640 1 -- 172.21.15.12:0/3238767560 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa1a809d460 msgr2=0x7fa1a81371c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1adf90640 1 --2- 172.21.15.12:0/3238767560 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa1a809d460 0x7fa1a81371c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1adf90640 1 -- 172.21.15.12:0/3238767560 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1a809d030 msgr2=0x7fa1a81ccfe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:52:55.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1adf90640 1 --2- 172.21.15.12:0/3238767560 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1a809d030 0x7fa1a81ccfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:55.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1adf90640 10 monclient: _finish_hunting 0 2024-09-11T01:52:55.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1adf90640 1 monclient: found mon.noname-a 2024-09-11T01:52:55.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1adf90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:55.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1adf90640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:52:55.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1adf90640 1 -- 172.21.15.12:0/3238767560 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1a80a1d50 con 0x7fa1a8138230 2024-09-11T01:52:55.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.069+0000 7fa1acf8e640 1 --2- 172.21.15.12:0/3238767560 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa1a809d460 0x7fa1a81371c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:52:55.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.071+0000 7fa1adf90640 1 -- 172.21.15.12:0/3238767560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1a8138230 msgr2=0x7fa1a8124f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:55.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.071+0000 7fa1adf90640 1 --2- 172.21.15.12:0/3238767560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1a8138230 0x7fa1a8124f70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa1a000bd60 tx=0x7fa1a000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:55.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.071+0000 7fa1adf90640 1 --2- 172.21.15.12:0/3238767560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1a8138230 0x7fa1a8124f70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa1a000bd60 tx=0x7fa1a000be50 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7fa1a0015670 mon_map magic: 0 2024-09-11T01:52:55.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.071+0000 7fa19dffb640 1 -- 172.21.15.12:0/3238767560 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a0015670 con 0x7fa1a8138230 2024-09-11T01:52:55.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.071+0000 7fa19dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:55.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.071+0000 7fa19dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:52:55.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.071+0000 7fa19dffb640 10 monclient: dump: 2024-09-11T01:52:55.085 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:55.085 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:55.085 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:55.085 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:55.085 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:55.085 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:55.085 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:55.085 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:55.085 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:55.085 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:55.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.071+0000 7fa19dffb640 10 monclient: _finish_auth 0 2024-09-11T01:52:55.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.071+0000 7fa19dffb640 10 monclient: _check_auth_tickets 2024-09-11T01:52:55.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.071+0000 7fa19dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:55.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.071+0000 7fa19dffb640 1 -- 172.21.15.12:0/3238767560 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1a0010070 con 0x7fa1a8138230 2024-09-11T01:52:55.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.071+0000 7fa1af9f3640 5 monclient: authenticate success, global_id 264162 2024-09-11T01:52:55.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.071+0000 7fa1af9f3640 10 monclient: _renew_subs 2024-09-11T01:52:55.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.071+0000 7fa1af9f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:52:55.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.071+0000 7fa1af9f3640 1 -- 172.21.15.12:0/3238767560 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1a8126880 con 0x7fa1a8138230 2024-09-11T01:52:55.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.071+0000 7fa19dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:55.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.072+0000 7fa1af9f3640 10 monclient: _renew_subs 2024-09-11T01:52:55.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.071+0000 7fa19f7fe640 4 set_mon_vals no callback set 2024-09-11T01:52:55.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.072+0000 7fa1af9f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:52:55.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.072+0000 7fa1af9f3640 1 -- 172.21.15.12:0/3238767560 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1a8128340 con 0x7fa1a8138230 2024-09-11T01:52:55.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.073+0000 7fa19dffb640 1 -- 172.21.15.12:0/3238767560 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a0015a50 con 0x7fa1a8138230 2024-09-11T01:52:55.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.073+0000 7fa19dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:55.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.073+0000 7fa19dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:52:55.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.073+0000 7fa19dffb640 10 monclient: dump: 2024-09-11T01:52:55.087 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:55.087 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:55.087 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:55.087 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:55.087 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:55.087 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:55.087 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:55.087 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:55.087 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:55.087 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:55.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.073+0000 7fa19dffb640 1 -- 172.21.15.12:0/3238767560 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fa1a001d030 con 0x7fa1a8138230 2024-09-11T01:52:55.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.073+0000 7fa1af9f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:55.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.073+0000 7fa1af9f3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:55.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.073+0000 7fa1af9f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:52:55.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.073+0000 7fa1af9f3640 1 -- 172.21.15.12:0/3238767560 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1a80a18d0 con 0x7fa1a8138230 2024-09-11T01:52:55.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.073+0000 7fa19dffb640 1 --2- 172.21.15.12:0/3238767560 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa18c03e1c0 0x7fa18c040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:55.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.073+0000 7fa1ad78f640 1 --2- 172.21.15.12:0/3238767560 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa18c03e1c0 0x7fa18c040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:55.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.073+0000 7fa1ad78f640 10 monclient: get_auth_request con 0x7fa18c03e1c0 auth_method 0 2024-09-11T01:52:55.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.076+0000 7fa1ad78f640 1 --2- 172.21.15.12:0/3238767560 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa18c03e1c0 0x7fa18c040680 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4006fd0 tx=0x7fa1a4006d80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:55.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.078+0000 7fa19dffb640 1 -- 172.21.15.12:0/3238767560 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(804..804 src has 676..804) ==== 4739+0+0 (secure 0 0 0) 0x7fa1a0052f40 con 0x7fa1a8138230 2024-09-11T01:52:55.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.694+0000 7fa19dffb640 1 -- 172.21.15.12:0/3238767560 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa1a0051960 con 0x7fa1a8138230 2024-09-11T01:52:55.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.694+0000 7fa19dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:55.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.694+0000 7fa19dffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:55.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.960+0000 7fa1af9f3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:52:55.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.960+0000 7fa1af9f3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:52:55.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.960+0000 7fa1af9f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:52:55.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.960+0000 7fa1af9f3640 1 -- 172.21.15.12:0/3238767560 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa1a8138610 con 0x7fa1a8138230 2024-09-11T01:52:55.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.960+0000 7fa19dffb640 1 -- 172.21.15.12:0/3238767560 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fa1a001eb50 con 0x7fa1a8138230 2024-09-11T01:52:55.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.960+0000 7fa19dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:52:55.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.960+0000 7fa19dffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:55.963 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":220,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:52:55.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.963+0000 7fa1af9f3640 1 -- 172.21.15.12:0/3238767560 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa18c03e1c0 msgr2=0x7fa18c040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:55.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.963+0000 7fa1af9f3640 1 --2- 172.21.15.12:0/3238767560 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa18c03e1c0 0x7fa18c040680 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4006fd0 tx=0x7fa1a4006d80 comp rx=0 tx=0).stop 2024-09-11T01:52:55.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.963+0000 7fa1af9f3640 10 monclient: shutdown 2024-09-11T01:52:55.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.963+0000 7fa1af9f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:55.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.963+0000 7fa1af9f3640 1 -- 172.21.15.12:0/3238767560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1a8138230 msgr2=0x7fa1a8124f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:55.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.963+0000 7fa1af9f3640 1 --2- 172.21.15.12:0/3238767560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1a8138230 0x7fa1a8124f70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa1a000bd60 tx=0x7fa1a000be50 comp rx=0 tx=0).stop 2024-09-11T01:52:55.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.963+0000 7fa1af9f3640 1 -- 172.21.15.12:0/3238767560 shutdown_connections 2024-09-11T01:52:55.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.963+0000 7fa1af9f3640 1 --2- 172.21.15.12:0/3238767560 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa18c03e1c0 0x7fa18c040680 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:55.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.963+0000 7fa1af9f3640 1 --2- 172.21.15.12:0/3238767560 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1a8138230 0x7fa1a8124f70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:55.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.963+0000 7fa1af9f3640 1 --2- 172.21.15.12:0/3238767560 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa1a809d460 0x7fa1a81371c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:55.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.963+0000 7fa1af9f3640 1 --2- 172.21.15.12:0/3238767560 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1a809d030 0x7fa1a81ccfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:55.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.963+0000 7fa1af9f3640 1 -- 172.21.15.12:0/3238767560 >> 172.21.15.12:0/3238767560 conn(0x7fa1a80a7120 msgr2=0x7fa1a80a3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:55.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.963+0000 7fa1af9f3640 1 -- 172.21.15.12:0/3238767560 shutdown_connections 2024-09-11T01:52:55.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:55.963+0000 7fa1af9f3640 1 -- 172.21.15.12:0/3238767560 wait complete. 2024-09-11T01:52:58.976 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:52:59.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.087+0000 7f570173d640 10 monclient: get_monmap_and_config 2024-09-11T01:52:59.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.087+0000 7f570173d640 10 monclient: build_initial_monmap 2024-09-11T01:52:59.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.087+0000 7f570173d640 10 monclient: monmap: 2024-09-11T01:52:59.089 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:59.089 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:59.089 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:52:59.088405+0000 2024-09-11T01:52:59.089 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:52:59.088405+0000 2024-09-11T01:52:59.090 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:59.090 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:59.090 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:52:59.090 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:52:59.090 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:52:59.090 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:52:59.090 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:52:59.090 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:59.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.088+0000 7f570173d640 1 Processor -- start 2024-09-11T01:52:59.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.088+0000 7f570173d640 1 -- start start 2024-09-11T01:52:59.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.088+0000 7f570173d640 10 monclient: init 2024-09-11T01:52:59.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.088+0000 7f570173d640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:59.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.088+0000 7f570173d640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-11T01:52:59.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.088+0000 7f570173d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f56fc0a4170 0x7f56fc073fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:59.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.088+0000 7f570173d640 10 monclient(hunting): picked mon.noname-c con 0x7f56fc0a4170 addr v2:172.21.15.71:3300/0 2024-09-11T01:52:59.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.088+0000 7f570173d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f56fc074500 addr v1:172.21.15.12:6789/0 2024-09-11T01:52:59.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.088+0000 7f570173d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f56fc0a2c70 addr v1:172.21.15.71:6789/0 2024-09-11T01:52:59.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.088+0000 7f570173d640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56fc1380d0 con 0x7f56fc074500 2024-09-11T01:52:59.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.088+0000 7f570173d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56fc09fff0 con 0x7f56fc0a2c70 2024-09-11T01:52:59.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.088+0000 7f570173d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:59.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.088+0000 7f570173d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f56fc0a4550 con 0x7f56fc0a4170 2024-09-11T01:52:59.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.088+0000 7f570173d640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:59.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.088+0000 7f570173d640 10 monclient(hunting): authenticate will time out at 2185.530948s 2024-09-11T01:52:59.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.089+0000 7f56fb7fe640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f56fc0a2c70 legacy=0x7f56fc13ffb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:52:59.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.089+0000 7f56faffd640 1 -- >> v2:172.21.15.71:3300/0 conn(0x7f56fc0a4170 msgr2=0x7f56fc073fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:52:59.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.089+0000 7f56f27fc640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f56fc074500 0x7f56fc0a2560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52688/0 (socket says 172.21.15.12:52688) 2024-09-11T01:52:59.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.089+0000 7f56f27fc640 1 -- 172.21.15.12:0/1887113212 learned_addr learned my addr 172.21.15.12:0/1887113212 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:52:59.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.089+0000 7f56faffd640 1 --2- 172.21.15.12:0/1887113212 >> v2:172.21.15.71:3300/0 conn(0x7f56fc0a4170 0x7f56fc073fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:59.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56f27fc640 1 -- 172.21.15.12:0/1887113212 >> v1:172.21.15.12:6789/0 conn(0x7f56fc074500 legacy=0x7f56fc0a2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:59.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 1 -- 172.21.15.12:0/1887113212 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1347939586 0 0) 0x7f56e4001450 con 0x7f56fc074500 2024-09-11T01:52:59.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 10 monclient(hunting): _negotiate 2024-09-11T01:52:59.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:59.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:59.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 10 monclient(hunting): my global_id is 264165 2024-09-11T01:52:59.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 1 -- 172.21.15.12:0/1887113212 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56e8003520 con 0x7f56fc074500 2024-09-11T01:52:59.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 1 -- 172.21.15.12:0/1887113212 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4275744935 0 0) 0x7f56e8003520 con 0x7f56fc074500 2024-09-11T01:52:59.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 1 -- 172.21.15.12:0/1887113212 >> v2:172.21.15.71:3300/0 conn(0x7f56fc0a4170 msgr2=0x7f56fc073fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:52:59.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 1 --2- 172.21.15.12:0/1887113212 >> v2:172.21.15.71:3300/0 conn(0x7f56fc0a4170 0x7f56fc073fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:59.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 1 -- 172.21.15.12:0/1887113212 >> v1:172.21.15.71:6789/0 conn(0x7f56fc0a2c70 legacy=0x7f56fc13ffb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:52:59.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 10 monclient: _finish_hunting 0 2024-09-11T01:52:59.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:52:59.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:59.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:52:59.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 1 -- 172.21.15.12:0/1887113212 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56fc138350 con 0x7f56fc074500 2024-09-11T01:52:59.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 10 monclient: _finish_auth 0 2024-09-11T01:52:59.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:52:59.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:59.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 1 -- 172.21.15.12:0/1887113212 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f56e4001b70 con 0x7f56fc074500 2024-09-11T01:52:59.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f570173d640 5 monclient: authenticate success, global_id 264165 2024-09-11T01:52:59.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f570173d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:52:59.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:59.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:52:59.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 10 monclient: dump: 2024-09-11T01:52:59.100 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:59.100 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:59.100 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:59.100 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:59.100 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:59.100 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:59.100 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:59.101 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:59.101 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:59.101 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:59.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:52:59.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:59.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 1 -- 172.21.15.12:0/1887113212 >> v1:172.21.15.12:6789/0 conn(0x7f56fc074500 legacy=0x7f56fc0a2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:59.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:52:59.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 1 --2- 172.21.15.12:0/1887113212 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56e8003a00 0x7f56e8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:59.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 10 monclient(hunting): picked mon.a con 0x7f56e8003a00 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:59.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 1 --2- 172.21.15.12:0/1887113212 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56e8004800 0x7f56e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:59.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 10 monclient(hunting): picked mon.c con 0x7f56e8004800 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:59.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 1 --2- 172.21.15.12:0/1887113212 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56e8005370 0x7f56e800b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:59.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 10 monclient(hunting): picked mon.b con 0x7f56e8005370 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:59.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:59.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 1 -- 172.21.15.12:0/1887113212 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc0a4550 con 0x7f56e8003a00 2024-09-11T01:52:59.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:59.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 1 -- 172.21.15.12:0/1887113212 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f56e800c2c0 con 0x7f56e8005370 2024-09-11T01:52:59.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:59.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 1 -- 172.21.15.12:0/1887113212 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f56e800c4c0 con 0x7f56e8004800 2024-09-11T01:52:59.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:59.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56fa7fc640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:52:59.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f570173d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:52:59.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56faffd640 1 --2- 172.21.15.12:0/1887113212 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56e8003a00 0x7f56e8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:59.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56f27fc640 1 --2- 172.21.15.12:0/1887113212 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56e8004800 0x7f56e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:59.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56faffd640 10 monclient(hunting): get_auth_request con 0x7f56e8003a00 auth_method 0 2024-09-11T01:52:59.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:59.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56faffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:59.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56faffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:52:59.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56f27fc640 10 monclient(hunting): get_auth_request con 0x7f56e8004800 auth_method 0 2024-09-11T01:52:59.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56f27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:59.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.091+0000 7f56f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:59.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.092+0000 7f56f27fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:52:59.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.092+0000 7f56fb7fe640 1 -- 172.21.15.12:0/1887113212 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56e8005370 msgr2=0x7f56e800b9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:52:59.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.092+0000 7f56fb7fe640 1 --2- 172.21.15.12:0/1887113212 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56e8005370 0x7f56e800b9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:59.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.092+0000 7f56faffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:59.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.092+0000 7f56faffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:59.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.092+0000 7f56faffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:52:59.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.092+0000 7f56f27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:59.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.092+0000 7f56f27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:59.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.092+0000 7f56f27fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:52:59.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.092+0000 7f56faffd640 10 monclient(hunting): handle_auth_done global_id 264165 payload 293 2024-09-11T01:52:59.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.092+0000 7f56faffd640 1 -- 172.21.15.12:0/1887113212 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56e8004800 msgr2=0x7f56e8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:52:59.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.092+0000 7f56faffd640 1 --2- 172.21.15.12:0/1887113212 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56e8004800 0x7f56e8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:59.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.092+0000 7f56faffd640 1 -- 172.21.15.12:0/1887113212 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56e8005370 msgr2=0x7f56e800b9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:52:59.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.092+0000 7f56faffd640 1 --2- 172.21.15.12:0/1887113212 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56e8005370 0x7f56e800b9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:59.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.092+0000 7f56faffd640 10 monclient: _finish_hunting 0 2024-09-11T01:52:59.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.092+0000 7f56faffd640 1 monclient: found mon.a 2024-09-11T01:52:59.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.092+0000 7f56faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:59.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.092+0000 7f56faffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:52:59.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.092+0000 7f56faffd640 1 -- 172.21.15.12:0/1887113212 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56e8003520 con 0x7f56e8003a00 2024-09-11T01:52:59.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.092+0000 7f56fb7fe640 1 -- 172.21.15.12:0/1887113212 reap_dead start 2024-09-11T01:52:59.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.092+0000 7f56faffd640 10 monclient: _finish_auth 0 2024-09-11T01:52:59.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.092+0000 7f56faffd640 10 monclient: _check_auth_tickets 2024-09-11T01:52:59.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.092+0000 7f56faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:59.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.092+0000 7f56f27fc640 1 --2- 172.21.15.12:0/1887113212 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56e8004800 0x7f56e8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:52:59.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.095+0000 7f56faffd640 1 -- 172.21.15.12:0/1887113212 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56e8003a00 msgr2=0x7f56e8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:59.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.095+0000 7f56faffd640 1 --2- 172.21.15.12:0/1887113212 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56e8003a00 0x7f56e8003e00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009830 tx=0x7f56e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9e88281f801af96 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:59.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.095+0000 7f56fa7fc640 1 -- 172.21.15.12:0/1887113212 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec009430 con 0x7f56e8003a00 2024-09-11T01:52:59.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.095+0000 7f56fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:59.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.095+0000 7f56fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:52:59.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.095+0000 7f56fa7fc640 10 monclient: dump: 2024-09-11T01:52:59.105 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:59.105 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:59.105 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:59.105 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:59.105 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:59.105 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:59.106 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:59.106 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:59.106 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:59.106 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:59.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.095+0000 7f56fa7fc640 1 -- 172.21.15.12:0/1887113212 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56ec02e070 con 0x7f56e8003a00 2024-09-11T01:52:59.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.095+0000 7f570173d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:52:59.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.095+0000 7f56fa7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:59.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.095+0000 7f56fa7fc640 1 -- 172.21.15.12:0/1887113212 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec02a850 con 0x7f56e8003a00 2024-09-11T01:52:59.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.095+0000 7f56fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:59.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.095+0000 7f56fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:52:59.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.095+0000 7f56fa7fc640 10 monclient: dump: 2024-09-11T01:52:59.106 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:59.106 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:59.106 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:59.107 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:59.107 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:59.107 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:59.107 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:59.107 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:59.107 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:59.107 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:59.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.095+0000 7f570173d640 10 monclient: get_monmap_and_config success 2024-09-11T01:52:59.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.095+0000 7f570173d640 4 set_mon_vals no callback set 2024-09-11T01:52:59.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.095+0000 7f570173d640 10 monclient: shutdown 2024-09-11T01:52:59.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.095+0000 7f570173d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:59.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.095+0000 7f570173d640 1 -- 172.21.15.12:0/1887113212 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56e8003a00 msgr2=0x7f56e8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:59.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.095+0000 7f570173d640 1 --2- 172.21.15.12:0/1887113212 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56e8003a00 0x7f56e8003e00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009830 tx=0x7f56e8005830 comp rx=0 tx=0).stop 2024-09-11T01:52:59.107 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.095+0000 7f570173d640 1 -- 172.21.15.12:0/1887113212 shutdown_connections 2024-09-11T01:52:59.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.095+0000 7f570173d640 1 --2- 172.21.15.12:0/1887113212 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56e8003a00 0x7f56e8003e00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:59.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.095+0000 7f570173d640 1 -- 172.21.15.12:0/1887113212 >> 172.21.15.12:0/1887113212 conn(0x7f56fc0a3650 msgr2=0x7f56fc0a3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:59.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.095+0000 7f570173d640 1 -- 172.21.15.12:0/1887113212 shutdown_connections 2024-09-11T01:52:59.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.095+0000 7f570173d640 1 -- 172.21.15.12:0/1887113212 wait complete. 2024-09-11T01:52:59.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.096+0000 7f570173d640 10 monclient: build_initial_monmap 2024-09-11T01:52:59.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.096+0000 7f570173d640 10 monclient: monmap: 2024-09-11T01:52:59.108 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:52:59.108 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:52:59.108 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:52:59.108 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:52:59.108 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:52:59.108 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:59.108 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:52:59.109 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:52:59.109 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:52:59.109 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:59.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.096+0000 7f570173d640 1 Processor -- start 2024-09-11T01:52:59.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f570173d640 1 -- start start 2024-09-11T01:52:59.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f570173d640 10 monclient: init 2024-09-11T01:52:59.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f570173d640 10 monclient: _reopen_session rank -1 2024-09-11T01:52:59.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f570173d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:52:59.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f570173d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56e8003a00 0x7f56fc1cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:59.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f570173d640 10 monclient(hunting): picked mon.noname-a con 0x7f56e8003a00 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:52:59.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f570173d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56fc1ce330 0x7f56fc11ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:59.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f570173d640 10 monclient(hunting): picked mon.noname-b con 0x7f56fc1ce330 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:52:59.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f570173d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56fc11f470 0x7f56fc11f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:59.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f570173d640 10 monclient(hunting): picked mon.noname-c con 0x7f56fc11f470 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:52:59.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f570173d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:59.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f570173d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc070840 con 0x7f56e8003a00 2024-09-11T01:52:59.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f570173d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:59.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f570173d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc070340 con 0x7f56fc1ce330 2024-09-11T01:52:59.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f570173d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:52:59.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f570173d640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f56fc0a0660 con 0x7f56fc11f470 2024-09-11T01:52:59.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f570173d640 10 monclient(hunting): _renew_subs 2024-09-11T01:52:59.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f570173d640 10 monclient(hunting): authenticate will time out at 2185.539677s 2024-09-11T01:52:59.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f56faffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56e8003a00 0x7f56fc1cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:59.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f56f27fc640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56fc1ce330 msgr2=0x7f56fc11ef30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:52:59.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f56fb7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56fc11f470 0x7f56fc11f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:59.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f56faffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56e8003a00 0x7f56fc1cddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34032/0 (socket says 172.21.15.12:34032) 2024-09-11T01:52:59.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f56faffd640 1 -- 172.21.15.12:0/3900171162 learned_addr learned my addr 172.21.15.12:0/3900171162 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:52:59.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f56f27fc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56fc1ce330 0x7f56fc11ef30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:52:59.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f56faffd640 10 monclient(hunting): get_auth_request con 0x7f56e8003a00 auth_method 0 2024-09-11T01:52:59.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f56faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:59.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f56faffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:59.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f56faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:59.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f56fb7fe640 10 monclient(hunting): get_auth_request con 0x7f56fc11f470 auth_method 0 2024-09-11T01:52:59.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f56fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:52:59.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f56fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:52:59.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f56fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:52:59.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f56faffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:59.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f56faffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:59.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f56faffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:59.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f56fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:52:59.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f56fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:52:59.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.097+0000 7f56fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:52:59.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.098+0000 7f56faffd640 10 monclient(hunting): handle_auth_done global_id 264168 payload 824 2024-09-11T01:52:59.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.098+0000 7f56faffd640 1 -- 172.21.15.12:0/3900171162 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56fc11f470 msgr2=0x7f56fc11f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:59.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.098+0000 7f56faffd640 1 --2- 172.21.15.12:0/3900171162 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56fc11f470 0x7f56fc11f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:59.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.098+0000 7f56faffd640 1 -- 172.21.15.12:0/3900171162 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56fc1ce330 msgr2=0x7f56fc11ef30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:52:59.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.098+0000 7f56faffd640 1 --2- 172.21.15.12:0/3900171162 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56fc1ce330 0x7f56fc11ef30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:59.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.098+0000 7f56faffd640 10 monclient: _finish_hunting 0 2024-09-11T01:52:59.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.098+0000 7f56faffd640 1 monclient: found mon.noname-a 2024-09-11T01:52:59.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.098+0000 7f56faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:52:59.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.098+0000 7f56faffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:52:59.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.098+0000 7f56faffd640 1 -- 172.21.15.12:0/3900171162 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56fc122d50 con 0x7f56e8003a00 2024-09-11T01:52:59.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.098+0000 7f56fb7fe640 1 --2- 172.21.15.12:0/3900171162 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56fc11f470 0x7f56fc11f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:52:59.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.100+0000 7f56faffd640 1 -- 172.21.15.12:0/3900171162 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56e8003a00 msgr2=0x7f56fc1cddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:52:59.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.100+0000 7f56faffd640 1 --2- 172.21.15.12:0/3900171162 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56e8003a00 0x7f56fc1cddf0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009950 tx=0x7f56ec005940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:59.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.100+0000 7f56f3fff640 1 -- 172.21.15.12:0/3900171162 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec005270 con 0x7f56e8003a00 2024-09-11T01:52:59.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.100+0000 7f56f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:59.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.100+0000 7f56f3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:52:59.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.100+0000 7f56f3fff640 10 monclient: dump: 2024-09-11T01:52:59.117 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:59.117 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:59.117 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:59.117 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:59.117 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:59.117 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:59.117 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:59.117 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:59.117 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:59.117 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:59.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.100+0000 7f56f3fff640 10 monclient: _finish_auth 0 2024-09-11T01:52:59.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.100+0000 7f56f3fff640 10 monclient: _check_auth_tickets 2024-09-11T01:52:59.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.100+0000 7f56f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:52:59.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.100+0000 7f56f3fff640 1 -- 172.21.15.12:0/3900171162 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56ec043070 con 0x7f56e8003a00 2024-09-11T01:52:59.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.100+0000 7f570173d640 5 monclient: authenticate success, global_id 264168 2024-09-11T01:52:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.100+0000 7f570173d640 10 monclient: _renew_subs 2024-09-11T01:52:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.100+0000 7f570173d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:52:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.100+0000 7f570173d640 1 -- 172.21.15.12:0/3900171162 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56fc0a6500 con 0x7f56e8003a00 2024-09-11T01:52:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.100+0000 7f56f3fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:52:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.100+0000 7f56f3fff640 1 -- 172.21.15.12:0/3900171162 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec005650 con 0x7f56e8003a00 2024-09-11T01:52:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.100+0000 7f56f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:52:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.100+0000 7f56f3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:52:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.100+0000 7f56f3fff640 10 monclient: dump: 2024-09-11T01:52:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:52:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:52:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:52:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:52:59.118 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:52:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:52:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:52:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:52:59.119 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:52:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.100+0000 7f56f9ffb640 4 set_mon_vals no callback set 2024-09-11T01:52:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.100+0000 7f570173d640 10 monclient: _renew_subs 2024-09-11T01:52:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.100+0000 7f570173d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:52:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.100+0000 7f570173d640 1 -- 172.21.15.12:0/3900171162 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56fc131d50 con 0x7f56e8003a00 2024-09-11T01:52:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.101+0000 7f570173d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.101+0000 7f570173d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.101+0000 7f570173d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:52:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.101+0000 7f56f3fff640 1 -- 172.21.15.12:0/3900171162 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f56ec03e470 con 0x7f56e8003a00 2024-09-11T01:52:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.101+0000 7f570173d640 1 -- 172.21.15.12:0/3900171162 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56fc074fb0 con 0x7f56e8003a00 2024-09-11T01:52:59.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.101+0000 7f56f3fff640 1 --2- 172.21.15.12:0/3900171162 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f56dc03e240 0x7f56dc040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:52:59.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.101+0000 7f56f3fff640 1 -- 172.21.15.12:0/3900171162 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(804..804 src has 676..804) ==== 4739+0+0 (secure 0 0 0) 0x7f56ec072fc0 con 0x7f56e8003a00 2024-09-11T01:52:59.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.102+0000 7f56f27fc640 1 --2- 172.21.15.12:0/3900171162 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f56dc03e240 0x7f56dc040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:52:59.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.102+0000 7f56f27fc640 10 monclient: get_auth_request con 0x7f56dc03e240 auth_method 0 2024-09-11T01:52:59.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.104+0000 7f56f27fc640 1 --2- 172.21.15.12:0/3900171162 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f56dc03e240 0x7f56dc040700 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f56e40015f0 tx=0x7f56e4004d10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:52:59.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.106+0000 7f56f3fff640 1 -- 172.21.15.12:0/3900171162 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f56ec04d020 con 0x7f56e8003a00 2024-09-11T01:52:59.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.106+0000 7f56f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:52:59.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.106+0000 7f56f3fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:52:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.365+0000 7f570173d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:52:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.365+0000 7f570173d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:52:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.366+0000 7f570173d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:52:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.366+0000 7f570173d640 1 -- 172.21.15.12:0/3900171162 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f56fc077460 con 0x7f56e8003a00 2024-09-11T01:52:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.366+0000 7f56f3fff640 1 -- 172.21.15.12:0/3900171162 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f56fc077460 con 0x7f56e8003a00 2024-09-11T01:52:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.366+0000 7f56f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:52:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.366+0000 7f56f3fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:52:59.368 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":220,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:52:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.368+0000 7f570173d640 1 -- 172.21.15.12:0/3900171162 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f56dc03e240 msgr2=0x7f56dc040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.368+0000 7f570173d640 1 --2- 172.21.15.12:0/3900171162 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f56dc03e240 0x7f56dc040700 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f56e40015f0 tx=0x7f56e4004d10 comp rx=0 tx=0).stop 2024-09-11T01:52:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.368+0000 7f570173d640 10 monclient: shutdown 2024-09-11T01:52:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.368+0000 7f570173d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:52:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.368+0000 7f570173d640 1 -- 172.21.15.12:0/3900171162 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56e8003a00 msgr2=0x7f56fc1cddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:52:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.368+0000 7f570173d640 1 --2- 172.21.15.12:0/3900171162 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56e8003a00 0x7f56fc1cddf0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009950 tx=0x7f56ec005940 comp rx=0 tx=0).stop 2024-09-11T01:52:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.368+0000 7f570173d640 1 -- 172.21.15.12:0/3900171162 shutdown_connections 2024-09-11T01:52:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.368+0000 7f570173d640 1 --2- 172.21.15.12:0/3900171162 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f56fc11f470 0x7f56fc11f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.368+0000 7f570173d640 1 --2- 172.21.15.12:0/3900171162 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56fc1ce330 0x7f56fc11ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.368+0000 7f570173d640 1 --2- 172.21.15.12:0/3900171162 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f56dc03e240 0x7f56dc040700 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.368+0000 7f570173d640 1 --2- 172.21.15.12:0/3900171162 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56e8003a00 0x7f56fc1cddf0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:52:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.368+0000 7f570173d640 1 -- 172.21.15.12:0/3900171162 >> 172.21.15.12:0/3900171162 conn(0x7f56fc0a3650 msgr2=0x7f56fc1bda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:52:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.368+0000 7f570173d640 1 -- 172.21.15.12:0/3900171162 shutdown_connections 2024-09-11T01:52:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:52:59.368+0000 7f570173d640 1 -- 172.21.15.12:0/3900171162 wait complete. 2024-09-11T01:53:02.381 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:53:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.492+0000 7f70545b4640 10 monclient: get_monmap_and_config 2024-09-11T01:53:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.492+0000 7f70545b4640 10 monclient: build_initial_monmap 2024-09-11T01:53:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.492+0000 7f70545b4640 10 monclient: monmap: 2024-09-11T01:53:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:53:02.493571+0000 2024-09-11T01:53:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:53:02.493571+0000 2024-09-11T01:53:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:53:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:53:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:53:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:53:02.495 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:53:02.495 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.493+0000 7f70545b4640 1 Processor -- start 2024-09-11T01:53:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.493+0000 7f70545b4640 1 -- start start 2024-09-11T01:53:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.493+0000 7f70545b4640 10 monclient: init 2024-09-11T01:53:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f70545b4640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f70545b4640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-11T01:53:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f70545b4640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f704c0a2e90 0x7f704c06dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f70545b4640 10 monclient(hunting): picked mon.noname-c con 0x7f704c0a2e90 addr v2:172.21.15.71:3300/0 2024-09-11T01:53:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f70545b4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f704c0a3270 0x7f704c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:02.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f70545b4640 10 monclient(hunting): picked mon.noname-a con 0x7f704c0a3270 addr v2:172.21.15.12:3300/0 2024-09-11T01:53:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f70545b4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f704c138040 addr v1:172.21.15.12:6789/0 2024-09-11T01:53:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f70545b4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f704c0a46c0 con 0x7f704c138040 2024-09-11T01:53:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f70545b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f70545b4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f704c0a4900 con 0x7f704c0a3270 2024-09-11T01:53:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f70545b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f70545b4640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f704c06ee50 con 0x7f704c0a2e90 2024-09-11T01:53:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f70545b4640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f70545b4640 10 monclient(hunting): authenticate will time out at 2188.936267s 2024-09-11T01:53:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f7051b4f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f704c0a3270 0x7f704c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f7052350640 1 -- >> v2:172.21.15.71:3300/0 conn(0x7f704c0a2e90 msgr2=0x7f704c06dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:53:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f7052350640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f704c0a2e90 0x7f704c06dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:53:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f7051b4f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f704c0a3270 0x7f704c0a3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34046/0 (socket says 172.21.15.12:34046) 2024-09-11T01:53:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f7051b4f640 1 -- 172.21.15.12:0/3697312534 learned_addr learned my addr 172.21.15.12:0/3697312534 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:53:02.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f7051b4f640 10 monclient(hunting): get_auth_request con 0x7f704c0a3270 auth_method 0 2024-09-11T01:53:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f7051b4f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f7051b4f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f7051b4f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f705134e640 1 -- 172.21.15.12:0/3697312534 DispatchQueue::entry inject delay of 0.002000 2024-09-11T01:53:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f7051b4f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f7051b4f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:02.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.494+0000 7f7051b4f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.496+0000 7f7052b51640 1 -- 172.21.15.12:0/3697312534 >> v1:172.21.15.12:6789/0 conn(0x7f704c138040 legacy=0x7f704c1384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.496+0000 7f7051b4f640 10 monclient(hunting): handle_auth_done global_id 264171 payload 274 2024-09-11T01:53:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.496+0000 7f7051b4f640 1 -- 172.21.15.12:0/3697312534 >> v2:172.21.15.71:3300/0 conn(0x7f704c0a2e90 msgr2=0x7f704c06dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:53:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.496+0000 7f7051b4f640 1 --2- 172.21.15.12:0/3697312534 >> v2:172.21.15.71:3300/0 conn(0x7f704c0a2e90 0x7f704c06dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.496+0000 7f7051b4f640 1 -- 172.21.15.12:0/3697312534 >> v1:172.21.15.12:6789/0 conn(0x7f704c138040 legacy=0x7f704c1384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.496+0000 7f7051b4f640 10 monclient: _finish_hunting 0 2024-09-11T01:53:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.496+0000 7f7051b4f640 1 monclient: found mon.noname-a 2024-09-11T01:53:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.496+0000 7f7051b4f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:02.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.496+0000 7f7051b4f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:53:02.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.496+0000 7f7051b4f640 1 -- 172.21.15.12:0/3697312534 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f704c06eb70 con 0x7f704c0a3270 2024-09-11T01:53:02.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.499+0000 7f7051b4f640 1 -- 172.21.15.12:0/3697312534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f704c0a3270 msgr2=0x7f704c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:02.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.499+0000 7f7051b4f640 1 --2- 172.21.15.12:0/3697312534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f704c0a3270 0x7f704c0a3680 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f704c071140 tx=0x7f703c02a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1631ef8df83d6a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:02.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.499+0000 7f705134e640 1 -- 172.21.15.12:0/3697312534 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703c037070 con 0x7f704c0a3270 2024-09-11T01:53:02.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.499+0000 7f705134e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:02.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.499+0000 7f705134e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:53:02.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.499+0000 7f705134e640 10 monclient: dump: 2024-09-11T01:53:02.503 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:02.503 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:02.503 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:02.503 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:02.503 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:02.503 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:02.503 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:02.503 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:02.503 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:02.503 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:02.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.499+0000 7f705134e640 10 monclient: _finish_auth 0 2024-09-11T01:53:02.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.499+0000 7f705134e640 10 monclient: _check_auth_tickets 2024-09-11T01:53:02.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.499+0000 7f705134e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:02.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.499+0000 7f70545b4640 5 monclient: authenticate success, global_id 264171 2024-09-11T01:53:02.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.499+0000 7f705134e640 1 -- 172.21.15.12:0/3697312534 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f703c032640 con 0x7f704c0a3270 2024-09-11T01:53:02.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.499+0000 7f70545b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:53:02.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.499+0000 7f705134e640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:02.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.499+0000 7f705134e640 1 -- 172.21.15.12:0/3697312534 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703c032a10 con 0x7f704c0a3270 2024-09-11T01:53:02.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.499+0000 7f70545b4640 10 monclient: get_monmap_and_config success 2024-09-11T01:53:02.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.499+0000 7f70545b4640 4 set_mon_vals no callback set 2024-09-11T01:53:02.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.499+0000 7f705134e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:02.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.499+0000 7f705134e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:53:02.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.499+0000 7f705134e640 10 monclient: dump: 2024-09-11T01:53:02.504 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:02.504 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:02.505 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:02.505 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:02.505 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:02.505 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:02.505 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:02.505 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:02.505 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:02.505 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:02.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.500+0000 7f70545b4640 10 monclient: shutdown 2024-09-11T01:53:02.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.500+0000 7f70545b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:02.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.500+0000 7f70545b4640 1 -- 172.21.15.12:0/3697312534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f704c0a3270 msgr2=0x7f704c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:02.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.500+0000 7f70545b4640 1 --2- 172.21.15.12:0/3697312534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f704c0a3270 0x7f704c0a3680 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f704c071140 tx=0x7f703c02a200 comp rx=0 tx=0).stop 2024-09-11T01:53:02.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.500+0000 7f70545b4640 1 -- 172.21.15.12:0/3697312534 shutdown_connections 2024-09-11T01:53:02.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.500+0000 7f70545b4640 1 --2- 172.21.15.12:0/3697312534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f704c0a3270 0x7f704c0a3680 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:02.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.500+0000 7f70545b4640 1 --2- 172.21.15.12:0/3697312534 >> v2:172.21.15.71:3300/0 conn(0x7f704c0a2e90 0x7f704c06dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:02.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.500+0000 7f70545b4640 1 -- 172.21.15.12:0/3697312534 >> 172.21.15.12:0/3697312534 conn(0x7f704c0a8490 msgr2=0x7f704c0a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:02.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.500+0000 7f70545b4640 1 -- 172.21.15.12:0/3697312534 shutdown_connections 2024-09-11T01:53:02.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.500+0000 7f70545b4640 1 -- 172.21.15.12:0/3697312534 wait complete. 2024-09-11T01:53:02.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.501+0000 7f70545b4640 10 monclient: build_initial_monmap 2024-09-11T01:53:02.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.501+0000 7f70545b4640 10 monclient: monmap: 2024-09-11T01:53:02.506 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:02.506 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:53:02.506 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:53:02.506 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:53:02.506 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:02.506 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:02.506 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:53:02.507 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:53:02.507 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:53:02.507 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:02.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.501+0000 7f70545b4640 1 Processor -- start 2024-09-11T01:53:02.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.501+0000 7f70545b4640 1 -- start start 2024-09-11T01:53:02.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.501+0000 7f70545b4640 10 monclient: init 2024-09-11T01:53:02.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f70545b4640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:02.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f70545b4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:53:02.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f70545b4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f704c0a2e90 0x7f704c1cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:02.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f70545b4640 10 monclient(hunting): picked mon.noname-a con 0x7f704c0a2e90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:53:02.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f70545b4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f704c0a3270 0x7f704c1cda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:02.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f70545b4640 10 monclient(hunting): picked mon.noname-b con 0x7f704c0a3270 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:53:02.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f70545b4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f704c138040 0x7f704c124830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:02.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f70545b4640 10 monclient(hunting): picked mon.noname-c con 0x7f704c138040 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:53:02.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f70545b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:02.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f70545b4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f704c06fba0 con 0x7f704c0a2e90 2024-09-11T01:53:02.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f70545b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:02.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f70545b4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f704c0a3bc0 con 0x7f704c0a3270 2024-09-11T01:53:02.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f70545b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:02.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f70545b4640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f704c06fd20 con 0x7f704c138040 2024-09-11T01:53:02.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f70545b4640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:02.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f70545b4640 10 monclient(hunting): authenticate will time out at 2188.944238s 2024-09-11T01:53:02.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052350640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f704c0a2e90 0x7f704c1cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:02.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052350640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f704c0a2e90 0x7f704c1cd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34054/0 (socket says 172.21.15.12:34054) 2024-09-11T01:53:02.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7051b4f640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f704c0a3270 msgr2=0x7f704c1cda20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:53:02.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052b51640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f704c138040 0x7f704c124830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:02.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052350640 1 -- 172.21.15.12:0/3047536123 learned_addr learned my addr 172.21.15.12:0/3047536123 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:53:02.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052350640 10 monclient(hunting): get_auth_request con 0x7f704c0a2e90 auth_method 0 2024-09-11T01:53:02.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052350640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:02.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052350640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:02.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052350640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:02.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7051b4f640 1 --2- 172.21.15.12:0/3047536123 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f704c0a3270 0x7f704c1cda20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:53:02.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052b51640 10 monclient(hunting): get_auth_request con 0x7f704c138040 auth_method 0 2024-09-11T01:53:02.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052b51640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:02.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052b51640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:02.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052b51640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:02.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052350640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:02.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052350640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:02.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052350640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:02.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052b51640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:02.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052b51640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:02.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052b51640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:02.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052350640 10 monclient(hunting): handle_auth_done global_id 264174 payload 824 2024-09-11T01:53:02.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052350640 1 -- 172.21.15.12:0/3047536123 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f704c138040 msgr2=0x7f704c124830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:02.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052350640 1 --2- 172.21.15.12:0/3047536123 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f704c138040 0x7f704c124830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:02.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052350640 1 -- 172.21.15.12:0/3047536123 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f704c0a3270 msgr2=0x7f704c1cda20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:53:02.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052350640 1 --2- 172.21.15.12:0/3047536123 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f704c0a3270 0x7f704c1cda20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:02.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052350640 10 monclient: _finish_hunting 0 2024-09-11T01:53:02.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052350640 1 monclient: found mon.noname-a 2024-09-11T01:53:02.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052350640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:02.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052350640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:53:02.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.502+0000 7f7052350640 1 -- 172.21.15.12:0/3047536123 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f704c128650 con 0x7f704c0a2e90 2024-09-11T01:53:02.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.504+0000 7f7052350640 1 -- 172.21.15.12:0/3047536123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f704c0a2e90 msgr2=0x7f704c1cd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:02.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.504+0000 7f7052350640 1 --2- 172.21.15.12:0/3047536123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f704c0a2e90 0x7f704c1cd4e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f704800b550 tx=0x7f704800ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:02.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.505+0000 7f7042ffd640 1 -- 172.21.15.12:0/3047536123 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7048013020 con 0x7f704c0a2e90 2024-09-11T01:53:02.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.505+0000 7f7042ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:02.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.505+0000 7f7042ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:53:02.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.505+0000 7f7042ffd640 10 monclient: dump: 2024-09-11T01:53:02.515 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:02.515 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:02.515 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:02.515 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:02.515 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:02.515 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:02.515 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:02.515 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:02.515 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:02.515 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:02.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.505+0000 7f7042ffd640 10 monclient: _finish_auth 0 2024-09-11T01:53:02.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.505+0000 7f7042ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:53:02.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.505+0000 7f7042ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:02.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.505+0000 7f7042ffd640 1 -- 172.21.15.12:0/3047536123 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7048005070 con 0x7f704c0a2e90 2024-09-11T01:53:02.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.505+0000 7f70545b4640 5 monclient: authenticate success, global_id 264174 2024-09-11T01:53:02.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.505+0000 7f70545b4640 10 monclient: _renew_subs 2024-09-11T01:53:02.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.505+0000 7f70545b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:02.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.505+0000 7f70545b4640 1 -- 172.21.15.12:0/3047536123 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f704c09f680 con 0x7f704c0a2e90 2024-09-11T01:53:02.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.505+0000 7f7042ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:02.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.505+0000 7f7042ffd640 1 -- 172.21.15.12:0/3047536123 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7048005480 con 0x7f704c0a2e90 2024-09-11T01:53:02.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.505+0000 7f7042ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:02.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.505+0000 7f7042ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:53:02.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.505+0000 7f7042ffd640 10 monclient: dump: 2024-09-11T01:53:02.516 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:02.516 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:02.517 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:02.517 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:02.517 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:02.517 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:02.517 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:02.517 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:02.517 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:02.517 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:02.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.505+0000 7f70545b4640 10 monclient: _renew_subs 2024-09-11T01:53:02.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.505+0000 7f70545b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:02.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.505+0000 7f7050b4d640 4 set_mon_vals no callback set 2024-09-11T01:53:02.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.505+0000 7f70545b4640 1 -- 172.21.15.12:0/3047536123 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f704c128370 con 0x7f704c0a2e90 2024-09-11T01:53:02.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.506+0000 7f7042ffd640 1 -- 172.21.15.12:0/3047536123 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f7048005620 con 0x7f704c0a2e90 2024-09-11T01:53:02.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.506+0000 7f70545b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:02.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.506+0000 7f70545b4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:02.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.506+0000 7f70545b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:02.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.506+0000 7f70545b4640 1 -- 172.21.15.12:0/3047536123 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f704c06dc60 con 0x7f704c0a2e90 2024-09-11T01:53:02.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.506+0000 7f7042ffd640 1 --2- 172.21.15.12:0/3047536123 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f703003e1c0 0x7f7030040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:02.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.506+0000 7f7042ffd640 1 -- 172.21.15.12:0/3047536123 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(804..804 src has 676..804) ==== 4739+0+0 (secure 0 0 0) 0x7f7048052a10 con 0x7f704c0a2e90 2024-09-11T01:53:02.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.506+0000 7f7051b4f640 1 --2- 172.21.15.12:0/3047536123 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f703003e1c0 0x7f7030040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:02.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.506+0000 7f7051b4f640 10 monclient: get_auth_request con 0x7f703003e1c0 auth_method 0 2024-09-11T01:53:02.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.509+0000 7f7051b4f640 1 --2- 172.21.15.12:0/3047536123 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f703003e1c0 0x7f7030040680 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f704c0a94a0 tx=0x7f703c02e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:02.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.510+0000 7f7042ffd640 1 -- 172.21.15.12:0/3047536123 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f704801d030 con 0x7f704c0a2e90 2024-09-11T01:53:02.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.510+0000 7f7042ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:02.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.510+0000 7f7042ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:53:02.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.767+0000 7f70545b4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:53:02.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.767+0000 7f70545b4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:53:02.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.767+0000 7f70545b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:02.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.767+0000 7f70545b4640 1 -- 172.21.15.12:0/3047536123 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f704c126e50 con 0x7f704c0a2e90 2024-09-11T01:53:02.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.767+0000 7f7042ffd640 1 -- 172.21.15.12:0/3047536123 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7f7048051430 con 0x7f704c0a2e90 2024-09-11T01:53:02.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.767+0000 7f7042ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:53:02.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.767+0000 7f7042ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:53:02.769 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":220,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:53:02.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.769+0000 7f70545b4640 1 -- 172.21.15.12:0/3047536123 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f703003e1c0 msgr2=0x7f7030040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:02.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.769+0000 7f70545b4640 1 --2- 172.21.15.12:0/3047536123 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f703003e1c0 0x7f7030040680 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f704c0a94a0 tx=0x7f703c02e000 comp rx=0 tx=0).stop 2024-09-11T01:53:02.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.769+0000 7f70545b4640 10 monclient: shutdown 2024-09-11T01:53:02.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.769+0000 7f70545b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:02.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.769+0000 7f70545b4640 1 -- 172.21.15.12:0/3047536123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f704c0a2e90 msgr2=0x7f704c1cd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:02.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.769+0000 7f70545b4640 1 --2- 172.21.15.12:0/3047536123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f704c0a2e90 0x7f704c1cd4e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f704800b550 tx=0x7f704800ba10 comp rx=0 tx=0).stop 2024-09-11T01:53:02.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.769+0000 7f70545b4640 1 -- 172.21.15.12:0/3047536123 shutdown_connections 2024-09-11T01:53:02.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.769+0000 7f70545b4640 1 --2- 172.21.15.12:0/3047536123 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f704c138040 0x7f704c124830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:02.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.769+0000 7f70545b4640 1 --2- 172.21.15.12:0/3047536123 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f704c0a3270 0x7f704c1cda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:02.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.769+0000 7f70545b4640 1 --2- 172.21.15.12:0/3047536123 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f703003e1c0 0x7f7030040680 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:02.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.769+0000 7f70545b4640 1 --2- 172.21.15.12:0/3047536123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f704c0a2e90 0x7f704c1cd4e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:02.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.769+0000 7f70545b4640 1 -- 172.21.15.12:0/3047536123 >> 172.21.15.12:0/3047536123 conn(0x7f704c0a8490 msgr2=0x7f704c072090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:02.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.769+0000 7f70545b4640 1 -- 172.21.15.12:0/3047536123 shutdown_connections 2024-09-11T01:53:02.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:02.769+0000 7f70545b4640 1 -- 172.21.15.12:0/3047536123 wait complete. 2024-09-11T01:53:05.783 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:53:05.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.895+0000 7feb2a9f8640 10 monclient: get_monmap_and_config 2024-09-11T01:53:05.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.895+0000 7feb2a9f8640 10 monclient: build_initial_monmap 2024-09-11T01:53:05.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.895+0000 7feb2a9f8640 10 monclient: monmap: 2024-09-11T01:53:05.897 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:05.897 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:05.897 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:53:05.895935+0000 2024-09-11T01:53:05.897 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:53:05.895935+0000 2024-09-11T01:53:05.897 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:05.897 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:05.897 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:53:05.897 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:53:05.897 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:53:05.897 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:53:05.897 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:53:05.897 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:05.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.895+0000 7feb2a9f8640 1 Processor -- start 2024-09-11T01:53:05.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb2a9f8640 1 -- start start 2024-09-11T01:53:05.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb2a9f8640 10 monclient: init 2024-09-11T01:53:05.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb2a9f8640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:05.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb2a9f8640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-11T01:53:05.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb2a9f8640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feb24138230 0x7feb24138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:05.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb2a9f8640 10 monclient(hunting): picked mon.noname-a con 0x7feb24138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:53:05.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb2a9f8640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7feb2409d030 0x7feb240a5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:05.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb2a9f8640 10 monclient(hunting): picked mon.noname-c con 0x7feb2409d030 addr v2:172.21.15.71:3300/0 2024-09-11T01:53:05.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb2a9f8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feb2409d460 addr v1:172.21.15.12:6789/0 2024-09-11T01:53:05.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb2a9f8640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb240a6e30 con 0x7feb2409d460 2024-09-11T01:53:05.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb2a9f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:05.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb2a9f8640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7feb2409c9c0 con 0x7feb24138230 2024-09-11T01:53:05.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb2a9f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:05.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb2a9f8640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7feb2409ba20 con 0x7feb2409d030 2024-09-11T01:53:05.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb2a9f8640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:05.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb2a9f8640 10 monclient(hunting): authenticate will time out at 2192.338702s 2024-09-11T01:53:05.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb23fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feb24138230 0x7feb24138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:05.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb237fe640 1 -- >> v2:172.21.15.71:3300/0 conn(0x7feb2409d030 msgr2=0x7feb240a5020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:53:05.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb237fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7feb2409d030 0x7feb240a5020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:53:05.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb23fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feb24138230 0x7feb24138610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34056/0 (socket says 172.21.15.12:34056) 2024-09-11T01:53:05.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb23fff640 1 -- 172.21.15.12:0/1873142170 learned_addr learned my addr 172.21.15.12:0/1873142170 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:53:05.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb23fff640 10 monclient(hunting): get_auth_request con 0x7feb24138230 auth_method 0 2024-09-11T01:53:05.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:05.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb23fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:05.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:05.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb28f95640 1 --1- 172.21.15.12:0/1873142170 >> v1:172.21.15.12:6789/0 conn(0x7feb2409d460 0x7feb240a5560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52702/0 (socket says 172.21.15.12:52702) 2024-09-11T01:53:05.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.896+0000 7feb22ffd640 1 -- 172.21.15.12:0/1873142170 DispatchQueue::entry inject delay of 0.002000 2024-09-11T01:53:05.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.898+0000 7feb28f95640 1 -- 172.21.15.12:0/1873142170 >> v1:172.21.15.12:6789/0 conn(0x7feb2409d460 legacy=0x7feb240a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:05.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.899+0000 7feb23fff640 1 -- 172.21.15.12:0/1873142170 >> v2:172.21.15.12:3300/0 conn(0x7feb24138230 msgr2=0x7feb24138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:53:05.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.899+0000 7feb23fff640 1 -- 172.21.15.12:0/1873142170 >> v2:172.21.15.12:3300/0 conn(0x7feb24138230 msgr2=0x7feb24138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:53:05.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.899+0000 7feb23fff640 1 --2- 172.21.15.12:0/1873142170 >> v2:172.21.15.12:3300/0 conn(0x7feb24138230 0x7feb24138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:53:05.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:05.899+0000 7feb23fff640 1 --2- 172.21.15.12:0/1873142170 >> v2:172.21.15.12:3300/0 conn(0x7feb24138230 0x7feb24138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:53:06.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:06.097+0000 7feb237fe640 1 -- 172.21.15.12:0/1873142170 >> v2:172.21.15.71:3300/0 conn(0x7feb2409d030 msgr2=0x7feb240a5020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:53:06.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:06.097+0000 7feb237fe640 1 --2- 172.21.15.12:0/1873142170 >> v2:172.21.15.71:3300/0 conn(0x7feb2409d030 0x7feb240a5020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:53:06.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:06.100+0000 7feb23fff640 1 --2- 172.21.15.12:0/1873142170 >> v2:172.21.15.12:3300/0 conn(0x7feb24138230 0x7feb24138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:06.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:06.100+0000 7feb23fff640 10 monclient(hunting): get_auth_request con 0x7feb24138230 auth_method 0 2024-09-11T01:53:06.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:06.100+0000 7feb23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:06.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:06.100+0000 7feb23fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:06.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:06.100+0000 7feb23fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:53:06.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:06.100+0000 7feb23fff640 1 -- 172.21.15.12:0/1873142170 >> v2:172.21.15.12:3300/0 conn(0x7feb24138230 msgr2=0x7feb24138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:53:06.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:06.100+0000 7feb23fff640 1 -- 172.21.15.12:0/1873142170 >> v2:172.21.15.12:3300/0 conn(0x7feb24138230 msgr2=0x7feb24138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:53:06.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:06.100+0000 7feb23fff640 1 --2- 172.21.15.12:0/1873142170 >> v2:172.21.15.12:3300/0 conn(0x7feb24138230 0x7feb24138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:53:06.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:06.100+0000 7feb23fff640 1 --2- 172.21.15.12:0/1873142170 >> v2:172.21.15.12:3300/0 conn(0x7feb24138230 0x7feb24138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:53:06.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:06.499+0000 7feb237fe640 1 -- 172.21.15.12:0/1873142170 >> v2:172.21.15.71:3300/0 conn(0x7feb2409d030 msgr2=0x7feb240a5020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:53:06.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:06.499+0000 7feb237fe640 1 --2- 172.21.15.12:0/1873142170 >> v2:172.21.15.71:3300/0 conn(0x7feb2409d030 0x7feb240a5020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:53:06.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:06.501+0000 7feb23fff640 1 --2- 172.21.15.12:0/1873142170 >> v2:172.21.15.12:3300/0 conn(0x7feb24138230 0x7feb24138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:06.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:06.501+0000 7feb23fff640 10 monclient(hunting): get_auth_request con 0x7feb24138230 auth_method 0 2024-09-11T01:53:06.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:06.501+0000 7feb23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:06.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:06.501+0000 7feb23fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:06.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:06.501+0000 7feb23fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:53:06.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:06.501+0000 7feb23fff640 1 -- 172.21.15.12:0/1873142170 >> v2:172.21.15.12:3300/0 conn(0x7feb24138230 msgr2=0x7feb24138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:53:06.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:06.501+0000 7feb23fff640 1 -- 172.21.15.12:0/1873142170 >> v2:172.21.15.12:3300/0 conn(0x7feb24138230 msgr2=0x7feb24138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:53:06.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:06.501+0000 7feb23fff640 1 --2- 172.21.15.12:0/1873142170 >> v2:172.21.15.12:3300/0 conn(0x7feb24138230 0x7feb24138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:53:06.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:06.501+0000 7feb23fff640 1 --2- 172.21.15.12:0/1873142170 >> v2:172.21.15.12:3300/0 conn(0x7feb24138230 0x7feb24138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:53:07.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:07.301+0000 7feb237fe640 1 -- 172.21.15.12:0/1873142170 >> v2:172.21.15.71:3300/0 conn(0x7feb2409d030 msgr2=0x7feb240a5020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:53:07.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:07.301+0000 7feb237fe640 1 --2- 172.21.15.12:0/1873142170 >> v2:172.21.15.71:3300/0 conn(0x7feb2409d030 0x7feb240a5020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:53:07.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:07.302+0000 7feb23fff640 1 --2- 172.21.15.12:0/1873142170 >> v2:172.21.15.12:3300/0 conn(0x7feb24138230 0x7feb24138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:07.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:07.302+0000 7feb23fff640 10 monclient(hunting): get_auth_request con 0x7feb24138230 auth_method 0 2024-09-11T01:53:07.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:07.302+0000 7feb23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:07.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:07.302+0000 7feb23fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:07.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:07.302+0000 7feb23fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:53:07.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:07.302+0000 7feb23fff640 1 -- 172.21.15.12:0/1873142170 >> v2:172.21.15.12:3300/0 conn(0x7feb24138230 msgr2=0x7feb24138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:53:07.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:07.302+0000 7feb23fff640 1 -- 172.21.15.12:0/1873142170 >> v2:172.21.15.12:3300/0 conn(0x7feb24138230 msgr2=0x7feb24138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:53:07.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:07.302+0000 7feb23fff640 1 --2- 172.21.15.12:0/1873142170 >> v2:172.21.15.12:3300/0 conn(0x7feb24138230 0x7feb24138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:53:07.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:07.302+0000 7feb23fff640 1 --2- 172.21.15.12:0/1873142170 >> v2:172.21.15.12:3300/0 conn(0x7feb24138230 0x7feb24138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:53:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.896+0000 7feb21ffb640 10 monclient(hunting): tick 2024-09-11T01:53:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.896+0000 7feb21ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:53:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.896+0000 7feb21ffb640 1 monclient(hunting): continuing hunt 2024-09-11T01:53:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.896+0000 7feb21ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:53:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.896+0000 7feb21ffb640 1 -- 172.21.15.12:0/1873142170 >> v2:172.21.15.71:3300/0 conn(0x7feb2409d030 msgr2=0x7feb240a5020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:53:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.896+0000 7feb21ffb640 1 --2- 172.21.15.12:0/1873142170 >> v2:172.21.15.71:3300/0 conn(0x7feb2409d030 0x7feb240a5020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.896+0000 7feb21ffb640 1 -- 172.21.15.12:0/1873142170 >> v2:172.21.15.12:3300/0 conn(0x7feb24138230 msgr2=0x7feb24138610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:53:08.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.896+0000 7feb21ffb640 1 --2- 172.21.15.12:0/1873142170 >> v2:172.21.15.12:3300/0 conn(0x7feb24138230 0x7feb24138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.896+0000 7feb21ffb640 1 -- 172.21.15.12:0/1873142170 >> v1:172.21.15.12:6789/0 conn(0x7feb2409d460 legacy=0x7feb240a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.896+0000 7feb21ffb640 10 monclient: _add_conns ranks=[2,1] 2024-09-11T01:53:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.896+0000 7feb21ffb640 1 --2- 172.21.15.12:0/1873142170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb10003730 0x7feb10005b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.896+0000 7feb21ffb640 10 monclient(hunting): picked mon.noname-b con 0x7feb10003730 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:53:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.896+0000 7feb21ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feb10006290 addr v1:172.21.15.71:6789/0 2024-09-11T01:53:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.896+0000 7feb21ffb640 1 -- 172.21.15.12:0/1873142170 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb240a66a0 con 0x7feb10006290 2024-09-11T01:53:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.896+0000 7feb21ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.896+0000 7feb21ffb640 1 -- 172.21.15.12:0/1873142170 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7feb2409c9c0 con 0x7feb10003730 2024-09-11T01:53:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.896+0000 7feb21ffb640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.896+0000 7feb237fe640 1 -- 172.21.15.12:0/1873142170 >> v1:172.21.15.71:6789/0 conn(0x7feb10006290 legacy=0x7feb10008750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:53:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.896+0000 7feb23fff640 1 --2- 172.21.15.12:0/1873142170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb10003730 0x7feb10005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.897+0000 7feb23fff640 10 monclient(hunting): get_auth_request con 0x7feb10003730 auth_method 0 2024-09-11T01:53:08.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.897+0000 7feb23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:08.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.897+0000 7feb23fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:08.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.897+0000 7feb23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:08.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.897+0000 7feb23fff640 1 -- 172.21.15.12:0/1873142170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb10003730 msgr2=0x7feb10005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:53:08.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.897+0000 7feb23fff640 1 -- 172.21.15.12:0/1873142170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb10003730 msgr2=0x7feb10005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:53:08.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.897+0000 7feb23fff640 1 --2- 172.21.15.12:0/1873142170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb10003730 0x7feb10005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:53:08.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:08.897+0000 7feb23fff640 1 --2- 172.21.15.12:0/1873142170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb10003730 0x7feb10005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:53:09.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.097+0000 7feb237fe640 1 -- 172.21.15.12:0/1873142170 >> v1:172.21.15.71:6789/0 conn(0x7feb10006290 legacy=0x7feb10008750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:53:09.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.098+0000 7feb23fff640 1 --2- 172.21.15.12:0/1873142170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb10003730 0x7feb10005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:09.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.098+0000 7feb23fff640 10 monclient(hunting): get_auth_request con 0x7feb10003730 auth_method 0 2024-09-11T01:53:09.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.098+0000 7feb23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:09.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.098+0000 7feb23fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:09.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.098+0000 7feb23fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:53:09.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.098+0000 7feb23fff640 1 -- 172.21.15.12:0/1873142170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb10003730 msgr2=0x7feb10005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:53:09.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.098+0000 7feb23fff640 1 -- 172.21.15.12:0/1873142170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb10003730 msgr2=0x7feb10005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:53:09.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.098+0000 7feb23fff640 1 --2- 172.21.15.12:0/1873142170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb10003730 0x7feb10005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:53:09.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.098+0000 7feb23fff640 1 --2- 172.21.15.12:0/1873142170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb10003730 0x7feb10005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:53:09.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.498+0000 7feb237fe640 1 -- 172.21.15.12:0/1873142170 >> v1:172.21.15.71:6789/0 conn(0x7feb10006290 legacy=0x7feb10008750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:53:09.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.499+0000 7feb23fff640 1 --2- 172.21.15.12:0/1873142170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb10003730 0x7feb10005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:09.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.499+0000 7feb23fff640 10 monclient(hunting): get_auth_request con 0x7feb10003730 auth_method 0 2024-09-11T01:53:09.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.499+0000 7feb23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:09.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.499+0000 7feb23fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:09.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.499+0000 7feb23fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:53:09.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.499+0000 7feb23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:09.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.499+0000 7feb23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:09.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.499+0000 7feb23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:09.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.499+0000 7feb23fff640 10 monclient(hunting): handle_auth_done global_id 244373 payload 274 2024-09-11T01:53:09.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.499+0000 7feb23fff640 1 -- 172.21.15.12:0/1873142170 >> v1:172.21.15.71:6789/0 conn(0x7feb10006290 legacy=0x7feb10008750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:53:09.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.499+0000 7feb23fff640 10 monclient: _finish_hunting 0 2024-09-11T01:53:09.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.499+0000 7feb23fff640 1 monclient: found mon.noname-b 2024-09-11T01:53:09.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.499+0000 7feb23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:09.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.499+0000 7feb23fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:53:09.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.499+0000 7feb23fff640 1 -- 172.21.15.12:0/1873142170 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb1000b0a0 con 0x7feb10003730 2024-09-11T01:53:09.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.502+0000 7feb23fff640 1 -- 172.21.15.12:0/1873142170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb10003730 msgr2=0x7feb10005b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:09.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.503+0000 7feb23fff640 1 --2- 172.21.15.12:0/1873142170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb10003730 0x7feb10005b60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7feb14016a60 tx=0x7feb1402ba90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e824c5f5ebf1db5e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:09.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.503+0000 7feb22ffd640 1 -- 172.21.15.12:0/1873142170 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb14035020 con 0x7feb10003730 2024-09-11T01:53:09.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.503+0000 7feb22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:09.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.503+0000 7feb22ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:53:09.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.503+0000 7feb22ffd640 10 monclient: dump: 2024-09-11T01:53:09.506 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:09.506 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:09.506 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:09.506 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:09.506 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:09.506 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:09.506 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:09.506 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:09.507 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.503+0000 7feb22ffd640 10 monclient: _finish_auth 0 2024-09-11T01:53:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.503+0000 7feb22ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:53:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.503+0000 7feb22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.503+0000 7feb2a9f8640 5 monclient: authenticate success, global_id 244373 2024-09-11T01:53:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.503+0000 7feb22ffd640 1 -- 172.21.15.12:0/1873142170 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb1402f910 con 0x7feb10003730 2024-09-11T01:53:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.503+0000 7feb2a9f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:53:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.503+0000 7feb22ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.503+0000 7feb2a9f8640 10 monclient: get_monmap_and_config success 2024-09-11T01:53:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.503+0000 7feb2a9f8640 4 set_mon_vals no callback set 2024-09-11T01:53:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.503+0000 7feb22ffd640 1 -- 172.21.15.12:0/1873142170 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb1400f040 con 0x7feb10003730 2024-09-11T01:53:09.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.503+0000 7feb22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:09.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.503+0000 7feb22ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:53:09.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.503+0000 7feb22ffd640 10 monclient: dump: 2024-09-11T01:53:09.508 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:09.508 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:09.508 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:09.508 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:09.508 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:09.508 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:09.508 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:09.508 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:09.508 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:09.508 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:09.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.504+0000 7feb2a9f8640 10 monclient: shutdown 2024-09-11T01:53:09.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.504+0000 7feb2a9f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:09.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.504+0000 7feb2a9f8640 1 -- 172.21.15.12:0/1873142170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb10003730 msgr2=0x7feb10005b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:09.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.504+0000 7feb2a9f8640 1 --2- 172.21.15.12:0/1873142170 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb10003730 0x7feb10005b60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7feb14016a60 tx=0x7feb1402ba90 comp rx=0 tx=0).stop 2024-09-11T01:53:09.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.504+0000 7feb28f95640 1 -- 172.21.15.12:0/1873142170 reap_dead start 2024-09-11T01:53:09.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.504+0000 7feb2a9f8640 1 -- 172.21.15.12:0/1873142170 shutdown_connections 2024-09-11T01:53:09.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.504+0000 7feb2a9f8640 1 -- 172.21.15.12:0/1873142170 >> 172.21.15.12:0/1873142170 conn(0x7feb240a7120 msgr2=0x7feb240a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:09.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.504+0000 7feb2a9f8640 1 -- 172.21.15.12:0/1873142170 shutdown_connections 2024-09-11T01:53:09.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.504+0000 7feb2a9f8640 1 -- 172.21.15.12:0/1873142170 wait complete. 2024-09-11T01:53:09.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.505+0000 7feb2a9f8640 10 monclient: build_initial_monmap 2024-09-11T01:53:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.505+0000 7feb2a9f8640 10 monclient: monmap: 2024-09-11T01:53:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:53:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:53:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:53:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:53:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:53:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:53:09.514 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:09.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.505+0000 7feb2a9f8640 1 Processor -- start 2024-09-11T01:53:09.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb2a9f8640 1 -- start start 2024-09-11T01:53:09.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb2a9f8640 10 monclient: init 2024-09-11T01:53:09.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb2a9f8640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:09.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb2a9f8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:53:09.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb2a9f8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb10003730 0x7feb241ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:09.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb2a9f8640 10 monclient(hunting): picked mon.noname-a con 0x7feb10003730 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:53:09.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb2a9f8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb24136f90 0x7feb24137410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:09.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb2a9f8640 10 monclient(hunting): picked mon.noname-c con 0x7feb24136f90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:53:09.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb2a9f8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb24120c70 0x7feb241210e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:09.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb2a9f8640 10 monclient(hunting): picked mon.noname-b con 0x7feb24120c70 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:53:09.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb2a9f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:09.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb2a9f8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7feb2406f970 con 0x7feb10003730 2024-09-11T01:53:09.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb2a9f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb2a9f8640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7feb2406f7f0 con 0x7feb24120c70 2024-09-11T01:53:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb2a9f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb2a9f8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7feb240a5c70 con 0x7feb24136f90 2024-09-11T01:53:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb2a9f8640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb2a9f8640 10 monclient(hunting): authenticate will time out at 2195.948692s 2024-09-11T01:53:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb23fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb10003730 0x7feb241ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb237fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb24136f90 0x7feb24137410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb23fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb10003730 0x7feb241ccfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59348/0 (socket says 172.21.15.12:59348) 2024-09-11T01:53:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb23fff640 1 -- 172.21.15.12:0/1172255209 learned_addr learned my addr 172.21.15.12:0/1172255209 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:53:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb23fff640 10 monclient(hunting): get_auth_request con 0x7feb10003730 auth_method 0 2024-09-11T01:53:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb23fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:09.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb237fe640 10 monclient(hunting): get_auth_request con 0x7feb24136f90 auth_method 0 2024-09-11T01:53:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb237fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb28f95640 1 -- 172.21.15.12:0/1172255209 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb24120c70 msgr2=0x7feb241210e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:53:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb28f95640 1 --2- 172.21.15.12:0/1172255209 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb24120c70 0x7feb241210e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:53:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb20ff9640 1 -- 172.21.15.12:0/1172255209 DispatchQueue::entry inject delay of 0.002000 2024-09-11T01:53:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.506+0000 7feb237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:09.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.507+0000 7feb237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:09.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.507+0000 7feb23fff640 10 monclient(hunting): handle_auth_done global_id 264177 payload 824 2024-09-11T01:53:09.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.507+0000 7feb23fff640 1 -- 172.21.15.12:0/1172255209 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb24136f90 msgr2=0x7feb24137410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:09.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.507+0000 7feb23fff640 1 --2- 172.21.15.12:0/1172255209 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb24136f90 0x7feb24137410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:09.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.507+0000 7feb23fff640 1 -- 172.21.15.12:0/1172255209 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb24120c70 msgr2=0x7feb241210e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:53:09.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.507+0000 7feb23fff640 1 --2- 172.21.15.12:0/1172255209 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb24120c70 0x7feb241210e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:09.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.507+0000 7feb23fff640 10 monclient: _finish_hunting 0 2024-09-11T01:53:09.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.507+0000 7feb23fff640 1 monclient: found mon.noname-a 2024-09-11T01:53:09.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.507+0000 7feb23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:09.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.507+0000 7feb23fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:53:09.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.507+0000 7feb23fff640 1 -- 172.21.15.12:0/1172255209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb241225f0 con 0x7feb10003730 2024-09-11T01:53:09.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.507+0000 7feb237fe640 1 --2- 172.21.15.12:0/1172255209 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb24136f90 0x7feb24137410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:53:09.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.509+0000 7feb23fff640 1 -- 172.21.15.12:0/1172255209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb10003730 msgr2=0x7feb241ccfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:09.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.509+0000 7feb23fff640 1 --2- 172.21.15.12:0/1172255209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb10003730 0x7feb241ccfe0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7feb14016a60 tx=0x7feb140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:09.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.509+0000 7feb20ff9640 1 -- 172.21.15.12:0/1172255209 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb14048070 con 0x7feb10003730 2024-09-11T01:53:09.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.509+0000 7feb20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:09.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.509+0000 7feb20ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:53:09.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.509+0000 7feb20ff9640 10 monclient: dump: 2024-09-11T01:53:09.519 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:09.519 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:09.519 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:09.519 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:09.519 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:09.519 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:09.519 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:09.519 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:09.520 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:09.520 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:09.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.509+0000 7feb20ff9640 10 monclient: _finish_auth 0 2024-09-11T01:53:09.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.509+0000 7feb20ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:53:09.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.509+0000 7feb20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:09.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.509+0000 7feb2a9f8640 5 monclient: authenticate success, global_id 264177 2024-09-11T01:53:09.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.509+0000 7feb20ff9640 1 -- 172.21.15.12:0/1172255209 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb1403a630 con 0x7feb10003730 2024-09-11T01:53:09.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.509+0000 7feb2a9f8640 10 monclient: _renew_subs 2024-09-11T01:53:09.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.509+0000 7feb2a9f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:09.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.509+0000 7feb2a9f8640 1 -- 172.21.15.12:0/1172255209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb24137950 con 0x7feb10003730 2024-09-11T01:53:09.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.510+0000 7feb20ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:09.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.510+0000 7feb20ff9640 1 -- 172.21.15.12:0/1172255209 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb14035040 con 0x7feb10003730 2024-09-11T01:53:09.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.510+0000 7feb20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:09.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.510+0000 7feb20ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:53:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.510+0000 7feb20ff9640 10 monclient: dump: 2024-09-11T01:53:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:09.521 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.510+0000 7feb2a9f8640 10 monclient: _renew_subs 2024-09-11T01:53:09.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.510+0000 7feb227fc640 4 set_mon_vals no callback set 2024-09-11T01:53:09.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.510+0000 7feb2a9f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:09.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.510+0000 7feb2a9f8640 1 -- 172.21.15.12:0/1172255209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb24124950 con 0x7feb10003730 2024-09-11T01:53:09.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.511+0000 7feb20ff9640 1 -- 172.21.15.12:0/1172255209 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7feb14015040 con 0x7feb10003730 2024-09-11T01:53:09.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.511+0000 7feb2a9f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:09.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.511+0000 7feb2a9f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:09.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.511+0000 7feb2a9f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:09.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.511+0000 7feb2a9f8640 1 -- 172.21.15.12:0/1172255209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb24138610 con 0x7feb10003730 2024-09-11T01:53:09.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.511+0000 7feb20ff9640 1 --2- 172.21.15.12:0/1172255209 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7feb1003f0a0 0x7feb10041560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:09.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.511+0000 7feb20ff9640 1 -- 172.21.15.12:0/1172255209 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(804..804 src has 676..804) ==== 4739+0+0 (secure 0 0 0) 0x7feb140744e0 con 0x7feb10003730 2024-09-11T01:53:09.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.511+0000 7feb237fe640 1 --2- 172.21.15.12:0/1172255209 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7feb1003f0a0 0x7feb10041560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:09.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.511+0000 7feb237fe640 10 monclient: get_auth_request con 0x7feb1003f0a0 auth_method 0 2024-09-11T01:53:09.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.514+0000 7feb237fe640 1 --2- 172.21.15.12:0/1172255209 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7feb1003f0a0 0x7feb10041560 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7feb18003840 tx=0x7feb18009cf0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:09.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.516+0000 7feb20ff9640 1 -- 172.21.15.12:0/1172255209 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7feb1402f8d0 con 0x7feb10003730 2024-09-11T01:53:09.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.516+0000 7feb20ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:09.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.516+0000 7feb20ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:53:09.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.779+0000 7feb2a9f8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:53:09.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.779+0000 7feb2a9f8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:53:09.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.779+0000 7feb2a9f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:09.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.779+0000 7feb2a9f8640 1 -- 172.21.15.12:0/1172255209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7feb241cd520 con 0x7feb10003730 2024-09-11T01:53:09.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.780+0000 7feb20ff9640 1 -- 172.21.15.12:0/1172255209 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7feb14072f00 con 0x7feb10003730 2024-09-11T01:53:09.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.780+0000 7feb20ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:53:09.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.780+0000 7feb20ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:53:09.782 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":224,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:53:09.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.782+0000 7feb2a9f8640 1 -- 172.21.15.12:0/1172255209 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7feb1003f0a0 msgr2=0x7feb10041560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:09.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.782+0000 7feb2a9f8640 1 --2- 172.21.15.12:0/1172255209 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7feb1003f0a0 0x7feb10041560 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7feb18003840 tx=0x7feb18009cf0 comp rx=0 tx=0).stop 2024-09-11T01:53:09.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.782+0000 7feb2a9f8640 10 monclient: shutdown 2024-09-11T01:53:09.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.782+0000 7feb2a9f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:09.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.782+0000 7feb2a9f8640 1 -- 172.21.15.12:0/1172255209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb10003730 msgr2=0x7feb241ccfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:09.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.782+0000 7feb2a9f8640 1 --2- 172.21.15.12:0/1172255209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb10003730 0x7feb241ccfe0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7feb14016a60 tx=0x7feb140046c0 comp rx=0 tx=0).stop 2024-09-11T01:53:09.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.782+0000 7feb2a9f8640 1 -- 172.21.15.12:0/1172255209 shutdown_connections 2024-09-11T01:53:09.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.782+0000 7feb2a9f8640 1 --2- 172.21.15.12:0/1172255209 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb24120c70 0x7feb241210e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:09.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.782+0000 7feb2a9f8640 1 --2- 172.21.15.12:0/1172255209 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7feb24136f90 0x7feb24137410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:09.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.782+0000 7feb2a9f8640 1 --2- 172.21.15.12:0/1172255209 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7feb1003f0a0 0x7feb10041560 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:09.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.782+0000 7feb2a9f8640 1 --2- 172.21.15.12:0/1172255209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb10003730 0x7feb241ccfe0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:09.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.782+0000 7feb2a9f8640 1 -- 172.21.15.12:0/1172255209 >> 172.21.15.12:0/1172255209 conn(0x7feb240a7120 msgr2=0x7feb240a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:09.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.782+0000 7feb2a9f8640 1 -- 172.21.15.12:0/1172255209 shutdown_connections 2024-09-11T01:53:09.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.782+0000 7feb2a9f8640 1 -- 172.21.15.12:0/1172255209 wait complete. 2024-09-11T01:53:09.795 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:53:09.795 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-11T01:53:09.796 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:53:09.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.896+0000 7f06cb100640 10 monclient: get_monmap_and_config 2024-09-11T01:53:09.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.896+0000 7f06cb100640 10 monclient: build_initial_monmap 2024-09-11T01:53:09.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.896+0000 7f06cb100640 10 monclient: monmap: 2024-09-11T01:53:09.897 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:53:09.896844+0000 2024-09-11T01:53:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:53:09.896844+0000 2024-09-11T01:53:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:53:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:53:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:53:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:53:09.898 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:53:09.898 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:09.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.896+0000 7f06cb100640 1 Processor -- start 2024-09-11T01:53:09.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.896+0000 7f06cb100640 1 -- start start 2024-09-11T01:53:09.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.896+0000 7f06cb100640 10 monclient: init 2024-09-11T01:53:09.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06cb100640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:09.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06cb100640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-11T01:53:09.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06cb100640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f06c4137e50 addr v1:172.21.15.71:6789/0 2024-09-11T01:53:09.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06cb100640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f06c4138230 0x7f06c4138640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:09.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06cb100640 10 monclient(hunting): picked mon.noname-c con 0x7f06c4138230 addr v2:172.21.15.71:3300/0 2024-09-11T01:53:09.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06cb100640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f06c413d380 0x7f06c413f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:09.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06cb100640 10 monclient(hunting): picked mon.noname-a con 0x7f06c413d380 addr v2:172.21.15.12:3300/0 2024-09-11T01:53:09.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06cb100640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06c40a62d0 con 0x7f06c4137e50 2024-09-11T01:53:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06cb100640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06cb100640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f06c409bb40 con 0x7f06c413d380 2024-09-11T01:53:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06cb100640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06cb100640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f06c409c4a0 con 0x7f06c4138230 2024-09-11T01:53:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06cb100640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06cb100640 10 monclient(hunting): authenticate will time out at 2196.339546s 2024-09-11T01:53:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06c8e9c640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f06c4137e50 legacy=0x7f06c409d160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:53:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06c3fff640 1 -- >> v2:172.21.15.71:3300/0 conn(0x7f06c4138230 msgr2=0x7f06c4138640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:53:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06c3fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f06c4138230 0x7f06c4138640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:53:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06c969d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f06c413d380 0x7f06c413f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06c969d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f06c413d380 0x7f06c413f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59352/0 (socket says 172.21.15.12:59352) 2024-09-11T01:53:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06c969d640 1 -- 172.21.15.12:0/2513725428 learned_addr learned my addr 172.21.15.12:0/2513725428 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:53:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06c969d640 10 monclient(hunting): get_auth_request con 0x7f06c413d380 auth_method 0 2024-09-11T01:53:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06c969d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06c969d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06c969d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06c969d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06c969d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06c969d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06c969d640 10 monclient(hunting): handle_auth_done global_id 264180 payload 274 2024-09-11T01:53:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06c969d640 1 -- 172.21.15.12:0/2513725428 >> v2:172.21.15.71:3300/0 conn(0x7f06c4138230 msgr2=0x7f06c4138640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:53:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06c969d640 1 --2- 172.21.15.12:0/2513725428 >> v2:172.21.15.71:3300/0 conn(0x7f06c4138230 0x7f06c4138640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06c969d640 1 -- 172.21.15.12:0/2513725428 >> v1:172.21.15.71:6789/0 conn(0x7f06c4137e50 legacy=0x7f06c409d160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:53:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06c969d640 10 monclient: _finish_hunting 0 2024-09-11T01:53:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06c969d640 1 monclient: found mon.noname-a 2024-09-11T01:53:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06c969d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06c969d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:53:09.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.897+0000 7f06c969d640 1 -- 172.21.15.12:0/2513725428 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06c409c780 con 0x7f06c413d380 2024-09-11T01:53:09.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.900+0000 7f06c969d640 1 -- 172.21.15.12:0/2513725428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c413d380 msgr2=0x7f06c413f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:09.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.900+0000 7f06c969d640 1 --2- 172.21.15.12:0/2513725428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c413d380 0x7f06c413f770 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f06c40a6f80 tx=0x7f06bc02a580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e816fd535a4a7331 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:09.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.900+0000 7f06c37fe640 1 -- 172.21.15.12:0/2513725428 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06bc036070 con 0x7f06c413d380 2024-09-11T01:53:09.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.901+0000 7f06c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:09.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.901+0000 7f06c37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:53:09.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.901+0000 7f06c37fe640 10 monclient: dump: 2024-09-11T01:53:09.910 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:09.910 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:09.910 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:09.910 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:09.910 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:09.910 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:09.910 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:09.910 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:09.911 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:09.911 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:09.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.901+0000 7f06c37fe640 10 monclient: _finish_auth 0 2024-09-11T01:53:09.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.901+0000 7f06c37fe640 10 monclient: _check_auth_tickets 2024-09-11T01:53:09.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.901+0000 7f06c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:09.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.901+0000 7f06c37fe640 1 -- 172.21.15.12:0/2513725428 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06bc007800 con 0x7f06c413d380 2024-09-11T01:53:09.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.901+0000 7f06cb100640 5 monclient: authenticate success, global_id 264180 2024-09-11T01:53:09.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.901+0000 7f06cb100640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:53:09.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.901+0000 7f06c37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:09.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.901+0000 7f06c37fe640 1 -- 172.21.15.12:0/2513725428 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06bc007bd0 con 0x7f06c413d380 2024-09-11T01:53:09.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.901+0000 7f06cb100640 10 monclient: get_monmap_and_config success 2024-09-11T01:53:09.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.901+0000 7f06cb100640 4 set_mon_vals no callback set 2024-09-11T01:53:09.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.901+0000 7f06c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:09.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.901+0000 7f06c37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:53:09.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.901+0000 7f06c37fe640 10 monclient: dump: 2024-09-11T01:53:09.912 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:09.912 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:09.912 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:09.912 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:09.912 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:09.912 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:09.912 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:09.912 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:09.912 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:09.912 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:09.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.901+0000 7f06cb100640 10 monclient: shutdown 2024-09-11T01:53:09.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.901+0000 7f06cb100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:09.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.901+0000 7f06cb100640 1 -- 172.21.15.12:0/2513725428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c413d380 msgr2=0x7f06c413f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:09.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.901+0000 7f06cb100640 1 --2- 172.21.15.12:0/2513725428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c413d380 0x7f06c413f770 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f06c40a6f80 tx=0x7f06bc02a580 comp rx=0 tx=0).stop 2024-09-11T01:53:09.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.901+0000 7f06cb100640 1 -- 172.21.15.12:0/2513725428 shutdown_connections 2024-09-11T01:53:09.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.901+0000 7f06cb100640 1 --2- 172.21.15.12:0/2513725428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c413d380 0x7f06c413f770 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:09.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.901+0000 7f06cb100640 1 --2- 172.21.15.12:0/2513725428 >> v2:172.21.15.71:3300/0 conn(0x7f06c4138230 0x7f06c4138640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:09.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.901+0000 7f06cb100640 1 -- 172.21.15.12:0/2513725428 >> 172.21.15.12:0/2513725428 conn(0x7f06c40a7020 msgr2=0x7f06c40a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:09.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.901+0000 7f06cb100640 1 -- 172.21.15.12:0/2513725428 shutdown_connections 2024-09-11T01:53:09.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.901+0000 7f06cb100640 1 -- 172.21.15.12:0/2513725428 wait complete. 2024-09-11T01:53:09.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.902+0000 7f06cb100640 10 monclient: build_initial_monmap 2024-09-11T01:53:09.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.902+0000 7f06cb100640 10 monclient: monmap: 2024-09-11T01:53:09.913 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:09.913 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:53:09.913 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:53:09.914 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:53:09.914 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:09.914 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:09.914 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:53:09.914 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:53:09.914 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:53:09.914 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:09.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.902+0000 7f06cb100640 1 Processor -- start 2024-09-11T01:53:09.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.902+0000 7f06cb100640 1 -- start start 2024-09-11T01:53:09.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.902+0000 7f06cb100640 10 monclient: init 2024-09-11T01:53:09.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06cb100640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:09.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06cb100640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:53:09.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06cb100640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f06c4137e50 0x7f06c41ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:09.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06cb100640 10 monclient(hunting): picked mon.noname-b con 0x7f06c4137e50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:53:09.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06cb100640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c4138230 0x7f06c4137060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:09.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06cb100640 10 monclient(hunting): picked mon.noname-a con 0x7f06c4138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:53:09.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06cb100640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f06c413d380 0x7f06c40a1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:09.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06cb100640 10 monclient(hunting): picked mon.noname-c con 0x7f06c413d380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:53:09.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06cb100640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:09.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06cb100640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f06c406f890 con 0x7f06c4138230 2024-09-11T01:53:09.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06cb100640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:09.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06cb100640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f06c406f710 con 0x7f06c4137e50 2024-09-11T01:53:09.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06cb100640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:09.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06cb100640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f06c413fcb0 con 0x7f06c413d380 2024-09-11T01:53:09.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06cb100640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:09.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06cb100640 10 monclient(hunting): authenticate will time out at 2196.345301s 2024-09-11T01:53:09.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c4138230 0x7f06c4137060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:09.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c8e9c640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f06c4137e50 msgr2=0x7f06c41ccff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:53:09.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c8e9c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f06c4137e50 0x7f06c41ccff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:53:09.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c4138230 0x7f06c4137060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59362/0 (socket says 172.21.15.12:59362) 2024-09-11T01:53:09.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c3fff640 1 -- 172.21.15.12:0/811178833 learned_addr learned my addr 172.21.15.12:0/811178833 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:53:09.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c3fff640 10 monclient(hunting): get_auth_request con 0x7f06c4138230 auth_method 0 2024-09-11T01:53:09.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:09.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:09.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:09.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c969d640 1 --2- 172.21.15.12:0/811178833 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f06c413d380 0x7f06c40a1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:09.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c969d640 10 monclient(hunting): get_auth_request con 0x7f06c413d380 auth_method 0 2024-09-11T01:53:09.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c969d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:09.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c969d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:09.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c969d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:09.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:09.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:09.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:09.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c969d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:09.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c969d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:09.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c969d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:09.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c3fff640 10 monclient(hunting): handle_auth_done global_id 264183 payload 824 2024-09-11T01:53:09.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c3fff640 1 -- 172.21.15.12:0/811178833 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f06c413d380 msgr2=0x7f06c40a1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:09.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c3fff640 1 --2- 172.21.15.12:0/811178833 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f06c413d380 0x7f06c40a1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:09.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c3fff640 1 -- 172.21.15.12:0/811178833 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f06c4137e50 msgr2=0x7f06c41ccff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:53:09.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c3fff640 1 --2- 172.21.15.12:0/811178833 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f06c4137e50 0x7f06c41ccff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:09.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c3fff640 10 monclient: _finish_hunting 0 2024-09-11T01:53:09.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c3fff640 1 monclient: found mon.noname-a 2024-09-11T01:53:09.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:09.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:53:09.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c3fff640 1 -- 172.21.15.12:0/811178833 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06c40a3640 con 0x7f06c4138230 2024-09-11T01:53:09.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.903+0000 7f06c969d640 1 --2- 172.21.15.12:0/811178833 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f06c413d380 0x7f06c40a1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:53:09.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.905+0000 7f06c3fff640 1 -- 172.21.15.12:0/811178833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c4138230 msgr2=0x7f06c4137060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:09.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.905+0000 7f06c3fff640 1 --2- 172.21.15.12:0/811178833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c4138230 0x7f06c4137060 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f06b800b580 tx=0x7f06b800ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:09.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.906+0000 7f06c17fa640 1 -- 172.21.15.12:0/811178833 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b80050f0 con 0x7f06c4138230 2024-09-11T01:53:09.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.906+0000 7f06c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:09.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.906+0000 7f06c17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:53:09.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.906+0000 7f06c17fa640 10 monclient: dump: 2024-09-11T01:53:09.918 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:09.918 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:09.919 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:09.919 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:09.919 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:09.919 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:09.919 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:09.919 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:09.919 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:09.919 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:09.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.906+0000 7f06c17fa640 10 monclient: _finish_auth 0 2024-09-11T01:53:09.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.906+0000 7f06c17fa640 10 monclient: _check_auth_tickets 2024-09-11T01:53:09.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.906+0000 7f06c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:09.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.906+0000 7f06cb100640 5 monclient: authenticate success, global_id 264183 2024-09-11T01:53:09.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.906+0000 7f06c17fa640 1 -- 172.21.15.12:0/811178833 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06b8015070 con 0x7f06c4138230 2024-09-11T01:53:09.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.906+0000 7f06cb100640 10 monclient: _renew_subs 2024-09-11T01:53:09.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.906+0000 7f06cb100640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:09.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.906+0000 7f06cb100640 1 -- 172.21.15.12:0/811178833 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06c40a41a0 con 0x7f06c4138230 2024-09-11T01:53:09.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.906+0000 7f06c17fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:09.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.906+0000 7f06c17fa640 1 -- 172.21.15.12:0/811178833 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b80054f0 con 0x7f06c4138230 2024-09-11T01:53:09.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.906+0000 7f06c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:09.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.906+0000 7f06c17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:53:09.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.906+0000 7f06c17fa640 10 monclient: dump: 2024-09-11T01:53:09.920 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:09.920 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:09.920 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:09.920 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:09.920 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:09.920 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:09.921 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:09.921 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:09.921 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:09.921 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:09.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.906+0000 7f06cb100640 10 monclient: _renew_subs 2024-09-11T01:53:09.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.906+0000 7f06cb100640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:09.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.906+0000 7f06c2ffd640 4 set_mon_vals no callback set 2024-09-11T01:53:09.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.906+0000 7f06cb100640 1 -- 172.21.15.12:0/811178833 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06c4120b70 con 0x7f06c4138230 2024-09-11T01:53:09.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.907+0000 7f06c17fa640 1 -- 172.21.15.12:0/811178833 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f06b8010480 con 0x7f06c4138230 2024-09-11T01:53:09.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.907+0000 7f06cb100640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:09.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.907+0000 7f06cb100640 1 --2- 172.21.15.12:0/811178833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c40a20c0 0x7f06c40a24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:09.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.907+0000 7f06cb100640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:09.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.907+0000 7f06cb100640 1 -- 172.21.15.12:0/811178833 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f06c4121970 con 0x7f06c40a20c0 2024-09-11T01:53:09.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.907+0000 7f06c8e9c640 1 --2- 172.21.15.12:0/811178833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c40a20c0 0x7f06c40a24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:09.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.907+0000 7f06c8e9c640 10 monclient: get_auth_request con 0x7f06c40a20c0 auth_method 0 2024-09-11T01:53:09.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.907+0000 7f06c8e9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:09.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.907+0000 7f06c8e9c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:09.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.907+0000 7f06c8e9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:09.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.907+0000 7f06c17fa640 1 --2- 172.21.15.12:0/811178833 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f06b003e1c0 0x7f06b0040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:09.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.907+0000 7f06c8e9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:09.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.907+0000 7f06c8e9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:09.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.907+0000 7f06c8e9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:09.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.907+0000 7f06c17fa640 1 -- 172.21.15.12:0/811178833 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(804..804 src has 676..804) ==== 4739+0+0 (secure 0 0 0) 0x7f06b8052ae0 con 0x7f06c4138230 2024-09-11T01:53:09.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.907+0000 7f06c8e9c640 10 monclient(hunting): handle_auth_done global_id 264186 payload 824 2024-09-11T01:53:09.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.907+0000 7f06c8e9c640 1 -- 172.21.15.12:0/811178833 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f06c413f960 con 0x7f06c40a20c0 2024-09-11T01:53:09.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.907+0000 7f06c969d640 1 --2- 172.21.15.12:0/811178833 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f06b003e1c0 0x7f06b0040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:09.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.907+0000 7f06c969d640 10 monclient: get_auth_request con 0x7f06b003e1c0 auth_method 0 2024-09-11T01:53:09.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.909+0000 7f06c8e9c640 1 -- 172.21.15.12:0/811178833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c40a20c0 msgr2=0x7f06c40a24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:09.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.909+0000 7f06c8e9c640 1 --2- 172.21.15.12:0/811178833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c40a20c0 0x7f06c40a24a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f06b4003930 tx=0x7f06b400b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:09.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.909+0000 7f06c17fa640 1 -- 172.21.15.12:0/811178833 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b4005270 con 0x7f06c40a20c0 2024-09-11T01:53:09.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.909+0000 7f06c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:09.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.909+0000 7f06c17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:53:09.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.909+0000 7f06c17fa640 10 monclient: dump: 2024-09-11T01:53:09.926 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:09.926 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:09.926 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:09.926 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:09.926 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:09.926 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:09.926 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:09.926 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:09.926 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:09.926 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:09.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.910+0000 7f06c17fa640 1 -- 172.21.15.12:0/811178833 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f06b4015070 con 0x7f06c40a20c0 2024-09-11T01:53:09.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.910+0000 7f06c17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:09.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.910+0000 7f06c17fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:53:09.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.910+0000 7f06c17fa640 1 -- 172.21.15.12:0/811178833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c40a20c0 msgr2=0x7f06c40a24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:09.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.910+0000 7f06c17fa640 1 --2- 172.21.15.12:0/811178833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c40a20c0 0x7f06c40a24a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f06b4003930 tx=0x7f06b400b8c0 comp rx=0 tx=0).stop 2024-09-11T01:53:09.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.910+0000 7f06c17fa640 1 -- 172.21.15.12:0/811178833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c40a20c0 msgr2=0x7f06c40a24a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:53:09.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.910+0000 7f06c969d640 1 --2- 172.21.15.12:0/811178833 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f06b003e1c0 0x7f06b0040680 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f06bc030000 tx=0x7f06bc009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:09.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.910+0000 7f06c17fa640 1 --2- 172.21.15.12:0/811178833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c40a20c0 0x7f06c40a24a0 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f06b4003930 tx=0x7f06b400b8c0 comp rx=0 tx=0).stop 2024-09-11T01:53:09.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.916+0000 7f06cb100640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:53:09.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.916+0000 7f06cb100640 1 --2- 172.21.15.12:0/811178833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c406e2e0 0x7f06c406e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:09.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.916+0000 7f06cb100640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:09.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.916+0000 7f06cb100640 1 -- 172.21.15.12:0/811178833 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f06c41213b0 con 0x7f06c406e2e0 2024-09-11T01:53:09.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.916+0000 7f06c8e9c640 1 --2- 172.21.15.12:0/811178833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c406e2e0 0x7f06c406e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:09.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.916+0000 7f06c8e9c640 10 monclient: get_auth_request con 0x7f06c406e2e0 auth_method 0 2024-09-11T01:53:09.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.916+0000 7f06c8e9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:09.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.916+0000 7f06c8e9c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:09.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.916+0000 7f06c8e9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:09.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.916+0000 7f06c8e9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:09.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.916+0000 7f06c8e9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:09.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.916+0000 7f06c8e9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:09.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.916+0000 7f06c8e9c640 10 monclient(hunting): handle_auth_done global_id 264189 payload 824 2024-09-11T01:53:09.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.916+0000 7f06c8e9c640 1 -- 172.21.15.12:0/811178833 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f06c4138640 con 0x7f06c406e2e0 2024-09-11T01:53:09.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.918+0000 7f06c8e9c640 1 -- 172.21.15.12:0/811178833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c406e2e0 msgr2=0x7f06c406e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:09.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.918+0000 7f06c8e9c640 1 --2- 172.21.15.12:0/811178833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c406e2e0 0x7f06c406e6f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f06b4003930 tx=0x7f06b4005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:09.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.919+0000 7f06c17fa640 1 -- 172.21.15.12:0/811178833 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b4010610 con 0x7f06c406e2e0 2024-09-11T01:53:09.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.919+0000 7f06c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:09.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.919+0000 7f06c17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:53:09.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.919+0000 7f06c17fa640 10 monclient: dump: 2024-09-11T01:53:09.928 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:09.928 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:09.929 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:09.929 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:09.929 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:09.929 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:09.929 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:09.929 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:09.929 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:09.929 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:09.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.919+0000 7f06c17fa640 1 -- 172.21.15.12:0/811178833 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4699 (secure 0 0 0) 0x7f06b4010610 con 0x7f06c406e2e0 2024-09-11T01:53:09.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.919+0000 7f06c17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:53:09.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.919+0000 7f06c17fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:53:09.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.919+0000 7f06c17fa640 1 -- 172.21.15.12:0/811178833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c406e2e0 msgr2=0x7f06c406e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:09.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.919+0000 7f06c17fa640 1 --2- 172.21.15.12:0/811178833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c406e2e0 0x7f06c406e6f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f06b4003930 tx=0x7f06b4005a20 comp rx=0 tx=0).stop 2024-09-11T01:53:09.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.919+0000 7f06c17fa640 1 -- 172.21.15.12:0/811178833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c406e2e0 msgr2=0x7f06c406e6f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:53:09.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.919+0000 7f06c17fa640 1 --2- 172.21.15.12:0/811178833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c406e2e0 0x7f06c406e6f0 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f06b4003930 tx=0x7f06b4005a20 comp rx=0 tx=0).stop 2024-09-11T01:53:09.930 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:53:09.930 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:53:09.930 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:53:09.930 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:53:09.930 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 224, 2024-09-11T01:53:09.930 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 31625, 2024-09-11T01:53:09.930 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:53:09.930 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:53:09.931 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:53:09.931 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:09.931 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:53:09.931 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:53:09.931 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:53:09.931 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:53:09.931 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:53:09.931 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:53:09.931 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:53:09.931 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:53:09.931 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:53:09.931 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:53:09.931 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:53:09.932 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:53:09.932 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:53:09.932 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:53:09.932 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:53:09.932 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:09.932 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:53:09.932 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:53:09.932 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:53:09.932 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:53:09.932 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:53:09.932 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:53:09.932 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:53:09.932 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:53:09.933 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:53:09.933 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:53:09.933 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:53:09.933 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:53:09.933 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:53:09.933 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:09.933 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:09.933 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:53:09.933 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:53:09.933 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:53:09.933 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:53:09.933 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:53:09.933 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:53:09.934 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:53:09.934 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:53:09.934 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:53:09.934 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:53:09.934 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:53:09.934 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:53:09.934 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:53:09.934 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:53:09.934 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:53:09.934 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:53:09.934 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:53:09.934 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:53:09.934 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:53:09.934 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:53:09.935 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:53:09.935 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:53:09.935 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:53:09.935 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:53:09.935 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:53:09.935 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:53:09.935 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:53:09.935 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:53:09.935 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:09.935 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:53:09.935 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:09.935 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:53:09.935 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:09.935 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:53:09.936 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:53:09.936 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:53:09.936 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:53:09.936 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:09.936 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:53:09.936 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:53:09.936 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:09.936 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:09.936 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:09.936 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:53:09.936 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:53:09.936 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:09.936 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:09.937 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:09.937 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:09.937 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:53:09.937 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:53:09.937 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:53:09.937 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:53:09.937 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:53:09.937 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:09.937 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:09.937 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:53:09.937 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:53:09.937 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:53:09.937 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:53:09.937 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:09.938 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:53:09.938 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:53:09.938 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:09.938 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:09.938 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:09.938 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:53:09.938 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:53:09.938 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:09.938 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:09.938 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:09.938 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:09.938 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:53:09.938 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:53:09.939 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:53:09.939 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:53:09.939 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:53:09.939 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:09.939 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:09.939 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:53:09.939 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:53:09.939 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:53:09.939 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:53:09.939 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:09.939 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:53:09.939 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:53:09.939 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:09.939 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:09.940 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:09.940 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:53:09.940 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:53:09.940 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:09.940 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:09.940 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:09.940 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:09.940 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:53:09.940 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:53:09.940 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:53:09.940 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:53:09.940 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:53:09.940 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:09.940 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:09.941 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:09.941 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:53:09.941 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:53:09.941 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:09.941 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:53:09.941 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:53:09.941 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:53:09.941 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:09.941 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:09.941 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:53:09.941 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:09.941 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:53:09.941 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:53:09.941 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:53:09.942 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:09.942 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:09.942 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:53:09.942 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:09.942 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:53:09.942 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:53:09.942 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:53:09.942 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:09.942 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:09.942 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:09.942 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:53:09.942 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:53:09.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.919+0000 7f06cb100640 1 -- 172.21.15.12:0/811178833 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f06b003e1c0 msgr2=0x7f06b0040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:09.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.919+0000 7f06cb100640 1 --2- 172.21.15.12:0/811178833 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f06b003e1c0 0x7f06b0040680 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f06bc030000 tx=0x7f06bc009000 comp rx=0 tx=0).stop 2024-09-11T01:53:09.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.919+0000 7f06cb100640 10 monclient: shutdown 2024-09-11T01:53:09.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.919+0000 7f06cb100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:09.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.919+0000 7f06cb100640 1 -- 172.21.15.12:0/811178833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c4138230 msgr2=0x7f06c4137060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:09.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.919+0000 7f06c969d640 1 -- 172.21.15.12:0/811178833 reap_dead start 2024-09-11T01:53:09.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.919+0000 7f06cb100640 1 --2- 172.21.15.12:0/811178833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c4138230 0x7f06c4137060 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f06b800b580 tx=0x7f06b800ba40 comp rx=0 tx=0).stop 2024-09-11T01:53:09.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.919+0000 7f06cb100640 1 -- 172.21.15.12:0/811178833 shutdown_connections 2024-09-11T01:53:09.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.919+0000 7f06cb100640 1 --2- 172.21.15.12:0/811178833 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c4138230 0x7f06c4137060 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:09.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.919+0000 7f06cb100640 1 -- 172.21.15.12:0/811178833 >> 172.21.15.12:0/811178833 conn(0x7f06c40a7020 msgr2=0x7f06c413e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:09.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.919+0000 7f06cb100640 1 -- 172.21.15.12:0/811178833 shutdown_connections 2024-09-11T01:53:09.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:09.919+0000 7f06cb100640 1 -- 172.21.15.12:0/811178833 wait complete. 2024-09-11T01:53:09.945 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:53:09.945 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T01:53:09.945 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:53:10.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.046+0000 7f28df9f3640 10 monclient: get_monmap_and_config 2024-09-11T01:53:10.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.046+0000 7f28df9f3640 10 monclient: build_initial_monmap 2024-09-11T01:53:10.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.046+0000 7f28df9f3640 10 monclient: monmap: 2024-09-11T01:53:10.048 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:10.048 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:10.048 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:53:10.047123+0000 2024-09-11T01:53:10.048 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:53:10.047123+0000 2024-09-11T01:53:10.048 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:10.048 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:10.048 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:53:10.048 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:53:10.049 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:53:10.049 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:53:10.049 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:53:10.049 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:10.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.046+0000 7f28df9f3640 1 Processor -- start 2024-09-11T01:53:10.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28df9f3640 1 -- start start 2024-09-11T01:53:10.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28df9f3640 10 monclient: init 2024-09-11T01:53:10.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28df9f3640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:10.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28df9f3640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-11T01:53:10.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28df9f3640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f28d80a08c0 0x7f28d8137b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:10.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28df9f3640 10 monclient(hunting): picked mon.noname-a con 0x7f28d80a08c0 addr v2:172.21.15.12:3300/0 2024-09-11T01:53:10.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28df9f3640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d8073ef0 0x7f28d8074310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:10.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28df9f3640 10 monclient(hunting): picked mon.noname-b con 0x7f28d8073ef0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:53:10.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28df9f3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f28d80a5690 addr v1:172.21.15.12:6789/0 2024-09-11T01:53:10.054 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28df9f3640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28d8138ab0 con 0x7f28d80a5690 2024-09-11T01:53:10.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28df9f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:10.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28df9f3640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f28d80a0680 con 0x7f28d80a08c0 2024-09-11T01:53:10.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28df9f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:10.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28df9f3640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f28d80a0ca0 con 0x7f28d8073ef0 2024-09-11T01:53:10.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28df9f3640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:10.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28df9f3640 10 monclient(hunting): authenticate will time out at 2196.489724s 2024-09-11T01:53:10.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28dd78f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f28d80a08c0 0x7f28d8137b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:10.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28dcf8e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d8073ef0 0x7f28d8074310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:10.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28dd78f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f28d80a08c0 0x7f28d8137b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59388/0 (socket says 172.21.15.12:59388) 2024-09-11T01:53:10.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28dd78f640 1 -- 172.21.15.12:0/3163118306 learned_addr learned my addr 172.21.15.12:0/3163118306 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:53:10.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28dd78f640 10 monclient(hunting): get_auth_request con 0x7f28d80a08c0 auth_method 0 2024-09-11T01:53:10.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28dd78f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:10.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28dd78f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:10.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28dd78f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:10.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28dcf8e640 10 monclient(hunting): get_auth_request con 0x7f28d8073ef0 auth_method 0 2024-09-11T01:53:10.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28dcf8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:10.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28dcf8e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:10.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28dcf8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:10.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28dd78f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:10.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.047+0000 7f28dd78f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:10.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.048+0000 7f28dd78f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:10.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.048+0000 7f28dcf8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:10.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.048+0000 7f28dcf8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:10.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.048+0000 7f28dcf8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:10.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.048+0000 7f28dcf8e640 10 monclient(hunting): handle_auth_done global_id 244382 payload 274 2024-09-11T01:53:10.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.048+0000 7f28dcf8e640 1 -- 172.21.15.12:0/3163118306 >> v2:172.21.15.12:3300/0 conn(0x7f28d80a08c0 msgr2=0x7f28d8137b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:53:10.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.048+0000 7f28dcf8e640 1 --2- 172.21.15.12:0/3163118306 >> v2:172.21.15.12:3300/0 conn(0x7f28d80a08c0 0x7f28d8137b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:10.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.048+0000 7f28dcf8e640 1 -- 172.21.15.12:0/3163118306 >> v1:172.21.15.12:6789/0 conn(0x7f28d80a5690 legacy=0x7f28d80a5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:53:10.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.048+0000 7f28dcf8e640 10 monclient: _finish_hunting 0 2024-09-11T01:53:10.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.048+0000 7f28dcf8e640 1 monclient: found mon.noname-b 2024-09-11T01:53:10.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.048+0000 7f28dcf8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:10.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.048+0000 7f28dcf8e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:53:10.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.048+0000 7f28dcf8e640 1 -- 172.21.15.12:0/3163118306 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d8070150 con 0x7f28d8073ef0 2024-09-11T01:53:10.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28dcf8e640 1 -- 172.21.15.12:0/3163118306 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d8073ef0 msgr2=0x7f28d8074310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:10.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28dcf8e640 1 --2- 172.21.15.12:0/3163118306 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d8073ef0 0x7f28d8074310 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f28c80097a0 tx=0x7f28c8029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3300c378227473f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:10.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28cffff640 1 -- 172.21.15.12:0/3163118306 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c8037070 con 0x7f28d8073ef0 2024-09-11T01:53:10.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:10.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28cffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:53:10.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28cffff640 10 monclient: dump: 2024-09-11T01:53:10.058 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:10.058 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:10.058 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:10.058 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:10.058 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:10.058 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:10.058 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:10.058 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:10.058 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:10.058 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:10.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28cffff640 10 monclient: _finish_auth 0 2024-09-11T01:53:10.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28cffff640 10 monclient: _check_auth_tickets 2024-09-11T01:53:10.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:10.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28df9f3640 5 monclient: authenticate success, global_id 244382 2024-09-11T01:53:10.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28df9f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:53:10.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28cffff640 1 -- 172.21.15.12:0/3163118306 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28c8032440 con 0x7f28d8073ef0 2024-09-11T01:53:10.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28cffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:10.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28cffff640 1 -- 172.21.15.12:0/3163118306 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c8032810 con 0x7f28d8073ef0 2024-09-11T01:53:10.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28df9f3640 10 monclient: get_monmap_and_config success 2024-09-11T01:53:10.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28df9f3640 4 set_mon_vals no callback set 2024-09-11T01:53:10.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:10.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28cffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:53:10.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28cffff640 10 monclient: dump: 2024-09-11T01:53:10.060 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:10.060 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:10.060 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:10.060 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:10.060 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:10.060 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:10.060 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:10.060 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:10.060 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:10.061 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:10.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28df9f3640 10 monclient: shutdown 2024-09-11T01:53:10.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28df9f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:10.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28df9f3640 1 -- 172.21.15.12:0/3163118306 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d8073ef0 msgr2=0x7f28d8074310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:10.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28df9f3640 1 --2- 172.21.15.12:0/3163118306 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d8073ef0 0x7f28d8074310 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f28c80097a0 tx=0x7f28c8029fd0 comp rx=0 tx=0).stop 2024-09-11T01:53:10.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28df9f3640 1 -- 172.21.15.12:0/3163118306 shutdown_connections 2024-09-11T01:53:10.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28df9f3640 1 --2- 172.21.15.12:0/3163118306 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d8073ef0 0x7f28d8074310 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:10.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28df9f3640 1 --2- 172.21.15.12:0/3163118306 >> v2:172.21.15.12:3300/0 conn(0x7f28d80a08c0 0x7f28d8137b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:10.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28df9f3640 1 -- 172.21.15.12:0/3163118306 >> 172.21.15.12:0/3163118306 conn(0x7f28d8068950 msgr2=0x7f28d8068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:10.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28df9f3640 1 -- 172.21.15.12:0/3163118306 shutdown_connections 2024-09-11T01:53:10.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.051+0000 7f28df9f3640 1 -- 172.21.15.12:0/3163118306 wait complete. 2024-09-11T01:53:10.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.052+0000 7f28df9f3640 10 monclient: build_initial_monmap 2024-09-11T01:53:10.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.052+0000 7f28df9f3640 10 monclient: monmap: 2024-09-11T01:53:10.062 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:10.062 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:53:10.062 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:53:10.062 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:53:10.062 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:10.062 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:10.062 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:53:10.062 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:53:10.062 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:53:10.062 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:10.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.052+0000 7f28df9f3640 1 Processor -- start 2024-09-11T01:53:10.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.052+0000 7f28df9f3640 1 -- start start 2024-09-11T01:53:10.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.052+0000 7f28df9f3640 10 monclient: init 2024-09-11T01:53:10.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28df9f3640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:10.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28df9f3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:53:10.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28df9f3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28d8073ef0 0x7f28d813f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:10.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28df9f3640 10 monclient(hunting): picked mon.noname-a con 0x7f28d8073ef0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:53:10.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28df9f3640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d80a08c0 0x7f28d813f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:10.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28df9f3640 10 monclient(hunting): picked mon.noname-c con 0x7f28d80a08c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:53:10.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28df9f3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28d80a5690 0x7f28d81ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:10.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28df9f3640 10 monclient(hunting): picked mon.noname-b con 0x7f28d80a5690 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:53:10.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28df9f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:10.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28df9f3640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f28d8138210 con 0x7f28d8073ef0 2024-09-11T01:53:10.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28df9f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:10.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28df9f3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f28d8138090 con 0x7f28d80a5690 2024-09-11T01:53:10.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28df9f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:10.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28df9f3640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f28d80a6230 con 0x7f28d80a08c0 2024-09-11T01:53:10.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28df9f3640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:10.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28df9f3640 10 monclient(hunting): authenticate will time out at 2196.495407s 2024-09-11T01:53:10.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dd78f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28d8073ef0 0x7f28d813f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:10.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dcf8e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d80a08c0 0x7f28d813f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:10.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dd78f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28d8073ef0 0x7f28d813f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59392/0 (socket says 172.21.15.12:59392) 2024-09-11T01:53:10.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dd78f640 1 -- 172.21.15.12:0/3605354118 learned_addr learned my addr 172.21.15.12:0/3605354118 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:53:10.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dd78f640 10 monclient(hunting): get_auth_request con 0x7f28d8073ef0 auth_method 0 2024-09-11T01:53:10.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dd78f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:10.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dd78f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:10.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dd78f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:10.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dcf8e640 10 monclient(hunting): get_auth_request con 0x7f28d80a08c0 auth_method 0 2024-09-11T01:53:10.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28ddf90640 1 -- 172.21.15.12:0/3605354118 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28d80a5690 msgr2=0x7f28d81ccee0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:53:10.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dcf8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:10.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dcf8e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:10.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dcf8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:10.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28ddf90640 1 --2- 172.21.15.12:0/3605354118 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28d80a5690 0x7f28d81ccee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:53:10.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dd78f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:10.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dd78f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:10.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dd78f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:10.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dcf8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:10.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dcf8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:10.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dcf8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:10.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dcf8e640 10 monclient(hunting): handle_auth_done global_id 244385 payload 824 2024-09-11T01:53:10.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dcf8e640 1 -- 172.21.15.12:0/3605354118 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28d80a5690 msgr2=0x7f28d81ccee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:53:10.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dcf8e640 1 --2- 172.21.15.12:0/3605354118 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28d80a5690 0x7f28d81ccee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:10.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dcf8e640 1 -- 172.21.15.12:0/3605354118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28d8073ef0 msgr2=0x7f28d813f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:10.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dcf8e640 1 --2- 172.21.15.12:0/3605354118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28d8073ef0 0x7f28d813f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:10.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dcf8e640 10 monclient: _finish_hunting 0 2024-09-11T01:53:10.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dcf8e640 1 monclient: found mon.noname-c 2024-09-11T01:53:10.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dcf8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:10.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dcf8e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:53:10.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dcf8e640 1 -- 172.21.15.12:0/3605354118 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d8074b90 con 0x7f28d80a08c0 2024-09-11T01:53:10.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.053+0000 7f28dd78f640 1 --2- 172.21.15.12:0/3605354118 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28d8073ef0 0x7f28d813f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:53:10.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.056+0000 7f28dcf8e640 1 -- 172.21.15.12:0/3605354118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d80a08c0 msgr2=0x7f28d813f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:10.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.056+0000 7f28dcf8e640 1 --2- 172.21.15.12:0/3605354118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d80a08c0 0x7f28d813f5c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f28c8031000 tx=0x7f28c80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:10.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.056+0000 7f28cdffb640 1 -- 172.21.15.12:0/3605354118 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c8041020 con 0x7f28d80a08c0 2024-09-11T01:53:10.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.056+0000 7f28cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:10.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.056+0000 7f28cdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:53:10.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.056+0000 7f28cdffb640 10 monclient: dump: 2024-09-11T01:53:10.073 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:10.073 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:10.073 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:10.073 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:10.073 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:10.073 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:10.073 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:10.073 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:10.073 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:10.073 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:10.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.056+0000 7f28cdffb640 10 monclient: _finish_auth 0 2024-09-11T01:53:10.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.056+0000 7f28cdffb640 10 monclient: _check_auth_tickets 2024-09-11T01:53:10.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.056+0000 7f28cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:10.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.056+0000 7f28df9f3640 5 monclient: authenticate success, global_id 244385 2024-09-11T01:53:10.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.056+0000 7f28cdffb640 1 -- 172.21.15.12:0/3605354118 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28c803b8c0 con 0x7f28d80a08c0 2024-09-11T01:53:10.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.056+0000 7f28df9f3640 10 monclient: _renew_subs 2024-09-11T01:53:10.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.056+0000 7f28df9f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:53:10.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.056+0000 7f28df9f3640 1 -- 172.21.15.12:0/3605354118 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28d813fb00 con 0x7f28d80a08c0 2024-09-11T01:53:10.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.056+0000 7f28cdffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:10.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.056+0000 7f28cdffb640 1 -- 172.21.15.12:0/3605354118 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c8037040 con 0x7f28d80a08c0 2024-09-11T01:53:10.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.056+0000 7f28cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:10.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.056+0000 7f28cdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:53:10.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.056+0000 7f28cdffb640 10 monclient: dump: 2024-09-11T01:53:10.074 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:10.074 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:10.074 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:10.074 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:10.075 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:10.075 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:10.075 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:10.075 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:10.075 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:10.075 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:10.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.056+0000 7f28df9f3640 10 monclient: _renew_subs 2024-09-11T01:53:10.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.056+0000 7f28df9f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:53:10.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.056+0000 7f28df9f3640 1 -- 172.21.15.12:0/3605354118 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f28d813b290 con 0x7f28d80a08c0 2024-09-11T01:53:10.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.056+0000 7f28ceffd640 4 set_mon_vals no callback set 2024-09-11T01:53:10.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.057+0000 7f28df9f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:10.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.057+0000 7f28df9f3640 1 --2- 172.21.15.12:0/3605354118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d802d440 0x7f28d802d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:10.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.057+0000 7f28df9f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:10.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.057+0000 7f28df9f3640 1 -- 172.21.15.12:0/3605354118 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f28d8074310 con 0x7f28d802d440 2024-09-11T01:53:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.057+0000 7f28dd78f640 1 --2- 172.21.15.12:0/3605354118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d802d440 0x7f28d802d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.057+0000 7f28dd78f640 10 monclient: get_auth_request con 0x7f28d802d440 auth_method 0 2024-09-11T01:53:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.057+0000 7f28dd78f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.057+0000 7f28dd78f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.057+0000 7f28dd78f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.057+0000 7f28cdffb640 1 -- 172.21.15.12:0/3605354118 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f28c803a030 con 0x7f28d80a08c0 2024-09-11T01:53:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.057+0000 7f28dd78f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.057+0000 7f28dd78f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.057+0000 7f28dd78f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.057+0000 7f28dd78f640 10 monclient(hunting): handle_auth_done global_id 244388 payload 824 2024-09-11T01:53:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.057+0000 7f28dd78f640 1 -- 172.21.15.12:0/3605354118 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f28d8130810 con 0x7f28d802d440 2024-09-11T01:53:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.057+0000 7f28cdffb640 1 --2- 172.21.15.12:0/3605354118 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f28bc03e240 0x7f28bc040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:10.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.057+0000 7f28cdffb640 1 -- 172.21.15.12:0/3605354118 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(804..804 src has 676..804) ==== 4739+0+0 (secure 0 0 0) 0x7f28c80727b0 con 0x7f28d80a08c0 2024-09-11T01:53:10.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.057+0000 7f28ddf90640 1 --2- 172.21.15.12:0/3605354118 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f28bc03e240 0x7f28bc040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:10.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.057+0000 7f28ddf90640 10 monclient: get_auth_request con 0x7f28bc03e240 auth_method 0 2024-09-11T01:53:10.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.059+0000 7f28dd78f640 1 -- 172.21.15.12:0/3605354118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d802d440 msgr2=0x7f28d802d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:10.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.059+0000 7f28dd78f640 1 --2- 172.21.15.12:0/3605354118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d802d440 0x7f28d802d820 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f28d4009f70 tx=0x7f28d400f610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:10.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.059+0000 7f28cdffb640 1 -- 172.21.15.12:0/3605354118 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d4004020 con 0x7f28d802d440 2024-09-11T01:53:10.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.059+0000 7f28cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:10.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.059+0000 7f28cdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:53:10.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.059+0000 7f28cdffb640 10 monclient: dump: 2024-09-11T01:53:10.077 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:10.077 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:10.077 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:10.077 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:10.077 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:10.077 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:10.078 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.060+0000 7f28cdffb640 1 -- 172.21.15.12:0/3605354118 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f28d4004020 con 0x7f28d802d440 2024-09-11T01:53:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.060+0000 7f28cdffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.060+0000 7f28cdffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:53:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.060+0000 7f28cdffb640 1 -- 172.21.15.12:0/3605354118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d802d440 msgr2=0x7f28d802d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.060+0000 7f28cdffb640 1 --2- 172.21.15.12:0/3605354118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d802d440 0x7f28d802d820 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f28d4009f70 tx=0x7f28d400f610 comp rx=0 tx=0).stop 2024-09-11T01:53:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.060+0000 7f28cdffb640 1 -- 172.21.15.12:0/3605354118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d802d440 msgr2=0x7f28d802d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:53:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.060+0000 7f28cdffb640 1 --2- 172.21.15.12:0/3605354118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d802d440 0x7f28d802d820 secure :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f28d4009f70 tx=0x7f28d400f610 comp rx=0 tx=0).stop 2024-09-11T01:53:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.060+0000 7f28ddf90640 1 --2- 172.21.15.12:0/3605354118 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f28bc03e240 0x7f28bc040700 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f28d8139df0 tx=0x7f28d0007d10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.066+0000 7f28df9f3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:53:10.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.066+0000 7f28df9f3640 1 --2- 172.21.15.12:0/3605354118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d802dd60 0x7f28d802e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:10.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.066+0000 7f28df9f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:10.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.066+0000 7f28df9f3640 1 -- 172.21.15.12:0/3605354118 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f28d813a390 con 0x7f28d802dd60 2024-09-11T01:53:10.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.066+0000 7f28dd78f640 1 --2- 172.21.15.12:0/3605354118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d802dd60 0x7f28d802e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:10.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.066+0000 7f28dd78f640 10 monclient: get_auth_request con 0x7f28d802dd60 auth_method 0 2024-09-11T01:53:10.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.066+0000 7f28dd78f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:10.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.066+0000 7f28dd78f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:10.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.066+0000 7f28dd78f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:10.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.066+0000 7f28dd78f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:10.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.066+0000 7f28dd78f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:10.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.066+0000 7f28dd78f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:10.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.066+0000 7f28dd78f640 10 monclient(hunting): handle_auth_done global_id 244391 payload 824 2024-09-11T01:53:10.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.066+0000 7f28dd78f640 1 -- 172.21.15.12:0/3605354118 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f28d806d8c0 con 0x7f28d802dd60 2024-09-11T01:53:10.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.069+0000 7f28dd78f640 1 -- 172.21.15.12:0/3605354118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d802dd60 msgr2=0x7f28d802e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.069+0000 7f28dd78f640 1 --2- 172.21.15.12:0/3605354118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d802dd60 0x7f28d802e170 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f28d4009f70 tx=0x7f28d40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.069+0000 7f28cdffb640 1 -- 172.21.15.12:0/3605354118 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d4003c30 con 0x7f28d802dd60 2024-09-11T01:53:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.069+0000 7f28cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.069+0000 7f28cdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:53:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.069+0000 7f28cdffb640 10 monclient: dump: 2024-09-11T01:53:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:10.081 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:10.082 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:10.082 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:10.082 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:10.082 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:10.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.069+0000 7f28cdffb640 1 -- 172.21.15.12:0/3605354118 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4865 (secure 0 0 0) 0x7f28d4003c30 con 0x7f28d802dd60 2024-09-11T01:53:10.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.069+0000 7f28cdffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:53:10.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.069+0000 7f28cdffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:53:10.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.069+0000 7f28cdffb640 1 -- 172.21.15.12:0/3605354118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d802dd60 msgr2=0x7f28d802e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:10.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.069+0000 7f28cdffb640 1 --2- 172.21.15.12:0/3605354118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d802dd60 0x7f28d802e170 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f28d4009f70 tx=0x7f28d40046c0 comp rx=0 tx=0).stop 2024-09-11T01:53:10.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.069+0000 7f28cdffb640 1 -- 172.21.15.12:0/3605354118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d802dd60 msgr2=0x7f28d802e170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:53:10.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.069+0000 7f28cdffb640 1 --2- 172.21.15.12:0/3605354118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d802dd60 0x7f28d802e170 secure :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f28d4009f70 tx=0x7f28d40046c0 comp rx=0 tx=0).stop 2024-09-11T01:53:10.082 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:53:10.083 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:53:10.083 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:53:10.083 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:53:10.083 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 224, 2024-09-11T01:53:10.083 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 143517, 2024-09-11T01:53:10.083 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:53:10.083 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:53:10.083 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:53:10.083 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:10.084 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:53:10.084 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:53:10.084 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:53:10.084 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:53:10.084 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:53:10.084 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:53:10.084 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:53:10.084 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:53:10.084 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:53:10.084 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:53:10.084 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:53:10.084 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:53:10.084 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:53:10.085 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:53:10.085 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:53:10.085 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:10.085 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:53:10.085 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:53:10.085 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:53:10.085 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:53:10.085 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:53:10.085 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:53:10.085 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:53:10.085 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:53:10.085 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:53:10.085 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:53:10.085 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:53:10.086 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:53:10.086 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:53:10.086 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:10.086 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:10.086 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:53:10.086 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:53:10.086 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:53:10.086 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:53:10.086 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:53:10.086 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:53:10.086 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:53:10.086 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:53:10.086 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:53:10.086 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:53:10.087 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:53:10.087 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:53:10.087 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:53:10.087 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:53:10.087 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:53:10.087 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:53:10.087 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:53:10.087 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:53:10.087 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:53:10.087 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:53:10.087 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:53:10.087 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:53:10.087 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:53:10.087 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:53:10.087 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:53:10.088 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:53:10.088 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:53:10.088 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:53:10.088 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:10.088 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:53:10.088 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:10.088 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:53:10.088 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:10.088 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:53:10.088 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:53:10.088 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:53:10.088 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:53:10.088 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:10.089 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:53:10.089 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:53:10.089 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:10.089 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:10.089 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:10.089 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:53:10.089 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:53:10.089 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:10.089 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:10.089 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:10.089 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:10.090 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:53:10.090 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:53:10.090 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:53:10.090 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:53:10.090 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:53:10.090 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:10.090 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:10.090 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:53:10.090 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:53:10.090 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:53:10.090 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:53:10.090 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:10.090 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:53:10.090 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:53:10.091 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:10.091 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:10.091 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:10.091 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:53:10.091 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:53:10.091 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:10.091 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:10.091 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:10.091 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:10.091 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:53:10.091 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:53:10.091 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:53:10.091 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:53:10.091 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:53:10.092 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:10.092 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:10.092 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:53:10.092 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:53:10.092 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:53:10.092 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:53:10.092 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:10.092 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:53:10.092 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:53:10.092 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:10.092 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:10.092 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:10.092 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:53:10.093 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:53:10.093 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:10.093 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:10.093 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:10.093 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:10.093 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:53:10.093 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:53:10.093 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:53:10.093 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:53:10.094 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:53:10.094 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:10.094 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:10.094 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:10.094 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:53:10.094 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:53:10.094 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:10.094 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:53:10.094 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:53:10.094 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:53:10.094 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:10.094 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:10.094 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:53:10.094 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:10.095 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:53:10.095 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:53:10.095 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:53:10.095 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:10.095 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:10.095 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:53:10.095 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:10.095 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:53:10.095 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:53:10.095 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:53:10.095 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:10.095 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:10.095 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:53:10.096 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:10.096 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:53:10.096 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:53:10.096 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:53:10.096 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:10.096 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:10.096 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:10.096 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:53:10.096 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:53:10.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.069+0000 7f28df9f3640 1 -- 172.21.15.12:0/3605354118 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f28bc03e240 msgr2=0x7f28bc040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:10.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.069+0000 7f28df9f3640 1 --2- 172.21.15.12:0/3605354118 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f28bc03e240 0x7f28bc040700 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f28d8139df0 tx=0x7f28d0007d10 comp rx=0 tx=0).stop 2024-09-11T01:53:10.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.069+0000 7f28df9f3640 10 monclient: shutdown 2024-09-11T01:53:10.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.069+0000 7f28df9f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:10.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.069+0000 7f28df9f3640 1 -- 172.21.15.12:0/3605354118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d80a08c0 msgr2=0x7f28d813f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:10.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.069+0000 7f28ddf90640 1 -- 172.21.15.12:0/3605354118 reap_dead start 2024-09-11T01:53:10.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.069+0000 7f28df9f3640 1 --2- 172.21.15.12:0/3605354118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d80a08c0 0x7f28d813f5c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f28c8031000 tx=0x7f28c80046c0 comp rx=0 tx=0).stop 2024-09-11T01:53:10.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.069+0000 7f28df9f3640 1 -- 172.21.15.12:0/3605354118 shutdown_connections 2024-09-11T01:53:10.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.069+0000 7f28df9f3640 1 --2- 172.21.15.12:0/3605354118 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28d80a08c0 0x7f28d813f5c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:10.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.069+0000 7f28df9f3640 1 -- 172.21.15.12:0/3605354118 >> 172.21.15.12:0/3605354118 conn(0x7f28d8068950 msgr2=0x7f28d8134500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:10.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.069+0000 7f28df9f3640 1 -- 172.21.15.12:0/3605354118 shutdown_connections 2024-09-11T01:53:10.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:10.069+0000 7f28df9f3640 1 -- 172.21.15.12:0/3605354118 wait complete. 2024-09-11T01:53:10.099 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:53:10.099 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:53:10.795 INFO:tasks.rados.rados.0.smithi071.stdout:3557: done (0 left) 2024-09-11T01:53:10.795 INFO:tasks.rados.rados.0.smithi071.stdout:3558: write oid 12 current snap is 403 2024-09-11T01:53:10.795 INFO:tasks.rados.rados.0.smithi071.stdout:3558: seq_num 1232 ranges {706224=755954,2160404=545820} 2024-09-11T01:53:10.804 INFO:tasks.rados.rados.0.smithi071.stdout:3558: writing smithi07134777-12 from 706224 to 1462178 tid 1 2024-09-11T01:53:10.811 INFO:tasks.rados.rados.0.smithi071.stdout:3558: writing smithi07134777-12 from 2160404 to 2706224 tid 2 2024-09-11T01:53:10.811 INFO:tasks.rados.rados.0.smithi071.stdout:3559: snap_create 2024-09-11T01:53:10.817 INFO:tasks.rados.rados.0.smithi071.stdout:3558: finishing write tid 1 to smithi07134777-12 2024-09-11T01:53:10.818 INFO:tasks.rados.rados.0.smithi071.stdout:3558: finishing write tid 2 to smithi07134777-12 2024-09-11T01:53:10.818 INFO:tasks.rados.rados.0.smithi071.stdout:3558: finishing write tid 3 to smithi07134777-12 2024-09-11T01:53:10.818 INFO:tasks.rados.rados.0.smithi071.stdout:3558: finishing write tid 4 to smithi07134777-12 2024-09-11T01:53:10.818 INFO:tasks.rados.rados.0.smithi071.stdout:3558: oid 12 updating version 0 to 2532 2024-09-11T01:53:10.818 INFO:tasks.rados.rados.0.smithi071.stdout:3558: oid 12 updating version 2532 to 2533 2024-09-11T01:53:10.818 INFO:tasks.rados.rados.0.smithi071.stdout:3558: oid 12 version 2533 is already newer than 2527 2024-09-11T01:53:10.818 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 2533 (ObjNum 1232 snap 403 seq_num 1232) dirty exists 2024-09-11T01:53:10.818 INFO:tasks.rados.rados.0.smithi071.stdout:3558: left oid 12 (ObjNum 1232 snap 403 seq_num 1232) 2024-09-11T01:53:11.806 INFO:tasks.rados.rados.0.smithi071.stdout:3558: done (1 left) 2024-09-11T01:53:11.807 INFO:tasks.rados.rados.0.smithi071.stdout:3559: done (0 left) 2024-09-11T01:53:11.808 INFO:tasks.rados.rados.0.smithi071.stdout:3560: snap_remove snap 395 2024-09-11T01:53:12.816 INFO:tasks.rados.rados.0.smithi071.stdout:3560: done (0 left) 2024-09-11T01:53:12.816 INFO:tasks.rados.rados.0.smithi071.stdout:3561: copy_from oid 44 from oid 7 current snap is 404 2024-09-11T01:53:12.817 INFO:tasks.rados.rados.0.smithi071.stdout:3562: snap_create 2024-09-11T01:53:12.822 INFO:tasks.rados.rados.0.smithi071.stdout:3561: finishing copy_from to smithi07134777-44 2024-09-11T01:53:12.823 INFO:tasks.rados.rados.0.smithi071.stdout:3561: got expected ENOENT (src dne) 2024-09-11T01:53:12.823 INFO:tasks.rados.rados.0.smithi071.stdout:3561: finishing copy_from racing read to smithi07134777-44 2024-09-11T01:53:13.310 INFO:tasks.rados.rados.0.smithi071.stdout:3561: done (1 left) 2024-09-11T01:53:13.310 INFO:tasks.rados.rados.0.smithi071.stdout:3562: done (0 left) 2024-09-11T01:53:13.310 INFO:tasks.rados.rados.0.smithi071.stdout:3563: rollback oid 20 current snap is 405 2024-09-11T01:53:13.310 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 20 to 397 2024-09-11T01:53:13.310 INFO:tasks.rados.rados.0.smithi071.stdout:3564: rollback oid 40 current snap is 405 2024-09-11T01:53:13.310 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 40 to 385 2024-09-11T01:53:13.310 INFO:tasks.rados.rados.0.smithi071.stdout:3565: snap_create 2024-09-11T01:53:13.314 INFO:tasks.rados.rados.0.smithi071.stdout:3564: finishing rollback tid 0 to smithi07134777-40 2024-09-11T01:53:13.316 INFO:tasks.rados.rados.0.smithi071.stdout:3564: finishing rollback tid 1 to smithi07134777-40 2024-09-11T01:53:13.316 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 2572 (ObjNum 145 snap 0 seq_num 2371125264) dirty dne 2024-09-11T01:53:13.318 INFO:tasks.rados.rados.0.smithi071.stdout:3563: finishing rollback tid 0 to smithi07134777-20 2024-09-11T01:53:13.319 INFO:tasks.rados.rados.0.smithi071.stdout:3563: finishing rollback tid 1 to smithi07134777-20 2024-09-11T01:53:13.319 INFO:tasks.rados.rados.0.smithi071.stdout:3563: finishing rollback tid 2 to smithi07134777-20 2024-09-11T01:53:13.319 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 1223 (ObjNum 1216 snap 397 seq_num 1216) dirty exists 2024-09-11T01:53:15.011 INFO:tasks.rados.rados.0.smithi071.stdout:3563: done (2 left) 2024-09-11T01:53:15.011 INFO:tasks.rados.rados.0.smithi071.stdout:3564: done (1 left) 2024-09-11T01:53:15.011 INFO:tasks.rados.rados.0.smithi071.stdout:3565: done (0 left) 2024-09-11T01:53:15.011 INFO:tasks.rados.rados.0.smithi071.stdout:3566: snap_create 2024-09-11T01:53:15.821 INFO:tasks.rados.rados.0.smithi071.stdout:3566: done (0 left) 2024-09-11T01:53:15.821 INFO:tasks.rados.rados.0.smithi071.stdout:3567: write (excl) oid 3 current snap is 407 2024-09-11T01:53:15.821 INFO:tasks.rados.rados.0.smithi071.stdout:3567: seq_num 1233 ranges {723031=431203,1818886=469759} 2024-09-11T01:53:15.827 INFO:tasks.rados.rados.0.smithi071.stdout:3567: writing smithi07134777-3 from 723031 to 1154234 tid 1 2024-09-11T01:53:15.834 INFO:tasks.rados.rados.0.smithi071.stdout:3567: writing smithi07134777-3 from 1818886 to 2288645 tid 2 2024-09-11T01:53:15.834 INFO:tasks.rados.rados.0.smithi071.stdout:3568: write oid 36 current snap is 407 2024-09-11T01:53:15.834 INFO:tasks.rados.rados.0.smithi071.stdout:3568: seq_num 1234 ranges {739838=506452,1877368=613417} 2024-09-11T01:53:15.841 INFO:tasks.rados.rados.0.smithi071.stdout:3568: writing smithi07134777-36 from 739838 to 1246290 tid 1 2024-09-11T01:53:15.848 INFO:tasks.rados.rados.0.smithi071.stdout:3568: writing smithi07134777-36 from 1877368 to 2490785 tid 2 2024-09-11T01:53:15.848 INFO:tasks.rados.rados.0.smithi071.stdout:3569: snap_create 2024-09-11T01:53:15.848 INFO:tasks.rados.rados.0.smithi071.stdout:3567: finishing write tid 1 to smithi07134777-3 2024-09-11T01:53:15.848 INFO:tasks.rados.rados.0.smithi071.stdout:3567: finishing write tid 2 to smithi07134777-3 2024-09-11T01:53:15.848 INFO:tasks.rados.rados.0.smithi071.stdout:3567: finishing write tid 3 to smithi07134777-3 2024-09-11T01:53:15.848 INFO:tasks.rados.rados.0.smithi071.stdout:3567: finishing write tid 4 to smithi07134777-3 2024-09-11T01:53:15.848 INFO:tasks.rados.rados.0.smithi071.stdout:3567: oid 3 updating version 0 to 2431 2024-09-11T01:53:15.848 INFO:tasks.rados.rados.0.smithi071.stdout:3567: oid 3 version 2431 is already newer than 2429 2024-09-11T01:53:15.848 INFO:tasks.rados.rados.0.smithi071.stdout:3567: oid 3 version 2431 is already newer than 2430 2024-09-11T01:53:15.849 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 2431 (ObjNum 1233 snap 407 seq_num 1233) dirty exists 2024-09-11T01:53:15.849 INFO:tasks.rados.rados.0.smithi071.stdout:3567: left oid 3 (ObjNum 1233 snap 407 seq_num 1233) 2024-09-11T01:53:15.849 INFO:tasks.rados.rados.0.smithi071.stdout:3568: finishing write tid 1 to smithi07134777-36 2024-09-11T01:53:15.852 INFO:tasks.rados.rados.0.smithi071.stdout:3568: finishing write tid 2 to smithi07134777-36 2024-09-11T01:53:15.853 INFO:tasks.rados.rados.0.smithi071.stdout:3568: finishing write tid 3 to smithi07134777-36 2024-09-11T01:53:15.853 INFO:tasks.rados.rados.0.smithi071.stdout:3568: finishing write tid 4 to smithi07134777-36 2024-09-11T01:53:15.853 INFO:tasks.rados.rados.0.smithi071.stdout:3568: oid 36 updating version 0 to 2094 2024-09-11T01:53:15.853 INFO:tasks.rados.rados.0.smithi071.stdout:3568: oid 36 updating version 2094 to 2095 2024-09-11T01:53:15.853 INFO:tasks.rados.rados.0.smithi071.stdout:3568: oid 36 updating version 2095 to 2096 2024-09-11T01:53:15.853 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 2096 (ObjNum 1234 snap 407 seq_num 1234) dirty exists 2024-09-11T01:53:15.853 INFO:tasks.rados.rados.0.smithi071.stdout:3568: left oid 36 (ObjNum 1234 snap 407 seq_num 1234) 2024-09-11T01:53:16.834 INFO:tasks.rados.rados.0.smithi071.stdout:3567: done (2 left) 2024-09-11T01:53:16.834 INFO:tasks.rados.rados.0.smithi071.stdout:3568: done (1 left) 2024-09-11T01:53:16.834 INFO:tasks.rados.rados.0.smithi071.stdout:3569: done (0 left) 2024-09-11T01:53:16.834 INFO:tasks.rados.rados.0.smithi071.stdout:3570: snap_remove snap 350 2024-09-11T01:53:17.839 INFO:tasks.rados.rados.0.smithi071.stdout:3570: done (0 left) 2024-09-11T01:53:17.839 INFO:tasks.rados.rados.0.smithi071.stdout:3571: delete oid 14 current snap is 408 2024-09-11T01:53:17.849 INFO:tasks.rados.rados.0.smithi071.stdout:3571: done (0 left) 2024-09-11T01:53:17.849 INFO:tasks.rados.rados.0.smithi071.stdout:3572: read oid 18 snap -1 2024-09-11T01:53:17.849 INFO:tasks.rados.rados.0.smithi071.stdout:3572: expect (ObjNum 618 snap 183 seq_num 618) 2024-09-11T01:53:17.849 INFO:tasks.rados.rados.0.smithi071.stdout:3573: snap_remove snap 382 2024-09-11T01:53:18.311 INFO:tasks.rados.rados.0.smithi071.stdout:3573: done (1 left) 2024-09-11T01:53:18.311 INFO:tasks.rados.rados.0.smithi071.stdout:3574: write (excl) oid 21 current snap is 408 2024-09-11T01:53:18.311 INFO:tasks.rados.rados.0.smithi071.stdout:3574: seq_num 1236 ranges {773452=656950,1877979=617086} 2024-09-11T01:53:18.318 INFO:tasks.rados.rados.0.smithi071.stdout:3574: writing smithi07134777-21 from 773452 to 1430402 tid 1 2024-09-11T01:53:18.325 INFO:tasks.rados.rados.0.smithi071.stdout:3574: writing smithi07134777-21 from 1877979 to 2495065 tid 2 2024-09-11T01:53:18.325 INFO:tasks.rados.rados.0.smithi071.stdout:3575: snap_remove snap 378 2024-09-11T01:53:19.316 INFO:tasks.rados.rados.0.smithi071.stdout:3575: done (2 left) 2024-09-11T01:53:19.316 INFO:tasks.rados.rados.0.smithi071.stdout:3576: snap_remove snap 375 2024-09-11T01:53:20.334 INFO:tasks.rados.rados.0.smithi071.stdout:3576: done (2 left) 2024-09-11T01:53:20.334 INFO:tasks.rados.rados.0.smithi071.stdout:3577: snap_create 2024-09-11T01:53:20.334 INFO:tasks.rados.rados.0.smithi071.stdout:3572: expect (ObjNum 618 snap 183 seq_num 618) 2024-09-11T01:53:20.511 INFO:tasks.rados.rados.0.smithi071.stdout:3574: finishing write tid 1 to smithi07134777-21 2024-09-11T01:53:20.511 INFO:tasks.rados.rados.0.smithi071.stdout:3574: finishing write tid 2 to smithi07134777-21 2024-09-11T01:53:20.511 INFO:tasks.rados.rados.0.smithi071.stdout:3574: finishing write tid 3 to smithi07134777-21 2024-09-11T01:53:20.511 INFO:tasks.rados.rados.0.smithi071.stdout:3574: finishing write tid 4 to smithi07134777-21 2024-09-11T01:53:20.512 INFO:tasks.rados.rados.0.smithi071.stdout:3574: oid 21 updating version 0 to 2222 2024-09-11T01:53:20.512 INFO:tasks.rados.rados.0.smithi071.stdout:3574: oid 21 version 2222 is already newer than 2220 2024-09-11T01:53:20.512 INFO:tasks.rados.rados.0.smithi071.stdout:3574: oid 21 version 2222 is already newer than 2221 2024-09-11T01:53:20.512 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 2222 (ObjNum 1236 snap 408 seq_num 1236) dirty exists 2024-09-11T01:53:20.512 INFO:tasks.rados.rados.0.smithi071.stdout:3574: left oid 21 (ObjNum 1236 snap 408 seq_num 1236) 2024-09-11T01:53:21.327 INFO:tasks.rados.rados.0.smithi071.stdout:3572: done (2 left) 2024-09-11T01:53:21.327 INFO:tasks.rados.rados.0.smithi071.stdout:3574: done (1 left) 2024-09-11T01:53:21.328 INFO:tasks.rados.rados.0.smithi071.stdout:3577: done (0 left) 2024-09-11T01:53:21.328 INFO:tasks.rados.rados.0.smithi071.stdout:3578: rollback oid 32 current snap is 409 2024-09-11T01:53:21.328 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 32 to 384 2024-09-11T01:53:21.328 INFO:tasks.rados.rados.0.smithi071.stdout:3579: rollback oid 28 current snap is 409 2024-09-11T01:53:21.328 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 28 to 401 2024-09-11T01:53:21.328 INFO:tasks.rados.rados.0.smithi071.stdout:3580: read oid 6 snap -1 2024-09-11T01:53:21.328 INFO:tasks.rados.rados.0.smithi071.stdout:3580: expect (ObjNum 1225 snap 397 seq_num 1225) 2024-09-11T01:53:21.329 INFO:tasks.rados.rados.0.smithi071.stdout:3581: delete oid 24 current snap is 409 2024-09-11T01:53:21.331 INFO:tasks.rados.rados.0.smithi071.stdout:3578: finishing rollback tid 0 to smithi07134777-32 2024-09-11T01:53:21.333 INFO:tasks.rados.rados.0.smithi071.stdout:3578: finishing rollback tid 1 to smithi07134777-32 2024-09-11T01:53:21.333 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 2101 (ObjNum 129 snap 0 seq_num 2371363568) dirty dne 2024-09-11T01:53:21.335 INFO:tasks.rados.rados.0.smithi071.stdout:3580: expect (ObjNum 1225 snap 397 seq_num 1225) 2024-09-11T01:53:21.403 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 581598 2024-09-11T01:53:21.484 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1411912 2024-09-11T01:53:21.523 INFO:tasks.rados.rados.0.smithi071.stdout:3579: finishing rollback tid 0 to smithi07134777-28 2024-09-11T01:53:21.523 INFO:tasks.rados.rados.0.smithi071.stdout:3579: finishing rollback tid 1 to smithi07134777-28 2024-09-11T01:53:21.523 INFO:tasks.rados.rados.0.smithi071.stdout:3579: finishing rollback tid 2 to smithi07134777-28 2024-09-11T01:53:21.523 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 2242 (ObjNum 1226 snap 401 seq_num 1226) dirty exists 2024-09-11T01:53:21.523 INFO:tasks.rados.rados.0.smithi071.stdout:3578: done (3 left) 2024-09-11T01:53:21.523 INFO:tasks.rados.rados.0.smithi071.stdout:3579: done (2 left) 2024-09-11T01:53:21.524 INFO:tasks.rados.rados.0.smithi071.stdout:3580: done (1 left) 2024-09-11T01:53:21.524 INFO:tasks.rados.rados.0.smithi071.stdout:3581: done (0 left) 2024-09-11T01:53:21.524 INFO:tasks.rados.rados.0.smithi071.stdout:3582: write (excl) oid 14 current snap is 409 2024-09-11T01:53:21.524 INFO:tasks.rados.rados.0.smithi071.stdout:3582: seq_num 1238 ranges {407066=407448,1594943=620755} 2024-09-11T01:53:21.526 INFO:tasks.rados.rados.0.smithi071.stdout:3582: writing smithi07134777-14 from 407066 to 814514 tid 1 2024-09-11T01:53:21.535 INFO:tasks.rados.rados.0.smithi071.stdout:3582: writing smithi07134777-14 from 1594943 to 2215698 tid 2 2024-09-11T01:53:21.535 INFO:tasks.rados.rados.0.smithi071.stdout:3583: rollback oid 38 current snap is 409 2024-09-11T01:53:21.535 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 38 to 393 2024-09-11T01:53:21.535 INFO:tasks.rados.rados.0.smithi071.stdout:3582: finishing write tid 1 to smithi07134777-14 2024-09-11T01:53:21.535 INFO:tasks.rados.rados.0.smithi071.stdout:3584: snap_remove snap 406 2024-09-11T01:53:22.326 INFO:tasks.rados.rados.0.smithi071.stdout:3584: done (2 left) 2024-09-11T01:53:22.326 INFO:tasks.rados.rados.0.smithi071.stdout:3585: copy_from oid 10 from oid 12 current snap is 409 2024-09-11T01:53:22.327 INFO:tasks.rados.rados.0.smithi071.stdout:3582: finishing write tid 2 to smithi07134777-14 2024-09-11T01:53:22.327 INFO:tasks.rados.rados.0.smithi071.stdout:3582: finishing write tid 3 to smithi07134777-14 2024-09-11T01:53:22.327 INFO:tasks.rados.rados.0.smithi071.stdout:3582: finishing write tid 4 to smithi07134777-14 2024-09-11T01:53:22.327 INFO:tasks.rados.rados.0.smithi071.stdout:3582: oid 14 updating version 0 to 2731 2024-09-11T01:53:22.327 INFO:tasks.rados.rados.0.smithi071.stdout:3582: oid 14 version 2731 is already newer than 2730 2024-09-11T01:53:22.327 INFO:tasks.rados.rados.0.smithi071.stdout:3582: oid 14 version 2731 is already newer than 2729 2024-09-11T01:53:22.327 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 2731 (ObjNum 1238 snap 409 seq_num 1238) dirty exists 2024-09-11T01:53:22.327 INFO:tasks.rados.rados.0.smithi071.stdout:3582: left oid 14 (ObjNum 1238 snap 409 seq_num 1238) 2024-09-11T01:53:22.327 INFO:tasks.rados.rados.0.smithi071.stdout:3583: finishing rollback tid 1 to smithi07134777-38 2024-09-11T01:53:22.327 INFO:tasks.rados.rados.0.smithi071.stdout:3583: finishing rollback tid 2 to smithi07134777-38 2024-09-11T01:53:22.327 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 2733 (ObjNum 1162 snap 370 seq_num 1162) dirty exists 2024-09-11T01:53:22.327 INFO:tasks.rados.rados.0.smithi071.stdout:3582: done (2 left) 2024-09-11T01:53:22.327 INFO:tasks.rados.rados.0.smithi071.stdout:3583: done (1 left) 2024-09-11T01:53:22.327 INFO:tasks.rados.rados.0.smithi071.stdout:3586: write (excl) oid 19 current snap is 409 2024-09-11T01:53:22.328 INFO:tasks.rados.rados.0.smithi071.stdout:3586: seq_num 1239 ranges {423873=482697,1653425=480766,2691441=132432} 2024-09-11T01:53:22.331 INFO:tasks.rados.rados.0.smithi071.stdout:3586: writing smithi07134777-19 from 423873 to 906570 tid 1 2024-09-11T01:53:22.338 INFO:tasks.rados.rados.0.smithi071.stdout:3586: writing smithi07134777-19 from 1653425 to 2134191 tid 2 2024-09-11T01:53:22.343 INFO:tasks.rados.rados.0.smithi071.stdout:3586: writing smithi07134777-19 from 2691441 to 2823873 tid 3 2024-09-11T01:53:22.343 INFO:tasks.rados.rados.0.smithi071.stdout:3587: read oid 27 snap 400 2024-09-11T01:53:22.343 INFO:tasks.rados.rados.0.smithi071.stdout:3587: expect (ObjNum 1154 snap 366 seq_num 1154) 2024-09-11T01:53:22.343 INFO:tasks.rados.rados.0.smithi071.stdout:3588: copy_from oid 14 from oid 43 current snap is 409 2024-09-11T01:53:22.343 INFO:tasks.rados.rados.0.smithi071.stdout:3589: snap_remove snap 370 2024-09-11T01:53:23.313 INFO:tasks.rados.rados.0.smithi071.stdout:3589: done (4 left) 2024-09-11T01:53:23.313 INFO:tasks.rados.rados.0.smithi071.stdout:3590: read oid 38 snap 399 2024-09-11T01:53:23.313 INFO:tasks.rados.rados.0.smithi071.stdout:3590: expect deleted 2024-09-11T01:53:23.313 INFO:tasks.rados.rados.0.smithi071.stdout:3591: rollback oid 42 current snap is 409 2024-09-11T01:53:23.313 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 42 to 391 2024-09-11T01:53:23.313 INFO:tasks.rados.rados.0.smithi071.stdout:3585: finishing copy_from racing read to smithi07134777-10 2024-09-11T01:53:23.313 INFO:tasks.rados.rados.0.smithi071.stdout:3585: finishing copy_from to smithi07134777-10 2024-09-11T01:53:23.313 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 2533 (ObjNum 1232 snap 403 seq_num 1232) dirty exists 2024-09-11T01:53:23.314 INFO:tasks.rados.rados.0.smithi071.stdout:3586: finishing write tid 1 to smithi07134777-19 2024-09-11T01:53:23.314 INFO:tasks.rados.rados.0.smithi071.stdout:3588: finishing copy_from to smithi07134777-14 2024-09-11T01:53:23.314 INFO:tasks.rados.rados.0.smithi071.stdout:3588: got expected ENOENT (src dne) 2024-09-11T01:53:23.314 INFO:tasks.rados.rados.0.smithi071.stdout:3588: finishing copy_from racing read to smithi07134777-14 2024-09-11T01:53:23.314 INFO:tasks.rados.rados.0.smithi071.stdout:3586: finishing write tid 2 to smithi07134777-19 2024-09-11T01:53:23.314 INFO:tasks.rados.rados.0.smithi071.stdout:3586: finishing write tid 3 to smithi07134777-19 2024-09-11T01:53:23.314 INFO:tasks.rados.rados.0.smithi071.stdout:3586: finishing write tid 4 to smithi07134777-19 2024-09-11T01:53:23.314 INFO:tasks.rados.rados.0.smithi071.stdout:3586: finishing write tid 5 to smithi07134777-19 2024-09-11T01:53:23.314 INFO:tasks.rados.rados.0.smithi071.stdout:3586: oid 19 updating version 0 to 2100 2024-09-11T01:53:23.314 INFO:tasks.rados.rados.0.smithi071.stdout:3586: oid 19 version 2100 is already newer than 2099 2024-09-11T01:53:23.314 INFO:tasks.rados.rados.0.smithi071.stdout:3586: oid 19 version 2100 is already newer than 2097 2024-09-11T01:53:23.314 INFO:tasks.rados.rados.0.smithi071.stdout:3586: oid 19 version 2100 is already newer than 2098 2024-09-11T01:53:23.314 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 2100 (ObjNum 1239 snap 409 seq_num 1239) dirty exists 2024-09-11T01:53:23.314 INFO:tasks.rados.rados.0.smithi071.stdout:3586: left oid 19 (ObjNum 1239 snap 409 seq_num 1239) 2024-09-11T01:53:23.315 INFO:tasks.rados.rados.0.smithi071.stdout:3587: expect (ObjNum 1154 snap 366 seq_num 1154) 2024-09-11T01:53:23.344 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3380432 2024-09-11T01:53:23.390 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 653449 2024-09-11T01:53:23.430 INFO:tasks.rados.rados.0.smithi071.stdout:3591: finishing rollback tid 0 to smithi07134777-42 2024-09-11T01:53:23.430 INFO:tasks.rados.rados.0.smithi071.stdout:3591: finishing rollback tid 1 to smithi07134777-42 2024-09-11T01:53:23.430 INFO:tasks.rados.rados.0.smithi071.stdout:3591: finishing rollback tid 2 to smithi07134777-42 2024-09-11T01:53:23.430 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 2544 (ObjNum 1143 snap 364 seq_num 1143) dirty exists 2024-09-11T01:53:23.430 INFO:tasks.rados.rados.0.smithi071.stdout:3585: done (5 left) 2024-09-11T01:53:23.431 INFO:tasks.rados.rados.0.smithi071.stdout:3586: done (4 left) 2024-09-11T01:53:23.431 INFO:tasks.rados.rados.0.smithi071.stdout:3587: done (3 left) 2024-09-11T01:53:23.431 INFO:tasks.rados.rados.0.smithi071.stdout:3588: done (2 left) 2024-09-11T01:53:23.431 INFO:tasks.rados.rados.0.smithi071.stdout:3590: done (1 left) 2024-09-11T01:53:23.431 INFO:tasks.rados.rados.0.smithi071.stdout:3591: done (0 left) 2024-09-11T01:53:23.431 INFO:tasks.rados.rados.0.smithi071.stdout:3592: write oid 46 current snap is 409 2024-09-11T01:53:23.431 INFO:tasks.rados.rados.0.smithi071.stdout:3592: seq_num 1240 ranges {440680=674299,1711907=624424} 2024-09-11T01:53:23.435 INFO:tasks.rados.rados.0.smithi071.stdout:3592: writing smithi07134777-46 from 440680 to 1114979 tid 1 2024-09-11T01:53:23.442 INFO:tasks.rados.rados.0.smithi071.stdout:3592: writing smithi07134777-46 from 1711907 to 2336331 tid 2 2024-09-11T01:53:23.442 INFO:tasks.rados.rados.0.smithi071.stdout:3593: snap_create 2024-09-11T01:53:23.443 INFO:tasks.rados.rados.0.smithi071.stdout:3592: finishing write tid 1 to smithi07134777-46 2024-09-11T01:53:23.448 INFO:tasks.rados.rados.0.smithi071.stdout:3592: finishing write tid 2 to smithi07134777-46 2024-09-11T01:53:23.448 INFO:tasks.rados.rados.0.smithi071.stdout:3592: finishing write tid 3 to smithi07134777-46 2024-09-11T01:53:23.448 INFO:tasks.rados.rados.0.smithi071.stdout:3592: finishing write tid 4 to smithi07134777-46 2024-09-11T01:53:23.449 INFO:tasks.rados.rados.0.smithi071.stdout:3592: oid 46 updating version 0 to 2575 2024-09-11T01:53:23.449 INFO:tasks.rados.rados.0.smithi071.stdout:3592: oid 46 version 2575 is already newer than 2574 2024-09-11T01:53:23.449 INFO:tasks.rados.rados.0.smithi071.stdout:3592: oid 46 version 2575 is already newer than 2573 2024-09-11T01:53:23.449 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 2575 (ObjNum 1240 snap 409 seq_num 1240) dirty exists 2024-09-11T01:53:23.449 INFO:tasks.rados.rados.0.smithi071.stdout:3592: left oid 46 (ObjNum 1240 snap 409 seq_num 1240) 2024-09-11T01:53:24.353 INFO:tasks.rados.rados.0.smithi071.stdout:3592: done (1 left) 2024-09-11T01:53:24.353 INFO:tasks.rados.rados.0.smithi071.stdout:3593: done (0 left) 2024-09-11T01:53:24.353 INFO:tasks.rados.rados.0.smithi071.stdout:3594: rollback oid 38 current snap is 410 2024-09-11T01:53:24.353 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 38 to 366 2024-09-11T01:53:24.353 INFO:tasks.rados.rados.0.smithi071.stdout:3595: read oid 16 snap -1 2024-09-11T01:53:24.353 INFO:tasks.rados.rados.0.smithi071.stdout:3595: expect (ObjNum 1217 snap 397 seq_num 1217) 2024-09-11T01:53:24.353 INFO:tasks.rados.rados.0.smithi071.stdout:3596: copy_from oid 34 from oid 25 current snap is 410 2024-09-11T01:53:24.353 INFO:tasks.rados.rados.0.smithi071.stdout:3597: copy_from oid 20 from oid 7 current snap is 410 2024-09-11T01:53:24.354 INFO:tasks.rados.rados.0.smithi071.stdout:3598: read oid 49 snap -1 2024-09-11T01:53:24.354 INFO:tasks.rados.rados.0.smithi071.stdout:3598: expect (ObjNum 1162 snap 370 seq_num 1162) 2024-09-11T01:53:24.354 INFO:tasks.rados.rados.0.smithi071.stdout:3599: read oid 8 snap -1 2024-09-11T01:53:24.354 INFO:tasks.rados.rados.0.smithi071.stdout:3599: expect (ObjNum 1187 snap 378 seq_num 1187) 2024-09-11T01:53:24.354 INFO:tasks.rados.rados.0.smithi071.stdout:3600: write oid 35 current snap is 410 2024-09-11T01:53:24.354 INFO:tasks.rados.rados.0.smithi071.stdout:3600: seq_num 1241 ranges {457487=749548,1770389=484435,2746287=111200} 2024-09-11T01:53:24.359 INFO:tasks.rados.rados.0.smithi071.stdout:3600: writing smithi07134777-35 from 457487 to 1207035 tid 1 2024-09-11T01:53:24.365 INFO:tasks.rados.rados.0.smithi071.stdout:3600: writing smithi07134777-35 from 1770389 to 2254824 tid 2 2024-09-11T01:53:24.369 INFO:tasks.rados.rados.0.smithi071.stdout:3600: writing smithi07134777-35 from 2746287 to 2857487 tid 3 2024-09-11T01:53:24.369 INFO:tasks.rados.rados.0.smithi071.stdout:3601: snap_create 2024-09-11T01:53:24.370 INFO:tasks.rados.rados.0.smithi071.stdout:3599: expect (ObjNum 1187 snap 378 seq_num 1187) 2024-09-11T01:53:24.432 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3629200 2024-09-11T01:53:24.470 INFO:tasks.rados.rados.0.smithi071.stdout:3594: finishing rollback tid 0 to smithi07134777-38 2024-09-11T01:53:24.470 INFO:tasks.rados.rados.0.smithi071.stdout:3597: finishing copy_from to smithi07134777-20 2024-09-11T01:53:24.470 INFO:tasks.rados.rados.0.smithi071.stdout:3597: got expected ENOENT (src dne) 2024-09-11T01:53:24.470 INFO:tasks.rados.rados.0.smithi071.stdout:3594: finishing rollback tid 1 to smithi07134777-38 2024-09-11T01:53:24.470 INFO:tasks.rados.rados.0.smithi071.stdout:3597: finishing copy_from racing read to smithi07134777-20 2024-09-11T01:53:24.470 INFO:tasks.rados.rados.0.smithi071.stdout:3594: finishing rollback tid 2 to smithi07134777-38 2024-09-11T01:53:24.470 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 2747 (ObjNum 1150 snap 366 seq_num 1150) dirty exists 2024-09-11T01:53:24.470 INFO:tasks.rados.rados.0.smithi071.stdout:3598: expect (ObjNum 1162 snap 370 seq_num 1162) 2024-09-11T01:53:24.614 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 904646 2024-09-11T01:53:24.672 INFO:tasks.rados.rados.0.smithi071.stdout:3600: finishing write tid 1 to smithi07134777-35 2024-09-11T01:53:24.672 INFO:tasks.rados.rados.0.smithi071.stdout:3600: finishing write tid 2 to smithi07134777-35 2024-09-11T01:53:24.673 INFO:tasks.rados.rados.0.smithi071.stdout:3600: finishing write tid 3 to smithi07134777-35 2024-09-11T01:53:24.673 INFO:tasks.rados.rados.0.smithi071.stdout:3600: finishing write tid 4 to smithi07134777-35 2024-09-11T01:53:24.673 INFO:tasks.rados.rados.0.smithi071.stdout:3595: expect (ObjNum 1217 snap 397 seq_num 1217) 2024-09-11T01:53:24.721 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 712759 2024-09-11T01:53:24.759 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:53:24.805 INFO:tasks.rados.rados.0.smithi071.stdout:3600: finishing write tid 5 to smithi07134777-35 2024-09-11T01:53:24.805 INFO:tasks.rados.rados.0.smithi071.stdout:3600: oid 35 updating version 0 to 2105 2024-09-11T01:53:24.805 INFO:tasks.rados.rados.0.smithi071.stdout:3600: oid 35 version 2105 is already newer than 2102 2024-09-11T01:53:24.805 INFO:tasks.rados.rados.0.smithi071.stdout:3600: oid 35 version 2105 is already newer than 2103 2024-09-11T01:53:24.806 INFO:tasks.rados.rados.0.smithi071.stdout:3600: oid 35 version 2105 is already newer than 2104 2024-09-11T01:53:24.806 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 2105 (ObjNum 1241 snap 410 seq_num 1241) dirty exists 2024-09-11T01:53:24.806 INFO:tasks.rados.rados.0.smithi071.stdout:3600: left oid 35 (ObjNum 1241 snap 410 seq_num 1241) 2024-09-11T01:53:24.806 INFO:tasks.rados.rados.0.smithi071.stdout:3596: finishing copy_from racing read to smithi07134777-34 2024-09-11T01:53:24.806 INFO:tasks.rados.rados.0.smithi071.stdout:3596: finishing copy_from to smithi07134777-34 2024-09-11T01:53:24.806 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 2572 (ObjNum 1136 snap 360 seq_num 1136) dirty exists 2024-09-11T01:53:25.376 INFO:tasks.rados.rados.0.smithi071.stdout:3594: done (7 left) 2024-09-11T01:53:25.376 INFO:tasks.rados.rados.0.smithi071.stdout:3595: done (6 left) 2024-09-11T01:53:25.376 INFO:tasks.rados.rados.0.smithi071.stdout:3596: done (5 left) 2024-09-11T01:53:25.376 INFO:tasks.rados.rados.0.smithi071.stdout:3597: done (4 left) 2024-09-11T01:53:25.377 INFO:tasks.rados.rados.0.smithi071.stdout:3598: done (3 left) 2024-09-11T01:53:25.377 INFO:tasks.rados.rados.0.smithi071.stdout:3599: done (2 left) 2024-09-11T01:53:25.377 INFO:tasks.rados.rados.0.smithi071.stdout:3600: done (1 left) 2024-09-11T01:53:25.377 INFO:tasks.rados.rados.0.smithi071.stdout:3601: done (0 left) 2024-09-11T01:53:25.377 INFO:tasks.rados.rados.0.smithi071.stdout:3602: snap_remove snap 401 2024-09-11T01:53:26.375 INFO:tasks.rados.rados.0.smithi071.stdout:3602: done (0 left) 2024-09-11T01:53:26.375 INFO:tasks.rados.rados.0.smithi071.stdout:3603: copy_from oid 31 from oid 9 current snap is 411 2024-09-11T01:53:26.375 INFO:tasks.rados.rados.0.smithi071.stdout:3604: read oid 18 snap -1 2024-09-11T01:53:26.376 INFO:tasks.rados.rados.0.smithi071.stdout:3604: expect (ObjNum 618 snap 183 seq_num 618) 2024-09-11T01:53:26.376 INFO:tasks.rados.rados.0.smithi071.stdout:3605: read oid 38 snap -1 2024-09-11T01:53:26.376 INFO:tasks.rados.rados.0.smithi071.stdout:3605: expect (ObjNum 1150 snap 366 seq_num 1150) 2024-09-11T01:53:26.376 INFO:tasks.rados.rados.0.smithi071.stdout:3606: snap_remove snap 369 2024-09-11T01:53:27.394 INFO:tasks.rados.rados.0.smithi071.stdout:3606: done (3 left) 2024-09-11T01:53:27.394 INFO:tasks.rados.rados.0.smithi071.stdout:3607: snap_remove snap 399 2024-09-11T01:53:28.314 INFO:tasks.rados.rados.0.smithi071.stdout:3607: done (3 left) 2024-09-11T01:53:28.315 INFO:tasks.rados.rados.0.smithi071.stdout:3608: snap_remove snap 386 2024-09-11T01:53:29.320 INFO:tasks.rados.rados.0.smithi071.stdout:3605: expect (ObjNum 1150 snap 366 seq_num 1150) 2024-09-11T01:53:29.460 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 917591 2024-09-11T01:53:29.512 INFO:tasks.rados.rados.0.smithi071.stdout:3604: expect (ObjNum 618 snap 183 seq_num 618) 2024-09-11T01:53:29.512 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:53:29.672 INFO:tasks.rados.rados.0.smithi071.stdout:3603: finishing copy_from racing read to smithi07134777-31 2024-09-11T01:53:29.672 INFO:tasks.rados.rados.0.smithi071.stdout:3603: finishing copy_from to smithi07134777-31 2024-09-11T01:53:29.672 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 1967 (ObjNum 832 snap 254 seq_num 832) dirty exists 2024-09-11T01:53:29.672 INFO:tasks.rados.rados.0.smithi071.stdout:3603: done (3 left) 2024-09-11T01:53:29.672 INFO:tasks.rados.rados.0.smithi071.stdout:3604: done (2 left) 2024-09-11T01:53:29.672 INFO:tasks.rados.rados.0.smithi071.stdout:3605: done (1 left) 2024-09-11T01:53:29.672 INFO:tasks.rados.rados.0.smithi071.stdout:3608: done (0 left) 2024-09-11T01:53:29.672 INFO:tasks.rados.rados.0.smithi071.stdout:3609: write oid 17 current snap is 411 2024-09-11T01:53:29.672 INFO:tasks.rados.rados.0.smithi071.stdout:3609: seq_num 1242 ranges {474294=424797,1428871=628093,2657357=216937} 2024-09-11T01:53:29.676 INFO:tasks.rados.rados.0.smithi071.stdout:3609: writing smithi07134777-17 from 474294 to 899091 tid 1 2024-09-11T01:53:29.682 INFO:tasks.rados.rados.0.smithi071.stdout:3609: writing smithi07134777-17 from 1428871 to 2056964 tid 2 2024-09-11T01:53:29.687 INFO:tasks.rados.rados.0.smithi071.stdout:3609: writing smithi07134777-17 from 2657357 to 2874294 tid 3 2024-09-11T01:53:29.687 INFO:tasks.rados.rados.0.smithi071.stdout:3610: write oid 25 current snap is 411 2024-09-11T01:53:29.687 INFO:tasks.rados.rados.0.smithi071.stdout:3609: finishing write tid 1 to smithi07134777-17 2024-09-11T01:53:29.687 INFO:tasks.rados.rados.0.smithi071.stdout:3610: seq_num 1243 ranges {491101=500046,1487353=488104,2401133=489968} 2024-09-11T01:53:29.692 INFO:tasks.rados.rados.0.smithi071.stdout:3610: writing smithi07134777-25 from 491101 to 991147 tid 1 2024-09-11T01:53:29.697 INFO:tasks.rados.rados.0.smithi071.stdout:3610: writing smithi07134777-25 from 1487353 to 1975457 tid 2 2024-09-11T01:53:29.703 INFO:tasks.rados.rados.0.smithi071.stdout:3610: writing smithi07134777-25 from 2401133 to 2891101 tid 3 2024-09-11T01:53:29.703 INFO:tasks.rados.rados.0.smithi071.stdout:3611: rollback oid 48 current snap is 411 2024-09-11T01:53:29.703 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 48 to 400 2024-09-11T01:53:29.703 INFO:tasks.rados.rados.0.smithi071.stdout:3609: finishing write tid 2 to smithi07134777-17 2024-09-11T01:53:29.703 INFO:tasks.rados.rados.0.smithi071.stdout:3609: finishing write tid 3 to smithi07134777-17 2024-09-11T01:53:29.703 INFO:tasks.rados.rados.0.smithi071.stdout:3609: finishing write tid 4 to smithi07134777-17 2024-09-11T01:53:29.703 INFO:tasks.rados.rados.0.smithi071.stdout:3609: finishing write tid 5 to smithi07134777-17 2024-09-11T01:53:29.704 INFO:tasks.rados.rados.0.smithi071.stdout:3609: oid 17 updating version 0 to 2565 2024-09-11T01:53:29.704 INFO:tasks.rados.rados.0.smithi071.stdout:3609: oid 17 updating version 2565 to 2566 2024-09-11T01:53:29.704 INFO:tasks.rados.rados.0.smithi071.stdout:3609: oid 17 updating version 2566 to 2567 2024-09-11T01:53:29.704 INFO:tasks.rados.rados.0.smithi071.stdout:3609: oid 17 updating version 2567 to 2568 2024-09-11T01:53:29.704 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 2568 (ObjNum 1242 snap 411 seq_num 1242) dirty exists 2024-09-11T01:53:29.704 INFO:tasks.rados.rados.0.smithi071.stdout:3609: left oid 17 (ObjNum 1242 snap 411 seq_num 1242) 2024-09-11T01:53:29.704 INFO:tasks.rados.rados.0.smithi071.stdout:3610: finishing write tid 1 to smithi07134777-25 2024-09-11T01:53:29.704 INFO:tasks.rados.rados.0.smithi071.stdout:3610: finishing write tid 2 to smithi07134777-25 2024-09-11T01:53:29.704 INFO:tasks.rados.rados.0.smithi071.stdout:3609: done (2 left) 2024-09-11T01:53:29.704 INFO:tasks.rados.rados.0.smithi071.stdout:3612: read oid 34 snap -1 2024-09-11T01:53:29.704 INFO:tasks.rados.rados.0.smithi071.stdout:3612: expect (ObjNum 1136 snap 360 seq_num 1136) 2024-09-11T01:53:29.704 INFO:tasks.rados.rados.0.smithi071.stdout:3613: delete oid 35 current snap is 411 2024-09-11T01:53:29.707 INFO:tasks.rados.rados.0.smithi071.stdout:3613: done (3 left) 2024-09-11T01:53:29.707 INFO:tasks.rados.rados.0.smithi071.stdout:3614: write oid 21 current snap is 411 2024-09-11T01:53:29.707 INFO:tasks.rados.rados.0.smithi071.stdout:3614: seq_num 1245 ranges {524715=650544,1887964=491773} 2024-09-11T01:53:29.714 INFO:tasks.rados.rados.0.smithi071.stdout:3614: writing smithi07134777-21 from 524715 to 1175259 tid 1 2024-09-11T01:53:29.721 INFO:tasks.rados.rados.0.smithi071.stdout:3614: writing smithi07134777-21 from 1887964 to 2379737 tid 2 2024-09-11T01:53:29.721 INFO:tasks.rados.rados.0.smithi071.stdout:3615: write (excl) oid 22 current snap is 411 2024-09-11T01:53:29.721 INFO:tasks.rados.rados.0.smithi071.stdout:3611: finishing rollback tid 0 to smithi07134777-48 2024-09-11T01:53:29.721 INFO:tasks.rados.rados.0.smithi071.stdout:3615: seq_num 1246 ranges {541522=725793,1946446=635431} 2024-09-11T01:53:29.728 INFO:tasks.rados.rados.0.smithi071.stdout:3615: writing smithi07134777-22 from 541522 to 1267315 tid 1 2024-09-11T01:53:29.736 INFO:tasks.rados.rados.0.smithi071.stdout:3615: writing smithi07134777-22 from 1946446 to 2581877 tid 2 2024-09-11T01:53:29.736 INFO:tasks.rados.rados.0.smithi071.stdout:3616: snap_create 2024-09-11T01:53:29.736 INFO:tasks.rados.rados.0.smithi071.stdout:3611: finishing rollback tid 1 to smithi07134777-48 2024-09-11T01:53:29.736 INFO:tasks.rados.rados.0.smithi071.stdout:3610: finishing write tid 3 to smithi07134777-25 2024-09-11T01:53:29.736 INFO:tasks.rados.rados.0.smithi071.stdout:3611: finishing rollback tid 2 to smithi07134777-48 2024-09-11T01:53:29.736 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 2771 (ObjNum 1178 snap 376 seq_num 1178) dirty exists 2024-09-11T01:53:29.736 INFO:tasks.rados.rados.0.smithi071.stdout:3610: finishing write tid 4 to smithi07134777-25 2024-09-11T01:53:29.736 INFO:tasks.rados.rados.0.smithi071.stdout:3610: finishing write tid 5 to smithi07134777-25 2024-09-11T01:53:29.737 INFO:tasks.rados.rados.0.smithi071.stdout:3610: oid 25 updating version 0 to 2575 2024-09-11T01:53:29.737 INFO:tasks.rados.rados.0.smithi071.stdout:3610: oid 25 updating version 2575 to 2576 2024-09-11T01:53:29.737 INFO:tasks.rados.rados.0.smithi071.stdout:3610: oid 25 version 2576 is already newer than 2574 2024-09-11T01:53:29.737 INFO:tasks.rados.rados.0.smithi071.stdout:3610: oid 25 version 2576 is already newer than 2573 2024-09-11T01:53:29.737 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 2576 (ObjNum 1243 snap 411 seq_num 1243) dirty exists 2024-09-11T01:53:29.737 INFO:tasks.rados.rados.0.smithi071.stdout:3610: left oid 25 (ObjNum 1243 snap 411 seq_num 1243) 2024-09-11T01:53:29.737 INFO:tasks.rados.rados.0.smithi071.stdout:3612: expect (ObjNum 1136 snap 360 seq_num 1136) 2024-09-11T01:53:29.792 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:53:29.827 INFO:tasks.rados.rados.0.smithi071.stdout:3614: finishing write tid 1 to smithi07134777-21 2024-09-11T01:53:29.827 INFO:tasks.rados.rados.0.smithi071.stdout:3614: finishing write tid 2 to smithi07134777-21 2024-09-11T01:53:29.827 INFO:tasks.rados.rados.0.smithi071.stdout:3614: finishing write tid 3 to smithi07134777-21 2024-09-11T01:53:29.827 INFO:tasks.rados.rados.0.smithi071.stdout:3614: finishing write tid 4 to smithi07134777-21 2024-09-11T01:53:29.827 INFO:tasks.rados.rados.0.smithi071.stdout:3614: oid 21 updating version 0 to 2276 2024-09-11T01:53:29.827 INFO:tasks.rados.rados.0.smithi071.stdout:3614: oid 21 updating version 2276 to 2278 2024-09-11T01:53:29.827 INFO:tasks.rados.rados.0.smithi071.stdout:3614: oid 21 version 2278 is already newer than 2277 2024-09-11T01:53:29.827 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 2278 (ObjNum 1245 snap 411 seq_num 1245) dirty exists 2024-09-11T01:53:29.827 INFO:tasks.rados.rados.0.smithi071.stdout:3614: left oid 21 (ObjNum 1245 snap 411 seq_num 1245) 2024-09-11T01:53:29.827 INFO:tasks.rados.rados.0.smithi071.stdout:3615: finishing write tid 1 to smithi07134777-22 2024-09-11T01:53:29.827 INFO:tasks.rados.rados.0.smithi071.stdout:3615: finishing write tid 2 to smithi07134777-22 2024-09-11T01:53:29.827 INFO:tasks.rados.rados.0.smithi071.stdout:3615: finishing write tid 3 to smithi07134777-22 2024-09-11T01:53:29.827 INFO:tasks.rados.rados.0.smithi071.stdout:3615: finishing write tid 4 to smithi07134777-22 2024-09-11T01:53:29.828 INFO:tasks.rados.rados.0.smithi071.stdout:3615: oid 22 updating version 0 to 2434 2024-09-11T01:53:29.828 INFO:tasks.rados.rados.0.smithi071.stdout:3615: oid 22 version 2434 is already newer than 2433 2024-09-11T01:53:29.828 INFO:tasks.rados.rados.0.smithi071.stdout:3615: oid 22 version 2434 is already newer than 2432 2024-09-11T01:53:29.828 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 2434 (ObjNum 1246 snap 411 seq_num 1246) dirty exists 2024-09-11T01:53:29.828 INFO:tasks.rados.rados.0.smithi071.stdout:3615: left oid 22 (ObjNum 1246 snap 411 seq_num 1246) 2024-09-11T01:53:30.099 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-11T01:53:30.100 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-11T01:53:30.100 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-11T01:53:30.102 INFO:tasks.ceph.mon.b:Started 2024-09-11T01:53:30.102 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:53:30.103 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:53:30.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.213+0000 7f6236c39640 10 monclient: get_monmap_and_config 2024-09-11T01:53:30.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.213+0000 7f6236c39640 10 monclient: build_initial_monmap 2024-09-11T01:53:30.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.213+0000 7f6236c39640 10 monclient: monmap: 2024-09-11T01:53:30.219 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:30.219 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:53:30.214397+0000 2024-09-11T01:53:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:53:30.214397+0000 2024-09-11T01:53:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:53:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:53:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:53:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:53:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:53:30.220 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.214+0000 7f6236c39640 1 Processor -- start 2024-09-11T01:53:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.214+0000 7f6236c39640 1 -- start start 2024-09-11T01:53:30.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.214+0000 7f6236c39640 10 monclient: init 2024-09-11T01:53:30.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.214+0000 7f6236c39640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:30.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.214+0000 7f6236c39640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-11T01:53:30.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.214+0000 7f6236c39640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f62300a2e90 0x7f623006dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:30.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f6236c39640 10 monclient(hunting): picked mon.noname-a con 0x7f62300a2e90 addr v2:172.21.15.12:3300/0 2024-09-11T01:53:30.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f6236c39640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f62300a3270 0x7f62300a3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:30.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f6236c39640 10 monclient(hunting): picked mon.noname-b con 0x7f62300a3270 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:53:30.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f6236c39640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f6230138040 0x7f6230138490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:30.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f6236c39640 10 monclient(hunting): picked mon.noname-c con 0x7f6230138040 addr v2:172.21.15.71:3300/0 2024-09-11T01:53:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f6236c39640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f6236c39640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f62300a4900 con 0x7f62300a2e90 2024-09-11T01:53:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f6236c39640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f6236c39640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f62300a46c0 con 0x7f6230138040 2024-09-11T01:53:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f6236c39640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f6236c39640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f623006ee50 con 0x7f62300a3270 2024-09-11T01:53:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f6236c39640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f6236c39640 10 monclient(hunting): authenticate will time out at 2216.657057s 2024-09-11T01:53:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62349d5640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f62300a2e90 0x7f623006dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f622ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f62300a3270 0x7f62300a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f622ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f62300a3270 0x7f62300a3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:33950/0 (socket says 172.21.15.12:33950) 2024-09-11T01:53:30.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62349d5640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f62300a2e90 0x7f623006dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48356/0 (socket says 172.21.15.12:48356) 2024-09-11T01:53:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f622ffff640 1 -- 172.21.15.12:0/2438467913 learned_addr learned my addr 172.21.15.12:0/2438467913 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:53:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f622ffff640 10 monclient(hunting): get_auth_request con 0x7f62300a3270 auth_method 0 2024-09-11T01:53:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f622ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f622ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f622ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62349d5640 10 monclient(hunting): get_auth_request con 0x7f62300a2e90 auth_method 0 2024-09-11T01:53:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62349d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62349d5640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62349d5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f622ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f622ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62351d6640 1 --2- 172.21.15.12:0/2438467913 >> v2:172.21.15.71:3300/0 conn(0x7f6230138040 0x7f6230138490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f622ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:30.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62351d6640 10 monclient(hunting): get_auth_request con 0x7f6230138040 auth_method 0 2024-09-11T01:53:30.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62351d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:30.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62351d6640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:30.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62351d6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:30.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62349d5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:30.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62349d5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:30.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62349d5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:30.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62351d6640 1 -- 172.21.15.12:0/2438467913 >> v2:172.21.15.71:3300/0 conn(0x7f6230138040 msgr2=0x7f6230138490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:53:30.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62351d6640 1 -- 172.21.15.12:0/2438467913 >> v2:172.21.15.71:3300/0 conn(0x7f6230138040 msgr2=0x7f6230138490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:53:30.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62351d6640 1 --2- 172.21.15.12:0/2438467913 >> v2:172.21.15.71:3300/0 conn(0x7f6230138040 0x7f6230138490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:53:30.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62351d6640 1 --2- 172.21.15.12:0/2438467913 >> v2:172.21.15.71:3300/0 conn(0x7f6230138040 0x7f6230138490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:53:30.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62349d5640 10 monclient(hunting): handle_auth_done global_id 264198 payload 274 2024-09-11T01:53:30.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62349d5640 1 -- 172.21.15.12:0/2438467913 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f62300a3270 msgr2=0x7f62300a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:53:30.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62349d5640 1 --2- 172.21.15.12:0/2438467913 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f62300a3270 0x7f62300a3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:30.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62349d5640 1 -- 172.21.15.12:0/2438467913 >> v2:172.21.15.71:3300/0 conn(0x7f6230138040 msgr2=0x7f6230138490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:53:30.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62349d5640 1 --2- 172.21.15.12:0/2438467913 >> v2:172.21.15.71:3300/0 conn(0x7f6230138040 0x7f6230138490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:30.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62349d5640 10 monclient: _finish_hunting 0 2024-09-11T01:53:30.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62349d5640 1 monclient: found mon.noname-a 2024-09-11T01:53:30.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62349d5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:30.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62349d5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:53:30.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f62349d5640 1 -- 172.21.15.12:0/2438467913 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f623006eb70 con 0x7f62300a2e90 2024-09-11T01:53:30.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.215+0000 7f622ffff640 1 --2- 172.21.15.12:0/2438467913 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f62300a3270 0x7f62300a3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:53:30.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.218+0000 7f62349d5640 1 -- 172.21.15.12:0/2438467913 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62300a2e90 msgr2=0x7f623006dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:30.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.218+0000 7f62349d5640 1 --2- 172.21.15.12:0/2438467913 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62300a2e90 0x7f623006dc60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f623006f340 tx=0x7f6224029fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb08fcfb0a1154c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:30.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.218+0000 7f622f7fe640 1 -- 172.21.15.12:0/2438467913 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6224037070 con 0x7f62300a2e90 2024-09-11T01:53:30.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.218+0000 7f622f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:30.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.218+0000 7f622f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:53:30.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.218+0000 7f622f7fe640 10 monclient: dump: 2024-09-11T01:53:30.231 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:30.232 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:30.232 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:30.232 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:30.232 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:30.232 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:30.232 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:30.232 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:30.232 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:30.232 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:30.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.218+0000 7f622f7fe640 10 monclient: _finish_auth 0 2024-09-11T01:53:30.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.218+0000 7f622f7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:53:30.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.218+0000 7f622f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:30.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.218+0000 7f622f7fe640 1 -- 172.21.15.12:0/2438467913 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6224032440 con 0x7f62300a2e90 2024-09-11T01:53:30.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.218+0000 7f622f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:30.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.218+0000 7f622f7fe640 1 -- 172.21.15.12:0/2438467913 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6224032810 con 0x7f62300a2e90 2024-09-11T01:53:30.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.218+0000 7f622f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:30.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.218+0000 7f622f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:53:30.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.218+0000 7f622f7fe640 10 monclient: dump: 2024-09-11T01:53:30.233 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:30.233 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:30.233 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:30.233 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:30.234 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:30.234 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:30.234 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:30.234 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:30.234 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:30.234 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:30.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.218+0000 7f6236c39640 5 monclient: authenticate success, global_id 264198 2024-09-11T01:53:30.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.218+0000 7f6236c39640 10 monclient: get_monmap_and_config success 2024-09-11T01:53:30.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.218+0000 7f6236c39640 4 set_mon_vals no callback set 2024-09-11T01:53:30.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.219+0000 7f6236c39640 10 monclient: shutdown 2024-09-11T01:53:30.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.219+0000 7f6236c39640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:30.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.219+0000 7f6236c39640 1 -- 172.21.15.12:0/2438467913 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62300a2e90 msgr2=0x7f623006dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:30.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.219+0000 7f6236c39640 1 --2- 172.21.15.12:0/2438467913 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62300a2e90 0x7f623006dc60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f623006f340 tx=0x7f6224029fe0 comp rx=0 tx=0).stop 2024-09-11T01:53:30.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.219+0000 7f6236c39640 1 -- 172.21.15.12:0/2438467913 shutdown_connections 2024-09-11T01:53:30.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.219+0000 7f6236c39640 1 --2- 172.21.15.12:0/2438467913 >> v2:172.21.15.71:3300/0 conn(0x7f6230138040 0x7f6230138490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:30.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.219+0000 7f6236c39640 1 --2- 172.21.15.12:0/2438467913 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f62300a3270 0x7f62300a3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:30.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.219+0000 7f6236c39640 1 --2- 172.21.15.12:0/2438467913 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62300a2e90 0x7f623006dc60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:30.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.219+0000 7f6236c39640 1 -- 172.21.15.12:0/2438467913 >> 172.21.15.12:0/2438467913 conn(0x7f62300a8490 msgr2=0x7f62300a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:30.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.219+0000 7f6236c39640 1 -- 172.21.15.12:0/2438467913 shutdown_connections 2024-09-11T01:53:30.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.219+0000 7f6236c39640 1 -- 172.21.15.12:0/2438467913 wait complete. 2024-09-11T01:53:30.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.219+0000 7f6236c39640 10 monclient: build_initial_monmap 2024-09-11T01:53:30.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.219+0000 7f6236c39640 10 monclient: monmap: 2024-09-11T01:53:30.236 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:30.236 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:53:30.236 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:53:30.236 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:53:30.236 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:30.236 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:30.236 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:53:30.236 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:53:30.237 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:53:30.237 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:30.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.220+0000 7f6236c39640 1 Processor -- start 2024-09-11T01:53:30.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.220+0000 7f6236c39640 1 -- start start 2024-09-11T01:53:30.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.220+0000 7f6236c39640 10 monclient: init 2024-09-11T01:53:30.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.220+0000 7f6236c39640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:30.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f6236c39640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:53:30.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f6236c39640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f62300a2e90 0x7f62301247c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:30.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f6236c39640 10 monclient(hunting): picked mon.noname-c con 0x7f62300a2e90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:53:30.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f6236c39640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62300a3270 0x7f6230124d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:30.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f6236c39640 10 monclient(hunting): picked mon.noname-a con 0x7f62300a3270 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:53:30.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f6236c39640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6230138040 0x7f6230125240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:30.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f6236c39640 10 monclient(hunting): picked mon.noname-b con 0x7f6230138040 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:53:30.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f6236c39640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:30.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f6236c39640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f623006fba0 con 0x7f62300a3270 2024-09-11T01:53:30.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f6236c39640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:30.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f6236c39640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f62300a3bc0 con 0x7f6230138040 2024-09-11T01:53:30.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f6236c39640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:30.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f6236c39640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f623006fd20 con 0x7f62300a2e90 2024-09-11T01:53:30.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f6236c39640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:30.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f6236c39640 10 monclient(hunting): authenticate will time out at 2216.663126s 2024-09-11T01:53:30.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f62349d5640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f62300a2e90 0x7f62301247c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:30.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f622ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62300a3270 0x7f6230124d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:30.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f62349d5640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f62300a2e90 0x7f62301247c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:33966/0 (socket says 172.21.15.12:33966) 2024-09-11T01:53:30.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f62349d5640 1 -- 172.21.15.12:0/1665111583 learned_addr learned my addr 172.21.15.12:0/1665111583 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:53:30.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f622ffff640 10 monclient(hunting): get_auth_request con 0x7f62300a3270 auth_method 0 2024-09-11T01:53:30.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f622ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:30.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f622ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:30.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f622ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:30.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f62349d5640 10 monclient(hunting): get_auth_request con 0x7f62300a2e90 auth_method 0 2024-09-11T01:53:30.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f62349d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:30.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f62349d5640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:30.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f62349d5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:30.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f622ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:30.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f622ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:30.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f622ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:30.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f62349d5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:30.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f62349d5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:30.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f62349d5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:30.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f62351d6640 1 --2- 172.21.15.12:0/1665111583 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6230138040 0x7f6230125240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:30.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f622ffff640 10 monclient(hunting): handle_auth_done global_id 264201 payload 824 2024-09-11T01:53:30.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f622ffff640 1 -- 172.21.15.12:0/1665111583 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f62300a2e90 msgr2=0x7f62301247c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:30.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f622ffff640 1 --2- 172.21.15.12:0/1665111583 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f62300a2e90 0x7f62301247c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:30.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f622ffff640 1 -- 172.21.15.12:0/1665111583 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6230138040 msgr2=0x7f6230125240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:30.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f622ffff640 1 --2- 172.21.15.12:0/1665111583 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6230138040 0x7f6230125240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:30.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f622ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:53:30.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f622ffff640 1 monclient: found mon.noname-a 2024-09-11T01:53:30.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f622ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:30.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f622ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:53:30.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f622ffff640 1 -- 172.21.15.12:0/1665111583 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f623009f680 con 0x7f62300a3270 2024-09-11T01:53:30.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f62351d6640 10 monclient: get_auth_request con 0x7f6230138040 auth_method 0 2024-09-11T01:53:30.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f62351d6640 1 --2- 172.21.15.12:0/1665111583 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6230138040 0x7f6230125240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:53:30.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.221+0000 7f62349d5640 1 --2- 172.21.15.12:0/1665111583 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f62300a2e90 0x7f62301247c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:53:30.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.223+0000 7f622ffff640 1 -- 172.21.15.12:0/1665111583 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62300a3270 msgr2=0x7f6230124d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:30.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.223+0000 7f622ffff640 1 --2- 172.21.15.12:0/1665111583 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62300a3270 0x7f6230124d00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f621c00e900 tx=0x7f621c00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:30.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.223+0000 7f622d7fa640 1 -- 172.21.15.12:0/1665111583 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c00cc10 con 0x7f62300a3270 2024-09-11T01:53:30.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.223+0000 7f622d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:30.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.223+0000 7f622d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:53:30.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.223+0000 7f622d7fa640 10 monclient: dump: 2024-09-11T01:53:30.241 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:30.241 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:30.241 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:30.241 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:30.241 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:30.242 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:30.242 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:30.242 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:30.242 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:30.242 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:30.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.223+0000 7f622d7fa640 10 monclient: _finish_auth 0 2024-09-11T01:53:30.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.223+0000 7f622d7fa640 10 monclient: _check_auth_tickets 2024-09-11T01:53:30.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.223+0000 7f622d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:30.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.223+0000 7f622d7fa640 1 -- 172.21.15.12:0/1665111583 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f621c00cdb0 con 0x7f62300a3270 2024-09-11T01:53:30.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.223+0000 7f6236c39640 5 monclient: authenticate success, global_id 264201 2024-09-11T01:53:30.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.223+0000 7f6236c39640 10 monclient: _renew_subs 2024-09-11T01:53:30.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.223+0000 7f6236c39640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:30.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.223+0000 7f6236c39640 1 -- 172.21.15.12:0/1665111583 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6230128300 con 0x7f62300a3270 2024-09-11T01:53:30.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.224+0000 7f622d7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:30.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.224+0000 7f622d7fa640 1 -- 172.21.15.12:0/1665111583 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c004250 con 0x7f62300a3270 2024-09-11T01:53:30.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.224+0000 7f622d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:30.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.224+0000 7f622d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:53:30.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.224+0000 7f622d7fa640 10 monclient: dump: 2024-09-11T01:53:30.243 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:30.243 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:30.243 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:30.243 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:30.243 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:30.243 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:30.243 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:30.243 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:30.243 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:30.243 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:30.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.224+0000 7f622effd640 4 set_mon_vals no callback set 2024-09-11T01:53:30.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.224+0000 7f6236c39640 10 monclient: _renew_subs 2024-09-11T01:53:30.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.224+0000 7f6236c39640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:30.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.224+0000 7f6236c39640 1 -- 172.21.15.12:0/1665111583 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6230073610 con 0x7f62300a3270 2024-09-11T01:53:30.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.225+0000 7f622d7fa640 1 -- 172.21.15.12:0/1665111583 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f621c023060 con 0x7f62300a3270 2024-09-11T01:53:30.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.225+0000 7f6236c39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:30.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.225+0000 7f6236c39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:30.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.225+0000 7f6236c39640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:30.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.225+0000 7f6236c39640 1 -- 172.21.15.12:0/1665111583 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62300a3680 con 0x7f62300a3270 2024-09-11T01:53:30.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.225+0000 7f622d7fa640 1 --2- 172.21.15.12:0/1665111583 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f621403e210 0x7f62140406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:30.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.225+0000 7f622d7fa640 1 -- 172.21.15.12:0/1665111583 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(824..824 src has 676..824) ==== 4815+0+0 (secure 0 0 0) 0x7f621c052aa0 con 0x7f62300a3270 2024-09-11T01:53:30.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.225+0000 7f62349d5640 1 --2- 172.21.15.12:0/1665111583 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f621403e210 0x7f62140406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:30.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.225+0000 7f62349d5640 10 monclient: get_auth_request con 0x7f621403e210 auth_method 0 2024-09-11T01:53:30.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.228+0000 7f62349d5640 1 --2- 172.21.15.12:0/1665111583 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f621403e210 0x7f62140406d0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f6224009560 tx=0x7f622402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:30.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.229+0000 7f622d7fa640 1 -- 172.21.15.12:0/1665111583 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f623009f680 con 0x7f62300a3270 2024-09-11T01:53:30.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.229+0000 7f622d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:30.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.229+0000 7f622d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:53:30.396 INFO:tasks.rados.rados.0.smithi071.stdout:3610: done (5 left) 2024-09-11T01:53:30.396 INFO:tasks.rados.rados.0.smithi071.stdout:3611: done (4 left) 2024-09-11T01:53:30.396 INFO:tasks.rados.rados.0.smithi071.stdout:3612: done (3 left) 2024-09-11T01:53:30.396 INFO:tasks.rados.rados.0.smithi071.stdout:3614: done (2 left) 2024-09-11T01:53:30.396 INFO:tasks.rados.rados.0.smithi071.stdout:3615: done (1 left) 2024-09-11T01:53:30.396 INFO:tasks.rados.rados.0.smithi071.stdout:3616: done (0 left) 2024-09-11T01:53:30.397 INFO:tasks.rados.rados.0.smithi071.stdout:3617: write oid 15 current snap is 412 2024-09-11T01:53:30.397 INFO:tasks.rados.rados.0.smithi071.stdout:3617: seq_num 1247 ranges {558329=517395,1721281=779089} 2024-09-11T01:53:30.402 INFO:tasks.rados.rados.0.smithi071.stdout:3617: writing smithi07134777-15 from 558329 to 1075724 tid 1 2024-09-11T01:53:30.410 INFO:tasks.rados.rados.0.smithi071.stdout:3617: writing smithi07134777-15 from 1721281 to 2500370 tid 2 2024-09-11T01:53:30.410 INFO:tasks.rados.rados.0.smithi071.stdout:3618: read oid 30 snap -1 2024-09-11T01:53:30.410 INFO:tasks.rados.rados.0.smithi071.stdout:3618: expect (ObjNum 1217 snap 397 seq_num 1217) 2024-09-11T01:53:30.410 INFO:tasks.rados.rados.0.smithi071.stdout:3619: write oid 7 current snap is 412 2024-09-11T01:53:30.410 INFO:tasks.rados.rados.0.smithi071.stdout:3617: finishing write tid 1 to smithi07134777-15 2024-09-11T01:53:30.410 INFO:tasks.rados.rados.0.smithi071.stdout:3619: seq_num 1248 ranges {575136=592644,1663410=639100,2705542=269594} 2024-09-11T01:53:30.417 INFO:tasks.rados.rados.0.smithi071.stdout:3619: writing smithi07134777-7 from 575136 to 1167780 tid 1 2024-09-11T01:53:30.423 INFO:tasks.rados.rados.0.smithi071.stdout:3619: writing smithi07134777-7 from 1663410 to 2302510 tid 2 2024-09-11T01:53:30.427 INFO:tasks.rados.rados.0.smithi071.stdout:3619: writing smithi07134777-7 from 2705542 to 2975136 tid 3 2024-09-11T01:53:30.427 INFO:tasks.rados.rados.0.smithi071.stdout:3620: snap_remove snap 397 2024-09-11T01:53:30.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.490+0000 7f6236c39640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:53:30.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.490+0000 7f6236c39640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:53:30.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.490+0000 7f6236c39640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:30.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.490+0000 7f6236c39640 1 -- 172.21.15.12:0/1665111583 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f62300698e0 con 0x7f62300a3270 2024-09-11T01:53:30.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.490+0000 7f622d7fa640 1 -- 172.21.15.12:0/1665111583 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f621c017020 con 0x7f62300a3270 2024-09-11T01:53:30.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.490+0000 7f622d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:53:30.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.490+0000 7f622d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:53:30.492 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":224,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:53:30.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.492+0000 7f6236c39640 1 -- 172.21.15.12:0/1665111583 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f621403e210 msgr2=0x7f62140406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:30.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.492+0000 7f6236c39640 1 --2- 172.21.15.12:0/1665111583 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f621403e210 0x7f62140406d0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f6224009560 tx=0x7f622402e000 comp rx=0 tx=0).stop 2024-09-11T01:53:30.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.492+0000 7f6236c39640 10 monclient: shutdown 2024-09-11T01:53:30.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.492+0000 7f6236c39640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:30.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.492+0000 7f6236c39640 1 -- 172.21.15.12:0/1665111583 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62300a3270 msgr2=0x7f6230124d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:30.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.492+0000 7f6236c39640 1 --2- 172.21.15.12:0/1665111583 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62300a3270 0x7f6230124d00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f621c00e900 tx=0x7f621c00edc0 comp rx=0 tx=0).stop 2024-09-11T01:53:30.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.492+0000 7f6236c39640 1 -- 172.21.15.12:0/1665111583 shutdown_connections 2024-09-11T01:53:30.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.492+0000 7f6236c39640 1 --2- 172.21.15.12:0/1665111583 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6230138040 0x7f6230125240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:30.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.492+0000 7f6236c39640 1 --2- 172.21.15.12:0/1665111583 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f621403e210 0x7f62140406d0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:30.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.492+0000 7f6236c39640 1 --2- 172.21.15.12:0/1665111583 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62300a3270 0x7f6230124d00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:30.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.492+0000 7f6236c39640 1 --2- 172.21.15.12:0/1665111583 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f62300a2e90 0x7f62301247c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:30.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.492+0000 7f6236c39640 1 -- 172.21.15.12:0/1665111583 >> 172.21.15.12:0/1665111583 conn(0x7f62300a8490 msgr2=0x7f6230072090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:30.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.492+0000 7f6236c39640 1 -- 172.21.15.12:0/1665111583 shutdown_connections 2024-09-11T01:53:30.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:30.492+0000 7f6236c39640 1 -- 172.21.15.12:0/1665111583 wait complete. 2024-09-11T01:53:31.402 INFO:tasks.rados.rados.0.smithi071.stdout:3620: done (3 left) 2024-09-11T01:53:31.402 INFO:tasks.rados.rados.0.smithi071.stdout:3621: snap_remove snap 398 2024-09-11T01:53:32.402 INFO:tasks.rados.rados.0.smithi071.stdout:3621: done (3 left) 2024-09-11T01:53:32.402 INFO:tasks.rados.rados.0.smithi071.stdout:3622: write (excl) oid 44 current snap is 412 2024-09-11T01:53:32.402 INFO:tasks.rados.rados.0.smithi071.stdout:3622: seq_num 1249 ranges {591943=667893,1721892=782758} 2024-09-11T01:53:32.409 INFO:tasks.rados.rados.0.smithi071.stdout:3622: writing smithi07134777-44 from 591943 to 1259836 tid 1 2024-09-11T01:53:32.417 INFO:tasks.rados.rados.0.smithi071.stdout:3622: writing smithi07134777-44 from 1721892 to 2504650 tid 2 2024-09-11T01:53:32.417 INFO:tasks.rados.rados.0.smithi071.stdout:3623: snap_create 2024-09-11T01:53:32.417 INFO:tasks.rados.rados.0.smithi071.stdout:3617: finishing write tid 2 to smithi07134777-15 2024-09-11T01:53:32.417 INFO:tasks.rados.rados.0.smithi071.stdout:3617: finishing write tid 3 to smithi07134777-15 2024-09-11T01:53:32.417 INFO:tasks.rados.rados.0.smithi071.stdout:3617: finishing write tid 4 to smithi07134777-15 2024-09-11T01:53:32.417 INFO:tasks.rados.rados.0.smithi071.stdout:3617: oid 15 updating version 0 to 2773 2024-09-11T01:53:32.417 INFO:tasks.rados.rados.0.smithi071.stdout:3617: oid 15 version 2773 is already newer than 2772 2024-09-11T01:53:32.417 INFO:tasks.rados.rados.0.smithi071.stdout:3617: oid 15 updating version 2773 to 2774 2024-09-11T01:53:32.418 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 2774 (ObjNum 1247 snap 412 seq_num 1247) dirty exists 2024-09-11T01:53:32.418 INFO:tasks.rados.rados.0.smithi071.stdout:3617: left oid 15 (ObjNum 1247 snap 412 seq_num 1247) 2024-09-11T01:53:32.418 INFO:tasks.rados.rados.0.smithi071.stdout:3618: expect (ObjNum 1217 snap 397 seq_num 1217) 2024-09-11T01:53:32.507 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:53:32.554 INFO:tasks.rados.rados.0.smithi071.stdout:3619: finishing write tid 1 to smithi07134777-7 2024-09-11T01:53:32.554 INFO:tasks.rados.rados.0.smithi071.stdout:3619: finishing write tid 2 to smithi07134777-7 2024-09-11T01:53:32.554 INFO:tasks.rados.rados.0.smithi071.stdout:3619: finishing write tid 3 to smithi07134777-7 2024-09-11T01:53:32.554 INFO:tasks.rados.rados.0.smithi071.stdout:3619: finishing write tid 4 to smithi07134777-7 2024-09-11T01:53:32.554 INFO:tasks.rados.rados.0.smithi071.stdout:3619: finishing write tid 5 to smithi07134777-7 2024-09-11T01:53:32.554 INFO:tasks.rados.rados.0.smithi071.stdout:3619: oid 7 updating version 0 to 2778 2024-09-11T01:53:32.554 INFO:tasks.rados.rados.0.smithi071.stdout:3619: oid 7 version 2778 is already newer than 2776 2024-09-11T01:53:32.554 INFO:tasks.rados.rados.0.smithi071.stdout:3619: oid 7 version 2778 is already newer than 2777 2024-09-11T01:53:32.554 INFO:tasks.rados.rados.0.smithi071.stdout:3619: oid 7 version 2778 is already newer than 2775 2024-09-11T01:53:32.555 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 2778 (ObjNum 1248 snap 412 seq_num 1248) dirty exists 2024-09-11T01:53:32.555 INFO:tasks.rados.rados.0.smithi071.stdout:3619: left oid 7 (ObjNum 1248 snap 412 seq_num 1248) 2024-09-11T01:53:32.555 INFO:tasks.rados.rados.0.smithi071.stdout:3622: finishing write tid 1 to smithi07134777-44 2024-09-11T01:53:32.555 INFO:tasks.rados.rados.0.smithi071.stdout:3622: finishing write tid 2 to smithi07134777-44 2024-09-11T01:53:32.555 INFO:tasks.rados.rados.0.smithi071.stdout:3622: finishing write tid 3 to smithi07134777-44 2024-09-11T01:53:32.555 INFO:tasks.rados.rados.0.smithi071.stdout:3622: finishing write tid 4 to smithi07134777-44 2024-09-11T01:53:32.555 INFO:tasks.rados.rados.0.smithi071.stdout:3622: oid 44 updating version 0 to 1258 2024-09-11T01:53:32.555 INFO:tasks.rados.rados.0.smithi071.stdout:3622: oid 44 updating version 1258 to 1263 2024-09-11T01:53:32.555 INFO:tasks.rados.rados.0.smithi071.stdout:3622: oid 44 updating version 1263 to 1264 2024-09-11T01:53:32.555 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 1264 (ObjNum 1249 snap 412 seq_num 1249) dirty exists 2024-09-11T01:53:32.555 INFO:tasks.rados.rados.0.smithi071.stdout:3622: left oid 44 (ObjNum 1249 snap 412 seq_num 1249) 2024-09-11T01:53:33.315 INFO:tasks.rados.rados.0.smithi071.stdout:3617: done (4 left) 2024-09-11T01:53:33.315 INFO:tasks.rados.rados.0.smithi071.stdout:3618: done (3 left) 2024-09-11T01:53:33.315 INFO:tasks.rados.rados.0.smithi071.stdout:3619: done (2 left) 2024-09-11T01:53:33.315 INFO:tasks.rados.rados.0.smithi071.stdout:3622: done (1 left) 2024-09-11T01:53:33.315 INFO:tasks.rados.rados.0.smithi071.stdout:3623: done (0 left) 2024-09-11T01:53:33.315 INFO:tasks.rados.rados.0.smithi071.stdout:3624: snap_create 2024-09-11T01:53:33.505 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:53:33.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.616+0000 7f8505a9b640 10 monclient: get_monmap_and_config 2024-09-11T01:53:33.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.616+0000 7f8505a9b640 10 monclient: build_initial_monmap 2024-09-11T01:53:33.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.616+0000 7f8505a9b640 10 monclient: monmap: 2024-09-11T01:53:33.618 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:33.618 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:33.618 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:53:33.617052+0000 2024-09-11T01:53:33.618 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:53:33.617052+0000 2024-09-11T01:53:33.618 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:33.618 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:33.618 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:53:33.618 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:53:33.618 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:53:33.618 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:53:33.619 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:53:33.619 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:33.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f8505a9b640 1 Processor -- start 2024-09-11T01:53:33.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f8505a9b640 1 -- start start 2024-09-11T01:53:33.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f8505a9b640 10 monclient: init 2024-09-11T01:53:33.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f8505a9b640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:33.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f8505a9b640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-11T01:53:33.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f8505a9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f850009d790 0x7f850009db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:33.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f8505a9b640 10 monclient(hunting): picked mon.noname-a con 0x7f850009d790 addr v2:172.21.15.12:3300/0 2024-09-11T01:53:33.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f8505a9b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f850009e0b0 0x7f850009e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:33.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f8505a9b640 10 monclient(hunting): picked mon.noname-c con 0x7f850009e0b0 addr v2:172.21.15.71:3300/0 2024-09-11T01:53:33.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f8505a9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8500134a60 0x7f8500134ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:33.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f8505a9b640 10 monclient(hunting): picked mon.noname-b con 0x7f8500134a60 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:53:33.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f8505a9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:33.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f8505a9b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f85000a1da0 con 0x7f850009d790 2024-09-11T01:53:33.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f8505a9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:33.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f8505a9b640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f85000a0760 con 0x7f850009e0b0 2024-09-11T01:53:33.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f8505a9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:33.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f8505a9b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f85000a1fa0 con 0x7f8500134a60 2024-09-11T01:53:33.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f8505a9b640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:33.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f8505a9b640 10 monclient(hunting): authenticate will time out at 2220.059725s 2024-09-11T01:53:33.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f84fffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8500134a60 0x7f8500134ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:33.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f84ff7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f850009d790 0x7f850009db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:33.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f84ff7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f850009d790 0x7f850009db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48404/0 (socket says 172.21.15.12:48404) 2024-09-11T01:53:33.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f84fffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8500134a60 0x7f8500134ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:33986/0 (socket says 172.21.15.12:33986) 2024-09-11T01:53:33.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f84ff7fe640 1 -- 172.21.15.12:0/4098298753 learned_addr learned my addr 172.21.15.12:0/4098298753 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:53:33.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f84ff7fe640 10 monclient(hunting): get_auth_request con 0x7f850009d790 auth_method 0 2024-09-11T01:53:33.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f84ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:33.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f84ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:33.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f84ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:33.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f84fffff640 10 monclient(hunting): get_auth_request con 0x7f8500134a60 auth_method 0 2024-09-11T01:53:33.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f84fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f84fffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.617+0000 7f84fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.618+0000 7f84fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.618+0000 7f84fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.618+0000 7f84fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.618+0000 7f84ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.618+0000 7f84ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.618+0000 7f84ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.618+0000 7f84feffd640 1 --2- 172.21.15.12:0/4098298753 >> v2:172.21.15.71:3300/0 conn(0x7f850009e0b0 0x7f850009e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.618+0000 7f84feffd640 10 monclient(hunting): get_auth_request con 0x7f850009e0b0 auth_method 0 2024-09-11T01:53:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.618+0000 7f84feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.618+0000 7f84feffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:33.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.618+0000 7f84feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.618+0000 7f84fffff640 10 monclient(hunting): handle_auth_done global_id 244400 payload 274 2024-09-11T01:53:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.618+0000 7f84fffff640 1 -- 172.21.15.12:0/4098298753 >> v2:172.21.15.71:3300/0 conn(0x7f850009e0b0 msgr2=0x7f850009e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:53:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.618+0000 7f84fffff640 1 --2- 172.21.15.12:0/4098298753 >> v2:172.21.15.71:3300/0 conn(0x7f850009e0b0 0x7f850009e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.618+0000 7f84fffff640 1 -- 172.21.15.12:0/4098298753 >> v2:172.21.15.12:3300/0 conn(0x7f850009d790 msgr2=0x7f850009db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:53:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.618+0000 7f84fffff640 1 --2- 172.21.15.12:0/4098298753 >> v2:172.21.15.12:3300/0 conn(0x7f850009d790 0x7f850009db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.618+0000 7f84fffff640 10 monclient: _finish_hunting 0 2024-09-11T01:53:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.618+0000 7f84fffff640 1 monclient: found mon.noname-b 2024-09-11T01:53:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.618+0000 7f84fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.618+0000 7f84fffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:53:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.618+0000 7f84fffff640 1 -- 172.21.15.12:0/4098298753 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f850009eda0 con 0x7f8500134a60 2024-09-11T01:53:33.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.618+0000 7f84ff7fe640 1 --2- 172.21.15.12:0/4098298753 >> v2:172.21.15.12:3300/0 conn(0x7f850009d790 0x7f850009db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:53:33.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f84fffff640 1 -- 172.21.15.12:0/4098298753 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8500134a60 msgr2=0x7f8500134ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:33.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f84fffff640 1 --2- 172.21.15.12:0/4098298753 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8500134a60 0x7f8500134ec0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f850009eca0 tx=0x7f84f8029fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2973f855d39a3708 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:33.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f84fe7fc640 1 -- 172.21.15.12:0/4098298753 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f8037070 con 0x7f8500134a60 2024-09-11T01:53:33.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f84fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:33.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f84fe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:53:33.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f84fe7fc640 10 monclient: dump: 2024-09-11T01:53:33.632 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:33.632 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:33.632 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:33.632 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:33.632 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:33.632 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:33.632 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:33.632 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:33.632 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:33.632 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:33.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f84fe7fc640 10 monclient: _finish_auth 0 2024-09-11T01:53:33.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f84fe7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:53:33.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f84fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:33.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f84fe7fc640 1 -- 172.21.15.12:0/4098298753 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84f8032440 con 0x7f8500134a60 2024-09-11T01:53:33.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f8505a9b640 5 monclient: authenticate success, global_id 244400 2024-09-11T01:53:33.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f8505a9b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:53:33.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f84fe7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:33.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f84fe7fc640 1 -- 172.21.15.12:0/4098298753 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f8032810 con 0x7f8500134a60 2024-09-11T01:53:33.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f8505a9b640 10 monclient: get_monmap_and_config success 2024-09-11T01:53:33.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f8505a9b640 4 set_mon_vals no callback set 2024-09-11T01:53:33.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f84fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:33.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f84fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:53:33.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f84fe7fc640 10 monclient: dump: 2024-09-11T01:53:33.633 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:33.633 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:33.634 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:33.634 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:33.634 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:33.634 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:33.634 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:33.634 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:33.634 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:33.634 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:33.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f8505a9b640 10 monclient: shutdown 2024-09-11T01:53:33.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f8505a9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:33.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f8505a9b640 1 -- 172.21.15.12:0/4098298753 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8500134a60 msgr2=0x7f8500134ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:33.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f8505a9b640 1 --2- 172.21.15.12:0/4098298753 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8500134a60 0x7f8500134ec0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f850009eca0 tx=0x7f84f8029fe0 comp rx=0 tx=0).stop 2024-09-11T01:53:33.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f8505a9b640 1 -- 172.21.15.12:0/4098298753 shutdown_connections 2024-09-11T01:53:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f8505a9b640 1 --2- 172.21.15.12:0/4098298753 >> v2:172.21.15.71:3300/0 conn(0x7f850009e0b0 0x7f850009e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f8505a9b640 1 --2- 172.21.15.12:0/4098298753 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8500134a60 0x7f8500134ec0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f8505a9b640 1 --2- 172.21.15.12:0/4098298753 >> v2:172.21.15.12:3300/0 conn(0x7f850009d790 0x7f850009db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f8505a9b640 1 -- 172.21.15.12:0/4098298753 >> 172.21.15.12:0/4098298753 conn(0x7f85000a2d90 msgr2=0x7f85000a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f8505a9b640 1 -- 172.21.15.12:0/4098298753 shutdown_connections 2024-09-11T01:53:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.621+0000 7f8505a9b640 1 -- 172.21.15.12:0/4098298753 wait complete. 2024-09-11T01:53:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.622+0000 7f8505a9b640 10 monclient: build_initial_monmap 2024-09-11T01:53:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.622+0000 7f8505a9b640 10 monclient: monmap: 2024-09-11T01:53:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:53:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:53:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:53:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:33.635 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:33.636 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:53:33.636 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:53:33.636 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:53:33.636 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:33.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.622+0000 7f8505a9b640 1 Processor -- start 2024-09-11T01:53:33.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f8505a9b640 1 -- start start 2024-09-11T01:53:33.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f8505a9b640 10 monclient: init 2024-09-11T01:53:33.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f8505a9b640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:33.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f8505a9b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:53:33.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f8505a9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f850009d790 0x7f85001caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:33.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f8505a9b640 10 monclient(hunting): picked mon.noname-c con 0x7f850009d790 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:53:33.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f8505a9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f850009e0b0 0x7f85001cb410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:33.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f8505a9b640 10 monclient(hunting): picked mon.noname-a con 0x7f850009e0b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:53:33.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f8505a9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8500134a60 0x7f85001cb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:33.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f8505a9b640 10 monclient(hunting): picked mon.noname-b con 0x7f8500134a60 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:53:33.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f8505a9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:33.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f8505a9b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f850009ce20 con 0x7f850009e0b0 2024-09-11T01:53:33.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f8505a9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:33.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f8505a9b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f850009cca0 con 0x7f8500134a60 2024-09-11T01:53:33.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f8505a9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:33.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f8505a9b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8500135400 con 0x7f850009d790 2024-09-11T01:53:33.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f8505a9b640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:33.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f8505a9b640 10 monclient(hunting): authenticate will time out at 2220.065719s 2024-09-11T01:53:33.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f84ff7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f850009d790 0x7f85001caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:33.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f84ff7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f850009d790 0x7f85001caed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:33998/0 (socket says 172.21.15.12:33998) 2024-09-11T01:53:33.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f84feffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f850009e0b0 0x7f85001cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:33.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f84ff7fe640 1 -- 172.21.15.12:0/1276634182 learned_addr learned my addr 172.21.15.12:0/1276634182 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:53:33.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f84ff7fe640 10 monclient(hunting): get_auth_request con 0x7f850009d790 auth_method 0 2024-09-11T01:53:33.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f84ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:33.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f84ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:33.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f84ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:33.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f84feffd640 10 monclient(hunting): get_auth_request con 0x7f850009e0b0 auth_method 0 2024-09-11T01:53:33.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f84feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:33.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f84feffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:33.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f84feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:33.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f84ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:33.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f84ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:33.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f84ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:33.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f84feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:33.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f84feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:33.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.623+0000 7f84feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:33.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.624+0000 7f84ff7fe640 10 monclient(hunting): handle_auth_done global_id 244403 payload 824 2024-09-11T01:53:33.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.624+0000 7f84ff7fe640 1 -- 172.21.15.12:0/1276634182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8500134a60 msgr2=0x7f85001cb950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:33.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.624+0000 7f84ff7fe640 1 --2- 172.21.15.12:0/1276634182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8500134a60 0x7f85001cb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:33.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.624+0000 7f84ff7fe640 1 -- 172.21.15.12:0/1276634182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f850009e0b0 msgr2=0x7f85001cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:33.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.624+0000 7f84ff7fe640 1 --2- 172.21.15.12:0/1276634182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f850009e0b0 0x7f85001cb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:33.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.624+0000 7f84ff7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:53:33.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.624+0000 7f84ff7fe640 1 monclient: found mon.noname-c 2024-09-11T01:53:33.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.624+0000 7f84ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:33.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.624+0000 7f84ff7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:53:33.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.624+0000 7f84ff7fe640 1 -- 172.21.15.12:0/1276634182 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f850011d8e0 con 0x7f850009d790 2024-09-11T01:53:33.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.624+0000 7f84feffd640 1 --2- 172.21.15.12:0/1276634182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f850009e0b0 0x7f85001cb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:53:33.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.626+0000 7f84ff7fe640 1 -- 172.21.15.12:0/1276634182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f850009d790 msgr2=0x7f85001caed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:33.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.626+0000 7f84ff7fe640 1 --2- 172.21.15.12:0/1276634182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f850009d790 0x7f85001caed0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f84f000e9d0 tx=0x7f84f000ee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:33.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.626+0000 7f84e7fff640 1 -- 172.21.15.12:0/1276634182 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f0017070 con 0x7f850009d790 2024-09-11T01:53:33.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.626+0000 7f84e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:33.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.626+0000 7f84e7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:53:33.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.626+0000 7f84e7fff640 10 monclient: dump: 2024-09-11T01:53:33.640 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:33.640 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:33.640 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:33.640 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:33.640 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:33.640 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:33.640 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:33.640 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:33.640 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:33.640 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:33.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.626+0000 7f84e7fff640 10 monclient: _finish_auth 0 2024-09-11T01:53:33.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.626+0000 7f84e7fff640 10 monclient: _check_auth_tickets 2024-09-11T01:53:33.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.626+0000 7f84e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:33.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.626+0000 7f84e7fff640 1 -- 172.21.15.12:0/1276634182 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84f000cbf0 con 0x7f850009d790 2024-09-11T01:53:33.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.626+0000 7f8505a9b640 5 monclient: authenticate success, global_id 244403 2024-09-11T01:53:33.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.626+0000 7f8505a9b640 10 monclient: _renew_subs 2024-09-11T01:53:33.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.626+0000 7f8505a9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:53:33.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.626+0000 7f8505a9b640 1 -- 172.21.15.12:0/1276634182 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8500134870 con 0x7f850009d790 2024-09-11T01:53:33.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.626+0000 7f84e7fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:33.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.626+0000 7f84e7fff640 1 -- 172.21.15.12:0/1276634182 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f0013470 con 0x7f850009d790 2024-09-11T01:53:33.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.626+0000 7f84e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:33.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.626+0000 7f84e7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:53:33.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.626+0000 7f84e7fff640 10 monclient: dump: 2024-09-11T01:53:33.641 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:33.642 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.626+0000 7f8505a9b640 10 monclient: _renew_subs 2024-09-11T01:53:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.626+0000 7f8505a9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:53:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.626+0000 7f84fdffb640 4 set_mon_vals no callback set 2024-09-11T01:53:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.626+0000 7f8505a9b640 1 -- 172.21.15.12:0/1276634182 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f850013a9f0 con 0x7f850009d790 2024-09-11T01:53:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.627+0000 7f8505a9b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:33.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.627+0000 7f8505a9b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.628+0000 7f8505a9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:53:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.628+0000 7f84e7fff640 1 -- 172.21.15.12:0/1276634182 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f84f0024060 con 0x7f850009d790 2024-09-11T01:53:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.628+0000 7f8505a9b640 1 -- 172.21.15.12:0/1276634182 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f850009e500 con 0x7f850009d790 2024-09-11T01:53:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.628+0000 7f84e7fff640 1 --2- 172.21.15.12:0/1276634182 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f84ec03e290 0x7f84ec040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.628+0000 7f84e7fff640 1 -- 172.21.15.12:0/1276634182 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(828..828 src has 676..828) ==== 4799+0+0 (secure 0 0 0) 0x7f84f0052be0 con 0x7f850009d790 2024-09-11T01:53:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.628+0000 7f84feffd640 1 --2- 172.21.15.12:0/1276634182 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f84ec03e290 0x7f84ec040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:33.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.628+0000 7f84feffd640 10 monclient: get_auth_request con 0x7f84ec03e290 auth_method 0 2024-09-11T01:53:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.631+0000 7f84feffd640 1 --2- 172.21.15.12:0/1276634182 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f84ec03e290 0x7f84ec040750 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f84f4009fd0 tx=0x7f84f40098d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.633+0000 7f84e7fff640 1 -- 172.21.15.12:0/1276634182 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f84f00515d0 con 0x7f850009d790 2024-09-11T01:53:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.633+0000 7f84e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:33.644 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.633+0000 7f84e7fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:53:33.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.889+0000 7f8505a9b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:53:33.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.889+0000 7f8505a9b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:53:33.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.889+0000 7f8505a9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:53:33.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.889+0000 7f8505a9b640 1 -- 172.21.15.12:0/1276634182 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8500073740 con 0x7f850009d790 2024-09-11T01:53:33.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.890+0000 7f84e7fff640 1 -- 172.21.15.12:0/1276634182 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f84f0023d00 con 0x7f850009d790 2024-09-11T01:53:33.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.890+0000 7f84e7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:53:33.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.890+0000 7f84e7fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:53:33.894 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":224,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:53:33.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.892+0000 7f8505a9b640 1 -- 172.21.15.12:0/1276634182 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f84ec03e290 msgr2=0x7f84ec040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:33.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.892+0000 7f8505a9b640 1 --2- 172.21.15.12:0/1276634182 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f84ec03e290 0x7f84ec040750 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f84f4009fd0 tx=0x7f84f40098d0 comp rx=0 tx=0).stop 2024-09-11T01:53:33.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.892+0000 7f8505a9b640 10 monclient: shutdown 2024-09-11T01:53:33.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.892+0000 7f8505a9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:33.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.892+0000 7f8505a9b640 1 -- 172.21.15.12:0/1276634182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f850009d790 msgr2=0x7f85001caed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:33.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.892+0000 7f8505a9b640 1 --2- 172.21.15.12:0/1276634182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f850009d790 0x7f85001caed0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f84f000e9d0 tx=0x7f84f000ee90 comp rx=0 tx=0).stop 2024-09-11T01:53:33.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.892+0000 7f8505a9b640 1 -- 172.21.15.12:0/1276634182 shutdown_connections 2024-09-11T01:53:33.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.892+0000 7f8505a9b640 1 --2- 172.21.15.12:0/1276634182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8500134a60 0x7f85001cb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:33.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.892+0000 7f8505a9b640 1 --2- 172.21.15.12:0/1276634182 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f84ec03e290 0x7f84ec040750 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:33.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.892+0000 7f8505a9b640 1 --2- 172.21.15.12:0/1276634182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f850009e0b0 0x7f85001cb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:33.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.892+0000 7f8505a9b640 1 --2- 172.21.15.12:0/1276634182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f850009d790 0x7f85001caed0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:33.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.892+0000 7f8505a9b640 1 -- 172.21.15.12:0/1276634182 >> 172.21.15.12:0/1276634182 conn(0x7f85000a2d90 msgr2=0x7f850013b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:33.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.892+0000 7f8505a9b640 1 -- 172.21.15.12:0/1276634182 shutdown_connections 2024-09-11T01:53:33.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:33.892+0000 7f8505a9b640 1 -- 172.21.15.12:0/1276634182 wait complete. 2024-09-11T01:53:34.315 INFO:tasks.rados.rados.0.smithi071.stdout:3624: done (0 left) 2024-09-11T01:53:34.316 INFO:tasks.rados.rados.0.smithi071.stdout:3625: copy_from oid 18 from oid 9 current snap is 414 2024-09-11T01:53:34.316 INFO:tasks.rados.rados.0.smithi071.stdout:3626: snap_remove snap 407 2024-09-11T01:53:35.317 INFO:tasks.rados.rados.0.smithi071.stdout:3626: done (1 left) 2024-09-11T01:53:35.317 INFO:tasks.rados.rados.0.smithi071.stdout:3627: rollback oid 35 current snap is 414 2024-09-11T01:53:35.317 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 35 to 325 2024-09-11T01:53:35.317 INFO:tasks.rados.rados.0.smithi071.stdout:3628: write oid 12 current snap is 414 2024-09-11T01:53:35.317 INFO:tasks.rados.rados.0.smithi071.stdout:3628: seq_num 1250 ranges {608750=743142,1780374=642769} 2024-09-11T01:53:35.324 INFO:tasks.rados.rados.0.smithi071.stdout:3628: writing smithi07134777-12 from 608750 to 1351892 tid 1 2024-09-11T01:53:35.332 INFO:tasks.rados.rados.0.smithi071.stdout:3628: writing smithi07134777-12 from 1780374 to 2423143 tid 2 2024-09-11T01:53:35.332 INFO:tasks.rados.rados.0.smithi071.stdout:3629: write (excl) oid 23 current snap is 414 2024-09-11T01:53:35.332 INFO:tasks.rados.rados.0.smithi071.stdout:3625: finishing copy_from racing read to smithi07134777-18 2024-09-11T01:53:35.332 INFO:tasks.rados.rados.0.smithi071.stdout:3625: finishing copy_from to smithi07134777-18 2024-09-11T01:53:35.332 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 1967 (ObjNum 832 snap 254 seq_num 832) dirty exists 2024-09-11T01:53:35.332 INFO:tasks.rados.rados.0.smithi071.stdout:3627: finishing rollback tid 1 to smithi07134777-35 2024-09-11T01:53:35.332 INFO:tasks.rados.rados.0.smithi071.stdout:3627: finishing rollback tid 2 to smithi07134777-35 2024-09-11T01:53:35.332 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 2108 (ObjNum 949 snap 300 seq_num 949) dirty exists 2024-09-11T01:53:35.332 INFO:tasks.rados.rados.0.smithi071.stdout:3629: seq_num 1251 ranges {625557=418391,1838856=786427} 2024-09-11T01:53:35.337 INFO:tasks.rados.rados.0.smithi071.stdout:3629: writing smithi07134777-23 from 625557 to 1043948 tid 1 2024-09-11T01:53:35.346 INFO:tasks.rados.rados.0.smithi071.stdout:3629: writing smithi07134777-23 from 1838856 to 2625283 tid 2 2024-09-11T01:53:35.346 INFO:tasks.rados.rados.0.smithi071.stdout:3625: done (3 left) 2024-09-11T01:53:35.346 INFO:tasks.rados.rados.0.smithi071.stdout:3627: done (2 left) 2024-09-11T01:53:35.347 INFO:tasks.rados.rados.0.smithi071.stdout:3630: snap_create 2024-09-11T01:53:35.347 INFO:tasks.rados.rados.0.smithi071.stdout:3628: finishing write tid 1 to smithi07134777-12 2024-09-11T01:53:35.347 INFO:tasks.rados.rados.0.smithi071.stdout:3628: finishing write tid 2 to smithi07134777-12 2024-09-11T01:53:35.347 INFO:tasks.rados.rados.0.smithi071.stdout:3628: finishing write tid 3 to smithi07134777-12 2024-09-11T01:53:35.347 INFO:tasks.rados.rados.0.smithi071.stdout:3628: finishing write tid 4 to smithi07134777-12 2024-09-11T01:53:35.347 INFO:tasks.rados.rados.0.smithi071.stdout:3628: oid 12 updating version 0 to 2650 2024-09-11T01:53:35.347 INFO:tasks.rados.rados.0.smithi071.stdout:3628: oid 12 updating version 2650 to 2651 2024-09-11T01:53:35.347 INFO:tasks.rados.rados.0.smithi071.stdout:3628: oid 12 version 2651 is already newer than 2649 2024-09-11T01:53:35.347 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 2651 (ObjNum 1250 snap 414 seq_num 1250) dirty exists 2024-09-11T01:53:35.347 INFO:tasks.rados.rados.0.smithi071.stdout:3628: left oid 12 (ObjNum 1250 snap 414 seq_num 1250) 2024-09-11T01:53:35.347 INFO:tasks.rados.rados.0.smithi071.stdout:3629: finishing write tid 1 to smithi07134777-23 2024-09-11T01:53:35.350 INFO:tasks.rados.rados.0.smithi071.stdout:3629: finishing write tid 2 to smithi07134777-23 2024-09-11T01:53:35.350 INFO:tasks.rados.rados.0.smithi071.stdout:3629: finishing write tid 3 to smithi07134777-23 2024-09-11T01:53:35.351 INFO:tasks.rados.rados.0.smithi071.stdout:3629: finishing write tid 4 to smithi07134777-23 2024-09-11T01:53:35.351 INFO:tasks.rados.rados.0.smithi071.stdout:3629: oid 23 updating version 0 to 1268 2024-09-11T01:53:35.351 INFO:tasks.rados.rados.0.smithi071.stdout:3629: oid 23 version 1268 is already newer than 1266 2024-09-11T01:53:35.351 INFO:tasks.rados.rados.0.smithi071.stdout:3629: oid 23 version 1268 is already newer than 1267 2024-09-11T01:53:35.351 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 1268 (ObjNum 1251 snap 414 seq_num 1251) dirty exists 2024-09-11T01:53:35.351 INFO:tasks.rados.rados.0.smithi071.stdout:3629: left oid 23 (ObjNum 1251 snap 414 seq_num 1251) 2024-09-11T01:53:36.421 INFO:tasks.rados.rados.0.smithi071.stdout:3628: done (2 left) 2024-09-11T01:53:36.421 INFO:tasks.rados.rados.0.smithi071.stdout:3629: done (1 left) 2024-09-11T01:53:36.422 INFO:tasks.rados.rados.0.smithi071.stdout:3630: done (0 left) 2024-09-11T01:53:36.422 INFO:tasks.rados.rados.0.smithi071.stdout:3631: read oid 11 snap -1 2024-09-11T01:53:36.422 INFO:tasks.rados.rados.0.smithi071.stdout:3631: expect (ObjNum 1115 snap 356 seq_num 1115) 2024-09-11T01:53:36.422 INFO:tasks.rados.rados.0.smithi071.stdout:3632: write oid 19 current snap is 415 2024-09-11T01:53:36.422 INFO:tasks.rados.rados.0.smithi071.stdout:3632: seq_num 1252 ranges {642364=493640,1897338=646438} 2024-09-11T01:53:36.427 INFO:tasks.rados.rados.0.smithi071.stdout:3632: writing smithi07134777-19 from 642364 to 1136004 tid 1 2024-09-11T01:53:36.435 INFO:tasks.rados.rados.0.smithi071.stdout:3632: writing smithi07134777-19 from 1897338 to 2543776 tid 2 2024-09-11T01:53:36.435 INFO:tasks.rados.rados.0.smithi071.stdout:3633: delete oid 15 current snap is 415 2024-09-11T01:53:36.435 INFO:tasks.rados.rados.0.smithi071.stdout:3631: expect (ObjNum 1115 snap 356 seq_num 1115) 2024-09-11T01:53:36.480 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:53:36.565 INFO:tasks.rados.rados.0.smithi071.stdout:3632: finishing write tid 1 to smithi07134777-19 2024-09-11T01:53:36.565 INFO:tasks.rados.rados.0.smithi071.stdout:3632: finishing write tid 2 to smithi07134777-19 2024-09-11T01:53:36.565 INFO:tasks.rados.rados.0.smithi071.stdout:3632: finishing write tid 3 to smithi07134777-19 2024-09-11T01:53:36.565 INFO:tasks.rados.rados.0.smithi071.stdout:3632: finishing write tid 4 to smithi07134777-19 2024-09-11T01:53:36.565 INFO:tasks.rados.rados.0.smithi071.stdout:3632: oid 19 updating version 0 to 2103 2024-09-11T01:53:36.566 INFO:tasks.rados.rados.0.smithi071.stdout:3632: oid 19 version 2103 is already newer than 2101 2024-09-11T01:53:36.566 INFO:tasks.rados.rados.0.smithi071.stdout:3632: oid 19 version 2103 is already newer than 2102 2024-09-11T01:53:36.566 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 2103 (ObjNum 1252 snap 415 seq_num 1252) dirty exists 2024-09-11T01:53:36.566 INFO:tasks.rados.rados.0.smithi071.stdout:3632: left oid 19 (ObjNum 1252 snap 415 seq_num 1252) 2024-09-11T01:53:36.566 INFO:tasks.rados.rados.0.smithi071.stdout:3631: done (2 left) 2024-09-11T01:53:36.566 INFO:tasks.rados.rados.0.smithi071.stdout:3632: done (1 left) 2024-09-11T01:53:36.566 INFO:tasks.rados.rados.0.smithi071.stdout:3633: done (0 left) 2024-09-11T01:53:36.566 INFO:tasks.rados.rados.0.smithi071.stdout:3634: snap_remove snap 403 2024-09-11T01:53:36.904 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:53:37.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.016+0000 7fbdcd7da640 10 monclient: get_monmap_and_config 2024-09-11T01:53:37.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.016+0000 7fbdcd7da640 10 monclient: build_initial_monmap 2024-09-11T01:53:37.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.016+0000 7fbdcd7da640 10 monclient: monmap: 2024-09-11T01:53:37.018 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:37.018 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.018 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:53:37.016881+0000 2024-09-11T01:53:37.018 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:53:37.016881+0000 2024-09-11T01:53:37.018 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:37.018 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.018 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:53:37.018 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:53:37.018 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:53:37.018 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:53:37.018 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:53:37.018 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.016+0000 7fbdcd7da640 1 Processor -- start 2024-09-11T01:53:37.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.016+0000 7fbdcd7da640 1 -- start start 2024-09-11T01:53:37.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.016+0000 7fbdcd7da640 10 monclient: init 2024-09-11T01:53:37.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdcd7da640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:37.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdcd7da640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-11T01:53:37.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdcd7da640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbdc8138210 0x7fbdc81385f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdcd7da640 10 monclient(hunting): picked mon.noname-b con 0x7fbdc8138210 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:53:37.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdcd7da640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbdc80a6810 addr v1:172.21.15.71:6789/0 2024-09-11T01:53:37.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdcd7da640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbdc80a57a0 addr v1:172.21.15.12:6789/0 2024-09-11T01:53:37.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdcd7da640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdc809ba20 con 0x7fbdc80a57a0 2024-09-11T01:53:37.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdcd7da640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdc80a6590 con 0x7fbdc80a6810 2024-09-11T01:53:37.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdcd7da640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdcd7da640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fbdc809c7a0 con 0x7fbdc8138210 2024-09-11T01:53:37.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdcd7da640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:37.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdcd7da640 10 monclient(hunting): authenticate will time out at 2223.459565s 2024-09-11T01:53:37.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdc6ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbdc8138210 0x7fbdc81385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdc6ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbdc8138210 0x7fbdc81385f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42850/0 (socket says 172.21.15.12:42850) 2024-09-11T01:53:37.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdc6ffd640 1 -- 172.21.15.12:0/4033168417 learned_addr learned my addr 172.21.15.12:0/4033168417 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:53:37.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdc6ffd640 10 monclient(hunting): get_auth_request con 0x7fbdc8138210 auth_method 0 2024-09-11T01:53:37.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdc6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdc6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdc6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:37.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdc6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:37.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdc6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:37.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdc6ffd640 10 monclient(hunting): handle_auth_done global_id 244406 payload 274 2024-09-11T01:53:37.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdc6ffd640 1 -- 172.21.15.12:0/4033168417 >> v1:172.21.15.71:6789/0 conn(0x7fbdc80a6810 legacy=0x7fbdc80a5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:53:37.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdc6ffd640 1 -- 172.21.15.12:0/4033168417 >> v1:172.21.15.12:6789/0 conn(0x7fbdc80a57a0 legacy=0x7fbdc8140120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:53:37.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdc6ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:53:37.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdc6ffd640 1 monclient: found mon.noname-b 2024-09-11T01:53:37.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:37.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdc6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:53:37.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.017+0000 7fbdc6ffd640 1 -- 172.21.15.12:0/4033168417 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdc809c1e0 con 0x7fbdc8138210 2024-09-11T01:53:37.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.020+0000 7fbdc6ffd640 1 -- 172.21.15.12:0/4033168417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbdc8138210 msgr2=0x7fbdc81385f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:37.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.020+0000 7fbdc6ffd640 1 --2- 172.21.15.12:0/4033168417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbdc8138210 0x7fbdc81385f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbdb80098a0 tx=0x7fbdb802a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34bc8384b5cbccd7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:37.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.020+0000 7fbdc5ffb640 1 -- 172.21.15.12:0/4033168417 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb8037070 con 0x7fbdc8138210 2024-09-11T01:53:37.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.020+0000 7fbdc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.020+0000 7fbdc5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:53:37.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.020+0000 7fbdc5ffb640 10 monclient: dump: 2024-09-11T01:53:37.027 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.027 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.028 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.028 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.028 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.028 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.028 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.028 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.028 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.028 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.021+0000 7fbdc5ffb640 10 monclient: _finish_auth 0 2024-09-11T01:53:37.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.021+0000 7fbdc5ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:53:37.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.021+0000 7fbdc5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:37.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.021+0000 7fbdcd7da640 5 monclient: authenticate success, global_id 244406 2024-09-11T01:53:37.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.021+0000 7fbdc5ffb640 1 -- 172.21.15.12:0/4033168417 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdb8032640 con 0x7fbdc8138210 2024-09-11T01:53:37.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.021+0000 7fbdcd7da640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:53:37.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.021+0000 7fbdc5ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:37.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.021+0000 7fbdcd7da640 10 monclient: get_monmap_and_config success 2024-09-11T01:53:37.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.021+0000 7fbdcd7da640 4 set_mon_vals no callback set 2024-09-11T01:53:37.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.021+0000 7fbdc5ffb640 1 -- 172.21.15.12:0/4033168417 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb8032a10 con 0x7fbdc8138210 2024-09-11T01:53:37.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.021+0000 7fbdc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.021+0000 7fbdc5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:53:37.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.021+0000 7fbdc5ffb640 10 monclient: dump: 2024-09-11T01:53:37.029 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.029 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.029 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.029 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.029 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.030 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.030 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.030 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.030 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.030 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.021+0000 7fbdcd7da640 10 monclient: shutdown 2024-09-11T01:53:37.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.021+0000 7fbdcd7da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:37.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.021+0000 7fbdcd7da640 1 -- 172.21.15.12:0/4033168417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbdc8138210 msgr2=0x7fbdc81385f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.021+0000 7fbdcd7da640 1 --2- 172.21.15.12:0/4033168417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbdc8138210 0x7fbdc81385f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbdb80098a0 tx=0x7fbdb802a080 comp rx=0 tx=0).stop 2024-09-11T01:53:37.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.021+0000 7fbdcd7da640 1 -- 172.21.15.12:0/4033168417 shutdown_connections 2024-09-11T01:53:37.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.021+0000 7fbdcd7da640 1 --2- 172.21.15.12:0/4033168417 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbdc8138210 0x7fbdc81385f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.021+0000 7fbdcd7da640 1 -- 172.21.15.12:0/4033168417 >> 172.21.15.12:0/4033168417 conn(0x7fbdc80aaa80 msgr2=0x7fbdc80aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:37.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.021+0000 7fbdcd7da640 1 -- 172.21.15.12:0/4033168417 shutdown_connections 2024-09-11T01:53:37.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.021+0000 7fbdcd7da640 1 -- 172.21.15.12:0/4033168417 wait complete. 2024-09-11T01:53:37.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.022+0000 7fbdcd7da640 10 monclient: build_initial_monmap 2024-09-11T01:53:37.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.022+0000 7fbdcd7da640 10 monclient: monmap: 2024-09-11T01:53:37.031 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:37.031 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:53:37.031 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:53:37.031 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:53:37.031 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:37.031 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.031 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:53:37.031 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:53:37.031 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:53:37.031 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.022+0000 7fbdcd7da640 1 Processor -- start 2024-09-11T01:53:37.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.022+0000 7fbdcd7da640 1 -- start start 2024-09-11T01:53:37.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.022+0000 7fbdcd7da640 10 monclient: init 2024-09-11T01:53:37.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdcd7da640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:37.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdcd7da640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:53:37.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdcd7da640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdc80a57a0 0x7fbdc8136c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdcd7da640 10 monclient(hunting): picked mon.noname-a con 0x7fbdc80a57a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:53:37.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdcd7da640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbdc80a6810 0x7fbdc8120ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdcd7da640 10 monclient(hunting): picked mon.noname-b con 0x7fbdc80a6810 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:53:37.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdcd7da640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbdc8138210 0x7fbdc8121220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdcd7da640 10 monclient(hunting): picked mon.noname-c con 0x7fbdc8138210 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:53:37.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdcd7da640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdcd7da640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc806f670 con 0x7fbdc80a57a0 2024-09-11T01:53:37.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdcd7da640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdcd7da640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc806f4f0 con 0x7fbdc80a6810 2024-09-11T01:53:37.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdcd7da640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdcd7da640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fbdc8138b30 con 0x7fbdc8138210 2024-09-11T01:53:37.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdcd7da640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:37.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdcd7da640 10 monclient(hunting): authenticate will time out at 2223.465396s 2024-09-11T01:53:37.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc6ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdc80a57a0 0x7fbdc8136c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc77fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbdc8138210 0x7fbdc8121220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc6ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdc80a57a0 0x7fbdc8136c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39416/0 (socket says 172.21.15.12:39416) 2024-09-11T01:53:37.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc6ffd640 1 -- 172.21.15.12:0/4090698384 learned_addr learned my addr 172.21.15.12:0/4090698384 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:53:37.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc6ffd640 10 monclient(hunting): get_auth_request con 0x7fbdc80a57a0 auth_method 0 2024-09-11T01:53:37.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc77fe640 10 monclient(hunting): get_auth_request con 0x7fbdc8138210 auth_method 0 2024-09-11T01:53:37.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:37.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:37.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:37.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:37.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:37.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc67fc640 1 --2- 172.21.15.12:0/4090698384 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbdc80a6810 0x7fbdc8120ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:37.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc67fc640 10 monclient(hunting): get_auth_request con 0x7fbdc80a6810 auth_method 0 2024-09-11T01:53:37.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc67fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc67fc640 1 -- 172.21.15.12:0/4090698384 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbdc80a6810 msgr2=0x7fbdc8120ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:53:37.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc6ffd640 10 monclient(hunting): handle_auth_done global_id 264210 payload 824 2024-09-11T01:53:37.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc67fc640 1 -- 172.21.15.12:0/4090698384 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbdc80a6810 msgr2=0x7fbdc8120ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:53:37.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc67fc640 1 --2- 172.21.15.12:0/4090698384 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbdc80a6810 0x7fbdc8120ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:53:37.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc67fc640 1 --2- 172.21.15.12:0/4090698384 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbdc80a6810 0x7fbdc8120ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:53:37.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc6ffd640 1 -- 172.21.15.12:0/4090698384 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbdc8138210 msgr2=0x7fbdc8121220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc6ffd640 1 --2- 172.21.15.12:0/4090698384 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbdc8138210 0x7fbdc8121220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc6ffd640 1 -- 172.21.15.12:0/4090698384 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbdc80a6810 msgr2=0x7fbdc8120ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:53:37.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc6ffd640 1 --2- 172.21.15.12:0/4090698384 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbdc80a6810 0x7fbdc8120ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc6ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:53:37.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc6ffd640 1 monclient: found mon.noname-a 2024-09-11T01:53:37.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:37.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:53:37.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.023+0000 7fbdc6ffd640 1 -- 172.21.15.12:0/4090698384 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdc80a1130 con 0x7fbdc80a57a0 2024-09-11T01:53:37.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.026+0000 7fbdc6ffd640 1 -- 172.21.15.12:0/4090698384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdc80a57a0 msgr2=0x7fbdc8136c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:37.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.026+0000 7fbdc6ffd640 1 --2- 172.21.15.12:0/4090698384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdc80a57a0 0x7fbdc8136c50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbdb80090b0 tx=0x7fbdb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:37.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.026+0000 7fbdaf7fe640 1 -- 172.21.15.12:0/4090698384 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb8041070 con 0x7fbdc80a57a0 2024-09-11T01:53:37.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.026+0000 7fbdaf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.026+0000 7fbdaf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:53:37.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.026+0000 7fbdaf7fe640 10 monclient: dump: 2024-09-11T01:53:37.038 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.038 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.038 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.038 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.038 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.038 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.038 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.038 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.038 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.038 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.026+0000 7fbdaf7fe640 10 monclient: _finish_auth 0 2024-09-11T01:53:37.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.026+0000 7fbdaf7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:53:37.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.026+0000 7fbdaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:37.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.026+0000 7fbdaf7fe640 1 -- 172.21.15.12:0/4090698384 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdb803c880 con 0x7fbdc80a57a0 2024-09-11T01:53:37.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.026+0000 7fbdcd7da640 5 monclient: authenticate success, global_id 264210 2024-09-11T01:53:37.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.026+0000 7fbdcd7da640 10 monclient: _renew_subs 2024-09-11T01:53:37.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.026+0000 7fbdcd7da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:37.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.026+0000 7fbdcd7da640 1 -- 172.21.15.12:0/4090698384 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdc8124490 con 0x7fbdc80a57a0 2024-09-11T01:53:37.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.026+0000 7fbdaf7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:37.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.026+0000 7fbdaf7fe640 1 -- 172.21.15.12:0/4090698384 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb8037040 con 0x7fbdc80a57a0 2024-09-11T01:53:37.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.026+0000 7fbdaf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.026+0000 7fbdaf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:53:37.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.026+0000 7fbdaf7fe640 10 monclient: dump: 2024-09-11T01:53:37.039 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.040 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.040 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.040 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.040 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.040 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.040 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.040 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.040 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.040 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.026+0000 7fbdcd7da640 10 monclient: _renew_subs 2024-09-11T01:53:37.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.026+0000 7fbdcd7da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:37.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.026+0000 7fbdcd7da640 1 -- 172.21.15.12:0/4090698384 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdc80a7b00 con 0x7fbdc80a57a0 2024-09-11T01:53:37.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.026+0000 7fbdc57fa640 4 set_mon_vals no callback set 2024-09-11T01:53:37.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.027+0000 7fbdcd7da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:37.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.027+0000 7fbdcd7da640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:37.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.027+0000 7fbdcd7da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:37.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.027+0000 7fbdaf7fe640 1 -- 172.21.15.12:0/4090698384 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fbdb803ccb0 con 0x7fbdc80a57a0 2024-09-11T01:53:37.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.027+0000 7fbdcd7da640 1 -- 172.21.15.12:0/4090698384 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdc80a5c60 con 0x7fbdc80a57a0 2024-09-11T01:53:37.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.027+0000 7fbdaf7fe640 1 --2- 172.21.15.12:0/4090698384 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbdb403e240 0x7fbdb4040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.027+0000 7fbdaf7fe640 1 -- 172.21.15.12:0/4090698384 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(831..831 src has 676..831) ==== 4771+0+0 (secure 0 0 0) 0x7fbdb80734e0 con 0x7fbdc80a57a0 2024-09-11T01:53:37.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.028+0000 7fbdc67fc640 1 --2- 172.21.15.12:0/4090698384 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbdb403e240 0x7fbdb4040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.028+0000 7fbdc67fc640 10 monclient: get_auth_request con 0x7fbdb403e240 auth_method 0 2024-09-11T01:53:37.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.030+0000 7fbdc67fc640 1 --2- 172.21.15.12:0/4090698384 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbdb403e240 0x7fbdb4040700 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8124980 tx=0x7fbdbc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:37.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.032+0000 7fbdaf7fe640 1 -- 172.21.15.12:0/4090698384 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbdb804b070 con 0x7fbdc80a57a0 2024-09-11T01:53:37.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.032+0000 7fbdaf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:37.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.032+0000 7fbdaf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:53:37.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.295+0000 7fbdcd7da640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:53:37.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.295+0000 7fbdcd7da640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:53:37.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.295+0000 7fbdcd7da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:37.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.295+0000 7fbdcd7da640 1 -- 172.21.15.12:0/4090698384 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbdc8074b50 con 0x7fbdc80a57a0 2024-09-11T01:53:37.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.295+0000 7fbdaf7fe640 1 -- 172.21.15.12:0/4090698384 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fbdc80a1130 con 0x7fbdc80a57a0 2024-09-11T01:53:37.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.295+0000 7fbdaf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:53:37.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.295+0000 7fbdaf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:53:37.298 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":226,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:53:37.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.298+0000 7fbdcd7da640 1 -- 172.21.15.12:0/4090698384 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbdb403e240 msgr2=0x7fbdb4040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.298+0000 7fbdcd7da640 1 --2- 172.21.15.12:0/4090698384 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbdb403e240 0x7fbdb4040700 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8124980 tx=0x7fbdbc009040 comp rx=0 tx=0).stop 2024-09-11T01:53:37.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.298+0000 7fbdcd7da640 10 monclient: shutdown 2024-09-11T01:53:37.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.298+0000 7fbdcd7da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:37.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.298+0000 7fbdcd7da640 1 -- 172.21.15.12:0/4090698384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdc80a57a0 msgr2=0x7fbdc8136c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.298+0000 7fbdcd7da640 1 --2- 172.21.15.12:0/4090698384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdc80a57a0 0x7fbdc8136c50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbdb80090b0 tx=0x7fbdb80046c0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.298+0000 7fbdcd7da640 1 -- 172.21.15.12:0/4090698384 shutdown_connections 2024-09-11T01:53:37.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.298+0000 7fbdcd7da640 1 --2- 172.21.15.12:0/4090698384 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbdc8138210 0x7fbdc8121220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.298+0000 7fbdcd7da640 1 --2- 172.21.15.12:0/4090698384 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbdc80a6810 0x7fbdc8120ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.298+0000 7fbdcd7da640 1 --2- 172.21.15.12:0/4090698384 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fbdb403e240 0x7fbdb4040700 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.298+0000 7fbdcd7da640 1 --2- 172.21.15.12:0/4090698384 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbdc80a57a0 0x7fbdc8136c50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.298+0000 7fbdcd7da640 1 -- 172.21.15.12:0/4090698384 >> 172.21.15.12:0/4090698384 conn(0x7fbdc80aaa80 msgr2=0x7fbdc80a3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:37.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.298+0000 7fbdcd7da640 1 -- 172.21.15.12:0/4090698384 shutdown_connections 2024-09-11T01:53:37.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.298+0000 7fbdcd7da640 1 -- 172.21.15.12:0/4090698384 wait complete. 2024-09-11T01:53:37.310 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:53:37.310 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:53:37.310 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:53:37.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.410+0000 7fe2fdd2b640 10 monclient: get_monmap_and_config 2024-09-11T01:53:37.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.410+0000 7fe2fdd2b640 10 monclient: build_initial_monmap 2024-09-11T01:53:37.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.410+0000 7fe2fdd2b640 10 monclient: monmap: 2024-09-11T01:53:37.412 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:37.412 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.412 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:53:37.411438+0000 2024-09-11T01:53:37.413 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:53:37.411438+0000 2024-09-11T01:53:37.413 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:37.413 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.413 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:53:37.413 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:53:37.413 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:53:37.413 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:53:37.413 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:53:37.413 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.411+0000 7fe2fdd2b640 1 Processor -- start 2024-09-11T01:53:37.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.411+0000 7fe2fdd2b640 1 -- start start 2024-09-11T01:53:37.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.411+0000 7fe2fdd2b640 10 monclient: init 2024-09-11T01:53:37.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.411+0000 7fe2fdd2b640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:37.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.411+0000 7fe2fdd2b640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-11T01:53:37.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.411+0000 7fe2fdd2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe2f80a2e20 addr v1:172.21.15.12:6789/0 2024-09-11T01:53:37.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2fdd2b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe2f80a4970 0x7fe2f806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2fdd2b640 10 monclient(hunting): picked mon.noname-c con 0x7fe2f80a4970 addr v2:172.21.15.71:3300/0 2024-09-11T01:53:37.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2fdd2b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe2f806f910 0x7fe2f813f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2fdd2b640 10 monclient(hunting): picked mon.noname-a con 0x7fe2f806f910 addr v2:172.21.15.12:3300/0 2024-09-11T01:53:37.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2fdd2b640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2f806f4f0 con 0x7fe2f80a2e20 2024-09-11T01:53:37.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2fdd2b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2fdd2b640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fe2f806ef30 con 0x7fe2f806f910 2024-09-11T01:53:37.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2fdd2b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2fdd2b640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fe2f806ec50 con 0x7fe2f80a4970 2024-09-11T01:53:37.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2fdd2b640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:37.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2fdd2b640 10 monclient(hunting): authenticate will time out at 2223.854082s 2024-09-11T01:53:37.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2f7fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe2f806f910 0x7fe2f813f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2f77fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe2f80a2e20 0x7fe2f80a3200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58974/0 (socket says 172.21.15.12:58974) 2024-09-11T01:53:37.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2f77fe640 1 -- 172.21.15.12:0/1090871585 learned_addr learned my addr 172.21.15.12:0/1090871585 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:53:37.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2f7fff640 10 monclient(hunting): get_auth_request con 0x7fe2f806f910 auth_method 0 2024-09-11T01:53:37.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:37.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:37.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:37.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2f6ffd640 1 --2- 172.21.15.12:0/1090871585 >> v2:172.21.15.71:3300/0 conn(0x7fe2f80a4970 0x7fe2f806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2f7fff640 10 monclient(hunting): handle_auth_done global_id 264213 payload 274 2024-09-11T01:53:37.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2f7fff640 1 -- 172.21.15.12:0/1090871585 >> v2:172.21.15.71:3300/0 conn(0x7fe2f80a4970 msgr2=0x7fe2f806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:53:37.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2f7fff640 1 --2- 172.21.15.12:0/1090871585 >> v2:172.21.15.71:3300/0 conn(0x7fe2f80a4970 0x7fe2f806ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2f7fff640 1 -- 172.21.15.12:0/1090871585 >> v1:172.21.15.12:6789/0 conn(0x7fe2f80a2e20 legacy=0x7fe2f80a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:53:37.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2f7fff640 10 monclient: _finish_hunting 0 2024-09-11T01:53:37.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2f7fff640 1 monclient: found mon.noname-a 2024-09-11T01:53:37.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:37.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2f7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:53:37.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.412+0000 7fe2f7fff640 1 -- 172.21.15.12:0/1090871585 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f80a3910 con 0x7fe2f806f910 2024-09-11T01:53:37.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.415+0000 7fe2f7fff640 1 -- 172.21.15.12:0/1090871585 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f806f910 msgr2=0x7fe2f813f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:37.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.415+0000 7fe2f7fff640 1 --2- 172.21.15.12:0/1090871585 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f806f910 0x7fe2f813f430 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8074f40 tx=0x7fe2f002a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e555c51faeadd6fb server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:37.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.415+0000 7fe2f67fc640 1 -- 172.21.15.12:0/1090871585 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f0037070 con 0x7fe2f806f910 2024-09-11T01:53:37.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.415+0000 7fe2f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.415+0000 7fe2f67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:53:37.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.415+0000 7fe2f67fc640 10 monclient: dump: 2024-09-11T01:53:37.423 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.423 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.423 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.423 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.423 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.423 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.423 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.424 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.424 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.424 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.415+0000 7fe2f67fc640 10 monclient: _finish_auth 0 2024-09-11T01:53:37.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.415+0000 7fe2f67fc640 10 monclient: _check_auth_tickets 2024-09-11T01:53:37.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.415+0000 7fe2f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:37.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.415+0000 7fe2f67fc640 1 -- 172.21.15.12:0/1090871585 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2f0032640 con 0x7fe2f806f910 2024-09-11T01:53:37.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.415+0000 7fe2fdd2b640 5 monclient: authenticate success, global_id 264213 2024-09-11T01:53:37.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.415+0000 7fe2fdd2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:53:37.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.415+0000 7fe2f67fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:37.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.415+0000 7fe2fdd2b640 10 monclient: get_monmap_and_config success 2024-09-11T01:53:37.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.415+0000 7fe2f67fc640 1 -- 172.21.15.12:0/1090871585 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f0032a10 con 0x7fe2f806f910 2024-09-11T01:53:37.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.415+0000 7fe2fdd2b640 4 set_mon_vals no callback set 2024-09-11T01:53:37.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.415+0000 7fe2f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.415+0000 7fe2f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:53:37.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.415+0000 7fe2f67fc640 10 monclient: dump: 2024-09-11T01:53:37.425 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.425 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.425 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.425 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.425 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.425 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.425 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.425 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.425 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.425 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.416+0000 7fe2fdd2b640 10 monclient: shutdown 2024-09-11T01:53:37.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.416+0000 7fe2fdd2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:37.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.416+0000 7fe2fdd2b640 1 -- 172.21.15.12:0/1090871585 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f806f910 msgr2=0x7fe2f813f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.416+0000 7fe2fdd2b640 1 --2- 172.21.15.12:0/1090871585 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f806f910 0x7fe2f813f430 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8074f40 tx=0x7fe2f002a000 comp rx=0 tx=0).stop 2024-09-11T01:53:37.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.416+0000 7fe2fdd2b640 1 -- 172.21.15.12:0/1090871585 shutdown_connections 2024-09-11T01:53:37.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.416+0000 7fe2fdd2b640 1 --2- 172.21.15.12:0/1090871585 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f806f910 0x7fe2f813f430 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.416+0000 7fe2fdd2b640 1 --2- 172.21.15.12:0/1090871585 >> v2:172.21.15.71:3300/0 conn(0x7fe2f80a4970 0x7fe2f806ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.416+0000 7fe2fdd2b640 1 -- 172.21.15.12:0/1090871585 >> 172.21.15.12:0/1090871585 conn(0x7fe2f80a8600 msgr2=0x7fe2f80a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:37.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.416+0000 7fe2fdd2b640 1 -- 172.21.15.12:0/1090871585 shutdown_connections 2024-09-11T01:53:37.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.416+0000 7fe2fdd2b640 1 -- 172.21.15.12:0/1090871585 wait complete. 2024-09-11T01:53:37.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.416+0000 7fe2fdd2b640 10 monclient: build_initial_monmap 2024-09-11T01:53:37.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.416+0000 7fe2fdd2b640 10 monclient: monmap: 2024-09-11T01:53:37.426 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:53:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:53:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:53:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:53:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:53:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:53:37.427 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.417+0000 7fe2fdd2b640 1 Processor -- start 2024-09-11T01:53:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.417+0000 7fe2fdd2b640 1 -- start start 2024-09-11T01:53:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.417+0000 7fe2fdd2b640 10 monclient: init 2024-09-11T01:53:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.417+0000 7fe2fdd2b640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:37.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.417+0000 7fe2fdd2b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:53:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.417+0000 7fe2fdd2b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f806f910 0x7fe2f809e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.417+0000 7fe2fdd2b640 10 monclient(hunting): picked mon.noname-c con 0x7fe2f806f910 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:53:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.417+0000 7fe2fdd2b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f80a2e20 0x7fe2f81398e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.417+0000 7fe2fdd2b640 10 monclient(hunting): picked mon.noname-a con 0x7fe2f80a2e20 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:53:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.417+0000 7fe2fdd2b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2f80a4970 0x7fe2f8139e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.417+0000 7fe2fdd2b640 10 monclient(hunting): picked mon.noname-b con 0x7fe2f80a4970 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:53:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.417+0000 7fe2fdd2b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.417+0000 7fe2fdd2b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f813fd00 con 0x7fe2f80a2e20 2024-09-11T01:53:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.417+0000 7fe2fdd2b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.417+0000 7fe2fdd2b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f813fb80 con 0x7fe2f80a4970 2024-09-11T01:53:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.417+0000 7fe2fdd2b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.417+0000 7fe2fdd2b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe2f813fe80 con 0x7fe2f806f910 2024-09-11T01:53:37.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.417+0000 7fe2fdd2b640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.417+0000 7fe2fdd2b640 10 monclient(hunting): authenticate will time out at 2223.859998s 2024-09-11T01:53:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f6ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f80a2e20 0x7fe2f81398e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f77fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f806f910 0x7fe2f809e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f6ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f80a2e20 0x7fe2f81398e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39444/0 (socket says 172.21.15.12:39444) 2024-09-11T01:53:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f6ffd640 1 -- 172.21.15.12:0/1362497611 learned_addr learned my addr 172.21.15.12:0/1362497611 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:53:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f6ffd640 10 monclient(hunting): get_auth_request con 0x7fe2f80a2e20 auth_method 0 2024-09-11T01:53:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f77fe640 10 monclient(hunting): get_auth_request con 0x7fe2f806f910 auth_method 0 2024-09-11T01:53:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.429 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:37.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:37.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:37.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:37.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:37.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:37.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f7fff640 1 --2- 172.21.15.12:0/1362497611 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2f80a4970 0x7fe2f8139e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f7fff640 10 monclient(hunting): get_auth_request con 0x7fe2f80a4970 auth_method 0 2024-09-11T01:53:37.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f77fe640 10 monclient(hunting): handle_auth_done global_id 244412 payload 824 2024-09-11T01:53:37.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f77fe640 1 -- 172.21.15.12:0/1362497611 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2f80a4970 msgr2=0x7fe2f8139e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f7fff640 1 -- 172.21.15.12:0/1362497611 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2f80a4970 msgr2=0x7fe2f8139e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:53:37.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f7fff640 1 -- 172.21.15.12:0/1362497611 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2f80a4970 msgr2=0x7fe2f8139e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:53:37.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f7fff640 1 --2- 172.21.15.12:0/1362497611 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2f80a4970 0x7fe2f8139e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:53:37.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f7fff640 1 --2- 172.21.15.12:0/1362497611 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2f80a4970 0x7fe2f8139e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:53:37.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f77fe640 1 --2- 172.21.15.12:0/1362497611 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2f80a4970 0x7fe2f8139e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f77fe640 1 -- 172.21.15.12:0/1362497611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f80a2e20 msgr2=0x7fe2f81398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f77fe640 1 --2- 172.21.15.12:0/1362497611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f80a2e20 0x7fe2f81398e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f77fe640 10 monclient: _finish_hunting 0 2024-09-11T01:53:37.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f77fe640 1 monclient: found mon.noname-c 2024-09-11T01:53:37.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:37.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:53:37.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f77fe640 1 -- 172.21.15.12:0/1362497611 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f813bd90 con 0x7fe2f806f910 2024-09-11T01:53:37.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.418+0000 7fe2f6ffd640 1 --2- 172.21.15.12:0/1362497611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f80a2e20 0x7fe2f81398e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:53:37.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.420+0000 7fe2f77fe640 1 -- 172.21.15.12:0/1362497611 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f806f910 msgr2=0x7fe2f809e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:37.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.420+0000 7fe2f77fe640 1 --2- 172.21.15.12:0/1362497611 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f806f910 0x7fe2f809e2b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe2e800caf0 tx=0x7fe2e800a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:37.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.420+0000 7fe2effff640 1 -- 172.21.15.12:0/1362497611 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e8012400 con 0x7fe2f806f910 2024-09-11T01:53:37.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.420+0000 7fe2effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.420+0000 7fe2effff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:53:37.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.420+0000 7fe2effff640 10 monclient: dump: 2024-09-11T01:53:37.432 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.432 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.432 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.432 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.432 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.432 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.432 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.432 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.433 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.433 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.420+0000 7fe2effff640 10 monclient: _finish_auth 0 2024-09-11T01:53:37.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.420+0000 7fe2effff640 10 monclient: _check_auth_tickets 2024-09-11T01:53:37.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.420+0000 7fe2effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:37.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.420+0000 7fe2effff640 1 -- 172.21.15.12:0/1362497611 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2e80125a0 con 0x7fe2f806f910 2024-09-11T01:53:37.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.420+0000 7fe2fdd2b640 5 monclient: authenticate success, global_id 244412 2024-09-11T01:53:37.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.420+0000 7fe2fdd2b640 10 monclient: _renew_subs 2024-09-11T01:53:37.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.420+0000 7fe2fdd2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:53:37.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.420+0000 7fe2fdd2b640 1 -- 172.21.15.12:0/1362497611 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2f809ed20 con 0x7fe2f806f910 2024-09-11T01:53:37.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.420+0000 7fe2effff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:37.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.420+0000 7fe2effff640 1 -- 172.21.15.12:0/1362497611 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e8012990 con 0x7fe2f806f910 2024-09-11T01:53:37.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.420+0000 7fe2effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.420+0000 7fe2effff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:53:37.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.420+0000 7fe2effff640 10 monclient: dump: 2024-09-11T01:53:37.434 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.434 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.434 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.434 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.434 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.434 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.434 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.434 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.434 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.434 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.421+0000 7fe2fdd2b640 10 monclient: _renew_subs 2024-09-11T01:53:37.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.421+0000 7fe2fdd2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:53:37.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.421+0000 7fe2fdd2b640 1 -- 172.21.15.12:0/1362497611 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2f813a360 con 0x7fe2f806f910 2024-09-11T01:53:37.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.420+0000 7fe2f5ffb640 4 set_mon_vals no callback set 2024-09-11T01:53:37.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.422+0000 7fe2fdd2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:37.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.422+0000 7fe2fdd2b640 1 --2- 172.21.15.12:0/1362497611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f81ccf90 0x7fe2f81cd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.422+0000 7fe2fdd2b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.422+0000 7fe2fdd2b640 1 -- 172.21.15.12:0/1362497611 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f813c0d0 con 0x7fe2f81ccf90 2024-09-11T01:53:37.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.422+0000 7fe2f6ffd640 1 --2- 172.21.15.12:0/1362497611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f81ccf90 0x7fe2f81cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.422+0000 7fe2f6ffd640 10 monclient: get_auth_request con 0x7fe2f81ccf90 auth_method 0 2024-09-11T01:53:37.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.422+0000 7fe2f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.422+0000 7fe2f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.422+0000 7fe2f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.422+0000 7fe2f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:37.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.422+0000 7fe2f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:37.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.422+0000 7fe2f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:37.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.422+0000 7fe2effff640 1 -- 172.21.15.12:0/1362497611 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fe2e8012bf0 con 0x7fe2f806f910 2024-09-11T01:53:37.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.422+0000 7fe2f6ffd640 10 monclient(hunting): handle_auth_done global_id 264219 payload 824 2024-09-11T01:53:37.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.422+0000 7fe2f6ffd640 1 -- 172.21.15.12:0/1362497611 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe2f813ba10 con 0x7fe2f81ccf90 2024-09-11T01:53:37.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.422+0000 7fe2effff640 1 --2- 172.21.15.12:0/1362497611 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe2d803e260 0x7fe2d8040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.422+0000 7fe2effff640 1 -- 172.21.15.12:0/1362497611 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(831..831 src has 676..831) ==== 4771+0+0 (secure 0 0 0) 0x7fe2e80533c0 con 0x7fe2f806f910 2024-09-11T01:53:37.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.422+0000 7fe2f7fff640 1 --2- 172.21.15.12:0/1362497611 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe2d803e260 0x7fe2d8040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.422+0000 7fe2f7fff640 10 monclient: get_auth_request con 0x7fe2d803e260 auth_method 0 2024-09-11T01:53:37.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.424+0000 7fe2f6ffd640 1 -- 172.21.15.12:0/1362497611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f81ccf90 msgr2=0x7fe2f81cd370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:37.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.424+0000 7fe2f6ffd640 1 --2- 172.21.15.12:0/1362497611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f81ccf90 0x7fe2f81cd370 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe2f809e990 tx=0x7fe2e000d590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:37.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.424+0000 7fe2effff640 1 -- 172.21.15.12:0/1362497611 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0004220 con 0x7fe2f81ccf90 2024-09-11T01:53:37.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.424+0000 7fe2effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.424+0000 7fe2effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:53:37.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.424+0000 7fe2effff640 10 monclient: dump: 2024-09-11T01:53:37.441 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.441 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.441 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.441 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.441 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.441 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.441 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.441 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.441 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.441 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.425+0000 7fe2effff640 1 -- 172.21.15.12:0/1362497611 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe2e000fe70 con 0x7fe2f81ccf90 2024-09-11T01:53:37.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.425+0000 7fe2effff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:37.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.425+0000 7fe2effff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:53:37.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.425+0000 7fe2effff640 1 -- 172.21.15.12:0/1362497611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f81ccf90 msgr2=0x7fe2f81cd370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.425+0000 7fe2effff640 1 --2- 172.21.15.12:0/1362497611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f81ccf90 0x7fe2f81cd370 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe2f809e990 tx=0x7fe2e000d590 comp rx=0 tx=0).stop 2024-09-11T01:53:37.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.425+0000 7fe2effff640 1 -- 172.21.15.12:0/1362497611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f81ccf90 msgr2=0x7fe2f81cd370 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:53:37.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.425+0000 7fe2effff640 1 --2- 172.21.15.12:0/1362497611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f81ccf90 0x7fe2f81cd370 secure :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe2f809e990 tx=0x7fe2e000d590 comp rx=0 tx=0).stop 2024-09-11T01:53:37.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.425+0000 7fe2f7fff640 1 --2- 172.21.15.12:0/1362497611 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe2d803e260 0x7fe2d8040720 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0034000 tx=0x7fe2f002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:37.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.433+0000 7fe2fdd2b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:53:37.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.433+0000 7fe2fdd2b640 1 --2- 172.21.15.12:0/1362497611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f81cd8b0 0x7fe2f80a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.433+0000 7fe2fdd2b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.433+0000 7fe2fdd2b640 1 -- 172.21.15.12:0/1362497611 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f813a690 con 0x7fe2f81cd8b0 2024-09-11T01:53:37.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.433+0000 7fe2f6ffd640 1 --2- 172.21.15.12:0/1362497611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f81cd8b0 0x7fe2f80a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.434+0000 7fe2f6ffd640 10 monclient: get_auth_request con 0x7fe2f81cd8b0 auth_method 0 2024-09-11T01:53:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.434+0000 7fe2f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.434+0000 7fe2f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.434+0000 7fe2f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.434+0000 7fe2f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.434+0000 7fe2f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.434+0000 7fe2f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.434+0000 7fe2f6ffd640 10 monclient(hunting): handle_auth_done global_id 264222 payload 824 2024-09-11T01:53:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.434+0000 7fe2f6ffd640 1 -- 172.21.15.12:0/1362497611 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe2f806ded0 con 0x7fe2f81cd8b0 2024-09-11T01:53:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.437+0000 7fe2f6ffd640 1 -- 172.21.15.12:0/1362497611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f81cd8b0 msgr2=0x7fe2f80a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.437+0000 7fe2f6ffd640 1 --2- 172.21.15.12:0/1362497611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f81cd8b0 0x7fe2f80a0830 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0019000 tx=0x7fe2e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.437+0000 7fe2effff640 1 -- 172.21.15.12:0/1362497611 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0013b50 con 0x7fe2f81cd8b0 2024-09-11T01:53:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.437+0000 7fe2effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.437+0000 7fe2effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:53:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.437+0000 7fe2effff640 10 monclient: dump: 2024-09-11T01:53:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.444 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.437+0000 7fe2effff640 1 -- 172.21.15.12:0/1362497611 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4710 (secure 0 0 0) 0x7fe2e0013b50 con 0x7fe2f81cd8b0 2024-09-11T01:53:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.437+0000 7fe2effff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:53:37.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.437+0000 7fe2effff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:53:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.437+0000 7fe2effff640 1 -- 172.21.15.12:0/1362497611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f81cd8b0 msgr2=0x7fe2f80a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.437+0000 7fe2effff640 1 --2- 172.21.15.12:0/1362497611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f81cd8b0 0x7fe2f80a0830 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0019000 tx=0x7fe2e00046c0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.437+0000 7fe2effff640 1 -- 172.21.15.12:0/1362497611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f81cd8b0 msgr2=0x7fe2f80a0830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:53:37.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.437+0000 7fe2effff640 1 --2- 172.21.15.12:0/1362497611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2f81cd8b0 0x7fe2f80a0830 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.445 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:53:37.445 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:53:37.445 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:53:37.445 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:53:37.445 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 226, 2024-09-11T01:53:37.446 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 59143, 2024-09-11T01:53:37.446 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:53:37.446 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:53:37.446 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:53:37.446 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:53:37.446 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:37.446 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:53:37.446 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:53:37.446 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:53:37.446 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:53:37.446 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:53:37.446 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:53:37.446 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:53:37.446 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:53:37.447 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:53:37.447 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:53:37.447 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:53:37.447 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:53:37.447 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:53:37.447 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:53:37.447 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:53:37.447 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:37.447 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:53:37.447 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:53:37.447 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:53:37.447 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:53:37.447 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:53:37.447 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:53:37.448 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:53:37.448 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:53:37.448 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:53:37.448 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:53:37.448 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:53:37.448 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:53:37.448 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:53:37.448 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:37.448 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.448 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:53:37.448 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:53:37.448 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:53:37.448 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:53:37.448 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:53:37.449 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:53:37.449 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:53:37.449 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:53:37.449 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:53:37.449 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:53:37.449 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:53:37.449 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:53:37.449 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:53:37.449 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:53:37.449 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:53:37.449 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:53:37.449 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:53:37.449 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:53:37.450 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:53:37.450 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:53:37.450 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:53:37.450 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:53:37.450 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:53:37.450 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:53:37.450 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:53:37.450 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:53:37.450 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:53:37.450 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:53:37.450 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:37.450 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:53:37.450 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.450 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:53:37.451 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.451 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:53:37.451 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:53:37.451 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:53:37.451 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:53:37.451 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.451 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:53:37.451 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:53:37.451 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:37.451 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.451 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.451 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:53:37.451 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:53:37.451 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:37.452 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:37.452 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:37.452 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.452 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:53:37.452 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:53:37.452 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:53:37.452 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:53:37.452 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:53:37.452 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.452 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.452 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:53:37.452 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:53:37.452 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:53:37.452 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:53:37.453 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.453 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:53:37.453 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:53:37.453 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:37.453 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.453 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.453 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:53:37.453 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:53:37.453 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:37.453 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:37.453 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:37.453 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.453 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:53:37.454 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:53:37.454 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:53:37.454 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:53:37.454 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:53:37.454 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.454 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.454 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:53:37.454 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:53:37.454 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:53:37.455 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:53:37.455 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.455 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:53:37.455 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:53:37.455 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:37.455 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.455 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.455 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:53:37.455 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:53:37.455 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:37.455 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:37.456 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:37.456 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.456 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:53:37.456 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:53:37.456 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:53:37.456 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:53:37.456 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:53:37.456 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:37.456 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:37.456 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.457 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:53:37.457 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:53:37.457 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.457 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:53:37.457 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:53:37.457 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:53:37.457 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:37.457 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:37.457 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:53:37.457 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.457 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:53:37.457 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:53:37.457 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:53:37.457 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:37.458 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:37.458 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:53:37.458 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.458 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:53:37.458 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:53:37.458 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:53:37.458 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:37.458 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:37.458 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.458 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:53:37.458 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:53:37.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.438+0000 7fe2fdd2b640 1 -- 172.21.15.12:0/1362497611 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe2d803e260 msgr2=0x7fe2d8040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.438+0000 7fe2fdd2b640 1 --2- 172.21.15.12:0/1362497611 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe2d803e260 0x7fe2d8040720 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0034000 tx=0x7fe2f002e000 comp rx=0 tx=0).stop 2024-09-11T01:53:37.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.438+0000 7fe2fdd2b640 10 monclient: shutdown 2024-09-11T01:53:37.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.438+0000 7fe2fdd2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:37.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.438+0000 7fe2fdd2b640 1 -- 172.21.15.12:0/1362497611 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f806f910 msgr2=0x7fe2f809e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.438+0000 7fe2fdd2b640 1 --2- 172.21.15.12:0/1362497611 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe2f806f910 0x7fe2f809e2b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe2e800caf0 tx=0x7fe2e800a060 comp rx=0 tx=0).stop 2024-09-11T01:53:37.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.438+0000 7fe2f7fff640 1 -- 172.21.15.12:0/1362497611 reap_dead start 2024-09-11T01:53:37.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.438+0000 7fe2f7fff640 1 -- 172.21.15.12:0/1362497611 reap_dead start 2024-09-11T01:53:37.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.438+0000 7fe2fdd2b640 1 -- 172.21.15.12:0/1362497611 shutdown_connections 2024-09-11T01:53:37.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.438+0000 7fe2fdd2b640 1 -- 172.21.15.12:0/1362497611 >> 172.21.15.12:0/1362497611 conn(0x7fe2f80a8600 msgr2=0x7fe2f81381d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:37.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.438+0000 7fe2fdd2b640 1 -- 172.21.15.12:0/1362497611 shutdown_connections 2024-09-11T01:53:37.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.438+0000 7fe2fdd2b640 1 -- 172.21.15.12:0/1362497611 wait complete. 2024-09-11T01:53:37.462 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:53:37.462 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:53:37.462 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:53:37.463 INFO:tasks.rados.rados.0.smithi071.stdout:3634: done (0 left) 2024-09-11T01:53:37.463 INFO:tasks.rados.rados.0.smithi071.stdout:3635: read oid 47 snap -1 2024-09-11T01:53:37.463 INFO:tasks.rados.rados.0.smithi071.stdout:3635: expect (ObjNum 1210 snap 395 seq_num 1210) 2024-09-11T01:53:37.463 INFO:tasks.rados.rados.0.smithi071.stdout:3636: write (excl) oid 40 current snap is 415 2024-09-11T01:53:37.463 INFO:tasks.rados.rados.0.smithi071.stdout:3636: seq_num 1254 ranges {675978=644138,1897949=650107} 2024-09-11T01:53:37.463 INFO:tasks.rados.rados.0.smithi071.stdout:3636: writing smithi07134777-40 from 675978 to 1320116 tid 1 2024-09-11T01:53:37.463 INFO:tasks.rados.rados.0.smithi071.stdout:3636: writing smithi07134777-40 from 1897949 to 2548056 tid 2 2024-09-11T01:53:37.463 INFO:tasks.rados.rados.0.smithi071.stdout:3637: read oid 9 snap -1 2024-09-11T01:53:37.463 INFO:tasks.rados.rados.0.smithi071.stdout:3637: expect (ObjNum 832 snap 254 seq_num 832) 2024-09-11T01:53:37.463 INFO:tasks.rados.rados.0.smithi071.stdout:3638: snap_remove snap 352 2024-09-11T01:53:37.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.563+0000 7fe6ee1ec640 10 monclient: get_monmap_and_config 2024-09-11T01:53:37.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.563+0000 7fe6ee1ec640 10 monclient: build_initial_monmap 2024-09-11T01:53:37.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.563+0000 7fe6ee1ec640 10 monclient: monmap: 2024-09-11T01:53:37.565 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:37.565 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.565 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:53:37.564529+0000 2024-09-11T01:53:37.565 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:53:37.564529+0000 2024-09-11T01:53:37.566 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:37.566 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.566 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:53:37.566 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:53:37.566 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:53:37.566 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:53:37.566 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:53:37.566 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.564+0000 7fe6ee1ec640 1 Processor -- start 2024-09-11T01:53:37.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.564+0000 7fe6ee1ec640 1 -- start start 2024-09-11T01:53:37.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.564+0000 7fe6ee1ec640 10 monclient: init 2024-09-11T01:53:37.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6ee1ec640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:37.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6ee1ec640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-11T01:53:37.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6ee1ec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe6e8135fc0 addr v1:172.21.15.71:6789/0 2024-09-11T01:53:37.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6ee1ec640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe6e8073fe0 0x7fe6e80743f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6ee1ec640 10 monclient(hunting): picked mon.noname-a con 0x7fe6e8073fe0 addr v2:172.21.15.12:3300/0 2024-09-11T01:53:37.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6ee1ec640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe6e813ae90 0x7fe6e813d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6ee1ec640 10 monclient(hunting): picked mon.noname-c con 0x7fe6e813ae90 addr v2:172.21.15.71:3300/0 2024-09-11T01:53:37.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6ee1ec640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6e8072fb0 con 0x7fe6e8135fc0 2024-09-11T01:53:37.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6ee1ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6ee1ec640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fe6e8073850 con 0x7fe6e8073fe0 2024-09-11T01:53:37.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6ee1ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6ee1ec640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fe6e8136ab0 con 0x7fe6e813ae90 2024-09-11T01:53:37.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6ee1ec640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:37.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6ee1ec640 10 monclient(hunting): authenticate will time out at 2224.007229s 2024-09-11T01:53:37.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6e6ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe6e8073fe0 0x7fe6e80743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6e6ffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe6e8073fe0 0x7fe6e80743f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39480/0 (socket says 172.21.15.12:39480) 2024-09-11T01:53:37.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6e6ffd640 1 -- 172.21.15.12:0/3241888253 learned_addr learned my addr 172.21.15.12:0/3241888253 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:53:37.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6e6ffd640 10 monclient(hunting): get_auth_request con 0x7fe6e8073fe0 auth_method 0 2024-09-11T01:53:37.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:37.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:37.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:37.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6e7fff640 1 --2- 172.21.15.12:0/3241888253 >> v2:172.21.15.71:3300/0 conn(0x7fe6e813ae90 0x7fe6e813d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6e6ffd640 10 monclient(hunting): handle_auth_done global_id 264225 payload 274 2024-09-11T01:53:37.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6e6ffd640 1 -- 172.21.15.12:0/3241888253 >> v2:172.21.15.71:3300/0 conn(0x7fe6e813ae90 msgr2=0x7fe6e813d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:53:37.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6e6ffd640 1 --2- 172.21.15.12:0/3241888253 >> v2:172.21.15.71:3300/0 conn(0x7fe6e813ae90 0x7fe6e813d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6e6ffd640 1 -- 172.21.15.12:0/3241888253 >> v1:172.21.15.71:6789/0 conn(0x7fe6e8135fc0 legacy=0x7fe6e81363a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:53:37.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6e6ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:53:37.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6e6ffd640 1 monclient: found mon.noname-a 2024-09-11T01:53:37.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:37.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6e6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:53:37.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6e6ffd640 1 -- 172.21.15.12:0/3241888253 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6e8073290 con 0x7fe6e8073fe0 2024-09-11T01:53:37.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6e7fff640 10 monclient: get_auth_request con 0x7fe6e813ae90 auth_method 0 2024-09-11T01:53:37.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.565+0000 7fe6e7fff640 1 --2- 172.21.15.12:0/3241888253 >> v2:172.21.15.71:3300/0 conn(0x7fe6e813ae90 0x7fe6e813d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:53:37.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.568+0000 7fe6e6ffd640 1 -- 172.21.15.12:0/3241888253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6e8073fe0 msgr2=0x7fe6e80743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:37.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.568+0000 7fe6e6ffd640 1 --2- 172.21.15.12:0/3241888253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6e8073fe0 0x7fe6e80743f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe6e80a4800 tx=0x7fe6dc02a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfddee0926f8cf4f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:37.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.568+0000 7fe6e67fc640 1 -- 172.21.15.12:0/3241888253 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc037070 con 0x7fe6e8073fe0 2024-09-11T01:53:37.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.568+0000 7fe6e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.568+0000 7fe6e67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:53:37.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.568+0000 7fe6e67fc640 10 monclient: dump: 2024-09-11T01:53:37.580 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.580 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.580 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.580 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.580 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.580 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.580 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.580 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.580 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.581 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.568+0000 7fe6e67fc640 10 monclient: _finish_auth 0 2024-09-11T01:53:37.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.568+0000 7fe6e67fc640 10 monclient: _check_auth_tickets 2024-09-11T01:53:37.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.568+0000 7fe6e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:37.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.568+0000 7fe6e67fc640 1 -- 172.21.15.12:0/3241888253 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6dc032640 con 0x7fe6e8073fe0 2024-09-11T01:53:37.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.568+0000 7fe6e67fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:37.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.568+0000 7fe6e67fc640 1 -- 172.21.15.12:0/3241888253 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc032a10 con 0x7fe6e8073fe0 2024-09-11T01:53:37.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.568+0000 7fe6ee1ec640 5 monclient: authenticate success, global_id 264225 2024-09-11T01:53:37.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.568+0000 7fe6ee1ec640 10 monclient: get_monmap_and_config success 2024-09-11T01:53:37.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.568+0000 7fe6e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.568+0000 7fe6ee1ec640 4 set_mon_vals no callback set 2024-09-11T01:53:37.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.568+0000 7fe6e67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:53:37.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.568+0000 7fe6e67fc640 10 monclient: dump: 2024-09-11T01:53:37.581 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.582 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.582 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.582 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.582 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.582 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.582 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.582 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.582 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.582 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.569+0000 7fe6ee1ec640 10 monclient: shutdown 2024-09-11T01:53:37.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.569+0000 7fe6ee1ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:37.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.569+0000 7fe6ee1ec640 1 -- 172.21.15.12:0/3241888253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6e8073fe0 msgr2=0x7fe6e80743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.569+0000 7fe6ee1ec640 1 --2- 172.21.15.12:0/3241888253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6e8073fe0 0x7fe6e80743f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe6e80a4800 tx=0x7fe6dc02a0b0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.569+0000 7fe6ee1ec640 1 -- 172.21.15.12:0/3241888253 shutdown_connections 2024-09-11T01:53:37.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.569+0000 7fe6ee1ec640 1 --2- 172.21.15.12:0/3241888253 >> v2:172.21.15.71:3300/0 conn(0x7fe6e813ae90 0x7fe6e813d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.569+0000 7fe6ee1ec640 1 --2- 172.21.15.12:0/3241888253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6e8073fe0 0x7fe6e80743f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.569+0000 7fe6ee1ec640 1 -- 172.21.15.12:0/3241888253 >> 172.21.15.12:0/3241888253 conn(0x7fe6e80a83b0 msgr2=0x7fe6e80a87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:37.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.569+0000 7fe6ee1ec640 1 -- 172.21.15.12:0/3241888253 shutdown_connections 2024-09-11T01:53:37.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.569+0000 7fe6ee1ec640 1 -- 172.21.15.12:0/3241888253 wait complete. 2024-09-11T01:53:37.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.569+0000 7fe6ee1ec640 10 monclient: build_initial_monmap 2024-09-11T01:53:37.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.569+0000 7fe6ee1ec640 10 monclient: monmap: 2024-09-11T01:53:37.583 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:37.583 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:53:37.583 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:53:37.583 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:53:37.583 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:37.583 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.584 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:53:37.584 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:53:37.584 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:53:37.584 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.570+0000 7fe6ee1ec640 1 Processor -- start 2024-09-11T01:53:37.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.570+0000 7fe6ee1ec640 1 -- start start 2024-09-11T01:53:37.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.570+0000 7fe6ee1ec640 10 monclient: init 2024-09-11T01:53:37.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.570+0000 7fe6ee1ec640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:37.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.570+0000 7fe6ee1ec640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:53:37.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.570+0000 7fe6ee1ec640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6e8073fe0 0x7fe6e80a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.570+0000 7fe6ee1ec640 10 monclient(hunting): picked mon.noname-b con 0x7fe6e8073fe0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:53:37.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.570+0000 7fe6ee1ec640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6e8135fc0 0x7fe6e80a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.570+0000 7fe6ee1ec640 10 monclient(hunting): picked mon.noname-a con 0x7fe6e8135fc0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:53:37.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.570+0000 7fe6ee1ec640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6e813ae90 0x7fe6e81cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.570+0000 7fe6ee1ec640 10 monclient(hunting): picked mon.noname-c con 0x7fe6e813ae90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:53:37.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.570+0000 7fe6ee1ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.570+0000 7fe6ee1ec640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e813d940 con 0x7fe6e8135fc0 2024-09-11T01:53:37.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.570+0000 7fe6ee1ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.570+0000 7fe6ee1ec640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e813d7c0 con 0x7fe6e8073fe0 2024-09-11T01:53:37.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.570+0000 7fe6ee1ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.570+0000 7fe6ee1ec640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe6e813dac0 con 0x7fe6e813ae90 2024-09-11T01:53:37.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.570+0000 7fe6ee1ec640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:37.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.570+0000 7fe6ee1ec640 10 monclient(hunting): authenticate will time out at 2224.012991s 2024-09-11T01:53:37.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e6ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6e8135fc0 0x7fe6e80a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e7fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6e813ae90 0x7fe6e81cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e6ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6e8135fc0 0x7fe6e80a10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39484/0 (socket says 172.21.15.12:39484) 2024-09-11T01:53:37.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e6ffd640 1 -- 172.21.15.12:0/4224028737 learned_addr learned my addr 172.21.15.12:0/4224028737 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:53:37.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e6ffd640 10 monclient(hunting): get_auth_request con 0x7fe6e8135fc0 auth_method 0 2024-09-11T01:53:37.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e7fff640 10 monclient(hunting): get_auth_request con 0x7fe6e813ae90 auth_method 0 2024-09-11T01:53:37.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e77fe640 1 --2- 172.21.15.12:0/4224028737 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6e8073fe0 0x7fe6e80a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:37.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:37.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:37.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e77fe640 10 monclient(hunting): get_auth_request con 0x7fe6e8073fe0 auth_method 0 2024-09-11T01:53:37.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:37.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:37.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:37.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e77fe640 1 -- 172.21.15.12:0/4224028737 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6e8073fe0 msgr2=0x7fe6e80a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:53:37.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e77fe640 1 -- 172.21.15.12:0/4224028737 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6e8073fe0 msgr2=0x7fe6e80a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:53:37.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e77fe640 1 --2- 172.21.15.12:0/4224028737 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6e8073fe0 0x7fe6e80a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:53:37.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e77fe640 1 --2- 172.21.15.12:0/4224028737 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6e8073fe0 0x7fe6e80a0b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:53:37.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e6ffd640 10 monclient(hunting): handle_auth_done global_id 264228 payload 824 2024-09-11T01:53:37.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e6ffd640 1 -- 172.21.15.12:0/4224028737 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6e813ae90 msgr2=0x7fe6e81cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e6ffd640 1 --2- 172.21.15.12:0/4224028737 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6e813ae90 0x7fe6e81cadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e6ffd640 1 -- 172.21.15.12:0/4224028737 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6e8073fe0 msgr2=0x7fe6e80a0b60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:53:37.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e6ffd640 1 --2- 172.21.15.12:0/4224028737 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6e8073fe0 0x7fe6e80a0b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e6ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:53:37.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e6ffd640 1 monclient: found mon.noname-a 2024-09-11T01:53:37.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:37.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:53:37.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.571+0000 7fe6e6ffd640 1 -- 172.21.15.12:0/4224028737 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6e8138520 con 0x7fe6e8135fc0 2024-09-11T01:53:37.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.573+0000 7fe6e6ffd640 1 -- 172.21.15.12:0/4224028737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6e8135fc0 msgr2=0x7fe6e80a10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:37.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.573+0000 7fe6e6ffd640 1 --2- 172.21.15.12:0/4224028737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6e8135fc0 0x7fe6e80a10a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc0090b0 tx=0x7fe6dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:37.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.573+0000 7fe6cffff640 1 -- 172.21.15.12:0/4224028737 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc041020 con 0x7fe6e8135fc0 2024-09-11T01:53:37.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.573+0000 7fe6cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.573+0000 7fe6cffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:53:37.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.573+0000 7fe6cffff640 10 monclient: dump: 2024-09-11T01:53:37.588 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.589 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.589 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.589 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.589 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.589 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.589 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.589 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.589 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.589 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.573+0000 7fe6cffff640 10 monclient: _finish_auth 0 2024-09-11T01:53:37.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.573+0000 7fe6cffff640 10 monclient: _check_auth_tickets 2024-09-11T01:53:37.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.573+0000 7fe6cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:37.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.573+0000 7fe6cffff640 1 -- 172.21.15.12:0/4224028737 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6dc03d910 con 0x7fe6e8135fc0 2024-09-11T01:53:37.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.573+0000 7fe6cffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:37.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.573+0000 7fe6cffff640 1 -- 172.21.15.12:0/4224028737 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc037040 con 0x7fe6e8135fc0 2024-09-11T01:53:37.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.573+0000 7fe6cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.573+0000 7fe6cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:53:37.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.573+0000 7fe6cffff640 10 monclient: dump: 2024-09-11T01:53:37.590 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.590 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.590 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.590 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.590 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.590 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.590 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.590 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.590 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.590 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.573+0000 7fe6ee1ec640 5 monclient: authenticate success, global_id 264228 2024-09-11T01:53:37.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.573+0000 7fe6ee1ec640 10 monclient: _renew_subs 2024-09-11T01:53:37.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.573+0000 7fe6ee1ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:37.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.573+0000 7fe6ee1ec640 1 -- 172.21.15.12:0/4224028737 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6e8138800 con 0x7fe6e8135fc0 2024-09-11T01:53:37.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.573+0000 7fe6e5ffb640 4 set_mon_vals no callback set 2024-09-11T01:53:37.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.573+0000 7fe6ee1ec640 10 monclient: _renew_subs 2024-09-11T01:53:37.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.573+0000 7fe6ee1ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:37.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.573+0000 7fe6ee1ec640 1 -- 172.21.15.12:0/4224028737 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6e80a75c0 con 0x7fe6e8135fc0 2024-09-11T01:53:37.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.574+0000 7fe6ee1ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:37.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.574+0000 7fe6ee1ec640 1 --2- 172.21.15.12:0/4224028737 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6e81cb2f0 0x7fe6e81cb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.574+0000 7fe6ee1ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.574+0000 7fe6ee1ec640 1 -- 172.21.15.12:0/4224028737 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe6e806a060 con 0x7fe6e81cb2f0 2024-09-11T01:53:37.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.574+0000 7fe6e77fe640 1 --2- 172.21.15.12:0/4224028737 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6e81cb2f0 0x7fe6e81cb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.574+0000 7fe6cffff640 1 -- 172.21.15.12:0/4224028737 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fe6dc03a030 con 0x7fe6e8135fc0 2024-09-11T01:53:37.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.574+0000 7fe6e77fe640 10 monclient: get_auth_request con 0x7fe6e81cb2f0 auth_method 0 2024-09-11T01:53:37.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.574+0000 7fe6e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.574+0000 7fe6e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.574+0000 7fe6e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.575+0000 7fe6e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:37.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.575+0000 7fe6e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:37.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.575+0000 7fe6e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:37.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.575+0000 7fe6e77fe640 10 monclient(hunting): handle_auth_done global_id 244418 payload 824 2024-09-11T01:53:37.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.575+0000 7fe6e77fe640 1 -- 172.21.15.12:0/4224028737 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe6e813a820 con 0x7fe6e81cb2f0 2024-09-11T01:53:37.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.575+0000 7fe6cffff640 1 --2- 172.21.15.12:0/4224028737 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe6d403e260 0x7fe6d4040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.575+0000 7fe6cffff640 1 -- 172.21.15.12:0/4224028737 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(832..832 src has 676..832) ==== 4815+0+0 (secure 0 0 0) 0x7fe6dc073310 con 0x7fe6e8135fc0 2024-09-11T01:53:37.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.575+0000 7fe6e7fff640 1 --2- 172.21.15.12:0/4224028737 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe6d403e260 0x7fe6d4040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.575+0000 7fe6e7fff640 10 monclient: get_auth_request con 0x7fe6d403e260 auth_method 0 2024-09-11T01:53:37.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.577+0000 7fe6e77fe640 1 -- 172.21.15.12:0/4224028737 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6e81cb2f0 msgr2=0x7fe6e81cb6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:37.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.577+0000 7fe6e77fe640 1 --2- 172.21.15.12:0/4224028737 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6e81cb2f0 0x7fe6e81cb6d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe6d800e420 tx=0x7fe6d800e8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:37.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.577+0000 7fe6cffff640 1 -- 172.21.15.12:0/4224028737 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d8007450 con 0x7fe6e81cb2f0 2024-09-11T01:53:37.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.577+0000 7fe6cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.577+0000 7fe6cffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:53:37.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.577+0000 7fe6cffff640 10 monclient: dump: 2024-09-11T01:53:37.593 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.593 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.593 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.593 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.593 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.593 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.594 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.594 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.594 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.594 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.577+0000 7fe6cffff640 1 -- 172.21.15.12:0/4224028737 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe6d8007450 con 0x7fe6e81cb2f0 2024-09-11T01:53:37.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.577+0000 7fe6cffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:37.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.577+0000 7fe6cffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:53:37.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.577+0000 7fe6cffff640 1 -- 172.21.15.12:0/4224028737 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6e81cb2f0 msgr2=0x7fe6e81cb6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.577+0000 7fe6cffff640 1 --2- 172.21.15.12:0/4224028737 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6e81cb2f0 0x7fe6e81cb6d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe6d800e420 tx=0x7fe6d800e8e0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.577+0000 7fe6cffff640 1 -- 172.21.15.12:0/4224028737 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6e81cb2f0 msgr2=0x7fe6e81cb6d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:53:37.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.577+0000 7fe6cffff640 1 --2- 172.21.15.12:0/4224028737 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6e81cb2f0 0x7fe6e81cb6d0 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe6d800e420 tx=0x7fe6d800e8e0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.578+0000 7fe6e7fff640 1 --2- 172.21.15.12:0/4224028737 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe6d403e260 0x7fe6d4040720 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fe6e000d540 tx=0x7fe6e0007780 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:37.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.583+0000 7fe6ee1ec640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:53:37.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.583+0000 7fe6ee1ec640 1 --2- 172.21.15.12:0/4224028737 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6e809c2b0 0x7fe6e809c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.583+0000 7fe6ee1ec640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.583+0000 7fe6ee1ec640 1 -- 172.21.15.12:0/4224028737 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe6e813d540 con 0x7fe6e809c2b0 2024-09-11T01:53:37.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.584+0000 7fe6e77fe640 1 --2- 172.21.15.12:0/4224028737 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6e809c2b0 0x7fe6e809c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.584+0000 7fe6e77fe640 10 monclient: get_auth_request con 0x7fe6e809c2b0 auth_method 0 2024-09-11T01:53:37.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.584+0000 7fe6e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.584+0000 7fe6e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.584+0000 7fe6e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.584+0000 7fe6e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:37.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.584+0000 7fe6e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:37.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.584+0000 7fe6e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:37.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.584+0000 7fe6e77fe640 10 monclient(hunting): handle_auth_done global_id 244421 payload 824 2024-09-11T01:53:37.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.584+0000 7fe6e77fe640 1 -- 172.21.15.12:0/4224028737 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe6e80743f0 con 0x7fe6e809c2b0 2024-09-11T01:53:37.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.586+0000 7fe6e77fe640 1 -- 172.21.15.12:0/4224028737 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6e809c2b0 msgr2=0x7fe6e809c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:37.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.586+0000 7fe6e77fe640 1 --2- 172.21.15.12:0/4224028737 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6e809c2b0 0x7fe6e809c6c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8018000 tx=0x7fe6d8003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:37.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.586+0000 7fe6cffff640 1 -- 172.21.15.12:0/4224028737 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d801eb70 con 0x7fe6e809c2b0 2024-09-11T01:53:37.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.586+0000 7fe6cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.586+0000 7fe6cffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:53:37.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.586+0000 7fe6cffff640 10 monclient: dump: 2024-09-11T01:53:37.598 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.598 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.598 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.598 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.598 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.598 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.598 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.598 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.598 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.598 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.586+0000 7fe6cffff640 1 -- 172.21.15.12:0/4224028737 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7fe6d8006e50 con 0x7fe6e809c2b0 2024-09-11T01:53:37.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.586+0000 7fe6cffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:53:37.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.586+0000 7fe6cffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:53:37.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.587+0000 7fe6cffff640 1 -- 172.21.15.12:0/4224028737 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6e809c2b0 msgr2=0x7fe6e809c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.587+0000 7fe6cffff640 1 --2- 172.21.15.12:0/4224028737 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6e809c2b0 0x7fe6e809c6c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8018000 tx=0x7fe6d8003760 comp rx=0 tx=0).stop 2024-09-11T01:53:37.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.587+0000 7fe6cffff640 1 -- 172.21.15.12:0/4224028737 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6e809c2b0 msgr2=0x7fe6e809c6c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:53:37.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.587+0000 7fe6cffff640 1 --2- 172.21.15.12:0/4224028737 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe6e809c2b0 0x7fe6e809c6c0 secure :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8018000 tx=0x7fe6d8003760 comp rx=0 tx=0).stop 2024-09-11T01:53:37.599 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:53:37.599 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:53:37.599 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:53:37.599 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:53:37.600 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 226, 2024-09-11T01:53:37.600 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 171034, 2024-09-11T01:53:37.600 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:53:37.600 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:53:37.600 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:53:37.600 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:53:37.600 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:37.600 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:53:37.600 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:53:37.600 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:53:37.600 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:53:37.600 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:53:37.600 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:53:37.601 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:53:37.601 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:53:37.601 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:53:37.601 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:53:37.601 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:53:37.601 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:53:37.601 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:53:37.601 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:53:37.601 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:53:37.601 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:37.601 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:53:37.601 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:53:37.601 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:53:37.602 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:53:37.602 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:53:37.602 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:53:37.602 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:53:37.602 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:53:37.602 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:53:37.602 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:53:37.602 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:53:37.602 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:53:37.602 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:53:37.602 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:37.602 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.602 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:53:37.602 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:53:37.603 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:53:37.603 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:53:37.603 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:53:37.603 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:53:37.603 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:53:37.603 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:53:37.603 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:53:37.603 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:53:37.603 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:53:37.603 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:53:37.603 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:53:37.603 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:53:37.603 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:53:37.603 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:53:37.604 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:53:37.604 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:53:37.604 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:53:37.604 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:53:37.604 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:53:37.604 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:53:37.604 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:53:37.604 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:53:37.604 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:53:37.604 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:53:37.604 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:53:37.604 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:53:37.604 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:37.604 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:53:37.605 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.605 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:53:37.605 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.605 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:53:37.605 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:53:37.605 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:53:37.605 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:53:37.605 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.605 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:53:37.605 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:53:37.605 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:37.605 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.605 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.606 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:53:37.606 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:53:37.606 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:37.606 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:37.606 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:37.606 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.606 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:53:37.606 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:53:37.606 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:53:37.606 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:53:37.606 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:53:37.606 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.606 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.606 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:53:37.607 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:53:37.607 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:53:37.607 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:53:37.607 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.607 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:53:37.607 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:53:37.607 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:37.607 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.607 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.607 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:53:37.607 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:53:37.607 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:37.607 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:37.607 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:37.608 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.608 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:53:37.608 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:53:37.608 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:53:37.608 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:53:37.608 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:53:37.608 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.608 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.608 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:53:37.609 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:53:37.609 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:53:37.609 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:53:37.609 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.609 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:53:37.609 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:53:37.609 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:37.609 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.609 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.609 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:53:37.609 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:53:37.609 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:37.609 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:37.609 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:37.610 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.610 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:53:37.610 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:53:37.610 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:53:37.610 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:53:37.610 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:53:37.610 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:37.610 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:37.610 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.610 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:53:37.610 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:53:37.610 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.610 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:53:37.610 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:53:37.611 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:53:37.611 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:37.611 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:37.611 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:53:37.611 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.611 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:53:37.611 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:53:37.611 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:53:37.611 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:37.611 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:37.611 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:53:37.611 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.611 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:53:37.611 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:53:37.611 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:53:37.612 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:37.612 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:37.612 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.612 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:53:37.612 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:53:37.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.587+0000 7fe6ee1ec640 1 -- 172.21.15.12:0/4224028737 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe6d403e260 msgr2=0x7fe6d4040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.587+0000 7fe6ee1ec640 1 --2- 172.21.15.12:0/4224028737 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe6d403e260 0x7fe6d4040720 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fe6e000d540 tx=0x7fe6e0007780 comp rx=0 tx=0).stop 2024-09-11T01:53:37.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.587+0000 7fe6ee1ec640 10 monclient: shutdown 2024-09-11T01:53:37.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.587+0000 7fe6ee1ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:37.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.587+0000 7fe6ee1ec640 1 -- 172.21.15.12:0/4224028737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6e8135fc0 msgr2=0x7fe6e80a10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.587+0000 7fe6ee1ec640 1 --2- 172.21.15.12:0/4224028737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6e8135fc0 0x7fe6e80a10a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc0090b0 tx=0x7fe6dc0046c0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.587+0000 7fe6e7fff640 1 -- 172.21.15.12:0/4224028737 reap_dead start 2024-09-11T01:53:37.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.587+0000 7fe6ee1ec640 1 -- 172.21.15.12:0/4224028737 shutdown_connections 2024-09-11T01:53:37.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.587+0000 7fe6ee1ec640 1 --2- 172.21.15.12:0/4224028737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe6e8135fc0 0x7fe6e80a10a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.587+0000 7fe6ee1ec640 1 -- 172.21.15.12:0/4224028737 >> 172.21.15.12:0/4224028737 conn(0x7fe6e80a83b0 msgr2=0x7fe6e8134d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:37.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.587+0000 7fe6ee1ec640 1 -- 172.21.15.12:0/4224028737 shutdown_connections 2024-09-11T01:53:37.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.587+0000 7fe6ee1ec640 1 -- 172.21.15.12:0/4224028737 wait complete. 2024-09-11T01:53:37.614 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:53:37.614 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:53:37.615 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:53:37.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.715+0000 7ff9342bd640 10 monclient: get_monmap_and_config 2024-09-11T01:53:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.715+0000 7ff9342bd640 10 monclient: build_initial_monmap 2024-09-11T01:53:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.715+0000 7ff9342bd640 10 monclient: monmap: 2024-09-11T01:53:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:53:37.715910+0000 2024-09-11T01:53:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:53:37.715910+0000 2024-09-11T01:53:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:53:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:53:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:53:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:53:37.717 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:53:37.718 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.715+0000 7ff9342bd640 1 Processor -- start 2024-09-11T01:53:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.715+0000 7ff9342bd640 1 -- start start 2024-09-11T01:53:37.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.715+0000 7ff9342bd640 10 monclient: init 2024-09-11T01:53:37.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.716+0000 7ff9342bd640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:37.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.716+0000 7ff9342bd640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-11T01:53:37.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.716+0000 7ff9342bd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff92c137e50 0x7ff92c09d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.716+0000 7ff9342bd640 10 monclient(hunting): picked mon.noname-c con 0x7ff92c137e50 addr v2:172.21.15.71:3300/0 2024-09-11T01:53:37.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.716+0000 7ff9342bd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff92c138230 addr v1:172.21.15.12:6789/0 2024-09-11T01:53:37.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.716+0000 7ff9342bd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff92c13d380 addr v1:172.21.15.71:6789/0 2024-09-11T01:53:37.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.716+0000 7ff9342bd640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff92c0a62d0 con 0x7ff92c138230 2024-09-11T01:53:37.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.716+0000 7ff9342bd640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff92c09bb40 con 0x7ff92c13d380 2024-09-11T01:53:37.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.716+0000 7ff9342bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.716+0000 7ff9342bd640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7ff92c09c4a0 con 0x7ff92c137e50 2024-09-11T01:53:37.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.716+0000 7ff9342bd640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:37.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.716+0000 7ff9342bd640 10 monclient(hunting): authenticate will time out at 2224.158499s 2024-09-11T01:53:37.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.716+0000 7ff931858640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff92c138230 0x7ff92c138640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58988/0 (socket says 172.21.15.12:58988) 2024-09-11T01:53:37.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.716+0000 7ff931858640 1 -- 172.21.15.12:0/3928869457 learned_addr learned my addr 172.21.15.12:0/3928869457 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:53:37.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.716+0000 7ff932059640 1 --2- 172.21.15.12:0/3928869457 >> v2:172.21.15.71:3300/0 conn(0x7ff92c137e50 0x7ff92c09d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.716+0000 7ff932059640 10 monclient(hunting): get_auth_request con 0x7ff92c137e50 auth_method 0 2024-09-11T01:53:37.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.716+0000 7ff932059640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.716+0000 7ff932059640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.716+0000 7ff932059640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.716+0000 7ff932059640 1 -- 172.21.15.12:0/3928869457 >> v2:172.21.15.71:3300/0 conn(0x7ff92c137e50 msgr2=0x7ff92c09d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:53:37.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.716+0000 7ff932059640 1 -- 172.21.15.12:0/3928869457 >> v2:172.21.15.71:3300/0 conn(0x7ff92c137e50 msgr2=0x7ff92c09d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:53:37.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.716+0000 7ff932059640 1 --2- 172.21.15.12:0/3928869457 >> v2:172.21.15.71:3300/0 conn(0x7ff92c137e50 0x7ff92c09d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:53:37.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.717+0000 7ff932059640 1 --2- 172.21.15.12:0/3928869457 >> v2:172.21.15.71:3300/0 conn(0x7ff92c137e50 0x7ff92c09d160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:53:37.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.718+0000 7ff931858640 1 -- 172.21.15.12:0/3928869457 >> v1:172.21.15.12:6789/0 conn(0x7ff92c138230 legacy=0x7ff92c138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:37.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.718+0000 7ff931057640 1 -- 172.21.15.12:0/3928869457 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4067812562 0 0) 0x7ff92c0a62d0 con 0x7ff92c138230 2024-09-11T01:53:37.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.718+0000 7ff931057640 10 monclient(hunting): _negotiate 2024-09-11T01:53:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.718+0000 7ff931057640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.718+0000 7ff931057640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.718+0000 7ff931057640 10 monclient(hunting): my global_id is 264231 2024-09-11T01:53:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.718+0000 7ff931057640 1 -- 172.21.15.12:0/3928869457 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff918003520 con 0x7ff92c138230 2024-09-11T01:53:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff93285a640 1 -- 172.21.15.12:0/3928869457 >> v1:172.21.15.71:6789/0 conn(0x7ff92c13d380 legacy=0x7ff92c13f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 1 -- 172.21.15.12:0/3928869457 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 116753687 0 0) 0x7ff918003520 con 0x7ff92c138230 2024-09-11T01:53:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 1 -- 172.21.15.12:0/3928869457 >> v2:172.21.15.71:3300/0 conn(0x7ff92c137e50 msgr2=0x7ff92c09d160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:53:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 1 --2- 172.21.15.12:0/3928869457 >> v2:172.21.15.71:3300/0 conn(0x7ff92c137e50 0x7ff92c09d160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 1 -- 172.21.15.12:0/3928869457 >> v1:172.21.15.71:6789/0 conn(0x7ff92c13d380 legacy=0x7ff92c13f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 10 monclient: _finish_hunting 0 2024-09-11T01:53:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:53:37.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:53:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 1 -- 172.21.15.12:0/3928869457 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff92c09c780 con 0x7ff92c138230 2024-09-11T01:53:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 10 monclient: _finish_auth 0 2024-09-11T01:53:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 10 monclient: _check_auth_tickets 2024-09-11T01:53:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff9342bd640 5 monclient: authenticate success, global_id 264231 2024-09-11T01:53:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 1 -- 172.21.15.12:0/3928869457 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7ff91c001990 con 0x7ff92c138230 2024-09-11T01:53:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:53:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 10 monclient: dump: 2024-09-11T01:53:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.733 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.734 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.734 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.734 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.734 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.734 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.734 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.734 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:53:37.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:37.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 1 -- 172.21.15.12:0/3928869457 >> v1:172.21.15.12:6789/0 conn(0x7ff92c138230 legacy=0x7ff92c138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:53:37.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 1 --2- 172.21.15.12:0/3928869457 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff918003b20 0x7ff918003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 10 monclient(hunting): picked mon.b con 0x7ff918003b20 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:53:37.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 1 --2- 172.21.15.12:0/3928869457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff918004800 0x7ff918004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 10 monclient(hunting): picked mon.a con 0x7ff918004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:53:37.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 1 --2- 172.21.15.12:0/3928869457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9180099b0 0x7ff91800bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 10 monclient(hunting): picked mon.c con 0x7ff9180099b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:53:37.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 1 -- 172.21.15.12:0/3928869457 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff92c09c4a0 con 0x7ff918004800 2024-09-11T01:53:37.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 1 -- 172.21.15.12:0/3928869457 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff91800c3c0 con 0x7ff918003b20 2024-09-11T01:53:37.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 1 -- 172.21.15.12:0/3928869457 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff91800c5c0 con 0x7ff9180099b0 2024-09-11T01:53:37.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:37.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931057640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:53:37.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff9342bd640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:53:37.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931858640 1 --2- 172.21.15.12:0/3928869457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff918004800 0x7ff918004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff93285a640 1 --2- 172.21.15.12:0/3928869457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9180099b0 0x7ff91800bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931858640 10 monclient(hunting): get_auth_request con 0x7ff918004800 auth_method 0 2024-09-11T01:53:37.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931858640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931858640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931858640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:53:37.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff93285a640 10 monclient(hunting): get_auth_request con 0x7ff9180099b0 auth_method 0 2024-09-11T01:53:37.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff93285a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff93285a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff93285a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:53:37.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff932059640 1 --2- 172.21.15.12:0/3928869457 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff918003b20 0x7ff918003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff932059640 10 monclient(hunting): get_auth_request con 0x7ff918003b20 auth_method 0 2024-09-11T01:53:37.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff932059640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff932059640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff932059640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:53:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931858640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931858640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931858640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:53:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff93285a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff93285a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff93285a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:53:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff932059640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff932059640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff932059640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:53:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931858640 10 monclient(hunting): handle_auth_done global_id 264231 payload 293 2024-09-11T01:53:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931858640 1 -- 172.21.15.12:0/3928869457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9180099b0 msgr2=0x7ff91800bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:53:37.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931858640 1 --2- 172.21.15.12:0/3928869457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9180099b0 0x7ff91800bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931858640 1 -- 172.21.15.12:0/3928869457 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff918003b20 msgr2=0x7ff918003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:53:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931858640 1 --2- 172.21.15.12:0/3928869457 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff918003b20 0x7ff918003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931858640 10 monclient: _finish_hunting 0 2024-09-11T01:53:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931858640 1 monclient: found mon.a 2024-09-11T01:53:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931858640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931858640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931858640 1 -- 172.21.15.12:0/3928869457 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff918003520 con 0x7ff918004800 2024-09-11T01:53:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931858640 10 monclient: _finish_auth 0 2024-09-11T01:53:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931858640 10 monclient: _check_auth_tickets 2024-09-11T01:53:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff931858640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff93285a640 1 --2- 172.21.15.12:0/3928869457 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9180099b0 0x7ff91800bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:53:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.719+0000 7ff93285a640 1 -- 172.21.15.12:0/3928869457 reap_dead start 2024-09-11T01:53:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.722+0000 7ff931858640 1 -- 172.21.15.12:0/3928869457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff918004800 msgr2=0x7ff918004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:37.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.722+0000 7ff931858640 1 --2- 172.21.15.12:0/3928869457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff918004800 0x7ff918004c80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff91c001bb0 tx=0x7ff918005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c04768ed9f5932df server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:37.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.722+0000 7ff931057640 1 -- 172.21.15.12:0/3928869457 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff91c036070 con 0x7ff918004800 2024-09-11T01:53:37.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.722+0000 7ff931057640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.722+0000 7ff931057640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:53:37.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.722+0000 7ff931057640 10 monclient: dump: 2024-09-11T01:53:37.739 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.739 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.739 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.739 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.739 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.739 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.739 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.739 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.739 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.739 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.722+0000 7ff931057640 1 -- 172.21.15.12:0/3928869457 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff91c005ad0 con 0x7ff918004800 2024-09-11T01:53:37.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.722+0000 7ff931057640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:37.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.722+0000 7ff9342bd640 10 monclient: get_monmap_and_config success 2024-09-11T01:53:37.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.722+0000 7ff931057640 1 -- 172.21.15.12:0/3928869457 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff91c00c0e0 con 0x7ff918004800 2024-09-11T01:53:37.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.722+0000 7ff931057640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.722+0000 7ff9342bd640 4 set_mon_vals no callback set 2024-09-11T01:53:37.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.722+0000 7ff931057640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:53:37.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.722+0000 7ff931057640 10 monclient: dump: 2024-09-11T01:53:37.740 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.740 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.740 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.740 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.740 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.740 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.741 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.741 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.741 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.741 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.723+0000 7ff9342bd640 10 monclient: shutdown 2024-09-11T01:53:37.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.723+0000 7ff9342bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:37.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.723+0000 7ff9342bd640 1 -- 172.21.15.12:0/3928869457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff918004800 msgr2=0x7ff918004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.723+0000 7ff9342bd640 1 --2- 172.21.15.12:0/3928869457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff918004800 0x7ff918004c80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff91c001bb0 tx=0x7ff918005830 comp rx=0 tx=0).stop 2024-09-11T01:53:37.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.723+0000 7ff9342bd640 1 -- 172.21.15.12:0/3928869457 shutdown_connections 2024-09-11T01:53:37.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.723+0000 7ff9342bd640 1 --2- 172.21.15.12:0/3928869457 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff918004800 0x7ff918004c80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.723+0000 7ff9342bd640 1 -- 172.21.15.12:0/3928869457 >> 172.21.15.12:0/3928869457 conn(0x7ff92c0a7020 msgr2=0x7ff92c0a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:37.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.723+0000 7ff9342bd640 1 -- 172.21.15.12:0/3928869457 shutdown_connections 2024-09-11T01:53:37.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.723+0000 7ff9342bd640 1 -- 172.21.15.12:0/3928869457 wait complete. 2024-09-11T01:53:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.724+0000 7ff9342bd640 10 monclient: build_initial_monmap 2024-09-11T01:53:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.724+0000 7ff9342bd640 10 monclient: monmap: 2024-09-11T01:53:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:53:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:53:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:53:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:53:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:53:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:53:37.742 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.724+0000 7ff9342bd640 1 Processor -- start 2024-09-11T01:53:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.724+0000 7ff9342bd640 1 -- start start 2024-09-11T01:53:37.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.724+0000 7ff9342bd640 10 monclient: init 2024-09-11T01:53:37.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff9342bd640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:37.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff9342bd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:53:37.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff9342bd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff918004800 0x7ff92c0a41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff9342bd640 10 monclient(hunting): picked mon.noname-a con 0x7ff918004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:53:37.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff9342bd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff92c120b70 0x7ff92c120fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff9342bd640 10 monclient(hunting): picked mon.noname-c con 0x7ff92c120b70 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:53:37.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff9342bd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff92c121510 0x7ff92c1cd520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff9342bd640 10 monclient(hunting): picked mon.noname-b con 0x7ff92c121510 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:53:37.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff9342bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff9342bd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff92c1387c0 con 0x7ff918004800 2024-09-11T01:53:37.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff9342bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff9342bd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff92c138640 con 0x7ff92c121510 2024-09-11T01:53:37.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff9342bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff9342bd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff92c138940 con 0x7ff92c120b70 2024-09-11T01:53:37.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff9342bd640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:37.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff9342bd640 10 monclient(hunting): authenticate will time out at 2224.167143s 2024-09-11T01:53:37.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff932059640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff918004800 0x7ff92c0a41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff931858640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff92c120b70 0x7ff92c120fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff932059640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff918004800 0x7ff92c0a41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39504/0 (socket says 172.21.15.12:39504) 2024-09-11T01:53:37.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff932059640 1 -- 172.21.15.12:0/2974433032 learned_addr learned my addr 172.21.15.12:0/2974433032 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:53:37.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff932059640 10 monclient(hunting): get_auth_request con 0x7ff918004800 auth_method 0 2024-09-11T01:53:37.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff931858640 1 --2- 172.21.15.12:0/2974433032 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff92c120b70 0x7ff92c120fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42906/0 (socket says 172.21.15.12:42906) 2024-09-11T01:53:37.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff932059640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff932059640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff932059640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff931858640 10 monclient(hunting): get_auth_request con 0x7ff92c120b70 auth_method 0 2024-09-11T01:53:37.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff931858640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff931858640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff931858640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff93285a640 1 --2- 172.21.15.12:0/2974433032 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff92c121510 0x7ff92c1cd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff932059640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:37.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff932059640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:37.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff932059640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:37.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff93285a640 10 monclient(hunting): get_auth_request con 0x7ff92c121510 auth_method 0 2024-09-11T01:53:37.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff93285a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff93285a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff93285a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff931858640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:37.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff931858640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:37.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff931858640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:37.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff93285a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:37.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff93285a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:37.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff93285a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:37.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff932059640 10 monclient(hunting): handle_auth_done global_id 264234 payload 824 2024-09-11T01:53:37.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff932059640 1 -- 172.21.15.12:0/2974433032 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff92c120b70 msgr2=0x7ff92c120fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff932059640 1 --2- 172.21.15.12:0/2974433032 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff92c120b70 0x7ff92c120fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff932059640 1 -- 172.21.15.12:0/2974433032 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff92c121510 msgr2=0x7ff92c1cd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff932059640 1 --2- 172.21.15.12:0/2974433032 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff92c121510 0x7ff92c1cd520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff932059640 10 monclient: _finish_hunting 0 2024-09-11T01:53:37.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff932059640 1 monclient: found mon.noname-a 2024-09-11T01:53:37.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff932059640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:37.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff932059640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:53:37.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff932059640 1 -- 172.21.15.12:0/2974433032 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff92c1239f0 con 0x7ff918004800 2024-09-11T01:53:37.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.725+0000 7ff931858640 1 --2- 172.21.15.12:0/2974433032 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff92c120b70 0x7ff92c120fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:53:37.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.727+0000 7ff932059640 1 -- 172.21.15.12:0/2974433032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff918004800 msgr2=0x7ff92c0a41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:37.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.727+0000 7ff932059640 1 --2- 172.21.15.12:0/2974433032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff918004800 0x7ff92c0a41a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff9280108d0 tx=0x7ff928010cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:37.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.727+0000 7ff922ffd640 1 -- 172.21.15.12:0/2974433032 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff92801a020 con 0x7ff918004800 2024-09-11T01:53:37.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.728+0000 7ff922ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.728+0000 7ff922ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:53:37.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.728+0000 7ff922ffd640 10 monclient: dump: 2024-09-11T01:53:37.747 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.747 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.747 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.747 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.747 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.747 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.747 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.747 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.748 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.748 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.728+0000 7ff922ffd640 10 monclient: _finish_auth 0 2024-09-11T01:53:37.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.728+0000 7ff922ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:53:37.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.728+0000 7ff922ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:37.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.728+0000 7ff9342bd640 5 monclient: authenticate success, global_id 264234 2024-09-11T01:53:37.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.728+0000 7ff922ffd640 1 -- 172.21.15.12:0/2974433032 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff92800eb70 con 0x7ff918004800 2024-09-11T01:53:37.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.728+0000 7ff9342bd640 10 monclient: _renew_subs 2024-09-11T01:53:37.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.728+0000 7ff9342bd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:37.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.728+0000 7ff9342bd640 1 -- 172.21.15.12:0/2974433032 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff92c123cd0 con 0x7ff918004800 2024-09-11T01:53:37.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.728+0000 7ff922ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:37.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.728+0000 7ff922ffd640 1 -- 172.21.15.12:0/2974433032 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff928016480 con 0x7ff918004800 2024-09-11T01:53:37.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.728+0000 7ff922ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.728+0000 7ff922ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:53:37.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.728+0000 7ff922ffd640 10 monclient: dump: 2024-09-11T01:53:37.749 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.749 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.749 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.749 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.749 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.749 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.749 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.749 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.749 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.749 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.728+0000 7ff9342bd640 10 monclient: _renew_subs 2024-09-11T01:53:37.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.728+0000 7ff9342bd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:37.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.728+0000 7ff9342bd640 1 -- 172.21.15.12:0/2974433032 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff92c0a9910 con 0x7ff918004800 2024-09-11T01:53:37.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.728+0000 7ff930856640 4 set_mon_vals no callback set 2024-09-11T01:53:37.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.728+0000 7ff922ffd640 1 -- 172.21.15.12:0/2974433032 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff9280166a0 con 0x7ff918004800 2024-09-11T01:53:37.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.729+0000 7ff9342bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:37.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.729+0000 7ff9342bd640 1 --2- 172.21.15.12:0/2974433032 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff92c1cda60 0x7ff92c1cde40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.729+0000 7ff9342bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.729+0000 7ff9342bd640 1 -- 172.21.15.12:0/2974433032 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff92c0a1770 con 0x7ff92c1cda60 2024-09-11T01:53:37.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.729+0000 7ff922ffd640 1 --2- 172.21.15.12:0/2974433032 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff91003e260 0x7ff910040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.729+0000 7ff922ffd640 1 -- 172.21.15.12:0/2974433032 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(832..832 src has 676..832) ==== 4815+0+0 (secure 0 0 0) 0x7ff928053920 con 0x7ff918004800 2024-09-11T01:53:37.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.729+0000 7ff931858640 1 --2- 172.21.15.12:0/2974433032 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff92c1cda60 0x7ff92c1cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.729+0000 7ff93285a640 1 --2- 172.21.15.12:0/2974433032 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff91003e260 0x7ff910040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.729+0000 7ff931858640 10 monclient: get_auth_request con 0x7ff92c1cda60 auth_method 0 2024-09-11T01:53:37.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.729+0000 7ff931858640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.729+0000 7ff931858640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.729+0000 7ff931858640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.729+0000 7ff93285a640 10 monclient: get_auth_request con 0x7ff91003e260 auth_method 0 2024-09-11T01:53:37.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.729+0000 7ff931858640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:37.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.729+0000 7ff931858640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:37.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.729+0000 7ff931858640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:37.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.729+0000 7ff931858640 10 monclient(hunting): handle_auth_done global_id 274102 payload 824 2024-09-11T01:53:37.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.729+0000 7ff931858640 1 -- 172.21.15.12:0/2974433032 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff92c1ce380 con 0x7ff92c1cda60 2024-09-11T01:53:37.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.732+0000 7ff93285a640 1 --2- 172.21.15.12:0/2974433032 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff91003e260 0x7ff910040720 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff92c138610 tx=0x7ff924009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:37.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.732+0000 7ff931858640 1 -- 172.21.15.12:0/2974433032 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff92c1cda60 msgr2=0x7ff92c1cde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:37.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.732+0000 7ff931858640 1 --2- 172.21.15.12:0/2974433032 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff92c1cda60 0x7ff92c1cde40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff91c033000 tx=0x7ff92c09c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:37.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.732+0000 7ff922ffd640 1 -- 172.21.15.12:0/2974433032 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff91c043070 con 0x7ff92c1cda60 2024-09-11T01:53:37.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.732+0000 7ff922ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.732+0000 7ff922ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:53:37.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.732+0000 7ff922ffd640 10 monclient: dump: 2024-09-11T01:53:37.753 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.753 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.754 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.754 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.754 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.754 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.754 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.754 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.754 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.754 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.732+0000 7ff922ffd640 1 -- 172.21.15.12:0/2974433032 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff91c0049e0 con 0x7ff92c1cda60 2024-09-11T01:53:37.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.732+0000 7ff922ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:37.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.732+0000 7ff922ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:53:37.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.732+0000 7ff922ffd640 1 -- 172.21.15.12:0/2974433032 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff92c1cda60 msgr2=0x7ff92c1cde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.732+0000 7ff922ffd640 1 --2- 172.21.15.12:0/2974433032 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff92c1cda60 0x7ff92c1cde40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff91c033000 tx=0x7ff92c09c950 comp rx=0 tx=0).stop 2024-09-11T01:53:37.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.732+0000 7ff922ffd640 1 -- 172.21.15.12:0/2974433032 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff92c1cda60 msgr2=0x7ff92c1cde40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:53:37.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.732+0000 7ff922ffd640 1 --2- 172.21.15.12:0/2974433032 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff92c1cda60 0x7ff92c1cde40 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff91c033000 tx=0x7ff92c09c950 comp rx=0 tx=0).stop 2024-09-11T01:53:37.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.738+0000 7ff9342bd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:53:37.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.738+0000 7ff9342bd640 1 --2- 172.21.15.12:0/2974433032 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff92c09ed20 0x7ff92c09f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.738+0000 7ff9342bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.738+0000 7ff9342bd640 1 -- 172.21.15.12:0/2974433032 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff92c09d160 con 0x7ff92c09ed20 2024-09-11T01:53:37.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.739+0000 7ff931858640 1 --2- 172.21.15.12:0/2974433032 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff92c09ed20 0x7ff92c09f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.739+0000 7ff931858640 10 monclient: get_auth_request con 0x7ff92c09ed20 auth_method 0 2024-09-11T01:53:37.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.739+0000 7ff931858640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.739+0000 7ff931858640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.739+0000 7ff931858640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.739+0000 7ff931858640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:37.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.739+0000 7ff931858640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:37.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.739+0000 7ff931858640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:37.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.739+0000 7ff931858640 10 monclient(hunting): handle_auth_done global_id 274105 payload 824 2024-09-11T01:53:37.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.739+0000 7ff931858640 1 -- 172.21.15.12:0/2974433032 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff92c123380 con 0x7ff92c09ed20 2024-09-11T01:53:37.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.742+0000 7ff931858640 1 -- 172.21.15.12:0/2974433032 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff92c09ed20 msgr2=0x7ff92c09f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:37.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.742+0000 7ff931858640 1 --2- 172.21.15.12:0/2974433032 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff92c09ed20 0x7ff92c09f150 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff91c0010e0 tx=0x7ff91c02f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:37.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.742+0000 7ff922ffd640 1 -- 172.21.15.12:0/2974433032 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff91c036040 con 0x7ff92c09ed20 2024-09-11T01:53:37.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.742+0000 7ff922ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.742+0000 7ff922ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:53:37.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.742+0000 7ff922ffd640 10 monclient: dump: 2024-09-11T01:53:37.756 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.756 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.756 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.757 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.757 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.757 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.757 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.757 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.757 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.757 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.742+0000 7ff922ffd640 1 -- 172.21.15.12:0/2974433032 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4707 (secure 0 0 0) 0x7ff91c03f450 con 0x7ff92c09ed20 2024-09-11T01:53:37.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.742+0000 7ff922ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:53:37.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.742+0000 7ff922ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:53:37.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.742+0000 7ff922ffd640 1 -- 172.21.15.12:0/2974433032 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff92c09ed20 msgr2=0x7ff92c09f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.742+0000 7ff922ffd640 1 --2- 172.21.15.12:0/2974433032 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff92c09ed20 0x7ff92c09f150 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff91c0010e0 tx=0x7ff91c02f000 comp rx=0 tx=0).stop 2024-09-11T01:53:37.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.742+0000 7ff922ffd640 1 -- 172.21.15.12:0/2974433032 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff92c09ed20 msgr2=0x7ff92c09f150 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:53:37.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.742+0000 7ff922ffd640 1 --2- 172.21.15.12:0/2974433032 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff92c09ed20 0x7ff92c09f150 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff91c0010e0 tx=0x7ff91c02f000 comp rx=0 tx=0).stop 2024-09-11T01:53:37.758 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:53:37.758 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:53:37.758 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:53:37.758 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:53:37.758 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 226, 2024-09-11T01:53:37.758 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 7560, 2024-09-11T01:53:37.758 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:53:37.758 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:53:37.758 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:53:37.758 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:53:37.759 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:37.759 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:53:37.759 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:53:37.759 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:53:37.759 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:53:37.759 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:53:37.759 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:53:37.759 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:53:37.759 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:53:37.759 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:53:37.759 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:53:37.759 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:53:37.759 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:53:37.759 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:53:37.760 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:53:37.760 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:53:37.760 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:37.760 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:53:37.760 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:53:37.760 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:53:37.760 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:53:37.760 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:53:37.760 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:53:37.760 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:53:37.760 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:53:37.760 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:53:37.760 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:53:37.760 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:53:37.761 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:53:37.761 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:53:37.761 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:37.761 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.761 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:53:37.761 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:53:37.761 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:53:37.761 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:53:37.761 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:53:37.761 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:53:37.761 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:53:37.761 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:53:37.761 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:53:37.761 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:53:37.762 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:53:37.762 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:53:37.762 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:53:37.762 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:53:37.762 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:53:37.762 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:53:37.762 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:53:37.762 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:53:37.762 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:53:37.762 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:53:37.762 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:53:37.762 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:53:37.762 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:53:37.762 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:53:37.763 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:53:37.763 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:53:37.763 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:53:37.763 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:53:37.763 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:37.763 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:53:37.763 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.763 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:53:37.763 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.763 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:53:37.763 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:53:37.763 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:53:37.763 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:53:37.763 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.764 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:53:37.764 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:53:37.764 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:37.764 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.764 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.764 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:53:37.764 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:53:37.764 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:37.764 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:37.764 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:37.764 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.764 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:53:37.764 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:53:37.765 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:53:37.765 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:53:37.765 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:53:37.765 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.765 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.765 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:53:37.765 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:53:37.765 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:53:37.765 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:53:37.765 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.765 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:53:37.765 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:53:37.765 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:37.765 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.765 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.766 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:53:37.766 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:53:37.766 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:37.766 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:37.766 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:37.766 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.766 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:53:37.766 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:53:37.766 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:53:37.766 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:53:37.766 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:53:37.766 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.766 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.766 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:53:37.767 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:53:37.767 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:53:37.767 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:53:37.767 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.767 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:53:37.767 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:53:37.767 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:37.767 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.767 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.767 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:53:37.767 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:53:37.767 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:37.767 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:37.767 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:37.768 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.768 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:53:37.768 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:53:37.768 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:53:37.768 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:53:37.768 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:53:37.768 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:37.768 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:37.768 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.768 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:53:37.768 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:53:37.768 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.768 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:53:37.768 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:53:37.769 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:53:37.769 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:37.769 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:37.769 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:53:37.769 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.769 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:53:37.769 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:53:37.769 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:53:37.769 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:37.769 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:37.769 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:53:37.769 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:37.769 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:53:37.770 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:53:37.770 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:53:37.770 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:37.770 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:37.770 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:37.770 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:53:37.770 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:53:37.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.742+0000 7ff9342bd640 1 -- 172.21.15.12:0/2974433032 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff91003e260 msgr2=0x7ff910040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.742+0000 7ff9342bd640 1 --2- 172.21.15.12:0/2974433032 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff91003e260 0x7ff910040720 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff92c138610 tx=0x7ff924009040 comp rx=0 tx=0).stop 2024-09-11T01:53:37.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.742+0000 7ff9342bd640 10 monclient: shutdown 2024-09-11T01:53:37.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.742+0000 7ff9342bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:37.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.742+0000 7ff9342bd640 1 -- 172.21.15.12:0/2974433032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff918004800 msgr2=0x7ff92c0a41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.742+0000 7ff9342bd640 1 --2- 172.21.15.12:0/2974433032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff918004800 0x7ff92c0a41a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff9280108d0 tx=0x7ff928010cd0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.742+0000 7ff93285a640 1 -- 172.21.15.12:0/2974433032 reap_dead start 2024-09-11T01:53:37.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.742+0000 7ff93285a640 1 -- 172.21.15.12:0/2974433032 reap_dead start 2024-09-11T01:53:37.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.742+0000 7ff9342bd640 1 -- 172.21.15.12:0/2974433032 shutdown_connections 2024-09-11T01:53:37.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.742+0000 7ff9342bd640 1 -- 172.21.15.12:0/2974433032 >> 172.21.15.12:0/2974433032 conn(0x7ff92c0a7020 msgr2=0x7ff92c0a0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:37.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.742+0000 7ff9342bd640 1 -- 172.21.15.12:0/2974433032 shutdown_connections 2024-09-11T01:53:37.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.743+0000 7ff9342bd640 1 -- 172.21.15.12:0/2974433032 wait complete. 2024-09-11T01:53:37.772 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:53:37.773 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:53:37.773 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:53:37.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.875+0000 7ff98d1cd640 10 monclient: get_monmap_and_config 2024-09-11T01:53:37.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.875+0000 7ff98d1cd640 10 monclient: build_initial_monmap 2024-09-11T01:53:37.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.875+0000 7ff98d1cd640 10 monclient: monmap: 2024-09-11T01:53:37.877 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:37.877 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.878 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:53:37.876612+0000 2024-09-11T01:53:37.878 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:53:37.876612+0000 2024-09-11T01:53:37.878 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:37.878 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.878 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:53:37.878 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:53:37.878 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:53:37.878 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:53:37.878 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:53:37.878 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.876+0000 7ff98d1cd640 1 Processor -- start 2024-09-11T01:53:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.876+0000 7ff98d1cd640 1 -- start start 2024-09-11T01:53:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.876+0000 7ff98d1cd640 10 monclient: init 2024-09-11T01:53:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.876+0000 7ff98d1cd640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.876+0000 7ff98d1cd640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-11T01:53:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.876+0000 7ff98d1cd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff9880a2d40 addr v1:172.21.15.12:6789/0 2024-09-11T01:53:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.876+0000 7ff98d1cd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff98806d260 0x7ff98813d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.876+0000 7ff98d1cd640 10 monclient(hunting): picked mon.noname-a con 0x7ff98806d260 addr v2:172.21.15.12:3300/0 2024-09-11T01:53:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.876+0000 7ff98d1cd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff98806d670 0x7ff98813fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.876+0000 7ff98d1cd640 10 monclient(hunting): picked mon.noname-b con 0x7ff98806d670 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:53:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98d1cd640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff98809aed0 con 0x7ff9880a2d40 2024-09-11T01:53:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98d1cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98d1cd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff98806dd10 con 0x7ff98806d260 2024-09-11T01:53:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98d1cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98d1cd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff98809bcf0 con 0x7ff98806d670 2024-09-11T01:53:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98d1cd640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98d1cd640 10 monclient(hunting): authenticate will time out at 2224.319030s 2024-09-11T01:53:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98759e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff98806d670 0x7ff98813fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98659c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff98806d260 0x7ff98813d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98759e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff98806d670 0x7ff98813fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42922/0 (socket says 172.21.15.12:42922) 2024-09-11T01:53:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98759e640 1 -- 172.21.15.12:0/1165452032 learned_addr learned my addr 172.21.15.12:0/1165452032 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:53:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98759e640 10 monclient(hunting): get_auth_request con 0x7ff98806d670 auth_method 0 2024-09-11T01:53:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff986d9d640 1 --1- 172.21.15.12:0/1165452032 >> v1:172.21.15.12:6789/0 conn(0x7ff9880a2d40 0x7ff98806cb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59000/0 (socket says 172.21.15.12:59000) 2024-09-11T01:53:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98659c640 10 monclient(hunting): get_auth_request con 0x7ff98806d260 auth_method 0 2024-09-11T01:53:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98759e640 10 monclient(hunting): handle_auth_done global_id 244427 payload 274 2024-09-11T01:53:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98759e640 1 -- 172.21.15.12:0/1165452032 >> v2:172.21.15.12:3300/0 conn(0x7ff98806d260 msgr2=0x7ff98813d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:53:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98759e640 1 --2- 172.21.15.12:0/1165452032 >> v2:172.21.15.12:3300/0 conn(0x7ff98806d260 0x7ff98813d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98759e640 1 -- 172.21.15.12:0/1165452032 >> v1:172.21.15.12:6789/0 conn(0x7ff9880a2d40 legacy=0x7ff98806cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:53:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98759e640 10 monclient: _finish_hunting 0 2024-09-11T01:53:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98759e640 1 monclient: found mon.noname-b 2024-09-11T01:53:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98759e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:53:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98759e640 1 -- 172.21.15.12:0/1165452032 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9880a3920 con 0x7ff98806d670 2024-09-11T01:53:37.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.877+0000 7ff98659c640 1 --2- 172.21.15.12:0/1165452032 >> v2:172.21.15.12:3300/0 conn(0x7ff98806d260 0x7ff98813d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:53:37.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.880+0000 7ff98759e640 1 -- 172.21.15.12:0/1165452032 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff98806d670 msgr2=0x7ff98813fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:37.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.880+0000 7ff98759e640 1 --2- 172.21.15.12:0/1165452032 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff98806d670 0x7ff98813fa30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff98806a070 tx=0x7ff98002a530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efda25d1f66d6e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:37.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.880+0000 7ff985d9b640 1 -- 172.21.15.12:0/1165452032 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff980036070 con 0x7ff98806d670 2024-09-11T01:53:37.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.880+0000 7ff985d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.880+0000 7ff985d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:53:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.880+0000 7ff985d9b640 10 monclient: dump: 2024-09-11T01:53:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.892 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.880+0000 7ff985d9b640 10 monclient: _finish_auth 0 2024-09-11T01:53:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.880+0000 7ff985d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:53:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.880+0000 7ff985d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.880+0000 7ff985d9b640 1 -- 172.21.15.12:0/1165452032 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff980007800 con 0x7ff98806d670 2024-09-11T01:53:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.880+0000 7ff98d1cd640 5 monclient: authenticate success, global_id 244427 2024-09-11T01:53:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.880+0000 7ff98d1cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:53:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.880+0000 7ff985d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.880+0000 7ff985d9b640 1 -- 172.21.15.12:0/1165452032 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff980007bd0 con 0x7ff98806d670 2024-09-11T01:53:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.880+0000 7ff985d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.880+0000 7ff985d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:53:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.880+0000 7ff985d9b640 10 monclient: dump: 2024-09-11T01:53:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.894 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.880+0000 7ff98d1cd640 10 monclient: get_monmap_and_config success 2024-09-11T01:53:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.880+0000 7ff98d1cd640 4 set_mon_vals no callback set 2024-09-11T01:53:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.881+0000 7ff98d1cd640 10 monclient: shutdown 2024-09-11T01:53:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.881+0000 7ff98d1cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.881+0000 7ff98d1cd640 1 -- 172.21.15.12:0/1165452032 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff98806d670 msgr2=0x7ff98813fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.881+0000 7ff98d1cd640 1 --2- 172.21.15.12:0/1165452032 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff98806d670 0x7ff98813fa30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff98806a070 tx=0x7ff98002a530 comp rx=0 tx=0).stop 2024-09-11T01:53:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.881+0000 7ff98d1cd640 1 -- 172.21.15.12:0/1165452032 shutdown_connections 2024-09-11T01:53:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.881+0000 7ff98d1cd640 1 --2- 172.21.15.12:0/1165452032 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff98806d670 0x7ff98813fa30 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.881+0000 7ff98d1cd640 1 --2- 172.21.15.12:0/1165452032 >> v2:172.21.15.12:3300/0 conn(0x7ff98806d260 0x7ff98813d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.881+0000 7ff98d1cd640 1 -- 172.21.15.12:0/1165452032 >> 172.21.15.12:0/1165452032 conn(0x7ff9880a3e30 msgr2=0x7ff9880a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.881+0000 7ff98d1cd640 1 -- 172.21.15.12:0/1165452032 shutdown_connections 2024-09-11T01:53:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.881+0000 7ff98d1cd640 1 -- 172.21.15.12:0/1165452032 wait complete. 2024-09-11T01:53:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.881+0000 7ff98d1cd640 10 monclient: build_initial_monmap 2024-09-11T01:53:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.881+0000 7ff98d1cd640 10 monclient: monmap: 2024-09-11T01:53:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:53:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:53:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:53:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:53:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:53:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:53:37.896 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.882+0000 7ff98d1cd640 1 Processor -- start 2024-09-11T01:53:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.882+0000 7ff98d1cd640 1 -- start start 2024-09-11T01:53:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.882+0000 7ff98d1cd640 10 monclient: init 2024-09-11T01:53:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.882+0000 7ff98d1cd640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:37.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.882+0000 7ff98d1cd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:53:37.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.882+0000 7ff98d1cd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff98806d260 0x7ff988072f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.882+0000 7ff98d1cd640 10 monclient(hunting): picked mon.noname-c con 0x7ff98806d260 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:53:37.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.882+0000 7ff98d1cd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff98806d670 0x7ff988099990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.882+0000 7ff98d1cd640 10 monclient(hunting): picked mon.noname-b con 0x7ff98806d670 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:53:37.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.882+0000 7ff98d1cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9880a2d40 0x7ff988099ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.882+0000 7ff98d1cd640 10 monclient(hunting): picked mon.noname-a con 0x7ff9880a2d40 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:53:37.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.882+0000 7ff98d1cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.882+0000 7ff98d1cd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff98809ba10 con 0x7ff9880a2d40 2024-09-11T01:53:37.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98d1cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98d1cd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff9880a3120 con 0x7ff98806d670 2024-09-11T01:53:37.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98d1cd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:37.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98d1cd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff98809b3a0 con 0x7ff98806d260 2024-09-11T01:53:37.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98d1cd640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:37.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98d1cd640 10 monclient(hunting): authenticate will time out at 2224.325019s 2024-09-11T01:53:37.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff986d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff98806d260 0x7ff988072f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98759e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9880a2d40 0x7ff988099ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff986d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff98806d260 0x7ff988072f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42936/0 (socket says 172.21.15.12:42936) 2024-09-11T01:53:37.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98759e640 10 monclient(hunting): get_auth_request con 0x7ff9880a2d40 auth_method 0 2024-09-11T01:53:37.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff986d9d640 1 -- 172.21.15.12:0/930999620 learned_addr learned my addr 172.21.15.12:0/930999620 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:53:37.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff986d9d640 10 monclient(hunting): get_auth_request con 0x7ff98806d260 auth_method 0 2024-09-11T01:53:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff986d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff986d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff986d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff986d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff986d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff986d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98659c640 1 --2- 172.21.15.12:0/930999620 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff98806d670 0x7ff988099990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98759e640 10 monclient(hunting): handle_auth_done global_id 264240 payload 824 2024-09-11T01:53:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98759e640 1 -- 172.21.15.12:0/930999620 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff98806d260 msgr2=0x7ff988072f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98759e640 1 --2- 172.21.15.12:0/930999620 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff98806d260 0x7ff988072f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98759e640 1 -- 172.21.15.12:0/930999620 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff98806d670 msgr2=0x7ff988099990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98759e640 1 --2- 172.21.15.12:0/930999620 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff98806d670 0x7ff988099990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98759e640 10 monclient: _finish_hunting 0 2024-09-11T01:53:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98759e640 1 monclient: found mon.noname-a 2024-09-11T01:53:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98759e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:53:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98759e640 1 -- 172.21.15.12:0/930999620 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff988073460 con 0x7ff9880a2d40 2024-09-11T01:53:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98659c640 10 monclient: get_auth_request con 0x7ff98806d670 auth_method 0 2024-09-11T01:53:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff986d9d640 1 --2- 172.21.15.12:0/930999620 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff98806d260 0x7ff988072f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:53:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.883+0000 7ff98659c640 1 --2- 172.21.15.12:0/930999620 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff98806d670 0x7ff988099990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:53:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.885+0000 7ff98759e640 1 -- 172.21.15.12:0/930999620 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9880a2d40 msgr2=0x7ff988099ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.885+0000 7ff98759e640 1 --2- 172.21.15.12:0/930999620 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9880a2d40 0x7ff988099ed0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff980033000 tx=0x7ff980004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.885+0000 7ff96f7fe640 1 -- 172.21.15.12:0/930999620 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff980041020 con 0x7ff9880a2d40 2024-09-11T01:53:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.885+0000 7ff96f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.885+0000 7ff96f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:53:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.885+0000 7ff96f7fe640 10 monclient: dump: 2024-09-11T01:53:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.902 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.885+0000 7ff96f7fe640 10 monclient: _finish_auth 0 2024-09-11T01:53:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.885+0000 7ff96f7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:53:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.885+0000 7ff96f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.885+0000 7ff96f7fe640 1 -- 172.21.15.12:0/930999620 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff980032950 con 0x7ff9880a2d40 2024-09-11T01:53:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.885+0000 7ff98d1cd640 5 monclient: authenticate success, global_id 264240 2024-09-11T01:53:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.885+0000 7ff98d1cd640 10 monclient: _renew_subs 2024-09-11T01:53:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.885+0000 7ff98d1cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.885+0000 7ff98d1cd640 1 -- 172.21.15.12:0/930999620 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff988074470 con 0x7ff9880a2d40 2024-09-11T01:53:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.885+0000 7ff96f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.885+0000 7ff96f7fe640 1 -- 172.21.15.12:0/930999620 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff980036040 con 0x7ff9880a2d40 2024-09-11T01:53:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.885+0000 7ff96f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.885+0000 7ff96f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:53:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.885+0000 7ff96f7fe640 10 monclient: dump: 2024-09-11T01:53:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:37.903 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.886+0000 7ff98d1cd640 10 monclient: _renew_subs 2024-09-11T01:53:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.886+0000 7ff98d1cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.886+0000 7ff98d1cd640 1 -- 172.21.15.12:0/930999620 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff988074a30 con 0x7ff9880a2d40 2024-09-11T01:53:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.885+0000 7ff98559a640 4 set_mon_vals no callback set 2024-09-11T01:53:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.886+0000 7ff96f7fe640 1 -- 172.21.15.12:0/930999620 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff98003c470 con 0x7ff9880a2d40 2024-09-11T01:53:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.887+0000 7ff96f7fe640 1 --2- 172.21.15.12:0/930999620 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff97403e210 0x7ff9740406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.887+0000 7ff98d1cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.887+0000 7ff98d1cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.887+0000 7ff98d1cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.887+0000 7ff96f7fe640 1 -- 172.21.15.12:0/930999620 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(832..832 src has 676..832) ==== 4815+0+0 (secure 0 0 0) 0x7ff980073d80 con 0x7ff9880a2d40 2024-09-11T01:53:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.887+0000 7ff98d1cd640 1 -- 172.21.15.12:0/930999620 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff98806cb50 con 0x7ff9880a2d40 2024-09-11T01:53:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.887+0000 7ff986d9d640 1 --2- 172.21.15.12:0/930999620 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff97403e210 0x7ff9740406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.887+0000 7ff986d9d640 10 monclient: get_auth_request con 0x7ff97403e210 auth_method 0 2024-09-11T01:53:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.890+0000 7ff986d9d640 1 --2- 172.21.15.12:0/930999620 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff97403e210 0x7ff9740406d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7ff978007fd0 tx=0x7ff978007f20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:37.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.891+0000 7ff96f7fe640 1 -- 172.21.15.12:0/930999620 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff98004b030 con 0x7ff9880a2d40 2024-09-11T01:53:37.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.891+0000 7ff96f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:37.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:37.891+0000 7ff96f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:53:38.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:38.145+0000 7ff98d1cd640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:53:38.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:38.145+0000 7ff98d1cd640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:53:38.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:38.145+0000 7ff98d1cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:53:38.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:38.145+0000 7ff98d1cd640 1 -- 172.21.15.12:0/930999620 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7ff98806e520 con 0x7ff9880a2d40 2024-09-11T01:53:38.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:38.146+0000 7ff96f7fe640 1 -- 172.21.15.12:0/930999620 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7ff988073460 con 0x7ff9880a2d40 2024-09-11T01:53:38.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:38.146+0000 7ff96f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:53:38.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:38.146+0000 7ff96f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:53:38.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.146+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.146+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=11,kv=12,logm=47} crc {auth=2250009983,config=2586702516,health=3724698710,kv=3263697420,logm=3170195735}) 2024-09-11T01:53:38.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.146+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=30,config=2,health=12,kv=12,logm=44} crc {auth=3356257432,config=2586702516,health=1443762948,kv=3263697420,logm=1664572750}) 2024-09-11T01:53:38.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.146+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.146+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=11,kv=12,logm=47} crc {auth=2250009983,config=2586702516,health=3724698710,kv=3263697420,logm=3170195735}) 2024-09-11T01:53:38.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.146+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=12,kv=12,logm=44} crc {auth=3950603835,config=2586702516,health=2612668575,kv=3263697420,logm=1465087273}) 2024-09-11T01:53:38.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.147+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.147+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3188413789}) 2024-09-11T01:53:38.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.147+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=950901562}) 2024-09-11T01:53:38.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.147+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.147+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3188413789}) 2024-09-11T01:53:38.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.147+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=470015890}) 2024-09-11T01:53:38.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:38.148+0000 7ff98d1cd640 1 -- 172.21.15.12:0/930999620 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff97403e210 msgr2=0x7ff9740406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:38.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:38.148+0000 7ff98d1cd640 1 --2- 172.21.15.12:0/930999620 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff97403e210 0x7ff9740406d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7ff978007fd0 tx=0x7ff978007f20 comp rx=0 tx=0).stop 2024-09-11T01:53:38.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:38.148+0000 7ff98d1cd640 10 monclient: shutdown 2024-09-11T01:53:38.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:38.148+0000 7ff98d1cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:38.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:38.148+0000 7ff98d1cd640 1 -- 172.21.15.12:0/930999620 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9880a2d40 msgr2=0x7ff988099ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:38.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:38.148+0000 7ff98d1cd640 1 --2- 172.21.15.12:0/930999620 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9880a2d40 0x7ff988099ed0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff980033000 tx=0x7ff980004a40 comp rx=0 tx=0).stop 2024-09-11T01:53:38.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:38.148+0000 7ff98d1cd640 1 -- 172.21.15.12:0/930999620 shutdown_connections 2024-09-11T01:53:38.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:38.148+0000 7ff98d1cd640 1 --2- 172.21.15.12:0/930999620 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff97403e210 0x7ff9740406d0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:38.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:38.148+0000 7ff98d1cd640 1 --2- 172.21.15.12:0/930999620 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9880a2d40 0x7ff988099ed0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:38.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:38.148+0000 7ff98d1cd640 1 --2- 172.21.15.12:0/930999620 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff98806d670 0x7ff988099990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:38.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:38.148+0000 7ff98d1cd640 1 --2- 172.21.15.12:0/930999620 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff98806d260 0x7ff988072f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:38.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:38.148+0000 7ff98d1cd640 1 -- 172.21.15.12:0/930999620 >> 172.21.15.12:0/930999620 conn(0x7ff9880a3e30 msgr2=0x7ff9880a5210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:38.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:38.148+0000 7ff98d1cd640 1 -- 172.21.15.12:0/930999620 shutdown_connections 2024-09-11T01:53:38.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:38.148+0000 7ff98d1cd640 1 -- 172.21.15.12:0/930999620 wait complete. 2024-09-11T01:53:38.154 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.148+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.154 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.148+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2765110540}) 2024-09-11T01:53:38.154 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.148+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=660859711}) 2024-09-11T01:53:38.154 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.148+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.154 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.148+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2765110540}) 2024-09-11T01:53:38.154 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.148+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2991969119}) 2024-09-11T01:53:38.154 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.149+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.154 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.149+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2084555862}) 2024-09-11T01:53:38.154 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.149+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3813868899}) 2024-09-11T01:53:38.154 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.149+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.154 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.149+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2084555862}) 2024-09-11T01:53:38.154 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.149+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3782921487}) 2024-09-11T01:53:38.155 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.151+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.155 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.151+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4151142806}) 2024-09-11T01:53:38.155 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.151+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=878331704}) 2024-09-11T01:53:38.155 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.151+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.155 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.151+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4151142806}) 2024-09-11T01:53:38.155 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.151+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1734940585}) 2024-09-11T01:53:38.155 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.151+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.155 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.151+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1815991040}) 2024-09-11T01:53:38.155 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.151+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1096682144}) 2024-09-11T01:53:38.155 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.151+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.155 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.151+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1815991040}) 2024-09-11T01:53:38.155 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.151+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2275962808}) 2024-09-11T01:53:38.155 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.152+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.156 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.152+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3867842121}) 2024-09-11T01:53:38.156 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.152+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2201855309}) 2024-09-11T01:53:38.156 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.152+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.156 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.152+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3867842121}) 2024-09-11T01:53:38.156 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.152+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3434549152}) 2024-09-11T01:53:38.158 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.153+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.158 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.153+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3567358328}) 2024-09-11T01:53:38.158 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.153+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1845808865}) 2024-09-11T01:53:38.158 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.153+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.158 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.153+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3567358328}) 2024-09-11T01:53:38.158 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.153+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3187037612}) 2024-09-11T01:53:38.158 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.153+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.158 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.153+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1413549451}) 2024-09-11T01:53:38.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.153+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3013589210}) 2024-09-11T01:53:38.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.153+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.153+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1413549451}) 2024-09-11T01:53:38.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.153+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2880917457}) 2024-09-11T01:53:38.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.154+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.154+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3130203926}) 2024-09-11T01:53:38.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.154+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1191551500}) 2024-09-11T01:53:38.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.154+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.154+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3130203926}) 2024-09-11T01:53:38.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.154+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3931862275}) 2024-09-11T01:53:38.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.155+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.155+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2340023398}) 2024-09-11T01:53:38.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.155+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4220631267}) 2024-09-11T01:53:38.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.155+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.155+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2340023398}) 2024-09-11T01:53:38.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.155+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3666235284}) 2024-09-11T01:53:38.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.155+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.155+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3547958654}) 2024-09-11T01:53:38.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.155+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=97340442}) 2024-09-11T01:53:38.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.155+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.155+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3547958654}) 2024-09-11T01:53:38.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.155+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=437563474}) 2024-09-11T01:53:38.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.156+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.156+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=64992897}) 2024-09-11T01:53:38.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.156+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1979180606}) 2024-09-11T01:53:38.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.156+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.156+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=64992897}) 2024-09-11T01:53:38.161 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.156+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4283558375}) 2024-09-11T01:53:38.161 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.157+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.162 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.157+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1087221963}) 2024-09-11T01:53:38.162 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.157+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2079709885}) 2024-09-11T01:53:38.162 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.157+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.162 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.157+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1087221963}) 2024-09-11T01:53:38.162 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.157+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3445774726}) 2024-09-11T01:53:38.162 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.157+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.162 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.157+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1067688631}) 2024-09-11T01:53:38.162 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.157+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2902724371}) 2024-09-11T01:53:38.162 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.157+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.162 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.157+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1067688631}) 2024-09-11T01:53:38.162 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.157+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1186340426}) 2024-09-11T01:53:38.163 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:53:38.167 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.165+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.167 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.165+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1850356816}) 2024-09-11T01:53:38.167 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.165+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=98409398}) 2024-09-11T01:53:38.167 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.165+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.168 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.165+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1850356816}) 2024-09-11T01:53:38.168 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.166+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3872295156}) 2024-09-11T01:53:38.169 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.166+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.169 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.166+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3998361771}) 2024-09-11T01:53:38.169 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.167+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=84692636}) 2024-09-11T01:53:38.169 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.167+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.169 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.167+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3998361771}) 2024-09-11T01:53:38.169 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.167+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=731731331}) 2024-09-11T01:53:38.169 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.167+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.169 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.167+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=705766277}) 2024-09-11T01:53:38.169 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.167+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=105411753}) 2024-09-11T01:53:38.169 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.167+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.170 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.167+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=705766277}) 2024-09-11T01:53:38.170 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.167+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1698596717}) 2024-09-11T01:53:38.171 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.168+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.171 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.168+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3191274185}) 2024-09-11T01:53:38.171 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.168+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=622307097}) 2024-09-11T01:53:38.171 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.168+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.171 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.168+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3191274185}) 2024-09-11T01:53:38.171 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.168+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=750596683}) 2024-09-11T01:53:38.172 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.169+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.172 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.169+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3727815384}) 2024-09-11T01:53:38.172 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.169+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=130366074}) 2024-09-11T01:53:38.172 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.169+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.172 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.169+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3727815384}) 2024-09-11T01:53:38.172 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.169+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=190063522}) 2024-09-11T01:53:38.172 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.169+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.172 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.169+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1406171289}) 2024-09-11T01:53:38.172 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.169+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=203280541}) 2024-09-11T01:53:38.172 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.169+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.172 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.169+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1406171289}) 2024-09-11T01:53:38.172 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.169+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=89540955}) 2024-09-11T01:53:38.175 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.170+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.175 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.170+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2351239641}) 2024-09-11T01:53:38.175 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.170+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3692927467}) 2024-09-11T01:53:38.175 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.170+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.175 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.170+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2351239641}) 2024-09-11T01:53:38.175 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.170+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3585612572}) 2024-09-11T01:53:38.175 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.171+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.175 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.171+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1363486968}) 2024-09-11T01:53:38.175 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.171+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2697604818}) 2024-09-11T01:53:38.175 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.171+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.175 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.171+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1363486968}) 2024-09-11T01:53:38.175 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.171+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2753445721}) 2024-09-11T01:53:38.176 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.172+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.176 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.172+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821983925}) 2024-09-11T01:53:38.176 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.172+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3329087141}) 2024-09-11T01:53:38.176 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.172+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.176 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.172+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821983925}) 2024-09-11T01:53:38.176 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.172+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=686621595}) 2024-09-11T01:53:38.176 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.172+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.176 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.172+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1955568603}) 2024-09-11T01:53:38.176 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.172+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=38934710}) 2024-09-11T01:53:38.176 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.172+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.176 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.172+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1955568603}) 2024-09-11T01:53:38.176 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.172+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1316379429}) 2024-09-11T01:53:38.176 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.173+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.176 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.173+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=496646100}) 2024-09-11T01:53:38.177 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.173+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1709970883}) 2024-09-11T01:53:38.177 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.173+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.177 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.173+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=496646100}) 2024-09-11T01:53:38.177 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.173+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3698099689}) 2024-09-11T01:53:38.181 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.174+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.184 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.174+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=115088266}) 2024-09-11T01:53:38.184 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.174+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3121487177}) 2024-09-11T01:53:38.184 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.174+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.174+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=115088266}) 2024-09-11T01:53:38.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.174+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3171596217}) 2024-09-11T01:53:38.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.174+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.174+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1201826222}) 2024-09-11T01:53:38.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.174+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1441579022}) 2024-09-11T01:53:38.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.174+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.174+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1201826222}) 2024-09-11T01:53:38.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.174+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1205025988}) 2024-09-11T01:53:38.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.175+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.175+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3773741652}) 2024-09-11T01:53:38.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.175+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=691480208}) 2024-09-11T01:53:38.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.175+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.175+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3773741652}) 2024-09-11T01:53:38.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.175+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3684510351}) 2024-09-11T01:53:38.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.176+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.176+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4002356525}) 2024-09-11T01:53:38.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.176+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4271341346}) 2024-09-11T01:53:38.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.176+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.176+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4002356525}) 2024-09-11T01:53:38.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.176+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=320229461}) 2024-09-11T01:53:38.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.177+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.177+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=742480322}) 2024-09-11T01:53:38.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.177+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3854278653}) 2024-09-11T01:53:38.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.177+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.177+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=742480322}) 2024-09-11T01:53:38.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.177+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1447798712}) 2024-09-11T01:53:38.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.177+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.177+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3643195193}) 2024-09-11T01:53:38.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.177+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3156225438}) 2024-09-11T01:53:38.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.177+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.177+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3643195193}) 2024-09-11T01:53:38.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.177+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3288732892}) 2024-09-11T01:53:38.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.178+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.178+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2986902801}) 2024-09-11T01:53:38.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.178+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3823268882}) 2024-09-11T01:53:38.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.178+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.178+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2986902801}) 2024-09-11T01:53:38.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.178+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1111015225}) 2024-09-11T01:53:38.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.179+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.179+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=513249675}) 2024-09-11T01:53:38.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.179+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1174699819}) 2024-09-11T01:53:38.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.179+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.179+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=513249675}) 2024-09-11T01:53:38.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.179+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1902504370}) 2024-09-11T01:53:38.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.179+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.179+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3410674526}) 2024-09-11T01:53:38.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.179+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1474270253}) 2024-09-11T01:53:38.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.180+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.180+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3410674526}) 2024-09-11T01:53:38.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.180+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3017330003}) 2024-09-11T01:53:38.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.180+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.180+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3131413488}) 2024-09-11T01:53:38.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.180+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2284303105}) 2024-09-11T01:53:38.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.180+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.180+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3131413488}) 2024-09-11T01:53:38.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.180+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3177370570}) 2024-09-11T01:53:38.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.181+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.181+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1520771958}) 2024-09-11T01:53:38.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.181+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2468510674}) 2024-09-11T01:53:38.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.181+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.181+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1520771958}) 2024-09-11T01:53:38.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.181+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2106931204}) 2024-09-11T01:53:38.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.182+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.182+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=65384180}) 2024-09-11T01:53:38.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.182+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3824774601}) 2024-09-11T01:53:38.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.182+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.182+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=65384180}) 2024-09-11T01:53:38.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.182+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2359747762}) 2024-09-11T01:53:38.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.182+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.182+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2387991638}) 2024-09-11T01:53:38.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.182+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3481749661}) 2024-09-11T01:53:38.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.183+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.183+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2387991638}) 2024-09-11T01:53:38.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.183+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4028005241}) 2024-09-11T01:53:38.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.183+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.183+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2329438293}) 2024-09-11T01:53:38.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.183+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2796158997}) 2024-09-11T01:53:38.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.183+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.183+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2329438293}) 2024-09-11T01:53:38.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.183+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3239350600}) 2024-09-11T01:53:38.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.184+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.184+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2445404767}) 2024-09-11T01:53:38.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.184+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3037201089}) 2024-09-11T01:53:38.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.184+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.184+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2445404767}) 2024-09-11T01:53:38.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.184+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1013687379}) 2024-09-11T01:53:38.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.185+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.185+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3889276692}) 2024-09-11T01:53:38.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.185+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=248084481}) 2024-09-11T01:53:38.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.185+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.185+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3889276692}) 2024-09-11T01:53:38.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.185+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3666144256}) 2024-09-11T01:53:38.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.186+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.186+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1361519267}) 2024-09-11T01:53:38.196 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.186+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=468034556}) 2024-09-11T01:53:38.196 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.186+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.196 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.186+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1361519267}) 2024-09-11T01:53:38.196 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.186+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=915362109}) 2024-09-11T01:53:38.196 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.186+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.196 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.186+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1962396060}) 2024-09-11T01:53:38.196 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.186+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3734272986}) 2024-09-11T01:53:38.196 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.186+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.196 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.186+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1962396060}) 2024-09-11T01:53:38.196 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.186+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=679058261}) 2024-09-11T01:53:38.313 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.311+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.313 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.311+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4233819599}) 2024-09-11T01:53:38.313 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.311+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3388328750}) 2024-09-11T01:53:38.313 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.311+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.313 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.311+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4233819599}) 2024-09-11T01:53:38.313 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.311+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2003992503}) 2024-09-11T01:53:38.315 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.314+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.315 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.314+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2740207902}) 2024-09-11T01:53:38.316 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.314+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1919887303}) 2024-09-11T01:53:38.316 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.314+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.316 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.314+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2740207902}) 2024-09-11T01:53:38.316 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.314+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=431852580}) 2024-09-11T01:53:38.319 INFO:tasks.rados.rados.0.smithi071.stdout:3638: done (3 left) 2024-09-11T01:53:38.319 INFO:tasks.rados.rados.0.smithi071.stdout:3639: read oid 1 snap -1 2024-09-11T01:53:38.319 INFO:tasks.rados.rados.0.smithi071.stdout:3639: expect (ObjNum 1154 snap 366 seq_num 1154) 2024-09-11T01:53:38.319 INFO:tasks.rados.rados.0.smithi071.stdout:3640: write oid 5 current snap is 415 2024-09-11T01:53:38.319 INFO:tasks.rados.rados.0.smithi071.stdout:3640: seq_num 1255 ranges {692785=435740,1672784=793765,2897503=195282} 2024-09-11T01:53:38.324 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.322+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.324 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.322+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3830554272}) 2024-09-11T01:53:38.324 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.322+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=903487989}) 2024-09-11T01:53:38.324 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.322+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.324 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.322+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3830554272}) 2024-09-11T01:53:38.324 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.322+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1912960628}) 2024-09-11T01:53:38.325 INFO:tasks.rados.rados.0.smithi071.stdout:3640: writing smithi07134777-5 from 692785 to 1128525 tid 1 2024-09-11T01:53:38.334 INFO:tasks.rados.rados.0.smithi071.stdout:3640: writing smithi07134777-5 from 1672784 to 2466549 tid 2 2024-09-11T01:53:38.338 INFO:tasks.rados.rados.0.smithi071.stdout:3640: writing smithi07134777-5 from 2897503 to 3092785 tid 3 2024-09-11T01:53:38.338 INFO:tasks.rados.rados.0.smithi071.stdout:3641: snap_remove snap 396 2024-09-11T01:53:38.430 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.428+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.430 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.428+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2674878987}) 2024-09-11T01:53:38.431 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.429+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3808237097}) 2024-09-11T01:53:38.431 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.429+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.431 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.429+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2674878987}) 2024-09-11T01:53:38.431 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.429+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=372641151}) 2024-09-11T01:53:38.439 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.437+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.439 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.437+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3249073190}) 2024-09-11T01:53:38.439 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.437+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=113370197}) 2024-09-11T01:53:38.439 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.437+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.439 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.437+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3249073190}) 2024-09-11T01:53:38.439 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.437+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1902615366}) 2024-09-11T01:53:38.921 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.919+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.921 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.919+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3940516513}) 2024-09-11T01:53:38.921 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.919+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=586891025}) 2024-09-11T01:53:38.921 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.919+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.921 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.919+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3940516513}) 2024-09-11T01:53:38.921 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.919+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1410877360}) 2024-09-11T01:53:38.922 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.920+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.922 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.920+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=294421244}) 2024-09-11T01:53:38.922 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.920+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=36448985}) 2024-09-11T01:53:38.922 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.920+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.923 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.920+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=294421244}) 2024-09-11T01:53:38.923 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.920+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2771198081}) 2024-09-11T01:53:38.923 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.921+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.923 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.921+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2190613877}) 2024-09-11T01:53:38.923 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.921+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2875984632}) 2024-09-11T01:53:38.923 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.921+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.923 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.921+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2190613877}) 2024-09-11T01:53:38.923 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.921+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1528829879}) 2024-09-11T01:53:38.924 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.922+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.924 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.922+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3686782990}) 2024-09-11T01:53:38.924 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.922+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=945199468}) 2024-09-11T01:53:38.924 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.922+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.924 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.922+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3686782990}) 2024-09-11T01:53:38.925 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.922+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=664785006}) 2024-09-11T01:53:38.925 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.922+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.925 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.922+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=569344083}) 2024-09-11T01:53:38.925 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.922+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2073947244}) 2024-09-11T01:53:38.925 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.922+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.925 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.922+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=569344083}) 2024-09-11T01:53:38.925 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.922+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2861962776}) 2024-09-11T01:53:38.928 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.923+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.928 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.923+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4072154179}) 2024-09-11T01:53:38.928 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.923+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=857366321}) 2024-09-11T01:53:38.928 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.923+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.928 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.923+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4072154179}) 2024-09-11T01:53:38.928 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.923+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3646428521}) 2024-09-11T01:53:38.928 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.924+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.928 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.924+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4249994082}) 2024-09-11T01:53:38.928 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.924+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=238969331}) 2024-09-11T01:53:38.928 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.924+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.928 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.924+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4249994082}) 2024-09-11T01:53:38.928 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.924+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3315059064}) 2024-09-11T01:53:38.929 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.925+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.929 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.925+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2593157889}) 2024-09-11T01:53:38.929 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.925+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=245416844}) 2024-09-11T01:53:38.929 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.925+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.929 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.925+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2593157889}) 2024-09-11T01:53:38.929 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.925+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1057707033}) 2024-09-11T01:53:38.929 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.926+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.929 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.926+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3572964647}) 2024-09-11T01:53:38.929 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.926+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=397299989}) 2024-09-11T01:53:38.929 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.926+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.929 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.926+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3572964647}) 2024-09-11T01:53:38.929 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.926+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3845310176}) 2024-09-11T01:53:38.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.927+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.927+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=743887027}) 2024-09-11T01:53:38.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.927+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2896468062}) 2024-09-11T01:53:38.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.927+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.927+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=743887027}) 2024-09-11T01:53:38.932 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.927+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2022362015}) 2024-09-11T01:53:38.932 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.928+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.932 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.928+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4262454190}) 2024-09-11T01:53:38.932 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.928+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1729257621}) 2024-09-11T01:53:38.932 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.928+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.932 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.928+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4262454190}) 2024-09-11T01:53:38.932 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.928+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=920427814}) 2024-09-11T01:53:38.932 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.928+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.932 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.928+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3753776398}) 2024-09-11T01:53:38.932 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.929+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3169595164}) 2024-09-11T01:53:38.932 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.929+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.932 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.929+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3753776398}) 2024-09-11T01:53:38.932 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.929+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3946604909}) 2024-09-11T01:53:38.933 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.929+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.933 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.929+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=669366588}) 2024-09-11T01:53:38.933 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.929+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2065498987}) 2024-09-11T01:53:38.933 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.929+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.933 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.929+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=669366588}) 2024-09-11T01:53:38.933 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.929+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3754233842}) 2024-09-11T01:53:38.935 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.930+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.935 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.930+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4157444262}) 2024-09-11T01:53:38.935 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.930+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1777248197}) 2024-09-11T01:53:38.935 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.930+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.935 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.930+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4157444262}) 2024-09-11T01:53:38.935 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.930+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3999816837}) 2024-09-11T01:53:38.935 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.931+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.935 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.931+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=780135388}) 2024-09-11T01:53:38.935 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.931+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3233984509}) 2024-09-11T01:53:38.935 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.931+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.935 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.931+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=780135388}) 2024-09-11T01:53:38.935 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.931+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2966836631}) 2024-09-11T01:53:38.936 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.932+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.936 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.932+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2959674406}) 2024-09-11T01:53:38.936 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.932+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=252969811}) 2024-09-11T01:53:38.936 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.932+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.936 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.932+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2959674406}) 2024-09-11T01:53:38.936 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.932+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=70437318}) 2024-09-11T01:53:38.936 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.933+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.936 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.933+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3373637384}) 2024-09-11T01:53:38.936 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.933+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3271932679}) 2024-09-11T01:53:38.936 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.933+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.936 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.933+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3373637384}) 2024-09-11T01:53:38.936 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.933+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=560771681}) 2024-09-11T01:53:38.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.934+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.934+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4145318478}) 2024-09-11T01:53:38.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.934+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1041476159}) 2024-09-11T01:53:38.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.934+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.934+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4145318478}) 2024-09-11T01:53:38.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.934+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3140670665}) 2024-09-11T01:53:38.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.935+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.935+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=709175372}) 2024-09-11T01:53:38.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.935+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3259756984}) 2024-09-11T01:53:38.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.935+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.935+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=709175372}) 2024-09-11T01:53:38.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.935+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=621946540}) 2024-09-11T01:53:38.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.935+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.935+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4048141791}) 2024-09-11T01:53:38.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.935+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1591510581}) 2024-09-11T01:53:38.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.935+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.935+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4048141791}) 2024-09-11T01:53:38.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.935+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4214115393}) 2024-09-11T01:53:38.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.936+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.936+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4274080976}) 2024-09-11T01:53:38.940 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.936+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2987419704}) 2024-09-11T01:53:38.940 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.936+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.940 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.936+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4274080976}) 2024-09-11T01:53:38.940 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.936+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=320651078}) 2024-09-11T01:53:38.942 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.937+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.942 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.937+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4027745414}) 2024-09-11T01:53:38.942 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.937+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2589621105}) 2024-09-11T01:53:38.942 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.937+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.942 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.937+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4027745414}) 2024-09-11T01:53:38.942 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.937+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3410500412}) 2024-09-11T01:53:38.942 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.938+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.942 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.938+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1882616001}) 2024-09-11T01:53:38.942 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.938+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2534853005}) 2024-09-11T01:53:38.942 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.938+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.942 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.938+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1882616001}) 2024-09-11T01:53:38.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.938+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3310281478}) 2024-09-11T01:53:38.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.939+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.939+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=466306770}) 2024-09-11T01:53:38.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.939+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1085059771}) 2024-09-11T01:53:38.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.939+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.939+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=466306770}) 2024-09-11T01:53:38.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.939+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3629676753}) 2024-09-11T01:53:38.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.940+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.940+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4036847555}) 2024-09-11T01:53:38.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.940+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=390923360}) 2024-09-11T01:53:38.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.940+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.940+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4036847555}) 2024-09-11T01:53:38.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.940+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3067909596}) 2024-09-11T01:53:38.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.941+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.941+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2324736464}) 2024-09-11T01:53:38.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.941+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1846746439}) 2024-09-11T01:53:38.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.941+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.941+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2324736464}) 2024-09-11T01:53:38.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.941+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1602781992}) 2024-09-11T01:53:38.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.942+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.942+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90590801}) 2024-09-11T01:53:38.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.942+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2496875454}) 2024-09-11T01:53:38.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.942+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.942+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90590801}) 2024-09-11T01:53:38.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.942+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1426536080}) 2024-09-11T01:53:38.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.942+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.942+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1392265095}) 2024-09-11T01:53:38.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.942+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=287915420}) 2024-09-11T01:53:38.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.942+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.942+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1392265095}) 2024-09-11T01:53:38.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.942+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1790919163}) 2024-09-11T01:53:38.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.943+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.943+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2586328288}) 2024-09-11T01:53:38.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.943+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1313568540}) 2024-09-11T01:53:38.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.943+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.943+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2586328288}) 2024-09-11T01:53:38.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.943+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1915092766}) 2024-09-11T01:53:38.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.944+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.944+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=83554578}) 2024-09-11T01:53:38.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.944+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1690026267}) 2024-09-11T01:53:38.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.944+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.944+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=83554578}) 2024-09-11T01:53:38.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.944+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1535929569}) 2024-09-11T01:53:38.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.945+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.945+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1135211056}) 2024-09-11T01:53:38.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.945+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=258091864}) 2024-09-11T01:53:38.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.945+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.945+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1135211056}) 2024-09-11T01:53:38.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.945+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3253622793}) 2024-09-11T01:53:38.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.946+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.946+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3748252063}) 2024-09-11T01:53:38.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.946+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2441201664}) 2024-09-11T01:53:38.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.946+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.946+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3748252063}) 2024-09-11T01:53:38.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.946+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1625008243}) 2024-09-11T01:53:38.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.947+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.947+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4100824009}) 2024-09-11T01:53:38.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.947+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3750481298}) 2024-09-11T01:53:38.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.947+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.947+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4100824009}) 2024-09-11T01:53:38.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.947+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=386102408}) 2024-09-11T01:53:38.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.948+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.948+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1149715970}) 2024-09-11T01:53:38.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.948+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1675753301}) 2024-09-11T01:53:38.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.948+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.948+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1149715970}) 2024-09-11T01:53:38.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.948+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4095672815}) 2024-09-11T01:53:38.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.949+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.949+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938859525}) 2024-09-11T01:53:38.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.949+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=29759645}) 2024-09-11T01:53:38.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.949+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.949+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938859525}) 2024-09-11T01:53:38.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.949+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1009740154}) 2024-09-11T01:53:38.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.950+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.950+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1004328299}) 2024-09-11T01:53:38.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.950+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2565888519}) 2024-09-11T01:53:38.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.950+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.950+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1004328299}) 2024-09-11T01:53:38.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.950+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1951823888}) 2024-09-11T01:53:38.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.950+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.950+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2568121439}) 2024-09-11T01:53:38.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.950+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1438802264}) 2024-09-11T01:53:38.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.950+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.950+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2568121439}) 2024-09-11T01:53:38.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.950+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2126935289}) 2024-09-11T01:53:38.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.951+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.951+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2319170703}) 2024-09-11T01:53:38.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.951+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3850545837}) 2024-09-11T01:53:38.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.951+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.951+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2319170703}) 2024-09-11T01:53:38.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.951+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3013587376}) 2024-09-11T01:53:38.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.952+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.952+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=373351612}) 2024-09-11T01:53:38.956 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.952+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3769398334}) 2024-09-11T01:53:38.956 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.952+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.956 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.952+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=373351612}) 2024-09-11T01:53:38.956 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.952+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1522629029}) 2024-09-11T01:53:38.958 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.953+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.958 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.953+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1702159314}) 2024-09-11T01:53:38.958 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.953+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=561368818}) 2024-09-11T01:53:38.958 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.953+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.958 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.953+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1702159314}) 2024-09-11T01:53:38.959 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.953+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2881105167}) 2024-09-11T01:53:38.959 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.954+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.959 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.954+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4257509294}) 2024-09-11T01:53:38.959 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.954+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3076352068}) 2024-09-11T01:53:38.959 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.954+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.959 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.954+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4257509294}) 2024-09-11T01:53:38.959 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.954+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2929875262}) 2024-09-11T01:53:38.959 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.954+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.959 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.954+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1575218077}) 2024-09-11T01:53:38.959 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.954+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3348045344}) 2024-09-11T01:53:38.959 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.954+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.959 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.954+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1575218077}) 2024-09-11T01:53:38.959 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.954+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1416609409}) 2024-09-11T01:53:38.959 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.955+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.955+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3627042371}) 2024-09-11T01:53:38.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.955+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2138097317}) 2024-09-11T01:53:38.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.955+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.955+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3627042371}) 2024-09-11T01:53:38.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.955+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1791375829}) 2024-09-11T01:53:38.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.956+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.956+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1929936975}) 2024-09-11T01:53:38.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.956+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3351511524}) 2024-09-11T01:53:38.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.956+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.956+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1929936975}) 2024-09-11T01:53:38.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.956+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4207100611}) 2024-09-11T01:53:38.960 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.956+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.961 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.956+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589287954}) 2024-09-11T01:53:38.961 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.956+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2922878348}) 2024-09-11T01:53:38.961 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.956+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.961 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.956+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589287954}) 2024-09-11T01:53:38.961 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.956+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3663853425}) 2024-09-11T01:53:38.963 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.957+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.963 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.957+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350062162}) 2024-09-11T01:53:38.963 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.957+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=704072778}) 2024-09-11T01:53:38.963 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.957+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.963 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.957+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350062162}) 2024-09-11T01:53:38.963 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.957+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=396133497}) 2024-09-11T01:53:38.963 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.958+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.964 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.958+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=518029604}) 2024-09-11T01:53:38.964 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.958+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2469283318}) 2024-09-11T01:53:38.964 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.958+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.964 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.958+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=518029604}) 2024-09-11T01:53:38.964 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.958+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3877270130}) 2024-09-11T01:53:38.964 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.959+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.964 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.959+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3293874760}) 2024-09-11T01:53:38.964 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.959+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3877342858}) 2024-09-11T01:53:38.964 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.959+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.964 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.959+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3293874760}) 2024-09-11T01:53:38.964 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.959+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=769217157}) 2024-09-11T01:53:38.964 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.959+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.964 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.959+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1917742983}) 2024-09-11T01:53:38.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.959+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3833563083}) 2024-09-11T01:53:38.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.959+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.959+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1917742983}) 2024-09-11T01:53:38.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.959+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2528261030}) 2024-09-11T01:53:38.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.960+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.960+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4024049280}) 2024-09-11T01:53:38.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.960+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1076640057}) 2024-09-11T01:53:38.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.960+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.960+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4024049280}) 2024-09-11T01:53:38.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.960+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3367742180}) 2024-09-11T01:53:38.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.961+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.961+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=466886087}) 2024-09-11T01:53:38.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.961+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1619299751}) 2024-09-11T01:53:38.965 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.961+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:38.966 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.961+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=466886087}) 2024-09-11T01:53:38.966 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:38.961+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4004288081}) 2024-09-11T01:53:39.064 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.062+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:39.064 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.062+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4102228847}) 2024-09-11T01:53:39.064 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.063+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2755849948}) 2024-09-11T01:53:39.065 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.063+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:39.065 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.063+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4102228847}) 2024-09-11T01:53:39.065 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.063+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2618071357}) 2024-09-11T01:53:39.066 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.063+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:39.066 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.063+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3138296220}) 2024-09-11T01:53:39.066 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.064+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4081408900}) 2024-09-11T01:53:39.066 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.064+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:39.066 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.064+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3138296220}) 2024-09-11T01:53:39.066 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.064+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2332951056}) 2024-09-11T01:53:39.066 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.064+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:39.066 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.064+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1551715628}) 2024-09-11T01:53:39.066 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.064+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1502372153}) 2024-09-11T01:53:39.067 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.064+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:39.067 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.064+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1551715628}) 2024-09-11T01:53:39.067 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.064+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=5564837}) 2024-09-11T01:53:39.068 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.065+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:39.068 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.065+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=9,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=16,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=42} crc {logm=1052178929,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=135905440,mon_config_key=4195014768,monmap=3140514009,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=2731214370,osd_snap=3064847345}) 2024-09-11T01:53:39.068 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.065+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=10,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=15,monmap=2,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=44} crc {logm=3305282973,mdsmap=3760429986,mgr=4204056853,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=3475191263,mon_config_key=2379583869,monmap=1324671969,nvmeofgw=1596038039,osd_metadata=2942852742,osd_pg_creating=2731214370,osd_snap=4197674456}) 2024-09-11T01:53:39.068 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.065+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:39.068 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.065+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=9,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=16,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=42} crc {logm=1052178929,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=135905440,mon_config_key=4195014768,monmap=3140514009,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=2731214370,osd_snap=3064847345}) 2024-09-11T01:53:39.068 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.065+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=10,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=42} crc {logm=3305282973,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2191625971,mon_config_key=2518676092,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=2731214370,osd_snap=4229016228}) 2024-09-11T01:53:39.069 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.066+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:39.069 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.066+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=2731820602}) 2024-09-11T01:53:39.069 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.066+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=100} crc {osd_snap=2990784355}) 2024-09-11T01:53:39.069 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.066+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:39.069 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.066+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=2731820602}) 2024-09-11T01:53:39.069 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.066+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=100} crc {osd_snap=1921100770}) 2024-09-11T01:53:39.069 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.066+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:39.069 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.066+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=28,osdmap=72} crc {osd_snap=439231340,osdmap=594989972}) 2024-09-11T01:53:39.069 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.066+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=28,osdmap=72} crc {osd_snap=251493801,osdmap=2222958931}) 2024-09-11T01:53:39.069 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.066+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:39.069 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.066+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=28,osdmap=72} crc {osd_snap=439231340,osdmap=594989972}) 2024-09-11T01:53:39.069 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.066+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=28,osdmap=72} crc {osd_snap=3234796726,osdmap=4200161520}) 2024-09-11T01:53:39.070 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.068+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:39.070 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.068+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2903039352}) 2024-09-11T01:53:39.070 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.068+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1715736529}) 2024-09-11T01:53:39.070 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.068+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:39.070 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.068+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2903039352}) 2024-09-11T01:53:39.071 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.068+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1581861322}) 2024-09-11T01:53:39.075 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.073+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:39.075 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.073+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=67} crc {osdmap=1740818871}) 2024-09-11T01:53:39.075 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.073+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=63} crc {osdmap=1007711871}) 2024-09-11T01:53:39.075 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.073+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:53:39.075 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.073+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=67} crc {osdmap=1740818871}) 2024-09-11T01:53:39.075 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:53:39.073+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=64} crc {osdmap=728738097}) 2024-09-11T01:53:39.164 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:53:39.165 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:53:39.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.270+0000 7efd1ceb6640 10 monclient: get_monmap_and_config 2024-09-11T01:53:39.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.270+0000 7efd1ceb6640 10 monclient: build_initial_monmap 2024-09-11T01:53:39.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.270+0000 7efd1ceb6640 10 monclient: monmap: 2024-09-11T01:53:39.273 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:39.273 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:39.273 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:53:39.271238+0000 2024-09-11T01:53:39.273 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:53:39.271238+0000 2024-09-11T01:53:39.273 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:39.273 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:39.273 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:53:39.273 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:53:39.273 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:53:39.273 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:53:39.273 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:53:39.274 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:39.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.271+0000 7efd1ceb6640 1 Processor -- start 2024-09-11T01:53:39.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.271+0000 7efd1ceb6640 1 -- start start 2024-09-11T01:53:39.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.271+0000 7efd1ceb6640 10 monclient: init 2024-09-11T01:53:39.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.271+0000 7efd1ceb6640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:39.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.271+0000 7efd1ceb6640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-11T01:53:39.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.271+0000 7efd1ceb6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efd180a2e90 addr v1:172.21.15.71:6789/0 2024-09-11T01:53:39.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.271+0000 7efd1ceb6640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7efd180a3270 0x7efd180a3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:39.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.271+0000 7efd1ceb6640 10 monclient(hunting): picked mon.noname-a con 0x7efd180a3270 addr v2:172.21.15.12:3300/0 2024-09-11T01:53:39.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.271+0000 7efd1ceb6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7efd1813d490 0x7efd1813f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:39.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.271+0000 7efd1ceb6640 10 monclient(hunting): picked mon.noname-c con 0x7efd1813d490 addr v2:172.21.15.71:3300/0 2024-09-11T01:53:39.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.271+0000 7efd1ceb6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd180a46c0 con 0x7efd180a2e90 2024-09-11T01:53:39.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.271+0000 7efd1ceb6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:39.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.271+0000 7efd1ceb6640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7efd180a4900 con 0x7efd180a3270 2024-09-11T01:53:39.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.271+0000 7efd1ceb6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:39.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.271+0000 7efd1ceb6640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7efd1806ee50 con 0x7efd1813d490 2024-09-11T01:53:39.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.271+0000 7efd1ceb6640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:39.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.271+0000 7efd1ceb6640 10 monclient(hunting): authenticate will time out at 2225.713808s 2024-09-11T01:53:39.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.271+0000 7efd15d9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7efd180a3270 0x7efd180a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:39.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.271+0000 7efd15d9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7efd180a3270 0x7efd180a3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39532/0 (socket says 172.21.15.12:39532) 2024-09-11T01:53:39.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.271+0000 7efd15d9b640 1 -- 172.21.15.12:0/3757016111 learned_addr learned my addr 172.21.15.12:0/3757016111 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:53:39.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.271+0000 7efd15d9b640 10 monclient(hunting): get_auth_request con 0x7efd180a3270 auth_method 0 2024-09-11T01:53:39.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.271+0000 7efd15d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:39.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.271+0000 7efd15d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:39.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.271+0000 7efd15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:39.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.272+0000 7efd15d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:39.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.272+0000 7efd15d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:39.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.272+0000 7efd15d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:39.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.272+0000 7efd16d9d640 1 --2- 172.21.15.12:0/3757016111 >> v2:172.21.15.71:3300/0 conn(0x7efd1813d490 0x7efd1813f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:39.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.272+0000 7efd16d9d640 10 monclient(hunting): get_auth_request con 0x7efd1813d490 auth_method 0 2024-09-11T01:53:39.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.272+0000 7efd16d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:39.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.272+0000 7efd16d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:39.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.272+0000 7efd16d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:39.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.272+0000 7efd15d9b640 10 monclient(hunting): handle_auth_done global_id 264243 payload 274 2024-09-11T01:53:39.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.272+0000 7efd15d9b640 1 -- 172.21.15.12:0/3757016111 >> v2:172.21.15.71:3300/0 conn(0x7efd1813d490 msgr2=0x7efd1813f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:53:39.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.272+0000 7efd15d9b640 1 --2- 172.21.15.12:0/3757016111 >> v2:172.21.15.71:3300/0 conn(0x7efd1813d490 0x7efd1813f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:39.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.272+0000 7efd15d9b640 1 -- 172.21.15.12:0/3757016111 >> v1:172.21.15.71:6789/0 conn(0x7efd180a2e90 legacy=0x7efd1806dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:53:39.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.272+0000 7efd15d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:53:39.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.272+0000 7efd15d9b640 1 monclient: found mon.noname-a 2024-09-11T01:53:39.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.272+0000 7efd15d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:39.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.272+0000 7efd15d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:53:39.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.272+0000 7efd15d9b640 1 -- 172.21.15.12:0/3757016111 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd1806eb70 con 0x7efd180a3270 2024-09-11T01:53:39.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd15d9b640 1 -- 172.21.15.12:0/3757016111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd180a3270 msgr2=0x7efd180a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:39.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd15d9b640 1 --2- 172.21.15.12:0/3757016111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd180a3270 0x7efd180a3680 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efd180a4a80 tx=0x7efd0002a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ae02bf81b4e538f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:39.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1559a640 1 -- 172.21.15.12:0/3757016111 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd00037070 con 0x7efd180a3270 2024-09-11T01:53:39.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:39.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:53:39.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1559a640 10 monclient: dump: 2024-09-11T01:53:39.285 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:39.285 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:39.285 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.285 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.285 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:39.286 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1559a640 10 monclient: _finish_auth 0 2024-09-11T01:53:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1559a640 10 monclient: _check_auth_tickets 2024-09-11T01:53:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1ceb6640 5 monclient: authenticate success, global_id 264243 2024-09-11T01:53:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1ceb6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:53:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1559a640 1 -- 172.21.15.12:0/3757016111 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd00032440 con 0x7efd180a3270 2024-09-11T01:53:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1559a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1ceb6640 10 monclient: get_monmap_and_config success 2024-09-11T01:53:39.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1ceb6640 4 set_mon_vals no callback set 2024-09-11T01:53:39.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1559a640 1 -- 172.21.15.12:0/3757016111 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd000327f0 con 0x7efd180a3270 2024-09-11T01:53:39.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:39.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:53:39.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1559a640 10 monclient: dump: 2024-09-11T01:53:39.287 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:39.287 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:39.287 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.287 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.287 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:39.287 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:39.287 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:39.287 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:39.288 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:39.288 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:39.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1ceb6640 10 monclient: shutdown 2024-09-11T01:53:39.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1ceb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:39.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1ceb6640 1 -- 172.21.15.12:0/3757016111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd180a3270 msgr2=0x7efd180a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:39.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1ceb6640 1 --2- 172.21.15.12:0/3757016111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd180a3270 0x7efd180a3680 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efd180a4a80 tx=0x7efd0002a080 comp rx=0 tx=0).stop 2024-09-11T01:53:39.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1ceb6640 1 -- 172.21.15.12:0/3757016111 shutdown_connections 2024-09-11T01:53:39.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1ceb6640 1 --2- 172.21.15.12:0/3757016111 >> v2:172.21.15.71:3300/0 conn(0x7efd1813d490 0x7efd1813f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:39.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1ceb6640 1 --2- 172.21.15.12:0/3757016111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd180a3270 0x7efd180a3680 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:39.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1ceb6640 1 -- 172.21.15.12:0/3757016111 >> 172.21.15.12:0/3757016111 conn(0x7efd180a8490 msgr2=0x7efd180a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:39.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1ceb6640 1 -- 172.21.15.12:0/3757016111 shutdown_connections 2024-09-11T01:53:39.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.275+0000 7efd1ceb6640 1 -- 172.21.15.12:0/3757016111 wait complete. 2024-09-11T01:53:39.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.276+0000 7efd1ceb6640 10 monclient: build_initial_monmap 2024-09-11T01:53:39.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.276+0000 7efd1ceb6640 10 monclient: monmap: 2024-09-11T01:53:39.289 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:39.289 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:53:39.289 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:53:39.289 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:53:39.289 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:39.289 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:39.289 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:53:39.289 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:53:39.290 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:53:39.290 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:39.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.276+0000 7efd1ceb6640 1 Processor -- start 2024-09-11T01:53:39.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1ceb6640 1 -- start start 2024-09-11T01:53:39.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1ceb6640 10 monclient: init 2024-09-11T01:53:39.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1ceb6640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:39.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1ceb6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:53:39.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1ceb6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efd180a2e90 0x7efd1813ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:39.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1ceb6640 10 monclient(hunting): picked mon.noname-c con 0x7efd180a2e90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:53:39.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1ceb6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efd180a3270 0x7efd18138320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:39.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1ceb6640 10 monclient(hunting): picked mon.noname-b con 0x7efd180a3270 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:53:39.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1ceb6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd1813d490 0x7efd181ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:39.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1ceb6640 10 monclient(hunting): picked mon.noname-a con 0x7efd1813d490 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:53:39.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1ceb6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:39.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1ceb6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7efd1806fd20 con 0x7efd1813d490 2024-09-11T01:53:39.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1ceb6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:39.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1ceb6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7efd1806fba0 con 0x7efd180a3270 2024-09-11T01:53:39.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1ceb6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:39.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1ceb6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7efd1813fdc0 con 0x7efd180a2e90 2024-09-11T01:53:39.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1ceb6640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:39.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1ceb6640 10 monclient(hunting): authenticate will time out at 2225.719485s 2024-09-11T01:53:39.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efd180a2e90 0x7efd1813ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:39.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd16d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd1813d490 0x7efd181ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efd180a2e90 0x7efd1813ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42946/0 (socket says 172.21.15.12:42946) 2024-09-11T01:53:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1659c640 1 -- 172.21.15.12:0/1669174093 learned_addr learned my addr 172.21.15.12:0/1669174093 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:53:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1659c640 10 monclient(hunting): get_auth_request con 0x7efd180a2e90 auth_method 0 2024-09-11T01:53:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd16d9d640 10 monclient(hunting): get_auth_request con 0x7efd1813d490 auth_method 0 2024-09-11T01:53:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd16d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd16d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd16d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:39.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd15d9b640 1 --2- 172.21.15.12:0/1669174093 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efd180a3270 0x7efd18138320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:39.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:39.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd15d9b640 10 monclient(hunting): get_auth_request con 0x7efd180a3270 auth_method 0 2024-09-11T01:53:39.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd15d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:39.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd15d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:39.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:39.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd16d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:39.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd16d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:39.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd16d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:39.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd15d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:39.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd15d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:39.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd15d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:39.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1659c640 10 monclient(hunting): handle_auth_done global_id 244433 payload 824 2024-09-11T01:53:39.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1659c640 1 -- 172.21.15.12:0/1669174093 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efd180a3270 msgr2=0x7efd18138320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:39.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1659c640 1 --2- 172.21.15.12:0/1669174093 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efd180a3270 0x7efd18138320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:39.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1659c640 1 -- 172.21.15.12:0/1669174093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd1813d490 msgr2=0x7efd181ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:39.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1659c640 1 --2- 172.21.15.12:0/1669174093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd1813d490 0x7efd181ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:39.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1659c640 10 monclient: _finish_hunting 0 2024-09-11T01:53:39.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1659c640 1 monclient: found mon.noname-c 2024-09-11T01:53:39.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:39.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:53:39.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd1659c640 1 -- 172.21.15.12:0/1669174093 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd1809f680 con 0x7efd180a2e90 2024-09-11T01:53:39.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.277+0000 7efd16d9d640 1 --2- 172.21.15.12:0/1669174093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd1813d490 0x7efd181ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:53:39.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.280+0000 7efd1659c640 1 -- 172.21.15.12:0/1669174093 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efd180a2e90 msgr2=0x7efd1813ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:39.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.280+0000 7efd1659c640 1 --2- 172.21.15.12:0/1669174093 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efd180a2e90 0x7efd1813ba40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efd0800c430 tx=0x7efd0800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:39.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.280+0000 7efd0effd640 1 -- 172.21.15.12:0/1669174093 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd080119a0 con 0x7efd180a2e90 2024-09-11T01:53:39.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.280+0000 7efd0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:39.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.280+0000 7efd0effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:53:39.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.280+0000 7efd0effd640 10 monclient: dump: 2024-09-11T01:53:39.295 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:39.295 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:39.295 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.295 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.295 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:39.295 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:39.295 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:39.295 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:39.296 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:39.296 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:39.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.280+0000 7efd0effd640 10 monclient: _finish_auth 0 2024-09-11T01:53:39.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.280+0000 7efd0effd640 10 monclient: _check_auth_tickets 2024-09-11T01:53:39.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.280+0000 7efd0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:39.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.280+0000 7efd0effd640 1 -- 172.21.15.12:0/1669174093 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd08017070 con 0x7efd180a2e90 2024-09-11T01:53:39.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.280+0000 7efd1ceb6640 5 monclient: authenticate success, global_id 244433 2024-09-11T01:53:39.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.280+0000 7efd1ceb6640 10 monclient: _renew_subs 2024-09-11T01:53:39.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.280+0000 7efd1ceb6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:53:39.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.280+0000 7efd1ceb6640 1 -- 172.21.15.12:0/1669174093 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd18138fc0 con 0x7efd180a2e90 2024-09-11T01:53:39.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.280+0000 7efd0effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:39.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.280+0000 7efd0effd640 1 -- 172.21.15.12:0/1669174093 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd08011d80 con 0x7efd180a2e90 2024-09-11T01:53:39.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.280+0000 7efd0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:39.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.280+0000 7efd0effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:53:39.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.280+0000 7efd0effd640 10 monclient: dump: 2024-09-11T01:53:39.297 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:39.297 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:39.297 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.297 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.297 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:39.297 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:39.297 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:39.298 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:39.298 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:39.298 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:39.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.280+0000 7efd14d99640 4 set_mon_vals no callback set 2024-09-11T01:53:39.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.280+0000 7efd1ceb6640 10 monclient: _renew_subs 2024-09-11T01:53:39.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.280+0000 7efd1ceb6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:53:39.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.280+0000 7efd1ceb6640 1 -- 172.21.15.12:0/1669174093 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efd1813a810 con 0x7efd180a2e90 2024-09-11T01:53:39.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.281+0000 7efd0effd640 1 -- 172.21.15.12:0/1669174093 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7efd0801c460 con 0x7efd180a2e90 2024-09-11T01:53:39.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.281+0000 7efd1ceb6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:39.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.281+0000 7efd1ceb6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:39.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.281+0000 7efd1ceb6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:53:39.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.281+0000 7efd1ceb6640 1 -- 172.21.15.12:0/1669174093 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd180a3680 con 0x7efd180a2e90 2024-09-11T01:53:39.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.281+0000 7efd0effd640 1 --2- 172.21.15.12:0/1669174093 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efcf803e260 0x7efcf8040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:39.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.281+0000 7efd0effd640 1 -- 172.21.15.12:0/1669174093 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(833..833 src has 676..833) ==== 4843+0+0 (secure 0 0 0) 0x7efd08052c10 con 0x7efd180a2e90 2024-09-11T01:53:39.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.281+0000 7efd15d9b640 1 --2- 172.21.15.12:0/1669174093 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efcf803e260 0x7efcf8040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:39.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.281+0000 7efd15d9b640 10 monclient: get_auth_request con 0x7efcf803e260 auth_method 0 2024-09-11T01:53:39.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.284+0000 7efd15d9b640 1 --2- 172.21.15.12:0/1669174093 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efcf803e260 0x7efcf8040720 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7efd00034000 tx=0x7efd0002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:39.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.286+0000 7efd0effd640 1 -- 172.21.15.12:0/1669174093 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7efd0801ac60 con 0x7efd180a2e90 2024-09-11T01:53:39.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.286+0000 7efd0effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:39.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.286+0000 7efd0effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:53:39.327 INFO:tasks.rados.rados.0.smithi071.stdout:3641: done (5 left) 2024-09-11T01:53:39.328 INFO:tasks.rados.rados.0.smithi071.stdout:3642: write (excl) oid 36 current snap is 415 2024-09-11T01:53:39.328 INFO:tasks.rados.rados.0.smithi071.stdout:3642: seq_num 1256 ranges {709592=510989,1731266=653776,3041279=68313} 2024-09-11T01:53:39.337 INFO:tasks.rados.rados.0.smithi071.stdout:3642: writing smithi07134777-36 from 709592 to 1220581 tid 1 2024-09-11T01:53:39.342 INFO:tasks.rados.rados.0.smithi071.stdout:3642: writing smithi07134777-36 from 1731266 to 2385042 tid 2 2024-09-11T01:53:39.346 INFO:tasks.rados.rados.0.smithi071.stdout:3642: writing smithi07134777-36 from 3041279 to 3109592 tid 3 2024-09-11T01:53:39.346 INFO:tasks.rados.rados.0.smithi071.stdout:3643: read oid 17 snap -1 2024-09-11T01:53:39.346 INFO:tasks.rados.rados.0.smithi071.stdout:3643: expect (ObjNum 1242 snap 411 seq_num 1242) 2024-09-11T01:53:39.346 INFO:tasks.rados.rados.0.smithi071.stdout:3644: snap_create 2024-09-11T01:53:39.346 INFO:tasks.rados.rados.0.smithi071.stdout:3635: expect (ObjNum 1210 snap 395 seq_num 1210) 2024-09-11T01:53:39.346 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 711135 2024-09-11T01:53:39.366 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:53:39.423 INFO:tasks.rados.rados.0.smithi071.stdout:3636: finishing write tid 1 to smithi07134777-40 2024-09-11T01:53:39.423 INFO:tasks.rados.rados.0.smithi071.stdout:3636: finishing write tid 2 to smithi07134777-40 2024-09-11T01:53:39.423 INFO:tasks.rados.rados.0.smithi071.stdout:3636: finishing write tid 3 to smithi07134777-40 2024-09-11T01:53:39.423 INFO:tasks.rados.rados.0.smithi071.stdout:3636: finishing write tid 4 to smithi07134777-40 2024-09-11T01:53:39.423 INFO:tasks.rados.rados.0.smithi071.stdout:3636: oid 40 updating version 0 to 2578 2024-09-11T01:53:39.423 INFO:tasks.rados.rados.0.smithi071.stdout:3636: oid 40 version 2578 is already newer than 2577 2024-09-11T01:53:39.423 INFO:tasks.rados.rados.0.smithi071.stdout:3636: oid 40 version 2578 is already newer than 2576 2024-09-11T01:53:39.424 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 2578 (ObjNum 1254 snap 415 seq_num 1254) dirty exists 2024-09-11T01:53:39.424 INFO:tasks.rados.rados.0.smithi071.stdout:3636: left oid 40 (ObjNum 1254 snap 415 seq_num 1254) 2024-09-11T01:53:39.424 INFO:tasks.rados.rados.0.smithi071.stdout:3637: expect (ObjNum 832 snap 254 seq_num 832) 2024-09-11T01:53:39.450 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2354195 2024-09-11T01:53:39.525 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3055274 2024-09-11T01:53:39.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.552+0000 7efd1ceb6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:53:39.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.552+0000 7efd1ceb6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:53:39.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.552+0000 7efd1ceb6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:53:39.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.552+0000 7efd1ceb6640 1 -- 172.21.15.12:0/1669174093 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7efd1813aee0 con 0x7efd180a2e90 2024-09-11T01:53:39.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.553+0000 7efd0effd640 1 -- 172.21.15.12:0/1669174093 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7efd080515d0 con 0x7efd180a2e90 2024-09-11T01:53:39.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.553+0000 7efd0effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:53:39.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.553+0000 7efd0effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:53:39.555 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":226,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:53:39.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.555+0000 7efd1ceb6640 1 -- 172.21.15.12:0/1669174093 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efcf803e260 msgr2=0x7efcf8040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:39.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.555+0000 7efd1ceb6640 1 --2- 172.21.15.12:0/1669174093 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efcf803e260 0x7efcf8040720 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7efd00034000 tx=0x7efd0002e000 comp rx=0 tx=0).stop 2024-09-11T01:53:39.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.555+0000 7efd1ceb6640 10 monclient: shutdown 2024-09-11T01:53:39.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.555+0000 7efd1ceb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:39.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.555+0000 7efd1ceb6640 1 -- 172.21.15.12:0/1669174093 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efd180a2e90 msgr2=0x7efd1813ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:39.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.555+0000 7efd1ceb6640 1 --2- 172.21.15.12:0/1669174093 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efd180a2e90 0x7efd1813ba40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efd0800c430 tx=0x7efd0800c8f0 comp rx=0 tx=0).stop 2024-09-11T01:53:39.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.555+0000 7efd1ceb6640 1 -- 172.21.15.12:0/1669174093 shutdown_connections 2024-09-11T01:53:39.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.555+0000 7efd1ceb6640 1 --2- 172.21.15.12:0/1669174093 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7efcf803e260 0x7efcf8040720 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:39.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.555+0000 7efd1ceb6640 1 --2- 172.21.15.12:0/1669174093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efd1813d490 0x7efd181ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:39.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.555+0000 7efd1ceb6640 1 --2- 172.21.15.12:0/1669174093 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efd180a3270 0x7efd18138320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:39.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.555+0000 7efd1ceb6640 1 --2- 172.21.15.12:0/1669174093 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efd180a2e90 0x7efd1813ba40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:39.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.555+0000 7efd1ceb6640 1 -- 172.21.15.12:0/1669174093 >> 172.21.15.12:0/1669174093 conn(0x7efd180a8490 msgr2=0x7efd18072090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:39.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.555+0000 7efd1ceb6640 1 -- 172.21.15.12:0/1669174093 shutdown_connections 2024-09-11T01:53:39.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.555+0000 7efd1ceb6640 1 -- 172.21.15.12:0/1669174093 wait complete. 2024-09-11T01:53:39.567 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:53:39.567 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:53:39.567 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:53:39.567 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:53:39.625 INFO:tasks.rados.rados.0.smithi071.stdout:3639: expect (ObjNum 1154 snap 366 seq_num 1154) 2024-09-11T01:53:39.625 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:53:39.669 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:53:39.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.668+0000 7f9d2b033640 10 monclient: get_monmap_and_config 2024-09-11T01:53:39.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.668+0000 7f9d2b033640 10 monclient: build_initial_monmap 2024-09-11T01:53:39.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.668+0000 7f9d2b033640 10 monclient: monmap: 2024-09-11T01:53:39.671 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:39.671 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:39.671 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:53:39.668914+0000 2024-09-11T01:53:39.671 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:53:39.668914+0000 2024-09-11T01:53:39.671 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:39.671 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:39.671 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:53:39.671 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:53:39.671 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:53:39.671 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:53:39.671 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:53:39.672 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:39.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.668+0000 7f9d2b033640 1 Processor -- start 2024-09-11T01:53:39.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.669+0000 7f9d2b033640 1 -- start start 2024-09-11T01:53:39.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.669+0000 7f9d2b033640 10 monclient: init 2024-09-11T01:53:39.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.669+0000 7f9d2b033640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:39.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.669+0000 7f9d2b033640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-11T01:53:39.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.669+0000 7f9d2b033640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d24137e50 addr v1:172.21.15.71:6789/0 2024-09-11T01:53:39.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.669+0000 7f9d2b033640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9d24138230 0x7f9d24138640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:39.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.669+0000 7f9d2b033640 10 monclient(hunting): picked mon.noname-c con 0x7f9d24138230 addr v2:172.21.15.71:3300/0 2024-09-11T01:53:39.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.669+0000 7f9d2b033640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9d2413d380 addr v1:172.21.15.12:6789/0 2024-09-11T01:53:39.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.669+0000 7f9d2b033640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d240a62d0 con 0x7f9d2413d380 2024-09-11T01:53:39.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.669+0000 7f9d2b033640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d2409bb40 con 0x7f9d24137e50 2024-09-11T01:53:39.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.669+0000 7f9d2b033640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:39.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.669+0000 7f9d2b033640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f9d2409c4a0 con 0x7f9d24138230 2024-09-11T01:53:39.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.669+0000 7f9d2b033640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:39.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.669+0000 7f9d2b033640 10 monclient(hunting): authenticate will time out at 2226.111600s 2024-09-11T01:53:39.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.669+0000 7f9d295d0640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9d2413d380 0x7f9d2413f770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59012/0 (socket says 172.21.15.12:59012) 2024-09-11T01:53:39.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.669+0000 7f9d295d0640 1 -- 172.21.15.12:0/346191998 learned_addr learned my addr 172.21.15.12:0/346191998 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:53:39.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.669+0000 7f9d23fff640 1 --2- 172.21.15.12:0/346191998 >> v2:172.21.15.71:3300/0 conn(0x7f9d24138230 0x7f9d24138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:39.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.669+0000 7f9d23fff640 10 monclient(hunting): get_auth_request con 0x7f9d24138230 auth_method 0 2024-09-11T01:53:39.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.669+0000 7f9d23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:39.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.669+0000 7f9d23fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:39.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.669+0000 7f9d23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:39.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.669+0000 7f9d23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:39.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.669+0000 7f9d23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:39.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.669+0000 7f9d23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:39.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.670+0000 7f9d23fff640 10 monclient(hunting): handle_auth_done global_id 274114 payload 274 2024-09-11T01:53:39.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.670+0000 7f9d23fff640 1 -- 172.21.15.12:0/346191998 >> v1:172.21.15.71:6789/0 conn(0x7f9d24137e50 legacy=0x7f9d2409d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:53:39.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.670+0000 7f9d23fff640 1 -- 172.21.15.12:0/346191998 >> v1:172.21.15.12:6789/0 conn(0x7f9d2413d380 legacy=0x7f9d2413f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:53:39.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.670+0000 7f9d23fff640 10 monclient: _finish_hunting 0 2024-09-11T01:53:39.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.670+0000 7f9d23fff640 1 monclient: found mon.noname-c 2024-09-11T01:53:39.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.670+0000 7f9d23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:39.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.670+0000 7f9d23fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:53:39.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.670+0000 7f9d23fff640 1 -- 172.21.15.12:0/346191998 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d2409c780 con 0x7f9d24138230 2024-09-11T01:53:39.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.672+0000 7f9d23fff640 1 -- 172.21.15.12:0/346191998 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d24138230 msgr2=0x7f9d24138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:39.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.672+0000 7f9d23fff640 1 --2- 172.21.15.12:0/346191998 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d24138230 0x7f9d24138640 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9d240ab580 tx=0x7f9d18029fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c651e7a23d3836be server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:39.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.673+0000 7f9d237fe640 1 -- 172.21.15.12:0/346191998 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d18037070 con 0x7f9d24138230 2024-09-11T01:53:39.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.673+0000 7f9d237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:39.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.673+0000 7f9d237fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:53:39.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.673+0000 7f9d237fe640 10 monclient: dump: 2024-09-11T01:53:39.688 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:39.688 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:39.688 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.688 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.689 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:39.689 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:39.689 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:39.689 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:39.689 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:39.689 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:39.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.673+0000 7f9d237fe640 10 monclient: _finish_auth 0 2024-09-11T01:53:39.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.673+0000 7f9d237fe640 10 monclient: _check_auth_tickets 2024-09-11T01:53:39.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.673+0000 7f9d237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:39.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.673+0000 7f9d237fe640 1 -- 172.21.15.12:0/346191998 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d18032440 con 0x7f9d24138230 2024-09-11T01:53:39.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.673+0000 7f9d237fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:39.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.673+0000 7f9d237fe640 1 -- 172.21.15.12:0/346191998 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d18032810 con 0x7f9d24138230 2024-09-11T01:53:39.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.673+0000 7f9d2b033640 5 monclient: authenticate success, global_id 274114 2024-09-11T01:53:39.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.673+0000 7f9d2b033640 10 monclient: get_monmap_and_config success 2024-09-11T01:53:39.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.673+0000 7f9d237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:39.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.673+0000 7f9d2b033640 4 set_mon_vals no callback set 2024-09-11T01:53:39.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.673+0000 7f9d237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:53:39.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.673+0000 7f9d237fe640 10 monclient: dump: 2024-09-11T01:53:39.691 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:39.691 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:39.691 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.691 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.691 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:39.691 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:39.692 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:39.692 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:39.692 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:39.692 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:39.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.673+0000 7f9d2b033640 10 monclient: shutdown 2024-09-11T01:53:39.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.673+0000 7f9d2b033640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:39.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.673+0000 7f9d2b033640 1 -- 172.21.15.12:0/346191998 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d24138230 msgr2=0x7f9d24138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:39.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.673+0000 7f9d2b033640 1 --2- 172.21.15.12:0/346191998 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d24138230 0x7f9d24138640 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9d240ab580 tx=0x7f9d18029fb0 comp rx=0 tx=0).stop 2024-09-11T01:53:39.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.673+0000 7f9d2b033640 1 -- 172.21.15.12:0/346191998 shutdown_connections 2024-09-11T01:53:39.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.673+0000 7f9d2b033640 1 --2- 172.21.15.12:0/346191998 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d24138230 0x7f9d24138640 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:39.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.673+0000 7f9d2b033640 1 -- 172.21.15.12:0/346191998 >> 172.21.15.12:0/346191998 conn(0x7f9d240a7020 msgr2=0x7f9d240a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:39.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.673+0000 7f9d2b033640 1 -- 172.21.15.12:0/346191998 shutdown_connections 2024-09-11T01:53:39.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.673+0000 7f9d2b033640 1 -- 172.21.15.12:0/346191998 wait complete. 2024-09-11T01:53:39.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.674+0000 7f9d2b033640 10 monclient: build_initial_monmap 2024-09-11T01:53:39.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.674+0000 7f9d2b033640 10 monclient: monmap: 2024-09-11T01:53:39.693 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:39.694 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:53:39.694 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:53:39.694 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:53:39.694 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:39.694 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:39.694 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:53:39.694 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:53:39.694 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:53:39.694 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:39.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.674+0000 7f9d2b033640 1 Processor -- start 2024-09-11T01:53:39.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.674+0000 7f9d2b033640 1 -- start start 2024-09-11T01:53:39.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.674+0000 7f9d2b033640 10 monclient: init 2024-09-11T01:53:39.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d2b033640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:39.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d2b033640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:53:39.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d2b033640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9d24137e50 0x7f9d2413b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:39.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d2b033640 10 monclient(hunting): picked mon.noname-c con 0x7f9d24137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:53:39.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d2b033640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d24138230 0x7f9d24139820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:39.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d2b033640 10 monclient(hunting): picked mon.noname-a con 0x7f9d24138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:53:39.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d2b033640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d2413d380 0x7f9d24136f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:39.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d2b033640 10 monclient(hunting): picked mon.noname-b con 0x7f9d2413d380 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:53:39.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d2b033640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:39.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d2b033640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9d2406f890 con 0x7f9d24138230 2024-09-11T01:53:39.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d2b033640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:39.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d2b033640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f9d2406f710 con 0x7f9d2413d380 2024-09-11T01:53:39.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d2b033640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:39.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d2b033640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f9d24140e90 con 0x7f9d24137e50 2024-09-11T01:53:39.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d2b033640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:39.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d2b033640 10 monclient(hunting): authenticate will time out at 2226.117507s 2024-09-11T01:53:39.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d23fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d24138230 0x7f9d24139820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:39.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d28dcf640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9d24137e50 0x7f9d2413b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:39.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d23fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d24138230 0x7f9d24139820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39556/0 (socket says 172.21.15.12:39556) 2024-09-11T01:53:39.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d28dcf640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9d24137e50 0x7f9d2413b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42958/0 (socket says 172.21.15.12:42958) 2024-09-11T01:53:39.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d23fff640 1 -- 172.21.15.12:0/1457973160 learned_addr learned my addr 172.21.15.12:0/1457973160 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:53:39.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d23fff640 10 monclient(hunting): get_auth_request con 0x7f9d24138230 auth_method 0 2024-09-11T01:53:39.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:39.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d23fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:39.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:39.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d28dcf640 10 monclient(hunting): get_auth_request con 0x7f9d24137e50 auth_method 0 2024-09-11T01:53:39.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d28dcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:39.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d28dcf640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:39.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d28dcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:39.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d295d0640 1 --2- 172.21.15.12:0/1457973160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d2413d380 0x7f9d24136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:39.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d295d0640 10 monclient(hunting): get_auth_request con 0x7f9d2413d380 auth_method 0 2024-09-11T01:53:39.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d295d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:39.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d295d0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:39.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d295d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:39.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:39.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:39.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:39.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d28dcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:39.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d28dcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:39.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d28dcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:39.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d295d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:39.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d295d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:39.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d295d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:39.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d295d0640 10 monclient(hunting): handle_auth_done global_id 274117 payload 824 2024-09-11T01:53:39.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d295d0640 1 -- 172.21.15.12:0/1457973160 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9d24137e50 msgr2=0x7f9d2413b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:39.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d295d0640 1 --2- 172.21.15.12:0/1457973160 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9d24137e50 0x7f9d2413b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:39.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d295d0640 1 -- 172.21.15.12:0/1457973160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d24138230 msgr2=0x7f9d24139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:39.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d295d0640 1 --2- 172.21.15.12:0/1457973160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d24138230 0x7f9d24139820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:39.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d295d0640 10 monclient: _finish_hunting 0 2024-09-11T01:53:39.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d295d0640 1 monclient: found mon.noname-b 2024-09-11T01:53:39.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d295d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:39.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d295d0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-09-11T01:53:39.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d295d0640 1 -- 172.21.15.12:0/1457973160 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d2413c260 con 0x7f9d2413d380 2024-09-11T01:53:39.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d28dcf640 1 --2- 172.21.15.12:0/1457973160 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9d24137e50 0x7f9d2413b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:53:39.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.675+0000 7f9d23fff640 1 --2- 172.21.15.12:0/1457973160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d24138230 0x7f9d24139820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:53:39.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.678+0000 7f9d295d0640 1 -- 172.21.15.12:0/1457973160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d2413d380 msgr2=0x7f9d24136f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:39.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.678+0000 7f9d295d0640 1 --2- 172.21.15.12:0/1457973160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d2413d380 0x7f9d24136f40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c00cb50 tx=0x7f9d1c00cf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:39.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.678+0000 7f9d217fa640 1 -- 172.21.15.12:0/1457973160 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d1c00ae40 con 0x7f9d2413d380 2024-09-11T01:53:39.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.678+0000 7f9d217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:39.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.678+0000 7f9d217fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:53:39.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.678+0000 7f9d217fa640 10 monclient: dump: 2024-09-11T01:53:39.705 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:39.705 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:39.705 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.705 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.706 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:39.706 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:39.706 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:39.706 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:39.706 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:39.707 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:39.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.678+0000 7f9d217fa640 10 monclient: _finish_auth 0 2024-09-11T01:53:39.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.678+0000 7f9d217fa640 10 monclient: _check_auth_tickets 2024-09-11T01:53:39.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.678+0000 7f9d217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:39.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.678+0000 7f9d217fa640 1 -- 172.21.15.12:0/1457973160 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d1c012400 con 0x7f9d2413d380 2024-09-11T01:53:39.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.678+0000 7f9d217fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:39.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.678+0000 7f9d2b033640 5 monclient: authenticate success, global_id 274117 2024-09-11T01:53:39.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.678+0000 7f9d2b033640 10 monclient: _renew_subs 2024-09-11T01:53:39.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.678+0000 7f9d217fa640 1 -- 172.21.15.12:0/1457973160 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d1c0127f0 con 0x7f9d2413d380 2024-09-11T01:53:39.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.678+0000 7f9d2b033640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:53:39.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.678+0000 7f9d2b033640 1 -- 172.21.15.12:0/1457973160 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d240a1710 con 0x7f9d2413d380 2024-09-11T01:53:39.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.678+0000 7f9d217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:39.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.678+0000 7f9d217fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:53:39.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.678+0000 7f9d217fa640 10 monclient: dump: 2024-09-11T01:53:39.709 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:39.709 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:39.709 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.710 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.710 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:39.710 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:39.710 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:39.710 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:39.710 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:39.711 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:39.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.678+0000 7f9d22ffd640 4 set_mon_vals no callback set 2024-09-11T01:53:39.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.678+0000 7f9d2b033640 10 monclient: _renew_subs 2024-09-11T01:53:39.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.678+0000 7f9d2b033640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:53:39.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.678+0000 7f9d2b033640 1 -- 172.21.15.12:0/1457973160 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d2413d0d0 con 0x7f9d2413d380 2024-09-11T01:53:39.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.679+0000 7f9d2b033640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:39.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.679+0000 7f9d2b033640 1 --2- 172.21.15.12:0/1457973160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d241cd0d0 0x7f9d241cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:39.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.679+0000 7f9d2b033640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:39.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.679+0000 7f9d2b033640 1 -- 172.21.15.12:0/1457973160 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9d240a0ab0 con 0x7f9d241cd0d0 2024-09-11T01:53:39.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.679+0000 7f9d295d0640 1 --2- 172.21.15.12:0/1457973160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d2413d380 0x7f9d24136f40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c00cb50 tx=0x7f9d1c00cf50 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7f9d1c053330 osd_map(834..834 src has 676..834) 2024-09-11T01:53:39.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.679+0000 7f9d28dcf640 1 --2- 172.21.15.12:0/1457973160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d241cd0d0 0x7f9d241cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:39.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.679+0000 7f9d217fa640 1 -- 172.21.15.12:0/1457973160 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f9d1c012990 con 0x7f9d2413d380 2024-09-11T01:53:39.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.679+0000 7f9d28dcf640 10 monclient: get_auth_request con 0x7f9d241cd0d0 auth_method 0 2024-09-11T01:53:39.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.679+0000 7f9d28dcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:39.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.679+0000 7f9d28dcf640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:39.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.679+0000 7f9d28dcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:39.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.679+0000 7f9d28dcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:39.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.679+0000 7f9d28dcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:39.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.679+0000 7f9d28dcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:39.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.679+0000 7f9d28dcf640 10 monclient(hunting): handle_auth_done global_id 264252 payload 824 2024-09-11T01:53:39.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.679+0000 7f9d28dcf640 1 -- 172.21.15.12:0/1457973160 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9d24138640 con 0x7f9d241cd0d0 2024-09-11T01:53:39.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.679+0000 7f9d217fa640 1 --2- 172.21.15.12:0/1457973160 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9d1003e260 0x7f9d10040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:39.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.679+0000 7f9d23fff640 1 --2- 172.21.15.12:0/1457973160 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9d1003e260 0x7f9d10040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:39.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.679+0000 7f9d23fff640 10 monclient: get_auth_request con 0x7f9d1003e260 auth_method 0 2024-09-11T01:53:39.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.681+0000 7f9d28dcf640 1 -- 172.21.15.12:0/1457973160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d241cd0d0 msgr2=0x7f9d241cd4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:39.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.681+0000 7f9d28dcf640 1 --2- 172.21.15.12:0/1457973160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d241cd0d0 0x7f9d241cd4b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9d1400fae0 tx=0x7f9d140044e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:39.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.682+0000 7f9d217fa640 1 -- 172.21.15.12:0/1457973160 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d14018020 con 0x7f9d241cd0d0 2024-09-11T01:53:39.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.682+0000 7f9d217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:39.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.682+0000 7f9d217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:53:39.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.682+0000 7f9d217fa640 10 monclient: dump: 2024-09-11T01:53:39.715 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:39.715 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:39.715 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.715 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.715 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:39.715 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:39.715 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:39.715 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:39.716 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:39.716 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:39.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.682+0000 7f9d217fa640 1 -- 172.21.15.12:0/1457973160 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9d1400dd60 con 0x7f9d241cd0d0 2024-09-11T01:53:39.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.682+0000 7f9d217fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:39.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.682+0000 7f9d217fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:53:39.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.682+0000 7f9d217fa640 1 -- 172.21.15.12:0/1457973160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d241cd0d0 msgr2=0x7f9d241cd4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:39.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.682+0000 7f9d217fa640 1 --2- 172.21.15.12:0/1457973160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d241cd0d0 0x7f9d241cd4b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9d1400fae0 tx=0x7f9d140044e0 comp rx=0 tx=0).stop 2024-09-11T01:53:39.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.682+0000 7f9d217fa640 1 -- 172.21.15.12:0/1457973160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d241cd0d0 msgr2=0x7f9d241cd4b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:53:39.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.682+0000 7f9d217fa640 1 --2- 172.21.15.12:0/1457973160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d241cd0d0 0x7f9d241cd4b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:39.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.682+0000 7f9d23fff640 1 --2- 172.21.15.12:0/1457973160 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9d1003e260 0x7f9d10040720 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f9d2413c450 tx=0x7f9d1802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:39.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.688+0000 7f9d2b033640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:53:39.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.688+0000 7f9d2b033640 1 --2- 172.21.15.12:0/1457973160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d241cd9f0 0x7f9d2409f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:39.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.688+0000 7f9d2b033640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:39.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.688+0000 7f9d2b033640 1 -- 172.21.15.12:0/1457973160 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9d24069950 con 0x7f9d241cd9f0 2024-09-11T01:53:39.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.688+0000 7f9d28dcf640 1 --2- 172.21.15.12:0/1457973160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d241cd9f0 0x7f9d2409f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:39.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.688+0000 7f9d28dcf640 10 monclient: get_auth_request con 0x7f9d241cd9f0 auth_method 0 2024-09-11T01:53:39.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.688+0000 7f9d28dcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:39.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.688+0000 7f9d28dcf640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:39.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.688+0000 7f9d28dcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:39.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.688+0000 7f9d28dcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:39.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.688+0000 7f9d28dcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:39.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.688+0000 7f9d28dcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:39.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.688+0000 7f9d28dcf640 10 monclient(hunting): handle_auth_done global_id 264255 payload 824 2024-09-11T01:53:39.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.688+0000 7f9d28dcf640 1 -- 172.21.15.12:0/1457973160 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9d2413c9a0 con 0x7f9d241cd9f0 2024-09-11T01:53:39.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.690+0000 7f9d28dcf640 1 -- 172.21.15.12:0/1457973160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d241cd9f0 msgr2=0x7f9d2409f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:39.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.690+0000 7f9d28dcf640 1 --2- 172.21.15.12:0/1457973160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d241cd9f0 0x7f9d2409f380 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9d1400fae0 tx=0x7f9d14003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:39.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.691+0000 7f9d217fa640 1 -- 172.21.15.12:0/1457973160 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d14018030 con 0x7f9d241cd9f0 2024-09-11T01:53:39.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.691+0000 7f9d217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:39.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.691+0000 7f9d217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:53:39.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.691+0000 7f9d217fa640 10 monclient: dump: 2024-09-11T01:53:39.721 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:39.721 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:39.721 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.721 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.721 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:39.722 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:39.722 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:39.722 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:39.722 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:39.722 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:39.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.691+0000 7f9d217fa640 1 -- 172.21.15.12:0/1457973160 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4710 (secure 0 0 0) 0x7f9d140069e0 con 0x7f9d241cd9f0 2024-09-11T01:53:39.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.691+0000 7f9d217fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:53:39.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.691+0000 7f9d217fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:53:39.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.691+0000 7f9d217fa640 1 -- 172.21.15.12:0/1457973160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d241cd9f0 msgr2=0x7f9d2409f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:39.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.691+0000 7f9d217fa640 1 --2- 172.21.15.12:0/1457973160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d241cd9f0 0x7f9d2409f380 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9d1400fae0 tx=0x7f9d14003760 comp rx=0 tx=0).stop 2024-09-11T01:53:39.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.691+0000 7f9d217fa640 1 -- 172.21.15.12:0/1457973160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d241cd9f0 msgr2=0x7f9d2409f380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:53:39.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.691+0000 7f9d217fa640 1 --2- 172.21.15.12:0/1457973160 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9d241cd9f0 0x7f9d2409f380 secure :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9d1400fae0 tx=0x7f9d14003760 comp rx=0 tx=0).stop 2024-09-11T01:53:39.723 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:53:39.724 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:53:39.724 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:53:39.724 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:53:39.724 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 226, 2024-09-11T01:53:39.724 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 61397, 2024-09-11T01:53:39.724 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:53:39.725 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:53:39.725 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:53:39.725 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:53:39.725 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:39.725 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 4, 2024-09-11T01:53:39.725 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:53:39.725 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:53:39.726 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:53:39.726 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:53:39.726 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:53:39.726 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:53:39.726 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:53:39.726 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:53:39.726 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:53:39.726 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:53:39.726 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:53:39.727 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:53:39.727 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:53:39.727 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:53:39.727 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:39.727 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:53:39.728 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:53:39.728 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:53:39.728 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:53:39.728 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:53:39.728 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:53:39.728 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:53:39.728 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:53:39.729 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:53:39.729 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:53:39.729 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:53:39.729 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:53:39.729 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:53:39.729 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:39.730 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:39.730 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:53:39.730 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:53:39.730 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:53:39.730 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:53:39.730 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:53:39.731 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:53:39.731 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:53:39.731 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:53:39.731 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:53:39.731 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:53:39.731 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:53:39.731 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:53:39.731 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:53:39.731 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:53:39.732 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:53:39.732 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:53:39.732 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:53:39.732 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:53:39.732 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:53:39.732 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:53:39.732 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:53:39.732 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:53:39.732 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:53:39.733 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:53:39.733 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:53:39.733 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:53:39.733 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:53:39.733 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:53:39.733 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:39.733 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:53:39.733 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:39.734 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:53:39.734 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:39.734 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:53:39.734 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:53:39.734 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:53:39.734 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:53:39.734 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:39.734 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:53:39.734 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:53:39.735 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:39.735 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:39.735 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:39.735 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:53:39.735 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:53:39.735 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:39.735 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:39.735 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:39.735 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:39.735 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:53:39.736 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:53:39.736 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:53:39.736 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:53:39.736 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:53:39.736 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:39.736 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:39.736 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:53:39.736 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:53:39.736 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:53:39.737 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:53:39.737 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:39.737 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:53:39.737 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:53:39.737 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:39.737 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:39.737 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:39.737 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:53:39.738 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:53:39.738 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:39.738 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:39.738 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:39.738 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:39.738 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:53:39.738 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:53:39.738 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:53:39.739 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:53:39.739 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:53:39.739 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:39.739 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:39.739 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:53:39.739 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:53:39.739 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:53:39.739 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:53:39.739 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:39.740 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:53:39.740 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:53:39.740 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:39.740 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:39.740 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:39.740 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:53:39.740 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:53:39.742 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:39.742 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:39.743 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:39.743 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:39.743 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:53:39.743 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:53:39.743 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:53:39.743 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:53:39.743 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:53:39.744 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:39.744 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:39.744 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:39.744 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:53:39.744 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:53:39.744 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:39.744 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:53:39.745 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:53:39.745 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:53:39.745 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:39.745 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:39.745 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:53:39.745 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:39.745 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:53:39.746 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:53:39.746 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:53:39.746 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:39.746 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:39.746 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:53:39.746 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:39.746 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:53:39.746 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:53:39.746 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:53:39.746 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:39.746 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:39.746 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:39.746 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:53:39.747 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:53:39.749 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:53:39.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.691+0000 7f9d2b033640 1 -- 172.21.15.12:0/1457973160 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9d1003e260 msgr2=0x7f9d10040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:39.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.691+0000 7f9d2b033640 1 --2- 172.21.15.12:0/1457973160 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9d1003e260 0x7f9d10040720 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f9d2413c450 tx=0x7f9d1802e000 comp rx=0 tx=0).stop 2024-09-11T01:53:39.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.691+0000 7f9d2b033640 10 monclient: shutdown 2024-09-11T01:53:39.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.691+0000 7f9d2b033640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:39.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.691+0000 7f9d2b033640 1 -- 172.21.15.12:0/1457973160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d2413d380 msgr2=0x7f9d24136f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:39.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.691+0000 7f9d2b033640 1 --2- 172.21.15.12:0/1457973160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d2413d380 0x7f9d24136f40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c00cb50 tx=0x7f9d1c00cf50 comp rx=0 tx=0).stop 2024-09-11T01:53:39.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.691+0000 7f9d295d0640 1 -- 172.21.15.12:0/1457973160 reap_dead start 2024-09-11T01:53:39.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.691+0000 7f9d295d0640 1 -- 172.21.15.12:0/1457973160 reap_dead start 2024-09-11T01:53:39.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.691+0000 7f9d217fa640 1 -- 172.21.15.12:0/1457973160 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(834..834 src has 676..834) ==== 4815+0+0 (unknown 0 0 0) 0x7f9d1c053330 con 0x7f9d2413d380 2024-09-11T01:53:39.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.691+0000 7f9d2b033640 1 -- 172.21.15.12:0/1457973160 shutdown_connections 2024-09-11T01:53:39.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.691+0000 7f9d2b033640 1 -- 172.21.15.12:0/1457973160 >> 172.21.15.12:0/1457973160 conn(0x7f9d240a7020 msgr2=0x7f9d2413f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:39.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.691+0000 7f9d2b033640 1 -- 172.21.15.12:0/1457973160 shutdown_connections 2024-09-11T01:53:39.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.691+0000 7f9d2b033640 1 -- 172.21.15.12:0/1457973160 wait complete. 2024-09-11T01:53:39.751 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:53:39.751 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:53:39.751 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:53:39.766 INFO:tasks.rados.rados.0.smithi071.stdout:3640: finishing write tid 1 to smithi07134777-5 2024-09-11T01:53:39.766 INFO:tasks.rados.rados.0.smithi071.stdout:3640: finishing write tid 2 to smithi07134777-5 2024-09-11T01:53:39.767 INFO:tasks.rados.rados.0.smithi071.stdout:3640: finishing write tid 3 to smithi07134777-5 2024-09-11T01:53:39.767 INFO:tasks.rados.rados.0.smithi071.stdout:3640: finishing write tid 4 to smithi07134777-5 2024-09-11T01:53:39.767 INFO:tasks.rados.rados.0.smithi071.stdout:3640: finishing write tid 5 to smithi07134777-5 2024-09-11T01:53:39.767 INFO:tasks.rados.rados.0.smithi071.stdout:3640: oid 5 updating version 0 to 1188 2024-09-11T01:53:39.767 INFO:tasks.rados.rados.0.smithi071.stdout:3640: oid 5 updating version 1188 to 1190 2024-09-11T01:53:39.768 INFO:tasks.rados.rados.0.smithi071.stdout:3640: oid 5 version 1190 is already newer than 1189 2024-09-11T01:53:39.768 INFO:tasks.rados.rados.0.smithi071.stdout:3640: oid 5 version 1190 is already newer than 1187 2024-09-11T01:53:39.768 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 1190 (ObjNum 1255 snap 415 seq_num 1255) dirty exists 2024-09-11T01:53:39.768 INFO:tasks.rados.rados.0.smithi071.stdout:3640: left oid 5 (ObjNum 1255 snap 415 seq_num 1255) 2024-09-11T01:53:39.768 INFO:tasks.rados.rados.0.smithi071.stdout:3642: finishing write tid 1 to smithi07134777-36 2024-09-11T01:53:39.768 INFO:tasks.rados.rados.0.smithi071.stdout:3642: finishing write tid 2 to smithi07134777-36 2024-09-11T01:53:39.769 INFO:tasks.rados.rados.0.smithi071.stdout:3642: finishing write tid 3 to smithi07134777-36 2024-09-11T01:53:39.769 INFO:tasks.rados.rados.0.smithi071.stdout:3642: finishing write tid 4 to smithi07134777-36 2024-09-11T01:53:39.769 INFO:tasks.rados.rados.0.smithi071.stdout:3642: finishing write tid 5 to smithi07134777-36 2024-09-11T01:53:39.769 INFO:tasks.rados.rados.0.smithi071.stdout:3642: oid 36 updating version 0 to 2107 2024-09-11T01:53:39.769 INFO:tasks.rados.rados.0.smithi071.stdout:3642: oid 36 version 2107 is already newer than 2104 2024-09-11T01:53:39.769 INFO:tasks.rados.rados.0.smithi071.stdout:3642: oid 36 version 2107 is already newer than 2106 2024-09-11T01:53:39.770 INFO:tasks.rados.rados.0.smithi071.stdout:3642: oid 36 version 2107 is already newer than 2105 2024-09-11T01:53:39.770 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 2107 (ObjNum 1256 snap 415 seq_num 1256) dirty exists 2024-09-11T01:53:39.770 INFO:tasks.rados.rados.0.smithi071.stdout:3642: left oid 36 (ObjNum 1256 snap 415 seq_num 1256) 2024-09-11T01:53:39.770 INFO:tasks.rados.rados.0.smithi071.stdout:3643: expect (ObjNum 1242 snap 411 seq_num 1242) 2024-09-11T01:53:39.792 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:53:39.825 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1326477 2024-09-11T01:53:39.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.856+0000 7fe59885a640 10 monclient: get_monmap_and_config 2024-09-11T01:53:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.856+0000 7fe59885a640 10 monclient: build_initial_monmap 2024-09-11T01:53:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.856+0000 7fe59885a640 10 monclient: monmap: 2024-09-11T01:53:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:53:39.856901+0000 2024-09-11T01:53:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:53:39.856901+0000 2024-09-11T01:53:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:53:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:53:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:53:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:53:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:53:39.859 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:39.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.856+0000 7fe59885a640 1 Processor -- start 2024-09-11T01:53:39.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.856+0000 7fe59885a640 1 -- start start 2024-09-11T01:53:39.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.856+0000 7fe59885a640 10 monclient: init 2024-09-11T01:53:39.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59885a640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:39.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59885a640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-11T01:53:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59885a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe594137e50 0x7fe59409d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59885a640 10 monclient(hunting): picked mon.noname-b con 0x7fe594137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:53:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59885a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe594138230 addr v1:172.21.15.12:6789/0 2024-09-11T01:53:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59885a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe59413d380 addr v1:172.21.15.71:6789/0 2024-09-11T01:53:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59885a640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5940a6580 con 0x7fe594138230 2024-09-11T01:53:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59885a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5940a62d0 con 0x7fe59413d380 2024-09-11T01:53:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59885a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59885a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe59409c4a0 con 0x7fe594137e50 2024-09-11T01:53:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59885a640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59885a640 10 monclient(hunting): authenticate will time out at 2226.299508s 2024-09-11T01:53:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59259c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe594137e50 0x7fe59409d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe589d9b640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe594138230 0x7fe594138630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:59022/0 (socket says 172.21.15.12:59022) 2024-09-11T01:53:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe589d9b640 1 -- 172.21.15.12:0/572045167 learned_addr learned my addr 172.21.15.12:0/572045167 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:53:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59259c640 10 monclient(hunting): get_auth_request con 0x7fe594137e50 auth_method 0 2024-09-11T01:53:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59259c640 10 monclient(hunting): handle_auth_done global_id 244439 payload 274 2024-09-11T01:53:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59259c640 1 -- 172.21.15.12:0/572045167 >> v1:172.21.15.71:6789/0 conn(0x7fe59413d380 legacy=0x7fe59413f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:53:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59259c640 1 -- 172.21.15.12:0/572045167 >> v1:172.21.15.12:6789/0 conn(0x7fe594138230 legacy=0x7fe594138630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:53:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59259c640 10 monclient: _finish_hunting 0 2024-09-11T01:53:39.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59259c640 1 monclient: found mon.noname-b 2024-09-11T01:53:39.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:39.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:53:39.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.857+0000 7fe59259c640 1 -- 172.21.15.12:0/572045167 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe59409c780 con 0x7fe594137e50 2024-09-11T01:53:39.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.860+0000 7fe59259c640 1 -- 172.21.15.12:0/572045167 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe594137e50 msgr2=0x7fe59409d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:39.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.860+0000 7fe59259c640 1 --2- 172.21.15.12:0/572045167 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe594137e50 0x7fe59409d160 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe5840097d0 tx=0x7fe584029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47cfaffe89196b53 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:39.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.860+0000 7fe591d9b640 1 -- 172.21.15.12:0/572045167 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe584037070 con 0x7fe594137e50 2024-09-11T01:53:39.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.860+0000 7fe591d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:39.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.860+0000 7fe591d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:53:39.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.860+0000 7fe591d9b640 10 monclient: dump: 2024-09-11T01:53:39.873 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:39.873 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:39.873 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.873 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.873 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:39.873 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:39.873 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:39.873 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:39.873 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:39.874 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:39.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.860+0000 7fe591d9b640 10 monclient: _finish_auth 0 2024-09-11T01:53:39.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.860+0000 7fe591d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:53:39.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.860+0000 7fe591d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:39.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.860+0000 7fe591d9b640 1 -- 172.21.15.12:0/572045167 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe584032440 con 0x7fe594137e50 2024-09-11T01:53:39.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.860+0000 7fe591d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:39.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.860+0000 7fe591d9b640 1 -- 172.21.15.12:0/572045167 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe584032810 con 0x7fe594137e50 2024-09-11T01:53:39.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.860+0000 7fe59885a640 5 monclient: authenticate success, global_id 244439 2024-09-11T01:53:39.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.860+0000 7fe59885a640 10 monclient: get_monmap_and_config success 2024-09-11T01:53:39.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.860+0000 7fe59885a640 4 set_mon_vals no callback set 2024-09-11T01:53:39.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.860+0000 7fe591d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:39.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.860+0000 7fe591d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:53:39.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.860+0000 7fe591d9b640 10 monclient: dump: 2024-09-11T01:53:39.875 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:39.875 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:39.875 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.875 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.875 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:39.875 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:39.876 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:39.876 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:39.876 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:39.876 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:39.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.861+0000 7fe59885a640 10 monclient: shutdown 2024-09-11T01:53:39.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.861+0000 7fe59885a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:39.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.861+0000 7fe59885a640 1 -- 172.21.15.12:0/572045167 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe594137e50 msgr2=0x7fe59409d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:39.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.861+0000 7fe59885a640 1 --2- 172.21.15.12:0/572045167 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe594137e50 0x7fe59409d160 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe5840097d0 tx=0x7fe584029fb0 comp rx=0 tx=0).stop 2024-09-11T01:53:39.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.861+0000 7fe59885a640 1 -- 172.21.15.12:0/572045167 shutdown_connections 2024-09-11T01:53:39.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.861+0000 7fe59885a640 1 --2- 172.21.15.12:0/572045167 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe594137e50 0x7fe59409d160 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:39.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.861+0000 7fe59885a640 1 -- 172.21.15.12:0/572045167 >> 172.21.15.12:0/572045167 conn(0x7fe5940a7020 msgr2=0x7fe5940a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:39.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.861+0000 7fe59885a640 1 -- 172.21.15.12:0/572045167 shutdown_connections 2024-09-11T01:53:39.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.861+0000 7fe59885a640 1 -- 172.21.15.12:0/572045167 wait complete. 2024-09-11T01:53:39.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.862+0000 7fe59885a640 10 monclient: build_initial_monmap 2024-09-11T01:53:39.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.862+0000 7fe59885a640 10 monclient: monmap: 2024-09-11T01:53:39.877 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:39.877 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:53:39.877 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:53:39.877 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:53:39.877 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:39.877 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:39.878 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:53:39.878 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:53:39.878 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:53:39.878 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:39.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.862+0000 7fe59885a640 1 Processor -- start 2024-09-11T01:53:39.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.862+0000 7fe59885a640 1 -- start start 2024-09-11T01:53:39.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.862+0000 7fe59885a640 10 monclient: init 2024-09-11T01:53:39.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe59885a640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:39.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe59885a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:53:39.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe59885a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe594137e50 0x7fe59413baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:39.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe59885a640 10 monclient(hunting): picked mon.noname-a con 0x7fe594137e50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:53:39.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe59885a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe594138230 0x7fe594139ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:39.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe59885a640 10 monclient(hunting): picked mon.noname-c con 0x7fe594138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:53:39.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe59885a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe59413d380 0x7fe594137160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:39.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe59885a640 10 monclient(hunting): picked mon.noname-b con 0x7fe59413d380 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:53:39.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe59885a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:39.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe59885a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe59406f890 con 0x7fe594137e50 2024-09-11T01:53:39.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe59885a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:39.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe59885a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe59406f710 con 0x7fe59413d380 2024-09-11T01:53:39.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe59885a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:39.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe59885a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe594140e90 con 0x7fe594138230 2024-09-11T01:53:39.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe59885a640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:39.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe59885a640 10 monclient(hunting): authenticate will time out at 2226.305100s 2024-09-11T01:53:39.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe59259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe594137e50 0x7fe59413baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:39.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe589d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe594138230 0x7fe594139ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:39.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe589d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe594138230 0x7fe594139ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42972/0 (socket says 172.21.15.12:42972) 2024-09-11T01:53:39.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe59259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe594137e50 0x7fe59413baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39588/0 (socket says 172.21.15.12:39588) 2024-09-11T01:53:39.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe589d9b640 1 -- 172.21.15.12:0/306385969 learned_addr learned my addr 172.21.15.12:0/306385969 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:53:39.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe589d9b640 10 monclient(hunting): get_auth_request con 0x7fe594138230 auth_method 0 2024-09-11T01:53:39.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe589d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:39.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe589d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:39.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe589d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:39.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe59259c640 10 monclient(hunting): get_auth_request con 0x7fe594137e50 auth_method 0 2024-09-11T01:53:39.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe59259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:39.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe59259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:39.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe59259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:39.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe589d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:39.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe589d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:39.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe592d9d640 1 --2- 172.21.15.12:0/306385969 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe59413d380 0x7fe594137160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:39.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe589d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:39.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe59259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:39.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe59259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:39.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe59259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:39.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe592d9d640 10 monclient(hunting): get_auth_request con 0x7fe59413d380 auth_method 0 2024-09-11T01:53:39.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe592d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:39.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe592d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:39.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe592d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:39.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe592d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:39.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe592d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:39.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe592d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:39.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe589d9b640 10 monclient(hunting): handle_auth_done global_id 244442 payload 824 2024-09-11T01:53:39.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe589d9b640 1 -- 172.21.15.12:0/306385969 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe59413d380 msgr2=0x7fe594137160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:39.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe589d9b640 1 --2- 172.21.15.12:0/306385969 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe59413d380 0x7fe594137160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:39.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe589d9b640 1 -- 172.21.15.12:0/306385969 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe594137e50 msgr2=0x7fe59413baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:39.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe589d9b640 1 --2- 172.21.15.12:0/306385969 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe594137e50 0x7fe59413baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:39.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe589d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:53:39.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe589d9b640 1 monclient: found mon.noname-c 2024-09-11T01:53:39.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe589d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:39.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe589d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:53:39.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe589d9b640 1 -- 172.21.15.12:0/306385969 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe594140a20 con 0x7fe594138230 2024-09-11T01:53:39.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.863+0000 7fe59259c640 1 --2- 172.21.15.12:0/306385969 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe594137e50 0x7fe59413baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:53:39.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.865+0000 7fe589d9b640 1 -- 172.21.15.12:0/306385969 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe594138230 msgr2=0x7fe594139ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:39.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.865+0000 7fe589d9b640 1 --2- 172.21.15.12:0/306385969 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe594138230 0x7fe594139ae0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe57c00c460 tx=0x7fe57c00c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:39.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.865+0000 7fe58b7fe640 1 -- 172.21.15.12:0/306385969 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c017070 con 0x7fe594138230 2024-09-11T01:53:39.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.865+0000 7fe58b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:39.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.865+0000 7fe58b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:53:39.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.865+0000 7fe58b7fe640 10 monclient: dump: 2024-09-11T01:53:39.883 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:39.883 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:39.883 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.883 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.883 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:39.883 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:39.883 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:39.884 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:39.884 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:39.884 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:39.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.865+0000 7fe58b7fe640 10 monclient: _finish_auth 0 2024-09-11T01:53:39.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.865+0000 7fe58b7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:53:39.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.865+0000 7fe58b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:39.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.865+0000 7fe59885a640 5 monclient: authenticate success, global_id 244442 2024-09-11T01:53:39.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.865+0000 7fe58b7fe640 1 -- 172.21.15.12:0/306385969 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe57c0070d0 con 0x7fe594138230 2024-09-11T01:53:39.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.865+0000 7fe58b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:39.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.865+0000 7fe58b7fe640 1 -- 172.21.15.12:0/306385969 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c0074c0 con 0x7fe594138230 2024-09-11T01:53:39.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.866+0000 7fe59885a640 10 monclient: _renew_subs 2024-09-11T01:53:39.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.866+0000 7fe59885a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:53:39.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.866+0000 7fe59885a640 1 -- 172.21.15.12:0/306385969 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5940a9530 con 0x7fe594138230 2024-09-11T01:53:39.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.866+0000 7fe59159a640 4 set_mon_vals no callback set 2024-09-11T01:53:39.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.866+0000 7fe58b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:39.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.866+0000 7fe58b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:53:39.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.866+0000 7fe58b7fe640 10 monclient: dump: 2024-09-11T01:53:39.885 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:39.885 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:39.885 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.885 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.885 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:39.885 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:39.885 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:39.885 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:39.886 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:39.886 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:39.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.866+0000 7fe59885a640 10 monclient: _renew_subs 2024-09-11T01:53:39.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.866+0000 7fe59885a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:53:39.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.866+0000 7fe59885a640 1 -- 172.21.15.12:0/306385969 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe59413a020 con 0x7fe594138230 2024-09-11T01:53:39.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.866+0000 7fe59885a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:39.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.866+0000 7fe59885a640 1 --2- 172.21.15.12:0/306385969 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe594137700 0x7fe5941cd360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:39.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.866+0000 7fe59885a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:39.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.866+0000 7fe59885a640 1 -- 172.21.15.12:0/306385969 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe594137ae0 con 0x7fe594137700 2024-09-11T01:53:39.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.867+0000 7fe59259c640 1 --2- 172.21.15.12:0/306385969 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe594137700 0x7fe5941cd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:39.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.867+0000 7fe58b7fe640 1 -- 172.21.15.12:0/306385969 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fe57c01b3b0 con 0x7fe594138230 2024-09-11T01:53:39.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.867+0000 7fe59259c640 10 monclient: get_auth_request con 0x7fe594137700 auth_method 0 2024-09-11T01:53:39.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.867+0000 7fe59259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:39.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.867+0000 7fe59259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:39.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.867+0000 7fe59259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:39.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.867+0000 7fe59259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:39.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.867+0000 7fe59259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:39.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.867+0000 7fe59259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:39.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.867+0000 7fe59259c640 10 monclient(hunting): handle_auth_done global_id 244445 payload 824 2024-09-11T01:53:39.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.867+0000 7fe59259c640 1 -- 172.21.15.12:0/306385969 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe59409d160 con 0x7fe594137700 2024-09-11T01:53:39.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.867+0000 7fe58b7fe640 1 --2- 172.21.15.12:0/306385969 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe57403e210 0x7fe5740406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:39.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.867+0000 7fe58b7fe640 1 -- 172.21.15.12:0/306385969 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(834..834 src has 676..834) ==== 4815+0+0 (secure 0 0 0) 0x7fe57c052590 con 0x7fe594138230 2024-09-11T01:53:39.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.867+0000 7fe592d9d640 1 --2- 172.21.15.12:0/306385969 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe57403e210 0x7fe5740406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:39.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.867+0000 7fe592d9d640 10 monclient: get_auth_request con 0x7fe57403e210 auth_method 0 2024-09-11T01:53:39.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.869+0000 7fe59259c640 1 -- 172.21.15.12:0/306385969 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe594137700 msgr2=0x7fe5941cd360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:39.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.869+0000 7fe59259c640 1 --2- 172.21.15.12:0/306385969 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe594137700 0x7fe5941cd360 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe584034000 tx=0x7fe5840046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:39.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.869+0000 7fe58b7fe640 1 -- 172.21.15.12:0/306385969 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe584037040 con 0x7fe594137700 2024-09-11T01:53:39.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.869+0000 7fe58b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:39.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.869+0000 7fe58b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:53:39.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.869+0000 7fe58b7fe640 10 monclient: dump: 2024-09-11T01:53:39.888 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:39.888 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:39.888 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.888 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.888 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:39.889 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:39.889 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:39.889 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:39.889 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:39.889 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:39.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.869+0000 7fe58b7fe640 1 -- 172.21.15.12:0/306385969 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe58403c820 con 0x7fe594137700 2024-09-11T01:53:39.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.869+0000 7fe58b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:39.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.869+0000 7fe58b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:53:39.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.869+0000 7fe58b7fe640 1 -- 172.21.15.12:0/306385969 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe594137700 msgr2=0x7fe5941cd360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:39.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.869+0000 7fe58b7fe640 1 --2- 172.21.15.12:0/306385969 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe594137700 0x7fe5941cd360 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe584034000 tx=0x7fe5840046c0 comp rx=0 tx=0).stop 2024-09-11T01:53:39.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.869+0000 7fe58b7fe640 1 -- 172.21.15.12:0/306385969 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe594137700 msgr2=0x7fe5941cd360 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:53:39.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.869+0000 7fe58b7fe640 1 --2- 172.21.15.12:0/306385969 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe594137700 0x7fe5941cd360 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:39.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.870+0000 7fe592d9d640 1 --2- 172.21.15.12:0/306385969 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe57403e210 0x7fe5740406d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fe59413b960 tx=0x7fe58c003d10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:39.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.876+0000 7fe59885a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:53:39.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.876+0000 7fe59885a640 1 --2- 172.21.15.12:0/306385969 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5941cd8a0 0x7fe5941cdc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:39.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.876+0000 7fe59885a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:39.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.876+0000 7fe59885a640 1 -- 172.21.15.12:0/306385969 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe5941ce1c0 con 0x7fe5941cd8a0 2024-09-11T01:53:39.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.876+0000 7fe59259c640 1 --2- 172.21.15.12:0/306385969 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5941cd8a0 0x7fe5941cdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:39.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.876+0000 7fe59259c640 10 monclient: get_auth_request con 0x7fe5941cd8a0 auth_method 0 2024-09-11T01:53:39.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.876+0000 7fe59259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:39.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.876+0000 7fe59259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:39.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.876+0000 7fe59259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:39.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.876+0000 7fe59259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:39.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.876+0000 7fe59259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:39.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.876+0000 7fe59259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:39.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.876+0000 7fe59259c640 10 monclient(hunting): handle_auth_done global_id 244448 payload 824 2024-09-11T01:53:39.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.876+0000 7fe59259c640 1 -- 172.21.15.12:0/306385969 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe594069950 con 0x7fe5941cd8a0 2024-09-11T01:53:39.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.878+0000 7fe59259c640 1 -- 172.21.15.12:0/306385969 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5941cd8a0 msgr2=0x7fe5941cdc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:39.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.878+0000 7fe59259c640 1 --2- 172.21.15.12:0/306385969 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5941cd8a0 0x7fe5941cdc80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe584000c00 tx=0x7fe584008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.878+0000 7fe58b7fe640 1 -- 172.21.15.12:0/306385969 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe584050070 con 0x7fe5941cd8a0 2024-09-11T01:53:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.878+0000 7fe58b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.878+0000 7fe58b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:53:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.878+0000 7fe58b7fe640 10 monclient: dump: 2024-09-11T01:53:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:39.895 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:39.895 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:39.895 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:39.895 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:39.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.879+0000 7fe58b7fe640 1 -- 172.21.15.12:0/306385969 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7fe584003ac0 con 0x7fe5941cd8a0 2024-09-11T01:53:39.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.879+0000 7fe58b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:53:39.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.879+0000 7fe58b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:53:39.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.879+0000 7fe58b7fe640 1 -- 172.21.15.12:0/306385969 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5941cd8a0 msgr2=0x7fe5941cdc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:39.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.879+0000 7fe58b7fe640 1 --2- 172.21.15.12:0/306385969 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5941cd8a0 0x7fe5941cdc80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe584000c00 tx=0x7fe584008030 comp rx=0 tx=0).stop 2024-09-11T01:53:39.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.879+0000 7fe58b7fe640 1 -- 172.21.15.12:0/306385969 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5941cd8a0 msgr2=0x7fe5941cdc80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:53:39.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.879+0000 7fe58b7fe640 1 --2- 172.21.15.12:0/306385969 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe5941cd8a0 0x7fe5941cdc80 secure :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe584000c00 tx=0x7fe584008030 comp rx=0 tx=0).stop 2024-09-11T01:53:39.897 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:53:39.897 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:53:39.897 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:53:39.897 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:53:39.897 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 226, 2024-09-11T01:53:39.897 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 173326, 2024-09-11T01:53:39.897 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:53:39.897 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:53:39.897 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:53:39.898 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:53:39.898 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:39.898 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 4, 2024-09-11T01:53:39.898 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:53:39.898 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:53:39.898 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:53:39.898 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:53:39.898 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:53:39.898 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:53:39.898 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:53:39.898 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:53:39.898 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:53:39.898 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:53:39.898 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:53:39.898 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:53:39.899 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:53:39.899 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:53:39.899 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:39.899 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:53:39.899 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:53:39.899 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:53:39.899 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:53:39.899 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:53:39.899 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:53:39.899 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:53:39.899 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:53:39.899 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:53:39.899 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:53:39.899 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:53:39.900 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:53:39.900 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:53:39.900 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:39.900 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:39.900 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:53:39.900 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:53:39.900 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:53:39.900 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:53:39.900 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:53:39.900 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:53:39.900 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:53:39.900 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:53:39.900 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:53:39.900 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:53:39.901 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:53:39.901 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:53:39.901 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:53:39.901 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:53:39.901 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:53:39.901 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:53:39.901 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:53:39.901 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:53:39.901 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:53:39.901 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:53:39.901 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:53:39.901 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:53:39.901 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:53:39.902 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:53:39.902 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:53:39.902 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:53:39.902 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:53:39.902 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:53:39.902 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:39.902 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:53:39.902 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:39.902 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:53:39.902 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:39.902 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:53:39.902 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:53:39.902 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:53:39.902 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:53:39.903 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:39.903 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:53:39.903 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:53:39.903 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:39.903 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:39.903 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:39.903 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:53:39.903 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:53:39.903 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:39.903 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:39.903 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:39.903 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:39.903 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:53:39.904 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:53:39.904 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:53:39.904 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:53:39.904 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:53:39.904 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:39.904 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:39.904 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:53:39.904 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:53:39.904 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:53:39.904 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:53:39.904 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:39.904 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:53:39.904 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:53:39.904 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:39.905 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:39.905 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:39.905 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:53:39.905 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:53:39.905 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:39.905 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:39.905 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:39.905 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:39.905 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:53:39.905 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:53:39.905 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:53:39.905 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:53:39.905 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:53:39.906 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:39.906 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:39.906 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:53:39.906 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:53:39.906 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:53:39.906 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:53:39.906 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:39.906 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:53:39.906 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:53:39.906 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:39.906 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:39.906 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:39.906 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:53:39.906 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:53:39.907 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:39.907 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:39.907 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:39.907 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:39.907 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:53:39.907 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:53:39.907 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:53:39.907 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:53:39.907 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:53:39.907 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:39.907 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:39.907 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:39.907 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:53:39.907 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:53:39.908 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:39.908 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:53:39.908 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:53:39.908 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:53:39.908 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:39.908 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:39.908 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:53:39.908 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:39.908 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:53:39.908 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:53:39.908 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:53:39.908 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:39.908 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:39.908 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:53:39.909 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:39.909 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:53:39.909 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:53:39.909 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:53:39.909 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:39.909 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:39.909 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:39.909 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:53:39.909 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:53:39.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.879+0000 7fe59885a640 1 -- 172.21.15.12:0/306385969 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe57403e210 msgr2=0x7fe5740406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:39.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.879+0000 7fe59885a640 1 --2- 172.21.15.12:0/306385969 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe57403e210 0x7fe5740406d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fe59413b960 tx=0x7fe58c003d10 comp rx=0 tx=0).stop 2024-09-11T01:53:39.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.879+0000 7fe59885a640 10 monclient: shutdown 2024-09-11T01:53:39.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.879+0000 7fe59885a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:39.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.879+0000 7fe59885a640 1 -- 172.21.15.12:0/306385969 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe594138230 msgr2=0x7fe594139ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:39.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.879+0000 7fe59885a640 1 --2- 172.21.15.12:0/306385969 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe594138230 0x7fe594139ae0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe57c00c460 tx=0x7fe57c00c920 comp rx=0 tx=0).stop 2024-09-11T01:53:39.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.879+0000 7fe592d9d640 1 -- 172.21.15.12:0/306385969 reap_dead start 2024-09-11T01:53:39.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.879+0000 7fe59885a640 1 -- 172.21.15.12:0/306385969 shutdown_connections 2024-09-11T01:53:39.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.879+0000 7fe59885a640 1 --2- 172.21.15.12:0/306385969 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe594138230 0x7fe594139ae0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:39.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.879+0000 7fe59885a640 1 -- 172.21.15.12:0/306385969 >> 172.21.15.12:0/306385969 conn(0x7fe5940a7020 msgr2=0x7fe59409eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:39.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.879+0000 7fe59885a640 1 -- 172.21.15.12:0/306385969 shutdown_connections 2024-09-11T01:53:39.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:39.879+0000 7fe59885a640 1 -- 172.21.15.12:0/306385969 wait complete. 2024-09-11T01:53:39.912 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:53:39.912 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:53:39.912 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:53:40.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.012+0000 7f8300532640 10 monclient: get_monmap_and_config 2024-09-11T01:53:40.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.012+0000 7f8300532640 10 monclient: build_initial_monmap 2024-09-11T01:53:40.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.012+0000 7f8300532640 10 monclient: monmap: 2024-09-11T01:53:40.014 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:40.014 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:40.014 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:53:40.013412+0000 2024-09-11T01:53:40.014 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:53:40.013412+0000 2024-09-11T01:53:40.014 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:40.015 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:40.015 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:53:40.015 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:53:40.015 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:53:40.015 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:53:40.015 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:53:40.015 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:40.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.013+0000 7f8300532640 1 Processor -- start 2024-09-11T01:53:40.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.013+0000 7f8300532640 1 -- start start 2024-09-11T01:53:40.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.013+0000 7f8300532640 10 monclient: init 2024-09-11T01:53:40.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.013+0000 7f8300532640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:40.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.013+0000 7f8300532640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-11T01:53:40.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.013+0000 7f8300532640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f82f8134ad0 0x7f82f8134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:40.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.013+0000 7f8300532640 10 monclient(hunting): picked mon.noname-c con 0x7f82f8134ad0 addr v2:172.21.15.71:3300/0 2024-09-11T01:53:40.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.013+0000 7f8300532640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f82f80a2200 addr v1:172.21.15.12:6789/0 2024-09-11T01:53:40.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.013+0000 7f8300532640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f82f81353f0 0x7f82f8134160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:40.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.013+0000 7f8300532640 10 monclient(hunting): picked mon.noname-a con 0x7f82f81353f0 addr v2:172.21.15.12:3300/0 2024-09-11T01:53:40.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.013+0000 7f8300532640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82f809ed30 con 0x7f82f80a2200 2024-09-11T01:53:40.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.013+0000 7f8300532640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:40.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.013+0000 7f8300532640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f82f809f5d0 con 0x7f82f81353f0 2024-09-11T01:53:40.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.013+0000 7f8300532640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:40.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.013+0000 7f8300532640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f82f809f010 con 0x7f82f8134ad0 2024-09-11T01:53:40.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.013+0000 7f8300532640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:40.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.013+0000 7f8300532640 10 monclient(hunting): authenticate will time out at 2226.455958s 2024-09-11T01:53:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82feacf640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f82f81353f0 0x7f82f8134160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82feacf640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f82f81353f0 0x7f82f8134160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39604/0 (socket says 172.21.15.12:39604) 2024-09-11T01:53:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82feacf640 1 -- 172.21.15.12:0/2219411535 learned_addr learned my addr 172.21.15.12:0/2219411535 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:53:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82feacf640 10 monclient(hunting): get_auth_request con 0x7f82f81353f0 auth_method 0 2024-09-11T01:53:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82feacf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82feacf640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82feacf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82feacf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82feacf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82fe2ce640 1 --2- 172.21.15.12:0/2219411535 >> v2:172.21.15.71:3300/0 conn(0x7f82f8134ad0 0x7f82f8134eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82feacf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82fe2ce640 10 monclient(hunting): get_auth_request con 0x7f82f8134ad0 auth_method 0 2024-09-11T01:53:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82fe2ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82fe2ce640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:40.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82fe2ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:40.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82fe2ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:40.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82fe2ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:40.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82fe2ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:40.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82feacf640 10 monclient(hunting): handle_auth_done global_id 264261 payload 274 2024-09-11T01:53:40.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82feacf640 1 -- 172.21.15.12:0/2219411535 >> v2:172.21.15.71:3300/0 conn(0x7f82f8134ad0 msgr2=0x7f82f8134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:53:40.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82feacf640 1 --2- 172.21.15.12:0/2219411535 >> v2:172.21.15.71:3300/0 conn(0x7f82f8134ad0 0x7f82f8134eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:40.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82feacf640 1 -- 172.21.15.12:0/2219411535 >> v1:172.21.15.12:6789/0 conn(0x7f82f80a2200 legacy=0x7f82f8133a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:53:40.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82feacf640 10 monclient: _finish_hunting 0 2024-09-11T01:53:40.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82feacf640 1 monclient: found mon.noname-a 2024-09-11T01:53:40.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82feacf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:40.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82feacf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:53:40.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.014+0000 7f82feacf640 1 -- 172.21.15.12:0/2219411535 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82f809f2f0 con 0x7f82f81353f0 2024-09-11T01:53:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f82feacf640 1 -- 172.21.15.12:0/2219411535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82f81353f0 msgr2=0x7f82f8134160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f82feacf640 1 --2- 172.21.15.12:0/2219411535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82f81353f0 0x7f82f8134160 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f82f0009e50 tx=0x7f82f002a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4e420dee6c718bc server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f82fd2cc640 1 -- 172.21.15.12:0/2219411535 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f0036070 con 0x7f82f81353f0 2024-09-11T01:53:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f82fd2cc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f82fd2cc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:53:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f82fd2cc640 10 monclient: dump: 2024-09-11T01:53:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:40.028 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f82fd2cc640 10 monclient: _finish_auth 0 2024-09-11T01:53:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f82fd2cc640 10 monclient: _check_auth_tickets 2024-09-11T01:53:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f82fd2cc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f82fd2cc640 1 -- 172.21.15.12:0/2219411535 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82f0007800 con 0x7f82f81353f0 2024-09-11T01:53:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f8300532640 5 monclient: authenticate success, global_id 264261 2024-09-11T01:53:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f8300532640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:53:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f82fd2cc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f8300532640 10 monclient: get_monmap_and_config success 2024-09-11T01:53:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f82fd2cc640 1 -- 172.21.15.12:0/2219411535 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f0007bd0 con 0x7f82f81353f0 2024-09-11T01:53:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f8300532640 4 set_mon_vals no callback set 2024-09-11T01:53:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f82fd2cc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f82fd2cc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:53:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f82fd2cc640 10 monclient: dump: 2024-09-11T01:53:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:40.029 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f8300532640 10 monclient: shutdown 2024-09-11T01:53:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f8300532640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f8300532640 1 -- 172.21.15.12:0/2219411535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82f81353f0 msgr2=0x7f82f8134160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f8300532640 1 --2- 172.21.15.12:0/2219411535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82f81353f0 0x7f82f8134160 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f82f0009e50 tx=0x7f82f002a630 comp rx=0 tx=0).stop 2024-09-11T01:53:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f8300532640 1 -- 172.21.15.12:0/2219411535 shutdown_connections 2024-09-11T01:53:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f8300532640 1 --2- 172.21.15.12:0/2219411535 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82f81353f0 0x7f82f8134160 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f8300532640 1 --2- 172.21.15.12:0/2219411535 >> v2:172.21.15.71:3300/0 conn(0x7f82f8134ad0 0x7f82f8134eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.017+0000 7f8300532640 1 -- 172.21.15.12:0/2219411535 >> 172.21.15.12:0/2219411535 conn(0x7f82f80a2f70 msgr2=0x7f82f80a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.018+0000 7f8300532640 1 -- 172.21.15.12:0/2219411535 shutdown_connections 2024-09-11T01:53:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.018+0000 7f8300532640 1 -- 172.21.15.12:0/2219411535 wait complete. 2024-09-11T01:53:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.018+0000 7f8300532640 10 monclient: build_initial_monmap 2024-09-11T01:53:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.018+0000 7f8300532640 10 monclient: monmap: 2024-09-11T01:53:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:53:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:53:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:53:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:53:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:53:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:53:40.034 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.018+0000 7f8300532640 1 Processor -- start 2024-09-11T01:53:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.018+0000 7f8300532640 1 -- start start 2024-09-11T01:53:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.018+0000 7f8300532640 10 monclient: init 2024-09-11T01:53:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f8300532640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f8300532640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:53:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f8300532640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f82f80a2200 0x7f82f806f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f8300532640 10 monclient(hunting): picked mon.noname-c con 0x7f82f80a2200 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:53:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f8300532640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82f8134ad0 0x7f82f806fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f8300532640 10 monclient(hunting): picked mon.noname-b con 0x7f82f8134ad0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:53:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f8300532640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82f81353f0 0x7f82f8070140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f8300532640 10 monclient(hunting): picked mon.noname-a con 0x7f82f81353f0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:53:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f8300532640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f8300532640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f82f81346a0 con 0x7f82f81353f0 2024-09-11T01:53:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f8300532640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f8300532640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f82f8135920 con 0x7f82f8134ad0 2024-09-11T01:53:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f8300532640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f8300532640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f82f8134820 con 0x7f82f80a2200 2024-09-11T01:53:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f8300532640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f8300532640 10 monclient(hunting): authenticate will time out at 2226.461509s 2024-09-11T01:53:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fe2ce640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f82f80a2200 0x7f82f806f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82feacf640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82f81353f0 0x7f82f8070140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fe2ce640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f82f80a2200 0x7f82f806f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42996/0 (socket says 172.21.15.12:42996) 2024-09-11T01:53:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fe2ce640 1 -- 172.21.15.12:0/973704161 learned_addr learned my addr 172.21.15.12:0/973704161 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:53:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fe2ce640 10 monclient(hunting): get_auth_request con 0x7f82f80a2200 auth_method 0 2024-09-11T01:53:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fe2ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fe2ce640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fe2ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82feacf640 10 monclient(hunting): get_auth_request con 0x7f82f81353f0 auth_method 0 2024-09-11T01:53:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82feacf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82feacf640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82feacf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fdacd640 1 --2- 172.21.15.12:0/973704161 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82f8134ad0 0x7f82f806fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fe2ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fe2ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fe2ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fdacd640 10 monclient(hunting): get_auth_request con 0x7f82f8134ad0 auth_method 0 2024-09-11T01:53:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fdacd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fdacd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fdacd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82feacf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82feacf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82feacf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fdacd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fdacd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fdacd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fe2ce640 10 monclient(hunting): handle_auth_done global_id 244451 payload 824 2024-09-11T01:53:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fe2ce640 1 -- 172.21.15.12:0/973704161 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82f8134ad0 msgr2=0x7f82f806fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fe2ce640 1 --2- 172.21.15.12:0/973704161 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82f8134ad0 0x7f82f806fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fe2ce640 1 -- 172.21.15.12:0/973704161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82f81353f0 msgr2=0x7f82f8070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fe2ce640 1 --2- 172.21.15.12:0/973704161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82f81353f0 0x7f82f8070140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fe2ce640 10 monclient: _finish_hunting 0 2024-09-11T01:53:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fe2ce640 1 monclient: found mon.noname-c 2024-09-11T01:53:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fe2ce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:53:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fe2ce640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:53:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82fe2ce640 1 -- 172.21.15.12:0/973704161 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82f8127200 con 0x7f82f80a2200 2024-09-11T01:53:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.019+0000 7f82feacf640 1 --2- 172.21.15.12:0/973704161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f82f81353f0 0x7f82f8070140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:53:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.022+0000 7f82fe2ce640 1 -- 172.21.15.12:0/973704161 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f82f80a2200 msgr2=0x7f82f806f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.022+0000 7f82fe2ce640 1 --2- 172.21.15.12:0/973704161 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f82f80a2200 0x7f82f806f6c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f82f400e9b0 tx=0x7f82f400ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.022+0000 7f82eeffd640 1 -- 172.21.15.12:0/973704161 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f4017070 con 0x7f82f80a2200 2024-09-11T01:53:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.022+0000 7f82eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.022+0000 7f82eeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:53:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.022+0000 7f82eeffd640 10 monclient: dump: 2024-09-11T01:53:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:40.039 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:40.039 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:40.039 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:40.039 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:40.039 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:40.039 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:40.039 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:40.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.022+0000 7f82eeffd640 10 monclient: _finish_auth 0 2024-09-11T01:53:40.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.022+0000 7f82eeffd640 10 monclient: _check_auth_tickets 2024-09-11T01:53:40.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.022+0000 7f82eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:53:40.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.022+0000 7f82eeffd640 1 -- 172.21.15.12:0/973704161 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82f400cbb0 con 0x7f82f80a2200 2024-09-11T01:53:40.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.022+0000 7f8300532640 5 monclient: authenticate success, global_id 244451 2024-09-11T01:53:40.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.022+0000 7f8300532640 10 monclient: _renew_subs 2024-09-11T01:53:40.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.022+0000 7f8300532640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:53:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.022+0000 7f8300532640 1 -- 172.21.15.12:0/973704161 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82f8125fc0 con 0x7f82f80a2200 2024-09-11T01:53:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.022+0000 7f82eeffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:53:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.022+0000 7f82eeffd640 1 -- 172.21.15.12:0/973704161 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f4013470 con 0x7f82f80a2200 2024-09-11T01:53:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.022+0000 7f82eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.022+0000 7f82eeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:53:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.022+0000 7f82eeffd640 10 monclient: dump: 2024-09-11T01:53:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:40.040 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:40.041 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.022+0000 7f8300532640 10 monclient: _renew_subs 2024-09-11T01:53:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.022+0000 7f8300532640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:53:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.022+0000 7f8300532640 1 -- 172.21.15.12:0/973704161 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f82f8126480 con 0x7f82f80a2200 2024-09-11T01:53:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.022+0000 7f82fcacb640 4 set_mon_vals no callback set 2024-09-11T01:53:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.023+0000 7f8300532640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.023+0000 7f8300532640 1 --2- 172.21.15.12:0/973704161 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82f81cd850 0x7f82f81cdc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.023+0000 7f8300532640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.023+0000 7f8300532640 1 -- 172.21.15.12:0/973704161 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f82f80a42b0 con 0x7f82f81cd850 2024-09-11T01:53:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.023+0000 7f82eeffd640 1 -- 172.21.15.12:0/973704161 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f82f4024060 con 0x7f82f80a2200 2024-09-11T01:53:40.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.023+0000 7f82eeffd640 1 --2- 172.21.15.12:0/973704161 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f82dc03e210 0x7f82dc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.023+0000 7f82fdacd640 1 --2- 172.21.15.12:0/973704161 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82f81cd850 0x7f82f81cdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.023+0000 7f82fdacd640 10 monclient: get_auth_request con 0x7f82f81cd850 auth_method 0 2024-09-11T01:53:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.023+0000 7f82fdacd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.023+0000 7f82fdacd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.023+0000 7f82fdacd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.023+0000 7f82eeffd640 1 -- 172.21.15.12:0/973704161 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(834..834 src has 676..834) ==== 4815+0+0 (secure 0 0 0) 0x7f82f4052dd0 con 0x7f82f80a2200 2024-09-11T01:53:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.023+0000 7f82fdacd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.023+0000 7f82fdacd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.023+0000 7f82fdacd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.023+0000 7f82feacf640 1 --2- 172.21.15.12:0/973704161 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f82dc03e210 0x7f82dc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.023+0000 7f82feacf640 10 monclient: get_auth_request con 0x7f82dc03e210 auth_method 0 2024-09-11T01:53:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.023+0000 7f82fdacd640 10 monclient(hunting): handle_auth_done global_id 274129 payload 824 2024-09-11T01:53:40.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.023+0000 7f82fdacd640 1 -- 172.21.15.12:0/973704161 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f82f80a00f0 con 0x7f82f81cd850 2024-09-11T01:53:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.026+0000 7f82fdacd640 1 -- 172.21.15.12:0/973704161 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82f81cd850 msgr2=0x7f82f81cdc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.026+0000 7f82fdacd640 1 --2- 172.21.15.12:0/973704161 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82f81cd850 0x7f82f81cdc30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f82f80a2630 tx=0x7f82e800fc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.026+0000 7f82feacf640 1 --2- 172.21.15.12:0/973704161 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f82dc03e210 0x7f82dc0406d0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f82f80a72b0 tx=0x7f82f0009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.026+0000 7f82eeffd640 1 -- 172.21.15.12:0/973704161 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e8018030 con 0x7f82f81cd850 2024-09-11T01:53:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.026+0000 7f82eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.026+0000 7f82eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:53:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.026+0000 7f82eeffd640 10 monclient: dump: 2024-09-11T01:53:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:40.045 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:40.046 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:40.046 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:40.046 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:40.046 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:40.046 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:40.046 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:40.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.026+0000 7f82eeffd640 1 -- 172.21.15.12:0/973704161 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f82e800d9b0 con 0x7f82f81cd850 2024-09-11T01:53:40.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.026+0000 7f82eeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:53:40.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.026+0000 7f82eeffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:53:40.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.026+0000 7f82eeffd640 1 -- 172.21.15.12:0/973704161 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82f81cd850 msgr2=0x7f82f81cdc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:40.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.026+0000 7f82eeffd640 1 --2- 172.21.15.12:0/973704161 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82f81cd850 0x7f82f81cdc30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f82f80a2630 tx=0x7f82e800fc00 comp rx=0 tx=0).stop 2024-09-11T01:53:40.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.026+0000 7f82eeffd640 1 -- 172.21.15.12:0/973704161 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82f81cd850 msgr2=0x7f82f81cdc30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:53:40.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.026+0000 7f82eeffd640 1 --2- 172.21.15.12:0/973704161 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82f81cd850 0x7f82f81cdc30 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f82f80a2630 tx=0x7f82e800fc00 comp rx=0 tx=0).stop 2024-09-11T01:53:40.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.032+0000 7f8300532640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:53:40.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.032+0000 7f8300532640 1 --2- 172.21.15.12:0/973704161 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82f802d440 0x7f82f802d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:40.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.032+0000 7f8300532640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:40.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.032+0000 7f8300532640 1 -- 172.21.15.12:0/973704161 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f82f8127900 con 0x7f82f802d440 2024-09-11T01:53:40.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.033+0000 7f82fdacd640 1 --2- 172.21.15.12:0/973704161 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82f802d440 0x7f82f802d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:40.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.033+0000 7f82fdacd640 10 monclient: get_auth_request con 0x7f82f802d440 auth_method 0 2024-09-11T01:53:40.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.033+0000 7f82fdacd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:40.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.033+0000 7f82fdacd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:40.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.033+0000 7f82fdacd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:40.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.033+0000 7f82fdacd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:53:40.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.033+0000 7f82fdacd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:53:40.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.033+0000 7f82fdacd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:53:40.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.033+0000 7f82fdacd640 10 monclient(hunting): handle_auth_done global_id 274132 payload 824 2024-09-11T01:53:40.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.033+0000 7f82fdacd640 1 -- 172.21.15.12:0/973704161 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f82f811dff0 con 0x7f82f802d440 2024-09-11T01:53:40.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.035+0000 7f82fdacd640 1 -- 172.21.15.12:0/973704161 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82f802d440 msgr2=0x7f82f802d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:40.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.035+0000 7f82fdacd640 1 --2- 172.21.15.12:0/973704161 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82f802d440 0x7f82f802d820 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f82f80a2630 tx=0x7f82e8003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:53:40.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.035+0000 7f82eeffd640 1 -- 172.21.15.12:0/973704161 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e801dd20 con 0x7f82f802d440 2024-09-11T01:53:40.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.035+0000 7f82eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:53:40.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.035+0000 7f82eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:53:40.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.035+0000 7f82eeffd640 10 monclient: dump: 2024-09-11T01:53:40.048 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:53:40.048 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:40.048 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:40.048 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:53:40.048 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:53:40.048 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:40.048 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:53:40.049 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:53:40.049 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:53:40.049 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:40.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.036+0000 7f82eeffd640 1 -- 172.21.15.12:0/973704161 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4707 (secure 0 0 0) 0x7f82e801dd20 con 0x7f82f802d440 2024-09-11T01:53:40.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.036+0000 7f82eeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:53:40.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.036+0000 7f82eeffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:53:40.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.036+0000 7f82eeffd640 1 -- 172.21.15.12:0/973704161 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82f802d440 msgr2=0x7f82f802d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:40.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.036+0000 7f82eeffd640 1 --2- 172.21.15.12:0/973704161 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82f802d440 0x7f82f802d820 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f82f80a2630 tx=0x7f82e8003760 comp rx=0 tx=0).stop 2024-09-11T01:53:40.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.036+0000 7f82eeffd640 1 -- 172.21.15.12:0/973704161 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82f802d440 msgr2=0x7f82f802d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:53:40.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.036+0000 7f82eeffd640 1 --2- 172.21.15.12:0/973704161 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82f802d440 0x7f82f802d820 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f82f80a2630 tx=0x7f82e8003760 comp rx=0 tx=0).stop 2024-09-11T01:53:40.049 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:53:40.050 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:53:40.050 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:53:40.050 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:53:40.050 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 226, 2024-09-11T01:53:40.050 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 9853, 2024-09-11T01:53:40.050 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:53:40.050 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:53:40.050 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:53:40.050 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:53:40.050 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:40.050 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 4, 2024-09-11T01:53:40.050 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:53:40.050 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:53:40.051 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:53:40.051 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:53:40.051 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:53:40.051 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:53:40.051 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:53:40.051 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:53:40.051 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:53:40.051 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:53:40.051 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:53:40.051 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:53:40.051 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:53:40.051 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:53:40.051 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:40.051 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:53:40.052 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:53:40.052 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:53:40.052 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:53:40.052 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:53:40.052 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:53:40.052 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:53:40.052 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:53:40.052 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:53:40.052 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:53:40.052 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:53:40.052 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:53:40.052 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:53:40.052 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:40.052 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:40.053 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:53:40.053 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:53:40.053 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:53:40.053 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:53:40.053 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:53:40.053 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:53:40.053 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:53:40.053 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:53:40.053 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:53:40.053 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:53:40.053 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:53:40.053 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:53:40.053 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:53:40.053 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:53:40.054 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:53:40.054 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:53:40.054 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:53:40.054 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:53:40.054 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:53:40.054 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:53:40.054 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:53:40.054 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:53:40.054 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:53:40.054 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:53:40.054 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:53:40.054 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:53:40.054 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:53:40.054 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:53:40.055 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:40.055 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:53:40.055 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:40.055 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:53:40.055 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:40.055 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:53:40.055 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:53:40.055 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:53:40.055 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:53:40.055 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:40.055 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:53:40.055 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:53:40.055 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:40.055 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:40.056 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:40.056 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:53:40.056 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:53:40.056 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:40.056 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:40.056 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:40.056 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:40.056 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:53:40.056 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:53:40.056 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:53:40.056 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:53:40.056 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:53:40.056 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:40.056 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:40.057 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:53:40.057 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:53:40.057 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:53:40.057 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:53:40.057 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:40.057 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:53:40.057 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:53:40.057 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:40.057 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:40.057 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:40.057 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:53:40.057 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:53:40.057 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:40.057 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:40.058 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:40.058 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:40.058 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:53:40.058 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:53:40.058 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:53:40.058 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:53:40.058 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:53:40.058 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:40.058 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:40.058 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:53:40.058 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:53:40.058 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:53:40.058 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:53:40.059 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:40.059 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:53:40.059 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:53:40.059 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:40.059 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:40.059 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:40.059 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:53:40.059 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:53:40.060 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:53:40.060 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:40.060 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:40.060 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:40.060 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:53:40.060 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:53:40.060 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:53:40.060 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:53:40.060 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:53:40.060 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:40.060 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:40.060 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:40.060 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:53:40.061 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:53:40.061 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:40.061 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:53:40.061 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:53:40.061 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:53:40.061 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:40.061 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:40.061 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:53:40.061 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:40.061 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:53:40.061 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:53:40.061 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:53:40.061 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:40.061 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:53:40.062 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:53:40.062 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:53:40.062 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:53:40.062 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:53:40.062 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:53:40.062 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:53:40.062 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:53:40.062 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:53:40.062 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:53:40.062 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:53:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.036+0000 7f8300532640 1 -- 172.21.15.12:0/973704161 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f82dc03e210 msgr2=0x7f82dc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.036+0000 7f8300532640 1 --2- 172.21.15.12:0/973704161 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f82dc03e210 0x7f82dc0406d0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f82f80a72b0 tx=0x7f82f0009000 comp rx=0 tx=0).stop 2024-09-11T01:53:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.036+0000 7f8300532640 10 monclient: shutdown 2024-09-11T01:53:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.036+0000 7f8300532640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:53:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.036+0000 7f8300532640 1 -- 172.21.15.12:0/973704161 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f82f80a2200 msgr2=0x7f82f806f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.036+0000 7f8300532640 1 --2- 172.21.15.12:0/973704161 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f82f80a2200 0x7f82f806f6c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f82f400e9b0 tx=0x7f82f400ee70 comp rx=0 tx=0).stop 2024-09-11T01:53:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.036+0000 7f82feacf640 1 -- 172.21.15.12:0/973704161 reap_dead start 2024-09-11T01:53:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.036+0000 7f8300532640 1 -- 172.21.15.12:0/973704161 shutdown_connections 2024-09-11T01:53:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.036+0000 7f8300532640 1 --2- 172.21.15.12:0/973704161 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f82f80a2200 0x7f82f806f6c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.036+0000 7f8300532640 1 -- 172.21.15.12:0/973704161 >> 172.21.15.12:0/973704161 conn(0x7f82f80a2f70 msgr2=0x7f82f80738b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:53:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.036+0000 7f8300532640 1 -- 172.21.15.12:0/973704161 shutdown_connections 2024-09-11T01:53:40.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:40.036+0000 7f8300532640 1 -- 172.21.15.12:0/973704161 wait complete. 2024-09-11T01:53:40.065 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:53:40.065 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-11T01:53:40.065 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-11T01:53:40.065 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-11T01:53:40.065 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-11T01:53:40.066 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-11T01:53:40.066 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:53:40.079 INFO:tasks.ceph.mon.c:Stopped 2024-09-11T01:53:40.079 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-11T01:53:40.079 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-11T01:53:40.079 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T01:53:55.080 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-11T01:53:55.081 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-11T01:53:55.082 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:53:55.082 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:53:55.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.193+0000 7fae9ad1f640 10 monclient: get_monmap_and_config 2024-09-11T01:53:55.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.193+0000 7fae9ad1f640 10 monclient: build_initial_monmap 2024-09-11T01:53:55.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.193+0000 7fae9ad1f640 10 monclient: monmap: 2024-09-11T01:53:55.195 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:53:55.195 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:53:55.195 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:53:55.194419+0000 2024-09-11T01:53:55.195 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:53:55.194419+0000 2024-09-11T01:53:55.195 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:53:55.196 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:53:55.196 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:53:55.196 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:53:55.196 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:53:55.196 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:53:55.196 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:53:55.196 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:53:55.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.194+0000 7fae9ad1f640 1 Processor -- start 2024-09-11T01:53:55.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.194+0000 7fae9ad1f640 1 -- start start 2024-09-11T01:53:55.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.194+0000 7fae9ad1f640 10 monclient: init 2024-09-11T01:53:55.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.194+0000 7fae9ad1f640 10 monclient: _reopen_session rank -1 2024-09-11T01:53:55.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.194+0000 7fae9ad1f640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-11T01:53:55.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.194+0000 7fae9ad1f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fae940a2db0 0x7fae9406cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:55.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.194+0000 7fae9ad1f640 10 monclient(hunting): picked mon.noname-b con 0x7fae940a2db0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:53:55.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.194+0000 7fae9ad1f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fae9409fbd0 0x7fae9406d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:55.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.194+0000 7fae9ad1f640 10 monclient(hunting): picked mon.noname-c con 0x7fae9409fbd0 addr v2:172.21.15.71:3300/0 2024-09-11T01:53:55.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.194+0000 7fae9ad1f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fae9406d5d0 addr v1:172.21.15.12:6789/0 2024-09-11T01:53:55.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.194+0000 7fae9ad1f640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae9409eea0 con 0x7fae9406d5d0 2024-09-11T01:53:55.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.195+0000 7fae9ad1f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:55.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.195+0000 7fae9ad1f640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fae940a3850 con 0x7fae9409fbd0 2024-09-11T01:53:55.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.195+0000 7fae9ad1f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:55.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.195+0000 7fae9ad1f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fae940a3190 con 0x7fae940a2db0 2024-09-11T01:53:55.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.195+0000 7fae9ad1f640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:55.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.195+0000 7fae9ad1f640 10 monclient(hunting): authenticate will time out at 2241.637024s 2024-09-11T01:53:55.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.195+0000 7fae98abb640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fae940a2db0 msgr2=0x7fae9406cb50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:53:55.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.195+0000 7fae98abb640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fae940a2db0 0x7fae9406cb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:53:55.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.195+0000 7fae992bc640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fae9406d5d0 0x7fae9413e550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:40580/0 (socket says 172.21.15.12:40580) 2024-09-11T01:53:55.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.195+0000 7fae992bc640 1 -- 172.21.15.12:0/1308584592 learned_addr learned my addr 172.21.15.12:0/1308584592 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:53:55.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.195+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.71:3300/0 conn(0x7fae9409fbd0 0x7fae9406d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:55.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.195+0000 7fae8bfff640 10 monclient(hunting): get_auth_request con 0x7fae9409fbd0 auth_method 0 2024-09-11T01:53:55.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.195+0000 7fae8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:55.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.195+0000 7fae8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:55.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.195+0000 7fae8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:55.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.195+0000 7fae8bfff640 1 -- 172.21.15.12:0/1308584592 >> v2:172.21.15.71:3300/0 conn(0x7fae9409fbd0 msgr2=0x7fae9406d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:53:55.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.195+0000 7fae8bfff640 1 -- 172.21.15.12:0/1308584592 >> v2:172.21.15.71:3300/0 conn(0x7fae9409fbd0 msgr2=0x7fae9406d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:53:55.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.195+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.71:3300/0 conn(0x7fae9409fbd0 0x7fae9406d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:53:55.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.195+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.71:3300/0 conn(0x7fae9409fbd0 0x7fae9406d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:53:55.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.197+0000 7fae992bc640 1 -- 172.21.15.12:0/1308584592 >> v1:172.21.15.12:6789/0 conn(0x7fae9406d5d0 legacy=0x7fae9413e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:55.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.396+0000 7fae98abb640 1 -- 172.21.15.12:0/1308584592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fae940a2db0 msgr2=0x7fae9406cb50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:53:55.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.396+0000 7fae98abb640 1 --2- 172.21.15.12:0/1308584592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fae940a2db0 0x7fae9406cb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:53:55.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.397+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.71:3300/0 conn(0x7fae9409fbd0 0x7fae9406d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:55.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.397+0000 7fae8bfff640 10 monclient(hunting): get_auth_request con 0x7fae9409fbd0 auth_method 0 2024-09-11T01:53:55.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.397+0000 7fae8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:55.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.397+0000 7fae8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:55.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.397+0000 7fae8bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:53:55.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.397+0000 7fae8bfff640 1 -- 172.21.15.12:0/1308584592 >> v2:172.21.15.71:3300/0 conn(0x7fae9409fbd0 msgr2=0x7fae9406d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:53:55.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.397+0000 7fae8bfff640 1 -- 172.21.15.12:0/1308584592 >> v2:172.21.15.71:3300/0 conn(0x7fae9409fbd0 msgr2=0x7fae9406d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:53:55.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.397+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.71:3300/0 conn(0x7fae9409fbd0 0x7fae9406d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:53:55.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.397+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.71:3300/0 conn(0x7fae9409fbd0 0x7fae9406d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:53:55.798 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.797+0000 7fae98abb640 1 -- 172.21.15.12:0/1308584592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fae940a2db0 msgr2=0x7fae9406cb50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:53:55.799 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.797+0000 7fae98abb640 1 --2- 172.21.15.12:0/1308584592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fae940a2db0 0x7fae9406cb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:53:55.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.799+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.71:3300/0 conn(0x7fae9409fbd0 0x7fae9406d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:55.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.799+0000 7fae8bfff640 10 monclient(hunting): get_auth_request con 0x7fae9409fbd0 auth_method 0 2024-09-11T01:53:55.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.799+0000 7fae8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:55.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.799+0000 7fae8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:55.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.799+0000 7fae8bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:53:55.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.799+0000 7fae8bfff640 1 -- 172.21.15.12:0/1308584592 >> v2:172.21.15.71:3300/0 conn(0x7fae9409fbd0 msgr2=0x7fae9406d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:53:55.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.799+0000 7fae8bfff640 1 -- 172.21.15.12:0/1308584592 >> v2:172.21.15.71:3300/0 conn(0x7fae9409fbd0 msgr2=0x7fae9406d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:53:55.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.799+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.71:3300/0 conn(0x7fae9409fbd0 0x7fae9406d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:53:55.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:55.799+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.71:3300/0 conn(0x7fae9409fbd0 0x7fae9406d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:53:56.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:56.598+0000 7fae98abb640 1 -- 172.21.15.12:0/1308584592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fae940a2db0 msgr2=0x7fae9406cb50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:53:56.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:56.598+0000 7fae98abb640 1 --2- 172.21.15.12:0/1308584592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fae940a2db0 0x7fae9406cb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:53:56.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:56.600+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.71:3300/0 conn(0x7fae9409fbd0 0x7fae9406d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:56.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:56.600+0000 7fae8bfff640 10 monclient(hunting): get_auth_request con 0x7fae9409fbd0 auth_method 0 2024-09-11T01:53:56.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:56.600+0000 7fae8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:56.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:56.600+0000 7fae8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:56.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:56.600+0000 7fae8bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:53:56.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:56.600+0000 7fae8bfff640 1 -- 172.21.15.12:0/1308584592 >> v2:172.21.15.71:3300/0 conn(0x7fae9409fbd0 msgr2=0x7fae9406d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:53:56.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:56.600+0000 7fae8bfff640 1 -- 172.21.15.12:0/1308584592 >> v2:172.21.15.71:3300/0 conn(0x7fae9409fbd0 msgr2=0x7fae9406d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:53:56.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:56.600+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.71:3300/0 conn(0x7fae9409fbd0 0x7fae9406d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:53:56.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:56.600+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.71:3300/0 conn(0x7fae9409fbd0 0x7fae9406d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:53:58.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.195+0000 7fae8a7fc640 10 monclient(hunting): tick 2024-09-11T01:53:58.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.195+0000 7fae8a7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:53:58.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.195+0000 7fae8a7fc640 1 monclient(hunting): continuing hunt 2024-09-11T01:53:58.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.195+0000 7fae8a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:53:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.195+0000 7fae8a7fc640 1 -- 172.21.15.12:0/1308584592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fae940a2db0 msgr2=0x7fae9406cb50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:53:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.195+0000 7fae8a7fc640 1 --2- 172.21.15.12:0/1308584592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fae940a2db0 0x7fae9406cb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.195+0000 7fae8a7fc640 1 -- 172.21.15.12:0/1308584592 >> v2:172.21.15.71:3300/0 conn(0x7fae9409fbd0 msgr2=0x7fae9406d090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:53:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.195+0000 7fae8a7fc640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.71:3300/0 conn(0x7fae9409fbd0 0x7fae9406d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:53:58.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.195+0000 7fae8a7fc640 1 -- 172.21.15.12:0/1308584592 >> v1:172.21.15.12:6789/0 conn(0x7fae9406d5d0 legacy=0x7fae9413e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:53:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.195+0000 7fae8a7fc640 10 monclient: _add_conns ranks=[1,3] 2024-09-11T01:53:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.195+0000 7fae8a7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fae80003790 addr v1:172.21.15.71:6789/0 2024-09-11T01:53:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.195+0000 7fae8a7fc640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.12:3300/0 conn(0x7fae800074c0 0x7fae80009980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:53:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.195+0000 7fae8a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7fae800074c0 addr v2:172.21.15.12:3300/0 2024-09-11T01:53:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.195+0000 7fae8a7fc640 1 -- 172.21.15.12:0/1308584592 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae9409f180 con 0x7fae80003790 2024-09-11T01:53:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.195+0000 7fae8a7fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:53:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.195+0000 7fae8a7fc640 1 -- 172.21.15.12:0/1308584592 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fae940a3190 con 0x7fae800074c0 2024-09-11T01:53:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.195+0000 7fae8a7fc640 10 monclient(hunting): _renew_subs 2024-09-11T01:53:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.195+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.12:3300/0 conn(0x7fae800074c0 0x7fae80009980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.195+0000 7fae8bfff640 10 monclient(hunting): get_auth_request con 0x7fae800074c0 auth_method 0 2024-09-11T01:53:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.195+0000 7fae8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.195+0000 7fae8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.195+0000 7fae8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:53:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.195+0000 7fae8bfff640 1 -- 172.21.15.12:0/1308584592 >> v2:172.21.15.12:3300/0 conn(0x7fae800074c0 msgr2=0x7fae80009980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:53:58.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.195+0000 7fae8bfff640 1 -- 172.21.15.12:0/1308584592 >> v2:172.21.15.12:3300/0 conn(0x7fae800074c0 msgr2=0x7fae80009980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:53:58.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.195+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.12:3300/0 conn(0x7fae800074c0 0x7fae80009980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:53:58.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.195+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.12:3300/0 conn(0x7fae800074c0 0x7fae80009980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:53:58.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.197+0000 7fae98abb640 1 -- 172.21.15.12:0/1308584592 >> v1:172.21.15.71:6789/0 conn(0x7fae80003790 legacy=0x7fae80005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:53:58.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.396+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.12:3300/0 conn(0x7fae800074c0 0x7fae80009980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:58.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.396+0000 7fae8bfff640 10 monclient(hunting): get_auth_request con 0x7fae800074c0 auth_method 0 2024-09-11T01:53:58.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.396+0000 7fae8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:58.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.396+0000 7fae8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:58.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.396+0000 7fae8bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:53:58.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.396+0000 7fae8bfff640 1 -- 172.21.15.12:0/1308584592 >> v2:172.21.15.12:3300/0 conn(0x7fae800074c0 msgr2=0x7fae80009980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:53:58.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.396+0000 7fae8bfff640 1 -- 172.21.15.12:0/1308584592 >> v2:172.21.15.12:3300/0 conn(0x7fae800074c0 msgr2=0x7fae80009980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:53:58.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.396+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.12:3300/0 conn(0x7fae800074c0 0x7fae80009980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:53:58.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.396+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.12:3300/0 conn(0x7fae800074c0 0x7fae80009980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:53:58.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.798+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.12:3300/0 conn(0x7fae800074c0 0x7fae80009980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:58.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.798+0000 7fae8bfff640 10 monclient(hunting): get_auth_request con 0x7fae800074c0 auth_method 0 2024-09-11T01:53:58.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.798+0000 7fae8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:58.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.798+0000 7fae8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:58.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.798+0000 7fae8bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:53:58.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.798+0000 7fae8bfff640 1 -- 172.21.15.12:0/1308584592 >> v2:172.21.15.12:3300/0 conn(0x7fae800074c0 msgr2=0x7fae80009980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:53:58.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.798+0000 7fae8bfff640 1 -- 172.21.15.12:0/1308584592 >> v2:172.21.15.12:3300/0 conn(0x7fae800074c0 msgr2=0x7fae80009980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:53:58.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.798+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.12:3300/0 conn(0x7fae800074c0 0x7fae80009980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:53:58.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:58.798+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.12:3300/0 conn(0x7fae800074c0 0x7fae80009980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:53:59.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:59.599+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.12:3300/0 conn(0x7fae800074c0 0x7fae80009980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:53:59.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:59.599+0000 7fae8bfff640 10 monclient(hunting): get_auth_request con 0x7fae800074c0 auth_method 0 2024-09-11T01:53:59.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:59.599+0000 7fae8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:53:59.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:59.599+0000 7fae8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:53:59.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:59.599+0000 7fae8bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:53:59.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:59.599+0000 7fae8bfff640 1 -- 172.21.15.12:0/1308584592 >> v2:172.21.15.12:3300/0 conn(0x7fae800074c0 msgr2=0x7fae80009980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:53:59.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:59.599+0000 7fae8bfff640 1 -- 172.21.15.12:0/1308584592 >> v2:172.21.15.12:3300/0 conn(0x7fae800074c0 msgr2=0x7fae80009980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:53:59.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:59.599+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.12:3300/0 conn(0x7fae800074c0 0x7fae80009980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:53:59.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:53:59.599+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.12:3300/0 conn(0x7fae800074c0 0x7fae80009980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:54:00.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.102+0000 7fae8b7fe640 1 -- 172.21.15.12:0/1308584592 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 73136473 0 0) 0x7fae9409f180 con 0x7fae80003790 2024-09-11T01:54:00.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.102+0000 7fae8b7fe640 10 monclient(hunting): _negotiate 2024-09-11T01:54:00.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.102+0000 7fae8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:00.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.102+0000 7fae8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:00.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.102+0000 7fae8b7fe640 10 monclient(hunting): my global_id is 274135 2024-09-11T01:54:00.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.102+0000 7fae8b7fe640 1 -- 172.21.15.12:0/1308584592 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae7c003520 con 0x7fae80003790 2024-09-11T01:54:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.106+0000 7fae8b7fe640 1 -- 172.21.15.12:0/1308584592 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2661038512 0 0) 0x7fae7c003520 con 0x7fae80003790 2024-09-11T01:54:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.106+0000 7fae8b7fe640 1 -- 172.21.15.12:0/1308584592 >> v2:172.21.15.12:3300/0 conn(0x7fae800074c0 msgr2=0x7fae80009980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:54:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.106+0000 7fae8b7fe640 1 --2- 172.21.15.12:0/1308584592 >> v2:172.21.15.12:3300/0 conn(0x7fae800074c0 0x7fae80009980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.106+0000 7fae8b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:54:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.106+0000 7fae8b7fe640 1 monclient: found mon.noname-c-legacy 2024-09-11T01:54:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.106+0000 7fae8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.106+0000 7fae8b7fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-09-11T01:54:00.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.106+0000 7fae8b7fe640 1 -- 172.21.15.12:0/1308584592 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae8000b110 con 0x7fae80003790 2024-09-11T01:54:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.106+0000 7fae8b7fe640 10 monclient: _finish_auth 0 2024-09-11T01:54:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.106+0000 7fae8b7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:54:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.106+0000 7fae8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae8b7fe640 1 -- 172.21.15.12:0/1308584592 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7fae90004600 con 0x7fae80003790 2024-09-11T01:54:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae9ad1f640 5 monclient: authenticate success, global_id 274135 2024-09-11T01:54:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae9ad1f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:54:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae8b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae8b7fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-11T01:54:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae8b7fe640 10 monclient: dump: 2024-09-11T01:54:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.110 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:00.111 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae8b7fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-09-11T01:54:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae8b7fe640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae8b7fe640 1 -- 172.21.15.12:0/1308584592 >> v1:172.21.15.71:6789/0 conn(0x7fae80003790 legacy=0x7fae80005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae8b7fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:54:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae992bc640 1 -- 172.21.15.12:0/1308584592 reap_dead start 2024-09-11T01:54:00.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae8b7fe640 1 --2- 172.21.15.12:0/1308584592 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fae7c0039b0 0x7fae7c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:00.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae8b7fe640 10 monclient(hunting): picked mon.b con 0x7fae7c0039b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:54:00.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae8b7fe640 1 --2- 172.21.15.12:0/1308584592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fae7c004800 0x7fae7c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:00.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae8b7fe640 10 monclient(hunting): picked mon.c con 0x7fae7c004800 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:00.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae8b7fe640 1 --2- 172.21.15.12:0/1308584592 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae7c005290 0x7fae7c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:00.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae8b7fe640 10 monclient(hunting): picked mon.a con 0x7fae7c005290 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:54:00.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae8b7fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:00.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae8b7fe640 1 -- 172.21.15.12:0/1308584592 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fae940a3190 con 0x7fae7c005290 2024-09-11T01:54:00.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae8b7fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:00.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae8b7fe640 1 -- 172.21.15.12:0/1308584592 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fae7c00c1e0 con 0x7fae7c0039b0 2024-09-11T01:54:00.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae8b7fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:00.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae8b7fe640 1 -- 172.21.15.12:0/1308584592 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fae7c00c3e0 con 0x7fae7c004800 2024-09-11T01:54:00.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae8b7fe640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:00.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae8b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:54:00.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae9ad1f640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:54:00.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae8bfff640 1 -- 172.21.15.12:0/1308584592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fae7c004800 msgr2=0x7fae7c004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:54:00.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae8b7fe640 1 -- 172.21.15.12:0/1308584592 DispatchQueue::entry inject delay of 0.002000 2024-09-11T01:54:00.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1308584592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fae7c004800 0x7fae7c004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:54:00.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae98abb640 1 --2- 172.21.15.12:0/1308584592 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fae7c0039b0 0x7fae7c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:00.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae992bc640 1 --2- 172.21.15.12:0/1308584592 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae7c005290 0x7fae7c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:00.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae98abb640 10 monclient(hunting): get_auth_request con 0x7fae7c0039b0 auth_method 0 2024-09-11T01:54:00.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae98abb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:00.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae98abb640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:00.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae98abb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:54:00.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae992bc640 10 monclient(hunting): get_auth_request con 0x7fae7c005290 auth_method 0 2024-09-11T01:54:00.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae992bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:00.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae992bc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:00.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae992bc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:54:00.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae992bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:00.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae992bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:00.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae992bc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:54:00.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae98abb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:00.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae98abb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:00.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae98abb640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:54:00.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae992bc640 10 monclient(hunting): handle_auth_done global_id 274135 payload 293 2024-09-11T01:54:00.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae992bc640 1 -- 172.21.15.12:0/1308584592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fae7c004800 msgr2=0x7fae7c004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:54:00.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae992bc640 1 --2- 172.21.15.12:0/1308584592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fae7c004800 0x7fae7c004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae992bc640 1 -- 172.21.15.12:0/1308584592 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fae7c0039b0 msgr2=0x7fae7c003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:54:00.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae992bc640 1 --2- 172.21.15.12:0/1308584592 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fae7c0039b0 0x7fae7c003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae992bc640 10 monclient: _finish_hunting 0 2024-09-11T01:54:00.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae992bc640 1 monclient: found mon.a 2024-09-11T01:54:00.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae992bc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:00.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae992bc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:54:00.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae992bc640 1 -- 172.21.15.12:0/1308584592 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae7c003520 con 0x7fae7c005290 2024-09-11T01:54:00.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae992bc640 10 monclient: _finish_auth 0 2024-09-11T01:54:00.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae992bc640 10 monclient: _check_auth_tickets 2024-09-11T01:54:00.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae992bc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.107+0000 7fae98abb640 1 --2- 172.21.15.12:0/1308584592 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fae7c0039b0 0x7fae7c003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:54:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.111+0000 7fae992bc640 1 -- 172.21.15.12:0/1308584592 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae7c005290 msgr2=0x7fae7c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.111+0000 7fae992bc640 1 --2- 172.21.15.12:0/1308584592 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae7c005290 0x7fae7c00b9c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fae800034f0 tx=0x7fae7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f9f1005692eca5c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.111+0000 7fae8b7fe640 1 -- 172.21.15.12:0/1308584592 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8c002080 con 0x7fae7c005290 2024-09-11T01:54:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.111+0000 7fae8b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.111+0000 7fae8b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:54:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.111+0000 7fae8b7fe640 10 monclient: dump: 2024-09-11T01:54:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.124 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:00.125 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.111+0000 7fae9ad1f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:54:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.111+0000 7fae8b7fe640 1 -- 172.21.15.12:0/1308584592 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae8c032630 con 0x7fae7c005290 2024-09-11T01:54:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.111+0000 7fae8b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.111+0000 7fae9ad1f640 10 monclient: get_monmap_and_config success 2024-09-11T01:54:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.111+0000 7fae9ad1f640 4 set_mon_vals no callback set 2024-09-11T01:54:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.111+0000 7fae8b7fe640 1 -- 172.21.15.12:0/1308584592 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8c032a20 con 0x7fae7c005290 2024-09-11T01:54:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.111+0000 7fae8b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.111+0000 7fae8b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:54:00.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.111+0000 7fae8b7fe640 10 monclient: dump: 2024-09-11T01:54:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:00.126 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.112+0000 7fae9ad1f640 10 monclient: shutdown 2024-09-11T01:54:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.112+0000 7fae9ad1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:00.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.112+0000 7fae9ad1f640 1 -- 172.21.15.12:0/1308584592 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae7c005290 msgr2=0x7fae7c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.112+0000 7fae9ad1f640 1 --2- 172.21.15.12:0/1308584592 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae7c005290 0x7fae7c00b9c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fae800034f0 tx=0x7fae7c005830 comp rx=0 tx=0).stop 2024-09-11T01:54:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.112+0000 7fae9ad1f640 1 -- 172.21.15.12:0/1308584592 shutdown_connections 2024-09-11T01:54:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.112+0000 7fae9ad1f640 1 --2- 172.21.15.12:0/1308584592 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae7c005290 0x7fae7c00b9c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.112+0000 7fae9ad1f640 1 --2- 172.21.15.12:0/1308584592 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fae7c004800 0x7fae7c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.112+0000 7fae9ad1f640 1 --2- 172.21.15.12:0/1308584592 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fae7c0039b0 0x7fae7c003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.112+0000 7fae9ad1f640 1 -- 172.21.15.12:0/1308584592 >> 172.21.15.12:0/1308584592 conn(0x7fae940a3ea0 msgr2=0x7fae940a42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.112+0000 7fae9ad1f640 1 -- 172.21.15.12:0/1308584592 shutdown_connections 2024-09-11T01:54:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.112+0000 7fae9ad1f640 1 -- 172.21.15.12:0/1308584592 wait complete. 2024-09-11T01:54:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.113+0000 7fae9ad1f640 10 monclient: build_initial_monmap 2024-09-11T01:54:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.113+0000 7fae9ad1f640 10 monclient: monmap: 2024-09-11T01:54:00.127 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:54:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:54:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:54:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:54:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:54:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:54:00.128 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.113+0000 7fae9ad1f640 1 Processor -- start 2024-09-11T01:54:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.113+0000 7fae9ad1f640 1 -- start start 2024-09-11T01:54:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.113+0000 7fae9ad1f640 10 monclient: init 2024-09-11T01:54:00.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae9ad1f640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:00.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae9ad1f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:54:00.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae9ad1f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae7c0039b0 0x7fae940939d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:00.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae9ad1f640 10 monclient(hunting): picked mon.noname-a con 0x7fae7c0039b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:54:00.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae9ad1f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fae7c004800 0x7fae94093f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:00.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae9ad1f640 10 monclient(hunting): picked mon.noname-b con 0x7fae7c004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:54:00.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae9ad1f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fae7c005290 0x7fae94094450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:00.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae9ad1f640 10 monclient(hunting): picked mon.noname-c con 0x7fae7c005290 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:00.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae9ad1f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:00.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae9ad1f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fae9409ea20 con 0x7fae7c0039b0 2024-09-11T01:54:00.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae9ad1f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:00.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae9ad1f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fae9409e8a0 con 0x7fae7c004800 2024-09-11T01:54:00.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae9ad1f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:00.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae9ad1f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fae9406cb50 con 0x7fae7c005290 2024-09-11T01:54:00.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae9ad1f640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:00.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae9ad1f640 10 monclient(hunting): authenticate will time out at 2246.556400s 2024-09-11T01:54:00.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae992bc640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fae7c005290 msgr2=0x7fae94094450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:54:00.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae992bc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fae7c005290 0x7fae94094450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:54:00.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae98abb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae7c0039b0 0x7fae940939d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:00.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae98abb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae7c0039b0 0x7fae940939d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43038/0 (socket says 172.21.15.12:43038) 2024-09-11T01:54:00.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae98abb640 1 -- 172.21.15.12:0/1661499654 learned_addr learned my addr 172.21.15.12:0/1661499654 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:00.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae98abb640 10 monclient(hunting): get_auth_request con 0x7fae7c0039b0 auth_method 0 2024-09-11T01:54:00.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae98abb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:00.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae98abb640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:00.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae98abb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:00.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1661499654 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fae7c004800 0x7fae94093f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:00.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae8bfff640 10 monclient(hunting): get_auth_request con 0x7fae7c004800 auth_method 0 2024-09-11T01:54:00.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:00.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:00.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:00.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae98abb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:00.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae98abb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:00.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae98abb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:00.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:00.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:00.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:00.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae98abb640 10 monclient(hunting): handle_auth_done global_id 264267 payload 824 2024-09-11T01:54:00.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae98abb640 1 -- 172.21.15.12:0/1661499654 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fae7c005290 msgr2=0x7fae94094450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:54:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae98abb640 1 --2- 172.21.15.12:0/1661499654 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fae7c005290 0x7fae94094450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae98abb640 1 -- 172.21.15.12:0/1661499654 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fae7c004800 msgr2=0x7fae94093f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae98abb640 1 --2- 172.21.15.12:0/1661499654 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fae7c004800 0x7fae94093f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae98abb640 10 monclient: _finish_hunting 0 2024-09-11T01:54:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae98abb640 1 monclient: found mon.noname-a 2024-09-11T01:54:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae98abb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae98abb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:54:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae98abb640 1 -- 172.21.15.12:0/1661499654 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae94097700 con 0x7fae7c0039b0 2024-09-11T01:54:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.114+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1661499654 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fae7c004800 0x7fae94093f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:54:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.117+0000 7fae98abb640 1 -- 172.21.15.12:0/1661499654 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae7c0039b0 msgr2=0x7fae940939d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.117+0000 7fae98abb640 1 --2- 172.21.15.12:0/1661499654 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae7c0039b0 0x7fae940939d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fae90000ea0 tx=0x7fae90006710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.117+0000 7fae897fa640 1 -- 172.21.15.12:0/1661499654 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae90017020 con 0x7fae7c0039b0 2024-09-11T01:54:00.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.117+0000 7fae897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.117+0000 7fae897fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:54:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.117+0000 7fae897fa640 10 monclient: dump: 2024-09-11T01:54:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:00.133 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:00.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.117+0000 7fae897fa640 10 monclient: _finish_auth 0 2024-09-11T01:54:00.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.117+0000 7fae897fa640 10 monclient: _check_auth_tickets 2024-09-11T01:54:00.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.117+0000 7fae897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:00.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.117+0000 7fae897fa640 1 -- 172.21.15.12:0/1661499654 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae9000cd10 con 0x7fae7c0039b0 2024-09-11T01:54:00.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.117+0000 7fae9ad1f640 5 monclient: authenticate success, global_id 264267 2024-09-11T01:54:00.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.117+0000 7fae9ad1f640 10 monclient: _renew_subs 2024-09-11T01:54:00.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.117+0000 7fae9ad1f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:54:00.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.117+0000 7fae9ad1f640 1 -- 172.21.15.12:0/1661499654 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae9409e100 con 0x7fae7c0039b0 2024-09-11T01:54:00.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.117+0000 7fae897fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:00.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.117+0000 7fae897fa640 1 -- 172.21.15.12:0/1661499654 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae90013630 con 0x7fae7c0039b0 2024-09-11T01:54:00.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.117+0000 7fae897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:00.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.117+0000 7fae897fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:54:00.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.117+0000 7fae897fa640 10 monclient: dump: 2024-09-11T01:54:00.134 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:00.134 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:00.135 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.135 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.135 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:00.135 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:00.135 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:00.135 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:00.135 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:00.135 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:00.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.117+0000 7fae9ad1f640 10 monclient: _renew_subs 2024-09-11T01:54:00.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.117+0000 7fae9ad1f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:54:00.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.117+0000 7fae9ad1f640 1 -- 172.21.15.12:0/1661499654 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae94097420 con 0x7fae7c0039b0 2024-09-11T01:54:00.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.117+0000 7fae8affd640 4 set_mon_vals no callback set 2024-09-11T01:54:00.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.118+0000 7fae897fa640 1 -- 172.21.15.12:0/1661499654 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fae900137d0 con 0x7fae7c0039b0 2024-09-11T01:54:00.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.118+0000 7fae9ad1f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:00.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.118+0000 7fae9ad1f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:00.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.118+0000 7fae9ad1f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:54:00.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.118+0000 7fae9ad1f640 1 -- 172.21.15.12:0/1661499654 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae9406cee0 con 0x7fae7c0039b0 2024-09-11T01:54:00.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.118+0000 7fae897fa640 1 --2- 172.21.15.12:0/1661499654 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fae8003eb10 0x7fae80040fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:00.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.118+0000 7fae897fa640 1 -- 172.21.15.12:0/1661499654 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(835..835 src has 676..835) ==== 4787+0+0 (secure 0 0 0) 0x7fae90053300 con 0x7fae7c0039b0 2024-09-11T01:54:00.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.119+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1661499654 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fae8003eb10 0x7fae80040fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:00.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.119+0000 7fae8bfff640 10 monclient: get_auth_request con 0x7fae8003eb10 auth_method 0 2024-09-11T01:54:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.121+0000 7fae8bfff640 1 --2- 172.21.15.12:0/1661499654 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fae8003eb10 0x7fae80040fd0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fae94096d50 tx=0x7fae84008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.123+0000 7fae897fa640 1 -- 172.21.15.12:0/1661499654 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fae90021020 con 0x7fae7c0039b0 2024-09-11T01:54:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.123+0000 7fae897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:00.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.123+0000 7fae897fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:54:00.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.384+0000 7fae9ad1f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:54:00.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.384+0000 7fae9ad1f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:54:00.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.384+0000 7fae9ad1f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:54:00.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.384+0000 7fae9ad1f640 1 -- 172.21.15.12:0/1661499654 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fae9406e570 con 0x7fae7c0039b0 2024-09-11T01:54:00.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.384+0000 7fae897fa640 1 -- 172.21.15.12:0/1661499654 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fae90051cf0 con 0x7fae7c0039b0 2024-09-11T01:54:00.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.384+0000 7fae897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:54:00.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.384+0000 7fae897fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:54:00.386 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":228,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:54:00.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.387+0000 7fae9ad1f640 1 -- 172.21.15.12:0/1661499654 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fae8003eb10 msgr2=0x7fae80040fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:00.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.387+0000 7fae9ad1f640 1 --2- 172.21.15.12:0/1661499654 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fae8003eb10 0x7fae80040fd0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fae94096d50 tx=0x7fae84008040 comp rx=0 tx=0).stop 2024-09-11T01:54:00.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.387+0000 7fae9ad1f640 10 monclient: shutdown 2024-09-11T01:54:00.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.387+0000 7fae9ad1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:00.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.387+0000 7fae9ad1f640 1 -- 172.21.15.12:0/1661499654 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae7c0039b0 msgr2=0x7fae940939d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:00.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.387+0000 7fae9ad1f640 1 --2- 172.21.15.12:0/1661499654 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae7c0039b0 0x7fae940939d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fae90000ea0 tx=0x7fae90006710 comp rx=0 tx=0).stop 2024-09-11T01:54:00.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.387+0000 7fae9ad1f640 1 -- 172.21.15.12:0/1661499654 shutdown_connections 2024-09-11T01:54:00.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.387+0000 7fae9ad1f640 1 --2- 172.21.15.12:0/1661499654 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fae7c005290 0x7fae94094450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.387+0000 7fae9ad1f640 1 --2- 172.21.15.12:0/1661499654 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fae7c004800 0x7fae94093f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.387+0000 7fae9ad1f640 1 --2- 172.21.15.12:0/1661499654 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fae8003eb10 0x7fae80040fd0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.387+0000 7fae9ad1f640 1 --2- 172.21.15.12:0/1661499654 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fae7c0039b0 0x7fae940939d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.387+0000 7fae9ad1f640 1 -- 172.21.15.12:0/1661499654 >> 172.21.15.12:0/1661499654 conn(0x7fae940a3ea0 msgr2=0x7fae94070f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:00.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.387+0000 7fae9ad1f640 1 -- 172.21.15.12:0/1661499654 shutdown_connections 2024-09-11T01:54:00.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.387+0000 7fae9ad1f640 1 -- 172.21.15.12:0/1661499654 wait complete. 2024-09-11T01:54:00.400 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:54:00.400 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-11T01:54:00.400 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:54:00.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.501+0000 7fd51a262640 10 monclient: get_monmap_and_config 2024-09-11T01:54:00.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.501+0000 7fd51a262640 10 monclient: build_initial_monmap 2024-09-11T01:54:00.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.501+0000 7fd51a262640 10 monclient: monmap: 2024-09-11T01:54:00.503 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:00.503 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:00.503 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:54:00.502017+0000 2024-09-11T01:54:00.503 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:54:00.502017+0000 2024-09-11T01:54:00.503 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:00.503 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:00.503 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:54:00.503 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:54:00.503 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:54:00.504 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:54:00.504 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:54:00.504 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:00.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.501+0000 7fd51a262640 1 Processor -- start 2024-09-11T01:54:00.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd51a262640 1 -- start start 2024-09-11T01:54:00.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd51a262640 10 monclient: init 2024-09-11T01:54:00.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd51a262640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:00.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd51a262640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-11T01:54:00.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd51a262640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd5140a03a0 0x7fd51409f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:00.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd51a262640 10 monclient(hunting): picked mon.noname-a con 0x7fd5140a03a0 addr v2:172.21.15.12:3300/0 2024-09-11T01:54:00.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd51a262640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd5140a2530 0x7fd5140a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:00.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd51a262640 10 monclient(hunting): picked mon.noname-c con 0x7fd5140a2530 addr v2:172.21.15.71:3300/0 2024-09-11T01:54:00.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd51a262640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd5140a11d0 addr v1:172.21.15.71:6789/0 2024-09-11T01:54:00.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd51a262640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd51409e510 con 0x7fd5140a11d0 2024-09-11T01:54:00.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd51a262640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:00.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd51a262640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fd5140a29b0 con 0x7fd5140a03a0 2024-09-11T01:54:00.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd51a262640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:00.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd51a262640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fd5140a0780 con 0x7fd5140a2530 2024-09-11T01:54:00.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd51a262640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:00.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd51a262640 10 monclient(hunting): authenticate will time out at 2246.944602s 2024-09-11T01:54:00.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd5137fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd5140a03a0 0x7fd51409f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:00.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd5137fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd5140a03a0 0x7fd51409f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43040/0 (socket says 172.21.15.12:43040) 2024-09-11T01:54:00.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd5137fe640 1 -- 172.21.15.12:0/2981024314 learned_addr learned my addr 172.21.15.12:0/2981024314 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:00.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd5137fe640 10 monclient(hunting): get_auth_request con 0x7fd5140a03a0 auth_method 0 2024-09-11T01:54:00.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd5137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:00.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd5137fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:00.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd5137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:00.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd5137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:00.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd5137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:00.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd512ffd640 1 --2- 172.21.15.12:0/2981024314 >> v2:172.21.15.71:3300/0 conn(0x7fd5140a2530 0x7fd5140a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:00.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd5137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:00.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd512ffd640 10 monclient(hunting): get_auth_request con 0x7fd5140a2530 auth_method 0 2024-09-11T01:54:00.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd512ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:00.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd512ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:00.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd512ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:00.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd512ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:00.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.502+0000 7fd512ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:00.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.503+0000 7fd512ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:00.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.503+0000 7fd5137fe640 10 monclient(hunting): handle_auth_done global_id 264270 payload 274 2024-09-11T01:54:00.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.503+0000 7fd5137fe640 1 -- 172.21.15.12:0/2981024314 >> v2:172.21.15.71:3300/0 conn(0x7fd5140a2530 msgr2=0x7fd5140a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:54:00.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.503+0000 7fd5137fe640 1 --2- 172.21.15.12:0/2981024314 >> v2:172.21.15.71:3300/0 conn(0x7fd5140a2530 0x7fd5140a0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.503+0000 7fd5137fe640 1 -- 172.21.15.12:0/2981024314 >> v1:172.21.15.71:6789/0 conn(0x7fd5140a11d0 legacy=0x7fd51413aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:54:00.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.503+0000 7fd5137fe640 10 monclient: _finish_hunting 0 2024-09-11T01:54:00.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.503+0000 7fd5137fe640 1 monclient: found mon.noname-a 2024-09-11T01:54:00.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.503+0000 7fd5137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:00.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.503+0000 7fd5137fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:54:00.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.503+0000 7fd5137fe640 1 -- 172.21.15.12:0/2981024314 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd51409edb0 con 0x7fd5140a03a0 2024-09-11T01:54:00.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.505+0000 7fd5137fe640 1 -- 172.21.15.12:0/2981024314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5140a03a0 msgr2=0x7fd51409f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:00.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.505+0000 7fd5137fe640 1 --2- 172.21.15.12:0/2981024314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5140a03a0 0x7fd51409f1d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd5140a0120 tx=0x7fd50402a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adca09e61f7a4df0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:00.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.506+0000 7fd5127fc640 1 -- 172.21.15.12:0/2981024314 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd504036070 con 0x7fd5140a03a0 2024-09-11T01:54:00.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.506+0000 7fd5127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:00.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.506+0000 7fd5127fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:54:00.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.506+0000 7fd5127fc640 10 monclient: dump: 2024-09-11T01:54:00.517 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:00.517 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:00.517 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.517 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.517 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:00.517 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:00.517 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:00.518 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:00.518 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:00.518 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:00.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.506+0000 7fd5127fc640 10 monclient: _finish_auth 0 2024-09-11T01:54:00.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.506+0000 7fd5127fc640 10 monclient: _check_auth_tickets 2024-09-11T01:54:00.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.506+0000 7fd5127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:00.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.506+0000 7fd5127fc640 1 -- 172.21.15.12:0/2981024314 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd504007800 con 0x7fd5140a03a0 2024-09-11T01:54:00.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.506+0000 7fd5127fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:00.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.506+0000 7fd5127fc640 1 -- 172.21.15.12:0/2981024314 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd504007bd0 con 0x7fd5140a03a0 2024-09-11T01:54:00.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.506+0000 7fd51a262640 5 monclient: authenticate success, global_id 264270 2024-09-11T01:54:00.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.506+0000 7fd51a262640 10 monclient: get_monmap_and_config success 2024-09-11T01:54:00.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.506+0000 7fd51a262640 4 set_mon_vals no callback set 2024-09-11T01:54:00.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.506+0000 7fd5127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:00.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.506+0000 7fd5127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:54:00.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.506+0000 7fd5127fc640 10 monclient: dump: 2024-09-11T01:54:00.519 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:00.519 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:00.519 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.519 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.519 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:00.519 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:00.519 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:00.519 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:00.519 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:00.519 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:00.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.506+0000 7fd51a262640 10 monclient: shutdown 2024-09-11T01:54:00.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.506+0000 7fd51a262640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:00.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.506+0000 7fd51a262640 1 -- 172.21.15.12:0/2981024314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5140a03a0 msgr2=0x7fd51409f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:00.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.506+0000 7fd51a262640 1 --2- 172.21.15.12:0/2981024314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5140a03a0 0x7fd51409f1d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd5140a0120 tx=0x7fd50402a570 comp rx=0 tx=0).stop 2024-09-11T01:54:00.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.506+0000 7fd51a262640 1 -- 172.21.15.12:0/2981024314 shutdown_connections 2024-09-11T01:54:00.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.506+0000 7fd51a262640 1 --2- 172.21.15.12:0/2981024314 >> v2:172.21.15.71:3300/0 conn(0x7fd5140a2530 0x7fd5140a0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.506+0000 7fd51a262640 1 --2- 172.21.15.12:0/2981024314 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5140a03a0 0x7fd51409f1d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.506+0000 7fd51a262640 1 -- 172.21.15.12:0/2981024314 >> 172.21.15.12:0/2981024314 conn(0x7fd5140a2f70 msgr2=0x7fd5140a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:00.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.506+0000 7fd51a262640 1 -- 172.21.15.12:0/2981024314 shutdown_connections 2024-09-11T01:54:00.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.506+0000 7fd51a262640 1 -- 172.21.15.12:0/2981024314 wait complete. 2024-09-11T01:54:00.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.507+0000 7fd51a262640 10 monclient: build_initial_monmap 2024-09-11T01:54:00.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.507+0000 7fd51a262640 10 monclient: monmap: 2024-09-11T01:54:00.520 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:00.520 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:54:00.520 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:54:00.521 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:54:00.521 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:00.521 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:00.521 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:54:00.521 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:54:00.521 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:54:00.521 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:00.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.507+0000 7fd51a262640 1 Processor -- start 2024-09-11T01:54:00.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.507+0000 7fd51a262640 1 -- start start 2024-09-11T01:54:00.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.507+0000 7fd51a262640 10 monclient: init 2024-09-11T01:54:00.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd51a262640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:00.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd51a262640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:54:00.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd51a262640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5140a03a0 0x7fd514128370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:00.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd51a262640 10 monclient(hunting): picked mon.noname-b con 0x7fd5140a03a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:54:00.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd51a262640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5140a11d0 0x7fd5141cd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:00.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd51a262640 10 monclient(hunting): picked mon.noname-c con 0x7fd5140a11d0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:00.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd51a262640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5140a2530 0x7fd514125d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:00.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd51a262640 10 monclient(hunting): picked mon.noname-a con 0x7fd5140a2530 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:54:00.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd51a262640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:00.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd51a262640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd5140a19f0 con 0x7fd5140a2530 2024-09-11T01:54:00.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd51a262640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:00.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd51a262640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd5140a1870 con 0x7fd5140a03a0 2024-09-11T01:54:00.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd51a262640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:00.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd51a262640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd51409ca30 con 0x7fd5140a11d0 2024-09-11T01:54:00.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd51a262640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:00.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd51a262640 10 monclient(hunting): authenticate will time out at 2246.950221s 2024-09-11T01:54:00.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd512ffd640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5140a11d0 msgr2=0x7fd5141cd8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:54:00.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd512ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5140a11d0 0x7fd5141cd8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:54:00.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd513fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5140a2530 0x7fd514125d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:00.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd513fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5140a2530 0x7fd514125d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43056/0 (socket says 172.21.15.12:43056) 2024-09-11T01:54:00.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd513fff640 1 -- 172.21.15.12:0/2808340857 learned_addr learned my addr 172.21.15.12:0/2808340857 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:00.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd513fff640 10 monclient(hunting): get_auth_request con 0x7fd5140a2530 auth_method 0 2024-09-11T01:54:00.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd513fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:00.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd513fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:00.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd513fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:00.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd513fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:00.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd513fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:00.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd513fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:00.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd5137fe640 1 --2- 172.21.15.12:0/2808340857 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5140a03a0 0x7fd514128370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:00.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd5137fe640 10 monclient(hunting): get_auth_request con 0x7fd5140a03a0 auth_method 0 2024-09-11T01:54:00.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd5137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:00.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd5137fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:00.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd5137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:00.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd513fff640 10 monclient(hunting): handle_auth_done global_id 264273 payload 824 2024-09-11T01:54:00.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd513fff640 1 -- 172.21.15.12:0/2808340857 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5140a11d0 msgr2=0x7fd5141cd8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:54:00.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd513fff640 1 --2- 172.21.15.12:0/2808340857 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd5140a11d0 0x7fd5141cd8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd513fff640 1 -- 172.21.15.12:0/2808340857 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5140a03a0 msgr2=0x7fd514128370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:00.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd513fff640 1 --2- 172.21.15.12:0/2808340857 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5140a03a0 0x7fd514128370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd513fff640 10 monclient: _finish_hunting 0 2024-09-11T01:54:00.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd513fff640 1 monclient: found mon.noname-a 2024-09-11T01:54:00.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd513fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:00.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd513fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:54:00.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd513fff640 1 -- 172.21.15.12:0/2808340857 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd514129890 con 0x7fd5140a2530 2024-09-11T01:54:00.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.508+0000 7fd5137fe640 1 --2- 172.21.15.12:0/2808340857 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd5140a03a0 0x7fd514128370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:54:00.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.510+0000 7fd513fff640 1 -- 172.21.15.12:0/2808340857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5140a2530 msgr2=0x7fd514125d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:00.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.510+0000 7fd513fff640 1 --2- 172.21.15.12:0/2808340857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5140a2530 0x7fd514125d50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd50c00c410 tx=0x7fd50c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:00.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.510+0000 7fd50bfff640 1 -- 172.21.15.12:0/2808340857 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50c0119a0 con 0x7fd5140a2530 2024-09-11T01:54:00.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.511+0000 7fd50bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:00.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.511+0000 7fd50bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:54:00.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.511+0000 7fd50bfff640 10 monclient: dump: 2024-09-11T01:54:00.525 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:00.525 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:00.525 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.525 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.525 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:00.526 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:00.526 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:00.526 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:00.526 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:00.526 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:00.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.511+0000 7fd50bfff640 10 monclient: _finish_auth 0 2024-09-11T01:54:00.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.511+0000 7fd50bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:54:00.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.511+0000 7fd50bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:00.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.511+0000 7fd50bfff640 1 -- 172.21.15.12:0/2808340857 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd50c017070 con 0x7fd5140a2530 2024-09-11T01:54:00.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.511+0000 7fd50bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:00.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.511+0000 7fd50bfff640 1 -- 172.21.15.12:0/2808340857 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50c011d80 con 0x7fd5140a2530 2024-09-11T01:54:00.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.511+0000 7fd51a262640 5 monclient: authenticate success, global_id 264273 2024-09-11T01:54:00.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.511+0000 7fd51a262640 10 monclient: _renew_subs 2024-09-11T01:54:00.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.511+0000 7fd51a262640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:54:00.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.511+0000 7fd51a262640 1 -- 172.21.15.12:0/2808340857 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd514129b70 con 0x7fd5140a2530 2024-09-11T01:54:00.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.511+0000 7fd50bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:00.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.511+0000 7fd50bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:54:00.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.511+0000 7fd50bfff640 10 monclient: dump: 2024-09-11T01:54:00.527 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:00.527 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:00.527 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.527 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.527 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:00.527 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:00.527 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:00.527 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:00.527 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:00.528 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:00.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.511+0000 7fd511ffb640 4 set_mon_vals no callback set 2024-09-11T01:54:00.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.511+0000 7fd51a262640 10 monclient: _renew_subs 2024-09-11T01:54:00.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.511+0000 7fd51a262640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:54:00.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.511+0000 7fd51a262640 1 -- 172.21.15.12:0/2808340857 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd51409d480 con 0x7fd5140a2530 2024-09-11T01:54:00.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.511+0000 7fd51a262640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:00.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.512+0000 7fd51a262640 1 --2- 172.21.15.12:0/2808340857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5141cde20 0x7fd51402d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:00.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.512+0000 7fd51a262640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:00.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.512+0000 7fd51a262640 1 -- 172.21.15.12:0/2808340857 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd51409ce40 con 0x7fd5141cde20 2024-09-11T01:54:00.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.512+0000 7fd50bfff640 1 -- 172.21.15.12:0/2808340857 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fd50c01c460 con 0x7fd5140a2530 2024-09-11T01:54:00.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.512+0000 7fd5137fe640 1 --2- 172.21.15.12:0/2808340857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5141cde20 0x7fd51402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:00.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.512+0000 7fd5137fe640 10 monclient: get_auth_request con 0x7fd5141cde20 auth_method 0 2024-09-11T01:54:00.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.512+0000 7fd5137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:00.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.512+0000 7fd5137fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:00.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.512+0000 7fd5137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:00.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.512+0000 7fd5137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:00.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.512+0000 7fd5137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:00.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.512+0000 7fd5137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:00.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.512+0000 7fd50bfff640 1 --2- 172.21.15.12:0/2808340857 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd4f403e1c0 0x7fd4f4040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:00.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.512+0000 7fd5137fe640 10 monclient(hunting): handle_auth_done global_id 264276 payload 824 2024-09-11T01:54:00.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.512+0000 7fd5137fe640 1 -- 172.21.15.12:0/2808340857 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd514128d00 con 0x7fd5141cde20 2024-09-11T01:54:00.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.512+0000 7fd50bfff640 1 -- 172.21.15.12:0/2808340857 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(835..835 src has 676..835) ==== 4787+0+0 (secure 0 0 0) 0x7fd50c052bb0 con 0x7fd5140a2530 2024-09-11T01:54:00.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.512+0000 7fd512ffd640 1 --2- 172.21.15.12:0/2808340857 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd4f403e1c0 0x7fd4f4040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:00.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.512+0000 7fd512ffd640 10 monclient: get_auth_request con 0x7fd4f403e1c0 auth_method 0 2024-09-11T01:54:00.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.514+0000 7fd5137fe640 1 -- 172.21.15.12:0/2808340857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5141cde20 msgr2=0x7fd51402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:00.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.514+0000 7fd5137fe640 1 --2- 172.21.15.12:0/2808340857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5141cde20 0x7fd51402d440 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd504033000 tx=0x7fd504004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:00.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.514+0000 7fd50bfff640 1 -- 172.21.15.12:0/2808340857 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd504040070 con 0x7fd5141cde20 2024-09-11T01:54:00.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.514+0000 7fd50bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:00.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.514+0000 7fd50bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:54:00.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.514+0000 7fd50bfff640 10 monclient: dump: 2024-09-11T01:54:00.530 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:00.530 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:00.530 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.530 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.530 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:00.530 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:00.530 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:00.530 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:00.531 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:00.531 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:00.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.515+0000 7fd50bfff640 1 -- 172.21.15.12:0/2808340857 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd504031810 con 0x7fd5141cde20 2024-09-11T01:54:00.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.515+0000 7fd50bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:00.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.515+0000 7fd50bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:54:00.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.515+0000 7fd50bfff640 1 -- 172.21.15.12:0/2808340857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5141cde20 msgr2=0x7fd51402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:00.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.515+0000 7fd50bfff640 1 --2- 172.21.15.12:0/2808340857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5141cde20 0x7fd51402d440 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd504033000 tx=0x7fd504004a40 comp rx=0 tx=0).stop 2024-09-11T01:54:00.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.515+0000 7fd50bfff640 1 -- 172.21.15.12:0/2808340857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5141cde20 msgr2=0x7fd51402d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:54:00.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.515+0000 7fd50bfff640 1 --2- 172.21.15.12:0/2808340857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5141cde20 0x7fd51402d440 secure :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd504033000 tx=0x7fd504004a40 comp rx=0 tx=0).stop 2024-09-11T01:54:00.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.515+0000 7fd512ffd640 1 --2- 172.21.15.12:0/2808340857 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd4f403e1c0 0x7fd4f4040680 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fd514129aa0 tx=0x7fd4fc009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:00.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.521+0000 7fd51a262640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:54:00.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.521+0000 7fd51a262640 1 --2- 172.21.15.12:0/2808340857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd51402d980 0x7fd51402dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:00.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.521+0000 7fd51a262640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:00.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.521+0000 7fd51a262640 1 -- 172.21.15.12:0/2808340857 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd51411dff0 con 0x7fd51402d980 2024-09-11T01:54:00.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.521+0000 7fd5137fe640 1 --2- 172.21.15.12:0/2808340857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd51402d980 0x7fd51402dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:00.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.521+0000 7fd5137fe640 10 monclient: get_auth_request con 0x7fd51402d980 auth_method 0 2024-09-11T01:54:00.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.521+0000 7fd5137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:00.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.521+0000 7fd5137fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:00.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.521+0000 7fd5137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:00.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.521+0000 7fd5137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:00.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.521+0000 7fd5137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:00.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.521+0000 7fd5137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:00.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.521+0000 7fd5137fe640 10 monclient(hunting): handle_auth_done global_id 264279 payload 824 2024-09-11T01:54:00.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.521+0000 7fd5137fe640 1 -- 172.21.15.12:0/2808340857 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd51413f810 con 0x7fd51402d980 2024-09-11T01:54:00.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.523+0000 7fd5137fe640 1 -- 172.21.15.12:0/2808340857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd51402d980 msgr2=0x7fd51402dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:00.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.523+0000 7fd5137fe640 1 --2- 172.21.15.12:0/2808340857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd51402d980 0x7fd51402dd90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd504000f80 tx=0x7fd50402f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:00.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.524+0000 7fd50bfff640 1 -- 172.21.15.12:0/2808340857 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd504007e60 con 0x7fd51402d980 2024-09-11T01:54:00.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.524+0000 7fd50bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:00.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.524+0000 7fd50bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:54:00.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.524+0000 7fd50bfff640 10 monclient: dump: 2024-09-11T01:54:00.534 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:00.534 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:00.535 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.535 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.535 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:00.535 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:00.535 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:00.535 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:00.535 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:00.535 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:00.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.524+0000 7fd50bfff640 1 -- 172.21.15.12:0/2808340857 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4699 (secure 0 0 0) 0x7fd50404f420 con 0x7fd51402d980 2024-09-11T01:54:00.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.524+0000 7fd50bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:54:00.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.524+0000 7fd50bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:54:00.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.524+0000 7fd50bfff640 1 -- 172.21.15.12:0/2808340857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd51402d980 msgr2=0x7fd51402dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:00.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.524+0000 7fd50bfff640 1 --2- 172.21.15.12:0/2808340857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd51402d980 0x7fd51402dd90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd504000f80 tx=0x7fd50402f000 comp rx=0 tx=0).stop 2024-09-11T01:54:00.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.524+0000 7fd50bfff640 1 -- 172.21.15.12:0/2808340857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd51402d980 msgr2=0x7fd51402dd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:54:00.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.524+0000 7fd50bfff640 1 --2- 172.21.15.12:0/2808340857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd51402d980 0x7fd51402dd90 secure :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd504000f80 tx=0x7fd50402f000 comp rx=0 tx=0).stop 2024-09-11T01:54:00.536 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:54:00.536 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:54:00.536 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:54:00.536 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:54:00.536 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 228, 2024-09-11T01:54:00.536 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 82230, 2024-09-11T01:54:00.536 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:54:00.537 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:54:00.537 INFO:teuthology.orchestra.run.smithi012.stdout: 1 2024-09-11T01:54:00.537 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:00.537 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:54:00.537 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:54:00.537 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:54:00.537 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:54:00.537 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:00.537 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:00.537 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:00.537 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:00.537 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:00.537 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:00.538 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:00.538 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:00.538 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:00.538 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:00.538 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:00.538 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:00.538 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:54:00.538 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:54:00.538 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:00.538 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:00.538 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:00.538 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:00.538 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:00.539 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:00.539 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:00.539 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:00.539 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:00.539 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:00.539 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:00.539 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:00.539 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:00.539 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:54:00.539 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:54:00.539 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:54:00.539 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:54:00.539 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:54:00.540 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:54:00.540 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:54:00.540 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:54:00.540 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:54:00.540 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:54:00.540 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:54:00.540 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:54:00.540 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:54:00.540 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:54:00.540 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:54:00.540 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:54:00.540 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:54:00.540 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:00.541 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:00.541 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:00.541 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:00.541 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:00.541 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:00.541 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:00.541 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:00.541 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:00.541 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:00.541 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:00.541 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:00.541 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:54:00.541 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:00.542 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:54:00.542 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:00.542 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:54:00.542 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:54:00.542 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:00.542 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:00.542 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:00.542 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:00.542 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:54:00.542 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:00.542 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:00.542 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:00.542 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:00.542 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:54:00.543 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:00.543 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:00.543 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:00.543 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:00.543 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:54:00.543 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:54:00.543 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:00.543 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:00.543 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:00.543 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:00.543 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:00.543 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:54:00.543 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:54:00.544 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:00.544 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:00.544 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:00.544 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:00.544 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:54:00.544 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:00.544 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:00.544 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:00.544 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:00.544 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:54:00.544 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:00.544 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:00.544 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:00.545 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:00.545 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:54:00.545 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:54:00.545 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:00.545 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:00.545 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:00.545 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:00.545 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:00.545 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:54:00.545 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:54:00.545 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:00.545 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:00.545 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:00.546 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:00.546 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:54:00.546 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:00.546 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:00.546 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:00.546 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:00.546 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:54:00.546 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:00.546 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:00.546 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:00.546 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:00.546 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:54:00.546 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:54:00.546 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:00.547 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:00.547 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:00.547 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:00.547 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:00.547 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:00.547 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:54:00.547 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:54:00.547 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:00.547 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:00.547 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:00.547 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:54:00.547 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:00.547 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:00.548 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:54:00.548 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:00.548 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:00.548 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:00.548 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:54:00.548 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:00.548 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:00.548 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:54:00.548 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:00.548 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:00.548 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:00.548 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:54:00.548 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:00.548 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:00.549 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:00.549 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:54:00.549 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:54:00.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.524+0000 7fd51a262640 1 -- 172.21.15.12:0/2808340857 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd4f403e1c0 msgr2=0x7fd4f4040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:00.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.524+0000 7fd51a262640 1 --2- 172.21.15.12:0/2808340857 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd4f403e1c0 0x7fd4f4040680 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fd514129aa0 tx=0x7fd4fc009190 comp rx=0 tx=0).stop 2024-09-11T01:54:00.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.524+0000 7fd51a262640 10 monclient: shutdown 2024-09-11T01:54:00.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.524+0000 7fd51a262640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:00.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.524+0000 7fd51a262640 1 -- 172.21.15.12:0/2808340857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5140a2530 msgr2=0x7fd514125d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:00.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.524+0000 7fd51a262640 1 --2- 172.21.15.12:0/2808340857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5140a2530 0x7fd514125d50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd50c00c410 tx=0x7fd50c00c8d0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.524+0000 7fd513fff640 1 -- 172.21.15.12:0/2808340857 reap_dead start 2024-09-11T01:54:00.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.524+0000 7fd513fff640 1 -- 172.21.15.12:0/2808340857 reap_dead start 2024-09-11T01:54:00.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.524+0000 7fd51a262640 1 -- 172.21.15.12:0/2808340857 shutdown_connections 2024-09-11T01:54:00.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.524+0000 7fd51a262640 1 -- 172.21.15.12:0/2808340857 >> 172.21.15.12:0/2808340857 conn(0x7fd5140a2f70 msgr2=0x7fd5140a4bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:00.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.524+0000 7fd51a262640 1 -- 172.21.15.12:0/2808340857 shutdown_connections 2024-09-11T01:54:00.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.524+0000 7fd51a262640 1 -- 172.21.15.12:0/2808340857 wait complete. 2024-09-11T01:54:00.551 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:54:00.551 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T01:54:00.551 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:54:00.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.651+0000 7f6391cef640 10 monclient: get_monmap_and_config 2024-09-11T01:54:00.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.651+0000 7f6391cef640 10 monclient: build_initial_monmap 2024-09-11T01:54:00.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.651+0000 7f6391cef640 10 monclient: monmap: 2024-09-11T01:54:00.653 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:00.653 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:00.654 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:54:00.652636+0000 2024-09-11T01:54:00.654 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:54:00.652636+0000 2024-09-11T01:54:00.654 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:00.654 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:00.654 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:54:00.654 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:54:00.654 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:54:00.654 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:54:00.654 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:54:00.654 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:00.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.652+0000 7f6391cef640 1 Processor -- start 2024-09-11T01:54:00.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.652+0000 7f6391cef640 1 -- start start 2024-09-11T01:54:00.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.652+0000 7f6391cef640 10 monclient: init 2024-09-11T01:54:00.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f6391cef640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:00.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f6391cef640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-11T01:54:00.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f6391cef640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f638c0a1380 0x7f638c0a1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:00.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f6391cef640 10 monclient(hunting): picked mon.noname-b con 0x7f638c0a1380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:00.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f6391cef640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f638c0a0370 0x7f638c135ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:00.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f6391cef640 10 monclient(hunting): picked mon.noname-c con 0x7f638c0a0370 addr v2:172.21.15.71:3300/0 2024-09-11T01:54:00.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f6391cef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f638c0a0c90 addr v1:172.21.15.12:6789/0 2024-09-11T01:54:00.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f6391cef640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f638c0a1ca0 con 0x7f638c0a0c90 2024-09-11T01:54:00.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f6391cef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:00.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f6391cef640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f638c0a21c0 con 0x7f638c0a0370 2024-09-11T01:54:00.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f6391cef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:00.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f6391cef640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f638c0a24a0 con 0x7f638c0a1380 2024-09-11T01:54:00.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f638b7fe640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f638c0a1380 msgr2=0x7f638c0a1760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:54:00.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f6391cef640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:00.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f6391cef640 10 monclient(hunting): authenticate will time out at 2247.095322s 2024-09-11T01:54:00.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f638b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f638c0a1380 0x7f638c0a1760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:54:00.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f638bfff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f638c0a0c90 0x7f638c136020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:52320/0 (socket says 172.21.15.12:52320) 2024-09-11T01:54:00.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f638bfff640 1 -- 172.21.15.12:0/3964294140 learned_addr learned my addr 172.21.15.12:0/3964294140 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:54:00.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f638affd640 1 --2- 172.21.15.12:0/3964294140 >> v2:172.21.15.71:3300/0 conn(0x7f638c0a0370 0x7f638c135ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:00.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f638affd640 10 monclient(hunting): get_auth_request con 0x7f638c0a0370 auth_method 0 2024-09-11T01:54:00.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f638affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:00.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f638affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:00.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f638affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:00.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f638affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:00.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f638affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:00.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f638affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:00.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f638affd640 10 monclient(hunting): handle_auth_done global_id 274147 payload 274 2024-09-11T01:54:00.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f638affd640 1 -- 172.21.15.12:0/3964294140 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f638c0a1380 msgr2=0x7f638c0a1760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:54:00.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f638affd640 1 --2- 172.21.15.12:0/3964294140 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f638c0a1380 0x7f638c0a1760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f638affd640 1 -- 172.21.15.12:0/3964294140 >> v1:172.21.15.12:6789/0 conn(0x7f638c0a0c90 legacy=0x7f638c136020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:54:00.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f638affd640 10 monclient: _finish_hunting 0 2024-09-11T01:54:00.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f638affd640 1 monclient: found mon.noname-c 2024-09-11T01:54:00.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f638affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:00.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f638affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:54:00.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.653+0000 7f638affd640 1 -- 172.21.15.12:0/3964294140 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f638c0a2780 con 0x7f638c0a0370 2024-09-11T01:54:00.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.656+0000 7f638affd640 1 -- 172.21.15.12:0/3964294140 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638c0a0370 msgr2=0x7f638c135ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:00.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.656+0000 7f638affd640 1 --2- 172.21.15.12:0/3964294140 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638c0a0370 0x7f638c135ae0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f63800097a0 tx=0x7f6380029fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6bd55e915b4cf2b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:00.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f638a7fc640 1 -- 172.21.15.12:0/3964294140 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6380037070 con 0x7f638c0a0370 2024-09-11T01:54:00.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f638a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:00.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f638a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:54:00.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f638a7fc640 10 monclient: dump: 2024-09-11T01:54:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:00.667 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f638a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:54:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f638a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:54:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f638a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:00.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f6391cef640 5 monclient: authenticate success, global_id 274147 2024-09-11T01:54:00.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f6391cef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:54:00.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f638a7fc640 1 -- 172.21.15.12:0/3964294140 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6380032640 con 0x7f638c0a0370 2024-09-11T01:54:00.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f638a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:00.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f6391cef640 10 monclient: get_monmap_and_config success 2024-09-11T01:54:00.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f638a7fc640 1 -- 172.21.15.12:0/3964294140 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6380032a10 con 0x7f638c0a0370 2024-09-11T01:54:00.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f6391cef640 4 set_mon_vals no callback set 2024-09-11T01:54:00.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f638a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:00.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f638a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:54:00.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f638a7fc640 10 monclient: dump: 2024-09-11T01:54:00.668 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:00.668 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:00.668 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.669 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.669 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:00.669 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:00.669 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:00.669 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:00.669 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:00.669 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:00.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f6391cef640 10 monclient: shutdown 2024-09-11T01:54:00.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f6391cef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:00.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f6391cef640 1 -- 172.21.15.12:0/3964294140 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638c0a0370 msgr2=0x7f638c135ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:00.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f6391cef640 1 --2- 172.21.15.12:0/3964294140 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638c0a0370 0x7f638c135ae0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f63800097a0 tx=0x7f6380029fe0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f6391cef640 1 -- 172.21.15.12:0/3964294140 shutdown_connections 2024-09-11T01:54:00.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f6391cef640 1 --2- 172.21.15.12:0/3964294140 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638c0a0370 0x7f638c135ae0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f6391cef640 1 --2- 172.21.15.12:0/3964294140 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f638c0a1380 0x7f638c0a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f6391cef640 1 -- 172.21.15.12:0/3964294140 >> 172.21.15.12:0/3964294140 conn(0x7f638c0a2f70 msgr2=0x7f638c0a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:00.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f6391cef640 1 -- 172.21.15.12:0/3964294140 shutdown_connections 2024-09-11T01:54:00.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.657+0000 7f6391cef640 1 -- 172.21.15.12:0/3964294140 wait complete. 2024-09-11T01:54:00.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.658+0000 7f6391cef640 10 monclient: build_initial_monmap 2024-09-11T01:54:00.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.658+0000 7f6391cef640 10 monclient: monmap: 2024-09-11T01:54:00.670 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:00.670 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:54:00.670 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:54:00.670 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:54:00.670 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:00.670 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:00.670 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:54:00.671 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:54:00.671 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:54:00.671 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:00.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.658+0000 7f6391cef640 1 Processor -- start 2024-09-11T01:54:00.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.658+0000 7f6391cef640 1 -- start start 2024-09-11T01:54:00.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.658+0000 7f6391cef640 10 monclient: init 2024-09-11T01:54:00.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f6391cef640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:00.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f6391cef640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:54:00.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f6391cef640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638c0a0370 0x7f638c1ccf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:00.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f6391cef640 10 monclient(hunting): picked mon.noname-b con 0x7f638c0a0370 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:54:00.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f6391cef640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f638c0a0c90 0x7f638c135a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:00.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f6391cef640 10 monclient(hunting): picked mon.noname-c con 0x7f638c0a0c90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:00.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f6391cef640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f638c0a1380 0x7f638c135f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:00.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f6391cef640 10 monclient(hunting): picked mon.noname-a con 0x7f638c0a1380 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:54:00.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f6391cef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:00.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f6391cef640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f638c09ce20 con 0x7f638c0a1380 2024-09-11T01:54:00.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f6391cef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:00.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f6391cef640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f638c09cca0 con 0x7f638c0a0370 2024-09-11T01:54:00.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f6391cef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:00.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f6391cef640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f638c136730 con 0x7f638c0a0c90 2024-09-11T01:54:00.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f6391cef640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:00.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f6391cef640 10 monclient(hunting): authenticate will time out at 2247.101319s 2024-09-11T01:54:00.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638affd640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f638c0a0c90 msgr2=0x7f638c135a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:54:00.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638affd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f638c0a0c90 0x7f638c135a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:54:00.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f638c0a1380 0x7f638c135f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:00.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f638c0a1380 0x7f638c135f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43094/0 (socket says 172.21.15.12:43094) 2024-09-11T01:54:00.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638bfff640 1 -- 172.21.15.12:0/1133185340 learned_addr learned my addr 172.21.15.12:0/1133185340 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:00.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638bfff640 10 monclient(hunting): get_auth_request con 0x7f638c0a1380 auth_method 0 2024-09-11T01:54:00.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:00.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:00.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:00.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638b7fe640 1 --2- 172.21.15.12:0/1133185340 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638c0a0370 0x7f638c1ccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:00.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638b7fe640 10 monclient(hunting): get_auth_request con 0x7f638c0a0370 auth_method 0 2024-09-11T01:54:00.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:00.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:00.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:00.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:00.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:00.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:00.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:00.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:00.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:00.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638bfff640 10 monclient(hunting): handle_auth_done global_id 264282 payload 824 2024-09-11T01:54:00.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638bfff640 1 -- 172.21.15.12:0/1133185340 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f638c0a0c90 msgr2=0x7f638c135a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:54:00.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638bfff640 1 --2- 172.21.15.12:0/1133185340 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f638c0a0c90 0x7f638c135a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638bfff640 1 -- 172.21.15.12:0/1133185340 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638c0a0370 msgr2=0x7f638c1ccf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:00.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638bfff640 1 --2- 172.21.15.12:0/1133185340 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638c0a0370 0x7f638c1ccf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:54:00.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638bfff640 1 monclient: found mon.noname-a 2024-09-11T01:54:00.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:00.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:54:00.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638bfff640 1 -- 172.21.15.12:0/1133185340 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f638c0732a0 con 0x7f638c0a1380 2024-09-11T01:54:00.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.659+0000 7f638b7fe640 1 --2- 172.21.15.12:0/1133185340 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638c0a0370 0x7f638c1ccf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:54:00.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.662+0000 7f638bfff640 1 -- 172.21.15.12:0/1133185340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f638c0a1380 msgr2=0x7f638c135f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:00.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.662+0000 7f638bfff640 1 --2- 172.21.15.12:0/1133185340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f638c0a1380 0x7f638c135f80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f638400cac0 tx=0x7f638400cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:00.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.662+0000 7f6373fff640 1 -- 172.21.15.12:0/1133185340 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638400ad80 con 0x7f638c0a1380 2024-09-11T01:54:00.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.662+0000 7f6373fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:00.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.662+0000 7f6373fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:54:00.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.662+0000 7f6373fff640 10 monclient: dump: 2024-09-11T01:54:00.675 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:00.675 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:00.675 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.675 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.675 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:00.675 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:00.676 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:00.676 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:00.676 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:00.676 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:00.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.662+0000 7f6373fff640 10 monclient: _finish_auth 0 2024-09-11T01:54:00.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.662+0000 7f6373fff640 10 monclient: _check_auth_tickets 2024-09-11T01:54:00.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.662+0000 7f6373fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:00.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.662+0000 7f6373fff640 1 -- 172.21.15.12:0/1133185340 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6384012400 con 0x7f638c0a1380 2024-09-11T01:54:00.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.662+0000 7f6391cef640 5 monclient: authenticate success, global_id 264282 2024-09-11T01:54:00.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.662+0000 7f6373fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:00.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.662+0000 7f6373fff640 1 -- 172.21.15.12:0/1133185340 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63840127f0 con 0x7f638c0a1380 2024-09-11T01:54:00.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.662+0000 7f6373fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:00.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.662+0000 7f6373fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:54:00.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.662+0000 7f6373fff640 10 monclient: dump: 2024-09-11T01:54:00.677 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:00.677 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:00.677 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.677 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.677 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:00.677 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:00.677 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:00.677 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:00.677 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:00.677 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:00.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.662+0000 7f6391cef640 10 monclient: _renew_subs 2024-09-11T01:54:00.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.662+0000 7f6391cef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:54:00.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.662+0000 7f6391cef640 1 -- 172.21.15.12:0/1133185340 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f638c09d480 con 0x7f638c0a1380 2024-09-11T01:54:00.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.662+0000 7f6389ffb640 4 set_mon_vals no callback set 2024-09-11T01:54:00.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.662+0000 7f6391cef640 10 monclient: _renew_subs 2024-09-11T01:54:00.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.662+0000 7f6391cef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:54:00.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.662+0000 7f6391cef640 1 -- 172.21.15.12:0/1133185340 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f638c122130 con 0x7f638c0a1380 2024-09-11T01:54:00.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.663+0000 7f6391cef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:00.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.663+0000 7f6391cef640 1 --2- 172.21.15.12:0/1133185340 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638c121990 0x7f638c09d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:00.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.663+0000 7f6391cef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:00.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.663+0000 7f6391cef640 1 -- 172.21.15.12:0/1133185340 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f638c120cf0 con 0x7f638c121990 2024-09-11T01:54:00.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.663+0000 7f6373fff640 1 -- 172.21.15.12:0/1133185340 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f6384012990 con 0x7f638c0a1380 2024-09-11T01:54:00.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.663+0000 7f638b7fe640 1 --2- 172.21.15.12:0/1133185340 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638c121990 0x7f638c09d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:00.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.663+0000 7f638b7fe640 10 monclient: get_auth_request con 0x7f638c121990 auth_method 0 2024-09-11T01:54:00.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.663+0000 7f638b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:00.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.663+0000 7f638b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:00.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.663+0000 7f638b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:00.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.663+0000 7f638b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:00.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.663+0000 7f638b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:00.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.663+0000 7f638b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:00.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.663+0000 7f6373fff640 1 --2- 172.21.15.12:0/1133185340 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f637803e240 0x7f6378040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:00.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.663+0000 7f6373fff640 1 -- 172.21.15.12:0/1133185340 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(835..835 src has 676..835) ==== 4787+0+0 (secure 0 0 0) 0x7f6384053350 con 0x7f638c0a1380 2024-09-11T01:54:00.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.663+0000 7f638b7fe640 10 monclient(hunting): handle_auth_done global_id 274153 payload 824 2024-09-11T01:54:00.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.663+0000 7f638b7fe640 1 -- 172.21.15.12:0/1133185340 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f638c0a1760 con 0x7f638c121990 2024-09-11T01:54:00.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.663+0000 7f638affd640 1 --2- 172.21.15.12:0/1133185340 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f637803e240 0x7f6378040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:00.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.663+0000 7f638affd640 10 monclient: get_auth_request con 0x7f637803e240 auth_method 0 2024-09-11T01:54:00.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.666+0000 7f638b7fe640 1 -- 172.21.15.12:0/1133185340 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638c121990 msgr2=0x7f638c09d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:00.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.666+0000 7f638b7fe640 1 --2- 172.21.15.12:0/1133185340 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638c121990 0x7f638c09d790 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f637c003fd0 tx=0x7f637c005780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:00.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.666+0000 7f638affd640 1 --2- 172.21.15.12:0/1133185340 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f637803e240 0x7f6378040700 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f638c06f900 tx=0x7f638002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:00.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.668+0000 7f6373fff640 1 -- 172.21.15.12:0/1133185340 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f637c016070 con 0x7f638c121990 2024-09-11T01:54:00.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.668+0000 7f6373fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:00.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.668+0000 7f6373fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:54:00.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.668+0000 7f6373fff640 10 monclient: dump: 2024-09-11T01:54:00.683 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:00.683 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:00.684 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.684 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.684 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:00.684 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:00.684 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:00.684 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:00.684 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:00.684 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:00.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.668+0000 7f6373fff640 1 -- 172.21.15.12:0/1133185340 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f637c00ecd0 con 0x7f638c121990 2024-09-11T01:54:00.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.668+0000 7f6373fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:00.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.668+0000 7f6373fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:54:00.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.668+0000 7f6373fff640 1 -- 172.21.15.12:0/1133185340 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638c121990 msgr2=0x7f638c09d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:00.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.668+0000 7f6373fff640 1 --2- 172.21.15.12:0/1133185340 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638c121990 0x7f638c09d790 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f637c003fd0 tx=0x7f637c005780 comp rx=0 tx=0).stop 2024-09-11T01:54:00.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.668+0000 7f6373fff640 1 -- 172.21.15.12:0/1133185340 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638c121990 msgr2=0x7f638c09d790 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:54:00.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.668+0000 7f6373fff640 1 --2- 172.21.15.12:0/1133185340 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638c121990 0x7f638c09d790 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f637c003fd0 tx=0x7f637c005780 comp rx=0 tx=0).stop 2024-09-11T01:54:00.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.674+0000 7f6391cef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:54:00.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.674+0000 7f6391cef640 1 --2- 172.21.15.12:0/1133185340 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638c09dcd0 0x7f638c09e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:00.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.674+0000 7f6391cef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:00.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.674+0000 7f6391cef640 1 -- 172.21.15.12:0/1133185340 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f638c09e620 con 0x7f638c09dcd0 2024-09-11T01:54:00.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.675+0000 7f638b7fe640 1 --2- 172.21.15.12:0/1133185340 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638c09dcd0 0x7f638c09e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:00.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.675+0000 7f638b7fe640 10 monclient: get_auth_request con 0x7f638c09dcd0 auth_method 0 2024-09-11T01:54:00.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.675+0000 7f638b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:00.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.675+0000 7f638b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:00.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.675+0000 7f638b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:00.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.675+0000 7f638b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:00.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.675+0000 7f638b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:00.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.675+0000 7f638b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:00.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.675+0000 7f638b7fe640 10 monclient(hunting): handle_auth_done global_id 274156 payload 824 2024-09-11T01:54:00.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.675+0000 7f638b7fe640 1 -- 172.21.15.12:0/1133185340 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f638c1d8300 con 0x7f638c09dcd0 2024-09-11T01:54:00.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.678+0000 7f638b7fe640 1 -- 172.21.15.12:0/1133185340 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638c09dcd0 msgr2=0x7f638c09e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:00.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.678+0000 7f638b7fe640 1 --2- 172.21.15.12:0/1133185340 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638c09dcd0 0x7f638c09e0e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f637c018000 tx=0x7f637c0059d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:00.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.678+0000 7f6373fff640 1 -- 172.21.15.12:0/1133185340 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f637c016040 con 0x7f638c09dcd0 2024-09-11T01:54:00.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.678+0000 7f6373fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:00.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.678+0000 7f6373fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:54:00.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.678+0000 7f6373fff640 10 monclient: dump: 2024-09-11T01:54:00.687 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:00.687 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:00.687 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.687 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:00.687 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:00.687 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:00.687 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:00.687 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:00.687 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:00.687 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:00.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.678+0000 7f6373fff640 1 -- 172.21.15.12:0/1133185340 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4864 (secure 0 0 0) 0x7f637c005a40 con 0x7f638c09dcd0 2024-09-11T01:54:00.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.678+0000 7f6373fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:54:00.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.678+0000 7f6373fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:54:00.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.678+0000 7f6373fff640 1 -- 172.21.15.12:0/1133185340 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638c09dcd0 msgr2=0x7f638c09e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:00.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.678+0000 7f6373fff640 1 --2- 172.21.15.12:0/1133185340 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638c09dcd0 0x7f638c09e0e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f637c018000 tx=0x7f637c0059d0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.678+0000 7f6373fff640 1 -- 172.21.15.12:0/1133185340 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638c09dcd0 msgr2=0x7f638c09e0e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:54:00.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.678+0000 7f6373fff640 1 --2- 172.21.15.12:0/1133185340 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638c09dcd0 0x7f638c09e0e0 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f637c018000 tx=0x7f637c0059d0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.688 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:54:00.688 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:54:00.688 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:54:00.688 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:54:00.688 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 228, 2024-09-11T01:54:00.688 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 30496, 2024-09-11T01:54:00.689 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:54:00.689 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:54:00.689 INFO:teuthology.orchestra.run.smithi012.stdout: 1 2024-09-11T01:54:00.689 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:00.689 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:54:00.689 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:54:00.689 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:54:00.689 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:54:00.689 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:00.689 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:00.689 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:00.689 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:00.690 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:00.690 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:00.690 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:00.690 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:00.690 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:00.690 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:00.690 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:00.690 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:00.690 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:54:00.690 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:54:00.690 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:00.690 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:00.690 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:00.690 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:00.691 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:00.691 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:00.691 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:00.691 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:00.691 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:00.691 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:00.691 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:00.691 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:00.691 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:00.691 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:54:00.691 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:54:00.691 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:54:00.691 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:54:00.691 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:54:00.692 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:54:00.692 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:54:00.692 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:54:00.692 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:54:00.692 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:54:00.692 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:54:00.692 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:54:00.692 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:54:00.692 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:54:00.692 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:54:00.692 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:54:00.692 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:54:00.693 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:00.693 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:00.693 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:00.693 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:00.693 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:00.693 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:00.693 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:00.693 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:00.693 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:00.693 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:00.693 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:00.693 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:00.693 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:54:00.693 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:00.694 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:54:00.694 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:00.694 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:54:00.694 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:54:00.694 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:00.694 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:00.694 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:00.694 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:00.694 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:54:00.694 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:00.694 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:00.694 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:00.694 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:00.694 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:54:00.695 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:00.695 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:00.695 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:00.695 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:00.695 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:54:00.695 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:54:00.695 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:00.695 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:00.695 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:00.695 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:00.695 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:00.695 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:54:00.695 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:54:00.696 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:00.696 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:00.696 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:00.696 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:00.696 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:54:00.696 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:00.696 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:00.696 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:00.696 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:00.696 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:54:00.696 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:00.696 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:00.696 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:00.696 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:00.697 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:54:00.697 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:54:00.697 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:00.697 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:00.697 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:00.697 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:00.697 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:00.697 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:54:00.697 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:54:00.697 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:00.697 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:00.697 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:00.697 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:00.698 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:54:00.698 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:00.698 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:00.698 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:00.698 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:00.698 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:54:00.698 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:00.698 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:00.698 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:00.698 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:00.698 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:54:00.698 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:54:00.698 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:00.698 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:00.699 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:00.699 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:00.699 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:00.699 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:00.699 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:54:00.699 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:54:00.699 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:00.699 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:00.699 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:00.699 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:54:00.699 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:00.699 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:00.699 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:54:00.700 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:00.700 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:00.700 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:00.700 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:54:00.700 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:00.700 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:00.700 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:54:00.700 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:00.700 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:00.700 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:00.700 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:54:00.700 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:00.700 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:00.700 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:54:00.701 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:00.701 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:00.701 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:00.701 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:54:00.701 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:00.701 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:00.701 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:00.701 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:54:00.701 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:54:00.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.678+0000 7f6391cef640 1 -- 172.21.15.12:0/1133185340 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f637803e240 msgr2=0x7f6378040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:00.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.678+0000 7f6391cef640 1 --2- 172.21.15.12:0/1133185340 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f637803e240 0x7f6378040700 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f638c06f900 tx=0x7f638002e000 comp rx=0 tx=0).stop 2024-09-11T01:54:00.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.678+0000 7f6391cef640 10 monclient: shutdown 2024-09-11T01:54:00.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.678+0000 7f6391cef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:00.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.678+0000 7f6391cef640 1 -- 172.21.15.12:0/1133185340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f638c0a1380 msgr2=0x7f638c135f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:00.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.678+0000 7f638bfff640 1 -- 172.21.15.12:0/1133185340 reap_dead start 2024-09-11T01:54:00.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.678+0000 7f6391cef640 1 --2- 172.21.15.12:0/1133185340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f638c0a1380 0x7f638c135f80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f638400cac0 tx=0x7f638400cec0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.678+0000 7f6391cef640 1 -- 172.21.15.12:0/1133185340 shutdown_connections 2024-09-11T01:54:00.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.678+0000 7f6391cef640 1 --2- 172.21.15.12:0/1133185340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f638c0a1380 0x7f638c135f80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:00.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.678+0000 7f6391cef640 1 -- 172.21.15.12:0/1133185340 >> 172.21.15.12:0/1133185340 conn(0x7f638c0a2f70 msgr2=0x7f638c070860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:00.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.678+0000 7f6391cef640 1 -- 172.21.15.12:0/1133185340 shutdown_connections 2024-09-11T01:54:00.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:00.678+0000 7f6391cef640 1 -- 172.21.15.12:0/1133185340 wait complete. 2024-09-11T01:54:00.704 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:54:00.704 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:54:01.104 INFO:tasks.rados.rados.0.smithi071.stdout:3635: done (7 left) 2024-09-11T01:54:01.104 INFO:tasks.rados.rados.0.smithi071.stdout:3636: done (6 left) 2024-09-11T01:54:01.105 INFO:tasks.rados.rados.0.smithi071.stdout:3637: done (5 left) 2024-09-11T01:54:01.105 INFO:tasks.rados.rados.0.smithi071.stdout:3639: done (4 left) 2024-09-11T01:54:01.105 INFO:tasks.rados.rados.0.smithi071.stdout:3640: done (3 left) 2024-09-11T01:54:01.105 INFO:tasks.rados.rados.0.smithi071.stdout:3642: done (2 left) 2024-09-11T01:54:01.105 INFO:tasks.rados.rados.0.smithi071.stdout:3643: done (1 left) 2024-09-11T01:54:01.105 INFO:tasks.rados.rados.0.smithi071.stdout:3644: done (0 left) 2024-09-11T01:54:01.105 INFO:tasks.rados.rados.0.smithi071.stdout:3645: snap_remove snap 394 2024-09-11T01:54:02.111 INFO:tasks.rados.rados.0.smithi071.stdout:3645: done (0 left) 2024-09-11T01:54:02.112 INFO:tasks.rados.rados.0.smithi071.stdout:3646: delete oid 33 current snap is 416 2024-09-11T01:54:02.124 INFO:tasks.rados.rados.0.smithi071.stdout:3646: done (0 left) 2024-09-11T01:54:02.124 INFO:tasks.rados.rados.0.smithi071.stdout:3647: read oid 3 snap -1 2024-09-11T01:54:02.124 INFO:tasks.rados.rados.0.smithi071.stdout:3647: expect (ObjNum 1233 snap 407 seq_num 1233) 2024-09-11T01:54:02.124 INFO:tasks.rados.rados.0.smithi071.stdout:3648: write (excl) oid 38 current snap is 416 2024-09-11T01:54:02.124 INFO:tasks.rados.rados.0.smithi071.stdout:3648: seq_num 1258 ranges {743206=661487,2131877=541092} 2024-09-11T01:54:02.131 INFO:tasks.rados.rados.0.smithi071.stdout:3648: writing smithi07134777-38 from 743206 to 1404693 tid 1 2024-09-11T01:54:02.139 INFO:tasks.rados.rados.0.smithi071.stdout:3648: writing smithi07134777-38 from 2131877 to 2672969 tid 2 2024-09-11T01:54:02.139 INFO:tasks.rados.rados.0.smithi071.stdout:3649: rollback oid 30 current snap is 416 2024-09-11T01:54:02.140 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 30 to 361 2024-09-11T01:54:02.140 INFO:tasks.rados.rados.0.smithi071.stdout:3647: expect (ObjNum 1233 snap 407 seq_num 1233) 2024-09-11T01:54:02.140 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:54:02.194 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1306855 2024-09-11T01:54:02.213 INFO:tasks.rados.rados.0.smithi071.stdout:3648: finishing write tid 1 to smithi07134777-38 2024-09-11T01:54:02.213 INFO:tasks.rados.rados.0.smithi071.stdout:3648: finishing write tid 2 to smithi07134777-38 2024-09-11T01:54:02.214 INFO:tasks.rados.rados.0.smithi071.stdout:3648: finishing write tid 3 to smithi07134777-38 2024-09-11T01:54:02.214 INFO:tasks.rados.rados.0.smithi071.stdout:3647: done (2 left) 2024-09-11T01:54:02.214 INFO:tasks.rados.rados.0.smithi071.stdout:3650: write oid 50 current snap is 416 2024-09-11T01:54:02.214 INFO:tasks.rados.rados.0.smithi071.stdout:3648: finishing write tid 4 to smithi07134777-38 2024-09-11T01:54:02.214 INFO:tasks.rados.rados.0.smithi071.stdout:3648: oid 38 updating version 0 to 2822 2024-09-11T01:54:02.214 INFO:tasks.rados.rados.0.smithi071.stdout:3648: oid 38 updating version 2822 to 2823 2024-09-11T01:54:02.214 INFO:tasks.rados.rados.0.smithi071.stdout:3648: oid 38 updating version 2823 to 2824 2024-09-11T01:54:02.214 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 2824 (ObjNum 1258 snap 416 seq_num 1258) dirty exists 2024-09-11T01:54:02.214 INFO:tasks.rados.rados.0.smithi071.stdout:3648: left oid 38 (ObjNum 1258 snap 416 seq_num 1258) 2024-09-11T01:54:02.214 INFO:tasks.rados.rados.0.smithi071.stdout:3649: finishing rollback tid 0 to smithi07134777-30 2024-09-11T01:54:02.214 INFO:tasks.rados.rados.0.smithi071.stdout:3650: seq_num 1259 ranges {760013=736736,2190359=401103} 2024-09-11T01:54:02.221 INFO:tasks.rados.rados.0.smithi071.stdout:3650: writing smithi07134777-50 from 760013 to 1496749 tid 1 2024-09-11T01:54:02.228 INFO:tasks.rados.rados.0.smithi071.stdout:3650: writing smithi07134777-50 from 2190359 to 2591462 tid 2 2024-09-11T01:54:02.228 INFO:tasks.rados.rados.0.smithi071.stdout:3648: done (2 left) 2024-09-11T01:54:02.228 INFO:tasks.rados.rados.0.smithi071.stdout:3651: copy_from oid 14 from oid 40 current snap is 416 2024-09-11T01:54:02.228 INFO:tasks.rados.rados.0.smithi071.stdout:3649: finishing rollback tid 1 to smithi07134777-30 2024-09-11T01:54:02.228 INFO:tasks.rados.rados.0.smithi071.stdout:3649: finishing rollback tid 2 to smithi07134777-30 2024-09-11T01:54:02.228 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 2212 (ObjNum 1003 snap 322 seq_num 1003) dirty exists 2024-09-11T01:54:02.228 INFO:tasks.rados.rados.0.smithi071.stdout:3650: finishing write tid 1 to smithi07134777-50 2024-09-11T01:54:02.228 INFO:tasks.rados.rados.0.smithi071.stdout:3649: done (2 left) 2024-09-11T01:54:02.228 INFO:tasks.rados.rados.0.smithi071.stdout:3652: read oid 18 snap -1 2024-09-11T01:54:02.229 INFO:tasks.rados.rados.0.smithi071.stdout:3652: expect (ObjNum 832 snap 254 seq_num 832) 2024-09-11T01:54:02.229 INFO:tasks.rados.rados.0.smithi071.stdout:3653: snap_remove snap 412 2024-09-11T01:54:03.395 INFO:tasks.rados.rados.0.smithi071.stdout:3650: finishing write tid 2 to smithi07134777-50 2024-09-11T01:54:03.396 INFO:tasks.rados.rados.0.smithi071.stdout:3650: finishing write tid 3 to smithi07134777-50 2024-09-11T01:54:03.396 INFO:tasks.rados.rados.0.smithi071.stdout:3650: finishing write tid 4 to smithi07134777-50 2024-09-11T01:54:03.396 INFO:tasks.rados.rados.0.smithi071.stdout:3650: oid 50 updating version 0 to 2828 2024-09-11T01:54:03.396 INFO:tasks.rados.rados.0.smithi071.stdout:3650: oid 50 version 2828 is already newer than 2827 2024-09-11T01:54:03.396 INFO:tasks.rados.rados.0.smithi071.stdout:3650: oid 50 version 2828 is already newer than 2826 2024-09-11T01:54:03.396 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 2828 (ObjNum 1259 snap 416 seq_num 1259) dirty exists 2024-09-11T01:54:03.396 INFO:tasks.rados.rados.0.smithi071.stdout:3650: left oid 50 (ObjNum 1259 snap 416 seq_num 1259) 2024-09-11T01:54:03.396 INFO:tasks.rados.rados.0.smithi071.stdout:3652: expect (ObjNum 832 snap 254 seq_num 832) 2024-09-11T01:54:03.455 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:54:03.612 INFO:tasks.rados.rados.0.smithi071.stdout:3651: finishing copy_from racing read to smithi07134777-14 2024-09-11T01:54:03.612 INFO:tasks.rados.rados.0.smithi071.stdout:3651: finishing copy_from to smithi07134777-14 2024-09-11T01:54:03.612 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 2578 (ObjNum 1254 snap 415 seq_num 1254) dirty exists 2024-09-11T01:54:03.612 INFO:tasks.rados.rados.0.smithi071.stdout:3650: done (3 left) 2024-09-11T01:54:03.612 INFO:tasks.rados.rados.0.smithi071.stdout:3651: done (2 left) 2024-09-11T01:54:03.612 INFO:tasks.rados.rados.0.smithi071.stdout:3652: done (1 left) 2024-09-11T01:54:03.612 INFO:tasks.rados.rados.0.smithi071.stdout:3653: done (0 left) 2024-09-11T01:54:03.612 INFO:tasks.rados.rados.0.smithi071.stdout:3654: read oid 47 snap 400 2024-09-11T01:54:03.612 INFO:tasks.rados.rados.0.smithi071.stdout:3654: expect (ObjNum 1210 snap 395 seq_num 1210) 2024-09-11T01:54:03.613 INFO:tasks.rados.rados.0.smithi071.stdout:3655: snap_create 2024-09-11T01:54:03.613 INFO:tasks.rados.rados.0.smithi071.stdout:3654: expect (ObjNum 1210 snap 395 seq_num 1210) 2024-09-11T01:54:03.643 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2333336 2024-09-11T01:54:03.647 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1701517 2024-09-11T01:54:04.404 INFO:tasks.rados.rados.0.smithi071.stdout:3654: done (1 left) 2024-09-11T01:54:04.404 INFO:tasks.rados.rados.0.smithi071.stdout:3655: done (0 left) 2024-09-11T01:54:04.404 INFO:tasks.rados.rados.0.smithi071.stdout:3656: read oid 3 snap 385 2024-09-11T01:54:04.405 INFO:tasks.rados.rados.0.smithi071.stdout:3656: expect deleted 2024-09-11T01:54:04.405 INFO:tasks.rados.rados.0.smithi071.stdout:3657: read oid 6 snap -1 2024-09-11T01:54:04.405 INFO:tasks.rados.rados.0.smithi071.stdout:3657: expect (ObjNum 1225 snap 397 seq_num 1225) 2024-09-11T01:54:04.405 INFO:tasks.rados.rados.0.smithi071.stdout:3658: read oid 26 snap 384 2024-09-11T01:54:04.405 INFO:tasks.rados.rados.0.smithi071.stdout:3658: expect deleted 2024-09-11T01:54:04.405 INFO:tasks.rados.rados.0.smithi071.stdout:3659: rollback oid 8 current snap is 417 2024-09-11T01:54:04.405 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 8 to 404 2024-09-11T01:54:04.405 INFO:tasks.rados.rados.0.smithi071.stdout:3660: read oid 22 snap 404 2024-09-11T01:54:04.405 INFO:tasks.rados.rados.0.smithi071.stdout:3660: expect (ObjNum 1218 snap 397 seq_num 1218) 2024-09-11T01:54:04.405 INFO:tasks.rados.rados.0.smithi071.stdout:3661: read oid 19 snap -1 2024-09-11T01:54:04.405 INFO:tasks.rados.rados.0.smithi071.stdout:3661: expect (ObjNum 1252 snap 415 seq_num 1252) 2024-09-11T01:54:04.405 INFO:tasks.rados.rados.0.smithi071.stdout:3662: delete oid 18 current snap is 417 2024-09-11T01:54:04.409 INFO:tasks.rados.rados.0.smithi071.stdout:3660: expect (ObjNum 1218 snap 397 seq_num 1218) 2024-09-11T01:54:04.429 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1567818 2024-09-11T01:54:04.560 INFO:tasks.rados.rados.0.smithi071.stdout:3656: done (6 left) 2024-09-11T01:54:04.561 INFO:tasks.rados.rados.0.smithi071.stdout:3660: done (5 left) 2024-09-11T01:54:04.561 INFO:tasks.rados.rados.0.smithi071.stdout:3662: done (4 left) 2024-09-11T01:54:04.561 INFO:tasks.rados.rados.0.smithi071.stdout:3663: read oid 3 snap 410 2024-09-11T01:54:04.561 INFO:tasks.rados.rados.0.smithi071.stdout:3663: expect (ObjNum 1233 snap 407 seq_num 1233) 2024-09-11T01:54:04.561 INFO:tasks.rados.rados.0.smithi071.stdout:3664: read oid 20 snap -1 2024-09-11T01:54:04.561 INFO:tasks.rados.rados.0.smithi071.stdout:3664: expect (ObjNum 1216 snap 397 seq_num 1216) 2024-09-11T01:54:04.561 INFO:tasks.rados.rados.0.smithi071.stdout:3665: rollback oid 2 current snap is 417 2024-09-11T01:54:04.561 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 2 to 377 2024-09-11T01:54:04.561 INFO:tasks.rados.rados.0.smithi071.stdout:3657: expect (ObjNum 1225 snap 397 seq_num 1225) 2024-09-11T01:54:04.561 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:54:04.628 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 7310 2024-09-11T01:54:04.756 INFO:tasks.rados.rados.0.smithi071.stdout:3659: finishing rollback tid 0 to smithi07134777-8 2024-09-11T01:54:04.756 INFO:tasks.rados.rados.0.smithi071.stdout:3659: finishing rollback tid 1 to smithi07134777-8 2024-09-11T01:54:04.756 INFO:tasks.rados.rados.0.smithi071.stdout:3661: expect (ObjNum 1252 snap 415 seq_num 1252) 2024-09-11T01:54:04.761 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1556966 2024-09-11T01:54:04.793 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:54:04.857 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1973678 2024-09-11T01:54:04.874 INFO:tasks.rados.rados.0.smithi071.stdout:3659: finishing rollback tid 2 to smithi07134777-8 2024-09-11T01:54:04.874 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 2110 (ObjNum 1187 snap 378 seq_num 1187) dirty exists 2024-09-11T01:54:04.874 INFO:tasks.rados.rados.0.smithi071.stdout:3663: expect (ObjNum 1233 snap 407 seq_num 1233) 2024-09-11T01:54:04.890 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:54:04.951 INFO:tasks.rados.rados.0.smithi071.stdout:3664: expect (ObjNum 1216 snap 397 seq_num 1216) 2024-09-11T01:54:05.018 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1385183 2024-09-11T01:54:05.080 INFO:tasks.rados.rados.0.smithi071.stdout:3665: finishing rollback tid 0 to smithi07134777-2 2024-09-11T01:54:05.080 INFO:tasks.rados.rados.0.smithi071.stdout:3665: finishing rollback tid 1 to smithi07134777-2 2024-09-11T01:54:05.080 INFO:tasks.rados.rados.0.smithi071.stdout:3665: finishing rollback tid 2 to smithi07134777-2 2024-09-11T01:54:05.080 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 2547 (ObjNum 1131 snap 357 seq_num 1131) dirty exists 2024-09-11T01:54:05.080 INFO:tasks.rados.rados.0.smithi071.stdout:3657: done (6 left) 2024-09-11T01:54:05.080 INFO:tasks.rados.rados.0.smithi071.stdout:3658: done (5 left) 2024-09-11T01:54:05.080 INFO:tasks.rados.rados.0.smithi071.stdout:3659: done (4 left) 2024-09-11T01:54:05.080 INFO:tasks.rados.rados.0.smithi071.stdout:3661: done (3 left) 2024-09-11T01:54:05.081 INFO:tasks.rados.rados.0.smithi071.stdout:3663: done (2 left) 2024-09-11T01:54:05.081 INFO:tasks.rados.rados.0.smithi071.stdout:3664: done (1 left) 2024-09-11T01:54:05.081 INFO:tasks.rados.rados.0.smithi071.stdout:3665: done (0 left) 2024-09-11T01:54:05.081 INFO:tasks.rados.rados.0.smithi071.stdout:3666: write (excl) oid 27 current snap is 417 2024-09-11T01:54:05.081 INFO:tasks.rados.rados.0.smithi071.stdout:3666: seq_num 1261 ranges {793627=487234,1790970=404772,2829335=364292} 2024-09-11T01:54:05.088 INFO:tasks.rados.rados.0.smithi071.stdout:3666: writing smithi07134777-27 from 793627 to 1280861 tid 1 2024-09-11T01:54:05.091 INFO:tasks.rados.rados.0.smithi071.stdout:3666: writing smithi07134777-27 from 1790970 to 2195742 tid 2 2024-09-11T01:54:05.097 INFO:tasks.rados.rados.0.smithi071.stdout:3666: writing smithi07134777-27 from 2829335 to 3193627 tid 3 2024-09-11T01:54:05.097 INFO:tasks.rados.rados.0.smithi071.stdout:3667: copy_from oid 14 from oid 43 current snap is 417 2024-09-11T01:54:05.097 INFO:tasks.rados.rados.0.smithi071.stdout:3666: finishing write tid 1 to smithi07134777-27 2024-09-11T01:54:05.097 INFO:tasks.rados.rados.0.smithi071.stdout:3666: finishing write tid 2 to smithi07134777-27 2024-09-11T01:54:05.097 INFO:tasks.rados.rados.0.smithi071.stdout:3668: copy_from oid 24 from oid 8 current snap is 417 2024-09-11T01:54:05.098 INFO:tasks.rados.rados.0.smithi071.stdout:3669: delete oid 41 current snap is 417 2024-09-11T01:54:05.103 INFO:tasks.rados.rados.0.smithi071.stdout:3667: finishing copy_from to smithi07134777-14 2024-09-11T01:54:05.103 INFO:tasks.rados.rados.0.smithi071.stdout:3667: got expected ENOENT (src dne) 2024-09-11T01:54:05.103 INFO:tasks.rados.rados.0.smithi071.stdout:3667: finishing copy_from racing read to smithi07134777-14 2024-09-11T01:54:05.103 INFO:tasks.rados.rados.0.smithi071.stdout:3667: done (3 left) 2024-09-11T01:54:05.103 INFO:tasks.rados.rados.0.smithi071.stdout:3669: done (2 left) 2024-09-11T01:54:05.103 INFO:tasks.rados.rados.0.smithi071.stdout:3670: snap_create 2024-09-11T01:54:05.105 INFO:tasks.rados.rados.0.smithi071.stdout:3666: finishing write tid 3 to smithi07134777-27 2024-09-11T01:54:05.105 INFO:tasks.rados.rados.0.smithi071.stdout:3666: finishing write tid 4 to smithi07134777-27 2024-09-11T01:54:05.105 INFO:tasks.rados.rados.0.smithi071.stdout:3666: finishing write tid 5 to smithi07134777-27 2024-09-11T01:54:05.105 INFO:tasks.rados.rados.0.smithi071.stdout:3666: oid 27 updating version 0 to 2696 2024-09-11T01:54:05.105 INFO:tasks.rados.rados.0.smithi071.stdout:3666: oid 27 version 2696 is already newer than 2695 2024-09-11T01:54:05.105 INFO:tasks.rados.rados.0.smithi071.stdout:3666: oid 27 updating version 2696 to 2698 2024-09-11T01:54:05.106 INFO:tasks.rados.rados.0.smithi071.stdout:3666: oid 27 version 2698 is already newer than 2697 2024-09-11T01:54:05.106 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 2698 (ObjNum 1261 snap 417 seq_num 1261) dirty exists 2024-09-11T01:54:05.106 INFO:tasks.rados.rados.0.smithi071.stdout:3666: left oid 27 (ObjNum 1261 snap 417 seq_num 1261) 2024-09-11T01:54:05.111 INFO:tasks.rados.rados.0.smithi071.stdout:3668: finishing copy_from racing read to smithi07134777-24 2024-09-11T01:54:05.112 INFO:tasks.rados.rados.0.smithi071.stdout:3668: finishing copy_from to smithi07134777-24 2024-09-11T01:54:05.113 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 2110 (ObjNum 1187 snap 378 seq_num 1187) dirty exists 2024-09-11T01:54:05.414 INFO:tasks.rados.rados.0.smithi071.stdout:3666: done (2 left) 2024-09-11T01:54:05.414 INFO:tasks.rados.rados.0.smithi071.stdout:3668: done (1 left) 2024-09-11T01:54:05.414 INFO:tasks.rados.rados.0.smithi071.stdout:3670: done (0 left) 2024-09-11T01:54:05.414 INFO:tasks.rados.rados.0.smithi071.stdout:3671: copy_from oid 12 from oid 18 current snap is 418 2024-09-11T01:54:05.414 INFO:tasks.rados.rados.0.smithi071.stdout:3672: rollback oid 28 current snap is 418 2024-09-11T01:54:05.415 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 28 to 392 2024-09-11T01:54:05.415 INFO:tasks.rados.rados.0.smithi071.stdout:3673: write oid 29 current snap is 418 2024-09-11T01:54:05.415 INFO:tasks.rados.rados.0.smithi071.stdout:3673: seq_num 1263 ranges {427241=754085,1624287=408441,2600534=549964} 2024-09-11T01:54:05.421 INFO:tasks.rados.rados.0.smithi071.stdout:3673: writing smithi07134777-29 from 427241 to 1181326 tid 1 2024-09-11T01:54:05.426 INFO:tasks.rados.rados.0.smithi071.stdout:3673: writing smithi07134777-29 from 1624287 to 2032728 tid 2 2024-09-11T01:54:05.432 INFO:tasks.rados.rados.0.smithi071.stdout:3673: writing smithi07134777-29 from 2600534 to 3150498 tid 3 2024-09-11T01:54:05.432 INFO:tasks.rados.rados.0.smithi071.stdout:3674: snap_remove snap 415 2024-09-11T01:54:06.415 INFO:tasks.rados.rados.0.smithi071.stdout:3674: done (3 left) 2024-09-11T01:54:06.415 INFO:tasks.rados.rados.0.smithi071.stdout:3675: copy_from oid 37 from oid 19 current snap is 418 2024-09-11T01:54:06.415 INFO:tasks.rados.rados.0.smithi071.stdout:3671: finishing copy_from to smithi07134777-12 2024-09-11T01:54:06.415 INFO:tasks.rados.rados.0.smithi071.stdout:3671: got expected ENOENT (src dne) 2024-09-11T01:54:06.415 INFO:tasks.rados.rados.0.smithi071.stdout:3671: finishing copy_from racing read to smithi07134777-12 2024-09-11T01:54:06.415 INFO:tasks.rados.rados.0.smithi071.stdout:3672: finishing rollback tid 0 to smithi07134777-28 2024-09-11T01:54:06.415 INFO:tasks.rados.rados.0.smithi071.stdout:3672: finishing rollback tid 1 to smithi07134777-28 2024-09-11T01:54:06.415 INFO:tasks.rados.rados.0.smithi071.stdout:3672: finishing rollback tid 2 to smithi07134777-28 2024-09-11T01:54:06.415 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 2329 (ObjNum 1170 snap 376 seq_num 1170) dirty exists 2024-09-11T01:54:06.415 INFO:tasks.rados.rados.0.smithi071.stdout:3673: finishing write tid 1 to smithi07134777-29 2024-09-11T01:54:06.415 INFO:tasks.rados.rados.0.smithi071.stdout:3673: finishing write tid 2 to smithi07134777-29 2024-09-11T01:54:06.416 INFO:tasks.rados.rados.0.smithi071.stdout:3673: finishing write tid 3 to smithi07134777-29 2024-09-11T01:54:06.416 INFO:tasks.rados.rados.0.smithi071.stdout:3673: finishing write tid 4 to smithi07134777-29 2024-09-11T01:54:06.416 INFO:tasks.rados.rados.0.smithi071.stdout:3673: finishing write tid 5 to smithi07134777-29 2024-09-11T01:54:06.416 INFO:tasks.rados.rados.0.smithi071.stdout:3673: oid 29 updating version 0 to 2840 2024-09-11T01:54:06.416 INFO:tasks.rados.rados.0.smithi071.stdout:3673: oid 29 updating version 2840 to 2842 2024-09-11T01:54:06.416 INFO:tasks.rados.rados.0.smithi071.stdout:3673: oid 29 version 2842 is already newer than 2841 2024-09-11T01:54:06.416 INFO:tasks.rados.rados.0.smithi071.stdout:3673: oid 29 version 2842 is already newer than 2839 2024-09-11T01:54:06.416 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 2842 (ObjNum 1263 snap 418 seq_num 1263) dirty exists 2024-09-11T01:54:06.416 INFO:tasks.rados.rados.0.smithi071.stdout:3673: left oid 29 (ObjNum 1263 snap 418 seq_num 1263) 2024-09-11T01:54:06.416 INFO:tasks.rados.rados.0.smithi071.stdout:3671: done (3 left) 2024-09-11T01:54:06.416 INFO:tasks.rados.rados.0.smithi071.stdout:3672: done (2 left) 2024-09-11T01:54:06.416 INFO:tasks.rados.rados.0.smithi071.stdout:3673: done (1 left) 2024-09-11T01:54:06.416 INFO:tasks.rados.rados.0.smithi071.stdout:3676: read oid 25 snap -1 2024-09-11T01:54:06.416 INFO:tasks.rados.rados.0.smithi071.stdout:3676: expect (ObjNum 1243 snap 411 seq_num 1243) 2024-09-11T01:54:06.417 INFO:tasks.rados.rados.0.smithi071.stdout:3677: rollback oid 21 current snap is 418 2024-09-11T01:54:06.417 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 21 to 400 2024-09-11T01:54:06.417 INFO:tasks.rados.rados.0.smithi071.stdout:3678: snap_remove snap 341 2024-09-11T01:54:07.418 INFO:tasks.rados.rados.0.smithi071.stdout:3678: done (3 left) 2024-09-11T01:54:07.418 INFO:tasks.rados.rados.0.smithi071.stdout:3679: rollback oid 38 current snap is 418 2024-09-11T01:54:07.418 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 38 to 404 2024-09-11T01:54:07.419 INFO:tasks.rados.rados.0.smithi071.stdout:3676: expect (ObjNum 1243 snap 411 seq_num 1243) 2024-09-11T01:54:07.447 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:54:07.496 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1302404 2024-09-11T01:54:07.522 INFO:tasks.rados.rados.0.smithi071.stdout:3677: finishing rollback tid 0 to smithi07134777-21 2024-09-11T01:54:07.522 INFO:tasks.rados.rados.0.smithi071.stdout:3677: finishing rollback tid 1 to smithi07134777-21 2024-09-11T01:54:07.522 INFO:tasks.rados.rados.0.smithi071.stdout:3677: finishing rollback tid 2 to smithi07134777-21 2024-09-11T01:54:07.523 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 2333 (ObjNum 1200 snap 389 seq_num 1200) dirty exists 2024-09-11T01:54:07.523 INFO:tasks.rados.rados.0.smithi071.stdout:3675: finishing copy_from racing read to smithi07134777-37 2024-09-11T01:54:07.523 INFO:tasks.rados.rados.0.smithi071.stdout:3675: finishing copy_from to smithi07134777-37 2024-09-11T01:54:07.523 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 2103 (ObjNum 1252 snap 415 seq_num 1252) dirty exists 2024-09-11T01:54:07.523 INFO:tasks.rados.rados.0.smithi071.stdout:3675: done (3 left) 2024-09-11T01:54:07.523 INFO:tasks.rados.rados.0.smithi071.stdout:3676: done (2 left) 2024-09-11T01:54:07.523 INFO:tasks.rados.rados.0.smithi071.stdout:3677: done (1 left) 2024-09-11T01:54:07.524 INFO:tasks.rados.rados.0.smithi071.stdout:3680: delete oid 7 current snap is 418 2024-09-11T01:54:07.524 INFO:tasks.rados.rados.0.smithi071.stdout:3679: finishing rollback tid 0 to smithi07134777-38 2024-09-11T01:54:07.524 INFO:tasks.rados.rados.0.smithi071.stdout:3679: finishing rollback tid 1 to smithi07134777-38 2024-09-11T01:54:07.524 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 2851 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-11T01:54:07.525 INFO:tasks.rados.rados.0.smithi071.stdout:3679: done (1 left) 2024-09-11T01:54:07.525 INFO:tasks.rados.rados.0.smithi071.stdout:3680: done (0 left) 2024-09-11T01:54:07.525 INFO:tasks.rados.rados.0.smithi071.stdout:3681: read oid 2 snap 416 2024-09-11T01:54:07.525 INFO:tasks.rados.rados.0.smithi071.stdout:3681: expect (ObjNum 1220 snap 397 seq_num 1220) 2024-09-11T01:54:07.525 INFO:tasks.rados.rados.0.smithi071.stdout:3682: read oid 22 snap 392 2024-09-11T01:54:07.525 INFO:tasks.rados.rados.0.smithi071.stdout:3682: expect (ObjNum 1167 snap 374 seq_num 1167) 2024-09-11T01:54:07.525 INFO:tasks.rados.rados.0.smithi071.stdout:3683: snap_create 2024-09-11T01:54:07.529 INFO:tasks.rados.rados.0.smithi071.stdout:3681: expect (ObjNum 1220 snap 397 seq_num 1220) 2024-09-11T01:54:07.668 INFO:tasks.rados.rados.0.smithi071.stdout:3682: expect (ObjNum 1167 snap 374 seq_num 1167) 2024-09-11T01:54:07.748 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2517111 2024-09-11T01:54:08.321 INFO:tasks.rados.rados.0.smithi071.stdout:3681: done (2 left) 2024-09-11T01:54:08.321 INFO:tasks.rados.rados.0.smithi071.stdout:3682: done (1 left) 2024-09-11T01:54:08.321 INFO:tasks.rados.rados.0.smithi071.stdout:3683: done (0 left) 2024-09-11T01:54:08.321 INFO:tasks.rados.rados.0.smithi071.stdout:3684: read oid 38 snap -1 2024-09-11T01:54:08.322 INFO:tasks.rados.rados.0.smithi071.stdout:3684: expect deleted 2024-09-11T01:54:08.322 INFO:tasks.rados.rados.0.smithi071.stdout:3685: snap_remove snap 410 2024-09-11T01:54:09.322 INFO:tasks.rados.rados.0.smithi071.stdout:3685: done (1 left) 2024-09-11T01:54:09.322 INFO:tasks.rados.rados.0.smithi071.stdout:3686: read oid 22 snap -1 2024-09-11T01:54:09.322 INFO:tasks.rados.rados.0.smithi071.stdout:3686: expect (ObjNum 1246 snap 411 seq_num 1246) 2024-09-11T01:54:09.322 INFO:tasks.rados.rados.0.smithi071.stdout:3687: copy_from oid 29 from oid 24 current snap is 419 2024-09-11T01:54:09.322 INFO:tasks.rados.rados.0.smithi071.stdout:3684: done (2 left) 2024-09-11T01:54:09.322 INFO:tasks.rados.rados.0.smithi071.stdout:3688: read oid 1 snap -1 2024-09-11T01:54:09.322 INFO:tasks.rados.rados.0.smithi071.stdout:3688: expect (ObjNum 1154 snap 366 seq_num 1154) 2024-09-11T01:54:09.323 INFO:tasks.rados.rados.0.smithi071.stdout:3689: rollback oid 34 current snap is 419 2024-09-11T01:54:09.323 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 34 to 402 2024-09-11T01:54:09.323 INFO:tasks.rados.rados.0.smithi071.stdout:3690: snap_create 2024-09-11T01:54:09.329 INFO:tasks.rados.rados.0.smithi071.stdout:3688: expect (ObjNum 1154 snap 366 seq_num 1154) 2024-09-11T01:54:09.380 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 539454 2024-09-11T01:54:09.471 INFO:tasks.rados.rados.0.smithi071.stdout:3686: expect (ObjNum 1246 snap 411 seq_num 1246) 2024-09-11T01:54:09.541 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1469381 2024-09-11T01:54:09.649 INFO:tasks.rados.rados.0.smithi071.stdout:3689: finishing rollback tid 0 to smithi07134777-34 2024-09-11T01:54:09.650 INFO:tasks.rados.rados.0.smithi071.stdout:3689: finishing rollback tid 1 to smithi07134777-34 2024-09-11T01:54:09.650 INFO:tasks.rados.rados.0.smithi071.stdout:3689: finishing rollback tid 2 to smithi07134777-34 2024-09-11T01:54:09.650 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 2718 (ObjNum 1121 snap 357 seq_num 1121) dirty exists 2024-09-11T01:54:09.650 INFO:tasks.rados.rados.0.smithi071.stdout:3687: finishing copy_from racing read to smithi07134777-29 2024-09-11T01:54:09.650 INFO:tasks.rados.rados.0.smithi071.stdout:3687: finishing copy_from to smithi07134777-29 2024-09-11T01:54:09.650 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 2110 (ObjNum 1187 snap 378 seq_num 1187) dirty exists 2024-09-11T01:54:10.322 INFO:tasks.rados.rados.0.smithi071.stdout:3686: done (4 left) 2024-09-11T01:54:10.323 INFO:tasks.rados.rados.0.smithi071.stdout:3687: done (3 left) 2024-09-11T01:54:10.323 INFO:tasks.rados.rados.0.smithi071.stdout:3688: done (2 left) 2024-09-11T01:54:10.323 INFO:tasks.rados.rados.0.smithi071.stdout:3689: done (1 left) 2024-09-11T01:54:10.323 INFO:tasks.rados.rados.0.smithi071.stdout:3690: done (0 left) 2024-09-11T01:54:10.323 INFO:tasks.rados.rados.0.smithi071.stdout:3691: copy_from oid 38 from oid 4 current snap is 420 2024-09-11T01:54:10.323 INFO:tasks.rados.rados.0.smithi071.stdout:3692: delete oid 41 current snap is 420 2024-09-11T01:54:10.329 INFO:tasks.rados.rados.0.smithi071.stdout:3692: done (1 left) 2024-09-11T01:54:10.329 INFO:tasks.rados.rados.0.smithi071.stdout:3693: write oid 6 current snap is 420 2024-09-11T01:54:10.329 INFO:tasks.rados.rados.0.smithi071.stdout:3693: seq_num 1266 ranges {477662=579832,1683380=555768,2966450=311212} 2024-09-11T01:54:10.335 INFO:tasks.rados.rados.0.smithi071.stdout:3693: writing smithi07134777-6 from 477662 to 1057494 tid 1 2024-09-11T01:54:10.341 INFO:tasks.rados.rados.0.smithi071.stdout:3693: writing smithi07134777-6 from 1683380 to 2239148 tid 2 2024-09-11T01:54:10.348 INFO:tasks.rados.rados.0.smithi071.stdout:3693: writing smithi07134777-6 from 2966450 to 3277662 tid 3 2024-09-11T01:54:10.348 INFO:tasks.rados.rados.0.smithi071.stdout:3694: rollback oid 50 current snap is 420 2024-09-11T01:54:10.348 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 50 to 391 2024-09-11T01:54:10.348 INFO:tasks.rados.rados.0.smithi071.stdout:3691: finishing copy_from to smithi07134777-38 2024-09-11T01:54:10.348 INFO:tasks.rados.rados.0.smithi071.stdout:3691: got expected ENOENT (src dne) 2024-09-11T01:54:10.348 INFO:tasks.rados.rados.0.smithi071.stdout:3691: finishing copy_from racing read to smithi07134777-38 2024-09-11T01:54:10.348 INFO:tasks.rados.rados.0.smithi071.stdout:3693: finishing write tid 1 to smithi07134777-6 2024-09-11T01:54:10.348 INFO:tasks.rados.rados.0.smithi071.stdout:3693: finishing write tid 2 to smithi07134777-6 2024-09-11T01:54:10.348 INFO:tasks.rados.rados.0.smithi071.stdout:3691: done (2 left) 2024-09-11T01:54:10.348 INFO:tasks.rados.rados.0.smithi071.stdout:3695: rollback oid 7 current snap is 420 2024-09-11T01:54:10.348 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 7 to 418 2024-09-11T01:54:10.348 INFO:tasks.rados.rados.0.smithi071.stdout:3696: write oid 27 current snap is 420 2024-09-11T01:54:10.349 INFO:tasks.rados.rados.0.smithi071.stdout:3696: seq_num 1267 ranges {494469=655081,1741862=415779,2710226=584243} 2024-09-11T01:54:10.353 INFO:tasks.rados.rados.0.smithi071.stdout:3696: writing smithi07134777-27 from 494469 to 1149550 tid 1 2024-09-11T01:54:10.359 INFO:tasks.rados.rados.0.smithi071.stdout:3696: writing smithi07134777-27 from 1741862 to 2157641 tid 2 2024-09-11T01:54:10.367 INFO:tasks.rados.rados.0.smithi071.stdout:3696: writing smithi07134777-27 from 2710226 to 3294469 tid 3 2024-09-11T01:54:10.367 INFO:tasks.rados.rados.0.smithi071.stdout:3697: read oid 12 snap -1 2024-09-11T01:54:10.367 INFO:tasks.rados.rados.0.smithi071.stdout:3697: expect (ObjNum 1250 snap 414 seq_num 1250) 2024-09-11T01:54:10.367 INFO:tasks.rados.rados.0.smithi071.stdout:3698: rollback oid 34 current snap is 420 2024-09-11T01:54:10.367 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 34 to 366 2024-09-11T01:54:10.367 INFO:tasks.rados.rados.0.smithi071.stdout:3694: finishing rollback tid 0 to smithi07134777-50 2024-09-11T01:54:10.367 INFO:tasks.rados.rados.0.smithi071.stdout:3693: finishing write tid 3 to smithi07134777-6 2024-09-11T01:54:10.367 INFO:tasks.rados.rados.0.smithi071.stdout:3693: finishing write tid 4 to smithi07134777-6 2024-09-11T01:54:10.367 INFO:tasks.rados.rados.0.smithi071.stdout:3693: finishing write tid 5 to smithi07134777-6 2024-09-11T01:54:10.367 INFO:tasks.rados.rados.0.smithi071.stdout:3693: oid 6 updating version 0 to 2726 2024-09-11T01:54:10.367 INFO:tasks.rados.rados.0.smithi071.stdout:3693: oid 6 updating version 2726 to 2729 2024-09-11T01:54:10.368 INFO:tasks.rados.rados.0.smithi071.stdout:3693: oid 6 version 2729 is already newer than 2727 2024-09-11T01:54:10.368 INFO:tasks.rados.rados.0.smithi071.stdout:3693: oid 6 version 2729 is already newer than 2728 2024-09-11T01:54:10.368 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 2729 (ObjNum 1266 snap 420 seq_num 1266) dirty exists 2024-09-11T01:54:10.368 INFO:tasks.rados.rados.0.smithi071.stdout:3693: left oid 6 (ObjNum 1266 snap 420 seq_num 1266) 2024-09-11T01:54:10.368 INFO:tasks.rados.rados.0.smithi071.stdout:3694: finishing rollback tid 1 to smithi07134777-50 2024-09-11T01:54:10.368 INFO:tasks.rados.rados.0.smithi071.stdout:3694: finishing rollback tid 2 to smithi07134777-50 2024-09-11T01:54:10.368 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 2877 (ObjNum 1191 snap 379 seq_num 1191) dirty exists 2024-09-11T01:54:10.368 INFO:tasks.rados.rados.0.smithi071.stdout:3695: finishing rollback tid 1 to smithi07134777-7 2024-09-11T01:54:10.368 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 2863 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-11T01:54:10.368 INFO:tasks.rados.rados.0.smithi071.stdout:3696: finishing write tid 1 to smithi07134777-27 2024-09-11T01:54:10.368 INFO:tasks.rados.rados.0.smithi071.stdout:3696: finishing write tid 2 to smithi07134777-27 2024-09-11T01:54:10.368 INFO:tasks.rados.rados.0.smithi071.stdout:3693: done (5 left) 2024-09-11T01:54:10.368 INFO:tasks.rados.rados.0.smithi071.stdout:3694: done (4 left) 2024-09-11T01:54:10.368 INFO:tasks.rados.rados.0.smithi071.stdout:3695: done (3 left) 2024-09-11T01:54:10.369 INFO:tasks.rados.rados.0.smithi071.stdout:3699: copy_from oid 8 from oid 15 current snap is 420 2024-09-11T01:54:10.369 INFO:tasks.rados.rados.0.smithi071.stdout:3700: rollback oid 46 current snap is 420 2024-09-11T01:54:10.369 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 46 to 384 2024-09-11T01:54:10.369 INFO:tasks.rados.rados.0.smithi071.stdout:3701: delete oid 24 current snap is 420 2024-09-11T01:54:10.369 INFO:tasks.rados.rados.0.smithi071.stdout:3699: finishing copy_from to smithi07134777-8 2024-09-11T01:54:10.369 INFO:tasks.rados.rados.0.smithi071.stdout:3699: got expected ENOENT (src dne) 2024-09-11T01:54:10.370 INFO:tasks.rados.rados.0.smithi071.stdout:3699: finishing copy_from racing read to smithi07134777-8 2024-09-11T01:54:10.372 INFO:tasks.rados.rados.0.smithi071.stdout:3700: finishing rollback tid 0 to smithi07134777-46 2024-09-11T01:54:10.372 INFO:tasks.rados.rados.0.smithi071.stdout:3700: finishing rollback tid 1 to smithi07134777-46 2024-09-11T01:54:10.372 INFO:tasks.rados.rados.0.smithi071.stdout:3700: finishing rollback tid 2 to smithi07134777-46 2024-09-11T01:54:10.373 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 2581 (ObjNum 1193 snap 381 seq_num 1193) dirty exists 2024-09-11T01:54:10.373 INFO:tasks.rados.rados.0.smithi071.stdout:3699: done (5 left) 2024-09-11T01:54:10.373 INFO:tasks.rados.rados.0.smithi071.stdout:3700: done (4 left) 2024-09-11T01:54:10.373 INFO:tasks.rados.rados.0.smithi071.stdout:3701: done (3 left) 2024-09-11T01:54:10.373 INFO:tasks.rados.rados.0.smithi071.stdout:3702: write oid 47 current snap is 420 2024-09-11T01:54:10.373 INFO:tasks.rados.rados.0.smithi071.stdout:3702: seq_num 1269 ranges {528083=405579,1342473=419448,2248719=733949} 2024-09-11T01:54:10.375 INFO:tasks.rados.rados.0.smithi071.stdout:3702: writing smithi07134777-47 from 528083 to 933662 tid 1 2024-09-11T01:54:10.380 INFO:tasks.rados.rados.0.smithi071.stdout:3702: writing smithi07134777-47 from 1342473 to 1761921 tid 2 2024-09-11T01:54:10.387 INFO:tasks.rados.rados.0.smithi071.stdout:3702: writing smithi07134777-47 from 2248719 to 2982668 tid 3 2024-09-11T01:54:10.387 INFO:tasks.rados.rados.0.smithi071.stdout:3703: rollback oid 24 current snap is 420 2024-09-11T01:54:10.388 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 24 to 339 2024-09-11T01:54:10.388 INFO:tasks.rados.rados.0.smithi071.stdout:3696: finishing write tid 3 to smithi07134777-27 2024-09-11T01:54:10.388 INFO:tasks.rados.rados.0.smithi071.stdout:3697: expect (ObjNum 1250 snap 414 seq_num 1250) 2024-09-11T01:54:10.413 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2141121 2024-09-11T01:54:10.534 INFO:tasks.rados.rados.0.smithi071.stdout:3696: finishing write tid 4 to smithi07134777-27 2024-09-11T01:54:10.534 INFO:tasks.rados.rados.0.smithi071.stdout:3696: finishing write tid 5 to smithi07134777-27 2024-09-11T01:54:10.534 INFO:tasks.rados.rados.0.smithi071.stdout:3696: oid 27 updating version 0 to 2734 2024-09-11T01:54:10.534 INFO:tasks.rados.rados.0.smithi071.stdout:3696: oid 27 version 2734 is already newer than 2732 2024-09-11T01:54:10.534 INFO:tasks.rados.rados.0.smithi071.stdout:3696: oid 27 version 2734 is already newer than 2731 2024-09-11T01:54:10.535 INFO:tasks.rados.rados.0.smithi071.stdout:3696: oid 27 version 2734 is already newer than 2733 2024-09-11T01:54:10.535 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 2734 (ObjNum 1267 snap 420 seq_num 1267) dirty exists 2024-09-11T01:54:10.535 INFO:tasks.rados.rados.0.smithi071.stdout:3696: left oid 27 (ObjNum 1267 snap 420 seq_num 1267) 2024-09-11T01:54:10.535 INFO:tasks.rados.rados.0.smithi071.stdout:3702: finishing write tid 1 to smithi07134777-47 2024-09-11T01:54:10.535 INFO:tasks.rados.rados.0.smithi071.stdout:3696: done (4 left) 2024-09-11T01:54:10.535 INFO:tasks.rados.rados.0.smithi071.stdout:3697: done (3 left) 2024-09-11T01:54:10.535 INFO:tasks.rados.rados.0.smithi071.stdout:3704: snap_remove snap 405 2024-09-11T01:54:11.332 INFO:tasks.rados.rados.0.smithi071.stdout:3704: done (3 left) 2024-09-11T01:54:11.332 INFO:tasks.rados.rados.0.smithi071.stdout:3705: write (excl) oid 7 current snap is 420 2024-09-11T01:54:11.333 INFO:tasks.rados.rados.0.smithi071.stdout:3698: finishing rollback tid 0 to smithi07134777-34 2024-09-11T01:54:11.333 INFO:tasks.rados.rados.0.smithi071.stdout:3705: seq_num 1270 ranges {544890=597181,1917308=563106,3076142=268748} 2024-09-11T01:54:11.339 INFO:tasks.rados.rados.0.smithi071.stdout:3705: writing smithi07134777-7 from 544890 to 1142071 tid 1 2024-09-11T01:54:11.347 INFO:tasks.rados.rados.0.smithi071.stdout:3705: writing smithi07134777-7 from 1917308 to 2480414 tid 2 2024-09-11T01:54:11.353 INFO:tasks.rados.rados.0.smithi071.stdout:3705: writing smithi07134777-7 from 3076142 to 3344890 tid 3 2024-09-11T01:54:11.353 INFO:tasks.rados.rados.0.smithi071.stdout:3706: read oid 11 snap -1 2024-09-11T01:54:11.353 INFO:tasks.rados.rados.0.smithi071.stdout:3706: expect (ObjNum 1115 snap 356 seq_num 1115) 2024-09-11T01:54:11.353 INFO:tasks.rados.rados.0.smithi071.stdout:3707: copy_from oid 25 from oid 36 current snap is 420 2024-09-11T01:54:11.353 INFO:tasks.rados.rados.0.smithi071.stdout:3698: finishing rollback tid 1 to smithi07134777-34 2024-09-11T01:54:11.353 INFO:tasks.rados.rados.0.smithi071.stdout:3698: finishing rollback tid 2 to smithi07134777-34 2024-09-11T01:54:11.353 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 2738 (ObjNum 1121 snap 357 seq_num 1121) dirty exists 2024-09-11T01:54:11.353 INFO:tasks.rados.rados.0.smithi071.stdout:3702: finishing write tid 2 to smithi07134777-47 2024-09-11T01:54:11.353 INFO:tasks.rados.rados.0.smithi071.stdout:3703: finishing rollback tid 1 to smithi07134777-24 2024-09-11T01:54:11.353 INFO:tasks.rados.rados.0.smithi071.stdout:3703: finishing rollback tid 2 to smithi07134777-24 2024-09-11T01:54:11.353 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 2353 (ObjNum 1024 snap 328 seq_num 1024) dirty exists 2024-09-11T01:54:11.353 INFO:tasks.rados.rados.0.smithi071.stdout:3702: finishing write tid 3 to smithi07134777-47 2024-09-11T01:54:11.354 INFO:tasks.rados.rados.0.smithi071.stdout:3702: finishing write tid 4 to smithi07134777-47 2024-09-11T01:54:11.354 INFO:tasks.rados.rados.0.smithi071.stdout:3702: finishing write tid 5 to smithi07134777-47 2024-09-11T01:54:11.354 INFO:tasks.rados.rados.0.smithi071.stdout:3702: oid 47 updating version 0 to 2572 2024-09-11T01:54:11.354 INFO:tasks.rados.rados.0.smithi071.stdout:3702: oid 47 version 2572 is already newer than 2571 2024-09-11T01:54:11.354 INFO:tasks.rados.rados.0.smithi071.stdout:3702: oid 47 version 2572 is already newer than 2569 2024-09-11T01:54:11.354 INFO:tasks.rados.rados.0.smithi071.stdout:3702: oid 47 version 2572 is already newer than 2570 2024-09-11T01:54:11.354 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 2572 (ObjNum 1269 snap 420 seq_num 1269) dirty exists 2024-09-11T01:54:11.354 INFO:tasks.rados.rados.0.smithi071.stdout:3702: left oid 47 (ObjNum 1269 snap 420 seq_num 1269) 2024-09-11T01:54:11.354 INFO:tasks.rados.rados.0.smithi071.stdout:3705: finishing write tid 1 to smithi07134777-7 2024-09-11T01:54:11.354 INFO:tasks.rados.rados.0.smithi071.stdout:3705: finishing write tid 2 to smithi07134777-7 2024-09-11T01:54:11.354 INFO:tasks.rados.rados.0.smithi071.stdout:3698: done (5 left) 2024-09-11T01:54:11.354 INFO:tasks.rados.rados.0.smithi071.stdout:3702: done (4 left) 2024-09-11T01:54:11.354 INFO:tasks.rados.rados.0.smithi071.stdout:3703: done (3 left) 2024-09-11T01:54:11.354 INFO:tasks.rados.rados.0.smithi071.stdout:3708: snap_remove snap 361 2024-09-11T01:54:12.335 INFO:tasks.rados.rados.0.smithi071.stdout:3708: done (3 left) 2024-09-11T01:54:12.335 INFO:tasks.rados.rados.0.smithi071.stdout:3709: read oid 33 snap -1 2024-09-11T01:54:12.335 INFO:tasks.rados.rados.0.smithi071.stdout:3709: expect deleted 2024-09-11T01:54:12.335 INFO:tasks.rados.rados.0.smithi071.stdout:3710: read oid 8 snap -1 2024-09-11T01:54:12.335 INFO:tasks.rados.rados.0.smithi071.stdout:3710: expect (ObjNum 1187 snap 378 seq_num 1187) 2024-09-11T01:54:12.336 INFO:tasks.rados.rados.0.smithi071.stdout:3711: write oid 46 current snap is 420 2024-09-11T01:54:12.336 INFO:tasks.rados.rados.0.smithi071.stdout:3711: seq_num 1271 ranges {561697=672430,1975790=706764,3103565=258132} 2024-09-11T01:54:12.343 INFO:tasks.rados.rados.0.smithi071.stdout:3711: writing smithi07134777-46 from 561697 to 1234127 tid 1 2024-09-11T01:54:12.352 INFO:tasks.rados.rados.0.smithi071.stdout:3711: writing smithi07134777-46 from 1975790 to 2682554 tid 2 2024-09-11T01:54:12.356 INFO:tasks.rados.rados.0.smithi071.stdout:3711: writing smithi07134777-46 from 3103565 to 3361697 tid 3 2024-09-11T01:54:12.356 INFO:tasks.rados.rados.0.smithi071.stdout:3712: snap_create 2024-09-11T01:54:12.356 INFO:tasks.rados.rados.0.smithi071.stdout:3705: finishing write tid 3 to smithi07134777-7 2024-09-11T01:54:12.356 INFO:tasks.rados.rados.0.smithi071.stdout:3706: expect (ObjNum 1115 snap 356 seq_num 1115) 2024-09-11T01:54:12.365 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1148582 2024-09-11T01:54:12.464 INFO:tasks.rados.rados.0.smithi071.stdout:3705: finishing write tid 4 to smithi07134777-7 2024-09-11T01:54:12.464 INFO:tasks.rados.rados.0.smithi071.stdout:3705: finishing write tid 5 to smithi07134777-7 2024-09-11T01:54:12.464 INFO:tasks.rados.rados.0.smithi071.stdout:3705: oid 7 updating version 0 to 2887 2024-09-11T01:54:12.464 INFO:tasks.rados.rados.0.smithi071.stdout:3705: oid 7 updating version 2887 to 2888 2024-09-11T01:54:12.464 INFO:tasks.rados.rados.0.smithi071.stdout:3705: oid 7 version 2888 is already newer than 2884 2024-09-11T01:54:12.465 INFO:tasks.rados.rados.0.smithi071.stdout:3705: oid 7 version 2888 is already newer than 2879 2024-09-11T01:54:12.465 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 2888 (ObjNum 1270 snap 420 seq_num 1270) dirty exists 2024-09-11T01:54:12.465 INFO:tasks.rados.rados.0.smithi071.stdout:3705: left oid 7 (ObjNum 1270 snap 420 seq_num 1270) 2024-09-11T01:54:12.465 INFO:tasks.rados.rados.0.smithi071.stdout:3707: finishing copy_from racing read to smithi07134777-25 2024-09-11T01:54:12.465 INFO:tasks.rados.rados.0.smithi071.stdout:3707: finishing copy_from to smithi07134777-25 2024-09-11T01:54:12.465 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 2107 (ObjNum 1256 snap 415 seq_num 1256) dirty exists 2024-09-11T01:54:12.465 INFO:tasks.rados.rados.0.smithi071.stdout:3710: expect (ObjNum 1187 snap 378 seq_num 1187) 2024-09-11T01:54:12.577 INFO:tasks.rados.rados.0.smithi071.stdout:3711: finishing write tid 1 to smithi07134777-46 2024-09-11T01:54:12.577 INFO:tasks.rados.rados.0.smithi071.stdout:3711: finishing write tid 2 to smithi07134777-46 2024-09-11T01:54:12.577 INFO:tasks.rados.rados.0.smithi071.stdout:3711: finishing write tid 3 to smithi07134777-46 2024-09-11T01:54:12.577 INFO:tasks.rados.rados.0.smithi071.stdout:3711: finishing write tid 4 to smithi07134777-46 2024-09-11T01:54:12.577 INFO:tasks.rados.rados.0.smithi071.stdout:3711: finishing write tid 5 to smithi07134777-46 2024-09-11T01:54:12.577 INFO:tasks.rados.rados.0.smithi071.stdout:3711: oid 46 updating version 0 to 2584 2024-09-11T01:54:12.577 INFO:tasks.rados.rados.0.smithi071.stdout:3711: oid 46 version 2584 is already newer than 2582 2024-09-11T01:54:12.577 INFO:tasks.rados.rados.0.smithi071.stdout:3711: oid 46 updating version 2584 to 2585 2024-09-11T01:54:12.577 INFO:tasks.rados.rados.0.smithi071.stdout:3711: oid 46 version 2585 is already newer than 2583 2024-09-11T01:54:12.578 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 2585 (ObjNum 1271 snap 420 seq_num 1271) dirty exists 2024-09-11T01:54:12.578 INFO:tasks.rados.rados.0.smithi071.stdout:3711: left oid 46 (ObjNum 1271 snap 420 seq_num 1271) 2024-09-11T01:54:13.322 INFO:tasks.rados.rados.0.smithi071.stdout:3705: done (6 left) 2024-09-11T01:54:13.323 INFO:tasks.rados.rados.0.smithi071.stdout:3706: done (5 left) 2024-09-11T01:54:13.323 INFO:tasks.rados.rados.0.smithi071.stdout:3707: done (4 left) 2024-09-11T01:54:13.323 INFO:tasks.rados.rados.0.smithi071.stdout:3709: done (3 left) 2024-09-11T01:54:13.323 INFO:tasks.rados.rados.0.smithi071.stdout:3710: done (2 left) 2024-09-11T01:54:13.323 INFO:tasks.rados.rados.0.smithi071.stdout:3711: done (1 left) 2024-09-11T01:54:13.323 INFO:tasks.rados.rados.0.smithi071.stdout:3712: done (0 left) 2024-09-11T01:54:13.323 INFO:tasks.rados.rados.0.smithi071.stdout:3713: delete oid 20 current snap is 421 2024-09-11T01:54:13.326 INFO:tasks.rados.rados.0.smithi071.stdout:3713: done (0 left) 2024-09-11T01:54:13.326 INFO:tasks.rados.rados.0.smithi071.stdout:3714: read oid 29 snap -1 2024-09-11T01:54:13.326 INFO:tasks.rados.rados.0.smithi071.stdout:3714: expect (ObjNum 1187 snap 378 seq_num 1187) 2024-09-11T01:54:13.326 INFO:tasks.rados.rados.0.smithi071.stdout:3715: snap_create 2024-09-11T01:54:13.329 INFO:tasks.rados.rados.0.smithi071.stdout:3714: expect (ObjNum 1187 snap 378 seq_num 1187) 2024-09-11T01:54:14.322 INFO:tasks.rados.rados.0.smithi071.stdout:3714: done (1 left) 2024-09-11T01:54:14.322 INFO:tasks.rados.rados.0.smithi071.stdout:3715: done (0 left) 2024-09-11T01:54:14.322 INFO:tasks.rados.rados.0.smithi071.stdout:3716: delete oid 3 current snap is 422 2024-09-11T01:54:14.325 INFO:tasks.rados.rados.0.smithi071.stdout:3716: done (0 left) 2024-09-11T01:54:14.325 INFO:tasks.rados.rados.0.smithi071.stdout:3717: delete oid 49 current snap is 422 2024-09-11T01:54:14.332 INFO:tasks.rados.rados.0.smithi071.stdout:3717: done (0 left) 2024-09-11T01:54:14.332 INFO:tasks.rados.rados.0.smithi071.stdout:3718: read oid 5 snap -1 2024-09-11T01:54:14.332 INFO:tasks.rados.rados.0.smithi071.stdout:3718: expect (ObjNum 1255 snap 415 seq_num 1255) 2024-09-11T01:54:14.332 INFO:tasks.rados.rados.0.smithi071.stdout:3719: delete oid 6 current snap is 422 2024-09-11T01:54:14.337 INFO:tasks.rados.rados.0.smithi071.stdout:3719: done (1 left) 2024-09-11T01:54:14.337 INFO:tasks.rados.rados.0.smithi071.stdout:3720: copy_from oid 13 from oid 4 current snap is 422 2024-09-11T01:54:14.337 INFO:tasks.rados.rados.0.smithi071.stdout:3721: copy_from oid 3 from oid 11 current snap is 422 2024-09-11T01:54:14.337 INFO:tasks.rados.rados.0.smithi071.stdout:3722: write (excl) oid 30 current snap is 422 2024-09-11T01:54:14.337 INFO:tasks.rados.rados.0.smithi071.stdout:3722: seq_num 1276 ranges {645732=648675,1751847=574113,3124327=321405} 2024-09-11T01:54:14.345 INFO:tasks.rados.rados.0.smithi071.stdout:3722: writing smithi07134777-30 from 645732 to 1294407 tid 1 2024-09-11T01:54:14.350 INFO:tasks.rados.rados.0.smithi071.stdout:3722: writing smithi07134777-30 from 1751847 to 2325960 tid 2 2024-09-11T01:54:14.357 INFO:tasks.rados.rados.0.smithi071.stdout:3722: writing smithi07134777-30 from 3124327 to 3445732 tid 3 2024-09-11T01:54:14.357 INFO:tasks.rados.rados.0.smithi071.stdout:3723: rollback oid 8 current snap is 422 2024-09-11T01:54:14.357 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 8 to 325 2024-09-11T01:54:14.357 INFO:tasks.rados.rados.0.smithi071.stdout:3718: expect (ObjNum 1255 snap 415 seq_num 1255) 2024-09-11T01:54:14.357 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:54:14.390 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:54:14.457 INFO:tasks.rados.rados.0.smithi071.stdout:3720: finishing copy_from to smithi07134777-13 2024-09-11T01:54:14.457 INFO:tasks.rados.rados.0.smithi071.stdout:3720: got expected ENOENT (src dne) 2024-09-11T01:54:14.457 INFO:tasks.rados.rados.0.smithi071.stdout:3720: finishing copy_from racing read to smithi07134777-13 2024-09-11T01:54:14.457 INFO:tasks.rados.rados.0.smithi071.stdout:3721: finishing copy_from racing read to smithi07134777-3 2024-09-11T01:54:14.457 INFO:tasks.rados.rados.0.smithi071.stdout:3721: finishing copy_from to smithi07134777-3 2024-09-11T01:54:14.458 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 2206 (ObjNum 1115 snap 356 seq_num 1115) dirty exists 2024-09-11T01:54:14.458 INFO:tasks.rados.rados.0.smithi071.stdout:3722: finishing write tid 1 to smithi07134777-30 2024-09-11T01:54:14.458 INFO:tasks.rados.rados.0.smithi071.stdout:3722: finishing write tid 2 to smithi07134777-30 2024-09-11T01:54:14.458 INFO:tasks.rados.rados.0.smithi071.stdout:3718: done (4 left) 2024-09-11T01:54:14.458 INFO:tasks.rados.rados.0.smithi071.stdout:3720: done (3 left) 2024-09-11T01:54:14.458 INFO:tasks.rados.rados.0.smithi071.stdout:3721: done (2 left) 2024-09-11T01:54:14.458 INFO:tasks.rados.rados.0.smithi071.stdout:3724: delete oid 26 current snap is 422 2024-09-11T01:54:14.458 INFO:tasks.rados.rados.0.smithi071.stdout:3722: finishing write tid 3 to smithi07134777-30 2024-09-11T01:54:14.458 INFO:tasks.rados.rados.0.smithi071.stdout:3722: finishing write tid 4 to smithi07134777-30 2024-09-11T01:54:14.458 INFO:tasks.rados.rados.0.smithi071.stdout:3723: finishing rollback tid 0 to smithi07134777-8 2024-09-11T01:54:14.459 INFO:tasks.rados.rados.0.smithi071.stdout:3722: finishing write tid 5 to smithi07134777-30 2024-09-11T01:54:14.459 INFO:tasks.rados.rados.0.smithi071.stdout:3722: oid 30 updating version 0 to 2217 2024-09-11T01:54:14.459 INFO:tasks.rados.rados.0.smithi071.stdout:3722: oid 30 version 2217 is already newer than 2215 2024-09-11T01:54:14.459 INFO:tasks.rados.rados.0.smithi071.stdout:3722: oid 30 version 2217 is already newer than 2216 2024-09-11T01:54:14.459 INFO:tasks.rados.rados.0.smithi071.stdout:3722: oid 30 version 2217 is already newer than 2214 2024-09-11T01:54:14.459 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 2217 (ObjNum 1276 snap 422 seq_num 1276) dirty exists 2024-09-11T01:54:14.459 INFO:tasks.rados.rados.0.smithi071.stdout:3722: left oid 30 (ObjNum 1276 snap 422 seq_num 1276) 2024-09-11T01:54:14.459 INFO:tasks.rados.rados.0.smithi071.stdout:3723: finishing rollback tid 1 to smithi07134777-8 2024-09-11T01:54:14.459 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 2112 (ObjNum 49 snap 0 seq_num 2342450832) dirty dne 2024-09-11T01:54:14.460 INFO:tasks.rados.rados.0.smithi071.stdout:3722: done (2 left) 2024-09-11T01:54:14.460 INFO:tasks.rados.rados.0.smithi071.stdout:3723: done (1 left) 2024-09-11T01:54:14.460 INFO:tasks.rados.rados.0.smithi071.stdout:3724: done (0 left) 2024-09-11T01:54:14.460 INFO:tasks.rados.rados.0.smithi071.stdout:3725: delete oid 37 current snap is 422 2024-09-11T01:54:14.463 INFO:tasks.rados.rados.0.smithi071.stdout:3725: done (0 left) 2024-09-11T01:54:14.463 INFO:tasks.rados.rados.0.smithi071.stdout:3726: delete oid 48 current snap is 422 2024-09-11T01:54:14.467 INFO:tasks.rados.rados.0.smithi071.stdout:3726: done (0 left) 2024-09-11T01:54:14.467 INFO:tasks.rados.rados.0.smithi071.stdout:3727: delete oid 26 current snap is 422 2024-09-11T01:54:14.471 INFO:tasks.rados.rados.0.smithi071.stdout:3727: done (0 left) 2024-09-11T01:54:14.471 INFO:tasks.rados.rados.0.smithi071.stdout:3728: write oid 39 current snap is 422 2024-09-11T01:54:14.471 INFO:tasks.rados.rados.0.smithi071.stdout:3728: seq_num 1281 ranges {729767=741273,1927904=725109,3145089=384678} 2024-09-11T01:54:14.478 INFO:tasks.rados.rados.0.smithi071.stdout:3728: writing smithi07134777-39 from 729767 to 1471040 tid 1 2024-09-11T01:54:14.485 INFO:tasks.rados.rados.0.smithi071.stdout:3728: writing smithi07134777-39 from 1927904 to 2653013 tid 2 2024-09-11T01:54:14.490 INFO:tasks.rados.rados.0.smithi071.stdout:3728: writing smithi07134777-39 from 3145089 to 3529767 tid 3 2024-09-11T01:54:14.490 INFO:tasks.rados.rados.0.smithi071.stdout:3729: rollback oid 2 current snap is 422 2024-09-11T01:54:14.490 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 2 to 389 2024-09-11T01:54:14.491 INFO:tasks.rados.rados.0.smithi071.stdout:3728: finishing write tid 1 to smithi07134777-39 2024-09-11T01:54:14.491 INFO:tasks.rados.rados.0.smithi071.stdout:3728: finishing write tid 2 to smithi07134777-39 2024-09-11T01:54:14.491 INFO:tasks.rados.rados.0.smithi071.stdout:3730: read oid 35 snap -1 2024-09-11T01:54:14.491 INFO:tasks.rados.rados.0.smithi071.stdout:3730: expect (ObjNum 949 snap 300 seq_num 949) 2024-09-11T01:54:14.491 INFO:tasks.rados.rados.0.smithi071.stdout:3731: write (excl) oid 23 current snap is 422 2024-09-11T01:54:14.491 INFO:tasks.rados.rados.0.smithi071.stdout:3731: seq_num 1282 ranges {746574=416522,1586386=468767,2772512=629544} 2024-09-11T01:54:14.496 INFO:tasks.rados.rados.0.smithi071.stdout:3731: writing smithi07134777-23 from 746574 to 1163096 tid 1 2024-09-11T01:54:14.501 INFO:tasks.rados.rados.0.smithi071.stdout:3731: writing smithi07134777-23 from 1586386 to 2055153 tid 2 2024-09-11T01:54:14.509 INFO:tasks.rados.rados.0.smithi071.stdout:3731: writing smithi07134777-23 from 2772512 to 3402056 tid 3 2024-09-11T01:54:14.509 INFO:tasks.rados.rados.0.smithi071.stdout:3732: read oid 24 snap -1 2024-09-11T01:54:14.510 INFO:tasks.rados.rados.0.smithi071.stdout:3732: expect (ObjNum 1024 snap 328 seq_num 1024) 2024-09-11T01:54:14.510 INFO:tasks.rados.rados.0.smithi071.stdout:3733: delete oid 41 current snap is 422 2024-09-11T01:54:14.510 INFO:tasks.rados.rados.0.smithi071.stdout:3729: finishing rollback tid 0 to smithi07134777-2 2024-09-11T01:54:14.510 INFO:tasks.rados.rados.0.smithi071.stdout:3729: finishing rollback tid 1 to smithi07134777-2 2024-09-11T01:54:14.510 INFO:tasks.rados.rados.0.smithi071.stdout:3729: finishing rollback tid 2 to smithi07134777-2 2024-09-11T01:54:14.510 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 2550 (ObjNum 1131 snap 357 seq_num 1131) dirty exists 2024-09-11T01:54:14.510 INFO:tasks.rados.rados.0.smithi071.stdout:3731: finishing write tid 1 to smithi07134777-23 2024-09-11T01:54:14.510 INFO:tasks.rados.rados.0.smithi071.stdout:3731: finishing write tid 2 to smithi07134777-23 2024-09-11T01:54:14.510 INFO:tasks.rados.rados.0.smithi071.stdout:3728: finishing write tid 3 to smithi07134777-39 2024-09-11T01:54:14.510 INFO:tasks.rados.rados.0.smithi071.stdout:3728: finishing write tid 4 to smithi07134777-39 2024-09-11T01:54:14.510 INFO:tasks.rados.rados.0.smithi071.stdout:3728: finishing write tid 5 to smithi07134777-39 2024-09-11T01:54:14.510 INFO:tasks.rados.rados.0.smithi071.stdout:3728: oid 39 updating version 0 to 2111 2024-09-11T01:54:14.510 INFO:tasks.rados.rados.0.smithi071.stdout:3728: oid 39 updating version 2111 to 2112 2024-09-11T01:54:14.511 INFO:tasks.rados.rados.0.smithi071.stdout:3728: oid 39 updating version 2112 to 2113 2024-09-11T01:54:14.511 INFO:tasks.rados.rados.0.smithi071.stdout:3728: oid 39 version 2113 is already newer than 2110 2024-09-11T01:54:14.511 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 2113 (ObjNum 1281 snap 422 seq_num 1281) dirty exists 2024-09-11T01:54:14.511 INFO:tasks.rados.rados.0.smithi071.stdout:3728: left oid 39 (ObjNum 1281 snap 422 seq_num 1281) 2024-09-11T01:54:14.511 INFO:tasks.rados.rados.0.smithi071.stdout:3730: expect (ObjNum 949 snap 300 seq_num 949) 2024-09-11T01:54:14.511 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:54:14.728 INFO:tasks.rados.rados.0.smithi071.stdout:3731: finishing write tid 3 to smithi07134777-23 2024-09-11T01:54:14.728 INFO:tasks.rados.rados.0.smithi071.stdout:3731: finishing write tid 4 to smithi07134777-23 2024-09-11T01:54:14.728 INFO:tasks.rados.rados.0.smithi071.stdout:3731: finishing write tid 5 to smithi07134777-23 2024-09-11T01:54:14.728 INFO:tasks.rados.rados.0.smithi071.stdout:3731: oid 23 updating version 0 to 1298 2024-09-11T01:54:14.728 INFO:tasks.rados.rados.0.smithi071.stdout:3731: oid 23 updating version 1298 to 1299 2024-09-11T01:54:14.729 INFO:tasks.rados.rados.0.smithi071.stdout:3731: oid 23 updating version 1299 to 1301 2024-09-11T01:54:14.729 INFO:tasks.rados.rados.0.smithi071.stdout:3731: oid 23 version 1301 is already newer than 1300 2024-09-11T01:54:14.729 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 1301 (ObjNum 1282 snap 422 seq_num 1282) dirty exists 2024-09-11T01:54:14.729 INFO:tasks.rados.rados.0.smithi071.stdout:3731: left oid 23 (ObjNum 1282 snap 422 seq_num 1282) 2024-09-11T01:54:14.729 INFO:tasks.rados.rados.0.smithi071.stdout:3732: expect (ObjNum 1024 snap 328 seq_num 1024) 2024-09-11T01:54:14.857 INFO:tasks.rados.rados.0.smithi071.stdout:3728: done (5 left) 2024-09-11T01:54:14.857 INFO:tasks.rados.rados.0.smithi071.stdout:3729: done (4 left) 2024-09-11T01:54:14.857 INFO:tasks.rados.rados.0.smithi071.stdout:3730: done (3 left) 2024-09-11T01:54:14.857 INFO:tasks.rados.rados.0.smithi071.stdout:3731: done (2 left) 2024-09-11T01:54:14.857 INFO:tasks.rados.rados.0.smithi071.stdout:3732: done (1 left) 2024-09-11T01:54:14.857 INFO:tasks.rados.rados.0.smithi071.stdout:3733: done (0 left) 2024-09-11T01:54:14.857 INFO:tasks.rados.rados.0.smithi071.stdout:3734: read oid 33 snap 421 2024-09-11T01:54:14.857 INFO:tasks.rados.rados.0.smithi071.stdout:3734: expect deleted 2024-09-11T01:54:14.857 INFO:tasks.rados.rados.0.smithi071.stdout:3735: read oid 16 snap -1 2024-09-11T01:54:14.857 INFO:tasks.rados.rados.0.smithi071.stdout:3735: expect (ObjNum 1217 snap 397 seq_num 1217) 2024-09-11T01:54:14.857 INFO:tasks.rados.rados.0.smithi071.stdout:3736: write (excl) oid 27 current snap is 422 2024-09-11T01:54:14.858 INFO:tasks.rados.rados.0.smithi071.stdout:3736: seq_num 1284 ranges {780188=567020,2103350=472436,3227358=352830} 2024-09-11T01:54:14.864 INFO:tasks.rados.rados.0.smithi071.stdout:3736: writing smithi07134777-27 from 780188 to 1347208 tid 1 2024-09-11T01:54:14.870 INFO:tasks.rados.rados.0.smithi071.stdout:3736: writing smithi07134777-27 from 2103350 to 2575786 tid 2 2024-09-11T01:54:14.876 INFO:tasks.rados.rados.0.smithi071.stdout:3736: writing smithi07134777-27 from 3227358 to 3580188 tid 3 2024-09-11T01:54:14.876 INFO:tasks.rados.rados.0.smithi071.stdout:3737: snap_remove snap 339 2024-09-11T01:54:15.356 INFO:tasks.rados.rados.0.smithi071.stdout:3737: done (3 left) 2024-09-11T01:54:15.357 INFO:tasks.rados.rados.0.smithi071.stdout:3738: snap_remove snap 409 2024-09-11T01:54:16.360 INFO:tasks.rados.rados.0.smithi071.stdout:3738: done (3 left) 2024-09-11T01:54:16.360 INFO:tasks.rados.rados.0.smithi071.stdout:3739: rollback oid 12 current snap is 422 2024-09-11T01:54:16.360 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 12 to 400 2024-09-11T01:54:16.360 INFO:tasks.rados.rados.0.smithi071.stdout:3740: read oid 34 snap -1 2024-09-11T01:54:16.360 INFO:tasks.rados.rados.0.smithi071.stdout:3740: expect (ObjNum 1121 snap 357 seq_num 1121) 2024-09-11T01:54:16.360 INFO:tasks.rados.rados.0.smithi071.stdout:3741: snap_remove snap 384 2024-09-11T01:54:17.362 INFO:tasks.rados.rados.0.smithi071.stdout:3741: done (5 left) 2024-09-11T01:54:17.362 INFO:tasks.rados.rados.0.smithi071.stdout:3742: snap_create 2024-09-11T01:54:17.362 INFO:tasks.rados.rados.0.smithi071.stdout:3735: expect (ObjNum 1217 snap 397 seq_num 1217) 2024-09-11T01:54:17.451 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:54:17.498 INFO:tasks.rados.rados.0.smithi071.stdout:3736: finishing write tid 1 to smithi07134777-27 2024-09-11T01:54:17.498 INFO:tasks.rados.rados.0.smithi071.stdout:3736: finishing write tid 2 to smithi07134777-27 2024-09-11T01:54:17.498 INFO:tasks.rados.rados.0.smithi071.stdout:3736: finishing write tid 3 to smithi07134777-27 2024-09-11T01:54:17.499 INFO:tasks.rados.rados.0.smithi071.stdout:3736: finishing write tid 4 to smithi07134777-27 2024-09-11T01:54:17.499 INFO:tasks.rados.rados.0.smithi071.stdout:3736: finishing write tid 5 to smithi07134777-27 2024-09-11T01:54:17.499 INFO:tasks.rados.rados.0.smithi071.stdout:3736: oid 27 updating version 0 to 2769 2024-09-11T01:54:17.499 INFO:tasks.rados.rados.0.smithi071.stdout:3736: oid 27 version 2769 is already newer than 2767 2024-09-11T01:54:17.499 INFO:tasks.rados.rados.0.smithi071.stdout:3736: oid 27 updating version 2769 to 2770 2024-09-11T01:54:17.499 INFO:tasks.rados.rados.0.smithi071.stdout:3736: oid 27 version 2770 is already newer than 2768 2024-09-11T01:54:17.499 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 2770 (ObjNum 1284 snap 422 seq_num 1284) dirty exists 2024-09-11T01:54:17.499 INFO:tasks.rados.rados.0.smithi071.stdout:3736: left oid 27 (ObjNum 1284 snap 422 seq_num 1284) 2024-09-11T01:54:17.499 INFO:tasks.rados.rados.0.smithi071.stdout:3739: finishing rollback tid 0 to smithi07134777-12 2024-09-11T01:54:17.499 INFO:tasks.rados.rados.0.smithi071.stdout:3739: finishing rollback tid 1 to smithi07134777-12 2024-09-11T01:54:17.499 INFO:tasks.rados.rados.0.smithi071.stdout:3739: finishing rollback tid 2 to smithi07134777-12 2024-09-11T01:54:17.499 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 2786 (ObjNum 1208 snap 393 seq_num 1208) dirty exists 2024-09-11T01:54:17.499 INFO:tasks.rados.rados.0.smithi071.stdout:3740: expect (ObjNum 1121 snap 357 seq_num 1121) 2024-09-11T01:54:17.513 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:54:18.322 INFO:tasks.rados.rados.0.smithi071.stdout:3734: done (5 left) 2024-09-11T01:54:18.322 INFO:tasks.rados.rados.0.smithi071.stdout:3735: done (4 left) 2024-09-11T01:54:18.322 INFO:tasks.rados.rados.0.smithi071.stdout:3736: done (3 left) 2024-09-11T01:54:18.323 INFO:tasks.rados.rados.0.smithi071.stdout:3739: done (2 left) 2024-09-11T01:54:18.323 INFO:tasks.rados.rados.0.smithi071.stdout:3740: done (1 left) 2024-09-11T01:54:18.323 INFO:tasks.rados.rados.0.smithi071.stdout:3742: done (0 left) 2024-09-11T01:54:18.323 INFO:tasks.rados.rados.0.smithi071.stdout:3743: write oid 29 current snap is 423 2024-09-11T01:54:18.323 INFO:tasks.rados.rados.0.smithi071.stdout:3743: seq_num 1285 ranges {796995=758622,2161832=732447} 2024-09-11T01:54:18.330 INFO:tasks.rados.rados.0.smithi071.stdout:3743: writing smithi07134777-29 from 796995 to 1555617 tid 1 2024-09-11T01:54:18.339 INFO:tasks.rados.rados.0.smithi071.stdout:3743: writing smithi07134777-29 from 2161832 to 2894279 tid 2 2024-09-11T01:54:18.339 INFO:tasks.rados.rados.0.smithi071.stdout:3744: rollback oid 45 current snap is 423 2024-09-11T01:54:18.339 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 45 to 377 2024-09-11T01:54:18.339 INFO:tasks.rados.rados.0.smithi071.stdout:3745: copy_from oid 12 from oid 32 current snap is 423 2024-09-11T01:54:18.339 INFO:tasks.rados.rados.0.smithi071.stdout:3746: read oid 38 snap -1 2024-09-11T01:54:18.339 INFO:tasks.rados.rados.0.smithi071.stdout:3746: expect deleted 2024-09-11T01:54:18.339 INFO:tasks.rados.rados.0.smithi071.stdout:3747: read oid 21 snap -1 2024-09-11T01:54:18.339 INFO:tasks.rados.rados.0.smithi071.stdout:3747: expect (ObjNum 1200 snap 389 seq_num 1200) 2024-09-11T01:54:18.339 INFO:tasks.rados.rados.0.smithi071.stdout:3748: snap_remove snap 366 2024-09-11T01:54:19.340 INFO:tasks.rados.rados.0.smithi071.stdout:3748: done (5 left) 2024-09-11T01:54:19.340 INFO:tasks.rados.rados.0.smithi071.stdout:3749: delete oid 34 current snap is 423 2024-09-11T01:54:19.340 INFO:tasks.rados.rados.0.smithi071.stdout:3743: finishing write tid 1 to smithi07134777-29 2024-09-11T01:54:19.340 INFO:tasks.rados.rados.0.smithi071.stdout:3745: finishing copy_from to smithi07134777-12 2024-09-11T01:54:19.341 INFO:tasks.rados.rados.0.smithi071.stdout:3745: got expected ENOENT (src dne) 2024-09-11T01:54:19.341 INFO:tasks.rados.rados.0.smithi071.stdout:3745: finishing copy_from racing read to smithi07134777-12 2024-09-11T01:54:19.341 INFO:tasks.rados.rados.0.smithi071.stdout:3743: finishing write tid 2 to smithi07134777-29 2024-09-11T01:54:19.341 INFO:tasks.rados.rados.0.smithi071.stdout:3743: finishing write tid 3 to smithi07134777-29 2024-09-11T01:54:19.341 INFO:tasks.rados.rados.0.smithi071.stdout:3743: finishing write tid 4 to smithi07134777-29 2024-09-11T01:54:19.341 INFO:tasks.rados.rados.0.smithi071.stdout:3743: oid 29 updating version 0 to 2928 2024-09-11T01:54:19.341 INFO:tasks.rados.rados.0.smithi071.stdout:3743: oid 29 updating version 2928 to 2929 2024-09-11T01:54:19.341 INFO:tasks.rados.rados.0.smithi071.stdout:3743: oid 29 updating version 2929 to 2930 2024-09-11T01:54:19.341 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 2930 (ObjNum 1285 snap 423 seq_num 1285) dirty exists 2024-09-11T01:54:19.341 INFO:tasks.rados.rados.0.smithi071.stdout:3743: left oid 29 (ObjNum 1285 snap 423 seq_num 1285) 2024-09-11T01:54:19.341 INFO:tasks.rados.rados.0.smithi071.stdout:3744: finishing rollback tid 0 to smithi07134777-45 2024-09-11T01:54:19.341 INFO:tasks.rados.rados.0.smithi071.stdout:3744: finishing rollback tid 1 to smithi07134777-45 2024-09-11T01:54:19.341 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 2386 (ObjNum 65 snap 0 seq_num 1953066355) dirty dne 2024-09-11T01:54:19.341 INFO:tasks.rados.rados.0.smithi071.stdout:3747: expect (ObjNum 1200 snap 389 seq_num 1200) 2024-09-11T01:54:19.399 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 715999 2024-09-11T01:54:19.499 INFO:tasks.rados.rados.0.smithi071.stdout:3743: done (5 left) 2024-09-11T01:54:19.499 INFO:tasks.rados.rados.0.smithi071.stdout:3744: done (4 left) 2024-09-11T01:54:19.499 INFO:tasks.rados.rados.0.smithi071.stdout:3745: done (3 left) 2024-09-11T01:54:19.499 INFO:tasks.rados.rados.0.smithi071.stdout:3746: done (2 left) 2024-09-11T01:54:19.499 INFO:tasks.rados.rados.0.smithi071.stdout:3747: done (1 left) 2024-09-11T01:54:19.499 INFO:tasks.rados.rados.0.smithi071.stdout:3749: done (0 left) 2024-09-11T01:54:19.500 INFO:tasks.rados.rados.0.smithi071.stdout:3750: snap_create 2024-09-11T01:54:20.376 INFO:tasks.rados.rados.0.smithi071.stdout:3750: done (0 left) 2024-09-11T01:54:20.376 INFO:tasks.rados.rados.0.smithi071.stdout:3751: read oid 18 snap -1 2024-09-11T01:54:20.376 INFO:tasks.rados.rados.0.smithi071.stdout:3751: expect deleted 2024-09-11T01:54:20.376 INFO:tasks.rados.rados.0.smithi071.stdout:3752: read oid 35 snap -1 2024-09-11T01:54:20.376 INFO:tasks.rados.rados.0.smithi071.stdout:3752: expect (ObjNum 949 snap 300 seq_num 949) 2024-09-11T01:54:20.377 INFO:tasks.rados.rados.0.smithi071.stdout:3753: copy_from oid 33 from oid 3 current snap is 424 2024-09-11T01:54:20.377 INFO:tasks.rados.rados.0.smithi071.stdout:3754: snap_remove snap 392 2024-09-11T01:54:20.705 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-11T01:54:20.705 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-11T01:54:20.705 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-11T01:54:20.712 INFO:tasks.ceph.mon.c:Started 2024-09-11T01:54:20.712 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-11T01:54:20.712 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-11T01:54:20.712 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T01:54:35.713 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-11T01:54:35.713 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-11T01:54:35.714 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:54:35.714 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:54:35.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.826+0000 7f2d69517640 10 monclient: get_monmap_and_config 2024-09-11T01:54:35.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.826+0000 7f2d69517640 10 monclient: build_initial_monmap 2024-09-11T01:54:35.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.826+0000 7f2d69517640 10 monclient: monmap: 2024-09-11T01:54:35.829 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:35.829 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:35.829 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:54:35.828249+0000 2024-09-11T01:54:35.829 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:54:35.828249+0000 2024-09-11T01:54:35.830 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:35.830 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:35.830 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:54:35.830 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:54:35.830 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:54:35.830 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:54:35.830 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:54:35.830 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:35.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d69517640 1 Processor -- start 2024-09-11T01:54:35.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d69517640 1 -- start start 2024-09-11T01:54:35.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d69517640 10 monclient: init 2024-09-11T01:54:35.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d69517640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:35.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d69517640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-11T01:54:35.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d69517640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2d640a2db0 0x7f2d6406cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:35.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d69517640 10 monclient(hunting): picked mon.noname-c con 0x7f2d640a2db0 addr v2:172.21.15.71:3300/0 2024-09-11T01:54:35.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d69517640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2d6409fbd0 addr v1:172.21.15.12:6789/0 2024-09-11T01:54:35.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d69517640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d6413d550 0x7f2d6413f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:35.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d69517640 10 monclient(hunting): picked mon.noname-b con 0x7f2d6413d550 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:35.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d69517640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d6409eea0 con 0x7f2d6409fbd0 2024-09-11T01:54:35.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d69517640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:35.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d69517640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f2d640a3850 con 0x7f2d640a2db0 2024-09-11T01:54:35.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d69517640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:35.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d69517640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2d640a3190 con 0x7f2d6413d550 2024-09-11T01:54:35.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d69517640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:35.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d69517640 10 monclient(hunting): authenticate will time out at 2282.270708s 2024-09-11T01:54:35.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d627fc640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2d6409fbd0 0x7f2d6406d090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44828/0 (socket says 172.21.15.12:44828) 2024-09-11T01:54:35.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d627fc640 1 -- 172.21.15.12:0/4211254271 learned_addr learned my addr 172.21.15.12:0/4211254271 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:54:35.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d637fe640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d6413d550 0x7f2d6413f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:35.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d637fe640 10 monclient(hunting): get_auth_request con 0x7f2d6413d550 auth_method 0 2024-09-11T01:54:35.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:35.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d637fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:35.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:35.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d637fe640 1 -- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d6413d550 msgr2=0x7f2d6413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:54:35.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d637fe640 1 -- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d6413d550 msgr2=0x7f2d6413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:54:35.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.827+0000 7f2d637fe640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d6413d550 0x7f2d6413f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:54:35.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.828+0000 7f2d637fe640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d6413d550 0x7f2d6413f940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:54:35.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.828+0000 7f2d62ffd640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.71:3300/0 conn(0x7f2d640a2db0 0x7f2d6406cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:35.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.828+0000 7f2d62ffd640 10 monclient(hunting): get_auth_request con 0x7f2d640a2db0 auth_method 0 2024-09-11T01:54:35.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.828+0000 7f2d62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:35.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.828+0000 7f2d62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:35.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.828+0000 7f2d62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:35.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.828+0000 7f2d62ffd640 1 -- 172.21.15.12:0/4211254271 >> v2:172.21.15.71:3300/0 conn(0x7f2d640a2db0 msgr2=0x7f2d6406cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:54:35.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.828+0000 7f2d62ffd640 1 -- 172.21.15.12:0/4211254271 >> v2:172.21.15.71:3300/0 conn(0x7f2d640a2db0 msgr2=0x7f2d6406cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:54:35.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.828+0000 7f2d62ffd640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.71:3300/0 conn(0x7f2d640a2db0 0x7f2d6406cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:54:35.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.828+0000 7f2d62ffd640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.71:3300/0 conn(0x7f2d640a2db0 0x7f2d6406cb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:54:35.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:35.829+0000 7f2d627fc640 1 -- 172.21.15.12:0/4211254271 >> v1:172.21.15.12:6789/0 conn(0x7f2d6409fbd0 legacy=0x7f2d6406d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:36.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.029+0000 7f2d637fe640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d6413d550 0x7f2d6413f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:36.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.029+0000 7f2d637fe640 10 monclient(hunting): get_auth_request con 0x7f2d6413d550 auth_method 0 2024-09-11T01:54:36.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.029+0000 7f2d637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:36.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.029+0000 7f2d637fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:36.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.029+0000 7f2d637fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:54:36.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.029+0000 7f2d637fe640 1 -- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d6413d550 msgr2=0x7f2d6413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:54:36.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.029+0000 7f2d637fe640 1 -- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d6413d550 msgr2=0x7f2d6413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:54:36.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.029+0000 7f2d637fe640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d6413d550 0x7f2d6413f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:54:36.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.029+0000 7f2d637fe640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d6413d550 0x7f2d6413f940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:54:36.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.029+0000 7f2d62ffd640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.71:3300/0 conn(0x7f2d640a2db0 0x7f2d6406cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:36.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.029+0000 7f2d62ffd640 10 monclient(hunting): get_auth_request con 0x7f2d640a2db0 auth_method 0 2024-09-11T01:54:36.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.029+0000 7f2d62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:36.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.029+0000 7f2d62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:36.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.029+0000 7f2d62ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:54:36.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.030+0000 7f2d62ffd640 1 -- 172.21.15.12:0/4211254271 >> v2:172.21.15.71:3300/0 conn(0x7f2d640a2db0 msgr2=0x7f2d6406cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:54:36.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.030+0000 7f2d62ffd640 1 -- 172.21.15.12:0/4211254271 >> v2:172.21.15.71:3300/0 conn(0x7f2d640a2db0 msgr2=0x7f2d6406cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:54:36.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.030+0000 7f2d62ffd640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.71:3300/0 conn(0x7f2d640a2db0 0x7f2d6406cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:54:36.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.030+0000 7f2d62ffd640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.71:3300/0 conn(0x7f2d640a2db0 0x7f2d6406cb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:54:36.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.431+0000 7f2d637fe640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d6413d550 0x7f2d6413f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:36.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.431+0000 7f2d637fe640 10 monclient(hunting): get_auth_request con 0x7f2d6413d550 auth_method 0 2024-09-11T01:54:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.431+0000 7f2d637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.431+0000 7f2d637fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.431+0000 7f2d637fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:54:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.431+0000 7f2d637fe640 1 -- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d6413d550 msgr2=0x7f2d6413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:54:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.431+0000 7f2d637fe640 1 -- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d6413d550 msgr2=0x7f2d6413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:54:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.431+0000 7f2d637fe640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d6413d550 0x7f2d6413f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:54:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.431+0000 7f2d637fe640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d6413d550 0x7f2d6413f940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:54:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.431+0000 7f2d62ffd640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.71:3300/0 conn(0x7f2d640a2db0 0x7f2d6406cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.431+0000 7f2d62ffd640 10 monclient(hunting): get_auth_request con 0x7f2d640a2db0 auth_method 0 2024-09-11T01:54:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.431+0000 7f2d62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.431+0000 7f2d62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.431+0000 7f2d62ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:54:36.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.431+0000 7f2d62ffd640 1 -- 172.21.15.12:0/4211254271 >> v2:172.21.15.71:3300/0 conn(0x7f2d640a2db0 msgr2=0x7f2d6406cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:54:36.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.431+0000 7f2d62ffd640 1 -- 172.21.15.12:0/4211254271 >> v2:172.21.15.71:3300/0 conn(0x7f2d640a2db0 msgr2=0x7f2d6406cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:54:36.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.431+0000 7f2d62ffd640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.71:3300/0 conn(0x7f2d640a2db0 0x7f2d6406cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:54:36.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:36.431+0000 7f2d62ffd640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.71:3300/0 conn(0x7f2d640a2db0 0x7f2d6406cb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:54:37.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:37.232+0000 7f2d637fe640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d6413d550 0x7f2d6413f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:37.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:37.232+0000 7f2d637fe640 10 monclient(hunting): get_auth_request con 0x7f2d6413d550 auth_method 0 2024-09-11T01:54:37.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:37.232+0000 7f2d637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:37.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:37.232+0000 7f2d637fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:37.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:37.232+0000 7f2d637fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:54:37.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:37.232+0000 7f2d637fe640 1 -- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d6413d550 msgr2=0x7f2d6413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:54:37.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:37.232+0000 7f2d637fe640 1 -- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d6413d550 msgr2=0x7f2d6413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:54:37.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:37.232+0000 7f2d637fe640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d6413d550 0x7f2d6413f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:54:37.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:37.232+0000 7f2d637fe640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d6413d550 0x7f2d6413f940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:54:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:37.232+0000 7f2d62ffd640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.71:3300/0 conn(0x7f2d640a2db0 0x7f2d6406cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:37.232+0000 7f2d62ffd640 10 monclient(hunting): get_auth_request con 0x7f2d640a2db0 auth_method 0 2024-09-11T01:54:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:37.232+0000 7f2d62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:37.232+0000 7f2d62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:37.232+0000 7f2d62ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:54:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:37.232+0000 7f2d62ffd640 1 -- 172.21.15.12:0/4211254271 >> v2:172.21.15.71:3300/0 conn(0x7f2d640a2db0 msgr2=0x7f2d6406cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:54:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:37.232+0000 7f2d62ffd640 1 -- 172.21.15.12:0/4211254271 >> v2:172.21.15.71:3300/0 conn(0x7f2d640a2db0 msgr2=0x7f2d6406cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:54:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:37.232+0000 7f2d62ffd640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.71:3300/0 conn(0x7f2d640a2db0 0x7f2d6406cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:54:37.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:37.232+0000 7f2d62ffd640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.71:3300/0 conn(0x7f2d640a2db0 0x7f2d6406cb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:54:38.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.827+0000 7f2d60ff9640 10 monclient(hunting): tick 2024-09-11T01:54:38.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.827+0000 7f2d60ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:54:38.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.827+0000 7f2d60ff9640 1 monclient(hunting): continuing hunt 2024-09-11T01:54:38.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.827+0000 7f2d60ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:54:38.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.827+0000 7f2d60ff9640 1 -- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d6413d550 msgr2=0x7f2d6413f940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:54:38.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.827+0000 7f2d60ff9640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d6413d550 0x7f2d6413f940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:38.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.827+0000 7f2d60ff9640 1 -- 172.21.15.12:0/4211254271 >> v2:172.21.15.71:3300/0 conn(0x7f2d640a2db0 msgr2=0x7f2d6406cb50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:54:38.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.827+0000 7f2d60ff9640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.71:3300/0 conn(0x7f2d640a2db0 0x7f2d6406cb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:38.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.827+0000 7f2d60ff9640 1 -- 172.21.15.12:0/4211254271 >> v1:172.21.15.12:6789/0 conn(0x7f2d6409fbd0 legacy=0x7f2d6406d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:38.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.827+0000 7f2d60ff9640 10 monclient: _add_conns ranks=[1,3] 2024-09-11T01:54:38.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.827+0000 7f2d60ff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2d50003790 addr v1:172.21.15.71:6789/0 2024-09-11T01:54:38.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.827+0000 7f2d60ff9640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.12:3300/0 conn(0x7f2d500074a0 0x7f2d50009960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:38.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.827+0000 7f2d60ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f2d500074a0 addr v2:172.21.15.12:3300/0 2024-09-11T01:54:38.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.827+0000 7f2d60ff9640 1 -- 172.21.15.12:0/4211254271 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d6409f180 con 0x7f2d50003790 2024-09-11T01:54:38.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.827+0000 7f2d60ff9640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:38.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.827+0000 7f2d60ff9640 1 -- 172.21.15.12:0/4211254271 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f2d640a3190 con 0x7f2d500074a0 2024-09-11T01:54:38.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.827+0000 7f2d60ff9640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:38.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.828+0000 7f2d627fc640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.12:3300/0 conn(0x7f2d500074a0 0x7f2d50009960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:38.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.828+0000 7f2d627fc640 10 monclient(hunting): get_auth_request con 0x7f2d500074a0 auth_method 0 2024-09-11T01:54:38.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.828+0000 7f2d627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:38.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.828+0000 7f2d627fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:38.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.828+0000 7f2d627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:38.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.828+0000 7f2d627fc640 1 -- 172.21.15.12:0/4211254271 >> v2:172.21.15.12:3300/0 conn(0x7f2d500074a0 msgr2=0x7f2d50009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:54:38.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.828+0000 7f2d627fc640 1 -- 172.21.15.12:0/4211254271 >> v2:172.21.15.12:3300/0 conn(0x7f2d500074a0 msgr2=0x7f2d50009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:54:38.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.828+0000 7f2d627fc640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.12:3300/0 conn(0x7f2d500074a0 0x7f2d50009960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:54:38.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.828+0000 7f2d627fc640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.12:3300/0 conn(0x7f2d500074a0 0x7f2d50009960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:54:38.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:38.830+0000 7f2d62ffd640 1 -- 172.21.15.12:0/4211254271 >> v1:172.21.15.71:6789/0 conn(0x7f2d50003790 legacy=0x7f2d50005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:39.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:39.029+0000 7f2d627fc640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.12:3300/0 conn(0x7f2d500074a0 0x7f2d50009960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:39.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:39.029+0000 7f2d627fc640 10 monclient(hunting): get_auth_request con 0x7f2d500074a0 auth_method 0 2024-09-11T01:54:39.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:39.029+0000 7f2d627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:39.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:39.029+0000 7f2d627fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:39.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:39.029+0000 7f2d627fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:54:39.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:39.029+0000 7f2d627fc640 1 -- 172.21.15.12:0/4211254271 >> v2:172.21.15.12:3300/0 conn(0x7f2d500074a0 msgr2=0x7f2d50009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:54:39.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:39.029+0000 7f2d627fc640 1 -- 172.21.15.12:0/4211254271 >> v2:172.21.15.12:3300/0 conn(0x7f2d500074a0 msgr2=0x7f2d50009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:54:39.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:39.029+0000 7f2d627fc640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.12:3300/0 conn(0x7f2d500074a0 0x7f2d50009960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:54:39.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:39.029+0000 7f2d627fc640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.12:3300/0 conn(0x7f2d500074a0 0x7f2d50009960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:54:39.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:39.430+0000 7f2d627fc640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.12:3300/0 conn(0x7f2d500074a0 0x7f2d50009960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:39.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:39.430+0000 7f2d627fc640 10 monclient(hunting): get_auth_request con 0x7f2d500074a0 auth_method 0 2024-09-11T01:54:39.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:39.430+0000 7f2d627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:39.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:39.430+0000 7f2d627fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:39.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:39.430+0000 7f2d627fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:54:39.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:39.430+0000 7f2d627fc640 1 -- 172.21.15.12:0/4211254271 >> v2:172.21.15.12:3300/0 conn(0x7f2d500074a0 msgr2=0x7f2d50009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:54:39.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:39.430+0000 7f2d627fc640 1 -- 172.21.15.12:0/4211254271 >> v2:172.21.15.12:3300/0 conn(0x7f2d500074a0 msgr2=0x7f2d50009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:54:39.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:39.430+0000 7f2d627fc640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.12:3300/0 conn(0x7f2d500074a0 0x7f2d50009960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:54:39.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:39.430+0000 7f2d627fc640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.12:3300/0 conn(0x7f2d500074a0 0x7f2d50009960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:54:40.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.231+0000 7f2d627fc640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.12:3300/0 conn(0x7f2d500074a0 0x7f2d50009960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:40.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.231+0000 7f2d627fc640 10 monclient(hunting): get_auth_request con 0x7f2d500074a0 auth_method 0 2024-09-11T01:54:40.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.231+0000 7f2d627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:40.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.231+0000 7f2d627fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:40.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.231+0000 7f2d627fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:54:40.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.231+0000 7f2d627fc640 1 -- 172.21.15.12:0/4211254271 >> v2:172.21.15.12:3300/0 conn(0x7f2d500074a0 msgr2=0x7f2d50009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:54:40.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.231+0000 7f2d627fc640 1 -- 172.21.15.12:0/4211254271 >> v2:172.21.15.12:3300/0 conn(0x7f2d500074a0 msgr2=0x7f2d50009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:54:40.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.231+0000 7f2d627fc640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.12:3300/0 conn(0x7f2d500074a0 0x7f2d50009960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:54:40.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.231+0000 7f2d627fc640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.12:3300/0 conn(0x7f2d500074a0 0x7f2d50009960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:54:40.726 INFO:tasks.rados.rados.0.smithi071.stdout:3754: done (3 left) 2024-09-11T01:54:40.727 INFO:tasks.rados.rados.0.smithi071.stdout:3755: read oid 4 snap 393 2024-09-11T01:54:40.727 INFO:tasks.rados.rados.0.smithi071.stdout:3755: expect deleted 2024-09-11T01:54:40.727 INFO:tasks.rados.rados.0.smithi071.stdout:3756: read oid 39 snap -1 2024-09-11T01:54:40.727 INFO:tasks.rados.rados.0.smithi071.stdout:3756: expect (ObjNum 1281 snap 422 seq_num 1281) 2024-09-11T01:54:40.727 INFO:tasks.rados.rados.0.smithi071.stdout:3757: snap_remove snap 391 2024-09-11T01:54:40.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.727+0000 7f2d61ffb640 1 -- 172.21.15.12:0/4211254271 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1263833991 0 0) 0x7f2d6409f180 con 0x7f2d50003790 2024-09-11T01:54:40.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.727+0000 7f2d61ffb640 10 monclient(hunting): _negotiate 2024-09-11T01:54:40.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.727+0000 7f2d61ffb640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:40.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.727+0000 7f2d61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:40.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.727+0000 7f2d61ffb640 10 monclient(hunting): my global_id is 274159 2024-09-11T01:54:40.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.727+0000 7f2d61ffb640 1 -- 172.21.15.12:0/4211254271 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d4c003520 con 0x7f2d50003790 2024-09-11T01:54:40.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 1 -- 172.21.15.12:0/4211254271 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 221940971 0 0) 0x7f2d4c003520 con 0x7f2d50003790 2024-09-11T01:54:40.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 1 -- 172.21.15.12:0/4211254271 >> v2:172.21.15.12:3300/0 conn(0x7f2d500074a0 msgr2=0x7f2d50009960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:54:40.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 1 --2- 172.21.15.12:0/4211254271 >> v2:172.21.15.12:3300/0 conn(0x7f2d500074a0 0x7f2d50009960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:40.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 10 monclient: _finish_hunting 0 2024-09-11T01:54:40.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 1 monclient: found mon.noname-c-legacy 2024-09-11T01:54:40.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:40.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-09-11T01:54:40.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 1 -- 172.21.15.12:0/4211254271 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d5000b0f0 con 0x7f2d50003790 2024-09-11T01:54:40.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 10 monclient: _finish_auth 0 2024-09-11T01:54:40.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:54:40.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:40.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 1 -- 172.21.15.12:0/4211254271 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f2d54009d10 con 0x7f2d50003790 2024-09-11T01:54:40.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:40.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-11T01:54:40.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 10 monclient: dump: 2024-09-11T01:54:40.752 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:40.752 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:40.752 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:40.752 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:40.752 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:40.752 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:40.752 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:40.752 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:40.752 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:40.752 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:40.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-09-11T01:54:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 1 -- 172.21.15.12:0/4211254271 >> v1:172.21.15.71:6789/0 conn(0x7f2d50003790 legacy=0x7f2d50005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d637fe640 1 -- 172.21.15.12:0/4211254271 reap_dead start 2024-09-11T01:54:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:54:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d4c003aa0 0x7f2d4c003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 10 monclient(hunting): picked mon.c con 0x7f2d4c003aa0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d4c004800 0x7f2d4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 10 monclient(hunting): picked mon.a con 0x7f2d4c004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:54:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d4c005410 0x7f2d4c00ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 10 monclient(hunting): picked mon.b con 0x7f2d4c005410 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:54:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 1 -- 172.21.15.12:0/4211254271 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2d640a3190 con 0x7f2d4c004800 2024-09-11T01:54:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:40.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 1 -- 172.21.15.12:0/4211254271 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c00c320 con 0x7f2d4c005410 2024-09-11T01:54:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 1 -- 172.21.15.12:0/4211254271 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2d4c00c520 con 0x7f2d4c003aa0 2024-09-11T01:54:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d61ffb640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:54:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d62ffd640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d4c003aa0 0x7f2d4c003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d62ffd640 10 monclient(hunting): get_auth_request con 0x7f2d4c003aa0 auth_method 0 2024-09-11T01:54:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d62ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:54:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d627fc640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d4c004800 0x7f2d4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:40.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d637fe640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d4c005410 0x7f2d4c00ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:54:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d637fe640 10 monclient(hunting): get_auth_request con 0x7f2d4c005410 auth_method 0 2024-09-11T01:54:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d637fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d637fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:54:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d627fc640 10 monclient(hunting): get_auth_request con 0x7f2d4c004800 auth_method 0 2024-09-11T01:54:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d627fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d627fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:54:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.746+0000 7f2d627fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:54:40.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.747+0000 7f2d62ffd640 10 monclient(hunting): handle_auth_done global_id 274159 payload 293 2024-09-11T01:54:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.747+0000 7f2d62ffd640 1 -- 172.21.15.12:0/4211254271 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d4c005410 msgr2=0x7f2d4c00ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:54:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.747+0000 7f2d62ffd640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d4c005410 0x7f2d4c00ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.747+0000 7f2d62ffd640 1 -- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d4c004800 msgr2=0x7f2d4c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:54:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.747+0000 7f2d62ffd640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d4c004800 0x7f2d4c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.747+0000 7f2d62ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:54:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.747+0000 7f2d62ffd640 1 monclient: found mon.c 2024-09-11T01:54:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.747+0000 7f2d62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.747+0000 7f2d62ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.747+0000 7f2d62ffd640 1 -- 172.21.15.12:0/4211254271 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d4c003520 con 0x7f2d4c003aa0 2024-09-11T01:54:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.747+0000 7f2d62ffd640 10 monclient: _finish_auth 0 2024-09-11T01:54:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.747+0000 7f2d62ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:54:40.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.747+0000 7f2d62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:40.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.747+0000 7f2d69517640 5 monclient: authenticate success, global_id 274159 2024-09-11T01:54:40.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.747+0000 7f2d69517640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:54:40.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.747+0000 7f2d637fe640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d4c005410 0x7f2d4c00ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:54:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.750+0000 7f2d62ffd640 1 -- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d4c003aa0 msgr2=0x7f2d4c003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.750+0000 7f2d62ffd640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d4c003aa0 0x7f2d4c003ea0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2d54009610 tx=0x7f2d4c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ca5d71445011f78 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:40.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.754+0000 7f2d61ffb640 1 -- 172.21.15.12:0/4211254271 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d54039070 con 0x7f2d4c003aa0 2024-09-11T01:54:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.754+0000 7f2d61ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.754+0000 7f2d61ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.754+0000 7f2d61ffb640 10 monclient: dump: 2024-09-11T01:54:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:40.759 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:40.759 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.754+0000 7f2d61ffb640 1 -- 172.21.15.12:0/4211254271 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d54034410 con 0x7f2d4c003aa0 2024-09-11T01:54:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.754+0000 7f2d61ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.754+0000 7f2d61ffb640 1 -- 172.21.15.12:0/4211254271 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d54004040 con 0x7f2d4c003aa0 2024-09-11T01:54:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.754+0000 7f2d61ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.754+0000 7f2d61ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.754+0000 7f2d61ffb640 10 monclient: dump: 2024-09-11T01:54:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:40.760 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:40.761 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:40.761 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:40.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.754+0000 7f2d69517640 10 monclient: get_monmap_and_config success 2024-09-11T01:54:40.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.754+0000 7f2d69517640 4 set_mon_vals no callback set 2024-09-11T01:54:40.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.754+0000 7f2d69517640 10 monclient: shutdown 2024-09-11T01:54:40.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.755+0000 7f2d69517640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:40.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.755+0000 7f2d69517640 1 -- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d4c003aa0 msgr2=0x7f2d4c003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:40.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.755+0000 7f2d69517640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d4c003aa0 0x7f2d4c003ea0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2d54009610 tx=0x7f2d4c005830 comp rx=0 tx=0).stop 2024-09-11T01:54:40.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.755+0000 7f2d69517640 1 -- 172.21.15.12:0/4211254271 shutdown_connections 2024-09-11T01:54:40.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.755+0000 7f2d69517640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d4c005410 0x7f2d4c00ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:40.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.755+0000 7f2d69517640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d4c004800 0x7f2d4c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:40.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.755+0000 7f2d69517640 1 --2- 172.21.15.12:0/4211254271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d4c003aa0 0x7f2d4c003ea0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:40.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.755+0000 7f2d69517640 1 -- 172.21.15.12:0/4211254271 >> 172.21.15.12:0/4211254271 conn(0x7f2d640a3ea0 msgr2=0x7f2d640a42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:40.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.755+0000 7f2d69517640 1 -- 172.21.15.12:0/4211254271 shutdown_connections 2024-09-11T01:54:40.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.755+0000 7f2d69517640 1 -- 172.21.15.12:0/4211254271 wait complete. 2024-09-11T01:54:40.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.755+0000 7f2d69517640 10 monclient: build_initial_monmap 2024-09-11T01:54:40.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.755+0000 7f2d69517640 10 monclient: monmap: 2024-09-11T01:54:40.762 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:40.762 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:54:40.762 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:54:40.762 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:54:40.762 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:40.762 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:40.762 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:54:40.763 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:54:40.763 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:54:40.763 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:40.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.756+0000 7f2d69517640 1 Processor -- start 2024-09-11T01:54:40.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.756+0000 7f2d69517640 1 -- start start 2024-09-11T01:54:40.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.756+0000 7f2d69517640 10 monclient: init 2024-09-11T01:54:40.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d69517640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:40.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d69517640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:54:40.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d69517640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d4c003aa0 0x7f2d641ccac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:40.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d69517640 10 monclient(hunting): picked mon.noname-a con 0x7f2d4c003aa0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:54:40.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d69517640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d4c004800 0x7f2d64099600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:40.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d69517640 10 monclient(hunting): picked mon.noname-b con 0x7f2d4c004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:54:40.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d69517640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d4c005410 0x7f2d64099b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:40.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d69517640 10 monclient(hunting): picked mon.noname-c con 0x7f2d4c005410 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:40.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d69517640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:40.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d69517640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6409ea20 con 0x7f2d4c003aa0 2024-09-11T01:54:40.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d69517640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:40.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d69517640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6409e8a0 con 0x7f2d4c004800 2024-09-11T01:54:40.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d69517640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:40.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d69517640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2d6406cb50 con 0x7f2d4c005410 2024-09-11T01:54:40.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d69517640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:40.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d69517640 10 monclient(hunting): authenticate will time out at 2287.200367s 2024-09-11T01:54:40.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d62ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d4c003aa0 0x7f2d641ccac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:40.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d627fc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d4c004800 0x7f2d64099600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:40.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d62ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d4c003aa0 0x7f2d641ccac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43512/0 (socket says 172.21.15.12:43512) 2024-09-11T01:54:40.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d62ffd640 1 -- 172.21.15.12:0/2565125453 learned_addr learned my addr 172.21.15.12:0/2565125453 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:40.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d62ffd640 10 monclient(hunting): get_auth_request con 0x7f2d4c003aa0 auth_method 0 2024-09-11T01:54:40.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:40.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:40.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:40.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d637fe640 1 --2- 172.21.15.12:0/2565125453 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d4c005410 0x7f2d64099b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:40.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d627fc640 10 monclient(hunting): get_auth_request con 0x7f2d4c004800 auth_method 0 2024-09-11T01:54:40.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:40.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d627fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:40.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:40.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d637fe640 10 monclient(hunting): get_auth_request con 0x7f2d4c005410 auth_method 0 2024-09-11T01:54:40.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:40.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d637fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:40.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:40.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:40.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:40.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:40.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d637fe640 1 -- 172.21.15.12:0/2565125453 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d4c005410 msgr2=0x7f2d64099b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:54:40.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d637fe640 1 -- 172.21.15.12:0/2565125453 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d4c005410 msgr2=0x7f2d64099b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:54:40.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d637fe640 1 --2- 172.21.15.12:0/2565125453 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d4c005410 0x7f2d64099b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:54:40.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d637fe640 1 --2- 172.21.15.12:0/2565125453 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d4c005410 0x7f2d64099b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:54:40.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:40.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:40.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:40.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d62ffd640 10 monclient(hunting): handle_auth_done global_id 264285 payload 824 2024-09-11T01:54:40.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d62ffd640 1 -- 172.21.15.12:0/2565125453 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d4c005410 msgr2=0x7f2d64099b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:54:40.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d62ffd640 1 --2- 172.21.15.12:0/2565125453 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d4c005410 0x7f2d64099b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:40.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d62ffd640 1 -- 172.21.15.12:0/2565125453 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d4c004800 msgr2=0x7f2d64099600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:40.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d62ffd640 1 --2- 172.21.15.12:0/2565125453 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d4c004800 0x7f2d64099600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:40.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.757+0000 7f2d62ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:54:40.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.758+0000 7f2d62ffd640 1 monclient: found mon.noname-a 2024-09-11T01:54:40.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.758+0000 7f2d62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:40.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.758+0000 7f2d62ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:54:40.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.758+0000 7f2d62ffd640 1 -- 172.21.15.12:0/2565125453 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d64093d50 con 0x7f2d4c003aa0 2024-09-11T01:54:40.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.760+0000 7f2d62ffd640 1 -- 172.21.15.12:0/2565125453 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d4c003aa0 msgr2=0x7f2d641ccac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:40.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.760+0000 7f2d62ffd640 1 --2- 172.21.15.12:0/2565125453 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d4c003aa0 0x7f2d641ccac0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2d5402bd50 tx=0x7f2d54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:40.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.760+0000 7f2d4b7fe640 1 -- 172.21.15.12:0/2565125453 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d54044020 con 0x7f2d4c003aa0 2024-09-11T01:54:40.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.760+0000 7f2d4b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:40.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.760+0000 7f2d4b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:54:40.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.760+0000 7f2d4b7fe640 10 monclient: dump: 2024-09-11T01:54:40.772 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:40.772 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:40.772 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:40.772 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:40.772 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:40.772 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:40.772 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:40.773 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:40.773 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:40.773 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:40.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.760+0000 7f2d4b7fe640 10 monclient: _finish_auth 0 2024-09-11T01:54:40.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.760+0000 7f2d4b7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:54:40.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.760+0000 7f2d4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:40.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.760+0000 7f2d69517640 5 monclient: authenticate success, global_id 264285 2024-09-11T01:54:40.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.760+0000 7f2d4b7fe640 1 -- 172.21.15.12:0/2565125453 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d54004000 con 0x7f2d4c003aa0 2024-09-11T01:54:40.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.760+0000 7f2d4b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:40.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.760+0000 7f2d4b7fe640 1 -- 172.21.15.12:0/2565125453 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d54039040 con 0x7f2d4c003aa0 2024-09-11T01:54:40.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.760+0000 7f2d69517640 10 monclient: _renew_subs 2024-09-11T01:54:40.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.760+0000 7f2d69517640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:54:40.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.760+0000 7f2d69517640 1 -- 172.21.15.12:0/2565125453 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d6409dea0 con 0x7f2d4c003aa0 2024-09-11T01:54:40.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.760+0000 7f2d4b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:40.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.760+0000 7f2d4b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:54:40.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.760+0000 7f2d4b7fe640 10 monclient: dump: 2024-09-11T01:54:40.774 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:40.774 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:40.774 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:40.774 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:40.774 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:40.774 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:40.774 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:40.774 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:40.774 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:40.774 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:40.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.760+0000 7f2d617fa640 4 set_mon_vals no callback set 2024-09-11T01:54:40.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.760+0000 7f2d69517640 10 monclient: _renew_subs 2024-09-11T01:54:40.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.760+0000 7f2d69517640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:54:40.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.760+0000 7f2d69517640 1 -- 172.21.15.12:0/2565125453 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d6409a080 con 0x7f2d4c003aa0 2024-09-11T01:54:40.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.761+0000 7f2d4b7fe640 1 -- 172.21.15.12:0/2565125453 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f2d5403f6f0 con 0x7f2d4c003aa0 2024-09-11T01:54:40.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.762+0000 7f2d69517640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:40.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.762+0000 7f2d69517640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:40.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.762+0000 7f2d69517640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:54:40.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.762+0000 7f2d4b7fe640 1 --2- 172.21.15.12:0/2565125453 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2d5003eca0 0x7f2d50041160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:40.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.762+0000 7f2d69517640 1 -- 172.21.15.12:0/2565125453 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d6406cee0 con 0x7f2d4c003aa0 2024-09-11T01:54:40.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.762+0000 7f2d4b7fe640 1 -- 172.21.15.12:0/2565125453 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(856..856 src has 676..856) ==== 4831+0+0 (secure 0 0 0) 0x7f2d540746b0 con 0x7f2d4c003aa0 2024-09-11T01:54:40.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.762+0000 7f2d627fc640 1 --2- 172.21.15.12:0/2565125453 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2d5003eca0 0x7f2d50041160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:40.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.762+0000 7f2d627fc640 10 monclient: get_auth_request con 0x7f2d5003eca0 auth_method 0 2024-09-11T01:54:40.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.765+0000 7f2d627fc640 1 --2- 172.21.15.12:0/2565125453 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2d5003eca0 0x7f2d50041160 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f2d6409d690 tx=0x7f2d5801a040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:40.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.766+0000 7f2d4b7fe640 1 -- 172.21.15.12:0/2565125453 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2d5404e070 con 0x7f2d4c003aa0 2024-09-11T01:54:40.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.766+0000 7f2d4b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:40.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:40.766+0000 7f2d4b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:54:41.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.024+0000 7f2d69517640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:54:41.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.024+0000 7f2d69517640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:54:41.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.024+0000 7f2d69517640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:54:41.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.024+0000 7f2d69517640 1 -- 172.21.15.12:0/2565125453 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2d64095c80 con 0x7f2d4c003aa0 2024-09-11T01:54:41.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.025+0000 7f2d4b7fe640 1 -- 172.21.15.12:0/2565125453 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f2d54073080 con 0x7f2d4c003aa0 2024-09-11T01:54:41.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.025+0000 7f2d4b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:54:41.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.025+0000 7f2d4b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:54:41.028 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":232,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:54:41.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.027+0000 7f2d69517640 1 -- 172.21.15.12:0/2565125453 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2d5003eca0 msgr2=0x7f2d50041160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.027+0000 7f2d69517640 1 --2- 172.21.15.12:0/2565125453 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2d5003eca0 0x7f2d50041160 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f2d6409d690 tx=0x7f2d5801a040 comp rx=0 tx=0).stop 2024-09-11T01:54:41.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.027+0000 7f2d69517640 10 monclient: shutdown 2024-09-11T01:54:41.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.027+0000 7f2d69517640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:41.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.027+0000 7f2d69517640 1 -- 172.21.15.12:0/2565125453 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d4c003aa0 msgr2=0x7f2d641ccac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.027+0000 7f2d69517640 1 --2- 172.21.15.12:0/2565125453 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d4c003aa0 0x7f2d641ccac0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2d5402bd50 tx=0x7f2d54004a40 comp rx=0 tx=0).stop 2024-09-11T01:54:41.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.027+0000 7f2d69517640 1 -- 172.21.15.12:0/2565125453 shutdown_connections 2024-09-11T01:54:41.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.027+0000 7f2d69517640 1 --2- 172.21.15.12:0/2565125453 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d4c005410 0x7f2d64099b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.027+0000 7f2d69517640 1 --2- 172.21.15.12:0/2565125453 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d4c004800 0x7f2d64099600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.027+0000 7f2d69517640 1 --2- 172.21.15.12:0/2565125453 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2d5003eca0 0x7f2d50041160 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.027+0000 7f2d69517640 1 --2- 172.21.15.12:0/2565125453 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d4c003aa0 0x7f2d641ccac0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.027+0000 7f2d69517640 1 -- 172.21.15.12:0/2565125453 >> 172.21.15.12:0/2565125453 conn(0x7f2d640a3ea0 msgr2=0x7f2d6413c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:41.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.027+0000 7f2d69517640 1 -- 172.21.15.12:0/2565125453 shutdown_connections 2024-09-11T01:54:41.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.027+0000 7f2d69517640 1 -- 172.21.15.12:0/2565125453 wait complete. 2024-09-11T01:54:41.041 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:54:41.041 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:54:41.041 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:54:41.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.140+0000 7f8645517640 10 monclient: get_monmap_and_config 2024-09-11T01:54:41.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.140+0000 7f8645517640 10 monclient: build_initial_monmap 2024-09-11T01:54:41.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.140+0000 7f8645517640 10 monclient: monmap: 2024-09-11T01:54:41.143 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:41.143 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:41.143 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:54:41.142528+0000 2024-09-11T01:54:41.144 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:54:41.142528+0000 2024-09-11T01:54:41.144 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:41.144 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.144 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:54:41.144 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:54:41.144 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:54:41.144 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:54:41.144 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:54:41.144 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.141+0000 7f8645517640 1 Processor -- start 2024-09-11T01:54:41.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.141+0000 7f8645517640 1 -- start start 2024-09-11T01:54:41.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.141+0000 7f8645517640 10 monclient: init 2024-09-11T01:54:41.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.141+0000 7f8645517640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:41.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f8645517640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-11T01:54:41.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f8645517640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86400a1380 0x7f86400a1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f8645517640 10 monclient(hunting): picked mon.noname-b con 0x7f86400a1380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:41.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f8645517640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f86400a0370 0x7f8640135ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f8645517640 10 monclient(hunting): picked mon.noname-c con 0x7f86400a0370 addr v2:172.21.15.71:3300/0 2024-09-11T01:54:41.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f8645517640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f86400a0c90 addr v1:172.21.15.12:6789/0 2024-09-11T01:54:41.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f8645517640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86400a1ca0 con 0x7f86400a0c90 2024-09-11T01:54:41.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f8645517640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:41.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f8645517640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f86400a21c0 con 0x7f86400a0370 2024-09-11T01:54:41.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f8645517640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:41.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f8645517640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f86400a24a0 con 0x7f86400a1380 2024-09-11T01:54:41.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f8645517640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:41.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f8645517640 10 monclient(hunting): authenticate will time out at 2287.585190s 2024-09-11T01:54:41.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f863effd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86400a1380 0x7f86400a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f863effd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86400a1380 0x7f86400a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41070/0 (socket says 172.21.15.12:41070) 2024-09-11T01:54:41.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f863effd640 1 -- 172.21.15.12:0/3634555775 learned_addr learned my addr 172.21.15.12:0/3634555775 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:41.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f863effd640 10 monclient(hunting): get_auth_request con 0x7f86400a1380 auth_method 0 2024-09-11T01:54:41.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f863effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:41.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f863effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:41.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f863effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:41.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f863effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:41.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f863effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:41.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f863effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:41.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f8637fff640 1 --2- 172.21.15.12:0/3634555775 >> v2:172.21.15.71:3300/0 conn(0x7f86400a0370 0x7f8640135ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f8637fff640 10 monclient(hunting): get_auth_request con 0x7f86400a0370 auth_method 0 2024-09-11T01:54:41.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f8637fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:41.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f8637fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:41.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f8637fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:41.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f863effd640 10 monclient(hunting): handle_auth_done global_id 284099 payload 274 2024-09-11T01:54:41.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f863effd640 1 -- 172.21.15.12:0/3634555775 >> v2:172.21.15.71:3300/0 conn(0x7f86400a0370 msgr2=0x7f8640135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:54:41.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f863effd640 1 --2- 172.21.15.12:0/3634555775 >> v2:172.21.15.71:3300/0 conn(0x7f86400a0370 0x7f8640135ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f863effd640 1 -- 172.21.15.12:0/3634555775 >> v1:172.21.15.12:6789/0 conn(0x7f86400a0c90 legacy=0x7f8640136020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:54:41.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f863effd640 10 monclient: _finish_hunting 0 2024-09-11T01:54:41.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f863effd640 1 monclient: found mon.noname-b 2024-09-11T01:54:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f863effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f863effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:54:41.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.142+0000 7f863effd640 1 -- 172.21.15.12:0/3634555775 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86400a2780 con 0x7f86400a1380 2024-09-11T01:54:41.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.145+0000 7f863effd640 1 -- 172.21.15.12:0/3634555775 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86400a1380 msgr2=0x7f86400a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:41.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.145+0000 7f863effd640 1 --2- 172.21.15.12:0/3634555775 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86400a1380 0x7f86400a1760 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8630009870 tx=0x7f863002a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2682c8880e6c3a1d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:41.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.145+0000 7f863e7fc640 1 -- 172.21.15.12:0/3634555775 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8630037070 con 0x7f86400a1380 2024-09-11T01:54:41.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.145+0000 7f863e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:41.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.145+0000 7f863e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:54:41.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.145+0000 7f863e7fc640 10 monclient: dump: 2024-09-11T01:54:41.156 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:41.156 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:41.156 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.156 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.156 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:41.156 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.156 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:41.156 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:41.156 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:41.156 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.145+0000 7f863e7fc640 10 monclient: _finish_auth 0 2024-09-11T01:54:41.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.145+0000 7f863e7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:54:41.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.145+0000 7f863e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:41.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.145+0000 7f8645517640 5 monclient: authenticate success, global_id 284099 2024-09-11T01:54:41.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.145+0000 7f8645517640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:54:41.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.145+0000 7f863e7fc640 1 -- 172.21.15.12:0/3634555775 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8630032440 con 0x7f86400a1380 2024-09-11T01:54:41.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.145+0000 7f863e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:41.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.145+0000 7f8645517640 10 monclient: get_monmap_and_config success 2024-09-11T01:54:41.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.145+0000 7f8645517640 4 set_mon_vals no callback set 2024-09-11T01:54:41.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.145+0000 7f863e7fc640 1 -- 172.21.15.12:0/3634555775 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8630032810 con 0x7f86400a1380 2024-09-11T01:54:41.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.145+0000 7f863e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:41.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.145+0000 7f863e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:41.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.145+0000 7f863e7fc640 10 monclient: dump: 2024-09-11T01:54:41.157 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:41.158 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:41.158 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.158 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.158 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:41.158 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.158 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:41.158 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:41.158 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:41.158 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.145+0000 7f8645517640 10 monclient: shutdown 2024-09-11T01:54:41.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.146+0000 7f8645517640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:41.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.146+0000 7f8645517640 1 -- 172.21.15.12:0/3634555775 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86400a1380 msgr2=0x7f86400a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.146+0000 7f8645517640 1 --2- 172.21.15.12:0/3634555775 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86400a1380 0x7f86400a1760 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8630009870 tx=0x7f863002a0a0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.146+0000 7f8645517640 1 -- 172.21.15.12:0/3634555775 shutdown_connections 2024-09-11T01:54:41.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.146+0000 7f8645517640 1 --2- 172.21.15.12:0/3634555775 >> v2:172.21.15.71:3300/0 conn(0x7f86400a0370 0x7f8640135ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.146+0000 7f8645517640 1 --2- 172.21.15.12:0/3634555775 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86400a1380 0x7f86400a1760 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.146+0000 7f8645517640 1 -- 172.21.15.12:0/3634555775 >> 172.21.15.12:0/3634555775 conn(0x7f86400a2f70 msgr2=0x7f86400a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:41.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.146+0000 7f8645517640 1 -- 172.21.15.12:0/3634555775 shutdown_connections 2024-09-11T01:54:41.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.146+0000 7f8645517640 1 -- 172.21.15.12:0/3634555775 wait complete. 2024-09-11T01:54:41.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.146+0000 7f8645517640 10 monclient: build_initial_monmap 2024-09-11T01:54:41.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.146+0000 7f8645517640 10 monclient: monmap: 2024-09-11T01:54:41.159 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:41.159 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:54:41.159 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:54:41.159 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:54:41.159 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:41.159 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.159 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:54:41.160 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:54:41.160 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:54:41.160 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8645517640 1 Processor -- start 2024-09-11T01:54:41.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8645517640 1 -- start start 2024-09-11T01:54:41.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8645517640 10 monclient: init 2024-09-11T01:54:41.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8645517640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:41.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8645517640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:54:41.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8645517640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86400a0370 0x7f8640135c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8645517640 10 monclient(hunting): picked mon.noname-c con 0x7f86400a0370 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:41.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8645517640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86400a0c90 0x7f86401361c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8645517640 10 monclient(hunting): picked mon.noname-a con 0x7f86400a0c90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:54:41.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8645517640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86400a1380 0x7f86401ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8645517640 10 monclient(hunting): picked mon.noname-b con 0x7f86400a1380 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:54:41.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8645517640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:41.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8645517640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f864009ce20 con 0x7f86400a0c90 2024-09-11T01:54:41.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8645517640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:41.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8645517640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f864009cca0 con 0x7f86400a1380 2024-09-11T01:54:41.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8645517640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:41.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8645517640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8640136730 con 0x7f86400a0370 2024-09-11T01:54:41.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8645517640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:41.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8645517640 10 monclient(hunting): authenticate will time out at 2287.590668s 2024-09-11T01:54:41.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f863effd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86400a0370 0x7f8640135c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f863effd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86400a0370 0x7f8640135c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41080/0 (socket says 172.21.15.12:41080) 2024-09-11T01:54:41.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f863effd640 1 -- 172.21.15.12:0/2872108694 learned_addr learned my addr 172.21.15.12:0/2872108694 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:41.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f863effd640 10 monclient(hunting): get_auth_request con 0x7f86400a0370 auth_method 0 2024-09-11T01:54:41.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8637fff640 1 --2- 172.21.15.12:0/2872108694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86400a0c90 0x7f86401361c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f863effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:41.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f863effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:41.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f863effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:41.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8637fff640 10 monclient(hunting): get_auth_request con 0x7f86400a0c90 auth_method 0 2024-09-11T01:54:41.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8637fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:41.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8637fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:41.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8637fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:41.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f863effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:41.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f863effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:41.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f863effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:41.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8637fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:41.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8637fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:41.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f8637fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:41.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f863f7fe640 1 --2- 172.21.15.12:0/2872108694 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86400a1380 0x7f86401ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f863f7fe640 10 monclient(hunting): get_auth_request con 0x7f86400a1380 auth_method 0 2024-09-11T01:54:41.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f863f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:41.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f863f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:41.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f863f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:41.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.147+0000 7f863effd640 10 monclient(hunting): handle_auth_done global_id 284102 payload 824 2024-09-11T01:54:41.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.148+0000 7f863effd640 1 -- 172.21.15.12:0/2872108694 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86400a1380 msgr2=0x7f86401ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.148+0000 7f863effd640 1 --2- 172.21.15.12:0/2872108694 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86400a1380 0x7f86401ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.148+0000 7f863effd640 1 -- 172.21.15.12:0/2872108694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86400a0c90 msgr2=0x7f86401361c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.148+0000 7f863effd640 1 --2- 172.21.15.12:0/2872108694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86400a0c90 0x7f86401361c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.148+0000 7f863effd640 10 monclient: _finish_hunting 0 2024-09-11T01:54:41.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.148+0000 7f863effd640 1 monclient: found mon.noname-c 2024-09-11T01:54:41.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.148+0000 7f863effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:41.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.148+0000 7f863effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:54:41.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.148+0000 7f863effd640 1 -- 172.21.15.12:0/2872108694 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f864009f1e0 con 0x7f86400a0370 2024-09-11T01:54:41.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.148+0000 7f8637fff640 1 --2- 172.21.15.12:0/2872108694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86400a0c90 0x7f86401361c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:54:41.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.148+0000 7f863f7fe640 1 --2- 172.21.15.12:0/2872108694 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86400a1380 0x7f86401ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:54:41.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.150+0000 7f863effd640 1 -- 172.21.15.12:0/2872108694 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86400a0370 msgr2=0x7f8640135c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:41.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.150+0000 7f863effd640 1 --2- 172.21.15.12:0/2872108694 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86400a0370 0x7f8640135c80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f86300090b0 tx=0x7f86300046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:41.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.150+0000 7f86377fe640 1 -- 172.21.15.12:0/2872108694 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8630041020 con 0x7f86400a0370 2024-09-11T01:54:41.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.150+0000 7f86377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:41.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.150+0000 7f86377fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:54:41.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.150+0000 7f86377fe640 10 monclient: dump: 2024-09-11T01:54:41.164 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:41.164 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:41.164 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.164 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.164 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:41.164 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.165 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:41.165 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:41.165 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:41.165 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.150+0000 7f86377fe640 10 monclient: _finish_auth 0 2024-09-11T01:54:41.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.150+0000 7f86377fe640 10 monclient: _check_auth_tickets 2024-09-11T01:54:41.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.150+0000 7f86377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:41.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.150+0000 7f8645517640 5 monclient: authenticate success, global_id 284102 2024-09-11T01:54:41.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.150+0000 7f86377fe640 1 -- 172.21.15.12:0/2872108694 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f863003d630 con 0x7f86400a0370 2024-09-11T01:54:41.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.150+0000 7f8645517640 10 monclient: _renew_subs 2024-09-11T01:54:41.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.150+0000 7f8645517640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:41.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.150+0000 7f8645517640 1 -- 172.21.15.12:0/2872108694 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86401ca340 con 0x7f86400a0370 2024-09-11T01:54:41.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.150+0000 7f86377fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:41.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.150+0000 7f86377fe640 1 -- 172.21.15.12:0/2872108694 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8630037040 con 0x7f86400a0370 2024-09-11T01:54:41.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.150+0000 7f86377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:41.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.150+0000 7f86377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:41.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.150+0000 7f86377fe640 10 monclient: dump: 2024-09-11T01:54:41.166 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:41.166 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:41.166 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.166 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.166 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:41.166 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.166 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:41.166 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:41.166 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:41.166 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.150+0000 7f8645517640 10 monclient: _renew_subs 2024-09-11T01:54:41.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.150+0000 7f8645517640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:41.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.150+0000 7f863dffb640 4 set_mon_vals no callback set 2024-09-11T01:54:41.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.150+0000 7f8645517640 1 -- 172.21.15.12:0/2872108694 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f86401293c0 con 0x7f86400a0370 2024-09-11T01:54:41.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.151+0000 7f86377fe640 1 -- 172.21.15.12:0/2872108694 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f863003a030 con 0x7f86400a0370 2024-09-11T01:54:41.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.151+0000 7f8645517640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:41.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.151+0000 7f8645517640 1 --2- 172.21.15.12:0/2872108694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f864002d440 0x7f864002d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.151+0000 7f8645517640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:41.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.151+0000 7f8645517640 1 -- 172.21.15.12:0/2872108694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8640073110 con 0x7f864002d440 2024-09-11T01:54:41.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.151+0000 7f8637fff640 1 --2- 172.21.15.12:0/2872108694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f864002d440 0x7f864002d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.151+0000 7f8637fff640 10 monclient: get_auth_request con 0x7f864002d440 auth_method 0 2024-09-11T01:54:41.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.151+0000 7f8637fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:41.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.151+0000 7f8637fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:41.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.151+0000 7f8637fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:41.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.151+0000 7f8637fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:41.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.151+0000 7f86377fe640 1 --2- 172.21.15.12:0/2872108694 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f862003e210 0x7f86200406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.151+0000 7f8637fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:41.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.151+0000 7f8637fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:41.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.151+0000 7f86377fe640 1 -- 172.21.15.12:0/2872108694 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(856..856 src has 676..856) ==== 4831+0+0 (secure 0 0 0) 0x7f86300730f0 con 0x7f86400a0370 2024-09-11T01:54:41.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.151+0000 7f8637fff640 10 monclient(hunting): handle_auth_done global_id 264291 payload 824 2024-09-11T01:54:41.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.151+0000 7f8637fff640 1 -- 172.21.15.12:0/2872108694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f864009d480 con 0x7f864002d440 2024-09-11T01:54:41.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.152+0000 7f863f7fe640 1 --2- 172.21.15.12:0/2872108694 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f862003e210 0x7f86200406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.152+0000 7f863f7fe640 10 monclient: get_auth_request con 0x7f862003e210 auth_method 0 2024-09-11T01:54:41.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.154+0000 7f8637fff640 1 -- 172.21.15.12:0/2872108694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f864002d440 msgr2=0x7f864002d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:41.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.154+0000 7f8637fff640 1 --2- 172.21.15.12:0/2872108694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f864002d440 0x7f864002d820 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f864009f970 tx=0x7f862800e5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:41.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.154+0000 7f86377fe640 1 -- 172.21.15.12:0/2872108694 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8628018020 con 0x7f864002d440 2024-09-11T01:54:41.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.154+0000 7f86377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:41.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.154+0000 7f86377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:54:41.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.154+0000 7f86377fe640 10 monclient: dump: 2024-09-11T01:54:41.171 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:41.171 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:41.171 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.171 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.171 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:41.171 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.171 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:41.172 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:41.172 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:41.172 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.154+0000 7f86377fe640 1 -- 172.21.15.12:0/2872108694 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8628010e70 con 0x7f864002d440 2024-09-11T01:54:41.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.154+0000 7f86377fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:41.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.154+0000 7f86377fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:54:41.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.154+0000 7f86377fe640 1 -- 172.21.15.12:0/2872108694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f864002d440 msgr2=0x7f864002d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.154+0000 7f86377fe640 1 --2- 172.21.15.12:0/2872108694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f864002d440 0x7f864002d820 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f864009f970 tx=0x7f862800e5a0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.154+0000 7f86377fe640 1 -- 172.21.15.12:0/2872108694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f864002d440 msgr2=0x7f864002d820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:54:41.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.154+0000 7f86377fe640 1 --2- 172.21.15.12:0/2872108694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f864002d440 0x7f864002d820 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.154+0000 7f863f7fe640 1 --2- 172.21.15.12:0/2872108694 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f862003e210 0x7f86200406d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f86400a1120 tx=0x7f8638009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:41.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.160+0000 7f8645517640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:54:41.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.160+0000 7f8645517640 1 --2- 172.21.15.12:0/2872108694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f864002dd60 0x7f864002e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.160+0000 7f8645517640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:41.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.160+0000 7f8645517640 1 -- 172.21.15.12:0/2872108694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f86401280a0 con 0x7f864002dd60 2024-09-11T01:54:41.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.160+0000 7f8637fff640 1 --2- 172.21.15.12:0/2872108694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f864002dd60 0x7f864002e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.161+0000 7f8637fff640 10 monclient: get_auth_request con 0x7f864002dd60 auth_method 0 2024-09-11T01:54:41.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.161+0000 7f8637fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:41.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.161+0000 7f8637fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:41.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.161+0000 7f8637fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:41.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.161+0000 7f8637fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:41.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.161+0000 7f8637fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:41.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.161+0000 7f8637fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:41.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.161+0000 7f8637fff640 10 monclient(hunting): handle_auth_done global_id 264294 payload 824 2024-09-11T01:54:41.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.161+0000 7f8637fff640 1 -- 172.21.15.12:0/2872108694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8640120000 con 0x7f864002dd60 2024-09-11T01:54:41.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.163+0000 7f8637fff640 1 -- 172.21.15.12:0/2872108694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f864002dd60 msgr2=0x7f864002e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:41.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.163+0000 7f8637fff640 1 --2- 172.21.15.12:0/2872108694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f864002dd60 0x7f864002e140 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f86401298b0 tx=0x7f86280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:41.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.163+0000 7f86377fe640 1 -- 172.21.15.12:0/2872108694 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8628018030 con 0x7f864002dd60 2024-09-11T01:54:41.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.163+0000 7f86377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:41.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.163+0000 7f86377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:54:41.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.163+0000 7f86377fe640 10 monclient: dump: 2024-09-11T01:54:41.174 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:41.174 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:41.174 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.174 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.174 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:41.174 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.174 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:41.174 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:41.174 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:41.174 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.163+0000 7f86377fe640 1 -- 172.21.15.12:0/2872108694 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f8628003820 con 0x7f864002dd60 2024-09-11T01:54:41.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.163+0000 7f86377fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:54:41.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.163+0000 7f86377fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:54:41.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.163+0000 7f86377fe640 1 -- 172.21.15.12:0/2872108694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f864002dd60 msgr2=0x7f864002e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.163+0000 7f86377fe640 1 --2- 172.21.15.12:0/2872108694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f864002dd60 0x7f864002e140 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f86401298b0 tx=0x7f86280046c0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.163+0000 7f86377fe640 1 -- 172.21.15.12:0/2872108694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f864002dd60 msgr2=0x7f864002e140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:54:41.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.163+0000 7f86377fe640 1 --2- 172.21.15.12:0/2872108694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f864002dd60 0x7f864002e140 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.175 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:54:41.175 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:54:41.175 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:54:41.175 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:54:41.175 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 232, 2024-09-11T01:54:41.175 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 122870, 2024-09-11T01:54:41.176 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:54:41.176 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:54:41.176 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:54:41.176 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:54:41.176 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:41.176 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:54:41.176 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:54:41.176 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:54:41.176 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:54:41.176 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:41.176 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:41.176 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:41.176 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:41.176 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:41.177 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:41.177 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:41.177 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:41.177 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:41.177 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:41.177 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:41.177 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:41.177 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:54:41.177 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:54:41.177 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:41.177 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:41.177 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:41.177 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:41.178 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:41.178 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:41.178 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:41.178 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:41.178 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:41.178 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:41.178 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:41.178 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:41.178 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.178 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:54:41.178 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:54:41.178 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:54:41.178 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:54:41.178 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:54:41.178 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:54:41.179 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:54:41.179 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:54:41.179 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:54:41.179 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:54:41.179 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:54:41.179 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:54:41.179 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:54:41.179 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:54:41.179 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:54:41.179 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:54:41.179 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:54:41.179 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:41.179 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:41.179 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:41.179 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:41.180 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:41.180 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:41.180 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:41.180 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:41.180 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:41.180 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:41.180 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:41.180 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:41.180 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:54:41.180 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.180 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:54:41.180 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.180 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:54:41.180 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:54:41.181 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:41.181 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:41.181 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.181 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:41.181 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:54:41.181 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:41.181 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.181 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.181 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:41.181 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:54:41.181 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:41.181 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:41.181 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:41.181 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.182 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:54:41.182 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:54:41.182 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:41.182 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:41.182 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:41.182 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.182 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.182 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:54:41.182 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:54:41.182 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:41.182 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:41.182 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.182 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:41.182 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:54:41.183 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:41.183 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.183 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.183 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:41.183 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:54:41.183 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:41.183 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:41.183 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:41.183 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.183 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:54:41.183 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:54:41.183 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:41.183 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:41.183 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:41.183 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.184 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.184 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:54:41.184 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:54:41.184 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:41.184 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:41.184 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.184 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:41.184 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:54:41.184 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:41.184 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.184 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.184 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:41.184 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:54:41.184 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:41.185 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:41.185 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:41.185 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.185 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:54:41.185 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:54:41.185 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:41.185 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:41.185 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:41.185 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:41.185 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:41.185 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.185 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:54:41.185 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:54:41.186 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.186 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:41.186 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:41.186 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:54:41.186 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:41.186 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:41.186 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:54:41.186 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.186 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:41.186 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:41.186 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:54:41.186 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:41.186 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:41.186 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:54:41.187 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.187 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:41.187 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:41.187 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:54:41.187 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:41.187 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:41.187 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.187 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:54:41.187 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:54:41.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.164+0000 7f8645517640 1 -- 172.21.15.12:0/2872108694 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f862003e210 msgr2=0x7f86200406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.164+0000 7f8645517640 1 --2- 172.21.15.12:0/2872108694 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f862003e210 0x7f86200406d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f86400a1120 tx=0x7f8638009040 comp rx=0 tx=0).stop 2024-09-11T01:54:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.164+0000 7f8645517640 10 monclient: shutdown 2024-09-11T01:54:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.164+0000 7f8645517640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.164+0000 7f8645517640 1 -- 172.21.15.12:0/2872108694 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86400a0370 msgr2=0x7f8640135c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.164+0000 7f8645517640 1 --2- 172.21.15.12:0/2872108694 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f86400a0370 0x7f8640135c80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f86300090b0 tx=0x7f86300046c0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.164+0000 7f863f7fe640 1 -- 172.21.15.12:0/2872108694 reap_dead start 2024-09-11T01:54:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.164+0000 7f863f7fe640 1 -- 172.21.15.12:0/2872108694 reap_dead start 2024-09-11T01:54:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.164+0000 7f8645517640 1 -- 172.21.15.12:0/2872108694 shutdown_connections 2024-09-11T01:54:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.164+0000 7f8645517640 1 -- 172.21.15.12:0/2872108694 >> 172.21.15.12:0/2872108694 conn(0x7f86400a2f70 msgr2=0x7f8640071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.164+0000 7f8645517640 1 -- 172.21.15.12:0/2872108694 shutdown_connections 2024-09-11T01:54:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.164+0000 7f8645517640 1 -- 172.21.15.12:0/2872108694 wait complete. 2024-09-11T01:54:41.189 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:54:41.190 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:54:41.190 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:54:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.289+0000 7f30207e2640 10 monclient: get_monmap_and_config 2024-09-11T01:54:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.289+0000 7f30207e2640 10 monclient: build_initial_monmap 2024-09-11T01:54:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.289+0000 7f30207e2640 10 monclient: monmap: 2024-09-11T01:54:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:54:41.291200+0000 2024-09-11T01:54:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:54:41.291200+0000 2024-09-11T01:54:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:54:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:54:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:54:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:54:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:54:41.293 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.290+0000 7f30207e2640 1 Processor -- start 2024-09-11T01:54:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.290+0000 7f30207e2640 1 -- start start 2024-09-11T01:54:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.290+0000 7f30207e2640 10 monclient: init 2024-09-11T01:54:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.290+0000 7f30207e2640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.290+0000 7f30207e2640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-11T01:54:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.290+0000 7f30207e2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f30180a2e20 addr v1:172.21.15.71:6789/0 2024-09-11T01:54:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.290+0000 7f30207e2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a4970 0x7f301806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.290+0000 7f30207e2640 10 monclient(hunting): picked mon.noname-b con 0x7f30180a4970 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.290+0000 7f30207e2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f301806f910 addr v1:172.21.15.12:6789/0 2024-09-11T01:54:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.290+0000 7f30207e2640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f301806f4f0 con 0x7f301806f910 2024-09-11T01:54:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.290+0000 7f30207e2640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30180a3910 con 0x7f30180a2e20 2024-09-11T01:54:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.290+0000 7f30207e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.290+0000 7f30207e2640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f301806ef30 con 0x7f30180a4970 2024-09-11T01:54:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.290+0000 7f30207e2640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.290+0000 7f30207e2640 10 monclient(hunting): authenticate will time out at 2287.733825s 2024-09-11T01:54:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.290+0000 7f301dd7d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a4970 0x7f301806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.290+0000 7f301ed7f640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f301806f910 0x7f301813f430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:60994/0 (socket says 172.21.15.12:60994) 2024-09-11T01:54:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.290+0000 7f301dd7d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a4970 0x7f301806ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41084/0 (socket says 172.21.15.12:41084) 2024-09-11T01:54:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.290+0000 7f301ed7f640 1 -- 172.21.15.12:0/2710445018 learned_addr learned my addr 172.21.15.12:0/2710445018 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:54:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.290+0000 7f301dd7d640 10 monclient(hunting): get_auth_request con 0x7f30180a4970 auth_method 0 2024-09-11T01:54:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.290+0000 7f301dd7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.290+0000 7f301dd7d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.290+0000 7f301dd7d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.291+0000 7f301dd7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.291+0000 7f301dd7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.291+0000 7f301dd7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.291+0000 7f301dd7d640 10 monclient(hunting): handle_auth_done global_id 284105 payload 274 2024-09-11T01:54:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.291+0000 7f301dd7d640 1 -- 172.21.15.12:0/2710445018 >> v1:172.21.15.71:6789/0 conn(0x7f30180a2e20 legacy=0x7f30180a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:54:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.291+0000 7f301dd7d640 1 -- 172.21.15.12:0/2710445018 >> v1:172.21.15.12:6789/0 conn(0x7f301806f910 legacy=0x7f301813f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:54:41.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.291+0000 7f301dd7d640 10 monclient: _finish_hunting 0 2024-09-11T01:54:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.291+0000 7f301dd7d640 1 monclient: found mon.noname-b 2024-09-11T01:54:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.291+0000 7f301dd7d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.291+0000 7f301dd7d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:54:41.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.291+0000 7f301dd7d640 1 -- 172.21.15.12:0/2710445018 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30180a3ae0 con 0x7f30180a4970 2024-09-11T01:54:41.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.294+0000 7f301dd7d640 1 -- 172.21.15.12:0/2710445018 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a4970 msgr2=0x7f301806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:41.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.294+0000 7f301dd7d640 1 --2- 172.21.15.12:0/2710445018 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a4970 0x7f301806ded0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f30080097d0 tx=0x7f3008029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14f1ff2f6389695e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:41.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.294+0000 7f301d57c640 1 -- 172.21.15.12:0/2710445018 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3008037070 con 0x7f30180a4970 2024-09-11T01:54:41.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.294+0000 7f301d57c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:41.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.294+0000 7f301d57c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:54:41.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.294+0000 7f301d57c640 10 monclient: dump: 2024-09-11T01:54:41.305 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:41.305 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:41.305 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.305 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.305 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:41.305 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.305 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:41.305 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:41.305 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:41.305 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.294+0000 7f301d57c640 10 monclient: _finish_auth 0 2024-09-11T01:54:41.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.294+0000 7f301d57c640 10 monclient: _check_auth_tickets 2024-09-11T01:54:41.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.294+0000 7f301d57c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:41.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.294+0000 7f30207e2640 5 monclient: authenticate success, global_id 284105 2024-09-11T01:54:41.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.294+0000 7f30207e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:54:41.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.294+0000 7f301d57c640 1 -- 172.21.15.12:0/2710445018 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3008032440 con 0x7f30180a4970 2024-09-11T01:54:41.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.294+0000 7f301d57c640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:41.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.294+0000 7f301d57c640 1 -- 172.21.15.12:0/2710445018 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3008032810 con 0x7f30180a4970 2024-09-11T01:54:41.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.294+0000 7f30207e2640 10 monclient: get_monmap_and_config success 2024-09-11T01:54:41.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.294+0000 7f30207e2640 4 set_mon_vals no callback set 2024-09-11T01:54:41.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.294+0000 7f301d57c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:41.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.294+0000 7f301d57c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:41.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.294+0000 7f301d57c640 10 monclient: dump: 2024-09-11T01:54:41.306 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:41.306 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:41.307 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.307 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.307 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:41.307 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.307 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:41.307 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:41.307 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:41.307 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.294+0000 7f30207e2640 10 monclient: shutdown 2024-09-11T01:54:41.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.294+0000 7f30207e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:41.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.294+0000 7f30207e2640 1 -- 172.21.15.12:0/2710445018 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a4970 msgr2=0x7f301806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.294+0000 7f30207e2640 1 --2- 172.21.15.12:0/2710445018 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a4970 0x7f301806ded0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f30080097d0 tx=0x7f3008029fb0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.295+0000 7f30207e2640 1 -- 172.21.15.12:0/2710445018 shutdown_connections 2024-09-11T01:54:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.295+0000 7f30207e2640 1 --2- 172.21.15.12:0/2710445018 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a4970 0x7f301806ded0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.295+0000 7f30207e2640 1 -- 172.21.15.12:0/2710445018 >> 172.21.15.12:0/2710445018 conn(0x7f30180a8600 msgr2=0x7f30180a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.295+0000 7f30207e2640 1 -- 172.21.15.12:0/2710445018 shutdown_connections 2024-09-11T01:54:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.295+0000 7f30207e2640 1 -- 172.21.15.12:0/2710445018 wait complete. 2024-09-11T01:54:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.295+0000 7f30207e2640 10 monclient: build_initial_monmap 2024-09-11T01:54:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.295+0000 7f30207e2640 10 monclient: monmap: 2024-09-11T01:54:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:54:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:54:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:54:41.308 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:54:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:54:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:54:41.309 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.295+0000 7f30207e2640 1 Processor -- start 2024-09-11T01:54:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f30207e2640 1 -- start start 2024-09-11T01:54:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f30207e2640 10 monclient: init 2024-09-11T01:54:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f30207e2640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f30207e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:54:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f30207e2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f301806f910 0x7f30181cd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f30207e2640 10 monclient(hunting): picked mon.noname-b con 0x7f301806f910 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:54:41.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f30207e2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a2e20 0x7f301813f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f30207e2640 10 monclient(hunting): picked mon.noname-c con 0x7f30180a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:41.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f30207e2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30180a4970 0x7f301813feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f30207e2640 10 monclient(hunting): picked mon.noname-a con 0x7f30180a4970 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:54:41.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f30207e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:41.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f30207e2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3018140d60 con 0x7f30180a4970 2024-09-11T01:54:41.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f30207e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:41.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f30207e2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f301806ec90 con 0x7f301806f910 2024-09-11T01:54:41.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f30207e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:41.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f30207e2640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f3018140ee0 con 0x7f30180a2e20 2024-09-11T01:54:41.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f30207e2640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:41.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f30207e2640 10 monclient(hunting): authenticate will time out at 2287.739474s 2024-09-11T01:54:41.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301dd7d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a2e20 0x7f301813f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301ed7f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30180a4970 0x7f301813feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301dd7d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a2e20 0x7f301813f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41096/0 (socket says 172.21.15.12:41096) 2024-09-11T01:54:41.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301dd7d640 1 -- 172.21.15.12:0/3353852810 learned_addr learned my addr 172.21.15.12:0/3353852810 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:41.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301dd7d640 10 monclient(hunting): get_auth_request con 0x7f30180a2e20 auth_method 0 2024-09-11T01:54:41.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301dd7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:41.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301dd7d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:41.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301dd7d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:41.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301ed7f640 10 monclient(hunting): get_auth_request con 0x7f30180a4970 auth_method 0 2024-09-11T01:54:41.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301ed7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:41.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301ed7f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:41.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301ed7f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:41.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301dd7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:41.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301dd7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:41.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301dd7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:41.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301ed7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:41.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301ed7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:41.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301ed7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:41.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301e57e640 1 --2- 172.21.15.12:0/3353852810 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f301806f910 0x7f30181cd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301e57e640 10 monclient(hunting): get_auth_request con 0x7f301806f910 auth_method 0 2024-09-11T01:54:41.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301e57e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:41.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301e57e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:41.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301e57e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:41.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301dd7d640 10 monclient(hunting): handle_auth_done global_id 284108 payload 824 2024-09-11T01:54:41.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301dd7d640 1 -- 172.21.15.12:0/3353852810 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f301806f910 msgr2=0x7f30181cd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301dd7d640 1 --2- 172.21.15.12:0/3353852810 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f301806f910 0x7f30181cd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301dd7d640 1 -- 172.21.15.12:0/3353852810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30180a4970 msgr2=0x7f301813feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301dd7d640 1 --2- 172.21.15.12:0/3353852810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30180a4970 0x7f301813feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301dd7d640 10 monclient: _finish_hunting 0 2024-09-11T01:54:41.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301dd7d640 1 monclient: found mon.noname-c 2024-09-11T01:54:41.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301dd7d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:41.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301dd7d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:54:41.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301dd7d640 1 -- 172.21.15.12:0/3353852810 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f301813b410 con 0x7f30180a2e20 2024-09-11T01:54:41.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.296+0000 7f301ed7f640 1 --2- 172.21.15.12:0/3353852810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f30180a4970 0x7f301813feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:54:41.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.299+0000 7f301dd7d640 1 -- 172.21.15.12:0/3353852810 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a2e20 msgr2=0x7f301813f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:41.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.299+0000 7f301dd7d640 1 --2- 172.21.15.12:0/3353852810 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a2e20 0x7f301813f970 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f30080098f0 tx=0x7f30080046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:41.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.299+0000 7f300effd640 1 -- 172.21.15.12:0/3353852810 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3008042020 con 0x7f30180a2e20 2024-09-11T01:54:41.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.299+0000 7f300effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:41.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.299+0000 7f300effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:54:41.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.299+0000 7f300effd640 10 monclient: dump: 2024-09-11T01:54:41.314 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:41.314 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:41.314 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.314 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.314 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:41.314 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.314 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:41.314 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:41.314 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:41.314 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.299+0000 7f300effd640 10 monclient: _finish_auth 0 2024-09-11T01:54:41.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.299+0000 7f300effd640 10 monclient: _check_auth_tickets 2024-09-11T01:54:41.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.299+0000 7f300effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:41.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.299+0000 7f300effd640 1 -- 172.21.15.12:0/3353852810 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f300803d8a0 con 0x7f30180a2e20 2024-09-11T01:54:41.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.299+0000 7f30207e2640 5 monclient: authenticate success, global_id 284108 2024-09-11T01:54:41.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.299+0000 7f30207e2640 10 monclient: _renew_subs 2024-09-11T01:54:41.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.299+0000 7f30207e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:41.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.299+0000 7f30207e2640 1 -- 172.21.15.12:0/3353852810 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f301809f610 con 0x7f30180a2e20 2024-09-11T01:54:41.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.299+0000 7f300effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:41.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.299+0000 7f300effd640 1 -- 172.21.15.12:0/3353852810 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3008037040 con 0x7f30180a2e20 2024-09-11T01:54:41.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.299+0000 7f300effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:41.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.299+0000 7f300effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:41.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.299+0000 7f300effd640 10 monclient: dump: 2024-09-11T01:54:41.315 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:41.315 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:41.316 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.316 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.316 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:41.316 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.316 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:41.316 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:41.316 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:41.316 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.299+0000 7f300ffff640 4 set_mon_vals no callback set 2024-09-11T01:54:41.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.299+0000 7f30207e2640 10 monclient: _renew_subs 2024-09-11T01:54:41.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.299+0000 7f30207e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:41.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.299+0000 7f30207e2640 1 -- 172.21.15.12:0/3353852810 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3018140860 con 0x7f30180a2e20 2024-09-11T01:54:41.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.300+0000 7f30207e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:41.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.300+0000 7f30207e2640 1 --2- 172.21.15.12:0/3353852810 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3018137c70 0x7f301802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.300+0000 7f30207e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:41.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.300+0000 7f30207e2640 1 -- 172.21.15.12:0/3353852810 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f301806a060 con 0x7f3018137c70 2024-09-11T01:54:41.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.300+0000 7f300effd640 1 -- 172.21.15.12:0/3353852810 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f300803dc90 con 0x7f30180a2e20 2024-09-11T01:54:41.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.300+0000 7f301e57e640 1 --2- 172.21.15.12:0/3353852810 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3018137c70 0x7f301802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.300+0000 7f301e57e640 10 monclient: get_auth_request con 0x7f3018137c70 auth_method 0 2024-09-11T01:54:41.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.300+0000 7f301e57e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:41.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.300+0000 7f301e57e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:41.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.300+0000 7f301e57e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:41.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.300+0000 7f301e57e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:41.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.300+0000 7f301e57e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:41.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.300+0000 7f301e57e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:41.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.300+0000 7f301e57e640 10 monclient(hunting): handle_auth_done global_id 284111 payload 824 2024-09-11T01:54:41.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.300+0000 7f301e57e640 1 -- 172.21.15.12:0/3353852810 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f301806ded0 con 0x7f3018137c70 2024-09-11T01:54:41.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.300+0000 7f300effd640 1 --2- 172.21.15.12:0/3353852810 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2ffc03e260 0x7f2ffc040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.300+0000 7f300effd640 1 -- 172.21.15.12:0/3353852810 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(856..856 src has 676..856) ==== 4831+0+0 (secure 0 0 0) 0x7f30080737f0 con 0x7f30180a2e20 2024-09-11T01:54:41.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.300+0000 7f301ed7f640 1 --2- 172.21.15.12:0/3353852810 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2ffc03e260 0x7f2ffc040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.300+0000 7f301ed7f640 10 monclient: get_auth_request con 0x7f2ffc03e260 auth_method 0 2024-09-11T01:54:41.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.302+0000 7f301e57e640 1 -- 172.21.15.12:0/3353852810 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3018137c70 msgr2=0x7f301802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:41.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.302+0000 7f301e57e640 1 --2- 172.21.15.12:0/3353852810 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3018137c70 0x7f301802d440 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f301400e4d0 tx=0x7f301400e990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:41.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.302+0000 7f300effd640 1 -- 172.21.15.12:0/3353852810 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3014007450 con 0x7f3018137c70 2024-09-11T01:54:41.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.302+0000 7f300effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:41.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.302+0000 7f300effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:41.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.302+0000 7f300effd640 10 monclient: dump: 2024-09-11T01:54:41.321 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:41.321 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:41.321 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.321 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.321 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:41.321 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.321 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:41.322 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:41.322 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:41.322 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.303+0000 7f300effd640 1 -- 172.21.15.12:0/3353852810 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3014007450 con 0x7f3018137c70 2024-09-11T01:54:41.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.303+0000 7f300effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:41.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.303+0000 7f300effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:54:41.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.303+0000 7f300effd640 1 -- 172.21.15.12:0/3353852810 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3018137c70 msgr2=0x7f301802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.303+0000 7f300effd640 1 --2- 172.21.15.12:0/3353852810 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3018137c70 0x7f301802d440 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f301400e4d0 tx=0x7f301400e990 comp rx=0 tx=0).stop 2024-09-11T01:54:41.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.303+0000 7f300effd640 1 -- 172.21.15.12:0/3353852810 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3018137c70 msgr2=0x7f301802d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:54:41.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.303+0000 7f300effd640 1 --2- 172.21.15.12:0/3353852810 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3018137c70 0x7f301802d440 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f301400e4d0 tx=0x7f301400e990 comp rx=0 tx=0).stop 2024-09-11T01:54:41.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.303+0000 7f301ed7f640 1 --2- 172.21.15.12:0/3353852810 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2ffc03e260 0x7f2ffc040720 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f3010005c00 tx=0x7f3010009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:41.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.309+0000 7f30207e2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:54:41.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.309+0000 7f30207e2640 1 --2- 172.21.15.12:0/3353852810 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3018138050 0x7f301802d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.309+0000 7f30207e2640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.309+0000 7f30207e2640 1 -- 172.21.15.12:0/3353852810 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f30180a0ef0 con 0x7f3018138050 2024-09-11T01:54:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.309+0000 7f301e57e640 1 --2- 172.21.15.12:0/3353852810 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3018138050 0x7f301802d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.309+0000 7f301e57e640 10 monclient: get_auth_request con 0x7f3018138050 auth_method 0 2024-09-11T01:54:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.309+0000 7f301e57e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.309+0000 7f301e57e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.309+0000 7f301e57e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.309+0000 7f301e57e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.309+0000 7f301e57e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.309+0000 7f301e57e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.309+0000 7f301e57e640 10 monclient(hunting): handle_auth_done global_id 284114 payload 824 2024-09-11T01:54:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.309+0000 7f301e57e640 1 -- 172.21.15.12:0/3353852810 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3018139da0 con 0x7f3018138050 2024-09-11T01:54:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.311+0000 7f301e57e640 1 -- 172.21.15.12:0/3353852810 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3018138050 msgr2=0x7f301802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.311+0000 7f301e57e640 1 --2- 172.21.15.12:0/3353852810 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3018138050 0x7f301802d980 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f301400e4d0 tx=0x7f3014003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.312+0000 7f300effd640 1 -- 172.21.15.12:0/3353852810 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f301401eb70 con 0x7f3018138050 2024-09-11T01:54:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.312+0000 7f300effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.312+0000 7f300effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.312+0000 7f300effd640 10 monclient: dump: 2024-09-11T01:54:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:41.325 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:41.325 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.312+0000 7f300effd640 1 -- 172.21.15.12:0/3353852810 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7f3014028430 con 0x7f3018138050 2024-09-11T01:54:41.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.312+0000 7f300effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:54:41.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.312+0000 7f300effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:54:41.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.312+0000 7f300effd640 1 -- 172.21.15.12:0/3353852810 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3018138050 msgr2=0x7f301802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.312+0000 7f300effd640 1 --2- 172.21.15.12:0/3353852810 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3018138050 0x7f301802d980 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f301400e4d0 tx=0x7f3014003760 comp rx=0 tx=0).stop 2024-09-11T01:54:41.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.312+0000 7f300effd640 1 -- 172.21.15.12:0/3353852810 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3018138050 msgr2=0x7f301802d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:54:41.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.312+0000 7f300effd640 1 --2- 172.21.15.12:0/3353852810 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3018138050 0x7f301802d980 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f301400e4d0 tx=0x7f3014003760 comp rx=0 tx=0).stop 2024-09-11T01:54:41.325 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:54:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:54:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:54:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:54:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 232, 2024-09-11T01:54:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 20508, 2024-09-11T01:54:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:54:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:54:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:54:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:54:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:54:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:54:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:54:41.326 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:54:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:41.327 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:54:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:54:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:41.328 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:54:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:54:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:54:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:54:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:54:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:54:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:54:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:54:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:54:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:54:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:54:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:54:41.329 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:54:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:54:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:54:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:54:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:54:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:41.330 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:54:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:54:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:54:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:54:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.331 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:54:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:54:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:54:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:54:41.332 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:54:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:54:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:54:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.333 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.334 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:41.334 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:54:41.334 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:41.334 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:41.334 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:41.334 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.334 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:54:41.334 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:54:41.334 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:41.334 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:41.334 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:41.334 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:54:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:54:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:54:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:54:41.335 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:54:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:54:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:54:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:54:41.336 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:54:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:54:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:54:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.337 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:54:41.338 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:54:41.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.312+0000 7f30207e2640 1 -- 172.21.15.12:0/3353852810 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2ffc03e260 msgr2=0x7f2ffc040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.312+0000 7f30207e2640 1 --2- 172.21.15.12:0/3353852810 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2ffc03e260 0x7f2ffc040720 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f3010005c00 tx=0x7f3010009040 comp rx=0 tx=0).stop 2024-09-11T01:54:41.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.312+0000 7f30207e2640 10 monclient: shutdown 2024-09-11T01:54:41.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.312+0000 7f30207e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:41.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.312+0000 7f30207e2640 1 -- 172.21.15.12:0/3353852810 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a2e20 msgr2=0x7f301813f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.312+0000 7f301ed7f640 1 -- 172.21.15.12:0/3353852810 reap_dead start 2024-09-11T01:54:41.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.312+0000 7f30207e2640 1 --2- 172.21.15.12:0/3353852810 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a2e20 0x7f301813f970 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f30080098f0 tx=0x7f30080046c0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.312+0000 7f30207e2640 1 -- 172.21.15.12:0/3353852810 shutdown_connections 2024-09-11T01:54:41.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.312+0000 7f30207e2640 1 --2- 172.21.15.12:0/3353852810 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f30180a2e20 0x7f301813f970 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.312+0000 7f30207e2640 1 -- 172.21.15.12:0/3353852810 >> 172.21.15.12:0/3353852810 conn(0x7f30180a8600 msgr2=0x7f301809e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:41.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.312+0000 7f30207e2640 1 -- 172.21.15.12:0/3353852810 shutdown_connections 2024-09-11T01:54:41.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.312+0000 7f30207e2640 1 -- 172.21.15.12:0/3353852810 wait complete. 2024-09-11T01:54:41.340 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:54:41.340 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:54:41.340 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:54:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.439+0000 7f36f4dad640 10 monclient: get_monmap_and_config 2024-09-11T01:54:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.439+0000 7f36f4dad640 10 monclient: build_initial_monmap 2024-09-11T01:54:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.439+0000 7f36f4dad640 10 monclient: monmap: 2024-09-11T01:54:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:54:41.441462+0000 2024-09-11T01:54:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:54:41.441462+0000 2024-09-11T01:54:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:54:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:54:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:54:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:54:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:54:41.443 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.440+0000 7f36f4dad640 1 Processor -- start 2024-09-11T01:54:41.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.440+0000 7f36f4dad640 1 -- start start 2024-09-11T01:54:41.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.440+0000 7f36f4dad640 10 monclient: init 2024-09-11T01:54:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.440+0000 7f36f4dad640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.440+0000 7f36f4dad640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-11T01:54:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.440+0000 7f36f4dad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f36f00a2e20 addr v1:172.21.15.71:6789/0 2024-09-11T01:54:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.440+0000 7f36f4dad640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f36f00a4970 0x7f36f006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.440+0000 7f36f4dad640 10 monclient(hunting): picked mon.noname-c con 0x7f36f00a4970 addr v2:172.21.15.71:3300/0 2024-09-11T01:54:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.440+0000 7f36f4dad640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f36f006f910 addr v1:172.21.15.12:6789/0 2024-09-11T01:54:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.440+0000 7f36f4dad640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36f006f4f0 con 0x7f36f006f910 2024-09-11T01:54:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.440+0000 7f36f4dad640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36f00a3910 con 0x7f36f00a2e20 2024-09-11T01:54:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.440+0000 7f36f4dad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.440+0000 7f36f4dad640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f36f006ef30 con 0x7f36f00a4970 2024-09-11T01:54:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.440+0000 7f36f4dad640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.440+0000 7f36f4dad640 10 monclient(hunting): authenticate will time out at 2287.883985s 2024-09-11T01:54:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.441+0000 7f36eed9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f36f006f910 0x7f36f013f430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:32774/0 (socket says 172.21.15.12:32774) 2024-09-11T01:54:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.441+0000 7f36eed9d640 1 -- 172.21.15.12:0/2347268567 learned_addr learned my addr 172.21.15.12:0/2347268567 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:54:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.441+0000 7f36e5d9b640 1 --2- 172.21.15.12:0/2347268567 >> v2:172.21.15.71:3300/0 conn(0x7f36f00a4970 0x7f36f006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.441+0000 7f36e5d9b640 10 monclient(hunting): get_auth_request con 0x7f36f00a4970 auth_method 0 2024-09-11T01:54:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.441+0000 7f36e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.441+0000 7f36e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.441+0000 7f36e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.441+0000 7f36e5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.441+0000 7f36e5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.441+0000 7f36e5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.441+0000 7f36e5d9b640 10 monclient(hunting): handle_auth_done global_id 274174 payload 274 2024-09-11T01:54:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.441+0000 7f36e5d9b640 1 -- 172.21.15.12:0/2347268567 >> v1:172.21.15.71:6789/0 conn(0x7f36f00a2e20 legacy=0x7f36f00a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:54:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.441+0000 7f36e5d9b640 1 -- 172.21.15.12:0/2347268567 >> v1:172.21.15.12:6789/0 conn(0x7f36f006f910 legacy=0x7f36f013f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:54:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.441+0000 7f36e5d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:54:41.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.441+0000 7f36e5d9b640 1 monclient: found mon.noname-c 2024-09-11T01:54:41.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.441+0000 7f36e5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:41.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.441+0000 7f36e5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:54:41.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.441+0000 7f36e5d9b640 1 -- 172.21.15.12:0/2347268567 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36f00a3ae0 con 0x7f36f00a4970 2024-09-11T01:54:41.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.444+0000 7f36e5d9b640 1 -- 172.21.15.12:0/2347268567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36f00a4970 msgr2=0x7f36f006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:41.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.444+0000 7f36e5d9b640 1 --2- 172.21.15.12:0/2347268567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36f00a4970 0x7f36f006ded0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f36d80097a0 tx=0x7f36d8029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b170a225eda5bfa server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:41.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.444+0000 7f36edd9b640 1 -- 172.21.15.12:0/2347268567 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d8037070 con 0x7f36f00a4970 2024-09-11T01:54:41.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.444+0000 7f36edd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.444+0000 7f36edd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:54:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.444+0000 7f36edd9b640 10 monclient: dump: 2024-09-11T01:54:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:41.452 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:41.452 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.444+0000 7f36edd9b640 10 monclient: _finish_auth 0 2024-09-11T01:54:41.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.444+0000 7f36edd9b640 10 monclient: _check_auth_tickets 2024-09-11T01:54:41.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.444+0000 7f36edd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:41.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.444+0000 7f36edd9b640 1 -- 172.21.15.12:0/2347268567 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36d8032640 con 0x7f36f00a4970 2024-09-11T01:54:41.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.444+0000 7f36f4dad640 5 monclient: authenticate success, global_id 274174 2024-09-11T01:54:41.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.444+0000 7f36f4dad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:54:41.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.444+0000 7f36edd9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:41.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.444+0000 7f36f4dad640 10 monclient: get_monmap_and_config success 2024-09-11T01:54:41.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.444+0000 7f36edd9b640 1 -- 172.21.15.12:0/2347268567 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d8032a10 con 0x7f36f00a4970 2024-09-11T01:54:41.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.444+0000 7f36f4dad640 4 set_mon_vals no callback set 2024-09-11T01:54:41.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.444+0000 7f36edd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:41.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.444+0000 7f36edd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:54:41.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.444+0000 7f36edd9b640 10 monclient: dump: 2024-09-11T01:54:41.453 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:41.454 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:41.454 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.454 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.454 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:41.454 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.454 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:41.454 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:41.454 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:41.454 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.445+0000 7f36f4dad640 10 monclient: shutdown 2024-09-11T01:54:41.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.445+0000 7f36f4dad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:41.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.445+0000 7f36f4dad640 1 -- 172.21.15.12:0/2347268567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36f00a4970 msgr2=0x7f36f006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.445+0000 7f36f4dad640 1 --2- 172.21.15.12:0/2347268567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36f00a4970 0x7f36f006ded0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f36d80097a0 tx=0x7f36d8029fd0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.445+0000 7f36f4dad640 1 -- 172.21.15.12:0/2347268567 shutdown_connections 2024-09-11T01:54:41.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.445+0000 7f36f4dad640 1 --2- 172.21.15.12:0/2347268567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36f00a4970 0x7f36f006ded0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.445+0000 7f36f4dad640 1 -- 172.21.15.12:0/2347268567 >> 172.21.15.12:0/2347268567 conn(0x7f36f00a8600 msgr2=0x7f36f00a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:41.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.445+0000 7f36f4dad640 1 -- 172.21.15.12:0/2347268567 shutdown_connections 2024-09-11T01:54:41.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.445+0000 7f36f4dad640 1 -- 172.21.15.12:0/2347268567 wait complete. 2024-09-11T01:54:41.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.446+0000 7f36f4dad640 10 monclient: build_initial_monmap 2024-09-11T01:54:41.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.446+0000 7f36f4dad640 10 monclient: monmap: 2024-09-11T01:54:41.455 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:41.455 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:54:41.455 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:54:41.455 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:54:41.455 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:41.455 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.456 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:54:41.456 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:54:41.456 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:54:41.456 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.446+0000 7f36f4dad640 1 Processor -- start 2024-09-11T01:54:41.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.446+0000 7f36f4dad640 1 -- start start 2024-09-11T01:54:41.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.446+0000 7f36f4dad640 10 monclient: init 2024-09-11T01:54:41.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36f4dad640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:41.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36f4dad640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:54:41.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36f4dad640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f36f006f910 0x7f36f0137140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36f4dad640 10 monclient(hunting): picked mon.noname-c con 0x7f36f006f910 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:41.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36f4dad640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36f00a2e20 0x7f36f0137680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36f4dad640 10 monclient(hunting): picked mon.noname-a con 0x7f36f00a2e20 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:54:41.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36f4dad640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36f00a4970 0x7f36f0137bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36f4dad640 10 monclient(hunting): picked mon.noname-b con 0x7f36f00a4970 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:54:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36f4dad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36f4dad640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f36f0140d60 con 0x7f36f00a2e20 2024-09-11T01:54:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36f4dad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36f4dad640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f36f006ec50 con 0x7f36f00a4970 2024-09-11T01:54:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36f4dad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36f4dad640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f36f0140ee0 con 0x7f36f006f910 2024-09-11T01:54:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36f4dad640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36f4dad640 10 monclient(hunting): authenticate will time out at 2287.890290s 2024-09-11T01:54:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36ee59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f36f006f910 0x7f36f0137140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36e5d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36f00a2e20 0x7f36f0137680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36e5d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36f00a2e20 0x7f36f0137680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43572/0 (socket says 172.21.15.12:43572) 2024-09-11T01:54:41.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36ee59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f36f006f910 0x7f36f0137140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41116/0 (socket says 172.21.15.12:41116) 2024-09-11T01:54:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36e5d9b640 1 -- 172.21.15.12:0/3041686502 learned_addr learned my addr 172.21.15.12:0/3041686502 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36e5d9b640 10 monclient(hunting): get_auth_request con 0x7f36f00a2e20 auth_method 0 2024-09-11T01:54:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36ee59c640 10 monclient(hunting): get_auth_request con 0x7f36f006f910 auth_method 0 2024-09-11T01:54:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36ee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36ee59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36ee59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36e5d9b640 1 -- 172.21.15.12:0/3041686502 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36f00a2e20 msgr2=0x7f36f0137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:54:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36e5d9b640 1 -- 172.21.15.12:0/3041686502 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36f00a2e20 msgr2=0x7f36f0137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:54:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36e5d9b640 1 --2- 172.21.15.12:0/3041686502 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36f00a2e20 0x7f36f0137680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:54:41.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36e5d9b640 1 --2- 172.21.15.12:0/3041686502 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36f00a2e20 0x7f36f0137680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:54:41.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36eed9d640 1 --2- 172.21.15.12:0/3041686502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36f00a4970 0x7f36f0137bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36ee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:41.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36ee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:41.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36ee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:41.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36eed9d640 10 monclient(hunting): get_auth_request con 0x7f36f00a4970 auth_method 0 2024-09-11T01:54:41.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36eed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:41.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36eed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:41.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36eed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:41.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36eed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:41.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36eed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:41.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36eed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:41.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36ee59c640 10 monclient(hunting): handle_auth_done global_id 284117 payload 824 2024-09-11T01:54:41.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36ee59c640 1 -- 172.21.15.12:0/3041686502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36f00a4970 msgr2=0x7f36f0137bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36ee59c640 1 --2- 172.21.15.12:0/3041686502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36f00a4970 0x7f36f0137bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36ee59c640 1 -- 172.21.15.12:0/3041686502 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36f00a2e20 msgr2=0x7f36f0137680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:54:41.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36ee59c640 1 --2- 172.21.15.12:0/3041686502 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36f00a2e20 0x7f36f0137680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36ee59c640 10 monclient: _finish_hunting 0 2024-09-11T01:54:41.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36ee59c640 1 monclient: found mon.noname-c 2024-09-11T01:54:41.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36ee59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:41.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36ee59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:54:41.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.447+0000 7f36ee59c640 1 -- 172.21.15.12:0/3041686502 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36f013abe0 con 0x7f36f006f910 2024-09-11T01:54:41.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.449+0000 7f36ee59c640 1 -- 172.21.15.12:0/3041686502 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f36f006f910 msgr2=0x7f36f0137140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:41.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.449+0000 7f36ee59c640 1 --2- 172.21.15.12:0/3041686502 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f36f006f910 0x7f36f0137140 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f36e000c410 tx=0x7f36e000c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:41.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.450+0000 7f36e77fe640 1 -- 172.21.15.12:0/3041686502 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e0017070 con 0x7f36f006f910 2024-09-11T01:54:41.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.450+0000 7f36e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:41.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.450+0000 7f36e77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:54:41.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.450+0000 7f36e77fe640 10 monclient: dump: 2024-09-11T01:54:41.464 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:41.464 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:41.464 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.464 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.464 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:41.464 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.464 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:41.464 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:41.465 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:41.465 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.450+0000 7f36e77fe640 10 monclient: _finish_auth 0 2024-09-11T01:54:41.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.450+0000 7f36e77fe640 10 monclient: _check_auth_tickets 2024-09-11T01:54:41.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.450+0000 7f36e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:41.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.450+0000 7f36e77fe640 1 -- 172.21.15.12:0/3041686502 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36e0005230 con 0x7f36f006f910 2024-09-11T01:54:41.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.450+0000 7f36f4dad640 5 monclient: authenticate success, global_id 284117 2024-09-11T01:54:41.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.450+0000 7f36f4dad640 10 monclient: _renew_subs 2024-09-11T01:54:41.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.450+0000 7f36f4dad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:41.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.450+0000 7f36f4dad640 1 -- 172.21.15.12:0/3041686502 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36f013aec0 con 0x7f36f006f910 2024-09-11T01:54:41.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.450+0000 7f36e77fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:41.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.450+0000 7f36e77fe640 1 -- 172.21.15.12:0/3041686502 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e00117a0 con 0x7f36f006f910 2024-09-11T01:54:41.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.450+0000 7f36e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:41.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.450+0000 7f36e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:41.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.450+0000 7f36e77fe640 10 monclient: dump: 2024-09-11T01:54:41.466 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:41.466 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:41.466 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.466 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.466 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:41.466 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.466 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:41.466 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:41.466 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:41.466 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.450+0000 7f36ed59a640 4 set_mon_vals no callback set 2024-09-11T01:54:41.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.450+0000 7f36f4dad640 10 monclient: _renew_subs 2024-09-11T01:54:41.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.450+0000 7f36f4dad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:41.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.450+0000 7f36f4dad640 1 -- 172.21.15.12:0/3041686502 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f36f0140800 con 0x7f36f006f910 2024-09-11T01:54:41.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.450+0000 7f36f4dad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:41.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.451+0000 7f36f4dad640 1 --2- 172.21.15.12:0/3041686502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36f01cda30 0x7f36f002d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.451+0000 7f36f4dad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:41.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.451+0000 7f36f4dad640 1 -- 172.21.15.12:0/3041686502 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f36f006a060 con 0x7f36f01cda30 2024-09-11T01:54:41.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.451+0000 7f36e77fe640 1 -- 172.21.15.12:0/3041686502 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f36e00119c0 con 0x7f36f006f910 2024-09-11T01:54:41.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.451+0000 7f36e5d9b640 1 --2- 172.21.15.12:0/3041686502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36f01cda30 0x7f36f002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.451+0000 7f36e5d9b640 10 monclient: get_auth_request con 0x7f36f01cda30 auth_method 0 2024-09-11T01:54:41.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.451+0000 7f36e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:41.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.451+0000 7f36e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:41.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.451+0000 7f36e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:41.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.451+0000 7f36e5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:41.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.451+0000 7f36e5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:41.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.451+0000 7f36e5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:41.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.451+0000 7f36e77fe640 1 --2- 172.21.15.12:0/3041686502 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f36d003e240 0x7f36d0040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.451+0000 7f36e77fe640 1 -- 172.21.15.12:0/3041686502 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(856..856 src has 676..856) ==== 4831+0+0 (secure 0 0 0) 0x7f36e0052260 con 0x7f36f006f910 2024-09-11T01:54:41.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.451+0000 7f36e5d9b640 10 monclient(hunting): handle_auth_done global_id 274180 payload 824 2024-09-11T01:54:41.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.451+0000 7f36e5d9b640 1 -- 172.21.15.12:0/3041686502 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f36f0072350 con 0x7f36f01cda30 2024-09-11T01:54:41.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.451+0000 7f36eed9d640 1 --2- 172.21.15.12:0/3041686502 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f36d003e240 0x7f36d0040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.451+0000 7f36eed9d640 10 monclient: get_auth_request con 0x7f36d003e240 auth_method 0 2024-09-11T01:54:41.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.453+0000 7f36e5d9b640 1 -- 172.21.15.12:0/3041686502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36f01cda30 msgr2=0x7f36f002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:41.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.453+0000 7f36e5d9b640 1 --2- 172.21.15.12:0/3041686502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36f01cda30 0x7f36f002d440 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f36d8000c00 tx=0x7f36d80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:41.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.454+0000 7f36e77fe640 1 -- 172.21.15.12:0/3041686502 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d8043070 con 0x7f36f01cda30 2024-09-11T01:54:41.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.454+0000 7f36e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:41.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.454+0000 7f36e77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:54:41.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.454+0000 7f36eed9d640 1 --2- 172.21.15.12:0/3041686502 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f36d003e240 0x7f36d0040700 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f36e80055a0 tx=0x7f36e8005530 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:41.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.454+0000 7f36e77fe640 10 monclient: dump: 2024-09-11T01:54:41.469 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:41.469 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:41.469 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.469 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.469 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:41.469 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.469 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:41.469 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:41.469 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:41.469 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.454+0000 7f36e77fe640 1 -- 172.21.15.12:0/3041686502 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f36d803d650 con 0x7f36f01cda30 2024-09-11T01:54:41.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.454+0000 7f36e77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:41.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.454+0000 7f36e77fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:54:41.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.454+0000 7f36e77fe640 1 -- 172.21.15.12:0/3041686502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36f01cda30 msgr2=0x7f36f002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.454+0000 7f36e77fe640 1 --2- 172.21.15.12:0/3041686502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36f01cda30 0x7f36f002d440 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f36d8000c00 tx=0x7f36d80046c0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.454+0000 7f36e77fe640 1 -- 172.21.15.12:0/3041686502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36f01cda30 msgr2=0x7f36f002d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:54:41.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.454+0000 7f36e77fe640 1 --2- 172.21.15.12:0/3041686502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36f01cda30 0x7f36f002d440 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f36d8000c00 tx=0x7f36d80046c0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.460+0000 7f36f4dad640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:54:41.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.460+0000 7f36f4dad640 1 --2- 172.21.15.12:0/3041686502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36f01cde10 0x7f36f002d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.460+0000 7f36f4dad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:41.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.460+0000 7f36f4dad640 1 -- 172.21.15.12:0/3041686502 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f36f006ded0 con 0x7f36f01cde10 2024-09-11T01:54:41.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.461+0000 7f36e5d9b640 1 --2- 172.21.15.12:0/3041686502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36f01cde10 0x7f36f002d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.461+0000 7f36e5d9b640 10 monclient: get_auth_request con 0x7f36f01cde10 auth_method 0 2024-09-11T01:54:41.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.461+0000 7f36e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:41.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.461+0000 7f36e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:41.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.461+0000 7f36e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:41.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.461+0000 7f36e5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:41.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.461+0000 7f36e5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:41.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.461+0000 7f36e5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:41.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.461+0000 7f36e5d9b640 10 monclient(hunting): handle_auth_done global_id 274183 payload 824 2024-09-11T01:54:41.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.461+0000 7f36e5d9b640 1 -- 172.21.15.12:0/3041686502 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f36f00a0f30 con 0x7f36f01cde10 2024-09-11T01:54:41.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.463+0000 7f36e5d9b640 1 -- 172.21.15.12:0/3041686502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36f01cde10 msgr2=0x7f36f002d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:41.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.463+0000 7f36e5d9b640 1 --2- 172.21.15.12:0/3041686502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36f01cde10 0x7f36f002d980 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f36d8042000 tx=0x7f36d8008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:41.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.463+0000 7f36e77fe640 1 -- 172.21.15.12:0/3041686502 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d8037040 con 0x7f36f01cde10 2024-09-11T01:54:41.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.464+0000 7f36e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:41.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.464+0000 7f36e77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:54:41.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.464+0000 7f36e77fe640 10 monclient: dump: 2024-09-11T01:54:41.472 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:41.472 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:41.473 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.473 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.473 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:41.473 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.473 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:41.473 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:41.473 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:41.473 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.464+0000 7f36e77fe640 1 -- 172.21.15.12:0/3041686502 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7f36d8003820 con 0x7f36f01cde10 2024-09-11T01:54:41.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.464+0000 7f36e77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:54:41.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.464+0000 7f36e77fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:54:41.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.464+0000 7f36e77fe640 1 -- 172.21.15.12:0/3041686502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36f01cde10 msgr2=0x7f36f002d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.464+0000 7f36e77fe640 1 --2- 172.21.15.12:0/3041686502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36f01cde10 0x7f36f002d980 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f36d8042000 tx=0x7f36d8008030 comp rx=0 tx=0).stop 2024-09-11T01:54:41.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.464+0000 7f36e77fe640 1 -- 172.21.15.12:0/3041686502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36f01cde10 msgr2=0x7f36f002d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:54:41.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.464+0000 7f36e77fe640 1 --2- 172.21.15.12:0/3041686502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36f01cde10 0x7f36f002d980 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f36d8042000 tx=0x7f36d8008030 comp rx=0 tx=0).stop 2024-09-11T01:54:41.474 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:54:41.474 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:54:41.474 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:54:41.474 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:54:41.474 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 232, 2024-09-11T01:54:41.474 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 71283, 2024-09-11T01:54:41.474 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:54:41.474 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:54:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:54:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:54:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:54:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:54:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:54:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:54:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:41.475 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:54:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:54:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:41.476 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:54:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:54:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:54:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:54:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:54:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:54:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:54:41.477 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:54:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:54:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:54:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:54:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:54:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:54:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:54:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:54:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:54:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:54:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:41.478 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:54:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:54:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:54:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:54:41.479 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:54:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:54:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:41.480 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:54:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:54:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:54:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:54:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:41.481 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:54:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:54:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:54:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:54:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:54:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:54:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:54:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.483 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:54:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:54:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:54:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:41.484 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:54:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:54:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:54:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:54:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:54:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:41.485 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:54:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:54:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:54:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:54:41.486 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:41.487 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:41.487 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:41.487 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:54:41.487 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:54:41.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.464+0000 7f36f4dad640 1 -- 172.21.15.12:0/3041686502 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f36d003e240 msgr2=0x7f36d0040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.464+0000 7f36f4dad640 1 --2- 172.21.15.12:0/3041686502 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f36d003e240 0x7f36d0040700 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f36e80055a0 tx=0x7f36e8005530 comp rx=0 tx=0).stop 2024-09-11T01:54:41.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.464+0000 7f36f4dad640 10 monclient: shutdown 2024-09-11T01:54:41.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.464+0000 7f36f4dad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:41.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.464+0000 7f36f4dad640 1 -- 172.21.15.12:0/3041686502 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f36f006f910 msgr2=0x7f36f0137140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.464+0000 7f36f4dad640 1 --2- 172.21.15.12:0/3041686502 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f36f006f910 0x7f36f0137140 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f36e000c410 tx=0x7f36e000c8d0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.464+0000 7f36eed9d640 1 -- 172.21.15.12:0/3041686502 reap_dead start 2024-09-11T01:54:41.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.464+0000 7f36eed9d640 1 -- 172.21.15.12:0/3041686502 reap_dead start 2024-09-11T01:54:41.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.464+0000 7f36f4dad640 1 -- 172.21.15.12:0/3041686502 shutdown_connections 2024-09-11T01:54:41.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.464+0000 7f36f4dad640 1 -- 172.21.15.12:0/3041686502 >> 172.21.15.12:0/3041686502 conn(0x7f36f00a8600 msgr2=0x7f36f009e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:41.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.464+0000 7f36f4dad640 1 -- 172.21.15.12:0/3041686502 shutdown_connections 2024-09-11T01:54:41.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.464+0000 7f36f4dad640 1 -- 172.21.15.12:0/3041686502 wait complete. 2024-09-11T01:54:41.489 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:54:41.489 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:54:41.489 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:54:41.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.587+0000 7fcc1c639640 10 monclient: get_monmap_and_config 2024-09-11T01:54:41.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.587+0000 7fcc1c639640 10 monclient: build_initial_monmap 2024-09-11T01:54:41.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.587+0000 7fcc1c639640 10 monclient: monmap: 2024-09-11T01:54:41.590 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:41.591 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:41.591 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:54:41.589645+0000 2024-09-11T01:54:41.591 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:54:41.589645+0000 2024-09-11T01:54:41.591 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:41.591 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.591 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:54:41.591 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:54:41.591 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:54:41.591 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:54:41.591 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:54:41.591 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.588+0000 7fcc1c639640 1 Processor -- start 2024-09-11T01:54:41.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.588+0000 7fcc1c639640 1 -- start start 2024-09-11T01:54:41.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.588+0000 7fcc1c639640 10 monclient: init 2024-09-11T01:54:41.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1c639640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:41.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1c639640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-11T01:54:41.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1c639640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcc141381c0 0x7fcc141385a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1c639640 10 monclient(hunting): picked mon.noname-c con 0x7fcc141381c0 addr v2:172.21.15.71:3300/0 2024-09-11T01:54:41.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1c639640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcc140a6aa0 addr v1:172.21.15.71:6789/0 2024-09-11T01:54:41.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1c639640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc1413d000 0x7fcc1413f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1c639640 10 monclient(hunting): picked mon.noname-b con 0x7fcc1413d000 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:41.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1c639640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc1409c310 con 0x7fcc140a6aa0 2024-09-11T01:54:41.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1c639640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:41.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1c639640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fcc1406fc90 con 0x7fcc141381c0 2024-09-11T01:54:41.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1c639640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:41.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1c639640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcc1409c5f0 con 0x7fcc1413d000 2024-09-11T01:54:41.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1c639640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:41.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1c639640 10 monclient(hunting): authenticate will time out at 2288.032266s 2024-09-11T01:54:41.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1abd6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc1413d000 0x7fcc1413f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1abd6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc1413d000 0x7fcc1413f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41118/0 (socket says 172.21.15.12:41118) 2024-09-11T01:54:41.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1abd6640 1 -- 172.21.15.12:0/2714053880 learned_addr learned my addr 172.21.15.12:0/2714053880 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:41.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1abd6640 10 monclient(hunting): get_auth_request con 0x7fcc1413d000 auth_method 0 2024-09-11T01:54:41.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1abd6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:41.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1abd6640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:41.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1abd6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:41.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1abd6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:41.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1abd6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:41.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1abd6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:41.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1a3d5640 1 --2- 172.21.15.12:0/2714053880 >> v2:172.21.15.71:3300/0 conn(0x7fcc141381c0 0x7fcc141385a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1a3d5640 10 monclient(hunting): get_auth_request con 0x7fcc141381c0 auth_method 0 2024-09-11T01:54:41.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1a3d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:41.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1a3d5640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:41.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1a3d5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:41.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1abd6640 10 monclient(hunting): handle_auth_done global_id 284120 payload 274 2024-09-11T01:54:41.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1abd6640 1 -- 172.21.15.12:0/2714053880 >> v2:172.21.15.71:3300/0 conn(0x7fcc141381c0 msgr2=0x7fcc141385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:54:41.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1abd6640 1 --2- 172.21.15.12:0/2714053880 >> v2:172.21.15.71:3300/0 conn(0x7fcc141381c0 0x7fcc141385a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1abd6640 1 -- 172.21.15.12:0/2714053880 >> v1:172.21.15.71:6789/0 conn(0x7fcc140a6aa0 legacy=0x7fcc1409cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:54:41.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1abd6640 10 monclient: _finish_hunting 0 2024-09-11T01:54:41.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1abd6640 1 monclient: found mon.noname-b 2024-09-11T01:54:41.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1abd6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:41.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1abd6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:54:41.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.589+0000 7fcc1abd6640 1 -- 172.21.15.12:0/2714053880 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc1409c8d0 con 0x7fcc1413d000 2024-09-11T01:54:41.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.592+0000 7fcc1abd6640 1 -- 172.21.15.12:0/2714053880 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc1413d000 msgr2=0x7fcc1413f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:41.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.592+0000 7fcc1abd6640 1 --2- 172.21.15.12:0/2714053880 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc1413d000 0x7fcc1413f3f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c009e50 tx=0x7fcc0c02a550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b3ae509c23b12f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:41.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.592+0000 7fcc193d3640 1 -- 172.21.15.12:0/2714053880 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c036070 con 0x7fcc1413d000 2024-09-11T01:54:41.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.592+0000 7fcc193d3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:41.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.592+0000 7fcc193d3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:54:41.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.592+0000 7fcc193d3640 10 monclient: dump: 2024-09-11T01:54:41.604 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:41.604 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:41.604 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.604 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.604 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:41.604 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.604 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:41.604 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:41.605 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:41.605 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.592+0000 7fcc193d3640 10 monclient: _finish_auth 0 2024-09-11T01:54:41.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.592+0000 7fcc193d3640 10 monclient: _check_auth_tickets 2024-09-11T01:54:41.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.592+0000 7fcc193d3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:41.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.592+0000 7fcc1c639640 5 monclient: authenticate success, global_id 284120 2024-09-11T01:54:41.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.592+0000 7fcc193d3640 1 -- 172.21.15.12:0/2714053880 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc0c007800 con 0x7fcc1413d000 2024-09-11T01:54:41.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.592+0000 7fcc1c639640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:54:41.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.592+0000 7fcc193d3640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:41.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.592+0000 7fcc1c639640 10 monclient: get_monmap_and_config success 2024-09-11T01:54:41.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.592+0000 7fcc1c639640 4 set_mon_vals no callback set 2024-09-11T01:54:41.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.592+0000 7fcc193d3640 1 -- 172.21.15.12:0/2714053880 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c007bd0 con 0x7fcc1413d000 2024-09-11T01:54:41.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.592+0000 7fcc193d3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:41.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.592+0000 7fcc193d3640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:41.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.592+0000 7fcc193d3640 10 monclient: dump: 2024-09-11T01:54:41.606 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:41.606 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:41.606 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.606 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.606 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:41.606 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.606 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:41.606 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:41.606 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:41.606 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.593+0000 7fcc1c639640 10 monclient: shutdown 2024-09-11T01:54:41.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.593+0000 7fcc1c639640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:41.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.593+0000 7fcc1c639640 1 -- 172.21.15.12:0/2714053880 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc1413d000 msgr2=0x7fcc1413f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.593+0000 7fcc1c639640 1 --2- 172.21.15.12:0/2714053880 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc1413d000 0x7fcc1413f3f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c009e50 tx=0x7fcc0c02a550 comp rx=0 tx=0).stop 2024-09-11T01:54:41.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.593+0000 7fcc1c639640 1 -- 172.21.15.12:0/2714053880 shutdown_connections 2024-09-11T01:54:41.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.593+0000 7fcc1c639640 1 --2- 172.21.15.12:0/2714053880 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc1413d000 0x7fcc1413f3f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.593+0000 7fcc1c639640 1 --2- 172.21.15.12:0/2714053880 >> v2:172.21.15.71:3300/0 conn(0x7fcc141381c0 0x7fcc141385a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.593+0000 7fcc1c639640 1 -- 172.21.15.12:0/2714053880 >> 172.21.15.12:0/2714053880 conn(0x7fcc140a7440 msgr2=0x7fcc140a7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:41.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.593+0000 7fcc1c639640 1 -- 172.21.15.12:0/2714053880 shutdown_connections 2024-09-11T01:54:41.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.593+0000 7fcc1c639640 1 -- 172.21.15.12:0/2714053880 wait complete. 2024-09-11T01:54:41.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.593+0000 7fcc1c639640 10 monclient: build_initial_monmap 2024-09-11T01:54:41.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.593+0000 7fcc1c639640 10 monclient: monmap: 2024-09-11T01:54:41.607 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:41.607 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:54:41.607 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:54:41.608 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:54:41.608 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:41.608 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.608 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:54:41.608 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:54:41.608 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:54:41.608 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1c639640 1 Processor -- start 2024-09-11T01:54:41.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1c639640 1 -- start start 2024-09-11T01:54:41.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1c639640 10 monclient: init 2024-09-11T01:54:41.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1c639640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:41.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1c639640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:54:41.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1c639640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc140a6aa0 0x7fcc14135100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1c639640 10 monclient(hunting): picked mon.noname-c con 0x7fcc140a6aa0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:41.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1c639640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc141381c0 0x7fcc14135640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1c639640 10 monclient(hunting): picked mon.noname-a con 0x7fcc141381c0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:54:41.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1c639640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc1413d000 0x7fcc14138dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1c639640 10 monclient(hunting): picked mon.noname-b con 0x7fcc1413d000 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:54:41.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1c639640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:41.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1c639640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1413fab0 con 0x7fcc141381c0 2024-09-11T01:54:41.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1c639640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:41.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1c639640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1413f930 con 0x7fcc1413d000 2024-09-11T01:54:41.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1c639640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:41.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1c639640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcc1413fc30 con 0x7fcc140a6aa0 2024-09-11T01:54:41.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1c639640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:41.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1c639640 10 monclient(hunting): authenticate will time out at 2288.037674s 2024-09-11T01:54:41.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1a3d5640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc140a6aa0 0x7fcc14135100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1a3d5640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc140a6aa0 0x7fcc14135100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41130/0 (socket says 172.21.15.12:41130) 2024-09-11T01:54:41.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1a3d5640 1 -- 172.21.15.12:0/1972352311 learned_addr learned my addr 172.21.15.12:0/1972352311 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:41.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1a3d5640 10 monclient(hunting): get_auth_request con 0x7fcc140a6aa0 auth_method 0 2024-09-11T01:54:41.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1a3d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:41.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1a3d5640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:41.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1a3d5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:41.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc19bd4640 1 --2- 172.21.15.12:0/1972352311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc141381c0 0x7fcc14135640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc19bd4640 10 monclient(hunting): get_auth_request con 0x7fcc141381c0 auth_method 0 2024-09-11T01:54:41.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc19bd4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:41.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc19bd4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:41.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc19bd4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:41.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1a3d5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:41.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1a3d5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:41.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1a3d5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:41.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc19bd4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:41.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc19bd4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:41.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc19bd4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:41.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1abd6640 1 --2- 172.21.15.12:0/1972352311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc1413d000 0x7fcc14138dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1abd6640 10 monclient(hunting): get_auth_request con 0x7fcc1413d000 auth_method 0 2024-09-11T01:54:41.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1abd6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:41.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1abd6640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:41.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.594+0000 7fcc1abd6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:41.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.595+0000 7fcc1a3d5640 10 monclient(hunting): handle_auth_done global_id 284123 payload 824 2024-09-11T01:54:41.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.595+0000 7fcc1a3d5640 1 -- 172.21.15.12:0/1972352311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc1413d000 msgr2=0x7fcc14138dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.595+0000 7fcc1a3d5640 1 --2- 172.21.15.12:0/1972352311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc1413d000 0x7fcc14138dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.595+0000 7fcc1a3d5640 1 -- 172.21.15.12:0/1972352311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc141381c0 msgr2=0x7fcc14135640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.595+0000 7fcc1a3d5640 1 --2- 172.21.15.12:0/1972352311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc141381c0 0x7fcc14135640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.595+0000 7fcc1a3d5640 10 monclient: _finish_hunting 0 2024-09-11T01:54:41.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.595+0000 7fcc1a3d5640 1 monclient: found mon.noname-c 2024-09-11T01:54:41.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.595+0000 7fcc1a3d5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:41.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.595+0000 7fcc1a3d5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:54:41.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.595+0000 7fcc1a3d5640 1 -- 172.21.15.12:0/1972352311 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc140a1710 con 0x7fcc140a6aa0 2024-09-11T01:54:41.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.595+0000 7fcc1abd6640 1 --2- 172.21.15.12:0/1972352311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc1413d000 0x7fcc14138dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:54:41.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.595+0000 7fcc19bd4640 1 --2- 172.21.15.12:0/1972352311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc141381c0 0x7fcc14135640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:54:41.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.597+0000 7fcc1a3d5640 1 -- 172.21.15.12:0/1972352311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc140a6aa0 msgr2=0x7fcc14135100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:41.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.597+0000 7fcc1a3d5640 1 --2- 172.21.15.12:0/1972352311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc140a6aa0 0x7fcc14135100 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000d830 tx=0x7fcc1000dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:41.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.597+0000 7fcc0affd640 1 -- 172.21.15.12:0/1972352311 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1000b8d0 con 0x7fcc140a6aa0 2024-09-11T01:54:41.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.597+0000 7fcc0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:41.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.597+0000 7fcc0affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:54:41.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.597+0000 7fcc0affd640 10 monclient: dump: 2024-09-11T01:54:41.613 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:41.613 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:41.613 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.613 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.613 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:41.613 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.613 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:41.613 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:41.613 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:41.613 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.597+0000 7fcc0affd640 10 monclient: _finish_auth 0 2024-09-11T01:54:41.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.597+0000 7fcc0affd640 10 monclient: _check_auth_tickets 2024-09-11T01:54:41.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.597+0000 7fcc0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:41.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.597+0000 7fcc1c639640 5 monclient: authenticate success, global_id 284123 2024-09-11T01:54:41.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.597+0000 7fcc0affd640 1 -- 172.21.15.12:0/1972352311 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc10019070 con 0x7fcc140a6aa0 2024-09-11T01:54:41.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.597+0000 7fcc0affd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:41.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.597+0000 7fcc1c639640 10 monclient: _renew_subs 2024-09-11T01:54:41.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.597+0000 7fcc0affd640 1 -- 172.21.15.12:0/1972352311 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1000bc90 con 0x7fcc140a6aa0 2024-09-11T01:54:41.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.597+0000 7fcc1c639640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:41.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.597+0000 7fcc1c639640 1 -- 172.21.15.12:0/1972352311 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc1413ca30 con 0x7fcc140a6aa0 2024-09-11T01:54:41.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.597+0000 7fcc0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:41.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.597+0000 7fcc0affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:41.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.597+0000 7fcc0affd640 10 monclient: dump: 2024-09-11T01:54:41.614 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:41.615 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:41.615 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.615 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:41.615 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:41.615 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:41.615 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:41.615 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:41.615 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:41.615 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:41.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.597+0000 7fcc18bd2640 4 set_mon_vals no callback set 2024-09-11T01:54:41.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.597+0000 7fcc1c639640 10 monclient: _renew_subs 2024-09-11T01:54:41.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.597+0000 7fcc1c639640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:41.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.597+0000 7fcc1c639640 1 -- 172.21.15.12:0/1972352311 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc1413b830 con 0x7fcc140a6aa0 2024-09-11T01:54:41.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.598+0000 7fcc0affd640 1 -- 172.21.15.12:0/1972352311 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fcc10013070 con 0x7fcc140a6aa0 2024-09-11T01:54:41.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.598+0000 7fcc1c639640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:41.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.598+0000 7fcc1c639640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:41.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.598+0000 7fcc1c639640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:41.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.598+0000 7fcc1c639640 1 -- 172.21.15.12:0/1972352311 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc141385a0 con 0x7fcc140a6aa0 2024-09-11T01:54:41.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.598+0000 7fcc0affd640 1 --2- 172.21.15.12:0/1972352311 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcbf803e260 0x7fcbf8040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:41.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.598+0000 7fcc0affd640 1 -- 172.21.15.12:0/1972352311 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(856..856 src has 676..856) ==== 4831+0+0 (secure 0 0 0) 0x7fcc10052e90 con 0x7fcc140a6aa0 2024-09-11T01:54:41.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.599+0000 7fcc19bd4640 1 --2- 172.21.15.12:0/1972352311 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcbf803e260 0x7fcbf8040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:41.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.599+0000 7fcc19bd4640 10 monclient: get_auth_request con 0x7fcbf803e260 auth_method 0 2024-09-11T01:54:41.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.601+0000 7fcc19bd4640 1 --2- 172.21.15.12:0/1972352311 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcbf803e260 0x7fcbf8040720 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fcc1413a720 tx=0x7fcc04007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:41.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.603+0000 7fcc0affd640 1 -- 172.21.15.12:0/1972352311 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fcc10051860 con 0x7fcc140a6aa0 2024-09-11T01:54:41.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.603+0000 7fcc0affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:41.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.603+0000 7fcc0affd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:54:41.749 INFO:tasks.rados.rados.0.smithi071.stdout:3757: done (5 left) 2024-09-11T01:54:41.749 INFO:tasks.rados.rados.0.smithi071.stdout:3758: snap_remove snap 284 2024-09-11T01:54:41.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.872+0000 7fcc1c639640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:54:41.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.872+0000 7fcc1c639640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:54:41.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.872+0000 7fcc1c639640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:41.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.872+0000 7fcc1c639640 1 -- 172.21.15.12:0/1972352311 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fcc14069840 con 0x7fcc140a6aa0 2024-09-11T01:54:41.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.873+0000 7fcc0affd640 1 -- 172.21.15.12:0/1972352311 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fcc1001ebc0 con 0x7fcc140a6aa0 2024-09-11T01:54:41.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.873+0000 7fcc0affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:54:41.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.873+0000 7fcc0affd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:54:41.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.873+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:41.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.873+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=1,health=11,kv=10,logm=48} crc {auth=1835580864,config=1747058506,health=4012094232,kv=287228308,logm=435699220}) 2024-09-11T01:54:41.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.873+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=30,config=2,health=12,kv=12,logm=44} crc {auth=36242455,config=2586702516,health=425501344,kv=3263697420,logm=3978039824}) 2024-09-11T01:54:41.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.873+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:41.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.873+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=1,health=11,kv=10,logm=48} crc {auth=1835580864,config=1747058506,health=4012094232,kv=287228308,logm=435699220}) 2024-09-11T01:54:41.877 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.873+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=31,config=2,health=11,kv=12,logm=44} crc {auth=3792016265,config=2586702516,health=4012094232,kv=3263697420,logm=1022180912}) 2024-09-11T01:54:41.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.874+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:41.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.874+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=954278697}) 2024-09-11T01:54:41.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.874+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4209206913}) 2024-09-11T01:54:41.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.874+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:41.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.874+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=954278697}) 2024-09-11T01:54:41.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.874+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=908690314}) 2024-09-11T01:54:41.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.875+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:41.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.875+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=392668671}) 2024-09-11T01:54:41.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.875+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=739242322}) 2024-09-11T01:54:41.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.875+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:41.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.875+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=392668671}) 2024-09-11T01:54:41.879 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.875+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=520091100}) 2024-09-11T01:54:41.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.875+0000 7fcc1c639640 1 -- 172.21.15.12:0/1972352311 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcbf803e260 msgr2=0x7fcbf8040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.875+0000 7fcc1c639640 1 --2- 172.21.15.12:0/1972352311 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcbf803e260 0x7fcbf8040720 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fcc1413a720 tx=0x7fcc04007e30 comp rx=0 tx=0).stop 2024-09-11T01:54:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.875+0000 7fcc1c639640 10 monclient: shutdown 2024-09-11T01:54:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.875+0000 7fcc1c639640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.875+0000 7fcc1c639640 1 -- 172.21.15.12:0/1972352311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc140a6aa0 msgr2=0x7fcc14135100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.875+0000 7fcc1c639640 1 --2- 172.21.15.12:0/1972352311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc140a6aa0 0x7fcc14135100 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000d830 tx=0x7fcc1000dcf0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.875+0000 7fcc1c639640 1 -- 172.21.15.12:0/1972352311 shutdown_connections 2024-09-11T01:54:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.875+0000 7fcc1c639640 1 --2- 172.21.15.12:0/1972352311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc1413d000 0x7fcc14138dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.875+0000 7fcc1c639640 1 --2- 172.21.15.12:0/1972352311 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcbf803e260 0x7fcbf8040720 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.875+0000 7fcc1c639640 1 --2- 172.21.15.12:0/1972352311 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc141381c0 0x7fcc14135640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.875+0000 7fcc1c639640 1 --2- 172.21.15.12:0/1972352311 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc140a6aa0 0x7fcc14135100 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:41.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.875+0000 7fcc1c639640 1 -- 172.21.15.12:0/1972352311 >> 172.21.15.12:0/1972352311 conn(0x7fcc140a7440 msgr2=0x7fcc140a9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:41.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.876+0000 7fcc1c639640 1 -- 172.21.15.12:0/1972352311 shutdown_connections 2024-09-11T01:54:41.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:41.876+0000 7fcc1c639640 1 -- 172.21.15.12:0/1972352311 wait complete. 2024-09-11T01:54:41.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.876+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:41.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.876+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=361901621}) 2024-09-11T01:54:41.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.876+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3557677073}) 2024-09-11T01:54:41.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.876+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:41.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.876+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=361901621}) 2024-09-11T01:54:41.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.876+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4196101507}) 2024-09-11T01:54:41.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.877+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:41.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.878+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4177377598}) 2024-09-11T01:54:41.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.878+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2916807357}) 2024-09-11T01:54:41.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.878+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.878+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4177377598}) 2024-09-11T01:54:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.878+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2172834658}) 2024-09-11T01:54:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.878+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.878+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=615397565}) 2024-09-11T01:54:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.878+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3801322664}) 2024-09-11T01:54:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.878+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.878+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=615397565}) 2024-09-11T01:54:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.878+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3908241652}) 2024-09-11T01:54:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.879+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.879+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=521939211}) 2024-09-11T01:54:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.879+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=979329607}) 2024-09-11T01:54:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.879+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.879+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=521939211}) 2024-09-11T01:54:41.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.879+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=473965498}) 2024-09-11T01:54:41.885 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.880+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:41.885 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.880+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=284237488}) 2024-09-11T01:54:41.885 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.880+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=675845654}) 2024-09-11T01:54:41.885 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.880+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:41.885 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.880+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=284237488}) 2024-09-11T01:54:41.885 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.880+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=59306213}) 2024-09-11T01:54:41.885 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.881+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:41.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.881+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3940921698}) 2024-09-11T01:54:41.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.881+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3254596839}) 2024-09-11T01:54:41.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.881+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:41.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.881+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3940921698}) 2024-09-11T01:54:41.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:41.881+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=101059107}) 2024-09-11T01:54:41.889 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:54:42.101 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.098+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.101 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.098+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2576062664}) 2024-09-11T01:54:42.101 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.098+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=268626660}) 2024-09-11T01:54:42.101 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.098+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.101 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.098+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2576062664}) 2024-09-11T01:54:42.101 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.098+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3897568338}) 2024-09-11T01:54:42.103 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.099+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.103 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.099+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3041959447}) 2024-09-11T01:54:42.103 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.099+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3827772078}) 2024-09-11T01:54:42.103 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.099+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.103 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.099+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3041959447}) 2024-09-11T01:54:42.103 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.099+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1199552409}) 2024-09-11T01:54:42.103 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.100+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.103 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.100+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1733234293}) 2024-09-11T01:54:42.103 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.100+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1558527308}) 2024-09-11T01:54:42.103 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.100+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.103 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.100+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1733234293}) 2024-09-11T01:54:42.104 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.100+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=643719691}) 2024-09-11T01:54:42.105 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.101+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.105 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.101+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=309575906}) 2024-09-11T01:54:42.105 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.101+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1845934697}) 2024-09-11T01:54:42.105 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.101+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.105 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.101+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=309575906}) 2024-09-11T01:54:42.105 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.101+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1475660680}) 2024-09-11T01:54:42.106 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.102+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.106 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.102+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1794963880}) 2024-09-11T01:54:42.106 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.102+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=218708573}) 2024-09-11T01:54:42.106 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.102+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.106 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.102+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1794963880}) 2024-09-11T01:54:42.106 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.102+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=572820387}) 2024-09-11T01:54:42.106 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.102+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.106 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.102+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068758280}) 2024-09-11T01:54:42.106 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.102+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3623742532}) 2024-09-11T01:54:42.106 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.102+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.106 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.102+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068758280}) 2024-09-11T01:54:42.106 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.102+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2932057255}) 2024-09-11T01:54:42.108 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.103+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.108 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.103+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4201200625}) 2024-09-11T01:54:42.109 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.103+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4198071625}) 2024-09-11T01:54:42.109 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.103+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.109 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.103+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4201200625}) 2024-09-11T01:54:42.109 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.103+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=981037311}) 2024-09-11T01:54:42.109 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.104+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.109 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.104+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2555865385}) 2024-09-11T01:54:42.109 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.104+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3484821988}) 2024-09-11T01:54:42.109 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.104+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.109 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.104+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2555865385}) 2024-09-11T01:54:42.109 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.104+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=987285334}) 2024-09-11T01:54:42.109 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.105+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.109 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.105+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3157057651}) 2024-09-11T01:54:42.110 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.105+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1718700956}) 2024-09-11T01:54:42.110 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.105+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.110 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.105+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3157057651}) 2024-09-11T01:54:42.110 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.105+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3431111555}) 2024-09-11T01:54:42.110 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.106+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.110 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.106+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4082411263}) 2024-09-11T01:54:42.110 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.106+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1123715764}) 2024-09-11T01:54:42.110 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.106+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.110 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.106+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4082411263}) 2024-09-11T01:54:42.110 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.106+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2086132274}) 2024-09-11T01:54:42.113 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.106+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.113 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.106+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2923636812}) 2024-09-11T01:54:42.113 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.106+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3432542203}) 2024-09-11T01:54:42.113 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.106+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.113 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.106+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2923636812}) 2024-09-11T01:54:42.113 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.106+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2400901444}) 2024-09-11T01:54:42.114 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.107+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.114 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.107+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1413361676}) 2024-09-11T01:54:42.114 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.107+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=445155936}) 2024-09-11T01:54:42.114 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.107+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.114 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.107+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1413361676}) 2024-09-11T01:54:42.114 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.107+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1499397981}) 2024-09-11T01:54:42.114 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.108+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.114 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.108+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3472459503}) 2024-09-11T01:54:42.114 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.108+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4266802966}) 2024-09-11T01:54:42.114 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.108+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.114 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.108+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3472459503}) 2024-09-11T01:54:42.114 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.108+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=841716597}) 2024-09-11T01:54:42.115 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.109+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.115 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.109+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3415932026}) 2024-09-11T01:54:42.115 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.109+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2309878504}) 2024-09-11T01:54:42.115 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.109+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.115 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.109+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3415932026}) 2024-09-11T01:54:42.115 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.109+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4192446903}) 2024-09-11T01:54:42.115 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.110+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.115 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.110+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2019168042}) 2024-09-11T01:54:42.115 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.110+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3737810454}) 2024-09-11T01:54:42.115 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.110+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.115 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.110+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2019168042}) 2024-09-11T01:54:42.115 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.110+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1478443102}) 2024-09-11T01:54:42.116 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.110+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.116 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.110+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1973392484}) 2024-09-11T01:54:42.116 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.110+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2790048962}) 2024-09-11T01:54:42.116 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.110+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.116 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.110+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1973392484}) 2024-09-11T01:54:42.116 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.110+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1358869733}) 2024-09-11T01:54:42.123 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.111+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.124 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.111+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2524007993}) 2024-09-11T01:54:42.124 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.111+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=715143723}) 2024-09-11T01:54:42.124 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.111+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.124 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.111+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2524007993}) 2024-09-11T01:54:42.124 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.111+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1433711398}) 2024-09-11T01:54:42.124 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.112+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.124 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.112+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4201526284}) 2024-09-11T01:54:42.124 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.112+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1737972595}) 2024-09-11T01:54:42.124 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.112+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.124 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.112+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4201526284}) 2024-09-11T01:54:42.124 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.112+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1612196661}) 2024-09-11T01:54:42.124 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.113+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.125 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.113+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=819560207}) 2024-09-11T01:54:42.125 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.113+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2104391269}) 2024-09-11T01:54:42.125 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.113+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.125 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.113+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=819560207}) 2024-09-11T01:54:42.125 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.113+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=818163852}) 2024-09-11T01:54:42.125 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.113+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.125 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.113+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2971547780}) 2024-09-11T01:54:42.125 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.113+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2999276467}) 2024-09-11T01:54:42.125 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.113+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.125 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.114+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2971547780}) 2024-09-11T01:54:42.125 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.114+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=713527154}) 2024-09-11T01:54:42.125 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.114+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.126 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.114+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2862094570}) 2024-09-11T01:54:42.126 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.114+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=327960618}) 2024-09-11T01:54:42.126 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.114+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.126 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.114+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2862094570}) 2024-09-11T01:54:42.126 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.114+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3734172703}) 2024-09-11T01:54:42.126 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.115+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.126 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.115+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1339575045}) 2024-09-11T01:54:42.126 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.115+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3559344765}) 2024-09-11T01:54:42.126 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.115+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.126 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.115+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1339575045}) 2024-09-11T01:54:42.126 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.115+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2371410482}) 2024-09-11T01:54:42.126 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.116+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.126 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.116+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=250071599}) 2024-09-11T01:54:42.127 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.116+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=608756654}) 2024-09-11T01:54:42.127 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.116+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.127 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.116+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=250071599}) 2024-09-11T01:54:42.127 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.116+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=609841152}) 2024-09-11T01:54:42.127 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.117+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.127 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.117+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3844817412}) 2024-09-11T01:54:42.127 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.117+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1828212123}) 2024-09-11T01:54:42.127 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.117+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.127 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.117+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3844817412}) 2024-09-11T01:54:42.127 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.117+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3323905189}) 2024-09-11T01:54:42.127 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.117+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.127 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.117+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3913119739}) 2024-09-11T01:54:42.128 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.117+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3182781993}) 2024-09-11T01:54:42.128 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.117+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.128 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.117+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3913119739}) 2024-09-11T01:54:42.128 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.117+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2157316999}) 2024-09-11T01:54:42.128 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.118+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.128 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.118+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254210901}) 2024-09-11T01:54:42.128 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.118+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4084507070}) 2024-09-11T01:54:42.128 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.118+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.128 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.118+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254210901}) 2024-09-11T01:54:42.128 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.118+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1886501357}) 2024-09-11T01:54:42.128 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.119+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.128 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.119+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3767549077}) 2024-09-11T01:54:42.128 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.119+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2737826986}) 2024-09-11T01:54:42.129 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.119+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.129 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.119+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3767549077}) 2024-09-11T01:54:42.129 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.119+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2884484905}) 2024-09-11T01:54:42.129 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.120+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.129 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.120+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3760520876}) 2024-09-11T01:54:42.129 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.120+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2381088794}) 2024-09-11T01:54:42.129 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.120+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.129 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.120+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3760520876}) 2024-09-11T01:54:42.129 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.120+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1711238001}) 2024-09-11T01:54:42.129 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.120+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.129 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.120+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=740759066}) 2024-09-11T01:54:42.129 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.120+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2278333507}) 2024-09-11T01:54:42.130 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.120+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.130 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.120+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=740759066}) 2024-09-11T01:54:42.130 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.120+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4204415024}) 2024-09-11T01:54:42.130 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.121+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.130 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.121+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1363756342}) 2024-09-11T01:54:42.130 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.121+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3091031203}) 2024-09-11T01:54:42.130 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.121+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.130 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.121+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1363756342}) 2024-09-11T01:54:42.130 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.121+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3605510763}) 2024-09-11T01:54:42.146 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.122+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.146 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.122+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2425570539}) 2024-09-11T01:54:42.146 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.122+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=445036602}) 2024-09-11T01:54:42.146 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.122+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.146 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.122+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2425570539}) 2024-09-11T01:54:42.146 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.122+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=161075657}) 2024-09-11T01:54:42.146 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.122+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.146 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.122+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1467178615}) 2024-09-11T01:54:42.147 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.122+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1819245673}) 2024-09-11T01:54:42.147 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.122+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.147 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.122+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1467178615}) 2024-09-11T01:54:42.147 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.122+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=414831480}) 2024-09-11T01:54:42.147 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.123+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.147 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.123+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=207246415}) 2024-09-11T01:54:42.147 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.123+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1808226108}) 2024-09-11T01:54:42.147 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.123+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.147 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.123+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=207246415}) 2024-09-11T01:54:42.147 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.123+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3952810364}) 2024-09-11T01:54:42.147 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.124+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.148 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.124+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1456388116}) 2024-09-11T01:54:42.148 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.124+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2228584647}) 2024-09-11T01:54:42.148 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.124+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.148 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.124+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1456388116}) 2024-09-11T01:54:42.148 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.124+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1384585439}) 2024-09-11T01:54:42.148 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.124+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.148 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.124+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=931768172}) 2024-09-11T01:54:42.148 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.124+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2458904819}) 2024-09-11T01:54:42.148 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.124+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.148 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.124+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=931768172}) 2024-09-11T01:54:42.148 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.124+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2343912420}) 2024-09-11T01:54:42.148 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.125+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.125+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1777050646}) 2024-09-11T01:54:42.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.125+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2147567074}) 2024-09-11T01:54:42.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.125+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.125+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1777050646}) 2024-09-11T01:54:42.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.125+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=686384861}) 2024-09-11T01:54:42.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.126+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.126+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2611759108}) 2024-09-11T01:54:42.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.126+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2578734450}) 2024-09-11T01:54:42.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.126+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.126+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2611759108}) 2024-09-11T01:54:42.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.126+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2694537096}) 2024-09-11T01:54:42.149 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.127+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.150 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.127+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2713379489}) 2024-09-11T01:54:42.150 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.127+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3993925977}) 2024-09-11T01:54:42.150 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.127+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.150 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.127+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2713379489}) 2024-09-11T01:54:42.150 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.127+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=55691110}) 2024-09-11T01:54:42.150 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.128+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.150 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.128+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3910835028}) 2024-09-11T01:54:42.150 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.128+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3788331838}) 2024-09-11T01:54:42.150 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.128+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.150 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.128+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3910835028}) 2024-09-11T01:54:42.150 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.128+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1791550778}) 2024-09-11T01:54:42.150 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.128+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.150 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.128+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4175254550}) 2024-09-11T01:54:42.150 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.128+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3302853013}) 2024-09-11T01:54:42.151 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.128+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.151 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.128+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4175254550}) 2024-09-11T01:54:42.151 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.128+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4153937502}) 2024-09-11T01:54:42.151 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.129+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.151 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.129+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=486361970}) 2024-09-11T01:54:42.151 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.129+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4024219296}) 2024-09-11T01:54:42.151 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.129+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.151 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.129+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=486361970}) 2024-09-11T01:54:42.151 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.129+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1804377173}) 2024-09-11T01:54:42.151 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.130+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.151 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.130+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2119423289}) 2024-09-11T01:54:42.152 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.130+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=898369011}) 2024-09-11T01:54:42.152 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.130+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.152 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.130+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2119423289}) 2024-09-11T01:54:42.152 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.130+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3647234506}) 2024-09-11T01:54:42.152 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.130+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.152 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.130+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1463737189}) 2024-09-11T01:54:42.152 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.130+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1970258094}) 2024-09-11T01:54:42.152 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.130+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.152 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.130+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1463737189}) 2024-09-11T01:54:42.152 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.130+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3103986314}) 2024-09-11T01:54:42.152 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.131+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.152 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.131+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3807802732}) 2024-09-11T01:54:42.152 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.131+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2025424640}) 2024-09-11T01:54:42.153 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.131+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.153 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.131+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3807802732}) 2024-09-11T01:54:42.153 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.131+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3183985169}) 2024-09-11T01:54:42.153 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.132+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.153 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.132+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2692548071}) 2024-09-11T01:54:42.153 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.132+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1440854895}) 2024-09-11T01:54:42.153 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.132+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.153 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.132+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2692548071}) 2024-09-11T01:54:42.153 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.132+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=598742993}) 2024-09-11T01:54:42.153 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.132+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.156 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.132+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1848054077}) 2024-09-11T01:54:42.156 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.132+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1110801276}) 2024-09-11T01:54:42.156 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.132+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.156 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.132+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1848054077}) 2024-09-11T01:54:42.157 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.132+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4022576702}) 2024-09-11T01:54:42.157 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.133+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.157 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.133+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2147938452}) 2024-09-11T01:54:42.157 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.133+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2270703024}) 2024-09-11T01:54:42.157 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.133+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.157 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.133+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2147938452}) 2024-09-11T01:54:42.157 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.133+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3565593964}) 2024-09-11T01:54:42.157 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.134+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.157 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.134+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3048776583}) 2024-09-11T01:54:42.157 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.134+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1569663374}) 2024-09-11T01:54:42.157 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.134+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.157 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.134+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3048776583}) 2024-09-11T01:54:42.157 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.134+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3583651752}) 2024-09-11T01:54:42.158 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.134+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.158 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.134+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=710945514}) 2024-09-11T01:54:42.158 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.134+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3845150881}) 2024-09-11T01:54:42.158 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.134+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.158 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.134+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=710945514}) 2024-09-11T01:54:42.158 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.134+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=499300751}) 2024-09-11T01:54:42.158 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.135+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.158 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.135+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4272658857}) 2024-09-11T01:54:42.158 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.135+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3535314205}) 2024-09-11T01:54:42.158 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.135+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.158 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.135+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4272658857}) 2024-09-11T01:54:42.158 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.135+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1970041371}) 2024-09-11T01:54:42.158 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.136+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.136+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=276248477}) 2024-09-11T01:54:42.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.136+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3377565927}) 2024-09-11T01:54:42.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.136+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.136+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=276248477}) 2024-09-11T01:54:42.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.136+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3189770}) 2024-09-11T01:54:42.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.136+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.136+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=421580125}) 2024-09-11T01:54:42.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.136+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1290302265}) 2024-09-11T01:54:42.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.136+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.136+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=421580125}) 2024-09-11T01:54:42.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.136+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=263556520}) 2024-09-11T01:54:42.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.137+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.159 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.137+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=751925643}) 2024-09-11T01:54:42.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.137+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2197032902}) 2024-09-11T01:54:42.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.137+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.137+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=751925643}) 2024-09-11T01:54:42.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.137+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=961814563}) 2024-09-11T01:54:42.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.138+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.138+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=67856461}) 2024-09-11T01:54:42.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.138+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1528280162}) 2024-09-11T01:54:42.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.138+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.138+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=67856461}) 2024-09-11T01:54:42.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.138+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=461937297}) 2024-09-11T01:54:42.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.138+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.160 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.138+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1051093679}) 2024-09-11T01:54:42.161 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.138+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=130306804}) 2024-09-11T01:54:42.161 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.138+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.161 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.138+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1051093679}) 2024-09-11T01:54:42.161 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.138+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=782657506}) 2024-09-11T01:54:42.161 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.139+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.161 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.139+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2613241313}) 2024-09-11T01:54:42.161 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.139+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2296790072}) 2024-09-11T01:54:42.161 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.139+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.161 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.139+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2613241313}) 2024-09-11T01:54:42.161 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.139+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=113839401}) 2024-09-11T01:54:42.161 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.140+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.161 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.140+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1612409793}) 2024-09-11T01:54:42.161 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.140+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3339882053}) 2024-09-11T01:54:42.162 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.140+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.162 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.140+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1612409793}) 2024-09-11T01:54:42.162 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.140+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756155988}) 2024-09-11T01:54:42.162 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.140+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.162 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.140+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3382377362}) 2024-09-11T01:54:42.162 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.140+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2709221043}) 2024-09-11T01:54:42.162 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.140+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.162 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.140+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3382377362}) 2024-09-11T01:54:42.162 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.140+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2449104478}) 2024-09-11T01:54:42.162 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.141+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.162 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.141+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1943620479}) 2024-09-11T01:54:42.162 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.141+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3057685130}) 2024-09-11T01:54:42.162 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.141+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.162 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.141+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1943620479}) 2024-09-11T01:54:42.163 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.141+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3415905398}) 2024-09-11T01:54:42.163 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.142+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.163 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.142+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3715883117}) 2024-09-11T01:54:42.163 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.142+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=733942913}) 2024-09-11T01:54:42.163 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.142+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.163 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.142+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3715883117}) 2024-09-11T01:54:42.163 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.142+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1819395133}) 2024-09-11T01:54:42.163 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.142+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.163 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.142+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=571919903}) 2024-09-11T01:54:42.163 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.142+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2501605181}) 2024-09-11T01:54:42.163 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.142+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.163 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.142+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=571919903}) 2024-09-11T01:54:42.163 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.142+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1859899659}) 2024-09-11T01:54:42.164 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.143+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.164 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.143+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1166098079}) 2024-09-11T01:54:42.164 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.143+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1017267105}) 2024-09-11T01:54:42.164 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.143+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.164 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.143+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1166098079}) 2024-09-11T01:54:42.164 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.143+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3032804429}) 2024-09-11T01:54:42.182 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.144+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.183 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.144+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4085076810}) 2024-09-11T01:54:42.183 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.144+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=666666171}) 2024-09-11T01:54:42.183 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.144+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.183 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.144+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4085076810}) 2024-09-11T01:54:42.183 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.144+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3764954248}) 2024-09-11T01:54:42.183 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.144+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.183 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.144+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3708265676}) 2024-09-11T01:54:42.183 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.144+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=113936449}) 2024-09-11T01:54:42.183 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.144+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.183 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.144+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3708265676}) 2024-09-11T01:54:42.183 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.144+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3409380642}) 2024-09-11T01:54:42.183 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.145+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.184 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.145+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2100095895}) 2024-09-11T01:54:42.184 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.145+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2821664923}) 2024-09-11T01:54:42.184 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.145+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.184 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.145+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2100095895}) 2024-09-11T01:54:42.184 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.145+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1618245450}) 2024-09-11T01:54:42.184 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.146+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.184 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.146+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3501938690}) 2024-09-11T01:54:42.184 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.146+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=879523769}) 2024-09-11T01:54:42.184 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.146+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.184 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.146+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3501938690}) 2024-09-11T01:54:42.184 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.146+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2314556262}) 2024-09-11T01:54:42.184 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.146+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.146+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=944714264}) 2024-09-11T01:54:42.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.146+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=631787651}) 2024-09-11T01:54:42.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.146+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.146+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=944714264}) 2024-09-11T01:54:42.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.146+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2938384815}) 2024-09-11T01:54:42.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.147+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.147+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1491128523}) 2024-09-11T01:54:42.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.147+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1235717874}) 2024-09-11T01:54:42.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.147+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.147+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1491128523}) 2024-09-11T01:54:42.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.147+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1507027964}) 2024-09-11T01:54:42.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.147+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.185 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.147+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4139367392}) 2024-09-11T01:54:42.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.147+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1083464653}) 2024-09-11T01:54:42.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.147+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.147+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4139367392}) 2024-09-11T01:54:42.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.148+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=293591002}) 2024-09-11T01:54:42.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.148+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.148+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=344025931}) 2024-09-11T01:54:42.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.148+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3162029236}) 2024-09-11T01:54:42.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.148+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.148+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=344025931}) 2024-09-11T01:54:42.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.148+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3895686767}) 2024-09-11T01:54:42.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.149+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.186 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.149+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612421241}) 2024-09-11T01:54:42.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.149+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1732681130}) 2024-09-11T01:54:42.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.149+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.149+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612421241}) 2024-09-11T01:54:42.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.149+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1599565895}) 2024-09-11T01:54:42.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.150+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.150+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3801955472}) 2024-09-11T01:54:42.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.150+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2363344278}) 2024-09-11T01:54:42.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.150+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.150+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3801955472}) 2024-09-11T01:54:42.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.150+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1030228060}) 2024-09-11T01:54:42.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.151+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.151+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3428712307}) 2024-09-11T01:54:42.187 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.151+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=131691025}) 2024-09-11T01:54:42.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.151+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.151+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3428712307}) 2024-09-11T01:54:42.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.151+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4062360155}) 2024-09-11T01:54:42.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.151+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.151+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2409610447}) 2024-09-11T01:54:42.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.151+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4180558307}) 2024-09-11T01:54:42.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.151+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.151+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2409610447}) 2024-09-11T01:54:42.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.151+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2229897650}) 2024-09-11T01:54:42.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.152+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.152+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305390978}) 2024-09-11T01:54:42.188 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.152+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2333756575}) 2024-09-11T01:54:42.189 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.152+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.189 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.152+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305390978}) 2024-09-11T01:54:42.189 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.152+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2071513121}) 2024-09-11T01:54:42.189 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.153+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.189 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.153+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2269055443}) 2024-09-11T01:54:42.189 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.153+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3049092150}) 2024-09-11T01:54:42.189 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.153+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.189 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.153+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2269055443}) 2024-09-11T01:54:42.189 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.153+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4145224903}) 2024-09-11T01:54:42.189 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.154+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.189 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.154+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4092708686}) 2024-09-11T01:54:42.189 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.154+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3367776699}) 2024-09-11T01:54:42.189 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.154+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.189 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.154+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4092708686}) 2024-09-11T01:54:42.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.154+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=460457593}) 2024-09-11T01:54:42.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.155+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.155+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=108389348}) 2024-09-11T01:54:42.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.155+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2206675389}) 2024-09-11T01:54:42.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.155+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.155+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=108389348}) 2024-09-11T01:54:42.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.155+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1405357100}) 2024-09-11T01:54:42.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.155+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.155+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2446482381}) 2024-09-11T01:54:42.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.155+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2890754545}) 2024-09-11T01:54:42.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.155+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.155+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2446482381}) 2024-09-11T01:54:42.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.155+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=436293281}) 2024-09-11T01:54:42.190 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.156+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.156+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4233731714}) 2024-09-11T01:54:42.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.156+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2950252819}) 2024-09-11T01:54:42.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.156+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.156+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4233731714}) 2024-09-11T01:54:42.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.156+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=358907792}) 2024-09-11T01:54:42.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.157+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.157+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1383586437}) 2024-09-11T01:54:42.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.157+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2211397523}) 2024-09-11T01:54:42.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.157+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.157+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1383586437}) 2024-09-11T01:54:42.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.157+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=984802246}) 2024-09-11T01:54:42.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.158+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.158+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=679818281}) 2024-09-11T01:54:42.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.158+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3349101804}) 2024-09-11T01:54:42.191 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.158+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.158+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=679818281}) 2024-09-11T01:54:42.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.158+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=84759071}) 2024-09-11T01:54:42.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.166+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.166+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3156194497}) 2024-09-11T01:54:42.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.166+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2576127625}) 2024-09-11T01:54:42.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.166+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.166+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3156194497}) 2024-09-11T01:54:42.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.166+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3527010372}) 2024-09-11T01:54:42.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.166+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.166+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4132783756}) 2024-09-11T01:54:42.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.166+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2313829201}) 2024-09-11T01:54:42.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.166+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.166+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4132783756}) 2024-09-11T01:54:42.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.166+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1365722544}) 2024-09-11T01:54:42.192 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.167+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.167+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2198371072}) 2024-09-11T01:54:42.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.167+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1272524595}) 2024-09-11T01:54:42.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.167+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.167+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2198371072}) 2024-09-11T01:54:42.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.167+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1176448739}) 2024-09-11T01:54:42.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.168+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.168+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1519235663}) 2024-09-11T01:54:42.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.168+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=299030321}) 2024-09-11T01:54:42.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.168+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.168+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1519235663}) 2024-09-11T01:54:42.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.168+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2410025156}) 2024-09-11T01:54:42.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.169+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.193 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.169+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2165846364}) 2024-09-11T01:54:42.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.169+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4010200264}) 2024-09-11T01:54:42.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.169+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.169+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2165846364}) 2024-09-11T01:54:42.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.169+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3345883004}) 2024-09-11T01:54:42.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.169+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.169+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2175288117}) 2024-09-11T01:54:42.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.169+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1278437784}) 2024-09-11T01:54:42.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.169+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.169+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2175288117}) 2024-09-11T01:54:42.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.169+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=526144755}) 2024-09-11T01:54:42.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.170+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.170+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3709854411}) 2024-09-11T01:54:42.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.170+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=552761782}) 2024-09-11T01:54:42.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.170+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.194 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.170+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3709854411}) 2024-09-11T01:54:42.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.170+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1665130977}) 2024-09-11T01:54:42.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.171+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.171+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3225192292}) 2024-09-11T01:54:42.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.171+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2241436552}) 2024-09-11T01:54:42.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.171+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.171+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3225192292}) 2024-09-11T01:54:42.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.171+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3400237912}) 2024-09-11T01:54:42.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.172+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.172+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2768876931}) 2024-09-11T01:54:42.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.172+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2853098028}) 2024-09-11T01:54:42.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.172+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.172+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2768876931}) 2024-09-11T01:54:42.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.172+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3405599787}) 2024-09-11T01:54:42.195 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.172+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.196 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.172+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2478976421}) 2024-09-11T01:54:42.196 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.172+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1103510869}) 2024-09-11T01:54:42.196 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.172+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.196 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.172+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2478976421}) 2024-09-11T01:54:42.196 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.172+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3763810328}) 2024-09-11T01:54:42.196 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.173+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.196 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.173+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=550110042}) 2024-09-11T01:54:42.196 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.173+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1719822702}) 2024-09-11T01:54:42.196 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.173+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.196 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.173+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=550110042}) 2024-09-11T01:54:42.196 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.173+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2787177424}) 2024-09-11T01:54:42.196 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.174+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.197 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.174+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2918428134}) 2024-09-11T01:54:42.197 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.174+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3149738780}) 2024-09-11T01:54:42.197 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.174+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.197 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.174+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2918428134}) 2024-09-11T01:54:42.197 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.174+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=332796711}) 2024-09-11T01:54:42.197 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.175+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.197 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.175+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1641610650}) 2024-09-11T01:54:42.197 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.175+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=12094819}) 2024-09-11T01:54:42.197 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.175+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.197 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.175+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1641610650}) 2024-09-11T01:54:42.197 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.175+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2375234204}) 2024-09-11T01:54:42.197 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.175+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.197 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.175+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3079975908}) 2024-09-11T01:54:42.197 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.175+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2017803133}) 2024-09-11T01:54:42.197 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.175+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.198 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.175+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3079975908}) 2024-09-11T01:54:42.198 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.175+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3017005247}) 2024-09-11T01:54:42.198 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.176+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.198 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.176+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3827923633}) 2024-09-11T01:54:42.198 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.176+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4081783225}) 2024-09-11T01:54:42.198 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.176+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.198 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.176+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3827923633}) 2024-09-11T01:54:42.198 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.176+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3340866205}) 2024-09-11T01:54:42.198 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.177+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.198 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.177+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3939331129}) 2024-09-11T01:54:42.198 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.177+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3036697668}) 2024-09-11T01:54:42.198 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.177+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.199 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.177+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3939331129}) 2024-09-11T01:54:42.199 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.177+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1284302847}) 2024-09-11T01:54:42.199 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.178+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.199 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.178+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=60,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=2} crc {logm=3838471485,mdsmap=3760429986,mgr=4080174641,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1977883,mon_config_key=4147331739,monmap=1324671969}) 2024-09-11T01:54:42.199 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.178+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=57,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=16,monmap=2,nvmeofgw=2} crc {logm=2864475985,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2674937487,mon_config_key=1809882168,monmap=1324671969,nvmeofgw=1596038039}) 2024-09-11T01:54:42.199 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.178+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.199 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.178+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=60,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=2} crc {logm=3838471485,mdsmap=3760429986,mgr=4080174641,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1977883,mon_config_key=4147331739,monmap=1324671969}) 2024-09-11T01:54:42.199 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.178+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=54,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=1} crc {logm=1317724297,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2674937487,mon_config_key=2516600898,monmap=2933275274,nvmeofgw=598337686,osd_metadata=3504647627}) 2024-09-11T01:54:42.199 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.179+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.199 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.179+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=1,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=89} crc {monmap=1228700967,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=2979909969,osd_snap=2371624867}) 2024-09-11T01:54:42.199 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.179+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {monmap=1,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=89} crc {monmap=1228700967,nvmeofgw=598337686,osd_metadata=2310555360,osd_pg_creating=2979909969,osd_snap=709157089}) 2024-09-11T01:54:42.199 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.179+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.199 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.179+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=1,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=89} crc {monmap=1228700967,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=2979909969,osd_snap=2371624867}) 2024-09-11T01:54:42.200 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.179+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {monmap=1,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=90} crc {monmap=1228700967,nvmeofgw=598337686,osd_metadata=4228979228,osd_pg_creating=2979909969,osd_snap=1822257192}) 2024-09-11T01:54:42.200 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.179+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.200 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.179+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=79,osdmap=21} crc {osd_snap=1415650560,osdmap=2301764349}) 2024-09-11T01:54:42.200 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.179+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=84,osdmap=16} crc {osd_snap=4046830347,osdmap=631084703}) 2024-09-11T01:54:42.200 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.180+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.200 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.180+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=79,osdmap=21} crc {osd_snap=1415650560,osdmap=2301764349}) 2024-09-11T01:54:42.200 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.180+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=81,osdmap=19} crc {osd_snap=2400557,osdmap=468300667}) 2024-09-11T01:54:42.201 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.180+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.202 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.180+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3907855883}) 2024-09-11T01:54:42.202 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.180+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3750302136}) 2024-09-11T01:54:42.202 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.180+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.202 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.180+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3907855883}) 2024-09-11T01:54:42.202 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.180+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3653130292}) 2024-09-11T01:54:42.202 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.182+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.202 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.182+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1728239068}) 2024-09-11T01:54:42.202 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.182+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=763078997}) 2024-09-11T01:54:42.202 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.182+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.202 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.182+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1728239068}) 2024-09-11T01:54:42.202 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.182+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2127930341}) 2024-09-11T01:54:42.202 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.186+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.202 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.186+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=49} crc {osdmap=3309120312}) 2024-09-11T01:54:42.203 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.186+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=50} crc {osdmap=294560449}) 2024-09-11T01:54:42.203 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.186+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:54:42.203 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.186+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=49} crc {osdmap=3309120312}) 2024-09-11T01:54:42.203 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:54:42.186+0000 7fc3d11d4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=51} crc {osdmap=2094067757}) 2024-09-11T01:54:42.752 INFO:tasks.rados.rados.0.smithi071.stdout:3758: done (5 left) 2024-09-11T01:54:42.752 INFO:tasks.rados.rados.0.smithi071.stdout:3759: rollback oid 28 current snap is 424 2024-09-11T01:54:42.753 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 28 to 364 2024-09-11T01:54:42.753 INFO:tasks.rados.rados.0.smithi071.stdout:3752: expect (ObjNum 949 snap 300 seq_num 949) 2024-09-11T01:54:42.891 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:54:42.891 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:54:42.895 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:54:42.968 INFO:tasks.rados.rados.0.smithi071.stdout:3753: finishing copy_from racing read to smithi07134777-33 2024-09-11T01:54:42.968 INFO:tasks.rados.rados.0.smithi071.stdout:3753: finishing copy_from to smithi07134777-33 2024-09-11T01:54:42.968 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 2206 (ObjNum 1115 snap 356 seq_num 1115) dirty exists 2024-09-11T01:54:42.968 INFO:tasks.rados.rados.0.smithi071.stdout:3756: expect (ObjNum 1281 snap 422 seq_num 1281) 2024-09-11T01:54:42.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.995+0000 7fcc3b433640 10 monclient: get_monmap_and_config 2024-09-11T01:54:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.995+0000 7fcc3b433640 10 monclient: build_initial_monmap 2024-09-11T01:54:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.995+0000 7fcc3b433640 10 monclient: monmap: 2024-09-11T01:54:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:54:42.996973+0000 2024-09-11T01:54:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:54:42.996973+0000 2024-09-11T01:54:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:54:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:54:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:54:42.999 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:54:42.999 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:54:42.999 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.995+0000 7fcc3b433640 1 Processor -- start 2024-09-11T01:54:43.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc3b433640 1 -- start start 2024-09-11T01:54:43.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc3b433640 10 monclient: init 2024-09-11T01:54:43.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc3b433640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:43.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc3b433640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-11T01:54:43.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc3b433640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcc34138230 0x7fcc34138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc3b433640 10 monclient(hunting): picked mon.noname-c con 0x7fcc34138230 addr v2:172.21.15.71:3300/0 2024-09-11T01:54:43.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc3b433640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcc3409d030 addr v1:172.21.15.71:6789/0 2024-09-11T01:54:43.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc3b433640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc3409d460 0x7fcc340a5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc3b433640 10 monclient(hunting): picked mon.noname-b con 0x7fcc3409d460 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:43.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc3b433640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc340a6e30 con 0x7fcc3409d030 2024-09-11T01:54:43.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc3b433640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc3b433640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fcc3409c9c0 con 0x7fcc34138230 2024-09-11T01:54:43.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc3b433640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc3b433640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcc3409ba20 con 0x7fcc3409d460 2024-09-11T01:54:43.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc3b433640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:43.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc3b433640 10 monclient(hunting): authenticate will time out at 2289.439628s 2024-09-11T01:54:43.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc399d0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc3409d460 0x7fcc340a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc399d0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc3409d460 0x7fcc340a5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41136/0 (socket says 172.21.15.12:41136) 2024-09-11T01:54:43.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc399d0640 1 -- 172.21.15.12:0/1716048107 learned_addr learned my addr 172.21.15.12:0/1716048107 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:43.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc399d0640 10 monclient(hunting): get_auth_request con 0x7fcc3409d460 auth_method 0 2024-09-11T01:54:43.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc399d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc399d0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc399d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc391cf640 1 --2- 172.21.15.12:0/1716048107 >> v2:172.21.15.71:3300/0 conn(0x7fcc34138230 0x7fcc34138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc391cf640 10 monclient(hunting): get_auth_request con 0x7fcc34138230 auth_method 0 2024-09-11T01:54:43.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc391cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc391cf640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc391cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc399d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc399d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc399d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc391cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc391cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.996+0000 7fcc391cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.997+0000 7fcc399d0640 10 monclient(hunting): handle_auth_done global_id 284126 payload 274 2024-09-11T01:54:43.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.997+0000 7fcc399d0640 1 -- 172.21.15.12:0/1716048107 >> v2:172.21.15.71:3300/0 conn(0x7fcc34138230 msgr2=0x7fcc34138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:54:43.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.997+0000 7fcc399d0640 1 --2- 172.21.15.12:0/1716048107 >> v2:172.21.15.71:3300/0 conn(0x7fcc34138230 0x7fcc34138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.997+0000 7fcc399d0640 1 -- 172.21.15.12:0/1716048107 >> v1:172.21.15.71:6789/0 conn(0x7fcc3409d030 legacy=0x7fcc340a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:54:43.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.997+0000 7fcc399d0640 10 monclient: _finish_hunting 0 2024-09-11T01:54:43.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.997+0000 7fcc399d0640 1 monclient: found mon.noname-b 2024-09-11T01:54:43.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.997+0000 7fcc399d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:43.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.997+0000 7fcc399d0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:54:43.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.997+0000 7fcc399d0640 1 -- 172.21.15.12:0/1716048107 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc340a66a0 con 0x7fcc3409d460 2024-09-11T01:54:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.999+0000 7fcc399d0640 1 -- 172.21.15.12:0/1716048107 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc3409d460 msgr2=0x7fcc340a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.999+0000 7fcc399d0640 1 --2- 172.21.15.12:0/1716048107 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc3409d460 0x7fcc340a5730 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcc340a17e0 tx=0x7fcc2c02a530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f264dce7ae361c30 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:42.999+0000 7fcc27fff640 1 -- 172.21.15.12:0/1716048107 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c036070 con 0x7fcc3409d460 2024-09-11T01:54:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.000+0000 7fcc27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.000+0000 7fcc27fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:54:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.000+0000 7fcc27fff640 10 monclient: dump: 2024-09-11T01:54:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:43.011 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.000+0000 7fcc27fff640 10 monclient: _finish_auth 0 2024-09-11T01:54:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.000+0000 7fcc27fff640 10 monclient: _check_auth_tickets 2024-09-11T01:54:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.000+0000 7fcc27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.000+0000 7fcc3b433640 5 monclient: authenticate success, global_id 284126 2024-09-11T01:54:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.000+0000 7fcc27fff640 1 -- 172.21.15.12:0/1716048107 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc2c007800 con 0x7fcc3409d460 2024-09-11T01:54:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.000+0000 7fcc3b433640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:54:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.000+0000 7fcc27fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.000+0000 7fcc3b433640 10 monclient: get_monmap_and_config success 2024-09-11T01:54:43.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.000+0000 7fcc27fff640 1 -- 172.21.15.12:0/1716048107 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c007bd0 con 0x7fcc3409d460 2024-09-11T01:54:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.000+0000 7fcc27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.000+0000 7fcc3b433640 4 set_mon_vals no callback set 2024-09-11T01:54:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.000+0000 7fcc27fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.000+0000 7fcc27fff640 10 monclient: dump: 2024-09-11T01:54:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:43.012 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:43.013 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.000+0000 7fcc3b433640 10 monclient: shutdown 2024-09-11T01:54:43.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.000+0000 7fcc3b433640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:43.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.000+0000 7fcc3b433640 1 -- 172.21.15.12:0/1716048107 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc3409d460 msgr2=0x7fcc340a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.000+0000 7fcc3b433640 1 --2- 172.21.15.12:0/1716048107 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc3409d460 0x7fcc340a5730 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcc340a17e0 tx=0x7fcc2c02a530 comp rx=0 tx=0).stop 2024-09-11T01:54:43.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.000+0000 7fcc3b433640 1 -- 172.21.15.12:0/1716048107 shutdown_connections 2024-09-11T01:54:43.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.000+0000 7fcc3b433640 1 --2- 172.21.15.12:0/1716048107 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc3409d460 0x7fcc340a5730 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.000+0000 7fcc3b433640 1 --2- 172.21.15.12:0/1716048107 >> v2:172.21.15.71:3300/0 conn(0x7fcc34138230 0x7fcc34138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.000+0000 7fcc3b433640 1 -- 172.21.15.12:0/1716048107 >> 172.21.15.12:0/1716048107 conn(0x7fcc340a7120 msgr2=0x7fcc340a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:43.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.000+0000 7fcc3b433640 1 -- 172.21.15.12:0/1716048107 shutdown_connections 2024-09-11T01:54:43.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.000+0000 7fcc3b433640 1 -- 172.21.15.12:0/1716048107 wait complete. 2024-09-11T01:54:43.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.001+0000 7fcc3b433640 10 monclient: build_initial_monmap 2024-09-11T01:54:43.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.001+0000 7fcc3b433640 10 monclient: monmap: 2024-09-11T01:54:43.014 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:43.014 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:54:43.014 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:54:43.014 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:54:43.014 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:43.014 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.014 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:54:43.014 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:54:43.014 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:54:43.014 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.001+0000 7fcc3b433640 1 Processor -- start 2024-09-11T01:54:43.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.001+0000 7fcc3b433640 1 -- start start 2024-09-11T01:54:43.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.001+0000 7fcc3b433640 10 monclient: init 2024-09-11T01:54:43.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc3b433640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc3b433640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:54:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc3b433640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc3409d030 0x7fcc34136d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc3b433640 10 monclient(hunting): picked mon.noname-a con 0x7fcc3409d030 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:54:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc3b433640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc3409d460 0x7fcc34137250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc3b433640 10 monclient(hunting): picked mon.noname-b con 0x7fcc3409d460 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:54:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc3b433640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc34138230 0x7fcc34120c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc3b433640 10 monclient(hunting): picked mon.noname-c con 0x7fcc34138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc3b433640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc3b433640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3406f970 con 0x7fcc3409d030 2024-09-11T01:54:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc3b433640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc3b433640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3406f7f0 con 0x7fcc3409d460 2024-09-11T01:54:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc3b433640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc3b433640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fcc340a5c70 con 0x7fcc34138230 2024-09-11T01:54:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc3b433640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc3b433640 10 monclient(hunting): authenticate will time out at 2289.445278s 2024-09-11T01:54:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc391cf640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc3409d030 0x7fcc34136d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc399d0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc34138230 0x7fcc34120c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc391cf640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc3409d030 0x7fcc34136d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43590/0 (socket says 172.21.15.12:43590) 2024-09-11T01:54:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc399d0640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc34138230 0x7fcc34120c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41148/0 (socket says 172.21.15.12:41148) 2024-09-11T01:54:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc391cf640 1 -- 172.21.15.12:0/458353219 learned_addr learned my addr 172.21.15.12:0/458353219 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc391cf640 10 monclient(hunting): get_auth_request con 0x7fcc3409d030 auth_method 0 2024-09-11T01:54:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc391cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc391cf640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc391cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc399d0640 10 monclient(hunting): get_auth_request con 0x7fcc34138230 auth_method 0 2024-09-11T01:54:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc399d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc399d0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc399d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc389ce640 1 --2- 172.21.15.12:0/458353219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc3409d460 0x7fcc34137250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc389ce640 10 monclient(hunting): get_auth_request con 0x7fcc3409d460 auth_method 0 2024-09-11T01:54:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc389ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc389ce640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc389ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc391cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc391cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc391cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc399d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc399d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc399d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc389ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc389ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc389ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc399d0640 10 monclient(hunting): handle_auth_done global_id 284129 payload 824 2024-09-11T01:54:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc399d0640 1 -- 172.21.15.12:0/458353219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc3409d460 msgr2=0x7fcc34137250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc399d0640 1 --2- 172.21.15.12:0/458353219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc3409d460 0x7fcc34137250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc399d0640 1 -- 172.21.15.12:0/458353219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc3409d030 msgr2=0x7fcc34136d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc399d0640 1 --2- 172.21.15.12:0/458353219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc3409d030 0x7fcc34136d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc399d0640 10 monclient: _finish_hunting 0 2024-09-11T01:54:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc399d0640 1 monclient: found mon.noname-c 2024-09-11T01:54:43.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc399d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc399d0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:54:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc399d0640 1 -- 172.21.15.12:0/458353219 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc34122ac0 con 0x7fcc34138230 2024-09-11T01:54:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.002+0000 7fcc391cf640 1 --2- 172.21.15.12:0/458353219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc3409d030 0x7fcc34136d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:54:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.004+0000 7fcc399d0640 1 -- 172.21.15.12:0/458353219 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc34138230 msgr2=0x7fcc34120c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.004+0000 7fcc399d0640 1 --2- 172.21.15.12:0/458353219 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc34138230 0x7fcc34120c70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c030000 tx=0x7fcc2c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.005+0000 7fcc25ffb640 1 -- 172.21.15.12:0/458353219 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c036050 con 0x7fcc34138230 2024-09-11T01:54:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.005+0000 7fcc25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.005+0000 7fcc25ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:54:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.005+0000 7fcc25ffb640 10 monclient: dump: 2024-09-11T01:54:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.019 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.020 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:43.020 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.020 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:43.020 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:43.020 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:43.020 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.005+0000 7fcc25ffb640 10 monclient: _finish_auth 0 2024-09-11T01:54:43.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.005+0000 7fcc25ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:54:43.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.005+0000 7fcc25ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:43.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.005+0000 7fcc3b433640 5 monclient: authenticate success, global_id 284129 2024-09-11T01:54:43.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.005+0000 7fcc3b433640 10 monclient: _renew_subs 2024-09-11T01:54:43.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.005+0000 7fcc3b433640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:43.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.005+0000 7fcc25ffb640 1 -- 172.21.15.12:0/458353219 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc2c042070 con 0x7fcc34138230 2024-09-11T01:54:43.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.005+0000 7fcc3b433640 1 -- 172.21.15.12:0/458353219 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc34123e30 con 0x7fcc34138230 2024-09-11T01:54:43.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.005+0000 7fcc25ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:43.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.005+0000 7fcc25ffb640 1 -- 172.21.15.12:0/458353219 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c031d90 con 0x7fcc34138230 2024-09-11T01:54:43.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.005+0000 7fcc25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:43.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.005+0000 7fcc25ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:43.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.005+0000 7fcc25ffb640 10 monclient: dump: 2024-09-11T01:54:43.021 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:43.021 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.021 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.021 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.021 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:43.021 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.021 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:43.021 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:43.022 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:43.022 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.005+0000 7fcc3b433640 10 monclient: _renew_subs 2024-09-11T01:54:43.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.005+0000 7fcc3b433640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:43.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.005+0000 7fcc3b433640 1 -- 172.21.15.12:0/458353219 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc340a1810 con 0x7fcc34138230 2024-09-11T01:54:43.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.005+0000 7fcc277fe640 4 set_mon_vals no callback set 2024-09-11T01:54:43.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.006+0000 7fcc25ffb640 1 -- 172.21.15.12:0/458353219 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fcc2c03d450 con 0x7fcc34138230 2024-09-11T01:54:43.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.006+0000 7fcc3b433640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:43.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.006+0000 7fcc3b433640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:43.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.006+0000 7fcc3b433640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:43.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.006+0000 7fcc3b433640 1 -- 172.21.15.12:0/458353219 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc34122390 con 0x7fcc34138230 2024-09-11T01:54:43.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.006+0000 7fcc25ffb640 1 --2- 172.21.15.12:0/458353219 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcc1803e260 0x7fcc18040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.006+0000 7fcc25ffb640 1 -- 172.21.15.12:0/458353219 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(858..858 src has 676..858) ==== 4847+0+0 (secure 0 0 0) 0x7fcc2c073600 con 0x7fcc34138230 2024-09-11T01:54:43.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.006+0000 7fcc391cf640 1 --2- 172.21.15.12:0/458353219 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcc1803e260 0x7fcc18040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.006+0000 7fcc391cf640 10 monclient: get_auth_request con 0x7fcc1803e260 auth_method 0 2024-09-11T01:54:43.023 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:54:43.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.009+0000 7fcc391cf640 1 --2- 172.21.15.12:0/458353219 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcc1803e260 0x7fcc18040720 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fcc300112c0 tx=0x7fcc30009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:43.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.011+0000 7fcc25ffb640 1 -- 172.21.15.12:0/458353219 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fcc2c04b020 con 0x7fcc34138230 2024-09-11T01:54:43.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.011+0000 7fcc25ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:43.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.011+0000 7fcc25ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:54:43.082 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1957055 2024-09-11T01:54:43.104 INFO:tasks.rados.rados.0.smithi071.stdout:3759: finishing rollback tid 0 to smithi07134777-28 2024-09-11T01:54:43.104 INFO:tasks.rados.rados.0.smithi071.stdout:3759: finishing rollback tid 1 to smithi07134777-28 2024-09-11T01:54:43.104 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 2411 (ObjNum 129 snap 0 seq_num 2371133616) dirty dne 2024-09-11T01:54:43.104 INFO:tasks.rados.rados.0.smithi071.stdout:3751: done (5 left) 2024-09-11T01:54:43.104 INFO:tasks.rados.rados.0.smithi071.stdout:3752: done (4 left) 2024-09-11T01:54:43.104 INFO:tasks.rados.rados.0.smithi071.stdout:3753: done (3 left) 2024-09-11T01:54:43.104 INFO:tasks.rados.rados.0.smithi071.stdout:3755: done (2 left) 2024-09-11T01:54:43.104 INFO:tasks.rados.rados.0.smithi071.stdout:3756: done (1 left) 2024-09-11T01:54:43.104 INFO:tasks.rados.rados.0.smithi071.stdout:3759: done (0 left) 2024-09-11T01:54:43.104 INFO:tasks.rados.rados.0.smithi071.stdout:3760: snap_remove snap 271 2024-09-11T01:54:43.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.267+0000 7fcc3b433640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:54:43.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.267+0000 7fcc3b433640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:54:43.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.267+0000 7fcc3b433640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:43.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.267+0000 7fcc3b433640 1 -- 172.21.15.12:0/458353219 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcc34138610 con 0x7fcc34138230 2024-09-11T01:54:43.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.267+0000 7fcc25ffb640 1 -- 172.21.15.12:0/458353219 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fcc2c003990 con 0x7fcc34138230 2024-09-11T01:54:43.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.267+0000 7fcc25ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:54:43.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.267+0000 7fcc25ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:54:43.271 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":232,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:54:43.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.269+0000 7fcc3b433640 1 -- 172.21.15.12:0/458353219 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcc1803e260 msgr2=0x7fcc18040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.269+0000 7fcc3b433640 1 --2- 172.21.15.12:0/458353219 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcc1803e260 0x7fcc18040720 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fcc300112c0 tx=0x7fcc30009190 comp rx=0 tx=0).stop 2024-09-11T01:54:43.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.269+0000 7fcc3b433640 10 monclient: shutdown 2024-09-11T01:54:43.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.269+0000 7fcc3b433640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:43.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.269+0000 7fcc3b433640 1 -- 172.21.15.12:0/458353219 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc34138230 msgr2=0x7fcc34120c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.269+0000 7fcc3b433640 1 --2- 172.21.15.12:0/458353219 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc34138230 0x7fcc34120c70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c030000 tx=0x7fcc2c004a40 comp rx=0 tx=0).stop 2024-09-11T01:54:43.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.269+0000 7fcc3b433640 1 -- 172.21.15.12:0/458353219 shutdown_connections 2024-09-11T01:54:43.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.269+0000 7fcc3b433640 1 --2- 172.21.15.12:0/458353219 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc34138230 0x7fcc34120c70 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.269+0000 7fcc3b433640 1 --2- 172.21.15.12:0/458353219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc3409d460 0x7fcc34137250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.269+0000 7fcc3b433640 1 --2- 172.21.15.12:0/458353219 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fcc1803e260 0x7fcc18040720 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.269+0000 7fcc3b433640 1 --2- 172.21.15.12:0/458353219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcc3409d030 0x7fcc34136d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.269+0000 7fcc3b433640 1 -- 172.21.15.12:0/458353219 >> 172.21.15.12:0/458353219 conn(0x7fcc340a7120 msgr2=0x7fcc340a3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:43.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.269+0000 7fcc3b433640 1 -- 172.21.15.12:0/458353219 shutdown_connections 2024-09-11T01:54:43.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.270+0000 7fcc3b433640 1 -- 172.21.15.12:0/458353219 wait complete. 2024-09-11T01:54:43.282 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:54:43.282 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:54:43.283 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:54:43.283 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:54:43.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.382+0000 7f2d10260640 10 monclient: get_monmap_and_config 2024-09-11T01:54:43.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.382+0000 7f2d10260640 10 monclient: build_initial_monmap 2024-09-11T01:54:43.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.382+0000 7f2d10260640 10 monclient: monmap: 2024-09-11T01:54:43.386 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:43.386 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.386 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:54:43.384201+0000 2024-09-11T01:54:43.386 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:54:43.384201+0000 2024-09-11T01:54:43.386 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:43.386 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.386 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:54:43.386 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:54:43.386 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:54:43.386 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:54:43.386 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:54:43.386 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.383+0000 7f2d10260640 1 Processor -- start 2024-09-11T01:54:43.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.383+0000 7f2d10260640 1 -- start start 2024-09-11T01:54:43.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.383+0000 7f2d10260640 10 monclient: init 2024-09-11T01:54:43.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.383+0000 7f2d10260640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:43.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.383+0000 7f2d10260640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-11T01:54:43.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.383+0000 7f2d10260640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2d08135fc0 addr v1:172.21.15.12:6789/0 2024-09-11T01:54:43.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.383+0000 7f2d10260640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2d08074000 addr v1:172.21.15.71:6789/0 2024-09-11T01:54:43.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.383+0000 7f2d10260640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d08074410 0x7f2d0813e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.383+0000 7f2d10260640 10 monclient(hunting): picked mon.noname-b con 0x7f2d08074410 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:43.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.383+0000 7f2d10260640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d080732d0 con 0x7f2d08135fc0 2024-09-11T01:54:43.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.383+0000 7f2d10260640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d08073890 con 0x7f2d08074000 2024-09-11T01:54:43.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.383+0000 7f2d10260640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.383+0000 7f2d10260640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2d08074820 con 0x7f2d08074410 2024-09-11T01:54:43.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.383+0000 7f2d10260640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:43.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.383+0000 7f2d10260640 10 monclient(hunting): authenticate will time out at 2289.826916s 2024-09-11T01:54:43.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.383+0000 7f2d0dffc640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2d08135fc0 0x7f2d081363a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:32782/0 (socket says 172.21.15.12:32782) 2024-09-11T01:54:43.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.383+0000 7f2d0dffc640 1 -- 172.21.15.12:0/2147171641 learned_addr learned my addr 172.21.15.12:0/2147171641 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:54:43.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.383+0000 7f2d0e7fd640 1 --2- 172.21.15.12:0/2147171641 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d08074410 0x7f2d0813e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.384+0000 7f2d0e7fd640 10 monclient(hunting): get_auth_request con 0x7f2d08074410 auth_method 0 2024-09-11T01:54:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.384+0000 7f2d0e7fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.384+0000 7f2d0e7fd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.384+0000 7f2d0e7fd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.384+0000 7f2d0e7fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.384+0000 7f2d0e7fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.384+0000 7f2d0e7fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.384+0000 7f2d0e7fd640 10 monclient(hunting): handle_auth_done global_id 284132 payload 274 2024-09-11T01:54:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.384+0000 7f2d0e7fd640 1 -- 172.21.15.12:0/2147171641 >> v1:172.21.15.71:6789/0 conn(0x7f2d08074000 legacy=0x7f2d0813ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:54:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.384+0000 7f2d0e7fd640 1 -- 172.21.15.12:0/2147171641 >> v1:172.21.15.12:6789/0 conn(0x7f2d08135fc0 legacy=0x7f2d081363a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:54:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.384+0000 7f2d0e7fd640 10 monclient: _finish_hunting 0 2024-09-11T01:54:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.384+0000 7f2d0e7fd640 1 monclient: found mon.noname-b 2024-09-11T01:54:43.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.384+0000 7f2d0e7fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:43.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.384+0000 7f2d0e7fd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:54:43.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.384+0000 7f2d0e7fd640 1 -- 172.21.15.12:0/2147171641 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d08072000 con 0x7f2d08074410 2024-09-11T01:54:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.387+0000 7f2d0e7fd640 1 -- 172.21.15.12:0/2147171641 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d08074410 msgr2=0x7f2d0813e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.387+0000 7f2d0e7fd640 1 --2- 172.21.15.12:0/2147171641 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d08074410 0x7f2d0813e570 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2d00009dc0 tx=0x7f2d0002a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9de26ff282f7453 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.387+0000 7f2d0cffa640 1 -- 172.21.15.12:0/2147171641 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d00007a30 con 0x7f2d08074410 2024-09-11T01:54:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.387+0000 7f2d0cffa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.387+0000 7f2d0cffa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:54:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.387+0000 7f2d0cffa640 10 monclient: dump: 2024-09-11T01:54:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:43.400 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.387+0000 7f2d0cffa640 10 monclient: _finish_auth 0 2024-09-11T01:54:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.387+0000 7f2d0cffa640 10 monclient: _check_auth_tickets 2024-09-11T01:54:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.387+0000 7f2d0cffa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.387+0000 7f2d10260640 5 monclient: authenticate success, global_id 284132 2024-09-11T01:54:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.387+0000 7f2d10260640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:54:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.387+0000 7f2d0cffa640 1 -- 172.21.15.12:0/2147171641 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d00007bd0 con 0x7f2d08074410 2024-09-11T01:54:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.387+0000 7f2d0cffa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.387+0000 7f2d0cffa640 1 -- 172.21.15.12:0/2147171641 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d00031400 con 0x7f2d08074410 2024-09-11T01:54:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.387+0000 7f2d0cffa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.387+0000 7f2d0cffa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.387+0000 7f2d0cffa640 10 monclient: dump: 2024-09-11T01:54:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:43.402 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.387+0000 7f2d10260640 10 monclient: get_monmap_and_config success 2024-09-11T01:54:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.387+0000 7f2d10260640 4 set_mon_vals no callback set 2024-09-11T01:54:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.387+0000 7f2d10260640 10 monclient: shutdown 2024-09-11T01:54:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.387+0000 7f2d10260640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.387+0000 7f2d10260640 1 -- 172.21.15.12:0/2147171641 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d08074410 msgr2=0x7f2d0813e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.387+0000 7f2d10260640 1 --2- 172.21.15.12:0/2147171641 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d08074410 0x7f2d0813e570 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2d00009dc0 tx=0x7f2d0002a480 comp rx=0 tx=0).stop 2024-09-11T01:54:43.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.387+0000 7f2d10260640 1 -- 172.21.15.12:0/2147171641 shutdown_connections 2024-09-11T01:54:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.387+0000 7f2d10260640 1 --2- 172.21.15.12:0/2147171641 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d08074410 0x7f2d0813e570 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.387+0000 7f2d10260640 1 -- 172.21.15.12:0/2147171641 >> 172.21.15.12:0/2147171641 conn(0x7f2d080a8600 msgr2=0x7f2d080a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.388+0000 7f2d10260640 1 -- 172.21.15.12:0/2147171641 shutdown_connections 2024-09-11T01:54:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.388+0000 7f2d10260640 1 -- 172.21.15.12:0/2147171641 wait complete. 2024-09-11T01:54:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.388+0000 7f2d10260640 10 monclient: build_initial_monmap 2024-09-11T01:54:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.388+0000 7f2d10260640 10 monclient: monmap: 2024-09-11T01:54:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:54:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:54:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:54:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:54:43.403 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:54:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:54:43.404 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.388+0000 7f2d10260640 1 Processor -- start 2024-09-11T01:54:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d10260640 1 -- start start 2024-09-11T01:54:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d10260640 10 monclient: init 2024-09-11T01:54:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d10260640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d10260640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:54:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d10260640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d08074000 0x7f2d08135540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d10260640 10 monclient(hunting): picked mon.noname-c con 0x7f2d08074000 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d10260640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d08074410 0x7f2d0813dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d10260640 10 monclient(hunting): picked mon.noname-b con 0x7f2d08074410 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:54:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d10260640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d08135fc0 0x7f2d0806dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d10260640 10 monclient(hunting): picked mon.noname-a con 0x7f2d08135fc0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:54:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d10260640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d10260640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2d0813ec30 con 0x7f2d08135fc0 2024-09-11T01:54:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d10260640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d10260640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f2d0813eab0 con 0x7f2d08074410 2024-09-11T01:54:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d10260640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d10260640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2d0813edb0 con 0x7f2d08074000 2024-09-11T01:54:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d10260640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d10260640 10 monclient(hunting): authenticate will time out at 2289.832534s 2024-09-11T01:54:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0dffc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d08074000 0x7f2d08135540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0e7fd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d08135fc0 0x7f2d0806dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0dffc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d08074000 0x7f2d08135540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41162/0 (socket says 172.21.15.12:41162) 2024-09-11T01:54:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0dffc640 1 -- 172.21.15.12:0/58773862 learned_addr learned my addr 172.21.15.12:0/58773862 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0dffc640 10 monclient(hunting): get_auth_request con 0x7f2d08074000 auth_method 0 2024-09-11T01:54:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0dffc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0dffc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0dffc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0e7fd640 10 monclient(hunting): get_auth_request con 0x7f2d08135fc0 auth_method 0 2024-09-11T01:54:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0e7fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0e7fd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0e7fd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0d7fb640 1 --2- 172.21.15.12:0/58773862 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d08074410 0x7f2d0813dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0dffc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0dffc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0dffc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0d7fb640 10 monclient(hunting): get_auth_request con 0x7f2d08074410 auth_method 0 2024-09-11T01:54:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0d7fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0d7fb640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0d7fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0e7fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0e7fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0e7fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0d7fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0d7fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0d7fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0dffc640 10 monclient(hunting): handle_auth_done global_id 284135 payload 824 2024-09-11T01:54:43.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0dffc640 1 -- 172.21.15.12:0/58773862 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d08074410 msgr2=0x7f2d0813dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0dffc640 1 --2- 172.21.15.12:0/58773862 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d08074410 0x7f2d0813dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0dffc640 1 -- 172.21.15.12:0/58773862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d08135fc0 msgr2=0x7f2d0806dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0dffc640 1 --2- 172.21.15.12:0/58773862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d08135fc0 0x7f2d0806dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0dffc640 10 monclient: _finish_hunting 0 2024-09-11T01:54:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0dffc640 1 monclient: found mon.noname-c 2024-09-11T01:54:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0dffc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0dffc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:54:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.389+0000 7f2d0dffc640 1 -- 172.21.15.12:0/58773862 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d080a7d40 con 0x7f2d08074000 2024-09-11T01:54:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.392+0000 7f2d0dffc640 1 -- 172.21.15.12:0/58773862 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d08074000 msgr2=0x7f2d08135540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.392+0000 7f2d0dffc640 1 --2- 172.21.15.12:0/58773862 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d08074000 0x7f2d08135540 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2d0400c460 tx=0x7f2d0400c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.392+0000 7f2cfa7fc640 1 -- 172.21.15.12:0/58773862 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d04007070 con 0x7f2d08074000 2024-09-11T01:54:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.392+0000 7f2cfa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:43.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.392+0000 7f2cfa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:54:43.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.392+0000 7f2cfa7fc640 10 monclient: dump: 2024-09-11T01:54:43.409 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:43.409 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.409 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.409 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.409 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:43.409 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.409 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:43.409 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:43.409 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:43.409 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.392+0000 7f2cfa7fc640 10 monclient: _finish_auth 0 2024-09-11T01:54:43.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.392+0000 7f2cfa7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:54:43.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.392+0000 7f2cfa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.392+0000 7f2d10260640 5 monclient: authenticate success, global_id 284135 2024-09-11T01:54:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.392+0000 7f2cfa7fc640 1 -- 172.21.15.12:0/58773862 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d04007230 con 0x7f2d08074000 2024-09-11T01:54:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.392+0000 7f2d10260640 10 monclient: _renew_subs 2024-09-11T01:54:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.392+0000 7f2d10260640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.392+0000 7f2d10260640 1 -- 172.21.15.12:0/58773862 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d0813bfd0 con 0x7f2d08074000 2024-09-11T01:54:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.392+0000 7f2cfa7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.392+0000 7f2cfa7fc640 1 -- 172.21.15.12:0/58773862 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d04007620 con 0x7f2d08074000 2024-09-11T01:54:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.392+0000 7f2cfa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.392+0000 7f2cfa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.392+0000 7f2cfa7fc640 10 monclient: dump: 2024-09-11T01:54:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:43.411 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.392+0000 7f2d10260640 10 monclient: _renew_subs 2024-09-11T01:54:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.392+0000 7f2cfbfff640 4 set_mon_vals no callback set 2024-09-11T01:54:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.392+0000 7f2d10260640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.392+0000 7f2d10260640 1 -- 172.21.15.12:0/58773862 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d0806f750 con 0x7f2d08074000 2024-09-11T01:54:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.393+0000 7f2cfa7fc640 1 -- 172.21.15.12:0/58773862 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f2d0401b400 con 0x7f2d08074000 2024-09-11T01:54:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.393+0000 7f2d10260640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.393+0000 7f2d10260640 1 --2- 172.21.15.12:0/58773862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d0806e1d0 0x7f2d0806e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.393+0000 7f2d10260640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.393+0000 7f2d10260640 1 -- 172.21.15.12:0/58773862 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2d0806a060 con 0x7f2d0806e1d0 2024-09-11T01:54:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.393+0000 7f2d0d7fb640 1 --2- 172.21.15.12:0/58773862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d0806e1d0 0x7f2d0806e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.393+0000 7f2d0d7fb640 10 monclient: get_auth_request con 0x7f2d0806e1d0 auth_method 0 2024-09-11T01:54:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.393+0000 7f2d0d7fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.393+0000 7f2d0d7fb640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.393+0000 7f2d0d7fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.393+0000 7f2d0d7fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.393+0000 7f2d0d7fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.393+0000 7f2d0d7fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.393+0000 7f2cfa7fc640 1 --2- 172.21.15.12:0/58773862 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2cec03e260 0x7f2cec040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.393+0000 7f2d0d7fb640 10 monclient(hunting): handle_auth_done global_id 264312 payload 824 2024-09-11T01:54:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.393+0000 7f2d0d7fb640 1 -- 172.21.15.12:0/58773862 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2d08076fc0 con 0x7f2d0806e1d0 2024-09-11T01:54:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.393+0000 7f2cfa7fc640 1 -- 172.21.15.12:0/58773862 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(858..858 src has 676..858) ==== 4847+0+0 (secure 0 0 0) 0x7f2d04052a40 con 0x7f2d08074000 2024-09-11T01:54:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.393+0000 7f2d0e7fd640 1 --2- 172.21.15.12:0/58773862 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2cec03e260 0x7f2cec040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.393+0000 7f2d0e7fd640 10 monclient: get_auth_request con 0x7f2cec03e260 auth_method 0 2024-09-11T01:54:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.395+0000 7f2d0d7fb640 1 -- 172.21.15.12:0/58773862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d0806e1d0 msgr2=0x7f2d0806e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.395+0000 7f2d0d7fb640 1 --2- 172.21.15.12:0/58773862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d0806e1d0 0x7f2d0806e5b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2d0813b720 tx=0x7f2cfc00fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.396+0000 7f2cfa7fc640 1 -- 172.21.15.12:0/58773862 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc018030 con 0x7f2d0806e1d0 2024-09-11T01:54:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.396+0000 7f2cfa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.396+0000 7f2cfa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:54:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.396+0000 7f2cfa7fc640 10 monclient: dump: 2024-09-11T01:54:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:43.414 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.396+0000 7f2d0e7fd640 1 --2- 172.21.15.12:0/58773862 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2cec03e260 0x7f2cec040720 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f2d08070ba0 tx=0x7f2d0002f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.396+0000 7f2cfa7fc640 1 -- 172.21.15.12:0/58773862 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2cfc00d9b0 con 0x7f2d0806e1d0 2024-09-11T01:54:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.396+0000 7f2cfa7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.396+0000 7f2cfa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:54:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.396+0000 7f2cfa7fc640 1 -- 172.21.15.12:0/58773862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d0806e1d0 msgr2=0x7f2d0806e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.396+0000 7f2cfa7fc640 1 --2- 172.21.15.12:0/58773862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d0806e1d0 0x7f2d0806e5b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2d0813b720 tx=0x7f2cfc00fc00 comp rx=0 tx=0).stop 2024-09-11T01:54:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.396+0000 7f2cfa7fc640 1 -- 172.21.15.12:0/58773862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d0806e1d0 msgr2=0x7f2d0806e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:54:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.396+0000 7f2cfa7fc640 1 --2- 172.21.15.12:0/58773862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d0806e1d0 0x7f2d0806e5b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.402+0000 7f2d10260640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:54:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.402+0000 7f2d10260640 1 --2- 172.21.15.12:0/58773862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d08136bf0 0x7f2d08137000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.402+0000 7f2d10260640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.402+0000 7f2d10260640 1 -- 172.21.15.12:0/58773862 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2d081363a0 con 0x7f2d08136bf0 2024-09-11T01:54:43.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.402+0000 7f2d0d7fb640 1 --2- 172.21.15.12:0/58773862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d08136bf0 0x7f2d08137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.402+0000 7f2d0d7fb640 10 monclient: get_auth_request con 0x7f2d08136bf0 auth_method 0 2024-09-11T01:54:43.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.402+0000 7f2d0d7fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.402+0000 7f2d0d7fb640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.402+0000 7f2d0d7fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.402+0000 7f2d0d7fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.402+0000 7f2d0d7fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.402+0000 7f2d0d7fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.402+0000 7f2d0d7fb640 10 monclient(hunting): handle_auth_done global_id 264315 payload 824 2024-09-11T01:54:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.402+0000 7f2d0d7fb640 1 -- 172.21.15.12:0/58773862 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2d0813c650 con 0x7f2d08136bf0 2024-09-11T01:54:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.405+0000 7f2d0d7fb640 1 -- 172.21.15.12:0/58773862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d08136bf0 msgr2=0x7f2d08137000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.405+0000 7f2d0d7fb640 1 --2- 172.21.15.12:0/58773862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d08136bf0 0x7f2d08137000 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2d0813b720 tx=0x7f2cfc003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.405+0000 7f2cfa7fc640 1 -- 172.21.15.12:0/58773862 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc01dd20 con 0x7f2d08136bf0 2024-09-11T01:54:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.405+0000 7f2cfa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.405+0000 7f2cfa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:54:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.405+0000 7f2cfa7fc640 10 monclient: dump: 2024-09-11T01:54:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:43.419 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.405+0000 7f2cfa7fc640 1 -- 172.21.15.12:0/58773862 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f2cfc01dd20 con 0x7f2d08136bf0 2024-09-11T01:54:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.405+0000 7f2cfa7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:54:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.405+0000 7f2cfa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:54:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.405+0000 7f2cfa7fc640 1 -- 172.21.15.12:0/58773862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d08136bf0 msgr2=0x7f2d08137000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.405+0000 7f2cfa7fc640 1 --2- 172.21.15.12:0/58773862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d08136bf0 0x7f2d08137000 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2d0813b720 tx=0x7f2cfc003760 comp rx=0 tx=0).stop 2024-09-11T01:54:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.405+0000 7f2cfa7fc640 1 -- 172.21.15.12:0/58773862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d08136bf0 msgr2=0x7f2d08137000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:54:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.405+0000 7f2cfa7fc640 1 --2- 172.21.15.12:0/58773862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d08136bf0 0x7f2d08137000 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.420 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:54:43.420 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:54:43.420 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:54:43.420 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:54:43.420 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 232, 2024-09-11T01:54:43.420 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 125112, 2024-09-11T01:54:43.420 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:54:43.421 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:54:43.421 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:54:43.421 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:54:43.421 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:43.421 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:54:43.421 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:54:43.421 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:54:43.421 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:54:43.421 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:43.421 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:43.421 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:43.421 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:43.421 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:43.422 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:43.422 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:43.422 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:43.422 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:43.422 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:43.422 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:43.422 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:43.422 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:54:43.422 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:54:43.422 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:43.422 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:43.422 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:43.422 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:54:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:54:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:54:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:54:43.423 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:54:43.424 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:54:43.424 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:54:43.424 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:54:43.424 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:54:43.424 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:54:43.424 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:54:43.424 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:54:43.424 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:54:43.424 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:54:43.424 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:54:43.424 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:54:43.424 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:54:43.425 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:43.425 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:43.425 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:43.425 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:43.425 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:43.425 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:43.426 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:43.426 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:43.426 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:43.426 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:43.426 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:43.426 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:43.426 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:54:43.426 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.426 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:54:43.426 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.426 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:54:43.426 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:54:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:54:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:54:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:43.427 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:43.428 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.428 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:54:43.428 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:54:43.428 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:43.428 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:43.428 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:43.428 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.428 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.428 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:54:43.428 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:54:43.428 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:43.428 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:43.428 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.429 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:43.429 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:54:43.429 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:43.429 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.429 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.429 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:43.429 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:54:43.429 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:43.429 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:43.429 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:43.429 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.429 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:54:43.429 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:54:43.429 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:43.430 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:43.430 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:43.430 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.430 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.430 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:54:43.430 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:54:43.430 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:43.430 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:43.430 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.430 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:43.430 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:54:43.430 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:43.430 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.431 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.431 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:43.431 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:54:43.431 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:43.431 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:43.431 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:43.431 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.431 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:54:43.431 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:54:43.431 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:43.431 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:43.431 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:43.431 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:43.431 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:43.432 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.432 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:54:43.432 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:54:43.432 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.432 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:43.432 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:43.432 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:54:43.432 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:43.432 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:43.432 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:54:43.432 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.432 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:43.432 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:43.432 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:54:43.433 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:43.433 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:43.433 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:54:43.433 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.433 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:43.433 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:43.433 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:54:43.433 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:43.433 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:43.433 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.433 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:54:43.433 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:54:43.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.405+0000 7f2d10260640 1 -- 172.21.15.12:0/58773862 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2cec03e260 msgr2=0x7f2cec040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.405+0000 7f2d10260640 1 --2- 172.21.15.12:0/58773862 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f2cec03e260 0x7f2cec040720 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f2d08070ba0 tx=0x7f2d0002f000 comp rx=0 tx=0).stop 2024-09-11T01:54:43.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.405+0000 7f2d10260640 10 monclient: shutdown 2024-09-11T01:54:43.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.405+0000 7f2d10260640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:43.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.405+0000 7f2d10260640 1 -- 172.21.15.12:0/58773862 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d08074000 msgr2=0x7f2d08135540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.405+0000 7f2d10260640 1 --2- 172.21.15.12:0/58773862 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d08074000 0x7f2d08135540 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2d0400c460 tx=0x7f2d0400c920 comp rx=0 tx=0).stop 2024-09-11T01:54:43.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.405+0000 7f2d0e7fd640 1 -- 172.21.15.12:0/58773862 reap_dead start 2024-09-11T01:54:43.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.405+0000 7f2d10260640 1 -- 172.21.15.12:0/58773862 shutdown_connections 2024-09-11T01:54:43.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.405+0000 7f2d10260640 1 --2- 172.21.15.12:0/58773862 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2d08074000 0x7f2d08135540 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.405+0000 7f2d10260640 1 -- 172.21.15.12:0/58773862 >> 172.21.15.12:0/58773862 conn(0x7f2d080a8600 msgr2=0x7f2d080a15b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:43.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.405+0000 7f2d10260640 1 -- 172.21.15.12:0/58773862 shutdown_connections 2024-09-11T01:54:43.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.405+0000 7f2d10260640 1 -- 172.21.15.12:0/58773862 wait complete. 2024-09-11T01:54:43.436 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:54:43.436 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:54:43.436 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:54:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.535+0000 7ff9ee479640 10 monclient: get_monmap_and_config 2024-09-11T01:54:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.535+0000 7ff9ee479640 10 monclient: build_initial_monmap 2024-09-11T01:54:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.535+0000 7ff9ee479640 10 monclient: monmap: 2024-09-11T01:54:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.538 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:54:43.537475+0000 2024-09-11T01:54:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:54:43.537475+0000 2024-09-11T01:54:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:54:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:54:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:54:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:54:43.539 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:54:43.539 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.536+0000 7ff9ee479640 1 Processor -- start 2024-09-11T01:54:43.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.536+0000 7ff9ee479640 1 -- start start 2024-09-11T01:54:43.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.536+0000 7ff9ee479640 10 monclient: init 2024-09-11T01:54:43.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9ee479640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:43.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9ee479640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-11T01:54:43.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9ee479640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff9e80a1380 0x7ff9e80a1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9ee479640 10 monclient(hunting): picked mon.noname-a con 0x7ff9e80a1380 addr v2:172.21.15.12:3300/0 2024-09-11T01:54:43.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9ee479640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff9e80a0370 0x7ff9e8135ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9ee479640 10 monclient(hunting): picked mon.noname-c con 0x7ff9e80a0370 addr v2:172.21.15.71:3300/0 2024-09-11T01:54:43.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9ee479640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff9e80a0c90 addr v1:172.21.15.71:6789/0 2024-09-11T01:54:43.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9ee479640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9e80a1ca0 con 0x7ff9e80a0c90 2024-09-11T01:54:43.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9ee479640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9ee479640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff9e80a21c0 con 0x7ff9e80a1380 2024-09-11T01:54:43.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9ee479640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9ee479640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7ff9e80a24a0 con 0x7ff9e80a0370 2024-09-11T01:54:43.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9ee479640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:43.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9ee479640 10 monclient(hunting): authenticate will time out at 2289.980203s 2024-09-11T01:54:43.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e7fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff9e80a1380 0x7ff9e80a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e7fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff9e80a1380 0x7ff9e80a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43628/0 (socket says 172.21.15.12:43628) 2024-09-11T01:54:43.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e7fff640 1 -- 172.21.15.12:0/1557963106 learned_addr learned my addr 172.21.15.12:0/1557963106 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:43.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e7fff640 10 monclient(hunting): get_auth_request con 0x7ff9e80a1380 auth_method 0 2024-09-11T01:54:43.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e77fe640 1 --2- 172.21.15.12:0/1557963106 >> v2:172.21.15.71:3300/0 conn(0x7ff9e80a0370 0x7ff9e8135ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e77fe640 10 monclient(hunting): get_auth_request con 0x7ff9e80a0370 auth_method 0 2024-09-11T01:54:43.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e7fff640 10 monclient(hunting): handle_auth_done global_id 264318 payload 274 2024-09-11T01:54:43.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e7fff640 1 -- 172.21.15.12:0/1557963106 >> v2:172.21.15.71:3300/0 conn(0x7ff9e80a0370 msgr2=0x7ff9e8135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:54:43.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e7fff640 1 --2- 172.21.15.12:0/1557963106 >> v2:172.21.15.71:3300/0 conn(0x7ff9e80a0370 0x7ff9e8135ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e7fff640 1 -- 172.21.15.12:0/1557963106 >> v1:172.21.15.71:6789/0 conn(0x7ff9e80a0c90 legacy=0x7ff9e8136020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:54:43.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e7fff640 10 monclient: _finish_hunting 0 2024-09-11T01:54:43.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e7fff640 1 monclient: found mon.noname-a 2024-09-11T01:54:43.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:43.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:54:43.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.537+0000 7ff9e7fff640 1 -- 172.21.15.12:0/1557963106 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9e80a2780 con 0x7ff9e80a1380 2024-09-11T01:54:43.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.540+0000 7ff9e7fff640 1 -- 172.21.15.12:0/1557963106 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9e80a1380 msgr2=0x7ff9e80a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:43.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.540+0000 7ff9e7fff640 1 --2- 172.21.15.12:0/1557963106 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9e80a1380 0x7ff9e80a1760 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff9e80a4140 tx=0x7ff9d802a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82e5ae35de8c0dc6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.540+0000 7ff9e6ffd640 1 -- 172.21.15.12:0/1557963106 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8037070 con 0x7ff9e80a1380 2024-09-11T01:54:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.540+0000 7ff9e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.540+0000 7ff9e6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:54:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.540+0000 7ff9e6ffd640 10 monclient: dump: 2024-09-11T01:54:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:43.553 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:43.554 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.540+0000 7ff9e6ffd640 10 monclient: _finish_auth 0 2024-09-11T01:54:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.540+0000 7ff9e6ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:54:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.540+0000 7ff9e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.540+0000 7ff9e6ffd640 1 -- 172.21.15.12:0/1557963106 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9d8032640 con 0x7ff9e80a1380 2024-09-11T01:54:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.540+0000 7ff9ee479640 5 monclient: authenticate success, global_id 264318 2024-09-11T01:54:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.540+0000 7ff9e6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.540+0000 7ff9ee479640 10 monclient: get_monmap_and_config success 2024-09-11T01:54:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.540+0000 7ff9e6ffd640 1 -- 172.21.15.12:0/1557963106 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8032a10 con 0x7ff9e80a1380 2024-09-11T01:54:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.540+0000 7ff9ee479640 4 set_mon_vals no callback set 2024-09-11T01:54:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.540+0000 7ff9e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.540+0000 7ff9e6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:54:43.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.540+0000 7ff9e6ffd640 10 monclient: dump: 2024-09-11T01:54:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:43.555 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.541+0000 7ff9ee479640 10 monclient: shutdown 2024-09-11T01:54:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.541+0000 7ff9ee479640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:43.555 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.541+0000 7ff9ee479640 1 -- 172.21.15.12:0/1557963106 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9e80a1380 msgr2=0x7ff9e80a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.541+0000 7ff9ee479640 1 --2- 172.21.15.12:0/1557963106 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9e80a1380 0x7ff9e80a1760 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff9e80a4140 tx=0x7ff9d802a0b0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.541+0000 7ff9ee479640 1 -- 172.21.15.12:0/1557963106 shutdown_connections 2024-09-11T01:54:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.541+0000 7ff9ee479640 1 --2- 172.21.15.12:0/1557963106 >> v2:172.21.15.71:3300/0 conn(0x7ff9e80a0370 0x7ff9e8135ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.541+0000 7ff9ee479640 1 --2- 172.21.15.12:0/1557963106 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9e80a1380 0x7ff9e80a1760 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.541+0000 7ff9ee479640 1 -- 172.21.15.12:0/1557963106 >> 172.21.15.12:0/1557963106 conn(0x7ff9e80a2f70 msgr2=0x7ff9e80a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.541+0000 7ff9ee479640 1 -- 172.21.15.12:0/1557963106 shutdown_connections 2024-09-11T01:54:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.541+0000 7ff9ee479640 1 -- 172.21.15.12:0/1557963106 wait complete. 2024-09-11T01:54:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.541+0000 7ff9ee479640 10 monclient: build_initial_monmap 2024-09-11T01:54:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.541+0000 7ff9ee479640 10 monclient: monmap: 2024-09-11T01:54:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:54:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:54:43.556 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:54:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:54:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:54:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:54:43.557 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.542+0000 7ff9ee479640 1 Processor -- start 2024-09-11T01:54:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.542+0000 7ff9ee479640 1 -- start start 2024-09-11T01:54:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.542+0000 7ff9ee479640 10 monclient: init 2024-09-11T01:54:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.542+0000 7ff9ee479640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.542+0000 7ff9ee479640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:54:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.542+0000 7ff9ee479640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9e80a0370 0x7ff9e8135c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.542+0000 7ff9ee479640 10 monclient(hunting): picked mon.noname-c con 0x7ff9e80a0370 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:43.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.542+0000 7ff9ee479640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9e80a0c90 0x7ff9e8136160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.542+0000 7ff9ee479640 10 monclient(hunting): picked mon.noname-a con 0x7ff9e80a0c90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:54:43.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.542+0000 7ff9ee479640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9e80a1380 0x7ff9e81cce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.542+0000 7ff9ee479640 10 monclient(hunting): picked mon.noname-b con 0x7ff9e80a1380 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:54:43.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.542+0000 7ff9ee479640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.542+0000 7ff9ee479640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e809ce20 con 0x7ff9e80a0c90 2024-09-11T01:54:43.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.542+0000 7ff9ee479640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.542+0000 7ff9ee479640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e809cca0 con 0x7ff9e80a1380 2024-09-11T01:54:43.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.542+0000 7ff9ee479640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.542+0000 7ff9ee479640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff9e8136730 con 0x7ff9e80a0370 2024-09-11T01:54:43.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.542+0000 7ff9ee479640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:43.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.542+0000 7ff9ee479640 10 monclient(hunting): authenticate will time out at 2289.985904s 2024-09-11T01:54:43.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.542+0000 7ff9e7fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9e80a0370 0x7ff9e8135c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.542+0000 7ff9e77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9e80a0c90 0x7ff9e8136160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.542+0000 7ff9e77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9e80a0c90 0x7ff9e8136160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43636/0 (socket says 172.21.15.12:43636) 2024-09-11T01:54:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.542+0000 7ff9e7fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9e80a0370 0x7ff9e8135c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41168/0 (socket says 172.21.15.12:41168) 2024-09-11T01:54:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9e77fe640 1 -- 172.21.15.12:0/784789668 learned_addr learned my addr 172.21.15.12:0/784789668 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9e77fe640 10 monclient(hunting): get_auth_request con 0x7ff9e80a0c90 auth_method 0 2024-09-11T01:54:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9e7fff640 10 monclient(hunting): get_auth_request con 0x7ff9e80a0370 auth_method 0 2024-09-11T01:54:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9eca16640 1 --2- 172.21.15.12:0/784789668 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9e80a1380 0x7ff9e81cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9e77fe640 10 monclient(hunting): handle_auth_done global_id 264321 payload 824 2024-09-11T01:54:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9e77fe640 1 -- 172.21.15.12:0/784789668 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9e80a0370 msgr2=0x7ff9e8135c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9e77fe640 1 --2- 172.21.15.12:0/784789668 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9e80a0370 0x7ff9e8135c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9e77fe640 1 -- 172.21.15.12:0/784789668 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9e80a1380 msgr2=0x7ff9e81cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9e77fe640 1 --2- 172.21.15.12:0/784789668 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9e80a1380 0x7ff9e81cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9e77fe640 10 monclient: _finish_hunting 0 2024-09-11T01:54:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9e77fe640 1 monclient: found mon.noname-a 2024-09-11T01:54:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:43.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9e77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:54:43.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9e77fe640 1 -- 172.21.15.12:0/784789668 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9e8129740 con 0x7ff9e80a0c90 2024-09-11T01:54:43.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9eca16640 10 monclient: get_auth_request con 0x7ff9e80a1380 auth_method 0 2024-09-11T01:54:43.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9e7fff640 1 --2- 172.21.15.12:0/784789668 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9e80a0370 0x7ff9e8135c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:54:43.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.543+0000 7ff9eca16640 1 --2- 172.21.15.12:0/784789668 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9e80a1380 0x7ff9e81cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:54:43.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.545+0000 7ff9e77fe640 1 -- 172.21.15.12:0/784789668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9e80a0c90 msgr2=0x7ff9e8136160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:43.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.545+0000 7ff9e77fe640 1 --2- 172.21.15.12:0/784789668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9e80a0c90 0x7ff9e8136160 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc00e980 tx=0x7ff9dc00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:43.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.545+0000 7ff9e4ff9640 1 -- 172.21.15.12:0/784789668 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc017070 con 0x7ff9e80a0c90 2024-09-11T01:54:43.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.545+0000 7ff9e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:43.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.545+0000 7ff9e4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:54:43.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.545+0000 7ff9e4ff9640 10 monclient: dump: 2024-09-11T01:54:43.561 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:43.562 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.545+0000 7ff9e4ff9640 10 monclient: _finish_auth 0 2024-09-11T01:54:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.545+0000 7ff9e4ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:54:43.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.545+0000 7ff9e4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:43.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.545+0000 7ff9ee479640 5 monclient: authenticate success, global_id 264321 2024-09-11T01:54:43.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.545+0000 7ff9e4ff9640 1 -- 172.21.15.12:0/784789668 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9dc00cc10 con 0x7ff9e80a0c90 2024-09-11T01:54:43.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.545+0000 7ff9e4ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:43.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.545+0000 7ff9e4ff9640 1 -- 172.21.15.12:0/784789668 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc013470 con 0x7ff9e80a0c90 2024-09-11T01:54:43.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.545+0000 7ff9ee479640 10 monclient: _renew_subs 2024-09-11T01:54:43.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.545+0000 7ff9ee479640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:54:43.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.545+0000 7ff9ee479640 1 -- 172.21.15.12:0/784789668 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9e809d480 con 0x7ff9e80a0c90 2024-09-11T01:54:43.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.545+0000 7ff9e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:43.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.545+0000 7ff9e4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:54:43.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.545+0000 7ff9e4ff9640 10 monclient: dump: 2024-09-11T01:54:43.563 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:43.563 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.563 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.564 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.564 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:43.564 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.564 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:43.564 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:43.564 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:43.564 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.545+0000 7ff9e67fc640 4 set_mon_vals no callback set 2024-09-11T01:54:43.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.545+0000 7ff9ee479640 10 monclient: _renew_subs 2024-09-11T01:54:43.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.545+0000 7ff9ee479640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:54:43.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.545+0000 7ff9ee479640 1 -- 172.21.15.12:0/784789668 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9e809f230 con 0x7ff9e80a0c90 2024-09-11T01:54:43.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.546+0000 7ff9ee479640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:43.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.546+0000 7ff9ee479640 1 --2- 172.21.15.12:0/784789668 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9e802d440 0x7ff9e802d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.546+0000 7ff9ee479640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.546+0000 7ff9ee479640 1 -- 172.21.15.12:0/784789668 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff9e80703d0 con 0x7ff9e802d440 2024-09-11T01:54:43.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.546+0000 7ff9e7fff640 1 --2- 172.21.15.12:0/784789668 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9e802d440 0x7ff9e802d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.546+0000 7ff9e7fff640 10 monclient: get_auth_request con 0x7ff9e802d440 auth_method 0 2024-09-11T01:54:43.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.546+0000 7ff9e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.546+0000 7ff9e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.546+0000 7ff9e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.546+0000 7ff9e77fe640 1 --2- 172.21.15.12:0/784789668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9e80a0c90 0x7ff9e8136160 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc00e980 tx=0x7ff9dc00ee40 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7ff9dc024060 mgrmap(e 5) 2024-09-11T01:54:43.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.546+0000 7ff9e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.546+0000 7ff9e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.546+0000 7ff9e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.546+0000 7ff9e4ff9640 1 -- 172.21.15.12:0/784789668 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff9dc024060 con 0x7ff9e80a0c90 2024-09-11T01:54:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.546+0000 7ff9e7fff640 10 monclient(hunting): handle_auth_done global_id 284141 payload 824 2024-09-11T01:54:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.546+0000 7ff9e7fff640 1 -- 172.21.15.12:0/784789668 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff9e80a1760 con 0x7ff9e802d440 2024-09-11T01:54:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.547+0000 7ff9e4ff9640 1 --2- 172.21.15.12:0/784789668 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff9d403e240 0x7ff9d4040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.547+0000 7ff9eca16640 1 --2- 172.21.15.12:0/784789668 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff9d403e240 0x7ff9d4040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.547+0000 7ff9eca16640 10 monclient: get_auth_request con 0x7ff9d403e240 auth_method 0 2024-09-11T01:54:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.549+0000 7ff9e7fff640 1 -- 172.21.15.12:0/784789668 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9e802d440 msgr2=0x7ff9e802d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.549+0000 7ff9e7fff640 1 --2- 172.21.15.12:0/784789668 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9e802d440 0x7ff9e802d820 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8034000 tx=0x7ff9d80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.549+0000 7ff9e4ff9640 1 -- 172.21.15.12:0/784789668 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8037040 con 0x7ff9e802d440 2024-09-11T01:54:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.549+0000 7ff9e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.549+0000 7ff9e4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:43.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.549+0000 7ff9e4ff9640 10 monclient: dump: 2024-09-11T01:54:43.567 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:43.567 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.567 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.567 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.567 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:43.567 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.567 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:43.567 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:43.567 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:43.567 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.549+0000 7ff9e4ff9640 1 -- 172.21.15.12:0/784789668 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff9d803ca60 con 0x7ff9e802d440 2024-09-11T01:54:43.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.549+0000 7ff9e4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:43.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.549+0000 7ff9e4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:54:43.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.549+0000 7ff9e4ff9640 1 -- 172.21.15.12:0/784789668 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9e802d440 msgr2=0x7ff9e802d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.549+0000 7ff9e4ff9640 1 --2- 172.21.15.12:0/784789668 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9e802d440 0x7ff9e802d820 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8034000 tx=0x7ff9d80046c0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.549+0000 7ff9e4ff9640 1 -- 172.21.15.12:0/784789668 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9e802d440 msgr2=0x7ff9e802d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:54:43.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.549+0000 7ff9e4ff9640 1 --2- 172.21.15.12:0/784789668 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9e802d440 0x7ff9e802d820 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8034000 tx=0x7ff9d80046c0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.549+0000 7ff9eca16640 1 --2- 172.21.15.12:0/784789668 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff9d403e240 0x7ff9d4040700 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ff9e8128880 tx=0x7ff9e0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:43.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.555+0000 7ff9ee479640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:54:43.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.555+0000 7ff9ee479640 1 --2- 172.21.15.12:0/784789668 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9e802dd60 0x7ff9e802e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.555+0000 7ff9ee479640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.555+0000 7ff9ee479640 1 -- 172.21.15.12:0/784789668 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff9e8120000 con 0x7ff9e802dd60 2024-09-11T01:54:43.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.555+0000 7ff9e7fff640 1 --2- 172.21.15.12:0/784789668 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9e802dd60 0x7ff9e802e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.555+0000 7ff9e7fff640 10 monclient: get_auth_request con 0x7ff9e802dd60 auth_method 0 2024-09-11T01:54:43.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.555+0000 7ff9e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.555+0000 7ff9e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.555+0000 7ff9e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.555+0000 7ff9e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.555+0000 7ff9e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.555+0000 7ff9e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.556+0000 7ff9e7fff640 10 monclient(hunting): handle_auth_done global_id 284144 payload 824 2024-09-11T01:54:43.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.556+0000 7ff9e7fff640 1 -- 172.21.15.12:0/784789668 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff9e81280d0 con 0x7ff9e802dd60 2024-09-11T01:54:43.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.558+0000 7ff9e7fff640 1 -- 172.21.15.12:0/784789668 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9e802dd60 msgr2=0x7ff9e802e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:43.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.558+0000 7ff9e7fff640 1 --2- 172.21.15.12:0/784789668 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9e802dd60 0x7ff9e802e140 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff9d802a0f0 tx=0x7ff9d8009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:43.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.558+0000 7ff9e4ff9640 1 -- 172.21.15.12:0/784789668 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8037030 con 0x7ff9e802dd60 2024-09-11T01:54:43.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.558+0000 7ff9e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:43.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.558+0000 7ff9e4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:43.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.558+0000 7ff9e4ff9640 10 monclient: dump: 2024-09-11T01:54:43.571 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:43.571 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.571 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.572 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.572 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:43.572 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.572 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:43.572 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:43.572 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:43.572 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.558+0000 7ff9e4ff9640 1 -- 172.21.15.12:0/784789668 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7ff9d8004440 con 0x7ff9e802dd60 2024-09-11T01:54:43.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.558+0000 7ff9e4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:54:43.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.558+0000 7ff9e4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:54:43.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.558+0000 7ff9e4ff9640 1 -- 172.21.15.12:0/784789668 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9e802dd60 msgr2=0x7ff9e802e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.558+0000 7ff9e4ff9640 1 --2- 172.21.15.12:0/784789668 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9e802dd60 0x7ff9e802e140 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff9d802a0f0 tx=0x7ff9d8009240 comp rx=0 tx=0).stop 2024-09-11T01:54:43.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.558+0000 7ff9e4ff9640 1 -- 172.21.15.12:0/784789668 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9e802dd60 msgr2=0x7ff9e802e140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:54:43.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.558+0000 7ff9e4ff9640 1 --2- 172.21.15.12:0/784789668 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff9e802dd60 0x7ff9e802e140 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff9d802a0f0 tx=0x7ff9d8009240 comp rx=0 tx=0).stop 2024-09-11T01:54:43.573 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:54:43.573 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:54:43.573 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:54:43.573 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:54:43.573 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 232, 2024-09-11T01:54:43.573 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 22754, 2024-09-11T01:54:43.573 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:54:43.573 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:54:43.573 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:54:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:54:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:54:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:54:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:54:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:54:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:43.574 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:54:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:54:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:43.575 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:54:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:54:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:54:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:54:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:54:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:54:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:54:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:54:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:54:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:54:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:54:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:54:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:54:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:54:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:54:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:54:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:54:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:54:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:54:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:54:43.578 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:54:43.579 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:43.579 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:43.579 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.579 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:43.579 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:54:43.579 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:43.579 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.579 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.579 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:43.579 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:54:43.579 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:43.579 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:43.579 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:43.579 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.580 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:54:43.580 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:54:43.580 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:43.580 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:43.580 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:43.580 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.580 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.580 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:54:43.580 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:54:43.580 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:43.580 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:43.580 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.580 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:43.581 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:54:43.581 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:43.581 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.581 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.581 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:43.581 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:54:43.581 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:43.581 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:43.581 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:43.581 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.581 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:54:43.581 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:54:43.581 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:43.581 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:43.582 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:43.582 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.582 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.582 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:54:43.582 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:54:43.582 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:43.582 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:43.582 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.582 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:43.583 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:54:43.583 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:43.583 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.583 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.583 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:43.583 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:54:43.583 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:43.583 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:43.583 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:43.583 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.583 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:54:43.583 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:54:43.583 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:43.584 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:43.584 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:43.584 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:43.584 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:43.584 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.584 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:54:43.584 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:54:43.584 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.584 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:43.584 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:43.584 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:54:43.584 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:43.584 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:43.584 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:54:43.585 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.585 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:43.585 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:43.585 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:54:43.585 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:43.585 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:43.585 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.585 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:54:43.585 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:54:43.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.558+0000 7ff9ee479640 1 -- 172.21.15.12:0/784789668 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff9d403e240 msgr2=0x7ff9d4040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.558+0000 7ff9ee479640 1 --2- 172.21.15.12:0/784789668 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff9d403e240 0x7ff9d4040700 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ff9e8128880 tx=0x7ff9e0009040 comp rx=0 tx=0).stop 2024-09-11T01:54:43.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.558+0000 7ff9ee479640 10 monclient: shutdown 2024-09-11T01:54:43.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.558+0000 7ff9ee479640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:43.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.558+0000 7ff9ee479640 1 -- 172.21.15.12:0/784789668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9e80a0c90 msgr2=0x7ff9e8136160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.558+0000 7ff9ee479640 1 --2- 172.21.15.12:0/784789668 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9e80a0c90 0x7ff9e8136160 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc00e980 tx=0x7ff9dc00ee40 comp rx=0 tx=0).stop 2024-09-11T01:54:43.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.558+0000 7ff9eca16640 1 -- 172.21.15.12:0/784789668 reap_dead start 2024-09-11T01:54:43.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.558+0000 7ff9e4ff9640 1 -- 172.21.15.12:0/784789668 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(858..858 src has 676..858) ==== 4847+0+0 (unknown 0 0 0) 0x7ff9dc052d80 con 0x7ff9e80a0c90 2024-09-11T01:54:43.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.558+0000 7ff9eca16640 1 -- 172.21.15.12:0/784789668 reap_dead start 2024-09-11T01:54:43.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.558+0000 7ff9ee479640 1 -- 172.21.15.12:0/784789668 shutdown_connections 2024-09-11T01:54:43.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.558+0000 7ff9ee479640 1 -- 172.21.15.12:0/784789668 >> 172.21.15.12:0/784789668 conn(0x7ff9e80a2f70 msgr2=0x7ff9e80a5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:43.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.559+0000 7ff9ee479640 1 -- 172.21.15.12:0/784789668 shutdown_connections 2024-09-11T01:54:43.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.559+0000 7ff9ee479640 1 -- 172.21.15.12:0/784789668 wait complete. 2024-09-11T01:54:43.588 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:54:43.588 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:54:43.588 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:54:43.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.687+0000 7f4252df3640 10 monclient: get_monmap_and_config 2024-09-11T01:54:43.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.687+0000 7f4252df3640 10 monclient: build_initial_monmap 2024-09-11T01:54:43.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.687+0000 7f4252df3640 10 monclient: monmap: 2024-09-11T01:54:43.691 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:43.691 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.692 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:54:43.689388+0000 2024-09-11T01:54:43.692 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:54:43.689388+0000 2024-09-11T01:54:43.692 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:43.692 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.692 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:54:43.692 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:54:43.692 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:54:43.693 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:54:43.693 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:54:43.693 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.688+0000 7f4252df3640 1 Processor -- start 2024-09-11T01:54:43.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.688+0000 7f4252df3640 1 -- start start 2024-09-11T01:54:43.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.688+0000 7f4252df3640 10 monclient: init 2024-09-11T01:54:43.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.688+0000 7f4252df3640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:43.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.688+0000 7f4252df3640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-11T01:54:43.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.688+0000 7f4252df3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f424c134ad0 addr v1:172.21.15.12:6789/0 2024-09-11T01:54:43.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.688+0000 7f4252df3640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f424c0a2200 0x7f424c13a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.688+0000 7f4252df3640 10 monclient(hunting): picked mon.noname-c con 0x7f424c0a2200 addr v2:172.21.15.71:3300/0 2024-09-11T01:54:43.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.688+0000 7f4252df3640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f424c1355c0 0x7f424c13c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.688+0000 7f4252df3640 10 monclient(hunting): picked mon.noname-a con 0x7f424c1355c0 addr v2:172.21.15.12:3300/0 2024-09-11T01:54:43.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.688+0000 7f4252df3640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f424c09ed30 con 0x7f424c134ad0 2024-09-11T01:54:43.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.688+0000 7f4252df3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.688+0000 7f4252df3640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f424c09f5d0 con 0x7f424c1355c0 2024-09-11T01:54:43.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.688+0000 7f4252df3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.688+0000 7f4252df3640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f424c09f010 con 0x7f424c0a2200 2024-09-11T01:54:43.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.688+0000 7f4252df3640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:43.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.688+0000 7f4252df3640 10 monclient(hunting): authenticate will time out at 2290.131950s 2024-09-11T01:54:43.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f4251390640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f424c1355c0 0x7f424c13c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f4251390640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f424c1355c0 0x7f424c13c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43650/0 (socket says 172.21.15.12:43650) 2024-09-11T01:54:43.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f4251390640 1 -- 172.21.15.12:0/3170379485 learned_addr learned my addr 172.21.15.12:0/3170379485 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:43.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f4251390640 10 monclient(hunting): get_auth_request con 0x7f424c1355c0 auth_method 0 2024-09-11T01:54:43.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f4251390640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f4251390640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f4251390640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f4251390640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f4251390640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f4251390640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f424bfff640 1 --2- 172.21.15.12:0/3170379485 >> v2:172.21.15.71:3300/0 conn(0x7f424c0a2200 0x7f424c13a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f424bfff640 10 monclient(hunting): get_auth_request con 0x7f424c0a2200 auth_method 0 2024-09-11T01:54:43.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f424bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f424bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f424bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f424bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f424bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f424bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f424bfff640 10 monclient(hunting): handle_auth_done global_id 274204 payload 274 2024-09-11T01:54:43.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f424bfff640 1 -- 172.21.15.12:0/3170379485 >> v2:172.21.15.12:3300/0 conn(0x7f424c1355c0 msgr2=0x7f424c13c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:54:43.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f424bfff640 1 --2- 172.21.15.12:0/3170379485 >> v2:172.21.15.12:3300/0 conn(0x7f424c1355c0 0x7f424c13c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f424bfff640 1 -- 172.21.15.12:0/3170379485 >> v1:172.21.15.12:6789/0 conn(0x7f424c134ad0 legacy=0x7f424c134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:54:43.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f424bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:54:43.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f424bfff640 1 monclient: found mon.noname-c 2024-09-11T01:54:43.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f424bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:43.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f424bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:54:43.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.689+0000 7f424bfff640 1 -- 172.21.15.12:0/3170379485 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f424c09f2f0 con 0x7f424c0a2200 2024-09-11T01:54:43.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.692+0000 7f424bfff640 1 -- 172.21.15.12:0/3170379485 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c0a2200 msgr2=0x7f424c13a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:43.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.692+0000 7f424bfff640 1 --2- 172.21.15.12:0/3170379485 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c0a2200 0x7f424c13a000 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f42380097a0 tx=0x7f423802a010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0596bae9bf9317 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:43.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.692+0000 7f424b7fe640 1 -- 172.21.15.12:0/3170379485 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4238037070 con 0x7f424c0a2200 2024-09-11T01:54:43.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.692+0000 7f424b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:43.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.692+0000 7f424b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:54:43.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.692+0000 7f424b7fe640 10 monclient: dump: 2024-09-11T01:54:43.708 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:43.708 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.708 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.708 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.708 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:43.708 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.708 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:43.708 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:43.708 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:43.708 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.692+0000 7f424b7fe640 10 monclient: _finish_auth 0 2024-09-11T01:54:43.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.692+0000 7f424b7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:54:43.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.692+0000 7f424b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:43.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.692+0000 7f424b7fe640 1 -- 172.21.15.12:0/3170379485 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4238032440 con 0x7f424c0a2200 2024-09-11T01:54:43.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.692+0000 7f424b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:43.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.692+0000 7f424b7fe640 1 -- 172.21.15.12:0/3170379485 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4238032810 con 0x7f424c0a2200 2024-09-11T01:54:43.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.692+0000 7f4252df3640 5 monclient: authenticate success, global_id 274204 2024-09-11T01:54:43.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.692+0000 7f4252df3640 10 monclient: get_monmap_and_config success 2024-09-11T01:54:43.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.692+0000 7f4252df3640 4 set_mon_vals no callback set 2024-09-11T01:54:43.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.692+0000 7f424b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:43.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.692+0000 7f424b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:54:43.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.692+0000 7f424b7fe640 10 monclient: dump: 2024-09-11T01:54:43.709 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:43.709 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.709 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.709 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:43.710 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.693+0000 7f4252df3640 10 monclient: shutdown 2024-09-11T01:54:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.693+0000 7f4252df3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.693+0000 7f4252df3640 1 -- 172.21.15.12:0/3170379485 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c0a2200 msgr2=0x7f424c13a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.693+0000 7f4252df3640 1 --2- 172.21.15.12:0/3170379485 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c0a2200 0x7f424c13a000 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f42380097a0 tx=0x7f423802a010 comp rx=0 tx=0).stop 2024-09-11T01:54:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.693+0000 7f4252df3640 1 -- 172.21.15.12:0/3170379485 shutdown_connections 2024-09-11T01:54:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.693+0000 7f4252df3640 1 --2- 172.21.15.12:0/3170379485 >> v2:172.21.15.12:3300/0 conn(0x7f424c1355c0 0x7f424c13c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.693+0000 7f4252df3640 1 --2- 172.21.15.12:0/3170379485 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c0a2200 0x7f424c13a000 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.693+0000 7f4252df3640 1 -- 172.21.15.12:0/3170379485 >> 172.21.15.12:0/3170379485 conn(0x7f424c0a2f70 msgr2=0x7f424c0a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:43.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.693+0000 7f4252df3640 1 -- 172.21.15.12:0/3170379485 shutdown_connections 2024-09-11T01:54:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.693+0000 7f4252df3640 1 -- 172.21.15.12:0/3170379485 wait complete. 2024-09-11T01:54:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.693+0000 7f4252df3640 10 monclient: build_initial_monmap 2024-09-11T01:54:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.693+0000 7f4252df3640 10 monclient: monmap: 2024-09-11T01:54:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:54:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:54:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:54:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:54:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:54:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:54:43.711 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.694+0000 7f4252df3640 1 Processor -- start 2024-09-11T01:54:43.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.694+0000 7f4252df3640 1 -- start start 2024-09-11T01:54:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.694+0000 7f4252df3640 10 monclient: init 2024-09-11T01:54:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.694+0000 7f4252df3640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.694+0000 7f4252df3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:54:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.694+0000 7f4252df3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c0a2200 0x7f424c06fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.694+0000 7f4252df3640 10 monclient(hunting): picked mon.noname-b con 0x7f424c0a2200 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:54:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.694+0000 7f4252df3640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f424c134ad0 0x7f424c0700e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.694+0000 7f4252df3640 10 monclient(hunting): picked mon.noname-c con 0x7f424c134ad0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.694+0000 7f4252df3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f424c1355c0 0x7f424c1cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.694+0000 7f4252df3640 10 monclient(hunting): picked mon.noname-a con 0x7f424c1355c0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:54:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.694+0000 7f4252df3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.694+0000 7f4252df3640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f424c0a0150 con 0x7f424c1355c0 2024-09-11T01:54:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.694+0000 7f4252df3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.694+0000 7f4252df3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f424c09ffd0 con 0x7f424c0a2200 2024-09-11T01:54:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.694+0000 7f4252df3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.694+0000 7f4252df3640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f424c13ca90 con 0x7f424c134ad0 2024-09-11T01:54:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.694+0000 7f4252df3640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.694+0000 7f4252df3640 10 monclient(hunting): authenticate will time out at 2290.137972s 2024-09-11T01:54:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f4251390640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f424c1355c0 0x7f424c1cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f424bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f424c134ad0 0x7f424c0700e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f424bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f424c134ad0 0x7f424c0700e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41186/0 (socket says 172.21.15.12:41186) 2024-09-11T01:54:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f4251390640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f424c1355c0 0x7f424c1cae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43664/0 (socket says 172.21.15.12:43664) 2024-09-11T01:54:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f424bfff640 1 -- 172.21.15.12:0/2002077471 learned_addr learned my addr 172.21.15.12:0/2002077471 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f424bfff640 10 monclient(hunting): get_auth_request con 0x7f424c134ad0 auth_method 0 2024-09-11T01:54:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f424bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f424bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f424bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f4251390640 10 monclient(hunting): get_auth_request con 0x7f424c1355c0 auth_method 0 2024-09-11T01:54:43.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f4251390640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f4251390640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f4251390640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f424bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f424bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f424bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f4251390640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f4251390640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f4251390640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f4250b8f640 1 --2- 172.21.15.12:0/2002077471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c0a2200 0x7f424c06fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f4250b8f640 10 monclient(hunting): get_auth_request con 0x7f424c0a2200 auth_method 0 2024-09-11T01:54:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f4250b8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f4250b8f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f4250b8f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f424bfff640 10 monclient(hunting): handle_auth_done global_id 284147 payload 824 2024-09-11T01:54:43.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f424bfff640 1 -- 172.21.15.12:0/2002077471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c0a2200 msgr2=0x7f424c06fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f424bfff640 1 --2- 172.21.15.12:0/2002077471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c0a2200 0x7f424c06fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f424bfff640 1 -- 172.21.15.12:0/2002077471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f424c1355c0 msgr2=0x7f424c1cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f424bfff640 1 --2- 172.21.15.12:0/2002077471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f424c1355c0 0x7f424c1cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f424bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:54:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f424bfff640 1 monclient: found mon.noname-c 2024-09-11T01:54:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f424bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f424bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:54:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f424bfff640 1 -- 172.21.15.12:0/2002077471 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f424c134790 con 0x7f424c134ad0 2024-09-11T01:54:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.695+0000 7f4251390640 1 --2- 172.21.15.12:0/2002077471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f424c1355c0 0x7f424c1cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:54:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.697+0000 7f424bfff640 1 -- 172.21.15.12:0/2002077471 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f424c134ad0 msgr2=0x7f424c0700e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.697+0000 7f424bfff640 1 --2- 172.21.15.12:0/2002077471 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f424c134ad0 0x7f424c0700e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4238000c00 tx=0x7f42380046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.697+0000 7f42497fa640 1 -- 172.21.15.12:0/2002077471 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4238037040 con 0x7f424c134ad0 2024-09-11T01:54:43.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.697+0000 7f42497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:43.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.697+0000 7f42497fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:54:43.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.697+0000 7f42497fa640 10 monclient: dump: 2024-09-11T01:54:43.716 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:43.716 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.716 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.716 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.716 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:43.716 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.716 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:43.716 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:43.716 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:43.716 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.697+0000 7f42497fa640 10 monclient: _finish_auth 0 2024-09-11T01:54:43.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.697+0000 7f42497fa640 10 monclient: _check_auth_tickets 2024-09-11T01:54:43.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.697+0000 7f42497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:43.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.697+0000 7f42497fa640 1 -- 172.21.15.12:0/2002077471 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4238042070 con 0x7f424c134ad0 2024-09-11T01:54:43.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.697+0000 7f42497fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:43.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.697+0000 7f42497fa640 1 -- 172.21.15.12:0/2002077471 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423803bb00 con 0x7f424c134ad0 2024-09-11T01:54:43.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.697+0000 7f4252df3640 5 monclient: authenticate success, global_id 284147 2024-09-11T01:54:43.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.697+0000 7f4252df3640 10 monclient: _renew_subs 2024-09-11T01:54:43.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.697+0000 7f4252df3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:43.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.697+0000 7f4252df3640 1 -- 172.21.15.12:0/2002077471 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f424c0726f0 con 0x7f424c134ad0 2024-09-11T01:54:43.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.697+0000 7f42497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:43.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.697+0000 7f42497fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:43.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.697+0000 7f42497fa640 10 monclient: dump: 2024-09-11T01:54:43.717 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:43.717 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:43.718 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.697+0000 7f424affd640 4 set_mon_vals no callback set 2024-09-11T01:54:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.697+0000 7f4252df3640 10 monclient: _renew_subs 2024-09-11T01:54:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.697+0000 7f4252df3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.697+0000 7f4252df3640 1 -- 172.21.15.12:0/2002077471 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f424c072410 con 0x7f424c134ad0 2024-09-11T01:54:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.698+0000 7f4252df3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.698+0000 7f4252df3640 1 --2- 172.21.15.12:0/2002077471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c1cb350 0x7f424c1cb730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.698+0000 7f4252df3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.698+0000 7f4252df3640 1 -- 172.21.15.12:0/2002077471 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f424c1cbc70 con 0x7f424c1cb350 2024-09-11T01:54:43.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.698+0000 7f42497fa640 1 -- 172.21.15.12:0/2002077471 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f423803a030 con 0x7f424c134ad0 2024-09-11T01:54:43.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.699+0000 7f4250b8f640 1 --2- 172.21.15.12:0/2002077471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c1cb350 0x7f424c1cb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.699+0000 7f4250b8f640 10 monclient: get_auth_request con 0x7f424c1cb350 auth_method 0 2024-09-11T01:54:43.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.699+0000 7f4250b8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.699+0000 7f4250b8f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.699+0000 7f4250b8f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.699+0000 7f42497fa640 1 --2- 172.21.15.12:0/2002077471 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f423003e210 0x7f42300406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.699+0000 7f42497fa640 1 -- 172.21.15.12:0/2002077471 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(858..858 src has 676..858) ==== 4847+0+0 (secure 0 0 0) 0x7f4238073240 con 0x7f424c134ad0 2024-09-11T01:54:43.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.699+0000 7f4250b8f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.699+0000 7f4250b8f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.699+0000 7f4250b8f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.699+0000 7f4251390640 1 --2- 172.21.15.12:0/2002077471 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f423003e210 0x7f42300406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.699+0000 7f4251390640 10 monclient: get_auth_request con 0x7f423003e210 auth_method 0 2024-09-11T01:54:43.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.699+0000 7f4250b8f640 10 monclient(hunting): handle_auth_done global_id 274210 payload 824 2024-09-11T01:54:43.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.699+0000 7f4250b8f640 1 -- 172.21.15.12:0/2002077471 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f424c134eb0 con 0x7f424c1cb350 2024-09-11T01:54:43.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.701+0000 7f4250b8f640 1 -- 172.21.15.12:0/2002077471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c1cb350 msgr2=0x7f424c1cb730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:43.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.701+0000 7f4250b8f640 1 --2- 172.21.15.12:0/2002077471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c1cb350 0x7f424c1cb730 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f424000e4d0 tx=0x7f424000e990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:43.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.701+0000 7f4251390640 1 --2- 172.21.15.12:0/2002077471 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f423003e210 0x7f42300406d0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f424c070c80 tx=0x7f424400b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:43.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.701+0000 7f42497fa640 1 -- 172.21.15.12:0/2002077471 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4240007450 con 0x7f424c1cb350 2024-09-11T01:54:43.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.701+0000 7f42497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:43.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.701+0000 7f42497fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:54:43.720 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.701+0000 7f42497fa640 10 monclient: dump: 2024-09-11T01:54:43.720 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:43.720 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.720 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.720 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.720 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:43.721 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.721 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:43.721 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:43.721 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:43.721 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.702+0000 7f42497fa640 1 -- 172.21.15.12:0/2002077471 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4240007450 con 0x7f424c1cb350 2024-09-11T01:54:43.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.702+0000 7f42497fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:43.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.702+0000 7f42497fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:54:43.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.702+0000 7f42497fa640 1 -- 172.21.15.12:0/2002077471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c1cb350 msgr2=0x7f424c1cb730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.702+0000 7f42497fa640 1 --2- 172.21.15.12:0/2002077471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c1cb350 0x7f424c1cb730 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f424000e4d0 tx=0x7f424000e990 comp rx=0 tx=0).stop 2024-09-11T01:54:43.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.702+0000 7f42497fa640 1 -- 172.21.15.12:0/2002077471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c1cb350 msgr2=0x7f424c1cb730 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:54:43.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.702+0000 7f42497fa640 1 --2- 172.21.15.12:0/2002077471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c1cb350 0x7f424c1cb730 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f424000e4d0 tx=0x7f424000e990 comp rx=0 tx=0).stop 2024-09-11T01:54:43.722 INFO:tasks.rados.rados.0.smithi071.stdout:3760: done (0 left) 2024-09-11T01:54:43.722 INFO:tasks.rados.rados.0.smithi071.stdout:3761: write (excl) oid 49 current snap is 424 2024-09-11T01:54:43.723 INFO:tasks.rados.rados.0.smithi071.stdout:3761: seq_num 1287 ranges {430609=509120,1478796=736116,2909627=602257} 2024-09-11T01:54:43.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.708+0000 7f4252df3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:54:43.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.708+0000 7f4252df3640 1 --2- 172.21.15.12:0/2002077471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c130000 0x7f424c1303e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.708+0000 7f4252df3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.708+0000 7f4252df3640 1 -- 172.21.15.12:0/2002077471 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f424c070ef0 con 0x7f424c130000 2024-09-11T01:54:43.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.708+0000 7f4250b8f640 1 --2- 172.21.15.12:0/2002077471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c130000 0x7f424c1303e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.708+0000 7f4250b8f640 10 monclient: get_auth_request con 0x7f424c130000 auth_method 0 2024-09-11T01:54:43.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.708+0000 7f4250b8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.708+0000 7f4250b8f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.708+0000 7f4250b8f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.708+0000 7f4250b8f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.708+0000 7f4250b8f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.708+0000 7f4250b8f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.708+0000 7f4250b8f640 10 monclient(hunting): handle_auth_done global_id 274213 payload 824 2024-09-11T01:54:43.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.708+0000 7f4250b8f640 1 -- 172.21.15.12:0/2002077471 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f424c06d480 con 0x7f424c130000 2024-09-11T01:54:43.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.711+0000 7f4250b8f640 1 -- 172.21.15.12:0/2002077471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c130000 msgr2=0x7f424c1303e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:43.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.711+0000 7f4250b8f640 1 --2- 172.21.15.12:0/2002077471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c130000 0x7f424c1303e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f424000e4d0 tx=0x7f4240003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:43.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.711+0000 7f42497fa640 1 -- 172.21.15.12:0/2002077471 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f424001e960 con 0x7f424c130000 2024-09-11T01:54:43.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.711+0000 7f42497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:43.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.711+0000 7f42497fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:54:43.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.711+0000 7f42497fa640 10 monclient: dump: 2024-09-11T01:54:43.725 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:43.725 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.725 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:43.726 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.711+0000 7f42497fa640 1 -- 172.21.15.12:0/2002077471 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7f4240013420 con 0x7f424c130000 2024-09-11T01:54:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.711+0000 7f42497fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:54:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.711+0000 7f42497fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:54:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.711+0000 7f42497fa640 1 -- 172.21.15.12:0/2002077471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c130000 msgr2=0x7f424c1303e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.711+0000 7f42497fa640 1 --2- 172.21.15.12:0/2002077471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c130000 0x7f424c1303e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f424000e4d0 tx=0x7f4240003760 comp rx=0 tx=0).stop 2024-09-11T01:54:43.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.711+0000 7f42497fa640 1 -- 172.21.15.12:0/2002077471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c130000 msgr2=0x7f424c1303e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:54:43.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.711+0000 7f42497fa640 1 --2- 172.21.15.12:0/2002077471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c130000 0x7f424c1303e0 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f424000e4d0 tx=0x7f4240003760 comp rx=0 tx=0).stop 2024-09-11T01:54:43.727 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:54:43.727 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:54:43.727 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:54:43.727 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:54:43.727 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 232, 2024-09-11T01:54:43.727 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 73531, 2024-09-11T01:54:43.727 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:54:43.727 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:54:43.727 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:54:43.728 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:54:43.728 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:43.728 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:54:43.728 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:54:43.728 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:54:43.728 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:54:43.728 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:43.728 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:43.728 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:43.728 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:43.728 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:43.728 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:43.728 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:43.728 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:43.729 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:43.729 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:43.729 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:43.729 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:43.729 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:54:43.729 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:54:43.729 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:43.729 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:43.729 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:43.729 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:43.729 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:43.729 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:43.729 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:43.730 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:43.730 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:43.730 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:43.730 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:43.730 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:43.730 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.730 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:54:43.730 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:54:43.730 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:54:43.730 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:54:43.730 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:54:43.730 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:54:43.730 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:54:43.730 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:54:43.731 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:54:43.731 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:54:43.731 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:54:43.731 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:54:43.731 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:54:43.731 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:54:43.731 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:54:43.731 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:54:43.731 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:54:43.731 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:43.731 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:43.731 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:43.731 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:43.731 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:43.732 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:43.732 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:43.732 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:43.732 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:43.732 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:43.732 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:43.732 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:43.732 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:54:43.732 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.732 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:54:43.732 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.732 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:54:43.732 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:54:43.733 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:43.733 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:43.733 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.733 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:43.733 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:54:43.733 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:43.733 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.733 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.733 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:43.733 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:54:43.733 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:43.733 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:43.733 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:43.733 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.734 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:54:43.734 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:54:43.734 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:43.734 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:43.734 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:43.734 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.734 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.734 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:54:43.734 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:54:43.734 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:43.734 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:43.734 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.734 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:43.734 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:54:43.735 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:43.735 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.735 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.735 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:43.735 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:54:43.735 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:43.735 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:43.735 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:43.735 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.735 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:54:43.735 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:54:43.735 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:43.735 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:43.735 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:43.735 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.736 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.736 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:54:43.736 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:54:43.736 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:43.736 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:43.736 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.736 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:43.736 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:54:43.736 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:43.736 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.736 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.736 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:43.736 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:54:43.736 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:43.737 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:43.737 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:43.737 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.737 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:54:43.737 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:54:43.737 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:43.737 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:43.737 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:43.737 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:43.737 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:43.737 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.737 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:54:43.737 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:54:43.738 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.738 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:43.738 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:43.738 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:54:43.738 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:43.738 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:43.738 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:54:43.738 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.738 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:43.738 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:43.738 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:54:43.738 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:43.738 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:43.738 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:54:43.738 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.739 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:43.739 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:43.739 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:54:43.739 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:43.739 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:43.739 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:54:43.739 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:43.739 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:43.739 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:43.739 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:54:43.739 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:43.740 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:43.740 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:43.740 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:54:43.740 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:54:43.741 INFO:tasks.rados.rados.0.smithi071.stdout:3761: writing smithi07134777-49 from 430609 to 939729 tid 1 2024-09-11T01:54:43.741 INFO:tasks.rados.rados.0.smithi071.stdout:3761: writing smithi07134777-49 from 1478796 to 2214912 tid 2 2024-09-11T01:54:43.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.711+0000 7f4252df3640 1 -- 172.21.15.12:0/2002077471 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f423003e210 msgr2=0x7f42300406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.711+0000 7f4252df3640 1 --2- 172.21.15.12:0/2002077471 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f423003e210 0x7f42300406d0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f424c070c80 tx=0x7f424400b040 comp rx=0 tx=0).stop 2024-09-11T01:54:43.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.711+0000 7f4252df3640 10 monclient: shutdown 2024-09-11T01:54:43.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.711+0000 7f4252df3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:43.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.711+0000 7f4252df3640 1 -- 172.21.15.12:0/2002077471 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f424c134ad0 msgr2=0x7f424c0700e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.711+0000 7f4252df3640 1 --2- 172.21.15.12:0/2002077471 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f424c134ad0 0x7f424c0700e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4238000c00 tx=0x7f42380046c0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.711+0000 7f4251390640 1 -- 172.21.15.12:0/2002077471 reap_dead start 2024-09-11T01:54:43.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.711+0000 7f4251390640 1 -- 172.21.15.12:0/2002077471 reap_dead start 2024-09-11T01:54:43.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.712+0000 7f4252df3640 1 -- 172.21.15.12:0/2002077471 shutdown_connections 2024-09-11T01:54:43.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.712+0000 7f4252df3640 1 -- 172.21.15.12:0/2002077471 >> 172.21.15.12:0/2002077471 conn(0x7f424c0a2f70 msgr2=0x7f424c09deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:43.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.712+0000 7f4252df3640 1 -- 172.21.15.12:0/2002077471 shutdown_connections 2024-09-11T01:54:43.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.712+0000 7f4252df3640 1 -- 172.21.15.12:0/2002077471 wait complete. 2024-09-11T01:54:43.743 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:54:43.743 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-11T01:54:43.743 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T01:54:43.743 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-11T01:54:43.743 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-11T01:54:43.743 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T01:54:43.744 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:54:43.744 INFO:tasks.rados.rados.0.smithi071.stdout:3761: writing smithi07134777-49 from 2909627 to 3511884 tid 3 2024-09-11T01:54:43.745 INFO:tasks.rados.rados.0.smithi071.stdout:3762: read oid 12 snap -1 2024-09-11T01:54:43.745 INFO:tasks.rados.rados.0.smithi071.stdout:3762: expect (ObjNum 1208 snap 393 seq_num 1208) 2024-09-11T01:54:43.745 INFO:tasks.rados.rados.0.smithi071.stdout:3763: read oid 25 snap 340 2024-09-11T01:54:43.745 INFO:tasks.rados.rados.0.smithi071.stdout:3763: expect (ObjNum 1023 snap 328 seq_num 1023) 2024-09-11T01:54:43.745 INFO:tasks.rados.rados.0.smithi071.stdout:3764: delete oid 29 current snap is 424 2024-09-11T01:54:43.745 INFO:tasks.rados.rados.0.smithi071.stdout:3761: finishing write tid 1 to smithi07134777-49 2024-09-11T01:54:43.747 INFO:tasks.rados.rados.0.smithi071.stdout:3761: finishing write tid 2 to smithi07134777-49 2024-09-11T01:54:43.747 INFO:tasks.rados.rados.0.smithi071.stdout:3761: finishing write tid 3 to smithi07134777-49 2024-09-11T01:54:43.747 INFO:tasks.rados.rados.0.smithi071.stdout:3761: finishing write tid 4 to smithi07134777-49 2024-09-11T01:54:43.747 INFO:tasks.rados.rados.0.smithi071.stdout:3761: finishing write tid 5 to smithi07134777-49 2024-09-11T01:54:43.748 INFO:tasks.rados.rados.0.smithi071.stdout:3761: oid 49 updating version 0 to 2454 2024-09-11T01:54:43.748 INFO:tasks.rados.rados.0.smithi071.stdout:3761: oid 49 updating version 2454 to 2457 2024-09-11T01:54:43.748 INFO:tasks.rados.rados.0.smithi071.stdout:3761: oid 49 version 2457 is already newer than 2455 2024-09-11T01:54:43.748 INFO:tasks.rados.rados.0.smithi071.stdout:3761: oid 49 version 2457 is already newer than 2456 2024-09-11T01:54:43.748 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 2457 (ObjNum 1287 snap 424 seq_num 1287) dirty exists 2024-09-11T01:54:43.748 INFO:tasks.rados.rados.0.smithi071.stdout:3761: left oid 49 (ObjNum 1287 snap 424 seq_num 1287) 2024-09-11T01:54:43.754 INFO:tasks.rados.rados.0.smithi071.stdout:3763: expect (ObjNum 1023 snap 328 seq_num 1023) 2024-09-11T01:54:43.761 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T01:54:43.761 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:54:43.761 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:54:43.803 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 474401 2024-09-11T01:54:43.875 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 3187055 2024-09-11T01:54:43.876 INFO:tasks.rados.rados.0.smithi071.stdout:3762: expect (ObjNum 1208 snap 393 seq_num 1208) 2024-09-11T01:54:43.876 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 112770 2024-09-11T01:54:43.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.898+0000 7f65d1d3d640 10 monclient: get_monmap_and_config 2024-09-11T01:54:43.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.898+0000 7f65d1d3d640 10 monclient: build_initial_monmap 2024-09-11T01:54:43.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.898+0000 7f65d1d3d640 10 monclient: monmap: 2024-09-11T01:54:43.902 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:43.902 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.902 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:54:43.900682+0000 2024-09-11T01:54:43.902 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:54:43.900682+0000 2024-09-11T01:54:43.902 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:43.902 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.902 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:54:43.902 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:54:43.902 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:54:43.902 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:54:43.902 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:54:43.902 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.899+0000 7f65d1d3d640 1 Processor -- start 2024-09-11T01:54:43.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.899+0000 7f65d1d3d640 1 -- start start 2024-09-11T01:54:43.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.899+0000 7f65d1d3d640 10 monclient: init 2024-09-11T01:54:43.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65d1d3d640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:43.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65d1d3d640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-11T01:54:43.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65d1d3d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f65cc0a2db0 addr v1:172.21.15.12:6789/0 2024-09-11T01:54:43.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65d1d3d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f65cc09fbd0 0x7f65cc06d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65d1d3d640 10 monclient(hunting): picked mon.noname-b con 0x7f65cc09fbd0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:43.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65d1d3d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f65cc13d550 0x7f65cc13f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65d1d3d640 10 monclient(hunting): picked mon.noname-c con 0x7f65cc13d550 addr v2:172.21.15.71:3300/0 2024-09-11T01:54:43.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65d1d3d640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65cc09eea0 con 0x7f65cc0a2db0 2024-09-11T01:54:43.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65d1d3d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65d1d3d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f65cc0a3850 con 0x7f65cc13d550 2024-09-11T01:54:43.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65d1d3d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65d1d3d640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f65cc0a3190 con 0x7f65cc09fbd0 2024-09-11T01:54:43.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65d1d3d640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:43.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65d1d3d640 10 monclient(hunting): authenticate will time out at 2290.343229s 2024-09-11T01:54:43.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65cb7fe640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7f65cc0a2db0 legacy=0x7f65cc06cb50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:54:43.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65caffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f65cc09fbd0 0x7f65cc06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65caffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f65cc09fbd0 0x7f65cc06d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41236/0 (socket says 172.21.15.12:41236) 2024-09-11T01:54:43.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65caffd640 1 -- 172.21.15.12:0/2148818830 learned_addr learned my addr 172.21.15.12:0/2148818830 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:43.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65caffd640 10 monclient(hunting): get_auth_request con 0x7f65cc09fbd0 auth_method 0 2024-09-11T01:54:43.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65caffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65cbfff640 1 --2- 172.21.15.12:0/2148818830 >> v2:172.21.15.71:3300/0 conn(0x7f65cc13d550 0x7f65cc13f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65cbfff640 10 monclient(hunting): get_auth_request con 0x7f65cc13d550 auth_method 0 2024-09-11T01:54:43.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65caffd640 10 monclient(hunting): handle_auth_done global_id 284150 payload 274 2024-09-11T01:54:43.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65caffd640 1 -- 172.21.15.12:0/2148818830 >> v2:172.21.15.71:3300/0 conn(0x7f65cc13d550 msgr2=0x7f65cc13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:54:43.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65caffd640 1 --2- 172.21.15.12:0/2148818830 >> v2:172.21.15.71:3300/0 conn(0x7f65cc13d550 0x7f65cc13f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65caffd640 1 -- 172.21.15.12:0/2148818830 >> v1:172.21.15.12:6789/0 conn(0x7f65cc0a2db0 legacy=0x7f65cc06cb50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:54:43.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65caffd640 10 monclient: _finish_hunting 0 2024-09-11T01:54:43.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65caffd640 1 monclient: found mon.noname-b 2024-09-11T01:54:43.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:43.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65caffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:54:43.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.900+0000 7f65caffd640 1 -- 172.21.15.12:0/2148818830 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65cc09f180 con 0x7f65cc09fbd0 2024-09-11T01:54:43.914 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:54:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.903+0000 7f65caffd640 1 -- 172.21.15.12:0/2148818830 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f65cc09fbd0 msgr2=0x7f65cc06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.903+0000 7f65caffd640 1 --2- 172.21.15.12:0/2148818830 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f65cc09fbd0 0x7f65cc06d260 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f65c0009870 tx=0x7f65c002a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46f412d8e833cd1b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.903+0000 7f65ca7fc640 1 -- 172.21.15.12:0/2148818830 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65c0037070 con 0x7f65cc09fbd0 2024-09-11T01:54:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.903+0000 7f65ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.903+0000 7f65ca7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:54:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.903+0000 7f65ca7fc640 10 monclient: dump: 2024-09-11T01:54:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.915 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:43.916 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.903+0000 7f65ca7fc640 10 monclient: _finish_auth 0 2024-09-11T01:54:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.903+0000 7f65ca7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:54:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.903+0000 7f65ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.903+0000 7f65ca7fc640 1 -- 172.21.15.12:0/2148818830 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65c0032440 con 0x7f65cc09fbd0 2024-09-11T01:54:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.903+0000 7f65d1d3d640 5 monclient: authenticate success, global_id 284150 2024-09-11T01:54:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.903+0000 7f65d1d3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:54:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.903+0000 7f65ca7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.903+0000 7f65ca7fc640 1 -- 172.21.15.12:0/2148818830 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65c0032810 con 0x7f65cc09fbd0 2024-09-11T01:54:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.903+0000 7f65d1d3d640 10 monclient: get_monmap_and_config success 2024-09-11T01:54:43.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.903+0000 7f65d1d3d640 4 set_mon_vals no callback set 2024-09-11T01:54:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.903+0000 7f65ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.903+0000 7f65ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.903+0000 7f65ca7fc640 10 monclient: dump: 2024-09-11T01:54:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:43.917 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:43.918 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.904+0000 7f65d1d3d640 10 monclient: shutdown 2024-09-11T01:54:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.904+0000 7f65d1d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.904+0000 7f65d1d3d640 1 -- 172.21.15.12:0/2148818830 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f65cc09fbd0 msgr2=0x7f65cc06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.904+0000 7f65d1d3d640 1 --2- 172.21.15.12:0/2148818830 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f65cc09fbd0 0x7f65cc06d260 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f65c0009870 tx=0x7f65c002a0a0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.904+0000 7f65d1d3d640 1 -- 172.21.15.12:0/2148818830 shutdown_connections 2024-09-11T01:54:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.904+0000 7f65d1d3d640 1 --2- 172.21.15.12:0/2148818830 >> v2:172.21.15.71:3300/0 conn(0x7f65cc13d550 0x7f65cc13f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.904+0000 7f65d1d3d640 1 --2- 172.21.15.12:0/2148818830 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f65cc09fbd0 0x7f65cc06d260 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.904+0000 7f65d1d3d640 1 -- 172.21.15.12:0/2148818830 >> 172.21.15.12:0/2148818830 conn(0x7f65cc0a3ea0 msgr2=0x7f65cc0a42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.904+0000 7f65d1d3d640 1 -- 172.21.15.12:0/2148818830 shutdown_connections 2024-09-11T01:54:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.904+0000 7f65d1d3d640 1 -- 172.21.15.12:0/2148818830 wait complete. 2024-09-11T01:54:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.904+0000 7f65d1d3d640 10 monclient: build_initial_monmap 2024-09-11T01:54:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.904+0000 7f65d1d3d640 10 monclient: monmap: 2024-09-11T01:54:43.918 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:54:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:54:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:54:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:54:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:54:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:54:43.919 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65d1d3d640 1 Processor -- start 2024-09-11T01:54:43.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65d1d3d640 1 -- start start 2024-09-11T01:54:43.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65d1d3d640 10 monclient: init 2024-09-11T01:54:43.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65d1d3d640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:43.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65d1d3d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:54:43.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65d1d3d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65cc09fbd0 0x7f65cc072230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65d1d3d640 10 monclient(hunting): picked mon.noname-a con 0x7f65cc09fbd0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:54:43.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65d1d3d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65cc0a2db0 0x7f65cc072770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65d1d3d640 10 monclient(hunting): picked mon.noname-b con 0x7f65cc0a2db0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:54:43.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65d1d3d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f65cc13d550 0x7f65cc072cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65d1d3d640 10 monclient(hunting): picked mon.noname-c con 0x7f65cc13d550 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:43.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65cb7fe640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65cc09fbd0 msgr2=0x7f65cc072230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:54:43.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65d1d3d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65d1d3d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f65cc09ea20 con 0x7f65cc09fbd0 2024-09-11T01:54:43.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65d1d3d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65d1d3d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f65cc09e8a0 con 0x7f65cc0a2db0 2024-09-11T01:54:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65d1d3d640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65d1d3d640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f65cc06d7a0 con 0x7f65cc13d550 2024-09-11T01:54:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65d1d3d640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65d1d3d640 10 monclient(hunting): authenticate will time out at 2290.348870s 2024-09-11T01:54:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65cb7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65cc09fbd0 0x7f65cc072230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:54:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65cbfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f65cc13d550 0x7f65cc072cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65cbfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f65cc13d550 0x7f65cc072cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41252/0 (socket says 172.21.15.12:41252) 2024-09-11T01:54:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65cbfff640 1 -- 172.21.15.12:0/1819503626 learned_addr learned my addr 172.21.15.12:0/1819503626 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65cbfff640 10 monclient(hunting): get_auth_request con 0x7f65cc13d550 auth_method 0 2024-09-11T01:54:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.905+0000 7f65cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.906+0000 7f65caffd640 1 --2- 172.21.15.12:0/1819503626 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65cc0a2db0 0x7f65cc072770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.906+0000 7f65caffd640 10 monclient(hunting): get_auth_request con 0x7f65cc0a2db0 auth_method 0 2024-09-11T01:54:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.906+0000 7f65caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.906+0000 7f65caffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.906+0000 7f65caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.906+0000 7f65cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.906+0000 7f65cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.906+0000 7f65cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.906+0000 7f65caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.906+0000 7f65caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.906+0000 7f65caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.906+0000 7f65cbfff640 10 monclient(hunting): handle_auth_done global_id 284153 payload 824 2024-09-11T01:54:43.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.906+0000 7f65cbfff640 1 -- 172.21.15.12:0/1819503626 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65cc0a2db0 msgr2=0x7f65cc072770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:43.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.906+0000 7f65cbfff640 1 --2- 172.21.15.12:0/1819503626 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65cc0a2db0 0x7f65cc072770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.906+0000 7f65cbfff640 1 -- 172.21.15.12:0/1819503626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65cc09fbd0 msgr2=0x7f65cc072230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:54:43.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.906+0000 7f65cbfff640 1 --2- 172.21.15.12:0/1819503626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65cc09fbd0 0x7f65cc072230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:43.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.906+0000 7f65cbfff640 10 monclient: _finish_hunting 0 2024-09-11T01:54:43.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.906+0000 7f65cbfff640 1 monclient: found mon.noname-c 2024-09-11T01:54:43.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.906+0000 7f65cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:43.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.906+0000 7f65cbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:54:43.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.906+0000 7f65cbfff640 1 -- 172.21.15.12:0/1819503626 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65cc091880 con 0x7f65cc13d550 2024-09-11T01:54:43.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.908+0000 7f65cbfff640 1 -- 172.21.15.12:0/1819503626 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f65cc13d550 msgr2=0x7f65cc072cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:43.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.908+0000 7f65cbfff640 1 --2- 172.21.15.12:0/1819503626 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f65cc13d550 0x7f65cc072cb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f65c400efc0 tx=0x7f65c400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:43.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.908+0000 7f65b3fff640 1 -- 172.21.15.12:0/1819503626 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65c4018020 con 0x7f65cc13d550 2024-09-11T01:54:43.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.908+0000 7f65b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:43.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.908+0000 7f65b3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:54:43.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.908+0000 7f65b3fff640 10 monclient: dump: 2024-09-11T01:54:43.924 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:43.924 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.924 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.924 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.924 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:43.924 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.924 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:43.924 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:43.924 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:43.924 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.908+0000 7f65b3fff640 10 monclient: _finish_auth 0 2024-09-11T01:54:43.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.908+0000 7f65b3fff640 10 monclient: _check_auth_tickets 2024-09-11T01:54:43.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.908+0000 7f65b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:43.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.908+0000 7f65b3fff640 1 -- 172.21.15.12:0/1819503626 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65c4014680 con 0x7f65cc13d550 2024-09-11T01:54:43.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.908+0000 7f65b3fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:43.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.908+0000 7f65b3fff640 1 -- 172.21.15.12:0/1819503626 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65c4014a70 con 0x7f65cc13d550 2024-09-11T01:54:43.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.908+0000 7f65d1d3d640 5 monclient: authenticate success, global_id 284153 2024-09-11T01:54:43.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.908+0000 7f65d1d3d640 10 monclient: _renew_subs 2024-09-11T01:54:43.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.908+0000 7f65d1d3d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:43.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.908+0000 7f65d1d3d640 1 -- 172.21.15.12:0/1819503626 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65cc091e40 con 0x7f65cc13d550 2024-09-11T01:54:43.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.908+0000 7f65b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:43.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.908+0000 7f65b3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:43.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.908+0000 7f65b3fff640 10 monclient: dump: 2024-09-11T01:54:43.925 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:43.925 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:43.925 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.925 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:43.925 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:43.926 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:43.926 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:43.926 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:43.926 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:43.926 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:43.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.908+0000 7f65c9ffb640 4 set_mon_vals no callback set 2024-09-11T01:54:43.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.908+0000 7f65d1d3d640 10 monclient: _renew_subs 2024-09-11T01:54:43.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.908+0000 7f65d1d3d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:43.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.908+0000 7f65d1d3d640 1 -- 172.21.15.12:0/1819503626 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f65cc0913c0 con 0x7f65cc13d550 2024-09-11T01:54:43.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.909+0000 7f65b3fff640 1 -- 172.21.15.12:0/1819503626 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f65c4014c10 con 0x7f65cc13d550 2024-09-11T01:54:43.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.909+0000 7f65d1d3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:43.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.909+0000 7f65d1d3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:43.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.909+0000 7f65d1d3d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:43.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.909+0000 7f65d1d3d640 1 -- 172.21.15.12:0/1819503626 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65cc06cb50 con 0x7f65cc13d550 2024-09-11T01:54:43.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.909+0000 7f65b3fff640 1 --2- 172.21.15.12:0/1819503626 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f65b803e1c0 0x7f65b8040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:43.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.910+0000 7f65b3fff640 1 -- 172.21.15.12:0/1819503626 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(859..859 src has 676..859) ==== 4875+0+0 (secure 0 0 0) 0x7f65c4053cb0 con 0x7f65cc13d550 2024-09-11T01:54:43.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.910+0000 7f65cb7fe640 1 --2- 172.21.15.12:0/1819503626 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f65b803e1c0 0x7f65b8040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:43.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.910+0000 7f65cb7fe640 10 monclient: get_auth_request con 0x7f65b803e1c0 auth_method 0 2024-09-11T01:54:43.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.912+0000 7f65cb7fe640 1 --2- 172.21.15.12:0/1819503626 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f65b803e1c0 0x7f65b8040680 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f65bc003980 tx=0x7f65bc006d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:43.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.914+0000 7f65b3fff640 1 -- 172.21.15.12:0/1819503626 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f65c4022070 con 0x7f65cc13d550 2024-09-11T01:54:43.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.914+0000 7f65b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:43.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:43.914+0000 7f65b3fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:54:43.951 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:54:43.989 INFO:tasks.rados.rados.0.smithi071.stdout:3761: done (3 left) 2024-09-11T01:54:43.989 INFO:tasks.rados.rados.0.smithi071.stdout:3762: done (2 left) 2024-09-11T01:54:43.990 INFO:tasks.rados.rados.0.smithi071.stdout:3763: done (1 left) 2024-09-11T01:54:43.990 INFO:tasks.rados.rados.0.smithi071.stdout:3764: done (0 left) 2024-09-11T01:54:43.990 INFO:tasks.rados.rados.0.smithi071.stdout:3765: write oid 24 current snap is 424 2024-09-11T01:54:43.990 INFO:tasks.rados.rados.0.smithi071.stdout:3765: seq_num 1289 ranges {464223=659618,1879407=739785,3248120=416103} 2024-09-11T01:54:43.996 INFO:tasks.rados.rados.0.smithi071.stdout:3765: writing smithi07134777-24 from 464223 to 1123841 tid 1 2024-09-11T01:54:44.003 INFO:tasks.rados.rados.0.smithi071.stdout:3765: writing smithi07134777-24 from 1879407 to 2619192 tid 2 2024-09-11T01:54:44.009 INFO:tasks.rados.rados.0.smithi071.stdout:3765: writing smithi07134777-24 from 3248120 to 3664223 tid 3 2024-09-11T01:54:44.009 INFO:tasks.rados.rados.0.smithi071.stdout:3766: write (excl) oid 16 current snap is 424 2024-09-11T01:54:44.009 INFO:tasks.rados.rados.0.smithi071.stdout:3765: finishing write tid 1 to smithi07134777-24 2024-09-11T01:54:44.009 INFO:tasks.rados.rados.0.smithi071.stdout:3766: seq_num 1290 ranges {481030=734867,1937889=483443,2875543=552426} 2024-09-11T01:54:44.015 INFO:tasks.rados.rados.0.smithi071.stdout:3766: writing smithi07134777-16 from 481030 to 1215897 tid 1 2024-09-11T01:54:44.022 INFO:tasks.rados.rados.0.smithi071.stdout:3766: writing smithi07134777-16 from 1937889 to 2421332 tid 2 2024-09-11T01:54:44.028 INFO:tasks.rados.rados.0.smithi071.stdout:3766: writing smithi07134777-16 from 2875543 to 3427969 tid 3 2024-09-11T01:54:44.028 INFO:tasks.rados.rados.0.smithi071.stdout:3767: read oid 17 snap -1 2024-09-11T01:54:44.028 INFO:tasks.rados.rados.0.smithi071.stdout:3767: expect (ObjNum 1242 snap 411 seq_num 1242) 2024-09-11T01:54:44.028 INFO:tasks.rados.rados.0.smithi071.stdout:3768: snap_remove snap 423 2024-09-11T01:54:44.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:44.183+0000 7f65d1d3d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:54:44.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:44.183+0000 7f65d1d3d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:54:44.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:44.183+0000 7f65d1d3d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:44.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:44.183+0000 7f65d1d3d640 1 -- 172.21.15.12:0/1819503626 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f65cc0921b0 con 0x7f65cc13d550 2024-09-11T01:54:44.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:44.183+0000 7f65b3fff640 1 -- 172.21.15.12:0/1819503626 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f65c4010040 con 0x7f65cc13d550 2024-09-11T01:54:44.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:44.183+0000 7f65b3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:54:44.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:44.183+0000 7f65b3fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:54:44.186 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":232,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:54:44.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:44.185+0000 7f65d1d3d640 1 -- 172.21.15.12:0/1819503626 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f65b803e1c0 msgr2=0x7f65b8040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:44.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:44.185+0000 7f65d1d3d640 1 --2- 172.21.15.12:0/1819503626 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f65b803e1c0 0x7f65b8040680 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f65bc003980 tx=0x7f65bc006d30 comp rx=0 tx=0).stop 2024-09-11T01:54:44.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:44.185+0000 7f65d1d3d640 10 monclient: shutdown 2024-09-11T01:54:44.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:44.185+0000 7f65d1d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:44.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:44.185+0000 7f65d1d3d640 1 -- 172.21.15.12:0/1819503626 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f65cc13d550 msgr2=0x7f65cc072cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:44.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:44.185+0000 7f65d1d3d640 1 --2- 172.21.15.12:0/1819503626 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f65cc13d550 0x7f65cc072cb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f65c400efc0 tx=0x7f65c400c490 comp rx=0 tx=0).stop 2024-09-11T01:54:44.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:44.185+0000 7f65d1d3d640 1 -- 172.21.15.12:0/1819503626 shutdown_connections 2024-09-11T01:54:44.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:44.185+0000 7f65d1d3d640 1 --2- 172.21.15.12:0/1819503626 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f65cc13d550 0x7f65cc072cb0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:44.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:44.185+0000 7f65d1d3d640 1 --2- 172.21.15.12:0/1819503626 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65cc0a2db0 0x7f65cc072770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:44.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:44.185+0000 7f65d1d3d640 1 --2- 172.21.15.12:0/1819503626 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f65b803e1c0 0x7f65b8040680 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:44.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:44.185+0000 7f65d1d3d640 1 --2- 172.21.15.12:0/1819503626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f65cc09fbd0 0x7f65cc072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:44.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:44.185+0000 7f65d1d3d640 1 -- 172.21.15.12:0/1819503626 >> 172.21.15.12:0/1819503626 conn(0x7f65cc0a3ea0 msgr2=0x7f65cc13c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:44.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:44.185+0000 7f65d1d3d640 1 -- 172.21.15.12:0/1819503626 shutdown_connections 2024-09-11T01:54:44.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:44.185+0000 7f65d1d3d640 1 -- 172.21.15.12:0/1819503626 wait complete. 2024-09-11T01:54:47.200 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:54:47.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.310+0000 7fe03046e640 10 monclient: get_monmap_and_config 2024-09-11T01:54:47.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.310+0000 7fe03046e640 10 monclient: build_initial_monmap 2024-09-11T01:54:47.313 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.310+0000 7fe03046e640 10 monclient: monmap: 2024-09-11T01:54:47.313 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:47.313 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:47.313 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:54:47.312003+0000 2024-09-11T01:54:47.313 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:54:47.312003+0000 2024-09-11T01:54:47.313 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:47.313 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:47.313 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:54:47.313 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:54:47.314 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:54:47.314 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:54:47.314 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:54:47.314 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.310+0000 7fe03046e640 1 Processor -- start 2024-09-11T01:54:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.310+0000 7fe03046e640 1 -- start start 2024-09-11T01:54:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.310+0000 7fe03046e640 10 monclient: init 2024-09-11T01:54:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe03046e640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe03046e640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-11T01:54:47.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe03046e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe0280a2e90 addr v1:172.21.15.12:6789/0 2024-09-11T01:54:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe03046e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe0280a3270 0x7fe0280a3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe03046e640 10 monclient(hunting): picked mon.noname-a con 0x7fe0280a3270 addr v2:172.21.15.12:3300/0 2024-09-11T01:54:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe03046e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe02813d490 0x7fe02813f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe03046e640 10 monclient(hunting): picked mon.noname-b con 0x7fe02813d490 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe03046e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0280a46c0 con 0x7fe0280a2e90 2024-09-11T01:54:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe03046e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe03046e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fe0280a4900 con 0x7fe0280a3270 2024-09-11T01:54:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe03046e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe03046e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe02806ee50 con 0x7fe02813d490 2024-09-11T01:54:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe03046e640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe03046e640 10 monclient(hunting): authenticate will time out at 2293.754644s 2024-09-11T01:54:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe02da09640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7fe0280a3270 msgr2=0x7fe0280a3680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:54:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe02e20a640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7fe0280a2e90 legacy=0x7fe02806dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:54:47.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe02da09640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe0280a3270 0x7fe0280a3680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:54:47.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe02ea0b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe02813d490 0x7fe02813f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:47.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe02ea0b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe02813d490 0x7fe02813f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:35888/0 (socket says 172.21.15.12:35888) 2024-09-11T01:54:47.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe02ea0b640 1 -- 172.21.15.12:0/366478727 learned_addr learned my addr 172.21.15.12:0/366478727 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:47.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe02ea0b640 10 monclient(hunting): get_auth_request con 0x7fe02813d490 auth_method 0 2024-09-11T01:54:47.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe02ea0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:47.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe02ea0b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:47.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe02ea0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:47.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe02ea0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:47.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe02ea0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:47.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.311+0000 7fe02ea0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:47.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.312+0000 7fe02ea0b640 10 monclient(hunting): handle_auth_done global_id 284156 payload 274 2024-09-11T01:54:47.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.312+0000 7fe02ea0b640 1 -- 172.21.15.12:0/366478727 >> v2:172.21.15.12:3300/0 conn(0x7fe0280a3270 msgr2=0x7fe0280a3680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:54:47.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.312+0000 7fe02ea0b640 1 --2- 172.21.15.12:0/366478727 >> v2:172.21.15.12:3300/0 conn(0x7fe0280a3270 0x7fe0280a3680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:47.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.312+0000 7fe02ea0b640 1 -- 172.21.15.12:0/366478727 >> v1:172.21.15.12:6789/0 conn(0x7fe0280a2e90 legacy=0x7fe02806dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:54:47.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.312+0000 7fe02ea0b640 10 monclient: _finish_hunting 0 2024-09-11T01:54:47.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.312+0000 7fe02ea0b640 1 monclient: found mon.noname-b 2024-09-11T01:54:47.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.312+0000 7fe02ea0b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:47.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.312+0000 7fe02ea0b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:54:47.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.312+0000 7fe02ea0b640 1 -- 172.21.15.12:0/366478727 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe02806eb70 con 0x7fe02813d490 2024-09-11T01:54:47.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.314+0000 7fe02ea0b640 1 -- 172.21.15.12:0/366478727 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe02813d490 msgr2=0x7fe02813f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:47.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.314+0000 7fe02ea0b640 1 --2- 172.21.15.12:0/366478727 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe02813d490 0x7fe02813f880 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe028071820 tx=0x7fe02002a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a72856d1f619475b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:47.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.315+0000 7fe02d208640 1 -- 172.21.15.12:0/366478727 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe020036070 con 0x7fe02813d490 2024-09-11T01:54:47.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.315+0000 7fe02d208640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:47.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.315+0000 7fe02d208640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:54:47.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.315+0000 7fe02d208640 10 monclient: dump: 2024-09-11T01:54:47.324 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:47.324 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:47.324 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:47.324 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:47.324 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:47.324 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:47.325 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:47.325 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:47.325 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:47.325 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:47.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.315+0000 7fe02d208640 10 monclient: _finish_auth 0 2024-09-11T01:54:47.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.315+0000 7fe02d208640 10 monclient: _check_auth_tickets 2024-09-11T01:54:47.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.315+0000 7fe02d208640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:47.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.315+0000 7fe02d208640 1 -- 172.21.15.12:0/366478727 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe020007800 con 0x7fe02813d490 2024-09-11T01:54:47.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.315+0000 7fe02d208640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:47.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.315+0000 7fe02d208640 1 -- 172.21.15.12:0/366478727 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe020007bd0 con 0x7fe02813d490 2024-09-11T01:54:47.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.315+0000 7fe03046e640 5 monclient: authenticate success, global_id 284156 2024-09-11T01:54:47.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.315+0000 7fe03046e640 10 monclient: get_monmap_and_config success 2024-09-11T01:54:47.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.315+0000 7fe03046e640 4 set_mon_vals no callback set 2024-09-11T01:54:47.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.315+0000 7fe02d208640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:47.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.315+0000 7fe02d208640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:47.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.315+0000 7fe02d208640 10 monclient: dump: 2024-09-11T01:54:47.326 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:47.326 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:47.326 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:47.326 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:47.326 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:47.326 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:47.326 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:47.327 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:47.327 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:47.327 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:47.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.315+0000 7fe03046e640 10 monclient: shutdown 2024-09-11T01:54:47.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.315+0000 7fe03046e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:47.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.315+0000 7fe03046e640 1 -- 172.21.15.12:0/366478727 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe02813d490 msgr2=0x7fe02813f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:47.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.315+0000 7fe03046e640 1 --2- 172.21.15.12:0/366478727 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe02813d490 0x7fe02813f880 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe028071820 tx=0x7fe02002a560 comp rx=0 tx=0).stop 2024-09-11T01:54:47.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.315+0000 7fe03046e640 1 -- 172.21.15.12:0/366478727 shutdown_connections 2024-09-11T01:54:47.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.315+0000 7fe03046e640 1 --2- 172.21.15.12:0/366478727 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe02813d490 0x7fe02813f880 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:47.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.315+0000 7fe03046e640 1 --2- 172.21.15.12:0/366478727 >> v2:172.21.15.12:3300/0 conn(0x7fe0280a3270 0x7fe0280a3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:47.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.315+0000 7fe03046e640 1 -- 172.21.15.12:0/366478727 >> 172.21.15.12:0/366478727 conn(0x7fe0280a8490 msgr2=0x7fe0280a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:47.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.315+0000 7fe03046e640 1 -- 172.21.15.12:0/366478727 shutdown_connections 2024-09-11T01:54:47.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.315+0000 7fe03046e640 1 -- 172.21.15.12:0/366478727 wait complete. 2024-09-11T01:54:47.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.316+0000 7fe03046e640 10 monclient: build_initial_monmap 2024-09-11T01:54:47.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.316+0000 7fe03046e640 10 monclient: monmap: 2024-09-11T01:54:47.328 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:47.328 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:54:47.328 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:54:47.328 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:54:47.328 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:47.329 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:47.329 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:54:47.329 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:54:47.329 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:54:47.329 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:47.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.316+0000 7fe03046e640 1 Processor -- start 2024-09-11T01:54:47.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.316+0000 7fe03046e640 1 -- start start 2024-09-11T01:54:47.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.316+0000 7fe03046e640 10 monclient: init 2024-09-11T01:54:47.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe03046e640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:47.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe03046e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:54:47.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe03046e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe0280a2e90 0x7fe028137fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:47.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe03046e640 10 monclient(hunting): picked mon.noname-c con 0x7fe0280a2e90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:47.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe03046e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0280a3270 0x7fe028138520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:47.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe03046e640 10 monclient(hunting): picked mon.noname-a con 0x7fe0280a3270 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:54:47.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe03046e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe02813d490 0x7fe02809c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:47.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe03046e640 10 monclient(hunting): picked mon.noname-b con 0x7fe02813d490 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:54:47.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe03046e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:47.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe03046e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe02806fd20 con 0x7fe0280a3270 2024-09-11T01:54:47.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe03046e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:47.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe03046e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe02806fba0 con 0x7fe02813d490 2024-09-11T01:54:47.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe03046e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:47.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe03046e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe02813fdc0 con 0x7fe0280a2e90 2024-09-11T01:54:47.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe03046e640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:47.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe03046e640 10 monclient(hunting): authenticate will time out at 2293.760368s 2024-09-11T01:54:47.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe02da09640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0280a3270 msgr2=0x7fe028138520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:54:47.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe02da09640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0280a3270 0x7fe028138520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:54:47.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe02e20a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe0280a2e90 0x7fe028137fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:47.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe02e20a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe0280a2e90 0x7fe028137fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:35892/0 (socket says 172.21.15.12:35892) 2024-09-11T01:54:47.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe02e20a640 1 -- 172.21.15.12:0/126333663 learned_addr learned my addr 172.21.15.12:0/126333663 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:47.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe02e20a640 10 monclient(hunting): get_auth_request con 0x7fe0280a2e90 auth_method 0 2024-09-11T01:54:47.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe02e20a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:47.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe02e20a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:47.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe02e20a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:47.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe02e20a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:47.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe02e20a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:47.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe02e20a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:47.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe02e20a640 10 monclient(hunting): handle_auth_done global_id 284159 payload 824 2024-09-11T01:54:47.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe02ea0b640 1 --2- 172.21.15.12:0/126333663 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe02813d490 0x7fe02809c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:47.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe02e20a640 1 -- 172.21.15.12:0/126333663 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe02813d490 msgr2=0x7fe02809c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:47.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe02e20a640 1 --2- 172.21.15.12:0/126333663 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe02813d490 0x7fe02809c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:47.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe02e20a640 1 -- 172.21.15.12:0/126333663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0280a3270 msgr2=0x7fe028138520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:54:47.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe02e20a640 1 --2- 172.21.15.12:0/126333663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0280a3270 0x7fe028138520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:47.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe02e20a640 10 monclient: _finish_hunting 0 2024-09-11T01:54:47.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe02e20a640 1 monclient: found mon.noname-c 2024-09-11T01:54:47.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe02e20a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:47.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe02e20a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:54:47.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe02e20a640 1 -- 172.21.15.12:0/126333663 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0280a65b0 con 0x7fe0280a2e90 2024-09-11T01:54:47.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe02ea0b640 10 monclient: get_auth_request con 0x7fe02813d490 auth_method 0 2024-09-11T01:54:47.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.317+0000 7fe02ea0b640 1 --2- 172.21.15.12:0/126333663 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe02813d490 0x7fe02809c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:54:47.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.320+0000 7fe02e20a640 1 -- 172.21.15.12:0/126333663 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe0280a2e90 msgr2=0x7fe028137fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:47.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.320+0000 7fe02e20a640 1 --2- 172.21.15.12:0/126333663 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe0280a2e90 0x7fe028137fe0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe02400b580 tx=0x7fe02400ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:47.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.320+0000 7fe01effd640 1 -- 172.21.15.12:0/126333663 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0240050f0 con 0x7fe0280a2e90 2024-09-11T01:54:47.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.320+0000 7fe01effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:47.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.320+0000 7fe01effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:54:47.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.320+0000 7fe01effd640 10 monclient: dump: 2024-09-11T01:54:47.333 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:47.334 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:47.334 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:47.334 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:47.334 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:47.334 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:47.334 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:47.334 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:47.334 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:47.334 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:47.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.320+0000 7fe01effd640 10 monclient: _finish_auth 0 2024-09-11T01:54:47.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.320+0000 7fe01effd640 10 monclient: _check_auth_tickets 2024-09-11T01:54:47.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.320+0000 7fe01effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:47.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.320+0000 7fe01effd640 1 -- 172.21.15.12:0/126333663 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe024015070 con 0x7fe0280a2e90 2024-09-11T01:54:47.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.320+0000 7fe03046e640 5 monclient: authenticate success, global_id 284159 2024-09-11T01:54:47.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.320+0000 7fe03046e640 10 monclient: _renew_subs 2024-09-11T01:54:47.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.320+0000 7fe03046e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:47.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.320+0000 7fe03046e640 1 -- 172.21.15.12:0/126333663 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe02809edf0 con 0x7fe0280a2e90 2024-09-11T01:54:47.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.320+0000 7fe01effd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:47.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.320+0000 7fe01effd640 1 -- 172.21.15.12:0/126333663 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0240054f0 con 0x7fe0280a2e90 2024-09-11T01:54:47.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.320+0000 7fe01effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:47.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.320+0000 7fe01effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:47.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.320+0000 7fe01effd640 10 monclient: dump: 2024-09-11T01:54:47.335 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:47.335 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:47.335 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:47.335 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:47.336 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:47.336 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:47.336 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:47.336 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:47.336 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:47.336 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:47.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.320+0000 7fe02ca07640 4 set_mon_vals no callback set 2024-09-11T01:54:47.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.320+0000 7fe03046e640 10 monclient: _renew_subs 2024-09-11T01:54:47.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.320+0000 7fe03046e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:47.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.320+0000 7fe03046e640 1 -- 172.21.15.12:0/126333663 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe02809f680 con 0x7fe0280a2e90 2024-09-11T01:54:47.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.321+0000 7fe03046e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:47.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.321+0000 7fe03046e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:47.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.321+0000 7fe03046e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:47.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.321+0000 7fe03046e640 1 -- 172.21.15.12:0/126333663 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0280a3680 con 0x7fe0280a2e90 2024-09-11T01:54:47.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.321+0000 7fe01effd640 1 -- 172.21.15.12:0/126333663 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fe024010480 con 0x7fe0280a2e90 2024-09-11T01:54:47.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.321+0000 7fe01effd640 1 --2- 172.21.15.12:0/126333663 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe00c03e1c0 0x7fe00c040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:47.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.321+0000 7fe01effd640 1 -- 172.21.15.12:0/126333663 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(859..859 src has 676..859) ==== 4875+0+0 (secure 0 0 0) 0x7fe024052df0 con 0x7fe0280a2e90 2024-09-11T01:54:47.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.321+0000 7fe02da09640 1 --2- 172.21.15.12:0/126333663 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe00c03e1c0 0x7fe00c040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:47.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.322+0000 7fe02da09640 10 monclient: get_auth_request con 0x7fe00c03e1c0 auth_method 0 2024-09-11T01:54:47.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.324+0000 7fe02da09640 1 --2- 172.21.15.12:0/126333663 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe00c03e1c0 0x7fe00c040680 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe02809f000 tx=0x7fe018006d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:47.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.326+0000 7fe01effd640 1 -- 172.21.15.12:0/126333663 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe0280a65b0 con 0x7fe0280a2e90 2024-09-11T01:54:47.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.326+0000 7fe01effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:47.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.326+0000 7fe01effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:54:47.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.582+0000 7fe03046e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:54:47.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.582+0000 7fe03046e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:54:47.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.582+0000 7fe03046e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:47.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.582+0000 7fe03046e640 1 -- 172.21.15.12:0/126333663 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe02809eb70 con 0x7fe0280a2e90 2024-09-11T01:54:47.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:47.582+0000 7fe02e20a640 1 --2- 172.21.15.12:0/126333663 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe0280a2e90 0x7fe028137fe0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe02400b580 tx=0x7fe02400ba40 comp rx=0 tx=0).queue_received will delay after 1.72602e+09 on 0x7fe0240056d0 mon_command_ack([{"prefix": "quorum_status"}]=0 v0) 2024-09-11T01:54:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:48.411+0000 7fe01effd640 1 -- 172.21.15.12:0/126333663 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fe0240056d0 con 0x7fe0280a2e90 2024-09-11T01:54:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:48.411+0000 7fe01effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:54:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:48.411+0000 7fe01effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:54:48.414 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":232,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:54:48.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:48.414+0000 7fe03046e640 1 -- 172.21.15.12:0/126333663 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe00c03e1c0 msgr2=0x7fe00c040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:48.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:48.414+0000 7fe03046e640 1 --2- 172.21.15.12:0/126333663 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe00c03e1c0 0x7fe00c040680 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe02809f000 tx=0x7fe018006d30 comp rx=0 tx=0).stop 2024-09-11T01:54:48.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:48.414+0000 7fe03046e640 10 monclient: shutdown 2024-09-11T01:54:48.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:48.414+0000 7fe03046e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:48.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:48.414+0000 7fe03046e640 1 -- 172.21.15.12:0/126333663 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe0280a2e90 msgr2=0x7fe028137fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:48.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:48.414+0000 7fe03046e640 1 --2- 172.21.15.12:0/126333663 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe0280a2e90 0x7fe028137fe0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe02400b580 tx=0x7fe02400ba40 comp rx=0 tx=0).stop 2024-09-11T01:54:48.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:48.414+0000 7fe03046e640 1 -- 172.21.15.12:0/126333663 shutdown_connections 2024-09-11T01:54:48.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:48.414+0000 7fe03046e640 1 --2- 172.21.15.12:0/126333663 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe02813d490 0x7fe02809c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:48.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:48.414+0000 7fe03046e640 1 --2- 172.21.15.12:0/126333663 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe00c03e1c0 0x7fe00c040680 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:48.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:48.414+0000 7fe03046e640 1 --2- 172.21.15.12:0/126333663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0280a3270 0x7fe028138520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:48.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:48.414+0000 7fe03046e640 1 --2- 172.21.15.12:0/126333663 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe0280a2e90 0x7fe028137fe0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:48.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:48.414+0000 7fe03046e640 1 -- 172.21.15.12:0/126333663 >> 172.21.15.12:0/126333663 conn(0x7fe0280a8490 msgr2=0x7fe02813c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:48.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:48.414+0000 7fe03046e640 1 -- 172.21.15.12:0/126333663 shutdown_connections 2024-09-11T01:54:48.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:48.414+0000 7fe03046e640 1 -- 172.21.15.12:0/126333663 wait complete. 2024-09-11T01:54:51.431 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:54:51.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.542+0000 7f39a4d0e640 10 monclient: get_monmap_and_config 2024-09-11T01:54:51.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.542+0000 7f39a4d0e640 10 monclient: build_initial_monmap 2024-09-11T01:54:51.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.542+0000 7f39a4d0e640 10 monclient: monmap: 2024-09-11T01:54:51.546 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:51.546 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:51.546 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:54:51.544498+0000 2024-09-11T01:54:51.546 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:54:51.544498+0000 2024-09-11T01:54:51.546 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:51.546 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:51.546 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:54:51.547 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:54:51.547 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:54:51.547 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:54:51.547 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:54:51.547 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:51.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.543+0000 7f39a4d0e640 1 Processor -- start 2024-09-11T01:54:51.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.543+0000 7f39a4d0e640 1 -- start start 2024-09-11T01:54:51.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.543+0000 7f39a4d0e640 10 monclient: init 2024-09-11T01:54:51.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f39a4d0e640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:51.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f39a4d0e640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-11T01:54:51.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f39a4d0e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f39a00a4170 0x7f39a0073fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:51.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f39a4d0e640 10 monclient(hunting): picked mon.noname-a con 0x7f39a00a4170 addr v2:172.21.15.12:3300/0 2024-09-11T01:54:51.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f39a4d0e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f39a0074500 0x7f39a00a2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:51.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f39a4d0e640 10 monclient(hunting): picked mon.noname-c con 0x7f39a0074500 addr v2:172.21.15.71:3300/0 2024-09-11T01:54:51.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f39a4d0e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f39a00a2aa0 addr v1:172.21.15.12:6789/0 2024-09-11T01:54:51.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f39a4d0e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39a01380d0 con 0x7f39a00a2aa0 2024-09-11T01:54:51.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f39a4d0e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:51.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f39a4d0e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f39a00a4550 con 0x7f39a00a4170 2024-09-11T01:54:51.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f39a4d0e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:51.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f39a4d0e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f39a009fff0 con 0x7f39a0074500 2024-09-11T01:54:51.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f39a4d0e640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:51.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f39a4d0e640 10 monclient(hunting): authenticate will time out at 2297.987181s 2024-09-11T01:54:51.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f399e59c640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7f39a00a4170 msgr2=0x7f39a0073fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:54:51.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f399ed9d640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7f39a00a2aa0 legacy=0x7f39a013efa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:54:51.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f399e59c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f39a00a4170 0x7f39a0073fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:54:51.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f3997fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f39a0074500 0x7f39a00a2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:51.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f3997fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f39a0074500 0x7f39a00a2560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:50032/0 (socket says 172.21.15.12:50032) 2024-09-11T01:54:51.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f3997fff640 1 -- 172.21.15.12:0/1497277921 learned_addr learned my addr 172.21.15.12:0/1497277921 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:51.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f3997fff640 10 monclient(hunting): get_auth_request con 0x7f39a0074500 auth_method 0 2024-09-11T01:54:51.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f3997fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:51.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f3997fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:51.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f3997fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:51.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f3997fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:51.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f3997fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:51.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f3997fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:51.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f3997fff640 10 monclient(hunting): handle_auth_done global_id 274222 payload 274 2024-09-11T01:54:51.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f3997fff640 1 -- 172.21.15.12:0/1497277921 >> v2:172.21.15.12:3300/0 conn(0x7f39a00a4170 msgr2=0x7f39a0073fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:54:51.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f3997fff640 1 --2- 172.21.15.12:0/1497277921 >> v2:172.21.15.12:3300/0 conn(0x7f39a00a4170 0x7f39a0073fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:51.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f3997fff640 1 -- 172.21.15.12:0/1497277921 >> v1:172.21.15.12:6789/0 conn(0x7f39a00a2aa0 legacy=0x7f39a013efa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:54:51.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f3997fff640 10 monclient: _finish_hunting 0 2024-09-11T01:54:51.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f3997fff640 1 monclient: found mon.noname-c 2024-09-11T01:54:51.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f3997fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:51.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f3997fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:54:51.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.544+0000 7f3997fff640 1 -- 172.21.15.12:0/1497277921 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39a0138350 con 0x7f39a0074500 2024-09-11T01:54:51.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.547+0000 7f3997fff640 1 -- 172.21.15.12:0/1497277921 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f39a0074500 msgr2=0x7f39a00a2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:51.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.547+0000 7f3997fff640 1 --2- 172.21.15.12:0/1497277921 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f39a0074500 0x7f39a00a2560 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f39a00a0f90 tx=0x7f398802a160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe44b77e29c69d05 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:51.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f399dd9b640 1 -- 172.21.15.12:0/1497277921 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3988037070 con 0x7f39a0074500 2024-09-11T01:54:51.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f399dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:51.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f399dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:54:51.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f399dd9b640 10 monclient: dump: 2024-09-11T01:54:51.566 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:51.566 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:51.566 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:51.566 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:51.567 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:51.567 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:51.567 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:51.567 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:51.567 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:51.567 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:51.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f399dd9b640 10 monclient: _finish_auth 0 2024-09-11T01:54:51.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f399dd9b640 10 monclient: _check_auth_tickets 2024-09-11T01:54:51.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f399dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:51.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f399dd9b640 1 -- 172.21.15.12:0/1497277921 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3988032640 con 0x7f39a0074500 2024-09-11T01:54:51.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f39a4d0e640 5 monclient: authenticate success, global_id 274222 2024-09-11T01:54:51.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f39a4d0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:54:51.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f399dd9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:51.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f39a4d0e640 10 monclient: get_monmap_and_config success 2024-09-11T01:54:51.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f399dd9b640 1 -- 172.21.15.12:0/1497277921 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3988032a10 con 0x7f39a0074500 2024-09-11T01:54:51.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f399dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:51.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f39a4d0e640 4 set_mon_vals no callback set 2024-09-11T01:54:51.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f399dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:54:51.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f399dd9b640 10 monclient: dump: 2024-09-11T01:54:51.569 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:51.569 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:51.569 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:51.569 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:51.569 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:51.569 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:51.569 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:51.570 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:51.570 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:51.570 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:51.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f39a4d0e640 10 monclient: shutdown 2024-09-11T01:54:51.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f39a4d0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:51.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f39a4d0e640 1 -- 172.21.15.12:0/1497277921 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f39a0074500 msgr2=0x7f39a00a2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:51.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f39a4d0e640 1 --2- 172.21.15.12:0/1497277921 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f39a0074500 0x7f39a00a2560 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f39a00a0f90 tx=0x7f398802a160 comp rx=0 tx=0).stop 2024-09-11T01:54:51.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f39a4d0e640 1 -- 172.21.15.12:0/1497277921 shutdown_connections 2024-09-11T01:54:51.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f39a4d0e640 1 --2- 172.21.15.12:0/1497277921 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f39a0074500 0x7f39a00a2560 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:51.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f39a4d0e640 1 --2- 172.21.15.12:0/1497277921 >> v2:172.21.15.12:3300/0 conn(0x7f39a00a4170 0x7f39a0073fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:51.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f39a4d0e640 1 -- 172.21.15.12:0/1497277921 >> 172.21.15.12:0/1497277921 conn(0x7f39a00a3650 msgr2=0x7f39a00a3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:51.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f39a4d0e640 1 -- 172.21.15.12:0/1497277921 shutdown_connections 2024-09-11T01:54:51.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.548+0000 7f39a4d0e640 1 -- 172.21.15.12:0/1497277921 wait complete. 2024-09-11T01:54:51.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.549+0000 7f39a4d0e640 10 monclient: build_initial_monmap 2024-09-11T01:54:51.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.549+0000 7f39a4d0e640 10 monclient: monmap: 2024-09-11T01:54:51.571 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:51.571 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:54:51.571 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:54:51.572 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:54:51.572 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:51.572 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:51.572 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:54:51.572 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:54:51.572 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:54:51.572 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:51.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.549+0000 7f39a4d0e640 1 Processor -- start 2024-09-11T01:54:51.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.549+0000 7f39a4d0e640 1 -- start start 2024-09-11T01:54:51.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.549+0000 7f39a4d0e640 10 monclient: init 2024-09-11T01:54:51.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f39a4d0e640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:51.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f39a4d0e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:54:51.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f39a4d0e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f39a0074500 0x7f39a01cd350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:51.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f39a4d0e640 10 monclient(hunting): picked mon.noname-c con 0x7f39a0074500 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:51.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f39a4d0e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39a00a2aa0 0x7f39a0131750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:51.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f39a4d0e640 10 monclient(hunting): picked mon.noname-a con 0x7f39a00a2aa0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:54:51.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f39a4d0e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f39a00a4170 0x7f39a0131c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:51.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f39a4d0e640 10 monclient(hunting): picked mon.noname-b con 0x7f39a00a4170 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:54:51.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f39a4d0e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:51.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f39a4d0e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f39a0070840 con 0x7f39a00a2aa0 2024-09-11T01:54:51.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f39a4d0e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:51.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f39a4d0e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f39a0070340 con 0x7f39a00a4170 2024-09-11T01:54:51.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f39a4d0e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:51.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f39a4d0e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f39a00a0660 con 0x7f39a0074500 2024-09-11T01:54:51.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f39a4d0e640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:51.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f39a4d0e640 10 monclient(hunting): authenticate will time out at 2297.993377s 2024-09-11T01:54:51.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f3997fff640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39a00a2aa0 msgr2=0x7f39a0131750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:54:51.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f3997fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39a00a2aa0 0x7f39a0131750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:54:51.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f399e59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f39a0074500 0x7f39a01cd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:51.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f399e59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f39a0074500 0x7f39a01cd350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:35906/0 (socket says 172.21.15.12:35906) 2024-09-11T01:54:51.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f399e59c640 1 -- 172.21.15.12:0/2961557290 learned_addr learned my addr 172.21.15.12:0/2961557290 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:51.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f399e59c640 10 monclient(hunting): get_auth_request con 0x7f39a0074500 auth_method 0 2024-09-11T01:54:51.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f399e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:51.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f399e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:51.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f399e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:51.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f399e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:51.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f399e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:51.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f399e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:51.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f399ed9d640 1 --2- 172.21.15.12:0/2961557290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f39a00a4170 0x7f39a0131c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:51.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f399ed9d640 10 monclient(hunting): get_auth_request con 0x7f39a00a4170 auth_method 0 2024-09-11T01:54:51.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f399ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:51.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f399ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:51.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f399ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:51.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f399e59c640 10 monclient(hunting): handle_auth_done global_id 284162 payload 824 2024-09-11T01:54:51.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f399e59c640 1 -- 172.21.15.12:0/2961557290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f39a00a4170 msgr2=0x7f39a0131c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:51.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f399e59c640 1 --2- 172.21.15.12:0/2961557290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f39a00a4170 0x7f39a0131c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:51.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f399e59c640 1 -- 172.21.15.12:0/2961557290 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39a00a2aa0 msgr2=0x7f39a0131750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:54:51.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f399e59c640 1 --2- 172.21.15.12:0/2961557290 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39a00a2aa0 0x7f39a0131750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:51.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f399e59c640 10 monclient: _finish_hunting 0 2024-09-11T01:54:51.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f399e59c640 1 monclient: found mon.noname-c 2024-09-11T01:54:51.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f399e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:51.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f399e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:54:51.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f399e59c640 1 -- 172.21.15.12:0/2961557290 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39a0135240 con 0x7f39a0074500 2024-09-11T01:54:51.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.550+0000 7f399ed9d640 1 --2- 172.21.15.12:0/2961557290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f39a00a4170 0x7f39a0131c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:54:51.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.553+0000 7f399e59c640 1 -- 172.21.15.12:0/2961557290 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f39a0074500 msgr2=0x7f39a01cd350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:51.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.553+0000 7f399e59c640 1 --2- 172.21.15.12:0/2961557290 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f39a0074500 0x7f39a01cd350 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f399000b550 tx=0x7f399000ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:51.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.553+0000 7f3996ffd640 1 -- 172.21.15.12:0/2961557290 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3990003dd0 con 0x7f39a0074500 2024-09-11T01:54:51.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.553+0000 7f3996ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:51.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.553+0000 7f3996ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:54:51.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.553+0000 7f3996ffd640 10 monclient: dump: 2024-09-11T01:54:51.578 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:51.578 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:51.578 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:51.578 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:51.578 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:51.578 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:51.578 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:51.578 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:51.579 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:51.579 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:51.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.553+0000 7f3996ffd640 10 monclient: _finish_auth 0 2024-09-11T01:54:51.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.553+0000 7f3996ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:54:51.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.553+0000 7f3996ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:51.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.553+0000 7f39a4d0e640 5 monclient: authenticate success, global_id 284162 2024-09-11T01:54:51.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.553+0000 7f3996ffd640 1 -- 172.21.15.12:0/2961557290 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3990015070 con 0x7f39a0074500 2024-09-11T01:54:51.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.553+0000 7f39a4d0e640 10 monclient: _renew_subs 2024-09-11T01:54:51.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.553+0000 7f39a4d0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:51.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.553+0000 7f39a4d0e640 1 -- 172.21.15.12:0/2961557290 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39a011f110 con 0x7f39a0074500 2024-09-11T01:54:51.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.553+0000 7f3996ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:51.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.553+0000 7f3996ffd640 1 -- 172.21.15.12:0/2961557290 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39900052b0 con 0x7f39a0074500 2024-09-11T01:54:51.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.553+0000 7f3996ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:51.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.553+0000 7f3996ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:51.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.553+0000 7f3996ffd640 10 monclient: dump: 2024-09-11T01:54:51.580 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:51.580 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:51.580 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:51.581 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:51.581 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:51.581 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:51.581 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:51.581 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:51.581 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:51.581 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:51.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.553+0000 7f399d59a640 4 set_mon_vals no callback set 2024-09-11T01:54:51.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.553+0000 7f39a4d0e640 10 monclient: _renew_subs 2024-09-11T01:54:51.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.553+0000 7f39a4d0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:51.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.553+0000 7f39a4d0e640 1 -- 172.21.15.12:0/2961557290 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f39a01cd890 con 0x7f39a0074500 2024-09-11T01:54:51.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.554+0000 7f39a4d0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:51.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.554+0000 7f39a4d0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:51.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.554+0000 7f39a4d0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:51.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.554+0000 7f39a4d0e640 1 -- 172.21.15.12:0/2961557290 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39a00742c0 con 0x7f39a0074500 2024-09-11T01:54:51.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.554+0000 7f3996ffd640 1 -- 172.21.15.12:0/2961557290 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f3990005490 con 0x7f39a0074500 2024-09-11T01:54:51.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.554+0000 7f3996ffd640 1 --2- 172.21.15.12:0/2961557290 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f398003e1c0 0x7f3980040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:51.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.554+0000 7f3996ffd640 1 -- 172.21.15.12:0/2961557290 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(859..859 src has 676..859) ==== 4875+0+0 (secure 0 0 0) 0x7f3990052810 con 0x7f39a0074500 2024-09-11T01:54:51.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.555+0000 7f3997fff640 1 --2- 172.21.15.12:0/2961557290 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f398003e1c0 0x7f3980040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:51.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.555+0000 7f3997fff640 10 monclient: get_auth_request con 0x7f398003e1c0 auth_method 0 2024-09-11T01:54:51.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.557+0000 7f3997fff640 1 --2- 172.21.15.12:0/2961557290 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f398003e1c0 0x7f3980040680 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3988031000 tx=0x7f398802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:51.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.559+0000 7f3996ffd640 1 -- 172.21.15.12:0/2961557290 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f39a00742c0 con 0x7f39a0074500 2024-09-11T01:54:51.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.559+0000 7f3996ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:51.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.559+0000 7f3996ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:54:51.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.813+0000 7f39a4d0e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:54:51.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.813+0000 7f39a4d0e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:54:51.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.813+0000 7f39a4d0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:51.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.813+0000 7f39a4d0e640 1 -- 172.21.15.12:0/2961557290 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f39a00a2560 con 0x7f39a0074500 2024-09-11T01:54:51.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.813+0000 7f3996ffd640 1 -- 172.21.15.12:0/2961557290 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7f3990052070 con 0x7f39a0074500 2024-09-11T01:54:51.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.813+0000 7f3996ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:54:51.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.813+0000 7f3996ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:54:51.816 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":232,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:54:51.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.815+0000 7f39a4d0e640 1 -- 172.21.15.12:0/2961557290 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f398003e1c0 msgr2=0x7f3980040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:51.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.815+0000 7f39a4d0e640 1 --2- 172.21.15.12:0/2961557290 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f398003e1c0 0x7f3980040680 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3988031000 tx=0x7f398802e000 comp rx=0 tx=0).stop 2024-09-11T01:54:51.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.815+0000 7f39a4d0e640 10 monclient: shutdown 2024-09-11T01:54:51.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.815+0000 7f39a4d0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:51.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.815+0000 7f39a4d0e640 1 -- 172.21.15.12:0/2961557290 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f39a0074500 msgr2=0x7f39a01cd350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:51.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.815+0000 7f39a4d0e640 1 --2- 172.21.15.12:0/2961557290 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f39a0074500 0x7f39a01cd350 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f399000b550 tx=0x7f399000ba10 comp rx=0 tx=0).stop 2024-09-11T01:54:51.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.815+0000 7f39a4d0e640 1 -- 172.21.15.12:0/2961557290 shutdown_connections 2024-09-11T01:54:51.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.815+0000 7f39a4d0e640 1 --2- 172.21.15.12:0/2961557290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f39a00a4170 0x7f39a0131c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:51.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.815+0000 7f39a4d0e640 1 --2- 172.21.15.12:0/2961557290 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f398003e1c0 0x7f3980040680 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:51.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.815+0000 7f39a4d0e640 1 --2- 172.21.15.12:0/2961557290 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39a00a2aa0 0x7f39a0131750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:51.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.815+0000 7f39a4d0e640 1 --2- 172.21.15.12:0/2961557290 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f39a0074500 0x7f39a01cd350 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:51.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.815+0000 7f39a4d0e640 1 -- 172.21.15.12:0/2961557290 >> 172.21.15.12:0/2961557290 conn(0x7f39a00a3650 msgr2=0x7f39a01bfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:51.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.815+0000 7f39a4d0e640 1 -- 172.21.15.12:0/2961557290 shutdown_connections 2024-09-11T01:54:51.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:51.815+0000 7f39a4d0e640 1 -- 172.21.15.12:0/2961557290 wait complete. 2024-09-11T01:54:54.833 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:54:54.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.944+0000 7fc3929ad640 10 monclient: get_monmap_and_config 2024-09-11T01:54:54.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.944+0000 7fc3929ad640 10 monclient: build_initial_monmap 2024-09-11T01:54:54.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.944+0000 7fc3929ad640 10 monclient: monmap: 2024-09-11T01:54:54.946 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:54.947 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:54.947 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:54:54.945796+0000 2024-09-11T01:54:54.947 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:54:54.945796+0000 2024-09-11T01:54:54.947 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:54.947 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:54.947 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:54:54.947 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:54:54.947 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:54:54.947 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:54:54.947 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:54:54.947 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:54.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.944+0000 7fc3929ad640 1 Processor -- start 2024-09-11T01:54:54.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.944+0000 7fc3929ad640 1 -- start start 2024-09-11T01:54:54.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.944+0000 7fc3929ad640 10 monclient: init 2024-09-11T01:54:54.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.945+0000 7fc3929ad640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:54.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.945+0000 7fc3929ad640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-11T01:54:54.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.945+0000 7fc3929ad640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc38c138230 0x7fc38c138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:54.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.945+0000 7fc3929ad640 10 monclient(hunting): picked mon.noname-b con 0x7fc38c138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:54.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.945+0000 7fc3929ad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc38c09d030 addr v1:172.21.15.71:6789/0 2024-09-11T01:54:54.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.945+0000 7fc3929ad640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc38c09d430 addr v1:172.21.15.12:6789/0 2024-09-11T01:54:54.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.945+0000 7fc3929ad640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc38c0a5e40 con 0x7fc38c09d430 2024-09-11T01:54:54.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.945+0000 7fc3929ad640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc38c0a6e30 con 0x7fc38c09d030 2024-09-11T01:54:54.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.945+0000 7fc3929ad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:54.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.945+0000 7fc390f4a640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7fc38c09d430 legacy=0x7fc38c0a5730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:54:54.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.945+0000 7fc3929ad640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc38c09c9c0 con 0x7fc38c138230 2024-09-11T01:54:54.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.945+0000 7fc3929ad640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:54.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.945+0000 7fc3929ad640 10 monclient(hunting): authenticate will time out at 2301.388703s 2024-09-11T01:54:54.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.945+0000 7fc38bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc38c138230 0x7fc38c138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:54.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.945+0000 7fc38bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc38c138230 0x7fc38c138610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:35920/0 (socket says 172.21.15.12:35920) 2024-09-11T01:54:54.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.945+0000 7fc38bfff640 1 -- 172.21.15.12:0/1384587470 learned_addr learned my addr 172.21.15.12:0/1384587470 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:54.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.945+0000 7fc38bfff640 10 monclient(hunting): get_auth_request con 0x7fc38c138230 auth_method 0 2024-09-11T01:54:54.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.945+0000 7fc38bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:54.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.945+0000 7fc38bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:54.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.945+0000 7fc38bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:54.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.945+0000 7fc38bfff640 1 -- 172.21.15.12:0/1384587470 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc38c138230 msgr2=0x7fc38c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:54:54.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.945+0000 7fc38bfff640 1 -- 172.21.15.12:0/1384587470 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc38c138230 msgr2=0x7fc38c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:54:54.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.945+0000 7fc38bfff640 1 --2- 172.21.15.12:0/1384587470 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc38c138230 0x7fc38c138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:54:54.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.945+0000 7fc38bfff640 1 --2- 172.21.15.12:0/1384587470 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc38c138230 0x7fc38c138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:54:54.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:54.948+0000 7fc38b7fe640 1 -- 172.21.15.12:0/1384587470 >> v1:172.21.15.71:6789/0 conn(0x7fc38c09d030 legacy=0x7fc38c0a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:55.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:55.146+0000 7fc390f4a640 1 -- 172.21.15.12:0/1384587470 >> v1:172.21.15.12:6789/0 conn(0x7fc38c09d430 legacy=0x7fc38c0a5730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:54:55.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:55.146+0000 7fc38bfff640 1 --2- 172.21.15.12:0/1384587470 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc38c138230 0x7fc38c138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:55.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:55.146+0000 7fc38bfff640 10 monclient(hunting): get_auth_request con 0x7fc38c138230 auth_method 0 2024-09-11T01:54:55.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:55.146+0000 7fc38bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:55.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:55.146+0000 7fc38bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:55.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:55.146+0000 7fc38bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:54:55.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:55.146+0000 7fc38bfff640 1 -- 172.21.15.12:0/1384587470 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc38c138230 msgr2=0x7fc38c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:54:55.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:55.146+0000 7fc38bfff640 1 -- 172.21.15.12:0/1384587470 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc38c138230 msgr2=0x7fc38c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:54:55.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:55.146+0000 7fc38bfff640 1 --2- 172.21.15.12:0/1384587470 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc38c138230 0x7fc38c138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:54:55.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:55.146+0000 7fc38bfff640 1 --2- 172.21.15.12:0/1384587470 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc38c138230 0x7fc38c138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:54:55.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:55.547+0000 7fc390f4a640 1 -- 172.21.15.12:0/1384587470 >> v1:172.21.15.12:6789/0 conn(0x7fc38c09d430 legacy=0x7fc38c0a5730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:54:55.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:55.548+0000 7fc38bfff640 1 --2- 172.21.15.12:0/1384587470 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc38c138230 0x7fc38c138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:55.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:55.548+0000 7fc38bfff640 10 monclient(hunting): get_auth_request con 0x7fc38c138230 auth_method 0 2024-09-11T01:54:55.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:55.548+0000 7fc38bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:55.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:55.548+0000 7fc38bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:55.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:55.548+0000 7fc38bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:54:55.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:55.548+0000 7fc38bfff640 1 -- 172.21.15.12:0/1384587470 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc38c138230 msgr2=0x7fc38c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:54:55.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:55.548+0000 7fc38bfff640 1 -- 172.21.15.12:0/1384587470 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc38c138230 msgr2=0x7fc38c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:54:55.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:55.548+0000 7fc38bfff640 1 --2- 172.21.15.12:0/1384587470 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc38c138230 0x7fc38c138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:54:55.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:55.548+0000 7fc38bfff640 1 --2- 172.21.15.12:0/1384587470 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc38c138230 0x7fc38c138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:54:56.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:56.348+0000 7fc390f4a640 1 -- 172.21.15.12:0/1384587470 >> v1:172.21.15.12:6789/0 conn(0x7fc38c09d430 legacy=0x7fc38c0a5730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:54:56.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:56.349+0000 7fc38bfff640 1 --2- 172.21.15.12:0/1384587470 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc38c138230 0x7fc38c138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:56.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:56.349+0000 7fc38bfff640 10 monclient(hunting): get_auth_request con 0x7fc38c138230 auth_method 0 2024-09-11T01:54:56.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:56.349+0000 7fc38bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:56.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:56.349+0000 7fc38bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:56.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:56.349+0000 7fc38bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:54:56.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:56.349+0000 7fc38bfff640 1 -- 172.21.15.12:0/1384587470 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc38c138230 msgr2=0x7fc38c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:54:56.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:56.349+0000 7fc38bfff640 1 -- 172.21.15.12:0/1384587470 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc38c138230 msgr2=0x7fc38c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:54:56.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:56.349+0000 7fc38bfff640 1 --2- 172.21.15.12:0/1384587470 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc38c138230 0x7fc38c138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:54:56.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:56.349+0000 7fc38bfff640 1 --2- 172.21.15.12:0/1384587470 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc38c138230 0x7fc38c138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:54:57.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc389ffb640 10 monclient(hunting): tick 2024-09-11T01:54:57.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc389ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:54:57.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc389ffb640 1 monclient(hunting): continuing hunt 2024-09-11T01:54:57.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc389ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:54:57.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc389ffb640 1 -- 172.21.15.12:0/1384587470 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc38c138230 msgr2=0x7fc38c138610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:54:57.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc389ffb640 1 --2- 172.21.15.12:0/1384587470 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc38c138230 0x7fc38c138610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:57.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc389ffb640 1 -- 172.21.15.12:0/1384587470 >> v1:172.21.15.71:6789/0 conn(0x7fc38c09d030 legacy=0x7fc38c0a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:57.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc389ffb640 1 -- 172.21.15.12:0/1384587470 >> v1:172.21.15.12:6789/0 conn(0x7fc38c09d430 legacy=0x7fc38c0a5730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:54:57.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc389ffb640 10 monclient: _add_conns ranks=[4,3] 2024-09-11T01:54:57.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc389ffb640 1 --2- 172.21.15.12:0/1384587470 >> v2:172.21.15.71:3300/0 conn(0x7fc378003770 0x7fc378005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:57.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc389ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fc378003770 addr v2:172.21.15.71:3300/0 2024-09-11T01:54:57.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc389ffb640 1 --2- 172.21.15.12:0/1384587470 >> v2:172.21.15.12:3300/0 conn(0x7fc3780062c0 0x7fc378008780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:57.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc389ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fc3780062c0 addr v2:172.21.15.12:3300/0 2024-09-11T01:54:57.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc389ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:57.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc389ffb640 1 -- 172.21.15.12:0/1384587470 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc38c09c9c0 con 0x7fc3780062c0 2024-09-11T01:54:57.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc389ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:57.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc389ffb640 1 -- 172.21.15.12:0/1384587470 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fc378008e40 con 0x7fc378003770 2024-09-11T01:54:57.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc389ffb640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:57.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc38b7fe640 1 -- 172.21.15.12:0/1384587470 >> v2:172.21.15.12:3300/0 conn(0x7fc3780062c0 msgr2=0x7fc378008780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:54:57.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc38b7fe640 1 --2- 172.21.15.12:0/1384587470 >> v2:172.21.15.12:3300/0 conn(0x7fc3780062c0 0x7fc378008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:54:57.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc38bfff640 1 --2- 172.21.15.12:0/1384587470 >> v2:172.21.15.71:3300/0 conn(0x7fc378003770 0x7fc378005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:57.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc38bfff640 10 monclient(hunting): get_auth_request con 0x7fc378003770 auth_method 0 2024-09-11T01:54:57.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc38bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:57.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc38bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:57.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc38bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:57.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc38bfff640 1 -- 172.21.15.12:0/1384587470 >> v2:172.21.15.71:3300/0 conn(0x7fc378003770 msgr2=0x7fc378005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:54:57.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc38bfff640 1 -- 172.21.15.12:0/1384587470 >> v2:172.21.15.71:3300/0 conn(0x7fc378003770 msgr2=0x7fc378005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:54:57.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc38bfff640 1 --2- 172.21.15.12:0/1384587470 >> v2:172.21.15.71:3300/0 conn(0x7fc378003770 0x7fc378005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:54:57.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:57.946+0000 7fc38bfff640 1 --2- 172.21.15.12:0/1384587470 >> v2:172.21.15.71:3300/0 conn(0x7fc378003770 0x7fc378005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:54:58.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:58.147+0000 7fc38b7fe640 1 -- 172.21.15.12:0/1384587470 >> v2:172.21.15.12:3300/0 conn(0x7fc3780062c0 msgr2=0x7fc378008780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:54:58.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:58.147+0000 7fc38b7fe640 1 --2- 172.21.15.12:0/1384587470 >> v2:172.21.15.12:3300/0 conn(0x7fc3780062c0 0x7fc378008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:54:58.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:58.147+0000 7fc38bfff640 1 --2- 172.21.15.12:0/1384587470 >> v2:172.21.15.71:3300/0 conn(0x7fc378003770 0x7fc378005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:58.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:58.147+0000 7fc38bfff640 10 monclient(hunting): get_auth_request con 0x7fc378003770 auth_method 0 2024-09-11T01:54:58.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:58.147+0000 7fc38bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:58.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:58.147+0000 7fc38bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:58.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:58.147+0000 7fc38bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:54:58.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:58.147+0000 7fc38bfff640 1 -- 172.21.15.12:0/1384587470 >> v2:172.21.15.71:3300/0 conn(0x7fc378003770 msgr2=0x7fc378005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:54:58.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:58.147+0000 7fc38bfff640 1 -- 172.21.15.12:0/1384587470 >> v2:172.21.15.71:3300/0 conn(0x7fc378003770 msgr2=0x7fc378005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:54:58.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:58.147+0000 7fc38bfff640 1 --2- 172.21.15.12:0/1384587470 >> v2:172.21.15.71:3300/0 conn(0x7fc378003770 0x7fc378005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:54:58.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:58.147+0000 7fc38bfff640 1 --2- 172.21.15.12:0/1384587470 >> v2:172.21.15.71:3300/0 conn(0x7fc378003770 0x7fc378005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:54:58.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:58.548+0000 7fc38b7fe640 1 -- 172.21.15.12:0/1384587470 >> v2:172.21.15.12:3300/0 conn(0x7fc3780062c0 msgr2=0x7fc378008780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:54:58.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:58.548+0000 7fc38bfff640 1 --2- 172.21.15.12:0/1384587470 >> v2:172.21.15.71:3300/0 conn(0x7fc378003770 0x7fc378005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:58.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:58.548+0000 7fc38b7fe640 1 --2- 172.21.15.12:0/1384587470 >> v2:172.21.15.12:3300/0 conn(0x7fc3780062c0 0x7fc378008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:54:58.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:58.548+0000 7fc38bfff640 10 monclient(hunting): get_auth_request con 0x7fc378003770 auth_method 0 2024-09-11T01:54:58.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:58.548+0000 7fc38bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:58.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:58.548+0000 7fc38bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:58.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:58.548+0000 7fc38bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:54:58.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:58.548+0000 7fc38bfff640 1 -- 172.21.15.12:0/1384587470 >> v2:172.21.15.71:3300/0 conn(0x7fc378003770 msgr2=0x7fc378005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:54:58.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:58.548+0000 7fc38bfff640 1 -- 172.21.15.12:0/1384587470 >> v2:172.21.15.71:3300/0 conn(0x7fc378003770 msgr2=0x7fc378005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:54:58.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:58.548+0000 7fc38bfff640 1 --2- 172.21.15.12:0/1384587470 >> v2:172.21.15.71:3300/0 conn(0x7fc378003770 0x7fc378005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:54:58.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:58.548+0000 7fc38bfff640 1 --2- 172.21.15.12:0/1384587470 >> v2:172.21.15.71:3300/0 conn(0x7fc378003770 0x7fc378005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:54:59.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.349+0000 7fc38b7fe640 1 -- 172.21.15.12:0/1384587470 >> v2:172.21.15.12:3300/0 conn(0x7fc3780062c0 msgr2=0x7fc378008780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:54:59.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.349+0000 7fc38b7fe640 1 --2- 172.21.15.12:0/1384587470 >> v2:172.21.15.12:3300/0 conn(0x7fc3780062c0 0x7fc378008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:54:59.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.349+0000 7fc38affd640 1 -- 172.21.15.12:0/1384587470 DispatchQueue::entry inject delay of 0.002000 2024-09-11T01:54:59.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.349+0000 7fc38bfff640 1 --2- 172.21.15.12:0/1384587470 >> v2:172.21.15.71:3300/0 conn(0x7fc378003770 0x7fc378005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:59.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.349+0000 7fc38bfff640 10 monclient(hunting): get_auth_request con 0x7fc378003770 auth_method 0 2024-09-11T01:54:59.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.349+0000 7fc38bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:59.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.349+0000 7fc38bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:59.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.349+0000 7fc38bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:54:59.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.349+0000 7fc38bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:59.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.349+0000 7fc38bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:59.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.349+0000 7fc38bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:59.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.349+0000 7fc38bfff640 10 monclient(hunting): handle_auth_done global_id 274228 payload 274 2024-09-11T01:54:59.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.349+0000 7fc38bfff640 1 -- 172.21.15.12:0/1384587470 >> v2:172.21.15.12:3300/0 conn(0x7fc3780062c0 msgr2=0x7fc378008780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:54:59.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.349+0000 7fc38bfff640 1 --2- 172.21.15.12:0/1384587470 >> v2:172.21.15.12:3300/0 conn(0x7fc3780062c0 0x7fc378008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:59.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.349+0000 7fc38bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:54:59.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.349+0000 7fc38bfff640 1 monclient: found mon.noname-c 2024-09-11T01:54:59.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.349+0000 7fc38bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:59.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.349+0000 7fc38bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:54:59.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.349+0000 7fc38bfff640 1 -- 172.21.15.12:0/1384587470 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc38c09ba20 con 0x7fc378003770 2024-09-11T01:54:59.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.353+0000 7fc38bfff640 1 -- 172.21.15.12:0/1384587470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc378003770 msgr2=0x7fc378005bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:59.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.353+0000 7fc38bfff640 1 --2- 172.21.15.12:0/1384587470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc378003770 0x7fc378005bf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc378003700 tx=0x7fc37c02bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1784a4e2acb82c65 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:59.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.353+0000 7fc38affd640 1 -- 172.21.15.12:0/1384587470 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c032070 con 0x7fc378003770 2024-09-11T01:54:59.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.353+0000 7fc38affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:59.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.353+0000 7fc38affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:54:59.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.353+0000 7fc38affd640 10 monclient: dump: 2024-09-11T01:54:59.356 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:59.356 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:59.356 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.356 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.356 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:59.356 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:59.357 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.353+0000 7fc38affd640 10 monclient: _finish_auth 0 2024-09-11T01:54:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.353+0000 7fc38affd640 10 monclient: _check_auth_tickets 2024-09-11T01:54:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.353+0000 7fc38affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.353+0000 7fc38affd640 1 -- 172.21.15.12:0/1384587470 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc37c030070 con 0x7fc378003770 2024-09-11T01:54:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.353+0000 7fc38affd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.353+0000 7fc38affd640 1 -- 172.21.15.12:0/1384587470 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c00f040 con 0x7fc378003770 2024-09-11T01:54:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.353+0000 7fc3929ad640 5 monclient: authenticate success, global_id 274228 2024-09-11T01:54:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.353+0000 7fc3929ad640 10 monclient: get_monmap_and_config success 2024-09-11T01:54:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.353+0000 7fc3929ad640 4 set_mon_vals no callback set 2024-09-11T01:54:59.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.353+0000 7fc38affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:59.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.353+0000 7fc38affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:54:59.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.353+0000 7fc38affd640 10 monclient: dump: 2024-09-11T01:54:59.358 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:59.358 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:59.358 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.358 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.358 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:59.358 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:59.358 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:59.358 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:59.358 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:59.358 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:59.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.354+0000 7fc3929ad640 10 monclient: shutdown 2024-09-11T01:54:59.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.354+0000 7fc3929ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:59.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.354+0000 7fc3929ad640 1 -- 172.21.15.12:0/1384587470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc378003770 msgr2=0x7fc378005bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:59.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.354+0000 7fc3929ad640 1 --2- 172.21.15.12:0/1384587470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc378003770 0x7fc378005bf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc378003700 tx=0x7fc37c02bec0 comp rx=0 tx=0).stop 2024-09-11T01:54:59.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.354+0000 7fc390f4a640 1 -- 172.21.15.12:0/1384587470 reap_dead start 2024-09-11T01:54:59.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.354+0000 7fc3929ad640 1 -- 172.21.15.12:0/1384587470 shutdown_connections 2024-09-11T01:54:59.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.354+0000 7fc3929ad640 1 --2- 172.21.15.12:0/1384587470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc378003770 0x7fc378005bf0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:59.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.354+0000 7fc3929ad640 1 -- 172.21.15.12:0/1384587470 >> 172.21.15.12:0/1384587470 conn(0x7fc38c0a7120 msgr2=0x7fc38c0a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:59.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.354+0000 7fc3929ad640 1 -- 172.21.15.12:0/1384587470 shutdown_connections 2024-09-11T01:54:59.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.354+0000 7fc3929ad640 1 -- 172.21.15.12:0/1384587470 wait complete. 2024-09-11T01:54:59.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.355+0000 7fc3929ad640 10 monclient: build_initial_monmap 2024-09-11T01:54:59.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.355+0000 7fc3929ad640 10 monclient: monmap: 2024-09-11T01:54:59.364 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:59.364 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:54:59.364 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:54:59.364 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:54:59.364 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:59.364 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:59.364 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:54:59.364 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:54:59.364 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:54:59.365 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.357+0000 7fc3929ad640 1 Processor -- start 2024-09-11T01:54:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.357+0000 7fc3929ad640 1 -- start start 2024-09-11T01:54:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.357+0000 7fc3929ad640 10 monclient: init 2024-09-11T01:54:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.357+0000 7fc3929ad640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.357+0000 7fc3929ad640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:54:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc3929ad640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc378003770 0x7fc38c13f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc3929ad640 10 monclient(hunting): picked mon.noname-c con 0x7fc378003770 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc3929ad640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc38c1cd5d0 0x7fc38c1cda30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc3929ad640 10 monclient(hunting): picked mon.noname-b con 0x7fc38c1cd5d0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:54:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc3929ad640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc38c13cc20 0x7fc38c13d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc3929ad640 10 monclient(hunting): picked mon.noname-a con 0x7fc38c13cc20 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:54:59.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc3929ad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc3929ad640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc38c138b50 con 0x7fc38c13cc20 2024-09-11T01:54:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc3929ad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc3929ad640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc38c06f7f0 con 0x7fc38c1cd5d0 2024-09-11T01:54:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc3929ad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc3929ad640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc38c1c5ef0 con 0x7fc378003770 2024-09-11T01:54:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc3929ad640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc390f4a640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc38c13cc20 msgr2=0x7fc38c13d0a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:54:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc3929ad640 10 monclient(hunting): authenticate will time out at 2305.800187s 2024-09-11T01:54:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc390f4a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc38c13cc20 0x7fc38c13d0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:54:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc378003770 0x7fc38c13f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc378003770 0x7fc38c13f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:59932/0 (socket says 172.21.15.12:59932) 2024-09-11T01:54:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38bfff640 1 -- 172.21.15.12:0/3909499751 learned_addr learned my addr 172.21.15.12:0/3909499751 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:59.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38bfff640 10 monclient(hunting): get_auth_request con 0x7fc378003770 auth_method 0 2024-09-11T01:54:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38b7fe640 1 --2- 172.21.15.12:0/3909499751 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc38c1cd5d0 0x7fc38c1cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38b7fe640 10 monclient(hunting): get_auth_request con 0x7fc38c1cd5d0 auth_method 0 2024-09-11T01:54:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:59.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38bfff640 10 monclient(hunting): handle_auth_done global_id 284165 payload 824 2024-09-11T01:54:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38bfff640 1 -- 172.21.15.12:0/3909499751 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc38c1cd5d0 msgr2=0x7fc38c1cda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38bfff640 1 --2- 172.21.15.12:0/3909499751 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc38c1cd5d0 0x7fc38c1cda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38bfff640 1 -- 172.21.15.12:0/3909499751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc38c13cc20 msgr2=0x7fc38c13d0a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:54:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38bfff640 1 --2- 172.21.15.12:0/3909499751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc38c13cc20 0x7fc38c13d0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:54:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38bfff640 1 monclient: found mon.noname-c 2024-09-11T01:54:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:54:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.358+0000 7fc38bfff640 1 -- 172.21.15.12:0/3909499751 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc38c137400 con 0x7fc378003770 2024-09-11T01:54:59.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.360+0000 7fc38bfff640 1 -- 172.21.15.12:0/3909499751 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc378003770 msgr2=0x7fc38c13f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.360+0000 7fc38bfff640 1 --2- 172.21.15.12:0/3909499751 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc378003770 0x7fc38c13f290 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc37c018ba0 tx=0x7fc37c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.360+0000 7fc388ff9640 1 -- 172.21.15.12:0/3909499751 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c046070 con 0x7fc378003770 2024-09-11T01:54:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.360+0000 7fc388ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.360+0000 7fc388ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:54:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.360+0000 7fc388ff9640 10 monclient: dump: 2024-09-11T01:54:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:59.369 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:59.370 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.360+0000 7fc388ff9640 10 monclient: _finish_auth 0 2024-09-11T01:54:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.360+0000 7fc388ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:54:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.360+0000 7fc388ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.360+0000 7fc388ff9640 1 -- 172.21.15.12:0/3909499751 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc37c037a80 con 0x7fc378003770 2024-09-11T01:54:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.360+0000 7fc3929ad640 5 monclient: authenticate success, global_id 284165 2024-09-11T01:54:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.360+0000 7fc3929ad640 10 monclient: _renew_subs 2024-09-11T01:54:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.360+0000 7fc3929ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.360+0000 7fc3929ad640 1 -- 172.21.15.12:0/3909499751 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc38c1407b0 con 0x7fc378003770 2024-09-11T01:54:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.360+0000 7fc388ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.360+0000 7fc388ff9640 1 -- 172.21.15.12:0/3909499751 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c006bf0 con 0x7fc378003770 2024-09-11T01:54:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.360+0000 7fc388ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.360+0000 7fc388ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:59.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.360+0000 7fc388ff9640 10 monclient: dump: 2024-09-11T01:54:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:59.371 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.360+0000 7fc38a7fc640 4 set_mon_vals no callback set 2024-09-11T01:54:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.360+0000 7fc3929ad640 10 monclient: _renew_subs 2024-09-11T01:54:59.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.360+0000 7fc3929ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:59.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.360+0000 7fc3929ad640 1 -- 172.21.15.12:0/3909499751 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc38c140a90 con 0x7fc378003770 2024-09-11T01:54:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.361+0000 7fc388ff9640 1 -- 172.21.15.12:0/3909499751 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fc37c040030 con 0x7fc378003770 2024-09-11T01:54:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.361+0000 7fc388ff9640 1 --2- 172.21.15.12:0/3909499751 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc37803ef30 0x7fc3780413f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.361+0000 7fc388ff9640 1 -- 172.21.15.12:0/3909499751 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(859..859 src has 676..859) ==== 4875+0+0 (secure 0 0 0) 0x7fc37c0742e0 con 0x7fc378003770 2024-09-11T01:54:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.361+0000 7fc3929ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.361+0000 7fc3929ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.361+0000 7fc3929ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.361+0000 7fc3929ad640 1 -- 172.21.15.12:0/3909499751 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc38c138610 con 0x7fc378003770 2024-09-11T01:54:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.362+0000 7fc38b7fe640 1 --2- 172.21.15.12:0/3909499751 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc37803ef30 0x7fc3780413f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.362+0000 7fc38b7fe640 10 monclient: get_auth_request con 0x7fc37803ef30 auth_method 0 2024-09-11T01:54:59.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.365+0000 7fc38b7fe640 1 --2- 172.21.15.12:0/3909499751 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc37803ef30 0x7fc3780413f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc380000f60 tx=0x7fc380009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.367+0000 7fc388ff9640 1 -- 172.21.15.12:0/3909499751 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc37c040d60 con 0x7fc378003770 2024-09-11T01:54:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.367+0000 7fc388ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:59.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.367+0000 7fc388ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:54:59.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.625+0000 7fc3929ad640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:54:59.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.625+0000 7fc3929ad640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:54:59.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.625+0000 7fc3929ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:59.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.625+0000 7fc3929ad640 1 -- 172.21.15.12:0/3909499751 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc38c13fc20 con 0x7fc378003770 2024-09-11T01:54:59.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.626+0000 7fc388ff9640 1 -- 172.21.15.12:0/3909499751 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fc37c050020 con 0x7fc378003770 2024-09-11T01:54:59.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.626+0000 7fc388ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:54:59.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.626+0000 7fc388ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:54:59.629 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":236,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:54:59.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.628+0000 7fc3929ad640 1 -- 172.21.15.12:0/3909499751 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc37803ef30 msgr2=0x7fc3780413f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:59.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.628+0000 7fc3929ad640 1 --2- 172.21.15.12:0/3909499751 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc37803ef30 0x7fc3780413f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc380000f60 tx=0x7fc380009040 comp rx=0 tx=0).stop 2024-09-11T01:54:59.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.628+0000 7fc3929ad640 10 monclient: shutdown 2024-09-11T01:54:59.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.628+0000 7fc3929ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:59.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.628+0000 7fc3929ad640 1 -- 172.21.15.12:0/3909499751 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc378003770 msgr2=0x7fc38c13f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:59.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.628+0000 7fc3929ad640 1 --2- 172.21.15.12:0/3909499751 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc378003770 0x7fc38c13f290 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc37c018ba0 tx=0x7fc37c004730 comp rx=0 tx=0).stop 2024-09-11T01:54:59.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.628+0000 7fc3929ad640 1 -- 172.21.15.12:0/3909499751 shutdown_connections 2024-09-11T01:54:59.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.628+0000 7fc3929ad640 1 --2- 172.21.15.12:0/3909499751 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc37803ef30 0x7fc3780413f0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:59.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.628+0000 7fc3929ad640 1 --2- 172.21.15.12:0/3909499751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc38c13cc20 0x7fc38c13d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:59.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.628+0000 7fc3929ad640 1 --2- 172.21.15.12:0/3909499751 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc38c1cd5d0 0x7fc38c1cda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:59.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.628+0000 7fc3929ad640 1 --2- 172.21.15.12:0/3909499751 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc378003770 0x7fc38c13f290 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:59.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.628+0000 7fc3929ad640 1 -- 172.21.15.12:0/3909499751 >> 172.21.15.12:0/3909499751 conn(0x7fc38c0a7120 msgr2=0x7fc38c0a1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:59.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.628+0000 7fc3929ad640 1 -- 172.21.15.12:0/3909499751 shutdown_connections 2024-09-11T01:54:59.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.628+0000 7fc3929ad640 1 -- 172.21.15.12:0/3909499751 wait complete. 2024-09-11T01:54:59.641 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:54:59.641 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T01:54:59.642 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:54:59.740 INFO:tasks.rados.rados.0.smithi071.stdout:3768: done (3 left) 2024-09-11T01:54:59.740 INFO:tasks.rados.rados.0.smithi071.stdout:3769: snap_create 2024-09-11T01:54:59.740 INFO:tasks.rados.rados.0.smithi071.stdout:3765: finishing write tid 2 to smithi07134777-24 2024-09-11T01:54:59.741 INFO:tasks.rados.rados.0.smithi071.stdout:3765: finishing write tid 3 to smithi07134777-24 2024-09-11T01:54:59.741 INFO:tasks.rados.rados.0.smithi071.stdout:3765: finishing write tid 4 to smithi07134777-24 2024-09-11T01:54:59.741 INFO:tasks.rados.rados.0.smithi071.stdout:3765: finishing write tid 5 to smithi07134777-24 2024-09-11T01:54:59.741 INFO:tasks.rados.rados.0.smithi071.stdout:3765: oid 24 updating version 0 to 2432 2024-09-11T01:54:59.741 INFO:tasks.rados.rados.0.smithi071.stdout:3765: oid 24 updating version 2432 to 2433 2024-09-11T01:54:59.741 INFO:tasks.rados.rados.0.smithi071.stdout:3765: oid 24 updating version 2433 to 2434 2024-09-11T01:54:59.741 INFO:tasks.rados.rados.0.smithi071.stdout:3765: oid 24 version 2434 is already newer than 2431 2024-09-11T01:54:59.741 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 2434 (ObjNum 1289 snap 424 seq_num 1289) dirty exists 2024-09-11T01:54:59.741 INFO:tasks.rados.rados.0.smithi071.stdout:3765: left oid 24 (ObjNum 1289 snap 424 seq_num 1289) 2024-09-11T01:54:59.741 INFO:tasks.rados.rados.0.smithi071.stdout:3766: finishing write tid 1 to smithi07134777-16 2024-09-11T01:54:59.741 INFO:tasks.rados.rados.0.smithi071.stdout:3766: finishing write tid 2 to smithi07134777-16 2024-09-11T01:54:59.741 INFO:tasks.rados.rados.0.smithi071.stdout:3767: expect (ObjNum 1242 snap 411 seq_num 1242) 2024-09-11T01:54:59.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.746+0000 7f46cd42a640 10 monclient: get_monmap_and_config 2024-09-11T01:54:59.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.746+0000 7f46cd42a640 10 monclient: build_initial_monmap 2024-09-11T01:54:59.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.746+0000 7f46cd42a640 10 monclient: monmap: 2024-09-11T01:54:59.748 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:59.748 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:59.748 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:54:59.747387+0000 2024-09-11T01:54:59.748 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:54:59.747387+0000 2024-09-11T01:54:59.748 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:59.749 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:59.749 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:54:59.749 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:54:59.749 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:54:59.749 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:54:59.749 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:54:59.749 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:59.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46cd42a640 1 Processor -- start 2024-09-11T01:54:59.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46cd42a640 1 -- start start 2024-09-11T01:54:59.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46cd42a640 10 monclient: init 2024-09-11T01:54:59.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46cd42a640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:59.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46cd42a640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-11T01:54:59.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46cd42a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f46c8134ad0 addr v1:172.21.15.12:6789/0 2024-09-11T01:54:59.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46cd42a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f46c80a2200 addr v1:172.21.15.71:6789/0 2024-09-11T01:54:59.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46cd42a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c81355c0 0x7f46c813d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:59.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46cd42a640 10 monclient(hunting): picked mon.noname-b con 0x7f46c81355c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:59.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46cd42a640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46c809ed30 con 0x7f46c8134ad0 2024-09-11T01:54:59.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46cd42a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46c809f010 con 0x7f46c80a2200 2024-09-11T01:54:59.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46cd42a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:59.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46c6ffd640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7f46c8134ad0 legacy=0x7f46c8134eb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:54:59.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46cd42a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f46c809f5d0 con 0x7f46c81355c0 2024-09-11T01:54:59.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46cd42a640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:59.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46cd42a640 10 monclient(hunting): authenticate will time out at 2306.189977s 2024-09-11T01:54:59.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46c77fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c81355c0 0x7f46c813d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:59.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46c77fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c81355c0 0x7f46c813d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:59936/0 (socket says 172.21.15.12:59936) 2024-09-11T01:54:59.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46c77fe640 1 -- 172.21.15.12:0/1157001807 learned_addr learned my addr 172.21.15.12:0/1157001807 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:59.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46c77fe640 10 monclient(hunting): get_auth_request con 0x7f46c81355c0 auth_method 0 2024-09-11T01:54:59.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:59.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:59.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:59.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:59.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:59.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:59.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46c77fe640 10 monclient(hunting): handle_auth_done global_id 284168 payload 274 2024-09-11T01:54:59.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46c77fe640 1 -- 172.21.15.12:0/1157001807 >> v1:172.21.15.71:6789/0 conn(0x7f46c80a2200 legacy=0x7f46c813a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:54:59.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46c77fe640 1 -- 172.21.15.12:0/1157001807 >> v1:172.21.15.12:6789/0 conn(0x7f46c8134ad0 legacy=0x7f46c8134eb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:54:59.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46c77fe640 10 monclient: _finish_hunting 0 2024-09-11T01:54:59.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46c77fe640 1 monclient: found mon.noname-b 2024-09-11T01:54:59.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:59.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46c77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:54:59.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.747+0000 7f46c77fe640 1 -- 172.21.15.12:0/1157001807 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46c809f2f0 con 0x7f46c81355c0 2024-09-11T01:54:59.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46c77fe640 1 -- 172.21.15.12:0/1157001807 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c81355c0 msgr2=0x7f46c813d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:59.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46c77fe640 1 --2- 172.21.15.12:0/1157001807 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c81355c0 0x7f46c813d730 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f46c0009eb0 tx=0x7f46c002a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0269e4c8e31b38 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:59.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46c67fc640 1 -- 172.21.15.12:0/1157001807 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c0036070 con 0x7f46c81355c0 2024-09-11T01:54:59.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:59.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46c67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:54:59.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46c67fc640 10 monclient: dump: 2024-09-11T01:54:59.761 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:59.761 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:59.761 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.761 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.761 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:59.761 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:59.762 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:59.762 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:59.762 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:59.762 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:59.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46c67fc640 10 monclient: _finish_auth 0 2024-09-11T01:54:59.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46c67fc640 10 monclient: _check_auth_tickets 2024-09-11T01:54:59.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:59.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46cd42a640 5 monclient: authenticate success, global_id 284168 2024-09-11T01:54:59.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46c67fc640 1 -- 172.21.15.12:0/1157001807 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46c0007800 con 0x7f46c81355c0 2024-09-11T01:54:59.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46cd42a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:54:59.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46c67fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:59.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46cd42a640 10 monclient: get_monmap_and_config success 2024-09-11T01:54:59.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46cd42a640 4 set_mon_vals no callback set 2024-09-11T01:54:59.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46c67fc640 1 -- 172.21.15.12:0/1157001807 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c0007bd0 con 0x7f46c81355c0 2024-09-11T01:54:59.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:59.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:59.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46c67fc640 10 monclient: dump: 2024-09-11T01:54:59.763 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:59.763 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:59.763 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.763 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.763 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:59.763 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:59.763 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:59.763 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:59.763 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:59.763 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:59.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46cd42a640 10 monclient: shutdown 2024-09-11T01:54:59.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46cd42a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:59.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46cd42a640 1 -- 172.21.15.12:0/1157001807 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c81355c0 msgr2=0x7f46c813d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:59.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46cd42a640 1 --2- 172.21.15.12:0/1157001807 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c81355c0 0x7f46c813d730 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f46c0009eb0 tx=0x7f46c002a560 comp rx=0 tx=0).stop 2024-09-11T01:54:59.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46cd42a640 1 -- 172.21.15.12:0/1157001807 shutdown_connections 2024-09-11T01:54:59.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46cd42a640 1 --2- 172.21.15.12:0/1157001807 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c81355c0 0x7f46c813d730 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:59.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46cd42a640 1 -- 172.21.15.12:0/1157001807 >> 172.21.15.12:0/1157001807 conn(0x7f46c80a2f70 msgr2=0x7f46c80a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:59.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46cd42a640 1 -- 172.21.15.12:0/1157001807 shutdown_connections 2024-09-11T01:54:59.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46cd42a640 1 -- 172.21.15.12:0/1157001807 wait complete. 2024-09-11T01:54:59.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46cd42a640 10 monclient: build_initial_monmap 2024-09-11T01:54:59.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.751+0000 7f46cd42a640 10 monclient: monmap: 2024-09-11T01:54:59.764 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:59.764 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:54:59.765 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:54:59.765 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:54:59.765 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:59.765 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:59.765 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:54:59.765 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:54:59.765 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:54:59.765 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:59.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46cd42a640 1 Processor -- start 2024-09-11T01:54:59.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46cd42a640 1 -- start start 2024-09-11T01:54:59.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46cd42a640 10 monclient: init 2024-09-11T01:54:59.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46cd42a640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:59.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46cd42a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:54:59.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46cd42a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46c80a2200 0x7f46c81cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:59.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46cd42a640 10 monclient(hunting): picked mon.noname-a con 0x7f46c80a2200 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:54:59.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46cd42a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c8134ad0 0x7f46c8133d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:59.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46cd42a640 10 monclient(hunting): picked mon.noname-c con 0x7f46c8134ad0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:59.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46cd42a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46c81355c0 0x7f46c8134290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:59.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46cd42a640 10 monclient(hunting): picked mon.noname-b con 0x7f46c81355c0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:54:59.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46cd42a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:59.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46cd42a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f46c809cde0 con 0x7f46c80a2200 2024-09-11T01:54:59.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46cd42a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:59.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46cd42a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f46c80a01e0 con 0x7f46c81355c0 2024-09-11T01:54:59.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46cd42a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:59.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46cd42a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f46c80a0c90 con 0x7f46c8134ad0 2024-09-11T01:54:59.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46cd42a640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:59.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46cd42a640 10 monclient(hunting): authenticate will time out at 2306.195653s 2024-09-11T01:54:59.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46c6ffd640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46c80a2200 msgr2=0x7f46c81cd1f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:54:59.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46c6ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46c80a2200 0x7f46c81cd1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:54:59.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46bffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c8134ad0 0x7f46c8133d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:59.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46bffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c8134ad0 0x7f46c8133d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:59946/0 (socket says 172.21.15.12:59946) 2024-09-11T01:54:59.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46bffff640 1 -- 172.21.15.12:0/1770365814 learned_addr learned my addr 172.21.15.12:0/1770365814 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:59.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46bffff640 10 monclient(hunting): get_auth_request con 0x7f46c8134ad0 auth_method 0 2024-09-11T01:54:59.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:59.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46bffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:59.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:59.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46c77fe640 1 --2- 172.21.15.12:0/1770365814 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46c81355c0 0x7f46c8134290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:59.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:59.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:59.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:59.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46c77fe640 10 monclient(hunting): get_auth_request con 0x7f46c81355c0 auth_method 0 2024-09-11T01:54:59.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:59.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:59.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:59.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.752+0000 7f46c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:59.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.753+0000 7f46c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:59.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.753+0000 7f46c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:59.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.753+0000 7f46bffff640 10 monclient(hunting): handle_auth_done global_id 284171 payload 824 2024-09-11T01:54:59.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.753+0000 7f46bffff640 1 -- 172.21.15.12:0/1770365814 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46c81355c0 msgr2=0x7f46c8134290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:59.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.753+0000 7f46bffff640 1 --2- 172.21.15.12:0/1770365814 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46c81355c0 0x7f46c8134290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:59.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.753+0000 7f46bffff640 1 -- 172.21.15.12:0/1770365814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46c80a2200 msgr2=0x7f46c81cd1f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:54:59.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.753+0000 7f46bffff640 1 --2- 172.21.15.12:0/1770365814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46c80a2200 0x7f46c81cd1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:59.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.753+0000 7f46bffff640 10 monclient: _finish_hunting 0 2024-09-11T01:54:59.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.753+0000 7f46bffff640 1 monclient: found mon.noname-c 2024-09-11T01:54:59.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.753+0000 7f46bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:59.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.753+0000 7f46bffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:54:59.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.753+0000 7f46bffff640 1 -- 172.21.15.12:0/1770365814 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46c8127db0 con 0x7f46c8134ad0 2024-09-11T01:54:59.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46bffff640 1 -- 172.21.15.12:0/1770365814 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c8134ad0 msgr2=0x7f46c8133d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:59.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46bffff640 1 --2- 172.21.15.12:0/1770365814 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c8134ad0 0x7f46c8133d50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f46b000c400 tx=0x7f46b000c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:59.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46bf7fe640 1 -- 172.21.15.12:0/1770365814 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b00119a0 con 0x7f46c8134ad0 2024-09-11T01:54:59.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:59.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46bf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:54:59.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46bf7fe640 10 monclient: dump: 2024-09-11T01:54:59.770 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:59.770 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:59.770 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.770 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.770 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:59.770 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:59.770 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:59.770 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:59.770 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:59.770 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:59.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46bf7fe640 10 monclient: _finish_auth 0 2024-09-11T01:54:59.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46bf7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:54:59.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:59.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46bf7fe640 1 -- 172.21.15.12:0/1770365814 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46b0017070 con 0x7f46c8134ad0 2024-09-11T01:54:59.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46bf7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:59.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46bf7fe640 1 -- 172.21.15.12:0/1770365814 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b0011dc0 con 0x7f46c8134ad0 2024-09-11T01:54:59.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46cd42a640 5 monclient: authenticate success, global_id 284171 2024-09-11T01:54:59.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:59.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46bf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:59.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46bf7fe640 10 monclient: dump: 2024-09-11T01:54:59.771 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:59.771 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:59.771 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.771 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.771 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:59.771 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:59.772 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:59.772 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:59.772 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:59.772 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:59.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46cd42a640 10 monclient: _renew_subs 2024-09-11T01:54:59.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46cd42a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:59.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46c5ffb640 4 set_mon_vals no callback set 2024-09-11T01:54:59.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46cd42a640 1 -- 172.21.15.12:0/1770365814 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46c80a49d0 con 0x7f46c8134ad0 2024-09-11T01:54:59.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46cd42a640 10 monclient: _renew_subs 2024-09-11T01:54:59.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46cd42a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:59.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46cd42a640 1 -- 172.21.15.12:0/1770365814 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f46c8128650 con 0x7f46c8134ad0 2024-09-11T01:54:59.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46cd42a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:59.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46cd42a640 1 --2- 172.21.15.12:0/1770365814 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c8124870 0x7f46c8124c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:59.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46cd42a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:59.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46bf7fe640 1 -- 172.21.15.12:0/1770365814 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f46b001c460 con 0x7f46c8134ad0 2024-09-11T01:54:59.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46cd42a640 1 -- 172.21.15.12:0/1770365814 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f46c8134eb0 con 0x7f46c8124870 2024-09-11T01:54:59.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46c6ffd640 1 --2- 172.21.15.12:0/1770365814 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c8124870 0x7f46c8124c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:59.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46c6ffd640 10 monclient: get_auth_request con 0x7f46c8124870 auth_method 0 2024-09-11T01:54:59.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:59.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:59.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:59.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:59.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:59.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:59.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46bf7fe640 1 --2- 172.21.15.12:0/1770365814 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f46a803e270 0x7f46a8040730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:59.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46c6ffd640 10 monclient(hunting): handle_auth_done global_id 284174 payload 824 2024-09-11T01:54:59.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46bf7fe640 1 -- 172.21.15.12:0/1770365814 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(860..860 src has 676..860) ==== 4815+0+0 (secure 0 0 0) 0x7f46b0052b40 con 0x7f46c8134ad0 2024-09-11T01:54:59.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46c6ffd640 1 -- 172.21.15.12:0/1770365814 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f46c8134890 con 0x7f46c8124870 2024-09-11T01:54:59.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46c77fe640 1 --2- 172.21.15.12:0/1770365814 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f46a803e270 0x7f46a8040730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:59.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.756+0000 7f46c77fe640 10 monclient: get_auth_request con 0x7f46a803e270 auth_method 0 2024-09-11T01:54:59.775 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:54:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.759+0000 7f46c6ffd640 1 -- 172.21.15.12:0/1770365814 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c8124870 msgr2=0x7f46c8124c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.759+0000 7f46c6ffd640 1 --2- 172.21.15.12:0/1770365814 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c8124870 0x7f46c8124c50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f46b800b450 tx=0x7f46b800b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.759+0000 7f46bf7fe640 1 -- 172.21.15.12:0/1770365814 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b8005270 con 0x7f46c8124870 2024-09-11T01:54:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.759+0000 7f46bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.759+0000 7f46bf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:59.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.759+0000 7f46bf7fe640 10 monclient: dump: 2024-09-11T01:54:59.778 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:59.778 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:59.778 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.778 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.778 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:59.778 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:59.778 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:59.778 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:59.778 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:59.778 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:59.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.759+0000 7f46bf7fe640 1 -- 172.21.15.12:0/1770365814 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f46b8015070 con 0x7f46c8124870 2024-09-11T01:54:59.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.759+0000 7f46bf7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:59.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.759+0000 7f46bf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:54:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.759+0000 7f46bf7fe640 1 -- 172.21.15.12:0/1770365814 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c8124870 msgr2=0x7f46c8124c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.759+0000 7f46bf7fe640 1 --2- 172.21.15.12:0/1770365814 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c8124870 0x7f46c8124c50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f46b800b450 tx=0x7f46b800b910 comp rx=0 tx=0).stop 2024-09-11T01:54:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.759+0000 7f46bf7fe640 1 -- 172.21.15.12:0/1770365814 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c8124870 msgr2=0x7f46c8124c50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:54:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.759+0000 7f46bf7fe640 1 --2- 172.21.15.12:0/1770365814 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c8124870 0x7f46c8124c50 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f46b800b450 tx=0x7f46b800b910 comp rx=0 tx=0).stop 2024-09-11T01:54:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.759+0000 7f46c77fe640 1 --2- 172.21.15.12:0/1770365814 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f46a803e270 0x7f46a8040730 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f46c80a72b0 tx=0x7f46c0009000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.766+0000 7f46cd42a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:54:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.766+0000 7f46cd42a640 1 --2- 172.21.15.12:0/1770365814 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c8125190 0x7f46c811c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.766+0000 7f46cd42a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.766+0000 7f46cd42a640 1 -- 172.21.15.12:0/1770365814 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f46c8126e50 con 0x7f46c8125190 2024-09-11T01:54:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.766+0000 7f46c6ffd640 1 --2- 172.21.15.12:0/1770365814 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c8125190 0x7f46c811c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.766+0000 7f46c6ffd640 10 monclient: get_auth_request con 0x7f46c8125190 auth_method 0 2024-09-11T01:54:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.766+0000 7f46c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.766+0000 7f46c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:59.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.766+0000 7f46c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.766+0000 7f46c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.766+0000 7f46c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.766+0000 7f46c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.766+0000 7f46c6ffd640 10 monclient(hunting): handle_auth_done global_id 284177 payload 824 2024-09-11T01:54:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.766+0000 7f46c6ffd640 1 -- 172.21.15.12:0/1770365814 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f46c806d480 con 0x7f46c8125190 2024-09-11T01:54:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.769+0000 7f46c6ffd640 1 -- 172.21.15.12:0/1770365814 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c8125190 msgr2=0x7f46c811c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.769+0000 7f46c6ffd640 1 --2- 172.21.15.12:0/1770365814 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c8125190 0x7f46c811c8e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f46b800b450 tx=0x7f46b80059d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.769+0000 7f46bf7fe640 1 -- 172.21.15.12:0/1770365814 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b8023410 con 0x7f46c8125190 2024-09-11T01:54:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.769+0000 7f46bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.769+0000 7f46bf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.769+0000 7f46bf7fe640 10 monclient: dump: 2024-09-11T01:54:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:59.781 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.769+0000 7f46bf7fe640 1 -- 172.21.15.12:0/1770365814 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4697 (secure 0 0 0) 0x7f46b8023410 con 0x7f46c8125190 2024-09-11T01:54:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.769+0000 7f46bf7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:54:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.769+0000 7f46bf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:54:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.769+0000 7f46bf7fe640 1 -- 172.21.15.12:0/1770365814 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c8125190 msgr2=0x7f46c811c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:59.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.769+0000 7f46bf7fe640 1 --2- 172.21.15.12:0/1770365814 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c8125190 0x7f46c811c8e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f46b800b450 tx=0x7f46b80059d0 comp rx=0 tx=0).stop 2024-09-11T01:54:59.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.769+0000 7f46bf7fe640 1 -- 172.21.15.12:0/1770365814 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c8125190 msgr2=0x7f46c811c8e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:54:59.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.769+0000 7f46bf7fe640 1 --2- 172.21.15.12:0/1770365814 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c8125190 0x7f46c811c8e0 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f46b800b450 tx=0x7f46b80059d0 comp rx=0 tx=0).stop 2024-09-11T01:54:59.782 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:54:59.782 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:54:59.782 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:54:59.782 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:54:59.782 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 236, 2024-09-11T01:54:59.782 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 38965, 2024-09-11T01:54:59.782 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:54:59.782 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:54:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:54:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:54:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:54:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:54:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:54:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:59.783 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:59.784 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:59.784 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:59.784 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:59.784 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:59.784 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:59.784 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:54:59.784 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:54:59.784 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:59.784 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:59.784 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:59.784 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:59.784 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:59.784 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:54:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:54:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:54:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:54:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:54:59.785 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:54:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:54:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:54:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:54:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:54:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:54:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:54:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:54:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:54:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:54:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:54:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:54:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:59.786 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:59.787 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:59.787 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:59.787 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:59.787 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:59.787 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:59.787 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:59.787 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:59.787 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:59.787 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:59.787 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:54:59.787 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:59.787 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:54:59.787 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:59.787 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:54:59.787 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:54:59.788 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:59.788 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:59.788 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:59.788 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:59.788 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:54:59.788 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:59.788 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:59.788 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:59.788 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:59.788 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:54:59.788 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:59.788 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:59.789 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:59.789 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:59.789 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:54:59.789 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:54:59.789 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:59.789 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:59.789 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:59.789 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:59.789 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:59.789 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:54:59.789 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:54:59.789 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:59.789 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:59.789 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:59.790 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:59.790 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:54:59.790 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:59.790 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:59.790 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:59.790 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:59.790 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:54:59.790 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:59.790 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:59.790 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:59.790 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:59.790 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:54:59.790 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:54:59.790 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:59.791 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:59.791 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:59.791 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:59.791 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:59.791 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:54:59.791 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:54:59.791 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:59.791 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:59.791 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:59.791 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:59.791 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:54:59.791 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:59.791 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:59.791 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:59.792 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:59.792 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:54:59.792 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:59.792 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:59.792 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:59.792 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:59.792 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:54:59.792 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:54:59.792 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:59.792 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:59.792 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:59.792 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:59.792 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:59.793 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:59.793 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:54:59.793 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:54:59.793 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:59.793 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:59.793 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:59.793 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:54:59.793 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:59.793 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:59.793 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:54:59.793 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:59.793 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:59.793 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:59.793 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:54:59.794 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:59.794 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:59.794 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:54:59.794 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:59.794 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:59.794 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:59.794 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:54:59.794 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:59.794 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:59.794 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:59.794 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:54:59.794 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:54:59.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.769+0000 7f46cd42a640 1 -- 172.21.15.12:0/1770365814 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f46a803e270 msgr2=0x7f46a8040730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:59.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.769+0000 7f46cd42a640 1 --2- 172.21.15.12:0/1770365814 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f46a803e270 0x7f46a8040730 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f46c80a72b0 tx=0x7f46c0009000 comp rx=0 tx=0).stop 2024-09-11T01:54:59.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.769+0000 7f46cd42a640 10 monclient: shutdown 2024-09-11T01:54:59.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.769+0000 7f46cd42a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:59.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.769+0000 7f46cd42a640 1 -- 172.21.15.12:0/1770365814 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c8134ad0 msgr2=0x7f46c8133d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:59.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.769+0000 7f46cd42a640 1 --2- 172.21.15.12:0/1770365814 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f46c8134ad0 0x7f46c8133d50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f46b000c400 tx=0x7f46b000c8c0 comp rx=0 tx=0).stop 2024-09-11T01:54:59.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.769+0000 7f46c77fe640 1 -- 172.21.15.12:0/1770365814 reap_dead start 2024-09-11T01:54:59.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.769+0000 7f46c77fe640 1 -- 172.21.15.12:0/1770365814 reap_dead start 2024-09-11T01:54:59.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.769+0000 7f46cd42a640 1 -- 172.21.15.12:0/1770365814 shutdown_connections 2024-09-11T01:54:59.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.769+0000 7f46cd42a640 1 -- 172.21.15.12:0/1770365814 >> 172.21.15.12:0/1770365814 conn(0x7f46c80a2f70 msgr2=0x7f46c8073ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:59.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.769+0000 7f46cd42a640 1 -- 172.21.15.12:0/1770365814 shutdown_connections 2024-09-11T01:54:59.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.769+0000 7f46cd42a640 1 -- 172.21.15.12:0/1770365814 wait complete. 2024-09-11T01:54:59.797 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:54:59.797 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T01:54:59.797 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:54:59.831 INFO:tasks.rados.rados.0.smithi071.stdout:3766: finishing write tid 3 to smithi07134777-16 2024-09-11T01:54:59.831 INFO:tasks.rados.rados.0.smithi071.stdout:3766: finishing write tid 4 to smithi07134777-16 2024-09-11T01:54:59.831 INFO:tasks.rados.rados.0.smithi071.stdout:3766: finishing write tid 5 to smithi07134777-16 2024-09-11T01:54:59.831 INFO:tasks.rados.rados.0.smithi071.stdout:3766: oid 16 updating version 0 to 2439 2024-09-11T01:54:59.831 INFO:tasks.rados.rados.0.smithi071.stdout:3766: oid 16 version 2439 is already newer than 2436 2024-09-11T01:54:59.831 INFO:tasks.rados.rados.0.smithi071.stdout:3766: oid 16 version 2439 is already newer than 2437 2024-09-11T01:54:59.831 INFO:tasks.rados.rados.0.smithi071.stdout:3766: oid 16 version 2439 is already newer than 2438 2024-09-11T01:54:59.831 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 2439 (ObjNum 1290 snap 424 seq_num 1290) dirty exists 2024-09-11T01:54:59.831 INFO:tasks.rados.rados.0.smithi071.stdout:3766: left oid 16 (ObjNum 1290 snap 424 seq_num 1290) 2024-09-11T01:54:59.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.898+0000 7f695dc16640 10 monclient: get_monmap_and_config 2024-09-11T01:54:59.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.898+0000 7f695dc16640 10 monclient: build_initial_monmap 2024-09-11T01:54:59.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.898+0000 7f695dc16640 10 monclient: monmap: 2024-09-11T01:54:59.900 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:59.900 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:59.900 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:54:59.899118+0000 2024-09-11T01:54:59.900 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:54:59.899118+0000 2024-09-11T01:54:59.900 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:59.900 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:59.900 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:54:59.900 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:54:59.901 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:54:59.901 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:54:59.901 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:54:59.901 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:59.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.898+0000 7f695dc16640 1 Processor -- start 2024-09-11T01:54:59.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.898+0000 7f695dc16640 1 -- start start 2024-09-11T01:54:59.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.898+0000 7f695dc16640 10 monclient: init 2024-09-11T01:54:59.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.898+0000 7f695dc16640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:59.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.898+0000 7f695dc16640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-11T01:54:59.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.898+0000 7f695dc16640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f69581381c0 0x7f69581385a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:59.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.898+0000 7f695dc16640 10 monclient(hunting): picked mon.noname-a con 0x7f69581381c0 addr v2:172.21.15.12:3300/0 2024-09-11T01:54:59.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.898+0000 7f695dc16640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f69580a6aa0 0x7f695809cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:59.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.898+0000 7f695dc16640 10 monclient(hunting): picked mon.noname-c con 0x7f69580a6aa0 addr v2:172.21.15.71:3300/0 2024-09-11T01:54:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.898+0000 7f695dc16640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f69580a4f20 addr v1:172.21.15.12:6789/0 2024-09-11T01:54:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.898+0000 7f695dc16640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f695809c310 con 0x7f69580a4f20 2024-09-11T01:54:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.898+0000 7f695dc16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.898+0000 7f695dc16640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f695806fc90 con 0x7f69581381c0 2024-09-11T01:54:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.898+0000 7f695dc16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.898+0000 7f695dc16640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f695809c5f0 con 0x7f69580a6aa0 2024-09-11T01:54:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.898+0000 7f695dc16640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.898+0000 7f695dc16640 10 monclient(hunting): authenticate will time out at 2306.341815s 2024-09-11T01:54:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.898+0000 7f69577fe640 1 -- >> v2:172.21.15.12:3300/0 conn(0x7f69581381c0 msgr2=0x7f69581385a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:54:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.898+0000 7f6957fff640 1 -- >> v1:172.21.15.12:6789/0 conn(0x7f69580a4f20 legacy=0x7f69580a5350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.12:6789/0 2024-09-11T01:54:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.898+0000 7f69577fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f69581381c0 0x7f69581385a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:54:59.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.899+0000 7f6956ffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f69580a6aa0 0x7f695809cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.899+0000 7f6956ffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f69580a6aa0 0x7f695809cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:58846/0 (socket says 172.21.15.12:58846) 2024-09-11T01:54:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.899+0000 7f6956ffd640 1 -- 172.21.15.12:0/3450310033 learned_addr learned my addr 172.21.15.12:0/3450310033 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.899+0000 7f6956ffd640 10 monclient(hunting): get_auth_request con 0x7f69580a6aa0 auth_method 0 2024-09-11T01:54:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.899+0000 7f6956ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.899+0000 7f6956ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.899+0000 7f6956ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.899+0000 7f6956ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.899+0000 7f6956ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.899+0000 7f6956ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.899+0000 7f6956ffd640 10 monclient(hunting): handle_auth_done global_id 274237 payload 274 2024-09-11T01:54:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.899+0000 7f6956ffd640 1 -- 172.21.15.12:0/3450310033 >> v2:172.21.15.12:3300/0 conn(0x7f69581381c0 msgr2=0x7f69581385a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:54:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.899+0000 7f6956ffd640 1 --2- 172.21.15.12:0/3450310033 >> v2:172.21.15.12:3300/0 conn(0x7f69581381c0 0x7f69581385a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:59.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.899+0000 7f6956ffd640 1 -- 172.21.15.12:0/3450310033 >> v1:172.21.15.12:6789/0 conn(0x7f69580a4f20 legacy=0x7f69580a5350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:54:59.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.899+0000 7f6956ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:54:59.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.899+0000 7f6956ffd640 1 monclient: found mon.noname-c 2024-09-11T01:54:59.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.899+0000 7f6956ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:59.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.899+0000 7f6956ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:54:59.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.899+0000 7f6956ffd640 1 -- 172.21.15.12:0/3450310033 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f695809c8d0 con 0x7f69580a6aa0 2024-09-11T01:54:59.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.902+0000 7f6956ffd640 1 -- 172.21.15.12:0/3450310033 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69580a6aa0 msgr2=0x7f695809cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:59.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.902+0000 7f6956ffd640 1 --2- 172.21.15.12:0/3450310033 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69580a6aa0 0x7f695809cfd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f69580a4be0 tx=0x7f694c02a0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3992fb4edb16d031 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:59.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.903+0000 7f69567fc640 1 -- 172.21.15.12:0/3450310033 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f694c037070 con 0x7f69580a6aa0 2024-09-11T01:54:59.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.903+0000 7f69567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:59.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.903+0000 7f69567fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:54:59.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.903+0000 7f69567fc640 10 monclient: dump: 2024-09-11T01:54:59.910 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:59.910 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:59.910 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.910 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.910 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:59.911 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:59.911 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:59.911 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:59.911 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:59.911 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:59.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.903+0000 7f69567fc640 10 monclient: _finish_auth 0 2024-09-11T01:54:59.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.903+0000 7f69567fc640 10 monclient: _check_auth_tickets 2024-09-11T01:54:59.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.903+0000 7f69567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:59.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.903+0000 7f69567fc640 1 -- 172.21.15.12:0/3450310033 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f694c032640 con 0x7f69580a6aa0 2024-09-11T01:54:59.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.903+0000 7f69567fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:59.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.903+0000 7f69567fc640 1 -- 172.21.15.12:0/3450310033 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f694c032a10 con 0x7f69580a6aa0 2024-09-11T01:54:59.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.903+0000 7f695dc16640 5 monclient: authenticate success, global_id 274237 2024-09-11T01:54:59.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.903+0000 7f695dc16640 10 monclient: get_monmap_and_config success 2024-09-11T01:54:59.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.903+0000 7f695dc16640 4 set_mon_vals no callback set 2024-09-11T01:54:59.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.903+0000 7f69567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:59.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.903+0000 7f69567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:54:59.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.903+0000 7f69567fc640 10 monclient: dump: 2024-09-11T01:54:59.912 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:59.912 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:59.912 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.912 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.912 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:59.912 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:59.912 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:59.913 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:59.913 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:59.913 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:59.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.903+0000 7f695dc16640 10 monclient: shutdown 2024-09-11T01:54:59.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.903+0000 7f695dc16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:59.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.903+0000 7f695dc16640 1 -- 172.21.15.12:0/3450310033 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69580a6aa0 msgr2=0x7f695809cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:59.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.903+0000 7f695dc16640 1 --2- 172.21.15.12:0/3450310033 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69580a6aa0 0x7f695809cfd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f69580a4be0 tx=0x7f694c02a0d0 comp rx=0 tx=0).stop 2024-09-11T01:54:59.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.903+0000 7f695dc16640 1 -- 172.21.15.12:0/3450310033 shutdown_connections 2024-09-11T01:54:59.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.903+0000 7f695dc16640 1 --2- 172.21.15.12:0/3450310033 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69580a6aa0 0x7f695809cfd0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:59.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.903+0000 7f695dc16640 1 --2- 172.21.15.12:0/3450310033 >> v2:172.21.15.12:3300/0 conn(0x7f69581381c0 0x7f69581385a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:59.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.903+0000 7f695dc16640 1 -- 172.21.15.12:0/3450310033 >> 172.21.15.12:0/3450310033 conn(0x7f69580a7440 msgr2=0x7f69580a7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:59.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.903+0000 7f695dc16640 1 -- 172.21.15.12:0/3450310033 shutdown_connections 2024-09-11T01:54:59.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.903+0000 7f695dc16640 1 -- 172.21.15.12:0/3450310033 wait complete. 2024-09-11T01:54:59.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.904+0000 7f695dc16640 10 monclient: build_initial_monmap 2024-09-11T01:54:59.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.904+0000 7f695dc16640 10 monclient: monmap: 2024-09-11T01:54:59.914 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:54:59.914 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:54:59.914 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:54:59.914 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:54:59.914 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:54:59.914 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:59.914 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:54:59.915 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:54:59.915 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:54:59.915 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:59.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.904+0000 7f695dc16640 1 Processor -- start 2024-09-11T01:54:59.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f695dc16640 1 -- start start 2024-09-11T01:54:59.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f695dc16640 10 monclient: init 2024-09-11T01:54:59.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f695dc16640 10 monclient: _reopen_session rank -1 2024-09-11T01:54:59.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f695dc16640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:54:59.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f695dc16640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69580a4f20 0x7f6958132f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:59.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f695dc16640 10 monclient(hunting): picked mon.noname-b con 0x7f69580a4f20 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:54:59.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f695dc16640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69580a6aa0 0x7f6958133440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:59.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f695dc16640 10 monclient(hunting): picked mon.noname-a con 0x7f69580a6aa0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:54:59.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f695dc16640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f69581381c0 0x7f6958138dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:59.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f695dc16640 10 monclient(hunting): picked mon.noname-c con 0x7f69581381c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:54:59.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f695dc16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:59.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f695dc16640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f69580a5a60 con 0x7f69580a6aa0 2024-09-11T01:54:59.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f695dc16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:59.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f695dc16640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6958138bf0 con 0x7f69580a4f20 2024-09-11T01:54:59.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f695dc16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:59.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f695dc16640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f69580a5be0 con 0x7f69581381c0 2024-09-11T01:54:59.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f695dc16640 10 monclient(hunting): _renew_subs 2024-09-11T01:54:59.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f695dc16640 10 monclient(hunting): authenticate will time out at 2306.348583s 2024-09-11T01:54:59.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f6956ffd640 1 -- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69580a6aa0 msgr2=0x7f6958133440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3300/0 2024-09-11T01:54:59.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f6956ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69580a6aa0 0x7f6958133440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:54:59.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f6957fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f69581381c0 0x7f6958138dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:59.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f6957fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f69581381c0 0x7f6958138dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:59974/0 (socket says 172.21.15.12:59974) 2024-09-11T01:54:59.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f6957fff640 1 -- 172.21.15.12:0/242403992 learned_addr learned my addr 172.21.15.12:0/242403992 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:54:59.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f6957fff640 10 monclient(hunting): get_auth_request con 0x7f69581381c0 auth_method 0 2024-09-11T01:54:59.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f6957fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:59.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f6957fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:59.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f6957fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:59.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f6957fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:59.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f6957fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:59.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f6957fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:59.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f69577fe640 1 --2- 172.21.15.12:0/242403992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69580a4f20 0x7f6958132f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:59.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f69577fe640 10 monclient(hunting): get_auth_request con 0x7f69580a4f20 auth_method 0 2024-09-11T01:54:59.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f69577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:59.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f69577fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:59.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.905+0000 7f69577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:59.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.906+0000 7f6957fff640 10 monclient(hunting): handle_auth_done global_id 284180 payload 824 2024-09-11T01:54:59.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.906+0000 7f6957fff640 1 -- 172.21.15.12:0/242403992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69580a4f20 msgr2=0x7f6958132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:59.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.906+0000 7f6957fff640 1 --2- 172.21.15.12:0/242403992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69580a4f20 0x7f6958132f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:59.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.906+0000 7f6957fff640 1 -- 172.21.15.12:0/242403992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69580a6aa0 msgr2=0x7f6958133440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:54:59.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.906+0000 7f6957fff640 1 --2- 172.21.15.12:0/242403992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f69580a6aa0 0x7f6958133440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:59.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.906+0000 7f6957fff640 10 monclient: _finish_hunting 0 2024-09-11T01:54:59.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.906+0000 7f6957fff640 1 monclient: found mon.noname-c 2024-09-11T01:54:59.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.906+0000 7f6957fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:54:59.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.906+0000 7f6957fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:54:59.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.906+0000 7f6957fff640 1 -- 172.21.15.12:0/242403992 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f695813a6a0 con 0x7f69581381c0 2024-09-11T01:54:59.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.908+0000 7f6957fff640 1 -- 172.21.15.12:0/242403992 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f69581381c0 msgr2=0x7f6958138dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:59.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.908+0000 7f6957fff640 1 --2- 172.21.15.12:0/242403992 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f69581381c0 0x7f6958138dd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f695000be60 tx=0x7f695000bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:59.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.908+0000 7f693ffff640 1 -- 172.21.15.12:0/242403992 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6950016630 con 0x7f69581381c0 2024-09-11T01:54:59.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.908+0000 7f693ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:59.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.908+0000 7f693ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:54:59.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.908+0000 7f693ffff640 10 monclient: dump: 2024-09-11T01:54:59.923 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:59.923 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:59.923 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.923 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.923 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:59.923 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:59.923 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:59.923 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:59.923 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:59.923 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:59.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.908+0000 7f693ffff640 10 monclient: _finish_auth 0 2024-09-11T01:54:59.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.908+0000 7f693ffff640 10 monclient: _check_auth_tickets 2024-09-11T01:54:59.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.908+0000 7f693ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:54:59.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.908+0000 7f693ffff640 1 -- 172.21.15.12:0/242403992 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6950010030 con 0x7f69581381c0 2024-09-11T01:54:59.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.908+0000 7f695dc16640 5 monclient: authenticate success, global_id 284180 2024-09-11T01:54:59.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.908+0000 7f695dc16640 10 monclient: _renew_subs 2024-09-11T01:54:59.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.908+0000 7f695dc16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:59.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.908+0000 7f695dc16640 1 -- 172.21.15.12:0/242403992 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f695813ca00 con 0x7f69581381c0 2024-09-11T01:54:59.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.908+0000 7f693ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:54:59.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.908+0000 7f693ffff640 1 -- 172.21.15.12:0/242403992 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6950016a10 con 0x7f69581381c0 2024-09-11T01:54:59.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.908+0000 7f693ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:59.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.908+0000 7f693ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:54:59.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.908+0000 7f693ffff640 10 monclient: dump: 2024-09-11T01:54:59.924 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:59.925 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:59.925 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.925 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.925 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:59.925 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:59.925 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:59.925 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:59.925 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:59.925 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:59.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.908+0000 7f6955ffb640 4 set_mon_vals no callback set 2024-09-11T01:54:59.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.908+0000 7f695dc16640 10 monclient: _renew_subs 2024-09-11T01:54:59.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.908+0000 7f695dc16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:54:59.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.908+0000 7f695dc16640 1 -- 172.21.15.12:0/242403992 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6958133980 con 0x7f69581381c0 2024-09-11T01:54:59.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.909+0000 7f695dc16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:59.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.909+0000 7f695dc16640 1 --2- 172.21.15.12:0/242403992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f695813b200 0x7f6958139310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:59.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.909+0000 7f695dc16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:59.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.909+0000 7f695dc16640 1 -- 172.21.15.12:0/242403992 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f69580a1710 con 0x7f695813b200 2024-09-11T01:54:59.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.909+0000 7f693ffff640 1 -- 172.21.15.12:0/242403992 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f6950008070 con 0x7f69581381c0 2024-09-11T01:54:59.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.909+0000 7f69577fe640 1 --2- 172.21.15.12:0/242403992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f695813b200 0x7f6958139310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:59.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.909+0000 7f69577fe640 10 monclient: get_auth_request con 0x7f695813b200 auth_method 0 2024-09-11T01:54:59.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.909+0000 7f693ffff640 1 --2- 172.21.15.12:0/242403992 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f694403e1c0 0x7f6944040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:59.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.909+0000 7f69577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:59.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.909+0000 7f69577fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:59.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.909+0000 7f69577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:59.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.909+0000 7f693ffff640 1 -- 172.21.15.12:0/242403992 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(860..860 src has 676..860) ==== 4815+0+0 (secure 0 0 0) 0x7f69500532d0 con 0x7f69581381c0 2024-09-11T01:54:59.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.909+0000 7f69577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:59.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.909+0000 7f69577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:59.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.909+0000 7f69577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:59.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.909+0000 7f6956ffd640 1 --2- 172.21.15.12:0/242403992 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f694403e1c0 0x7f6944040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:59.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.909+0000 7f6956ffd640 10 monclient: get_auth_request con 0x7f694403e1c0 auth_method 0 2024-09-11T01:54:59.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.909+0000 7f69577fe640 10 monclient(hunting): handle_auth_done global_id 274243 payload 824 2024-09-11T01:54:59.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.909+0000 7f69577fe640 1 -- 172.21.15.12:0/242403992 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f695809cfd0 con 0x7f695813b200 2024-09-11T01:54:59.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.912+0000 7f69577fe640 1 -- 172.21.15.12:0/242403992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f695813b200 msgr2=0x7f6958139310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:59.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.912+0000 7f69577fe640 1 --2- 172.21.15.12:0/242403992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f695813b200 0x7f6958139310 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f694800d700 tx=0x7f694800dbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:59.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.912+0000 7f6956ffd640 1 --2- 172.21.15.12:0/242403992 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f694403e1c0 0x7f6944040680 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f69580a4be0 tx=0x7f694c02e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:59.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.912+0000 7f693ffff640 1 -- 172.21.15.12:0/242403992 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6948003bc0 con 0x7f695813b200 2024-09-11T01:54:59.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.912+0000 7f693ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:59.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.912+0000 7f693ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:54:59.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.912+0000 7f693ffff640 10 monclient: dump: 2024-09-11T01:54:59.927 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:59.927 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:59.927 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.928 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.928 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:59.928 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:59.928 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:59.928 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:59.928 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:59.928 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:59.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.912+0000 7f693ffff640 1 -- 172.21.15.12:0/242403992 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6948003bc0 con 0x7f695813b200 2024-09-11T01:54:59.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.912+0000 7f693ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:54:59.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.912+0000 7f693ffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:54:59.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.912+0000 7f693ffff640 1 -- 172.21.15.12:0/242403992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f695813b200 msgr2=0x7f6958139310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:59.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.912+0000 7f693ffff640 1 --2- 172.21.15.12:0/242403992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f695813b200 0x7f6958139310 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f694800d700 tx=0x7f694800dbc0 comp rx=0 tx=0).stop 2024-09-11T01:54:59.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.912+0000 7f693ffff640 1 -- 172.21.15.12:0/242403992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f695813b200 msgr2=0x7f6958139310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:54:59.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.912+0000 7f693ffff640 1 --2- 172.21.15.12:0/242403992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f695813b200 0x7f6958139310 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f694800d700 tx=0x7f694800dbc0 comp rx=0 tx=0).stop 2024-09-11T01:54:59.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.918+0000 7f695dc16640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:54:59.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.918+0000 7f695dc16640 1 --2- 172.21.15.12:0/242403992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6958139850 0x7f6958139c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:54:59.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.918+0000 7f695dc16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:54:59.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.918+0000 7f695dc16640 1 -- 172.21.15.12:0/242403992 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f695813a170 con 0x7f6958139850 2024-09-11T01:54:59.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.919+0000 7f69577fe640 1 --2- 172.21.15.12:0/242403992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6958139850 0x7f6958139c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:54:59.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.919+0000 7f69577fe640 10 monclient: get_auth_request con 0x7f6958139850 auth_method 0 2024-09-11T01:54:59.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.919+0000 7f69577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:54:59.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.919+0000 7f69577fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:54:59.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.919+0000 7f69577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:54:59.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.919+0000 7f69577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:54:59.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.919+0000 7f69577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:54:59.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.919+0000 7f69577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:54:59.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.919+0000 7f69577fe640 10 monclient(hunting): handle_auth_done global_id 274246 payload 824 2024-09-11T01:54:59.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.919+0000 7f69577fe640 1 -- 172.21.15.12:0/242403992 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6958069840 con 0x7f6958139850 2024-09-11T01:54:59.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.921+0000 7f69577fe640 1 -- 172.21.15.12:0/242403992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6958139850 msgr2=0x7f6958139c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:54:59.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.921+0000 7f69577fe640 1 --2- 172.21.15.12:0/242403992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6958139850 0x7f6958139c30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f694800d700 tx=0x7f69480059d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:54:59.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.922+0000 7f693ffff640 1 -- 172.21.15.12:0/242403992 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f694800be10 con 0x7f6958139850 2024-09-11T01:54:59.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.922+0000 7f693ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:54:59.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.922+0000 7f693ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:54:59.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.922+0000 7f693ffff640 10 monclient: dump: 2024-09-11T01:54:59.932 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:54:59.932 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:54:59.932 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.932 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:54:59.932 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:54:59.932 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:54:59.932 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:54:59.932 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:54:59.932 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:54:59.932 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:54:59.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.922+0000 7f693ffff640 1 -- 172.21.15.12:0/242403992 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4866 (secure 0 0 0) 0x7f694800be10 con 0x7f6958139850 2024-09-11T01:54:59.932 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.922+0000 7f693ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:54:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.922+0000 7f693ffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:54:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.922+0000 7f693ffff640 1 -- 172.21.15.12:0/242403992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6958139850 msgr2=0x7f6958139c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.922+0000 7f693ffff640 1 --2- 172.21.15.12:0/242403992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6958139850 0x7f6958139c30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f694800d700 tx=0x7f69480059d0 comp rx=0 tx=0).stop 2024-09-11T01:54:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.922+0000 7f693ffff640 1 -- 172.21.15.12:0/242403992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6958139850 msgr2=0x7f6958139c30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:54:59.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.922+0000 7f693ffff640 1 --2- 172.21.15.12:0/242403992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6958139850 0x7f6958139c30 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f694800d700 tx=0x7f69480059d0 comp rx=0 tx=0).stop 2024-09-11T01:54:59.933 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:54:59.933 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:54:59.933 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:54:59.933 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:54:59.933 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 236, 2024-09-11T01:54:59.934 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 89742, 2024-09-11T01:54:59.934 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:54:59.934 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:54:59.934 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:54:59.934 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:59.934 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:54:59.934 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:54:59.934 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:54:59.934 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:54:59.934 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:59.934 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:59.934 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:59.935 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:59.935 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:59.935 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:59.935 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:59.935 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:59.935 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:59.935 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:59.935 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:59.935 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:59.935 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:54:59.935 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:54:59.935 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:59.935 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:59.935 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:59.936 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:59.936 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:59.936 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:59.936 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:59.936 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:59.936 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:59.936 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:59.936 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:59.936 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:59.936 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:59.936 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:54:59.936 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:54:59.936 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:54:59.937 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:54:59.937 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:54:59.937 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:54:59.937 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:54:59.937 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:54:59.937 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:54:59.937 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:54:59.937 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:54:59.937 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:54:59.937 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:54:59.937 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:54:59.937 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:54:59.937 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:54:59.938 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:54:59.938 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:54:59.938 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:54:59.938 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:54:59.938 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:54:59.938 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:54:59.938 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:54:59.938 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:54:59.938 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:54:59.938 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:54:59.938 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:54:59.938 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:54:59.938 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:59.938 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:54:59.938 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:59.939 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:54:59.939 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:59.939 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:54:59.939 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:54:59.939 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:59.939 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:59.939 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:59.939 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:59.939 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:54:59.939 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:59.939 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:59.939 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:59.939 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:59.939 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:54:59.940 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:59.940 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:59.940 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:59.940 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:59.940 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:54:59.940 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:54:59.940 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:59.940 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:59.940 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:59.940 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:59.940 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:59.940 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:54:59.940 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:54:59.940 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:59.941 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:59.941 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:59.941 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:59.941 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:54:59.941 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:59.941 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:59.941 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:59.941 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:59.941 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:54:59.941 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:59.941 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:59.941 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:59.941 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:59.941 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:54:59.942 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:54:59.942 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:59.942 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:59.942 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:59.942 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:59.942 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:59.942 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:54:59.942 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:54:59.942 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:54:59.942 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:54:59.942 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:59.942 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:54:59.942 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:54:59.942 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:59.943 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:59.943 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:59.943 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:54:59.943 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:54:59.943 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:54:59.943 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:59.943 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:59.943 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:59.943 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:54:59.943 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:54:59.943 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:54:59.943 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:54:59.943 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:54:59.943 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:59.943 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:59.944 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:59.944 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:54:59.944 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:54:59.944 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:59.944 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:59.944 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:59.944 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:54:59.944 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:59.944 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:59.944 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:54:59.944 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:59.944 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:59.944 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:59.944 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:54:59.944 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:59.945 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:59.945 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:54:59.945 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:59.945 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:59.945 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:59.945 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:54:59.945 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:59.945 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:54:59.945 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:54:59.945 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:54:59.945 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:54:59.945 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:54:59.945 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:54:59.945 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:54:59.945 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:54:59.946 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:54:59.946 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:54:59.946 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:54:59.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.923+0000 7f695dc16640 1 -- 172.21.15.12:0/242403992 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f694403e1c0 msgr2=0x7f6944040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:59.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.923+0000 7f695dc16640 1 --2- 172.21.15.12:0/242403992 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f694403e1c0 0x7f6944040680 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f69580a4be0 tx=0x7f694c02e000 comp rx=0 tx=0).stop 2024-09-11T01:54:59.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.923+0000 7f695dc16640 10 monclient: shutdown 2024-09-11T01:54:59.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.923+0000 7f695dc16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:54:59.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.923+0000 7f695dc16640 1 -- 172.21.15.12:0/242403992 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f69581381c0 msgr2=0x7f6958138dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:54:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.923+0000 7f6957fff640 1 -- 172.21.15.12:0/242403992 reap_dead start 2024-09-11T01:54:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.923+0000 7f695dc16640 1 --2- 172.21.15.12:0/242403992 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f69581381c0 0x7f6958138dd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f695000be60 tx=0x7f695000bf50 comp rx=0 tx=0).stop 2024-09-11T01:54:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.923+0000 7f695dc16640 1 -- 172.21.15.12:0/242403992 shutdown_connections 2024-09-11T01:54:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.923+0000 7f695dc16640 1 --2- 172.21.15.12:0/242403992 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f69581381c0 0x7f6958138dd0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:54:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.923+0000 7f695dc16640 1 -- 172.21.15.12:0/242403992 >> 172.21.15.12:0/242403992 conn(0x7f69580a7440 msgr2=0x7f695809ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:54:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.923+0000 7f695dc16640 1 -- 172.21.15.12:0/242403992 shutdown_connections 2024-09-11T01:54:59.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:54:59.923+0000 7f695dc16640 1 -- 172.21.15.12:0/242403992 wait complete. 2024-09-11T01:54:59.948 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:54:59.948 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:55:00.722 INFO:tasks.rados.rados.0.smithi071.stdout:3765: done (3 left) 2024-09-11T01:55:00.723 INFO:tasks.rados.rados.0.smithi071.stdout:3766: done (2 left) 2024-09-11T01:55:00.723 INFO:tasks.rados.rados.0.smithi071.stdout:3767: done (1 left) 2024-09-11T01:55:00.723 INFO:tasks.rados.rados.0.smithi071.stdout:3769: done (0 left) 2024-09-11T01:55:00.723 INFO:tasks.rados.rados.0.smithi071.stdout:3770: read oid 26 snap 420 2024-09-11T01:55:00.723 INFO:tasks.rados.rados.0.smithi071.stdout:3770: expect (ObjNum 1229 snap 401 seq_num 1229) 2024-09-11T01:55:00.723 INFO:tasks.rados.rados.0.smithi071.stdout:3771: rollback oid 24 current snap is 425 2024-09-11T01:55:00.723 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 24 to 404 2024-09-11T01:55:00.723 INFO:tasks.rados.rados.0.smithi071.stdout:3772: read oid 14 snap 417 2024-09-11T01:55:00.723 INFO:tasks.rados.rados.0.smithi071.stdout:3772: expect (ObjNum 1254 snap 415 seq_num 1254) 2024-09-11T01:55:00.723 INFO:tasks.rados.rados.0.smithi071.stdout:3773: read oid 11 snap 411 2024-09-11T01:55:00.723 INFO:tasks.rados.rados.0.smithi071.stdout:3773: expect (ObjNum 1115 snap 356 seq_num 1115) 2024-09-11T01:55:00.723 INFO:tasks.rados.rados.0.smithi071.stdout:3774: copy_from oid 43 from oid 44 current snap is 425 2024-09-11T01:55:00.724 INFO:tasks.rados.rados.0.smithi071.stdout:3775: snap_remove snap 416 2024-09-11T01:55:01.727 INFO:tasks.rados.rados.0.smithi071.stdout:3775: done (5 left) 2024-09-11T01:55:01.727 INFO:tasks.rados.rados.0.smithi071.stdout:3776: write oid 27 current snap is 425 2024-09-11T01:55:01.727 INFO:tasks.rados.rados.0.smithi071.stdout:3776: seq_num 1291 ranges {497837=410116,1596371=743454,2902966=763698} 2024-09-11T01:55:01.732 INFO:tasks.rados.rados.0.smithi071.stdout:3776: writing smithi07134777-27 from 497837 to 907953 tid 1 2024-09-11T01:55:01.740 INFO:tasks.rados.rados.0.smithi071.stdout:3776: writing smithi07134777-27 from 1596371 to 2339825 tid 2 2024-09-11T01:55:01.750 INFO:tasks.rados.rados.0.smithi071.stdout:3776: writing smithi07134777-27 from 2902966 to 3666664 tid 3 2024-09-11T01:55:01.750 INFO:tasks.rados.rados.0.smithi071.stdout:3777: delete oid 25 current snap is 425 2024-09-11T01:55:01.750 INFO:tasks.rados.rados.0.smithi071.stdout:3770: expect (ObjNum 1229 snap 401 seq_num 1229) 2024-09-11T01:55:01.798 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:55:01.828 INFO:tasks.rados.rados.0.smithi071.stdout:3771: finishing rollback tid 0 to smithi07134777-24 2024-09-11T01:55:01.828 INFO:tasks.rados.rados.0.smithi071.stdout:3773: expect (ObjNum 1115 snap 356 seq_num 1115) 2024-09-11T01:55:01.829 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 45553 2024-09-11T01:55:01.871 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:55:01.913 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:55:01.956 INFO:tasks.rados.rados.0.smithi071.stdout:3771: finishing rollback tid 1 to smithi07134777-24 2024-09-11T01:55:01.956 INFO:tasks.rados.rados.0.smithi071.stdout:3771: finishing rollback tid 2 to smithi07134777-24 2024-09-11T01:55:01.956 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 2449 (ObjNum 1187 snap 378 seq_num 1187) dirty exists 2024-09-11T01:55:01.956 INFO:tasks.rados.rados.0.smithi071.stdout:3772: expect (ObjNum 1254 snap 415 seq_num 1254) 2024-09-11T01:55:02.016 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:55:02.050 INFO:tasks.rados.rados.0.smithi071.stdout:3774: finishing copy_from racing read to smithi07134777-43 2024-09-11T01:55:02.050 INFO:tasks.rados.rados.0.smithi071.stdout:3774: finishing copy_from to smithi07134777-43 2024-09-11T01:55:02.050 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 1264 (ObjNum 1249 snap 412 seq_num 1249) dirty exists 2024-09-11T01:55:02.050 INFO:tasks.rados.rados.0.smithi071.stdout:3776: finishing write tid 1 to smithi07134777-27 2024-09-11T01:55:02.050 INFO:tasks.rados.rados.0.smithi071.stdout:3776: finishing write tid 2 to smithi07134777-27 2024-09-11T01:55:02.050 INFO:tasks.rados.rados.0.smithi071.stdout:3776: finishing write tid 3 to smithi07134777-27 2024-09-11T01:55:02.050 INFO:tasks.rados.rados.0.smithi071.stdout:3776: finishing write tid 4 to smithi07134777-27 2024-09-11T01:55:02.050 INFO:tasks.rados.rados.0.smithi071.stdout:3776: finishing write tid 5 to smithi07134777-27 2024-09-11T01:55:02.050 INFO:tasks.rados.rados.0.smithi071.stdout:3776: oid 27 updating version 0 to 2884 2024-09-11T01:55:02.050 INFO:tasks.rados.rados.0.smithi071.stdout:3776: oid 27 version 2884 is already newer than 2874 2024-09-11T01:55:02.051 INFO:tasks.rados.rados.0.smithi071.stdout:3776: oid 27 updating version 2884 to 2885 2024-09-11T01:55:02.051 INFO:tasks.rados.rados.0.smithi071.stdout:3776: oid 27 version 2885 is already newer than 2879 2024-09-11T01:55:02.051 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 2885 (ObjNum 1291 snap 425 seq_num 1291) dirty exists 2024-09-11T01:55:02.051 INFO:tasks.rados.rados.0.smithi071.stdout:3776: left oid 27 (ObjNum 1291 snap 425 seq_num 1291) 2024-09-11T01:55:02.051 INFO:tasks.rados.rados.0.smithi071.stdout:3770: done (6 left) 2024-09-11T01:55:02.051 INFO:tasks.rados.rados.0.smithi071.stdout:3771: done (5 left) 2024-09-11T01:55:02.051 INFO:tasks.rados.rados.0.smithi071.stdout:3772: done (4 left) 2024-09-11T01:55:02.051 INFO:tasks.rados.rados.0.smithi071.stdout:3773: done (3 left) 2024-09-11T01:55:02.051 INFO:tasks.rados.rados.0.smithi071.stdout:3774: done (2 left) 2024-09-11T01:55:02.051 INFO:tasks.rados.rados.0.smithi071.stdout:3776: done (1 left) 2024-09-11T01:55:02.051 INFO:tasks.rados.rados.0.smithi071.stdout:3777: done (0 left) 2024-09-11T01:55:02.051 INFO:tasks.rados.rados.0.smithi071.stdout:3778: copy_from oid 32 from oid 44 current snap is 425 2024-09-11T01:55:02.051 INFO:tasks.rados.rados.0.smithi071.stdout:3779: snap_create 2024-09-11T01:55:02.061 INFO:tasks.rados.rados.0.smithi071.stdout:3778: finishing copy_from racing read to smithi07134777-32 2024-09-11T01:55:02.062 INFO:tasks.rados.rados.0.smithi071.stdout:3778: finishing copy_from to smithi07134777-32 2024-09-11T01:55:02.062 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 1264 (ObjNum 1249 snap 412 seq_num 1249) dirty exists 2024-09-11T01:55:02.744 INFO:tasks.rados.rados.0.smithi071.stdout:3778: done (1 left) 2024-09-11T01:55:02.744 INFO:tasks.rados.rados.0.smithi071.stdout:3779: done (0 left) 2024-09-11T01:55:02.745 INFO:tasks.rados.rados.0.smithi071.stdout:3780: delete oid 15 current snap is 426 2024-09-11T01:55:02.749 INFO:tasks.rados.rados.0.smithi071.stdout:3780: done (0 left) 2024-09-11T01:55:02.749 INFO:tasks.rados.rados.0.smithi071.stdout:3781: write oid 37 current snap is 426 2024-09-11T01:55:02.749 INFO:tasks.rados.rados.0.smithi071.stdout:3781: seq_num 1294 ranges {548258=752216,1771817=490781,2985235=713867} 2024-09-11T01:55:02.756 INFO:tasks.rados.rados.0.smithi071.stdout:3781: writing smithi07134777-37 from 548258 to 1300474 tid 1 2024-09-11T01:55:02.761 INFO:tasks.rados.rados.0.smithi071.stdout:3781: writing smithi07134777-37 from 1771817 to 2262598 tid 2 2024-09-11T01:55:02.769 INFO:tasks.rados.rados.0.smithi071.stdout:3781: writing smithi07134777-37 from 2985235 to 3699102 tid 3 2024-09-11T01:55:02.769 INFO:tasks.rados.rados.0.smithi071.stdout:3782: read oid 26 snap -1 2024-09-11T01:55:02.769 INFO:tasks.rados.rados.0.smithi071.stdout:3782: expect deleted 2024-09-11T01:55:02.769 INFO:tasks.rados.rados.0.smithi071.stdout:3783: rollback oid 9 current snap is 426 2024-09-11T01:55:02.769 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 9 to 325 2024-09-11T01:55:02.769 INFO:tasks.rados.rados.0.smithi071.stdout:3781: finishing write tid 1 to smithi07134777-37 2024-09-11T01:55:02.769 INFO:tasks.rados.rados.0.smithi071.stdout:3784: write (excl) oid 34 current snap is 426 2024-09-11T01:55:02.769 INFO:tasks.rados.rados.0.smithi071.stdout:3784: seq_num 1295 ranges {565065=427465,1430299=634439,2496305=525139,3472104=292961} 2024-09-11T01:55:02.774 INFO:tasks.rados.rados.0.smithi071.stdout:3784: writing smithi07134777-34 from 565065 to 992530 tid 1 2024-09-11T01:55:02.780 INFO:tasks.rados.rados.0.smithi071.stdout:3784: writing smithi07134777-34 from 1430299 to 2064738 tid 2 2024-09-11T01:55:02.785 INFO:tasks.rados.rados.0.smithi071.stdout:3784: writing smithi07134777-34 from 2496305 to 3021444 tid 3 2024-09-11T01:55:02.789 INFO:tasks.rados.rados.0.smithi071.stdout:3784: writing smithi07134777-34 from 3472104 to 3765065 tid 4 2024-09-11T01:55:02.789 INFO:tasks.rados.rados.0.smithi071.stdout:3785: delete oid 14 current snap is 426 2024-09-11T01:55:02.790 INFO:tasks.rados.rados.0.smithi071.stdout:3781: finishing write tid 2 to smithi07134777-37 2024-09-11T01:55:02.790 INFO:tasks.rados.rados.0.smithi071.stdout:3781: finishing write tid 3 to smithi07134777-37 2024-09-11T01:55:02.790 INFO:tasks.rados.rados.0.smithi071.stdout:3781: finishing write tid 4 to smithi07134777-37 2024-09-11T01:55:02.790 INFO:tasks.rados.rados.0.smithi071.stdout:3781: finishing write tid 5 to smithi07134777-37 2024-09-11T01:55:02.790 INFO:tasks.rados.rados.0.smithi071.stdout:3781: oid 37 updating version 0 to 2116 2024-09-11T01:55:02.790 INFO:tasks.rados.rados.0.smithi071.stdout:3781: oid 37 version 2116 is already newer than 2115 2024-09-11T01:55:02.790 INFO:tasks.rados.rados.0.smithi071.stdout:3781: oid 37 version 2116 is already newer than 2114 2024-09-11T01:55:02.790 INFO:tasks.rados.rados.0.smithi071.stdout:3781: oid 37 updating version 2116 to 2117 2024-09-11T01:55:02.790 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 2117 (ObjNum 1294 snap 426 seq_num 1294) dirty exists 2024-09-11T01:55:02.790 INFO:tasks.rados.rados.0.smithi071.stdout:3781: left oid 37 (ObjNum 1294 snap 426 seq_num 1294) 2024-09-11T01:55:02.790 INFO:tasks.rados.rados.0.smithi071.stdout:3783: finishing rollback tid 0 to smithi07134777-9 2024-09-11T01:55:02.790 INFO:tasks.rados.rados.0.smithi071.stdout:3783: finishing rollback tid 1 to smithi07134777-9 2024-09-11T01:55:02.791 INFO:tasks.rados.rados.0.smithi071.stdout:3783: finishing rollback tid 2 to smithi07134777-9 2024-09-11T01:55:02.791 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 2580 (ObjNum 1002 snap 322 seq_num 1002) dirty exists 2024-09-11T01:55:02.791 INFO:tasks.rados.rados.0.smithi071.stdout:3784: finishing write tid 1 to smithi07134777-34 2024-09-11T01:55:02.791 INFO:tasks.rados.rados.0.smithi071.stdout:3784: finishing write tid 2 to smithi07134777-34 2024-09-11T01:55:02.791 INFO:tasks.rados.rados.0.smithi071.stdout:3784: finishing write tid 3 to smithi07134777-34 2024-09-11T01:55:02.793 INFO:tasks.rados.rados.0.smithi071.stdout:3781: done (4 left) 2024-09-11T01:55:02.793 INFO:tasks.rados.rados.0.smithi071.stdout:3782: done (3 left) 2024-09-11T01:55:02.793 INFO:tasks.rados.rados.0.smithi071.stdout:3783: done (2 left) 2024-09-11T01:55:02.793 INFO:tasks.rados.rados.0.smithi071.stdout:3785: done (1 left) 2024-09-11T01:55:02.794 INFO:tasks.rados.rados.0.smithi071.stdout:3786: read oid 38 snap -1 2024-09-11T01:55:02.794 INFO:tasks.rados.rados.0.smithi071.stdout:3786: expect deleted 2024-09-11T01:55:02.794 INFO:tasks.rados.rados.0.smithi071.stdout:3787: rollback oid 20 current snap is 426 2024-09-11T01:55:02.794 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 20 to 414 2024-09-11T01:55:02.794 INFO:tasks.rados.rados.0.smithi071.stdout:3788: write oid 31 current snap is 426 2024-09-11T01:55:02.794 INFO:tasks.rados.rados.0.smithi071.stdout:3788: seq_num 1297 ranges {598679=577963,1947263=638108,3351151=447528} 2024-09-11T01:55:02.799 INFO:tasks.rados.rados.0.smithi071.stdout:3788: writing smithi07134777-31 from 598679 to 1176642 tid 1 2024-09-11T01:55:02.807 INFO:tasks.rados.rados.0.smithi071.stdout:3788: writing smithi07134777-31 from 1947263 to 2585371 tid 2 2024-09-11T01:55:02.814 INFO:tasks.rados.rados.0.smithi071.stdout:3788: writing smithi07134777-31 from 3351151 to 3798679 tid 3 2024-09-11T01:55:02.814 INFO:tasks.rados.rados.0.smithi071.stdout:3789: delete oid 7 current snap is 426 2024-09-11T01:55:02.814 INFO:tasks.rados.rados.0.smithi071.stdout:3784: finishing write tid 4 to smithi07134777-34 2024-09-11T01:55:02.814 INFO:tasks.rados.rados.0.smithi071.stdout:3784: finishing write tid 5 to smithi07134777-34 2024-09-11T01:55:02.814 INFO:tasks.rados.rados.0.smithi071.stdout:3784: finishing write tid 6 to smithi07134777-34 2024-09-11T01:55:02.814 INFO:tasks.rados.rados.0.smithi071.stdout:3784: oid 34 updating version 0 to 2888 2024-09-11T01:55:02.814 INFO:tasks.rados.rados.0.smithi071.stdout:3784: oid 34 updating version 2888 to 2892 2024-09-11T01:55:02.814 INFO:tasks.rados.rados.0.smithi071.stdout:3784: oid 34 version 2892 is already newer than 2889 2024-09-11T01:55:02.815 INFO:tasks.rados.rados.0.smithi071.stdout:3784: oid 34 version 2892 is already newer than 2890 2024-09-11T01:55:02.815 INFO:tasks.rados.rados.0.smithi071.stdout:3784: oid 34 version 2892 is already newer than 2891 2024-09-11T01:55:02.815 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 2892 (ObjNum 1295 snap 426 seq_num 1295) dirty exists 2024-09-11T01:55:02.815 INFO:tasks.rados.rados.0.smithi071.stdout:3784: left oid 34 (ObjNum 1295 snap 426 seq_num 1295) 2024-09-11T01:55:02.815 INFO:tasks.rados.rados.0.smithi071.stdout:3787: finishing rollback tid 1 to smithi07134777-20 2024-09-11T01:55:02.815 INFO:tasks.rados.rados.0.smithi071.stdout:3787: finishing rollback tid 2 to smithi07134777-20 2024-09-11T01:55:02.815 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 1341 (ObjNum 1216 snap 397 seq_num 1216) dirty exists 2024-09-11T01:55:02.815 INFO:tasks.rados.rados.0.smithi071.stdout:3788: finishing write tid 1 to smithi07134777-31 2024-09-11T01:55:02.815 INFO:tasks.rados.rados.0.smithi071.stdout:3788: finishing write tid 2 to smithi07134777-31 2024-09-11T01:55:02.818 INFO:tasks.rados.rados.0.smithi071.stdout:3784: done (4 left) 2024-09-11T01:55:02.818 INFO:tasks.rados.rados.0.smithi071.stdout:3786: done (3 left) 2024-09-11T01:55:02.818 INFO:tasks.rados.rados.0.smithi071.stdout:3787: done (2 left) 2024-09-11T01:55:02.818 INFO:tasks.rados.rados.0.smithi071.stdout:3789: done (1 left) 2024-09-11T01:55:02.818 INFO:tasks.rados.rados.0.smithi071.stdout:3790: read oid 26 snap 421 2024-09-11T01:55:02.819 INFO:tasks.rados.rados.0.smithi071.stdout:3790: expect (ObjNum 1229 snap 401 seq_num 1229) 2024-09-11T01:55:02.819 INFO:tasks.rados.rados.0.smithi071.stdout:3791: write (excl) oid 16 current snap is 426 2024-09-11T01:55:02.819 INFO:tasks.rados.rados.0.smithi071.stdout:3791: seq_num 1299 ranges {632293=728461,1947874=641777,3005997=686580} 2024-09-11T01:55:02.825 INFO:tasks.rados.rados.0.smithi071.stdout:3791: writing smithi07134777-16 from 632293 to 1360754 tid 1 2024-09-11T01:55:02.833 INFO:tasks.rados.rados.0.smithi071.stdout:3791: writing smithi07134777-16 from 1947874 to 2589651 tid 2 2024-09-11T01:55:02.839 INFO:tasks.rados.rados.0.smithi071.stdout:3791: writing smithi07134777-16 from 3005997 to 3692577 tid 3 2024-09-11T01:55:02.839 INFO:tasks.rados.rados.0.smithi071.stdout:3792: write oid 3 current snap is 426 2024-09-11T01:55:02.839 INFO:tasks.rados.rados.0.smithi071.stdout:3788: finishing write tid 3 to smithi07134777-31 2024-09-11T01:55:02.840 INFO:tasks.rados.rados.0.smithi071.stdout:3792: seq_num 1300 ranges {649100=403710,1606356=501788,2633420=497852,3636005=213095} 2024-09-11T01:55:02.845 INFO:tasks.rados.rados.0.smithi071.stdout:3792: writing smithi07134777-3 from 649100 to 1052810 tid 1 2024-09-11T01:55:02.851 INFO:tasks.rados.rados.0.smithi071.stdout:3792: writing smithi07134777-3 from 1606356 to 2108144 tid 2 2024-09-11T01:55:02.857 INFO:tasks.rados.rados.0.smithi071.stdout:3792: writing smithi07134777-3 from 2633420 to 3131272 tid 3 2024-09-11T01:55:02.861 INFO:tasks.rados.rados.0.smithi071.stdout:3792: writing smithi07134777-3 from 3636005 to 3849100 tid 4 2024-09-11T01:55:02.861 INFO:tasks.rados.rados.0.smithi071.stdout:3793: snap_remove snap 418 2024-09-11T01:55:03.749 INFO:tasks.rados.rados.0.smithi071.stdout:3793: done (4 left) 2024-09-11T01:55:03.749 INFO:tasks.rados.rados.0.smithi071.stdout:3794: read oid 4 snap -1 2024-09-11T01:55:03.749 INFO:tasks.rados.rados.0.smithi071.stdout:3794: expect deleted 2024-09-11T01:55:03.749 INFO:tasks.rados.rados.0.smithi071.stdout:3795: snap_remove snap 381 2024-09-11T01:55:04.754 INFO:tasks.rados.rados.0.smithi071.stdout:3795: done (5 left) 2024-09-11T01:55:04.754 INFO:tasks.rados.rados.0.smithi071.stdout:3796: read oid 27 snap -1 2024-09-11T01:55:04.754 INFO:tasks.rados.rados.0.smithi071.stdout:3796: expect (ObjNum 1291 snap 425 seq_num 1291) 2024-09-11T01:55:04.754 INFO:tasks.rados.rados.0.smithi071.stdout:3797: snap_remove snap 400 2024-09-11T01:55:05.723 INFO:tasks.rados.rados.0.smithi071.stdout:3797: done (6 left) 2024-09-11T01:55:05.723 INFO:tasks.rados.rados.0.smithi071.stdout:3798: write (excl) oid 17 current snap is 426 2024-09-11T01:55:05.723 INFO:tasks.rados.rados.0.smithi071.stdout:3798: seq_num 1301 ranges {665907=595312,1781191=645446,3177196=688711} 2024-09-11T01:55:05.731 INFO:tasks.rados.rados.0.smithi071.stdout:3798: writing smithi07134777-17 from 665907 to 1261219 tid 1 2024-09-11T01:55:05.738 INFO:tasks.rados.rados.0.smithi071.stdout:3798: writing smithi07134777-17 from 1781191 to 2426637 tid 2 2024-09-11T01:55:05.747 INFO:tasks.rados.rados.0.smithi071.stdout:3798: writing smithi07134777-17 from 3177196 to 3865907 tid 3 2024-09-11T01:55:05.747 INFO:tasks.rados.rados.0.smithi071.stdout:3799: copy_from oid 41 from oid 48 current snap is 426 2024-09-11T01:55:05.747 INFO:tasks.rados.rados.0.smithi071.stdout:3788: finishing write tid 4 to smithi07134777-31 2024-09-11T01:55:05.747 INFO:tasks.rados.rados.0.smithi071.stdout:3788: finishing write tid 5 to smithi07134777-31 2024-09-11T01:55:05.747 INFO:tasks.rados.rados.0.smithi071.stdout:3788: oid 31 updating version 0 to 2574 2024-09-11T01:55:05.748 INFO:tasks.rados.rados.0.smithi071.stdout:3788: oid 31 updating version 2574 to 2576 2024-09-11T01:55:05.748 INFO:tasks.rados.rados.0.smithi071.stdout:3788: oid 31 version 2576 is already newer than 2575 2024-09-11T01:55:05.748 INFO:tasks.rados.rados.0.smithi071.stdout:3788: oid 31 version 2576 is already newer than 2573 2024-09-11T01:55:05.748 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 2576 (ObjNum 1297 snap 426 seq_num 1297) dirty exists 2024-09-11T01:55:05.748 INFO:tasks.rados.rados.0.smithi071.stdout:3788: left oid 31 (ObjNum 1297 snap 426 seq_num 1297) 2024-09-11T01:55:05.748 INFO:tasks.rados.rados.0.smithi071.stdout:3790: expect (ObjNum 1229 snap 401 seq_num 1229) 2024-09-11T01:55:05.828 INFO:tasks.rados.rados.0.smithi071.stdout:3791: finishing write tid 1 to smithi07134777-16 2024-09-11T01:55:05.828 INFO:tasks.rados.rados.0.smithi071.stdout:3791: finishing write tid 2 to smithi07134777-16 2024-09-11T01:55:05.828 INFO:tasks.rados.rados.0.smithi071.stdout:3788: done (7 left) 2024-09-11T01:55:05.828 INFO:tasks.rados.rados.0.smithi071.stdout:3790: done (6 left) 2024-09-11T01:55:05.828 INFO:tasks.rados.rados.0.smithi071.stdout:3800: read oid 20 snap -1 2024-09-11T01:55:05.828 INFO:tasks.rados.rados.0.smithi071.stdout:3800: expect (ObjNum 1216 snap 397 seq_num 1216) 2024-09-11T01:55:05.828 INFO:tasks.rados.rados.0.smithi071.stdout:3801: write (excl) oid 47 current snap is 426 2024-09-11T01:55:05.828 INFO:tasks.rados.rados.0.smithi071.stdout:3791: finishing write tid 3 to smithi07134777-16 2024-09-11T01:55:05.829 INFO:tasks.rados.rados.0.smithi071.stdout:3791: finishing write tid 4 to smithi07134777-16 2024-09-11T01:55:05.829 INFO:tasks.rados.rados.0.smithi071.stdout:3791: finishing write tid 5 to smithi07134777-16 2024-09-11T01:55:05.829 INFO:tasks.rados.rados.0.smithi071.stdout:3791: oid 16 updating version 0 to 2459 2024-09-11T01:55:05.829 INFO:tasks.rados.rados.0.smithi071.stdout:3791: oid 16 updating version 2459 to 2460 2024-09-11T01:55:05.829 INFO:tasks.rados.rados.0.smithi071.stdout:3791: oid 16 updating version 2460 to 2461 2024-09-11T01:55:05.829 INFO:tasks.rados.rados.0.smithi071.stdout:3791: oid 16 updating version 2461 to 2462 2024-09-11T01:55:05.829 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 2462 (ObjNum 1299 snap 426 seq_num 1299) dirty exists 2024-09-11T01:55:05.829 INFO:tasks.rados.rados.0.smithi071.stdout:3791: left oid 16 (ObjNum 1299 snap 426 seq_num 1299) 2024-09-11T01:55:05.829 INFO:tasks.rados.rados.0.smithi071.stdout:3792: finishing write tid 1 to smithi07134777-3 2024-09-11T01:55:05.829 INFO:tasks.rados.rados.0.smithi071.stdout:3792: finishing write tid 2 to smithi07134777-3 2024-09-11T01:55:05.829 INFO:tasks.rados.rados.0.smithi071.stdout:3792: finishing write tid 3 to smithi07134777-3 2024-09-11T01:55:05.829 INFO:tasks.rados.rados.0.smithi071.stdout:3792: finishing write tid 4 to smithi07134777-3 2024-09-11T01:55:05.829 INFO:tasks.rados.rados.0.smithi071.stdout:3792: finishing write tid 5 to smithi07134777-3 2024-09-11T01:55:05.830 INFO:tasks.rados.rados.0.smithi071.stdout:3801: seq_num 1302 ranges {682714=670561,2123320=505457,3088266=636749} 2024-09-11T01:55:05.835 INFO:tasks.rados.rados.0.smithi071.stdout:3801: writing smithi07134777-47 from 682714 to 1353275 tid 1 2024-09-11T01:55:05.842 INFO:tasks.rados.rados.0.smithi071.stdout:3801: writing smithi07134777-47 from 2123320 to 2628777 tid 2 2024-09-11T01:55:05.849 INFO:tasks.rados.rados.0.smithi071.stdout:3801: writing smithi07134777-47 from 3088266 to 3725015 tid 3 2024-09-11T01:55:05.849 INFO:tasks.rados.rados.0.smithi071.stdout:3791: done (7 left) 2024-09-11T01:55:05.849 INFO:tasks.rados.rados.0.smithi071.stdout:3802: write (excl) oid 44 current snap is 426 2024-09-11T01:55:05.849 INFO:tasks.rados.rados.0.smithi071.stdout:3802: seq_num 1303 ranges {699521=745810,2181802=649115,3515689=383832} 2024-09-11T01:55:05.857 INFO:tasks.rados.rados.0.smithi071.stdout:3802: writing smithi07134777-44 from 699521 to 1445331 tid 1 2024-09-11T01:55:05.865 INFO:tasks.rados.rados.0.smithi071.stdout:3802: writing smithi07134777-44 from 2181802 to 2830917 tid 2 2024-09-11T01:55:05.871 INFO:tasks.rados.rados.0.smithi071.stdout:3802: writing smithi07134777-44 from 3515689 to 3899521 tid 3 2024-09-11T01:55:05.871 INFO:tasks.rados.rados.0.smithi071.stdout:3803: read oid 14 snap -1 2024-09-11T01:55:05.871 INFO:tasks.rados.rados.0.smithi071.stdout:3803: expect deleted 2024-09-11T01:55:05.871 INFO:tasks.rados.rados.0.smithi071.stdout:3804: snap_remove snap 421 2024-09-11T01:55:06.772 INFO:tasks.rados.rados.0.smithi071.stdout:3804: done (9 left) 2024-09-11T01:55:06.773 INFO:tasks.rados.rados.0.smithi071.stdout:3805: write oid 33 current snap is 426 2024-09-11T01:55:06.773 INFO:tasks.rados.rados.0.smithi071.stdout:3805: seq_num 1304 ranges {716328=421059,1840284=509126,3143112=659293} 2024-09-11T01:55:06.778 INFO:tasks.rados.rados.0.smithi071.stdout:3805: writing smithi07134777-33 from 716328 to 1137387 tid 1 2024-09-11T01:55:06.786 INFO:tasks.rados.rados.0.smithi071.stdout:3805: writing smithi07134777-33 from 1840284 to 2349410 tid 2 2024-09-11T01:55:06.794 INFO:tasks.rados.rados.0.smithi071.stdout:3805: writing smithi07134777-33 from 3143112 to 3802405 tid 3 2024-09-11T01:55:06.794 INFO:tasks.rados.rados.0.smithi071.stdout:3806: snap_remove snap 408 2024-09-11T01:55:07.778 INFO:tasks.rados.rados.0.smithi071.stdout:3806: done (10 left) 2024-09-11T01:55:07.864 INFO:tasks.rados.rados.0.smithi071.stdout:3807: snap_remove snap 377 2024-09-11T01:55:08.779 INFO:tasks.rados.rados.0.smithi071.stdout:3807: done (10 left) 2024-09-11T01:55:08.780 INFO:tasks.rados.rados.0.smithi071.stdout:3808: write oid 32 current snap is 426 2024-09-11T01:55:08.780 INFO:tasks.rados.rados.0.smithi071.stdout:3808: seq_num 1305 ranges {733135=496308,1898766=652784,3170535=470565} 2024-09-11T01:55:08.786 INFO:tasks.rados.rados.0.smithi071.stdout:3808: writing smithi07134777-32 from 733135 to 1229443 tid 1 2024-09-11T01:55:08.794 INFO:tasks.rados.rados.0.smithi071.stdout:3808: writing smithi07134777-32 from 1898766 to 2551550 tid 2 2024-09-11T01:55:08.801 INFO:tasks.rados.rados.0.smithi071.stdout:3808: writing smithi07134777-32 from 3170535 to 3641100 tid 3 2024-09-11T01:55:08.801 INFO:tasks.rados.rados.0.smithi071.stdout:3809: write oid 45 current snap is 426 2024-09-11T01:55:08.801 INFO:tasks.rados.rados.0.smithi071.stdout:3809: seq_num 1306 ranges {749942=571557,1840895=796442,3365252=584690} 2024-09-11T01:55:08.808 INFO:tasks.rados.rados.0.smithi071.stdout:3809: writing smithi07134777-45 from 749942 to 1321499 tid 1 2024-09-11T01:55:08.815 INFO:tasks.rados.rados.0.smithi071.stdout:3809: writing smithi07134777-45 from 1840895 to 2637337 tid 2 2024-09-11T01:55:08.824 INFO:tasks.rados.rados.0.smithi071.stdout:3809: writing smithi07134777-45 from 3365252 to 3949942 tid 3 2024-09-11T01:55:08.824 INFO:tasks.rados.rados.0.smithi071.stdout:3810: copy_from oid 37 from oid 10 current snap is 426 2024-09-11T01:55:08.824 INFO:tasks.rados.rados.0.smithi071.stdout:3792: finishing write tid 6 to smithi07134777-3 2024-09-11T01:55:08.824 INFO:tasks.rados.rados.0.smithi071.stdout:3792: oid 3 updating version 0 to 2440 2024-09-11T01:55:08.824 INFO:tasks.rados.rados.0.smithi071.stdout:3792: oid 3 version 2440 is already newer than 2437 2024-09-11T01:55:08.824 INFO:tasks.rados.rados.0.smithi071.stdout:3792: oid 3 version 2440 is already newer than 2439 2024-09-11T01:55:08.824 INFO:tasks.rados.rados.0.smithi071.stdout:3792: oid 3 version 2440 is already newer than 2436 2024-09-11T01:55:08.824 INFO:tasks.rados.rados.0.smithi071.stdout:3792: oid 3 version 2440 is already newer than 2438 2024-09-11T01:55:08.824 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 2440 (ObjNum 1300 snap 426 seq_num 1300) dirty exists 2024-09-11T01:55:08.824 INFO:tasks.rados.rados.0.smithi071.stdout:3792: left oid 3 (ObjNum 1300 snap 426 seq_num 1300) 2024-09-11T01:55:08.825 INFO:tasks.rados.rados.0.smithi071.stdout:3792: done (12 left) 2024-09-11T01:55:08.825 INFO:tasks.rados.rados.0.smithi071.stdout:3794: done (11 left) 2024-09-11T01:55:08.825 INFO:tasks.rados.rados.0.smithi071.stdout:3811: read oid 28 snap -1 2024-09-11T01:55:08.825 INFO:tasks.rados.rados.0.smithi071.stdout:3811: expect deleted 2024-09-11T01:55:08.825 INFO:tasks.rados.rados.0.smithi071.stdout:3812: write oid 39 current snap is 426 2024-09-11T01:55:08.825 INFO:tasks.rados.rados.0.smithi071.stdout:3796: expect (ObjNum 1291 snap 425 seq_num 1291) 2024-09-11T01:55:09.015 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:55:09.115 INFO:tasks.rados.rados.0.smithi071.stdout:3798: finishing write tid 1 to smithi07134777-17 2024-09-11T01:55:09.115 INFO:tasks.rados.rados.0.smithi071.stdout:3798: finishing write tid 2 to smithi07134777-17 2024-09-11T01:55:09.115 INFO:tasks.rados.rados.0.smithi071.stdout:3799: finishing copy_from to smithi07134777-41 2024-09-11T01:55:09.115 INFO:tasks.rados.rados.0.smithi071.stdout:3799: got expected ENOENT (src dne) 2024-09-11T01:55:09.115 INFO:tasks.rados.rados.0.smithi071.stdout:3799: finishing copy_from racing read to smithi07134777-41 2024-09-11T01:55:09.115 INFO:tasks.rados.rados.0.smithi071.stdout:3798: finishing write tid 3 to smithi07134777-17 2024-09-11T01:55:09.115 INFO:tasks.rados.rados.0.smithi071.stdout:3798: finishing write tid 4 to smithi07134777-17 2024-09-11T01:55:09.115 INFO:tasks.rados.rados.0.smithi071.stdout:3798: finishing write tid 5 to smithi07134777-17 2024-09-11T01:55:09.115 INFO:tasks.rados.rados.0.smithi071.stdout:3798: oid 17 updating version 0 to 2577 2024-09-11T01:55:09.116 INFO:tasks.rados.rados.0.smithi071.stdout:3798: oid 17 updating version 2577 to 2580 2024-09-11T01:55:09.116 INFO:tasks.rados.rados.0.smithi071.stdout:3798: oid 17 version 2580 is already newer than 2579 2024-09-11T01:55:09.116 INFO:tasks.rados.rados.0.smithi071.stdout:3798: oid 17 version 2580 is already newer than 2578 2024-09-11T01:55:09.116 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 2580 (ObjNum 1301 snap 426 seq_num 1301) dirty exists 2024-09-11T01:55:09.116 INFO:tasks.rados.rados.0.smithi071.stdout:3798: left oid 17 (ObjNum 1301 snap 426 seq_num 1301) 2024-09-11T01:55:09.116 INFO:tasks.rados.rados.0.smithi071.stdout:3812: seq_num 1307 ranges {766749=646806,1899377=656453,3109028=609462} 2024-09-11T01:55:09.122 INFO:tasks.rados.rados.0.smithi071.stdout:3812: writing smithi07134777-39 from 766749 to 1413555 tid 1 2024-09-11T01:55:09.128 INFO:tasks.rados.rados.0.smithi071.stdout:3812: writing smithi07134777-39 from 1899377 to 2555830 tid 2 2024-09-11T01:55:09.136 INFO:tasks.rados.rados.0.smithi071.stdout:3812: writing smithi07134777-39 from 3109028 to 3718490 tid 3 2024-09-11T01:55:09.136 INFO:tasks.rados.rados.0.smithi071.stdout:3796: done (12 left) 2024-09-11T01:55:09.136 INFO:tasks.rados.rados.0.smithi071.stdout:3798: done (11 left) 2024-09-11T01:55:09.136 INFO:tasks.rados.rados.0.smithi071.stdout:3799: done (10 left) 2024-09-11T01:55:09.137 INFO:tasks.rados.rados.0.smithi071.stdout:3813: read oid 40 snap -1 2024-09-11T01:55:09.137 INFO:tasks.rados.rados.0.smithi071.stdout:3813: expect (ObjNum 1254 snap 415 seq_num 1254) 2024-09-11T01:55:09.137 INFO:tasks.rados.rados.0.smithi071.stdout:3814: write oid 49 current snap is 426 2024-09-11T01:55:09.137 INFO:tasks.rados.rados.0.smithi071.stdout:3800: expect (ObjNum 1216 snap 397 seq_num 1216) 2024-09-11T01:55:09.137 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 126435 2024-09-11T01:55:09.183 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:55:09.278 INFO:tasks.rados.rados.0.smithi071.stdout:3801: finishing write tid 1 to smithi07134777-47 2024-09-11T01:55:09.278 INFO:tasks.rados.rados.0.smithi071.stdout:3801: finishing write tid 2 to smithi07134777-47 2024-09-11T01:55:09.278 INFO:tasks.rados.rados.0.smithi071.stdout:3801: finishing write tid 3 to smithi07134777-47 2024-09-11T01:55:09.278 INFO:tasks.rados.rados.0.smithi071.stdout:3801: finishing write tid 4 to smithi07134777-47 2024-09-11T01:55:09.278 INFO:tasks.rados.rados.0.smithi071.stdout:3801: finishing write tid 5 to smithi07134777-47 2024-09-11T01:55:09.278 INFO:tasks.rados.rados.0.smithi071.stdout:3801: oid 47 updating version 0 to 2581 2024-09-11T01:55:09.278 INFO:tasks.rados.rados.0.smithi071.stdout:3801: oid 47 updating version 2581 to 2584 2024-09-11T01:55:09.279 INFO:tasks.rados.rados.0.smithi071.stdout:3801: oid 47 version 2584 is already newer than 2582 2024-09-11T01:55:09.279 INFO:tasks.rados.rados.0.smithi071.stdout:3801: oid 47 version 2584 is already newer than 2583 2024-09-11T01:55:09.279 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 2584 (ObjNum 1302 snap 426 seq_num 1302) dirty exists 2024-09-11T01:55:09.279 INFO:tasks.rados.rados.0.smithi071.stdout:3801: left oid 47 (ObjNum 1302 snap 426 seq_num 1302) 2024-09-11T01:55:09.279 INFO:tasks.rados.rados.0.smithi071.stdout:3802: finishing write tid 1 to smithi07134777-44 2024-09-11T01:55:09.279 INFO:tasks.rados.rados.0.smithi071.stdout:3802: finishing write tid 2 to smithi07134777-44 2024-09-11T01:55:09.279 INFO:tasks.rados.rados.0.smithi071.stdout:3802: finishing write tid 3 to smithi07134777-44 2024-09-11T01:55:09.279 INFO:tasks.rados.rados.0.smithi071.stdout:3802: finishing write tid 4 to smithi07134777-44 2024-09-11T01:55:09.279 INFO:tasks.rados.rados.0.smithi071.stdout:3802: finishing write tid 5 to smithi07134777-44 2024-09-11T01:55:09.279 INFO:tasks.rados.rados.0.smithi071.stdout:3802: oid 44 updating version 0 to 1357 2024-09-11T01:55:09.279 INFO:tasks.rados.rados.0.smithi071.stdout:3802: oid 44 version 1357 is already newer than 1355 2024-09-11T01:55:09.279 INFO:tasks.rados.rados.0.smithi071.stdout:3802: oid 44 version 1357 is already newer than 1356 2024-09-11T01:55:09.279 INFO:tasks.rados.rados.0.smithi071.stdout:3802: oid 44 updating version 1357 to 1358 2024-09-11T01:55:09.279 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 1358 (ObjNum 1303 snap 426 seq_num 1303) dirty exists 2024-09-11T01:55:09.280 INFO:tasks.rados.rados.0.smithi071.stdout:3802: left oid 44 (ObjNum 1303 snap 426 seq_num 1303) 2024-09-11T01:55:09.280 INFO:tasks.rados.rados.0.smithi071.stdout:3805: finishing write tid 1 to smithi07134777-33 2024-09-11T01:55:09.280 INFO:tasks.rados.rados.0.smithi071.stdout:3805: finishing write tid 2 to smithi07134777-33 2024-09-11T01:55:09.280 INFO:tasks.rados.rados.0.smithi071.stdout:3805: finishing write tid 3 to smithi07134777-33 2024-09-11T01:55:09.280 INFO:tasks.rados.rados.0.smithi071.stdout:3814: seq_num 1308 ranges {783556=438408,1674212=400111,2736451=420734,3882270=101286} 2024-09-11T01:55:09.284 INFO:tasks.rados.rados.0.smithi071.stdout:3814: writing smithi07134777-49 from 783556 to 1221964 tid 1 2024-09-11T01:55:09.289 INFO:tasks.rados.rados.0.smithi071.stdout:3814: writing smithi07134777-49 from 1674212 to 2074323 tid 2 2024-09-11T01:55:09.295 INFO:tasks.rados.rados.0.smithi071.stdout:3814: writing smithi07134777-49 from 2736451 to 3157185 tid 3 2024-09-11T01:55:09.299 INFO:tasks.rados.rados.0.smithi071.stdout:3814: writing smithi07134777-49 from 3882270 to 3983556 tid 4 2024-09-11T01:55:09.299 INFO:tasks.rados.rados.0.smithi071.stdout:3800: done (11 left) 2024-09-11T01:55:09.299 INFO:tasks.rados.rados.0.smithi071.stdout:3801: done (10 left) 2024-09-11T01:55:09.300 INFO:tasks.rados.rados.0.smithi071.stdout:3802: done (9 left) 2024-09-11T01:55:09.300 INFO:tasks.rados.rados.0.smithi071.stdout:3803: done (8 left) 2024-09-11T01:55:09.300 INFO:tasks.rados.rados.0.smithi071.stdout:3815: snap_remove snap 422 2024-09-11T01:55:09.788 INFO:tasks.rados.rados.0.smithi071.stdout:3815: done (8 left) 2024-09-11T01:55:09.788 INFO:tasks.rados.rados.0.smithi071.stdout:3816: write oid 46 current snap is 426 2024-09-11T01:55:09.788 INFO:tasks.rados.rados.0.smithi071.stdout:3816: seq_num 1309 ranges {400363=513657,1332694=660122} 2024-09-11T01:55:09.793 INFO:tasks.rados.rados.0.smithi071.stdout:3816: writing smithi07134777-46 from 400363 to 914020 tid 1 2024-09-11T01:55:09.800 INFO:tasks.rados.rados.0.smithi071.stdout:3816: writing smithi07134777-46 from 1332694 to 1992816 tid 2 2024-09-11T01:55:09.801 INFO:tasks.rados.rados.0.smithi071.stdout:3817: write (excl) oid 44 current snap is 426 2024-09-11T01:55:09.801 INFO:tasks.rados.rados.0.smithi071.stdout:3805: finishing write tid 4 to smithi07134777-33 2024-09-11T01:55:09.801 INFO:tasks.rados.rados.0.smithi071.stdout:3805: finishing write tid 5 to smithi07134777-33 2024-09-11T01:55:09.801 INFO:tasks.rados.rados.0.smithi071.stdout:3805: oid 33 updating version 0 to 2936 2024-09-11T01:55:09.801 INFO:tasks.rados.rados.0.smithi071.stdout:3805: oid 33 version 2936 is already newer than 2926 2024-09-11T01:55:09.801 INFO:tasks.rados.rados.0.smithi071.stdout:3805: oid 33 version 2936 is already newer than 2931 2024-09-11T01:55:09.801 INFO:tasks.rados.rados.0.smithi071.stdout:3805: oid 33 updating version 2936 to 2937 2024-09-11T01:55:09.801 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 2937 (ObjNum 1304 snap 426 seq_num 1304) dirty exists 2024-09-11T01:55:09.801 INFO:tasks.rados.rados.0.smithi071.stdout:3805: left oid 33 (ObjNum 1304 snap 426 seq_num 1304) 2024-09-11T01:55:09.801 INFO:tasks.rados.rados.0.smithi071.stdout:3808: finishing write tid 1 to smithi07134777-32 2024-09-11T01:55:09.801 INFO:tasks.rados.rados.0.smithi071.stdout:3808: finishing write tid 2 to smithi07134777-32 2024-09-11T01:55:09.801 INFO:tasks.rados.rados.0.smithi071.stdout:3808: finishing write tid 3 to smithi07134777-32 2024-09-11T01:55:09.801 INFO:tasks.rados.rados.0.smithi071.stdout:3808: finishing write tid 4 to smithi07134777-32 2024-09-11T01:55:09.802 INFO:tasks.rados.rados.0.smithi071.stdout:3808: finishing write tid 5 to smithi07134777-32 2024-09-11T01:55:09.802 INFO:tasks.rados.rados.0.smithi071.stdout:3808: oid 32 updating version 0 to 2121 2024-09-11T01:55:09.802 INFO:tasks.rados.rados.0.smithi071.stdout:3808: oid 32 version 2121 is already newer than 2119 2024-09-11T01:55:09.802 INFO:tasks.rados.rados.0.smithi071.stdout:3808: oid 32 version 2121 is already newer than 2120 2024-09-11T01:55:09.802 INFO:tasks.rados.rados.0.smithi071.stdout:3808: oid 32 version 2121 is already newer than 2118 2024-09-11T01:55:09.802 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 2121 (ObjNum 1305 snap 426 seq_num 1305) dirty exists 2024-09-11T01:55:09.802 INFO:tasks.rados.rados.0.smithi071.stdout:3808: left oid 32 (ObjNum 1305 snap 426 seq_num 1305) 2024-09-11T01:55:09.802 INFO:tasks.rados.rados.0.smithi071.stdout:3809: finishing write tid 1 to smithi07134777-45 2024-09-11T01:55:09.802 INFO:tasks.rados.rados.0.smithi071.stdout:3809: finishing write tid 2 to smithi07134777-45 2024-09-11T01:55:09.802 INFO:tasks.rados.rados.0.smithi071.stdout:3809: finishing write tid 3 to smithi07134777-45 2024-09-11T01:55:09.802 INFO:tasks.rados.rados.0.smithi071.stdout:3809: finishing write tid 4 to smithi07134777-45 2024-09-11T01:55:09.802 INFO:tasks.rados.rados.0.smithi071.stdout:3809: finishing write tid 5 to smithi07134777-45 2024-09-11T01:55:09.802 INFO:tasks.rados.rados.0.smithi071.stdout:3809: oid 45 updating version 0 to 2516 2024-09-11T01:55:09.802 INFO:tasks.rados.rados.0.smithi071.stdout:3809: oid 45 version 2516 is already newer than 2515 2024-09-11T01:55:09.803 INFO:tasks.rados.rados.0.smithi071.stdout:3809: oid 45 version 2516 is already newer than 2513 2024-09-11T01:55:09.803 INFO:tasks.rados.rados.0.smithi071.stdout:3809: oid 45 version 2516 is already newer than 2514 2024-09-11T01:55:09.803 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 2516 (ObjNum 1306 snap 426 seq_num 1306) dirty exists 2024-09-11T01:55:09.803 INFO:tasks.rados.rados.0.smithi071.stdout:3809: left oid 45 (ObjNum 1306 snap 426 seq_num 1306) 2024-09-11T01:55:09.803 INFO:tasks.rados.rados.0.smithi071.stdout:3810: finishing copy_from racing read to smithi07134777-37 2024-09-11T01:55:09.803 INFO:tasks.rados.rados.0.smithi071.stdout:3810: finishing copy_from to smithi07134777-37 2024-09-11T01:55:09.803 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 2533 (ObjNum 1232 snap 403 seq_num 1232) dirty exists 2024-09-11T01:55:09.803 INFO:tasks.rados.rados.0.smithi071.stdout:3812: finishing write tid 1 to smithi07134777-39 2024-09-11T01:55:09.803 INFO:tasks.rados.rados.0.smithi071.stdout:3812: finishing write tid 2 to smithi07134777-39 2024-09-11T01:55:09.803 INFO:tasks.rados.rados.0.smithi071.stdout:3813: expect (ObjNum 1254 snap 415 seq_num 1254) 2024-09-11T01:55:09.810 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1987138 2024-09-11T01:55:09.887 INFO:tasks.rados.rados.0.smithi071.stdout:3812: finishing write tid 3 to smithi07134777-39 2024-09-11T01:55:09.887 INFO:tasks.rados.rados.0.smithi071.stdout:3812: finishing write tid 4 to smithi07134777-39 2024-09-11T01:55:09.887 INFO:tasks.rados.rados.0.smithi071.stdout:3812: finishing write tid 5 to smithi07134777-39 2024-09-11T01:55:09.887 INFO:tasks.rados.rados.0.smithi071.stdout:3812: oid 39 updating version 0 to 2534 2024-09-11T01:55:09.887 INFO:tasks.rados.rados.0.smithi071.stdout:3812: oid 39 updating version 2534 to 2537 2024-09-11T01:55:09.888 INFO:tasks.rados.rados.0.smithi071.stdout:3812: oid 39 version 2537 is already newer than 2536 2024-09-11T01:55:09.888 INFO:tasks.rados.rados.0.smithi071.stdout:3812: oid 39 version 2537 is already newer than 2535 2024-09-11T01:55:09.888 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 2537 (ObjNum 1307 snap 426 seq_num 1307) dirty exists 2024-09-11T01:55:09.888 INFO:tasks.rados.rados.0.smithi071.stdout:3812: left oid 39 (ObjNum 1307 snap 426 seq_num 1307) 2024-09-11T01:55:09.888 INFO:tasks.rados.rados.0.smithi071.stdout:3814: finishing write tid 1 to smithi07134777-49 2024-09-11T01:55:09.888 INFO:tasks.rados.rados.0.smithi071.stdout:3817: seq_num 1310 ranges {417170=588906,1674823=342347} 2024-09-11T01:55:09.892 INFO:tasks.rados.rados.0.smithi071.stdout:3817: writing smithi07134777-44 from 417170 to 1006076 tid 1 2024-09-11T01:55:09.898 INFO:tasks.rados.rados.0.smithi071.stdout:3817: writing smithi07134777-44 from 1674823 to 2017170 tid 2 2024-09-11T01:55:09.899 INFO:tasks.rados.rados.0.smithi071.stdout:3805: done (9 left) 2024-09-11T01:55:09.899 INFO:tasks.rados.rados.0.smithi071.stdout:3808: done (8 left) 2024-09-11T01:55:09.899 INFO:tasks.rados.rados.0.smithi071.stdout:3809: done (7 left) 2024-09-11T01:55:09.899 INFO:tasks.rados.rados.0.smithi071.stdout:3810: done (6 left) 2024-09-11T01:55:09.899 INFO:tasks.rados.rados.0.smithi071.stdout:3811: done (5 left) 2024-09-11T01:55:09.899 INFO:tasks.rados.rados.0.smithi071.stdout:3812: done (4 left) 2024-09-11T01:55:09.899 INFO:tasks.rados.rados.0.smithi071.stdout:3813: done (3 left) 2024-09-11T01:55:09.899 INFO:tasks.rados.rados.0.smithi071.stdout:3818: snap_create 2024-09-11T01:55:09.899 INFO:tasks.rados.rados.0.smithi071.stdout:3814: finishing write tid 2 to smithi07134777-49 2024-09-11T01:55:09.899 INFO:tasks.rados.rados.0.smithi071.stdout:3814: finishing write tid 3 to smithi07134777-49 2024-09-11T01:55:09.899 INFO:tasks.rados.rados.0.smithi071.stdout:3814: finishing write tid 4 to smithi07134777-49 2024-09-11T01:55:09.899 INFO:tasks.rados.rados.0.smithi071.stdout:3814: finishing write tid 5 to smithi07134777-49 2024-09-11T01:55:09.900 INFO:tasks.rados.rados.0.smithi071.stdout:3814: finishing write tid 6 to smithi07134777-49 2024-09-11T01:55:09.900 INFO:tasks.rados.rados.0.smithi071.stdout:3814: oid 49 updating version 0 to 2460 2024-09-11T01:55:09.900 INFO:tasks.rados.rados.0.smithi071.stdout:3814: oid 49 updating version 2460 to 2462 2024-09-11T01:55:09.900 INFO:tasks.rados.rados.0.smithi071.stdout:3814: oid 49 version 2462 is already newer than 2458 2024-09-11T01:55:09.900 INFO:tasks.rados.rados.0.smithi071.stdout:3814: oid 49 version 2462 is already newer than 2461 2024-09-11T01:55:09.900 INFO:tasks.rados.rados.0.smithi071.stdout:3814: oid 49 version 2462 is already newer than 2459 2024-09-11T01:55:09.900 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 2462 (ObjNum 1308 snap 426 seq_num 1308) dirty exists 2024-09-11T01:55:09.900 INFO:tasks.rados.rados.0.smithi071.stdout:3814: left oid 49 (ObjNum 1308 snap 426 seq_num 1308) 2024-09-11T01:55:09.900 INFO:tasks.rados.rados.0.smithi071.stdout:3816: finishing write tid 1 to smithi07134777-46 2024-09-11T01:55:09.900 INFO:tasks.rados.rados.0.smithi071.stdout:3816: finishing write tid 2 to smithi07134777-46 2024-09-11T01:55:09.900 INFO:tasks.rados.rados.0.smithi071.stdout:3816: finishing write tid 3 to smithi07134777-46 2024-09-11T01:55:09.900 INFO:tasks.rados.rados.0.smithi071.stdout:3816: finishing write tid 4 to smithi07134777-46 2024-09-11T01:55:09.900 INFO:tasks.rados.rados.0.smithi071.stdout:3816: oid 46 updating version 0 to 2588 2024-09-11T01:55:09.900 INFO:tasks.rados.rados.0.smithi071.stdout:3816: oid 46 version 2588 is already newer than 2587 2024-09-11T01:55:09.901 INFO:tasks.rados.rados.0.smithi071.stdout:3816: oid 46 version 2588 is already newer than 2586 2024-09-11T01:55:09.901 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 2588 (ObjNum 1309 snap 426 seq_num 1309) dirty exists 2024-09-11T01:55:09.901 INFO:tasks.rados.rados.0.smithi071.stdout:3816: left oid 46 (ObjNum 1309 snap 426 seq_num 1309) 2024-09-11T01:55:09.901 INFO:tasks.rados.rados.0.smithi071.stdout:3817: finishing write tid 1 to smithi07134777-44 2024-09-11T01:55:09.901 INFO:tasks.rados.rados.0.smithi071.stdout:3817: finishing write tid 2 to smithi07134777-44 2024-09-11T01:55:09.904 INFO:tasks.rados.rados.0.smithi071.stdout:3817: finishing write tid 3 to smithi07134777-44 2024-09-11T01:55:09.904 INFO:tasks.rados.rados.0.smithi071.stdout:3817: finishing write tid 4 to smithi07134777-44 2024-09-11T01:55:09.904 INFO:tasks.rados.rados.0.smithi071.stdout:3817: oid 44 updating version 0 to 1374 2024-09-11T01:55:09.904 INFO:tasks.rados.rados.0.smithi071.stdout:3817: oid 44 updating version 1374 to 1375 2024-09-11T01:55:09.904 INFO:tasks.rados.rados.0.smithi071.stdout:3817: oid 44 version 1375 is already newer than 1373 2024-09-11T01:55:09.904 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 1375 (ObjNum 1310 snap 426 seq_num 1310) dirty exists 2024-09-11T01:55:09.904 INFO:tasks.rados.rados.0.smithi071.stdout:3817: left oid 44 (ObjNum 1310 snap 426 seq_num 1310) 2024-09-11T01:55:10.725 INFO:tasks.rados.rados.0.smithi071.stdout:3814: done (3 left) 2024-09-11T01:55:10.725 INFO:tasks.rados.rados.0.smithi071.stdout:3816: done (2 left) 2024-09-11T01:55:10.725 INFO:tasks.rados.rados.0.smithi071.stdout:3817: done (1 left) 2024-09-11T01:55:10.725 INFO:tasks.rados.rados.0.smithi071.stdout:3818: done (0 left) 2024-09-11T01:55:10.725 INFO:tasks.rados.rados.0.smithi071.stdout:3819: write (excl) oid 32 current snap is 427 2024-09-11T01:55:10.725 INFO:tasks.rados.rados.0.smithi071.stdout:3819: seq_num 1311 ranges {433977=664155,1733305=300672} 2024-09-11T01:55:10.731 INFO:tasks.rados.rados.0.smithi071.stdout:3819: writing smithi07134777-32 from 433977 to 1098132 tid 1 2024-09-11T01:55:10.737 INFO:tasks.rados.rados.0.smithi071.stdout:3819: writing smithi07134777-32 from 1733305 to 2033977 tid 2 2024-09-11T01:55:10.737 INFO:tasks.rados.rados.0.smithi071.stdout:3820: rollback oid 27 current snap is 427 2024-09-11T01:55:10.737 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 27 to 385 2024-09-11T01:55:10.737 INFO:tasks.rados.rados.0.smithi071.stdout:3821: rollback oid 50 current snap is 427 2024-09-11T01:55:10.737 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 50 to 413 2024-09-11T01:55:10.737 INFO:tasks.rados.rados.0.smithi071.stdout:3822: copy_from oid 5 from oid 18 current snap is 427 2024-09-11T01:55:10.737 INFO:tasks.rados.rados.0.smithi071.stdout:3823: write (excl) oid 41 current snap is 427 2024-09-11T01:55:10.737 INFO:tasks.rados.rados.0.smithi071.stdout:3823: seq_num 1312 ranges {450784=739404,1791787=258997} 2024-09-11T01:55:10.744 INFO:tasks.rados.rados.0.smithi071.stdout:3823: writing smithi07134777-41 from 450784 to 1190188 tid 1 2024-09-11T01:55:10.749 INFO:tasks.rados.rados.0.smithi071.stdout:3823: writing smithi07134777-41 from 1791787 to 2050784 tid 2 2024-09-11T01:55:10.750 INFO:tasks.rados.rados.0.smithi071.stdout:3824: snap_create 2024-09-11T01:55:10.750 INFO:tasks.rados.rados.0.smithi071.stdout:3819: finishing write tid 1 to smithi07134777-32 2024-09-11T01:55:10.750 INFO:tasks.rados.rados.0.smithi071.stdout:3822: finishing copy_from to smithi07134777-5 2024-09-11T01:55:10.750 INFO:tasks.rados.rados.0.smithi071.stdout:3822: got expected ENOENT (src dne) 2024-09-11T01:55:10.750 INFO:tasks.rados.rados.0.smithi071.stdout:3819: finishing write tid 2 to smithi07134777-32 2024-09-11T01:55:10.750 INFO:tasks.rados.rados.0.smithi071.stdout:3819: finishing write tid 3 to smithi07134777-32 2024-09-11T01:55:10.750 INFO:tasks.rados.rados.0.smithi071.stdout:3822: finishing copy_from racing read to smithi07134777-5 2024-09-11T01:55:10.750 INFO:tasks.rados.rados.0.smithi071.stdout:3819: finishing write tid 4 to smithi07134777-32 2024-09-11T01:55:10.750 INFO:tasks.rados.rados.0.smithi071.stdout:3819: oid 32 updating version 0 to 2540 2024-09-11T01:55:10.750 INFO:tasks.rados.rados.0.smithi071.stdout:3819: oid 32 version 2540 is already newer than 2539 2024-09-11T01:55:10.750 INFO:tasks.rados.rados.0.smithi071.stdout:3819: oid 32 version 2540 is already newer than 2538 2024-09-11T01:55:10.750 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 2540 (ObjNum 1311 snap 427 seq_num 1311) dirty exists 2024-09-11T01:55:10.750 INFO:tasks.rados.rados.0.smithi071.stdout:3819: left oid 32 (ObjNum 1311 snap 427 seq_num 1311) 2024-09-11T01:55:10.751 INFO:tasks.rados.rados.0.smithi071.stdout:3821: finishing rollback tid 0 to smithi07134777-50 2024-09-11T01:55:10.751 INFO:tasks.rados.rados.0.smithi071.stdout:3821: finishing rollback tid 1 to smithi07134777-50 2024-09-11T01:55:10.751 INFO:tasks.rados.rados.0.smithi071.stdout:3821: finishing rollback tid 2 to smithi07134777-50 2024-09-11T01:55:10.751 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 3057 (ObjNum 1191 snap 379 seq_num 1191) dirty exists 2024-09-11T01:55:10.751 INFO:tasks.rados.rados.0.smithi071.stdout:3820: finishing rollback tid 0 to smithi07134777-27 2024-09-11T01:55:10.751 INFO:tasks.rados.rados.0.smithi071.stdout:3820: finishing rollback tid 1 to smithi07134777-27 2024-09-11T01:55:10.751 INFO:tasks.rados.rados.0.smithi071.stdout:3820: finishing rollback tid 2 to smithi07134777-27 2024-09-11T01:55:10.751 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 2971 (ObjNum 1154 snap 366 seq_num 1154) dirty exists 2024-09-11T01:55:10.751 INFO:tasks.rados.rados.0.smithi071.stdout:3823: finishing write tid 1 to smithi07134777-41 2024-09-11T01:55:10.757 INFO:tasks.rados.rados.0.smithi071.stdout:3823: finishing write tid 2 to smithi07134777-41 2024-09-11T01:55:10.757 INFO:tasks.rados.rados.0.smithi071.stdout:3823: finishing write tid 3 to smithi07134777-41 2024-09-11T01:55:10.758 INFO:tasks.rados.rados.0.smithi071.stdout:3823: finishing write tid 4 to smithi07134777-41 2024-09-11T01:55:10.758 INFO:tasks.rados.rados.0.smithi071.stdout:3823: oid 41 updating version 0 to 2537 2024-09-11T01:55:10.758 INFO:tasks.rados.rados.0.smithi071.stdout:3823: oid 41 version 2537 is already newer than 2535 2024-09-11T01:55:10.758 INFO:tasks.rados.rados.0.smithi071.stdout:3823: oid 41 version 2537 is already newer than 2536 2024-09-11T01:55:10.758 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 2537 (ObjNum 1312 snap 427 seq_num 1312) dirty exists 2024-09-11T01:55:10.758 INFO:tasks.rados.rados.0.smithi071.stdout:3823: left oid 41 (ObjNum 1312 snap 427 seq_num 1312) 2024-09-11T01:55:11.733 INFO:tasks.rados.rados.0.smithi071.stdout:3819: done (5 left) 2024-09-11T01:55:11.733 INFO:tasks.rados.rados.0.smithi071.stdout:3820: done (4 left) 2024-09-11T01:55:11.733 INFO:tasks.rados.rados.0.smithi071.stdout:3821: done (3 left) 2024-09-11T01:55:11.733 INFO:tasks.rados.rados.0.smithi071.stdout:3822: done (2 left) 2024-09-11T01:55:11.733 INFO:tasks.rados.rados.0.smithi071.stdout:3823: done (1 left) 2024-09-11T01:55:11.733 INFO:tasks.rados.rados.0.smithi071.stdout:3824: done (0 left) 2024-09-11T01:55:11.733 INFO:tasks.rados.rados.0.smithi071.stdout:3825: snap_remove snap 424 2024-09-11T01:55:12.737 INFO:tasks.rados.rados.0.smithi071.stdout:3825: done (0 left) 2024-09-11T01:55:12.737 INFO:tasks.rados.rados.0.smithi071.stdout:3826: snap_remove snap 393 2024-09-11T01:55:13.736 INFO:tasks.rados.rados.0.smithi071.stdout:3826: done (0 left) 2024-09-11T01:55:13.736 INFO:tasks.rados.rados.0.smithi071.stdout:3827: write (excl) oid 21 current snap is 428 2024-09-11T01:55:13.736 INFO:tasks.rados.rados.0.smithi071.stdout:3827: seq_num 1313 ranges {467591=414653,1450269=617322} 2024-09-11T01:55:13.741 INFO:tasks.rados.rados.0.smithi071.stdout:3827: writing smithi07134777-21 from 467591 to 882244 tid 1 2024-09-11T01:55:13.748 INFO:tasks.rados.rados.0.smithi071.stdout:3827: writing smithi07134777-21 from 1450269 to 2067591 tid 2 2024-09-11T01:55:13.749 INFO:tasks.rados.rados.0.smithi071.stdout:3828: write oid 44 current snap is 428 2024-09-11T01:55:13.749 INFO:tasks.rados.rados.0.smithi071.stdout:3828: seq_num 1314 ranges {484398=489902,1392398=411118} 2024-09-11T01:55:13.754 INFO:tasks.rados.rados.0.smithi071.stdout:3828: writing smithi07134777-44 from 484398 to 974300 tid 1 2024-09-11T01:55:13.760 INFO:tasks.rados.rados.0.smithi071.stdout:3828: writing smithi07134777-44 from 1392398 to 1803516 tid 2 2024-09-11T01:55:13.760 INFO:tasks.rados.rados.0.smithi071.stdout:3829: copy_from oid 14 from oid 29 current snap is 428 2024-09-11T01:55:13.760 INFO:tasks.rados.rados.0.smithi071.stdout:3828: finishing write tid 1 to smithi07134777-44 2024-09-11T01:55:13.760 INFO:tasks.rados.rados.0.smithi071.stdout:3830: read oid 47 snap -1 2024-09-11T01:55:13.760 INFO:tasks.rados.rados.0.smithi071.stdout:3830: expect (ObjNum 1302 snap 426 seq_num 1302) 2024-09-11T01:55:13.760 INFO:tasks.rados.rados.0.smithi071.stdout:3831: snap_create 2024-09-11T01:55:13.761 INFO:tasks.rados.rados.0.smithi071.stdout:3827: finishing write tid 1 to smithi07134777-21 2024-09-11T01:55:13.763 INFO:tasks.rados.rados.0.smithi071.stdout:3827: finishing write tid 2 to smithi07134777-21 2024-09-11T01:55:13.763 INFO:tasks.rados.rados.0.smithi071.stdout:3827: finishing write tid 3 to smithi07134777-21 2024-09-11T01:55:13.765 INFO:tasks.rados.rados.0.smithi071.stdout:3829: finishing copy_from to smithi07134777-14 2024-09-11T01:55:13.765 INFO:tasks.rados.rados.0.smithi071.stdout:3829: got expected ENOENT (src dne) 2024-09-11T01:55:13.765 INFO:tasks.rados.rados.0.smithi071.stdout:3827: finishing write tid 4 to smithi07134777-21 2024-09-11T01:55:13.765 INFO:tasks.rados.rados.0.smithi071.stdout:3827: oid 21 updating version 0 to 2535 2024-09-11T01:55:13.765 INFO:tasks.rados.rados.0.smithi071.stdout:3827: oid 21 updating version 2535 to 2536 2024-09-11T01:55:13.765 INFO:tasks.rados.rados.0.smithi071.stdout:3827: oid 21 version 2536 is already newer than 2534 2024-09-11T01:55:13.765 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 2536 (ObjNum 1313 snap 428 seq_num 1313) dirty exists 2024-09-11T01:55:13.765 INFO:tasks.rados.rados.0.smithi071.stdout:3827: left oid 21 (ObjNum 1313 snap 428 seq_num 1313) 2024-09-11T01:55:13.765 INFO:tasks.rados.rados.0.smithi071.stdout:3828: finishing write tid 2 to smithi07134777-44 2024-09-11T01:55:13.765 INFO:tasks.rados.rados.0.smithi071.stdout:3829: finishing copy_from racing read to smithi07134777-14 2024-09-11T01:55:13.766 INFO:tasks.rados.rados.0.smithi071.stdout:3828: finishing write tid 3 to smithi07134777-44 2024-09-11T01:55:13.766 INFO:tasks.rados.rados.0.smithi071.stdout:3828: finishing write tid 4 to smithi07134777-44 2024-09-11T01:55:13.766 INFO:tasks.rados.rados.0.smithi071.stdout:3828: oid 44 updating version 0 to 1383 2024-09-11T01:55:13.766 INFO:tasks.rados.rados.0.smithi071.stdout:3828: oid 44 updating version 1383 to 1385 2024-09-11T01:55:13.767 INFO:tasks.rados.rados.0.smithi071.stdout:3828: oid 44 version 1385 is already newer than 1384 2024-09-11T01:55:13.767 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 1385 (ObjNum 1314 snap 428 seq_num 1314) dirty exists 2024-09-11T01:55:13.767 INFO:tasks.rados.rados.0.smithi071.stdout:3828: left oid 44 (ObjNum 1314 snap 428 seq_num 1314) 2024-09-11T01:55:13.770 INFO:tasks.rados.rados.0.smithi071.stdout:3830: expect (ObjNum 1302 snap 426 seq_num 1302) 2024-09-11T01:55:13.833 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:55:14.752 INFO:tasks.rados.rados.0.smithi071.stdout:3827: done (4 left) 2024-09-11T01:55:14.752 INFO:tasks.rados.rados.0.smithi071.stdout:3828: done (3 left) 2024-09-11T01:55:14.752 INFO:tasks.rados.rados.0.smithi071.stdout:3829: done (2 left) 2024-09-11T01:55:14.752 INFO:tasks.rados.rados.0.smithi071.stdout:3830: done (1 left) 2024-09-11T01:55:14.752 INFO:tasks.rados.rados.0.smithi071.stdout:3831: done (0 left) 2024-09-11T01:55:14.752 INFO:tasks.rados.rados.0.smithi071.stdout:3832: delete oid 33 current snap is 429 2024-09-11T01:55:14.759 INFO:tasks.rados.rados.0.smithi071.stdout:3832: done (0 left) 2024-09-11T01:55:14.759 INFO:tasks.rados.rados.0.smithi071.stdout:3833: rollback oid 25 current snap is 429 2024-09-11T01:55:14.759 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 25 to 414 2024-09-11T01:55:14.759 INFO:tasks.rados.rados.0.smithi071.stdout:3834: read oid 17 snap -1 2024-09-11T01:55:14.759 INFO:tasks.rados.rados.0.smithi071.stdout:3834: expect (ObjNum 1301 snap 426 seq_num 1301) 2024-09-11T01:55:14.759 INFO:tasks.rados.rados.0.smithi071.stdout:3835: write oid 20 current snap is 429 2024-09-11T01:55:14.759 INFO:tasks.rados.rados.0.smithi071.stdout:3835: seq_num 1316 ranges {518012=756753,2025715=92297} 2024-09-11T01:55:14.766 INFO:tasks.rados.rados.0.smithi071.stdout:3835: writing smithi07134777-20 from 518012 to 1274765 tid 1 2024-09-11T01:55:14.771 INFO:tasks.rados.rados.0.smithi071.stdout:3835: writing smithi07134777-20 from 2025715 to 2118012 tid 2 2024-09-11T01:55:14.771 INFO:tasks.rados.rados.0.smithi071.stdout:3836: write (excl) oid 27 current snap is 429 2024-09-11T01:55:14.771 INFO:tasks.rados.rados.0.smithi071.stdout:3833: finishing rollback tid 1 to smithi07134777-25 2024-09-11T01:55:14.771 INFO:tasks.rados.rados.0.smithi071.stdout:3833: finishing rollback tid 2 to smithi07134777-25 2024-09-11T01:55:14.771 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 2582 (ObjNum 1243 snap 411 seq_num 1243) dirty exists 2024-09-11T01:55:14.771 INFO:tasks.rados.rados.0.smithi071.stdout:3834: expect (ObjNum 1301 snap 426 seq_num 1301) 2024-09-11T01:55:14.868 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:55:14.918 INFO:tasks.rados.rados.0.smithi071.stdout:3835: finishing write tid 1 to smithi07134777-20 2024-09-11T01:55:14.918 INFO:tasks.rados.rados.0.smithi071.stdout:3835: finishing write tid 2 to smithi07134777-20 2024-09-11T01:55:14.918 INFO:tasks.rados.rados.0.smithi071.stdout:3835: finishing write tid 3 to smithi07134777-20 2024-09-11T01:55:14.918 INFO:tasks.rados.rados.0.smithi071.stdout:3835: finishing write tid 4 to smithi07134777-20 2024-09-11T01:55:14.918 INFO:tasks.rados.rados.0.smithi071.stdout:3835: oid 20 updating version 0 to 1388 2024-09-11T01:55:14.918 INFO:tasks.rados.rados.0.smithi071.stdout:3835: oid 20 updating version 1388 to 1389 2024-09-11T01:55:14.918 INFO:tasks.rados.rados.0.smithi071.stdout:3835: oid 20 version 1389 is already newer than 1387 2024-09-11T01:55:14.918 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 1389 (ObjNum 1316 snap 429 seq_num 1316) dirty exists 2024-09-11T01:55:14.918 INFO:tasks.rados.rados.0.smithi071.stdout:3835: left oid 20 (ObjNum 1316 snap 429 seq_num 1316) 2024-09-11T01:55:14.918 INFO:tasks.rados.rados.0.smithi071.stdout:3836: seq_num 1317 ranges {534819=432002,1684197=450622} 2024-09-11T01:55:14.923 INFO:tasks.rados.rados.0.smithi071.stdout:3836: writing smithi07134777-27 from 534819 to 966821 tid 1 2024-09-11T01:55:14.929 INFO:tasks.rados.rados.0.smithi071.stdout:3836: writing smithi07134777-27 from 1684197 to 2134819 tid 2 2024-09-11T01:55:14.929 INFO:tasks.rados.rados.0.smithi071.stdout:3833: done (3 left) 2024-09-11T01:55:14.929 INFO:tasks.rados.rados.0.smithi071.stdout:3834: done (2 left) 2024-09-11T01:55:14.929 INFO:tasks.rados.rados.0.smithi071.stdout:3835: done (1 left) 2024-09-11T01:55:14.930 INFO:tasks.rados.rados.0.smithi071.stdout:3837: write oid 19 current snap is 429 2024-09-11T01:55:14.930 INFO:tasks.rados.rados.0.smithi071.stdout:3836: finishing write tid 1 to smithi07134777-27 2024-09-11T01:55:14.930 INFO:tasks.rados.rados.0.smithi071.stdout:3837: seq_num 1318 ranges {551626=507251,1626326=418456} 2024-09-11T01:55:14.934 INFO:tasks.rados.rados.0.smithi071.stdout:3837: writing smithi07134777-19 from 551626 to 1058877 tid 1 2024-09-11T01:55:14.940 INFO:tasks.rados.rados.0.smithi071.stdout:3837: writing smithi07134777-19 from 1626326 to 2044782 tid 2 2024-09-11T01:55:14.941 INFO:tasks.rados.rados.0.smithi071.stdout:3838: read oid 15 snap -1 2024-09-11T01:55:14.941 INFO:tasks.rados.rados.0.smithi071.stdout:3838: expect deleted 2024-09-11T01:55:14.941 INFO:tasks.rados.rados.0.smithi071.stdout:3839: copy_from oid 21 from oid 13 current snap is 429 2024-09-11T01:55:14.941 INFO:tasks.rados.rados.0.smithi071.stdout:3836: finishing write tid 2 to smithi07134777-27 2024-09-11T01:55:14.941 INFO:tasks.rados.rados.0.smithi071.stdout:3836: finishing write tid 3 to smithi07134777-27 2024-09-11T01:55:14.941 INFO:tasks.rados.rados.0.smithi071.stdout:3836: finishing write tid 4 to smithi07134777-27 2024-09-11T01:55:14.941 INFO:tasks.rados.rados.0.smithi071.stdout:3836: oid 27 updating version 0 to 2991 2024-09-11T01:55:14.941 INFO:tasks.rados.rados.0.smithi071.stdout:3836: oid 27 updating version 2991 to 2992 2024-09-11T01:55:14.941 INFO:tasks.rados.rados.0.smithi071.stdout:3836: oid 27 updating version 2992 to 2993 2024-09-11T01:55:14.941 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 2993 (ObjNum 1317 snap 429 seq_num 1317) dirty exists 2024-09-11T01:55:14.941 INFO:tasks.rados.rados.0.smithi071.stdout:3836: left oid 27 (ObjNum 1317 snap 429 seq_num 1317) 2024-09-11T01:55:14.941 INFO:tasks.rados.rados.0.smithi071.stdout:3836: done (3 left) 2024-09-11T01:55:14.941 INFO:tasks.rados.rados.0.smithi071.stdout:3840: delete oid 50 current snap is 429 2024-09-11T01:55:14.942 INFO:tasks.rados.rados.0.smithi071.stdout:3837: finishing write tid 1 to smithi07134777-19 2024-09-11T01:55:14.942 INFO:tasks.rados.rados.0.smithi071.stdout:3839: finishing copy_from to smithi07134777-21 2024-09-11T01:55:14.943 INFO:tasks.rados.rados.0.smithi071.stdout:3839: got expected ENOENT (src dne) 2024-09-11T01:55:14.943 INFO:tasks.rados.rados.0.smithi071.stdout:3839: finishing copy_from racing read to smithi07134777-21 2024-09-11T01:55:14.945 INFO:tasks.rados.rados.0.smithi071.stdout:3838: done (3 left) 2024-09-11T01:55:14.945 INFO:tasks.rados.rados.0.smithi071.stdout:3839: done (2 left) 2024-09-11T01:55:14.945 INFO:tasks.rados.rados.0.smithi071.stdout:3840: done (1 left) 2024-09-11T01:55:14.946 INFO:tasks.rados.rados.0.smithi071.stdout:3841: read oid 43 snap 325 2024-09-11T01:55:14.946 INFO:tasks.rados.rados.0.smithi071.stdout:3841: expect (ObjNum 1018 snap 323 seq_num 1018) 2024-09-11T01:55:14.946 INFO:tasks.rados.rados.0.smithi071.stdout:3842: delete oid 46 current snap is 429 2024-09-11T01:55:14.947 INFO:tasks.rados.rados.0.smithi071.stdout:3842: done (2 left) 2024-09-11T01:55:14.948 INFO:tasks.rados.rados.0.smithi071.stdout:3843: snap_create 2024-09-11T01:55:14.951 INFO:tasks.rados.rados.0.smithi071.stdout:3837: finishing write tid 2 to smithi07134777-19 2024-09-11T01:55:14.951 INFO:tasks.rados.rados.0.smithi071.stdout:3837: finishing write tid 3 to smithi07134777-19 2024-09-11T01:55:14.951 INFO:tasks.rados.rados.0.smithi071.stdout:3837: finishing write tid 4 to smithi07134777-19 2024-09-11T01:55:14.951 INFO:tasks.rados.rados.0.smithi071.stdout:3837: oid 19 updating version 0 to 2114 2024-09-11T01:55:14.951 INFO:tasks.rados.rados.0.smithi071.stdout:3837: oid 19 version 2114 is already newer than 2113 2024-09-11T01:55:14.951 INFO:tasks.rados.rados.0.smithi071.stdout:3837: oid 19 updating version 2114 to 2115 2024-09-11T01:55:14.951 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 2115 (ObjNum 1318 snap 429 seq_num 1318) dirty exists 2024-09-11T01:55:14.951 INFO:tasks.rados.rados.0.smithi071.stdout:3837: left oid 19 (ObjNum 1318 snap 429 seq_num 1318) 2024-09-11T01:55:14.955 INFO:tasks.rados.rados.0.smithi071.stdout:3841: expect (ObjNum 1018 snap 323 seq_num 1018) 2024-09-11T01:55:14.989 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2689013 2024-09-11T01:55:14.998 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 144886 2024-09-11T01:55:15.725 INFO:tasks.rados.rados.0.smithi071.stdout:3837: done (2 left) 2024-09-11T01:55:15.725 INFO:tasks.rados.rados.0.smithi071.stdout:3841: done (1 left) 2024-09-11T01:55:15.725 INFO:tasks.rados.rados.0.smithi071.stdout:3843: done (0 left) 2024-09-11T01:55:15.725 INFO:tasks.rados.rados.0.smithi071.stdout:3844: write oid 5 current snap is 430 2024-09-11T01:55:15.725 INFO:tasks.rados.rados.0.smithi071.stdout:3844: seq_num 1321 ranges {602047=732998,1801772=400275} 2024-09-11T01:55:15.732 INFO:tasks.rados.rados.0.smithi071.stdout:3844: writing smithi07134777-5 from 602047 to 1335045 tid 1 2024-09-11T01:55:15.737 INFO:tasks.rados.rados.0.smithi071.stdout:3844: writing smithi07134777-5 from 1801772 to 2202047 tid 2 2024-09-11T01:55:15.737 INFO:tasks.rados.rados.0.smithi071.stdout:3845: rollback oid 9 current snap is 430 2024-09-11T01:55:15.737 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 9 to 411 2024-09-11T01:55:15.737 INFO:tasks.rados.rados.0.smithi071.stdout:3846: write (excl) oid 26 current snap is 430 2024-09-11T01:55:15.737 INFO:tasks.rados.rados.0.smithi071.stdout:3846: seq_num 1322 ranges {618854=408247,1743901=425794} 2024-09-11T01:55:15.743 INFO:tasks.rados.rados.0.smithi071.stdout:3846: writing smithi07134777-26 from 618854 to 1027101 tid 1 2024-09-11T01:55:15.750 INFO:tasks.rados.rados.0.smithi071.stdout:3846: writing smithi07134777-26 from 1743901 to 2169695 tid 2 2024-09-11T01:55:15.750 INFO:tasks.rados.rados.0.smithi071.stdout:3847: read oid 20 snap 404 2024-09-11T01:55:15.750 INFO:tasks.rados.rados.0.smithi071.stdout:3847: expect (ObjNum 1216 snap 397 seq_num 1216) 2024-09-11T01:55:15.750 INFO:tasks.rados.rados.0.smithi071.stdout:3848: snap_create 2024-09-11T01:55:15.750 INFO:tasks.rados.rados.0.smithi071.stdout:3844: finishing write tid 1 to smithi07134777-5 2024-09-11T01:55:15.750 INFO:tasks.rados.rados.0.smithi071.stdout:3844: finishing write tid 2 to smithi07134777-5 2024-09-11T01:55:15.750 INFO:tasks.rados.rados.0.smithi071.stdout:3844: finishing write tid 3 to smithi07134777-5 2024-09-11T01:55:15.750 INFO:tasks.rados.rados.0.smithi071.stdout:3844: finishing write tid 4 to smithi07134777-5 2024-09-11T01:55:15.750 INFO:tasks.rados.rados.0.smithi071.stdout:3844: oid 5 updating version 0 to 1192 2024-09-11T01:55:15.750 INFO:tasks.rados.rados.0.smithi071.stdout:3844: oid 5 updating version 1192 to 1193 2024-09-11T01:55:15.750 INFO:tasks.rados.rados.0.smithi071.stdout:3844: oid 5 version 1193 is already newer than 1191 2024-09-11T01:55:15.751 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 1193 (ObjNum 1321 snap 430 seq_num 1321) dirty exists 2024-09-11T01:55:15.751 INFO:tasks.rados.rados.0.smithi071.stdout:3844: left oid 5 (ObjNum 1321 snap 430 seq_num 1321) 2024-09-11T01:55:15.751 INFO:tasks.rados.rados.0.smithi071.stdout:3846: finishing write tid 1 to smithi07134777-26 2024-09-11T01:55:15.751 INFO:tasks.rados.rados.0.smithi071.stdout:3845: finishing rollback tid 0 to smithi07134777-9 2024-09-11T01:55:15.751 INFO:tasks.rados.rados.0.smithi071.stdout:3845: finishing rollback tid 1 to smithi07134777-9 2024-09-11T01:55:15.751 INFO:tasks.rados.rados.0.smithi071.stdout:3845: finishing rollback tid 2 to smithi07134777-9 2024-09-11T01:55:15.751 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 2585 (ObjNum 832 snap 254 seq_num 832) dirty exists 2024-09-11T01:55:15.755 INFO:tasks.rados.rados.0.smithi071.stdout:3847: expect (ObjNum 1216 snap 397 seq_num 1216) 2024-09-11T01:55:15.780 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1404984 2024-09-11T01:55:15.839 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:55:15.886 INFO:tasks.rados.rados.0.smithi071.stdout:3846: finishing write tid 2 to smithi07134777-26 2024-09-11T01:55:15.886 INFO:tasks.rados.rados.0.smithi071.stdout:3846: finishing write tid 3 to smithi07134777-26 2024-09-11T01:55:15.887 INFO:tasks.rados.rados.0.smithi071.stdout:3846: finishing write tid 4 to smithi07134777-26 2024-09-11T01:55:15.887 INFO:tasks.rados.rados.0.smithi071.stdout:3846: oid 26 updating version 0 to 2996 2024-09-11T01:55:15.887 INFO:tasks.rados.rados.0.smithi071.stdout:3846: oid 26 version 2996 is already newer than 2994 2024-09-11T01:55:15.887 INFO:tasks.rados.rados.0.smithi071.stdout:3846: oid 26 version 2996 is already newer than 2995 2024-09-11T01:55:15.887 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 2996 (ObjNum 1322 snap 430 seq_num 1322) dirty exists 2024-09-11T01:55:15.887 INFO:tasks.rados.rados.0.smithi071.stdout:3846: left oid 26 (ObjNum 1322 snap 430 seq_num 1322) 2024-09-11T01:55:16.741 INFO:tasks.rados.rados.0.smithi071.stdout:3844: done (4 left) 2024-09-11T01:55:16.741 INFO:tasks.rados.rados.0.smithi071.stdout:3845: done (3 left) 2024-09-11T01:55:16.742 INFO:tasks.rados.rados.0.smithi071.stdout:3846: done (2 left) 2024-09-11T01:55:16.742 INFO:tasks.rados.rados.0.smithi071.stdout:3847: done (1 left) 2024-09-11T01:55:16.742 INFO:tasks.rados.rados.0.smithi071.stdout:3848: done (0 left) 2024-09-11T01:55:16.742 INFO:tasks.rados.rados.0.smithi071.stdout:3849: snap_create 2024-09-11T01:55:17.744 INFO:tasks.rados.rados.0.smithi071.stdout:3849: done (0 left) 2024-09-11T01:55:17.744 INFO:tasks.rados.rados.0.smithi071.stdout:3850: rollback oid 11 current snap is 432 2024-09-11T01:55:17.744 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 11 to 417 2024-09-11T01:55:17.744 INFO:tasks.rados.rados.0.smithi071.stdout:3851: write (excl) oid 8 current snap is 432 2024-09-11T01:55:17.744 INFO:tasks.rados.rados.0.smithi071.stdout:3851: seq_num 1323 ranges {635661=599849,1918736=316925} 2024-09-11T01:55:17.750 INFO:tasks.rados.rados.0.smithi071.stdout:3851: writing smithi07134777-8 from 635661 to 1235510 tid 1 2024-09-11T01:55:17.756 INFO:tasks.rados.rados.0.smithi071.stdout:3851: writing smithi07134777-8 from 1918736 to 2235661 tid 2 2024-09-11T01:55:17.756 INFO:tasks.rados.rados.0.smithi071.stdout:3852: write oid 36 current snap is 432 2024-09-11T01:55:17.756 INFO:tasks.rados.rados.0.smithi071.stdout:3850: finishing rollback tid 0 to smithi07134777-11 2024-09-11T01:55:17.756 INFO:tasks.rados.rados.0.smithi071.stdout:3850: finishing rollback tid 1 to smithi07134777-11 2024-09-11T01:55:17.756 INFO:tasks.rados.rados.0.smithi071.stdout:3850: finishing rollback tid 2 to smithi07134777-11 2024-09-11T01:55:17.756 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 2220 (ObjNum 1115 snap 356 seq_num 1115) dirty exists 2024-09-11T01:55:17.756 INFO:tasks.rados.rados.0.smithi071.stdout:3851: finishing write tid 1 to smithi07134777-8 2024-09-11T01:55:17.756 INFO:tasks.rados.rados.0.smithi071.stdout:3852: seq_num 1324 ranges {652468=675098,1977218=275250} 2024-09-11T01:55:17.763 INFO:tasks.rados.rados.0.smithi071.stdout:3852: writing smithi07134777-36 from 652468 to 1327566 tid 1 2024-09-11T01:55:17.768 INFO:tasks.rados.rados.0.smithi071.stdout:3852: writing smithi07134777-36 from 1977218 to 2252468 tid 2 2024-09-11T01:55:17.768 INFO:tasks.rados.rados.0.smithi071.stdout:3850: done (2 left) 2024-09-11T01:55:17.768 INFO:tasks.rados.rados.0.smithi071.stdout:3853: snap_remove snap 426 2024-09-11T01:55:19.680 INFO:tasks.rados.rados.0.smithi071.stdout:3853: done (2 left) 2024-09-11T01:55:19.680 INFO:tasks.rados.rados.0.smithi071.stdout:3854: snap_remove snap 385 2024-09-11T01:55:19.949 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-11T01:55:19.950 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T01:55:19.950 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T01:55:19.953 INFO:tasks.ceph.mon.a:Started 2024-09-11T01:55:19.953 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:55:19.953 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:55:20.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.063+0000 7f61d11f3640 10 monclient: get_monmap_and_config 2024-09-11T01:55:20.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.063+0000 7f61d11f3640 10 monclient: build_initial_monmap 2024-09-11T01:55:20.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.063+0000 7f61d11f3640 10 monclient: monmap: 2024-09-11T01:55:20.065 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:20.065 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:20.065 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:55:20.064436+0000 2024-09-11T01:55:20.065 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:55:20.064436+0000 2024-09-11T01:55:20.065 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:20.066 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:20.066 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:55:20.066 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:55:20.066 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:55:20.066 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:55:20.066 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:55:20.066 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:20.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.064+0000 7f61d11f3640 1 Processor -- start 2024-09-11T01:55:20.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.064+0000 7f61d11f3640 1 -- start start 2024-09-11T01:55:20.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.064+0000 7f61d11f3640 10 monclient: init 2024-09-11T01:55:20.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.064+0000 7f61d11f3640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:20.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.064+0000 7f61d11f3640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-11T01:55:20.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.064+0000 7f61d11f3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f61cc0a2e90 addr v1:172.21.15.12:6789/0 2024-09-11T01:55:20.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.064+0000 7f61d11f3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f61cc0a3270 addr v1:172.21.15.71:6789/0 2024-09-11T01:55:20.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.064+0000 7f61d11f3640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f61cc13e4a0 0x7f61cc140890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:20.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.065+0000 7f61d11f3640 10 monclient(hunting): picked mon.noname-c con 0x7f61cc13e4a0 addr v2:172.21.15.71:3300/0 2024-09-11T01:55:20.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.065+0000 7f61d11f3640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61cc0a46c0 con 0x7f61cc0a2e90 2024-09-11T01:55:20.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.065+0000 7f61d11f3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61cc06ee50 con 0x7f61cc0a3270 2024-09-11T01:55:20.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.065+0000 7f61d11f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:20.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.065+0000 7f61d11f3640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f61cc0a4900 con 0x7f61cc13e4a0 2024-09-11T01:55:20.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.065+0000 7f61d11f3640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:20.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.065+0000 7f61d11f3640 10 monclient(hunting): authenticate will time out at 2326.507061s 2024-09-11T01:55:20.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.065+0000 7f61cad9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f61cc0a2e90 0x7f61cc06dc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44088/0 (socket says 172.21.15.12:44088) 2024-09-11T01:55:20.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.065+0000 7f61cad9d640 1 -- 172.21.15.12:0/2141089347 learned_addr learned my addr 172.21.15.12:0/2141089347 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:55:20.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.065+0000 7f61cb59e640 1 --2- 172.21.15.12:0/2141089347 >> v2:172.21.15.71:3300/0 conn(0x7f61cc13e4a0 0x7f61cc140890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:20.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.065+0000 7f61cb59e640 10 monclient(hunting): get_auth_request con 0x7f61cc13e4a0 auth_method 0 2024-09-11T01:55:20.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.065+0000 7f61cb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:20.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.065+0000 7f61cb59e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:20.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.065+0000 7f61cb59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:20.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.065+0000 7f61cb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:20.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.065+0000 7f61cb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:20.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.065+0000 7f61cb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:20.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.065+0000 7f61cb59e640 10 monclient(hunting): handle_auth_done global_id 274249 payload 274 2024-09-11T01:55:20.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.065+0000 7f61cb59e640 1 -- 172.21.15.12:0/2141089347 >> v1:172.21.15.71:6789/0 conn(0x7f61cc0a3270 legacy=0x7f61cc0a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:55:20.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.065+0000 7f61cb59e640 1 -- 172.21.15.12:0/2141089347 >> v1:172.21.15.12:6789/0 conn(0x7f61cc0a2e90 legacy=0x7f61cc06dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:55:20.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.065+0000 7f61cb59e640 10 monclient: _finish_hunting 0 2024-09-11T01:55:20.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.065+0000 7f61cb59e640 1 monclient: found mon.noname-c 2024-09-11T01:55:20.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.065+0000 7f61cb59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:20.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.065+0000 7f61cb59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:55:20.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.065+0000 7f61cb59e640 1 -- 172.21.15.12:0/2141089347 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61cc06eb70 con 0x7f61cc13e4a0 2024-09-11T01:55:20.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.068+0000 7f61cb59e640 1 -- 172.21.15.12:0/2141089347 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f61cc13e4a0 msgr2=0x7f61cc140890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:20.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.068+0000 7f61cb59e640 1 --2- 172.21.15.12:0/2141089347 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f61cc13e4a0 0x7f61cc140890 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f61cc071820 tx=0x7f61c402a460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0b5801eb50a0ae3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:20.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.068+0000 7f61ca59c640 1 -- 172.21.15.12:0/2141089347 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c4007a30 con 0x7f61cc13e4a0 2024-09-11T01:55:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.068+0000 7f61ca59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.068+0000 7f61ca59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:55:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.068+0000 7f61ca59c640 10 monclient: dump: 2024-09-11T01:55:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:20.078 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:20.079 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:20.079 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:20.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.068+0000 7f61ca59c640 10 monclient: _finish_auth 0 2024-09-11T01:55:20.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.068+0000 7f61ca59c640 10 monclient: _check_auth_tickets 2024-09-11T01:55:20.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.068+0000 7f61ca59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:20.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.068+0000 7f61ca59c640 1 -- 172.21.15.12:0/2141089347 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61c4007bd0 con 0x7f61cc13e4a0 2024-09-11T01:55:20.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.068+0000 7f61ca59c640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:20.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.068+0000 7f61ca59c640 1 -- 172.21.15.12:0/2141089347 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c4031400 con 0x7f61cc13e4a0 2024-09-11T01:55:20.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.068+0000 7f61d11f3640 5 monclient: authenticate success, global_id 274249 2024-09-11T01:55:20.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.068+0000 7f61d11f3640 10 monclient: get_monmap_and_config success 2024-09-11T01:55:20.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.068+0000 7f61d11f3640 4 set_mon_vals no callback set 2024-09-11T01:55:20.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.068+0000 7f61ca59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:20.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.068+0000 7f61ca59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:55:20.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.068+0000 7f61ca59c640 10 monclient: dump: 2024-09-11T01:55:20.080 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:20.080 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:20.080 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:20.080 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:20.080 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:20.080 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:20.081 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:20.081 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:20.081 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:20.081 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:20.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.069+0000 7f61d11f3640 10 monclient: shutdown 2024-09-11T01:55:20.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.069+0000 7f61d11f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:20.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.069+0000 7f61d11f3640 1 -- 172.21.15.12:0/2141089347 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f61cc13e4a0 msgr2=0x7f61cc140890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:20.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.069+0000 7f61d11f3640 1 --2- 172.21.15.12:0/2141089347 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f61cc13e4a0 0x7f61cc140890 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f61cc071820 tx=0x7f61c402a460 comp rx=0 tx=0).stop 2024-09-11T01:55:20.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.069+0000 7f61d11f3640 1 -- 172.21.15.12:0/2141089347 shutdown_connections 2024-09-11T01:55:20.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.069+0000 7f61d11f3640 1 --2- 172.21.15.12:0/2141089347 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f61cc13e4a0 0x7f61cc140890 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:20.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.069+0000 7f61d11f3640 1 -- 172.21.15.12:0/2141089347 >> 172.21.15.12:0/2141089347 conn(0x7f61cc0a8490 msgr2=0x7f61cc0a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:20.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.069+0000 7f61d11f3640 1 -- 172.21.15.12:0/2141089347 shutdown_connections 2024-09-11T01:55:20.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.069+0000 7f61d11f3640 1 -- 172.21.15.12:0/2141089347 wait complete. 2024-09-11T01:55:20.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.070+0000 7f61d11f3640 10 monclient: build_initial_monmap 2024-09-11T01:55:20.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.070+0000 7f61d11f3640 10 monclient: monmap: 2024-09-11T01:55:20.082 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:20.082 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:55:20.082 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:55:20.082 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:55:20.082 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:20.082 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:20.082 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:55:20.082 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:55:20.083 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:55:20.083 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:20.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.070+0000 7f61d11f3640 1 Processor -- start 2024-09-11T01:55:20.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.070+0000 7f61d11f3640 1 -- start start 2024-09-11T01:55:20.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.070+0000 7f61d11f3640 10 monclient: init 2024-09-11T01:55:20.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61d11f3640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:20.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61d11f3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:55:20.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61d11f3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f61cc0a2e90 0x7f61cc138490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:20.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61d11f3640 10 monclient(hunting): picked mon.noname-b con 0x7f61cc0a2e90 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:55:20.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61d11f3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61cc0a3270 0x7f61cc1389d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:20.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61d11f3640 10 monclient(hunting): picked mon.noname-a con 0x7f61cc0a3270 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:55:20.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61d11f3640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f61cc13e4a0 0x7f61cc139470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:20.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61d11f3640 10 monclient(hunting): picked mon.noname-c con 0x7f61cc13e4a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:20.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61d11f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:20.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61d11f3640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f61cc140f50 con 0x7f61cc0a3270 2024-09-11T01:55:20.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61d11f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:20.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61d11f3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f61cc140dd0 con 0x7f61cc0a2e90 2024-09-11T01:55:20.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61d11f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:20.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61d11f3640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f61cc1410d0 con 0x7f61cc13e4a0 2024-09-11T01:55:20.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61d11f3640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:20.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61d11f3640 10 monclient(hunting): authenticate will time out at 2326.513408s 2024-09-11T01:55:20.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cb59e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f61cc13e4a0 0x7f61cc139470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:20.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cb59e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f61cc13e4a0 0x7f61cc139470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:60748/0 (socket says 172.21.15.12:60748) 2024-09-11T01:55:20.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cb59e640 1 -- 172.21.15.12:0/1274264732 learned_addr learned my addr 172.21.15.12:0/1274264732 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:55:20.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cb59e640 10 monclient(hunting): get_auth_request con 0x7f61cc13e4a0 auth_method 0 2024-09-11T01:55:20.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:20.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cb59e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:20.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cb59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:20.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61c259c640 1 --2- 172.21.15.12:0/1274264732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61cc0a3270 0x7f61cc1389d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:20.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61c259c640 10 monclient(hunting): get_auth_request con 0x7f61cc0a3270 auth_method 0 2024-09-11T01:55:20.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61c259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:20.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61c259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:20.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61c259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:20.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cad9d640 1 --2- 172.21.15.12:0/1274264732 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f61cc0a2e90 0x7f61cc138490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:20.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cad9d640 10 monclient(hunting): get_auth_request con 0x7f61cc0a2e90 auth_method 0 2024-09-11T01:55:20.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:20.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:20.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:20.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61c259c640 1 -- 172.21.15.12:0/1274264732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61cc0a3270 msgr2=0x7f61cc1389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:55:20.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61c259c640 1 -- 172.21.15.12:0/1274264732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61cc0a3270 msgr2=0x7f61cc1389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:55:20.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:20.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:20.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61c259c640 1 --2- 172.21.15.12:0/1274264732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61cc0a3270 0x7f61cc1389d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:55:20.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:20.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61c259c640 1 --2- 172.21.15.12:0/1274264732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61cc0a3270 0x7f61cc1389d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:55:20.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:20.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:20.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:20.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cb59e640 10 monclient(hunting): handle_auth_done global_id 284183 payload 824 2024-09-11T01:55:20.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cb59e640 1 -- 172.21.15.12:0/1274264732 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f61cc0a2e90 msgr2=0x7f61cc138490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:20.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cb59e640 1 --2- 172.21.15.12:0/1274264732 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f61cc0a2e90 0x7f61cc138490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:20.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cb59e640 1 -- 172.21.15.12:0/1274264732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61cc0a3270 msgr2=0x7f61cc1389d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:55:20.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cb59e640 1 --2- 172.21.15.12:0/1274264732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61cc0a3270 0x7f61cc1389d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:20.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cb59e640 10 monclient: _finish_hunting 0 2024-09-11T01:55:20.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cb59e640 1 monclient: found mon.noname-c 2024-09-11T01:55:20.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cb59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:20.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cb59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:55:20.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.071+0000 7f61cb59e640 1 -- 172.21.15.12:0/1274264732 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61cc13cc00 con 0x7f61cc13e4a0 2024-09-11T01:55:20.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.074+0000 7f61cb59e640 1 -- 172.21.15.12:0/1274264732 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f61cc13e4a0 msgr2=0x7f61cc139470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:20.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.074+0000 7f61cb59e640 1 --2- 172.21.15.12:0/1274264732 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f61cc13e4a0 0x7f61cc139470 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f61c402aa50 tx=0x7f61c4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:20.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.074+0000 7f61c3fff640 1 -- 172.21.15.12:0/1274264732 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c4040070 con 0x7f61cc13e4a0 2024-09-11T01:55:20.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.074+0000 7f61c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:20.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.074+0000 7f61c3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:55:20.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.074+0000 7f61c3fff640 10 monclient: dump: 2024-09-11T01:55:20.088 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:20.088 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:20.088 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:20.089 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:20.089 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:20.089 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:20.089 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:20.089 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:20.089 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:20.089 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:20.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.074+0000 7f61c3fff640 10 monclient: _finish_auth 0 2024-09-11T01:55:20.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.074+0000 7f61c3fff640 10 monclient: _check_auth_tickets 2024-09-11T01:55:20.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.074+0000 7f61c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:20.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.074+0000 7f61d11f3640 5 monclient: authenticate success, global_id 284183 2024-09-11T01:55:20.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.074+0000 7f61c3fff640 1 -- 172.21.15.12:0/1274264732 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61c4045660 con 0x7f61cc13e4a0 2024-09-11T01:55:20.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.074+0000 7f61c3fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:20.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.074+0000 7f61c3fff640 1 -- 172.21.15.12:0/1274264732 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c4045a30 con 0x7f61cc13e4a0 2024-09-11T01:55:20.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.074+0000 7f61c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:20.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.074+0000 7f61c3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:55:20.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.074+0000 7f61c3fff640 10 monclient: dump: 2024-09-11T01:55:20.090 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:20.090 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:20.090 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:20.090 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:20.090 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:20.090 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:20.090 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:20.091 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:20.091 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:20.091 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:20.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.074+0000 7f61d11f3640 10 monclient: _renew_subs 2024-09-11T01:55:20.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.074+0000 7f61d11f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:55:20.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.074+0000 7f61d11f3640 1 -- 172.21.15.12:0/1274264732 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61cc13bf40 con 0x7f61cc13e4a0 2024-09-11T01:55:20.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.074+0000 7f61c9d9b640 4 set_mon_vals no callback set 2024-09-11T01:55:20.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.074+0000 7f61d11f3640 10 monclient: _renew_subs 2024-09-11T01:55:20.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.074+0000 7f61d11f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:55:20.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.074+0000 7f61d11f3640 1 -- 172.21.15.12:0/1274264732 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f61cc09efa0 con 0x7f61cc13e4a0 2024-09-11T01:55:20.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.075+0000 7f61c3fff640 1 -- 172.21.15.12:0/1274264732 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f61c4007a40 con 0x7f61cc13e4a0 2024-09-11T01:55:20.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.075+0000 7f61d11f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:20.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.075+0000 7f61d11f3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:20.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.075+0000 7f61d11f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:55:20.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.075+0000 7f61d11f3640 1 -- 172.21.15.12:0/1274264732 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61cc06dc60 con 0x7f61cc13e4a0 2024-09-11T01:55:20.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.075+0000 7f61c3fff640 1 --2- 172.21.15.12:0/1274264732 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f61ac03e210 0x7f61ac0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:20.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.075+0000 7f61c3fff640 1 -- 172.21.15.12:0/1274264732 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(879..879 src has 676..879) ==== 4815+0+0 (secure 0 0 0) 0x7f61c4074b80 con 0x7f61cc13e4a0 2024-09-11T01:55:20.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.076+0000 7f61cad9d640 1 --2- 172.21.15.12:0/1274264732 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f61ac03e210 0x7f61ac0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:20.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.076+0000 7f61cad9d640 10 monclient: get_auth_request con 0x7f61ac03e210 auth_method 0 2024-09-11T01:55:20.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.078+0000 7f61cad9d640 1 --2- 172.21.15.12:0/1274264732 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f61ac03e210 0x7f61ac0406d0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f61bc007c90 tx=0x7f61bc007be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:20.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.080+0000 7f61c3fff640 1 -- 172.21.15.12:0/1274264732 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f61cc13cc00 con 0x7f61cc13e4a0 2024-09-11T01:55:20.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.080+0000 7f61c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:20.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.080+0000 7f61c3fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:55:20.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.337+0000 7f61d11f3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:55:20.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.337+0000 7f61d11f3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:55:20.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.337+0000 7f61d11f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:55:20.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.337+0000 7f61d11f3640 1 -- 172.21.15.12:0/1274264732 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f61cc0a5e10 con 0x7f61cc13e4a0 2024-09-11T01:55:20.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.337+0000 7f61c3fff640 1 -- 172.21.15.12:0/1274264732 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f61c4073560 con 0x7f61cc13e4a0 2024-09-11T01:55:20.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.337+0000 7f61c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:55:20.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.337+0000 7f61c3fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:55:20.340 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":236,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:55:20.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.340+0000 7f61d11f3640 1 -- 172.21.15.12:0/1274264732 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f61ac03e210 msgr2=0x7f61ac0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:20.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.340+0000 7f61d11f3640 1 --2- 172.21.15.12:0/1274264732 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f61ac03e210 0x7f61ac0406d0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f61bc007c90 tx=0x7f61bc007be0 comp rx=0 tx=0).stop 2024-09-11T01:55:20.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.340+0000 7f61d11f3640 10 monclient: shutdown 2024-09-11T01:55:20.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.340+0000 7f61d11f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:20.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.340+0000 7f61d11f3640 1 -- 172.21.15.12:0/1274264732 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f61cc13e4a0 msgr2=0x7f61cc139470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:20.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.340+0000 7f61d11f3640 1 --2- 172.21.15.12:0/1274264732 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f61cc13e4a0 0x7f61cc139470 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f61c402aa50 tx=0x7f61c4004a40 comp rx=0 tx=0).stop 2024-09-11T01:55:20.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.340+0000 7f61d11f3640 1 -- 172.21.15.12:0/1274264732 shutdown_connections 2024-09-11T01:55:20.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.340+0000 7f61d11f3640 1 --2- 172.21.15.12:0/1274264732 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f61cc13e4a0 0x7f61cc139470 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:20.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.340+0000 7f61d11f3640 1 --2- 172.21.15.12:0/1274264732 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f61ac03e210 0x7f61ac0406d0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:20.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.340+0000 7f61d11f3640 1 --2- 172.21.15.12:0/1274264732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61cc0a3270 0x7f61cc1389d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:20.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.340+0000 7f61d11f3640 1 --2- 172.21.15.12:0/1274264732 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f61cc0a2e90 0x7f61cc138490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:20.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.340+0000 7f61d11f3640 1 -- 172.21.15.12:0/1274264732 >> 172.21.15.12:0/1274264732 conn(0x7f61cc0a8490 msgr2=0x7f61cc0a7920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:20.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.340+0000 7f61d11f3640 1 -- 172.21.15.12:0/1274264732 shutdown_connections 2024-09-11T01:55:20.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:20.340+0000 7f61d11f3640 1 -- 172.21.15.12:0/1274264732 wait complete. 2024-09-11T01:55:20.679 INFO:tasks.rados.rados.0.smithi071.stdout:3854: done (2 left) 2024-09-11T01:55:20.679 INFO:tasks.rados.rados.0.smithi071.stdout:3855: snap_remove snap 419 2024-09-11T01:55:20.727 INFO:tasks.rados.rados.0.smithi071.stdout:3855: done (2 left) 2024-09-11T01:55:20.727 INFO:tasks.rados.rados.0.smithi071.stdout:3856: delete oid 1 current snap is 432 2024-09-11T01:55:20.727 INFO:tasks.rados.rados.0.smithi071.stdout:3851: finishing write tid 2 to smithi07134777-8 2024-09-11T01:55:20.727 INFO:tasks.rados.rados.0.smithi071.stdout:3851: finishing write tid 3 to smithi07134777-8 2024-09-11T01:55:20.727 INFO:tasks.rados.rados.0.smithi071.stdout:3851: finishing write tid 4 to smithi07134777-8 2024-09-11T01:55:20.727 INFO:tasks.rados.rados.0.smithi071.stdout:3851: oid 8 updating version 0 to 2118 2024-09-11T01:55:20.728 INFO:tasks.rados.rados.0.smithi071.stdout:3851: oid 8 version 2118 is already newer than 2116 2024-09-11T01:55:20.728 INFO:tasks.rados.rados.0.smithi071.stdout:3851: oid 8 version 2118 is already newer than 2117 2024-09-11T01:55:20.728 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 2118 (ObjNum 1323 snap 432 seq_num 1323) dirty exists 2024-09-11T01:55:20.728 INFO:tasks.rados.rados.0.smithi071.stdout:3851: left oid 8 (ObjNum 1323 snap 432 seq_num 1323) 2024-09-11T01:55:20.728 INFO:tasks.rados.rados.0.smithi071.stdout:3852: finishing write tid 1 to smithi07134777-36 2024-09-11T01:55:20.728 INFO:tasks.rados.rados.0.smithi071.stdout:3852: finishing write tid 2 to smithi07134777-36 2024-09-11T01:55:20.728 INFO:tasks.rados.rados.0.smithi071.stdout:3852: finishing write tid 3 to smithi07134777-36 2024-09-11T01:55:20.728 INFO:tasks.rados.rados.0.smithi071.stdout:3852: finishing write tid 4 to smithi07134777-36 2024-09-11T01:55:20.728 INFO:tasks.rados.rados.0.smithi071.stdout:3852: oid 36 updating version 0 to 2119 2024-09-11T01:55:20.728 INFO:tasks.rados.rados.0.smithi071.stdout:3852: oid 36 updating version 2119 to 2121 2024-09-11T01:55:20.728 INFO:tasks.rados.rados.0.smithi071.stdout:3852: oid 36 version 2121 is already newer than 2120 2024-09-11T01:55:20.728 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 2121 (ObjNum 1324 snap 432 seq_num 1324) dirty exists 2024-09-11T01:55:20.728 INFO:tasks.rados.rados.0.smithi071.stdout:3852: left oid 36 (ObjNum 1324 snap 432 seq_num 1324) 2024-09-11T01:55:20.737 INFO:tasks.rados.rados.0.smithi071.stdout:3851: done (2 left) 2024-09-11T01:55:20.737 INFO:tasks.rados.rados.0.smithi071.stdout:3852: done (1 left) 2024-09-11T01:55:20.738 INFO:tasks.rados.rados.0.smithi071.stdout:3856: done (0 left) 2024-09-11T01:55:20.738 INFO:tasks.rados.rados.0.smithi071.stdout:3857: write oid 48 current snap is 432 2024-09-11T01:55:20.738 INFO:tasks.rados.rados.0.smithi071.stdout:3857: seq_num 1326 ranges {686082=425596,1577829=433132} 2024-09-11T01:55:20.744 INFO:tasks.rados.rados.0.smithi071.stdout:3857: writing smithi07134777-48 from 686082 to 1111678 tid 1 2024-09-11T01:55:20.750 INFO:tasks.rados.rados.0.smithi071.stdout:3857: writing smithi07134777-48 from 1577829 to 2010961 tid 2 2024-09-11T01:55:20.750 INFO:tasks.rados.rados.0.smithi071.stdout:3858: snap_remove snap 340 2024-09-11T01:55:21.748 INFO:tasks.rados.rados.0.smithi071.stdout:3858: done (1 left) 2024-09-11T01:55:21.749 INFO:tasks.rados.rados.0.smithi071.stdout:3859: read oid 28 snap -1 2024-09-11T01:55:21.749 INFO:tasks.rados.rados.0.smithi071.stdout:3859: expect deleted 2024-09-11T01:55:21.749 INFO:tasks.rados.rados.0.smithi071.stdout:3860: write (excl) oid 26 current snap is 432 2024-09-11T01:55:21.749 INFO:tasks.rados.rados.0.smithi071.stdout:3857: finishing write tid 1 to smithi07134777-48 2024-09-11T01:55:21.749 INFO:tasks.rados.rados.0.smithi071.stdout:3860: seq_num 1327 ranges {702889=500845,1636311=576790} 2024-09-11T01:55:21.755 INFO:tasks.rados.rados.0.smithi071.stdout:3860: writing smithi07134777-26 from 702889 to 1203734 tid 1 2024-09-11T01:55:21.761 INFO:tasks.rados.rados.0.smithi071.stdout:3860: writing smithi07134777-26 from 1636311 to 2213101 tid 2 2024-09-11T01:55:21.761 INFO:tasks.rados.rados.0.smithi071.stdout:3861: write (excl) oid 2 current snap is 432 2024-09-11T01:55:21.761 INFO:tasks.rados.rados.0.smithi071.stdout:3861: seq_num 1328 ranges {719696=576094,2094793=224903} 2024-09-11T01:55:21.768 INFO:tasks.rados.rados.0.smithi071.stdout:3861: writing smithi07134777-2 from 719696 to 1295790 tid 1 2024-09-11T01:55:21.774 INFO:tasks.rados.rados.0.smithi071.stdout:3861: writing smithi07134777-2 from 2094793 to 2319696 tid 2 2024-09-11T01:55:21.774 INFO:tasks.rados.rados.0.smithi071.stdout:3862: write oid 43 current snap is 432 2024-09-11T01:55:21.774 INFO:tasks.rados.rados.0.smithi071.stdout:3862: seq_num 1329 ranges {736503=651343,2153275=183228} 2024-09-11T01:55:21.782 INFO:tasks.rados.rados.0.smithi071.stdout:3862: writing smithi07134777-43 from 736503 to 1387846 tid 1 2024-09-11T01:55:21.788 INFO:tasks.rados.rados.0.smithi071.stdout:3862: writing smithi07134777-43 from 2153275 to 2336503 tid 2 2024-09-11T01:55:21.788 INFO:tasks.rados.rados.0.smithi071.stdout:3863: snap_remove snap 417 2024-09-11T01:55:22.753 INFO:tasks.rados.rados.0.smithi071.stdout:3863: done (5 left) 2024-09-11T01:55:22.753 INFO:tasks.rados.rados.0.smithi071.stdout:3864: delete oid 44 current snap is 432 2024-09-11T01:55:22.753 INFO:tasks.rados.rados.0.smithi071.stdout:3857: finishing write tid 2 to smithi07134777-48 2024-09-11T01:55:22.753 INFO:tasks.rados.rados.0.smithi071.stdout:3857: finishing write tid 3 to smithi07134777-48 2024-09-11T01:55:22.753 INFO:tasks.rados.rados.0.smithi071.stdout:3857: finishing write tid 4 to smithi07134777-48 2024-09-11T01:55:22.753 INFO:tasks.rados.rados.0.smithi071.stdout:3857: oid 48 updating version 0 to 3098 2024-09-11T01:55:22.754 INFO:tasks.rados.rados.0.smithi071.stdout:3857: oid 48 version 3098 is already newer than 3096 2024-09-11T01:55:22.754 INFO:tasks.rados.rados.0.smithi071.stdout:3857: oid 48 version 3098 is already newer than 3097 2024-09-11T01:55:22.754 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 3098 (ObjNum 1326 snap 432 seq_num 1326) dirty exists 2024-09-11T01:55:22.754 INFO:tasks.rados.rados.0.smithi071.stdout:3857: left oid 48 (ObjNum 1326 snap 432 seq_num 1326) 2024-09-11T01:55:22.754 INFO:tasks.rados.rados.0.smithi071.stdout:3860: finishing write tid 1 to smithi07134777-26 2024-09-11T01:55:22.754 INFO:tasks.rados.rados.0.smithi071.stdout:3860: finishing write tid 2 to smithi07134777-26 2024-09-11T01:55:22.754 INFO:tasks.rados.rados.0.smithi071.stdout:3860: finishing write tid 3 to smithi07134777-26 2024-09-11T01:55:22.754 INFO:tasks.rados.rados.0.smithi071.stdout:3860: finishing write tid 4 to smithi07134777-26 2024-09-11T01:55:22.754 INFO:tasks.rados.rados.0.smithi071.stdout:3860: oid 26 updating version 0 to 3025 2024-09-11T01:55:22.754 INFO:tasks.rados.rados.0.smithi071.stdout:3860: oid 26 version 3025 is already newer than 3024 2024-09-11T01:55:22.754 INFO:tasks.rados.rados.0.smithi071.stdout:3860: oid 26 updating version 3025 to 3026 2024-09-11T01:55:22.754 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 3026 (ObjNum 1327 snap 432 seq_num 1327) dirty exists 2024-09-11T01:55:22.754 INFO:tasks.rados.rados.0.smithi071.stdout:3860: left oid 26 (ObjNum 1327 snap 432 seq_num 1327) 2024-09-11T01:55:22.754 INFO:tasks.rados.rados.0.smithi071.stdout:3861: finishing write tid 1 to smithi07134777-2 2024-09-11T01:55:22.755 INFO:tasks.rados.rados.0.smithi071.stdout:3861: finishing write tid 2 to smithi07134777-2 2024-09-11T01:55:22.755 INFO:tasks.rados.rados.0.smithi071.stdout:3861: finishing write tid 3 to smithi07134777-2 2024-09-11T01:55:22.755 INFO:tasks.rados.rados.0.smithi071.stdout:3861: finishing write tid 4 to smithi07134777-2 2024-09-11T01:55:22.755 INFO:tasks.rados.rados.0.smithi071.stdout:3861: oid 2 updating version 0 to 2552 2024-09-11T01:55:22.755 INFO:tasks.rados.rados.0.smithi071.stdout:3861: oid 2 version 2552 is already newer than 2551 2024-09-11T01:55:22.755 INFO:tasks.rados.rados.0.smithi071.stdout:3861: oid 2 updating version 2552 to 2553 2024-09-11T01:55:22.755 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 2553 (ObjNum 1328 snap 432 seq_num 1328) dirty exists 2024-09-11T01:55:22.755 INFO:tasks.rados.rados.0.smithi071.stdout:3861: left oid 2 (ObjNum 1328 snap 432 seq_num 1328) 2024-09-11T01:55:22.755 INFO:tasks.rados.rados.0.smithi071.stdout:3862: finishing write tid 1 to smithi07134777-43 2024-09-11T01:55:22.755 INFO:tasks.rados.rados.0.smithi071.stdout:3862: finishing write tid 2 to smithi07134777-43 2024-09-11T01:55:22.755 INFO:tasks.rados.rados.0.smithi071.stdout:3862: finishing write tid 3 to smithi07134777-43 2024-09-11T01:55:22.755 INFO:tasks.rados.rados.0.smithi071.stdout:3862: finishing write tid 4 to smithi07134777-43 2024-09-11T01:55:22.755 INFO:tasks.rados.rados.0.smithi071.stdout:3862: oid 43 updating version 0 to 3041 2024-09-11T01:55:22.755 INFO:tasks.rados.rados.0.smithi071.stdout:3862: oid 43 updating version 3041 to 3042 2024-09-11T01:55:22.755 INFO:tasks.rados.rados.0.smithi071.stdout:3862: oid 43 version 3042 is already newer than 3040 2024-09-11T01:55:22.755 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 3042 (ObjNum 1329 snap 432 seq_num 1329) dirty exists 2024-09-11T01:55:22.756 INFO:tasks.rados.rados.0.smithi071.stdout:3862: left oid 43 (ObjNum 1329 snap 432 seq_num 1329) 2024-09-11T01:55:22.763 INFO:tasks.rados.rados.0.smithi071.stdout:3857: done (5 left) 2024-09-11T01:55:22.763 INFO:tasks.rados.rados.0.smithi071.stdout:3859: done (4 left) 2024-09-11T01:55:22.763 INFO:tasks.rados.rados.0.smithi071.stdout:3860: done (3 left) 2024-09-11T01:55:22.763 INFO:tasks.rados.rados.0.smithi071.stdout:3861: done (2 left) 2024-09-11T01:55:22.763 INFO:tasks.rados.rados.0.smithi071.stdout:3862: done (1 left) 2024-09-11T01:55:22.763 INFO:tasks.rados.rados.0.smithi071.stdout:3864: done (0 left) 2024-09-11T01:55:22.763 INFO:tasks.rados.rados.0.smithi071.stdout:3865: snap_create 2024-09-11T01:55:23.354 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:55:23.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.471+0000 7ff879e52640 10 monclient: get_monmap_and_config 2024-09-11T01:55:23.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.471+0000 7ff879e52640 10 monclient: build_initial_monmap 2024-09-11T01:55:23.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.471+0000 7ff879e52640 10 monclient: monmap: 2024-09-11T01:55:23.473 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:23.473 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:23.473 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:55:23.472102+0000 2024-09-11T01:55:23.473 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:55:23.472102+0000 2024-09-11T01:55:23.473 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:23.473 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:23.473 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:55:23.474 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:55:23.474 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:55:23.474 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:55:23.474 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:55:23.474 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:23.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.472+0000 7ff879e52640 1 Processor -- start 2024-09-11T01:55:23.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.472+0000 7ff879e52640 1 -- start start 2024-09-11T01:55:23.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.472+0000 7ff879e52640 10 monclient: init 2024-09-11T01:55:23.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.472+0000 7ff879e52640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:23.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.472+0000 7ff879e52640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-11T01:55:23.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.472+0000 7ff879e52640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff8740a2db0 0x7ff87406cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:23.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.472+0000 7ff879e52640 10 monclient(hunting): picked mon.noname-a con 0x7ff8740a2db0 addr v2:172.21.15.12:3300/0 2024-09-11T01:55:23.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.472+0000 7ff879e52640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff87409fbd0 addr v1:172.21.15.71:6789/0 2024-09-11T01:55:23.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.472+0000 7ff879e52640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff87413d550 addr v1:172.21.15.12:6789/0 2024-09-11T01:55:23.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.472+0000 7ff879e52640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff87409eea0 con 0x7ff87413d550 2024-09-11T01:55:23.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.472+0000 7ff879e52640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff87409f180 con 0x7ff87409fbd0 2024-09-11T01:55:23.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.472+0000 7ff879e52640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:23.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.472+0000 7ff879e52640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff8740a3850 con 0x7ff8740a2db0 2024-09-11T01:55:23.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.472+0000 7ff879e52640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:23.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.472+0000 7ff879e52640 10 monclient(hunting): authenticate will time out at 2329.914850s 2024-09-11T01:55:23.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.472+0000 7ff8737fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff8740a2db0 0x7ff87406cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:23.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.473+0000 7ff873fff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff87413d550 0x7ff87413f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:44102/0 (socket says 172.21.15.12:44102) 2024-09-11T01:55:23.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.473+0000 7ff873fff640 1 -- 172.21.15.12:0/1345003005 learned_addr learned my addr 172.21.15.12:0/1345003005 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:55:23.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.473+0000 7ff8737fe640 10 monclient(hunting): get_auth_request con 0x7ff8740a2db0 auth_method 0 2024-09-11T01:55:23.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.473+0000 7ff8737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:23.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.473+0000 7ff8737fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:23.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.473+0000 7ff8737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:23.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.473+0000 7ff8737fe640 1 -- 172.21.15.12:0/1345003005 >> v2:172.21.15.12:3300/0 conn(0x7ff8740a2db0 msgr2=0x7ff87406cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:55:23.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.473+0000 7ff8737fe640 1 -- 172.21.15.12:0/1345003005 >> v2:172.21.15.12:3300/0 conn(0x7ff8740a2db0 msgr2=0x7ff87406cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:55:23.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.473+0000 7ff8737fe640 1 --2- 172.21.15.12:0/1345003005 >> v2:172.21.15.12:3300/0 conn(0x7ff8740a2db0 0x7ff87406cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:55:23.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.473+0000 7ff8737fe640 1 --2- 172.21.15.12:0/1345003005 >> v2:172.21.15.12:3300/0 conn(0x7ff8740a2db0 0x7ff87406cb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:55:23.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff873fff640 1 -- 172.21.15.12:0/1345003005 >> v1:172.21.15.12:6789/0 conn(0x7ff87413d550 legacy=0x7ff87413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:23.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff872ffd640 1 -- 172.21.15.12:0/1345003005 >> v1:172.21.15.71:6789/0 conn(0x7ff87409fbd0 legacy=0x7ff87406d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:23.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 1 -- 172.21.15.12:0/1345003005 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3911717812 0 0) 0x7ff87409f180 con 0x7ff87409fbd0 2024-09-11T01:55:23.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 10 monclient(hunting): _negotiate 2024-09-11T01:55:23.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:23.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:23.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 10 monclient(hunting): my global_id is 274255 2024-09-11T01:55:23.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 1 -- 172.21.15.12:0/1345003005 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff85c003520 con 0x7ff87409fbd0 2024-09-11T01:55:23.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 1 -- 172.21.15.12:0/1345003005 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2902109604 0 0) 0x7ff85c003520 con 0x7ff87409fbd0 2024-09-11T01:55:23.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 1 -- 172.21.15.12:0/1345003005 >> v2:172.21.15.12:3300/0 conn(0x7ff8740a2db0 msgr2=0x7ff87406cb50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:55:23.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 1 --2- 172.21.15.12:0/1345003005 >> v2:172.21.15.12:3300/0 conn(0x7ff8740a2db0 0x7ff87406cb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:23.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 1 -- 172.21.15.12:0/1345003005 >> v1:172.21.15.12:6789/0 conn(0x7ff87413d550 legacy=0x7ff87413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:23.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 10 monclient: _finish_hunting 0 2024-09-11T01:55:23.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 1 monclient: found mon.noname-c-legacy 2024-09-11T01:55:23.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:23.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-09-11T01:55:23.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 1 -- 172.21.15.12:0/1345003005 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8740a4ce0 con 0x7ff87409fbd0 2024-09-11T01:55:23.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 10 monclient: _finish_auth 0 2024-09-11T01:55:23.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 10 monclient: _check_auth_tickets 2024-09-11T01:55:23.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:23.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 1 -- 172.21.15.12:0/1345003005 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7ff8680019f0 con 0x7ff87409fbd0 2024-09-11T01:55:23.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff879e52640 5 monclient: authenticate success, global_id 274255 2024-09-11T01:55:23.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff879e52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:55:23.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:23.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-11T01:55:23.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 10 monclient: dump: 2024-09-11T01:55:23.484 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:23.484 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:23.484 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:23.484 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:23.485 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:23.485 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:23.485 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:23.485 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:23.485 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:23.485 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:23.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-09-11T01:55:23.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:23.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 1 -- 172.21.15.12:0/1345003005 >> v1:172.21.15.71:6789/0 conn(0x7ff87409fbd0 legacy=0x7ff87406d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:23.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:55:23.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.475+0000 7ff8727fc640 1 --2- 172.21.15.12:0/1345003005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff85c003b20 0x7ff85c003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:23.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff8727fc640 10 monclient(hunting): picked mon.a con 0x7ff85c003b20 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:55:23.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff8727fc640 1 --2- 172.21.15.12:0/1345003005 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff85c004800 0x7ff85c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:23.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff8727fc640 10 monclient(hunting): picked mon.b con 0x7ff85c004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:55:23.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff8727fc640 1 --2- 172.21.15.12:0/1345003005 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff85c0053f0 0x7ff85c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:23.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff8727fc640 10 monclient(hunting): picked mon.c con 0x7ff85c0053f0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:23.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff8727fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:23.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff8727fc640 1 -- 172.21.15.12:0/1345003005 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff85c00c310 con 0x7ff85c003b20 2024-09-11T01:55:23.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff8727fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:23.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff8727fc640 1 -- 172.21.15.12:0/1345003005 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff85c00c4c0 con 0x7ff85c004800 2024-09-11T01:55:23.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff8727fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:23.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff8727fc640 1 -- 172.21.15.12:0/1345003005 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff85c00c6c0 con 0x7ff85c0053f0 2024-09-11T01:55:23.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff8727fc640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:23.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff8727fc640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:55:23.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff879e52640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:55:23.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff8737fe640 1 --2- 172.21.15.12:0/1345003005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff85c003b20 0x7ff85c003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:23.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff8737fe640 10 monclient(hunting): get_auth_request con 0x7ff85c003b20 auth_method 0 2024-09-11T01:55:23.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff8737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:23.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff8737fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:23.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff8737fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:55:23.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff873fff640 1 --2- 172.21.15.12:0/1345003005 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff85c0053f0 0x7ff85c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:23.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff873fff640 10 monclient(hunting): get_auth_request con 0x7ff85c0053f0 auth_method 0 2024-09-11T01:55:23.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff873fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:23.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff873fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:23.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff873fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:55:23.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff872ffd640 1 --2- 172.21.15.12:0/1345003005 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff85c004800 0x7ff85c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:23.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff8737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:23.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff8737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:23.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff8737fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:55:23.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff872ffd640 10 monclient(hunting): get_auth_request con 0x7ff85c004800 auth_method 0 2024-09-11T01:55:23.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff872ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:23.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff872ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:23.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff872ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:55:23.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff873fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:23.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff873fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:23.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff873fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:55:23.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff872ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:23.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff872ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:23.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff872ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:55:23.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff873fff640 10 monclient(hunting): handle_auth_done global_id 274255 payload 293 2024-09-11T01:55:23.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff873fff640 1 -- 172.21.15.12:0/1345003005 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff85c004800 msgr2=0x7ff85c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:55:23.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff873fff640 1 --2- 172.21.15.12:0/1345003005 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff85c004800 0x7ff85c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:23.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff873fff640 1 -- 172.21.15.12:0/1345003005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff85c003b20 msgr2=0x7ff85c003f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:55:23.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff873fff640 1 --2- 172.21.15.12:0/1345003005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff85c003b20 0x7ff85c003f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:23.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff873fff640 10 monclient: _finish_hunting 0 2024-09-11T01:55:23.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff873fff640 1 monclient: found mon.c 2024-09-11T01:55:23.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff873fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:23.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff873fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:55:23.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff873fff640 1 -- 172.21.15.12:0/1345003005 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff85c003520 con 0x7ff85c0053f0 2024-09-11T01:55:23.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff873fff640 10 monclient: _finish_auth 0 2024-09-11T01:55:23.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff873fff640 10 monclient: _check_auth_tickets 2024-09-11T01:55:23.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff873fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:23.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.476+0000 7ff8737fe640 1 --2- 172.21.15.12:0/1345003005 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff85c003b20 0x7ff85c003f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:55:23.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.477+0000 7ff873fff640 1 -- 172.21.15.12:0/1345003005 reap_dead start 2024-09-11T01:55:23.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.479+0000 7ff873fff640 1 -- 172.21.15.12:0/1345003005 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff85c0053f0 msgr2=0x7ff85c00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:23.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.479+0000 7ff873fff640 1 --2- 172.21.15.12:0/1345003005 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff85c0053f0 0x7ff85c00ba60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff86c00c360 tx=0x7ff85c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64941f0a5eb4c64d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:23.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.479+0000 7ff8727fc640 1 -- 172.21.15.12:0/1345003005 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86c035070 con 0x7ff85c0053f0 2024-09-11T01:55:23.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.479+0000 7ff8727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:23.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.479+0000 7ff8727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:55:23.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.479+0000 7ff8727fc640 10 monclient: dump: 2024-09-11T01:55:23.497 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:23.497 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:23.497 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:23.497 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:23.497 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:23.497 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:23.497 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:23.498 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:23.498 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:23.498 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:23.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.479+0000 7ff879e52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:55:23.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.479+0000 7ff8727fc640 1 -- 172.21.15.12:0/1345003005 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff86c030600 con 0x7ff85c0053f0 2024-09-11T01:55:23.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.479+0000 7ff8727fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:23.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.479+0000 7ff879e52640 10 monclient: get_monmap_and_config success 2024-09-11T01:55:23.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.479+0000 7ff879e52640 4 set_mon_vals no callback set 2024-09-11T01:55:23.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.479+0000 7ff8727fc640 1 -- 172.21.15.12:0/1345003005 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86400b5e0 con 0x7ff85c0053f0 2024-09-11T01:55:23.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.479+0000 7ff8727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:23.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.479+0000 7ff8727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:55:23.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.479+0000 7ff8727fc640 10 monclient: dump: 2024-09-11T01:55:23.498 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:23.499 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:23.499 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:23.499 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:23.499 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:23.499 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:23.499 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:23.499 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:23.499 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:23.499 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:23.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.480+0000 7ff879e52640 10 monclient: shutdown 2024-09-11T01:55:23.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.480+0000 7ff879e52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:23.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.480+0000 7ff879e52640 1 -- 172.21.15.12:0/1345003005 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff85c0053f0 msgr2=0x7ff85c00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:23.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.480+0000 7ff879e52640 1 --2- 172.21.15.12:0/1345003005 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff85c0053f0 0x7ff85c00ba60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff86c00c360 tx=0x7ff85c005830 comp rx=0 tx=0).stop 2024-09-11T01:55:23.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.480+0000 7ff879e52640 1 -- 172.21.15.12:0/1345003005 shutdown_connections 2024-09-11T01:55:23.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.480+0000 7ff879e52640 1 --2- 172.21.15.12:0/1345003005 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff85c0053f0 0x7ff85c00ba60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:23.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.480+0000 7ff879e52640 1 -- 172.21.15.12:0/1345003005 >> 172.21.15.12:0/1345003005 conn(0x7ff8740a3ea0 msgr2=0x7ff8740a42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:23.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.480+0000 7ff879e52640 1 -- 172.21.15.12:0/1345003005 shutdown_connections 2024-09-11T01:55:23.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.480+0000 7ff879e52640 1 -- 172.21.15.12:0/1345003005 wait complete. 2024-09-11T01:55:23.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.480+0000 7ff879e52640 10 monclient: build_initial_monmap 2024-09-11T01:55:23.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.480+0000 7ff879e52640 10 monclient: monmap: 2024-09-11T01:55:23.500 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:23.501 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:55:23.501 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:55:23.501 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:55:23.501 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:23.501 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:23.501 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:55:23.501 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:55:23.501 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:55:23.501 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:23.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff879e52640 1 Processor -- start 2024-09-11T01:55:23.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff879e52640 1 -- start start 2024-09-11T01:55:23.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff879e52640 10 monclient: init 2024-09-11T01:55:23.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff879e52640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:23.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff879e52640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:55:23.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff879e52640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff85c0053f0 0x7ff874099c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:23.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff879e52640 10 monclient(hunting): picked mon.noname-a con 0x7ff85c0053f0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:55:23.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff879e52640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff874093520 0x7ff8740939a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:23.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff879e52640 10 monclient(hunting): picked mon.noname-b con 0x7ff874093520 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:55:23.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff879e52640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff874093ee0 0x7ff8741d3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:23.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff879e52640 10 monclient(hunting): picked mon.noname-c con 0x7ff874093ee0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:23.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff879e52640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:23.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff879e52640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff87406cb50 con 0x7ff85c0053f0 2024-09-11T01:55:23.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff879e52640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:23.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff879e52640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff8740a3190 con 0x7ff874093520 2024-09-11T01:55:23.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff879e52640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:23.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff879e52640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff87406ccd0 con 0x7ff874093ee0 2024-09-11T01:55:23.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff879e52640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:23.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff879e52640 10 monclient(hunting): authenticate will time out at 2329.923832s 2024-09-11T01:55:23.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff8737fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff85c0053f0 0x7ff874099c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:23.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff873fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff874093ee0 0x7ff8741d3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:23.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff8737fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff85c0053f0 0x7ff874099c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48192/0 (socket says 172.21.15.12:48192) 2024-09-11T01:55:23.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff8737fe640 1 -- 172.21.15.12:0/1052290373 learned_addr learned my addr 172.21.15.12:0/1052290373 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:55:23.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff8737fe640 10 monclient(hunting): get_auth_request con 0x7ff85c0053f0 auth_method 0 2024-09-11T01:55:23.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff8737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:23.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff8737fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:23.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff8737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:23.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff873fff640 10 monclient(hunting): get_auth_request con 0x7ff874093ee0 auth_method 0 2024-09-11T01:55:23.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff873fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:23.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff873fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:23.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.481+0000 7ff873fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:23.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.482+0000 7ff8737fe640 1 -- 172.21.15.12:0/1052290373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff85c0053f0 msgr2=0x7ff874099c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:55:23.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.482+0000 7ff8737fe640 1 -- 172.21.15.12:0/1052290373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff85c0053f0 msgr2=0x7ff874099c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:55:23.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.482+0000 7ff8737fe640 1 --2- 172.21.15.12:0/1052290373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff85c0053f0 0x7ff874099c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:55:23.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.482+0000 7ff873fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:23.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.482+0000 7ff873fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:23.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.482+0000 7ff872ffd640 1 --2- 172.21.15.12:0/1052290373 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff874093520 0x7ff8740939a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:23.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.482+0000 7ff873fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:23.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.482+0000 7ff8737fe640 1 --2- 172.21.15.12:0/1052290373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff85c0053f0 0x7ff874099c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:55:23.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.482+0000 7ff872ffd640 10 monclient(hunting): get_auth_request con 0x7ff874093520 auth_method 0 2024-09-11T01:55:23.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.482+0000 7ff872ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:23.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.482+0000 7ff872ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:23.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.482+0000 7ff872ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:23.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.482+0000 7ff872ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:23.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.482+0000 7ff872ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:23.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.482+0000 7ff872ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:23.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.482+0000 7ff873fff640 10 monclient(hunting): handle_auth_done global_id 284186 payload 824 2024-09-11T01:55:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.482+0000 7ff873fff640 1 -- 172.21.15.12:0/1052290373 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff874093520 msgr2=0x7ff8740939a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.482+0000 7ff873fff640 1 --2- 172.21.15.12:0/1052290373 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff874093520 0x7ff8740939a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.482+0000 7ff873fff640 1 -- 172.21.15.12:0/1052290373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff85c0053f0 msgr2=0x7ff874099c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:55:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.482+0000 7ff873fff640 1 --2- 172.21.15.12:0/1052290373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff85c0053f0 0x7ff874099c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.482+0000 7ff873fff640 10 monclient: _finish_hunting 0 2024-09-11T01:55:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.482+0000 7ff873fff640 1 monclient: found mon.noname-c 2024-09-11T01:55:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.482+0000 7ff873fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.482+0000 7ff873fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:55:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.482+0000 7ff873fff640 1 -- 172.21.15.12:0/1052290373 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff874097220 con 0x7ff874093ee0 2024-09-11T01:55:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.484+0000 7ff873fff640 1 -- 172.21.15.12:0/1052290373 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff874093ee0 msgr2=0x7ff8741d3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.484+0000 7ff873fff640 1 --2- 172.21.15.12:0/1052290373 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff874093ee0 0x7ff8741d3720 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff87409e560 tx=0x7ff86c0017e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.484+0000 7ff85bfff640 1 -- 172.21.15.12:0/1052290373 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86c035070 con 0x7ff874093ee0 2024-09-11T01:55:23.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.484+0000 7ff85bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.484+0000 7ff85bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:55:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.484+0000 7ff85bfff640 10 monclient: dump: 2024-09-11T01:55:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:23.507 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.484+0000 7ff85bfff640 10 monclient: _finish_auth 0 2024-09-11T01:55:23.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.484+0000 7ff85bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:55:23.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.484+0000 7ff85bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:23.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.484+0000 7ff85bfff640 1 -- 172.21.15.12:0/1052290373 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff86c007270 con 0x7ff874093ee0 2024-09-11T01:55:23.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.484+0000 7ff879e52640 5 monclient: authenticate success, global_id 284186 2024-09-11T01:55:23.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.484+0000 7ff879e52640 10 monclient: _renew_subs 2024-09-11T01:55:23.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.484+0000 7ff879e52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:55:23.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.484+0000 7ff879e52640 1 -- 172.21.15.12:0/1052290373 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff87409e160 con 0x7ff874093ee0 2024-09-11T01:55:23.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.484+0000 7ff85bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:23.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.484+0000 7ff85bfff640 1 -- 172.21.15.12:0/1052290373 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86c00c0e0 con 0x7ff874093ee0 2024-09-11T01:55:23.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.484+0000 7ff85bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:23.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.484+0000 7ff85bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:55:23.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.484+0000 7ff85bfff640 10 monclient: dump: 2024-09-11T01:55:23.508 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:23.508 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:23.509 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:23.509 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:23.509 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:23.509 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:23.509 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:23.509 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:23.509 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:23.509 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:23.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.484+0000 7ff879e52640 10 monclient: _renew_subs 2024-09-11T01:55:23.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.484+0000 7ff879e52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:55:23.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.484+0000 7ff8717fa640 4 set_mon_vals no callback set 2024-09-11T01:55:23.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.484+0000 7ff879e52640 1 -- 172.21.15.12:0/1052290373 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff874095a20 con 0x7ff874093ee0 2024-09-11T01:55:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.485+0000 7ff879e52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.485+0000 7ff879e52640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.485+0000 7ff879e52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:55:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.485+0000 7ff85bfff640 1 -- 172.21.15.12:0/1052290373 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff86c03b070 con 0x7ff874093ee0 2024-09-11T01:55:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.485+0000 7ff879e52640 1 -- 172.21.15.12:0/1052290373 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff87406d130 con 0x7ff874093ee0 2024-09-11T01:55:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.486+0000 7ff85bfff640 1 --2- 172.21.15.12:0/1052290373 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff86003e210 0x7ff8600406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.486+0000 7ff85bfff640 1 -- 172.21.15.12:0/1052290373 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(883..883 src has 676..883) ==== 4831+0+0 (secure 0 0 0) 0x7ff86c073010 con 0x7ff874093ee0 2024-09-11T01:55:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.486+0000 7ff8737fe640 1 --2- 172.21.15.12:0/1052290373 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff86003e210 0x7ff8600406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.486+0000 7ff8737fe640 10 monclient: get_auth_request con 0x7ff86003e210 auth_method 0 2024-09-11T01:55:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.489+0000 7ff8737fe640 1 --2- 172.21.15.12:0/1052290373 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff86003e210 0x7ff8600406d0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff86400b3f0 tx=0x7ff864008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.490+0000 7ff85bfff640 1 -- 172.21.15.12:0/1052290373 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff86c0076a0 con 0x7ff874093ee0 2024-09-11T01:55:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.490+0000 7ff85bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:23.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.490+0000 7ff85bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:55:23.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.746+0000 7ff879e52640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:55:23.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.746+0000 7ff879e52640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:55:23.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.746+0000 7ff879e52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:55:23.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.746+0000 7ff879e52640 1 -- 172.21.15.12:0/1052290373 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff874096ea0 con 0x7ff874093ee0 2024-09-11T01:55:23.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.747+0000 7ff85bfff640 1 -- 172.21.15.12:0/1052290373 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7ff86c0719e0 con 0x7ff874093ee0 2024-09-11T01:55:23.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.747+0000 7ff85bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:55:23.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.747+0000 7ff85bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:55:23.749 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":236,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":25,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:55:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.749+0000 7ff879e52640 1 -- 172.21.15.12:0/1052290373 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff86003e210 msgr2=0x7ff8600406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.749+0000 7ff879e52640 1 --2- 172.21.15.12:0/1052290373 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff86003e210 0x7ff8600406d0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff86400b3f0 tx=0x7ff864008040 comp rx=0 tx=0).stop 2024-09-11T01:55:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.749+0000 7ff879e52640 10 monclient: shutdown 2024-09-11T01:55:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.749+0000 7ff879e52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.749+0000 7ff879e52640 1 -- 172.21.15.12:0/1052290373 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff874093ee0 msgr2=0x7ff8741d3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.749+0000 7ff879e52640 1 --2- 172.21.15.12:0/1052290373 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff874093ee0 0x7ff8741d3720 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff87409e560 tx=0x7ff86c0017e0 comp rx=0 tx=0).stop 2024-09-11T01:55:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.749+0000 7ff879e52640 1 -- 172.21.15.12:0/1052290373 shutdown_connections 2024-09-11T01:55:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.749+0000 7ff879e52640 1 --2- 172.21.15.12:0/1052290373 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff874093ee0 0x7ff8741d3720 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.749+0000 7ff879e52640 1 --2- 172.21.15.12:0/1052290373 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff874093520 0x7ff8740939a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.749+0000 7ff879e52640 1 --2- 172.21.15.12:0/1052290373 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff86003e210 0x7ff8600406d0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.749+0000 7ff879e52640 1 --2- 172.21.15.12:0/1052290373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff85c0053f0 0x7ff874099c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.749+0000 7ff879e52640 1 -- 172.21.15.12:0/1052290373 >> 172.21.15.12:0/1052290373 conn(0x7ff8740a3ea0 msgr2=0x7ff87409b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.749+0000 7ff879e52640 1 -- 172.21.15.12:0/1052290373 shutdown_connections 2024-09-11T01:55:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:23.749+0000 7ff879e52640 1 -- 172.21.15.12:0/1052290373 wait complete. 2024-09-11T01:55:23.760 INFO:tasks.rados.rados.0.smithi071.stdout:3865: done (0 left) 2024-09-11T01:55:23.760 INFO:tasks.rados.rados.0.smithi071.stdout:3866: read oid 47 snap -1 2024-09-11T01:55:23.760 INFO:tasks.rados.rados.0.smithi071.stdout:3866: expect (ObjNum 1302 snap 426 seq_num 1302) 2024-09-11T01:55:23.760 INFO:tasks.rados.rados.0.smithi071.stdout:3867: copy_from oid 22 from oid 10 current snap is 433 2024-09-11T01:55:23.761 INFO:tasks.rados.rados.0.smithi071.stdout:3868: write oid 48 current snap is 433 2024-09-11T01:55:23.761 INFO:tasks.rados.rados.0.smithi071.stdout:3868: seq_num 1331 ranges {770117=518194,1870239=499878} 2024-09-11T01:55:23.767 INFO:tasks.rados.rados.0.smithi071.stdout:3868: writing smithi07134777-48 from 770117 to 1288311 tid 1 2024-09-11T01:55:23.774 INFO:tasks.rados.rados.0.smithi071.stdout:3868: writing smithi07134777-48 from 1870239 to 2370117 tid 2 2024-09-11T01:55:23.774 INFO:tasks.rados.rados.0.smithi071.stdout:3869: delete oid 38 current snap is 433 2024-09-11T01:55:23.774 INFO:tasks.rados.rados.0.smithi071.stdout:3866: expect (ObjNum 1302 snap 426 seq_num 1302) 2024-09-11T01:55:23.774 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:55:23.858 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1945235 2024-09-11T01:55:23.975 INFO:tasks.rados.rados.0.smithi071.stdout:3868: finishing write tid 1 to smithi07134777-48 2024-09-11T01:55:23.975 INFO:tasks.rados.rados.0.smithi071.stdout:3867: finishing copy_from racing read to smithi07134777-22 2024-09-11T01:55:23.976 INFO:tasks.rados.rados.0.smithi071.stdout:3867: finishing copy_from to smithi07134777-22 2024-09-11T01:55:23.976 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 2533 (ObjNum 1232 snap 403 seq_num 1232) dirty exists 2024-09-11T01:55:23.976 INFO:tasks.rados.rados.0.smithi071.stdout:3868: finishing write tid 2 to smithi07134777-48 2024-09-11T01:55:23.976 INFO:tasks.rados.rados.0.smithi071.stdout:3868: finishing write tid 3 to smithi07134777-48 2024-09-11T01:55:23.976 INFO:tasks.rados.rados.0.smithi071.stdout:3868: finishing write tid 4 to smithi07134777-48 2024-09-11T01:55:23.977 INFO:tasks.rados.rados.0.smithi071.stdout:3868: oid 48 updating version 0 to 3122 2024-09-11T01:55:23.977 INFO:tasks.rados.rados.0.smithi071.stdout:3868: oid 48 version 3122 is already newer than 3120 2024-09-11T01:55:23.977 INFO:tasks.rados.rados.0.smithi071.stdout:3868: oid 48 version 3122 is already newer than 3121 2024-09-11T01:55:23.977 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 3122 (ObjNum 1331 snap 433 seq_num 1331) dirty exists 2024-09-11T01:55:23.977 INFO:tasks.rados.rados.0.smithi071.stdout:3868: left oid 48 (ObjNum 1331 snap 433 seq_num 1331) 2024-09-11T01:55:23.977 INFO:tasks.rados.rados.0.smithi071.stdout:3866: done (3 left) 2024-09-11T01:55:23.978 INFO:tasks.rados.rados.0.smithi071.stdout:3867: done (2 left) 2024-09-11T01:55:23.978 INFO:tasks.rados.rados.0.smithi071.stdout:3868: done (1 left) 2024-09-11T01:55:23.978 INFO:tasks.rados.rados.0.smithi071.stdout:3869: done (0 left) 2024-09-11T01:55:23.978 INFO:tasks.rados.rados.0.smithi071.stdout:3870: copy_from oid 10 from oid 21 current snap is 433 2024-09-11T01:55:23.978 INFO:tasks.rados.rados.0.smithi071.stdout:3871: copy_from oid 19 from oid 25 current snap is 433 2024-09-11T01:55:23.978 INFO:tasks.rados.rados.0.smithi071.stdout:3872: snap_create 2024-09-11T01:55:23.984 INFO:tasks.rados.rados.0.smithi071.stdout:3870: finishing copy_from racing read to smithi07134777-10 2024-09-11T01:55:23.988 INFO:tasks.rados.rados.0.smithi071.stdout:3871: finishing copy_from racing read to smithi07134777-19 2024-09-11T01:55:23.991 INFO:tasks.rados.rados.0.smithi071.stdout:3871: finishing copy_from to smithi07134777-19 2024-09-11T01:55:23.991 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 2582 (ObjNum 1243 snap 411 seq_num 1243) dirty exists 2024-09-11T01:55:23.991 INFO:tasks.rados.rados.0.smithi071.stdout:3870: finishing copy_from to smithi07134777-10 2024-09-11T01:55:23.992 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 2536 (ObjNum 1313 snap 428 seq_num 1313) dirty exists 2024-09-11T01:55:24.762 INFO:tasks.rados.rados.0.smithi071.stdout:3870: done (2 left) 2024-09-11T01:55:24.762 INFO:tasks.rados.rados.0.smithi071.stdout:3871: done (1 left) 2024-09-11T01:55:24.762 INFO:tasks.rados.rados.0.smithi071.stdout:3872: done (0 left) 2024-09-11T01:55:24.762 INFO:tasks.rados.rados.0.smithi071.stdout:3873: snap_create 2024-09-11T01:55:25.728 INFO:tasks.rados.rados.0.smithi071.stdout:3873: done (0 left) 2024-09-11T01:55:25.728 INFO:tasks.rados.rados.0.smithi071.stdout:3874: write oid 41 current snap is 435 2024-09-11T01:55:25.728 INFO:tasks.rados.rados.0.smithi071.stdout:3874: seq_num 1333 ranges {403731=668692,1587203=587797} 2024-09-11T01:55:25.733 INFO:tasks.rados.rados.0.smithi071.stdout:3874: writing smithi07134777-41 from 403731 to 1072423 tid 1 2024-09-11T01:55:25.740 INFO:tasks.rados.rados.0.smithi071.stdout:3874: writing smithi07134777-41 from 1587203 to 2175000 tid 2 2024-09-11T01:55:25.740 INFO:tasks.rados.rados.0.smithi071.stdout:3875: write (excl) oid 13 current snap is 435 2024-09-11T01:55:25.740 INFO:tasks.rados.rados.0.smithi071.stdout:3875: seq_num 1334 ranges {420538=743941,1929332=491206} 2024-09-11T01:55:25.747 INFO:tasks.rados.rados.0.smithi071.stdout:3875: writing smithi07134777-13 from 420538 to 1164479 tid 1 2024-09-11T01:55:25.755 INFO:tasks.rados.rados.0.smithi071.stdout:3875: writing smithi07134777-13 from 1929332 to 2420538 tid 2 2024-09-11T01:55:25.755 INFO:tasks.rados.rados.0.smithi071.stdout:3876: read oid 8 snap -1 2024-09-11T01:55:25.755 INFO:tasks.rados.rados.0.smithi071.stdout:3876: expect (ObjNum 1323 snap 432 seq_num 1323) 2024-09-11T01:55:25.755 INFO:tasks.rados.rados.0.smithi071.stdout:3877: read oid 15 snap 404 2024-09-11T01:55:25.755 INFO:tasks.rados.rados.0.smithi071.stdout:3877: expect deleted 2024-09-11T01:55:25.755 INFO:tasks.rados.rados.0.smithi071.stdout:3878: read oid 5 snap 429 2024-09-11T01:55:25.755 INFO:tasks.rados.rados.0.smithi071.stdout:3878: expect (ObjNum 1255 snap 415 seq_num 1255) 2024-09-11T01:55:25.755 INFO:tasks.rados.rados.0.smithi071.stdout:3879: read oid 28 snap 434 2024-09-11T01:55:25.755 INFO:tasks.rados.rados.0.smithi071.stdout:3879: expect deleted 2024-09-11T01:55:25.755 INFO:tasks.rados.rados.0.smithi071.stdout:3880: write (excl) oid 9 current snap is 435 2024-09-11T01:55:25.756 INFO:tasks.rados.rados.0.smithi071.stdout:3874: finishing write tid 1 to smithi07134777-41 2024-09-11T01:55:25.756 INFO:tasks.rados.rados.0.smithi071.stdout:3880: seq_num 1335 ranges {437345=419190,1587814=591466} 2024-09-11T01:55:25.759 INFO:tasks.rados.rados.0.smithi071.stdout:3880: writing smithi07134777-9 from 437345 to 856535 tid 1 2024-09-11T01:55:25.767 INFO:tasks.rados.rados.0.smithi071.stdout:3880: writing smithi07134777-9 from 1587814 to 2179280 tid 2 2024-09-11T01:55:25.768 INFO:tasks.rados.rados.0.smithi071.stdout:3881: delete oid 48 current snap is 435 2024-09-11T01:55:25.768 INFO:tasks.rados.rados.0.smithi071.stdout:3874: finishing write tid 2 to smithi07134777-41 2024-09-11T01:55:25.768 INFO:tasks.rados.rados.0.smithi071.stdout:3874: finishing write tid 3 to smithi07134777-41 2024-09-11T01:55:25.768 INFO:tasks.rados.rados.0.smithi071.stdout:3874: finishing write tid 4 to smithi07134777-41 2024-09-11T01:55:25.768 INFO:tasks.rados.rados.0.smithi071.stdout:3874: oid 41 updating version 0 to 2539 2024-09-11T01:55:25.768 INFO:tasks.rados.rados.0.smithi071.stdout:3874: oid 41 version 2539 is already newer than 2538 2024-09-11T01:55:25.768 INFO:tasks.rados.rados.0.smithi071.stdout:3874: oid 41 updating version 2539 to 2540 2024-09-11T01:55:25.768 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 2540 (ObjNum 1333 snap 435 seq_num 1333) dirty exists 2024-09-11T01:55:25.768 INFO:tasks.rados.rados.0.smithi071.stdout:3874: left oid 41 (ObjNum 1333 snap 435 seq_num 1333) 2024-09-11T01:55:25.768 INFO:tasks.rados.rados.0.smithi071.stdout:3875: finishing write tid 1 to smithi07134777-13 2024-09-11T01:55:25.768 INFO:tasks.rados.rados.0.smithi071.stdout:3875: finishing write tid 2 to smithi07134777-13 2024-09-11T01:55:25.768 INFO:tasks.rados.rados.0.smithi071.stdout:3875: finishing write tid 3 to smithi07134777-13 2024-09-11T01:55:25.769 INFO:tasks.rados.rados.0.smithi071.stdout:3875: finishing write tid 4 to smithi07134777-13 2024-09-11T01:55:25.769 INFO:tasks.rados.rados.0.smithi071.stdout:3875: oid 13 updating version 0 to 2586 2024-09-11T01:55:25.769 INFO:tasks.rados.rados.0.smithi071.stdout:3875: oid 13 updating version 2586 to 2587 2024-09-11T01:55:25.769 INFO:tasks.rados.rados.0.smithi071.stdout:3875: oid 13 updating version 2587 to 2588 2024-09-11T01:55:25.769 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 2588 (ObjNum 1334 snap 435 seq_num 1334) dirty exists 2024-09-11T01:55:25.769 INFO:tasks.rados.rados.0.smithi071.stdout:3875: left oid 13 (ObjNum 1334 snap 435 seq_num 1334) 2024-09-11T01:55:25.769 INFO:tasks.rados.rados.0.smithi071.stdout:3878: expect (ObjNum 1255 snap 415 seq_num 1255) 2024-09-11T01:55:25.841 INFO:tasks.rados.rados.0.smithi071.stdout:3880: finishing write tid 1 to smithi07134777-9 2024-09-11T01:55:25.841 INFO:tasks.rados.rados.0.smithi071.stdout:3876: expect (ObjNum 1323 snap 432 seq_num 1323) 2024-09-11T01:55:25.841 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:55:25.883 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:55:25.909 INFO:tasks.rados.rados.0.smithi071.stdout:3880: finishing write tid 2 to smithi07134777-9 2024-09-11T01:55:25.909 INFO:tasks.rados.rados.0.smithi071.stdout:3880: finishing write tid 3 to smithi07134777-9 2024-09-11T01:55:25.909 INFO:tasks.rados.rados.0.smithi071.stdout:3880: finishing write tid 4 to smithi07134777-9 2024-09-11T01:55:25.909 INFO:tasks.rados.rados.0.smithi071.stdout:3880: oid 9 updating version 0 to 2586 2024-09-11T01:55:25.909 INFO:tasks.rados.rados.0.smithi071.stdout:3880: oid 9 updating version 2586 to 2588 2024-09-11T01:55:25.909 INFO:tasks.rados.rados.0.smithi071.stdout:3880: oid 9 version 2588 is already newer than 2587 2024-09-11T01:55:25.910 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 2588 (ObjNum 1335 snap 435 seq_num 1335) dirty exists 2024-09-11T01:55:25.910 INFO:tasks.rados.rados.0.smithi071.stdout:3880: left oid 9 (ObjNum 1335 snap 435 seq_num 1335) 2024-09-11T01:55:25.910 INFO:tasks.rados.rados.0.smithi071.stdout:3874: done (7 left) 2024-09-11T01:55:25.910 INFO:tasks.rados.rados.0.smithi071.stdout:3875: done (6 left) 2024-09-11T01:55:25.910 INFO:tasks.rados.rados.0.smithi071.stdout:3876: done (5 left) 2024-09-11T01:55:25.910 INFO:tasks.rados.rados.0.smithi071.stdout:3877: done (4 left) 2024-09-11T01:55:25.910 INFO:tasks.rados.rados.0.smithi071.stdout:3878: done (3 left) 2024-09-11T01:55:25.910 INFO:tasks.rados.rados.0.smithi071.stdout:3879: done (2 left) 2024-09-11T01:55:25.910 INFO:tasks.rados.rados.0.smithi071.stdout:3880: done (1 left) 2024-09-11T01:55:25.910 INFO:tasks.rados.rados.0.smithi071.stdout:3881: done (0 left) 2024-09-11T01:55:25.910 INFO:tasks.rados.rados.0.smithi071.stdout:3882: snap_create 2024-09-11T01:55:26.497 INFO:tasks.rados.rados.0.smithi071.stdout:3882: done (0 left) 2024-09-11T01:55:26.497 INFO:tasks.rados.rados.0.smithi071.stdout:3883: delete oid 40 current snap is 436 2024-09-11T01:55:26.501 INFO:tasks.rados.rados.0.smithi071.stdout:3883: done (0 left) 2024-09-11T01:55:26.501 INFO:tasks.rados.rados.0.smithi071.stdout:3884: snap_remove snap 427 2024-09-11T01:55:26.762 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:55:26.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.874+0000 7fc84a1f8640 10 monclient: get_monmap_and_config 2024-09-11T01:55:26.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.874+0000 7fc84a1f8640 10 monclient: build_initial_monmap 2024-09-11T01:55:26.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.874+0000 7fc84a1f8640 10 monclient: monmap: 2024-09-11T01:55:26.876 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:26.876 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:26.876 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:55:26.875394+0000 2024-09-11T01:55:26.876 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:55:26.875394+0000 2024-09-11T01:55:26.877 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:26.877 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:26.877 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:55:26.877 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:55:26.877 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:55:26.877 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:55:26.877 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:55:26.877 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:26.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.875+0000 7fc84a1f8640 1 Processor -- start 2024-09-11T01:55:26.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.875+0000 7fc84a1f8640 1 -- start start 2024-09-11T01:55:26.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.875+0000 7fc84a1f8640 10 monclient: init 2024-09-11T01:55:26.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.875+0000 7fc84a1f8640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:26.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.875+0000 7fc84a1f8640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-11T01:55:26.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc84a1f8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8440a2e90 0x7fc844073d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:26.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc84a1f8640 10 monclient(hunting): picked mon.noname-b con 0x7fc8440a2e90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:26.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc84a1f8640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc8440a3270 0x7fc8440a3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:26.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc84a1f8640 10 monclient(hunting): picked mon.noname-c con 0x7fc8440a3270 addr v2:172.21.15.71:3300/0 2024-09-11T01:55:26.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc84a1f8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc844135c50 addr v1:172.21.15.71:6789/0 2024-09-11T01:55:26.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc84a1f8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8440a46c0 con 0x7fc844135c50 2024-09-11T01:55:26.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc84a1f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:26.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc84a1f8640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fc844072730 con 0x7fc8440a3270 2024-09-11T01:55:26.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc84a1f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:26.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc84a1f8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc844072fa0 con 0x7fc8440a2e90 2024-09-11T01:55:26.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc84a1f8640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:26.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc84a1f8640 10 monclient(hunting): authenticate will time out at 2333.318086s 2024-09-11T01:55:26.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc8437fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8440a2e90 0x7fc844073d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:26.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc8437fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8440a2e90 0x7fc844073d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:44860/0 (socket says 172.21.15.12:44860) 2024-09-11T01:55:26.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc8437fe640 1 -- 172.21.15.12:0/2624041099 learned_addr learned my addr 172.21.15.12:0/2624041099 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:55:26.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc8437fe640 10 monclient(hunting): get_auth_request con 0x7fc8440a2e90 auth_method 0 2024-09-11T01:55:26.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc8437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:26.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc8437fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:26.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc8437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:26.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc8437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:26.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc8437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:26.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc8437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:26.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc842ffd640 1 --2- 172.21.15.12:0/2624041099 >> v2:172.21.15.71:3300/0 conn(0x7fc8440a3270 0x7fc8440a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:26.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc842ffd640 10 monclient(hunting): get_auth_request con 0x7fc8440a3270 auth_method 0 2024-09-11T01:55:26.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc842ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:26.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc842ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:26.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc842ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:26.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc8437fe640 10 monclient(hunting): handle_auth_done global_id 284189 payload 274 2024-09-11T01:55:26.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc8437fe640 1 -- 172.21.15.12:0/2624041099 >> v2:172.21.15.71:3300/0 conn(0x7fc8440a3270 msgr2=0x7fc8440a3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:55:26.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc8437fe640 1 --2- 172.21.15.12:0/2624041099 >> v2:172.21.15.71:3300/0 conn(0x7fc8440a3270 0x7fc8440a3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:26.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc8437fe640 1 -- 172.21.15.12:0/2624041099 >> v1:172.21.15.71:6789/0 conn(0x7fc844135c50 legacy=0x7fc844136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:55:26.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc8437fe640 10 monclient: _finish_hunting 0 2024-09-11T01:55:26.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc8437fe640 1 monclient: found mon.noname-b 2024-09-11T01:55:26.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc8437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:26.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc8437fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:55:26.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.876+0000 7fc8437fe640 1 -- 172.21.15.12:0/2624041099 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc844073280 con 0x7fc8440a2e90 2024-09-11T01:55:26.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.879+0000 7fc8437fe640 1 -- 172.21.15.12:0/2624041099 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8440a2e90 msgr2=0x7fc844073d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:26.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.879+0000 7fc8437fe640 1 --2- 172.21.15.12:0/2624041099 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8440a2e90 0x7fc844073d40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc834009870 tx=0x7fc83402a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=480f9539cddedfa2 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:26.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.879+0000 7fc8427fc640 1 -- 172.21.15.12:0/2624041099 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc834037070 con 0x7fc8440a2e90 2024-09-11T01:55:26.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.879+0000 7fc8427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:26.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.879+0000 7fc8427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:55:26.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.879+0000 7fc8427fc640 10 monclient: dump: 2024-09-11T01:55:26.889 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:26.889 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:26.889 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:26.889 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:26.889 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:26.890 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:26.890 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:26.890 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:26.890 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:26.890 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:26.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.879+0000 7fc8427fc640 10 monclient: _finish_auth 0 2024-09-11T01:55:26.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.879+0000 7fc8427fc640 10 monclient: _check_auth_tickets 2024-09-11T01:55:26.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.879+0000 7fc8427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:26.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.879+0000 7fc84a1f8640 5 monclient: authenticate success, global_id 284189 2024-09-11T01:55:26.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.879+0000 7fc8427fc640 1 -- 172.21.15.12:0/2624041099 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc834032440 con 0x7fc8440a2e90 2024-09-11T01:55:26.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.879+0000 7fc84a1f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:55:26.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.879+0000 7fc8427fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:26.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.879+0000 7fc84a1f8640 10 monclient: get_monmap_and_config success 2024-09-11T01:55:26.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.879+0000 7fc84a1f8640 4 set_mon_vals no callback set 2024-09-11T01:55:26.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.879+0000 7fc8427fc640 1 -- 172.21.15.12:0/2624041099 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc834032810 con 0x7fc8440a2e90 2024-09-11T01:55:26.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.879+0000 7fc8427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:26.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.879+0000 7fc8427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:55:26.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.879+0000 7fc8427fc640 10 monclient: dump: 2024-09-11T01:55:26.891 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:26.891 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:26.891 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:26.891 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:26.891 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:26.891 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:26.891 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:26.891 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:26.891 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:26.892 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:26.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.880+0000 7fc84a1f8640 10 monclient: shutdown 2024-09-11T01:55:26.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.880+0000 7fc84a1f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:26.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.880+0000 7fc84a1f8640 1 -- 172.21.15.12:0/2624041099 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8440a2e90 msgr2=0x7fc844073d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:26.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.880+0000 7fc84a1f8640 1 --2- 172.21.15.12:0/2624041099 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8440a2e90 0x7fc844073d40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc834009870 tx=0x7fc83402a0a0 comp rx=0 tx=0).stop 2024-09-11T01:55:26.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.880+0000 7fc84a1f8640 1 -- 172.21.15.12:0/2624041099 shutdown_connections 2024-09-11T01:55:26.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.880+0000 7fc84a1f8640 1 --2- 172.21.15.12:0/2624041099 >> v2:172.21.15.71:3300/0 conn(0x7fc8440a3270 0x7fc8440a3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:26.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.880+0000 7fc84a1f8640 1 --2- 172.21.15.12:0/2624041099 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc8440a2e90 0x7fc844073d40 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:26.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.880+0000 7fc84a1f8640 1 -- 172.21.15.12:0/2624041099 >> 172.21.15.12:0/2624041099 conn(0x7fc8440a8490 msgr2=0x7fc8440a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:26.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.880+0000 7fc84a1f8640 1 -- 172.21.15.12:0/2624041099 shutdown_connections 2024-09-11T01:55:26.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.880+0000 7fc84a1f8640 1 -- 172.21.15.12:0/2624041099 wait complete. 2024-09-11T01:55:26.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.881+0000 7fc84a1f8640 10 monclient: build_initial_monmap 2024-09-11T01:55:26.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.881+0000 7fc84a1f8640 10 monclient: monmap: 2024-09-11T01:55:26.893 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:26.893 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:55:26.893 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:55:26.893 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:55:26.893 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:26.893 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:26.893 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:55:26.893 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:55:26.893 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:55:26.893 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:26.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.881+0000 7fc84a1f8640 1 Processor -- start 2024-09-11T01:55:26.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.881+0000 7fc84a1f8640 1 -- start start 2024-09-11T01:55:26.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.881+0000 7fc84a1f8640 10 monclient: init 2024-09-11T01:55:26.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc84a1f8640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:26.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc84a1f8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:55:26.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc84a1f8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8440a2e90 0x7fc844124b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:26.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc84a1f8640 10 monclient(hunting): picked mon.noname-a con 0x7fc8440a2e90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:55:26.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc84a1f8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc8440a3270 0x7fc844125070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:26.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc84a1f8640 10 monclient(hunting): picked mon.noname-b con 0x7fc8440a3270 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:55:26.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc84a1f8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc844135c50 0x7fc8441255b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:26.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc84a1f8640 10 monclient(hunting): picked mon.noname-c con 0x7fc844135c50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:26.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc84a1f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:26.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc84a1f8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc844072300 con 0x7fc8440a2e90 2024-09-11T01:55:26.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc84a1f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:26.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc84a1f8640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc844074310 con 0x7fc8440a3270 2024-09-11T01:55:26.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc84a1f8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:26.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc84a1f8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc844072480 con 0x7fc844135c50 2024-09-11T01:55:26.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc84a1f8640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:26.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc84a1f8640 10 monclient(hunting): authenticate will time out at 2333.324122s 2024-09-11T01:55:26.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc8437fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8440a2e90 0x7fc844124b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:26.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc843fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc844135c50 0x7fc8441255b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:26.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc8437fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8440a2e90 0x7fc844124b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45964/0 (socket says 172.21.15.12:45964) 2024-09-11T01:55:26.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc8437fe640 1 -- 172.21.15.12:0/4237455380 learned_addr learned my addr 172.21.15.12:0/4237455380 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:55:26.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc8437fe640 10 monclient(hunting): get_auth_request con 0x7fc8440a2e90 auth_method 0 2024-09-11T01:55:26.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc8437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:26.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc8437fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:26.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc8437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:26.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc843fff640 10 monclient(hunting): get_auth_request con 0x7fc844135c50 auth_method 0 2024-09-11T01:55:26.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc843fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:26.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc843fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:26.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc843fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:26.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc8437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:26.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc8437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:26.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc8437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:26.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc843fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:26.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc843fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:26.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc843fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:26.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc842ffd640 1 --2- 172.21.15.12:0/4237455380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc8440a3270 0x7fc844125070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:26.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc842ffd640 10 monclient(hunting): get_auth_request con 0x7fc8440a3270 auth_method 0 2024-09-11T01:55:26.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc842ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:26.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc842ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:26.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc842ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:26.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc8437fe640 10 monclient(hunting): handle_auth_done global_id 294099 payload 824 2024-09-11T01:55:26.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc8437fe640 1 -- 172.21.15.12:0/4237455380 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc844135c50 msgr2=0x7fc8441255b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:26.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc8437fe640 1 --2- 172.21.15.12:0/4237455380 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc844135c50 0x7fc8441255b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:26.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc8437fe640 1 -- 172.21.15.12:0/4237455380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc8440a3270 msgr2=0x7fc844125070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:26.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc8437fe640 1 --2- 172.21.15.12:0/4237455380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc8440a3270 0x7fc844125070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:26.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc8437fe640 10 monclient: _finish_hunting 0 2024-09-11T01:55:26.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc8437fe640 1 monclient: found mon.noname-a 2024-09-11T01:55:26.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc8437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:26.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc8437fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:55:26.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc8437fe640 1 -- 172.21.15.12:0/4237455380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc844126bb0 con 0x7fc8440a2e90 2024-09-11T01:55:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc842ffd640 1 --2- 172.21.15.12:0/4237455380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc8440a3270 0x7fc844125070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:55:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.882+0000 7fc843fff640 1 --2- 172.21.15.12:0/4237455380 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc844135c50 0x7fc8441255b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:55:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.884+0000 7fc8437fe640 1 -- 172.21.15.12:0/4237455380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8440a2e90 msgr2=0x7fc844124b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.884+0000 7fc8437fe640 1 --2- 172.21.15.12:0/4237455380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8440a2e90 0x7fc844124b30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc8340090b0 tx=0x7fc8340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.885+0000 7fc82bfff640 1 -- 172.21.15.12:0/4237455380 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc834041020 con 0x7fc8440a2e90 2024-09-11T01:55:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.885+0000 7fc82bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.885+0000 7fc82bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:55:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.885+0000 7fc82bfff640 10 monclient: dump: 2024-09-11T01:55:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:26.898 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:26.899 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:26.899 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:26.899 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:26.899 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:26.899 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:26.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.885+0000 7fc82bfff640 10 monclient: _finish_auth 0 2024-09-11T01:55:26.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.885+0000 7fc82bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:55:26.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.885+0000 7fc82bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:26.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.885+0000 7fc82bfff640 1 -- 172.21.15.12:0/4237455380 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc83403d630 con 0x7fc8440a2e90 2024-09-11T01:55:26.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.885+0000 7fc82bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:26.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.885+0000 7fc82bfff640 1 -- 172.21.15.12:0/4237455380 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc834037040 con 0x7fc8440a2e90 2024-09-11T01:55:26.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.885+0000 7fc82bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:26.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.885+0000 7fc82bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:55:26.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.885+0000 7fc82bfff640 10 monclient: dump: 2024-09-11T01:55:26.900 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:26.900 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:26.900 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:26.900 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:26.900 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:26.900 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:26.900 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:26.900 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:26.900 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:26.901 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:26.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.885+0000 7fc84a1f8640 5 monclient: authenticate success, global_id 294099 2024-09-11T01:55:26.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.885+0000 7fc84a1f8640 10 monclient: _renew_subs 2024-09-11T01:55:26.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.885+0000 7fc84a1f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:26.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.885+0000 7fc84a1f8640 1 -- 172.21.15.12:0/4237455380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8441284e0 con 0x7fc8440a2e90 2024-09-11T01:55:26.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.885+0000 7fc841ffb640 4 set_mon_vals no callback set 2024-09-11T01:55:26.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.885+0000 7fc84a1f8640 10 monclient: _renew_subs 2024-09-11T01:55:26.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.885+0000 7fc84a1f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:26.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.885+0000 7fc84a1f8640 1 -- 172.21.15.12:0/4237455380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc84409f680 con 0x7fc8440a2e90 2024-09-11T01:55:26.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.886+0000 7fc82bfff640 1 -- 172.21.15.12:0/4237455380 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fc83403a030 con 0x7fc8440a2e90 2024-09-11T01:55:26.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.886+0000 7fc84a1f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:26.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.886+0000 7fc84a1f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:26.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.886+0000 7fc84a1f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:26.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.886+0000 7fc84a1f8640 1 -- 172.21.15.12:0/4237455380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8441369e0 con 0x7fc8440a2e90 2024-09-11T01:55:26.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.886+0000 7fc82bfff640 1 --2- 172.21.15.12:0/4237455380 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc83003e290 0x7fc830040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:26.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.886+0000 7fc82bfff640 1 -- 172.21.15.12:0/4237455380 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(887..887 src has 676..887) ==== 4803+0+0 (secure 0 0 0) 0x7fc834072e00 con 0x7fc8440a2e90 2024-09-11T01:55:26.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.886+0000 7fc842ffd640 1 --2- 172.21.15.12:0/4237455380 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc83003e290 0x7fc830040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:26.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.886+0000 7fc842ffd640 10 monclient: get_auth_request con 0x7fc83003e290 auth_method 0 2024-09-11T01:55:26.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.889+0000 7fc842ffd640 1 --2- 172.21.15.12:0/4237455380 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc83003e290 0x7fc830040750 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc84406faf0 tx=0x7fc838008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:26.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.891+0000 7fc82bfff640 1 -- 172.21.15.12:0/4237455380 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc83404b070 con 0x7fc8440a2e90 2024-09-11T01:55:26.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.891+0000 7fc82bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:26.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:26.891+0000 7fc82bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:55:27.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.147+0000 7fc84a1f8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:55:27.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.147+0000 7fc84a1f8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:55:27.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.147+0000 7fc84a1f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:27.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.147+0000 7fc84a1f8640 1 -- 172.21.15.12:0/4237455380 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc844135780 con 0x7fc8440a2e90 2024-09-11T01:55:27.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.148+0000 7fc82bfff640 1 -- 172.21.15.12:0/4237455380 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fc8340717e0 con 0x7fc8440a2e90 2024-09-11T01:55:27.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.148+0000 7fc82bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:55:27.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.148+0000 7fc82bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:55:27.150 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":240,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:55:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.150+0000 7fc84a1f8640 1 -- 172.21.15.12:0/4237455380 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc83003e290 msgr2=0x7fc830040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.150+0000 7fc84a1f8640 1 --2- 172.21.15.12:0/4237455380 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc83003e290 0x7fc830040750 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc84406faf0 tx=0x7fc838008040 comp rx=0 tx=0).stop 2024-09-11T01:55:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.150+0000 7fc84a1f8640 10 monclient: shutdown 2024-09-11T01:55:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.150+0000 7fc84a1f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.150+0000 7fc84a1f8640 1 -- 172.21.15.12:0/4237455380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8440a2e90 msgr2=0x7fc844124b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.150+0000 7fc84a1f8640 1 --2- 172.21.15.12:0/4237455380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8440a2e90 0x7fc844124b30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc8340090b0 tx=0x7fc8340046c0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.150+0000 7fc84a1f8640 1 -- 172.21.15.12:0/4237455380 shutdown_connections 2024-09-11T01:55:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.150+0000 7fc84a1f8640 1 --2- 172.21.15.12:0/4237455380 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc844135c50 0x7fc8441255b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.150+0000 7fc84a1f8640 1 --2- 172.21.15.12:0/4237455380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc8440a3270 0x7fc844125070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.150+0000 7fc84a1f8640 1 --2- 172.21.15.12:0/4237455380 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc83003e290 0x7fc830040750 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.150+0000 7fc84a1f8640 1 --2- 172.21.15.12:0/4237455380 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc8440a2e90 0x7fc844124b30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.150+0000 7fc84a1f8640 1 -- 172.21.15.12:0/4237455380 >> 172.21.15.12:0/4237455380 conn(0x7fc8440a8490 msgr2=0x7fc8440a7220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:27.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.150+0000 7fc84a1f8640 1 -- 172.21.15.12:0/4237455380 shutdown_connections 2024-09-11T01:55:27.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.150+0000 7fc84a1f8640 1 -- 172.21.15.12:0/4237455380 wait complete. 2024-09-11T01:55:27.162 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:55:27.162 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:55:27.162 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:55:27.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.262+0000 7ff4d5994640 10 monclient: get_monmap_and_config 2024-09-11T01:55:27.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.262+0000 7ff4d5994640 10 monclient: build_initial_monmap 2024-09-11T01:55:27.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.262+0000 7ff4d5994640 10 monclient: monmap: 2024-09-11T01:55:27.264 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:27.265 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:27.265 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:55:27.263548+0000 2024-09-11T01:55:27.265 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:55:27.263548+0000 2024-09-11T01:55:27.265 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:27.265 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.265 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:55:27.265 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:55:27.265 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:55:27.265 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:55:27.265 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:55:27.265 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.263+0000 7ff4d5994640 1 Processor -- start 2024-09-11T01:55:27.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.263+0000 7ff4d5994640 1 -- start start 2024-09-11T01:55:27.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.263+0000 7ff4d5994640 10 monclient: init 2024-09-11T01:55:27.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.263+0000 7ff4d5994640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:27.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.263+0000 7ff4d5994640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-11T01:55:27.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.263+0000 7ff4d5994640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff4d00a2e20 0x7ff4d006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.263+0000 7ff4d5994640 10 monclient(hunting): picked mon.noname-a con 0x7ff4d00a2e20 addr v2:172.21.15.12:3300/0 2024-09-11T01:55:27.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.263+0000 7ff4d5994640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff4d00a4970 addr v1:172.21.15.12:6789/0 2024-09-11T01:55:27.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.263+0000 7ff4d5994640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff4d006f910 0x7ff4d013f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.263+0000 7ff4d5994640 10 monclient(hunting): picked mon.noname-c con 0x7ff4d006f910 addr v2:172.21.15.71:3300/0 2024-09-11T01:55:27.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.263+0000 7ff4d5994640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4d006f4f0 con 0x7ff4d00a4970 2024-09-11T01:55:27.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.263+0000 7ff4d5994640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:27.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.263+0000 7ff4d5994640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ff4d006ef30 con 0x7ff4d00a2e20 2024-09-11T01:55:27.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.263+0000 7ff4d5994640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:27.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.263+0000 7ff4d5994640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7ff4d006ec50 con 0x7ff4d006f910 2024-09-11T01:55:27.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.263+0000 7ff4d5994640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:27.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.263+0000 7ff4d5994640 10 monclient(hunting): authenticate will time out at 2333.705994s 2024-09-11T01:55:27.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.264+0000 7ff4ceffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff4d00a2e20 0x7ff4d006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.264+0000 7ff4ceffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff4d00a2e20 0x7ff4d006ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45974/0 (socket says 172.21.15.12:45974) 2024-09-11T01:55:27.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.264+0000 7ff4ceffd640 1 -- 172.21.15.12:0/1129191398 learned_addr learned my addr 172.21.15.12:0/1129191398 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:55:27.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.264+0000 7ff4ceffd640 10 monclient(hunting): get_auth_request con 0x7ff4d00a2e20 auth_method 0 2024-09-11T01:55:27.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.264+0000 7ff4ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:27.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.264+0000 7ff4ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:27.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.264+0000 7ff4ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:27.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.264+0000 7ff4cf7fe640 1 --2- 172.21.15.12:0/1129191398 >> v2:172.21.15.71:3300/0 conn(0x7ff4d006f910 0x7ff4d013f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.264+0000 7ff4cf7fe640 10 monclient(hunting): get_auth_request con 0x7ff4d006f910 auth_method 0 2024-09-11T01:55:27.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.264+0000 7ff4cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:27.270 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.264+0000 7ff4cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:27.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.264+0000 7ff4cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:27.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.264+0000 7ff4cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:27.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.264+0000 7ff4cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:27.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.264+0000 7ff4cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:27.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.264+0000 7ff4cf7fe640 10 monclient(hunting): handle_auth_done global_id 274267 payload 274 2024-09-11T01:55:27.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.264+0000 7ff4cf7fe640 1 -- 172.21.15.12:0/1129191398 >> v2:172.21.15.12:3300/0 conn(0x7ff4d00a2e20 msgr2=0x7ff4d006ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:55:27.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.264+0000 7ff4cf7fe640 1 --2- 172.21.15.12:0/1129191398 >> v2:172.21.15.12:3300/0 conn(0x7ff4d00a2e20 0x7ff4d006ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.264+0000 7ff4cf7fe640 1 -- 172.21.15.12:0/1129191398 >> v1:172.21.15.12:6789/0 conn(0x7ff4d00a4970 legacy=0x7ff4d00a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:55:27.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.264+0000 7ff4cf7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:55:27.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.264+0000 7ff4cf7fe640 1 monclient: found mon.noname-c 2024-09-11T01:55:27.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.264+0000 7ff4cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:27.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.264+0000 7ff4cf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:55:27.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.264+0000 7ff4cf7fe640 1 -- 172.21.15.12:0/1129191398 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4d00a3910 con 0x7ff4d006f910 2024-09-11T01:55:27.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.267+0000 7ff4cf7fe640 1 -- 172.21.15.12:0/1129191398 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d006f910 msgr2=0x7ff4d013f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:27.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.267+0000 7ff4cf7fe640 1 --2- 172.21.15.12:0/1129191398 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d006f910 0x7ff4d013f430 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff4d00a62d0 tx=0x7ff4c802a4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=116adaa7b6c73e6f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:27.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.268+0000 7ff4cdffb640 1 -- 172.21.15.12:0/1129191398 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c8007950 con 0x7ff4d006f910 2024-09-11T01:55:27.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.268+0000 7ff4cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:27.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.268+0000 7ff4cdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:55:27.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.268+0000 7ff4cdffb640 10 monclient: dump: 2024-09-11T01:55:27.279 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:27.279 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:27.279 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.280 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.280 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:27.280 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.280 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:27.280 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:27.280 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:27.280 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.268+0000 7ff4cdffb640 10 monclient: _finish_auth 0 2024-09-11T01:55:27.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.268+0000 7ff4cdffb640 10 monclient: _check_auth_tickets 2024-09-11T01:55:27.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.268+0000 7ff4cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:27.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.268+0000 7ff4d5994640 5 monclient: authenticate success, global_id 274267 2024-09-11T01:55:27.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.268+0000 7ff4d5994640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:55:27.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.268+0000 7ff4cdffb640 1 -- 172.21.15.12:0/1129191398 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4c8007af0 con 0x7ff4d006f910 2024-09-11T01:55:27.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.268+0000 7ff4cdffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:27.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.268+0000 7ff4d5994640 10 monclient: get_monmap_and_config success 2024-09-11T01:55:27.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.268+0000 7ff4cdffb640 1 -- 172.21.15.12:0/1129191398 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c8031400 con 0x7ff4d006f910 2024-09-11T01:55:27.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.268+0000 7ff4cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:27.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.268+0000 7ff4d5994640 4 set_mon_vals no callback set 2024-09-11T01:55:27.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.268+0000 7ff4cdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:55:27.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.268+0000 7ff4cdffb640 10 monclient: dump: 2024-09-11T01:55:27.281 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:27.282 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:27.282 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.282 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.282 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:27.282 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.282 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:27.282 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:27.282 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:27.282 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.268+0000 7ff4d5994640 10 monclient: shutdown 2024-09-11T01:55:27.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.268+0000 7ff4d5994640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:27.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.268+0000 7ff4d5994640 1 -- 172.21.15.12:0/1129191398 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d006f910 msgr2=0x7ff4d013f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.268+0000 7ff4d5994640 1 --2- 172.21.15.12:0/1129191398 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d006f910 0x7ff4d013f430 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff4d00a62d0 tx=0x7ff4c802a4b0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.268+0000 7ff4d5994640 1 -- 172.21.15.12:0/1129191398 shutdown_connections 2024-09-11T01:55:27.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.268+0000 7ff4d5994640 1 --2- 172.21.15.12:0/1129191398 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d006f910 0x7ff4d013f430 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.268+0000 7ff4d5994640 1 --2- 172.21.15.12:0/1129191398 >> v2:172.21.15.12:3300/0 conn(0x7ff4d00a2e20 0x7ff4d006ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.268+0000 7ff4d5994640 1 -- 172.21.15.12:0/1129191398 >> 172.21.15.12:0/1129191398 conn(0x7ff4d00a8600 msgr2=0x7ff4d00a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:27.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.269+0000 7ff4d5994640 1 -- 172.21.15.12:0/1129191398 shutdown_connections 2024-09-11T01:55:27.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.269+0000 7ff4d5994640 1 -- 172.21.15.12:0/1129191398 wait complete. 2024-09-11T01:55:27.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.269+0000 7ff4d5994640 10 monclient: build_initial_monmap 2024-09-11T01:55:27.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.269+0000 7ff4d5994640 10 monclient: monmap: 2024-09-11T01:55:27.283 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:27.283 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:55:27.283 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:55:27.283 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:55:27.283 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:27.283 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.283 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:55:27.284 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:55:27.284 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:55:27.284 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.269+0000 7ff4d5994640 1 Processor -- start 2024-09-11T01:55:27.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4d5994640 1 -- start start 2024-09-11T01:55:27.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4d5994640 10 monclient: init 2024-09-11T01:55:27.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4d5994640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:27.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4d5994640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:55:27.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4d5994640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d006f910 0x7ff4d0137eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4d5994640 10 monclient(hunting): picked mon.noname-b con 0x7ff4d006f910 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:55:27.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4d5994640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff4d00a2e20 0x7ff4d01383f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4d5994640 10 monclient(hunting): picked mon.noname-c con 0x7ff4d00a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:27.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4d5994640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4d00a4970 0x7ff4d01cce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4d5994640 10 monclient(hunting): picked mon.noname-a con 0x7ff4d00a4970 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:55:27.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4d5994640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:27.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4d5994640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d013fd00 con 0x7ff4d00a4970 2024-09-11T01:55:27.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4d5994640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:27.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4d5994640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d013fb80 con 0x7ff4d006f910 2024-09-11T01:55:27.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4d5994640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:27.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4d5994640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff4d013fe80 con 0x7ff4d00a2e20 2024-09-11T01:55:27.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4d5994640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:27.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4d5994640 10 monclient(hunting): authenticate will time out at 2333.712553s 2024-09-11T01:55:27.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4ce7fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff4d00a2e20 0x7ff4d01383f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4cf7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4d00a4970 0x7ff4d01cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4ce7fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff4d00a2e20 0x7ff4d01383f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:44878/0 (socket says 172.21.15.12:44878) 2024-09-11T01:55:27.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4cf7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4d00a4970 0x7ff4d01cce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45990/0 (socket says 172.21.15.12:45990) 2024-09-11T01:55:27.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4ce7fc640 1 -- 172.21.15.12:0/1228399447 learned_addr learned my addr 172.21.15.12:0/1228399447 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:55:27.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4ce7fc640 10 monclient(hunting): get_auth_request con 0x7ff4d00a2e20 auth_method 0 2024-09-11T01:55:27.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4ce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:27.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:27.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:27.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4cf7fe640 10 monclient(hunting): get_auth_request con 0x7ff4d00a4970 auth_method 0 2024-09-11T01:55:27.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:27.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:27.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:27.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4ce7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:27.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4ce7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:27.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4ce7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:27.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:27.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:27.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4ceffd640 1 --2- 172.21.15.12:0/1228399447 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d006f910 0x7ff4d0137eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:27.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4ceffd640 10 monclient(hunting): get_auth_request con 0x7ff4d006f910 auth_method 0 2024-09-11T01:55:27.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:27.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:27.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.270+0000 7ff4ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:27.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.271+0000 7ff4ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:27.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.271+0000 7ff4ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:27.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.271+0000 7ff4ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:27.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.271+0000 7ff4ce7fc640 10 monclient(hunting): handle_auth_done global_id 284195 payload 824 2024-09-11T01:55:27.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.271+0000 7ff4ce7fc640 1 -- 172.21.15.12:0/1228399447 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d006f910 msgr2=0x7ff4d0137eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.271+0000 7ff4ce7fc640 1 --2- 172.21.15.12:0/1228399447 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d006f910 0x7ff4d0137eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.271+0000 7ff4ce7fc640 1 -- 172.21.15.12:0/1228399447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4d00a4970 msgr2=0x7ff4d01cce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.271+0000 7ff4ce7fc640 1 --2- 172.21.15.12:0/1228399447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4d00a4970 0x7ff4d01cce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.271+0000 7ff4ce7fc640 10 monclient: _finish_hunting 0 2024-09-11T01:55:27.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.271+0000 7ff4ce7fc640 1 monclient: found mon.noname-c 2024-09-11T01:55:27.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.271+0000 7ff4ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:27.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.271+0000 7ff4ce7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:55:27.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.271+0000 7ff4ce7fc640 1 -- 172.21.15.12:0/1228399447 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4d009f610 con 0x7ff4d00a2e20 2024-09-11T01:55:27.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.271+0000 7ff4cf7fe640 1 --2- 172.21.15.12:0/1228399447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4d00a4970 0x7ff4d01cce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:55:27.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.273+0000 7ff4ce7fc640 1 -- 172.21.15.12:0/1228399447 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff4d00a2e20 msgr2=0x7ff4d01383f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:27.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.273+0000 7ff4ce7fc640 1 --2- 172.21.15.12:0/1228399447 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff4d00a2e20 0x7ff4d01383f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff4b800c430 tx=0x7ff4b800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:27.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.273+0000 7ff4c7fff640 1 -- 172.21.15.12:0/1228399447 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b80119a0 con 0x7ff4d00a2e20 2024-09-11T01:55:27.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.273+0000 7ff4c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:27.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.273+0000 7ff4c7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:55:27.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.273+0000 7ff4c7fff640 10 monclient: dump: 2024-09-11T01:55:27.289 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:27.289 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:27.289 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.289 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.289 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:27.289 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.289 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:27.289 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:27.289 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:27.289 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.273+0000 7ff4c7fff640 10 monclient: _finish_auth 0 2024-09-11T01:55:27.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.273+0000 7ff4c7fff640 10 monclient: _check_auth_tickets 2024-09-11T01:55:27.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.273+0000 7ff4c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:27.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.273+0000 7ff4c7fff640 1 -- 172.21.15.12:0/1228399447 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b8017070 con 0x7ff4d00a2e20 2024-09-11T01:55:27.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.273+0000 7ff4d5994640 5 monclient: authenticate success, global_id 284195 2024-09-11T01:55:27.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.273+0000 7ff4d5994640 10 monclient: _renew_subs 2024-09-11T01:55:27.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.273+0000 7ff4d5994640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:55:27.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.273+0000 7ff4d5994640 1 -- 172.21.15.12:0/1228399447 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4d0138eb0 con 0x7ff4d00a2e20 2024-09-11T01:55:27.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.273+0000 7ff4c7fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:27.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.273+0000 7ff4c7fff640 1 -- 172.21.15.12:0/1228399447 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8011d80 con 0x7ff4d00a2e20 2024-09-11T01:55:27.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.273+0000 7ff4c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:27.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.273+0000 7ff4c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:55:27.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.273+0000 7ff4c7fff640 10 monclient: dump: 2024-09-11T01:55:27.291 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:27.291 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:27.291 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.291 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.291 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:27.291 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.291 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:27.291 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:27.291 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:27.291 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.273+0000 7ff4cd7fa640 4 set_mon_vals no callback set 2024-09-11T01:55:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.273+0000 7ff4d5994640 10 monclient: _renew_subs 2024-09-11T01:55:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.273+0000 7ff4d5994640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:55:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.273+0000 7ff4d5994640 1 -- 172.21.15.12:0/1228399447 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4d013f1e0 con 0x7ff4d00a2e20 2024-09-11T01:55:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.274+0000 7ff4d5994640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.274+0000 7ff4d5994640 1 --2- 172.21.15.12:0/1228399447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4d002d440 0x7ff4d002d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.274+0000 7ff4d5994640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.274+0000 7ff4d5994640 1 -- 172.21.15.12:0/1228399447 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d006a060 con 0x7ff4d002d440 2024-09-11T01:55:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.274+0000 7ff4ceffd640 1 --2- 172.21.15.12:0/1228399447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4d002d440 0x7ff4d002d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.274+0000 7ff4c7fff640 1 -- 172.21.15.12:0/1228399447 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff4b801c460 con 0x7ff4d00a2e20 2024-09-11T01:55:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.274+0000 7ff4ceffd640 10 monclient: get_auth_request con 0x7ff4d002d440 auth_method 0 2024-09-11T01:55:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.274+0000 7ff4ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.274+0000 7ff4ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:27.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.274+0000 7ff4ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:27.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.274+0000 7ff4ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:27.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.274+0000 7ff4ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:27.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.274+0000 7ff4ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:27.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.274+0000 7ff4ceffd640 10 monclient(hunting): handle_auth_done global_id 294108 payload 824 2024-09-11T01:55:27.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.274+0000 7ff4ceffd640 1 -- 172.21.15.12:0/1228399447 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff4d006ded0 con 0x7ff4d002d440 2024-09-11T01:55:27.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.274+0000 7ff4c7fff640 1 --2- 172.21.15.12:0/1228399447 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff4b003e290 0x7ff4b0040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.274+0000 7ff4c7fff640 1 -- 172.21.15.12:0/1228399447 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(887..887 src has 676..887) ==== 4803+0+0 (secure 0 0 0) 0x7ff4b8052b40 con 0x7ff4d00a2e20 2024-09-11T01:55:27.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.275+0000 7ff4cf7fe640 1 --2- 172.21.15.12:0/1228399447 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff4b003e290 0x7ff4b0040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.275+0000 7ff4cf7fe640 10 monclient: get_auth_request con 0x7ff4b003e290 auth_method 0 2024-09-11T01:55:27.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.277+0000 7ff4ceffd640 1 -- 172.21.15.12:0/1228399447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4d002d440 msgr2=0x7ff4d002d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:27.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.277+0000 7ff4ceffd640 1 --2- 172.21.15.12:0/1228399447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4d002d440 0x7ff4d002d820 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0011600 tx=0x7ff4c0011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:27.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.277+0000 7ff4c7fff640 1 -- 172.21.15.12:0/1228399447 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0019070 con 0x7ff4d002d440 2024-09-11T01:55:27.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.277+0000 7ff4c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.277+0000 7ff4c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:55:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.277+0000 7ff4c7fff640 10 monclient: dump: 2024-09-11T01:55:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:27.294 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.277+0000 7ff4c7fff640 1 -- 172.21.15.12:0/1228399447 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff4c000fce0 con 0x7ff4d002d440 2024-09-11T01:55:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.277+0000 7ff4c7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:27.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.277+0000 7ff4c7fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:55:27.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.277+0000 7ff4c7fff640 1 -- 172.21.15.12:0/1228399447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4d002d440 msgr2=0x7ff4d002d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.277+0000 7ff4c7fff640 1 --2- 172.21.15.12:0/1228399447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4d002d440 0x7ff4d002d820 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0011600 tx=0x7ff4c0011ac0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.277+0000 7ff4c7fff640 1 -- 172.21.15.12:0/1228399447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4d002d440 msgr2=0x7ff4d002d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:55:27.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.277+0000 7ff4c7fff640 1 --2- 172.21.15.12:0/1228399447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4d002d440 0x7ff4d002d820 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0011600 tx=0x7ff4c0011ac0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.277+0000 7ff4cf7fe640 1 --2- 172.21.15.12:0/1228399447 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff4b003e290 0x7ff4b0040750 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8031000 tx=0x7ff4c802f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.283+0000 7ff4d5994640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:55:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.283+0000 7ff4d5994640 1 --2- 172.21.15.12:0/1228399447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4d002dd60 0x7ff4d002e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.283+0000 7ff4d5994640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.283+0000 7ff4d5994640 1 -- 172.21.15.12:0/1228399447 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d013ce80 con 0x7ff4d002dd60 2024-09-11T01:55:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.283+0000 7ff4ceffd640 1 --2- 172.21.15.12:0/1228399447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4d002dd60 0x7ff4d002e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.283+0000 7ff4ceffd640 10 monclient: get_auth_request con 0x7ff4d002dd60 auth_method 0 2024-09-11T01:55:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.283+0000 7ff4ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.283+0000 7ff4ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.283+0000 7ff4ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.283+0000 7ff4ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.283+0000 7ff4ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:27.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.283+0000 7ff4ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:27.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.283+0000 7ff4ceffd640 10 monclient(hunting): handle_auth_done global_id 294111 payload 824 2024-09-11T01:55:27.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.283+0000 7ff4ceffd640 1 -- 172.21.15.12:0/1228399447 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff4d00a1f40 con 0x7ff4d002dd60 2024-09-11T01:55:27.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.286+0000 7ff4ceffd640 1 -- 172.21.15.12:0/1228399447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4d002dd60 msgr2=0x7ff4d002e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:27.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.286+0000 7ff4ceffd640 1 --2- 172.21.15.12:0/1228399447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4d002dd60 0x7ff4d002e170 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0011600 tx=0x7ff4c0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:27.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.286+0000 7ff4c7fff640 1 -- 172.21.15.12:0/1228399447 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c000f040 con 0x7ff4d002dd60 2024-09-11T01:55:27.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.286+0000 7ff4c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:27.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.286+0000 7ff4c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:55:27.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.286+0000 7ff4c7fff640 10 monclient: dump: 2024-09-11T01:55:27.298 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:27.298 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:27.299 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.299 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.299 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:27.299 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.299 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:27.299 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:27.299 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:27.299 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.286+0000 7ff4c7fff640 1 -- 172.21.15.12:0/1228399447 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4542 (secure 0 0 0) 0x7ff4c0015e30 con 0x7ff4d002dd60 2024-09-11T01:55:27.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.286+0000 7ff4c7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:55:27.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.286+0000 7ff4c7fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:55:27.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.286+0000 7ff4c7fff640 1 -- 172.21.15.12:0/1228399447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4d002dd60 msgr2=0x7ff4d002e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.286+0000 7ff4c7fff640 1 --2- 172.21.15.12:0/1228399447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4d002dd60 0x7ff4d002e170 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0011600 tx=0x7ff4c0004730 comp rx=0 tx=0).stop 2024-09-11T01:55:27.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.286+0000 7ff4c7fff640 1 -- 172.21.15.12:0/1228399447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4d002dd60 msgr2=0x7ff4d002e170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:55:27.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.286+0000 7ff4c7fff640 1 --2- 172.21.15.12:0/1228399447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff4d002dd60 0x7ff4d002e170 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0011600 tx=0x7ff4c0004730 comp rx=0 tx=0).stop 2024-09-11T01:55:27.300 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:55:27.300 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:55:27.300 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:55:27.300 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:55:27.300 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 240, 2024-09-11T01:55:27.300 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 7239, 2024-09-11T01:55:27.301 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:55:27.301 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:55:27.301 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:55:27.301 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:55:27.301 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:27.301 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:55:27.301 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:55:27.301 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:55:27.301 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:55:27.301 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:55:27.301 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:55:27.301 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:55:27.301 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:55:27.301 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:55:27.302 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:55:27.302 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:55:27.302 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:55:27.302 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:55:27.302 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:55:27.302 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:55:27.302 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:27.302 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:55:27.302 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:55:27.302 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:55:27.302 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:55:27.302 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:55:27.302 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:55:27.302 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:55:27.302 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:55:27.303 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:55:27.303 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:55:27.303 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:55:27.303 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:55:27.303 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:55:27.303 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:27.303 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.303 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:55:27.303 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:55:27.303 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:55:27.303 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:55:27.303 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:55:27.304 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:55:27.304 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:55:27.304 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:55:27.304 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:55:27.304 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:55:27.304 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:55:27.304 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:55:27.304 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:55:27.304 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:55:27.304 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:55:27.304 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:55:27.304 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:55:27.305 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:55:27.305 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:55:27.305 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:55:27.305 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:55:27.305 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:55:27.305 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:55:27.305 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:55:27.305 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:55:27.305 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:55:27.305 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:55:27.305 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:55:27.305 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:27.306 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:55:27.306 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.306 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:55:27.306 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.306 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:55:27.306 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:55:27.306 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:55:27.306 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:55:27.306 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.306 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:55:27.306 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:55:27.306 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:27.306 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.306 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.307 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:55:27.307 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:55:27.307 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:27.307 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:27.307 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:27.307 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.307 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:55:27.307 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:55:27.307 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:55:27.307 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:55:27.307 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:55:27.307 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.307 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.307 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:55:27.307 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:55:27.308 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:55:27.308 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:55:27.308 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.308 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:55:27.308 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:55:27.308 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:27.308 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.308 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.308 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:55:27.308 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:55:27.308 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:27.309 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:27.309 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:27.309 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.309 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:55:27.309 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:55:27.309 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:55:27.309 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:55:27.309 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:55:27.309 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.309 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.309 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:55:27.310 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:55:27.310 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:55:27.310 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:55:27.310 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.310 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:55:27.310 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:55:27.310 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:27.310 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.310 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.310 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:55:27.310 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:55:27.310 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:27.310 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:27.310 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:27.310 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.311 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:55:27.311 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:55:27.311 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:55:27.311 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:55:27.311 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:55:27.311 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:27.311 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:27.311 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.311 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:55:27.311 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:55:27.311 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.311 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:55:27.312 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:55:27.312 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:55:27.312 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:27.312 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:27.312 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:55:27.312 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.312 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:55:27.312 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:55:27.312 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:55:27.312 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:27.312 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:27.312 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.312 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:55:27.312 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:55:27.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.286+0000 7ff4d5994640 1 -- 172.21.15.12:0/1228399447 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff4b003e290 msgr2=0x7ff4b0040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.286+0000 7ff4d5994640 1 --2- 172.21.15.12:0/1228399447 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff4b003e290 0x7ff4b0040750 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8031000 tx=0x7ff4c802f000 comp rx=0 tx=0).stop 2024-09-11T01:55:27.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.286+0000 7ff4d5994640 10 monclient: shutdown 2024-09-11T01:55:27.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.286+0000 7ff4d5994640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:27.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.286+0000 7ff4d5994640 1 -- 172.21.15.12:0/1228399447 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff4d00a2e20 msgr2=0x7ff4d01383f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.286+0000 7ff4d5994640 1 --2- 172.21.15.12:0/1228399447 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff4d00a2e20 0x7ff4d01383f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff4b800c430 tx=0x7ff4b800c8f0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.286+0000 7ff4cf7fe640 1 -- 172.21.15.12:0/1228399447 reap_dead start 2024-09-11T01:55:27.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.287+0000 7ff4d5994640 1 -- 172.21.15.12:0/1228399447 shutdown_connections 2024-09-11T01:55:27.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.287+0000 7ff4d5994640 1 --2- 172.21.15.12:0/1228399447 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff4d00a2e20 0x7ff4d01383f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.314 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.287+0000 7ff4d5994640 1 -- 172.21.15.12:0/1228399447 >> 172.21.15.12:0/1228399447 conn(0x7ff4d00a8600 msgr2=0x7ff4d009e730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:27.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.287+0000 7ff4d5994640 1 -- 172.21.15.12:0/1228399447 shutdown_connections 2024-09-11T01:55:27.315 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.287+0000 7ff4d5994640 1 -- 172.21.15.12:0/1228399447 wait complete. 2024-09-11T01:55:27.315 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:55:27.315 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:55:27.315 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:55:27.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.415+0000 7f4c483b4640 10 monclient: get_monmap_and_config 2024-09-11T01:55:27.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.415+0000 7f4c483b4640 10 monclient: build_initial_monmap 2024-09-11T01:55:27.418 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.415+0000 7f4c483b4640 10 monclient: monmap: 2024-09-11T01:55:27.418 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:27.418 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:27.418 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:55:27.416653+0000 2024-09-11T01:55:27.418 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:55:27.416653+0000 2024-09-11T01:55:27.419 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:27.419 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.419 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:55:27.419 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:55:27.419 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:55:27.419 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:55:27.419 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:55:27.420 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.416+0000 7f4c483b4640 1 Processor -- start 2024-09-11T01:55:27.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.416+0000 7f4c483b4640 1 -- start start 2024-09-11T01:55:27.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.416+0000 7f4c483b4640 10 monclient: init 2024-09-11T01:55:27.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c483b4640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:27.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c483b4640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-11T01:55:27.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c483b4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4c401381c0 addr v1:172.21.15.12:6789/0 2024-09-11T01:55:27.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c483b4640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4c400a6aa0 0x7f4c4009cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c483b4640 10 monclient(hunting): picked mon.noname-c con 0x7f4c400a6aa0 addr v2:172.21.15.71:3300/0 2024-09-11T01:55:27.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c483b4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4c4013d000 addr v1:172.21.15.71:6789/0 2024-09-11T01:55:27.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c483b4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c4009c310 con 0x7f4c401381c0 2024-09-11T01:55:27.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c483b4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c4009c8d0 con 0x7f4c4013d000 2024-09-11T01:55:27.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c483b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:27.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c483b4640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f4c4006fc90 con 0x7f4c400a6aa0 2024-09-11T01:55:27.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c483b4640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:27.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c483b4640 10 monclient(hunting): authenticate will time out at 2333.859272s 2024-09-11T01:55:27.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c46150640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4c401381c0 0x7f4c401385a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55618/0 (socket says 172.21.15.12:55618) 2024-09-11T01:55:27.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c46150640 1 -- 172.21.15.12:0/1765053059 learned_addr learned my addr 172.21.15.12:0/1765053059 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:55:27.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c4594f640 1 --2- 172.21.15.12:0/1765053059 >> v2:172.21.15.71:3300/0 conn(0x7f4c400a6aa0 0x7f4c4009cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c4594f640 10 monclient(hunting): get_auth_request con 0x7f4c400a6aa0 auth_method 0 2024-09-11T01:55:27.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c4594f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:27.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c4594f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:27.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c4594f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:27.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c4594f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:27.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c4594f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:27.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c4594f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:27.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c4594f640 10 monclient(hunting): handle_auth_done global_id 274273 payload 274 2024-09-11T01:55:27.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c4594f640 1 -- 172.21.15.12:0/1765053059 >> v1:172.21.15.71:6789/0 conn(0x7f4c4013d000 legacy=0x7f4c4013f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:55:27.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c4594f640 1 -- 172.21.15.12:0/1765053059 >> v1:172.21.15.12:6789/0 conn(0x7f4c401381c0 legacy=0x7f4c401385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:55:27.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c4594f640 10 monclient: _finish_hunting 0 2024-09-11T01:55:27.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c4594f640 1 monclient: found mon.noname-c 2024-09-11T01:55:27.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c4594f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:27.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c4594f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:55:27.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.417+0000 7f4c4594f640 1 -- 172.21.15.12:0/1765053059 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c4009d510 con 0x7f4c400a6aa0 2024-09-11T01:55:27.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.420+0000 7f4c4594f640 1 -- 172.21.15.12:0/1765053059 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c400a6aa0 msgr2=0x7f4c4009cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:27.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.420+0000 7f4c4594f640 1 --2- 172.21.15.12:0/1765053059 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c400a6aa0 0x7f4c4009cfd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4c300097a0 tx=0x7f4c30029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc60251326c6808e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:27.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.421+0000 7f4c4514e640 1 -- 172.21.15.12:0/1765053059 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c30037070 con 0x7f4c400a6aa0 2024-09-11T01:55:27.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.421+0000 7f4c4514e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:27.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.421+0000 7f4c4514e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:55:27.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.421+0000 7f4c4514e640 10 monclient: dump: 2024-09-11T01:55:27.443 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:27.443 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:27.443 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.443 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.443 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:27.444 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.444 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:27.444 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:27.444 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:27.444 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.421+0000 7f4c4514e640 10 monclient: _finish_auth 0 2024-09-11T01:55:27.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.421+0000 7f4c4514e640 10 monclient: _check_auth_tickets 2024-09-11T01:55:27.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.421+0000 7f4c4514e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:27.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.421+0000 7f4c4514e640 1 -- 172.21.15.12:0/1765053059 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c30032640 con 0x7f4c400a6aa0 2024-09-11T01:55:27.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.421+0000 7f4c4514e640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:27.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.421+0000 7f4c4514e640 1 -- 172.21.15.12:0/1765053059 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c30032a10 con 0x7f4c400a6aa0 2024-09-11T01:55:27.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.421+0000 7f4c4514e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:27.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.421+0000 7f4c4514e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:55:27.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.421+0000 7f4c4514e640 10 monclient: dump: 2024-09-11T01:55:27.446 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:27.446 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:27.447 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.447 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.447 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:27.447 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.447 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:27.447 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:27.448 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:27.448 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.421+0000 7f4c483b4640 5 monclient: authenticate success, global_id 274273 2024-09-11T01:55:27.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.421+0000 7f4c483b4640 10 monclient: get_monmap_and_config success 2024-09-11T01:55:27.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.421+0000 7f4c483b4640 4 set_mon_vals no callback set 2024-09-11T01:55:27.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.421+0000 7f4c483b4640 10 monclient: shutdown 2024-09-11T01:55:27.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.421+0000 7f4c483b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:27.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.421+0000 7f4c483b4640 1 -- 172.21.15.12:0/1765053059 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c400a6aa0 msgr2=0x7f4c4009cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.421+0000 7f4c483b4640 1 --2- 172.21.15.12:0/1765053059 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c400a6aa0 0x7f4c4009cfd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4c300097a0 tx=0x7f4c30029fd0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.422+0000 7f4c483b4640 1 -- 172.21.15.12:0/1765053059 shutdown_connections 2024-09-11T01:55:27.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.422+0000 7f4c483b4640 1 --2- 172.21.15.12:0/1765053059 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c400a6aa0 0x7f4c4009cfd0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.422+0000 7f4c483b4640 1 -- 172.21.15.12:0/1765053059 >> 172.21.15.12:0/1765053059 conn(0x7f4c400a7440 msgr2=0x7f4c400a7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:27.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.422+0000 7f4c483b4640 1 -- 172.21.15.12:0/1765053059 shutdown_connections 2024-09-11T01:55:27.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.422+0000 7f4c483b4640 1 -- 172.21.15.12:0/1765053059 wait complete. 2024-09-11T01:55:27.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.422+0000 7f4c483b4640 10 monclient: build_initial_monmap 2024-09-11T01:55:27.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.422+0000 7f4c483b4640 10 monclient: monmap: 2024-09-11T01:55:27.451 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:27.451 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:55:27.451 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:55:27.451 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:55:27.451 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:27.452 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.452 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:55:27.452 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:55:27.452 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:55:27.452 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.422+0000 7f4c483b4640 1 Processor -- start 2024-09-11T01:55:27.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c483b4640 1 -- start start 2024-09-11T01:55:27.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c483b4640 10 monclient: init 2024-09-11T01:55:27.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c483b4640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:27.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c483b4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:55:27.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c483b4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4c400a6aa0 0x7f4c40138d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c483b4640 10 monclient(hunting): picked mon.noname-c con 0x7f4c400a6aa0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:27.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c483b4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c401381c0 0x7f4c401392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c483b4640 10 monclient(hunting): picked mon.noname-a con 0x7f4c401381c0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:55:27.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c483b4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c4013d000 0x7f4c40139810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c483b4640 10 monclient(hunting): picked mon.noname-b con 0x7f4c4013d000 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:55:27.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c483b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:27.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c483b4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4c40140d20 con 0x7f4c401381c0 2024-09-11T01:55:27.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c483b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:27.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c483b4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4009c5f0 con 0x7f4c4013d000 2024-09-11T01:55:27.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c483b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:27.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c483b4640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4c40140ea0 con 0x7f4c400a6aa0 2024-09-11T01:55:27.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c483b4640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:27.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c483b4640 10 monclient(hunting): authenticate will time out at 2333.865455s 2024-09-11T01:55:27.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c4594f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c401381c0 0x7f4c401392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46150640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4c400a6aa0 0x7f4c40138d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46150640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4c400a6aa0 0x7f4c40138d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:44884/0 (socket says 172.21.15.12:44884) 2024-09-11T01:55:27.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c4594f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c401381c0 0x7f4c401392d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46028/0 (socket says 172.21.15.12:46028) 2024-09-11T01:55:27.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46150640 1 -- 172.21.15.12:0/3299772046 learned_addr learned my addr 172.21.15.12:0/3299772046 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:55:27.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46150640 10 monclient(hunting): get_auth_request con 0x7f4c400a6aa0 auth_method 0 2024-09-11T01:55:27.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46150640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:27.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46150640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:27.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46150640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:27.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c4594f640 10 monclient(hunting): get_auth_request con 0x7f4c401381c0 auth_method 0 2024-09-11T01:55:27.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c4594f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:27.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c4594f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:27.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c4594f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:27.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46951640 1 --2- 172.21.15.12:0/3299772046 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c4013d000 0x7f4c40139810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46150640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:27.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46150640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:27.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46150640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:27.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c4594f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:27.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c4594f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:27.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c4594f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:27.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46951640 10 monclient(hunting): get_auth_request con 0x7f4c4013d000 auth_method 0 2024-09-11T01:55:27.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46951640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46951640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46951640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46951640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46951640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46951640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46150640 10 monclient(hunting): handle_auth_done global_id 284198 payload 824 2024-09-11T01:55:27.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46150640 1 -- 172.21.15.12:0/3299772046 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c4013d000 msgr2=0x7f4c40139810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46150640 1 --2- 172.21.15.12:0/3299772046 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c4013d000 0x7f4c40139810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46150640 1 -- 172.21.15.12:0/3299772046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c401381c0 msgr2=0x7f4c401392d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46150640 1 --2- 172.21.15.12:0/3299772046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c401381c0 0x7f4c401392d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46150640 10 monclient: _finish_hunting 0 2024-09-11T01:55:27.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46150640 1 monclient: found mon.noname-c 2024-09-11T01:55:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46150640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46150640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:55:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c46150640 1 -- 172.21.15.12:0/3299772046 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c4013bee0 con 0x7f4c400a6aa0 2024-09-11T01:55:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.423+0000 7f4c4594f640 1 --2- 172.21.15.12:0/3299772046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4c401381c0 0x7f4c401392d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:55:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.426+0000 7f4c46150640 1 -- 172.21.15.12:0/3299772046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4c400a6aa0 msgr2=0x7f4c40138d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.426+0000 7f4c46150640 1 --2- 172.21.15.12:0/3299772046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4c400a6aa0 0x7f4c40138d90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c00c460 tx=0x7f4c3c00c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.426+0000 7f4c36ffd640 1 -- 172.21.15.12:0/3299772046 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c007070 con 0x7f4c400a6aa0 2024-09-11T01:55:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.426+0000 7f4c36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:27.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.426+0000 7f4c36ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:55:27.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.426+0000 7f4c36ffd640 10 monclient: dump: 2024-09-11T01:55:27.463 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:27.463 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:27.463 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.463 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.463 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:27.463 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.463 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:27.464 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:27.464 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:27.464 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.426+0000 7f4c36ffd640 10 monclient: _finish_auth 0 2024-09-11T01:55:27.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.426+0000 7f4c36ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:55:27.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.426+0000 7f4c36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:27.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.426+0000 7f4c483b4640 5 monclient: authenticate success, global_id 284198 2024-09-11T01:55:27.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.426+0000 7f4c36ffd640 1 -- 172.21.15.12:0/3299772046 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c3c007230 con 0x7f4c400a6aa0 2024-09-11T01:55:27.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.426+0000 7f4c483b4640 10 monclient: _renew_subs 2024-09-11T01:55:27.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.426+0000 7f4c483b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:55:27.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.426+0000 7f4c483b4640 1 -- 172.21.15.12:0/3299772046 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c400a4950 con 0x7f4c400a6aa0 2024-09-11T01:55:27.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.426+0000 7f4c36ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:27.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.426+0000 7f4c36ffd640 1 -- 172.21.15.12:0/3299772046 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c007620 con 0x7f4c400a6aa0 2024-09-11T01:55:27.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.426+0000 7f4c36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:27.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.426+0000 7f4c36ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:55:27.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.426+0000 7f4c36ffd640 10 monclient: dump: 2024-09-11T01:55:27.465 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:27.465 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:27.465 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.465 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.465 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:27.465 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.465 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:27.465 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:27.465 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:27.466 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.426+0000 7f4c4494d640 4 set_mon_vals no callback set 2024-09-11T01:55:27.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.426+0000 7f4c483b4640 10 monclient: _renew_subs 2024-09-11T01:55:27.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.426+0000 7f4c483b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:55:27.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.426+0000 7f4c483b4640 1 -- 172.21.15.12:0/3299772046 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c400a4160 con 0x7f4c400a6aa0 2024-09-11T01:55:27.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.427+0000 7f4c483b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:27.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.427+0000 7f4c483b4640 1 --2- 172.21.15.12:0/3299772046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4c40135ec0 0x7f4c401362a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.427+0000 7f4c483b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:27.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.427+0000 7f4c483b4640 1 -- 172.21.15.12:0/3299772046 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4c4009cfd0 con 0x7f4c40135ec0 2024-09-11T01:55:27.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.427+0000 7f4c4594f640 1 --2- 172.21.15.12:0/3299772046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4c40135ec0 0x7f4c401362a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.427+0000 7f4c4594f640 10 monclient: get_auth_request con 0x7f4c40135ec0 auth_method 0 2024-09-11T01:55:27.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.427+0000 7f4c4594f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:27.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.427+0000 7f4c4594f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:27.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.427+0000 7f4c4594f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:27.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.427+0000 7f4c4594f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:27.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.427+0000 7f4c4594f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:27.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.427+0000 7f4c4594f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:27.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.427+0000 7f4c36ffd640 1 -- 172.21.15.12:0/3299772046 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f4c3c01b400 con 0x7f4c400a6aa0 2024-09-11T01:55:27.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.427+0000 7f4c4594f640 10 monclient(hunting): handle_auth_done global_id 284201 payload 824 2024-09-11T01:55:27.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.427+0000 7f4c4594f640 1 -- 172.21.15.12:0/3299772046 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4c400008d0 con 0x7f4c40135ec0 2024-09-11T01:55:27.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.427+0000 7f4c36ffd640 1 --2- 172.21.15.12:0/3299772046 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4c2403e210 0x7f4c240406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.427+0000 7f4c36ffd640 1 -- 172.21.15.12:0/3299772046 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(887..887 src has 676..887) ==== 4803+0+0 (secure 0 0 0) 0x7f4c3c052a80 con 0x7f4c400a6aa0 2024-09-11T01:55:27.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.427+0000 7f4c46951640 1 --2- 172.21.15.12:0/3299772046 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4c2403e210 0x7f4c240406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.428+0000 7f4c46951640 10 monclient: get_auth_request con 0x7f4c2403e210 auth_method 0 2024-09-11T01:55:27.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.429+0000 7f4c4594f640 1 -- 172.21.15.12:0/3299772046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4c40135ec0 msgr2=0x7f4c401362a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:27.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.429+0000 7f4c4594f640 1 --2- 172.21.15.12:0/3299772046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4c40135ec0 0x7f4c401362a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4c30034000 tx=0x7f4c300046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:27.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.429+0000 7f4c36ffd640 1 -- 172.21.15.12:0/3299772046 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c30037040 con 0x7f4c40135ec0 2024-09-11T01:55:27.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.429+0000 7f4c36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:27.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.429+0000 7f4c36ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:55:27.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.429+0000 7f4c36ffd640 10 monclient: dump: 2024-09-11T01:55:27.468 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:27.468 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:27.468 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.468 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.468 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:27.468 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.468 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:27.468 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:27.468 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:27.469 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.430+0000 7f4c36ffd640 1 -- 172.21.15.12:0/3299772046 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4c3003cb10 con 0x7f4c40135ec0 2024-09-11T01:55:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.430+0000 7f4c36ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.430+0000 7f4c36ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:55:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.430+0000 7f4c36ffd640 1 -- 172.21.15.12:0/3299772046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4c40135ec0 msgr2=0x7f4c401362a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.430+0000 7f4c36ffd640 1 --2- 172.21.15.12:0/3299772046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4c40135ec0 0x7f4c401362a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4c30034000 tx=0x7f4c300046c0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.430+0000 7f4c36ffd640 1 -- 172.21.15.12:0/3299772046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4c40135ec0 msgr2=0x7f4c401362a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:55:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.430+0000 7f4c36ffd640 1 --2- 172.21.15.12:0/3299772046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4c40135ec0 0x7f4c401362a0 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4c30034000 tx=0x7f4c300046c0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.430+0000 7f4c46951640 1 --2- 172.21.15.12:0/3299772046 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4c2403e210 0x7f4c240406d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f4c4009d9f0 tx=0x7f4c38003c60 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:27.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.436+0000 7f4c483b4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:55:27.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.436+0000 7f4c483b4640 1 --2- 172.21.15.12:0/3299772046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4c401367e0 0x7f4c4009df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.436+0000 7f4c483b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:27.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.436+0000 7f4c483b4640 1 -- 172.21.15.12:0/3299772046 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4c40136bc0 con 0x7f4c401367e0 2024-09-11T01:55:27.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.436+0000 7f4c4594f640 1 --2- 172.21.15.12:0/3299772046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4c401367e0 0x7f4c4009df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.436+0000 7f4c4594f640 10 monclient: get_auth_request con 0x7f4c401367e0 auth_method 0 2024-09-11T01:55:27.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.436+0000 7f4c4594f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:27.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.436+0000 7f4c4594f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:27.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.436+0000 7f4c4594f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:27.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.436+0000 7f4c4594f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:27.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.436+0000 7f4c4594f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:27.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.436+0000 7f4c4594f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:27.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.436+0000 7f4c4594f640 10 monclient(hunting): handle_auth_done global_id 284204 payload 824 2024-09-11T01:55:27.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.436+0000 7f4c4594f640 1 -- 172.21.15.12:0/3299772046 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4c4013c8f0 con 0x7f4c401367e0 2024-09-11T01:55:27.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.438+0000 7f4c4594f640 1 -- 172.21.15.12:0/3299772046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4c401367e0 msgr2=0x7f4c4009df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:27.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.438+0000 7f4c4594f640 1 --2- 172.21.15.12:0/3299772046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4c401367e0 0x7f4c4009df90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4c30000c00 tx=0x7f4c30008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:27.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.439+0000 7f4c36ffd640 1 -- 172.21.15.12:0/3299772046 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c3004b040 con 0x7f4c401367e0 2024-09-11T01:55:27.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.439+0000 7f4c36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:27.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.439+0000 7f4c36ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:55:27.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.439+0000 7f4c36ffd640 10 monclient: dump: 2024-09-11T01:55:27.473 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:27.473 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:27.473 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.474 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.474 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:27.474 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.474 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:27.474 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:27.474 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:27.474 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.439+0000 7f4c36ffd640 1 -- 172.21.15.12:0/3299772046 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7f4c3004c610 con 0x7f4c401367e0 2024-09-11T01:55:27.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.439+0000 7f4c36ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:55:27.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.439+0000 7f4c36ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:55:27.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.439+0000 7f4c36ffd640 1 -- 172.21.15.12:0/3299772046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4c401367e0 msgr2=0x7f4c4009df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.439+0000 7f4c36ffd640 1 --2- 172.21.15.12:0/3299772046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4c401367e0 0x7f4c4009df90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4c30000c00 tx=0x7f4c30008030 comp rx=0 tx=0).stop 2024-09-11T01:55:27.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.439+0000 7f4c36ffd640 1 -- 172.21.15.12:0/3299772046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4c401367e0 msgr2=0x7f4c4009df90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:55:27.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.439+0000 7f4c36ffd640 1 --2- 172.21.15.12:0/3299772046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4c401367e0 0x7f4c4009df90 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4c30000c00 tx=0x7f4c30008030 comp rx=0 tx=0).stop 2024-09-11T01:55:27.475 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:55:27.475 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:55:27.475 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:55:27.475 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:55:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 240, 2024-09-11T01:55:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 66635, 2024-09-11T01:55:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:55:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:55:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:55:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:55:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:55:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:55:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:55:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:55:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:55:27.476 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:55:27.477 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:55:27.477 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:55:27.477 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:55:27.477 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:55:27.477 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:55:27.477 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:55:27.477 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:55:27.477 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:55:27.477 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:55:27.477 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:27.477 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:55:27.477 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:55:27.478 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:55:27.478 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:55:27.478 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:55:27.478 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:55:27.478 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:55:27.478 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:55:27.478 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:55:27.478 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:55:27.478 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:55:27.478 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:55:27.478 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:55:27.479 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:27.479 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.479 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:55:27.479 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:55:27.479 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:55:27.479 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:55:27.479 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:55:27.479 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:55:27.479 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:55:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:55:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:55:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:55:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:55:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:55:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:55:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:55:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:55:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:55:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:55:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:55:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:55:27.480 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:55:27.481 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:55:27.481 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:55:27.481 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:55:27.481 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:55:27.481 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:55:27.481 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:55:27.481 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:55:27.481 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:55:27.481 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:27.481 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:55:27.481 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.481 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:55:27.482 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.482 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:55:27.482 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:55:27.482 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:55:27.482 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:55:27.482 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.482 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:55:27.482 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:55:27.482 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:27.482 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:55:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:55:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:55:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:55:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:55:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:55:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:55:27.483 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.484 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.484 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:55:27.484 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:55:27.484 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:55:27.484 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:55:27.484 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.484 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:55:27.484 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:55:27.484 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:27.484 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.484 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.484 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:55:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:55:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:55:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:55:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:55:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:55:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:55:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.485 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:55:27.486 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:55:27.486 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:55:27.486 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:55:27.486 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.486 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:55:27.486 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:55:27.486 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:27.486 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.486 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.486 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:55:27.486 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:55:27.486 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:27.486 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:27.487 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:27.487 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.487 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:55:27.487 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:55:27.487 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:55:27.487 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:55:27.487 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:55:27.487 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:27.487 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:27.487 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.487 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:55:27.487 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:55:27.487 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.487 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:55:27.488 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:55:27.488 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:55:27.488 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:27.488 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:27.488 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:55:27.488 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.488 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:55:27.488 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:55:27.488 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:55:27.488 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:27.488 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:27.488 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:55:27.488 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.489 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:55:27.489 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:55:27.489 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:55:27.489 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:27.489 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:27.489 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.489 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:55:27.489 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:55:27.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.439+0000 7f4c483b4640 1 -- 172.21.15.12:0/3299772046 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4c2403e210 msgr2=0x7f4c240406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.439+0000 7f4c483b4640 1 --2- 172.21.15.12:0/3299772046 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4c2403e210 0x7f4c240406d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f4c4009d9f0 tx=0x7f4c38003c60 comp rx=0 tx=0).stop 2024-09-11T01:55:27.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.439+0000 7f4c483b4640 10 monclient: shutdown 2024-09-11T01:55:27.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.439+0000 7f4c483b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:27.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.439+0000 7f4c483b4640 1 -- 172.21.15.12:0/3299772046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4c400a6aa0 msgr2=0x7f4c40138d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.439+0000 7f4c483b4640 1 --2- 172.21.15.12:0/3299772046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4c400a6aa0 0x7f4c40138d90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c00c460 tx=0x7f4c3c00c920 comp rx=0 tx=0).stop 2024-09-11T01:55:27.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.439+0000 7f4c46951640 1 -- 172.21.15.12:0/3299772046 reap_dead start 2024-09-11T01:55:27.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.439+0000 7f4c483b4640 1 -- 172.21.15.12:0/3299772046 shutdown_connections 2024-09-11T01:55:27.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.439+0000 7f4c483b4640 1 --2- 172.21.15.12:0/3299772046 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4c400a6aa0 0x7f4c40138d90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.439+0000 7f4c483b4640 1 -- 172.21.15.12:0/3299772046 >> 172.21.15.12:0/3299772046 conn(0x7f4c400a7440 msgr2=0x7f4c4013fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:27.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.439+0000 7f4c483b4640 1 -- 172.21.15.12:0/3299772046 shutdown_connections 2024-09-11T01:55:27.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.439+0000 7f4c483b4640 1 -- 172.21.15.12:0/3299772046 wait complete. 2024-09-11T01:55:27.492 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:55:27.492 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:55:27.492 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:55:27.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.592+0000 7fdea1764640 10 monclient: get_monmap_and_config 2024-09-11T01:55:27.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.592+0000 7fdea1764640 10 monclient: build_initial_monmap 2024-09-11T01:55:27.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.592+0000 7fdea1764640 10 monclient: monmap: 2024-09-11T01:55:27.594 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:27.594 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:27.595 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:55:27.593531+0000 2024-09-11T01:55:27.595 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:55:27.593531+0000 2024-09-11T01:55:27.595 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:27.595 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.595 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:55:27.595 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:55:27.595 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:55:27.595 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:55:27.595 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:55:27.595 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.593+0000 7fdea1764640 1 Processor -- start 2024-09-11T01:55:27.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.593+0000 7fdea1764640 1 -- start start 2024-09-11T01:55:27.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.593+0000 7fdea1764640 10 monclient: init 2024-09-11T01:55:27.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.593+0000 7fdea1764640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:27.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.593+0000 7fdea1764640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-11T01:55:27.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.593+0000 7fdea1764640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fde9c0a2e20 addr v1:172.21.15.12:6789/0 2024-09-11T01:55:27.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.593+0000 7fdea1764640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fde9c0a4970 addr v1:172.21.15.71:6789/0 2024-09-11T01:55:27.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.593+0000 7fdea1764640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fde9c06f910 0x7fde9c06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.593+0000 7fdea1764640 10 monclient(hunting): picked mon.noname-c con 0x7fde9c06f910 addr v2:172.21.15.71:3300/0 2024-09-11T01:55:27.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.593+0000 7fdea1764640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde9c06f4f0 con 0x7fde9c0a2e20 2024-09-11T01:55:27.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.593+0000 7fdea1764640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde9c0a3910 con 0x7fde9c0a4970 2024-09-11T01:55:27.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.593+0000 7fdea1764640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:27.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.593+0000 7fdea1764640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fde9c06ef30 con 0x7fde9c06f910 2024-09-11T01:55:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.593+0000 7fdea1764640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.593+0000 7fdea1764640 10 monclient(hunting): authenticate will time out at 2334.035969s 2024-09-11T01:55:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.594+0000 7fde9affd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fde9c0a2e20 0x7fde9c0a3200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55622/0 (socket says 172.21.15.12:55622) 2024-09-11T01:55:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.594+0000 7fde9affd640 1 -- 172.21.15.12:0/3967688736 learned_addr learned my addr 172.21.15.12:0/3967688736 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:55:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.594+0000 7fde9b7fe640 1 --2- 172.21.15.12:0/3967688736 >> v2:172.21.15.71:3300/0 conn(0x7fde9c06f910 0x7fde9c06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.594+0000 7fde9b7fe640 10 monclient(hunting): get_auth_request con 0x7fde9c06f910 auth_method 0 2024-09-11T01:55:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.594+0000 7fde9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.594+0000 7fde9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.594+0000 7fde9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.594+0000 7fde9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.594+0000 7fde9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.594+0000 7fde9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.594+0000 7fde9b7fe640 10 monclient(hunting): handle_auth_done global_id 274279 payload 274 2024-09-11T01:55:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.594+0000 7fde9b7fe640 1 -- 172.21.15.12:0/3967688736 >> v1:172.21.15.71:6789/0 conn(0x7fde9c0a4970 legacy=0x7fde9c13d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:55:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.594+0000 7fde9b7fe640 1 -- 172.21.15.12:0/3967688736 >> v1:172.21.15.12:6789/0 conn(0x7fde9c0a2e20 legacy=0x7fde9c0a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:55:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.594+0000 7fde9b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:55:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.594+0000 7fde9b7fe640 1 monclient: found mon.noname-c 2024-09-11T01:55:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.594+0000 7fde9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.594+0000 7fde9b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:55:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.594+0000 7fde9b7fe640 1 -- 172.21.15.12:0/3967688736 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde9c0a3ae0 con 0x7fde9c06f910 2024-09-11T01:55:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.597+0000 7fde9b7fe640 1 -- 172.21.15.12:0/3967688736 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde9c06f910 msgr2=0x7fde9c06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.597+0000 7fde9b7fe640 1 --2- 172.21.15.12:0/3967688736 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde9c06f910 0x7fde9c06ded0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fde94009d90 tx=0x7fde9402a490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3d9c9384eef8e0a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.598+0000 7fde9a7fc640 1 -- 172.21.15.12:0/3967688736 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde94007a30 con 0x7fde9c06f910 2024-09-11T01:55:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.598+0000 7fde9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.598+0000 7fde9a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:55:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.598+0000 7fde9a7fc640 10 monclient: dump: 2024-09-11T01:55:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:27.608 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:27.608 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.598+0000 7fde9a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:55:27.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.598+0000 7fde9a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:55:27.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.598+0000 7fde9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:27.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.598+0000 7fde9a7fc640 1 -- 172.21.15.12:0/3967688736 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde94007bd0 con 0x7fde9c06f910 2024-09-11T01:55:27.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.598+0000 7fde9a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:27.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.598+0000 7fde9a7fc640 1 -- 172.21.15.12:0/3967688736 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde94031400 con 0x7fde9c06f910 2024-09-11T01:55:27.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.598+0000 7fde9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:27.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.598+0000 7fde9a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:55:27.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.598+0000 7fde9a7fc640 10 monclient: dump: 2024-09-11T01:55:27.609 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:27.609 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:27.609 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.609 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:27.610 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.598+0000 7fdea1764640 5 monclient: authenticate success, global_id 274279 2024-09-11T01:55:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.598+0000 7fdea1764640 10 monclient: get_monmap_and_config success 2024-09-11T01:55:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.598+0000 7fdea1764640 4 set_mon_vals no callback set 2024-09-11T01:55:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.598+0000 7fdea1764640 10 monclient: shutdown 2024-09-11T01:55:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.598+0000 7fdea1764640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.598+0000 7fdea1764640 1 -- 172.21.15.12:0/3967688736 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde9c06f910 msgr2=0x7fde9c06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.598+0000 7fdea1764640 1 --2- 172.21.15.12:0/3967688736 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde9c06f910 0x7fde9c06ded0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fde94009d90 tx=0x7fde9402a490 comp rx=0 tx=0).stop 2024-09-11T01:55:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.598+0000 7fdea1764640 1 -- 172.21.15.12:0/3967688736 shutdown_connections 2024-09-11T01:55:27.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.598+0000 7fdea1764640 1 --2- 172.21.15.12:0/3967688736 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde9c06f910 0x7fde9c06ded0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.598+0000 7fdea1764640 1 -- 172.21.15.12:0/3967688736 >> 172.21.15.12:0/3967688736 conn(0x7fde9c0a8600 msgr2=0x7fde9c0a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.599+0000 7fdea1764640 1 -- 172.21.15.12:0/3967688736 shutdown_connections 2024-09-11T01:55:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.599+0000 7fdea1764640 1 -- 172.21.15.12:0/3967688736 wait complete. 2024-09-11T01:55:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.599+0000 7fdea1764640 10 monclient: build_initial_monmap 2024-09-11T01:55:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.599+0000 7fdea1764640 10 monclient: monmap: 2024-09-11T01:55:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:55:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:55:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:55:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:55:27.611 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:55:27.612 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:55:27.612 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.599+0000 7fdea1764640 1 Processor -- start 2024-09-11T01:55:27.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fdea1764640 1 -- start start 2024-09-11T01:55:27.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fdea1764640 10 monclient: init 2024-09-11T01:55:27.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fdea1764640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:27.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fdea1764640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:55:27.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fdea1764640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde9c06f910 0x7fde9c13f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fdea1764640 10 monclient(hunting): picked mon.noname-b con 0x7fde9c06f910 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:55:27.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fdea1764640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde9c0a2e20 0x7fde9c13fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fdea1764640 10 monclient(hunting): picked mon.noname-a con 0x7fde9c0a2e20 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:55:27.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fdea1764640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fde9c0a4970 0x7fde9c1371b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fdea1764640 10 monclient(hunting): picked mon.noname-c con 0x7fde9c0a4970 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:27.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fdea1764640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:27.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fdea1764640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fde9c140d60 con 0x7fde9c0a2e20 2024-09-11T01:55:27.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fdea1764640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:27.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fdea1764640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fde9c06ec50 con 0x7fde9c06f910 2024-09-11T01:55:27.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fdea1764640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:27.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fdea1764640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fde9c140ee0 con 0x7fde9c0a4970 2024-09-11T01:55:27.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fdea1764640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:27.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fdea1764640 10 monclient(hunting): authenticate will time out at 2334.042513s 2024-09-11T01:55:27.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde927fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde9c0a2e20 0x7fde9c13fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde927fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde9c0a2e20 0x7fde9c13fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46034/0 (socket says 172.21.15.12:46034) 2024-09-11T01:55:27.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde9b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fde9c0a4970 0x7fde9c1371b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde927fc640 1 -- 172.21.15.12:0/957019379 learned_addr learned my addr 172.21.15.12:0/957019379 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:55:27.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde927fc640 10 monclient(hunting): get_auth_request con 0x7fde9c0a2e20 auth_method 0 2024-09-11T01:55:27.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde927fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:27.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde927fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:27.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:27.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde9b7fe640 10 monclient(hunting): get_auth_request con 0x7fde9c0a4970 auth_method 0 2024-09-11T01:55:27.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:27.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:27.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:27.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde927fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:27.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde927fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:27.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde927fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:27.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:27.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:27.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde9affd640 1 --2- 172.21.15.12:0/957019379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde9c06f910 0x7fde9c13f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:27.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde9affd640 10 monclient(hunting): get_auth_request con 0x7fde9c06f910 auth_method 0 2024-09-11T01:55:27.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:27.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde9affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:27.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:27.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:27.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:27.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:27.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde927fc640 10 monclient(hunting): handle_auth_done global_id 294117 payload 824 2024-09-11T01:55:27.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde927fc640 1 -- 172.21.15.12:0/957019379 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fde9c0a4970 msgr2=0x7fde9c1371b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde927fc640 1 --2- 172.21.15.12:0/957019379 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fde9c0a4970 0x7fde9c1371b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde927fc640 1 -- 172.21.15.12:0/957019379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde9c06f910 msgr2=0x7fde9c13f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde927fc640 1 --2- 172.21.15.12:0/957019379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde9c06f910 0x7fde9c13f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde927fc640 10 monclient: _finish_hunting 0 2024-09-11T01:55:27.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.600+0000 7fde927fc640 1 monclient: found mon.noname-a 2024-09-11T01:55:27.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.601+0000 7fde927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:27.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.601+0000 7fde927fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:55:27.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.601+0000 7fde927fc640 1 -- 172.21.15.12:0/957019379 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde9c09f610 con 0x7fde9c0a2e20 2024-09-11T01:55:27.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.601+0000 7fde9b7fe640 1 --2- 172.21.15.12:0/957019379 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fde9c0a4970 0x7fde9c1371b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:55:27.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.603+0000 7fde927fc640 1 -- 172.21.15.12:0/957019379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde9c0a2e20 msgr2=0x7fde9c13fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:27.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.603+0000 7fde927fc640 1 --2- 172.21.15.12:0/957019379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde9c0a2e20 0x7fde9c13fb20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fde8400c430 tx=0x7fde8400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:27.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.603+0000 7fde93fff640 1 -- 172.21.15.12:0/957019379 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde840119a0 con 0x7fde9c0a2e20 2024-09-11T01:55:27.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.603+0000 7fde93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:27.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.603+0000 7fde93fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:55:27.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.603+0000 7fde93fff640 10 monclient: dump: 2024-09-11T01:55:27.617 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:27.618 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:27.618 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.618 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.618 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:27.618 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.618 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:27.618 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:27.618 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:27.618 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.603+0000 7fde93fff640 10 monclient: _finish_auth 0 2024-09-11T01:55:27.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.603+0000 7fde93fff640 10 monclient: _check_auth_tickets 2024-09-11T01:55:27.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.603+0000 7fde93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:27.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.603+0000 7fdea1764640 5 monclient: authenticate success, global_id 294117 2024-09-11T01:55:27.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.603+0000 7fde93fff640 1 -- 172.21.15.12:0/957019379 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde84017070 con 0x7fde9c0a2e20 2024-09-11T01:55:27.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.603+0000 7fdea1764640 10 monclient: _renew_subs 2024-09-11T01:55:27.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.603+0000 7fdea1764640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:27.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.603+0000 7fdea1764640 1 -- 172.21.15.12:0/957019379 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde9c140060 con 0x7fde9c0a2e20 2024-09-11T01:55:27.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.603+0000 7fde93fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:27.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.603+0000 7fde93fff640 1 -- 172.21.15.12:0/957019379 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84011d80 con 0x7fde9c0a2e20 2024-09-11T01:55:27.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.603+0000 7fde93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:27.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.603+0000 7fde93fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:55:27.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.603+0000 7fde93fff640 10 monclient: dump: 2024-09-11T01:55:27.619 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:27.619 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:27.619 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.619 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.619 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:27.620 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.620 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:27.620 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:27.620 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:27.620 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.603+0000 7fde997fa640 4 set_mon_vals no callback set 2024-09-11T01:55:27.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.603+0000 7fdea1764640 10 monclient: _renew_subs 2024-09-11T01:55:27.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.603+0000 7fdea1764640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:27.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.603+0000 7fdea1764640 1 -- 172.21.15.12:0/957019379 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde9c1398a0 con 0x7fde9c0a2e20 2024-09-11T01:55:27.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.604+0000 7fdea1764640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:27.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.604+0000 7fdea1764640 1 --2- 172.21.15.12:0/957019379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde9c138b70 0x7fde9c137eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.604+0000 7fdea1764640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:27.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.604+0000 7fdea1764640 1 -- 172.21.15.12:0/957019379 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fde9c06a060 con 0x7fde9c138b70 2024-09-11T01:55:27.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.604+0000 7fde93fff640 1 -- 172.21.15.12:0/957019379 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fde8401c460 con 0x7fde9c0a2e20 2024-09-11T01:55:27.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.604+0000 7fde9affd640 1 --2- 172.21.15.12:0/957019379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde9c138b70 0x7fde9c137eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.604+0000 7fde9affd640 10 monclient: get_auth_request con 0x7fde9c138b70 auth_method 0 2024-09-11T01:55:27.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.604+0000 7fde9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:27.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.604+0000 7fde9affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:27.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.604+0000 7fde9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:27.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.604+0000 7fde9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:27.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.604+0000 7fde9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:27.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.604+0000 7fde9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:27.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.604+0000 7fde93fff640 1 --2- 172.21.15.12:0/957019379 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fde7c03e210 0x7fde7c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.604+0000 7fde93fff640 1 -- 172.21.15.12:0/957019379 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(887..887 src has 676..887) ==== 4803+0+0 (secure 0 0 0) 0x7fde84052c10 con 0x7fde9c0a2e20 2024-09-11T01:55:27.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.604+0000 7fde9affd640 10 monclient(hunting): handle_auth_done global_id 274285 payload 824 2024-09-11T01:55:27.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.604+0000 7fde9affd640 1 -- 172.21.15.12:0/957019379 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fde9c06ded0 con 0x7fde9c138b70 2024-09-11T01:55:27.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.604+0000 7fde9b7fe640 1 --2- 172.21.15.12:0/957019379 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fde7c03e210 0x7fde7c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.604+0000 7fde9b7fe640 10 monclient: get_auth_request con 0x7fde7c03e210 auth_method 0 2024-09-11T01:55:27.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.607+0000 7fde9affd640 1 -- 172.21.15.12:0/957019379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde9c138b70 msgr2=0x7fde9c137eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:27.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.607+0000 7fde9affd640 1 --2- 172.21.15.12:0/957019379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde9c138b70 0x7fde9c137eb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fde8c00ea70 tx=0x7fde8c00ef30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:27.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.607+0000 7fde93fff640 1 -- 172.21.15.12:0/957019379 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8c01a070 con 0x7fde9c138b70 2024-09-11T01:55:27.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.607+0000 7fde93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:27.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.607+0000 7fde9b7fe640 1 --2- 172.21.15.12:0/957019379 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fde7c03e210 0x7fde7c0406d0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fde9c070ba0 tx=0x7fde9402f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:27.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.607+0000 7fde93fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:55:27.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.607+0000 7fde93fff640 10 monclient: dump: 2024-09-11T01:55:27.625 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:27.625 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:27.625 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.625 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.625 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:27.625 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.625 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:27.625 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:27.625 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:27.625 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.607+0000 7fde93fff640 1 -- 172.21.15.12:0/957019379 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fde8c00cc50 con 0x7fde9c138b70 2024-09-11T01:55:27.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.607+0000 7fde93fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:27.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.607+0000 7fde93fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:55:27.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.607+0000 7fde93fff640 1 -- 172.21.15.12:0/957019379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde9c138b70 msgr2=0x7fde9c137eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.607+0000 7fde93fff640 1 --2- 172.21.15.12:0/957019379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde9c138b70 0x7fde9c137eb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fde8c00ea70 tx=0x7fde8c00ef30 comp rx=0 tx=0).stop 2024-09-11T01:55:27.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.607+0000 7fde93fff640 1 -- 172.21.15.12:0/957019379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde9c138b70 msgr2=0x7fde9c137eb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:55:27.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.607+0000 7fde93fff640 1 --2- 172.21.15.12:0/957019379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde9c138b70 0x7fde9c137eb0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.613+0000 7fdea1764640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:55:27.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.614+0000 7fdea1764640 1 --2- 172.21.15.12:0/957019379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde9c1383f0 0x7fde9c02d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.614+0000 7fdea1764640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:27.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.614+0000 7fdea1764640 1 -- 172.21.15.12:0/957019379 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fde9c13a8b0 con 0x7fde9c1383f0 2024-09-11T01:55:27.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.614+0000 7fde9affd640 1 --2- 172.21.15.12:0/957019379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde9c1383f0 0x7fde9c02d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.614+0000 7fde9affd640 10 monclient: get_auth_request con 0x7fde9c1383f0 auth_method 0 2024-09-11T01:55:27.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.614+0000 7fde9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:27.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.614+0000 7fde9affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:27.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.614+0000 7fde9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:27.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.614+0000 7fde9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:27.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.614+0000 7fde9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:27.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.614+0000 7fde9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:27.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.614+0000 7fde9affd640 10 monclient(hunting): handle_auth_done global_id 274288 payload 824 2024-09-11T01:55:27.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.614+0000 7fde9affd640 1 -- 172.21.15.12:0/957019379 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fde9c0a0ef0 con 0x7fde9c1383f0 2024-09-11T01:55:27.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.616+0000 7fde9affd640 1 -- 172.21.15.12:0/957019379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde9c1383f0 msgr2=0x7fde9c02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:27.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.617+0000 7fde9affd640 1 --2- 172.21.15.12:0/957019379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde9c1383f0 0x7fde9c02d440 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fde8c017000 tx=0x7fde8c004940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:27.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.617+0000 7fde93fff640 1 -- 172.21.15.12:0/957019379 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8c01a050 con 0x7fde9c1383f0 2024-09-11T01:55:27.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.617+0000 7fde93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:27.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.617+0000 7fde93fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:55:27.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.617+0000 7fde93fff640 10 monclient: dump: 2024-09-11T01:55:27.628 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:27.628 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:27.628 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.629 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.629 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:27.629 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.629 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:27.629 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:27.629 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:27.629 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.617+0000 7fde93fff640 1 -- 172.21.15.12:0/957019379 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7fde8c01b040 con 0x7fde9c1383f0 2024-09-11T01:55:27.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.617+0000 7fde93fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:55:27.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.617+0000 7fde93fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:55:27.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.617+0000 7fde93fff640 1 -- 172.21.15.12:0/957019379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde9c1383f0 msgr2=0x7fde9c02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.617+0000 7fde93fff640 1 --2- 172.21.15.12:0/957019379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde9c1383f0 0x7fde9c02d440 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fde8c017000 tx=0x7fde8c004940 comp rx=0 tx=0).stop 2024-09-11T01:55:27.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.617+0000 7fde93fff640 1 -- 172.21.15.12:0/957019379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde9c1383f0 msgr2=0x7fde9c02d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:55:27.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.617+0000 7fde93fff640 1 --2- 172.21.15.12:0/957019379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde9c1383f0 0x7fde9c02d440 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fde8c017000 tx=0x7fde8c004940 comp rx=0 tx=0).stop 2024-09-11T01:55:27.630 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:55:27.630 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:55:27.630 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:55:27.630 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:55:27.630 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 240, 2024-09-11T01:55:27.630 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 117434, 2024-09-11T01:55:27.630 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:55:27.630 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:55:27.631 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:55:27.631 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:55:27.631 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:27.631 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:55:27.631 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:55:27.631 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:55:27.631 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:55:27.631 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:55:27.631 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:55:27.631 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:55:27.631 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:55:27.631 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:55:27.631 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:55:27.632 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:55:27.632 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:55:27.632 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:55:27.632 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:55:27.632 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:55:27.632 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:27.632 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:55:27.632 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:55:27.632 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:55:27.632 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:55:27.632 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:55:27.632 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:55:27.632 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:55:27.633 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:55:27.633 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:55:27.633 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:55:27.633 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:55:27.633 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:55:27.633 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:55:27.633 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:27.633 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.633 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:55:27.633 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:55:27.633 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:55:27.633 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:55:27.633 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:55:27.633 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:55:27.634 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:55:27.634 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:55:27.634 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:55:27.634 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:55:27.634 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:55:27.634 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:55:27.634 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:55:27.634 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:55:27.634 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:55:27.634 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:55:27.634 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:55:27.634 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:55:27.634 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:55:27.634 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:55:27.635 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:55:27.635 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:55:27.635 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:55:27.635 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:55:27.635 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:55:27.635 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:55:27.635 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:55:27.635 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:55:27.635 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:27.635 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:55:27.636 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.636 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:55:27.636 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.636 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:55:27.636 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:55:27.636 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:55:27.636 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:55:27.636 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.636 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:55:27.636 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:55:27.636 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:27.636 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.636 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.637 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:55:27.637 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:55:27.637 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:27.637 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:27.637 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:27.637 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.637 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:55:27.637 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:55:27.637 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:55:27.637 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:55:27.638 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:55:27.638 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.638 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.638 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:55:27.638 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:55:27.638 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:55:27.638 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:55:27.638 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.638 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:55:27.638 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:55:27.638 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:27.639 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.639 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.639 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:55:27.639 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:55:27.639 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:27.639 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:27.639 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:27.639 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.639 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:55:27.639 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:55:27.639 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:55:27.639 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:55:27.639 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:55:27.640 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.640 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.640 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:55:27.640 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:55:27.640 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:55:27.640 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:55:27.640 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.640 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:55:27.640 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:55:27.640 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:27.640 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.641 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.641 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:55:27.641 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:55:27.641 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:27.641 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:27.641 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:27.641 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.641 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:55:27.641 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:55:27.641 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:55:27.641 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:55:27.641 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:55:27.641 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:27.642 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:27.642 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.642 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:55:27.642 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:55:27.642 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.642 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:55:27.642 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:55:27.642 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:55:27.642 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:27.642 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:27.642 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:55:27.642 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.643 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:55:27.643 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:55:27.643 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:55:27.643 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:27.643 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:27.643 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:55:27.643 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.643 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:55:27.643 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:55:27.643 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:55:27.643 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:27.643 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:27.643 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:55:27.643 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:27.643 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:55:27.644 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:55:27.644 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:55:27.644 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:27.644 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:27.644 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:27.644 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:55:27.644 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:55:27.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.617+0000 7fdea1764640 1 -- 172.21.15.12:0/957019379 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fde7c03e210 msgr2=0x7fde7c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.617+0000 7fdea1764640 1 --2- 172.21.15.12:0/957019379 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fde7c03e210 0x7fde7c0406d0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fde9c070ba0 tx=0x7fde9402f000 comp rx=0 tx=0).stop 2024-09-11T01:55:27.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.617+0000 7fdea1764640 10 monclient: shutdown 2024-09-11T01:55:27.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.617+0000 7fdea1764640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:27.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.617+0000 7fdea1764640 1 -- 172.21.15.12:0/957019379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde9c0a2e20 msgr2=0x7fde9c13fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.617+0000 7fdea1764640 1 --2- 172.21.15.12:0/957019379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde9c0a2e20 0x7fde9c13fb20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fde8400c430 tx=0x7fde8400c8f0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.617+0000 7fde9b7fe640 1 -- 172.21.15.12:0/957019379 reap_dead start 2024-09-11T01:55:27.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.617+0000 7fde9b7fe640 1 -- 172.21.15.12:0/957019379 reap_dead start 2024-09-11T01:55:27.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.617+0000 7fdea1764640 1 -- 172.21.15.12:0/957019379 shutdown_connections 2024-09-11T01:55:27.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.617+0000 7fdea1764640 1 -- 172.21.15.12:0/957019379 >> 172.21.15.12:0/957019379 conn(0x7fde9c0a8600 msgr2=0x7fde9c09e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:27.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.618+0000 7fdea1764640 1 -- 172.21.15.12:0/957019379 shutdown_connections 2024-09-11T01:55:27.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.618+0000 7fdea1764640 1 -- 172.21.15.12:0/957019379 wait complete. 2024-09-11T01:55:27.646 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:55:27.647 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:55:27.647 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:55:27.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.745+0000 7fa9ffbbb640 10 monclient: get_monmap_and_config 2024-09-11T01:55:27.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.745+0000 7fa9ffbbb640 10 monclient: build_initial_monmap 2024-09-11T01:55:27.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.745+0000 7fa9ffbbb640 10 monclient: monmap: 2024-09-11T01:55:27.749 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:27.749 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:27.750 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:55:27.746690+0000 2024-09-11T01:55:27.750 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:55:27.746690+0000 2024-09-11T01:55:27.750 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:27.750 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.750 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:55:27.750 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:55:27.750 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:55:27.750 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:55:27.750 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:55:27.750 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.746+0000 7fa9ffbbb640 1 Processor -- start 2024-09-11T01:55:27.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.746+0000 7fa9ffbbb640 1 -- start start 2024-09-11T01:55:27.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.746+0000 7fa9ffbbb640 10 monclient: init 2024-09-11T01:55:27.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.747+0000 7fa9ffbbb640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:27.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.747+0000 7fa9ffbbb640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-11T01:55:27.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.747+0000 7fa9ffbbb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa9f80a2d40 addr v1:172.21.15.12:6789/0 2024-09-11T01:55:27.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.747+0000 7fa9ffbbb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa9f806d260 addr v1:172.21.15.71:6789/0 2024-09-11T01:55:27.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.747+0000 7fa9ffbbb640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa9f806d670 0x7fa9f8140c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.747+0000 7fa9ffbbb640 10 monclient(hunting): picked mon.noname-a con 0x7fa9f806d670 addr v2:172.21.15.12:3300/0 2024-09-11T01:55:27.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.747+0000 7fa9ffbbb640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9f809aed0 con 0x7fa9f80a2d40 2024-09-11T01:55:27.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.747+0000 7fa9ffbbb640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9f80a3920 con 0x7fa9f806d260 2024-09-11T01:55:27.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.747+0000 7fa9ffbbb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:27.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.747+0000 7fa9ffbbb640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fa9f806dd10 con 0x7fa9f806d670 2024-09-11T01:55:27.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.747+0000 7fa9ffbbb640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:27.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.747+0000 7fa9ffbbb640 10 monclient(hunting): authenticate will time out at 2334.189221s 2024-09-11T01:55:27.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.747+0000 7fa9fe158640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa9f806d670 0x7fa9f8140c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.747+0000 7fa9fe158640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fa9f806d670 0x7fa9f8140c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46042/0 (socket says 172.21.15.12:46042) 2024-09-11T01:55:27.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.747+0000 7fa9fe158640 1 -- 172.21.15.12:0/1710195798 learned_addr learned my addr 172.21.15.12:0/1710195798 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:55:27.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.747+0000 7fa9fe158640 10 monclient(hunting): get_auth_request con 0x7fa9f806d670 auth_method 0 2024-09-11T01:55:27.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.747+0000 7fa9fe158640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:27.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.747+0000 7fa9fe158640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:27.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.747+0000 7fa9fe158640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:27.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.747+0000 7fa9fe158640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:27.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.747+0000 7fa9fe158640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:27.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.747+0000 7fa9fe158640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:27.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.749+0000 7fa9fd957640 1 -- 172.21.15.12:0/1710195798 >> v1:172.21.15.12:6789/0 conn(0x7fa9f80a2d40 legacy=0x7fa9f806cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:27.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.749+0000 7fa9fe158640 10 monclient(hunting): handle_auth_done global_id 294120 payload 274 2024-09-11T01:55:27.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.749+0000 7fa9fe158640 1 -- 172.21.15.12:0/1710195798 >> v1:172.21.15.71:6789/0 conn(0x7fa9f806d260 legacy=0x7fa9f813d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:55:27.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.749+0000 7fa9fe158640 1 -- 172.21.15.12:0/1710195798 >> v1:172.21.15.12:6789/0 conn(0x7fa9f80a2d40 legacy=0x7fa9f806cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.749+0000 7fa9fe158640 10 monclient: _finish_hunting 0 2024-09-11T01:55:27.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.749+0000 7fa9fe158640 1 monclient: found mon.noname-a 2024-09-11T01:55:27.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.749+0000 7fa9fe158640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:27.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.749+0000 7fa9fe158640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:55:27.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.749+0000 7fa9fe158640 1 -- 172.21.15.12:0/1710195798 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9f80a3320 con 0x7fa9f806d670 2024-09-11T01:55:27.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.752+0000 7fa9fe158640 1 -- 172.21.15.12:0/1710195798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9f806d670 msgr2=0x7fa9f8140c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:27.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.752+0000 7fa9fe158640 1 --2- 172.21.15.12:0/1710195798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9f806d670 0x7fa9f8140c10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0009ed0 tx=0x7fa9f002a6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb845f2ef0fb5e1f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:27.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.752+0000 7fa9fc955640 1 -- 172.21.15.12:0/1710195798 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f0036070 con 0x7fa9f806d670 2024-09-11T01:55:27.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.752+0000 7fa9fc955640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:27.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.752+0000 7fa9fc955640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:55:27.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.752+0000 7fa9fc955640 10 monclient: dump: 2024-09-11T01:55:27.755 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:27.755 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:27.755 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.755 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.755 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:27.755 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.755 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:27.755 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:27.756 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:27.756 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.752+0000 7fa9fc955640 10 monclient: _finish_auth 0 2024-09-11T01:55:27.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.752+0000 7fa9fc955640 10 monclient: _check_auth_tickets 2024-09-11T01:55:27.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.752+0000 7fa9fc955640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:27.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.752+0000 7fa9fc955640 1 -- 172.21.15.12:0/1710195798 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9f0007800 con 0x7fa9f806d670 2024-09-11T01:55:27.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.752+0000 7fa9fc955640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:27.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.752+0000 7fa9fc955640 1 -- 172.21.15.12:0/1710195798 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f0007bd0 con 0x7fa9f806d670 2024-09-11T01:55:27.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.752+0000 7fa9ffbbb640 5 monclient: authenticate success, global_id 294120 2024-09-11T01:55:27.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.752+0000 7fa9ffbbb640 10 monclient: get_monmap_and_config success 2024-09-11T01:55:27.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.752+0000 7fa9ffbbb640 4 set_mon_vals no callback set 2024-09-11T01:55:27.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.752+0000 7fa9fc955640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:27.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.752+0000 7fa9fc955640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:55:27.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.752+0000 7fa9fc955640 10 monclient: dump: 2024-09-11T01:55:27.757 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:27.757 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:27.757 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.757 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.757 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:27.757 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.757 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:27.757 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:27.757 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:27.757 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.753+0000 7fa9ffbbb640 10 monclient: shutdown 2024-09-11T01:55:27.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.753+0000 7fa9ffbbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:27.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.753+0000 7fa9ffbbb640 1 -- 172.21.15.12:0/1710195798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9f806d670 msgr2=0x7fa9f8140c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.753+0000 7fa9ffbbb640 1 --2- 172.21.15.12:0/1710195798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9f806d670 0x7fa9f8140c10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0009ed0 tx=0x7fa9f002a6b0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.753+0000 7fa9ffbbb640 1 -- 172.21.15.12:0/1710195798 shutdown_connections 2024-09-11T01:55:27.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.753+0000 7fa9ffbbb640 1 --2- 172.21.15.12:0/1710195798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9f806d670 0x7fa9f8140c10 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.753+0000 7fa9ffbbb640 1 -- 172.21.15.12:0/1710195798 >> 172.21.15.12:0/1710195798 conn(0x7fa9f80a3e30 msgr2=0x7fa9f80a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:27.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.753+0000 7fa9ffbbb640 1 -- 172.21.15.12:0/1710195798 shutdown_connections 2024-09-11T01:55:27.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.753+0000 7fa9ffbbb640 1 -- 172.21.15.12:0/1710195798 wait complete. 2024-09-11T01:55:27.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.754+0000 7fa9ffbbb640 10 monclient: build_initial_monmap 2024-09-11T01:55:27.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.754+0000 7fa9ffbbb640 10 monclient: monmap: 2024-09-11T01:55:27.764 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:27.764 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:55:27.764 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:55:27.764 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:55:27.764 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:27.765 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.765 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:55:27.765 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:55:27.765 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:55:27.765 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.754+0000 7fa9ffbbb640 1 Processor -- start 2024-09-11T01:55:27.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.754+0000 7fa9ffbbb640 1 -- start start 2024-09-11T01:55:27.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.754+0000 7fa9ffbbb640 10 monclient: init 2024-09-11T01:55:27.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.754+0000 7fa9ffbbb640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:27.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.754+0000 7fa9ffbbb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:55:27.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.754+0000 7fa9ffbbb640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9f806d260 0x7fa9f813ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.754+0000 7fa9ffbbb640 10 monclient(hunting): picked mon.noname-c con 0x7fa9f806d260 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:27.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.754+0000 7fa9ffbbb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9f806d670 0x7fa9f813bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.754+0000 7fa9ffbbb640 10 monclient(hunting): picked mon.noname-a con 0x7fa9f806d670 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:55:27.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.754+0000 7fa9ffbbb640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9f80a2d40 0x7fa9f813c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.754+0000 7fa9ffbbb640 10 monclient(hunting): picked mon.noname-b con 0x7fa9f80a2d40 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:55:27.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.754+0000 7fa9ffbbb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:27.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.754+0000 7fa9ffbbb640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f80a3120 con 0x7fa9f806d670 2024-09-11T01:55:27.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.754+0000 7fa9ffbbb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:27.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.754+0000 7fa9ffbbb640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f809bcf0 con 0x7fa9f80a2d40 2024-09-11T01:55:27.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.754+0000 7fa9ffbbb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:27.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.754+0000 7fa9ffbbb640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa9f809ba10 con 0x7fa9f806d260 2024-09-11T01:55:27.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.754+0000 7fa9ffbbb640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:27.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.754+0000 7fa9ffbbb640 10 monclient(hunting): authenticate will time out at 2334.196854s 2024-09-11T01:55:27.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.754+0000 7fa9fd156640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9f806d670 0x7fa9f813bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.754+0000 7fa9fd957640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9f806d260 0x7fa9f813ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.754+0000 7fa9fd156640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9f806d670 0x7fa9f813bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46056/0 (socket says 172.21.15.12:46056) 2024-09-11T01:55:27.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.754+0000 7fa9fd957640 10 monclient(hunting): get_auth_request con 0x7fa9f806d260 auth_method 0 2024-09-11T01:55:27.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.754+0000 7fa9fd156640 1 -- 172.21.15.12:0/2158198598 learned_addr learned my addr 172.21.15.12:0/2158198598 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:55:27.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fd957640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:27.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fd957640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:27.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fd957640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:27.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fd156640 10 monclient(hunting): get_auth_request con 0x7fa9f806d670 auth_method 0 2024-09-11T01:55:27.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fd156640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:27.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fd156640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:27.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fd156640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:27.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fd957640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:27.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fd957640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:27.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fe158640 1 --2- 172.21.15.12:0/2158198598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9f80a2d40 0x7fa9f813c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fd957640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:27.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fd156640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:27.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fd156640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:27.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fd156640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:27.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fe158640 10 monclient(hunting): get_auth_request con 0x7fa9f80a2d40 auth_method 0 2024-09-11T01:55:27.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fe158640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:27.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fe158640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:27.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fe158640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:27.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fe158640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:27.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fe158640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:27.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fe158640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:27.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fd156640 10 monclient(hunting): handle_auth_done global_id 294123 payload 824 2024-09-11T01:55:27.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fd156640 1 -- 172.21.15.12:0/2158198598 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9f806d260 msgr2=0x7fa9f813ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fd156640 1 --2- 172.21.15.12:0/2158198598 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9f806d260 0x7fa9f813ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fd156640 1 -- 172.21.15.12:0/2158198598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9f80a2d40 msgr2=0x7fa9f813c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:27.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fd156640 1 --2- 172.21.15.12:0/2158198598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9f80a2d40 0x7fa9f813c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:27.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fd156640 10 monclient: _finish_hunting 0 2024-09-11T01:55:27.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fd156640 1 monclient: found mon.noname-a 2024-09-11T01:55:27.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fd156640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:27.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fd156640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:55:27.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fd156640 1 -- 172.21.15.12:0/2158198598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9f813e5a0 con 0x7fa9f806d670 2024-09-11T01:55:27.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.755+0000 7fa9fd957640 1 --2- 172.21.15.12:0/2158198598 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9f806d260 0x7fa9f813ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:55:27.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.757+0000 7fa9fd156640 1 -- 172.21.15.12:0/2158198598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9f806d670 msgr2=0x7fa9f813bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:27.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.757+0000 7fa9fd156640 1 --2- 172.21.15.12:0/2158198598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9f806d670 0x7fa9f813bf70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa9e800c430 tx=0x7fa9e800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:27.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.757+0000 7fa9ee7fc640 1 -- 172.21.15.12:0/2158198598 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e80119a0 con 0x7fa9f806d670 2024-09-11T01:55:27.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.757+0000 7fa9ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:27.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.757+0000 7fa9ee7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:55:27.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.757+0000 7fa9ee7fc640 10 monclient: dump: 2024-09-11T01:55:27.770 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:27.770 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:27.770 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.770 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.771 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:27.771 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.771 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:27.771 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:27.771 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:27.771 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.757+0000 7fa9ee7fc640 10 monclient: _finish_auth 0 2024-09-11T01:55:27.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.757+0000 7fa9ee7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:55:27.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.757+0000 7fa9ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:27.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.757+0000 7fa9ee7fc640 1 -- 172.21.15.12:0/2158198598 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9e8017070 con 0x7fa9f806d670 2024-09-11T01:55:27.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.757+0000 7fa9ffbbb640 5 monclient: authenticate success, global_id 294123 2024-09-11T01:55:27.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.757+0000 7fa9ffbbb640 10 monclient: _renew_subs 2024-09-11T01:55:27.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.757+0000 7fa9ffbbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:27.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.757+0000 7fa9ffbbb640 1 -- 172.21.15.12:0/2158198598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9f809e520 con 0x7fa9f806d670 2024-09-11T01:55:27.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.757+0000 7fa9ee7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:27.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.757+0000 7fa9ee7fc640 1 -- 172.21.15.12:0/2158198598 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e8011d80 con 0x7fa9f806d670 2024-09-11T01:55:27.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.757+0000 7fa9ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:27.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.757+0000 7fa9ee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:55:27.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.757+0000 7fa9ee7fc640 10 monclient: dump: 2024-09-11T01:55:27.772 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:27.772 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:27.772 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.772 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:27.772 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:27.772 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:27.772 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:27.773 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:27.773 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:27.773 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:27.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.757+0000 7fa9effff640 4 set_mon_vals no callback set 2024-09-11T01:55:27.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.757+0000 7fa9ffbbb640 10 monclient: _renew_subs 2024-09-11T01:55:27.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.757+0000 7fa9ffbbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:27.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.757+0000 7fa9ffbbb640 1 -- 172.21.15.12:0/2158198598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9f813cda0 con 0x7fa9f806d670 2024-09-11T01:55:27.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.758+0000 7fa9ee7fc640 1 -- 172.21.15.12:0/2158198598 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fa9e801c460 con 0x7fa9f806d670 2024-09-11T01:55:27.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.758+0000 7fa9ffbbb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:27.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.758+0000 7fa9ffbbb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:27.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.758+0000 7fa9ffbbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:27.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.758+0000 7fa9ffbbb640 1 -- 172.21.15.12:0/2158198598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9f806cbd0 con 0x7fa9f806d670 2024-09-11T01:55:27.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.759+0000 7fa9ee7fc640 1 --2- 172.21.15.12:0/2158198598 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa9dc03e260 0x7fa9dc040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:27.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.759+0000 7fa9ee7fc640 1 -- 172.21.15.12:0/2158198598 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(887..887 src has 676..887) ==== 4803+0+0 (secure 0 0 0) 0x7fa9e8052bc0 con 0x7fa9f806d670 2024-09-11T01:55:27.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.759+0000 7fa9fd957640 1 --2- 172.21.15.12:0/2158198598 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa9dc03e260 0x7fa9dc040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:27.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.759+0000 7fa9fd957640 10 monclient: get_auth_request con 0x7fa9dc03e260 auth_method 0 2024-09-11T01:55:27.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.762+0000 7fa9fd957640 1 --2- 172.21.15.12:0/2158198598 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa9dc03e260 0x7fa9dc040720 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4000f60 tx=0x7fa9f4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:27.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.763+0000 7fa9ee7fc640 1 -- 172.21.15.12:0/2158198598 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa9e801ac60 con 0x7fa9f806d670 2024-09-11T01:55:27.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.763+0000 7fa9ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:27.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:27.763+0000 7fa9ee7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:55:28.025 INFO:tasks.rados.rados.0.smithi071.stdout:3884: done (0 left) 2024-09-11T01:55:28.025 INFO:tasks.rados.rados.0.smithi071.stdout:3885: copy_from oid 20 from oid 26 current snap is 436 2024-09-11T01:55:28.025 INFO:tasks.rados.rados.0.smithi071.stdout:3886: copy_from oid 47 from oid 3 current snap is 436 2024-09-11T01:55:28.025 INFO:tasks.rados.rados.0.smithi071.stdout:3887: copy_from oid 5 from oid 44 current snap is 436 2024-09-11T01:55:28.026 INFO:tasks.rados.rados.0.smithi071.stdout:3888: delete oid 29 current snap is 436 2024-09-11T01:55:28.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:28.025+0000 7fa9ffbbb640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:55:28.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:28.025+0000 7fa9ffbbb640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:55:28.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:28.025+0000 7fa9ffbbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:28.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:28.025+0000 7fa9ffbbb640 1 -- 172.21.15.12:0/2158198598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fa9f813e100 con 0x7fa9f806d670 2024-09-11T01:55:28.028 INFO:tasks.rados.rados.0.smithi071.stdout:3887: finishing copy_from to smithi07134777-5 2024-09-11T01:55:28.028 INFO:tasks.rados.rados.0.smithi071.stdout:3887: got expected ENOENT (src dne) 2024-09-11T01:55:28.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:28.027+0000 7fa9ee7fc640 1 -- 172.21.15.12:0/2158198598 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fa9e80515a0 con 0x7fa9f806d670 2024-09-11T01:55:28.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:28.027+0000 7fa9ee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:55:28.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:28.027+0000 7fa9ee7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:55:28.029 INFO:tasks.rados.rados.0.smithi071.stdout:3887: finishing copy_from racing read to smithi07134777-5 2024-09-11T01:55:28.030 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.029+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.030 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.029+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=10,logm=49} crc {auth=4106507714,config=2586702516,health=3854736661,kv=2884365833,logm=766081585}) 2024-09-11T01:55:28.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.029+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=23,config=2,health=12,kv=11,logm=52} crc {auth=1572654008,config=2586702516,health=3854736661,kv=58696370,logm=1389727148}) 2024-09-11T01:55:28.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.029+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.029+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=10,logm=49} crc {auth=4106507714,config=2586702516,health=3854736661,kv=2884365833,logm=766081585}) 2024-09-11T01:55:28.031 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.029+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=12,kv=10,logm=51} crc {auth=1654632109,config=2586702516,health=3854736661,kv=1566819644,logm=415981291}) 2024-09-11T01:55:28.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:28.030+0000 7fa9ffbbb640 1 -- 172.21.15.12:0/2158198598 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa9dc03e260 msgr2=0x7fa9dc040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:28.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:28.030+0000 7fa9ffbbb640 1 --2- 172.21.15.12:0/2158198598 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa9dc03e260 0x7fa9dc040720 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4000f60 tx=0x7fa9f4009040 comp rx=0 tx=0).stop 2024-09-11T01:55:28.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:28.030+0000 7fa9ffbbb640 10 monclient: shutdown 2024-09-11T01:55:28.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:28.030+0000 7fa9ffbbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:28.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:28.030+0000 7fa9ffbbb640 1 -- 172.21.15.12:0/2158198598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9f806d670 msgr2=0x7fa9f813bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:28.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:28.030+0000 7fa9ffbbb640 1 --2- 172.21.15.12:0/2158198598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9f806d670 0x7fa9f813bf70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa9e800c430 tx=0x7fa9e800c8f0 comp rx=0 tx=0).stop 2024-09-11T01:55:28.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:28.030+0000 7fa9ffbbb640 1 -- 172.21.15.12:0/2158198598 shutdown_connections 2024-09-11T01:55:28.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:28.030+0000 7fa9ffbbb640 1 --2- 172.21.15.12:0/2158198598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9f80a2d40 0x7fa9f813c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:28.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:28.030+0000 7fa9ffbbb640 1 --2- 172.21.15.12:0/2158198598 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa9dc03e260 0x7fa9dc040720 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:28.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:28.030+0000 7fa9ffbbb640 1 --2- 172.21.15.12:0/2158198598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9f806d670 0x7fa9f813bf70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:28.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:28.030+0000 7fa9ffbbb640 1 --2- 172.21.15.12:0/2158198598 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9f806d260 0x7fa9f813ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:28.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:28.030+0000 7fa9ffbbb640 1 -- 172.21.15.12:0/2158198598 >> 172.21.15.12:0/2158198598 conn(0x7fa9f80a3e30 msgr2=0x7fa9f81386c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:28.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:28.030+0000 7fa9ffbbb640 1 -- 172.21.15.12:0/2158198598 shutdown_connections 2024-09-11T01:55:28.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:28.030+0000 7fa9ffbbb640 1 -- 172.21.15.12:0/2158198598 wait complete. 2024-09-11T01:55:28.033 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.031+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.033 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.031+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4203998332}) 2024-09-11T01:55:28.033 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.031+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4083389802}) 2024-09-11T01:55:28.033 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.031+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.034 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.031+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4203998332}) 2024-09-11T01:55:28.034 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.031+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2246870957}) 2024-09-11T01:55:28.034 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.032+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.034 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.032+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1263304850}) 2024-09-11T01:55:28.034 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.032+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3802577966}) 2024-09-11T01:55:28.034 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.032+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.034 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.032+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1263304850}) 2024-09-11T01:55:28.034 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.032+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3451325800}) 2024-09-11T01:55:28.036 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.035+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.036 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.035+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=101545098}) 2024-09-11T01:55:28.037 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.035+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1389612836}) 2024-09-11T01:55:28.037 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.035+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.037 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.035+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=101545098}) 2024-09-11T01:55:28.037 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.035+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2129221723}) 2024-09-11T01:55:28.037 INFO:tasks.rados.rados.0.smithi071.stdout:3887: done (3 left) 2024-09-11T01:55:28.037 INFO:tasks.rados.rados.0.smithi071.stdout:3888: done (2 left) 2024-09-11T01:55:28.038 INFO:tasks.rados.rados.0.smithi071.stdout:3889: snap_create 2024-09-11T01:55:28.038 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.037+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.038 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.037+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2775290392}) 2024-09-11T01:55:28.038 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.037+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1417342327}) 2024-09-11T01:55:28.038 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.037+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.038 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.037+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2775290392}) 2024-09-11T01:55:28.038 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.037+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2345658064}) 2024-09-11T01:55:28.040 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.038+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.040 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.038+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2843886360}) 2024-09-11T01:55:28.040 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.038+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=62226272}) 2024-09-11T01:55:28.040 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.038+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.040 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.038+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2843886360}) 2024-09-11T01:55:28.040 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.038+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1931273369}) 2024-09-11T01:55:28.041 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.039+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.041 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.039+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1978888403}) 2024-09-11T01:55:28.041 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.039+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2369710787}) 2024-09-11T01:55:28.041 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.039+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.041 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.039+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1978888403}) 2024-09-11T01:55:28.041 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.039+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2334354942}) 2024-09-11T01:55:28.042 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.040+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.042 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.040+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1328251930}) 2024-09-11T01:55:28.042 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.040+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1189399725}) 2024-09-11T01:55:28.042 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.040+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.042 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.040+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1328251930}) 2024-09-11T01:55:28.042 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.040+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4001620622}) 2024-09-11T01:55:28.043 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.041+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.043 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.041+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4052889893}) 2024-09-11T01:55:28.043 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.041+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1991807498}) 2024-09-11T01:55:28.043 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.041+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.043 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.041+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4052889893}) 2024-09-11T01:55:28.043 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.041+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2527715434}) 2024-09-11T01:55:28.044 INFO:tasks.rados.rados.0.smithi071.stdout:3885: finishing copy_from racing read to smithi07134777-20 2024-09-11T01:55:28.045 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.042+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.045 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.042+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=427483564}) 2024-09-11T01:55:28.045 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.042+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1000659686}) 2024-09-11T01:55:28.045 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.042+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.045 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.042+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=427483564}) 2024-09-11T01:55:28.045 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.042+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2388634732}) 2024-09-11T01:55:28.045 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.043+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.046 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.043+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1894047395}) 2024-09-11T01:55:28.046 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.043+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1301855561}) 2024-09-11T01:55:28.046 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.043+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.046 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.043+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1894047395}) 2024-09-11T01:55:28.046 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.043+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3565842248}) 2024-09-11T01:55:28.046 INFO:tasks.rados.rados.0.smithi071.stdout:3885: finishing copy_from to smithi07134777-20 2024-09-11T01:55:28.046 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 3026 (ObjNum 1327 snap 432 seq_num 1327) dirty exists 2024-09-11T01:55:28.047 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.044+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.047 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.044+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3753586735}) 2024-09-11T01:55:28.047 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.044+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=157267346}) 2024-09-11T01:55:28.047 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.044+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.048 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.044+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3753586735}) 2024-09-11T01:55:28.048 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.044+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1818512304}) 2024-09-11T01:55:28.048 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.045+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.048 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.045+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3635276308}) 2024-09-11T01:55:28.048 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.045+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=503250723}) 2024-09-11T01:55:28.048 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.045+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.048 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.045+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3635276308}) 2024-09-11T01:55:28.048 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.045+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2972415862}) 2024-09-11T01:55:28.049 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:55:28.050 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.046+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.050 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.046+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2421705957}) 2024-09-11T01:55:28.050 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.046+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=225392500}) 2024-09-11T01:55:28.050 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.046+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.050 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.046+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2421705957}) 2024-09-11T01:55:28.050 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.046+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=858367135}) 2024-09-11T01:55:28.051 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.047+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.051 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.047+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3466665131}) 2024-09-11T01:55:28.051 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.047+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=349271224}) 2024-09-11T01:55:28.051 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.047+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.051 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.047+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3466665131}) 2024-09-11T01:55:28.051 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.047+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3663517443}) 2024-09-11T01:55:28.051 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.048+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.051 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.048+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=485865248}) 2024-09-11T01:55:28.051 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.048+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2602296881}) 2024-09-11T01:55:28.051 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.048+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.051 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.048+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=485865248}) 2024-09-11T01:55:28.051 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.048+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1107069891}) 2024-09-11T01:55:28.052 INFO:tasks.rados.rados.0.smithi071.stdout:3886: finishing copy_from racing read to smithi07134777-47 2024-09-11T01:55:28.052 INFO:tasks.rados.rados.0.smithi071.stdout:3886: finishing copy_from to smithi07134777-47 2024-09-11T01:55:28.052 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 2440 (ObjNum 1300 snap 426 seq_num 1300) dirty exists 2024-09-11T01:55:28.053 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.049+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.053 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.049+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1370314548}) 2024-09-11T01:55:28.053 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.049+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3336774667}) 2024-09-11T01:55:28.054 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.049+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.054 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.049+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1370314548}) 2024-09-11T01:55:28.054 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.049+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4046866284}) 2024-09-11T01:55:28.054 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.050+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.054 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.050+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2157290385}) 2024-09-11T01:55:28.054 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.050+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1318322882}) 2024-09-11T01:55:28.054 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.050+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.054 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.050+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2157290385}) 2024-09-11T01:55:28.054 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.050+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=975418054}) 2024-09-11T01:55:28.054 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.051+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.054 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.051+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2921597346}) 2024-09-11T01:55:28.054 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.051+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=603255442}) 2024-09-11T01:55:28.054 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.051+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.054 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.051+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2921597346}) 2024-09-11T01:55:28.055 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.051+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2715678976}) 2024-09-11T01:55:28.055 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.052+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.055 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.052+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2506744624}) 2024-09-11T01:55:28.056 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.052+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2589160517}) 2024-09-11T01:55:28.056 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.052+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.056 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.052+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2506744624}) 2024-09-11T01:55:28.056 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.052+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1609496239}) 2024-09-11T01:55:28.056 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.053+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.056 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.053+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=852010972}) 2024-09-11T01:55:28.056 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.053+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2949526304}) 2024-09-11T01:55:28.056 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.053+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.056 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.053+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=852010972}) 2024-09-11T01:55:28.056 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.053+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3782289872}) 2024-09-11T01:55:28.057 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.054+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.057 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.054+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3500191620}) 2024-09-11T01:55:28.057 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.054+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=84179007}) 2024-09-11T01:55:28.057 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.054+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.058 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.054+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3500191620}) 2024-09-11T01:55:28.058 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.054+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2508001973}) 2024-09-11T01:55:28.058 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.055+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.058 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.055+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2865796624}) 2024-09-11T01:55:28.058 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.055+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4078914012}) 2024-09-11T01:55:28.058 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.055+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.058 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.055+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2865796624}) 2024-09-11T01:55:28.058 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.055+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=128715896}) 2024-09-11T01:55:28.285 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.283+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.286 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.283+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=12205618}) 2024-09-11T01:55:28.286 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.283+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3307210752}) 2024-09-11T01:55:28.286 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.283+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.286 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.284+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=12205618}) 2024-09-11T01:55:28.286 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.284+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1218096049}) 2024-09-11T01:55:28.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.875+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.875+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=184007463}) 2024-09-11T01:55:28.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.876+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2551171164}) 2024-09-11T01:55:28.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.876+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.876+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=184007463}) 2024-09-11T01:55:28.878 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.876+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2490764161}) 2024-09-11T01:55:28.879 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.877+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.879 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.877+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2982746853}) 2024-09-11T01:55:28.879 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.877+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2532707473}) 2024-09-11T01:55:28.879 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.877+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.879 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.877+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2982746853}) 2024-09-11T01:55:28.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.877+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=944813096}) 2024-09-11T01:55:28.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.878+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.878+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4002338803}) 2024-09-11T01:55:28.880 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.878+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2879279840}) 2024-09-11T01:55:28.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.878+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.878+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4002338803}) 2024-09-11T01:55:28.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.878+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1751063736}) 2024-09-11T01:55:28.881 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.879+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.882 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.879+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3213136083}) 2024-09-11T01:55:28.882 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.879+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=993115326}) 2024-09-11T01:55:28.882 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.879+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.882 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.879+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3213136083}) 2024-09-11T01:55:28.882 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.879+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1690098136}) 2024-09-11T01:55:28.883 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.881+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.881+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2333498208}) 2024-09-11T01:55:28.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.881+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3272236372}) 2024-09-11T01:55:28.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.881+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.881+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2333498208}) 2024-09-11T01:55:28.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.881+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3971556142}) 2024-09-11T01:55:28.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.882+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.882+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2743543248}) 2024-09-11T01:55:28.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.882+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1933291962}) 2024-09-11T01:55:28.884 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.882+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.885 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.882+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2743543248}) 2024-09-11T01:55:28.885 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.882+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2770721176}) 2024-09-11T01:55:28.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.883+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.883+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=207019832}) 2024-09-11T01:55:28.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.883+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2356486581}) 2024-09-11T01:55:28.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.883+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.886 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.883+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=207019832}) 2024-09-11T01:55:28.887 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.883+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2265304719}) 2024-09-11T01:55:28.887 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.884+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.887 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.884+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=893551589}) 2024-09-11T01:55:28.887 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.884+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=850339994}) 2024-09-11T01:55:28.887 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.884+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.887 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.884+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=893551589}) 2024-09-11T01:55:28.887 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.884+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1193582216}) 2024-09-11T01:55:28.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.886+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.886+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3018810521}) 2024-09-11T01:55:28.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.886+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=819354617}) 2024-09-11T01:55:28.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.886+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.886+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3018810521}) 2024-09-11T01:55:28.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.886+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3376584620}) 2024-09-11T01:55:28.889 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.887+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.890 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.887+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1245594129}) 2024-09-11T01:55:28.890 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.887+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3560963976}) 2024-09-11T01:55:28.890 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.887+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.890 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.887+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1245594129}) 2024-09-11T01:55:28.890 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.887+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1937305414}) 2024-09-11T01:55:28.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.888+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.888+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3154967959}) 2024-09-11T01:55:28.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.888+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2256797663}) 2024-09-11T01:55:28.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.888+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.888+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3154967959}) 2024-09-11T01:55:28.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.888+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2877943473}) 2024-09-11T01:55:28.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.889+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.889+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4063641730}) 2024-09-11T01:55:28.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.889+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1606236223}) 2024-09-11T01:55:28.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.889+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.889+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4063641730}) 2024-09-11T01:55:28.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.889+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3773346732}) 2024-09-11T01:55:28.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.891+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.891+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=454371205}) 2024-09-11T01:55:28.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.891+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=141932513}) 2024-09-11T01:55:28.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.891+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.891+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=454371205}) 2024-09-11T01:55:28.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.891+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2884876962}) 2024-09-11T01:55:28.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.892+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.892+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1716366587}) 2024-09-11T01:55:28.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.892+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2715410447}) 2024-09-11T01:55:28.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.892+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.892+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1716366587}) 2024-09-11T01:55:28.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.892+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4069465694}) 2024-09-11T01:55:28.895 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.893+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.895 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.893+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2751793189}) 2024-09-11T01:55:28.895 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.893+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2779739431}) 2024-09-11T01:55:28.895 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.893+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.895 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.893+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2751793189}) 2024-09-11T01:55:28.896 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.893+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=151766334}) 2024-09-11T01:55:28.896 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.895+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.896 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.895+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=868790273}) 2024-09-11T01:55:28.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.895+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3208461144}) 2024-09-11T01:55:28.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.895+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.895+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=868790273}) 2024-09-11T01:55:28.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.895+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2357182411}) 2024-09-11T01:55:28.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.896+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.896+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1301290308}) 2024-09-11T01:55:28.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.896+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1777987042}) 2024-09-11T01:55:28.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.896+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.896+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1301290308}) 2024-09-11T01:55:28.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.896+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=348821518}) 2024-09-11T01:55:28.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.897+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.897+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2401034512}) 2024-09-11T01:55:28.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.897+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1967638482}) 2024-09-11T01:55:28.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.897+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.897+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2401034512}) 2024-09-11T01:55:28.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.897+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3627456675}) 2024-09-11T01:55:28.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.898+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.898+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2367975344}) 2024-09-11T01:55:28.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.898+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=331473227}) 2024-09-11T01:55:28.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.898+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.898+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2367975344}) 2024-09-11T01:55:28.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.898+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1122390849}) 2024-09-11T01:55:28.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.899+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.899+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=245008379}) 2024-09-11T01:55:28.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.899+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2616887383}) 2024-09-11T01:55:28.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.899+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.899+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=245008379}) 2024-09-11T01:55:28.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.899+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1742322332}) 2024-09-11T01:55:28.902 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.901+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.902 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.901+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1695875379}) 2024-09-11T01:55:28.902 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.901+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3848182738}) 2024-09-11T01:55:28.902 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.901+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.902 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.901+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1695875379}) 2024-09-11T01:55:28.903 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.901+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2814975968}) 2024-09-11T01:55:28.903 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.902+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.903 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.902+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3490553950}) 2024-09-11T01:55:28.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.902+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2328734915}) 2024-09-11T01:55:28.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.902+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.902+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3490553950}) 2024-09-11T01:55:28.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.902+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2373424725}) 2024-09-11T01:55:28.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.903+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.903+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2963179021}) 2024-09-11T01:55:28.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.903+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1382857332}) 2024-09-11T01:55:28.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.903+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.903+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2963179021}) 2024-09-11T01:55:28.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.903+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=305117147}) 2024-09-11T01:55:28.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.904+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.904+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2592863245}) 2024-09-11T01:55:28.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.904+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3277412681}) 2024-09-11T01:55:28.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.904+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.904+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2592863245}) 2024-09-11T01:55:28.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.904+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=637246925}) 2024-09-11T01:55:28.907 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.905+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.907 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.905+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1945092800}) 2024-09-11T01:55:28.907 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.905+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2146761079}) 2024-09-11T01:55:28.907 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.905+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.907 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.905+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1945092800}) 2024-09-11T01:55:28.907 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.905+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=543753761}) 2024-09-11T01:55:28.908 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.907+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.908 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.907+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4238985628}) 2024-09-11T01:55:28.908 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.907+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=981306884}) 2024-09-11T01:55:28.908 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.907+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.908 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.907+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4238985628}) 2024-09-11T01:55:28.909 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.907+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=811063522}) 2024-09-11T01:55:28.909 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.908+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.909 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.908+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1464218089}) 2024-09-11T01:55:28.909 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.908+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=488645392}) 2024-09-11T01:55:28.910 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.908+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.910 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.908+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1464218089}) 2024-09-11T01:55:28.910 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.908+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1097118554}) 2024-09-11T01:55:28.910 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.909+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.910 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.909+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=494791003}) 2024-09-11T01:55:28.911 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.909+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=29602898}) 2024-09-11T01:55:28.911 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.909+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.911 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.909+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=494791003}) 2024-09-11T01:55:28.911 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.909+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3388083127}) 2024-09-11T01:55:28.912 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.910+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.912 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.910+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3644162387}) 2024-09-11T01:55:28.912 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.910+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1778257859}) 2024-09-11T01:55:28.912 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.910+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.912 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.910+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3644162387}) 2024-09-11T01:55:28.912 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.910+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=370006349}) 2024-09-11T01:55:28.913 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.911+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.913 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.911+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=552144753}) 2024-09-11T01:55:28.913 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.911+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2449126201}) 2024-09-11T01:55:28.913 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.911+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.913 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.911+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=552144753}) 2024-09-11T01:55:28.913 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.911+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3487950315}) 2024-09-11T01:55:28.914 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.912+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.914 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.912+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2760136198}) 2024-09-11T01:55:28.914 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.912+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=143495787}) 2024-09-11T01:55:28.914 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.912+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.914 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.912+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2760136198}) 2024-09-11T01:55:28.915 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.912+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=184884890}) 2024-09-11T01:55:28.915 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.913+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.915 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.914+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=578701748}) 2024-09-11T01:55:28.915 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.914+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3994311439}) 2024-09-11T01:55:28.915 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.914+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.915 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.914+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=578701748}) 2024-09-11T01:55:28.916 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.914+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3985766552}) 2024-09-11T01:55:28.916 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.915+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.916 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.915+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2081089836}) 2024-09-11T01:55:28.917 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.915+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3315847590}) 2024-09-11T01:55:28.917 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.915+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.917 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.915+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2081089836}) 2024-09-11T01:55:28.917 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.915+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2861084070}) 2024-09-11T01:55:28.918 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.916+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.918 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.916+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1585531218}) 2024-09-11T01:55:28.918 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.916+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=129910608}) 2024-09-11T01:55:28.918 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.916+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.918 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.916+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1585531218}) 2024-09-11T01:55:28.918 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.916+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1824931695}) 2024-09-11T01:55:28.919 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.917+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.919 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.917+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1694711481}) 2024-09-11T01:55:28.919 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.917+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=443052615}) 2024-09-11T01:55:28.919 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.917+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.919 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.917+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1694711481}) 2024-09-11T01:55:28.919 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.917+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2627299942}) 2024-09-11T01:55:28.920 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.918+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.920 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.918+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4268130285}) 2024-09-11T01:55:28.920 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.918+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=47259198}) 2024-09-11T01:55:28.920 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.918+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.921 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.918+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4268130285}) 2024-09-11T01:55:28.921 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.918+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=620795404}) 2024-09-11T01:55:28.922 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.919+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.923 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.919+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2229742916}) 2024-09-11T01:55:28.923 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.919+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1865844299}) 2024-09-11T01:55:28.923 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.919+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.923 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.919+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2229742916}) 2024-09-11T01:55:28.923 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.919+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1016389652}) 2024-09-11T01:55:28.923 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.920+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.923 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.920+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2537159106}) 2024-09-11T01:55:28.923 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.920+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3183828668}) 2024-09-11T01:55:28.923 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.920+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.923 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.920+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2537159106}) 2024-09-11T01:55:28.923 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.920+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=849998126}) 2024-09-11T01:55:28.925 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.921+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.925 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.922+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3429809683}) 2024-09-11T01:55:28.925 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.922+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=846111511}) 2024-09-11T01:55:28.925 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.922+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.925 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.922+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3429809683}) 2024-09-11T01:55:28.925 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.922+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=815858690}) 2024-09-11T01:55:28.925 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.923+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.926 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.923+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1652000896}) 2024-09-11T01:55:28.926 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.923+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=642582255}) 2024-09-11T01:55:28.926 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.923+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.926 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.923+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1652000896}) 2024-09-11T01:55:28.926 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.923+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1523703255}) 2024-09-11T01:55:28.928 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.924+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.928 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.924+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3029702855}) 2024-09-11T01:55:28.928 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.924+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2701159283}) 2024-09-11T01:55:28.928 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.924+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.928 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.924+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3029702855}) 2024-09-11T01:55:28.928 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.924+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2902480560}) 2024-09-11T01:55:28.929 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.925+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.929 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.925+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2966381764}) 2024-09-11T01:55:28.929 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.925+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3323511766}) 2024-09-11T01:55:28.929 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.925+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.929 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.925+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2966381764}) 2024-09-11T01:55:28.929 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.925+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=434392602}) 2024-09-11T01:55:28.929 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.926+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.929 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.926+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=346058309}) 2024-09-11T01:55:28.929 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.926+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=718469275}) 2024-09-11T01:55:28.929 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.926+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.929 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.926+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=346058309}) 2024-09-11T01:55:28.929 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.926+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3802581709}) 2024-09-11T01:55:28.930 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.927+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.927+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=440707216}) 2024-09-11T01:55:28.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.927+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1980593222}) 2024-09-11T01:55:28.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.927+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.927+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=440707216}) 2024-09-11T01:55:28.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.927+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3617702780}) 2024-09-11T01:55:28.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.928+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.928+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=205373489}) 2024-09-11T01:55:28.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.928+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=538825595}) 2024-09-11T01:55:28.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.928+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.928+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=205373489}) 2024-09-11T01:55:28.931 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.928+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2768986653}) 2024-09-11T01:55:28.932 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.930+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.932 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.930+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3796286819}) 2024-09-11T01:55:28.933 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.930+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3566292348}) 2024-09-11T01:55:28.933 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.930+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.933 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.930+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3796286819}) 2024-09-11T01:55:28.933 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.930+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4207833751}) 2024-09-11T01:55:28.933 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.931+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.933 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.931+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980443437}) 2024-09-11T01:55:28.933 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.931+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2621915039}) 2024-09-11T01:55:28.933 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.931+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.933 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.931+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980443437}) 2024-09-11T01:55:28.933 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.931+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3233321890}) 2024-09-11T01:55:28.934 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.932+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.934 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.932+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=688028345}) 2024-09-11T01:55:28.934 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.932+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2555768970}) 2024-09-11T01:55:28.934 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.932+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.934 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.932+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=688028345}) 2024-09-11T01:55:28.934 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.932+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=500477875}) 2024-09-11T01:55:28.935 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.933+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.935 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.933+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1073929532}) 2024-09-11T01:55:28.935 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.933+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3741988033}) 2024-09-11T01:55:28.935 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.933+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.935 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.933+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1073929532}) 2024-09-11T01:55:28.935 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.933+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3551448315}) 2024-09-11T01:55:28.936 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.934+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.936 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.934+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1639882761}) 2024-09-11T01:55:28.936 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.934+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2159149860}) 2024-09-11T01:55:28.936 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.934+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.936 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.934+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1639882761}) 2024-09-11T01:55:28.936 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.934+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4086060116}) 2024-09-11T01:55:28.937 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.935+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.937 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.935+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597282291}) 2024-09-11T01:55:28.937 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.935+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1332407562}) 2024-09-11T01:55:28.937 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.935+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.937 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.935+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597282291}) 2024-09-11T01:55:28.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.935+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3858673498}) 2024-09-11T01:55:28.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.936+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.936+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3548641785}) 2024-09-11T01:55:28.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.936+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1698940731}) 2024-09-11T01:55:28.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.936+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:28.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.936+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3548641785}) 2024-09-11T01:55:28.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:28.936+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3047984863}) 2024-09-11T01:55:29.025 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.023+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.025 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.023+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3640831230}) 2024-09-11T01:55:29.025 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.023+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2421867708}) 2024-09-11T01:55:29.025 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.023+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.026 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.023+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3640831230}) 2024-09-11T01:55:29.026 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.023+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3248062574}) 2024-09-11T01:55:29.050 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:55:29.050 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:55:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.157+0000 7f598e66e640 10 monclient: get_monmap_and_config 2024-09-11T01:55:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.157+0000 7f598e66e640 10 monclient: build_initial_monmap 2024-09-11T01:55:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.157+0000 7f598e66e640 10 monclient: monmap: 2024-09-11T01:55:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:55:29.158339+0000 2024-09-11T01:55:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:55:29.158339+0000 2024-09-11T01:55:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:55:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:55:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:55:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:55:29.160 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:55:29.160 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.158+0000 7f598e66e640 1 Processor -- start 2024-09-11T01:55:29.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.158+0000 7f598e66e640 1 -- start start 2024-09-11T01:55:29.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.158+0000 7f598e66e640 10 monclient: init 2024-09-11T01:55:29.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.158+0000 7f598e66e640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.158+0000 7f598e66e640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-11T01:55:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.158+0000 7f598e66e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f59880a2e90 addr v1:172.21.15.12:6789/0 2024-09-11T01:55:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.158+0000 7f598e66e640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f59880a3270 0x7f59880a3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.158+0000 7f598e66e640 10 monclient(hunting): picked mon.noname-a con 0x7f59880a3270 addr v2:172.21.15.12:3300/0 2024-09-11T01:55:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.158+0000 7f598e66e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f598813d490 0x7f598813f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.158+0000 7f598e66e640 10 monclient(hunting): picked mon.noname-b con 0x7f598813d490 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f598e66e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59880a46c0 con 0x7f59880a2e90 2024-09-11T01:55:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f598e66e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f598e66e640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f59880a4900 con 0x7f59880a3270 2024-09-11T01:55:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f598e66e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f598e66e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f598806ee50 con 0x7f598813d490 2024-09-11T01:55:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f598e66e640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f598e66e640 10 monclient(hunting): authenticate will time out at 2335.601036s 2024-09-11T01:55:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f598cc0b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f598813d490 0x7f598813f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f59877fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f59880a3270 0x7f59880a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f59877fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f59880a3270 0x7f59880a3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46072/0 (socket says 172.21.15.12:46072) 2024-09-11T01:55:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f598cc0b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f598813d490 0x7f598813f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:44918/0 (socket says 172.21.15.12:44918) 2024-09-11T01:55:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f59877fe640 1 -- 172.21.15.12:0/1300336830 learned_addr learned my addr 172.21.15.12:0/1300336830 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:55:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f59877fe640 10 monclient(hunting): get_auth_request con 0x7f59880a3270 auth_method 0 2024-09-11T01:55:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f59877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f59877fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f59877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f598cc0b640 10 monclient(hunting): get_auth_request con 0x7f598813d490 auth_method 0 2024-09-11T01:55:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f598cc0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f598cc0b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f598cc0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f59877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f59877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f59877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f598cc0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f598cc0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f598cc0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f598cc0b640 10 monclient(hunting): handle_auth_done global_id 284213 payload 274 2024-09-11T01:55:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f598cc0b640 1 -- 172.21.15.12:0/1300336830 >> v2:172.21.15.12:3300/0 conn(0x7f59880a3270 msgr2=0x7f59880a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:55:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f598cc0b640 1 --2- 172.21.15.12:0/1300336830 >> v2:172.21.15.12:3300/0 conn(0x7f59880a3270 0x7f59880a3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f598cc0b640 1 -- 172.21.15.12:0/1300336830 >> v1:172.21.15.12:6789/0 conn(0x7f59880a2e90 legacy=0x7f598806dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:55:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f598cc0b640 10 monclient: _finish_hunting 0 2024-09-11T01:55:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f598cc0b640 1 monclient: found mon.noname-b 2024-09-11T01:55:29.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f598cc0b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:29.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f598cc0b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:55:29.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.159+0000 7f598cc0b640 1 -- 172.21.15.12:0/1300336830 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f598806eb70 con 0x7f598813d490 2024-09-11T01:55:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.162+0000 7f598cc0b640 1 -- 172.21.15.12:0/1300336830 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f598813d490 msgr2=0x7f598813f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.162+0000 7f598cc0b640 1 --2- 172.21.15.12:0/1300336830 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f598813d490 0x7f598813f880 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5988071820 tx=0x7f598002a460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cc18b00ba15cb91 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.162+0000 7f5986ffd640 1 -- 172.21.15.12:0/1300336830 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5980007950 con 0x7f598813d490 2024-09-11T01:55:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.162+0000 7f5986ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.162+0000 7f5986ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:55:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.162+0000 7f5986ffd640 10 monclient: dump: 2024-09-11T01:55:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:29.173 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.173 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:29.173 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:29.173 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:29.173 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.162+0000 7f5986ffd640 10 monclient: _finish_auth 0 2024-09-11T01:55:29.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.162+0000 7f5986ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:55:29.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.162+0000 7f5986ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:29.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.162+0000 7f5986ffd640 1 -- 172.21.15.12:0/1300336830 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5980007af0 con 0x7f598813d490 2024-09-11T01:55:29.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.162+0000 7f5986ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:29.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.162+0000 7f5986ffd640 1 -- 172.21.15.12:0/1300336830 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5980031400 con 0x7f598813d490 2024-09-11T01:55:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.162+0000 7f598e66e640 5 monclient: authenticate success, global_id 284213 2024-09-11T01:55:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.162+0000 7f598e66e640 10 monclient: get_monmap_and_config success 2024-09-11T01:55:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.162+0000 7f598e66e640 4 set_mon_vals no callback set 2024-09-11T01:55:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.162+0000 7f5986ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.162+0000 7f5986ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:55:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.162+0000 7f5986ffd640 10 monclient: dump: 2024-09-11T01:55:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.175 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:29.175 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.175 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:29.175 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:29.175 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:29.175 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.163+0000 7f598e66e640 10 monclient: shutdown 2024-09-11T01:55:29.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.163+0000 7f598e66e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:29.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.163+0000 7f598e66e640 1 -- 172.21.15.12:0/1300336830 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f598813d490 msgr2=0x7f598813f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:29.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.163+0000 7f598e66e640 1 --2- 172.21.15.12:0/1300336830 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f598813d490 0x7f598813f880 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5988071820 tx=0x7f598002a460 comp rx=0 tx=0).stop 2024-09-11T01:55:29.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.163+0000 7f598e66e640 1 -- 172.21.15.12:0/1300336830 shutdown_connections 2024-09-11T01:55:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.163+0000 7f598e66e640 1 --2- 172.21.15.12:0/1300336830 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f598813d490 0x7f598813f880 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.163+0000 7f598e66e640 1 --2- 172.21.15.12:0/1300336830 >> v2:172.21.15.12:3300/0 conn(0x7f59880a3270 0x7f59880a3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.163+0000 7f598e66e640 1 -- 172.21.15.12:0/1300336830 >> 172.21.15.12:0/1300336830 conn(0x7f59880a8490 msgr2=0x7f59880a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.163+0000 7f598e66e640 1 -- 172.21.15.12:0/1300336830 shutdown_connections 2024-09-11T01:55:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.163+0000 7f598e66e640 1 -- 172.21.15.12:0/1300336830 wait complete. 2024-09-11T01:55:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.163+0000 7f598e66e640 10 monclient: build_initial_monmap 2024-09-11T01:55:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.163+0000 7f598e66e640 10 monclient: monmap: 2024-09-11T01:55:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:55:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:55:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:55:29.177 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:29.177 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.177 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:55:29.177 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:55:29.177 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:55:29.177 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.164+0000 7f598e66e640 1 Processor -- start 2024-09-11T01:55:29.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.164+0000 7f598e66e640 1 -- start start 2024-09-11T01:55:29.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.164+0000 7f598e66e640 10 monclient: init 2024-09-11T01:55:29.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.164+0000 7f598e66e640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:29.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.164+0000 7f598e66e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:55:29.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.164+0000 7f598e66e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59880a2e90 0x7f59881380b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.164+0000 7f598e66e640 10 monclient(hunting): picked mon.noname-b con 0x7f59880a2e90 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:55:29.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.164+0000 7f598e66e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f59880a3270 0x7f5988139470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.164+0000 7f598e66e640 10 monclient(hunting): picked mon.noname-c con 0x7f59880a3270 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:29.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.164+0000 7f598e66e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f598813d490 0x7f59881399b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.164+0000 7f598e66e640 10 monclient(hunting): picked mon.noname-a con 0x7f598813d490 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:55:29.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.164+0000 7f598e66e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:29.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.164+0000 7f598e66e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f598806fd20 con 0x7f598813d490 2024-09-11T01:55:29.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.164+0000 7f598e66e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:29.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.164+0000 7f598e66e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f598806fba0 con 0x7f59880a2e90 2024-09-11T01:55:29.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.164+0000 7f598e66e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:29.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.164+0000 7f598e66e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f598813fdc0 con 0x7f59880a3270 2024-09-11T01:55:29.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.164+0000 7f598e66e640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:29.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.164+0000 7f598e66e640 10 monclient(hunting): authenticate will time out at 2335.606890s 2024-09-11T01:55:29.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.164+0000 7f59877fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f59880a3270 0x7f5988139470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.164+0000 7f598cc0b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f598813d490 0x7f59881399b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f59877fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f59880a3270 0x7f5988139470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:44930/0 (socket says 172.21.15.12:44930) 2024-09-11T01:55:29.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f59877fe640 1 -- 172.21.15.12:0/1799945483 learned_addr learned my addr 172.21.15.12:0/1799945483 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:55:29.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f59877fe640 10 monclient(hunting): get_auth_request con 0x7f59880a3270 auth_method 0 2024-09-11T01:55:29.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f59877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:29.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f59877fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:29.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f59877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:29.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f598cc0b640 10 monclient(hunting): get_auth_request con 0x7f598813d490 auth_method 0 2024-09-11T01:55:29.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f598cc0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:29.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f598cc0b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f598cc0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f59877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f59877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f59877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f598cc0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f598cc0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f598cc0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f59877fe640 10 monclient(hunting): handle_auth_done global_id 284216 payload 824 2024-09-11T01:55:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f5987fff640 1 --2- 172.21.15.12:0/1799945483 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59880a2e90 0x7f59881380b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f59877fe640 1 -- 172.21.15.12:0/1799945483 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59880a2e90 msgr2=0x7f59881380b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f59877fe640 1 --2- 172.21.15.12:0/1799945483 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59880a2e90 0x7f59881380b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f59877fe640 1 -- 172.21.15.12:0/1799945483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f598813d490 msgr2=0x7f59881399b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f59877fe640 1 --2- 172.21.15.12:0/1799945483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f598813d490 0x7f59881399b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f59877fe640 10 monclient: _finish_hunting 0 2024-09-11T01:55:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f59877fe640 1 monclient: found mon.noname-c 2024-09-11T01:55:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f59877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f59877fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:55:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f59877fe640 1 -- 172.21.15.12:0/1799945483 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f598813bbe0 con 0x7f59880a3270 2024-09-11T01:55:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f5987fff640 10 monclient: get_auth_request con 0x7f59880a2e90 auth_method 0 2024-09-11T01:55:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.165+0000 7f5987fff640 1 --2- 172.21.15.12:0/1799945483 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59880a2e90 0x7f59881380b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:55:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.167+0000 7f59877fe640 1 -- 172.21.15.12:0/1799945483 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f59880a3270 msgr2=0x7f5988139470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.167+0000 7f59877fe640 1 --2- 172.21.15.12:0/1799945483 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f59880a3270 0x7f5988139470 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f597c009fd0 tx=0x7f597c00ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.167+0000 7f5984ff9640 1 -- 172.21.15.12:0/1799945483 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f597c017070 con 0x7f59880a3270 2024-09-11T01:55:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.167+0000 7f5984ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.167+0000 7f5984ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:55:29.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.167+0000 7f5984ff9640 10 monclient: dump: 2024-09-11T01:55:29.183 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:29.183 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:29.183 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.183 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.183 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:29.183 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.183 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:29.183 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:29.183 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:29.183 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.167+0000 7f5984ff9640 10 monclient: _finish_auth 0 2024-09-11T01:55:29.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.167+0000 7f5984ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:55:29.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.167+0000 7f5984ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:29.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.167+0000 7f5984ff9640 1 -- 172.21.15.12:0/1799945483 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f597c00cb50 con 0x7f59880a3270 2024-09-11T01:55:29.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.167+0000 7f598e66e640 5 monclient: authenticate success, global_id 284216 2024-09-11T01:55:29.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.167+0000 7f598e66e640 10 monclient: _renew_subs 2024-09-11T01:55:29.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.167+0000 7f598e66e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:55:29.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.167+0000 7f598e66e640 1 -- 172.21.15.12:0/1799945483 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f598813ce30 con 0x7f59880a3270 2024-09-11T01:55:29.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.167+0000 7f5984ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:29.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.167+0000 7f5984ff9640 1 -- 172.21.15.12:0/1799945483 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f597c013470 con 0x7f59880a3270 2024-09-11T01:55:29.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.167+0000 7f5984ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:29.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.167+0000 7f5984ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:55:29.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.167+0000 7f5984ff9640 10 monclient: dump: 2024-09-11T01:55:29.185 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:29.185 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:29.185 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.185 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.185 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:29.185 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.185 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:29.185 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:29.185 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:29.185 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.167+0000 7f59867fc640 4 set_mon_vals no callback set 2024-09-11T01:55:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.167+0000 7f598e66e640 10 monclient: _renew_subs 2024-09-11T01:55:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.167+0000 7f598e66e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:55:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.167+0000 7f598e66e640 1 -- 172.21.15.12:0/1799945483 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f59880a6420 con 0x7f59880a3270 2024-09-11T01:55:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.168+0000 7f5984ff9640 1 -- 172.21.15.12:0/1799945483 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f597c024060 con 0x7f59880a3270 2024-09-11T01:55:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.168+0000 7f598e66e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.168+0000 7f598e66e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.168+0000 7f598e66e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:55:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.168+0000 7f598e66e640 1 -- 172.21.15.12:0/1799945483 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59880a3680 con 0x7f59880a3270 2024-09-11T01:55:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.169+0000 7f5984ff9640 1 --2- 172.21.15.12:0/1799945483 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f597403e290 0x7f5974040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.169+0000 7f5984ff9640 1 -- 172.21.15.12:0/1799945483 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(888..888 src has 676..888) ==== 4831+0+0 (secure 0 0 0) 0x7f597c052c70 con 0x7f59880a3270 2024-09-11T01:55:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.169+0000 7f5987fff640 1 --2- 172.21.15.12:0/1799945483 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f597403e290 0x7f5974040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.169+0000 7f5987fff640 10 monclient: get_auth_request con 0x7f597403e290 auth_method 0 2024-09-11T01:55:29.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.172+0000 7f5987fff640 1 --2- 172.21.15.12:0/1799945483 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f597403e290 0x7f5974040750 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5978003eb0 tx=0x7f5978009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:29.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.173+0000 7f5984ff9640 1 -- 172.21.15.12:0/1799945483 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f597c051640 con 0x7f59880a3270 2024-09-11T01:55:29.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.173+0000 7f5984ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:29.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.173+0000 7f5984ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:55:29.203 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.201+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.203 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.201+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1534266277}) 2024-09-11T01:55:29.203 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.201+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=39378129}) 2024-09-11T01:55:29.203 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.201+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.204 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.201+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1534266277}) 2024-09-11T01:55:29.204 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.201+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1291589520}) 2024-09-11T01:55:29.293 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.291+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.293 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.291+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=432025430}) 2024-09-11T01:55:29.293 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.291+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2195477312}) 2024-09-11T01:55:29.293 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.291+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.294 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.291+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=432025430}) 2024-09-11T01:55:29.294 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.291+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3284082117}) 2024-09-11T01:55:29.295 INFO:tasks.rados.rados.0.smithi071.stdout:3885: done (2 left) 2024-09-11T01:55:29.295 INFO:tasks.rados.rados.0.smithi071.stdout:3886: done (1 left) 2024-09-11T01:55:29.295 INFO:tasks.rados.rados.0.smithi071.stdout:3889: done (0 left) 2024-09-11T01:55:29.295 INFO:tasks.rados.rados.0.smithi071.stdout:3890: snap_remove snap 411 2024-09-11T01:55:29.329 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.327+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.329 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.327+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2790766447}) 2024-09-11T01:55:29.329 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.327+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=350524903}) 2024-09-11T01:55:29.329 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.327+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.329 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.327+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2790766447}) 2024-09-11T01:55:29.329 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.327+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3624390376}) 2024-09-11T01:55:29.336 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.335+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.336 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.335+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=844868539}) 2024-09-11T01:55:29.337 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.335+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1742179517}) 2024-09-11T01:55:29.337 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.335+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.337 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.335+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=844868539}) 2024-09-11T01:55:29.337 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.335+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2599124960}) 2024-09-11T01:55:29.338 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.336+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.338 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.336+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2209916728}) 2024-09-11T01:55:29.338 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.336+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3768531649}) 2024-09-11T01:55:29.338 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.336+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.338 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.336+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2209916728}) 2024-09-11T01:55:29.339 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.336+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2471059494}) 2024-09-11T01:55:29.339 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.336+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.339 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.336+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3501484179}) 2024-09-11T01:55:29.339 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.336+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1177608831}) 2024-09-11T01:55:29.339 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.336+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.339 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.336+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3501484179}) 2024-09-11T01:55:29.339 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.336+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3642265762}) 2024-09-11T01:55:29.341 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.337+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.341 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.337+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1838817789}) 2024-09-11T01:55:29.341 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.337+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1234318792}) 2024-09-11T01:55:29.342 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.337+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.342 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.337+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1838817789}) 2024-09-11T01:55:29.342 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.337+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=760582070}) 2024-09-11T01:55:29.342 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.338+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.342 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.338+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1724711161}) 2024-09-11T01:55:29.342 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.338+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1712833155}) 2024-09-11T01:55:29.342 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.338+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.342 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.338+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1724711161}) 2024-09-11T01:55:29.342 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.338+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3108008098}) 2024-09-11T01:55:29.342 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.339+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.342 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.339+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3725554354}) 2024-09-11T01:55:29.342 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.339+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2908315005}) 2024-09-11T01:55:29.342 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.339+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.342 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.339+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3725554354}) 2024-09-11T01:55:29.343 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.339+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1543453962}) 2024-09-11T01:55:29.343 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.339+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.343 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.339+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1418389031}) 2024-09-11T01:55:29.343 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.339+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1123055033}) 2024-09-11T01:55:29.343 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.340+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.343 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.340+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1418389031}) 2024-09-11T01:55:29.343 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.340+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=914145803}) 2024-09-11T01:55:29.345 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.340+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.345 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.340+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1480949162}) 2024-09-11T01:55:29.345 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.340+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2884021517}) 2024-09-11T01:55:29.346 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.340+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.346 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.340+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1480949162}) 2024-09-11T01:55:29.346 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.340+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=650308408}) 2024-09-11T01:55:29.346 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.341+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.346 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.341+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4143737688}) 2024-09-11T01:55:29.346 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.341+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=98654501}) 2024-09-11T01:55:29.346 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.341+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.346 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.341+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4143737688}) 2024-09-11T01:55:29.346 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.341+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1683294687}) 2024-09-11T01:55:29.346 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.342+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.346 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.342+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=789848772}) 2024-09-11T01:55:29.346 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.342+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3635480707}) 2024-09-11T01:55:29.346 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.342+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.342+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=789848772}) 2024-09-11T01:55:29.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.342+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4280576895}) 2024-09-11T01:55:29.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.343+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.343+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=655689980}) 2024-09-11T01:55:29.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.343+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=873449058}) 2024-09-11T01:55:29.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.343+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.343+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=655689980}) 2024-09-11T01:55:29.347 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.343+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=668481509}) 2024-09-11T01:55:29.362 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.361+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.362 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.361+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=918122185}) 2024-09-11T01:55:29.363 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.361+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1525998063}) 2024-09-11T01:55:29.363 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.361+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.363 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.361+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=918122185}) 2024-09-11T01:55:29.363 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.361+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=677841671}) 2024-09-11T01:55:29.364 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.362+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.364 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.362+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1218399880}) 2024-09-11T01:55:29.364 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.362+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3728826486}) 2024-09-11T01:55:29.364 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.362+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.364 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.362+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1218399880}) 2024-09-11T01:55:29.364 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.362+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4218227313}) 2024-09-11T01:55:29.365 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.363+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.365 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.363+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3617992695}) 2024-09-11T01:55:29.365 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.363+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1047552566}) 2024-09-11T01:55:29.366 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.363+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.366 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.363+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3617992695}) 2024-09-11T01:55:29.366 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.363+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2236156388}) 2024-09-11T01:55:29.366 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.363+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.366 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.363+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1102841688}) 2024-09-11T01:55:29.366 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.364+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2454812607}) 2024-09-11T01:55:29.366 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.364+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.366 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.364+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1102841688}) 2024-09-11T01:55:29.366 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.364+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2576828540}) 2024-09-11T01:55:29.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.364+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.364+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3456711639}) 2024-09-11T01:55:29.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.364+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2582885730}) 2024-09-11T01:55:29.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.364+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.364+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3456711639}) 2024-09-11T01:55:29.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.364+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3445356598}) 2024-09-11T01:55:29.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.365+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.365+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=230596746}) 2024-09-11T01:55:29.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.365+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2593181999}) 2024-09-11T01:55:29.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.365+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.369 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.365+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=230596746}) 2024-09-11T01:55:29.369 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.365+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1660026735}) 2024-09-11T01:55:29.369 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.366+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.369 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.366+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3550685244}) 2024-09-11T01:55:29.369 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.366+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2073980025}) 2024-09-11T01:55:29.369 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.366+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.369 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.366+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3550685244}) 2024-09-11T01:55:29.369 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.366+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3359891492}) 2024-09-11T01:55:29.374 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.367+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.374 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.367+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=771865969}) 2024-09-11T01:55:29.375 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.367+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3403457368}) 2024-09-11T01:55:29.375 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.367+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.375 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.367+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=771865969}) 2024-09-11T01:55:29.375 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.367+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2682113566}) 2024-09-11T01:55:29.375 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.368+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.375 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.368+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1238757958}) 2024-09-11T01:55:29.375 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.368+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=342116317}) 2024-09-11T01:55:29.375 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.368+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.375 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.368+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1238757958}) 2024-09-11T01:55:29.375 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.368+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1909072185}) 2024-09-11T01:55:29.375 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.368+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.375 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.368+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=583586768}) 2024-09-11T01:55:29.375 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.368+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2157344574}) 2024-09-11T01:55:29.375 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.368+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.376 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.369+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=583586768}) 2024-09-11T01:55:29.376 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.369+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2729203048}) 2024-09-11T01:55:29.447 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.446+0000 7f598e66e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:55:29.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.446+0000 7f598e66e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:55:29.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.446+0000 7f598e66e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:55:29.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.446+0000 7f598e66e640 1 -- 172.21.15.12:0/1799945483 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f59880698e0 con 0x7f59880a3270 2024-09-11T01:55:29.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.446+0000 7f5984ff9640 1 -- 172.21.15.12:0/1799945483 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f597c013dc0 con 0x7f59880a3270 2024-09-11T01:55:29.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.446+0000 7f5984ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:55:29.448 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.446+0000 7f5984ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:55:29.448 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":240,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:55:29.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.448+0000 7f598e66e640 1 -- 172.21.15.12:0/1799945483 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f597403e290 msgr2=0x7f5974040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:29.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.448+0000 7f598e66e640 1 --2- 172.21.15.12:0/1799945483 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f597403e290 0x7f5974040750 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5978003eb0 tx=0x7f5978009040 comp rx=0 tx=0).stop 2024-09-11T01:55:29.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.448+0000 7f598e66e640 10 monclient: shutdown 2024-09-11T01:55:29.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.448+0000 7f598e66e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:29.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.448+0000 7f598e66e640 1 -- 172.21.15.12:0/1799945483 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f59880a3270 msgr2=0x7f5988139470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:29.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.448+0000 7f598e66e640 1 --2- 172.21.15.12:0/1799945483 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f59880a3270 0x7f5988139470 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f597c009fd0 tx=0x7f597c00ed40 comp rx=0 tx=0).stop 2024-09-11T01:55:29.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.448+0000 7f598e66e640 1 -- 172.21.15.12:0/1799945483 shutdown_connections 2024-09-11T01:55:29.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.448+0000 7f598e66e640 1 --2- 172.21.15.12:0/1799945483 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f597403e290 0x7f5974040750 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.448+0000 7f598e66e640 1 --2- 172.21.15.12:0/1799945483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f598813d490 0x7f59881399b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.448+0000 7f598e66e640 1 --2- 172.21.15.12:0/1799945483 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f59880a3270 0x7f5988139470 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.448+0000 7f598e66e640 1 --2- 172.21.15.12:0/1799945483 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59880a2e90 0x7f59881380b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.448+0000 7f598e66e640 1 -- 172.21.15.12:0/1799945483 >> 172.21.15.12:0/1799945483 conn(0x7f59880a8490 msgr2=0x7f59880a6950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:29.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.448+0000 7f598e66e640 1 -- 172.21.15.12:0/1799945483 shutdown_connections 2024-09-11T01:55:29.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.448+0000 7f598e66e640 1 -- 172.21.15.12:0/1799945483 wait complete. 2024-09-11T01:55:29.460 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:55:29.460 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:55:29.460 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:55:29.460 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:55:29.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.560+0000 7f71d2847640 10 monclient: get_monmap_and_config 2024-09-11T01:55:29.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.560+0000 7f71d2847640 10 monclient: build_initial_monmap 2024-09-11T01:55:29.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.560+0000 7f71d2847640 10 monclient: monmap: 2024-09-11T01:55:29.562 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:29.562 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:29.563 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:55:29.561680+0000 2024-09-11T01:55:29.563 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:55:29.561680+0000 2024-09-11T01:55:29.563 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:29.563 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.563 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:55:29.563 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:55:29.563 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:55:29.563 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:55:29.563 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:55:29.563 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.561+0000 7f71d2847640 1 Processor -- start 2024-09-11T01:55:29.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.561+0000 7f71d2847640 1 -- start start 2024-09-11T01:55:29.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.561+0000 7f71d2847640 10 monclient: init 2024-09-11T01:55:29.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d2847640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:29.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d2847640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-11T01:55:29.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d2847640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f71cc0a2d40 addr v1:172.21.15.12:6789/0 2024-09-11T01:55:29.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d2847640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f71cc06d260 0x7f71cc13d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d2847640 10 monclient(hunting): picked mon.noname-a con 0x7f71cc06d260 addr v2:172.21.15.12:3300/0 2024-09-11T01:55:29.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d2847640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f71cc06d670 0x7f71cc13fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d2847640 10 monclient(hunting): picked mon.noname-c con 0x7f71cc06d670 addr v2:172.21.15.71:3300/0 2024-09-11T01:55:29.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d2847640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71cc09aed0 con 0x7f71cc0a2d40 2024-09-11T01:55:29.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d2847640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:29.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d2847640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f71cc06dd10 con 0x7f71cc06d260 2024-09-11T01:55:29.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d2847640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:29.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d2847640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f71cc09bcf0 con 0x7f71cc06d670 2024-09-11T01:55:29.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d2847640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:29.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d2847640 10 monclient(hunting): authenticate will time out at 2336.004245s 2024-09-11T01:55:29.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71cb7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f71cc06d260 0x7f71cc13d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71cb7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f71cc06d260 0x7f71cc13d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46090/0 (socket says 172.21.15.12:46090) 2024-09-11T01:55:29.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71cbfff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f71cc0a2d40 0x7f71cc06cb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55632/0 (socket says 172.21.15.12:55632) 2024-09-11T01:55:29.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71cb7fe640 1 -- 172.21.15.12:0/2458233888 learned_addr learned my addr 172.21.15.12:0/2458233888 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:55:29.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71cb7fe640 10 monclient(hunting): get_auth_request con 0x7f71cc06d260 auth_method 0 2024-09-11T01:55:29.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:29.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:29.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:29.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:29.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:29.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:29.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d0de4640 1 --2- 172.21.15.12:0/2458233888 >> v2:172.21.15.71:3300/0 conn(0x7f71cc06d670 0x7f71cc13fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d0de4640 10 monclient(hunting): get_auth_request con 0x7f71cc06d670 auth_method 0 2024-09-11T01:55:29.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d0de4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:29.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d0de4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:29.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d0de4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:29.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d0de4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:29.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d0de4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:29.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d0de4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:29.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d0de4640 10 monclient(hunting): handle_auth_done global_id 274294 payload 274 2024-09-11T01:55:29.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d0de4640 1 -- 172.21.15.12:0/2458233888 >> v2:172.21.15.12:3300/0 conn(0x7f71cc06d260 msgr2=0x7f71cc13d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:55:29.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d0de4640 1 --2- 172.21.15.12:0/2458233888 >> v2:172.21.15.12:3300/0 conn(0x7f71cc06d260 0x7f71cc13d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d0de4640 1 -- 172.21.15.12:0/2458233888 >> v1:172.21.15.12:6789/0 conn(0x7f71cc0a2d40 legacy=0x7f71cc06cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:55:29.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d0de4640 10 monclient: _finish_hunting 0 2024-09-11T01:55:29.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d0de4640 1 monclient: found mon.noname-c 2024-09-11T01:55:29.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d0de4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:29.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d0de4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:55:29.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.562+0000 7f71d0de4640 1 -- 172.21.15.12:0/2458233888 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71cc0a3920 con 0x7f71cc06d670 2024-09-11T01:55:29.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.565+0000 7f71d0de4640 1 -- 172.21.15.12:0/2458233888 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71cc06d670 msgr2=0x7f71cc13fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:29.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.565+0000 7f71d0de4640 1 --2- 172.21.15.12:0/2458233888 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71cc06d670 0x7f71cc13fa30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f71cc09be70 tx=0x7f71c402a4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f8b3de4feedcddb server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:29.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71caffd640 1 -- 172.21.15.12:0/2458233888 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c4007950 con 0x7f71cc06d670 2024-09-11T01:55:29.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:29.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71caffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:55:29.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71caffd640 10 monclient: dump: 2024-09-11T01:55:29.576 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:29.576 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:29.576 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.576 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.576 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:29.576 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.576 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:29.576 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:29.576 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:29.576 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71caffd640 10 monclient: _finish_auth 0 2024-09-11T01:55:29.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71caffd640 10 monclient: _check_auth_tickets 2024-09-11T01:55:29.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:29.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71caffd640 1 -- 172.21.15.12:0/2458233888 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71c4007af0 con 0x7f71cc06d670 2024-09-11T01:55:29.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71d2847640 5 monclient: authenticate success, global_id 274294 2024-09-11T01:55:29.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71d2847640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:55:29.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71caffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:29.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71d2847640 10 monclient: get_monmap_and_config success 2024-09-11T01:55:29.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71caffd640 1 -- 172.21.15.12:0/2458233888 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c4031400 con 0x7f71cc06d670 2024-09-11T01:55:29.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71d2847640 4 set_mon_vals no callback set 2024-09-11T01:55:29.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:29.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71caffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:55:29.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71caffd640 10 monclient: dump: 2024-09-11T01:55:29.577 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:29.578 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:29.578 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.578 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.578 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:29.578 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.578 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:29.578 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:29.578 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:29.578 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71d2847640 10 monclient: shutdown 2024-09-11T01:55:29.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71d2847640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:29.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71d2847640 1 -- 172.21.15.12:0/2458233888 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71cc06d670 msgr2=0x7f71cc13fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:29.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71d2847640 1 --2- 172.21.15.12:0/2458233888 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71cc06d670 0x7f71cc13fa30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f71cc09be70 tx=0x7f71c402a4b0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71d2847640 1 -- 172.21.15.12:0/2458233888 shutdown_connections 2024-09-11T01:55:29.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71d2847640 1 --2- 172.21.15.12:0/2458233888 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71cc06d670 0x7f71cc13fa30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71d2847640 1 --2- 172.21.15.12:0/2458233888 >> v2:172.21.15.12:3300/0 conn(0x7f71cc06d260 0x7f71cc13d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71d2847640 1 -- 172.21.15.12:0/2458233888 >> 172.21.15.12:0/2458233888 conn(0x7f71cc0a3e30 msgr2=0x7f71cc0a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:29.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71d2847640 1 -- 172.21.15.12:0/2458233888 shutdown_connections 2024-09-11T01:55:29.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.566+0000 7f71d2847640 1 -- 172.21.15.12:0/2458233888 wait complete. 2024-09-11T01:55:29.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.567+0000 7f71d2847640 10 monclient: build_initial_monmap 2024-09-11T01:55:29.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.567+0000 7f71d2847640 10 monclient: monmap: 2024-09-11T01:55:29.579 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:29.579 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:55:29.579 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:55:29.579 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:55:29.579 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:29.579 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.580 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:55:29.580 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:55:29.580 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:55:29.580 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.567+0000 7f71d2847640 1 Processor -- start 2024-09-11T01:55:29.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.567+0000 7f71d2847640 1 -- start start 2024-09-11T01:55:29.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.567+0000 7f71d2847640 10 monclient: init 2024-09-11T01:55:29.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71d2847640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:29.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71d2847640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:55:29.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71d2847640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71cc06d260 0x7f71cc13f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71d2847640 10 monclient(hunting): picked mon.noname-a con 0x7f71cc06d260 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:55:29.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71d2847640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f71cc06d670 0x7f71cc137a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71d2847640 10 monclient(hunting): picked mon.noname-c con 0x7f71cc06d670 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:29.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71d2847640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71cc0a2d40 0x7f71cc137fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71d2847640 10 monclient(hunting): picked mon.noname-b con 0x7f71cc0a2d40 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:55:29.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71d2847640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:29.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71d2847640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f71cc09ba10 con 0x7f71cc06d260 2024-09-11T01:55:29.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71d2847640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:29.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71d2847640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f71cc0a3120 con 0x7f71cc0a2d40 2024-09-11T01:55:29.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71d2847640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:29.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71d2847640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f71cc09b3a0 con 0x7f71cc06d670 2024-09-11T01:55:29.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71d2847640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:29.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71d2847640 10 monclient(hunting): authenticate will time out at 2336.010316s 2024-09-11T01:55:29.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71cc06d260 0x7f71cc13f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71cc06d260 0x7f71cc13f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46092/0 (socket says 172.21.15.12:46092) 2024-09-11T01:55:29.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cbfff640 1 -- 172.21.15.12:0/3042846897 learned_addr learned my addr 172.21.15.12:0/3042846897 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:55:29.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cbfff640 10 monclient(hunting): get_auth_request con 0x7f71cc06d260 auth_method 0 2024-09-11T01:55:29.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:29.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:29.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:29.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cb7fe640 1 --2- 172.21.15.12:0/3042846897 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f71cc06d670 0x7f71cc137a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cb7fe640 10 monclient(hunting): get_auth_request con 0x7f71cc06d670 auth_method 0 2024-09-11T01:55:29.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:29.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:29.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:29.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:29.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:29.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71d0de4640 1 --2- 172.21.15.12:0/3042846897 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71cc0a2d40 0x7f71cc137fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:29.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71d0de4640 10 monclient(hunting): get_auth_request con 0x7f71cc0a2d40 auth_method 0 2024-09-11T01:55:29.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71d0de4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:29.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71d0de4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:29.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71d0de4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:29.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:29.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:29.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:29.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71d0de4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:29.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71d0de4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:29.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71d0de4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:29.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cbfff640 10 monclient(hunting): handle_auth_done global_id 294135 payload 824 2024-09-11T01:55:29.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cbfff640 1 -- 172.21.15.12:0/3042846897 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f71cc06d670 msgr2=0x7f71cc137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:29.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cbfff640 1 --2- 172.21.15.12:0/3042846897 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f71cc06d670 0x7f71cc137a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cbfff640 1 -- 172.21.15.12:0/3042846897 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71cc0a2d40 msgr2=0x7f71cc137fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:29.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cbfff640 1 --2- 172.21.15.12:0/3042846897 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71cc0a2d40 0x7f71cc137fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cbfff640 10 monclient: _finish_hunting 0 2024-09-11T01:55:29.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cbfff640 1 monclient: found mon.noname-a 2024-09-11T01:55:29.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:29.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:55:29.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.568+0000 7f71cbfff640 1 -- 172.21.15.12:0/3042846897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71cc074d40 con 0x7f71cc06d260 2024-09-11T01:55:29.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.570+0000 7f71cbfff640 1 -- 172.21.15.12:0/3042846897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71cc06d260 msgr2=0x7f71cc13f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:29.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.570+0000 7f71cbfff640 1 --2- 172.21.15.12:0/3042846897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71cc06d260 0x7f71cc13f390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f71bc00c450 tx=0x7f71bc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:29.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.571+0000 7f71c8ff9640 1 -- 172.21.15.12:0/3042846897 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71bc017070 con 0x7f71cc06d260 2024-09-11T01:55:29.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.571+0000 7f71c8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:29.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.571+0000 7f71c8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:55:29.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.571+0000 7f71c8ff9640 10 monclient: dump: 2024-09-11T01:55:29.584 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:29.584 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:29.585 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.585 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.585 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:29.585 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.585 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:29.585 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:29.585 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:29.585 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.571+0000 7f71c8ff9640 10 monclient: _finish_auth 0 2024-09-11T01:55:29.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.571+0000 7f71c8ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:55:29.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.571+0000 7f71c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:29.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.571+0000 7f71c8ff9640 1 -- 172.21.15.12:0/3042846897 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71bc005810 con 0x7f71cc06d260 2024-09-11T01:55:29.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.571+0000 7f71d2847640 5 monclient: authenticate success, global_id 294135 2024-09-11T01:55:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.571+0000 7f71d2847640 10 monclient: _renew_subs 2024-09-11T01:55:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.571+0000 7f71d2847640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.571+0000 7f71d2847640 1 -- 172.21.15.12:0/3042846897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71cc13e120 con 0x7f71cc06d260 2024-09-11T01:55:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.571+0000 7f71c8ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.571+0000 7f71c8ff9640 1 -- 172.21.15.12:0/3042846897 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71bc007690 con 0x7f71cc06d260 2024-09-11T01:55:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.571+0000 7f71c8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.571+0000 7f71c8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:55:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.571+0000 7f71c8ff9640 10 monclient: dump: 2024-09-11T01:55:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:29.586 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.587 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:29.587 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:29.587 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:29.587 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.571+0000 7f71d2847640 10 monclient: _renew_subs 2024-09-11T01:55:29.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.571+0000 7f71d2847640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:29.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.571+0000 7f71d2847640 1 -- 172.21.15.12:0/3042846897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71cc13d1c0 con 0x7f71cc06d260 2024-09-11T01:55:29.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.571+0000 7f71ca7fc640 4 set_mon_vals no callback set 2024-09-11T01:55:29.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.571+0000 7f71d2847640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:29.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.571+0000 7f71d2847640 1 --2- 172.21.15.12:0/3042846897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71cc13e640 0x7f71cc1cd110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.571+0000 7f71d2847640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:29.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.571+0000 7f71d2847640 1 -- 172.21.15.12:0/3042846897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f71cc09a980 con 0x7f71cc13e640 2024-09-11T01:55:29.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.572+0000 7f71cb7fe640 1 --2- 172.21.15.12:0/3042846897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71cc13e640 0x7f71cc1cd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.572+0000 7f71cb7fe640 10 monclient: get_auth_request con 0x7f71cc13e640 auth_method 0 2024-09-11T01:55:29.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.572+0000 7f71cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:29.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.572+0000 7f71cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:29.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.572+0000 7f71cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:29.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.572+0000 7f71c8ff9640 1 -- 172.21.15.12:0/3042846897 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f71bc005c40 con 0x7f71cc06d260 2024-09-11T01:55:29.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.572+0000 7f71cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:29.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.572+0000 7f71cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:29.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.572+0000 7f71cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:29.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.572+0000 7f71cb7fe640 10 monclient(hunting): handle_auth_done global_id 294138 payload 824 2024-09-11T01:55:29.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.572+0000 7f71cb7fe640 1 -- 172.21.15.12:0/3042846897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f71cc0008d0 con 0x7f71cc13e640 2024-09-11T01:55:29.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.572+0000 7f71c8ff9640 1 --2- 172.21.15.12:0/3042846897 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f71ac03e210 0x7f71ac0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.572+0000 7f71c8ff9640 1 -- 172.21.15.12:0/3042846897 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(889..889 src has 676..889) ==== 4803+0+0 (secure 0 0 0) 0x7f71bc0524a0 con 0x7f71cc06d260 2024-09-11T01:55:29.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.572+0000 7f71d0de4640 1 --2- 172.21.15.12:0/3042846897 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f71ac03e210 0x7f71ac0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.572+0000 7f71d0de4640 10 monclient: get_auth_request con 0x7f71ac03e210 auth_method 0 2024-09-11T01:55:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.574+0000 7f71cb7fe640 1 -- 172.21.15.12:0/3042846897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71cc13e640 msgr2=0x7f71cc1cd110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.574+0000 7f71cb7fe640 1 --2- 172.21.15.12:0/3042846897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71cc13e640 0x7f71cc1cd110 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f71b40118f0 tx=0x7f71b4011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.574+0000 7f71c8ff9640 1 -- 172.21.15.12:0/3042846897 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b400fe50 con 0x7f71cc13e640 2024-09-11T01:55:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.574+0000 7f71c8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.574+0000 7f71c8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:55:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.574+0000 7f71c8ff9640 10 monclient: dump: 2024-09-11T01:55:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.589 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:29.590 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:29.590 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:29.590 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.575+0000 7f71c8ff9640 1 -- 172.21.15.12:0/3042846897 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f71b400fe50 con 0x7f71cc13e640 2024-09-11T01:55:29.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.575+0000 7f71c8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:29.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.575+0000 7f71c8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:55:29.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.575+0000 7f71c8ff9640 1 -- 172.21.15.12:0/3042846897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71cc13e640 msgr2=0x7f71cc1cd110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:29.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.575+0000 7f71c8ff9640 1 --2- 172.21.15.12:0/3042846897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71cc13e640 0x7f71cc1cd110 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f71b40118f0 tx=0x7f71b4011db0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.575+0000 7f71c8ff9640 1 -- 172.21.15.12:0/3042846897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71cc13e640 msgr2=0x7f71cc1cd110 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:55:29.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.575+0000 7f71c8ff9640 1 --2- 172.21.15.12:0/3042846897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71cc13e640 0x7f71cc1cd110 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f71b40118f0 tx=0x7f71b4011db0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.575+0000 7f71d0de4640 1 --2- 172.21.15.12:0/3042846897 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f71ac03e210 0x7f71ac0406d0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f71cc0700d0 tx=0x7f71c402f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:29.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.581+0000 7f71d2847640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:55:29.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.581+0000 7f71d2847640 1 --2- 172.21.15.12:0/3042846897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71cc1cd650 0x7f71cc1cda30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.581+0000 7f71d2847640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:29.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.581+0000 7f71d2847640 1 -- 172.21.15.12:0/3042846897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f71cc1cdf70 con 0x7f71cc1cd650 2024-09-11T01:55:29.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.581+0000 7f71cb7fe640 1 --2- 172.21.15.12:0/3042846897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71cc1cd650 0x7f71cc1cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.581+0000 7f71cb7fe640 10 monclient: get_auth_request con 0x7f71cc1cd650 auth_method 0 2024-09-11T01:55:29.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.581+0000 7f71cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:29.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.581+0000 7f71cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:29.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.581+0000 7f71cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:29.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.581+0000 7f71cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:29.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.581+0000 7f71cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:29.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.581+0000 7f71cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:29.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.581+0000 7f71cb7fe640 10 monclient(hunting): handle_auth_done global_id 294141 payload 824 2024-09-11T01:55:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.581+0000 7f71cb7fe640 1 -- 172.21.15.12:0/3042846897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f71cc074600 con 0x7f71cc1cd650 2024-09-11T01:55:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.583+0000 7f71cb7fe640 1 -- 172.21.15.12:0/3042846897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71cc1cd650 msgr2=0x7f71cc1cda30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.583+0000 7f71cb7fe640 1 --2- 172.21.15.12:0/3042846897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71cc1cd650 0x7f71cc1cda30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f71b401b000 tx=0x7f71b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.583+0000 7f71c8ff9640 1 -- 172.21.15.12:0/3042846897 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b4014660 con 0x7f71cc1cd650 2024-09-11T01:55:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.583+0000 7f71c8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.583+0000 7f71c8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:55:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.583+0000 7f71c8ff9640 10 monclient: dump: 2024-09-11T01:55:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.594 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:29.595 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:29.595 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:29.595 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.584+0000 7f71c8ff9640 1 -- 172.21.15.12:0/3042846897 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4542 (secure 0 0 0) 0x7f71b40273a0 con 0x7f71cc1cd650 2024-09-11T01:55:29.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.584+0000 7f71c8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:55:29.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.584+0000 7f71c8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:55:29.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.584+0000 7f71c8ff9640 1 -- 172.21.15.12:0/3042846897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71cc1cd650 msgr2=0x7f71cc1cda30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:29.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.584+0000 7f71c8ff9640 1 --2- 172.21.15.12:0/3042846897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71cc1cd650 0x7f71cc1cda30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f71b401b000 tx=0x7f71b40046c0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.584+0000 7f71c8ff9640 1 -- 172.21.15.12:0/3042846897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71cc1cd650 msgr2=0x7f71cc1cda30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:55:29.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.584+0000 7f71c8ff9640 1 --2- 172.21.15.12:0/3042846897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71cc1cd650 0x7f71cc1cda30 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f71b401b000 tx=0x7f71b40046c0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.595 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:55:29.596 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:55:29.596 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:55:29.596 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:55:29.596 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 240, 2024-09-11T01:55:29.596 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 9536, 2024-09-11T01:55:29.596 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:55:29.596 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:55:29.596 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:55:29.596 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:55:29.596 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:29.596 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:55:29.596 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:55:29.597 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:55:29.597 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:55:29.597 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:55:29.597 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:55:29.597 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:55:29.597 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:55:29.597 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:55:29.597 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:55:29.597 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:55:29.597 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:55:29.597 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:55:29.597 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:55:29.597 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:55:29.598 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:29.598 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:55:29.598 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:55:29.598 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:55:29.598 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:55:29.598 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:55:29.598 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:55:29.598 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:55:29.598 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:55:29.598 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:55:29.598 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:55:29.598 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:55:29.598 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:55:29.598 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:55:29.599 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:29.599 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.599 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:55:29.599 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:55:29.599 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:55:29.599 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:55:29.599 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:55:29.599 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:55:29.599 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:55:29.599 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:55:29.599 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:55:29.599 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:55:29.599 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:55:29.599 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:55:29.600 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:55:29.600 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:55:29.600 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:55:29.600 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:55:29.600 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:55:29.600 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:55:29.600 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:55:29.600 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:55:29.600 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:55:29.600 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:55:29.600 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:55:29.600 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:55:29.600 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:55:29.601 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:55:29.601 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:55:29.601 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:55:29.601 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:29.601 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:55:29.601 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.601 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:55:29.601 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.601 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:55:29.601 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:55:29.601 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:55:29.601 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:55:29.601 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.601 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:55:29.602 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:55:29.602 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:29.602 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.602 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.602 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:55:29.602 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:55:29.602 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:29.602 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:29.602 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:29.602 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.602 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:55:29.602 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:55:29.602 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:55:29.602 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:55:29.603 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:55:29.603 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.603 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.603 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:55:29.603 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:55:29.603 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:55:29.603 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:55:29.603 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.603 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:55:29.603 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:55:29.603 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:29.603 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.603 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.603 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:55:29.604 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:55:29.604 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:29.604 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:29.604 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:29.604 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.604 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:55:29.604 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:55:29.604 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:55:29.604 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:55:29.604 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:55:29.604 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.604 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.604 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:55:29.605 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:55:29.605 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:55:29.605 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:55:29.605 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.605 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:55:29.605 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:55:29.605 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:29.605 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.605 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.605 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:55:29.605 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:55:29.605 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:29.605 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:29.606 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:29.606 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.606 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:55:29.606 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:55:29.606 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:55:29.606 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:55:29.606 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:55:29.606 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:29.606 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:29.606 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.606 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:55:29.606 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:55:29.606 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.606 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:55:29.606 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:55:29.607 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:55:29.607 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:29.607 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:29.607 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:55:29.607 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.607 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:55:29.607 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:55:29.607 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:55:29.607 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:29.607 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:29.607 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.607 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:55:29.607 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:55:29.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.584+0000 7f71d2847640 1 -- 172.21.15.12:0/3042846897 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f71ac03e210 msgr2=0x7f71ac0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:29.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.584+0000 7f71d2847640 1 --2- 172.21.15.12:0/3042846897 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f71ac03e210 0x7f71ac0406d0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f71cc0700d0 tx=0x7f71c402f000 comp rx=0 tx=0).stop 2024-09-11T01:55:29.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.584+0000 7f71d2847640 10 monclient: shutdown 2024-09-11T01:55:29.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.584+0000 7f71d2847640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:29.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.584+0000 7f71d2847640 1 -- 172.21.15.12:0/3042846897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71cc06d260 msgr2=0x7f71cc13f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:29.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.584+0000 7f71d2847640 1 --2- 172.21.15.12:0/3042846897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f71cc06d260 0x7f71cc13f390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f71bc00c450 tx=0x7f71bc00c910 comp rx=0 tx=0).stop 2024-09-11T01:55:29.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.584+0000 7f71d0de4640 1 -- 172.21.15.12:0/3042846897 reap_dead start 2024-09-11T01:55:29.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.584+0000 7f71d0de4640 1 -- 172.21.15.12:0/3042846897 reap_dead start 2024-09-11T01:55:29.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.584+0000 7f71d2847640 1 -- 172.21.15.12:0/3042846897 shutdown_connections 2024-09-11T01:55:29.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.584+0000 7f71d2847640 1 -- 172.21.15.12:0/3042846897 >> 172.21.15.12:0/3042846897 conn(0x7f71cc0a3e30 msgr2=0x7f71cc0a5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:29.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.584+0000 7f71d2847640 1 -- 172.21.15.12:0/3042846897 shutdown_connections 2024-09-11T01:55:29.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.584+0000 7f71d2847640 1 -- 172.21.15.12:0/3042846897 wait complete. 2024-09-11T01:55:29.610 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:55:29.610 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:55:29.610 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:55:29.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.710+0000 7f9fdaa23640 10 monclient: get_monmap_and_config 2024-09-11T01:55:29.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.710+0000 7f9fdaa23640 10 monclient: build_initial_monmap 2024-09-11T01:55:29.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.710+0000 7f9fdaa23640 10 monclient: monmap: 2024-09-11T01:55:29.712 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:29.712 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:29.712 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:55:29.711342+0000 2024-09-11T01:55:29.712 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:55:29.711342+0000 2024-09-11T01:55:29.712 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:29.712 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.712 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:55:29.713 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:55:29.713 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:55:29.713 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:55:29.713 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:55:29.713 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.711+0000 7f9fdaa23640 1 Processor -- start 2024-09-11T01:55:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.711+0000 7f9fdaa23640 1 -- start start 2024-09-11T01:55:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.711+0000 7f9fdaa23640 10 monclient: init 2024-09-11T01:55:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.711+0000 7f9fdaa23640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.711+0000 7f9fdaa23640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-11T01:55:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.711+0000 7f9fdaa23640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9fd40a2d40 addr v1:172.21.15.12:6789/0 2024-09-11T01:55:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.711+0000 7f9fdaa23640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9fd406d260 0x7f9fd413d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.711+0000 7f9fdaa23640 10 monclient(hunting): picked mon.noname-a con 0x7f9fd406d260 addr v2:172.21.15.12:3300/0 2024-09-11T01:55:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.711+0000 7f9fdaa23640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9fd406d670 0x7f9fd413fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.711+0000 7f9fdaa23640 10 monclient(hunting): picked mon.noname-c con 0x7f9fd406d670 addr v2:172.21.15.71:3300/0 2024-09-11T01:55:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.711+0000 7f9fdaa23640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fd409aed0 con 0x7f9fd40a2d40 2024-09-11T01:55:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.711+0000 7f9fdaa23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.711+0000 7f9fdaa23640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f9fd406dd10 con 0x7f9fd406d260 2024-09-11T01:55:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.711+0000 7f9fdaa23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.711+0000 7f9fdaa23640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f9fd409bcf0 con 0x7f9fd406d670 2024-09-11T01:55:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.711+0000 7f9fdaa23640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.711+0000 7f9fdaa23640 10 monclient(hunting): authenticate will time out at 2336.153919s 2024-09-11T01:55:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd37fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9fd406d260 0x7f9fd413d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd37fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9fd406d260 0x7f9fd413d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46104/0 (socket says 172.21.15.12:46104) 2024-09-11T01:55:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd37fe640 1 -- 172.21.15.12:0/3580524942 learned_addr learned my addr 172.21.15.12:0/3580524942 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:55:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd37fe640 10 monclient(hunting): get_auth_request con 0x7f9fd406d260 auth_method 0 2024-09-11T01:55:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd8fc0640 1 --2- 172.21.15.12:0/3580524942 >> v2:172.21.15.71:3300/0 conn(0x7f9fd406d670 0x7f9fd413fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd8fc0640 10 monclient(hunting): get_auth_request con 0x7f9fd406d670 auth_method 0 2024-09-11T01:55:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd8fc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd8fc0640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd8fc0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd8fc0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd8fc0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd8fc0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd8fc0640 10 monclient(hunting): handle_auth_done global_id 274300 payload 274 2024-09-11T01:55:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd8fc0640 1 -- 172.21.15.12:0/3580524942 >> v2:172.21.15.12:3300/0 conn(0x7f9fd406d260 msgr2=0x7f9fd413d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:55:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd8fc0640 1 --2- 172.21.15.12:0/3580524942 >> v2:172.21.15.12:3300/0 conn(0x7f9fd406d260 0x7f9fd413d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd8fc0640 1 -- 172.21.15.12:0/3580524942 >> v1:172.21.15.12:6789/0 conn(0x7f9fd40a2d40 legacy=0x7f9fd406cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:55:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd8fc0640 10 monclient: _finish_hunting 0 2024-09-11T01:55:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd8fc0640 1 monclient: found mon.noname-c 2024-09-11T01:55:29.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd8fc0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:29.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd8fc0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:55:29.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.712+0000 7f9fd8fc0640 1 -- 172.21.15.12:0/3580524942 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fd40a3920 con 0x7f9fd406d670 2024-09-11T01:55:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.715+0000 7f9fd8fc0640 1 -- 172.21.15.12:0/3580524942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fd406d670 msgr2=0x7f9fd413fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.715+0000 7f9fd8fc0640 1 --2- 172.21.15.12:0/3580524942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fd406d670 0x7f9fd413fa30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9fd409be70 tx=0x7f9fcc02a4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1d0dcae67502b97 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.715+0000 7f9fd2ffd640 1 -- 172.21.15.12:0/3580524942 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc007950 con 0x7f9fd406d670 2024-09-11T01:55:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.715+0000 7f9fd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.715+0000 7f9fd2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:55:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.715+0000 7f9fd2ffd640 10 monclient: dump: 2024-09-11T01:55:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.725 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:29.726 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.715+0000 7f9fd2ffd640 10 monclient: _finish_auth 0 2024-09-11T01:55:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.715+0000 7f9fd2ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:55:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.715+0000 7f9fd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.715+0000 7f9fd2ffd640 1 -- 172.21.15.12:0/3580524942 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fcc007af0 con 0x7f9fd406d670 2024-09-11T01:55:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.715+0000 7f9fd2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.715+0000 7f9fd2ffd640 1 -- 172.21.15.12:0/3580524942 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc031400 con 0x7f9fd406d670 2024-09-11T01:55:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.715+0000 7f9fd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:29.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.715+0000 7f9fd2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:55:29.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.715+0000 7f9fd2ffd640 10 monclient: dump: 2024-09-11T01:55:29.727 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:29.727 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:29.727 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.727 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.727 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:29.727 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.727 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:29.727 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:29.727 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:29.727 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.715+0000 7f9fdaa23640 5 monclient: authenticate success, global_id 274300 2024-09-11T01:55:29.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.715+0000 7f9fdaa23640 10 monclient: get_monmap_and_config success 2024-09-11T01:55:29.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.715+0000 7f9fdaa23640 4 set_mon_vals no callback set 2024-09-11T01:55:29.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.716+0000 7f9fdaa23640 10 monclient: shutdown 2024-09-11T01:55:29.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.716+0000 7f9fdaa23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:29.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.716+0000 7f9fdaa23640 1 -- 172.21.15.12:0/3580524942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fd406d670 msgr2=0x7f9fd413fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:29.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.716+0000 7f9fdaa23640 1 --2- 172.21.15.12:0/3580524942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fd406d670 0x7f9fd413fa30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9fd409be70 tx=0x7f9fcc02a4b0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.716+0000 7f9fdaa23640 1 -- 172.21.15.12:0/3580524942 shutdown_connections 2024-09-11T01:55:29.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.716+0000 7f9fdaa23640 1 --2- 172.21.15.12:0/3580524942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fd406d670 0x7f9fd413fa30 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.716+0000 7f9fdaa23640 1 --2- 172.21.15.12:0/3580524942 >> v2:172.21.15.12:3300/0 conn(0x7f9fd406d260 0x7f9fd413d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.716+0000 7f9fdaa23640 1 -- 172.21.15.12:0/3580524942 >> 172.21.15.12:0/3580524942 conn(0x7f9fd40a3e30 msgr2=0x7f9fd40a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:29.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.716+0000 7f9fdaa23640 1 -- 172.21.15.12:0/3580524942 shutdown_connections 2024-09-11T01:55:29.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.716+0000 7f9fdaa23640 1 -- 172.21.15.12:0/3580524942 wait complete. 2024-09-11T01:55:29.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.717+0000 7f9fdaa23640 10 monclient: build_initial_monmap 2024-09-11T01:55:29.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.717+0000 7f9fdaa23640 10 monclient: monmap: 2024-09-11T01:55:29.729 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:29.729 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:55:29.729 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:55:29.729 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:55:29.729 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:29.729 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.729 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:55:29.729 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:55:29.729 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:55:29.729 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.717+0000 7f9fdaa23640 1 Processor -- start 2024-09-11T01:55:29.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.717+0000 7f9fdaa23640 1 -- start start 2024-09-11T01:55:29.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.717+0000 7f9fdaa23640 10 monclient: init 2024-09-11T01:55:29.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fdaa23640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:29.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fdaa23640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:55:29.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fdaa23640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fd406d260 0x7f9fd413f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fdaa23640 10 monclient(hunting): picked mon.noname-c con 0x7f9fd406d260 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:29.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fdaa23640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fd406d670 0x7f9fd4137a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fdaa23640 10 monclient(hunting): picked mon.noname-a con 0x7f9fd406d670 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:55:29.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fdaa23640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fd40a2d40 0x7f9fd4137fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fdaa23640 10 monclient(hunting): picked mon.noname-b con 0x7f9fd40a2d40 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:55:29.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fdaa23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:29.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fdaa23640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd409ba10 con 0x7f9fd406d670 2024-09-11T01:55:29.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fdaa23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:29.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fdaa23640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd40a3120 con 0x7f9fd40a2d40 2024-09-11T01:55:29.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fdaa23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:29.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fdaa23640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f9fd409b3a0 con 0x7f9fd406d260 2024-09-11T01:55:29.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fdaa23640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:29.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fdaa23640 10 monclient(hunting): authenticate will time out at 2336.160086s 2024-09-11T01:55:29.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fd406d670 0x7f9fd4137a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fd406d670 0x7f9fd4137a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46114/0 (socket says 172.21.15.12:46114) 2024-09-11T01:55:29.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd3fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fd406d260 0x7f9fd413f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd37fe640 1 -- 172.21.15.12:0/2443890949 learned_addr learned my addr 172.21.15.12:0/2443890949 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:55:29.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd37fe640 10 monclient(hunting): get_auth_request con 0x7f9fd406d670 auth_method 0 2024-09-11T01:55:29.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:29.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:29.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:29.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd3fff640 10 monclient(hunting): get_auth_request con 0x7f9fd406d260 auth_method 0 2024-09-11T01:55:29.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:29.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:29.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:29.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:29.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:29.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:29.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:29.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:29.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:29.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd37fe640 10 monclient(hunting): handle_auth_done global_id 294147 payload 824 2024-09-11T01:55:29.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd37fe640 1 -- 172.21.15.12:0/2443890949 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fd406d260 msgr2=0x7f9fd413f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:29.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd37fe640 1 --2- 172.21.15.12:0/2443890949 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fd406d260 0x7f9fd413f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd37fe640 1 -- 172.21.15.12:0/2443890949 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fd40a2d40 msgr2=0x7f9fd4137fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:29.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd8fc0640 1 --2- 172.21.15.12:0/2443890949 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fd40a2d40 0x7f9fd4137fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd37fe640 1 --2- 172.21.15.12:0/2443890949 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fd40a2d40 0x7f9fd4137fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd37fe640 10 monclient: _finish_hunting 0 2024-09-11T01:55:29.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd37fe640 1 monclient: found mon.noname-a 2024-09-11T01:55:29.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:29.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:55:29.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd37fe640 1 -- 172.21.15.12:0/2443890949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fd4074d40 con 0x7f9fd406d670 2024-09-11T01:55:29.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd8fc0640 10 monclient: get_auth_request con 0x7f9fd40a2d40 auth_method 0 2024-09-11T01:55:29.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.718+0000 7f9fd8fc0640 1 --2- 172.21.15.12:0/2443890949 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fd40a2d40 0x7f9fd4137fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:55:29.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.720+0000 7f9fd37fe640 1 -- 172.21.15.12:0/2443890949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fd406d670 msgr2=0x7f9fd4137a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:29.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.720+0000 7f9fd37fe640 1 --2- 172.21.15.12:0/2443890949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fd406d670 0x7f9fd4137a90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8009fd0 tx=0x7f9fc800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:29.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.720+0000 7f9fd0ff9640 1 -- 172.21.15.12:0/2443890949 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc8017070 con 0x7f9fd406d670 2024-09-11T01:55:29.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.720+0000 7f9fd0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:29.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.720+0000 7f9fd0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:55:29.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.720+0000 7f9fd0ff9640 10 monclient: dump: 2024-09-11T01:55:29.734 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:29.734 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:29.734 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.734 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.734 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:29.734 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.734 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:29.734 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:29.734 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:29.734 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.720+0000 7f9fd0ff9640 10 monclient: _finish_auth 0 2024-09-11T01:55:29.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.720+0000 7f9fd0ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:55:29.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.720+0000 7f9fd0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.720+0000 7f9fdaa23640 5 monclient: authenticate success, global_id 294147 2024-09-11T01:55:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.720+0000 7f9fd0ff9640 1 -- 172.21.15.12:0/2443890949 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fc800cb50 con 0x7f9fd406d670 2024-09-11T01:55:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.720+0000 7f9fdaa23640 10 monclient: _renew_subs 2024-09-11T01:55:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.720+0000 7f9fdaa23640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.720+0000 7f9fdaa23640 1 -- 172.21.15.12:0/2443890949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fd413e120 con 0x7f9fd406d670 2024-09-11T01:55:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.721+0000 7f9fd0ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.721+0000 7f9fd0ff9640 1 -- 172.21.15.12:0/2443890949 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc8013470 con 0x7f9fd406d670 2024-09-11T01:55:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.721+0000 7f9fd0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.721+0000 7f9fd0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:55:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.721+0000 7f9fd0ff9640 10 monclient: dump: 2024-09-11T01:55:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.735 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.736 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:29.736 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.736 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:29.736 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:29.736 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:29.736 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.721+0000 7f9fdaa23640 10 monclient: _renew_subs 2024-09-11T01:55:29.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.721+0000 7f9fdaa23640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:29.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.721+0000 7f9fdaa23640 1 -- 172.21.15.12:0/2443890949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fd413d1c0 con 0x7f9fd406d670 2024-09-11T01:55:29.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.721+0000 7f9fd27fc640 4 set_mon_vals no callback set 2024-09-11T01:55:29.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.721+0000 7f9fdaa23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:29.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.721+0000 7f9fdaa23640 1 --2- 172.21.15.12:0/2443890949 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fd41cd110 0x7f9fd41cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.721+0000 7f9fdaa23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:29.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.721+0000 7f9fdaa23640 1 -- 172.21.15.12:0/2443890949 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f9fd413e920 con 0x7f9fd41cd110 2024-09-11T01:55:29.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.722+0000 7f9fd3fff640 1 --2- 172.21.15.12:0/2443890949 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fd41cd110 0x7f9fd41cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.722+0000 7f9fd3fff640 10 monclient: get_auth_request con 0x7f9fd41cd110 auth_method 0 2024-09-11T01:55:29.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.722+0000 7f9fd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:29.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.722+0000 7f9fd3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:29.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.722+0000 7f9fd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:29.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.722+0000 7f9fd0ff9640 1 -- 172.21.15.12:0/2443890949 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f9fc8024060 con 0x7f9fd406d670 2024-09-11T01:55:29.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.722+0000 7f9fd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:29.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.722+0000 7f9fd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:29.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.722+0000 7f9fd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:29.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.722+0000 7f9fd0ff9640 1 --2- 172.21.15.12:0/2443890949 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9fc003e210 0x7f9fc00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.722+0000 7f9fd3fff640 10 monclient(hunting): handle_auth_done global_id 284225 payload 824 2024-09-11T01:55:29.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.722+0000 7f9fd3fff640 1 -- 172.21.15.12:0/2443890949 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9fd413e640 con 0x7f9fd41cd110 2024-09-11T01:55:29.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.722+0000 7f9fd0ff9640 1 -- 172.21.15.12:0/2443890949 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(889..889 src has 676..889) ==== 4803+0+0 (secure 0 0 0) 0x7f9fc8052d70 con 0x7f9fd406d670 2024-09-11T01:55:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.722+0000 7f9fd8fc0640 1 --2- 172.21.15.12:0/2443890949 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9fc003e210 0x7f9fc00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.722+0000 7f9fd8fc0640 10 monclient: get_auth_request con 0x7f9fc003e210 auth_method 0 2024-09-11T01:55:29.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.724+0000 7f9fd3fff640 1 -- 172.21.15.12:0/2443890949 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fd41cd110 msgr2=0x7f9fd41cd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:29.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.724+0000 7f9fd3fff640 1 --2- 172.21.15.12:0/2443890949 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fd41cd110 0x7f9fd41cd4f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9fc400f770 tx=0x7f9fc400fc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:29.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.724+0000 7f9fd0ff9640 1 -- 172.21.15.12:0/2443890949 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc4018030 con 0x7f9fd41cd110 2024-09-11T01:55:29.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.724+0000 7f9fd0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:29.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.724+0000 7f9fd0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:55:29.740 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.724+0000 7f9fd0ff9640 10 monclient: dump: 2024-09-11T01:55:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:29.741 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.724+0000 7f9fd0ff9640 1 -- 172.21.15.12:0/2443890949 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9fc400fe40 con 0x7f9fd41cd110 2024-09-11T01:55:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.724+0000 7f9fd0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.724+0000 7f9fd0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:55:29.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.724+0000 7f9fd0ff9640 1 -- 172.21.15.12:0/2443890949 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fd41cd110 msgr2=0x7f9fd41cd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.724+0000 7f9fd0ff9640 1 --2- 172.21.15.12:0/2443890949 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fd41cd110 0x7f9fd41cd4f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9fc400f770 tx=0x7f9fc400fc30 comp rx=0 tx=0).stop 2024-09-11T01:55:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.724+0000 7f9fd0ff9640 1 -- 172.21.15.12:0/2443890949 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fd41cd110 msgr2=0x7f9fd41cd4f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:55:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.724+0000 7f9fd0ff9640 1 --2- 172.21.15.12:0/2443890949 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fd41cd110 0x7f9fd41cd4f0 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9fc400f770 tx=0x7f9fc400fc30 comp rx=0 tx=0).stop 2024-09-11T01:55:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.725+0000 7f9fd8fc0640 1 --2- 172.21.15.12:0/2443890949 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9fc003e210 0x7f9fc00406d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f9fd40700d0 tx=0x7f9fcc02f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.731+0000 7f9fdaa23640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:55:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.731+0000 7f9fdaa23640 1 --2- 172.21.15.12:0/2443890949 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fd41cda30 0x7f9fd4070f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.731+0000 7f9fdaa23640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.731+0000 7f9fdaa23640 1 -- 172.21.15.12:0/2443890949 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f9fd409a980 con 0x7f9fd41cda30 2024-09-11T01:55:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.731+0000 7f9fd3fff640 1 --2- 172.21.15.12:0/2443890949 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fd41cda30 0x7f9fd4070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.731+0000 7f9fd3fff640 10 monclient: get_auth_request con 0x7f9fd41cda30 auth_method 0 2024-09-11T01:55:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.731+0000 7f9fd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.731+0000 7f9fd3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:29.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.731+0000 7f9fd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.731+0000 7f9fd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.731+0000 7f9fd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.731+0000 7f9fd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.731+0000 7f9fd3fff640 10 monclient(hunting): handle_auth_done global_id 284228 payload 824 2024-09-11T01:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.731+0000 7f9fd3fff640 1 -- 172.21.15.12:0/2443890949 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9fd406e520 con 0x7f9fd41cda30 2024-09-11T01:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.733+0000 7f9fd3fff640 1 -- 172.21.15.12:0/2443890949 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fd41cda30 msgr2=0x7f9fd4070f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.733+0000 7f9fd3fff640 1 --2- 172.21.15.12:0/2443890949 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fd41cda30 0x7f9fd4070f30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9fc400f770 tx=0x7f9fc4003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.733+0000 7f9fd0ff9640 1 -- 172.21.15.12:0/2443890949 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc4007440 con 0x7f9fd41cda30 2024-09-11T01:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.733+0000 7f9fd0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.733+0000 7f9fd0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.733+0000 7f9fd0ff9640 10 monclient: dump: 2024-09-11T01:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:29.743 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.744 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.744 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:29.744 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.744 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:29.744 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:29.744 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:29.744 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.734+0000 7f9fd0ff9640 1 -- 172.21.15.12:0/2443890949 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7f9fc4007440 con 0x7f9fd41cda30 2024-09-11T01:55:29.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.734+0000 7f9fd0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:55:29.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.734+0000 7f9fd0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:55:29.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.734+0000 7f9fd0ff9640 1 -- 172.21.15.12:0/2443890949 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fd41cda30 msgr2=0x7f9fd4070f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:29.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.734+0000 7f9fd0ff9640 1 --2- 172.21.15.12:0/2443890949 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fd41cda30 0x7f9fd4070f30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9fc400f770 tx=0x7f9fc4003760 comp rx=0 tx=0).stop 2024-09-11T01:55:29.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.734+0000 7f9fd0ff9640 1 -- 172.21.15.12:0/2443890949 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fd41cda30 msgr2=0x7f9fd4070f30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:55:29.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.734+0000 7f9fd0ff9640 1 --2- 172.21.15.12:0/2443890949 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9fd41cda30 0x7f9fd4070f30 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9fc400f770 tx=0x7f9fc4003760 comp rx=0 tx=0).stop 2024-09-11T01:55:29.745 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:55:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:55:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:55:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:55:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 240, 2024-09-11T01:55:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 68929, 2024-09-11T01:55:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:55:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:55:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:55:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:55:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:55:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:55:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:55:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:55:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:55:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:55:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:55:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:55:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:55:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:55:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:55:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:55:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:55:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:55:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:55:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:55:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:55:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:55:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:55:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:55:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:55:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:55:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:55:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:55:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:55:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:55:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:55:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:55:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:55:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:55:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:55:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:55:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:55:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:55:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:55:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:55:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:55:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:55:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:55:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:55:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:55:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:55:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:55:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:55:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:55:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:55:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:55:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:55:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:55:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:55:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:55:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:55:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:55:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:55:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:55:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:55:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:55:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:55:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:55:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:55:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:55:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:55:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:55:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:55:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:55:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:55:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:55:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:55:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:55:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:55:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:55:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:55:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:55:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:55:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:55:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:55:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:55:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:55:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:55:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:55:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:55:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:55:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:55:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:55:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:55:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:55:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:55:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:55:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:55:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:55:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:55:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:55:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:55:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:55:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:55:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:55:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:55:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:55:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:55:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:55:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:55:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:55:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:55:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:55:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:55:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:55:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:55:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:55:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:55:29.757 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:55:29.757 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:29.757 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:29.760 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.760 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:55:29.760 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:55:29.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.734+0000 7f9fdaa23640 1 -- 172.21.15.12:0/2443890949 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9fc003e210 msgr2=0x7f9fc00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:29.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.734+0000 7f9fdaa23640 1 --2- 172.21.15.12:0/2443890949 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9fc003e210 0x7f9fc00406d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f9fd40700d0 tx=0x7f9fcc02f000 comp rx=0 tx=0).stop 2024-09-11T01:55:29.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.734+0000 7f9fdaa23640 10 monclient: shutdown 2024-09-11T01:55:29.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.734+0000 7f9fdaa23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:29.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.734+0000 7f9fdaa23640 1 -- 172.21.15.12:0/2443890949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fd406d670 msgr2=0x7f9fd4137a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:29.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.734+0000 7f9fdaa23640 1 --2- 172.21.15.12:0/2443890949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fd406d670 0x7f9fd4137a90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8009fd0 tx=0x7f9fc800ed40 comp rx=0 tx=0).stop 2024-09-11T01:55:29.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.734+0000 7f9fd8fc0640 1 -- 172.21.15.12:0/2443890949 reap_dead start 2024-09-11T01:55:29.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.734+0000 7f9fdaa23640 1 -- 172.21.15.12:0/2443890949 shutdown_connections 2024-09-11T01:55:29.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.734+0000 7f9fdaa23640 1 --2- 172.21.15.12:0/2443890949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9fd406d670 0x7f9fd4137a90 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.734+0000 7f9fdaa23640 1 -- 172.21.15.12:0/2443890949 >> 172.21.15.12:0/2443890949 conn(0x7f9fd40a3e30 msgr2=0x7f9fd40a5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:29.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.734+0000 7f9fdaa23640 1 -- 172.21.15.12:0/2443890949 shutdown_connections 2024-09-11T01:55:29.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.734+0000 7f9fdaa23640 1 -- 172.21.15.12:0/2443890949 wait complete. 2024-09-11T01:55:29.763 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:55:29.763 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:55:29.763 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:55:29.846 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.845+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.846 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.845+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3807820294}) 2024-09-11T01:55:29.847 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.845+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=946972201}) 2024-09-11T01:55:29.847 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.845+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.847 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.845+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3807820294}) 2024-09-11T01:55:29.847 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.845+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2765028822}) 2024-09-11T01:55:29.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.864+0000 7f151d03c640 10 monclient: get_monmap_and_config 2024-09-11T01:55:29.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.864+0000 7f151d03c640 10 monclient: build_initial_monmap 2024-09-11T01:55:29.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.864+0000 7f151d03c640 10 monclient: monmap: 2024-09-11T01:55:29.866 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:29.866 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:29.866 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:55:29.865416+0000 2024-09-11T01:55:29.866 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:55:29.865416+0000 2024-09-11T01:55:29.866 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:29.866 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.867 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:55:29.867 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:55:29.867 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:55:29.867 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:55:29.867 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:55:29.867 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.865+0000 7f151d03c640 1 Processor -- start 2024-09-11T01:55:29.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.865+0000 7f151d03c640 1 -- start start 2024-09-11T01:55:29.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.865+0000 7f151d03c640 10 monclient: init 2024-09-11T01:55:29.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.865+0000 7f151d03c640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:29.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.865+0000 7f151d03c640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-11T01:55:29.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.865+0000 7f151d03c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1518134ad0 addr v1:172.21.15.71:6789/0 2024-09-11T01:55:29.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.865+0000 7f151d03c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f15180a2200 addr v1:172.21.15.12:6789/0 2024-09-11T01:55:29.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.865+0000 7f151d03c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15181355c0 0x7f151813d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.865+0000 7f151d03c640 10 monclient(hunting): picked mon.noname-b con 0x7f15181355c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:29.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.866+0000 7f151d03c640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f151809ed30 con 0x7f15180a2200 2024-09-11T01:55:29.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.866+0000 7f151d03c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f151809f010 con 0x7f1518134ad0 2024-09-11T01:55:29.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.866+0000 7f151d03c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:29.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.866+0000 7f151d03c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f151809f5d0 con 0x7f15181355c0 2024-09-11T01:55:29.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.866+0000 7f151d03c640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:29.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.866+0000 7f151d03c640 10 monclient(hunting): authenticate will time out at 2336.308044s 2024-09-11T01:55:29.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.866+0000 7f150ffff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f15180a2200 0x7f151813a000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55656/0 (socket says 172.21.15.12:55656) 2024-09-11T01:55:29.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.866+0000 7f150ffff640 1 -- 172.21.15.12:0/388597440 learned_addr learned my addr 172.21.15.12:0/388597440 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:55:29.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.866+0000 7f151759e640 1 --2- 172.21.15.12:0/388597440 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15181355c0 0x7f151813d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.866+0000 7f151759e640 10 monclient(hunting): get_auth_request con 0x7f15181355c0 auth_method 0 2024-09-11T01:55:29.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.866+0000 7f151759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:29.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.866+0000 7f151759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:29.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.866+0000 7f151759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:29.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.866+0000 7f151759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:29.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.866+0000 7f151759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:29.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.866+0000 7f151759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:29.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.866+0000 7f151759e640 10 monclient(hunting): handle_auth_done global_id 284231 payload 274 2024-09-11T01:55:29.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.866+0000 7f151759e640 1 -- 172.21.15.12:0/388597440 >> v1:172.21.15.71:6789/0 conn(0x7f1518134ad0 legacy=0x7f1518134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:55:29.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.866+0000 7f151759e640 1 -- 172.21.15.12:0/388597440 >> v1:172.21.15.12:6789/0 conn(0x7f15180a2200 legacy=0x7f151813a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:55:29.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.866+0000 7f151759e640 10 monclient: _finish_hunting 0 2024-09-11T01:55:29.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.866+0000 7f151759e640 1 monclient: found mon.noname-b 2024-09-11T01:55:29.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.866+0000 7f151759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:29.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.866+0000 7f151759e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:55:29.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.866+0000 7f151759e640 1 -- 172.21.15.12:0/388597440 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f151809f2f0 con 0x7f15181355c0 2024-09-11T01:55:29.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.869+0000 7f151759e640 1 -- 172.21.15.12:0/388597440 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15181355c0 msgr2=0x7f151813d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:29.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.869+0000 7f151759e640 1 --2- 172.21.15.12:0/388597440 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15181355c0 0x7f151813d730 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1510009df0 tx=0x7f151002a4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24af2cb6be84f92e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:29.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.869+0000 7f151659c640 1 -- 172.21.15.12:0/388597440 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1510007a30 con 0x7f15181355c0 2024-09-11T01:55:29.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.869+0000 7f151659c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:29.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.869+0000 7f151659c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:55:29.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.869+0000 7f151659c640 10 monclient: dump: 2024-09-11T01:55:29.875 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:29.875 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:29.875 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.875 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.875 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:29.875 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.876 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:29.876 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:29.876 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:29.876 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.869+0000 7f151659c640 10 monclient: _finish_auth 0 2024-09-11T01:55:29.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.869+0000 7f151659c640 10 monclient: _check_auth_tickets 2024-09-11T01:55:29.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.869+0000 7f151659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:29.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.869+0000 7f151659c640 1 -- 172.21.15.12:0/388597440 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1510007bd0 con 0x7f15181355c0 2024-09-11T01:55:29.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.869+0000 7f151d03c640 5 monclient: authenticate success, global_id 284231 2024-09-11T01:55:29.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.869+0000 7f151d03c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:55:29.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.869+0000 7f151659c640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:29.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.869+0000 7f151d03c640 10 monclient: get_monmap_and_config success 2024-09-11T01:55:29.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.869+0000 7f151d03c640 4 set_mon_vals no callback set 2024-09-11T01:55:29.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.869+0000 7f151659c640 1 -- 172.21.15.12:0/388597440 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1510031400 con 0x7f15181355c0 2024-09-11T01:55:29.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.869+0000 7f151659c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:29.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.869+0000 7f151659c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:55:29.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.869+0000 7f151659c640 10 monclient: dump: 2024-09-11T01:55:29.877 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:29.877 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:29.877 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.877 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.877 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:29.877 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.877 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:29.877 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:29.877 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:29.877 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.869+0000 7f151d03c640 10 monclient: shutdown 2024-09-11T01:55:29.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.869+0000 7f151d03c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:29.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.869+0000 7f151d03c640 1 -- 172.21.15.12:0/388597440 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15181355c0 msgr2=0x7f151813d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:29.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.869+0000 7f151d03c640 1 --2- 172.21.15.12:0/388597440 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15181355c0 0x7f151813d730 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1510009df0 tx=0x7f151002a4b0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.869+0000 7f151d03c640 1 -- 172.21.15.12:0/388597440 shutdown_connections 2024-09-11T01:55:29.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.870+0000 7f151d03c640 1 --2- 172.21.15.12:0/388597440 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15181355c0 0x7f151813d730 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.870+0000 7f151d03c640 1 -- 172.21.15.12:0/388597440 >> 172.21.15.12:0/388597440 conn(0x7f15180a2f70 msgr2=0x7f15180a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:29.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.870+0000 7f151d03c640 1 -- 172.21.15.12:0/388597440 shutdown_connections 2024-09-11T01:55:29.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.870+0000 7f151d03c640 1 -- 172.21.15.12:0/388597440 wait complete. 2024-09-11T01:55:29.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.870+0000 7f151d03c640 10 monclient: build_initial_monmap 2024-09-11T01:55:29.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.870+0000 7f151d03c640 10 monclient: monmap: 2024-09-11T01:55:29.878 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:29.878 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:55:29.878 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:55:29.879 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:55:29.879 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:29.879 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.879 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:55:29.879 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:55:29.879 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:55:29.879 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f151d03c640 1 Processor -- start 2024-09-11T01:55:29.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f151d03c640 1 -- start start 2024-09-11T01:55:29.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f151d03c640 10 monclient: init 2024-09-11T01:55:29.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f151d03c640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:29.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f151d03c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:55:29.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f151d03c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15180a2200 0x7f151811cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f151d03c640 10 monclient(hunting): picked mon.noname-c con 0x7f15180a2200 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:29.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f151d03c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1518134ad0 0x7f15181339b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f151d03c640 10 monclient(hunting): picked mon.noname-a con 0x7f1518134ad0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:55:29.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f151d03c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15181355c0 0x7f151813bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f151d03c640 10 monclient(hunting): picked mon.noname-b con 0x7f15181355c0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:55:29.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f151d03c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:29.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f151d03c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f151809cde0 con 0x7f1518134ad0 2024-09-11T01:55:29.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f151d03c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:29.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f151d03c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f15180a01e0 con 0x7f15181355c0 2024-09-11T01:55:29.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f151d03c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:29.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f151d03c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f151813dc70 con 0x7f15180a2200 2024-09-11T01:55:29.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f151d03c640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:29.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f151d03c640 10 monclient(hunting): authenticate will time out at 2336.313604s 2024-09-11T01:55:29.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f150ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1518134ad0 0x7f15181339b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f1516d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15180a2200 0x7f151811cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f150ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1518134ad0 0x7f15181339b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46124/0 (socket says 172.21.15.12:46124) 2024-09-11T01:55:29.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f150ffff640 1 -- 172.21.15.12:0/411261847 learned_addr learned my addr 172.21.15.12:0/411261847 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:55:29.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f150ffff640 10 monclient(hunting): get_auth_request con 0x7f1518134ad0 auth_method 0 2024-09-11T01:55:29.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f150ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:29.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f150ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:29.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f150ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:29.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f1516d9d640 10 monclient(hunting): get_auth_request con 0x7f15180a2200 auth_method 0 2024-09-11T01:55:29.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f1516d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:29.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f1516d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:29.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f1516d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:29.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f150ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:29.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f150ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:29.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f150ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:29.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f1516d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:29.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f1516d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:29.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f1516d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:29.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f150ffff640 10 monclient(hunting): handle_auth_done global_id 294150 payload 824 2024-09-11T01:55:29.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f151759e640 1 --2- 172.21.15.12:0/411261847 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15181355c0 0x7f151813bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f150ffff640 1 -- 172.21.15.12:0/411261847 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15180a2200 msgr2=0x7f151811cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:29.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f150ffff640 1 --2- 172.21.15.12:0/411261847 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f15180a2200 0x7f151811cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f150ffff640 1 -- 172.21.15.12:0/411261847 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15181355c0 msgr2=0x7f151813bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:29.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f150ffff640 1 --2- 172.21.15.12:0/411261847 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15181355c0 0x7f151813bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f150ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:55:29.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f150ffff640 1 monclient: found mon.noname-a 2024-09-11T01:55:29.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f150ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:29.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f150ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:55:29.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.871+0000 7f150ffff640 1 -- 172.21.15.12:0/411261847 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f151811d4d0 con 0x7f1518134ad0 2024-09-11T01:55:29.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.874+0000 7f150ffff640 1 -- 172.21.15.12:0/411261847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1518134ad0 msgr2=0x7f15181339b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:29.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.874+0000 7f150ffff640 1 --2- 172.21.15.12:0/411261847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1518134ad0 0x7f15181339b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f150000c430 tx=0x7f150000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:29.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.874+0000 7f150f7fe640 1 -- 172.21.15.12:0/411261847 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1500017070 con 0x7f1518134ad0 2024-09-11T01:55:29.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.874+0000 7f150f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:29.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.874+0000 7f150f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:55:29.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.874+0000 7f150f7fe640 10 monclient: dump: 2024-09-11T01:55:29.886 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:29.886 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:29.886 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.886 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.886 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:29.886 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.886 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:29.886 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:29.886 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:29.886 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.874+0000 7f150f7fe640 10 monclient: _finish_auth 0 2024-09-11T01:55:29.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.874+0000 7f150f7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:55:29.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.874+0000 7f150f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:29.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.874+0000 7f151d03c640 5 monclient: authenticate success, global_id 294150 2024-09-11T01:55:29.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.874+0000 7f151d03c640 10 monclient: _renew_subs 2024-09-11T01:55:29.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.874+0000 7f150f7fe640 1 -- 172.21.15.12:0/411261847 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15000070d0 con 0x7f1518134ad0 2024-09-11T01:55:29.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.874+0000 7f151d03c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:29.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.874+0000 7f151d03c640 1 -- 172.21.15.12:0/411261847 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15181cbc00 con 0x7f1518134ad0 2024-09-11T01:55:29.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.874+0000 7f150f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:29.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.874+0000 7f150f7fe640 1 -- 172.21.15.12:0/411261847 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15000074c0 con 0x7f1518134ad0 2024-09-11T01:55:29.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.874+0000 7f150f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:29.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.874+0000 7f150f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:55:29.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.874+0000 7f150f7fe640 10 monclient: dump: 2024-09-11T01:55:29.887 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:29.887 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:29.888 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.888 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.888 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:29.888 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.888 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:29.888 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:29.888 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:29.888 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.874+0000 7f1515d9b640 4 set_mon_vals no callback set 2024-09-11T01:55:29.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.874+0000 7f151d03c640 10 monclient: _renew_subs 2024-09-11T01:55:29.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.874+0000 7f151d03c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:29.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.874+0000 7f151d03c640 1 -- 172.21.15.12:0/411261847 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f151809d480 con 0x7f1518134ad0 2024-09-11T01:55:29.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.875+0000 7f150f7fe640 1 -- 172.21.15.12:0/411261847 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f150001c440 con 0x7f1518134ad0 2024-09-11T01:55:29.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.875+0000 7f151d03c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:29.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.875+0000 7f151d03c640 1 --2- 172.21.15.12:0/411261847 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f151811dfc0 0x7f151813c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.875+0000 7f151d03c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:29.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.875+0000 7f151d03c640 1 -- 172.21.15.12:0/411261847 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1518134eb0 con 0x7f151811dfc0 2024-09-11T01:55:29.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.875+0000 7f150f7fe640 1 --2- 172.21.15.12:0/411261847 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f14f803e210 0x7f14f80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.875+0000 7f150f7fe640 1 -- 172.21.15.12:0/411261847 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(889..889 src has 676..889) ==== 4803+0+0 (secure 0 0 0) 0x7f1500052530 con 0x7f1518134ad0 2024-09-11T01:55:29.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.875+0000 7f1516d9d640 1 --2- 172.21.15.12:0/411261847 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f151811dfc0 0x7f151813c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.875+0000 7f1516d9d640 10 monclient: get_auth_request con 0x7f151811dfc0 auth_method 0 2024-09-11T01:55:29.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.875+0000 7f1516d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:29.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.875+0000 7f1516d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:29.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.875+0000 7f1516d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:29.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.875+0000 7f151759e640 1 --2- 172.21.15.12:0/411261847 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f14f803e210 0x7f14f80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.875+0000 7f1516d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:29.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.875+0000 7f1516d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:29.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.875+0000 7f1516d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:29.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.875+0000 7f151759e640 10 monclient: get_auth_request con 0x7f14f803e210 auth_method 0 2024-09-11T01:55:29.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.875+0000 7f1516d9d640 10 monclient(hunting): handle_auth_done global_id 274303 payload 824 2024-09-11T01:55:29.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.876+0000 7f1516d9d640 1 -- 172.21.15.12:0/411261847 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f151811dba0 con 0x7f151811dfc0 2024-09-11T01:55:29.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.878+0000 7f1516d9d640 1 -- 172.21.15.12:0/411261847 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f151811dfc0 msgr2=0x7f151813c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:29.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.878+0000 7f1516d9d640 1 --2- 172.21.15.12:0/411261847 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f151811dfc0 0x7f151813c440 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f150800f7a0 tx=0x7f150800fc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:29.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.878+0000 7f151759e640 1 --2- 172.21.15.12:0/411261847 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f14f803e210 0x7f14f80406d0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f1510009dc0 tx=0x7f151002f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:29.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.878+0000 7f150f7fe640 1 -- 172.21.15.12:0/411261847 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1508018030 con 0x7f151811dfc0 2024-09-11T01:55:29.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.878+0000 7f150f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:29.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.878+0000 7f150f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:55:29.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.878+0000 7f150f7fe640 10 monclient: dump: 2024-09-11T01:55:29.890 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:29.891 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:29.891 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.891 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.891 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:29.891 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.891 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:29.891 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:29.891 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:29.891 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.878+0000 7f150f7fe640 1 -- 172.21.15.12:0/411261847 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f150800d9b0 con 0x7f151811dfc0 2024-09-11T01:55:29.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.878+0000 7f150f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:29.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.878+0000 7f150f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:55:29.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.878+0000 7f150f7fe640 1 -- 172.21.15.12:0/411261847 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f151811dfc0 msgr2=0x7f151813c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:29.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.878+0000 7f150f7fe640 1 --2- 172.21.15.12:0/411261847 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f151811dfc0 0x7f151813c440 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f150800f7a0 tx=0x7f150800fc60 comp rx=0 tx=0).stop 2024-09-11T01:55:29.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.878+0000 7f150f7fe640 1 -- 172.21.15.12:0/411261847 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f151811dfc0 msgr2=0x7f151813c440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:55:29.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.878+0000 7f150f7fe640 1 --2- 172.21.15.12:0/411261847 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f151811dfc0 0x7f151813c440 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f150800f7a0 tx=0x7f150800fc60 comp rx=0 tx=0).stop 2024-09-11T01:55:29.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.885+0000 7f151d03c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:55:29.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.885+0000 7f151d03c640 1 --2- 172.21.15.12:0/411261847 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f151813c980 0x7f1518135fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:29.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.885+0000 7f151d03c640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:29.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.885+0000 7f151d03c640 1 -- 172.21.15.12:0/411261847 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f151813cd90 con 0x7f151813c980 2024-09-11T01:55:29.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.885+0000 7f1516d9d640 1 --2- 172.21.15.12:0/411261847 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f151813c980 0x7f1518135fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:29.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.885+0000 7f1516d9d640 10 monclient: get_auth_request con 0x7f151813c980 auth_method 0 2024-09-11T01:55:29.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.885+0000 7f1516d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:29.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.885+0000 7f1516d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:29.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.885+0000 7f1516d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:29.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.885+0000 7f1516d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:29.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.885+0000 7f1516d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:29.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.885+0000 7f1516d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:29.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.885+0000 7f1516d9d640 10 monclient(hunting): handle_auth_done global_id 274306 payload 824 2024-09-11T01:55:29.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.885+0000 7f1516d9d640 1 -- 172.21.15.12:0/411261847 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f15181c2d60 con 0x7f151813c980 2024-09-11T01:55:29.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.888+0000 7f1516d9d640 1 -- 172.21.15.12:0/411261847 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f151813c980 msgr2=0x7f1518135fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:29.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.888+0000 7f1516d9d640 1 --2- 172.21.15.12:0/411261847 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f151813c980 0x7f1518135fe0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f150800f7a0 tx=0x7f1508003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:29.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.888+0000 7f150f7fe640 1 -- 172.21.15.12:0/411261847 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f150801dbe0 con 0x7f151813c980 2024-09-11T01:55:29.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.888+0000 7f150f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:29.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.888+0000 7f150f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:55:29.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.888+0000 7f150f7fe640 10 monclient: dump: 2024-09-11T01:55:29.895 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:29.895 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:29.895 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.895 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:29.895 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:29.895 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:29.895 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:29.895 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:29.895 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:29.895 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:29.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.888+0000 7f150f7fe640 1 -- 172.21.15.12:0/411261847 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f150801dbe0 con 0x7f151813c980 2024-09-11T01:55:29.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.888+0000 7f150f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:55:29.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.888+0000 7f150f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:55:29.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.888+0000 7f150f7fe640 1 -- 172.21.15.12:0/411261847 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f151813c980 msgr2=0x7f1518135fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:29.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.888+0000 7f150f7fe640 1 --2- 172.21.15.12:0/411261847 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f151813c980 0x7f1518135fe0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f150800f7a0 tx=0x7f1508003760 comp rx=0 tx=0).stop 2024-09-11T01:55:29.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.888+0000 7f150f7fe640 1 -- 172.21.15.12:0/411261847 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f151813c980 msgr2=0x7f1518135fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:55:29.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.888+0000 7f150f7fe640 1 --2- 172.21.15.12:0/411261847 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f151813c980 0x7f1518135fe0 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f150800f7a0 tx=0x7f1508003760 comp rx=0 tx=0).stop 2024-09-11T01:55:29.896 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.888+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.897 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:55:29.897 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:55:29.897 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:55:29.897 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:55:29.897 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 240, 2024-09-11T01:55:29.897 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 119707, 2024-09-11T01:55:29.897 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:55:29.897 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:55:29.897 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:55:29.897 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:55:29.897 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:29.897 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:55:29.897 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:55:29.898 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:55:29.898 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:55:29.898 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:55:29.898 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:55:29.898 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:55:29.898 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:55:29.898 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:55:29.898 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:55:29.898 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:55:29.898 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:55:29.898 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:55:29.898 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:55:29.898 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:55:29.899 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:29.899 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:55:29.899 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:55:29.899 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:55:29.899 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:55:29.899 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:55:29.899 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:55:29.899 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:55:29.899 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:55:29.899 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:55:29.899 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:55:29.899 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:55:29.899 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:55:29.899 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:55:29.900 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:29.900 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.900 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:55:29.900 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:55:29.900 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:55:29.900 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:55:29.900 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:55:29.900 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:55:29.900 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:55:29.900 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:55:29.900 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:55:29.900 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:55:29.900 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:55:29.901 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:55:29.901 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:55:29.901 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:55:29.901 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:55:29.901 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:55:29.901 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:55:29.901 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:55:29.901 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:55:29.901 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:55:29.901 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:55:29.901 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:55:29.901 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:55:29.901 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:55:29.902 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:55:29.902 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:55:29.902 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:55:29.902 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:55:29.902 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:29.902 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:55:29.902 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.902 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:55:29.902 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.902 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:55:29.902 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:55:29.902 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:55:29.902 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:55:29.902 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.903 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:55:29.903 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:55:29.903 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:29.903 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.903 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.903 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:55:29.903 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:55:29.903 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:29.903 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:29.903 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:29.903 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.903 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:55:29.903 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:55:29.904 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:55:29.904 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:55:29.904 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:55:29.904 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.904 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.904 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:55:29.904 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:55:29.904 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:55:29.904 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:55:29.904 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.904 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:55:29.904 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:55:29.904 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:29.905 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.905 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.905 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:55:29.905 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:55:29.905 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:29.905 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:29.905 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:29.905 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.905 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:55:29.905 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:55:29.905 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:55:29.905 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:55:29.905 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:55:29.906 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.906 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.906 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:55:29.906 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:55:29.906 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:55:29.906 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:55:29.906 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.906 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:55:29.906 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:55:29.906 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:29.906 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.906 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.906 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:55:29.906 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:55:29.907 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:29.907 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:29.907 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:29.907 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.907 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:55:29.907 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:55:29.907 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:55:29.907 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:55:29.907 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:55:29.907 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:29.907 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:29.907 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.907 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:55:29.907 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:55:29.907 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.908 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:55:29.908 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:55:29.908 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:55:29.908 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:29.908 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:29.908 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:55:29.908 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.908 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:55:29.908 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:55:29.908 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:55:29.908 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:29.908 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:29.908 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:55:29.909 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.909 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:55:29.909 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:55:29.909 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:55:29.909 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:29.909 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:29.909 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:55:29.909 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:29.909 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:55:29.909 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:55:29.909 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:55:29.909 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:29.909 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:29.909 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:29.910 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:55:29.910 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:55:29.910 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.888+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2592255725}) 2024-09-11T01:55:29.910 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.888+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1707265510}) 2024-09-11T01:55:29.910 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.888+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.910 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.888+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2592255725}) 2024-09-11T01:55:29.910 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.888+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2813006934}) 2024-09-11T01:55:29.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.889+0000 7f151d03c640 1 -- 172.21.15.12:0/411261847 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f14f803e210 msgr2=0x7f14f80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:29.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.889+0000 7f151d03c640 1 --2- 172.21.15.12:0/411261847 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f14f803e210 0x7f14f80406d0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f1510009dc0 tx=0x7f151002f000 comp rx=0 tx=0).stop 2024-09-11T01:55:29.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.889+0000 7f151d03c640 10 monclient: shutdown 2024-09-11T01:55:29.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.889+0000 7f151d03c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:29.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.889+0000 7f151d03c640 1 -- 172.21.15.12:0/411261847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1518134ad0 msgr2=0x7f15181339b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:29.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.889+0000 7f151d03c640 1 --2- 172.21.15.12:0/411261847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1518134ad0 0x7f15181339b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f150000c430 tx=0x7f150000c8f0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.889+0000 7f151759e640 1 -- 172.21.15.12:0/411261847 reap_dead start 2024-09-11T01:55:29.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.889+0000 7f151d03c640 1 -- 172.21.15.12:0/411261847 shutdown_connections 2024-09-11T01:55:29.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.889+0000 7f151d03c640 1 --2- 172.21.15.12:0/411261847 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1518134ad0 0x7f15181339b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:29.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.889+0000 7f151d03c640 1 -- 172.21.15.12:0/411261847 >> 172.21.15.12:0/411261847 conn(0x7f15180a2f70 msgr2=0x7f15180a49b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:29.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.889+0000 7f151d03c640 1 -- 172.21.15.12:0/411261847 shutdown_connections 2024-09-11T01:55:29.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:29.889+0000 7f151d03c640 1 -- 172.21.15.12:0/411261847 wait complete. 2024-09-11T01:55:29.913 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:55:29.913 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-11T01:55:29.913 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T01:55:29.913 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-11T01:55:29.914 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-11T01:55:29.914 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-11T01:55:29.914 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:55:29.922 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.920+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.922 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.920+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3841358952}) 2024-09-11T01:55:29.922 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.920+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3274064708}) 2024-09-11T01:55:29.922 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.920+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:55:29.922 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.921+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3841358952}) 2024-09-11T01:55:29.922 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:55:29.921+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2009904790}) 2024-09-11T01:55:29.926 INFO:tasks.ceph.mon.c:Stopped 2024-09-11T01:55:29.926 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:55:29.926 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:55:30.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.065+0000 7f4bd0bf8640 10 monclient: get_monmap_and_config 2024-09-11T01:55:30.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.065+0000 7f4bd0bf8640 10 monclient: build_initial_monmap 2024-09-11T01:55:30.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.065+0000 7f4bd0bf8640 10 monclient: monmap: 2024-09-11T01:55:30.067 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:30.068 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:30.068 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:55:30.066613+0000 2024-09-11T01:55:30.068 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:55:30.066613+0000 2024-09-11T01:55:30.068 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:30.068 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:30.068 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:55:30.068 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:55:30.068 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:55:30.068 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:55:30.068 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:55:30.068 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:30.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.066+0000 7f4bd0bf8640 1 Processor -- start 2024-09-11T01:55:30.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.066+0000 7f4bd0bf8640 1 -- start start 2024-09-11T01:55:30.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.066+0000 7f4bd0bf8640 10 monclient: init 2024-09-11T01:55:30.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.067+0000 7f4bd0bf8640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:30.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.067+0000 7f4bd0bf8640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-11T01:55:30.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.067+0000 7f4bd0bf8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4bcc0a2e90 addr v1:172.21.15.12:6789/0 2024-09-11T01:55:30.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.067+0000 7f4bd0bf8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bcc0a3270 0x7f4bcc0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.067+0000 7f4bd0bf8640 10 monclient(hunting): picked mon.noname-b con 0x7f4bcc0a3270 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.067+0000 7f4bd0bf8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4bcc13d490 addr v1:172.21.15.71:6789/0 2024-09-11T01:55:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.067+0000 7f4bd0bf8640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bcc0a46c0 con 0x7f4bcc0a2e90 2024-09-11T01:55:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.067+0000 7f4bd0bf8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bcc06ee50 con 0x7f4bcc13d490 2024-09-11T01:55:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.067+0000 7f4bd0bf8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.067+0000 7f4bd0bf8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4bcc0a4900 con 0x7f4bcc0a3270 2024-09-11T01:55:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.067+0000 7f4bd0bf8640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.067+0000 7f4bd0bf8640 10 monclient(hunting): authenticate will time out at 2336.509192s 2024-09-11T01:55:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.067+0000 7f4bc3fff640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bcc0a3270 msgr2=0x7f4bcc0a3680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:55:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.067+0000 7f4bc3fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bcc0a3270 0x7f4bcc0a3680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:55:30.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.067+0000 7f4bca59c640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4bcc0a2e90 0x7f4bcc06dc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:55664/0 (socket says 172.21.15.12:55664) 2024-09-11T01:55:30.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.067+0000 7f4bca59c640 1 -- 172.21.15.12:0/1150566509 learned_addr learned my addr 172.21.15.12:0/1150566509 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:55:30.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bca59c640 1 -- 172.21.15.12:0/1150566509 >> v1:172.21.15.12:6789/0 conn(0x7f4bcc0a2e90 legacy=0x7f4bcc06dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:30.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 1 -- 172.21.15.12:0/1150566509 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 128634077 0 0) 0x7f4bcc0a46c0 con 0x7f4bcc0a2e90 2024-09-11T01:55:30.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 10 monclient(hunting): _negotiate 2024-09-11T01:55:30.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:30.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:30.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 10 monclient(hunting): my global_id is 294153 2024-09-11T01:55:30.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 1 -- 172.21.15.12:0/1150566509 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4bb8003520 con 0x7f4bcc0a2e90 2024-09-11T01:55:30.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bcad9d640 1 -- 172.21.15.12:0/1150566509 >> v1:172.21.15.71:6789/0 conn(0x7f4bcc13d490 legacy=0x7f4bcc13f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:30.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 1 -- 172.21.15.12:0/1150566509 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2056518786 0 0) 0x7f4bb8003520 con 0x7f4bcc0a2e90 2024-09-11T01:55:30.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 1 -- 172.21.15.12:0/1150566509 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bcc0a3270 msgr2=0x7f4bcc0a3680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:55:30.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 1 --2- 172.21.15.12:0/1150566509 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bcc0a3270 0x7f4bcc0a3680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:30.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 1 -- 172.21.15.12:0/1150566509 >> v1:172.21.15.71:6789/0 conn(0x7f4bcc13d490 legacy=0x7f4bcc13f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:30.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:55:30.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:55:30.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:30.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:55:30.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 1 -- 172.21.15.12:0/1150566509 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bcc06eb70 con 0x7f4bcc0a2e90 2024-09-11T01:55:30.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 10 monclient: _finish_auth 0 2024-09-11T01:55:30.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:55:30.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:30.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 1 -- 172.21.15.12:0/1150566509 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f4bbc001990 con 0x7f4bcc0a2e90 2024-09-11T01:55:30.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bd0bf8640 5 monclient: authenticate success, global_id 294153 2024-09-11T01:55:30.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bd0bf8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:55:30.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:30.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:55:30.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 10 monclient: dump: 2024-09-11T01:55:30.083 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:30.083 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:30.083 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:30.084 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:30.084 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:30.084 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:30.084 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:30.084 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:30.084 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:30.084 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:30.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:55:30.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:30.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 1 -- 172.21.15.12:0/1150566509 >> v1:172.21.15.12:6789/0 conn(0x7f4bcc0a2e90 legacy=0x7f4bcc06dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:30.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:55:30.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 1 --2- 172.21.15.12:0/1150566509 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bb8003a30 0x7f4bb8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:30.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 10 monclient(hunting): picked mon.a con 0x7f4bb8003a30 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:55:30.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 1 --2- 172.21.15.12:0/1150566509 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4bb8004800 0x7f4bb8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:30.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 10 monclient(hunting): picked mon.b con 0x7f4bb8004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:55:30.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 1 --2- 172.21.15.12:0/1150566509 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bcc0a3270 0x7f4bb800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:30.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 10 monclient(hunting): picked mon.c con 0x7f4bcc0a3270 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:30.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:30.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 1 -- 172.21.15.12:0/1150566509 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc0a4900 con 0x7f4bb8003a30 2024-09-11T01:55:30.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:30.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 1 -- 172.21.15.12:0/1150566509 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb800bf00 con 0x7f4bb8004800 2024-09-11T01:55:30.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:30.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.069+0000 7f4bc9d9b640 1 -- 172.21.15.12:0/1150566509 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4bb800c080 con 0x7f4bcc0a3270 2024-09-11T01:55:30.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bc9d9b640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:30.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bc9d9b640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:55:30.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bd0bf8640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T01:55:30.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bcad9d640 1 -- 172.21.15.12:0/1150566509 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bcc0a3270 msgr2=0x7f4bb800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:55:30.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bcad9d640 1 --2- 172.21.15.12:0/1150566509 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bcc0a3270 0x7f4bb800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:55:30.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bca59c640 1 --2- 172.21.15.12:0/1150566509 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bb8003a30 0x7f4bb8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:30.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bca59c640 10 monclient(hunting): get_auth_request con 0x7f4bb8003a30 auth_method 0 2024-09-11T01:55:30.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:30.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bca59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:30.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bca59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:55:30.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:30.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:30.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bc3fff640 1 --2- 172.21.15.12:0/1150566509 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4bb8004800 0x7f4bb8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:30.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bca59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:55:30.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bc3fff640 10 monclient(hunting): get_auth_request con 0x7f4bb8004800 auth_method 0 2024-09-11T01:55:30.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:30.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bc3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:30.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bc3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:55:30.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bca59c640 10 monclient(hunting): handle_auth_done global_id 294153 payload 293 2024-09-11T01:55:30.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bca59c640 1 -- 172.21.15.12:0/1150566509 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bcc0a3270 msgr2=0x7f4bb800b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:55:30.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bca59c640 1 --2- 172.21.15.12:0/1150566509 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bcc0a3270 0x7f4bb800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:30.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bca59c640 1 -- 172.21.15.12:0/1150566509 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4bb8004800 msgr2=0x7f4bb8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:55:30.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bca59c640 1 --2- 172.21.15.12:0/1150566509 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4bb8004800 0x7f4bb8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:30.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bca59c640 10 monclient: _finish_hunting 0 2024-09-11T01:55:30.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bca59c640 1 monclient: found mon.a 2024-09-11T01:55:30.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bca59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:30.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bca59c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:30.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bca59c640 1 -- 172.21.15.12:0/1150566509 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bb8003520 con 0x7f4bb8003a30 2024-09-11T01:55:30.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bca59c640 10 monclient: _finish_auth 0 2024-09-11T01:55:30.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bca59c640 10 monclient: _check_auth_tickets 2024-09-11T01:55:30.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bca59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:30.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.070+0000 7f4bc3fff640 1 --2- 172.21.15.12:0/1150566509 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4bb8004800 0x7f4bb8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:55:30.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.073+0000 7f4bca59c640 1 -- 172.21.15.12:0/1150566509 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bb8003a30 msgr2=0x7f4bb8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:30.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.073+0000 7f4bca59c640 1 --2- 172.21.15.12:0/1150566509 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bb8003a30 0x7f4bb8003e70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc06e400 tx=0x7f4bb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7764ae3588417ead server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:30.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.073+0000 7f4bc9d9b640 1 -- 172.21.15.12:0/1150566509 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc00c0e0 con 0x7f4bb8003a30 2024-09-11T01:55:30.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.073+0000 7f4bc9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:30.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.073+0000 7f4bc9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:55:30.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.073+0000 7f4bc9d9b640 10 monclient: dump: 2024-09-11T01:55:30.089 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:30.090 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:30.090 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:30.090 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:30.090 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:30.090 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:30.090 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:30.090 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:30.090 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:30.090 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:30.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.073+0000 7f4bc9d9b640 1 -- 172.21.15.12:0/1150566509 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bbc02e070 con 0x7f4bb8003a30 2024-09-11T01:55:30.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.073+0000 7f4bc9d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:30.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.073+0000 7f4bc9d9b640 1 -- 172.21.15.12:0/1150566509 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc005d30 con 0x7f4bb8003a30 2024-09-11T01:55:30.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.073+0000 7f4bc9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:30.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.073+0000 7f4bc9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:55:30.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.073+0000 7f4bc9d9b640 10 monclient: dump: 2024-09-11T01:55:30.091 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:30.091 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:30.091 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:30.091 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:30.091 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:30.091 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:30.091 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:30.091 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:30.091 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:30.092 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:30.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.073+0000 7f4bd0bf8640 10 monclient: get_monmap_and_config success 2024-09-11T01:55:30.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.073+0000 7f4bd0bf8640 4 set_mon_vals no callback set 2024-09-11T01:55:30.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.073+0000 7f4bd0bf8640 10 monclient: shutdown 2024-09-11T01:55:30.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.073+0000 7f4bd0bf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:30.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.073+0000 7f4bd0bf8640 1 -- 172.21.15.12:0/1150566509 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bb8003a30 msgr2=0x7f4bb8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:30.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.073+0000 7f4bd0bf8640 1 --2- 172.21.15.12:0/1150566509 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bb8003a30 0x7f4bb8003e70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc06e400 tx=0x7f4bb8005830 comp rx=0 tx=0).stop 2024-09-11T01:55:30.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.073+0000 7f4bcad9d640 1 -- 172.21.15.12:0/1150566509 reap_dead start 2024-09-11T01:55:30.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.073+0000 7f4bd0bf8640 1 -- 172.21.15.12:0/1150566509 shutdown_connections 2024-09-11T01:55:30.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.073+0000 7f4bd0bf8640 1 -- 172.21.15.12:0/1150566509 >> 172.21.15.12:0/1150566509 conn(0x7f4bcc0a8490 msgr2=0x7f4bcc0a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:30.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.073+0000 7f4bd0bf8640 1 -- 172.21.15.12:0/1150566509 shutdown_connections 2024-09-11T01:55:30.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.073+0000 7f4bd0bf8640 1 -- 172.21.15.12:0/1150566509 wait complete. 2024-09-11T01:55:30.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.074+0000 7f4bd0bf8640 10 monclient: build_initial_monmap 2024-09-11T01:55:30.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.074+0000 7f4bd0bf8640 10 monclient: monmap: 2024-09-11T01:55:30.093 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:30.093 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:55:30.093 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:55:30.093 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:55:30.093 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:30.093 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:30.093 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:55:30.093 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:55:30.093 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:55:30.094 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:30.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.074+0000 7f4bd0bf8640 1 Processor -- start 2024-09-11T01:55:30.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bd0bf8640 1 -- start start 2024-09-11T01:55:30.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bd0bf8640 10 monclient: init 2024-09-11T01:55:30.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bd0bf8640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:30.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bd0bf8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:55:30.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bd0bf8640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bb8003a30 0x7f4bcc13bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:30.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bd0bf8640 10 monclient(hunting): picked mon.noname-c con 0x7f4bb8003a30 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:30.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bd0bf8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4bcc13c020 0x7f4bcc13c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:30.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bd0bf8640 10 monclient(hunting): picked mon.noname-b con 0x7f4bcc13c020 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:55:30.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bd0bf8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bcc1d1650 0x7f4bcc1d3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:30.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bd0bf8640 10 monclient(hunting): picked mon.noname-a con 0x7f4bcc1d1650 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:55:30.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bd0bf8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:30.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bd0bf8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc0a3800 con 0x7f4bcc1d1650 2024-09-11T01:55:30.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bd0bf8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:30.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bca59c640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bb8003a30 msgr2=0x7f4bcc13bae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:55:30.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bd0bf8640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc0a3680 con 0x7f4bcc13c020 2024-09-11T01:55:30.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bd0bf8640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:30.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bd0bf8640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4bcc0a3980 con 0x7f4bb8003a30 2024-09-11T01:55:30.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bca59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bb8003a30 0x7f4bcc13bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:55:30.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bd0bf8640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:30.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bd0bf8640 10 monclient(hunting): authenticate will time out at 2336.517559s 2024-09-11T01:55:30.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bcad9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bcc1d1650 0x7f4bcc1d3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:30.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bcad9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bcc1d1650 0x7f4bcc1d3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46178/0 (socket says 172.21.15.12:46178) 2024-09-11T01:55:30.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bcad9d640 1 -- 172.21.15.12:0/2590139447 learned_addr learned my addr 172.21.15.12:0/2590139447 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:55:30.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bcad9d640 10 monclient(hunting): get_auth_request con 0x7f4bcc1d1650 auth_method 0 2024-09-11T01:55:30.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bcad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:30.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bcad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:30.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bcad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:30.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bc3fff640 1 --2- 172.21.15.12:0/2590139447 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4bcc13c020 0x7f4bcc13c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:30.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bcad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:30.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bcad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:30.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bcad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:30.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bc3fff640 10 monclient(hunting): get_auth_request con 0x7f4bcc13c020 auth_method 0 2024-09-11T01:55:30.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:30.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bc3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:30.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:30.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:30.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:30.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:30.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bcad9d640 10 monclient(hunting): handle_auth_done global_id 294156 payload 824 2024-09-11T01:55:30.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bcad9d640 1 -- 172.21.15.12:0/2590139447 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bb8003a30 msgr2=0x7f4bcc13bae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:55:30.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bcad9d640 1 --2- 172.21.15.12:0/2590139447 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bb8003a30 0x7f4bcc13bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:30.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bcad9d640 1 -- 172.21.15.12:0/2590139447 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4bcc13c020 msgr2=0x7f4bcc13c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:30.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bcad9d640 1 --2- 172.21.15.12:0/2590139447 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4bcc13c020 0x7f4bcc13c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:30.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bcad9d640 10 monclient: _finish_hunting 0 2024-09-11T01:55:30.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bcad9d640 1 monclient: found mon.noname-a 2024-09-11T01:55:30.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bcad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:30.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bcad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:55:30.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.075+0000 7f4bcad9d640 1 -- 172.21.15.12:0/2590139447 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bcc138070 con 0x7f4bcc1d1650 2024-09-11T01:55:30.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.078+0000 7f4bcad9d640 1 -- 172.21.15.12:0/2590139447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bcc1d1650 msgr2=0x7f4bcc1d3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:30.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.078+0000 7f4bcad9d640 1 --2- 172.21.15.12:0/2590139447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bcc1d1650 0x7f4bcc1d3a40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc06e450 tx=0x7f4bc400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:30.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.078+0000 7f4bc2ffd640 1 -- 172.21.15.12:0/2590139447 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc4017070 con 0x7f4bcc1d1650 2024-09-11T01:55:30.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.078+0000 7f4bc2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:30.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.078+0000 7f4bc2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:55:30.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.078+0000 7f4bc2ffd640 10 monclient: dump: 2024-09-11T01:55:30.098 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:30.098 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:30.098 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:30.098 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:30.098 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:30.099 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:30.099 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:30.099 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:30.099 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:30.099 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:30.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.078+0000 7f4bc2ffd640 10 monclient: _finish_auth 0 2024-09-11T01:55:30.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.078+0000 7f4bc2ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:55:30.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.078+0000 7f4bc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:30.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.078+0000 7f4bc2ffd640 1 -- 172.21.15.12:0/2590139447 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bc400a610 con 0x7f4bcc1d1650 2024-09-11T01:55:30.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.078+0000 7f4bd0bf8640 5 monclient: authenticate success, global_id 294156 2024-09-11T01:55:30.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.078+0000 7f4bd0bf8640 10 monclient: _renew_subs 2024-09-11T01:55:30.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.078+0000 7f4bd0bf8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:30.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.078+0000 7f4bd0bf8640 1 -- 172.21.15.12:0/2590139447 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bcc125320 con 0x7f4bcc1d1650 2024-09-11T01:55:30.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.078+0000 7f4bc2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.078+0000 7f4bc2ffd640 1 -- 172.21.15.12:0/2590139447 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc400aa00 con 0x7f4bcc1d1650 2024-09-11T01:55:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.078+0000 7f4bc2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.078+0000 7f4bc2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:55:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.078+0000 7f4bc2ffd640 10 monclient: dump: 2024-09-11T01:55:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:30.100 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:30.100 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:30.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.078+0000 7f4bd0bf8640 10 monclient: _renew_subs 2024-09-11T01:55:30.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.078+0000 7f4bd0bf8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:30.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.078+0000 7f4bc959a640 4 set_mon_vals no callback set 2024-09-11T01:55:30.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.078+0000 7f4bd0bf8640 1 -- 172.21.15.12:0/2590139447 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bcc1244b0 con 0x7f4bcc1d1650 2024-09-11T01:55:30.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.079+0000 7f4bc2ffd640 1 -- 172.21.15.12:0/2590139447 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f4bc400aba0 con 0x7f4bcc1d1650 2024-09-11T01:55:30.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.079+0000 7f4bd0bf8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:30.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.079+0000 7f4bc2ffd640 1 --2- 172.21.15.12:0/2590139447 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4bac03e1c0 0x7f4bac040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:30.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.079+0000 7f4bd0bf8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:30.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.079+0000 7f4bd0bf8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:30.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.079+0000 7f4bd0bf8640 1 -- 172.21.15.12:0/2590139447 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bcc06dc60 con 0x7f4bcc1d1650 2024-09-11T01:55:30.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.079+0000 7f4bc2ffd640 1 -- 172.21.15.12:0/2590139447 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(889..889 src has 676..889) ==== 4803+0+0 (secure 0 0 0) 0x7f4bc4052370 con 0x7f4bcc1d1650 2024-09-11T01:55:30.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.079+0000 7f4bca59c640 1 --2- 172.21.15.12:0/2590139447 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4bac03e1c0 0x7f4bac040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:30.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.079+0000 7f4bca59c640 10 monclient: get_auth_request con 0x7f4bac03e1c0 auth_method 0 2024-09-11T01:55:30.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.082+0000 7f4bca59c640 1 --2- 172.21.15.12:0/2590139447 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4bac03e1c0 0x7f4bac040680 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc001bb0 tx=0x7f4bbc02e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:30.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.084+0000 7f4bc2ffd640 1 -- 172.21.15.12:0/2590139447 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4bc401b3b0 con 0x7f4bcc1d1650 2024-09-11T01:55:30.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.084+0000 7f4bc2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:30.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.084+0000 7f4bc2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:55:30.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.339+0000 7f4bd0bf8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:55:30.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.339+0000 7f4bd0bf8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:55:30.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.339+0000 7f4bd0bf8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:30.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.339+0000 7f4bd0bf8640 1 -- 172.21.15.12:0/2590139447 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4bcc0698e0 con 0x7f4bcc1d1650 2024-09-11T01:55:30.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.340+0000 7f4bc2ffd640 1 -- 172.21.15.12:0/2590139447 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f4bc4052080 con 0x7f4bcc1d1650 2024-09-11T01:55:30.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.340+0000 7f4bc2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:55:30.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.340+0000 7f4bc2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:55:30.342 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":240,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:55:30.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.342+0000 7f4bd0bf8640 1 -- 172.21.15.12:0/2590139447 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4bac03e1c0 msgr2=0x7f4bac040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:30.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.342+0000 7f4bd0bf8640 1 --2- 172.21.15.12:0/2590139447 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4bac03e1c0 0x7f4bac040680 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc001bb0 tx=0x7f4bbc02e000 comp rx=0 tx=0).stop 2024-09-11T01:55:30.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.342+0000 7f4bd0bf8640 10 monclient: shutdown 2024-09-11T01:55:30.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.342+0000 7f4bd0bf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:30.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.342+0000 7f4bd0bf8640 1 -- 172.21.15.12:0/2590139447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bcc1d1650 msgr2=0x7f4bcc1d3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:30.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.342+0000 7f4bd0bf8640 1 --2- 172.21.15.12:0/2590139447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bcc1d1650 0x7f4bcc1d3a40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc06e450 tx=0x7f4bc400cb90 comp rx=0 tx=0).stop 2024-09-11T01:55:30.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.342+0000 7f4bd0bf8640 1 -- 172.21.15.12:0/2590139447 shutdown_connections 2024-09-11T01:55:30.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.342+0000 7f4bd0bf8640 1 --2- 172.21.15.12:0/2590139447 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4bac03e1c0 0x7f4bac040680 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:30.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.342+0000 7f4bd0bf8640 1 --2- 172.21.15.12:0/2590139447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4bcc1d1650 0x7f4bcc1d3a40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:30.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.342+0000 7f4bd0bf8640 1 --2- 172.21.15.12:0/2590139447 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4bcc13c020 0x7f4bcc13c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:30.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.342+0000 7f4bd0bf8640 1 --2- 172.21.15.12:0/2590139447 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bb8003a30 0x7f4bcc13bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:30.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.342+0000 7f4bd0bf8640 1 -- 172.21.15.12:0/2590139447 >> 172.21.15.12:0/2590139447 conn(0x7f4bcc0a8490 msgr2=0x7f4bcc072f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:30.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.342+0000 7f4bd0bf8640 1 -- 172.21.15.12:0/2590139447 shutdown_connections 2024-09-11T01:55:30.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:30.342+0000 7f4bd0bf8640 1 -- 172.21.15.12:0/2590139447 wait complete. 2024-09-11T01:55:33.355 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:55:33.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.467+0000 7ffab9408640 10 monclient: get_monmap_and_config 2024-09-11T01:55:33.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.467+0000 7ffab9408640 10 monclient: build_initial_monmap 2024-09-11T01:55:33.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.467+0000 7ffab9408640 10 monclient: monmap: 2024-09-11T01:55:33.469 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:33.469 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:33.470 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:55:33.468456+0000 2024-09-11T01:55:33.470 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:55:33.468456+0000 2024-09-11T01:55:33.470 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:33.470 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:33.470 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:55:33.470 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:55:33.470 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:55:33.470 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:55:33.470 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:55:33.470 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:33.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.468+0000 7ffab9408640 1 Processor -- start 2024-09-11T01:55:33.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.468+0000 7ffab9408640 1 -- start start 2024-09-11T01:55:33.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.468+0000 7ffab9408640 10 monclient: init 2024-09-11T01:55:33.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.468+0000 7ffab9408640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:33.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.468+0000 7ffab9408640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-11T01:55:33.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.468+0000 7ffab9408640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffab4073d90 addr v1:172.21.15.71:6789/0 2024-09-11T01:55:33.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.468+0000 7ffab9408640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffab40a0660 addr v1:172.21.15.12:6789/0 2024-09-11T01:55:33.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.468+0000 7ffab9408640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ffab41380d0 0x7ffab41406c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:33.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.468+0000 7ffab9408640 10 monclient(hunting): picked mon.noname-a con 0x7ffab41380d0 addr v2:172.21.15.12:3300/0 2024-09-11T01:55:33.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.468+0000 7ffab9408640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffab4070280 con 0x7ffab40a0660 2024-09-11T01:55:33.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.468+0000 7ffab9408640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffab4070560 con 0x7ffab4073d90 2024-09-11T01:55:33.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.469+0000 7ffab9408640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:33.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.469+0000 7ffab9408640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ffab4070840 con 0x7ffab41380d0 2024-09-11T01:55:33.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.469+0000 7ffab9408640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:33.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.469+0000 7ffab9408640 10 monclient(hunting): authenticate will time out at 2339.911026s 2024-09-11T01:55:33.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.469+0000 7ffab37fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ffab41380d0 0x7ffab41406c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:33.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.469+0000 7ffab37fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ffab41380d0 0x7ffab41406c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46182/0 (socket says 172.21.15.12:46182) 2024-09-11T01:55:33.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.469+0000 7ffab37fe640 1 -- 172.21.15.12:0/1117910781 learned_addr learned my addr 172.21.15.12:0/1117910781 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:55:33.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.469+0000 7ffab37fe640 10 monclient(hunting): get_auth_request con 0x7ffab41380d0 auth_method 0 2024-09-11T01:55:33.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.469+0000 7ffab37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:33.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.469+0000 7ffab37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:33.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.469+0000 7ffab37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:33.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.469+0000 7ffab37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:33.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.469+0000 7ffab37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:33.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.469+0000 7ffab37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:33.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.471+0000 7ffaaa7fc640 1 -- 172.21.15.12:0/1117910781 >> v1:172.21.15.12:6789/0 conn(0x7ffab40a0660 legacy=0x7ffab413cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:33.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.471+0000 7ffab2ffd640 1 -- 172.21.15.12:0/1117910781 >> v1:172.21.15.71:6789/0 conn(0x7ffab4073d90 legacy=0x7ffab4074170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:33.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.471+0000 7ffab37fe640 10 monclient(hunting): handle_auth_done global_id 294159 payload 274 2024-09-11T01:55:33.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.471+0000 7ffab37fe640 1 -- 172.21.15.12:0/1117910781 >> v1:172.21.15.71:6789/0 conn(0x7ffab4073d90 legacy=0x7ffab4074170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:33.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.471+0000 7ffab37fe640 1 -- 172.21.15.12:0/1117910781 >> v1:172.21.15.12:6789/0 conn(0x7ffab40a0660 legacy=0x7ffab413cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:33.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.471+0000 7ffab37fe640 10 monclient: _finish_hunting 0 2024-09-11T01:55:33.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.471+0000 7ffab37fe640 1 monclient: found mon.noname-a 2024-09-11T01:55:33.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.471+0000 7ffab37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:33.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.471+0000 7ffab37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:55:33.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.471+0000 7ffab37fe640 1 -- 172.21.15.12:0/1117910781 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab409fff0 con 0x7ffab41380d0 2024-09-11T01:55:33.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.474+0000 7ffab37fe640 1 -- 172.21.15.12:0/1117910781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffab41380d0 msgr2=0x7ffab41406c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:33.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.474+0000 7ffab37fe640 1 --2- 172.21.15.12:0/1117910781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffab41380d0 0x7ffab41406c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ffab40a0220 tx=0x7ffaac02a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d32376f42e284be server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:33.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.474+0000 7ffab27fc640 1 -- 172.21.15.12:0/1117910781 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaac007800 con 0x7ffab41380d0 2024-09-11T01:55:33.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.474+0000 7ffab27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:33.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.474+0000 7ffab27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:55:33.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.474+0000 7ffab27fc640 10 monclient: dump: 2024-09-11T01:55:33.480 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:33.480 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:33.481 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:33.481 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:33.481 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:33.481 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:33.481 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:33.481 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:33.481 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:33.481 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:33.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.474+0000 7ffab27fc640 10 monclient: _finish_auth 0 2024-09-11T01:55:33.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.474+0000 7ffab27fc640 10 monclient: _check_auth_tickets 2024-09-11T01:55:33.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.474+0000 7ffab27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:33.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.474+0000 7ffab27fc640 1 -- 172.21.15.12:0/1117910781 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaac0079a0 con 0x7ffab41380d0 2024-09-11T01:55:33.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.474+0000 7ffab9408640 5 monclient: authenticate success, global_id 294159 2024-09-11T01:55:33.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.474+0000 7ffab9408640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:55:33.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.474+0000 7ffab27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:33.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.474+0000 7ffab9408640 10 monclient: get_monmap_and_config success 2024-09-11T01:55:33.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.474+0000 7ffab9408640 4 set_mon_vals no callback set 2024-09-11T01:55:33.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.474+0000 7ffab27fc640 1 -- 172.21.15.12:0/1117910781 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaac007d70 con 0x7ffab41380d0 2024-09-11T01:55:33.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.474+0000 7ffab27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:33.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.474+0000 7ffab27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:55:33.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.474+0000 7ffab27fc640 10 monclient: dump: 2024-09-11T01:55:33.483 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:33.483 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:33.483 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:33.483 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:33.483 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:33.483 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:33.483 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:33.483 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:33.483 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:33.483 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:33.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.475+0000 7ffab9408640 10 monclient: shutdown 2024-09-11T01:55:33.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.475+0000 7ffab9408640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:33.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.475+0000 7ffab9408640 1 -- 172.21.15.12:0/1117910781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffab41380d0 msgr2=0x7ffab41406c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:33.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.475+0000 7ffab9408640 1 --2- 172.21.15.12:0/1117910781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffab41380d0 0x7ffab41406c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ffab40a0220 tx=0x7ffaac02a780 comp rx=0 tx=0).stop 2024-09-11T01:55:33.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.475+0000 7ffab9408640 1 -- 172.21.15.12:0/1117910781 shutdown_connections 2024-09-11T01:55:33.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.475+0000 7ffab9408640 1 --2- 172.21.15.12:0/1117910781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffab41380d0 0x7ffab41406c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:33.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.475+0000 7ffab9408640 1 -- 172.21.15.12:0/1117910781 >> 172.21.15.12:0/1117910781 conn(0x7ffab40a3650 msgr2=0x7ffab40a3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:33.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.475+0000 7ffab9408640 1 -- 172.21.15.12:0/1117910781 shutdown_connections 2024-09-11T01:55:33.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.475+0000 7ffab9408640 1 -- 172.21.15.12:0/1117910781 wait complete. 2024-09-11T01:55:33.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.476+0000 7ffab9408640 10 monclient: build_initial_monmap 2024-09-11T01:55:33.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.476+0000 7ffab9408640 10 monclient: monmap: 2024-09-11T01:55:33.484 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:33.484 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:55:33.484 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:55:33.484 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:55:33.485 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:33.485 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:33.485 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:55:33.485 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:55:33.485 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:55:33.485 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:33.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.476+0000 7ffab9408640 1 Processor -- start 2024-09-11T01:55:33.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.476+0000 7ffab9408640 1 -- start start 2024-09-11T01:55:33.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.476+0000 7ffab9408640 10 monclient: init 2024-09-11T01:55:33.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab9408640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:33.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab9408640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:55:33.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab9408640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffab4073d90 0x7ffab413f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:33.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab9408640 10 monclient(hunting): picked mon.noname-a con 0x7ffab4073d90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:55:33.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab9408640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffab40a0660 0x7ffab41316e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:33.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab9408640 10 monclient(hunting): picked mon.noname-b con 0x7ffab40a0660 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:55:33.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab9408640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffab41380d0 0x7ffab4131c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:33.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab9408640 10 monclient(hunting): picked mon.noname-c con 0x7ffab41380d0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:33.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab9408640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:33.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab9408640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ffab4140c00 con 0x7ffab4073d90 2024-09-11T01:55:33.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab9408640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:33.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab9408640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ffab40a4560 con 0x7ffab40a0660 2024-09-11T01:55:33.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab9408640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:33.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab9408640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ffab4140d80 con 0x7ffab41380d0 2024-09-11T01:55:33.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab9408640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:33.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab9408640 10 monclient(hunting): authenticate will time out at 2339.919152s 2024-09-11T01:55:33.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab37fe640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffab41380d0 msgr2=0x7ffab4131c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:55:33.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab37fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffab41380d0 0x7ffab4131c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:55:33.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab2ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffab4073d90 0x7ffab413f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:33.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab2ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffab4073d90 0x7ffab413f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46186/0 (socket says 172.21.15.12:46186) 2024-09-11T01:55:33.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab2ffd640 1 -- 172.21.15.12:0/182062845 learned_addr learned my addr 172.21.15.12:0/182062845 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:55:33.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab2ffd640 10 monclient(hunting): get_auth_request con 0x7ffab4073d90 auth_method 0 2024-09-11T01:55:33.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:33.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:33.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:33.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:33.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:33.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:33.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffaaa7fc640 1 --2- 172.21.15.12:0/182062845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffab40a0660 0x7ffab41316e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:33.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab2ffd640 10 monclient(hunting): handle_auth_done global_id 294162 payload 824 2024-09-11T01:55:33.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab2ffd640 1 -- 172.21.15.12:0/182062845 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffab41380d0 msgr2=0x7ffab4131c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:55:33.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab2ffd640 1 --2- 172.21.15.12:0/182062845 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffab41380d0 0x7ffab4131c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:33.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab2ffd640 1 -- 172.21.15.12:0/182062845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffab40a0660 msgr2=0x7ffab41316e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:33.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab2ffd640 1 --2- 172.21.15.12:0/182062845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffab40a0660 0x7ffab41316e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:33.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab2ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:55:33.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab2ffd640 1 monclient: found mon.noname-a 2024-09-11T01:55:33.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:33.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:55:33.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffab2ffd640 1 -- 172.21.15.12:0/182062845 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab413c480 con 0x7ffab4073d90 2024-09-11T01:55:33.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffaaa7fc640 10 monclient: get_auth_request con 0x7ffab40a0660 auth_method 0 2024-09-11T01:55:33.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.477+0000 7ffaaa7fc640 1 --2- 172.21.15.12:0/182062845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffab40a0660 0x7ffab41316e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:55:33.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.479+0000 7ffab2ffd640 1 -- 172.21.15.12:0/182062845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffab4073d90 msgr2=0x7ffab413f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:33.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.479+0000 7ffab2ffd640 1 --2- 172.21.15.12:0/182062845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffab4073d90 0x7ffab413f1e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ffaa400c490 tx=0x7ffaa400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:33.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.479+0000 7ffaabfff640 1 -- 172.21.15.12:0/182062845 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa4017070 con 0x7ffab4073d90 2024-09-11T01:55:33.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.479+0000 7ffaabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:33.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.480+0000 7ffaabfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:55:33.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.480+0000 7ffaabfff640 10 monclient: dump: 2024-09-11T01:55:33.492 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:33.492 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:33.492 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:33.492 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:33.492 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:33.492 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:33.492 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:33.492 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:33.492 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:33.492 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:33.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.480+0000 7ffaabfff640 10 monclient: _finish_auth 0 2024-09-11T01:55:33.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.480+0000 7ffaabfff640 10 monclient: _check_auth_tickets 2024-09-11T01:55:33.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.480+0000 7ffaabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:33.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.480+0000 7ffaabfff640 1 -- 172.21.15.12:0/182062845 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaa40057c0 con 0x7ffab4073d90 2024-09-11T01:55:33.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.480+0000 7ffab9408640 5 monclient: authenticate success, global_id 294162 2024-09-11T01:55:33.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.480+0000 7ffab9408640 10 monclient: _renew_subs 2024-09-11T01:55:33.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.480+0000 7ffab9408640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:33.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.480+0000 7ffab9408640 1 -- 172.21.15.12:0/182062845 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffab413cfe0 con 0x7ffab4073d90 2024-09-11T01:55:33.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.480+0000 7ffaabfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:33.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.480+0000 7ffaabfff640 1 -- 172.21.15.12:0/182062845 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa4005bb0 con 0x7ffab4073d90 2024-09-11T01:55:33.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.480+0000 7ffaabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:33.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.480+0000 7ffaabfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:55:33.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.480+0000 7ffaabfff640 10 monclient: dump: 2024-09-11T01:55:33.493 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:33.494 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:33.494 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:33.494 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:33.494 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:33.494 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:33.494 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:33.494 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:33.494 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:33.494 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:33.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.480+0000 7ffab9408640 10 monclient: _renew_subs 2024-09-11T01:55:33.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.480+0000 7ffab1ffb640 4 set_mon_vals no callback set 2024-09-11T01:55:33.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.480+0000 7ffab9408640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:33.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.480+0000 7ffab9408640 1 -- 172.21.15.12:0/182062845 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffab413aa50 con 0x7ffab4073d90 2024-09-11T01:55:33.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.481+0000 7ffab9408640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:33.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.481+0000 7ffab9408640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:33.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.481+0000 7ffab9408640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:33.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.481+0000 7ffab9408640 1 -- 172.21.15.12:0/182062845 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffab4074fb0 con 0x7ffab4073d90 2024-09-11T01:55:33.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.481+0000 7ffaabfff640 1 -- 172.21.15.12:0/182062845 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ffaa401b3b0 con 0x7ffab4073d90 2024-09-11T01:55:33.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.481+0000 7ffaabfff640 1 --2- 172.21.15.12:0/182062845 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffa9403e1c0 0x7ffa94040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:33.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.481+0000 7ffaabfff640 1 -- 172.21.15.12:0/182062845 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(889..889 src has 676..889) ==== 4803+0+0 (secure 0 0 0) 0x7ffaa40523e0 con 0x7ffab4073d90 2024-09-11T01:55:33.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.481+0000 7ffaaa7fc640 1 --2- 172.21.15.12:0/182062845 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffa9403e1c0 0x7ffa94040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:33.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.482+0000 7ffaaa7fc640 10 monclient: get_auth_request con 0x7ffa9403e1c0 auth_method 0 2024-09-11T01:55:33.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.484+0000 7ffaaa7fc640 1 --2- 172.21.15.12:0/182062845 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffa9403e1c0 0x7ffa94040680 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c007fd0 tx=0x7ffa9c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:33.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.486+0000 7ffaabfff640 1 -- 172.21.15.12:0/182062845 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ffaa401be80 con 0x7ffab4073d90 2024-09-11T01:55:33.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.486+0000 7ffaabfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:33.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.486+0000 7ffaabfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:55:33.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.743+0000 7ffab9408640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:55:33.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.743+0000 7ffab9408640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:55:33.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.743+0000 7ffab9408640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.743+0000 7ffab9408640 1 -- 172.21.15.12:0/182062845 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ffab4074170 con 0x7ffab4073d90 2024-09-11T01:55:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.744+0000 7ffaabfff640 1 -- 172.21.15.12:0/182062845 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7ffaa401a360 con 0x7ffab4073d90 2024-09-11T01:55:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.744+0000 7ffaabfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:55:33.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.744+0000 7ffaabfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:55:33.746 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":240,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:55:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.746+0000 7ffab9408640 1 -- 172.21.15.12:0/182062845 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffa9403e1c0 msgr2=0x7ffa94040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.746+0000 7ffab9408640 1 --2- 172.21.15.12:0/182062845 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffa9403e1c0 0x7ffa94040680 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c007fd0 tx=0x7ffa9c009040 comp rx=0 tx=0).stop 2024-09-11T01:55:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.746+0000 7ffab9408640 10 monclient: shutdown 2024-09-11T01:55:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.746+0000 7ffab9408640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.746+0000 7ffab9408640 1 -- 172.21.15.12:0/182062845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffab4073d90 msgr2=0x7ffab413f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.746+0000 7ffab9408640 1 --2- 172.21.15.12:0/182062845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffab4073d90 0x7ffab413f1e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ffaa400c490 tx=0x7ffaa400c950 comp rx=0 tx=0).stop 2024-09-11T01:55:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.746+0000 7ffab9408640 1 -- 172.21.15.12:0/182062845 shutdown_connections 2024-09-11T01:55:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.746+0000 7ffab9408640 1 --2- 172.21.15.12:0/182062845 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffab41380d0 0x7ffab4131c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:33.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.746+0000 7ffab9408640 1 --2- 172.21.15.12:0/182062845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffab40a0660 0x7ffab41316e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.746+0000 7ffab9408640 1 --2- 172.21.15.12:0/182062845 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffa9403e1c0 0x7ffa94040680 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.746+0000 7ffab9408640 1 --2- 172.21.15.12:0/182062845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffab4073d90 0x7ffab413f1e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.746+0000 7ffab9408640 1 -- 172.21.15.12:0/182062845 >> 172.21.15.12:0/182062845 conn(0x7ffab40a3650 msgr2=0x7ffab4073290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.746+0000 7ffab9408640 1 -- 172.21.15.12:0/182062845 shutdown_connections 2024-09-11T01:55:33.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:33.746+0000 7ffab9408640 1 -- 172.21.15.12:0/182062845 wait complete. 2024-09-11T01:55:36.759 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:55:36.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.870+0000 7f0bc4a85640 10 monclient: get_monmap_and_config 2024-09-11T01:55:36.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.870+0000 7f0bc4a85640 10 monclient: build_initial_monmap 2024-09-11T01:55:36.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.870+0000 7f0bc4a85640 10 monclient: monmap: 2024-09-11T01:55:36.872 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:36.873 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:36.873 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:55:36.871586+0000 2024-09-11T01:55:36.873 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:55:36.871586+0000 2024-09-11T01:55:36.873 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:36.873 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:36.873 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:55:36.873 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:55:36.873 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:55:36.873 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:55:36.873 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:55:36.873 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:36.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.871+0000 7f0bc4a85640 1 Processor -- start 2024-09-11T01:55:36.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.871+0000 7f0bc4a85640 1 -- start start 2024-09-11T01:55:36.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.871+0000 7f0bc4a85640 10 monclient: init 2024-09-11T01:55:36.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bc4a85640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:36.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bc4a85640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-11T01:55:36.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bc4a85640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0bc00a2e90 0x7f0bc006dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:36.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bc4a85640 10 monclient(hunting): picked mon.noname-b con 0x7f0bc00a2e90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:36.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bc4a85640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0bc00a3270 0x7f0bc00a3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:36.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bc4a85640 10 monclient(hunting): picked mon.noname-a con 0x7f0bc00a3270 addr v2:172.21.15.12:3300/0 2024-09-11T01:55:36.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bc4a85640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0bc0138040 0x7f0bc0138460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:36.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bc4a85640 10 monclient(hunting): picked mon.noname-c con 0x7f0bc0138040 addr v2:172.21.15.71:3300/0 2024-09-11T01:55:36.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bc4a85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:36.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bc4a85640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f0bc00a4900 con 0x7f0bc00a3270 2024-09-11T01:55:36.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bc4a85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:36.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bc4a85640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f0bc00a46c0 con 0x7f0bc0138040 2024-09-11T01:55:36.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bc4a85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:36.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bc4a85640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f0bc006ee50 con 0x7f0bc00a2e90 2024-09-11T01:55:36.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bc4a85640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:36.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bc4a85640 10 monclient(hunting): authenticate will time out at 2343.314295s 2024-09-11T01:55:36.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbe59c640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0bc00a2e90 msgr2=0x7f0bc006dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:55:36.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbe59c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0bc00a2e90 0x7f0bc006dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:55:36.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbdd9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0bc00a3270 0x7f0bc00a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:36.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbdd9b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0bc00a3270 0x7f0bc00a3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49020/0 (socket says 172.21.15.12:49020) 2024-09-11T01:55:36.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbdd9b640 1 -- 172.21.15.12:0/2168178597 learned_addr learned my addr 172.21.15.12:0/2168178597 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:55:36.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbdd9b640 10 monclient(hunting): get_auth_request con 0x7f0bc00a3270 auth_method 0 2024-09-11T01:55:36.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:36.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:36.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:36.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:36.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:36.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:36.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbed9d640 1 --2- 172.21.15.12:0/2168178597 >> v2:172.21.15.71:3300/0 conn(0x7f0bc0138040 0x7f0bc0138460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:36.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbed9d640 10 monclient(hunting): get_auth_request con 0x7f0bc0138040 auth_method 0 2024-09-11T01:55:36.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:36.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:36.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:36.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbdd9b640 10 monclient(hunting): handle_auth_done global_id 294165 payload 274 2024-09-11T01:55:36.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbdd9b640 1 -- 172.21.15.12:0/2168178597 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0bc00a2e90 msgr2=0x7f0bc006dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:55:36.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbdd9b640 1 --2- 172.21.15.12:0/2168178597 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0bc00a2e90 0x7f0bc006dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:36.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbdd9b640 1 -- 172.21.15.12:0/2168178597 >> v2:172.21.15.71:3300/0 conn(0x7f0bc0138040 msgr2=0x7f0bc0138460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:55:36.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbdd9b640 1 --2- 172.21.15.12:0/2168178597 >> v2:172.21.15.71:3300/0 conn(0x7f0bc0138040 0x7f0bc0138460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:36.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbdd9b640 10 monclient: _finish_hunting 0 2024-09-11T01:55:36.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbdd9b640 1 monclient: found mon.noname-a 2024-09-11T01:55:36.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:36.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbdd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:55:36.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.872+0000 7f0bbdd9b640 1 -- 172.21.15.12:0/2168178597 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bc006eb70 con 0x7f0bc00a3270 2024-09-11T01:55:36.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.875+0000 7f0bbdd9b640 1 -- 172.21.15.12:0/2168178597 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bc00a3270 msgr2=0x7f0bc00a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:36.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.875+0000 7f0bbdd9b640 1 --2- 172.21.15.12:0/2168178597 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bc00a3270 0x7f0bc00a3650 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0bc006e860 tx=0x7f0ba802a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91cdc991f3a45b4e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:36.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.875+0000 7f0bbd59a640 1 -- 172.21.15.12:0/2168178597 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba8037070 con 0x7f0bc00a3270 2024-09-11T01:55:36.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.875+0000 7f0bbd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:36.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.875+0000 7f0bbd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:55:36.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.875+0000 7f0bbd59a640 10 monclient: dump: 2024-09-11T01:55:36.885 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:36.885 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:36.885 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:36.885 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:36.885 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:36.885 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:36.885 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:36.885 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:36.885 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:36.886 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:36.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.875+0000 7f0bbd59a640 10 monclient: _finish_auth 0 2024-09-11T01:55:36.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.875+0000 7f0bbd59a640 10 monclient: _check_auth_tickets 2024-09-11T01:55:36.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.875+0000 7f0bbd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:36.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.875+0000 7f0bbd59a640 1 -- 172.21.15.12:0/2168178597 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ba8032640 con 0x7f0bc00a3270 2024-09-11T01:55:36.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.875+0000 7f0bc4a85640 5 monclient: authenticate success, global_id 294165 2024-09-11T01:55:36.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.875+0000 7f0bc4a85640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:55:36.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.875+0000 7f0bbd59a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:36.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.875+0000 7f0bc4a85640 10 monclient: get_monmap_and_config success 2024-09-11T01:55:36.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.875+0000 7f0bbd59a640 1 -- 172.21.15.12:0/2168178597 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba8032a10 con 0x7f0bc00a3270 2024-09-11T01:55:36.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.875+0000 7f0bc4a85640 4 set_mon_vals no callback set 2024-09-11T01:55:36.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.875+0000 7f0bbd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:36.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.875+0000 7f0bbd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:55:36.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.875+0000 7f0bbd59a640 10 monclient: dump: 2024-09-11T01:55:36.887 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:36.887 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:36.887 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:36.887 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:36.887 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:36.887 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:36.887 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:36.887 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:36.887 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:36.887 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:36.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.876+0000 7f0bc4a85640 10 monclient: shutdown 2024-09-11T01:55:36.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.876+0000 7f0bc4a85640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:36.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.876+0000 7f0bc4a85640 1 -- 172.21.15.12:0/2168178597 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bc00a3270 msgr2=0x7f0bc00a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:36.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.876+0000 7f0bc4a85640 1 --2- 172.21.15.12:0/2168178597 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bc00a3270 0x7f0bc00a3650 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0bc006e860 tx=0x7f0ba802a0d0 comp rx=0 tx=0).stop 2024-09-11T01:55:36.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.876+0000 7f0bc4a85640 1 -- 172.21.15.12:0/2168178597 shutdown_connections 2024-09-11T01:55:36.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.876+0000 7f0bc4a85640 1 --2- 172.21.15.12:0/2168178597 >> v2:172.21.15.71:3300/0 conn(0x7f0bc0138040 0x7f0bc0138460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:36.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.876+0000 7f0bc4a85640 1 --2- 172.21.15.12:0/2168178597 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bc00a3270 0x7f0bc00a3650 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:36.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.876+0000 7f0bc4a85640 1 --2- 172.21.15.12:0/2168178597 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0bc00a2e90 0x7f0bc006dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:36.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.876+0000 7f0bc4a85640 1 -- 172.21.15.12:0/2168178597 >> 172.21.15.12:0/2168178597 conn(0x7f0bc00a8490 msgr2=0x7f0bc00a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:36.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.876+0000 7f0bc4a85640 1 -- 172.21.15.12:0/2168178597 shutdown_connections 2024-09-11T01:55:36.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.876+0000 7f0bc4a85640 1 -- 172.21.15.12:0/2168178597 wait complete. 2024-09-11T01:55:36.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.877+0000 7f0bc4a85640 10 monclient: build_initial_monmap 2024-09-11T01:55:36.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.877+0000 7f0bc4a85640 10 monclient: monmap: 2024-09-11T01:55:36.888 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:36.889 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:55:36.889 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:55:36.889 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:55:36.889 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:36.889 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:36.889 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:55:36.889 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:55:36.889 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:55:36.889 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:36.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.877+0000 7f0bc4a85640 1 Processor -- start 2024-09-11T01:55:36.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.877+0000 7f0bc4a85640 1 -- start start 2024-09-11T01:55:36.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.877+0000 7f0bc4a85640 10 monclient: init 2024-09-11T01:55:36.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bc4a85640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:36.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bc4a85640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:55:36.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bc4a85640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bc00a2e90 0x7f0bc0124880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:36.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bc4a85640 10 monclient(hunting): picked mon.noname-a con 0x7f0bc00a2e90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:55:36.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bc4a85640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0bc00a3270 0x7f0bc0124dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:36.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bc4a85640 10 monclient(hunting): picked mon.noname-c con 0x7f0bc00a3270 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:36.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bc4a85640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0bc0138040 0x7f0bc0125300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:36.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bc4a85640 10 monclient(hunting): picked mon.noname-b con 0x7f0bc0138040 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:55:36.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bc4a85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:36.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bc4a85640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0bc00a3b90 con 0x7f0bc00a2e90 2024-09-11T01:55:36.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bc4a85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:36.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bc4a85640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0bc006fba0 con 0x7f0bc0138040 2024-09-11T01:55:36.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bc4a85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:36.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bc4a85640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f0bc009f680 con 0x7f0bc00a3270 2024-09-11T01:55:36.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bc4a85640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:36.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bc4a85640 10 monclient(hunting): authenticate will time out at 2343.320210s 2024-09-11T01:55:36.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bbdd9b640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0bc00a3270 msgr2=0x7f0bc0124dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:55:36.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bbdd9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0bc00a3270 0x7f0bc0124dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:55:36.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bbe59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bc00a2e90 0x7f0bc0124880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:36.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bbe59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bc00a2e90 0x7f0bc0124880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49030/0 (socket says 172.21.15.12:49030) 2024-09-11T01:55:36.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bbe59c640 1 -- 172.21.15.12:0/1311065216 learned_addr learned my addr 172.21.15.12:0/1311065216 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:55:36.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bbe59c640 10 monclient(hunting): get_auth_request con 0x7f0bc00a2e90 auth_method 0 2024-09-11T01:55:36.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bbe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:36.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bbe59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:36.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bbe59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:36.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bbe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:36.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bbe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:36.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bbe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:36.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bbe59c640 10 monclient(hunting): handle_auth_done global_id 294168 payload 824 2024-09-11T01:55:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bbed9d640 1 --2- 172.21.15.12:0/1311065216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0bc0138040 0x7f0bc0125300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bbe59c640 1 -- 172.21.15.12:0/1311065216 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0bc00a3270 msgr2=0x7f0bc0124dc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:55:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bbe59c640 1 --2- 172.21.15.12:0/1311065216 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0bc00a3270 0x7f0bc0124dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bbe59c640 1 -- 172.21.15.12:0/1311065216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0bc0138040 msgr2=0x7f0bc0125300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bbe59c640 1 --2- 172.21.15.12:0/1311065216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0bc0138040 0x7f0bc0125300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bbe59c640 10 monclient: _finish_hunting 0 2024-09-11T01:55:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bbe59c640 1 monclient: found mon.noname-a 2024-09-11T01:55:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bbe59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bbe59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:55:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.878+0000 7f0bbe59c640 1 -- 172.21.15.12:0/1311065216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bc0128320 con 0x7f0bc00a2e90 2024-09-11T01:55:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.880+0000 7f0bbe59c640 1 -- 172.21.15.12:0/1311065216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bc00a2e90 msgr2=0x7f0bc0124880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.880+0000 7f0bbe59c640 1 --2- 172.21.15.12:0/1311065216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bc00a2e90 0x7f0bc0124880 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0bb000b580 tx=0x7f0bb000ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.881+0000 7f0bb6ffd640 1 -- 172.21.15.12:0/1311065216 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bb00050f0 con 0x7f0bc00a2e90 2024-09-11T01:55:36.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.881+0000 7f0bb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:36.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.881+0000 7f0bb6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:55:36.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.881+0000 7f0bb6ffd640 10 monclient: dump: 2024-09-11T01:55:36.893 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:36.893 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:36.893 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:36.893 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:36.893 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:36.893 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:36.893 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:36.893 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:36.893 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:36.893 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:36.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.881+0000 7f0bb6ffd640 10 monclient: _finish_auth 0 2024-09-11T01:55:36.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.881+0000 7f0bb6ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:55:36.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.881+0000 7f0bb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:36.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.881+0000 7f0bb6ffd640 1 -- 172.21.15.12:0/1311065216 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bb0015070 con 0x7f0bc00a2e90 2024-09-11T01:55:36.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.881+0000 7f0bc4a85640 5 monclient: authenticate success, global_id 294168 2024-09-11T01:55:36.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.881+0000 7f0bc4a85640 10 monclient: _renew_subs 2024-09-11T01:55:36.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.881+0000 7f0bc4a85640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:36.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.881+0000 7f0bc4a85640 1 -- 172.21.15.12:0/1311065216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bc0128600 con 0x7f0bc00a2e90 2024-09-11T01:55:36.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.881+0000 7f0bb6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:36.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.881+0000 7f0bb6ffd640 1 -- 172.21.15.12:0/1311065216 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bb00054d0 con 0x7f0bc00a2e90 2024-09-11T01:55:36.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.881+0000 7f0bb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:36.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.881+0000 7f0bb6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:55:36.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.881+0000 7f0bb6ffd640 10 monclient: dump: 2024-09-11T01:55:36.894 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:36.894 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:36.895 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:36.895 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:36.895 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:36.895 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:36.895 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:36.895 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:36.895 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:36.895 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:36.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.881+0000 7f0bc4a85640 10 monclient: _renew_subs 2024-09-11T01:55:36.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.881+0000 7f0bbcd99640 4 set_mon_vals no callback set 2024-09-11T01:55:36.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.881+0000 7f0bc4a85640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:36.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.881+0000 7f0bc4a85640 1 -- 172.21.15.12:0/1311065216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bc0128040 con 0x7f0bc00a2e90 2024-09-11T01:55:36.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.882+0000 7f0bb6ffd640 1 -- 172.21.15.12:0/1311065216 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f0bb0005670 con 0x7f0bc00a2e90 2024-09-11T01:55:36.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.882+0000 7f0bc4a85640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:36.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.882+0000 7f0bc4a85640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:36.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.882+0000 7f0bc4a85640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:36.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.882+0000 7f0bc4a85640 1 -- 172.21.15.12:0/1311065216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bc00a3650 con 0x7f0bc00a2e90 2024-09-11T01:55:36.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.882+0000 7f0bb6ffd640 1 --2- 172.21.15.12:0/1311065216 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0ba003e210 0x7f0ba00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:36.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.882+0000 7f0bb6ffd640 1 -- 172.21.15.12:0/1311065216 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(889..889 src has 676..889) ==== 4803+0+0 (secure 0 0 0) 0x7f0bb0052cd0 con 0x7f0bc00a2e90 2024-09-11T01:55:36.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.882+0000 7f0bbdd9b640 1 --2- 172.21.15.12:0/1311065216 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0ba003e210 0x7f0ba00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:36.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.882+0000 7f0bbdd9b640 10 monclient: get_auth_request con 0x7f0ba003e210 auth_method 0 2024-09-11T01:55:36.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.885+0000 7f0bbdd9b640 1 --2- 172.21.15.12:0/1311065216 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0ba003e210 0x7f0ba00406d0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f0bc00a92c0 tx=0x7f0ba802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:36.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.887+0000 7f0bb6ffd640 1 -- 172.21.15.12:0/1311065216 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0bb000ee20 con 0x7f0bc00a2e90 2024-09-11T01:55:36.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.887+0000 7f0bb6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:36.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:36.887+0000 7f0bb6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:55:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:37.143+0000 7f0bc4a85640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:55:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:37.143+0000 7f0bc4a85640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:55:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:37.143+0000 7f0bc4a85640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:37.143+0000 7f0bc4a85640 1 -- 172.21.15.12:0/1311065216 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0bc00737c0 con 0x7f0bc00a2e90 2024-09-11T01:55:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:37.143+0000 7f0bb6ffd640 1 -- 172.21.15.12:0/1311065216 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7f0bb0010e20 con 0x7f0bc00a2e90 2024-09-11T01:55:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:37.143+0000 7f0bb6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:55:37.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:37.143+0000 7f0bb6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:55:37.146 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":240,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:55:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:37.146+0000 7f0bc4a85640 1 -- 172.21.15.12:0/1311065216 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0ba003e210 msgr2=0x7f0ba00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:37.146+0000 7f0bc4a85640 1 --2- 172.21.15.12:0/1311065216 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0ba003e210 0x7f0ba00406d0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f0bc00a92c0 tx=0x7f0ba802e000 comp rx=0 tx=0).stop 2024-09-11T01:55:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:37.146+0000 7f0bc4a85640 10 monclient: shutdown 2024-09-11T01:55:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:37.146+0000 7f0bc4a85640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:37.146+0000 7f0bc4a85640 1 -- 172.21.15.12:0/1311065216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bc00a2e90 msgr2=0x7f0bc0124880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:37.146+0000 7f0bc4a85640 1 --2- 172.21.15.12:0/1311065216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bc00a2e90 0x7f0bc0124880 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0bb000b580 tx=0x7f0bb000ba40 comp rx=0 tx=0).stop 2024-09-11T01:55:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:37.146+0000 7f0bc4a85640 1 -- 172.21.15.12:0/1311065216 shutdown_connections 2024-09-11T01:55:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:37.146+0000 7f0bc4a85640 1 --2- 172.21.15.12:0/1311065216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0bc0138040 0x7f0bc0125300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:37.146+0000 7f0bc4a85640 1 --2- 172.21.15.12:0/1311065216 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0bc00a3270 0x7f0bc0124dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:37.146+0000 7f0bc4a85640 1 --2- 172.21.15.12:0/1311065216 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0ba003e210 0x7f0ba00406d0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:37.146+0000 7f0bc4a85640 1 --2- 172.21.15.12:0/1311065216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0bc00a2e90 0x7f0bc0124880 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:37.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:37.146+0000 7f0bc4a85640 1 -- 172.21.15.12:0/1311065216 >> 172.21.15.12:0/1311065216 conn(0x7f0bc00a8490 msgr2=0x7f0bc0072090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:37.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:37.146+0000 7f0bc4a85640 1 -- 172.21.15.12:0/1311065216 shutdown_connections 2024-09-11T01:55:37.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:37.146+0000 7f0bc4a85640 1 -- 172.21.15.12:0/1311065216 wait complete. 2024-09-11T01:55:40.159 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:55:40.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.270+0000 7f0b79a65640 10 monclient: get_monmap_and_config 2024-09-11T01:55:40.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.270+0000 7f0b79a65640 10 monclient: build_initial_monmap 2024-09-11T01:55:40.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.270+0000 7f0b79a65640 10 monclient: monmap: 2024-09-11T01:55:40.272 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:40.272 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:40.272 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:55:40.271076+0000 2024-09-11T01:55:40.272 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:55:40.271076+0000 2024-09-11T01:55:40.272 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:40.272 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:40.272 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:55:40.272 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:55:40.273 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:55:40.273 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:55:40.273 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:55:40.273 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:40.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.271+0000 7f0b79a65640 1 Processor -- start 2024-09-11T01:55:40.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.271+0000 7f0b79a65640 1 -- start start 2024-09-11T01:55:40.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.271+0000 7f0b79a65640 10 monclient: init 2024-09-11T01:55:40.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.271+0000 7f0b79a65640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:40.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.271+0000 7f0b79a65640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-11T01:55:40.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.271+0000 7f0b79a65640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0b740a2e90 0x7f0b7406dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:40.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.271+0000 7f0b79a65640 10 monclient(hunting): picked mon.noname-b con 0x7f0b740a2e90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:40.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.271+0000 7f0b79a65640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0b740a3270 addr v1:172.21.15.71:6789/0 2024-09-11T01:55:40.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.271+0000 7f0b79a65640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0b7413d490 0x7f0b7413f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:40.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.271+0000 7f0b79a65640 10 monclient(hunting): picked mon.noname-c con 0x7f0b7413d490 addr v2:172.21.15.71:3300/0 2024-09-11T01:55:40.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.271+0000 7f0b79a65640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b740a46c0 con 0x7f0b740a3270 2024-09-11T01:55:40.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.271+0000 7f0b79a65640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:40.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.271+0000 7f0b79a65640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f0b740a4900 con 0x7f0b7413d490 2024-09-11T01:55:40.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.271+0000 7f0b79a65640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:40.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.271+0000 7f0b79a65640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f0b7406ee50 con 0x7f0b740a2e90 2024-09-11T01:55:40.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.271+0000 7f0b79a65640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:40.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.271+0000 7f0b737fe640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0b740a2e90 msgr2=0x7f0b7406dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:55:40.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.271+0000 7f0b79a65640 10 monclient(hunting): authenticate will time out at 2346.713801s 2024-09-11T01:55:40.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.271+0000 7f0b737fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0b740a2e90 0x7f0b7406dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:55:40.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.272+0000 7f0b73fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0b7413d490 0x7f0b7413f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:40.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.272+0000 7f0b72ffd640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f0b740a3270 0x7f0b740a3650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.12:47968/0 (socket says 172.21.15.12:47968) 2024-09-11T01:55:40.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.272+0000 7f0b72ffd640 1 -- 172.21.15.12:0/1904928984 learned_addr learned my addr 172.21.15.12:0/1904928984 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:55:40.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.272+0000 7f0b73fff640 10 monclient(hunting): get_auth_request con 0x7f0b7413d490 auth_method 0 2024-09-11T01:55:40.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.272+0000 7f0b73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:40.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.272+0000 7f0b73fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:40.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.272+0000 7f0b73fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:40.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.272+0000 7f0b73fff640 1 -- 172.21.15.12:0/1904928984 >> v2:172.21.15.71:3300/0 conn(0x7f0b7413d490 msgr2=0x7f0b7413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:55:40.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.272+0000 7f0b73fff640 1 -- 172.21.15.12:0/1904928984 >> v2:172.21.15.71:3300/0 conn(0x7f0b7413d490 msgr2=0x7f0b7413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:55:40.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.272+0000 7f0b73fff640 1 --2- 172.21.15.12:0/1904928984 >> v2:172.21.15.71:3300/0 conn(0x7f0b7413d490 0x7f0b7413f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:55:40.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.272+0000 7f0b73fff640 1 --2- 172.21.15.12:0/1904928984 >> v2:172.21.15.71:3300/0 conn(0x7f0b7413d490 0x7f0b7413f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:55:40.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.274+0000 7f0b72ffd640 1 -- 172.21.15.12:0/1904928984 >> v1:172.21.15.71:6789/0 conn(0x7f0b740a3270 legacy=0x7f0b740a3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:40.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.472+0000 7f0b737fe640 1 -- 172.21.15.12:0/1904928984 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0b740a2e90 msgr2=0x7f0b7406dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:55:40.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.472+0000 7f0b737fe640 1 --2- 172.21.15.12:0/1904928984 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0b740a2e90 0x7f0b7406dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:55:40.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.474+0000 7f0b73fff640 1 --2- 172.21.15.12:0/1904928984 >> v2:172.21.15.71:3300/0 conn(0x7f0b7413d490 0x7f0b7413f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:40.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.474+0000 7f0b73fff640 10 monclient(hunting): get_auth_request con 0x7f0b7413d490 auth_method 0 2024-09-11T01:55:40.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.474+0000 7f0b73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:40.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.474+0000 7f0b73fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:40.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.474+0000 7f0b73fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:55:40.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.474+0000 7f0b73fff640 1 -- 172.21.15.12:0/1904928984 >> v2:172.21.15.71:3300/0 conn(0x7f0b7413d490 msgr2=0x7f0b7413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:55:40.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.474+0000 7f0b73fff640 1 -- 172.21.15.12:0/1904928984 >> v2:172.21.15.71:3300/0 conn(0x7f0b7413d490 msgr2=0x7f0b7413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:55:40.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.474+0000 7f0b73fff640 1 --2- 172.21.15.12:0/1904928984 >> v2:172.21.15.71:3300/0 conn(0x7f0b7413d490 0x7f0b7413f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:55:40.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.474+0000 7f0b73fff640 1 --2- 172.21.15.12:0/1904928984 >> v2:172.21.15.71:3300/0 conn(0x7f0b7413d490 0x7f0b7413f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:55:40.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.873+0000 7f0b737fe640 1 -- 172.21.15.12:0/1904928984 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0b740a2e90 msgr2=0x7f0b7406dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:55:40.875 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.873+0000 7f0b737fe640 1 --2- 172.21.15.12:0/1904928984 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0b740a2e90 0x7f0b7406dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:55:40.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.876+0000 7f0b73fff640 1 --2- 172.21.15.12:0/1904928984 >> v2:172.21.15.71:3300/0 conn(0x7f0b7413d490 0x7f0b7413f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:40.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.876+0000 7f0b73fff640 10 monclient(hunting): get_auth_request con 0x7f0b7413d490 auth_method 0 2024-09-11T01:55:40.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.876+0000 7f0b73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:40.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.876+0000 7f0b73fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:40.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.876+0000 7f0b73fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:55:40.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.876+0000 7f0b73fff640 1 -- 172.21.15.12:0/1904928984 >> v2:172.21.15.71:3300/0 conn(0x7f0b7413d490 msgr2=0x7f0b7413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:55:40.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.876+0000 7f0b73fff640 1 -- 172.21.15.12:0/1904928984 >> v2:172.21.15.71:3300/0 conn(0x7f0b7413d490 msgr2=0x7f0b7413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:55:40.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.876+0000 7f0b73fff640 1 --2- 172.21.15.12:0/1904928984 >> v2:172.21.15.71:3300/0 conn(0x7f0b7413d490 0x7f0b7413f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:55:40.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:40.876+0000 7f0b73fff640 1 --2- 172.21.15.12:0/1904928984 >> v2:172.21.15.71:3300/0 conn(0x7f0b7413d490 0x7f0b7413f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:55:41.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:41.675+0000 7f0b737fe640 1 -- 172.21.15.12:0/1904928984 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0b740a2e90 msgr2=0x7f0b7406dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:55:41.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:41.675+0000 7f0b737fe640 1 --2- 172.21.15.12:0/1904928984 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0b740a2e90 0x7f0b7406dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:55:41.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:41.677+0000 7f0b73fff640 1 --2- 172.21.15.12:0/1904928984 >> v2:172.21.15.71:3300/0 conn(0x7f0b7413d490 0x7f0b7413f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:41.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:41.678+0000 7f0b73fff640 10 monclient(hunting): get_auth_request con 0x7f0b7413d490 auth_method 0 2024-09-11T01:55:41.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:41.678+0000 7f0b73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:41.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:41.678+0000 7f0b73fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:41.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:41.678+0000 7f0b73fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:55:41.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:41.678+0000 7f0b73fff640 1 -- 172.21.15.12:0/1904928984 >> v2:172.21.15.71:3300/0 conn(0x7f0b7413d490 msgr2=0x7f0b7413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:55:41.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:41.678+0000 7f0b73fff640 1 -- 172.21.15.12:0/1904928984 >> v2:172.21.15.71:3300/0 conn(0x7f0b7413d490 msgr2=0x7f0b7413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:55:41.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:41.678+0000 7f0b73fff640 1 --2- 172.21.15.12:0/1904928984 >> v2:172.21.15.71:3300/0 conn(0x7f0b7413d490 0x7f0b7413f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:55:41.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:41.678+0000 7f0b73fff640 1 --2- 172.21.15.12:0/1904928984 >> v2:172.21.15.71:3300/0 conn(0x7f0b7413d490 0x7f0b7413f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:55:43.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.272+0000 7f0b717fa640 10 monclient(hunting): tick 2024-09-11T01:55:43.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.272+0000 7f0b717fa640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:55:43.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.272+0000 7f0b717fa640 1 monclient(hunting): continuing hunt 2024-09-11T01:55:43.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.272+0000 7f0b717fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:55:43.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.272+0000 7f0b717fa640 1 -- 172.21.15.12:0/1904928984 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0b740a2e90 msgr2=0x7f0b7406dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:55:43.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.272+0000 7f0b717fa640 1 --2- 172.21.15.12:0/1904928984 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0b740a2e90 0x7f0b7406dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:43.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.272+0000 7f0b717fa640 1 -- 172.21.15.12:0/1904928984 >> v2:172.21.15.71:3300/0 conn(0x7f0b7413d490 msgr2=0x7f0b7413f880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:55:43.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.272+0000 7f0b717fa640 1 --2- 172.21.15.12:0/1904928984 >> v2:172.21.15.71:3300/0 conn(0x7f0b7413d490 0x7f0b7413f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:43.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.272+0000 7f0b717fa640 1 -- 172.21.15.12:0/1904928984 >> v1:172.21.15.71:6789/0 conn(0x7f0b740a3270 legacy=0x7f0b740a3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:43.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.272+0000 7f0b717fa640 10 monclient: _add_conns ranks=[0,3] 2024-09-11T01:55:43.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.272+0000 7f0b717fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0b60003760 addr v1:172.21.15.12:6789/0 2024-09-11T01:55:43.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.272+0000 7f0b717fa640 1 --2- 172.21.15.12:0/1904928984 >> v2:172.21.15.12:3300/0 conn(0x7f0b60007490 0x7f0b60009930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:43.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.272+0000 7f0b717fa640 10 monclient(hunting): picked mon.noname-a con 0x7f0b60007490 addr v2:172.21.15.12:3300/0 2024-09-11T01:55:43.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.272+0000 7f0b717fa640 1 -- 172.21.15.12:0/1904928984 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b7406eb70 con 0x7f0b60003760 2024-09-11T01:55:43.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.272+0000 7f0b717fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:43.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.272+0000 7f0b717fa640 1 -- 172.21.15.12:0/1904928984 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f0b740a4900 con 0x7f0b60007490 2024-09-11T01:55:43.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.272+0000 7f0b717fa640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:43.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.272+0000 7f0b72ffd640 1 --2- 172.21.15.12:0/1904928984 >> v2:172.21.15.12:3300/0 conn(0x7f0b60007490 0x7f0b60009930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.272+0000 7f0b72ffd640 10 monclient(hunting): get_auth_request con 0x7f0b60007490 auth_method 0 2024-09-11T01:55:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.272+0000 7f0b72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.272+0000 7f0b72ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.272+0000 7f0b72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.272+0000 7f0b72ffd640 1 -- 172.21.15.12:0/1904928984 >> v2:172.21.15.12:3300/0 conn(0x7f0b60007490 msgr2=0x7f0b60009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:55:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.272+0000 7f0b72ffd640 1 -- 172.21.15.12:0/1904928984 >> v2:172.21.15.12:3300/0 conn(0x7f0b60007490 msgr2=0x7f0b60009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:55:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.272+0000 7f0b72ffd640 1 --2- 172.21.15.12:0/1904928984 >> v2:172.21.15.12:3300/0 conn(0x7f0b60007490 0x7f0b60009930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:55:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.272+0000 7f0b72ffd640 1 --2- 172.21.15.12:0/1904928984 >> v2:172.21.15.12:3300/0 conn(0x7f0b60007490 0x7f0b60009930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:55:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.274+0000 7f0b737fe640 1 -- 172.21.15.12:0/1904928984 >> v1:172.21.15.12:6789/0 conn(0x7f0b60003760 legacy=0x7f0b60005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:43.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.473+0000 7f0b72ffd640 1 --2- 172.21.15.12:0/1904928984 >> v2:172.21.15.12:3300/0 conn(0x7f0b60007490 0x7f0b60009930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:43.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.473+0000 7f0b72ffd640 10 monclient(hunting): get_auth_request con 0x7f0b60007490 auth_method 0 2024-09-11T01:55:43.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.473+0000 7f0b72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:43.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.473+0000 7f0b72ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:43.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.473+0000 7f0b72ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:55:43.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.473+0000 7f0b72ffd640 1 -- 172.21.15.12:0/1904928984 >> v2:172.21.15.12:3300/0 conn(0x7f0b60007490 msgr2=0x7f0b60009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:55:43.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.473+0000 7f0b72ffd640 1 -- 172.21.15.12:0/1904928984 >> v2:172.21.15.12:3300/0 conn(0x7f0b60007490 msgr2=0x7f0b60009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:55:43.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.473+0000 7f0b72ffd640 1 --2- 172.21.15.12:0/1904928984 >> v2:172.21.15.12:3300/0 conn(0x7f0b60007490 0x7f0b60009930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:55:43.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.473+0000 7f0b72ffd640 1 --2- 172.21.15.12:0/1904928984 >> v2:172.21.15.12:3300/0 conn(0x7f0b60007490 0x7f0b60009930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:55:43.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.875+0000 7f0b72ffd640 1 --2- 172.21.15.12:0/1904928984 >> v2:172.21.15.12:3300/0 conn(0x7f0b60007490 0x7f0b60009930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:43.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.875+0000 7f0b72ffd640 10 monclient(hunting): get_auth_request con 0x7f0b60007490 auth_method 0 2024-09-11T01:55:43.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.875+0000 7f0b72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:43.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.875+0000 7f0b72ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:43.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.875+0000 7f0b72ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:55:43.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.875+0000 7f0b72ffd640 1 -- 172.21.15.12:0/1904928984 >> v2:172.21.15.12:3300/0 conn(0x7f0b60007490 msgr2=0x7f0b60009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:55:43.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.875+0000 7f0b72ffd640 1 -- 172.21.15.12:0/1904928984 >> v2:172.21.15.12:3300/0 conn(0x7f0b60007490 msgr2=0x7f0b60009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:55:43.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.875+0000 7f0b72ffd640 1 --2- 172.21.15.12:0/1904928984 >> v2:172.21.15.12:3300/0 conn(0x7f0b60007490 0x7f0b60009930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:55:43.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:43.875+0000 7f0b72ffd640 1 --2- 172.21.15.12:0/1904928984 >> v2:172.21.15.12:3300/0 conn(0x7f0b60007490 0x7f0b60009930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:55:44.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.346+0000 7f0b727fc640 1 -- 172.21.15.12:0/1904928984 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3076749161 0 0) 0x7f0b7406eb70 con 0x7f0b60003760 2024-09-11T01:55:44.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.346+0000 7f0b727fc640 10 monclient(hunting): _negotiate 2024-09-11T01:55:44.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.346+0000 7f0b727fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:44.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.346+0000 7f0b727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:44.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.346+0000 7f0b727fc640 10 monclient(hunting): my global_id is 294171 2024-09-11T01:55:44.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.346+0000 7f0b727fc640 1 -- 172.21.15.12:0/1904928984 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b5c003520 con 0x7f0b60003760 2024-09-11T01:55:44.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 1 -- 172.21.15.12:0/1904928984 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 39913565 0 0) 0x7f0b5c003520 con 0x7f0b60003760 2024-09-11T01:55:44.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 1 -- 172.21.15.12:0/1904928984 >> v2:172.21.15.12:3300/0 conn(0x7f0b60007490 msgr2=0x7f0b60009930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:55:44.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 1 --2- 172.21.15.12:0/1904928984 >> v2:172.21.15.12:3300/0 conn(0x7f0b60007490 0x7f0b60009930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 10 monclient: _finish_hunting 0 2024-09-11T01:55:44.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:55:44.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:44.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:55:44.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 1 -- 172.21.15.12:0/1904928984 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b6000b050 con 0x7f0b60003760 2024-09-11T01:55:44.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 10 monclient: _finish_auth 0 2024-09-11T01:55:44.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 10 monclient: _check_auth_tickets 2024-09-11T01:55:44.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:44.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 1 -- 172.21.15.12:0/1904928984 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7f0b64004600 con 0x7f0b60003760 2024-09-11T01:55:44.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:44.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:55:44.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 10 monclient: dump: 2024-09-11T01:55:44.353 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:44.353 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:44.353 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.354 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.354 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:44.354 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:44.354 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:44.354 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:44.354 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:44.354 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:44.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:55:44.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:44.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 1 -- 172.21.15.12:0/1904928984 >> v1:172.21.15.12:6789/0 conn(0x7f0b60003760 legacy=0x7f0b60005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:44.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b73fff640 1 -- 172.21.15.12:0/1904928984 reap_dead start 2024-09-11T01:55:44.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:55:44.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 1 --2- 172.21.15.12:0/1904928984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b5c0039b0 0x7f0b5c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:44.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 10 monclient(hunting): picked mon.a con 0x7f0b5c0039b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:55:44.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 1 --2- 172.21.15.12:0/1904928984 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b5c004800 0x7f0b5c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:44.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 10 monclient(hunting): picked mon.b con 0x7f0b5c004800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:55:44.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 1 --2- 172.21.15.12:0/1904928984 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0b5c005290 0x7f0b5c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:44.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 10 monclient(hunting): picked mon.c con 0x7f0b5c005290 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:44.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:44.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 1 -- 172.21.15.12:0/1904928984 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0b740a4900 con 0x7f0b5c0039b0 2024-09-11T01:55:44.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:44.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 1 -- 172.21.15.12:0/1904928984 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0b5c00c1e0 con 0x7f0b5c004800 2024-09-11T01:55:44.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:44.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 1 -- 172.21.15.12:0/1904928984 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f0b5c00c3e0 con 0x7f0b5c005290 2024-09-11T01:55:44.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:44.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.348+0000 7f0b727fc640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:55:44.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b73fff640 1 -- 172.21.15.12:0/1904928984 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0b5c005290 msgr2=0x7f0b5c00b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:55:44.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b73fff640 1 --2- 172.21.15.12:0/1904928984 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0b5c005290 0x7f0b5c00b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:55:44.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b737fe640 1 --2- 172.21.15.12:0/1904928984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b5c0039b0 0x7f0b5c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:44.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b737fe640 10 monclient(hunting): get_auth_request con 0x7f0b5c0039b0 auth_method 0 2024-09-11T01:55:44.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:44.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b737fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:44.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b737fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:55:44.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b72ffd640 1 --2- 172.21.15.12:0/1904928984 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b5c004800 0x7f0b5c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:44.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b72ffd640 10 monclient(hunting): get_auth_request con 0x7f0b5c004800 auth_method 0 2024-09-11T01:55:44.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:44.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b72ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:44.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b72ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:55:44.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:44.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:44.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b737fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:55:44.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:44.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:44.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:55:44.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b737fe640 10 monclient(hunting): handle_auth_done global_id 294171 payload 293 2024-09-11T01:55:44.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b737fe640 1 -- 172.21.15.12:0/1904928984 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0b5c005290 msgr2=0x7f0b5c00b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:55:44.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b737fe640 1 --2- 172.21.15.12:0/1904928984 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0b5c005290 0x7f0b5c00b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b737fe640 1 -- 172.21.15.12:0/1904928984 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b5c004800 msgr2=0x7f0b5c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:55:44.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b737fe640 1 --2- 172.21.15.12:0/1904928984 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b5c004800 0x7f0b5c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b737fe640 10 monclient: _finish_hunting 0 2024-09-11T01:55:44.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b737fe640 1 monclient: found mon.a 2024-09-11T01:55:44.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:44.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b737fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:44.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b737fe640 1 -- 172.21.15.12:0/1904928984 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b5c003520 con 0x7f0b5c0039b0 2024-09-11T01:55:44.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b737fe640 10 monclient: _finish_auth 0 2024-09-11T01:55:44.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b737fe640 10 monclient: _check_auth_tickets 2024-09-11T01:55:44.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:44.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b79a65640 5 monclient: authenticate success, global_id 294171 2024-09-11T01:55:44.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.349+0000 7f0b79a65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:55:44.360 INFO:tasks.rados.rados.0.smithi071.stdout:3890: done (0 left) 2024-09-11T01:55:44.360 INFO:tasks.rados.rados.0.smithi071.stdout:3891: read oid 26 snap -1 2024-09-11T01:55:44.360 INFO:tasks.rados.rados.0.smithi071.stdout:3891: expect (ObjNum 1327 snap 432 seq_num 1327) 2024-09-11T01:55:44.360 INFO:tasks.rados.rados.0.smithi071.stdout:3892: snap_remove snap 413 2024-09-11T01:55:44.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.352+0000 7f0b737fe640 1 -- 172.21.15.12:0/1904928984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b5c0039b0 msgr2=0x7f0b5c003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:44.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.352+0000 7f0b737fe640 1 --2- 172.21.15.12:0/1904928984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b5c0039b0 0x7f0b5c003db0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0b64004820 tx=0x7f0b5c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f8203e8a397cb83 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:44.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.354+0000 7f0b727fc640 1 -- 172.21.15.12:0/1904928984 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6400c200 con 0x7f0b5c0039b0 2024-09-11T01:55:44.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.354+0000 7f0b727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:44.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.354+0000 7f0b727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:55:44.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.354+0000 7f0b727fc640 10 monclient: dump: 2024-09-11T01:55:44.363 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:44.363 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:44.363 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.364 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.364 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:44.364 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:44.364 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:44.364 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:44.364 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:44.364 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:44.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.354+0000 7f0b727fc640 1 -- 172.21.15.12:0/1904928984 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b64004b90 con 0x7f0b5c0039b0 2024-09-11T01:55:44.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.354+0000 7f0b727fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:44.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.354+0000 7f0b727fc640 1 -- 172.21.15.12:0/1904928984 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b640060d0 con 0x7f0b5c0039b0 2024-09-11T01:55:44.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.354+0000 7f0b727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:44.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.354+0000 7f0b727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:55:44.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.354+0000 7f0b727fc640 10 monclient: dump: 2024-09-11T01:55:44.365 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:44.365 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:44.365 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.366 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.366 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:44.366 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:44.366 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:44.366 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:44.367 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:44.367 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:44.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.354+0000 7f0b79a65640 10 monclient: get_monmap_and_config success 2024-09-11T01:55:44.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.354+0000 7f0b79a65640 4 set_mon_vals no callback set 2024-09-11T01:55:44.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.355+0000 7f0b79a65640 10 monclient: shutdown 2024-09-11T01:55:44.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.355+0000 7f0b79a65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:44.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.355+0000 7f0b79a65640 1 -- 172.21.15.12:0/1904928984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b5c0039b0 msgr2=0x7f0b5c003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:44.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.355+0000 7f0b79a65640 1 --2- 172.21.15.12:0/1904928984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b5c0039b0 0x7f0b5c003db0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0b64004820 tx=0x7f0b5c005830 comp rx=0 tx=0).stop 2024-09-11T01:55:44.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.356+0000 7f0b79a65640 1 -- 172.21.15.12:0/1904928984 shutdown_connections 2024-09-11T01:55:44.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.356+0000 7f0b79a65640 1 --2- 172.21.15.12:0/1904928984 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0b5c005290 0x7f0b5c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.356+0000 7f0b79a65640 1 --2- 172.21.15.12:0/1904928984 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b5c004800 0x7f0b5c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.356+0000 7f0b79a65640 1 --2- 172.21.15.12:0/1904928984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b5c0039b0 0x7f0b5c003db0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.356+0000 7f0b79a65640 1 -- 172.21.15.12:0/1904928984 >> 172.21.15.12:0/1904928984 conn(0x7f0b740a8490 msgr2=0x7f0b740a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:44.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.356+0000 7f0b79a65640 1 -- 172.21.15.12:0/1904928984 shutdown_connections 2024-09-11T01:55:44.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.356+0000 7f0b79a65640 1 -- 172.21.15.12:0/1904928984 wait complete. 2024-09-11T01:55:44.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.357+0000 7f0b79a65640 10 monclient: build_initial_monmap 2024-09-11T01:55:44.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.357+0000 7f0b79a65640 10 monclient: monmap: 2024-09-11T01:55:44.369 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:44.369 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:55:44.369 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:55:44.370 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:55:44.370 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:44.370 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:44.370 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:55:44.370 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:55:44.370 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:55:44.370 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:44.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.358+0000 7f0b79a65640 1 Processor -- start 2024-09-11T01:55:44.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.358+0000 7f0b79a65640 1 -- start start 2024-09-11T01:55:44.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.358+0000 7f0b79a65640 10 monclient: init 2024-09-11T01:55:44.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.359+0000 7f0b79a65640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:44.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.359+0000 7f0b79a65640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:55:44.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.359+0000 7f0b79a65640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b5c0039b0 0x7f0b7411fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:44.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.359+0000 7f0b79a65640 10 monclient(hunting): picked mon.noname-a con 0x7f0b5c0039b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:55:44.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.359+0000 7f0b79a65640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0b5c004800 0x7f0b7413b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:44.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.359+0000 7f0b79a65640 10 monclient(hunting): picked mon.noname-c con 0x7f0b5c004800 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:44.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.359+0000 7f0b79a65640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b5c005290 0x7f0b7413ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:44.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.359+0000 7f0b72ffd640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0b5c004800 msgr2=0x7f0b7413b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:55:44.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.359+0000 7f0b72ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0b5c004800 0x7f0b7413b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:55:44.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.359+0000 7f0b79a65640 10 monclient(hunting): picked mon.noname-b con 0x7f0b5c005290 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:55:44.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.359+0000 7f0b79a65640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:44.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.359+0000 7f0b79a65640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7406dde0 con 0x7f0b5c0039b0 2024-09-11T01:55:44.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.359+0000 7f0b79a65640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:44.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.359+0000 7f0b79a65640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7406dc60 con 0x7f0b5c005290 2024-09-11T01:55:44.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.359+0000 7f0b79a65640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:44.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.359+0000 7f0b79a65640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f0b7406df60 con 0x7f0b5c004800 2024-09-11T01:55:44.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.359+0000 7f0b79a65640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:44.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.359+0000 7f0b79a65640 10 monclient(hunting): authenticate will time out at 2350.801853s 2024-09-11T01:55:44.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.359+0000 7f0b73fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b5c005290 0x7f0b7413ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:44.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.360+0000 7f0b73fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b5c005290 0x7f0b7413ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:34750/0 (socket says 172.21.15.12:34750) 2024-09-11T01:55:44.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.360+0000 7f0b73fff640 1 -- 172.21.15.12:0/2963307237 learned_addr learned my addr 172.21.15.12:0/2963307237 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:55:44.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.360+0000 7f0b73fff640 10 monclient(hunting): get_auth_request con 0x7f0b5c005290 auth_method 0 2024-09-11T01:55:44.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.360+0000 7f0b73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:44.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.360+0000 7f0b73fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:44.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.360+0000 7f0b73fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:44.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.360+0000 7f0b73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:44.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.360+0000 7f0b73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:44.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.360+0000 7f0b73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:44.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.360+0000 7f0b73fff640 10 monclient(hunting): handle_auth_done global_id 274315 payload 824 2024-09-11T01:55:44.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.360+0000 7f0b73fff640 1 -- 172.21.15.12:0/2963307237 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0b5c004800 msgr2=0x7f0b7413b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:55:44.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.360+0000 7f0b73fff640 1 --2- 172.21.15.12:0/2963307237 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0b5c004800 0x7f0b7413b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.360+0000 7f0b73fff640 1 -- 172.21.15.12:0/2963307237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b5c0039b0 msgr2=0x7f0b7411fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:44.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.360+0000 7f0b73fff640 1 --2- 172.21.15.12:0/2963307237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b5c0039b0 0x7f0b7411fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.360+0000 7f0b73fff640 10 monclient: _finish_hunting 0 2024-09-11T01:55:44.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.360+0000 7f0b73fff640 1 monclient: found mon.noname-b 2024-09-11T01:55:44.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.360+0000 7f0b73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:44.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.360+0000 7f0b73fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-09-11T01:55:44.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.360+0000 7f0b73fff640 1 -- 172.21.15.12:0/2963307237 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b7409f680 con 0x7f0b5c005290 2024-09-11T01:55:44.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.363+0000 7f0b73fff640 1 -- 172.21.15.12:0/2963307237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b5c005290 msgr2=0x7f0b7413ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:44.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.363+0000 7f0b73fff640 1 --2- 172.21.15.12:0/2963307237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b5c005290 0x7f0b7413ba30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0b60007400 tx=0x7f0b6c00ad60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:44.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.363+0000 7f0b5bfff640 1 -- 172.21.15.12:0/2963307237 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c01c020 con 0x7f0b5c005290 2024-09-11T01:55:44.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.363+0000 7f0b5bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:44.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.363+0000 7f0b5bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:55:44.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.363+0000 7f0b5bfff640 10 monclient: dump: 2024-09-11T01:55:44.377 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:44.377 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:44.377 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.377 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.377 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:44.377 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:44.377 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:44.378 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:44.378 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:44.378 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:44.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.363+0000 7f0b5bfff640 10 monclient: _finish_auth 0 2024-09-11T01:55:44.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.363+0000 7f0b5bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:55:44.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.363+0000 7f0b5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:44.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.363+0000 7f0b79a65640 5 monclient: authenticate success, global_id 274315 2024-09-11T01:55:44.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.363+0000 7f0b79a65640 10 monclient: _renew_subs 2024-09-11T01:55:44.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.363+0000 7f0b5bfff640 1 -- 172.21.15.12:0/2963307237 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b6c012db0 con 0x7f0b5c005290 2024-09-11T01:55:44.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.363+0000 7f0b79a65640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:55:44.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.363+0000 7f0b79a65640 1 -- 172.21.15.12:0/2963307237 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b741203a0 con 0x7f0b5c005290 2024-09-11T01:55:44.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.363+0000 7f0b5bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:44.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.363+0000 7f0b5bfff640 1 -- 172.21.15.12:0/2963307237 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c018690 con 0x7f0b5c005290 2024-09-11T01:55:44.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.364+0000 7f0b79a65640 10 monclient: _renew_subs 2024-09-11T01:55:44.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.364+0000 7f0b79a65640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:55:44.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.364+0000 7f0b79a65640 1 -- 172.21.15.12:0/2963307237 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b7411f160 con 0x7f0b5c005290 2024-09-11T01:55:44.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.364+0000 7f0b5bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:44.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.364+0000 7f0b5bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:55:44.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.364+0000 7f0b5bfff640 10 monclient: dump: 2024-09-11T01:55:44.379 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:44.379 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:44.379 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.379 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.379 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:44.379 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:44.379 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:44.379 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:44.379 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:44.380 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:44.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.363+0000 7f0b71ffb640 4 set_mon_vals no callback set 2024-09-11T01:55:44.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.365+0000 7f0b5bfff640 1 -- 172.21.15.12:0/2963307237 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f0b6c029060 con 0x7f0b5c005290 2024-09-11T01:55:44.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.365+0000 7f0b79a65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:44.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.365+0000 7f0b79a65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:44.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.365+0000 7f0b79a65640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:55:44.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.365+0000 7f0b79a65640 1 -- 172.21.15.12:0/2963307237 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b740a8040 con 0x7f0b5c005290 2024-09-11T01:55:44.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.366+0000 7f0b5bfff640 1 --2- 172.21.15.12:0/2963307237 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0b6003eae0 0x7f0b60040fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:44.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.366+0000 7f0b5bfff640 1 -- 172.21.15.12:0/2963307237 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(890..890 src has 676..890) ==== 4875+0+0 (secure 0 0 0) 0x7f0b6c053e70 con 0x7f0b5c005290 2024-09-11T01:55:44.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.366+0000 7f0b737fe640 1 --2- 172.21.15.12:0/2963307237 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0b6003eae0 0x7f0b60040fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:44.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.366+0000 7f0b737fe640 10 monclient: get_auth_request con 0x7f0b6003eae0 auth_method 0 2024-09-11T01:55:44.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.369+0000 7f0b737fe640 1 --2- 172.21.15.12:0/2963307237 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0b6003eae0 0x7f0b60040fa0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f0b6402f000 tx=0x7f0b64009070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:44.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.374+0000 7f0b5bfff640 1 -- 172.21.15.12:0/2963307237 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0b7409f680 con 0x7f0b5c005290 2024-09-11T01:55:44.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.374+0000 7f0b5bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:44.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.374+0000 7f0b5bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:55:44.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.632+0000 7f0b79a65640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:55:44.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.632+0000 7f0b79a65640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:55:44.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.632+0000 7f0b79a65640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:55:44.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.632+0000 7f0b79a65640 1 -- 172.21.15.12:0/2963307237 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0b74120a60 con 0x7f0b5c005290 2024-09-11T01:55:44.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.633+0000 7f0b5bfff640 1 -- 172.21.15.12:0/2963307237 <== mon.1 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f0b74120a60 con 0x7f0b5c005290 2024-09-11T01:55:44.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.633+0000 7f0b5bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:55:44.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.633+0000 7f0b5bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:55:44.635 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":244,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:55:44.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.635+0000 7f0b79a65640 1 -- 172.21.15.12:0/2963307237 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0b6003eae0 msgr2=0x7f0b60040fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:44.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.635+0000 7f0b79a65640 1 --2- 172.21.15.12:0/2963307237 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0b6003eae0 0x7f0b60040fa0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f0b6402f000 tx=0x7f0b64009070 comp rx=0 tx=0).stop 2024-09-11T01:55:44.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.635+0000 7f0b79a65640 10 monclient: shutdown 2024-09-11T01:55:44.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.635+0000 7f0b79a65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:44.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.635+0000 7f0b79a65640 1 -- 172.21.15.12:0/2963307237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b5c005290 msgr2=0x7f0b7413ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:44.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.635+0000 7f0b79a65640 1 --2- 172.21.15.12:0/2963307237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b5c005290 0x7f0b7413ba30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0b60007400 tx=0x7f0b6c00ad60 comp rx=0 tx=0).stop 2024-09-11T01:55:44.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.635+0000 7f0b79a65640 1 -- 172.21.15.12:0/2963307237 shutdown_connections 2024-09-11T01:55:44.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.635+0000 7f0b79a65640 1 --2- 172.21.15.12:0/2963307237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b5c005290 0x7f0b7413ba30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.635+0000 7f0b79a65640 1 --2- 172.21.15.12:0/2963307237 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0b5c004800 0x7f0b7413b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.635+0000 7f0b79a65640 1 --2- 172.21.15.12:0/2963307237 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f0b6003eae0 0x7f0b60040fa0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.635+0000 7f0b79a65640 1 --2- 172.21.15.12:0/2963307237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b5c0039b0 0x7f0b7411fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.635+0000 7f0b79a65640 1 -- 172.21.15.12:0/2963307237 >> 172.21.15.12:0/2963307237 conn(0x7f0b740a8490 msgr2=0x7f0b7409e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:44.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.635+0000 7f0b79a65640 1 -- 172.21.15.12:0/2963307237 shutdown_connections 2024-09-11T01:55:44.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.635+0000 7f0b79a65640 1 -- 172.21.15.12:0/2963307237 wait complete. 2024-09-11T01:55:44.648 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:55:44.648 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-11T01:55:44.648 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:55:44.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.748+0000 7ff98cc60640 10 monclient: get_monmap_and_config 2024-09-11T01:55:44.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.748+0000 7ff98cc60640 10 monclient: build_initial_monmap 2024-09-11T01:55:44.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.748+0000 7ff98cc60640 10 monclient: monmap: 2024-09-11T01:55:44.750 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:44.750 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:44.750 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:55:44.748975+0000 2024-09-11T01:55:44.750 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:55:44.748975+0000 2024-09-11T01:55:44.750 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:44.750 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:44.750 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:55:44.750 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:55:44.750 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:55:44.750 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:55:44.750 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:55:44.751 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:44.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.748+0000 7ff98cc60640 1 Processor -- start 2024-09-11T01:55:44.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.748+0000 7ff98cc60640 1 -- start start 2024-09-11T01:55:44.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.748+0000 7ff98cc60640 10 monclient: init 2024-09-11T01:55:44.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff98cc60640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:44.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff98cc60640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-11T01:55:44.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff98cc60640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff988137e50 0x7ff98809d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:44.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff98cc60640 10 monclient(hunting): picked mon.noname-b con 0x7ff988137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:44.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff98cc60640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff988138230 0x7ff988138630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:44.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff98cc60640 10 monclient(hunting): picked mon.noname-c con 0x7ff988138230 addr v2:172.21.15.71:3300/0 2024-09-11T01:55:44.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff98cc60640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff988134cb0 addr v1:172.21.15.71:6789/0 2024-09-11T01:55:44.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff98cc60640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9880a62d0 con 0x7ff988134cb0 2024-09-11T01:55:44.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff98cc60640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:44.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff98cc60640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7ff9880a65b0 con 0x7ff988138230 2024-09-11T01:55:44.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff98cc60640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:44.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff98cc60640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff98809bb90 con 0x7ff988137e50 2024-09-11T01:55:44.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff98cc60640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:44.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff98659c640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff988137e50 msgr2=0x7ff98809d160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:55:44.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff98cc60640 10 monclient(hunting): authenticate will time out at 2351.191639s 2024-09-11T01:55:44.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff98659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff988137e50 0x7ff98809d160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:55:44.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff985d9b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff988138230 0x7ff988138630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:44.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff985d9b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff988138230 0x7ff988138630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:34762/0 (socket says 172.21.15.12:34762) 2024-09-11T01:55:44.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff985d9b640 1 -- 172.21.15.12:0/684445785 learned_addr learned my addr 172.21.15.12:0/684445785 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:55:44.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff985d9b640 10 monclient(hunting): get_auth_request con 0x7ff988138230 auth_method 0 2024-09-11T01:55:44.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff985d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:44.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff985d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:44.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff985d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:44.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff986d9d640 1 --1- 172.21.15.12:0/684445785 >> v1:172.21.15.71:6789/0 conn(0x7ff988134cb0 0x7ff988135120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.12:47976/0 (socket says 172.21.15.12:47976) 2024-09-11T01:55:44.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff985d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:44.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff985d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:44.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.749+0000 7ff985d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:44.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.750+0000 7ff985d9b640 10 monclient(hunting): handle_auth_done global_id 274318 payload 274 2024-09-11T01:55:44.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.750+0000 7ff985d9b640 1 -- 172.21.15.12:0/684445785 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff988137e50 msgr2=0x7ff98809d160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:55:44.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.750+0000 7ff985d9b640 1 --2- 172.21.15.12:0/684445785 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff988137e50 0x7ff98809d160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.750+0000 7ff985d9b640 1 -- 172.21.15.12:0/684445785 >> v1:172.21.15.71:6789/0 conn(0x7ff988134cb0 legacy=0x7ff988135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:55:44.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.750+0000 7ff985d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:55:44.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.750+0000 7ff985d9b640 1 monclient: found mon.noname-c 2024-09-11T01:55:44.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.750+0000 7ff985d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:44.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.750+0000 7ff985d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:55:44.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.750+0000 7ff985d9b640 1 -- 172.21.15.12:0/684445785 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff98809c4f0 con 0x7ff988138230 2024-09-11T01:55:44.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.753+0000 7ff985d9b640 1 -- 172.21.15.12:0/684445785 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff988138230 msgr2=0x7ff988138630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:44.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.753+0000 7ff985d9b640 1 --2- 172.21.15.12:0/684445785 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff988138230 0x7ff988138630 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff97c009870 tx=0x7ff97c02a0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=304256bee5918503 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:44.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.753+0000 7ff98559a640 1 -- 172.21.15.12:0/684445785 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97c037070 con 0x7ff988138230 2024-09-11T01:55:44.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.753+0000 7ff98559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:44.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.753+0000 7ff98559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:55:44.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.753+0000 7ff98559a640 10 monclient: dump: 2024-09-11T01:55:44.760 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:44.761 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:44.761 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.761 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.761 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:44.761 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:44.761 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:44.761 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:44.761 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:44.761 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:44.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.753+0000 7ff98559a640 10 monclient: _finish_auth 0 2024-09-11T01:55:44.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.753+0000 7ff98559a640 10 monclient: _check_auth_tickets 2024-09-11T01:55:44.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.753+0000 7ff98559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:44.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.753+0000 7ff98559a640 1 -- 172.21.15.12:0/684445785 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff97c032640 con 0x7ff988138230 2024-09-11T01:55:44.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.753+0000 7ff98cc60640 5 monclient: authenticate success, global_id 274318 2024-09-11T01:55:44.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.753+0000 7ff98cc60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:55:44.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.753+0000 7ff98559a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:44.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.753+0000 7ff98cc60640 10 monclient: get_monmap_and_config success 2024-09-11T01:55:44.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.753+0000 7ff98559a640 1 -- 172.21.15.12:0/684445785 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97c032a10 con 0x7ff988138230 2024-09-11T01:55:44.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.753+0000 7ff98cc60640 4 set_mon_vals no callback set 2024-09-11T01:55:44.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.753+0000 7ff98559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:44.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.753+0000 7ff98559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:55:44.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.753+0000 7ff98559a640 10 monclient: dump: 2024-09-11T01:55:44.762 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:44.762 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:44.762 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.762 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.762 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:44.763 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:44.763 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:44.763 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:44.763 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:44.763 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:44.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.753+0000 7ff98cc60640 10 monclient: shutdown 2024-09-11T01:55:44.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.753+0000 7ff98cc60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:44.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.753+0000 7ff98cc60640 1 -- 172.21.15.12:0/684445785 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff988138230 msgr2=0x7ff988138630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:44.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.753+0000 7ff98cc60640 1 --2- 172.21.15.12:0/684445785 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff988138230 0x7ff988138630 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff97c009870 tx=0x7ff97c02a0d0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.754+0000 7ff98cc60640 1 -- 172.21.15.12:0/684445785 shutdown_connections 2024-09-11T01:55:44.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.754+0000 7ff98cc60640 1 --2- 172.21.15.12:0/684445785 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff988138230 0x7ff988138630 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.754+0000 7ff98cc60640 1 --2- 172.21.15.12:0/684445785 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff988137e50 0x7ff98809d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.754+0000 7ff98cc60640 1 -- 172.21.15.12:0/684445785 >> 172.21.15.12:0/684445785 conn(0x7ff9880a7020 msgr2=0x7ff9880a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:44.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.754+0000 7ff98cc60640 1 -- 172.21.15.12:0/684445785 shutdown_connections 2024-09-11T01:55:44.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.754+0000 7ff98cc60640 1 -- 172.21.15.12:0/684445785 wait complete. 2024-09-11T01:55:44.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.754+0000 7ff98cc60640 10 monclient: build_initial_monmap 2024-09-11T01:55:44.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.754+0000 7ff98cc60640 10 monclient: monmap: 2024-09-11T01:55:44.764 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:44.764 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:55:44.764 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:55:44.764 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:55:44.764 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:44.764 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:44.764 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:55:44.764 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:55:44.764 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:55:44.764 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:44.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff98cc60640 1 Processor -- start 2024-09-11T01:55:44.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff98cc60640 1 -- start start 2024-09-11T01:55:44.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff98cc60640 10 monclient: init 2024-09-11T01:55:44.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff98cc60640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:44.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff98cc60640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:55:44.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff98cc60640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff988134cb0 0x7ff9881cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:44.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff98cc60640 10 monclient(hunting): picked mon.noname-c con 0x7ff988134cb0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:44.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff98cc60640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff988137e50 0x7ff988133ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:44.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff98cc60640 10 monclient(hunting): picked mon.noname-a con 0x7ff988137e50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:55:44.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff98cc60640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff988138230 0x7ff9881341e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:44.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff98cc60640 10 monclient(hunting): picked mon.noname-b con 0x7ff988138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:55:44.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff98cc60640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:44.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff98cc60640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff98806f710 con 0x7ff988137e50 2024-09-11T01:55:44.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff98cc60640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:44.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff98cc60640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff988138b70 con 0x7ff988138230 2024-09-11T01:55:44.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff98cc60640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:44.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff98659c640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff988134cb0 msgr2=0x7ff9881cd1d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:55:44.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff98cc60640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff98806f890 con 0x7ff988134cb0 2024-09-11T01:55:44.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff98cc60640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:44.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff98cc60640 10 monclient(hunting): authenticate will time out at 2351.197822s 2024-09-11T01:55:44.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff98659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff988134cb0 0x7ff9881cd1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:55:44.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff985d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff988137e50 0x7ff988133ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:44.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff985d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff988137e50 0x7ff988133ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49088/0 (socket says 172.21.15.12:49088) 2024-09-11T01:55:44.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff985d9b640 1 -- 172.21.15.12:0/2313029414 learned_addr learned my addr 172.21.15.12:0/2313029414 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:55:44.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff985d9b640 10 monclient(hunting): get_auth_request con 0x7ff988137e50 auth_method 0 2024-09-11T01:55:44.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff985d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:44.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff985d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:44.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.755+0000 7ff985d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:44.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.756+0000 7ff985d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:44.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.756+0000 7ff985d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:44.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.756+0000 7ff985d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:44.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.756+0000 7ff986d9d640 1 --2- 172.21.15.12:0/2313029414 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff988138230 0x7ff9881341e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:44.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.756+0000 7ff985d9b640 10 monclient(hunting): handle_auth_done global_id 294174 payload 824 2024-09-11T01:55:44.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.756+0000 7ff985d9b640 1 -- 172.21.15.12:0/2313029414 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff988134cb0 msgr2=0x7ff9881cd1d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:55:44.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.756+0000 7ff985d9b640 1 --2- 172.21.15.12:0/2313029414 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff988134cb0 0x7ff9881cd1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.756+0000 7ff985d9b640 1 -- 172.21.15.12:0/2313029414 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff988138230 msgr2=0x7ff9881341e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:44.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.756+0000 7ff985d9b640 1 --2- 172.21.15.12:0/2313029414 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff988138230 0x7ff9881341e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.756+0000 7ff985d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:55:44.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.756+0000 7ff985d9b640 1 monclient: found mon.noname-a 2024-09-11T01:55:44.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.756+0000 7ff985d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:44.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.756+0000 7ff985d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:55:44.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.756+0000 7ff985d9b640 1 -- 172.21.15.12:0/2313029414 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9880a1710 con 0x7ff988137e50 2024-09-11T01:55:44.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.756+0000 7ff986d9d640 10 monclient: get_auth_request con 0x7ff988138230 auth_method 0 2024-09-11T01:55:44.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.756+0000 7ff986d9d640 1 --2- 172.21.15.12:0/2313029414 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff988138230 0x7ff9881341e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:55:44.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.758+0000 7ff985d9b640 1 -- 172.21.15.12:0/2313029414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff988137e50 msgr2=0x7ff988133ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:44.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.758+0000 7ff985d9b640 1 --2- 172.21.15.12:0/2313029414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff988137e50 0x7ff988133ca0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff97c0090b0 tx=0x7ff97c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:44.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.758+0000 7ff976ffd640 1 -- 172.21.15.12:0/2313029414 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97c041070 con 0x7ff988137e50 2024-09-11T01:55:44.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.758+0000 7ff976ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:44.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.758+0000 7ff976ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:55:44.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.758+0000 7ff976ffd640 10 monclient: dump: 2024-09-11T01:55:44.768 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:44.768 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:44.768 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.768 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.769 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:44.769 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:44.769 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:44.769 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:44.769 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:44.769 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:44.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.758+0000 7ff976ffd640 10 monclient: _finish_auth 0 2024-09-11T01:55:44.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.758+0000 7ff976ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:55:44.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.758+0000 7ff976ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:44.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.758+0000 7ff976ffd640 1 -- 172.21.15.12:0/2313029414 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff97c03ca80 con 0x7ff988137e50 2024-09-11T01:55:44.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.758+0000 7ff976ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:44.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.758+0000 7ff976ffd640 1 -- 172.21.15.12:0/2313029414 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97c037040 con 0x7ff988137e50 2024-09-11T01:55:44.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.758+0000 7ff976ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:44.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.758+0000 7ff976ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:55:44.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.758+0000 7ff976ffd640 10 monclient: dump: 2024-09-11T01:55:44.770 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:44.770 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:44.770 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.770 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.770 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:44.770 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:44.770 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:44.770 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:44.770 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:44.770 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:44.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.758+0000 7ff98cc60640 5 monclient: authenticate success, global_id 294174 2024-09-11T01:55:44.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.758+0000 7ff98cc60640 10 monclient: _renew_subs 2024-09-11T01:55:44.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.758+0000 7ff98cc60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:44.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.758+0000 7ff984d99640 4 set_mon_vals no callback set 2024-09-11T01:55:44.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.758+0000 7ff98cc60640 1 -- 172.21.15.12:0/2313029414 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9881256a0 con 0x7ff988137e50 2024-09-11T01:55:44.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.758+0000 7ff98cc60640 10 monclient: _renew_subs 2024-09-11T01:55:44.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.758+0000 7ff98cc60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:44.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.758+0000 7ff98cc60640 1 -- 172.21.15.12:0/2313029414 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff988123340 con 0x7ff988137e50 2024-09-11T01:55:44.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.759+0000 7ff98cc60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:44.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.759+0000 7ff98cc60640 1 --2- 172.21.15.12:0/2313029414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff988121880 0x7ff988121c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:44.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.759+0000 7ff98cc60640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:44.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.759+0000 7ff98cc60640 1 -- 172.21.15.12:0/2313029414 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff9881359a0 con 0x7ff988121880 2024-09-11T01:55:44.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.759+0000 7ff98659c640 1 --2- 172.21.15.12:0/2313029414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff988121880 0x7ff988121c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:44.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.759+0000 7ff976ffd640 1 -- 172.21.15.12:0/2313029414 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff97c03d3f0 con 0x7ff988137e50 2024-09-11T01:55:44.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.759+0000 7ff98659c640 10 monclient: get_auth_request con 0x7ff988121880 auth_method 0 2024-09-11T01:55:44.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.759+0000 7ff98659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:44.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.759+0000 7ff98659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:44.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.759+0000 7ff98659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:44.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.759+0000 7ff98659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:44.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.759+0000 7ff98659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:44.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.759+0000 7ff98659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:44.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.760+0000 7ff98659c640 10 monclient(hunting): handle_auth_done global_id 294177 payload 824 2024-09-11T01:55:44.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.760+0000 7ff98659c640 1 -- 172.21.15.12:0/2313029414 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff988125220 con 0x7ff988121880 2024-09-11T01:55:44.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.760+0000 7ff976ffd640 1 --2- 172.21.15.12:0/2313029414 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff96803e1c0 0x7ff968040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:44.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.760+0000 7ff976ffd640 1 -- 172.21.15.12:0/2313029414 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(890..890 src has 676..890) ==== 4875+0+0 (secure 0 0 0) 0x7ff97c073690 con 0x7ff988137e50 2024-09-11T01:55:44.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.760+0000 7ff986d9d640 1 --2- 172.21.15.12:0/2313029414 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff96803e1c0 0x7ff968040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:44.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.760+0000 7ff986d9d640 10 monclient: get_auth_request con 0x7ff96803e1c0 auth_method 0 2024-09-11T01:55:44.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.762+0000 7ff98659c640 1 -- 172.21.15.12:0/2313029414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff988121880 msgr2=0x7ff988121c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:44.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.762+0000 7ff98659c640 1 --2- 172.21.15.12:0/2313029414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff988121880 0x7ff988121c60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff97800b450 tx=0x7ff97800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:44.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.762+0000 7ff976ffd640 1 -- 172.21.15.12:0/2313029414 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff978005270 con 0x7ff988121880 2024-09-11T01:55:44.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.762+0000 7ff976ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:44.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.762+0000 7ff976ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:55:44.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.762+0000 7ff976ffd640 10 monclient: dump: 2024-09-11T01:55:44.776 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:44.776 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:44.777 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.777 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.777 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:44.777 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:44.777 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:44.777 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:44.777 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:44.777 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:44.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.762+0000 7ff976ffd640 1 -- 172.21.15.12:0/2313029414 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff978015070 con 0x7ff988121880 2024-09-11T01:55:44.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.762+0000 7ff976ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:44.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.762+0000 7ff976ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:55:44.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.762+0000 7ff976ffd640 1 -- 172.21.15.12:0/2313029414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff988121880 msgr2=0x7ff988121c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:44.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.762+0000 7ff976ffd640 1 --2- 172.21.15.12:0/2313029414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff988121880 0x7ff988121c60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff97800b450 tx=0x7ff97800b910 comp rx=0 tx=0).stop 2024-09-11T01:55:44.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.762+0000 7ff976ffd640 1 -- 172.21.15.12:0/2313029414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff988121880 msgr2=0x7ff988121c60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:55:44.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.762+0000 7ff976ffd640 1 --2- 172.21.15.12:0/2313029414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff988121880 0x7ff988121c60 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff97800b450 tx=0x7ff97800b910 comp rx=0 tx=0).stop 2024-09-11T01:55:44.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.763+0000 7ff986d9d640 1 --2- 172.21.15.12:0/2313029414 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff96803e1c0 0x7ff968040680 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7ff980007fd0 tx=0x7ff980009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:44.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.768+0000 7ff98cc60640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:55:44.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.768+0000 7ff98cc60640 1 --2- 172.21.15.12:0/2313029414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9881221a0 0x7ff9880a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:44.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.768+0000 7ff98cc60640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:44.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.768+0000 7ff98cc60640 1 -- 172.21.15.12:0/2313029414 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff9881225b0 con 0x7ff9881221a0 2024-09-11T01:55:44.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.768+0000 7ff98659c640 1 --2- 172.21.15.12:0/2313029414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9881221a0 0x7ff9880a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:44.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.768+0000 7ff98659c640 10 monclient: get_auth_request con 0x7ff9881221a0 auth_method 0 2024-09-11T01:55:44.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.768+0000 7ff98659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:44.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.768+0000 7ff98659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:44.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.768+0000 7ff98659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:44.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.768+0000 7ff98659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:44.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.768+0000 7ff98659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:44.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.768+0000 7ff98659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:44.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.769+0000 7ff98659c640 10 monclient(hunting): handle_auth_done global_id 294180 payload 824 2024-09-11T01:55:44.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.769+0000 7ff98659c640 1 -- 172.21.15.12:0/2313029414 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff988124e00 con 0x7ff9881221a0 2024-09-11T01:55:44.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.771+0000 7ff98659c640 1 -- 172.21.15.12:0/2313029414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9881221a0 msgr2=0x7ff9880a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:44.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.771+0000 7ff98659c640 1 --2- 172.21.15.12:0/2313029414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9881221a0 0x7ff9880a1920 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff97800b450 tx=0x7ff9780059d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:44.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.771+0000 7ff976ffd640 1 -- 172.21.15.12:0/2313029414 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff978023410 con 0x7ff9881221a0 2024-09-11T01:55:44.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.771+0000 7ff976ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:44.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.771+0000 7ff976ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:55:44.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.771+0000 7ff976ffd640 10 monclient: dump: 2024-09-11T01:55:44.779 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:44.779 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:44.779 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.780 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.780 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:44.780 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:44.780 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:44.780 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:44.780 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:44.780 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:44.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.771+0000 7ff976ffd640 1 -- 172.21.15.12:0/2313029414 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4699 (secure 0 0 0) 0x7ff978010400 con 0x7ff9881221a0 2024-09-11T01:55:44.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.771+0000 7ff976ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:55:44.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.771+0000 7ff976ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:55:44.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.771+0000 7ff976ffd640 1 -- 172.21.15.12:0/2313029414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9881221a0 msgr2=0x7ff9880a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:44.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.771+0000 7ff976ffd640 1 --2- 172.21.15.12:0/2313029414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9881221a0 0x7ff9880a1920 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff97800b450 tx=0x7ff9780059d0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.771+0000 7ff976ffd640 1 -- 172.21.15.12:0/2313029414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9881221a0 msgr2=0x7ff9880a1920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:55:44.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.771+0000 7ff976ffd640 1 --2- 172.21.15.12:0/2313029414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff9881221a0 0x7ff9880a1920 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff97800b450 tx=0x7ff9780059d0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.781 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:55:44.781 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:55:44.781 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:55:44.781 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:55:44.781 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 244, 2024-09-11T01:55:44.781 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 24724, 2024-09-11T01:55:44.781 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:55:44.781 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:55:44.781 INFO:teuthology.orchestra.run.smithi012.stdout: 1 2024-09-11T01:55:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:55:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:55:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:55:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:55:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:55:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:55:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:55:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:55:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:55:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:55:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:55:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:55:44.782 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:55:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:55:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:55:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:55:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:55:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:55:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:55:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:55:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:55:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:55:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:55:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:55:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:55:44.783 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:55:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:55:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:55:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:55:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:55:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:55:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:55:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:55:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:55:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:55:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:55:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:55:44.784 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:55:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:55:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:55:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:55:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:55:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:55:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:55:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:55:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:55:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:55:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:55:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:55:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:55:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:55:44.785 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:55:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:55:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:55:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:55:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:55:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:55:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:55:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:55:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:55:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:55:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:55:44.786 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:44.787 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:55:44.787 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:55:44.787 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:44.787 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:44.787 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:44.787 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:55:44.787 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:55:44.787 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:44.787 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:44.787 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:44.787 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:44.787 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:55:44.787 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:55:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:55:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:55:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:55:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:55:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:55:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:55:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:55:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:55:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:55:44.788 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:55:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:55:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:55:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:55:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:55:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:55:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:55:44.789 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:55:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:55:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:55:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:55:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:55:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:55:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:55:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:55:44.790 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:55:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:55:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:55:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:55:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:55:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:55:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:55:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:44.791 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:55:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:55:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:55:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:55:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:55:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:55:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:55:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:55:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:44.792 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:55:44.793 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:55:44.793 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:55:44.793 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:44.793 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:44.793 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:44.793 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:55:44.793 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:55:44.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.772+0000 7ff98cc60640 1 -- 172.21.15.12:0/2313029414 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff96803e1c0 msgr2=0x7ff968040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:44.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.772+0000 7ff98cc60640 1 --2- 172.21.15.12:0/2313029414 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff96803e1c0 0x7ff968040680 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7ff980007fd0 tx=0x7ff980009040 comp rx=0 tx=0).stop 2024-09-11T01:55:44.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.772+0000 7ff98cc60640 10 monclient: shutdown 2024-09-11T01:55:44.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.772+0000 7ff98cc60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:44.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.772+0000 7ff98cc60640 1 -- 172.21.15.12:0/2313029414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff988137e50 msgr2=0x7ff988133ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:44.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.772+0000 7ff98cc60640 1 --2- 172.21.15.12:0/2313029414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff988137e50 0x7ff988133ca0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff97c0090b0 tx=0x7ff97c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.772+0000 7ff986d9d640 1 -- 172.21.15.12:0/2313029414 reap_dead start 2024-09-11T01:55:44.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.772+0000 7ff98cc60640 1 -- 172.21.15.12:0/2313029414 shutdown_connections 2024-09-11T01:55:44.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.772+0000 7ff98cc60640 1 --2- 172.21.15.12:0/2313029414 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff988137e50 0x7ff988133ca0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.772+0000 7ff98cc60640 1 -- 172.21.15.12:0/2313029414 >> 172.21.15.12:0/2313029414 conn(0x7ff9880a7020 msgr2=0x7ff9881361c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:44.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.772+0000 7ff98cc60640 1 -- 172.21.15.12:0/2313029414 shutdown_connections 2024-09-11T01:55:44.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.772+0000 7ff98cc60640 1 -- 172.21.15.12:0/2313029414 wait complete. 2024-09-11T01:55:44.795 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:55:44.795 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T01:55:44.796 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:55:44.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.896+0000 7f59c8b75640 10 monclient: get_monmap_and_config 2024-09-11T01:55:44.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.896+0000 7f59c8b75640 10 monclient: build_initial_monmap 2024-09-11T01:55:44.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.896+0000 7f59c8b75640 10 monclient: monmap: 2024-09-11T01:55:44.898 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:44.898 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:44.898 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:55:44.897605+0000 2024-09-11T01:55:44.899 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:55:44.897605+0000 2024-09-11T01:55:44.899 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:44.899 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:44.899 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:55:44.899 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:55:44.899 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:55:44.899 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:55:44.899 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:55:44.899 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:44.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.897+0000 7f59c8b75640 1 Processor -- start 2024-09-11T01:55:44.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.897+0000 7f59c8b75640 1 -- start start 2024-09-11T01:55:44.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.897+0000 7f59c8b75640 10 monclient: init 2024-09-11T01:55:44.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c8b75640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:44.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c8b75640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-11T01:55:44.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c8b75640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f59c4135fc0 0x7f59c41363a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:44.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c8b75640 10 monclient(hunting): picked mon.noname-b con 0x7f59c4135fc0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:44.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c8b75640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f59c4073fe0 addr v1:172.21.15.12:6789/0 2024-09-11T01:55:44.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c8b75640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f59c40743c0 0x7f59c40a34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:44.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c8b75640 10 monclient(hunting): picked mon.noname-c con 0x7f59c40743c0 addr v2:172.21.15.71:3300/0 2024-09-11T01:55:44.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c8b75640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59c4072fb0 con 0x7f59c4073fe0 2024-09-11T01:55:44.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c259c640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f59c4135fc0 msgr2=0x7f59c41363a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:55:44.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c8b75640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:44.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c8b75640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f59c4073850 con 0x7f59c40743c0 2024-09-11T01:55:44.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c8b75640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:44.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c8b75640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f59c4073290 con 0x7f59c4135fc0 2024-09-11T01:55:44.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c259c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f59c4135fc0 0x7f59c41363a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:55:44.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c8b75640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:44.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c8b75640 10 monclient(hunting): authenticate will time out at 2351.340341s 2024-09-11T01:55:44.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c1d9b640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f59c4073fe0 0x7f59c40a2db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:46128/0 (socket says 172.21.15.12:46128) 2024-09-11T01:55:44.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c1d9b640 1 -- 172.21.15.12:0/1945302983 learned_addr learned my addr 172.21.15.12:0/1945302983 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:55:44.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c2d9d640 1 --2- 172.21.15.12:0/1945302983 >> v2:172.21.15.71:3300/0 conn(0x7f59c40743c0 0x7f59c40a34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:44.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c2d9d640 10 monclient(hunting): get_auth_request con 0x7f59c40743c0 auth_method 0 2024-09-11T01:55:44.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:44.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:44.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:44.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:44.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:44.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:44.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c2d9d640 10 monclient(hunting): handle_auth_done global_id 274321 payload 274 2024-09-11T01:55:44.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c2d9d640 1 -- 172.21.15.12:0/1945302983 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f59c4135fc0 msgr2=0x7f59c41363a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:55:44.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c2d9d640 1 --2- 172.21.15.12:0/1945302983 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f59c4135fc0 0x7f59c41363a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c2d9d640 1 -- 172.21.15.12:0/1945302983 >> v1:172.21.15.12:6789/0 conn(0x7f59c4073fe0 legacy=0x7f59c40a2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:55:44.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c2d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:55:44.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c2d9d640 1 monclient: found mon.noname-c 2024-09-11T01:55:44.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:44.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:55:44.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.898+0000 7f59c2d9d640 1 -- 172.21.15.12:0/1945302983 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59c4073570 con 0x7f59c40743c0 2024-09-11T01:55:44.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.901+0000 7f59c2d9d640 1 -- 172.21.15.12:0/1945302983 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59c40743c0 msgr2=0x7f59c40a34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:44.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.901+0000 7f59c2d9d640 1 --2- 172.21.15.12:0/1945302983 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59c40743c0 0x7f59c40a34c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f59bc009d90 tx=0x7f59bc02a4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=187747cf9c3a4b3a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:44.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.902+0000 7f59c159a640 1 -- 172.21.15.12:0/1945302983 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59bc007a30 con 0x7f59c40743c0 2024-09-11T01:55:44.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.902+0000 7f59c159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:44.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.902+0000 7f59c159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:55:44.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.902+0000 7f59c159a640 10 monclient: dump: 2024-09-11T01:55:44.911 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:44.911 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:44.911 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.911 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.911 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:44.911 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:44.912 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.902+0000 7f59c159a640 10 monclient: _finish_auth 0 2024-09-11T01:55:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.902+0000 7f59c159a640 10 monclient: _check_auth_tickets 2024-09-11T01:55:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.902+0000 7f59c159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.902+0000 7f59c159a640 1 -- 172.21.15.12:0/1945302983 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59bc007bd0 con 0x7f59c40743c0 2024-09-11T01:55:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.902+0000 7f59c159a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.902+0000 7f59c159a640 1 -- 172.21.15.12:0/1945302983 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59bc031400 con 0x7f59c40743c0 2024-09-11T01:55:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.902+0000 7f59c8b75640 5 monclient: authenticate success, global_id 274321 2024-09-11T01:55:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.902+0000 7f59c8b75640 10 monclient: get_monmap_and_config success 2024-09-11T01:55:44.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.902+0000 7f59c8b75640 4 set_mon_vals no callback set 2024-09-11T01:55:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.902+0000 7f59c159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.902+0000 7f59c159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:55:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.902+0000 7f59c159a640 10 monclient: dump: 2024-09-11T01:55:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:44.913 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:44.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.902+0000 7f59c8b75640 10 monclient: shutdown 2024-09-11T01:55:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.902+0000 7f59c8b75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.902+0000 7f59c8b75640 1 -- 172.21.15.12:0/1945302983 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59c40743c0 msgr2=0x7f59c40a34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.902+0000 7f59c8b75640 1 --2- 172.21.15.12:0/1945302983 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59c40743c0 0x7f59c40a34c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f59bc009d90 tx=0x7f59bc02a4b0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.902+0000 7f59c8b75640 1 -- 172.21.15.12:0/1945302983 shutdown_connections 2024-09-11T01:55:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.902+0000 7f59c8b75640 1 --2- 172.21.15.12:0/1945302983 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59c40743c0 0x7f59c40a34c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.902+0000 7f59c8b75640 1 --2- 172.21.15.12:0/1945302983 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f59c4135fc0 0x7f59c41363a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.902+0000 7f59c8b75640 1 -- 172.21.15.12:0/1945302983 >> 172.21.15.12:0/1945302983 conn(0x7f59c40a83b0 msgr2=0x7f59c40a87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.902+0000 7f59c8b75640 1 -- 172.21.15.12:0/1945302983 shutdown_connections 2024-09-11T01:55:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.902+0000 7f59c8b75640 1 -- 172.21.15.12:0/1945302983 wait complete. 2024-09-11T01:55:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.903+0000 7f59c8b75640 10 monclient: build_initial_monmap 2024-09-11T01:55:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.903+0000 7f59c8b75640 10 monclient: monmap: 2024-09-11T01:55:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:55:44.914 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:55:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:55:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:55:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:55:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:55:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:55:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:55:44.915 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.903+0000 7f59c8b75640 1 Processor -- start 2024-09-11T01:55:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c8b75640 1 -- start start 2024-09-11T01:55:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c8b75640 10 monclient: init 2024-09-11T01:55:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c8b75640 10 monclient: _reopen_session rank -1 2024-09-11T01:55:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c8b75640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:55:44.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c8b75640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59c4073fe0 0x7f59c41cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:44.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c8b75640 10 monclient(hunting): picked mon.noname-a con 0x7f59c4073fe0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:55:44.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c8b75640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59c40743c0 0x7f59c413cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:44.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c8b75640 10 monclient(hunting): picked mon.noname-b con 0x7f59c40743c0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:55:44.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c8b75640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f59c4135fc0 0x7f59c413d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:44.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c8b75640 10 monclient(hunting): picked mon.noname-c con 0x7f59c4135fc0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:55:44.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c8b75640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:44.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c8b75640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f59c4136a60 con 0x7f59c4073fe0 2024-09-11T01:55:44.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c8b75640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:44.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c8b75640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f59c41368e0 con 0x7f59c40743c0 2024-09-11T01:55:44.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c8b75640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c8b75640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f59c40a3a00 con 0x7f59c4135fc0 2024-09-11T01:55:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c8b75640 10 monclient(hunting): _renew_subs 2024-09-11T01:55:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c8b75640 10 monclient(hunting): authenticate will time out at 2351.346580s 2024-09-11T01:55:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c2d9d640 1 -- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f59c4135fc0 msgr2=0x7f59c413d4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:3301/0 2024-09-11T01:55:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c2d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f59c4135fc0 0x7f59c413d4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:55:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59c4073fe0 0x7f59c41cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59c4073fe0 0x7f59c41cb1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49116/0 (socket says 172.21.15.12:49116) 2024-09-11T01:55:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c259c640 1 -- 172.21.15.12:0/4085790401 learned_addr learned my addr 172.21.15.12:0/4085790401 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:55:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c259c640 10 monclient(hunting): get_auth_request con 0x7f59c4073fe0 auth_method 0 2024-09-11T01:55:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:44.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:44.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:44.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:44.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c1d9b640 1 --2- 172.21.15.12:0/4085790401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59c40743c0 0x7f59c413cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:44.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c1d9b640 10 monclient(hunting): get_auth_request con 0x7f59c40743c0 auth_method 0 2024-09-11T01:55:44.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:44.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:44.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:44.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:44.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:44.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:44.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c259c640 10 monclient(hunting): handle_auth_done global_id 294183 payload 824 2024-09-11T01:55:44.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c259c640 1 -- 172.21.15.12:0/4085790401 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f59c4135fc0 msgr2=0x7f59c413d4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:55:44.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c259c640 1 --2- 172.21.15.12:0/4085790401 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f59c4135fc0 0x7f59c413d4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c259c640 1 -- 172.21.15.12:0/4085790401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59c40743c0 msgr2=0x7f59c413cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:44.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c259c640 1 --2- 172.21.15.12:0/4085790401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59c40743c0 0x7f59c413cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c259c640 10 monclient: _finish_hunting 0 2024-09-11T01:55:44.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c259c640 1 monclient: found mon.noname-a 2024-09-11T01:55:44.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.904+0000 7f59c259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:55:44.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.905+0000 7f59c259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:55:44.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.905+0000 7f59c259c640 1 -- 172.21.15.12:0/4085790401 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59c40a2200 con 0x7f59c4073fe0 2024-09-11T01:55:44.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.907+0000 7f59c259c640 1 -- 172.21.15.12:0/4085790401 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59c4073fe0 msgr2=0x7f59c41cb1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:44.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.907+0000 7f59c259c640 1 --2- 172.21.15.12:0/4085790401 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59c4073fe0 0x7f59c41cb1c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f59b400b580 tx=0x7f59b400ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:44.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.907+0000 7f59b2ffd640 1 -- 172.21.15.12:0/4085790401 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b40050f0 con 0x7f59c4073fe0 2024-09-11T01:55:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.907+0000 7f59b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.907+0000 7f59b2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:55:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.907+0000 7f59b2ffd640 10 monclient: dump: 2024-09-11T01:55:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:44.920 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:44.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.907+0000 7f59b2ffd640 10 monclient: _finish_auth 0 2024-09-11T01:55:44.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.907+0000 7f59b2ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:55:44.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.907+0000 7f59b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:55:44.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.907+0000 7f59b2ffd640 1 -- 172.21.15.12:0/4085790401 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59b4015070 con 0x7f59c4073fe0 2024-09-11T01:55:44.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.907+0000 7f59c8b75640 5 monclient: authenticate success, global_id 294183 2024-09-11T01:55:44.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.907+0000 7f59c8b75640 10 monclient: _renew_subs 2024-09-11T01:55:44.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.907+0000 7f59c8b75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:44.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.907+0000 7f59c8b75640 1 -- 172.21.15.12:0/4085790401 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59c4139720 con 0x7f59c4073fe0 2024-09-11T01:55:44.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.907+0000 7f59b2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:55:44.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.907+0000 7f59b2ffd640 1 -- 172.21.15.12:0/4085790401 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b40054d0 con 0x7f59c4073fe0 2024-09-11T01:55:44.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.907+0000 7f59b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:44.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.907+0000 7f59b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:55:44.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.907+0000 7f59b2ffd640 10 monclient: dump: 2024-09-11T01:55:44.921 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:44.922 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:44.922 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.922 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.922 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:44.922 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:44.922 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:44.922 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:44.922 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:44.922 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:44.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.907+0000 7f59c0d99640 4 set_mon_vals no callback set 2024-09-11T01:55:44.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.907+0000 7f59c8b75640 10 monclient: _renew_subs 2024-09-11T01:55:44.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.907+0000 7f59c8b75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:55:44.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.907+0000 7f59c8b75640 1 -- 172.21.15.12:0/4085790401 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59c413b1e0 con 0x7f59c4073fe0 2024-09-11T01:55:44.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.908+0000 7f59c8b75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.908+0000 7f59c8b75640 1 --2- 172.21.15.12:0/4085790401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59c4137c60 0x7f59c4138040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.908+0000 7f59c8b75640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.908+0000 7f59c8b75640 1 -- 172.21.15.12:0/4085790401 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f59c406a060 con 0x7f59c4137c60 2024-09-11T01:55:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.908+0000 7f59b2ffd640 1 -- 172.21.15.12:0/4085790401 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f59b40056b0 con 0x7f59c4073fe0 2024-09-11T01:55:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.908+0000 7f59c1d9b640 1 --2- 172.21.15.12:0/4085790401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59c4137c60 0x7f59c4138040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.908+0000 7f59c1d9b640 10 monclient: get_auth_request con 0x7f59c4137c60 auth_method 0 2024-09-11T01:55:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.908+0000 7f59c1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.908+0000 7f59c1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.908+0000 7f59c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.908+0000 7f59c1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.908+0000 7f59c1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.908+0000 7f59c1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.908+0000 7f59b2ffd640 1 --2- 172.21.15.12:0/4085790401 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f59a403e1c0 0x7f59a4040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:44.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.908+0000 7f59b2ffd640 1 -- 172.21.15.12:0/4085790401 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(890..890 src has 676..890) ==== 4875+0+0 (secure 0 0 0) 0x7f59b4052bf0 con 0x7f59c4073fe0 2024-09-11T01:55:44.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.908+0000 7f59c1d9b640 10 monclient(hunting): handle_auth_done global_id 274327 payload 824 2024-09-11T01:55:44.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.908+0000 7f59c1d9b640 1 -- 172.21.15.12:0/4085790401 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f59c40a0430 con 0x7f59c4137c60 2024-09-11T01:55:44.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.908+0000 7f59c2d9d640 1 --2- 172.21.15.12:0/4085790401 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f59a403e1c0 0x7f59a4040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:44.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.908+0000 7f59c2d9d640 10 monclient: get_auth_request con 0x7f59a403e1c0 auth_method 0 2024-09-11T01:55:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.911+0000 7f59c1d9b640 1 -- 172.21.15.12:0/4085790401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59c4137c60 msgr2=0x7f59c4138040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.911+0000 7f59c1d9b640 1 --2- 172.21.15.12:0/4085790401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59c4137c60 0x7f59c4138040 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f59b8007fd0 tx=0x7f59b800ef00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.911+0000 7f59b2ffd640 1 -- 172.21.15.12:0/4085790401 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b801a070 con 0x7f59c4137c60 2024-09-11T01:55:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.911+0000 7f59b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.911+0000 7f59b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:55:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.911+0000 7f59b2ffd640 10 monclient: dump: 2024-09-11T01:55:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:44.926 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.927 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.927 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:44.927 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:44.927 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:44.927 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:44.927 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:44.927 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:44.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.911+0000 7f59c2d9d640 1 --2- 172.21.15.12:0/4085790401 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f59a403e1c0 0x7f59a4040680 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f59c4070c30 tx=0x7f59bc02f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:44.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.911+0000 7f59b2ffd640 1 -- 172.21.15.12:0/4085790401 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f59b800cc50 con 0x7f59c4137c60 2024-09-11T01:55:44.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.911+0000 7f59b2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:55:44.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.911+0000 7f59b2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:55:44.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.911+0000 7f59b2ffd640 1 -- 172.21.15.12:0/4085790401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59c4137c60 msgr2=0x7f59c4138040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:44.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.911+0000 7f59b2ffd640 1 --2- 172.21.15.12:0/4085790401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59c4137c60 0x7f59c4138040 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f59b8007fd0 tx=0x7f59b800ef00 comp rx=0 tx=0).stop 2024-09-11T01:55:44.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.911+0000 7f59b2ffd640 1 -- 172.21.15.12:0/4085790401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59c4137c60 msgr2=0x7f59c4138040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:55:44.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.911+0000 7f59b2ffd640 1 --2- 172.21.15.12:0/4085790401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59c4137c60 0x7f59c4138040 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.917+0000 7f59c8b75640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:55:44.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.917+0000 7f59c8b75640 1 --2- 172.21.15.12:0/4085790401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59c4138580 0x7f59c409c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:55:44.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.917+0000 7f59c8b75640 10 monclient(hunting): start opening mon connection 2024-09-11T01:55:44.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.917+0000 7f59c8b75640 1 -- 172.21.15.12:0/4085790401 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f59c413db50 con 0x7f59c4138580 2024-09-11T01:55:44.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.918+0000 7f59c1d9b640 1 --2- 172.21.15.12:0/4085790401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59c4138580 0x7f59c409c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:55:44.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.918+0000 7f59c1d9b640 10 monclient: get_auth_request con 0x7f59c4138580 auth_method 0 2024-09-11T01:55:44.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.918+0000 7f59c1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:55:44.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.918+0000 7f59c1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:55:44.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.918+0000 7f59c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:55:44.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.918+0000 7f59c1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:55:44.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.918+0000 7f59c1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:55:44.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.918+0000 7f59c1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:55:44.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.918+0000 7f59c1d9b640 10 monclient(hunting): handle_auth_done global_id 274330 payload 824 2024-09-11T01:55:44.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.918+0000 7f59c1d9b640 1 -- 172.21.15.12:0/4085790401 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f59c41363a0 con 0x7f59c4138580 2024-09-11T01:55:44.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.920+0000 7f59c1d9b640 1 -- 172.21.15.12:0/4085790401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59c4138580 msgr2=0x7f59c409c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:55:44.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.920+0000 7f59c1d9b640 1 --2- 172.21.15.12:0/4085790401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59c4138580 0x7f59c409c2b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f59b8017000 tx=0x7f59b8004940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:55:44.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.921+0000 7f59b2ffd640 1 -- 172.21.15.12:0/4085790401 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b801a070 con 0x7f59c4138580 2024-09-11T01:55:44.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.921+0000 7f59b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:55:44.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.921+0000 7f59b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:55:44.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.921+0000 7f59b2ffd640 10 monclient: dump: 2024-09-11T01:55:44.929 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:55:44.929 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:55:44.929 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.929 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:55:44.929 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:55:44.929 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:55:44.930 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:55:44.930 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:55:44.930 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:55:44.930 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:55:44.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.921+0000 7f59b2ffd640 1 -- 172.21.15.12:0/4085790401 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4865 (secure 0 0 0) 0x7f59b801b040 con 0x7f59c4138580 2024-09-11T01:55:44.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.921+0000 7f59b2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:55:44.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.921+0000 7f59b2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:55:44.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.921+0000 7f59b2ffd640 1 -- 172.21.15.12:0/4085790401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59c4138580 msgr2=0x7f59c409c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:44.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.921+0000 7f59b2ffd640 1 --2- 172.21.15.12:0/4085790401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59c4138580 0x7f59c409c2b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f59b8017000 tx=0x7f59b8004940 comp rx=0 tx=0).stop 2024-09-11T01:55:44.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.921+0000 7f59b2ffd640 1 -- 172.21.15.12:0/4085790401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59c4138580 msgr2=0x7f59c409c2b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:55:44.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.921+0000 7f59b2ffd640 1 --2- 172.21.15.12:0/4085790401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59c4138580 0x7f59c409c2b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:55:44.931 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:55:44.931 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:55:44.931 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:55:44.931 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:55:44.931 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 244, 2024-09-11T01:55:44.931 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 134739, 2024-09-11T01:55:44.931 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:55:44.931 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:55:44.931 INFO:teuthology.orchestra.run.smithi012.stdout: 1 2024-09-11T01:55:44.931 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:44.931 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:55:44.931 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:55:44.931 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:55:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:55:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:55:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:55:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:55:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:55:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:55:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:55:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:55:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:55:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:55:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:55:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:55:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:44.932 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:55:44.933 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:55:44.933 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:55:44.933 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:55:44.933 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:55:44.933 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:55:44.933 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:55:44.933 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:55:44.933 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:55:44.933 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:55:44.933 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:55:44.933 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:55:44.933 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:55:44.933 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:44.934 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:44.934 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:55:44.934 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:55:44.934 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:55:44.934 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:55:44.934 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:55:44.934 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:55:44.934 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:55:44.934 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:55:44.934 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:55:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:55:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:55:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:55:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:55:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:55:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:55:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:55:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:55:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:55:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:55:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:55:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:55:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:55:44.935 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:55:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:55:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:55:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:55:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:55:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:55:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:55:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:55:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:55:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:55:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:55:44.936 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:55:44.937 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:44.937 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:55:44.937 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:55:44.937 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:44.937 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:44.937 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:44.937 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:55:44.937 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:55:44.937 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:44.937 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:44.937 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:44.937 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:44.937 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:55:44.937 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:55:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:55:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:55:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:55:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:55:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:55:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:55:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:55:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:55:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:55:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:44.938 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:55:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:55:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:55:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:55:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:55:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:55:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:55:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:44.939 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:55:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:55:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:55:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:55:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:55:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:55:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:55:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:55:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:55:44.940 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:55:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:55:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:55:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:55:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:55:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:55:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:55:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:44.941 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:55:44.942 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:55:44.942 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:55:44.942 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:44.942 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:44.942 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:55:44.942 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:44.942 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:55:44.942 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:55:44.942 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:55:44.942 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:44.942 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:44.942 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:55:44.942 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:44.942 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:55:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:55:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:55:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:55:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:55:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:55:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:55:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:55:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:55:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:55:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:55:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:55:44.943 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:55:44.944 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:55:44.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.921+0000 7f59c8b75640 1 -- 172.21.15.12:0/4085790401 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f59a403e1c0 msgr2=0x7f59a4040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:44.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.921+0000 7f59c8b75640 1 --2- 172.21.15.12:0/4085790401 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f59a403e1c0 0x7f59a4040680 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f59c4070c30 tx=0x7f59bc02f000 comp rx=0 tx=0).stop 2024-09-11T01:55:44.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.921+0000 7f59c8b75640 10 monclient: shutdown 2024-09-11T01:55:44.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.921+0000 7f59c8b75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:55:44.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.921+0000 7f59c8b75640 1 -- 172.21.15.12:0/4085790401 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59c4073fe0 msgr2=0x7f59c41cb1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:55:44.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.921+0000 7f59c8b75640 1 --2- 172.21.15.12:0/4085790401 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59c4073fe0 0x7f59c41cb1c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f59b400b580 tx=0x7f59b400ba40 comp rx=0 tx=0).stop 2024-09-11T01:55:44.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.921+0000 7f59c2d9d640 1 -- 172.21.15.12:0/4085790401 reap_dead start 2024-09-11T01:55:44.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.921+0000 7f59c2d9d640 1 -- 172.21.15.12:0/4085790401 reap_dead start 2024-09-11T01:55:44.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.921+0000 7f59c8b75640 1 -- 172.21.15.12:0/4085790401 shutdown_connections 2024-09-11T01:55:44.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.921+0000 7f59c8b75640 1 -- 172.21.15.12:0/4085790401 >> 172.21.15.12:0/4085790401 conn(0x7f59c40a83b0 msgr2=0x7f59c4134d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:55:44.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.921+0000 7f59c8b75640 1 -- 172.21.15.12:0/4085790401 shutdown_connections 2024-09-11T01:55:44.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:55:44.921+0000 7f59c8b75640 1 -- 172.21.15.12:0/4085790401 wait complete. 2024-09-11T01:55:44.946 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:55:44.946 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:55:46.292 INFO:tasks.rados.rados.0.smithi071.stdout:3892: done (1 left) 2024-09-11T01:55:46.292 INFO:tasks.rados.rados.0.smithi071.stdout:3893: rollback oid 18 current snap is 437 2024-09-11T01:55:46.292 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 18 to 436 2024-09-11T01:55:46.292 INFO:tasks.rados.rados.0.smithi071.stdout:3891: expect (ObjNum 1327 snap 432 seq_num 1327) 2024-09-11T01:55:46.385 INFO:tasks.rados.rados.0.smithi071.stdout:3893: finishing rollback tid 1 to smithi07134777-18 2024-09-11T01:55:46.385 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 2213 (ObjNum 97 snap 0 seq_num 2370842768) dirty dne 2024-09-11T01:55:46.385 INFO:tasks.rados.rados.0.smithi071.stdout:3891: done (1 left) 2024-09-11T01:55:46.385 INFO:tasks.rados.rados.0.smithi071.stdout:3893: done (0 left) 2024-09-11T01:55:46.385 INFO:tasks.rados.rados.0.smithi071.stdout:3894: snap_remove snap 389 2024-09-11T01:55:46.976 INFO:tasks.rados.rados.0.smithi071.stdout:3894: done (0 left) 2024-09-11T01:55:46.976 INFO:tasks.rados.rados.0.smithi071.stdout:3895: snap_remove snap 434 2024-09-11T01:55:48.920 INFO:tasks.rados.rados.0.smithi071.stdout:3895: done (0 left) 2024-09-11T01:55:48.921 INFO:tasks.rados.rados.0.smithi071.stdout:3896: snap_create 2024-09-11T01:55:48.990 INFO:tasks.rados.rados.0.smithi071.stdout:3896: done (0 left) 2024-09-11T01:55:48.990 INFO:tasks.rados.rados.0.smithi071.stdout:3897: delete oid 20 current snap is 438 2024-09-11T01:55:48.994 INFO:tasks.rados.rados.0.smithi071.stdout:3897: done (0 left) 2024-09-11T01:55:48.994 INFO:tasks.rados.rados.0.smithi071.stdout:3898: delete oid 29 current snap is 438 2024-09-11T01:55:49.000 INFO:tasks.rados.rados.0.smithi071.stdout:3898: done (0 left) 2024-09-11T01:55:49.000 INFO:tasks.rados.rados.0.smithi071.stdout:3899: copy_from oid 9 from oid 23 current snap is 438 2024-09-11T01:55:49.001 INFO:tasks.rados.rados.0.smithi071.stdout:3900: read oid 34 snap -1 2024-09-11T01:55:49.001 INFO:tasks.rados.rados.0.smithi071.stdout:3900: expect (ObjNum 1295 snap 426 seq_num 1295) 2024-09-11T01:55:49.001 INFO:tasks.rados.rados.0.smithi071.stdout:3901: delete oid 49 current snap is 438 2024-09-11T01:55:49.005 INFO:tasks.rados.rados.0.smithi071.stdout:3901: done (2 left) 2024-09-11T01:55:49.006 INFO:tasks.rados.rados.0.smithi071.stdout:3902: write oid 42 current snap is 438 2024-09-11T01:55:49.006 INFO:tasks.rados.rados.0.smithi071.stdout:3902: seq_num 1342 ranges {554994=662286,1997188=557806} 2024-09-11T01:55:49.012 INFO:tasks.rados.rados.0.smithi071.stdout:3902: writing smithi07134777-42 from 554994 to 1217280 tid 1 2024-09-11T01:55:49.021 INFO:tasks.rados.rados.0.smithi071.stdout:3902: writing smithi07134777-42 from 1997188 to 2554994 tid 2 2024-09-11T01:55:49.021 INFO:tasks.rados.rados.0.smithi071.stdout:3903: snap_remove snap 433 2024-09-11T01:55:49.996 INFO:tasks.rados.rados.0.smithi071.stdout:3903: done (3 left) 2024-09-11T01:55:49.996 INFO:tasks.rados.rados.0.smithi071.stdout:3904: read oid 10 snap -1 2024-09-11T01:55:49.996 INFO:tasks.rados.rados.0.smithi071.stdout:3904: expect (ObjNum 1313 snap 428 seq_num 1313) 2024-09-11T01:55:49.996 INFO:tasks.rados.rados.0.smithi071.stdout:3905: rollback oid 15 current snap is 438 2024-09-11T01:55:49.996 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 15 to 364 2024-09-11T01:55:49.996 INFO:tasks.rados.rados.0.smithi071.stdout:3900: expect (ObjNum 1295 snap 426 seq_num 1295) 2024-09-11T01:55:50.039 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:55:50.081 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:55:50.122 INFO:tasks.rados.rados.0.smithi071.stdout:3902: finishing write tid 1 to smithi07134777-42 2024-09-11T01:55:50.122 INFO:tasks.rados.rados.0.smithi071.stdout:3899: finishing copy_from racing read to smithi07134777-9 2024-09-11T01:55:50.122 INFO:tasks.rados.rados.0.smithi071.stdout:3899: finishing copy_from to smithi07134777-9 2024-09-11T01:55:50.122 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 1301 (ObjNum 1282 snap 422 seq_num 1282) dirty exists 2024-09-11T01:55:50.122 INFO:tasks.rados.rados.0.smithi071.stdout:3902: finishing write tid 2 to smithi07134777-42 2024-09-11T01:55:50.122 INFO:tasks.rados.rados.0.smithi071.stdout:3902: finishing write tid 3 to smithi07134777-42 2024-09-11T01:55:50.122 INFO:tasks.rados.rados.0.smithi071.stdout:3902: finishing write tid 4 to smithi07134777-42 2024-09-11T01:55:50.122 INFO:tasks.rados.rados.0.smithi071.stdout:3902: oid 42 updating version 0 to 2555 2024-09-11T01:55:50.122 INFO:tasks.rados.rados.0.smithi071.stdout:3902: oid 42 updating version 2555 to 2556 2024-09-11T01:55:50.122 INFO:tasks.rados.rados.0.smithi071.stdout:3902: oid 42 version 2556 is already newer than 2554 2024-09-11T01:55:50.123 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 2556 (ObjNum 1342 snap 438 seq_num 1342) dirty exists 2024-09-11T01:55:50.123 INFO:tasks.rados.rados.0.smithi071.stdout:3902: left oid 42 (ObjNum 1342 snap 438 seq_num 1342) 2024-09-11T01:55:50.123 INFO:tasks.rados.rados.0.smithi071.stdout:3899: done (4 left) 2024-09-11T01:55:50.123 INFO:tasks.rados.rados.0.smithi071.stdout:3900: done (3 left) 2024-09-11T01:55:50.123 INFO:tasks.rados.rados.0.smithi071.stdout:3902: done (2 left) 2024-09-11T01:55:50.123 INFO:tasks.rados.rados.0.smithi071.stdout:3906: snap_create 2024-09-11T01:55:50.123 INFO:tasks.rados.rados.0.smithi071.stdout:3904: expect (ObjNum 1313 snap 428 seq_num 1313) 2024-09-11T01:55:50.227 INFO:tasks.rados.rados.0.smithi071.stdout:3905: finishing rollback tid 1 to smithi07134777-15 2024-09-11T01:55:50.227 INFO:tasks.rados.rados.0.smithi071.stdout:3905: finishing rollback tid 2 to smithi07134777-15 2024-09-11T01:55:50.227 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 3163 (ObjNum 1017 snap 322 seq_num 1017) dirty exists 2024-09-11T01:55:51.073 INFO:tasks.rados.rados.0.smithi071.stdout:3904: done (2 left) 2024-09-11T01:55:51.073 INFO:tasks.rados.rados.0.smithi071.stdout:3905: done (1 left) 2024-09-11T01:55:51.073 INFO:tasks.rados.rados.0.smithi071.stdout:3906: done (0 left) 2024-09-11T01:55:51.073 INFO:tasks.rados.rados.0.smithi071.stdout:3907: snap_create 2024-09-11T01:55:52.083 INFO:tasks.rados.rados.0.smithi071.stdout:3907: done (0 left) 2024-09-11T01:55:52.083 INFO:tasks.rados.rados.0.smithi071.stdout:3908: rollback oid 37 current snap is 440 2024-09-11T01:55:52.083 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 37 to 420 2024-09-11T01:55:52.083 INFO:tasks.rados.rados.0.smithi071.stdout:3909: snap_create 2024-09-11T01:55:52.089 INFO:tasks.rados.rados.0.smithi071.stdout:3908: finishing rollback tid 0 to smithi07134777-37 2024-09-11T01:55:52.092 INFO:tasks.rados.rados.0.smithi071.stdout:3908: finishing rollback tid 1 to smithi07134777-37 2024-09-11T01:55:52.092 INFO:tasks.rados.rados.0.smithi071.stdout:3908: finishing rollback tid 2 to smithi07134777-37 2024-09-11T01:55:52.092 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 2543 (ObjNum 1252 snap 415 seq_num 1252) dirty exists 2024-09-11T01:55:53.085 INFO:tasks.rados.rados.0.smithi071.stdout:3908: done (1 left) 2024-09-11T01:55:53.085 INFO:tasks.rados.rados.0.smithi071.stdout:3909: done (0 left) 2024-09-11T01:55:53.085 INFO:tasks.rados.rados.0.smithi071.stdout:3910: copy_from oid 42 from oid 13 current snap is 441 2024-09-11T01:55:53.085 INFO:tasks.rados.rados.0.smithi071.stdout:3911: write oid 47 current snap is 441 2024-09-11T01:55:53.086 INFO:tasks.rados.rados.0.smithi071.stdout:3911: seq_num 1343 ranges {571801=737535,1939317=489789} 2024-09-11T01:55:53.092 INFO:tasks.rados.rados.0.smithi071.stdout:3911: writing smithi07134777-47 from 571801 to 1309336 tid 1 2024-09-11T01:55:53.099 INFO:tasks.rados.rados.0.smithi071.stdout:3911: writing smithi07134777-47 from 1939317 to 2429106 tid 2 2024-09-11T01:55:53.100 INFO:tasks.rados.rados.0.smithi071.stdout:3912: write oid 14 current snap is 441 2024-09-11T01:55:53.100 INFO:tasks.rados.rados.0.smithi071.stdout:3910: finishing copy_from racing read to smithi07134777-42 2024-09-11T01:55:53.100 INFO:tasks.rados.rados.0.smithi071.stdout:3912: seq_num 1344 ranges {588608=412784,1597799=749800} 2024-09-11T01:55:53.105 INFO:tasks.rados.rados.0.smithi071.stdout:3912: writing smithi07134777-14 from 588608 to 1001392 tid 1 2024-09-11T01:55:53.114 INFO:tasks.rados.rados.0.smithi071.stdout:3912: writing smithi07134777-14 from 1597799 to 2347599 tid 2 2024-09-11T01:55:53.114 INFO:tasks.rados.rados.0.smithi071.stdout:3913: delete oid 38 current snap is 441 2024-09-11T01:55:53.114 INFO:tasks.rados.rados.0.smithi071.stdout:3910: finishing copy_from to smithi07134777-42 2024-09-11T01:55:53.114 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 2588 (ObjNum 1334 snap 435 seq_num 1334) dirty exists 2024-09-11T01:55:53.115 INFO:tasks.rados.rados.0.smithi071.stdout:3911: finishing write tid 1 to smithi07134777-47 2024-09-11T01:55:53.115 INFO:tasks.rados.rados.0.smithi071.stdout:3911: finishing write tid 2 to smithi07134777-47 2024-09-11T01:55:53.115 INFO:tasks.rados.rados.0.smithi071.stdout:3911: finishing write tid 3 to smithi07134777-47 2024-09-11T01:55:53.115 INFO:tasks.rados.rados.0.smithi071.stdout:3911: finishing write tid 4 to smithi07134777-47 2024-09-11T01:55:53.115 INFO:tasks.rados.rados.0.smithi071.stdout:3911: oid 47 updating version 0 to 2591 2024-09-11T01:55:53.115 INFO:tasks.rados.rados.0.smithi071.stdout:3911: oid 47 version 2591 is already newer than 2589 2024-09-11T01:55:53.116 INFO:tasks.rados.rados.0.smithi071.stdout:3911: oid 47 version 2591 is already newer than 2590 2024-09-11T01:55:53.116 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 2591 (ObjNum 1343 snap 441 seq_num 1343) dirty exists 2024-09-11T01:55:53.116 INFO:tasks.rados.rados.0.smithi071.stdout:3911: left oid 47 (ObjNum 1343 snap 441 seq_num 1343) 2024-09-11T01:55:53.116 INFO:tasks.rados.rados.0.smithi071.stdout:3912: finishing write tid 1 to smithi07134777-14 2024-09-11T01:55:53.119 INFO:tasks.rados.rados.0.smithi071.stdout:3912: finishing write tid 2 to smithi07134777-14 2024-09-11T01:55:53.119 INFO:tasks.rados.rados.0.smithi071.stdout:3912: finishing write tid 3 to smithi07134777-14 2024-09-11T01:55:53.119 INFO:tasks.rados.rados.0.smithi071.stdout:3912: finishing write tid 4 to smithi07134777-14 2024-09-11T01:55:53.120 INFO:tasks.rados.rados.0.smithi071.stdout:3912: oid 14 updating version 0 to 3168 2024-09-11T01:55:53.120 INFO:tasks.rados.rados.0.smithi071.stdout:3912: oid 14 version 3168 is already newer than 3167 2024-09-11T01:55:53.120 INFO:tasks.rados.rados.0.smithi071.stdout:3912: oid 14 version 3168 is already newer than 3166 2024-09-11T01:55:53.120 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 3168 (ObjNum 1344 snap 441 seq_num 1344) dirty exists 2024-09-11T01:55:53.120 INFO:tasks.rados.rados.0.smithi071.stdout:3912: left oid 14 (ObjNum 1344 snap 441 seq_num 1344) 2024-09-11T01:55:53.127 INFO:tasks.rados.rados.0.smithi071.stdout:3910: done (3 left) 2024-09-11T01:55:53.127 INFO:tasks.rados.rados.0.smithi071.stdout:3911: done (2 left) 2024-09-11T01:55:53.127 INFO:tasks.rados.rados.0.smithi071.stdout:3912: done (1 left) 2024-09-11T01:55:53.128 INFO:tasks.rados.rados.0.smithi071.stdout:3913: done (0 left) 2024-09-11T01:55:53.128 INFO:tasks.rados.rados.0.smithi071.stdout:3914: write (excl) oid 23 current snap is 441 2024-09-11T01:55:53.128 INFO:tasks.rados.rados.0.smithi071.stdout:3914: seq_num 1346 ranges {622222=679635,1831116=753469} 2024-09-11T01:55:53.134 INFO:tasks.rados.rados.0.smithi071.stdout:3914: writing smithi07134777-23 from 622222 to 1301857 tid 1 2024-09-11T01:55:53.141 INFO:tasks.rados.rados.0.smithi071.stdout:3914: writing smithi07134777-23 from 1831116 to 2584585 tid 2 2024-09-11T01:55:53.141 INFO:tasks.rados.rados.0.smithi071.stdout:3915: snap_remove snap 425 2024-09-11T01:55:54.098 INFO:tasks.rados.rados.0.smithi071.stdout:3915: done (1 left) 2024-09-11T01:55:54.098 INFO:tasks.rados.rados.0.smithi071.stdout:3916: delete oid 41 current snap is 441 2024-09-11T01:55:54.098 INFO:tasks.rados.rados.0.smithi071.stdout:3914: finishing write tid 1 to smithi07134777-23 2024-09-11T01:55:54.098 INFO:tasks.rados.rados.0.smithi071.stdout:3914: finishing write tid 2 to smithi07134777-23 2024-09-11T01:55:54.099 INFO:tasks.rados.rados.0.smithi071.stdout:3914: finishing write tid 3 to smithi07134777-23 2024-09-11T01:55:54.099 INFO:tasks.rados.rados.0.smithi071.stdout:3914: finishing write tid 4 to smithi07134777-23 2024-09-11T01:55:54.099 INFO:tasks.rados.rados.0.smithi071.stdout:3914: oid 23 updating version 0 to 3029 2024-09-11T01:55:54.099 INFO:tasks.rados.rados.0.smithi071.stdout:3914: oid 23 updating version 3029 to 3030 2024-09-11T01:55:54.099 INFO:tasks.rados.rados.0.smithi071.stdout:3914: oid 23 version 3030 is already newer than 3028 2024-09-11T01:55:54.099 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 3030 (ObjNum 1346 snap 441 seq_num 1346) dirty exists 2024-09-11T01:55:54.099 INFO:tasks.rados.rados.0.smithi071.stdout:3914: left oid 23 (ObjNum 1346 snap 441 seq_num 1346) 2024-09-11T01:55:54.105 INFO:tasks.rados.rados.0.smithi071.stdout:3914: done (1 left) 2024-09-11T01:55:54.105 INFO:tasks.rados.rados.0.smithi071.stdout:3916: done (0 left) 2024-09-11T01:55:54.105 INFO:tasks.rados.rados.0.smithi071.stdout:3917: read oid 10 snap -1 2024-09-11T01:55:54.105 INFO:tasks.rados.rados.0.smithi071.stdout:3917: expect (ObjNum 1313 snap 428 seq_num 1313) 2024-09-11T01:55:54.105 INFO:tasks.rados.rados.0.smithi071.stdout:3918: snap_remove snap 428 2024-09-11T01:55:55.068 INFO:tasks.rados.rados.0.smithi071.stdout:3918: done (1 left) 2024-09-11T01:55:55.069 INFO:tasks.rados.rados.0.smithi071.stdout:3919: read oid 23 snap -1 2024-09-11T01:55:55.069 INFO:tasks.rados.rados.0.smithi071.stdout:3919: expect (ObjNum 1346 snap 441 seq_num 1346) 2024-09-11T01:55:55.069 INFO:tasks.rados.rados.0.smithi071.stdout:3920: snap_create 2024-09-11T01:55:55.069 INFO:tasks.rados.rados.0.smithi071.stdout:3917: expect (ObjNum 1313 snap 428 seq_num 1313) 2024-09-11T01:55:55.180 INFO:tasks.rados.rados.0.smithi071.stdout:3919: expect (ObjNum 1346 snap 441 seq_num 1346) 2024-09-11T01:55:55.223 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:55:55.272 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:55:56.122 INFO:tasks.rados.rados.0.smithi071.stdout:3917: done (2 left) 2024-09-11T01:55:56.122 INFO:tasks.rados.rados.0.smithi071.stdout:3919: done (1 left) 2024-09-11T01:55:56.122 INFO:tasks.rados.rados.0.smithi071.stdout:3920: done (0 left) 2024-09-11T01:55:56.122 INFO:tasks.rados.rados.0.smithi071.stdout:3921: delete oid 24 current snap is 442 2024-09-11T01:55:56.128 INFO:tasks.rados.rados.0.smithi071.stdout:3921: done (0 left) 2024-09-11T01:55:56.128 INFO:tasks.rados.rados.0.smithi071.stdout:3922: snap_create 2024-09-11T01:55:57.103 INFO:tasks.rados.rados.0.smithi071.stdout:3922: done (0 left) 2024-09-11T01:55:57.103 INFO:tasks.rados.rados.0.smithi071.stdout:3923: read oid 37 snap -1 2024-09-11T01:55:57.103 INFO:tasks.rados.rados.0.smithi071.stdout:3923: expect (ObjNum 1252 snap 415 seq_num 1252) 2024-09-11T01:55:57.103 INFO:tasks.rados.rados.0.smithi071.stdout:3924: rollback oid 28 current snap is 443 2024-09-11T01:55:57.103 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 28 to 429 2024-09-11T01:55:57.103 INFO:tasks.rados.rados.0.smithi071.stdout:3925: delete oid 13 current snap is 443 2024-09-11T01:55:57.108 INFO:tasks.rados.rados.0.smithi071.stdout:3925: done (2 left) 2024-09-11T01:55:57.108 INFO:tasks.rados.rados.0.smithi071.stdout:3926: snap_remove snap 364 2024-09-11T01:55:58.107 INFO:tasks.rados.rados.0.smithi071.stdout:3926: done (2 left) 2024-09-11T01:55:58.107 INFO:tasks.rados.rados.0.smithi071.stdout:3927: rollback oid 18 current snap is 443 2024-09-11T01:55:58.108 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 18 to 436 2024-09-11T01:55:58.108 INFO:tasks.rados.rados.0.smithi071.stdout:3924: finishing rollback tid 1 to smithi07134777-28 2024-09-11T01:55:58.108 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 2411 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-11T01:55:58.108 INFO:tasks.rados.rados.0.smithi071.stdout:3923: expect (ObjNum 1252 snap 415 seq_num 1252) 2024-09-11T01:55:58.108 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 282312 2024-09-11T01:55:58.240 INFO:tasks.rados.rados.0.smithi071.stdout:3927: finishing rollback tid 1 to smithi07134777-18 2024-09-11T01:55:58.240 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 2213 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-11T01:55:58.240 INFO:tasks.rados.rados.0.smithi071.stdout:3923: done (2 left) 2024-09-11T01:55:58.240 INFO:tasks.rados.rados.0.smithi071.stdout:3924: done (1 left) 2024-09-11T01:55:58.240 INFO:tasks.rados.rados.0.smithi071.stdout:3927: done (0 left) 2024-09-11T01:55:58.240 INFO:tasks.rados.rados.0.smithi071.stdout:3928: read oid 18 snap -1 2024-09-11T01:55:58.240 INFO:tasks.rados.rados.0.smithi071.stdout:3928: expect deleted 2024-09-11T01:55:58.240 INFO:tasks.rados.rados.0.smithi071.stdout:3929: read oid 27 snap 439 2024-09-11T01:55:58.240 INFO:tasks.rados.rados.0.smithi071.stdout:3929: expect (ObjNum 1317 snap 429 seq_num 1317) 2024-09-11T01:55:58.240 INFO:tasks.rados.rados.0.smithi071.stdout:3930: read oid 10 snap 431 2024-09-11T01:55:58.241 INFO:tasks.rados.rados.0.smithi071.stdout:3930: expect (ObjNum 1232 snap 403 seq_num 1232) 2024-09-11T01:55:58.241 INFO:tasks.rados.rados.0.smithi071.stdout:3931: rollback oid 3 current snap is 443 2024-09-11T01:55:58.241 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 3 to 432 2024-09-11T01:55:58.241 INFO:tasks.rados.rados.0.smithi071.stdout:3932: snap_create 2024-09-11T01:55:58.243 INFO:tasks.rados.rados.0.smithi071.stdout:3931: finishing rollback tid 0 to smithi07134777-3 2024-09-11T01:55:58.244 INFO:tasks.rados.rados.0.smithi071.stdout:3931: finishing rollback tid 1 to smithi07134777-3 2024-09-11T01:55:58.244 INFO:tasks.rados.rados.0.smithi071.stdout:3931: finishing rollback tid 2 to smithi07134777-3 2024-09-11T01:55:58.244 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 2536 (ObjNum 1300 snap 426 seq_num 1300) dirty exists 2024-09-11T01:55:58.246 INFO:tasks.rados.rados.0.smithi071.stdout:3930: expect (ObjNum 1232 snap 403 seq_num 1232) 2024-09-11T01:55:58.281 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 2637117 2024-09-11T01:55:58.297 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1117036 2024-09-11T01:55:58.370 INFO:tasks.rados.rados.0.smithi071.stdout:3929: expect (ObjNum 1317 snap 429 seq_num 1317) 2024-09-11T01:55:58.370 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:55:59.110 INFO:tasks.rados.rados.0.smithi071.stdout:3928: done (4 left) 2024-09-11T01:55:59.110 INFO:tasks.rados.rados.0.smithi071.stdout:3929: done (3 left) 2024-09-11T01:55:59.110 INFO:tasks.rados.rados.0.smithi071.stdout:3930: done (2 left) 2024-09-11T01:55:59.111 INFO:tasks.rados.rados.0.smithi071.stdout:3931: done (1 left) 2024-09-11T01:55:59.111 INFO:tasks.rados.rados.0.smithi071.stdout:3932: done (0 left) 2024-09-11T01:55:59.111 INFO:tasks.rados.rados.0.smithi071.stdout:3933: snap_create 2024-09-11T01:56:00.069 INFO:tasks.rados.rados.0.smithi071.stdout:3933: done (0 left) 2024-09-11T01:56:00.070 INFO:tasks.rados.rados.0.smithi071.stdout:3934: read oid 44 snap 325 2024-09-11T01:56:00.070 INFO:tasks.rados.rados.0.smithi071.stdout:3934: expect (ObjNum 1009 snap 322 seq_num 1009) 2024-09-11T01:56:00.070 INFO:tasks.rados.rados.0.smithi071.stdout:3935: rollback oid 19 current snap is 445 2024-09-11T01:56:00.070 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 19 to 440 2024-09-11T01:56:00.070 INFO:tasks.rados.rados.0.smithi071.stdout:3936: snap_create 2024-09-11T01:56:00.077 INFO:tasks.rados.rados.0.smithi071.stdout:3935: finishing rollback tid 0 to smithi07134777-19 2024-09-11T01:56:00.077 INFO:tasks.rados.rados.0.smithi071.stdout:3935: finishing rollback tid 1 to smithi07134777-19 2024-09-11T01:56:00.078 INFO:tasks.rados.rados.0.smithi071.stdout:3935: finishing rollback tid 2 to smithi07134777-19 2024-09-11T01:56:00.078 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 2585 (ObjNum 1243 snap 411 seq_num 1243) dirty exists 2024-09-11T01:56:00.079 INFO:tasks.rados.rados.0.smithi071.stdout:3934: expect (ObjNum 1009 snap 322 seq_num 1009) 2024-09-11T01:56:00.225 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 787318 2024-09-11T01:56:01.075 INFO:tasks.rados.rados.0.smithi071.stdout:3934: done (2 left) 2024-09-11T01:56:01.075 INFO:tasks.rados.rados.0.smithi071.stdout:3935: done (1 left) 2024-09-11T01:56:01.075 INFO:tasks.rados.rados.0.smithi071.stdout:3936: done (0 left) 2024-09-11T01:56:01.075 INFO:tasks.rados.rados.0.smithi071.stdout:3937: delete oid 44 current snap is 446 2024-09-11T01:56:01.078 INFO:tasks.rados.rados.0.smithi071.stdout:3937: done (0 left) 2024-09-11T01:56:01.078 INFO:tasks.rados.rados.0.smithi071.stdout:3938: rollback oid 15 current snap is 446 2024-09-11T01:56:01.078 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 15 to 438 2024-09-11T01:56:01.078 INFO:tasks.rados.rados.0.smithi071.stdout:3939: read oid 30 snap -1 2024-09-11T01:56:01.078 INFO:tasks.rados.rados.0.smithi071.stdout:3939: expect (ObjNum 1276 snap 422 seq_num 1276) 2024-09-11T01:56:01.078 INFO:tasks.rados.rados.0.smithi071.stdout:3940: write (excl) oid 36 current snap is 446 2024-09-11T01:56:01.078 INFO:tasks.rados.rados.0.smithi071.stdout:3940: seq_num 1351 ranges {706257=655880,1890820=504465} 2024-09-11T01:56:01.085 INFO:tasks.rados.rados.0.smithi071.stdout:3940: writing smithi07134777-36 from 706257 to 1362137 tid 1 2024-09-11T01:56:01.091 INFO:tasks.rados.rados.0.smithi071.stdout:3940: writing smithi07134777-36 from 1890820 to 2395285 tid 2 2024-09-11T01:56:01.091 INFO:tasks.rados.rados.0.smithi071.stdout:3941: rollback oid 47 current snap is 446 2024-09-11T01:56:01.092 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 47 to 438 2024-09-11T01:56:01.092 INFO:tasks.rados.rados.0.smithi071.stdout:3938: finishing rollback tid 0 to smithi07134777-15 2024-09-11T01:56:01.092 INFO:tasks.rados.rados.0.smithi071.stdout:3938: finishing rollback tid 1 to smithi07134777-15 2024-09-11T01:56:01.092 INFO:tasks.rados.rados.0.smithi071.stdout:3938: finishing rollback tid 2 to smithi07134777-15 2024-09-11T01:56:01.092 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 3193 (ObjNum 1017 snap 322 seq_num 1017) dirty exists 2024-09-11T01:56:01.092 INFO:tasks.rados.rados.0.smithi071.stdout:3939: expect (ObjNum 1276 snap 422 seq_num 1276) 2024-09-11T01:56:01.104 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1299479 2024-09-11T01:56:01.147 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 247875 2024-09-11T01:56:01.207 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:56:01.269 INFO:tasks.rados.rados.0.smithi071.stdout:3940: finishing write tid 1 to smithi07134777-36 2024-09-11T01:56:01.269 INFO:tasks.rados.rados.0.smithi071.stdout:3940: finishing write tid 2 to smithi07134777-36 2024-09-11T01:56:01.269 INFO:tasks.rados.rados.0.smithi071.stdout:3941: finishing rollback tid 0 to smithi07134777-47 2024-09-11T01:56:01.269 INFO:tasks.rados.rados.0.smithi071.stdout:3941: finishing rollback tid 1 to smithi07134777-47 2024-09-11T01:56:01.269 INFO:tasks.rados.rados.0.smithi071.stdout:3940: finishing write tid 3 to smithi07134777-36 2024-09-11T01:56:01.269 INFO:tasks.rados.rados.0.smithi071.stdout:3941: finishing rollback tid 2 to smithi07134777-47 2024-09-11T01:56:01.269 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 2595 (ObjNum 1300 snap 426 seq_num 1300) dirty exists 2024-09-11T01:56:01.269 INFO:tasks.rados.rados.0.smithi071.stdout:3940: finishing write tid 4 to smithi07134777-36 2024-09-11T01:56:01.269 INFO:tasks.rados.rados.0.smithi071.stdout:3940: oid 36 updating version 0 to 2588 2024-09-11T01:56:01.270 INFO:tasks.rados.rados.0.smithi071.stdout:3940: oid 36 version 2588 is already newer than 2586 2024-09-11T01:56:01.270 INFO:tasks.rados.rados.0.smithi071.stdout:3940: oid 36 version 2588 is already newer than 2587 2024-09-11T01:56:01.270 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 2588 (ObjNum 1351 snap 446 seq_num 1351) dirty exists 2024-09-11T01:56:01.270 INFO:tasks.rados.rados.0.smithi071.stdout:3940: left oid 36 (ObjNum 1351 snap 446 seq_num 1351) 2024-09-11T01:56:01.270 INFO:tasks.rados.rados.0.smithi071.stdout:3938: done (3 left) 2024-09-11T01:56:01.270 INFO:tasks.rados.rados.0.smithi071.stdout:3939: done (2 left) 2024-09-11T01:56:01.270 INFO:tasks.rados.rados.0.smithi071.stdout:3940: done (1 left) 2024-09-11T01:56:01.270 INFO:tasks.rados.rados.0.smithi071.stdout:3941: done (0 left) 2024-09-11T01:56:01.270 INFO:tasks.rados.rados.0.smithi071.stdout:3942: snap_create 2024-09-11T01:56:02.079 INFO:tasks.rados.rados.0.smithi071.stdout:3942: done (0 left) 2024-09-11T01:56:02.079 INFO:tasks.rados.rados.0.smithi071.stdout:3943: write (excl) oid 10 current snap is 447 2024-09-11T01:56:02.079 INFO:tasks.rados.rados.0.smithi071.stdout:3943: seq_num 1352 ranges {723064=731129,1949302=764476} 2024-09-11T01:56:02.086 INFO:tasks.rados.rados.0.smithi071.stdout:3943: writing smithi07134777-10 from 723064 to 1454193 tid 1 2024-09-11T01:56:02.094 INFO:tasks.rados.rados.0.smithi071.stdout:3943: writing smithi07134777-10 from 1949302 to 2713778 tid 2 2024-09-11T01:56:02.094 INFO:tasks.rados.rados.0.smithi071.stdout:3944: write oid 15 current snap is 447 2024-09-11T01:56:02.094 INFO:tasks.rados.rados.0.smithi071.stdout:3944: seq_num 1353 ranges {739871=406378,1607784=508134} 2024-09-11T01:56:02.100 INFO:tasks.rados.rados.0.smithi071.stdout:3944: writing smithi07134777-15 from 739871 to 1146249 tid 1 2024-09-11T01:56:02.106 INFO:tasks.rados.rados.0.smithi071.stdout:3944: writing smithi07134777-15 from 1607784 to 2115918 tid 2 2024-09-11T01:56:02.106 INFO:tasks.rados.rados.0.smithi071.stdout:3943: finishing write tid 1 to smithi07134777-10 2024-09-11T01:56:02.106 INFO:tasks.rados.rados.0.smithi071.stdout:3943: finishing write tid 2 to smithi07134777-10 2024-09-11T01:56:02.107 INFO:tasks.rados.rados.0.smithi071.stdout:3943: finishing write tid 3 to smithi07134777-10 2024-09-11T01:56:02.107 INFO:tasks.rados.rados.0.smithi071.stdout:3945: read oid 16 snap 438 2024-09-11T01:56:02.107 INFO:tasks.rados.rados.0.smithi071.stdout:3945: expect (ObjNum 1299 snap 426 seq_num 1299) 2024-09-11T01:56:02.107 INFO:tasks.rados.rados.0.smithi071.stdout:3946: snap_create 2024-09-11T01:56:02.107 INFO:tasks.rados.rados.0.smithi071.stdout:3943: finishing write tid 4 to smithi07134777-10 2024-09-11T01:56:02.107 INFO:tasks.rados.rados.0.smithi071.stdout:3943: oid 10 updating version 0 to 2544 2024-09-11T01:56:02.107 INFO:tasks.rados.rados.0.smithi071.stdout:3943: oid 10 version 2544 is already newer than 2543 2024-09-11T01:56:02.107 INFO:tasks.rados.rados.0.smithi071.stdout:3943: oid 10 version 2544 is already newer than 2542 2024-09-11T01:56:02.107 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 2544 (ObjNum 1352 snap 447 seq_num 1352) dirty exists 2024-09-11T01:56:02.107 INFO:tasks.rados.rados.0.smithi071.stdout:3943: left oid 10 (ObjNum 1352 snap 447 seq_num 1352) 2024-09-11T01:56:02.111 INFO:tasks.rados.rados.0.smithi071.stdout:3944: finishing write tid 1 to smithi07134777-15 2024-09-11T01:56:02.111 INFO:tasks.rados.rados.0.smithi071.stdout:3944: finishing write tid 2 to smithi07134777-15 2024-09-11T01:56:02.112 INFO:tasks.rados.rados.0.smithi071.stdout:3944: finishing write tid 3 to smithi07134777-15 2024-09-11T01:56:02.112 INFO:tasks.rados.rados.0.smithi071.stdout:3944: finishing write tid 4 to smithi07134777-15 2024-09-11T01:56:02.112 INFO:tasks.rados.rados.0.smithi071.stdout:3944: oid 15 updating version 0 to 3196 2024-09-11T01:56:02.112 INFO:tasks.rados.rados.0.smithi071.stdout:3944: oid 15 version 3196 is already newer than 3195 2024-09-11T01:56:02.112 INFO:tasks.rados.rados.0.smithi071.stdout:3944: oid 15 updating version 3196 to 3197 2024-09-11T01:56:02.112 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 3197 (ObjNum 1353 snap 447 seq_num 1353) dirty exists 2024-09-11T01:56:02.112 INFO:tasks.rados.rados.0.smithi071.stdout:3944: left oid 15 (ObjNum 1353 snap 447 seq_num 1353) 2024-09-11T01:56:02.123 INFO:tasks.rados.rados.0.smithi071.stdout:3945: expect (ObjNum 1299 snap 426 seq_num 1299) 2024-09-11T01:56:02.123 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:56:02.286 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1086093 2024-09-11T01:56:03.083 INFO:tasks.rados.rados.0.smithi071.stdout:3943: done (3 left) 2024-09-11T01:56:03.083 INFO:tasks.rados.rados.0.smithi071.stdout:3944: done (2 left) 2024-09-11T01:56:03.083 INFO:tasks.rados.rados.0.smithi071.stdout:3945: done (1 left) 2024-09-11T01:56:03.083 INFO:tasks.rados.rados.0.smithi071.stdout:3946: done (0 left) 2024-09-11T01:56:03.083 INFO:tasks.rados.rados.0.smithi071.stdout:3947: write (excl) oid 42 current snap is 448 2024-09-11T01:56:03.083 INFO:tasks.rados.rados.0.smithi071.stdout:3947: seq_num 1354 ranges {756678=597980,1782619=768145} 2024-09-11T01:56:03.090 INFO:tasks.rados.rados.0.smithi071.stdout:3947: writing smithi07134777-42 from 756678 to 1354658 tid 1 2024-09-11T01:56:03.097 INFO:tasks.rados.rados.0.smithi071.stdout:3947: writing smithi07134777-42 from 1782619 to 2550764 tid 2 2024-09-11T01:56:03.097 INFO:tasks.rados.rados.0.smithi071.stdout:3948: snap_create 2024-09-11T01:56:03.097 INFO:tasks.rados.rados.0.smithi071.stdout:3947: finishing write tid 1 to smithi07134777-42 2024-09-11T01:56:03.101 INFO:tasks.rados.rados.0.smithi071.stdout:3947: finishing write tid 2 to smithi07134777-42 2024-09-11T01:56:03.103 INFO:tasks.rados.rados.0.smithi071.stdout:3947: finishing write tid 3 to smithi07134777-42 2024-09-11T01:56:03.103 INFO:tasks.rados.rados.0.smithi071.stdout:3947: finishing write tid 4 to smithi07134777-42 2024-09-11T01:56:03.103 INFO:tasks.rados.rados.0.smithi071.stdout:3947: oid 42 updating version 0 to 2590 2024-09-11T01:56:03.104 INFO:tasks.rados.rados.0.smithi071.stdout:3947: oid 42 updating version 2590 to 2591 2024-09-11T01:56:03.104 INFO:tasks.rados.rados.0.smithi071.stdout:3947: oid 42 version 2591 is already newer than 2589 2024-09-11T01:56:03.104 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 2591 (ObjNum 1354 snap 448 seq_num 1354) dirty exists 2024-09-11T01:56:03.104 INFO:tasks.rados.rados.0.smithi071.stdout:3947: left oid 42 (ObjNum 1354 snap 448 seq_num 1354) 2024-09-11T01:56:04.098 INFO:tasks.rados.rados.0.smithi071.stdout:3947: done (1 left) 2024-09-11T01:56:04.098 INFO:tasks.rados.rados.0.smithi071.stdout:3948: done (0 left) 2024-09-11T01:56:04.098 INFO:tasks.rados.rados.0.smithi071.stdout:3949: read oid 12 snap -1 2024-09-11T01:56:04.098 INFO:tasks.rados.rados.0.smithi071.stdout:3949: expect (ObjNum 1208 snap 393 seq_num 1208) 2024-09-11T01:56:04.098 INFO:tasks.rados.rados.0.smithi071.stdout:3950: rollback oid 47 current snap is 449 2024-09-11T01:56:04.098 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 47 to 448 2024-09-11T01:56:04.098 INFO:tasks.rados.rados.0.smithi071.stdout:3951: snap_remove snap 402 2024-09-11T01:56:04.947 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-11T01:56:04.947 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-11T01:56:04.947 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-11T01:56:04.950 INFO:tasks.ceph.mon.c:Started 2024-09-11T01:56:04.950 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:56:04.950 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:56:05.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.063+0000 7f37590f7640 10 monclient: get_monmap_and_config 2024-09-11T01:56:05.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.063+0000 7f37590f7640 10 monclient: build_initial_monmap 2024-09-11T01:56:05.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.063+0000 7f37590f7640 10 monclient: monmap: 2024-09-11T01:56:05.064 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:05.064 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:05.065 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:56:05.063800+0000 2024-09-11T01:56:05.065 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:56:05.063800+0000 2024-09-11T01:56:05.065 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:05.065 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:05.065 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:56:05.065 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:56:05.065 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:56:05.065 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:56:05.065 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:56:05.065 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:05.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.063+0000 7f37590f7640 1 Processor -- start 2024-09-11T01:56:05.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.064+0000 7f37590f7640 1 -- start start 2024-09-11T01:56:05.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.064+0000 7f37590f7640 10 monclient: init 2024-09-11T01:56:05.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.064+0000 7f37590f7640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:05.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.064+0000 7f37590f7640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-11T01:56:05.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.064+0000 7f37590f7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3754138230 addr v1:172.21.15.12:6789/0 2024-09-11T01:56:05.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.064+0000 7f37590f7640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f375409d030 0x7f375413d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:05.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.064+0000 7f37590f7640 10 monclient(hunting): picked mon.noname-c con 0x7f375409d030 addr v2:172.21.15.71:3300/0 2024-09-11T01:56:05.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.064+0000 7f37590f7640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f375409d460 0x7f375413f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:05.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.064+0000 7f37590f7640 10 monclient(hunting): picked mon.noname-b con 0x7f375409d460 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:05.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.064+0000 7f37590f7640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37540a6e30 con 0x7f3754138230 2024-09-11T01:56:05.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.064+0000 7f37590f7640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:05.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.064+0000 7f37590f7640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f375409c9c0 con 0x7f375409d030 2024-09-11T01:56:05.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.064+0000 7f37590f7640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:05.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.064+0000 7f37590f7640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f375409ba20 con 0x7f375409d460 2024-09-11T01:56:05.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.064+0000 7f37590f7640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:05.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.064+0000 7f37590f7640 10 monclient(hunting): authenticate will time out at 2371.506939s 2024-09-11T01:56:05.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f375359e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f375409d460 0x7f375413f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:05.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f3752d9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3754138230 0x7f3754138610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43328/0 (socket says 172.21.15.12:43328) 2024-09-11T01:56:05.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f3752d9d640 1 -- 172.21.15.12:0/1123761188 learned_addr learned my addr 172.21.15.12:0/1123761188 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:56:05.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f375359e640 10 monclient(hunting): get_auth_request con 0x7f375409d460 auth_method 0 2024-09-11T01:56:05.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f375359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:05.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f375359e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:05.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f375359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:05.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f374bfff640 1 --2- 172.21.15.12:0/1123761188 >> v2:172.21.15.71:3300/0 conn(0x7f375409d030 0x7f375413d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:05.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f374bfff640 10 monclient(hunting): get_auth_request con 0x7f375409d030 auth_method 0 2024-09-11T01:56:05.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f374bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:05.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f374bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:05.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f374bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:05.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f375359e640 1 -- 172.21.15.12:0/1123761188 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f375409d460 msgr2=0x7f375413f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:56:05.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f375359e640 1 -- 172.21.15.12:0/1123761188 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f375409d460 msgr2=0x7f375413f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:56:05.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f375359e640 1 --2- 172.21.15.12:0/1123761188 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f375409d460 0x7f375413f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:56:05.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f374bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:05.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f374bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:05.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f374bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:05.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f375359e640 1 --2- 172.21.15.12:0/1123761188 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f375409d460 0x7f375413f5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:56:05.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f374bfff640 10 monclient(hunting): handle_auth_done global_id 274333 payload 274 2024-09-11T01:56:05.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f374bfff640 1 -- 172.21.15.12:0/1123761188 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f375409d460 msgr2=0x7f375413f5e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:56:05.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f374bfff640 1 --2- 172.21.15.12:0/1123761188 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f375409d460 0x7f375413f5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:05.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f374bfff640 1 -- 172.21.15.12:0/1123761188 >> v1:172.21.15.12:6789/0 conn(0x7f3754138230 legacy=0x7f3754138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:05.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f374bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:56:05.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f374bfff640 1 monclient: found mon.noname-c 2024-09-11T01:56:05.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f374bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:05.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f374bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:56:05.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.065+0000 7f374bfff640 1 -- 172.21.15.12:0/1123761188 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37540a66a0 con 0x7f375409d030 2024-09-11T01:56:05.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.069+0000 7f374bfff640 1 -- 172.21.15.12:0/1123761188 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f375409d030 msgr2=0x7f375413d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:05.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.069+0000 7f374bfff640 1 --2- 172.21.15.12:0/1123761188 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f375409d030 0x7f375413d090 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f375409dae0 tx=0x7f373c029f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a22ab99cbd7d22a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:05.071 INFO:tasks.rados.rados.0.smithi071.stdout:3951: done (2 left) 2024-09-11T01:56:05.072 INFO:tasks.rados.rados.0.smithi071.stdout:3952: rollback oid 2 current snap is 449 2024-09-11T01:56:05.072 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 2 to 444 2024-09-11T01:56:05.072 INFO:tasks.rados.rados.0.smithi071.stdout:3950: finishing rollback tid 0 to smithi07134777-47 2024-09-11T01:56:05.072 INFO:tasks.rados.rados.0.smithi071.stdout:3950: finishing rollback tid 1 to smithi07134777-47 2024-09-11T01:56:05.072 INFO:tasks.rados.rados.0.smithi071.stdout:3950: finishing rollback tid 2 to smithi07134777-47 2024-09-11T01:56:05.072 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 2598 (ObjNum 1300 snap 426 seq_num 1300) dirty exists 2024-09-11T01:56:05.072 INFO:tasks.rados.rados.0.smithi071.stdout:3949: expect (ObjNum 1208 snap 393 seq_num 1208) 2024-09-11T01:56:05.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.072+0000 7f375259c640 1 -- 172.21.15.12:0/1123761188 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373c037070 con 0x7f375409d030 2024-09-11T01:56:05.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.072+0000 7f375259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:05.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.072+0000 7f375259c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:56:05.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.072+0000 7f375259c640 10 monclient: dump: 2024-09-11T01:56:05.075 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:05.075 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:05.075 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:05.075 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:05.075 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:05.075 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:05.075 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:05.075 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:05.075 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:05.076 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:05.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.073+0000 7f375259c640 10 monclient: _finish_auth 0 2024-09-11T01:56:05.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.073+0000 7f375259c640 10 monclient: _check_auth_tickets 2024-09-11T01:56:05.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.073+0000 7f375259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:05.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.073+0000 7f375259c640 1 -- 172.21.15.12:0/1123761188 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f373c032440 con 0x7f375409d030 2024-09-11T01:56:05.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.073+0000 7f375259c640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:05.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.073+0000 7f375259c640 1 -- 172.21.15.12:0/1123761188 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373c032810 con 0x7f375409d030 2024-09-11T01:56:05.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.073+0000 7f375259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:05.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.073+0000 7f375259c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:56:05.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.073+0000 7f375259c640 10 monclient: dump: 2024-09-11T01:56:05.076 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:05.076 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:05.077 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:05.077 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:05.077 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:05.077 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:05.077 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:05.077 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:05.077 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:05.077 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:05.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.073+0000 7f37590f7640 5 monclient: authenticate success, global_id 274333 2024-09-11T01:56:05.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.073+0000 7f37590f7640 10 monclient: get_monmap_and_config success 2024-09-11T01:56:05.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.073+0000 7f37590f7640 4 set_mon_vals no callback set 2024-09-11T01:56:05.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.074+0000 7f37590f7640 10 monclient: shutdown 2024-09-11T01:56:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.074+0000 7f37590f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.074+0000 7f37590f7640 1 -- 172.21.15.12:0/1123761188 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f375409d030 msgr2=0x7f375413d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.074+0000 7f37590f7640 1 --2- 172.21.15.12:0/1123761188 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f375409d030 0x7f375413d090 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f375409dae0 tx=0x7f373c029f90 comp rx=0 tx=0).stop 2024-09-11T01:56:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.075+0000 7f37590f7640 1 -- 172.21.15.12:0/1123761188 shutdown_connections 2024-09-11T01:56:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.075+0000 7f37590f7640 1 --2- 172.21.15.12:0/1123761188 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f375409d460 0x7f375413f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.075+0000 7f37590f7640 1 --2- 172.21.15.12:0/1123761188 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f375409d030 0x7f375413d090 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.075+0000 7f37590f7640 1 -- 172.21.15.12:0/1123761188 >> 172.21.15.12:0/1123761188 conn(0x7f37540a7120 msgr2=0x7f37540a7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.075+0000 7f37590f7640 1 -- 172.21.15.12:0/1123761188 shutdown_connections 2024-09-11T01:56:05.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.076+0000 7f37590f7640 1 -- 172.21.15.12:0/1123761188 wait complete. 2024-09-11T01:56:05.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.077+0000 7f37590f7640 10 monclient: build_initial_monmap 2024-09-11T01:56:05.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.077+0000 7f37590f7640 10 monclient: monmap: 2024-09-11T01:56:05.080 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:05.080 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:56:05.080 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:56:05.080 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:56:05.080 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:05.080 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:05.080 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:56:05.080 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:56:05.080 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:56:05.080 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:05.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.077+0000 7f37590f7640 1 Processor -- start 2024-09-11T01:56:05.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.079+0000 7f37590f7640 1 -- start start 2024-09-11T01:56:05.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.079+0000 7f37590f7640 10 monclient: init 2024-09-11T01:56:05.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.080+0000 7f37590f7640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:05.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.080+0000 7f37590f7640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:56:05.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.080+0000 7f37590f7640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f375409d030 0x7f3754131c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:05.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.080+0000 7f37590f7640 10 monclient(hunting): picked mon.noname-c con 0x7f375409d030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:05.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.080+0000 7f37590f7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f375409d460 0x7f37541321c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:05.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.080+0000 7f37590f7640 10 monclient(hunting): picked mon.noname-a con 0x7f375409d460 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:56:05.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.080+0000 7f37590f7640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3754138230 0x7f3754132700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:05.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.080+0000 7f37590f7640 10 monclient(hunting): picked mon.noname-b con 0x7f3754138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:56:05.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.080+0000 7f37590f7640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:05.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.080+0000 7f37590f7640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f375413fca0 con 0x7f375409d460 2024-09-11T01:56:05.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.080+0000 7f37590f7640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:05.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.080+0000 7f37590f7640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f375413fb20 con 0x7f3754138230 2024-09-11T01:56:05.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.080+0000 7f37590f7640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:05.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.080+0000 7f37590f7640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f375413fe20 con 0x7f375409d030 2024-09-11T01:56:05.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.080+0000 7f37590f7640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:05.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.080+0000 7f37590f7640 10 monclient(hunting): authenticate will time out at 2371.522640s 2024-09-11T01:56:05.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.080+0000 7f375359e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3754138230 0x7f3754132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:05.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.080+0000 7f375359e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3754138230 0x7f3754132700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.12:59222/0 (socket says 172.21.15.12:59222) 2024-09-11T01:56:05.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.080+0000 7f375359e640 1 -- 172.21.15.12:0/3808364662 learned_addr learned my addr 172.21.15.12:0/3808364662 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:05.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.080+0000 7f375359e640 10 monclient(hunting): get_auth_request con 0x7f3754138230 auth_method 0 2024-09-11T01:56:05.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.080+0000 7f375359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:05.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.080+0000 7f375359e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:05.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.080+0000 7f375359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:05.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.080+0000 7f374bfff640 1 --2- 172.21.15.12:0/3808364662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f375409d460 0x7f37541321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:05.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.081+0000 7f374bfff640 10 monclient(hunting): get_auth_request con 0x7f375409d460 auth_method 0 2024-09-11T01:56:05.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.081+0000 7f374bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:05.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.081+0000 7f374bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:05.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.081+0000 7f374bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:05.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.081+0000 7f375359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:05.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.081+0000 7f375359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:05.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.081+0000 7f375359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:05.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.081+0000 7f374bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:05.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.081+0000 7f374bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:05.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.081+0000 7f374bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:05.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.081+0000 7f3752d9d640 1 --2- 172.21.15.12:0/3808364662 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f375409d030 0x7f3754131c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:05.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.081+0000 7f375359e640 10 monclient(hunting): handle_auth_done global_id 274336 payload 824 2024-09-11T01:56:05.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.081+0000 7f375359e640 1 -- 172.21.15.12:0/3808364662 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f375409d030 msgr2=0x7f3754131c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:05.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.081+0000 7f375359e640 1 --2- 172.21.15.12:0/3808364662 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f375409d030 0x7f3754131c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:05.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.081+0000 7f375359e640 1 -- 172.21.15.12:0/3808364662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f375409d460 msgr2=0x7f37541321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:05.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.081+0000 7f375359e640 1 --2- 172.21.15.12:0/3808364662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f375409d460 0x7f37541321c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:05.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.081+0000 7f375359e640 10 monclient: _finish_hunting 0 2024-09-11T01:56:05.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.081+0000 7f375359e640 1 monclient: found mon.noname-b 2024-09-11T01:56:05.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.081+0000 7f375359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:05.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.081+0000 7f375359e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-09-11T01:56:05.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.081+0000 7f375359e640 1 -- 172.21.15.12:0/3808364662 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3754135090 con 0x7f3754138230 2024-09-11T01:56:05.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.081+0000 7f3752d9d640 10 monclient: get_auth_request con 0x7f375409d030 auth_method 0 2024-09-11T01:56:05.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.081+0000 7f3752d9d640 1 --2- 172.21.15.12:0/3808364662 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f375409d030 0x7f3754131c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:56:05.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.081+0000 7f374bfff640 1 --2- 172.21.15.12:0/3808364662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f375409d460 0x7f37541321c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:56:05.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.083+0000 7f375359e640 1 -- 172.21.15.12:0/3808364662 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3754138230 msgr2=0x7f3754132700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:05.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.083+0000 7f375359e640 1 --2- 172.21.15.12:0/3808364662 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3754138230 0x7f3754132700 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f374c007d40 tx=0x7f374c00e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:05.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.084+0000 7f374b7fe640 1 -- 172.21.15.12:0/3808364662 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374c014620 con 0x7f3754138230 2024-09-11T01:56:05.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.084+0000 7f374b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:05.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.084+0000 7f374b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:56:05.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.084+0000 7f374b7fe640 10 monclient: dump: 2024-09-11T01:56:05.088 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:05.088 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:05.089 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:05.089 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:05.089 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:05.089 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:05.089 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:05.089 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:05.089 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:05.089 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:05.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.084+0000 7f374b7fe640 10 monclient: _finish_auth 0 2024-09-11T01:56:05.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.084+0000 7f374b7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:56:05.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.084+0000 7f374b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:05.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.084+0000 7f374b7fe640 1 -- 172.21.15.12:0/3808364662 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f374c019070 con 0x7f3754138230 2024-09-11T01:56:05.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.084+0000 7f37590f7640 5 monclient: authenticate success, global_id 274336 2024-09-11T01:56:05.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.084+0000 7f374b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:05.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.084+0000 7f374b7fe640 1 -- 172.21.15.12:0/3808364662 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374c014a00 con 0x7f3754138230 2024-09-11T01:56:05.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.084+0000 7f374b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:05.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.084+0000 7f374b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:56:05.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.084+0000 7f374b7fe640 10 monclient: dump: 2024-09-11T01:56:05.090 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:05.090 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:05.090 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:05.090 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:05.090 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:05.090 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:05.090 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:05.090 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:05.090 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:05.090 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:05.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.084+0000 7f37590f7640 10 monclient: _renew_subs 2024-09-11T01:56:05.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.084+0000 7f37590f7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:56:05.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.084+0000 7f37590f7640 1 -- 172.21.15.12:0/3808364662 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3754135820 con 0x7f3754138230 2024-09-11T01:56:05.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.084+0000 7f3751d9b640 4 set_mon_vals no callback set 2024-09-11T01:56:05.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.084+0000 7f37590f7640 10 monclient: _renew_subs 2024-09-11T01:56:05.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.084+0000 7f37590f7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:56:05.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.084+0000 7f37590f7640 1 -- 172.21.15.12:0/3808364662 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37540a14a0 con 0x7f3754138230 2024-09-11T01:56:05.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.085+0000 7f37590f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:05.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.085+0000 7f37590f7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:05.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.085+0000 7f37590f7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:56:05.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.085+0000 7f37590f7640 1 -- 172.21.15.12:0/3808364662 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37540a1810 con 0x7f3754138230 2024-09-11T01:56:05.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.085+0000 7f374b7fe640 1 -- 172.21.15.12:0/3808364662 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f374c014c20 con 0x7f3754138230 2024-09-11T01:56:05.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.086+0000 7f374b7fe640 1 --2- 172.21.15.12:0/3808364662 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f373403e210 0x7f37340406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:05.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.086+0000 7f374b7fe640 1 -- 172.21.15.12:0/3808364662 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(911..911 src has 676..911) ==== 4843+0+0 (secure 0 0 0) 0x7f374c0535a0 con 0x7f3754138230 2024-09-11T01:56:05.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.086+0000 7f3752d9d640 1 --2- 172.21.15.12:0/3808364662 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f373403e210 0x7f37340406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:05.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.086+0000 7f3752d9d640 10 monclient: get_auth_request con 0x7f373403e210 auth_method 0 2024-09-11T01:56:05.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.089+0000 7f3752d9d640 1 --2- 172.21.15.12:0/3808364662 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f373403e210 0x7f37340406d0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f37440050b0 tx=0x7f3744009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:05.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.095+0000 7f374b7fe640 1 -- 172.21.15.12:0/3808364662 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f374c012cf0 con 0x7f3754138230 2024-09-11T01:56:05.096 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.095+0000 7f374b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:05.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.095+0000 7f374b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:56:05.113 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:56:05.190 INFO:tasks.rados.rados.0.smithi071.stdout:3952: finishing rollback tid 0 to smithi07134777-2 2024-09-11T01:56:05.190 INFO:tasks.rados.rados.0.smithi071.stdout:3952: finishing rollback tid 1 to smithi07134777-2 2024-09-11T01:56:05.190 INFO:tasks.rados.rados.0.smithi071.stdout:3952: finishing rollback tid 2 to smithi07134777-2 2024-09-11T01:56:05.190 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 2594 (ObjNum 1328 snap 432 seq_num 1328) dirty exists 2024-09-11T01:56:05.190 INFO:tasks.rados.rados.0.smithi071.stdout:3949: done (2 left) 2024-09-11T01:56:05.190 INFO:tasks.rados.rados.0.smithi071.stdout:3950: done (1 left) 2024-09-11T01:56:05.190 INFO:tasks.rados.rados.0.smithi071.stdout:3952: done (0 left) 2024-09-11T01:56:05.190 INFO:tasks.rados.rados.0.smithi071.stdout:3953: copy_from oid 38 from oid 43 current snap is 449 2024-09-11T01:56:05.191 INFO:tasks.rados.rados.0.smithi071.stdout:3954: write (excl) oid 32 current snap is 449 2024-09-11T01:56:05.191 INFO:tasks.rados.rados.0.smithi071.stdout:3954: seq_num 1355 ranges {773485=673229,2124748=511803} 2024-09-11T01:56:05.198 INFO:tasks.rados.rados.0.smithi071.stdout:3954: writing smithi07134777-32 from 773485 to 1446714 tid 1 2024-09-11T01:56:05.205 INFO:tasks.rados.rados.0.smithi071.stdout:3954: writing smithi07134777-32 from 2124748 to 2636551 tid 2 2024-09-11T01:56:05.205 INFO:tasks.rados.rados.0.smithi071.stdout:3955: snap_remove snap 442 2024-09-11T01:56:05.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.355+0000 7f37590f7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:56:05.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.355+0000 7f37590f7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:56:05.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.355+0000 7f37590f7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-09-11T01:56:05.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.355+0000 7f37590f7640 1 -- 172.21.15.12:0/3808364662 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3754138610 con 0x7f3754138230 2024-09-11T01:56:05.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.356+0000 7f374b7fe640 1 -- 172.21.15.12:0/3808364662 <== mon.1 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f374c01b310 con 0x7f3754138230 2024-09-11T01:56:05.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.356+0000 7f374b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:56:05.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.356+0000 7f374b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:56:05.358 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":244,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:56:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.358+0000 7f37590f7640 1 -- 172.21.15.12:0/3808364662 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f373403e210 msgr2=0x7f37340406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.358+0000 7f37590f7640 1 --2- 172.21.15.12:0/3808364662 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f373403e210 0x7f37340406d0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f37440050b0 tx=0x7f3744009040 comp rx=0 tx=0).stop 2024-09-11T01:56:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.358+0000 7f37590f7640 10 monclient: shutdown 2024-09-11T01:56:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.358+0000 7f37590f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.358+0000 7f37590f7640 1 -- 172.21.15.12:0/3808364662 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3754138230 msgr2=0x7f3754132700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.358+0000 7f37590f7640 1 --2- 172.21.15.12:0/3808364662 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3754138230 0x7f3754132700 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f374c007d40 tx=0x7f374c00e4b0 comp rx=0 tx=0).stop 2024-09-11T01:56:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.358+0000 7f37590f7640 1 -- 172.21.15.12:0/3808364662 shutdown_connections 2024-09-11T01:56:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.358+0000 7f37590f7640 1 --2- 172.21.15.12:0/3808364662 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3754138230 0x7f3754132700 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.358+0000 7f37590f7640 1 --2- 172.21.15.12:0/3808364662 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f373403e210 0x7f37340406d0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:05.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.358+0000 7f37590f7640 1 --2- 172.21.15.12:0/3808364662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f375409d460 0x7f37541321c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:05.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.358+0000 7f37590f7640 1 --2- 172.21.15.12:0/3808364662 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f375409d030 0x7f3754131c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:05.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.358+0000 7f37590f7640 1 -- 172.21.15.12:0/3808364662 >> 172.21.15.12:0/3808364662 conn(0x7f37540a7120 msgr2=0x7f37540a0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:05.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.358+0000 7f37590f7640 1 -- 172.21.15.12:0/3808364662 shutdown_connections 2024-09-11T01:56:05.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:05.358+0000 7f37590f7640 1 -- 172.21.15.12:0/3808364662 wait complete. 2024-09-11T01:56:06.105 INFO:tasks.rados.rados.0.smithi071.stdout:3955: done (2 left) 2024-09-11T01:56:06.105 INFO:tasks.rados.rados.0.smithi071.stdout:3956: snap_remove snap 325 2024-09-11T01:56:07.108 INFO:tasks.rados.rados.0.smithi071.stdout:3956: done (2 left) 2024-09-11T01:56:07.108 INFO:tasks.rados.rados.0.smithi071.stdout:3957: snap_remove snap 414 2024-09-11T01:56:08.371 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:56:08.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.482+0000 7fb9da789640 10 monclient: get_monmap_and_config 2024-09-11T01:56:08.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.482+0000 7fb9da789640 10 monclient: build_initial_monmap 2024-09-11T01:56:08.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.483+0000 7fb9da789640 10 monclient: monmap: 2024-09-11T01:56:08.485 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:08.485 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:08.485 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:56:08.483735+0000 2024-09-11T01:56:08.485 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:56:08.483735+0000 2024-09-11T01:56:08.485 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:08.485 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:08.485 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:56:08.485 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:56:08.485 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:56:08.485 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:56:08.486 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:56:08.486 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:08.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.483+0000 7fb9da789640 1 Processor -- start 2024-09-11T01:56:08.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.483+0000 7fb9da789640 1 -- start start 2024-09-11T01:56:08.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.483+0000 7fb9da789640 10 monclient: init 2024-09-11T01:56:08.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9da789640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:08.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9da789640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-11T01:56:08.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9da789640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb9d4132b30 0x7fb9d4132f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:08.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9da789640 10 monclient(hunting): picked mon.noname-c con 0x7fb9d4132b30 addr v2:172.21.15.71:3300/0 2024-09-11T01:56:08.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9da789640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb9d4070310 addr v1:172.21.15.12:6789/0 2024-09-11T01:56:08.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9da789640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9d4136160 0x7fb9d413ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:08.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9da789640 10 monclient(hunting): picked mon.noname-b con 0x7fb9d4136160 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:08.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9da789640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9d40738f0 con 0x7fb9d4070310 2024-09-11T01:56:08.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9da789640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:08.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9da789640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fb9d4073610 con 0x7fb9d4132b30 2024-09-11T01:56:08.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9da789640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:08.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9da789640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb9d4073bd0 con 0x7fb9d4136160 2024-09-11T01:56:08.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9da789640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:08.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9da789640 10 monclient(hunting): authenticate will time out at 2374.926525s 2024-09-11T01:56:08.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9d8d26640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9d4136160 0x7fb9d413ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:08.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9d37fe640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb9d4070310 0x7fb9d4135a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:41950/0 (socket says 172.21.15.12:41950) 2024-09-11T01:56:08.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9d37fe640 1 -- 172.21.15.12:0/878896858 learned_addr learned my addr 172.21.15.12:0/878896858 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:56:08.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9d8d26640 10 monclient(hunting): get_auth_request con 0x7fb9d4136160 auth_method 0 2024-09-11T01:56:08.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9d8d26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:08.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9d8d26640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:08.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9d8d26640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:08.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9d8d26640 1 -- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9d4136160 msgr2=0x7fb9d413ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:56:08.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9d8d26640 1 -- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9d4136160 msgr2=0x7fb9d413ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:56:08.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9d8d26640 1 --2- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9d4136160 0x7fb9d413ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:56:08.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9d8d26640 1 --2- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9d4136160 0x7fb9d413ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:56:08.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9d3fff640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.71:3300/0 conn(0x7fb9d4132b30 0x7fb9d4132f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:08.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9d3fff640 10 monclient(hunting): get_auth_request con 0x7fb9d4132b30 auth_method 0 2024-09-11T01:56:08.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:08.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:08.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.484+0000 7fb9d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:08.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.485+0000 7fb9d3fff640 1 -- 172.21.15.12:0/878896858 >> v2:172.21.15.71:3300/0 conn(0x7fb9d4132b30 msgr2=0x7fb9d4132f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:56:08.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.485+0000 7fb9d3fff640 1 -- 172.21.15.12:0/878896858 >> v2:172.21.15.71:3300/0 conn(0x7fb9d4132b30 msgr2=0x7fb9d4132f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:56:08.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.485+0000 7fb9d3fff640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.71:3300/0 conn(0x7fb9d4132b30 0x7fb9d4132f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:56:08.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.485+0000 7fb9d3fff640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.71:3300/0 conn(0x7fb9d4132b30 0x7fb9d4132f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:56:08.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.486+0000 7fb9d37fe640 1 -- 172.21.15.12:0/878896858 >> v1:172.21.15.12:6789/0 conn(0x7fb9d4070310 legacy=0x7fb9d4135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:08.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.685+0000 7fb9d8d26640 1 --2- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9d4136160 0x7fb9d413ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:08.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.685+0000 7fb9d8d26640 10 monclient(hunting): get_auth_request con 0x7fb9d4136160 auth_method 0 2024-09-11T01:56:08.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.685+0000 7fb9d8d26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:08.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.685+0000 7fb9d8d26640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:08.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.685+0000 7fb9d8d26640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:56:08.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.685+0000 7fb9d8d26640 1 -- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9d4136160 msgr2=0x7fb9d413ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:56:08.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.685+0000 7fb9d8d26640 1 -- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9d4136160 msgr2=0x7fb9d413ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:56:08.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.685+0000 7fb9d8d26640 1 --2- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9d4136160 0x7fb9d413ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:56:08.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.685+0000 7fb9d8d26640 1 --2- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9d4136160 0x7fb9d413ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:56:08.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.686+0000 7fb9d3fff640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.71:3300/0 conn(0x7fb9d4132b30 0x7fb9d4132f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:08.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.686+0000 7fb9d3fff640 10 monclient(hunting): get_auth_request con 0x7fb9d4132b30 auth_method 0 2024-09-11T01:56:08.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.686+0000 7fb9d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:08.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.686+0000 7fb9d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:08.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.686+0000 7fb9d3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:56:08.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.686+0000 7fb9d3fff640 1 -- 172.21.15.12:0/878896858 >> v2:172.21.15.71:3300/0 conn(0x7fb9d4132b30 msgr2=0x7fb9d4132f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:56:08.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.686+0000 7fb9d3fff640 1 -- 172.21.15.12:0/878896858 >> v2:172.21.15.71:3300/0 conn(0x7fb9d4132b30 msgr2=0x7fb9d4132f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:56:08.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.686+0000 7fb9d3fff640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.71:3300/0 conn(0x7fb9d4132b30 0x7fb9d4132f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:56:08.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:08.686+0000 7fb9d3fff640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.71:3300/0 conn(0x7fb9d4132b30 0x7fb9d4132f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:56:09.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.087+0000 7fb9d8d26640 1 --2- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9d4136160 0x7fb9d413ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:09.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.087+0000 7fb9d8d26640 10 monclient(hunting): get_auth_request con 0x7fb9d4136160 auth_method 0 2024-09-11T01:56:09.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.087+0000 7fb9d8d26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:09.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.087+0000 7fb9d8d26640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:09.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.087+0000 7fb9d8d26640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:56:09.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.087+0000 7fb9d8d26640 1 -- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9d4136160 msgr2=0x7fb9d413ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:56:09.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.087+0000 7fb9d8d26640 1 -- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9d4136160 msgr2=0x7fb9d413ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:56:09.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.087+0000 7fb9d8d26640 1 --2- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9d4136160 0x7fb9d413ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:56:09.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.087+0000 7fb9d8d26640 1 --2- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9d4136160 0x7fb9d413ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:56:09.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.087+0000 7fb9d3fff640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.71:3300/0 conn(0x7fb9d4132b30 0x7fb9d4132f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:09.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.087+0000 7fb9d3fff640 10 monclient(hunting): get_auth_request con 0x7fb9d4132b30 auth_method 0 2024-09-11T01:56:09.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.087+0000 7fb9d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:09.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.087+0000 7fb9d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:09.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.087+0000 7fb9d3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:56:09.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.087+0000 7fb9d3fff640 1 -- 172.21.15.12:0/878896858 >> v2:172.21.15.71:3300/0 conn(0x7fb9d4132b30 msgr2=0x7fb9d4132f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:56:09.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.087+0000 7fb9d3fff640 1 -- 172.21.15.12:0/878896858 >> v2:172.21.15.71:3300/0 conn(0x7fb9d4132b30 msgr2=0x7fb9d4132f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:56:09.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.087+0000 7fb9d3fff640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.71:3300/0 conn(0x7fb9d4132b30 0x7fb9d4132f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:56:09.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.087+0000 7fb9d3fff640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.71:3300/0 conn(0x7fb9d4132b30 0x7fb9d4132f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:56:09.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.888+0000 7fb9d8d26640 1 --2- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9d4136160 0x7fb9d413ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:09.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.888+0000 7fb9d8d26640 10 monclient(hunting): get_auth_request con 0x7fb9d4136160 auth_method 0 2024-09-11T01:56:09.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.888+0000 7fb9d8d26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:09.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.888+0000 7fb9d8d26640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.888+0000 7fb9d8d26640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:56:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.888+0000 7fb9d8d26640 1 -- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9d4136160 msgr2=0x7fb9d413ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:56:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.888+0000 7fb9d8d26640 1 -- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9d4136160 msgr2=0x7fb9d413ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:56:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.888+0000 7fb9d8d26640 1 --2- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9d4136160 0x7fb9d413ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:56:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.888+0000 7fb9d8d26640 1 --2- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9d4136160 0x7fb9d413ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:56:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.889+0000 7fb9d3fff640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.71:3300/0 conn(0x7fb9d4132b30 0x7fb9d4132f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.889+0000 7fb9d3fff640 10 monclient(hunting): get_auth_request con 0x7fb9d4132b30 auth_method 0 2024-09-11T01:56:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.889+0000 7fb9d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.889+0000 7fb9d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.889+0000 7fb9d3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:56:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.889+0000 7fb9d3fff640 1 -- 172.21.15.12:0/878896858 >> v2:172.21.15.71:3300/0 conn(0x7fb9d4132b30 msgr2=0x7fb9d4132f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:56:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.889+0000 7fb9d3fff640 1 -- 172.21.15.12:0/878896858 >> v2:172.21.15.71:3300/0 conn(0x7fb9d4132b30 msgr2=0x7fb9d4132f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:56:09.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.889+0000 7fb9d3fff640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.71:3300/0 conn(0x7fb9d4132b30 0x7fb9d4132f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:56:09.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:09.889+0000 7fb9d3fff640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.71:3300/0 conn(0x7fb9d4132b30 0x7fb9d4132f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:56:11.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.485+0000 7fb9d1ffb640 10 monclient(hunting): tick 2024-09-11T01:56:11.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.485+0000 7fb9d1ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:56:11.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.485+0000 7fb9d1ffb640 1 monclient(hunting): continuing hunt 2024-09-11T01:56:11.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.485+0000 7fb9d1ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:56:11.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.485+0000 7fb9d1ffb640 1 -- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9d4136160 msgr2=0x7fb9d413ddf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:56:11.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.485+0000 7fb9d1ffb640 1 --2- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9d4136160 0x7fb9d413ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:11.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.485+0000 7fb9d1ffb640 1 -- 172.21.15.12:0/878896858 >> v2:172.21.15.71:3300/0 conn(0x7fb9d4132b30 msgr2=0x7fb9d4132f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:56:11.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.485+0000 7fb9d1ffb640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.71:3300/0 conn(0x7fb9d4132b30 0x7fb9d4132f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:11.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.485+0000 7fb9d1ffb640 1 -- 172.21.15.12:0/878896858 >> v1:172.21.15.12:6789/0 conn(0x7fb9d4070310 legacy=0x7fb9d4135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:11.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.485+0000 7fb9d1ffb640 10 monclient: _add_conns ranks=[1,3] 2024-09-11T01:56:11.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.485+0000 7fb9d1ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb9c0003740 addr v1:172.21.15.71:6789/0 2024-09-11T01:56:11.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.485+0000 7fb9d1ffb640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.12:3300/0 conn(0x7fb9c0007470 0x7fb9c0009930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:11.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.485+0000 7fb9d1ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fb9c0007470 addr v2:172.21.15.12:3300/0 2024-09-11T01:56:11.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.485+0000 7fb9d1ffb640 1 -- 172.21.15.12:0/878896858 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9d40a43e0 con 0x7fb9c0003740 2024-09-11T01:56:11.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.485+0000 7fb9d1ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:11.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.485+0000 7fb9d1ffb640 1 -- 172.21.15.12:0/878896858 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fb9d4073610 con 0x7fb9c0007470 2024-09-11T01:56:11.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.485+0000 7fb9d1ffb640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:11.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.485+0000 7fb9d37fe640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.12:3300/0 conn(0x7fb9c0007470 0x7fb9c0009930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:11.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.485+0000 7fb9d37fe640 10 monclient(hunting): get_auth_request con 0x7fb9c0007470 auth_method 0 2024-09-11T01:56:11.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.485+0000 7fb9d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:11.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.485+0000 7fb9d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:11.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.485+0000 7fb9d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:11.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.485+0000 7fb9d37fe640 1 -- 172.21.15.12:0/878896858 >> v2:172.21.15.12:3300/0 conn(0x7fb9c0007470 msgr2=0x7fb9c0009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:56:11.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.485+0000 7fb9d37fe640 1 -- 172.21.15.12:0/878896858 >> v2:172.21.15.12:3300/0 conn(0x7fb9c0007470 msgr2=0x7fb9c0009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:56:11.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.485+0000 7fb9d37fe640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.12:3300/0 conn(0x7fb9c0007470 0x7fb9c0009930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:56:11.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.485+0000 7fb9d37fe640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.12:3300/0 conn(0x7fb9c0007470 0x7fb9c0009930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:56:11.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.487+0000 7fb9d3fff640 1 -- 172.21.15.12:0/878896858 >> v1:172.21.15.71:6789/0 conn(0x7fb9c0003740 legacy=0x7fb9c0005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:11.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.686+0000 7fb9d37fe640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.12:3300/0 conn(0x7fb9c0007470 0x7fb9c0009930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:11.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.686+0000 7fb9d37fe640 10 monclient(hunting): get_auth_request con 0x7fb9c0007470 auth_method 0 2024-09-11T01:56:11.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.686+0000 7fb9d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:11.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.686+0000 7fb9d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:11.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.686+0000 7fb9d37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:56:11.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.686+0000 7fb9d37fe640 1 -- 172.21.15.12:0/878896858 >> v2:172.21.15.12:3300/0 conn(0x7fb9c0007470 msgr2=0x7fb9c0009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:56:11.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.686+0000 7fb9d37fe640 1 -- 172.21.15.12:0/878896858 >> v2:172.21.15.12:3300/0 conn(0x7fb9c0007470 msgr2=0x7fb9c0009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:56:11.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.686+0000 7fb9d37fe640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.12:3300/0 conn(0x7fb9c0007470 0x7fb9c0009930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:56:11.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:11.687+0000 7fb9d37fe640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.12:3300/0 conn(0x7fb9c0007470 0x7fb9c0009930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:56:12.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:12.088+0000 7fb9d37fe640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.12:3300/0 conn(0x7fb9c0007470 0x7fb9c0009930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:12.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:12.088+0000 7fb9d37fe640 10 monclient(hunting): get_auth_request con 0x7fb9c0007470 auth_method 0 2024-09-11T01:56:12.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:12.088+0000 7fb9d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:12.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:12.088+0000 7fb9d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:12.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:12.088+0000 7fb9d37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:56:12.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:12.088+0000 7fb9d37fe640 1 -- 172.21.15.12:0/878896858 >> v2:172.21.15.12:3300/0 conn(0x7fb9c0007470 msgr2=0x7fb9c0009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:56:12.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:12.088+0000 7fb9d37fe640 1 -- 172.21.15.12:0/878896858 >> v2:172.21.15.12:3300/0 conn(0x7fb9c0007470 msgr2=0x7fb9c0009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:56:12.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:12.088+0000 7fb9d37fe640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.12:3300/0 conn(0x7fb9c0007470 0x7fb9c0009930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:56:12.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:12.088+0000 7fb9d37fe640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.12:3300/0 conn(0x7fb9c0007470 0x7fb9c0009930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:56:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:12.889+0000 7fb9d37fe640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.12:3300/0 conn(0x7fb9c0007470 0x7fb9c0009930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:12.889+0000 7fb9d37fe640 10 monclient(hunting): get_auth_request con 0x7fb9c0007470 auth_method 0 2024-09-11T01:56:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:12.889+0000 7fb9d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:12.889+0000 7fb9d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:12.889+0000 7fb9d37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:56:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:12.889+0000 7fb9d37fe640 1 -- 172.21.15.12:0/878896858 >> v2:172.21.15.12:3300/0 conn(0x7fb9c0007470 msgr2=0x7fb9c0009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:56:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:12.889+0000 7fb9d37fe640 1 -- 172.21.15.12:0/878896858 >> v2:172.21.15.12:3300/0 conn(0x7fb9c0007470 msgr2=0x7fb9c0009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:56:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:12.889+0000 7fb9d37fe640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.12:3300/0 conn(0x7fb9c0007470 0x7fb9c0009930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:56:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:12.889+0000 7fb9d37fe640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.12:3300/0 conn(0x7fb9c0007470 0x7fb9c0009930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:56:13.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.093+0000 7fb9d2ffd640 1 -- 172.21.15.12:0/878896858 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2696360743 0 0) 0x7fb9d40a43e0 con 0x7fb9c0003740 2024-09-11T01:56:13.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.093+0000 7fb9d2ffd640 10 monclient(hunting): _negotiate 2024-09-11T01:56:13.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.093+0000 7fb9d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:13.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.093+0000 7fb9d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:13.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.093+0000 7fb9d2ffd640 10 monclient(hunting): my global_id is 274339 2024-09-11T01:56:13.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.093+0000 7fb9d2ffd640 1 -- 172.21.15.12:0/878896858 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9bc003520 con 0x7fb9c0003740 2024-09-11T01:56:13.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.093+0000 7fb9d2ffd640 1 -- 172.21.15.12:0/878896858 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 363836213 0 0) 0x7fb9bc003520 con 0x7fb9c0003740 2024-09-11T01:56:13.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 1 -- 172.21.15.12:0/878896858 >> v2:172.21.15.12:3300/0 conn(0x7fb9c0007470 msgr2=0x7fb9c0009930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:56:13.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 1 --2- 172.21.15.12:0/878896858 >> v2:172.21.15.12:3300/0 conn(0x7fb9c0007470 0x7fb9c0009930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:13.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:56:13.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 1 monclient: found mon.noname-c-legacy 2024-09-11T01:56:13.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:13.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-09-11T01:56:13.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 1 -- 172.21.15.12:0/878896858 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9c000b000 con 0x7fb9c0003740 2024-09-11T01:56:13.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 10 monclient: _finish_auth 0 2024-09-11T01:56:13.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:56:13.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:13.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 1 -- 172.21.15.12:0/878896858 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7fb9c400d680 con 0x7fb9c0003740 2024-09-11T01:56:13.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:13.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-11T01:56:13.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 10 monclient: dump: 2024-09-11T01:56:13.099 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:13.099 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:13.099 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:13.099 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:13.099 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:13.099 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:13.099 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:13.099 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:13.100 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:13.100 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:13.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-09-11T01:56:13.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:13.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 1 -- 172.21.15.12:0/878896858 >> v1:172.21.15.71:6789/0 conn(0x7fb9c0003740 legacy=0x7fb9c0005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:13.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:56:13.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d8d26640 1 -- 172.21.15.12:0/878896858 reap_dead start 2024-09-11T01:56:13.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 1 --2- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9bc003aa0 0x7fb9bc003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:13.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 10 monclient(hunting): picked mon.a con 0x7fb9bc003aa0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:56:13.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 1 --2- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9bc004800 0x7fb9bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:13.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 10 monclient(hunting): picked mon.c con 0x7fb9bc004800 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:13.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 1 --2- 172.21.15.12:0/878896858 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9bc005410 0x7fb9bc00ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:13.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 10 monclient(hunting): picked mon.b con 0x7fb9bc005410 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:56:13.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:13.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 1 -- 172.21.15.12:0/878896858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d4073610 con 0x7fb9bc003aa0 2024-09-11T01:56:13.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:13.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 1 -- 172.21.15.12:0/878896858 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc00c320 con 0x7fb9bc005410 2024-09-11T01:56:13.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:13.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 1 -- 172.21.15.12:0/878896858 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb9bc00c520 con 0x7fb9bc004800 2024-09-11T01:56:13.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:13.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:56:13.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d3fff640 1 --2- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9bc003aa0 0x7fb9bc003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:13.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d37fe640 1 --2- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9bc004800 0x7fb9bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:13.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d3fff640 10 monclient(hunting): get_auth_request con 0x7fb9bc003aa0 auth_method 0 2024-09-11T01:56:13.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:13.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:13.102 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:56:13.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d37fe640 10 monclient(hunting): get_auth_request con 0x7fb9bc004800 auth_method 0 2024-09-11T01:56:13.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:13.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:13.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:56:13.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:13.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:13.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d8d26640 1 --2- 172.21.15.12:0/878896858 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9bc005410 0x7fb9bc00ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:13.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:56:13.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d8d26640 10 monclient(hunting): get_auth_request con 0x7fb9bc005410 auth_method 0 2024-09-11T01:56:13.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d8d26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:13.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d8d26640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:13.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d8d26640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:56:13.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:13.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:13.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:56:13.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d8d26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:13.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d8d26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:13.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d8d26640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:56:13.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d3fff640 10 monclient(hunting): handle_auth_done global_id 274339 payload 293 2024-09-11T01:56:13.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d3fff640 1 -- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9bc004800 msgr2=0x7fb9bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:13.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d3fff640 1 --2- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9bc004800 0x7fb9bc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:13.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d3fff640 1 -- 172.21.15.12:0/878896858 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9bc005410 msgr2=0x7fb9bc00ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:13.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d3fff640 1 --2- 172.21.15.12:0/878896858 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9bc005410 0x7fb9bc00ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:13.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d3fff640 10 monclient: _finish_hunting 0 2024-09-11T01:56:13.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d3fff640 1 monclient: found mon.a 2024-09-11T01:56:13.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:13.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d3fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:13.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d3fff640 1 -- 172.21.15.12:0/878896858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9bc003520 con 0x7fb9bc003aa0 2024-09-11T01:56:13.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d3fff640 10 monclient: _finish_auth 0 2024-09-11T01:56:13.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d3fff640 10 monclient: _check_auth_tickets 2024-09-11T01:56:13.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:13.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9da789640 5 monclient: authenticate success, global_id 274339 2024-09-11T01:56:13.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.094+0000 7fb9da789640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:56:13.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.097+0000 7fb9d3fff640 1 -- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9bc003aa0 msgr2=0x7fb9bc003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:13.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.097+0000 7fb9d3fff640 1 --2- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9bc003aa0 0x7fb9bc003ea0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb9c400d8a0 tx=0x7fb9bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37d4e692158aa95 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:13.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.098+0000 7fb9d2ffd640 1 -- 172.21.15.12:0/878896858 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4031070 con 0x7fb9bc003aa0 2024-09-11T01:56:13.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.098+0000 7fb9d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:13.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.098+0000 7fb9d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:56:13.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.098+0000 7fb9d2ffd640 10 monclient: dump: 2024-09-11T01:56:13.114 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:13.114 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:13.114 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:13.115 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:13.115 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:13.115 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:13.115 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:13.115 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:13.115 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:13.115 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:13.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.098+0000 7fb9da789640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:56:13.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.098+0000 7fb9d2ffd640 1 -- 172.21.15.12:0/878896858 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9c400de90 con 0x7fb9bc003aa0 2024-09-11T01:56:13.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.099+0000 7fb9d2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:13.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.099+0000 7fb9d2ffd640 1 -- 172.21.15.12:0/878896858 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4035400 con 0x7fb9bc003aa0 2024-09-11T01:56:13.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.099+0000 7fb9d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:13.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.099+0000 7fb9d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:56:13.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.099+0000 7fb9d2ffd640 10 monclient: dump: 2024-09-11T01:56:13.116 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:13.116 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:13.116 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:13.116 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:13.116 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:13.116 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:13.116 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:13.116 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:13.116 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:13.116 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:13.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.099+0000 7fb9da789640 10 monclient: get_monmap_and_config success 2024-09-11T01:56:13.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.099+0000 7fb9da789640 4 set_mon_vals no callback set 2024-09-11T01:56:13.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.099+0000 7fb9da789640 10 monclient: shutdown 2024-09-11T01:56:13.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.099+0000 7fb9da789640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:13.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.099+0000 7fb9da789640 1 -- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9bc003aa0 msgr2=0x7fb9bc003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:13.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.099+0000 7fb9da789640 1 --2- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9bc003aa0 0x7fb9bc003ea0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb9c400d8a0 tx=0x7fb9bc005830 comp rx=0 tx=0).stop 2024-09-11T01:56:13.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.099+0000 7fb9da789640 1 -- 172.21.15.12:0/878896858 shutdown_connections 2024-09-11T01:56:13.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.099+0000 7fb9da789640 1 --2- 172.21.15.12:0/878896858 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9bc005410 0x7fb9bc00ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:13.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.099+0000 7fb9da789640 1 --2- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9bc004800 0x7fb9bc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:13.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.099+0000 7fb9da789640 1 --2- 172.21.15.12:0/878896858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9bc003aa0 0x7fb9bc003ea0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:13.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.099+0000 7fb9da789640 1 -- 172.21.15.12:0/878896858 >> 172.21.15.12:0/878896858 conn(0x7fb9d40a3650 msgr2=0x7fb9d40a3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:13.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.099+0000 7fb9da789640 1 -- 172.21.15.12:0/878896858 shutdown_connections 2024-09-11T01:56:13.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.100+0000 7fb9da789640 1 -- 172.21.15.12:0/878896858 wait complete. 2024-09-11T01:56:13.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.100+0000 7fb9da789640 10 monclient: build_initial_monmap 2024-09-11T01:56:13.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.100+0000 7fb9da789640 10 monclient: monmap: 2024-09-11T01:56:13.118 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:13.118 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:56:13.118 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:56:13.118 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:56:13.118 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:13.119 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:13.119 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:56:13.119 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:56:13.119 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:56:13.119 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:13.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.101+0000 7fb9da789640 1 Processor -- start 2024-09-11T01:56:13.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.101+0000 7fb9da789640 1 -- start start 2024-09-11T01:56:13.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.101+0000 7fb9da789640 10 monclient: init 2024-09-11T01:56:13.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9da789640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:13.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9da789640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:56:13.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9da789640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9bc003aa0 0x7fb9d41ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:13.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9da789640 10 monclient(hunting): picked mon.noname-a con 0x7fb9bc003aa0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:56:13.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9da789640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9bc004800 0x7fb9d412f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:13.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9da789640 10 monclient(hunting): picked mon.noname-c con 0x7fb9bc004800 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:13.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9da789640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9bc005410 0x7fb9d411f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:13.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9da789640 10 monclient(hunting): picked mon.noname-b con 0x7fb9bc005410 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:56:13.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9da789640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:13.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9da789640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d41365f0 con 0x7fb9bc003aa0 2024-09-11T01:56:13.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9da789640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:13.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9da789640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d4136770 con 0x7fb9bc005410 2024-09-11T01:56:13.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9da789640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:13.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9da789640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb9d4132f10 con 0x7fb9bc004800 2024-09-11T01:56:13.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9da789640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:13.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9da789640 10 monclient(hunting): authenticate will time out at 2379.544380s 2024-09-11T01:56:13.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9bc003aa0 0x7fb9d41ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:13.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d37fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9bc004800 0x7fb9d412f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:13.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9bc003aa0 0x7fb9d41ccdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50984/0 (socket says 172.21.15.12:50984) 2024-09-11T01:56:13.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d3fff640 1 -- 172.21.15.12:0/1458620739 learned_addr learned my addr 172.21.15.12:0/1458620739 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:13.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d37fe640 10 monclient(hunting): get_auth_request con 0x7fb9bc004800 auth_method 0 2024-09-11T01:56:13.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:13.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:13.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:13.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d3fff640 10 monclient(hunting): get_auth_request con 0x7fb9bc003aa0 auth_method 0 2024-09-11T01:56:13.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:13.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:13.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:13.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d8d26640 1 --2- 172.21.15.12:0/1458620739 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9bc005410 0x7fb9d411f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:13.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:13.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:13.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d37fe640 1 -- 172.21.15.12:0/1458620739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9bc004800 msgr2=0x7fb9d412f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:56:13.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d37fe640 1 -- 172.21.15.12:0/1458620739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9bc004800 msgr2=0x7fb9d412f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:56:13.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d37fe640 1 --2- 172.21.15.12:0/1458620739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9bc004800 0x7fb9d412f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:56:13.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:13.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d8d26640 10 monclient(hunting): get_auth_request con 0x7fb9bc005410 auth_method 0 2024-09-11T01:56:13.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d8d26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:13.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d8d26640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:13.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d8d26640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:13.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d37fe640 1 --2- 172.21.15.12:0/1458620739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9bc004800 0x7fb9d412f690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:56:13.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d8d26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:13.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d8d26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:13.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d8d26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:13.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d3fff640 10 monclient(hunting): handle_auth_done global_id 294189 payload 824 2024-09-11T01:56:13.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d3fff640 1 -- 172.21.15.12:0/1458620739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9bc004800 msgr2=0x7fb9d412f690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:56:13.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d3fff640 1 --2- 172.21.15.12:0/1458620739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9bc004800 0x7fb9d412f690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:13.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d3fff640 1 -- 172.21.15.12:0/1458620739 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9bc005410 msgr2=0x7fb9d411f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:13.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d3fff640 1 --2- 172.21.15.12:0/1458620739 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9bc005410 0x7fb9d411f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:13.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d3fff640 10 monclient: _finish_hunting 0 2024-09-11T01:56:13.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d3fff640 1 monclient: found mon.noname-a 2024-09-11T01:56:13.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:13.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:56:13.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.102+0000 7fb9d3fff640 1 -- 172.21.15.12:0/1458620739 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9d41cdaa0 con 0x7fb9bc003aa0 2024-09-11T01:56:13.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.105+0000 7fb9d3fff640 1 -- 172.21.15.12:0/1458620739 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9bc003aa0 msgr2=0x7fb9d41ccdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:13.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.105+0000 7fb9d3fff640 1 --2- 172.21.15.12:0/1458620739 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9bc003aa0 0x7fb9d41ccdf0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb9c4037040 tx=0x7fb9c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:13.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.105+0000 7fb9d0ff9640 1 -- 172.21.15.12:0/1458620739 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4044020 con 0x7fb9bc003aa0 2024-09-11T01:56:13.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.105+0000 7fb9d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:13.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.105+0000 7fb9d0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:56:13.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.105+0000 7fb9d0ff9640 10 monclient: dump: 2024-09-11T01:56:13.124 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:13.124 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:13.124 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:13.124 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:13.124 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:13.124 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:13.125 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:13.125 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:13.125 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:13.125 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:13.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.105+0000 7fb9d0ff9640 10 monclient: _finish_auth 0 2024-09-11T01:56:13.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.105+0000 7fb9d0ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:56:13.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.105+0000 7fb9d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:13.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.105+0000 7fb9d0ff9640 1 -- 172.21.15.12:0/1458620739 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9c4040880 con 0x7fb9bc003aa0 2024-09-11T01:56:13.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.105+0000 7fb9da789640 5 monclient: authenticate success, global_id 294189 2024-09-11T01:56:13.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.105+0000 7fb9da789640 10 monclient: _renew_subs 2024-09-11T01:56:13.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.105+0000 7fb9da789640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:13.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.105+0000 7fb9da789640 1 -- 172.21.15.12:0/1458620739 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9d4120b20 con 0x7fb9bc003aa0 2024-09-11T01:56:13.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.105+0000 7fb9d0ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:13.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.105+0000 7fb9d0ff9640 1 -- 172.21.15.12:0/1458620739 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4040c30 con 0x7fb9bc003aa0 2024-09-11T01:56:13.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.105+0000 7fb9d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:13.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.105+0000 7fb9d0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:56:13.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.105+0000 7fb9d0ff9640 10 monclient: dump: 2024-09-11T01:56:13.126 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:13.126 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:13.126 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:13.126 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:13.126 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:13.126 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:13.126 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:13.126 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:13.126 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:13.127 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:13.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.105+0000 7fb9d27fc640 4 set_mon_vals no callback set 2024-09-11T01:56:13.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.105+0000 7fb9da789640 10 monclient: _renew_subs 2024-09-11T01:56:13.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.105+0000 7fb9da789640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:13.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.105+0000 7fb9da789640 1 -- 172.21.15.12:0/1458620739 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9d4122e80 con 0x7fb9bc003aa0 2024-09-11T01:56:13.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.106+0000 7fb9d0ff9640 1 -- 172.21.15.12:0/1458620739 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fb9c4005ef0 con 0x7fb9bc003aa0 2024-09-11T01:56:13.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.106+0000 7fb9da789640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:13.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.106+0000 7fb9da789640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:13.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.106+0000 7fb9da789640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:13.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.106+0000 7fb9da789640 1 -- 172.21.15.12:0/1458620739 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9d4133340 con 0x7fb9bc003aa0 2024-09-11T01:56:13.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.107+0000 7fb9d0ff9640 1 --2- 172.21.15.12:0/1458620739 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb9c003ebf0 0x7fb9c00410b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:13.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.107+0000 7fb9d0ff9640 1 -- 172.21.15.12:0/1458620739 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(913..913 src has 676..913) ==== 4799+0+0 (secure 0 0 0) 0x7fb9c40743a0 con 0x7fb9bc003aa0 2024-09-11T01:56:13.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.107+0000 7fb9d37fe640 1 --2- 172.21.15.12:0/1458620739 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb9c003ebf0 0x7fb9c00410b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:13.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.107+0000 7fb9d37fe640 10 monclient: get_auth_request con 0x7fb9c003ebf0 auth_method 0 2024-09-11T01:56:13.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.110+0000 7fb9d37fe640 1 --2- 172.21.15.12:0/1458620739 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb9c003ebf0 0x7fb9c00410b0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4131650 tx=0x7fb9c8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:13.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.111+0000 7fb9d0ff9640 1 -- 172.21.15.12:0/1458620739 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb9c404e030 con 0x7fb9bc003aa0 2024-09-11T01:56:13.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.112+0000 7fb9d0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:13.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.112+0000 7fb9d0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:56:13.154 INFO:tasks.rados.rados.0.smithi071.stdout:3957: done (2 left) 2024-09-11T01:56:13.154 INFO:tasks.rados.rados.0.smithi071.stdout:3958: delete oid 50 current snap is 449 2024-09-11T01:56:13.154 INFO:tasks.rados.rados.0.smithi071.stdout:3953: finishing copy_from racing read to smithi07134777-38 2024-09-11T01:56:13.154 INFO:tasks.rados.rados.0.smithi071.stdout:3953: finishing copy_from to smithi07134777-38 2024-09-11T01:56:13.155 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 3042 (ObjNum 1329 snap 432 seq_num 1329) dirty exists 2024-09-11T01:56:13.155 INFO:tasks.rados.rados.0.smithi071.stdout:3954: finishing write tid 1 to smithi07134777-32 2024-09-11T01:56:13.155 INFO:tasks.rados.rados.0.smithi071.stdout:3954: finishing write tid 2 to smithi07134777-32 2024-09-11T01:56:13.155 INFO:tasks.rados.rados.0.smithi071.stdout:3954: finishing write tid 3 to smithi07134777-32 2024-09-11T01:56:13.155 INFO:tasks.rados.rados.0.smithi071.stdout:3954: finishing write tid 4 to smithi07134777-32 2024-09-11T01:56:13.155 INFO:tasks.rados.rados.0.smithi071.stdout:3954: oid 32 updating version 0 to 2544 2024-09-11T01:56:13.155 INFO:tasks.rados.rados.0.smithi071.stdout:3954: oid 32 updating version 2544 to 2546 2024-09-11T01:56:13.155 INFO:tasks.rados.rados.0.smithi071.stdout:3954: oid 32 version 2546 is already newer than 2545 2024-09-11T01:56:13.155 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 2546 (ObjNum 1355 snap 449 seq_num 1355) dirty exists 2024-09-11T01:56:13.155 INFO:tasks.rados.rados.0.smithi071.stdout:3954: left oid 32 (ObjNum 1355 snap 449 seq_num 1355) 2024-09-11T01:56:13.165 INFO:tasks.rados.rados.0.smithi071.stdout:3953: done (2 left) 2024-09-11T01:56:13.165 INFO:tasks.rados.rados.0.smithi071.stdout:3954: done (1 left) 2024-09-11T01:56:13.165 INFO:tasks.rados.rados.0.smithi071.stdout:3958: done (0 left) 2024-09-11T01:56:13.165 INFO:tasks.rados.rados.0.smithi071.stdout:3959: read oid 42 snap 445 2024-09-11T01:56:13.165 INFO:tasks.rados.rados.0.smithi071.stdout:3959: expect (ObjNum 1334 snap 435 seq_num 1334) 2024-09-11T01:56:13.165 INFO:tasks.rados.rados.0.smithi071.stdout:3960: delete oid 14 current snap is 449 2024-09-11T01:56:13.168 INFO:tasks.rados.rados.0.smithi071.stdout:3959: expect (ObjNum 1334 snap 435 seq_num 1334) 2024-09-11T01:56:13.228 INFO:tasks.rados.rados.0.smithi071.stdout:3959: done (1 left) 2024-09-11T01:56:13.228 INFO:tasks.rados.rados.0.smithi071.stdout:3960: done (0 left) 2024-09-11T01:56:13.228 INFO:tasks.rados.rados.0.smithi071.stdout:3961: snap_remove snap 440 2024-09-11T01:56:13.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.384+0000 7fb9da789640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:56:13.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.384+0000 7fb9da789640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:56:13.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.384+0000 7fb9da789640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:13.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.384+0000 7fb9da789640 1 -- 172.21.15.12:0/1458620739 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb9d40700b0 con 0x7fb9bc003aa0 2024-09-11T01:56:13.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.385+0000 7fb9d0ff9640 1 -- 172.21.15.12:0/1458620739 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fb9c4040e10 con 0x7fb9bc003aa0 2024-09-11T01:56:13.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.385+0000 7fb9d0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:56:13.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.385+0000 7fb9d0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:56:13.387 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":248,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:56:13.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.387+0000 7fb9da789640 1 -- 172.21.15.12:0/1458620739 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb9c003ebf0 msgr2=0x7fb9c00410b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:13.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.387+0000 7fb9da789640 1 --2- 172.21.15.12:0/1458620739 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb9c003ebf0 0x7fb9c00410b0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4131650 tx=0x7fb9c8009040 comp rx=0 tx=0).stop 2024-09-11T01:56:13.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.387+0000 7fb9da789640 10 monclient: shutdown 2024-09-11T01:56:13.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.387+0000 7fb9da789640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:13.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.387+0000 7fb9da789640 1 -- 172.21.15.12:0/1458620739 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9bc003aa0 msgr2=0x7fb9d41ccdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:13.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.387+0000 7fb9da789640 1 --2- 172.21.15.12:0/1458620739 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9bc003aa0 0x7fb9d41ccdf0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb9c4037040 tx=0x7fb9c40046c0 comp rx=0 tx=0).stop 2024-09-11T01:56:13.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.387+0000 7fb9da789640 1 -- 172.21.15.12:0/1458620739 shutdown_connections 2024-09-11T01:56:13.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.387+0000 7fb9da789640 1 --2- 172.21.15.12:0/1458620739 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9bc005410 0x7fb9d411f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:13.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.387+0000 7fb9da789640 1 --2- 172.21.15.12:0/1458620739 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb9bc004800 0x7fb9d412f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:13.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.387+0000 7fb9da789640 1 --2- 172.21.15.12:0/1458620739 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb9c003ebf0 0x7fb9c00410b0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:13.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.387+0000 7fb9da789640 1 --2- 172.21.15.12:0/1458620739 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb9bc003aa0 0x7fb9d41ccdf0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:13.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.387+0000 7fb9da789640 1 -- 172.21.15.12:0/1458620739 >> 172.21.15.12:0/1458620739 conn(0x7fb9d40a3650 msgr2=0x7fb9d409fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:13.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.387+0000 7fb9da789640 1 -- 172.21.15.12:0/1458620739 shutdown_connections 2024-09-11T01:56:13.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:13.387+0000 7fb9da789640 1 -- 172.21.15.12:0/1458620739 wait complete. 2024-09-11T01:56:14.161 INFO:tasks.rados.rados.0.smithi071.stdout:3961: done (0 left) 2024-09-11T01:56:14.161 INFO:tasks.rados.rados.0.smithi071.stdout:3962: read oid 25 snap 432 2024-09-11T01:56:14.161 INFO:tasks.rados.rados.0.smithi071.stdout:3962: expect (ObjNum 1243 snap 411 seq_num 1243) 2024-09-11T01:56:14.161 INFO:tasks.rados.rados.0.smithi071.stdout:3963: snap_create 2024-09-11T01:56:14.170 INFO:tasks.rados.rados.0.smithi071.stdout:3962: expect (ObjNum 1243 snap 411 seq_num 1243) 2024-09-11T01:56:14.211 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:56:14.267 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1974589 2024-09-11T01:56:15.074 INFO:tasks.rados.rados.0.smithi071.stdout:3962: done (1 left) 2024-09-11T01:56:15.074 INFO:tasks.rados.rados.0.smithi071.stdout:3963: done (0 left) 2024-09-11T01:56:15.074 INFO:tasks.rados.rados.0.smithi071.stdout:3964: read oid 29 snap -1 2024-09-11T01:56:15.074 INFO:tasks.rados.rados.0.smithi071.stdout:3964: expect deleted 2024-09-11T01:56:15.074 INFO:tasks.rados.rados.0.smithi071.stdout:3965: write (excl) oid 48 current snap is 450 2024-09-11T01:56:15.074 INFO:tasks.rados.rados.0.smithi071.stdout:3965: seq_num 1358 ranges {423906=498976,1500194=659130,2609483=214423} 2024-09-11T01:56:15.079 INFO:tasks.rados.rados.0.smithi071.stdout:3965: writing smithi07134777-48 from 423906 to 922882 tid 1 2024-09-11T01:56:15.086 INFO:tasks.rados.rados.0.smithi071.stdout:3965: writing smithi07134777-48 from 1500194 to 2159324 tid 2 2024-09-11T01:56:15.090 INFO:tasks.rados.rados.0.smithi071.stdout:3965: writing smithi07134777-48 from 2609483 to 2823906 tid 3 2024-09-11T01:56:15.090 INFO:tasks.rados.rados.0.smithi071.stdout:3966: snap_create 2024-09-11T01:56:15.090 INFO:tasks.rados.rados.0.smithi071.stdout:3965: finishing write tid 1 to smithi07134777-48 2024-09-11T01:56:15.095 INFO:tasks.rados.rados.0.smithi071.stdout:3965: finishing write tid 2 to smithi07134777-48 2024-09-11T01:56:15.096 INFO:tasks.rados.rados.0.smithi071.stdout:3965: finishing write tid 3 to smithi07134777-48 2024-09-11T01:56:15.096 INFO:tasks.rados.rados.0.smithi071.stdout:3965: finishing write tid 4 to smithi07134777-48 2024-09-11T01:56:15.096 INFO:tasks.rados.rados.0.smithi071.stdout:3965: finishing write tid 5 to smithi07134777-48 2024-09-11T01:56:15.096 INFO:tasks.rados.rados.0.smithi071.stdout:3965: oid 48 updating version 0 to 3234 2024-09-11T01:56:15.096 INFO:tasks.rados.rados.0.smithi071.stdout:3965: oid 48 version 3234 is already newer than 3233 2024-09-11T01:56:15.096 INFO:tasks.rados.rados.0.smithi071.stdout:3965: oid 48 updating version 3234 to 3235 2024-09-11T01:56:15.096 INFO:tasks.rados.rados.0.smithi071.stdout:3965: oid 48 version 3235 is already newer than 3232 2024-09-11T01:56:15.096 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 3235 (ObjNum 1358 snap 450 seq_num 1358) dirty exists 2024-09-11T01:56:15.096 INFO:tasks.rados.rados.0.smithi071.stdout:3965: left oid 48 (ObjNum 1358 snap 450 seq_num 1358) 2024-09-11T01:56:16.082 INFO:tasks.rados.rados.0.smithi071.stdout:3964: done (2 left) 2024-09-11T01:56:16.082 INFO:tasks.rados.rados.0.smithi071.stdout:3965: done (1 left) 2024-09-11T01:56:16.082 INFO:tasks.rados.rados.0.smithi071.stdout:3966: done (0 left) 2024-09-11T01:56:16.082 INFO:tasks.rados.rados.0.smithi071.stdout:3967: delete oid 33 current snap is 451 2024-09-11T01:56:16.088 INFO:tasks.rados.rados.0.smithi071.stdout:3967: done (0 left) 2024-09-11T01:56:16.088 INFO:tasks.rados.rados.0.smithi071.stdout:3968: read oid 2 snap -1 2024-09-11T01:56:16.088 INFO:tasks.rados.rados.0.smithi071.stdout:3968: expect (ObjNum 1328 snap 432 seq_num 1328) 2024-09-11T01:56:16.089 INFO:tasks.rados.rados.0.smithi071.stdout:3969: write (excl) oid 8 current snap is 451 2024-09-11T01:56:16.089 INFO:tasks.rados.rados.0.smithi071.stdout:3969: seq_num 1360 ranges {457520=649474,1900805=662799} 2024-09-11T01:56:16.094 INFO:tasks.rados.rados.0.smithi071.stdout:3969: writing smithi07134777-8 from 457520 to 1106994 tid 1 2024-09-11T01:56:16.102 INFO:tasks.rados.rados.0.smithi071.stdout:3969: writing smithi07134777-8 from 1900805 to 2563604 tid 2 2024-09-11T01:56:16.102 INFO:tasks.rados.rados.0.smithi071.stdout:3970: write oid 12 current snap is 451 2024-09-11T01:56:16.102 INFO:tasks.rados.rados.0.smithi071.stdout:3970: seq_num 1361 ranges {474327=441076,1675640=522810,2808105=66222} 2024-09-11T01:56:16.107 INFO:tasks.rados.rados.0.smithi071.stdout:3970: writing smithi07134777-12 from 474327 to 915403 tid 1 2024-09-11T01:56:16.114 INFO:tasks.rados.rados.0.smithi071.stdout:3970: writing smithi07134777-12 from 1675640 to 2198450 tid 2 2024-09-11T01:56:16.118 INFO:tasks.rados.rados.0.smithi071.stdout:3970: writing smithi07134777-12 from 2808105 to 2874327 tid 3 2024-09-11T01:56:16.118 INFO:tasks.rados.rados.0.smithi071.stdout:3971: delete oid 48 current snap is 451 2024-09-11T01:56:16.118 INFO:tasks.rados.rados.0.smithi071.stdout:3968: expect (ObjNum 1328 snap 432 seq_num 1328) 2024-09-11T01:56:16.118 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:56:16.154 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:56:16.224 INFO:tasks.rados.rados.0.smithi071.stdout:3969: finishing write tid 1 to smithi07134777-8 2024-09-11T01:56:16.224 INFO:tasks.rados.rados.0.smithi071.stdout:3969: finishing write tid 2 to smithi07134777-8 2024-09-11T01:56:16.224 INFO:tasks.rados.rados.0.smithi071.stdout:3969: finishing write tid 3 to smithi07134777-8 2024-09-11T01:56:16.224 INFO:tasks.rados.rados.0.smithi071.stdout:3969: finishing write tid 4 to smithi07134777-8 2024-09-11T01:56:16.224 INFO:tasks.rados.rados.0.smithi071.stdout:3969: oid 8 updating version 0 to 2591 2024-09-11T01:56:16.224 INFO:tasks.rados.rados.0.smithi071.stdout:3969: oid 8 version 2591 is already newer than 2590 2024-09-11T01:56:16.224 INFO:tasks.rados.rados.0.smithi071.stdout:3969: oid 8 version 2591 is already newer than 2589 2024-09-11T01:56:16.224 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 2591 (ObjNum 1360 snap 451 seq_num 1360) dirty exists 2024-09-11T01:56:16.224 INFO:tasks.rados.rados.0.smithi071.stdout:3969: left oid 8 (ObjNum 1360 snap 451 seq_num 1360) 2024-09-11T01:56:16.224 INFO:tasks.rados.rados.0.smithi071.stdout:3970: finishing write tid 1 to smithi07134777-12 2024-09-11T01:56:16.224 INFO:tasks.rados.rados.0.smithi071.stdout:3970: finishing write tid 2 to smithi07134777-12 2024-09-11T01:56:16.225 INFO:tasks.rados.rados.0.smithi071.stdout:3968: done (3 left) 2024-09-11T01:56:16.225 INFO:tasks.rados.rados.0.smithi071.stdout:3969: done (2 left) 2024-09-11T01:56:16.225 INFO:tasks.rados.rados.0.smithi071.stdout:3971: done (1 left) 2024-09-11T01:56:16.225 INFO:tasks.rados.rados.0.smithi071.stdout:3972: read oid 25 snap -1 2024-09-11T01:56:16.225 INFO:tasks.rados.rados.0.smithi071.stdout:3972: expect (ObjNum 1243 snap 411 seq_num 1243) 2024-09-11T01:56:16.225 INFO:tasks.rados.rados.0.smithi071.stdout:3973: delete oid 15 current snap is 451 2024-09-11T01:56:16.225 INFO:tasks.rados.rados.0.smithi071.stdout:3970: finishing write tid 3 to smithi07134777-12 2024-09-11T01:56:16.225 INFO:tasks.rados.rados.0.smithi071.stdout:3970: finishing write tid 4 to smithi07134777-12 2024-09-11T01:56:16.225 INFO:tasks.rados.rados.0.smithi071.stdout:3970: finishing write tid 5 to smithi07134777-12 2024-09-11T01:56:16.225 INFO:tasks.rados.rados.0.smithi071.stdout:3970: oid 12 updating version 0 to 3158 2024-09-11T01:56:16.225 INFO:tasks.rados.rados.0.smithi071.stdout:3970: oid 12 version 3158 is already newer than 3155 2024-09-11T01:56:16.225 INFO:tasks.rados.rados.0.smithi071.stdout:3970: oid 12 version 3158 is already newer than 3156 2024-09-11T01:56:16.225 INFO:tasks.rados.rados.0.smithi071.stdout:3970: oid 12 version 3158 is already newer than 3157 2024-09-11T01:56:16.225 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 3158 (ObjNum 1361 snap 451 seq_num 1361) dirty exists 2024-09-11T01:56:16.226 INFO:tasks.rados.rados.0.smithi071.stdout:3970: left oid 12 (ObjNum 1361 snap 451 seq_num 1361) 2024-09-11T01:56:16.228 INFO:tasks.rados.rados.0.smithi071.stdout:3970: done (2 left) 2024-09-11T01:56:16.228 INFO:tasks.rados.rados.0.smithi071.stdout:3973: done (1 left) 2024-09-11T01:56:16.228 INFO:tasks.rados.rados.0.smithi071.stdout:3974: delete oid 8 current snap is 451 2024-09-11T01:56:16.230 INFO:tasks.rados.rados.0.smithi071.stdout:3974: done (1 left) 2024-09-11T01:56:16.231 INFO:tasks.rados.rados.0.smithi071.stdout:3975: read oid 18 snap 420 2024-09-11T01:56:16.231 INFO:tasks.rados.rados.0.smithi071.stdout:3975: expect deleted 2024-09-11T01:56:16.231 INFO:tasks.rados.rados.0.smithi071.stdout:3976: rollback oid 24 current snap is 451 2024-09-11T01:56:16.231 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 24 to 449 2024-09-11T01:56:16.231 INFO:tasks.rados.rados.0.smithi071.stdout:3977: write (excl) oid 14 current snap is 451 2024-09-11T01:56:16.231 INFO:tasks.rados.rados.0.smithi071.stdout:3977: seq_num 1365 ranges {541555=742072,1793215=530148,2801444=140111} 2024-09-11T01:56:16.237 INFO:tasks.rados.rados.0.smithi071.stdout:3977: writing smithi07134777-14 from 541555 to 1283627 tid 1 2024-09-11T01:56:16.243 INFO:tasks.rados.rados.0.smithi071.stdout:3977: writing smithi07134777-14 from 1793215 to 2323363 tid 2 2024-09-11T01:56:16.247 INFO:tasks.rados.rados.0.smithi071.stdout:3977: writing smithi07134777-14 from 2801444 to 2941555 tid 3 2024-09-11T01:56:16.247 INFO:tasks.rados.rados.0.smithi071.stdout:3978: rollback oid 21 current snap is 451 2024-09-11T01:56:16.247 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 21 to 441 2024-09-11T01:56:16.247 INFO:tasks.rados.rados.0.smithi071.stdout:3972: expect (ObjNum 1243 snap 411 seq_num 1243) 2024-09-11T01:56:16.289 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 430787 2024-09-11T01:56:16.368 INFO:tasks.rados.rados.0.smithi071.stdout:3976: finishing rollback tid 1 to smithi07134777-24 2024-09-11T01:56:16.368 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 2610 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-11T01:56:16.368 INFO:tasks.rados.rados.0.smithi071.stdout:3977: finishing write tid 1 to smithi07134777-14 2024-09-11T01:56:16.369 INFO:tasks.rados.rados.0.smithi071.stdout:3977: finishing write tid 2 to smithi07134777-14 2024-09-11T01:56:16.369 INFO:tasks.rados.rados.0.smithi071.stdout:3972: done (4 left) 2024-09-11T01:56:16.369 INFO:tasks.rados.rados.0.smithi071.stdout:3975: done (3 left) 2024-09-11T01:56:16.369 INFO:tasks.rados.rados.0.smithi071.stdout:3976: done (2 left) 2024-09-11T01:56:16.369 INFO:tasks.rados.rados.0.smithi071.stdout:3979: copy_from oid 49 from oid 10 current snap is 451 2024-09-11T01:56:16.369 INFO:tasks.rados.rados.0.smithi071.stdout:3977: finishing write tid 3 to smithi07134777-14 2024-09-11T01:56:16.369 INFO:tasks.rados.rados.0.smithi071.stdout:3977: finishing write tid 4 to smithi07134777-14 2024-09-11T01:56:16.369 INFO:tasks.rados.rados.0.smithi071.stdout:3977: finishing write tid 5 to smithi07134777-14 2024-09-11T01:56:16.369 INFO:tasks.rados.rados.0.smithi071.stdout:3977: oid 14 updating version 0 to 3241 2024-09-11T01:56:16.369 INFO:tasks.rados.rados.0.smithi071.stdout:3977: oid 14 version 3241 is already newer than 3240 2024-09-11T01:56:16.369 INFO:tasks.rados.rados.0.smithi071.stdout:3977: oid 14 updating version 3241 to 3242 2024-09-11T01:56:16.369 INFO:tasks.rados.rados.0.smithi071.stdout:3977: oid 14 updating version 3242 to 3243 2024-09-11T01:56:16.369 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 3243 (ObjNum 1365 snap 451 seq_num 1365) dirty exists 2024-09-11T01:56:16.369 INFO:tasks.rados.rados.0.smithi071.stdout:3977: left oid 14 (ObjNum 1365 snap 451 seq_num 1365) 2024-09-11T01:56:16.369 INFO:tasks.rados.rados.0.smithi071.stdout:3978: finishing rollback tid 0 to smithi07134777-21 2024-09-11T01:56:16.370 INFO:tasks.rados.rados.0.smithi071.stdout:3978: finishing rollback tid 1 to smithi07134777-21 2024-09-11T01:56:16.370 INFO:tasks.rados.rados.0.smithi071.stdout:3978: finishing rollback tid 2 to smithi07134777-21 2024-09-11T01:56:16.370 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 2648 (ObjNum 1313 snap 428 seq_num 1313) dirty exists 2024-09-11T01:56:16.370 INFO:tasks.rados.rados.0.smithi071.stdout:3977: done (2 left) 2024-09-11T01:56:16.370 INFO:tasks.rados.rados.0.smithi071.stdout:3978: done (1 left) 2024-09-11T01:56:16.370 INFO:tasks.rados.rados.0.smithi071.stdout:3980: write oid 27 current snap is 451 2024-09-11T01:56:16.370 INFO:tasks.rados.rados.0.smithi071.stdout:3980: seq_num 1366 ranges {558362=417321,1451697=673806,2712514=245848} 2024-09-11T01:56:16.373 INFO:tasks.rados.rados.0.smithi071.stdout:3980: writing smithi07134777-27 from 558362 to 975683 tid 1 2024-09-11T01:56:16.379 INFO:tasks.rados.rados.0.smithi071.stdout:3980: writing smithi07134777-27 from 1451697 to 2125503 tid 2 2024-09-11T01:56:16.384 INFO:tasks.rados.rados.0.smithi071.stdout:3980: writing smithi07134777-27 from 2712514 to 2958362 tid 3 2024-09-11T01:56:16.384 INFO:tasks.rados.rados.0.smithi071.stdout:3981: copy_from oid 13 from oid 29 current snap is 451 2024-09-11T01:56:16.384 INFO:tasks.rados.rados.0.smithi071.stdout:3980: finishing write tid 1 to smithi07134777-27 2024-09-11T01:56:16.384 INFO:tasks.rados.rados.0.smithi071.stdout:3979: finishing copy_from racing read to smithi07134777-49 2024-09-11T01:56:16.384 INFO:tasks.rados.rados.0.smithi071.stdout:3982: read oid 12 snap -1 2024-09-11T01:56:16.384 INFO:tasks.rados.rados.0.smithi071.stdout:3982: expect (ObjNum 1361 snap 451 seq_num 1361) 2024-09-11T01:56:16.384 INFO:tasks.rados.rados.0.smithi071.stdout:3983: write (excl) oid 8 current snap is 451 2024-09-11T01:56:16.385 INFO:tasks.rados.rados.0.smithi071.stdout:3983: seq_num 1367 ranges {575169=492570,1793826=417464,2623584=351585} 2024-09-11T01:56:16.389 INFO:tasks.rados.rados.0.smithi071.stdout:3983: writing smithi07134777-8 from 575169 to 1067739 tid 1 2024-09-11T01:56:16.395 INFO:tasks.rados.rados.0.smithi071.stdout:3983: writing smithi07134777-8 from 1793826 to 2211290 tid 2 2024-09-11T01:56:16.400 INFO:tasks.rados.rados.0.smithi071.stdout:3983: writing smithi07134777-8 from 2623584 to 2975169 tid 3 2024-09-11T01:56:16.400 INFO:tasks.rados.rados.0.smithi071.stdout:3984: read oid 30 snap -1 2024-09-11T01:56:16.400 INFO:tasks.rados.rados.0.smithi071.stdout:3984: expect (ObjNum 1276 snap 422 seq_num 1276) 2024-09-11T01:56:16.400 INFO:tasks.rados.rados.0.smithi071.stdout:3985: write oid 23 current snap is 451 2024-09-11T01:56:16.400 INFO:tasks.rados.rados.0.smithi071.stdout:3981: finishing copy_from to smithi07134777-13 2024-09-11T01:56:16.400 INFO:tasks.rados.rados.0.smithi071.stdout:3981: got expected ENOENT (src dne) 2024-09-11T01:56:16.400 INFO:tasks.rados.rados.0.smithi071.stdout:3981: finishing copy_from racing read to smithi07134777-13 2024-09-11T01:56:16.401 INFO:tasks.rados.rados.0.smithi071.stdout:3979: finishing copy_from to smithi07134777-49 2024-09-11T01:56:16.401 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 2544 (ObjNum 1352 snap 447 seq_num 1352) dirty exists 2024-09-11T01:56:16.401 INFO:tasks.rados.rados.0.smithi071.stdout:3980: finishing write tid 2 to smithi07134777-27 2024-09-11T01:56:16.401 INFO:tasks.rados.rados.0.smithi071.stdout:3985: seq_num 1368 ranges {591976=567819,1852308=677475} 2024-09-11T01:56:16.401 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:56:16.406 INFO:tasks.rados.rados.0.smithi071.stdout:3985: writing smithi07134777-23 from 591976 to 1159795 tid 1 2024-09-11T01:56:16.414 INFO:tasks.rados.rados.0.smithi071.stdout:3985: writing smithi07134777-23 from 1852308 to 2529783 tid 2 2024-09-11T01:56:16.414 INFO:tasks.rados.rados.0.smithi071.stdout:3979: done (6 left) 2024-09-11T01:56:16.414 INFO:tasks.rados.rados.0.smithi071.stdout:3981: done (5 left) 2024-09-11T01:56:16.415 INFO:tasks.rados.rados.0.smithi071.stdout:3986: snap_remove snap 430 2024-09-11T01:56:16.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.505+0000 7fd2515ad640 10 monclient: get_monmap_and_config 2024-09-11T01:56:16.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.505+0000 7fd2515ad640 10 monclient: build_initial_monmap 2024-09-11T01:56:16.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.505+0000 7fd2515ad640 10 monclient: monmap: 2024-09-11T01:56:16.507 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:16.507 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:16.507 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:56:16.506256+0000 2024-09-11T01:56:16.507 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:56:16.506256+0000 2024-09-11T01:56:16.507 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:16.507 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:16.507 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:56:16.507 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:56:16.507 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:56:16.507 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:56:16.508 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:56:16.508 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:16.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd2515ad640 1 Processor -- start 2024-09-11T01:56:16.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd2515ad640 1 -- start start 2024-09-11T01:56:16.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd2515ad640 10 monclient: init 2024-09-11T01:56:16.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd2515ad640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:16.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd2515ad640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-11T01:56:16.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd2515ad640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd24c0a2e90 0x7fd24c06dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:16.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd2515ad640 10 monclient(hunting): picked mon.noname-c con 0x7fd24c0a2e90 addr v2:172.21.15.71:3300/0 2024-09-11T01:56:16.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd2515ad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd24c0a3270 addr v1:172.21.15.71:6789/0 2024-09-11T01:56:16.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd2515ad640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd24c13d490 0x7fd24c13f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:16.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd2515ad640 10 monclient(hunting): picked mon.noname-a con 0x7fd24c13d490 addr v2:172.21.15.12:3300/0 2024-09-11T01:56:16.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd2515ad640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd24c0a46c0 con 0x7fd24c0a3270 2024-09-11T01:56:16.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd2515ad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:16.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd2515ad640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fd24c0a4900 con 0x7fd24c13d490 2024-09-11T01:56:16.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd2515ad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:16.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd2515ad640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fd24c06ee50 con 0x7fd24c0a2e90 2024-09-11T01:56:16.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd2515ad640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:16.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd2515ad640 10 monclient(hunting): authenticate will time out at 2382.948704s 2024-09-11T01:56:16.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd24b7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd24c13d490 0x7fd24c13f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:16.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd24b7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd24c13d490 0x7fd24c13f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49684/0 (socket says 172.21.15.12:49684) 2024-09-11T01:56:16.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd24b7fe640 1 -- 172.21.15.12:0/1569016738 learned_addr learned my addr 172.21.15.12:0/1569016738 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:16.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd24b7fe640 10 monclient(hunting): get_auth_request con 0x7fd24c13d490 auth_method 0 2024-09-11T01:56:16.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd24b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:16.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd24b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:16.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd24b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:16.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd24b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:16.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd24b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:16.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.506+0000 7fd24b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:16.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.507+0000 7fd24affd640 1 --2- 172.21.15.12:0/1569016738 >> v2:172.21.15.71:3300/0 conn(0x7fd24c0a2e90 0x7fd24c06dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:16.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.507+0000 7fd24affd640 10 monclient(hunting): get_auth_request con 0x7fd24c0a2e90 auth_method 0 2024-09-11T01:56:16.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.507+0000 7fd24affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:16.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.507+0000 7fd24affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:16.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.507+0000 7fd24affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:16.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.507+0000 7fd24b7fe640 10 monclient(hunting): handle_auth_done global_id 294192 payload 274 2024-09-11T01:56:16.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.507+0000 7fd24b7fe640 1 -- 172.21.15.12:0/1569016738 >> v2:172.21.15.71:3300/0 conn(0x7fd24c0a2e90 msgr2=0x7fd24c06dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:16.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.507+0000 7fd24b7fe640 1 --2- 172.21.15.12:0/1569016738 >> v2:172.21.15.71:3300/0 conn(0x7fd24c0a2e90 0x7fd24c06dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:16.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.507+0000 7fd24b7fe640 1 -- 172.21.15.12:0/1569016738 >> v1:172.21.15.71:6789/0 conn(0x7fd24c0a3270 legacy=0x7fd24c0a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:16.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.507+0000 7fd24b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:56:16.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.507+0000 7fd24b7fe640 1 monclient: found mon.noname-a 2024-09-11T01:56:16.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.507+0000 7fd24b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:16.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.507+0000 7fd24b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:56:16.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.507+0000 7fd24b7fe640 1 -- 172.21.15.12:0/1569016738 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd24c06eb70 con 0x7fd24c13d490 2024-09-11T01:56:16.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.509+0000 7fd24b7fe640 1 -- 172.21.15.12:0/1569016738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd24c13d490 msgr2=0x7fd24c13f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:16.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.509+0000 7fd24b7fe640 1 --2- 172.21.15.12:0/1569016738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd24c13d490 0x7fd24c13f880 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd24c0a94a0 tx=0x7fd24402a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a3c4bb8b02e4d83 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:16.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd249ffb640 1 -- 172.21.15.12:0/1569016738 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd244037070 con 0x7fd24c13d490 2024-09-11T01:56:16.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd249ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:16.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd249ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:56:16.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd249ffb640 10 monclient: dump: 2024-09-11T01:56:16.519 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:16.519 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:16.519 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:16.519 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:16.519 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:16.519 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:16.519 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:16.519 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:16.519 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:16.519 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:16.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd249ffb640 10 monclient: _finish_auth 0 2024-09-11T01:56:16.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd249ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:56:16.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd249ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:16.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd249ffb640 1 -- 172.21.15.12:0/1569016738 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd244032440 con 0x7fd24c13d490 2024-09-11T01:56:16.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd2515ad640 5 monclient: authenticate success, global_id 294192 2024-09-11T01:56:16.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd2515ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:56:16.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd249ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:16.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd249ffb640 1 -- 172.21.15.12:0/1569016738 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2440327f0 con 0x7fd24c13d490 2024-09-11T01:56:16.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd2515ad640 10 monclient: get_monmap_and_config success 2024-09-11T01:56:16.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd2515ad640 4 set_mon_vals no callback set 2024-09-11T01:56:16.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd249ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:16.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd249ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:56:16.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd249ffb640 10 monclient: dump: 2024-09-11T01:56:16.520 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:16.520 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:16.521 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:16.521 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:16.521 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:16.521 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:16.521 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:16.521 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:16.521 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:16.521 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:16.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd2515ad640 10 monclient: shutdown 2024-09-11T01:56:16.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd2515ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:16.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd2515ad640 1 -- 172.21.15.12:0/1569016738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd24c13d490 msgr2=0x7fd24c13f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:16.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd2515ad640 1 --2- 172.21.15.12:0/1569016738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd24c13d490 0x7fd24c13f880 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd24c0a94a0 tx=0x7fd24402a080 comp rx=0 tx=0).stop 2024-09-11T01:56:16.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd2515ad640 1 -- 172.21.15.12:0/1569016738 shutdown_connections 2024-09-11T01:56:16.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd2515ad640 1 --2- 172.21.15.12:0/1569016738 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd24c13d490 0x7fd24c13f880 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:16.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd2515ad640 1 --2- 172.21.15.12:0/1569016738 >> v2:172.21.15.71:3300/0 conn(0x7fd24c0a2e90 0x7fd24c06dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:16.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd2515ad640 1 -- 172.21.15.12:0/1569016738 >> 172.21.15.12:0/1569016738 conn(0x7fd24c0a8490 msgr2=0x7fd24c0a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:16.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd2515ad640 1 -- 172.21.15.12:0/1569016738 shutdown_connections 2024-09-11T01:56:16.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.510+0000 7fd2515ad640 1 -- 172.21.15.12:0/1569016738 wait complete. 2024-09-11T01:56:16.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.511+0000 7fd2515ad640 10 monclient: build_initial_monmap 2024-09-11T01:56:16.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.511+0000 7fd2515ad640 10 monclient: monmap: 2024-09-11T01:56:16.522 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:16.522 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:56:16.522 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:56:16.522 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:56:16.522 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:16.522 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:16.522 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:56:16.523 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:56:16.523 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:56:16.523 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:16.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.511+0000 7fd2515ad640 1 Processor -- start 2024-09-11T01:56:16.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd2515ad640 1 -- start start 2024-09-11T01:56:16.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd2515ad640 10 monclient: init 2024-09-11T01:56:16.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd2515ad640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:16.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd2515ad640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:56:16.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd2515ad640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd24c0a2e90 0x7fd24c13b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:16.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd2515ad640 10 monclient(hunting): picked mon.noname-a con 0x7fd24c0a2e90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:56:16.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd2515ad640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd24c0a3270 0x7fd24c1381d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:16.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd2515ad640 10 monclient(hunting): picked mon.noname-c con 0x7fd24c0a3270 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:16.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd2515ad640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd24c13d490 0x7fd24c09c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:16.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd2515ad640 10 monclient(hunting): picked mon.noname-b con 0x7fd24c13d490 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:56:16.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd2515ad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:16.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd2515ad640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd24c06fd20 con 0x7fd24c0a2e90 2024-09-11T01:56:16.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd2515ad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:16.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd2515ad640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd24c06fba0 con 0x7fd24c13d490 2024-09-11T01:56:16.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd2515ad640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:16.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd2515ad640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fd24c13fdc0 con 0x7fd24c0a3270 2024-09-11T01:56:16.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd2515ad640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:16.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd2515ad640 10 monclient(hunting): authenticate will time out at 2382.954527s 2024-09-11T01:56:16.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd24c0a2e90 0x7fd24c13b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:16.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd24c0a2e90 0x7fd24c13b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49692/0 (socket says 172.21.15.12:49692) 2024-09-11T01:56:16.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24affd640 1 -- 172.21.15.12:0/2128257736 learned_addr learned my addr 172.21.15.12:0/2128257736 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:16.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24affd640 10 monclient(hunting): get_auth_request con 0x7fd24c0a2e90 auth_method 0 2024-09-11T01:56:16.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:16.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:16.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:16.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24a7fc640 1 --2- 172.21.15.12:0/2128257736 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd24c0a3270 0x7fd24c1381d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:16.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:16.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:16.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:16.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24a7fc640 10 monclient(hunting): get_auth_request con 0x7fd24c0a3270 auth_method 0 2024-09-11T01:56:16.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:16.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:16.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:16.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24b7fe640 1 --2- 172.21.15.12:0/2128257736 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd24c13d490 0x7fd24c09c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:16.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24b7fe640 10 monclient(hunting): get_auth_request con 0x7fd24c13d490 auth_method 0 2024-09-11T01:56:16.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:16.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:16.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:16.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:16.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:16.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:16.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24affd640 10 monclient(hunting): handle_auth_done global_id 294195 payload 824 2024-09-11T01:56:16.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24affd640 1 -- 172.21.15.12:0/2128257736 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd24c0a3270 msgr2=0x7fd24c1381d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:16.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24affd640 1 --2- 172.21.15.12:0/2128257736 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd24c0a3270 0x7fd24c1381d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:16.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24affd640 1 -- 172.21.15.12:0/2128257736 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd24c13d490 msgr2=0x7fd24c09c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:16.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24affd640 1 --2- 172.21.15.12:0/2128257736 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd24c13d490 0x7fd24c09c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:16.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24affd640 10 monclient: _finish_hunting 0 2024-09-11T01:56:16.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24affd640 1 monclient: found mon.noname-a 2024-09-11T01:56:16.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:16.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:56:16.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24affd640 1 -- 172.21.15.12:0/2128257736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd24c09f220 con 0x7fd24c0a2e90 2024-09-11T01:56:16.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.512+0000 7fd24b7fe640 1 --2- 172.21.15.12:0/2128257736 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd24c13d490 0x7fd24c09c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:56:16.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.515+0000 7fd24affd640 1 -- 172.21.15.12:0/2128257736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd24c0a2e90 msgr2=0x7fd24c13b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:16.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.515+0000 7fd24affd640 1 --2- 172.21.15.12:0/2128257736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd24c0a2e90 0x7fd24c13b550 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd23c007ef0 tx=0x7fd23c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:16.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.515+0000 7fd2437fe640 1 -- 172.21.15.12:0/2128257736 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd23c013790 con 0x7fd24c0a2e90 2024-09-11T01:56:16.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.515+0000 7fd2437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:16.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.515+0000 7fd2437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:56:16.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.515+0000 7fd2437fe640 10 monclient: dump: 2024-09-11T01:56:16.527 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:16.527 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:16.527 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:16.527 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:16.527 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:16.527 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:16.528 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:16.528 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:16.528 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:16.528 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:16.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.515+0000 7fd2437fe640 10 monclient: _finish_auth 0 2024-09-11T01:56:16.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.515+0000 7fd2437fe640 10 monclient: _check_auth_tickets 2024-09-11T01:56:16.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.515+0000 7fd2437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:16.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.515+0000 7fd2515ad640 5 monclient: authenticate success, global_id 294195 2024-09-11T01:56:16.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.515+0000 7fd2437fe640 1 -- 172.21.15.12:0/2128257736 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd23c019070 con 0x7fd24c0a2e90 2024-09-11T01:56:16.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.515+0000 7fd2515ad640 10 monclient: _renew_subs 2024-09-11T01:56:16.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.515+0000 7fd2515ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:16.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.515+0000 7fd2515ad640 1 -- 172.21.15.12:0/2128257736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd24c0a5700 con 0x7fd24c0a2e90 2024-09-11T01:56:16.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.515+0000 7fd2437fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:16.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.515+0000 7fd2437fe640 1 -- 172.21.15.12:0/2128257736 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd23c013b70 con 0x7fd24c0a2e90 2024-09-11T01:56:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.515+0000 7fd2437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.515+0000 7fd2437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:56:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.515+0000 7fd2437fe640 10 monclient: dump: 2024-09-11T01:56:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:16.529 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:16.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.515+0000 7fd2515ad640 10 monclient: _renew_subs 2024-09-11T01:56:16.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.515+0000 7fd2497fa640 4 set_mon_vals no callback set 2024-09-11T01:56:16.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.515+0000 7fd2515ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:16.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.515+0000 7fd2515ad640 1 -- 172.21.15.12:0/2128257736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd24c09ef40 con 0x7fd24c0a2e90 2024-09-11T01:56:16.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.516+0000 7fd2515ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:16.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.516+0000 7fd2515ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:16.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.516+0000 7fd2515ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:16.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.516+0000 7fd2515ad640 1 -- 172.21.15.12:0/2128257736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd24c06dc60 con 0x7fd24c0a2e90 2024-09-11T01:56:16.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.516+0000 7fd2437fe640 1 -- 172.21.15.12:0/2128257736 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fd23c01e460 con 0x7fd24c0a2e90 2024-09-11T01:56:16.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.516+0000 7fd2437fe640 1 --2- 172.21.15.12:0/2128257736 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd22c03e260 0x7fd22c040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:16.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.516+0000 7fd2437fe640 1 -- 172.21.15.12:0/2128257736 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(917..917 src has 676..917) ==== 4755+0+0 (secure 0 0 0) 0x7fd23c0535c0 con 0x7fd24c0a2e90 2024-09-11T01:56:16.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.516+0000 7fd24a7fc640 1 --2- 172.21.15.12:0/2128257736 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd22c03e260 0x7fd22c040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:16.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.516+0000 7fd24a7fc640 10 monclient: get_auth_request con 0x7fd22c03e260 auth_method 0 2024-09-11T01:56:16.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.519+0000 7fd24a7fc640 1 --2- 172.21.15.12:0/2128257736 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd22c03e260 0x7fd22c040720 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fd24c09d9b0 tx=0x7fd234007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:16.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.520+0000 7fd2437fe640 1 -- 172.21.15.12:0/2128257736 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd23c051fd0 con 0x7fd24c0a2e90 2024-09-11T01:56:16.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.520+0000 7fd2437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:16.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.520+0000 7fd2437fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:56:16.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.778+0000 7fd2515ad640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:56:16.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.778+0000 7fd2515ad640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:56:16.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.778+0000 7fd2515ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:16.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.778+0000 7fd2515ad640 1 -- 172.21.15.12:0/2128257736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd24c0698e0 con 0x7fd24c0a2e90 2024-09-11T01:56:16.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.778+0000 7fd2437fe640 1 -- 172.21.15.12:0/2128257736 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fd23c053300 con 0x7fd24c0a2e90 2024-09-11T01:56:16.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.778+0000 7fd2437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:56:16.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.778+0000 7fd2437fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:56:16.780 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":250,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:56:16.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.780+0000 7fd2515ad640 1 -- 172.21.15.12:0/2128257736 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd22c03e260 msgr2=0x7fd22c040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:16.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.780+0000 7fd2515ad640 1 --2- 172.21.15.12:0/2128257736 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd22c03e260 0x7fd22c040720 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fd24c09d9b0 tx=0x7fd234007e30 comp rx=0 tx=0).stop 2024-09-11T01:56:16.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.780+0000 7fd2515ad640 10 monclient: shutdown 2024-09-11T01:56:16.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.780+0000 7fd2515ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:16.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.780+0000 7fd2515ad640 1 -- 172.21.15.12:0/2128257736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd24c0a2e90 msgr2=0x7fd24c13b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:16.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.780+0000 7fd2515ad640 1 --2- 172.21.15.12:0/2128257736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd24c0a2e90 0x7fd24c13b550 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd23c007ef0 tx=0x7fd23c00e4b0 comp rx=0 tx=0).stop 2024-09-11T01:56:16.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.780+0000 7fd2515ad640 1 -- 172.21.15.12:0/2128257736 shutdown_connections 2024-09-11T01:56:16.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.780+0000 7fd2515ad640 1 --2- 172.21.15.12:0/2128257736 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd24c13d490 0x7fd24c09c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:16.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.780+0000 7fd2515ad640 1 --2- 172.21.15.12:0/2128257736 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd24c0a3270 0x7fd24c1381d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:16.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.780+0000 7fd2515ad640 1 --2- 172.21.15.12:0/2128257736 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fd22c03e260 0x7fd22c040720 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:16.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.780+0000 7fd2515ad640 1 --2- 172.21.15.12:0/2128257736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd24c0a2e90 0x7fd24c13b550 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:16.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.780+0000 7fd2515ad640 1 -- 172.21.15.12:0/2128257736 >> 172.21.15.12:0/2128257736 conn(0x7fd24c0a8490 msgr2=0x7fd24c13c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:16.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.780+0000 7fd2515ad640 1 -- 172.21.15.12:0/2128257736 shutdown_connections 2024-09-11T01:56:16.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.780+0000 7fd2515ad640 1 -- 172.21.15.12:0/2128257736 wait complete. 2024-09-11T01:56:16.792 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:56:16.792 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:56:16.793 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:56:16.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.893+0000 7f1551039640 10 monclient: get_monmap_and_config 2024-09-11T01:56:16.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.893+0000 7f1551039640 10 monclient: build_initial_monmap 2024-09-11T01:56:16.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.893+0000 7f1551039640 10 monclient: monmap: 2024-09-11T01:56:16.895 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:16.895 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:16.895 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:56:16.893795+0000 2024-09-11T01:56:16.895 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:56:16.893795+0000 2024-09-11T01:56:16.895 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:16.895 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:16.895 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:56:16.895 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:56:16.895 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:56:16.895 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:56:16.895 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:56:16.895 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:16.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.893+0000 7f1551039640 1 Processor -- start 2024-09-11T01:56:16.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.893+0000 7f1551039640 1 -- start start 2024-09-11T01:56:16.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.893+0000 7f1551039640 10 monclient: init 2024-09-11T01:56:16.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f1551039640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:16.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f1551039640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-11T01:56:16.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f1551039640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f154c137e50 0x7f154c09d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:16.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f1551039640 10 monclient(hunting): picked mon.noname-a con 0x7f154c137e50 addr v2:172.21.15.12:3300/0 2024-09-11T01:56:16.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f1551039640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f154c138230 addr v1:172.21.15.12:6789/0 2024-09-11T01:56:16.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f1551039640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f154c13d380 0x7f154c13f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:16.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f1551039640 10 monclient(hunting): picked mon.noname-c con 0x7f154c13d380 addr v2:172.21.15.71:3300/0 2024-09-11T01:56:16.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f1551039640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f154c0a62d0 con 0x7f154c138230 2024-09-11T01:56:16.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f1551039640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:16.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f1551039640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f154c09bb40 con 0x7f154c137e50 2024-09-11T01:56:16.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f1551039640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:16.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f1551039640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f154c09c4a0 con 0x7f154c13d380 2024-09-11T01:56:16.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f1551039640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:16.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f1551039640 10 monclient(hunting): authenticate will time out at 2383.336228s 2024-09-11T01:56:16.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f154ad9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f154c137e50 0x7f154c09d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:16.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f154a59c640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f154c138230 0x7f154c138640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43466/0 (socket says 172.21.15.12:43466) 2024-09-11T01:56:16.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f154a59c640 1 -- 172.21.15.12:0/2366480728 learned_addr learned my addr 172.21.15.12:0/2366480728 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:56:16.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f154ad9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f154c137e50 0x7f154c09d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49704/0 (socket says 172.21.15.12:49704) 2024-09-11T01:56:16.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f154ad9d640 10 monclient(hunting): get_auth_request con 0x7f154c137e50 auth_method 0 2024-09-11T01:56:16.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f154ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:16.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f154ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:16.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f154ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:16.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f154b59e640 1 --2- 172.21.15.12:0/2366480728 >> v2:172.21.15.71:3300/0 conn(0x7f154c13d380 0x7f154c13f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:16.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f154b59e640 10 monclient(hunting): get_auth_request con 0x7f154c13d380 auth_method 0 2024-09-11T01:56:16.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f154b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:16.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f154b59e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:16.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f154b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:16.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f154b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:16.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f154b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:16.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f154b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:16.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f154b59e640 10 monclient(hunting): handle_auth_done global_id 274351 payload 274 2024-09-11T01:56:16.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f154b59e640 1 -- 172.21.15.12:0/2366480728 >> v2:172.21.15.12:3300/0 conn(0x7f154c137e50 msgr2=0x7f154c09d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:16.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f154b59e640 1 --2- 172.21.15.12:0/2366480728 >> v2:172.21.15.12:3300/0 conn(0x7f154c137e50 0x7f154c09d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:16.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f154b59e640 1 -- 172.21.15.12:0/2366480728 >> v1:172.21.15.12:6789/0 conn(0x7f154c138230 legacy=0x7f154c138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:16.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f154b59e640 10 monclient: _finish_hunting 0 2024-09-11T01:56:16.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f154b59e640 1 monclient: found mon.noname-c 2024-09-11T01:56:16.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f154b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:16.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f154b59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:56:16.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.894+0000 7f154b59e640 1 -- 172.21.15.12:0/2366480728 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f154c09c780 con 0x7f154c13d380 2024-09-11T01:56:16.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.897+0000 7f154b59e640 1 -- 172.21.15.12:0/2366480728 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f154c13d380 msgr2=0x7f154c13f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:16.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.897+0000 7f154b59e640 1 --2- 172.21.15.12:0/2366480728 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f154c13d380 0x7f154c13f770 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f154c0ab580 tx=0x7f154402a480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da3edec0c22c9725 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:16.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1549d9b640 1 -- 172.21.15.12:0/2366480728 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1544007950 con 0x7f154c13d380 2024-09-11T01:56:16.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1549d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:16.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1549d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:56:16.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1549d9b640 10 monclient: dump: 2024-09-11T01:56:16.907 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:16.907 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:16.907 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:16.907 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:16.907 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:16.907 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:16.907 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:16.907 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:16.908 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1549d9b640 10 monclient: _finish_auth 0 2024-09-11T01:56:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1549d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:56:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1549d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1551039640 5 monclient: authenticate success, global_id 274351 2024-09-11T01:56:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1551039640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:56:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1549d9b640 1 -- 172.21.15.12:0/2366480728 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1544007af0 con 0x7f154c13d380 2024-09-11T01:56:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1549d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1551039640 10 monclient: get_monmap_and_config success 2024-09-11T01:56:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1551039640 4 set_mon_vals no callback set 2024-09-11T01:56:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1549d9b640 1 -- 172.21.15.12:0/2366480728 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1544031400 con 0x7f154c13d380 2024-09-11T01:56:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1549d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1549d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:56:16.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1549d9b640 10 monclient: dump: 2024-09-11T01:56:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:16.909 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1551039640 10 monclient: shutdown 2024-09-11T01:56:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1551039640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:16.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1551039640 1 -- 172.21.15.12:0/2366480728 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f154c13d380 msgr2=0x7f154c13f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:16.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1551039640 1 --2- 172.21.15.12:0/2366480728 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f154c13d380 0x7f154c13f770 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f154c0ab580 tx=0x7f154402a480 comp rx=0 tx=0).stop 2024-09-11T01:56:16.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1551039640 1 -- 172.21.15.12:0/2366480728 shutdown_connections 2024-09-11T01:56:16.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1551039640 1 --2- 172.21.15.12:0/2366480728 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f154c13d380 0x7f154c13f770 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:16.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1551039640 1 --2- 172.21.15.12:0/2366480728 >> v2:172.21.15.12:3300/0 conn(0x7f154c137e50 0x7f154c09d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:16.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1551039640 1 -- 172.21.15.12:0/2366480728 >> 172.21.15.12:0/2366480728 conn(0x7f154c0a7020 msgr2=0x7f154c0a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:16.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1551039640 1 -- 172.21.15.12:0/2366480728 shutdown_connections 2024-09-11T01:56:16.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.898+0000 7f1551039640 1 -- 172.21.15.12:0/2366480728 wait complete. 2024-09-11T01:56:16.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.899+0000 7f1551039640 10 monclient: build_initial_monmap 2024-09-11T01:56:16.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.899+0000 7f1551039640 10 monclient: monmap: 2024-09-11T01:56:16.910 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:16.910 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:56:16.910 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:56:16.910 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:56:16.910 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:16.910 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:56:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:56:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:56:16.911 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f1551039640 1 Processor -- start 2024-09-11T01:56:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f1551039640 1 -- start start 2024-09-11T01:56:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f1551039640 10 monclient: init 2024-09-11T01:56:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f1551039640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f1551039640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:56:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f1551039640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f154c137e50 0x7f154c1cd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f1551039640 10 monclient(hunting): picked mon.noname-a con 0x7f154c137e50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:56:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f1551039640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f154c138230 0x7f154c13ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f1551039640 10 monclient(hunting): picked mon.noname-c con 0x7f154c138230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f1551039640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f154c13d380 0x7f154c13bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f1551039640 10 monclient(hunting): picked mon.noname-b con 0x7f154c13d380 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:56:16.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f1551039640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:16.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f1551039640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f154c06f890 con 0x7f154c137e50 2024-09-11T01:56:16.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f1551039640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:16.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f1551039640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f154c06f710 con 0x7f154c13d380 2024-09-11T01:56:16.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f1551039640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:16.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f1551039640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f154c13fcb0 con 0x7f154c138230 2024-09-11T01:56:16.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f1551039640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:16.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f1551039640 10 monclient(hunting): authenticate will time out at 2383.342605s 2024-09-11T01:56:16.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f154ad9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f154c137e50 0x7f154c1cd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:16.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f154ad9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f154c137e50 0x7f154c1cd2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49708/0 (socket says 172.21.15.12:49708) 2024-09-11T01:56:16.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f154ad9d640 1 -- 172.21.15.12:0/3467535325 learned_addr learned my addr 172.21.15.12:0/3467535325 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:16.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f154ad9d640 10 monclient(hunting): get_auth_request con 0x7f154c137e50 auth_method 0 2024-09-11T01:56:16.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f154a59c640 1 --2- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f154c138230 0x7f154c13ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:16.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f154ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:16.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f154ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:16.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f154ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:16.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f154a59c640 1 -- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f154c138230 msgr2=0x7f154c13ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T01:56:16.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f154a59c640 1 -- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f154c138230 msgr2=0x7f154c13ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:56:16.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f154a59c640 1 --2- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f154c138230 0x7f154c13ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:56:16.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f154a59c640 1 --2- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f154c138230 0x7f154c13ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:56:16.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f154ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:16.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f154ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:16.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f154ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:16.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f154b59e640 1 --2- 172.21.15.12:0/3467535325 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f154c13d380 0x7f154c13bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:16.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f154b59e640 10 monclient(hunting): get_auth_request con 0x7f154c13d380 auth_method 0 2024-09-11T01:56:16.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f154b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:16.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f154b59e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:16.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f154b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:16.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.900+0000 7f154ad9d640 10 monclient(hunting): handle_auth_done global_id 294201 payload 824 2024-09-11T01:56:16.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.901+0000 7f154ad9d640 1 -- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f154c138230 msgr2=0x7f154c13ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:56:16.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.901+0000 7f154ad9d640 1 --2- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f154c138230 0x7f154c13ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:16.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.901+0000 7f154ad9d640 1 -- 172.21.15.12:0/3467535325 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f154c13d380 msgr2=0x7f154c13bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:16.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.901+0000 7f154ad9d640 1 --2- 172.21.15.12:0/3467535325 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f154c13d380 0x7f154c13bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:16.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.901+0000 7f154ad9d640 10 monclient: _finish_hunting 0 2024-09-11T01:56:16.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.901+0000 7f154ad9d640 1 monclient: found mon.noname-a 2024-09-11T01:56:16.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.901+0000 7f154ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:16.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.901+0000 7f154ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:56:16.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.901+0000 7f154ad9d640 1 -- 172.21.15.12:0/3467535325 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f154c0a0540 con 0x7f154c137e50 2024-09-11T01:56:16.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.903+0000 7f154ad9d640 1 -- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f154c137e50 msgr2=0x7f154c1cd2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:16.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.903+0000 7f154ad9d640 1 --2- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f154c137e50 0x7f154c1cd2d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f153400d800 tx=0x7f153400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:16.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.903+0000 7f15437fe640 1 -- 172.21.15.12:0/3467535325 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153400b870 con 0x7f154c137e50 2024-09-11T01:56:16.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.903+0000 7f15437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:16.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.903+0000 7f15437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:56:16.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.903+0000 7f15437fe640 10 monclient: dump: 2024-09-11T01:56:16.915 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:16.916 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:16.916 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:16.916 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:16.916 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:16.916 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:16.916 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:16.916 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:16.916 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:16.916 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:16.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.903+0000 7f15437fe640 10 monclient: _finish_auth 0 2024-09-11T01:56:16.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.903+0000 7f15437fe640 10 monclient: _check_auth_tickets 2024-09-11T01:56:16.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.903+0000 7f15437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.903+0000 7f1551039640 5 monclient: authenticate success, global_id 294201 2024-09-11T01:56:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.903+0000 7f15437fe640 1 -- 172.21.15.12:0/3467535325 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1534019070 con 0x7f154c137e50 2024-09-11T01:56:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.903+0000 7f15437fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.903+0000 7f1551039640 10 monclient: _renew_subs 2024-09-11T01:56:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.903+0000 7f15437fe640 1 -- 172.21.15.12:0/3467535325 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153400bc30 con 0x7f154c137e50 2024-09-11T01:56:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.903+0000 7f1551039640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.903+0000 7f1551039640 1 -- 172.21.15.12:0/3467535325 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f154c13f8a0 con 0x7f154c137e50 2024-09-11T01:56:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.903+0000 7f15437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.903+0000 7f15437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:56:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.903+0000 7f15437fe640 10 monclient: dump: 2024-09-11T01:56:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:16.917 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:16.918 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:16.918 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:16.918 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:16.918 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:16.918 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:16.918 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:16.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.903+0000 7f154959a640 4 set_mon_vals no callback set 2024-09-11T01:56:16.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.903+0000 7f1551039640 10 monclient: _renew_subs 2024-09-11T01:56:16.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.903+0000 7f1551039640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:16.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.903+0000 7f1551039640 1 -- 172.21.15.12:0/3467535325 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f154c139da0 con 0x7f154c137e50 2024-09-11T01:56:16.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.904+0000 7f1551039640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:16.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.904+0000 7f1551039640 1 --2- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f154c09eb70 0x7f154c09ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:16.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.904+0000 7f1551039640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:16.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.904+0000 7f1551039640 1 -- 172.21.15.12:0/3467535325 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f154c06a060 con 0x7f154c09eb70 2024-09-11T01:56:16.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.904+0000 7f154a59c640 1 --2- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f154c09eb70 0x7f154c09ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:16.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.904+0000 7f154a59c640 10 monclient: get_auth_request con 0x7f154c09eb70 auth_method 0 2024-09-11T01:56:16.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.904+0000 7f154a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:16.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.904+0000 7f154a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:16.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.904+0000 7f154a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:16.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.904+0000 7f15437fe640 1 -- 172.21.15.12:0/3467535325 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f1534013070 con 0x7f154c137e50 2024-09-11T01:56:16.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.904+0000 7f154a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:16.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.904+0000 7f154a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:16.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.904+0000 7f154a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:16.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.904+0000 7f154a59c640 10 monclient(hunting): handle_auth_done global_id 294204 payload 824 2024-09-11T01:56:16.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.904+0000 7f154a59c640 1 -- 172.21.15.12:0/3467535325 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f154c13a210 con 0x7f154c09eb70 2024-09-11T01:56:16.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.904+0000 7f15437fe640 1 --2- 172.21.15.12:0/3467535325 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f152c03e210 0x7f152c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:16.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.904+0000 7f15437fe640 1 -- 172.21.15.12:0/3467535325 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(917..917 src has 676..917) ==== 4755+0+0 (secure 0 0 0) 0x7f1534052f40 con 0x7f154c137e50 2024-09-11T01:56:16.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.905+0000 7f154b59e640 1 --2- 172.21.15.12:0/3467535325 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f152c03e210 0x7f152c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:16.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.905+0000 7f154b59e640 10 monclient: get_auth_request con 0x7f152c03e210 auth_method 0 2024-09-11T01:56:16.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.906+0000 7f154a59c640 1 -- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f154c09eb70 msgr2=0x7f154c09ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:16.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.906+0000 7f154a59c640 1 --2- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f154c09eb70 0x7f154c09ef50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f154c0a1320 tx=0x7f153c00d810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:16.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.907+0000 7f15437fe640 1 -- 172.21.15.12:0/3467535325 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153c018070 con 0x7f154c09eb70 2024-09-11T01:56:16.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.907+0000 7f15437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:16.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.907+0000 7f15437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:56:16.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.907+0000 7f15437fe640 10 monclient: dump: 2024-09-11T01:56:16.923 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:16.923 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:16.923 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:16.923 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:16.923 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:16.923 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:16.923 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:16.923 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:16.923 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:16.923 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:16.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.907+0000 7f15437fe640 1 -- 172.21.15.12:0/3467535325 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f153c0070d0 con 0x7f154c09eb70 2024-09-11T01:56:16.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.907+0000 7f15437fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:16.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.907+0000 7f15437fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:56:16.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.907+0000 7f15437fe640 1 -- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f154c09eb70 msgr2=0x7f154c09ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:16.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.907+0000 7f15437fe640 1 --2- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f154c09eb70 0x7f154c09ef50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f154c0a1320 tx=0x7f153c00d810 comp rx=0 tx=0).stop 2024-09-11T01:56:16.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.907+0000 7f15437fe640 1 -- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f154c09eb70 msgr2=0x7f154c09ef50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:56:16.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.907+0000 7f15437fe640 1 --2- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f154c09eb70 0x7f154c09ef50 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f154c0a1320 tx=0x7f153c00d810 comp rx=0 tx=0).stop 2024-09-11T01:56:16.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.907+0000 7f154b59e640 1 --2- 172.21.15.12:0/3467535325 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f152c03e210 0x7f152c0406d0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f154c09cf30 tx=0x7f154402f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:16.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.913+0000 7f1551039640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:56:16.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.913+0000 7f1551039640 1 --2- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f154c09f490 0x7f154c13ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:16.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.913+0000 7f1551039640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:16.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.913+0000 7f1551039640 1 -- 172.21.15.12:0/3467535325 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f154c077410 con 0x7f154c09f490 2024-09-11T01:56:16.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.913+0000 7f154a59c640 1 --2- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f154c09f490 0x7f154c13ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:16.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.913+0000 7f154a59c640 10 monclient: get_auth_request con 0x7f154c09f490 auth_method 0 2024-09-11T01:56:16.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.913+0000 7f154a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:16.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.913+0000 7f154a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:16.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.913+0000 7f154a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:16.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.913+0000 7f154a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:16.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.913+0000 7f154a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:16.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.913+0000 7f154a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:16.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.913+0000 7f154a59c640 10 monclient(hunting): handle_auth_done global_id 294207 payload 824 2024-09-11T01:56:16.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.913+0000 7f154a59c640 1 -- 172.21.15.12:0/3467535325 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f154c09d160 con 0x7f154c09f490 2024-09-11T01:56:16.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.916+0000 7f154a59c640 1 -- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f154c09f490 msgr2=0x7f154c13ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:16.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.916+0000 7f154a59c640 1 --2- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f154c09f490 0x7f154c13ddb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f153c015000 tx=0x7f153c004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:16.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.916+0000 7f15437fe640 1 -- 172.21.15.12:0/3467535325 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153c018040 con 0x7f154c09f490 2024-09-11T01:56:16.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.916+0000 7f15437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:16.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.916+0000 7f15437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:56:16.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.916+0000 7f15437fe640 10 monclient: dump: 2024-09-11T01:56:16.926 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:16.926 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:16.926 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:16.926 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:16.926 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:16.926 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:16.926 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:16.926 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:16.926 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:16.926 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:16.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.916+0000 7f15437fe640 1 -- 172.21.15.12:0/3467535325 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4710 (secure 0 0 0) 0x7f153c01a830 con 0x7f154c09f490 2024-09-11T01:56:16.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.916+0000 7f15437fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:56:16.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.916+0000 7f15437fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:56:16.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.916+0000 7f15437fe640 1 -- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f154c09f490 msgr2=0x7f154c13ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:16.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.916+0000 7f15437fe640 1 --2- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f154c09f490 0x7f154c13ddb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f153c015000 tx=0x7f153c004940 comp rx=0 tx=0).stop 2024-09-11T01:56:16.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.916+0000 7f15437fe640 1 -- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f154c09f490 msgr2=0x7f154c13ddb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:56:16.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.916+0000 7f15437fe640 1 --2- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f154c09f490 0x7f154c13ddb0 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f153c015000 tx=0x7f153c004940 comp rx=0 tx=0).stop 2024-09-11T01:56:16.927 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:56:16.927 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:56:16.927 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:56:16.927 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:56:16.927 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 250, 2024-09-11T01:56:16.928 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 56869, 2024-09-11T01:56:16.928 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:56:16.928 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:56:16.928 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:56:16.928 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:56:16.928 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:16.928 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:56:16.928 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:56:16.928 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:56:16.928 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:56:16.928 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:56:16.928 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:56:16.928 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:56:16.929 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:56:16.929 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:56:16.929 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:56:16.929 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:56:16.929 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:56:16.929 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:56:16.929 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:56:16.929 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:56:16.929 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:16.929 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:56:16.929 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:56:16.929 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:56:16.929 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:56:16.929 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:56:16.930 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:56:16.930 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:56:16.930 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:56:16.930 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:56:16.930 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:56:16.930 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:56:16.930 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:56:16.930 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:56:16.930 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:16.930 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:16.930 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:56:16.930 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:56:16.930 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:56:16.931 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:56:16.931 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:56:16.931 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:56:16.931 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:56:16.931 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:56:16.931 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:56:16.931 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:56:16.931 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:56:16.931 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:56:16.931 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:56:16.931 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:56:16.931 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:56:16.931 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:56:16.932 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:56:16.932 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:56:16.932 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:56:16.932 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:56:16.932 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:56:16.932 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:56:16.932 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:56:16.932 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:56:16.932 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:56:16.932 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:56:16.932 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:56:16.932 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:56:16.932 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:16.932 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:56:16.933 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:16.933 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:56:16.933 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:16.933 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:56:16.933 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:56:16.933 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:56:16.933 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:56:16.933 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:16.933 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:56:16.933 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:56:16.933 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:16.933 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:16.933 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:16.934 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:56:16.934 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:56:16.934 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:16.934 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:16.934 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:16.934 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:16.934 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:56:16.934 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:56:16.934 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:56:16.934 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:56:16.934 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:56:16.934 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:16.934 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:16.934 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:56:16.935 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:56:16.935 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:56:16.935 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:56:16.935 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:16.935 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:56:16.935 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:56:16.935 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:16.935 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:16.935 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:16.935 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:56:16.935 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:56:16.935 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:16.935 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:16.936 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:16.936 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:16.936 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:56:16.936 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:56:16.936 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:56:16.936 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:56:16.936 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:56:16.936 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:16.936 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:16.936 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:56:16.936 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:56:16.936 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:56:16.936 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:56:16.937 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:16.937 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:56:16.937 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:56:16.937 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:16.937 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:16.937 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:16.937 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:56:16.937 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:56:16.937 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:16.937 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:16.937 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:16.937 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:16.937 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:56:16.937 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:56:16.938 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:56:16.938 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:56:16.938 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:56:16.938 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:16.938 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:16.938 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:16.938 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:56:16.938 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:56:16.938 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:16.938 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:56:16.938 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:56:16.938 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:56:16.938 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:16.939 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:16.939 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:56:16.939 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:16.939 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:56:16.939 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:56:16.939 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:56:16.939 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:16.939 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:16.939 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:56:16.939 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:16.939 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:56:16.939 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:56:16.939 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:56:16.939 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:16.939 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:16.940 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:16.940 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:56:16.940 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:56:16.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.916+0000 7f1551039640 1 -- 172.21.15.12:0/3467535325 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f152c03e210 msgr2=0x7f152c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:16.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.916+0000 7f1551039640 1 --2- 172.21.15.12:0/3467535325 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f152c03e210 0x7f152c0406d0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f154c09cf30 tx=0x7f154402f000 comp rx=0 tx=0).stop 2024-09-11T01:56:16.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.916+0000 7f1551039640 10 monclient: shutdown 2024-09-11T01:56:16.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.916+0000 7f1551039640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:16.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.916+0000 7f1551039640 1 -- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f154c137e50 msgr2=0x7f154c1cd2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:16.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.916+0000 7f1551039640 1 --2- 172.21.15.12:0/3467535325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f154c137e50 0x7f154c1cd2d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f153400d800 tx=0x7f153400dcc0 comp rx=0 tx=0).stop 2024-09-11T01:56:16.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.916+0000 7f154b59e640 1 -- 172.21.15.12:0/3467535325 reap_dead start 2024-09-11T01:56:16.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.916+0000 7f154b59e640 1 -- 172.21.15.12:0/3467535325 reap_dead start 2024-09-11T01:56:16.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.916+0000 7f1551039640 1 -- 172.21.15.12:0/3467535325 shutdown_connections 2024-09-11T01:56:16.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.916+0000 7f1551039640 1 -- 172.21.15.12:0/3467535325 >> 172.21.15.12:0/3467535325 conn(0x7f154c0a7020 msgr2=0x7f154c0a8420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:16.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.916+0000 7f1551039640 1 -- 172.21.15.12:0/3467535325 shutdown_connections 2024-09-11T01:56:16.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:16.916+0000 7f1551039640 1 -- 172.21.15.12:0/3467535325 wait complete. 2024-09-11T01:56:16.942 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:56:16.942 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:56:16.942 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:56:17.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.043+0000 7fc03d49b640 10 monclient: get_monmap_and_config 2024-09-11T01:56:17.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.043+0000 7fc03d49b640 10 monclient: build_initial_monmap 2024-09-11T01:56:17.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.043+0000 7fc03d49b640 10 monclient: monmap: 2024-09-11T01:56:17.045 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:17.045 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:17.045 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:56:17.044052+0000 2024-09-11T01:56:17.045 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:56:17.044052+0000 2024-09-11T01:56:17.045 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:17.045 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:17.045 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:56:17.045 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:56:17.046 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:56:17.046 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:56:17.046 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:56:17.046 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:17.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.043+0000 7fc03d49b640 1 Processor -- start 2024-09-11T01:56:17.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.043+0000 7fc03d49b640 1 -- start start 2024-09-11T01:56:17.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.043+0000 7fc03d49b640 10 monclient: init 2024-09-11T01:56:17.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc03d49b640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:17.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc03d49b640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-11T01:56:17.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc03d49b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc0380a2e20 addr v1:172.21.15.71:6789/0 2024-09-11T01:56:17.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc03d49b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc0380a4970 0x7fc03806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:17.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc03d49b640 10 monclient(hunting): picked mon.noname-c con 0x7fc0380a4970 addr v2:172.21.15.71:3300/0 2024-09-11T01:56:17.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc03d49b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc03806f910 0x7fc03813f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:17.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc03d49b640 10 monclient(hunting): picked mon.noname-b con 0x7fc03806f910 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:17.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc03d49b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc03806f4f0 con 0x7fc0380a2e20 2024-09-11T01:56:17.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc03d49b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:17.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc03d49b640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fc03806ef30 con 0x7fc0380a4970 2024-09-11T01:56:17.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc03d49b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:17.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc03d49b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc03806ec50 con 0x7fc03806f910 2024-09-11T01:56:17.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc03d49b640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:17.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc03d49b640 10 monclient(hunting): authenticate will time out at 2383.486504s 2024-09-11T01:56:17.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc0377fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc03806f910 0x7fc03813f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:17.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc0377fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc03806f910 0x7fc03813f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46690/0 (socket says 172.21.15.12:46690) 2024-09-11T01:56:17.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc0377fe640 1 -- 172.21.15.12:0/2469204110 learned_addr learned my addr 172.21.15.12:0/2469204110 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:17.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc0377fe640 10 monclient(hunting): get_auth_request con 0x7fc03806f910 auth_method 0 2024-09-11T01:56:17.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc0377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:17.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc0377fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:17.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc0377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:17.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc0377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:17.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc0377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:17.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc0377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:17.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc0367fc640 1 --2- 172.21.15.12:0/2469204110 >> v2:172.21.15.71:3300/0 conn(0x7fc0380a4970 0x7fc03806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:17.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc0367fc640 10 monclient(hunting): get_auth_request con 0x7fc0380a4970 auth_method 0 2024-09-11T01:56:17.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc0367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:17.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc0367fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:17.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc0367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:17.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc0377fe640 10 monclient(hunting): handle_auth_done global_id 304103 payload 274 2024-09-11T01:56:17.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc0377fe640 1 -- 172.21.15.12:0/2469204110 >> v2:172.21.15.71:3300/0 conn(0x7fc0380a4970 msgr2=0x7fc03806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:17.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc0377fe640 1 --2- 172.21.15.12:0/2469204110 >> v2:172.21.15.71:3300/0 conn(0x7fc0380a4970 0x7fc03806ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:17.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc0377fe640 1 -- 172.21.15.12:0/2469204110 >> v1:172.21.15.71:6789/0 conn(0x7fc0380a2e20 legacy=0x7fc0380a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:17.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc0377fe640 10 monclient: _finish_hunting 0 2024-09-11T01:56:17.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc0377fe640 1 monclient: found mon.noname-b 2024-09-11T01:56:17.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc0377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:17.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc0377fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:56:17.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.044+0000 7fc0377fe640 1 -- 172.21.15.12:0/2469204110 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0380a3910 con 0x7fc03806f910 2024-09-11T01:56:17.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.047+0000 7fc0377fe640 1 -- 172.21.15.12:0/2469204110 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc03806f910 msgr2=0x7fc03813f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:17.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.047+0000 7fc0377fe640 1 --2- 172.21.15.12:0/2469204110 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc03806f910 0x7fc03813f430 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc030009e50 tx=0x7fc03002a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8dac1f1007dc12e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:17.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.047+0000 7fc035ffb640 1 -- 172.21.15.12:0/2469204110 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc030036070 con 0x7fc03806f910 2024-09-11T01:56:17.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.047+0000 7fc035ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:17.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.047+0000 7fc035ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:56:17.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.047+0000 7fc035ffb640 10 monclient: dump: 2024-09-11T01:56:17.059 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:17.059 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:17.059 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.059 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.059 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:17.059 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:17.059 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:17.059 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:17.059 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:17.059 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:17.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.047+0000 7fc035ffb640 10 monclient: _finish_auth 0 2024-09-11T01:56:17.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.047+0000 7fc035ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:56:17.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.047+0000 7fc035ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:17.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.047+0000 7fc03d49b640 5 monclient: authenticate success, global_id 304103 2024-09-11T01:56:17.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.047+0000 7fc03d49b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:56:17.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.048+0000 7fc035ffb640 1 -- 172.21.15.12:0/2469204110 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc030007800 con 0x7fc03806f910 2024-09-11T01:56:17.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.048+0000 7fc035ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:17.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.048+0000 7fc03d49b640 10 monclient: get_monmap_and_config success 2024-09-11T01:56:17.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.048+0000 7fc03d49b640 4 set_mon_vals no callback set 2024-09-11T01:56:17.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.048+0000 7fc035ffb640 1 -- 172.21.15.12:0/2469204110 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc030007bd0 con 0x7fc03806f910 2024-09-11T01:56:17.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.048+0000 7fc035ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:17.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.048+0000 7fc035ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:56:17.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.048+0000 7fc035ffb640 10 monclient: dump: 2024-09-11T01:56:17.060 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:17.061 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:17.061 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.061 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.061 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:17.061 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:17.061 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:17.061 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:17.061 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:17.061 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:17.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.048+0000 7fc03d49b640 10 monclient: shutdown 2024-09-11T01:56:17.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.048+0000 7fc03d49b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:17.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.048+0000 7fc03d49b640 1 -- 172.21.15.12:0/2469204110 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc03806f910 msgr2=0x7fc03813f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:17.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.048+0000 7fc03d49b640 1 --2- 172.21.15.12:0/2469204110 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc03806f910 0x7fc03813f430 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc030009e50 tx=0x7fc03002a560 comp rx=0 tx=0).stop 2024-09-11T01:56:17.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.048+0000 7fc03d49b640 1 -- 172.21.15.12:0/2469204110 shutdown_connections 2024-09-11T01:56:17.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.048+0000 7fc03d49b640 1 --2- 172.21.15.12:0/2469204110 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc03806f910 0x7fc03813f430 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:17.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.048+0000 7fc03d49b640 1 --2- 172.21.15.12:0/2469204110 >> v2:172.21.15.71:3300/0 conn(0x7fc0380a4970 0x7fc03806ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:17.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.048+0000 7fc03d49b640 1 -- 172.21.15.12:0/2469204110 >> 172.21.15.12:0/2469204110 conn(0x7fc0380a8600 msgr2=0x7fc0380a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:17.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.048+0000 7fc03d49b640 1 -- 172.21.15.12:0/2469204110 shutdown_connections 2024-09-11T01:56:17.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.048+0000 7fc03d49b640 1 -- 172.21.15.12:0/2469204110 wait complete. 2024-09-11T01:56:17.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.049+0000 7fc03d49b640 10 monclient: build_initial_monmap 2024-09-11T01:56:17.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.049+0000 7fc03d49b640 10 monclient: monmap: 2024-09-11T01:56:17.062 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:17.062 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:56:17.062 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:56:17.062 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:56:17.062 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:17.063 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:17.063 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:56:17.063 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:56:17.063 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:56:17.063 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:17.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.049+0000 7fc03d49b640 1 Processor -- start 2024-09-11T01:56:17.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.049+0000 7fc03d49b640 1 -- start start 2024-09-11T01:56:17.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.049+0000 7fc03d49b640 10 monclient: init 2024-09-11T01:56:17.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc03d49b640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:17.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc03d49b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:56:17.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc03d49b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc03806f910 0x7fc0381398c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:17.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc03d49b640 10 monclient(hunting): picked mon.noname-b con 0x7fc03806f910 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:56:17.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc03d49b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0380a2e20 0x7fc038139e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:17.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc03d49b640 10 monclient(hunting): picked mon.noname-c con 0x7fc0380a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:17.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc03d49b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0380a4970 0x7fc0381cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:17.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc03d49b640 10 monclient(hunting): picked mon.noname-a con 0x7fc0380a4970 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:56:17.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc03d49b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:17.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc03d49b640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc03813fd00 con 0x7fc0380a4970 2024-09-11T01:56:17.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc03d49b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:17.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc03d49b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc03813fb80 con 0x7fc03806f910 2024-09-11T01:56:17.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc03d49b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:17.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc03d49b640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc03813fe80 con 0x7fc0380a2e20 2024-09-11T01:56:17.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc03d49b640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:17.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc03d49b640 10 monclient(hunting): authenticate will time out at 2383.492115s 2024-09-11T01:56:17.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0367fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0380a2e20 0x7fc038139e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:17.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0377fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0380a4970 0x7fc0381cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:17.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0367fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0380a2e20 0x7fc038139e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46692/0 (socket says 172.21.15.12:46692) 2024-09-11T01:56:17.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0377fe640 10 monclient(hunting): get_auth_request con 0x7fc0380a4970 auth_method 0 2024-09-11T01:56:17.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0367fc640 1 -- 172.21.15.12:0/3940706152 learned_addr learned my addr 172.21.15.12:0/3940706152 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:17.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:17.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0377fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:17.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:17.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0367fc640 10 monclient(hunting): get_auth_request con 0x7fc0380a2e20 auth_method 0 2024-09-11T01:56:17.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:17.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0367fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:17.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:17.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:17.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:17.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:17.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:17.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:17.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:17.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc036ffd640 1 --2- 172.21.15.12:0/3940706152 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc03806f910 0x7fc0381398c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:17.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc036ffd640 10 monclient(hunting): get_auth_request con 0x7fc03806f910 auth_method 0 2024-09-11T01:56:17.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc036ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:17.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc036ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:17.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:17.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0377fe640 10 monclient(hunting): handle_auth_done global_id 294210 payload 824 2024-09-11T01:56:17.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0377fe640 1 -- 172.21.15.12:0/3940706152 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0380a2e20 msgr2=0x7fc038139e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:17.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0377fe640 1 --2- 172.21.15.12:0/3940706152 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0380a2e20 0x7fc038139e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:17.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0377fe640 1 -- 172.21.15.12:0/3940706152 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc03806f910 msgr2=0x7fc0381398c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:17.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0377fe640 1 --2- 172.21.15.12:0/3940706152 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc03806f910 0x7fc0381398c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:17.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0377fe640 10 monclient: _finish_hunting 0 2024-09-11T01:56:17.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0377fe640 1 monclient: found mon.noname-a 2024-09-11T01:56:17.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:17.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0377fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:56:17.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0377fe640 1 -- 172.21.15.12:0/3940706152 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc03813ef90 con 0x7fc0380a4970 2024-09-11T01:56:17.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc0367fc640 1 --2- 172.21.15.12:0/3940706152 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc0380a2e20 0x7fc038139e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:56:17.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.050+0000 7fc036ffd640 1 --2- 172.21.15.12:0/3940706152 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc03806f910 0x7fc0381398c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:56:17.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.052+0000 7fc0377fe640 1 -- 172.21.15.12:0/3940706152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0380a4970 msgr2=0x7fc0381cd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:17.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.052+0000 7fc0377fe640 1 --2- 172.21.15.12:0/3940706152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0380a4970 0x7fc0381cd030 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc030030000 tx=0x7fc030004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:17.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.052+0000 7fc01f7fe640 1 -- 172.21.15.12:0/3940706152 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc030036050 con 0x7fc0380a4970 2024-09-11T01:56:17.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.053+0000 7fc01f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:17.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.053+0000 7fc01f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:56:17.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.053+0000 7fc01f7fe640 10 monclient: dump: 2024-09-11T01:56:17.067 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:17.067 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:17.068 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.068 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.068 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:17.068 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:17.068 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:17.068 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:17.068 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:17.068 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:17.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.053+0000 7fc01f7fe640 10 monclient: _finish_auth 0 2024-09-11T01:56:17.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.053+0000 7fc01f7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:56:17.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.053+0000 7fc01f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:17.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.053+0000 7fc03d49b640 5 monclient: authenticate success, global_id 294210 2024-09-11T01:56:17.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.053+0000 7fc01f7fe640 1 -- 172.21.15.12:0/3940706152 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc030042070 con 0x7fc0380a4970 2024-09-11T01:56:17.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.053+0000 7fc01f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:17.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.053+0000 7fc01f7fe640 1 -- 172.21.15.12:0/3940706152 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc030031d10 con 0x7fc0380a4970 2024-09-11T01:56:17.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.053+0000 7fc03d49b640 10 monclient: _renew_subs 2024-09-11T01:56:17.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.053+0000 7fc03d49b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:17.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.053+0000 7fc03d49b640 1 -- 172.21.15.12:0/3940706152 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0380a5b20 con 0x7fc0380a4970 2024-09-11T01:56:17.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.053+0000 7fc01f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:17.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.053+0000 7fc01f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:56:17.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.053+0000 7fc0357fa640 4 set_mon_vals no callback set 2024-09-11T01:56:17.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.053+0000 7fc01f7fe640 10 monclient: dump: 2024-09-11T01:56:17.069 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:17.069 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:17.069 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.069 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.069 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:17.070 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:17.070 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:17.070 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:17.070 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:17.070 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:17.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.053+0000 7fc03d49b640 10 monclient: _renew_subs 2024-09-11T01:56:17.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.053+0000 7fc03d49b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:17.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.053+0000 7fc03d49b640 1 -- 172.21.15.12:0/3940706152 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc03813bd00 con 0x7fc0380a4970 2024-09-11T01:56:17.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.053+0000 7fc03d49b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:17.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.053+0000 7fc03d49b640 1 --2- 172.21.15.12:0/3940706152 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc03802d440 0x7fc03802d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:17.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.053+0000 7fc03d49b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:17.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.053+0000 7fc03d49b640 1 -- 172.21.15.12:0/3940706152 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc03806a060 con 0x7fc03802d440 2024-09-11T01:56:17.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.054+0000 7fc01f7fe640 1 -- 172.21.15.12:0/3940706152 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fc030047460 con 0x7fc0380a4970 2024-09-11T01:56:17.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.054+0000 7fc036ffd640 1 --2- 172.21.15.12:0/3940706152 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc03802d440 0x7fc03802d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:17.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.054+0000 7fc036ffd640 10 monclient: get_auth_request con 0x7fc03802d440 auth_method 0 2024-09-11T01:56:17.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.054+0000 7fc036ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:17.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.054+0000 7fc036ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:17.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.054+0000 7fc036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:17.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.054+0000 7fc036ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:17.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.054+0000 7fc036ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:17.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.054+0000 7fc036ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:17.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.054+0000 7fc036ffd640 10 monclient(hunting): handle_auth_done global_id 304109 payload 824 2024-09-11T01:56:17.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.054+0000 7fc01f7fe640 1 --2- 172.21.15.12:0/3940706152 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc02403e260 0x7fc024040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:17.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.054+0000 7fc036ffd640 1 -- 172.21.15.12:0/3940706152 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc03806ded0 con 0x7fc03802d440 2024-09-11T01:56:17.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.054+0000 7fc01f7fe640 1 -- 172.21.15.12:0/3940706152 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(917..917 src has 676..917) ==== 4755+0+0 (secure 0 0 0) 0x7fc030073c70 con 0x7fc0380a4970 2024-09-11T01:56:17.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.054+0000 7fc0367fc640 1 --2- 172.21.15.12:0/3940706152 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc02403e260 0x7fc024040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:17.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.054+0000 7fc0367fc640 10 monclient: get_auth_request con 0x7fc02403e260 auth_method 0 2024-09-11T01:56:17.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.056+0000 7fc036ffd640 1 -- 172.21.15.12:0/3940706152 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc03802d440 msgr2=0x7fc03802d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:17.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.056+0000 7fc036ffd640 1 --2- 172.21.15.12:0/3940706152 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc03802d440 0x7fc03802d820 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc02800e500 tx=0x7fc02800e9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:17.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.056+0000 7fc01f7fe640 1 -- 172.21.15.12:0/3940706152 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc028013800 con 0x7fc03802d440 2024-09-11T01:56:17.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.056+0000 7fc01f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:17.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.056+0000 7fc01f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:56:17.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.056+0000 7fc01f7fe640 10 monclient: dump: 2024-09-11T01:56:17.072 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:17.072 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:17.072 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.072 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.072 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:17.072 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:17.073 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:17.073 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:17.073 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:17.073 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:17.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.056+0000 7fc01f7fe640 1 -- 172.21.15.12:0/3940706152 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc028013800 con 0x7fc03802d440 2024-09-11T01:56:17.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.056+0000 7fc01f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:17.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.056+0000 7fc01f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:56:17.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.056+0000 7fc01f7fe640 1 -- 172.21.15.12:0/3940706152 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc03802d440 msgr2=0x7fc03802d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:17.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.056+0000 7fc01f7fe640 1 --2- 172.21.15.12:0/3940706152 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc03802d440 0x7fc03802d820 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc02800e500 tx=0x7fc02800e9c0 comp rx=0 tx=0).stop 2024-09-11T01:56:17.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.056+0000 7fc01f7fe640 1 -- 172.21.15.12:0/3940706152 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc03802d440 msgr2=0x7fc03802d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:56:17.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.056+0000 7fc01f7fe640 1 --2- 172.21.15.12:0/3940706152 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc03802d440 0x7fc03802d820 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc02800e500 tx=0x7fc02800e9c0 comp rx=0 tx=0).stop 2024-09-11T01:56:17.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.057+0000 7fc0367fc640 1 --2- 172.21.15.12:0/3940706152 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc02403e260 0x7fc024040720 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fc038074830 tx=0x7fc02c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:17.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.063+0000 7fc03d49b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:56:17.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.063+0000 7fc03d49b640 1 --2- 172.21.15.12:0/3940706152 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc03802dd60 0x7fc03802e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:17.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.063+0000 7fc03d49b640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:17.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.063+0000 7fc03d49b640 1 -- 172.21.15.12:0/3940706152 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc0381378d0 con 0x7fc03802dd60 2024-09-11T01:56:17.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.063+0000 7fc036ffd640 1 --2- 172.21.15.12:0/3940706152 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc03802dd60 0x7fc03802e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:17.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.063+0000 7fc036ffd640 10 monclient: get_auth_request con 0x7fc03802dd60 auth_method 0 2024-09-11T01:56:17.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.063+0000 7fc036ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:17.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.063+0000 7fc036ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:17.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.063+0000 7fc036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:17.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.063+0000 7fc036ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:17.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.063+0000 7fc036ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:17.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.063+0000 7fc036ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:17.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.063+0000 7fc036ffd640 10 monclient(hunting): handle_auth_done global_id 304112 payload 824 2024-09-11T01:56:17.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.063+0000 7fc036ffd640 1 -- 172.21.15.12:0/3940706152 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc03813a340 con 0x7fc03802dd60 2024-09-11T01:56:17.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.065+0000 7fc036ffd640 1 -- 172.21.15.12:0/3940706152 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc03802dd60 msgr2=0x7fc03802e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:17.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.065+0000 7fc036ffd640 1 --2- 172.21.15.12:0/3940706152 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc03802dd60 0x7fc03802e140 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc02800e500 tx=0x7fc028003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:17.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.065+0000 7fc01f7fe640 1 -- 172.21.15.12:0/3940706152 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0280069d0 con 0x7fc03802dd60 2024-09-11T01:56:17.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.065+0000 7fc01f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:17.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.065+0000 7fc01f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:56:17.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.065+0000 7fc01f7fe640 10 monclient: dump: 2024-09-11T01:56:17.077 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:17.077 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:17.077 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.077 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.077 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:17.077 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:17.077 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:17.077 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:17.077 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:17.077 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:17.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.065+0000 7fc01f7fe640 1 -- 172.21.15.12:0/3940706152 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7fc028007680 con 0x7fc03802dd60 2024-09-11T01:56:17.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.065+0000 7fc01f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:56:17.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.065+0000 7fc01f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:56:17.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.065+0000 7fc01f7fe640 1 -- 172.21.15.12:0/3940706152 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc03802dd60 msgr2=0x7fc03802e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:17.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.065+0000 7fc01f7fe640 1 --2- 172.21.15.12:0/3940706152 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc03802dd60 0x7fc03802e140 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc02800e500 tx=0x7fc028003760 comp rx=0 tx=0).stop 2024-09-11T01:56:17.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.065+0000 7fc01f7fe640 1 -- 172.21.15.12:0/3940706152 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc03802dd60 msgr2=0x7fc03802e140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:56:17.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.066+0000 7fc01f7fe640 1 --2- 172.21.15.12:0/3940706152 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc03802dd60 0x7fc03802e140 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc02800e500 tx=0x7fc028003760 comp rx=0 tx=0).stop 2024-09-11T01:56:17.078 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:56:17.078 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:56:17.078 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:56:17.078 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:56:17.079 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 250, 2024-09-11T01:56:17.079 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 12020, 2024-09-11T01:56:17.079 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:56:17.079 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:56:17.079 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:56:17.079 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:56:17.079 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:17.079 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 2, 2024-09-11T01:56:17.079 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:56:17.079 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:56:17.079 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:56:17.079 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:56:17.079 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:56:17.080 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:56:17.080 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:56:17.080 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:56:17.080 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:56:17.080 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:56:17.080 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:56:17.080 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:56:17.080 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:56:17.080 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:56:17.080 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:17.080 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:56:17.080 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:56:17.081 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:56:17.081 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:56:17.081 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:56:17.081 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:56:17.081 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:56:17.081 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:56:17.081 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:56:17.081 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:56:17.081 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:56:17.081 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:56:17.081 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:56:17.081 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:17.081 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:17.082 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:56:17.082 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:56:17.082 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:56:17.082 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:56:17.082 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:56:17.082 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:56:17.082 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:56:17.082 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:56:17.082 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:56:17.082 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:56:17.082 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:56:17.082 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:56:17.082 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:56:17.082 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:56:17.083 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:56:17.083 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:56:17.083 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:56:17.083 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:56:17.083 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:56:17.083 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:56:17.083 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:56:17.083 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:56:17.083 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:56:17.083 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:56:17.083 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:56:17.083 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:56:17.083 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:56:17.083 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:56:17.083 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:17.083 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:56:17.084 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:17.084 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:56:17.084 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:17.084 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:56:17.084 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:56:17.084 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:56:17.084 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:56:17.084 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:17.084 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:56:17.084 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:56:17.084 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:17.084 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:17.084 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:17.084 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:56:17.085 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:56:17.085 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:17.085 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:17.085 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:17.085 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:17.085 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:56:17.085 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:56:17.085 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:56:17.085 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:56:17.085 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:56:17.085 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:17.085 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:17.085 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:56:17.085 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:56:17.086 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:56:17.086 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:56:17.086 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:17.086 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:56:17.086 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:56:17.086 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:17.086 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:17.086 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:17.086 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:56:17.086 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:56:17.086 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:17.086 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:17.086 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:17.086 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:17.087 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:56:17.087 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:56:17.087 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:56:17.087 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:56:17.087 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:56:17.087 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:17.087 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:17.087 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:56:17.087 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:56:17.087 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:56:17.087 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:56:17.087 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:17.087 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:56:17.087 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:56:17.088 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:17.088 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:17.088 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:17.088 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:56:17.088 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:56:17.088 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:17.088 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:17.088 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:17.088 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:17.088 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:56:17.088 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:56:17.088 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:56:17.089 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:56:17.089 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:56:17.089 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:17.089 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:17.089 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:17.089 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:56:17.089 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:56:17.089 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:17.089 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:56:17.089 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:56:17.089 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:56:17.089 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:17.089 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:17.089 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:56:17.090 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:17.090 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:56:17.090 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:56:17.090 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:56:17.090 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:17.090 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:17.090 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:17.090 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:56:17.090 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:56:17.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.066+0000 7fc03d49b640 1 -- 172.21.15.12:0/3940706152 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc02403e260 msgr2=0x7fc024040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:17.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.066+0000 7fc03d49b640 1 --2- 172.21.15.12:0/3940706152 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc02403e260 0x7fc024040720 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fc038074830 tx=0x7fc02c008040 comp rx=0 tx=0).stop 2024-09-11T01:56:17.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.066+0000 7fc03d49b640 10 monclient: shutdown 2024-09-11T01:56:17.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.066+0000 7fc03d49b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:17.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.066+0000 7fc03d49b640 1 -- 172.21.15.12:0/3940706152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0380a4970 msgr2=0x7fc0381cd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:17.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.066+0000 7fc03d49b640 1 --2- 172.21.15.12:0/3940706152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc0380a4970 0x7fc0381cd030 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc030030000 tx=0x7fc030004a40 comp rx=0 tx=0).stop 2024-09-11T01:56:17.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.066+0000 7fc0377fe640 1 -- 172.21.15.12:0/3940706152 reap_dead start 2024-09-11T01:56:17.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.066+0000 7fc0377fe640 1 -- 172.21.15.12:0/3940706152 reap_dead start 2024-09-11T01:56:17.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.066+0000 7fc03d49b640 1 -- 172.21.15.12:0/3940706152 shutdown_connections 2024-09-11T01:56:17.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.066+0000 7fc03d49b640 1 -- 172.21.15.12:0/3940706152 >> 172.21.15.12:0/3940706152 conn(0x7fc0380a8600 msgr2=0x7fc0380a1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:17.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.066+0000 7fc03d49b640 1 -- 172.21.15.12:0/3940706152 shutdown_connections 2024-09-11T01:56:17.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.066+0000 7fc03d49b640 1 -- 172.21.15.12:0/3940706152 wait complete. 2024-09-11T01:56:17.092 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:56:17.093 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:56:17.093 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:56:17.180 INFO:tasks.rados.rados.0.smithi071.stdout:3986: done (5 left) 2024-09-11T01:56:17.180 INFO:tasks.rados.rados.0.smithi071.stdout:3987: write oid 14 current snap is 451 2024-09-11T01:56:17.180 INFO:tasks.rados.rados.0.smithi071.stdout:3980: finishing write tid 3 to smithi07134777-27 2024-09-11T01:56:17.181 INFO:tasks.rados.rados.0.smithi071.stdout:3980: finishing write tid 4 to smithi07134777-27 2024-09-11T01:56:17.181 INFO:tasks.rados.rados.0.smithi071.stdout:3980: finishing write tid 5 to smithi07134777-27 2024-09-11T01:56:17.181 INFO:tasks.rados.rados.0.smithi071.stdout:3980: oid 27 updating version 0 to 3162 2024-09-11T01:56:17.181 INFO:tasks.rados.rados.0.smithi071.stdout:3980: oid 27 version 3162 is already newer than 3160 2024-09-11T01:56:17.181 INFO:tasks.rados.rados.0.smithi071.stdout:3980: oid 27 version 3162 is already newer than 3161 2024-09-11T01:56:17.181 INFO:tasks.rados.rados.0.smithi071.stdout:3980: oid 27 updating version 3162 to 3163 2024-09-11T01:56:17.181 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 3163 (ObjNum 1366 snap 451 seq_num 1366) dirty exists 2024-09-11T01:56:17.181 INFO:tasks.rados.rados.0.smithi071.stdout:3980: left oid 27 (ObjNum 1366 snap 451 seq_num 1366) 2024-09-11T01:56:17.181 INFO:tasks.rados.rados.0.smithi071.stdout:3982: expect (ObjNum 1361 snap 451 seq_num 1361) 2024-09-11T01:56:17.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.199+0000 7f25c0a7f640 10 monclient: get_monmap_and_config 2024-09-11T01:56:17.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.199+0000 7f25c0a7f640 10 monclient: build_initial_monmap 2024-09-11T01:56:17.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.199+0000 7f25c0a7f640 10 monclient: monmap: 2024-09-11T01:56:17.201 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:17.201 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:17.201 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:56:17.200229+0000 2024-09-11T01:56:17.201 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:56:17.200229+0000 2024-09-11T01:56:17.201 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:17.201 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:17.201 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:56:17.201 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:56:17.201 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:56:17.201 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:56:17.202 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:56:17.202 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:17.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.200+0000 7f25c0a7f640 1 Processor -- start 2024-09-11T01:56:17.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.200+0000 7f25c0a7f640 1 -- start start 2024-09-11T01:56:17.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.200+0000 7f25c0a7f640 10 monclient: init 2024-09-11T01:56:17.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.200+0000 7f25c0a7f640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:17.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.200+0000 7f25c0a7f640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-11T01:56:17.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.200+0000 7f25c0a7f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f25bc0a2db0 addr v1:172.21.15.71:6789/0 2024-09-11T01:56:17.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.200+0000 7f25c0a7f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f25bc0a38a0 0x7f25bc06f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:17.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.200+0000 7f25c0a7f640 10 monclient(hunting): picked mon.noname-b con 0x7f25bc0a38a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:17.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.200+0000 7f25c0a7f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f25bc06ded0 addr v1:172.21.15.12:6789/0 2024-09-11T01:56:17.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.200+0000 7f25c0a7f640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25bc06e320 con 0x7f25bc06ded0 2024-09-11T01:56:17.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.200+0000 7f25c0a7f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25bc0a4650 con 0x7f25bc0a2db0 2024-09-11T01:56:17.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.200+0000 7f25c0a7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:17.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.200+0000 7f25c0a7f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f25bc06f1a0 con 0x7f25bc0a38a0 2024-09-11T01:56:17.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.201+0000 7f25c0a7f640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:17.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.201+0000 7f25c0a7f640 10 monclient(hunting): authenticate will time out at 2383.643020s 2024-09-11T01:56:17.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.201+0000 7f25bad9d640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f25bc06ded0 0x7f25bc13f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43474/0 (socket says 172.21.15.12:43474) 2024-09-11T01:56:17.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.201+0000 7f25bad9d640 1 -- 172.21.15.12:0/3500729169 learned_addr learned my addr 172.21.15.12:0/3500729169 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:56:17.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.201+0000 7f25b9d9b640 1 --2- 172.21.15.12:0/3500729169 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f25bc0a38a0 0x7f25bc06f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:17.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.201+0000 7f25b9d9b640 10 monclient(hunting): get_auth_request con 0x7f25bc0a38a0 auth_method 0 2024-09-11T01:56:17.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.201+0000 7f25b9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:17.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.201+0000 7f25b9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:17.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.201+0000 7f25b9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:17.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.201+0000 7f25b9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:17.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.201+0000 7f25b9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:17.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.201+0000 7f25b9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:17.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.201+0000 7f25b9d9b640 10 monclient(hunting): handle_auth_done global_id 304115 payload 274 2024-09-11T01:56:17.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.201+0000 7f25b9d9b640 1 -- 172.21.15.12:0/3500729169 >> v1:172.21.15.71:6789/0 conn(0x7f25bc0a2db0 legacy=0x7f25bc0a3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:17.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.201+0000 7f25b9d9b640 1 -- 172.21.15.12:0/3500729169 >> v1:172.21.15.12:6789/0 conn(0x7f25bc06ded0 legacy=0x7f25bc13f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:17.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.201+0000 7f25b9d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:56:17.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.201+0000 7f25b9d9b640 1 monclient: found mon.noname-b 2024-09-11T01:56:17.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.201+0000 7f25b9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:17.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.201+0000 7f25b9d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:56:17.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.201+0000 7f25b9d9b640 1 -- 172.21.15.12:0/3500729169 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25bc06ebe0 con 0x7f25bc0a38a0 2024-09-11T01:56:17.207 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1600158 2024-09-11T01:56:17.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.204+0000 7f25b9d9b640 1 -- 172.21.15.12:0/3500729169 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f25bc0a38a0 msgr2=0x7f25bc06f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:17.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.204+0000 7f25b9d9b640 1 --2- 172.21.15.12:0/3500729169 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f25bc0a38a0 0x7f25bc06f8a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f25ac0097d0 tx=0x7f25ac02a030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db9fd65446e4a3b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:17.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.204+0000 7f25b959a640 1 -- 172.21.15.12:0/3500729169 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ac037070 con 0x7f25bc0a38a0 2024-09-11T01:56:17.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.204+0000 7f25b959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:17.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.204+0000 7f25b959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:56:17.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.204+0000 7f25b959a640 10 monclient: dump: 2024-09-11T01:56:17.213 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:17.213 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:17.213 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.213 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.213 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:17.213 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:17.214 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:17.214 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:17.214 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:17.214 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:17.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.204+0000 7f25b959a640 10 monclient: _finish_auth 0 2024-09-11T01:56:17.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.204+0000 7f25b959a640 10 monclient: _check_auth_tickets 2024-09-11T01:56:17.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.204+0000 7f25b959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:17.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.204+0000 7f25b959a640 1 -- 172.21.15.12:0/3500729169 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25ac032640 con 0x7f25bc0a38a0 2024-09-11T01:56:17.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.204+0000 7f25b959a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:17.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.204+0000 7f25b959a640 1 -- 172.21.15.12:0/3500729169 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ac032a10 con 0x7f25bc0a38a0 2024-09-11T01:56:17.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.204+0000 7f25b959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:17.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.204+0000 7f25b959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:56:17.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.204+0000 7f25b959a640 10 monclient: dump: 2024-09-11T01:56:17.215 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:17.215 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:17.215 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.215 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.215 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:17.215 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:17.215 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:17.215 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:17.215 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:17.215 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:17.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.204+0000 7f25c0a7f640 5 monclient: authenticate success, global_id 304115 2024-09-11T01:56:17.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.204+0000 7f25c0a7f640 10 monclient: get_monmap_and_config success 2024-09-11T01:56:17.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.204+0000 7f25c0a7f640 4 set_mon_vals no callback set 2024-09-11T01:56:17.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.205+0000 7f25c0a7f640 10 monclient: shutdown 2024-09-11T01:56:17.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.205+0000 7f25c0a7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:17.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.205+0000 7f25c0a7f640 1 -- 172.21.15.12:0/3500729169 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f25bc0a38a0 msgr2=0x7f25bc06f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:17.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.205+0000 7f25c0a7f640 1 --2- 172.21.15.12:0/3500729169 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f25bc0a38a0 0x7f25bc06f8a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f25ac0097d0 tx=0x7f25ac02a030 comp rx=0 tx=0).stop 2024-09-11T01:56:17.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.205+0000 7f25c0a7f640 1 -- 172.21.15.12:0/3500729169 shutdown_connections 2024-09-11T01:56:17.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.205+0000 7f25c0a7f640 1 --2- 172.21.15.12:0/3500729169 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f25bc0a38a0 0x7f25bc06f8a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:17.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.205+0000 7f25c0a7f640 1 -- 172.21.15.12:0/3500729169 >> 172.21.15.12:0/3500729169 conn(0x7f25bc0a83b0 msgr2=0x7f25bc0a87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:17.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.205+0000 7f25c0a7f640 1 -- 172.21.15.12:0/3500729169 shutdown_connections 2024-09-11T01:56:17.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.205+0000 7f25c0a7f640 1 -- 172.21.15.12:0/3500729169 wait complete. 2024-09-11T01:56:17.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25c0a7f640 10 monclient: build_initial_monmap 2024-09-11T01:56:17.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25c0a7f640 10 monclient: monmap: 2024-09-11T01:56:17.217 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:17.217 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:56:17.217 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:56:17.217 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:56:17.217 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:17.217 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:17.217 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:56:17.217 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:56:17.217 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:56:17.218 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:17.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25c0a7f640 1 Processor -- start 2024-09-11T01:56:17.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25c0a7f640 1 -- start start 2024-09-11T01:56:17.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25c0a7f640 10 monclient: init 2024-09-11T01:56:17.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25c0a7f640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:17.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25c0a7f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:56:17.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25c0a7f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25bc06ded0 0x7f25bc138490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:17.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25c0a7f640 10 monclient(hunting): picked mon.noname-a con 0x7f25bc06ded0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:56:17.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25c0a7f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f25bc0a2db0 0x7f25bc1389d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:17.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25c0a7f640 10 monclient(hunting): picked mon.noname-c con 0x7f25bc0a2db0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:17.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25c0a7f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25bc0a38a0 0x7f25bc1cd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:17.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25c0a7f640 10 monclient(hunting): picked mon.noname-b con 0x7f25bc0a38a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:56:17.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25c0a7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:17.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25c0a7f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f25bc140e70 con 0x7f25bc06ded0 2024-09-11T01:56:17.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25c0a7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:17.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25c0a7f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f25bc140cf0 con 0x7f25bc0a38a0 2024-09-11T01:56:17.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25c0a7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:17.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25c0a7f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f25bc140ff0 con 0x7f25bc0a2db0 2024-09-11T01:56:17.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25c0a7f640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:17.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25c0a7f640 10 monclient(hunting): authenticate will time out at 2383.648882s 2024-09-11T01:56:17.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25ba59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25bc06ded0 0x7f25bc138490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:17.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25b9d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f25bc0a2db0 0x7f25bc1389d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:17.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25b9d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f25bc0a2db0 0x7f25bc1389d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46720/0 (socket says 172.21.15.12:46720) 2024-09-11T01:56:17.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25ba59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25bc06ded0 0x7f25bc138490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49740/0 (socket says 172.21.15.12:49740) 2024-09-11T01:56:17.219 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25b9d9b640 1 -- 172.21.15.12:0/747619335 learned_addr learned my addr 172.21.15.12:0/747619335 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:17.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.206+0000 7f25b9d9b640 10 monclient(hunting): get_auth_request con 0x7f25bc0a2db0 auth_method 0 2024-09-11T01:56:17.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25b9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:17.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25b9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:17.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25b9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:17.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25ba59c640 10 monclient(hunting): get_auth_request con 0x7f25bc06ded0 auth_method 0 2024-09-11T01:56:17.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25ba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:17.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25ba59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:17.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25ba59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:17.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25b9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:17.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25b9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:17.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25b9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:17.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25ba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:17.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25ba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:17.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25bad9d640 1 --2- 172.21.15.12:0/747619335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25bc0a38a0 0x7f25bc1cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:17.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25ba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:17.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25bad9d640 10 monclient(hunting): get_auth_request con 0x7f25bc0a38a0 auth_method 0 2024-09-11T01:56:17.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25bad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:17.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25bad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:17.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25bad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:17.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25bad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:17.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25bad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:17.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25bad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:17.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25b9d9b640 10 monclient(hunting): handle_auth_done global_id 304118 payload 824 2024-09-11T01:56:17.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25b9d9b640 1 -- 172.21.15.12:0/747619335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25bc0a38a0 msgr2=0x7f25bc1cd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:17.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25b9d9b640 1 --2- 172.21.15.12:0/747619335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25bc0a38a0 0x7f25bc1cd280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:17.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25b9d9b640 1 -- 172.21.15.12:0/747619335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25bc06ded0 msgr2=0x7f25bc138490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:17.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25b9d9b640 1 --2- 172.21.15.12:0/747619335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25bc06ded0 0x7f25bc138490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:17.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25b9d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:56:17.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25b9d9b640 1 monclient: found mon.noname-c 2024-09-11T01:56:17.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25b9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:17.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25b9d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:56:17.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25b9d9b640 1 -- 172.21.15.12:0/747619335 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25bc138f10 con 0x7f25bc0a2db0 2024-09-11T01:56:17.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.207+0000 7f25ba59c640 1 --2- 172.21.15.12:0/747619335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25bc06ded0 0x7f25bc138490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:56:17.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.209+0000 7f25b9d9b640 1 -- 172.21.15.12:0/747619335 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f25bc0a2db0 msgr2=0x7f25bc1389d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:17.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.209+0000 7f25b9d9b640 1 --2- 172.21.15.12:0/747619335 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f25bc0a2db0 0x7f25bc1389d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f25ac035040 tx=0x7f25ac0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:17.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.209+0000 7f25b2ffd640 1 -- 172.21.15.12:0/747619335 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ac042020 con 0x7f25bc0a2db0 2024-09-11T01:56:17.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.209+0000 7f25b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:17.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.209+0000 7f25b2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:56:17.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.209+0000 7f25b2ffd640 10 monclient: dump: 2024-09-11T01:56:17.223 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:17.223 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:17.223 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.223 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.223 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:17.223 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:17.223 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:17.223 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:17.223 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:17.223 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:17.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.209+0000 7f25b2ffd640 10 monclient: _finish_auth 0 2024-09-11T01:56:17.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.209+0000 7f25b2ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:56:17.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.209+0000 7f25b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:17.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.209+0000 7f25b2ffd640 1 -- 172.21.15.12:0/747619335 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25ac030900 con 0x7f25bc0a2db0 2024-09-11T01:56:17.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.209+0000 7f25c0a7f640 5 monclient: authenticate success, global_id 304118 2024-09-11T01:56:17.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.209+0000 7f25b2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:17.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.209+0000 7f25b2ffd640 1 -- 172.21.15.12:0/747619335 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ac037040 con 0x7f25bc0a2db0 2024-09-11T01:56:17.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.209+0000 7f25c0a7f640 10 monclient: _renew_subs 2024-09-11T01:56:17.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.209+0000 7f25c0a7f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:56:17.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.209+0000 7f25c0a7f640 1 -- 172.21.15.12:0/747619335 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25bc09f5a0 con 0x7f25bc0a2db0 2024-09-11T01:56:17.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.209+0000 7f25b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:17.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.209+0000 7f25b3fff640 4 set_mon_vals no callback set 2024-09-11T01:56:17.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.209+0000 7f25b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:56:17.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.209+0000 7f25b2ffd640 10 monclient: dump: 2024-09-11T01:56:17.224 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:17.224 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:17.224 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.225 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.225 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:17.225 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:17.225 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:17.225 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:17.225 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:17.225 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:17.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.209+0000 7f25c0a7f640 10 monclient: _renew_subs 2024-09-11T01:56:17.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.209+0000 7f25c0a7f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:56:17.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.209+0000 7f25c0a7f640 1 -- 172.21.15.12:0/747619335 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f25bc13d090 con 0x7f25bc0a2db0 2024-09-11T01:56:17.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.210+0000 7f25b2ffd640 1 -- 172.21.15.12:0/747619335 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f25ac03d470 con 0x7f25bc0a2db0 2024-09-11T01:56:17.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.210+0000 7f25c0a7f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:17.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.210+0000 7f25c0a7f640 1 --2- 172.21.15.12:0/747619335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25bc02d440 0x7f25bc02d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:17.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.210+0000 7f25c0a7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:17.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.210+0000 7f25c0a7f640 1 -- 172.21.15.12:0/747619335 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f25bc06a060 con 0x7f25bc02d440 2024-09-11T01:56:17.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.210+0000 7f25b2ffd640 1 --2- 172.21.15.12:0/747619335 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f259c03e210 0x7f259c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:17.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.210+0000 7f25b2ffd640 1 -- 172.21.15.12:0/747619335 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(918..918 src has 676..918) ==== 4799+0+0 (secure 0 0 0) 0x7f25ac073a00 con 0x7f25bc0a2db0 2024-09-11T01:56:17.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.210+0000 7f25ba59c640 1 --2- 172.21.15.12:0/747619335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25bc02d440 0x7f25bc02d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:17.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.211+0000 7f25ba59c640 10 monclient: get_auth_request con 0x7f25bc02d440 auth_method 0 2024-09-11T01:56:17.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.211+0000 7f25ba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:17.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.211+0000 7f25ba59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:17.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.211+0000 7f25ba59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:17.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.211+0000 7f25bad9d640 1 --2- 172.21.15.12:0/747619335 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f259c03e210 0x7f259c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:17.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.211+0000 7f25ba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:17.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.211+0000 7f25ba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:17.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.211+0000 7f25ba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:17.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.211+0000 7f25bad9d640 10 monclient: get_auth_request con 0x7f259c03e210 auth_method 0 2024-09-11T01:56:17.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.211+0000 7f25ba59c640 10 monclient(hunting): handle_auth_done global_id 274366 payload 824 2024-09-11T01:56:17.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.211+0000 7f25ba59c640 1 -- 172.21.15.12:0/747619335 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f25bc06f8a0 con 0x7f25bc02d440 2024-09-11T01:56:17.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.213+0000 7f25ba59c640 1 -- 172.21.15.12:0/747619335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25bc02d440 msgr2=0x7f25bc02d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:17.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.213+0000 7f25ba59c640 1 --2- 172.21.15.12:0/747619335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25bc02d440 0x7f25bc02d820 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f25a400fb10 tx=0x7f25a40044c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:17.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.213+0000 7f25bad9d640 1 --2- 172.21.15.12:0/747619335 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f259c03e210 0x7f259c0406d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f25bc13ea30 tx=0x7f25b4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:17.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.213+0000 7f25b2ffd640 1 -- 172.21.15.12:0/747619335 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a4018020 con 0x7f25bc02d440 2024-09-11T01:56:17.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.213+0000 7f25b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:17.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.213+0000 7f25b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:56:17.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.213+0000 7f25b2ffd640 10 monclient: dump: 2024-09-11T01:56:17.229 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:17.229 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:17.229 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.230 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.230 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:17.230 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:17.230 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:17.230 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:17.230 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:17.230 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:17.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.214+0000 7f25b2ffd640 1 -- 172.21.15.12:0/747619335 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f25a400dd90 con 0x7f25bc02d440 2024-09-11T01:56:17.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.214+0000 7f25b2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:17.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.214+0000 7f25b2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:56:17.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.214+0000 7f25b2ffd640 1 -- 172.21.15.12:0/747619335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25bc02d440 msgr2=0x7f25bc02d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:17.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.214+0000 7f25b2ffd640 1 --2- 172.21.15.12:0/747619335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25bc02d440 0x7f25bc02d820 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f25a400fb10 tx=0x7f25a40044c0 comp rx=0 tx=0).stop 2024-09-11T01:56:17.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.214+0000 7f25b2ffd640 1 -- 172.21.15.12:0/747619335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25bc02d440 msgr2=0x7f25bc02d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:56:17.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.214+0000 7f25b2ffd640 1 --2- 172.21.15.12:0/747619335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25bc02d440 0x7f25bc02d820 secure :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f25a400fb10 tx=0x7f25a40044c0 comp rx=0 tx=0).stop 2024-09-11T01:56:17.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.220+0000 7f25c0a7f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:56:17.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.220+0000 7f25c0a7f640 1 --2- 172.21.15.12:0/747619335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25bc02dd60 0x7f25bc02e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:17.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.220+0000 7f25c0a7f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:17.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.220+0000 7f25c0a7f640 1 -- 172.21.15.12:0/747619335 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f25bc13dba0 con 0x7f25bc02dd60 2024-09-11T01:56:17.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.220+0000 7f25ba59c640 1 --2- 172.21.15.12:0/747619335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25bc02dd60 0x7f25bc02e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:17.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.220+0000 7f25ba59c640 10 monclient: get_auth_request con 0x7f25bc02dd60 auth_method 0 2024-09-11T01:56:17.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.220+0000 7f25ba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:17.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.220+0000 7f25ba59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:17.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.220+0000 7f25ba59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:17.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.220+0000 7f25ba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:17.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.220+0000 7f25ba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:17.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.220+0000 7f25ba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:17.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.220+0000 7f25ba59c640 10 monclient(hunting): handle_auth_done global_id 274369 payload 824 2024-09-11T01:56:17.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.220+0000 7f25ba59c640 1 -- 172.21.15.12:0/747619335 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f25bc0a0e80 con 0x7f25bc02dd60 2024-09-11T01:56:17.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.223+0000 7f25ba59c640 1 -- 172.21.15.12:0/747619335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25bc02dd60 msgr2=0x7f25bc02e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:17.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.223+0000 7f25ba59c640 1 --2- 172.21.15.12:0/747619335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25bc02dd60 0x7f25bc02e170 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f25a4019000 tx=0x7f25a40036f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:17.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.223+0000 7f25b2ffd640 1 -- 172.21.15.12:0/747619335 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a4018030 con 0x7f25bc02dd60 2024-09-11T01:56:17.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.223+0000 7f25b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:17.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.223+0000 7f25b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:56:17.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.223+0000 7f25b2ffd640 10 monclient: dump: 2024-09-11T01:56:17.232 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:17.232 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:17.232 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.232 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.233 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:17.233 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:17.233 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:17.233 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:17.233 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:17.233 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:17.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.223+0000 7f25b2ffd640 1 -- 172.21.15.12:0/747619335 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f25a4014b80 con 0x7f25bc02dd60 2024-09-11T01:56:17.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.223+0000 7f25b2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:56:17.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.223+0000 7f25b2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:56:17.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.223+0000 7f25b2ffd640 1 -- 172.21.15.12:0/747619335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25bc02dd60 msgr2=0x7f25bc02e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:17.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.223+0000 7f25b2ffd640 1 --2- 172.21.15.12:0/747619335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25bc02dd60 0x7f25bc02e170 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f25a4019000 tx=0x7f25a40036f0 comp rx=0 tx=0).stop 2024-09-11T01:56:17.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.223+0000 7f25b2ffd640 1 -- 172.21.15.12:0/747619335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25bc02dd60 msgr2=0x7f25bc02e170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:56:17.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.223+0000 7f25b2ffd640 1 --2- 172.21.15.12:0/747619335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25bc02dd60 0x7f25bc02e170 secure :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f25a4019000 tx=0x7f25a40036f0 comp rx=0 tx=0).stop 2024-09-11T01:56:17.234 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:56:17.234 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:56:17.234 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:56:17.234 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:56:17.234 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 250, 2024-09-11T01:56:17.234 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 167041, 2024-09-11T01:56:17.234 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:56:17.234 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:56:17.234 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:56:17.235 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:56:17.235 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:17.235 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 3, 2024-09-11T01:56:17.235 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:56:17.235 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:56:17.235 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:56:17.235 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:56:17.235 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:56:17.235 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:56:17.235 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:56:17.235 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:56:17.235 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:56:17.235 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:56:17.235 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:56:17.236 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:56:17.236 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:56:17.236 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:56:17.236 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:17.236 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:56:17.236 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:56:17.236 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:56:17.236 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:56:17.236 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:56:17.236 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:56:17.236 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:56:17.236 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:56:17.236 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:56:17.237 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:56:17.237 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:56:17.237 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:56:17.237 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:56:17.237 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:17.237 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:17.237 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:56:17.237 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:56:17.237 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:56:17.237 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:56:17.237 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:56:17.237 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:56:17.237 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:56:17.237 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:56:17.238 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:56:17.238 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:56:17.238 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:56:17.238 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:56:17.238 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:56:17.238 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:56:17.238 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:56:17.238 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:56:17.238 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:56:17.238 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:56:17.238 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:56:17.238 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:56:17.238 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:56:17.239 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:56:17.239 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:56:17.239 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:56:17.239 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:56:17.239 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:56:17.239 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:56:17.239 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:56:17.239 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:17.239 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:56:17.239 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:17.239 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:56:17.239 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:17.239 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:56:17.240 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:56:17.240 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:56:17.240 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:56:17.240 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:17.240 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:56:17.240 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:56:17.240 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:17.240 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:17.240 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:17.240 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:56:17.240 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:56:17.240 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:17.241 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:17.241 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:17.241 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:17.241 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:56:17.241 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:56:17.241 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:56:17.241 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:56:17.241 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:56:17.241 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:17.241 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:17.241 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:56:17.241 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:56:17.241 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:56:17.241 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:56:17.242 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:17.242 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:56:17.242 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:56:17.242 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:17.242 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:17.242 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:17.242 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:56:17.242 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:56:17.242 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:17.242 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:17.242 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:17.242 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:17.242 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:56:17.242 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:56:17.242 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:56:17.243 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:56:17.243 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:56:17.243 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:17.243 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:17.243 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:56:17.243 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:56:17.243 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:56:17.243 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:56:17.243 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:17.243 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:56:17.243 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:56:17.243 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:17.243 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:17.243 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:17.244 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:56:17.244 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:56:17.244 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:17.244 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:17.244 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:17.244 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:17.244 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:56:17.244 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:56:17.244 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:56:17.244 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:56:17.244 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:56:17.244 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:17.244 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:17.245 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:17.245 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:56:17.245 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:56:17.245 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:17.245 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:56:17.245 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:56:17.245 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:56:17.245 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:17.245 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:17.245 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:56:17.245 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:17.246 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:56:17.246 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:56:17.246 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:56:17.246 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:17.246 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:17.246 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:56:17.246 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:17.246 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:56:17.246 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:56:17.246 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:56:17.246 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:17.246 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:17.247 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:56:17.247 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:17.247 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:56:17.247 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:56:17.247 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:56:17.247 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:17.247 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:17.247 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:17.247 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:56:17.247 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:56:17.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.223+0000 7f25c0a7f640 1 -- 172.21.15.12:0/747619335 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f259c03e210 msgr2=0x7f259c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:17.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.223+0000 7f25c0a7f640 1 --2- 172.21.15.12:0/747619335 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f259c03e210 0x7f259c0406d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f25bc13ea30 tx=0x7f25b4009040 comp rx=0 tx=0).stop 2024-09-11T01:56:17.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.223+0000 7f25c0a7f640 10 monclient: shutdown 2024-09-11T01:56:17.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.223+0000 7f25c0a7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:17.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.223+0000 7f25c0a7f640 1 -- 172.21.15.12:0/747619335 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f25bc0a2db0 msgr2=0x7f25bc1389d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:17.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.223+0000 7f25c0a7f640 1 --2- 172.21.15.12:0/747619335 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f25bc0a2db0 0x7f25bc1389d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f25ac035040 tx=0x7f25ac0046c0 comp rx=0 tx=0).stop 2024-09-11T01:56:17.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.223+0000 7f25bad9d640 1 -- 172.21.15.12:0/747619335 reap_dead start 2024-09-11T01:56:17.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.223+0000 7f25bad9d640 1 -- 172.21.15.12:0/747619335 reap_dead start 2024-09-11T01:56:17.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.223+0000 7f25c0a7f640 1 -- 172.21.15.12:0/747619335 shutdown_connections 2024-09-11T01:56:17.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.223+0000 7f25c0a7f640 1 -- 172.21.15.12:0/747619335 >> 172.21.15.12:0/747619335 conn(0x7f25bc0a83b0 msgr2=0x7f25bc09e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:17.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.224+0000 7f25c0a7f640 1 -- 172.21.15.12:0/747619335 shutdown_connections 2024-09-11T01:56:17.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.224+0000 7f25c0a7f640 1 -- 172.21.15.12:0/747619335 wait complete. 2024-09-11T01:56:17.250 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:56:17.250 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:56:17.250 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:56:17.320 INFO:tasks.rados.rados.0.smithi071.stdout:3983: finishing write tid 1 to smithi07134777-8 2024-09-11T01:56:17.320 INFO:tasks.rados.rados.0.smithi071.stdout:3983: finishing write tid 2 to smithi07134777-8 2024-09-11T01:56:17.321 INFO:tasks.rados.rados.0.smithi071.stdout:3984: expect (ObjNum 1276 snap 422 seq_num 1276) 2024-09-11T01:56:17.321 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:56:17.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.349+0000 7f7afcc85640 10 monclient: get_monmap_and_config 2024-09-11T01:56:17.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.349+0000 7f7afcc85640 10 monclient: build_initial_monmap 2024-09-11T01:56:17.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.349+0000 7f7afcc85640 10 monclient: monmap: 2024-09-11T01:56:17.351 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:17.351 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:17.351 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:56:17.350198+0000 2024-09-11T01:56:17.351 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:56:17.350198+0000 2024-09-11T01:56:17.351 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:17.351 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:17.351 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:56:17.351 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:56:17.352 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:56:17.352 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:56:17.352 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:56:17.352 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:17.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.350+0000 7f7afcc85640 1 Processor -- start 2024-09-11T01:56:17.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.350+0000 7f7afcc85640 1 -- start start 2024-09-11T01:56:17.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.350+0000 7f7afcc85640 10 monclient: init 2024-09-11T01:56:17.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.350+0000 7f7afcc85640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:17.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.350+0000 7f7afcc85640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-11T01:56:17.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.350+0000 7f7afcc85640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7af8134ad0 addr v1:172.21.15.71:6789/0 2024-09-11T01:56:17.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.350+0000 7f7afcc85640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7af80a2200 addr v1:172.21.15.12:6789/0 2024-09-11T01:56:17.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.350+0000 7f7afcc85640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7af81355c0 0x7f7af813d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:17.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.350+0000 7f7afcc85640 10 monclient(hunting): picked mon.noname-c con 0x7f7af81355c0 addr v2:172.21.15.71:3300/0 2024-09-11T01:56:17.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.350+0000 7f7afcc85640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7af809ed30 con 0x7f7af80a2200 2024-09-11T01:56:17.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.350+0000 7f7afcc85640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7af809f010 con 0x7f7af8134ad0 2024-09-11T01:56:17.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.350+0000 7f7afcc85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:17.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.350+0000 7f7afcc85640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f7af809f5d0 con 0x7f7af81355c0 2024-09-11T01:56:17.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.350+0000 7f7afcc85640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:17.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.350+0000 7f7afcc85640 10 monclient(hunting): authenticate will time out at 2383.792784s 2024-09-11T01:56:17.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.350+0000 7f7af5d9b640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7af80a2200 0x7f7af813a000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:43478/0 (socket says 172.21.15.12:43478) 2024-09-11T01:56:17.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.350+0000 7f7af5d9b640 1 -- 172.21.15.12:0/3198299736 learned_addr learned my addr 172.21.15.12:0/3198299736 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:56:17.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.350+0000 7f7af6d9d640 1 --2- 172.21.15.12:0/3198299736 >> v2:172.21.15.71:3300/0 conn(0x7f7af81355c0 0x7f7af813d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:17.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.351+0000 7f7af6d9d640 10 monclient(hunting): get_auth_request con 0x7f7af81355c0 auth_method 0 2024-09-11T01:56:17.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.351+0000 7f7af6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:17.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.351+0000 7f7af6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:17.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.351+0000 7f7af6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:17.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.351+0000 7f7af6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:17.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.351+0000 7f7af6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:17.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.351+0000 7f7af6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:17.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.351+0000 7f7af6d9d640 10 monclient(hunting): handle_auth_done global_id 274372 payload 274 2024-09-11T01:56:17.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.351+0000 7f7af6d9d640 1 -- 172.21.15.12:0/3198299736 >> v1:172.21.15.71:6789/0 conn(0x7f7af8134ad0 legacy=0x7f7af8134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:17.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.351+0000 7f7af6d9d640 1 -- 172.21.15.12:0/3198299736 >> v1:172.21.15.12:6789/0 conn(0x7f7af80a2200 legacy=0x7f7af813a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:17.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.351+0000 7f7af6d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:56:17.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.351+0000 7f7af6d9d640 1 monclient: found mon.noname-c 2024-09-11T01:56:17.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.351+0000 7f7af6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:17.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.351+0000 7f7af6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:56:17.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.351+0000 7f7af6d9d640 1 -- 172.21.15.12:0/3198299736 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af809f2f0 con 0x7f7af81355c0 2024-09-11T01:56:17.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.354+0000 7f7af6d9d640 1 -- 172.21.15.12:0/3198299736 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7af81355c0 msgr2=0x7f7af813d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:17.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.354+0000 7f7af6d9d640 1 --2- 172.21.15.12:0/3198299736 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7af81355c0 0x7f7af813d730 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7af809f750 tx=0x7f7af0029fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cca2776e4a1fea4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:17.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.354+0000 7f7af559a640 1 -- 172.21.15.12:0/3198299736 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af0037070 con 0x7f7af81355c0 2024-09-11T01:56:17.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.354+0000 7f7af559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:17.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.354+0000 7f7af559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:56:17.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.354+0000 7f7af559a640 10 monclient: dump: 2024-09-11T01:56:17.370 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:17.371 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:17.371 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.371 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.371 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:17.371 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:17.371 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:17.372 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:17.372 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:17.372 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:17.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.354+0000 7f7af559a640 10 monclient: _finish_auth 0 2024-09-11T01:56:17.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.354+0000 7f7af559a640 10 monclient: _check_auth_tickets 2024-09-11T01:56:17.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.354+0000 7f7af559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:17.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.354+0000 7f7afcc85640 5 monclient: authenticate success, global_id 274372 2024-09-11T01:56:17.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.354+0000 7f7afcc85640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:56:17.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.354+0000 7f7af559a640 1 -- 172.21.15.12:0/3198299736 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7af0032440 con 0x7f7af81355c0 2024-09-11T01:56:17.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.354+0000 7f7af559a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:17.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.354+0000 7f7afcc85640 10 monclient: get_monmap_and_config success 2024-09-11T01:56:17.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.354+0000 7f7afcc85640 4 set_mon_vals no callback set 2024-09-11T01:56:17.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.354+0000 7f7af559a640 1 -- 172.21.15.12:0/3198299736 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af00327f0 con 0x7f7af81355c0 2024-09-11T01:56:17.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.354+0000 7f7af559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:17.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.354+0000 7f7af559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:56:17.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.354+0000 7f7af559a640 10 monclient: dump: 2024-09-11T01:56:17.374 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:17.375 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:17.375 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.375 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.375 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:17.375 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:17.375 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:17.376 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:17.376 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:17.376 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:17.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.354+0000 7f7afcc85640 10 monclient: shutdown 2024-09-11T01:56:17.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.354+0000 7f7afcc85640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:17.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.354+0000 7f7afcc85640 1 -- 172.21.15.12:0/3198299736 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7af81355c0 msgr2=0x7f7af813d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:17.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.354+0000 7f7afcc85640 1 --2- 172.21.15.12:0/3198299736 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7af81355c0 0x7f7af813d730 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7af809f750 tx=0x7f7af0029fe0 comp rx=0 tx=0).stop 2024-09-11T01:56:17.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.354+0000 7f7afcc85640 1 -- 172.21.15.12:0/3198299736 shutdown_connections 2024-09-11T01:56:17.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.354+0000 7f7afcc85640 1 --2- 172.21.15.12:0/3198299736 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7af81355c0 0x7f7af813d730 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:17.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.354+0000 7f7afcc85640 1 -- 172.21.15.12:0/3198299736 >> 172.21.15.12:0/3198299736 conn(0x7f7af80a2f70 msgr2=0x7f7af80a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:17.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.355+0000 7f7afcc85640 1 -- 172.21.15.12:0/3198299736 shutdown_connections 2024-09-11T01:56:17.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.355+0000 7f7afcc85640 1 -- 172.21.15.12:0/3198299736 wait complete. 2024-09-11T01:56:17.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.355+0000 7f7afcc85640 10 monclient: build_initial_monmap 2024-09-11T01:56:17.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.355+0000 7f7afcc85640 10 monclient: monmap: 2024-09-11T01:56:17.378 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:17.378 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:56:17.378 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:56:17.379 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:56:17.379 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:17.379 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:17.379 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:56:17.380 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:56:17.380 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:56:17.380 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:17.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7afcc85640 1 Processor -- start 2024-09-11T01:56:17.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7afcc85640 1 -- start start 2024-09-11T01:56:17.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7afcc85640 10 monclient: init 2024-09-11T01:56:17.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7afcc85640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:17.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7afcc85640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:56:17.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7afcc85640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7af80a2200 0x7f7af813bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:17.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7afcc85640 10 monclient(hunting): picked mon.noname-b con 0x7f7af80a2200 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:56:17.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7afcc85640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7af8134ad0 0x7f7af813c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:17.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7afcc85640 10 monclient(hunting): picked mon.noname-a con 0x7f7af8134ad0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:56:17.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7afcc85640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7af81355c0 0x7f7af813c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:17.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7afcc85640 10 monclient(hunting): picked mon.noname-c con 0x7f7af81355c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:17.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7afcc85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:17.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7afcc85640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7af80a0150 con 0x7f7af8134ad0 2024-09-11T01:56:17.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7afcc85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:17.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7afcc85640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f7af809ffd0 con 0x7f7af80a2200 2024-09-11T01:56:17.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7afcc85640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:17.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7afcc85640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7af813dc70 con 0x7f7af81355c0 2024-09-11T01:56:17.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7afcc85640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:17.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7afcc85640 10 monclient(hunting): authenticate will time out at 2383.798642s 2024-09-11T01:56:17.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7af5d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7af8134ad0 0x7f7af813c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:17.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7af5d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7af8134ad0 0x7f7af813c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49750/0 (socket says 172.21.15.12:49750) 2024-09-11T01:56:17.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7af5d9b640 1 -- 172.21.15.12:0/2592726715 learned_addr learned my addr 172.21.15.12:0/2592726715 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:17.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7af5d9b640 10 monclient(hunting): get_auth_request con 0x7f7af8134ad0 auth_method 0 2024-09-11T01:56:17.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7af5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:17.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7af5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:17.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7af5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:17.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7af6d9d640 1 --2- 172.21.15.12:0/2592726715 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7af81355c0 0x7f7af813c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:17.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7af6d9d640 10 monclient(hunting): get_auth_request con 0x7f7af81355c0 auth_method 0 2024-09-11T01:56:17.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7af6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:17.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7af6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:17.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7af6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:17.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7af659c640 1 --2- 172.21.15.12:0/2592726715 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7af80a2200 0x7f7af813bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:17.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7af5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:17.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7af5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:17.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7af5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:17.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7af659c640 10 monclient(hunting): get_auth_request con 0x7f7af80a2200 auth_method 0 2024-09-11T01:56:17.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7af659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:17.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7af659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:17.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7af659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:17.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7af6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:17.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7af6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:17.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7af6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:17.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7af659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:17.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7af659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:17.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.356+0000 7f7af659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:17.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.357+0000 7f7af5d9b640 10 monclient(hunting): handle_auth_done global_id 294216 payload 824 2024-09-11T01:56:17.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.357+0000 7f7af5d9b640 1 -- 172.21.15.12:0/2592726715 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7af81355c0 msgr2=0x7f7af813c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:17.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.357+0000 7f7af5d9b640 1 --2- 172.21.15.12:0/2592726715 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7af81355c0 0x7f7af813c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:17.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.357+0000 7f7af5d9b640 1 -- 172.21.15.12:0/2592726715 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7af80a2200 msgr2=0x7f7af813bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:17.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.357+0000 7f7af5d9b640 1 --2- 172.21.15.12:0/2592726715 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7af80a2200 0x7f7af813bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:17.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.357+0000 7f7af5d9b640 10 monclient: _finish_hunting 0 2024-09-11T01:56:17.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.357+0000 7f7af5d9b640 1 monclient: found mon.noname-a 2024-09-11T01:56:17.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.357+0000 7f7af5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:17.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.357+0000 7f7af5d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:56:17.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.357+0000 7f7af5d9b640 1 -- 172.21.15.12:0/2592726715 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af811df90 con 0x7f7af8134ad0 2024-09-11T01:56:17.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.359+0000 7f7af5d9b640 1 -- 172.21.15.12:0/2592726715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7af8134ad0 msgr2=0x7f7af813c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:17.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.359+0000 7f7af5d9b640 1 --2- 172.21.15.12:0/2592726715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7af8134ad0 0x7f7af813c1d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7aec00c460 tx=0x7f7aec00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:17.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.359+0000 7f7ae6ffd640 1 -- 172.21.15.12:0/2592726715 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aec007070 con 0x7f7af8134ad0 2024-09-11T01:56:17.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.359+0000 7f7ae6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:17.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.359+0000 7f7ae6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:56:17.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.359+0000 7f7ae6ffd640 10 monclient: dump: 2024-09-11T01:56:17.390 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:17.390 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:17.390 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.390 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.390 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:17.390 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:17.390 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:17.390 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:17.391 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:17.391 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:17.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.359+0000 7f7ae6ffd640 10 monclient: _finish_auth 0 2024-09-11T01:56:17.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.359+0000 7f7ae6ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:56:17.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.359+0000 7f7ae6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:17.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.359+0000 7f7ae6ffd640 1 -- 172.21.15.12:0/2592726715 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7aec007210 con 0x7f7af8134ad0 2024-09-11T01:56:17.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.359+0000 7f7ae6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:17.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.359+0000 7f7ae6ffd640 1 -- 172.21.15.12:0/2592726715 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aec007600 con 0x7f7af8134ad0 2024-09-11T01:56:17.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.359+0000 7f7afcc85640 5 monclient: authenticate success, global_id 294216 2024-09-11T01:56:17.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.359+0000 7f7afcc85640 10 monclient: _renew_subs 2024-09-11T01:56:17.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.359+0000 7f7afcc85640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:17.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.359+0000 7f7afcc85640 1 -- 172.21.15.12:0/2592726715 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7af80a13e0 con 0x7f7af8134ad0 2024-09-11T01:56:17.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.359+0000 7f7ae6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:17.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.359+0000 7f7ae6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:56:17.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.359+0000 7f7ae6ffd640 10 monclient: dump: 2024-09-11T01:56:17.392 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:17.392 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:17.392 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.392 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:17.392 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:17.392 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:17.392 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:17.393 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:17.393 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:17.393 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:17.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.359+0000 7f7af4d99640 4 set_mon_vals no callback set 2024-09-11T01:56:17.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.359+0000 7f7afcc85640 10 monclient: _renew_subs 2024-09-11T01:56:17.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.359+0000 7f7afcc85640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:17.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.359+0000 7f7afcc85640 1 -- 172.21.15.12:0/2592726715 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7af809d480 con 0x7f7af8134ad0 2024-09-11T01:56:17.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.360+0000 7f7afcc85640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:17.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.360+0000 7f7afcc85640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:17.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.360+0000 7f7afcc85640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:17.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.360+0000 7f7ae6ffd640 1 -- 172.21.15.12:0/2592726715 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f7aec01b400 con 0x7f7af8134ad0 2024-09-11T01:56:17.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.360+0000 7f7afcc85640 1 -- 172.21.15.12:0/2592726715 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7af8134eb0 con 0x7f7af8134ad0 2024-09-11T01:56:17.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.360+0000 7f7ae6ffd640 1 --2- 172.21.15.12:0/2592726715 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7ad803e210 0x7f7ad80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:17.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.360+0000 7f7ae6ffd640 1 -- 172.21.15.12:0/2592726715 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(918..918 src has 676..918) ==== 4799+0+0 (secure 0 0 0) 0x7f7aec052a00 con 0x7f7af8134ad0 2024-09-11T01:56:17.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.360+0000 7f7af659c640 1 --2- 172.21.15.12:0/2592726715 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7ad803e210 0x7f7ad80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:17.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.360+0000 7f7af659c640 10 monclient: get_auth_request con 0x7f7ad803e210 auth_method 0 2024-09-11T01:56:17.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.363+0000 7f7af659c640 1 --2- 172.21.15.12:0/2592726715 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7ad803e210 0x7f7ad80406d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f7ae8005660 tx=0x7f7ae8003d90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:17.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.365+0000 7f7ae6ffd640 1 -- 172.21.15.12:0/2592726715 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7aec0513f0 con 0x7f7af8134ad0 2024-09-11T01:56:17.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.365+0000 7f7ae6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:17.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.365+0000 7f7ae6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:56:17.501 INFO:tasks.rados.rados.0.smithi071.stdout:3983: finishing write tid 3 to smithi07134777-8 2024-09-11T01:56:17.501 INFO:tasks.rados.rados.0.smithi071.stdout:3983: finishing write tid 4 to smithi07134777-8 2024-09-11T01:56:17.501 INFO:tasks.rados.rados.0.smithi071.stdout:3983: finishing write tid 5 to smithi07134777-8 2024-09-11T01:56:17.501 INFO:tasks.rados.rados.0.smithi071.stdout:3983: oid 8 updating version 0 to 2595 2024-09-11T01:56:17.501 INFO:tasks.rados.rados.0.smithi071.stdout:3983: oid 8 updating version 2595 to 2596 2024-09-11T01:56:17.501 INFO:tasks.rados.rados.0.smithi071.stdout:3983: oid 8 version 2596 is already newer than 2594 2024-09-11T01:56:17.501 INFO:tasks.rados.rados.0.smithi071.stdout:3983: oid 8 version 2596 is already newer than 2593 2024-09-11T01:56:17.501 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 2596 (ObjNum 1367 snap 451 seq_num 1367) dirty exists 2024-09-11T01:56:17.502 INFO:tasks.rados.rados.0.smithi071.stdout:3983: left oid 8 (ObjNum 1367 snap 451 seq_num 1367) 2024-09-11T01:56:17.502 INFO:tasks.rados.rados.0.smithi071.stdout:3985: finishing write tid 1 to smithi07134777-23 2024-09-11T01:56:17.502 INFO:tasks.rados.rados.0.smithi071.stdout:3985: finishing write tid 2 to smithi07134777-23 2024-09-11T01:56:17.502 INFO:tasks.rados.rados.0.smithi071.stdout:3985: finishing write tid 3 to smithi07134777-23 2024-09-11T01:56:17.502 INFO:tasks.rados.rados.0.smithi071.stdout:3987: seq_num 1369 ranges {608783=759421,2027143=421133} 2024-09-11T01:56:17.508 INFO:tasks.rados.rados.0.smithi071.stdout:3987: writing smithi07134777-14 from 608783 to 1368204 tid 1 2024-09-11T01:56:17.515 INFO:tasks.rados.rados.0.smithi071.stdout:3987: writing smithi07134777-14 from 2027143 to 2448276 tid 2 2024-09-11T01:56:17.515 INFO:tasks.rados.rados.0.smithi071.stdout:3980: done (5 left) 2024-09-11T01:56:17.515 INFO:tasks.rados.rados.0.smithi071.stdout:3982: done (4 left) 2024-09-11T01:56:17.515 INFO:tasks.rados.rados.0.smithi071.stdout:3983: done (3 left) 2024-09-11T01:56:17.515 INFO:tasks.rados.rados.0.smithi071.stdout:3984: done (2 left) 2024-09-11T01:56:17.515 INFO:tasks.rados.rados.0.smithi071.stdout:3988: write oid 45 current snap is 451 2024-09-11T01:56:17.516 INFO:tasks.rados.rados.0.smithi071.stdout:3985: finishing write tid 4 to smithi07134777-23 2024-09-11T01:56:17.516 INFO:tasks.rados.rados.0.smithi071.stdout:3985: oid 23 updating version 0 to 3031 2024-09-11T01:56:17.516 INFO:tasks.rados.rados.0.smithi071.stdout:3985: oid 23 updating version 3031 to 3032 2024-09-11T01:56:17.516 INFO:tasks.rados.rados.0.smithi071.stdout:3985: oid 23 updating version 3032 to 3033 2024-09-11T01:56:17.516 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 3033 (ObjNum 1368 snap 451 seq_num 1368) dirty exists 2024-09-11T01:56:17.516 INFO:tasks.rados.rados.0.smithi071.stdout:3985: left oid 23 (ObjNum 1368 snap 451 seq_num 1368) 2024-09-11T01:56:17.516 INFO:tasks.rados.rados.0.smithi071.stdout:3987: finishing write tid 1 to smithi07134777-14 2024-09-11T01:56:17.516 INFO:tasks.rados.rados.0.smithi071.stdout:3988: seq_num 1370 ranges {625590=434670,1685625=681144,2822206=203384} 2024-09-11T01:56:17.520 INFO:tasks.rados.rados.0.smithi071.stdout:3988: writing smithi07134777-45 from 625590 to 1060260 tid 1 2024-09-11T01:56:17.527 INFO:tasks.rados.rados.0.smithi071.stdout:3988: writing smithi07134777-45 from 1685625 to 2366769 tid 2 2024-09-11T01:56:17.532 INFO:tasks.rados.rados.0.smithi071.stdout:3988: writing smithi07134777-45 from 2822206 to 3025590 tid 3 2024-09-11T01:56:17.532 INFO:tasks.rados.rados.0.smithi071.stdout:3985: done (2 left) 2024-09-11T01:56:17.532 INFO:tasks.rados.rados.0.smithi071.stdout:3989: write oid 42 current snap is 451 2024-09-11T01:56:17.532 INFO:tasks.rados.rados.0.smithi071.stdout:3987: finishing write tid 2 to smithi07134777-14 2024-09-11T01:56:17.532 INFO:tasks.rados.rados.0.smithi071.stdout:3987: finishing write tid 3 to smithi07134777-14 2024-09-11T01:56:17.532 INFO:tasks.rados.rados.0.smithi071.stdout:3987: finishing write tid 4 to smithi07134777-14 2024-09-11T01:56:17.532 INFO:tasks.rados.rados.0.smithi071.stdout:3987: oid 14 updating version 0 to 3245 2024-09-11T01:56:17.532 INFO:tasks.rados.rados.0.smithi071.stdout:3987: oid 14 updating version 3245 to 3246 2024-09-11T01:56:17.532 INFO:tasks.rados.rados.0.smithi071.stdout:3987: oid 14 version 3246 is already newer than 3244 2024-09-11T01:56:17.533 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 3246 (ObjNum 1369 snap 451 seq_num 1369) dirty exists 2024-09-11T01:56:17.533 INFO:tasks.rados.rados.0.smithi071.stdout:3987: left oid 14 (ObjNum 1369 snap 451 seq_num 1369) 2024-09-11T01:56:17.533 INFO:tasks.rados.rados.0.smithi071.stdout:3988: finishing write tid 1 to smithi07134777-45 2024-09-11T01:56:17.533 INFO:tasks.rados.rados.0.smithi071.stdout:3989: seq_num 1371 ranges {642397=509919,1627754=424802,2733276=309121} 2024-09-11T01:56:17.538 INFO:tasks.rados.rados.0.smithi071.stdout:3989: writing smithi07134777-42 from 642397 to 1152316 tid 1 2024-09-11T01:56:17.543 INFO:tasks.rados.rados.0.smithi071.stdout:3989: writing smithi07134777-42 from 1627754 to 2052556 tid 2 2024-09-11T01:56:17.549 INFO:tasks.rados.rados.0.smithi071.stdout:3989: writing smithi07134777-42 from 2733276 to 3042397 tid 3 2024-09-11T01:56:17.549 INFO:tasks.rados.rados.0.smithi071.stdout:3987: done (2 left) 2024-09-11T01:56:17.549 INFO:tasks.rados.rados.0.smithi071.stdout:3990: read oid 34 snap -1 2024-09-11T01:56:17.549 INFO:tasks.rados.rados.0.smithi071.stdout:3990: expect (ObjNum 1295 snap 426 seq_num 1295) 2024-09-11T01:56:17.550 INFO:tasks.rados.rados.0.smithi071.stdout:3988: finishing write tid 2 to smithi07134777-45 2024-09-11T01:56:17.550 INFO:tasks.rados.rados.0.smithi071.stdout:3991: read oid 23 snap 444 2024-09-11T01:56:17.550 INFO:tasks.rados.rados.0.smithi071.stdout:3991: expect (ObjNum 1346 snap 441 seq_num 1346) 2024-09-11T01:56:17.550 INFO:tasks.rados.rados.0.smithi071.stdout:3992: snap_create 2024-09-11T01:56:17.550 INFO:tasks.rados.rados.0.smithi071.stdout:3988: finishing write tid 3 to smithi07134777-45 2024-09-11T01:56:17.550 INFO:tasks.rados.rados.0.smithi071.stdout:3988: finishing write tid 4 to smithi07134777-45 2024-09-11T01:56:17.550 INFO:tasks.rados.rados.0.smithi071.stdout:3988: finishing write tid 5 to smithi07134777-45 2024-09-11T01:56:17.550 INFO:tasks.rados.rados.0.smithi071.stdout:3988: oid 45 updating version 0 to 2655 2024-09-11T01:56:17.550 INFO:tasks.rados.rados.0.smithi071.stdout:3988: oid 45 version 2655 is already newer than 2654 2024-09-11T01:56:17.550 INFO:tasks.rados.rados.0.smithi071.stdout:3988: oid 45 updating version 2655 to 2657 2024-09-11T01:56:17.550 INFO:tasks.rados.rados.0.smithi071.stdout:3988: oid 45 version 2657 is already newer than 2656 2024-09-11T01:56:17.550 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 2657 (ObjNum 1370 snap 451 seq_num 1370) dirty exists 2024-09-11T01:56:17.550 INFO:tasks.rados.rados.0.smithi071.stdout:3988: left oid 45 (ObjNum 1370 snap 451 seq_num 1370) 2024-09-11T01:56:17.550 INFO:tasks.rados.rados.0.smithi071.stdout:3989: finishing write tid 1 to smithi07134777-42 2024-09-11T01:56:17.550 INFO:tasks.rados.rados.0.smithi071.stdout:3989: finishing write tid 2 to smithi07134777-42 2024-09-11T01:56:17.551 INFO:tasks.rados.rados.0.smithi071.stdout:3989: finishing write tid 3 to smithi07134777-42 2024-09-11T01:56:17.551 INFO:tasks.rados.rados.0.smithi071.stdout:3989: finishing write tid 4 to smithi07134777-42 2024-09-11T01:56:17.552 INFO:tasks.rados.rados.0.smithi071.stdout:3989: finishing write tid 5 to smithi07134777-42 2024-09-11T01:56:17.552 INFO:tasks.rados.rados.0.smithi071.stdout:3989: oid 42 updating version 0 to 2598 2024-09-11T01:56:17.552 INFO:tasks.rados.rados.0.smithi071.stdout:3989: oid 42 version 2598 is already newer than 2595 2024-09-11T01:56:17.552 INFO:tasks.rados.rados.0.smithi071.stdout:3989: oid 42 version 2598 is already newer than 2597 2024-09-11T01:56:17.552 INFO:tasks.rados.rados.0.smithi071.stdout:3989: oid 42 version 2598 is already newer than 2596 2024-09-11T01:56:17.552 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 2598 (ObjNum 1371 snap 451 seq_num 1371) dirty exists 2024-09-11T01:56:17.552 INFO:tasks.rados.rados.0.smithi071.stdout:3989: left oid 42 (ObjNum 1371 snap 451 seq_num 1371) 2024-09-11T01:56:17.552 INFO:tasks.rados.rados.0.smithi071.stdout:3991: expect (ObjNum 1346 snap 441 seq_num 1346) 2024-09-11T01:56:17.603 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1194619 2024-09-11T01:56:17.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.626+0000 7f7afcc85640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:56:17.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.626+0000 7f7afcc85640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:56:17.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.626+0000 7f7afcc85640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:17.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.626+0000 7f7afcc85640 1 -- 172.21.15.12:0/2592726715 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f7af806d480 con 0x7f7af8134ad0 2024-09-11T01:56:17.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.627+0000 7f7ae6ffd640 1 -- 172.21.15.12:0/2592726715 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f7aec010d10 con 0x7f7af8134ad0 2024-09-11T01:56:17.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.627+0000 7f7ae6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:56:17.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.627+0000 7f7ae6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:56:17.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.629+0000 7f7afcc85640 1 -- 172.21.15.12:0/2592726715 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7ad803e210 msgr2=0x7f7ad80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:17.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.629+0000 7f7afcc85640 1 --2- 172.21.15.12:0/2592726715 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7ad803e210 0x7f7ad80406d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f7ae8005660 tx=0x7f7ae8003d90 comp rx=0 tx=0).stop 2024-09-11T01:56:17.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.629+0000 7f7afcc85640 10 monclient: shutdown 2024-09-11T01:56:17.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.629+0000 7f7afcc85640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:17.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.629+0000 7f7afcc85640 1 -- 172.21.15.12:0/2592726715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7af8134ad0 msgr2=0x7f7af813c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:17.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.629+0000 7f7afcc85640 1 --2- 172.21.15.12:0/2592726715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7af8134ad0 0x7f7af813c1d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7aec00c460 tx=0x7f7aec00c920 comp rx=0 tx=0).stop 2024-09-11T01:56:17.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.629+0000 7f7afcc85640 1 -- 172.21.15.12:0/2592726715 shutdown_connections 2024-09-11T01:56:17.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.629+0000 7f7afcc85640 1 --2- 172.21.15.12:0/2592726715 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7af81355c0 0x7f7af813c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:17.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.629+0000 7f7afcc85640 1 --2- 172.21.15.12:0/2592726715 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7ad803e210 0x7f7ad80406d0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:17.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.629+0000 7f7afcc85640 1 --2- 172.21.15.12:0/2592726715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7af8134ad0 0x7f7af813c1d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:17.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.629+0000 7f7afcc85640 1 --2- 172.21.15.12:0/2592726715 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7af80a2200 0x7f7af813bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:17.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.629+0000 7f7afcc85640 1 -- 172.21.15.12:0/2592726715 >> 172.21.15.12:0/2592726715 conn(0x7f7af80a2f70 msgr2=0x7f7af80a4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:17.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.629+0000 7f7afcc85640 1 -- 172.21.15.12:0/2592726715 shutdown_connections 2024-09-11T01:56:17.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:17.629+0000 7f7afcc85640 1 -- 172.21.15.12:0/2592726715 wait complete. 2024-09-11T01:56:17.641 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:56:17.681 INFO:tasks.rados.rados.0.smithi071.stdout:3990: expect (ObjNum 1295 snap 426 seq_num 1295) 2024-09-11T01:56:17.681 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:56:17.835 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.833+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.835 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.833+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=12,logm=46} crc {auth=1917994987,config=2586702516,health=615225419,kv=3263697421,logm=344873525}) 2024-09-11T01:56:17.836 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.833+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=11,kv=12,logm=48} crc {auth=839664195,config=2586702516,health=2519180625,kv=3263697420,logm=1813763847}) 2024-09-11T01:56:17.836 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.833+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.836 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.833+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=12,logm=46} crc {auth=1917994987,config=2586702516,health=615225419,kv=3263697421,logm=344873525}) 2024-09-11T01:56:17.836 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.833+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=11,kv=12,logm=47} crc {auth=1917994987,config=2586702516,health=1089369259,kv=3263697420,logm=4250390013}) 2024-09-11T01:56:17.837 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.835+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.837 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.835+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2690687731}) 2024-09-11T01:56:17.837 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.835+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2038773540}) 2024-09-11T01:56:17.837 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.835+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.837 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.835+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2690687731}) 2024-09-11T01:56:17.837 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.835+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2792986778}) 2024-09-11T01:56:17.838 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.836+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.838 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.836+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=345647164}) 2024-09-11T01:56:17.839 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.836+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=902725541}) 2024-09-11T01:56:17.839 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.836+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.839 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.836+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=345647164}) 2024-09-11T01:56:17.839 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.836+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3076790935}) 2024-09-11T01:56:17.839 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.837+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.839 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.837+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=35202262}) 2024-09-11T01:56:17.839 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.837+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1959661501}) 2024-09-11T01:56:17.839 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.837+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.839 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.837+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=35202262}) 2024-09-11T01:56:17.839 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.837+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2016105385}) 2024-09-11T01:56:17.840 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.838+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.840 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.838+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=623198388}) 2024-09-11T01:56:17.840 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.838+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1107412189}) 2024-09-11T01:56:17.840 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.838+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.840 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.838+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=623198388}) 2024-09-11T01:56:17.840 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.838+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3177305672}) 2024-09-11T01:56:17.843 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.839+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.843 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.839+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255467470}) 2024-09-11T01:56:17.843 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.839+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=70180877}) 2024-09-11T01:56:17.843 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.839+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.843 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.839+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255467470}) 2024-09-11T01:56:17.843 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.839+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3661664639}) 2024-09-11T01:56:17.843 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.840+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.843 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.840+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2260457439}) 2024-09-11T01:56:17.843 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.840+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=300795087}) 2024-09-11T01:56:17.843 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.840+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.840+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2260457439}) 2024-09-11T01:56:17.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.840+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2628470458}) 2024-09-11T01:56:17.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.840+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.840+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3679541508}) 2024-09-11T01:56:17.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.840+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2924604106}) 2024-09-11T01:56:17.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.840+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.840+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3679541508}) 2024-09-11T01:56:17.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.840+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1110736538}) 2024-09-11T01:56:17.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.841+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.841+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3917829023}) 2024-09-11T01:56:17.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.841+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1848257587}) 2024-09-11T01:56:17.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.841+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.841+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3917829023}) 2024-09-11T01:56:17.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.841+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1849940937}) 2024-09-11T01:56:17.851 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.842+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.851 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.842+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2055137986}) 2024-09-11T01:56:17.851 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.842+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2226923653}) 2024-09-11T01:56:17.851 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.842+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.851 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.842+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2055137986}) 2024-09-11T01:56:17.851 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.842+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1553308565}) 2024-09-11T01:56:17.851 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.843+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.851 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.843+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=47811005}) 2024-09-11T01:56:17.851 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.843+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1468612688}) 2024-09-11T01:56:17.851 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.843+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.851 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.843+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=47811005}) 2024-09-11T01:56:17.852 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.843+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=847240530}) 2024-09-11T01:56:17.852 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.843+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.852 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.843+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4177832929}) 2024-09-11T01:56:17.852 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.843+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4004280829}) 2024-09-11T01:56:17.852 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.843+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.852 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.843+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4177832929}) 2024-09-11T01:56:17.852 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.843+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2643417837}) 2024-09-11T01:56:17.852 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.844+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.852 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.844+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2719301462}) 2024-09-11T01:56:17.852 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.844+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2700192932}) 2024-09-11T01:56:17.852 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.844+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.852 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.844+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2719301462}) 2024-09-11T01:56:17.852 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.844+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1321059510}) 2024-09-11T01:56:17.853 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.845+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.853 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.845+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2301678352}) 2024-09-11T01:56:17.853 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.845+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1542884170}) 2024-09-11T01:56:17.853 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.845+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.853 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.845+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2301678352}) 2024-09-11T01:56:17.853 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.845+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1035629032}) 2024-09-11T01:56:17.853 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.846+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.853 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.846+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1686227029}) 2024-09-11T01:56:17.853 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.846+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3323647439}) 2024-09-11T01:56:17.853 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.846+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.853 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.846+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1686227029}) 2024-09-11T01:56:17.853 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.846+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3537001783}) 2024-09-11T01:56:17.853 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.846+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.853 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.846+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=897653619}) 2024-09-11T01:56:17.854 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.846+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1517903451}) 2024-09-11T01:56:17.854 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.846+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.854 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.846+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=897653619}) 2024-09-11T01:56:17.854 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.846+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=357179835}) 2024-09-11T01:56:17.854 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.847+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.854 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.847+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4199619342}) 2024-09-11T01:56:17.854 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.847+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=635393191}) 2024-09-11T01:56:17.854 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.847+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.854 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.847+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4199619342}) 2024-09-11T01:56:17.854 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.847+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2230469111}) 2024-09-11T01:56:17.854 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.848+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.854 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.848+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3092032057}) 2024-09-11T01:56:17.854 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.848+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2176888921}) 2024-09-11T01:56:17.855 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.848+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.855 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.848+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3092032057}) 2024-09-11T01:56:17.855 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.848+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3121786351}) 2024-09-11T01:56:17.855 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.848+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.855 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.848+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=814813768}) 2024-09-11T01:56:17.855 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.848+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1855898532}) 2024-09-11T01:56:17.855 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.848+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.855 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.848+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=814813768}) 2024-09-11T01:56:17.855 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.848+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1040874466}) 2024-09-11T01:56:17.855 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.849+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.855 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.849+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2887131141}) 2024-09-11T01:56:17.855 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.849+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1620228680}) 2024-09-11T01:56:17.855 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.849+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.855 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.849+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2887131141}) 2024-09-11T01:56:17.856 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.849+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2497713172}) 2024-09-11T01:56:17.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.850+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.850+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252398308}) 2024-09-11T01:56:17.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.850+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3208251788}) 2024-09-11T01:56:17.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.850+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.850+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252398308}) 2024-09-11T01:56:17.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.850+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4087803481}) 2024-09-11T01:56:17.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.850+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.867 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.850+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1452288071}) 2024-09-11T01:56:17.868 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.851+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4241774068}) 2024-09-11T01:56:17.868 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.851+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.868 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.851+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1452288071}) 2024-09-11T01:56:17.868 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.851+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2839451678}) 2024-09-11T01:56:17.868 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.851+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.868 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.851+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3732212289}) 2024-09-11T01:56:17.868 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.852+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2517911147}) 2024-09-11T01:56:17.868 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.852+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.868 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.852+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3732212289}) 2024-09-11T01:56:17.868 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.852+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3161246876}) 2024-09-11T01:56:17.868 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.852+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.868 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.852+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=255486352}) 2024-09-11T01:56:17.868 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.852+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3255368236}) 2024-09-11T01:56:17.869 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.852+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.869 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.852+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=255486352}) 2024-09-11T01:56:17.869 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.852+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3876757928}) 2024-09-11T01:56:17.869 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.853+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.869 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.853+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3576881354}) 2024-09-11T01:56:17.869 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.853+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3274860754}) 2024-09-11T01:56:17.869 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.853+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.869 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.853+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3576881354}) 2024-09-11T01:56:17.869 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.853+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2676768640}) 2024-09-11T01:56:17.869 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.854+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.869 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.854+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252910993}) 2024-09-11T01:56:17.869 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.854+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=964113088}) 2024-09-11T01:56:17.869 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.854+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.869 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.854+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252910993}) 2024-09-11T01:56:17.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.854+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1841510804}) 2024-09-11T01:56:17.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.855+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.855+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2657025980}) 2024-09-11T01:56:17.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.855+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3596115175}) 2024-09-11T01:56:17.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.855+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.855+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2657025980}) 2024-09-11T01:56:17.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.855+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4106472163}) 2024-09-11T01:56:17.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.856+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.856+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2278015952}) 2024-09-11T01:56:17.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.856+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=828684017}) 2024-09-11T01:56:17.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.856+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.856+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2278015952}) 2024-09-11T01:56:17.870 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.856+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1783594936}) 2024-09-11T01:56:17.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.856+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.856+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2314673443}) 2024-09-11T01:56:17.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.856+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=209334014}) 2024-09-11T01:56:17.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.856+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.856+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2314673443}) 2024-09-11T01:56:17.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.856+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2202282088}) 2024-09-11T01:56:17.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.857+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.857+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3231560276}) 2024-09-11T01:56:17.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.857+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=86023983}) 2024-09-11T01:56:17.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.857+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.857+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3231560276}) 2024-09-11T01:56:17.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.857+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3301120533}) 2024-09-11T01:56:17.871 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.858+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.858+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2089379843}) 2024-09-11T01:56:17.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.858+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4186364188}) 2024-09-11T01:56:17.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.858+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.858+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2089379843}) 2024-09-11T01:56:17.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.858+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=51910925}) 2024-09-11T01:56:17.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.859+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.859+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1062708111}) 2024-09-11T01:56:17.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.859+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2204266586}) 2024-09-11T01:56:17.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.859+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.859+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1062708111}) 2024-09-11T01:56:17.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.859+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=306305495}) 2024-09-11T01:56:17.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.860+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.872 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.860+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=971552204}) 2024-09-11T01:56:17.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.860+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1444951355}) 2024-09-11T01:56:17.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.860+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.860+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=971552204}) 2024-09-11T01:56:17.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.860+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1598758049}) 2024-09-11T01:56:17.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.860+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.860+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2159854437}) 2024-09-11T01:56:17.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.860+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2549319631}) 2024-09-11T01:56:17.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.860+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.860+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2159854437}) 2024-09-11T01:56:17.873 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.860+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3164624657}) 2024-09-11T01:56:17.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.861+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.861+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1622389426}) 2024-09-11T01:56:17.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.861+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3751678405}) 2024-09-11T01:56:17.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.861+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.861+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1622389426}) 2024-09-11T01:56:17.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.861+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2100181550}) 2024-09-11T01:56:17.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.862+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.862+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3850728037}) 2024-09-11T01:56:17.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.862+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3815708981}) 2024-09-11T01:56:17.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.862+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.862+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3850728037}) 2024-09-11T01:56:17.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.862+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=37896557}) 2024-09-11T01:56:17.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.863+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.874 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.863+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988604918}) 2024-09-11T01:56:17.875 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.863+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=236048939}) 2024-09-11T01:56:17.875 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.863+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.875 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.863+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988604918}) 2024-09-11T01:56:17.875 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.863+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3234489569}) 2024-09-11T01:56:17.875 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.864+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.875 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.864+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4023793745}) 2024-09-11T01:56:17.875 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.864+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1691223224}) 2024-09-11T01:56:17.875 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.864+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.875 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.864+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4023793745}) 2024-09-11T01:56:17.875 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.864+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1598994788}) 2024-09-11T01:56:17.875 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.864+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.875 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.864+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1713826370}) 2024-09-11T01:56:17.875 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.864+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1011313897}) 2024-09-11T01:56:17.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.864+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.864+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1713826370}) 2024-09-11T01:56:17.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.864+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2299826179}) 2024-09-11T01:56:17.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.865+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.865+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3441874153}) 2024-09-11T01:56:17.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.865+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3172568852}) 2024-09-11T01:56:17.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.865+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.865+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3441874153}) 2024-09-11T01:56:17.876 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.865+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1977117254}) 2024-09-11T01:56:17.890 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.866+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.866+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3234396718}) 2024-09-11T01:56:17.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.866+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2256107738}) 2024-09-11T01:56:17.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.866+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.866+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3234396718}) 2024-09-11T01:56:17.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.866+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4256287798}) 2024-09-11T01:56:17.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.867+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.867+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126849737}) 2024-09-11T01:56:17.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.867+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2886599534}) 2024-09-11T01:56:17.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.867+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.867+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126849737}) 2024-09-11T01:56:17.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.867+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4004487066}) 2024-09-11T01:56:17.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.868+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.891 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.868+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=132748896}) 2024-09-11T01:56:17.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.868+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1430253045}) 2024-09-11T01:56:17.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.868+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.868+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=132748896}) 2024-09-11T01:56:17.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.868+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3806517035}) 2024-09-11T01:56:17.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.868+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.868+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=686616023}) 2024-09-11T01:56:17.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.868+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3770832165}) 2024-09-11T01:56:17.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.868+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.868+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=686616023}) 2024-09-11T01:56:17.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.868+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3919573614}) 2024-09-11T01:56:17.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.869+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.869+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=330782119}) 2024-09-11T01:56:17.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.869+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2573942889}) 2024-09-11T01:56:17.892 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.869+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.869+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=330782119}) 2024-09-11T01:56:17.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.869+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2995012477}) 2024-09-11T01:56:17.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.870+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.870+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=159977130}) 2024-09-11T01:56:17.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.870+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1456337032}) 2024-09-11T01:56:17.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.870+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.870+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=159977130}) 2024-09-11T01:56:17.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.870+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2500639197}) 2024-09-11T01:56:17.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.870+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.870+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2436895062}) 2024-09-11T01:56:17.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.870+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3302602445}) 2024-09-11T01:56:17.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.870+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.870+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2436895062}) 2024-09-11T01:56:17.893 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.870+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4064431278}) 2024-09-11T01:56:17.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.871+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.871+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3958647807}) 2024-09-11T01:56:17.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.871+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=83312537}) 2024-09-11T01:56:17.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.871+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.871+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3958647807}) 2024-09-11T01:56:17.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.871+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3670399867}) 2024-09-11T01:56:17.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.871+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.871+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1870483775}) 2024-09-11T01:56:17.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.871+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=595953189}) 2024-09-11T01:56:17.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.872+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.872+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1870483775}) 2024-09-11T01:56:17.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.872+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3244176513}) 2024-09-11T01:56:17.894 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.872+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.895 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.872+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=64607358}) 2024-09-11T01:56:17.895 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.872+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=38746275}) 2024-09-11T01:56:17.895 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.872+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.895 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.872+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=64607358}) 2024-09-11T01:56:17.895 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.872+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=499745909}) 2024-09-11T01:56:17.895 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.873+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.895 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.873+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3616492642}) 2024-09-11T01:56:17.895 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.873+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=636263869}) 2024-09-11T01:56:17.895 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.873+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.895 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.873+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3616492642}) 2024-09-11T01:56:17.895 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.873+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2679429533}) 2024-09-11T01:56:17.896 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.874+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.896 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.874+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1552512300}) 2024-09-11T01:56:17.896 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.874+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=614256224}) 2024-09-11T01:56:17.896 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.874+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.896 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.874+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1552512300}) 2024-09-11T01:56:17.896 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.874+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=176350230}) 2024-09-11T01:56:17.896 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.874+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.896 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.874+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2848727671}) 2024-09-11T01:56:17.896 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.874+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2259199106}) 2024-09-11T01:56:17.896 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.874+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.896 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.874+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2848727671}) 2024-09-11T01:56:17.896 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.874+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1575897554}) 2024-09-11T01:56:17.896 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.875+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.875+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4118952180}) 2024-09-11T01:56:17.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.875+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2504607691}) 2024-09-11T01:56:17.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.875+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.875+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4118952180}) 2024-09-11T01:56:17.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.875+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2699856444}) 2024-09-11T01:56:17.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.876+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.876+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3408871652}) 2024-09-11T01:56:17.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.876+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1028796825}) 2024-09-11T01:56:17.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.876+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.876+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3408871652}) 2024-09-11T01:56:17.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.876+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3742169339}) 2024-09-11T01:56:17.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.876+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.876+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2381187575}) 2024-09-11T01:56:17.897 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.876+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3868864078}) 2024-09-11T01:56:17.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.876+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.876+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2381187575}) 2024-09-11T01:56:17.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.876+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=180844825}) 2024-09-11T01:56:17.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.877+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.877+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4041285755}) 2024-09-11T01:56:17.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.877+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=679930836}) 2024-09-11T01:56:17.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.877+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.877+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4041285755}) 2024-09-11T01:56:17.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.877+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2120330005}) 2024-09-11T01:56:17.898 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.878+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.878+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4048318256}) 2024-09-11T01:56:17.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.878+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=836356981}) 2024-09-11T01:56:17.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.878+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.878+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4048318256}) 2024-09-11T01:56:17.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.878+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1710866284}) 2024-09-11T01:56:17.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.878+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.878+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2492787685}) 2024-09-11T01:56:17.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.878+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=909467197}) 2024-09-11T01:56:17.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.878+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.878+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2492787685}) 2024-09-11T01:56:17.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.878+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3676973879}) 2024-09-11T01:56:17.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.879+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.879+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=659248922}) 2024-09-11T01:56:17.899 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.879+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=564366738}) 2024-09-11T01:56:17.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.879+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.879+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=659248922}) 2024-09-11T01:56:17.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.879+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3508751207}) 2024-09-11T01:56:17.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.880+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.880+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2363426711}) 2024-09-11T01:56:17.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.880+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2045364619}) 2024-09-11T01:56:17.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.880+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.880+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2363426711}) 2024-09-11T01:56:17.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.880+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2796513059}) 2024-09-11T01:56:17.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.880+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.880+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3177062453}) 2024-09-11T01:56:17.900 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.880+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3010184032}) 2024-09-11T01:56:17.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.880+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.880+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3177062453}) 2024-09-11T01:56:17.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.880+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3088270471}) 2024-09-11T01:56:17.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.881+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.881+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3201650883}) 2024-09-11T01:56:17.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.881+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2335388321}) 2024-09-11T01:56:17.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.881+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.881+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3201650883}) 2024-09-11T01:56:17.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.881+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3976745789}) 2024-09-11T01:56:17.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.882+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.882+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1758181273}) 2024-09-11T01:56:17.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.882+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3186912037}) 2024-09-11T01:56:17.901 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.882+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.902 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.882+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1758181273}) 2024-09-11T01:56:17.902 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.882+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=80279465}) 2024-09-11T01:56:17.902 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.882+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.902 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.882+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1602636047}) 2024-09-11T01:56:17.902 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.882+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=904687485}) 2024-09-11T01:56:17.902 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.882+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.902 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.882+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1602636047}) 2024-09-11T01:56:17.902 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.882+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=671260235}) 2024-09-11T01:56:17.902 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.883+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.902 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.883+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3480290777}) 2024-09-11T01:56:17.902 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.883+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2208912802}) 2024-09-11T01:56:17.902 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.883+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.903 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.883+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3480290777}) 2024-09-11T01:56:17.903 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.883+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=523433673}) 2024-09-11T01:56:17.903 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.883+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.903 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.883+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1998127800}) 2024-09-11T01:56:17.903 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.883+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=127548703}) 2024-09-11T01:56:17.903 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.883+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.903 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.883+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1998127800}) 2024-09-11T01:56:17.903 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.883+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1339618342}) 2024-09-11T01:56:17.903 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.884+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.903 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.884+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3909855211}) 2024-09-11T01:56:17.903 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.884+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=84851675}) 2024-09-11T01:56:17.903 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.884+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.903 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.884+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3909855211}) 2024-09-11T01:56:17.903 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.884+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3107432320}) 2024-09-11T01:56:17.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.885+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.885+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=727444455}) 2024-09-11T01:56:17.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.885+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1357053895}) 2024-09-11T01:56:17.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.885+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.885+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=727444455}) 2024-09-11T01:56:17.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.885+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3696044328}) 2024-09-11T01:56:17.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.885+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.885+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2631215617}) 2024-09-11T01:56:17.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.885+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1345578238}) 2024-09-11T01:56:17.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.885+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.885+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2631215617}) 2024-09-11T01:56:17.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.885+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2757893771}) 2024-09-11T01:56:17.904 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.886+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.886+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1517048204}) 2024-09-11T01:56:17.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.886+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=787309482}) 2024-09-11T01:56:17.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.886+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.886+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1517048204}) 2024-09-11T01:56:17.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.886+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1464260092}) 2024-09-11T01:56:17.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.887+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.887+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2248209020}) 2024-09-11T01:56:17.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.887+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=516566201}) 2024-09-11T01:56:17.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.887+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.887+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2248209020}) 2024-09-11T01:56:17.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.887+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2728116146}) 2024-09-11T01:56:17.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.888+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.888+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1771967846}) 2024-09-11T01:56:17.905 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.888+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1087508645}) 2024-09-11T01:56:17.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.888+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.888+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1771967846}) 2024-09-11T01:56:17.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.888+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3557197512}) 2024-09-11T01:56:17.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.888+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.888+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2455667398}) 2024-09-11T01:56:17.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.888+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=983572293}) 2024-09-11T01:56:17.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.888+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.888+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2455667398}) 2024-09-11T01:56:17.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.888+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2559481856}) 2024-09-11T01:56:17.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.889+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.889+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1941327288}) 2024-09-11T01:56:17.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.889+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3173446993}) 2024-09-11T01:56:17.906 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.889+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.907 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.889+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1941327288}) 2024-09-11T01:56:17.907 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.889+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=662678151}) 2024-09-11T01:56:17.932 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.889+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.932 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.889+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1472728963}) 2024-09-11T01:56:17.933 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.889+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1779509738}) 2024-09-11T01:56:17.933 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.889+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.933 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.889+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1472728963}) 2024-09-11T01:56:17.933 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.890+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1651317843}) 2024-09-11T01:56:17.933 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.890+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.934 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.890+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2701760612}) 2024-09-11T01:56:17.934 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.890+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=927910623}) 2024-09-11T01:56:17.934 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.890+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.934 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.890+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2701760612}) 2024-09-11T01:56:17.934 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.890+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=850031641}) 2024-09-11T01:56:17.934 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.891+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.934 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.891+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3516064164}) 2024-09-11T01:56:17.935 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.891+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2320712736}) 2024-09-11T01:56:17.935 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.891+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.935 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.891+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3516064164}) 2024-09-11T01:56:17.935 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.891+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1585535372}) 2024-09-11T01:56:17.935 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.891+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.936 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.891+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=160528501}) 2024-09-11T01:56:17.936 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.891+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=39317392}) 2024-09-11T01:56:17.936 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.891+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.936 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.891+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=160528501}) 2024-09-11T01:56:17.936 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.891+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=310021040}) 2024-09-11T01:56:17.936 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.892+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.937 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.892+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1439311808}) 2024-09-11T01:56:17.937 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.892+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2139342776}) 2024-09-11T01:56:17.937 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.892+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.937 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.892+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1439311808}) 2024-09-11T01:56:17.937 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.892+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3015115459}) 2024-09-11T01:56:17.937 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.893+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.893+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=108375715}) 2024-09-11T01:56:17.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.893+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3846870383}) 2024-09-11T01:56:17.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.893+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.893+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=108375715}) 2024-09-11T01:56:17.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.893+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3176307054}) 2024-09-11T01:56:17.938 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.893+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.893+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4272578310}) 2024-09-11T01:56:17.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.893+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1141679108}) 2024-09-11T01:56:17.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.893+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.893+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4272578310}) 2024-09-11T01:56:17.939 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.893+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3384813677}) 2024-09-11T01:56:17.940 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.894+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.940 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.894+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4012196244}) 2024-09-11T01:56:17.940 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.894+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1601493290}) 2024-09-11T01:56:17.940 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.894+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.940 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.894+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4012196244}) 2024-09-11T01:56:17.940 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.894+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3440909714}) 2024-09-11T01:56:17.940 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.895+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.940 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.895+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2639027382}) 2024-09-11T01:56:17.941 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.895+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2970327809}) 2024-09-11T01:56:17.941 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.895+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.941 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.895+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2639027382}) 2024-09-11T01:56:17.941 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.895+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1056225979}) 2024-09-11T01:56:17.941 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.895+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.941 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.895+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3131143538}) 2024-09-11T01:56:17.942 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.895+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3619500479}) 2024-09-11T01:56:17.942 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.895+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.942 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.895+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3131143538}) 2024-09-11T01:56:17.942 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.895+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4082959482}) 2024-09-11T01:56:17.942 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.896+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.942 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.896+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1408083306}) 2024-09-11T01:56:17.942 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.896+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3543448744}) 2024-09-11T01:56:17.942 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.896+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.942 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.896+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1408083306}) 2024-09-11T01:56:17.942 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.896+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4275741906}) 2024-09-11T01:56:17.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.897+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.897+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2782176622}) 2024-09-11T01:56:17.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.897+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2365443968}) 2024-09-11T01:56:17.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.897+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.897+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2782176622}) 2024-09-11T01:56:17.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.897+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1320151970}) 2024-09-11T01:56:17.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.897+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.897+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562452025}) 2024-09-11T01:56:17.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.897+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2998305253}) 2024-09-11T01:56:17.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.897+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.897+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562452025}) 2024-09-11T01:56:17.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.897+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4224516506}) 2024-09-11T01:56:17.943 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.905+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.944 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.905+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=59587794}) 2024-09-11T01:56:17.944 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.905+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=853442792}) 2024-09-11T01:56:17.944 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.905+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.944 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.905+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=59587794}) 2024-09-11T01:56:17.944 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.905+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1107376194}) 2024-09-11T01:56:17.944 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.906+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.944 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.906+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=670906709}) 2024-09-11T01:56:17.944 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.906+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2726766796}) 2024-09-11T01:56:17.944 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.906+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.944 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.906+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=670906709}) 2024-09-11T01:56:17.944 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.906+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1174272735}) 2024-09-11T01:56:17.944 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.906+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.944 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.906+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1395597536}) 2024-09-11T01:56:17.944 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.906+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1502659129}) 2024-09-11T01:56:17.945 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.906+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.945 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.906+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1395597536}) 2024-09-11T01:56:17.945 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.906+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=805803651}) 2024-09-11T01:56:17.945 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.907+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.945 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.907+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3405644170}) 2024-09-11T01:56:17.945 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.907+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1859902187}) 2024-09-11T01:56:17.945 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.907+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.945 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.907+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3405644170}) 2024-09-11T01:56:17.945 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.907+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4248627995}) 2024-09-11T01:56:17.945 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.908+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.945 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.908+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4226997790}) 2024-09-11T01:56:17.945 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.908+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2449476287}) 2024-09-11T01:56:17.945 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.908+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.908+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4226997790}) 2024-09-11T01:56:17.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.908+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3497090926}) 2024-09-11T01:56:17.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.909+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.909+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1347104288}) 2024-09-11T01:56:17.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.909+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3413669991}) 2024-09-11T01:56:17.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.909+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.909+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1347104288}) 2024-09-11T01:56:17.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.909+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=75164221}) 2024-09-11T01:56:17.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.909+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.909+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3231227749}) 2024-09-11T01:56:17.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.909+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1030883867}) 2024-09-11T01:56:17.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.909+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.909+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3231227749}) 2024-09-11T01:56:17.946 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.909+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3298741709}) 2024-09-11T01:56:17.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.910+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.910+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2870111062}) 2024-09-11T01:56:17.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.910+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=795668548}) 2024-09-11T01:56:17.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.910+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.910+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2870111062}) 2024-09-11T01:56:17.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.910+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1105543541}) 2024-09-11T01:56:17.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.911+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.911+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1386184659}) 2024-09-11T01:56:17.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.911+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3225154310}) 2024-09-11T01:56:17.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.911+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.911+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1386184659}) 2024-09-11T01:56:17.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.911+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1629585699}) 2024-09-11T01:56:17.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.911+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.947 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.911+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1199023940}) 2024-09-11T01:56:17.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.911+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3447514203}) 2024-09-11T01:56:17.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.911+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.911+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1199023940}) 2024-09-11T01:56:17.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.911+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3557757382}) 2024-09-11T01:56:17.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.912+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.912+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2664444623}) 2024-09-11T01:56:17.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.912+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3606252025}) 2024-09-11T01:56:17.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.912+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.912+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2664444623}) 2024-09-11T01:56:17.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.912+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3972706753}) 2024-09-11T01:56:17.948 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.913+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.913+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3321224912}) 2024-09-11T01:56:17.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.913+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1237861400}) 2024-09-11T01:56:17.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.913+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.913+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3321224912}) 2024-09-11T01:56:17.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.913+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3541182827}) 2024-09-11T01:56:17.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.914+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.914+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1806587154}) 2024-09-11T01:56:17.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.914+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1331203579}) 2024-09-11T01:56:17.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.914+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.914+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1806587154}) 2024-09-11T01:56:17.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.914+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3410624471}) 2024-09-11T01:56:17.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.914+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.914+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=922778098}) 2024-09-11T01:56:17.949 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.914+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=609604587}) 2024-09-11T01:56:17.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.914+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.914+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=922778098}) 2024-09-11T01:56:17.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.914+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2158252907}) 2024-09-11T01:56:17.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.915+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.915+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1426847523}) 2024-09-11T01:56:17.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.915+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=912746898}) 2024-09-11T01:56:17.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.915+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.915+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1426847523}) 2024-09-11T01:56:17.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.915+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3116350379}) 2024-09-11T01:56:17.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.915+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.915+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=429092868}) 2024-09-11T01:56:17.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.915+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2138590947}) 2024-09-11T01:56:17.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.916+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.950 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.916+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=429092868}) 2024-09-11T01:56:17.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.916+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2807787845}) 2024-09-11T01:56:17.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.916+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.916+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2113571688}) 2024-09-11T01:56:17.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.916+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=67879042}) 2024-09-11T01:56:17.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.916+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.916+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2113571688}) 2024-09-11T01:56:17.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.916+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3897187332}) 2024-09-11T01:56:17.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.917+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.917+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3792699670}) 2024-09-11T01:56:17.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.917+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3795719911}) 2024-09-11T01:56:17.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.917+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.917+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3792699670}) 2024-09-11T01:56:17.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.917+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=695254407}) 2024-09-11T01:56:17.951 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.917+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.917+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2783594333}) 2024-09-11T01:56:17.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.917+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3430720368}) 2024-09-11T01:56:17.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.917+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.917+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2783594333}) 2024-09-11T01:56:17.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.917+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3216880807}) 2024-09-11T01:56:17.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.918+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.918+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1359721282}) 2024-09-11T01:56:17.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.918+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3997568950}) 2024-09-11T01:56:17.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.918+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.918+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1359721282}) 2024-09-11T01:56:17.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.918+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=824056591}) 2024-09-11T01:56:17.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.919+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.952 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.919+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=64,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=13} crc {logm=20890074,mdsmap=3760429986,mgr=3526296668,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=749007249,mon_config_key=1552104035}) 2024-09-11T01:56:17.953 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.919+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=66,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11} crc {logm=1760954461,mdsmap=3760429986,mgr=4080174641,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=749007249,mon_config_key=3207299444}) 2024-09-11T01:56:17.953 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.919+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.953 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.919+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=64,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=13} crc {logm=20890074,mdsmap=3760429986,mgr=3526296668,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=749007249,mon_config_key=1552104035}) 2024-09-11T01:56:17.953 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.919+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=60,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=15,monmap=3} crc {logm=219749598,mdsmap=3760429986,mgr=3108067518,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1381888844,mon_config_key=301872969,monmap=2933275274}) 2024-09-11T01:56:17.953 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.920+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.953 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.920+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=3,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=85} crc {mon_config_key=2844315700,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2239697408,osd_pg_creating=2109091619,osd_snap=3858899184}) 2024-09-11T01:56:17.953 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.920+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=4,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=84} crc {mon_config_key=2571357278,monmap=2933275274,nvmeofgw=3140514009,osd_metadata=2942852742,osd_pg_creating=2109091619,osd_snap=653966940}) 2024-09-11T01:56:17.953 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.920+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.953 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.920+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=3,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=85} crc {mon_config_key=2844315700,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2239697408,osd_pg_creating=2109091619,osd_snap=3858899184}) 2024-09-11T01:56:17.953 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.920+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=4,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=84} crc {mon_config_key=2571357278,monmap=2933275274,nvmeofgw=598337686,osd_metadata=950107459,osd_pg_creating=2109091619,osd_snap=210862773}) 2024-09-11T01:56:17.953 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.921+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.953 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.921+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=89,osdmap=11} crc {osd_snap=2884890153,osdmap=3286580103}) 2024-09-11T01:56:17.953 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.921+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=90,osdmap=10} crc {osd_snap=4072153018,osdmap=955854241}) 2024-09-11T01:56:17.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.921+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.921+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=89,osdmap=11} crc {osd_snap=2884890153,osdmap=3286580103}) 2024-09-11T01:56:17.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.921+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=87,osdmap=13} crc {osd_snap=3418652957,osdmap=795138114}) 2024-09-11T01:56:17.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.921+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.921+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1363365625}) 2024-09-11T01:56:17.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.921+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1508726041}) 2024-09-11T01:56:17.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.921+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.921+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1363365625}) 2024-09-11T01:56:17.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.921+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3617034234}) 2024-09-11T01:56:17.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.922+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.922+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3618380722}) 2024-09-11T01:56:17.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.922+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1291951641}) 2024-09-11T01:56:17.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.922+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.954 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.922+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3618380722}) 2024-09-11T01:56:17.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.922+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3613565040}) 2024-09-11T01:56:17.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.923+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.923+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2614582986}) 2024-09-11T01:56:17.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.923+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=487507611}) 2024-09-11T01:56:17.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.923+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.923+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2614582986}) 2024-09-11T01:56:17.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.923+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2313010951}) 2024-09-11T01:56:17.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.929+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.929+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=49} crc {osdmap=2005461595}) 2024-09-11T01:56:17.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.929+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=47} crc {osdmap=2305942695}) 2024-09-11T01:56:17.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.929+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:56:17.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.929+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=49} crc {osdmap=2005461595}) 2024-09-11T01:56:17.955 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:56:17.929+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=49} crc {osdmap=3803531089}) 2024-09-11T01:56:18.198 INFO:tasks.rados.rados.0.smithi071.stdout:3988: done (4 left) 2024-09-11T01:56:18.198 INFO:tasks.rados.rados.0.smithi071.stdout:3989: done (3 left) 2024-09-11T01:56:18.198 INFO:tasks.rados.rados.0.smithi071.stdout:3990: done (2 left) 2024-09-11T01:56:18.198 INFO:tasks.rados.rados.0.smithi071.stdout:3991: done (1 left) 2024-09-11T01:56:18.199 INFO:tasks.rados.rados.0.smithi071.stdout:3992: done (0 left) 2024-09-11T01:56:18.199 INFO:tasks.rados.rados.0.smithi071.stdout:3993: read oid 17 snap -1 2024-09-11T01:56:18.199 INFO:tasks.rados.rados.0.smithi071.stdout:3993: expect (ObjNum 1301 snap 426 seq_num 1301) 2024-09-11T01:56:18.199 INFO:tasks.rados.rados.0.smithi071.stdout:3994: snap_remove snap 447 2024-09-11T01:56:18.641 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:56:18.642 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:56:18.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.749+0000 7fb963ebe640 10 monclient: get_monmap_and_config 2024-09-11T01:56:18.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.749+0000 7fb963ebe640 10 monclient: build_initial_monmap 2024-09-11T01:56:18.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.749+0000 7fb963ebe640 10 monclient: monmap: 2024-09-11T01:56:18.751 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:18.751 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:18.751 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:56:18.750488+0000 2024-09-11T01:56:18.751 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:56:18.750488+0000 2024-09-11T01:56:18.752 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:18.752 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:18.752 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:56:18.752 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:56:18.752 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:56:18.752 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:56:18.752 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:56:18.752 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:18.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.750+0000 7fb963ebe640 1 Processor -- start 2024-09-11T01:56:18.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.750+0000 7fb963ebe640 1 -- start start 2024-09-11T01:56:18.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.750+0000 7fb963ebe640 10 monclient: init 2024-09-11T01:56:18.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb963ebe640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:18.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb963ebe640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-11T01:56:18.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb963ebe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb95c073d90 0x7fb95c074170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:18.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb963ebe640 10 monclient(hunting): picked mon.noname-b con 0x7fb95c073d90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:18.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb963ebe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb95c0a0660 0x7fb95c0a2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:18.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb963ebe640 10 monclient(hunting): picked mon.noname-a con 0x7fb95c0a0660 addr v2:172.21.15.12:3300/0 2024-09-11T01:56:18.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb963ebe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb95c1380d0 0x7fb95c0a2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:18.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb963ebe640 10 monclient(hunting): picked mon.noname-c con 0x7fb95c1380d0 addr v2:172.21.15.71:3300/0 2024-09-11T01:56:18.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb963ebe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:18.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb963ebe640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fb95c070310 con 0x7fb95c0a0660 2024-09-11T01:56:18.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb963ebe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:18.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb963ebe640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fb95c070560 con 0x7fb95c1380d0 2024-09-11T01:56:18.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb963ebe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:18.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb963ebe640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb95c070840 con 0x7fb95c073d90 2024-09-11T01:56:18.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb963ebe640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:18.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb963ebe640 10 monclient(hunting): authenticate will time out at 2385.193160s 2024-09-11T01:56:18.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961c5a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb95c073d90 0x7fb95c074170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:18.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961459640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb95c0a0660 0x7fb95c0a2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:18.756 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961459640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb95c0a0660 0x7fb95c0a2560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49768/0 (socket says 172.21.15.12:49768) 2024-09-11T01:56:18.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961c5a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb95c073d90 0x7fb95c074170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46744/0 (socket says 172.21.15.12:46744) 2024-09-11T01:56:18.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961459640 1 -- 172.21.15.12:0/674211344 learned_addr learned my addr 172.21.15.12:0/674211344 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:18.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961459640 10 monclient(hunting): get_auth_request con 0x7fb95c0a0660 auth_method 0 2024-09-11T01:56:18.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961459640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:18.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961459640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:18.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961459640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:18.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961c5a640 10 monclient(hunting): get_auth_request con 0x7fb95c073d90 auth_method 0 2024-09-11T01:56:18.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961c5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:18.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961c5a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:18.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961c5a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:18.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961c5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:18.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961c5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:18.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961c5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:18.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961459640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:18.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961459640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:18.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961459640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:18.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb96245b640 1 --2- 172.21.15.12:0/674211344 >> v2:172.21.15.71:3300/0 conn(0x7fb95c1380d0 0x7fb95c0a2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:18.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961c5a640 10 monclient(hunting): handle_auth_done global_id 304124 payload 274 2024-09-11T01:56:18.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961c5a640 1 -- 172.21.15.12:0/674211344 >> v2:172.21.15.71:3300/0 conn(0x7fb95c1380d0 msgr2=0x7fb95c0a2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:18.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961c5a640 1 --2- 172.21.15.12:0/674211344 >> v2:172.21.15.71:3300/0 conn(0x7fb95c1380d0 0x7fb95c0a2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:18.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961c5a640 1 -- 172.21.15.12:0/674211344 >> v2:172.21.15.12:3300/0 conn(0x7fb95c0a0660 msgr2=0x7fb95c0a2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:18.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961c5a640 1 --2- 172.21.15.12:0/674211344 >> v2:172.21.15.12:3300/0 conn(0x7fb95c0a0660 0x7fb95c0a2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:18.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961c5a640 10 monclient: _finish_hunting 0 2024-09-11T01:56:18.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961c5a640 1 monclient: found mon.noname-b 2024-09-11T01:56:18.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961c5a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:18.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961c5a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:56:18.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961c5a640 1 -- 172.21.15.12:0/674211344 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb95c09fff0 con 0x7fb95c073d90 2024-09-11T01:56:18.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb96245b640 10 monclient: get_auth_request con 0x7fb95c1380d0 auth_method 0 2024-09-11T01:56:18.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb961459640 1 --2- 172.21.15.12:0/674211344 >> v2:172.21.15.12:3300/0 conn(0x7fb95c0a0660 0x7fb95c0a2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:56:18.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.751+0000 7fb96245b640 1 --2- 172.21.15.12:0/674211344 >> v2:172.21.15.71:3300/0 conn(0x7fb95c1380d0 0x7fb95c0a2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:56:18.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.754+0000 7fb961c5a640 1 -- 172.21.15.12:0/674211344 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb95c073d90 msgr2=0x7fb95c074170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:18.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.754+0000 7fb961c5a640 1 --2- 172.21.15.12:0/674211344 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb95c073d90 0x7fb95c074170 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb95c0a0220 tx=0x7fb958029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d6c60d5b6e2422a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:18.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.754+0000 7fb960c58640 1 -- 172.21.15.12:0/674211344 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb958037070 con 0x7fb95c073d90 2024-09-11T01:56:18.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.754+0000 7fb960c58640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:18.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.754+0000 7fb960c58640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:56:18.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.754+0000 7fb960c58640 10 monclient: dump: 2024-09-11T01:56:18.767 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:18.767 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:18.767 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:18.767 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:18.767 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:18.767 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:18.767 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:18.767 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:18.767 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:18.767 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:18.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.754+0000 7fb960c58640 10 monclient: _finish_auth 0 2024-09-11T01:56:18.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.754+0000 7fb960c58640 10 monclient: _check_auth_tickets 2024-09-11T01:56:18.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.754+0000 7fb960c58640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:18.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.754+0000 7fb960c58640 1 -- 172.21.15.12:0/674211344 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb958032440 con 0x7fb95c073d90 2024-09-11T01:56:18.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.754+0000 7fb963ebe640 5 monclient: authenticate success, global_id 304124 2024-09-11T01:56:18.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.754+0000 7fb963ebe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:56:18.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.754+0000 7fb960c58640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:18.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.754+0000 7fb963ebe640 10 monclient: get_monmap_and_config success 2024-09-11T01:56:18.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.754+0000 7fb960c58640 1 -- 172.21.15.12:0/674211344 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb958032810 con 0x7fb95c073d90 2024-09-11T01:56:18.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.754+0000 7fb960c58640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:18.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.754+0000 7fb963ebe640 4 set_mon_vals no callback set 2024-09-11T01:56:18.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.754+0000 7fb960c58640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:56:18.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.754+0000 7fb960c58640 10 monclient: dump: 2024-09-11T01:56:18.769 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:18.769 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:18.769 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:18.769 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:18.769 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:18.769 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:18.769 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:18.769 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:18.769 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:18.769 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:18.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.755+0000 7fb963ebe640 10 monclient: shutdown 2024-09-11T01:56:18.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.755+0000 7fb963ebe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:18.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.755+0000 7fb963ebe640 1 -- 172.21.15.12:0/674211344 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb95c073d90 msgr2=0x7fb95c074170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:18.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.755+0000 7fb963ebe640 1 --2- 172.21.15.12:0/674211344 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb95c073d90 0x7fb95c074170 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb95c0a0220 tx=0x7fb958029fd0 comp rx=0 tx=0).stop 2024-09-11T01:56:18.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.755+0000 7fb963ebe640 1 -- 172.21.15.12:0/674211344 shutdown_connections 2024-09-11T01:56:18.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.755+0000 7fb963ebe640 1 --2- 172.21.15.12:0/674211344 >> v2:172.21.15.71:3300/0 conn(0x7fb95c1380d0 0x7fb95c0a2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:18.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.755+0000 7fb963ebe640 1 --2- 172.21.15.12:0/674211344 >> v2:172.21.15.12:3300/0 conn(0x7fb95c0a0660 0x7fb95c0a2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:18.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.755+0000 7fb963ebe640 1 --2- 172.21.15.12:0/674211344 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb95c073d90 0x7fb95c074170 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:18.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.755+0000 7fb963ebe640 1 -- 172.21.15.12:0/674211344 >> 172.21.15.12:0/674211344 conn(0x7fb95c0a3650 msgr2=0x7fb95c0a3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:18.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.755+0000 7fb963ebe640 1 -- 172.21.15.12:0/674211344 shutdown_connections 2024-09-11T01:56:18.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.755+0000 7fb963ebe640 1 -- 172.21.15.12:0/674211344 wait complete. 2024-09-11T01:56:18.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.755+0000 7fb963ebe640 10 monclient: build_initial_monmap 2024-09-11T01:56:18.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.755+0000 7fb963ebe640 10 monclient: monmap: 2024-09-11T01:56:18.770 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:18.770 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:56:18.770 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:56:18.771 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:56:18.771 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:18.771 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:18.771 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:56:18.771 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:56:18.771 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:56:18.771 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:18.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.756+0000 7fb963ebe640 1 Processor -- start 2024-09-11T01:56:18.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.756+0000 7fb963ebe640 1 -- start start 2024-09-11T01:56:18.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.756+0000 7fb963ebe640 10 monclient: init 2024-09-11T01:56:18.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.756+0000 7fb963ebe640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:18.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.756+0000 7fb963ebe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:56:18.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.756+0000 7fb963ebe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb95c073d90 0x7fb95c130b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:18.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.756+0000 7fb963ebe640 10 monclient(hunting): picked mon.noname-a con 0x7fb95c073d90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:56:18.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.756+0000 7fb963ebe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb95c0a0660 0x7fb95c131040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:18.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.756+0000 7fb963ebe640 10 monclient(hunting): picked mon.noname-c con 0x7fb95c0a0660 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:18.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.756+0000 7fb963ebe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb95c1380d0 0x7fb95c1bf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:18.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.756+0000 7fb963ebe640 10 monclient(hunting): picked mon.noname-b con 0x7fb95c1380d0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:56:18.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.756+0000 7fb963ebe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:18.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.756+0000 7fb963ebe640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb95c0a44d0 con 0x7fb95c073d90 2024-09-11T01:56:18.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.756+0000 7fb963ebe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:18.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.756+0000 7fb963ebe640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb95c0a4350 con 0x7fb95c1380d0 2024-09-11T01:56:18.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.756+0000 7fb963ebe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:18.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.756+0000 7fb963ebe640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb95c0a2fe0 con 0x7fb95c0a0660 2024-09-11T01:56:18.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.756+0000 7fb963ebe640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:18.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.756+0000 7fb963ebe640 10 monclient(hunting): authenticate will time out at 2385.198993s 2024-09-11T01:56:18.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961c5a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb95c073d90 0x7fb95c130b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:18.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961c5a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb95c073d90 0x7fb95c130b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49778/0 (socket says 172.21.15.12:49778) 2024-09-11T01:56:18.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961459640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb95c0a0660 0x7fb95c131040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:18.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961c5a640 1 -- 172.21.15.12:0/2265726379 learned_addr learned my addr 172.21.15.12:0/2265726379 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:18.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961c5a640 10 monclient(hunting): get_auth_request con 0x7fb95c073d90 auth_method 0 2024-09-11T01:56:18.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961c5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:18.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961c5a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:18.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961c5a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:18.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961459640 10 monclient(hunting): get_auth_request con 0x7fb95c0a0660 auth_method 0 2024-09-11T01:56:18.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961459640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:18.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961459640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:18.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961459640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:18.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961c5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:18.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961c5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:18.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961c5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:18.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961459640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:18.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961459640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:18.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961459640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:18.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb96245b640 1 --2- 172.21.15.12:0/2265726379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb95c1380d0 0x7fb95c1bf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:18.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961459640 10 monclient(hunting): handle_auth_done global_id 304127 payload 824 2024-09-11T01:56:18.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961459640 1 -- 172.21.15.12:0/2265726379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb95c1380d0 msgr2=0x7fb95c1bf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:18.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961459640 1 --2- 172.21.15.12:0/2265726379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb95c1380d0 0x7fb95c1bf5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:18.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961459640 1 -- 172.21.15.12:0/2265726379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb95c073d90 msgr2=0x7fb95c130b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:18.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961459640 1 --2- 172.21.15.12:0/2265726379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb95c073d90 0x7fb95c130b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:18.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961459640 10 monclient: _finish_hunting 0 2024-09-11T01:56:18.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961459640 1 monclient: found mon.noname-c 2024-09-11T01:56:18.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961459640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:18.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961459640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:56:18.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961459640 1 -- 172.21.15.12:0/2265726379 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb95c131ac0 con 0x7fb95c0a0660 2024-09-11T01:56:18.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb96245b640 10 monclient: get_auth_request con 0x7fb95c1380d0 auth_method 0 2024-09-11T01:56:18.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb961c5a640 1 --2- 172.21.15.12:0/2265726379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb95c073d90 0x7fb95c130b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:56:18.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.757+0000 7fb96245b640 1 --2- 172.21.15.12:0/2265726379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb95c1380d0 0x7fb95c1bf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:56:18.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.759+0000 7fb961459640 1 -- 172.21.15.12:0/2265726379 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb95c0a0660 msgr2=0x7fb95c131040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:18.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.759+0000 7fb961459640 1 --2- 172.21.15.12:0/2265726379 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb95c0a0660 0x7fb95c131040 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb95c0a0a80 tx=0x7fb94c00ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:18.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.759+0000 7fb9527fc640 1 -- 172.21.15.12:0/2265726379 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94c00cac0 con 0x7fb95c0a0660 2024-09-11T01:56:18.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.759+0000 7fb9527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:18.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.759+0000 7fb9527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:56:18.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.759+0000 7fb9527fc640 10 monclient: dump: 2024-09-11T01:56:18.776 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:18.776 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:18.776 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:18.776 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:18.776 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:18.776 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:18.776 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:18.776 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:18.776 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:18.777 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:18.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.759+0000 7fb9527fc640 10 monclient: _finish_auth 0 2024-09-11T01:56:18.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.759+0000 7fb9527fc640 10 monclient: _check_auth_tickets 2024-09-11T01:56:18.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.759+0000 7fb9527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:18.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.759+0000 7fb963ebe640 5 monclient: authenticate success, global_id 304127 2024-09-11T01:56:18.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.759+0000 7fb9527fc640 1 -- 172.21.15.12:0/2265726379 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb94c00cc60 con 0x7fb95c0a0660 2024-09-11T01:56:18.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.759+0000 7fb9527fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:18.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.759+0000 7fb9527fc640 1 -- 172.21.15.12:0/2265726379 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94c004250 con 0x7fb95c0a0660 2024-09-11T01:56:18.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.759+0000 7fb963ebe640 10 monclient: _renew_subs 2024-09-11T01:56:18.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.759+0000 7fb963ebe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:56:18.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.759+0000 7fb963ebe640 1 -- 172.21.15.12:0/2265726379 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb95c11e830 con 0x7fb95c0a0660 2024-09-11T01:56:18.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.759+0000 7fb9527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:18.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.759+0000 7fb9527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:56:18.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.759+0000 7fb9527fc640 10 monclient: dump: 2024-09-11T01:56:18.778 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:18.778 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:18.778 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:18.778 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:18.778 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:18.778 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:18.778 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:18.778 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:18.778 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:18.778 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:18.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.759+0000 7fb953fff640 4 set_mon_vals no callback set 2024-09-11T01:56:18.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.759+0000 7fb963ebe640 10 monclient: _renew_subs 2024-09-11T01:56:18.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.759+0000 7fb963ebe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:56:18.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.759+0000 7fb963ebe640 1 -- 172.21.15.12:0/2265726379 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb95c11e550 con 0x7fb95c0a0660 2024-09-11T01:56:18.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.760+0000 7fb9527fc640 1 -- 172.21.15.12:0/2265726379 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fb94c023060 con 0x7fb95c0a0660 2024-09-11T01:56:18.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.760+0000 7fb963ebe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:18.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.760+0000 7fb963ebe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:18.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.760+0000 7fb963ebe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:56:18.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.760+0000 7fb963ebe640 1 -- 172.21.15.12:0/2265726379 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb95c074170 con 0x7fb95c0a0660 2024-09-11T01:56:18.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.761+0000 7fb9527fc640 1 --2- 172.21.15.12:0/2265726379 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb94003e290 0x7fb940040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:18.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.761+0000 7fb9527fc640 1 -- 172.21.15.12:0/2265726379 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(919..919 src has 676..919) ==== 4771+0+0 (secure 0 0 0) 0x7fb94c052a00 con 0x7fb95c0a0660 2024-09-11T01:56:18.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.761+0000 7fb961c5a640 1 --2- 172.21.15.12:0/2265726379 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb94003e290 0x7fb940040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:18.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.761+0000 7fb961c5a640 10 monclient: get_auth_request con 0x7fb94003e290 auth_method 0 2024-09-11T01:56:18.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.763+0000 7fb961c5a640 1 --2- 172.21.15.12:0/2265726379 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb94003e290 0x7fb940040750 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fb958009560 tx=0x7fb95802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:18.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.765+0000 7fb9527fc640 1 -- 172.21.15.12:0/2265726379 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb94c017020 con 0x7fb95c0a0660 2024-09-11T01:56:18.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.765+0000 7fb9527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:18.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:18.765+0000 7fb9527fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:56:19.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.024+0000 7fb963ebe640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:56:19.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.024+0000 7fb963ebe640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:56:19.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.024+0000 7fb963ebe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:56:19.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.024+0000 7fb963ebe640 1 -- 172.21.15.12:0/2265726379 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb95c06dca0 con 0x7fb95c0a0660 2024-09-11T01:56:19.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.025+0000 7fb9527fc640 1 -- 172.21.15.12:0/2265726379 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fb94c011350 con 0x7fb95c0a0660 2024-09-11T01:56:19.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.025+0000 7fb9527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:56:19.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.025+0000 7fb9527fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:56:19.027 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":250,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:56:19.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.027+0000 7fb963ebe640 1 -- 172.21.15.12:0/2265726379 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb94003e290 msgr2=0x7fb940040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:19.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.027+0000 7fb963ebe640 1 --2- 172.21.15.12:0/2265726379 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb94003e290 0x7fb940040750 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fb958009560 tx=0x7fb95802e000 comp rx=0 tx=0).stop 2024-09-11T01:56:19.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.027+0000 7fb963ebe640 10 monclient: shutdown 2024-09-11T01:56:19.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.027+0000 7fb963ebe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:19.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.027+0000 7fb963ebe640 1 -- 172.21.15.12:0/2265726379 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb95c0a0660 msgr2=0x7fb95c131040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:19.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.027+0000 7fb963ebe640 1 --2- 172.21.15.12:0/2265726379 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb95c0a0660 0x7fb95c131040 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb95c0a0a80 tx=0x7fb94c00ed90 comp rx=0 tx=0).stop 2024-09-11T01:56:19.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.027+0000 7fb963ebe640 1 -- 172.21.15.12:0/2265726379 shutdown_connections 2024-09-11T01:56:19.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.027+0000 7fb963ebe640 1 --2- 172.21.15.12:0/2265726379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb95c1380d0 0x7fb95c1bf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.027+0000 7fb963ebe640 1 --2- 172.21.15.12:0/2265726379 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb95c0a0660 0x7fb95c131040 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.027+0000 7fb963ebe640 1 --2- 172.21.15.12:0/2265726379 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb94003e290 0x7fb940040750 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.027+0000 7fb963ebe640 1 --2- 172.21.15.12:0/2265726379 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb95c073d90 0x7fb95c130b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.027+0000 7fb963ebe640 1 -- 172.21.15.12:0/2265726379 >> 172.21.15.12:0/2265726379 conn(0x7fb95c0a3650 msgr2=0x7fb95c073330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:19.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.027+0000 7fb963ebe640 1 -- 172.21.15.12:0/2265726379 shutdown_connections 2024-09-11T01:56:19.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.027+0000 7fb963ebe640 1 -- 172.21.15.12:0/2265726379 wait complete. 2024-09-11T01:56:19.039 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:56:19.039 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T01:56:19.039 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:56:19.039 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:56:19.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.140+0000 7f08a7f1f640 10 monclient: get_monmap_and_config 2024-09-11T01:56:19.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.140+0000 7f08a7f1f640 10 monclient: build_initial_monmap 2024-09-11T01:56:19.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.140+0000 7f08a7f1f640 10 monclient: monmap: 2024-09-11T01:56:19.141 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:19.142 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:19.142 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:56:19.140765+0000 2024-09-11T01:56:19.142 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:56:19.140765+0000 2024-09-11T01:56:19.142 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:19.142 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:19.142 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:56:19.142 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:56:19.142 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:56:19.142 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:56:19.142 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:56:19.142 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:19.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.140+0000 7f08a7f1f640 1 Processor -- start 2024-09-11T01:56:19.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.140+0000 7f08a7f1f640 1 -- start start 2024-09-11T01:56:19.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.140+0000 7f08a7f1f640 10 monclient: init 2024-09-11T01:56:19.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a7f1f640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:19.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a7f1f640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-11T01:56:19.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a7f1f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f08a01381c0 addr v1:172.21.15.71:6789/0 2024-09-11T01:56:19.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a7f1f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f08a00a6aa0 0x7f08a009cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:19.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a7f1f640 10 monclient(hunting): picked mon.noname-b con 0x7f08a00a6aa0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:19.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a7f1f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f08a013d000 0x7f08a013f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:19.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a7f1f640 10 monclient(hunting): picked mon.noname-c con 0x7f08a013d000 addr v2:172.21.15.71:3300/0 2024-09-11T01:56:19.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a7f1f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08a009c310 con 0x7f08a01381c0 2024-09-11T01:56:19.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a7f1f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:19.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a7f1f640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f08a006fc90 con 0x7f08a013d000 2024-09-11T01:56:19.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a7f1f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:19.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a7f1f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f08a009c5f0 con 0x7f08a00a6aa0 2024-09-11T01:56:19.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a7f1f640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:19.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a7f1f640 10 monclient(hunting): authenticate will time out at 2385.583350s 2024-09-11T01:56:19.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a54ba640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f08a00a6aa0 0x7f08a009cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:19.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a54ba640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f08a00a6aa0 0x7f08a009cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46768/0 (socket says 172.21.15.12:46768) 2024-09-11T01:56:19.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a54ba640 1 -- 172.21.15.12:0/722464994 learned_addr learned my addr 172.21.15.12:0/722464994 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:19.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a54ba640 10 monclient(hunting): get_auth_request con 0x7f08a00a6aa0 auth_method 0 2024-09-11T01:56:19.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a54ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:19.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a54ba640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:19.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a54ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:19.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a54ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:19.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a54ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:19.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a54ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:19.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a64bc640 1 --2- 172.21.15.12:0/722464994 >> v2:172.21.15.71:3300/0 conn(0x7f08a013d000 0x7f08a013f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:19.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a64bc640 10 monclient(hunting): get_auth_request con 0x7f08a013d000 auth_method 0 2024-09-11T01:56:19.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a64bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:19.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a64bc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:19.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a64bc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:19.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a54ba640 10 monclient(hunting): handle_auth_done global_id 304130 payload 274 2024-09-11T01:56:19.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a54ba640 1 -- 172.21.15.12:0/722464994 >> v2:172.21.15.71:3300/0 conn(0x7f08a013d000 msgr2=0x7f08a013f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:19.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a54ba640 1 --2- 172.21.15.12:0/722464994 >> v2:172.21.15.71:3300/0 conn(0x7f08a013d000 0x7f08a013f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a54ba640 1 -- 172.21.15.12:0/722464994 >> v1:172.21.15.71:6789/0 conn(0x7f08a01381c0 legacy=0x7f08a01385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:19.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a54ba640 10 monclient: _finish_hunting 0 2024-09-11T01:56:19.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a54ba640 1 monclient: found mon.noname-b 2024-09-11T01:56:19.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a54ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:19.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a54ba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:56:19.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.141+0000 7f08a54ba640 1 -- 172.21.15.12:0/722464994 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a009c8d0 con 0x7f08a00a6aa0 2024-09-11T01:56:19.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.144+0000 7f08a54ba640 1 -- 172.21.15.12:0/722464994 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f08a00a6aa0 msgr2=0x7f08a009cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:19.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.144+0000 7f08a54ba640 1 --2- 172.21.15.12:0/722464994 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f08a00a6aa0 0x7f08a009cfd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0894009870 tx=0x7f089402a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8136048a7df27f9e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:19.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.144+0000 7f08a4cb9640 1 -- 172.21.15.12:0/722464994 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0894037070 con 0x7f08a00a6aa0 2024-09-11T01:56:19.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.144+0000 7f08a4cb9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:19.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.144+0000 7f08a4cb9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:56:19.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.144+0000 7f08a4cb9640 10 monclient: dump: 2024-09-11T01:56:19.154 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:19.154 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:19.154 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.154 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.154 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:19.154 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:19.154 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:19.155 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:19.155 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:19.155 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:19.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.144+0000 7f08a4cb9640 10 monclient: _finish_auth 0 2024-09-11T01:56:19.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.144+0000 7f08a4cb9640 10 monclient: _check_auth_tickets 2024-09-11T01:56:19.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.144+0000 7f08a4cb9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:19.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.144+0000 7f08a4cb9640 1 -- 172.21.15.12:0/722464994 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0894032640 con 0x7f08a00a6aa0 2024-09-11T01:56:19.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.144+0000 7f08a7f1f640 5 monclient: authenticate success, global_id 304130 2024-09-11T01:56:19.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.144+0000 7f08a7f1f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:56:19.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.144+0000 7f08a4cb9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:19.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.144+0000 7f08a7f1f640 10 monclient: get_monmap_and_config success 2024-09-11T01:56:19.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.144+0000 7f08a4cb9640 1 -- 172.21.15.12:0/722464994 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0894032a10 con 0x7f08a00a6aa0 2024-09-11T01:56:19.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.144+0000 7f08a4cb9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:19.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.144+0000 7f08a7f1f640 4 set_mon_vals no callback set 2024-09-11T01:56:19.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.144+0000 7f08a4cb9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:56:19.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.144+0000 7f08a4cb9640 10 monclient: dump: 2024-09-11T01:56:19.156 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:19.156 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:19.156 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.156 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.156 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:19.156 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:19.156 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:19.156 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:19.156 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:19.157 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:19.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.145+0000 7f08a7f1f640 10 monclient: shutdown 2024-09-11T01:56:19.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.145+0000 7f08a7f1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:19.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.145+0000 7f08a7f1f640 1 -- 172.21.15.12:0/722464994 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f08a00a6aa0 msgr2=0x7f08a009cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:19.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.145+0000 7f08a7f1f640 1 --2- 172.21.15.12:0/722464994 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f08a00a6aa0 0x7f08a009cfd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0894009870 tx=0x7f089402a130 comp rx=0 tx=0).stop 2024-09-11T01:56:19.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.145+0000 7f08a7f1f640 1 -- 172.21.15.12:0/722464994 shutdown_connections 2024-09-11T01:56:19.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.145+0000 7f08a7f1f640 1 --2- 172.21.15.12:0/722464994 >> v2:172.21.15.71:3300/0 conn(0x7f08a013d000 0x7f08a013f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.145+0000 7f08a7f1f640 1 --2- 172.21.15.12:0/722464994 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f08a00a6aa0 0x7f08a009cfd0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.145+0000 7f08a7f1f640 1 -- 172.21.15.12:0/722464994 >> 172.21.15.12:0/722464994 conn(0x7f08a00a7440 msgr2=0x7f08a00a7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:19.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.145+0000 7f08a7f1f640 1 -- 172.21.15.12:0/722464994 shutdown_connections 2024-09-11T01:56:19.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.145+0000 7f08a7f1f640 1 -- 172.21.15.12:0/722464994 wait complete. 2024-09-11T01:56:19.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.145+0000 7f08a7f1f640 10 monclient: build_initial_monmap 2024-09-11T01:56:19.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.145+0000 7f08a7f1f640 10 monclient: monmap: 2024-09-11T01:56:19.158 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:19.158 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:56:19.158 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:56:19.158 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:56:19.158 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:19.158 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:19.158 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:56:19.158 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:56:19.158 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:56:19.158 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:19.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a7f1f640 1 Processor -- start 2024-09-11T01:56:19.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a7f1f640 1 -- start start 2024-09-11T01:56:19.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a7f1f640 10 monclient: init 2024-09-11T01:56:19.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a7f1f640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:19.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a7f1f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:56:19.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a7f1f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08a00a6aa0 0x7f08a0139810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:19.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a7f1f640 10 monclient(hunting): picked mon.noname-a con 0x7f08a00a6aa0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:56:19.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a7f1f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f08a01381c0 0x7f08a013b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:19.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a7f1f640 10 monclient(hunting): picked mon.noname-c con 0x7f08a01381c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:19.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a7f1f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f08a013d000 0x7f08a0136340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:19.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a7f1f640 10 monclient(hunting): picked mon.noname-b con 0x7f08a013d000 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:56:19.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a7f1f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:19.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a7f1f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f08a013fab0 con 0x7f08a00a6aa0 2024-09-11T01:56:19.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a7f1f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:19.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a7f1f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f08a013f930 con 0x7f08a013d000 2024-09-11T01:56:19.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a7f1f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:19.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a7f1f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f08a013fc30 con 0x7f08a01381c0 2024-09-11T01:56:19.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a7f1f640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:19.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a7f1f640 10 monclient(hunting): authenticate will time out at 2385.588806s 2024-09-11T01:56:19.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a54ba640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f08a01381c0 0x7f08a013b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:19.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a5cbb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08a00a6aa0 0x7f08a0139810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:19.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a54ba640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f08a01381c0 0x7f08a013b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46772/0 (socket says 172.21.15.12:46772) 2024-09-11T01:56:19.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a54ba640 1 -- 172.21.15.12:0/2216999456 learned_addr learned my addr 172.21.15.12:0/2216999456 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:19.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a54ba640 10 monclient(hunting): get_auth_request con 0x7f08a01381c0 auth_method 0 2024-09-11T01:56:19.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a54ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:19.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a54ba640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:19.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a54ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:19.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a5cbb640 10 monclient(hunting): get_auth_request con 0x7f08a00a6aa0 auth_method 0 2024-09-11T01:56:19.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a5cbb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:19.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a5cbb640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:19.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a5cbb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:19.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a54ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:19.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.146+0000 7f08a54ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:19.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.147+0000 7f08a54ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:19.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.147+0000 7f08a5cbb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:19.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.147+0000 7f08a5cbb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:19.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.147+0000 7f08a5cbb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:19.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.147+0000 7f08a64bc640 1 --2- 172.21.15.12:0/2216999456 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f08a013d000 0x7f08a0136340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:19.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.147+0000 7f08a64bc640 10 monclient(hunting): get_auth_request con 0x7f08a013d000 auth_method 0 2024-09-11T01:56:19.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.147+0000 7f08a64bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:19.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.147+0000 7f08a64bc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:19.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.147+0000 7f08a64bc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:19.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.147+0000 7f08a54ba640 10 monclient(hunting): handle_auth_done global_id 304133 payload 824 2024-09-11T01:56:19.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.147+0000 7f08a54ba640 1 -- 172.21.15.12:0/2216999456 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f08a013d000 msgr2=0x7f08a0136340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:19.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.147+0000 7f08a54ba640 1 --2- 172.21.15.12:0/2216999456 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f08a013d000 0x7f08a0136340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.147+0000 7f08a54ba640 1 -- 172.21.15.12:0/2216999456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08a00a6aa0 msgr2=0x7f08a0139810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:19.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.147+0000 7f08a54ba640 1 --2- 172.21.15.12:0/2216999456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08a00a6aa0 0x7f08a0139810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.147+0000 7f08a54ba640 10 monclient: _finish_hunting 0 2024-09-11T01:56:19.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.147+0000 7f08a54ba640 1 monclient: found mon.noname-c 2024-09-11T01:56:19.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.147+0000 7f08a54ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:19.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.147+0000 7f08a54ba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:56:19.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.147+0000 7f08a54ba640 1 -- 172.21.15.12:0/2216999456 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a0120d10 con 0x7f08a01381c0 2024-09-11T01:56:19.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.147+0000 7f08a5cbb640 1 --2- 172.21.15.12:0/2216999456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08a00a6aa0 0x7f08a0139810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:56:19.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.149+0000 7f08a54ba640 1 -- 172.21.15.12:0/2216999456 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f08a01381c0 msgr2=0x7f08a013b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:19.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.149+0000 7f08a54ba640 1 --2- 172.21.15.12:0/2216999456 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f08a01381c0 0x7f08a013b8e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f08940090b0 tx=0x7f08940046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:19.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.149+0000 7f089e7fc640 1 -- 172.21.15.12:0/2216999456 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0894041020 con 0x7f08a01381c0 2024-09-11T01:56:19.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.149+0000 7f089e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:19.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.149+0000 7f089e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:56:19.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.149+0000 7f089e7fc640 10 monclient: dump: 2024-09-11T01:56:19.168 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:19.168 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:19.168 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.168 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.168 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:19.169 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:19.169 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:19.169 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:19.169 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:19.169 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:19.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.149+0000 7f089e7fc640 10 monclient: _finish_auth 0 2024-09-11T01:56:19.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.149+0000 7f089e7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:56:19.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.149+0000 7f089e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:19.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.149+0000 7f089e7fc640 1 -- 172.21.15.12:0/2216999456 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f089403d830 con 0x7f08a01381c0 2024-09-11T01:56:19.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.149+0000 7f089e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:19.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.149+0000 7f089e7fc640 1 -- 172.21.15.12:0/2216999456 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0894037040 con 0x7f08a01381c0 2024-09-11T01:56:19.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.149+0000 7f089e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:19.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.149+0000 7f089e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:56:19.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.149+0000 7f089e7fc640 10 monclient: dump: 2024-09-11T01:56:19.170 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:19.170 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:19.170 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.170 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.170 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:19.170 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:19.170 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:19.170 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:19.170 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:19.170 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:19.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.149+0000 7f08a7f1f640 5 monclient: authenticate success, global_id 304133 2024-09-11T01:56:19.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.149+0000 7f08a7f1f640 10 monclient: _renew_subs 2024-09-11T01:56:19.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.149+0000 7f08a7f1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:56:19.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.149+0000 7f08a7f1f640 1 -- 172.21.15.12:0/2216999456 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08a013f400 con 0x7f08a01381c0 2024-09-11T01:56:19.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.149+0000 7f089f7fe640 4 set_mon_vals no callback set 2024-09-11T01:56:19.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.149+0000 7f08a7f1f640 10 monclient: _renew_subs 2024-09-11T01:56:19.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.149+0000 7f08a7f1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:56:19.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.149+0000 7f08a7f1f640 1 -- 172.21.15.12:0/2216999456 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f08a013cb20 con 0x7f08a01381c0 2024-09-11T01:56:19.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.150+0000 7f08a7f1f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:19.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.150+0000 7f08a7f1f640 1 --2- 172.21.15.12:0/2216999456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08a0136880 0x7f08a01cd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:19.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.150+0000 7f08a7f1f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:19.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.150+0000 7f08a7f1f640 1 -- 172.21.15.12:0/2216999456 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f08a0137040 con 0x7f08a0136880 2024-09-11T01:56:19.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.150+0000 7f08a5cbb640 1 --2- 172.21.15.12:0/2216999456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08a0136880 0x7f08a01cd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:19.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.150+0000 7f08a5cbb640 10 monclient: get_auth_request con 0x7f08a0136880 auth_method 0 2024-09-11T01:56:19.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.150+0000 7f08a5cbb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:19.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.150+0000 7f08a5cbb640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:19.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.150+0000 7f08a5cbb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:19.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.150+0000 7f089e7fc640 1 -- 172.21.15.12:0/2216999456 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f089403a030 con 0x7f08a01381c0 2024-09-11T01:56:19.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.150+0000 7f08a5cbb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:19.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.150+0000 7f08a5cbb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:19.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.150+0000 7f08a5cbb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:19.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.150+0000 7f08a5cbb640 10 monclient(hunting): handle_auth_done global_id 294228 payload 824 2024-09-11T01:56:19.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.150+0000 7f08a5cbb640 1 -- 172.21.15.12:0/2216999456 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f08a009cfd0 con 0x7f08a0136880 2024-09-11T01:56:19.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.150+0000 7f089e7fc640 1 --2- 172.21.15.12:0/2216999456 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f088c03e210 0x7f088c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:19.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.151+0000 7f089e7fc640 1 -- 172.21.15.12:0/2216999456 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(919..919 src has 676..919) ==== 4771+0+0 (secure 0 0 0) 0x7f0894073290 con 0x7f08a01381c0 2024-09-11T01:56:19.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.151+0000 7f08a64bc640 1 --2- 172.21.15.12:0/2216999456 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f088c03e210 0x7f088c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:19.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.151+0000 7f08a64bc640 10 monclient: get_auth_request con 0x7f088c03e210 auth_method 0 2024-09-11T01:56:19.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.153+0000 7f08a5cbb640 1 -- 172.21.15.12:0/2216999456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08a0136880 msgr2=0x7f08a01cd0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:19.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.153+0000 7f08a5cbb640 1 --2- 172.21.15.12:0/2216999456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08a0136880 0x7f08a01cd0f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f08900048c0 tx=0x7f089000d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:19.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.153+0000 7f089e7fc640 1 -- 172.21.15.12:0/2216999456 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0890004950 con 0x7f08a0136880 2024-09-11T01:56:19.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.153+0000 7f089e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:19.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.153+0000 7f089e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:56:19.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.153+0000 7f089e7fc640 10 monclient: dump: 2024-09-11T01:56:19.173 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:19.173 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:19.173 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.173 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.173 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:19.173 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:19.173 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:19.173 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:19.173 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:19.174 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:19.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.153+0000 7f089e7fc640 1 -- 172.21.15.12:0/2216999456 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0890004950 con 0x7f08a0136880 2024-09-11T01:56:19.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.153+0000 7f089e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:19.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.153+0000 7f089e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:56:19.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.153+0000 7f089e7fc640 1 -- 172.21.15.12:0/2216999456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08a0136880 msgr2=0x7f08a01cd0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:19.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.153+0000 7f089e7fc640 1 --2- 172.21.15.12:0/2216999456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08a0136880 0x7f08a01cd0f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f08900048c0 tx=0x7f089000d520 comp rx=0 tx=0).stop 2024-09-11T01:56:19.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.153+0000 7f089e7fc640 1 -- 172.21.15.12:0/2216999456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08a0136880 msgr2=0x7f08a01cd0f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:56:19.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.153+0000 7f089e7fc640 1 --2- 172.21.15.12:0/2216999456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08a0136880 0x7f08a01cd0f0 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f08900048c0 tx=0x7f089000d520 comp rx=0 tx=0).stop 2024-09-11T01:56:19.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.153+0000 7f08a64bc640 1 --2- 172.21.15.12:0/2216999456 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f088c03e210 0x7f088c0406d0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f08a013ce40 tx=0x7f0898008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:19.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.159+0000 7f08a7f1f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:56:19.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.159+0000 7f08a7f1f640 1 --2- 172.21.15.12:0/2216999456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08a01cd630 0x7f08a01cda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:19.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.159+0000 7f08a7f1f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:19.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.159+0000 7f08a7f1f640 1 -- 172.21.15.12:0/2216999456 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f08a01cdf80 con 0x7f08a01cd630 2024-09-11T01:56:19.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.159+0000 7f08a5cbb640 1 --2- 172.21.15.12:0/2216999456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08a01cd630 0x7f08a01cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:19.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.160+0000 7f08a5cbb640 10 monclient: get_auth_request con 0x7f08a01cd630 auth_method 0 2024-09-11T01:56:19.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.160+0000 7f08a5cbb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:19.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.160+0000 7f08a5cbb640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:19.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.160+0000 7f08a5cbb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:19.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.160+0000 7f08a5cbb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:19.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.160+0000 7f08a5cbb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:19.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.160+0000 7f08a5cbb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:19.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.160+0000 7f08a5cbb640 10 monclient(hunting): handle_auth_done global_id 294231 payload 824 2024-09-11T01:56:19.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.160+0000 7f08a5cbb640 1 -- 172.21.15.12:0/2216999456 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f08a0069840 con 0x7f08a01cd630 2024-09-11T01:56:19.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.162+0000 7f08a5cbb640 1 -- 172.21.15.12:0/2216999456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08a01cd630 msgr2=0x7f08a01cda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:19.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.162+0000 7f08a5cbb640 1 --2- 172.21.15.12:0/2216999456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08a01cd630 0x7f08a01cda40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0890018000 tx=0x7f0890003ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:19.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.162+0000 7f089e7fc640 1 -- 172.21.15.12:0/2216999456 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089001e030 con 0x7f08a01cd630 2024-09-11T01:56:19.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.162+0000 7f089e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:19.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.162+0000 7f089e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:56:19.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.162+0000 7f089e7fc640 10 monclient: dump: 2024-09-11T01:56:19.176 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:19.176 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:19.176 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.176 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.176 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:19.176 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:19.176 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:19.176 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:19.176 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:19.176 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:19.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.162+0000 7f089e7fc640 1 -- 172.21.15.12:0/2216999456 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4710 (secure 0 0 0) 0x7f0890006b50 con 0x7f08a01cd630 2024-09-11T01:56:19.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.162+0000 7f089e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:56:19.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.162+0000 7f089e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:56:19.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.162+0000 7f089e7fc640 1 -- 172.21.15.12:0/2216999456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08a01cd630 msgr2=0x7f08a01cda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:19.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.162+0000 7f089e7fc640 1 --2- 172.21.15.12:0/2216999456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08a01cd630 0x7f08a01cda40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0890018000 tx=0x7f0890003ae0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.162+0000 7f089e7fc640 1 -- 172.21.15.12:0/2216999456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08a01cd630 msgr2=0x7f08a01cda40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:56:19.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.162+0000 7f089e7fc640 1 --2- 172.21.15.12:0/2216999456 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08a01cd630 0x7f08a01cda40 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0890018000 tx=0x7f0890003ae0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.177 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:56:19.177 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:56:19.178 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:56:19.178 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:56:19.178 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 250, 2024-09-11T01:56:19.178 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 59115, 2024-09-11T01:56:19.178 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:56:19.178 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:56:19.178 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:56:19.178 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:56:19.178 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:19.178 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 5, 2024-09-11T01:56:19.178 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:56:19.178 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:56:19.178 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:56:19.179 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:56:19.179 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:56:19.179 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:56:19.179 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:56:19.179 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:56:19.179 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:56:19.179 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:56:19.179 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:56:19.179 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:56:19.179 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:56:19.179 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:56:19.179 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:19.179 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:56:19.180 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:56:19.180 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:56:19.180 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:56:19.180 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:56:19.180 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:56:19.180 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:56:19.180 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:56:19.180 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:56:19.180 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:56:19.180 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:56:19.180 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:56:19.180 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:56:19.180 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:19.180 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.181 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:56:19.181 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:56:19.181 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:56:19.181 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:56:19.181 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:56:19.181 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:56:19.181 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:56:19.181 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:56:19.181 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:56:19.181 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:56:19.181 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:56:19.181 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:56:19.181 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:56:19.182 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:56:19.182 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:56:19.182 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:56:19.182 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:56:19.182 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:56:19.182 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:56:19.182 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:56:19.182 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:56:19.182 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:56:19.182 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:56:19.182 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:56:19.182 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:56:19.182 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:56:19.182 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:56:19.183 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:56:19.183 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:19.183 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:56:19.183 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.183 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:56:19.183 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.183 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:56:19.183 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:56:19.183 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:56:19.183 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:56:19.183 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.183 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:56:19.183 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:56:19.184 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:19.184 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.184 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.184 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:56:19.184 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:56:19.184 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:19.184 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:19.184 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:19.184 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.184 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:56:19.184 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:56:19.184 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:56:19.184 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:56:19.185 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:56:19.185 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.185 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.185 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:56:19.185 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:56:19.185 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:56:19.185 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:56:19.185 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.185 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:56:19.185 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:56:19.185 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:19.185 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.185 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.185 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:56:19.186 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:56:19.186 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:19.186 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:19.186 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:19.186 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.186 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:56:19.186 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:56:19.186 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:56:19.186 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:56:19.186 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:56:19.186 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.186 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.186 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:56:19.186 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:56:19.187 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:56:19.187 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:56:19.187 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.187 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:56:19.187 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:56:19.187 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:19.187 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.187 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.187 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:56:19.187 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:56:19.187 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:19.187 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:19.187 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:19.188 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.188 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:56:19.188 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:56:19.188 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:56:19.188 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:56:19.188 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:56:19.188 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:19.188 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:19.188 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.188 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:56:19.188 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:56:19.189 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.189 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:56:19.189 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:56:19.189 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:56:19.189 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:19.189 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:19.189 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:56:19.189 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.189 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:56:19.189 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:56:19.189 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:56:19.189 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:19.190 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:19.190 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:56:19.190 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.190 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:56:19.190 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:56:19.190 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:56:19.190 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:19.190 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:19.190 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.190 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:56:19.190 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:56:19.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.163+0000 7f08a7f1f640 1 -- 172.21.15.12:0/2216999456 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f088c03e210 msgr2=0x7f088c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:19.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.163+0000 7f08a7f1f640 1 --2- 172.21.15.12:0/2216999456 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f088c03e210 0x7f088c0406d0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f08a013ce40 tx=0x7f0898008040 comp rx=0 tx=0).stop 2024-09-11T01:56:19.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.163+0000 7f08a7f1f640 10 monclient: shutdown 2024-09-11T01:56:19.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.163+0000 7f08a7f1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:19.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.163+0000 7f08a7f1f640 1 -- 172.21.15.12:0/2216999456 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f08a01381c0 msgr2=0x7f08a013b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:19.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.163+0000 7f08a7f1f640 1 --2- 172.21.15.12:0/2216999456 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f08a01381c0 0x7f08a013b8e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f08940090b0 tx=0x7f08940046c0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.163+0000 7f08a64bc640 1 -- 172.21.15.12:0/2216999456 reap_dead start 2024-09-11T01:56:19.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.163+0000 7f08a64bc640 1 -- 172.21.15.12:0/2216999456 reap_dead start 2024-09-11T01:56:19.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.163+0000 7f08a7f1f640 1 -- 172.21.15.12:0/2216999456 shutdown_connections 2024-09-11T01:56:19.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.163+0000 7f08a7f1f640 1 -- 172.21.15.12:0/2216999456 >> 172.21.15.12:0/2216999456 conn(0x7f08a00a7440 msgr2=0x7f08a013d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:19.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.163+0000 7f08a7f1f640 1 -- 172.21.15.12:0/2216999456 shutdown_connections 2024-09-11T01:56:19.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.163+0000 7f08a7f1f640 1 -- 172.21.15.12:0/2216999456 wait complete. 2024-09-11T01:56:19.192 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:56:19.193 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:56:19.193 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:56:19.194 INFO:tasks.rados.rados.0.smithi071.stdout:3994: done (1 left) 2024-09-11T01:56:19.194 INFO:tasks.rados.rados.0.smithi071.stdout:3995: read oid 50 snap 436 2024-09-11T01:56:19.194 INFO:tasks.rados.rados.0.smithi071.stdout:3995: expect deleted 2024-09-11T01:56:19.194 INFO:tasks.rados.rados.0.smithi071.stdout:3996: snap_create 2024-09-11T01:56:19.194 INFO:tasks.rados.rados.0.smithi071.stdout:3993: expect (ObjNum 1301 snap 426 seq_num 1301) 2024-09-11T01:56:19.194 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:56:19.245 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:56:19.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.295+0000 7f218ac71640 10 monclient: get_monmap_and_config 2024-09-11T01:56:19.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.295+0000 7f218ac71640 10 monclient: build_initial_monmap 2024-09-11T01:56:19.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.295+0000 7f218ac71640 10 monclient: monmap: 2024-09-11T01:56:19.297 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:19.297 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:19.297 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:56:19.296599+0000 2024-09-11T01:56:19.298 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:56:19.296599+0000 2024-09-11T01:56:19.298 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:19.298 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:19.298 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:56:19.298 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:56:19.298 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:56:19.298 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:56:19.298 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:56:19.298 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:19.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.296+0000 7f218ac71640 1 Processor -- start 2024-09-11T01:56:19.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.296+0000 7f218ac71640 1 -- start start 2024-09-11T01:56:19.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.296+0000 7f218ac71640 10 monclient: init 2024-09-11T01:56:19.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f218ac71640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:19.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f218ac71640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-11T01:56:19.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f218ac71640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184137e50 0x7f218409d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:19.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f218ac71640 10 monclient(hunting): picked mon.noname-b con 0x7f2184137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:19.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f218ac71640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2184138230 0x7f2184138630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:19.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f218ac71640 10 monclient(hunting): picked mon.noname-a con 0x7f2184138230 addr v2:172.21.15.12:3300/0 2024-09-11T01:56:19.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f218ac71640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2184134cb0 addr v1:172.21.15.71:6789/0 2024-09-11T01:56:19.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f218ac71640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21840a62d0 con 0x7f2184134cb0 2024-09-11T01:56:19.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f218ac71640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:19.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f218ac71640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f21840a65b0 con 0x7f2184138230 2024-09-11T01:56:19.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f218ac71640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:19.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f218ac71640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f218409bb90 con 0x7f2184137e50 2024-09-11T01:56:19.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f218ac71640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:19.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f218ac71640 10 monclient(hunting): authenticate will time out at 2385.739174s 2024-09-11T01:56:19.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2183fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2184138230 0x7f2184138630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:19.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2188a0d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184137e50 0x7f218409d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:19.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2183fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2184138230 0x7f2184138630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49800/0 (socket says 172.21.15.12:49800) 2024-09-11T01:56:19.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2188a0d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184137e50 0x7f218409d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46788/0 (socket says 172.21.15.12:46788) 2024-09-11T01:56:19.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2183fff640 1 -- 172.21.15.12:0/1322056271 learned_addr learned my addr 172.21.15.12:0/1322056271 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:19.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2183fff640 10 monclient(hunting): get_auth_request con 0x7f2184138230 auth_method 0 2024-09-11T01:56:19.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2183fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:19.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2183fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:19.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2183fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:19.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2188a0d640 10 monclient(hunting): get_auth_request con 0x7f2184137e50 auth_method 0 2024-09-11T01:56:19.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2188a0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:19.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2188a0d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:19.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2188a0d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:19.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2183fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:19.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2183fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:19.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2183fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:19.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2188a0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:19.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2188a0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:19.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2188a0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:19.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2183fff640 10 monclient(hunting): handle_auth_done global_id 294234 payload 274 2024-09-11T01:56:19.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2183fff640 1 -- 172.21.15.12:0/1322056271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184137e50 msgr2=0x7f218409d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:19.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2183fff640 1 --2- 172.21.15.12:0/1322056271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184137e50 0x7f218409d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2183fff640 1 -- 172.21.15.12:0/1322056271 >> v1:172.21.15.71:6789/0 conn(0x7f2184134cb0 legacy=0x7f2184135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:19.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2183fff640 10 monclient: _finish_hunting 0 2024-09-11T01:56:19.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2183fff640 1 monclient: found mon.noname-a 2024-09-11T01:56:19.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2183fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:19.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2183fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:56:19.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2183fff640 1 -- 172.21.15.12:0/1322056271 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f218409c4f0 con 0x7f2184138230 2024-09-11T01:56:19.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.297+0000 7f2188a0d640 1 --2- 172.21.15.12:0/1322056271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184137e50 0x7f218409d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:56:19.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.300+0000 7f2183fff640 1 -- 172.21.15.12:0/1322056271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2184138230 msgr2=0x7f2184138630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:19.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.300+0000 7f2183fff640 1 --2- 172.21.15.12:0/1322056271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2184138230 0x7f2184138630 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2178009870 tx=0x7f217802a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed847b9b73901338 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:19.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.300+0000 7f21837fe640 1 -- 172.21.15.12:0/1322056271 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2178037070 con 0x7f2184138230 2024-09-11T01:56:19.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.300+0000 7f21837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:19.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.300+0000 7f21837fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:56:19.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.300+0000 7f21837fe640 10 monclient: dump: 2024-09-11T01:56:19.307 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:19.307 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:19.307 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.307 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.307 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:19.307 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:19.307 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:19.307 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:19.307 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:19.308 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:19.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.300+0000 7f21837fe640 10 monclient: _finish_auth 0 2024-09-11T01:56:19.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.300+0000 7f21837fe640 10 monclient: _check_auth_tickets 2024-09-11T01:56:19.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.300+0000 7f21837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:19.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.300+0000 7f21837fe640 1 -- 172.21.15.12:0/1322056271 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2178032440 con 0x7f2184138230 2024-09-11T01:56:19.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.300+0000 7f218ac71640 5 monclient: authenticate success, global_id 294234 2024-09-11T01:56:19.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.300+0000 7f218ac71640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:56:19.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.300+0000 7f21837fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:19.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.300+0000 7f218ac71640 10 monclient: get_monmap_and_config success 2024-09-11T01:56:19.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.300+0000 7f21837fe640 1 -- 172.21.15.12:0/1322056271 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2178032810 con 0x7f2184138230 2024-09-11T01:56:19.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.300+0000 7f21837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:19.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.300+0000 7f218ac71640 4 set_mon_vals no callback set 2024-09-11T01:56:19.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.300+0000 7f21837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:56:19.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.300+0000 7f21837fe640 10 monclient: dump: 2024-09-11T01:56:19.309 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:19.309 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:19.309 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.309 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.309 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:19.309 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:19.309 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:19.309 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:19.309 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:19.309 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:19.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.301+0000 7f218ac71640 10 monclient: shutdown 2024-09-11T01:56:19.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.301+0000 7f218ac71640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:19.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.301+0000 7f218ac71640 1 -- 172.21.15.12:0/1322056271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2184138230 msgr2=0x7f2184138630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:19.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.301+0000 7f218ac71640 1 --2- 172.21.15.12:0/1322056271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2184138230 0x7f2184138630 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2178009870 tx=0x7f217802a0a0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.301+0000 7f218ac71640 1 -- 172.21.15.12:0/1322056271 shutdown_connections 2024-09-11T01:56:19.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.301+0000 7f218ac71640 1 --2- 172.21.15.12:0/1322056271 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2184138230 0x7f2184138630 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.301+0000 7f218ac71640 1 --2- 172.21.15.12:0/1322056271 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184137e50 0x7f218409d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.301+0000 7f218ac71640 1 -- 172.21.15.12:0/1322056271 >> 172.21.15.12:0/1322056271 conn(0x7f21840a7020 msgr2=0x7f21840a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:19.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.301+0000 7f218ac71640 1 -- 172.21.15.12:0/1322056271 shutdown_connections 2024-09-11T01:56:19.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.301+0000 7f218ac71640 1 -- 172.21.15.12:0/1322056271 wait complete. 2024-09-11T01:56:19.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.301+0000 7f218ac71640 10 monclient: build_initial_monmap 2024-09-11T01:56:19.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.301+0000 7f218ac71640 10 monclient: monmap: 2024-09-11T01:56:19.310 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:19.310 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:56:19.310 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:56:19.311 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:56:19.311 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:19.311 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:19.311 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:56:19.311 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:56:19.311 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:56:19.311 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:19.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218ac71640 1 Processor -- start 2024-09-11T01:56:19.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218ac71640 1 -- start start 2024-09-11T01:56:19.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218ac71640 10 monclient: init 2024-09-11T01:56:19.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218ac71640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:19.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218ac71640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:56:19.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218ac71640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184134cb0 0x7f21841cd750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:19.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218ac71640 10 monclient(hunting): picked mon.noname-c con 0x7f2184134cb0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:19.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218ac71640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2184137e50 0x7f2184134020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:19.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218ac71640 10 monclient(hunting): picked mon.noname-b con 0x7f2184137e50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:56:19.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218ac71640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2184138230 0x7f2184134560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:19.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218ac71640 10 monclient(hunting): picked mon.noname-a con 0x7f2184138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:56:19.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218ac71640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:19.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218ac71640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f218406f710 con 0x7f2184138230 2024-09-11T01:56:19.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218ac71640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:19.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218ac71640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f2184138b70 con 0x7f2184137e50 2024-09-11T01:56:19.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218ac71640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:19.317 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218ac71640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f218406f890 con 0x7f2184134cb0 2024-09-11T01:56:19.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218ac71640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:19.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218ac71640 10 monclient(hunting): authenticate will time out at 2385.744760s 2024-09-11T01:56:19.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f2188a0d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184134cb0 0x7f21841cd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:19.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218920e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2184138230 0x7f2184134560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:19.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f2188a0d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184134cb0 0x7f21841cd750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46792/0 (socket says 172.21.15.12:46792) 2024-09-11T01:56:19.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218920e640 10 monclient(hunting): get_auth_request con 0x7f2184138230 auth_method 0 2024-09-11T01:56:19.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f2188a0d640 1 -- 172.21.15.12:0/2212863051 learned_addr learned my addr 172.21.15.12:0/2212863051 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:19.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218920e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:19.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218920e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:19.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218920e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:19.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f2188a0d640 10 monclient(hunting): get_auth_request con 0x7f2184134cb0 auth_method 0 2024-09-11T01:56:19.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f2188a0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:19.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f2188a0d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f2188a0d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f2183fff640 1 --2- 172.21.15.12:0/2212863051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2184137e50 0x7f2184134020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218920e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218920e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.302+0000 7f218920e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.303+0000 7f2183fff640 10 monclient(hunting): get_auth_request con 0x7f2184137e50 auth_method 0 2024-09-11T01:56:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.303+0000 7f2183fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.303+0000 7f2183fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.303+0000 7f2183fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.303+0000 7f2188a0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.303+0000 7f2188a0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:19.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.303+0000 7f2188a0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.303+0000 7f2183fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.303+0000 7f2183fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.303+0000 7f2183fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.303+0000 7f218920e640 10 monclient(hunting): handle_auth_done global_id 294237 payload 824 2024-09-11T01:56:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.303+0000 7f218920e640 1 -- 172.21.15.12:0/2212863051 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184134cb0 msgr2=0x7f21841cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.303+0000 7f218920e640 1 --2- 172.21.15.12:0/2212863051 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184134cb0 0x7f21841cd750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.303+0000 7f218920e640 1 -- 172.21.15.12:0/2212863051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2184137e50 msgr2=0x7f2184134020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.303+0000 7f218920e640 1 --2- 172.21.15.12:0/2212863051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2184137e50 0x7f2184134020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.303+0000 7f218920e640 10 monclient: _finish_hunting 0 2024-09-11T01:56:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.303+0000 7f218920e640 1 monclient: found mon.noname-a 2024-09-11T01:56:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.303+0000 7f218920e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.303+0000 7f218920e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:56:19.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.303+0000 7f218920e640 1 -- 172.21.15.12:0/2212863051 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2184125740 con 0x7f2184138230 2024-09-11T01:56:19.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.305+0000 7f218920e640 1 -- 172.21.15.12:0/2212863051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2184138230 msgr2=0x7f2184134560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:19.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.305+0000 7f218920e640 1 --2- 172.21.15.12:0/2212863051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2184138230 0x7f2184134560 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f217c00c780 tx=0x7f217c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:19.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.305+0000 7f21817fa640 1 -- 172.21.15.12:0/2212863051 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217c016020 con 0x7f2184138230 2024-09-11T01:56:19.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.305+0000 7f21817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:19.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.305+0000 7f21817fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:56:19.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.305+0000 7f21817fa640 10 monclient: dump: 2024-09-11T01:56:19.321 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:19.321 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:19.321 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.321 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.321 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:19.321 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:19.321 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:19.322 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:19.322 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:19.322 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:19.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.305+0000 7f21817fa640 10 monclient: _finish_auth 0 2024-09-11T01:56:19.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.305+0000 7f21817fa640 10 monclient: _check_auth_tickets 2024-09-11T01:56:19.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.305+0000 7f21817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:19.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.305+0000 7f21817fa640 1 -- 172.21.15.12:0/2212863051 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f217c00ac60 con 0x7f2184138230 2024-09-11T01:56:19.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.305+0000 7f218ac71640 5 monclient: authenticate success, global_id 294237 2024-09-11T01:56:19.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.305+0000 7f218ac71640 10 monclient: _renew_subs 2024-09-11T01:56:19.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.305+0000 7f218ac71640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:19.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.305+0000 7f218ac71640 1 -- 172.21.15.12:0/2212863051 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2184124220 con 0x7f2184138230 2024-09-11T01:56:19.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.305+0000 7f21817fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:19.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.305+0000 7f21817fa640 1 -- 172.21.15.12:0/2212863051 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217c012660 con 0x7f2184138230 2024-09-11T01:56:19.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.305+0000 7f21817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:19.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.305+0000 7f21817fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:56:19.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.305+0000 7f21817fa640 10 monclient: dump: 2024-09-11T01:56:19.323 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:19.323 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:19.323 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.323 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.323 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:19.323 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:19.323 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:19.323 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:19.324 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:19.324 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:19.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.305+0000 7f21827fc640 4 set_mon_vals no callback set 2024-09-11T01:56:19.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.305+0000 7f218ac71640 10 monclient: _renew_subs 2024-09-11T01:56:19.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.305+0000 7f218ac71640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:19.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.305+0000 7f218ac71640 1 -- 172.21.15.12:0/2212863051 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2184124800 con 0x7f2184138230 2024-09-11T01:56:19.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.306+0000 7f218ac71640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:19.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.306+0000 7f218ac71640 1 --2- 172.21.15.12:0/2212863051 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184125180 0x7f2184121c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:19.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.306+0000 7f218ac71640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:19.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.306+0000 7f218ac71640 1 -- 172.21.15.12:0/2212863051 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f218406a060 con 0x7f2184125180 2024-09-11T01:56:19.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.306+0000 7f2188a0d640 1 --2- 172.21.15.12:0/2212863051 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184125180 0x7f2184121c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:19.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.306+0000 7f2188a0d640 10 monclient: get_auth_request con 0x7f2184125180 auth_method 0 2024-09-11T01:56:19.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.306+0000 7f2188a0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:19.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.306+0000 7f2188a0d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:19.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.306+0000 7f2188a0d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:19.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.306+0000 7f2188a0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:19.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.306+0000 7f2188a0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:19.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.306+0000 7f2188a0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:19.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.306+0000 7f21817fa640 1 -- 172.21.15.12:0/2212863051 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f217c0128c0 con 0x7f2184138230 2024-09-11T01:56:19.325 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.306+0000 7f2188a0d640 10 monclient(hunting): handle_auth_done global_id 304142 payload 824 2024-09-11T01:56:19.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.306+0000 7f2188a0d640 1 -- 172.21.15.12:0/2212863051 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f21840a1600 con 0x7f2184125180 2024-09-11T01:56:19.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.307+0000 7f21817fa640 1 --2- 172.21.15.12:0/2212863051 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f217003e210 0x7f21700406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:19.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.307+0000 7f21817fa640 1 -- 172.21.15.12:0/2212863051 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(920..920 src has 676..920) ==== 4815+0+0 (secure 0 0 0) 0x7f217c052f40 con 0x7f2184138230 2024-09-11T01:56:19.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.307+0000 7f2183fff640 1 --2- 172.21.15.12:0/2212863051 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f217003e210 0x7f21700406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:19.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.307+0000 7f2183fff640 10 monclient: get_auth_request con 0x7f217003e210 auth_method 0 2024-09-11T01:56:19.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.309+0000 7f2188a0d640 1 -- 172.21.15.12:0/2212863051 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184125180 msgr2=0x7f2184121c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:19.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.309+0000 7f2188a0d640 1 --2- 172.21.15.12:0/2212863051 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184125180 0x7f2184121c00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2174011c00 tx=0x7f217400f610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:19.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.309+0000 7f21817fa640 1 -- 172.21.15.12:0/2212863051 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217400fe70 con 0x7f2184125180 2024-09-11T01:56:19.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.309+0000 7f21817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:19.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.309+0000 7f21817fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:56:19.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.309+0000 7f21817fa640 10 monclient: dump: 2024-09-11T01:56:19.327 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:19.327 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:19.327 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.327 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.327 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:19.327 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:19.327 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:19.327 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:19.327 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:19.327 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:19.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.309+0000 7f21817fa640 1 -- 172.21.15.12:0/2212863051 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f217400fe70 con 0x7f2184125180 2024-09-11T01:56:19.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.309+0000 7f21817fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:19.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.309+0000 7f21817fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:56:19.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.309+0000 7f21817fa640 1 -- 172.21.15.12:0/2212863051 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184125180 msgr2=0x7f2184121c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:19.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.309+0000 7f21817fa640 1 --2- 172.21.15.12:0/2212863051 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184125180 0x7f2184121c00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2174011c00 tx=0x7f217400f610 comp rx=0 tx=0).stop 2024-09-11T01:56:19.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.309+0000 7f21817fa640 1 -- 172.21.15.12:0/2212863051 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184125180 msgr2=0x7f2184121c00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:56:19.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.309+0000 7f21817fa640 1 --2- 172.21.15.12:0/2212863051 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184125180 0x7f2184121c00 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2174011c00 tx=0x7f217400f610 comp rx=0 tx=0).stop 2024-09-11T01:56:19.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.309+0000 7f2183fff640 1 --2- 172.21.15.12:0/2212863051 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f217003e210 0x7f21700406d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f2178034000 tx=0x7f217802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:19.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.315+0000 7f218ac71640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:56:19.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.315+0000 7f218ac71640 1 --2- 172.21.15.12:0/2212863051 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184122140 0x7f2184122550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:19.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.315+0000 7f218ac71640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:19.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.315+0000 7f218ac71640 1 -- 172.21.15.12:0/2212863051 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f2184122a90 con 0x7f2184122140 2024-09-11T01:56:19.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.315+0000 7f2188a0d640 1 --2- 172.21.15.12:0/2212863051 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184122140 0x7f2184122550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:19.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.315+0000 7f2188a0d640 10 monclient: get_auth_request con 0x7f2184122140 auth_method 0 2024-09-11T01:56:19.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.315+0000 7f2188a0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:19.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.315+0000 7f2188a0d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:19.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.315+0000 7f2188a0d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:19.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.315+0000 7f2188a0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:19.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.315+0000 7f2188a0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:19.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.315+0000 7f2188a0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:19.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.316+0000 7f2188a0d640 10 monclient(hunting): handle_auth_done global_id 304145 payload 824 2024-09-11T01:56:19.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.316+0000 7f2188a0d640 1 -- 172.21.15.12:0/2212863051 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f21841236c0 con 0x7f2184122140 2024-09-11T01:56:19.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.318+0000 7f2188a0d640 1 -- 172.21.15.12:0/2212863051 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184122140 msgr2=0x7f2184122550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:19.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.318+0000 7f2188a0d640 1 --2- 172.21.15.12:0/2212863051 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184122140 0x7f2184122550 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f217401c000 tx=0x7f2174004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:19.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.318+0000 7f21817fa640 1 -- 172.21.15.12:0/2212863051 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2174004220 con 0x7f2184122140 2024-09-11T01:56:19.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.318+0000 7f21817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:19.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.318+0000 7f21817fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:56:19.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.318+0000 7f21817fa640 10 monclient: dump: 2024-09-11T01:56:19.331 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:19.331 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:19.331 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.331 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.331 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:19.332 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:19.332 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:19.332 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:19.332 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:19.332 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:19.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.318+0000 7f21817fa640 1 -- 172.21.15.12:0/2212863051 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7f2174004220 con 0x7f2184122140 2024-09-11T01:56:19.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.318+0000 7f21817fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:56:19.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.318+0000 7f21817fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:56:19.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.318+0000 7f21817fa640 1 -- 172.21.15.12:0/2212863051 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184122140 msgr2=0x7f2184122550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:19.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.318+0000 7f21817fa640 1 --2- 172.21.15.12:0/2212863051 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184122140 0x7f2184122550 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f217401c000 tx=0x7f2174004730 comp rx=0 tx=0).stop 2024-09-11T01:56:19.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.318+0000 7f21817fa640 1 -- 172.21.15.12:0/2212863051 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184122140 msgr2=0x7f2184122550 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:56:19.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.318+0000 7f21817fa640 1 --2- 172.21.15.12:0/2212863051 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2184122140 0x7f2184122550 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f217401c000 tx=0x7f2174004730 comp rx=0 tx=0).stop 2024-09-11T01:56:19.333 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:56:19.333 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:56:19.333 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:56:19.333 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:56:19.333 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 250, 2024-09-11T01:56:19.334 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 14273, 2024-09-11T01:56:19.334 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:56:19.334 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:56:19.334 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:56:19.334 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:56:19.334 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:19.334 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 5, 2024-09-11T01:56:19.334 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:56:19.334 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:56:19.334 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:56:19.334 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:56:19.334 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:56:19.334 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:56:19.335 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:56:19.335 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:56:19.335 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:56:19.335 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:56:19.335 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:56:19.335 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:56:19.335 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:56:19.335 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:56:19.335 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:19.335 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:56:19.335 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:56:19.335 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:56:19.336 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:56:19.336 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:56:19.336 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:56:19.336 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:56:19.336 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:56:19.336 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:56:19.336 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:56:19.336 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:56:19.336 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:56:19.336 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:56:19.336 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:19.336 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.336 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:56:19.337 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:56:19.337 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:56:19.337 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:56:19.337 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:56:19.337 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:56:19.337 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:56:19.337 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:56:19.337 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:56:19.337 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:56:19.337 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:56:19.337 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:56:19.337 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:56:19.337 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:56:19.338 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:56:19.338 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:56:19.338 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:56:19.338 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:56:19.338 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:56:19.338 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:56:19.338 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:56:19.338 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:56:19.338 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:56:19.338 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:56:19.338 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:56:19.338 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:56:19.338 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:56:19.338 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:56:19.339 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:19.339 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:56:19.339 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.339 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:56:19.339 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.339 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:56:19.339 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:56:19.339 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:56:19.339 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:56:19.339 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.339 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:56:19.339 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:56:19.339 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:19.340 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.340 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.340 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:56:19.340 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:56:19.340 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:19.340 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:19.340 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:19.340 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.340 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:56:19.340 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:56:19.340 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:56:19.340 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:56:19.340 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:56:19.341 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.341 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.341 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:56:19.341 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:56:19.341 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:56:19.341 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:56:19.341 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.341 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:56:19.341 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:56:19.341 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:19.341 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.341 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.341 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:56:19.342 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:56:19.342 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:19.342 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:19.342 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:19.342 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.342 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:56:19.342 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:56:19.342 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:56:19.342 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:56:19.342 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:56:19.342 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.343 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.343 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:56:19.343 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:56:19.343 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:56:19.343 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:56:19.343 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.343 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:56:19.343 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:56:19.343 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:19.343 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.343 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.343 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:56:19.344 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:56:19.344 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:19.344 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:19.344 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:19.344 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.344 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:56:19.344 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:56:19.344 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:56:19.344 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:56:19.344 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:56:19.344 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:19.344 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:19.344 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.345 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:56:19.345 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:56:19.345 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.345 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:56:19.345 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:56:19.345 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:56:19.345 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:19.345 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:19.345 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:56:19.345 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.345 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:56:19.345 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:56:19.345 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:56:19.346 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:19.346 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:19.346 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.346 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:56:19.346 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:56:19.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.319+0000 7f218ac71640 1 -- 172.21.15.12:0/2212863051 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f217003e210 msgr2=0x7f21700406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:19.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.319+0000 7f218ac71640 1 --2- 172.21.15.12:0/2212863051 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f217003e210 0x7f21700406d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f2178034000 tx=0x7f217802e000 comp rx=0 tx=0).stop 2024-09-11T01:56:19.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.319+0000 7f218ac71640 10 monclient: shutdown 2024-09-11T01:56:19.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.319+0000 7f218ac71640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:19.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.319+0000 7f218ac71640 1 -- 172.21.15.12:0/2212863051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2184138230 msgr2=0x7f2184134560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:19.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.319+0000 7f218920e640 1 -- 172.21.15.12:0/2212863051 reap_dead start 2024-09-11T01:56:19.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.319+0000 7f218ac71640 1 --2- 172.21.15.12:0/2212863051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2184138230 0x7f2184134560 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f217c00c780 tx=0x7f217c00cc40 comp rx=0 tx=0).stop 2024-09-11T01:56:19.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.319+0000 7f218ac71640 1 -- 172.21.15.12:0/2212863051 shutdown_connections 2024-09-11T01:56:19.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.319+0000 7f218ac71640 1 --2- 172.21.15.12:0/2212863051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2184138230 0x7f2184134560 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.319+0000 7f218ac71640 1 -- 172.21.15.12:0/2212863051 >> 172.21.15.12:0/2212863051 conn(0x7f21840a7020 msgr2=0x7f2184136000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:19.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.319+0000 7f218ac71640 1 -- 172.21.15.12:0/2212863051 shutdown_connections 2024-09-11T01:56:19.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.319+0000 7f218ac71640 1 -- 172.21.15.12:0/2212863051 wait complete. 2024-09-11T01:56:19.348 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:56:19.349 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:56:19.349 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:56:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.449+0000 7f760cd05640 10 monclient: get_monmap_and_config 2024-09-11T01:56:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.449+0000 7f760cd05640 10 monclient: build_initial_monmap 2024-09-11T01:56:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.449+0000 7f760cd05640 10 monclient: monmap: 2024-09-11T01:56:19.450 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:19.451 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:19.451 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:56:19.449823+0000 2024-09-11T01:56:19.451 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:56:19.449823+0000 2024-09-11T01:56:19.451 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:19.451 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:19.451 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:56:19.451 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:56:19.451 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:56:19.451 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:56:19.451 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:56:19.451 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:19.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.449+0000 7f760cd05640 1 Processor -- start 2024-09-11T01:56:19.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.449+0000 7f760cd05640 1 -- start start 2024-09-11T01:56:19.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.449+0000 7f760cd05640 10 monclient: init 2024-09-11T01:56:19.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760cd05640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:19.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760cd05640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-11T01:56:19.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760cd05640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7608137e50 0x7f760809d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:19.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760cd05640 10 monclient(hunting): picked mon.noname-a con 0x7f7608137e50 addr v2:172.21.15.12:3300/0 2024-09-11T01:56:19.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760cd05640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7608138230 addr v1:172.21.15.71:6789/0 2024-09-11T01:56:19.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760cd05640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f760813d380 0x7f760813f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:19.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760cd05640 10 monclient(hunting): picked mon.noname-c con 0x7f760813d380 addr v2:172.21.15.71:3300/0 2024-09-11T01:56:19.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760cd05640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76080a62d0 con 0x7f7608138230 2024-09-11T01:56:19.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760cd05640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:19.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760cd05640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f760809bb40 con 0x7f7608137e50 2024-09-11T01:56:19.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760cd05640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:19.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760cd05640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f760809c4a0 con 0x7f760813d380 2024-09-11T01:56:19.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760cd05640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:19.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760cd05640 10 monclient(hunting): authenticate will time out at 2385.892408s 2024-09-11T01:56:19.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760659c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7608137e50 0x7f760809d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:19.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760659c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7608137e50 0x7f760809d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49820/0 (socket says 172.21.15.12:49820) 2024-09-11T01:56:19.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760659c640 1 -- 172.21.15.12:0/881998322 learned_addr learned my addr 172.21.15.12:0/881998322 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:19.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760659c640 10 monclient(hunting): get_auth_request con 0x7f7608137e50 auth_method 0 2024-09-11T01:56:19.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:19.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:19.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:19.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:19.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:19.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:19.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f7606d9d640 1 --2- 172.21.15.12:0/881998322 >> v2:172.21.15.71:3300/0 conn(0x7f760813d380 0x7f760813f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:19.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f7606d9d640 10 monclient(hunting): get_auth_request con 0x7f760813d380 auth_method 0 2024-09-11T01:56:19.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f7606d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:19.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f7606d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:19.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f7606d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:19.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760659c640 10 monclient(hunting): handle_auth_done global_id 294240 payload 274 2024-09-11T01:56:19.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760659c640 1 -- 172.21.15.12:0/881998322 >> v2:172.21.15.71:3300/0 conn(0x7f760813d380 msgr2=0x7f760813f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:19.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760659c640 1 --2- 172.21.15.12:0/881998322 >> v2:172.21.15.71:3300/0 conn(0x7f760813d380 0x7f760813f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760659c640 1 -- 172.21.15.12:0/881998322 >> v1:172.21.15.71:6789/0 conn(0x7f7608138230 legacy=0x7f7608138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:19.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760659c640 10 monclient: _finish_hunting 0 2024-09-11T01:56:19.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760659c640 1 monclient: found mon.noname-a 2024-09-11T01:56:19.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:19.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:56:19.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f760659c640 1 -- 172.21.15.12:0/881998322 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f760809c780 con 0x7f7608137e50 2024-09-11T01:56:19.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.450+0000 7f7606d9d640 1 --2- 172.21.15.12:0/881998322 >> v2:172.21.15.71:3300/0 conn(0x7f760813d380 0x7f760813f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:56:19.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.453+0000 7f760659c640 1 -- 172.21.15.12:0/881998322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7608137e50 msgr2=0x7f760809d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:19.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.453+0000 7f760659c640 1 --2- 172.21.15.12:0/881998322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7608137e50 0x7f760809d160 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f76080ab580 tx=0x7f75fc02a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bd5730705919c4a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:19.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.453+0000 7f760559a640 1 -- 172.21.15.12:0/881998322 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75fc037070 con 0x7f7608137e50 2024-09-11T01:56:19.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.453+0000 7f760559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:19.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.453+0000 7f760559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:56:19.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.453+0000 7f760559a640 10 monclient: dump: 2024-09-11T01:56:19.464 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:19.464 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:19.464 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.464 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.464 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:19.464 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:19.464 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:19.464 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:19.464 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:19.465 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:19.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.453+0000 7f760559a640 10 monclient: _finish_auth 0 2024-09-11T01:56:19.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.453+0000 7f760559a640 10 monclient: _check_auth_tickets 2024-09-11T01:56:19.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.453+0000 7f760559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:19.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.453+0000 7f760cd05640 5 monclient: authenticate success, global_id 294240 2024-09-11T01:56:19.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.453+0000 7f760cd05640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:56:19.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.453+0000 7f760559a640 1 -- 172.21.15.12:0/881998322 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75fc032440 con 0x7f7608137e50 2024-09-11T01:56:19.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.453+0000 7f760559a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:19.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.453+0000 7f760cd05640 10 monclient: get_monmap_and_config success 2024-09-11T01:56:19.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.453+0000 7f760cd05640 4 set_mon_vals no callback set 2024-09-11T01:56:19.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.453+0000 7f760559a640 1 -- 172.21.15.12:0/881998322 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75fc0327f0 con 0x7f7608137e50 2024-09-11T01:56:19.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.453+0000 7f760559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:19.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.453+0000 7f760559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:56:19.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.453+0000 7f760559a640 10 monclient: dump: 2024-09-11T01:56:19.466 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:19.466 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:19.466 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.466 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.466 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:19.466 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:19.467 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:19.467 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:19.467 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:19.467 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:19.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.454+0000 7f760cd05640 10 monclient: shutdown 2024-09-11T01:56:19.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.454+0000 7f760cd05640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:19.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.454+0000 7f760cd05640 1 -- 172.21.15.12:0/881998322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7608137e50 msgr2=0x7f760809d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:19.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.454+0000 7f760cd05640 1 --2- 172.21.15.12:0/881998322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7608137e50 0x7f760809d160 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f76080ab580 tx=0x7f75fc02a080 comp rx=0 tx=0).stop 2024-09-11T01:56:19.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.454+0000 7f760cd05640 1 -- 172.21.15.12:0/881998322 shutdown_connections 2024-09-11T01:56:19.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.454+0000 7f760cd05640 1 --2- 172.21.15.12:0/881998322 >> v2:172.21.15.71:3300/0 conn(0x7f760813d380 0x7f760813f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.454+0000 7f760cd05640 1 --2- 172.21.15.12:0/881998322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7608137e50 0x7f760809d160 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.454+0000 7f760cd05640 1 -- 172.21.15.12:0/881998322 >> 172.21.15.12:0/881998322 conn(0x7f76080a7020 msgr2=0x7f76080a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:19.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.454+0000 7f760cd05640 1 -- 172.21.15.12:0/881998322 shutdown_connections 2024-09-11T01:56:19.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.454+0000 7f760cd05640 1 -- 172.21.15.12:0/881998322 wait complete. 2024-09-11T01:56:19.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.455+0000 7f760cd05640 10 monclient: build_initial_monmap 2024-09-11T01:56:19.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.455+0000 7f760cd05640 10 monclient: monmap: 2024-09-11T01:56:19.468 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:19.468 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:56:19.468 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:56:19.468 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:56:19.468 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:19.468 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:19.469 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:56:19.469 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:56:19.469 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:56:19.469 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:19.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.455+0000 7f760cd05640 1 Processor -- start 2024-09-11T01:56:19.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.455+0000 7f760cd05640 1 -- start start 2024-09-11T01:56:19.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.455+0000 7f760cd05640 10 monclient: init 2024-09-11T01:56:19.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f760cd05640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:19.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f760cd05640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:56:19.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f760cd05640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7608137e50 0x7f76081ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:19.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f760cd05640 10 monclient(hunting): picked mon.noname-b con 0x7f7608137e50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:56:19.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f760cd05640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7608138230 0x7f76081cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:19.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f760cd05640 10 monclient(hunting): picked mon.noname-a con 0x7f7608138230 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:56:19.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f760cd05640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f760813d380 0x7f76081cd920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:19.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f760cd05640 10 monclient(hunting): picked mon.noname-c con 0x7f760813d380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:19.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f760cd05640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:19.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f760cd05640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f760806f890 con 0x7f7608138230 2024-09-11T01:56:19.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f760cd05640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:19.470 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f760cd05640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f760806f710 con 0x7f7608137e50 2024-09-11T01:56:19.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f760cd05640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:19.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f760cd05640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f760813fcb0 con 0x7f760813d380 2024-09-11T01:56:19.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f760cd05640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:19.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f760cd05640 10 monclient(hunting): authenticate will time out at 2385.898133s 2024-09-11T01:56:19.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7605d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7608138230 0x7f76081cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:19.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7606d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f760813d380 0x7f76081cd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:19.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7605d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7608138230 0x7f76081cd3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49826/0 (socket says 172.21.15.12:49826) 2024-09-11T01:56:19.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7605d9b640 1 -- 172.21.15.12:0/2154941216 learned_addr learned my addr 172.21.15.12:0/2154941216 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:19.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7605d9b640 10 monclient(hunting): get_auth_request con 0x7f7608138230 auth_method 0 2024-09-11T01:56:19.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7605d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:19.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7605d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:19.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7605d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:19.471 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f760659c640 1 --2- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7608137e50 0x7f76081ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:19.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f760659c640 10 monclient(hunting): get_auth_request con 0x7f7608137e50 auth_method 0 2024-09-11T01:56:19.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f760659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:19.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f760659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:19.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f760659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:19.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7606d9d640 10 monclient(hunting): get_auth_request con 0x7f760813d380 auth_method 0 2024-09-11T01:56:19.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7606d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:19.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7606d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:19.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7606d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:19.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7605d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:19.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7605d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:19.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7605d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:19.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7606d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:19.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7606d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:19.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7606d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:19.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f760659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:19.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f760659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:19.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f760659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:19.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7606d9d640 10 monclient(hunting): handle_auth_done global_id 304148 payload 824 2024-09-11T01:56:19.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7606d9d640 1 -- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7608137e50 msgr2=0x7f76081ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:19.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7606d9d640 1 --2- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7608137e50 0x7f76081ccea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7606d9d640 1 -- 172.21.15.12:0/2154941216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7608138230 msgr2=0x7f76081cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:19.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7606d9d640 1 --2- 172.21.15.12:0/2154941216 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7608138230 0x7f76081cd3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7606d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:56:19.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7606d9d640 1 monclient: found mon.noname-c 2024-09-11T01:56:19.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7606d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:19.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7606d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:56:19.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.456+0000 7f7606d9d640 1 -- 172.21.15.12:0/2154941216 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f760813b0b0 con 0x7f760813d380 2024-09-11T01:56:19.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.458+0000 7f7606d9d640 1 -- 172.21.15.12:0/2154941216 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f760813d380 msgr2=0x7f76081cd920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:19.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.458+0000 7f7606d9d640 1 --2- 172.21.15.12:0/2154941216 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f760813d380 0x7f76081cd920 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7600007ef0 tx=0x7f760000d4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:19.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.458+0000 7f75f6ffd640 1 -- 172.21.15.12:0/2154941216 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f760001a070 con 0x7f760813d380 2024-09-11T01:56:19.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.458+0000 7f75f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:19.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.458+0000 7f75f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:56:19.474 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.458+0000 7f75f6ffd640 10 monclient: dump: 2024-09-11T01:56:19.474 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:19.474 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:19.474 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.474 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.474 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:19.475 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:19.475 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:19.475 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:19.475 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:19.475 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:19.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.458+0000 7f75f6ffd640 10 monclient: _finish_auth 0 2024-09-11T01:56:19.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.458+0000 7f75f6ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:56:19.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.458+0000 7f75f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:19.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.458+0000 7f75f6ffd640 1 -- 172.21.15.12:0/2154941216 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7600003b90 con 0x7f760813d380 2024-09-11T01:56:19.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.458+0000 7f760cd05640 5 monclient: authenticate success, global_id 304148 2024-09-11T01:56:19.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.459+0000 7f760cd05640 10 monclient: _renew_subs 2024-09-11T01:56:19.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.459+0000 7f760cd05640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:56:19.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.459+0000 7f760cd05640 1 -- 172.21.15.12:0/2154941216 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f760813c0d0 con 0x7f760813d380 2024-09-11T01:56:19.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.459+0000 7f75f6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:19.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.459+0000 7f75f6ffd640 1 -- 172.21.15.12:0/2154941216 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7600004030 con 0x7f760813d380 2024-09-11T01:56:19.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.459+0000 7f75f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:19.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.459+0000 7f75f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:56:19.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.459+0000 7f75f6ffd640 10 monclient: dump: 2024-09-11T01:56:19.476 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:19.476 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:19.476 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.476 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.476 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:19.476 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:19.477 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:19.477 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:19.477 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:19.477 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:19.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.459+0000 7f760cd05640 10 monclient: _renew_subs 2024-09-11T01:56:19.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.459+0000 7f760cd05640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:56:19.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.459+0000 7f760cd05640 1 -- 172.21.15.12:0/2154941216 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f760813bc10 con 0x7f760813d380 2024-09-11T01:56:19.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.459+0000 7f7604d99640 4 set_mon_vals no callback set 2024-09-11T01:56:19.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.459+0000 7f760cd05640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:19.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.459+0000 7f760cd05640 1 --2- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f760813dde0 0x7f760813e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:19.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.459+0000 7f760cd05640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:19.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.459+0000 7f760cd05640 1 -- 172.21.15.12:0/2154941216 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f760806a060 con 0x7f760813dde0 2024-09-11T01:56:19.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.460+0000 7f75f6ffd640 1 -- 172.21.15.12:0/2154941216 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f7600004250 con 0x7f760813d380 2024-09-11T01:56:19.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.460+0000 7f760659c640 1 --2- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f760813dde0 0x7f760813e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:19.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.460+0000 7f760659c640 10 monclient: get_auth_request con 0x7f760813dde0 auth_method 0 2024-09-11T01:56:19.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.460+0000 7f760659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:19.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.460+0000 7f760659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:19.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.460+0000 7f760659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:19.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.460+0000 7f760659c640 1 -- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f760813dde0 msgr2=0x7f760813e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T01:56:19.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.460+0000 7f760659c640 1 -- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f760813dde0 msgr2=0x7f760813e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:56:19.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.460+0000 7f760659c640 1 --2- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f760813dde0 0x7f760813e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:56:19.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.460+0000 7f760659c640 1 --2- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f760813dde0 0x7f760813e1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:56:19.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.460+0000 7f75f6ffd640 1 --2- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f75e803e210 0x7f75e80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:19.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.460+0000 7f75f6ffd640 1 -- 172.21.15.12:0/2154941216 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(920..920 src has 676..920) ==== 4815+0+0 (secure 0 0 0) 0x7f76000533f0 con 0x7f760813d380 2024-09-11T01:56:19.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.460+0000 7f7605d9b640 1 --2- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f75e803e210 0x7f75e80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:19.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.460+0000 7f7605d9b640 10 monclient: get_auth_request con 0x7f75e803e210 auth_method 0 2024-09-11T01:56:19.481 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.463+0000 7f7605d9b640 1 --2- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f75e803e210 0x7f75e80406d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f760813f7c0 tx=0x7f75f8007ef0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:19.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.661+0000 7f760659c640 1 --2- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f760813dde0 0x7f760813e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:19.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.661+0000 7f760659c640 10 monclient: get_auth_request con 0x7f760813dde0 auth_method 0 2024-09-11T01:56:19.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.661+0000 7f760659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:19.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.661+0000 7f760659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:19.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.661+0000 7f760659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:56:19.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.661+0000 7f760659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:19.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.661+0000 7f760659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:19.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.661+0000 7f760659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:19.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.661+0000 7f760659c640 10 monclient(hunting): handle_auth_done global_id 274393 payload 824 2024-09-11T01:56:19.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.661+0000 7f760659c640 1 -- 172.21.15.12:0/2154941216 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f760809d160 con 0x7f760813dde0 2024-09-11T01:56:19.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.663+0000 7f760659c640 1 -- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f760813dde0 msgr2=0x7f760813e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:19.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.663+0000 7f760659c640 1 --2- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f760813dde0 0x7f760813e1c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f75fc034000 tx=0x7f75fc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:19.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.664+0000 7f75f6ffd640 1 -- 172.21.15.12:0/2154941216 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75fc042070 con 0x7f760813dde0 2024-09-11T01:56:19.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.664+0000 7f75f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:19.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.664+0000 7f75f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:56:19.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.664+0000 7f75f6ffd640 10 monclient: dump: 2024-09-11T01:56:19.665 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:19.666 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:19.666 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.666 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.666 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:19.666 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:19.666 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:19.666 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:19.666 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:19.666 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:19.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.664+0000 7f75f6ffd640 1 -- 172.21.15.12:0/2154941216 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f75fc03c820 con 0x7f760813dde0 2024-09-11T01:56:19.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.664+0000 7f75f6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:19.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.664+0000 7f75f6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:56:19.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.664+0000 7f75f6ffd640 1 -- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f760813dde0 msgr2=0x7f760813e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:19.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.664+0000 7f75f6ffd640 1 --2- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f760813dde0 0x7f760813e1c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f75fc034000 tx=0x7f75fc0046c0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.664+0000 7f75f6ffd640 1 -- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f760813dde0 msgr2=0x7f760813e1c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:56:19.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.664+0000 7f75f6ffd640 1 --2- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f760813dde0 0x7f760813e1c0 secure :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f75fc034000 tx=0x7f75fc0046c0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.671+0000 7f760cd05640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:56:19.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.671+0000 7f760cd05640 1 --2- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f760813e700 0x7f7608139c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:19.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.671+0000 7f760cd05640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:19.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.671+0000 7f760cd05640 1 -- 172.21.15.12:0/2154941216 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f76080a2410 con 0x7f760813e700 2024-09-11T01:56:19.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.671+0000 7f760659c640 1 --2- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f760813e700 0x7f7608139c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:19.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.671+0000 7f760659c640 10 monclient: get_auth_request con 0x7f760813e700 auth_method 0 2024-09-11T01:56:19.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.671+0000 7f760659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:19.673 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.671+0000 7f760659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:19.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.671+0000 7f760659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:19.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.671+0000 7f760659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:19.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.671+0000 7f760659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:19.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.671+0000 7f760659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:19.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.671+0000 7f760659c640 10 monclient(hunting): handle_auth_done global_id 274396 payload 824 2024-09-11T01:56:19.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.671+0000 7f760659c640 1 -- 172.21.15.12:0/2154941216 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7608069950 con 0x7f760813e700 2024-09-11T01:56:19.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.674+0000 7f760659c640 1 -- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f760813e700 msgr2=0x7f7608139c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:19.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.674+0000 7f760659c640 1 --2- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f760813e700 0x7f7608139c80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f75fc02a0c0 tx=0x7f75fc009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:19.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.674+0000 7f75f6ffd640 1 -- 172.21.15.12:0/2154941216 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75fc037040 con 0x7f760813e700 2024-09-11T01:56:19.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.674+0000 7f75f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:19.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.674+0000 7f75f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:56:19.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.674+0000 7f75f6ffd640 10 monclient: dump: 2024-09-11T01:56:19.676 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:19.676 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:19.676 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.676 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:19.676 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:19.676 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:19.676 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:19.677 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:19.677 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:19.677 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:19.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.674+0000 7f75f6ffd640 1 -- 172.21.15.12:0/2154941216 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f75fc04ca30 con 0x7f760813e700 2024-09-11T01:56:19.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.674+0000 7f75f6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:56:19.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.674+0000 7f75f6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:56:19.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.674+0000 7f75f6ffd640 1 -- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f760813e700 msgr2=0x7f7608139c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:19.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.674+0000 7f75f6ffd640 1 --2- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f760813e700 0x7f7608139c80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f75fc02a0c0 tx=0x7f75fc009240 comp rx=0 tx=0).stop 2024-09-11T01:56:19.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.674+0000 7f75f6ffd640 1 -- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f760813e700 msgr2=0x7f7608139c80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:56:19.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.674+0000 7f75f6ffd640 1 --2- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f760813e700 0x7f7608139c80 secure :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f75fc02a0c0 tx=0x7f75fc009240 comp rx=0 tx=0).stop 2024-09-11T01:56:19.678 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:56:19.678 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:56:19.678 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:56:19.678 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:56:19.678 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 250, 2024-09-11T01:56:19.678 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 169492, 2024-09-11T01:56:19.678 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:56:19.678 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:56:19.678 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:56:19.678 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:56:19.678 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:19.678 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 5, 2024-09-11T01:56:19.679 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:56:19.679 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:56:19.679 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:56:19.679 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:56:19.679 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:56:19.679 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:56:19.679 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:56:19.679 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:56:19.679 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:56:19.679 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:56:19.679 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:56:19.679 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:56:19.679 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:56:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:56:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:56:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:56:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:56:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:56:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:56:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:56:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:56:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:56:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:56:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:56:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:56:19.680 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:56:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:56:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:56:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:56:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:56:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:56:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:56:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:56:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:56:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:56:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:56:19.681 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:56:19.682 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:56:19.682 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:56:19.682 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:56:19.682 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:56:19.682 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:56:19.682 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:56:19.682 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:56:19.682 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:56:19.682 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:56:19.682 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:56:19.682 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:56:19.682 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:56:19.682 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:56:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:56:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:56:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:56:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:56:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:56:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:56:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:56:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:56:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:56:19.683 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:56:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:56:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:56:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:56:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:56:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:56:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:19.684 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:56:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:56:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:56:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:56:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:56:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:56:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:56:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:56:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:56:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:56:19.685 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:56:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:56:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:56:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:56:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:56:19.686 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:56:19.687 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:56:19.687 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:56:19.687 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.687 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.687 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:56:19.687 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:56:19.687 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:56:19.687 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:56:19.687 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.687 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:56:19.687 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:56:19.687 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:19.687 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.687 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.688 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:56:19.688 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:56:19.688 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:19.688 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:19.688 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:19.688 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.688 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:56:19.688 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:56:19.688 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:56:19.688 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:56:19.688 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:56:19.688 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:19.688 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:19.689 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.689 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:56:19.689 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:56:19.689 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.689 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:56:19.689 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:56:19.689 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:56:19.689 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:19.689 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:19.689 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:56:19.689 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.689 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:56:19.689 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:56:19.689 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 5 2024-09-11T01:56:19.690 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:19.690 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:19.690 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:56:19.690 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.690 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:56:19.690 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:56:19.690 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 6 2024-09-11T01:56:19.690 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:19.690 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:19.690 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:56:19.690 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:19.690 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:56:19.690 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:56:19.691 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:56:19.691 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:19.691 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:19.691 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:19.691 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:56:19.691 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:56:19.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.674+0000 7f760cd05640 1 -- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f75e803e210 msgr2=0x7f75e80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:19.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.674+0000 7f760cd05640 1 --2- 172.21.15.12:0/2154941216 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f75e803e210 0x7f75e80406d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f760813f7c0 tx=0x7f75f8007ef0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.674+0000 7f760cd05640 10 monclient: shutdown 2024-09-11T01:56:19.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.674+0000 7f760cd05640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:19.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.674+0000 7f760cd05640 1 -- 172.21.15.12:0/2154941216 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f760813d380 msgr2=0x7f76081cd920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:19.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.674+0000 7f760cd05640 1 --2- 172.21.15.12:0/2154941216 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f760813d380 0x7f76081cd920 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7600007ef0 tx=0x7f760000d4a0 comp rx=0 tx=0).stop 2024-09-11T01:56:19.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.674+0000 7f7606d9d640 1 -- 172.21.15.12:0/2154941216 reap_dead start 2024-09-11T01:56:19.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.674+0000 7f7606d9d640 1 -- 172.21.15.12:0/2154941216 reap_dead start 2024-09-11T01:56:19.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.675+0000 7f760cd05640 1 -- 172.21.15.12:0/2154941216 shutdown_connections 2024-09-11T01:56:19.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.675+0000 7f760cd05640 1 -- 172.21.15.12:0/2154941216 >> 172.21.15.12:0/2154941216 conn(0x7f76080a7020 msgr2=0x7f7608136ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:19.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.675+0000 7f760cd05640 1 -- 172.21.15.12:0/2154941216 shutdown_connections 2024-09-11T01:56:19.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:19.675+0000 7f760cd05640 1 -- 172.21.15.12:0/2154941216 wait complete. 2024-09-11T01:56:19.693 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:56:19.693 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-11T01:56:19.693 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-11T01:56:19.694 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-11T01:56:19.694 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-11T01:56:19.694 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-11T01:56:19.694 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:56:19.707 INFO:tasks.ceph.mon.b:Stopped 2024-09-11T01:56:19.707 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-11T01:56:19.707 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-11T01:56:19.707 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T01:56:34.708 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-11T01:56:34.709 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-11T01:56:34.709 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T01:56:34.709 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:56:34.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.820+0000 7fc189ec6640 10 monclient: get_monmap_and_config 2024-09-11T01:56:34.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.820+0000 7fc189ec6640 10 monclient: build_initial_monmap 2024-09-11T01:56:34.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.820+0000 7fc189ec6640 10 monclient: monmap: 2024-09-11T01:56:34.822 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:34.822 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:34.822 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:56:34.821340+0000 2024-09-11T01:56:34.822 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:56:34.821340+0000 2024-09-11T01:56:34.823 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:34.823 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:34.823 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:56:34.823 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:56:34.823 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:56:34.823 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:56:34.823 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:56:34.823 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:34.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.821+0000 7fc189ec6640 1 Processor -- start 2024-09-11T01:56:34.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.821+0000 7fc189ec6640 1 -- start start 2024-09-11T01:56:34.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.821+0000 7fc189ec6640 10 monclient: init 2024-09-11T01:56:34.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.821+0000 7fc189ec6640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:34.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.821+0000 7fc189ec6640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-11T01:56:34.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.821+0000 7fc189ec6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc1840a2e90 0x7fc18406dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:34.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.821+0000 7fc189ec6640 10 monclient(hunting): picked mon.noname-c con 0x7fc1840a2e90 addr v2:172.21.15.71:3300/0 2024-09-11T01:56:34.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.822+0000 7fc189ec6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc1840a3270 addr v1:172.21.15.12:6789/0 2024-09-11T01:56:34.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.822+0000 7fc189ec6640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 0x7fc18413f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:34.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.822+0000 7fc189ec6640 10 monclient(hunting): picked mon.noname-a con 0x7fc18413d490 addr v2:172.21.15.12:3300/0 2024-09-11T01:56:34.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.822+0000 7fc189ec6640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1840a46c0 con 0x7fc1840a3270 2024-09-11T01:56:34.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.822+0000 7fc189ec6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:34.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.822+0000 7fc189ec6640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc1840a4900 con 0x7fc18413d490 2024-09-11T01:56:34.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.822+0000 7fc189ec6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:34.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.822+0000 7fc189ec6640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fc18406ee50 con 0x7fc1840a2e90 2024-09-11T01:56:34.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.822+0000 7fc189ec6640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:34.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.822+0000 7fc189ec6640 10 monclient(hunting): authenticate will time out at 2401.264050s 2024-09-11T01:56:34.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.822+0000 7fc1837fe640 1 -- >> v2:172.21.15.71:3300/0 conn(0x7fc1840a2e90 msgr2=0x7fc18406dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:56:34.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.822+0000 7fc1837fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc1840a2e90 0x7fc18406dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:56:34.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.822+0000 7fc182ffd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc1840a3270 0x7fc1840a3680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:48078/0 (socket says 172.21.15.12:48078) 2024-09-11T01:56:34.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.822+0000 7fc182ffd640 1 -- 172.21.15.12:0/3682125754 learned_addr learned my addr 172.21.15.12:0/3682125754 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:56:34.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.822+0000 7fc183fff640 1 --2- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 0x7fc18413f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:34.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.822+0000 7fc183fff640 10 monclient(hunting): get_auth_request con 0x7fc18413d490 auth_method 0 2024-09-11T01:56:34.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.822+0000 7fc183fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:34.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.822+0000 7fc183fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:34.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.822+0000 7fc183fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:34.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.822+0000 7fc183fff640 1 -- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 msgr2=0x7fc18413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:56:34.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.822+0000 7fc183fff640 1 -- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 msgr2=0x7fc18413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:56:34.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.822+0000 7fc183fff640 1 --2- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 0x7fc18413f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:56:34.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.822+0000 7fc183fff640 1 --2- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 0x7fc18413f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:56:34.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:34.824+0000 7fc182ffd640 1 -- 172.21.15.12:0/3682125754 >> v1:172.21.15.12:6789/0 conn(0x7fc1840a3270 legacy=0x7fc1840a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:35.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:35.023+0000 7fc1837fe640 1 -- 172.21.15.12:0/3682125754 >> v2:172.21.15.71:3300/0 conn(0x7fc1840a2e90 msgr2=0x7fc18406dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:56:35.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:35.023+0000 7fc1837fe640 1 --2- 172.21.15.12:0/3682125754 >> v2:172.21.15.71:3300/0 conn(0x7fc1840a2e90 0x7fc18406dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:56:35.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:35.023+0000 7fc183fff640 1 --2- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 0x7fc18413f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:35.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:35.023+0000 7fc183fff640 10 monclient(hunting): get_auth_request con 0x7fc18413d490 auth_method 0 2024-09-11T01:56:35.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:35.023+0000 7fc183fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:35.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:35.023+0000 7fc183fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:35.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:35.023+0000 7fc183fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:56:35.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:35.023+0000 7fc183fff640 1 -- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 msgr2=0x7fc18413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:56:35.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:35.023+0000 7fc183fff640 1 -- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 msgr2=0x7fc18413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:56:35.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:35.023+0000 7fc183fff640 1 --2- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 0x7fc18413f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:56:35.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:35.023+0000 7fc183fff640 1 --2- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 0x7fc18413f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:56:35.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:35.424+0000 7fc1837fe640 1 -- 172.21.15.12:0/3682125754 >> v2:172.21.15.71:3300/0 conn(0x7fc1840a2e90 msgr2=0x7fc18406dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:56:35.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:35.424+0000 7fc1837fe640 1 --2- 172.21.15.12:0/3682125754 >> v2:172.21.15.71:3300/0 conn(0x7fc1840a2e90 0x7fc18406dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:56:35.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:35.424+0000 7fc183fff640 1 --2- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 0x7fc18413f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:35.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:35.424+0000 7fc183fff640 10 monclient(hunting): get_auth_request con 0x7fc18413d490 auth_method 0 2024-09-11T01:56:35.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:35.424+0000 7fc183fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:35.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:35.424+0000 7fc183fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:35.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:35.424+0000 7fc183fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:56:35.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:35.424+0000 7fc183fff640 1 -- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 msgr2=0x7fc18413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T01:56:35.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:35.424+0000 7fc183fff640 1 -- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 msgr2=0x7fc18413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:56:35.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:35.424+0000 7fc183fff640 1 --2- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 0x7fc18413f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:56:35.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:35.424+0000 7fc183fff640 1 --2- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 0x7fc18413f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:56:36.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:36.225+0000 7fc1837fe640 1 -- 172.21.15.12:0/3682125754 >> v2:172.21.15.71:3300/0 conn(0x7fc1840a2e90 msgr2=0x7fc18406dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:56:36.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:36.225+0000 7fc1837fe640 1 --2- 172.21.15.12:0/3682125754 >> v2:172.21.15.71:3300/0 conn(0x7fc1840a2e90 0x7fc18406dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:56:36.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:36.225+0000 7fc183fff640 1 --2- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 0x7fc18413f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:36.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:36.225+0000 7fc183fff640 10 monclient(hunting): get_auth_request con 0x7fc18413d490 auth_method 0 2024-09-11T01:56:36.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:36.225+0000 7fc183fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:36.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:36.225+0000 7fc183fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:36.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:36.225+0000 7fc183fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:56:36.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:36.225+0000 7fc183fff640 1 -- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 msgr2=0x7fc18413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:56:36.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:36.225+0000 7fc183fff640 1 -- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 msgr2=0x7fc18413f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:56:36.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:36.225+0000 7fc183fff640 1 --2- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 0x7fc18413f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:56:36.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:36.225+0000 7fc183fff640 1 --2- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 0x7fc18413f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:56:37.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc1817fa640 10 monclient(hunting): tick 2024-09-11T01:56:37.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc1817fa640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:56:37.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc1817fa640 1 monclient(hunting): continuing hunt 2024-09-11T01:56:37.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc1817fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:56:37.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc1817fa640 1 -- 172.21.15.12:0/3682125754 >> v2:172.21.15.71:3300/0 conn(0x7fc1840a2e90 msgr2=0x7fc18406dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:56:37.824 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc1817fa640 1 --2- 172.21.15.12:0/3682125754 >> v2:172.21.15.71:3300/0 conn(0x7fc1840a2e90 0x7fc18406dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:37.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc1817fa640 1 -- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 msgr2=0x7fc18413f880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:56:37.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc1817fa640 1 --2- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 0x7fc18413f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:37.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc1817fa640 1 -- 172.21.15.12:0/3682125754 >> v1:172.21.15.12:6789/0 conn(0x7fc1840a3270 legacy=0x7fc1840a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:37.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc1817fa640 10 monclient: _add_conns ranks=[2,1] 2024-09-11T01:56:37.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc1817fa640 1 --2- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170003730 0x7fc170005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:37.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc1817fa640 10 monclient(hunting): picked mon.noname-b con 0x7fc170003730 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:37.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc1817fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc1700062b0 addr v1:172.21.15.71:6789/0 2024-09-11T01:56:37.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc1817fa640 1 -- 172.21.15.12:0/3682125754 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc18406eb70 con 0x7fc1700062b0 2024-09-11T01:56:37.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc1817fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:37.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc1817fa640 1 -- 172.21.15.12:0/3682125754 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc1840a4900 con 0x7fc170003730 2024-09-11T01:56:37.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc1817fa640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:37.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc1837fe640 1 --2- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170003730 0x7fc170005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:37.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc182ffd640 1 -- 172.21.15.12:0/3682125754 >> v1:172.21.15.71:6789/0 conn(0x7fc1700062b0 legacy=0x7fc170008750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:56:37.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc1837fe640 10 monclient(hunting): get_auth_request con 0x7fc170003730 auth_method 0 2024-09-11T01:56:37.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc1837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:37.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc1837fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:37.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc1837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:37.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc1837fe640 1 -- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170003730 msgr2=0x7fc170005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:56:37.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc1837fe640 1 -- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170003730 msgr2=0x7fc170005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:56:37.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc1837fe640 1 --2- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170003730 0x7fc170005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:56:37.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:37.821+0000 7fc1837fe640 1 --2- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170003730 0x7fc170005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:56:38.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:38.022+0000 7fc182ffd640 1 -- 172.21.15.12:0/3682125754 >> v1:172.21.15.71:6789/0 conn(0x7fc1700062b0 legacy=0x7fc170008750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:56:38.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:38.022+0000 7fc1837fe640 1 --2- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170003730 0x7fc170005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:38.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:38.022+0000 7fc1837fe640 10 monclient(hunting): get_auth_request con 0x7fc170003730 auth_method 0 2024-09-11T01:56:38.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:38.022+0000 7fc1837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:38.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:38.022+0000 7fc1837fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:38.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:38.022+0000 7fc1837fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:56:38.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:38.022+0000 7fc1837fe640 1 -- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170003730 msgr2=0x7fc170005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:56:38.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:38.022+0000 7fc1837fe640 1 -- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170003730 msgr2=0x7fc170005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:56:38.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:38.022+0000 7fc1837fe640 1 --2- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170003730 0x7fc170005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:56:38.025 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:38.022+0000 7fc1837fe640 1 --2- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170003730 0x7fc170005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:56:38.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:38.423+0000 7fc182ffd640 1 -- 172.21.15.12:0/3682125754 >> v1:172.21.15.71:6789/0 conn(0x7fc1700062b0 legacy=0x7fc170008750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:56:38.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:38.424+0000 7fc1837fe640 1 --2- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170003730 0x7fc170005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:38.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:38.424+0000 7fc1837fe640 10 monclient(hunting): get_auth_request con 0x7fc170003730 auth_method 0 2024-09-11T01:56:38.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:38.424+0000 7fc1837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:38.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:38.424+0000 7fc1837fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:38.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:38.424+0000 7fc1837fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:56:38.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:38.424+0000 7fc1837fe640 1 -- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170003730 msgr2=0x7fc170005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:56:38.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:38.424+0000 7fc1837fe640 1 -- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170003730 msgr2=0x7fc170005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:56:38.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:38.424+0000 7fc1837fe640 1 --2- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170003730 0x7fc170005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:56:38.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:38.424+0000 7fc1837fe640 1 --2- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170003730 0x7fc170005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:56:39.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:39.224+0000 7fc182ffd640 1 -- 172.21.15.12:0/3682125754 >> v1:172.21.15.71:6789/0 conn(0x7fc1700062b0 legacy=0x7fc170008750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:56:39.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:39.224+0000 7fc1827fc640 1 -- 172.21.15.12:0/3682125754 DispatchQueue::entry inject delay of 0.002000 2024-09-11T01:56:39.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:39.225+0000 7fc1837fe640 1 --2- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170003730 0x7fc170005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:39.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:39.225+0000 7fc1837fe640 10 monclient(hunting): get_auth_request con 0x7fc170003730 auth_method 0 2024-09-11T01:56:39.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:39.225+0000 7fc1837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:39.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:39.225+0000 7fc1837fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:39.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:39.225+0000 7fc1837fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:56:39.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:39.225+0000 7fc1837fe640 1 -- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170003730 msgr2=0x7fc170005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:56:39.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:39.225+0000 7fc1837fe640 1 -- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170003730 msgr2=0x7fc170005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:56:39.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:39.225+0000 7fc1837fe640 1 --2- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170003730 0x7fc170005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:56:39.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:39.225+0000 7fc1837fe640 1 --2- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170003730 0x7fc170005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:56:40.737 INFO:tasks.rados.rados.0.smithi071.stdout:3993: done (2 left) 2024-09-11T01:56:40.737 INFO:tasks.rados.rados.0.smithi071.stdout:3995: done (1 left) 2024-09-11T01:56:40.737 INFO:tasks.rados.rados.0.smithi071.stdout:3996: done (0 left) 2024-09-11T01:56:40.737 INFO:tasks.rados.rados.0.smithi071.stdout:3997: read oid 43 snap 431 2024-09-11T01:56:40.737 INFO:tasks.rados.rados.0.smithi071.stdout:3997: expect (ObjNum 1249 snap 412 seq_num 1249) 2024-09-11T01:56:40.737 INFO:tasks.rados.rados.0.smithi071.stdout:3998: snap_remove snap 448 2024-09-11T01:56:40.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1817fa640 10 monclient(hunting): tick 2024-09-11T01:56:40.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1817fa640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:56:40.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1817fa640 1 monclient(hunting): continuing hunt 2024-09-11T01:56:40.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1817fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:56:40.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1817fa640 1 -- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170003730 msgr2=0x7fc170005b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:56:40.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1817fa640 1 --2- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170003730 0x7fc170005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:40.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1817fa640 1 -- 172.21.15.12:0/3682125754 >> v1:172.21.15.71:6789/0 conn(0x7fc1700062b0 legacy=0x7fc170008750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:56:40.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1817fa640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-11T01:56:40.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc183fff640 1 -- 172.21.15.12:0/3682125754 reap_dead start 2024-09-11T01:56:40.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1817fa640 1 --2- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 0x7fc170009ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:40.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1817fa640 10 monclient(hunting): picked mon.noname-a con 0x7fc18413d490 addr v2:172.21.15.12:3300/0 2024-09-11T01:56:40.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1817fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc17000a530 addr v1:172.21.15.12:6789/0 2024-09-11T01:56:40.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1817fa640 1 --2- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170005c20 0x7fc17000a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:40.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1817fa640 10 monclient(hunting): picked mon.noname-b con 0x7fc170005c20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:40.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1817fa640 1 -- 172.21.15.12:0/3682125754 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc17000b160 con 0x7fc17000a530 2024-09-11T01:56:40.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1817fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:40.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1817fa640 1 -- 172.21.15.12:0/3682125754 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fc1840a4900 con 0x7fc18413d490 2024-09-11T01:56:40.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1817fa640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:40.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1817fa640 1 -- 172.21.15.12:0/3682125754 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc18406ee50 con 0x7fc170005c20 2024-09-11T01:56:40.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1817fa640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:40.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1837fe640 1 --2- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 0x7fc170009ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:40.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc183fff640 1 --2- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170005c20 0x7fc17000a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:40.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1837fe640 10 monclient(hunting): get_auth_request con 0x7fc18413d490 auth_method 0 2024-09-11T01:56:40.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:40.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1837fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:40.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:40.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc183fff640 10 monclient(hunting): get_auth_request con 0x7fc170005c20 auth_method 0 2024-09-11T01:56:40.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc183fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:40.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc183fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:40.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc183fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:40.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:40.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:40.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc1837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:40.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc183fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:40.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc183fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:40.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.821+0000 7fc183fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:40.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.822+0000 7fc183fff640 10 monclient(hunting): handle_auth_done global_id 304151 payload 274 2024-09-11T01:56:40.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.822+0000 7fc183fff640 1 -- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 msgr2=0x7fc170009ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:40.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.822+0000 7fc183fff640 1 --2- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 0x7fc170009ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:40.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.822+0000 7fc183fff640 1 -- 172.21.15.12:0/3682125754 >> v1:172.21.15.12:6789/0 conn(0x7fc17000a530 legacy=0x7fc170008f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:40.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.822+0000 7fc183fff640 10 monclient: _finish_hunting 0 2024-09-11T01:56:40.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.822+0000 7fc183fff640 1 monclient: found mon.noname-b 2024-09-11T01:56:40.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.822+0000 7fc183fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:40.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.822+0000 7fc183fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:56:40.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.822+0000 7fc183fff640 1 -- 172.21.15.12:0/3682125754 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc18406eb70 con 0x7fc170005c20 2024-09-11T01:56:40.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.822+0000 7fc1837fe640 1 --2- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 0x7fc170009ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:56:40.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.825+0000 7fc183fff640 1 -- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170005c20 msgr2=0x7fc17000a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:40.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.825+0000 7fc183fff640 1 --2- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170005c20 0x7fc17000a930 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc18406f310 tx=0x7fc17c02af60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a7e4843877c2d4f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:40.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.825+0000 7fc1827fc640 1 -- 172.21.15.12:0/3682125754 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17c031070 con 0x7fc170005c20 2024-09-11T01:56:40.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.825+0000 7fc1827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:40.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.825+0000 7fc1827fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:56:40.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.825+0000 7fc1827fc640 10 monclient: dump: 2024-09-11T01:56:40.833 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:40.833 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:40.833 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:40.833 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:40.833 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:40.833 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:40.833 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:40.834 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:40.834 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:40.834 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:40.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.825+0000 7fc1827fc640 10 monclient: _finish_auth 0 2024-09-11T01:56:40.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.825+0000 7fc1827fc640 10 monclient: _check_auth_tickets 2024-09-11T01:56:40.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.825+0000 7fc1827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:40.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.825+0000 7fc1827fc640 1 -- 172.21.15.12:0/3682125754 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc17c02bc80 con 0x7fc170005c20 2024-09-11T01:56:40.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.825+0000 7fc189ec6640 5 monclient: authenticate success, global_id 304151 2024-09-11T01:56:40.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.825+0000 7fc189ec6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:56:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.825+0000 7fc1827fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.825+0000 7fc1827fc640 1 -- 172.21.15.12:0/3682125754 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17c035620 con 0x7fc170005c20 2024-09-11T01:56:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.825+0000 7fc189ec6640 10 monclient: get_monmap_and_config success 2024-09-11T01:56:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.825+0000 7fc189ec6640 4 set_mon_vals no callback set 2024-09-11T01:56:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.825+0000 7fc1827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.825+0000 7fc1827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:56:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.825+0000 7fc1827fc640 10 monclient: dump: 2024-09-11T01:56:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:40.835 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:40.836 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:40.836 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:40.836 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:40.836 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:40.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.826+0000 7fc189ec6640 10 monclient: shutdown 2024-09-11T01:56:40.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.826+0000 7fc189ec6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:40.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.826+0000 7fc189ec6640 1 -- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170005c20 msgr2=0x7fc17000a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:40.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.826+0000 7fc189ec6640 1 --2- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170005c20 0x7fc17000a930 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc18406f310 tx=0x7fc17c02af60 comp rx=0 tx=0).stop 2024-09-11T01:56:40.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.826+0000 7fc189ec6640 1 -- 172.21.15.12:0/3682125754 shutdown_connections 2024-09-11T01:56:40.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.826+0000 7fc189ec6640 1 --2- 172.21.15.12:0/3682125754 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc170005c20 0x7fc17000a930 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:40.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.826+0000 7fc189ec6640 1 --2- 172.21.15.12:0/3682125754 >> v2:172.21.15.12:3300/0 conn(0x7fc18413d490 0x7fc170009ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:40.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.826+0000 7fc189ec6640 1 -- 172.21.15.12:0/3682125754 >> 172.21.15.12:0/3682125754 conn(0x7fc1840a8490 msgr2=0x7fc1840a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:40.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.826+0000 7fc189ec6640 1 -- 172.21.15.12:0/3682125754 shutdown_connections 2024-09-11T01:56:40.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.826+0000 7fc189ec6640 1 -- 172.21.15.12:0/3682125754 wait complete. 2024-09-11T01:56:40.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.827+0000 7fc189ec6640 10 monclient: build_initial_monmap 2024-09-11T01:56:40.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.827+0000 7fc189ec6640 10 monclient: monmap: 2024-09-11T01:56:40.837 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:40.837 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:56:40.837 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:56:40.837 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:56:40.837 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:40.837 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:40.837 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:56:40.837 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:56:40.837 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:56:40.838 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:40.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.827+0000 7fc189ec6640 1 Processor -- start 2024-09-11T01:56:40.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.827+0000 7fc189ec6640 1 -- start start 2024-09-11T01:56:40.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.827+0000 7fc189ec6640 10 monclient: init 2024-09-11T01:56:40.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc189ec6640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:40.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc189ec6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:56:40.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc189ec6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc170005c20 0x7fc1841ccc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:40.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc189ec6640 10 monclient(hunting): picked mon.noname-a con 0x7fc170005c20 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:56:40.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc189ec6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc17000a530 0x7fc1841386f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:40.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc189ec6640 10 monclient(hunting): picked mon.noname-b con 0x7fc17000a530 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:56:40.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc189ec6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc18413d490 0x7fc1841c9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:40.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc189ec6640 10 monclient(hunting): picked mon.noname-c con 0x7fc18413d490 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:40.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc189ec6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:40.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc189ec6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc18406dde0 con 0x7fc170005c20 2024-09-11T01:56:40.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc189ec6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:40.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc189ec6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc18406dc60 con 0x7fc17000a530 2024-09-11T01:56:40.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc189ec6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:40.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc189ec6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fc18406df60 con 0x7fc18413d490 2024-09-11T01:56:40.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc189ec6640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:40.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc189ec6640 10 monclient(hunting): authenticate will time out at 2407.271430s 2024-09-11T01:56:40.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc182ffd640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc17000a530 msgr2=0x7fc1841386f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:56:40.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc183fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc18413d490 0x7fc1841c9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:40.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc182ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc17000a530 0x7fc1841386f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:56:40.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc1837fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc170005c20 0x7fc1841ccc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:40.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc183fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc18413d490 0x7fc1841c9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:38022/0 (socket says 172.21.15.12:38022) 2024-09-11T01:56:40.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc183fff640 1 -- 172.21.15.12:0/3107373827 learned_addr learned my addr 172.21.15.12:0/3107373827 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:40.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc183fff640 10 monclient(hunting): get_auth_request con 0x7fc18413d490 auth_method 0 2024-09-11T01:56:40.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc183fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:40.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc183fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:40.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc183fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:40.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc1837fe640 10 monclient(hunting): get_auth_request con 0x7fc170005c20 auth_method 0 2024-09-11T01:56:40.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc1837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:40.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc1837fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:40.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc1837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:40.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc183fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:40.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc183fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:40.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc183fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:40.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc1837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:40.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc1837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:40.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc1837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:40.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc183fff640 10 monclient(hunting): handle_auth_done global_id 304154 payload 824 2024-09-11T01:56:40.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc183fff640 1 -- 172.21.15.12:0/3107373827 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc17000a530 msgr2=0x7fc1841386f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:56:40.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc183fff640 1 --2- 172.21.15.12:0/3107373827 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc17000a530 0x7fc1841386f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:40.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc183fff640 1 -- 172.21.15.12:0/3107373827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc170005c20 msgr2=0x7fc1841ccc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:40.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc183fff640 1 --2- 172.21.15.12:0/3107373827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc170005c20 0x7fc1841ccc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:40.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc183fff640 10 monclient: _finish_hunting 0 2024-09-11T01:56:40.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc183fff640 1 monclient: found mon.noname-c 2024-09-11T01:56:40.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc183fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:40.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc183fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:56:40.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.828+0000 7fc183fff640 1 -- 172.21.15.12:0/3107373827 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc18409f680 con 0x7fc18413d490 2024-09-11T01:56:40.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.831+0000 7fc183fff640 1 -- 172.21.15.12:0/3107373827 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc18413d490 msgr2=0x7fc1841c9760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:40.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.831+0000 7fc183fff640 1 --2- 172.21.15.12:0/3107373827 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc18413d490 0x7fc1841c9760 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc17c037040 tx=0x7fc17c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:40.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.831+0000 7fc16bfff640 1 -- 172.21.15.12:0/3107373827 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17c043020 con 0x7fc18413d490 2024-09-11T01:56:40.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.831+0000 7fc16bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:40.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.831+0000 7fc16bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:56:40.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.831+0000 7fc16bfff640 10 monclient: dump: 2024-09-11T01:56:40.847 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:40.847 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:40.847 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:40.847 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:40.847 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:40.847 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:40.847 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:40.847 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:40.847 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:40.847 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:40.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.831+0000 7fc16bfff640 10 monclient: _finish_auth 0 2024-09-11T01:56:40.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.831+0000 7fc16bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:56:40.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.831+0000 7fc16bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.831+0000 7fc16bfff640 1 -- 172.21.15.12:0/3107373827 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc17c03e870 con 0x7fc18413d490 2024-09-11T01:56:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.831+0000 7fc189ec6640 5 monclient: authenticate success, global_id 304154 2024-09-11T01:56:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.831+0000 7fc189ec6640 10 monclient: _renew_subs 2024-09-11T01:56:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.831+0000 7fc189ec6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:56:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.831+0000 7fc189ec6640 1 -- 172.21.15.12:0/3107373827 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1841250a0 con 0x7fc18413d490 2024-09-11T01:56:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.831+0000 7fc16bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.831+0000 7fc16bfff640 1 -- 172.21.15.12:0/3107373827 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17c03ec60 con 0x7fc18413d490 2024-09-11T01:56:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.831+0000 7fc16bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.831+0000 7fc16bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:56:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.831+0000 7fc16bfff640 10 monclient: dump: 2024-09-11T01:56:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:40.849 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.831+0000 7fc189ec6640 10 monclient: _renew_subs 2024-09-11T01:56:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.831+0000 7fc181ffb640 4 set_mon_vals no callback set 2024-09-11T01:56:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.831+0000 7fc189ec6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:56:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.831+0000 7fc189ec6640 1 -- 172.21.15.12:0/3107373827 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc184123a80 con 0x7fc18413d490 2024-09-11T01:56:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.832+0000 7fc16bfff640 1 -- 172.21.15.12:0/3107373827 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fc17c03f3f0 con 0x7fc18413d490 2024-09-11T01:56:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.832+0000 7fc189ec6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:40.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.832+0000 7fc189ec6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:40.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.832+0000 7fc189ec6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:56:40.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.833+0000 7fc189ec6640 1 -- 172.21.15.12:0/3107373827 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1840a3680 con 0x7fc18413d490 2024-09-11T01:56:40.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.833+0000 7fc16bfff640 1 --2- 172.21.15.12:0/3107373827 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc17003fad0 0x7fc170041f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:40.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.833+0000 7fc16bfff640 1 -- 172.21.15.12:0/3107373827 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(921..921 src has 889..921) ==== 4787+0+0 (secure 0 0 0) 0x7fc17c074400 con 0x7fc18413d490 2024-09-11T01:56:40.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.833+0000 7fc1837fe640 1 --2- 172.21.15.12:0/3107373827 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc17003fad0 0x7fc170041f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:40.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.833+0000 7fc1837fe640 10 monclient: get_auth_request con 0x7fc17003fad0 auth_method 0 2024-09-11T01:56:40.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.836+0000 7fc1837fe640 1 --2- 172.21.15.12:0/3107373827 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc17003fad0 0x7fc170041f90 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc174005010 tx=0x7fc17400d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:40.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.838+0000 7fc16bfff640 1 -- 172.21.15.12:0/3107373827 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc17c04d030 con 0x7fc18413d490 2024-09-11T01:56:40.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.838+0000 7fc16bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:40.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:40.838+0000 7fc16bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:56:41.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.096+0000 7fc189ec6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:56:41.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.096+0000 7fc189ec6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:56:41.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.096+0000 7fc189ec6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:56:41.100 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.096+0000 7fc189ec6640 1 -- 172.21.15.12:0/3107373827 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc1840698e0 con 0x7fc18413d490 2024-09-11T01:56:41.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.097+0000 7fc16bfff640 1 -- 172.21.15.12:0/3107373827 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fc17c072df0 con 0x7fc18413d490 2024-09-11T01:56:41.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.097+0000 7fc16bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:56:41.101 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.097+0000 7fc16bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:56:41.101 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":254,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:56:41.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.099+0000 7fc189ec6640 1 -- 172.21.15.12:0/3107373827 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc17003fad0 msgr2=0x7fc170041f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:41.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.099+0000 7fc189ec6640 1 --2- 172.21.15.12:0/3107373827 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc17003fad0 0x7fc170041f90 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc174005010 tx=0x7fc17400d040 comp rx=0 tx=0).stop 2024-09-11T01:56:41.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.099+0000 7fc189ec6640 10 monclient: shutdown 2024-09-11T01:56:41.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.099+0000 7fc189ec6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:41.103 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.099+0000 7fc189ec6640 1 -- 172.21.15.12:0/3107373827 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc18413d490 msgr2=0x7fc1841c9760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:41.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.099+0000 7fc189ec6640 1 --2- 172.21.15.12:0/3107373827 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc18413d490 0x7fc1841c9760 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc17c037040 tx=0x7fc17c004a40 comp rx=0 tx=0).stop 2024-09-11T01:56:41.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.099+0000 7fc189ec6640 1 -- 172.21.15.12:0/3107373827 shutdown_connections 2024-09-11T01:56:41.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.099+0000 7fc189ec6640 1 --2- 172.21.15.12:0/3107373827 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc18413d490 0x7fc1841c9760 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.099+0000 7fc189ec6640 1 --2- 172.21.15.12:0/3107373827 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc17000a530 0x7fc1841386f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.099+0000 7fc189ec6640 1 --2- 172.21.15.12:0/3107373827 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fc17003fad0 0x7fc170041f90 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.099+0000 7fc189ec6640 1 --2- 172.21.15.12:0/3107373827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc170005c20 0x7fc1841ccc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.099+0000 7fc189ec6640 1 -- 172.21.15.12:0/3107373827 >> 172.21.15.12:0/3107373827 conn(0x7fc1840a8490 msgr2=0x7fc1840a7d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:41.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.099+0000 7fc189ec6640 1 -- 172.21.15.12:0/3107373827 shutdown_connections 2024-09-11T01:56:41.104 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.099+0000 7fc189ec6640 1 -- 172.21.15.12:0/3107373827 wait complete. 2024-09-11T01:56:41.113 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T01:56:41.113 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-11T01:56:41.113 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:56:41.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.228+0000 7f8f26d8a640 10 monclient: get_monmap_and_config 2024-09-11T01:56:41.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.228+0000 7f8f26d8a640 10 monclient: build_initial_monmap 2024-09-11T01:56:41.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.228+0000 7f8f26d8a640 10 monclient: monmap: 2024-09-11T01:56:41.231 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:41.231 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:41.231 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:56:41.230000+0000 2024-09-11T01:56:41.231 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:56:41.230000+0000 2024-09-11T01:56:41.231 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:41.231 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:41.231 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:56:41.231 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:56:41.232 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:56:41.232 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:56:41.232 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:56:41.232 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:41.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.228+0000 7f8f26d8a640 1 Processor -- start 2024-09-11T01:56:41.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f26d8a640 1 -- start start 2024-09-11T01:56:41.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f26d8a640 10 monclient: init 2024-09-11T01:56:41.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f26d8a640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:41.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f26d8a640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-11T01:56:41.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f26d8a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8f20137e50 0x7f8f2009d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:41.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f26d8a640 10 monclient(hunting): picked mon.noname-b con 0x7f8f20137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:41.234 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f26d8a640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f8f20138230 0x7f8f20138630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:41.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f26d8a640 10 monclient(hunting): picked mon.noname-c con 0x7f8f20138230 addr v2:172.21.15.71:3300/0 2024-09-11T01:56:41.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f26d8a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8f20134cb0 0x7f8f20135120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:41.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f26d8a640 10 monclient(hunting): picked mon.noname-a con 0x7f8f20134cb0 addr v2:172.21.15.12:3300/0 2024-09-11T01:56:41.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f26d8a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:41.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f26d8a640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f8f200a65b0 con 0x7f8f20134cb0 2024-09-11T01:56:41.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f26d8a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:41.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f26d8a640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f8f200a62d0 con 0x7f8f20138230 2024-09-11T01:56:41.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f26d8a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:41.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f26d8a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8f2009bb40 con 0x7f8f20137e50 2024-09-11T01:56:41.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f26d8a640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:41.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f26d8a640 10 monclient(hunting): authenticate will time out at 2407.672550s 2024-09-11T01:56:41.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f24b26640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8f20137e50 0x7f8f2009d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:41.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f25327640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8f20134cb0 0x7f8f20135120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:41.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f1ffff640 1 -- >> v2:172.21.15.71:3300/0 conn(0x7f8f20138230 msgr2=0x7f8f20138630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:56:41.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f24b26640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8f20137e50 0x7f8f2009d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:38028/0 (socket says 172.21.15.12:38028) 2024-09-11T01:56:41.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f24b26640 1 -- 172.21.15.12:0/4249719084 learned_addr learned my addr 172.21.15.12:0/4249719084 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:41.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f24b26640 10 monclient(hunting): get_auth_request con 0x7f8f20137e50 auth_method 0 2024-09-11T01:56:41.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f24b26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:41.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f24b26640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:41.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f24b26640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:41.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f1ffff640 1 --2- 172.21.15.12:0/4249719084 >> v2:172.21.15.71:3300/0 conn(0x7f8f20138230 0x7f8f20138630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:56:41.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f25327640 10 monclient(hunting): get_auth_request con 0x7f8f20134cb0 auth_method 0 2024-09-11T01:56:41.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f25327640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:41.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f25327640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:41.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f25327640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:41.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f24b26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:41.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f24b26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:41.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f24b26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:41.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f25327640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:41.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f25327640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:41.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f25327640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:41.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f25327640 10 monclient(hunting): handle_auth_done global_id 294252 payload 274 2024-09-11T01:56:41.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f25327640 1 -- 172.21.15.12:0/4249719084 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8f20137e50 msgr2=0x7f8f2009d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:41.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f25327640 1 --2- 172.21.15.12:0/4249719084 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8f20137e50 0x7f8f2009d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f25327640 1 -- 172.21.15.12:0/4249719084 >> v2:172.21.15.71:3300/0 conn(0x7f8f20138230 msgr2=0x7f8f20138630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:56:41.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f25327640 1 --2- 172.21.15.12:0/4249719084 >> v2:172.21.15.71:3300/0 conn(0x7f8f20138230 0x7f8f20138630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f25327640 10 monclient: _finish_hunting 0 2024-09-11T01:56:41.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f25327640 1 monclient: found mon.noname-a 2024-09-11T01:56:41.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f25327640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:41.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.229+0000 7f8f25327640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:56:41.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.230+0000 7f8f25327640 1 -- 172.21.15.12:0/4249719084 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f2009c4a0 con 0x7f8f20134cb0 2024-09-11T01:56:41.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.230+0000 7f8f24b26640 1 --2- 172.21.15.12:0/4249719084 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8f20137e50 0x7f8f2009d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:56:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.232+0000 7f8f25327640 1 -- 172.21.15.12:0/4249719084 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20134cb0 msgr2=0x7f8f20135120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.232+0000 7f8f25327640 1 --2- 172.21.15.12:0/4249719084 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20134cb0 0x7f8f20135120 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8f2006fc00 tx=0x7f8f1802a4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20aa4fe3981d2711 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f1f7fe640 1 -- 172.21.15.12:0/4249719084 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f18007950 con 0x7f8f20134cb0 2024-09-11T01:56:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f1f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:56:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f1f7fe640 10 monclient: dump: 2024-09-11T01:56:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:41.244 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f1f7fe640 10 monclient: _finish_auth 0 2024-09-11T01:56:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f1f7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:56:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f1f7fe640 1 -- 172.21.15.12:0/4249719084 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f18007af0 con 0x7f8f20134cb0 2024-09-11T01:56:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f26d8a640 5 monclient: authenticate success, global_id 294252 2024-09-11T01:56:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f26d8a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:56:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f1f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f26d8a640 10 monclient: get_monmap_and_config success 2024-09-11T01:56:41.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f1f7fe640 1 -- 172.21.15.12:0/4249719084 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f18031400 con 0x7f8f20134cb0 2024-09-11T01:56:41.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:41.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f26d8a640 4 set_mon_vals no callback set 2024-09-11T01:56:41.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f1f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:56:41.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f1f7fe640 10 monclient: dump: 2024-09-11T01:56:41.245 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:41.245 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:41.248 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.248 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.248 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:41.248 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:41.248 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:41.249 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:41.249 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:41.249 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:41.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f26d8a640 10 monclient: shutdown 2024-09-11T01:56:41.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f26d8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:41.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f26d8a640 1 -- 172.21.15.12:0/4249719084 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20134cb0 msgr2=0x7f8f20135120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:41.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f26d8a640 1 --2- 172.21.15.12:0/4249719084 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20134cb0 0x7f8f20135120 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8f2006fc00 tx=0x7f8f1802a4d0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f26d8a640 1 -- 172.21.15.12:0/4249719084 shutdown_connections 2024-09-11T01:56:41.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f26d8a640 1 --2- 172.21.15.12:0/4249719084 >> v2:172.21.15.71:3300/0 conn(0x7f8f20138230 0x7f8f20138630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f26d8a640 1 --2- 172.21.15.12:0/4249719084 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20134cb0 0x7f8f20135120 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f26d8a640 1 --2- 172.21.15.12:0/4249719084 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8f20137e50 0x7f8f2009d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f26d8a640 1 -- 172.21.15.12:0/4249719084 >> 172.21.15.12:0/4249719084 conn(0x7f8f200a7020 msgr2=0x7f8f200a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:41.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f26d8a640 1 -- 172.21.15.12:0/4249719084 shutdown_connections 2024-09-11T01:56:41.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.233+0000 7f8f26d8a640 1 -- 172.21.15.12:0/4249719084 wait complete. 2024-09-11T01:56:41.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.234+0000 7f8f26d8a640 10 monclient: build_initial_monmap 2024-09-11T01:56:41.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.234+0000 7f8f26d8a640 10 monclient: monmap: 2024-09-11T01:56:41.250 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:41.250 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:56:41.250 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:56:41.250 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:56:41.250 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:41.250 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:41.250 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:56:41.250 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:56:41.250 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:56:41.250 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:41.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.234+0000 7f8f26d8a640 1 Processor -- start 2024-09-11T01:56:41.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.234+0000 7f8f26d8a640 1 -- start start 2024-09-11T01:56:41.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.234+0000 7f8f26d8a640 10 monclient: init 2024-09-11T01:56:41.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f26d8a640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:41.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f26d8a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:56:41.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f26d8a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20134cb0 0x7f8f201cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:41.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f26d8a640 10 monclient(hunting): picked mon.noname-a con 0x7f8f20134cb0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:56:41.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f26d8a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8f20137e50 0x7f8f201cd910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:41.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f26d8a640 10 monclient(hunting): picked mon.noname-c con 0x7f8f20137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:41.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f26d8a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f20138230 0x7f8f201215e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:41.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f26d8a640 10 monclient(hunting): picked mon.noname-b con 0x7f8f20138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:56:41.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f26d8a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:41.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f26d8a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8f2006f710 con 0x7f8f20134cb0 2024-09-11T01:56:41.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f26d8a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:41.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f26d8a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8f20138b70 con 0x7f8f20138230 2024-09-11T01:56:41.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f26d8a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:41.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f26d8a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8f2006f890 con 0x7f8f20137e50 2024-09-11T01:56:41.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f26d8a640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:41.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f26d8a640 10 monclient(hunting): authenticate will time out at 2407.678336s 2024-09-11T01:56:41.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f24b26640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20134cb0 0x7f8f201cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:41.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f1ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8f20137e50 0x7f8f201cd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:41.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f25327640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f20138230 msgr2=0x7f8f201215e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:56:41.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f1ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8f20137e50 0x7f8f201cd910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:38042/0 (socket says 172.21.15.12:38042) 2024-09-11T01:56:41.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f24b26640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20134cb0 0x7f8f201cd3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50382/0 (socket says 172.21.15.12:50382) 2024-09-11T01:56:41.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f1ffff640 1 -- 172.21.15.12:0/3073228736 learned_addr learned my addr 172.21.15.12:0/3073228736 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:41.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f1ffff640 10 monclient(hunting): get_auth_request con 0x7f8f20137e50 auth_method 0 2024-09-11T01:56:41.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f25327640 1 --2- 172.21.15.12:0/3073228736 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f20138230 0x7f8f201215e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:56:41.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:41.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:41.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:41.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f24b26640 10 monclient(hunting): get_auth_request con 0x7f8f20134cb0 auth_method 0 2024-09-11T01:56:41.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f24b26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:41.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f24b26640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:41.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f24b26640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:41.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:41.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:41.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:41.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f24b26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:41.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f24b26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:41.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f24b26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:41.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f1ffff640 10 monclient(hunting): handle_auth_done global_id 304160 payload 824 2024-09-11T01:56:41.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f1ffff640 1 -- 172.21.15.12:0/3073228736 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f20138230 msgr2=0x7f8f201215e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:56:41.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f1ffff640 1 --2- 172.21.15.12:0/3073228736 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f20138230 0x7f8f201215e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f1ffff640 1 -- 172.21.15.12:0/3073228736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20134cb0 msgr2=0x7f8f201cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:41.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f1ffff640 1 --2- 172.21.15.12:0/3073228736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20134cb0 0x7f8f201cd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f1ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:56:41.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f1ffff640 1 monclient: found mon.noname-c 2024-09-11T01:56:41.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:41.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f1ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:56:41.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f1ffff640 1 -- 172.21.15.12:0/3073228736 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f201253b0 con 0x7f8f20137e50 2024-09-11T01:56:41.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.235+0000 7f8f24b26640 1 --2- 172.21.15.12:0/3073228736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20134cb0 0x7f8f201cd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:56:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.238+0000 7f8f1ffff640 1 -- 172.21.15.12:0/3073228736 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8f20137e50 msgr2=0x7f8f201cd910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.238+0000 7f8f1ffff640 1 --2- 172.21.15.12:0/3073228736 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8f20137e50 0x7f8f201cd910 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c00b550 tx=0x7f8f0c00ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.238+0000 7f8f1d7fa640 1 -- 172.21.15.12:0/3073228736 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c013020 con 0x7f8f20137e50 2024-09-11T01:56:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.238+0000 7f8f1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.238+0000 7f8f1d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:56:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.238+0000 7f8f1d7fa640 10 monclient: dump: 2024-09-11T01:56:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:41.256 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.238+0000 7f8f1d7fa640 10 monclient: _finish_auth 0 2024-09-11T01:56:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.238+0000 7f8f1d7fa640 10 monclient: _check_auth_tickets 2024-09-11T01:56:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.238+0000 7f8f1d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.238+0000 7f8f26d8a640 5 monclient: authenticate success, global_id 304160 2024-09-11T01:56:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.238+0000 7f8f26d8a640 10 monclient: _renew_subs 2024-09-11T01:56:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.238+0000 7f8f26d8a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:56:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.238+0000 7f8f26d8a640 1 -- 172.21.15.12:0/3073228736 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f201247c0 con 0x7f8f20137e50 2024-09-11T01:56:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.238+0000 7f8f1d7fa640 1 -- 172.21.15.12:0/3073228736 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f0c005070 con 0x7f8f20137e50 2024-09-11T01:56:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.238+0000 7f8f26d8a640 10 monclient: _renew_subs 2024-09-11T01:56:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.238+0000 7f8f26d8a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:56:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.238+0000 7f8f26d8a640 1 -- 172.21.15.12:0/3073228736 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f20123050 con 0x7f8f20137e50 2024-09-11T01:56:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.238+0000 7f8f1d7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.238+0000 7f8f1d7fa640 1 -- 172.21.15.12:0/3073228736 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c005480 con 0x7f8f20137e50 2024-09-11T01:56:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.238+0000 7f8f1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.238+0000 7f8f1d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:56:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.238+0000 7f8f1d7fa640 10 monclient: dump: 2024-09-11T01:56:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:41.258 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.238+0000 7f8f1effd640 4 set_mon_vals no callback set 2024-09-11T01:56:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.239+0000 7f8f1d7fa640 1 -- 172.21.15.12:0/3073228736 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f8f0c005620 con 0x7f8f20137e50 2024-09-11T01:56:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.239+0000 7f8f26d8a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.239+0000 7f8f26d8a640 1 --2- 172.21.15.12:0/3073228736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20124b10 0x7f8f20121b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.239+0000 7f8f26d8a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.239+0000 7f8f26d8a640 1 -- 172.21.15.12:0/3073228736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8f2006a060 con 0x7f8f20124b10 2024-09-11T01:56:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.239+0000 7f8f1d7fa640 1 --2- 172.21.15.12:0/3073228736 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8f0403e1c0 0x7f8f04040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.239+0000 7f8f1d7fa640 1 -- 172.21.15.12:0/3073228736 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(921..921 src has 889..921) ==== 4787+0+0 (secure 0 0 0) 0x7f8f0c052c40 con 0x7f8f20137e50 2024-09-11T01:56:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.239+0000 7f8f24b26640 1 --2- 172.21.15.12:0/3073228736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20124b10 0x7f8f20121b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.239+0000 7f8f24b26640 10 monclient: get_auth_request con 0x7f8f20124b10 auth_method 0 2024-09-11T01:56:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.239+0000 7f8f24b26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.239+0000 7f8f24b26640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.239+0000 7f8f24b26640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.239+0000 7f8f24b26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.239+0000 7f8f24b26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.239+0000 7f8f24b26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.239+0000 7f8f25327640 1 --2- 172.21.15.12:0/3073228736 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8f0403e1c0 0x7f8f04040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.239+0000 7f8f25327640 10 monclient: get_auth_request con 0x7f8f0403e1c0 auth_method 0 2024-09-11T01:56:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.239+0000 7f8f24b26640 10 monclient(hunting): handle_auth_done global_id 294258 payload 824 2024-09-11T01:56:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.239+0000 7f8f24b26640 1 -- 172.21.15.12:0/3073228736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8f2009d160 con 0x7f8f20124b10 2024-09-11T01:56:41.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.241+0000 7f8f24b26640 1 -- 172.21.15.12:0/3073228736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20124b10 msgr2=0x7f8f20121b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:41.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.241+0000 7f8f24b26640 1 --2- 172.21.15.12:0/3073228736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20124b10 0x7f8f20121b20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8f14006ca0 tx=0x7f8f1400f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:41.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.241+0000 7f8f1d7fa640 1 -- 172.21.15.12:0/3073228736 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f14014650 con 0x7f8f20124b10 2024-09-11T01:56:41.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.241+0000 7f8f1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:41.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.241+0000 7f8f1d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:56:41.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.241+0000 7f8f1d7fa640 10 monclient: dump: 2024-09-11T01:56:41.263 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:41.263 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:41.263 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.263 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.263 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:41.263 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:41.263 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:41.264 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.242+0000 7f8f25327640 1 --2- 172.21.15.12:0/3073228736 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8f0403e1c0 0x7f8f04040680 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f8f200a8360 tx=0x7f8f1802f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.242+0000 7f8f1d7fa640 1 -- 172.21.15.12:0/3073228736 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8f14014650 con 0x7f8f20124b10 2024-09-11T01:56:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.242+0000 7f8f1d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.242+0000 7f8f1d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:56:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.242+0000 7f8f1d7fa640 1 -- 172.21.15.12:0/3073228736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20124b10 msgr2=0x7f8f20121b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.242+0000 7f8f1d7fa640 1 --2- 172.21.15.12:0/3073228736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20124b10 0x7f8f20121b20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8f14006ca0 tx=0x7f8f1400f610 comp rx=0 tx=0).stop 2024-09-11T01:56:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.242+0000 7f8f1d7fa640 1 -- 172.21.15.12:0/3073228736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20124b10 msgr2=0x7f8f20121b20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:56:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.242+0000 7f8f1d7fa640 1 --2- 172.21.15.12:0/3073228736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20124b10 0x7f8f20121b20 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.248+0000 7f8f26d8a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:56:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.248+0000 7f8f26d8a640 1 --2- 172.21.15.12:0/3073228736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20122060 0x7f8f20122470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.248+0000 7f8f26d8a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.248+0000 7f8f26d8a640 1 -- 172.21.15.12:0/3073228736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8f201229b0 con 0x7f8f20122060 2024-09-11T01:56:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.248+0000 7f8f24b26640 1 --2- 172.21.15.12:0/3073228736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20122060 0x7f8f20122470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.248+0000 7f8f24b26640 10 monclient: get_auth_request con 0x7f8f20122060 auth_method 0 2024-09-11T01:56:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.248+0000 7f8f24b26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.248+0000 7f8f24b26640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.248+0000 7f8f24b26640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.248+0000 7f8f24b26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.248+0000 7f8f24b26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.248+0000 7f8f24b26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:41.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.248+0000 7f8f24b26640 10 monclient(hunting): handle_auth_done global_id 294261 payload 824 2024-09-11T01:56:41.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.248+0000 7f8f24b26640 1 -- 172.21.15.12:0/3073228736 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8f201cde50 con 0x7f8f20122060 2024-09-11T01:56:41.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.250+0000 7f8f24b26640 1 -- 172.21.15.12:0/3073228736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20122060 msgr2=0x7f8f20122470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:41.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.250+0000 7f8f24b26640 1 --2- 172.21.15.12:0/3073228736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20122060 0x7f8f20122470 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8f14006ca0 tx=0x7f8f140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:41.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.251+0000 7f8f1d7fa640 1 -- 172.21.15.12:0/3073228736 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f14022dc0 con 0x7f8f20122060 2024-09-11T01:56:41.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.251+0000 7f8f1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:41.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.251+0000 7f8f1d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:56:41.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.251+0000 7f8f1d7fa640 10 monclient: dump: 2024-09-11T01:56:41.267 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:41.267 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:41.267 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.267 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.267 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:41.267 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:41.267 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:41.267 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:41.267 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:41.267 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:41.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.251+0000 7f8f1d7fa640 1 -- 172.21.15.12:0/3073228736 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4699 (secure 0 0 0) 0x7f8f1402c420 con 0x7f8f20122060 2024-09-11T01:56:41.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.251+0000 7f8f1d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:56:41.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.251+0000 7f8f1d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:56:41.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.251+0000 7f8f1d7fa640 1 -- 172.21.15.12:0/3073228736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20122060 msgr2=0x7f8f20122470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:41.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.251+0000 7f8f1d7fa640 1 --2- 172.21.15.12:0/3073228736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20122060 0x7f8f20122470 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8f14006ca0 tx=0x7f8f140046c0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.251+0000 7f8f1d7fa640 1 -- 172.21.15.12:0/3073228736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20122060 msgr2=0x7f8f20122470 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:56:41.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.251+0000 7f8f1d7fa640 1 --2- 172.21.15.12:0/3073228736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f20122060 0x7f8f20122470 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8f14006ca0 tx=0x7f8f140046c0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.268 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:56:41.268 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:56:41.268 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:56:41.268 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:56:41.268 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 254, 2024-09-11T01:56:41.269 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 81205, 2024-09-11T01:56:41.269 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:56:41.269 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:56:41.269 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:56:41.269 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:41.269 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:56:41.269 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:56:41.269 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:56:41.269 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:56:41.269 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:56:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:56:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:56:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:56:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:56:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:56:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:56:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:56:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:56:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:56:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:56:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:56:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:56:41.270 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:56:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:56:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:56:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:56:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:56:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:56:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:56:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:56:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:56:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:56:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:56:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:41.271 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:56:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:56:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:56:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:56:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:56:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:56:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:56:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:56:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:56:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:56:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:56:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:56:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:56:41.272 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:56:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:56:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:56:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:56:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:56:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:56:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:56:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:56:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:56:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:56:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:56:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:56:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:56:41.273 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:56:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:56:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:56:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:56:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:56:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:56:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:56:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:56:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:56:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:56:41.274 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:56:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:56:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:56:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:56:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:56:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:56:41.275 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:56:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:56:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:56:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:56:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:56:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:56:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:56:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:41.276 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:56:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:56:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:56:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:56:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:56:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:56:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:56:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:41.277 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:41.278 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:56:41.278 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:56:41.278 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:56:41.278 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:56:41.278 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:41.278 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:56:41.278 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:56:41.278 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:41.278 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:41.278 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:41.278 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:56:41.278 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:56:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:56:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:56:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:56:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:56:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:56:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:41.279 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:56:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:56:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:56:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:56:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:56:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:56:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:56:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:56:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:56:41.280 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:41.281 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:41.281 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:56:41.281 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:41.281 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:56:41.281 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:56:41.281 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:56:41.281 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:41.281 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:41.281 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:41.281 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:56:41.281 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:56:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.251+0000 7f8f26d8a640 1 -- 172.21.15.12:0/3073228736 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8f0403e1c0 msgr2=0x7f8f04040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.251+0000 7f8f26d8a640 1 --2- 172.21.15.12:0/3073228736 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8f0403e1c0 0x7f8f04040680 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f8f200a8360 tx=0x7f8f1802f000 comp rx=0 tx=0).stop 2024-09-11T01:56:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.251+0000 7f8f26d8a640 10 monclient: shutdown 2024-09-11T01:56:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.251+0000 7f8f26d8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.251+0000 7f8f26d8a640 1 -- 172.21.15.12:0/3073228736 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8f20137e50 msgr2=0x7f8f201cd910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.251+0000 7f8f26d8a640 1 --2- 172.21.15.12:0/3073228736 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8f20137e50 0x7f8f201cd910 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c00b550 tx=0x7f8f0c00ba10 comp rx=0 tx=0).stop 2024-09-11T01:56:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.251+0000 7f8f25327640 1 -- 172.21.15.12:0/3073228736 reap_dead start 2024-09-11T01:56:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.251+0000 7f8f25327640 1 -- 172.21.15.12:0/3073228736 reap_dead start 2024-09-11T01:56:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.251+0000 7f8f26d8a640 1 -- 172.21.15.12:0/3073228736 shutdown_connections 2024-09-11T01:56:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.251+0000 7f8f26d8a640 1 -- 172.21.15.12:0/3073228736 >> 172.21.15.12:0/3073228736 conn(0x7f8f200a7020 msgr2=0x7f8f20135970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.251+0000 7f8f26d8a640 1 -- 172.21.15.12:0/3073228736 shutdown_connections 2024-09-11T01:56:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.251+0000 7f8f26d8a640 1 -- 172.21.15.12:0/3073228736 wait complete. 2024-09-11T01:56:41.284 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:56:41.284 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T01:56:41.284 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:56:41.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.383+0000 7f8da89b4640 10 monclient: get_monmap_and_config 2024-09-11T01:56:41.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.383+0000 7f8da89b4640 10 monclient: build_initial_monmap 2024-09-11T01:56:41.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.383+0000 7f8da89b4640 10 monclient: monmap: 2024-09-11T01:56:41.386 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:41.386 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:41.387 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:56:41.385697+0000 2024-09-11T01:56:41.387 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:56:41.385697+0000 2024-09-11T01:56:41.387 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:41.387 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:41.387 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:56:41.387 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:56:41.387 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:56:41.387 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:56:41.387 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:56:41.387 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:41.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.384+0000 7f8da89b4640 1 Processor -- start 2024-09-11T01:56:41.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.384+0000 7f8da89b4640 1 -- start start 2024-09-11T01:56:41.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.384+0000 7f8da89b4640 10 monclient: init 2024-09-11T01:56:41.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da89b4640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:41.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da89b4640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-11T01:56:41.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da89b4640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8da40a2e20 0x7f8da406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:41.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da89b4640 10 monclient(hunting): picked mon.noname-a con 0x7f8da40a2e20 addr v2:172.21.15.12:3300/0 2024-09-11T01:56:41.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da89b4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da40a4970 0x7f8da40a3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:41.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da89b4640 10 monclient(hunting): picked mon.noname-b con 0x7f8da40a4970 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:41.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da89b4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8da406f910 addr v1:172.21.15.71:6789/0 2024-09-11T01:56:41.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da89b4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8da406f4f0 con 0x7f8da406f910 2024-09-11T01:56:41.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da89b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:41.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da89b4640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f8da406ef30 con 0x7f8da40a2e20 2024-09-11T01:56:41.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da89b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:41.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da89b4640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8da406ec50 con 0x7f8da40a4970 2024-09-11T01:56:41.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da89b4640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:41.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da89b4640 10 monclient(hunting): authenticate will time out at 2407.828298s 2024-09-11T01:56:41.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da1d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da40a4970 0x7f8da40a3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:41.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da259c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8da40a2e20 0x7f8da406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:41.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da2d9d640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f8da406f910 legacy=0x7f8da4137fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:56:41.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da1d9b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da40a4970 0x7f8da40a3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:38058/0 (socket says 172.21.15.12:38058) 2024-09-11T01:56:41.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da259c640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8da40a2e20 0x7f8da406ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50408/0 (socket says 172.21.15.12:50408) 2024-09-11T01:56:41.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da1d9b640 1 -- 172.21.15.12:0/658051459 learned_addr learned my addr 172.21.15.12:0/658051459 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:41.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da1d9b640 10 monclient(hunting): get_auth_request con 0x7f8da40a4970 auth_method 0 2024-09-11T01:56:41.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:41.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:41.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:41.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da259c640 10 monclient(hunting): get_auth_request con 0x7f8da40a2e20 auth_method 0 2024-09-11T01:56:41.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:41.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:41.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:41.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:41.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:41.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:41.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:41.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:41.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:41.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da259c640 10 monclient(hunting): handle_auth_done global_id 294264 payload 274 2024-09-11T01:56:41.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da259c640 1 -- 172.21.15.12:0/658051459 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da40a4970 msgr2=0x7f8da40a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:41.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da259c640 1 --2- 172.21.15.12:0/658051459 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da40a4970 0x7f8da40a3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da259c640 1 -- 172.21.15.12:0/658051459 >> v1:172.21.15.71:6789/0 conn(0x7f8da406f910 legacy=0x7f8da4137fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:56:41.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da259c640 10 monclient: _finish_hunting 0 2024-09-11T01:56:41.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da259c640 1 monclient: found mon.noname-a 2024-09-11T01:56:41.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:41.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:56:41.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da259c640 1 -- 172.21.15.12:0/658051459 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8da40a3740 con 0x7f8da40a2e20 2024-09-11T01:56:41.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.385+0000 7f8da1d9b640 1 --2- 172.21.15.12:0/658051459 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da40a4970 0x7f8da40a3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:56:41.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.388+0000 7f8da259c640 1 -- 172.21.15.12:0/658051459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8da40a2e20 msgr2=0x7f8da406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.388+0000 7f8da259c640 1 --2- 172.21.15.12:0/658051459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8da40a2e20 0x7f8da406ded0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8d940097a0 tx=0x7f8d94029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38087cf48a65f8f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.388+0000 7f8da159a640 1 -- 172.21.15.12:0/658051459 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d94037070 con 0x7f8da40a2e20 2024-09-11T01:56:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.388+0000 7f8da159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.388+0000 7f8da159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:56:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.388+0000 7f8da159a640 10 monclient: dump: 2024-09-11T01:56:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:41.402 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:41.402 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:41.402 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:41.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.388+0000 7f8da159a640 10 monclient: _finish_auth 0 2024-09-11T01:56:41.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.388+0000 7f8da159a640 10 monclient: _check_auth_tickets 2024-09-11T01:56:41.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.388+0000 7f8da159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:41.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.388+0000 7f8da159a640 1 -- 172.21.15.12:0/658051459 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d94032440 con 0x7f8da40a2e20 2024-09-11T01:56:41.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.388+0000 7f8da89b4640 5 monclient: authenticate success, global_id 294264 2024-09-11T01:56:41.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.388+0000 7f8da89b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:56:41.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.388+0000 7f8da159a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:41.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.388+0000 7f8da89b4640 10 monclient: get_monmap_and_config success 2024-09-11T01:56:41.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.388+0000 7f8da159a640 1 -- 172.21.15.12:0/658051459 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d94032810 con 0x7f8da40a2e20 2024-09-11T01:56:41.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.388+0000 7f8da89b4640 4 set_mon_vals no callback set 2024-09-11T01:56:41.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.388+0000 7f8da159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:41.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.388+0000 7f8da159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:56:41.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.388+0000 7f8da159a640 10 monclient: dump: 2024-09-11T01:56:41.403 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:41.403 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:41.403 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.403 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.403 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:41.403 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:41.403 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:41.403 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:41.404 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.389+0000 7f8da89b4640 10 monclient: shutdown 2024-09-11T01:56:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.389+0000 7f8da89b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.389+0000 7f8da89b4640 1 -- 172.21.15.12:0/658051459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8da40a2e20 msgr2=0x7f8da406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.389+0000 7f8da89b4640 1 --2- 172.21.15.12:0/658051459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8da40a2e20 0x7f8da406ded0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8d940097a0 tx=0x7f8d94029fd0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.389+0000 7f8da89b4640 1 -- 172.21.15.12:0/658051459 shutdown_connections 2024-09-11T01:56:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.389+0000 7f8da89b4640 1 --2- 172.21.15.12:0/658051459 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da40a4970 0x7f8da40a3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.389+0000 7f8da89b4640 1 --2- 172.21.15.12:0/658051459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8da40a2e20 0x7f8da406ded0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.389+0000 7f8da89b4640 1 -- 172.21.15.12:0/658051459 >> 172.21.15.12:0/658051459 conn(0x7f8da40a8600 msgr2=0x7f8da40a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.389+0000 7f8da89b4640 1 -- 172.21.15.12:0/658051459 shutdown_connections 2024-09-11T01:56:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.389+0000 7f8da89b4640 1 -- 172.21.15.12:0/658051459 wait complete. 2024-09-11T01:56:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.390+0000 7f8da89b4640 10 monclient: build_initial_monmap 2024-09-11T01:56:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.390+0000 7f8da89b4640 10 monclient: monmap: 2024-09-11T01:56:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:56:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:56:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:56:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:56:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:56:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:56:41.405 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:41.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.390+0000 7f8da89b4640 1 Processor -- start 2024-09-11T01:56:41.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.390+0000 7f8da89b4640 1 -- start start 2024-09-11T01:56:41.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.390+0000 7f8da89b4640 10 monclient: init 2024-09-11T01:56:41.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da89b4640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:41.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da89b4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:56:41.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da89b4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8da406f910 0x7f8da41cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:41.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da89b4640 10 monclient(hunting): picked mon.noname-a con 0x7f8da406f910 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:56:41.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da89b4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8da40a2e20 0x7f8da41cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:41.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da89b4640 10 monclient(hunting): picked mon.noname-b con 0x7f8da40a2e20 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:56:41.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da89b4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da40a4970 0x7f8da4124e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:41.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da89b4640 10 monclient(hunting): picked mon.noname-c con 0x7f8da40a4970 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da89b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da89b4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8da4138860 con 0x7f8da406f910 2024-09-11T01:56:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da89b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da89b4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8da41386e0 con 0x7f8da40a2e20 2024-09-11T01:56:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da89b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da89b4640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8da41389e0 con 0x7f8da40a4970 2024-09-11T01:56:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da89b4640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da89b4640 10 monclient(hunting): authenticate will time out at 2407.834186s 2024-09-11T01:56:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8da406f910 0x7f8da41cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da2d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da40a4970 0x7f8da4124e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8da406f910 0x7f8da41cd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50416/0 (socket says 172.21.15.12:50416) 2024-09-11T01:56:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da2d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da40a4970 0x7f8da4124e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:38066/0 (socket says 172.21.15.12:38066) 2024-09-11T01:56:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da259c640 1 -- 172.21.15.12:0/4124414333 learned_addr learned my addr 172.21.15.12:0/4124414333 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da259c640 10 monclient(hunting): get_auth_request con 0x7f8da406f910 auth_method 0 2024-09-11T01:56:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da2d9d640 10 monclient(hunting): get_auth_request con 0x7f8da40a4970 auth_method 0 2024-09-11T01:56:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da1d9b640 1 -- 172.21.15.12:0/4124414333 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8da40a2e20 msgr2=0x7f8da41cd9b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:56:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da1d9b640 1 --2- 172.21.15.12:0/4124414333 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8da40a2e20 0x7f8da41cd9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:56:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:41.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:41.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:41.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:41.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:41.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da259c640 10 monclient(hunting): handle_auth_done global_id 294267 payload 824 2024-09-11T01:56:41.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da259c640 1 -- 172.21.15.12:0/4124414333 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da40a4970 msgr2=0x7f8da4124e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:41.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da259c640 1 --2- 172.21.15.12:0/4124414333 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da40a4970 0x7f8da4124e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da259c640 1 -- 172.21.15.12:0/4124414333 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8da40a2e20 msgr2=0x7f8da41cd9b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:56:41.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da259c640 1 --2- 172.21.15.12:0/4124414333 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8da40a2e20 0x7f8da41cd9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da259c640 10 monclient: _finish_hunting 0 2024-09-11T01:56:41.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da259c640 1 monclient: found mon.noname-a 2024-09-11T01:56:41.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:41.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:56:41.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.391+0000 7f8da259c640 1 -- 172.21.15.12:0/4124414333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8da40a6220 con 0x7f8da406f910 2024-09-11T01:56:41.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.393+0000 7f8da259c640 1 -- 172.21.15.12:0/4124414333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8da406f910 msgr2=0x7f8da41cd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:41.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.393+0000 7f8da259c640 1 --2- 172.21.15.12:0/4124414333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8da406f910 0x7f8da41cd470 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8d94000c00 tx=0x7f8d940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:41.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.394+0000 7f8d92ffd640 1 -- 172.21.15.12:0/4124414333 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d94037040 con 0x7f8da406f910 2024-09-11T01:56:41.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.394+0000 7f8d92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:41.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.394+0000 7f8d92ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:56:41.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.394+0000 7f8d92ffd640 10 monclient: dump: 2024-09-11T01:56:41.410 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:41.410 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:41.410 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:41.411 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.394+0000 7f8d92ffd640 10 monclient: _finish_auth 0 2024-09-11T01:56:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.394+0000 7f8d92ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:56:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.394+0000 7f8d92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.394+0000 7f8da89b4640 5 monclient: authenticate success, global_id 294267 2024-09-11T01:56:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.394+0000 7f8d92ffd640 1 -- 172.21.15.12:0/4124414333 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d94042070 con 0x7f8da406f910 2024-09-11T01:56:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.394+0000 7f8da89b4640 10 monclient: _renew_subs 2024-09-11T01:56:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.394+0000 7f8da89b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.394+0000 7f8da89b4640 1 -- 172.21.15.12:0/4124414333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8da409f610 con 0x7f8da406f910 2024-09-11T01:56:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.394+0000 7f8d92ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.394+0000 7f8d92ffd640 1 -- 172.21.15.12:0/4124414333 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d9403bb00 con 0x7f8da406f910 2024-09-11T01:56:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.394+0000 7f8d92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.394+0000 7f8d92ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:56:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.394+0000 7f8d92ffd640 10 monclient: dump: 2024-09-11T01:56:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:41.414 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:41.414 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:41.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.394+0000 7f8da89b4640 10 monclient: _renew_subs 2024-09-11T01:56:41.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.394+0000 7f8da89b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:41.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.394+0000 7f8da0d99640 4 set_mon_vals no callback set 2024-09-11T01:56:41.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.394+0000 7f8da89b4640 1 -- 172.21.15.12:0/4124414333 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8da4128b40 con 0x7f8da406f910 2024-09-11T01:56:41.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.394+0000 7f8da89b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:41.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.394+0000 7f8da89b4640 1 --2- 172.21.15.12:0/4124414333 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da4125b70 0x7f8da402d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:41.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.394+0000 7f8da89b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:41.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.394+0000 7f8da89b4640 1 -- 172.21.15.12:0/4124414333 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8da406a060 con 0x7f8da4125b70 2024-09-11T01:56:41.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.395+0000 7f8da1d9b640 1 --2- 172.21.15.12:0/4124414333 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da4125b70 0x7f8da402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:41.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.395+0000 7f8da1d9b640 10 monclient: get_auth_request con 0x7f8da4125b70 auth_method 0 2024-09-11T01:56:41.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.395+0000 7f8da1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.395+0000 7f8da1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.395+0000 7f8da1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.395+0000 7f8d92ffd640 1 -- 172.21.15.12:0/4124414333 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f8d9403a030 con 0x7f8da406f910 2024-09-11T01:56:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.395+0000 7f8da1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.395+0000 7f8da1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.395+0000 7f8da1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.395+0000 7f8da1d9b640 10 monclient(hunting): handle_auth_done global_id 304169 payload 824 2024-09-11T01:56:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.395+0000 7f8da1d9b640 1 -- 172.21.15.12:0/4124414333 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8da406ded0 con 0x7f8da4125b70 2024-09-11T01:56:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.395+0000 7f8d92ffd640 1 --2- 172.21.15.12:0/4124414333 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8d8403e240 0x7f8d84040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.395+0000 7f8d92ffd640 1 -- 172.21.15.12:0/4124414333 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(921..921 src has 889..921) ==== 4787+0+0 (secure 0 0 0) 0x7f8d94072b20 con 0x7f8da406f910 2024-09-11T01:56:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.395+0000 7f8da2d9d640 1 --2- 172.21.15.12:0/4124414333 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8d8403e240 0x7f8d84040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.395+0000 7f8da2d9d640 10 monclient: get_auth_request con 0x7f8d8403e240 auth_method 0 2024-09-11T01:56:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.397+0000 7f8da1d9b640 1 -- 172.21.15.12:0/4124414333 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da4125b70 msgr2=0x7f8da402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:41.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.397+0000 7f8da1d9b640 1 --2- 172.21.15.12:0/4124414333 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da4125b70 0x7f8da402d440 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8da4127a60 tx=0x7f8d9800ec00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:41.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.397+0000 7f8d92ffd640 1 -- 172.21.15.12:0/4124414333 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d98017070 con 0x7f8da4125b70 2024-09-11T01:56:41.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.397+0000 7f8d92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:41.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.397+0000 7f8d92ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:56:41.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.397+0000 7f8d92ffd640 10 monclient: dump: 2024-09-11T01:56:41.416 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:41.416 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:41.416 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.416 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.416 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:41.416 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:41.416 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:41.417 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:41.417 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:41.417 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:41.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.398+0000 7f8d92ffd640 1 -- 172.21.15.12:0/4124414333 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8d9800cb90 con 0x7f8da4125b70 2024-09-11T01:56:41.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.398+0000 7f8d92ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:41.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.398+0000 7f8d92ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:56:41.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.398+0000 7f8d92ffd640 1 -- 172.21.15.12:0/4124414333 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da4125b70 msgr2=0x7f8da402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:41.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.398+0000 7f8d92ffd640 1 --2- 172.21.15.12:0/4124414333 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da4125b70 0x7f8da402d440 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8da4127a60 tx=0x7f8d9800ec00 comp rx=0 tx=0).stop 2024-09-11T01:56:41.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.398+0000 7f8d92ffd640 1 -- 172.21.15.12:0/4124414333 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da4125b70 msgr2=0x7f8da402d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:56:41.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.398+0000 7f8d92ffd640 1 --2- 172.21.15.12:0/4124414333 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da4125b70 0x7f8da402d440 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8da4127a60 tx=0x7f8d9800ec00 comp rx=0 tx=0).stop 2024-09-11T01:56:41.417 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.398+0000 7f8da2d9d640 1 --2- 172.21.15.12:0/4124414333 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8d8403e240 0x7f8d84040700 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c003bc0 tx=0x7f8d9c00d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:41.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.404+0000 7f8da89b4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:56:41.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.404+0000 7f8da89b4640 1 --2- 172.21.15.12:0/4124414333 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da4125f50 0x7f8da402d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:41.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.404+0000 7f8da89b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:41.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.404+0000 7f8da89b4640 1 -- 172.21.15.12:0/4124414333 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8da4138100 con 0x7f8da4125f50 2024-09-11T01:56:41.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.404+0000 7f8da1d9b640 1 --2- 172.21.15.12:0/4124414333 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da4125f50 0x7f8da402d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:41.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.404+0000 7f8da1d9b640 10 monclient: get_auth_request con 0x7f8da4125f50 auth_method 0 2024-09-11T01:56:41.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.404+0000 7f8da1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:41.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.404+0000 7f8da1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:41.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.404+0000 7f8da1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:41.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.404+0000 7f8da1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:41.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.404+0000 7f8da1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:41.421 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.404+0000 7f8da1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:41.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.404+0000 7f8da1d9b640 10 monclient(hunting): handle_auth_done global_id 304172 payload 824 2024-09-11T01:56:41.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.404+0000 7f8da1d9b640 1 -- 172.21.15.12:0/4124414333 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8da40a0ef0 con 0x7f8da4125f50 2024-09-11T01:56:41.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.406+0000 7f8da1d9b640 1 -- 172.21.15.12:0/4124414333 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da4125f50 msgr2=0x7f8da402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:41.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.406+0000 7f8da1d9b640 1 --2- 172.21.15.12:0/4124414333 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da4125f50 0x7f8da402d980 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8da40a4e30 tx=0x7f8d98004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:41.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.406+0000 7f8d92ffd640 1 -- 172.21.15.12:0/4124414333 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d98017070 con 0x7f8da4125f50 2024-09-11T01:56:41.422 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.406+0000 7f8d92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:41.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.406+0000 7f8d92ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:56:41.423 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.406+0000 7f8d92ffd640 10 monclient: dump: 2024-09-11T01:56:41.423 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:41.423 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:41.423 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.423 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.424 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:41.424 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:41.424 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:41.424 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:41.424 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:41.424 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:41.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.407+0000 7f8d92ffd640 1 -- 172.21.15.12:0/4124414333 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4864 (secure 0 0 0) 0x7f8d98006770 con 0x7f8da4125f50 2024-09-11T01:56:41.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.407+0000 7f8d92ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:56:41.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.407+0000 7f8d92ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:56:41.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.407+0000 7f8d92ffd640 1 -- 172.21.15.12:0/4124414333 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da4125f50 msgr2=0x7f8da402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:41.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.407+0000 7f8d92ffd640 1 --2- 172.21.15.12:0/4124414333 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da4125f50 0x7f8da402d980 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8da40a4e30 tx=0x7f8d98004730 comp rx=0 tx=0).stop 2024-09-11T01:56:41.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.407+0000 7f8d92ffd640 1 -- 172.21.15.12:0/4124414333 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da4125f50 msgr2=0x7f8da402d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:56:41.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.407+0000 7f8d92ffd640 1 --2- 172.21.15.12:0/4124414333 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8da4125f50 0x7f8da402d980 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8da40a4e30 tx=0x7f8d98004730 comp rx=0 tx=0).stop 2024-09-11T01:56:41.426 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:56:41.426 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:56:41.426 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:56:41.427 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:56:41.427 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 254, 2024-09-11T01:56:41.427 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 36362, 2024-09-11T01:56:41.427 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:56:41.427 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:56:41.427 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:56:41.428 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:41.428 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 1, 2024-09-11T01:56:41.428 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:56:41.428 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:56:41.428 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:56:41.428 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:56:41.429 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:56:41.429 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:56:41.429 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:56:41.429 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:56:41.429 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:56:41.429 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:56:41.430 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:56:41.430 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:56:41.430 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:56:41.430 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:56:41.430 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:41.431 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:56:41.431 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:56:41.431 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:56:41.431 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:56:41.431 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:56:41.432 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:56:41.432 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:56:41.432 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:56:41.432 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:56:41.432 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:56:41.432 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:56:41.433 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:56:41.433 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:56:41.433 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:41.433 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:41.433 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:56:41.433 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:56:41.434 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:56:41.434 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:56:41.434 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:56:41.434 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:56:41.434 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:56:41.434 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:56:41.435 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:56:41.435 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:56:41.435 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:56:41.435 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:56:41.435 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:56:41.435 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:56:41.436 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:56:41.436 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:56:41.436 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:56:41.436 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:56:41.436 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:56:41.436 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:56:41.437 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:56:41.437 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:56:41.437 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:56:41.437 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:56:41.437 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:56:41.437 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:56:41.438 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:56:41.438 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:56:41.438 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:41.438 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:56:41.438 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:41.438 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:56:41.439 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:41.439 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:56:41.439 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:56:41.439 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:56:41.439 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:56:41.439 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:41.440 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:56:41.440 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:56:41.440 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:41.440 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:41.440 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:41.440 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:56:41.441 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:56:41.441 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:41.441 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:41.441 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:41.441 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:41.442 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:56:41.442 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:56:41.442 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:56:41.442 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:56:41.442 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:56:41.442 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:41.443 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:41.443 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:56:41.443 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:56:41.443 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:56:41.443 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:56:41.443 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:41.444 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:56:41.444 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:56:41.444 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:41.444 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:41.444 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:41.444 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:56:41.445 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:56:41.445 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:41.445 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:41.445 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:41.445 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:41.446 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:56:41.446 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:56:41.446 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:56:41.446 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:56:41.446 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:56:41.446 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:41.447 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:41.447 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:56:41.447 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:56:41.447 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:56:41.447 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:56:41.447 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:41.448 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:56:41.448 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:56:41.448 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:41.448 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:41.448 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:41.449 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:56:41.449 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:56:41.452 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:56:41.452 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:41.452 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:41.453 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:41.453 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:56:41.453 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:56:41.453 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:56:41.453 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:56:41.454 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:56:41.454 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:41.454 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:41.454 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:41.454 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:56:41.454 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:56:41.455 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:41.455 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:56:41.455 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:56:41.455 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:56:41.455 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:41.456 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:41.456 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:56:41.456 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:41.456 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:56:41.456 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:56:41.456 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:56:41.457 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:41.457 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:41.457 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:56:41.457 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:41.457 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:56:41.457 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:56:41.458 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 3 2024-09-11T01:56:41.458 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:41.458 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:56:41.458 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:56:41.458 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:56:41.458 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:56:41.458 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:56:41.458 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:56:41.458 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:56:41.459 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:56:41.459 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:56:41.459 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:56:41.459 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:56:41.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.407+0000 7f8da89b4640 1 -- 172.21.15.12:0/4124414333 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8d8403e240 msgr2=0x7f8d84040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:41.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.407+0000 7f8da89b4640 1 --2- 172.21.15.12:0/4124414333 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8d8403e240 0x7f8d84040700 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c003bc0 tx=0x7f8d9c00d040 comp rx=0 tx=0).stop 2024-09-11T01:56:41.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.407+0000 7f8da89b4640 10 monclient: shutdown 2024-09-11T01:56:41.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.407+0000 7f8da89b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:41.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.407+0000 7f8da89b4640 1 -- 172.21.15.12:0/4124414333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8da406f910 msgr2=0x7f8da41cd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:41.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.407+0000 7f8da89b4640 1 --2- 172.21.15.12:0/4124414333 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8da406f910 0x7f8da41cd470 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8d94000c00 tx=0x7f8d940046c0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.407+0000 7f8da2d9d640 1 -- 172.21.15.12:0/4124414333 reap_dead start 2024-09-11T01:56:41.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.407+0000 7f8da2d9d640 1 -- 172.21.15.12:0/4124414333 reap_dead start 2024-09-11T01:56:41.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.407+0000 7f8da89b4640 1 -- 172.21.15.12:0/4124414333 shutdown_connections 2024-09-11T01:56:41.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.407+0000 7f8da89b4640 1 -- 172.21.15.12:0/4124414333 >> 172.21.15.12:0/4124414333 conn(0x7f8da40a8600 msgr2=0x7f8da409e330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:41.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.407+0000 7f8da89b4640 1 -- 172.21.15.12:0/4124414333 shutdown_connections 2024-09-11T01:56:41.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.407+0000 7f8da89b4640 1 -- 172.21.15.12:0/4124414333 wait complete. 2024-09-11T01:56:41.463 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T01:56:41.463 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-11T01:56:41.748 INFO:tasks.rados.rados.0.smithi071.stdout:3998: done (1 left) 2024-09-11T01:56:41.748 INFO:tasks.rados.rados.0.smithi071.stdout:3999: rollback oid 49 current snap is 453 2024-09-11T01:56:41.748 INFO:tasks.rados.rados.0.smithi071.stdout:rollback oid 49 to 446 2024-09-11T01:56:41.748 INFO:tasks.rados.rados.0.smithi071.stdout:3997: expect (ObjNum 1249 snap 412 seq_num 1249) 2024-09-11T01:56:41.776 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 0 2024-09-11T01:56:41.811 INFO:tasks.rados.rados.0.smithi071.stdout:seeking to 1635853 2024-09-11T01:56:41.830 INFO:tasks.rados.rados.0.smithi071.stdout:3999: finishing rollback tid 0 to smithi07134777-49 2024-09-11T01:56:41.831 INFO:tasks.rados.rados.0.smithi071.stdout:3999: finishing rollback tid 1 to smithi07134777-49 2024-09-11T01:56:41.831 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 2546 (ObjNum 33 snap 0 seq_num 2371290624) dirty dne 2024-09-11T01:56:41.831 INFO:tasks.rados.rados.0.smithi071.stdout:3997: done (1 left) 2024-09-11T01:56:41.831 INFO:tasks.rados.rados.0.smithi071.stdout:3999: done (0 left) 2024-09-11T01:56:41.832 INFO:tasks.rados.rados.0.smithi071.stderr:0 errors. 2024-09-11T01:56:41.832 INFO:tasks.rados.rados.0.smithi071.stderr: 2024-09-11T01:56:41.835 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T01:56:41.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.936+0000 7f4a8b78a640 10 monclient: get_monmap_and_config 2024-09-11T01:56:41.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.936+0000 7f4a8b78a640 10 monclient: build_initial_monmap 2024-09-11T01:56:41.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.936+0000 7f4a8b78a640 10 monclient: monmap: 2024-09-11T01:56:41.938 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:41.939 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:41.939 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:56:41.937840+0000 2024-09-11T01:56:41.939 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:56:41.937840+0000 2024-09-11T01:56:41.939 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:41.939 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:41.939 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:56:41.939 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:56:41.939 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:56:41.939 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:56:41.939 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:56:41.939 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:41.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.936+0000 7f4a8b78a640 1 Processor -- start 2024-09-11T01:56:41.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.936+0000 7f4a8b78a640 1 -- start start 2024-09-11T01:56:41.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.936+0000 7f4a8b78a640 10 monclient: init 2024-09-11T01:56:41.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a8b78a640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:41.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a8b78a640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-11T01:56:41.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a8b78a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4a84137e50 0x7f4a8409d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:41.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a8b78a640 10 monclient(hunting): picked mon.noname-b con 0x7f4a84137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:41.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a8b78a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4a84138230 addr v1:172.21.15.12:6789/0 2024-09-11T01:56:41.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a8b78a640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f4a8413d380 0x7f4a8413f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:41.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a8b78a640 10 monclient(hunting): picked mon.noname-a con 0x7f4a8413d380 addr v2:172.21.15.12:3300/0 2024-09-11T01:56:41.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a8b78a640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a840a6580 con 0x7f4a84138230 2024-09-11T01:56:41.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a8b78a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:41.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a8b78a640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f4a840a62d0 con 0x7f4a8413d380 2024-09-11T01:56:41.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a8b78a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:41.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a8b78a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4a8409bb40 con 0x7f4a84137e50 2024-09-11T01:56:41.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a8b78a640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:41.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a8b78a640 10 monclient(hunting): authenticate will time out at 2408.380408s 2024-09-11T01:56:41.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89526640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4a84137e50 0x7f4a8409d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:41.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89526640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4a84137e50 0x7f4a8409d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:38104/0 (socket says 172.21.15.12:38104) 2024-09-11T01:56:41.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89526640 1 -- 172.21.15.12:0/2677156098 learned_addr learned my addr 172.21.15.12:0/2677156098 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:41.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89526640 10 monclient(hunting): get_auth_request con 0x7f4a84137e50 auth_method 0 2024-09-11T01:56:41.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89d27640 1 --2- 172.21.15.12:0/2677156098 >> v2:172.21.15.12:3300/0 conn(0x7f4a8413d380 0x7f4a8413f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:41.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89526640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:41.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89526640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:41.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89526640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:41.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89d27640 10 monclient(hunting): get_auth_request con 0x7f4a8413d380 auth_method 0 2024-09-11T01:56:41.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89d27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89d27640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89d27640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89526640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89526640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89526640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89d27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89d27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89d27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89526640 10 monclient(hunting): handle_auth_done global_id 304175 payload 274 2024-09-11T01:56:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89526640 1 -- 172.21.15.12:0/2677156098 >> v2:172.21.15.12:3300/0 conn(0x7f4a8413d380 msgr2=0x7f4a8413f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89526640 1 --2- 172.21.15.12:0/2677156098 >> v2:172.21.15.12:3300/0 conn(0x7f4a8413d380 0x7f4a8413f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89526640 1 -- 172.21.15.12:0/2677156098 >> v1:172.21.15.12:6789/0 conn(0x7f4a84138230 legacy=0x7f4a84138630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:41.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89526640 10 monclient: _finish_hunting 0 2024-09-11T01:56:41.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89526640 1 monclient: found mon.noname-b 2024-09-11T01:56:41.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89526640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:41.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89526640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:56:41.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89526640 1 -- 172.21.15.12:0/2677156098 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a8409c4a0 con 0x7f4a84137e50 2024-09-11T01:56:41.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.937+0000 7f4a89d27640 1 --2- 172.21.15.12:0/2677156098 >> v2:172.21.15.12:3300/0 conn(0x7f4a8413d380 0x7f4a8413f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:56:41.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.940+0000 7f4a89526640 1 -- 172.21.15.12:0/2677156098 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4a84137e50 msgr2=0x7f4a8409d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:41.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.940+0000 7f4a89526640 1 --2- 172.21.15.12:0/2677156098 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4a84137e50 0x7f4a8409d160 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4a80009870 tx=0x7f4a8002a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ea969f40e3225e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:41.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.940+0000 7f4a77fff640 1 -- 172.21.15.12:0/2677156098 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a80037070 con 0x7f4a84137e50 2024-09-11T01:56:41.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.940+0000 7f4a77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:41.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.940+0000 7f4a77fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:56:41.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.940+0000 7f4a77fff640 10 monclient: dump: 2024-09-11T01:56:41.952 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:41.952 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:41.952 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.952 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.952 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:41.953 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:41.953 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:41.953 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:41.953 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:41.953 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:41.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.940+0000 7f4a77fff640 10 monclient: _finish_auth 0 2024-09-11T01:56:41.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.940+0000 7f4a77fff640 10 monclient: _check_auth_tickets 2024-09-11T01:56:41.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.941+0000 7f4a77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:41.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.941+0000 7f4a8b78a640 5 monclient: authenticate success, global_id 304175 2024-09-11T01:56:41.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.941+0000 7f4a8b78a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:56:41.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.941+0000 7f4a77fff640 1 -- 172.21.15.12:0/2677156098 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a80032440 con 0x7f4a84137e50 2024-09-11T01:56:41.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.941+0000 7f4a77fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:41.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.941+0000 7f4a8b78a640 10 monclient: get_monmap_and_config success 2024-09-11T01:56:41.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.941+0000 7f4a8b78a640 4 set_mon_vals no callback set 2024-09-11T01:56:41.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.941+0000 7f4a77fff640 1 -- 172.21.15.12:0/2677156098 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a80032810 con 0x7f4a84137e50 2024-09-11T01:56:41.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.941+0000 7f4a77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:41.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.941+0000 7f4a77fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:56:41.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.941+0000 7f4a77fff640 10 monclient: dump: 2024-09-11T01:56:41.954 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:41.954 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:41.954 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.954 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.954 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:41.954 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:41.955 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:41.955 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:41.955 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:41.955 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:41.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.941+0000 7f4a8b78a640 10 monclient: shutdown 2024-09-11T01:56:41.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.941+0000 7f4a8b78a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:41.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.941+0000 7f4a8b78a640 1 -- 172.21.15.12:0/2677156098 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4a84137e50 msgr2=0x7f4a8409d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:41.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.941+0000 7f4a8b78a640 1 --2- 172.21.15.12:0/2677156098 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4a84137e50 0x7f4a8409d160 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4a80009870 tx=0x7f4a8002a0a0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.941+0000 7f4a8b78a640 1 -- 172.21.15.12:0/2677156098 shutdown_connections 2024-09-11T01:56:41.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.941+0000 7f4a8b78a640 1 --2- 172.21.15.12:0/2677156098 >> v2:172.21.15.12:3300/0 conn(0x7f4a8413d380 0x7f4a8413f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.941+0000 7f4a8b78a640 1 --2- 172.21.15.12:0/2677156098 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4a84137e50 0x7f4a8409d160 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.941+0000 7f4a8b78a640 1 -- 172.21.15.12:0/2677156098 >> 172.21.15.12:0/2677156098 conn(0x7f4a840a7020 msgr2=0x7f4a840a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:41.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.941+0000 7f4a8b78a640 1 -- 172.21.15.12:0/2677156098 shutdown_connections 2024-09-11T01:56:41.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.941+0000 7f4a8b78a640 1 -- 172.21.15.12:0/2677156098 wait complete. 2024-09-11T01:56:41.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.942+0000 7f4a8b78a640 10 monclient: build_initial_monmap 2024-09-11T01:56:41.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.942+0000 7f4a8b78a640 10 monclient: monmap: 2024-09-11T01:56:41.956 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:41.956 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:56:41.956 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:56:41.956 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:56:41.956 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:41.956 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:41.956 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:56:41.956 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:56:41.956 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:56:41.956 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:41.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.942+0000 7f4a8b78a640 1 Processor -- start 2024-09-11T01:56:41.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.942+0000 7f4a8b78a640 1 -- start start 2024-09-11T01:56:41.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.942+0000 7f4a8b78a640 10 monclient: init 2024-09-11T01:56:41.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.942+0000 7f4a8b78a640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:41.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.942+0000 7f4a8b78a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:56:41.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.942+0000 7f4a8b78a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a84137e50 0x7f4a841cd010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:41.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.942+0000 7f4a8b78a640 10 monclient(hunting): picked mon.noname-a con 0x7f4a84137e50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:56:41.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a8b78a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a84138230 0x7f4a84137070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:41.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a8b78a640 10 monclient(hunting): picked mon.noname-b con 0x7f4a84138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:56:41.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a8b78a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4a8413d380 0x7f4a840a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:41.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a8b78a640 10 monclient(hunting): picked mon.noname-c con 0x7f4a8413d380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:41.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a8b78a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:41.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a8b78a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8406f890 con 0x7f4a84137e50 2024-09-11T01:56:41.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a8b78a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a8b78a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8406f710 con 0x7f4a84138230 2024-09-11T01:56:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a8b78a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a8b78a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4a8413fcb0 con 0x7f4a8413d380 2024-09-11T01:56:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a8b78a640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a8b78a640 10 monclient(hunting): authenticate will time out at 2408.386067s 2024-09-11T01:56:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89526640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a84137e50 0x7f4a841cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89526640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a84137e50 0x7f4a841cd010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50446/0 (socket says 172.21.15.12:50446) 2024-09-11T01:56:41.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89d27640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4a8413d380 0x7f4a840a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:41.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89526640 1 -- 172.21.15.12:0/3137675501 learned_addr learned my addr 172.21.15.12:0/3137675501 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:41.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89526640 10 monclient(hunting): get_auth_request con 0x7f4a84137e50 auth_method 0 2024-09-11T01:56:41.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89526640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:41.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89526640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:41.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89526640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:41.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89d27640 10 monclient(hunting): get_auth_request con 0x7f4a8413d380 auth_method 0 2024-09-11T01:56:41.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89d27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:41.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89d27640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:41.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89d27640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:41.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a88d25640 1 -- 172.21.15.12:0/3137675501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a84138230 msgr2=0x7f4a84137070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:56:41.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89526640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:41.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a88d25640 1 --2- 172.21.15.12:0/3137675501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a84138230 0x7f4a84137070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:56:41.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89526640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:41.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89526640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:41.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89d27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:41.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89d27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:41.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89d27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:41.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89526640 10 monclient(hunting): handle_auth_done global_id 294273 payload 824 2024-09-11T01:56:41.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89526640 1 -- 172.21.15.12:0/3137675501 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4a8413d380 msgr2=0x7f4a840a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:41.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89526640 1 --2- 172.21.15.12:0/3137675501 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4a8413d380 0x7f4a840a1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89526640 1 -- 172.21.15.12:0/3137675501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a84138230 msgr2=0x7f4a84137070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:56:41.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89526640 1 --2- 172.21.15.12:0/3137675501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a84138230 0x7f4a84137070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89526640 10 monclient: _finish_hunting 0 2024-09-11T01:56:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89526640 1 monclient: found mon.noname-a 2024-09-11T01:56:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89526640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89526640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:56:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89526640 1 -- 172.21.15.12:0/3137675501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a84121820 con 0x7f4a84137e50 2024-09-11T01:56:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.943+0000 7f4a89d27640 1 --2- 172.21.15.12:0/3137675501 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4a8413d380 0x7f4a840a1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:56:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.945+0000 7f4a89526640 1 -- 172.21.15.12:0/3137675501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a84137e50 msgr2=0x7f4a841cd010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.945+0000 7f4a89526640 1 --2- 172.21.15.12:0/3137675501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a84137e50 0x7f4a841cd010 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4a800090b0 tx=0x7f4a800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.945+0000 7f4a75ffb640 1 -- 172.21.15.12:0/3137675501 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a80041020 con 0x7f4a84137e50 2024-09-11T01:56:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.945+0000 7f4a75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.945+0000 7f4a75ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:56:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.945+0000 7f4a75ffb640 10 monclient: dump: 2024-09-11T01:56:41.962 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:41.963 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:41.963 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.963 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.963 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:41.963 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:41.963 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:41.963 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:41.963 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:41.963 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:41.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.945+0000 7f4a75ffb640 10 monclient: _finish_auth 0 2024-09-11T01:56:41.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.945+0000 7f4a75ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:56:41.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.945+0000 7f4a75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:41.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.945+0000 7f4a75ffb640 1 -- 172.21.15.12:0/3137675501 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a8003d630 con 0x7f4a84137e50 2024-09-11T01:56:41.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.945+0000 7f4a8b78a640 5 monclient: authenticate success, global_id 294273 2024-09-11T01:56:41.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.945+0000 7f4a8b78a640 10 monclient: _renew_subs 2024-09-11T01:56:41.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.945+0000 7f4a8b78a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:41.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.945+0000 7f4a8b78a640 1 -- 172.21.15.12:0/3137675501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a84121b00 con 0x7f4a84137e50 2024-09-11T01:56:41.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.945+0000 7f4a75ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:41.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.945+0000 7f4a75ffb640 1 -- 172.21.15.12:0/3137675501 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a80037040 con 0x7f4a84137e50 2024-09-11T01:56:41.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.945+0000 7f4a75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:41.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.945+0000 7f4a75ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:56:41.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.945+0000 7f4a75ffb640 10 monclient: dump: 2024-09-11T01:56:41.964 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:41.965 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:41.965 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.965 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:41.965 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:41.965 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:41.965 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:41.966 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:41.966 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:41.966 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:41.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.945+0000 7f4a777fe640 4 set_mon_vals no callback set 2024-09-11T01:56:41.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.946+0000 7f4a8b78a640 10 monclient: _renew_subs 2024-09-11T01:56:41.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.946+0000 7f4a8b78a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:41.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.946+0000 7f4a8b78a640 1 -- 172.21.15.12:0/3137675501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a840a1710 con 0x7f4a84137e50 2024-09-11T01:56:41.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.947+0000 7f4a8b78a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:41.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.947+0000 7f4a8b78a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:41.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.947+0000 7f4a8b78a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:41.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.947+0000 7f4a8b78a640 1 -- 172.21.15.12:0/3137675501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a8409d160 con 0x7f4a84137e50 2024-09-11T01:56:41.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.947+0000 7f4a75ffb640 1 -- 172.21.15.12:0/3137675501 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f4a8003a030 con 0x7f4a84137e50 2024-09-11T01:56:41.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.947+0000 7f4a75ffb640 1 --2- 172.21.15.12:0/3137675501 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4a6803e210 0x7f4a680406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:41.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.947+0000 7f4a75ffb640 1 -- 172.21.15.12:0/3137675501 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(922..922 src has 889..922) ==== 4815+0+0 (secure 0 0 0) 0x7f4a80073050 con 0x7f4a84137e50 2024-09-11T01:56:41.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.947+0000 7f4a88d25640 1 --2- 172.21.15.12:0/3137675501 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4a6803e210 0x7f4a680406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:41.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.947+0000 7f4a88d25640 10 monclient: get_auth_request con 0x7f4a6803e210 auth_method 0 2024-09-11T01:56:41.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.950+0000 7f4a88d25640 1 --2- 172.21.15.12:0/3137675501 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4a6803e210 0x7f4a680406d0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f4a840a3410 tx=0x7f4a78007d90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:41.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.952+0000 7f4a75ffb640 1 -- 172.21.15.12:0/3137675501 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4a8004b030 con 0x7f4a84137e50 2024-09-11T01:56:41.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.952+0000 7f4a75ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:41.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:41.952+0000 7f4a75ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:56:42.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.140+0000 7f4a8b78a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-11T01:56:42.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.140+0000 7f4a8b78a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T01:56:42.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.140+0000 7f4a8b78a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:42.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.140+0000 7f4a8b78a640 1 -- 172.21.15.12:0/3137675501 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4a84120b90 con 0x7f4a84137e50 2024-09-11T01:56:42.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.141+0000 7f4a75ffb640 1 -- 172.21.15.12:0/3137675501 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v922) ==== 74+0+14287 (secure 0 0 0) 0x7f4a8003b3f0 con 0x7f4a84137e50 2024-09-11T01:56:42.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.141+0000 7f4a75ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T01:56:42.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.141+0000 7f4a75ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:56:42.144 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:56:42.145 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":922,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","created":"2024-09-11T01:29:02.540751+0000","modified":"2024-09-11T01:56:41.734018+0000","last_up_change":"2024-09-11T01:29:13.320573+0000","last_in_change":"2024-09-11T01:29:05.075851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-11T01:29:14.157646+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-11T01:29:14.872725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-09-11T01:29:25.089463+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"922","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":463,"snap_epoch":922,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"616ba1d0-adf6-4fe1-85e3-266e53e21867","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":4266003406},{"type":"v1","addr":"172.21.15.12:6817","nonce":4266003406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":4266003406},{"type":"v1","addr":"172.21.15.12:6819","nonce":4266003406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":4266003406},{"type":"v1","addr":"172.21.15.12:6823","nonce":4266003406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":4266003406},{"type":"v1","addr":"172.21.15.12:6821","nonce":4266003406}]},"public_addr":"172.21.15.12:6817/4266003406","cluster_addr":"172.21.15.12:6819/4266003406","heartbeat_back_addr":"172.21.15.12:6823/4266003406","heartbeat_front_addr":"172.21.15.12:6821/4266003406","state":["exists","up"]},{"osd":1,"uuid":"9b570874-7a3d-4867-8c44-29084a8bf649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3563357280},{"type":"v1","addr":"172.21.15.12:6809","nonce":3563357280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3563357280},{"type":"v1","addr":"172.21.15.12:6811","nonce":3563357280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3563357280},{"type":"v1","addr":"172.21.15.12:6815","nonce":3563357280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3563357280},{"type":"v1","addr":"172.21.15.12:6813","nonce":3563357280}]},"public_addr":"172.21.15.12:6809/3563357280","cluster_addr":"172.21.15.12:6811/3563357280","heartbeat_back_addr":"172.21.15.12:6815/3563357280","heartbeat_front_addr":"172.21.15.12:6813/3563357280","state":["exists","up"]},{"osd":2,"uuid":"b12ee288-ce57-41de-8b8e-6fad509311b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2893036867},{"type":"v1","addr":"172.21.15.12:6801","nonce":2893036867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2893036867},{"type":"v1","addr":"172.21.15.12:6803","nonce":2893036867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2893036867},{"type":"v1","addr":"172.21.15.12:6807","nonce":2893036867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2893036867},{"type":"v1","addr":"172.21.15.12:6805","nonce":2893036867}]},"public_addr":"172.21.15.12:6801/2893036867","cluster_addr":"172.21.15.12:6803/2893036867","heartbeat_back_addr":"172.21.15.12:6807/2893036867","heartbeat_front_addr":"172.21.15.12:6805/2893036867","state":["exists","up"]},{"osd":3,"uuid":"a8e81aa4-0dee-487d-981d-1f593e363eee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":3091488141},{"type":"v1","addr":"172.21.15.71:6819","nonce":3091488141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":3091488141},{"type":"v1","addr":"172.21.15.71:6821","nonce":3091488141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":3091488141},{"type":"v1","addr":"172.21.15.71:6825","nonce":3091488141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":3091488141},{"type":"v1","addr":"172.21.15.71:6823","nonce":3091488141}]},"public_addr":"172.21.15.71:6819/3091488141","cluster_addr":"172.21.15.71:6821/3091488141","heartbeat_back_addr":"172.21.15.71:6825/3091488141","heartbeat_front_addr":"172.21.15.71:6823/3091488141","state":["exists","up"]},{"osd":4,"uuid":"9db77181-fc18-408f-a35c-87bf4b4ba2d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":1790415427},{"type":"v1","addr":"172.21.15.71:6801","nonce":1790415427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":1790415427},{"type":"v1","addr":"172.21.15.71:6803","nonce":1790415427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":1790415427},{"type":"v1","addr":"172.21.15.71:6807","nonce":1790415427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":1790415427},{"type":"v1","addr":"172.21.15.71:6805","nonce":1790415427}]},"public_addr":"172.21.15.71:6801/1790415427","cluster_addr":"172.21.15.71:6803/1790415427","heartbeat_back_addr":"172.21.15.71:6807/1790415427","heartbeat_front_addr":"172.21.15.71:6805/1790415427","state":["exists","up"]},{"osd":5,"uuid":"fadf2191-f31a-4880-a529-5b2d1023d236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":2032569317},{"type":"v1","addr":"172.21.15.71:6809","nonce":2032569317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":2032569317},{"type":"v1","addr":"172.21.15.71:6811","nonce":2032569317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":2032569317},{"type":"v1","addr":"172.21.15.71:6815","nonce":2032569317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":2032569317},{"type":"v1","addr":"172.21.15.71:6813","nonce":2032569317}]},"public_addr":"172.21.15.71:6809/2032569317","cluster_addr":"172.21.15.71:6811/2032569317","heartbeat_back_addr":"172.21.15.71:6815/2032569317","heartbeat_front_addr":"172.21.15.71:6813/2032569317","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:11.548818+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:11.310103+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:10.578802+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:11.748432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:11.098916+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:11.150538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":423,"length":1},{"begin":441,"length":1},{"begin":451,"length":1},{"begin":458,"length":2}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":459,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T01:56:42.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.144+0000 7f4a8b78a640 1 -- 172.21.15.12:0/3137675501 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4a6803e210 msgr2=0x7f4a680406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:42.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.144+0000 7f4a8b78a640 1 --2- 172.21.15.12:0/3137675501 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4a6803e210 0x7f4a680406d0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f4a840a3410 tx=0x7f4a78007d90 comp rx=0 tx=0).stop 2024-09-11T01:56:42.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.144+0000 7f4a8b78a640 10 monclient: shutdown 2024-09-11T01:56:42.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.144+0000 7f4a8b78a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:42.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.144+0000 7f4a8b78a640 1 -- 172.21.15.12:0/3137675501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a84137e50 msgr2=0x7f4a841cd010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:42.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.144+0000 7f4a8b78a640 1 --2- 172.21.15.12:0/3137675501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a84137e50 0x7f4a841cd010 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4a800090b0 tx=0x7f4a800046c0 comp rx=0 tx=0).stop 2024-09-11T01:56:42.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.144+0000 7f4a8b78a640 1 -- 172.21.15.12:0/3137675501 shutdown_connections 2024-09-11T01:56:42.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.144+0000 7f4a8b78a640 1 --2- 172.21.15.12:0/3137675501 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4a8413d380 0x7f4a840a1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:42.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.144+0000 7f4a8b78a640 1 --2- 172.21.15.12:0/3137675501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a84138230 0x7f4a84137070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:42.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.144+0000 7f4a8b78a640 1 --2- 172.21.15.12:0/3137675501 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4a6803e210 0x7f4a680406d0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:42.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.144+0000 7f4a8b78a640 1 --2- 172.21.15.12:0/3137675501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4a84137e50 0x7f4a841cd010 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:42.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.144+0000 7f4a8b78a640 1 -- 172.21.15.12:0/3137675501 >> 172.21.15.12:0/3137675501 conn(0x7f4a840a7020 msgr2=0x7f4a8413c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:42.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.144+0000 7f4a8b78a640 1 -- 172.21.15.12:0/3137675501 shutdown_connections 2024-09-11T01:56:42.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.144+0000 7f4a8b78a640 1 -- 172.21.15.12:0/3137675501 wait complete. 2024-09-11T01:56:42.157 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T01:56:42.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.256+0000 7fba2567f640 10 monclient: get_monmap_and_config 2024-09-11T01:56:42.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.256+0000 7fba2567f640 10 monclient: build_initial_monmap 2024-09-11T01:56:42.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.256+0000 7fba2567f640 10 monclient: monmap: 2024-09-11T01:56:42.258 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:42.259 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:42.259 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:56:42.257803+0000 2024-09-11T01:56:42.259 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:56:42.257803+0000 2024-09-11T01:56:42.259 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:42.259 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:42.259 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:56:42.259 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:56:42.259 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:56:42.259 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:56:42.259 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:56:42.259 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:42.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.256+0000 7fba2567f640 1 Processor -- start 2024-09-11T01:56:42.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.256+0000 7fba2567f640 1 -- start start 2024-09-11T01:56:42.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.256+0000 7fba2567f640 10 monclient: init 2024-09-11T01:56:42.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba2567f640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:42.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba2567f640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-11T01:56:42.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba2567f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fba20137d50 0x7fba2009d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:42.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba2567f640 10 monclient(hunting): picked mon.noname-a con 0x7fba20137d50 addr v2:172.21.15.12:3300/0 2024-09-11T01:56:42.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba2567f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fba20138130 addr v1:172.21.15.71:6789/0 2024-09-11T01:56:42.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba2567f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fba2013d280 0x7fba2013f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:42.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba2567f640 10 monclient(hunting): picked mon.noname-c con 0x7fba2013d280 addr v2:172.21.15.71:3300/0 2024-09-11T01:56:42.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba2567f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba2009ba20 con 0x7fba20138130 2024-09-11T01:56:42.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba2567f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:42.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba2567f640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fba2009c650 con 0x7fba20137d50 2024-09-11T01:56:42.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba2567f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:42.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba2567f640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fba2009c370 con 0x7fba2013d280 2024-09-11T01:56:42.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba2567f640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:42.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba2567f640 10 monclient(hunting): authenticate will time out at 2408.700220s 2024-09-11T01:56:42.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba1effd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fba20137d50 0x7fba2009d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:42.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba1f7fe640 1 -- >> v2:172.21.15.71:3300/0 conn(0x7fba2013d280 msgr2=0x7fba2013f670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:56:42.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba1e7fc640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7fba20138130 legacy=0x7fba20138580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-09-11T01:56:42.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba1f7fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fba2013d280 0x7fba2013f670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:56:42.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba1effd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fba20137d50 0x7fba2009d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50454/0 (socket says 172.21.15.12:50454) 2024-09-11T01:56:42.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba1effd640 1 -- 172.21.15.12:0/431107666 learned_addr learned my addr 172.21.15.12:0/431107666 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:42.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba1effd640 10 monclient(hunting): get_auth_request con 0x7fba20137d50 auth_method 0 2024-09-11T01:56:42.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:42.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba1effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:42.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:42.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:42.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:42.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba1effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:42.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba1effd640 10 monclient(hunting): handle_auth_done global_id 294276 payload 274 2024-09-11T01:56:42.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba1effd640 1 -- 172.21.15.12:0/431107666 >> v2:172.21.15.71:3300/0 conn(0x7fba2013d280 msgr2=0x7fba2013f670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:56:42.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba1effd640 1 --2- 172.21.15.12:0/431107666 >> v2:172.21.15.71:3300/0 conn(0x7fba2013d280 0x7fba2013f670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:42.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba1effd640 1 -- 172.21.15.12:0/431107666 >> v1:172.21.15.71:6789/0 conn(0x7fba20138130 legacy=0x7fba20138580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:56:42.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba1effd640 10 monclient: _finish_hunting 0 2024-09-11T01:56:42.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba1effd640 1 monclient: found mon.noname-a 2024-09-11T01:56:42.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:42.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba1effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:56:42.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.257+0000 7fba1effd640 1 -- 172.21.15.12:0/431107666 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba200a6ce0 con 0x7fba20137d50 2024-09-11T01:56:42.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.260+0000 7fba1effd640 1 -- 172.21.15.12:0/431107666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba20137d50 msgr2=0x7fba2009d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:42.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.260+0000 7fba1effd640 1 --2- 172.21.15.12:0/431107666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba20137d50 0x7fba2009d110 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fba2009c7d0 tx=0x7fba1002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ce151ba65f5d7c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:42.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.260+0000 7fba1dffb640 1 -- 172.21.15.12:0/431107666 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba10037070 con 0x7fba20137d50 2024-09-11T01:56:42.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.260+0000 7fba1dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:42.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.260+0000 7fba1dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:56:42.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.260+0000 7fba1dffb640 10 monclient: dump: 2024-09-11T01:56:42.273 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:42.274 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:42.274 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:42.274 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:42.274 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:42.274 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:42.274 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:42.274 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:42.274 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:42.274 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:42.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.260+0000 7fba1dffb640 10 monclient: _finish_auth 0 2024-09-11T01:56:42.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.260+0000 7fba1dffb640 10 monclient: _check_auth_tickets 2024-09-11T01:56:42.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.260+0000 7fba1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:42.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.260+0000 7fba2567f640 5 monclient: authenticate success, global_id 294276 2024-09-11T01:56:42.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.260+0000 7fba1dffb640 1 -- 172.21.15.12:0/431107666 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba10032640 con 0x7fba20137d50 2024-09-11T01:56:42.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.260+0000 7fba2567f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:56:42.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.260+0000 7fba1dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:42.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.260+0000 7fba2567f640 10 monclient: get_monmap_and_config success 2024-09-11T01:56:42.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.260+0000 7fba2567f640 4 set_mon_vals no callback set 2024-09-11T01:56:42.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.260+0000 7fba1dffb640 1 -- 172.21.15.12:0/431107666 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba10032a10 con 0x7fba20137d50 2024-09-11T01:56:42.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.260+0000 7fba1dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:42.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.260+0000 7fba1dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:56:42.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.260+0000 7fba1dffb640 10 monclient: dump: 2024-09-11T01:56:42.275 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:42.275 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:42.275 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:42.275 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:42.275 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:42.275 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:42.275 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:42.276 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:42.276 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:42.276 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:42.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.261+0000 7fba2567f640 10 monclient: shutdown 2024-09-11T01:56:42.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.261+0000 7fba2567f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:42.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.261+0000 7fba2567f640 1 -- 172.21.15.12:0/431107666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba20137d50 msgr2=0x7fba2009d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:42.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.261+0000 7fba2567f640 1 --2- 172.21.15.12:0/431107666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba20137d50 0x7fba2009d110 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fba2009c7d0 tx=0x7fba1002a1e0 comp rx=0 tx=0).stop 2024-09-11T01:56:42.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.261+0000 7fba2567f640 1 -- 172.21.15.12:0/431107666 shutdown_connections 2024-09-11T01:56:42.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.261+0000 7fba2567f640 1 --2- 172.21.15.12:0/431107666 >> v2:172.21.15.71:3300/0 conn(0x7fba2013d280 0x7fba2013f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:42.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.261+0000 7fba2567f640 1 --2- 172.21.15.12:0/431107666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba20137d50 0x7fba2009d110 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:42.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.261+0000 7fba2567f640 1 -- 172.21.15.12:0/431107666 >> 172.21.15.12:0/431107666 conn(0x7fba200a7020 msgr2=0x7fba200a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:42.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.261+0000 7fba2567f640 1 -- 172.21.15.12:0/431107666 shutdown_connections 2024-09-11T01:56:42.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.261+0000 7fba2567f640 1 -- 172.21.15.12:0/431107666 wait complete. 2024-09-11T01:56:42.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.261+0000 7fba2567f640 10 monclient: build_initial_monmap 2024-09-11T01:56:42.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.261+0000 7fba2567f640 10 monclient: monmap: 2024-09-11T01:56:42.277 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:42.277 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:56:42.277 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:56:42.277 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:56:42.277 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:42.277 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:42.277 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:56:42.277 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:56:42.277 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:56:42.277 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:42.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.262+0000 7fba2567f640 1 Processor -- start 2024-09-11T01:56:42.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.262+0000 7fba2567f640 1 -- start start 2024-09-11T01:56:42.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.262+0000 7fba2567f640 10 monclient: init 2024-09-11T01:56:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.262+0000 7fba2567f640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.262+0000 7fba2567f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:56:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.262+0000 7fba2567f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fba20137d50 0x7fba200a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.262+0000 7fba2567f640 10 monclient(hunting): picked mon.noname-c con 0x7fba20137d50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.262+0000 7fba2567f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba20138130 0x7fba2013b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.262+0000 7fba2567f640 10 monclient(hunting): picked mon.noname-b con 0x7fba20138130 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:56:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.262+0000 7fba2567f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba2013d280 0x7fba2013be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.262+0000 7fba2567f640 10 monclient(hunting): picked mon.noname-a con 0x7fba2013d280 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:56:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.262+0000 7fba2567f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.262+0000 7fba2567f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fba2009d7d0 con 0x7fba2013d280 2024-09-11T01:56:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.262+0000 7fba2567f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.262+0000 7fba2567f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fba2009d650 con 0x7fba20138130 2024-09-11T01:56:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.262+0000 7fba2567f640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.262+0000 7fba2567f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fba2013fbb0 con 0x7fba20137d50 2024-09-11T01:56:42.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.262+0000 7fba2567f640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:42.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.262+0000 7fba2567f640 10 monclient(hunting): authenticate will time out at 2408.705947s 2024-09-11T01:56:42.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1effd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fba20137d50 0x7fba200a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:42.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba2013d280 0x7fba2013be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:42.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1e7fc640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba20138130 msgr2=0x7fba2013b8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:56:42.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1effd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fba20137d50 0x7fba200a0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:38130/0 (socket says 172.21.15.12:38130) 2024-09-11T01:56:42.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1e7fc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba20138130 0x7fba2013b8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:56:42.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba2013d280 0x7fba2013be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50464/0 (socket says 172.21.15.12:50464) 2024-09-11T01:56:42.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1effd640 1 -- 172.21.15.12:0/2294008332 learned_addr learned my addr 172.21.15.12:0/2294008332 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:42.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1effd640 10 monclient(hunting): get_auth_request con 0x7fba20137d50 auth_method 0 2024-09-11T01:56:42.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:42.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:42.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:42.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1f7fe640 10 monclient(hunting): get_auth_request con 0x7fba2013d280 auth_method 0 2024-09-11T01:56:42.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:42.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:42.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:42.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:42.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:42.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:42.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:42.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:42.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:42.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1f7fe640 10 monclient(hunting): handle_auth_done global_id 294279 payload 824 2024-09-11T01:56:42.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1f7fe640 1 -- 172.21.15.12:0/2294008332 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fba20137d50 msgr2=0x7fba200a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:42.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1f7fe640 1 --2- 172.21.15.12:0/2294008332 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fba20137d50 0x7fba200a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:42.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1f7fe640 1 -- 172.21.15.12:0/2294008332 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba20138130 msgr2=0x7fba2013b8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:56:42.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1f7fe640 1 --2- 172.21.15.12:0/2294008332 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba20138130 0x7fba2013b8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:42.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1f7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:56:42.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1f7fe640 1 monclient: found mon.noname-a 2024-09-11T01:56:42.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:42.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:56:42.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.263+0000 7fba1f7fe640 1 -- 172.21.15.12:0/2294008332 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba200a98c0 con 0x7fba2013d280 2024-09-11T01:56:42.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.265+0000 7fba1f7fe640 1 -- 172.21.15.12:0/2294008332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba2013d280 msgr2=0x7fba2013be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:42.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.265+0000 7fba1f7fe640 1 --2- 172.21.15.12:0/2294008332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba2013d280 0x7fba2013be10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fba1800bd80 tx=0x7fba18009ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:42.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.265+0000 7fba177fe640 1 -- 172.21.15.12:0/2294008332 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba18015470 con 0x7fba2013d280 2024-09-11T01:56:42.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.265+0000 7fba177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:42.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.265+0000 7fba177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:56:42.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.265+0000 7fba177fe640 10 monclient: dump: 2024-09-11T01:56:42.282 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:42.282 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:42.282 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:42.282 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:42.282 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:42.282 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:42.282 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:42.282 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:42.282 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:42.282 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.265+0000 7fba177fe640 10 monclient: _finish_auth 0 2024-09-11T01:56:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.265+0000 7fba177fe640 10 monclient: _check_auth_tickets 2024-09-11T01:56:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.265+0000 7fba177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.265+0000 7fba177fe640 1 -- 172.21.15.12:0/2294008332 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba18010070 con 0x7fba2013d280 2024-09-11T01:56:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.265+0000 7fba2567f640 5 monclient: authenticate success, global_id 294279 2024-09-11T01:56:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.265+0000 7fba2567f640 10 monclient: _renew_subs 2024-09-11T01:56:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.265+0000 7fba2567f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.265+0000 7fba2567f640 1 -- 172.21.15.12:0/2294008332 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba2006ed80 con 0x7fba2013d280 2024-09-11T01:56:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.265+0000 7fba177fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.265+0000 7fba177fe640 1 -- 172.21.15.12:0/2294008332 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba18015850 con 0x7fba2013d280 2024-09-11T01:56:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.265+0000 7fba177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.265+0000 7fba177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:56:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.265+0000 7fba177fe640 10 monclient: dump: 2024-09-11T01:56:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:42.283 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:42.284 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:42.284 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:42.284 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:42.284 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:42.284 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:42.284 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:42.284 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:42.284 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:42.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.265+0000 7fba1d7fa640 4 set_mon_vals no callback set 2024-09-11T01:56:42.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.265+0000 7fba2567f640 10 monclient: _renew_subs 2024-09-11T01:56:42.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.265+0000 7fba2567f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:42.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.265+0000 7fba2567f640 1 -- 172.21.15.12:0/2294008332 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba200a1bf0 con 0x7fba2013d280 2024-09-11T01:56:42.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.266+0000 7fba2567f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:42.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.266+0000 7fba2567f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.266+0000 7fba2567f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:56:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.266+0000 7fba2567f640 1 -- 172.21.15.12:0/2294008332 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba2009d190 con 0x7fba2013d280 2024-09-11T01:56:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.266+0000 7fba177fe640 1 -- 172.21.15.12:0/2294008332 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fba1801d030 con 0x7fba2013d280 2024-09-11T01:56:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.267+0000 7fba177fe640 1 --2- 172.21.15.12:0/2294008332 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fba0003e1f0 0x7fba000406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.267+0000 7fba177fe640 1 -- 172.21.15.12:0/2294008332 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(922..922 src has 889..922) ==== 4815+0+0 (secure 0 0 0) 0x7fba18052da0 con 0x7fba2013d280 2024-09-11T01:56:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.267+0000 7fba1effd640 1 --2- 172.21.15.12:0/2294008332 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fba0003e1f0 0x7fba000406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.267+0000 7fba1effd640 10 monclient: get_auth_request con 0x7fba0003e1f0 auth_method 0 2024-09-11T01:56:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.269+0000 7fba1effd640 1 --2- 172.21.15.12:0/2294008332 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fba0003e1f0 0x7fba000406b0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fba1002a220 tx=0x7fba1002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.271+0000 7fba177fe640 1 -- 172.21.15.12:0/2294008332 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fba18005a90 con 0x7fba2013d280 2024-09-11T01:56:42.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.271+0000 7fba177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:42.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.271+0000 7fba177fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:56:42.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.451+0000 7fba2567f640 1 -- 172.21.15.12:0/2294008332 --> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fba200a2b10 con 0x7fba0003e1f0 2024-09-11T01:56:42.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.453+0000 7fba177fe640 1 -- 172.21.15.12:0/2294008332 <== mgr.4101 v2:172.21.15.71:6816/3189259519 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+78387 (secure 0 0 0) 0x7fba200a2b10 con 0x7fba0003e1f0 2024-09-11T01:56:42.456 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:56:42.456 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-09-11T01:56:42.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.455+0000 7fba2567f640 1 -- 172.21.15.12:0/2294008332 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fba0003e1f0 msgr2=0x7fba000406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:42.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.455+0000 7fba2567f640 1 --2- 172.21.15.12:0/2294008332 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fba0003e1f0 0x7fba000406b0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fba1002a220 tx=0x7fba1002e000 comp rx=0 tx=0).stop 2024-09-11T01:56:42.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.455+0000 7fba2567f640 10 monclient: shutdown 2024-09-11T01:56:42.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.455+0000 7fba2567f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:42.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.455+0000 7fba2567f640 1 -- 172.21.15.12:0/2294008332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba2013d280 msgr2=0x7fba2013be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:42.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.455+0000 7fba2567f640 1 --2- 172.21.15.12:0/2294008332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba2013d280 0x7fba2013be10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fba1800bd80 tx=0x7fba18009ef0 comp rx=0 tx=0).stop 2024-09-11T01:56:42.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.455+0000 7fba2567f640 1 -- 172.21.15.12:0/2294008332 shutdown_connections 2024-09-11T01:56:42.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.455+0000 7fba2567f640 1 --2- 172.21.15.12:0/2294008332 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fba0003e1f0 0x7fba000406b0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:42.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.455+0000 7fba2567f640 1 --2- 172.21.15.12:0/2294008332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba2013d280 0x7fba2013be10 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:42.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.455+0000 7fba2567f640 1 --2- 172.21.15.12:0/2294008332 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba20138130 0x7fba2013b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:42.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.455+0000 7fba2567f640 1 --2- 172.21.15.12:0/2294008332 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fba20137d50 0x7fba200a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:42.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.455+0000 7fba2567f640 1 -- 172.21.15.12:0/2294008332 >> 172.21.15.12:0/2294008332 conn(0x7fba200a7020 msgr2=0x7fba20137080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:42.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.455+0000 7fba2567f640 1 -- 172.21.15.12:0/2294008332 shutdown_connections 2024-09-11T01:56:42.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.455+0000 7fba2567f640 1 -- 172.21.15.12:0/2294008332 wait complete. 2024-09-11T01:56:42.468 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":1744,"stamp":"2024-09-11T01:56:41.743950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":337300922,"num_objects":164,"num_object_clones":111,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":17,"num_read":7864,"num_read_kb":5962163,"num_write":5686,"num_write_kb":2148385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23740,"ondisk_log_size":23740,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":712412,"kb_used_data":411920,"kb_used_omap":9,"kb_used_meta":299382,"kb_avail":565518628,"statfs":{"total":579820584960,"available":579091075072,"internally_reserved":0,"allocated":421806080,"data_stored":534527518,"data_compressed":15648240,"data_compressed_allocated":10510336,"data_compressed_original":30735016,"omap_allocated":9568,"internal_metadata":306567840},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.545876"},"pg_stats":[{"pgid":"3.f","version":"920'3250","reported_seq":7491,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-11T01:56:19.198988+0000","last_change":"2024-09-11T01:56:19.198988+0000","last_active":"2024-09-11T01:56:19.198988+0000","last_peered":"2024-09-11T01:56:19.198988+0000","last_clean":"2024-09-11T01:56:19.198988+0000","last_became_active":"2024-09-11T01:29:45.769850+0000","last_became_peered":"2024-09-11T01:29:45.769850+0000","last_unstale":"2024-09-11T01:56:19.198988+0000","last_undegraded":"2024-09-11T01:56:19.198988+0000","last_fullsized":"2024-09-11T01:56:19.198988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_clean_scrub_stamp":"2024-09-11T01:29:25.600224+0000","objects_scrubbed":0,"log_size":3250,"log_dups_size":0,"ondisk_log_size":3250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:24:35.861813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089582949999999998,"stat_sum":{"num_bytes":37067161,"num_objects":21,"num_object_clones":14,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":5,"num_read":1081,"num_read_kb":920071,"num_write":797,"num_write_kb":297730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1a7","length":"1"},{"start":"1b9","length":"1"},{"start":"1c3","length":"1"},{"start":"1ca","length":"1"}]},{"pgid":"3.e","version":"920'2446","reported_seq":6699,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-11T01:56:19.199174+0000","last_change":"2024-09-11T01:56:19.199174+0000","last_active":"2024-09-11T01:56:19.199174+0000","last_peered":"2024-09-11T01:56:19.199174+0000","last_clean":"2024-09-11T01:56:19.199174+0000","last_became_active":"2024-09-11T01:29:45.767947+0000","last_became_peered":"2024-09-11T01:29:45.767947+0000","last_unstale":"2024-09-11T01:56:19.199174+0000","last_undegraded":"2024-09-11T01:56:19.199174+0000","last_fullsized":"2024-09-11T01:56:19.199174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_clean_scrub_stamp":"2024-09-11T01:29:25.600224+0000","objects_scrubbed":0,"log_size":2446,"log_dups_size":0,"ondisk_log_size":2446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:03:29.531689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0090193479999999999,"stat_sum":{"num_bytes":32967194,"num_objects":16,"num_object_clones":11,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":2,"num_read":782,"num_read_kb":623054,"num_write":538,"num_write_kb":193684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1a7","length":"1"},{"start":"1b9","length":"1"},{"start":"1c3","length":"1"},{"start":"1ca","length":"1"}]},{"pgid":"3.d","version":"918'942","reported_seq":5326,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-11T01:56:19.194181+0000","last_change":"2024-09-11T01:56:19.194181+0000","last_active":"2024-09-11T01:56:19.194181+0000","last_peered":"2024-09-11T01:56:19.194181+0000","last_clean":"2024-09-11T01:56:19.194181+0000","last_became_active":"2024-09-11T01:29:45.775407+0000","last_became_peered":"2024-09-11T01:29:45.775407+0000","last_unstale":"2024-09-11T01:56:19.194181+0000","last_undegraded":"2024-09-11T01:56:19.194181+0000","last_fullsized":"2024-09-11T01:56:19.194181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_clean_scrub_stamp":"2024-09-11T01:29:25.600224+0000","objects_scrubbed":0,"log_size":942,"log_dups_size":0,"ondisk_log_size":942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:49:52.218551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097478500000000004,"stat_sum":{"num_bytes":13470150,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":312,"num_read_kb":237002,"num_write":224,"num_write_kb":97301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1a7","length":"1"},{"start":"1b9","length":"1"},{"start":"1c3","length":"1"},{"start":"1ca","length":"1"}]},{"pgid":"3.c","version":"913'475","reported_seq":4899,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-11T01:56:19.190599+0000","last_change":"2024-09-11T01:56:19.190599+0000","last_active":"2024-09-11T01:56:19.190599+0000","last_peered":"2024-09-11T01:56:19.190599+0000","last_clean":"2024-09-11T01:56:19.190599+0000","last_became_active":"2024-09-11T01:29:45.779412+0000","last_became_peered":"2024-09-11T01:29:45.779412+0000","last_unstale":"2024-09-11T01:56:19.190599+0000","last_undegraded":"2024-09-11T01:56:19.190599+0000","last_fullsized":"2024-09-11T01:56:19.190599+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_clean_scrub_stamp":"2024-09-11T01:29:25.600224+0000","objects_scrubbed":0,"log_size":475,"log_dups_size":0,"ondisk_log_size":475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:54:01.309098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064735699999999999,"stat_sum":{"num_bytes":4226058,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":142,"num_read_kb":94400,"num_write":115,"num_write_kb":38870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1a7","length":"1"},{"start":"1b9","length":"1"},{"start":"1c3","length":"1"},{"start":"1ca","length":"1"}]},{"pgid":"3.b","version":"918'438","reported_seq":4877,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-11T01:56:19.191723+0000","last_change":"2024-09-11T01:56:19.191723+0000","last_active":"2024-09-11T01:56:19.191723+0000","last_peered":"2024-09-11T01:56:19.191723+0000","last_clean":"2024-09-11T01:56:19.191723+0000","last_became_active":"2024-09-11T01:29:45.780171+0000","last_became_peered":"2024-09-11T01:29:45.780171+0000","last_unstale":"2024-09-11T01:56:19.191723+0000","last_undegraded":"2024-09-11T01:56:19.191723+0000","last_fullsized":"2024-09-11T01:56:19.191723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_clean_scrub_stamp":"2024-09-11T01:29:25.600224+0000","objects_scrubbed":0,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:43:02.430916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055860999999999997,"stat_sum":{"num_bytes":10236354,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":157,"num_read_kb":132158,"num_write":112,"num_write_kb":40633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a7","length":"1"},{"start":"1b9","length":"1"},{"start":"1c3","length":"1"},{"start":"1ca","length":"1"}]},{"pgid":"3.a","version":"920'1488","reported_seq":5829,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-11T01:56:19.197576+0000","last_change":"2024-09-11T01:56:19.197852+0000","last_active":"2024-09-11T01:56:19.197576+0000","last_peered":"2024-09-11T01:56:19.197576+0000","last_clean":"2024-09-11T01:56:19.197576+0000","last_became_active":"2024-09-11T01:29:45.767686+0000","last_became_peered":"2024-09-11T01:29:45.767686+0000","last_unstale":"2024-09-11T01:56:19.197576+0000","last_undegraded":"2024-09-11T01:56:19.197576+0000","last_fullsized":"2024-09-11T01:56:19.197576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_clean_scrub_stamp":"2024-09-11T01:29:25.600224+0000","objects_scrubbed":0,"log_size":1488,"log_dups_size":0,"ondisk_log_size":1488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:02:33.437465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041353020000000004,"stat_sum":{"num_bytes":20925825,"num_objects":13,"num_object_clones":10,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":2,"num_read":492,"num_read_kb":420184,"num_write":344,"num_write_kb":124106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1a7","length":"1"},{"start":"1b9","length":"1"},{"start":"1c3","length":"1"},{"start":"1ca","length":"1"}]},{"pgid":"3.9","version":"918'973","reported_seq":5363,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-11T01:56:19.196871+0000","last_change":"2024-09-11T01:56:19.197273+0000","last_active":"2024-09-11T01:56:19.196871+0000","last_peered":"2024-09-11T01:56:19.196871+0000","last_clean":"2024-09-11T01:56:19.196871+0000","last_became_active":"2024-09-11T01:29:45.775201+0000","last_became_peered":"2024-09-11T01:29:45.775201+0000","last_unstale":"2024-09-11T01:56:19.196871+0000","last_undegraded":"2024-09-11T01:56:19.196871+0000","last_fullsized":"2024-09-11T01:56:19.196871+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_clean_scrub_stamp":"2024-09-11T01:29:25.600224+0000","objects_scrubbed":0,"log_size":973,"log_dups_size":0,"ondisk_log_size":973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:51:21.402445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076152999999999995,"stat_sum":{"num_bytes":16218738,"num_objects":8,"num_object_clones":6,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":300,"num_read_kb":180678,"num_write":247,"num_write_kb":92364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1a7","length":"1"},{"start":"1b9","length":"1"},{"start":"1c3","length":"1"},{"start":"1ca","length":"1"}]},{"pgid":"3.8","version":"920'1401","reported_seq":5741,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-11T01:56:19.190092+0000","last_change":"2024-09-11T01:56:19.190261+0000","last_active":"2024-09-11T01:56:19.190092+0000","last_peered":"2024-09-11T01:56:19.190092+0000","last_clean":"2024-09-11T01:56:19.190092+0000","last_became_active":"2024-09-11T01:29:45.777170+0000","last_became_peered":"2024-09-11T01:29:45.777170+0000","last_unstale":"2024-09-11T01:56:19.190092+0000","last_undegraded":"2024-09-11T01:56:19.190092+0000","last_fullsized":"2024-09-11T01:56:19.190092+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_clean_scrub_stamp":"2024-09-11T01:29:25.600224+0000","objects_scrubbed":0,"log_size":1401,"log_dups_size":0,"ondisk_log_size":1401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:02:41.734219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029326159999999999,"stat_sum":{"num_bytes":23317948,"num_objects":12,"num_object_clones":9,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":413,"num_read_kb":320850,"num_write":341,"num_write_kb":118310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a7","length":"1"},{"start":"1b9","length":"1"},{"start":"1c3","length":"1"},{"start":"1ca","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":1801,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-11T01:56:19.192851+0000","last_change":"2024-09-11T01:29:17.579548+0000","last_active":"2024-09-11T01:56:19.192851+0000","last_peered":"2024-09-11T01:56:19.192851+0000","last_clean":"2024-09-11T01:56:19.192851+0000","last_became_active":"2024-09-11T01:29:15.546952+0000","last_became_peered":"2024-09-11T01:29:15.546952+0000","last_unstale":"2024-09-11T01:56:19.192851+0000","last_undegraded":"2024-09-11T01:56:19.192851+0000","last_fullsized":"2024-09-11T01:56:19.192851+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:45:28.460820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038245800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"920'2661","reported_seq":6912,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-11T01:56:19.198072+0000","last_change":"2024-09-11T01:56:19.198072+0000","last_active":"2024-09-11T01:56:19.198072+0000","last_peered":"2024-09-11T01:56:19.198072+0000","last_clean":"2024-09-11T01:56:19.198072+0000","last_became_active":"2024-09-11T01:29:45.772821+0000","last_became_peered":"2024-09-11T01:29:45.772821+0000","last_unstale":"2024-09-11T01:56:19.198072+0000","last_undegraded":"2024-09-11T01:56:19.198072+0000","last_fullsized":"2024-09-11T01:56:19.198072+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_clean_scrub_stamp":"2024-09-11T01:29:25.600224+0000","objects_scrubbed":0,"log_size":2661,"log_dups_size":0,"ondisk_log_size":2661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:35:45.697312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0069095190000000003,"stat_sum":{"num_bytes":33791001,"num_objects":15,"num_object_clones":10,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":2,"num_read":876,"num_read_kb":697825,"num_write":538,"num_write_kb":224769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a7","length":"1"},{"start":"1b9","length":"1"},{"start":"1c3","length":"1"},{"start":"1ca","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":1806,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-11T01:56:19.189871+0000","last_change":"2024-09-11T01:29:17.597459+0000","last_active":"2024-09-11T01:56:19.189871+0000","last_peered":"2024-09-11T01:56:19.189871+0000","last_clean":"2024-09-11T01:56:19.189871+0000","last_became_active":"2024-09-11T01:29:15.549013+0000","last_became_peered":"2024-09-11T01:29:15.549013+0000","last_unstale":"2024-09-11T01:56:19.189871+0000","last_undegraded":"2024-09-11T01:56:19.189871+0000","last_fullsized":"2024-09-11T01:56:19.189871+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:46:02.130174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041667199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"918'859","reported_seq":5253,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-11T01:56:19.197120+0000","last_change":"2024-09-11T01:56:19.197120+0000","last_active":"2024-09-11T01:56:19.197120+0000","last_peered":"2024-09-11T01:56:19.197120+0000","last_clean":"2024-09-11T01:56:19.197120+0000","last_became_active":"2024-09-11T01:29:45.780171+0000","last_became_peered":"2024-09-11T01:29:45.780171+0000","last_unstale":"2024-09-11T01:56:19.197120+0000","last_undegraded":"2024-09-11T01:56:19.197120+0000","last_fullsized":"2024-09-11T01:56:19.197120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_clean_scrub_stamp":"2024-09-11T01:29:25.600224+0000","objects_scrubbed":0,"log_size":859,"log_dups_size":0,"ondisk_log_size":859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:18:28.212352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067209300000000002,"stat_sum":{"num_bytes":10992966,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":273,"num_read_kb":176553,"num_write":199,"num_write_kb":69322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1a7","length":"1"},{"start":"1b9","length":"1"},{"start":"1c3","length":"1"},{"start":"1ca","length":"1"}]},{"pgid":"1.2","version":"16'2","reported_seq":1816,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-11T01:56:19.196579+0000","last_change":"2024-09-11T01:29:17.577463+0000","last_active":"2024-09-11T01:56:19.196579+0000","last_peered":"2024-09-11T01:56:19.196579+0000","last_clean":"2024-09-11T01:56:19.196579+0000","last_became_active":"2024-09-11T01:29:15.548884+0000","last_became_peered":"2024-09-11T01:29:15.548884+0000","last_unstale":"2024-09-11T01:56:19.196579+0000","last_undegraded":"2024-09-11T01:56:19.196579+0000","last_fullsized":"2024-09-11T01:56:19.196579+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:09.120675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094832499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"920'3175","reported_seq":7410,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-11T01:56:19.195994+0000","last_change":"2024-09-11T01:56:19.196176+0000","last_active":"2024-09-11T01:56:19.195994+0000","last_peered":"2024-09-11T01:56:19.195994+0000","last_clean":"2024-09-11T01:56:19.195994+0000","last_became_active":"2024-09-11T01:29:45.776149+0000","last_became_peered":"2024-09-11T01:29:45.776149+0000","last_unstale":"2024-09-11T01:56:19.195994+0000","last_undegraded":"2024-09-11T01:56:19.195994+0000","last_fullsized":"2024-09-11T01:56:19.195994+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_clean_scrub_stamp":"2024-09-11T01:29:25.600224+0000","objects_scrubbed":0,"log_size":3175,"log_dups_size":0,"ondisk_log_size":3175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:39:09.815092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047131389999999999,"stat_sum":{"num_bytes":43558428,"num_objects":21,"num_object_clones":14,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":1053,"num_read_kb":750989,"num_write":749,"num_write_kb":347351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a7","length":"1"},{"start":"1b9","length":"1"},{"start":"1c3","length":"1"},{"start":"1ca","length":"1"}]},{"pgid":"1.3","version":"14'1","reported_seq":1811,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-11T01:56:19.191130+0000","last_change":"2024-09-11T01:29:17.583117+0000","last_active":"2024-09-11T01:56:19.191130+0000","last_peered":"2024-09-11T01:56:19.191130+0000","last_clean":"2024-09-11T01:56:19.191130+0000","last_became_active":"2024-09-11T01:29:15.546142+0000","last_became_peered":"2024-09-11T01:29:15.546142+0000","last_unstale":"2024-09-11T01:56:19.191130+0000","last_undegraded":"2024-09-11T01:56:19.191130+0000","last_fullsized":"2024-09-11T01:56:19.191130+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:16:49.024029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044411199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"16'72","reported_seq":1884,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-11T01:56:19.192736+0000","last_change":"2024-09-11T01:29:16.691671+0000","last_active":"2024-09-11T01:56:19.192736+0000","last_peered":"2024-09-11T01:56:19.192736+0000","last_clean":"2024-09-11T01:56:19.192736+0000","last_became_active":"2024-09-11T01:29:16.690647+0000","last_became_peered":"2024-09-11T01:29:16.690647+0000","last_unstale":"2024-09-11T01:56:19.192736+0000","last_undegraded":"2024-09-11T01:56:19.192736+0000","last_fullsized":"2024-09-11T01:56:19.192736+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:15.533298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:15.533298+0000","last_clean_scrub_stamp":"2024-09-11T01:29:15.533298+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:41:07.416896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":4391,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-11T01:56:19.186956+0000","last_change":"2024-09-11T01:56:19.186956+0000","last_active":"2024-09-11T01:56:19.186956+0000","last_peered":"2024-09-11T01:56:19.186956+0000","last_clean":"2024-09-11T01:56:19.186956+0000","last_became_active":"2024-09-11T01:29:45.778869+0000","last_became_peered":"2024-09-11T01:29:45.778869+0000","last_unstale":"2024-09-11T01:56:19.186956+0000","last_undegraded":"2024-09-11T01:56:19.186956+0000","last_fullsized":"2024-09-11T01:56:19.186956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_clean_scrub_stamp":"2024-09-11T01:29:25.600224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:50:41.084800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052852499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1a7","length":"1"},{"start":"1b9","length":"1"},{"start":"1c3","length":"1"},{"start":"1ca","length":"1"}]},{"pgid":"3.6","version":"920'1830","reported_seq":6185,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-11T01:56:19.189600+0000","last_change":"2024-09-11T01:56:19.189825+0000","last_active":"2024-09-11T01:56:19.189600+0000","last_peered":"2024-09-11T01:56:19.189600+0000","last_clean":"2024-09-11T01:56:19.189600+0000","last_became_active":"2024-09-11T01:29:45.779451+0000","last_became_peered":"2024-09-11T01:29:45.779451+0000","last_unstale":"2024-09-11T01:56:19.189600+0000","last_undegraded":"2024-09-11T01:56:19.189600+0000","last_fullsized":"2024-09-11T01:56:19.189600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_clean_scrub_stamp":"2024-09-11T01:29:25.600224+0000","objects_scrubbed":0,"log_size":1830,"log_dups_size":0,"ondisk_log_size":1830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:02:14.283287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029679799999999998,"stat_sum":{"num_bytes":30442876,"num_objects":11,"num_object_clones":7,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":576,"num_read_kb":420347,"num_write":505,"num_write_kb":186365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1a7","length":"1"},{"start":"1b9","length":"1"},{"start":"1c3","length":"1"},{"start":"1ca","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":1810,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-11T01:56:19.191285+0000","last_change":"2024-09-11T01:29:17.583012+0000","last_active":"2024-09-11T01:56:19.191285+0000","last_peered":"2024-09-11T01:56:19.191285+0000","last_clean":"2024-09-11T01:56:19.191285+0000","last_became_active":"2024-09-11T01:29:15.547447+0000","last_became_peered":"2024-09-11T01:29:15.547447+0000","last_unstale":"2024-09-11T01:56:19.191285+0000","last_undegraded":"2024-09-11T01:56:19.191285+0000","last_fullsized":"2024-09-11T01:56:19.191285+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:17:48.616008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037448800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"918'1723","reported_seq":6056,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-11T01:56:19.187536+0000","last_change":"2024-09-11T01:56:19.187802+0000","last_active":"2024-09-11T01:56:19.187536+0000","last_peered":"2024-09-11T01:56:19.187536+0000","last_clean":"2024-09-11T01:56:19.187536+0000","last_became_active":"2024-09-11T01:29:45.776998+0000","last_became_peered":"2024-09-11T01:29:45.776998+0000","last_unstale":"2024-09-11T01:56:19.187536+0000","last_undegraded":"2024-09-11T01:56:19.187536+0000","last_fullsized":"2024-09-11T01:56:19.187536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_clean_scrub_stamp":"2024-09-11T01:29:25.600224+0000","objects_scrubbed":0,"log_size":1723,"log_dups_size":0,"ondisk_log_size":1723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:09:08.692234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050832700000000004,"stat_sum":{"num_bytes":12249170,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":583,"num_read_kb":400414,"num_write":413,"num_write_kb":160590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a7","length":"1"},{"start":"1b9","length":"1"},{"start":"1c3","length":"1"},{"start":"1ca","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":1814,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-11T01:56:19.196734+0000","last_change":"2024-09-11T01:29:17.576853+0000","last_active":"2024-09-11T01:56:19.196734+0000","last_peered":"2024-09-11T01:56:19.196734+0000","last_clean":"2024-09-11T01:56:19.196734+0000","last_became_active":"2024-09-11T01:29:15.546476+0000","last_became_peered":"2024-09-11T01:29:15.546476+0000","last_unstale":"2024-09-11T01:56:19.196734+0000","last_undegraded":"2024-09-11T01:56:19.196734+0000","last_fullsized":"2024-09-11T01:56:19.196734+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:11:54.706642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037031899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"918'938","reported_seq":5332,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-11T01:56:19.192058+0000","last_change":"2024-09-11T01:56:19.192058+0000","last_active":"2024-09-11T01:56:19.192058+0000","last_peered":"2024-09-11T01:56:19.192058+0000","last_clean":"2024-09-11T01:56:19.192058+0000","last_became_active":"2024-09-11T01:29:45.776069+0000","last_became_peered":"2024-09-11T01:29:45.776069+0000","last_unstale":"2024-09-11T01:56:19.192058+0000","last_undegraded":"2024-09-11T01:56:19.192058+0000","last_fullsized":"2024-09-11T01:56:19.192058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_clean_scrub_stamp":"2024-09-11T01:29:25.600224+0000","objects_scrubbed":0,"log_size":938,"log_dups_size":0,"ondisk_log_size":938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:47:48.950504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749524,"stat_sum":{"num_bytes":24458137,"num_objects":8,"num_object_clones":6,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":372,"num_read_kb":297161,"num_write":201,"num_write_kb":67553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a7","length":"1"},{"start":"1b9","length":"1"},{"start":"1c3","length":"1"},{"start":"1ca","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":1800,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-11T01:56:19.192997+0000","last_change":"2024-09-11T01:29:17.579590+0000","last_active":"2024-09-11T01:56:19.192997+0000","last_peered":"2024-09-11T01:56:19.192997+0000","last_clean":"2024-09-11T01:56:19.192997+0000","last_became_active":"2024-09-11T01:29:15.542800+0000","last_became_peered":"2024-09-11T01:29:15.542800+0000","last_unstale":"2024-09-11T01:56:19.192997+0000","last_undegraded":"2024-09-11T01:56:19.192997+0000","last_fullsized":"2024-09-11T01:56:19.192997+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:23:55.773396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040701099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"920'1066","reported_seq":5445,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-11T01:56:19.194579+0000","last_change":"2024-09-11T01:56:19.194579+0000","last_active":"2024-09-11T01:56:19.194579+0000","last_peered":"2024-09-11T01:56:19.194579+0000","last_clean":"2024-09-11T01:56:19.194579+0000","last_became_active":"2024-09-11T01:29:45.775427+0000","last_became_peered":"2024-09-11T01:29:45.775427+0000","last_unstale":"2024-09-11T01:56:19.194579+0000","last_undegraded":"2024-09-11T01:56:19.194579+0000","last_fullsized":"2024-09-11T01:56:19.194579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:25.600224+0000","last_clean_scrub_stamp":"2024-09-11T01:29:25.600224+0000","objects_scrubbed":0,"log_size":1066,"log_dups_size":0,"ondisk_log_size":1066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:41:57.785806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045932569999999999,"stat_sum":{"num_bytes":22788529,"num_objects":11,"num_object_clones":9,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":386,"num_read_kb":290422,"num_write":260,"num_write_kb":87807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1a7","length":"1"},{"start":"1b9","length":"1"},{"start":"1c3","length":"1"},{"start":"1ca","length":"1"}]},{"pgid":"1.7","version":"0'0","reported_seq":1805,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-11T01:56:19.189954+0000","last_change":"2024-09-11T01:29:17.597492+0000","last_active":"2024-09-11T01:56:19.189954+0000","last_peered":"2024-09-11T01:56:19.189954+0000","last_clean":"2024-09-11T01:56:19.189954+0000","last_became_active":"2024-09-11T01:29:15.546444+0000","last_became_peered":"2024-09-11T01:29:15.546444+0000","last_unstale":"2024-09-11T01:56:19.189954+0000","last_undegraded":"2024-09-11T01:56:19.189954+0000","last_fullsized":"2024-09-11T01:56:19.189954+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:52:43.624650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000396053,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":336710535,"num_objects":160,"num_object_clones":111,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":17,"num_read":7798,"num_read_kb":5962108,"num_write":5583,"num_write_kb":2146755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":403562496,"data_stored":525565902,"data_compressed":15589104,"data_compressed_allocated":10428416,"data_compressed_original":29711016,"omap_allocated":0,"internal_metadata":0},"log_size":23665,"ondisk_log_size":23665,"up":32,"acting":32,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59136,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607886,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":123276,"kb_used_data":78196,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94248564,"statfs":{"total":96636764160,"available":96510529536,"internally_reserved":0,"allocated":80072704,"data_stored":106159609,"data_compressed":3544588,"data_compressed_allocated":2297856,"data_compressed_original":6712660,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607884,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":106344,"kb_used_data":52944,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94265496,"statfs":{"total":96636764160,"available":96527867904,"internally_reserved":0,"allocated":54214656,"data_stored":66508644,"data_compressed":1624398,"data_compressed_allocated":946176,"data_compressed_original":2996633,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607885,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97648,"kb_used_data":43292,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94274192,"statfs":{"total":96636764160,"available":96536772608,"internally_reserved":0,"allocated":44331008,"data_stored":58468158,"data_compressed":1315766,"data_compressed_allocated":1019904,"data_compressed_original":3179451,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607885,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":174852,"kb_used_data":107628,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94196988,"statfs":{"total":96636764160,"available":96457715712,"internally_reserved":0,"allocated":110211072,"data_stored":139792616,"data_compressed":4489857,"data_compressed_allocated":3002368,"data_compressed_original":9113600,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607885,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67160,"kb_used_data":35972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94304680,"statfs":{"total":96636764160,"available":96567992320,"internally_reserved":0,"allocated":36835328,"data_stored":39154563,"data_compressed":1087863,"data_compressed_allocated":950272,"data_compressed_original":2187264,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640591,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":143132,"kb_used_data":93888,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94228708,"statfs":{"total":96636764160,"available":96490196992,"internally_reserved":0,"allocated":96141312,"data_stored":124443928,"data_compressed":3585768,"data_compressed_allocated":2293760,"data_compressed_original":6545408,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33812480,"data_stored":37837257,"data_compressed":1087863,"data_compressed_allocated":950272,"data_compressed_original":2187264,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":107130880,"data_stored":137946401,"data_compressed":4460289,"data_compressed_allocated":2961408,"data_compressed_original":8601600,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":93122560,"data_stored":123126641,"data_compressed":3585768,"data_compressed_allocated":2293760,"data_compressed_original":6545408,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":41250816,"data_stored":56621943,"data_compressed":1286198,"data_compressed_allocated":978944,"data_compressed_original":2667451,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":51191808,"data_stored":65191338,"data_compressed":1624398,"data_compressed_allocated":946176,"data_compressed_original":2996633,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":77053952,"data_stored":104842322,"data_compressed":3544588,"data_compressed_allocated":2297856,"data_compressed_original":6712660,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T01:56:42.470 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-11T01:56:42.470 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-11T01:56:42.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.573+0000 7f3bdd616640 10 monclient: get_monmap_and_config 2024-09-11T01:56:42.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.573+0000 7f3bdd616640 10 monclient: build_initial_monmap 2024-09-11T01:56:42.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.573+0000 7f3bdd616640 10 monclient: monmap: 2024-09-11T01:56:42.576 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:42.576 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:42.576 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:56:42.575649+0000 2024-09-11T01:56:42.577 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:56:42.575649+0000 2024-09-11T01:56:42.577 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:42.577 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:42.577 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:56:42.577 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:56:42.577 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:56:42.577 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:56:42.577 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:56:42.577 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:42.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.574+0000 7f3bdd616640 1 Processor -- start 2024-09-11T01:56:42.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.574+0000 7f3bdd616640 1 -- start start 2024-09-11T01:56:42.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.574+0000 7f3bdd616640 10 monclient: init 2024-09-11T01:56:42.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.574+0000 7f3bdd616640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:42.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bdd616640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-11T01:56:42.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bdd616640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3bd80a5020 0x7f3bd809d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:42.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bdd616640 10 monclient(hunting): picked mon.noname-c con 0x7f3bd80a5020 addr v2:172.21.15.71:3300/0 2024-09-11T01:56:42.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bdd616640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3bd80a5400 addr v1:172.21.15.12:6789/0 2024-09-11T01:56:42.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bdd616640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3bd813d230 0x7f3bd813f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:42.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bdd616640 10 monclient(hunting): picked mon.noname-b con 0x7f3bd813d230 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:42.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bdd616640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bd80a6b50 con 0x7f3bd80a5400 2024-09-11T01:56:42.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bdd616640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:42.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bdd616640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f3bd809d8e0 con 0x7f3bd80a5020 2024-09-11T01:56:42.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bdd616640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:42.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bdd616640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f3bd806fa80 con 0x7f3bd813d230 2024-09-11T01:56:42.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bdd616640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:42.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bdd616640 10 monclient(hunting): authenticate will time out at 2409.018099s 2024-09-11T01:56:42.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bd77fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3bd813d230 0x7f3bd813f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:42.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bd67fc640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3bd80a5400 0x7f3bd80a57e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:39442/0 (socket says 172.21.15.12:39442) 2024-09-11T01:56:42.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bd6ffd640 1 -- >> v2:172.21.15.71:3300/0 conn(0x7f3bd80a5020 msgr2=0x7f3bd809d3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:56:42.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bd67fc640 1 -- 172.21.15.12:0/948540469 learned_addr learned my addr 172.21.15.12:0/948540469 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:56:42.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bd77fe640 10 monclient(hunting): get_auth_request con 0x7f3bd813d230 auth_method 0 2024-09-11T01:56:42.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bd6ffd640 1 --2- 172.21.15.12:0/948540469 >> v2:172.21.15.71:3300/0 conn(0x7f3bd80a5020 0x7f3bd809d3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:56:42.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bd77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:42.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bd77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:42.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:42.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bd77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:42.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bd77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:42.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bd77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:42.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bd77fe640 10 monclient(hunting): handle_auth_done global_id 304184 payload 274 2024-09-11T01:56:42.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bd77fe640 1 -- 172.21.15.12:0/948540469 >> v2:172.21.15.71:3300/0 conn(0x7f3bd80a5020 msgr2=0x7f3bd809d3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:56:42.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bd77fe640 1 --2- 172.21.15.12:0/948540469 >> v2:172.21.15.71:3300/0 conn(0x7f3bd80a5020 0x7f3bd809d3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:42.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bd77fe640 1 -- 172.21.15.12:0/948540469 >> v1:172.21.15.12:6789/0 conn(0x7f3bd80a5400 legacy=0x7f3bd80a57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:56:42.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bd77fe640 10 monclient: _finish_hunting 0 2024-09-11T01:56:42.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bd77fe640 1 monclient: found mon.noname-b 2024-09-11T01:56:42.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bd77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:42.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bd77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:56:42.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.575+0000 7f3bd77fe640 1 -- 172.21.15.12:0/948540469 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bd80a6910 con 0x7f3bd813d230 2024-09-11T01:56:42.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bd77fe640 1 -- 172.21.15.12:0/948540469 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3bd813d230 msgr2=0x7f3bd813f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:42.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bd77fe640 1 --2- 172.21.15.12:0/948540469 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3bd813d230 0x7f3bd813f620 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3bd00097a0 tx=0x7f3bd0029fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fc3cdcce0ebd3ae server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:42.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bd5ffb640 1 -- 172.21.15.12:0/948540469 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0037070 con 0x7f3bd813d230 2024-09-11T01:56:42.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bd5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:42.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bd5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:56:42.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bd5ffb640 10 monclient: dump: 2024-09-11T01:56:42.588 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:42.588 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:42.588 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:42.588 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:42.588 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:42.588 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:42.588 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:42.588 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:42.588 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:42.588 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:42.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bd5ffb640 10 monclient: _finish_auth 0 2024-09-11T01:56:42.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bd5ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:56:42.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:42.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bdd616640 5 monclient: authenticate success, global_id 304184 2024-09-11T01:56:42.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bd5ffb640 1 -- 172.21.15.12:0/948540469 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bd0032640 con 0x7f3bd813d230 2024-09-11T01:56:42.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bdd616640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:56:42.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bd5ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:42.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bdd616640 10 monclient: get_monmap_and_config success 2024-09-11T01:56:42.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bd5ffb640 1 -- 172.21.15.12:0/948540469 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0032a10 con 0x7f3bd813d230 2024-09-11T01:56:42.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bdd616640 4 set_mon_vals no callback set 2024-09-11T01:56:42.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bd5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:42.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bd5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:56:42.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bd5ffb640 10 monclient: dump: 2024-09-11T01:56:42.589 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:42.589 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:42.589 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:42.590 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:42.590 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:42.590 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:42.590 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:42.590 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:42.590 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:42.590 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:42.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bdd616640 10 monclient: shutdown 2024-09-11T01:56:42.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bdd616640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:42.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bdd616640 1 -- 172.21.15.12:0/948540469 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3bd813d230 msgr2=0x7f3bd813f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:42.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bdd616640 1 --2- 172.21.15.12:0/948540469 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3bd813d230 0x7f3bd813f620 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3bd00097a0 tx=0x7f3bd0029fe0 comp rx=0 tx=0).stop 2024-09-11T01:56:42.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bdd616640 1 -- 172.21.15.12:0/948540469 shutdown_connections 2024-09-11T01:56:42.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bdd616640 1 --2- 172.21.15.12:0/948540469 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3bd813d230 0x7f3bd813f620 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:42.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bdd616640 1 --2- 172.21.15.12:0/948540469 >> v2:172.21.15.71:3300/0 conn(0x7f3bd80a5020 0x7f3bd809d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.578+0000 7f3bdd616640 1 -- 172.21.15.12:0/948540469 >> 172.21.15.12:0/948540469 conn(0x7f3bd80a7540 msgr2=0x7f3bd80a7950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.579+0000 7f3bdd616640 1 -- 172.21.15.12:0/948540469 shutdown_connections 2024-09-11T01:56:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.579+0000 7f3bdd616640 1 -- 172.21.15.12:0/948540469 wait complete. 2024-09-11T01:56:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.579+0000 7f3bdd616640 10 monclient: build_initial_monmap 2024-09-11T01:56:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.579+0000 7f3bdd616640 10 monclient: monmap: 2024-09-11T01:56:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:56:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:56:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:56:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:56:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:56:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:56:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:56:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:56:42.592 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bdd616640 1 Processor -- start 2024-09-11T01:56:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bdd616640 1 -- start start 2024-09-11T01:56:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bdd616640 10 monclient: init 2024-09-11T01:56:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bdd616640 10 monclient: _reopen_session rank -1 2024-09-11T01:56:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bdd616640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:56:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bdd616640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3bd80a5020 0x7f3bd8121280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bdd616640 10 monclient(hunting): picked mon.noname-c con 0x7f3bd80a5020 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:56:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bdd616640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bd80a5400 0x7f3bd81380f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bdd616640 10 monclient(hunting): picked mon.noname-a con 0x7f3bd80a5400 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:56:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bdd616640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bd813d230 0x7f3bd81cce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bdd616640 10 monclient(hunting): picked mon.noname-b con 0x7f3bd813d230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:56:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bdd616640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bdd616640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd813fb60 con 0x7f3bd80a5400 2024-09-11T01:56:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bdd616640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bdd616640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd80a6690 con 0x7f3bd813d230 2024-09-11T01:56:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bdd616640 10 monclient(hunting): start opening mon connection 2024-09-11T01:56:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bdd616640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f3bd813fce0 con 0x7f3bd80a5020 2024-09-11T01:56:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bdd616640 10 monclient(hunting): _renew_subs 2024-09-11T01:56:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bdd616640 10 monclient(hunting): authenticate will time out at 2409.023622s 2024-09-11T01:56:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd6ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3bd80a5020 0x7f3bd8121280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd67fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bd80a5400 0x7f3bd81380f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd6ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3bd80a5020 0x7f3bd8121280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:38154/0 (socket says 172.21.15.12:38154) 2024-09-11T01:56:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd6ffd640 1 -- 172.21.15.12:0/247601465 learned_addr learned my addr 172.21.15.12:0/247601465 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:56:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd6ffd640 10 monclient(hunting): get_auth_request con 0x7f3bd80a5020 auth_method 0 2024-09-11T01:56:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:42.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:42.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd67fc640 10 monclient(hunting): get_auth_request con 0x7f3bd80a5400 auth_method 0 2024-09-11T01:56:42.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:56:42.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd67fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:56:42.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:56:42.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd77fe640 1 -- 172.21.15.12:0/247601465 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bd813d230 msgr2=0x7f3bd81cce70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-09-11T01:56:42.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:42.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:42.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd77fe640 1 --2- 172.21.15.12:0/247601465 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bd813d230 0x7f3bd81cce70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:56:42.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:42.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:56:42.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:56:42.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:56:42.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd6ffd640 10 monclient(hunting): handle_auth_done global_id 304187 payload 824 2024-09-11T01:56:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd6ffd640 1 -- 172.21.15.12:0/247601465 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bd813d230 msgr2=0x7f3bd81cce70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:56:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd6ffd640 1 --2- 172.21.15.12:0/247601465 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bd813d230 0x7f3bd81cce70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd6ffd640 1 -- 172.21.15.12:0/247601465 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bd80a5400 msgr2=0x7f3bd81380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd6ffd640 1 --2- 172.21.15.12:0/247601465 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bd80a5400 0x7f3bd81380f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd6ffd640 10 monclient: _finish_hunting 0 2024-09-11T01:56:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd6ffd640 1 monclient: found mon.noname-c 2024-09-11T01:56:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:56:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:56:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd6ffd640 1 -- 172.21.15.12:0/247601465 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bd80a1230 con 0x7f3bd80a5020 2024-09-11T01:56:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.580+0000 7f3bd67fc640 1 --2- 172.21.15.12:0/247601465 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bd80a5400 0x7f3bd81380f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:56:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.583+0000 7f3bd6ffd640 1 -- 172.21.15.12:0/247601465 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3bd80a5020 msgr2=0x7f3bd8121280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:56:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.583+0000 7f3bd6ffd640 1 --2- 172.21.15.12:0/247601465 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3bd80a5020 0x7f3bd8121280 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3bc800ba10 tx=0x7f3bc800bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.583+0000 7f3bcf7fe640 1 -- 172.21.15.12:0/247601465 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc800cd80 con 0x7f3bd80a5020 2024-09-11T01:56:42.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.583+0000 7f3bcf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:42.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.583+0000 7f3bcf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:56:42.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.583+0000 7f3bcf7fe640 10 monclient: dump: 2024-09-11T01:56:42.596 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:42.596 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:42.596 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:42.596 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:42.596 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:42.596 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:42.596 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:42.596 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:42.596 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:42.596 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:42.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.583+0000 7f3bcf7fe640 10 monclient: _finish_auth 0 2024-09-11T01:56:42.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.583+0000 7f3bcf7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:56:42.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.583+0000 7f3bcf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:56:42.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.583+0000 7f3bdd616640 5 monclient: authenticate success, global_id 304187 2024-09-11T01:56:42.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.583+0000 7f3bcf7fe640 1 -- 172.21.15.12:0/247601465 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bc8010070 con 0x7f3bd80a5020 2024-09-11T01:56:42.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.583+0000 7f3bcf7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:56:42.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.583+0000 7f3bcf7fe640 1 -- 172.21.15.12:0/247601465 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc80156b0 con 0x7f3bd80a5020 2024-09-11T01:56:42.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.583+0000 7f3bcf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:56:42.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.583+0000 7f3bcf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:56:42.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.583+0000 7f3bcf7fe640 10 monclient: dump: 2024-09-11T01:56:42.597 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:56:42.597 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:56:42.597 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:42.597 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:56:42.597 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:56:42.598 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:56:42.598 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:56:42.598 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:56:42.598 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:56:42.598 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:56:42.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.583+0000 7f3bd57fa640 4 set_mon_vals no callback set 2024-09-11T01:56:42.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.583+0000 7f3bdd616640 10 monclient: _renew_subs 2024-09-11T01:56:42.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.583+0000 7f3bdd616640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:56:42.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.583+0000 7f3bdd616640 1 -- 172.21.15.12:0/247601465 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bd80a0c10 con 0x7f3bd80a5020 2024-09-11T01:56:42.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.583+0000 7f3bdd616640 10 monclient: _renew_subs 2024-09-11T01:56:42.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.583+0000 7f3bdd616640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:56:42.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.583+0000 7f3bdd616640 1 -- 172.21.15.12:0/247601465 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bd81224c0 con 0x7f3bd80a5020 2024-09-11T01:56:42.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.584+0000 7f3bcf7fe640 1 -- 172.21.15.12:0/247601465 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f3bc801e3a0 con 0x7f3bd80a5020 2024-09-11T01:56:42.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.584+0000 7f3bdd616640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:42.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.584+0000 7f3bdd616640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:42.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.584+0000 7f3bdd616640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:56:42.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.584+0000 7f3bdd616640 1 -- 172.21.15.12:0/247601465 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bd809d420 con 0x7f3bd80a5020 2024-09-11T01:56:42.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.584+0000 7f3bcf7fe640 1 --2- 172.21.15.12:0/247601465 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3bb803e1c0 0x7f3bb8040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:56:42.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.584+0000 7f3bcf7fe640 1 -- 172.21.15.12:0/247601465 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(922..922 src has 889..922) ==== 4815+0+0 (secure 0 0 0) 0x7f3bc8052f40 con 0x7f3bd80a5020 2024-09-11T01:56:42.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.584+0000 7f3bd67fc640 1 --2- 172.21.15.12:0/247601465 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3bb803e1c0 0x7f3bb8040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:56:42.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.585+0000 7f3bd67fc640 10 monclient: get_auth_request con 0x7f3bb803e1c0 auth_method 0 2024-09-11T01:56:42.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.587+0000 7f3bd67fc640 1 --2- 172.21.15.12:0/247601465 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3bb803e1c0 0x7f3bb8040680 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0007c90 tx=0x7f3bc0007be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:56:42.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.589+0000 7f3bcf7fe640 1 -- 172.21.15.12:0/247601465 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3bc8051920 con 0x7f3bd80a5020 2024-09-11T01:56:42.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.589+0000 7f3bcf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:56:42.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.589+0000 7f3bcf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:56:42.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.772+0000 7f3bdd616640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}] 2024-09-11T01:56:42.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.772+0000 7f3bdd616640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}] 2024-09-11T01:56:42.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.772+0000 7f3bdd616640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:56:42.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:42.772+0000 7f3bdd616640 1 -- 172.21.15.12:0/247601465 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3bd80a1870 con 0x7f3bd80a5020 2024-09-11T01:56:43.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:43.744+0000 7f3bcf7fe640 1 -- 172.21.15.12:0/247601465 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]=0 pool 'unique_pool_0' removed v923) ==== 175+0+0 (secure 0 0 0) 0x7f3bc8005a60 con 0x7f3bd80a5020 2024-09-11T01:56:43.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:43.744+0000 7f3bcf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}] 2024-09-11T01:56:43.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:43.744+0000 7f3bcf7fe640 10 monclient: _finish_command 2 = system:0 pool 'unique_pool_0' removed 2024-09-11T01:56:43.747 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'unique_pool_0' removed 2024-09-11T01:56:43.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:43.748+0000 7f3bdd616640 1 -- 172.21.15.12:0/247601465 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3bb803e1c0 msgr2=0x7f3bb8040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:43.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:43.749+0000 7f3bdd616640 1 --2- 172.21.15.12:0/247601465 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3bb803e1c0 0x7f3bb8040680 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0007c90 tx=0x7f3bc0007be0 comp rx=0 tx=0).stop 2024-09-11T01:56:43.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:43.749+0000 7f3bdd616640 10 monclient: shutdown 2024-09-11T01:56:43.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:43.749+0000 7f3bdd616640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:56:43.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:43.749+0000 7f3bdd616640 1 -- 172.21.15.12:0/247601465 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3bd80a5020 msgr2=0x7f3bd8121280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:56:43.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:43.749+0000 7f3bdd616640 1 --2- 172.21.15.12:0/247601465 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3bd80a5020 0x7f3bd8121280 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3bc800ba10 tx=0x7f3bc800bed0 comp rx=0 tx=0).stop 2024-09-11T01:56:43.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:43.749+0000 7f3bdd616640 1 -- 172.21.15.12:0/247601465 shutdown_connections 2024-09-11T01:56:43.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:43.749+0000 7f3bdd616640 1 --2- 172.21.15.12:0/247601465 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bd813d230 0x7f3bd81cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:43.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:43.749+0000 7f3bdd616640 1 --2- 172.21.15.12:0/247601465 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f3bb803e1c0 0x7f3bb8040680 secure :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0007c90 tx=0x7f3bc0007be0 comp rx=0 tx=0).stop 2024-09-11T01:56:43.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:43.749+0000 7f3bdd616640 1 --2- 172.21.15.12:0/247601465 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bd80a5400 0x7f3bd81380f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:56:43.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:43.749+0000 7f3bdd616640 1 --2- 172.21.15.12:0/247601465 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3bd80a5020 0x7f3bd8121280 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3bc800ba10 tx=0x7f3bc800bed0 comp rx=0 tx=0).stop 2024-09-11T01:56:43.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:43.749+0000 7f3bdd616640 1 -- 172.21.15.12:0/247601465 >> 172.21.15.12:0/247601465 conn(0x7f3bd80a7540 msgr2=0x7f3bd813ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:56:43.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:43.750+0000 7f3bdd616640 1 -- 172.21.15.12:0/247601465 shutdown_connections 2024-09-11T01:56:43.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:56:43.750+0000 7f3bdd616640 1 -- 172.21.15.12:0/247601465 wait complete. 2024-09-11T01:56:43.771 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-11T01:56:43.785 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-11T01:57:01.464 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-11T01:57:01.465 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-11T01:57:01.465 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-11T01:57:01.468 INFO:tasks.ceph.mon.b:Started 2024-09-11T01:57:01.468 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-11T01:57:01.468 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-11T01:57:01.468 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T01:57:16.469 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-11T01:57:16.470 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-11T01:57:16.470 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:57:16.470 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:57:16.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.582+0000 7fb14f1f3640 10 monclient: get_monmap_and_config 2024-09-11T01:57:16.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.582+0000 7fb14f1f3640 10 monclient: build_initial_monmap 2024-09-11T01:57:16.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.582+0000 7fb14f1f3640 10 monclient: monmap: 2024-09-11T01:57:16.584 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:16.584 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:16.584 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:57:16.582830+0000 2024-09-11T01:57:16.584 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:57:16.582830+0000 2024-09-11T01:57:16.584 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:16.584 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:16.584 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:57:16.584 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:57:16.584 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:57:16.584 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:57:16.585 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:57:16.585 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:16.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.583+0000 7fb14f1f3640 1 Processor -- start 2024-09-11T01:57:16.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.583+0000 7fb14f1f3640 1 -- start start 2024-09-11T01:57:16.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.583+0000 7fb14f1f3640 10 monclient: init 2024-09-11T01:57:16.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.583+0000 7fb14f1f3640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:16.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.583+0000 7fb14f1f3640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-11T01:57:16.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.583+0000 7fb14f1f3640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb148138210 0x7fb1481385f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:16.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.583+0000 7fb14f1f3640 10 monclient(hunting): picked mon.noname-a con 0x7fb148138210 addr v2:172.21.15.12:3300/0 2024-09-11T01:57:16.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.583+0000 7fb14f1f3640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb1480a6810 0x7fb1480a5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:16.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.583+0000 7fb14f1f3640 10 monclient(hunting): picked mon.noname-b con 0x7fb1480a6810 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:16.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.583+0000 7fb14f1f3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb1480a55d0 addr v1:172.21.15.71:6789/0 2024-09-11T01:57:16.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.583+0000 7fb14f1f3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb14809ba20 con 0x7fb1480a55d0 2024-09-11T01:57:16.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.583+0000 7fb14f1f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:16.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.583+0000 7fb14f1f3640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fb14809c7a0 con 0x7fb148138210 2024-09-11T01:57:16.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.583+0000 7fb14f1f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:16.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.583+0000 7fb14f1f3640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb14809c4c0 con 0x7fb1480a6810 2024-09-11T01:57:16.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.583+0000 7fb14f1f3640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:16.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.583+0000 7fb14f1f3640 10 monclient(hunting): authenticate will time out at 2443.025852s 2024-09-11T01:57:16.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.583+0000 7fb147fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb1480a6810 0x7fb1480a5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:16.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.583+0000 7fb14cf8f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb148138210 0x7fb1481385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:16.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.584+0000 7fb147fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb1480a6810 0x7fb1480a5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52700/0 (socket says 172.21.15.12:52700) 2024-09-11T01:57:16.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.584+0000 7fb14cf8f640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb148138210 0x7fb1481385f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45206/0 (socket says 172.21.15.12:45206) 2024-09-11T01:57:16.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.584+0000 7fb147fff640 1 -- 172.21.15.12:0/1945532287 learned_addr learned my addr 172.21.15.12:0/1945532287 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:16.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.584+0000 7fb147fff640 10 monclient(hunting): get_auth_request con 0x7fb1480a6810 auth_method 0 2024-09-11T01:57:16.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.584+0000 7fb147fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:16.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.584+0000 7fb147fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:16.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.584+0000 7fb147fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:16.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.584+0000 7fb14cf8f640 10 monclient(hunting): get_auth_request con 0x7fb148138210 auth_method 0 2024-09-11T01:57:16.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.584+0000 7fb14cf8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:16.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.584+0000 7fb14cf8f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:16.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.584+0000 7fb14cf8f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:16.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.584+0000 7fb147fff640 1 -- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb1480a6810 msgr2=0x7fb1480a5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:57:16.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.584+0000 7fb14cf8f640 1 -- 172.21.15.12:0/1945532287 >> v2:172.21.15.12:3300/0 conn(0x7fb148138210 msgr2=0x7fb1481385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:57:16.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.584+0000 7fb147fff640 1 -- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb1480a6810 msgr2=0x7fb1480a5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:57:16.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.584+0000 7fb14cf8f640 1 -- 172.21.15.12:0/1945532287 >> v2:172.21.15.12:3300/0 conn(0x7fb148138210 msgr2=0x7fb1481385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:57:16.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.584+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb1480a6810 0x7fb1480a5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:57:16.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.584+0000 7fb14cf8f640 1 --2- 172.21.15.12:0/1945532287 >> v2:172.21.15.12:3300/0 conn(0x7fb148138210 0x7fb1481385f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:57:16.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.584+0000 7fb14cf8f640 1 --2- 172.21.15.12:0/1945532287 >> v2:172.21.15.12:3300/0 conn(0x7fb148138210 0x7fb1481385f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:57:16.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.584+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb1480a6810 0x7fb1480a5090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:57:16.591 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.586+0000 7fb14d790640 1 -- 172.21.15.12:0/1945532287 >> v1:172.21.15.71:6789/0 conn(0x7fb1480a55d0 legacy=0x7fb14813f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:16.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.785+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb1480a6810 0x7fb1480a5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:16.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.785+0000 7fb14cf8f640 1 --2- 172.21.15.12:0/1945532287 >> v2:172.21.15.12:3300/0 conn(0x7fb148138210 0x7fb1481385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:16.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.785+0000 7fb147fff640 10 monclient(hunting): get_auth_request con 0x7fb1480a6810 auth_method 0 2024-09-11T01:57:16.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.785+0000 7fb147fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:16.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.785+0000 7fb147fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:16.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.785+0000 7fb147fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:57:16.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.785+0000 7fb14cf8f640 10 monclient(hunting): get_auth_request con 0x7fb148138210 auth_method 0 2024-09-11T01:57:16.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.785+0000 7fb14cf8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:16.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.785+0000 7fb14cf8f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:16.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.785+0000 7fb14cf8f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:57:16.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.786+0000 7fb147fff640 1 -- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb1480a6810 msgr2=0x7fb1480a5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:57:16.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.786+0000 7fb147fff640 1 -- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb1480a6810 msgr2=0x7fb1480a5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:57:16.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.786+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb1480a6810 0x7fb1480a5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:57:16.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.786+0000 7fb14cf8f640 1 -- 172.21.15.12:0/1945532287 >> v2:172.21.15.12:3300/0 conn(0x7fb148138210 msgr2=0x7fb1481385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:57:16.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.786+0000 7fb14cf8f640 1 -- 172.21.15.12:0/1945532287 >> v2:172.21.15.12:3300/0 conn(0x7fb148138210 msgr2=0x7fb1481385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:57:16.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.786+0000 7fb14cf8f640 1 --2- 172.21.15.12:0/1945532287 >> v2:172.21.15.12:3300/0 conn(0x7fb148138210 0x7fb1481385f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:57:16.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.786+0000 7fb14cf8f640 1 --2- 172.21.15.12:0/1945532287 >> v2:172.21.15.12:3300/0 conn(0x7fb148138210 0x7fb1481385f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:57:16.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:16.786+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb1480a6810 0x7fb1480a5090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:57:17.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.187+0000 7fb14cf8f640 1 --2- 172.21.15.12:0/1945532287 >> v2:172.21.15.12:3300/0 conn(0x7fb148138210 0x7fb1481385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:17.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.187+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb1480a6810 0x7fb1480a5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:17.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.188+0000 7fb14cf8f640 10 monclient(hunting): get_auth_request con 0x7fb148138210 auth_method 0 2024-09-11T01:57:17.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.188+0000 7fb14cf8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:17.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.188+0000 7fb14cf8f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:17.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.188+0000 7fb14cf8f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:57:17.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.188+0000 7fb147fff640 10 monclient(hunting): get_auth_request con 0x7fb1480a6810 auth_method 0 2024-09-11T01:57:17.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.188+0000 7fb147fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:17.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.188+0000 7fb147fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:17.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.188+0000 7fb147fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:57:17.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.188+0000 7fb147fff640 1 -- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb1480a6810 msgr2=0x7fb1480a5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:57:17.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.188+0000 7fb147fff640 1 -- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb1480a6810 msgr2=0x7fb1480a5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:57:17.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.188+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb1480a6810 0x7fb1480a5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:57:17.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.188+0000 7fb14cf8f640 1 -- 172.21.15.12:0/1945532287 >> v2:172.21.15.12:3300/0 conn(0x7fb148138210 msgr2=0x7fb1481385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:57:17.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.188+0000 7fb14cf8f640 1 -- 172.21.15.12:0/1945532287 >> v2:172.21.15.12:3300/0 conn(0x7fb148138210 msgr2=0x7fb1481385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:57:17.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.188+0000 7fb14cf8f640 1 --2- 172.21.15.12:0/1945532287 >> v2:172.21.15.12:3300/0 conn(0x7fb148138210 0x7fb1481385f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:57:17.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.188+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb1480a6810 0x7fb1480a5090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:57:17.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.188+0000 7fb14cf8f640 1 --2- 172.21.15.12:0/1945532287 >> v2:172.21.15.12:3300/0 conn(0x7fb148138210 0x7fb1481385f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:57:17.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.989+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb1480a6810 0x7fb1480a5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:17.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.989+0000 7fb14cf8f640 1 --2- 172.21.15.12:0/1945532287 >> v2:172.21.15.12:3300/0 conn(0x7fb148138210 0x7fb1481385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:17.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.989+0000 7fb147fff640 10 monclient(hunting): get_auth_request con 0x7fb1480a6810 auth_method 0 2024-09-11T01:57:17.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.989+0000 7fb147fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:17.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.989+0000 7fb147fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:17.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.989+0000 7fb147fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:57:17.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.989+0000 7fb14cf8f640 10 monclient(hunting): get_auth_request con 0x7fb148138210 auth_method 0 2024-09-11T01:57:17.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.989+0000 7fb14cf8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:17.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.989+0000 7fb14cf8f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:17.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.989+0000 7fb14cf8f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:57:17.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.989+0000 7fb147fff640 1 -- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb1480a6810 msgr2=0x7fb1480a5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T01:57:17.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.989+0000 7fb147fff640 1 -- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb1480a6810 msgr2=0x7fb1480a5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:57:17.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.989+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb1480a6810 0x7fb1480a5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:57:17.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.989+0000 7fb14cf8f640 1 -- 172.21.15.12:0/1945532287 >> v2:172.21.15.12:3300/0 conn(0x7fb148138210 msgr2=0x7fb1481385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:57:17.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.989+0000 7fb14cf8f640 1 -- 172.21.15.12:0/1945532287 >> v2:172.21.15.12:3300/0 conn(0x7fb148138210 msgr2=0x7fb1481385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:57:17.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.989+0000 7fb14cf8f640 1 --2- 172.21.15.12:0/1945532287 >> v2:172.21.15.12:3300/0 conn(0x7fb148138210 0x7fb1481385f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:57:17.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.989+0000 7fb14cf8f640 1 --2- 172.21.15.12:0/1945532287 >> v2:172.21.15.12:3300/0 conn(0x7fb148138210 0x7fb1481385f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:57:17.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:17.989+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb1480a6810 0x7fb1480a5090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:57:19.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.583+0000 7fb1467fc640 10 monclient(hunting): tick 2024-09-11T01:57:19.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.583+0000 7fb1467fc640 10 monclient(hunting): _check_auth_tickets 2024-09-11T01:57:19.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.583+0000 7fb1467fc640 1 monclient(hunting): continuing hunt 2024-09-11T01:57:19.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.583+0000 7fb1467fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T01:57:19.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.583+0000 7fb1467fc640 1 -- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb1480a6810 msgr2=0x7fb1480a5090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:57:19.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.583+0000 7fb1467fc640 1 --2- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb1480a6810 0x7fb1480a5090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:19.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.583+0000 7fb1467fc640 1 -- 172.21.15.12:0/1945532287 >> v2:172.21.15.12:3300/0 conn(0x7fb148138210 msgr2=0x7fb1481385f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:57:19.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.583+0000 7fb1467fc640 1 --2- 172.21.15.12:0/1945532287 >> v2:172.21.15.12:3300/0 conn(0x7fb148138210 0x7fb1481385f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:19.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.583+0000 7fb1467fc640 1 -- 172.21.15.12:0/1945532287 >> v1:172.21.15.71:6789/0 conn(0x7fb1480a55d0 legacy=0x7fb14813f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:19.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.584+0000 7fb1467fc640 10 monclient: _add_conns ranks=[0,4] 2024-09-11T01:57:19.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.584+0000 7fb1467fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb12c003790 addr v1:172.21.15.12:6789/0 2024-09-11T01:57:19.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.584+0000 7fb1467fc640 1 --2- 172.21.15.12:0/1945532287 >> v2:172.21.15.71:3300/0 conn(0x7fb12c0074a0 0x7fb12c009940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:19.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.584+0000 7fb1467fc640 10 monclient(hunting): picked mon.noname-c con 0x7fb12c0074a0 addr v2:172.21.15.71:3300/0 2024-09-11T01:57:19.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.584+0000 7fb1467fc640 1 -- 172.21.15.12:0/1945532287 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1480a6590 con 0x7fb12c003790 2024-09-11T01:57:19.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.584+0000 7fb1467fc640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:19.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.584+0000 7fb1467fc640 1 -- 172.21.15.12:0/1945532287 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fb14809c4c0 con 0x7fb12c0074a0 2024-09-11T01:57:19.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.584+0000 7fb1467fc640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:19.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.584+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> v2:172.21.15.71:3300/0 conn(0x7fb12c0074a0 0x7fb12c009940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:19.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.584+0000 7fb147fff640 10 monclient(hunting): get_auth_request con 0x7fb12c0074a0 auth_method 0 2024-09-11T01:57:19.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.584+0000 7fb147fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:19.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.584+0000 7fb147fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:19.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.584+0000 7fb147fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:19.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.584+0000 7fb147fff640 1 -- 172.21.15.12:0/1945532287 >> v2:172.21.15.71:3300/0 conn(0x7fb12c0074a0 msgr2=0x7fb12c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:57:19.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.584+0000 7fb147fff640 1 -- 172.21.15.12:0/1945532287 >> v2:172.21.15.71:3300/0 conn(0x7fb12c0074a0 msgr2=0x7fb12c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:57:19.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.584+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> v2:172.21.15.71:3300/0 conn(0x7fb12c0074a0 0x7fb12c009940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:57:19.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.584+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> v2:172.21.15.71:3300/0 conn(0x7fb12c0074a0 0x7fb12c009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:57:19.588 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.586+0000 7fb14cf8f640 1 -- 172.21.15.12:0/1945532287 >> v1:172.21.15.12:6789/0 conn(0x7fb12c003790 legacy=0x7fb12c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:19.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.786+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> v2:172.21.15.71:3300/0 conn(0x7fb12c0074a0 0x7fb12c009940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:19.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.786+0000 7fb147fff640 10 monclient(hunting): get_auth_request con 0x7fb12c0074a0 auth_method 0 2024-09-11T01:57:19.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.786+0000 7fb147fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:19.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.786+0000 7fb147fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:19.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.786+0000 7fb147fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:57:19.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.786+0000 7fb147fff640 1 -- 172.21.15.12:0/1945532287 >> v2:172.21.15.71:3300/0 conn(0x7fb12c0074a0 msgr2=0x7fb12c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:57:19.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.786+0000 7fb147fff640 1 -- 172.21.15.12:0/1945532287 >> v2:172.21.15.71:3300/0 conn(0x7fb12c0074a0 msgr2=0x7fb12c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:57:19.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.786+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> v2:172.21.15.71:3300/0 conn(0x7fb12c0074a0 0x7fb12c009940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:57:19.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:19.786+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> v2:172.21.15.71:3300/0 conn(0x7fb12c0074a0 0x7fb12c009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T01:57:20.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:20.187+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> v2:172.21.15.71:3300/0 conn(0x7fb12c0074a0 0x7fb12c009940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:20.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:20.187+0000 7fb147fff640 10 monclient(hunting): get_auth_request con 0x7fb12c0074a0 auth_method 0 2024-09-11T01:57:20.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:20.187+0000 7fb147fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:20.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:20.187+0000 7fb147fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:20.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:20.187+0000 7fb147fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:57:20.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:20.187+0000 7fb147fff640 1 -- 172.21.15.12:0/1945532287 >> v2:172.21.15.71:3300/0 conn(0x7fb12c0074a0 msgr2=0x7fb12c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:57:20.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:20.187+0000 7fb147fff640 1 -- 172.21.15.12:0/1945532287 >> v2:172.21.15.71:3300/0 conn(0x7fb12c0074a0 msgr2=0x7fb12c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:57:20.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:20.187+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> v2:172.21.15.71:3300/0 conn(0x7fb12c0074a0 0x7fb12c009940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:57:20.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:20.187+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> v2:172.21.15.71:3300/0 conn(0x7fb12c0074a0 0x7fb12c009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T01:57:20.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:20.988+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> v2:172.21.15.71:3300/0 conn(0x7fb12c0074a0 0x7fb12c009940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:20.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:20.988+0000 7fb147fff640 10 monclient(hunting): get_auth_request con 0x7fb12c0074a0 auth_method 0 2024-09-11T01:57:20.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:20.988+0000 7fb147fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:20.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:20.988+0000 7fb147fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:20.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:20.988+0000 7fb147fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:57:20.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:20.989+0000 7fb147fff640 1 -- 172.21.15.12:0/1945532287 >> v2:172.21.15.71:3300/0 conn(0x7fb12c0074a0 msgr2=0x7fb12c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T01:57:20.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:20.989+0000 7fb147fff640 1 -- 172.21.15.12:0/1945532287 >> v2:172.21.15.71:3300/0 conn(0x7fb12c0074a0 msgr2=0x7fb12c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T01:57:20.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:20.989+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> v2:172.21.15.71:3300/0 conn(0x7fb12c0074a0 0x7fb12c009940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:57:20.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:20.989+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> v2:172.21.15.71:3300/0 conn(0x7fb12c0074a0 0x7fb12c009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T01:57:21.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.481+0000 7fb1477fe640 1 -- 172.21.15.12:0/1945532287 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1519594500 0 0) 0x7fb1480a6590 con 0x7fb12c003790 2024-09-11T01:57:21.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.481+0000 7fb1477fe640 10 monclient(hunting): _negotiate 2024-09-11T01:57:21.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.481+0000 7fb1477fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:21.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.481+0000 7fb1477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:21.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.481+0000 7fb1477fe640 10 monclient(hunting): my global_id is 294285 2024-09-11T01:57:21.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.481+0000 7fb1477fe640 1 -- 172.21.15.12:0/1945532287 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb138003520 con 0x7fb12c003790 2024-09-11T01:57:21.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 1 -- 172.21.15.12:0/1945532287 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 900875654 0 0) 0x7fb138003520 con 0x7fb12c003790 2024-09-11T01:57:21.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 1 -- 172.21.15.12:0/1945532287 >> v2:172.21.15.71:3300/0 conn(0x7fb12c0074a0 msgr2=0x7fb12c009940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T01:57:21.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 1 --2- 172.21.15.12:0/1945532287 >> v2:172.21.15.71:3300/0 conn(0x7fb12c0074a0 0x7fb12c009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:21.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 10 monclient: _finish_hunting 0 2024-09-11T01:57:21.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 1 monclient: found mon.noname-a-legacy 2024-09-11T01:57:21.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:21.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2024-09-11T01:57:21.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 1 -- 172.21.15.12:0/1945532287 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb12c00b070 con 0x7fb12c003790 2024-09-11T01:57:21.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 10 monclient: _finish_auth 0 2024-09-11T01:57:21.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 10 monclient: _check_auth_tickets 2024-09-11T01:57:21.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:21.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 1 -- 172.21.15.12:0/1945532287 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 140280696 0 0) 0x7fb13c00d680 con 0x7fb12c003790 2024-09-11T01:57:21.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:21.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T01:57:21.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 10 monclient: dump: 2024-09-11T01:57:21.486 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:21.486 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:21.487 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:21.487 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:21.487 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:21.487 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:21.487 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:21.487 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:21.487 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:21.487 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:21.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2024-09-11T01:57:21.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:21.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 1 -- 172.21.15.12:0/1945532287 >> v1:172.21.15.12:6789/0 conn(0x7fb12c003790 legacy=0x7fb12c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:21.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:57:21.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb14d790640 1 -- 172.21.15.12:0/1945532287 reap_dead start 2024-09-11T01:57:21.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 1 --2- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb138003aa0 0x7fb138003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:21.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 10 monclient(hunting): picked mon.c con 0x7fb138003aa0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:21.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 1 --2- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb138004800 0x7fb138004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:21.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 10 monclient(hunting): picked mon.a con 0x7fb138004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:57:21.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 1 --2- 172.21.15.12:0/1945532287 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb138005410 0x7fb13800ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:21.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 10 monclient(hunting): picked mon.b con 0x7fb138005410 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:57:21.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:21.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 1 -- 172.21.15.12:0/1945532287 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb14809c4c0 con 0x7fb138004800 2024-09-11T01:57:21.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:21.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 1 -- 172.21.15.12:0/1945532287 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb13800c320 con 0x7fb138005410 2024-09-11T01:57:21.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:21.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 1 -- 172.21.15.12:0/1945532287 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb13800c520 con 0x7fb138003aa0 2024-09-11T01:57:21.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:21.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb1477fe640 10 monclient(hunting): _finish_auth 0 2024-09-11T01:57:21.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb138004800 0x7fb138004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:21.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb14cf8f640 1 --2- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb138003aa0 0x7fb138003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:21.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb147fff640 10 monclient(hunting): get_auth_request con 0x7fb138004800 auth_method 0 2024-09-11T01:57:21.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb147fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:21.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb147fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:21.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb147fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:57:21.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb14cf8f640 10 monclient(hunting): get_auth_request con 0x7fb138003aa0 auth_method 0 2024-09-11T01:57:21.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb14cf8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:21.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb14cf8f640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:21.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb14cf8f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:57:21.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb147fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:21.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb147fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:21.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb147fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:57:21.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb14cf8f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:21.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb14cf8f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:21.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb14cf8f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T01:57:21.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.482+0000 7fb14d790640 1 --2- 172.21.15.12:0/1945532287 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb138005410 0x7fb13800ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:21.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.483+0000 7fb14d790640 10 monclient(hunting): get_auth_request con 0x7fb138005410 auth_method 0 2024-09-11T01:57:21.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.483+0000 7fb14d790640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:21.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.483+0000 7fb14d790640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:21.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.483+0000 7fb14d790640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T01:57:21.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.483+0000 7fb147fff640 10 monclient(hunting): handle_auth_done global_id 294285 payload 293 2024-09-11T01:57:21.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.483+0000 7fb147fff640 1 -- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb138003aa0 msgr2=0x7fb138003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:21.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.483+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb138003aa0 0x7fb138003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:21.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.483+0000 7fb147fff640 1 -- 172.21.15.12:0/1945532287 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb138005410 msgr2=0x7fb13800ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:21.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.483+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb138005410 0x7fb13800ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:21.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.483+0000 7fb147fff640 10 monclient: _finish_hunting 0 2024-09-11T01:57:21.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.483+0000 7fb147fff640 1 monclient: found mon.a 2024-09-11T01:57:21.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.483+0000 7fb147fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:21.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.483+0000 7fb147fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:21.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.483+0000 7fb147fff640 1 -- 172.21.15.12:0/1945532287 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb138003520 con 0x7fb138004800 2024-09-11T01:57:21.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.483+0000 7fb147fff640 10 monclient: _finish_auth 0 2024-09-11T01:57:21.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.483+0000 7fb147fff640 10 monclient: _check_auth_tickets 2024-09-11T01:57:21.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.483+0000 7fb147fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:21.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.483+0000 7fb14f1f3640 5 monclient: authenticate success, global_id 294285 2024-09-11T01:57:21.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.483+0000 7fb14f1f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:57:21.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.483+0000 7fb14cf8f640 1 --2- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb138003aa0 0x7fb138003ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:57:21.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.486+0000 7fb147fff640 1 -- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb138004800 msgr2=0x7fb138004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:21.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.486+0000 7fb147fff640 1 --2- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb138004800 0x7fb138004c80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb134019ac0 tx=0x7fb138005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a55d7648a08e0145 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:21.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.486+0000 7fb1477fe640 1 -- 172.21.15.12:0/1945532287 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb134039070 con 0x7fb138004800 2024-09-11T01:57:21.499 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.486+0000 7fb1477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:21.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.486+0000 7fb1477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:21.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.486+0000 7fb1477fe640 10 monclient: dump: 2024-09-11T01:57:21.500 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:21.500 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:21.500 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:21.500 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:21.500 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:21.500 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:21.500 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:21.500 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:21.500 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:21.500 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:21.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.486+0000 7fb14f1f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:57:21.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.486+0000 7fb1477fe640 1 -- 172.21.15.12:0/1945532287 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb13402c840 con 0x7fb138004800 2024-09-11T01:57:21.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.486+0000 7fb1477fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:21.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.486+0000 7fb14f1f3640 10 monclient: get_monmap_and_config success 2024-09-11T01:57:21.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.486+0000 7fb1477fe640 1 -- 172.21.15.12:0/1945532287 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13400f040 con 0x7fb138004800 2024-09-11T01:57:21.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.486+0000 7fb14f1f3640 4 set_mon_vals no callback set 2024-09-11T01:57:21.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.486+0000 7fb1477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:21.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.487+0000 7fb1477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:21.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.487+0000 7fb1477fe640 10 monclient: dump: 2024-09-11T01:57:21.501 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:21.501 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:21.501 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:21.501 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:21.501 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:21.502 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:21.502 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:21.502 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:21.502 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:21.502 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:21.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.487+0000 7fb14f1f3640 10 monclient: shutdown 2024-09-11T01:57:21.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.487+0000 7fb14f1f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:21.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.487+0000 7fb14f1f3640 1 -- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb138004800 msgr2=0x7fb138004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:21.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.487+0000 7fb14f1f3640 1 --2- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb138004800 0x7fb138004c80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb134019ac0 tx=0x7fb138005830 comp rx=0 tx=0).stop 2024-09-11T01:57:21.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.487+0000 7fb14f1f3640 1 -- 172.21.15.12:0/1945532287 shutdown_connections 2024-09-11T01:57:21.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.487+0000 7fb14f1f3640 1 --2- 172.21.15.12:0/1945532287 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb138005410 0x7fb13800ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:21.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.487+0000 7fb14f1f3640 1 --2- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb138004800 0x7fb138004c80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:21.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.487+0000 7fb14f1f3640 1 --2- 172.21.15.12:0/1945532287 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb138003aa0 0x7fb138003ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:21.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.487+0000 7fb14f1f3640 1 -- 172.21.15.12:0/1945532287 >> 172.21.15.12:0/1945532287 conn(0x7fb1480aaa80 msgr2=0x7fb1480aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:21.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.487+0000 7fb14f1f3640 1 -- 172.21.15.12:0/1945532287 shutdown_connections 2024-09-11T01:57:21.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.487+0000 7fb14f1f3640 1 -- 172.21.15.12:0/1945532287 wait complete. 2024-09-11T01:57:21.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.488+0000 7fb14f1f3640 10 monclient: build_initial_monmap 2024-09-11T01:57:21.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.488+0000 7fb14f1f3640 10 monclient: monmap: 2024-09-11T01:57:21.503 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:21.503 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:57:21.503 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:57:21.503 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:57:21.503 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:21.503 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:21.503 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:57:21.503 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:57:21.504 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:57:21.504 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:21.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.489+0000 7fb14f1f3640 1 Processor -- start 2024-09-11T01:57:21.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.489+0000 7fb14f1f3640 1 -- start start 2024-09-11T01:57:21.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.489+0000 7fb14f1f3640 10 monclient: init 2024-09-11T01:57:21.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb14f1f3640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:21.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb14f1f3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:57:21.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb14f1f3640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb138003aa0 0x7fb148123120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:21.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb14f1f3640 10 monclient(hunting): picked mon.noname-c con 0x7fb138003aa0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:21.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb14f1f3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb138004800 0x7fb148136e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:21.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb14f1f3640 10 monclient(hunting): picked mon.noname-a con 0x7fb138004800 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:57:21.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb14f1f3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb138005410 0x7fb1481cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:21.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb14f1f3640 10 monclient(hunting): picked mon.noname-b con 0x7fb138005410 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:57:21.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb14f1f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:21.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb14f1f3640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb14806f4f0 con 0x7fb138004800 2024-09-11T01:57:21.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb14f1f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:21.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb14f1f3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb14809c2a0 con 0x7fb138005410 2024-09-11T01:57:21.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb14f1f3640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:21.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb14f1f3640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb14806f670 con 0x7fb138003aa0 2024-09-11T01:57:21.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb14f1f3640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:21.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb14f1f3640 10 monclient(hunting): authenticate will time out at 2447.932182s 2024-09-11T01:57:21.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb147fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb138004800 0x7fb148136e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:21.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb147fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb138004800 0x7fb148136e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45272/0 (socket says 172.21.15.12:45272) 2024-09-11T01:57:21.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb147fff640 1 -- 172.21.15.12:0/1665663644 learned_addr learned my addr 172.21.15.12:0/1665663644 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:21.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb147fff640 10 monclient(hunting): get_auth_request con 0x7fb138004800 auth_method 0 2024-09-11T01:57:21.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb147fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:21.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb147fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:21.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb147fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:21.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb147fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:21.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb147fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:21.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb147fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:21.506 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb14d790640 1 --2- 172.21.15.12:0/1665663644 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb138005410 0x7fb1481cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:21.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb14d790640 10 monclient(hunting): get_auth_request con 0x7fb138005410 auth_method 0 2024-09-11T01:57:21.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb14d790640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:21.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb14d790640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:21.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb14d790640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:21.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb14d790640 1 -- 172.21.15.12:0/1665663644 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb138005410 msgr2=0x7fb1481cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T01:57:21.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb14d790640 1 -- 172.21.15.12:0/1665663644 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb138005410 msgr2=0x7fb1481cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T01:57:21.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb14d790640 1 --2- 172.21.15.12:0/1665663644 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb138005410 0x7fb1481cc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T01:57:21.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb14d790640 1 --2- 172.21.15.12:0/1665663644 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb138005410 0x7fb1481cc9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T01:57:21.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb147fff640 10 monclient(hunting): handle_auth_done global_id 294288 payload 824 2024-09-11T01:57:21.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb147fff640 1 -- 172.21.15.12:0/1665663644 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb138003aa0 msgr2=0x7fb148123120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:21.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb147fff640 1 --2- 172.21.15.12:0/1665663644 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb138003aa0 0x7fb148123120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:21.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb147fff640 1 -- 172.21.15.12:0/1665663644 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb138005410 msgr2=0x7fb1481cc9f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T01:57:21.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb147fff640 1 --2- 172.21.15.12:0/1665663644 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb138005410 0x7fb1481cc9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:21.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb147fff640 10 monclient: _finish_hunting 0 2024-09-11T01:57:21.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb147fff640 1 monclient: found mon.noname-a 2024-09-11T01:57:21.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb147fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:21.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb147fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:57:21.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.490+0000 7fb147fff640 1 -- 172.21.15.12:0/1665663644 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb148137860 con 0x7fb138004800 2024-09-11T01:57:21.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.493+0000 7fb147fff640 1 -- 172.21.15.12:0/1665663644 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb138004800 msgr2=0x7fb148136e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:21.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.493+0000 7fb147fff640 1 --2- 172.21.15.12:0/1665663644 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb138004800 0x7fb148136e60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb134019ac0 tx=0x7fb1340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:21.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.493+0000 7fb1457fa640 1 -- 172.21.15.12:0/1665663644 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb134043070 con 0x7fb138004800 2024-09-11T01:57:21.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.493+0000 7fb1457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:21.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.493+0000 7fb1457fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:57:21.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.493+0000 7fb1457fa640 10 monclient: dump: 2024-09-11T01:57:21.508 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:21.509 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:21.509 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:21.509 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:21.509 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:21.509 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:21.509 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:21.509 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:21.509 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:21.509 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:21.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.493+0000 7fb1457fa640 10 monclient: _finish_auth 0 2024-09-11T01:57:21.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.493+0000 7fb1457fa640 10 monclient: _check_auth_tickets 2024-09-11T01:57:21.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.493+0000 7fb1457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:21.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.493+0000 7fb1457fa640 1 -- 172.21.15.12:0/1665663644 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb13403e890 con 0x7fb138004800 2024-09-11T01:57:21.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.493+0000 7fb1457fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:21.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.493+0000 7fb1457fa640 1 -- 172.21.15.12:0/1665663644 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb134039040 con 0x7fb138004800 2024-09-11T01:57:21.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.493+0000 7fb1457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:21.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.493+0000 7fb1457fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:21.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.493+0000 7fb1457fa640 10 monclient: dump: 2024-09-11T01:57:21.510 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:21.510 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:21.510 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:21.510 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:21.510 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:21.510 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:21.510 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:21.510 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:21.511 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:21.511 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:21.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.493+0000 7fb14f1f3640 5 monclient: authenticate success, global_id 294288 2024-09-11T01:57:21.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.493+0000 7fb14f1f3640 10 monclient: _renew_subs 2024-09-11T01:57:21.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.493+0000 7fb14f1f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:21.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.493+0000 7fb14f1f3640 1 -- 172.21.15.12:0/1665663644 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb148124920 con 0x7fb138004800 2024-09-11T01:57:21.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.493+0000 7fb146ffd640 4 set_mon_vals no callback set 2024-09-11T01:57:21.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.493+0000 7fb14f1f3640 10 monclient: _renew_subs 2024-09-11T01:57:21.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.493+0000 7fb14f1f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:21.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.493+0000 7fb14f1f3640 1 -- 172.21.15.12:0/1665663644 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1480a1880 con 0x7fb138004800 2024-09-11T01:57:21.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.494+0000 7fb14f1f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:21.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.494+0000 7fb14f1f3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:21.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.494+0000 7fb14f1f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:21.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.494+0000 7fb14f1f3640 1 -- 172.21.15.12:0/1665663644 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1480a52e0 con 0x7fb138004800 2024-09-11T01:57:21.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.495+0000 7fb1457fa640 1 -- 172.21.15.12:0/1665663644 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fb134015040 con 0x7fb138004800 2024-09-11T01:57:21.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.495+0000 7fb1457fa640 1 --2- 172.21.15.12:0/1665663644 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb12c03eca0 0x7fb12c041160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:21.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.495+0000 7fb1457fa640 1 -- 172.21.15.12:0/1665663644 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(923..923 src has 889..923) ==== 4422+0+0 (secure 0 0 0) 0x7fb134074a90 con 0x7fb138004800 2024-09-11T01:57:21.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.495+0000 7fb14cf8f640 1 --2- 172.21.15.12:0/1665663644 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb12c03eca0 0x7fb12c041160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:21.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.495+0000 7fb14cf8f640 10 monclient: get_auth_request con 0x7fb12c03eca0 auth_method 0 2024-09-11T01:57:21.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.498+0000 7fb14cf8f640 1 --2- 172.21.15.12:0/1665663644 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb12c03eca0 0x7fb12c041160 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb13c00d310 tx=0x7fb13c00dd70 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:21.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.500+0000 7fb1457fa640 1 -- 172.21.15.12:0/1665663644 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb13403dce0 con 0x7fb138004800 2024-09-11T01:57:21.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.500+0000 7fb1457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:21.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.500+0000 7fb1457fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:57:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.761+0000 7fb14f1f3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:57:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.761+0000 7fb14f1f3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:57:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.761+0000 7fb14f1f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.761+0000 7fb14f1f3640 1 -- 172.21.15.12:0/1665663644 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb148122110 con 0x7fb138004800 2024-09-11T01:57:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.762+0000 7fb1457fa640 1 -- 172.21.15.12:0/1665663644 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fb134034440 con 0x7fb138004800 2024-09-11T01:57:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.762+0000 7fb1457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:57:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.762+0000 7fb1457fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:57:21.764 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":260,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:57:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.764+0000 7fb14f1f3640 1 -- 172.21.15.12:0/1665663644 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb12c03eca0 msgr2=0x7fb12c041160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.764+0000 7fb14f1f3640 1 --2- 172.21.15.12:0/1665663644 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb12c03eca0 0x7fb12c041160 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb13c00d310 tx=0x7fb13c00dd70 comp rx=0 tx=0).stop 2024-09-11T01:57:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.764+0000 7fb14f1f3640 10 monclient: shutdown 2024-09-11T01:57:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.764+0000 7fb14f1f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.764+0000 7fb14f1f3640 1 -- 172.21.15.12:0/1665663644 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb138004800 msgr2=0x7fb148136e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.764+0000 7fb14f1f3640 1 --2- 172.21.15.12:0/1665663644 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb138004800 0x7fb148136e60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb134019ac0 tx=0x7fb1340046c0 comp rx=0 tx=0).stop 2024-09-11T01:57:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.764+0000 7fb14f1f3640 1 -- 172.21.15.12:0/1665663644 shutdown_connections 2024-09-11T01:57:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.764+0000 7fb14f1f3640 1 --2- 172.21.15.12:0/1665663644 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb138005410 0x7fb1481cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.764+0000 7fb14f1f3640 1 --2- 172.21.15.12:0/1665663644 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb12c03eca0 0x7fb12c041160 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.764+0000 7fb14f1f3640 1 --2- 172.21.15.12:0/1665663644 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb138004800 0x7fb148136e60 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:21.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.764+0000 7fb14f1f3640 1 --2- 172.21.15.12:0/1665663644 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb138003aa0 0x7fb148123120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.764+0000 7fb14f1f3640 1 -- 172.21.15.12:0/1665663644 >> 172.21.15.12:0/1665663644 conn(0x7fb1480aaa80 msgr2=0x7fb1480a8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.764+0000 7fb14f1f3640 1 -- 172.21.15.12:0/1665663644 shutdown_connections 2024-09-11T01:57:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.764+0000 7fb14f1f3640 1 -- 172.21.15.12:0/1665663644 wait complete. 2024-09-11T01:57:21.776 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:57:21.776 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T01:57:21.777 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T01:57:21.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.876+0000 7fb705c16640 10 monclient: get_monmap_and_config 2024-09-11T01:57:21.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.876+0000 7fb705c16640 10 monclient: build_initial_monmap 2024-09-11T01:57:21.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.876+0000 7fb705c16640 10 monclient: monmap: 2024-09-11T01:57:21.878 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:21.878 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:21.879 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:57:21.877663+0000 2024-09-11T01:57:21.879 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:57:21.877663+0000 2024-09-11T01:57:21.879 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:21.879 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:21.879 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:57:21.879 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:57:21.879 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:57:21.879 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:57:21.879 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:57:21.879 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:21.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.877+0000 7fb705c16640 1 Processor -- start 2024-09-11T01:57:21.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.877+0000 7fb705c16640 1 -- start start 2024-09-11T01:57:21.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.877+0000 7fb705c16640 10 monclient: init 2024-09-11T01:57:21.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb705c16640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:21.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb705c16640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-11T01:57:21.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb705c16640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb700074160 0x7fb700074540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:21.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb705c16640 10 monclient(hunting): picked mon.noname-c con 0x7fb700074160 addr v2:172.21.15.71:3300/0 2024-09-11T01:57:21.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb705c16640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb7000a0540 0x7fb700137c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:21.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb705c16640 10 monclient(hunting): picked mon.noname-a con 0x7fb7000a0540 addr v2:172.21.15.12:3300/0 2024-09-11T01:57:21.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb705c16640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb7000a2ab0 addr v1:172.21.15.71:6789/0 2024-09-11T01:57:21.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb705c16640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7000a0230 con 0x7fb7000a2ab0 2024-09-11T01:57:21.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb705c16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:21.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb705c16640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fb700070160 con 0x7fb7000a0540 2024-09-11T01:57:21.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb705c16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:21.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb705c16640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fb700070440 con 0x7fb700074160 2024-09-11T01:57:21.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb705c16640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:21.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb705c16640 10 monclient(hunting): authenticate will time out at 2448.320397s 2024-09-11T01:57:21.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb6f7fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb7000a0540 0x7fb700137c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:21.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb6f7fff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb7000a0540 0x7fb700137c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45280/0 (socket says 172.21.15.12:45280) 2024-09-11T01:57:21.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb6f7fff640 1 -- 172.21.15.12:0/943109032 learned_addr learned my addr 172.21.15.12:0/943109032 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:21.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb6f7fff640 10 monclient(hunting): get_auth_request con 0x7fb7000a0540 auth_method 0 2024-09-11T01:57:21.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb6f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:21.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb6f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:21.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb6f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:21.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb6f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:21.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb6f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:21.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb6f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:21.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb6f7fff640 10 monclient(hunting): handle_auth_done global_id 294291 payload 274 2024-09-11T01:57:21.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb6ff7fe640 1 --2- 172.21.15.12:0/943109032 >> v2:172.21.15.71:3300/0 conn(0x7fb700074160 0x7fb700074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:21.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb6f7fff640 1 -- 172.21.15.12:0/943109032 >> v2:172.21.15.71:3300/0 conn(0x7fb700074160 msgr2=0x7fb700074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:21.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb6f7fff640 1 --2- 172.21.15.12:0/943109032 >> v2:172.21.15.71:3300/0 conn(0x7fb700074160 0x7fb700074540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:21.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb6f7fff640 1 -- 172.21.15.12:0/943109032 >> v1:172.21.15.71:6789/0 conn(0x7fb7000a2ab0 legacy=0x7fb700138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:21.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb6f7fff640 10 monclient: _finish_hunting 0 2024-09-11T01:57:21.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb6f7fff640 1 monclient: found mon.noname-a 2024-09-11T01:57:21.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb6f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:21.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb6f7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:57:21.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb6f7fff640 1 -- 172.21.15.12:0/943109032 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb700070720 con 0x7fb7000a0540 2024-09-11T01:57:21.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb6ff7fe640 10 monclient: get_auth_request con 0x7fb700074160 auth_method 0 2024-09-11T01:57:21.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.878+0000 7fb6ff7fe640 1 --2- 172.21.15.12:0/943109032 >> v2:172.21.15.71:3300/0 conn(0x7fb700074160 0x7fb700074540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:57:21.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.881+0000 7fb6f7fff640 1 -- 172.21.15.12:0/943109032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7000a0540 msgr2=0x7fb700137c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:21.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.881+0000 7fb6f7fff640 1 --2- 172.21.15.12:0/943109032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7000a0540 0x7fb700137c20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb70009ffc0 tx=0x7fb6e802a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b92d0f7795f36a2d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:21.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.881+0000 7fb6feffd640 1 -- 172.21.15.12:0/943109032 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e8037070 con 0x7fb7000a0540 2024-09-11T01:57:21.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.881+0000 7fb6feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:21.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.881+0000 7fb6feffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:57:21.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.881+0000 7fb6feffd640 10 monclient: dump: 2024-09-11T01:57:21.891 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:21.891 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:21.891 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:21.891 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:21.891 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:21.891 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:21.891 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:21.892 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:21.892 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:21.892 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:21.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.881+0000 7fb6feffd640 10 monclient: _finish_auth 0 2024-09-11T01:57:21.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.881+0000 7fb6feffd640 10 monclient: _check_auth_tickets 2024-09-11T01:57:21.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.881+0000 7fb6feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:21.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.881+0000 7fb6feffd640 1 -- 172.21.15.12:0/943109032 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6e8032640 con 0x7fb7000a0540 2024-09-11T01:57:21.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.881+0000 7fb705c16640 5 monclient: authenticate success, global_id 294291 2024-09-11T01:57:21.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.881+0000 7fb705c16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:57:21.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.881+0000 7fb6feffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:21.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.881+0000 7fb705c16640 10 monclient: get_monmap_and_config success 2024-09-11T01:57:21.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.881+0000 7fb6feffd640 1 -- 172.21.15.12:0/943109032 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e8032a10 con 0x7fb7000a0540 2024-09-11T01:57:21.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.881+0000 7fb6feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:21.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.881+0000 7fb705c16640 4 set_mon_vals no callback set 2024-09-11T01:57:21.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.881+0000 7fb6feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:21.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.881+0000 7fb6feffd640 10 monclient: dump: 2024-09-11T01:57:21.893 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:21.893 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:21.893 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:21.893 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:21.893 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:21.893 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:21.893 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:21.893 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:21.893 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:21.893 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:21.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.882+0000 7fb705c16640 10 monclient: shutdown 2024-09-11T01:57:21.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.882+0000 7fb705c16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:21.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.882+0000 7fb705c16640 1 -- 172.21.15.12:0/943109032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7000a0540 msgr2=0x7fb700137c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:21.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.882+0000 7fb705c16640 1 --2- 172.21.15.12:0/943109032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7000a0540 0x7fb700137c20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb70009ffc0 tx=0x7fb6e802a0b0 comp rx=0 tx=0).stop 2024-09-11T01:57:21.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.882+0000 7fb705c16640 1 -- 172.21.15.12:0/943109032 shutdown_connections 2024-09-11T01:57:21.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.882+0000 7fb705c16640 1 --2- 172.21.15.12:0/943109032 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7000a0540 0x7fb700137c20 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:21.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.882+0000 7fb705c16640 1 --2- 172.21.15.12:0/943109032 >> v2:172.21.15.71:3300/0 conn(0x7fb700074160 0x7fb700074540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:21.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.882+0000 7fb705c16640 1 -- 172.21.15.12:0/943109032 >> 172.21.15.12:0/943109032 conn(0x7fb7000a36c0 msgr2=0x7fb7000a3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:21.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.882+0000 7fb705c16640 1 -- 172.21.15.12:0/943109032 shutdown_connections 2024-09-11T01:57:21.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.882+0000 7fb705c16640 1 -- 172.21.15.12:0/943109032 wait complete. 2024-09-11T01:57:21.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.883+0000 7fb705c16640 10 monclient: build_initial_monmap 2024-09-11T01:57:21.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.883+0000 7fb705c16640 10 monclient: monmap: 2024-09-11T01:57:21.894 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:21.894 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:57:21.895 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:57:21.895 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:57:21.895 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:21.895 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:21.895 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:57:21.895 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:57:21.895 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:57:21.895 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:21.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.883+0000 7fb705c16640 1 Processor -- start 2024-09-11T01:57:21.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.883+0000 7fb705c16640 1 -- start start 2024-09-11T01:57:21.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.883+0000 7fb705c16640 10 monclient: init 2024-09-11T01:57:21.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.883+0000 7fb705c16640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:21.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.883+0000 7fb705c16640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:57:21.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.883+0000 7fb705c16640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb700074160 0x7fb7001ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:21.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.883+0000 7fb705c16640 10 monclient(hunting): picked mon.noname-c con 0x7fb700074160 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:21.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.883+0000 7fb705c16640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7000a0540 0x7fb700131a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:21.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.883+0000 7fb705c16640 10 monclient(hunting): picked mon.noname-b con 0x7fb7000a0540 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:57:21.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.883+0000 7fb705c16640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7000a2ab0 0x7fb70011e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:21.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.883+0000 7fb705c16640 10 monclient(hunting): picked mon.noname-a con 0x7fb7000a2ab0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:57:21.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.883+0000 7fb705c16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:21.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.883+0000 7fb705c16640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb700138870 con 0x7fb7000a2ab0 2024-09-11T01:57:21.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.883+0000 7fb705c16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:21.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.883+0000 7fb705c16640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb700068c20 con 0x7fb7000a0540 2024-09-11T01:57:21.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.883+0000 7fb705c16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:21.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb705c16640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb7001389f0 con 0x7fb700074160 2024-09-11T01:57:21.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb705c16640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:21.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb705c16640 10 monclient(hunting): authenticate will time out at 2448.326026s 2024-09-11T01:57:21.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6ff7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb700074160 0x7fb7001ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:21.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6fffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7000a2ab0 0x7fb70011e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:21.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6ff7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb700074160 0x7fb7001ccd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52766/0 (socket says 172.21.15.12:52766) 2024-09-11T01:57:21.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6ff7fe640 1 -- 172.21.15.12:0/453933576 learned_addr learned my addr 172.21.15.12:0/453933576 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:21.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6ff7fe640 10 monclient(hunting): get_auth_request con 0x7fb700074160 auth_method 0 2024-09-11T01:57:21.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:21.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:21.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:21.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6fffff640 10 monclient(hunting): get_auth_request con 0x7fb7000a2ab0 auth_method 0 2024-09-11T01:57:21.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:21.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6fffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:21.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:21.897 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6f7fff640 1 --2- 172.21.15.12:0/453933576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7000a0540 0x7fb700131a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:21.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:21.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:21.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:21.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6f7fff640 10 monclient(hunting): get_auth_request con 0x7fb7000a0540 auth_method 0 2024-09-11T01:57:21.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:21.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:21.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:21.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:21.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:21.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:21.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:21.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:21.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:21.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6fffff640 10 monclient(hunting): handle_auth_done global_id 294294 payload 824 2024-09-11T01:57:21.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6fffff640 1 -- 172.21.15.12:0/453933576 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb700074160 msgr2=0x7fb7001ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:21.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6fffff640 1 --2- 172.21.15.12:0/453933576 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb700074160 0x7fb7001ccd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:21.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6fffff640 1 -- 172.21.15.12:0/453933576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7000a0540 msgr2=0x7fb700131a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:21.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6fffff640 1 --2- 172.21.15.12:0/453933576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7000a0540 0x7fb700131a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:21.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6fffff640 10 monclient: _finish_hunting 0 2024-09-11T01:57:21.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6fffff640 1 monclient: found mon.noname-a 2024-09-11T01:57:21.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:21.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6fffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:57:21.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6fffff640 1 -- 172.21.15.12:0/453933576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb700075000 con 0x7fb7000a2ab0 2024-09-11T01:57:21.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.884+0000 7fb6ff7fe640 1 --2- 172.21.15.12:0/453933576 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb700074160 0x7fb7001ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:57:21.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.886+0000 7fb6fffff640 1 -- 172.21.15.12:0/453933576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7000a2ab0 msgr2=0x7fb70011e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:21.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.886+0000 7fb6fffff640 1 --2- 172.21.15.12:0/453933576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7000a2ab0 0x7fb70011e860 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb6f800c7b0 tx=0x7fb6f800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:21.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.886+0000 7fb6fcff9640 1 -- 172.21.15.12:0/453933576 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8017070 con 0x7fb7000a2ab0 2024-09-11T01:57:21.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.886+0000 7fb6fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:21.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.886+0000 7fb6fcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:57:21.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.886+0000 7fb6fcff9640 10 monclient: dump: 2024-09-11T01:57:21.900 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:21.900 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:21.900 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:21.900 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:21.900 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:21.900 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:21.900 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:21.900 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:21.900 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:21.900 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:21.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.886+0000 7fb6fcff9640 10 monclient: _finish_auth 0 2024-09-11T01:57:21.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.886+0000 7fb6fcff9640 10 monclient: _check_auth_tickets 2024-09-11T01:57:21.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.886+0000 7fb6fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:21.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.886+0000 7fb705c16640 5 monclient: authenticate success, global_id 294294 2024-09-11T01:57:21.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.886+0000 7fb6fcff9640 1 -- 172.21.15.12:0/453933576 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6f8003da0 con 0x7fb7000a2ab0 2024-09-11T01:57:21.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.886+0000 7fb6fcff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:21.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.886+0000 7fb6fcff9640 1 -- 172.21.15.12:0/453933576 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f80052b0 con 0x7fb7000a2ab0 2024-09-11T01:57:21.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.886+0000 7fb705c16640 10 monclient: _renew_subs 2024-09-11T01:57:21.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.886+0000 7fb705c16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:21.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.886+0000 7fb705c16640 1 -- 172.21.15.12:0/453933576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb700121fa0 con 0x7fb7000a2ab0 2024-09-11T01:57:21.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.886+0000 7fb6fdffb640 4 set_mon_vals no callback set 2024-09-11T01:57:21.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.886+0000 7fb6fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:21.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.886+0000 7fb6fcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:21.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.886+0000 7fb6fcff9640 10 monclient: dump: 2024-09-11T01:57:21.902 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:21.902 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:21.902 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:21.902 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:21.902 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:21.902 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:21.902 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:21.902 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:21.902 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:21.902 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:21.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.886+0000 7fb705c16640 10 monclient: _renew_subs 2024-09-11T01:57:21.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.886+0000 7fb705c16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:21.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.886+0000 7fb705c16640 1 -- 172.21.15.12:0/453933576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb700120d60 con 0x7fb7000a2ab0 2024-09-11T01:57:21.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.887+0000 7fb6fcff9640 1 -- 172.21.15.12:0/453933576 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fb6f8005490 con 0x7fb7000a2ab0 2024-09-11T01:57:21.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.887+0000 7fb705c16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:21.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.887+0000 7fb705c16640 1 --2- 172.21.15.12:0/453933576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb700131fb0 0x7fb70011eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:21.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.887+0000 7fb705c16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:21.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.887+0000 7fb705c16640 1 -- 172.21.15.12:0/453933576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb700121130 con 0x7fb700131fb0 2024-09-11T01:57:21.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.887+0000 7fb6ff7fe640 1 --2- 172.21.15.12:0/453933576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb700131fb0 0x7fb70011eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:21.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.887+0000 7fb6ff7fe640 10 monclient: get_auth_request con 0x7fb700131fb0 auth_method 0 2024-09-11T01:57:21.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.887+0000 7fb6ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:21.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.887+0000 7fb6ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:21.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.887+0000 7fb6ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:21.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.888+0000 7fb6ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:21.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.888+0000 7fb6ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:21.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.888+0000 7fb6ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:21.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.888+0000 7fb6fcff9640 1 --2- 172.21.15.12:0/453933576 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb6e003e260 0x7fb6e0040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:21.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.888+0000 7fb6fcff9640 1 -- 172.21.15.12:0/453933576 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(923..923 src has 889..923) ==== 4422+0+0 (secure 0 0 0) 0x7fb6f80525a0 con 0x7fb7000a2ab0 2024-09-11T01:57:21.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.888+0000 7fb6ff7fe640 10 monclient(hunting): handle_auth_done global_id 294297 payload 824 2024-09-11T01:57:21.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.888+0000 7fb6ff7fe640 1 -- 172.21.15.12:0/453933576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb7001360e0 con 0x7fb700131fb0 2024-09-11T01:57:21.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.888+0000 7fb6f7fff640 1 --2- 172.21.15.12:0/453933576 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb6e003e260 0x7fb6e0040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:21.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.888+0000 7fb6f7fff640 10 monclient: get_auth_request con 0x7fb6e003e260 auth_method 0 2024-09-11T01:57:21.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.890+0000 7fb6ff7fe640 1 -- 172.21.15.12:0/453933576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb700131fb0 msgr2=0x7fb70011eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:21.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.890+0000 7fb6ff7fe640 1 --2- 172.21.15.12:0/453933576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb700131fb0 0x7fb70011eda0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb6f000fd10 tx=0x7fb6f000d590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.890+0000 7fb6fcff9640 1 -- 172.21.15.12:0/453933576 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0013850 con 0x7fb700131fb0 2024-09-11T01:57:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.890+0000 7fb6fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.890+0000 7fb6fcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.890+0000 7fb6fcff9640 10 monclient: dump: 2024-09-11T01:57:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:21.907 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:21.908 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:21.908 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:21.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.890+0000 7fb6fcff9640 1 -- 172.21.15.12:0/453933576 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb6f0013850 con 0x7fb700131fb0 2024-09-11T01:57:21.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.890+0000 7fb6fcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:21.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.890+0000 7fb6fcff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:57:21.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.890+0000 7fb6fcff9640 1 -- 172.21.15.12:0/453933576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb700131fb0 msgr2=0x7fb70011eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:21.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.890+0000 7fb6fcff9640 1 --2- 172.21.15.12:0/453933576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb700131fb0 0x7fb70011eda0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb6f000fd10 tx=0x7fb6f000d590 comp rx=0 tx=0).stop 2024-09-11T01:57:21.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.890+0000 7fb6fcff9640 1 -- 172.21.15.12:0/453933576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb700131fb0 msgr2=0x7fb70011eda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:57:21.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.890+0000 7fb6fcff9640 1 --2- 172.21.15.12:0/453933576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb700131fb0 0x7fb70011eda0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:21.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.890+0000 7fb6f7fff640 1 --2- 172.21.15.12:0/453933576 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb6e003e260 0x7fb6e0040720 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8034000 tx=0x7fb6e802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:21.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.897+0000 7fb705c16640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:57:21.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.897+0000 7fb705c16640 1 --2- 172.21.15.12:0/453933576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb70011f2e0 0x7fb700132670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:21.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.897+0000 7fb705c16640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:21.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.897+0000 7fb705c16640 1 -- 172.21.15.12:0/453933576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb70011f6f0 con 0x7fb70011f2e0 2024-09-11T01:57:21.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.897+0000 7fb6ff7fe640 1 --2- 172.21.15.12:0/453933576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb70011f2e0 0x7fb700132670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:21.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.897+0000 7fb6ff7fe640 10 monclient: get_auth_request con 0x7fb70011f2e0 auth_method 0 2024-09-11T01:57:21.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.897+0000 7fb6ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:21.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.897+0000 7fb6ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:21.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.897+0000 7fb6ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:21.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.897+0000 7fb6ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:21.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.897+0000 7fb6ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:21.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.897+0000 7fb6ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:21.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.897+0000 7fb6ff7fe640 10 monclient(hunting): handle_auth_done global_id 294300 payload 824 2024-09-11T01:57:21.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.897+0000 7fb6ff7fe640 1 -- 172.21.15.12:0/453933576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb700074540 con 0x7fb70011f2e0 2024-09-11T01:57:21.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.899+0000 7fb6ff7fe640 1 -- 172.21.15.12:0/453933576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb70011f2e0 msgr2=0x7fb700132670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:21.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.899+0000 7fb6ff7fe640 1 --2- 172.21.15.12:0/453933576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb70011f2e0 0x7fb700132670 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb6f000fd10 tx=0x7fb6f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:21.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.899+0000 7fb6fcff9640 1 -- 172.21.15.12:0/453933576 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0000e60 con 0x7fb70011f2e0 2024-09-11T01:57:21.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.899+0000 7fb6fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:21.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.899+0000 7fb6fcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:21.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.899+0000 7fb6fcff9640 10 monclient: dump: 2024-09-11T01:57:21.910 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:21.910 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:21.910 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:21.910 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:21.910 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:21.910 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:21.910 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:21.910 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:21.911 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:21.911 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:21.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.900+0000 7fb6fcff9640 1 -- 172.21.15.12:0/453933576 <== mon.0 v2:172.21.15.12:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7fb6f0026070 con 0x7fb70011f2e0 2024-09-11T01:57:21.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.900+0000 7fb6fcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:57:21.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.900+0000 7fb6fcff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:57:21.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.900+0000 7fb6fcff9640 1 -- 172.21.15.12:0/453933576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb70011f2e0 msgr2=0x7fb700132670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:21.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.900+0000 7fb6fcff9640 1 --2- 172.21.15.12:0/453933576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb70011f2e0 0x7fb700132670 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb6f000fd10 tx=0x7fb6f00046c0 comp rx=0 tx=0).stop 2024-09-11T01:57:21.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.900+0000 7fb6fcff9640 1 -- 172.21.15.12:0/453933576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb70011f2e0 msgr2=0x7fb700132670 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:57:21.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.900+0000 7fb6fcff9640 1 --2- 172.21.15.12:0/453933576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb70011f2e0 0x7fb700132670 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb6f000fd10 tx=0x7fb6f00046c0 comp rx=0 tx=0).stop 2024-09-11T01:57:21.911 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:57:21.912 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:57:21.912 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:57:21.912 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "leader", 2024-09-11T01:57:21.912 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 260, 2024-09-11T01:57:21.912 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 121853, 2024-09-11T01:57:21.912 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:57:21.912 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:57:21.912 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:57:21.912 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:57:21.912 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:57:21.912 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:57:21.912 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:57:21.913 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:57:21.913 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:57:21.913 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:57:21.913 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:57:21.913 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:57:21.913 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:57:21.913 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:57:21.913 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:57:21.913 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:57:21.913 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:57:21.913 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:57:21.913 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:57:21.913 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:57:21.913 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:57:21.914 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:57:21.914 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:57:21.914 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:57:21.914 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:57:21.914 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:57:21.914 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:57:21.914 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:57:21.914 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:57:21.914 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:57:21.914 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:57:21.914 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:57:21.914 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:57:21.914 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:57:21.914 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:57:21.915 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:21.915 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:57:21.915 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:57:21.915 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:57:21.915 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:57:21.915 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:57:21.915 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:57:21.915 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:57:21.915 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:57:21.915 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:57:21.915 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:57:21.915 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:57:21.915 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:57:21.916 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:57:21.916 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:57:21.916 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:57:21.916 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:57:21.916 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:57:21.916 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:57:21.916 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:57:21.916 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:57:21.916 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:57:21.916 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:57:21.916 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:57:21.916 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:57:21.916 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:57:21.917 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:57:21.917 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:57:21.917 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:57:21.917 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:57:21.917 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:57:21.917 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:21.917 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:57:21.917 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:21.917 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:57:21.917 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:57:21.917 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:57:21.917 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:57:21.917 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:21.917 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:57:21.918 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:57:21.918 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:57:21.918 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:21.918 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:21.918 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:57:21.918 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:57:21.918 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:57:21.918 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:57:21.918 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:57:21.918 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:21.918 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:57:21.918 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:57:21.918 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:57:21.918 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:57:21.919 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:57:21.919 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:21.919 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:21.919 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:57:21.919 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:57:21.919 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:57:21.919 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:57:21.919 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:21.919 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:57:21.919 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:57:21.919 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:57:21.919 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:21.919 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:21.919 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:57:21.920 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:57:21.920 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:57:21.920 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:57:21.920 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:57:21.920 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:21.920 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:57:21.920 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:57:21.920 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:57:21.920 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:57:21.920 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:57:21.920 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:21.920 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:21.920 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:57:21.921 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:57:21.921 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:57:21.921 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:57:21.921 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:21.921 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:57:21.921 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:57:21.921 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:57:21.921 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:21.921 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:21.921 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:57:21.921 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:57:21.921 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:57:21.921 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:57:21.921 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:57:21.922 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:21.922 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:57:21.922 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:57:21.922 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:57:21.922 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:57:21.922 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:57:21.922 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:57:21.922 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:57:21.922 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:21.922 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:57:21.922 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:57:21.922 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:21.922 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:57:21.922 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:57:21.923 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:57:21.923 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:57:21.923 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:57:21.923 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:57:21.923 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:21.923 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:57:21.923 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:57:21.923 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:57:21.923 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:57:21.923 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:57:21.923 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:57:21.923 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:21.923 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:57:21.923 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:57:21.924 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:57:21.924 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:57:21.924 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:57:21.924 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:21.924 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:57:21.924 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:57:21.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.900+0000 7fb705c16640 1 -- 172.21.15.12:0/453933576 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb6e003e260 msgr2=0x7fb6e0040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:21.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.900+0000 7fb705c16640 1 --2- 172.21.15.12:0/453933576 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb6e003e260 0x7fb6e0040720 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8034000 tx=0x7fb6e802e000 comp rx=0 tx=0).stop 2024-09-11T01:57:21.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.900+0000 7fb705c16640 10 monclient: shutdown 2024-09-11T01:57:21.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.900+0000 7fb705c16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:21.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.900+0000 7fb705c16640 1 -- 172.21.15.12:0/453933576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7000a2ab0 msgr2=0x7fb70011e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:21.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.900+0000 7fb705c16640 1 --2- 172.21.15.12:0/453933576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb7000a2ab0 0x7fb70011e860 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb6f800c7b0 tx=0x7fb6f800cc70 comp rx=0 tx=0).stop 2024-09-11T01:57:21.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.900+0000 7fb6fffff640 1 -- 172.21.15.12:0/453933576 reap_dead start 2024-09-11T01:57:21.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.900+0000 7fb6fffff640 1 -- 172.21.15.12:0/453933576 reap_dead start 2024-09-11T01:57:21.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.900+0000 7fb705c16640 1 -- 172.21.15.12:0/453933576 shutdown_connections 2024-09-11T01:57:21.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.900+0000 7fb705c16640 1 -- 172.21.15.12:0/453933576 >> 172.21.15.12:0/453933576 conn(0x7fb7000a36c0 msgr2=0x7fb7001bd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:21.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.900+0000 7fb705c16640 1 -- 172.21.15.12:0/453933576 shutdown_connections 2024-09-11T01:57:21.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:21.900+0000 7fb705c16640 1 -- 172.21.15.12:0/453933576 wait complete. 2024-09-11T01:57:21.927 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:57:21.927 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T01:57:21.927 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T01:57:22.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.026+0000 7f4b81ac6640 10 monclient: get_monmap_and_config 2024-09-11T01:57:22.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.026+0000 7f4b81ac6640 10 monclient: build_initial_monmap 2024-09-11T01:57:22.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.026+0000 7f4b81ac6640 10 monclient: monmap: 2024-09-11T01:57:22.028 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:22.028 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:22.028 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:57:22.027628+0000 2024-09-11T01:57:22.029 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:57:22.027628+0000 2024-09-11T01:57:22.029 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:22.029 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:22.029 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:57:22.029 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:57:22.029 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:57:22.029 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:57:22.029 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:57:22.029 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:22.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.027+0000 7f4b81ac6640 1 Processor -- start 2024-09-11T01:57:22.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.027+0000 7f4b81ac6640 1 -- start start 2024-09-11T01:57:22.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.027+0000 7f4b81ac6640 10 monclient: init 2024-09-11T01:57:22.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b81ac6640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:22.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b81ac6640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-11T01:57:22.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b81ac6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4b7c135fc0 addr v1:172.21.15.71:6789/0 2024-09-11T01:57:22.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b81ac6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4b7c074000 addr v1:172.21.15.12:6789/0 2024-09-11T01:57:22.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b81ac6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c074410 0x7f4b7c13e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:22.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b81ac6640 10 monclient(hunting): picked mon.noname-b con 0x7f4b7c074410 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:22.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b81ac6640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b7c0732d0 con 0x7f4b7c074000 2024-09-11T01:57:22.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b81ac6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b7c073890 con 0x7f4b7c135fc0 2024-09-11T01:57:22.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b81ac6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:22.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b81ac6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4b7c074820 con 0x7f4b7c074410 2024-09-11T01:57:22.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b81ac6640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:22.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b81ac6640 10 monclient(hunting): authenticate will time out at 2448.470363s 2024-09-11T01:57:22.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b7affd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4b7c074000 0x7f4b7c13ae40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:39148/0 (socket says 172.21.15.12:39148) 2024-09-11T01:57:22.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b7affd640 1 -- 172.21.15.12:0/4125508086 learned_addr learned my addr 172.21.15.12:0/4125508086 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:57:22.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b7bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c074410 0x7f4b7c13e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:22.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b7bfff640 10 monclient(hunting): get_auth_request con 0x7f4b7c074410 auth_method 0 2024-09-11T01:57:22.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:22.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:22.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:22.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:22.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:22.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:22.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b7bfff640 10 monclient(hunting): handle_auth_done global_id 304193 payload 274 2024-09-11T01:57:22.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b7bfff640 1 -- 172.21.15.12:0/4125508086 >> v1:172.21.15.71:6789/0 conn(0x7f4b7c135fc0 legacy=0x7f4b7c1363a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:22.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b7bfff640 1 -- 172.21.15.12:0/4125508086 >> v1:172.21.15.12:6789/0 conn(0x7f4b7c074000 legacy=0x7f4b7c13ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:22.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b7bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:57:22.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b7bfff640 1 monclient: found mon.noname-b 2024-09-11T01:57:22.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:22.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b7bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:57:22.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.028+0000 7f4b7bfff640 1 -- 172.21.15.12:0/4125508086 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b7c072000 con 0x7f4b7c074410 2024-09-11T01:57:22.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.031+0000 7f4b7bfff640 1 -- 172.21.15.12:0/4125508086 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c074410 msgr2=0x7f4b7c13e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:22.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.031+0000 7f4b7bfff640 1 --2- 172.21.15.12:0/4125508086 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c074410 0x7f4b7c13e570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4b74009dc0 tx=0x7f4b7402a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16ba459b63770a6a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:22.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.031+0000 7f4b7a7fc640 1 -- 172.21.15.12:0/4125508086 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b74007a30 con 0x7f4b7c074410 2024-09-11T01:57:22.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.031+0000 7f4b7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:22.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.031+0000 7f4b7a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:57:22.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.031+0000 7f4b7a7fc640 10 monclient: dump: 2024-09-11T01:57:22.041 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:22.041 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:22.041 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.041 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.041 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:22.041 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:22.041 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:22.041 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:22.041 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:22.041 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:22.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.031+0000 7f4b7a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:57:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.031+0000 7f4b7a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:57:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.031+0000 7f4b7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.031+0000 7f4b7a7fc640 1 -- 172.21.15.12:0/4125508086 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b74007bd0 con 0x7f4b7c074410 2024-09-11T01:57:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.031+0000 7f4b81ac6640 5 monclient: authenticate success, global_id 304193 2024-09-11T01:57:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.031+0000 7f4b81ac6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:57:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.031+0000 7f4b7a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.031+0000 7f4b7a7fc640 1 -- 172.21.15.12:0/4125508086 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b74031400 con 0x7f4b7c074410 2024-09-11T01:57:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.031+0000 7f4b7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.031+0000 7f4b7a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:57:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.031+0000 7f4b7a7fc640 10 monclient: dump: 2024-09-11T01:57:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:22.042 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.043 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.043 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:22.043 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:22.043 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:22.043 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:22.043 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:22.043 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:22.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.031+0000 7f4b81ac6640 10 monclient: get_monmap_and_config success 2024-09-11T01:57:22.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.031+0000 7f4b81ac6640 4 set_mon_vals no callback set 2024-09-11T01:57:22.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.032+0000 7f4b81ac6640 10 monclient: shutdown 2024-09-11T01:57:22.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.032+0000 7f4b81ac6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:22.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.032+0000 7f4b81ac6640 1 -- 172.21.15.12:0/4125508086 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c074410 msgr2=0x7f4b7c13e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:22.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.032+0000 7f4b81ac6640 1 --2- 172.21.15.12:0/4125508086 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c074410 0x7f4b7c13e570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4b74009dc0 tx=0x7f4b7402a480 comp rx=0 tx=0).stop 2024-09-11T01:57:22.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.032+0000 7f4b81ac6640 1 -- 172.21.15.12:0/4125508086 shutdown_connections 2024-09-11T01:57:22.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.032+0000 7f4b81ac6640 1 --2- 172.21.15.12:0/4125508086 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c074410 0x7f4b7c13e570 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.032+0000 7f4b81ac6640 1 -- 172.21.15.12:0/4125508086 >> 172.21.15.12:0/4125508086 conn(0x7f4b7c0a8600 msgr2=0x7f4b7c0a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:22.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.032+0000 7f4b81ac6640 1 -- 172.21.15.12:0/4125508086 shutdown_connections 2024-09-11T01:57:22.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.032+0000 7f4b81ac6640 1 -- 172.21.15.12:0/4125508086 wait complete. 2024-09-11T01:57:22.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.033+0000 7f4b81ac6640 10 monclient: build_initial_monmap 2024-09-11T01:57:22.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.033+0000 7f4b81ac6640 10 monclient: monmap: 2024-09-11T01:57:22.044 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:22.044 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:57:22.044 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:57:22.044 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:57:22.044 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:22.044 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:22.045 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:57:22.045 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:57:22.045 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:57:22.045 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:22.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.033+0000 7f4b81ac6640 1 Processor -- start 2024-09-11T01:57:22.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.033+0000 7f4b81ac6640 1 -- start start 2024-09-11T01:57:22.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.033+0000 7f4b81ac6640 10 monclient: init 2024-09-11T01:57:22.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.033+0000 7f4b81ac6640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:22.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.033+0000 7f4b81ac6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:57:22.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.033+0000 7f4b81ac6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c074000 0x7f4b7c13cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:22.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.033+0000 7f4b81ac6640 10 monclient(hunting): picked mon.noname-c con 0x7f4b7c074000 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:22.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.033+0000 7f4b81ac6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b7c074410 0x7f4b7c134d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:22.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b81ac6640 10 monclient(hunting): picked mon.noname-b con 0x7f4b7c074410 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:57:22.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b81ac6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4b7c135fc0 0x7f4b7c1caf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:22.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b81ac6640 10 monclient(hunting): picked mon.noname-a con 0x7f4b7c135fc0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:57:22.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b81ac6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:22.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b81ac6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7c13ec30 con 0x7f4b7c135fc0 2024-09-11T01:57:22.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b81ac6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:22.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b81ac6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7c13eab0 con 0x7f4b7c074410 2024-09-11T01:57:22.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b81ac6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:22.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b81ac6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4b7c13edb0 con 0x7f4b7c074000 2024-09-11T01:57:22.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b81ac6640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:22.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b81ac6640 10 monclient(hunting): authenticate will time out at 2448.476042s 2024-09-11T01:57:22.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c074000 0x7f4b7c13cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:22.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4b7c135fc0 0x7f4b7c1caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:22.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4b7c135fc0 0x7f4b7c1caf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45326/0 (socket says 172.21.15.12:45326) 2024-09-11T01:57:22.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c074000 0x7f4b7c13cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52782/0 (socket says 172.21.15.12:52782) 2024-09-11T01:57:22.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7bfff640 1 -- 172.21.15.12:0/587184052 learned_addr learned my addr 172.21.15.12:0/587184052 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:22.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7bfff640 10 monclient(hunting): get_auth_request con 0x7f4b7c135fc0 auth_method 0 2024-09-11T01:57:22.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:22.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:22.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:22.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7b7fe640 10 monclient(hunting): get_auth_request con 0x7f4b7c074000 auth_method 0 2024-09-11T01:57:22.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:22.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:22.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:22.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:22.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:22.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:22.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:22.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:22.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:22.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7affd640 1 --2- 172.21.15.12:0/587184052 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b7c074410 0x7f4b7c134d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:22.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7affd640 10 monclient(hunting): get_auth_request con 0x7f4b7c074410 auth_method 0 2024-09-11T01:57:22.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:22.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:22.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:22.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7b7fe640 10 monclient(hunting): handle_auth_done global_id 304196 payload 824 2024-09-11T01:57:22.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7b7fe640 1 -- 172.21.15.12:0/587184052 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b7c074410 msgr2=0x7f4b7c134d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:22.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7b7fe640 1 --2- 172.21.15.12:0/587184052 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b7c074410 0x7f4b7c134d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7b7fe640 1 -- 172.21.15.12:0/587184052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4b7c135fc0 msgr2=0x7f4b7c1caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:22.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7b7fe640 1 --2- 172.21.15.12:0/587184052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4b7c135fc0 0x7f4b7c1caf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:57:22.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7b7fe640 1 monclient: found mon.noname-c 2024-09-11T01:57:22.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:22.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:57:22.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7b7fe640 1 -- 172.21.15.12:0/587184052 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b7c13e420 con 0x7f4b7c074000 2024-09-11T01:57:22.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7affd640 1 --2- 172.21.15.12:0/587184052 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b7c074410 0x7f4b7c134d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:57:22.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.034+0000 7f4b7bfff640 1 --2- 172.21.15.12:0/587184052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4b7c135fc0 0x7f4b7c1caf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:57:22.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.036+0000 7f4b7b7fe640 1 -- 172.21.15.12:0/587184052 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c074000 msgr2=0x7f4b7c13cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:22.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.036+0000 7f4b7b7fe640 1 --2- 172.21.15.12:0/587184052 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c074000 0x7f4b7c13cf00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c00c410 tx=0x7f4b6c00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:22.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.036+0000 7f4b63fff640 1 -- 172.21.15.12:0/587184052 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c017070 con 0x7f4b7c074000 2024-09-11T01:57:22.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.036+0000 7f4b63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:22.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.036+0000 7f4b63fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:57:22.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.036+0000 7f4b63fff640 10 monclient: dump: 2024-09-11T01:57:22.049 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:22.049 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:22.050 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.036+0000 7f4b63fff640 10 monclient: _finish_auth 0 2024-09-11T01:57:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.036+0000 7f4b63fff640 10 monclient: _check_auth_tickets 2024-09-11T01:57:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.036+0000 7f4b63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.036+0000 7f4b63fff640 1 -- 172.21.15.12:0/587184052 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b6c005230 con 0x7f4b7c074000 2024-09-11T01:57:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.036+0000 7f4b63fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:22.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.036+0000 7f4b63fff640 1 -- 172.21.15.12:0/587184052 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c0117a0 con 0x7f4b7c074000 2024-09-11T01:57:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.036+0000 7f4b81ac6640 5 monclient: authenticate success, global_id 304196 2024-09-11T01:57:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.036+0000 7f4b63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.036+0000 7f4b63fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:57:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.036+0000 7f4b63fff640 10 monclient: dump: 2024-09-11T01:57:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:22.051 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:22.051 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:22.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.036+0000 7f4b81ac6640 10 monclient: _renew_subs 2024-09-11T01:57:22.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.036+0000 7f4b79ffb640 4 set_mon_vals no callback set 2024-09-11T01:57:22.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.036+0000 7f4b81ac6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:22.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.036+0000 7f4b81ac6640 1 -- 172.21.15.12:0/587184052 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b7c13e700 con 0x7f4b7c074000 2024-09-11T01:57:22.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.037+0000 7f4b81ac6640 10 monclient: _renew_subs 2024-09-11T01:57:22.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.037+0000 7f4b81ac6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:22.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.037+0000 7f4b81ac6640 1 -- 172.21.15.12:0/587184052 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b7c13c3a0 con 0x7f4b7c074000 2024-09-11T01:57:22.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.037+0000 7f4b81ac6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:22.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.037+0000 7f4b81ac6640 1 --2- 172.21.15.12:0/587184052 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c13adc0 0x7f4b7c1cb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:22.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.037+0000 7f4b81ac6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:22.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.037+0000 7f4b81ac6640 1 -- 172.21.15.12:0/587184052 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4b7c06a060 con 0x7f4b7c13adc0 2024-09-11T01:57:22.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.037+0000 7f4b7affd640 1 --2- 172.21.15.12:0/587184052 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c13adc0 0x7f4b7c1cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:22.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.037+0000 7f4b7affd640 10 monclient: get_auth_request con 0x7f4b7c13adc0 auth_method 0 2024-09-11T01:57:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.037+0000 7f4b7affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.037+0000 7f4b7affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.037+0000 7f4b7affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.038+0000 7f4b7affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.038+0000 7f4b63fff640 1 -- 172.21.15.12:0/587184052 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f4b6c011980 con 0x7f4b7c074000 2024-09-11T01:57:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.038+0000 7f4b7affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.038+0000 7f4b7affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.038+0000 7f4b7affd640 10 monclient(hunting): handle_auth_done global_id 304199 payload 824 2024-09-11T01:57:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.038+0000 7f4b7affd640 1 -- 172.21.15.12:0/587184052 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4b7c13dfe0 con 0x7f4b7c13adc0 2024-09-11T01:57:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.038+0000 7f4b63fff640 1 --2- 172.21.15.12:0/587184052 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4b6803e260 0x7f4b68040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.038+0000 7f4b63fff640 1 -- 172.21.15.12:0/587184052 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(923..923 src has 889..923) ==== 4422+0+0 (secure 0 0 0) 0x7f4b6c052220 con 0x7f4b7c074000 2024-09-11T01:57:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.038+0000 7f4b7bfff640 1 --2- 172.21.15.12:0/587184052 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4b6803e260 0x7f4b68040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:22.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.038+0000 7f4b7bfff640 10 monclient: get_auth_request con 0x7f4b6803e260 auth_method 0 2024-09-11T01:57:22.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.040+0000 7f4b7affd640 1 -- 172.21.15.12:0/587184052 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c13adc0 msgr2=0x7f4b7c1cb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:22.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.040+0000 7f4b7affd640 1 --2- 172.21.15.12:0/587184052 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c13adc0 0x7f4b7c1cb490 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4b7000e4f0 tx=0x7f4b7000e9b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:22.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.040+0000 7f4b63fff640 1 -- 172.21.15.12:0/587184052 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b70005cf0 con 0x7f4b7c13adc0 2024-09-11T01:57:22.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.040+0000 7f4b63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:22.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.040+0000 7f4b63fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:57:22.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.040+0000 7f4b63fff640 10 monclient: dump: 2024-09-11T01:57:22.056 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:22.057 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.040+0000 7f4b63fff640 1 -- 172.21.15.12:0/587184052 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4b70018070 con 0x7f4b7c13adc0 2024-09-11T01:57:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.040+0000 7f4b63fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.040+0000 7f4b63fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:57:22.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.040+0000 7f4b63fff640 1 -- 172.21.15.12:0/587184052 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c13adc0 msgr2=0x7f4b7c1cb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.040+0000 7f4b63fff640 1 --2- 172.21.15.12:0/587184052 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c13adc0 0x7f4b7c1cb490 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4b7000e4f0 tx=0x7f4b7000e9b0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.040+0000 7f4b63fff640 1 -- 172.21.15.12:0/587184052 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c13adc0 msgr2=0x7f4b7c1cb490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:57:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.040+0000 7f4b63fff640 1 --2- 172.21.15.12:0/587184052 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c13adc0 0x7f4b7c1cb490 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4b7000e4f0 tx=0x7f4b7000e9b0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.041+0000 7f4b7bfff640 1 --2- 172.21.15.12:0/587184052 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4b6803e260 0x7f4b68040720 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f4b74009bd0 tx=0x7f4b7402f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.047+0000 7f4b81ac6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:57:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.047+0000 7f4b81ac6640 1 --2- 172.21.15.12:0/587184052 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c1cb9d0 0x7f4b7c136bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.047+0000 7f4b81ac6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.047+0000 7f4b81ac6640 1 -- 172.21.15.12:0/587184052 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4b7c1cbde0 con 0x7f4b7c1cb9d0 2024-09-11T01:57:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.047+0000 7f4b7affd640 1 --2- 172.21.15.12:0/587184052 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c1cb9d0 0x7f4b7c136bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.047+0000 7f4b7affd640 10 monclient: get_auth_request con 0x7f4b7c1cb9d0 auth_method 0 2024-09-11T01:57:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.047+0000 7f4b7affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.047+0000 7f4b7affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:22.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.047+0000 7f4b7affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.047+0000 7f4b7affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.047+0000 7f4b7affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.047+0000 7f4b7affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.047+0000 7f4b7affd640 10 monclient(hunting): handle_auth_done global_id 304202 payload 824 2024-09-11T01:57:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.047+0000 7f4b7affd640 1 -- 172.21.15.12:0/587184052 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4b7c13bde0 con 0x7f4b7c1cb9d0 2024-09-11T01:57:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.049+0000 7f4b7affd640 1 -- 172.21.15.12:0/587184052 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c1cb9d0 msgr2=0x7f4b7c136bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.049+0000 7f4b7affd640 1 --2- 172.21.15.12:0/587184052 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c1cb9d0 0x7f4b7c136bf0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4b70017000 tx=0x7f4b70004940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.049+0000 7f4b63fff640 1 -- 172.21.15.12:0/587184052 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b700072e0 con 0x7f4b7c1cb9d0 2024-09-11T01:57:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.049+0000 7f4b63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.049+0000 7f4b63fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:57:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.049+0000 7f4b63fff640 10 monclient: dump: 2024-09-11T01:57:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:22.059 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:22.060 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.049+0000 7f4b63fff640 1 -- 172.21.15.12:0/587184052 <== mon.2 v2:172.21.15.12:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7f4b700072e0 con 0x7f4b7c1cb9d0 2024-09-11T01:57:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.049+0000 7f4b63fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:57:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.049+0000 7f4b63fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:57:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.050+0000 7f4b63fff640 1 -- 172.21.15.12:0/587184052 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c1cb9d0 msgr2=0x7f4b7c136bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.050+0000 7f4b63fff640 1 --2- 172.21.15.12:0/587184052 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c1cb9d0 0x7f4b7c136bf0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4b70017000 tx=0x7f4b70004940 comp rx=0 tx=0).stop 2024-09-11T01:57:22.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.050+0000 7f4b63fff640 1 -- 172.21.15.12:0/587184052 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c1cb9d0 msgr2=0x7f4b7c136bf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:57:22.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.050+0000 7f4b63fff640 1 --2- 172.21.15.12:0/587184052 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c1cb9d0 0x7f4b7c136bf0 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4b70017000 tx=0x7f4b70004940 comp rx=0 tx=0).stop 2024-09-11T01:57:22.061 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:57:22.061 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:57:22.061 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:57:22.061 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:57:22.061 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 260, 2024-09-11T01:57:22.061 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 77005, 2024-09-11T01:57:22.061 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:57:22.061 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:57:22.061 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:57:22.062 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:57:22.062 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:57:22.062 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:57:22.062 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:57:22.062 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:57:22.062 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:57:22.062 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:57:22.062 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:57:22.062 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:57:22.062 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:57:22.062 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:57:22.062 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:57:22.062 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:57:22.062 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:57:22.063 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:57:22.063 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:57:22.063 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:57:22.063 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:57:22.063 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:57:22.063 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:57:22.063 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:57:22.063 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:57:22.063 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:57:22.063 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:57:22.063 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:57:22.063 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:57:22.063 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:57:22.064 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:57:22.064 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:57:22.064 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:57:22.064 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:57:22.064 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:57:22.064 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:22.064 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:57:22.064 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:57:22.064 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:57:22.064 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:57:22.064 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:57:22.064 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:57:22.064 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:57:22.065 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:57:22.065 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:57:22.065 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:57:22.065 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:57:22.065 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:57:22.065 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:57:22.065 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:57:22.065 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:57:22.065 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:57:22.065 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:57:22.065 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:57:22.065 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:57:22.065 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:57:22.065 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:57:22.066 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:57:22.066 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:57:22.066 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:57:22.066 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:57:22.066 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:57:22.066 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:57:22.066 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:57:22.066 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:57:22.066 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:57:22.066 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:22.066 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:57:22.066 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:22.066 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:57:22.066 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:57:22.067 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:57:22.067 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:57:22.067 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:22.067 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:57:22.067 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:57:22.067 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:57:22.067 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:22.067 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:22.067 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:57:22.067 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:57:22.067 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:57:22.067 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:57:22.067 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:57:22.067 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:22.068 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:57:22.068 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:57:22.068 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:57:22.068 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:57:22.068 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:57:22.068 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:22.068 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:22.068 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:57:22.068 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:57:22.068 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:57:22.068 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:57:22.068 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:22.068 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:57:22.069 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:57:22.069 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:57:22.069 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:22.069 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:22.069 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:57:22.069 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:57:22.069 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:57:22.069 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:57:22.069 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:57:22.069 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:22.069 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:57:22.069 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:57:22.069 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:57:22.069 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:57:22.070 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:57:22.070 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:22.070 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:22.070 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:57:22.070 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:57:22.070 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:57:22.070 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:57:22.070 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:22.070 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:57:22.070 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:57:22.070 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:57:22.070 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:22.070 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:22.070 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:57:22.071 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:57:22.071 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:57:22.071 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:57:22.071 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:57:22.071 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:22.071 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:57:22.071 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:57:22.071 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:57:22.071 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:57:22.071 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:57:22.071 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:57:22.071 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:57:22.071 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:22.071 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:57:22.072 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:57:22.072 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:22.072 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:57:22.072 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:57:22.072 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:57:22.072 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:57:22.072 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:57:22.072 INFO:teuthology.orchestra.run.smithi012.stdout: "osd": [ 2024-09-11T01:57:22.072 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:22.072 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:57:22.072 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:57:22.072 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 2 2024-09-11T01:57:22.072 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:57:22.073 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:57:22.073 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:57:22.073 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:22.073 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:57:22.073 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:57:22.073 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 4 2024-09-11T01:57:22.073 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:57:22.073 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:57:22.073 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr": [ 2024-09-11T01:57:22.073 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:22.073 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:57:22.073 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:57:22.073 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:57:22.073 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:57:22.074 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:57:22.074 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:22.074 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:57:22.074 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:57:22.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.050+0000 7f4b81ac6640 1 -- 172.21.15.12:0/587184052 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4b6803e260 msgr2=0x7f4b68040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:22.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.050+0000 7f4b81ac6640 1 --2- 172.21.15.12:0/587184052 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4b6803e260 0x7f4b68040720 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f4b74009bd0 tx=0x7f4b7402f000 comp rx=0 tx=0).stop 2024-09-11T01:57:22.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.050+0000 7f4b81ac6640 10 monclient: shutdown 2024-09-11T01:57:22.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.050+0000 7f4b81ac6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:22.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.050+0000 7f4b81ac6640 1 -- 172.21.15.12:0/587184052 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c074000 msgr2=0x7f4b7c13cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:22.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.050+0000 7f4b81ac6640 1 --2- 172.21.15.12:0/587184052 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b7c074000 0x7f4b7c13cf00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c00c410 tx=0x7f4b6c00c8d0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.050+0000 7f4b7bfff640 1 -- 172.21.15.12:0/587184052 reap_dead start 2024-09-11T01:57:22.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.050+0000 7f4b7bfff640 1 -- 172.21.15.12:0/587184052 reap_dead start 2024-09-11T01:57:22.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.050+0000 7f4b81ac6640 1 -- 172.21.15.12:0/587184052 shutdown_connections 2024-09-11T01:57:22.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.050+0000 7f4b81ac6640 1 -- 172.21.15.12:0/587184052 >> 172.21.15.12:0/587184052 conn(0x7f4b7c0a8600 msgr2=0x7f4b7c09e330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:22.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.050+0000 7f4b81ac6640 1 -- 172.21.15.12:0/587184052 shutdown_connections 2024-09-11T01:57:22.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.050+0000 7f4b81ac6640 1 -- 172.21.15.12:0/587184052 wait complete. 2024-09-11T01:57:22.076 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:57:22.076 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T01:57:22.077 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T01:57:22.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.176+0000 7f1a72460640 10 monclient: get_monmap_and_config 2024-09-11T01:57:22.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.176+0000 7f1a72460640 10 monclient: build_initial_monmap 2024-09-11T01:57:22.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.176+0000 7f1a72460640 10 monclient: monmap: 2024-09-11T01:57:22.179 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:22.179 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:22.179 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:57:22.177697+0000 2024-09-11T01:57:22.179 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:57:22.177697+0000 2024-09-11T01:57:22.179 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:22.179 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:22.179 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:57:22.179 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:57:22.179 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:57:22.179 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:57:22.179 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:57:22.179 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:22.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.177+0000 7f1a72460640 1 Processor -- start 2024-09-11T01:57:22.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.177+0000 7f1a72460640 1 -- start start 2024-09-11T01:57:22.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.177+0000 7f1a72460640 10 monclient: init 2024-09-11T01:57:22.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a72460640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:22.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a72460640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-11T01:57:22.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a72460640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1a6c134ad0 0x7f1a6c134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:22.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a72460640 10 monclient(hunting): picked mon.noname-a con 0x7f1a6c134ad0 addr v2:172.21.15.12:3300/0 2024-09-11T01:57:22.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a72460640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1a6c0a2200 0x7f1a6c1353f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:22.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a72460640 10 monclient(hunting): picked mon.noname-c con 0x7f1a6c0a2200 addr v2:172.21.15.71:3300/0 2024-09-11T01:57:22.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a72460640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1a6c133a50 0x7f1a6c133f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:22.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a72460640 10 monclient(hunting): picked mon.noname-b con 0x7f1a6c133a50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a72460640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a72460640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f1a6c09f5d0 con 0x7f1a6c134ad0 2024-09-11T01:57:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a72460640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a72460640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f1a6c135930 con 0x7f1a6c0a2200 2024-09-11T01:57:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a72460640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a72460640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1a6c09ed30 con 0x7f1a6c133a50 2024-09-11T01:57:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a72460640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a72460640 10 monclient(hunting): authenticate will time out at 2448.620296s 2024-09-11T01:57:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a6bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1a6c134ad0 0x7f1a6c134eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a709fd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1a6c133a50 0x7f1a6c133f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a6bfff640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1a6c134ad0 0x7f1a6c134eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45328/0 (socket says 172.21.15.12:45328) 2024-09-11T01:57:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a6bfff640 1 -- 172.21.15.12:0/1485948352 learned_addr learned my addr 172.21.15.12:0/1485948352 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a6bfff640 10 monclient(hunting): get_auth_request con 0x7f1a6c134ad0 auth_method 0 2024-09-11T01:57:22.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a709fd640 10 monclient(hunting): get_auth_request con 0x7f1a6c133a50 auth_method 0 2024-09-11T01:57:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a709fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a709fd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a709fd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a709fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a709fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:22.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a709fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a6bfff640 10 monclient(hunting): handle_auth_done global_id 294306 payload 274 2024-09-11T01:57:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a6b7fe640 1 --2- 172.21.15.12:0/1485948352 >> v2:172.21.15.71:3300/0 conn(0x7f1a6c0a2200 0x7f1a6c1353f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a6bfff640 1 -- 172.21.15.12:0/1485948352 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1a6c133a50 msgr2=0x7f1a6c133f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a6bfff640 1 --2- 172.21.15.12:0/1485948352 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1a6c133a50 0x7f1a6c133f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a6bfff640 1 -- 172.21.15.12:0/1485948352 >> v2:172.21.15.71:3300/0 conn(0x7f1a6c0a2200 msgr2=0x7f1a6c1353f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a6bfff640 1 --2- 172.21.15.12:0/1485948352 >> v2:172.21.15.71:3300/0 conn(0x7f1a6c0a2200 0x7f1a6c1353f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a6bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:57:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a6bfff640 1 monclient: found mon.noname-a 2024-09-11T01:57:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a6bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:57:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a6bfff640 1 -- 172.21.15.12:0/1485948352 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a6c09f010 con 0x7f1a6c134ad0 2024-09-11T01:57:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a6b7fe640 10 monclient: get_auth_request con 0x7f1a6c0a2200 auth_method 0 2024-09-11T01:57:22.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.178+0000 7f1a6b7fe640 1 --2- 172.21.15.12:0/1485948352 >> v2:172.21.15.71:3300/0 conn(0x7f1a6c0a2200 0x7f1a6c1353f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:57:22.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.181+0000 7f1a6bfff640 1 -- 172.21.15.12:0/1485948352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a6c134ad0 msgr2=0x7f1a6c134eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:22.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.181+0000 7f1a6bfff640 1 --2- 172.21.15.12:0/1485948352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a6c134ad0 0x7f1a6c134eb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009870 tx=0x7f1a5c02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfbd063766d5a888 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:22.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.181+0000 7f1a6affd640 1 -- 172.21.15.12:0/1485948352 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c037070 con 0x7f1a6c134ad0 2024-09-11T01:57:22.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.181+0000 7f1a6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:22.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.181+0000 7f1a6affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:57:22.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.181+0000 7f1a6affd640 10 monclient: dump: 2024-09-11T01:57:22.191 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:22.192 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:22.192 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.192 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.192 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:22.192 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:22.192 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:22.192 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:22.192 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:22.192 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:22.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.181+0000 7f1a6affd640 10 monclient: _finish_auth 0 2024-09-11T01:57:22.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.181+0000 7f1a6affd640 10 monclient: _check_auth_tickets 2024-09-11T01:57:22.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.181+0000 7f1a6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:22.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.181+0000 7f1a72460640 5 monclient: authenticate success, global_id 294306 2024-09-11T01:57:22.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.181+0000 7f1a6affd640 1 -- 172.21.15.12:0/1485948352 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a5c032640 con 0x7f1a6c134ad0 2024-09-11T01:57:22.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.181+0000 7f1a6affd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:22.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.181+0000 7f1a72460640 10 monclient: get_monmap_and_config success 2024-09-11T01:57:22.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.181+0000 7f1a72460640 4 set_mon_vals no callback set 2024-09-11T01:57:22.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.181+0000 7f1a6affd640 1 -- 172.21.15.12:0/1485948352 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c032a10 con 0x7f1a6c134ad0 2024-09-11T01:57:22.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.181+0000 7f1a6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:22.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.181+0000 7f1a6affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:22.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.181+0000 7f1a6affd640 10 monclient: dump: 2024-09-11T01:57:22.193 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:22.193 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:22.193 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.193 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.193 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:22.193 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:22.194 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:22.194 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:22.194 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:22.194 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:22.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.182+0000 7f1a72460640 10 monclient: shutdown 2024-09-11T01:57:22.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.182+0000 7f1a72460640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:22.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.182+0000 7f1a72460640 1 -- 172.21.15.12:0/1485948352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a6c134ad0 msgr2=0x7f1a6c134eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:22.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.182+0000 7f1a72460640 1 --2- 172.21.15.12:0/1485948352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a6c134ad0 0x7f1a6c134eb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009870 tx=0x7f1a5c02a100 comp rx=0 tx=0).stop 2024-09-11T01:57:22.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.182+0000 7f1a72460640 1 -- 172.21.15.12:0/1485948352 shutdown_connections 2024-09-11T01:57:22.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.182+0000 7f1a72460640 1 --2- 172.21.15.12:0/1485948352 >> v2:172.21.15.71:3300/0 conn(0x7f1a6c0a2200 0x7f1a6c1353f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.182+0000 7f1a72460640 1 --2- 172.21.15.12:0/1485948352 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1a6c133a50 0x7f1a6c133f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.182+0000 7f1a72460640 1 --2- 172.21.15.12:0/1485948352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a6c134ad0 0x7f1a6c134eb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.182+0000 7f1a72460640 1 -- 172.21.15.12:0/1485948352 >> 172.21.15.12:0/1485948352 conn(0x7f1a6c0a2f70 msgr2=0x7f1a6c0a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:22.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.182+0000 7f1a72460640 1 -- 172.21.15.12:0/1485948352 shutdown_connections 2024-09-11T01:57:22.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.182+0000 7f1a72460640 1 -- 172.21.15.12:0/1485948352 wait complete. 2024-09-11T01:57:22.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.182+0000 7f1a72460640 10 monclient: build_initial_monmap 2024-09-11T01:57:22.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.182+0000 7f1a72460640 10 monclient: monmap: 2024-09-11T01:57:22.195 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:22.195 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:57:22.195 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:57:22.195 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:57:22.195 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:22.195 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:22.195 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:57:22.195 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:57:22.195 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:57:22.195 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:22.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.183+0000 7f1a72460640 1 Processor -- start 2024-09-11T01:57:22.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.183+0000 7f1a72460640 1 -- start start 2024-09-11T01:57:22.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.183+0000 7f1a72460640 10 monclient: init 2024-09-11T01:57:22.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.183+0000 7f1a72460640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:22.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.183+0000 7f1a72460640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:57:22.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.183+0000 7f1a72460640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a6c0a2200 0x7f1a6c1cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:22.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.183+0000 7f1a72460640 10 monclient(hunting): picked mon.noname-a con 0x7f1a6c0a2200 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:57:22.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.183+0000 7f1a72460640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a6c133a50 0x7f1a6c132b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:22.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.183+0000 7f1a72460640 10 monclient(hunting): picked mon.noname-b con 0x7f1a6c133a50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:57:22.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.183+0000 7f1a72460640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1a6c134ad0 0x7f1a6c1330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:22.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.183+0000 7f1a72460640 10 monclient(hunting): picked mon.noname-c con 0x7f1a6c134ad0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:22.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.183+0000 7f1a72460640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:22.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.183+0000 7f1a72460640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6c09ffd0 con 0x7f1a6c0a2200 2024-09-11T01:57:22.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.183+0000 7f1a72460640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:22.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.183+0000 7f1a72460640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6c09f380 con 0x7f1a6c133a50 2024-09-11T01:57:22.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.183+0000 7f1a72460640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:22.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.183+0000 7f1a72460640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f1a6c0a0150 con 0x7f1a6c134ad0 2024-09-11T01:57:22.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.183+0000 7f1a72460640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:22.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.183+0000 7f1a72460640 10 monclient(hunting): authenticate will time out at 2448.625964s 2024-09-11T01:57:22.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a6bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a6c0a2200 0x7f1a6c1cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:22.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a6bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a6c0a2200 0x7f1a6c1cd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45340/0 (socket says 172.21.15.12:45340) 2024-09-11T01:57:22.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a6bfff640 1 -- 172.21.15.12:0/3794449006 learned_addr learned my addr 172.21.15.12:0/3794449006 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:22.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a6bfff640 10 monclient(hunting): get_auth_request con 0x7f1a6c0a2200 auth_method 0 2024-09-11T01:57:22.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:22.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:22.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:22.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a709fd640 1 --2- 172.21.15.12:0/3794449006 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1a6c134ad0 0x7f1a6c1330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:22.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a709fd640 10 monclient(hunting): get_auth_request con 0x7f1a6c134ad0 auth_method 0 2024-09-11T01:57:22.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a709fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:22.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a709fd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:22.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a709fd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:22.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:22.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:22.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:22.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a709fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:22.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a709fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:22.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a709fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:22.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a6bfff640 10 monclient(hunting): handle_auth_done global_id 294309 payload 824 2024-09-11T01:57:22.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a6b7fe640 1 --2- 172.21.15.12:0/3794449006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a6c133a50 0x7f1a6c132b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:22.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a6bfff640 1 -- 172.21.15.12:0/3794449006 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1a6c134ad0 msgr2=0x7f1a6c1330b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:22.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a6bfff640 1 --2- 172.21.15.12:0/3794449006 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1a6c134ad0 0x7f1a6c1330b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a6bfff640 1 -- 172.21.15.12:0/3794449006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a6c133a50 msgr2=0x7f1a6c132b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:22.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a6bfff640 1 --2- 172.21.15.12:0/3794449006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a6c133a50 0x7f1a6c132b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a6bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:57:22.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a6bfff640 1 monclient: found mon.noname-a 2024-09-11T01:57:22.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:22.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a6bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:57:22.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a6bfff640 1 -- 172.21.15.12:0/3794449006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a6c125240 con 0x7f1a6c0a2200 2024-09-11T01:57:22.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a6b7fe640 10 monclient: get_auth_request con 0x7f1a6c133a50 auth_method 0 2024-09-11T01:57:22.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.184+0000 7f1a6b7fe640 1 --2- 172.21.15.12:0/3794449006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a6c133a50 0x7f1a6c132b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T01:57:22.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.186+0000 7f1a6bfff640 1 -- 172.21.15.12:0/3794449006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a6c0a2200 msgr2=0x7f1a6c1cd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:22.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.186+0000 7f1a6bfff640 1 --2- 172.21.15.12:0/3794449006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a6c0a2200 0x7f1a6c1cd1f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c000c00 tx=0x7f1a5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:22.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.186+0000 7f1a68ff9640 1 -- 172.21.15.12:0/3794449006 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c041020 con 0x7f1a6c0a2200 2024-09-11T01:57:22.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.186+0000 7f1a68ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:22.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.186+0000 7f1a68ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:57:22.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.186+0000 7f1a68ff9640 10 monclient: dump: 2024-09-11T01:57:22.201 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:22.201 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:22.201 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.201 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.201 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:22.201 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:22.201 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:22.201 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:22.201 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:22.201 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:22.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.186+0000 7f1a68ff9640 10 monclient: _finish_auth 0 2024-09-11T01:57:22.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.186+0000 7f1a68ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:57:22.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.186+0000 7f1a68ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:22.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.186+0000 7f1a68ff9640 1 -- 172.21.15.12:0/3794449006 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a5c03d850 con 0x7f1a6c0a2200 2024-09-11T01:57:22.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.186+0000 7f1a72460640 5 monclient: authenticate success, global_id 294309 2024-09-11T01:57:22.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.186+0000 7f1a72460640 10 monclient: _renew_subs 2024-09-11T01:57:22.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.186+0000 7f1a72460640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:22.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.186+0000 7f1a72460640 1 -- 172.21.15.12:0/3794449006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a6c0a15c0 con 0x7f1a6c0a2200 2024-09-11T01:57:22.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.186+0000 7f1a68ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:22.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.186+0000 7f1a68ff9640 1 -- 172.21.15.12:0/3794449006 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c037040 con 0x7f1a6c0a2200 2024-09-11T01:57:22.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.186+0000 7f1a68ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:22.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.186+0000 7f1a68ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:22.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.186+0000 7f1a68ff9640 10 monclient: dump: 2024-09-11T01:57:22.202 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:22.203 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:22.203 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.203 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.203 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:22.203 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:22.203 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:22.203 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:22.203 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:22.203 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:22.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.186+0000 7f1a72460640 10 monclient: _renew_subs 2024-09-11T01:57:22.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.186+0000 7f1a72460640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:22.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.186+0000 7f1a72460640 1 -- 172.21.15.12:0/3794449006 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a6c1335f0 con 0x7f1a6c0a2200 2024-09-11T01:57:22.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.186+0000 7f1a6a7fc640 4 set_mon_vals no callback set 2024-09-11T01:57:22.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.187+0000 7f1a72460640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:22.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.187+0000 7f1a72460640 1 --2- 172.21.15.12:0/3794449006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a6c127420 0x7f1a6c123780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:22.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.187+0000 7f1a72460640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:22.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.187+0000 7f1a72460640 1 -- 172.21.15.12:0/3794449006 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6c134560 con 0x7f1a6c127420 2024-09-11T01:57:22.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.187+0000 7f1a68ff9640 1 -- 172.21.15.12:0/3794449006 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f1a5c03a030 con 0x7f1a6c0a2200 2024-09-11T01:57:22.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.187+0000 7f1a6b7fe640 1 --2- 172.21.15.12:0/3794449006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a6c127420 0x7f1a6c123780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:22.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.187+0000 7f1a6b7fe640 10 monclient: get_auth_request con 0x7f1a6c127420 auth_method 0 2024-09-11T01:57:22.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.187+0000 7f1a6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:22.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.187+0000 7f1a6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:22.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.187+0000 7f1a6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:22.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.187+0000 7f1a68ff9640 1 --2- 172.21.15.12:0/3794449006 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1a5803e240 0x7f1a58040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:22.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.188+0000 7f1a6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:22.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.188+0000 7f1a6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:22.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.188+0000 7f1a6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:22.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.188+0000 7f1a68ff9640 1 -- 172.21.15.12:0/3794449006 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(923..923 src has 889..923) ==== 4422+0+0 (secure 0 0 0) 0x7f1a5c073070 con 0x7f1a6c0a2200 2024-09-11T01:57:22.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.188+0000 7f1a709fd640 1 --2- 172.21.15.12:0/3794449006 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1a5803e240 0x7f1a58040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:22.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.188+0000 7f1a709fd640 10 monclient: get_auth_request con 0x7f1a5803e240 auth_method 0 2024-09-11T01:57:22.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.188+0000 7f1a6b7fe640 10 monclient(hunting): handle_auth_done global_id 314104 payload 824 2024-09-11T01:57:22.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.188+0000 7f1a6b7fe640 1 -- 172.21.15.12:0/3794449006 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1a6c125da0 con 0x7f1a6c127420 2024-09-11T01:57:22.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.190+0000 7f1a6b7fe640 1 -- 172.21.15.12:0/3794449006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a6c127420 msgr2=0x7f1a6c123780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:22.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.190+0000 7f1a6b7fe640 1 --2- 172.21.15.12:0/3794449006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a6c127420 0x7f1a6c123780 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c0a2630 tx=0x7f1a6000dbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:22.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.190+0000 7f1a709fd640 1 --2- 172.21.15.12:0/3794449006 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1a5803e240 0x7f1a58040700 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f1a640113e0 tx=0x7f1a6400b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:22.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.190+0000 7f1a68ff9640 1 -- 172.21.15.12:0/3794449006 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a60016070 con 0x7f1a6c127420 2024-09-11T01:57:22.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.190+0000 7f1a68ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:22.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.190+0000 7f1a68ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:57:22.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.190+0000 7f1a68ff9640 10 monclient: dump: 2024-09-11T01:57:22.208 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:22.208 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:22.208 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.208 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.208 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:22.208 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:22.208 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:22.208 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:22.209 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:22.209 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:22.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.191+0000 7f1a68ff9640 1 -- 172.21.15.12:0/3794449006 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1a60008d20 con 0x7f1a6c127420 2024-09-11T01:57:22.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.191+0000 7f1a68ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:22.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.191+0000 7f1a68ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:57:22.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.191+0000 7f1a68ff9640 1 -- 172.21.15.12:0/3794449006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a6c127420 msgr2=0x7f1a6c123780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:22.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.191+0000 7f1a68ff9640 1 --2- 172.21.15.12:0/3794449006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a6c127420 0x7f1a6c123780 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c0a2630 tx=0x7f1a6000dbf0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.191+0000 7f1a68ff9640 1 -- 172.21.15.12:0/3794449006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a6c127420 msgr2=0x7f1a6c123780 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:57:22.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.191+0000 7f1a68ff9640 1 --2- 172.21.15.12:0/3794449006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a6c127420 0x7f1a6c123780 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c0a2630 tx=0x7f1a6000dbf0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.197+0000 7f1a72460640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T01:57:22.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.197+0000 7f1a72460640 1 --2- 172.21.15.12:0/3794449006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a6c123cc0 0x7f1a6c1240d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:22.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.197+0000 7f1a72460640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:22.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.197+0000 7f1a72460640 1 -- 172.21.15.12:0/3794449006 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6c124610 con 0x7f1a6c123cc0 2024-09-11T01:57:22.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.197+0000 7f1a6b7fe640 1 --2- 172.21.15.12:0/3794449006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a6c123cc0 0x7f1a6c1240d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:22.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.197+0000 7f1a6b7fe640 10 monclient: get_auth_request con 0x7f1a6c123cc0 auth_method 0 2024-09-11T01:57:22.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.197+0000 7f1a6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:22.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.197+0000 7f1a6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:22.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.197+0000 7f1a6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:22.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.197+0000 7f1a6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:22.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.197+0000 7f1a6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:22.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.197+0000 7f1a6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:22.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.197+0000 7f1a6b7fe640 10 monclient(hunting): handle_auth_done global_id 314107 payload 824 2024-09-11T01:57:22.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.197+0000 7f1a6b7fe640 1 -- 172.21.15.12:0/3794449006 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1a6c06d480 con 0x7f1a6c123cc0 2024-09-11T01:57:22.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.200+0000 7f1a6b7fe640 1 -- 172.21.15.12:0/3794449006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a6c123cc0 msgr2=0x7f1a6c1240d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:22.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.200+0000 7f1a6b7fe640 1 --2- 172.21.15.12:0/3794449006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a6c123cc0 0x7f1a6c1240d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1a60018000 tx=0x7f1a60004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:22.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.200+0000 7f1a68ff9640 1 -- 172.21.15.12:0/3794449006 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6001ba70 con 0x7f1a6c123cc0 2024-09-11T01:57:22.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.200+0000 7f1a68ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:22.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.200+0000 7f1a68ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:57:22.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.200+0000 7f1a68ff9640 10 monclient: dump: 2024-09-11T01:57:22.211 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:22.211 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:22.211 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.212 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.212 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:22.212 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:22.212 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:22.212 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:22.212 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:22.212 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:22.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.200+0000 7f1a68ff9640 1 -- 172.21.15.12:0/3794449006 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7f1a600113c0 con 0x7f1a6c123cc0 2024-09-11T01:57:22.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.200+0000 7f1a68ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T01:57:22.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.200+0000 7f1a68ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:57:22.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.200+0000 7f1a68ff9640 1 -- 172.21.15.12:0/3794449006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a6c123cc0 msgr2=0x7f1a6c1240d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:22.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.200+0000 7f1a68ff9640 1 --2- 172.21.15.12:0/3794449006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a6c123cc0 0x7f1a6c1240d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1a60018000 tx=0x7f1a60004730 comp rx=0 tx=0).stop 2024-09-11T01:57:22.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.200+0000 7f1a68ff9640 1 -- 172.21.15.12:0/3794449006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a6c123cc0 msgr2=0x7f1a6c1240d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T01:57:22.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.200+0000 7f1a68ff9640 1 --2- 172.21.15.12:0/3794449006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1a6c123cc0 0x7f1a6c1240d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.213 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:57:22.213 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:57:22.213 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:57:22.213 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "peon", 2024-09-11T01:57:22.213 INFO:teuthology.orchestra.run.smithi012.stdout: "election_epoch": 260, 2024-09-11T01:57:22.213 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 20643, 2024-09-11T01:57:22.213 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum": [ 2024-09-11T01:57:22.213 INFO:teuthology.orchestra.run.smithi012.stdout: 0, 2024-09-11T01:57:22.214 INFO:teuthology.orchestra.run.smithi012.stdout: 1, 2024-09-11T01:57:22.214 INFO:teuthology.orchestra.run.smithi012.stdout: 2 2024-09-11T01:57:22.214 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:57:22.214 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_age": 0, 2024-09-11T01:57:22.214 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:57:22.214 INFO:teuthology.orchestra.run.smithi012.stdout: "required_con": "2450521708007866372", 2024-09-11T01:57:22.214 INFO:teuthology.orchestra.run.smithi012.stdout: "required_mon": [ 2024-09-11T01:57:22.214 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:57:22.214 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:57:22.214 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:57:22.215 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:57:22.215 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:57:22.215 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:57:22.215 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:57:22.215 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:57:22.215 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:57:22.215 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:57:22.215 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:57:22.215 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:57:22.215 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_con": "4540719139924082687", 2024-09-11T01:57:22.215 INFO:teuthology.orchestra.run.smithi012.stdout: "quorum_mon": [ 2024-09-11T01:57:22.215 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:57:22.215 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:57:22.215 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:57:22.216 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:57:22.216 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:57:22.216 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:57:22.216 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:57:22.216 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:57:22.216 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:57:22.216 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:57:22.216 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:57:22.216 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:57:22.216 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:22.216 INFO:teuthology.orchestra.run.smithi012.stdout: "outside_quorum": [], 2024-09-11T01:57:22.216 INFO:teuthology.orchestra.run.smithi012.stdout: "extra_probe_peers": [], 2024-09-11T01:57:22.216 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_provider": [], 2024-09-11T01:57:22.217 INFO:teuthology.orchestra.run.smithi012.stdout: "monmap": { 2024-09-11T01:57:22.217 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-09-11T01:57:22.217 INFO:teuthology.orchestra.run.smithi012.stdout: "fsid": "ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab", 2024-09-11T01:57:22.217 INFO:teuthology.orchestra.run.smithi012.stdout: "modified": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:57:22.217 INFO:teuthology.orchestra.run.smithi012.stdout: "created": "2024-09-11T01:28:30.421598Z", 2024-09-11T01:57:22.217 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release": 19, 2024-09-11T01:57:22.217 INFO:teuthology.orchestra.run.smithi012.stdout: "min_mon_release_name": "squid", 2024-09-11T01:57:22.217 INFO:teuthology.orchestra.run.smithi012.stdout: "election_strategy": 1, 2024-09-11T01:57:22.217 INFO:teuthology.orchestra.run.smithi012.stdout: "disallowed_leaders: ": "", 2024-09-11T01:57:22.217 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false, 2024-09-11T01:57:22.217 INFO:teuthology.orchestra.run.smithi012.stdout: "tiebreaker_mon": "", 2024-09-11T01:57:22.217 INFO:teuthology.orchestra.run.smithi012.stdout: "removed_ranks: ": "", 2024-09-11T01:57:22.217 INFO:teuthology.orchestra.run.smithi012.stdout: "features": { 2024-09-11T01:57:22.218 INFO:teuthology.orchestra.run.smithi012.stdout: "persistent": [ 2024-09-11T01:57:22.218 INFO:teuthology.orchestra.run.smithi012.stdout: "kraken", 2024-09-11T01:57:22.218 INFO:teuthology.orchestra.run.smithi012.stdout: "luminous", 2024-09-11T01:57:22.218 INFO:teuthology.orchestra.run.smithi012.stdout: "mimic", 2024-09-11T01:57:22.218 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap-prune", 2024-09-11T01:57:22.218 INFO:teuthology.orchestra.run.smithi012.stdout: "nautilus", 2024-09-11T01:57:22.218 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus", 2024-09-11T01:57:22.218 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific", 2024-09-11T01:57:22.218 INFO:teuthology.orchestra.run.smithi012.stdout: "elector-pinging", 2024-09-11T01:57:22.218 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy", 2024-09-11T01:57:22.218 INFO:teuthology.orchestra.run.smithi012.stdout: "reef", 2024-09-11T01:57:22.218 INFO:teuthology.orchestra.run.smithi012.stdout: "squid" 2024-09-11T01:57:22.218 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:57:22.218 INFO:teuthology.orchestra.run.smithi012.stdout: "optional": [] 2024-09-11T01:57:22.219 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:22.219 INFO:teuthology.orchestra.run.smithi012.stdout: "mons": [ 2024-09-11T01:57:22.219 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:22.219 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 0, 2024-09-11T01:57:22.219 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "a", 2024-09-11T01:57:22.219 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:57:22.219 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:57:22.219 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:22.219 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:57:22.219 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3300", 2024-09-11T01:57:22.219 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:57:22.219 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:22.219 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:22.220 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:57:22.220 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789", 2024-09-11T01:57:22.220 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:57:22.220 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:57:22.220 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:57:22.220 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:22.220 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6789/0", 2024-09-11T01:57:22.220 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6789/0", 2024-09-11T01:57:22.220 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:57:22.220 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:57:22.220 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:57:22.220 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:22.220 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:22.220 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 1, 2024-09-11T01:57:22.221 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "b", 2024-09-11T01:57:22.221 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:57:22.221 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:57:22.221 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:22.221 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:57:22.221 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:3300", 2024-09-11T01:57:22.221 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:57:22.221 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:22.221 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:22.221 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:57:22.221 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789", 2024-09-11T01:57:22.221 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:57:22.221 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:57:22.222 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:57:22.222 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:22.222 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.71:6789/0", 2024-09-11T01:57:22.222 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.71:6789/0", 2024-09-11T01:57:22.222 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:57:22.222 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:57:22.222 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:57:22.222 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:22.222 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:22.222 INFO:teuthology.orchestra.run.smithi012.stdout: "rank": 2, 2024-09-11T01:57:22.222 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "c", 2024-09-11T01:57:22.222 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addrs": { 2024-09-11T01:57:22.222 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-09-11T01:57:22.222 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:22.223 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-09-11T01:57:22.223 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:3301", 2024-09-11T01:57:22.223 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:57:22.223 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:22.223 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:22.223 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-09-11T01:57:22.223 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790", 2024-09-11T01:57:22.223 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 0 2024-09-11T01:57:22.223 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:57:22.223 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:57:22.223 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:22.223 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6790/0", 2024-09-11T01:57:22.223 INFO:teuthology.orchestra.run.smithi012.stdout: "public_addr": "172.21.15.12:6790/0", 2024-09-11T01:57:22.223 INFO:teuthology.orchestra.run.smithi012.stdout: "priority": 0, 2024-09-11T01:57:22.224 INFO:teuthology.orchestra.run.smithi012.stdout: "weight": 0, 2024-09-11T01:57:22.224 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_location": "{}" 2024-09-11T01:57:22.224 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:57:22.224 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:57:22.224 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:22.224 INFO:teuthology.orchestra.run.smithi012.stdout: "feature_map": { 2024-09-11T01:57:22.224 INFO:teuthology.orchestra.run.smithi012.stdout: "mon": [ 2024-09-11T01:57:22.224 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:22.224 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:57:22.224 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:57:22.224 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:57:22.224 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:57:22.224 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-09-11T01:57:22.225 INFO:teuthology.orchestra.run.smithi012.stdout: "client": [ 2024-09-11T01:57:22.225 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-09-11T01:57:22.225 INFO:teuthology.orchestra.run.smithi012.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T01:57:22.225 INFO:teuthology.orchestra.run.smithi012.stdout: "release": "squid", 2024-09-11T01:57:22.225 INFO:teuthology.orchestra.run.smithi012.stdout: "num": 1 2024-09-11T01:57:22.225 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-09-11T01:57:22.225 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-09-11T01:57:22.225 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-09-11T01:57:22.225 INFO:teuthology.orchestra.run.smithi012.stdout: "stretch_mode": false 2024-09-11T01:57:22.225 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:57:22.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.201+0000 7f1a72460640 1 -- 172.21.15.12:0/3794449006 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1a5803e240 msgr2=0x7f1a58040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:22.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.201+0000 7f1a72460640 1 --2- 172.21.15.12:0/3794449006 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f1a5803e240 0x7f1a58040700 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f1a640113e0 tx=0x7f1a6400b040 comp rx=0 tx=0).stop 2024-09-11T01:57:22.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.201+0000 7f1a72460640 10 monclient: shutdown 2024-09-11T01:57:22.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.201+0000 7f1a72460640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:22.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.201+0000 7f1a72460640 1 -- 172.21.15.12:0/3794449006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a6c0a2200 msgr2=0x7f1a6c1cd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:22.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.201+0000 7f1a72460640 1 --2- 172.21.15.12:0/3794449006 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a6c0a2200 0x7f1a6c1cd1f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c000c00 tx=0x7f1a5c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.201+0000 7f1a709fd640 1 -- 172.21.15.12:0/3794449006 reap_dead start 2024-09-11T01:57:22.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.201+0000 7f1a709fd640 1 -- 172.21.15.12:0/3794449006 reap_dead start 2024-09-11T01:57:22.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.201+0000 7f1a72460640 1 -- 172.21.15.12:0/3794449006 shutdown_connections 2024-09-11T01:57:22.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.201+0000 7f1a72460640 1 -- 172.21.15.12:0/3794449006 >> 172.21.15.12:0/3794449006 conn(0x7f1a6c0a2f70 msgr2=0x7f1a6c09d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:22.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.201+0000 7f1a72460640 1 -- 172.21.15.12:0/3794449006 shutdown_connections 2024-09-11T01:57:22.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.201+0000 7f1a72460640 1 -- 172.21.15.12:0/3794449006 wait complete. 2024-09-11T01:57:22.228 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T01:57:22.228 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T01:57:22.228 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T01:57:22.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.327+0000 7fe3056bd640 10 monclient: get_monmap_and_config 2024-09-11T01:57:22.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.327+0000 7fe3056bd640 10 monclient: build_initial_monmap 2024-09-11T01:57:22.329 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.327+0000 7fe3056bd640 10 monclient: monmap: 2024-09-11T01:57:22.329 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:22.329 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:22.329 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:57:22.328700+0000 2024-09-11T01:57:22.330 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:57:22.328700+0000 2024-09-11T01:57:22.330 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:22.330 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:22.330 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:57:22.330 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:57:22.330 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:57:22.330 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:57:22.330 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:57:22.330 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:22.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.328+0000 7fe3056bd640 1 Processor -- start 2024-09-11T01:57:22.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.328+0000 7fe3056bd640 1 -- start start 2024-09-11T01:57:22.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.328+0000 7fe3056bd640 10 monclient: init 2024-09-11T01:57:22.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.328+0000 7fe3056bd640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:22.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe3056bd640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-11T01:57:22.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe3056bd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe3000a2d40 0x7fe30006cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:22.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe3056bd640 10 monclient(hunting): picked mon.noname-a con 0x7fe3000a2d40 addr v2:172.21.15.12:3300/0 2024-09-11T01:57:22.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe3056bd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe30006d090 addr v1:172.21.15.71:6789/0 2024-09-11T01:57:22.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe3056bd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe30006d4a0 0x7fe30013fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:22.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe3056bd640 10 monclient(hunting): picked mon.noname-c con 0x7fe30006d4a0 addr v2:172.21.15.71:3300/0 2024-09-11T01:57:22.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe3056bd640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe30009aed0 con 0x7fe30006d090 2024-09-11T01:57:22.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe3056bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:22.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe3056bd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fe30006dd10 con 0x7fe3000a2d40 2024-09-11T01:57:22.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe3056bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:22.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe3056bd640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fe30009bcf0 con 0x7fe30006d4a0 2024-09-11T01:57:22.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe3056bd640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:22.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe3056bd640 10 monclient(hunting): authenticate will time out at 2448.771098s 2024-09-11T01:57:22.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe2feffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe3000a2d40 0x7fe30006cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:22.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe2feffd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe3000a2d40 0x7fe30006cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45346/0 (socket says 172.21.15.12:45346) 2024-09-11T01:57:22.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe2feffd640 1 -- 172.21.15.12:0/2315537158 learned_addr learned my addr 172.21.15.12:0/2315537158 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:22.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe2feffd640 10 monclient(hunting): get_auth_request con 0x7fe3000a2d40 auth_method 0 2024-09-11T01:57:22.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe2feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:22.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe2feffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:22.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe2feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:22.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe2feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:22.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe2feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:22.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe2feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:22.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe2ff7fe640 1 --2- 172.21.15.12:0/2315537158 >> v2:172.21.15.71:3300/0 conn(0x7fe30006d4a0 0x7fe30013fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:22.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe2ff7fe640 10 monclient(hunting): get_auth_request con 0x7fe30006d4a0 auth_method 0 2024-09-11T01:57:22.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe2ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:22.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe2ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:22.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe2ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:22.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe2feffd640 10 monclient(hunting): handle_auth_done global_id 294312 payload 274 2024-09-11T01:57:22.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe2feffd640 1 -- 172.21.15.12:0/2315537158 >> v2:172.21.15.71:3300/0 conn(0x7fe30006d4a0 msgr2=0x7fe30013fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:22.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe2feffd640 1 --2- 172.21.15.12:0/2315537158 >> v2:172.21.15.71:3300/0 conn(0x7fe30006d4a0 0x7fe30013fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe2feffd640 1 -- 172.21.15.12:0/2315537158 >> v1:172.21.15.71:6789/0 conn(0x7fe30006d090 legacy=0x7fe30013c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:22.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe2feffd640 10 monclient: _finish_hunting 0 2024-09-11T01:57:22.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe2feffd640 1 monclient: found mon.noname-a 2024-09-11T01:57:22.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe2feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:22.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe2feffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:57:22.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.329+0000 7fe2feffd640 1 -- 172.21.15.12:0/2315537158 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3000a3920 con 0x7fe3000a2d40 2024-09-11T01:57:22.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe2feffd640 1 -- 172.21.15.12:0/2315537158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3000a2d40 msgr2=0x7fe30006cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:22.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe2feffd640 1 --2- 172.21.15.12:0/2315537158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3000a2d40 0x7fe30006cb50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe30009be70 tx=0x7fe2f002a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9dd27e7a55cb107 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:22.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe2fdffb640 1 -- 172.21.15.12:0/2315537158 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f0037070 con 0x7fe3000a2d40 2024-09-11T01:57:22.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe2fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:22.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe2fdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:57:22.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe2fdffb640 10 monclient: dump: 2024-09-11T01:57:22.341 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:22.341 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:22.341 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.341 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.341 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:22.341 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:22.341 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:22.341 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:22.341 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:22.341 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:22.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe2fdffb640 10 monclient: _finish_auth 0 2024-09-11T01:57:22.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe2fdffb640 10 monclient: _check_auth_tickets 2024-09-11T01:57:22.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe2fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:22.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe3056bd640 5 monclient: authenticate success, global_id 294312 2024-09-11T01:57:22.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe2fdffb640 1 -- 172.21.15.12:0/2315537158 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2f0032440 con 0x7fe3000a2d40 2024-09-11T01:57:22.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe3056bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:57:22.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe2fdffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:22.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe3056bd640 10 monclient: get_monmap_and_config success 2024-09-11T01:57:22.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe2fdffb640 1 -- 172.21.15.12:0/2315537158 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f0032810 con 0x7fe3000a2d40 2024-09-11T01:57:22.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe2fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:22.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe3056bd640 4 set_mon_vals no callback set 2024-09-11T01:57:22.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe2fdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:22.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe2fdffb640 10 monclient: dump: 2024-09-11T01:57:22.342 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:22.342 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:22.343 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.343 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.343 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:22.343 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:22.343 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:22.343 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:22.343 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:22.343 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:22.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe3056bd640 10 monclient: shutdown 2024-09-11T01:57:22.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe3056bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:22.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe3056bd640 1 -- 172.21.15.12:0/2315537158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3000a2d40 msgr2=0x7fe30006cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:22.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe3056bd640 1 --2- 172.21.15.12:0/2315537158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3000a2d40 0x7fe30006cb50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe30009be70 tx=0x7fe2f002a0a0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe3056bd640 1 -- 172.21.15.12:0/2315537158 shutdown_connections 2024-09-11T01:57:22.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe3056bd640 1 --2- 172.21.15.12:0/2315537158 >> v2:172.21.15.71:3300/0 conn(0x7fe30006d4a0 0x7fe30013fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe3056bd640 1 --2- 172.21.15.12:0/2315537158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3000a2d40 0x7fe30006cb50 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.332+0000 7fe3056bd640 1 -- 172.21.15.12:0/2315537158 >> 172.21.15.12:0/2315537158 conn(0x7fe3000a3e30 msgr2=0x7fe3000a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:22.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.333+0000 7fe3056bd640 1 -- 172.21.15.12:0/2315537158 shutdown_connections 2024-09-11T01:57:22.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.333+0000 7fe3056bd640 1 -- 172.21.15.12:0/2315537158 wait complete. 2024-09-11T01:57:22.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.333+0000 7fe3056bd640 10 monclient: build_initial_monmap 2024-09-11T01:57:22.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.333+0000 7fe3056bd640 10 monclient: monmap: 2024-09-11T01:57:22.344 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:22.344 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:57:22.344 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:57:22.344 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:57:22.344 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:22.344 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:22.344 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:57:22.344 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:57:22.345 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:57:22.345 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:22.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe3056bd640 1 Processor -- start 2024-09-11T01:57:22.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe3056bd640 1 -- start start 2024-09-11T01:57:22.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe3056bd640 10 monclient: init 2024-09-11T01:57:22.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe3056bd640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:22.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe3056bd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:57:22.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe3056bd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe30006d090 0x7fe3001cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:22.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe3056bd640 10 monclient(hunting): picked mon.noname-c con 0x7fe30006d090 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:22.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe3056bd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe30006d4a0 0x7fe3001cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:22.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe3056bd640 10 monclient(hunting): picked mon.noname-b con 0x7fe30006d4a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:57:22.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe3056bd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3000a2d40 0x7fe300093dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:22.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe3056bd640 10 monclient(hunting): picked mon.noname-a con 0x7fe3000a2d40 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:57:22.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe3056bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:22.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe3056bd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe30009ba10 con 0x7fe3000a2d40 2024-09-11T01:57:22.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe3056bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:22.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe3056bd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe30006d8a0 con 0x7fe30006d4a0 2024-09-11T01:57:22.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe3056bd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:22.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe3056bd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe30009b3a0 con 0x7fe30006d090 2024-09-11T01:57:22.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe3056bd640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:22.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe3056bd640 10 monclient(hunting): authenticate will time out at 2448.776581s 2024-09-11T01:57:22.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2ff7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3000a2d40 0x7fe300093dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:22.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2feffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe30006d090 0x7fe3001cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:22.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2ff7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3000a2d40 0x7fe300093dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45360/0 (socket says 172.21.15.12:45360) 2024-09-11T01:57:22.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2ff7fe640 1 -- 172.21.15.12:0/3895429931 learned_addr learned my addr 172.21.15.12:0/3895429931 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:22.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2ff7fe640 10 monclient(hunting): get_auth_request con 0x7fe3000a2d40 auth_method 0 2024-09-11T01:57:22.346 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:22.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:22.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:22.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2feffd640 1 --2- 172.21.15.12:0/3895429931 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe30006d090 0x7fe3001cd530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52834/0 (socket says 172.21.15.12:52834) 2024-09-11T01:57:22.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2feffd640 10 monclient(hunting): get_auth_request con 0x7fe30006d090 auth_method 0 2024-09-11T01:57:22.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:22.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2feffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:22.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:22.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:22.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:22.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:22.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:22.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:22.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:22.347 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2fe7fc640 1 --2- 172.21.15.12:0/3895429931 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe30006d4a0 0x7fe3001cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:22.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2fe7fc640 10 monclient(hunting): get_auth_request con 0x7fe30006d4a0 auth_method 0 2024-09-11T01:57:22.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2fe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:22.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:22.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:22.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2ff7fe640 10 monclient(hunting): handle_auth_done global_id 294315 payload 824 2024-09-11T01:57:22.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2ff7fe640 1 -- 172.21.15.12:0/3895429931 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe30006d090 msgr2=0x7fe3001cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:22.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2ff7fe640 1 --2- 172.21.15.12:0/3895429931 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe30006d090 0x7fe3001cd530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2ff7fe640 1 -- 172.21.15.12:0/3895429931 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe30006d4a0 msgr2=0x7fe3001cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:22.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2ff7fe640 1 --2- 172.21.15.12:0/3895429931 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe30006d4a0 0x7fe3001cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2ff7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:57:22.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2ff7fe640 1 monclient: found mon.noname-a 2024-09-11T01:57:22.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:22.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2ff7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:57:22.348 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.334+0000 7fe2ff7fe640 1 -- 172.21.15.12:0/3895429931 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe300097b90 con 0x7fe3000a2d40 2024-09-11T01:57:22.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.337+0000 7fe2ff7fe640 1 -- 172.21.15.12:0/3895429931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3000a2d40 msgr2=0x7fe300093dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:22.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.337+0000 7fe2ff7fe640 1 --2- 172.21.15.12:0/3895429931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3000a2d40 0x7fe300093dc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8007e70 tx=0x7fe2f800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:22.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.337+0000 7fe2f77fe640 1 -- 172.21.15.12:0/3895429931 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f8013790 con 0x7fe3000a2d40 2024-09-11T01:57:22.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.337+0000 7fe2f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:22.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.337+0000 7fe2f77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:57:22.349 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.337+0000 7fe2f77fe640 10 monclient: dump: 2024-09-11T01:57:22.349 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:22.349 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:22.349 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.349 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.349 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:22.349 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:22.349 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:22.349 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:22.350 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:22.350 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:22.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.337+0000 7fe2f77fe640 10 monclient: _finish_auth 0 2024-09-11T01:57:22.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.337+0000 7fe2f77fe640 10 monclient: _check_auth_tickets 2024-09-11T01:57:22.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.337+0000 7fe2f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:22.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.337+0000 7fe2f77fe640 1 -- 172.21.15.12:0/3895429931 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2f8019070 con 0x7fe3000a2d40 2024-09-11T01:57:22.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.337+0000 7fe2f77fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:22.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.337+0000 7fe2f77fe640 1 -- 172.21.15.12:0/3895429931 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f8013b90 con 0x7fe3000a2d40 2024-09-11T01:57:22.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.337+0000 7fe3056bd640 5 monclient: authenticate success, global_id 294315 2024-09-11T01:57:22.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.337+0000 7fe3056bd640 10 monclient: _renew_subs 2024-09-11T01:57:22.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.337+0000 7fe3056bd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:22.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.337+0000 7fe3056bd640 1 -- 172.21.15.12:0/3895429931 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe30009a080 con 0x7fe3000a2d40 2024-09-11T01:57:22.350 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.337+0000 7fe2f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:22.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.337+0000 7fe2f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:22.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.337+0000 7fe2f77fe640 10 monclient: dump: 2024-09-11T01:57:22.351 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:22.351 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:22.351 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.351 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:22.351 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:22.351 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:22.351 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:22.351 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:22.351 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:22.351 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:22.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.337+0000 7fe2fd7fa640 4 set_mon_vals no callback set 2024-09-11T01:57:22.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.337+0000 7fe3056bd640 10 monclient: _renew_subs 2024-09-11T01:57:22.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.337+0000 7fe3056bd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:22.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.337+0000 7fe3056bd640 1 -- 172.21.15.12:0/3895429931 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3000972f0 con 0x7fe3000a2d40 2024-09-11T01:57:22.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.338+0000 7fe2f77fe640 1 -- 172.21.15.12:0/3895429931 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fe2f801e460 con 0x7fe3000a2d40 2024-09-11T01:57:22.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.338+0000 7fe3056bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:22.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.338+0000 7fe3056bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:22.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.338+0000 7fe3056bd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:22.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.338+0000 7fe3056bd640 1 -- 172.21.15.12:0/3895429931 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe30006cb50 con 0x7fe3000a2d40 2024-09-11T01:57:22.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.338+0000 7fe2f77fe640 1 --2- 172.21.15.12:0/3895429931 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe2e003e210 0x7fe2e00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:22.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.338+0000 7fe2f77fe640 1 -- 172.21.15.12:0/3895429931 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(923..923 src has 889..923) ==== 4422+0+0 (secure 0 0 0) 0x7fe2f8053540 con 0x7fe3000a2d40 2024-09-11T01:57:22.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.339+0000 7fe2feffd640 1 --2- 172.21.15.12:0/3895429931 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe2e003e210 0x7fe2e00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:22.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.339+0000 7fe2feffd640 10 monclient: get_auth_request con 0x7fe2e003e210 auth_method 0 2024-09-11T01:57:22.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.341+0000 7fe2feffd640 1 --2- 172.21.15.12:0/3895429931 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe2e003e210 0x7fe2e00406d0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0034000 tx=0x7fe2f002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:22.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.343+0000 7fe2f77fe640 1 -- 172.21.15.12:0/3895429931 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe2f80520a0 con 0x7fe3000a2d40 2024-09-11T01:57:22.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.343+0000 7fe2f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:22.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.343+0000 7fe2f77fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:57:22.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.603+0000 7fe3056bd640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T01:57:22.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.603+0000 7fe3056bd640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T01:57:22.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.603+0000 7fe3056bd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:22.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.603+0000 7fe3056bd640 1 -- 172.21.15.12:0/3895429931 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fe300095830 con 0x7fe3000a2d40 2024-09-11T01:57:22.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.604+0000 7fe2f77fe640 1 -- 172.21.15.12:0/3895429931 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fe2f8013d30 con 0x7fe3000a2d40 2024-09-11T01:57:22.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.604+0000 7fe2f77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T01:57:22.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.604+0000 7fe2f77fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:57:22.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.606+0000 7fe3056bd640 1 -- 172.21.15.12:0/3895429931 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe2e003e210 msgr2=0x7fe2e00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:22.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.606+0000 7fe3056bd640 1 --2- 172.21.15.12:0/3895429931 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe2e003e210 0x7fe2e00406d0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0034000 tx=0x7fe2f002e000 comp rx=0 tx=0).stop 2024-09-11T01:57:22.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.606+0000 7fe3056bd640 10 monclient: shutdown 2024-09-11T01:57:22.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.606+0000 7fe3056bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:22.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.606+0000 7fe3056bd640 1 -- 172.21.15.12:0/3895429931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3000a2d40 msgr2=0x7fe300093dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:22.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.606+0000 7fe3056bd640 1 --2- 172.21.15.12:0/3895429931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3000a2d40 0x7fe300093dc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8007e70 tx=0x7fe2f800e4b0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.606+0000 7fe3056bd640 1 -- 172.21.15.12:0/3895429931 shutdown_connections 2024-09-11T01:57:22.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.606+0000 7fe3056bd640 1 --2- 172.21.15.12:0/3895429931 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe2e003e210 0x7fe2e00406d0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.606+0000 7fe3056bd640 1 --2- 172.21.15.12:0/3895429931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3000a2d40 0x7fe300093dc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.606+0000 7fe3056bd640 1 --2- 172.21.15.12:0/3895429931 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe30006d4a0 0x7fe3001cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.606+0000 7fe3056bd640 1 --2- 172.21.15.12:0/3895429931 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe30006d090 0x7fe3001cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:22.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.606+0000 7fe3056bd640 1 -- 172.21.15.12:0/3895429931 >> 172.21.15.12:0/3895429931 conn(0x7fe3000a3e30 msgr2=0x7fe30009f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:22.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.606+0000 7fe3056bd640 1 -- 172.21.15.12:0/3895429931 shutdown_connections 2024-09-11T01:57:22.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:22.606+0000 7fe3056bd640 1 -- 172.21.15.12:0/3895429931 wait complete. 2024-09-11T01:57:22.619 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T01:57:23.422 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.420+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.422 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.420+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=1,health=11,kv=11,logm=49} crc {auth=63854780,config=1228700967,health=2322625438,kv=4243670985,logm=2095727147}) 2024-09-11T01:57:23.422 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.420+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=11,kv=12,logm=47} crc {auth=1300653968,config=2586702516,health=1896717033,kv=3263697420,logm=2691006058}) 2024-09-11T01:57:23.422 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.420+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.422 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.420+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=1,health=11,kv=11,logm=49} crc {auth=63854780,config=1228700967,health=2322625438,kv=4243670985,logm=2095727147}) 2024-09-11T01:57:23.423 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.420+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=11,kv=12,logm=45} crc {auth=982009561,config=2586702516,health=2322625438,kv=3263697420,logm=1990324859}) 2024-09-11T01:57:23.424 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.422+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.424 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.422+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1053907068}) 2024-09-11T01:57:23.424 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.422+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2669757866}) 2024-09-11T01:57:23.424 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.422+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.424 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.422+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1053907068}) 2024-09-11T01:57:23.424 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.422+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4223488926}) 2024-09-11T01:57:23.426 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.424+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.426 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.424+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4244335564}) 2024-09-11T01:57:23.426 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.424+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4152674252}) 2024-09-11T01:57:23.426 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.424+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.426 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.424+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4244335564}) 2024-09-11T01:57:23.426 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.424+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2976586742}) 2024-09-11T01:57:23.428 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.426+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.428 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.426+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2905679161}) 2024-09-11T01:57:23.428 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.426+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3489823942}) 2024-09-11T01:57:23.428 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.426+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.428 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.426+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2905679161}) 2024-09-11T01:57:23.428 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.426+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1657173166}) 2024-09-11T01:57:23.430 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.428+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.430 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.428+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3898059836}) 2024-09-11T01:57:23.430 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.428+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=126467802}) 2024-09-11T01:57:23.430 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.428+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.430 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.428+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3898059836}) 2024-09-11T01:57:23.430 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.428+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=326988934}) 2024-09-11T01:57:23.432 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.430+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.432 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.430+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3072591342}) 2024-09-11T01:57:23.432 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.430+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4284077354}) 2024-09-11T01:57:23.432 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.430+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.432 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.430+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3072591342}) 2024-09-11T01:57:23.432 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.430+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=180150906}) 2024-09-11T01:57:23.485 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.484+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.485 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.484+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4126782817}) 2024-09-11T01:57:23.486 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.484+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=208293190}) 2024-09-11T01:57:23.486 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.484+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.486 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.484+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4126782817}) 2024-09-11T01:57:23.486 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.484+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=672902474}) 2024-09-11T01:57:23.508 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.507+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.508 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.507+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2906080191}) 2024-09-11T01:57:23.509 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.507+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2578971542}) 2024-09-11T01:57:23.509 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.507+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.509 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.507+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2906080191}) 2024-09-11T01:57:23.509 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.507+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1898227869}) 2024-09-11T01:57:23.512 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.510+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.512 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.510+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3306517516}) 2024-09-11T01:57:23.512 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.510+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2254425768}) 2024-09-11T01:57:23.512 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.510+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.513 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.510+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3306517516}) 2024-09-11T01:57:23.513 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.511+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=711420371}) 2024-09-11T01:57:23.516 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.515+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.516 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.515+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1851461155}) 2024-09-11T01:57:23.517 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.515+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2417693495}) 2024-09-11T01:57:23.517 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.515+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.517 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.515+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1851461155}) 2024-09-11T01:57:23.517 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.515+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1871020054}) 2024-09-11T01:57:23.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.579+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.579+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3957613762}) 2024-09-11T01:57:23.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.579+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1664615192}) 2024-09-11T01:57:23.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.579+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.579+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3957613762}) 2024-09-11T01:57:23.581 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.579+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2562383214}) 2024-09-11T01:57:23.582 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.580+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.582 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.580+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4086448311}) 2024-09-11T01:57:23.582 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.580+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1796010055}) 2024-09-11T01:57:23.582 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.580+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.582 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.580+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4086448311}) 2024-09-11T01:57:23.582 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.580+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3568456778}) 2024-09-11T01:57:23.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.581+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.581+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3147518746}) 2024-09-11T01:57:23.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.581+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2343393317}) 2024-09-11T01:57:23.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.581+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.581+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3147518746}) 2024-09-11T01:57:23.583 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.581+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=326243977}) 2024-09-11T01:57:23.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.581+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.581+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1031309503}) 2024-09-11T01:57:23.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.581+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1128942476}) 2024-09-11T01:57:23.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.581+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.581+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1031309503}) 2024-09-11T01:57:23.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.581+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3353264729}) 2024-09-11T01:57:23.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.582+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.582+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=469979530}) 2024-09-11T01:57:23.584 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.582+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3569879921}) 2024-09-11T01:57:23.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.582+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.582+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=469979530}) 2024-09-11T01:57:23.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.582+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2981976190}) 2024-09-11T01:57:23.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.583+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.583+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1478778789}) 2024-09-11T01:57:23.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.583+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4100454004}) 2024-09-11T01:57:23.585 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.583+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.586 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.583+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1478778789}) 2024-09-11T01:57:23.586 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.583+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1215768255}) 2024-09-11T01:57:23.619 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T01:57:23.619 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T01:57:23.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.731+0000 7f5829990640 10 monclient: get_monmap_and_config 2024-09-11T01:57:23.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.731+0000 7f5829990640 10 monclient: build_initial_monmap 2024-09-11T01:57:23.733 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.731+0000 7f5829990640 10 monclient: monmap: 2024-09-11T01:57:23.733 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:23.733 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:23.733 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:57:23.731994+0000 2024-09-11T01:57:23.733 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:57:23.731994+0000 2024-09-11T01:57:23.733 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:23.734 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:23.734 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:57:23.734 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:57:23.734 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:57:23.734 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:57:23.734 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:57:23.734 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:23.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.731+0000 7f5829990640 1 Processor -- start 2024-09-11T01:57:23.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.731+0000 7f5829990640 1 -- start start 2024-09-11T01:57:23.734 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.731+0000 7f5829990640 10 monclient: init 2024-09-11T01:57:23.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f5829990640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:23.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f5829990640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-11T01:57:23.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f5829990640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5824138210 addr v1:172.21.15.71:6789/0 2024-09-11T01:57:23.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f5829990640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f58240a6810 addr v1:172.21.15.12:6789/0 2024-09-11T01:57:23.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f5829990640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f582413e820 0x7f5824140c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:23.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f5829990640 10 monclient(hunting): picked mon.noname-a con 0x7f582413e820 addr v2:172.21.15.12:3300/0 2024-09-11T01:57:23.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f5829990640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5824138d00 con 0x7f58240a6810 2024-09-11T01:57:23.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f5829990640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f582409ba20 con 0x7f5824138210 2024-09-11T01:57:23.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f5829990640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:23.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f5829990640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f582409c4c0 con 0x7f582413e820 2024-09-11T01:57:23.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f5829990640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:23.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f5829990640 10 monclient(hunting): authenticate will time out at 2450.174514s 2024-09-11T01:57:23.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f58237fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f582413e820 0x7f5824140c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:23.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f58227fc640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f58240a6810 0x7f582413d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:39162/0 (socket says 172.21.15.12:39162) 2024-09-11T01:57:23.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f58227fc640 1 -- 172.21.15.12:0/1188827104 learned_addr learned my addr 172.21.15.12:0/1188827104 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:57:23.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f58237fe640 10 monclient(hunting): get_auth_request con 0x7f582413e820 auth_method 0 2024-09-11T01:57:23.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f58237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:23.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f58237fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:23.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f58237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:23.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f58237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:23.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f58237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:23.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f58237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:23.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f58237fe640 10 monclient(hunting): handle_auth_done global_id 294318 payload 274 2024-09-11T01:57:23.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f58237fe640 1 -- 172.21.15.12:0/1188827104 >> v1:172.21.15.71:6789/0 conn(0x7f5824138210 legacy=0x7f58241385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:23.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f58237fe640 1 -- 172.21.15.12:0/1188827104 >> v1:172.21.15.12:6789/0 conn(0x7f58240a6810 legacy=0x7f582413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:23.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f58237fe640 10 monclient: _finish_hunting 0 2024-09-11T01:57:23.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f58237fe640 1 monclient: found mon.noname-a 2024-09-11T01:57:23.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f58237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:23.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f58237fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:57:23.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.732+0000 7f58237fe640 1 -- 172.21.15.12:0/1188827104 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58240a6590 con 0x7f582413e820 2024-09-11T01:57:23.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.735+0000 7f58237fe640 1 -- 172.21.15.12:0/1188827104 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f582413e820 msgr2=0x7f5824140c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:23.742 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.735+0000 7f58237fe640 1 --2- 172.21.15.12:0/1188827104 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f582413e820 0x7f5824140c10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f581c0097a0 tx=0x7f581c029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cda0b7d72e8b8c8f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:23.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.736+0000 7f5821ffb640 1 -- 172.21.15.12:0/1188827104 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f581c037070 con 0x7f582413e820 2024-09-11T01:57:23.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.736+0000 7f5821ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:23.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.736+0000 7f5821ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:57:23.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.736+0000 7f5821ffb640 10 monclient: dump: 2024-09-11T01:57:23.743 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:23.743 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:23.743 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:23.743 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:23.743 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:23.743 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:23.743 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:23.743 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:23.743 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:23.744 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:23.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.736+0000 7f5821ffb640 10 monclient: _finish_auth 0 2024-09-11T01:57:23.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.736+0000 7f5821ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:57:23.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.736+0000 7f5821ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:23.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.736+0000 7f5821ffb640 1 -- 172.21.15.12:0/1188827104 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f581c032640 con 0x7f582413e820 2024-09-11T01:57:23.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.736+0000 7f5829990640 5 monclient: authenticate success, global_id 294318 2024-09-11T01:57:23.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.736+0000 7f5829990640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:57:23.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.736+0000 7f5821ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:23.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.736+0000 7f5829990640 10 monclient: get_monmap_and_config success 2024-09-11T01:57:23.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.736+0000 7f5821ffb640 1 -- 172.21.15.12:0/1188827104 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f581c032a10 con 0x7f582413e820 2024-09-11T01:57:23.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.736+0000 7f5821ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:23.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.736+0000 7f5829990640 4 set_mon_vals no callback set 2024-09-11T01:57:23.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.736+0000 7f5821ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:23.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.736+0000 7f5821ffb640 10 monclient: dump: 2024-09-11T01:57:23.745 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:23.745 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:23.745 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:23.745 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:23.745 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:23.745 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:23.745 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:23.745 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:23.745 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:23.745 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:23.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.736+0000 7f5829990640 10 monclient: shutdown 2024-09-11T01:57:23.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.736+0000 7f5829990640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:23.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.736+0000 7f5829990640 1 -- 172.21.15.12:0/1188827104 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f582413e820 msgr2=0x7f5824140c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:23.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.736+0000 7f5829990640 1 --2- 172.21.15.12:0/1188827104 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f582413e820 0x7f5824140c10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f581c0097a0 tx=0x7f581c029fd0 comp rx=0 tx=0).stop 2024-09-11T01:57:23.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.736+0000 7f5829990640 1 -- 172.21.15.12:0/1188827104 shutdown_connections 2024-09-11T01:57:23.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.736+0000 7f5829990640 1 --2- 172.21.15.12:0/1188827104 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f582413e820 0x7f5824140c10 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:23.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.736+0000 7f5829990640 1 -- 172.21.15.12:0/1188827104 >> 172.21.15.12:0/1188827104 conn(0x7f58240aaa80 msgr2=0x7f58240aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:23.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.736+0000 7f5829990640 1 -- 172.21.15.12:0/1188827104 shutdown_connections 2024-09-11T01:57:23.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.736+0000 7f5829990640 1 -- 172.21.15.12:0/1188827104 wait complete. 2024-09-11T01:57:23.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.737+0000 7f5829990640 10 monclient: build_initial_monmap 2024-09-11T01:57:23.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.737+0000 7f5829990640 10 monclient: monmap: 2024-09-11T01:57:23.746 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:23.746 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:57:23.746 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:57:23.746 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:57:23.747 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:23.747 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:23.747 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:57:23.747 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:57:23.747 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:57:23.747 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:23.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.737+0000 7f5829990640 1 Processor -- start 2024-09-11T01:57:23.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f5829990640 1 -- start start 2024-09-11T01:57:23.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f5829990640 10 monclient: init 2024-09-11T01:57:23.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f5829990640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:23.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f5829990640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:57:23.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f5829990640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f58240a6810 0x7f58241cd430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:23.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f5829990640 10 monclient(hunting): picked mon.noname-c con 0x7f58240a6810 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:23.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f5829990640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5824138210 0x7f58241cd970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:23.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f5829990640 10 monclient(hunting): picked mon.noname-b con 0x7f5824138210 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:57:23.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f5829990640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f582413e820 0x7f5824139a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:23.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f5829990640 10 monclient(hunting): picked mon.noname-a con 0x7f582413e820 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:57:23.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f5829990640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:23.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f5829990640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f582406f670 con 0x7f582413e820 2024-09-11T01:57:23.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f5829990640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:23.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f5829990640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f582406f4f0 con 0x7f5824138210 2024-09-11T01:57:23.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f5829990640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:23.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f5829990640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5824141150 con 0x7f58240a6810 2024-09-11T01:57:23.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f5829990640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:23.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f5829990640 10 monclient(hunting): authenticate will time out at 2450.180526s 2024-09-11T01:57:23.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f58237fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f582413e820 0x7f5824139a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:23.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f5822ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f58240a6810 0x7f58241cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:23.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f58237fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f582413e820 0x7f5824139a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45372/0 (socket says 172.21.15.12:45372) 2024-09-11T01:57:23.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f58237fe640 1 -- 172.21.15.12:0/1375355330 learned_addr learned my addr 172.21.15.12:0/1375355330 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:23.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f58237fe640 10 monclient(hunting): get_auth_request con 0x7f582413e820 auth_method 0 2024-09-11T01:57:23.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f58237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:23.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f58237fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:23.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f58237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:23.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f5822ffd640 10 monclient(hunting): get_auth_request con 0x7f58240a6810 auth_method 0 2024-09-11T01:57:23.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f5822ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:23.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f5822ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:23.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f5822ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:23.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f58237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f58237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f58237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f5822ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f5822ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f58227fc640 1 --2- 172.21.15.12:0/1375355330 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5824138210 0x7f58241cd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f5822ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f58227fc640 10 monclient(hunting): get_auth_request con 0x7f5824138210 auth_method 0 2024-09-11T01:57:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f58227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f58227fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f58227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f58237fe640 10 monclient(hunting): handle_auth_done global_id 294321 payload 824 2024-09-11T01:57:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f58237fe640 1 -- 172.21.15.12:0/1375355330 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f58240a6810 msgr2=0x7f58241cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f58237fe640 1 --2- 172.21.15.12:0/1375355330 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f58240a6810 0x7f58241cd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:23.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f58237fe640 1 -- 172.21.15.12:0/1375355330 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5824138210 msgr2=0x7f58241cd970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f58237fe640 1 --2- 172.21.15.12:0/1375355330 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5824138210 0x7f58241cd970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f58237fe640 10 monclient: _finish_hunting 0 2024-09-11T01:57:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f58237fe640 1 monclient: found mon.noname-a 2024-09-11T01:57:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f58237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f58237fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:57:23.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.738+0000 7f58237fe640 1 -- 172.21.15.12:0/1375355330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f582413d770 con 0x7f582413e820 2024-09-11T01:57:23.752 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.740+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.752 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.740+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=156549803}) 2024-09-11T01:57:23.752 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.740+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2881571144}) 2024-09-11T01:57:23.752 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.740+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.752 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.740+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=156549803}) 2024-09-11T01:57:23.752 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.740+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3655980993}) 2024-09-11T01:57:23.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.741+0000 7f58237fe640 1 -- 172.21.15.12:0/1375355330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f582413e820 msgr2=0x7f5824139a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:23.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.741+0000 7f58237fe640 1 --2- 172.21.15.12:0/1375355330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f582413e820 0x7f5824139a40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f581c0098c0 tx=0x7f581c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:23.775 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.741+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.776 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.741+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1837638443}) 2024-09-11T01:57:23.776 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.741+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2651228491}) 2024-09-11T01:57:23.776 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.741+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.776 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.741+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1837638443}) 2024-09-11T01:57:23.776 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.741+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2710237241}) 2024-09-11T01:57:23.776 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.742+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.776 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.742+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1973878415}) 2024-09-11T01:57:23.776 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.742+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3123738529}) 2024-09-11T01:57:23.776 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.742+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.776 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.742+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1973878415}) 2024-09-11T01:57:23.776 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.742+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1366057121}) 2024-09-11T01:57:23.776 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.743+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.743+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=911235777}) 2024-09-11T01:57:23.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.743+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1196806719}) 2024-09-11T01:57:23.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.743+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.743+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=911235777}) 2024-09-11T01:57:23.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.743+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=444615803}) 2024-09-11T01:57:23.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.743+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.743+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3782203873}) 2024-09-11T01:57:23.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.744+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2792123165}) 2024-09-11T01:57:23.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.744+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.744+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3782203873}) 2024-09-11T01:57:23.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.744+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1733634599}) 2024-09-11T01:57:23.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.744+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.777 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.744+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2742714013}) 2024-09-11T01:57:23.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.744+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=525914277}) 2024-09-11T01:57:23.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.744+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.744+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2742714013}) 2024-09-11T01:57:23.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.744+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3621589963}) 2024-09-11T01:57:23.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.745+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.745+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2644551310}) 2024-09-11T01:57:23.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.745+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1953233393}) 2024-09-11T01:57:23.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.745+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.745+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2644551310}) 2024-09-11T01:57:23.778 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.745+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4288473600}) 2024-09-11T01:57:23.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.746+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.746+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1303167119}) 2024-09-11T01:57:23.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.746+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3014067791}) 2024-09-11T01:57:23.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.746+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.746+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1303167119}) 2024-09-11T01:57:23.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.746+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=281627167}) 2024-09-11T01:57:23.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.747+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.747+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3768106063}) 2024-09-11T01:57:23.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.747+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1512497732}) 2024-09-11T01:57:23.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.747+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.747+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3768106063}) 2024-09-11T01:57:23.779 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.747+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=429319616}) 2024-09-11T01:57:23.780 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.747+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.780 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.747+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3643595377}) 2024-09-11T01:57:23.780 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.747+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1350843997}) 2024-09-11T01:57:23.780 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.747+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.780 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.747+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3643595377}) 2024-09-11T01:57:23.780 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.747+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3721360740}) 2024-09-11T01:57:23.780 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.748+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.780 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.748+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2217063127}) 2024-09-11T01:57:23.780 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.748+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2289539569}) 2024-09-11T01:57:23.780 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.748+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.780 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.748+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2217063127}) 2024-09-11T01:57:23.780 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.748+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1613289472}) 2024-09-11T01:57:23.780 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.749+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.749+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=826350287}) 2024-09-11T01:57:23.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.749+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=45174694}) 2024-09-11T01:57:23.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.749+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.749+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=826350287}) 2024-09-11T01:57:23.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.749+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2275340176}) 2024-09-11T01:57:23.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.750+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.750+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3275572211}) 2024-09-11T01:57:23.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.750+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=265089550}) 2024-09-11T01:57:23.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.750+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.750+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3275572211}) 2024-09-11T01:57:23.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.750+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2043404031}) 2024-09-11T01:57:23.781 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.750+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.782 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.750+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949512205}) 2024-09-11T01:57:23.782 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.750+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=947760732}) 2024-09-11T01:57:23.782 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.750+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.782 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.750+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949512205}) 2024-09-11T01:57:23.782 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.750+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2492467267}) 2024-09-11T01:57:23.782 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.751+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.782 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.751+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4219338971}) 2024-09-11T01:57:23.782 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.751+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3125010387}) 2024-09-11T01:57:23.782 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.751+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.782 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.751+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4219338971}) 2024-09-11T01:57:23.782 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.751+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3283828378}) 2024-09-11T01:57:23.782 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.752+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.782 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.752+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4271117589}) 2024-09-11T01:57:23.783 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.752+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3150477299}) 2024-09-11T01:57:23.783 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.752+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.783 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.752+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4271117589}) 2024-09-11T01:57:23.783 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.752+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1250453663}) 2024-09-11T01:57:23.783 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.753+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.783 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.753+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3761119194}) 2024-09-11T01:57:23.783 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.753+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4221386109}) 2024-09-11T01:57:23.783 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.753+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.783 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.753+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3761119194}) 2024-09-11T01:57:23.783 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.753+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1506672319}) 2024-09-11T01:57:23.783 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.754+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.783 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.754+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=59960233}) 2024-09-11T01:57:23.783 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.754+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1633626807}) 2024-09-11T01:57:23.784 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.754+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.784 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.754+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=59960233}) 2024-09-11T01:57:23.784 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.754+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=737898769}) 2024-09-11T01:57:23.784 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.754+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.784 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.754+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=721789459}) 2024-09-11T01:57:23.784 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.754+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1768185223}) 2024-09-11T01:57:23.784 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.754+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.784 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.754+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=721789459}) 2024-09-11T01:57:23.784 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.754+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2126894391}) 2024-09-11T01:57:23.784 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.755+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.784 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.755+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1254860038}) 2024-09-11T01:57:23.784 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.755+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=206909412}) 2024-09-11T01:57:23.784 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.755+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.785 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.755+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1254860038}) 2024-09-11T01:57:23.785 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.755+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3548679547}) 2024-09-11T01:57:23.785 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.756+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.785 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.756+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=953055142}) 2024-09-11T01:57:23.785 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.756+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1139913663}) 2024-09-11T01:57:23.785 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.756+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.785 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.756+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=953055142}) 2024-09-11T01:57:23.785 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.756+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2660752369}) 2024-09-11T01:57:23.785 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.757+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.785 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.757+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4080171680}) 2024-09-11T01:57:23.785 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.757+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3442055380}) 2024-09-11T01:57:23.785 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.757+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.785 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.757+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4080171680}) 2024-09-11T01:57:23.786 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.757+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=199630493}) 2024-09-11T01:57:23.786 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.758+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.786 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.758+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3568816275}) 2024-09-11T01:57:23.786 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.758+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3388825967}) 2024-09-11T01:57:23.786 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.758+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.786 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.758+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3568816275}) 2024-09-11T01:57:23.786 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.758+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=184964672}) 2024-09-11T01:57:23.786 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.758+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.786 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.758+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3950185646}) 2024-09-11T01:57:23.786 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.758+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=192647701}) 2024-09-11T01:57:23.786 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.758+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.786 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.758+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3950185646}) 2024-09-11T01:57:23.786 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.758+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=462144781}) 2024-09-11T01:57:23.787 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.759+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.787 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.759+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=703100965}) 2024-09-11T01:57:23.787 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.759+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=398774484}) 2024-09-11T01:57:23.787 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.759+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.787 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.759+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=703100965}) 2024-09-11T01:57:23.787 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.759+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1185392423}) 2024-09-11T01:57:23.787 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.760+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.787 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.760+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=811670625}) 2024-09-11T01:57:23.787 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.760+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3784274656}) 2024-09-11T01:57:23.787 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.760+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.760+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=811670625}) 2024-09-11T01:57:23.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.760+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1413623837}) 2024-09-11T01:57:23.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.761+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.761+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2847793007}) 2024-09-11T01:57:23.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.761+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2441175224}) 2024-09-11T01:57:23.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.761+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.761+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2847793007}) 2024-09-11T01:57:23.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.761+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2242170934}) 2024-09-11T01:57:23.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.761+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.761+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=510474884}) 2024-09-11T01:57:23.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.762+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2247542670}) 2024-09-11T01:57:23.788 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.762+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.789 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.762+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=510474884}) 2024-09-11T01:57:23.789 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.762+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2904854799}) 2024-09-11T01:57:23.789 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.762+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.789 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.762+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2189370897}) 2024-09-11T01:57:23.789 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.762+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3217523357}) 2024-09-11T01:57:23.789 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.762+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.789 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.762+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2189370897}) 2024-09-11T01:57:23.789 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.762+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2960426855}) 2024-09-11T01:57:23.789 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.763+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.789 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.763+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=507346002}) 2024-09-11T01:57:23.789 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.763+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4293504007}) 2024-09-11T01:57:23.789 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.763+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.789 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.763+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=507346002}) 2024-09-11T01:57:23.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.763+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2814104933}) 2024-09-11T01:57:23.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.764+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.764+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=954042142}) 2024-09-11T01:57:23.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.764+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3033405632}) 2024-09-11T01:57:23.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.764+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.764+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=954042142}) 2024-09-11T01:57:23.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.764+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=951929355}) 2024-09-11T01:57:23.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.765+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.765+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=577158405}) 2024-09-11T01:57:23.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.765+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3878796926}) 2024-09-11T01:57:23.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.765+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.765+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=577158405}) 2024-09-11T01:57:23.790 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.765+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3172764508}) 2024-09-11T01:57:23.791 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.766+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.791 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.766+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431740075}) 2024-09-11T01:57:23.791 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.766+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2916165791}) 2024-09-11T01:57:23.791 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.766+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.791 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.766+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431740075}) 2024-09-11T01:57:23.791 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.766+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=929720027}) 2024-09-11T01:57:23.791 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.766+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.791 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.766+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1792669986}) 2024-09-11T01:57:23.791 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.766+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1765968199}) 2024-09-11T01:57:23.791 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.766+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.791 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.766+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1792669986}) 2024-09-11T01:57:23.791 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.766+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2854431280}) 2024-09-11T01:57:23.791 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.767+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.792 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.767+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1704191768}) 2024-09-11T01:57:23.792 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.767+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2464182664}) 2024-09-11T01:57:23.792 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.767+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.792 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.767+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1704191768}) 2024-09-11T01:57:23.792 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.767+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3505767269}) 2024-09-11T01:57:23.792 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.768+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.792 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.768+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2067579320}) 2024-09-11T01:57:23.792 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.768+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=742587834}) 2024-09-11T01:57:23.792 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.768+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.793 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.768+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2067579320}) 2024-09-11T01:57:23.793 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.768+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3340911061}) 2024-09-11T01:57:23.793 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.769+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.793 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.769+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2925441575}) 2024-09-11T01:57:23.793 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.769+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2509682493}) 2024-09-11T01:57:23.793 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.769+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.793 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.769+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2925441575}) 2024-09-11T01:57:23.793 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.769+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2056370215}) 2024-09-11T01:57:23.793 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.769+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.793 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.769+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1794771079}) 2024-09-11T01:57:23.793 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.769+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3512704693}) 2024-09-11T01:57:23.793 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.769+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.794 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.769+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1794771079}) 2024-09-11T01:57:23.794 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.769+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3039498461}) 2024-09-11T01:57:23.794 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.770+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.794 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.770+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3304035334}) 2024-09-11T01:57:23.794 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.770+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2740541826}) 2024-09-11T01:57:23.794 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.770+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.794 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.770+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3304035334}) 2024-09-11T01:57:23.794 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.770+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2767708265}) 2024-09-11T01:57:23.794 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.771+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.794 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.771+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2001377098}) 2024-09-11T01:57:23.794 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.771+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3629385078}) 2024-09-11T01:57:23.794 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.771+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.794 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.771+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2001377098}) 2024-09-11T01:57:23.795 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.771+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3367664413}) 2024-09-11T01:57:23.795 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.772+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.795 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.772+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1662373718}) 2024-09-11T01:57:23.795 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.772+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4026396995}) 2024-09-11T01:57:23.795 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.772+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.795 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.772+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1662373718}) 2024-09-11T01:57:23.795 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.772+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1955734344}) 2024-09-11T01:57:23.795 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.772+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.795 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.772+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3517824765}) 2024-09-11T01:57:23.795 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.772+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1638046806}) 2024-09-11T01:57:23.795 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.772+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.795 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.772+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3517824765}) 2024-09-11T01:57:23.795 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.772+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3077796634}) 2024-09-11T01:57:23.796 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.773+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.796 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.773+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1751029899}) 2024-09-11T01:57:23.796 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.773+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2975701022}) 2024-09-11T01:57:23.796 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.773+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.796 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.773+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1751029899}) 2024-09-11T01:57:23.796 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.773+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4204711024}) 2024-09-11T01:57:23.796 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.774+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.796 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.774+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4249397274}) 2024-09-11T01:57:23.796 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.774+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4286514077}) 2024-09-11T01:57:23.796 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.774+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.796 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.774+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4249397274}) 2024-09-11T01:57:23.796 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.774+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1042588288}) 2024-09-11T01:57:23.813 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.775+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.813 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.775+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=204521381}) 2024-09-11T01:57:23.814 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.775+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=435517192}) 2024-09-11T01:57:23.814 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.775+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.814 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.775+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=204521381}) 2024-09-11T01:57:23.814 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.775+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=629816168}) 2024-09-11T01:57:23.814 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.775+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.814 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.775+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3430209635}) 2024-09-11T01:57:23.814 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.775+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3256261681}) 2024-09-11T01:57:23.814 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.775+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.814 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.775+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3430209635}) 2024-09-11T01:57:23.814 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.775+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1731927935}) 2024-09-11T01:57:23.814 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.776+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.814 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.776+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2505119237}) 2024-09-11T01:57:23.814 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.776+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=202084075}) 2024-09-11T01:57:23.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.776+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.776+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2505119237}) 2024-09-11T01:57:23.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.776+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=9123947}) 2024-09-11T01:57:23.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.777+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.777+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=257707195}) 2024-09-11T01:57:23.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.777+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3954374423}) 2024-09-11T01:57:23.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.777+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.777+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=257707195}) 2024-09-11T01:57:23.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.777+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2026148484}) 2024-09-11T01:57:23.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.777+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.777+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2991971025}) 2024-09-11T01:57:23.815 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.777+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2646853237}) 2024-09-11T01:57:23.816 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.777+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.816 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.778+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2991971025}) 2024-09-11T01:57:23.816 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.778+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3066551404}) 2024-09-11T01:57:23.816 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.778+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.816 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.778+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=463869692}) 2024-09-11T01:57:23.816 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.778+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3449605536}) 2024-09-11T01:57:23.816 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.778+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.816 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.778+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=463869692}) 2024-09-11T01:57:23.816 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.778+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=748735591}) 2024-09-11T01:57:23.816 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.779+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.816 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.779+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=601983294}) 2024-09-11T01:57:23.816 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.779+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3672135644}) 2024-09-11T01:57:23.817 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.779+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.817 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.779+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=601983294}) 2024-09-11T01:57:23.817 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.779+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1815968186}) 2024-09-11T01:57:23.817 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.780+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.817 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.780+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2333142498}) 2024-09-11T01:57:23.817 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.780+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=734751776}) 2024-09-11T01:57:23.817 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.780+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.817 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.780+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2333142498}) 2024-09-11T01:57:23.817 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.780+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3450387413}) 2024-09-11T01:57:23.817 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.780+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.817 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.780+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1098263734}) 2024-09-11T01:57:23.817 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.780+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=627767671}) 2024-09-11T01:57:23.817 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.780+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.780+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1098263734}) 2024-09-11T01:57:23.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.780+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2599564691}) 2024-09-11T01:57:23.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.781+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.781+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1390588831}) 2024-09-11T01:57:23.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.781+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2478563099}) 2024-09-11T01:57:23.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.781+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.781+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1390588831}) 2024-09-11T01:57:23.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.781+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1498461128}) 2024-09-11T01:57:23.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.782+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.782+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1225687175}) 2024-09-11T01:57:23.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.782+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=169739687}) 2024-09-11T01:57:23.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.782+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.818 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.782+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1225687175}) 2024-09-11T01:57:23.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.782+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2986087661}) 2024-09-11T01:57:23.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.783+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.783+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432547469}) 2024-09-11T01:57:23.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.783+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3579909271}) 2024-09-11T01:57:23.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.783+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.783+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432547469}) 2024-09-11T01:57:23.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.783+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3326272643}) 2024-09-11T01:57:23.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.784+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.784+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1915482069}) 2024-09-11T01:57:23.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.784+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1148781569}) 2024-09-11T01:57:23.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.784+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.819 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.784+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1915482069}) 2024-09-11T01:57:23.820 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.784+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1352963731}) 2024-09-11T01:57:23.820 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.784+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.820 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.784+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1907551373}) 2024-09-11T01:57:23.820 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.784+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3942832483}) 2024-09-11T01:57:23.820 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.784+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.820 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.784+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1907551373}) 2024-09-11T01:57:23.820 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.784+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4021266971}) 2024-09-11T01:57:23.820 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.785+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.820 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.785+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2804060714}) 2024-09-11T01:57:23.820 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.785+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2366458476}) 2024-09-11T01:57:23.820 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.785+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.820 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.785+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2804060714}) 2024-09-11T01:57:23.820 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.785+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1935768041}) 2024-09-11T01:57:23.821 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.786+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.821 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.786+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3287100881}) 2024-09-11T01:57:23.821 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.786+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3332789977}) 2024-09-11T01:57:23.821 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.786+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.821 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.786+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3287100881}) 2024-09-11T01:57:23.821 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.786+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1573966043}) 2024-09-11T01:57:23.821 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.786+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.821 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.787+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=758090744}) 2024-09-11T01:57:23.821 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.787+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=106722836}) 2024-09-11T01:57:23.821 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.787+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.821 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.787+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=758090744}) 2024-09-11T01:57:23.821 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.787+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2922123888}) 2024-09-11T01:57:23.821 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.787+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.822 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.787+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=462177921}) 2024-09-11T01:57:23.822 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.787+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1913596546}) 2024-09-11T01:57:23.822 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.787+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.822 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.787+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=462177921}) 2024-09-11T01:57:23.822 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.787+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3718091395}) 2024-09-11T01:57:23.822 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.788+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.822 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.788+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3891333457}) 2024-09-11T01:57:23.822 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.788+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=541669166}) 2024-09-11T01:57:23.822 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.788+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.822 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.788+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3891333457}) 2024-09-11T01:57:23.822 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.788+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1344865345}) 2024-09-11T01:57:23.822 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.788+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.822 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.789+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2942295663}) 2024-09-11T01:57:23.823 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.789+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3492272148}) 2024-09-11T01:57:23.823 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.789+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.823 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.789+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2942295663}) 2024-09-11T01:57:23.823 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.789+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2496077631}) 2024-09-11T01:57:23.823 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.789+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.823 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.789+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2475251369}) 2024-09-11T01:57:23.823 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.789+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=690126240}) 2024-09-11T01:57:23.823 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.789+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.823 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.789+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2475251369}) 2024-09-11T01:57:23.823 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.790+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1453814492}) 2024-09-11T01:57:23.823 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.790+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.823 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.790+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1733746244}) 2024-09-11T01:57:23.823 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.790+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4161886927}) 2024-09-11T01:57:23.824 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.790+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.824 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.790+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1733746244}) 2024-09-11T01:57:23.824 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.790+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4100770207}) 2024-09-11T01:57:23.824 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.791+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.824 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.791+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1330172813}) 2024-09-11T01:57:23.824 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.791+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2018272967}) 2024-09-11T01:57:23.824 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.791+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.824 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.791+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1330172813}) 2024-09-11T01:57:23.824 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.791+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1392158049}) 2024-09-11T01:57:23.824 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.791+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.824 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.791+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1185052548}) 2024-09-11T01:57:23.824 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.791+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2199540210}) 2024-09-11T01:57:23.824 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.791+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.825 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.791+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1185052548}) 2024-09-11T01:57:23.825 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.791+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2692810155}) 2024-09-11T01:57:23.825 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.792+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.825 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.792+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2235311939}) 2024-09-11T01:57:23.825 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.792+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1053208845}) 2024-09-11T01:57:23.825 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.792+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.825 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.792+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2235311939}) 2024-09-11T01:57:23.825 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.792+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=155855023}) 2024-09-11T01:57:23.825 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.793+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.825 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.793+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3148001048}) 2024-09-11T01:57:23.825 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.793+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=346370716}) 2024-09-11T01:57:23.825 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.793+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.825 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.793+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3148001048}) 2024-09-11T01:57:23.826 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.793+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3849662002}) 2024-09-11T01:57:23.826 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.794+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.826 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.794+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=446691230}) 2024-09-11T01:57:23.826 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.794+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=697764803}) 2024-09-11T01:57:23.826 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.794+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.826 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.794+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=446691230}) 2024-09-11T01:57:23.826 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.794+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4173211204}) 2024-09-11T01:57:23.826 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.795+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.826 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.795+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3938826339}) 2024-09-11T01:57:23.826 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.795+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2344404226}) 2024-09-11T01:57:23.826 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.795+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.826 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.795+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3938826339}) 2024-09-11T01:57:23.826 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.795+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1296747566}) 2024-09-11T01:57:23.827 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.795+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.827 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.795+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3952810800}) 2024-09-11T01:57:23.827 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.795+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1349273513}) 2024-09-11T01:57:23.827 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.795+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.827 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.795+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3952810800}) 2024-09-11T01:57:23.827 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.795+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3656412195}) 2024-09-11T01:57:23.827 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.796+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.827 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.796+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4070474892}) 2024-09-11T01:57:23.827 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.796+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1523885912}) 2024-09-11T01:57:23.827 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.796+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.827 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.796+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4070474892}) 2024-09-11T01:57:23.827 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.796+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1623910915}) 2024-09-11T01:57:23.827 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.797+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.828 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.797+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329651591}) 2024-09-11T01:57:23.828 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.797+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1435409865}) 2024-09-11T01:57:23.828 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.797+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.828 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.797+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329651591}) 2024-09-11T01:57:23.828 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.797+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1399517998}) 2024-09-11T01:57:23.828 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.798+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.828 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.798+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=608585294}) 2024-09-11T01:57:23.828 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.798+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2955648541}) 2024-09-11T01:57:23.828 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.798+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.828 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.798+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=608585294}) 2024-09-11T01:57:23.828 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.798+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2410092338}) 2024-09-11T01:57:23.828 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.798+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.829 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.798+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3966986044}) 2024-09-11T01:57:23.829 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.798+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=928359668}) 2024-09-11T01:57:23.829 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.798+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.829 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.798+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3966986044}) 2024-09-11T01:57:23.829 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.798+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=151387339}) 2024-09-11T01:57:23.829 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.799+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.829 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.799+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2720532095}) 2024-09-11T01:57:23.829 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.799+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3748468408}) 2024-09-11T01:57:23.829 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.799+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.829 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.799+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2720532095}) 2024-09-11T01:57:23.829 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.799+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3114065210}) 2024-09-11T01:57:23.829 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.799+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.829 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.799+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562570920}) 2024-09-11T01:57:23.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.799+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1562957380}) 2024-09-11T01:57:23.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.800+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.800+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562570920}) 2024-09-11T01:57:23.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.800+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2687000047}) 2024-09-11T01:57:23.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.800+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.800+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4237430293}) 2024-09-11T01:57:23.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.800+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=728463748}) 2024-09-11T01:57:23.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.800+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.800+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4237430293}) 2024-09-11T01:57:23.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.800+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=362852046}) 2024-09-11T01:57:23.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.801+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.830 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.801+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=468520998}) 2024-09-11T01:57:23.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.801+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2606433029}) 2024-09-11T01:57:23.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.801+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.801+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=468520998}) 2024-09-11T01:57:23.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.801+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3424362792}) 2024-09-11T01:57:23.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.801+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.801+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1171815330}) 2024-09-11T01:57:23.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.801+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2449446408}) 2024-09-11T01:57:23.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.801+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.801+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1171815330}) 2024-09-11T01:57:23.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.802+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1678050498}) 2024-09-11T01:57:23.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.802+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.802+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2638097884}) 2024-09-11T01:57:23.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.802+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4085054668}) 2024-09-11T01:57:23.831 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.802+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.832 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.802+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2638097884}) 2024-09-11T01:57:23.832 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.802+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2168001149}) 2024-09-11T01:57:23.832 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.803+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.832 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.803+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=107178456}) 2024-09-11T01:57:23.832 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.803+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3815193639}) 2024-09-11T01:57:23.832 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.803+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.832 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.803+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=107178456}) 2024-09-11T01:57:23.832 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.803+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=19797629}) 2024-09-11T01:57:23.832 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.804+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.832 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.804+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2035129227}) 2024-09-11T01:57:23.832 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.804+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=188042767}) 2024-09-11T01:57:23.832 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.804+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.832 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.804+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2035129227}) 2024-09-11T01:57:23.833 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.804+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=661287494}) 2024-09-11T01:57:23.833 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.804+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.833 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.804+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3505917046}) 2024-09-11T01:57:23.833 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.804+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1754234077}) 2024-09-11T01:57:23.833 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.804+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.833 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.804+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3505917046}) 2024-09-11T01:57:23.833 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.804+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3664889710}) 2024-09-11T01:57:23.833 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.805+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.833 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.805+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=28099800}) 2024-09-11T01:57:23.833 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.805+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3913654078}) 2024-09-11T01:57:23.833 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.805+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.833 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.805+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=28099800}) 2024-09-11T01:57:23.833 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.805+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3013995925}) 2024-09-11T01:57:23.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f580bfff640 1 -- 172.21.15.12:0/1375355330 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f581c042020 con 0x7f582413e820 2024-09-11T01:57:23.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f580bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:23.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f580bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:57:23.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f580bfff640 10 monclient: dump: 2024-09-11T01:57:23.835 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:23.835 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:23.835 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:23.835 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:23.835 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:23.835 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:23.835 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:23.835 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:23.835 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:23.835 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:23.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f580bfff640 10 monclient: _finish_auth 0 2024-09-11T01:57:23.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f580bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:57:23.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f580bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:23.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f580bfff640 1 -- 172.21.15.12:0/1375355330 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f581c030880 con 0x7f582413e820 2024-09-11T01:57:23.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f5829990640 5 monclient: authenticate success, global_id 294321 2024-09-11T01:57:23.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f5829990640 10 monclient: _renew_subs 2024-09-11T01:57:23.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f5829990640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:23.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f5829990640 1 -- 172.21.15.12:0/1375355330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58240a9730 con 0x7f582413e820 2024-09-11T01:57:23.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f580bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:23.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f580bfff640 1 -- 172.21.15.12:0/1375355330 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f581c037040 con 0x7f582413e820 2024-09-11T01:57:23.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f580bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:23.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f580bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:23.836 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f580bfff640 10 monclient: dump: 2024-09-11T01:57:23.837 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:23.837 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:23.837 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:23.837 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:23.837 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:23.837 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:23.837 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:23.837 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:23.838 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:23.838 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:23.838 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=306733957}) 2024-09-11T01:57:23.838 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4124844743}) 2024-09-11T01:57:23.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f58217fa640 4 set_mon_vals no callback set 2024-09-11T01:57:23.839 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f5829990640 10 monclient: _renew_subs 2024-09-11T01:57:23.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f5829990640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:23.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f5829990640 1 -- 172.21.15.12:0/1375355330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f582413bf70 con 0x7f582413e820 2024-09-11T01:57:23.839 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=306733957}) 2024-09-11T01:57:23.839 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.806+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=863790160}) 2024-09-11T01:57:23.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.807+0000 7f5829990640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:23.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.807+0000 7f5829990640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:23.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.807+0000 7f5829990640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:23.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.807+0000 7f5829990640 1 -- 172.21.15.12:0/1375355330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58241385f0 con 0x7f582413e820 2024-09-11T01:57:23.841 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.807+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.841 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.807+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3804069672}) 2024-09-11T01:57:23.841 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.807+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3201406402}) 2024-09-11T01:57:23.841 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.807+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.841 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.807+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3804069672}) 2024-09-11T01:57:23.842 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.807+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3402790490}) 2024-09-11T01:57:23.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.808+0000 7f580bfff640 1 -- 172.21.15.12:0/1375355330 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f581c03d470 con 0x7f582413e820 2024-09-11T01:57:23.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.808+0000 7f580bfff640 1 --2- 172.21.15.12:0/1375355330 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f581003e290 0x7f5810040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:23.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.808+0000 7f5822ffd640 1 --2- 172.21.15.12:0/1375355330 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f581003e290 0x7f5810040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:23.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.808+0000 7f5822ffd640 10 monclient: get_auth_request con 0x7f581003e290 auth_method 0 2024-09-11T01:57:23.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.808+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.808+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3627611773}) 2024-09-11T01:57:23.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.808+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=609835247}) 2024-09-11T01:57:23.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.808+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.808+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3627611773}) 2024-09-11T01:57:23.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.808+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1711701997}) 2024-09-11T01:57:23.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.809+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.809+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2985248179}) 2024-09-11T01:57:23.844 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.809+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1651639537}) 2024-09-11T01:57:23.845 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.809+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.845 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.809+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2985248179}) 2024-09-11T01:57:23.845 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.809+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3160386919}) 2024-09-11T01:57:23.845 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.810+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.845 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.810+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1975422666}) 2024-09-11T01:57:23.845 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.810+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1317264987}) 2024-09-11T01:57:23.845 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.810+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.845 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.810+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1975422666}) 2024-09-11T01:57:23.845 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.810+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=586369679}) 2024-09-11T01:57:23.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.811+0000 7f5822ffd640 1 --2- 172.21.15.12:0/1375355330 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f581003e290 0x7f5810040750 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f5814004f60 tx=0x7f5814007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:23.848 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.811+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.848 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.811+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2015420441}) 2024-09-11T01:57:23.848 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.811+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=975387625}) 2024-09-11T01:57:23.848 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.811+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.848 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.811+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2015420441}) 2024-09-11T01:57:23.848 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.811+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=139049334}) 2024-09-11T01:57:23.848 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.813+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.848 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.813+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=26,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=20} crc {logm=3763270696,mdsmap=161958762,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2429886782,mon_config_key=3148220264,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=173707361,osd_snap=4215103296}) 2024-09-11T01:57:23.849 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.813+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=26,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=22} crc {logm=3763270696,mdsmap=3760429986,mgr=454318940,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2429886782,mon_config_key=2664027860,monmap=2933275274,nvmeofgw=598337686,osd_metadata=950107459,osd_pg_creating=173707361,osd_snap=3274092666}) 2024-09-11T01:57:23.849 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.813+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.849 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.813+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=26,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=20} crc {logm=3763270696,mdsmap=161958762,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=2429886782,mon_config_key=3148220264,monmap=2933275274,nvmeofgw=598337686,osd_metadata=2942852742,osd_pg_creating=173707361,osd_snap=4215103296}) 2024-09-11T01:57:23.849 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.813+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=26,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=21} crc {logm=3763270696,mdsmap=3760429986,mgr=4124675564,mgr_command_descs=3192186153,mgr_metadata=3338432716,mgrstat=1261996355,mon_config_key=272090419,monmap=1972551139,nvmeofgw=2613832008,osd_metadata=2239697408,osd_pg_creating=173707361,osd_snap=2902858640}) 2024-09-11T01:57:23.849 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.814+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.849 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.814+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=1023336929}) 2024-09-11T01:57:23.849 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.814+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=100} crc {osd_snap=141548728}) 2024-09-11T01:57:23.849 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.814+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.849 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.814+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=1023336929}) 2024-09-11T01:57:23.849 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.814+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=100} crc {osd_snap=1239042812}) 2024-09-11T01:57:23.849 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.815+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.849 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.815+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=56,osdmap=44} crc {osd_snap=730026513,osdmap=2431040962}) 2024-09-11T01:57:23.849 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.815+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=56,osdmap=44} crc {osd_snap=812789646,osdmap=699286680}) 2024-09-11T01:57:23.850 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.815+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.850 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.815+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=56,osdmap=44} crc {osd_snap=730026513,osdmap=2431040962}) 2024-09-11T01:57:23.850 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.815+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=57,osdmap=43} crc {osd_snap=94732934,osdmap=1374606956}) 2024-09-11T01:57:23.850 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.823+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.850 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.823+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=22} crc {osdmap=1161414956}) 2024-09-11T01:57:23.850 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.823+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=21} crc {osdmap=3986077236}) 2024-09-11T01:57:23.850 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.823+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T01:57:23.850 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.823+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=22} crc {osdmap=1161414956}) 2024-09-11T01:57:23.850 INFO:tasks.ceph.mon.a.smithi012.stderr:2024-09-11T01:57:23.823+0000 7f1e53d43640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=23} crc {osdmap=276170909}) 2024-09-11T01:57:23.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.823+0000 7f580bfff640 1 -- 172.21.15.12:0/1375355330 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(923..923 src has 889..923) ==== 4422+0+0 (secure 0 0 0) 0x7f581c030cf0 con 0x7f582413e820 2024-09-11T01:57:23.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.828+0000 7f580bfff640 1 -- 172.21.15.12:0/1375355330 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f581c04c070 con 0x7f582413e820 2024-09-11T01:57:23.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.828+0000 7f580bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:23.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:23.828+0000 7f580bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:57:24.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.084+0000 7f5829990640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T01:57:24.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.084+0000 7f5829990640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T01:57:24.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.084+0000 7f5829990640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:24.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.084+0000 7f5829990640 1 -- 172.21.15.12:0/1375355330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5824074b50 con 0x7f582413e820 2024-09-11T01:57:24.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.084+0000 7f580bfff640 1 -- 172.21.15.12:0/1375355330 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f582409ba20 con 0x7f582413e820 2024-09-11T01:57:24.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.084+0000 7f580bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T01:57:24.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.084+0000 7f580bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:57:24.086 INFO:teuthology.orchestra.run.smithi012.stdout:{"election_epoch":260,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","modified":"2024-09-11T01:28:30.421598Z","created":"2024-09-11T01:28:30.421598Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T01:57:24.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.086+0000 7f5829990640 1 -- 172.21.15.12:0/1375355330 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f581003e290 msgr2=0x7f5810040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:24.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.086+0000 7f5829990640 1 --2- 172.21.15.12:0/1375355330 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f581003e290 0x7f5810040750 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f5814004f60 tx=0x7f5814007e30 comp rx=0 tx=0).stop 2024-09-11T01:57:24.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.086+0000 7f5829990640 10 monclient: shutdown 2024-09-11T01:57:24.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.086+0000 7f5829990640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:24.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.086+0000 7f5829990640 1 -- 172.21.15.12:0/1375355330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f582413e820 msgr2=0x7f5824139a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:24.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.086+0000 7f5829990640 1 --2- 172.21.15.12:0/1375355330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f582413e820 0x7f5824139a40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f581c0098c0 tx=0x7f581c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.086+0000 7f5829990640 1 -- 172.21.15.12:0/1375355330 shutdown_connections 2024-09-11T01:57:24.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.086+0000 7f5829990640 1 --2- 172.21.15.12:0/1375355330 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f581003e290 0x7f5810040750 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.086+0000 7f5829990640 1 --2- 172.21.15.12:0/1375355330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f582413e820 0x7f5824139a40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.086+0000 7f5829990640 1 --2- 172.21.15.12:0/1375355330 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5824138210 0x7f58241cd970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.086+0000 7f5829990640 1 --2- 172.21.15.12:0/1375355330 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f58240a6810 0x7f58241cd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.086+0000 7f5829990640 1 -- 172.21.15.12:0/1375355330 >> 172.21.15.12:0/1375355330 conn(0x7f58240aaa80 msgr2=0x7f582413ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:24.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.086+0000 7f5829990640 1 -- 172.21.15.12:0/1375355330 shutdown_connections 2024-09-11T01:57:24.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.086+0000 7f5829990640 1 -- 172.21.15.12:0/1375355330 wait complete. 2024-09-11T01:57:24.098 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T01:57:24.099 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-11T01:57:24.108 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T01:57:24.108 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T01:57:24.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.237+0000 7f4329984640 10 monclient: get_monmap_and_config 2024-09-11T01:57:24.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.237+0000 7f4329984640 10 monclient: build_initial_monmap 2024-09-11T01:57:24.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.237+0000 7f4329984640 10 monclient: monmap: 2024-09-11T01:57:24.239 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:24.239 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:57:24.238663+0000 2024-09-11T01:57:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:57:24.238663+0000 2024-09-11T01:57:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:57:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:57:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:57:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:57:24.240 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:57:24.240 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:24.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.238+0000 7f4329984640 1 Processor -- start 2024-09-11T01:57:24.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.238+0000 7f4329984640 1 -- start start 2024-09-11T01:57:24.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.238+0000 7f4329984640 10 monclient: init 2024-09-11T01:57:24.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.238+0000 7f4329984640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:24.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f4329984640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-11T01:57:24.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f4329984640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4324134ad0 addr v1:172.21.15.12:6789/0 2024-09-11T01:57:24.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f4329984640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f43240a2200 addr v1:172.21.15.71:6789/0 2024-09-11T01:57:24.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f4329984640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f43241355c0 0x7f432413d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:24.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f4329984640 10 monclient(hunting): picked mon.noname-a con 0x7f43241355c0 addr v2:172.21.15.12:3300/0 2024-09-11T01:57:24.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f4329984640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f432409ed30 con 0x7f4324134ad0 2024-09-11T01:57:24.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f4329984640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f432409f010 con 0x7f43240a2200 2024-09-11T01:57:24.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f4329984640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:24.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f4329984640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f432409f5d0 con 0x7f43241355c0 2024-09-11T01:57:24.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f4329984640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:24.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f4329984640 10 monclient(hunting): authenticate will time out at 2450.681100s 2024-09-11T01:57:24.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f43237fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f43241355c0 0x7f432413d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:24.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f4322ffd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4324134ad0 0x7f4324134eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:39172/0 (socket says 172.21.15.12:39172) 2024-09-11T01:57:24.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f4322ffd640 1 -- 172.21.15.12:0/1173090547 learned_addr learned my addr 172.21.15.12:0/1173090547 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:57:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f43237fe640 10 monclient(hunting): get_auth_request con 0x7f43241355c0 auth_method 0 2024-09-11T01:57:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f43237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f43237fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f43237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f43237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f43237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f43237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f43237fe640 10 monclient(hunting): handle_auth_done global_id 294324 payload 274 2024-09-11T01:57:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f43237fe640 1 -- 172.21.15.12:0/1173090547 >> v1:172.21.15.71:6789/0 conn(0x7f43240a2200 legacy=0x7f432413a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f43237fe640 1 -- 172.21.15.12:0/1173090547 >> v1:172.21.15.12:6789/0 conn(0x7f4324134ad0 legacy=0x7f4324134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f43237fe640 10 monclient: _finish_hunting 0 2024-09-11T01:57:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f43237fe640 1 monclient: found mon.noname-a 2024-09-11T01:57:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f43237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f43237fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:57:24.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.239+0000 7f43237fe640 1 -- 172.21.15.12:0/1173090547 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f432409f2f0 con 0x7f43241355c0 2024-09-11T01:57:24.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.242+0000 7f43237fe640 1 -- 172.21.15.12:0/1173090547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43241355c0 msgr2=0x7f432413d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:24.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.242+0000 7f43237fe640 1 --2- 172.21.15.12:0/1173090547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43241355c0 0x7f432413d730 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f432409f750 tx=0x7f4314029fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dc7be04b1fa912e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:24.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.242+0000 7f4321ffb640 1 -- 172.21.15.12:0/1173090547 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4314037070 con 0x7f43241355c0 2024-09-11T01:57:24.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.242+0000 7f4321ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:24.248 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.242+0000 7f4321ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:57:24.249 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.242+0000 7f4321ffb640 10 monclient: dump: 2024-09-11T01:57:24.249 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:24.249 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:24.249 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:24.249 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:24.249 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:24.249 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:24.249 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:24.249 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:24.249 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:24.249 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:24.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.242+0000 7f4321ffb640 10 monclient: _finish_auth 0 2024-09-11T01:57:24.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.242+0000 7f4321ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:57:24.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.242+0000 7f4321ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:24.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.242+0000 7f4321ffb640 1 -- 172.21.15.12:0/1173090547 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4314032440 con 0x7f43241355c0 2024-09-11T01:57:24.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.242+0000 7f4329984640 5 monclient: authenticate success, global_id 294324 2024-09-11T01:57:24.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.242+0000 7f4329984640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:57:24.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.242+0000 7f4321ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:24.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.242+0000 7f4329984640 10 monclient: get_monmap_and_config success 2024-09-11T01:57:24.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.242+0000 7f4321ffb640 1 -- 172.21.15.12:0/1173090547 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43140327f0 con 0x7f43241355c0 2024-09-11T01:57:24.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.242+0000 7f4329984640 4 set_mon_vals no callback set 2024-09-11T01:57:24.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.242+0000 7f4321ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:24.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.242+0000 7f4321ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:24.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.242+0000 7f4321ffb640 10 monclient: dump: 2024-09-11T01:57:24.251 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:24.251 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:24.251 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:24.251 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:24.251 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:24.251 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:24.251 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:24.251 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:24.251 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:24.251 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:24.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.243+0000 7f4329984640 10 monclient: shutdown 2024-09-11T01:57:24.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.243+0000 7f4329984640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:24.251 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.243+0000 7f4329984640 1 -- 172.21.15.12:0/1173090547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43241355c0 msgr2=0x7f432413d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:24.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.243+0000 7f4329984640 1 --2- 172.21.15.12:0/1173090547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43241355c0 0x7f432413d730 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f432409f750 tx=0x7f4314029fe0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.243+0000 7f4329984640 1 -- 172.21.15.12:0/1173090547 shutdown_connections 2024-09-11T01:57:24.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.243+0000 7f4329984640 1 --2- 172.21.15.12:0/1173090547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43241355c0 0x7f432413d730 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.243+0000 7f4329984640 1 -- 172.21.15.12:0/1173090547 >> 172.21.15.12:0/1173090547 conn(0x7f43240a2f70 msgr2=0x7f43240a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:24.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.243+0000 7f4329984640 1 -- 172.21.15.12:0/1173090547 shutdown_connections 2024-09-11T01:57:24.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.243+0000 7f4329984640 1 -- 172.21.15.12:0/1173090547 wait complete. 2024-09-11T01:57:24.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.243+0000 7f4329984640 10 monclient: build_initial_monmap 2024-09-11T01:57:24.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.243+0000 7f4329984640 10 monclient: monmap: 2024-09-11T01:57:24.252 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:24.252 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:57:24.252 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:57:24.252 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:57:24.252 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:24.252 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:24.253 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:57:24.253 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:57:24.253 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:57:24.253 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:24.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f4329984640 1 Processor -- start 2024-09-11T01:57:24.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f4329984640 1 -- start start 2024-09-11T01:57:24.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f4329984640 10 monclient: init 2024-09-11T01:57:24.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f4329984640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:24.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f4329984640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:57:24.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f4329984640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f43240a2200 0x7f4324133a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:24.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f4329984640 10 monclient(hunting): picked mon.noname-c con 0x7f43240a2200 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:24.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f4329984640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4324134ad0 0x7f4324133f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:24.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f4329984640 10 monclient(hunting): picked mon.noname-b con 0x7f4324134ad0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:57:24.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f4329984640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43241355c0 0x7f43241ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:24.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f4329984640 10 monclient(hunting): picked mon.noname-a con 0x7f43241355c0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:57:24.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f4329984640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:24.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f4329984640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f43240a0150 con 0x7f43241355c0 2024-09-11T01:57:24.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f4329984640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:24.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f4329984640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f432409ffd0 con 0x7f4324134ad0 2024-09-11T01:57:24.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f4329984640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:24.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f4329984640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f432413dc70 con 0x7f43240a2200 2024-09-11T01:57:24.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f4329984640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:24.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f4329984640 10 monclient(hunting): authenticate will time out at 2450.686776s 2024-09-11T01:57:24.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f4322ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f43240a2200 0x7f4324133a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:24.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f43237fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43241355c0 0x7f43241ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:24.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f43237fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43241355c0 0x7f43241ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45380/0 (socket says 172.21.15.12:45380) 2024-09-11T01:57:24.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f43237fe640 1 -- 172.21.15.12:0/849980339 learned_addr learned my addr 172.21.15.12:0/849980339 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:24.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f43237fe640 10 monclient(hunting): get_auth_request con 0x7f43241355c0 auth_method 0 2024-09-11T01:57:24.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f43237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:24.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f43237fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:24.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f43237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:24.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f4322ffd640 10 monclient(hunting): get_auth_request con 0x7f43240a2200 auth_method 0 2024-09-11T01:57:24.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f4322ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:24.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f4322ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:24.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f4322ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:24.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f43237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:24.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f43237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:24.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f43237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:24.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f4322ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:24.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.244+0000 7f4322ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:24.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.245+0000 7f4322ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:24.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.245+0000 7f43237fe640 10 monclient(hunting): handle_auth_done global_id 294327 payload 824 2024-09-11T01:57:24.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.245+0000 7f43227fc640 1 --2- 172.21.15.12:0/849980339 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4324134ad0 0x7f4324133f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:24.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.245+0000 7f43237fe640 1 -- 172.21.15.12:0/849980339 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f43240a2200 msgr2=0x7f4324133a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:24.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.245+0000 7f43237fe640 1 --2- 172.21.15.12:0/849980339 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f43240a2200 0x7f4324133a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.245+0000 7f43237fe640 1 -- 172.21.15.12:0/849980339 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4324134ad0 msgr2=0x7f4324133f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:24.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.245+0000 7f43237fe640 1 --2- 172.21.15.12:0/849980339 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4324134ad0 0x7f4324133f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.245+0000 7f43237fe640 10 monclient: _finish_hunting 0 2024-09-11T01:57:24.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.245+0000 7f43237fe640 1 monclient: found mon.noname-a 2024-09-11T01:57:24.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.245+0000 7f43237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:24.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.245+0000 7f43237fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:57:24.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.245+0000 7f43237fe640 1 -- 172.21.15.12:0/849980339 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f432409d480 con 0x7f43241355c0 2024-09-11T01:57:24.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.245+0000 7f4322ffd640 1 --2- 172.21.15.12:0/849980339 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f43240a2200 0x7f4324133a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:57:24.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.247+0000 7f43237fe640 1 -- 172.21.15.12:0/849980339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43241355c0 msgr2=0x7f43241ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:24.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.247+0000 7f43237fe640 1 --2- 172.21.15.12:0/849980339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43241355c0 0x7f43241ccf60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f43140097a0 tx=0x7f43140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:24.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.247+0000 7f431bfff640 1 -- 172.21.15.12:0/849980339 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4314041070 con 0x7f43241355c0 2024-09-11T01:57:24.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.247+0000 7f431bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:24.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.247+0000 7f431bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:57:24.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.247+0000 7f431bfff640 10 monclient: dump: 2024-09-11T01:57:24.259 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:24.259 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:24.259 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:24.259 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:24.259 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:24.259 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:24.259 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:24.259 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:24.259 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:24.259 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:24.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.247+0000 7f431bfff640 10 monclient: _finish_auth 0 2024-09-11T01:57:24.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.247+0000 7f431bfff640 10 monclient: _check_auth_tickets 2024-09-11T01:57:24.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.247+0000 7f431bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:24.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.247+0000 7f431bfff640 1 -- 172.21.15.12:0/849980339 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f431403c870 con 0x7f43241355c0 2024-09-11T01:57:24.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.247+0000 7f4329984640 5 monclient: authenticate success, global_id 294327 2024-09-11T01:57:24.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.247+0000 7f4329984640 10 monclient: _renew_subs 2024-09-11T01:57:24.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.247+0000 7f4329984640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:24.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.247+0000 7f4329984640 1 -- 172.21.15.12:0/849980339 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43240a56a0 con 0x7f43241355c0 2024-09-11T01:57:24.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.247+0000 7f431bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:24.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.247+0000 7f431bfff640 1 -- 172.21.15.12:0/849980339 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4314037040 con 0x7f43241355c0 2024-09-11T01:57:24.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.247+0000 7f431bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:24.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.247+0000 7f431bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:24.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.247+0000 7f431bfff640 10 monclient: dump: 2024-09-11T01:57:24.261 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:24.261 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:24.261 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:24.261 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:24.261 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:24.261 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:24.261 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:24.261 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:24.261 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:24.261 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:24.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.247+0000 7f43217fa640 4 set_mon_vals no callback set 2024-09-11T01:57:24.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.247+0000 7f4329984640 10 monclient: _renew_subs 2024-09-11T01:57:24.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.247+0000 7f4329984640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:24.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.247+0000 7f4329984640 1 -- 172.21.15.12:0/849980339 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f432413c610 con 0x7f43241355c0 2024-09-11T01:57:24.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.248+0000 7f4329984640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:24.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.248+0000 7f4329984640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:24.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.248+0000 7f4329984640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:24.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.248+0000 7f431bfff640 1 -- 172.21.15.12:0/849980339 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f431403cc60 con 0x7f43241355c0 2024-09-11T01:57:24.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.248+0000 7f4329984640 1 -- 172.21.15.12:0/849980339 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4324134eb0 con 0x7f43241355c0 2024-09-11T01:57:24.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.248+0000 7f431bfff640 1 --2- 172.21.15.12:0/849980339 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f430403e240 0x7f4304040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:24.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.248+0000 7f431bfff640 1 -- 172.21.15.12:0/849980339 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(923..923 src has 889..923) ==== 4422+0+0 (secure 0 0 0) 0x7f4314073050 con 0x7f43241355c0 2024-09-11T01:57:24.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.249+0000 7f4322ffd640 1 --2- 172.21.15.12:0/849980339 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f430403e240 0x7f4304040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:24.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.249+0000 7f4322ffd640 10 monclient: get_auth_request con 0x7f430403e240 auth_method 0 2024-09-11T01:57:24.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.251+0000 7f4322ffd640 1 --2- 172.21.15.12:0/849980339 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f430403e240 0x7f4304040700 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f431c00d5a0 tx=0x7f431c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:24.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.253+0000 7f431bfff640 1 -- 172.21.15.12:0/849980339 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f431404b020 con 0x7f43241355c0 2024-09-11T01:57:24.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.253+0000 7f431bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:24.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.253+0000 7f431bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:57:24.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.432+0000 7f4329984640 1 -- 172.21.15.12:0/849980339 --> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f432413cd80 con 0x7f430403e240 2024-09-11T01:57:24.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.434+0000 7f431bfff640 1 -- 172.21.15.12:0/849980339 <== mgr.4101 v2:172.21.15.71:6816/3189259519 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34024 (secure 0 0 0) 0x7f432413cd80 con 0x7f430403e240 2024-09-11T01:57:24.436 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:57:24.436 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-09-11T01:57:24.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.436+0000 7f4329984640 1 -- 172.21.15.12:0/849980339 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f430403e240 msgr2=0x7f4304040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:24.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.436+0000 7f4329984640 1 --2- 172.21.15.12:0/849980339 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f430403e240 0x7f4304040700 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f431c00d5a0 tx=0x7f431c009040 comp rx=0 tx=0).stop 2024-09-11T01:57:24.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.436+0000 7f4329984640 10 monclient: shutdown 2024-09-11T01:57:24.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.436+0000 7f4329984640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:24.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.436+0000 7f4329984640 1 -- 172.21.15.12:0/849980339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43241355c0 msgr2=0x7f43241ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:24.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.436+0000 7f4329984640 1 --2- 172.21.15.12:0/849980339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43241355c0 0x7f43241ccf60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f43140097a0 tx=0x7f43140046c0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.436+0000 7f4329984640 1 -- 172.21.15.12:0/849980339 shutdown_connections 2024-09-11T01:57:24.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.436+0000 7f4329984640 1 --2- 172.21.15.12:0/849980339 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f430403e240 0x7f4304040700 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.436+0000 7f4329984640 1 --2- 172.21.15.12:0/849980339 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43241355c0 0x7f43241ccf60 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.436+0000 7f4329984640 1 --2- 172.21.15.12:0/849980339 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4324134ad0 0x7f4324133f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.436+0000 7f4329984640 1 --2- 172.21.15.12:0/849980339 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f43240a2200 0x7f4324133a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.436+0000 7f4329984640 1 -- 172.21.15.12:0/849980339 >> 172.21.15.12:0/849980339 conn(0x7f43240a2f70 msgr2=0x7f43240a48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:24.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.436+0000 7f4329984640 1 -- 172.21.15.12:0/849980339 shutdown_connections 2024-09-11T01:57:24.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.436+0000 7f4329984640 1 -- 172.21.15.12:0/849980339 wait complete. 2024-09-11T01:57:24.448 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":1766,"stamp":"2024-09-11T01:57:23.208437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":333208,"kb_used_data":14816,"kb_used_omap":9,"kb_used_meta":317366,"kb_avail":565897832,"statfs":{"total":579820584960,"available":579479379968,"internally_reserved":0,"allocated":15171584,"data_stored":7598212,"data_compressed":59136,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":324983456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002313"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":1807,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:44.358489+0000","last_change":"2024-09-11T01:29:17.579548+0000","last_active":"2024-09-11T01:56:44.358489+0000","last_peered":"2024-09-11T01:56:44.358489+0000","last_clean":"2024-09-11T01:56:44.358489+0000","last_became_active":"2024-09-11T01:29:15.546952+0000","last_became_peered":"2024-09-11T01:29:15.546952+0000","last_unstale":"2024-09-11T01:56:44.358489+0000","last_undegraded":"2024-09-11T01:56:44.358489+0000","last_fullsized":"2024-09-11T01:56:44.358489+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:45:28.460820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038245800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":1812,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:43.746370+0000","last_change":"2024-09-11T01:29:17.597459+0000","last_active":"2024-09-11T01:56:43.746370+0000","last_peered":"2024-09-11T01:56:43.746370+0000","last_clean":"2024-09-11T01:56:43.746370+0000","last_became_active":"2024-09-11T01:29:15.549013+0000","last_became_peered":"2024-09-11T01:29:15.549013+0000","last_unstale":"2024-09-11T01:56:43.746370+0000","last_undegraded":"2024-09-11T01:56:43.746370+0000","last_fullsized":"2024-09-11T01:56:43.746370+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:46:02.130174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041667199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":1822,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:43.904328+0000","last_change":"2024-09-11T01:29:17.577463+0000","last_active":"2024-09-11T01:56:43.904328+0000","last_peered":"2024-09-11T01:56:43.904328+0000","last_clean":"2024-09-11T01:56:43.904328+0000","last_became_active":"2024-09-11T01:29:15.548884+0000","last_became_peered":"2024-09-11T01:29:15.548884+0000","last_unstale":"2024-09-11T01:56:43.904328+0000","last_undegraded":"2024-09-11T01:56:43.904328+0000","last_fullsized":"2024-09-11T01:56:43.904328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:09.120675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094832499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":1817,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:44.360535+0000","last_change":"2024-09-11T01:29:17.583117+0000","last_active":"2024-09-11T01:56:44.360535+0000","last_peered":"2024-09-11T01:56:44.360535+0000","last_clean":"2024-09-11T01:56:44.360535+0000","last_became_active":"2024-09-11T01:29:15.546142+0000","last_became_peered":"2024-09-11T01:29:15.546142+0000","last_unstale":"2024-09-11T01:56:44.360535+0000","last_undegraded":"2024-09-11T01:56:44.360535+0000","last_fullsized":"2024-09-11T01:56:44.360535+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:16:49.024029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044411199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"16'72","reported_seq":1890,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:44.358484+0000","last_change":"2024-09-11T01:29:16.691671+0000","last_active":"2024-09-11T01:56:44.358484+0000","last_peered":"2024-09-11T01:56:44.358484+0000","last_clean":"2024-09-11T01:56:44.358484+0000","last_became_active":"2024-09-11T01:29:16.690647+0000","last_became_peered":"2024-09-11T01:29:16.690647+0000","last_unstale":"2024-09-11T01:56:44.358484+0000","last_undegraded":"2024-09-11T01:56:44.358484+0000","last_fullsized":"2024-09-11T01:56:44.358484+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:15.533298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:15.533298+0000","last_clean_scrub_stamp":"2024-09-11T01:29:15.533298+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:41:07.416896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":1816,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:44.360506+0000","last_change":"2024-09-11T01:29:17.583012+0000","last_active":"2024-09-11T01:56:44.360506+0000","last_peered":"2024-09-11T01:56:44.360506+0000","last_clean":"2024-09-11T01:56:44.360506+0000","last_became_active":"2024-09-11T01:29:15.547447+0000","last_became_peered":"2024-09-11T01:29:15.547447+0000","last_unstale":"2024-09-11T01:56:44.360506+0000","last_undegraded":"2024-09-11T01:56:44.360506+0000","last_fullsized":"2024-09-11T01:56:44.360506+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:17:48.616008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037448800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":1820,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:43.904334+0000","last_change":"2024-09-11T01:29:17.576853+0000","last_active":"2024-09-11T01:56:43.904334+0000","last_peered":"2024-09-11T01:56:43.904334+0000","last_clean":"2024-09-11T01:56:43.904334+0000","last_became_active":"2024-09-11T01:29:15.546476+0000","last_became_peered":"2024-09-11T01:29:15.546476+0000","last_unstale":"2024-09-11T01:56:43.904334+0000","last_undegraded":"2024-09-11T01:56:43.904334+0000","last_fullsized":"2024-09-11T01:56:43.904334+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:11:54.706642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037031899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":1806,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:44.358504+0000","last_change":"2024-09-11T01:29:17.579590+0000","last_active":"2024-09-11T01:56:44.358504+0000","last_peered":"2024-09-11T01:56:44.358504+0000","last_clean":"2024-09-11T01:56:44.358504+0000","last_became_active":"2024-09-11T01:29:15.542800+0000","last_became_peered":"2024-09-11T01:29:15.542800+0000","last_unstale":"2024-09-11T01:56:44.358504+0000","last_undegraded":"2024-09-11T01:56:44.358504+0000","last_fullsized":"2024-09-11T01:56:44.358504+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:23:55.773396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040701099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":1811,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:43.746418+0000","last_change":"2024-09-11T01:29:17.597492+0000","last_active":"2024-09-11T01:56:43.746418+0000","last_peered":"2024-09-11T01:56:43.746418+0000","last_clean":"2024-09-11T01:56:43.746418+0000","last_became_active":"2024-09-11T01:29:15.546444+0000","last_became_peered":"2024-09-11T01:29:15.546444+0000","last_unstale":"2024-09-11T01:56:43.746418+0000","last_undegraded":"2024-09-11T01:56:43.746418+0000","last_fullsized":"2024-09-11T01:56:43.746418+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:52:43.624650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000396053,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59136,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607894,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47704,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324136,"statfs":{"total":96636764160,"available":96587915264,"internally_reserved":0,"allocated":2506752,"data_stored":1090053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607892,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56028,"kb_used_data":2452,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315812,"statfs":{"total":96636764160,"available":96579391488,"internally_reserved":0,"allocated":2510848,"data_stored":1090072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607893,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56084,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315756,"statfs":{"total":96636764160,"available":96579334144,"internally_reserved":0,"allocated":2568192,"data_stored":1618981,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607893,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69724,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302116,"statfs":{"total":96636764160,"available":96565366784,"internally_reserved":0,"allocated":2568192,"data_stored":1618981,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607893,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33820,"kb_used_data":2452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338020,"statfs":{"total":96636764160,"available":96602132480,"internally_reserved":0,"allocated":2510848,"data_stored":1090072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640600,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69848,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94301992,"statfs":{"total":96636764160,"available":96565239808,"internally_reserved":0,"allocated":2506752,"data_stored":1090053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T01:57:24.449 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T01:57:24.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.551+0000 7f9eaf2e6640 10 monclient: get_monmap_and_config 2024-09-11T01:57:24.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.551+0000 7f9eaf2e6640 10 monclient: build_initial_monmap 2024-09-11T01:57:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.551+0000 7f9eaf2e6640 10 monclient: monmap: 2024-09-11T01:57:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:57:24.551903+0000 2024-09-11T01:57:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:57:24.551903+0000 2024-09-11T01:57:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:57:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:57:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:57:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:57:24.553 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:57:24.553 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.551+0000 7f9eaf2e6640 1 Processor -- start 2024-09-11T01:57:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.551+0000 7f9eaf2e6640 1 -- start start 2024-09-11T01:57:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.551+0000 7f9eaf2e6640 10 monclient: init 2024-09-11T01:57:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eaf2e6640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eaf2e6640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-11T01:57:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eaf2e6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9ea8134ad0 addr v1:172.21.15.12:6789/0 2024-09-11T01:57:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eaf2e6640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9ea80a2200 0x7f9ea813a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eaf2e6640 10 monclient(hunting): picked mon.noname-a con 0x7f9ea80a2200 addr v2:172.21.15.12:3300/0 2024-09-11T01:57:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eaf2e6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ea81355c0 0x7f9ea813c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eaf2e6640 10 monclient(hunting): picked mon.noname-b con 0x7f9ea81355c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:24.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eaf2e6640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ea809ed30 con 0x7f9ea8134ad0 2024-09-11T01:57:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eaf2e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eaf2e6640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f9ea809f5d0 con 0x7f9ea80a2200 2024-09-11T01:57:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eaf2e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eaf2e6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f9ea809f010 con 0x7f9ea81355c0 2024-09-11T01:57:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eaf2e6640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eaf2e6640 10 monclient(hunting): authenticate will time out at 2450.994517s 2024-09-11T01:57:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eac881640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9ea80a2200 0x7f9ea813a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9ead883640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ea81355c0 0x7f9ea813c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9ead082640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f9ea8134ad0 0x7f9ea8134eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:39178/0 (socket says 172.21.15.12:39178) 2024-09-11T01:57:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9ead082640 1 -- 172.21.15.12:0/3600595774 learned_addr learned my addr 172.21.15.12:0/3600595774 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:57:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9ead883640 10 monclient(hunting): get_auth_request con 0x7f9ea81355c0 auth_method 0 2024-09-11T01:57:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9ead883640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:24.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9ead883640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9ead883640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eac881640 10 monclient(hunting): get_auth_request con 0x7f9ea80a2200 auth_method 0 2024-09-11T01:57:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eac881640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eac881640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eac881640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eac881640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eac881640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eac881640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9ead883640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9ead883640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9ead883640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eac881640 10 monclient(hunting): handle_auth_done global_id 294330 payload 274 2024-09-11T01:57:24.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eac881640 1 -- 172.21.15.12:0/3600595774 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ea81355c0 msgr2=0x7f9ea813c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eac881640 1 --2- 172.21.15.12:0/3600595774 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ea81355c0 0x7f9ea813c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eac881640 1 -- 172.21.15.12:0/3600595774 >> v1:172.21.15.12:6789/0 conn(0x7f9ea8134ad0 legacy=0x7f9ea8134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eac881640 10 monclient: _finish_hunting 0 2024-09-11T01:57:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eac881640 1 monclient: found mon.noname-a 2024-09-11T01:57:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eac881640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eac881640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:57:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9eac881640 1 -- 172.21.15.12:0/3600595774 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ea809f2f0 con 0x7f9ea80a2200 2024-09-11T01:57:24.559 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.552+0000 7f9ead883640 1 --2- 172.21.15.12:0/3600595774 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ea81355c0 0x7f9ea813c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:57:24.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.555+0000 7f9eac881640 1 -- 172.21.15.12:0/3600595774 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ea80a2200 msgr2=0x7f9ea813a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:24.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.555+0000 7f9eac881640 1 --2- 172.21.15.12:0/3600595774 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ea80a2200 0x7f9ea813a000 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9e980097d0 tx=0x7f9e9802a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36b5eb07e7dbc3bb server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:24.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.555+0000 7f9e9ffff640 1 -- 172.21.15.12:0/3600595774 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e98037070 con 0x7f9ea80a2200 2024-09-11T01:57:24.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.555+0000 7f9e9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:24.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.555+0000 7f9e9ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:57:24.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.555+0000 7f9e9ffff640 10 monclient: dump: 2024-09-11T01:57:24.564 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:24.564 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:24.564 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:24.565 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.555+0000 7f9e9ffff640 10 monclient: _finish_auth 0 2024-09-11T01:57:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.555+0000 7f9e9ffff640 10 monclient: _check_auth_tickets 2024-09-11T01:57:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.555+0000 7f9e9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.555+0000 7f9eaf2e6640 5 monclient: authenticate success, global_id 294330 2024-09-11T01:57:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.555+0000 7f9eaf2e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:57:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.555+0000 7f9e9ffff640 1 -- 172.21.15.12:0/3600595774 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e98032440 con 0x7f9ea80a2200 2024-09-11T01:57:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.555+0000 7f9e9ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.555+0000 7f9e9ffff640 1 -- 172.21.15.12:0/3600595774 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e98032810 con 0x7f9ea80a2200 2024-09-11T01:57:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.555+0000 7f9eaf2e6640 10 monclient: get_monmap_and_config success 2024-09-11T01:57:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.555+0000 7f9eaf2e6640 4 set_mon_vals no callback set 2024-09-11T01:57:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.555+0000 7f9e9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.555+0000 7f9e9ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.555+0000 7f9e9ffff640 10 monclient: dump: 2024-09-11T01:57:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:24.567 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.556+0000 7f9eaf2e6640 10 monclient: shutdown 2024-09-11T01:57:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.556+0000 7f9eaf2e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.556+0000 7f9eaf2e6640 1 -- 172.21.15.12:0/3600595774 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ea80a2200 msgr2=0x7f9ea813a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.556+0000 7f9eaf2e6640 1 --2- 172.21.15.12:0/3600595774 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ea80a2200 0x7f9ea813a000 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9e980097d0 tx=0x7f9e9802a000 comp rx=0 tx=0).stop 2024-09-11T01:57:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.556+0000 7f9eaf2e6640 1 -- 172.21.15.12:0/3600595774 shutdown_connections 2024-09-11T01:57:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.556+0000 7f9eaf2e6640 1 --2- 172.21.15.12:0/3600595774 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ea81355c0 0x7f9ea813c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.556+0000 7f9eaf2e6640 1 --2- 172.21.15.12:0/3600595774 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ea80a2200 0x7f9ea813a000 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.556+0000 7f9eaf2e6640 1 -- 172.21.15.12:0/3600595774 >> 172.21.15.12:0/3600595774 conn(0x7f9ea80a2f70 msgr2=0x7f9ea80a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.556+0000 7f9eaf2e6640 1 -- 172.21.15.12:0/3600595774 shutdown_connections 2024-09-11T01:57:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.556+0000 7f9eaf2e6640 1 -- 172.21.15.12:0/3600595774 wait complete. 2024-09-11T01:57:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.557+0000 7f9eaf2e6640 10 monclient: build_initial_monmap 2024-09-11T01:57:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.557+0000 7f9eaf2e6640 10 monclient: monmap: 2024-09-11T01:57:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:57:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:57:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:57:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:57:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:57:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:57:24.569 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.557+0000 7f9eaf2e6640 1 Processor -- start 2024-09-11T01:57:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.557+0000 7f9eaf2e6640 1 -- start start 2024-09-11T01:57:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.557+0000 7f9eaf2e6640 10 monclient: init 2024-09-11T01:57:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.557+0000 7f9eaf2e6640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.557+0000 7f9eaf2e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:57:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.557+0000 7f9eaf2e6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ea80a2200 0x7f9ea81cd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.557+0000 7f9eaf2e6640 10 monclient(hunting): picked mon.noname-c con 0x7f9ea80a2200 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.557+0000 7f9eaf2e6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ea8134ad0 0x7f9ea8133c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.557+0000 7f9eaf2e6640 10 monclient(hunting): picked mon.noname-a con 0x7f9ea8134ad0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:57:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.557+0000 7f9eaf2e6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ea81355c0 0x7f9ea8134190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.557+0000 7f9eaf2e6640 10 monclient(hunting): picked mon.noname-b con 0x7f9ea81355c0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:57:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.557+0000 7f9eaf2e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:24.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.557+0000 7f9eaf2e6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea80a0150 con 0x7f9ea8134ad0 2024-09-11T01:57:24.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.557+0000 7f9eaf2e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:24.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.557+0000 7f9eaf2e6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea809ffd0 con 0x7f9ea81355c0 2024-09-11T01:57:24.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.557+0000 7f9eaf2e6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:24.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.557+0000 7f9eaf2e6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f9ea80a0c90 con 0x7f9ea80a2200 2024-09-11T01:57:24.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.557+0000 7f9eaf2e6640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:24.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.557+0000 7f9eaf2e6640 10 monclient(hunting): authenticate will time out at 2450.999990s 2024-09-11T01:57:24.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9ead082640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ea80a2200 0x7f9ea81cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:24.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9eac881640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ea8134ad0 0x7f9ea8133c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:24.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9eac881640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ea8134ad0 0x7f9ea8133c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45394/0 (socket says 172.21.15.12:45394) 2024-09-11T01:57:24.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9eac881640 1 -- 172.21.15.12:0/340814995 learned_addr learned my addr 172.21.15.12:0/340814995 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:24.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9eac881640 10 monclient(hunting): get_auth_request con 0x7f9ea8134ad0 auth_method 0 2024-09-11T01:57:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9eac881640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9eac881640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9eac881640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9ead082640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ea80a2200 0x7f9ea81cd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52876/0 (socket says 172.21.15.12:52876) 2024-09-11T01:57:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9ead082640 10 monclient(hunting): get_auth_request con 0x7f9ea80a2200 auth_method 0 2024-09-11T01:57:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9ead082640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9ead082640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9ead082640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9eac881640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9eac881640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9eac881640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9ead082640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9ead082640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:24.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9ead082640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:24.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9ead082640 10 monclient(hunting): handle_auth_done global_id 304223 payload 824 2024-09-11T01:57:24.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9ead082640 1 -- 172.21.15.12:0/340814995 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ea81355c0 msgr2=0x7f9ea8134190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:24.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9ead082640 1 --2- 172.21.15.12:0/340814995 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ea81355c0 0x7f9ea8134190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9ead082640 1 -- 172.21.15.12:0/340814995 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ea8134ad0 msgr2=0x7f9ea8133c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:24.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9ead082640 1 --2- 172.21.15.12:0/340814995 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ea8134ad0 0x7f9ea8133c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9ead082640 10 monclient: _finish_hunting 0 2024-09-11T01:57:24.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9ead082640 1 monclient: found mon.noname-c 2024-09-11T01:57:24.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9ead082640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:24.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9ead082640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:57:24.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9ead082640 1 -- 172.21.15.12:0/340814995 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ea8127e00 con 0x7f9ea80a2200 2024-09-11T01:57:24.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.558+0000 7f9eac881640 1 --2- 172.21.15.12:0/340814995 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ea8134ad0 0x7f9ea8133c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:57:24.572 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.560+0000 7f9ead082640 1 -- 172.21.15.12:0/340814995 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ea80a2200 msgr2=0x7f9ea81cd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:24.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.560+0000 7f9ead082640 1 --2- 172.21.15.12:0/340814995 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ea80a2200 0x7f9ea81cd260 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9ea400c390 tx=0x7f9ea400c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:24.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.560+0000 7f9e9dffb640 1 -- 172.21.15.12:0/340814995 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4017070 con 0x7f9ea80a2200 2024-09-11T01:57:24.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.560+0000 7f9e9dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:24.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.560+0000 7f9e9dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:57:24.573 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.560+0000 7f9e9dffb640 10 monclient: dump: 2024-09-11T01:57:24.573 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:24.573 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:24.573 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:24.573 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:24.573 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:24.573 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:24.573 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:24.573 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:24.573 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:24.574 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:24.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.560+0000 7f9e9dffb640 10 monclient: _finish_auth 0 2024-09-11T01:57:24.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.560+0000 7f9e9dffb640 10 monclient: _check_auth_tickets 2024-09-11T01:57:24.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.560+0000 7f9e9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:24.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.560+0000 7f9e9dffb640 1 -- 172.21.15.12:0/340814995 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ea40070d0 con 0x7f9ea80a2200 2024-09-11T01:57:24.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.560+0000 7f9eaf2e6640 5 monclient: authenticate success, global_id 304223 2024-09-11T01:57:24.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.560+0000 7f9eaf2e6640 10 monclient: _renew_subs 2024-09-11T01:57:24.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.560+0000 7f9eaf2e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:24.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.560+0000 7f9eaf2e6640 1 -- 172.21.15.12:0/340814995 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ea8126340 con 0x7f9ea80a2200 2024-09-11T01:57:24.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.560+0000 7f9e9dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:24.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.560+0000 7f9e9dffb640 1 -- 172.21.15.12:0/340814995 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea40074c0 con 0x7f9ea80a2200 2024-09-11T01:57:24.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.560+0000 7f9e9dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:24.574 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.560+0000 7f9e9dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:57:24.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.560+0000 7f9e9dffb640 10 monclient: dump: 2024-09-11T01:57:24.575 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:24.575 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:24.575 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:24.575 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:24.575 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:24.575 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:24.575 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:24.575 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:24.575 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:24.575 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:24.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.560+0000 7f9e9f7fe640 4 set_mon_vals no callback set 2024-09-11T01:57:24.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.561+0000 7f9eaf2e6640 10 monclient: _renew_subs 2024-09-11T01:57:24.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.561+0000 7f9eaf2e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:24.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.561+0000 7f9eaf2e6640 1 -- 172.21.15.12:0/340814995 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ea81286a0 con 0x7f9ea80a2200 2024-09-11T01:57:24.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.561+0000 7f9eaf2e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:24.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.561+0000 7f9eaf2e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:24.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.561+0000 7f9eaf2e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:24.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.561+0000 7f9e9dffb640 1 -- 172.21.15.12:0/340814995 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f9ea401b3b0 con 0x7f9ea80a2200 2024-09-11T01:57:24.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.561+0000 7f9eaf2e6640 1 -- 172.21.15.12:0/340814995 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ea8134eb0 con 0x7f9ea80a2200 2024-09-11T01:57:24.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.562+0000 7f9e9dffb640 1 --2- 172.21.15.12:0/340814995 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9e8c03e210 0x7f9e8c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:24.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.562+0000 7f9e9dffb640 1 -- 172.21.15.12:0/340814995 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(923..923 src has 889..923) ==== 4422+0+0 (secure 0 0 0) 0x7f9ea40522f0 con 0x7f9ea80a2200 2024-09-11T01:57:24.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.562+0000 7f9eac881640 1 --2- 172.21.15.12:0/340814995 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9e8c03e210 0x7f9e8c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:24.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.562+0000 7f9eac881640 10 monclient: get_auth_request con 0x7f9e8c03e210 auth_method 0 2024-09-11T01:57:24.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.565+0000 7f9eac881640 1 --2- 172.21.15.12:0/340814995 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9e8c03e210 0x7f9e8c0406d0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8127120 tx=0x7f9e9802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:24.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.566+0000 7f9e9dffb640 1 -- 172.21.15.12:0/340814995 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9ea8127e00 con 0x7f9ea80a2200 2024-09-11T01:57:24.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.566+0000 7f9e9dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:24.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.566+0000 7f9e9dffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:57:24.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.747+0000 7f9eaf2e6640 1 -- 172.21.15.12:0/340814995 --> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9ea80a4960 con 0x7f9e8c03e210 2024-09-11T01:57:24.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.748+0000 7f9e9dffb640 1 -- 172.21.15.12:0/340814995 <== mgr.4101 v2:172.21.15.71:6816/3189259519 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34024 (secure 0 0 0) 0x7f9ea80a4960 con 0x7f9e8c03e210 2024-09-11T01:57:24.750 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:57:24.751 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-09-11T01:57:24.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.751+0000 7f9eaf2e6640 1 -- 172.21.15.12:0/340814995 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9e8c03e210 msgr2=0x7f9e8c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:24.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.751+0000 7f9eaf2e6640 1 --2- 172.21.15.12:0/340814995 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9e8c03e210 0x7f9e8c0406d0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8127120 tx=0x7f9e9802e000 comp rx=0 tx=0).stop 2024-09-11T01:57:24.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.751+0000 7f9eaf2e6640 10 monclient: shutdown 2024-09-11T01:57:24.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.751+0000 7f9eaf2e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:24.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.751+0000 7f9eaf2e6640 1 -- 172.21.15.12:0/340814995 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ea80a2200 msgr2=0x7f9ea81cd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:24.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.751+0000 7f9eaf2e6640 1 --2- 172.21.15.12:0/340814995 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ea80a2200 0x7f9ea81cd260 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9ea400c390 tx=0x7f9ea400c850 comp rx=0 tx=0).stop 2024-09-11T01:57:24.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.751+0000 7f9eaf2e6640 1 -- 172.21.15.12:0/340814995 shutdown_connections 2024-09-11T01:57:24.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.751+0000 7f9eaf2e6640 1 --2- 172.21.15.12:0/340814995 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ea81355c0 0x7f9ea8134190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.751+0000 7f9eaf2e6640 1 --2- 172.21.15.12:0/340814995 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9e8c03e210 0x7f9e8c0406d0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.751+0000 7f9eaf2e6640 1 --2- 172.21.15.12:0/340814995 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ea8134ad0 0x7f9ea8133c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.751+0000 7f9eaf2e6640 1 --2- 172.21.15.12:0/340814995 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ea80a2200 0x7f9ea81cd260 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.751+0000 7f9eaf2e6640 1 -- 172.21.15.12:0/340814995 >> 172.21.15.12:0/340814995 conn(0x7f9ea80a2f70 msgr2=0x7f9ea809d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:24.753 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.751+0000 7f9eaf2e6640 1 -- 172.21.15.12:0/340814995 shutdown_connections 2024-09-11T01:57:24.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.751+0000 7f9eaf2e6640 1 -- 172.21.15.12:0/340814995 wait complete. 2024-09-11T01:57:24.763 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":1766,"stamp":"2024-09-11T01:57:23.208437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":333208,"kb_used_data":14816,"kb_used_omap":9,"kb_used_meta":317366,"kb_avail":565897832,"statfs":{"total":579820584960,"available":579479379968,"internally_reserved":0,"allocated":15171584,"data_stored":7598212,"data_compressed":59136,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":324983456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002313"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":1807,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:44.358489+0000","last_change":"2024-09-11T01:29:17.579548+0000","last_active":"2024-09-11T01:56:44.358489+0000","last_peered":"2024-09-11T01:56:44.358489+0000","last_clean":"2024-09-11T01:56:44.358489+0000","last_became_active":"2024-09-11T01:29:15.546952+0000","last_became_peered":"2024-09-11T01:29:15.546952+0000","last_unstale":"2024-09-11T01:56:44.358489+0000","last_undegraded":"2024-09-11T01:56:44.358489+0000","last_fullsized":"2024-09-11T01:56:44.358489+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:45:28.460820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038245800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":1812,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:43.746370+0000","last_change":"2024-09-11T01:29:17.597459+0000","last_active":"2024-09-11T01:56:43.746370+0000","last_peered":"2024-09-11T01:56:43.746370+0000","last_clean":"2024-09-11T01:56:43.746370+0000","last_became_active":"2024-09-11T01:29:15.549013+0000","last_became_peered":"2024-09-11T01:29:15.549013+0000","last_unstale":"2024-09-11T01:56:43.746370+0000","last_undegraded":"2024-09-11T01:56:43.746370+0000","last_fullsized":"2024-09-11T01:56:43.746370+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:46:02.130174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041667199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":1822,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:43.904328+0000","last_change":"2024-09-11T01:29:17.577463+0000","last_active":"2024-09-11T01:56:43.904328+0000","last_peered":"2024-09-11T01:56:43.904328+0000","last_clean":"2024-09-11T01:56:43.904328+0000","last_became_active":"2024-09-11T01:29:15.548884+0000","last_became_peered":"2024-09-11T01:29:15.548884+0000","last_unstale":"2024-09-11T01:56:43.904328+0000","last_undegraded":"2024-09-11T01:56:43.904328+0000","last_fullsized":"2024-09-11T01:56:43.904328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:09.120675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094832499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":1817,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:44.360535+0000","last_change":"2024-09-11T01:29:17.583117+0000","last_active":"2024-09-11T01:56:44.360535+0000","last_peered":"2024-09-11T01:56:44.360535+0000","last_clean":"2024-09-11T01:56:44.360535+0000","last_became_active":"2024-09-11T01:29:15.546142+0000","last_became_peered":"2024-09-11T01:29:15.546142+0000","last_unstale":"2024-09-11T01:56:44.360535+0000","last_undegraded":"2024-09-11T01:56:44.360535+0000","last_fullsized":"2024-09-11T01:56:44.360535+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:16:49.024029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044411199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"16'72","reported_seq":1890,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:44.358484+0000","last_change":"2024-09-11T01:29:16.691671+0000","last_active":"2024-09-11T01:56:44.358484+0000","last_peered":"2024-09-11T01:56:44.358484+0000","last_clean":"2024-09-11T01:56:44.358484+0000","last_became_active":"2024-09-11T01:29:16.690647+0000","last_became_peered":"2024-09-11T01:29:16.690647+0000","last_unstale":"2024-09-11T01:56:44.358484+0000","last_undegraded":"2024-09-11T01:56:44.358484+0000","last_fullsized":"2024-09-11T01:56:44.358484+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:15.533298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:15.533298+0000","last_clean_scrub_stamp":"2024-09-11T01:29:15.533298+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:41:07.416896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":1816,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:44.360506+0000","last_change":"2024-09-11T01:29:17.583012+0000","last_active":"2024-09-11T01:56:44.360506+0000","last_peered":"2024-09-11T01:56:44.360506+0000","last_clean":"2024-09-11T01:56:44.360506+0000","last_became_active":"2024-09-11T01:29:15.547447+0000","last_became_peered":"2024-09-11T01:29:15.547447+0000","last_unstale":"2024-09-11T01:56:44.360506+0000","last_undegraded":"2024-09-11T01:56:44.360506+0000","last_fullsized":"2024-09-11T01:56:44.360506+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:17:48.616008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037448800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":1820,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:43.904334+0000","last_change":"2024-09-11T01:29:17.576853+0000","last_active":"2024-09-11T01:56:43.904334+0000","last_peered":"2024-09-11T01:56:43.904334+0000","last_clean":"2024-09-11T01:56:43.904334+0000","last_became_active":"2024-09-11T01:29:15.546476+0000","last_became_peered":"2024-09-11T01:29:15.546476+0000","last_unstale":"2024-09-11T01:56:43.904334+0000","last_undegraded":"2024-09-11T01:56:43.904334+0000","last_fullsized":"2024-09-11T01:56:43.904334+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:11:54.706642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037031899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":1806,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:44.358504+0000","last_change":"2024-09-11T01:29:17.579590+0000","last_active":"2024-09-11T01:56:44.358504+0000","last_peered":"2024-09-11T01:56:44.358504+0000","last_clean":"2024-09-11T01:56:44.358504+0000","last_became_active":"2024-09-11T01:29:15.542800+0000","last_became_peered":"2024-09-11T01:29:15.542800+0000","last_unstale":"2024-09-11T01:56:44.358504+0000","last_undegraded":"2024-09-11T01:56:44.358504+0000","last_fullsized":"2024-09-11T01:56:44.358504+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:23:55.773396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040701099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":1811,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:43.746418+0000","last_change":"2024-09-11T01:29:17.597492+0000","last_active":"2024-09-11T01:56:43.746418+0000","last_peered":"2024-09-11T01:56:43.746418+0000","last_clean":"2024-09-11T01:56:43.746418+0000","last_became_active":"2024-09-11T01:29:15.546444+0000","last_became_peered":"2024-09-11T01:29:15.546444+0000","last_unstale":"2024-09-11T01:56:43.746418+0000","last_undegraded":"2024-09-11T01:56:43.746418+0000","last_fullsized":"2024-09-11T01:56:43.746418+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:52:43.624650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000396053,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59136,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607894,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47704,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324136,"statfs":{"total":96636764160,"available":96587915264,"internally_reserved":0,"allocated":2506752,"data_stored":1090053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607892,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56028,"kb_used_data":2452,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315812,"statfs":{"total":96636764160,"available":96579391488,"internally_reserved":0,"allocated":2510848,"data_stored":1090072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607893,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56084,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315756,"statfs":{"total":96636764160,"available":96579334144,"internally_reserved":0,"allocated":2568192,"data_stored":1618981,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607893,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69724,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302116,"statfs":{"total":96636764160,"available":96565366784,"internally_reserved":0,"allocated":2568192,"data_stored":1618981,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607893,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33820,"kb_used_data":2452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338020,"statfs":{"total":96636764160,"available":96602132480,"internally_reserved":0,"allocated":2510848,"data_stored":1090072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640600,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69848,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94301992,"statfs":{"total":96636764160,"available":96565239808,"internally_reserved":0,"allocated":2506752,"data_stored":1090053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T01:57:24.763 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T01:57:24.764 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T01:57:24.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.863+0000 7f83bbc2a640 10 monclient: get_monmap_and_config 2024-09-11T01:57:24.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.863+0000 7f83bbc2a640 10 monclient: build_initial_monmap 2024-09-11T01:57:24.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.863+0000 7f83bbc2a640 10 monclient: monmap: 2024-09-11T01:57:24.865 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:24.865 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:24.865 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:57:24.864307+0000 2024-09-11T01:57:24.865 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:57:24.864307+0000 2024-09-11T01:57:24.865 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:24.865 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:24.865 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:57:24.865 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:57:24.866 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:57:24.866 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:57:24.866 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:57:24.866 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:24.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.864+0000 7f83bbc2a640 1 Processor -- start 2024-09-11T01:57:24.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.864+0000 7f83bbc2a640 1 -- start start 2024-09-11T01:57:24.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.864+0000 7f83bbc2a640 10 monclient: init 2024-09-11T01:57:24.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.864+0000 7f83bbc2a640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:24.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.864+0000 7f83bbc2a640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-11T01:57:24.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.864+0000 7f83bbc2a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b4074180 0x7f83b4074560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:24.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.864+0000 7f83bbc2a640 10 monclient(hunting): picked mon.noname-b con 0x7f83b4074180 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:24.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.864+0000 7f83bbc2a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f83b40a06d0 addr v1:172.21.15.71:6789/0 2024-09-11T01:57:24.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.864+0000 7f83bbc2a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f83b4134ac0 addr v1:172.21.15.12:6789/0 2024-09-11T01:57:24.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.864+0000 7f83bbc2a640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83b4074aa0 con 0x7f83b4134ac0 2024-09-11T01:57:24.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.864+0000 7f83bbc2a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83b40a33f0 con 0x7f83b40a06d0 2024-09-11T01:57:24.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.864+0000 7f83bbc2a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:24.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.864+0000 7f83bbc2a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f83b4135560 con 0x7f83b4074180 2024-09-11T01:57:24.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.864+0000 7f83bbc2a640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:24.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.864+0000 7f83bbc2a640 10 monclient(hunting): authenticate will time out at 2451.306913s 2024-09-11T01:57:24.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.865+0000 7f83b99c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b4074180 0x7f83b4074560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:24.869 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.865+0000 7f83ba1c7640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f83b4134ac0 0x7f83b40a2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:39182/0 (socket says 172.21.15.12:39182) 2024-09-11T01:57:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.865+0000 7f83ba1c7640 1 -- 172.21.15.12:0/1059473256 learned_addr learned my addr 172.21.15.12:0/1059473256 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:57:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.865+0000 7f83b99c6640 10 monclient(hunting): get_auth_request con 0x7f83b4074180 auth_method 0 2024-09-11T01:57:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.865+0000 7f83b99c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.865+0000 7f83b99c6640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.865+0000 7f83b99c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.865+0000 7f83b99c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.865+0000 7f83b99c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.865+0000 7f83b99c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.865+0000 7f83b99c6640 10 monclient(hunting): handle_auth_done global_id 304226 payload 274 2024-09-11T01:57:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.865+0000 7f83b99c6640 1 -- 172.21.15.12:0/1059473256 >> v1:172.21.15.71:6789/0 conn(0x7f83b40a06d0 legacy=0x7f83b40a25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.865+0000 7f83b99c6640 1 -- 172.21.15.12:0/1059473256 >> v1:172.21.15.12:6789/0 conn(0x7f83b4134ac0 legacy=0x7f83b40a2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.865+0000 7f83b99c6640 10 monclient: _finish_hunting 0 2024-09-11T01:57:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.865+0000 7f83b99c6640 1 monclient: found mon.noname-b 2024-09-11T01:57:24.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.865+0000 7f83b99c6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.865+0000 7f83b99c6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:57:24.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.865+0000 7f83b99c6640 1 -- 172.21.15.12:0/1059473256 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83b4070160 con 0x7f83b4074180 2024-09-11T01:57:24.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83b99c6640 1 -- 172.21.15.12:0/1059473256 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b4074180 msgr2=0x7f83b4074560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:24.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83b99c6640 1 --2- 172.21.15.12:0/1059473256 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b4074180 0x7f83b4074560 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f83b00097d0 tx=0x7f83b0029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0fceec6374fadae server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:24.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83b89c4640 1 -- 172.21.15.12:0/1059473256 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b0037070 con 0x7f83b4074180 2024-09-11T01:57:24.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83b89c4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:24.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83b89c4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:57:24.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83b89c4640 10 monclient: dump: 2024-09-11T01:57:24.876 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:24.877 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:24.877 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:24.877 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:24.877 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:24.877 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:24.877 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:24.877 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:24.877 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:24.877 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:24.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83b89c4640 10 monclient: _finish_auth 0 2024-09-11T01:57:24.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83b89c4640 10 monclient: _check_auth_tickets 2024-09-11T01:57:24.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83b89c4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:24.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83bbc2a640 5 monclient: authenticate success, global_id 304226 2024-09-11T01:57:24.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83bbc2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:57:24.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83b89c4640 1 -- 172.21.15.12:0/1059473256 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83b0032440 con 0x7f83b4074180 2024-09-11T01:57:24.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83b89c4640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:24.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83bbc2a640 10 monclient: get_monmap_and_config success 2024-09-11T01:57:24.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83bbc2a640 4 set_mon_vals no callback set 2024-09-11T01:57:24.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83b89c4640 1 -- 172.21.15.12:0/1059473256 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b0032810 con 0x7f83b4074180 2024-09-11T01:57:24.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83b89c4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:24.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83b89c4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:57:24.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83b89c4640 10 monclient: dump: 2024-09-11T01:57:24.878 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:24.878 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:24.878 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:24.878 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:24.878 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:24.879 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:24.879 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:24.879 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:24.879 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:24.879 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:24.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83bbc2a640 10 monclient: shutdown 2024-09-11T01:57:24.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83bbc2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:24.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83bbc2a640 1 -- 172.21.15.12:0/1059473256 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b4074180 msgr2=0x7f83b4074560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:24.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83bbc2a640 1 --2- 172.21.15.12:0/1059473256 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b4074180 0x7f83b4074560 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f83b00097d0 tx=0x7f83b0029fb0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83bbc2a640 1 -- 172.21.15.12:0/1059473256 shutdown_connections 2024-09-11T01:57:24.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83bbc2a640 1 --2- 172.21.15.12:0/1059473256 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b4074180 0x7f83b4074560 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83bbc2a640 1 -- 172.21.15.12:0/1059473256 >> 172.21.15.12:0/1059473256 conn(0x7f83b40a36c0 msgr2=0x7f83b40a3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:24.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83bbc2a640 1 -- 172.21.15.12:0/1059473256 shutdown_connections 2024-09-11T01:57:24.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.868+0000 7f83bbc2a640 1 -- 172.21.15.12:0/1059473256 wait complete. 2024-09-11T01:57:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.869+0000 7f83bbc2a640 10 monclient: build_initial_monmap 2024-09-11T01:57:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.869+0000 7f83bbc2a640 10 monclient: monmap: 2024-09-11T01:57:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:57:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:57:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:57:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:57:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:57:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:57:24.880 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.869+0000 7f83bbc2a640 1 Processor -- start 2024-09-11T01:57:24.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.869+0000 7f83bbc2a640 1 -- start start 2024-09-11T01:57:24.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.869+0000 7f83bbc2a640 10 monclient: init 2024-09-11T01:57:24.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83bbc2a640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:24.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83bbc2a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:57:24.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83bbc2a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b4074180 0x7f83b413fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:24.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83bbc2a640 10 monclient(hunting): picked mon.noname-c con 0x7f83b4074180 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:24.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83bbc2a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83b40a06d0 0x7f83b4140020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:24.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83bbc2a640 10 monclient(hunting): picked mon.noname-a con 0x7f83b40a06d0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:57:24.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83bbc2a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83b4134ac0 0x7f83b4136790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:24.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83bbc2a640 10 monclient(hunting): picked mon.noname-b con 0x7f83b4134ac0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:57:24.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83bbc2a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:24.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83bbc2a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f83b4068d40 con 0x7f83b40a06d0 2024-09-11T01:57:24.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83bbc2a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:24.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83bbc2a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f83b4068bc0 con 0x7f83b4134ac0 2024-09-11T01:57:24.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83bbc2a640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:24.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83bbc2a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f83b4140ef0 con 0x7f83b4074180 2024-09-11T01:57:24.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83bbc2a640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:24.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83bbc2a640 10 monclient(hunting): authenticate will time out at 2451.312353s 2024-09-11T01:57:24.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b99c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b4074180 0x7f83b413fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:24.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b99c6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b4074180 0x7f83b413fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52892/0 (socket says 172.21.15.12:52892) 2024-09-11T01:57:24.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b99c6640 1 -- 172.21.15.12:0/1686251100 learned_addr learned my addr 172.21.15.12:0/1686251100 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:24.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b99c6640 10 monclient(hunting): get_auth_request con 0x7f83b4074180 auth_method 0 2024-09-11T01:57:24.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b99c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:24.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b99c6640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:24.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b99c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:24.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b91c5640 1 --2- 172.21.15.12:0/1686251100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83b40a06d0 0x7f83b4140020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:24.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b91c5640 10 monclient(hunting): get_auth_request con 0x7f83b40a06d0 auth_method 0 2024-09-11T01:57:24.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b91c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b91c5640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b91c5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b99c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b99c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b99c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b91c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b91c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b91c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83ba1c7640 1 --2- 172.21.15.12:0/1686251100 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83b4134ac0 0x7f83b4136790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83ba1c7640 10 monclient(hunting): get_auth_request con 0x7f83b4134ac0 auth_method 0 2024-09-11T01:57:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83ba1c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83ba1c7640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83ba1c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:24.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b99c6640 10 monclient(hunting): handle_auth_done global_id 304229 payload 824 2024-09-11T01:57:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b99c6640 1 -- 172.21.15.12:0/1686251100 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83b4134ac0 msgr2=0x7f83b4136790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b99c6640 1 --2- 172.21.15.12:0/1686251100 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83b4134ac0 0x7f83b4136790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b99c6640 1 -- 172.21.15.12:0/1686251100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83b40a06d0 msgr2=0x7f83b4140020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b99c6640 1 --2- 172.21.15.12:0/1686251100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83b40a06d0 0x7f83b4140020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b99c6640 10 monclient: _finish_hunting 0 2024-09-11T01:57:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b99c6640 1 monclient: found mon.noname-c 2024-09-11T01:57:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b99c6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b99c6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:57:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.870+0000 7f83b99c6640 1 -- 172.21.15.12:0/1686251100 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83b41ce040 con 0x7f83b4074180 2024-09-11T01:57:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.872+0000 7f83b99c6640 1 -- 172.21.15.12:0/1686251100 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b4074180 msgr2=0x7f83b413fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.872+0000 7f83b99c6640 1 --2- 172.21.15.12:0/1686251100 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b4074180 0x7f83b413fae0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f83b00098f0 tx=0x7f83b00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.873+0000 7f83aa7fc640 1 -- 172.21.15.12:0/1686251100 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b0042020 con 0x7f83b4074180 2024-09-11T01:57:24.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.873+0000 7f83aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.873+0000 7f83aa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:57:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.873+0000 7f83aa7fc640 10 monclient: dump: 2024-09-11T01:57:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:24.885 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.873+0000 7f83aa7fc640 10 monclient: _finish_auth 0 2024-09-11T01:57:24.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.873+0000 7f83aa7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:57:24.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.873+0000 7f83aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:24.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.873+0000 7f83aa7fc640 1 -- 172.21.15.12:0/1686251100 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83b003d670 con 0x7f83b4074180 2024-09-11T01:57:24.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.873+0000 7f83bbc2a640 5 monclient: authenticate success, global_id 304229 2024-09-11T01:57:24.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.873+0000 7f83bbc2a640 10 monclient: _renew_subs 2024-09-11T01:57:24.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.873+0000 7f83bbc2a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:24.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.873+0000 7f83bbc2a640 1 -- 172.21.15.12:0/1686251100 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83b4140b40 con 0x7f83b4074180 2024-09-11T01:57:24.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.873+0000 7f83aa7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:24.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.873+0000 7f83aa7fc640 1 -- 172.21.15.12:0/1686251100 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b0037040 con 0x7f83b4074180 2024-09-11T01:57:24.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.873+0000 7f83aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:24.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.873+0000 7f83aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:57:24.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.873+0000 7f83aa7fc640 10 monclient: dump: 2024-09-11T01:57:24.886 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:24.886 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:24.886 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:24.887 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.873+0000 7f83abfff640 4 set_mon_vals no callback set 2024-09-11T01:57:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.873+0000 7f83bbc2a640 10 monclient: _renew_subs 2024-09-11T01:57:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.873+0000 7f83bbc2a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.873+0000 7f83bbc2a640 1 -- 172.21.15.12:0/1686251100 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f83b413a570 con 0x7f83b4074180 2024-09-11T01:57:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.874+0000 7f83aa7fc640 1 -- 172.21.15.12:0/1686251100 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f83b003da60 con 0x7f83b4074180 2024-09-11T01:57:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.874+0000 7f83bbc2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:24.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.874+0000 7f83bbc2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.874+0000 7f83bbc2a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.874+0000 7f83bbc2a640 1 -- 172.21.15.12:0/1686251100 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83b4077150 con 0x7f83b4074180 2024-09-11T01:57:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.874+0000 7f83aa7fc640 1 --2- 172.21.15.12:0/1686251100 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f839803e210 0x7f83980406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.874+0000 7f83aa7fc640 1 -- 172.21.15.12:0/1686251100 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(923..923 src has 889..923) ==== 4422+0+0 (secure 0 0 0) 0x7f83b00736b0 con 0x7f83b4074180 2024-09-11T01:57:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.874+0000 7f83b91c5640 1 --2- 172.21.15.12:0/1686251100 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f839803e210 0x7f83980406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.874+0000 7f83b91c5640 10 monclient: get_auth_request con 0x7f839803e210 auth_method 0 2024-09-11T01:57:24.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.877+0000 7f83b91c5640 1 --2- 172.21.15.12:0/1686251100 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f839803e210 0x7f83980406d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f83b40a5e40 tx=0x7f83a4007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.878+0000 7f83aa7fc640 1 -- 172.21.15.12:0/1686251100 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f83b4077150 con 0x7f83b4074180 2024-09-11T01:57:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.879+0000 7f83aa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:24.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:24.879+0000 7f83aa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:57:25.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.060+0000 7f83bbc2a640 1 -- 172.21.15.12:0/1686251100 --> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f83b4074560 con 0x7f839803e210 2024-09-11T01:57:25.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.061+0000 7f83aa7fc640 1 -- 172.21.15.12:0/1686251100 <== mgr.4101 v2:172.21.15.71:6816/3189259519 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34024 (secure 0 0 0) 0x7f83b4074560 con 0x7f839803e210 2024-09-11T01:57:25.063 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:57:25.064 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-09-11T01:57:25.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.063+0000 7f83bbc2a640 1 -- 172.21.15.12:0/1686251100 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f839803e210 msgr2=0x7f83980406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:25.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.063+0000 7f83bbc2a640 1 --2- 172.21.15.12:0/1686251100 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f839803e210 0x7f83980406d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f83b40a5e40 tx=0x7f83a4007e30 comp rx=0 tx=0).stop 2024-09-11T01:57:25.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.063+0000 7f83bbc2a640 10 monclient: shutdown 2024-09-11T01:57:25.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.063+0000 7f83bbc2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:25.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.063+0000 7f83bbc2a640 1 -- 172.21.15.12:0/1686251100 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b4074180 msgr2=0x7f83b413fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:25.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.063+0000 7f83bbc2a640 1 --2- 172.21.15.12:0/1686251100 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b4074180 0x7f83b413fae0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f83b00098f0 tx=0x7f83b00046c0 comp rx=0 tx=0).stop 2024-09-11T01:57:25.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.063+0000 7f83bbc2a640 1 -- 172.21.15.12:0/1686251100 shutdown_connections 2024-09-11T01:57:25.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.063+0000 7f83bbc2a640 1 --2- 172.21.15.12:0/1686251100 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83b4134ac0 0x7f83b4136790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:25.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.063+0000 7f83bbc2a640 1 --2- 172.21.15.12:0/1686251100 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f839803e210 0x7f83980406d0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:25.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.063+0000 7f83bbc2a640 1 --2- 172.21.15.12:0/1686251100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83b40a06d0 0x7f83b4140020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:25.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.063+0000 7f83bbc2a640 1 --2- 172.21.15.12:0/1686251100 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f83b4074180 0x7f83b413fae0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:25.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.063+0000 7f83bbc2a640 1 -- 172.21.15.12:0/1686251100 >> 172.21.15.12:0/1686251100 conn(0x7f83b40a36c0 msgr2=0x7f83b4132a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:25.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.063+0000 7f83bbc2a640 1 -- 172.21.15.12:0/1686251100 shutdown_connections 2024-09-11T01:57:25.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.063+0000 7f83bbc2a640 1 -- 172.21.15.12:0/1686251100 wait complete. 2024-09-11T01:57:25.075 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":1766,"stamp":"2024-09-11T01:57:23.208437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":333208,"kb_used_data":14816,"kb_used_omap":9,"kb_used_meta":317366,"kb_avail":565897832,"statfs":{"total":579820584960,"available":579479379968,"internally_reserved":0,"allocated":15171584,"data_stored":7598212,"data_compressed":59136,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":324983456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002313"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":1807,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:44.358489+0000","last_change":"2024-09-11T01:29:17.579548+0000","last_active":"2024-09-11T01:56:44.358489+0000","last_peered":"2024-09-11T01:56:44.358489+0000","last_clean":"2024-09-11T01:56:44.358489+0000","last_became_active":"2024-09-11T01:29:15.546952+0000","last_became_peered":"2024-09-11T01:29:15.546952+0000","last_unstale":"2024-09-11T01:56:44.358489+0000","last_undegraded":"2024-09-11T01:56:44.358489+0000","last_fullsized":"2024-09-11T01:56:44.358489+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:45:28.460820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038245800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":1812,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:43.746370+0000","last_change":"2024-09-11T01:29:17.597459+0000","last_active":"2024-09-11T01:56:43.746370+0000","last_peered":"2024-09-11T01:56:43.746370+0000","last_clean":"2024-09-11T01:56:43.746370+0000","last_became_active":"2024-09-11T01:29:15.549013+0000","last_became_peered":"2024-09-11T01:29:15.549013+0000","last_unstale":"2024-09-11T01:56:43.746370+0000","last_undegraded":"2024-09-11T01:56:43.746370+0000","last_fullsized":"2024-09-11T01:56:43.746370+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:46:02.130174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041667199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":1822,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:43.904328+0000","last_change":"2024-09-11T01:29:17.577463+0000","last_active":"2024-09-11T01:56:43.904328+0000","last_peered":"2024-09-11T01:56:43.904328+0000","last_clean":"2024-09-11T01:56:43.904328+0000","last_became_active":"2024-09-11T01:29:15.548884+0000","last_became_peered":"2024-09-11T01:29:15.548884+0000","last_unstale":"2024-09-11T01:56:43.904328+0000","last_undegraded":"2024-09-11T01:56:43.904328+0000","last_fullsized":"2024-09-11T01:56:43.904328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:09.120675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094832499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":1817,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:44.360535+0000","last_change":"2024-09-11T01:29:17.583117+0000","last_active":"2024-09-11T01:56:44.360535+0000","last_peered":"2024-09-11T01:56:44.360535+0000","last_clean":"2024-09-11T01:56:44.360535+0000","last_became_active":"2024-09-11T01:29:15.546142+0000","last_became_peered":"2024-09-11T01:29:15.546142+0000","last_unstale":"2024-09-11T01:56:44.360535+0000","last_undegraded":"2024-09-11T01:56:44.360535+0000","last_fullsized":"2024-09-11T01:56:44.360535+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:16:49.024029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044411199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"16'72","reported_seq":1890,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:44.358484+0000","last_change":"2024-09-11T01:29:16.691671+0000","last_active":"2024-09-11T01:56:44.358484+0000","last_peered":"2024-09-11T01:56:44.358484+0000","last_clean":"2024-09-11T01:56:44.358484+0000","last_became_active":"2024-09-11T01:29:16.690647+0000","last_became_peered":"2024-09-11T01:29:16.690647+0000","last_unstale":"2024-09-11T01:56:44.358484+0000","last_undegraded":"2024-09-11T01:56:44.358484+0000","last_fullsized":"2024-09-11T01:56:44.358484+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:15.533298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:15.533298+0000","last_clean_scrub_stamp":"2024-09-11T01:29:15.533298+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:41:07.416896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":1816,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:44.360506+0000","last_change":"2024-09-11T01:29:17.583012+0000","last_active":"2024-09-11T01:56:44.360506+0000","last_peered":"2024-09-11T01:56:44.360506+0000","last_clean":"2024-09-11T01:56:44.360506+0000","last_became_active":"2024-09-11T01:29:15.547447+0000","last_became_peered":"2024-09-11T01:29:15.547447+0000","last_unstale":"2024-09-11T01:56:44.360506+0000","last_undegraded":"2024-09-11T01:56:44.360506+0000","last_fullsized":"2024-09-11T01:56:44.360506+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:17:48.616008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037448800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":1820,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:43.904334+0000","last_change":"2024-09-11T01:29:17.576853+0000","last_active":"2024-09-11T01:56:43.904334+0000","last_peered":"2024-09-11T01:56:43.904334+0000","last_clean":"2024-09-11T01:56:43.904334+0000","last_became_active":"2024-09-11T01:29:15.546476+0000","last_became_peered":"2024-09-11T01:29:15.546476+0000","last_unstale":"2024-09-11T01:56:43.904334+0000","last_undegraded":"2024-09-11T01:56:43.904334+0000","last_fullsized":"2024-09-11T01:56:43.904334+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:11:54.706642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037031899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":1806,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:44.358504+0000","last_change":"2024-09-11T01:29:17.579590+0000","last_active":"2024-09-11T01:56:44.358504+0000","last_peered":"2024-09-11T01:56:44.358504+0000","last_clean":"2024-09-11T01:56:44.358504+0000","last_became_active":"2024-09-11T01:29:15.542800+0000","last_became_peered":"2024-09-11T01:29:15.542800+0000","last_unstale":"2024-09-11T01:56:44.358504+0000","last_undegraded":"2024-09-11T01:56:44.358504+0000","last_fullsized":"2024-09-11T01:56:44.358504+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:23:55.773396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040701099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":1811,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:43.746418+0000","last_change":"2024-09-11T01:29:17.597492+0000","last_active":"2024-09-11T01:56:43.746418+0000","last_peered":"2024-09-11T01:56:43.746418+0000","last_clean":"2024-09-11T01:56:43.746418+0000","last_became_active":"2024-09-11T01:29:15.546444+0000","last_became_peered":"2024-09-11T01:29:15.546444+0000","last_unstale":"2024-09-11T01:56:43.746418+0000","last_undegraded":"2024-09-11T01:56:43.746418+0000","last_fullsized":"2024-09-11T01:56:43.746418+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:52:43.624650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000396053,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59136,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607894,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47704,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324136,"statfs":{"total":96636764160,"available":96587915264,"internally_reserved":0,"allocated":2506752,"data_stored":1090053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607892,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56028,"kb_used_data":2452,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315812,"statfs":{"total":96636764160,"available":96579391488,"internally_reserved":0,"allocated":2510848,"data_stored":1090072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607893,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56084,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315756,"statfs":{"total":96636764160,"available":96579334144,"internally_reserved":0,"allocated":2568192,"data_stored":1618981,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607893,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69724,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302116,"statfs":{"total":96636764160,"available":96565366784,"internally_reserved":0,"allocated":2568192,"data_stored":1618981,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607893,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33820,"kb_used_data":2452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338020,"statfs":{"total":96636764160,"available":96602132480,"internally_reserved":0,"allocated":2510848,"data_stored":1090072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640600,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69848,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94301992,"statfs":{"total":96636764160,"available":96565239808,"internally_reserved":0,"allocated":2506752,"data_stored":1090053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T01:57:25.076 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T01:57:25.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.175+0000 7f4fffc52640 10 monclient: get_monmap_and_config 2024-09-11T01:57:25.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.175+0000 7f4fffc52640 10 monclient: build_initial_monmap 2024-09-11T01:57:25.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.175+0000 7f4fffc52640 10 monclient: monmap: 2024-09-11T01:57:25.177 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:25.177 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:25.178 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:57:25.176293+0000 2024-09-11T01:57:25.178 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:57:25.176293+0000 2024-09-11T01:57:25.178 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:25.178 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:25.178 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:57:25.178 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:57:25.178 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:57:25.178 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:57:25.178 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:57:25.178 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:25.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.176+0000 7f4fffc52640 1 Processor -- start 2024-09-11T01:57:25.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.176+0000 7f4fffc52640 1 -- start start 2024-09-11T01:57:25.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.176+0000 7f4fffc52640 10 monclient: init 2024-09-11T01:57:25.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.176+0000 7f4fffc52640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:25.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.176+0000 7f4fffc52640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-11T01:57:25.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.176+0000 7f4fffc52640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4ff80a1380 addr v1:172.21.15.71:6789/0 2024-09-11T01:57:25.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.176+0000 7f4fffc52640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4ff80a0370 addr v1:172.21.15.12:6789/0 2024-09-11T01:57:25.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.176+0000 7f4fffc52640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ff80a0c90 0x7f4ff813e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:25.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.176+0000 7f4fffc52640 10 monclient(hunting): picked mon.noname-b con 0x7f4ff80a0c90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:25.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.176+0000 7f4fffc52640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ff80a21c0 con 0x7f4ff80a0370 2024-09-11T01:57:25.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.176+0000 7f4fffc52640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ff80a24a0 con 0x7f4ff80a1380 2024-09-11T01:57:25.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.176+0000 7f4fffc52640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:25.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.176+0000 7f4fffc52640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4ff80a2780 con 0x7f4ff80a0c90 2024-09-11T01:57:25.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.176+0000 7f4fffc52640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:25.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.176+0000 7f4fffc52640 10 monclient(hunting): authenticate will time out at 2451.618944s 2024-09-11T01:57:25.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.177+0000 7f4ffe1ef640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ff80a0c90 0x7f4ff813e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:25.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.177+0000 7f4ffd1ed640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4ff80a0370 0x7f4ff813ae00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:39198/0 (socket says 172.21.15.12:39198) 2024-09-11T01:57:25.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.177+0000 7f4ffd1ed640 1 -- 172.21.15.12:0/2210347941 learned_addr learned my addr 172.21.15.12:0/2210347941 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:57:25.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.177+0000 7f4ffe1ef640 1 --2- 172.21.15.12:0/2210347941 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ff80a0c90 0x7f4ff813e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52904/0 (socket says 172.21.15.12:52904) 2024-09-11T01:57:25.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.177+0000 7f4ffe1ef640 10 monclient(hunting): get_auth_request con 0x7f4ff80a0c90 auth_method 0 2024-09-11T01:57:25.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.177+0000 7f4ffe1ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:25.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.177+0000 7f4ffe1ef640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:25.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.177+0000 7f4ffe1ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:25.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.177+0000 7f4ffe1ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:25.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.177+0000 7f4ffe1ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:25.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.177+0000 7f4ffe1ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:25.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.177+0000 7f4ffe1ef640 10 monclient(hunting): handle_auth_done global_id 304232 payload 274 2024-09-11T01:57:25.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.177+0000 7f4ffe1ef640 1 -- 172.21.15.12:0/2210347941 >> v1:172.21.15.71:6789/0 conn(0x7f4ff80a1380 legacy=0x7f4ff80a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:25.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.177+0000 7f4ffe1ef640 1 -- 172.21.15.12:0/2210347941 >> v1:172.21.15.12:6789/0 conn(0x7f4ff80a0370 legacy=0x7f4ff813ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:25.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.177+0000 7f4ffe1ef640 10 monclient: _finish_hunting 0 2024-09-11T01:57:25.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.177+0000 7f4ffe1ef640 1 monclient: found mon.noname-b 2024-09-11T01:57:25.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.177+0000 7f4ffe1ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:25.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.177+0000 7f4ffe1ef640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:57:25.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.177+0000 7f4ffe1ef640 1 -- 172.21.15.12:0/2210347941 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ff809fb60 con 0x7f4ff80a0c90 2024-09-11T01:57:25.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.180+0000 7f4ffe1ef640 1 -- 172.21.15.12:0/2210347941 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ff80a0c90 msgr2=0x7f4ff813e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:25.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.180+0000 7f4ffe1ef640 1 --2- 172.21.15.12:0/2210347941 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ff80a0c90 0x7f4ff813e530 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0009dc0 tx=0x7f4ff002a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92ab2521797b422c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:25.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.180+0000 7f4ffc9ec640 1 -- 172.21.15.12:0/2210347941 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0007a30 con 0x7f4ff80a0c90 2024-09-11T01:57:25.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.180+0000 7f4ffc9ec640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:25.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.180+0000 7f4ffc9ec640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:57:25.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.180+0000 7f4ffc9ec640 10 monclient: dump: 2024-09-11T01:57:25.190 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:25.190 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:25.190 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:25.190 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:25.190 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:25.190 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:25.190 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:25.190 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:25.190 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:25.190 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:25.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.180+0000 7f4ffc9ec640 10 monclient: _finish_auth 0 2024-09-11T01:57:25.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.180+0000 7f4ffc9ec640 10 monclient: _check_auth_tickets 2024-09-11T01:57:25.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.180+0000 7f4ffc9ec640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:25.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.180+0000 7f4ffc9ec640 1 -- 172.21.15.12:0/2210347941 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ff0007bd0 con 0x7f4ff80a0c90 2024-09-11T01:57:25.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.180+0000 7f4fffc52640 5 monclient: authenticate success, global_id 304232 2024-09-11T01:57:25.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.180+0000 7f4fffc52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:57:25.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.180+0000 7f4ffc9ec640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:25.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.180+0000 7f4fffc52640 10 monclient: get_monmap_and_config success 2024-09-11T01:57:25.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.180+0000 7f4ffc9ec640 1 -- 172.21.15.12:0/2210347941 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0031400 con 0x7f4ff80a0c90 2024-09-11T01:57:25.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.180+0000 7f4fffc52640 4 set_mon_vals no callback set 2024-09-11T01:57:25.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.180+0000 7f4ffc9ec640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:25.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.180+0000 7f4ffc9ec640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:57:25.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.180+0000 7f4ffc9ec640 10 monclient: dump: 2024-09-11T01:57:25.191 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:25.191 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:25.192 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:25.192 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:25.192 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:25.192 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:25.192 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:25.192 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:25.192 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:25.192 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:25.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.181+0000 7f4fffc52640 10 monclient: shutdown 2024-09-11T01:57:25.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.181+0000 7f4fffc52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:25.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.181+0000 7f4fffc52640 1 -- 172.21.15.12:0/2210347941 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ff80a0c90 msgr2=0x7f4ff813e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:25.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.181+0000 7f4fffc52640 1 --2- 172.21.15.12:0/2210347941 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ff80a0c90 0x7f4ff813e530 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0009dc0 tx=0x7f4ff002a480 comp rx=0 tx=0).stop 2024-09-11T01:57:25.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.181+0000 7f4fffc52640 1 -- 172.21.15.12:0/2210347941 shutdown_connections 2024-09-11T01:57:25.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.181+0000 7f4fffc52640 1 --2- 172.21.15.12:0/2210347941 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ff80a0c90 0x7f4ff813e530 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:25.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.181+0000 7f4fffc52640 1 -- 172.21.15.12:0/2210347941 >> 172.21.15.12:0/2210347941 conn(0x7f4ff80a2f70 msgr2=0x7f4ff80a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:25.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.181+0000 7f4fffc52640 1 -- 172.21.15.12:0/2210347941 shutdown_connections 2024-09-11T01:57:25.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.181+0000 7f4fffc52640 1 -- 172.21.15.12:0/2210347941 wait complete. 2024-09-11T01:57:25.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.181+0000 7f4fffc52640 10 monclient: build_initial_monmap 2024-09-11T01:57:25.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.181+0000 7f4fffc52640 10 monclient: monmap: 2024-09-11T01:57:25.193 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:25.193 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:57:25.193 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:57:25.193 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:57:25.193 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:25.193 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:25.194 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:57:25.194 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:57:25.194 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:57:25.194 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:25.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4fffc52640 1 Processor -- start 2024-09-11T01:57:25.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4fffc52640 1 -- start start 2024-09-11T01:57:25.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4fffc52640 10 monclient: init 2024-09-11T01:57:25.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4fffc52640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:25.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4fffc52640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T01:57:25.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4fffc52640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ff80a0370 0x7f4ff8127e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:25.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4fffc52640 10 monclient(hunting): picked mon.noname-b con 0x7f4ff80a0370 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:57:25.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4fffc52640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ff80a0c90 0x7f4ff8125d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:25.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4fffc52640 10 monclient(hunting): picked mon.noname-a con 0x7f4ff80a0c90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:57:25.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4fffc52640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ff80a1380 0x7f4ff813dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:25.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4fffc52640 10 monclient(hunting): picked mon.noname-c con 0x7f4ff80a1380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:25.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4fffc52640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:25.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4fffc52640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff813ebf0 con 0x7f4ff80a0c90 2024-09-11T01:57:25.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4fffc52640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:25.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4fffc52640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff813ea70 con 0x7f4ff80a0370 2024-09-11T01:57:25.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4fffc52640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:25.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4fffc52640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f4ff813ed70 con 0x7f4ff80a1380 2024-09-11T01:57:25.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4fffc52640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:25.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4fffc52640 10 monclient(hunting): authenticate will time out at 2451.624623s 2024-09-11T01:57:25.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4ffd1ed640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ff80a0c90 0x7f4ff8125d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:25.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4ffe1ef640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ff80a1380 0x7f4ff813dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:25.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4ffd1ed640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ff80a0c90 0x7f4ff8125d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45426/0 (socket says 172.21.15.12:45426) 2024-09-11T01:57:25.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4ffe1ef640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ff80a1380 0x7f4ff813dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:52908/0 (socket says 172.21.15.12:52908) 2024-09-11T01:57:25.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4ffd1ed640 1 -- 172.21.15.12:0/3363775600 learned_addr learned my addr 172.21.15.12:0/3363775600 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:25.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4ffd1ed640 10 monclient(hunting): get_auth_request con 0x7f4ff80a0c90 auth_method 0 2024-09-11T01:57:25.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4ffd1ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:25.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4ffd1ed640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:25.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4ffd1ed640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:25.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4ffe1ef640 10 monclient(hunting): get_auth_request con 0x7f4ff80a1380 auth_method 0 2024-09-11T01:57:25.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4ffe1ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:25.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4ffe1ef640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:25.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4ffe1ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:25.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4ffd1ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:25.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4ffd1ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:25.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4ffd1ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:25.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4ffe1ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:25.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4ffe1ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4ffe1ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4ffd9ee640 1 --2- 172.21.15.12:0/3363775600 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ff80a0370 0x7f4ff8127e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4ffd9ee640 10 monclient(hunting): get_auth_request con 0x7f4ff80a0370 auth_method 0 2024-09-11T01:57:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4ffd9ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4ffd9ee640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.182+0000 7f4ffd9ee640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.183+0000 7f4ffd1ed640 10 monclient(hunting): handle_auth_done global_id 294339 payload 824 2024-09-11T01:57:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.183+0000 7f4ffd1ed640 1 -- 172.21.15.12:0/3363775600 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ff80a1380 msgr2=0x7f4ff813dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.183+0000 7f4ffd1ed640 1 --2- 172.21.15.12:0/3363775600 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ff80a1380 0x7f4ff813dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.183+0000 7f4ffd1ed640 1 -- 172.21.15.12:0/3363775600 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ff80a0370 msgr2=0x7f4ff8127e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.183+0000 7f4ffd1ed640 1 --2- 172.21.15.12:0/3363775600 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ff80a0370 0x7f4ff8127e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.183+0000 7f4ffd1ed640 10 monclient: _finish_hunting 0 2024-09-11T01:57:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.183+0000 7f4ffd1ed640 1 monclient: found mon.noname-a 2024-09-11T01:57:25.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.183+0000 7f4ffd1ed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:25.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.183+0000 7f4ffd1ed640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:57:25.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.183+0000 7f4ffd1ed640 1 -- 172.21.15.12:0/3363775600 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ff81272c0 con 0x7f4ff80a0c90 2024-09-11T01:57:25.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.183+0000 7f4ffe1ef640 1 --2- 172.21.15.12:0/3363775600 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ff80a1380 0x7f4ff813dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:57:25.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.183+0000 7f4ffd9ee640 1 --2- 172.21.15.12:0/3363775600 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ff80a0370 0x7f4ff8127e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:57:25.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.185+0000 7f4ffd1ed640 1 -- 172.21.15.12:0/3363775600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ff80a0c90 msgr2=0x7f4ff8125d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:25.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.185+0000 7f4ffd1ed640 1 --2- 172.21.15.12:0/3363775600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ff80a0c90 0x7f4ff8125d10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4fe800c490 tx=0x7f4fe800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:25.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.185+0000 7f4fee7fc640 1 -- 172.21.15.12:0/3363775600 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe8007070 con 0x7f4ff80a0c90 2024-09-11T01:57:25.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.185+0000 7f4fee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:25.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.185+0000 7f4fee7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:57:25.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.185+0000 7f4fee7fc640 10 monclient: dump: 2024-09-11T01:57:25.198 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:25.198 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:25.199 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.185+0000 7f4fee7fc640 10 monclient: _finish_auth 0 2024-09-11T01:57:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.185+0000 7f4fee7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:57:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.185+0000 7f4fee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.185+0000 7f4fee7fc640 1 -- 172.21.15.12:0/3363775600 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fe8007230 con 0x7f4ff80a0c90 2024-09-11T01:57:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.185+0000 7f4fffc52640 5 monclient: authenticate success, global_id 294339 2024-09-11T01:57:25.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.185+0000 7f4fffc52640 10 monclient: _renew_subs 2024-09-11T01:57:25.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.185+0000 7f4fffc52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:25.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.185+0000 7f4fffc52640 1 -- 172.21.15.12:0/3363775600 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ff80702b0 con 0x7f4ff80a0c90 2024-09-11T01:57:25.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.185+0000 7f4fee7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:25.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.185+0000 7f4fee7fc640 1 -- 172.21.15.12:0/3363775600 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe8007620 con 0x7f4ff80a0c90 2024-09-11T01:57:25.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.185+0000 7f4fee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:25.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.185+0000 7f4fee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:25.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.185+0000 7f4fee7fc640 10 monclient: dump: 2024-09-11T01:57:25.200 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:25.200 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:25.200 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:25.200 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:25.200 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:25.200 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:25.200 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:25.201 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.185+0000 7f4feffff640 4 set_mon_vals no callback set 2024-09-11T01:57:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.185+0000 7f4fffc52640 10 monclient: _renew_subs 2024-09-11T01:57:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.185+0000 7f4fffc52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.185+0000 7f4fffc52640 1 -- 172.21.15.12:0/3363775600 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ff813e1c0 con 0x7f4ff80a0c90 2024-09-11T01:57:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.186+0000 7f4fee7fc640 1 -- 172.21.15.12:0/3363775600 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f4fe801b400 con 0x7f4ff80a0c90 2024-09-11T01:57:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.186+0000 7f4fffc52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.186+0000 7f4fffc52640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.186+0000 7f4fffc52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.186+0000 7f4fffc52640 1 -- 172.21.15.12:0/3363775600 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ff80a1760 con 0x7f4ff80a0c90 2024-09-11T01:57:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.186+0000 7f4fee7fc640 1 --2- 172.21.15.12:0/3363775600 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4fdc03e260 0x7f4fdc040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:25.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.186+0000 7f4fee7fc640 1 -- 172.21.15.12:0/3363775600 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(923..923 src has 889..923) ==== 4422+0+0 (secure 0 0 0) 0x7f4fe8052820 con 0x7f4ff80a0c90 2024-09-11T01:57:25.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.187+0000 7f4ffd9ee640 1 --2- 172.21.15.12:0/3363775600 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4fdc03e260 0x7f4fdc040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:25.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.187+0000 7f4ffd9ee640 10 monclient: get_auth_request con 0x7f4fdc03e260 auth_method 0 2024-09-11T01:57:25.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.189+0000 7f4ffd9ee640 1 --2- 172.21.15.12:0/3363775600 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4fdc03e260 0x7f4fdc040720 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4003eb0 tx=0x7f4ff4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:25.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.191+0000 7f4fee7fc640 1 -- 172.21.15.12:0/3363775600 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4fe8051380 con 0x7f4ff80a0c90 2024-09-11T01:57:25.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.191+0000 7f4fee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:25.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.191+0000 7f4fee7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:57:25.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.379+0000 7f4fffc52640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-11T01:57:25.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.379+0000 7f4fffc52640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T01:57:25.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.379+0000 7f4fffc52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:25.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.379+0000 7f4fffc52640 1 -- 172.21.15.12:0/3363775600 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4ff806d480 con 0x7f4ff80a0c90 2024-09-11T01:57:25.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.380+0000 7f4fee7fc640 1 -- 172.21.15.12:0/3363775600 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v923) ==== 74+0+12410 (secure 0 0 0) 0x7f4fe8012400 con 0x7f4ff80a0c90 2024-09-11T01:57:25.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.380+0000 7f4fee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T01:57:25.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.380+0000 7f4fee7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:57:25.382 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:57:25.382 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":923,"fsid":"ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab","created":"2024-09-11T01:29:02.540751+0000","modified":"2024-09-11T01:56:43.741526+0000","last_up_change":"2024-09-11T01:29:13.320573+0000","last_in_change":"2024-09-11T01:29:05.075851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-11T01:29:14.157646+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-11T01:29:14.872725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"616ba1d0-adf6-4fe1-85e3-266e53e21867","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":4266003406},{"type":"v1","addr":"172.21.15.12:6817","nonce":4266003406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":4266003406},{"type":"v1","addr":"172.21.15.12:6819","nonce":4266003406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":4266003406},{"type":"v1","addr":"172.21.15.12:6823","nonce":4266003406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":4266003406},{"type":"v1","addr":"172.21.15.12:6821","nonce":4266003406}]},"public_addr":"172.21.15.12:6817/4266003406","cluster_addr":"172.21.15.12:6819/4266003406","heartbeat_back_addr":"172.21.15.12:6823/4266003406","heartbeat_front_addr":"172.21.15.12:6821/4266003406","state":["exists","up"]},{"osd":1,"uuid":"9b570874-7a3d-4867-8c44-29084a8bf649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3563357280},{"type":"v1","addr":"172.21.15.12:6809","nonce":3563357280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3563357280},{"type":"v1","addr":"172.21.15.12:6811","nonce":3563357280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3563357280},{"type":"v1","addr":"172.21.15.12:6815","nonce":3563357280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3563357280},{"type":"v1","addr":"172.21.15.12:6813","nonce":3563357280}]},"public_addr":"172.21.15.12:6809/3563357280","cluster_addr":"172.21.15.12:6811/3563357280","heartbeat_back_addr":"172.21.15.12:6815/3563357280","heartbeat_front_addr":"172.21.15.12:6813/3563357280","state":["exists","up"]},{"osd":2,"uuid":"b12ee288-ce57-41de-8b8e-6fad509311b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2893036867},{"type":"v1","addr":"172.21.15.12:6801","nonce":2893036867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2893036867},{"type":"v1","addr":"172.21.15.12:6803","nonce":2893036867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2893036867},{"type":"v1","addr":"172.21.15.12:6807","nonce":2893036867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2893036867},{"type":"v1","addr":"172.21.15.12:6805","nonce":2893036867}]},"public_addr":"172.21.15.12:6801/2893036867","cluster_addr":"172.21.15.12:6803/2893036867","heartbeat_back_addr":"172.21.15.12:6807/2893036867","heartbeat_front_addr":"172.21.15.12:6805/2893036867","state":["exists","up"]},{"osd":3,"uuid":"a8e81aa4-0dee-487d-981d-1f593e363eee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":3091488141},{"type":"v1","addr":"172.21.15.71:6819","nonce":3091488141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":3091488141},{"type":"v1","addr":"172.21.15.71:6821","nonce":3091488141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":3091488141},{"type":"v1","addr":"172.21.15.71:6825","nonce":3091488141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":3091488141},{"type":"v1","addr":"172.21.15.71:6823","nonce":3091488141}]},"public_addr":"172.21.15.71:6819/3091488141","cluster_addr":"172.21.15.71:6821/3091488141","heartbeat_back_addr":"172.21.15.71:6825/3091488141","heartbeat_front_addr":"172.21.15.71:6823/3091488141","state":["exists","up"]},{"osd":4,"uuid":"9db77181-fc18-408f-a35c-87bf4b4ba2d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":1790415427},{"type":"v1","addr":"172.21.15.71:6801","nonce":1790415427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":1790415427},{"type":"v1","addr":"172.21.15.71:6803","nonce":1790415427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":1790415427},{"type":"v1","addr":"172.21.15.71:6807","nonce":1790415427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":1790415427},{"type":"v1","addr":"172.21.15.71:6805","nonce":1790415427}]},"public_addr":"172.21.15.71:6801/1790415427","cluster_addr":"172.21.15.71:6803/1790415427","heartbeat_back_addr":"172.21.15.71:6807/1790415427","heartbeat_front_addr":"172.21.15.71:6805/1790415427","state":["exists","up"]},{"osd":5,"uuid":"fadf2191-f31a-4880-a529-5b2d1023d236","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":2032569317},{"type":"v1","addr":"172.21.15.71:6809","nonce":2032569317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":2032569317},{"type":"v1","addr":"172.21.15.71:6811","nonce":2032569317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":2032569317},{"type":"v1","addr":"172.21.15.71:6815","nonce":2032569317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":2032569317},{"type":"v1","addr":"172.21.15.71:6813","nonce":2032569317}]},"public_addr":"172.21.15.71:6809/2032569317","cluster_addr":"172.21.15.71:6811/2032569317","heartbeat_back_addr":"172.21.15.71:6815/2032569317","heartbeat_front_addr":"172.21.15.71:6813/2032569317","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:11.548818+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:11.310103+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:10.578802+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:11.748432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:11.098916+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T01:29:11.150538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":423,"length":1},{"begin":441,"length":1},{"begin":451,"length":1},{"begin":458,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T01:57:25.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.382+0000 7f4fffc52640 1 -- 172.21.15.12:0/3363775600 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4fdc03e260 msgr2=0x7f4fdc040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:25.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.382+0000 7f4fffc52640 1 --2- 172.21.15.12:0/3363775600 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4fdc03e260 0x7f4fdc040720 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4003eb0 tx=0x7f4ff4009040 comp rx=0 tx=0).stop 2024-09-11T01:57:25.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.382+0000 7f4fffc52640 10 monclient: shutdown 2024-09-11T01:57:25.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.382+0000 7f4fffc52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:25.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.382+0000 7f4fffc52640 1 -- 172.21.15.12:0/3363775600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ff80a0c90 msgr2=0x7f4ff8125d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:25.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.382+0000 7f4fffc52640 1 --2- 172.21.15.12:0/3363775600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ff80a0c90 0x7f4ff8125d10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4fe800c490 tx=0x7f4fe800c950 comp rx=0 tx=0).stop 2024-09-11T01:57:25.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.382+0000 7f4fffc52640 1 -- 172.21.15.12:0/3363775600 shutdown_connections 2024-09-11T01:57:25.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.382+0000 7f4fffc52640 1 --2- 172.21.15.12:0/3363775600 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4ff80a1380 0x7f4ff813dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:25.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.382+0000 7f4fffc52640 1 --2- 172.21.15.12:0/3363775600 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f4fdc03e260 0x7f4fdc040720 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:25.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.382+0000 7f4fffc52640 1 --2- 172.21.15.12:0/3363775600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ff80a0c90 0x7f4ff8125d10 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:25.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.382+0000 7f4fffc52640 1 --2- 172.21.15.12:0/3363775600 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ff80a0370 0x7f4ff8127e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:25.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.382+0000 7f4fffc52640 1 -- 172.21.15.12:0/3363775600 >> 172.21.15.12:0/3363775600 conn(0x7f4ff80a2f70 msgr2=0x7f4ff80717d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:25.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.383+0000 7f4fffc52640 1 -- 172.21.15.12:0/3363775600 shutdown_connections 2024-09-11T01:57:25.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:25.383+0000 7f4fffc52640 1 -- 172.21.15.12:0/3363775600 wait complete. 2024-09-11T01:57:26.396 INFO:tasks.ceph:Scrubbing osd.0 2024-09-11T01:57:26.396 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-11T01:57:26.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.516+0000 7f9df8589640 10 monclient: get_monmap_and_config 2024-09-11T01:57:26.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.516+0000 7f9df8589640 10 monclient: build_initial_monmap 2024-09-11T01:57:26.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.517+0000 7f9df8589640 10 monclient: monmap: 2024-09-11T01:57:26.519 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:26.519 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:26.519 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:57:26.517753+0000 2024-09-11T01:57:26.519 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:57:26.517753+0000 2024-09-11T01:57:26.519 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:26.519 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:26.519 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:57:26.519 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:57:26.519 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:57:26.519 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:57:26.520 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:57:26.520 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:26.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.517+0000 7f9df8589640 1 Processor -- start 2024-09-11T01:57:26.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.517+0000 7f9df8589640 1 -- start start 2024-09-11T01:57:26.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.517+0000 7f9df8589640 10 monclient: init 2024-09-11T01:57:26.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df8589640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:26.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df8589640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-11T01:57:26.522 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df8589640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9df00a5090 0x7f9df00a5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:26.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df8589640 10 monclient(hunting): picked mon.noname-b con 0x7f9df00a5090 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:26.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df8589640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9df00a67d0 addr v1:172.21.15.71:6789/0 2024-09-11T01:57:26.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df8589640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9df013d310 0x7f9df013f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:26.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df8589640 10 monclient(hunting): picked mon.noname-a con 0x7f9df013d310 addr v2:172.21.15.12:3300/0 2024-09-11T01:57:26.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df8589640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9df00a6f60 con 0x7f9df00a67d0 2024-09-11T01:57:26.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df8589640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:26.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df8589640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f9df009c170 con 0x7f9df013d310 2024-09-11T01:57:26.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df8589640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:26.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df8589640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f9df009c450 con 0x7f9df00a5090 2024-09-11T01:57:26.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df8589640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:26.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df8589640 10 monclient(hunting): authenticate will time out at 2452.960574s 2024-09-11T01:57:26.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df6325640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9df00a5090 0x7f9df00a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:26.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df6b26640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9df013d310 0x7f9df013f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:26.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df6325640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9df00a5090 0x7f9df00a5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46504/0 (socket says 172.21.15.12:46504) 2024-09-11T01:57:26.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df6325640 1 -- 172.21.15.12:0/4249915728 learned_addr learned my addr 172.21.15.12:0/4249915728 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:26.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df6325640 10 monclient(hunting): get_auth_request con 0x7f9df00a5090 auth_method 0 2024-09-11T01:57:26.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df6325640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:26.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df6325640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:26.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df6325640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:26.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df6b26640 10 monclient(hunting): get_auth_request con 0x7f9df013d310 auth_method 0 2024-09-11T01:57:26.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df6b26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:26.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df6b26640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:26.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df6b26640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:26.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df6325640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:26.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df6325640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:26.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df6325640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:26.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df6b26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:26.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df6b26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:26.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.518+0000 7f9df6b26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:26.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.519+0000 7f9df6325640 10 monclient(hunting): handle_auth_done global_id 304238 payload 274 2024-09-11T01:57:26.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.519+0000 7f9df6325640 1 -- 172.21.15.12:0/4249915728 >> v2:172.21.15.12:3300/0 conn(0x7f9df013d310 msgr2=0x7f9df013f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:26.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.519+0000 7f9df6325640 1 --2- 172.21.15.12:0/4249915728 >> v2:172.21.15.12:3300/0 conn(0x7f9df013d310 0x7f9df013f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:26.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.519+0000 7f9df6325640 1 -- 172.21.15.12:0/4249915728 >> v1:172.21.15.71:6789/0 conn(0x7f9df00a67d0 legacy=0x7f9df00a59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:26.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.519+0000 7f9df6325640 10 monclient: _finish_hunting 0 2024-09-11T01:57:26.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.519+0000 7f9df6325640 1 monclient: found mon.noname-b 2024-09-11T01:57:26.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.519+0000 7f9df6325640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:26.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.519+0000 7f9df6325640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:57:26.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.519+0000 7f9df6325640 1 -- 172.21.15.12:0/4249915728 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df00a60c0 con 0x7f9df00a5090 2024-09-11T01:57:26.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.521+0000 7f9df6325640 1 -- 172.21.15.12:0/4249915728 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9df00a5090 msgr2=0x7f9df00a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:26.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.521+0000 7f9df6325640 1 --2- 172.21.15.12:0/4249915728 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9df00a5090 0x7f9df00a5470 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9de0009870 tx=0x7f9de002a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c2443b7d1236bc4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:26.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.522+0000 7f9df5323640 1 -- 172.21.15.12:0/4249915728 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de0037070 con 0x7f9df00a5090 2024-09-11T01:57:26.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.522+0000 7f9df5323640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:26.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.522+0000 7f9df5323640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:57:26.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.522+0000 7f9df5323640 10 monclient: dump: 2024-09-11T01:57:26.531 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:26.531 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:26.531 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:26.531 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:26.531 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:26.531 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:26.531 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:26.531 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:26.532 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:26.532 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:26.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.522+0000 7f9df5323640 10 monclient: _finish_auth 0 2024-09-11T01:57:26.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.522+0000 7f9df5323640 10 monclient: _check_auth_tickets 2024-09-11T01:57:26.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.522+0000 7f9df5323640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:26.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.522+0000 7f9df5323640 1 -- 172.21.15.12:0/4249915728 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9de0032440 con 0x7f9df00a5090 2024-09-11T01:57:26.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.522+0000 7f9df5323640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:26.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.522+0000 7f9df5323640 1 -- 172.21.15.12:0/4249915728 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de0032810 con 0x7f9df00a5090 2024-09-11T01:57:26.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.522+0000 7f9df8589640 5 monclient: authenticate success, global_id 304238 2024-09-11T01:57:26.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.522+0000 7f9df8589640 10 monclient: get_monmap_and_config success 2024-09-11T01:57:26.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.522+0000 7f9df8589640 4 set_mon_vals no callback set 2024-09-11T01:57:26.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.522+0000 7f9df5323640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:26.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.522+0000 7f9df5323640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:57:26.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.522+0000 7f9df5323640 10 monclient: dump: 2024-09-11T01:57:26.533 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:26.533 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:26.533 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:26.533 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:26.533 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:26.533 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:26.533 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:26.533 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:26.533 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:26.533 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:26.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.522+0000 7f9df8589640 10 monclient: shutdown 2024-09-11T01:57:26.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.522+0000 7f9df8589640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:26.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.522+0000 7f9df8589640 1 -- 172.21.15.12:0/4249915728 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9df00a5090 msgr2=0x7f9df00a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:26.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.522+0000 7f9df8589640 1 --2- 172.21.15.12:0/4249915728 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9df00a5090 0x7f9df00a5470 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9de0009870 tx=0x7f9de002a0a0 comp rx=0 tx=0).stop 2024-09-11T01:57:26.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.523+0000 7f9df8589640 1 -- 172.21.15.12:0/4249915728 shutdown_connections 2024-09-11T01:57:26.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.523+0000 7f9df8589640 1 --2- 172.21.15.12:0/4249915728 >> v2:172.21.15.12:3300/0 conn(0x7f9df013d310 0x7f9df013f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:26.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.523+0000 7f9df8589640 1 --2- 172.21.15.12:0/4249915728 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9df00a5090 0x7f9df00a5470 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:26.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.523+0000 7f9df8589640 1 -- 172.21.15.12:0/4249915728 >> 172.21.15.12:0/4249915728 conn(0x7f9df006ed00 msgr2=0x7f9df006f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:26.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.523+0000 7f9df8589640 1 -- 172.21.15.12:0/4249915728 shutdown_connections 2024-09-11T01:57:26.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.523+0000 7f9df8589640 1 -- 172.21.15.12:0/4249915728 wait complete. 2024-09-11T01:57:26.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.523+0000 7f9df8589640 10 monclient: build_initial_monmap 2024-09-11T01:57:26.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.523+0000 7f9df8589640 10 monclient: monmap: 2024-09-11T01:57:26.534 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:26.534 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:57:26.535 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:57:26.535 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:57:26.535 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:26.535 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:26.535 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:57:26.535 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:57:26.535 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:57:26.535 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:26.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df8589640 1 Processor -- start 2024-09-11T01:57:26.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df8589640 1 -- start start 2024-09-11T01:57:26.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df8589640 10 monclient: init 2024-09-11T01:57:26.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df8589640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:26.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df8589640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:57:26.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df8589640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9df00a5090 0x7f9df00a1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:26.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df8589640 10 monclient(hunting): picked mon.noname-b con 0x7f9df00a5090 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:57:26.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df8589640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9df00a67d0 0x7f9df00a2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:26.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df8589640 10 monclient(hunting): picked mon.noname-c con 0x7f9df00a67d0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:26.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df8589640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9df013d310 0x7f9df01cced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:26.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df8589640 10 monclient(hunting): picked mon.noname-a con 0x7f9df013d310 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:57:26.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df8589640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:26.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df8589640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9df013fc40 con 0x7f9df013d310 2024-09-11T01:57:26.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df8589640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:26.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df8589640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f9df009d3b0 con 0x7f9df00a5090 2024-09-11T01:57:26.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df8589640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:26.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df8589640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f9df013fdc0 con 0x7f9df00a67d0 2024-09-11T01:57:26.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df8589640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:26.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df8589640 10 monclient(hunting): authenticate will time out at 2452.966780s 2024-09-11T01:57:26.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df6b26640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9df013d310 0x7f9df01cced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:26.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df5b24640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9df00a67d0 0x7f9df00a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:26.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df6b26640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9df013d310 0x7f9df01cced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54102/0 (socket says 172.21.15.12:54102) 2024-09-11T01:57:26.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df6b26640 1 -- 172.21.15.12:0/1724715027 learned_addr learned my addr 172.21.15.12:0/1724715027 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:26.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df5b24640 10 monclient(hunting): get_auth_request con 0x7f9df00a67d0 auth_method 0 2024-09-11T01:57:26.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df5b24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:26.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df5b24640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:26.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df5b24640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:26.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df6b26640 10 monclient(hunting): get_auth_request con 0x7f9df013d310 auth_method 0 2024-09-11T01:57:26.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df6b26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:26.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df6b26640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:26.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.524+0000 7f9df6b26640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:26.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.525+0000 7f9df6325640 1 --2- 172.21.15.12:0/1724715027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9df00a5090 0x7f9df00a1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:26.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.525+0000 7f9df6325640 10 monclient(hunting): get_auth_request con 0x7f9df00a5090 auth_method 0 2024-09-11T01:57:26.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.525+0000 7f9df6325640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:26.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.525+0000 7f9df6325640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:26.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.525+0000 7f9df6325640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:26.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.525+0000 7f9df5b24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:26.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.525+0000 7f9df5b24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:26.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.525+0000 7f9df5b24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:26.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.525+0000 7f9df6b26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:26.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.525+0000 7f9df6b26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:26.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.525+0000 7f9df6b26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:26.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.525+0000 7f9df6325640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:26.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.525+0000 7f9df6325640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:26.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.525+0000 7f9df6325640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:26.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.525+0000 7f9df5b24640 10 monclient(hunting): handle_auth_done global_id 304241 payload 824 2024-09-11T01:57:26.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.525+0000 7f9df5b24640 1 -- 172.21.15.12:0/1724715027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9df00a5090 msgr2=0x7f9df00a1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:26.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.525+0000 7f9df5b24640 1 --2- 172.21.15.12:0/1724715027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9df00a5090 0x7f9df00a1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:26.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.525+0000 7f9df5b24640 1 -- 172.21.15.12:0/1724715027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9df013d310 msgr2=0x7f9df01cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:26.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.525+0000 7f9df5b24640 1 --2- 172.21.15.12:0/1724715027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9df013d310 0x7f9df01cced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:26.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.525+0000 7f9df5b24640 10 monclient: _finish_hunting 0 2024-09-11T01:57:26.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.525+0000 7f9df5b24640 1 monclient: found mon.noname-c 2024-09-11T01:57:26.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.525+0000 7f9df5b24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:26.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.525+0000 7f9df5b24640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:57:26.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.525+0000 7f9df5b24640 1 -- 172.21.15.12:0/1724715027 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df013db90 con 0x7f9df00a67d0 2024-09-11T01:57:26.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.525+0000 7f9df6b26640 1 --2- 172.21.15.12:0/1724715027 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9df013d310 0x7f9df01cced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:57:26.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.525+0000 7f9df6325640 1 --2- 172.21.15.12:0/1724715027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9df00a5090 0x7f9df00a1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:57:26.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.527+0000 7f9df5b24640 1 -- 172.21.15.12:0/1724715027 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9df00a67d0 msgr2=0x7f9df00a2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:26.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.527+0000 7f9df5b24640 1 --2- 172.21.15.12:0/1724715027 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9df00a67d0 0x7f9df00a2530 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9de400c410 tx=0x7f9de400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:26.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.528+0000 7f9deeffd640 1 -- 172.21.15.12:0/1724715027 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de4017070 con 0x7f9df00a67d0 2024-09-11T01:57:26.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.528+0000 7f9deeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:26.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.528+0000 7f9deeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:57:26.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.528+0000 7f9deeffd640 10 monclient: dump: 2024-09-11T01:57:26.540 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:26.540 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:26.541 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:26.541 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:26.541 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:26.541 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:26.541 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:26.541 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:26.541 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:26.541 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:26.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.528+0000 7f9deeffd640 10 monclient: _finish_auth 0 2024-09-11T01:57:26.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.528+0000 7f9deeffd640 10 monclient: _check_auth_tickets 2024-09-11T01:57:26.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.528+0000 7f9deeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:26.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.528+0000 7f9deeffd640 1 -- 172.21.15.12:0/1724715027 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9de4005230 con 0x7f9df00a67d0 2024-09-11T01:57:26.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.528+0000 7f9df8589640 5 monclient: authenticate success, global_id 304241 2024-09-11T01:57:26.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.528+0000 7f9df8589640 10 monclient: _renew_subs 2024-09-11T01:57:26.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.528+0000 7f9df8589640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:26.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.528+0000 7f9df8589640 1 -- 172.21.15.12:0/1724715027 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9df013d760 con 0x7f9df00a67d0 2024-09-11T01:57:26.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.528+0000 7f9deeffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:26.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.528+0000 7f9deeffd640 1 -- 172.21.15.12:0/1724715027 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de40117a0 con 0x7f9df00a67d0 2024-09-11T01:57:26.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.528+0000 7f9deeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:26.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.528+0000 7f9deeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:57:26.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.528+0000 7f9deeffd640 10 monclient: dump: 2024-09-11T01:57:26.542 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:26.542 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:26.542 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:26.543 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:26.543 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:26.543 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:26.543 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:26.543 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:26.543 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:26.543 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:26.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.528+0000 7f9df4b22640 4 set_mon_vals no callback set 2024-09-11T01:57:26.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.528+0000 7f9df8589640 10 monclient: _renew_subs 2024-09-11T01:57:26.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.528+0000 7f9df8589640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:26.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.528+0000 7f9df8589640 1 -- 172.21.15.12:0/1724715027 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9df013a280 con 0x7f9df00a67d0 2024-09-11T01:57:26.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.529+0000 7f9df8589640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:26.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.529+0000 7f9df8589640 1 -- 172.21.15.12:0/1724715027 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9df00a39f0 con 0x7f9df00a67d0 2024-09-11T01:57:26.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.529+0000 7f9deeffd640 1 -- 172.21.15.12:0/1724715027 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f9de40119c0 con 0x7f9df00a67d0 2024-09-11T01:57:26.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.530+0000 7f9deeffd640 1 --2- 172.21.15.12:0/1724715027 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9ddc03e210 0x7f9ddc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:26.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.530+0000 7f9deeffd640 1 -- 172.21.15.12:0/1724715027 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(923..923 src has 889..923) ==== 4422+0+0 (secure 0 0 0) 0x7f9de4052170 con 0x7f9df00a67d0 2024-09-11T01:57:26.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.530+0000 7f9deeffd640 1 --2- 172.21.15.12:0/1724715027 >> [v2:172.21.15.12:6816/4266003406,v1:172.21.15.12:6817/4266003406] conn(0x7f9ddc044090 0x7f9ddc046480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:26.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.530+0000 7f9deeffd640 1 -- 172.21.15.12:0/1724715027 --> [v2:172.21.15.12:6816/4266003406,v1:172.21.15.12:6817/4266003406] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9ddc046b30 con 0x7f9ddc044090 2024-09-11T01:57:26.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.530+0000 7f9df6b26640 1 --2- 172.21.15.12:0/1724715027 >> [v2:172.21.15.12:6816/4266003406,v1:172.21.15.12:6817/4266003406] conn(0x7f9ddc044090 0x7f9ddc046480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:26.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.530+0000 7f9df6b26640 10 monclient: get_auth_request con 0x7f9ddc044090 auth_method 0 2024-09-11T01:57:26.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.530+0000 7f9df6325640 1 --2- 172.21.15.12:0/1724715027 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9ddc03e210 0x7f9ddc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:26.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.530+0000 7f9df6325640 10 monclient: get_auth_request con 0x7f9ddc03e210 auth_method 0 2024-09-11T01:57:26.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.532+0000 7f9deeffd640 1 -- 172.21.15.12:0/1724715027 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_get_version_reply(handle=1 version=923) ==== 24+0+0 (secure 0 0 0) 0x7f9de4059050 con 0x7f9df00a67d0 2024-09-11T01:57:26.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.532+0000 7f9deeffd640 10 monclient: handle_get_version_reply finishing 1 version 923 2024-09-11T01:57:26.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.533+0000 7f9df6325640 1 --2- 172.21.15.12:0/1724715027 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9ddc03e210 0x7f9ddc0406d0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f9de0031000 tx=0x7f9de002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:26.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.535+0000 7f9df6b26640 1 --2- 172.21.15.12:0/1724715027 >> [v2:172.21.15.12:6816/4266003406,v1:172.21.15.12:6817/4266003406] conn(0x7f9ddc044090 0x7f9ddc046480 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:26.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.535+0000 7f9deeffd640 1 -- 172.21.15.12:0/1724715027 <== osd.0 v2:172.21.15.12:6816/4266003406 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9ddc046b30 con 0x7f9ddc044090 2024-09-11T01:57:26.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.548+0000 7f9df8589640 1 -- 172.21.15.12:0/1724715027 --> [v2:172.21.15.12:6816/4266003406,v1:172.21.15.12:6817/4266003406] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f9df00a5470 con 0x7f9ddc044090 2024-09-11T01:57:26.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.548+0000 7f9deeffd640 1 -- 172.21.15.12:0/1724715027 <== osd.0 v2:172.21.15.12:6816/4266003406 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f9df00a5470 con 0x7f9ddc044090 2024-09-11T01:57:26.550 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:57:26.550 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T01:57:26.550 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:57:26.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.548+0000 7f9df8589640 1 -- 172.21.15.12:0/1724715027 >> [v2:172.21.15.12:6816/4266003406,v1:172.21.15.12:6817/4266003406] conn(0x7f9ddc044090 msgr2=0x7f9ddc046480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:26.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.548+0000 7f9df8589640 1 --2- 172.21.15.12:0/1724715027 >> [v2:172.21.15.12:6816/4266003406,v1:172.21.15.12:6817/4266003406] conn(0x7f9ddc044090 0x7f9ddc046480 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:26.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.549+0000 7f9df8589640 1 -- 172.21.15.12:0/1724715027 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9ddc03e210 msgr2=0x7f9ddc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:26.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.549+0000 7f9df8589640 1 --2- 172.21.15.12:0/1724715027 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f9ddc03e210 0x7f9ddc0406d0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f9de0031000 tx=0x7f9de002e000 comp rx=0 tx=0).stop 2024-09-11T01:57:26.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.549+0000 7f9df8589640 10 monclient: shutdown 2024-09-11T01:57:26.551 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.549+0000 7f9df8589640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:26.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.549+0000 7f9df8589640 1 -- 172.21.15.12:0/1724715027 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9df00a67d0 msgr2=0x7f9df00a2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:26.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.549+0000 7f9df8589640 1 --2- 172.21.15.12:0/1724715027 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9df00a67d0 0x7f9df00a2530 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9de400c410 tx=0x7f9de400c8d0 comp rx=0 tx=0).stop 2024-09-11T01:57:26.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.549+0000 7f9df6b26640 1 -- 172.21.15.12:0/1724715027 reap_dead start 2024-09-11T01:57:26.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.549+0000 7f9df8589640 1 -- 172.21.15.12:0/1724715027 shutdown_connections 2024-09-11T01:57:26.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.549+0000 7f9df8589640 1 -- 172.21.15.12:0/1724715027 >> 172.21.15.12:0/1724715027 conn(0x7f9df006ed00 msgr2=0x7f9df013f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:26.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.549+0000 7f9df8589640 1 -- 172.21.15.12:0/1724715027 shutdown_connections 2024-09-11T01:57:26.552 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.549+0000 7f9df8589640 1 -- 172.21.15.12:0/1724715027 wait complete. 2024-09-11T01:57:26.560 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-11T01:57:26.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.661+0000 7fe589060640 10 monclient: get_monmap_and_config 2024-09-11T01:57:26.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.661+0000 7fe589060640 10 monclient: build_initial_monmap 2024-09-11T01:57:26.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.661+0000 7fe589060640 10 monclient: monmap: 2024-09-11T01:57:26.663 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:26.663 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:26.663 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:57:26.661842+0000 2024-09-11T01:57:26.663 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:57:26.661842+0000 2024-09-11T01:57:26.663 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:26.663 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:26.663 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:57:26.663 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:57:26.663 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:57:26.663 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:57:26.663 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:57:26.663 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:26.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.661+0000 7fe589060640 1 Processor -- start 2024-09-11T01:57:26.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.661+0000 7fe589060640 1 -- start start 2024-09-11T01:57:26.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.661+0000 7fe589060640 10 monclient: init 2024-09-11T01:57:26.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe589060640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:26.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe589060640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-11T01:57:26.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe589060640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe584137e50 0x7fe58409d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:26.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe589060640 10 monclient(hunting): picked mon.noname-a con 0x7fe584137e50 addr v2:172.21.15.12:3300/0 2024-09-11T01:57:26.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe589060640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe584138230 addr v1:172.21.15.12:6789/0 2024-09-11T01:57:26.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe589060640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe58413d380 0x7fe58413f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:26.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe589060640 10 monclient(hunting): picked mon.noname-c con 0x7fe58413d380 addr v2:172.21.15.71:3300/0 2024-09-11T01:57:26.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe589060640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5840a62d0 con 0x7fe584138230 2024-09-11T01:57:26.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe589060640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:26.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe589060640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7fe58409bb40 con 0x7fe584137e50 2024-09-11T01:57:26.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe589060640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:26.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe589060640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fe58409c4a0 con 0x7fe58413d380 2024-09-11T01:57:26.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe589060640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:26.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe589060640 10 monclient(hunting): authenticate will time out at 2453.104258s 2024-09-11T01:57:26.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe582d9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe584137e50 0x7fe58409d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:26.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe58259c640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe584138230 0x7fe584138640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33894/0 (socket says 172.21.15.12:33894) 2024-09-11T01:57:26.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe582d9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe584137e50 0x7fe58409d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54114/0 (socket says 172.21.15.12:54114) 2024-09-11T01:57:26.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe58259c640 1 -- 172.21.15.12:0/3224759730 learned_addr learned my addr 172.21.15.12:0/3224759730 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:57:26.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe582d9d640 10 monclient(hunting): get_auth_request con 0x7fe584137e50 auth_method 0 2024-09-11T01:57:26.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe582d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:26.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe582d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:26.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe582d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:26.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe582d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:26.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe582d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:26.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe582d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:26.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe58359e640 1 --2- 172.21.15.12:0/3224759730 >> v2:172.21.15.71:3300/0 conn(0x7fe58413d380 0x7fe58413f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:26.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe58359e640 10 monclient(hunting): get_auth_request con 0x7fe58413d380 auth_method 0 2024-09-11T01:57:26.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe58359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:26.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe58359e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:26.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe58359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:26.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe58359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:26.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe58359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:26.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe58359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:26.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe582d9d640 10 monclient(hunting): handle_auth_done global_id 294348 payload 274 2024-09-11T01:57:26.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe582d9d640 1 -- 172.21.15.12:0/3224759730 >> v2:172.21.15.71:3300/0 conn(0x7fe58413d380 msgr2=0x7fe58413f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:26.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe582d9d640 1 --2- 172.21.15.12:0/3224759730 >> v2:172.21.15.71:3300/0 conn(0x7fe58413d380 0x7fe58413f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:26.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe582d9d640 1 -- 172.21.15.12:0/3224759730 >> v1:172.21.15.12:6789/0 conn(0x7fe584138230 legacy=0x7fe584138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:26.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe582d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:57:26.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe582d9d640 1 monclient: found mon.noname-a 2024-09-11T01:57:26.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe582d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:26.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe582d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:57:26.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.662+0000 7fe582d9d640 1 -- 172.21.15.12:0/3224759730 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe58409c780 con 0x7fe584137e50 2024-09-11T01:57:26.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.665+0000 7fe582d9d640 1 -- 172.21.15.12:0/3224759730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe584137e50 msgr2=0x7fe58409d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:26.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.665+0000 7fe582d9d640 1 --2- 172.21.15.12:0/3224759730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe584137e50 0x7fe58409d160 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe5840ab580 tx=0x7fe56c029fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4893a03005a62eeb server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:26.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.665+0000 7fe581d9b640 1 -- 172.21.15.12:0/3224759730 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56c037070 con 0x7fe584137e50 2024-09-11T01:57:26.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.665+0000 7fe581d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:26.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.665+0000 7fe581d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:57:26.675 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.665+0000 7fe581d9b640 10 monclient: dump: 2024-09-11T01:57:26.675 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:26.675 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:26.675 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:26.676 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:26.676 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:26.676 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:26.676 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:26.676 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:26.676 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:26.676 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:26.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.665+0000 7fe581d9b640 10 monclient: _finish_auth 0 2024-09-11T01:57:26.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.665+0000 7fe581d9b640 10 monclient: _check_auth_tickets 2024-09-11T01:57:26.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.665+0000 7fe581d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:26.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.665+0000 7fe581d9b640 1 -- 172.21.15.12:0/3224759730 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe56c032440 con 0x7fe584137e50 2024-09-11T01:57:26.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.665+0000 7fe589060640 5 monclient: authenticate success, global_id 294348 2024-09-11T01:57:26.676 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.665+0000 7fe589060640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:57:26.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.665+0000 7fe581d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:26.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.665+0000 7fe589060640 10 monclient: get_monmap_and_config success 2024-09-11T01:57:26.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.665+0000 7fe589060640 4 set_mon_vals no callback set 2024-09-11T01:57:26.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.665+0000 7fe581d9b640 1 -- 172.21.15.12:0/3224759730 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56c032810 con 0x7fe584137e50 2024-09-11T01:57:26.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.665+0000 7fe581d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:26.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.665+0000 7fe581d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:26.677 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.665+0000 7fe581d9b640 10 monclient: dump: 2024-09-11T01:57:26.677 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:26.677 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:26.677 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:26.677 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:26.677 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:26.677 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:26.677 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:26.678 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:26.678 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:26.678 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:26.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.666+0000 7fe589060640 10 monclient: shutdown 2024-09-11T01:57:26.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.666+0000 7fe589060640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:26.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.666+0000 7fe589060640 1 -- 172.21.15.12:0/3224759730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe584137e50 msgr2=0x7fe58409d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:26.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.666+0000 7fe589060640 1 --2- 172.21.15.12:0/3224759730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe584137e50 0x7fe58409d160 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe5840ab580 tx=0x7fe56c029fb0 comp rx=0 tx=0).stop 2024-09-11T01:57:26.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.666+0000 7fe589060640 1 -- 172.21.15.12:0/3224759730 shutdown_connections 2024-09-11T01:57:26.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.666+0000 7fe589060640 1 --2- 172.21.15.12:0/3224759730 >> v2:172.21.15.71:3300/0 conn(0x7fe58413d380 0x7fe58413f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:26.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.666+0000 7fe589060640 1 --2- 172.21.15.12:0/3224759730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe584137e50 0x7fe58409d160 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:26.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.666+0000 7fe589060640 1 -- 172.21.15.12:0/3224759730 >> 172.21.15.12:0/3224759730 conn(0x7fe5840a7020 msgr2=0x7fe5840a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:26.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.666+0000 7fe589060640 1 -- 172.21.15.12:0/3224759730 shutdown_connections 2024-09-11T01:57:26.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.666+0000 7fe589060640 1 -- 172.21.15.12:0/3224759730 wait complete. 2024-09-11T01:57:26.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.666+0000 7fe589060640 10 monclient: build_initial_monmap 2024-09-11T01:57:26.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.667+0000 7fe589060640 10 monclient: monmap: 2024-09-11T01:57:26.678 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:57:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:57:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:57:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:57:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:57:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:57:26.679 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.667+0000 7fe589060640 1 Processor -- start 2024-09-11T01:57:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.667+0000 7fe589060640 1 -- start start 2024-09-11T01:57:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.667+0000 7fe589060640 10 monclient: init 2024-09-11T01:57:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.667+0000 7fe589060640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.667+0000 7fe589060640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T01:57:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.667+0000 7fe589060640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe584137e50 0x7fe5841cd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.667+0000 7fe589060640 10 monclient(hunting): picked mon.noname-a con 0x7fe584137e50 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:57:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.667+0000 7fe589060640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe584138230 0x7fe5840a40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.667+0000 7fe589060640 10 monclient(hunting): picked mon.noname-b con 0x7fe584138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:57:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.667+0000 7fe589060640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe58413d380 0x7fe5840a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.667+0000 7fe589060640 10 monclient(hunting): picked mon.noname-c con 0x7fe58413d380 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.667+0000 7fe589060640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.667+0000 7fe589060640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe58406f890 con 0x7fe584137e50 2024-09-11T01:57:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.667+0000 7fe589060640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.667+0000 7fe589060640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe58406f710 con 0x7fe584138230 2024-09-11T01:57:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.667+0000 7fe589060640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.667+0000 7fe589060640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fe58413fcb0 con 0x7fe58413d380 2024-09-11T01:57:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.667+0000 7fe589060640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.667+0000 7fe589060640 10 monclient(hunting): authenticate will time out at 2453.109996s 2024-09-11T01:57:26.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe582d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe584137e50 0x7fe5841cd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:26.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe58359e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe58413d380 0x7fe5840a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:26.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe58359e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe58413d380 0x7fe5840a45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46518/0 (socket says 172.21.15.12:46518) 2024-09-11T01:57:26.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe582d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe584137e50 0x7fe5841cd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54122/0 (socket says 172.21.15.12:54122) 2024-09-11T01:57:26.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe58359e640 1 -- 172.21.15.12:0/2742353182 learned_addr learned my addr 172.21.15.12:0/2742353182 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:26.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe58359e640 10 monclient(hunting): get_auth_request con 0x7fe58413d380 auth_method 0 2024-09-11T01:57:26.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe58359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:26.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe58359e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:26.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe58359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:26.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe582d9d640 10 monclient(hunting): get_auth_request con 0x7fe584137e50 auth_method 0 2024-09-11T01:57:26.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe582d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:26.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe582d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:26.681 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe582d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:26.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe582d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:26.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe582d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:26.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe582d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:26.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe58359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:26.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe58359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:26.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe58359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:26.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe58259c640 1 --2- 172.21.15.12:0/2742353182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe584138230 0x7fe5840a40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:26.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe58259c640 10 monclient(hunting): get_auth_request con 0x7fe584138230 auth_method 0 2024-09-11T01:57:26.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe58259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:26.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe58259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:26.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe58259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:26.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe582d9d640 10 monclient(hunting): handle_auth_done global_id 294351 payload 824 2024-09-11T01:57:26.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe582d9d640 1 -- 172.21.15.12:0/2742353182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe58413d380 msgr2=0x7fe5840a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:26.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe582d9d640 1 --2- 172.21.15.12:0/2742353182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe58413d380 0x7fe5840a45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe582d9d640 1 -- 172.21.15.12:0/2742353182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe584138230 msgr2=0x7fe5840a40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe582d9d640 1 --2- 172.21.15.12:0/2742353182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe584138230 0x7fe5840a40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe582d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:57:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe582d9d640 1 monclient: found mon.noname-a 2024-09-11T01:57:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe582d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe582d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:57:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe582d9d640 1 -- 172.21.15.12:0/2742353182 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5840a1710 con 0x7fe584137e50 2024-09-11T01:57:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe58359e640 1 --2- 172.21.15.12:0/2742353182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe58413d380 0x7fe5840a45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:57:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.668+0000 7fe58259c640 1 --2- 172.21.15.12:0/2742353182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe584138230 0x7fe5840a40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:57:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.670+0000 7fe582d9d640 1 -- 172.21.15.12:0/2742353182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe584137e50 msgr2=0x7fe5841cd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.670+0000 7fe582d9d640 1 --2- 172.21.15.12:0/2742353182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe584137e50 0x7fe5841cd120 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe56c031000 tx=0x7fe56c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.670+0000 7fe57b7fe640 1 -- 172.21.15.12:0/2742353182 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56c037040 con 0x7fe584137e50 2024-09-11T01:57:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.670+0000 7fe57b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:26.683 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.670+0000 7fe57b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:57:26.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.670+0000 7fe57b7fe640 10 monclient: dump: 2024-09-11T01:57:26.684 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:26.684 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:26.684 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:26.684 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:26.684 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:26.684 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:26.684 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:26.684 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:26.684 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:26.684 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:26.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.670+0000 7fe57b7fe640 10 monclient: _finish_auth 0 2024-09-11T01:57:26.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.670+0000 7fe57b7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:57:26.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.670+0000 7fe57b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:26.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.670+0000 7fe57b7fe640 1 -- 172.21.15.12:0/2742353182 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe56c042070 con 0x7fe584137e50 2024-09-11T01:57:26.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.670+0000 7fe589060640 5 monclient: authenticate success, global_id 294351 2024-09-11T01:57:26.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.670+0000 7fe589060640 10 monclient: _renew_subs 2024-09-11T01:57:26.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.670+0000 7fe589060640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:26.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.670+0000 7fe589060640 1 -- 172.21.15.12:0/2742353182 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe58413b500 con 0x7fe584137e50 2024-09-11T01:57:26.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.670+0000 7fe57b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:26.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.670+0000 7fe57b7fe640 1 -- 172.21.15.12:0/2742353182 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56c03bb20 con 0x7fe584137e50 2024-09-11T01:57:26.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.670+0000 7fe57b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:26.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.670+0000 7fe57b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:26.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.670+0000 7fe57b7fe640 10 monclient: dump: 2024-09-11T01:57:26.685 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:26.685 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:26.685 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:26.685 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:26.686 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:26.686 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:26.686 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:26.686 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:26.686 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:26.686 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:26.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.670+0000 7fe580d99640 4 set_mon_vals no callback set 2024-09-11T01:57:26.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.670+0000 7fe589060640 10 monclient: _renew_subs 2024-09-11T01:57:26.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.670+0000 7fe589060640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:26.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.670+0000 7fe589060640 1 -- 172.21.15.12:0/2742353182 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5841cd660 con 0x7fe584137e50 2024-09-11T01:57:26.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.671+0000 7fe589060640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:26.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.671+0000 7fe589060640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:26.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.671+0000 7fe589060640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:26.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.671+0000 7fe57b7fe640 1 -- 172.21.15.12:0/2742353182 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fe56c03a030 con 0x7fe584137e50 2024-09-11T01:57:26.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.671+0000 7fe589060640 1 -- 172.21.15.12:0/2742353182 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe58409d160 con 0x7fe584137e50 2024-09-11T01:57:26.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.672+0000 7fe57b7fe640 1 --2- 172.21.15.12:0/2742353182 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe56403e210 0x7fe5640406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:26.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.672+0000 7fe57b7fe640 1 -- 172.21.15.12:0/2742353182 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(923..923 src has 889..923) ==== 4422+0+0 (secure 0 0 0) 0x7fe56c073120 con 0x7fe584137e50 2024-09-11T01:57:26.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.672+0000 7fe58259c640 1 --2- 172.21.15.12:0/2742353182 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe56403e210 0x7fe5640406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:26.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.672+0000 7fe58259c640 10 monclient: get_auth_request con 0x7fe56403e210 auth_method 0 2024-09-11T01:57:26.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.674+0000 7fe58259c640 1 --2- 172.21.15.12:0/2742353182 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe56403e210 0x7fe5640406d0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fe58409ded0 tx=0x7fe574009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:26.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.676+0000 7fe57b7fe640 1 -- 172.21.15.12:0/2742353182 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe56c071c80 con 0x7fe584137e50 2024-09-11T01:57:26.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.676+0000 7fe57b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:26.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.676+0000 7fe57b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:57:26.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.858+0000 7fe589060640 1 -- 172.21.15.12:0/2742353182 --> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fe5840a5890 con 0x7fe56403e210 2024-09-11T01:57:26.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.859+0000 7fe57b7fe640 1 -- 172.21.15.12:0/2742353182 <== mgr.4101 v2:172.21.15.71:6816/3189259519 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe5840a5890 con 0x7fe56403e210 2024-09-11T01:57:26.860 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 0 to deep-scrub 2024-09-11T01:57:26.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.861+0000 7fe589060640 1 -- 172.21.15.12:0/2742353182 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe56403e210 msgr2=0x7fe5640406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:26.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.861+0000 7fe589060640 1 --2- 172.21.15.12:0/2742353182 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe56403e210 0x7fe5640406d0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fe58409ded0 tx=0x7fe574009040 comp rx=0 tx=0).stop 2024-09-11T01:57:26.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.861+0000 7fe589060640 10 monclient: shutdown 2024-09-11T01:57:26.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.861+0000 7fe589060640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:26.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.861+0000 7fe589060640 1 -- 172.21.15.12:0/2742353182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe584137e50 msgr2=0x7fe5841cd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:26.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.861+0000 7fe589060640 1 --2- 172.21.15.12:0/2742353182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe584137e50 0x7fe5841cd120 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe56c031000 tx=0x7fe56c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:57:26.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.861+0000 7fe589060640 1 -- 172.21.15.12:0/2742353182 shutdown_connections 2024-09-11T01:57:26.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.861+0000 7fe589060640 1 --2- 172.21.15.12:0/2742353182 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe58413d380 0x7fe5840a45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:26.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.861+0000 7fe589060640 1 --2- 172.21.15.12:0/2742353182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe584138230 0x7fe5840a40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:26.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.861+0000 7fe589060640 1 --2- 172.21.15.12:0/2742353182 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fe56403e210 0x7fe5640406d0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:26.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.861+0000 7fe589060640 1 --2- 172.21.15.12:0/2742353182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe584137e50 0x7fe5841cd120 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:26.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.861+0000 7fe589060640 1 -- 172.21.15.12:0/2742353182 >> 172.21.15.12:0/2742353182 conn(0x7fe5840a7020 msgr2=0x7fe584136ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:26.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.861+0000 7fe589060640 1 -- 172.21.15.12:0/2742353182 shutdown_connections 2024-09-11T01:57:26.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.861+0000 7fe589060640 1 -- 172.21.15.12:0/2742353182 wait complete. 2024-09-11T01:57:26.873 INFO:tasks.ceph:Scrubbing osd.1 2024-09-11T01:57:26.873 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-11T01:57:26.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.976+0000 7ff6d426e640 10 monclient: get_monmap_and_config 2024-09-11T01:57:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.976+0000 7ff6d426e640 10 monclient: build_initial_monmap 2024-09-11T01:57:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.976+0000 7ff6d426e640 10 monclient: monmap: 2024-09-11T01:57:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:57:26.977226+0000 2024-09-11T01:57:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:57:26.977226+0000 2024-09-11T01:57:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:57:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:57:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:57:26.978 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:57:26.979 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:57:26.979 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:26.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.977+0000 7ff6d426e640 1 Processor -- start 2024-09-11T01:57:26.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.977+0000 7ff6d426e640 1 -- start start 2024-09-11T01:57:26.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.977+0000 7ff6d426e640 10 monclient: init 2024-09-11T01:57:26.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.977+0000 7ff6d426e640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:26.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.977+0000 7ff6d426e640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-11T01:57:26.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.977+0000 7ff6d426e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff6cc069890 addr v1:172.21.15.71:6789/0 2024-09-11T01:57:26.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.977+0000 7ff6d426e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff6cc0a2690 0x7ff6cc0a2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:26.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.977+0000 7ff6d426e640 10 monclient(hunting): picked mon.noname-c con 0x7ff6cc0a2690 addr v2:172.21.15.71:3300/0 2024-09-11T01:57:26.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.977+0000 7ff6d426e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6cc13add0 0x7ff6cc13d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:26.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.977+0000 7ff6d426e640 10 monclient(hunting): picked mon.noname-b con 0x7ff6cc13add0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:26.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.977+0000 7ff6d426e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6cc09cb80 con 0x7ff6cc069890 2024-09-11T01:57:26.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.977+0000 7ff6d426e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:26.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.977+0000 7ff6d426e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7ff6cc069c70 con 0x7ff6cc0a2690 2024-09-11T01:57:26.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.977+0000 7ff6d426e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:26.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.977+0000 7ff6d426e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff6cc09c8a0 con 0x7ff6cc13add0 2024-09-11T01:57:26.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.977+0000 7ff6d426e640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:26.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.977+0000 7ff6d426e640 10 monclient(hunting): authenticate will time out at 2453.419838s 2024-09-11T01:57:26.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.977+0000 7ff6d280b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6cc13add0 0x7ff6cc13d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:26.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.977+0000 7ff6d280b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6cc13add0 0x7ff6cc13d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46526/0 (socket says 172.21.15.12:46526) 2024-09-11T01:57:26.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.977+0000 7ff6d280b640 1 -- 172.21.15.12:0/3905563456 learned_addr learned my addr 172.21.15.12:0/3905563456 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:26.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.977+0000 7ff6d280b640 10 monclient(hunting): get_auth_request con 0x7ff6cc13add0 auth_method 0 2024-09-11T01:57:26.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.977+0000 7ff6d280b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:26.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.977+0000 7ff6d280b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:26.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.977+0000 7ff6d280b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:26.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.978+0000 7ff6d280b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:26.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.978+0000 7ff6d280b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:26.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.978+0000 7ff6d280b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:26.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.978+0000 7ff6d1809640 1 --2- 172.21.15.12:0/3905563456 >> v2:172.21.15.71:3300/0 conn(0x7ff6cc0a2690 0x7ff6cc0a2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:26.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.978+0000 7ff6d1809640 10 monclient(hunting): get_auth_request con 0x7ff6cc0a2690 auth_method 0 2024-09-11T01:57:26.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.978+0000 7ff6d1809640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:26.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.978+0000 7ff6d1809640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:26.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.978+0000 7ff6d1809640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:26.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.978+0000 7ff6d280b640 10 monclient(hunting): handle_auth_done global_id 304247 payload 274 2024-09-11T01:57:26.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.978+0000 7ff6d280b640 1 -- 172.21.15.12:0/3905563456 >> v2:172.21.15.71:3300/0 conn(0x7ff6cc0a2690 msgr2=0x7ff6cc0a2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:26.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.978+0000 7ff6d280b640 1 --2- 172.21.15.12:0/3905563456 >> v2:172.21.15.71:3300/0 conn(0x7ff6cc0a2690 0x7ff6cc0a2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:26.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.978+0000 7ff6d280b640 1 -- 172.21.15.12:0/3905563456 >> v1:172.21.15.71:6789/0 conn(0x7ff6cc069890 legacy=0x7ff6cc135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:26.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.978+0000 7ff6d280b640 10 monclient: _finish_hunting 0 2024-09-11T01:57:26.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.978+0000 7ff6d280b640 1 monclient: found mon.noname-b 2024-09-11T01:57:26.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.978+0000 7ff6d280b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:26.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.978+0000 7ff6d280b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:57:26.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.978+0000 7ff6d280b640 1 -- 172.21.15.12:0/3905563456 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6cc09ce60 con 0x7ff6cc13add0 2024-09-11T01:57:26.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d280b640 1 -- 172.21.15.12:0/3905563456 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6cc13add0 msgr2=0x7ff6cc13d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:26.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d280b640 1 --2- 172.21.15.12:0/3905563456 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6cc13add0 0x7ff6cc13d1c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4009e50 tx=0x7ff6c402a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93a8200db702d10d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:26.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d1008640 1 -- 172.21.15.12:0/3905563456 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4036070 con 0x7ff6cc13add0 2024-09-11T01:57:26.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d1008640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:26.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d1008640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:57:26.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d1008640 10 monclient: dump: 2024-09-11T01:57:26.990 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:26.990 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:26.990 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:26.990 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:26.990 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:26.990 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:26.990 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:26.990 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:26.990 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:26.991 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:26.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d1008640 10 monclient: _finish_auth 0 2024-09-11T01:57:26.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d1008640 10 monclient: _check_auth_tickets 2024-09-11T01:57:26.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d1008640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:26.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d1008640 1 -- 172.21.15.12:0/3905563456 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6c4007800 con 0x7ff6cc13add0 2024-09-11T01:57:26.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d426e640 5 monclient: authenticate success, global_id 304247 2024-09-11T01:57:26.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d426e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:57:26.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d1008640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:26.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d1008640 1 -- 172.21.15.12:0/3905563456 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4007bd0 con 0x7ff6cc13add0 2024-09-11T01:57:26.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d1008640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:26.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d1008640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:57:26.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d1008640 10 monclient: dump: 2024-09-11T01:57:26.991 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:26.991 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:26.992 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:26.992 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:26.992 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:26.992 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:26.992 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:26.992 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:26.992 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:26.992 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:26.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d426e640 10 monclient: get_monmap_and_config success 2024-09-11T01:57:26.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d426e640 4 set_mon_vals no callback set 2024-09-11T01:57:26.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d426e640 10 monclient: shutdown 2024-09-11T01:57:26.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d426e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:26.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d426e640 1 -- 172.21.15.12:0/3905563456 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6cc13add0 msgr2=0x7ff6cc13d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:26.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d426e640 1 --2- 172.21.15.12:0/3905563456 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6cc13add0 0x7ff6cc13d1c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4009e50 tx=0x7ff6c402a5f0 comp rx=0 tx=0).stop 2024-09-11T01:57:26.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d426e640 1 -- 172.21.15.12:0/3905563456 shutdown_connections 2024-09-11T01:57:26.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d426e640 1 --2- 172.21.15.12:0/3905563456 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6cc13add0 0x7ff6cc13d1c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:26.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d426e640 1 --2- 172.21.15.12:0/3905563456 >> v2:172.21.15.71:3300/0 conn(0x7ff6cc0a2690 0x7ff6cc0a2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:26.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d426e640 1 -- 172.21.15.12:0/3905563456 >> 172.21.15.12:0/3905563456 conn(0x7ff6cc068950 msgr2=0x7ff6cc068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:26.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d426e640 1 -- 172.21.15.12:0/3905563456 shutdown_connections 2024-09-11T01:57:26.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.981+0000 7ff6d426e640 1 -- 172.21.15.12:0/3905563456 wait complete. 2024-09-11T01:57:26.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.982+0000 7ff6d426e640 10 monclient: build_initial_monmap 2024-09-11T01:57:26.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.982+0000 7ff6d426e640 10 monclient: monmap: 2024-09-11T01:57:26.993 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:26.993 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:57:26.993 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:57:26.994 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:57:26.994 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:26.994 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:26.994 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:57:26.994 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:57:26.994 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:57:26.994 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:26.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.982+0000 7ff6d426e640 1 Processor -- start 2024-09-11T01:57:26.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.982+0000 7ff6d426e640 1 -- start start 2024-09-11T01:57:26.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.982+0000 7ff6d426e640 10 monclient: init 2024-09-11T01:57:26.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d426e640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:26.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d426e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T01:57:26.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d426e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6cc069890 0x7ff6cc12f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:26.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d426e640 10 monclient(hunting): picked mon.noname-c con 0x7ff6cc069890 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:26.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d426e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6cc0a2690 0x7ff6cc12f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:26.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d426e640 10 monclient(hunting): picked mon.noname-a con 0x7ff6cc0a2690 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:57:26.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d426e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6cc13add0 0x7ff6cc138ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:26.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d426e640 10 monclient(hunting): picked mon.noname-b con 0x7ff6cc13add0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:57:26.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d426e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:26.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d426e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc13d910 con 0x7ff6cc0a2690 2024-09-11T01:57:26.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d426e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:26.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d426e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc13da90 con 0x7ff6cc13add0 2024-09-11T01:57:26.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d426e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:26.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d426e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff6cc13dc10 con 0x7ff6cc069890 2024-09-11T01:57:26.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d426e640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:26.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d426e640 10 monclient(hunting): authenticate will time out at 2453.425305s 2024-09-11T01:57:26.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d200a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6cc069890 0x7ff6cc12f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:26.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d200a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6cc069890 0x7ff6cc12f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46534/0 (socket says 172.21.15.12:46534) 2024-09-11T01:57:26.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d200a640 1 -- 172.21.15.12:0/4293969925 learned_addr learned my addr 172.21.15.12:0/4293969925 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:26.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d200a640 10 monclient(hunting): get_auth_request con 0x7ff6cc069890 auth_method 0 2024-09-11T01:57:26.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d200a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:26.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d200a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:26.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d200a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:26.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d1809640 1 --2- 172.21.15.12:0/4293969925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6cc0a2690 0x7ff6cc12f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:26.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d1809640 10 monclient(hunting): get_auth_request con 0x7ff6cc0a2690 auth_method 0 2024-09-11T01:57:26.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d1809640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:26.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d1809640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:26.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d1809640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:26.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d200a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:26.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d200a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:26.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d200a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:26.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d1809640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:26.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d1809640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:26.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d1809640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:26.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d280b640 1 --2- 172.21.15.12:0/4293969925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6cc13add0 0x7ff6cc138ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:26.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d200a640 10 monclient(hunting): handle_auth_done global_id 304250 payload 824 2024-09-11T01:57:26.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d200a640 1 -- 172.21.15.12:0/4293969925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6cc13add0 msgr2=0x7ff6cc138ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:26.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d200a640 1 --2- 172.21.15.12:0/4293969925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6cc13add0 0x7ff6cc138ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:26.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d200a640 1 -- 172.21.15.12:0/4293969925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6cc0a2690 msgr2=0x7ff6cc12f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:26.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d200a640 1 --2- 172.21.15.12:0/4293969925 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff6cc0a2690 0x7ff6cc12f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:26.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d200a640 10 monclient: _finish_hunting 0 2024-09-11T01:57:26.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d200a640 1 monclient: found mon.noname-c 2024-09-11T01:57:26.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d200a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:26.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d200a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:57:26.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.983+0000 7ff6d200a640 1 -- 172.21.15.12:0/4293969925 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6cc09ef00 con 0x7ff6cc069890 2024-09-11T01:57:26.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.985+0000 7ff6d200a640 1 -- 172.21.15.12:0/4293969925 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6cc069890 msgr2=0x7ff6cc12f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:26.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.985+0000 7ff6d200a640 1 --2- 172.21.15.12:0/4293969925 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6cc069890 0x7ff6cc12f0d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800c460 tx=0x7ff6c800c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:26.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.986+0000 7ff6c2ffd640 1 -- 172.21.15.12:0/4293969925 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c8016020 con 0x7ff6cc069890 2024-09-11T01:57:26.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.986+0000 7ff6c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:26.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.986+0000 7ff6c2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:57:26.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.986+0000 7ff6c2ffd640 10 monclient: dump: 2024-09-11T01:57:26.998 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:26.998 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:26.998 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:26.998 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:26.998 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:26.998 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:26.999 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.986+0000 7ff6c2ffd640 10 monclient: _finish_auth 0 2024-09-11T01:57:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.986+0000 7ff6c2ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:57:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.986+0000 7ff6c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.986+0000 7ff6c2ffd640 1 -- 172.21.15.12:0/4293969925 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6c8012a80 con 0x7ff6cc069890 2024-09-11T01:57:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.986+0000 7ff6d426e640 5 monclient: authenticate success, global_id 304250 2024-09-11T01:57:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.986+0000 7ff6d426e640 10 monclient: _renew_subs 2024-09-11T01:57:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.986+0000 7ff6d426e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.986+0000 7ff6d426e640 1 -- 172.21.15.12:0/4293969925 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6cc09f1e0 con 0x7ff6cc069890 2024-09-11T01:57:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.986+0000 7ff6c2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.986+0000 7ff6c2ffd640 1 -- 172.21.15.12:0/4293969925 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c8011380 con 0x7ff6cc069890 2024-09-11T01:57:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.986+0000 7ff6c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.986+0000 7ff6c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:57:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.986+0000 7ff6c2ffd640 10 monclient: dump: 2024-09-11T01:57:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:27.001 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:27.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.986+0000 7ff6c3fff640 4 set_mon_vals no callback set 2024-09-11T01:57:27.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.986+0000 7ff6d426e640 10 monclient: _renew_subs 2024-09-11T01:57:27.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.986+0000 7ff6d426e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:27.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.986+0000 7ff6d426e640 1 -- 172.21.15.12:0/4293969925 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6cc09ec20 con 0x7ff6cc069890 2024-09-11T01:57:27.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.986+0000 7ff6d426e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:27.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.986+0000 7ff6d426e640 1 -- 172.21.15.12:0/4293969925 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff6cc135b50 con 0x7ff6cc069890 2024-09-11T01:57:27.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.986+0000 7ff6c2ffd640 1 -- 172.21.15.12:0/4293969925 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff6c8011560 con 0x7ff6cc069890 2024-09-11T01:57:27.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.987+0000 7ff6c2ffd640 1 --2- 172.21.15.12:0/4293969925 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff6b003e210 0x7ff6b00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.987+0000 7ff6c2ffd640 1 -- 172.21.15.12:0/4293969925 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(923..923 src has 889..923) ==== 4422+0+0 (secure 0 0 0) 0x7ff6c8052900 con 0x7ff6cc069890 2024-09-11T01:57:27.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.987+0000 7ff6c2ffd640 1 --2- 172.21.15.12:0/4293969925 >> [v2:172.21.15.12:6808/3563357280,v1:172.21.15.12:6809/3563357280] conn(0x7ff6b0044100 0x7ff6b00464f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.987+0000 7ff6c2ffd640 1 -- 172.21.15.12:0/4293969925 --> [v2:172.21.15.12:6808/3563357280,v1:172.21.15.12:6809/3563357280] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff6b0046ba0 con 0x7ff6b0044100 2024-09-11T01:57:27.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.987+0000 7ff6c2ffd640 1 -- 172.21.15.12:0/4293969925 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_get_version_reply(handle=1 version=923) ==== 24+0+0 (secure 0 0 0) 0x7ff6c80572a0 con 0x7ff6cc069890 2024-09-11T01:57:27.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.987+0000 7ff6c2ffd640 10 monclient: handle_get_version_reply finishing 1 version 923 2024-09-11T01:57:27.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.987+0000 7ff6d280b640 1 --2- 172.21.15.12:0/4293969925 >> [v2:172.21.15.12:6808/3563357280,v1:172.21.15.12:6809/3563357280] conn(0x7ff6b0044100 0x7ff6b00464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.987+0000 7ff6d1809640 1 --2- 172.21.15.12:0/4293969925 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff6b003e210 0x7ff6b00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.987+0000 7ff6d1809640 10 monclient: get_auth_request con 0x7ff6b003e210 auth_method 0 2024-09-11T01:57:27.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.987+0000 7ff6d280b640 10 monclient: get_auth_request con 0x7ff6b0044100 auth_method 0 2024-09-11T01:57:27.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.990+0000 7ff6d1809640 1 --2- 172.21.15.12:0/4293969925 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff6b003e210 0x7ff6b00406d0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc09e870 tx=0x7ff6bc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:27.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.991+0000 7ff6d280b640 1 --2- 172.21.15.12:0/4293969925 >> [v2:172.21.15.12:6808/3563357280,v1:172.21.15.12:6809/3563357280] conn(0x7ff6b0044100 0x7ff6b00464f0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:27.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:26.992+0000 7ff6c2ffd640 1 -- 172.21.15.12:0/4293969925 <== osd.1 v2:172.21.15.12:6808/3563357280 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff6b0046ba0 con 0x7ff6b0044100 2024-09-11T01:57:27.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.005+0000 7ff6d426e640 1 -- 172.21.15.12:0/4293969925 --> [v2:172.21.15.12:6808/3563357280,v1:172.21.15.12:6809/3563357280] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff6cc074b60 con 0x7ff6b0044100 2024-09-11T01:57:27.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.005+0000 7ff6c2ffd640 1 -- 172.21.15.12:0/4293969925 <== osd.1 v2:172.21.15.12:6808/3563357280 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff6cc074b60 con 0x7ff6b0044100 2024-09-11T01:57:27.007 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:57:27.007 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T01:57:27.007 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:57:27.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.005+0000 7ff6d426e640 1 -- 172.21.15.12:0/4293969925 >> [v2:172.21.15.12:6808/3563357280,v1:172.21.15.12:6809/3563357280] conn(0x7ff6b0044100 msgr2=0x7ff6b00464f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:27.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.005+0000 7ff6d426e640 1 --2- 172.21.15.12:0/4293969925 >> [v2:172.21.15.12:6808/3563357280,v1:172.21.15.12:6809/3563357280] conn(0x7ff6b0044100 0x7ff6b00464f0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.005+0000 7ff6d426e640 1 -- 172.21.15.12:0/4293969925 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff6b003e210 msgr2=0x7ff6b00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:27.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.005+0000 7ff6d426e640 1 --2- 172.21.15.12:0/4293969925 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff6b003e210 0x7ff6b00406d0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc09e870 tx=0x7ff6bc008040 comp rx=0 tx=0).stop 2024-09-11T01:57:27.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.005+0000 7ff6d426e640 10 monclient: shutdown 2024-09-11T01:57:27.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.005+0000 7ff6d426e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:27.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.005+0000 7ff6d426e640 1 -- 172.21.15.12:0/4293969925 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6cc069890 msgr2=0x7ff6cc12f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:27.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.005+0000 7ff6d426e640 1 --2- 172.21.15.12:0/4293969925 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff6cc069890 0x7ff6cc12f0d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800c460 tx=0x7ff6c800c920 comp rx=0 tx=0).stop 2024-09-11T01:57:27.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.005+0000 7ff6d280b640 1 -- 172.21.15.12:0/4293969925 reap_dead start 2024-09-11T01:57:27.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.006+0000 7ff6d426e640 1 -- 172.21.15.12:0/4293969925 shutdown_connections 2024-09-11T01:57:27.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.006+0000 7ff6d426e640 1 -- 172.21.15.12:0/4293969925 >> 172.21.15.12:0/4293969925 conn(0x7ff6cc068950 msgr2=0x7ff6cc0a5dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:27.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.006+0000 7ff6d426e640 1 -- 172.21.15.12:0/4293969925 shutdown_connections 2024-09-11T01:57:27.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.006+0000 7ff6d426e640 1 -- 172.21.15.12:0/4293969925 wait complete. 2024-09-11T01:57:27.017 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-11T01:57:27.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.120+0000 7f03dc0ef640 10 monclient: get_monmap_and_config 2024-09-11T01:57:27.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.120+0000 7f03dc0ef640 10 monclient: build_initial_monmap 2024-09-11T01:57:27.122 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.120+0000 7f03dc0ef640 10 monclient: monmap: 2024-09-11T01:57:27.122 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:27.122 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:27.122 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:57:27.121054+0000 2024-09-11T01:57:27.122 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:57:27.121054+0000 2024-09-11T01:57:27.122 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:27.122 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:27.122 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:57:27.122 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:57:27.122 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:57:27.123 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:57:27.123 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:57:27.123 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:27.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.120+0000 7f03dc0ef640 1 Processor -- start 2024-09-11T01:57:27.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03dc0ef640 1 -- start start 2024-09-11T01:57:27.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03dc0ef640 10 monclient: init 2024-09-11T01:57:27.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03dc0ef640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:27.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03dc0ef640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-11T01:57:27.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03dc0ef640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f03d40a2d40 0x7f03d406cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03dc0ef640 10 monclient(hunting): picked mon.noname-a con 0x7f03d40a2d40 addr v2:172.21.15.12:3300/0 2024-09-11T01:57:27.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03dc0ef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f03d406d090 addr v1:172.21.15.12:6789/0 2024-09-11T01:57:27.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03dc0ef640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f03d406d4a0 0x7f03d413fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03dc0ef640 10 monclient(hunting): picked mon.noname-c con 0x7f03d406d4a0 addr v2:172.21.15.71:3300/0 2024-09-11T01:57:27.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03dc0ef640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03d409aed0 con 0x7f03d406d090 2024-09-11T01:57:27.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03dc0ef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:27.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03dc0ef640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f03d406dd10 con 0x7f03d40a2d40 2024-09-11T01:57:27.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03dc0ef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:27.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03dc0ef640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f03d409bcf0 con 0x7f03d406d4a0 2024-09-11T01:57:27.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03dc0ef640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:27.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03dc0ef640 10 monclient(hunting): authenticate will time out at 2453.563583s 2024-09-11T01:57:27.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03d9e8b640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f03d40a2d40 0x7f03d406cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03d968a640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f03d406d090 0x7f03d413c4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33896/0 (socket says 172.21.15.12:33896) 2024-09-11T01:57:27.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03d968a640 1 -- 172.21.15.12:0/1791364964 learned_addr learned my addr 172.21.15.12:0/1791364964 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:57:27.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03d9e8b640 10 monclient(hunting): get_auth_request con 0x7f03d40a2d40 auth_method 0 2024-09-11T01:57:27.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03d9e8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:27.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03d9e8b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:27.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03d9e8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:27.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03d9e8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:27.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03d9e8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:27.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03d9e8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:27.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03da68c640 1 --2- 172.21.15.12:0/1791364964 >> v2:172.21.15.71:3300/0 conn(0x7f03d406d4a0 0x7f03d413fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03da68c640 10 monclient(hunting): get_auth_request con 0x7f03d406d4a0 auth_method 0 2024-09-11T01:57:27.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03da68c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:27.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03da68c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:27.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03da68c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:27.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03da68c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:27.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03da68c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:27.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.121+0000 7f03da68c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:27.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.122+0000 7f03d9e8b640 10 monclient(hunting): handle_auth_done global_id 294357 payload 274 2024-09-11T01:57:27.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.122+0000 7f03d9e8b640 1 -- 172.21.15.12:0/1791364964 >> v2:172.21.15.71:3300/0 conn(0x7f03d406d4a0 msgr2=0x7f03d413fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:27.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.122+0000 7f03d9e8b640 1 --2- 172.21.15.12:0/1791364964 >> v2:172.21.15.71:3300/0 conn(0x7f03d406d4a0 0x7f03d413fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.122+0000 7f03d9e8b640 1 -- 172.21.15.12:0/1791364964 >> v1:172.21.15.12:6789/0 conn(0x7f03d406d090 legacy=0x7f03d413c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:27.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.122+0000 7f03d9e8b640 10 monclient: _finish_hunting 0 2024-09-11T01:57:27.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.122+0000 7f03d9e8b640 1 monclient: found mon.noname-a 2024-09-11T01:57:27.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.122+0000 7f03d9e8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:27.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.122+0000 7f03d9e8b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:57:27.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.122+0000 7f03d9e8b640 1 -- 172.21.15.12:0/1791364964 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03d40a3920 con 0x7f03d40a2d40 2024-09-11T01:57:27.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.124+0000 7f03d9e8b640 1 -- 172.21.15.12:0/1791364964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03d40a2d40 msgr2=0x7f03d406cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:27.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.124+0000 7f03d9e8b640 1 --2- 172.21.15.12:0/1791364964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03d40a2d40 0x7f03d406cb50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f03d409be70 tx=0x7f03cc029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91388678e7292a7b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:27.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03d8e89640 1 -- 172.21.15.12:0/1791364964 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03cc037070 con 0x7f03d40a2d40 2024-09-11T01:57:27.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03d8e89640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:27.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03d8e89640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:57:27.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03d8e89640 10 monclient: dump: 2024-09-11T01:57:27.134 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:27.134 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:27.134 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.134 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.134 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:27.134 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:27.134 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:27.135 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:27.135 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:27.135 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:27.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03d8e89640 10 monclient: _finish_auth 0 2024-09-11T01:57:27.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03d8e89640 10 monclient: _check_auth_tickets 2024-09-11T01:57:27.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03d8e89640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:27.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03d8e89640 1 -- 172.21.15.12:0/1791364964 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03cc032440 con 0x7f03d40a2d40 2024-09-11T01:57:27.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03dc0ef640 5 monclient: authenticate success, global_id 294357 2024-09-11T01:57:27.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03dc0ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:57:27.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03d8e89640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:27.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03d8e89640 1 -- 172.21.15.12:0/1791364964 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03cc032810 con 0x7f03d40a2d40 2024-09-11T01:57:27.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03dc0ef640 10 monclient: get_monmap_and_config success 2024-09-11T01:57:27.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03dc0ef640 4 set_mon_vals no callback set 2024-09-11T01:57:27.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03d8e89640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:27.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03d8e89640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:27.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03d8e89640 10 monclient: dump: 2024-09-11T01:57:27.136 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:27.136 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:27.136 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.136 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.136 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:27.136 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:27.136 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:27.136 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:27.136 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:27.136 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:27.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03dc0ef640 10 monclient: shutdown 2024-09-11T01:57:27.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03dc0ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:27.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03dc0ef640 1 -- 172.21.15.12:0/1791364964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03d40a2d40 msgr2=0x7f03d406cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:27.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03dc0ef640 1 --2- 172.21.15.12:0/1791364964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03d40a2d40 0x7f03d406cb50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f03d409be70 tx=0x7f03cc029fd0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03dc0ef640 1 -- 172.21.15.12:0/1791364964 shutdown_connections 2024-09-11T01:57:27.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03dc0ef640 1 --2- 172.21.15.12:0/1791364964 >> v2:172.21.15.71:3300/0 conn(0x7f03d406d4a0 0x7f03d413fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03dc0ef640 1 --2- 172.21.15.12:0/1791364964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03d40a2d40 0x7f03d406cb50 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03dc0ef640 1 -- 172.21.15.12:0/1791364964 >> 172.21.15.12:0/1791364964 conn(0x7f03d40a3e30 msgr2=0x7f03d40a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:27.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03dc0ef640 1 -- 172.21.15.12:0/1791364964 shutdown_connections 2024-09-11T01:57:27.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.125+0000 7f03dc0ef640 1 -- 172.21.15.12:0/1791364964 wait complete. 2024-09-11T01:57:27.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.126+0000 7f03dc0ef640 10 monclient: build_initial_monmap 2024-09-11T01:57:27.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.126+0000 7f03dc0ef640 10 monclient: monmap: 2024-09-11T01:57:27.137 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:27.137 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:57:27.137 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:57:27.138 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:57:27.138 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:27.138 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:27.138 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:57:27.138 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:57:27.138 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:57:27.138 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:27.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.126+0000 7f03dc0ef640 1 Processor -- start 2024-09-11T01:57:27.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.126+0000 7f03dc0ef640 1 -- start start 2024-09-11T01:57:27.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.126+0000 7f03dc0ef640 10 monclient: init 2024-09-11T01:57:27.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03dc0ef640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:27.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03dc0ef640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:57:27.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03dc0ef640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03d406d090 0x7f03d41cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03dc0ef640 10 monclient(hunting): picked mon.noname-c con 0x7f03d406d090 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:27.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03dc0ef640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03d406d4a0 0x7f03d41cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03dc0ef640 10 monclient(hunting): picked mon.noname-b con 0x7f03d406d4a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:57:27.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03dc0ef640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03d40a2d40 0x7f03d4093dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03dc0ef640 10 monclient(hunting): picked mon.noname-a con 0x7f03d40a2d40 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:57:27.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03dc0ef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:27.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03dc0ef640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f03d409ba10 con 0x7f03d40a2d40 2024-09-11T01:57:27.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03dc0ef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:27.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03dc0ef640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f03d406d8a0 con 0x7f03d406d4a0 2024-09-11T01:57:27.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03dc0ef640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:27.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03dc0ef640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f03d409b3a0 con 0x7f03d406d090 2024-09-11T01:57:27.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03dc0ef640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:27.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03dc0ef640 10 monclient(hunting): authenticate will time out at 2453.569247s 2024-09-11T01:57:27.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03d9e8b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03d406d090 0x7f03d41cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03da68c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03d40a2d40 0x7f03d4093dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03d9e8b640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03d406d090 0x7f03d41cd530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46546/0 (socket says 172.21.15.12:46546) 2024-09-11T01:57:27.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03d9e8b640 1 -- 172.21.15.12:0/4084824089 learned_addr learned my addr 172.21.15.12:0/4084824089 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:27.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03d9e8b640 10 monclient(hunting): get_auth_request con 0x7f03d406d090 auth_method 0 2024-09-11T01:57:27.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03d9e8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:27.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03d9e8b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:27.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03d9e8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:27.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03da68c640 10 monclient(hunting): get_auth_request con 0x7f03d40a2d40 auth_method 0 2024-09-11T01:57:27.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03da68c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:27.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03da68c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:27.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03da68c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:27.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03d9e8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:27.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03d9e8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:27.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03d9e8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:27.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03da68c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:27.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03da68c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:27.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03d968a640 1 --2- 172.21.15.12:0/4084824089 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03d406d4a0 0x7f03d41cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03da68c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:27.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03d968a640 10 monclient(hunting): get_auth_request con 0x7f03d406d4a0 auth_method 0 2024-09-11T01:57:27.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03d968a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:27.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03d968a640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:27.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03d968a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:27.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03d968a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:27.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03d968a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:27.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03d968a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:27.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03da68c640 10 monclient(hunting): handle_auth_done global_id 294360 payload 824 2024-09-11T01:57:27.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03da68c640 1 -- 172.21.15.12:0/4084824089 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03d406d090 msgr2=0x7f03d41cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:27.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03da68c640 1 --2- 172.21.15.12:0/4084824089 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03d406d090 0x7f03d41cd530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03da68c640 1 -- 172.21.15.12:0/4084824089 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03d406d4a0 msgr2=0x7f03d41cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:27.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03da68c640 1 --2- 172.21.15.12:0/4084824089 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03d406d4a0 0x7f03d41cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03da68c640 10 monclient: _finish_hunting 0 2024-09-11T01:57:27.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03da68c640 1 monclient: found mon.noname-a 2024-09-11T01:57:27.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03da68c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:27.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03da68c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:57:27.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03da68c640 1 -- 172.21.15.12:0/4084824089 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03d4097b90 con 0x7f03d40a2d40 2024-09-11T01:57:27.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.127+0000 7f03d9e8b640 1 --2- 172.21.15.12:0/4084824089 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03d406d090 0x7f03d41cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:57:27.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.129+0000 7f03da68c640 1 -- 172.21.15.12:0/4084824089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03d40a2d40 msgr2=0x7f03d4093dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:27.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.129+0000 7f03da68c640 1 --2- 172.21.15.12:0/4084824089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03d40a2d40 0x7f03d4093dc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f03c4009d90 tx=0x7f03c400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:27.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.130+0000 7f03d27fc640 1 -- 172.21.15.12:0/4084824089 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c4018020 con 0x7f03d40a2d40 2024-09-11T01:57:27.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.130+0000 7f03d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:27.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.130+0000 7f03d27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:57:27.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.130+0000 7f03d27fc640 10 monclient: dump: 2024-09-11T01:57:27.142 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:27.143 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:27.143 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.143 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.143 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:27.143 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:27.143 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:27.143 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:27.143 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:27.143 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:27.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.130+0000 7f03d27fc640 10 monclient: _finish_auth 0 2024-09-11T01:57:27.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.130+0000 7f03d27fc640 10 monclient: _check_auth_tickets 2024-09-11T01:57:27.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.130+0000 7f03d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:27.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.130+0000 7f03d27fc640 1 -- 172.21.15.12:0/4084824089 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03c40146a0 con 0x7f03d40a2d40 2024-09-11T01:57:27.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.130+0000 7f03dc0ef640 5 monclient: authenticate success, global_id 294360 2024-09-11T01:57:27.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.130+0000 7f03dc0ef640 10 monclient: _renew_subs 2024-09-11T01:57:27.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.130+0000 7f03dc0ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:27.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.130+0000 7f03dc0ef640 1 -- 172.21.15.12:0/4084824089 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03d409a080 con 0x7f03d40a2d40 2024-09-11T01:57:27.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.130+0000 7f03d27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:27.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.130+0000 7f03d27fc640 1 -- 172.21.15.12:0/4084824089 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c4014a90 con 0x7f03d40a2d40 2024-09-11T01:57:27.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.130+0000 7f03d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:27.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.130+0000 7f03d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:27.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.130+0000 7f03d27fc640 10 monclient: dump: 2024-09-11T01:57:27.144 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:27.144 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:27.144 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.144 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.144 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:27.144 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:27.145 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:27.145 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:27.145 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:27.145 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:27.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.130+0000 7f03dc0ef640 10 monclient: _renew_subs 2024-09-11T01:57:27.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.130+0000 7f03dc0ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:27.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.130+0000 7f03dc0ef640 1 -- 172.21.15.12:0/4084824089 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03d40972f0 con 0x7f03d40a2d40 2024-09-11T01:57:27.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.130+0000 7f03d3fff640 4 set_mon_vals no callback set 2024-09-11T01:57:27.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.131+0000 7f03dc0ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:27.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.131+0000 7f03dc0ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:27.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.131+0000 7f03dc0ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:27.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.131+0000 7f03d27fc640 1 -- 172.21.15.12:0/4084824089 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f03c4014c30 con 0x7f03d40a2d40 2024-09-11T01:57:27.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.131+0000 7f03dc0ef640 1 -- 172.21.15.12:0/4084824089 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03d406cb50 con 0x7f03d40a2d40 2024-09-11T01:57:27.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.131+0000 7f03d27fc640 1 --2- 172.21.15.12:0/4084824089 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f03c003e260 0x7f03c0040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.131+0000 7f03d27fc640 1 -- 172.21.15.12:0/4084824089 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(923..923 src has 889..923) ==== 4422+0+0 (secure 0 0 0) 0x7f03c4053dc0 con 0x7f03d40a2d40 2024-09-11T01:57:27.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.131+0000 7f03d9e8b640 1 --2- 172.21.15.12:0/4084824089 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f03c003e260 0x7f03c0040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.131+0000 7f03d9e8b640 10 monclient: get_auth_request con 0x7f03c003e260 auth_method 0 2024-09-11T01:57:27.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.134+0000 7f03d9e8b640 1 --2- 172.21.15.12:0/4084824089 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f03c003e260 0x7f03c0040720 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f03cc009560 tx=0x7f03cc02e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:27.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.135+0000 7f03d27fc640 1 -- 172.21.15.12:0/4084824089 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f03c4022030 con 0x7f03d40a2d40 2024-09-11T01:57:27.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.135+0000 7f03d27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:27.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.135+0000 7f03d27fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:57:27.320 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.319+0000 7f03dc0ef640 1 -- 172.21.15.12:0/4084824089 --> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f03d40a3120 con 0x7f03c003e260 2024-09-11T01:57:27.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.319+0000 7f03d27fc640 1 -- 172.21.15.12:0/4084824089 <== mgr.4101 v2:172.21.15.71:6816/3189259519 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f03cc038040 con 0x7f03c003e260 2024-09-11T01:57:27.321 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 1 to deep-scrub 2024-09-11T01:57:27.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.321+0000 7f03dc0ef640 1 -- 172.21.15.12:0/4084824089 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f03c003e260 msgr2=0x7f03c0040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:27.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.321+0000 7f03dc0ef640 1 --2- 172.21.15.12:0/4084824089 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f03c003e260 0x7f03c0040720 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f03cc009560 tx=0x7f03cc02e000 comp rx=0 tx=0).stop 2024-09-11T01:57:27.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.321+0000 7f03dc0ef640 10 monclient: shutdown 2024-09-11T01:57:27.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.321+0000 7f03dc0ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:27.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.321+0000 7f03dc0ef640 1 -- 172.21.15.12:0/4084824089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03d40a2d40 msgr2=0x7f03d4093dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:27.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.321+0000 7f03dc0ef640 1 --2- 172.21.15.12:0/4084824089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03d40a2d40 0x7f03d4093dc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f03c4009d90 tx=0x7f03c400c490 comp rx=0 tx=0).stop 2024-09-11T01:57:27.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.321+0000 7f03dc0ef640 1 -- 172.21.15.12:0/4084824089 shutdown_connections 2024-09-11T01:57:27.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.321+0000 7f03dc0ef640 1 --2- 172.21.15.12:0/4084824089 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f03c003e260 0x7f03c0040720 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.321+0000 7f03dc0ef640 1 --2- 172.21.15.12:0/4084824089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03d40a2d40 0x7f03d4093dc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.321+0000 7f03dc0ef640 1 --2- 172.21.15.12:0/4084824089 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03d406d4a0 0x7f03d41cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.321+0000 7f03dc0ef640 1 --2- 172.21.15.12:0/4084824089 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f03d406d090 0x7f03d41cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.321+0000 7f03dc0ef640 1 -- 172.21.15.12:0/4084824089 >> 172.21.15.12:0/4084824089 conn(0x7f03d40a3e30 msgr2=0x7f03d409f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:27.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.322+0000 7f03dc0ef640 1 -- 172.21.15.12:0/4084824089 shutdown_connections 2024-09-11T01:57:27.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.322+0000 7f03dc0ef640 1 -- 172.21.15.12:0/4084824089 wait complete. 2024-09-11T01:57:27.333 INFO:tasks.ceph:Scrubbing osd.2 2024-09-11T01:57:27.333 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-11T01:57:27.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.436+0000 7f8accf17640 10 monclient: get_monmap_and_config 2024-09-11T01:57:27.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.436+0000 7f8accf17640 10 monclient: build_initial_monmap 2024-09-11T01:57:27.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.437+0000 7f8accf17640 10 monclient: monmap: 2024-09-11T01:57:27.439 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:27.439 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:27.439 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:57:27.437736+0000 2024-09-11T01:57:27.439 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:57:27.437736+0000 2024-09-11T01:57:27.439 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:27.439 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:27.439 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:57:27.439 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:57:27.439 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:57:27.439 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:57:27.439 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:57:27.439 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:27.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.437+0000 7f8accf17640 1 Processor -- start 2024-09-11T01:57:27.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.437+0000 7f8accf17640 1 -- start start 2024-09-11T01:57:27.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.437+0000 7f8accf17640 10 monclient: init 2024-09-11T01:57:27.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8accf17640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:27.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8accf17640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-11T01:57:27.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8accf17640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f8ac806cb50 0x7f8ac806cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8accf17640 10 monclient(hunting): picked mon.noname-c con 0x7f8ac806cb50 addr v2:172.21.15.71:3300/0 2024-09-11T01:57:27.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8accf17640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8ac809c840 addr v1:172.21.15.71:6789/0 2024-09-11T01:57:27.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8accf17640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8ac806d470 0x7f8ac813fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8accf17640 10 monclient(hunting): picked mon.noname-a con 0x7f8ac806d470 addr v2:172.21.15.12:3300/0 2024-09-11T01:57:27.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8accf17640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ac809c400 con 0x7f8ac809c840 2024-09-11T01:57:27.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8accf17640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:27.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8accf17640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f8ac806dd10 con 0x7f8ac806d470 2024-09-11T01:57:27.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8accf17640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:27.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8accf17640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f8ac80a3570 con 0x7f8ac806cb50 2024-09-11T01:57:27.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8accf17640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:27.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8accf17640 10 monclient(hunting): authenticate will time out at 2453.880396s 2024-09-11T01:57:27.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8ac6d9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8ac806d470 0x7f8ac813fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8ac6d9d640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8ac806d470 0x7f8ac813fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54160/0 (socket says 172.21.15.12:54160) 2024-09-11T01:57:27.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8ac6d9d640 1 -- 172.21.15.12:0/3121229297 learned_addr learned my addr 172.21.15.12:0/3121229297 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:27.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8ac6d9d640 10 monclient(hunting): get_auth_request con 0x7f8ac806d470 auth_method 0 2024-09-11T01:57:27.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8ac6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:27.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8ac6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:27.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8ac6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:27.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8ac6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:27.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8ac6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:27.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8ac6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:27.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8ac659c640 1 --2- 172.21.15.12:0/3121229297 >> v2:172.21.15.71:3300/0 conn(0x7f8ac806cb50 0x7f8ac806cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8ac659c640 10 monclient(hunting): get_auth_request con 0x7f8ac806cb50 auth_method 0 2024-09-11T01:57:27.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8ac659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:27.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8ac659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:27.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8ac659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:27.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8ac6d9d640 10 monclient(hunting): handle_auth_done global_id 294363 payload 274 2024-09-11T01:57:27.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8ac6d9d640 1 -- 172.21.15.12:0/3121229297 >> v2:172.21.15.71:3300/0 conn(0x7f8ac806cb50 msgr2=0x7f8ac806cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:27.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8ac6d9d640 1 --2- 172.21.15.12:0/3121229297 >> v2:172.21.15.71:3300/0 conn(0x7f8ac806cb50 0x7f8ac806cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8ac6d9d640 1 -- 172.21.15.12:0/3121229297 >> v1:172.21.15.71:6789/0 conn(0x7f8ac809c840 legacy=0x7f8ac813c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:27.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8ac6d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:57:27.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8ac6d9d640 1 monclient: found mon.noname-a 2024-09-11T01:57:27.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8ac6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:27.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8ac6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:57:27.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.438+0000 7f8ac6d9d640 1 -- 172.21.15.12:0/3121229297 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ac809baf0 con 0x7f8ac806d470 2024-09-11T01:57:27.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.441+0000 7f8ac6d9d640 1 -- 172.21.15.12:0/3121229297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ac806d470 msgr2=0x7f8ac813fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:27.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.441+0000 7f8ac6d9d640 1 --2- 172.21.15.12:0/3121229297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ac806d470 0x7f8ac813fc00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8070c70 tx=0x7f8ac002a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fce558b276b2d0b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:27.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.441+0000 7f8ac559a640 1 -- 172.21.15.12:0/3121229297 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac0037070 con 0x7f8ac806d470 2024-09-11T01:57:27.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.441+0000 7f8ac559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:27.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.441+0000 7f8ac559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:57:27.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.441+0000 7f8ac559a640 10 monclient: dump: 2024-09-11T01:57:27.451 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:27.451 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:27.451 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.451 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.452 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:27.452 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:27.452 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:27.452 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:27.452 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:27.452 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:27.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.441+0000 7f8ac559a640 10 monclient: _finish_auth 0 2024-09-11T01:57:27.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.441+0000 7f8ac559a640 10 monclient: _check_auth_tickets 2024-09-11T01:57:27.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.441+0000 7f8ac559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:27.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.441+0000 7f8accf17640 5 monclient: authenticate success, global_id 294363 2024-09-11T01:57:27.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.441+0000 7f8accf17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:57:27.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.441+0000 7f8ac559a640 1 -- 172.21.15.12:0/3121229297 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ac0032640 con 0x7f8ac806d470 2024-09-11T01:57:27.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.441+0000 7f8ac559a640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:27.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.441+0000 7f8accf17640 10 monclient: get_monmap_and_config success 2024-09-11T01:57:27.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.441+0000 7f8accf17640 4 set_mon_vals no callback set 2024-09-11T01:57:27.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.441+0000 7f8ac559a640 1 -- 172.21.15.12:0/3121229297 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac0032a10 con 0x7f8ac806d470 2024-09-11T01:57:27.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.441+0000 7f8ac559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:27.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.441+0000 7f8ac559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:27.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.441+0000 7f8ac559a640 10 monclient: dump: 2024-09-11T01:57:27.453 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:27.453 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:27.453 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.453 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.453 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:27.453 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:27.453 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:27.453 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:27.454 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:27.454 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:27.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.442+0000 7f8accf17640 10 monclient: shutdown 2024-09-11T01:57:27.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.442+0000 7f8accf17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:27.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.442+0000 7f8accf17640 1 -- 172.21.15.12:0/3121229297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ac806d470 msgr2=0x7f8ac813fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:27.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.442+0000 7f8accf17640 1 --2- 172.21.15.12:0/3121229297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ac806d470 0x7f8ac813fc00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8070c70 tx=0x7f8ac002a0b0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.442+0000 7f8accf17640 1 -- 172.21.15.12:0/3121229297 shutdown_connections 2024-09-11T01:57:27.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.442+0000 7f8accf17640 1 --2- 172.21.15.12:0/3121229297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ac806d470 0x7f8ac813fc00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.442+0000 7f8accf17640 1 --2- 172.21.15.12:0/3121229297 >> v2:172.21.15.71:3300/0 conn(0x7f8ac806cb50 0x7f8ac806cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.442+0000 7f8accf17640 1 -- 172.21.15.12:0/3121229297 >> 172.21.15.12:0/3121229297 conn(0x7f8ac80a3e30 msgr2=0x7f8ac80a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:27.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.442+0000 7f8accf17640 1 -- 172.21.15.12:0/3121229297 shutdown_connections 2024-09-11T01:57:27.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.442+0000 7f8accf17640 1 -- 172.21.15.12:0/3121229297 wait complete. 2024-09-11T01:57:27.454 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.443+0000 7f8accf17640 10 monclient: build_initial_monmap 2024-09-11T01:57:27.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.443+0000 7f8accf17640 10 monclient: monmap: 2024-09-11T01:57:27.455 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:27.455 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:57:27.455 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:57:27.455 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:57:27.455 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:27.455 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:27.455 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:57:27.455 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:57:27.455 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:57:27.455 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:27.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.443+0000 7f8accf17640 1 Processor -- start 2024-09-11T01:57:27.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.443+0000 7f8accf17640 1 -- start start 2024-09-11T01:57:27.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.443+0000 7f8accf17640 10 monclient: init 2024-09-11T01:57:27.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.443+0000 7f8accf17640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:27.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.443+0000 7f8accf17640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:57:27.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.443+0000 7f8accf17640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8ac806cb50 0x7f8ac81cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.443+0000 7f8accf17640 10 monclient(hunting): picked mon.noname-c con 0x7f8ac806cb50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:27.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.443+0000 7f8accf17640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ac806d470 0x7f8ac81cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.443+0000 7f8accf17640 10 monclient(hunting): picked mon.noname-b con 0x7f8ac806d470 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:57:27.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.443+0000 7f8accf17640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ac809c840 0x7f8ac8093de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.443+0000 7f8accf17640 10 monclient(hunting): picked mon.noname-a con 0x7f8ac809c840 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:57:27.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.443+0000 7f8accf17640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:27.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.443+0000 7f8accf17640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac809af90 con 0x7f8ac809c840 2024-09-11T01:57:27.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.443+0000 7f8accf17640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:27.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.443+0000 7f8accf17640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac806d880 con 0x7f8ac806d470 2024-09-11T01:57:27.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.443+0000 7f8accf17640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:27.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.443+0000 7f8accf17640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f8ac809a0f0 con 0x7f8ac806cb50 2024-09-11T01:57:27.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.443+0000 7f8accf17640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:27.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.443+0000 7f8accf17640 10 monclient(hunting): authenticate will time out at 2453.885993s 2024-09-11T01:57:27.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac6d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ac809c840 0x7f8ac8093de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8ac806cb50 0x7f8ac81cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac6d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ac809c840 0x7f8ac8093de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54172/0 (socket says 172.21.15.12:54172) 2024-09-11T01:57:27.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac6d9d640 1 -- 172.21.15.12:0/1153194842 learned_addr learned my addr 172.21.15.12:0/1153194842 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:27.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac6d9d640 10 monclient(hunting): get_auth_request con 0x7f8ac809c840 auth_method 0 2024-09-11T01:57:27.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac659c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8ac806cb50 0x7f8ac81cd530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46562/0 (socket says 172.21.15.12:46562) 2024-09-11T01:57:27.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:27.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:27.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:27.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac659c640 10 monclient(hunting): get_auth_request con 0x7f8ac806cb50 auth_method 0 2024-09-11T01:57:27.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:27.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:27.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:27.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:27.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:27.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:27.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:27.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:27.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:27.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac5d9b640 1 --2- 172.21.15.12:0/1153194842 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ac806d470 0x7f8ac81cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac5d9b640 10 monclient(hunting): get_auth_request con 0x7f8ac806d470 auth_method 0 2024-09-11T01:57:27.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:27.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:27.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:27.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac6d9d640 10 monclient(hunting): handle_auth_done global_id 294366 payload 824 2024-09-11T01:57:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac6d9d640 1 -- 172.21.15.12:0/1153194842 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8ac806cb50 msgr2=0x7f8ac81cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac6d9d640 1 --2- 172.21.15.12:0/1153194842 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8ac806cb50 0x7f8ac81cd530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac6d9d640 1 -- 172.21.15.12:0/1153194842 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ac806d470 msgr2=0x7f8ac81cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac6d9d640 1 --2- 172.21.15.12:0/1153194842 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ac806d470 0x7f8ac81cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac6d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:57:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac6d9d640 1 monclient: found mon.noname-a 2024-09-11T01:57:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:57:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac6d9d640 1 -- 172.21.15.12:0/1153194842 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ac8097c00 con 0x7f8ac809c840 2024-09-11T01:57:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.444+0000 7f8ac5d9b640 1 --2- 172.21.15.12:0/1153194842 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ac806d470 0x7f8ac81cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:57:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.446+0000 7f8ac6d9d640 1 -- 172.21.15.12:0/1153194842 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ac809c840 msgr2=0x7f8ac8093de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:27.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.446+0000 7f8ac6d9d640 1 --2- 172.21.15.12:0/1153194842 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ac809c840 0x7f8ac8093de0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0009960 tx=0x7f8ac00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:27.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.446+0000 7f8ab6ffd640 1 -- 172.21.15.12:0/1153194842 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac0041030 con 0x7f8ac809c840 2024-09-11T01:57:27.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.446+0000 7f8ab6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:27.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.446+0000 7f8ab6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:57:27.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.446+0000 7f8ab6ffd640 10 monclient: dump: 2024-09-11T01:57:27.461 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:27.461 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:27.461 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.461 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.461 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:27.461 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:27.461 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:27.461 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:27.461 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:27.461 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:27.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.446+0000 7f8ab6ffd640 10 monclient: _finish_auth 0 2024-09-11T01:57:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.446+0000 7f8ab6ffd640 10 monclient: _check_auth_tickets 2024-09-11T01:57:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.446+0000 7f8ab6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.446+0000 7f8accf17640 5 monclient: authenticate success, global_id 294366 2024-09-11T01:57:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.446+0000 7f8ab6ffd640 1 -- 172.21.15.12:0/1153194842 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ac003d630 con 0x7f8ac809c840 2024-09-11T01:57:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.446+0000 7f8accf17640 10 monclient: _renew_subs 2024-09-11T01:57:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.446+0000 7f8accf17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.446+0000 7f8accf17640 1 -- 172.21.15.12:0/1153194842 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ac809e520 con 0x7f8ac809c840 2024-09-11T01:57:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.446+0000 7f8ab6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.446+0000 7f8ab6ffd640 1 -- 172.21.15.12:0/1153194842 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac0037040 con 0x7f8ac809c840 2024-09-11T01:57:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.446+0000 7f8ab6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.446+0000 7f8ab6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.446+0000 7f8ab6ffd640 10 monclient: dump: 2024-09-11T01:57:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:27.462 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.463 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.463 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:27.463 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:27.463 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:27.463 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:27.463 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:27.463 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:27.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.446+0000 7f8ac4d99640 4 set_mon_vals no callback set 2024-09-11T01:57:27.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.446+0000 7f8accf17640 10 monclient: _renew_subs 2024-09-11T01:57:27.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.446+0000 7f8accf17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:27.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.446+0000 7f8accf17640 1 -- 172.21.15.12:0/1153194842 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ac8097920 con 0x7f8ac809c840 2024-09-11T01:57:27.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.447+0000 7f8accf17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:27.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.447+0000 7f8accf17640 1 -- 172.21.15.12:0/1153194842 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8ac80a3040 con 0x7f8ac809c840 2024-09-11T01:57:27.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.447+0000 7f8ab6ffd640 1 -- 172.21.15.12:0/1153194842 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f8ac003a030 con 0x7f8ac809c840 2024-09-11T01:57:27.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.448+0000 7f8ab6ffd640 1 --2- 172.21.15.12:0/1153194842 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8aa803e240 0x7f8aa8040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.448+0000 7f8ab6ffd640 1 -- 172.21.15.12:0/1153194842 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(923..923 src has 889..923) ==== 4422+0+0 (secure 0 0 0) 0x7f8ac0072d00 con 0x7f8ac809c840 2024-09-11T01:57:27.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.448+0000 7f8ab6ffd640 1 --2- 172.21.15.12:0/1153194842 >> [v2:172.21.15.12:6800/2893036867,v1:172.21.15.12:6801/2893036867] conn(0x7f8aa8044130 0x7f8aa8046520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.448+0000 7f8ab6ffd640 1 -- 172.21.15.12:0/1153194842 --> [v2:172.21.15.12:6800/2893036867,v1:172.21.15.12:6801/2893036867] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8ac0032640 con 0x7f8aa8044130 2024-09-11T01:57:27.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.448+0000 7f8ab6ffd640 1 -- 172.21.15.12:0/1153194842 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=923) ==== 24+0+0 (secure 0 0 0) 0x7f8ac0075230 con 0x7f8ac809c840 2024-09-11T01:57:27.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.448+0000 7f8ab6ffd640 10 monclient: handle_get_version_reply finishing 1 version 923 2024-09-11T01:57:27.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.448+0000 7f8ac5d9b640 1 --2- 172.21.15.12:0/1153194842 >> [v2:172.21.15.12:6800/2893036867,v1:172.21.15.12:6801/2893036867] conn(0x7f8aa8044130 0x7f8aa8046520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.448+0000 7f8ac5d9b640 10 monclient: get_auth_request con 0x7f8aa8044130 auth_method 0 2024-09-11T01:57:27.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.448+0000 7f8ac659c640 1 --2- 172.21.15.12:0/1153194842 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8aa803e240 0x7f8aa8040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.448+0000 7f8ac659c640 10 monclient: get_auth_request con 0x7f8aa803e240 auth_method 0 2024-09-11T01:57:27.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.450+0000 7f8ac659c640 1 --2- 172.21.15.12:0/1153194842 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8aa803e240 0x7f8aa8040700 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8ab8003b80 tx=0x7f8ab80098d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:27.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.452+0000 7f8ac5d9b640 1 --2- 172.21.15.12:0/1153194842 >> [v2:172.21.15.12:6800/2893036867,v1:172.21.15.12:6801/2893036867] conn(0x7f8aa8044130 0x7f8aa8046520 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:27.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.453+0000 7f8ab6ffd640 1 -- 172.21.15.12:0/1153194842 <== osd.2 v2:172.21.15.12:6800/2893036867 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8ac0032640 con 0x7f8aa8044130 2024-09-11T01:57:27.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.466+0000 7f8accf17640 1 -- 172.21.15.12:0/1153194842 --> [v2:172.21.15.12:6800/2893036867,v1:172.21.15.12:6801/2893036867] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f8ac81cdfb0 con 0x7f8aa8044130 2024-09-11T01:57:27.467 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.466+0000 7f8ab6ffd640 1 -- 172.21.15.12:0/1153194842 <== osd.2 v2:172.21.15.12:6800/2893036867 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f8ac81cdfb0 con 0x7f8aa8044130 2024-09-11T01:57:27.467 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:57:27.467 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T01:57:27.468 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:57:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.466+0000 7f8accf17640 1 -- 172.21.15.12:0/1153194842 >> [v2:172.21.15.12:6800/2893036867,v1:172.21.15.12:6801/2893036867] conn(0x7f8aa8044130 msgr2=0x7f8aa8046520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.466+0000 7f8accf17640 1 --2- 172.21.15.12:0/1153194842 >> [v2:172.21.15.12:6800/2893036867,v1:172.21.15.12:6801/2893036867] conn(0x7f8aa8044130 0x7f8aa8046520 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.466+0000 7f8accf17640 1 -- 172.21.15.12:0/1153194842 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8aa803e240 msgr2=0x7f8aa8040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.466+0000 7f8accf17640 1 --2- 172.21.15.12:0/1153194842 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f8aa803e240 0x7f8aa8040700 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8ab8003b80 tx=0x7f8ab80098d0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.466+0000 7f8accf17640 10 monclient: shutdown 2024-09-11T01:57:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.466+0000 7f8accf17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.466+0000 7f8accf17640 1 -- 172.21.15.12:0/1153194842 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ac809c840 msgr2=0x7f8ac8093de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.466+0000 7f8accf17640 1 --2- 172.21.15.12:0/1153194842 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8ac809c840 0x7f8ac8093de0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0009960 tx=0x7f8ac00046c0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.466+0000 7f8ac6d9d640 1 -- 172.21.15.12:0/1153194842 reap_dead start 2024-09-11T01:57:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.466+0000 7f8accf17640 1 -- 172.21.15.12:0/1153194842 shutdown_connections 2024-09-11T01:57:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.466+0000 7f8accf17640 1 -- 172.21.15.12:0/1153194842 >> 172.21.15.12:0/1153194842 conn(0x7f8ac80a3e30 msgr2=0x7f8ac809f8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.466+0000 7f8accf17640 1 -- 172.21.15.12:0/1153194842 shutdown_connections 2024-09-11T01:57:27.469 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.466+0000 7f8accf17640 1 -- 172.21.15.12:0/1153194842 wait complete. 2024-09-11T01:57:27.478 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-11T01:57:27.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.579+0000 7f752d816640 10 monclient: get_monmap_and_config 2024-09-11T01:57:27.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.579+0000 7f752d816640 10 monclient: build_initial_monmap 2024-09-11T01:57:27.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.579+0000 7f752d816640 10 monclient: monmap: 2024-09-11T01:57:27.581 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:27.581 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:27.581 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:57:27.580296+0000 2024-09-11T01:57:27.581 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:57:27.580296+0000 2024-09-11T01:57:27.581 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:27.581 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:27.582 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:57:27.582 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:57:27.582 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:57:27.582 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:57:27.582 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:57:27.582 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:27.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.580+0000 7f752d816640 1 Processor -- start 2024-09-11T01:57:27.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.580+0000 7f752d816640 1 -- start start 2024-09-11T01:57:27.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.580+0000 7f752d816640 10 monclient: init 2024-09-11T01:57:27.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.580+0000 7f752d816640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:27.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.580+0000 7f752d816640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-11T01:57:27.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.580+0000 7f752d816640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f75281381c0 addr v1:172.21.15.12:6789/0 2024-09-11T01:57:27.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.580+0000 7f752d816640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f75280a6aa0 0x7f752809cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.580+0000 7f752d816640 10 monclient(hunting): picked mon.noname-c con 0x7f75280a6aa0 addr v2:172.21.15.71:3300/0 2024-09-11T01:57:27.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.580+0000 7f752d816640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f752813d000 0x7f752813f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.580+0000 7f752d816640 10 monclient(hunting): picked mon.noname-b con 0x7f752813d000 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:27.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.580+0000 7f752d816640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f752809c310 con 0x7f75281381c0 2024-09-11T01:57:27.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.580+0000 7f752d816640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:27.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.580+0000 7f752d816640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f752806fc90 con 0x7f75280a6aa0 2024-09-11T01:57:27.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.580+0000 7f752d816640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:27.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.580+0000 7f752d816640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f752809c5f0 con 0x7f752813d000 2024-09-11T01:57:27.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.580+0000 7f752d816640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:27.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.580+0000 7f752d816640 10 monclient(hunting): authenticate will time out at 2454.022861s 2024-09-11T01:57:27.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.580+0000 7f75277fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f752813d000 0x7f752813f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.580+0000 7f7526ffd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f75281381c0 0x7f75281385a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33900/0 (socket says 172.21.15.12:33900) 2024-09-11T01:57:27.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.581+0000 7f7526ffd640 1 -- 172.21.15.12:0/886573437 learned_addr learned my addr 172.21.15.12:0/886573437 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:57:27.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.581+0000 7f75277fe640 10 monclient(hunting): get_auth_request con 0x7f752813d000 auth_method 0 2024-09-11T01:57:27.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.581+0000 7f75277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:27.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.581+0000 7f75277fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:27.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.581+0000 7f75277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:27.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.581+0000 7f75277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:27.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.581+0000 7f75277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:27.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.581+0000 7f75277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:27.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.581+0000 7f751ffff640 1 --2- 172.21.15.12:0/886573437 >> v2:172.21.15.71:3300/0 conn(0x7f75280a6aa0 0x7f752809cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.581+0000 7f751ffff640 10 monclient(hunting): get_auth_request con 0x7f75280a6aa0 auth_method 0 2024-09-11T01:57:27.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.581+0000 7f751ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:27.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.581+0000 7f751ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:27.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.581+0000 7f751ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:27.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.581+0000 7f75277fe640 10 monclient(hunting): handle_auth_done global_id 304259 payload 274 2024-09-11T01:57:27.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.581+0000 7f75277fe640 1 -- 172.21.15.12:0/886573437 >> v2:172.21.15.71:3300/0 conn(0x7f75280a6aa0 msgr2=0x7f752809cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:27.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.581+0000 7f75277fe640 1 --2- 172.21.15.12:0/886573437 >> v2:172.21.15.71:3300/0 conn(0x7f75280a6aa0 0x7f752809cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.581+0000 7f75277fe640 1 -- 172.21.15.12:0/886573437 >> v1:172.21.15.12:6789/0 conn(0x7f75281381c0 legacy=0x7f75281385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:27.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.581+0000 7f75277fe640 10 monclient: _finish_hunting 0 2024-09-11T01:57:27.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.581+0000 7f75277fe640 1 monclient: found mon.noname-b 2024-09-11T01:57:27.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.581+0000 7f75277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:27.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.581+0000 7f75277fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:57:27.587 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.581+0000 7f75277fe640 1 -- 172.21.15.12:0/886573437 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f752809c8d0 con 0x7f752813d000 2024-09-11T01:57:27.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f75277fe640 1 -- 172.21.15.12:0/886573437 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f752813d000 msgr2=0x7f752813f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:27.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f75277fe640 1 --2- 172.21.15.12:0/886573437 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f752813d000 0x7f752813f3f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7520009d90 tx=0x7f752002a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=195a3fe482c8c135 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:27.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f75267fc640 1 -- 172.21.15.12:0/886573437 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7520007950 con 0x7f752813d000 2024-09-11T01:57:27.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f75267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:27.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f75267fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:57:27.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f75267fc640 10 monclient: dump: 2024-09-11T01:57:27.593 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:27.594 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:27.594 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.594 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.594 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:27.594 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:27.594 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:27.594 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:27.594 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:27.594 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:27.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f75267fc640 10 monclient: _finish_auth 0 2024-09-11T01:57:27.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f75267fc640 10 monclient: _check_auth_tickets 2024-09-11T01:57:27.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f75267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:27.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f75267fc640 1 -- 172.21.15.12:0/886573437 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7520007af0 con 0x7f752813d000 2024-09-11T01:57:27.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f752d816640 5 monclient: authenticate success, global_id 304259 2024-09-11T01:57:27.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f752d816640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:57:27.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f75267fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:27.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f752d816640 10 monclient: get_monmap_and_config success 2024-09-11T01:57:27.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f75267fc640 1 -- 172.21.15.12:0/886573437 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7520031400 con 0x7f752813d000 2024-09-11T01:57:27.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f75267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:27.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f752d816640 4 set_mon_vals no callback set 2024-09-11T01:57:27.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f75267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:57:27.595 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f75267fc640 10 monclient: dump: 2024-09-11T01:57:27.595 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:27.595 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:27.595 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.595 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.595 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:27.595 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:27.596 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:27.596 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:27.596 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:27.596 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:27.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f752d816640 10 monclient: shutdown 2024-09-11T01:57:27.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f752d816640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:27.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f752d816640 1 -- 172.21.15.12:0/886573437 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f752813d000 msgr2=0x7f752813f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:27.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f752d816640 1 --2- 172.21.15.12:0/886573437 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f752813d000 0x7f752813f3f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7520009d90 tx=0x7f752002a480 comp rx=0 tx=0).stop 2024-09-11T01:57:27.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f752d816640 1 -- 172.21.15.12:0/886573437 shutdown_connections 2024-09-11T01:57:27.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f752d816640 1 --2- 172.21.15.12:0/886573437 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f752813d000 0x7f752813f3f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f752d816640 1 --2- 172.21.15.12:0/886573437 >> v2:172.21.15.71:3300/0 conn(0x7f75280a6aa0 0x7f752809cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f752d816640 1 -- 172.21.15.12:0/886573437 >> 172.21.15.12:0/886573437 conn(0x7f75280a7440 msgr2=0x7f75280a7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:27.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f752d816640 1 -- 172.21.15.12:0/886573437 shutdown_connections 2024-09-11T01:57:27.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.584+0000 7f752d816640 1 -- 172.21.15.12:0/886573437 wait complete. 2024-09-11T01:57:27.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.585+0000 7f752d816640 10 monclient: build_initial_monmap 2024-09-11T01:57:27.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.585+0000 7f752d816640 10 monclient: monmap: 2024-09-11T01:57:27.597 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:27.597 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:57:27.597 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:57:27.597 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:57:27.597 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:27.598 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:27.598 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:57:27.598 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:57:27.598 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:57:27.598 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:27.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.585+0000 7f752d816640 1 Processor -- start 2024-09-11T01:57:27.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.585+0000 7f752d816640 1 -- start start 2024-09-11T01:57:27.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.585+0000 7f752d816640 10 monclient: init 2024-09-11T01:57:27.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f752d816640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:27.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f752d816640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:57:27.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f752d816640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75280a6aa0 0x7f75281ccef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f752d816640 10 monclient(hunting): picked mon.noname-a con 0x7f75280a6aa0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:57:27.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f752d816640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f75281381c0 0x7f7528136030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f752d816640 10 monclient(hunting): picked mon.noname-c con 0x7f75281381c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f752d816640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f752813d000 0x7f7528139080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f752d816640 10 monclient(hunting): picked mon.noname-b con 0x7f752813d000 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:57:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f752d816640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f752d816640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f752813fab0 con 0x7f75280a6aa0 2024-09-11T01:57:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f752d816640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f752d816640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f752813f930 con 0x7f752813d000 2024-09-11T01:57:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f752d816640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f752d816640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f752813fc30 con 0x7f75281381c0 2024-09-11T01:57:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f752d816640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f752d816640 10 monclient(hunting): authenticate will time out at 2454.028368s 2024-09-11T01:57:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f751ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f75281381c0 0x7f7528136030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f7526ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75280a6aa0 0x7f75281ccef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f751ffff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f75281381c0 0x7f7528136030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46584/0 (socket says 172.21.15.12:46584) 2024-09-11T01:57:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f751ffff640 1 -- 172.21.15.12:0/718892773 learned_addr learned my addr 172.21.15.12:0/718892773 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f7526ffd640 10 monclient(hunting): get_auth_request con 0x7f75280a6aa0 auth_method 0 2024-09-11T01:57:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f7526ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f7526ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f7526ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f751ffff640 10 monclient(hunting): get_auth_request con 0x7f75281381c0 auth_method 0 2024-09-11T01:57:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f751ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f751ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f751ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f75277fe640 1 --2- 172.21.15.12:0/718892773 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f752813d000 0x7f7528139080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f75277fe640 10 monclient(hunting): get_auth_request con 0x7f752813d000 auth_method 0 2024-09-11T01:57:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f75277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:27.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f75277fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f75277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f7526ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f7526ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f7526ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f751ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f751ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f751ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f75277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f75277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f75277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f751ffff640 10 monclient(hunting): handle_auth_done global_id 304262 payload 824 2024-09-11T01:57:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f751ffff640 1 -- 172.21.15.12:0/718892773 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f752813d000 msgr2=0x7f7528139080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f751ffff640 1 --2- 172.21.15.12:0/718892773 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f752813d000 0x7f7528139080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f751ffff640 1 -- 172.21.15.12:0/718892773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75280a6aa0 msgr2=0x7f75281ccef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:27.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f751ffff640 1 --2- 172.21.15.12:0/718892773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75280a6aa0 0x7f75281ccef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f751ffff640 10 monclient: _finish_hunting 0 2024-09-11T01:57:27.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f751ffff640 1 monclient: found mon.noname-c 2024-09-11T01:57:27.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f751ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:27.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f751ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:57:27.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f751ffff640 1 -- 172.21.15.12:0/718892773 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f752813b130 con 0x7f75281381c0 2024-09-11T01:57:27.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.586+0000 7f7526ffd640 1 --2- 172.21.15.12:0/718892773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75280a6aa0 0x7f75281ccef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:57:27.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.589+0000 7f751ffff640 1 -- 172.21.15.12:0/718892773 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f75281381c0 msgr2=0x7f7528136030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:27.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.589+0000 7f751ffff640 1 --2- 172.21.15.12:0/718892773 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f75281381c0 0x7f7528136030 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f751000d800 tx=0x7f751000dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:27.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.589+0000 7f7524ff9640 1 -- 172.21.15.12:0/718892773 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751000b8d0 con 0x7f75281381c0 2024-09-11T01:57:27.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.589+0000 7f7524ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:27.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.589+0000 7f7524ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:57:27.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.589+0000 7f7524ff9640 10 monclient: dump: 2024-09-11T01:57:27.603 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:27.603 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:27.603 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.603 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.603 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:27.603 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:27.603 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:27.603 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:27.604 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.589+0000 7f7524ff9640 10 monclient: _finish_auth 0 2024-09-11T01:57:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.589+0000 7f7524ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:57:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.589+0000 7f7524ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.589+0000 7f7524ff9640 1 -- 172.21.15.12:0/718892773 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7510019070 con 0x7f75281381c0 2024-09-11T01:57:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.589+0000 7f7524ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.589+0000 7f7524ff9640 1 -- 172.21.15.12:0/718892773 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751000bc90 con 0x7f75281381c0 2024-09-11T01:57:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.589+0000 7f7524ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.589+0000 7f7524ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:57:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.589+0000 7f7524ff9640 10 monclient: dump: 2024-09-11T01:57:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:27.605 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.589+0000 7f752d816640 5 monclient: authenticate success, global_id 304262 2024-09-11T01:57:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.589+0000 7f752d816640 10 monclient: _renew_subs 2024-09-11T01:57:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.589+0000 7f752d816640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.589+0000 7f752d816640 1 -- 172.21.15.12:0/718892773 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75281cd430 con 0x7f75281381c0 2024-09-11T01:57:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.589+0000 7f7525ffb640 4 set_mon_vals no callback set 2024-09-11T01:57:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.589+0000 7f752d816640 10 monclient: _renew_subs 2024-09-11T01:57:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.589+0000 7f752d816640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.589+0000 7f752d816640 1 -- 172.21.15.12:0/718892773 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f752813b540 con 0x7f75281381c0 2024-09-11T01:57:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.590+0000 7f7524ff9640 1 -- 172.21.15.12:0/718892773 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f7510013070 con 0x7f75281381c0 2024-09-11T01:57:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.590+0000 7f752d816640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.590+0000 7f752d816640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.590+0000 7f752d816640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.590+0000 7f752d816640 1 -- 172.21.15.12:0/718892773 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f752809cfd0 con 0x7f75281381c0 2024-09-11T01:57:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.590+0000 7f7524ff9640 1 --2- 172.21.15.12:0/718892773 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f750803e210 0x7f75080406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.590+0000 7f7524ff9640 1 -- 172.21.15.12:0/718892773 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(923..923 src has 889..923) ==== 4422+0+0 (secure 0 0 0) 0x7f7510052d40 con 0x7f75281381c0 2024-09-11T01:57:27.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.590+0000 7f7526ffd640 1 --2- 172.21.15.12:0/718892773 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f750803e210 0x7f75080406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.590+0000 7f7526ffd640 10 monclient: get_auth_request con 0x7f750803e210 auth_method 0 2024-09-11T01:57:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.593+0000 7f7526ffd640 1 --2- 172.21.15.12:0/718892773 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f750803e210 0x7f75080406d0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f7518007c90 tx=0x7f7518007be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.595+0000 7f7524ff9640 1 -- 172.21.15.12:0/718892773 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f75100518a0 con 0x7f75281381c0 2024-09-11T01:57:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.595+0000 7f7524ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:27.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.595+0000 7f7524ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:57:27.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.782+0000 7f752d816640 1 -- 172.21.15.12:0/718892773 --> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f75280a1710 con 0x7f750803e210 2024-09-11T01:57:27.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.782+0000 7f7524ff9640 1 -- 172.21.15.12:0/718892773 <== mgr.4101 v2:172.21.15.71:6816/3189259519 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f75280a1710 con 0x7f750803e210 2024-09-11T01:57:27.784 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 2 to deep-scrub 2024-09-11T01:57:27.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.784+0000 7f752d816640 1 -- 172.21.15.12:0/718892773 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f750803e210 msgr2=0x7f75080406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:27.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.784+0000 7f752d816640 1 --2- 172.21.15.12:0/718892773 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f750803e210 0x7f75080406d0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f7518007c90 tx=0x7f7518007be0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.784+0000 7f752d816640 10 monclient: shutdown 2024-09-11T01:57:27.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.784+0000 7f752d816640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:27.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.784+0000 7f752d816640 1 -- 172.21.15.12:0/718892773 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f75281381c0 msgr2=0x7f7528136030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:27.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.784+0000 7f752d816640 1 --2- 172.21.15.12:0/718892773 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f75281381c0 0x7f7528136030 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f751000d800 tx=0x7f751000dcc0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.784+0000 7f752d816640 1 -- 172.21.15.12:0/718892773 shutdown_connections 2024-09-11T01:57:27.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.784+0000 7f752d816640 1 --2- 172.21.15.12:0/718892773 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f752813d000 0x7f7528139080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.784+0000 7f752d816640 1 --2- 172.21.15.12:0/718892773 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f75281381c0 0x7f7528136030 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.784+0000 7f752d816640 1 --2- 172.21.15.12:0/718892773 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f750803e210 0x7f75080406d0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.784+0000 7f752d816640 1 --2- 172.21.15.12:0/718892773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75280a6aa0 0x7f75281ccef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.784+0000 7f752d816640 1 -- 172.21.15.12:0/718892773 >> 172.21.15.12:0/718892773 conn(0x7f75280a7440 msgr2=0x7f75280a4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:27.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.784+0000 7f752d816640 1 -- 172.21.15.12:0/718892773 shutdown_connections 2024-09-11T01:57:27.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.784+0000 7f752d816640 1 -- 172.21.15.12:0/718892773 wait complete. 2024-09-11T01:57:27.796 INFO:tasks.ceph:Scrubbing osd.3 2024-09-11T01:57:27.797 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-11T01:57:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.899+0000 7fa587216640 10 monclient: get_monmap_and_config 2024-09-11T01:57:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.899+0000 7fa587216640 10 monclient: build_initial_monmap 2024-09-11T01:57:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.899+0000 7fa587216640 10 monclient: monmap: 2024-09-11T01:57:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:57:27.900426+0000 2024-09-11T01:57:27.901 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:57:27.900426+0000 2024-09-11T01:57:27.902 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:27.902 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:27.902 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:57:27.902 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:57:27.902 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:57:27.902 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:57:27.902 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:57:27.902 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:27.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.900+0000 7fa587216640 1 Processor -- start 2024-09-11T01:57:27.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.900+0000 7fa587216640 1 -- start start 2024-09-11T01:57:27.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.900+0000 7fa587216640 10 monclient: init 2024-09-11T01:57:27.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.900+0000 7fa587216640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:27.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.900+0000 7fa587216640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-11T01:57:27.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.900+0000 7fa587216640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa5800a1210 0x7fa5800a15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.900+0000 7fa587216640 10 monclient(hunting): picked mon.noname-c con 0x7fa5800a1210 addr v2:172.21.15.71:3300/0 2024-09-11T01:57:27.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.900+0000 7fa587216640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa5800a1b30 addr v1:172.21.15.12:6789/0 2024-09-11T01:57:27.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.900+0000 7fa587216640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa5800a03f0 addr v1:172.21.15.71:6789/0 2024-09-11T01:57:27.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.900+0000 7fa587216640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa580136a00 con 0x7fa5800a1b30 2024-09-11T01:57:27.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.900+0000 7fa587216640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5800a21c0 con 0x7fa5800a03f0 2024-09-11T01:57:27.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.900+0000 7fa587216640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:27.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.900+0000 7fa587216640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fa58009fdf0 con 0x7fa5800a1210 2024-09-11T01:57:27.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.900+0000 7fa587216640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:27.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.901+0000 7fa587216640 10 monclient(hunting): authenticate will time out at 2454.343009s 2024-09-11T01:57:27.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.901+0000 7fa57ffff640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fa5800a1b30 0x7fa580135be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33906/0 (socket says 172.21.15.12:33906) 2024-09-11T01:57:27.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.901+0000 7fa57ffff640 1 -- 172.21.15.12:0/3780676258 learned_addr learned my addr 172.21.15.12:0/3780676258 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:57:27.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.901+0000 7fa584fb2640 1 --2- 172.21.15.12:0/3780676258 >> v2:172.21.15.71:3300/0 conn(0x7fa5800a1210 0x7fa5800a15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.901+0000 7fa584fb2640 10 monclient(hunting): get_auth_request con 0x7fa5800a1210 auth_method 0 2024-09-11T01:57:27.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.901+0000 7fa584fb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:27.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.901+0000 7fa584fb2640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:27.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.901+0000 7fa584fb2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:27.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.901+0000 7fa584fb2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:27.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.901+0000 7fa584fb2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:27.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.901+0000 7fa584fb2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:27.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.901+0000 7fa584fb2640 10 monclient(hunting): handle_auth_done global_id 314155 payload 274 2024-09-11T01:57:27.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.901+0000 7fa584fb2640 1 -- 172.21.15.12:0/3780676258 >> v1:172.21.15.71:6789/0 conn(0x7fa5800a03f0 legacy=0x7fa5801362f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:27.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.901+0000 7fa584fb2640 1 -- 172.21.15.12:0/3780676258 >> v1:172.21.15.12:6789/0 conn(0x7fa5800a1b30 legacy=0x7fa580135be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:27.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.901+0000 7fa584fb2640 10 monclient: _finish_hunting 0 2024-09-11T01:57:27.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.901+0000 7fa584fb2640 1 monclient: found mon.noname-c 2024-09-11T01:57:27.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.901+0000 7fa584fb2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:27.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.901+0000 7fa584fb2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:57:27.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.901+0000 7fa584fb2640 1 -- 172.21.15.12:0/3780676258 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5800a24a0 con 0x7fa5800a1210 2024-09-11T01:57:27.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.904+0000 7fa584fb2640 1 -- 172.21.15.12:0/3780676258 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa5800a1210 msgr2=0x7fa5800a15f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:27.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.904+0000 7fa584fb2640 1 --2- 172.21.15.12:0/3780676258 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa5800a1210 0x7fa5800a15f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa5740097a0 tx=0x7fa574029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2388b4ff3a237675 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:27.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.905+0000 7fa57f7fe640 1 -- 172.21.15.12:0/3780676258 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa574037070 con 0x7fa5800a1210 2024-09-11T01:57:27.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.905+0000 7fa57f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:27.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.905+0000 7fa57f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:57:27.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.905+0000 7fa57f7fe640 10 monclient: dump: 2024-09-11T01:57:27.914 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:27.914 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:27.914 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.914 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.914 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:27.914 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:27.914 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:27.914 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:27.914 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:27.914 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:27.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.905+0000 7fa57f7fe640 10 monclient: _finish_auth 0 2024-09-11T01:57:27.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.905+0000 7fa57f7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:57:27.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.905+0000 7fa57f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:27.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.905+0000 7fa57f7fe640 1 -- 172.21.15.12:0/3780676258 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa574032640 con 0x7fa5800a1210 2024-09-11T01:57:27.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.905+0000 7fa587216640 5 monclient: authenticate success, global_id 314155 2024-09-11T01:57:27.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.905+0000 7fa587216640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:57:27.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.905+0000 7fa57f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:27.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.905+0000 7fa587216640 10 monclient: get_monmap_and_config success 2024-09-11T01:57:27.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.905+0000 7fa57f7fe640 1 -- 172.21.15.12:0/3780676258 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa574032a10 con 0x7fa5800a1210 2024-09-11T01:57:27.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.905+0000 7fa57f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:27.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.905+0000 7fa587216640 4 set_mon_vals no callback set 2024-09-11T01:57:27.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.905+0000 7fa57f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:57:27.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.905+0000 7fa57f7fe640 10 monclient: dump: 2024-09-11T01:57:27.915 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:27.915 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:27.915 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.916 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.916 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:27.916 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:27.916 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:27.916 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:27.916 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:27.916 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:27.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.905+0000 7fa587216640 10 monclient: shutdown 2024-09-11T01:57:27.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.905+0000 7fa587216640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:27.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.905+0000 7fa587216640 1 -- 172.21.15.12:0/3780676258 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa5800a1210 msgr2=0x7fa5800a15f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:27.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.905+0000 7fa587216640 1 --2- 172.21.15.12:0/3780676258 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa5800a1210 0x7fa5800a15f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa5740097a0 tx=0x7fa574029fd0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.905+0000 7fa587216640 1 -- 172.21.15.12:0/3780676258 shutdown_connections 2024-09-11T01:57:27.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.905+0000 7fa587216640 1 --2- 172.21.15.12:0/3780676258 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa5800a1210 0x7fa5800a15f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.905+0000 7fa587216640 1 -- 172.21.15.12:0/3780676258 >> 172.21.15.12:0/3780676258 conn(0x7fa5800a2f00 msgr2=0x7fa5800a3310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:27.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.905+0000 7fa587216640 1 -- 172.21.15.12:0/3780676258 shutdown_connections 2024-09-11T01:57:27.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.905+0000 7fa587216640 1 -- 172.21.15.12:0/3780676258 wait complete. 2024-09-11T01:57:27.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.906+0000 7fa587216640 10 monclient: build_initial_monmap 2024-09-11T01:57:27.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.906+0000 7fa587216640 10 monclient: monmap: 2024-09-11T01:57:27.917 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:27.917 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:57:27.917 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:57:27.917 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:57:27.917 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:27.917 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:27.917 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:57:27.917 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:57:27.918 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:57:27.918 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:27.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.906+0000 7fa587216640 1 Processor -- start 2024-09-11T01:57:27.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.906+0000 7fa587216640 1 -- start start 2024-09-11T01:57:27.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.906+0000 7fa587216640 10 monclient: init 2024-09-11T01:57:27.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa587216640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:27.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa587216640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:57:27.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa587216640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa5800a03f0 0x7fa58013dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa587216640 10 monclient(hunting): picked mon.noname-c con 0x7fa5800a03f0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:27.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa587216640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa5800a1210 0x7fa580135d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa587216640 10 monclient(hunting): picked mon.noname-b con 0x7fa5800a1210 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:57:27.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa587216640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5800a1b30 0x7fa5801362a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa587216640 10 monclient(hunting): picked mon.noname-a con 0x7fa5800a1b30 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:57:27.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa587216640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:27.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa587216640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa5800a01b0 con 0x7fa5800a1b30 2024-09-11T01:57:27.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa587216640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:27.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa587216640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa5800a0030 con 0x7fa5800a1210 2024-09-11T01:57:27.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa587216640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:27.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa587216640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fa58013ef30 con 0x7fa5800a03f0 2024-09-11T01:57:27.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa587216640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:27.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa587216640 10 monclient(hunting): authenticate will time out at 2454.349315s 2024-09-11T01:57:27.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa584fb2640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa5800a03f0 0x7fa58013dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa5857b3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5800a1b30 0x7fa5801362a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa5857b3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5800a1b30 0x7fa5801362a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54180/0 (socket says 172.21.15.12:54180) 2024-09-11T01:57:27.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa584fb2640 10 monclient(hunting): get_auth_request con 0x7fa5800a03f0 auth_method 0 2024-09-11T01:57:27.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa5857b3640 1 -- 172.21.15.12:0/4190851734 learned_addr learned my addr 172.21.15.12:0/4190851734 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:27.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa584fb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:27.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa584fb2640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:27.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa584fb2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:27.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa5857b3640 10 monclient(hunting): get_auth_request con 0x7fa5800a1b30 auth_method 0 2024-09-11T01:57:27.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa5857b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:27.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa5857b3640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:27.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa5857b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:27.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa584fb2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:27.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa584fb2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:27.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa584fb2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:27.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa5857b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:27.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa5857b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:27.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa5857b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:27.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa57ffff640 1 --2- 172.21.15.12:0/4190851734 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa5800a1210 0x7fa580135d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa57ffff640 10 monclient(hunting): get_auth_request con 0x7fa5800a1210 auth_method 0 2024-09-11T01:57:27.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa57ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:27.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa57ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:27.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa57ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:27.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa5857b3640 10 monclient(hunting): handle_auth_done global_id 294372 payload 824 2024-09-11T01:57:27.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa5857b3640 1 -- 172.21.15.12:0/4190851734 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa5800a03f0 msgr2=0x7fa58013dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:27.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa5857b3640 1 --2- 172.21.15.12:0/4190851734 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa5800a03f0 0x7fa58013dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa5857b3640 1 -- 172.21.15.12:0/4190851734 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa5800a1210 msgr2=0x7fa580135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:27.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa5857b3640 1 --2- 172.21.15.12:0/4190851734 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa5800a1210 0x7fa580135d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa5857b3640 10 monclient: _finish_hunting 0 2024-09-11T01:57:27.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa5857b3640 1 monclient: found mon.noname-a 2024-09-11T01:57:27.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa5857b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:27.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa5857b3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:57:27.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa5857b3640 1 -- 172.21.15.12:0/4190851734 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa58011e2b0 con 0x7fa5800a1b30 2024-09-11T01:57:27.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa57ffff640 1 --2- 172.21.15.12:0/4190851734 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa5800a1210 0x7fa580135d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:57:27.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.907+0000 7fa584fb2640 1 --2- 172.21.15.12:0/4190851734 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa5800a03f0 0x7fa58013dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:57:27.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.910+0000 7fa5857b3640 1 -- 172.21.15.12:0/4190851734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5800a1b30 msgr2=0x7fa5801362a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:27.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.910+0000 7fa5857b3640 1 --2- 172.21.15.12:0/4190851734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5800a1b30 0x7fa5801362a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa57800c700 tx=0x7fa57800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:27.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.910+0000 7fa57d7fa640 1 -- 172.21.15.12:0/4190851734 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa578016020 con 0x7fa5800a1b30 2024-09-11T01:57:27.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.910+0000 7fa57d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:27.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.910+0000 7fa57d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:57:27.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.910+0000 7fa57d7fa640 10 monclient: dump: 2024-09-11T01:57:27.922 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:27.922 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:27.922 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.922 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.922 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:27.923 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:27.923 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:27.923 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:27.923 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:27.923 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:27.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.910+0000 7fa57d7fa640 10 monclient: _finish_auth 0 2024-09-11T01:57:27.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.910+0000 7fa57d7fa640 10 monclient: _check_auth_tickets 2024-09-11T01:57:27.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.910+0000 7fa57d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:27.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.910+0000 7fa57d7fa640 1 -- 172.21.15.12:0/4190851734 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa57800ac00 con 0x7fa5800a1b30 2024-09-11T01:57:27.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.910+0000 7fa587216640 5 monclient: authenticate success, global_id 294372 2024-09-11T01:57:27.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.910+0000 7fa587216640 10 monclient: _renew_subs 2024-09-11T01:57:27.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.910+0000 7fa587216640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:27.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.910+0000 7fa587216640 1 -- 172.21.15.12:0/4190851734 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa58009d410 con 0x7fa5800a1b30 2024-09-11T01:57:27.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.910+0000 7fa57d7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:27.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.910+0000 7fa57d7fa640 1 -- 172.21.15.12:0/4190851734 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa57800ce10 con 0x7fa5800a1b30 2024-09-11T01:57:27.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.910+0000 7fa57d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:27.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.910+0000 7fa57d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:27.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.910+0000 7fa57d7fa640 10 monclient: dump: 2024-09-11T01:57:27.924 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:27.924 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:27.924 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.924 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:27.924 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:27.924 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:27.924 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:27.924 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:27.924 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:27.925 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:27.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.910+0000 7fa57effd640 4 set_mon_vals no callback set 2024-09-11T01:57:27.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.910+0000 7fa587216640 10 monclient: _renew_subs 2024-09-11T01:57:27.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.910+0000 7fa587216640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:27.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.910+0000 7fa587216640 1 -- 172.21.15.12:0/4190851734 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa58011f700 con 0x7fa5800a1b30 2024-09-11T01:57:27.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.911+0000 7fa57d7fa640 1 -- 172.21.15.12:0/4190851734 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fa578012420 con 0x7fa5800a1b30 2024-09-11T01:57:27.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.911+0000 7fa587216640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:27.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.911+0000 7fa587216640 1 -- 172.21.15.12:0/4190851734 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa580074d30 con 0x7fa5800a1b30 2024-09-11T01:57:27.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.911+0000 7fa57d7fa640 1 --2- 172.21.15.12:0/4190851734 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa56403e240 0x7fa564040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.911+0000 7fa57d7fa640 1 -- 172.21.15.12:0/4190851734 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(923..923 src has 889..923) ==== 4422+0+0 (secure 0 0 0) 0x7fa578052e80 con 0x7fa5800a1b30 2024-09-11T01:57:27.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.911+0000 7fa57d7fa640 1 --2- 172.21.15.12:0/4190851734 >> [v2:172.21.15.71:6818/3091488141,v1:172.21.15.71:6819/3091488141] conn(0x7fa564044130 0x7fa564046520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:27.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.911+0000 7fa57d7fa640 1 -- 172.21.15.12:0/4190851734 --> [v2:172.21.15.71:6818/3091488141,v1:172.21.15.71:6819/3091488141] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa57801eeb0 con 0x7fa564044130 2024-09-11T01:57:27.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.911+0000 7fa57d7fa640 1 -- 172.21.15.12:0/4190851734 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=923) ==== 24+0+0 (secure 0 0 0) 0x7fa578058410 con 0x7fa5800a1b30 2024-09-11T01:57:27.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.911+0000 7fa57d7fa640 10 monclient: handle_get_version_reply finishing 1 version 923 2024-09-11T01:57:27.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.911+0000 7fa584fb2640 1 --2- 172.21.15.12:0/4190851734 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa56403e240 0x7fa564040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.911+0000 7fa584fb2640 10 monclient: get_auth_request con 0x7fa56403e240 auth_method 0 2024-09-11T01:57:27.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.911+0000 7fa57ffff640 1 --2- 172.21.15.12:0/4190851734 >> [v2:172.21.15.71:6818/3091488141,v1:172.21.15.71:6819/3091488141] conn(0x7fa564044130 0x7fa564046520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:27.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.911+0000 7fa57ffff640 10 monclient: get_auth_request con 0x7fa564044130 auth_method 0 2024-09-11T01:57:27.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.914+0000 7fa584fb2640 1 --2- 172.21.15.12:0/4190851734 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa56403e240 0x7fa564040700 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa574031000 tx=0x7fa57402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:27.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.916+0000 7fa57ffff640 1 --2- 172.21.15.12:0/4190851734 >> [v2:172.21.15.71:6818/3091488141,v1:172.21.15.71:6819/3091488141] conn(0x7fa564044130 0x7fa564046520 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:27.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.917+0000 7fa57d7fa640 1 -- 172.21.15.12:0/4190851734 <== osd.3 v2:172.21.15.71:6818/3091488141 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa57801eeb0 con 0x7fa564044130 2024-09-11T01:57:27.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.930+0000 7fa587216640 1 -- 172.21.15.12:0/4190851734 --> [v2:172.21.15.71:6818/3091488141,v1:172.21.15.71:6819/3091488141] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa580074290 con 0x7fa564044130 2024-09-11T01:57:27.931 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.930+0000 7fa57d7fa640 1 -- 172.21.15.12:0/4190851734 <== osd.3 v2:172.21.15.71:6818/3091488141 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fa580074290 con 0x7fa564044130 2024-09-11T01:57:27.932 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:57:27.932 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T01:57:27.932 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:57:27.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.930+0000 7fa587216640 1 -- 172.21.15.12:0/4190851734 >> [v2:172.21.15.71:6818/3091488141,v1:172.21.15.71:6819/3091488141] conn(0x7fa564044130 msgr2=0x7fa564046520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:27.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.930+0000 7fa587216640 1 --2- 172.21.15.12:0/4190851734 >> [v2:172.21.15.71:6818/3091488141,v1:172.21.15.71:6819/3091488141] conn(0x7fa564044130 0x7fa564046520 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.931+0000 7fa587216640 1 -- 172.21.15.12:0/4190851734 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa56403e240 msgr2=0x7fa564040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:27.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.931+0000 7fa587216640 1 --2- 172.21.15.12:0/4190851734 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fa56403e240 0x7fa564040700 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa574031000 tx=0x7fa57402e000 comp rx=0 tx=0).stop 2024-09-11T01:57:27.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.931+0000 7fa587216640 10 monclient: shutdown 2024-09-11T01:57:27.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.931+0000 7fa587216640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:27.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.931+0000 7fa587216640 1 -- 172.21.15.12:0/4190851734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5800a1b30 msgr2=0x7fa5801362a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:27.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.931+0000 7fa587216640 1 --2- 172.21.15.12:0/4190851734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5800a1b30 0x7fa5801362a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa57800c700 tx=0x7fa57800cbc0 comp rx=0 tx=0).stop 2024-09-11T01:57:27.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.931+0000 7fa5857b3640 1 -- 172.21.15.12:0/4190851734 reap_dead start 2024-09-11T01:57:27.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.931+0000 7fa587216640 1 -- 172.21.15.12:0/4190851734 shutdown_connections 2024-09-11T01:57:27.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.931+0000 7fa587216640 1 -- 172.21.15.12:0/4190851734 >> 172.21.15.12:0/4190851734 conn(0x7fa5800a2f00 msgr2=0x7fa580071d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:27.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.931+0000 7fa587216640 1 -- 172.21.15.12:0/4190851734 shutdown_connections 2024-09-11T01:57:27.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:27.931+0000 7fa587216640 1 -- 172.21.15.12:0/4190851734 wait complete. 2024-09-11T01:57:27.942 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-11T01:57:28.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.042+0000 7f72b9216640 10 monclient: get_monmap_and_config 2024-09-11T01:57:28.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.042+0000 7f72b9216640 10 monclient: build_initial_monmap 2024-09-11T01:57:28.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.042+0000 7f72b9216640 10 monclient: monmap: 2024-09-11T01:57:28.047 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:28.047 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:28.047 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:57:28.043445+0000 2024-09-11T01:57:28.047 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:57:28.043445+0000 2024-09-11T01:57:28.047 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:28.047 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.047 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:57:28.047 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:57:28.047 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:57:28.048 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:57:28.048 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:57:28.048 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.043+0000 7f72b9216640 1 Processor -- start 2024-09-11T01:57:28.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.043+0000 7f72b9216640 1 -- start start 2024-09-11T01:57:28.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.043+0000 7f72b9216640 10 monclient: init 2024-09-11T01:57:28.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.043+0000 7f72b9216640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:28.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.043+0000 7f72b9216640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-11T01:57:28.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.043+0000 7f72b9216640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72b40a2e20 0x7f72b406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.043+0000 7f72b9216640 10 monclient(hunting): picked mon.noname-b con 0x7f72b40a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:28.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.043+0000 7f72b9216640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f72b40a4970 addr v1:172.21.15.71:6789/0 2024-09-11T01:57:28.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.043+0000 7f72b9216640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f72b406f910 0x7f72b413f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.043+0000 7f72b9216640 10 monclient(hunting): picked mon.noname-c con 0x7f72b406f910 addr v2:172.21.15.71:3300/0 2024-09-11T01:57:28.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.043+0000 7f72b9216640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72b406f4f0 con 0x7f72b40a4970 2024-09-11T01:57:28.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.043+0000 7f72b9216640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:28.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.043+0000 7f72b9216640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f72b406ef30 con 0x7f72b406f910 2024-09-11T01:57:28.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.043+0000 7f72b9216640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:28.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.043+0000 7f72b9216640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f72b406ec50 con 0x7f72b40a2e20 2024-09-11T01:57:28.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b9216640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:28.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b9216640 10 monclient(hunting): authenticate will time out at 2454.486014s 2024-09-11T01:57:28.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b2d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72b40a2e20 0x7f72b406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b2d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72b40a2e20 0x7f72b406ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46608/0 (socket says 172.21.15.12:46608) 2024-09-11T01:57:28.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b2d9d640 1 -- 172.21.15.12:0/1603218955 learned_addr learned my addr 172.21.15.12:0/1603218955 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:28.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b2d9d640 10 monclient(hunting): get_auth_request con 0x7f72b40a2e20 auth_method 0 2024-09-11T01:57:28.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:28.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:28.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:28.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:28.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:28.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:28.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b359e640 1 --2- 172.21.15.12:0/1603218955 >> v2:172.21.15.71:3300/0 conn(0x7f72b406f910 0x7f72b413f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b359e640 10 monclient(hunting): get_auth_request con 0x7f72b406f910 auth_method 0 2024-09-11T01:57:28.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:28.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b359e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:28.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:28.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:28.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:28.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:28.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b2d9d640 10 monclient(hunting): handle_auth_done global_id 304268 payload 274 2024-09-11T01:57:28.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b2d9d640 1 -- 172.21.15.12:0/1603218955 >> v2:172.21.15.71:3300/0 conn(0x7f72b406f910 msgr2=0x7f72b413f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:28.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b2d9d640 1 --2- 172.21.15.12:0/1603218955 >> v2:172.21.15.71:3300/0 conn(0x7f72b406f910 0x7f72b413f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b2d9d640 1 -- 172.21.15.12:0/1603218955 >> v1:172.21.15.71:6789/0 conn(0x7f72b40a4970 legacy=0x7f72b40a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:28.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b2d9d640 10 monclient: _finish_hunting 0 2024-09-11T01:57:28.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b2d9d640 1 monclient: found mon.noname-b 2024-09-11T01:57:28.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:28.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:57:28.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.044+0000 7f72b2d9d640 1 -- 172.21.15.12:0/1603218955 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72b40a3910 con 0x7f72b40a2e20 2024-09-11T01:57:28.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.047+0000 7f72b2d9d640 1 -- 172.21.15.12:0/1603218955 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72b40a2e20 msgr2=0x7f72b406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:28.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.047+0000 7f72b2d9d640 1 --2- 172.21.15.12:0/1603218955 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72b40a2e20 0x7f72b406ded0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f72a4009870 tx=0x7f72a402a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=963bde70902443aa server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:28.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.047+0000 7f72b259c640 1 -- 172.21.15.12:0/1603218955 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a4037070 con 0x7f72b40a2e20 2024-09-11T01:57:28.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.047+0000 7f72b259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:28.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.047+0000 7f72b259c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:57:28.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.047+0000 7f72b259c640 10 monclient: dump: 2024-09-11T01:57:28.057 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:28.057 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:28.057 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.057 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.057 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:28.057 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.057 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:28.057 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:28.057 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:28.057 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.047+0000 7f72b259c640 10 monclient: _finish_auth 0 2024-09-11T01:57:28.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.047+0000 7f72b259c640 10 monclient: _check_auth_tickets 2024-09-11T01:57:28.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.047+0000 7f72b259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:28.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.047+0000 7f72b259c640 1 -- 172.21.15.12:0/1603218955 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72a4032440 con 0x7f72b40a2e20 2024-09-11T01:57:28.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.047+0000 7f72b9216640 5 monclient: authenticate success, global_id 304268 2024-09-11T01:57:28.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.047+0000 7f72b9216640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:57:28.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.047+0000 7f72b259c640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:28.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.047+0000 7f72b9216640 10 monclient: get_monmap_and_config success 2024-09-11T01:57:28.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.047+0000 7f72b259c640 1 -- 172.21.15.12:0/1603218955 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a4032810 con 0x7f72b40a2e20 2024-09-11T01:57:28.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.047+0000 7f72b9216640 4 set_mon_vals no callback set 2024-09-11T01:57:28.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.047+0000 7f72b259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:28.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.047+0000 7f72b259c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:57:28.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.047+0000 7f72b259c640 10 monclient: dump: 2024-09-11T01:57:28.058 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:28.058 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:28.059 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.059 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.059 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:28.059 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.059 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:28.059 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:28.059 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:28.059 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.047+0000 7f72b9216640 10 monclient: shutdown 2024-09-11T01:57:28.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.047+0000 7f72b9216640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:28.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.047+0000 7f72b9216640 1 -- 172.21.15.12:0/1603218955 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72b40a2e20 msgr2=0x7f72b406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:28.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.047+0000 7f72b9216640 1 --2- 172.21.15.12:0/1603218955 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72b40a2e20 0x7f72b406ded0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f72a4009870 tx=0x7f72a402a0a0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.047+0000 7f72b9216640 1 -- 172.21.15.12:0/1603218955 shutdown_connections 2024-09-11T01:57:28.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.047+0000 7f72b9216640 1 --2- 172.21.15.12:0/1603218955 >> v2:172.21.15.71:3300/0 conn(0x7f72b406f910 0x7f72b413f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.047+0000 7f72b9216640 1 --2- 172.21.15.12:0/1603218955 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72b40a2e20 0x7f72b406ded0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.048+0000 7f72b9216640 1 -- 172.21.15.12:0/1603218955 >> 172.21.15.12:0/1603218955 conn(0x7f72b40a8600 msgr2=0x7f72b40a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:28.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.048+0000 7f72b9216640 1 -- 172.21.15.12:0/1603218955 shutdown_connections 2024-09-11T01:57:28.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.048+0000 7f72b9216640 1 -- 172.21.15.12:0/1603218955 wait complete. 2024-09-11T01:57:28.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.048+0000 7f72b9216640 10 monclient: build_initial_monmap 2024-09-11T01:57:28.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.048+0000 7f72b9216640 10 monclient: monmap: 2024-09-11T01:57:28.060 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:28.060 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:57:28.060 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:57:28.060 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:57:28.060 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:28.060 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.061 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:57:28.061 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:57:28.061 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:57:28.061 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b9216640 1 Processor -- start 2024-09-11T01:57:28.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b9216640 1 -- start start 2024-09-11T01:57:28.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b9216640 10 monclient: init 2024-09-11T01:57:28.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b9216640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:28.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b9216640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:57:28.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b9216640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72b406f910 0x7f72b409e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b9216640 10 monclient(hunting): picked mon.noname-b con 0x7f72b406f910 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:57:28.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b9216640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72b40a2e20 0x7f72b413e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b9216640 10 monclient(hunting): picked mon.noname-c con 0x7f72b40a2e20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:28.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b9216640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72b40a4970 0x7f72b413ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b9216640 10 monclient(hunting): picked mon.noname-a con 0x7f72b40a4970 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:57:28.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b9216640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:28.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b9216640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f72b413fd00 con 0x7f72b40a4970 2024-09-11T01:57:28.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b9216640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:28.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b9216640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f72b413fb80 con 0x7f72b406f910 2024-09-11T01:57:28.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b9216640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:28.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b9216640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f72b413fe80 con 0x7f72b40a2e20 2024-09-11T01:57:28.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b9216640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:28.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b9216640 10 monclient(hunting): authenticate will time out at 2454.491597s 2024-09-11T01:57:28.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72abfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72b40a2e20 0x7f72b413e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b359e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72b40a4970 0x7f72b413ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72abfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72b40a2e20 0x7f72b413e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46622/0 (socket says 172.21.15.12:46622) 2024-09-11T01:57:28.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72abfff640 1 -- 172.21.15.12:0/1568309313 learned_addr learned my addr 172.21.15.12:0/1568309313 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:28.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72abfff640 10 monclient(hunting): get_auth_request con 0x7f72b40a2e20 auth_method 0 2024-09-11T01:57:28.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:28.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72abfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:28.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:28.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b359e640 10 monclient(hunting): get_auth_request con 0x7f72b40a4970 auth_method 0 2024-09-11T01:57:28.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:28.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b359e640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:28.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:28.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b2d9d640 1 --2- 172.21.15.12:0/1568309313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72b406f910 0x7f72b409e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:28.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:28.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:28.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:28.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:28.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:28.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b2d9d640 10 monclient(hunting): get_auth_request con 0x7f72b406f910 auth_method 0 2024-09-11T01:57:28.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:28.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:28.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:28.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:28.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:28.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.049+0000 7f72b2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:28.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.050+0000 7f72abfff640 10 monclient(hunting): handle_auth_done global_id 304271 payload 824 2024-09-11T01:57:28.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.050+0000 7f72abfff640 1 -- 172.21.15.12:0/1568309313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72b406f910 msgr2=0x7f72b409e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:28.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.050+0000 7f72abfff640 1 --2- 172.21.15.12:0/1568309313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72b406f910 0x7f72b409e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.050+0000 7f72abfff640 1 -- 172.21.15.12:0/1568309313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72b40a4970 msgr2=0x7f72b413ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:28.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.050+0000 7f72abfff640 1 --2- 172.21.15.12:0/1568309313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72b40a4970 0x7f72b413ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.050+0000 7f72abfff640 10 monclient: _finish_hunting 0 2024-09-11T01:57:28.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.050+0000 7f72abfff640 1 monclient: found mon.noname-c 2024-09-11T01:57:28.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.050+0000 7f72abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:28.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.050+0000 7f72abfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:57:28.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.050+0000 7f72abfff640 1 -- 172.21.15.12:0/1568309313 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72b409eee0 con 0x7f72b40a2e20 2024-09-11T01:57:28.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.050+0000 7f72b359e640 1 --2- 172.21.15.12:0/1568309313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72b40a4970 0x7f72b413ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:57:28.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.052+0000 7f72abfff640 1 -- 172.21.15.12:0/1568309313 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72b40a2e20 msgr2=0x7f72b413e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:28.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.052+0000 7f72abfff640 1 --2- 172.21.15.12:0/1568309313 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72b40a2e20 0x7f72b413e690 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f729c00c430 tx=0x7f729c00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:28.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.052+0000 7f72ab7fe640 1 -- 172.21.15.12:0/1568309313 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729c0119a0 con 0x7f72b40a2e20 2024-09-11T01:57:28.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.052+0000 7f72ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:28.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.052+0000 7f72ab7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:57:28.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.052+0000 7f72ab7fe640 10 monclient: dump: 2024-09-11T01:57:28.065 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:28.065 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:28.065 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.066 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.066 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:28.066 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.066 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:28.066 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:28.066 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:28.066 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.052+0000 7f72ab7fe640 10 monclient: _finish_auth 0 2024-09-11T01:57:28.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.052+0000 7f72ab7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:57:28.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.052+0000 7f72ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:28.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.052+0000 7f72ab7fe640 1 -- 172.21.15.12:0/1568309313 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f729c017070 con 0x7f72b40a2e20 2024-09-11T01:57:28.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.052+0000 7f72b9216640 5 monclient: authenticate success, global_id 304271 2024-09-11T01:57:28.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.052+0000 7f72b9216640 10 monclient: _renew_subs 2024-09-11T01:57:28.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.052+0000 7f72b9216640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:28.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.052+0000 7f72b9216640 1 -- 172.21.15.12:0/1568309313 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72b413a970 con 0x7f72b40a2e20 2024-09-11T01:57:28.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.052+0000 7f72ab7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:28.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.052+0000 7f72ab7fe640 1 -- 172.21.15.12:0/1568309313 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729c011d80 con 0x7f72b40a2e20 2024-09-11T01:57:28.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.052+0000 7f72ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:28.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.052+0000 7f72ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:57:28.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.052+0000 7f72ab7fe640 10 monclient: dump: 2024-09-11T01:57:28.067 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:28.067 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:28.067 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.067 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.067 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:28.067 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.067 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:28.068 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:28.068 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:28.068 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.052+0000 7f72b9216640 10 monclient: _renew_subs 2024-09-11T01:57:28.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.052+0000 7f72b1d9b640 4 set_mon_vals no callback set 2024-09-11T01:57:28.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.052+0000 7f72b9216640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:28.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.052+0000 7f72b9216640 1 -- 172.21.15.12:0/1568309313 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f72b413b8d0 con 0x7f72b40a2e20 2024-09-11T01:57:28.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.053+0000 7f72ab7fe640 1 -- 172.21.15.12:0/1568309313 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f729c01c460 con 0x7f72b40a2e20 2024-09-11T01:57:28.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.053+0000 7f72b9216640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:28.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.053+0000 7f72b9216640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:28.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.053+0000 7f72b9216640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:28.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.053+0000 7f72b9216640 1 -- 172.21.15.12:0/1568309313 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72b406ded0 con 0x7f72b40a2e20 2024-09-11T01:57:28.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.053+0000 7f72ab7fe640 1 --2- 172.21.15.12:0/1568309313 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f729403e260 0x7f7294040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.053+0000 7f72ab7fe640 1 -- 172.21.15.12:0/1568309313 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(923..923 src has 889..923) ==== 4422+0+0 (secure 0 0 0) 0x7f729c052a70 con 0x7f72b40a2e20 2024-09-11T01:57:28.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.054+0000 7f72b2d9d640 1 --2- 172.21.15.12:0/1568309313 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f729403e260 0x7f7294040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.054+0000 7f72b2d9d640 10 monclient: get_auth_request con 0x7f729403e260 auth_method 0 2024-09-11T01:57:28.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.056+0000 7f72b2d9d640 1 --2- 172.21.15.12:0/1568309313 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f729403e260 0x7f7294040720 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f72b4070c50 tx=0x7f72a402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:28.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.058+0000 7f72ab7fe640 1 -- 172.21.15.12:0/1568309313 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f729c01ac60 con 0x7f72b40a2e20 2024-09-11T01:57:28.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.058+0000 7f72ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:28.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.058+0000 7f72ab7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:57:28.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.240+0000 7f72b9216640 1 -- 172.21.15.12:0/1568309313 --> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f72b413bc10 con 0x7f729403e260 2024-09-11T01:57:28.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.240+0000 7f72ab7fe640 1 -- 172.21.15.12:0/1568309313 <== mgr.4101 v2:172.21.15.71:6816/3189259519 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f72b413bc10 con 0x7f729403e260 2024-09-11T01:57:28.242 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 3 to deep-scrub 2024-09-11T01:57:28.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.243+0000 7f72b9216640 1 -- 172.21.15.12:0/1568309313 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f729403e260 msgr2=0x7f7294040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:28.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.243+0000 7f72b9216640 1 --2- 172.21.15.12:0/1568309313 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f729403e260 0x7f7294040720 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f72b4070c50 tx=0x7f72a402e000 comp rx=0 tx=0).stop 2024-09-11T01:57:28.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.243+0000 7f72b9216640 10 monclient: shutdown 2024-09-11T01:57:28.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.243+0000 7f72b9216640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:28.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.243+0000 7f72b9216640 1 -- 172.21.15.12:0/1568309313 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72b40a2e20 msgr2=0x7f72b413e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:28.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.243+0000 7f72b9216640 1 --2- 172.21.15.12:0/1568309313 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72b40a2e20 0x7f72b413e690 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f729c00c430 tx=0x7f729c00c8f0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.243+0000 7f72b9216640 1 -- 172.21.15.12:0/1568309313 shutdown_connections 2024-09-11T01:57:28.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.243+0000 7f72b9216640 1 --2- 172.21.15.12:0/1568309313 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f729403e260 0x7f7294040720 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.243+0000 7f72b9216640 1 --2- 172.21.15.12:0/1568309313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72b40a4970 0x7f72b413ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.245 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.243+0000 7f72b9216640 1 --2- 172.21.15.12:0/1568309313 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f72b40a2e20 0x7f72b413e690 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.243+0000 7f72b9216640 1 --2- 172.21.15.12:0/1568309313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72b406f910 0x7f72b409e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.243+0000 7f72b9216640 1 -- 172.21.15.12:0/1568309313 >> 172.21.15.12:0/1568309313 conn(0x7f72b40a8600 msgr2=0x7f72b41382f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:28.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.243+0000 7f72b9216640 1 -- 172.21.15.12:0/1568309313 shutdown_connections 2024-09-11T01:57:28.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.243+0000 7f72b9216640 1 -- 172.21.15.12:0/1568309313 wait complete. 2024-09-11T01:57:28.255 INFO:tasks.ceph:Scrubbing osd.4 2024-09-11T01:57:28.256 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-11T01:57:28.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.360+0000 7f7a22447640 10 monclient: get_monmap_and_config 2024-09-11T01:57:28.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.360+0000 7f7a22447640 10 monclient: build_initial_monmap 2024-09-11T01:57:28.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.360+0000 7f7a22447640 10 monclient: monmap: 2024-09-11T01:57:28.362 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:28.362 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:28.362 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:57:28.361074+0000 2024-09-11T01:57:28.362 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:57:28.361074+0000 2024-09-11T01:57:28.362 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:28.362 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.362 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:57:28.362 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:57:28.362 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:57:28.362 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:57:28.363 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:57:28.363 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.361+0000 7f7a22447640 1 Processor -- start 2024-09-11T01:57:28.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.361+0000 7f7a22447640 1 -- start start 2024-09-11T01:57:28.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.361+0000 7f7a22447640 10 monclient: init 2024-09-11T01:57:28.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.361+0000 7f7a22447640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:28.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.361+0000 7f7a22447640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-11T01:57:28.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.361+0000 7f7a22447640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a1c137ef0 0x7f7a1c1382d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.361+0000 7f7a22447640 10 monclient(hunting): picked mon.noname-b con 0x7f7a1c137ef0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:28.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.361+0000 7f7a22447640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7a1c138810 addr v1:172.21.15.71:6789/0 2024-09-11T01:57:28.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.361+0000 7f7a22447640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7a1c0a3450 addr v1:172.21.15.12:6789/0 2024-09-11T01:57:28.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.361+0000 7f7a22447640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a1c06f090 con 0x7f7a1c0a3450 2024-09-11T01:57:28.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.361+0000 7f7a22447640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a1c0a4760 con 0x7f7a1c138810 2024-09-11T01:57:28.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.361+0000 7f7a22447640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:28.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.361+0000 7f7a22447640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7a1c09c8c0 con 0x7f7a1c137ef0 2024-09-11T01:57:28.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.361+0000 7f7a22447640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:28.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.361+0000 7f7a22447640 10 monclient(hunting): authenticate will time out at 2454.803646s 2024-09-11T01:57:28.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.361+0000 7f7a1bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a1c137ef0 0x7f7a1c1382d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.361+0000 7f7a1bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a1c137ef0 0x7f7a1c1382d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46638/0 (socket says 172.21.15.12:46638) 2024-09-11T01:57:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.361+0000 7f7a209e4640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7a1c0a3450 0x7f7a1c0a3860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33912/0 (socket says 172.21.15.12:33912) 2024-09-11T01:57:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.361+0000 7f7a1bfff640 1 -- 172.21.15.12:0/844190630 learned_addr learned my addr 172.21.15.12:0/844190630 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.361+0000 7f7a1bfff640 10 monclient(hunting): get_auth_request con 0x7f7a1c137ef0 auth_method 0 2024-09-11T01:57:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.361+0000 7f7a1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.361+0000 7f7a1bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.361+0000 7f7a1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.361+0000 7f7a1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.361+0000 7f7a1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.361+0000 7f7a1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.362+0000 7f7a1bfff640 10 monclient(hunting): handle_auth_done global_id 304274 payload 274 2024-09-11T01:57:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.362+0000 7f7a1bfff640 1 -- 172.21.15.12:0/844190630 >> v1:172.21.15.71:6789/0 conn(0x7f7a1c138810 legacy=0x7f7a1c0a2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.362+0000 7f7a1bfff640 1 -- 172.21.15.12:0/844190630 >> v1:172.21.15.12:6789/0 conn(0x7f7a1c0a3450 legacy=0x7f7a1c0a3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:28.366 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.362+0000 7f7a1bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:57:28.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.362+0000 7f7a1bfff640 1 monclient: found mon.noname-b 2024-09-11T01:57:28.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.362+0000 7f7a1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:28.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.362+0000 7f7a1bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:57:28.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.362+0000 7f7a1bfff640 1 -- 172.21.15.12:0/844190630 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a1c06edb0 con 0x7f7a1c137ef0 2024-09-11T01:57:28.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.364+0000 7f7a1bfff640 1 -- 172.21.15.12:0/844190630 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a1c137ef0 msgr2=0x7f7a1c1382d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:28.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.364+0000 7f7a1bfff640 1 --2- 172.21.15.12:0/844190630 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a1c137ef0 0x7f7a1c1382d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c0098a0 tx=0x7f7a0c02a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e6b913cb66c4e9e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:28.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.365+0000 7f7a1affd640 1 -- 172.21.15.12:0/844190630 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c037070 con 0x7f7a1c137ef0 2024-09-11T01:57:28.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.365+0000 7f7a1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:28.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.365+0000 7f7a1affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:57:28.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.365+0000 7f7a1affd640 10 monclient: dump: 2024-09-11T01:57:28.374 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:28.374 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:28.374 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.374 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.374 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:28.374 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.374 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:28.374 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:28.374 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:28.374 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.365+0000 7f7a1affd640 10 monclient: _finish_auth 0 2024-09-11T01:57:28.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.365+0000 7f7a1affd640 10 monclient: _check_auth_tickets 2024-09-11T01:57:28.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.365+0000 7f7a1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:28.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.365+0000 7f7a1affd640 1 -- 172.21.15.12:0/844190630 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a0c032640 con 0x7f7a1c137ef0 2024-09-11T01:57:28.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.365+0000 7f7a22447640 5 monclient: authenticate success, global_id 304274 2024-09-11T01:57:28.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.365+0000 7f7a22447640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:57:28.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.365+0000 7f7a1affd640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:28.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.365+0000 7f7a1affd640 1 -- 172.21.15.12:0/844190630 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c032a10 con 0x7f7a1c137ef0 2024-09-11T01:57:28.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.365+0000 7f7a22447640 10 monclient: get_monmap_and_config success 2024-09-11T01:57:28.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.365+0000 7f7a22447640 4 set_mon_vals no callback set 2024-09-11T01:57:28.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.365+0000 7f7a1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:28.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.365+0000 7f7a1affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:57:28.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.365+0000 7f7a1affd640 10 monclient: dump: 2024-09-11T01:57:28.375 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:28.376 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:28.376 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.376 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.376 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:28.376 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.376 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:28.376 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:28.376 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:28.376 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.365+0000 7f7a22447640 10 monclient: shutdown 2024-09-11T01:57:28.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.365+0000 7f7a22447640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:28.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.365+0000 7f7a22447640 1 -- 172.21.15.12:0/844190630 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a1c137ef0 msgr2=0x7f7a1c1382d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:28.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.365+0000 7f7a22447640 1 --2- 172.21.15.12:0/844190630 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a1c137ef0 0x7f7a1c1382d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c0098a0 tx=0x7f7a0c02a100 comp rx=0 tx=0).stop 2024-09-11T01:57:28.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.365+0000 7f7a22447640 1 -- 172.21.15.12:0/844190630 shutdown_connections 2024-09-11T01:57:28.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.365+0000 7f7a22447640 1 --2- 172.21.15.12:0/844190630 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a1c137ef0 0x7f7a1c1382d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.365+0000 7f7a22447640 1 -- 172.21.15.12:0/844190630 >> 172.21.15.12:0/844190630 conn(0x7f7a1c0a8340 msgr2=0x7f7a1c0a8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:28.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.365+0000 7f7a22447640 1 -- 172.21.15.12:0/844190630 shutdown_connections 2024-09-11T01:57:28.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.365+0000 7f7a22447640 1 -- 172.21.15.12:0/844190630 wait complete. 2024-09-11T01:57:28.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.366+0000 7f7a22447640 10 monclient: build_initial_monmap 2024-09-11T01:57:28.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.366+0000 7f7a22447640 10 monclient: monmap: 2024-09-11T01:57:28.377 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:28.377 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:57:28.377 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:57:28.377 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:57:28.377 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:28.377 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.377 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:57:28.377 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:57:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:57:28.378 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.366+0000 7f7a22447640 1 Processor -- start 2024-09-11T01:57:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.366+0000 7f7a22447640 1 -- start start 2024-09-11T01:57:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.366+0000 7f7a22447640 10 monclient: init 2024-09-11T01:57:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a22447640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a22447640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:57:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a22447640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a1c0a3450 0x7f7a1c1cd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a22447640 10 monclient(hunting): picked mon.noname-c con 0x7f7a1c0a3450 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a22447640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a1c137ef0 0x7f7a1c1cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a22447640 10 monclient(hunting): picked mon.noname-b con 0x7f7a1c137ef0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:57:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a22447640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a1c138810 0x7f7a1c124810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a22447640 10 monclient(hunting): picked mon.noname-a con 0x7f7a1c138810 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:57:28.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a22447640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:28.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a22447640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1c0a3f70 con 0x7f7a1c138810 2024-09-11T01:57:28.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a22447640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:28.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a22447640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1c06ddf0 con 0x7f7a1c137ef0 2024-09-11T01:57:28.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a22447640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:28.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a22447640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f7a1c0a40f0 con 0x7f7a1c0a3450 2024-09-11T01:57:28.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a22447640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:28.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a22447640 10 monclient(hunting): authenticate will time out at 2454.809344s 2024-09-11T01:57:28.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a1c0a3450 0x7f7a1c1cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a1c0a3450 0x7f7a1c1cd390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46644/0 (socket says 172.21.15.12:46644) 2024-09-11T01:57:28.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1bfff640 1 -- 172.21.15.12:0/3668679715 learned_addr learned my addr 172.21.15.12:0/3668679715 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:28.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1bfff640 10 monclient(hunting): get_auth_request con 0x7f7a1c0a3450 auth_method 0 2024-09-11T01:57:28.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:28.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:28.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a209e4640 1 --2- 172.21.15.12:0/3668679715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a1c138810 0x7f7a1c124810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a209e4640 10 monclient(hunting): get_auth_request con 0x7f7a1c138810 auth_method 0 2024-09-11T01:57:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a209e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a209e4640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a209e4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1b7fe640 1 --2- 172.21.15.12:0/3668679715 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a1c137ef0 0x7f7a1c1cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1b7fe640 10 monclient(hunting): get_auth_request con 0x7f7a1c137ef0 auth_method 0 2024-09-11T01:57:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:28.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:28.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:28.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a209e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:28.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a209e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:28.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a209e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:28.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:28.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:28.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:28.381 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1bfff640 10 monclient(hunting): handle_auth_done global_id 304277 payload 824 2024-09-11T01:57:28.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1bfff640 1 -- 172.21.15.12:0/3668679715 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a1c137ef0 msgr2=0x7f7a1c1cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:28.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1bfff640 1 --2- 172.21.15.12:0/3668679715 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a1c137ef0 0x7f7a1c1cd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1bfff640 1 -- 172.21.15.12:0/3668679715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a1c138810 msgr2=0x7f7a1c124810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:28.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1bfff640 1 --2- 172.21.15.12:0/3668679715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7a1c138810 0x7f7a1c124810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:57:28.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1bfff640 1 monclient: found mon.noname-c 2024-09-11T01:57:28.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:28.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:57:28.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.367+0000 7f7a1bfff640 1 -- 172.21.15.12:0/3668679715 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a1c1262d0 con 0x7f7a1c0a3450 2024-09-11T01:57:28.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.369+0000 7f7a1bfff640 1 -- 172.21.15.12:0/3668679715 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a1c0a3450 msgr2=0x7f7a1c1cd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:28.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.369+0000 7f7a1bfff640 1 --2- 172.21.15.12:0/3668679715 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a1c0a3450 0x7f7a1c1cd390 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c0090b0 tx=0x7f7a0c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:28.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.370+0000 7f7a18ff9640 1 -- 172.21.15.12:0/3668679715 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c041070 con 0x7f7a1c0a3450 2024-09-11T01:57:28.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.370+0000 7f7a18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:28.382 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.370+0000 7f7a18ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:57:28.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.370+0000 7f7a18ff9640 10 monclient: dump: 2024-09-11T01:57:28.383 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:28.383 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:28.383 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.383 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.383 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:28.383 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.383 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:28.383 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:28.383 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:28.383 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.370+0000 7f7a18ff9640 10 monclient: _finish_auth 0 2024-09-11T01:57:28.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.370+0000 7f7a18ff9640 10 monclient: _check_auth_tickets 2024-09-11T01:57:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.370+0000 7f7a18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.370+0000 7f7a18ff9640 1 -- 172.21.15.12:0/3668679715 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a0c03c880 con 0x7f7a1c0a3450 2024-09-11T01:57:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.370+0000 7f7a18ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.370+0000 7f7a18ff9640 1 -- 172.21.15.12:0/3668679715 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c037040 con 0x7f7a1c0a3450 2024-09-11T01:57:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.370+0000 7f7a18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.370+0000 7f7a18ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:57:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.370+0000 7f7a18ff9640 10 monclient: dump: 2024-09-11T01:57:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:28.384 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:28.385 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.370+0000 7f7a22447640 5 monclient: authenticate success, global_id 304277 2024-09-11T01:57:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.370+0000 7f7a22447640 10 monclient: _renew_subs 2024-09-11T01:57:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.370+0000 7f7a22447640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.370+0000 7f7a22447640 1 -- 172.21.15.12:0/3668679715 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a1c137bf0 con 0x7f7a1c0a3450 2024-09-11T01:57:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.370+0000 7f7a1a7fc640 4 set_mon_vals no callback set 2024-09-11T01:57:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.370+0000 7f7a22447640 10 monclient: _renew_subs 2024-09-11T01:57:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.370+0000 7f7a22447640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.370+0000 7f7a22447640 1 -- 172.21.15.12:0/3668679715 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a1c128070 con 0x7f7a1c0a3450 2024-09-11T01:57:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.371+0000 7f7a22447640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.371+0000 7f7a22447640 1 -- 172.21.15.12:0/3668679715 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7a1c1382d0 con 0x7f7a1c0a3450 2024-09-11T01:57:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.371+0000 7f7a18ff9640 1 -- 172.21.15.12:0/3668679715 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f7a0c03cc70 con 0x7f7a1c0a3450 2024-09-11T01:57:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.371+0000 7f7a18ff9640 1 --2- 172.21.15.12:0/3668679715 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7a0803e260 0x7f7a08040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.371+0000 7f7a18ff9640 1 -- 172.21.15.12:0/3668679715 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(923..923 src has 889..923) ==== 4422+0+0 (secure 0 0 0) 0x7f7a0c073560 con 0x7f7a1c0a3450 2024-09-11T01:57:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.371+0000 7f7a18ff9640 1 --2- 172.21.15.12:0/3668679715 >> [v2:172.21.15.71:6800/1790415427,v1:172.21.15.71:6801/1790415427] conn(0x7f7a080441d0 0x7f7a080465c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.371+0000 7f7a18ff9640 1 -- 172.21.15.12:0/3668679715 --> [v2:172.21.15.71:6800/1790415427,v1:172.21.15.71:6801/1790415427] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7a0c041e70 con 0x7f7a080441d0 2024-09-11T01:57:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.371+0000 7f7a18ff9640 1 -- 172.21.15.12:0/3668679715 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_get_version_reply(handle=1 version=923) ==== 24+0+0 (secure 0 0 0) 0x7f7a0c07a050 con 0x7f7a1c0a3450 2024-09-11T01:57:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.371+0000 7f7a18ff9640 10 monclient: handle_get_version_reply finishing 1 version 923 2024-09-11T01:57:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.371+0000 7f7a1b7fe640 1 --2- 172.21.15.12:0/3668679715 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7a0803e260 0x7f7a08040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.371+0000 7f7a1b7fe640 10 monclient: get_auth_request con 0x7f7a0803e260 auth_method 0 2024-09-11T01:57:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.371+0000 7f7a209e4640 1 --2- 172.21.15.12:0/3668679715 >> [v2:172.21.15.71:6800/1790415427,v1:172.21.15.71:6801/1790415427] conn(0x7f7a080441d0 0x7f7a080465c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.371+0000 7f7a209e4640 10 monclient: get_auth_request con 0x7f7a080441d0 auth_method 0 2024-09-11T01:57:28.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.374+0000 7f7a1b7fe640 1 --2- 172.21.15.12:0/3668679715 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7a0803e260 0x7f7a08040720 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f7a10004f60 tx=0x7f7a10007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:28.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.376+0000 7f7a209e4640 1 --2- 172.21.15.12:0/3668679715 >> [v2:172.21.15.71:6800/1790415427,v1:172.21.15.71:6801/1790415427] conn(0x7f7a080441d0 0x7f7a080465c0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:28.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.377+0000 7f7a18ff9640 1 -- 172.21.15.12:0/3668679715 <== osd.4 v2:172.21.15.71:6800/1790415427 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7a0c041e70 con 0x7f7a080441d0 2024-09-11T01:57:28.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.389+0000 7f7a22447640 1 -- 172.21.15.12:0/3668679715 --> [v2:172.21.15.71:6800/1790415427,v1:172.21.15.71:6801/1790415427] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f7a1c1284e0 con 0x7f7a080441d0 2024-09-11T01:57:28.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.390+0000 7f7a18ff9640 1 -- 172.21.15.12:0/3668679715 <== osd.4 v2:172.21.15.71:6800/1790415427 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f7a1400c4a0 con 0x7f7a080441d0 2024-09-11T01:57:28.392 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:57:28.392 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T01:57:28.392 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:57:28.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.390+0000 7f7a22447640 1 -- 172.21.15.12:0/3668679715 >> [v2:172.21.15.71:6800/1790415427,v1:172.21.15.71:6801/1790415427] conn(0x7f7a080441d0 msgr2=0x7f7a080465c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:28.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.390+0000 7f7a22447640 1 --2- 172.21.15.12:0/3668679715 >> [v2:172.21.15.71:6800/1790415427,v1:172.21.15.71:6801/1790415427] conn(0x7f7a080441d0 0x7f7a080465c0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.391+0000 7f7a22447640 1 -- 172.21.15.12:0/3668679715 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7a0803e260 msgr2=0x7f7a08040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:28.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.391+0000 7f7a22447640 1 --2- 172.21.15.12:0/3668679715 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f7a0803e260 0x7f7a08040720 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f7a10004f60 tx=0x7f7a10007e30 comp rx=0 tx=0).stop 2024-09-11T01:57:28.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.391+0000 7f7a22447640 10 monclient: shutdown 2024-09-11T01:57:28.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.391+0000 7f7a22447640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:28.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.391+0000 7f7a22447640 1 -- 172.21.15.12:0/3668679715 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a1c0a3450 msgr2=0x7f7a1c1cd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:28.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.391+0000 7f7a22447640 1 --2- 172.21.15.12:0/3668679715 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7a1c0a3450 0x7f7a1c1cd390 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c0090b0 tx=0x7f7a0c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.391+0000 7f7a209e4640 1 -- 172.21.15.12:0/3668679715 reap_dead start 2024-09-11T01:57:28.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.391+0000 7f7a22447640 1 -- 172.21.15.12:0/3668679715 shutdown_connections 2024-09-11T01:57:28.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.391+0000 7f7a22447640 1 -- 172.21.15.12:0/3668679715 >> 172.21.15.12:0/3668679715 conn(0x7f7a1c0a8340 msgr2=0x7f7a1c09d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:28.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.391+0000 7f7a22447640 1 -- 172.21.15.12:0/3668679715 shutdown_connections 2024-09-11T01:57:28.394 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.391+0000 7f7a22447640 1 -- 172.21.15.12:0/3668679715 wait complete. 2024-09-11T01:57:28.402 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-11T01:57:28.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.503+0000 7ffb692dd640 10 monclient: get_monmap_and_config 2024-09-11T01:57:28.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.503+0000 7ffb692dd640 10 monclient: build_initial_monmap 2024-09-11T01:57:28.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.503+0000 7ffb692dd640 10 monclient: monmap: 2024-09-11T01:57:28.505 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:28.505 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:28.505 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:57:28.503758+0000 2024-09-11T01:57:28.505 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:57:28.503758+0000 2024-09-11T01:57:28.505 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:28.505 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.505 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:57:28.505 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:57:28.505 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:57:28.505 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:57:28.505 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:57:28.505 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.503+0000 7ffb692dd640 1 Processor -- start 2024-09-11T01:57:28.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.503+0000 7ffb692dd640 1 -- start start 2024-09-11T01:57:28.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.503+0000 7ffb692dd640 10 monclient: init 2024-09-11T01:57:28.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb692dd640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:28.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb692dd640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-11T01:57:28.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb692dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffb64074160 addr v1:172.21.15.71:6789/0 2024-09-11T01:57:28.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb692dd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb640a0540 0x7ffb6413cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb692dd640 10 monclient(hunting): picked mon.noname-b con 0x7ffb640a0540 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:28.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb692dd640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ffb640a2ab0 0x7ffb6413f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb692dd640 10 monclient(hunting): picked mon.noname-a con 0x7ffb640a2ab0 addr v2:172.21.15.12:3300/0 2024-09-11T01:57:28.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb692dd640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb640a34f0 con 0x7ffb64074160 2024-09-11T01:57:28.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb692dd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:28.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb692dd640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7ffb640a0230 con 0x7ffb640a2ab0 2024-09-11T01:57:28.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb692dd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:28.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb692dd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ffb64070160 con 0x7ffb640a0540 2024-09-11T01:57:28.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb692dd640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:28.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb692dd640 10 monclient(hunting): authenticate will time out at 2454.946315s 2024-09-11T01:57:28.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb5bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb640a0540 0x7ffb6413cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb637fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ffb640a2ab0 0x7ffb6413f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb5bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb640a0540 0x7ffb6413cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46660/0 (socket says 172.21.15.12:46660) 2024-09-11T01:57:28.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb637fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ffb640a2ab0 0x7ffb6413f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54202/0 (socket says 172.21.15.12:54202) 2024-09-11T01:57:28.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb5bfff640 1 -- 172.21.15.12:0/2980574374 learned_addr learned my addr 172.21.15.12:0/2980574374 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:28.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb5bfff640 10 monclient(hunting): get_auth_request con 0x7ffb640a0540 auth_method 0 2024-09-11T01:57:28.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:28.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:28.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:28.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb637fe640 10 monclient(hunting): get_auth_request con 0x7ffb640a2ab0 auth_method 0 2024-09-11T01:57:28.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:28.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb637fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:28.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:28.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:28.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:28.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:28.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:28.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:28.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:28.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb5bfff640 10 monclient(hunting): handle_auth_done global_id 304280 payload 274 2024-09-11T01:57:28.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb5bfff640 1 -- 172.21.15.12:0/2980574374 >> v2:172.21.15.12:3300/0 conn(0x7ffb640a2ab0 msgr2=0x7ffb6413f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:28.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb5bfff640 1 --2- 172.21.15.12:0/2980574374 >> v2:172.21.15.12:3300/0 conn(0x7ffb640a2ab0 0x7ffb6413f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb5bfff640 1 -- 172.21.15.12:0/2980574374 >> v1:172.21.15.71:6789/0 conn(0x7ffb64074160 legacy=0x7ffb64074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:28.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb5bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:57:28.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb5bfff640 1 monclient: found mon.noname-b 2024-09-11T01:57:28.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:28.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb5bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:57:28.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb5bfff640 1 -- 172.21.15.12:0/2980574374 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb64070440 con 0x7ffb640a0540 2024-09-11T01:57:28.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.504+0000 7ffb637fe640 1 --2- 172.21.15.12:0/2980574374 >> v2:172.21.15.12:3300/0 conn(0x7ffb640a2ab0 0x7ffb6413f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:57:28.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.507+0000 7ffb5bfff640 1 -- 172.21.15.12:0/2980574374 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb640a0540 msgr2=0x7ffb6413cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:28.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.507+0000 7ffb5bfff640 1 --2- 172.21.15.12:0/2980574374 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb640a0540 0x7ffb6413cfa0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c009870 tx=0x7ffb4c02a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4654b1942b5b5f85 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:28.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.507+0000 7ffb627fc640 1 -- 172.21.15.12:0/2980574374 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c037070 con 0x7ffb640a0540 2024-09-11T01:57:28.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.507+0000 7ffb627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:28.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.507+0000 7ffb627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:57:28.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.507+0000 7ffb627fc640 10 monclient: dump: 2024-09-11T01:57:28.513 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:28.514 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:28.514 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.514 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.514 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:28.514 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.514 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:28.514 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:28.514 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:28.514 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.507+0000 7ffb627fc640 10 monclient: _finish_auth 0 2024-09-11T01:57:28.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.507+0000 7ffb627fc640 10 monclient: _check_auth_tickets 2024-09-11T01:57:28.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.507+0000 7ffb627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:28.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.507+0000 7ffb627fc640 1 -- 172.21.15.12:0/2980574374 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb4c032440 con 0x7ffb640a0540 2024-09-11T01:57:28.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.507+0000 7ffb692dd640 5 monclient: authenticate success, global_id 304280 2024-09-11T01:57:28.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.507+0000 7ffb692dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:57:28.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.507+0000 7ffb627fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:28.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.507+0000 7ffb692dd640 10 monclient: get_monmap_and_config success 2024-09-11T01:57:28.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.507+0000 7ffb627fc640 1 -- 172.21.15.12:0/2980574374 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c032810 con 0x7ffb640a0540 2024-09-11T01:57:28.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.507+0000 7ffb627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:28.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.507+0000 7ffb692dd640 4 set_mon_vals no callback set 2024-09-11T01:57:28.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.507+0000 7ffb627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:57:28.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.507+0000 7ffb627fc640 10 monclient: dump: 2024-09-11T01:57:28.515 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:28.515 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:28.515 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.515 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.515 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:28.515 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.516 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:28.516 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:28.516 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:28.516 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.508+0000 7ffb692dd640 10 monclient: shutdown 2024-09-11T01:57:28.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.508+0000 7ffb692dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:28.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.508+0000 7ffb692dd640 1 -- 172.21.15.12:0/2980574374 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb640a0540 msgr2=0x7ffb6413cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:28.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.508+0000 7ffb692dd640 1 --2- 172.21.15.12:0/2980574374 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb640a0540 0x7ffb6413cfa0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c009870 tx=0x7ffb4c02a0a0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.508+0000 7ffb692dd640 1 -- 172.21.15.12:0/2980574374 shutdown_connections 2024-09-11T01:57:28.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.508+0000 7ffb692dd640 1 --2- 172.21.15.12:0/2980574374 >> v2:172.21.15.12:3300/0 conn(0x7ffb640a2ab0 0x7ffb6413f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.508+0000 7ffb692dd640 1 --2- 172.21.15.12:0/2980574374 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb640a0540 0x7ffb6413cfa0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.508+0000 7ffb692dd640 1 -- 172.21.15.12:0/2980574374 >> 172.21.15.12:0/2980574374 conn(0x7ffb640a36c0 msgr2=0x7ffb640a3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:28.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.508+0000 7ffb692dd640 1 -- 172.21.15.12:0/2980574374 shutdown_connections 2024-09-11T01:57:28.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.508+0000 7ffb692dd640 1 -- 172.21.15.12:0/2980574374 wait complete. 2024-09-11T01:57:28.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.508+0000 7ffb692dd640 10 monclient: build_initial_monmap 2024-09-11T01:57:28.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.508+0000 7ffb692dd640 10 monclient: monmap: 2024-09-11T01:57:28.522 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:28.522 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:57:28.522 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:57:28.522 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:57:28.522 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:28.522 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.522 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:57:28.522 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:57:28.522 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:57:28.523 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb692dd640 1 Processor -- start 2024-09-11T01:57:28.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb692dd640 1 -- start start 2024-09-11T01:57:28.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb692dd640 10 monclient: init 2024-09-11T01:57:28.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb692dd640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:28.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb692dd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:57:28.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb692dd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb64074160 0x7ffb641cd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb692dd640 10 monclient(hunting): picked mon.noname-a con 0x7ffb64074160 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:57:28.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb692dd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb640a0540 0x7ffb641cd660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb692dd640 10 monclient(hunting): picked mon.noname-c con 0x7ffb640a0540 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:28.523 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb692dd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffb640a2ab0 0x7ffb641cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb692dd640 10 monclient(hunting): picked mon.noname-b con 0x7ffb640a2ab0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:57:28.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb692dd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:28.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb692dd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6413fa30 con 0x7ffb64074160 2024-09-11T01:57:28.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb692dd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:28.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb692dd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ffb64070780 con 0x7ffb640a2ab0 2024-09-11T01:57:28.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb692dd640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:28.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb692dd640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ffb6413fbb0 con 0x7ffb640a0540 2024-09-11T01:57:28.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb692dd640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:28.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb692dd640 10 monclient(hunting): authenticate will time out at 2454.951754s 2024-09-11T01:57:28.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb62ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb64074160 0x7ffb641cd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb5bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb640a0540 0x7ffb641cd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb5bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb640a0540 0x7ffb641cd660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46668/0 (socket says 172.21.15.12:46668) 2024-09-11T01:57:28.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb62ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb64074160 0x7ffb641cd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54212/0 (socket says 172.21.15.12:54212) 2024-09-11T01:57:28.524 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb5bfff640 1 -- 172.21.15.12:0/2416489483 learned_addr learned my addr 172.21.15.12:0/2416489483 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:28.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb5bfff640 10 monclient(hunting): get_auth_request con 0x7ffb640a0540 auth_method 0 2024-09-11T01:57:28.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:28.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:28.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:28.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb62ffd640 10 monclient(hunting): get_auth_request con 0x7ffb64074160 auth_method 0 2024-09-11T01:57:28.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:28.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:28.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:28.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:28.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:28.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:28.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:28.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:28.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.509+0000 7ffb62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:28.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.510+0000 7ffb5bfff640 10 monclient(hunting): handle_auth_done global_id 304283 payload 824 2024-09-11T01:57:28.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.510+0000 7ffb637fe640 1 --2- 172.21.15.12:0/2416489483 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffb640a2ab0 0x7ffb641cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.510+0000 7ffb5bfff640 1 -- 172.21.15.12:0/2416489483 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffb640a2ab0 msgr2=0x7ffb641cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:28.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.510+0000 7ffb5bfff640 1 --2- 172.21.15.12:0/2416489483 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffb640a2ab0 0x7ffb641cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.510+0000 7ffb5bfff640 1 -- 172.21.15.12:0/2416489483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb64074160 msgr2=0x7ffb641cd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:28.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.510+0000 7ffb5bfff640 1 --2- 172.21.15.12:0/2416489483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb64074160 0x7ffb641cd120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.510+0000 7ffb5bfff640 10 monclient: _finish_hunting 0 2024-09-11T01:57:28.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.510+0000 7ffb5bfff640 1 monclient: found mon.noname-c 2024-09-11T01:57:28.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.510+0000 7ffb5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:28.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.510+0000 7ffb5bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:57:28.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.510+0000 7ffb5bfff640 1 -- 172.21.15.12:0/2416489483 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb6413e100 con 0x7ffb640a0540 2024-09-11T01:57:28.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.512+0000 7ffb5bfff640 1 -- 172.21.15.12:0/2416489483 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb640a0540 msgr2=0x7ffb641cd660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:28.526 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.512+0000 7ffb5bfff640 1 --2- 172.21.15.12:0/2416489483 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb640a0540 0x7ffb641cd660 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c0090b0 tx=0x7ffb4c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:28.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.512+0000 7ffb5b7fe640 1 -- 172.21.15.12:0/2416489483 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c041020 con 0x7ffb640a0540 2024-09-11T01:57:28.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.512+0000 7ffb5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:28.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.512+0000 7ffb5b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:57:28.527 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.512+0000 7ffb5b7fe640 10 monclient: dump: 2024-09-11T01:57:28.527 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:28.527 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:28.527 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.527 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.527 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:28.527 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.527 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:28.527 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:28.527 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:28.527 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.512+0000 7ffb5b7fe640 10 monclient: _finish_auth 0 2024-09-11T01:57:28.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.512+0000 7ffb5b7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:57:28.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.512+0000 7ffb5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:28.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.512+0000 7ffb5b7fe640 1 -- 172.21.15.12:0/2416489483 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb4c03d630 con 0x7ffb640a0540 2024-09-11T01:57:28.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.512+0000 7ffb5b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:28.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.512+0000 7ffb5b7fe640 1 -- 172.21.15.12:0/2416489483 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c037040 con 0x7ffb640a0540 2024-09-11T01:57:28.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.512+0000 7ffb5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:28.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.512+0000 7ffb5b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:57:28.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.512+0000 7ffb5b7fe640 10 monclient: dump: 2024-09-11T01:57:28.528 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:28.528 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:28.528 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.528 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.529 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:28.529 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.529 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:28.529 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:28.529 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:28.529 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.512+0000 7ffb692dd640 5 monclient: authenticate success, global_id 304283 2024-09-11T01:57:28.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.512+0000 7ffb692dd640 10 monclient: _renew_subs 2024-09-11T01:57:28.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.512+0000 7ffb692dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:28.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.512+0000 7ffb692dd640 1 -- 172.21.15.12:0/2416489483 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb6413e3e0 con 0x7ffb640a0540 2024-09-11T01:57:28.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.512+0000 7ffb61ffb640 4 set_mon_vals no callback set 2024-09-11T01:57:28.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.512+0000 7ffb692dd640 10 monclient: _renew_subs 2024-09-11T01:57:28.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.512+0000 7ffb692dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:28.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.512+0000 7ffb692dd640 1 -- 172.21.15.12:0/2416489483 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb6413de20 con 0x7ffb640a0540 2024-09-11T01:57:28.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.513+0000 7ffb5b7fe640 1 -- 172.21.15.12:0/2416489483 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ffb4c03a030 con 0x7ffb640a0540 2024-09-11T01:57:28.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.513+0000 7ffb692dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:28.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.513+0000 7ffb692dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:28.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.513+0000 7ffb692dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:28.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.513+0000 7ffb692dd640 1 -- 172.21.15.12:0/2416489483 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb640728a0 con 0x7ffb640a0540 2024-09-11T01:57:28.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.513+0000 7ffb5b7fe640 1 --2- 172.21.15.12:0/2416489483 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffb4403e210 0x7ffb440406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.513+0000 7ffb5b7fe640 1 -- 172.21.15.12:0/2416489483 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(923..923 src has 889..923) ==== 4422+0+0 (secure 0 0 0) 0x7ffb4c072eb0 con 0x7ffb640a0540 2024-09-11T01:57:28.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.514+0000 7ffb62ffd640 1 --2- 172.21.15.12:0/2416489483 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffb4403e210 0x7ffb440406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.514+0000 7ffb62ffd640 10 monclient: get_auth_request con 0x7ffb4403e210 auth_method 0 2024-09-11T01:57:28.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.516+0000 7ffb62ffd640 1 --2- 172.21.15.12:0/2416489483 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffb4403e210 0x7ffb440406d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ffb54004f60 tx=0x7ffb54007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:28.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.518+0000 7ffb5b7fe640 1 -- 172.21.15.12:0/2416489483 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ffb640728a0 con 0x7ffb640a0540 2024-09-11T01:57:28.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.518+0000 7ffb5b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:28.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.518+0000 7ffb5b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:57:28.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.704+0000 7ffb692dd640 1 -- 172.21.15.12:0/2416489483 --> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7ffb64071710 con 0x7ffb4403e210 2024-09-11T01:57:28.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.704+0000 7ffb5b7fe640 1 -- 172.21.15.12:0/2416489483 <== mgr.4101 v2:172.21.15.71:6816/3189259519 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ffb64071710 con 0x7ffb4403e210 2024-09-11T01:57:28.706 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 4 to deep-scrub 2024-09-11T01:57:28.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.706+0000 7ffb692dd640 1 -- 172.21.15.12:0/2416489483 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffb4403e210 msgr2=0x7ffb440406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:28.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.706+0000 7ffb692dd640 1 --2- 172.21.15.12:0/2416489483 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffb4403e210 0x7ffb440406d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ffb54004f60 tx=0x7ffb54007e30 comp rx=0 tx=0).stop 2024-09-11T01:57:28.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.707+0000 7ffb692dd640 10 monclient: shutdown 2024-09-11T01:57:28.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.707+0000 7ffb692dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:28.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.707+0000 7ffb692dd640 1 -- 172.21.15.12:0/2416489483 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb640a0540 msgr2=0x7ffb641cd660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:28.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.707+0000 7ffb692dd640 1 --2- 172.21.15.12:0/2416489483 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb640a0540 0x7ffb641cd660 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c0090b0 tx=0x7ffb4c0046c0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.707+0000 7ffb692dd640 1 -- 172.21.15.12:0/2416489483 shutdown_connections 2024-09-11T01:57:28.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.707+0000 7ffb692dd640 1 --2- 172.21.15.12:0/2416489483 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffb640a2ab0 0x7ffb641cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.707+0000 7ffb692dd640 1 --2- 172.21.15.12:0/2416489483 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ffb640a0540 0x7ffb641cd660 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.707+0000 7ffb692dd640 1 --2- 172.21.15.12:0/2416489483 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ffb4403e210 0x7ffb440406d0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.707+0000 7ffb692dd640 1 --2- 172.21.15.12:0/2416489483 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffb64074160 0x7ffb641cd120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.707+0000 7ffb692dd640 1 -- 172.21.15.12:0/2416489483 >> 172.21.15.12:0/2416489483 conn(0x7ffb640a36c0 msgr2=0x7ffb64072da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:28.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.707+0000 7ffb692dd640 1 -- 172.21.15.12:0/2416489483 shutdown_connections 2024-09-11T01:57:28.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.707+0000 7ffb692dd640 1 -- 172.21.15.12:0/2416489483 wait complete. 2024-09-11T01:57:28.719 INFO:tasks.ceph:Scrubbing osd.5 2024-09-11T01:57:28.719 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-11T01:57:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.823+0000 7ff8f15b4640 10 monclient: get_monmap_and_config 2024-09-11T01:57:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.823+0000 7ff8f15b4640 10 monclient: build_initial_monmap 2024-09-11T01:57:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.823+0000 7ff8f15b4640 10 monclient: monmap: 2024-09-11T01:57:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:57:28.824373+0000 2024-09-11T01:57:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:57:28.824373+0000 2024-09-11T01:57:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.825 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:57:28.826 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:57:28.826 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:57:28.826 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:57:28.826 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:57:28.826 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.824+0000 7ff8f15b4640 1 Processor -- start 2024-09-11T01:57:28.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.824+0000 7ff8f15b4640 1 -- start start 2024-09-11T01:57:28.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.824+0000 7ff8f15b4640 10 monclient: init 2024-09-11T01:57:28.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.824+0000 7ff8f15b4640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:28.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.824+0000 7ff8f15b4640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-11T01:57:28.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.824+0000 7ff8f15b4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff8ec0a0f10 addr v1:172.21.15.12:6789/0 2024-09-11T01:57:28.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.824+0000 7ff8f15b4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff8ec136090 addr v1:172.21.15.71:6789/0 2024-09-11T01:57:28.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.824+0000 7ff8f15b4640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff8ec0a12f0 0x7ff8ec13ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.824+0000 7ff8f15b4640 10 monclient(hunting): picked mon.noname-c con 0x7ff8ec0a12f0 addr v2:172.21.15.71:3300/0 2024-09-11T01:57:28.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.824+0000 7ff8f15b4640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8ec0a3b60 con 0x7ff8ec0a0f10 2024-09-11T01:57:28.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.824+0000 7ff8f15b4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8ec1364a0 con 0x7ff8ec136090 2024-09-11T01:57:28.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.824+0000 7ff8f15b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:28.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.824+0000 7ff8f15b4640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7ff8ec136b00 con 0x7ff8ec0a12f0 2024-09-11T01:57:28.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.824+0000 7ff8f15b4640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:28.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.824+0000 7ff8f15b4640 10 monclient(hunting): authenticate will time out at 2455.266806s 2024-09-11T01:57:28.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.824+0000 7ff8eaffd640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7ff8ec0a0f10 0x7ff8ec0a2d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33928/0 (socket says 172.21.15.12:33928) 2024-09-11T01:57:28.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.824+0000 7ff8eaffd640 1 -- 172.21.15.12:0/971415694 learned_addr learned my addr 172.21.15.12:0/971415694 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:57:28.829 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.825+0000 7ff8eb7fe640 1 --2- 172.21.15.12:0/971415694 >> v2:172.21.15.71:3300/0 conn(0x7ff8ec0a12f0 0x7ff8ec13ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.825+0000 7ff8eb7fe640 10 monclient(hunting): get_auth_request con 0x7ff8ec0a12f0 auth_method 0 2024-09-11T01:57:28.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.825+0000 7ff8eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:28.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.825+0000 7ff8eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:28.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.825+0000 7ff8eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:28.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.825+0000 7ff8eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:28.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.825+0000 7ff8eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:28.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.825+0000 7ff8eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:28.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.825+0000 7ff8eb7fe640 10 monclient(hunting): handle_auth_done global_id 314170 payload 274 2024-09-11T01:57:28.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.825+0000 7ff8eb7fe640 1 -- 172.21.15.12:0/971415694 >> v1:172.21.15.71:6789/0 conn(0x7ff8ec136090 legacy=0x7ff8ec0a3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:28.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.825+0000 7ff8eb7fe640 1 -- 172.21.15.12:0/971415694 >> v1:172.21.15.12:6789/0 conn(0x7ff8ec0a0f10 legacy=0x7ff8ec0a2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:28.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.825+0000 7ff8eb7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:57:28.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.825+0000 7ff8eb7fe640 1 monclient: found mon.noname-c 2024-09-11T01:57:28.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.825+0000 7ff8eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:28.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.825+0000 7ff8eb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:57:28.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.825+0000 7ff8eb7fe640 1 -- 172.21.15.12:0/971415694 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8ec06d2d0 con 0x7ff8ec0a12f0 2024-09-11T01:57:28.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.828+0000 7ff8eb7fe640 1 -- 172.21.15.12:0/971415694 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8ec0a12f0 msgr2=0x7ff8ec13ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:28.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.828+0000 7ff8eb7fe640 1 --2- 172.21.15.12:0/971415694 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8ec0a12f0 0x7ff8ec13ef20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff8e4009d90 tx=0x7ff8e402a490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e371641ec5095b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:28.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.828+0000 7ff8ea7fc640 1 -- 172.21.15.12:0/971415694 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e4007a30 con 0x7ff8ec0a12f0 2024-09-11T01:57:28.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.828+0000 7ff8ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:28.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.828+0000 7ff8ea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:57:28.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.828+0000 7ff8ea7fc640 10 monclient: dump: 2024-09-11T01:57:28.837 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:28.837 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:28.837 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.838 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.838 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:28.838 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.838 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:28.838 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:28.838 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:28.838 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.828+0000 7ff8ea7fc640 10 monclient: _finish_auth 0 2024-09-11T01:57:28.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.828+0000 7ff8ea7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:57:28.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.828+0000 7ff8ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:28.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.828+0000 7ff8ea7fc640 1 -- 172.21.15.12:0/971415694 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8e4007bd0 con 0x7ff8ec0a12f0 2024-09-11T01:57:28.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.828+0000 7ff8ea7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:28.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.828+0000 7ff8ea7fc640 1 -- 172.21.15.12:0/971415694 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e4031400 con 0x7ff8ec0a12f0 2024-09-11T01:57:28.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.828+0000 7ff8ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:28.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.828+0000 7ff8ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:57:28.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.828+0000 7ff8ea7fc640 10 monclient: dump: 2024-09-11T01:57:28.839 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:28.839 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:28.839 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.839 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.839 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:28.839 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.839 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:28.839 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:28.839 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:28.839 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.828+0000 7ff8f15b4640 5 monclient: authenticate success, global_id 314170 2024-09-11T01:57:28.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.828+0000 7ff8f15b4640 10 monclient: get_monmap_and_config success 2024-09-11T01:57:28.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.828+0000 7ff8f15b4640 4 set_mon_vals no callback set 2024-09-11T01:57:28.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.829+0000 7ff8f15b4640 10 monclient: shutdown 2024-09-11T01:57:28.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.829+0000 7ff8f15b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:28.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.829+0000 7ff8f15b4640 1 -- 172.21.15.12:0/971415694 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8ec0a12f0 msgr2=0x7ff8ec13ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:28.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.829+0000 7ff8f15b4640 1 --2- 172.21.15.12:0/971415694 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8ec0a12f0 0x7ff8ec13ef20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff8e4009d90 tx=0x7ff8e402a490 comp rx=0 tx=0).stop 2024-09-11T01:57:28.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.829+0000 7ff8f15b4640 1 -- 172.21.15.12:0/971415694 shutdown_connections 2024-09-11T01:57:28.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.829+0000 7ff8f15b4640 1 --2- 172.21.15.12:0/971415694 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8ec0a12f0 0x7ff8ec13ef20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.829+0000 7ff8f15b4640 1 -- 172.21.15.12:0/971415694 >> 172.21.15.12:0/971415694 conn(0x7ff8ec0a3e30 msgr2=0x7ff8ec0a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:28.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.829+0000 7ff8f15b4640 1 -- 172.21.15.12:0/971415694 shutdown_connections 2024-09-11T01:57:28.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.829+0000 7ff8f15b4640 1 -- 172.21.15.12:0/971415694 wait complete. 2024-09-11T01:57:28.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.830+0000 7ff8f15b4640 10 monclient: build_initial_monmap 2024-09-11T01:57:28.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.830+0000 7ff8f15b4640 10 monclient: monmap: 2024-09-11T01:57:28.841 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:28.841 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:57:28.841 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:57:28.841 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:57:28.841 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:28.841 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.841 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:57:28.841 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:57:28.841 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:57:28.841 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.830+0000 7ff8f15b4640 1 Processor -- start 2024-09-11T01:57:28.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.830+0000 7ff8f15b4640 1 -- start start 2024-09-11T01:57:28.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.830+0000 7ff8f15b4640 10 monclient: init 2024-09-11T01:57:28.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.830+0000 7ff8f15b4640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:28.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.830+0000 7ff8f15b4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:57:28.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.830+0000 7ff8f15b4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8ec0a0f10 0x7ff8ec09a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.830+0000 7ff8f15b4640 10 monclient(hunting): picked mon.noname-a con 0x7ff8ec0a0f10 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:57:28.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.830+0000 7ff8f15b4640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff8ec0a12f0 0x7ff8ec09ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8f15b4640 10 monclient(hunting): picked mon.noname-c con 0x7ff8ec0a12f0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:28.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8f15b4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8ec136090 0x7ff8ec072eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8f15b4640 10 monclient(hunting): picked mon.noname-b con 0x7ff8ec136090 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:57:28.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8f15b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:28.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8f15b4640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ec0a1740 con 0x7ff8ec0a0f10 2024-09-11T01:57:28.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8f15b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:28.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8f15b4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ec06d890 con 0x7ff8ec136090 2024-09-11T01:57:28.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8f15b4640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:28.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8f15b4640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7ff8ec06cd00 con 0x7ff8ec0a12f0 2024-09-11T01:57:28.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8f15b4640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:28.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8f15b4640 10 monclient(hunting): authenticate will time out at 2455.273037s 2024-09-11T01:57:28.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8eaffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8ec0a0f10 0x7ff8ec09a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8eaffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8ec0a0f10 0x7ff8ec09a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54228/0 (socket says 172.21.15.12:54228) 2024-09-11T01:57:28.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8eaffd640 1 -- 172.21.15.12:0/1871977164 learned_addr learned my addr 172.21.15.12:0/1871977164 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:28.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8eaffd640 10 monclient(hunting): get_auth_request con 0x7ff8ec0a0f10 auth_method 0 2024-09-11T01:57:28.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:28.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:28.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:28.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8e27fc640 1 --2- 172.21.15.12:0/1871977164 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff8ec0a12f0 0x7ff8ec09ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8e27fc640 10 monclient(hunting): get_auth_request con 0x7ff8ec0a12f0 auth_method 0 2024-09-11T01:57:28.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8e27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:28.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:28.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:28.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:28.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:28.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:28.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8e27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:28.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8e27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:28.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8e27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:28.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8eb7fe640 1 --2- 172.21.15.12:0/1871977164 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8ec136090 0x7ff8ec072eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8eb7fe640 10 monclient(hunting): get_auth_request con 0x7ff8ec136090 auth_method 0 2024-09-11T01:57:28.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:28.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:28.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:28.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8eaffd640 10 monclient(hunting): handle_auth_done global_id 294387 payload 824 2024-09-11T01:57:28.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8eaffd640 1 -- 172.21.15.12:0/1871977164 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff8ec0a12f0 msgr2=0x7ff8ec09ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:28.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8eaffd640 1 --2- 172.21.15.12:0/1871977164 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff8ec0a12f0 0x7ff8ec09ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8eaffd640 1 -- 172.21.15.12:0/1871977164 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8ec136090 msgr2=0x7ff8ec072eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:28.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8eaffd640 1 --2- 172.21.15.12:0/1871977164 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8ec136090 0x7ff8ec072eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8eaffd640 10 monclient: _finish_hunting 0 2024-09-11T01:57:28.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8eaffd640 1 monclient: found mon.noname-a 2024-09-11T01:57:28.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:28.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8eaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:57:28.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8eaffd640 1 -- 172.21.15.12:0/1871977164 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8ec074970 con 0x7ff8ec0a0f10 2024-09-11T01:57:28.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8e27fc640 1 --2- 172.21.15.12:0/1871977164 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff8ec0a12f0 0x7ff8ec09ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:57:28.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.831+0000 7ff8eb7fe640 1 --2- 172.21.15.12:0/1871977164 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8ec136090 0x7ff8ec072eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:57:28.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.833+0000 7ff8eaffd640 1 -- 172.21.15.12:0/1871977164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8ec0a0f10 msgr2=0x7ff8ec09a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:28.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.833+0000 7ff8eaffd640 1 --2- 172.21.15.12:0/1871977164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8ec0a0f10 0x7ff8ec09a640 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc00c460 tx=0x7ff8dc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:28.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.833+0000 7ff8e3fff640 1 -- 172.21.15.12:0/1871977164 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc017070 con 0x7ff8ec0a0f10 2024-09-11T01:57:28.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.833+0000 7ff8e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:28.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.833+0000 7ff8e3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:57:28.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.833+0000 7ff8e3fff640 10 monclient: dump: 2024-09-11T01:57:28.846 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:28.846 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:28.846 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.846 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.846 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:28.846 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.846 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:28.846 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:28.846 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:28.847 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.833+0000 7ff8e3fff640 10 monclient: _finish_auth 0 2024-09-11T01:57:28.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.833+0000 7ff8e3fff640 10 monclient: _check_auth_tickets 2024-09-11T01:57:28.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.833+0000 7ff8e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:28.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.833+0000 7ff8e3fff640 1 -- 172.21.15.12:0/1871977164 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8dc0070d0 con 0x7ff8ec0a0f10 2024-09-11T01:57:28.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.833+0000 7ff8e3fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:28.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.833+0000 7ff8e3fff640 1 -- 172.21.15.12:0/1871977164 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc0074c0 con 0x7ff8ec0a0f10 2024-09-11T01:57:28.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.833+0000 7ff8e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:28.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.833+0000 7ff8e3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:28.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.833+0000 7ff8e3fff640 10 monclient: dump: 2024-09-11T01:57:28.848 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:28.848 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:28.848 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.848 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.848 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:28.848 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.848 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:28.848 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:28.848 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:28.848 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.833+0000 7ff8f15b4640 5 monclient: authenticate success, global_id 294387 2024-09-11T01:57:28.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.833+0000 7ff8f15b4640 10 monclient: _renew_subs 2024-09-11T01:57:28.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.833+0000 7ff8f15b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:28.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.833+0000 7ff8f15b4640 1 -- 172.21.15.12:0/1871977164 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8ec12d7d0 con 0x7ff8ec0a0f10 2024-09-11T01:57:28.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.833+0000 7ff8e97fa640 4 set_mon_vals no callback set 2024-09-11T01:57:28.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.833+0000 7ff8f15b4640 10 monclient: _renew_subs 2024-09-11T01:57:28.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.834+0000 7ff8f15b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:28.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.834+0000 7ff8f15b4640 1 -- 172.21.15.12:0/1871977164 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8ec12c870 con 0x7ff8ec0a0f10 2024-09-11T01:57:28.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.834+0000 7ff8f15b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:28.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.834+0000 7ff8f15b4640 1 -- 172.21.15.12:0/1871977164 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff8ec12ca70 con 0x7ff8ec0a0f10 2024-09-11T01:57:28.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.834+0000 7ff8e3fff640 1 -- 172.21.15.12:0/1871977164 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7ff8dc01b3b0 con 0x7ff8ec0a0f10 2024-09-11T01:57:28.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.835+0000 7ff8e3fff640 1 --2- 172.21.15.12:0/1871977164 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff8cc03e210 0x7ff8cc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.835+0000 7ff8e3fff640 1 -- 172.21.15.12:0/1871977164 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(923..923 src has 889..923) ==== 4422+0+0 (secure 0 0 0) 0x7ff8dc052410 con 0x7ff8ec0a0f10 2024-09-11T01:57:28.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.835+0000 7ff8e3fff640 1 --2- 172.21.15.12:0/1871977164 >> [v2:172.21.15.71:6808/2032569317,v1:172.21.15.71:6809/2032569317] conn(0x7ff8cc044090 0x7ff8cc046480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.835+0000 7ff8e3fff640 1 -- 172.21.15.12:0/1871977164 --> [v2:172.21.15.71:6808/2032569317,v1:172.21.15.71:6809/2032569317] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff8cc046b30 con 0x7ff8cc044090 2024-09-11T01:57:28.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.835+0000 7ff8e3fff640 1 -- 172.21.15.12:0/1871977164 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=923) ==== 24+0+0 (secure 0 0 0) 0x7ff8dc059050 con 0x7ff8ec0a0f10 2024-09-11T01:57:28.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.835+0000 7ff8e3fff640 10 monclient: handle_get_version_reply finishing 1 version 923 2024-09-11T01:57:28.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.835+0000 7ff8e27fc640 1 --2- 172.21.15.12:0/1871977164 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff8cc03e210 0x7ff8cc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.835+0000 7ff8eb7fe640 1 --2- 172.21.15.12:0/1871977164 >> [v2:172.21.15.71:6808/2032569317,v1:172.21.15.71:6809/2032569317] conn(0x7ff8cc044090 0x7ff8cc046480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.835+0000 7ff8e27fc640 10 monclient: get_auth_request con 0x7ff8cc03e210 auth_method 0 2024-09-11T01:57:28.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.835+0000 7ff8eb7fe640 10 monclient: get_auth_request con 0x7ff8cc044090 auth_method 0 2024-09-11T01:57:28.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.838+0000 7ff8e27fc640 1 --2- 172.21.15.12:0/1871977164 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff8cc03e210 0x7ff8cc0406d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec0a0ba0 tx=0x7ff8d4007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:28.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.840+0000 7ff8eb7fe640 1 --2- 172.21.15.12:0/1871977164 >> [v2:172.21.15.71:6808/2032569317,v1:172.21.15.71:6809/2032569317] conn(0x7ff8cc044090 0x7ff8cc046480 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:28.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.841+0000 7ff8e3fff640 1 -- 172.21.15.12:0/1871977164 <== osd.5 v2:172.21.15.71:6808/2032569317 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff8cc046b30 con 0x7ff8cc044090 2024-09-11T01:57:28.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.853+0000 7ff8f15b4640 1 -- 172.21.15.12:0/1871977164 --> [v2:172.21.15.71:6808/2032569317,v1:172.21.15.71:6809/2032569317] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff8ec12ec30 con 0x7ff8cc044090 2024-09-11T01:57:28.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.854+0000 7ff8e3fff640 1 -- 172.21.15.12:0/1871977164 <== osd.5 v2:172.21.15.71:6808/2032569317 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff8ec12ec30 con 0x7ff8cc044090 2024-09-11T01:57:28.856 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-09-11T01:57:28.856 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T01:57:28.856 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-09-11T01:57:28.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.854+0000 7ff8f15b4640 1 -- 172.21.15.12:0/1871977164 >> [v2:172.21.15.71:6808/2032569317,v1:172.21.15.71:6809/2032569317] conn(0x7ff8cc044090 msgr2=0x7ff8cc046480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:28.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.855+0000 7ff8f15b4640 1 --2- 172.21.15.12:0/1871977164 >> [v2:172.21.15.71:6808/2032569317,v1:172.21.15.71:6809/2032569317] conn(0x7ff8cc044090 0x7ff8cc046480 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.855+0000 7ff8f15b4640 1 -- 172.21.15.12:0/1871977164 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff8cc03e210 msgr2=0x7ff8cc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:28.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.855+0000 7ff8f15b4640 1 --2- 172.21.15.12:0/1871977164 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7ff8cc03e210 0x7ff8cc0406d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec0a0ba0 tx=0x7ff8d4007e30 comp rx=0 tx=0).stop 2024-09-11T01:57:28.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.855+0000 7ff8f15b4640 10 monclient: shutdown 2024-09-11T01:57:28.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.855+0000 7ff8f15b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:28.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.855+0000 7ff8f15b4640 1 -- 172.21.15.12:0/1871977164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8ec0a0f10 msgr2=0x7ff8ec09a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:28.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.855+0000 7ff8f15b4640 1 --2- 172.21.15.12:0/1871977164 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8ec0a0f10 0x7ff8ec09a640 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc00c460 tx=0x7ff8dc00c920 comp rx=0 tx=0).stop 2024-09-11T01:57:28.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.855+0000 7ff8eb7fe640 1 -- 172.21.15.12:0/1871977164 reap_dead start 2024-09-11T01:57:28.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.855+0000 7ff8f15b4640 1 -- 172.21.15.12:0/1871977164 shutdown_connections 2024-09-11T01:57:28.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.855+0000 7ff8f15b4640 1 -- 172.21.15.12:0/1871977164 >> 172.21.15.12:0/1871977164 conn(0x7ff8ec0a3e30 msgr2=0x7ff8ec1bb9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:28.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.855+0000 7ff8f15b4640 1 -- 172.21.15.12:0/1871977164 shutdown_connections 2024-09-11T01:57:28.858 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.855+0000 7ff8f15b4640 1 -- 172.21.15.12:0/1871977164 wait complete. 2024-09-11T01:57:28.866 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-11T01:57:28.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.966+0000 7f3771c57640 10 monclient: get_monmap_and_config 2024-09-11T01:57:28.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.966+0000 7f3771c57640 10 monclient: build_initial_monmap 2024-09-11T01:57:28.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.966+0000 7f3771c57640 10 monclient: monmap: 2024-09-11T01:57:28.968 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:28.968 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:28.969 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:57:28.967572+0000 2024-09-11T01:57:28.969 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:57:28.967572+0000 2024-09-11T01:57:28.969 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:28.969 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.969 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:57:28.969 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:57:28.969 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:57:28.969 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:57:28.969 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:57:28.969 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.967+0000 7f3771c57640 1 Processor -- start 2024-09-11T01:57:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.967+0000 7f3771c57640 1 -- start start 2024-09-11T01:57:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.967+0000 7f3771c57640 10 monclient: init 2024-09-11T01:57:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f3771c57640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f3771c57640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-11T01:57:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f3771c57640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f376c09dcd0 0x7f376c0a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f3771c57640 10 monclient(hunting): picked mon.noname-a con 0x7f376c09dcd0 addr v2:172.21.15.12:3300/0 2024-09-11T01:57:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f3771c57640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f376c09e0b0 addr v1:172.21.15.12:6789/0 2024-09-11T01:57:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f3771c57640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f376c13b6b0 0x7f376c13daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f3771c57640 10 monclient(hunting): picked mon.noname-b con 0x7f376c13b6b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f3771c57640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f376c09caa0 con 0x7f376c09e0b0 2024-09-11T01:57:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f3771c57640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f3771c57640 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 -- 0x7f376c0a3b30 con 0x7f376c09dcd0 2024-09-11T01:57:28.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f3771c57640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f3771c57640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f376c09c4e0 con 0x7f376c13b6b0 2024-09-11T01:57:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f3771c57640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f3771c57640 10 monclient(hunting): authenticate will time out at 2455.410155s 2024-09-11T01:57:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f376c13b6b0 0x7f376c13daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376b7fe640 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f376c09dcd0 0x7f376c0a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f376c13b6b0 0x7f376c13daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46682/0 (socket says 172.21.15.12:46682) 2024-09-11T01:57:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376bfff640 1 -- 172.21.15.12:0/1458270295 learned_addr learned my addr 172.21.15.12:0/1458270295 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376bfff640 10 monclient(hunting): get_auth_request con 0x7f376c13b6b0 auth_method 0 2024-09-11T01:57:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376affd640 1 --1- 172.21.15.12:0/1458270295 >> v1:172.21.15.12:6789/0 conn(0x7f376c09e0b0 0x7f376c0a2270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33934/0 (socket says 172.21.15.12:33934) 2024-09-11T01:57:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376b7fe640 10 monclient(hunting): get_auth_request con 0x7f376c09dcd0 auth_method 0 2024-09-11T01:57:28.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:28.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:28.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:28.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:28.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:28.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:28.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:28.975 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:28.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:28.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376b7fe640 10 monclient(hunting): handle_auth_done global_id 294390 payload 274 2024-09-11T01:57:28.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376b7fe640 1 -- 172.21.15.12:0/1458270295 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f376c13b6b0 msgr2=0x7f376c13daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:28.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376b7fe640 1 --2- 172.21.15.12:0/1458270295 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f376c13b6b0 0x7f376c13daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376b7fe640 1 -- 172.21.15.12:0/1458270295 >> v1:172.21.15.12:6789/0 conn(0x7f376c09e0b0 legacy=0x7f376c0a2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:28.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:57:28.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376b7fe640 1 monclient: found mon.noname-a 2024-09-11T01:57:28.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:28.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:57:28.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376b7fe640 1 -- 172.21.15.12:0/1458270295 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376c0a3470 con 0x7f376c09dcd0 2024-09-11T01:57:28.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.968+0000 7f376bfff640 1 --2- 172.21.15.12:0/1458270295 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f376c13b6b0 0x7f376c13daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:57:28.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.971+0000 7f376b7fe640 1 -- 172.21.15.12:0/1458270295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f376c09dcd0 msgr2=0x7f376c0a1d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:28.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.971+0000 7f376b7fe640 1 --2- 172.21.15.12:0/1458270295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f376c09dcd0 0x7f376c0a1d30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f375c009d90 tx=0x7f375c02a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4f5f6462c21c90b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:28.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.971+0000 7f376a7fc640 1 -- 172.21.15.12:0/1458270295 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375c007950 con 0x7f376c09dcd0 2024-09-11T01:57:28.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.971+0000 7f376a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:28.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.971+0000 7f376a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:57:28.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.971+0000 7f376a7fc640 10 monclient: dump: 2024-09-11T01:57:28.984 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:28.984 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:28.984 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.984 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.984 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:28.984 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.984 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:28.984 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:28.984 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:28.984 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.971+0000 7f376a7fc640 10 monclient: _finish_auth 0 2024-09-11T01:57:28.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.971+0000 7f376a7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:57:28.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.971+0000 7f376a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:28.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.971+0000 7f376a7fc640 1 -- 172.21.15.12:0/1458270295 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f375c007af0 con 0x7f376c09dcd0 2024-09-11T01:57:28.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.971+0000 7f3771c57640 5 monclient: authenticate success, global_id 294390 2024-09-11T01:57:28.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.971+0000 7f3771c57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:57:28.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.971+0000 7f376a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:28.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.971+0000 7f3771c57640 10 monclient: get_monmap_and_config success 2024-09-11T01:57:28.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.971+0000 7f376a7fc640 1 -- 172.21.15.12:0/1458270295 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375c031400 con 0x7f376c09dcd0 2024-09-11T01:57:28.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.971+0000 7f376a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:28.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.971+0000 7f3771c57640 4 set_mon_vals no callback set 2024-09-11T01:57:28.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.971+0000 7f376a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:28.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.971+0000 7f376a7fc640 10 monclient: dump: 2024-09-11T01:57:28.985 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:28.985 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:28.985 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.986 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.986 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:28.986 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.986 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:28.986 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:28.986 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:28.986 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.972+0000 7f3771c57640 10 monclient: shutdown 2024-09-11T01:57:28.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.972+0000 7f3771c57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:28.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.972+0000 7f3771c57640 1 -- 172.21.15.12:0/1458270295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f376c09dcd0 msgr2=0x7f376c0a1d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:28.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.972+0000 7f3771c57640 1 --2- 172.21.15.12:0/1458270295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f376c09dcd0 0x7f376c0a1d30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f375c009d90 tx=0x7f375c02a490 comp rx=0 tx=0).stop 2024-09-11T01:57:28.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.972+0000 7f3771c57640 1 -- 172.21.15.12:0/1458270295 shutdown_connections 2024-09-11T01:57:28.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.972+0000 7f3771c57640 1 --2- 172.21.15.12:0/1458270295 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f376c13b6b0 0x7f376c13daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.972+0000 7f3771c57640 1 --2- 172.21.15.12:0/1458270295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f376c09dcd0 0x7f376c0a1d30 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.972+0000 7f3771c57640 1 -- 172.21.15.12:0/1458270295 >> 172.21.15.12:0/1458270295 conn(0x7f376c0a7da0 msgr2=0x7f376c0a3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:28.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.972+0000 7f3771c57640 1 -- 172.21.15.12:0/1458270295 shutdown_connections 2024-09-11T01:57:28.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.972+0000 7f3771c57640 1 -- 172.21.15.12:0/1458270295 wait complete. 2024-09-11T01:57:28.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.973+0000 7f3771c57640 10 monclient: build_initial_monmap 2024-09-11T01:57:28.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.973+0000 7f3771c57640 10 monclient: monmap: 2024-09-11T01:57:28.987 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:28.987 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:57:28.987 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:57:28.987 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:57:28.987 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:28.987 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:57:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:57:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:57:28.988 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.973+0000 7f3771c57640 1 Processor -- start 2024-09-11T01:57:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.973+0000 7f3771c57640 1 -- start start 2024-09-11T01:57:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.973+0000 7f3771c57640 10 monclient: init 2024-09-11T01:57:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.973+0000 7f3771c57640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f3771c57640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:57:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f3771c57640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f376c09dcd0 0x7f376c073130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f3771c57640 10 monclient(hunting): picked mon.noname-c con 0x7f376c09dcd0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f3771c57640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f376c09e0b0 0x7f376c073670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f3771c57640 10 monclient(hunting): picked mon.noname-b con 0x7f376c09e0b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:57:28.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f3771c57640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f376c13b6b0 0x7f376c073bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f3771c57640 10 monclient(hunting): picked mon.noname-a con 0x7f376c13b6b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:57:28.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f3771c57640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:28.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f3771c57640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f376c074cd0 con 0x7f376c13b6b0 2024-09-11T01:57:28.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f3771c57640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:28.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f3771c57640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f376c074b50 con 0x7f376c09e0b0 2024-09-11T01:57:28.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f3771c57640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:28.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f3771c57640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f376c0a2980 con 0x7f376c09dcd0 2024-09-11T01:57:28.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f3771c57640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:28.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f3771c57640 10 monclient(hunting): authenticate will time out at 2455.416124s 2024-09-11T01:57:28.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f376c09dcd0 0x7f376c073130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f376c13b6b0 0x7f376c073bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f376c09dcd0 0x7f376c073130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46694/0 (socket says 172.21.15.12:46694) 2024-09-11T01:57:28.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376b7fe640 1 -- 172.21.15.12:0/3842412205 learned_addr learned my addr 172.21.15.12:0/3842412205 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:28.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376b7fe640 10 monclient(hunting): get_auth_request con 0x7f376c09dcd0 auth_method 0 2024-09-11T01:57:28.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:28.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:28.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:28.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376bfff640 10 monclient(hunting): get_auth_request con 0x7f376c13b6b0 auth_method 0 2024-09-11T01:57:28.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:28.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:28.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:28.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:28.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:28.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:28.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:28.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376affd640 1 --2- 172.21.15.12:0/3842412205 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f376c09e0b0 0x7f376c073670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376affd640 10 monclient(hunting): get_auth_request con 0x7f376c09e0b0 auth_method 0 2024-09-11T01:57:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376b7fe640 10 monclient(hunting): handle_auth_done global_id 304292 payload 824 2024-09-11T01:57:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376b7fe640 1 -- 172.21.15.12:0/3842412205 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f376c09e0b0 msgr2=0x7f376c073670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376b7fe640 1 --2- 172.21.15.12:0/3842412205 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f376c09e0b0 0x7f376c073670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376b7fe640 1 -- 172.21.15.12:0/3842412205 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f376c13b6b0 msgr2=0x7f376c073bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376b7fe640 1 --2- 172.21.15.12:0/3842412205 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f376c13b6b0 0x7f376c073bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376b7fe640 10 monclient: _finish_hunting 0 2024-09-11T01:57:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376b7fe640 1 monclient: found mon.noname-c 2024-09-11T01:57:28.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:28.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:57:28.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376b7fe640 1 -- 172.21.15.12:0/3842412205 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376c08f330 con 0x7f376c09dcd0 2024-09-11T01:57:28.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.974+0000 7f376bfff640 1 --2- 172.21.15.12:0/3842412205 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f376c13b6b0 0x7f376c073bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:57:28.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.976+0000 7f376b7fe640 1 -- 172.21.15.12:0/3842412205 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f376c09dcd0 msgr2=0x7f376c073130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:28.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.976+0000 7f376b7fe640 1 --2- 172.21.15.12:0/3842412205 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f376c09dcd0 0x7f376c073130 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f375c009d90 tx=0x7f375c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:28.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.976+0000 7f3753fff640 1 -- 172.21.15.12:0/3842412205 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375c040070 con 0x7f376c09dcd0 2024-09-11T01:57:28.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.976+0000 7f3753fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:28.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.976+0000 7f3753fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:57:28.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.976+0000 7f3753fff640 10 monclient: dump: 2024-09-11T01:57:28.992 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:28.992 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:28.992 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.992 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.993 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:28.993 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.993 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:28.993 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:28.993 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:28.993 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.976+0000 7f3753fff640 10 monclient: _finish_auth 0 2024-09-11T01:57:28.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.976+0000 7f3753fff640 10 monclient: _check_auth_tickets 2024-09-11T01:57:28.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.976+0000 7f3753fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:28.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.976+0000 7f3753fff640 1 -- 172.21.15.12:0/3842412205 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f375c045660 con 0x7f376c09dcd0 2024-09-11T01:57:28.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.976+0000 7f3771c57640 5 monclient: authenticate success, global_id 304292 2024-09-11T01:57:28.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.977+0000 7f3771c57640 10 monclient: _renew_subs 2024-09-11T01:57:28.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.977+0000 7f3771c57640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:28.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.977+0000 7f3771c57640 1 -- 172.21.15.12:0/3842412205 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f376c0a16b0 con 0x7f376c09dcd0 2024-09-11T01:57:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.977+0000 7f3753fff640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.977+0000 7f3753fff640 1 -- 172.21.15.12:0/3842412205 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375c045a30 con 0x7f376c09dcd0 2024-09-11T01:57:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.977+0000 7f3753fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.977+0000 7f3753fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:57:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.977+0000 7f3753fff640 10 monclient: dump: 2024-09-11T01:57:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:28.995 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.977+0000 7f37697fa640 4 set_mon_vals no callback set 2024-09-11T01:57:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.977+0000 7f3771c57640 10 monclient: _renew_subs 2024-09-11T01:57:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.977+0000 7f3771c57640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.977+0000 7f3771c57640 1 -- 172.21.15.12:0/3842412205 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f376c08d040 con 0x7f376c09dcd0 2024-09-11T01:57:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.978+0000 7f3753fff640 1 -- 172.21.15.12:0/3842412205 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f375c007960 con 0x7f376c09dcd0 2024-09-11T01:57:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.978+0000 7f3771c57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.978+0000 7f3771c57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.978+0000 7f3771c57640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.978+0000 7f3771c57640 1 -- 172.21.15.12:0/3842412205 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f376c0a1d30 con 0x7f376c09dcd0 2024-09-11T01:57:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.978+0000 7f3753fff640 1 --2- 172.21.15.12:0/3842412205 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f375803e210 0x7f37580406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.978+0000 7f3753fff640 1 -- 172.21.15.12:0/3842412205 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(923..923 src has 889..923) ==== 4422+0+0 (secure 0 0 0) 0x7f375c0748c0 con 0x7f376c09dcd0 2024-09-11T01:57:28.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.978+0000 7f376affd640 1 --2- 172.21.15.12:0/3842412205 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f375803e210 0x7f37580406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:28.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.978+0000 7f376affd640 10 monclient: get_auth_request con 0x7f375803e210 auth_method 0 2024-09-11T01:57:28.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.981+0000 7f376affd640 1 --2- 172.21.15.12:0/3842412205 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f375803e210 0x7f37580406d0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f376c08e970 tx=0x7f3760009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:28.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.982+0000 7f3753fff640 1 -- 172.21.15.12:0/3842412205 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f375c031e10 con 0x7f376c09dcd0 2024-09-11T01:57:28.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.982+0000 7f3753fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:28.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:28.982+0000 7f3753fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:57:29.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.168+0000 7f3771c57640 1 -- 172.21.15.12:0/3842412205 --> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f376c06e520 con 0x7f375803e210 2024-09-11T01:57:29.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.169+0000 7f3753fff640 1 -- 172.21.15.12:0/3842412205 <== mgr.4101 v2:172.21.15.71:6816/3189259519 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f376c06e520 con 0x7f375803e210 2024-09-11T01:57:29.171 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 5 to deep-scrub 2024-09-11T01:57:29.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.171+0000 7f3771c57640 1 -- 172.21.15.12:0/3842412205 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f375803e210 msgr2=0x7f37580406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:29.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.171+0000 7f3771c57640 1 --2- 172.21.15.12:0/3842412205 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f375803e210 0x7f37580406d0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f376c08e970 tx=0x7f3760009040 comp rx=0 tx=0).stop 2024-09-11T01:57:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.171+0000 7f3771c57640 10 monclient: shutdown 2024-09-11T01:57:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.171+0000 7f3771c57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.171+0000 7f3771c57640 1 -- 172.21.15.12:0/3842412205 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f376c09dcd0 msgr2=0x7f376c073130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.171+0000 7f3771c57640 1 --2- 172.21.15.12:0/3842412205 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f376c09dcd0 0x7f376c073130 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f375c009d90 tx=0x7f375c004a40 comp rx=0 tx=0).stop 2024-09-11T01:57:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.171+0000 7f3771c57640 1 -- 172.21.15.12:0/3842412205 shutdown_connections 2024-09-11T01:57:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.171+0000 7f3771c57640 1 --2- 172.21.15.12:0/3842412205 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f375803e210 0x7f37580406d0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.171+0000 7f3771c57640 1 --2- 172.21.15.12:0/3842412205 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f376c13b6b0 0x7f376c073bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.171+0000 7f3771c57640 1 --2- 172.21.15.12:0/3842412205 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f376c09e0b0 0x7f376c073670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.171+0000 7f3771c57640 1 --2- 172.21.15.12:0/3842412205 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f376c09dcd0 0x7f376c073130 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.171+0000 7f3771c57640 1 -- 172.21.15.12:0/3842412205 >> 172.21.15.12:0/3842412205 conn(0x7f376c0a7da0 msgr2=0x7f376c09a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.171+0000 7f3771c57640 1 -- 172.21.15.12:0/3842412205 shutdown_connections 2024-09-11T01:57:29.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.171+0000 7f3771c57640 1 -- 172.21.15.12:0/3842412205 wait complete. 2024-09-11T01:57:29.183 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T01:57:29.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.282+0000 7f5bd28fe640 10 monclient: get_monmap_and_config 2024-09-11T01:57:29.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.282+0000 7f5bd28fe640 10 monclient: build_initial_monmap 2024-09-11T01:57:29.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.282+0000 7f5bd28fe640 10 monclient: monmap: 2024-09-11T01:57:29.285 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:29.285 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:29.285 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:57:29.283599+0000 2024-09-11T01:57:29.285 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:57:29.283599+0000 2024-09-11T01:57:29.286 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:29.286 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:29.286 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:57:29.286 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:57:29.286 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:57:29.286 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:57:29.286 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:57:29.286 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:29.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.283+0000 7f5bd28fe640 1 Processor -- start 2024-09-11T01:57:29.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.283+0000 7f5bd28fe640 1 -- start start 2024-09-11T01:57:29.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.283+0000 7f5bd28fe640 10 monclient: init 2024-09-11T01:57:29.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bd28fe640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:29.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bd28fe640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-11T01:57:29.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bd28fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5bcc1381c0 0x7f5bcc1385a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:29.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bd28fe640 10 monclient(hunting): picked mon.noname-b con 0x7f5bcc1381c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:29.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bd28fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5bcc0a6aa0 addr v1:172.21.15.71:6789/0 2024-09-11T01:57:29.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bd28fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5bcc13d000 addr v1:172.21.15.12:6789/0 2024-09-11T01:57:29.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bd28fe640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bcc09c310 con 0x7f5bcc13d000 2024-09-11T01:57:29.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bd28fe640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bcc09c8d0 con 0x7f5bcc0a6aa0 2024-09-11T01:57:29.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bd28fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:29.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bd28fe640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5bcc06fc90 con 0x7f5bcc1381c0 2024-09-11T01:57:29.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bd28fe640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:29.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bd28fe640 10 monclient(hunting): authenticate will time out at 2455.726316s 2024-09-11T01:57:29.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bcbfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5bcc1381c0 0x7f5bcc1385a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:29.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bd0e9b640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5bcc13d000 0x7f5bcc13f3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33936/0 (socket says 172.21.15.12:33936) 2024-09-11T01:57:29.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bd0e9b640 1 -- 172.21.15.12:0/2790420953 learned_addr learned my addr 172.21.15.12:0/2790420953 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:57:29.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bcbfff640 10 monclient(hunting): get_auth_request con 0x7f5bcc1381c0 auth_method 0 2024-09-11T01:57:29.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bcbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:29.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bcbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:29.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:29.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bcbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:29.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bcbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:29.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bcbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:29.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bcbfff640 10 monclient(hunting): handle_auth_done global_id 304295 payload 274 2024-09-11T01:57:29.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bcbfff640 1 -- 172.21.15.12:0/2790420953 >> v1:172.21.15.71:6789/0 conn(0x7f5bcc0a6aa0 legacy=0x7f5bcc09cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:29.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bcbfff640 1 -- 172.21.15.12:0/2790420953 >> v1:172.21.15.12:6789/0 conn(0x7f5bcc13d000 legacy=0x7f5bcc13f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:29.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bcbfff640 10 monclient: _finish_hunting 0 2024-09-11T01:57:29.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bcbfff640 1 monclient: found mon.noname-b 2024-09-11T01:57:29.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bcbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:29.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bcbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:57:29.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.284+0000 7f5bcbfff640 1 -- 172.21.15.12:0/2790420953 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bcc138ae0 con 0x7f5bcc1381c0 2024-09-11T01:57:29.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.287+0000 7f5bcbfff640 1 -- 172.21.15.12:0/2790420953 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5bcc1381c0 msgr2=0x7f5bcc1385a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:29.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.287+0000 7f5bcbfff640 1 --2- 172.21.15.12:0/2790420953 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5bcc1381c0 0x7f5bcc1385a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc0097d0 tx=0x7f5bbc02a030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4a1c5145e9ee9d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:29.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.287+0000 7f5bcb7fe640 1 -- 172.21.15.12:0/2790420953 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc037070 con 0x7f5bcc1381c0 2024-09-11T01:57:29.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.287+0000 7f5bcb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:29.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.287+0000 7f5bcb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:57:29.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.287+0000 7f5bcb7fe640 10 monclient: dump: 2024-09-11T01:57:29.293 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:29.294 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:29.294 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:29.294 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:29.294 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:29.294 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:29.294 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:29.294 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:29.294 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:29.294 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:29.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.287+0000 7f5bcb7fe640 10 monclient: _finish_auth 0 2024-09-11T01:57:29.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.287+0000 7f5bcb7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:57:29.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.287+0000 7f5bcb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:29.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.287+0000 7f5bd28fe640 5 monclient: authenticate success, global_id 304295 2024-09-11T01:57:29.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.287+0000 7f5bcb7fe640 1 -- 172.21.15.12:0/2790420953 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bbc032640 con 0x7f5bcc1381c0 2024-09-11T01:57:29.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.287+0000 7f5bd28fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:57:29.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.287+0000 7f5bcb7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:29.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.287+0000 7f5bd28fe640 10 monclient: get_monmap_and_config success 2024-09-11T01:57:29.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.287+0000 7f5bcb7fe640 1 -- 172.21.15.12:0/2790420953 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc032a10 con 0x7f5bcc1381c0 2024-09-11T01:57:29.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.287+0000 7f5bd28fe640 4 set_mon_vals no callback set 2024-09-11T01:57:29.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.287+0000 7f5bcb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:29.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.287+0000 7f5bcb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:57:29.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.287+0000 7f5bcb7fe640 10 monclient: dump: 2024-09-11T01:57:29.295 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:29.295 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:29.295 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:29.296 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:29.296 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:29.296 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:29.296 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:29.296 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:29.296 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:29.296 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:29.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.288+0000 7f5bd28fe640 10 monclient: shutdown 2024-09-11T01:57:29.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.288+0000 7f5bd28fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:29.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.288+0000 7f5bd28fe640 1 -- 172.21.15.12:0/2790420953 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5bcc1381c0 msgr2=0x7f5bcc1385a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:29.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.288+0000 7f5bd28fe640 1 --2- 172.21.15.12:0/2790420953 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5bcc1381c0 0x7f5bcc1385a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc0097d0 tx=0x7f5bbc02a030 comp rx=0 tx=0).stop 2024-09-11T01:57:29.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.288+0000 7f5bd28fe640 1 -- 172.21.15.12:0/2790420953 shutdown_connections 2024-09-11T01:57:29.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.288+0000 7f5bd28fe640 1 --2- 172.21.15.12:0/2790420953 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5bcc1381c0 0x7f5bcc1385a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:29.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.288+0000 7f5bd28fe640 1 -- 172.21.15.12:0/2790420953 >> 172.21.15.12:0/2790420953 conn(0x7f5bcc0a7440 msgr2=0x7f5bcc0a7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:29.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.288+0000 7f5bd28fe640 1 -- 172.21.15.12:0/2790420953 shutdown_connections 2024-09-11T01:57:29.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.288+0000 7f5bd28fe640 1 -- 172.21.15.12:0/2790420953 wait complete. 2024-09-11T01:57:29.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.288+0000 7f5bd28fe640 10 monclient: build_initial_monmap 2024-09-11T01:57:29.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.288+0000 7f5bd28fe640 10 monclient: monmap: 2024-09-11T01:57:29.297 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:29.297 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:57:29.297 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:57:29.297 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:57:29.297 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:29.297 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:29.297 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:57:29.297 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:57:29.297 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:57:29.297 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:29.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bd28fe640 1 Processor -- start 2024-09-11T01:57:29.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bd28fe640 1 -- start start 2024-09-11T01:57:29.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bd28fe640 10 monclient: init 2024-09-11T01:57:29.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bd28fe640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:29.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bd28fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T01:57:29.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bd28fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bcc0a6aa0 0x7f5bcc131fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:29.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bd28fe640 10 monclient(hunting): picked mon.noname-a con 0x7f5bcc0a6aa0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:57:29.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bd28fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5bcc1381c0 0x7f5bcc132510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:29.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bd28fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5bcc1381c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:29.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bd28fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5bcc13d000 0x7f5bcc132ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:29.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bd28fe640 10 monclient(hunting): picked mon.noname-b con 0x7f5bcc13d000 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:57:29.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bd28fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:29.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bd28fe640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5bcc140d20 con 0x7f5bcc0a6aa0 2024-09-11T01:57:29.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bd28fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:29.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bd28fe640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5bcc09c5f0 con 0x7f5bcc13d000 2024-09-11T01:57:29.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bd28fe640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:29.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bd28fe640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f5bcc140ea0 con 0x7f5bcc1381c0 2024-09-11T01:57:29.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bd28fe640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:29.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bd28fe640 10 monclient(hunting): authenticate will time out at 2455.731811s 2024-09-11T01:57:29.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bc3fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5bcc1381c0 0x7f5bcc132510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:29.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bcbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bcc0a6aa0 0x7f5bcc131fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:29.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bcbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bcc0a6aa0 0x7f5bcc131fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54262/0 (socket says 172.21.15.12:54262) 2024-09-11T01:57:29.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bc3fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5bcc1381c0 0x7f5bcc132510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46712/0 (socket says 172.21.15.12:46712) 2024-09-11T01:57:29.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bcbfff640 1 -- 172.21.15.12:0/3592444146 learned_addr learned my addr 172.21.15.12:0/3592444146 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:29.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bcbfff640 10 monclient(hunting): get_auth_request con 0x7f5bcc0a6aa0 auth_method 0 2024-09-11T01:57:29.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bcbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:29.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bcbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:29.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:29.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bc3fff640 10 monclient(hunting): get_auth_request con 0x7f5bcc1381c0 auth_method 0 2024-09-11T01:57:29.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:29.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bc3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:29.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.289+0000 7f5bc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:29.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.290+0000 7f5bcbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:29.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.290+0000 7f5bd0e9b640 1 --2- 172.21.15.12:0/3592444146 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5bcc13d000 0x7f5bcc132ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:29.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.290+0000 7f5bcbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:29.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.290+0000 7f5bcbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:29.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.290+0000 7f5bc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:29.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.290+0000 7f5bc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:29.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.290+0000 7f5bc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:29.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.290+0000 7f5bd0e9b640 10 monclient(hunting): get_auth_request con 0x7f5bcc13d000 auth_method 0 2024-09-11T01:57:29.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.290+0000 7f5bd0e9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:29.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.290+0000 7f5bd0e9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:29.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.290+0000 7f5bd0e9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:29.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.290+0000 7f5bd0e9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:29.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.290+0000 7f5bd0e9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:29.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.290+0000 7f5bd0e9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:29.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.290+0000 7f5bcbfff640 10 monclient(hunting): handle_auth_done global_id 294396 payload 824 2024-09-11T01:57:29.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.290+0000 7f5bcbfff640 1 -- 172.21.15.12:0/3592444146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5bcc1381c0 msgr2=0x7f5bcc132510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:29.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.290+0000 7f5bcbfff640 1 --2- 172.21.15.12:0/3592444146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5bcc1381c0 0x7f5bcc132510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:29.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.290+0000 7f5bcbfff640 1 -- 172.21.15.12:0/3592444146 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5bcc13d000 msgr2=0x7f5bcc132ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:29.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.290+0000 7f5bcbfff640 1 --2- 172.21.15.12:0/3592444146 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5bcc13d000 0x7f5bcc132ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:29.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.290+0000 7f5bcbfff640 10 monclient: _finish_hunting 0 2024-09-11T01:57:29.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.290+0000 7f5bcbfff640 1 monclient: found mon.noname-a 2024-09-11T01:57:29.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.290+0000 7f5bcbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:29.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.290+0000 7f5bcbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:57:29.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.290+0000 7f5bcbfff640 1 -- 172.21.15.12:0/3592444146 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bcc135b30 con 0x7f5bcc0a6aa0 2024-09-11T01:57:29.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.290+0000 7f5bc3fff640 1 --2- 172.21.15.12:0/3592444146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5bcc1381c0 0x7f5bcc132510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:57:29.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.292+0000 7f5bcbfff640 1 -- 172.21.15.12:0/3592444146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bcc0a6aa0 msgr2=0x7f5bcc131fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:29.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.292+0000 7f5bcbfff640 1 --2- 172.21.15.12:0/3592444146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bcc0a6aa0 0x7f5bcc131fd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc009720 tx=0x7f5bbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:29.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.292+0000 7f5bc97fa640 1 -- 172.21.15.12:0/3592444146 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc041070 con 0x7f5bcc0a6aa0 2024-09-11T01:57:29.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.292+0000 7f5bc97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:29.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.292+0000 7f5bc97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:57:29.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.292+0000 7f5bc97fa640 10 monclient: dump: 2024-09-11T01:57:29.306 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:29.306 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:29.306 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:29.307 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:29.307 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:29.307 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:29.307 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:29.307 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:29.307 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:29.307 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:29.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.292+0000 7f5bc97fa640 10 monclient: _finish_auth 0 2024-09-11T01:57:29.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.292+0000 7f5bc97fa640 10 monclient: _check_auth_tickets 2024-09-11T01:57:29.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.292+0000 7f5bc97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:29.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.292+0000 7f5bc97fa640 1 -- 172.21.15.12:0/3592444146 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bbc03c880 con 0x7f5bcc0a6aa0 2024-09-11T01:57:29.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.292+0000 7f5bd28fe640 5 monclient: authenticate success, global_id 294396 2024-09-11T01:57:29.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.292+0000 7f5bd28fe640 10 monclient: _renew_subs 2024-09-11T01:57:29.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.292+0000 7f5bd28fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:29.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.292+0000 7f5bd28fe640 1 -- 172.21.15.12:0/3592444146 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bcc140410 con 0x7f5bcc0a6aa0 2024-09-11T01:57:29.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.292+0000 7f5bc97fa640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:29.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.292+0000 7f5bc97fa640 1 -- 172.21.15.12:0/3592444146 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc037040 con 0x7f5bcc0a6aa0 2024-09-11T01:57:29.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.292+0000 7f5bc97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:29.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.292+0000 7f5bc97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:29.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.292+0000 7f5bc97fa640 10 monclient: dump: 2024-09-11T01:57:29.308 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:29.308 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:29.308 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:29.308 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:29.308 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:29.308 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:29.308 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:29.308 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:29.309 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:29.309 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:29.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.292+0000 7f5bcaffd640 4 set_mon_vals no callback set 2024-09-11T01:57:29.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.292+0000 7f5bd28fe640 10 monclient: _renew_subs 2024-09-11T01:57:29.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.292+0000 7f5bd28fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:29.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.292+0000 7f5bd28fe640 1 -- 172.21.15.12:0/3592444146 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bcc0a09e0 con 0x7f5bcc0a6aa0 2024-09-11T01:57:29.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.293+0000 7f5bd28fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:29.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.293+0000 7f5bd28fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:29.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.293+0000 7f5bd28fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:29.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.293+0000 7f5bd28fe640 1 -- 172.21.15.12:0/3592444146 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bcc1385a0 con 0x7f5bcc0a6aa0 2024-09-11T01:57:29.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.294+0000 7f5bc97fa640 1 -- 172.21.15.12:0/3592444146 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f5bbc03cc70 con 0x7f5bcc0a6aa0 2024-09-11T01:57:29.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.294+0000 7f5bc97fa640 1 --2- 172.21.15.12:0/3592444146 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5bac03e260 0x7f5bac040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:29.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.294+0000 7f5bc97fa640 1 -- 172.21.15.12:0/3592444146 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(923..923 src has 889..923) ==== 4422+0+0 (secure 0 0 0) 0x7f5bbc073380 con 0x7f5bcc0a6aa0 2024-09-11T01:57:29.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.294+0000 7f5bc3fff640 1 --2- 172.21.15.12:0/3592444146 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5bac03e260 0x7f5bac040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:29.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.294+0000 7f5bc3fff640 10 monclient: get_auth_request con 0x7f5bac03e260 auth_method 0 2024-09-11T01:57:29.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.297+0000 7f5bc3fff640 1 --2- 172.21.15.12:0/3592444146 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5bac03e260 0x7f5bac040720 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc1342c0 tx=0x7f5bb4007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:29.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.298+0000 7f5bc97fa640 1 -- 172.21.15.12:0/3592444146 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5bbc033d30 con 0x7f5bcc0a6aa0 2024-09-11T01:57:29.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.298+0000 7f5bc97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:29.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.298+0000 7f5bc97fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:57:29.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.480+0000 7f5bd28fe640 1 -- 172.21.15.12:0/3592444146 --> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5bcc1385a0 con 0x7f5bac03e260 2024-09-11T01:57:29.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.482+0000 7f5bc97fa640 1 -- 172.21.15.12:0/3592444146 <== mgr.4101 v2:172.21.15.71:6816/3189259519 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34024 (secure 0 0 0) 0x7f5bcc1385a0 con 0x7f5bac03e260 2024-09-11T01:57:29.484 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:57:29.484 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-09-11T01:57:29.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.484+0000 7f5bd28fe640 1 -- 172.21.15.12:0/3592444146 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5bac03e260 msgr2=0x7f5bac040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:29.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.484+0000 7f5bd28fe640 1 --2- 172.21.15.12:0/3592444146 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5bac03e260 0x7f5bac040720 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc1342c0 tx=0x7f5bb4007e30 comp rx=0 tx=0).stop 2024-09-11T01:57:29.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.484+0000 7f5bd28fe640 10 monclient: shutdown 2024-09-11T01:57:29.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.484+0000 7f5bd28fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:29.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.484+0000 7f5bd28fe640 1 -- 172.21.15.12:0/3592444146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bcc0a6aa0 msgr2=0x7f5bcc131fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:29.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.484+0000 7f5bd28fe640 1 --2- 172.21.15.12:0/3592444146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bcc0a6aa0 0x7f5bcc131fd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc009720 tx=0x7f5bbc0046c0 comp rx=0 tx=0).stop 2024-09-11T01:57:29.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.484+0000 7f5bd28fe640 1 -- 172.21.15.12:0/3592444146 shutdown_connections 2024-09-11T01:57:29.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.484+0000 7f5bd28fe640 1 --2- 172.21.15.12:0/3592444146 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5bcc13d000 0x7f5bcc132ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:29.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.484+0000 7f5bd28fe640 1 --2- 172.21.15.12:0/3592444146 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5bcc1381c0 0x7f5bcc132510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:29.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.484+0000 7f5bd28fe640 1 --2- 172.21.15.12:0/3592444146 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f5bac03e260 0x7f5bac040720 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:29.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.484+0000 7f5bd28fe640 1 --2- 172.21.15.12:0/3592444146 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5bcc0a6aa0 0x7f5bcc131fd0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:29.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.484+0000 7f5bd28fe640 1 -- 172.21.15.12:0/3592444146 >> 172.21.15.12:0/3592444146 conn(0x7f5bcc0a7440 msgr2=0x7f5bcc13f970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:29.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.484+0000 7f5bd28fe640 1 -- 172.21.15.12:0/3592444146 shutdown_connections 2024-09-11T01:57:29.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:29.484+0000 7f5bd28fe640 1 -- 172.21.15.12:0/3592444146 wait complete. 2024-09-11T01:57:29.496 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":1769,"stamp":"2024-09-11T01:57:29.209974+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":333208,"kb_used_data":14816,"kb_used_omap":9,"kb_used_meta":317366,"kb_avail":565897832,"statfs":{"total":579820584960,"available":579479379968,"internally_reserved":0,"allocated":15171584,"data_stored":7598212,"data_compressed":59136,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":324983456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002728"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":1808,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:44.358595+0000","last_change":"2024-09-11T01:29:17.579548+0000","last_active":"2024-09-11T01:56:44.358595+0000","last_peered":"2024-09-11T01:56:44.358595+0000","last_clean":"2024-09-11T01:56:44.358595+0000","last_became_active":"2024-09-11T01:29:15.546952+0000","last_became_peered":"2024-09-11T01:29:15.546952+0000","last_unstale":"2024-09-11T01:56:44.358595+0000","last_undegraded":"2024-09-11T01:56:44.358595+0000","last_fullsized":"2024-09-11T01:56:44.358595+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:45:28.460820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038245800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":1812,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:43.746370+0000","last_change":"2024-09-11T01:29:17.597459+0000","last_active":"2024-09-11T01:56:43.746370+0000","last_peered":"2024-09-11T01:56:43.746370+0000","last_clean":"2024-09-11T01:56:43.746370+0000","last_became_active":"2024-09-11T01:29:15.549013+0000","last_became_peered":"2024-09-11T01:29:15.549013+0000","last_unstale":"2024-09-11T01:56:43.746370+0000","last_undegraded":"2024-09-11T01:56:43.746370+0000","last_fullsized":"2024-09-11T01:56:43.746370+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:46:02.130174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041667199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":1822,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:43.904328+0000","last_change":"2024-09-11T01:29:17.577463+0000","last_active":"2024-09-11T01:56:43.904328+0000","last_peered":"2024-09-11T01:56:43.904328+0000","last_clean":"2024-09-11T01:56:43.904328+0000","last_became_active":"2024-09-11T01:29:15.548884+0000","last_became_peered":"2024-09-11T01:29:15.548884+0000","last_unstale":"2024-09-11T01:56:43.904328+0000","last_undegraded":"2024-09-11T01:56:43.904328+0000","last_fullsized":"2024-09-11T01:56:43.904328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:09.120675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094832499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":1817,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:44.360535+0000","last_change":"2024-09-11T01:29:17.583117+0000","last_active":"2024-09-11T01:56:44.360535+0000","last_peered":"2024-09-11T01:56:44.360535+0000","last_clean":"2024-09-11T01:56:44.360535+0000","last_became_active":"2024-09-11T01:29:15.546142+0000","last_became_peered":"2024-09-11T01:29:15.546142+0000","last_unstale":"2024-09-11T01:56:44.360535+0000","last_undegraded":"2024-09-11T01:56:44.360535+0000","last_fullsized":"2024-09-11T01:56:44.360535+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:16:49.024029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044411199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"16'72","reported_seq":1891,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:44.358555+0000","last_change":"2024-09-11T01:29:16.691671+0000","last_active":"2024-09-11T01:56:44.358555+0000","last_peered":"2024-09-11T01:56:44.358555+0000","last_clean":"2024-09-11T01:56:44.358555+0000","last_became_active":"2024-09-11T01:29:16.690647+0000","last_became_peered":"2024-09-11T01:29:16.690647+0000","last_unstale":"2024-09-11T01:56:44.358555+0000","last_undegraded":"2024-09-11T01:56:44.358555+0000","last_fullsized":"2024-09-11T01:56:44.358555+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:15.533298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:15.533298+0000","last_clean_scrub_stamp":"2024-09-11T01:29:15.533298+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:41:07.416896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":1816,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:44.360506+0000","last_change":"2024-09-11T01:29:17.583012+0000","last_active":"2024-09-11T01:56:44.360506+0000","last_peered":"2024-09-11T01:56:44.360506+0000","last_clean":"2024-09-11T01:56:44.360506+0000","last_became_active":"2024-09-11T01:29:15.547447+0000","last_became_peered":"2024-09-11T01:29:15.547447+0000","last_unstale":"2024-09-11T01:56:44.360506+0000","last_undegraded":"2024-09-11T01:56:44.360506+0000","last_fullsized":"2024-09-11T01:56:44.360506+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:17:48.616008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037448800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":1820,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:43.904334+0000","last_change":"2024-09-11T01:29:17.576853+0000","last_active":"2024-09-11T01:56:43.904334+0000","last_peered":"2024-09-11T01:56:43.904334+0000","last_clean":"2024-09-11T01:56:43.904334+0000","last_became_active":"2024-09-11T01:29:15.546476+0000","last_became_peered":"2024-09-11T01:29:15.546476+0000","last_unstale":"2024-09-11T01:56:43.904334+0000","last_undegraded":"2024-09-11T01:56:43.904334+0000","last_fullsized":"2024-09-11T01:56:43.904334+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:11:54.706642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037031899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":1806,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:44.358504+0000","last_change":"2024-09-11T01:29:17.579590+0000","last_active":"2024-09-11T01:56:44.358504+0000","last_peered":"2024-09-11T01:56:44.358504+0000","last_clean":"2024-09-11T01:56:44.358504+0000","last_became_active":"2024-09-11T01:29:15.542800+0000","last_became_peered":"2024-09-11T01:29:15.542800+0000","last_unstale":"2024-09-11T01:56:44.358504+0000","last_undegraded":"2024-09-11T01:56:44.358504+0000","last_fullsized":"2024-09-11T01:56:44.358504+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:23:55.773396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040701099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":1811,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:56:43.746418+0000","last_change":"2024-09-11T01:29:17.597492+0000","last_active":"2024-09-11T01:56:43.746418+0000","last_peered":"2024-09-11T01:56:43.746418+0000","last_clean":"2024-09-11T01:56:43.746418+0000","last_became_active":"2024-09-11T01:29:15.546444+0000","last_became_peered":"2024-09-11T01:29:15.546444+0000","last_unstale":"2024-09-11T01:56:43.746418+0000","last_undegraded":"2024-09-11T01:56:43.746418+0000","last_fullsized":"2024-09-11T01:56:43.746418+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:29:14.531573+0000","last_clean_scrub_stamp":"2024-09-11T01:29:14.531573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:52:43.624650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000396053,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59136,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607895,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47704,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324136,"statfs":{"total":96636764160,"available":96587915264,"internally_reserved":0,"allocated":2506752,"data_stored":1090053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607893,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56028,"kb_used_data":2452,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315812,"statfs":{"total":96636764160,"available":96579391488,"internally_reserved":0,"allocated":2510848,"data_stored":1090072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607894,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56084,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315756,"statfs":{"total":96636764160,"available":96579334144,"internally_reserved":0,"allocated":2568192,"data_stored":1618981,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607894,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69724,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302116,"statfs":{"total":96636764160,"available":96565366784,"internally_reserved":0,"allocated":2568192,"data_stored":1618981,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607894,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33820,"kb_used_data":2452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338020,"statfs":{"total":96636764160,"available":96602132480,"internally_reserved":0,"allocated":2510848,"data_stored":1090072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640601,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69848,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94301992,"statfs":{"total":96636764160,"available":96565239808,"internally_reserved":0,"allocated":2506752,"data_stored":1090053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T01:57:29.500 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-11T01:29:14.531573+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=1, tm_min=29, tm_sec=14, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=1, tm_min=57, tm_sec=25, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T01:57:29.500 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-09-11T01:29:14.531573+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=1, tm_min=29, tm_sec=14, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=1, tm_min=57, tm_sec=25, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T01:57:29.501 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-09-11T01:29:14.531573+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=1, tm_min=29, tm_sec=14, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=1, tm_min=57, tm_sec=25, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T01:57:29.501 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-09-11T01:29:14.531573+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=1, tm_min=29, tm_sec=14, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=1, tm_min=57, tm_sec=25, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T01:57:29.501 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-11T01:29:15.533298+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=1, tm_min=29, tm_sec=15, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=1, tm_min=57, tm_sec=25, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T01:57:29.501 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-09-11T01:29:14.531573+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=1, tm_min=29, tm_sec=14, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=1, tm_min=57, tm_sec=25, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T01:57:29.501 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-09-11T01:29:14.531573+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=1, tm_min=29, tm_sec=14, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=1, tm_min=57, tm_sec=25, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T01:57:29.501 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-09-11T01:29:14.531573+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=1, tm_min=29, tm_sec=14, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=1, tm_min=57, tm_sec=25, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T01:57:29.501 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-09-11T01:29:14.531573+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=1, tm_min=29, tm_sec=14, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=1, tm_min=57, tm_sec=25, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T01:57:29.502 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-11T01:57:49.503 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T01:57:49.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.615+0000 7f68dd6b6640 10 monclient: get_monmap_and_config 2024-09-11T01:57:49.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.615+0000 7f68dd6b6640 10 monclient: build_initial_monmap 2024-09-11T01:57:49.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.615+0000 7f68dd6b6640 10 monclient: monmap: 2024-09-11T01:57:49.617 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:49.617 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:49.617 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:57:49.615807+0000 2024-09-11T01:57:49.617 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:57:49.615807+0000 2024-09-11T01:57:49.617 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:49.617 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:49.617 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:57:49.617 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:57:49.618 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:57:49.618 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:57:49.618 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:57:49.618 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:49.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.615+0000 7f68dd6b6640 1 Processor -- start 2024-09-11T01:57:49.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.615+0000 7f68dd6b6640 1 -- start start 2024-09-11T01:57:49.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.615+0000 7f68dd6b6640 10 monclient: init 2024-09-11T01:57:49.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68dd6b6640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:49.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68dd6b6640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-11T01:57:49.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68dd6b6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f68d8137e50 addr v1:172.21.15.71:6789/0 2024-09-11T01:57:49.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68dd6b6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f68d8138230 addr v1:172.21.15.12:6789/0 2024-09-11T01:57:49.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68dd6b6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f68d813e390 0x7f68d8140780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:49.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68dd6b6640 10 monclient(hunting): picked mon.noname-c con 0x7f68d813e390 addr v2:172.21.15.71:3300/0 2024-09-11T01:57:49.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68dd6b6640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68d80a6580 con 0x7f68d8138230 2024-09-11T01:57:49.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68dd6b6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68d80a62d0 con 0x7f68d8137e50 2024-09-11T01:57:49.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68dd6b6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:49.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68dd6b6640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f68d809bb40 con 0x7f68d813e390 2024-09-11T01:57:49.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68dd6b6640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:49.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68dd6b6640 10 monclient(hunting): authenticate will time out at 2476.058267s 2024-09-11T01:57:49.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68d67fc640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f68d8138230 0x7f68d8138640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42932/0 (socket says 172.21.15.12:42932) 2024-09-11T01:57:49.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68d67fc640 1 -- 172.21.15.12:0/845778201 learned_addr learned my addr 172.21.15.12:0/845778201 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:57:49.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68d77fe640 1 --2- 172.21.15.12:0/845778201 >> v2:172.21.15.71:3300/0 conn(0x7f68d813e390 0x7f68d8140780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:49.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68d77fe640 10 monclient(hunting): get_auth_request con 0x7f68d813e390 auth_method 0 2024-09-11T01:57:49.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:49.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:49.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:49.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:49.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:49.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:49.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68d77fe640 10 monclient(hunting): handle_auth_done global_id 314182 payload 274 2024-09-11T01:57:49.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68d77fe640 1 -- 172.21.15.12:0/845778201 >> v1:172.21.15.71:6789/0 conn(0x7f68d8137e50 legacy=0x7f68d809d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:49.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68d77fe640 1 -- 172.21.15.12:0/845778201 >> v1:172.21.15.12:6789/0 conn(0x7f68d8138230 legacy=0x7f68d8138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:49.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68d77fe640 10 monclient: _finish_hunting 0 2024-09-11T01:57:49.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68d77fe640 1 monclient: found mon.noname-c 2024-09-11T01:57:49.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:49.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68d77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-09-11T01:57:49.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.616+0000 7f68d77fe640 1 -- 172.21.15.12:0/845778201 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68d809c4a0 con 0x7f68d813e390 2024-09-11T01:57:49.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.620+0000 7f68d77fe640 1 -- 172.21.15.12:0/845778201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68d813e390 msgr2=0x7f68d8140780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:49.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.620+0000 7f68d77fe640 1 --2- 172.21.15.12:0/845778201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68d813e390 0x7f68d8140780 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f68d809bd20 tx=0x7f68d002a480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=964e3a41fa6f02cc server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:49.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.620+0000 7f68d5ffb640 1 -- 172.21.15.12:0/845778201 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d0007a30 con 0x7f68d813e390 2024-09-11T01:57:49.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.620+0000 7f68d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:49.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.620+0000 7f68d5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:57:49.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.620+0000 7f68d5ffb640 10 monclient: dump: 2024-09-11T01:57:49.629 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:49.629 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:49.629 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:49.629 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:49.629 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:49.630 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:49.630 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:49.630 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:49.630 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:49.630 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:49.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.620+0000 7f68d5ffb640 10 monclient: _finish_auth 0 2024-09-11T01:57:49.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.620+0000 7f68d5ffb640 10 monclient: _check_auth_tickets 2024-09-11T01:57:49.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.620+0000 7f68d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:49.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.620+0000 7f68d5ffb640 1 -- 172.21.15.12:0/845778201 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68d0007bd0 con 0x7f68d813e390 2024-09-11T01:57:49.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.620+0000 7f68d5ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:49.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.620+0000 7f68d5ffb640 1 -- 172.21.15.12:0/845778201 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d0031400 con 0x7f68d813e390 2024-09-11T01:57:49.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.620+0000 7f68d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:49.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.620+0000 7f68d5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T01:57:49.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.620+0000 7f68d5ffb640 10 monclient: dump: 2024-09-11T01:57:49.631 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:49.631 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:49.631 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:49.631 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:49.631 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:49.632 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:49.632 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:49.632 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:49.632 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:49.632 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:49.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.620+0000 7f68dd6b6640 5 monclient: authenticate success, global_id 314182 2024-09-11T01:57:49.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.620+0000 7f68dd6b6640 10 monclient: get_monmap_and_config success 2024-09-11T01:57:49.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.620+0000 7f68dd6b6640 4 set_mon_vals no callback set 2024-09-11T01:57:49.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.620+0000 7f68dd6b6640 10 monclient: shutdown 2024-09-11T01:57:49.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.620+0000 7f68dd6b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:49.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.620+0000 7f68dd6b6640 1 -- 172.21.15.12:0/845778201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68d813e390 msgr2=0x7f68d8140780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:49.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.620+0000 7f68dd6b6640 1 --2- 172.21.15.12:0/845778201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68d813e390 0x7f68d8140780 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f68d809bd20 tx=0x7f68d002a480 comp rx=0 tx=0).stop 2024-09-11T01:57:49.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.621+0000 7f68dd6b6640 1 -- 172.21.15.12:0/845778201 shutdown_connections 2024-09-11T01:57:49.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.621+0000 7f68dd6b6640 1 --2- 172.21.15.12:0/845778201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68d813e390 0x7f68d8140780 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:49.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.621+0000 7f68dd6b6640 1 -- 172.21.15.12:0/845778201 >> 172.21.15.12:0/845778201 conn(0x7f68d80a7020 msgr2=0x7f68d80a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:49.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.621+0000 7f68dd6b6640 1 -- 172.21.15.12:0/845778201 shutdown_connections 2024-09-11T01:57:49.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.621+0000 7f68dd6b6640 1 -- 172.21.15.12:0/845778201 wait complete. 2024-09-11T01:57:49.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.621+0000 7f68dd6b6640 10 monclient: build_initial_monmap 2024-09-11T01:57:49.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.621+0000 7f68dd6b6640 10 monclient: monmap: 2024-09-11T01:57:49.633 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:49.633 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:57:49.633 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:57:49.633 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:57:49.633 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:49.633 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:49.633 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:57:49.634 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:57:49.634 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:57:49.634 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:49.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68dd6b6640 1 Processor -- start 2024-09-11T01:57:49.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68dd6b6640 1 -- start start 2024-09-11T01:57:49.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68dd6b6640 10 monclient: init 2024-09-11T01:57:49.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68dd6b6640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:49.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68dd6b6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T01:57:49.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68dd6b6640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68d8137e50 0x7f68d813c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:49.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68dd6b6640 10 monclient(hunting): picked mon.noname-c con 0x7f68d8137e50 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:49.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68dd6b6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68d8138230 0x7f68d8136d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:49.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68dd6b6640 10 monclient(hunting): picked mon.noname-b con 0x7f68d8138230 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:57:49.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68dd6b6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68d813e390 0x7f68d813a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:49.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68dd6b6640 10 monclient(hunting): picked mon.noname-a con 0x7f68d813e390 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:57:49.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68dd6b6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:49.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68dd6b6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f68d8140e40 con 0x7f68d813e390 2024-09-11T01:57:49.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68dd6b6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:49.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68dd6b6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f68d8140cc0 con 0x7f68d8138230 2024-09-11T01:57:49.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68dd6b6640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:49.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68dd6b6640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7f68d8140fc0 con 0x7f68d8137e50 2024-09-11T01:57:49.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68dd6b6640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:49.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68dd6b6640 10 monclient(hunting): authenticate will time out at 2476.064779s 2024-09-11T01:57:49.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68d77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68d813e390 0x7f68d813a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:49.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68d6ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68d8137e50 0x7f68d813c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:49.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68d77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68d813e390 0x7f68d813a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52446/0 (socket says 172.21.15.12:52446) 2024-09-11T01:57:49.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68d77fe640 1 -- 172.21.15.12:0/166294996 learned_addr learned my addr 172.21.15.12:0/166294996 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:49.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68d77fe640 10 monclient(hunting): get_auth_request con 0x7f68d813e390 auth_method 0 2024-09-11T01:57:49.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:49.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:49.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:49.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68d6ffd640 10 monclient(hunting): get_auth_request con 0x7f68d8137e50 auth_method 0 2024-09-11T01:57:49.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:49.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:49.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.622+0000 7f68d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:49.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.623+0000 7f68d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:49.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.623+0000 7f68d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:49.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.623+0000 7f68d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:49.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.623+0000 7f68d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:49.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.623+0000 7f68d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:49.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.623+0000 7f68d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:49.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.623+0000 7f68d67fc640 1 --2- 172.21.15.12:0/166294996 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68d8138230 0x7f68d8136d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:49.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.623+0000 7f68d67fc640 10 monclient(hunting): get_auth_request con 0x7f68d8138230 auth_method 0 2024-09-11T01:57:49.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.623+0000 7f68d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:49.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.623+0000 7f68d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:49.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.623+0000 7f68d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:49.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.623+0000 7f68d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:49.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.623+0000 7f68d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:49.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.623+0000 7f68d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:49.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.623+0000 7f68d77fe640 10 monclient(hunting): handle_auth_done global_id 294399 payload 824 2024-09-11T01:57:49.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.623+0000 7f68d77fe640 1 -- 172.21.15.12:0/166294996 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68d8137e50 msgr2=0x7f68d813c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:49.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.623+0000 7f68d77fe640 1 --2- 172.21.15.12:0/166294996 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68d8137e50 0x7f68d813c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:49.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.623+0000 7f68d77fe640 1 -- 172.21.15.12:0/166294996 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68d8138230 msgr2=0x7f68d8136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:49.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.623+0000 7f68d77fe640 1 --2- 172.21.15.12:0/166294996 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68d8138230 0x7f68d8136d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:49.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.623+0000 7f68d77fe640 10 monclient: _finish_hunting 0 2024-09-11T01:57:49.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.623+0000 7f68d77fe640 1 monclient: found mon.noname-a 2024-09-11T01:57:49.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.623+0000 7f68d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:49.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.623+0000 7f68d77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2024-09-11T01:57:49.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.623+0000 7f68d77fe640 1 -- 172.21.15.12:0/166294996 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68d81cde90 con 0x7f68d813e390 2024-09-11T01:57:49.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.623+0000 7f68d6ffd640 1 --2- 172.21.15.12:0/166294996 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68d8137e50 0x7f68d813c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:57:49.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.625+0000 7f68d77fe640 1 -- 172.21.15.12:0/166294996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68d813e390 msgr2=0x7f68d813a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:49.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.625+0000 7f68d77fe640 1 --2- 172.21.15.12:0/166294996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68d813e390 0x7f68d813a130 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f68d002aa70 tx=0x7f68d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:49.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.625+0000 7f68bf7fe640 1 -- 172.21.15.12:0/166294996 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d0040070 con 0x7f68d813e390 2024-09-11T01:57:49.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.625+0000 7f68bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:49.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.625+0000 7f68bf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T01:57:49.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.625+0000 7f68bf7fe640 10 monclient: dump: 2024-09-11T01:57:49.643 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:49.644 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:49.644 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:49.644 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:49.644 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:49.644 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:49.644 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:49.644 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:49.645 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:49.645 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:49.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.625+0000 7f68bf7fe640 10 monclient: _finish_auth 0 2024-09-11T01:57:49.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.625+0000 7f68bf7fe640 10 monclient: _check_auth_tickets 2024-09-11T01:57:49.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.625+0000 7f68bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:49.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.625+0000 7f68bf7fe640 1 -- 172.21.15.12:0/166294996 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68d0045660 con 0x7f68d813e390 2024-09-11T01:57:49.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.625+0000 7f68bf7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:49.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.625+0000 7f68bf7fe640 1 -- 172.21.15.12:0/166294996 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d0045a30 con 0x7f68d813e390 2024-09-11T01:57:49.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.625+0000 7f68dd6b6640 5 monclient: authenticate success, global_id 294399 2024-09-11T01:57:49.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.625+0000 7f68bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:49.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.625+0000 7f68bf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T01:57:49.645 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.625+0000 7f68bf7fe640 10 monclient: dump: 2024-09-11T01:57:49.646 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:49.646 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:49.646 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:49.646 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:49.646 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:49.646 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:49.646 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:49.646 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:49.646 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:49.646 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:49.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.625+0000 7f68dd6b6640 10 monclient: _renew_subs 2024-09-11T01:57:49.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.625+0000 7f68dd6b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:49.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.625+0000 7f68d57fa640 4 set_mon_vals no callback set 2024-09-11T01:57:49.646 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.625+0000 7f68dd6b6640 1 -- 172.21.15.12:0/166294996 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68d813d090 con 0x7f68d813e390 2024-09-11T01:57:49.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.625+0000 7f68dd6b6640 10 monclient: _renew_subs 2024-09-11T01:57:49.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.625+0000 7f68dd6b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:49.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.625+0000 7f68dd6b6640 1 -- 172.21.15.12:0/166294996 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68d813bba0 con 0x7f68d813e390 2024-09-11T01:57:49.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.626+0000 7f68dd6b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:49.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.626+0000 7f68dd6b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:49.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.626+0000 7f68dd6b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2024-09-11T01:57:49.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.626+0000 7f68dd6b6640 1 -- 172.21.15.12:0/166294996 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68d809d160 con 0x7f68d813e390 2024-09-11T01:57:49.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.627+0000 7f68bf7fe640 1 -- 172.21.15.12:0/166294996 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7f68d0007a40 con 0x7f68d813e390 2024-09-11T01:57:49.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.627+0000 7f68bf7fe640 1 --2- 172.21.15.12:0/166294996 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f68c403e210 0x7f68c40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:49.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.627+0000 7f68bf7fe640 1 -- 172.21.15.12:0/166294996 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(923..923 src has 889..923) ==== 4422+0+0 (secure 0 0 0) 0x7f68d00749b0 con 0x7f68d813e390 2024-09-11T01:57:49.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.627+0000 7f68d6ffd640 1 --2- 172.21.15.12:0/166294996 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f68c403e210 0x7f68c40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:49.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.627+0000 7f68d6ffd640 10 monclient: get_auth_request con 0x7f68c403e210 auth_method 0 2024-09-11T01:57:49.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.630+0000 7f68d6ffd640 1 --2- 172.21.15.12:0/166294996 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f68c403e210 0x7f68c40406d0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f68c8004320 tx=0x7f68c8007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:49.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.631+0000 7f68bf7fe640 1 -- 172.21.15.12:0/166294996 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f68d003cbd0 con 0x7f68d813e390 2024-09-11T01:57:49.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.631+0000 7f68bf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:49.648 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.631+0000 7f68bf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:57:49.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.814+0000 7f68dd6b6640 1 -- 172.21.15.12:0/166294996 --> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f68d8069950 con 0x7f68c403e210 2024-09-11T01:57:49.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.816+0000 7f68bf7fe640 1 -- 172.21.15.12:0/166294996 <== mgr.4101 v2:172.21.15.71:6816/3189259519 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34041 (secure 0 0 0) 0x7f68d8069950 con 0x7f68c403e210 2024-09-11T01:57:49.818 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-09-11T01:57:49.818 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-09-11T01:57:49.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.818+0000 7f68dd6b6640 1 -- 172.21.15.12:0/166294996 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f68c403e210 msgr2=0x7f68c40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:49.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.818+0000 7f68dd6b6640 1 --2- 172.21.15.12:0/166294996 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f68c403e210 0x7f68c40406d0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f68c8004320 tx=0x7f68c8007e30 comp rx=0 tx=0).stop 2024-09-11T01:57:49.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.818+0000 7f68dd6b6640 10 monclient: shutdown 2024-09-11T01:57:49.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.818+0000 7f68dd6b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:49.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.818+0000 7f68dd6b6640 1 -- 172.21.15.12:0/166294996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68d813e390 msgr2=0x7f68d813a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:49.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.818+0000 7f68dd6b6640 1 --2- 172.21.15.12:0/166294996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68d813e390 0x7f68d813a130 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f68d002aa70 tx=0x7f68d0004a40 comp rx=0 tx=0).stop 2024-09-11T01:57:49.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.818+0000 7f68dd6b6640 1 -- 172.21.15.12:0/166294996 shutdown_connections 2024-09-11T01:57:49.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.818+0000 7f68dd6b6640 1 --2- 172.21.15.12:0/166294996 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7f68c403e210 0x7f68c40406d0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:49.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.818+0000 7f68dd6b6640 1 --2- 172.21.15.12:0/166294996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68d813e390 0x7f68d813a130 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:49.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.818+0000 7f68dd6b6640 1 --2- 172.21.15.12:0/166294996 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68d8138230 0x7f68d8136d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:49.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.818+0000 7f68dd6b6640 1 --2- 172.21.15.12:0/166294996 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f68d8137e50 0x7f68d813c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:49.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.818+0000 7f68dd6b6640 1 -- 172.21.15.12:0/166294996 >> 172.21.15.12:0/166294996 conn(0x7f68d80a7020 msgr2=0x7f68d813f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:49.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.819+0000 7f68dd6b6640 1 -- 172.21.15.12:0/166294996 shutdown_connections 2024-09-11T01:57:49.821 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.819+0000 7f68dd6b6640 1 -- 172.21.15.12:0/166294996 wait complete. 2024-09-11T01:57:49.831 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":1779,"stamp":"2024-09-11T01:57:49.214335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":332288,"kb_used_data":14816,"kb_used_omap":9,"kb_used_meta":317366,"kb_avail":565898752,"statfs":{"total":579820584960,"available":579480322048,"internally_reserved":0,"allocated":15171584,"data_stored":7598212,"data_compressed":59136,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":324983456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002696"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":1816,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:57:29.973868+0000","last_change":"2024-09-11T01:57:29.973868+0000","last_active":"2024-09-11T01:57:29.973868+0000","last_peered":"2024-09-11T01:57:29.973868+0000","last_clean":"2024-09-11T01:57:29.973868+0000","last_became_active":"2024-09-11T01:29:15.546952+0000","last_became_peered":"2024-09-11T01:29:15.546952+0000","last_unstale":"2024-09-11T01:57:29.973868+0000","last_undegraded":"2024-09-11T01:57:29.973868+0000","last_fullsized":"2024-09-11T01:57:29.973868+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:57:29.973794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:57:29.973794+0000","last_clean_scrub_stamp":"2024-09-11T01:57:29.973794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:22:11.690907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038245800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":1820,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:57:30.926500+0000","last_change":"2024-09-11T01:57:30.926500+0000","last_active":"2024-09-11T01:57:30.926500+0000","last_peered":"2024-09-11T01:57:30.926500+0000","last_clean":"2024-09-11T01:57:30.926500+0000","last_became_active":"2024-09-11T01:29:15.549013+0000","last_became_peered":"2024-09-11T01:29:15.549013+0000","last_unstale":"2024-09-11T01:57:30.926500+0000","last_undegraded":"2024-09-11T01:57:30.926500+0000","last_fullsized":"2024-09-11T01:57:30.926500+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:57:30.926405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:57:30.926405+0000","last_clean_scrub_stamp":"2024-09-11T01:57:30.926405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:25:28.420806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00041667199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"923'3","reported_seq":1831,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:57:30.051334+0000","last_change":"2024-09-11T01:57:30.051334+0000","last_active":"2024-09-11T01:57:30.051334+0000","last_peered":"2024-09-11T01:57:30.051334+0000","last_clean":"2024-09-11T01:57:30.051334+0000","last_became_active":"2024-09-11T01:29:15.548884+0000","last_became_peered":"2024-09-11T01:29:15.548884+0000","last_unstale":"2024-09-11T01:57:30.051334+0000","last_undegraded":"2024-09-11T01:57:30.051334+0000","last_fullsized":"2024-09-11T01:57:30.051334+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"923'3","last_scrub_stamp":"2024-09-11T01:57:30.051253+0000","last_deep_scrub":"923'3","last_deep_scrub_stamp":"2024-09-11T01:57:30.051253+0000","last_clean_scrub_stamp":"2024-09-11T01:57:30.051253+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:13:12.203643+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00094832499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"923'2","reported_seq":1826,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:57:27.624025+0000","last_change":"2024-09-11T01:57:27.624025+0000","last_active":"2024-09-11T01:57:27.624025+0000","last_peered":"2024-09-11T01:57:27.624025+0000","last_clean":"2024-09-11T01:57:27.624025+0000","last_became_active":"2024-09-11T01:29:15.546142+0000","last_became_peered":"2024-09-11T01:29:15.546142+0000","last_unstale":"2024-09-11T01:57:27.624025+0000","last_undegraded":"2024-09-11T01:57:27.624025+0000","last_fullsized":"2024-09-11T01:57:27.624025+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"923'2","last_scrub_stamp":"2024-09-11T01:57:27.623962+0000","last_deep_scrub":"923'2","last_deep_scrub_stamp":"2024-09-11T01:57:27.623962+0000","last_clean_scrub_stamp":"2024-09-11T01:57:27.623962+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:33:47.506941+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00044411199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"923'74","reported_seq":1901,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:57:31.022545+0000","last_change":"2024-09-11T01:57:31.022545+0000","last_active":"2024-09-11T01:57:31.022545+0000","last_peered":"2024-09-11T01:57:31.022545+0000","last_clean":"2024-09-11T01:57:31.022545+0000","last_became_active":"2024-09-11T01:29:16.690647+0000","last_became_peered":"2024-09-11T01:29:16.690647+0000","last_unstale":"2024-09-11T01:57:31.022545+0000","last_undegraded":"2024-09-11T01:57:31.022545+0000","last_fullsized":"2024-09-11T01:57:31.022545+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"923'74","last_scrub_stamp":"2024-09-11T01:57:31.022460+0000","last_deep_scrub":"923'74","last_deep_scrub_stamp":"2024-09-11T01:57:31.022460+0000","last_clean_scrub_stamp":"2024-09-11T01:57:31.022460+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:17:26.314718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":1824,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:57:28.657032+0000","last_change":"2024-09-11T01:57:28.657032+0000","last_active":"2024-09-11T01:57:28.657032+0000","last_peered":"2024-09-11T01:57:28.657032+0000","last_clean":"2024-09-11T01:57:28.657032+0000","last_became_active":"2024-09-11T01:29:15.547447+0000","last_became_peered":"2024-09-11T01:29:15.547447+0000","last_unstale":"2024-09-11T01:57:28.657032+0000","last_undegraded":"2024-09-11T01:57:28.657032+0000","last_fullsized":"2024-09-11T01:57:28.657032+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:57:28.656967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:57:28.656967+0000","last_clean_scrub_stamp":"2024-09-11T01:57:28.656967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:10:48.472977+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00037448800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":1828,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:57:29.072316+0000","last_change":"2024-09-11T01:57:29.072316+0000","last_active":"2024-09-11T01:57:29.072316+0000","last_peered":"2024-09-11T01:57:29.072316+0000","last_clean":"2024-09-11T01:57:29.072316+0000","last_became_active":"2024-09-11T01:29:15.546476+0000","last_became_peered":"2024-09-11T01:29:15.546476+0000","last_unstale":"2024-09-11T01:57:29.072316+0000","last_undegraded":"2024-09-11T01:57:29.072316+0000","last_fullsized":"2024-09-11T01:57:29.072316+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:57:29.072237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:57:29.072237+0000","last_clean_scrub_stamp":"2024-09-11T01:57:29.072237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:08:18.052058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00037031899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":1815,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:57:28.965556+0000","last_change":"2024-09-11T01:57:28.965486+0000","last_active":"2024-09-11T01:57:28.965556+0000","last_peered":"2024-09-11T01:57:28.965556+0000","last_clean":"2024-09-11T01:57:28.965556+0000","last_became_active":"2024-09-11T01:29:15.542800+0000","last_became_peered":"2024-09-11T01:29:15.542800+0000","last_unstale":"2024-09-11T01:57:28.965556+0000","last_undegraded":"2024-09-11T01:57:28.965556+0000","last_fullsized":"2024-09-11T01:57:28.965556+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:57:28.965390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:57:28.965390+0000","last_clean_scrub_stamp":"2024-09-11T01:57:28.965390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:09:42.587610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00040701099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":1820,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T01:57:29.952373+0000","last_change":"2024-09-11T01:57:29.952373+0000","last_active":"2024-09-11T01:57:29.952373+0000","last_peered":"2024-09-11T01:57:29.952373+0000","last_clean":"2024-09-11T01:57:29.952373+0000","last_became_active":"2024-09-11T01:29:15.546444+0000","last_became_peered":"2024-09-11T01:29:15.546444+0000","last_unstale":"2024-09-11T01:57:29.952373+0000","last_undegraded":"2024-09-11T01:57:29.952373+0000","last_fullsized":"2024-09-11T01:57:29.952373+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T01:57:29.952288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T01:57:29.952288+0000","last_clean_scrub_stamp":"2024-09-11T01:57:29.952288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:24:27.260956+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000396053,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59136,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607899,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47520,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324320,"statfs":{"total":96636764160,"available":96588103680,"internally_reserved":0,"allocated":2506752,"data_stored":1090053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607897,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55844,"kb_used_data":2452,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315996,"statfs":{"total":96636764160,"available":96579579904,"internally_reserved":0,"allocated":2510848,"data_stored":1090072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55900,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315940,"statfs":{"total":96636764160,"available":96579522560,"internally_reserved":0,"allocated":2568192,"data_stored":1618981,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607898,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69724,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302116,"statfs":{"total":96636764160,"available":96565366784,"internally_reserved":0,"allocated":2568192,"data_stored":1618981,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33636,"kb_used_data":2452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338204,"statfs":{"total":96636764160,"available":96602320896,"internally_reserved":0,"allocated":2510848,"data_stored":1090072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640605,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69664,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302176,"statfs":{"total":96636764160,"available":96565428224,"internally_reserved":0,"allocated":2506752,"data_stored":1090053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T01:57:49.832 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-11T01:57:49.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.932+0000 7fb45ff0e640 10 monclient: get_monmap_and_config 2024-09-11T01:57:49.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.932+0000 7fb45ff0e640 10 monclient: build_initial_monmap 2024-09-11T01:57:49.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.932+0000 7fb45ff0e640 10 monclient: monmap: 2024-09-11T01:57:49.933 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:49.934 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:49.934 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:57:49.932841+0000 2024-09-11T01:57:49.934 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:57:49.932841+0000 2024-09-11T01:57:49.934 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:49.934 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:49.934 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2024-09-11T01:57:49.934 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-09-11T01:57:49.934 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2024-09-11T01:57:49.934 INFO:teuthology.orchestra.run.smithi012.stderr:3: v2:172.21.15.12:3300/0 mon.noname-a 2024-09-11T01:57:49.934 INFO:teuthology.orchestra.run.smithi012.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-09-11T01:57:49.934 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:49.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.932+0000 7fb45ff0e640 1 Processor -- start 2024-09-11T01:57:49.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.932+0000 7fb45ff0e640 1 -- start start 2024-09-11T01:57:49.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.932+0000 7fb45ff0e640 10 monclient: init 2024-09-11T01:57:49.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45ff0e640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:49.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45ff0e640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-11T01:57:49.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45ff0e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb458137ec0 addr v1:172.21.15.71:6789/0 2024-09-11T01:57:49.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45ff0e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4581389b0 0x7fb4580a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:49.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45ff0e640 10 monclient(hunting): picked mon.noname-b con 0x7fb4581389b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:49.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45ff0e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb45813d460 addr v1:172.21.15.12:6789/0 2024-09-11T01:57:49.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45ff0e640 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb45809c2c0 con 0x7fb45813d460 2024-09-11T01:57:49.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45ff0e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb45809c530 con 0x7fb458137ec0 2024-09-11T01:57:49.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45ff0e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:49.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45ff0e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb4580a68f0 con 0x7fb4581389b0 2024-09-11T01:57:49.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45ff0e640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:49.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45ff0e640 10 monclient(hunting): authenticate will time out at 2476.375454s 2024-09-11T01:57:49.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45d4a9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4581389b0 0x7fb4580a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:49.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45e4ab640 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb45813d460 0x7fb45813f850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:42944/0 (socket says 172.21.15.12:42944) 2024-09-11T01:57:49.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45e4ab640 1 -- 172.21.15.12:0/3862318427 learned_addr learned my addr 172.21.15.12:0/3862318427 (peer_addr_for_me v1:172.21.15.12:0/0) 2024-09-11T01:57:49.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45d4a9640 10 monclient(hunting): get_auth_request con 0x7fb4581389b0 auth_method 0 2024-09-11T01:57:49.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45d4a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:49.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45d4a9640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:49.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45d4a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:49.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45d4a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:49.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45d4a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:49.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45d4a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:49.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45d4a9640 10 monclient(hunting): handle_auth_done global_id 304304 payload 274 2024-09-11T01:57:49.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45d4a9640 1 -- 172.21.15.12:0/3862318427 >> v1:172.21.15.71:6789/0 conn(0x7fb458137ec0 legacy=0x7fb4581382a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:49.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45d4a9640 1 -- 172.21.15.12:0/3862318427 >> v1:172.21.15.12:6789/0 conn(0x7fb45813d460 legacy=0x7fb45813f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T01:57:49.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45d4a9640 10 monclient: _finish_hunting 0 2024-09-11T01:57:49.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45d4a9640 1 monclient: found mon.noname-b 2024-09-11T01:57:49.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45d4a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:49.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45d4a9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2024-09-11T01:57:49.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.933+0000 7fb45d4a9640 1 -- 172.21.15.12:0/3862318427 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb45809c810 con 0x7fb4581389b0 2024-09-11T01:57:49.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.936+0000 7fb45d4a9640 1 -- 172.21.15.12:0/3862318427 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4581389b0 msgr2=0x7fb4580a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:49.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.936+0000 7fb45d4a9640 1 --2- 172.21.15.12:0/3862318427 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4581389b0 0x7fb4580a4e90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb4480097d0 tx=0x7fb448029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86ff85af4747879a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:49.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.937+0000 7fb45cca8640 1 -- 172.21.15.12:0/3862318427 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb448037070 con 0x7fb4581389b0 2024-09-11T01:57:49.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.937+0000 7fb45cca8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:49.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.937+0000 7fb45cca8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T01:57:49.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.937+0000 7fb45cca8640 10 monclient: dump: 2024-09-11T01:57:49.944 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:49.944 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:49.944 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:49.944 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:49.945 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:49.945 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:49.945 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:49.945 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:49.945 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:49.945 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:49.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.937+0000 7fb45cca8640 10 monclient: _finish_auth 0 2024-09-11T01:57:49.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.937+0000 7fb45cca8640 10 monclient: _check_auth_tickets 2024-09-11T01:57:49.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.937+0000 7fb45cca8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:49.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.937+0000 7fb45cca8640 1 -- 172.21.15.12:0/3862318427 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb448032440 con 0x7fb4581389b0 2024-09-11T01:57:49.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.937+0000 7fb45ff0e640 5 monclient: authenticate success, global_id 304304 2024-09-11T01:57:49.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.937+0000 7fb45ff0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T01:57:49.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.937+0000 7fb45cca8640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:49.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.937+0000 7fb45ff0e640 10 monclient: get_monmap_and_config success 2024-09-11T01:57:49.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.937+0000 7fb45cca8640 1 -- 172.21.15.12:0/3862318427 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb448032810 con 0x7fb4581389b0 2024-09-11T01:57:49.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.937+0000 7fb45ff0e640 4 set_mon_vals no callback set 2024-09-11T01:57:49.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.937+0000 7fb45cca8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:49.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.937+0000 7fb45cca8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:57:49.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.937+0000 7fb45cca8640 10 monclient: dump: 2024-09-11T01:57:49.946 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:49.946 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:49.946 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:49.946 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:49.946 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:49.946 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:49.947 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:49.947 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:49.947 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:49.947 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:49.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.937+0000 7fb45ff0e640 10 monclient: shutdown 2024-09-11T01:57:49.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.937+0000 7fb45ff0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:49.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.937+0000 7fb45ff0e640 1 -- 172.21.15.12:0/3862318427 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4581389b0 msgr2=0x7fb4580a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:49.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.937+0000 7fb45ff0e640 1 --2- 172.21.15.12:0/3862318427 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4581389b0 0x7fb4580a4e90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb4480097d0 tx=0x7fb448029fb0 comp rx=0 tx=0).stop 2024-09-11T01:57:49.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.937+0000 7fb45ff0e640 1 -- 172.21.15.12:0/3862318427 shutdown_connections 2024-09-11T01:57:49.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.937+0000 7fb45ff0e640 1 --2- 172.21.15.12:0/3862318427 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4581389b0 0x7fb4580a4e90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:49.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.937+0000 7fb45ff0e640 1 -- 172.21.15.12:0/3862318427 >> 172.21.15.12:0/3862318427 conn(0x7fb45806ec70 msgr2=0x7fb45806f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:49.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.937+0000 7fb45ff0e640 1 -- 172.21.15.12:0/3862318427 shutdown_connections 2024-09-11T01:57:49.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.937+0000 7fb45ff0e640 1 -- 172.21.15.12:0/3862318427 wait complete. 2024-09-11T01:57:49.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.938+0000 7fb45ff0e640 10 monclient: build_initial_monmap 2024-09-11T01:57:49.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.938+0000 7fb45ff0e640 10 monclient: monmap: 2024-09-11T01:57:49.948 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2024-09-11T01:57:49.948 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T01:57:49.948 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2024-09-11T01:57:49.948 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2024-09-11T01:57:49.948 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2024-09-11T01:57:49.948 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:49.948 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2024-09-11T01:57:49.948 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-09-11T01:57:49.948 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2024-09-11T01:57:49.948 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:49.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.938+0000 7fb45ff0e640 1 Processor -- start 2024-09-11T01:57:49.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.938+0000 7fb45ff0e640 1 -- start start 2024-09-11T01:57:49.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.938+0000 7fb45ff0e640 10 monclient: init 2024-09-11T01:57:49.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45ff0e640 10 monclient: _reopen_session rank -1 2024-09-11T01:57:49.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45ff0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T01:57:49.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45ff0e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb458137ec0 0x7fb458136c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:49.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45ff0e640 10 monclient(hunting): picked mon.noname-b con 0x7fb458137ec0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-11T01:57:49.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45ff0e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4581389b0 0x7fb458139440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:49.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45ff0e640 10 monclient(hunting): picked mon.noname-c con 0x7fb4581389b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-09-11T01:57:49.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45ff0e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb45813d460 0x7fb458139980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:49.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45ff0e640 10 monclient(hunting): picked mon.noname-a con 0x7fb45813d460 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2024-09-11T01:57:49.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45ff0e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:49.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45ff0e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb45809d810 con 0x7fb45813d460 2024-09-11T01:57:49.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45ff0e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:49.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45ff0e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb45809d690 con 0x7fb458137ec0 2024-09-11T01:57:49.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45ff0e640 10 monclient(hunting): start opening mon connection 2024-09-11T01:57:49.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45ff0e640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 -- 0x7fb458140f70 con 0x7fb4581389b0 2024-09-11T01:57:49.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45ff0e640 10 monclient(hunting): _renew_subs 2024-09-11T01:57:49.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45ff0e640 10 monclient(hunting): authenticate will time out at 2476.381163s 2024-09-11T01:57:49.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45d4a9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4581389b0 0x7fb458139440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:49.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45e4ab640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb45813d460 0x7fb458139980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:49.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45d4a9640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4581389b0 0x7fb458139440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:57390/0 (socket says 172.21.15.12:57390) 2024-09-11T01:57:49.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45d4a9640 1 -- 172.21.15.12:0/303848284 learned_addr learned my addr 172.21.15.12:0/303848284 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-09-11T01:57:49.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45d4a9640 10 monclient(hunting): get_auth_request con 0x7fb4581389b0 auth_method 0 2024-09-11T01:57:49.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45d4a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:49.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45d4a9640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:49.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45d4a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:49.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45e4ab640 10 monclient(hunting): get_auth_request con 0x7fb45813d460 auth_method 0 2024-09-11T01:57:49.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45e4ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:49.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45e4ab640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:49.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45e4ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:49.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45d4a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:49.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45d4a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:49.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45d4a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:49.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45e4ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T01:57:49.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45e4ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T01:57:49.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45e4ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T01:57:49.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45dcaa640 1 --2- 172.21.15.12:0/303848284 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb458137ec0 0x7fb458136c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:49.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45dcaa640 10 monclient(hunting): get_auth_request con 0x7fb458137ec0 auth_method 0 2024-09-11T01:57:49.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45dcaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T01:57:49.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45dcaa640 10 monclient(hunting): _init_auth method 2 2024-09-11T01:57:49.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45dcaa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T01:57:49.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45d4a9640 10 monclient(hunting): handle_auth_done global_id 304307 payload 824 2024-09-11T01:57:49.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45d4a9640 1 -- 172.21.15.12:0/303848284 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb458137ec0 msgr2=0x7fb458136c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:49.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45d4a9640 1 --2- 172.21.15.12:0/303848284 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb458137ec0 0x7fb458136c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:49.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45d4a9640 1 -- 172.21.15.12:0/303848284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb45813d460 msgr2=0x7fb458139980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:49.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45d4a9640 1 --2- 172.21.15.12:0/303848284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb45813d460 0x7fb458139980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:49.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45d4a9640 10 monclient: _finish_hunting 0 2024-09-11T01:57:49.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45d4a9640 1 monclient: found mon.noname-c 2024-09-11T01:57:49.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45d4a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T01:57:49.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45d4a9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2024-09-11T01:57:49.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45d4a9640 1 -- 172.21.15.12:0/303848284 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb45813cc90 con 0x7fb4581389b0 2024-09-11T01:57:49.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45dcaa640 1 --2- 172.21.15.12:0/303848284 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb458137ec0 0x7fb458136c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T01:57:49.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.939+0000 7fb45e4ab640 1 --2- 172.21.15.12:0/303848284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb45813d460 0x7fb458139980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T01:57:49.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.941+0000 7fb45d4a9640 1 -- 172.21.15.12:0/303848284 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4581389b0 msgr2=0x7fb458139440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T01:57:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.941+0000 7fb45d4a9640 1 --2- 172.21.15.12:0/303848284 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4581389b0 0x7fb458139440 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb4480098f0 tx=0x7fb4480046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.942+0000 7fb44e7fc640 1 -- 172.21.15.12:0/303848284 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb448042020 con 0x7fb4581389b0 2024-09-11T01:57:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.942+0000 7fb44e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.942+0000 7fb44e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T01:57:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.942+0000 7fb44e7fc640 10 monclient: dump: 2024-09-11T01:57:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:49.956 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:49.957 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:49.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.942+0000 7fb44e7fc640 10 monclient: _finish_auth 0 2024-09-11T01:57:49.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.942+0000 7fb44e7fc640 10 monclient: _check_auth_tickets 2024-09-11T01:57:49.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.942+0000 7fb44e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T01:57:49.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.942+0000 7fb44e7fc640 1 -- 172.21.15.12:0/303848284 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb44803d670 con 0x7fb4581389b0 2024-09-11T01:57:49.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.942+0000 7fb45ff0e640 5 monclient: authenticate success, global_id 304307 2024-09-11T01:57:49.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.942+0000 7fb45ff0e640 10 monclient: _renew_subs 2024-09-11T01:57:49.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.942+0000 7fb45ff0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:49.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.942+0000 7fb45ff0e640 1 -- 172.21.15.12:0/303848284 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb45813cf70 con 0x7fb4581389b0 2024-09-11T01:57:49.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.942+0000 7fb44e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T01:57:49.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.942+0000 7fb44e7fc640 1 -- 172.21.15.12:0/303848284 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb448037040 con 0x7fb4581389b0 2024-09-11T01:57:49.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.942+0000 7fb44e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T01:57:49.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.942+0000 7fb44e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T01:57:49.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.942+0000 7fb44e7fc640 10 monclient: dump: 2024-09-11T01:57:49.958 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2024-09-11T01:57:49.958 INFO:teuthology.orchestra.run.smithi012.stderr:fsid ef0d30d5-33dd-4ea5-8f4e-e0a3c2d113ab 2024-09-11T01:57:49.958 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:49.958 INFO:teuthology.orchestra.run.smithi012.stderr:created 2024-09-11T01:28:30.421598+0000 2024-09-11T01:57:49.958 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 19 (squid) 2024-09-11T01:57:49.958 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2024-09-11T01:57:49.958 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-09-11T01:57:49.958 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-11T01:57:49.958 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2024-09-11T01:57:49.958 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:49.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.942+0000 7fb44ffff640 4 set_mon_vals no callback set 2024-09-11T01:57:49.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.942+0000 7fb45ff0e640 10 monclient: _renew_subs 2024-09-11T01:57:49.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.942+0000 7fb45ff0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:49.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.942+0000 7fb45ff0e640 1 -- 172.21.15.12:0/303848284 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4580a7970 con 0x7fb4581389b0 2024-09-11T01:57:49.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.942+0000 7fb44e7fc640 1 -- 172.21.15.12:0/303848284 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) ==== 50613+0+0 (secure 0 0 0) 0x7fb44803da60 con 0x7fb4581389b0 2024-09-11T01:57:49.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.943+0000 7fb45ff0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:49.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.943+0000 7fb45ff0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:49.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.943+0000 7fb45ff0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:49.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.943+0000 7fb44e7fc640 1 --2- 172.21.15.12:0/303848284 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb43c03e260 0x7fb43c040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T01:57:49.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.943+0000 7fb45ff0e640 1 -- 172.21.15.12:0/303848284 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4580a4e90 con 0x7fb4581389b0 2024-09-11T01:57:49.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.943+0000 7fb44e7fc640 1 -- 172.21.15.12:0/303848284 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(923..923 src has 889..923) ==== 4422+0+0 (secure 0 0 0) 0x7fb448073640 con 0x7fb4581389b0 2024-09-11T01:57:49.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.943+0000 7fb45dcaa640 1 --2- 172.21.15.12:0/303848284 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb43c03e260 0x7fb43c040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T01:57:49.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.943+0000 7fb45dcaa640 10 monclient: get_auth_request con 0x7fb43c03e260 auth_method 0 2024-09-11T01:57:49.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.946+0000 7fb45dcaa640 1 --2- 172.21.15.12:0/303848284 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb43c03e260 0x7fb43c040720 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb454003eb0 tx=0x7fb454009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T01:57:49.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.947+0000 7fb44e7fc640 1 -- 172.21.15.12:0/303848284 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb44804c030 con 0x7fb4581389b0 2024-09-11T01:57:49.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.947+0000 7fb44e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T01:57:49.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:49.947+0000 7fb44e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T01:57:50.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:50.131+0000 7fb45ff0e640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-11T01:57:50.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:50.131+0000 7fb45ff0e640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-11T01:57:50.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:50.131+0000 7fb45ff0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2024-09-11T01:57:50.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:50.131+0000 7fb45ff0e640 1 -- 172.21.15.12:0/303848284 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fb4580698c0 con 0x7fb4581389b0 2024-09-11T01:57:50.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:50.135+0000 7fb44e7fc640 1 -- 172.21.15.12:0/303848284 <== mon.2 v2:172.21.15.12:3301/0 7 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fb44803b370 con 0x7fb4581389b0 2024-09-11T01:57:50.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:50.135+0000 7fb44e7fc640 10 monclient: handle_config config(1 keys) 2024-09-11T01:57:50.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:50.135+0000 7fb44ffff640 4 set_mon_vals no callback set 2024-09-11T01:57:50.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:50.135+0000 7fb44ffff640 10 set_mon_vals mon_health_to_clog = false 2024-09-11T01:57:50.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:50.135+0000 7fb44e7fc640 1 -- 172.21.15.12:0/303848284 <== mon.2 v2:172.21.15.12:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v6)=0 v6) ==== 123+0+0 (secure 0 0 0) 0x7fb4480721a0 con 0x7fb4581389b0 2024-09-11T01:57:50.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:50.135+0000 7fb44e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-11T01:57:50.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:50.135+0000 7fb44e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T01:57:50.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:50.138+0000 7fb45ff0e640 1 -- 172.21.15.12:0/303848284 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb43c03e260 msgr2=0x7fb43c040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:50.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:50.138+0000 7fb45ff0e640 1 --2- 172.21.15.12:0/303848284 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb43c03e260 0x7fb43c040720 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb454003eb0 tx=0x7fb454009040 comp rx=0 tx=0).stop 2024-09-11T01:57:50.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:50.138+0000 7fb45ff0e640 10 monclient: shutdown 2024-09-11T01:57:50.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:50.138+0000 7fb45ff0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T01:57:50.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:50.138+0000 7fb45ff0e640 1 -- 172.21.15.12:0/303848284 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4581389b0 msgr2=0x7fb458139440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T01:57:50.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:50.138+0000 7fb45ff0e640 1 --2- 172.21.15.12:0/303848284 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4581389b0 0x7fb458139440 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb4480098f0 tx=0x7fb4480046c0 comp rx=0 tx=0).stop 2024-09-11T01:57:50.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:50.138+0000 7fb45ff0e640 1 -- 172.21.15.12:0/303848284 shutdown_connections 2024-09-11T01:57:50.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:50.138+0000 7fb45ff0e640 1 --2- 172.21.15.12:0/303848284 >> [v2:172.21.15.71:6816/3189259519,v1:172.21.15.71:6817/3189259519] conn(0x7fb43c03e260 0x7fb43c040720 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:50.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:50.138+0000 7fb45ff0e640 1 --2- 172.21.15.12:0/303848284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb45813d460 0x7fb458139980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:50.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:50.138+0000 7fb45ff0e640 1 --2- 172.21.15.12:0/303848284 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4581389b0 0x7fb458139440 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:50.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:50.138+0000 7fb45ff0e640 1 --2- 172.21.15.12:0/303848284 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb458137ec0 0x7fb458136c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T01:57:50.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:50.138+0000 7fb45ff0e640 1 -- 172.21.15.12:0/303848284 >> 172.21.15.12:0/303848284 conn(0x7fb45806ec70 msgr2=0x7fb4580a20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T01:57:50.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:50.138+0000 7fb45ff0e640 1 -- 172.21.15.12:0/303848284 shutdown_connections 2024-09-11T01:57:50.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-09-11T01:57:50.138+0000 7fb45ff0e640 1 -- 172.21.15.12:0/303848284 wait complete. 2024-09-11T01:57:50.150 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-11T01:57:50.150 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-11T01:57:50.150 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-11T01:57:50.151 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:57:50.187 INFO:tasks.ceph.osd.0:Stopped 2024-09-11T01:57:50.187 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-11T01:57:50.187 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:57:50.260 INFO:tasks.ceph.osd.1:Stopped 2024-09-11T01:57:50.260 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-11T01:57:50.260 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:57:50.332 INFO:tasks.ceph.osd.2:Stopped 2024-09-11T01:57:50.332 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-11T01:57:50.332 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:57:50.375 INFO:tasks.ceph.osd.3:Stopped 2024-09-11T01:57:50.375 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-11T01:57:50.376 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:57:50.413 INFO:tasks.ceph.osd.4:Stopped 2024-09-11T01:57:50.414 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-11T01:57:50.414 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:57:50.487 INFO:tasks.ceph.osd.5:Stopped 2024-09-11T01:57:50.487 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-11T01:57:50.488 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-11T01:57:50.488 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:57:50.559 INFO:tasks.ceph.mgr.x:Stopped 2024-09-11T01:57:50.559 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-11T01:57:50.559 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T01:57:50.559 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:57:50.573 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T01:57:50.574 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-11T01:57:50.574 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:57:50.626 INFO:tasks.ceph.mon.c:Stopped 2024-09-11T01:57:50.626 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-11T01:57:50.626 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T01:57:50.638 INFO:tasks.ceph.mon.b:Stopped 2024-09-11T01:57:50.638 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-11T01:57:50.639 DEBUG:teuthology.orchestra.run.smithi012:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-11T01:57:50.704 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi012.front.sepia.ceph.com 2024-09-11T01:57:50.704 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-11T01:57:50.958 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi012.front.sepia.ceph.com 2024-09-11T01:57:50.958 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-11T01:57:51.056 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi012.front.sepia.ceph.com 2024-09-11T01:57:51.056 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-11T01:57:51.115 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi071.front.sepia.ceph.com 2024-09-11T01:57:51.115 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-11T01:57:51.246 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi071.front.sepia.ceph.com 2024-09-11T01:57:51.247 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-11T01:57:51.334 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi071.front.sepia.ceph.com 2024-09-11T01:57:51.334 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-11T01:57:51.409 INFO:tasks.ceph:Archiving mon data... 2024-09-11T01:57:51.411 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7899900/data/mon.a.tgz 2024-09-11T01:57:51.411 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2024-09-11T01:57:51.429 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.kaucvSRIbn 2024-09-11T01:57:51.429 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.kaucvSRIbn 2024-09-11T01:57:51.780 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.kaucvSRIbn 2024-09-11T01:57:51.865 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.kaucvSRIbn is 801KB 2024-09-11T01:57:51.967 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.kaucvSRIbn 2024-09-11T01:57:51.983 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7899900/data/mon.c.tgz 2024-09-11T01:57:51.983 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2024-09-11T01:57:52.037 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.CBlk48YxZr 2024-09-11T01:57:52.038 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.CBlk48YxZr 2024-09-11T01:57:52.223 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.CBlk48YxZr 2024-09-11T01:57:52.307 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.CBlk48YxZr is 804KB 2024-09-11T01:57:52.409 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.CBlk48YxZr 2024-09-11T01:57:52.425 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7899900/data/mon.b.tgz 2024-09-11T01:57:52.425 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2024-09-11T01:57:52.444 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.hNwWuJoc71 2024-09-11T01:57:52.445 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.hNwWuJoc71 2024-09-11T01:57:52.646 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.hNwWuJoc71 2024-09-11T01:57:52.728 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.hNwWuJoc71 is 801KB 2024-09-11T01:57:52.835 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.hNwWuJoc71 2024-09-11T01:57:52.849 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-11T01:57:52.850 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T01:57:52.852 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T01:57:52.934 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T01:57:52.969 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T01:57:52.969 INFO:tasks.ceph:Archiving crash dumps... 2024-09-11T01:57:52.971 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/crash to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7899900/remote/smithi012/crash 2024-09-11T01:57:52.972 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T01:57:53.001 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/crash to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7899900/remote/smithi071/crash 2024-09-11T01:57:53.002 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T01:57:53.033 INFO:tasks.ceph:Compressing logs... 2024-09-11T01:57:53.033 DEBUG:teuthology.orchestra.run.smithi012:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T01:57:53.046 DEBUG:teuthology.orchestra.run.smithi071:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T01:57:53.070 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-11T01:57:53.070 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36027.log 2024-09-11T01:57:53.107 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31717.log 2024-09-11T01:57:53.107 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31677.log 2024-09-11T01:57:53.108 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31637.log 2024-09-11T01:57:53.108 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31677.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-11T01:57:53.108 INFO:teuthology.orchestra.run.smithi071.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.31717.log.gz 2024-09-11T01:57:53.108 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31637.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-11T01:57:53.108 INFO:teuthology.orchestra.run.smithi071.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31677.log.gz 2024-09-11T01:57:53.108 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-11T01:57:53.108 INFO:teuthology.orchestra.run.smithi071.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31637.log.gz 2024-09-11T01:57:53.108 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-11T01:57:53.108 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-11T01:57:53.108 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31506.log 2024-09-11T01:57:53.108 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.34777.log 2024-09-11T01:57:53.109 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31506.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-11T01:57:53.109 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31597.log 2024-09-11T01:57:53.109 INFO:teuthology.orchestra.run.smithi071.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.31506.log.gz 2024-09-11T01:57:53.109 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.0.34777.log: /var/log/ceph/ceph-mon.b.log: 90.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-11T01:57:53.109 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-11T01:57:53.109 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31552.log 2024-09-11T01:57:53.109 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.3.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31597.log.gz 2024-09-11T01:57:53.128 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31552.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.31552.log.gz 2024-09-11T01:57:53.128 INFO:teuthology.orchestra.run.smithi071.stderr: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-11T01:57:53.189 INFO:teuthology.orchestra.run.smithi071.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.0.34777.log.gz 2024-09-11T01:57:53.328 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37880.log 2024-09-11T01:57:53.328 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.36027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40374.log 2024-09-11T01:57:53.328 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.37880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54633.log 2024-09-11T01:57:53.328 INFO:teuthology.orchestra.run.smithi012.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36027.log.gz 2024-09-11T01:57:53.329 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.40374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54075.log 2024-09-11T01:57:53.329 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37880.log.gz 2024-09-11T01:57:53.329 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50971.log 2024-09-11T01:57:53.329 INFO:teuthology.orchestra.run.smithi012.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.40374.log.gz 2024-09-11T01:57:53.329 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35263.log 2024-09-11T01:57:53.329 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54633.log.gz 2024-09-11T01:57:53.329 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46499.log 2024-09-11T01:57:53.329 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50971.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54075.log.gz 2024-09-11T01:57:53.329 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.35263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41496.log 2024-09-11T01:57:53.329 INFO:teuthology.orchestra.run.smithi012.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.50971.log.gz 2024-09-11T01:57:53.329 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46499.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35263.log.gz 2024-09-11T01:57:53.329 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53762.log 2024-09-11T01:57:53.329 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50270.log 2024-09-11T01:57:53.330 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.41496.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.46499.log.gz 2024-09-11T01:57:53.330 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41214.log 2024-09-11T01:57:53.330 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35014.log 2024-09-11T01:57:53.330 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53762.log.gz 2024-09-11T01:57:53.330 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41496.log.gz 2024-09-11T01:57:53.330 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.41214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41835.log 2024-09-11T01:57:53.330 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.35014.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.50270.log.gz 2024-09-11T01:57:53.330 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55221.log 2024-09-11T01:57:53.330 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41214.log.gz 2024-09-11T01:57:53.330 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35014.log.gz 2024-09-11T01:57:53.331 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35881.log 2024-09-11T01:57:53.331 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.41835.log: /var/log/ceph/ceph-client.admin.55221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52947.log 2024-09-11T01:57:53.331 INFO:teuthology.orchestra.run.smithi012.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.41835.log.gz 2024-09-11T01:57:53.331 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.35881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51920.log 2024-09-11T01:57:53.331 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.55221.log.gz 2024-09-11T01:57:53.331 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52947.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35881.log.gz 2024-09-11T01:57:53.331 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50228.log 2024-09-11T01:57:53.331 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52947.log.gz 2024-09-11T01:57:53.331 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44776.log 2024-09-11T01:57:53.331 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51920.log: /var/log/ceph/ceph-client.admin.50228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39979.log 2024-09-11T01:57:53.331 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48155.log 2024-09-11T01:57:53.331 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51920.log.gz 2024-09-11T01:57:53.331 INFO:teuthology.orchestra.run.smithi012.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.50228.log.gz 2024-09-11T01:57:53.332 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.39979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43451.log 2024-09-11T01:57:53.332 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44776.log.gz 2024-09-11T01:57:53.332 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47255.log 2024-09-11T01:57:53.332 INFO:teuthology.orchestra.run.smithi012.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.39979.log.gz 2024-09-11T01:57:53.332 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56391.log 2024-09-11T01:57:53.332 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.48155.log.gz 2024-09-11T01:57:53.332 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53300.log 2024-09-11T01:57:53.332 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43451.log.gz 2024-09-11T01:57:53.332 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54507.log 2024-09-11T01:57:53.332 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47255.log.gz 2024-09-11T01:57:53.332 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36828.log 2024-09-11T01:57:53.332 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56391.log.gz 2024-09-11T01:57:53.332 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56997.log 2024-09-11T01:57:53.333 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53300.log.gz 2024-09-11T01:57:53.333 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.36828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49706.log 2024-09-11T01:57:53.333 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.54507.log.gz 2024-09-11T01:57:53.333 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56997.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36828.log.gz 2024-09-11T01:57:53.333 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44818.log 2024-09-11T01:57:53.333 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55700.log 2024-09-11T01:57:53.333 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56997.log.gz 2024-09-11T01:57:53.333 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49285.log 2024-09-11T01:57:53.333 INFO:teuthology.orchestra.run.smithi012.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.49706.log.gz 2024-09-11T01:57:53.333 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47860.log 2024-09-11T01:57:53.333 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49285.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.44818.log.gz 2024-09-11T01:57:53.333 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54465.log 2024-09-11T01:57:53.333 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55700.log.gz 2024-09-11T01:57:53.334 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47860.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49285.log.gz 2024-09-11T01:57:53.334 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54549.log 2024-09-11T01:57:53.334 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47860.log.gz 2024-09-11T01:57:53.334 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38578.log 2024-09-11T01:57:53.334 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53552.log 2024-09-11T01:57:53.334 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5%/var/log/ceph/ceph-client.admin.38578.log: -- replaced with /var/log/ceph/ceph-client.admin.54465.log.gz 2024-09-11T01:57:53.334 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54549.log.gz 2024-09-11T01:57:53.334 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40794.log 2024-09-11T01:57:53.334 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 89.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.38578.log.gz -- 2024-09-11T01:57:53.334 INFO:teuthology.orchestra.run.smithi012.stderr: /var/log/ceph/ceph-client.admin.37521.log 2024-09-11T01:57:53.334 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53552.log: /var/log/ceph/ceph-client.admin.40794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34789.log 2024-09-11T01:57:53.334 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.37521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53073.log 2024-09-11T01:57:53.335 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.53552.log.gz 2024-09-11T01:57:53.335 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40794.log.gz 2024-09-11T01:57:53.335 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.34789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37391.log 2024-09-11T01:57:53.335 INFO:teuthology.orchestra.run.smithi012.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.37521.log.gz 2024-09-11T01:57:53.335 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50845.log 2024-09-11T01:57:53.335 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53073.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34789.log.gz 2024-09-11T01:57:53.335 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.37391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57123.log 2024-09-11T01:57:53.335 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53073.log.gz 2024-09-11T01:57:53.335 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34654.log 2024-09-11T01:57:53.335 INFO:teuthology.orchestra.run.smithi012.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37391.log.gz 2024-09-11T01:57:53.335 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57123.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50845.log.gz 2024-09-11T01:57:53.335 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50144.log 2024-09-11T01:57:53.335 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.34654.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.57123.log.gz 2024-09-11T01:57:53.336 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50761.log 2024-09-11T01:57:53.336 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50102.log 2024-09-11T01:57:53.336 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50761.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34654.log.gz 2024-09-11T01:57:53.336 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40021.log 2024-09-11T01:57:53.336 INFO:teuthology.orchestra.run.smithi012.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.50144.log.gz 2024-09-11T01:57:53.336 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50677.log 2024-09-11T01:57:53.336 INFO:teuthology.orchestra.run.smithi012.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.50761.log.gz 2024-09-11T01:57:53.336 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50102.log: /var/log/ceph/ceph-client.admin.40021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46542.log 2024-09-11T01:57:53.336 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50677.log: gzip 88.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.50102.log.gz --verbose 2024-09-11T01:57:53.336 INFO:teuthology.orchestra.run.smithi012.stderr: -- /var/log/ceph/ceph-client.admin.40836.log 2024-09-11T01:57:53.336 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58257.log 2024-09-11T01:57:53.336 INFO:teuthology.orchestra.run.smithi012.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50677.log.gz 2024-09-11T01:57:53.337 INFO:teuthology.orchestra.run.smithi012.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.40021.log.gz 2024-09-11T01:57:53.337 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.40836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38511.log 2024-09-11T01:57:53.337 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.46542.log.gz 2024-09-11T01:57:53.337 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42735.log 2024-09-11T01:57:53.337 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.38511.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.40836.log.gz 2024-09-11T01:57:53.337 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44101.log 2024-09-11T01:57:53.337 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58257.log.gz 2024-09-11T01:57:53.337 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.42735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47004.log 2024-09-11T01:57:53.337 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38511.log.gz 2024-09-11T01:57:53.337 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46098.log 2024-09-11T01:57:53.337 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42735.log.gz 2024-09-11T01:57:53.337 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44482.log 2024-09-11T01:57:53.337 INFO:teuthology.orchestra.run.smithi012.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.44101.log.gz 2024-09-11T01:57:53.338 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56811.log 2024-09-11T01:57:53.338 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47004.log.gz 2024-09-11T01:57:53.338 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37731.log 2024-09-11T01:57:53.338 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44482.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46098.log.gz 2024-09-11T01:57:53.338 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34629.log 2024-09-11T01:57:53.338 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44482.log.gz 2024-09-11T01:57:53.338 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.37731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54717.log 2024-09-11T01:57:53.338 INFO:teuthology.orchestra.run.smithi012.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56811.log.gz 2024-09-11T01:57:53.338 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43891.log 2024-09-11T01:57:53.339 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.34629.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37731.log.gz 2024-09-11T01:57:53.339 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57627.log 2024-09-11T01:57:53.339 INFO:teuthology.orchestra.run.smithi012.stderr: 88.9%/var/log/ceph/ceph-client.admin.43891.log: -- replaced with /var/log/ceph/ceph-client.admin.34629.log.gz 2024-09-11T01:57:53.339 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45028.log 2024-09-11T01:57:53.339 INFO:teuthology.orchestra.run.smithi012.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.54717.log.gz 2024-09-11T01:57:53.339 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41046.log 2024-09-11T01:57:53.339 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43891.log.gz 2024-09-11T01:57:53.339 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49496.log 2024-09-11T01:57:53.339 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 91.4% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.57627.log.gz 2024-09-11T01:57:53.339 INFO:teuthology.orchestra.run.smithi012.stderr: /var/log/ceph/ceph-client.admin.47507.log 2024-09-11T01:57:53.340 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.41046.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45028.log.gz 2024-09-11T01:57:53.340 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39811.log 2024-09-11T01:57:53.340 INFO:teuthology.orchestra.run.smithi012.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.41046.log.gz 2024-09-11T01:57:53.340 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47507.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49496.log.gz 2024-09-11T01:57:53.340 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52736.log 2024-09-11T01:57:53.340 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.39811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55868.log 2024-09-11T01:57:53.341 INFO:teuthology.orchestra.run.smithi012.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.47507.log.gz 2024-09-11T01:57:53.341 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52736.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39811.log.gz 2024-09-11T01:57:53.341 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38301.log 2024-09-11T01:57:53.341 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37965.log 2024-09-11T01:57:53.341 INFO:teuthology.orchestra.run.smithi012.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52736.log.gz 2024-09-11T01:57:53.341 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.38301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54927.log 2024-09-11T01:57:53.342 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55868.log.gz 2024-09-11T01:57:53.342 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.37965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36534.log 2024-09-11T01:57:53.342 INFO:teuthology.orchestra.run.smithi012.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38301.log.gz 2024-09-11T01:57:53.342 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48552.log 2024-09-11T01:57:53.343 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37965.log.gz 2024-09-11T01:57:53.343 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.36534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38259.log 2024-09-11T01:57:53.343 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54927.log.gz 2024-09-11T01:57:53.343 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55448.log 2024-09-11T01:57:53.343 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.36534.log.gz 2024-09-11T01:57:53.343 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.38259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34831.log 2024-09-11T01:57:53.344 INFO:teuthology.orchestra.run.smithi012.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48552.log.gz 2024-09-11T01:57:53.344 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46794.log 2024-09-11T01:57:53.344 INFO:teuthology.orchestra.run.smithi012.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.38259.log.gz 2024-09-11T01:57:53.344 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.34831.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-11T01:57:53.344 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46794.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.55448.log.gz 2024-09-11T01:57:53.344 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37838.log 2024-09-11T01:57:53.344 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34831.log.gz 2024-09-11T01:57:53.345 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 88.5% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.46794.log.gz 2024-09-11T01:57:53.345 INFO:teuthology.orchestra.run.smithi012.stderr: /var/log/ceph/ceph-client.admin.46920.log 2024-09-11T01:57:53.345 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.37838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47465.log 2024-09-11T01:57:53.345 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43325.log 2024-09-11T01:57:53.346 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37838.log.gz 2024-09-11T01:57:53.346 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42398.log 2024-09-11T01:57:53.346 INFO:teuthology.orchestra.run.smithi012.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.46920.log.gz 2024-09-11T01:57:53.346 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49538.log 2024-09-11T01:57:53.346 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47465.log.gz 2024-09-11T01:57:53.346 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.42398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36912.log 2024-09-11T01:57:53.347 INFO:teuthology.orchestra.run.smithi012.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.43325.log.gz 2024-09-11T01:57:53.347 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47423.log 2024-09-11T01:57:53.347 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55347.log 2024-09-11T01:57:53.347 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5%/var/log/ceph/ceph-client.admin.36912.log: -- replaced with /var/log/ceph/ceph-client.admin.49538.log.gz 2024-09-11T01:57:53.347 INFO:teuthology.orchestra.run.smithi012.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.42398.log.gz 2024-09-11T01:57:53.347 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49580.log 2024-09-11T01:57:53.348 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36912.log.gz 2024-09-11T01:57:53.348 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50719.log 2024-09-11T01:57:53.348 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47423.log.gz 2024-09-11T01:57:53.348 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44902.log 2024-09-11T01:57:53.348 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55347.log.gz 2024-09-11T01:57:53.348 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32112.log 2024-09-11T01:57:53.349 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49580.log.gz 2024-09-11T01:57:53.349 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58633.log 2024-09-11T01:57:53.349 INFO:teuthology.orchestra.run.smithi012.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.50719.log.gz 2024-09-11T01:57:53.349 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.32112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46836.log 2024-09-11T01:57:53.349 INFO:teuthology.orchestra.run.smithi012.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.44902.log.gz 2024-09-11T01:57:53.350 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58633.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32112.log.gz 2024-09-11T01:57:53.350 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44986.log 2024-09-11T01:57:53.350 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58633.log.gz 2024-09-11T01:57:53.350 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37647.log 2024-09-11T01:57:53.350 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46836.log: /var/log/ceph/ceph-client.admin.44986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48763.log 2024-09-11T01:57:53.351 INFO:teuthology.orchestra.run.smithi012.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46836.log.gz 2024-09-11T01:57:53.351 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.37647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40458.log 2024-09-11T01:57:53.351 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44986.log.gz 2024-09-11T01:57:53.351 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52610.log 2024-09-11T01:57:53.351 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48763.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37647.log.gz 2024-09-11T01:57:53.351 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.40458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38343.log 2024-09-11T01:57:53.352 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48763.log.gz 2024-09-11T01:57:53.352 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57795.log 2024-09-11T01:57:53.352 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40458.log.gz 2024-09-11T01:57:53.352 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56601.log 2024-09-11T01:57:53.353 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.38343.log: /var/log/ceph/ceph-client.admin.57795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47065.log 2024-09-11T01:57:53.353 INFO:teuthology.orchestra.run.smithi012.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.52610.log.gz 2024-09-11T01:57:53.353 INFO:teuthology.orchestra.run.smithi012.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.38343.log.gz 2024-09-11T01:57:53.353 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41340.log 2024-09-11T01:57:53.353 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57795.log.gz 2024-09-11T01:57:53.354 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34747.log 2024-09-11T01:57:53.354 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47065.log: /var/log/ceph/ceph-client.admin.41340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35553.log 2024-09-11T01:57:53.354 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56601.log.gz 2024-09-11T01:57:53.354 INFO:teuthology.orchestra.run.smithi012.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47065.log.gz 2024-09-11T01:57:53.354 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.34747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52652.log 2024-09-11T01:57:53.354 INFO:teuthology.orchestra.run.smithi012.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.41340.log.gz 2024-09-11T01:57:53.355 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41751.log 2024-09-11T01:57:53.355 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.35553.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34747.log.gz 2024-09-11T01:57:53.355 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46626.log 2024-09-11T01:57:53.355 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35553.log.gz 2024-09-11T01:57:53.355 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.41751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31809.log 2024-09-11T01:57:53.355 INFO:teuthology.orchestra.run.smithi012.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.52652.log.gz 2024-09-11T01:57:53.355 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54033.log 2024-09-11T01:57:53.356 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46626.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41751.log.gz 2024-09-11T01:57:53.356 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35010.log 2024-09-11T01:57:53.356 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.31809.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46626.log.gz 2024-09-11T01:57:53.356 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46056.log 2024-09-11T01:57:53.356 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56517.log 2024-09-11T01:57:53.357 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.35010.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54033.log.gz 2024-09-11T01:57:53.357 INFO:teuthology.orchestra.run.smithi012.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.31809.log.gz 2024-09-11T01:57:53.357 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55952.log 2024-09-11T01:57:53.357 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35010.log.gz 2024-09-11T01:57:53.357 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39769.log 2024-09-11T01:57:53.357 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56517.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46056.log.gz 2024-09-11T01:57:53.357 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50018.log 2024-09-11T01:57:53.358 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55952.log: /var/log/ceph/ceph-client.admin.39769.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.56517.log.gz 2024-09-11T01:57:53.358 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56643.log 2024-09-11T01:57:53.358 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55263.log 2024-09-11T01:57:53.358 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50018.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.55952.log.gz 2024-09-11T01:57:53.358 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.39769.log.gz 2024-09-11T01:57:53.358 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43723.log 2024-09-11T01:57:53.358 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.50018.log.gz 2024-09-11T01:57:53.358 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43157.log 2024-09-11T01:57:53.358 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55263.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56643.log.gz 2024-09-11T01:57:53.358 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45280.log 2024-09-11T01:57:53.358 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40105.log 2024-09-11T01:57:53.358 INFO:teuthology.orchestra.run.smithi012.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.55263.log.gz 2024-09-11T01:57:53.359 INFO:teuthology.orchestra.run.smithi012.stderr: 89.8%/var/log/ceph/ceph-client.admin.45280.log: -- replaced with /var/log/ceph/ceph-client.admin.43723.log.gz 2024-09-11T01:57:53.359 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39206.log 2024-09-11T01:57:53.359 INFO:teuthology.orchestra.run.smithi012.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.43157.log.gz 2024-09-11T01:57:53.359 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.40105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45449.log 2024-09-11T01:57:53.359 INFO:teuthology.orchestra.run.smithi012.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45280.log.gz 2024-09-11T01:57:53.359 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.39206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49328.log 2024-09-11T01:57:53.359 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40105.log.gz 2024-09-11T01:57:53.359 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52862.log 2024-09-11T01:57:53.359 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39206.log.gz 2024-09-11T01:57:53.359 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48323.log 2024-09-11T01:57:53.359 INFO:teuthology.orchestra.run.smithi012.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.45449.log.gz 2024-09-11T01:57:53.359 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46224.log 2024-09-11T01:57:53.359 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49328.log.gz 2024-09-11T01:57:53.360 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52046.log 2024-09-11T01:57:53.360 INFO:teuthology.orchestra.run.smithi012.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.52862.log.gz 2024-09-11T01:57:53.360 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57963.log 2024-09-11T01:57:53.360 INFO:teuthology.orchestra.run.smithi012.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.48323.log.gz 2024-09-11T01:57:53.360 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36954.log 2024-09-11T01:57:53.360 INFO:teuthology.orchestra.run.smithi012.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46224.log.gz 2024-09-11T01:57:53.360 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52130.log 2024-09-11T01:57:53.360 INFO:teuthology.orchestra.run.smithi012.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.52046.log.gz 2024-09-11T01:57:53.360 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.36954.log: gzip -5 87.9% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.57963.log.gz 2024-09-11T01:57:53.360 INFO:teuthology.orchestra.run.smithi012.stderr: /var/log/ceph/ceph-client.admin.55910.log 2024-09-11T01:57:53.360 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57543.log 2024-09-11T01:57:53.360 INFO:teuthology.orchestra.run.smithi012.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.36954.log.gz 2024-09-11T01:57:53.361 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39937.log 2024-09-11T01:57:53.361 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52130.log.gz 2024-09-11T01:57:53.361 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42525.log 2024-09-11T01:57:53.361 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57543.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55910.log.gz 2024-09-11T01:57:53.361 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.39937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44566.log 2024-09-11T01:57:53.361 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57543.log.gz 2024-09-11T01:57:53.361 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.42525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56872.log 2024-09-11T01:57:53.361 INFO:teuthology.orchestra.run.smithi012.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.39937.log.gz 2024-09-11T01:57:53.361 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38091.log 2024-09-11T01:57:53.361 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42525.log.gz 2024-09-11T01:57:53.361 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35797.log 2024-09-11T01:57:53.361 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56872.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.44566.log.gz 2024-09-11T01:57:53.361 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.38091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42045.log 2024-09-11T01:57:53.362 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56872.log.gz 2024-09-11T01:57:53.362 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.35797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53888.log 2024-09-11T01:57:53.362 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38091.log.gz 2024-09-11T01:57:53.362 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.42045.log: gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35797.log.gz -5 2024-09-11T01:57:53.362 INFO:teuthology.orchestra.run.smithi012.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42190.log 2024-09-11T01:57:53.362 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42482.log 2024-09-11T01:57:53.362 INFO:teuthology.orchestra.run.smithi012.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.42045.log.gz 2024-09-11T01:57:53.362 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.42190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42003.log 2024-09-11T01:57:53.362 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53888.log.gz 2024-09-11T01:57:53.362 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44734.log 2024-09-11T01:57:53.362 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.42482.log: gzip/var/log/ceph/ceph-client.admin.42003.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45154.log 2024-09-11T01:57:53.362 INFO:teuthology.orchestra.run.smithi012.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.42190.log.gz 2024-09-11T01:57:53.362 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42482.log.gz 2024-09-11T01:57:53.362 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38828.log 2024-09-11T01:57:53.363 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42003.log.gz 2024-09-11T01:57:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56099.log 2024-09-11T01:57:53.363 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44734.log.gz 2024-09-11T01:57:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.38828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35015.log 2024-09-11T01:57:53.363 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45154.log.gz 2024-09-11T01:57:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54969.log 2024-09-11T01:57:53.363 INFO:teuthology.orchestra.run.smithi012.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.38828.log.gz 2024-09-11T01:57:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.35015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46668.log 2024-09-11T01:57:53.363 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56099.log.gz 2024-09-11T01:57:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49204.log 2024-09-11T01:57:53.363 INFO:teuthology.orchestra.run.smithi012.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35015.log.gz 2024-09-11T01:57:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46668.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.54969.log.gz 2024-09-11T01:57:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46014.log 2024-09-11T01:57:53.363 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58467.log 2024-09-11T01:57:53.364 INFO:teuthology.orchestra.run.smithi012.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.46668.log.gz 2024-09-11T01:57:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48239.log 2024-09-11T01:57:53.364 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49204.log.gz 2024-09-11T01:57:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52172.log 2024-09-11T01:57:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58467.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46014.log.gz 2024-09-11T01:57:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53257.log 2024-09-11T01:57:53.364 INFO:teuthology.orchestra.run.smithi012.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58467.log.gz 2024-09-11T01:57:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38744.log 2024-09-11T01:57:53.364 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48239.log.gz 2024-09-11T01:57:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40500.log 2024-09-11T01:57:53.364 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52172.log.gz 2024-09-11T01:57:53.364 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.38744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54245.log 2024-09-11T01:57:53.364 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53257.log.gz 2024-09-11T01:57:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.40500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51284.log 2024-09-11T01:57:53.365 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38744.log.gz 2024-09-11T01:57:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55826.log 2024-09-11T01:57:53.365 INFO:teuthology.orchestra.run.smithi012.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.40500.log.gz 2024-09-11T01:57:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36618.log 2024-09-11T01:57:53.365 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.54245.log.gz 2024-09-11T01:57:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45888.log 2024-09-11T01:57:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55826.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.51284.log.gz 2024-09-11T01:57:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.36618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52568.log 2024-09-11T01:57:53.365 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55826.log.gz 2024-09-11T01:57:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44524.log 2024-09-11T01:57:53.365 INFO:teuthology.orchestra.run.smithi012.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36618.log.gz 2024-09-11T01:57:53.365 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39517.log 2024-09-11T01:57:53.366 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45888.log.gz 2024-09-11T01:57:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42777.log 2024-09-11T01:57:53.366 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.52568.log.gz 2024-09-11T01:57:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.39517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55658.log 2024-09-11T01:57:53.366 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44524.log.gz 2024-09-11T01:57:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.42777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41961.log 2024-09-11T01:57:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52214.log 2024-09-11T01:57:53.366 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42777.log.gz 2024-09-11T01:57:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55658.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.39517.log.gz 2024-09-11T01:57:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.41961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56265.log 2024-09-11T01:57:53.366 INFO:teuthology.orchestra.run.smithi012.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55658.log.gz 2024-09-11T01:57:53.366 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57669.log 2024-09-11T01:57:53.366 INFO:teuthology.orchestra.run.smithi012.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41961.log.gz 2024-09-11T01:57:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48197.log 2024-09-11T01:57:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56265.log: /var/log/ceph/ceph-client.admin.57669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45364.log 2024-09-11T01:57:53.367 INFO:teuthology.orchestra.run.smithi012.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.52214.log.gz 2024-09-11T01:57:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48197.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.56265.log.gz 2024-09-11T01:57:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36576.log 2024-09-11T01:57:53.367 INFO:teuthology.orchestra.run.smithi012.stderr: 89.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44229.log 2024-09-11T01:57:53.367 INFO:teuthology.orchestra.run.smithi012.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48197.log.gz 2024-09-11T01:57:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45364.log: -- replaced with /var/log/ceph/ceph-client.admin.57669.log.gz 2024-09-11T01:57:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.36576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58131.log 2024-09-11T01:57:53.367 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39290.log 2024-09-11T01:57:53.367 INFO:teuthology.orchestra.run.smithi012.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.45364.log.gz 2024-09-11T01:57:53.367 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.36576.log.gz 2024-09-11T01:57:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44059.log 2024-09-11T01:57:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58131.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44229.log.gz 2024-09-11T01:57:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:gzip/var/log/ceph/ceph-client.admin.39290.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51705.log 2024-09-11T01:57:53.368 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58131.log.gz 2024-09-11T01:57:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35839.log 2024-09-11T01:57:53.368 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39290.log.gz 2024-09-11T01:57:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40248.log 2024-09-11T01:57:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51705.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44059.log.gz 2024-09-11T01:57:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.35839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36346.log 2024-09-11T01:57:53.368 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51705.log.gz 2024-09-11T01:57:53.368 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.40248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41004.log 2024-09-11T01:57:53.368 INFO:teuthology.orchestra.run.smithi012.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35839.log.gz 2024-09-11T01:57:53.369 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.36346.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-11T01:57:53.369 INFO:teuthology.orchestra.run.smithi012.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40248.log.gz 2024-09-11T01:57:53.369 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.41004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48931.log 2024-09-11T01:57:53.369 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36346.log.gz 2024-09-11T01:57:53.369 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35265.log 2024-09-11T01:57:53.369 INFO:teuthology.orchestra.run.smithi012.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.41004.log.gz 2024-09-11T01:57:53.369 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55616.log 2024-09-11T01:57:53.369 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.35265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55053.log 2024-09-11T01:57:53.369 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48931.log.gz 2024-09-11T01:57:53.369 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55616.log: gzip 88.5% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.35265.log.gz -- 2024-09-11T01:57:53.369 INFO:teuthology.orchestra.run.smithi012.stderr: /var/log/ceph/ceph-client.admin.38912.log 2024-09-11T01:57:53.369 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35713.log 2024-09-11T01:57:53.370 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55053.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55616.log.gz 2024-09-11T01:57:53.370 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.38912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53720.log 2024-09-11T01:57:53.370 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55053.log.gz 2024-09-11T01:57:53.370 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.35713.log: gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38912.log.gz 2024-09-11T01:57:53.370 INFO:teuthology.orchestra.run.smithi012.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37605.log 2024-09-11T01:57:53.370 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35267.log 2024-09-11T01:57:53.370 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35713.log.gz 2024-09-11T01:57:53.370 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.37605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39080.log 2024-09-11T01:57:53.370 INFO:teuthology.orchestra.run.smithi012.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53720.log.gz 2024-09-11T01:57:53.370 INFO:teuthology.orchestra.run.smithi012.stderr: 94.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-11T01:57:53.370 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.35267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58383.log 2024-09-11T01:57:53.371 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37605.log.gz 2024-09-11T01:57:53.371 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.39080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49099.log 2024-09-11T01:57:53.371 INFO:teuthology.orchestra.run.smithi012.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35267.log.gz 2024-09-11T01:57:53.371 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37923.log 2024-09-11T01:57:53.371 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39080.log.gz 2024-09-11T01:57:53.371 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40332.log 2024-09-11T01:57:53.371 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58383.log.gz 2024-09-11T01:57:53.371 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.37923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38385.log 2024-09-11T01:57:53.371 INFO:teuthology.orchestra.run.smithi012.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.49099.log.gz 2024-09-11T01:57:53.371 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.40332.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37923.log.gz 2024-09-11T01:57:53.371 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53510.log 2024-09-11T01:57:53.371 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.38385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47297.log 2024-09-11T01:57:53.371 INFO:teuthology.orchestra.run.smithi012.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.40332.log.gz 2024-09-11T01:57:53.372 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49057.log 2024-09-11T01:57:53.372 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38385.log.gz 2024-09-11T01:57:53.372 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50593.log 2024-09-11T01:57:53.372 INFO:teuthology.orchestra.run.smithi012.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.53510.log.gz 2024-09-11T01:57:53.372 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41539.log 2024-09-11T01:57:53.372 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47297.log.gz 2024-09-11T01:57:53.372 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45407.log 2024-09-11T01:57:53.372 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50593.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.49057.log.gz 2024-09-11T01:57:53.372 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.41539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55095.log 2024-09-11T01:57:53.372 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50635.log 2024-09-11T01:57:53.372 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50593.log.gz 2024-09-11T01:57:53.372 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45407.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41539.log.gz 2024-09-11T01:57:53.372 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50060.log 2024-09-11T01:57:53.373 INFO:teuthology.orchestra.run.smithi012.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.45407.log.gz 2024-09-11T01:57:53.373 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52004.log 2024-09-11T01:57:53.373 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55095.log.gz 2024-09-11T01:57:53.373 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50060.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50635.log.gz 2024-09-11T01:57:53.373 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57921.log 2024-09-11T01:57:53.373 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49664.log 2024-09-11T01:57:53.373 INFO:teuthology.orchestra.run.smithi012.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50060.log.gz 2024-09-11T01:57:53.373 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58593.log 2024-09-11T01:57:53.373 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49664.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.52004.log.gz 2024-09-11T01:57:53.373 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45762.log 2024-09-11T01:57:53.373 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57921.log.gz 2024-09-11T01:57:53.374 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44650.log 2024-09-11T01:57:53.374 INFO:teuthology.orchestra.run.smithi012.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.49664.log.gz 2024-09-11T01:57:53.374 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55994.log 2024-09-11T01:57:53.374 INFO:teuthology.orchestra.run.smithi012.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58593.log.gz 2024-09-11T01:57:53.374 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36702.log 2024-09-11T01:57:53.374 INFO:teuthology.orchestra.run.smithi012.stderr: 89.8%/var/log/ceph/ceph-client.admin.55994.log: -- replaced with /var/log/ceph/ceph-client.admin.45762.log.gz 2024-09-11T01:57:53.374 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41581.log 2024-09-11T01:57:53.374 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44650.log.gz 2024-09-11T01:57:53.374 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.36702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40878.log 2024-09-11T01:57:53.374 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.55994.log.gz 2024-09-11T01:57:53.374 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.41581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52778.log 2024-09-11T01:57:53.374 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36702.log.gz 2024-09-11T01:57:53.374 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.40878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40063.log 2024-09-11T01:57:53.375 INFO:teuthology.orchestra.run.smithi012.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41581.log.gz 2024-09-11T01:57:53.375 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58089.log 2024-09-11T01:57:53.375 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58215.log 2024-09-11T01:57:53.375 INFO:teuthology.orchestra.run.smithi012.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.40878.log.gz 2024-09-11T01:57:53.375 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.40063.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.52778.log.gz 2024-09-11T01:57:53.375 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55742.log 2024-09-11T01:57:53.375 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40063.log.gz 2024-09-11T01:57:53.375 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36744.log 2024-09-11T01:57:53.375 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58215.log: /var/log/ceph/ceph-client.admin.55742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48721.log 2024-09-11T01:57:53.375 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58089.log.gz 2024-09-11T01:57:53.375 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58215.log.gz 2024-09-11T01:57:53.375 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.36744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55532.log 2024-09-11T01:57:53.376 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.55742.log.gz 2024-09-11T01:57:53.376 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57081.log 2024-09-11T01:57:53.376 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48721.log: /var/log/ceph/ceph-client.admin.55532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39895.log 2024-09-11T01:57:53.376 INFO:teuthology.orchestra.run.smithi012.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.36744.log.gz 2024-09-11T01:57:53.376 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48721.log.gz 2024-09-11T01:57:53.376 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41256.log 2024-09-11T01:57:53.376 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55137.log 2024-09-11T01:57:53.376 INFO:teuthology.orchestra.run.smithi012.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.55532.log.gz 2024-09-11T01:57:53.376 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.39895.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.57081.log.gz 2024-09-11T01:57:53.376 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.41256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56475.log 2024-09-11T01:57:53.376 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39895.log.gz 2024-09-11T01:57:53.377 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57249.log 2024-09-11T01:57:53.377 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42903.log 2024-09-11T01:57:53.377 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56475.log: 89.4% 90.6% -- replaced with /var/log/ceph/ceph-client.admin.41256.log.gz 2024-09-11T01:57:53.377 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55137.log.gz 2024-09-11T01:57:53.377 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57585.log 2024-09-11T01:57:53.377 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.56475.log.gz 2024-09-11T01:57:53.377 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47818.log 2024-09-11T01:57:53.377 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.42903.log: /var/log/ceph/ceph-client.admin.57585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43765.log 2024-09-11T01:57:53.377 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.57249.log.gz 2024-09-11T01:57:53.377 INFO:teuthology.orchestra.run.smithi012.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.42903.log.gz 2024-09-11T01:57:53.377 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54287.log 2024-09-11T01:57:53.378 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57585.log.gz 2024-09-11T01:57:53.378 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40626.log 2024-09-11T01:57:53.378 INFO:teuthology.orchestra.run.smithi012.stderr:gzip/var/log/ceph/ceph-client.admin.54287.log: -5 89.6% -- replaced with /var/log/ceph/ceph-client.admin.47818.log.gz 2024-09-11T01:57:53.378 INFO:teuthology.orchestra.run.smithi012.stderr: --verbose -- /var/log/ceph/ceph-client.admin.35264.log 2024-09-11T01:57:53.378 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43765.log.gz 2024-09-11T01:57:53.378 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.40626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37475.log 89.4% 2024-09-11T01:57:53.378 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54287.log.gz 2024-09-11T01:57:53.378 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.35264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39164.log 2024-09-11T01:57:53.378 INFO:teuthology.orchestra.run.smithi012.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.40626.log.gz 2024-09-11T01:57:53.378 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43639.log 2024-09-11T01:57:53.378 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.37475.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35264.log.gz 2024-09-11T01:57:53.378 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52256.log 2024-09-11T01:57:53.379 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.39164.log: /var/log/ceph/ceph-client.admin.43639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36026.log 2024-09-11T01:57:53.379 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37475.log.gz 2024-09-11T01:57:53.379 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39164.log.gz 2024-09-11T01:57:53.379 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57039.log 2024-09-11T01:57:53.379 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43639.log.gz 2024-09-11T01:57:53.379 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35559.log 2024-09-11T01:57:53.379 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.36026.log: /var/log/ceph/ceph-client.admin.57039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35513.log 2024-09-11T01:57:53.379 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.52256.log.gz 2024-09-11T01:57:53.379 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36026.log.gz 2024-09-11T01:57:53.379 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.35559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46415.log 2024-09-11T01:57:53.379 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.57039.log.gz 2024-09-11T01:57:53.379 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54161.log 2024-09-11T01:57:53.379 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.35513.log: /var/log/ceph/ceph-client.admin.46415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32038.log 2024-09-11T01:57:53.380 INFO:teuthology.orchestra.run.smithi012.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35559.log.gz 2024-09-11T01:57:53.380 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35513.log.gz 2024-09-11T01:57:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49748.log 2024-09-11T01:57:53.380 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46415.log.gz 2024-09-11T01:57:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45112.log 2024-09-11T01:57:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.32038.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54161.log.gz 2024-09-11T01:57:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36304.log 2024-09-11T01:57:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49748.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32038.log.gz 2024-09-11T01:57:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41919.log 2024-09-11T01:57:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36176.log 2024-09-11T01:57:53.380 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49748.log.gz 2024-09-11T01:57:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.36304.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45112.log.gz 2024-09-11T01:57:53.380 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.41919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39038.log 2024-09-11T01:57:53.381 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36304.log.gz 2024-09-11T01:57:53.381 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54675.log 2024-09-11T01:57:53.381 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.36176.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41919.log.gz 2024-09-11T01:57:53.381 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.39038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48595.log 2024-09-11T01:57:53.381 INFO:teuthology.orchestra.run.smithi012.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.36176.log.gz 2024-09-11T01:57:53.381 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54675.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-11T01:57:53.381 INFO:teuthology.orchestra.run.smithi012.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.39038.log.gz 2024-09-11T01:57:53.381 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44944.log 2024-09-11T01:57:53.381 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47986.log 2024-09-11T01:57:53.381 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-mon.c.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.54675.log.gz 2024-09-11T01:57:53.381 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44944.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.48595.log.gz 2024-09-11T01:57:53.381 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47902.log 2024-09-11T01:57:53.381 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45636.log 2024-09-11T01:57:53.382 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47986.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.44944.log.gz 2024-09-11T01:57:53.382 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45972.log 2024-09-11T01:57:53.382 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45636.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47986.log.gz 2024-09-11T01:57:53.382 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54801.log 2024-09-11T01:57:53.382 INFO:teuthology.orchestra.run.smithi012.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.47902.log.gz 2024-09-11T01:57:53.382 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45636.log.gz 2024-09-11T01:57:53.382 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38469.log 2024-09-11T01:57:53.382 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44017.log 2024-09-11T01:57:53.382 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54801.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.45972.log.gz 2024-09-11T01:57:53.382 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.38469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56769.log 2024-09-11T01:57:53.382 INFO:teuthology.orchestra.run.smithi012.stderr:gzip 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54801.log.gz 2024-09-11T01:57:53.382 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44017.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40962.log 2024-09-11T01:57:53.382 INFO:teuthology.orchestra.run.smithi012.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.38469.log.gz 2024-09-11T01:57:53.382 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56769.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44017.log.gz 2024-09-11T01:57:53.383 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53384.log 2024-09-11T01:57:53.383 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56685.log 2024-09-11T01:57:53.383 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56769.log.gz 2024-09-11T01:57:53.383 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.40962.log: /var/log/ceph/ceph-client.admin.53384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50312.log 2024-09-11T01:57:53.383 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56685.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40962.log.gz 2024-09-11T01:57:53.383 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42271.log 2024-09-11T01:57:53.383 INFO:teuthology.orchestra.run.smithi012.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.53384.log.gz 2024-09-11T01:57:53.383 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50312.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56685.log.gz 2024-09-11T01:57:53.383 INFO:teuthology.orchestra.run.smithi012.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52340.log 2024-09-11T01:57:53.383 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.42271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39474.log 2024-09-11T01:57:53.383 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.50312.log.gz 2024-09-11T01:57:53.383 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45804.log 2024-09-11T01:57:53.383 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42271.log.gz 2024-09-11T01:57:53.384 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.39474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46878.log 2024-09-11T01:57:53.384 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52340.log.gz 2024-09-11T01:57:53.384 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45804.log: gzip -5 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39474.log.gz 2024-09-11T01:57:53.384 INFO:teuthology.orchestra.run.smithi012.stderr: --verbose -- /var/log/ceph/ceph-client.admin.56727.log 2024-09-11T01:57:53.384 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46878.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45804.log.gz 2024-09-11T01:57:53.384 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53678.log 2024-09-11T01:57:53.384 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46878.log.gz 2024-09-11T01:57:53.384 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46286.log 2024-09-11T01:57:53.384 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56727.log: gzip/var/log/ceph/ceph-client.admin.53678.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53804.log 2024-09-11T01:57:53.384 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39853.log 2024-09-11T01:57:53.384 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% 90.4% -- replaced with /var/log/ceph/ceph-client.admin.56727.log.gz 2024-09-11T01:57:53.384 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53678.log.gz 2024-09-11T01:57:53.385 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53804.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46286.log.gz 2024-09-11T01:57:53.385 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35006.log 2024-09-11T01:57:53.385 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.39853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44314.log 2024-09-11T01:57:53.385 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.35006.log: gzip -5 --verbose -- 90.9% /var/log/ceph/ceph-client.admin.47944.log 2024-09-11T01:57:53.385 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53804.log.gz 2024-09-11T01:57:53.385 INFO:teuthology.orchestra.run.smithi012.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39853.log.gz 2024-09-11T01:57:53.385 INFO:teuthology.orchestra.run.smithi012.stderr:gzip/var/log/ceph/ceph-client.admin.44314.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.56349.log 2024-09-11T01:57:53.385 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35006.log.gz/var/log/ceph/ceph-client.admin.47944.log: 2024-09-11T01:57:53.385 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45238.log 2024-09-11T01:57:53.385 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44314.log.gz 2024-09-11T01:57:53.385 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56349.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47944.log.gz 2024-09-11T01:57:53.385 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55179.log 2024-09-11T01:57:53.385 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40416.log 2024-09-11T01:57:53.386 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56349.log.gz 2024-09-11T01:57:53.386 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55179.log: gzip -5 --verbose -- 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45238.log.gz 2024-09-11T01:57:53.386 INFO:teuthology.orchestra.run.smithi012.stderr: /var/log/ceph/ceph-client.admin.50509.log 2024-09-11T01:57:53.386 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.40416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49412.log 2024-09-11T01:57:53.386 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.55179.log.gz 2024-09-11T01:57:53.386 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38007.log 2024-09-11T01:57:53.386 INFO:teuthology.orchestra.run.smithi012.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40416.log.gz 2024-09-11T01:57:53.386 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46182.log 2024-09-11T01:57:53.386 INFO:teuthology.orchestra.run.smithi012.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.50509.log.gz 2024-09-11T01:57:53.386 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.38007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46752.log 2024-09-11T01:57:53.386 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49412.log.gz 2024-09-11T01:57:53.386 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36660.log 2024-09-11T01:57:53.386 INFO:teuthology.orchestra.run.smithi012.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.38007.log.gz 2024-09-11T01:57:53.386 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35923.log 2024-09-11T01:57:53.387 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.36660.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.46182.log.gz 2024-09-11T01:57:53.387 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49790.log 2024-09-11T01:57:53.387 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.46752.log.gz 2024-09-11T01:57:53.387 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.35923.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36660.log.gz 2024-09-11T01:57:53.387 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48029.log 2024-09-11T01:57:53.387 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35923.log.gz 2024-09-11T01:57:53.387 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56307.log 2024-09-11T01:57:53.387 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43807.log 2024-09-11T01:57:53.387 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49790.log.gz 2024-09-11T01:57:53.387 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38659.log 2024-09-11T01:57:53.387 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48029.log.gz 2024-09-11T01:57:53.387 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43807.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.56307.log.gz 2024-09-11T01:57:53.387 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47170.log 2024-09-11T01:57:53.387 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5/var/log/ceph/ceph-client.admin.38659.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43807.log.gz 2024-09-11T01:57:53.388 INFO:teuthology.orchestra.run.smithi012.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39248.log 2024-09-11T01:57:53.388 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47170.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38659.log.gz 2024-09-11T01:57:53.388 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57291.log 2024-09-11T01:57:53.388 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45720.log 2024-09-11T01:57:53.388 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.39248.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47170.log.gz 2024-09-11T01:57:53.388 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44692.log 2024-09-11T01:57:53.388 INFO:teuthology.orchestra.run.smithi012.stderr:gzip/var/log/ceph/ceph-client.admin.45720.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.54423.log 2024-09-11T01:57:53.388 INFO:teuthology.orchestra.run.smithi012.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.39248.log.gz 2024-09-11T01:57:53.388 INFO:teuthology.orchestra.run.smithi012.stderr: 90.0%/var/log/ceph/ceph-client.admin.44692.log: -- replaced with /var/log/ceph/ceph-client.admin.57291.log.gz 2024-09-11T01:57:53.388 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46584.log 2024-09-11T01:57:53.388 INFO:teuthology.orchestra.run.smithi012.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.45720.log.gz 2024-09-11T01:57:53.388 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51579.log 2024-09-11T01:57:53.388 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46584.log: 91.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38133.log 2024-09-11T01:57:53.388 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44692.log.gz 2024-09-11T01:57:53.389 INFO:teuthology.orchestra.run.smithi012.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.54423.log.gz 2024-09-11T01:57:53.389 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48637.log 2024-09-11T01:57:53.389 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46584.log.gz 2024-09-11T01:57:53.389 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5/var/log/ceph/ceph-client.admin.38133.log: --verbose -- /var/log/ceph/ceph-client.admin.48281.log 2024-09-11T01:57:53.389 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48637.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51579.log.gz 2024-09-11T01:57:53.389 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53636.log 2024-09-11T01:57:53.389 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38133.log.gz 2024-09-11T01:57:53.389 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35439.log 2024-09-11T01:57:53.389 INFO:teuthology.orchestra.run.smithi012.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.48637.log.gz 2024-09-11T01:57:53.389 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48281.log.gz 2024-09-11T01:57:53.389 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36870.log 2024-09-11T01:57:53.389 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.35439.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.53636.log.gz 2024-09-11T01:57:53.389 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47675.log 2024-09-11T01:57:53.389 INFO:teuthology.orchestra.run.smithi012.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35439.log.gz 2024-09-11T01:57:53.390 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.36870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58173.log 2024-09-11T01:57:53.390 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36870.log.gz 2024-09-11T01:57:53.390 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47675.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-11T01:57:53.390 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47339.log 2024-09-11T01:57:53.390 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58173.log: /var/log/ceph/ceph-mon.a.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.47675.log.gz 2024-09-11T01:57:53.390 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57459.log 2024-09-11T01:57:53.390 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58173.log.gz 2024-09-11T01:57:53.390 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58341.log 2024-09-11T01:57:53.390 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51199.log 90.6% -- replaced with /var/log/ceph/ceph-client.admin.47339.log.gz 2024-09-11T01:57:53.390 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:53.390 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58341.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.57459.log.gz 2024-09-11T01:57:53.390 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42314.log 2024-09-11T01:57:53.390 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6%/var/log/ceph/ceph-client.admin.51199.log: -- replaced with /var/log/ceph/ceph-client.admin.58341.log.gz 2024-09-11T01:57:53.391 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44185.log 2024-09-11T01:57:53.391 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.42314.log: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43199.log 2024-09-11T01:57:53.391 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51199.log.gz 2024-09-11T01:57:53.391 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36220.log 2024-09-11T01:57:53.391 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42314.log.gz 2024-09-11T01:57:53.391 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.43199.log: -- /var/log/ceph/ceph-client.admin.49892.log 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44185.log.gz 2024-09-11T01:57:53.391 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:53.391 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.36220.log: gzip -5 --verbose 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43199.log.gz 2024-09-11T01:57:53.391 INFO:teuthology.orchestra.run.smithi012.stderr: -- /var/log/ceph/ceph-client.admin.55490.log 2024-09-11T01:57:53.391 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49892.log: gzip 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36220.log.gz 2024-09-11T01:57:53.391 INFO:teuthology.orchestra.run.smithi012.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36092.log 2024-09-11T01:57:53.391 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55490.log: 91.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58299.log 2024-09-11T01:57:53.391 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49892.log.gz 2024-09-11T01:57:53.391 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.36092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51877.log 2024-09-11T01:57:53.392 INFO:teuthology.orchestra.run.smithi012.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.55490.log.gz 2024-09-11T01:57:53.392 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50551.log 2024-09-11T01:57:53.392 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37349.log 2024-09-11T01:57:53.392 INFO:teuthology.orchestra.run.smithi012.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.36092.log.gz 2024-09-11T01:57:53.392 INFO:teuthology.orchestra.run.smithi012.stderr: 87.9%/var/log/ceph/ceph-client.admin.51877.log: -- replaced with /var/log/ceph/ceph-client.admin.58299.log.gz 2024-09-11T01:57:53.392 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37223.log 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51877.log.gz 2024-09-11T01:57:53.392 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:53.392 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.37349.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50551.log.gz 2024-09-11T01:57:53.392 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51013.log 2024-09-11T01:57:53.392 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.37223.log: gzip -5 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37349.log.gz 2024-09-11T01:57:53.392 INFO:teuthology.orchestra.run.smithi012.stderr: --verbose -- /var/log/ceph/ceph-client.admin.57753.log 2024-09-11T01:57:53.392 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51013.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.37223.log.gz 2024-09-11T01:57:53.392 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53952.log 2024-09-11T01:57:53.393 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51013.log.gz 2024-09-11T01:57:53.393 INFO:teuthology.orchestra.run.smithi012.stderr: /var/log/ceph/ceph-client.admin.57837.log/var/log/ceph/ceph-client.admin.57753.log: 2024-09-11T01:57:53.393 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35965.log 2024-09-11T01:57:53.393 INFO:teuthology.orchestra.run.smithi012.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.57753.log.gz/var/log/ceph/ceph-client.admin.57837.log: 2024-09-11T01:57:53.393 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53952.log.gz 2024-09-11T01:57:53.393 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42989.log 2024-09-11T01:57:53.393 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.35965.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51789.log 2024-09-11T01:57:53.393 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57837.log.gz 2024-09-11T01:57:53.393 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.42989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43493.log 2024-09-11T01:57:53.393 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35965.log.gz 2024-09-11T01:57:53.393 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55574.log 2024-09-11T01:57:53.393 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51789.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42989.log.gz 2024-09-11T01:57:53.393 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43933.log 2024-09-11T01:57:53.393 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46710.log 2024-09-11T01:57:53.394 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55574.log: 89.8%/var/log/ceph/ceph-client.admin.43933.log: -- replaced with /var/log/ceph/ceph-client.admin.51789.log.gz 2024-09-11T01:57:53.394 INFO:teuthology.orchestra.run.smithi012.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.43493.log.gz 2024-09-11T01:57:53.394 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35013.log 2024-09-11T01:57:53.394 INFO:teuthology.orchestra.run.smithi012.stderr: 89.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56954.log 2024-09-11T01:57:53.394 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46710.log: -- replaced with /var/log/ceph/ceph-client.admin.43933.log.gz 2024-09-11T01:57:53.394 INFO:teuthology.orchestra.run.smithi012.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55574.log.gz 2024-09-11T01:57:53.394 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.35013.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-11T01:57:53.394 INFO:teuthology.orchestra.run.smithi012.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.46710.log.gz 2024-09-11T01:57:53.394 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39122.log 2024-09-11T01:57:53.394 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35013.log.gz 2024-09-11T01:57:53.394 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56954.log: /var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35266.log 2024-09-11T01:57:53.394 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.39122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51369.log 2024-09-11T01:57:53.394 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.35266.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39122.log.gz 2024-09-11T01:57:53.395 INFO:teuthology.orchestra.run.smithi012.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.56954.log.gz 2024-09-11T01:57:53.395 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35266.log.gz 2024-09-11T01:57:53.395 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57711.log 2024-09-11T01:57:53.395 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49976.log 2024-09-11T01:57:53.395 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57711.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51369.log.gz 2024-09-11T01:57:53.395 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49622.log 2024-09-11T01:57:53.395 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35262.log 2024-09-11T01:57:53.395 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.57711.log.gz 2024-09-11T01:57:53.395 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49976.log: /var/log/ceph/ceph-client.admin.49622.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.49976.log.gz 2024-09-11T01:57:53.395 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.49622.log.gz 2024-09-11T01:57:53.395 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53426.log 2024-09-11T01:57:53.395 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.35262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53031.log 2024-09-11T01:57:53.395 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53426.log: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54759.log 2024-09-11T01:57:53.396 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35262.log.gz 2024-09-11T01:57:53.396 INFO:teuthology.orchestra.run.smithi012.stderr:gzip 90.1% -5 --verbose -- /var/log/ceph/ceph-client.admin.43115.log 2024-09-11T01:57:53.396 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53426.log.gz 2024-09-11T01:57:53.396 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53031.log: /var/log/ceph/ceph-client.admin.54759.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.53031.log.gz 2024-09-11T01:57:53.396 INFO:teuthology.orchestra.run.smithi012.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.54759.log.gz 2024-09-11T01:57:53.396 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51118.log 2024-09-11T01:57:53.396 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43681.log 2024-09-11T01:57:53.396 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42947.log 2024-09-11T01:57:53.396 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.43115.log.gz 2024-09-11T01:57:53.396 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7%/var/log/ceph/ceph-client.admin.43681.log: -- replaced with /var/log/ceph/ceph-client.admin.51118.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46140.log 2024-09-11T01:57:53.396 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:53.396 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.42947.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.43681.log.gz 2024-09-11T01:57:53.396 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42947.log.gz 2024-09-11T01:57:53.397 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41403.log 2024-09-11T01:57:53.397 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51326.log 2024-09-11T01:57:53.397 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.41403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57333.log 2024-09-11T01:57:53.397 INFO:teuthology.orchestra.run.smithi012.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.46140.log.gz 2024-09-11T01:57:53.397 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51326.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41403.log.gz 2024-09-11T01:57:53.397 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58509.log 2024-09-11T01:57:53.397 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41667.log 2024-09-11T01:57:53.397 INFO:teuthology.orchestra.run.smithi012.stderr: 90.9%/var/log/ceph/ceph-client.admin.58509.log: -- replaced with /var/log/ceph/ceph-client.admin.51326.log.gz 2024-09-11T01:57:53.397 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43597.log 2024-09-11T01:57:53.397 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57333.log.gz 2024-09-11T01:57:53.397 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58509.log.gz 2024-09-11T01:57:53.397 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.41667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39332.log 2024-09-11T01:57:53.397 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55305.log 2024-09-11T01:57:53.398 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.39332.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.41667.log.gz 2024-09-11T01:57:53.398 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51962.log 2024-09-11T01:57:53.398 INFO:teuthology.orchestra.run.smithi012.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.43597.log.gz 2024-09-11T01:57:53.398 INFO:teuthology.orchestra.run.smithi012.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.39332.log.gz 2024-09-11T01:57:53.398 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52088.log 2024-09-11T01:57:53.398 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41793.log 2024-09-11T01:57:53.398 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.55305.log.gz 2024-09-11T01:57:53.398 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52088.log: gzip -5 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51962.log.gz 2024-09-11T01:57:53.398 INFO:teuthology.orchestra.run.smithi012.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43975.log 2024-09-11T01:57:53.398 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.41793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51055.log 2024-09-11T01:57:53.398 INFO:teuthology.orchestra.run.smithi012.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52088.log.gz 2024-09-11T01:57:53.398 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43975.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.41793.log.gz 2024-09-11T01:57:53.398 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43975.log.gz 2024-09-11T01:57:53.399 INFO:teuthology.orchestra.run.smithi012.stderr: /var/log/ceph/ceph-client.admin.38996.log 2024-09-11T01:57:53.399 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42356.log 2024-09-11T01:57:53.399 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.38996.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.51055.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38954.log 2024-09-11T01:57:53.399 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:53.399 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose 89.5% -- /var/log/ceph/ceph-client.admin.51831.log 2024-09-11T01:57:53.399 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38996.log.gz 2024-09-11T01:57:53.399 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.42356.log: /var/log/ceph/ceph-client.admin.38954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48471.log 2024-09-11T01:57:53.399 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51831.log: 89.7% 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38954.log.gz 2024-09-11T01:57:53.399 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42356.log.gz 2024-09-11T01:57:53.399 INFO:teuthology.orchestra.run.smithi012.stderr: 89.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47633.log 2024-09-11T01:57:53.399 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51831.log.gz 2024-09-11T01:57:53.399 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35755.log 2024-09-11T01:57:53.399 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35009.log 2024-09-11T01:57:53.399 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48471.log.gz 2024-09-11T01:57:53.400 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.35755.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.47633.log.gz 2024-09-11T01:57:53.400 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35755.log.gz 2024-09-11T01:57:53.400 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50186.log 2024-09-11T01:57:53.400 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.35009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45593.log 2024-09-11T01:57:53.400 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40206.log 2024-09-11T01:57:53.400 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35009.log.gz 2024-09-11T01:57:53.400 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37689.log 2024-09-11T01:57:53.400 INFO:teuthology.orchestra.run.smithi012.stderr: 89.8%/var/log/ceph/ceph-client.admin.45593.log: -- replaced with /var/log/ceph/ceph-client.admin.50186.log.gz 2024-09-11T01:57:53.400 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.40206.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45593.log.gz 2024-09-11T01:57:53.400 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- 91.4% -- replaced with /var/log/ceph/ceph-client.admin.40206.log.gz 2024-09-11T01:57:53.400 INFO:teuthology.orchestra.run.smithi012.stderr: /var/log/ceph/ceph-client.admin.45322.log 2024-09-11T01:57:53.400 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.37689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53468.log 2024-09-11T01:57:53.401 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52904.log 2024-09-11T01:57:53.401 INFO:teuthology.orchestra.run.smithi012.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.37689.log.gz 2024-09-11T01:57:53.401 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53468.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45322.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35419.log 2024-09-11T01:57:53.401 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:53.401 INFO:teuthology.orchestra.run.smithi012.stderr:gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53468.log.gz 2024-09-11T01:57:53.401 INFO:teuthology.orchestra.run.smithi012.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.55784.log 2024-09-11T01:57:53.401 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52904.log: /var/log/ceph/ceph-client.admin.35419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52820.log 2024-09-11T01:57:53.401 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.55784.log: 88.3% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35419.log.gz -- replaced with /var/log/ceph/ceph-client.admin.52904.log.gz 2024-09-11T01:57:53.401 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:53.401 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43367.log 89.8% -- replaced with /var/log/ceph/ceph-client.admin.55784.log.gz 2024-09-11T01:57:53.401 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:53.401 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49934.log 2024-09-11T01:57:53.401 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43367.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.52820.log.gz 2024-09-11T01:57:53.402 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36492.log 2024-09-11T01:57:53.402 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57417.log 2024-09-11T01:57:53.402 INFO:teuthology.orchestra.run.smithi012.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43367.log.gz 2024-09-11T01:57:53.402 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49934.log: /var/log/ceph/ceph-client.admin.36492.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.49934.log.gz 2024-09-11T01:57:53.402 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36492.log.gz 2024-09-11T01:57:53.402 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37307.log 2024-09-11T01:57:53.402 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38049.log 2024-09-11T01:57:53.402 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.37307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50887.log 2024-09-11T01:57:53.402 INFO:teuthology.orchestra.run.smithi012.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.57417.log.gz 2024-09-11T01:57:53.402 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51621.log 2024-09-11T01:57:53.402 INFO:teuthology.orchestra.run.smithi012.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.37307.log.gz 2024-09-11T01:57:53.402 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.38049.log: /var/log/ceph/ceph-client.admin.50887.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50887.log.gz 90.4% 2024-09-11T01:57:53.402 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38049.log.gz 2024-09-11T01:57:53.403 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48847.log 2024-09-11T01:57:53.403 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51621.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-11T01:57:53.403 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58551.log 2024-09-11T01:57:53.403 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51621.log.gz 2024-09-11T01:57:53.403 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph.log: 89.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54329.log 2024-09-11T01:57:53.403 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48847.log.gz 2024-09-11T01:57:53.403 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58551.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58551.log.gz 2024-09-11T01:57:53.403 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44143.log 2024-09-11T01:57:53.403 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44271.log 2024-09-11T01:57:53.403 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54329.log: /var/log/ceph/ceph-client.admin.44143.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54329.log.gz 2024-09-11T01:57:53.403 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41877.log 2024-09-11T01:57:53.403 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44143.log.gz 2024-09-11T01:57:53.403 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44271.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44271.log.gz 2024-09-11T01:57:53.403 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53176.log 2024-09-11T01:57:53.404 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45930.log 2024-09-11T01:57:53.404 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.41877.log: /var/log/ceph/ceph-client.admin.53176.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.41877.log.gz 2024-09-11T01:57:53.404 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53176.log.gzgzip 2024-09-11T01:57:53.404 INFO:teuthology.orchestra.run.smithi012.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43073.log 2024-09-11T01:57:53.404 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45930.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.45930.log.gz 2024-09-11T01:57:53.404 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40920.log 2024-09-11T01:57:53.404 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58425.log 2024-09-11T01:57:53.404 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43073.log: /var/log/ceph/ceph-client.admin.40920.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.43073.log.gz 2024-09-11T01:57:53.404 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40920.log.gz 2024-09-11T01:57:53.404 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55011.log 2024-09-11T01:57:53.404 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42609.log 2024-09-11T01:57:53.404 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58425.log: /var/log/ceph/ceph-client.admin.55011.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58425.log.gz 2024-09-11T01:57:53.404 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37181.log 2024-09-11T01:57:53.404 INFO:teuthology.orchestra.run.smithi012.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.55011.log.gz 2024-09-11T01:57:53.405 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.42609.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.42609.log.gz 2024-09-11T01:57:53.405 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42861.log 2024-09-11T01:57:53.405 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38217.log 2024-09-11T01:57:53.405 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.37181.log: /var/log/ceph/ceph-client.admin.42861.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.37181.log.gz 2024-09-11T01:57:53.405 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.42861.log.gz 2024-09-11T01:57:53.405 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41623.log 2024-09-11T01:57:53.405 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35671.log 2024-09-11T01:57:53.405 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.38217.log: /var/log/ceph/ceph-client.admin.41623.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.38217.log.gz 2024-09-11T01:57:53.405 INFO:teuthology.orchestra.run.smithi012.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.41623.log.gz 2024-09-11T01:57:53.405 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45846.log 2024-09-11T01:57:53.405 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37099.log 2024-09-11T01:57:53.405 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.35671.log: /var/log/ceph/ceph-client.admin.45846.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35671.log.gz 2024-09-11T01:57:53.406 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45846.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50929.log 2024-09-11T01:57:53.406 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:53.406 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.37099.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37099.log.gz 2024-09-11T01:57:53.406 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41130.log 2024-09-11T01:57:53.406 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.50929.log: /var/log/ceph/ceph-client.admin.56559.log 2024-09-11T01:57:53.406 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.41130.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50929.log.gz 2024-09-11T01:57:53.406 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53342.log 2024-09-11T01:57:53.406 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56559.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41130.log.gz 2024-09-11T01:57:53.406 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56559.log.gz 2024-09-11T01:57:53.406 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54118.log 2024-09-11T01:57:53.406 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44440.log 2024-09-11T01:57:53.406 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54118.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.53342.log.gz 2024-09-11T01:57:53.406 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56433.log 2024-09-11T01:57:53.406 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54118.log.gz 2024-09-11T01:57:53.407 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44440.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.44440.log.gz 2024-09-11T01:57:53.407 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41088.log 2024-09-11T01:57:53.407 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56180.log 2024-09-11T01:57:53.407 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.41088.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56433.log.gz 2024-09-11T01:57:53.407 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- 90.2% -- replaced with /var/log/ceph/ceph-client.admin.41088.log.gz 2024-09-11T01:57:53.407 INFO:teuthology.orchestra.run.smithi012.stderr: /var/log/ceph/ceph-client.admin.45512.log 2024-09-11T01:57:53.407 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40668.log 2024-09-11T01:57:53.407 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56180.log: /var/log/ceph/ceph-client.admin.45512.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.56180.log.gz 2024-09-11T01:57:53.407 INFO:teuthology.orchestra.run.smithi012.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45512.log.gz 2024-09-11T01:57:53.407 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54843.log 2024-09-11T01:57:53.407 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.40668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49015.log 2024-09-11T01:57:53.407 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54843.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.40668.log.gz 2024-09-11T01:57:53.407 INFO:teuthology.orchestra.run.smithi012.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54843.log.gz 2024-09-11T01:57:53.408 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37773.log 2024-09-11T01:57:53.408 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48407.log 2024-09-11T01:57:53.408 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49015.log: /var/log/ceph/ceph-client.admin.37773.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49015.log.gz 2024-09-11T01:57:53.408 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42087.log 2024-09-11T01:57:53.408 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37773.log.gz 2024-09-11T01:57:53.408 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48407.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.48407.log.gz 2024-09-11T01:57:53.408 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40752.log 2024-09-11T01:57:53.408 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.42087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51537.log 2024-09-11T01:57:53.408 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.40752.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.42087.log.gz 2024-09-11T01:57:53.408 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43031.log 2024-09-11T01:57:53.408 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51537.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.40752.log.gz 2024-09-11T01:57:53.408 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39601.log 2024-09-11T01:57:53.409 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51537.log.gz 2024-09-11T01:57:53.409 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43031.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43031.log.gz 2024-09-11T01:57:53.409 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56223.log 2024-09-11T01:57:53.409 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47591.log 2024-09-11T01:57:53.409 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.39601.log: /var/log/ceph/ceph-client.admin.56223.log: 88.8% 90.7% -- replaced with /var/log/ceph/ceph-client.admin.39601.log.gz 2024-09-11T01:57:53.409 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56223.log.gz 2024-09-11T01:57:53.409 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46372.log 2024-09-11T01:57:53.409 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49454.log 2024-09-11T01:57:53.409 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47591.log: /var/log/ceph/ceph-client.admin.46372.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47591.log.gz 2024-09-11T01:57:53.409 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37433.log 2024-09-11T01:57:53.409 INFO:teuthology.orchestra.run.smithi012.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.46372.log.gz 2024-09-11T01:57:53.409 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37265.log 2024-09-11T01:57:53.409 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.37433.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.49454.log.gz 2024-09-11T01:57:53.410 INFO:teuthology.orchestra.run.smithi012.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.37433.log.gz 2024-09-11T01:57:53.410 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45678.log 2024-09-11T01:57:53.410 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.37265.log: /var/log/ceph/ceph-client.admin.49370.log 2024-09-11T01:57:53.410 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45678.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.37265.log.gz 2024-09-11T01:57:53.410 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36996.log 2024-09-11T01:57:53.410 INFO:teuthology.orchestra.run.smithi012.stderr: 89.9%/var/log/ceph/ceph-client.admin.49370.log: -- replaced with /var/log/ceph/ceph-client.admin.45678.log.gz 2024-09-11T01:57:53.410 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38175.log 2024-09-11T01:57:53.410 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49370.log.gz 2024-09-11T01:57:53.410 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37563.log 2024-09-11T01:57:53.410 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.36996.log: /var/log/ceph/ceph-client.admin.38175.log: 91.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-11T01:57:53.410 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56036.log 2024-09-11T01:57:53.410 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.36996.log.gz 2024-09-11T01:57:53.410 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.37563.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.38175.log.gz 2024-09-11T01:57:53.411 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40710.log 2024-09-11T01:57:53.411 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.56036.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43241.log 2024-09-11T01:57:53.411 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37563.log.gz 2024-09-11T01:57:53.411 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44608.log 2024-09-11T01:57:53.411 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.40710.log: 89.5%/var/log/ceph/ceph-client.admin.43241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49141.log 2024-09-11T01:57:53.411 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56036.log.gz 2024-09-11T01:57:53.411 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44608.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40710.log.gz 2024-09-11T01:57:53.411 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43241.log.gz 2024-09-11T01:57:53.411 INFO:teuthology.orchestra.run.smithi012.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.44608.log.gz 2024-09-11T01:57:53.411 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39727.log 2024-09-11T01:57:53.411 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.49141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44356.log 2024-09-11T01:57:53.411 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.39727.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.49141.log.gz 2024-09-11T01:57:53.411 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42567.log 2024-09-11T01:57:53.411 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39393.log 2024-09-11T01:57:53.412 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.39727.log.gz/var/log/ceph/ceph-client.admin.44356.log: 2024-09-11T01:57:53.412 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.42567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48973.log 2024-09-11T01:57:53.412 INFO:teuthology.orchestra.run.smithi012.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44356.log.gz 2024-09-11T01:57:53.412 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.39393.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.42567.log.gz 2024-09-11T01:57:53.412 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57501.log 2024-09-11T01:57:53.412 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39393.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52989.log 2024-09-11T01:57:53.412 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:53.412 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48973.log: /var/log/ceph/ceph-client.admin.57501.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57501.log.gz 2024-09-11T01:57:53.412 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48973.log.gz 2024-09-11T01:57:53.412 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42819.log 2024-09-11T01:57:53.412 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41709.log 2024-09-11T01:57:53.412 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.42819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51242.log 2024-09-11T01:57:53.412 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52989.log.gz 2024-09-11T01:57:53.413 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47549.log 2024-09-11T01:57:53.413 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.41709.log: 88.5%/var/log/ceph/ceph-client.admin.51242.log: -- replaced with /var/log/ceph/ceph-client.admin.42819.log.gz 2024-09-11T01:57:53.413 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52694.log 2024-09-11T01:57:53.413 INFO:teuthology.orchestra.run.smithi012.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51242.log.gz 2024-09-11T01:57:53.413 INFO:teuthology.orchestra.run.smithi012.stderr: 88.7%/var/log/ceph/ceph-client.admin.47549.log: -- replaced with /var/log/ceph/ceph-client.admin.41709.log.gz 2024-09-11T01:57:53.413 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46457.log 2024-09-11T01:57:53.413 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52694.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.47549.log.gz 2024-09-11T01:57:53.413 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52526.log 2024-09-11T01:57:53.413 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46457.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52694.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54885.log 2024-09-11T01:57:53.413 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:53.413 INFO:teuthology.orchestra.run.smithi012.stderr:gzip 89.9% -- replaced with /var/log/ceph/ceph-client.admin.46457.log.gz 2024-09-11T01:57:53.413 INFO:teuthology.orchestra.run.smithi012.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.57207.log 2024-09-11T01:57:53.414 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54885.log: /var/log/ceph/ceph-client.admin.52526.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27427.log 2024-09-11T01:57:53.414 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3%/var/log/ceph/ceph-client.admin.57207.log: -- replaced with /var/log/ceph/ceph-client.admin.54885.log.gz 2024-09-11T01:57:53.414 INFO:teuthology.orchestra.run.smithi012.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.52526.log.gz 2024-09-11T01:57:53.414 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48805.log 2024-09-11T01:57:53.414 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57207.log.gz 2024-09-11T01:57:53.414 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph.tmp-client.admin.27427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46962.log 2024-09-11T01:57:53.414 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27427.log.gz 2024-09-11T01:57:53.414 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.48805.log: -- /var/log/ceph/ceph-client.admin.35536.log 2024-09-11T01:57:53.414 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.46962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53115.log 2024-09-11T01:57:53.414 INFO:teuthology.orchestra.run.smithi012.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.48805.log.gz 2024-09-11T01:57:53.414 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.35536.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.46962.log.gz 2024-09-11T01:57:53.414 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45196.log 2024-09-11T01:57:53.414 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53846.log 2024-09-11T01:57:53.414 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35536.log.gz 2024-09-11T01:57:53.415 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.45196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38427.log 2024-09-11T01:57:53.415 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53115.log.gz 2024-09-11T01:57:53.415 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.45196.log.gz 2024-09-11T01:57:53.415 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44860.log 2024-09-11T01:57:53.415 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.38427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58047.log 2024-09-11T01:57:53.415 INFO:teuthology.orchestra.run.smithi012.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.53846.log.gz 2024-09-11T01:57:53.415 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48365.log 2024-09-11T01:57:53.415 INFO:teuthology.orchestra.run.smithi012.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.38427.log.gz 2024-09-11T01:57:53.415 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44860.log.gz 2024-09-11T01:57:53.415 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.58047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36786.log 2024-09-11T01:57:53.415 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48365.log: gzip 88.7% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.58047.log.gz -- 2024-09-11T01:57:53.415 INFO:teuthology.orchestra.run.smithi012.stderr: /var/log/ceph/ceph-client.admin.52298.log 2024-09-11T01:57:53.415 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40542.log 2024-09-11T01:57:53.416 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.36786.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48365.log.gz 2024-09-11T01:57:53.416 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36450.log 2024-09-11T01:57:53.416 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.40542.log: 89.3% 90.9% -- replaced with /var/log/ceph/ceph-client.admin.52298.log.gz 2024-09-11T01:57:53.416 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36786.log.gz 2024-09-11T01:57:53.416 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40542.log.gz 2024-09-11T01:57:53.416 INFO:teuthology.orchestra.run.smithi012.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52484.log 2024-09-11T01:57:53.416 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.36450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36134.log 2024-09-11T01:57:53.416 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51411.log 2024-09-11T01:57:53.416 INFO:teuthology.orchestra.run.smithi012.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.36450.log.gz 2024-09-11T01:57:53.416 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.36134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40290.log 2024-09-11T01:57:53.416 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51411.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.52484.log.gz 2024-09-11T01:57:53.416 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41298.log 2024-09-11T01:57:53.416 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% 88.8%/var/log/ceph/ceph-client.admin.40290.log: -- replaced with /var/log/ceph/ceph-client.admin.51411.log.gz 2024-09-11T01:57:53.417 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36134.log.gz 2024-09-11T01:57:53.417 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54591.log 2024-09-11T01:57:53.417 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.41298.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.40290.log.gz 2024-09-11T01:57:53.417 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48889.log 2024-09-11T01:57:53.417 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51663.log 2024-09-11T01:57:53.417 INFO:teuthology.orchestra.run.smithi012.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.41298.log.gz 2024-09-11T01:57:53.417 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48889.log: gzip 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54591.log.gz 2024-09-11T01:57:53.417 INFO:teuthology.orchestra.run.smithi012.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47381.log 2024-09-11T01:57:53.417 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50354.log 2024-09-11T01:57:53.417 INFO:teuthology.orchestra.run.smithi012.stderr: 90.4% 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51663.log.gz/var/log/ceph/ceph-client.admin.47381.log: 2024-09-11T01:57:53.417 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48889.log.gz 2024-09-11T01:57:53.417 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43849.log 2024-09-11T01:57:53.417 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47381.log.gz/var/log/ceph/ceph-client.admin.50354.log: 2024-09-11T01:57:53.418 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50803.log 2024-09-11T01:57:53.418 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36261.log 2024-09-11T01:57:53.418 INFO:teuthology.orchestra.run.smithi012.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.50354.log.gz 2024-09-11T01:57:53.418 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48679.log 2024-09-11T01:57:53.418 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50803.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43849.log.gz/var/log/ceph/ceph-client.admin.36261.log: 2024-09-11T01:57:53.418 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38702.log 2024-09-11T01:57:53.418 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48679.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.50803.log.gz 2024-09-11T01:57:53.418 INFO:teuthology.orchestra.run.smithi012.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.36261.log.gz 2024-09-11T01:57:53.418 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58005.log 2024-09-11T01:57:53.418 INFO:teuthology.orchestra.run.smithi012.stderr: 90.5%/var/log/ceph/ceph-client.admin.38702.log: -- replaced with /var/log/ceph/ceph-client.admin.48679.log.gz 2024-09-11T01:57:53.418 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57375.log 2024-09-11T01:57:53.418 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.58005.log: -- /var/log/ceph/ceph-client.admin.57165.log 2024-09-11T01:57:53.418 INFO:teuthology.orchestra.run.smithi012.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.38702.log.gz 2024-09-11T01:57:53.418 INFO:teuthology.orchestra.run.smithi012.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58005.log.gz/var/log/ceph/ceph-client.admin.57375.log: 2024-09-11T01:57:53.418 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54381.log 2024-09-11T01:57:53.419 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57165.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57165.log.gz 2024-09-11T01:57:53.419 INFO:teuthology.orchestra.run.smithi012.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.57375.log.gz 2024-09-11T01:57:53.419 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39559.log 2024-09-11T01:57:53.419 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53594.log 2024-09-11T01:57:53.419 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.39559.log: gzip -5 --verbose -- 88.8% /var/log/ceph/ceph-client.admin.48071.log 2024-09-11T01:57:53.419 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54381.log.gz 2024-09-11T01:57:53.419 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47717.log 2024-09-11T01:57:53.419 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.53594.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.39559.log.gz 2024-09-11T01:57:53.419 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48071.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.53594.log.gz 2024-09-11T01:57:53.419 INFO:teuthology.orchestra.run.smithi012.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.48071.log.gz 2024-09-11T01:57:53.419 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54203.log 2024-09-11T01:57:53.419 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.47717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42440.log 2024-09-11T01:57:53.419 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.54203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43283.log 2024-09-11T01:57:53.419 INFO:teuthology.orchestra.run.smithi012.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47717.log.gz 2024-09-11T01:57:53.420 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.42440.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.54203.log.gz 2024-09-11T01:57:53.420 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34573.log 2024-09-11T01:57:53.420 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41172.log 2024-09-11T01:57:53.420 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.43283.log: /var/log/ceph/ceph-client.admin.34573.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.42440.log.gz 2024-09-11T01:57:53.420 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51453.log 2024-09-11T01:57:53.420 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43283.log.gz 2024-09-11T01:57:53.420 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.41172.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34573.log.gz 2024-09-11T01:57:53.420 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40584.log 2024-09-11T01:57:53.420 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51453.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41172.log.gz 2024-09-11T01:57:53.420 INFO:teuthology.orchestra.run.smithi012.stderr:gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51453.log.gz 2024-09-11T01:57:53.420 INFO:teuthology.orchestra.run.smithi012.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42129.log 2024-09-11T01:57:53.420 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.40584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38786.log 2024-09-11T01:57:53.420 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.42129.log: 90.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42693.log 2024-09-11T01:57:53.420 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40584.log.gz 2024-09-11T01:57:53.421 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43409.log 2024-09-11T01:57:53.421 INFO:teuthology.orchestra.run.smithi012.stderr: 89.8%/var/log/ceph/ceph-client.admin.38786.log: -- replaced with /var/log/ceph/ceph-client.admin.42129.log.gz 2024-09-11T01:57:53.421 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.42693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52382.log 2024-09-11T01:57:53.421 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.38786.log.gz/var/log/ceph/ceph-client.admin.43409.log: 2024-09-11T01:57:53.421 INFO:teuthology.orchestra.run.smithi012.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42693.log.gz 2024-09-11T01:57:53.421 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45070.log 2024-09-11T01:57:53.421 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.52382.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.43409.log.gz 2024-09-11T01:57:53.421 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51747.log 2024-09-11T01:57:53.421 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57879.log 2024-09-11T01:57:53.421 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.52382.log.gz 2024-09-11T01:57:53.421 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51747.log: /var/log/ceph/ceph-client.admin.45070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34873.log 2024-09-11T01:57:53.421 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.57879.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45070.log.gz 2024-09-11T01:57:53.421 INFO:teuthology.orchestra.run.smithi012.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.51747.log.gz 2024-09-11T01:57:53.422 INFO:teuthology.orchestra.run.smithi012.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57879.log.gz 2024-09-11T01:57:53.422 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37038.log 2024-09-11T01:57:53.422 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.34873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48113.log 2024-09-11T01:57:53.422 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.37038.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42651.log 2024-09-11T01:57:53.422 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34873.log.gz 2024-09-11T01:57:53.422 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44398.log 2024-09-11T01:57:53.422 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.48113.log: /var/log/ceph/ceph-client.admin.42651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47213.log 2024-09-11T01:57:53.422 INFO:teuthology.orchestra.run.smithi012.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37038.log.gz 2024-09-11T01:57:53.422 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.44398.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.48113.log.gz 2024-09-11T01:57:53.422 INFO:teuthology.orchestra.run.smithi012.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42651.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50467.log 2024-09-11T01:57:53.422 INFO:teuthology.orchestra.run.smithi012.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.44398.log.gz 2024-09-11T01:57:53.422 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:53.422 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5/var/log/ceph/ceph-client.admin.47213.log: --verbose -- /var/log/ceph/ceph-client.admin.38870.log 2024-09-11T01:57:53.423 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.50467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39643.log 2024-09-11T01:57:53.423 INFO:teuthology.orchestra.run.smithi012.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.47213.log.gz 2024-09-11T01:57:53.423 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.38870.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50467.log.gz 2024-09-11T01:57:53.423 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51495.log 2024-09-11T01:57:53.423 INFO:teuthology.orchestra.run.smithi012.stderr: 89.9%/var/log/ceph/ceph-client.admin.39643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39685.log 2024-09-11T01:57:53.423 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38870.log.gz 2024-09-11T01:57:53.423 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.51495.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39643.log.gz 2024-09-11T01:57:53.423 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.39685.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.51495.log.gz 2024-09-11T01:57:53.423 INFO:teuthology.orchestra.run.smithi012.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.39685.log.gz 2024-09-11T01:57:53.462 INFO:teuthology.orchestra.run.smithi071.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-11T01:57:53.777 INFO:teuthology.orchestra.run.smithi071.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-11T01:57:53.870 INFO:teuthology.orchestra.run.smithi012.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-11T01:57:54.367 INFO:teuthology.orchestra.run.smithi012.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-11T01:57:56.145 INFO:teuthology.orchestra.run.smithi012.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-11T01:57:56.926 INFO:teuthology.orchestra.run.smithi071.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-11T01:57:57.005 INFO:teuthology.orchestra.run.smithi071.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-11T01:57:58.940 INFO:teuthology.orchestra.run.smithi012.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-11T01:57:58.997 INFO:teuthology.orchestra.run.smithi071.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-11T01:57:58.999 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-11T01:57:58.999 INFO:teuthology.orchestra.run.smithi071.stderr:real 0m5.910s 2024-09-11T01:57:58.999 INFO:teuthology.orchestra.run.smithi071.stderr:user 0m14.311s 2024-09-11T01:57:58.999 INFO:teuthology.orchestra.run.smithi071.stderr:sys 0m0.462s 2024-09-11T01:57:59.691 INFO:teuthology.orchestra.run.smithi012.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-11T01:57:59.693 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-09-11T01:57:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:real 0m6.633s 2024-09-11T01:57:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:user 0m17.162s 2024-09-11T01:57:59.693 INFO:teuthology.orchestra.run.smithi012.stderr:sys 0m0.896s 2024-09-11T01:57:59.693 INFO:tasks.ceph:Archiving logs... 2024-09-11T01:57:59.693 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/log/ceph to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7899900/remote/smithi012/log 2024-09-11T01:57:59.694 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T01:58:02.848 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/log/ceph to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7899900/remote/smithi071/log 2024-09-11T01:58:02.849 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T01:58:05.204 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-11T01:58:05.214 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-11T01:58:05.214 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T01:58:05.217 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T01:58:05.314 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-11T01:58:05.323 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-11T01:58:05.323 DEBUG:teuthology.orchestra.run.smithi012:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T01:58:05.326 DEBUG:teuthology.orchestra.run.smithi071:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T01:58:05.341 INFO:teuthology.orchestra.run.smithi071.stderr:bash: line 1: ntpq: command not found 2024-09-11T01:58:05.342 INFO:teuthology.orchestra.run.smithi012.stderr:bash: line 1: ntpq: command not found 2024-09-11T01:58:05.345 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T01:58:05.345 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2024-09-11T01:58:05.345 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 41 -280us[ -279us] +/- 43ms 2024-09-11T01:58:05.345 INFO:teuthology.orchestra.run.smithi071.stdout:^* hv02.front.sepia.ceph.com 3 6 377 39 +847us[ +849us] +/- 49ms 2024-09-11T01:58:05.345 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 42 -934us[ -933us] +/- 90ms 2024-09-11T01:58:05.345 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T01:58:05.346 INFO:teuthology.orchestra.run.smithi012.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T01:58:05.346 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================== 2024-09-11T01:58:05.346 INFO:teuthology.orchestra.run.smithi012.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 41 -328us[ -313us] +/- 43ms 2024-09-11T01:58:05.346 INFO:teuthology.orchestra.run.smithi012.stdout:^* hv02.front.sepia.ceph.com 3 7 377 41 +845us[ +860us] +/- 49ms 2024-09-11T01:58:05.346 INFO:teuthology.orchestra.run.smithi012.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 39 -967us[ -967us] +/- 90ms 2024-09-11T01:58:05.346 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T01:58:05.348 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-11T01:58:05.356 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-11T01:58:05.357 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-11T01:58:05.365 DEBUG:teuthology.orchestra.run.smithi012:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-11T01:58:05.470 DEBUG:teuthology.orchestra.run.smithi071:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-11T01:58:05.534 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T01:58:05.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T01:58:05.566 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T01:58:05.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T01:58:05.594 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-11T01:58:05.604 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-11T01:58:05.614 INFO:teuthology.task.internal:Duration was 2146.653472 seconds 2024-09-11T01:58:05.614 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-11T01:58:05.622 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-11T01:58:05.623 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T01:58:05.625 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T01:58:05.665 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T01:58:05.672 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T01:58:05.934 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-11T01:58:05.934 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi012.front.sepia.ceph.com 2024-09-11T01:58:05.935 DEBUG:teuthology.orchestra.run.smithi012:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T01:58:05.996 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi071.front.sepia.ceph.com 2024-09-11T01:58:05.997 DEBUG:teuthology.orchestra.run.smithi071:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T01:58:06.018 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-11T01:58:06.018 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T01:58:06.039 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T01:58:06.084 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-11T01:58:06.085 DEBUG:teuthology.orchestra.run.smithi012:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T01:58:06.107 DEBUG:teuthology.orchestra.run.smithi071:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T01:58:06.233 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-11T01:58:06.243 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-11T01:58:06.244 DEBUG:teuthology.orchestra.run.smithi012:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T01:58:06.269 DEBUG:teuthology.orchestra.run.smithi071:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T01:58:06.294 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-11T01:58:06.306 DEBUG:teuthology.orchestra.run.smithi012:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T01:58:06.312 DEBUG:teuthology.orchestra.run.smithi071:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T01:58:06.333 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = core 2024-09-11T01:58:06.358 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = core 2024-09-11T01:58:06.379 DEBUG:teuthology.orchestra.run.smithi012:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T01:58:06.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T01:58:06.412 DEBUG:teuthology.orchestra.run.smithi071:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T01:58:06.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T01:58:06.434 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-11T01:58:06.445 INFO:teuthology.task.internal:Transferring archived files... 2024-09-11T01:58:06.446 DEBUG:teuthology.misc:Transferring archived files from smithi012:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7899900/remote/smithi012 2024-09-11T01:58:06.446 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T01:58:06.488 DEBUG:teuthology.misc:Transferring archived files from smithi071:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7899900/remote/smithi071 2024-09-11T01:58:06.489 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T01:58:06.523 INFO:teuthology.task.internal:Removing archive directory... 2024-09-11T01:58:06.523 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T01:58:06.526 DEBUG:teuthology.orchestra.run.smithi071:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T01:58:06.572 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-11T01:58:06.586 INFO:teuthology.task.internal:Not uploading archives. 2024-09-11T01:58:06.586 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-11T01:58:06.596 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-11T01:58:06.596 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T01:58:06.599 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T01:58:06.612 INFO:teuthology.orchestra.run.smithi012.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 01:58 /home/ubuntu/cephtest 2024-09-11T01:58:06.627 INFO:teuthology.orchestra.run.smithi071.stdout: 266063 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 01:58 /home/ubuntu/cephtest 2024-09-11T01:58:06.628 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-11T01:58:06.639 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-11T01:58:06.661 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest} thrashers/one workloads/snaps-few-objects} duration: 2146.6534724235535 flavor: default owner: scheduled_hyelloji@teuthology success: true 2024-09-11T01:58:06.661 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T01:58:06.758 INFO:teuthology.run:pass